How to install Graylog 1.x on Ubuntu 14.04.

Introduction

In this tutorial, we will show how to install Graylog v1.3.x (sometimes called Graylog2) on Ubuntu 14.04 and configure it to collect the system's syslog in a centralized location. Graylog is a powerful log management and analysis tool with many use cases, from monitoring SSH logins and abnormal activity to debugging applications. It is based on Elasticsearch, Java and MongoDB.

You can use Graylog to collect and monitor various logs, but we limit the scope of this tutorial to syslog collection. Also, because we are demonstrating the basics of Graylog, we will install all the components on a single server.

About Graylog Components

Graylog has four main components:

• Graylog Server node: Used as a worker to receive and process messages, and communicate with all other non-server components. Its performance depends on the CPU
• Elasticsearch node: Store all logs/messages. Its performance depends on RAM and disk I/O.
• MongoDB: Store metadata without experiencing too much load
• Web interface: User interface

The following is a schematic diagram of Graylog components (please note that messages are sent from other servers):

This tutorial will implement a very basic Graylog setup, all components are installed on the same server. For larger production settings, for performance reasons, it is recommended to install the components on different servers.

Course Preparation

The setup described in this tutorial requires an Ubuntu 14.04 server with at least 2GB RAM. You also need root access. Students who don’t have a server can buy it from here, but I personally recommend you to use the free Tencent Cloud Developer Lab to experiment and learn to install Then Buy Server.

If you use a VPS with less than 2GB RAM, you cannot start all Graylog components.

Let's install the software!

Install MongoDB

MongoDB installation is simple and quick. Run the following command to import the MongoDB public GPG key into apt:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

Create a MongoDB source list:

echo "deb http://repo.mongodb.org/apt/ubuntu "$(lsb_release -sc)"/mongodb-org/3.0 multiverse"| sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list

Update your apt package database:

sudo apt-get update

Install the latest stable version of MongoDB using the following command:

sudo apt-get install mongodb-org

MongoDB should now be up and running. Let's continue to install Java.

Install Java

Elasticsearch requires Java, so we will install it now. We will install Oracle Java 8 because it is recommended by Elastic. However, if you decide to go this route, it should work with OpenJDK.

sudo add-apt-repository ppa:webupd8team/java

Update your apt package database:

sudo apt-get update

Use this command to install the latest stable version of Oracle Java 8 (and accept the pop-up license agreement):

sudo apt-get install oracle-java8-installer

Now that Java is installed, let's install Elasticsearch.

Install Elasticsearch

Graylog 1.x is only available for version 2.0 of Elasticsearch, so we will install Elasticsearch 1.7.x. By adding Elastic's package source list, Elasticsearch can be installed with the package manager.

Run the following command to import the Elasticsearch public GPG key into apt:

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

If your prompt just hangs there, it may be waiting for your user password (authorize the sudo command). If this is the case, please enter your password.

echo "deb http://packages.elastic.co/elasticsearch/1.7/debian stable main"| sudo tee -a /etc/apt/sources.list.d/elasticsearch-1.7.x.list

Update your apt package database:

sudo apt-get update

Install Elasticsearch using the following command:

sudo apt-get-y install elasticsearch

Elasticsearch is now installed. Let's edit the configuration:

sudo vi /etc/elasticsearch/elasticsearch.yml

Find the specified cluster.name part. Uncomment and replace the default value with "graylog-development" as shown below:

cluster.name: graylog-development

You need to restrict external access to the Elasticsearch instance (port 9200), so outsiders cannot read your data through HTTP API or shut down your Elasticsearch cluster. Find the specified network.host line, uncomment it, and replace its value with "localhost" so that it looks like this:

network.host: localhost

Save and exit elasticsearch.yml.

Now start Elasticsearch:

sudo service elasticsearch restart

Then run the following command to start Elasticsearch at startup:

sudo update-rc.d elasticsearch defaults 9510

After a while, run the following command to test whether Elasticsearch is working properly:

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

Now that Elasticsearch is up and running, let's install the Graylog server.

Install Graylog Server

Now that we have installed the other required software, let's install Graylog's server component graylog-server.

First, download the Graylog Debian package to your home directory using the following command:

cd ~
wget https://packages.graylog2.org/repo/packages/graylog-1.3-repository-ubuntu14.04_latest.deb

Then use the following command to add the package to the package manager:

sudo dpkg -i graylog-1.3-repository-ubuntu14.04_latest.deb

Then install the graylog-server package using the following command:

sudo apt-get update
sudo apt-get install apt-transport-https
sudo apt-get install graylog-server

Install pwgen, we will use it to generate cryptographic keys:

sudo apt-get install pwgen

Now we must configure the administrator password and key. The password key is configured by the password_secret parameter in server.conf. We can generate a random key and insert it into the Graylog configuration using the following two commands:

SECRET=$(pwgen -s 961)
sudo -E sed -i -e 's/password_secret =.*/password_secret = '$SECRET'/'/etc/graylog/server/server.conf

By creating a password required to specify shasum and assigning it to the parameter in the Graylog configuration file described in root_password_sha2, the administrator password can be assigned. Use the following command to create a shasum of the required password, replacing the highlighted "password" with your own password. The sed command inserts it into the Graylog configuration for you:

PASSWORD=$(echo -n password | shasum -a 256| awk '{print $1}')
sudo -E sed -i -e 's/root_password_sha2 =.*/root_password_sha2 = '$PASSWORD'/'/etc/graylog/server/server.conf

Now that the administrator password is set, let's open the Graylog configuration and make some changes:

sudo vi /etc/graylog/server/server.conf

Because of the commands you ran in the above steps, you should see that password_secret and root_password_sha2 will have random strings.

Now we will configure rest_transport_uri, which is how the Graylog web interface communicates with the server. Because we want to install all components on a single server, let's set the value to 127.0.0.1, or localhost. Find and uncomment rest_transport_uri, and change its value so that it looks like this:

/etc/graylog/server/server.conf - 1/4

rest_transport_uri = http://127.0.0.1:12900/

Next, because we only have one Elasticsearch shard (running on this server), we change the value of elasticsearch_shards to 1:

elasticsearch_shards =1

Next, change the value of elasticsearch_cluster_name to "graylog-development" (same as Elasticsearch cluster.name):

elasticsearch_cluster_name = graylog-development

Uncomment these two lines to use unicast instead of multicast to discover Elasticsearch instances:

/etc/graylog/server/server.conf - 4 of 4

elasticsearch_discovery_zen_ping_multicast_enabled =false
elasticsearch_discovery_zen_ping_unicast_hosts =127.0.0.1:9300

Save and exit. Now graylog-server is configured and ready to start.

Use the service command to start the Graylog server:

sudo start graylog-server

The next step is to install the Graylog web interface. Let's do it now!

Install Graylog Web

Install Graylog Web using the following command:

sudo apt-get install graylog-web

Next, we need to configure the key of the web interface, which is the application.secret parameter in web.conf. We will generate another key as we did with the Graylog server configuration and insert it using sed as follows:

SECRET=$(pwgen -s 961)
sudo -E sed -i -e 's/application\.secret=""/application\.secret="'$SECRET'"/'/etc/graylog/web/web.conf

Now use the following command to open the web interface configuration file:

sudo vi /etc/graylog/web/web.conf

Now we need to update the configuration of the web interface to specify the graylog2-server.uris parameter. This is a comma separated list of server REST URIs. Since we only have one Graylog server node, this value should match the value in the rest_listen_uri Graylog server configuration.

/etc/graylog/web/web.conf excerpt

graylog2-server.uris="http://127.0.0.1:12900/"

The Graylog web interface is now configured. Start the Graylog web interface:

sudo start graylog-web

Now we can use the Graylog web interface. We will do it now.

Configure Graylog to receive syslog messages

Log in to Graylog web interface

In your favorite web browser, go to the 9000 port of the server's public IP address:

In a web browser:http://graylog_public_IP:9000/

You should see a login screen. Enter admin the username and administrator password you set earlier.

After logging in, you will see the following:

The red number at the top is the notification. If you click it, you will see a message stating that you have a node without any run input. Let's add an input to receive syslog messages on UDP.

Create Syslog UDP input

To add input to receive system log messages, click the System drop-down list in the top menu.

Now, select Inputs from the drop-down menu.

Select Syslog UDP from the drop-down menu, and then click the Launch new input button.

The "Start new input: Syslog UDP" mode window will pop up. Enter the following information (replace the binding address in the private IP address of the server):

• **Title: **syslog
• **Port: **8514
• **Bind address: **graylog_private_IP

Then click Launch.

You should now see an input named "syslog" in the Local inputs section (it should have a green box with "running" next to it), as shown below:

Now, our Graylog server is ready to receive system log messages on port 8514 from your server. Let us configure your server to send their system log messages to Graylog immediately.

Configure Rsyslog to send Syslog to Graylog Server

On all client servers, to send system log messages to Graylog's server, follow the steps below.

Create the rsyslog configuration file in /etc/rsyslog.d. We will call 90-graylog.conf:

sudo vi /etc/rsyslog.d/90-graylog.conf

In this file, add the following line to configure rsyslog to send system log messages to the Graylog server (replace graylog_private_IP with the private IP address of the Graylog server):

/etc/rsyslog.d/90-graylog.conf

$template GRAYLOGRFC5424,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msg%\n"*.* @graylog_private_IP:8514;GRAYLOGRFC5424

Save and exit. From now on, this file will be loaded as part of the rsyslog configuration. Now, you need to restart rsyslog for the changes to take effect.

sudo service rsyslog restart

After completing the rsyslog configuration on all servers to be monitored, please return to the Graylog web interface.

View your Graylog source

In your favorite web browser, go to the 9000 port of the server's public IP address:

In a web browser:http://graylog_public_IP:9000/

Click on "Source" in the top bar. You will see a list of all servers that have been configured with rsyslog.

The host name of the source is on the left, and the number of messages received by Graylog is on the right.

Search your Graylog data

After letting Graylog collect mail for a period of time, you will be able to search mail. For example, let's search for "sshd" and see what kind of SSH activity is happening on our server. Here is a snippet of our results:

As you can see, our example search results show sshd logs of various servers, as well as many failed root login attempts. Your results may vary, but it can help you identify many problems, including how unauthorized users are trying to access your server.

In addition to the basic search function on all sources, you can also search logs of a specific host or logs within a specific time range.

For example, if you want to view the logs of a server or multiple servers after an event, searching for data in Graylog is very useful. Centralized logging makes it easier to correlate related events, because you can view all events that have occurred without logging in to multiple servers.

For more information on how the search bar works, please check the official document: Graylog Search

in conclusion

Now that you have set up Graylog, please feel free to browse the other features it provides. You can send other types of logs to Graylog and set up an extractor (or use software such as logstash to reformat the logs) to make the logs more structured and searchable. You can also expand the Graylog environment by separating components and adding redundancy to improve performance and availability.

For more Ubuntu tutorials, please go to [Tencent Cloud + Community] (https://cloud.tencent.com/developer?from=10680) to learn more.

Reference: "How To Install Graylog 1.x on Ubuntu 14.04"