How to Install the ELK Stack on Google Cloud Platform

Datetime:2016-08-23 02:07:52          Topic: Elastic Search           Share

In this article, I will guide you through the process of installing the ELK Stack (Elasticsearch 2.2.x, Logstash 2.2.x and Kibana 4.4.x) on Google Cloud Platform (GCP).

While still lagging far behind Amazon Web Services, GCP is slowly gaining popularity, especially among early adopters and developers but also among a number of enterprises. Among the reasons for this trend are the full ability to customize virtual machines before provisioning them, positive performance benchmarking compared to other cloud providers, and overall reduced cost.

These reasons caused me to test the installation of the world’s most popular open source log analysis platform, theELK Stack, on this cloud offering. The steps below describe how to install the stack on a vanilla Ubuntu 14.04 virtual machine and establish an initial pipeline of system logs. Don’t worry about the costs of testing this workflow — GCP offers a nice sum of $300 for a trial (but don’t forget to delete the VM once you’re done!).

Setting up your environment

For the purposes of this article, I launched an Ubuntu 14.04 virtual machine instance in GCP’s Compute Engine. I enabled HTTP/HTTPS traffic to the instance and changed the default machine type to 7.5 GB.

Also, I created firewall rules within the Networking console to allow incoming TCP traffic to Elasticsearch and Kibana ports 9200 and 5601 respectively.

Installing Java

All of the packages we are going to install require Java, so this is the first step we’re going to describe (skip to the next step if you’ve already got Java installed).

Use this command to install Java:

$ sudoapt-getinstalldefault-jre

Verify that Java is installed:

$ java -version

If the output of the previous command is similar to this, you’ll know that you’re on track:

javaversion "1.7.0_101"OpenJDKRuntimeEnvironment (IcedTea 2.6.6) (7u101-2.6.6-0ubuntu0.14.04.1)OpenJDK 64-BitServerVM (build 24.95-b01, mixedmode)

Installing Elasticsearch

Elasticsearch is in charge of indexing and storing the data shipped from the various data sources, and can be called the “heart” of the ELK Stack.

To begin the process of installing Elasticsearch, add the following repository key:

$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Add the following Elasticsearch list to the key:

$ echo "deb http://packages.elastic.co/elasticsearch/1.7/debian stable main" | sudotee -a /etc/apt/sources.list.d/elasticsearch-1.7.list
 
$ sudoapt-getupdate

And finally, install:

$ sudoapt-getinstallelasticsearch

Before we start the service, we’re going to open the Elasticsearch configuration file and define the host on our network:

$ sudovi /etc/elasticsearch/elasticsearch.yml

In the Network section of the file, locate the line that specifies the ‘ network.host ’, uncomment it, and replace its value with “0.0.0.0”:

network.host: 0.0.0.0

Last but not least, restart the service:

$ sudoserviceelasticsearchrestart

To make sure that Elasticsearch is running as expected, issue the following cURL:

$ curllocalhost:9200

If the output is similar to the output below, you will know that Elasticsearch is running properly:

{
   "name" : "Hannah Levy",
   "cluster_name" : "elasticsearch",
   "version" :
      {     "number" : "2.3.4",
             "build_hash" : "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f",
             "build_timestamp" : "2016-06-30T11:24:31Z",
             "build_snapshot" : false,
             "lucene_version" : "5.5.0"
         },
   "tagline" : "You Know, for Search"
}

Production tip: DO NOT open any other ports, like 9200, to the world! There are bots that search for 9200 and execute groovy scripts to overtake machines.

Logstash Installation

Moving on, it’s time to install Logstash — the stack’s log shipper.

Using Logstash to parse and forward your logs into Elasticsearch is, of course, optional. There are other log shippers that can output to Elasticsearch directly, such as Filebeat and Fluentd, so I would recommend some research before you opt for using Logstash.

Since Logstash is available from the same repository as Elasticsearch and we have already installed that public key in the previous section, we’re going to start by creating the Logstash source list:

$ echo 'deb http://packages.elastic.co/logstash/2.2/debian stable main' | sudotee /etc/apt/sources.list.d/logstash-2.2.x.list

Next, we’re going to update the package database:

$ sudoapt-getupdate

Finally — we’re going to install Logstash:

$ sudoapt-getinstalllogstash

To start Logstash, execute:

$ sudoservicelogstashstart

And to make sure Logstash is running, use:

$ sudoservicelogstashstatus

The output should be:

logstashis running

We’ll get back to Logstash later to configure log shipping into Elasticsearch.

Kibana Installation

The process for installing Kibana, ELK’s pretty user interface, is identical to that of installing Logstash.

Create the Kibana source list:

$ echo "deb http://packages.elastic.co/kibana/4.4/debian stable main" | sudotee -a /etc/apt/sources.list.d/kibana-4.4.x.list

Update the apt package database:

$ sudoapt-getupdate

Then, install Kibana with this command:

$ sudoapt-get -y installkibana

Kibana is now installed.

We now need to configure the Kibana configuration file at /opt/kibana/config/kibana.yml :

$ sudovi /opt/kibana/config/kibana.yml

Uncomment the following lines:

server.port: 5601
 server.host: “0.0.0.0”

Last but not least, start Kibana:

$ sudoservicekibanastart

You should be able to access Kibana in your browser at http://<serverIP>:5601/ like this:

By default, Kibana connects to the Elasticsearch instance running on localhost, but you can connect to a different Elasticsearch instance instead. Simply modify the Elasticsearch URL in the Kibana configuration file that we had edited earlier ( /opt/kibana/config/kibana.yml ) and then restart Kibana.

If you cannot see Kibana, there is most likely an issue with GCP networking or firewalls. Please verify the firewall rules that you defined in GCP’s Networking console .

Establishing a pipeline

To start analyzing logs in Kibana, at least one Elasticsearch index pattern needs to be defined (you can read more about Elasticsearch concepts ) — and you will notice that since we have not yet shipped any logs, Kibana is unable to fetch mapping (as indicated by the grey button at the bottom of the page).

Our last and final step in this tutorial is to establish a pipeline of logs, in this case system logs, from syslog to Elasticsearch via Logstash.

First, create a new Logstash configuration file:

$ sudovim /etc/logstash/conf.d/10-syslog.conf

Use the following configuration:

input {
   file {
      type => “syslog”
      path => [  “/var/log/messages”, “/var/log/*.log”]
   }
}
filter {}
output {
   stdout {
      codec => rubydebug
   }
   elasticsearch {
      hosts => “localhost”
   }
}

A few words on this configuration.

Put simply, we’re telling Logstash to store the local syslog file ‘ /var/log/syslog ’ and all the files under ‘ /var/log*.log ’ on Elasticsearch.

The input section specifies which files to collect (path) and what format to expect (syslog). The output section uses two outputs – stdout and elasticsearch.

I left the filter section empty in this case, but usually this is where you would define rules to beautify the log messages usingLogstash plugins such as grok. Learn more aboutLogstash grokking.

The stdout output is used to debug Logstash, and the result is nicely-formatted log messages under ‘ /var/log/logstash/logstash.stdout ’. The Elasticsearch output is what actually stores the logs in Elasticsearch.

Please note that in this example I am using ‘ localhost ’ as the Elasticsearch hostname. In a real production setup, however, it is recommended to have Elasticsearch and Logstash installed on separate machines so the hostname would be different.

Next, run Logstash with this configuration:

$ /opt/logstash/bin/logstash -f /etc/logstash/conf.d/10-syslog.conf

You should see JSON output in your terminal indicating Logstash is performing as expected.

Refresh Kibana in your browser, and you’ll notice that the Create button is now green, meaning Kibana has found an Elasticsearch index. Click it to create the index and select the Discover tab.

Your logs will now begin to appear in Kibana:

Last, but not least

Installing ELK on GCP was smooth going — even easy — compared to AWS. Of course, as my goal was only to test installation and establish an initial pipeline. I didn’t stretch the stack to its limits. Logstash and Elasticsearch can cave under heavy loads, and the challenge, of course, is scaling and maintaining the stack on the long run. In a future post, I will compare the performance of ELK on GCP versus AWS.

Daniel Berman

Daniel Berman is Product Evangelist at Logz.io. He is passionate about log analytics, big data, cloud, and family and loves running, Liverpool FC, and writing about disruptive tech stuff.





About List