Making Elasticsearch Cluster Work In AWS Mumbai Region

Datetime:2016-08-23 02:08:03          Topic: Elastic Search           Share

July 31, 2016 by Aakash Agarwal | 0 comments

With the recently launched Mumbai region in AWS, many of us might need to migrate our infrastructure there. In one such scenario we had to setup Elasticsearch cluster on EC2s and test it in Mumbai region.

The Challenge

There is a plugin which enables Elasticsearch nodes to ‘discover’ each other on the basis of Region and form a cluster. It is ‘AWS Cloud Plugin for Elasticsearch https://github.com/elastic/elasticsearch/tree/master/plugins/discovery-ec2 ’. Once you have downloaded and setup the plugin, you need to configure the elasticsearch .yml file. The plugin segment should look similar to this:

 plugin.mandatory: cloud-aws
 cluster.name: mynewcluster
 cloud.aws.access_key: mykey
 cloud.aws.secret_key: mysecret
 cloud.aws.region: us-east-1
 discovery.type: ec2
 discovery.ec2.tag.elasticsearch: mynewcluster 

Here, we updated the cloud.aws.region to ‘ap-south-1’ (Mumbai) and restarted our Elasticsearch nodes. But, nodes could not be added to the cluster and there were errors which lead us to conclude that the plugin does not support Mumbai region yet .

 The Solution

Download/Clone the plugin and get ready to make some changes. Basically we want to update the source files and add support for Mumbai region. Once we have downloaded the plugin, we will search for the files that contain Regions in them for example:

$ grep -rnw ./ -e ap-southeast-1

We will get all the filenames and paths that contain region names and now we know where to make changes. To be precise there are three files viz.

S3Repository.java, InternalAwsS3Service.java, AwsEc2Service.java

There is a segment of code in all these files where ‘if’ operator checks for the Region and Endpoint and it is here where all the Regions and Endpoints are mentioned. Just add one more entry here for Mumbai region (ap-south-1) in all the files:

else if (region.equals("ap-south") || region.equals("ap-south-1")) {
endpoint = "ec2.ap-south-1.amazonaws.com";
}

That is it! We can now use Jenkins to do a Maven build on the plugin’s local git repository. Once done, we need to replace the old java files with newer ones in the existing plugin on each Elasticsearch node.

On restarting Elasticsearch we will see that the nodes are getting ‘discovered’ in the Region and are added to the cluster. Things work fine for Mumbai region just like any other. See-you with another write-up!  

(Special thanks to Nitin Bhadauria and Mohit Gupta for making this happen.)

Tag:





About List