Getting started with Elasticsearch and Node.js - Part5

Datetime:2016-08-23 02:07:18          Topic: Elastic Search  Node.js           Share

In the final step of the series, we create a web application to access the Elasticsearch managed data and show how to host the web app on IBM's Bluemix.

In theprevious article we ran some queries on the nested fields in our petitions data. In this article - the last in the series - we're going to turn our existing code into a fully-fledged app and deploy it using IBM Bluemix . To get an idea of what you'll have by the end of this article, check out our own Petitioneering app.

Before we get to that we'll add in the postcode lookup we promised at the end of the last article. We'll need the get-json library for this so install it before you start.

npm install get-json

We'll also move the search query into a separate file which we'll call from nestedQuery.js . This will set us up nicely for when we are putting our application together since it will allow us to keep our functions separate from our logic.

Start by creating a new file and call it functions.js . Add in the following lines to connect to Elasticsearch and use the get-json library.

var client = require ('./connection.js');  
var getJSON = require('get-json');

Now copy your search function from nestedQuery.js into functions.js . We need to modify it very slightly because of the way we'll be calling it from now on so change

var results = function(constitLookup) {

to

function results(constitLookup,callback) {

To allow nestedQuery.js to use this function we need to export it using module.exports. Add this to the end of functions.js :

module.exports = {  
  results: results
};

Back in nestedQuery.js we need to add a require statement to use functions.js and tweak how we call the results function. Replace everything in nestedQuery.js with the following:

var argv = require('yargs').argv;  
var functions = require('./functions.js');

if (argv.search) {  
  functions.results(argv.search, function(results) {
      console.log(results);
  });
}

Run nestedQuery.js to test everything looks ok, and when you're happy we'll move on to adding the postcode lookup.

Adding a postcode lookup

Not everyone knows the exact name of their constituency (easy enough to type in when it's Ipswich, not so much if you live in the constituency of "Inverness, Nairn, Badenoch and Strathspey") so to make it easier for a user to find out what petitions are popular where they live we'll add a lookup that works out a user's constituency from their postcode. For this, we'll use the API from http://postcodes.io/ . First we'll use their validate method to check we've got a valid postcode, and then we'll use their postcode method to establish the constituency name. We'll then pass this into our results function.

In functions.js add the following:

function getConstituency(postcode,callback) {  
  getJSON('https://api.postcodes.io/postcodes/'+postcode, function(error, response){
    if(error) {
      console.log(error);
    }
    else {
      results(response.result.parliamentary_constituency,function(response){
        callback(response);
      });
    }
  });
}

function validatePostcode(postcode, callback) {  
  getJSON('https://api.postcodes.io/postcodes/'+postcode+'/validate',function(error,response){
      if(response.result){
        getConstituency(postcode,function(response){
          callback(response);
        });
      }
      else {
        console.log("Please enter a valid postcode");
      }
  });
}

function getResults(userinput, cb) {  
  var results = validatePostcode(userinput,function(response){
    cb(response);
  });
}

We also need to add getResults to our module exports so we can call it from nestedQuery.js :

module.exports = {  
  getResults: getResults,
  results: results
};

Finally, we just need to add a function to nestedQuery.js to invoke the postcode lookup when required. It shouldn't be too much of a surprise that it looks a lot like our existing search function:

if (argv.postcode) {  
  functions.getResults(argv.postcode, function(results) {
      console.log(results);
  });
}

Now you can run nestedQuery.js by supplying it with a postcode, like so:

node nestedQuery --postcode="KA22 8NG"

Later we'll be able to drop this code right into our application: we'll need to modify how we get the search term to our function and we'll turn the results into some html we can output but the workflow will be essentially the same. Before we get to that stage, though, we need to create a Bluemix account and set up our application.

Creating the app in Bluemix

If you don't already have a Bluemix account, you can quickly get started by signing up for a free 30-day trial . After you've registered and confirmed your email, log in to your account and create your first organization when prompted. We've called ours petitioneering . Create a space and get ready to create your app.

From your Console choose Compute from the menu, and Cloud Foundry Applications . We'll be creating a web app using the SDK for Node.js so choose that, give your app a name and click Create . At this point Bluemix will start staging your app, and if you return to your Console again you'll see you have one item under Compute . As soon as Bluemix completes the staging you can click Open URL to see the default home page for your app and verify that your app is now running.

Next we need to connect our Elasticsearch index to our app. From your console, choose Data & Analytics . Click the icon to create a new service and choose Elasticsearch by Compose . Give your service an appropriate name, and in the Connect to field select the Node.js app you just created. The values you need to enter for Username, Password and Public hostname/Port are the same values that you've been using in connection.js all throughthis series.

Click create and restage your app when prompted by Bluemix.

Developing your app

Now it's time to download your app and start dropping in some code. Go to your app's Getting Started page, and download the CF Command Line Interface. Download your starter code and follow the rest of the instructions on the page to make sure your basic configuration is set up correctly.

When you're happy with the basic app, it's time to start adding our code. The first task is just to copy your functions.js file from earlier in this article into your application directory. Most of the app's functions are already in functions.js - we just need to pass user input into them and format the output for displaying in a web page.

Now, as well as entering their postcode to get information on their own constituency, a user might be interested in the results from any other constituency, so let's add a select box that allows them to do that. To populate the select box we'll use the output from a new Elasticsearch query that returns the constituency names from the documents in our constituencies index.

Add this to functions.js :

function getConstituencies(callback){  
  client.search({
    index: 'gov',
    type: 'constituencies',
    size: 650,
    fields: 'constituencyname',
    body: {
      sort:
        {
          "constituencyname": {
            order: "asc"
          }
        }
    }
  },function (error, response,status) {
      if (error){
        console.log("search error: "+error)
      }
      if (response){
        var constitList = [];
        response.hits.hits.forEach(function(hit){
          constitList.push(hit.fields.constituencyname);
        })
        callback(constitList.sort());
      }
      else {
        console.log("<p>No results</p>");
      }
  });
}

This function will be called when the app's homepage is loaded. That will happen over in app.js , so we'll need to export this function, along with getresults and results . To do that, we need to update module.exports in functions.js again:

module.exports = {  
  getResults: getResults,
  getConstituencies: getConstituencies,
  results: results
};

Back over in app.js , we need to require our new functions.js file so we can pass user input into it as function arguments:

var functions = require('./functions.js');

When we want to use any of the functions from functions.js we reference them by their name in module.exports :

functions.getResults(argv.postcode, function(results) {  
  console.log("results output...");
    console.log(results);
});

Now it's time to tell our application what to do when a user arrives at the homepage. We're going to use that getConstituencies function to populate a select box so users check the results from any constituency without having to know a valid postcode. Add the following to app.js :

app.get('/', function(request, response) {  
  functions.getConstituencies(function(constituencyList){
    if(constituencyList){
      response.render('index', {
          constituencies: constituencyList
      });
    }
  });
});

This fires when a request is made for the app's index page, and passes the response from getConstituencies (which will be a list of the 650 UK constituencies in our Elasticsearch index) to whatever is going to render our web page.

On the subject of rendering web pages, now might be a good time to introduce Pug, our template engine for this app.

Pug (the template engine formerly known as Jade)

You'll have probably noticed that our basic code from Bluemix uses express.js as its web application framework. To make it even easier (or harder, depending on your viewpoint) to create our html pages we're going to use a template engine as well. We've chosen to use Pug , but express supports many others as well.

Pug allows us to generate static html pages using templates that we define, and variables in those templates that we can pass values into. Starting with a base layout, which we'll use to define a header and a footer for every page, we can then insert different blocks of content depending on which page we want to display to the user. Our app only really has one page, but it should give you an idea of how you could extend it.

Let's add the following to app.js to tell our app to use Pug and specify the directory where it can find the templates (also known as views):

app.set('view engine', 'pug');  
app.set('views', __dirname + '/public/views');

And let's define our base layout. Save this in your app folder as public/views/layout.jade :

doctype html  
html  
    head
        title='Petitioneering'
        link(rel='stylesheet', href='/stylesheets/style.css')
        link(rel='stylesheet', href='/stylesheets/petitioneering.css')
    body
        block content
        script(src='http://ajax.googleapis.com/ajax/libs/jquery/2.0.3/jquery.min.js')
        script(src='/javascripts/main.js')
    footer
        block footer
          div#footer
            div#footer-content
              p
                | Petitioneering uses open data from
                a(href='https://petition.parliament.uk') UK Government and Parliament
                | , indexed in
                a(href='https://elastic.co') Elasticsearch
                | , hosted by
                a(href='https://compose.com') Compose
              p
                | Application source available from
                a(href='https://github.com/compose-ex/petitioneering') Github

We've defined a fairly straightforward html page, with <head>, <body> and <footer> sections. We've added a couple of links to css and JavaScript files, and added some footer content. Note the indenting in the file and make sure it's preserved when in your file. Pug uses this indenting to generate the right hierarchy and nesting in its generated html. So, on our page, the footer will look like this:

<footer>  
  <div id="footer">
    <div id="footer-content">
      <p>Petitioneering uses open data from <a href="https://petition.parliament.uk">UK Government and Parliament</a>, indexed in  <a href="https://elastic.co">Elasticsearch</a>, hosted by <a href="https://compose.com">Compose</a></p>
      <p>Source available from <a href="https://github.com/compose-ex/petitioneering">Github</a></p>
    </div>
  </div>
</footer>

The other key element in our layout is block content . Pug allows layouts to be extended; to extend this layout we need to create a new file, specifying that it extends layout and providing something for block content . We can do this by creating public/views/index.jade with the following:

extends layout

block content  
  div#header
    div#title
      h1 Petitioneering
  div#main
    div#content
      div#search-div
        p Want to know what really matters to voters in your constituency? Type your postcode into the search box to find out...
        h2 Search by Postcode
        div#intro-div
        form
        input(id='postcode',name='postcode',type='text', placeholder="Enter postcode")
        span(id='pcbutton', value="postcode") Submit
        h2 Search by Constituency
        form
          select(id='constitlist')
            option(value='')
            each constituency, c in constituencies
              option(value=constituency) #{constituency}
      div#results-div
        div#results

This will give us our search box for users to type in their postcode, and a select box containing the names of all our constituencies. Remember our function to render the home page?

res.render('index', {  
    constituencies: response
});

That constituencies variable is what's being passed into this template:

each constituency, c in constituencies  
  option(value=constituency) #{constituency}

At this point we're almost ready to check on our progress by running the app locally again. Before doing that we have a little housekeeping: first, delete index.html from the /public directory so that our layouts are used. Next, we need to install all the new libraries to run the app locally:

npm install express cfenv elasticsearch get-json pug

Don't worry too much about the style at this point - as long as you have a page with a heading of 'Petitioneering' and sections for 'Search by Postcode' and 'Search by Constituency' we're doing fine.

If you want to make it look a bit prettier you can download the CSS from our Github repo or create your own.

You might notice if you try to enter a postcode or select a constituency that a whole heap of nothing happens. That's because we haven't yet told our app what to do when a user actually does something on this home page. For that we need to add some client-side JavaScript to respond to events and some functions to handle those events and return data to that JavaScript.

First, the JavaScript. We'll define a quick function that responds to a click of the 'Submit' button or a change in the value of the select box (which will happen when a user chooses a different constituency from the list). We'll then send the postcode or constituency variable over to a page called 'search'.

$(function() {
  $('#pcbutton').click(function(){
    var parameters = { postcode: $('#postcode').val() };
      $.get( '/search',parameters, function(data) {
        $('#results').html(data);
      });
  });
  $('#constitlist').change(function(){
    var parameters = { constituency: $('#constitlist').val() };
      $.get( '/search',parameters, function(data) {
        $('#results').html(data);
      });
  });
});

Now we need to tell our application to do something when these events are triggered. Back in app.js we can tell Express to respond to a request for the 'search' page using app.get like we did when we generated the select box when the homepage was loaded.

app.get('/search', function(request, response) {  
  if (!request.query.postcode && !request.query.constituency) {
    response.send("<p>Please enter a postcode or parliamentary constituency</p>");
  }
  if (request.query.postcode){
    functions.getResults(request.query.postcode, function(htmllist) {
      response.send(htmllist);
    });
  }
  if (request.query.constituency) {
    functions.results(request.query.constituency,function(htmllist){
      response.send(htmllist);
    });
  }
});

Add that to app.js and stop and then re-run your app. Pop a valid postcode into the search box and click Submit or choose a constituency from the list and check the output in your terminal. You should be seeing results like you did earlier in the article when running nestedQuery.js . At the moment we're getting input from main.js in the form of a postcode or constituency, but we're not passing that back in the form of a response, so we're not yet sending any html back to main.js for outputting.

We need to turn the response from Elasticsearch into html and pass that back to our client-side JavaScript, which will insert it into the page. To do this we need to tweak our results function so that instead of outputting content to the console it sends it to a new function that will create html, which will then get passed back down the callback chain and into main.js where express picks it up. Using res.send. So here's a function we're going to call to do just that. Add it to functions.js :

function makeHtmlList(constituency,results,callback) {  
  var htmllist = '<h2>Results for '+constituency+'</h2><ol class="petition-results">';
  results.forEach(function(petitiondetails){
    htmllist+='<li><span class="list-item-head"><a href="https://petition.parliament.uk/petitions/'+petitiondetails._id+'">'+petitiondetails.fields.action+'</a></span><span class="list-item-info">'+petitiondetails.sort[1]+' signatures from a total of '+petitiondetails.fields.signature_count+' </span></li>';
  })
  htmllist+='</ol>';
  callback(htmllist);
}

To call it, we need to change how our results function handles the response it gets from Elasticsearch:

if (error){  
  console.log("search error: "+error)
}
else {  
  makeHtmlList(constitLookup,response.hits.hits,function(response){
    callback(response);
  });
}

Drop that into functions.js and re-run the app. You should see a list of petitions with details of how many signatures each has attracted in the constituency. Click on the petition titles to view the petition in full.

Uploading the finished app

Before we can upload our finished application we need to add in some dependencies to our package.json file so the app can be built using the modules it needs before being deployed by Bluemix. You can do that by replacing the dependencies section of package.json with the following:

"dependencies": {
  "cfenv": "1.0.x",
  "elasticsearch": "^10.1.3",
  "express": "4.13.x",
  "get-json": "0.0.2",
  "pug": "^0.1.0"
}

Now you can push your changes up to Bluemix. When you downloaded the basic app from Bluemix it included a manifest.yml file which contains all the information needed to upload your updated app so all you need to type is:

cf push

...and your updated app will be uploaded and automatically restaged by Bluemix.

Wrapping Up

We've come a long way from creating our Elasticsearch deployment in part 1. We've defined mappings , looked at non-analyzed and nested datatypes, we've taken a brief look at Elasticsearch queries , and we've queriednested fields.

And now we've put it all together to create a web app.

We hope this series has given you a solid introduction to Elasticsearch and howCompose can help you do awesome things with it.





About List