How to convert a logfile to JSONp with jq the JSON processor

Datetime:2016-08-22 23:49:14          Topic: JSONP           Share

jq is a new tool for command-line data mining . I’ve been using jq for a couple of months and I am very impressed.

JSONp is a data-exchange format for HTML5 applications that works around the Same Source policy. It’s a very powerful resource foraggregating Web services data into HTML5 dashboards.

In this post I present a simple example of transforming plain text logfiles into JSONp with jq.

For example purposes, I am using a Web log data set from The Web Server Workload Characterization project. as I have previouslydiscussed here.

How to convert lines of plain text into a JSONp array

Since a standard Web log file contains information broken up into lines, I just need to split the logfile on newlines in order to arrive at an array which I then wrap in a call to a predefined JavaScript function I have chosen to call jsonpHelper

head -n5 NASA_access_log_Jul95 | \
    jq --slurp --raw-input --raw-output \
    'split("\n") | @text "jsonpHelper(\(.));"'

Which produces JavaScript output like the following (I’ve added whitespace for readability).

jsonpHelper(["in24.inetnebr.com - - [01/Aug/1995:00:00:01 -0400] \"GET /shuttle/missions/sts-68/news/sts-68-mcc-05.txt HTTP/1.0\" 200 1839",
             "uplherc.upl.com - - [01/Aug/1995:00:00:07 -0400] \"GET / HTTP/1.0\" 304 0",
             "uplherc.upl.com - - [01/Aug/1995:00:00:08 -0400] \"GET /images/ksclogo-medium.gif HTTP/1.0\" 304 0",
             "uplherc.upl.com - - [01/Aug/1995:00:00:08 -0400] \"GET /images/MOSAIC-logosmall.gif HTTP/1.0\" 304 0",
             "uplherc.upl.com - - [01/Aug/1995:00:00:08 -0400] \"GET /images/USA-logosmall.gif HTTP/1.0\" 304 0"]);

Once this output has been saved to a file, I can immediately serve that file from any Web server anywhere and HTML5 applications will be able to take advantage of it!

Sending JSONp over the wire with a few lines of JavaScript!

Just drop the following snippet of jQuery into any HTML page in order to have access to the logfile data in the JavaScript DOM.

var myLogFile;

var jsonpHelper = function(payload){return payload;};

$.ajax({
    url: '<URL OF YOUR JSONP FILE>',
    dataType: 'jsonp',
    jsonp: false,
    jsonpCallback: 'jsonpHelper',
    success: function(data){myLogFile = data;}
});

I have chosen to store the data in a global variable called myLogFile so that I can work with it later.

Here is a screenshot of retrieving generated JSONp via Chrome Inspector

Thanks for reading all the way to the bottom =D

If you found this information useful then you might also be interested in a more advanced example of using the jq JSON processor to work with log files and JSON data consumers!





About List