Exporting Logentries Data With Leexportpy

Datetime:2016-08-23 03:29:28          Topic: Apache Kafka           Share

Leexportpy, the Logentries utility that exports your log data to 3 rd parties, has built-in support for various services such as Kafka, Geckoboard and Hosted Graphite. Without any modification to the current code, you can use these services to extract your Logentries data.

To begin, make sure your read-write or read-only API key is correctly placed in the LE section of your configuration file as shown below. Also make sure you have read the first blog post of this series.

[LE]
    ro_api_key = <your Logentries read-only api key>
    rw_api_key = <your Logentries read-write api key>

Kafka

Leexportpy provides easy to use support for appending transformed Logentries statistics data to a Kafka topic. The current implementation of the Kafka Service is a great reference for showing the querying, transformation and export of Logentries data to a Kafka topic using Leexportpy. The data extracted from Logentries is based on querying your log data using a LEQL query, the results, which are either time series, or groupby statistics is then appended to Kafka topic. The structure of the service and the destination configuration represents what a typical Leexportpy Kafka service should look like.

[[[destination]]]
            service = kafka
            brokers = <comma separated host:port of brokers e.g: “localhost:9092,localhost:9093”>
            topic = <name of the kafka topic>

Statistics data from a query will be appended to Kafka Topic in the order they received in the response, for example see the images below:

Kafka Service can be followed from application logs very easily as other services. Here 118.0, 104.0, 100.0, 31.0 is appended to Kafka topic by Leexportpy.

Kafka consumer consumes the results in the same order.

Geckoboard

We support various Geckoboard custom widgets in leexportpy. To export Logentries data to Geckoboard, you should include a “geckoboard” entry in the services section of your configuration file along with your Geckoboard API key, in this format:

[Services]
    [[geckoboard]]
        api_key = <your geckoboard api key>

This format, as you will see in other services, applies to all used services even if no API key is needed for that service. Destination services that are used by searches should be present in the Services section with the same service name(service)  in the destination configuration.

Some of the currently supported geckoboard widgets are highlighted below along with the corresponding destination configuration that the related Search should have.

Note:To get the push URL for your custom widget, in the Geckoboard UI go to edit page of your widget and click “Push” in the “Method” section. You can also get the push URL during the creation of a widget.

At geckoboard widget creating page, you can find the push URL to be used by leexportpy.

Line Chart:

The Line Chart service of Leexportpy expects a timeseries (non groupby) query response. The configuration of a line chart widget should look like:

[[[destination]]]
    push_url = <push url for line chart widget>
    service = geckoboard
    widget_type = line_chart
    name = <name of your series in line chart>

Bar Chart:

The query that provides data to Bar Chart can be either timeseries or groupby. The configuration of a bar chart widget should look like:

[[[destination]]]
    push_url = <push url for bar chart widget>
    service = geckoboard
    widget_type = bar_chart

Pie Chart:

The Pie Chart expects groupby query response to transform and push data to Geckoboard. The configuration of a pie chart widget should look like:

[[[destination]]]
    service = geckoboard
    push_url = <your geckoboard push url for pie chart widget>
    widget_type = pie_chart

Number and Secondary Stat:

The Number and Secondary Stat widget expects time series (non groupby) query because global count can be found only on timeseries queries.

[[[destination]]]
    service = geckoboard
    push_url = <push url for number stat widget>
    widget_type = number_stat
    text = <name of your number stat>

An image of Geckoboard dashboard with widgets after data was exported by leexportpy.

Hosted Graphite

Hosted Graphite is another 3rd party service currently supported in Leexportpy. You can now export your Logentries timeseries data for processing to Hosted Graphite.

To setup Hosted Graphite integration the hosted_graphite section with a valid Hosted Graphite API key in the Services section of your configuration file as shown below.

[Services]
    [[hosted_graphite]]
    api_key = <your Hosted Graphite API key>

Note: Only timeseries query results will be pushed to hosted graphite. Timestamps in the response will be pushed to Hosted Graphite along with their corresponding “count” values. The destination format of your search should be this:

[[[destination]]]
    push_url = https://www.hostedgraphite.com/api/v1/sink
    service = hosted_graphite
    metric_name = <metric name in the hosted graphite>

An example hosted graphite graph after Logentries data exported by leexportpy.

Conclusion

Currently supported services are production ready and provide examples of what can be done with the data provided by the Logentries REST Query API and the scheduling & processing mechanisms of Leexportpy . What you can do with Leexportpy is not limited to the currently supported services. Any service that is capable of accepting and processing your transformed log data can be implemented as a service in Leexportpy and give you the ability to synchronize and process your Logentries data.

In the following and last blog post of this series we will be showing how to create a new service from scratch and synchronizing our log data with a new external system.





About List