Archive for the ‘logstash’ Category

Big data in minutes with the ELK Stack

Friday, November 21st, 2014

Big data in minutes with the ELK Stack by Philippe Creux.

From the post:

We’ve built a data analysis and dashboarding infrastructure for one of our clients over the past few weeks. They collect about 10 million data points a day. Yes, that’s big data.

My highest priority was to allow them to browse the data they collect so that they can ensure that the data points are consistent and contain all the attributes required to generate the reports and dashboards they need.

I chose to give the ELK stack a try: ElasticSearch, logstash and Kibana.

Is it just me or does processing “big data” seem to have gotten easier over the past several years?

But however easy or hard the processing, the value-add question is what do we know post data processing that we didn’t know before?

Using Solr to Search and Analyze Logs

Tuesday, November 12th, 2013

Using Solr to Search and Analyze Logs by Radu Gheorghe.

From the description:

Since we’ve added Solr output for Logstash, indexing logs via Logstash has become a possibility. But what if you are not using (only) Logstash? Are there other ways you can index logs in Solr? Oh yeah, there are! The following slides are from Lucene Revolution conference that just took place in Dublin where we talked about indexing and searching logs with Solr.

Slides but a very good set of slides.

Radu’s post reminds me I over looked logs in the Hadoop eco-system when describing semantic diversity (Hadoop Ecosystem Configuration Woes?).

Or for that matter, how do you link up the logs with particular configuration or job settings?

Emails to the support desk and sticky notes don’t seem equal to the occasion.


Wednesday, March 28th, 2012


From the webpage:

You have logs. Billions of lines of data. You shipped, dated it, parsed it and stored it. Now what do you do with it? Now you make sense of it. Kibana helps you do that. Kibana is an alternative browser based interface for Logstash and ElasticSearch that allows you to efficiently search, graph, analyze and otherwise make sense of a mountain of logs.

Any thoughts of what data you would map to such an interface? Or map to the aggregations that it offers?

Elasticsearch Using index templates & dynamic mappings

Saturday, February 4th, 2012

Elasticsearch Using index templates & dynamic mappings

Enables faceted searches of logs using logstash.

If you don’t know logstash, you might want to take a quick tour.

I found it interesting that you can now parse events on a TCP socket.

What you want to add to logs, events, etc., for mapping purposes is entirely up to you.