Real-time Twitter heat map with MongoDB
From the post:
Over the last few weeks I got in touch with the fascinating field of data visualisation which offers great ways to play around with the perception of information.
In a more formal approach data visualisation denotes “The representation and presentation of data that exploits our visual perception abilities in order to amplify cognition“
Nowadays there is a huge flood of information that hit’s us everyday. Enormous amounts of data collected from various sources are freely available on the internet. One of these data gargoyles is Twitter producing around 400 million (400 000 000!) tweets per day!
Tweets basically offer two “layers” of information. The obvious direct information within the text of the Tweet itself and also a second layer that is not directly perceived which is the Tweets’ metadata. In this case Twitter offers a large number of additional information like user data, retweet count, hashtags, etc. This metadata can be leveraged to experience data from Twitter in a lot of exciting new ways!
So as a little weekend project I have decided to build a small piece of software that generates real-time heat maps of certain keywords from Twitter data.
Yes, “…in a lot of exciting new ways!” +1!
What about maintenance issues on such a heat map? The capture of terms to the map is fairly obvious, but a subsequent user may be left in the dark as to why this term and not some other term? Or some then current synonym for a term that is being captured?
Or imposing semantics on tweets or terms that are unexpected or non-obvious to a casual or not so casual observer.
You and I can agree red means go and green means stop in a tweet. That’s difficult to maintain as the number of participants and terms go up.
A great starting place to experiment with topic maps to address such issues.
I first saw this in the NoSQL Weekly Newsletter.