Inside Horizon: interactive analysis at cloud scale
From the website:
Late last year, we were honored to be invited to talk at Reflections|Projections, ACM@UIUC’s annual student-run computing conference. We decided to bring a talk about Horizon, our system for doing aggregate analysis and filtering across very large amounts of data. The video of the talk was posted a few weeks back on the conference website.
Horizon started as research project / technology demonstrator built as part of Palantir’s Hack Week – a periodic innovation sprint that our engineering team uses to build brand new ideas from whole cloth. It was then used to by the Center For Public Integrity in their Who’s Behind The Subprime Meltdown report. We produced a short video on the subject, Beyond the Cloud: Project Horizon, released on our analysis blog. Subsequently, it was folded into our product offering, under the name Object Explorer.
In this hour-long talk, two of the engineers that built this technology tell the story of how Horizon came to be, how it works, and show a live demo of doing analysis on hundreds of millions of records in interactive time.
From the presentation:
Mission statement: Organize the world’s information and make it universally accessible and useful. -> Google’s statement
Which should say:
Organize the world’s [public] information and make it universally accessible and useful.
Palantir’s misson:
Organize the world’s [private] information and make it universally accessible and useful.
Closes on human-driven analysis.
A couple of points:
The demo was of a pre-beta version even though the product version shipped several months prior to the presentation. What’s with that?
Long on general statements and short on any specifics.
Did mention this is a column-store solution. Appears to work well with very clean data, but then what solution doesn’t?
Good emphasis on user interface and interactive responses to queries.
I wonder if the emphasis on interactive responses creates unrealistic expectations among customers?
Or an emphasis on problems that can be solved or appear to be solvable, interactively?
My comments about intelligence community bias the other day for example. You can measure and visualize tweets that originate in Tahrir Square, but if they are mostly from Western media, how meaningful is that?