Crossfilter: Fast Multidimensional Filtering for Coordinated Views
From the webpage:
Crossfilter is a JavaScript library for exploring large multivariate datasets in the browser. Crossfilter supports extremely fast (<30ms) interaction with coordinated views, even with datasets containing a million or more records; we built it to power analytics for Square Register, allowing merchants to slice and dice their payment history fluidly.
Since most interactions only involve a single dimension, and then only small adjustments are made to the filter values, incremental filtering and reducing is significantly faster than starting from scratch. Crossfilter uses sorted indexes (and a few bit-twiddling hacks) to make this possible, dramatically increasing the performance of live histograms and top-K lists. For more details on how Crossfilter works, see the API reference.
See the webpage for an impressive demonstration with a 5.3 MB dataset.
Is there a trend towards “big data” manipulation on clusters and “less big data” in browsers?
Will be interesting to see how the benchmarks for “big” and “less big” move over time.
I first saw this in Nat Torkington’s Four Short links: 4 March 2013.