Introducing Atlas: Netflix’s Primary Telemetry Platform
From the post:
Various previous Tech Blog posts have referred to our centralized monitoring system, and we’ve presented at least one talk about it previously. Today, we want to both discuss the platform and ecosystem we built for time-series telemetry and its capabilities and announce the open-sourcing of its underlying foundation.
How We Got Here
While working in the datacenter, telemetry was split between an IT-provisioned commercial product and a tool a Netflix engineer wrote that allowed engineers to send in arbitrary time-series data and then query that data. This tool’s flexibility was very attractive to engineers, so it became the primary system of record for time series data. Sadly, even in the datacenter we found that we had significant problems scaling it to about two million distinct time series. Our global expansion, increase in platforms and customers and desire to improve our production systems’ visibility required us to scale much higher, by an order of magnitude (to 20M metrics) or more. In 2012, we started building Atlas, our next-generation monitoring platform. In late 2012, it started being phased into production, with production deployment completed in early 2013.
The use of arbitrary key/value pairs to determine a metrics identity merits a slow read. As does the query language for metrics, said “…to allow arbitrarily complex graph expressions to be encoded in a URL friendly way.”
Posted to Github with a longer introduction here.
The Wikipedia entry on time series offers this synopsis on time series data:
A time series is a sequence of data points, typically consisting of successive measurements made over a time interval. Examples of time series are ocean tides, counts of sunspots, and the daily closing value of the Dow Jones Industrial Average. Time series are very frequently plotted via line charts. Time series are used in statistics, signal processing, pattern recognition, econometrics, mathematical finance, weather forecasting, earthquake prediction, electroencephalography, control engineering, astronomy, communications engineering, and largely in any domain of applied science and engineering which involves temporal measurements.
It looks to me like a number of users communities should be interested in this release from Netflix!
Speaking of series, it occurs to me that is you count the character lengths of blanks in the Senate CIA torture report, you should be able to make some fairly good guesses on some of the names.
I am hopeful it doesn’t come to that because anyone with access to the full 6,000 page uncensored report has a moral obligation to post it to public servers. Surely there is one person with access to that report with a moral conscience.
I first saw this in a tweet by Roy Rapoport