The Traffic Factories: Metrics at Chartbeat, Gawker Media, and The New York Times by Caitlan Petre.
From the executive summary:
In a 2010 New Yorker profile, founder and CEO of Gawker Media Nick Denton argued, “probably the biggest change in Internet media isn’t the immediacy of it, or the low costs, but the measurability.” 1Digital media scholars and commentators could debate this claim exhaustively (and have), but there is little doubt that the ability to extensively track news readers’ behavior online is indeed a profound shift from the pre-Internet era. Newsrooms can now access real-time data on how readers arrive at a particular site or article, how often they visit, and what they do once they get there (e.g., how long they spend on a page, how far they scroll, and whether they are moving their mouse or pressing any keys).
What does all this data mean for the production of news? In the earlier days of web analytics, editorial metrics had both enthusiastic proponents and impassioned detractors. Nowadays the prevailing view is that metrics aren’t, by definition, good or bad for journalism. Rather, the thinking goes, it all depends what is measured: Some metrics, like page views, incentivize the production of celebrity slide shows and other vapid content, while others, like time on a page, reward high-quality journalism. Still, there are some who doubt that even so-called “engagement metrics” can peacefully coexist with (let alone bolster) journalistic values.
This report’s premise is that it will be impossible to settle these debates until we understand how people and organizations are producing, interpreting, and using metrics. I conducted an ethnographic study of the role of metrics in contemporary news by examining three case studies: Chartbeat, Gawker Media, and The New York Times. Through a combination of observation and interviews with product managers, data scientists, reporters, bloggers, editors, and others, my intention was to unearth the assumptions and values that underlie audience measures, the effect of metrics on journalists’ daily work, and the ways in which metrics interact with organizational culture. Among the central discoveries:
…
No, you need to go read the discoveries for yourself! 😉
Suffice it to say that it isn’t just the presence of metrics that is addressed but the influence of the presentation of metrics as well.
Petre asks of news readers:
…
Are they aware that their behavior on news sites is being tracked to the extent that it is? If so, how (if at all) does this affect their behavior?
…
On observation impacting behavior, see the Hawthorne effect. Subject to varying interpretations, the original study found that showing increased attention to workers boosted their productivity, which fell when the increased observation ceased. Research continues on this topic until the present day.
A harder to answer question would be how to measure the impact of observation, which in and of itself, has to impact the people being observed. The observational impact is in interaction with a large number of other variables, which are as difficult to measure as the impact of observation.
This is a great read for journalists or anyone who is interested in effective communication and the potential of metrics to measure the same.