Thoughts on ICDM I: Negative Results (part A) by Suresh Venkatasubramanian.
From (part A):
I just got back from ICDM (the IEEE conference on Data Mining). Data mining conferences are quite different from theory conferences (and much more similar to ML or DB conferences): there are numerous satellite events (workshops, tutorials and panels in this case), many more people (551 for ICDM, and that’s on the smaller side), and a wide variety of papers that range from SODA-ish results to user studies and industrial case studies.
While your typical data mining paper is still a string of techniques cobbled together without rhyme or reason (anyone for spectral manifold-based correlation clustering with outliers using MapReduce?), there are some general themes that might be of interest to an outside viewer. What I’d like to highlight here is a trend (that I hope grows) in negative results.
It’s not particularly hard to invent a new method for doing data mining. It’s much harder to show why certain methods will fail, or why certain models don’t make sense. But in my view, the latter is exactly what the field needs in order to give it a strong inferential foundation to build on (I’ll note here that I’m talking specifically about data mining, NOT machine learning – the difference between the two is left for another post).
From (part B):
Continuing where I left off on the idea of negative results in data mining, there was a beautiful paper at ICDM 2011 on the use of Stochastic Kronecker graphs to model social networks. And in this case, the key result of the paper came from theory, so stay tuned !
One of the problems that bedevils research in social networking is the lack of good graph models. Ideally, one would like a random graph model that evolves into structures that look like social networks. Having such a graph model is nice because
- you can target your algorithms to graphs that look like this, hopefully making them more efficient
- You can re-express an actual social network as a set of parameters to a graph model: it compacts the graph, and also gives you a better way of understanding different kinds of social networks: Twitter is a (0.8, 1, 2.5) and Facebook is a (1, 0.1, 0.5), and so on.
- If you’re lucky, the model describes not just reality, but how it forms. In other words, the model captures the actual social processes that lead to the formation of a social network. This last one is of great interest to sociologists.
But there aren’t even good graph models that capture known properties of social networks. For example, the classic Erdos-Renyi (ER) model of a random graph doesn’t have the heavy-tailed degree distribution that’s common in social networks. It also doesn’t have a property that’s common to large social networks: densification, or the fact that even as the network grows, the diameter stays small (implying that the network seems to get denser over time).
Part C – forthcoming –
I am perhaps more sceptical of modeling than the author but this is a very readable and interesting set of blog posts. I will be posting Part C as soon as it appears.
Update: Thoughts on ICDM I: Negative results (part C)
From Part C:
If you come up with a better way of doing classification (for now let’s just consider classification, but these remarks apply to clustering and other tasks as well), you have to compare it to prior methods to see which works better. (note: this is a tricky problem in clustering that my student Parasaran Raman has been working on: more on that later.).
The obvious way to compare two classification methods is how well they do compared to some ground truth (i.e labelled data), but this is a one-parameter system, because by changing the threshold of the classifier (or if you like, translating the hyperplane around),you can change the false positive and false negative rates.
Now the more smug folks reading these are waiting with ‘ROC’ and “AUC” at the tip of their tongues, and they’d be right ! You can plot a curve of the false positive vs false negative rate and take the area under the curve (AUC) as a measure of the effectiveness of the classifier.
For example, if the y axis measured increase false negatives, and the x-axis measured increasing false positives, you’d want a curve that looked like an L with the apex at the origin, and a random classifier would look like the line x+y = 1. The AUC score would be zero for the good classifier and 0.5 for the bad one (there are ways of scaling this to be between 0 and 1).
The AUC is a popular way of comparing methods in order to balance the different error rates. It’s also attractive because it’s parameter-free and is objective: seemingly providing a neutral method for comparing classifiers independent of data sets, cost measures and so on.
But is it ?