General Purpose Computer-Assisted Clustering and Conceptualization by Justin Grimmer and Gary King.
Abstract:
We develop a computer-assisted method for the discovery of insightful conceptualizations, in the form of clusterings (i.e., partitions) of input objects. Each of the numerous fully automated methods of cluster analysis proposed in statistics, computer science, and biology optimize a different objective function. Almost all are well defined, but how to determine before the fact which one, if any, will partition a given set of objects in an “insightful” or “useful” way for a given user is unknown and difficult, if not logically impossible. We develop a metric space of partitions from all existing cluster analysis methods applied to a given data set (along with millions of other solutions we add based on combinations of existing clusterings), and enable a user to explore and interact with it, and quickly reveal or prompt useful or insightful conceptualizations. In addition, although uncommon in unsupervised learning problems, we offer and implement evaluation designs that make our computer-assisted approach vulnerable to being proven suboptimal in specific data types. We demonstrate that our approach facilitates more efficient and insightful discovery of useful information than either expert human coders or many existing fully automated methods.
Despite my misgivings about metric spaces for semantics, the central theme that clustering (dare I say merging?) cannot be determined in advance of some user viewing the data, makes sense to me. Not every user will want or perhaps even need to do interactive clustering but I think this theme represents a substantial advance in this area.
The publication appeared in the Proceeding of the National Academy of Sciences of the United States of America and the authors are from Stanford and Harvard, respectively. Institutions that value technical and scientific brilliance.