Consensus of Ambiguity: Theory and Application of Active Learning for Biomedical Image Analysis Authors: Scott Doyle, Anant Madabhushi Keywords:
Abstract:
Supervised classifiers require manually labeled training samples to classify unlabeled objects. Active Learning (AL) can be used to selectively label only “ambiguous” samples, ensuring that each labeled sample is maximally informative. This is invaluable in applications where manual labeling is expensive, as in medical images where annotation of specific pathologies or anatomical structures is usually only possible by an expert physician. Existing AL methods use a single definition of ambiguity, but there can be significant variation among individual methods. In this paper we present a consensus of ambiguity (CoA) approach to AL, where only samples which are consistently labeled as ambiguous across multiple AL schemes are selected for annotation. CoA-based AL uses fewer samples than Random Learning (RL) while exploiting the variance between individual AL schemes to efficiently label training sets for classifier training. We use a consensus ratio to determine the variance between AL methods, and the CoA approach is used to train classifiers for three different medical image datasets: 100 prostate histopathology images, 18 prostate DCE-MRI patient studies, and 9,000 breast histopathology regions of interest from 2 patients. We use a Probabilistic Boosting Tree (PBT) to classify each dataset as either cancer or non-cancer (prostate), or high or low grade cancer (breast). Trained is done using CoA-based AL, and is evaluated in terms of accuracy and area under the receiver operating characteristic curve (AUC). CoA training yielded between 0.01-0.05% greater performance than RL for the same training set size; approximately 5-10 more samples were required for RL to match the performance of CoA, suggesting that CoA is a more efficient training strategy.
The consensus of ambiguity (CoA) is trivially extensible to other image analysis. Intelligence photos anyone?
What intrigues me is extension of that approach to other types of data analysis.
Such as having multiple AL schemes process textual data and follow the CoA approach on what to bounce to experts for annotation.
Questions:
- What types of ambiguity would this approach miss?
- How would you apply this method to other data?
- How would you measure success/failure of application to other data?
- Design and apply this concept to specified data set. (project)