Majority voting and information theory: What am I missing? by Panos Ipeirotis.
From the post:
In crowdsourcing, redundancy is a common approach to ensure quality. One of the questions that arises in this setting is the question of equivalence. Let’s assume that a worker has a known probability q of giving a correct answer, when presented with a choice of n possible answers. If I want to simulate one high-quality worker workers of quality q, how many workers of quality q′ < q do we need?
If you step away from match / no match type merging tests for topics, the question that Panos poses comes into play.
There has been prior work in the area where the question was the impact of quality (q) being less than or greater than 0.5. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers by Victor S. Sheng, Foster Provost, Panagiotis G. Ipeirotis.
Panos’ question is why can’t he achieve a theoretical quality of 1.0 if he uses two workers with q = 0.85?
I agree that using high quality workers in series can improve over all results. However, as I respond to his blog post, probabilities are not additive.
They are ever probabilities. Could have, on occasion, two 0.85 workers in series transmit an answer perfectly. But that is only one possible outcome out of number of possible outcomes.
What would your response be?