Interannotator Agreement for Chunking Tasks liked Named Entities and Phrases
Bob Carpenter writes:
Krishna writes,
I have a question about using the chunking evaluation class for inter annotation agreement : how can you use it when the annotators might have missing chunks I.e., if one of the files contains more chunks than the other.
The answer’s not immediately obvious because the usual application of interannotator agreement statistics is to classification tasks (including things like part-of-speech tagging) that have a fixed number of items being annotated.
An issue that is likely to come up in crowd sourcing analysis/annotation of text as well.