The unreasonable effectiveness of simplicity from Panos Ipeirotis suggests that simplicity should be considered in the construction of information resources.
The simplest aggregation technique: Use the majority vote as the correct answer.
I am mindful of the discussion several years ago about visual topic maps. Which was a proposal to use images as identifiers. Certainly doable now but the simplicity angle suggests an interesting possibility.
Would not work for highly abstract subjects, but what if users were presented with images when called upon to make identification choices for a topic map?
For example, marking entities in a newspaper account, the user is presented with images near each marked entity and chooses yes/no.
Or in legal discovery or research, a similar mechanism, along with the ability to annotate any string with an image/marker and that image/marker appears with that string in the rest of the corpus.
Unknown to the user is further information about the subject they have identified that forms the basis for merging identifications, linking into associations, etc.
A must read!