Another Word For It Patrick Durusau on Topic Maps and Semantic Diversity

August 10, 2012

[C]rowdsourcing … knowledge base construction

Filed under: Biomedical,Crowd Sourcing,Data Mining,Medical Informatics — Patrick Durusau @ 1:48 pm

Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications by Allison B McCoy, Adam Wright, Archana Laxmisan, Madelene J Ottosen, Jacob A McCoy, David Butten, and Dean F Sittig. (J Am Med Inform Assoc 2012; 19:713-718 doi:10.1136/amiajnl-2012-000852)

Abstract:

Objective We describe a novel, crowdsourcing method for generating a knowledge base of problem–medication pairs that takes advantage of manually asserted links between medications and problems.

Methods Through iterative review, we developed metrics to estimate the appropriateness of manually entered problem–medication links for inclusion in a knowledge base that can be used to infer previously unasserted links between problems and medications.

Results Clinicians manually linked 231 223 medications (55.30% of prescribed medications) to problems within the electronic health record, generating 41 203 distinct problem–medication pairs, although not all were accurate. We developed methods to evaluate the accuracy of the pairs, and after limiting the pairs to those meeting an estimated 95% appropriateness threshold, 11 166 pairs remained. The pairs in the knowledge base accounted for 183 127 total links asserted (76.47% of all links). Retrospective application of the knowledge base linked 68 316 medications not previously linked by a clinician to an indicated problem (36.53% of unlinked medications). Expert review of the combined knowledge base, including inferred and manually linked problem–medication pairs, found a sensitivity of 65.8% and a specificity of 97.9%.

Conclusion Crowdsourcing is an effective, inexpensive method for generating a knowledge base of problem–medication pairs that is automatically mapped to local terminologies, up-to-date, and reflective of local prescribing practices and trends.

I would not apply the term “crowdsourcing,” here, in part because the “crowd” is hardly unknown. Not a crowd at all, but an identifiable group of clinicians.

Doesn’t invalidate the results, which shows the utility of data mining for creating knowledge bases.

As a matter of usage, let’s not confuse anonymous “crowds,” with specific groups of people.

July 2, 2012

Readersourcing—a manifesto

Filed under: Crowd Sourcing,Publishing,Reviews — Patrick Durusau @ 5:24 pm

Readersourcing—a manifesto by Stefano Mizzaro. (Mizzaro, S. (2012), Readersourcing—a manifesto. J. Am. Soc. Inf. Sci.. doi: 10.1002/asi.22668)

Abstract:

This position paper analyzes the current situation in scholarly publishing and peer review practices and presents three theses: (a) we are going to run out of peer reviewers; (b) it is possible to replace referees with readers, an approach that I have named “Readersourcing”; and (c) it is possible to avoid potential weaknesses in the Readersourcing model by adopting an appropriate quality control mechanism. The readersourcing.org system is then presented as an independent, third-party, nonprofit, and academic/scientific endeavor aimed at quality rating of scholarly literature and scholars, and some possible criticisms are discussed.

Mizzaro touches a number of issues that have speculative answers in his call for “readersourcing” of research. There is a website in progress, www.readersourcing.org.

I am interested in the approach as an aspect of crowdsourcing the creation of topic maps.

FYI, his statement that:

Readersourcing is a solution to a problem, but it immediately raises another problem, for which we need a solution: how to distinguish good readers from bad readers. If 200 undergraduate students say that a paper is good, but five experts (by reputation) in the field say that it is not, then it seems obvious that the latter should be given more importance when calculating the paper’s quality.

Seems problematic to me. Particularly for graduate students. If professors at their school rate research high or low, that should be calculated into a rating for that particular reader.

If that seems pessimistic, read: Fish, Stanley, “Transmuting the Lump: Paradise Lost, 1942-1979,” in Doing What Comes Naturally. Fish, Stanley (ed.), Duke University Press, 1989), which treats changing “expert” opinions on the closing chapters of Paradise Lost. So far as I know, the text did not change between 1942 and 1979 but “expert” opinion certainly did.

I offer that as a caution that all of our judgements are a matter of social consensus that changes over time. On some issues more quickly than others. Our information systems should reflect the ebb and flow of that semantic renegotiation.

June 10, 2012

Citizen Archivist Dashboard [“…help the next person discover that record”]

Filed under: Archives,Crowd Sourcing,Indexing,Tagging — Patrick Durusau @ 8:15 pm

Citizen Archivist Dashboard

What’s the common theme of these interfaces from the National Archives (United States)?

  • Tag – Tagging is a fun and easy way for you to help make National Archives records found more easily online. By adding keywords, terms, and labels to a record, you can do your part to help the next person discover that record. For more information about tagging National Archives records, follow “Tag It Tuesdays,” a weekly feature on the NARAtions Blog. [includes “missions” (sets of materials for tagging), rated as “beginner,” “intermediate,” and “advanced.” Or you can create your own mission.]
  • Transcribe – By contributing to transcriptions, you can help the National Archives make historical documents more accessible. Transcriptions help in searching for the document as well as in reading and understanding the document. The work you do transcribing a handwritten or typed document will help the next person discover and use that record.

    The transcription tool features over 300 documents ranging from the late 18th century through the 20th century for citizen archivists to transcribe. Documents include letters to a civil war spy, presidential records, suffrage petitions, and fugitive slave case files.

    [A pilot project with 300 documents but one you should follow. Public transcription (crowd-sourced if you want the popular term) of documents has the potential to open up vast archives of materials.]

  • Edit Articles – Our Archives Wiki is an online space for researchers, educators, genealogists, and Archives staff to share information and knowledge about the records of the National Archives and about their research.

    Here are just a few of the ways you may want to participate:

    • Create new pages and edit pre-existing pages
    • Share your research tips
    • Store useful information discovered during research
    • Expand upon a description in our online catalog

    Check out the “Getting Started” page. When you’re ready to edit, you’ll need to log in by creating a username and password.

  • Upload & Share – Calling all researchers! Start sharing your digital copies of National Archives records on the Citizen Archivist Research group on Flickr today.

    Researchers scan and photograph National Archives records every day in our research rooms across the country — that’s a lot of digital images for records that are not yet available online. If you have taken scans or photographs of records you can help make them accessible to the public and other researchers by sharing your images with the National Archives Citizen Archivist Research Group on Flickr.

  • Index the Census – Citizen Archivists, you can help index the 1940 census!

    The National Archives is supporting the 1940 census community indexing project along with other archives, societies, and genealogical organizations. The release of the decennial census is one of the most eagerly awaited record openings. The 1940 census is available to search and browse, free of charge, on the National Archives 1940 Census web site. But, the 1940 census is not yet indexed by name.

    You can help index the 1940 census by joining the 1940 census community indexing project. To get started you will need to download and install the indexing software, register as an indexing volunteer, and download a batch of images to transcribe. When the index is completed, the National Archives will make the named index available for free.

The common theme?

The tagging entry sums it up with: “…you can do your part to help the next person discover that record.”

That’s the “trick” of topic maps. Once a fact about a subject is found, you can preserve your “finding” for the next person.

May 16, 2012

Identifying And Weighting Integration Hypotheses On Open Data Platforms

Filed under: Crowd Sourcing,Data Integration,Integration,Open Data — Patrick Durusau @ 12:58 pm

Identifying And Weighting Integration Hypotheses On Open Data Platforms by Julian Eberius, Katrin Braunschweig, Maik Thiele, and Wolfgang Lehner.

Abstract:

Open data platforms such as data.gov or opendata.socrata.com provide a huge amount of valuable information. Their free-for-all nature, the lack of publishing standards and the multitude of domains and authors represented on these platforms lead to new integration and standardization problems. At the same time, crowd-based data integration techniques are emerging as new way of dealing with these problems. However, these methods still require input in form of specific questions or tasks that can be passed to the crowd. This paper discusses integration problems on Open Data Platforms, and proposes a method for identifying and ranking integration hypotheses in this context. We will evaluate our findings by conducting a comprehensive evaluation using on one of the largest Open Data platforms.

This is interesting work on Open Data platforms but it is marred by claims such as:

Open Data Platforms have some unique integration problems that do not appear in classical integration scenarios and which can only be identi ed using a global view on the level of datasets. These problems include partial- or duplicated datasets, partitioned datasets, versioned datasets and others, which will be described in detail in Section 4.

Really?

Would come as a surprise to the World Data Centre for Aerosols which had Synthesis and INtegration of Global Aerosol Data Sets. Contract No. ENV4-CT98-0780 (DG 12 –EHKN) produced on data sets from 1999 to 2001. One of the specific issues they addressed were duplicate data sets.

More than a decade ago counts for a “classical integration scenario” I think.

Another quibble. Cited sources do not support the text.

New forms of data management such as dataspaces and pay-as-you-go data integration [2, 6] are a hot topic in database research. They are strongly related to Open Data Platforms in that they assume large sets of heterogeneous data sources lacking a global or mediated schemata, which still should be queried uniformly.

…

2 M. Franklin, A. Halevy, and D. Maier. From databases to dataspaces: a new abstraction for information management. SIGMOD Rec., 34:27{33, December 2005.

…

6 J. Madhavan, S. R. Je ery, S. Cohen, X. . Dong, D. Ko, C. Yu, A. Halevy, and G. Inc. Web-scale Data Integration: You Can Only A fford to Pay As You Go. In Proc. of CIDR-07, 2007.

Articles written seven (7) and five (5) years ago, do not justify a “hot topic(s) in database research.” claim today.

There are other issues, major and minor but for all that, this is important work.

I want to see reports that do justice to its importance.

May 12, 2012

TREC 2012 Crowdsourcing Track

Filed under: Crowd Sourcing,TREC — Patrick Durusau @ 6:22 pm

TREC 2012 Crowdsourcing Track

Panos Ipeirotis writes:

TREC 2012 Crowdsourcing Track – Call for Participation
 June 2012 – November 2012

https://sites.google.com/site/treccrowd/

Goals

As part of the National Institute of Standards and Technology (NIST)‘s annual Text REtrieval Conference (TREC), the Crowdsourcing track investigates emerging crowd-based methods for search evaluation and/or developing hybrid automation and crowd search systems.

This year, our goal is to evaluate approaches to crowdsourcing high quality relevance judgments for two different types of media:

  1. textual documents
  2. images

For each of the two tasks, participants will be expected to crowdsource relevance labels for approximately 20k topic-document pairs (i.e., 40k labels when taking part in both tasks). In the first task, the documents will be from an English news text corpora, while in the second task the documents will be images from Flickr and from a European news agency.

Participants may use any crowdsourcing methods and platforms, including home-grown systems. Submissions will be evaluated against a gold standard set of labels and against consensus labels over all participating teams.

Tentative Schedule

  • Jun 1: Document corpora, training topics (for image task) and task guidelines available
  • Jul 1: Training labels for the image task
  • Aug 1: Test data released
  • Sep 15: Submissions due
  • Oct 1: Preliminary results released
  • Oct 15: Conference notebook papers due
  • Nov 6-9: TREC 2012 conference at NIST, Gaithersburg, MD, USA
  • Nov 15: Final results released
  • Jan 15, 2013: Final papers due

As you know, I am interested in crowd sourcing of paths through data and assignment of semantics.

Although I am puzzled why we continue to put emphasis on post-creation assignment of semantics?

After data is created, we look around surprised the data has no explicit semantics.

Like realizing you are on Main Street without your pants.

Why don’t we look to the data creation process to assign explicit semantics?

Thoughts?

May 11, 2012

Crowdsourcing – A Solution to your “Bad Data” Problems

Filed under: Crowd Sourcing,Data Quality — Patrick Durusau @ 3:11 pm

Crowdsourcing – A Solution to your “Bad Data” Problems by Hollis Tibbetts.

Hollis writes:

Data problems – whether they be inaccurate data, incomplete data, data categorization issues, duplicate data, data in need of enrichment – are age-old.

IT executives consistently agree that data quality/data consistency is one of the biggest roadblocks to them getting full value from their data. Especially in today’s information-driven businesses, this issue is more critical than ever.

Technology, however, has not done much to help us solve the problem – in fact, technology has resulted in the increasingly fast creation of mountains of “bad data”, while doing very little to help organizations deal with the problem.

One “technology” holds much promise in helping organizations mitigate this issue – crowdsourcing. I put the word technology in quotation marks – as it’s really people that solve the problem, but it’s an underlying technology layer that makes it accurate, scalable, distributed, connectable, elastic and fast. In an article earlier this week, I referred to it as “Crowd Computing”.

Crowd Computing – for Data Problems

The Human “Crowd Computing” model is an ideal approach for newly entered data that needs to either be validated or enriched in near-realtime, or for existing data that needs to be cleansed, validated, de-duplicated and enriched. Typical data issues where this model is applicable include:

  • Verification of correctness
  • Data conflict and resolution between different data sources
  • Judgment calls (such as determining relevance, format or general “moderation”)
  • “Fuzzy” referential integrity judgment
  • Data error corrections
  • Data enrichment or enhancement
  • Classification of data based on attributes into categories
  • De-duplication of data items
  • Sentiment analysis
  • Data merging
  • Image data – correctness, appropriateness, appeal, quality
  • Transcription (e.g. hand-written comments, scanned content)
  • Translation

In areas such as the Data Warehouse, Master Data Management or Customer Data Management, Marketing databases, catalogs, sales force automation data, inventory data – this approach is ideal – or any time that business data needs to be enriched as part of a business process.

Hollis has a number of good points. But the choice doesn’t have to be “big data/iron” versus “crowd computing.”

More likely to get useful results out of some combination of the two.

Make “big data/iron” responsible for raw access, processing, visualization in an interactive environment with semantics supplied by the “crowd computers.”

And vet participants on both sides in real time. Would be a novel thing to have firms competing to supply the interactive environment and being paid on the basis of the “crowd computers” that preferred it or got better results.

That is a ways past where Hollis is going but I think it leads naturally in that direction.

April 1, 2012

Syrian crowdmapping project documents reports of rape

Filed under: Crowd Sourcing — Patrick Durusau @ 7:13 pm

Syrian crowdmapping project documents reports of rape

Niall Firth, technology editor for the New Scientist, writes:

Earlier this month, an unnamed woman in the village of Sahl Al-Rawj, Syria, left the safety of her hiding place to plead for the lives of her husband and son as government forces advanced. She was captured and five soldiers took turns raping her as she was forced to watch her husband die.

Her shocking story – officially unverified – is just one of many reports of sexual violence against women that has come out of Syria as fighting continues between government forces and rebels. Now a crowd-mapping website, launched this week, will attempt to detail every such rape and incident of sexual violence against women throughout the conflict.

The map is the creation of the Women under Siege initiative, and uses the same crowdsourcing technology developed by Washington DC-based Ushaidi, which is also being used to calculate the death toll in the recent fighting.

I read not all that long ago that under reporting of rape is 60% among civilians and 80% among the military. Military Sexual Abuse: A Greater Menace Than Combat

Would a mapping service such as the one created for the conflict in Syria help with the under reporting of rape in the United States? That would at least document the accounts of rape victims and the locations of their attacks.

Greater reporting of rapes and their locations is a first step.

Topic maps could help with the next step: Outing Rapists.

Outing Rapists means binding the accounts and locations of rapes to Facebook, faculty, department, government, listings of rapists.

Reporting a rape will help you help yourself. Anonymously or otherwise.

Outing a rapist may prevent a future rape.

A couple of resources out of thousands on domestic or sexual violence: National Center on Domestic and Sexual Violence or U.S. Military Violence Against Women.

March 21, 2012

SoSlang Crowdsources a Dictionary

Filed under: Crowd Sourcing,Dictionary — Patrick Durusau @ 3:31 pm

SoSlang Crowdsources a Dictionary

Stephen E. Arnold writes:

Here’s a surprising and interesting approach to dictionaries: have users build their own. SoSlang allows anyone to add a slang term and its definition. Beware, though, this site is not for everyone. Entries can be salty. R-rated, even. You’ve been warned.

I would compare this approach:

speakers -> usages -> dictionary

to a formal dictionary:

speakers -> usages -> editors -> formal dictionary

That is to say a formal dictionary reflects the editor’s sense of the language and not the raw input of the speakers of a language.

It would be a very interesting text mining tasks to eliminate duplicate usages of terms so that the changing uses of a term can be tracked.

March 1, 2012

Crowdsourcing and the end of job interviews

Filed under: Authoring Topic Maps,Crowd Sourcing — Patrick Durusau @ 9:00 pm

Crowdsourcing and the end of job interviews by Panos Ipeirotis.

From the post:

When you discuss crowdsourcing solutions with people that have not heard the concept before, they tend to ask the question: “Why is crowdsourcing so much cheaper than existing solutions that depend on ‘classic’ outsourcing?”

Interestingly enough, this is not a phenomenon that appears only in crowdsourcing. The Sunday edition of the New York Times has an article titled Why Are Harvard Graduates in the Mailroom?. The article discusses the job searching strategy in some fields (e.g., Hollywood, academic, etc), where talented young applicants are willing to start with jobs that are paying well below what their skills deserve, in exchange for having the ability to make it big later in the future:

[This is] the model lottery industry. For most companies in the business, it doesn’t make economic sense to, as Google does, put promising young applicants through a series of tests and then hire only the small number who pass. Instead, it’s cheaper for talent agencies and studios to hire a lot of young workers and run them through a few years of low-paying drudgery…. This occupational centrifuge allows workers to effectively sort themselves out based on skill and drive. Over time, some will lose their commitment; others will realize that they don’t have the right talent set; others will find that they’re better at something else.

Interestingly enough, this occupational centrifuge is very close to the model of employment in crowdsourcing.

The author’s take is that esoteric interview questions aren’t as effective as using a crowdsourcing model. I suspect he may be right.

If that is true, how would you go about structuring a topic map authoring project for crowdsourcing? What framework would you erect going into the project? What sort of quality checks would you implement? Would you “prime the pump” with already public data to be refined?

Are we on the verge of a meritocracy of performance?

As opposed to once meritocracies of performance, now the lands of clannish and odd questions in interviews?

December 13, 2011

Orev: The Apache OpenRelevance Viewer

Filed under: Crowd Sourcing,Natural Language Processing,Relevance — Patrick Durusau @ 9:50 pm

Orev: The Apache OpenRelevance Viewer

From the webpage:

The OpenRelevance project is an Apache project, aimed at making materials for doing relevance testing for information retrieval (IR), Machine Learning and Natural Language Processing (NLP). Think TREC, but open-source.

These materials require a lot of managing work and many human hours to be put into collecting corpora and topics, and then judging them. Without going into too many details here about the actual process, it essentially means crowd-sourcing a lot of work, and that is assuming the OpenRelevance project had the proper tools to offer the people recruited for the work.

Having no such tool, the Viewer – Orev – is meant for being exactly that, and so to minimize the overhead required from both the project managers and the people who will be doing the actual work. By providing nice and easy facilities to add new Topics and Corpora, and to feed documents into a corpus, it will make it very easy to manage the surrounding infrastructure. And with a nice web UI to be judging documents with, the work of the recruits is going to be very easy to grok.

Focuses on judging of documents but that is a common level of granularity these days for relevance.

I don’t know of anything more granular but if you find such a tool, please sing out!

November 23, 2011

Crowdsourcing Maps

Filed under: Authoring Topic Maps,Crowd Sourcing,Maps — Patrick Durusau @ 7:35 pm

Crowdsourcing Maps by Mikhil Masli appears in the November 2011 issue of Computer.

Mikhil describes geowikis as having three characteristics that enable crowdsourcing of maps:

  • simple, WYSIWYG editing of geographic features like roads and landmarks
  • versioning that works with a network of tightly coupled objects rather than independent documents, and
  • spatial monitoring tools that make it easier for users to “watch” a geographic area for possibly malicious edits and to interpret map changes visually.

How would those translate into characeristics of topic maps?

  • simple WYSIWYG interface
  • versioning at lowest level
  • subject monitoring tools to enable watching for edits

Oh, I forgot, the topic map originator would have to supply the basic content of the map. Not going to be very interesting to have an empty map for other to fill in.

That is where geographic maps have the advantage is that there is already some framework, into which any user can add their smaller bit of information.

In creating environments where we want users to add to topic maps, we need to populate those “maps” and make it easy for users to contribute.

For example, a library catalog is already populated with information and one possible goal (it may or may not be yours) would be to annotate library holdings with commentary by anonymous or non-anonymous comments/reviews by library patrons. The binding could be based on the library’s internal identifier with other subjects (such as roles) being populated transparently to the user.

Could you do that without a topic map? Possibly, depending on your access to the internals of your library catalog software. But could you then also associate all those reviews with a particular author and not a particular book they had written? 😉 Yes, gets dicey when requirements for information delivery change over time. Topic maps excel at such situations because the subjects you want need only be defined. (Well, there is a bit more to it than that but the margin is too small to write it all down.)

My point here is that topic maps can be authored and vetted by small groups of experts but that they can also, with some planning, be usefully authored by large groups of individuals. That places a greater burden on the implementer of the authoring interface but experience with that sort of thing appears to be growing.

November 20, 2011

On Data and Jargon

Filed under: Crowd Sourcing,Data,Jargon — Patrick Durusau @ 4:19 pm

On Data and Jargon by Phil Simon.

From the post:

I was recently viewing an online presentation from my friend Scott Berkun. In it, Scott uses austere slides like the one with this simple bromide:

Whoever uses the most jargon has the least confidence in their ideas.

I really like that.

Are we hiding from crowds behind our jargon?

If yes, why? What do we have to lose? What do we have to gain by not hiding?

November 19, 2011

Crowdsourcing Scientific Research: Leveraging the Crowd for Scientific Discovery

Filed under: Authoring Topic Maps,Crowd Sourcing — Patrick Durusau @ 10:25 pm

Crowdsourcing Scientific Research: Leveraging the Crowd for Scientific Discovery by Dave Oleson.

From the post:

Lab scientists spend countless hours manually reviewing and annotating cells. What if we could give these hours back, and replace the tedious parts of science with a hands-off, fast, cheap, and scalable solution?

That’s exactly what we did when we used the crowd to count neurons, an activity that computer vision can’t yet solve. Building on the work we recently did with the Harvard Tuberculosis lab, we were able to take untrained people all over the world (people who might never have learned that DNA Helicase unzips genes…), turn them into image analysts with our task design and quality control, and get results comparable to those provided by trained lab workers.

So, do you think authoring your topic map is more difficult than reliable identification of neurons? Really?

Maybe the lesson of crowd sourcing is that we need to be as smart at coming up new ways to do old tasks as we think we are.

What do you think?

November 12, 2011

Humans Plus Computers Equals Better Crowdsourcing

Filed under: Crowd Sourcing,Human Cognition — Patrick Durusau @ 8:38 pm

Humans Plus Computers Equals Better Crowdsourcing by Karen Weise.

Business Week isn’t a place I frequent for technology news. This article may change my attitude about it. Not its editorial policy but its technical content, at least sometimes.

From the article.

Greek-born computer scientist Panagiotis Ipeirotis is developing technology that gets computers to help people work smarter, and vice versa

If computer scientist Panagiotis Ipeirotis were to write a profile of himself, he’d start by hiring people online to summarize the key concepts in his published papers. Then he’d write a program to download every word in his 187 blog entries and examine which posts visitors to the site read most. Ipeirotis, an associate professor at New York University’s Stern School of Business, would do all that because his research shows that pairing computer and human intelligence can unearth discoveries neither can find alone. Ipeirotis, 35, is an expert on crowdsourcing, a way to break down big projects into small tasks that many people perform online. He tries to find ways, as he puts it, of using computer databases to augment human inputs.

Ipeirotis describes a recent real-world success with Magnum Photos. The renowned photo agency had hundreds of thousands of images scanned into its digital archive that it couldn’t search because they weren’t tagged with keywords. So Magnum hired Tagasauris, a startup Ipeirotis co-founded, to begin annotating. As Tagasauris’s online workers typed in tags, its analytical software queried databases to make the descriptions more specific. For example, when workers tagged a photo with the word “chicken,” the software tried to clarify whether the worker meant the feathery animal, the raw meat, or the death-defying game.

I really like the line:

He tries to find ways, as he puts it, of using computer databases to augment human inputs.

Rather than either humans or computers trying to do any task along, divide it up so that each is doing stuff it does well. For example, if photos are sorted down to a few possible matches, why not ask a human? Or if you have thousands of records to roughly sort, why not ask a computer?

Augmenting human inputs is something topic maps do well. They provide access to content that may have been input differently than at present. They can also enhance human knowledge of the data structures that hold information, augmenting our knowledge there as well.

November 4, 2011

Confidence Bias: Evidence from Crowdsourcing

Filed under: Bias,Confidence Bias,Crowd Sourcing,Interface Research/Design — Patrick Durusau @ 6:10 pm

Confidence Bias: Evidence from Crowdsourcing Crowdflower

From the post:

Evidence in experimental psychology suggests that most people overestimate their own ability to complete objective tasks accurately. This phenomenon, often called confidence bias, refers to “a systematic error of judgment made by individuals when they assess the correctness of their responses to questions related to intellectual or perceptual problems.” 1 But does this hold up in crowdsourcing?

We ran an experiment to test for a persistent difference between people’s perceptions of their own accuracy and their actual objective accuracy. We used a set of standardized questions, focusing on the Verbal and Math sections of a common standardized test. For the 829 individuals who answered more than 10 of these questions, we asked for the correct answer as well as an indication of how confident they were of the answer they supplied.

We didn’t use any Gold in this experiment. Instead, we incentivized performance by rewarding those finishing in the top 10%, based on objective accuracy.

I am not sure why crowdsourcing would make a difference on the question of overestimation of ability but now the answer is in, N0. But do read the post for the details, I think you will find it useful when doing user studies.

For example, when you ask a user if some task is too complex as designed, are they likely to overestimate their ability to complete it, either to avoid being embarrassed in front of others or admitting that they really didn’t follow your explanation?

My suspicion is yes and so in addition to simply asking users if they understand particular search or other functions with an interface, you need to also film them using the interface with no help from you (or others).

You will remember in Size Really Does Matter… that Blair and Maron reported that lawyers over estimated their accuracy in document retrieval by 55%. Of course, the question of retrieval is harder to evaluate than those in the Crowdflower experiment but it is a bias you need to keep in mind.

October 26, 2011

Collective Intelligence 2012: Deadline November 4, 2011

Filed under: Conferences,Crowd Sourcing — Patrick Durusau @ 6:59 pm

Collective Intelligence 2012: Deadline November 4, 2011 by Panos Ipeirotis.

From the post:

For all those of you interested in crowdsourcing, I would like to bring your attention to a new conference, named Collective Intelligence 2012, being organized at MIT this spring (April 18-20, 2012) by Tom Malone and Luis von Ahn. The conference is expected to have a set of 15-20 invited speakers (disclaimer: I am one of them), and also accepts papers submitted for publication. The deadline is November 4th, 2011, so if you have something that you would be willing to share with a wide audience interested in collective intelligence, this may be a place to consider.

If you do attend, please share your thoughts on the papers as relevant to crowdsourcing and topic map authoring. Thanks!

September 29, 2011

Human Computation: Core Research Questions and State of the Art

Filed under: Artificial Intelligence,Crowd Sourcing,Human Computation — Patrick Durusau @ 6:33 pm

Human Computation: Core Research Questions and State of the Art by Luis von Ahn and Edith Law. (> 300 slide tutorial) See also: Human Computation by Edith Law and Luis von Ahn.

Abstract from the book:

Human computation is a newand evolving research area that centers around harnessing human intelligence to solve computational problems that are beyond the scope of existing Artificial Intelligence (AI) algorithms.With the growth of the Web, human computation systems can now leverage the abilities of an unprecedented number of people via the Web to perform complex computation.There are various genres of human computation applications that exist today. Games with a purpose (e.g., the ESP Game) specifically target online gamers who generate useful data (e.g., image tags) while playing an enjoyable game.Crowdsourcing marketplaces (e.g.,Amazon MechanicalTurk) are human computation systems that coordinate workers to perform tasks in exchange for monetary rewards. In identity verification tasks, users perform computation in order to gain access to some online content; an example is reCAPTCHA, which leverages millions of users who solve CAPTCHAs every day to correct words in books that optical character recognition (OCR) programs fail to recognize with certainty.

This book is aimed at achieving four goals: (1) defining human computation as a research area; (2) providing a comprehensive review of existing work; (3) drawing connections to a wide variety of disciplines, including AI, Machine Learning, HCI, Mechanism/Market Design and Psychology, and capturing their unique perspectives on the core research questions in human computation; and (4) suggesting promising research directions for the future.

You may also want to see Luis van Ahn in a Google Techtalk video from about five years ago:

July 26, 2006 Luis von Ahn is an assistant professor in the Computer Science Department at Carnegie Mellon University, where he also received his Ph.D. in 2005. Previously, Luis obtained a B.S. in mathematics from Duke University in 2000. He is the recipient of a Microsoft Research Fellowship. ABSTRACT Tasks like image recognition are trivial for humans, but continue to challenge even the most sophisticated computer programs. This talk introduces a paradigm for utilizing human processing power to solve problems that computers cannot yet solve. Traditional approaches to solving such problems focus on improving software. I advocate a novel approach: constructively channel human brainpower using computer games. For example, the ESP Game, described in this talk, is an enjoyable online game — many people play over 40 hours a week — and when people play, they help label images on the Web with descriptive keywords. These keywords can be used to significantly improve the accuracy of image search. People play the game not because they want to help, but because they enjoy it. I describe other examples of “games with a purpose”: Peekaboom, which helps determine the location of objects in images, and Verbosity, which collects common-sense knowledge. I also explain a general approach for constructing games with a purpose.

A rapidly developing and exciting area of research. Perhaps your next topic map may be authored or maintained by a combination of entities.

September 25, 2011

Tang and Lease (2011) Semi-Supervised Consensus Labeling for Crowdsourcing

Filed under: Crowd Sourcing,LingPipe — Patrick Durusau @ 7:49 pm

Tang and Lease (2011) Semi-Supervised Consensus Labeling for Crowdsourcing

From the post:

I came across this paper, which, among other things, describes the data collection being used for the 2011 TREC Crowdsourcing Track:

But that’s not why we’re here today. I want to talk about their modeling decisions.

Tang and Lease apply a Dawid-and-Skene-style model to crowdsourced binary relevance judgments for highly-ranked system responses from a previous TREC information retrieval evaluation. The workers judge document/query pairs as highly relevant, relevant, or irrelevant (though highly relevant and relevant are collapsed in the paper).

The Dawid and Skene model was relatively unsupervised, imputing all of the categories for items being classified as well as the response distribution for each annotator for each category of input (thus characterizing both bias and accuracy of each annotator).

I post this in part for the review of the model in question and also as a warning that competent people really do read research papers in their areas. Yes, on the WWW you can publish anything you want, of whatever quality. But, others in your field will notice. Is that what you want?

September 17, 2011

The Revolution(s) Are Being Televised

Filed under: Crowd Sourcing,Image Recognition,Image Understanding,Marketing — Patrick Durusau @ 8:17 pm

Revolutions usually mean human rights violations, lots of them.

Patrick Meier has a project to collect evidence of mass human rights violations in Syria.

See: Help Crowdsource Satellite Imagery Analysis for Syria: Building a Library of Evidence

Topic maps are an ideal solution to link objects in dated satellite images to eye witness accounts, captured military documents, ground photos, news accounts and other information.

I say that for two reasons:

First, with a topic map you can start from any linked object in a photo, a witness account, ground photo or news account and see all related evidence for that location. Granted that takes someone authoring that collation but it doesn’t have to be only one someone.

Second, topic maps offer parallel subject processing, which can distribute the authoring task in a crowd-sourced project, for instance. For example, I could be doing photo analysis and marking the location of military checkpoints. That would generate topics and associations for the geographic location, the type of installation, dates (from the photos), etc. Someone else could be interviewing witnesses and taking their testimony. As part of the processing of that testimony, another volunteer codes an approximate date and geographic location in connection with part of that testimony. Still another person is coding military orders by identified individuals for checkpoints that include the one in question. Associations between all these separately encoded bits of evidence, each unknown to the individual volunteers becomes a mouse-click away from coming to the attention of anyone reviewing the evidence. And determining responsibility.

The alternative, the one most commonly used, is to have an under-staffed international group piece together the best evidence it can from a sea of documents, photos, witness accounts, etc. An adequate job for the resources they have, but why settle for an “adequate” job when it can be done properly with 21st century technology?

September 9, 2011

Chess@home Building the Largest Chess AI ever

Filed under: Artificial Intelligence,Collaboration,Crowd Sourcing — Patrick Durusau @ 7:11 pm

Chess@home Building the Largest Chess AI ever

From the post:

Many people are familiar with the SETI@home project: a very large scale effort to search for patterns from alien civilizations in the ocean of data we receive from the sky, using the computing power of millions of computers around the globe (“the grid”).

SETI@home has been a success, obviously not in finding aliens, but in demonstrating the potential of large-scale distributed computing. Projects like BOINC have been expanding this effort to other fields like biology, medicine and physics.

Last weekend, a team at Joshfire (Thomas, Nathan, Mickael and myself) participated in a 48-hour coding contest called Node Knockout. Rules were simple: code the most amazing thing you can in the weekend, as long as it uses server-side JavaScript.

JavaScript is an old but exciting technology, currently undergoing a major revival through performance breakthroughs, server-side engines, and an array of new possibilities offered by HTML5. It is also the language that has the biggest scale. Any computer connected to the web can run your JavaScript code extremely easily, just by typing an URL in a browser or running a script in a command line.

We decided to exploit this scale, together with Node’s high I/O performance, to build a large-scale distributed computing project that people could join just by visiting a web page. Searching for aliens was quickly eliminated as we didn’t have our own antenna array available at the time. So we went with a somewhat easier problem: Chess.

Easier problem: Take the coming weekend and sketch out how you think Javascript and/or HTML5 are going to impact the authoring/delivery of topic maps.

May 27, 2011

GAMIFY – SETI Contest

Filed under: Crowd Sourcing,Interface Research/Design — Patrick Durusau @ 12:33 pm

GAMIFY – SETI Contest

From the webpage:

Are you a gamification expert[1] or interested in becoming one? Want to help solve a problem of epic proportions that could have a major impact on the world?

The SETI Institute and Gamify[2] together have created an EPIC Contest to explore possible ways to gamify SETI. We’re asking the most brilliant Earthlings to come up with ideas on how to apply gamification[3] to increase participation in the SETI program.

The primary goal of this social/scientific challenge is to help SETI empower global citizens to participate in the search for cosmic company and to help SETI become financially sustainable so it can live long and prosper. This article explains our problem and what we are looking to accomplish. We invite everyone to answer the question, “How would you gamify SETI?”.

To be more specific:

  • Can we create a fun and compelling app or set of apps that allow people to aid us in identifying signals?
  • Do you have any ideas to make this process a fun game, while also solving our problem, by applying game mechanics and game-thinking?
  • Can we incorporate sharing and social interaction between players?
  • Is monetization possible through virtual goods, “status short-cuts” or other methods popularized by social games?
  • Are there any angles of looking at the problem and gamifying that we have not thought of?

The scientific principles involved in this field of science can be very complicated. A conscious attempt has been made to explain the challenge we face with a minimum of scientific explanation or jargon. We wish to be able to clearly explain our unique problems and desired outcomes to the scientific and non-scientific audience.

….

  1. http://gamify.com/experts
  2. http://gamify.com
  3. http://gamification.org

You will see from the presentations at Web 2.0 Expo SF 2011 that gamification is a growing theme in UI development.

I mention this because:

  1. Gamification has the potential to ease the authoring and use(?) of topic maps.
  2. SETI is an complex and important project and so a good proving ground for gamification.
  3. Insights found here maybe applicable to more complex data, like texts.

May 12, 2011

Lessons of History? Crowdsourcing

Filed under: Crowd Sourcing — Patrick Durusau @ 8:00 am

The post by Panos Ipeirotis, Crowdsourcing: Lessons from Henry Ford on his presentation (and slides), reminded me of Will and Auriel Durant’s Lessons of History observation (paraphrasing):

If you could select them, 10% of the population produces as much as the other 90% combined. History does exactly that.

So Panos saying that “A few workers contribute the majority of the work…” is no surprise.

If you don’t think asking people for their opinions is all that weird, you may enjoy his presentation.*

His summary:

The main points that I wanted to make:

  • It is common to consider crowdsourcing as the “assembly line for knowledge work” and think of the workers as simple cogs in a big machine. It is almost a knee-jerk reaction to think negatively about the concept. However, it was the proper use of the assembly line (together with the proper automation) by Henry Ford that led to the first significant improvement in the level of living for the masses.
  • Crowdsourcing suffers a lot due to significant worker turnover: Everyone who experimented with large tasks on MTurk knows that the participation distribution is very skewed. A few workers contribute the majority of the work, while a large number of workers contribute only minimally. Dealing with these hit-and-run workers is a pain, as we cannot apply any statistically meaningful mechanism for quality control.
  • We ignore the fact that workers give back what they are given. Pay peanuts, get monkeys. Pay well, and get good workers. Needless to say, reputation and other quality signaling mechanisms are of fundamental importance for this task.
  • Keeping the same workers around can give significant improvements in quality. Today on MTurk we have a tremendous turnover of workers, wasting significant effort and efficiencies. Whomever builds a strong base of a few good workers can pay the workers much better and, at the same time, generate a better product for lower cost than relying on an army of inexperienced, noisy workers.

Yes, at the end, crowdourcing is not about the crowd. It is about the individuals in the crowd. And we can now search for these valuable individuals very effectively. Crowdsourcing is crowdsearching.

*It isn’t that people are the best judges of semantics. They are the only judges of semantics.

Automated systems for searching, indexing, sorting, etc., are critical to modern information infrastructures. What they are not doing, appearances to the contrary notwithstanding, is judging semantics.

April 29, 2011

Duolingo: The Next Chapter in Human Communication

Filed under: Crowd Sourcing,Data Mining,Education,Entity Resolution,Knowledge Capture,Subject Identity — Patrick Durusau @ 1:16 pm

Duolingo: The Next Chapter in Human Communication

By one of the co-inventors of CAPTCHA and reCAPTCHA, Luis von Ahn, so his arguments should give us pause.

Luis wants to address the problem of translating the web into multiple languages.

Yes, you heard that right, translate the web into multiple languages.

Whatever you think now, watch the video and decide if you still feel the same way.

My question is how to adapt his techniques to subject identification?

April 5, 2011

Tutorial on Crowdsourcing and Human Computation

Filed under: Crowd Sourcing — Patrick Durusau @ 4:30 pm

Tutorial on Crowdsourcing and Human Computation

From the post:

Last week, together with Praveen Paritosh from Google, we presented a 6-hour tutorial at the WWW 20111 conference, on crowdsourcing and human computation. The title of the tutorial was “Managing Crowdsourced Human Computation”.

Check the post for other links, resources.

Perhaps the lesson is to automate when possible and to use human computation when necessary. And the trick is to know when to switch.

February 21, 2011

Soylent: A Word Processor with a Crowd Inside

Filed under: Authoring Topic Maps,Crowd Sourcing,Interface Research/Design — Patrick Durusau @ 4:31 pm

Soylent: A Word Processor with a Crowd Inside

I know, I know, won’t even go there. As the librarians say: “Look it up!”

From the abstract:

This paper introduces architectural and interaction patterns for integrating crowdsourced human contributions directly into user interfaces. We focus on writing and editing, complex endeavors that span many levels of conceptual and pragmatic activity. Authoring tools offer help with pragmatics, but for higher-level help, writers commonly turn to other people. We thus present Soylent, a word processing interface that enables writers to call on Mechanical Turk workers to shorten, proofread, and otherwise edit parts of their documents on demand. To improve worker quality, we introduce the Find-Fix-Verify crowd programming pattern, which splits tasks into a series of generation and review stages. Evaluation studies demonstrate the feasibility of crowdsourced editing and investigate questions of reliability, cost, wait time, and work time for edits.

When I first started reading the article, it seemed obvious to me that the Human Macro option could be useful for topic map authoring. At least if the tasks were sufficiently constrained.

I was startled to see a 30% error rate for the “corrections” was considered a baseline, hence the necessity for correction/control mechanisms.

The authors acknowledge that the bottom line cost of out-sourcing may weigh against its use in commercial contexts.

Perhaps so but I would run the same tests against published papers and books. To determine the error rate without an out-sourced correction loop.

I think the idea is basically sound, although for some topic maps it might be better to place qualification requirements on the outsourcing.

February 13, 2011

Software for Non-Human Users?

Filed under: Collaboration,Conferences,Crowd Sourcing,Data Structures,Mashups,Networks,Social Networks — Patrick Durusau @ 5:45 am

The description of: Emerging Intelligent Data and Web Technologies (EIDWT-2011) is a call for software designed for non-human users.

The Social Life of Information by John Seely Brown and Paul Duguid, makes it clear that human users don’t want to share data because sharing data represents a loss of power/status.

A poll of the readers of CACM or Computer would report a universal experience of working in an office where information is hoarded up by individuals in order to increase their own status or power.

9/11 was preceded and followed by, to this day, by a non-sharing of intelligence data. Even national peril cannot overcome the non-sharing reflex with regard to data.

EIDWT-2011 and conferences like it, are predicated on a sharing of data known to not exist, at least among human users.

Hence, I suspect the call must be directed at software for non-human users.

Emerging Intelligent Data and Web Technologies (EIDWT-2011)

Filed under: Collaboration,Conferences,Crowd Sourcing,Data Structures,Mashups,Social Networks — Patrick Durusau @ 5:41 am

2nd International Conference on Emerging Intelligent Data and Web Technologies (EIDWT-2011)

From the announcement:

The 2-nd International Conference on Emerging Intelligent Data and Web Technologies (EIDWT-2011) is dedicated to the dissemination of original contributions that are related to the theories, practices and concepts of emerging data technologies yet most importantly of their applicability in business and academia towards a collective intelligence approach. In particular, EIDWT-2011 will discuss advances about utilizing and exploiting data generated from emerging data technologies such as Data Centers, Data Grids, Clouds, Crowds, Mashups, Social Networks and/or other Web 2.0 implementations towards a collaborative and collective intelligence approach leading to advancements of virtual organizations and their user communities. This is because, current and future Web and Web 2.0 implementations will store and continuously produce a vast amount of data, which if combined and analyzed through a collective intelligence manner will make a difference in the organizational settings and their user communities. Thus, the scope of EIDWT-2011 is to discuss methods and practices (including P2P) which bring various emerging data technologies together to capture, integrate, analyze, mine, annotate and visualize data – made available from various community users – in a meaningful and collaborative for the organization manner. Finally, EIDWT-2011 aims to provide a forum for original discussion and prompt future directions in the area.

Important Dates:

Submission Deadline: March 10, 2011
Authors Notification: May 10, 2011
Author Registration: June 10, 2011
Final Manuscript: July 1, 2011
Conference Dates: September 7 – 9, 2011

February 10, 2011

The unreasonable effectiveness of simplicity

Filed under: Authoring Topic Maps,Crowd Sourcing,Data Analysis,Subject Identity — Patrick Durusau @ 1:50 pm

The unreasonable effectiveness of simplicity from Panos Ipeirotis suggests that simplicity should be considered in the construction of information resources.

The simplest aggregation technique: Use the majority vote as the correct answer.

I am mindful of the discussion several years ago about visual topic maps. Which was a proposal to use images as identifiers. Certainly doable now but the simplicity angle suggests an interesting possibility.

Would not work for highly abstract subjects, but what if users were presented with images when called upon to make identification choices for a topic map?

For example, marking entities in a newspaper account, the user is presented with images near each marked entity and chooses yes/no.

Or in legal discovery or research, a similar mechanism, along with the ability to annotate any string with an image/marker and that image/marker appears with that string in the rest of the corpus.

Unknown to the user is further information about the subject they have identified that forms the basis for merging identifications, linking into associations, etc.

A must read!

February 2, 2011

CrowdFlower

Filed under: Authoring Topic Maps,Crowd Sourcing,Interface Research/Design — Patrick Durusau @ 9:16 am

CrowdFlower

From the website:

Like Cloud computing with People.

Computers can’t do every task. Luckily, we have people to help.

We provide instant access to an elastic labor force. And our statistical quality control technology yields results you can trust.

From CrowdFlower Gets Gamers to Do Real Work for Virtual Pay

Here’s how it works. CrowdFlower embeds tasks in online games like FarmVille, Restaurant City, It Girl, Happy Aquarium, Happy Pets, Happy Island and Pop Boom. This means that the estimated 80 million gamers — from teens to homemakers — who are hooked on FarmVille, Zynga’s popular virtual farming game on Facebook, can be transformed into a virtual workforce.

To get to the next level in FarmVille, for example, the gamer might need 600 XP (XP means “experience” in Farmville parlance). So the gamer might buy a bed and breakfast building for $60 in FarmVille cash, which would earn him 600 XP. But for many gamers, revenue — and XP — from crop harvesting comes too slowly.

To earn game money quickly, the gamer can click a tab on the FarmVille page that links to real-world tasks to be performed by crowdsourced workers. Once the task is successfully completed, the gamer gets his FarmVille cash and CrowdFlower is paid by the client. The latter pays in real money, usually with a 10 percent markup.

Like any number of crowd sourcing services but I was struck by the notion of embedding tasks inside games for virtual payment.

Not the answer to all topic map authoring tasks but certainly worth thinking about.

Question: Does anyone have experience with creating topic maps by embedding tasks in online games?

January 20, 2011

80-50 Rule?

Filed under: Crowd Sourcing,Interface Research/Design,Search Interface,Uncategorized — Patrick Durusau @ 6:18 am

Watzlawick1 recounts the following experiment:

That there is no necessary connection between fact and explanation was illustrated in a recent experiment by Bavelas (20): Each subject was told he was participating in an experimental investigation of “concept formation” and was given the same gray, pebbly card about which he was to “formulate concepts.” Of every pair of subjects (seen separately but concurrently) one was told eight out of ten times at random that what he said about the card was correct; the other was told five out of ten times at random what he said about the card was correct. The ideas of the subject who was “rewarded” with a frequency of 80 per cent remained on a simple level, which the subject who was “rewarded” only at a frequency of 50 per cent evolved complex, subtle, and abstruse theories about the card, taking into consideration the tiniest detail of the card’s composition. When the two subjects were brought together and asked to discuss their findings, the subject with the simpler ideas immediately succumbed to the “brilliance” of the other’s concepts and agreed the other had analyzed the card correctly.

I repeat this account because it illustrates the impact that “reward” systems can have on results.

Whether the “rewards” are members of a crowd or experts.

Questions:

  1. Should you randomly reject searches in training to search for subjects?
  2. What literature supports your conclusion in #1? (3-5 pages)

This study does raise the interesting question of whether conferences should track and randomly reject authors to encourage innovation.

1. Watzlawick, Paul, Janet Beavin Bavelas, and Don D. Jackson. 1967. Pragmatics of human communication; a study of interactional patterns, pathologies, and paradoxes. New York: Norton.

« Newer Posts

Powered by WordPress