Archive for the ‘Human Computation’ Category

Human Computation

Saturday, May 24th, 2014

Human Computation

From the homepage:

Human Computation is an international and interdisciplinary forum for the electronic publication and print archiving of high-quality scholarly articles in all areas of human computation, which concerns the design or analysis of information processing systems in which humans participate as computational elements.

Submission Topics

(Editorial keywords are in boldface – please see author guidelines for details)

Applications – novel or transformative applications
Interfaces – HCI or related human factors methods or issues
Modalities – general interaction paradigms (e.g., gaming) and related methods
Techniques – repeatable methods, analogous to design patterns for OOP
Algorithms – wisdom of crowds, aggregation, reputation, crowdsourced analysis, and ML/HC
Architecture – development platforms, architectures, languages, APIs, IDEs, and compilers
Infrastructure – relevant networks, protocols, state space, and services
Participation – factors that influence human participation
Analysis – techniques for identifying typical characteristics and patterns in human computation systems
Epistemology – the role, source, representation, and construction of information
Policy – ethical, regulatory, and economic considerations
Security – security issues, including surreptitious behavior to influence system outcomes
Society – cultural, evolutionary, existential, psychological, and social impact
Organization – taxonomies of concepts, terminology, problem spaces, algorithms, and methods
Surveys – state of the art assessments of various facets
Meta-topics – insightful commentary on the future, philosophy, charter, and purpose of HC.

Looks like a journal for topic map articles to me.


I first saw this in a tweet by Matt Lease.

Improving Twitter search with real-time human computation [“semantics supplied”]

Tuesday, April 9th, 2013

Improving Twitter search with real-time human computation by Edwin Chen.

From the post:

Before we delve into the details, here’s an overview of how the system works.

(1) First, we monitor for which search queries are currently popular.

Behind the scenes: we run a Storm topology that tracks statistics on search queries.

For example: the query “Big Bird” may be averaging zero searches a day, but at 6pm on October 3, we suddenly see a spike in searches from the US.

(2) Next, as soon as we discover a new popular search query, we send it to our human evaluation systems, where judges are asked a variety of questions about the query.

Behind the scenes: when the Storm topology detects that a query has reached sufficient popularity, it connects to a Thrift API that dispatches the query to Amazon’s Mechanical Turk service, and then polls Mechanical Turk for a response.

For example: as soon as we notice “Big Bird” spiking, we may ask judges on Mechanical Turk to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps us serve relevant tweets and ads.

Finally, after a response from a judge is received, we push the information to our backend systems, so that the next time a user searches for a query, our machine learning models will make use of the additional information. For example, suppose our human judges tell us that “Big Bird” is related to politics; the next time someone performs this search, we know to surface ads by @barackobama or @mittromney, not ads about Dora the Explorer.

Let’s now explore the first two sections above in more detail.


The post is quite awesome and I suggest you read it in full.

This resonates with a recent comment about Lotus Agenda.

The short version is a user creates a thesaurus in Agenda that enables searches enriched by the thesaurus. The user supplied semantics to enhance the searches.

In the Twitter case, human reviewers supply semantics to enhance the searches.

In both cases, Agenda and Twitter, humans are supplying semantics to enhance the searches.

I emphasize “supplying semantics” as a contrast to mechanistic searches that rely on text.

Mechanistic searches can be quite valuable but they pale beside searches where semantics have been “supplied.”

The Twitter experience is a an important clue.

The answer to semantics for searches lies somewhere between ask an expert (you get his/her semantics) and ask ask all of us (too many answers to be useful).

More to follow.

Human Computation and Crowdsourcing

Saturday, January 26th, 2013

Announcing HCOMP 2013 – Conference on Human Computation and Crowdsourcing by Eric Horvitz.

From the conference website:


Palm Springs, California
Venue information coming soon


November 7-9, 2013

Important Dates

All deadlines are 5pm Pacific time unless otherwise noted.


Submission deadline: May 1, 2013
Author rebuttal period: June 21-28
Notification: July 16, 2013
Camera Ready: September 4, 2013

Workshops & Tutorials

Proposal deadline: May 10, 2013
Notification: July 16, 2013
Camera Ready: September 4, 2013

Posters & Demonstrations

Submission deadline: July 25, 2013
Notification: August 26, 2013
Camera Ready: September 4, 2013

From the post:

Announcing HCOMP 2013, the Conference on Human Computation and Crowdsourcing, Palm Springs, November 7-9, 2013. Paper submission deadline is May 1, 2013. Thanks to the HCOMP community for bringing HCOMP to life as a full conference, following on the successful workshop series.

The First AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2013) will be held November 7-9, 2013 in Palm Springs, California, USA. The conference was created by researchers from diverse fields to serve as a key focal point and scholarly venue for the review and presentation of the highest quality work on principles, studies, and applications of human computation. The conference is aimed at promoting the scientific exchange of advances in human computation and crowdsourcing among researchers, engineers, and practitioners across a spectrum of disciplines. Papers submissions are due May 1, 2013 with author notification on July 16, 2013. Workshop and tutorial proposals are due May 10, 2013. Posters & demonstrations submissions are due July 25, 2013.

I suppose it had to happen.

Instead of asking adding machines for their opinions, someone would decide to ask the creators of adding machines for theirs.

I first saw this at: New AAAI Conference on Human Computation and Crowdsourcing by Shar Steed.

Human Computation: Core Research Questions and State of the Art

Thursday, September 29th, 2011

Human Computation: Core Research Questions and State of the Art by Luis von Ahn and Edith Law. (> 300 slide tutorial) See also: Human Computation by Edith Law and Luis von Ahn.

Abstract from the book:

Human computation is a newand evolving research area that centers around harnessing human intelligence to solve computational problems that are beyond the scope of existing Artificial Intelligence (AI) algorithms.With the growth of the Web, human computation systems can now leverage the abilities of an unprecedented number of people via the Web to perform complex computation.There are various genres of human computation applications that exist today. Games with a purpose (e.g., the ESP Game) specifically target online gamers who generate useful data (e.g., image tags) while playing an enjoyable game.Crowdsourcing marketplaces (e.g.,Amazon MechanicalTurk) are human computation systems that coordinate workers to perform tasks in exchange for monetary rewards. In identity verification tasks, users perform computation in order to gain access to some online content; an example is reCAPTCHA, which leverages millions of users who solve CAPTCHAs every day to correct words in books that optical character recognition (OCR) programs fail to recognize with certainty.

This book is aimed at achieving four goals: (1) defining human computation as a research area; (2) providing a comprehensive review of existing work; (3) drawing connections to a wide variety of disciplines, including AI, Machine Learning, HCI, Mechanism/Market Design and Psychology, and capturing their unique perspectives on the core research questions in human computation; and (4) suggesting promising research directions for the future.

You may also want to see Luis van Ahn in a Google Techtalk video from about five years ago:

July 26, 2006 Luis von Ahn is an assistant professor in the Computer Science Department at Carnegie Mellon University, where he also received his Ph.D. in 2005. Previously, Luis obtained a B.S. in mathematics from Duke University in 2000. He is the recipient of a Microsoft Research Fellowship. ABSTRACT Tasks like image recognition are trivial for humans, but continue to challenge even the most sophisticated computer programs. This talk introduces a paradigm for utilizing human processing power to solve problems that computers cannot yet solve. Traditional approaches to solving such problems focus on improving software. I advocate a novel approach: constructively channel human brainpower using computer games. For example, the ESP Game, described in this talk, is an enjoyable online game — many people play over 40 hours a week — and when people play, they help label images on the Web with descriptive keywords. These keywords can be used to significantly improve the accuracy of image search. People play the game not because they want to help, but because they enjoy it. I describe other examples of “games with a purpose”: Peekaboom, which helps determine the location of objects in images, and Verbosity, which collects common-sense knowledge. I also explain a general approach for constructing games with a purpose.

A rapidly developing and exciting area of research. Perhaps your next topic map may be authored or maintained by a combination of entities.