Archive for the ‘Mechanical Turk’ Category

New version of Get-Another-Label available

Monday, October 22nd, 2012

New version of Get-Another-Label available by Panos Ipeirotis.

From the post:

I am often asked what type of technique I use for evaluating the quality of the workers on Mechanical Turk (or on oDesk, or …). Do I use gold tests? Do I use redundancy?

Well, the answer is that I use both. In fact, I use the code “Get-Another-Label” that I have developed together with my PhD students and a few other developers. The code is publicly available on Github.

We have updated the code recently, to add some useful functionality, such as the ability to pass (for evaluation purposes) the true answers for the different tasks, and get back answers about the quality of the estimates of the different algorithms.

Panos continues his series on the use of crowd sourcing.

Just a thought experiment at the moment but could semantic gaps between populations be “discovered” by use of crowd sourcing?

That is to create tasks that require “understanding” some implicit semantic in the task and then collecting the answer.

There being no “incorrect” answers but answers that reflect the differing perceptions of the semantics of the task.

A way to get away from using small groups of college students for such research? (Nothing against small groups of college students but they best represent small groups of college students. May need a broader semantic range.)

Clockwork Raven uses humans to crunch your Big Data

Saturday, August 18th, 2012

Clockwork Raven uses humans to crunch your Big Data (Powered by an army of twits) by Elliot Bentley.

Twitter, the folks with the friendly API, ;-), have open sourced Clockwork Raven, a Twitter based means to upload small tasks to Mechanical Turk.

You can give users a full topic map editing/ontology creation tool (and train them in its use) or, you can ask very precise questions and crunch the output.

Not appropriate for every task but I suspect good enough for a number of them.

New Mechanical Turk Categorization App

Saturday, May 19th, 2012

New Mechanical Turk Categorization App

Categorization is one of the more popular use cases for the Amazon Mechanical Turk. A categorization HIT (Human Intelligence Task) asks the Worker to select from a list of options. Our customers use HITs of this type to assign product categories, match URLs to business listings, and to discriminate between line art and photographs.

Using our new Categorization App, you can start categorizing your own items or data in minutes, eliminating the learning curve that has traditionally accompanied this type of activity. The app includes everything that you need to be successful including:

  1. Predefined HITs (no HTML editing required).
  2. Pre-qualified Master Workers (see Jinesh’s previous blog post on Mechanical Turk Masters).
  3. Price recommendations based on complexity and comparable HITs.
  4. Analysis tools.

The Categorization App guides you through the four simple steps that are needed to create your categorization project.

I thought the contrast between gamers (the GPU post) and MTurkers would be a nice to close the day. ­čśë

Although, there are efforts to create games where useful activity happens, whether intended or not. (Would that take some of the joy out of a game?)

If you use this particular app, please blog or post a note about your experieince.


Mechanical Turk vs oDesk: My experiences

Sunday, February 19th, 2012

Mechanical Turk vs oDesk: My experiences by Panos Ipeirotis.

From the post:

A question that I receive often is how to structure tasks on Mechanical Turk for which it is necessary for the workers to pass training before doing the task. My common answer to most such question is that Mechanical Turk is not the ideal environment for such tasks: When training and frequent interaction is required, an employer is typically better off by using a site such as oDesk to hire people for the long term to do the job.

Like most things, whether you choose oDesk or Amazon’s Mechanical Turk, should not be an automatic or knee-jerk reaction.

Panos is an “academic-in-residence” with oDesk but even handedly points out when oDesk or Mechanical Turk would be the better choice. Depends on the task at hand and a number of other factors.

If you are considering using either service now or in the future, this is definitely an article you need to keep close at hand.

Geo Analytics Tutorial – Where 2.0 2011

Friday, April 22nd, 2011

Geo Analytics Tutorial – Where 2.0 2011

Very cool set of slides on geo analytics from Pete Skomoroch.

Includes use of Hadoop, Pig, Mechanical Turk.