Archive for the ‘oDesk’ Category

New version of Get-Another-Label available

Monday, October 22nd, 2012

New version of Get-Another-Label available by Panos Ipeirotis.

From the post:

I am often asked what type of technique I use for evaluating the quality of the workers on Mechanical Turk (or on oDesk, or …). Do I use gold tests? Do I use redundancy?

Well, the answer is that I use both. In fact, I use the code “Get-Another-Label” that I have developed together with my PhD students and a few other developers. The code is publicly available on Github.

We have updated the code recently, to add some useful functionality, such as the ability to pass (for evaluation purposes) the true answers for the different tasks, and get back answers about the quality of the estimates of the different algorithms.

Panos continues his series on the use of crowd sourcing.

Just a thought experiment at the moment but could semantic gaps between populations be “discovered” by use of crowd sourcing?

That is to create tasks that require “understanding” some implicit semantic in the task and then collecting the answer.

There being no “incorrect” answers but answers that reflect the differing perceptions of the semantics of the task.

A way to get away from using small groups of college students for such research? (Nothing against small groups of college students but they best represent small groups of college students. May need a broader semantic range.)

Why oDesk has no spammers

Saturday, October 20th, 2012

Why oDesk has no spammers by Panos Ipeirotis.

From the post:

So, in my last blog post, I described a brief outline on how to use oDesk to execute automatically a set of tasks, in a “Mechanical Turk” style (i.e., no interviews for hiring and completely computer-mediated process for posting a job, hiring, and ending a contract).

A legitimate question by appeared in the comments:

“Well, the concept is certainly interesting. But is there a compelling reason to do microstasks on oDesk? Is it because oDesk has a rating system?”

So, here is my answer: If you hire contractors on oDesk you will not run into any spammers, even without any quality control. Why is that? Is there a magic ingredient at oDesk? Short answer: Yes, there is an ingredient: Lack of anonymity!

Well, when you put it that way. ­čśë

Question: How open are your topic maps?

Question: Would you use lack of anonymity to prevent spam in a publicly curated topic map?

Question: If we want a lack of anonymity to provide transparency and accountability in government, why isn’t that the case with public speech?

Using oDesk for microtasks [Data Semantics – A Permanent Wait?]

Monday, October 15th, 2012

Using oDesk for microtasks by Panos Ipeirotis.

From the post:

Quite a few people keep asking me about Mechanical Turk. Truth be told, I have not used MTurk for my own work for quite some time. Instead I use oDesk to get workers for my tasks, and, increasingly, for my microtasks as well.

When I mention that people can use oDesk for micro-tasks, people get often surprised: “oDesk cannot be used through an API, it is designed for human interaction, right?” Oh well, yes and no. Yes, most jobs require some form of interviewing, but there are certainly jobs where you do not need to manually interview a worker before engaging them. In fact, with most crowdsourcing jobs having both the training and the evaluation component built in the working process, the manual interview is often not needed.

For such crowdsourcing-style jobs, you can use the oDesk API to automate the hiring of workers to work on your tasks. You can find the API at (Saying that the API page is, ahem, badly designed, is an understatement. Nevertheless, it is possible to figure out how to use it, relatively quickly, so let’s move on.)

Panos promises future posts with the results of crowd-sourcing experiments with oDesk.

Looking forward to it because waiting for owners of data to disclose semantics looks like a long wait.

Perhaps a permanent wait.

And why not?

If the owners of data “know” the semantics of their data, what advantage do they get from telling you? What is their benefit?

If you guessed “none,” go to the head of the class.

We can either wait for crumbs of semantics to drop off the table or we can setup our own table to produce semantics.

Which one sounds quicker to you?

Mechanical Turk vs oDesk: My experiences

Sunday, February 19th, 2012

Mechanical Turk vs oDesk: My experiences by Panos Ipeirotis.

From the post:

A question that I receive often is how to structure tasks on Mechanical Turk for which it is necessary for the workers to pass training before doing the task. My common answer to most such question is that Mechanical Turk is not the ideal environment for such tasks: When training and frequent interaction is required, an employer is typically better off by using a site such as oDesk to hire people for the long term to do the job.

Like most things, whether you choose oDesk or Amazon’s Mechanical Turk, should not be an automatic or knee-jerk reaction.

Panos is an “academic-in-residence” with oDesk but even handedly points out when oDesk or Mechanical Turk would be the better choice. Depends on the task at hand and a number of other factors.

If you are considering using either service now or in the future, this is definitely an article you need to keep close at hand.