image

Amazon Mechanical Turk is a great tool; however, it is relatively difficult to use if you want to create and manage your HITs.

MIT’s CSAIL’s group has created a very cool tool called TurKit to help you to manage the task much easier. However, it may take you sometime to get started. Let me walk you through this process (it takes me almost a day to figure out the details).

Preparation

Before you start, set up your Amazon Mechanical Turk (MTurk) account and an S3 server account.

Step 1: To sign up for MTurk account, visit https://www.mturk.com/mturk/welcome. You need to use a US address to set it up. You also need a valid credit card. If you are not living in the US, it’s fine as long as you can provide a valid US address.

Step 2: You also need to setup a SandBox account for testing purposes: go to https://requestersandbox.mturk.com to setup your SandBox account.

Step 3: Sign up an S3 server account with Amazon http://aws.amazon.com/s3/.

Download and install TurKit

Step 4: visit http://code.google.com/p/turkit/downloads/list, and download the latest .jar file (current version is TurKit-0.2.7.jar)

Step 5: Make sure you have Java installed on your machine. If you haven’t done that, visit http://java.com/en/download/index.jsp to install Java on your machine.

Step 6: Move the jar file to a folder you like, and double click on it to launch the application. You will see the following application interface.

image

In this interface, you first need to set you Access ID and Key in the property tab. Follow the link provided to retrieve your ID and Key and copy-paste them in the proper location.

image

Now, enter the following piece of code. This piece of code is simplified and modified version from this code http://groups.csail.mit.edu/uid/turkit/samples/sample-writing-s3.js

<<insert code here>>

If you copy and paste the original code from the above code, you will receive a “Failed to create HIT” error (see image below). The reason is that this piece of code did not add an important parameter in the code.

image

The missing statement is: “autoApprovalDelayInSeconds : 60” in the createHIT method. Amazon has changed its API to require this parameter to be set to create a HIT.

The correct code is the following.

[javascript]
print("Your balance is: " + mturk.getAccountBalance())

var text = "…please improve me…"

var webpage = createWebpageFromTemplate(<div>
<img src="http://groups.csail.mit.edu/uid/turkit/www/nut_people.jpg" alt="description not available"></img>
<ul>
<li>Please improve the description for this image.</li>
<li>People will vote whether to approve your work.</li>
</ul>
<textarea style="width:500px;height:170px" name="newText">{text}</textarea>
<input type="submit" value="Submit"></input>
</div>)

// create a HIT on MTurk using the webpage
var hitId = mturk.createHIT({
title : "Improve Text",
desc : "Improve a small paragraph toward a goal.",
url : webpage,
height : 800,
reward : 0.02,
autoApprovalDelayInSeconds : 60
})

// wait for the HIT to be done,
// (NOTE: the program will probably stop here
// the first time it is executed,
// since people will not have had
// a chance yet to complete the HIT)
var hit = mturk.waitForHIT(hitId)

[/javascript]

image

Click “Run” to execute the script. You will have a web page created on the S3 server that looks like the following.

image

And a HIT task created on the SandBox that looks like the following

image

Once you have finished the task, you can delete the webpage and the task by clicking on reset database.

Additional example scripts for TurKit: http://groups.csail.mit.edu/uid/deneme/files/proofread/proofread.js

This is very cool, right? Just to let you know, TurKit is not the only tool that can connect with MTurk. There are two other tools:

Written by Shengdong Zhao

Shen is an Associate Professor in the Computer Science Department, National University of Singapore (NUS). He is the founding director of the NUS-HCI Lab, specializing in research and innovation in the area of human computer interaction.