├── .gitignore ├── LICENSE ├── README.md ├── easyturk ├── __init__.py ├── easyturk.py ├── evaluate.py ├── interface.py ├── render.py └── templates │ ├── annotate_bbox.html │ ├── easyturk.html │ ├── evaluation │ ├── easyturk.html │ ├── home.html │ └── verify_relationship.html │ ├── javascript │ ├── bbox-drawer.js │ ├── bleu.js │ └── iou.js │ ├── verify_bbox.html │ ├── verify_caption.html │ ├── verify_question_answer.html │ ├── verify_relationship.html │ └── write_caption.html └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | env/ 2 | *.pyc 3 | rendered_template.html 4 | easyturk/rendered_template.html 5 | easyturk/data 6 | easyturk/scripts 7 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | COPYRIGHT 2 | 3 | Copyright (c) 2019, Ranjay Krishna. 4 | All rights reserved. 5 | 6 | Each contributor holds copyright over their respective contributions. 7 | The project versioning (Git) records all such contribution source information. 8 | 9 | LICENSE 10 | 11 | The MIT License (MIT) 12 | 13 | Permission is hereby granted, free of charge, to any person obtaining a copy 14 | of this software and associated documentation files (the "Software"), to deal 15 | in the Software without restriction, including without limitation the rights 16 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 17 | copies of the Software, and to permit persons to whom the Software is 18 | furnished to do so, subject to the following conditions: 19 | 20 | The above copyright notice and this permission notice shall be included in all 21 | copies or substantial portions of the Software. 22 | 23 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 24 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 25 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 26 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 27 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 28 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 29 | SOFTWARE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # EasyTurk: A Wrapper for Custom AMT Tasks 2 | 3 | EasyTurk is an easy to use wrapper to launch and get responses for custom Amazon Mechanical Turk tasks. 4 | 5 | If you are using this repository for your research, please use the following citation: 6 | 7 | ``` 8 | @misc{krishna2019easyturk, 9 | author = {Krishna, Ranjay}, 10 | title = {EasyTurk: A Wrapper for Custom AMT Tasks}, 11 | year = {2019}, 12 | publisher = {GitHub}, 13 | journal = {GitHub repository}, 14 | howpublished = {\url{https://github.com/ranjaykrishna/easyturk}} 15 | } 16 | ``` 17 | 18 | #### Why should you use EasyTurk? 19 | 20 | EasyTurk is meant for Amazon Mechanical Turk requestors who want to programmatically create, launch and retrieve Amazon Mechanical Turk microtasks. EasyTurk allows you to build custom html/javascript tasks and control your tasks using an easy python wrapper. 21 | 22 | 23 | #### Dependencies 24 | 25 | EasyTurk requires jinja2 to compile your custom html and javascript tasks. It requires python2.7 or higher to use the API. It also requires that you have boto3 (AMT's python interface) installed. 26 | 27 | 28 | ## Installing EasyTurk 29 | 30 | First let's install the code. 31 | ``` 32 | # Clone the repository. 33 | git clone git@github.com:ranjaykrishna/easyturk.git 34 | cd easyturk 35 | 36 | # Create a virtual environment. 37 | # You might want to use -p python3 option 38 | # if you want to use EasyTurk with python 3. 39 | virtualenv env 40 | 41 | # Activate your environment. 42 | source env/bin/activate 43 | 44 | # Install all the requirements. 45 | pip install -r requirements 46 | ``` 47 | 48 | 49 | Now, let's install your AMT access keys. 50 | ``` 51 | aws configure --profile mturk 52 | ``` 53 | This will prompt you to enter your access key and secret key for your AMT account. 54 | If you don’t have an AWS account already, visit [this website](https://aws.amazon.com) and create an account. 55 | If you don’t have an MTurk Requester account already, visit [this website](https://requester.mturk.com) and create a new account. 56 | 57 | 58 | #### Easy check to make sure everything is in place. 59 | 60 | Run python in your terminal. Let's see if we can access your account information. 61 | ``` 62 | from easyturk import EasyTurk 63 | et = EasyTurk(sandbox=False) 64 | print(et.get_account_balance()) 65 | ``` 66 | This should print out how much balance you have left in your account. If you do not see the amount, then there setup was not successful. 67 | 68 | #### Example tasks that you can use out of the box: 69 | 70 | 1. Image captioning: `easyturk/templates/write_caption.html` contains a image captioning task where workers are asked to write captions for each image. 71 | 2. Verification: `easyturk/templates/verify_caption.html` contains a verification task that asks users to verify that a certain caption is correct for a given image. 72 | 3. Bounding box: `easyturk/templates/annotate_bbox.html` contains a bounding box task that asks users to annotate multiple objects per image. 73 | 3. Verify bounding box: `easyturk/templates/verify_bbox.html` contains a bounding box task that asks users verify whether a bounding box has been annotated correctly. 74 | 4. Verify visual relationships: `easyturk/templates/verify_relationship.html` contains a task that asks users to verify whether a visual relationship has been annotated correctly. 75 | 76 | In the remainer of this tutorial, we will explain how you can launch the captioning task and how you can create your own tasks. The same workflow can be used for the other tasks as well. 77 | 78 | 79 | ## Launching an example image-captioning task on AMT. 80 | 81 | In `easyturk/templates/write_caption.html`, there is an example image-captioning task I have already created for you. You can use that task as a reference to create your own task later. For now, let's try and see if we can render the task, launch it to AMT, then retrieve the results, and approve the assignment. 82 | 83 | #### Step 1: Render the task. 84 | The following command should render the template you built. This should allow you to locally open the task in your browser and check to make sure the functionality works. Of course, when you press the submit button, nothing will happen as you are running the task locally. But once we launch this task on AMT, the submit button will pull the worker's responses to your task and send them to AMT. We will later be able to retrieve those results. 85 | ``` 86 | python easyturk/render.py --template write_caption.html --output rendered_template.html 87 | open rendered_template.html 88 | ``` 89 | 90 | #### Step 2: Launching the task. 91 | The following block of code will launch the task to AMT. We start by creating a list of image urls we want to caption. The example task expects the data to be in a list of objects, where each element in the list is a dictionary containing a `url` field. Once a worker finishes the task, the task will return a similar list, except each element will contain a `caption` field. Run the following in your python interpretor: 92 | ``` 93 | from easyturk import interface 94 | data = [ 95 | { 96 | 'url': 'http://visualgenome.org/static/images/collect_sentences/dogs-playing.jpg', 97 | }, 98 | { 99 | 'url': 'http://visualgenome.org/static/images/collect_sentences/computer.png', 100 | } 101 | ] 102 | hit_ids = interface.launch_caption(data, reward=1, tasks_per_hit=10) 103 | print(hit_ids) 104 | ``` 105 | The above code will launch one HIT that will pay a reward of $1 and caption the two images in the list. Once the HIT is launched, it will return and print out the HITId. This HITId will be used to later retrieve and approve worker's responses. So make sure to NOT lose it. It's good practice to save your HITIds in a database or logfile. But if you do lose it, you can always get it back by queries for all the active HITs you have on AMT. 106 | 107 | `launch_caption` is a custom launch script that sets the title, description, keywords, tasks_per_hit fields. When you later write your own HIT, I recommend create a custom launch function like this one. You can see the source code for the function in `easyturk/interface.py`. 108 | 109 | 110 | #### Step 3: Tracking your task's progress. 111 | You can query for your HIT's progress with the following: 112 | ``` 113 | from easyturk import EasyTurk 114 | et = EasyTurk(sandbox=False) 115 | progress = et.show_hit_progress(hit_ids) 116 | print(progress[hit_ids[0]]) 117 | ``` 118 | The above will print out the progress made for the first HIT in the list. It should print something like: 119 | ``` 120 | {'completed': 0, 'max_assignment': 1} 121 | ``` 122 | 123 | 124 | #### Step 4: Retrieving worker responses. 125 | You can retrieve the work done by workers for the submitted assignments to a HIT using the following code: 126 | ``` 127 | from easyturk import interface 128 | 129 | # approve is set to False because right now we are only fetching the results. You can set it to True to auto-approve the assignments. 130 | results = interface.fetch_completed_hits(hit_ids, approve=False) 131 | print(results[hit_ids[0]]) 132 | ``` 133 | The above code will parse out the responses made by the worker and show you something like the following: 134 | ``` 135 | [ 136 | { 137 | 'url': 'http://visualgenome.org/static/images/collect_sentences/dogs-playing.jpg', 138 | 'caption': 'Two dogs playing with a stick.' 139 | }, 140 | { 141 | 'url': 'http://visualgenome.org/static/images/collect_sentences/computer.png', 142 | 'caption': 'A cluttered room', 143 | } 144 | ] 145 | ``` 146 | 147 | #### Step 5: Approving their work. 148 | If you are happy with the work, you can approve and pay your workers by issuing the following command: 149 | ``` 150 | from easyturk import EasyTurk 151 | et = EasyTurk(sandbox=False) 152 | for hit_id in hit_ids: 153 | et.approve_hit(hit_id) 154 | ``` 155 | 156 | ## Designing your own AMT task. 157 | The best way to learn to create your own tasks is to mimic the high level interface of `easyturk/templates/write_caption.html`. The main contraint to adhere to is making sure that you are using the API provided by `easyturk/templates/easyturk.html`. Currently, EasyTurk assumes that all your tasks you design will reside in one single HTML files in the `easyturk/templates/` directory 158 | 159 | 160 | #### Importing EasyTurk into your custom task. 161 | Simply add the following line before your custom javascript to use the EasyTurk API: 162 | ``` 163 | {% include "easyturk.html" %} 164 | ``` 165 | 166 | #### Loading the data from EasyTurk. 167 | The task contains a `DEFAULT_INPUT` variable that should be set when designing your task such that it follows the input schema you expect. In our example captioning task, that variable is a random list of image urls. It allows us to render and test the functionality of the task locally before launching it. When launched, make sure to call: 168 | ``` 169 | var input = easyturk.getInput(DEFAULT_INPUT); 170 | ``` 171 | This API call will update the input variable with the actual input for the given task. So, each worker will see a specific input, based on the `data` variable you set in Step 2 above. 172 | 173 | #### Enabling the task. 174 | The following API call allows workers to preview the HIT without being able to submit it. Make sure to include it so that when the HIT is accepted by a worker, the task will be enabled. 175 | ``` 176 | if (!easyturk.isPreview()){ 177 | enableTask(); 178 | } 179 | ``` 180 | 181 | #### Setup submission and submit. 182 | To make sure that the task is submittable, keep the `enableTask()` function and make sure it includes the `easyturk.setupSubmit()` call and the `easyturk.setOutput(output)` calls within. Any python object you set the `output` variable to will be the response you get back from the worker. 183 | 184 | #### Overall. 185 | When making your custom task, make sure to import EasyTurk's APIs. Get the input from EasyTurk, enable the hit, setup submission and then set the output that you want returned to you. It's easy! 186 | 187 | 188 | #### More customization. 189 | There is a lot more you can do. Check out the fully documented code in `easyturk.py`. There are more complicated workflows you can create by using those functions. Have fun. 190 | 191 | 192 | ## Contributing to this repository. 193 | 194 | This wrapper is by no means a complete list of functionality offered by AMT. If you feel inclined to contribute and enable more functionality, please send me a message over email (firstnamelastname [at] gmail [dot] com) or twitter ([@RanjayKrishna](https://twitter.com/RanjayKrishna)). Feel free to also send me pull requests. 195 | -------------------------------------------------------------------------------- /easyturk/__init__.py: -------------------------------------------------------------------------------- 1 | from easyturk import EasyTurk 2 | import interface 3 | -------------------------------------------------------------------------------- /easyturk/easyturk.py: -------------------------------------------------------------------------------- 1 | """A bunch of api calls functions that we used to talk to MTurk. 2 | """ 3 | 4 | from boto3 import Session 5 | from datetime import datetime 6 | from jinja2 import Environment 7 | from jinja2 import FileSystemLoader 8 | 9 | import json 10 | import os 11 | import xml 12 | 13 | 14 | class EasyTurk(object): 15 | """Class that contains all the api calls to interface with MTurk. 16 | """ 17 | 18 | def __init__(self, sandbox=True): 19 | """Constructor for EasyTurk. 20 | 21 | Args: 22 | sandbox: Whether we are launching on sandbox. 23 | """ 24 | environments = { 25 | "production": { 26 | "endpoint": "https://mturk-requester.us-east-1.amazonaws.com", 27 | "preview": "https://www.mturk.com/mturk/preview" 28 | }, 29 | "sandbox": { 30 | "endpoint": "https://mturk-requester-sandbox.us-east-1.amazonaws.com", 31 | "preview": "https://workersandbox.mturk.com/mturk/preview" 32 | }, 33 | } 34 | self.sandbox = sandbox 35 | env = (environments['sandbox'] if sandbox 36 | else environments["production"]) 37 | self.session = Session(profile_name='mturk') 38 | self.mtc = self.session.client( 39 | service_name='mturk', 40 | region_name='us-east-1', 41 | endpoint_url=env['endpoint'], 42 | ) 43 | 44 | def create_html_question(self, html, frame_height): 45 | head = ("" 47 | "") 48 | tail = "" 49 | xml = head + str(frame_height) + tail 50 | return xml.format(html) 51 | 52 | def get_jinja_env(self): 53 | """Get a jinja2 Environment object that we can use to find templates. 54 | """ 55 | dir_location = os.path.dirname(os.path.abspath(__file__)) 56 | templates = os.path.join(dir_location, 'templates') 57 | return Environment(loader=FileSystemLoader(templates)) 58 | 59 | def get_account_balance(self): 60 | """Retrieves the account balance. 61 | 62 | Returns: 63 | available account balance. 64 | """ 65 | return self.mtc.get_account_balance()['AvailableBalance'] 66 | 67 | def launch_hit(self, template_location, input_data, reward=0, 68 | frame_height=9000, title=None, description=None, 69 | keywords=None, duration=900, max_assignments=1, 70 | country='US', hits_approved=10000, lifetime=604800, 71 | percent_approved=95): 72 | """Launches a HIT. 73 | 74 | Make sure that none of the arguments are None. 75 | 76 | Returns: 77 | A hit_id. 78 | """ 79 | if self.sandbox: 80 | percent_approved = 0 81 | hits_approved = 0 82 | hit_properties = {'Title': title, 83 | 'Description': description, 84 | 'Keywords': keywords, 85 | 'MaxAssignments': max_assignments, 86 | 'LifetimeInSeconds': lifetime, 87 | 'AssignmentDurationInSeconds': duration, 88 | 'QualificationRequirements': [ 89 | { 90 | 'QualificationTypeId': '00000000000000000040', 91 | 'Comparator': 'GreaterThanOrEqualTo', 92 | 'IntegerValues': [hits_approved] 93 | }, 94 | { 95 | 'QualificationTypeId': '00000000000000000071', 96 | 'Comparator': 'EqualTo', 97 | 'LocaleValues': [ 98 | {'Country': country}, 99 | ], 100 | }, 101 | { 102 | 'QualificationTypeId': '000000000000000000L0', 103 | 'Comparator': 'GreaterThanOrEqualTo', 104 | 'IntegerValues': [percent_approved], 105 | } 106 | ], 107 | 'Reward': str(reward)} 108 | 109 | # Setup HTML Question. 110 | env = self.get_jinja_env() 111 | template = env.get_template(template_location) 112 | template_params = {'input': json.dumps(input_data)} 113 | html = template.render(template_params) 114 | html_question = self.create_html_question(html, frame_height) 115 | 116 | hit_properties['Question'] = html_question 117 | 118 | hit = self.mtc.create_hit(**hit_properties) 119 | return hit 120 | 121 | def _parse_response_from_assignment(self, assignment): 122 | """Parses out the worker's response from the assignment received. 123 | 124 | Args: 125 | assignment: A dictionary describing the assignment from boto. 126 | 127 | Returns: 128 | A Python object or list of the worker's response. 129 | """ 130 | try: 131 | output_tree = xml.etree.ElementTree.fromstring(assignment['Answer']) 132 | output = json.loads(output_tree[0][1].text) 133 | return output 134 | except ValueError as e: 135 | print(e) 136 | return None 137 | 138 | def get_results(self, hit_id, reject_on_fail=False): 139 | """Retrives the output of a hit if it has finished. 140 | 141 | Args: 142 | hit_id: The hit id of the HIT. 143 | reject_on_fail: If the hit returns unparsable answers, 144 | then reject it. 145 | 146 | Returns: 147 | A list of dictionaries with the following fields: 148 | - assignment_id 149 | - hit_id 150 | - worker_id 151 | - output 152 | - submit_time 153 | The number of dictionaries is equal to the number of assignments. 154 | """ 155 | status = ['Approved', 'Submitted', 'Rejected'] 156 | try: 157 | assignments = self.mtc.list_assignments_for_hit( 158 | HITId=hit_id, AssignmentStatuses=status) 159 | except Exception: 160 | return [] 161 | results = [] 162 | for a in assignments['Assignments']: 163 | output = self._parse_response_from_assignment(a) 164 | if output is not None: 165 | results.append({ 166 | 'assignment_id': a['AssignmentId'], 167 | 'hit_id': hit_id, 168 | 'worker_id': a['WorkerId'], 169 | 'output': output, 170 | 'submit_time': a['SubmitTime'], 171 | }) 172 | elif reject_on_fail: 173 | self.mtc.reject_assignment( 174 | AssignmentId=a['AssignmentId'], 175 | RequesterFeedback='Invalid results') 176 | return results 177 | 178 | def delete_hit(self, hit_id): 179 | """Disables a hit. 180 | 181 | Args: 182 | hit_id: The hit id to disable. 183 | 184 | Returns: 185 | A boolean indicating success. 186 | """ 187 | try: 188 | self.mtc.delete_hit(HITId=hit_id) 189 | return True 190 | except Exception: 191 | try: 192 | self.mtc.update_expiration_for_hit( 193 | HITId=hit_id, 194 | ExpireAt=datetime.now()) 195 | self.mtc.delete_hit(HITId=hit_id) 196 | return True 197 | except Exception as e: 198 | print e 199 | return False 200 | 201 | def approve_hit(self, hit_id, reject_on_fail=False, 202 | override_rejection=False): 203 | """Approves a hit so that the worker can get paid. 204 | 205 | Args: 206 | hit_id: The hit id to disable. 207 | reject_on_fail: If the hit returns unparsable answers, 208 | then reject it. 209 | override_rejection: overrides a previous rejection if it exists. 210 | 211 | Returns: 212 | Tuple of approved and rejected assignment ids. 213 | """ 214 | status = ['Approved', 'Submitted', 'Rejected'] 215 | try: 216 | assignments = self.mtc.list_assignments_for_hit( 217 | HITId=hit_id, AssignmentStatuses=status) 218 | except: 219 | return [], [] 220 | approve_ids = [] 221 | reject_ids = [] 222 | for a in assignments['Assignments']: 223 | if a['AssignmentStatus'] == 'Submitted': 224 | output = self._parse_response_from_assignment(a) 225 | if output is not None: 226 | approve_ids.append(a['AssignmentId']) 227 | elif reject_on_fail: 228 | reject_ids.append(a['AssignmentId']) 229 | 230 | for assignment_id in approve_ids: 231 | self.mtc.approve_assignment( 232 | AssignmentId=assignment_id, 233 | RequesterFeedback='Good job', 234 | OverrideRejection=override_rejection) 235 | for assignment_id in reject_ids: 236 | self.mtc.reject_assignment( 237 | AssignmentId=assignment_id, RequesterFeedback='Invalid results') 238 | return approve_ids, reject_ids 239 | 240 | def reject_assignment(self, assignment_id): 241 | """Reject an assignment so that the worker can get paid. 242 | 243 | Args: 244 | assignment_id: An assignment id. 245 | 246 | Returns: 247 | A boolean indicating success. 248 | """ 249 | a = self.mtc.get_assignment(AssignmentId=assignment_id)['Assignment'] 250 | if a['AssignmentStatus'] == 'Submitted': 251 | self.mtc.reject_assignment( 252 | AssignmentId=assignment_id, 253 | RequesterFeedback='Invalid results') 254 | return True 255 | return False 256 | 257 | def approve_assignment(self, assignment_id, reject_on_fail=False, 258 | override_rejection=False): 259 | """Approves an assignment so that the worker can get paid. 260 | 261 | Args: 262 | assignment_id: An assignment id. 263 | reject_on_fail: If the hit returns unparsable answers, 264 | then reject it. 265 | override_rejection: overrides a previous rejection if it exists. 266 | 267 | Returns: 268 | A boolean indicating success. 269 | """ 270 | a = self.mtc.get_assignment(AssignmentId=assignment_id)['Assignment'] 271 | if a['AssignmentStatus'] == 'Submitted': 272 | output = self._parse_response_from_assignment(a) 273 | if output is not None: 274 | self.mtc.approve_assignment( 275 | AssignmentId=assignment_id, 276 | RequesterFeedback='Good job', 277 | OverrideRejection=override_rejection) 278 | return True 279 | elif reject_on_fail: 280 | self.mtc.reject_assignment( 281 | AssignmentId=assignment_id, 282 | RequesterFeedback='Invalid results') 283 | return False 284 | return False 285 | 286 | def show_hit_progress(self, hit_ids): 287 | """Show the progress of the hits. 288 | 289 | Args: 290 | hit_ids: A list of HIT ids. 291 | 292 | Returns: 293 | A dictionary from hit_id to a dictionary of completed 294 | and maximum assignments. 295 | """ 296 | output = {} 297 | for hit_id in hit_ids: 298 | hit = self.mtc.get_hit(HITId=hit_id) 299 | assignments = self.mtc.list_assignments_for_hit( 300 | HITId=hit_id, AssignmentStatuses=['Submitted']) 301 | completed = len(assignments['Assignments']) 302 | max_assignments = hit['HIT']['MaxAssignments'] 303 | output[hit_id] = {'completed': completed, 304 | 'max_assignments': max_assignments} 305 | return output 306 | 307 | def list_hits(self): 308 | """Lists the HITs that have already been launched. 309 | 310 | Returns: 311 | A list of HITs. 312 | """ 313 | hits = self.mtc.list_hits()['HITs'] 314 | return hits 315 | -------------------------------------------------------------------------------- /easyturk/evaluate.py: -------------------------------------------------------------------------------- 1 | """Script to render a completed EasyTurk task. 2 | """ 3 | 4 | from flask import Flask 5 | from flask import render_template 6 | from flask import request 7 | 8 | import json 9 | import os 10 | 11 | from easyturk import EasyTurk 12 | 13 | 14 | # Global server variables. 15 | app = Flask(__name__) 16 | 17 | 18 | def get_e_filename(filename): 19 | """Modifies the filename to include _e_ in front. 20 | 21 | Args: 22 | filename: A filename (ex, directory/filename.json) 23 | 24 | Returns: 25 | A string directory/_e_filename.json 26 | """ 27 | dirname = os.path.dirname(filename) 28 | index = len(dirname) + 1 29 | dirname = filename[0:index] 30 | filename = filename[index:] 31 | return dirname + '_e_' + filename 32 | 33 | 34 | def convert(results): 35 | """Converts the results to include metadata for navigation. 36 | 37 | Args: 38 | results: A dictionary mapping from hit_id to hit data. 39 | 40 | Returns: 41 | A dictionary containing: 42 | - hits: All hits ordered by workers. 43 | - workers: Mapping from worker_id to indices into hits. 44 | """ 45 | # Organize all the data by worker id. 46 | worker_map = {} 47 | for hit_id, hit in results.items(): 48 | for assignment in hit: 49 | worker_id = assignment['worker_id'] 50 | if worker_id not in worker_map: 51 | worker_map[worker_id] = [] 52 | worker_map[worker_id].append(assignment) 53 | 54 | # Create the hits and worker index mapping structures. 55 | hits = [] 56 | worker_indices = {} 57 | worker_ids = [] 58 | for worker_id, assignments in worker_map.items(): 59 | worker_indices[worker_id] = range( 60 | len(hits), len(hits) + len(assignments)) 61 | hits.extend(assignments) 62 | worker_ids.append(worker_id) 63 | output = {'hits': hits, 64 | 'workers': worker_indices, 65 | 'worker_ids': worker_ids} 66 | return output 67 | 68 | 69 | @app.route('/task') 70 | def task(): 71 | """Visualizes the results received for a given task. 72 | """ 73 | # Compile the template. 74 | task = request.args['task'] 75 | results_file = request.args['results'] 76 | eresults_file = get_e_filename(results_file) 77 | if os.path.exists(eresults_file): 78 | results = json.load(open(eresults_file)) 79 | else: 80 | results = json.load(open(results_file)) 81 | results = convert(results) 82 | json.dump(results, open(eresults_file, 'w')) 83 | return render_template( 84 | 'evaluation/' + task, 85 | results=results, 86 | task=results_file, 87 | eresults_file=eresults_file) 88 | 89 | 90 | @app.route('/interface', methods=['POST']) 91 | def interface(): 92 | """Endpoint that rejects and approves work. 93 | """ 94 | et = EasyTurk(sandbox=False) 95 | if request.method != 'POST': 96 | return 'Fail' 97 | assignment_ids = json.loads(request.form['assignment_ids']) 98 | approve = json.loads(request.form['approve']) 99 | for assignment_id in assignment_ids: 100 | if approve: 101 | et.approve_assignment(assignment_id) 102 | else: 103 | et.reject_assignment(assignment_id) 104 | eresults_file = request.form['eresults_file'] 105 | results = json.load(open(eresults_file)) 106 | for hit in results['hits']: 107 | if hit['assignment_id'] in assignment_ids: 108 | hit['approve'] = approve 109 | json.dump(results, open(eresults_file, 'w')) 110 | return 'Succcess' 111 | 112 | 113 | @app.route('/') 114 | def home(): 115 | return render_template('evaluation/home.html') 116 | 117 | 118 | if __name__=='__main__': 119 | app.run() 120 | -------------------------------------------------------------------------------- /easyturk/interface.py: -------------------------------------------------------------------------------- 1 | """Functions to launch, retrieve, and parse specific EasyTurk tasks. 2 | """ 3 | 4 | from easyturk import EasyTurk 5 | 6 | 7 | def launch_verify_question_answer(data, reward=1.00, tasks_per_hit=50, sandbox=False): 8 | """Launches HITs to ask workers to verify bounding boxes. 9 | 10 | Args: 11 | data: List containing image urls, questions and answers, for the task. 12 | reward: A postive valued dollar amount per task. 13 | tasks_per_hit: Number of images per hit. 14 | sandbox: Whether to interact on sandbox or production. 15 | 16 | Returns: 17 | A list of hit ids that have been launched. 18 | """ 19 | et = EasyTurk(sandbox=sandbox) 20 | template = 'verify_question_answer.html' 21 | hit_ids = [] 22 | i = 0 23 | while i < len(data): 24 | hit = et.launch_hit( 25 | template, data[i:i+tasks_per_hit], reward=reward, 26 | title='Verify the answer to a question about an picture', 27 | description=('Verify whether an answer to a question about a picture is correct.'), 28 | keywords='image, text, picture, answer, question, relationship') 29 | hit_id = hit['HIT']['HITId'] 30 | hit_ids.append(hit_id) 31 | i += tasks_per_hit 32 | return hit_ids 33 | 34 | 35 | def launch_verify_relationship(data, reward=1.00, tasks_per_hit=30, sandbox=False): 36 | """Launches HITs to ask workers to verify bounding boxes. 37 | 38 | Args: 39 | data: List containing image urls, relationships, for the task. 40 | reward: A postive valued dollar amount per task. 41 | tasks_per_hit: Number of images per hit. 42 | sandbox: Whether to interact on sandbox or production. 43 | 44 | Returns: 45 | A list of hit ids that have been launched. 46 | """ 47 | et = EasyTurk(sandbox=sandbox) 48 | template = 'verify_relationship.html' 49 | hit_ids = [] 50 | i = 0 51 | while i < len(data): 52 | hit = et.launch_hit( 53 | template, data[i:i+tasks_per_hit], reward=reward, 54 | title='Verify relationships between objects in pictures', 55 | description=('Verify whether the relationships are correctly identified in pictures.'), 56 | keywords='image, text, picture, object, bounding box, relationship') 57 | hit_id = hit['HIT']['HITId'] 58 | hit_ids.append(hit_id) 59 | i += tasks_per_hit 60 | return hit_ids 61 | 62 | 63 | def launch_verify_bbox(data, reward=1.00, tasks_per_hit=30, sandbox=False): 64 | """Launches HITs to ask workers to verify bounding boxes. 65 | 66 | Args: 67 | data: List containing image urls, objects, for the task. 68 | reward: A postive valued dollar amount per task. 69 | tasks_per_hit: Number of images per hit. 70 | sandbox: Whether to interact on sandbox or production. 71 | 72 | Returns: 73 | A list of hit ids that have been launched. 74 | """ 75 | et = EasyTurk(sandbox=sandbox) 76 | template = 'verify_bbox.html' 77 | hit_ids = [] 78 | i = 0 79 | while i < len(data): 80 | hit = et.launch_hit( 81 | template, data[i:i+tasks_per_hit], reward=reward, 82 | title='Verify objects in pictures', 83 | description=('Verify whether objects are correctly identified in pictures.'), 84 | keywords='image, text, picture, object, bounding box') 85 | hit_id = hit['HIT']['HITId'] 86 | hit_ids.append(hit_id) 87 | i += tasks_per_hit 88 | return hit_ids 89 | 90 | 91 | def launch_caption(data, reward=1.00, tasks_per_hit=10, sandbox=False): 92 | """Launches HITs to ask workers to caption images. 93 | 94 | Args: 95 | data: List containing image urls for the task. 96 | reward: A postive valued dollar amount per task. 97 | tasks_per_hit: Number of images per hit. 98 | sandbox: Whether to interact on sandbox or production. 99 | 100 | Returns: 101 | A list of hit ids that have been launched. 102 | """ 103 | et = EasyTurk(sandbox=sandbox) 104 | template = 'write_caption.html' 105 | hit_ids = [] 106 | i = 0 107 | while i < len(data): 108 | hit = et.launch_hit( 109 | template, data[i:i+tasks_per_hit], reward=reward, 110 | title='Caption some pictures', 111 | description=('Write captions about the contents of images.'), 112 | keywords='image, caption, text') 113 | hit_id = hit['HIT']['HITId'] 114 | hit_ids.append(hit_id) 115 | i += tasks_per_hit 116 | return hit_ids 117 | 118 | 119 | def fetch_completed_hits(hit_ids, approve=True, sandbox=False): 120 | """Grabs the results for the hit ids. 121 | 122 | Args: 123 | hit_ids: A list of hit ids to fetch. 124 | approve: Whether to approve the hits that have been submitted. 125 | sandbox: Whether to interact on sandbox or production. 126 | 127 | Returns: 128 | A dictionary from hit_id to the result, if that hit_id has 129 | been submitted. 130 | """ 131 | et = EasyTurk(sandbox=sandbox) 132 | output = {} 133 | for hit_id in hit_ids: 134 | results = et.get_results(hit_id, reject_on_fail=False) 135 | if len(results) > 0: 136 | output[hit_id] = results 137 | if approve: 138 | for assignment in results: 139 | assignment_id = assignment['assignment_id'] 140 | et.approve_assignment(assignment_id) 141 | return output 142 | -------------------------------------------------------------------------------- /easyturk/render.py: -------------------------------------------------------------------------------- 1 | """Script to render a given EasyTurk task. 2 | """ 3 | 4 | import argparse 5 | 6 | from easyturk import EasyTurk 7 | 8 | 9 | if __name__ == '__main__': 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument('--template', required=True) 12 | parser.add_argument('--output', required=True) 13 | args = parser.parse_args() 14 | 15 | # Compile the template. 16 | et = EasyTurk() 17 | env = et.get_jinja_env() 18 | template = env.get_template(args.template) 19 | 20 | html = template.render({'input': ''}) 21 | 22 | # Save to output. 23 | with open(args.output, 'w') as f: 24 | f.write(html) 25 | -------------------------------------------------------------------------------- /easyturk/templates/annotate_bbox.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | Annotate objects in images 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 40 | 41 | 42 | 43 | 44 |
45 |

Describe image regions with text snippets

46 | 47 |
48 | Hi there, we need your help annotating objects in images.
49 | 50 | 105 | 106 |
107 | 108 |
109 | 110 |
111 |

Your task is below

112 |
113 |
114 |
115 |
116 |
117 |
118 |
119 |
120 |
121 |
122 |
123 |
124 | 125 | 126 | / 127 | 128 | 129 |
130 |
131 |
132 | 133 | 134 | 135 | 136 | {% include "easyturk.html" %} 137 | 140 | 141 | 426 | 427 | 428 | -------------------------------------------------------------------------------- /easyturk/templates/easyturk.html: -------------------------------------------------------------------------------- 1 | 4 |
5 | 6 | 7 | 8 |
9 | 53 | -------------------------------------------------------------------------------- /easyturk/templates/evaluation/easyturk.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | Verify whether relationships between objects in pictures are accurate 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 54 | 55 | 56 | 57 |
58 |
59 |

Showing tasks for {{task}}.

60 |
61 |
62 |
63 | 64 |
65 |
66 | 67 |
68 |
69 | 70 |
71 |
72 | 73 |
74 |
75 |
76 |
77 | 78 |
79 |
80 | 81 |
82 |
83 | 84 |
85 |
86 | 87 |
88 |
89 |
90 |

91 | Assignment: / , ---- 92 | Worker: / 93 |

94 |

Assignment id:

95 |

Worker id:

96 |

Approved?:

97 |
98 |
99 | 100 | 101 | {% block content %}{% endblock %} 102 | 103 | 276 | 277 | 278 | 279 | -------------------------------------------------------------------------------- /easyturk/templates/evaluation/home.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | Verify whether relationships between objects in pictures are accurate 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 29 | 30 | 31 |
32 |
33 |

Welcome to EasyTurk's evaluation server.

34 |
35 |
36 |
37 |

Task name:

38 |
39 |
40 | 41 |
42 |
43 |
44 |
45 |

Results location:

46 |
47 |
48 | 49 |
50 |
51 |
52 |
53 | 54 |
55 |
56 |
57 | 64 | 65 | 66 | -------------------------------------------------------------------------------- /easyturk/templates/evaluation/verify_relationship.html: -------------------------------------------------------------------------------- 1 | {% extends "evaluation/easyturk.html" %} 2 | {% block content %} 3 | 11 |
12 |
13 |
14 | 15 | 16 | 19 | 20 | 70 | {% endblock %} 71 | -------------------------------------------------------------------------------- /easyturk/templates/javascript/bbox-drawer.js: -------------------------------------------------------------------------------- 1 | /**This file enables utility functions for drawing and displaying bounding boxes on images. 2 | * 3 | * Requirements: 4 | * - JQuery 5 | */ 6 | 7 | var ETJS = (function(etjs) { 8 | 9 | etjs.BBoxDrawer = function(div, image_url, canvas_width, options) { 10 | /**Initialize the bbox drawing utility tool. 11 | * 12 | * Args: 13 | * div: The HTML element where the drawer will be initialized. 14 | * image_url: The image to display in that div. 15 | * canvas_width: The width of the image to be displayed. 16 | * options: A dictionary of possible customizations. Check DEFAULT_OPTIONS above. 17 | */ 18 | 19 | var DEFAULT_OPTIONS = { 20 | static_box_color: '#f00', 21 | bbox_color: '#ff0', 22 | bbox_line_width: 2.0, 23 | click_radius: 15, 24 | handle_opacity: 0.2, 25 | dot_small_radius: 5, 26 | dot_big_radius: 10, 27 | image_opacity: 1.0, 28 | max_height: null, 29 | max_width: null, 30 | image_width: null, 31 | callback: function() {}, 32 | text_box_padding: 4, 33 | text_font_size: 20, 34 | text_font: 20 + 'px sans-serif', 35 | }; 36 | var that = (this === etjs ? {} : this); 37 | 38 | that.merge_options = function(options, default_options) { 39 | if (typeof(options) === 'undefined') { 40 | options = {}; 41 | } 42 | 43 | for (var opt in default_options) { 44 | if (default_options.hasOwnProperty(opt) 45 | && !options.hasOwnProperty(opt)) { 46 | options[opt] = default_options[opt]; 47 | } 48 | } 49 | return options; 50 | } 51 | 52 | options = that.merge_options(options, DEFAULT_OPTIONS); 53 | 54 | var scale = null; 55 | var canvas = null; 56 | var canvas_pos = null; 57 | var ctx = null; 58 | 59 | // Most recent positions of the mouse and of a click. 60 | var click_pos = null; 61 | var mouse_pos = null; 62 | 63 | // Whether the box has been been finalized after the initial drag. 64 | var bbox_drawn = false; 65 | 66 | // Object with properties x, y, w, h. 67 | var bbox = null; 68 | var static_boxes = []; 69 | 70 | var object_pos = null; 71 | var object_name = null; 72 | 73 | var resize_direction = null; 74 | var old_bbox = null; 75 | 76 | var disabled = true; 77 | 78 | var cursors = { 79 | 'UL': 'nw-resize', 80 | 'UR': 'ne-resize', 81 | 'BL': 'sw-resize', 82 | 'BR': 'se-resize', 83 | 'U': 'n-resize', 84 | 'R': 'e-resize', 85 | 'L': 'w-resize', 86 | 'B': 's-resize', 87 | }; 88 | 89 | var img = new Image(); 90 | img.onload = setup; 91 | img.src = image_url; 92 | 93 | that.getBoxPosition = function() { 94 | /** Scales and returns the bbox location relative to the original image size. 95 | * 96 | * Returns: 97 | * A dictionary containing x, y, w and h. 98 | */ 99 | if (bbox) { 100 | // Convert to image coordinates 101 | return { 102 | 'x': Math.floor(toImageCoords(bbox.x)), 103 | 'y': Math.floor(toImageCoords(bbox.y)), 104 | 'w': Math.floor(toImageCoords(bbox.w)), 105 | 'h': Math.floor(toImageCoords(bbox.h)) 106 | }; 107 | 108 | } 109 | return null; 110 | } 111 | 112 | that.setBoxPosition = function(b) { 113 | /** Set the bbox position and draws the bbox. 114 | * 115 | * Args: 116 | * b: A bbox with x, y, w, h values. 117 | */ 118 | scale = choose_scale(img) 119 | bbox = {}; 120 | bbox.x = Math.floor(toCanvasCoords(b.x)); 121 | bbox.y = Math.floor(toCanvasCoords(b.y)); 122 | bbox.w = Math.floor(toCanvasCoords(b.w)); 123 | bbox.h = Math.floor(toCanvasCoords(b.h)); 124 | draw(); 125 | } 126 | 127 | that.enable = function() { 128 | disabled = false; 129 | draw(); 130 | } 131 | 132 | that.disable = function() { 133 | disabled = true; 134 | mouseup(); 135 | draw(); 136 | } 137 | 138 | that.setObject = function(x, y, name) { 139 | /** Set the object location and name. 140 | * 141 | * Args: 142 | * x: The x co-ordinate. 143 | * y: The y co-ordinate. 144 | * name: The text of the object. 145 | */ 146 | if (scale) { 147 | object_pos = {'x': toCanvasCoords(x), 'y': toCanvasCoords(y)}; 148 | } else { 149 | object_pos = {'x': x, 'y': y}; 150 | } 151 | object_name = name; 152 | } 153 | 154 | that.addStaticBox = function(box) { 155 | /** Set the object location and name. 156 | * 157 | * Args: 158 | * box: A dictionary containing x, y, w, h. 159 | * 160 | * Returns: 161 | * Index of static box so that the user can delete it if needed. 162 | */ 163 | static_box = {} 164 | static_box.x = box.x; 165 | static_box.y = box.y; 166 | static_box.w = box.w; 167 | static_box.h = box.h; 168 | if (box.color != null) { 169 | static_box.color = box.color; 170 | } 171 | //scale = choose_scale(img); 172 | if (scale) { 173 | static_box.x = toCanvasCoords(box.x); 174 | static_box.y = toCanvasCoords(box.y); 175 | static_box.w = toCanvasCoords(box.w); 176 | static_box.h = toCanvasCoords(box.h); 177 | } 178 | static_boxes.push(static_box); 179 | draw(); 180 | return static_boxes.length - 1; 181 | } 182 | 183 | that.removeStaticBox = function(index) { 184 | /** Deletes the static box at that index. 185 | * 186 | * Args: 187 | * index: Index of static box to delete. 188 | */ 189 | if (static_boxes.length <= index) return; 190 | static_boxes.splice(index, 1); 191 | } 192 | 193 | function choose_scale(img) { 194 | /** Helper function to scale the images based on canvas width. 195 | * 196 | * Args: 197 | * img: The image element. 198 | */ 199 | if (_.isNumber(options.max_width) && _.isNumber(options.max_height)) { 200 | var scale_h = img.width / options.max_width; 201 | var scale_v = img.height / options.max_height; 202 | return Math.max(scale_h, scale_v); 203 | } else { 204 | return img.width / canvas_width; 205 | } 206 | } 207 | 208 | that.restoring = function() { 209 | /** Sets the bbox to be drawn. 210 | */ 211 | bbox_drawn = true; 212 | } 213 | 214 | that.removeBox = function() { 215 | bbox = null; 216 | draw(); 217 | } 218 | 219 | that.reset = function() { 220 | /** Reset everything to no boxes and get the crosshairs. 221 | */ 222 | click_pos = null; 223 | mouse_pos = null; 224 | bbox_drawn = false; 225 | bbox = null; 226 | static_boxes = []; 227 | object_pos = null; 228 | object_name = null; 229 | resize_direction = null; 230 | old_bbox = null; 231 | $("#drawcanv").css({'cursor': 'crosshair'}); // reset cursor 232 | draw(); 233 | } 234 | 235 | function setup() { 236 | /* Setup the drawing and draw the object and bbox if already set. 237 | */ 238 | scale = choose_scale(img); 239 | canvas_width = img.width / scale; 240 | var canvas_height = img.height / scale; 241 | canvas = $('') 242 | .attr({'width': canvas_width, 243 | 'height': canvas_height}) 244 | .attr('id', 'drawcanv') 245 | .css({'cursor': 'crosshair'}) 246 | .appendTo(div); 247 | 248 | canvas_pos = {'x': canvas.offset().left, 249 | 'y': canvas.offset().top}; 250 | ctx = canvas[0].getContext('2d'); 251 | canvas.mousemove(mousemove); 252 | canvas.mousedown(mousedown); 253 | canvas.mouseup(mouseup); 254 | canvas.mouseout(mouseup); 255 | 256 | if (object_pos) { 257 | object_pos.x = toCanvasCoords(object_pos.x); 258 | object_pos.y = toCanvasCoords(object_pos.y); 259 | } 260 | 261 | // Convert the static box from image coordinates to canvas coordinates 262 | for (var k = 0; k < static_boxes.length; k++) { 263 | static_box = static_boxes[k]; 264 | var props = ['x', 'y', 'w', 'h']; 265 | for (var i = 0; i < props.length; i++) { 266 | static_box[props[i]] = toCanvasCoords(static_box[props[i]]); 267 | } 268 | } 269 | 270 | draw(); 271 | options.callback(); 272 | } 273 | 274 | function toCanvasCoords(x) { return x / scale; } 275 | function toImageCoords(x) { return x * scale; } 276 | function setCursor(cursor) { canvas.css('cursor', cursor); } 277 | 278 | function getPosition(e) { 279 | var x = e.pageX - canvas_pos.x; 280 | var y = e.pageY - canvas_pos.y; 281 | return {'x': x, 'y': y}; 282 | } 283 | 284 | function mousedown(e) { 285 | if (disabled) return; 286 | click_pos = getPosition(e); 287 | if (bbox) { 288 | resize_direction = detectCollision(click_pos.x, click_pos.y, bbox); 289 | if (resize_direction) { 290 | old_bbox = $.extend({}, bbox); 291 | } 292 | } 293 | } 294 | 295 | function mouseup(e) { 296 | click_pos = null; 297 | resize_direction = null; 298 | if (bbox) { 299 | bbox_drawn = true; 300 | setCursor('pointer'); 301 | } 302 | draw(); 303 | } 304 | 305 | function mousemove(e) { 306 | mouse_pos = getPosition(e); 307 | updateBBox(); 308 | 309 | if (bbox && !click_pos && !disabled) { 310 | var collision = detectCollision(mouse_pos.x, mouse_pos.y, bbox); 311 | if (cursors.hasOwnProperty(collision)) { 312 | setCursor(cursors[collision]); 313 | } else { 314 | setCursor('pointer'); 315 | } 316 | } 317 | 318 | draw(); 319 | } 320 | 321 | function detectCollision(x, y, box) { 322 | function dist(x1, y1, x2, y2) { 323 | var dx = x2 - x1; 324 | var dy = y2 - y1; 325 | return Math.sqrt(dx * dx + dy * dy); 326 | } 327 | if (dist(x, y, box.x, box.y) < options.click_radius) { 328 | return 'UL'; 329 | } else if (dist(x, y, box.x + box.w, box.y) < options.click_radius) { 330 | return 'UR'; 331 | } else if (dist(x, y, box.x, box.y + box.h) < options.click_radius) { 332 | return 'BL'; 333 | } else if (dist(x, y, box.x + box.w, box.y + box.h) 334 | < options.click_radius) { 335 | return 'BR'; 336 | } else if (Math.abs(box.x - x) < options.click_radius) { 337 | return 'L'; 338 | } else if (Math.abs(box.x + box.w - x) < options.click_radius) { 339 | return 'R'; 340 | } else if (Math.abs(box.y - y) < options.click_radius 341 | && x >= bbox.x && x <= bbox.x + bbox.w) { 342 | return 'U'; 343 | } else if (Math.abs(box.y + box.h - y) < options.click_radius 344 | && y >= bbox.y && y <= bbox.y + bbox.h) { 345 | return 'B'; 346 | } 347 | return null; 348 | } 349 | 350 | function updateBBox() { 351 | function makeBBox(x1, y1, x2, y2) { 352 | var x = Math.min(x1, x2); 353 | var y = Math.min(y1, y2); 354 | var w = Math.max(x1, x2) - x; 355 | var h = Math.max(y1, y2) - y; 356 | return {'x': x, 'y': y, 'w': w, 'h': h}; 357 | } 358 | if (!bbox_drawn && click_pos && mouse_pos) { 359 | bbox = makeBBox(click_pos.x, click_pos.y, mouse_pos.x, mouse_pos.y); 360 | } else if (bbox_drawn && click_pos && mouse_pos && resize_direction) { 361 | var x1 = old_bbox.x; 362 | var x2 = old_bbox.x + old_bbox.w; 363 | var y1 = old_bbox.y; 364 | var y2 = old_bbox.y + old_bbox.h; 365 | if (resize_direction === 'L' 366 | || resize_direction === 'UL' 367 | || resize_direction === 'BL') { 368 | x1 = mouse_pos.x; 369 | } 370 | if (resize_direction === 'R' 371 | || resize_direction === 'UR' 372 | || resize_direction === 'BR') { 373 | x2 = mouse_pos.x; 374 | } 375 | if (resize_direction === 'U' 376 | || resize_direction === 'UR' 377 | || resize_direction === 'UL') { 378 | y1 = mouse_pos.y; 379 | } 380 | if (resize_direction === 'B' 381 | || resize_direction === 'BR' 382 | || resize_direction === 'BL') { 383 | y2 = mouse_pos.y; 384 | } 385 | bbox = makeBBox(x1, y1, x2, y2); 386 | } 387 | } 388 | 389 | function drawBox(box, box_color) { 390 | ctx.save(); 391 | 392 | function r(x) { return Math.floor(x); } 393 | 394 | ctx.strokeStyle = box_color; 395 | ctx.lineWidth = options.bbox_line_width; 396 | ctx.strokeRect(r(box.x), r(box.y), r(box.w), r(box.h)); 397 | 398 | ctx.restore(); 399 | } 400 | 401 | function drawCrosshair(x, y) { 402 | y = Math.floor(y) + 0.5; 403 | x = Math.floor(x) + 0.5; 404 | 405 | ctx.save(); 406 | ctx.lineWidth = 3; 407 | ctx.strokeStyle = '#fff'; 408 | ctx.globalAlpha = 0.3; 409 | 410 | ctx.beginPath(); 411 | ctx.moveTo(x, 0); 412 | ctx.lineTo(x, canvas.height()); 413 | ctx.stroke(); 414 | 415 | ctx.beginPath(); 416 | ctx.moveTo(0, y); 417 | ctx.lineTo(canvas.width(), y); 418 | ctx.stroke(); 419 | 420 | ctx.lineWidth = 1; 421 | ctx.strokeStyle = '#000'; 422 | ctx.globalAlpha = 1.0; 423 | 424 | ctx.beginPath(); 425 | ctx.moveTo(x, 0); 426 | ctx.lineTo(x, canvas.height()); 427 | ctx.stroke(); 428 | 429 | ctx.beginPath(); 430 | ctx.moveTo(0, y); 431 | ctx.lineTo(canvas.width(), y); 432 | ctx.stroke(); 433 | 434 | ctx.restore(); 435 | } 436 | 437 | function drawHandle(x, y) { 438 | ctx.save(); 439 | ctx.globalAlpha = options.handle_opacity; 440 | 441 | ctx.beginPath(); 442 | ctx.arc(x, y, options.click_radius, 0, 2 * Math.PI); 443 | ctx.closePath(); 444 | ctx.fill(); 445 | 446 | ctx.restore(); 447 | } 448 | 449 | function drawObject(x, y, name, emphasized) { 450 | ctx.save(); 451 | ctx.fillStyle = options.bbox_color; 452 | 453 | var r = emphasized ? options.dot_big_radius : options.dot_small_radius; 454 | 455 | ctx.beginPath(); 456 | ctx.arc(x, y, r, 0, 2 * Math.PI); 457 | ctx.closePath(); 458 | ctx.fill(); 459 | ctx.stroke(); 460 | 461 | if (emphasized) { 462 | // Draw the text box 463 | var width = ctx.measureText(name).width; 464 | 465 | ctx.fillStyle = '#fff'; 466 | ctx.strokeStyle = '#000'; 467 | ctx.beginPath(); 468 | var x1 = x - width - TEXT_BOX_PADDING; 469 | var x2 = x + width + TEXT_BOX_PADDING; 470 | var y1 = y + options.dot_big_radius + 1.5 * TEXT_BOX_PADDING; 471 | var y2 = y1 + 1.5 * TEXT_BOX_PADDING + TEXT_FONT_SIZE; 472 | ctx.moveTo(x1, y1); 473 | ctx.lineTo(x1, y2); 474 | ctx.lineTo(x2, y2); 475 | ctx.lineTo(x2, y1); 476 | ctx.closePath(); 477 | ctx.fill(); 478 | ctx.stroke(); 479 | 480 | ctx.font = TEXT_FONT; 481 | ctx.fillStyle = '#000'; 482 | var tx = x - width; 483 | var ty = y + TEXT_FONT_SIZE + 2.5 * TEXT_BOX_PADDING + options.dot_big_radius; 484 | ctx.fillText(name, tx, ty); 485 | } 486 | 487 | ctx.restore(); 488 | } 489 | 490 | function draw() { 491 | if (!ctx) return; 492 | 493 | ctx.clearRect(0, 0, canvas.width(), canvas.height()); 494 | ctx.save(); 495 | ctx.globalAlpha = options.image_opacity; 496 | ctx.drawImage(img, 0, 0, canvas.width(), canvas.height()); 497 | ctx.restore(); 498 | 499 | for (var k = 0; k < static_boxes.length; k++) { 500 | static_box = static_boxes[k]; 501 | box_color = options.static_box_color; 502 | if (static_box.color != null) { 503 | box_color = static_box.color; 504 | } 505 | drawBox(static_box, box_color); 506 | } 507 | 508 | if (object_name && object_pos) { 509 | var emphasized = false; 510 | if (mouse_pos) { 511 | var dx = mouse_pos.x - object_pos.x; 512 | var dy = mouse_pos.y - object_pos.y; 513 | if (Math.sqrt(dx * dx + dy * dy) < options.click_radius) { 514 | emphasized = true; 515 | } 516 | } 517 | 518 | drawObject(object_pos.x, object_pos.y, object_name, emphasized); 519 | } 520 | 521 | if (disabled) return; 522 | 523 | if (bbox) { 524 | if (bbox_drawn) { 525 | drawHandle(bbox.x, bbox.y); 526 | drawHandle(bbox.x + bbox.w, bbox.y); 527 | drawHandle(bbox.x, bbox.y + bbox.h); 528 | drawHandle(bbox.x + bbox.w, bbox.y + bbox.h); 529 | } else { 530 | drawCrosshair(bbox.x, bbox.y); 531 | drawCrosshair(bbox.x + bbox.w, bbox.y + bbox.h); 532 | } 533 | drawBox(bbox, options.bbox_color); 534 | } else if (mouse_pos) { 535 | drawCrosshair(mouse_pos.x, mouse_pos.y); 536 | } 537 | } 538 | 539 | that.remove = function() { 540 | /** Delete the canvas. 541 | */ 542 | canvas.remove(); 543 | } 544 | 545 | return that; 546 | } 547 | 548 | return etjs; 549 | 550 | }(ETJS || {})); 551 | -------------------------------------------------------------------------------- /easyturk/templates/javascript/bleu.js: -------------------------------------------------------------------------------- 1 | var ETJS = (function(etjs) { 2 | 3 | etjs.bleu_score = function(sentence, reference, N) { 4 | /**Calculates the Bleu score between the sentence and a the reference. 5 | * 6 | * Args: 7 | * sentence: A string of words. 8 | * reference: A string of words. 9 | * 10 | * Returns: 11 | * The bleu-N score between the sentence and reference [0, 1]. 12 | **/ 13 | var matched = 0; 14 | var total = 0; 15 | for (var n = 1; n <= N; n++) { 16 | for (var s_index = 0; s_index + n <= sentence.length; s_index = s_index + n) { 17 | for (var r_index = 0; r_index + n <= reference.length; r_index = r_index + n) { 18 | all_matched = true; 19 | for (var offset = 0; offset < n; offset++) { 20 | //console.log(s_index+"::"+r_index+"::"+s_elem +"::"+r_elem+"::"+sentence[s_elem] + "::" + reference[r_elem]) 21 | if (sentence[s_index + offset] != reference[r_index + offset]) { 22 | all_matched = false; 23 | break; 24 | } 25 | } 26 | if (all_matched) { 27 | matched += 1; 28 | break; 29 | } 30 | } 31 | total += 1 32 | } 33 | } 34 | return (matched)/total; 35 | } 36 | 37 | 38 | etjs.max_bleu_score = function(sentence, references, N) { 39 | /** Calculates the maximum Bleu score between a sentence and a set of references. 40 | * 41 | * Args: 42 | * sentence: A string of words. 43 | * references: A list of strings of words. 44 | * 45 | * Returns: 46 | * The bleu-N score between the sentence and references [0, 1]. 47 | **/ 48 | if (sentence == undefined || sentence.length == 0) { 49 | return 0; 50 | } 51 | var score = 0; 52 | var s = sentence.toLowerCase().split(" "); 53 | for (var index in references) { 54 | var ref = references[index]; 55 | if (ref == undefined || ref.length == 0) { 56 | continue; 57 | } 58 | ref = ref.toLowerCase().split(" "); 59 | score = Math.max(score, vg.bleu_score(s, ref, N)); 60 | } 61 | return score 62 | }; 63 | 64 | 65 | etjs.is_unique_sentence = function(sentence, references, threshold, N) { 66 | /** Decides if the sentence is new, which is decided based on if: 67 | * Bleu-N(sentence, references) > threshold. 68 | * 69 | * Args: 70 | * sentence: A string of words. 71 | * references: A list of strings of words. 72 | * threshold: A float between [0, 1]. 73 | * N: Positive Integer. 74 | * 75 | * Returns: 76 | * A boolean indicates if the sentence is unlike the references. 77 | **/ 78 | var score = vg.max_bleu_score(sentence, references, N); 79 | return score < threshold; 80 | }; 81 | 82 | 83 | return etjs; 84 | }(ETJS || {})); 85 | -------------------------------------------------------------------------------- /easyturk/templates/javascript/iou.js: -------------------------------------------------------------------------------- 1 | var ETJS = (function(etjs) { 2 | 3 | etjs.intersection_over_union = function(bbox, other) { 4 | /** Calculates IoU between two bboxes. 5 | * 6 | * Args: 7 | * bbox: A bbox dictionary with x, y, w, h. 8 | * other: Same as bbox. 9 | * 10 | * Returns: 11 | * IoU between bbox and other. 12 | **/ 13 | if (!bbox || !other) return 0; 14 | var x1 = Math.max(bbox['x'], other['x']); 15 | var x2 = Math.min(bbox['x'] + bbox['w'], 16 | other['x'] + other['w']); 17 | var y1 = Math.max(bbox['y'], other['y']); 18 | var y2 = Math.min(bbox['y'] + bbox['h'], 19 | other['y'] + other['h']); 20 | if (x1 > x2 || y1 > y2) return 0; 21 | var a1 = bbox['w'] * bbox['h']; 22 | var a2 = other['w'] * other['h']; 23 | var o = (x2 - x1) * (y2 - y1); 24 | var iou = o / (a1 + a2 - o + 1e-9); 25 | return iou; 26 | }; 27 | 28 | return etjs; 29 | }(ETJS || {})); 30 | -------------------------------------------------------------------------------- /easyturk/templates/verify_bbox.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | Verify whether objects in pictures are accurate 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 54 | 55 | 56 |
57 |
58 |
59 |
60 |
61 |

Instructions

62 |
63 |
64 |
65 |
66 |
    67 |
  • Verify whether objects in pictures are accurate.
    • 68 |
  • In this task, you will be shown a list of pictures and objects in them. Your task is to tell us: 69 |
      70 |
    • Whether the name of the object is correct. (For example, if someone marked a "person" as a "car", tell us that the name is wrong.)
      • 71 |
    • Whether the box covers multiple objects or a single object. (For example, if the object is a "person", tell us if the box covers multiple people or just one person. If the box mainly covers one person but part of another person is in the box, then just say that it covers one person.)
      • 72 |
    • Whether the box is too loose or excludes some parts of the object. (For example, if the box doesn't cover the "person" completely and their hand is outside the box.
      • 73 |
    74 |
    • 75 |
  • 76 | Each box should be tight around its object, and should completely cover the object. 77 |
    78 |
    79 | Good: 80 |
    81 | 82 |
    83 |
    84 | Bad: 85 |
    86 | 87 |
    88 |
    89 | Bad: 90 |
    91 | 92 |
    93 |
    94 |
    • 95 |
  • 96 | Each box should only cover the visible part of all objects mentioned in the text snippet. 97 |
    98 |
    99 | Good: 100 |
    101 | 102 |
    103 |
    104 | Bad: 105 |
    106 | 107 |
    108 |
    109 |
    • 110 |
  • 111 | Objects that take up a large portion of the image should be completey covered 112 | by their boxes, such as the boxes for the road below: 113 |
    114 |
    115 | Good: 116 |
    117 | 118 |
    119 |
    120 | Bad: 121 |
    122 | 123 |
    124 |
    125 |
    • 126 |
127 |
128 |
129 |
130 |
131 |
132 | 133 |
134 |
135 |

Here are the objects and pictures we would like you to verify:

136 |
137 | 138 |
139 |
140 |
141 |
142 |
143 |
144 |
145 |
146 |
147 |
148 |
149 |

Answer the three questions below.

150 |

The name of the object is

151 |
152 |
153 | 154 |
155 |
156 |
157 |
158 | 159 |
160 |
161 |
162 |
163 | 164 |
165 |
166 |
167 |
168 | 169 |
170 |
171 |
172 |
173 | 174 |
175 |
176 |
177 |
178 | 179 |
180 |
181 |
182 |
183 |
184 |
185 | 186 |
187 |
188 | 189 | / 190 | 191 |
192 |
193 | 194 |
195 |
196 |
197 |
198 | 199 | 200 | {% include "easyturk.html" %} 201 | 204 | 205 | 390 | 391 | 392 | -------------------------------------------------------------------------------- /easyturk/templates/verify_caption.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | Verify whether captions for pictures are accurate 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 50 | 51 | 52 |
53 |
54 |
55 |
56 |
57 |

Instructions

58 |
59 |
60 |
61 |
62 |
    63 |
  • Verify whether captions for pictures are accurate.
    • 64 |
  • In this task, you will be shown a picture and a caption your task is to say whether the caption is a correct way to describe the image.
    • 65 |
66 |
67 |
68 |
69 |
70 |
71 | 72 |
73 |
74 |

Here are the captions and pictures we would like you to verify:

75 |
76 | 77 |
78 |
79 |
80 |
81 |
82 |
83 |
84 |
85 |
86 |
87 |
88 |

Verify a caption describing the contents of the image:

89 |

90 |
91 |
92 | 93 |
94 |
95 |
96 |
97 | 98 |
99 |
100 |
101 |
102 |
103 |
104 | 105 |
106 |
107 | 108 | / 109 | 110 |
111 |
112 | 113 |
114 |
115 |
116 |
117 | 118 | 119 | {% include "easyturk.html" %} 120 | 121 | 275 | 276 | 277 | -------------------------------------------------------------------------------- /easyturk/templates/verify_question_answer.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | Verify whether the answers to questions about a pictures are accurate 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 50 | 51 | 52 |
53 |
54 |
55 |
56 |
57 |

Instructions

58 |
59 |
60 |
61 |
62 |
    63 |
  • Verify whether answers to questions about pictures are accurate.
    • 64 |
  • In this task, you will be shown a picture and a questions and answer. Your task is to say whether the answer is correct.
    • 65 |
66 |
67 |
68 |
69 |
70 |
71 | 72 |
73 |
74 |

Here are the answers with questions and pictures we would like you to verify:

75 |
76 | 77 |
78 |
79 |
80 |
81 |
82 |
83 |
84 |
85 |
86 |
87 |
88 |

Verify the answer for the question about the contents of the image:

89 |

90 |

91 |
92 |
93 | 94 |
95 |
96 |
97 |
98 | 99 |
100 |
101 |
102 |
103 |
104 |
105 | 106 |
107 |
108 | 109 | / 110 | 111 |
112 |
113 | 114 |
115 |
116 |
117 |
118 | 119 | 120 | {% include "easyturk.html" %} 121 | 122 | 278 | 279 | 280 | -------------------------------------------------------------------------------- /easyturk/templates/verify_relationship.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | Verify whether relationships between objects in pictures are accurate 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 54 | 55 | 56 |
57 |
58 |
59 |
60 |
61 |

Instructions

62 |
63 |
64 |
65 |
66 |
    67 |
  • Verify whether relationships between objects in pictures are accurate.
    • 68 |
  • In this task, you will be shown a list of pictures and and asked to verify if a relationship is accurate.
    • 69 |
  • First, let's explain what a RELATIONSHIP is. A relationship consists of 3 parts: <object_1, predicate, object_2>. 70 | The predicate explains how object_1 is related to object_2. For example, <dog, holding, stick> is a relationship where the 71 | two objects are "dog" and "stick" and they are related to each other by the predicate "holding". The way to explain a 72 | relationship in english is by writing it as "a dog is holding a stick". Similarly, <man, jumping over, fire hydrant> 73 | can be read as "A man is jumping over a fire hydrant". It connects the two objects, "man" and "fire hydrant" with the predicate 74 | "jumping over".
    • 75 |
  • You will be shown a series of relationships in each picture and asked to verify the three questions below: 76 |
      77 |
    • Whether the name of object_1 is correct. (For example, if someone marked a "dog" as a "car", tell us that the name is wrong.)
      • 78 |
    • Whether the name of object_2 is correct. (For example, if someone marked a "stick" as a "pole", tell us that the name is wrong.)
      • 79 |
    • Whether the name of predicate is correct. (For example, if someone marked a "holding" as a "throwing", tell us that the name is wrong.)
      • 80 |
    81 |
    • 82 |
  • For this task, we don't care if the boxes are not tightly drawn around the objects. So don't worry about how well the boxes are drawn.
    • 83 |
84 |
85 |
86 |
87 |
88 |
89 | 90 |
91 |
92 |

Here are the relationships and pictures we would like you to verify:

93 |
94 | 95 |
96 |
97 |
98 |
99 |
100 |
101 |
102 |
103 |
104 |
105 |
106 |

Answer the three questions below.

107 |

The relationship is

108 |
109 |
110 | 111 |
112 |
113 |
114 |
115 | 116 |
117 |
118 |
119 |
120 | 121 |
122 |
123 |
124 |
125 | 126 |
127 |
128 |
129 |
130 | 131 |
132 |
133 |
134 |
135 | 136 |
137 |
138 |
139 |
140 |
141 |
142 | 143 |
144 |
145 | 146 | / 147 | 148 |
149 |
150 | 151 |
152 |
153 |
154 |
155 | 156 | 157 | {% include "easyturk.html" %} 158 | 161 | 162 | 373 | 374 | 375 | -------------------------------------------------------------------------------- /easyturk/templates/write_caption.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | Write captions for pictures 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 50 | 51 | 52 |
53 |
54 |
55 |
56 |
57 |

Instructions

58 |
59 |
60 |
61 |
62 |
    63 |
  • Write captions for pictures.
    • 64 |
  • In this task, you will be shown a picture and your task is to write a grammatically correct sentence describing the contents of the picture
    • 65 |
66 |
67 |
68 |
69 |
70 |
71 | 72 |
73 |
74 |

Here are the pictures we would like you to write captions for:

75 |
76 | 77 |
78 |
79 |
80 |
81 |
82 |
83 |
84 |
85 |
86 |
87 |
88 |

Write a caption describing the contents of the image:

89 | 90 |
91 |
92 |
93 |
94 | 95 |
96 |
97 | 98 | / 99 | 100 |
101 |
102 | 103 |
104 |
105 |
106 |
107 | 108 | 109 | {% include "easyturk.html" %} 110 | 111 | 244 | 245 | 246 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | awscli==1.16.198 2 | boto3==1.9.188 3 | botocore==1.12.188 4 | colorama==0.3.9 5 | docutils==0.14 6 | futures==3.3.0 7 | Jinja2==2.10.1 8 | jmespath==0.9.4 9 | MarkupSafe==1.1.1 10 | pyasn1==0.4.5 11 | python-dateutil==2.8.0 12 | PyYAML==5.1 13 | rsa==3.4.2 14 | s3transfer==0.2.1 15 | six==1.12.0 16 | urllib3==1.25.3 17 | --------------------------------------------------------------------------------