├── external └── README.md ├── .gitignore ├── evaluation ├── format_generation_eval.md ├── format_comprehension_eval.md ├── AMT_interface │ └── AMT_template_generated_vs_GT.html └── sample_results │ └── sample_results_generation.json ├── google_refexp_py_lib ├── common_utils.py ├── refexp.py └── refexp_eval.py ├── README.md └── setup.py /external/README.md: -------------------------------------------------------------------------------- 1 | This directory is used to store external packages. 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | # C extensions 6 | *.so 7 | # Distribution / packaging 8 | .Python 9 | env/ 10 | build/ 11 | develop-eggs/ 12 | dist/ 13 | downloads/ 14 | eggs/ 15 | .eggs/ 16 | lib/ 17 | lib64/ 18 | parts/ 19 | sdist/ 20 | var/ 21 | *.egg-info/ 22 | .installed.cfg 23 | *.egg 24 | # PyInstaller 25 | # Usually these files are written by a python script from a template 26 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 27 | *.manifest 28 | *.spec 29 | # Installer logs 30 | pip-log.txt 31 | pip-delete-this-directory.txt 32 | # Unit test / coverage reports 33 | htmlcov/ 34 | .tox/ 35 | .coverage 36 | .coverage.* 37 | .cache 38 | nosetests.xml 39 | coverage.xml 40 | *,cover 41 | .hypothesis/ 42 | # Translations 43 | *.mo 44 | *.pot 45 | # Django stuff: 46 | *.log 47 | # Sphinx documentation 48 | docs/_build/ 49 | # PyBuilder 50 | target/ 51 | 52 | # Junhua Add 53 | *~ 54 | external/coco 55 | external/PythonAPI 56 | cache 57 | EXP_Junhua_code 58 | google_refexp_py_lib/coco 59 | google_refexp_py_lib/pycocotools 60 | google_refexp_dataset_release 61 | .ipynb_checkpoints 62 | cache_evaluation 63 | -------------------------------------------------------------------------------- /evaluation/format_generation_eval.md: -------------------------------------------------------------------------------- 1 | Introduce the format to use the evaluation code in google_refexp_py_lib.eval 2 | that compares the generated sentences with the groundtruth annotation. 3 | 4 | ## Format for AMT evaluation of the generation task 5 | 6 | The algorithm should output a json file with a list. Each element in the list 7 | should be a dictionary with the following format: 8 | 9 | { 10 | "annotation_id": int, (required, MSCOCO object annotation id) 11 | "generated_refexp": string (required, generated referring expressions) 12 | } 13 | 14 | It is allowed that there are duplicated "annotation_id" in the list. The script 15 | will treat the corresponding referring expressions independently. 16 | 17 | If you want to use the *end-to-end evaluation* (evaluate the performance of the 18 | generation-comprehension pipeline) as described in [1], please use your generated 19 | sentences as the input to your comprehension model and evaluate the results under 20 | the comprehension settings. 21 | 22 | [1]. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and 23 | Kevin Murphy. "Generation and Comprehension of Unambiguous Object Descriptions." 24 | arXiv preprint arXiv:1511.02283 (2015). 25 | -------------------------------------------------------------------------------- /evaluation/format_comprehension_eval.md: -------------------------------------------------------------------------------- 1 | Introduce the format to use the evaluation code in google_refexp_py_lib.eval 2 | to output the prec@k score of the bounding boxes of your method. 3 | 4 | ## format for the prec@1 evaluation of the comprehension task 5 | 6 | The algorithm should output a json file with a list. Each element in the list 7 | should be a dictionary with the following format: 8 | 9 | { 10 | "annotation_id" : int, (required, MSCOCO object annotation id) 11 | "predicted_bounding_boxes": obj, (required, list of predicted bounding boxes) 12 | "refexp_id" : int, (optional) 13 | "refexp": string, (optional) 14 | } 15 | 16 | - "annotation_id" denotes the original MS COCO object annotation id for an 17 | object in Google Refexp dataset. 18 | - "predicted_bounding_boxes" is list of bounding boxes. Each bounding box is 19 | represented as a 4-int list: \[x, y, w, h\]. (x, y) is the coordinates of the 20 | top left corner of the bounding box. w and h are the width and height of the 21 | bounding box respectively. The order of the bounding box in this list 22 | should reflect the method's confident level. E.g. The evaluation code will treat 23 | the first bounding box as the most probrable bounding box output by the algorithm. 24 | - "refexp_id" is the id for the groundtruth referring expression in the Google 25 | Refexp Dataset. It will be used for the visualization of the results. If the 26 | predicted bounding box does not cooresponding to a groundtruth referring 27 | expression, please do NOT add this field. 28 | - "refexp" is the string for the referring expression that input to the comprehension 29 | model. If you want to visualize the results (e.g. see for which referring expression 30 | your model performs badly), you should have either "refexp_id" (if the refexp is a 31 | groundtruth one) or "refexp" (if the refexp is a customized or generated one). 32 | 33 | -------------------------------------------------------------------------------- /google_refexp_py_lib/common_utils.py: -------------------------------------------------------------------------------- 1 | """Common util functions. 2 | 3 | All the bounding boxes in this script should be represented by a 4-int list: 4 | \[x, y, w, h\]. 5 | (x, y) is the coordinates of the top left corner of the bounding box. 6 | w and h are the width and height of the bounding box respectively. 7 | """ 8 | 9 | import numpy 10 | 11 | def iou_bboxes(bbox1, bbox2): 12 | """Standard intersection over Union ratio between two bounding boxes.""" 13 | bbox_ov_x = max(bbox1[0], bbox2[0]) 14 | bbox_ov_y = max(bbox1[1], bbox2[1]) 15 | bbox_ov_w = min(bbox1[0] + bbox1[2] - 1, bbox2[0] + bbox2[2] - 1) - bbox_ov_x + 1 16 | bbox_ov_h = min(bbox1[1] + bbox1[3] - 1, bbox2[1] + bbox2[3] - 1) - bbox_ov_y + 1 17 | 18 | area1 = area_bbox(bbox1) 19 | area2 = area_bbox(bbox2) 20 | area_o = area_bbox([bbox_ov_x, bbox_ov_y, bbox_ov_w, bbox_ov_h]) 21 | area_u = area1 + area2 - area_o 22 | if area_u < 0.000001: 23 | return 0.0 24 | else: 25 | return area_o / area_u 26 | 27 | def area_bbox(bbox): 28 | """Return the area of a bounding box.""" 29 | if bbox[2] <= 0 or bbox[3] <= 0: 30 | return 0.0 31 | return float(bbox[2]) * float(bbox[3]) 32 | 33 | def apply_mask_to_image(img_o, mask, factor_mask = 0.5, channel_mask = 0): 34 | """Apply a transparent mask to an image.""" 35 | if len(img_o.shape) == 2: # grey-scale image 36 | img_o_c = numpy.zeros(tuple(list(img_o.shape) + [3]), numpy.uint8) 37 | for i in xrange(3): 38 | img_o_c[:, :, i] = img_o 39 | img_o = img_o_c 40 | img_mask = numpy.zeros(img_o.shape, numpy.uint8) 41 | img_mask[:, :, channel_mask] = mask 42 | img_t = img_o * (1.0 - factor_mask) + img_mask * factor_mask 43 | img_t = img_t.astype(img_o.dtype) 44 | return img_t 45 | 46 | def draw_bbox(ax, bbox, edge_color='red', line_width=3): 47 | """Draw one bounding box on a matplotlib axis object (ax).""" 48 | import matplotlib.patches as mpatches 49 | import matplotlib.pyplot as plt 50 | 51 | bbox_plot = mpatches.Rectangle((bbox[0], bbox[1]), bbox[2], bbox[3], 52 | fill=False, edgecolor=edge_color, linewidth=line_width) 53 | ax.add_patch(bbox_plot) 54 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Python toolbox for the Google Referring Expressions Dataset 2 | 3 | 4 | The Google RefExp dataset is a collection of text descriptions of objects in 5 | images which builds on the publicly available [MS-COCO](http://mscoco.org/) 6 | dataset. Whereas the image captions in MS-COCO apply to the entire image, this 7 | dataset focuses on 8 | text descriptions that allow one to uniquely identify a single object or region 9 | within an image. 10 | See more details in this paper: [Generation and Comprehension of Unambiguous Object Descriptions](http://arxiv.org/abs/1511.02283) 11 | 12 | ## Sample of the data 13 | 14 | Green dot is the object that is being referred to. 15 | Sentences are generated by humans in a way that uniquely describes the 16 | chosen object. 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 32 | 38 | 44 | 45 |
Mountain ViewMountain ViewMountain View
27 |
    28 |
  • A girl wearing glasses and a pink shirt.
    • 29 |
  • An Asian girl with a pink shirt eating at the table.
    • 30 |
31 | 		33 |
    34 |
  • A boy brushing his hair while looking at his reflection.
    • 35 |
  • A young male child in pajamas shaking around a hairbrush in the mirror.
    • 36 |
37 | 		39 |
    40 |
  • Zebra looking towards the camera.
    • 41 |
  • A zebra third from the left.
    • 42 |
43 |
46 | 47 | 48 | ## Requirements 49 | - python 2.7 (Need numpy, scipy, matlabplot, PIL packages. All included in 50 | [Anaconda](https://store.continuum.io/cshop/anaconda/)) 51 | 52 | ## Setup and data downloading 53 | 54 | 55 | ### Easy setup 56 | 57 | ``` 58 | cd $YOUR_PATH_TO_THIS_TOOLBOX 59 | python setup.py 60 | ``` 61 | 62 | Running the setup.py script will do the following five things: 63 | 64 | 1. Download Google Refexp Data 65 | 2. Download and compile the COCO toolbox 66 | 3. Download COCO annotations 67 | 4. Download COCO images 68 | 5. Align the Google Refexp Data with COCO annotations 69 | 70 | At each step you will be prompted by keyboard whether or not you would like to 71 | skip a step. 72 | Note that the MS COCO images (13GB) and annotations (158MB) are very large and 73 | it takes some time to download all of them. 74 | 75 | ### Manual downloading and setup 76 | 77 | You can 78 | download the GoogleRefexp data directly from this 79 | [link](https://storage.googleapis.com/refexp/google_refexp_dataset_release.zip). 80 | 81 | 82 | If you have already played with MS COCO and do not want to have two copies of 83 | them, you can choose to create a symbolic link from external to your COCO toolkit. E.g.: 84 | 85 | ``` 86 | cd $YOUR_PATH_TO_THIS_TOOLBOX 87 | ln -sf $YOUR_PATH_TO_COCO_TOOLBOX ./external/coco 88 | ``` 89 | 90 | Please make sure the following on are on your path: 91 | 92 | 1. compiled PythonAPI at 93 | external/coco/PythonAPI 94 | 2. the annotation file at 95 | external/coco/annotations/instances_train2014.json 96 | 3. COCO images at external/coco/images/train2014/. 97 | 98 | You can create symbolic links if you have 99 | already downloaded the data and compiled the COCO toolbox. 100 | 101 | Then run *setup.py* to download the Google Refexp data and compile this toolbox. 102 | You can skip steps 2, 3, 4. 103 | 104 | ## Demos 105 | 106 | For **visualization** and **utility** functions, please see 107 | *google_refexp_dataset_demo.ipynb*. 108 | 109 | For automatic and Amazon Mechanical Turk (AMT) **evaluation** of the comprehension 110 | and generation tasks, please see *google_refexp_eval_demo.ipynb*; The 111 | appropriate output format for a comprehension/generation algorithm is described 112 | in ./evaluation/format_comprehension_eval.md and 113 | ./evaluation/format_generation_eval.md 114 | 115 | We also provide two sample outputs for reference. For the comprehension task, 116 | we use a naive baseline which is a random shuffle of the region candidates 117 | (./evaluation/sample_results/sample_results_comprehension.json). For the 118 | generation task, we use a naive baseline which outputs the class name of an 119 | object (./evaluation/sample_results/sample_results_generation.json). 120 | 121 | If you are not familiar with AMT evaluations, please see this 122 | [tutorial](http://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/amt-ui.pdf) 123 | The interface and APIs provided by this toolbox have already grouped 5 124 | evaluations into one HIT. In our experiment, paying 2 cents for one HIT leads to 125 | reasonable results. 126 | 127 | 128 | ## Citation 129 | 130 | If you find the dataset and toolbox useful in your research, 131 | please consider citing: 132 | 133 | @inproceedings{mao2016generation, 134 | title={Generation and Comprehension of Unambiguous Object Descriptions}, 135 | author={Mao, Junhua and Huang, Jonathan and Toshev, Alexander and Camburu, Oana and Yuille, Alan and Murphy, Kevin}, 136 | booktitle={CVPR}, 137 | year={2016} 138 | } 139 | 140 | ## License 141 | 142 | This data is released by Google under the following license: 143 | Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License. 144 | 145 | ## Toolbox Developers 146 | 147 | [Junhua Mao](https://www.stat.ucla.edu/~junhua.mao/) and [Oana Camburu](https://www.cs.ox.ac.uk/people/oana-maria.camburu/) 148 | -------------------------------------------------------------------------------- /evaluation/AMT_interface/AMT_template_generated_vs_GT.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 12 | 13 | 14 |
15 |
16 |
17 |
Instructions
18 | 19 |
20 |

You are given 5 pairs of images (original image and the highlighted object) and two sentences for each pair. Select the more discriminative sentence that would allow someone to more easily identify the highlighted object in the image. If they are all good, choose "Similar".

21 | 22 |
    23 |
  • A good discriminative sentence should provide enough information for you to identify the highlighted object.
    • 24 |
  • Whether a sentence is good or not depends only on the accuracy of the sentence and whether you can identify the highlighted object, NOT its length.
    • 25 |
  • We will check your annotations afterwards and reject the ones that violate any of the instructions.
    • 26 |
27 | 28 | 29 | 30 | 31 | 32 | 47 | 48 | 49 |
example 33 |

Examples1:

34 | 35 |

(A). "A man wearing a blue sweater whose hand is touching his head."
36 | (B). "A man in blue."
37 | (C). Similar
38 | In this case, (A) is preferred because it is more precise than (B), as there are two men wearing blue.

39 | 40 |

Examples2:

41 | 42 |

(A). "A man wearing a blue sweater whose hand is touching his head."
43 | (B). "A man in a blue sweater."
44 | (C). Similar
45 | In this case, (C) is preferred since both (A) and (B) are sufficiently discriminative to identify the highlighted object.

46 |
50 |
51 |
52 | 53 |
54 |
Choose the more discriminative sentence for the five image pairs below.
55 | 56 |
57 |
58 |
image_url
59 | 60 |
image_url
61 |
62 | 63 |
64 |
65 | 66 |
67 | 68 |
69 |
70 | 71 |
72 |
image_url
73 | 74 |
image_url
75 |
76 | 77 |
78 |
79 | 80 |
81 | 82 |
83 |
84 | 85 |
86 |
image_url
87 | 88 |
image_url
89 |
90 | 91 |
92 |
93 | 94 |
95 | 96 |
97 |
98 | 99 |
100 |
image_url
101 | 102 |
image_url
103 |
104 | 105 |
106 |
107 | 108 |
109 | 110 |
111 |
112 | 113 |
114 |
image_url
115 | 116 |
image_url
117 |
118 | 119 |
120 |
121 | 122 |
123 | 124 |
125 |
126 |
127 | 141 |
142 |
143 |
144 | 145 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | """Setup files for Google Refexp Toolbox 2 | 3 | This script has the following functions: 4 | 1. Download Google Refexp Data 5 | 2. Download and compile coco toolbox 6 | 3. Download coco annotations 7 | 4. Download coco images 8 | 5. Align Google Refexp Data with coco annotations 9 | """ 10 | 11 | import os 12 | import sys 13 | import json 14 | import numpy as np 15 | import logging 16 | import copy 17 | 18 | logger = logging.getLogger('DatasetRefexpGoogle') 19 | FORMAT = "[%(filename)s:line %(lineno)4s - %(funcName)20s() ] %(message)s" 20 | logging.basicConfig(format=FORMAT) 21 | logger.setLevel(logging.INFO) 22 | 23 | ## Step1. Download Google Refexp Data 24 | while True: 25 | ch = raw_input('Do you want to download Google Refexp Datset? (Y/N)') 26 | if ch == 'Y': 27 | os.system('mkdir google_refexp_dataset_release') 28 | os.system('cd google_refexp_dataset_release && ' 29 | 'wget https://storage.googleapis.com/refexp/google_refexp_dataset_release.zip') 30 | os.system('cd google_refexp_dataset_release && ' 31 | 'unzip google_refexp_dataset_release.zip') 32 | os.system('cd google_refexp_dataset_release && ' 33 | 'rm -f google_refexp_dataset_release.zip') 34 | print 'Google Refexp Dataset is now available at ./google_refexp_dataset_release' 35 | if ch == 'Y' or ch == 'N': 36 | break 37 | else: 38 | print 'Please type Y or N' 39 | 40 | ## Step2. Download coco toolbox 41 | while True: 42 | ch = raw_input('Do you want to download and install MS COCO toolbox? (Y/N)') 43 | if ch == 'Y': 44 | os.system('cd ./external && git clone https://github.com/pdollar/coco.git') 45 | os.system('cd ./external/coco/PythonAPI && python setup.py build_ext --inplace') 46 | print 'COCO toolbox is installed at ./external/coco/' 47 | if ch == 'Y' or ch == 'N': 48 | break 49 | else: 50 | print 'Please type Y or N' 51 | 52 | ## Step3. Download coco annotations 53 | while True: 54 | ch = raw_input('Do you want to download MS COCO train2014 annotations? (Y/N)') 55 | if ch == 'Y': 56 | os.system('cd ./external/coco && wget http://msvocds.blob.core.windows.net/' 57 | 'annotations-1-0-3/instances_train-val2014.zip && ' 58 | 'unzip instances_train-val2014.zip') 59 | os.system('rm -f ./external/coco/instances_train-val2014.zip') 60 | print ('COCO train2014 annotations are downloaded at ' 61 | './external/annotations/instances_train2014.json') 62 | if ch == 'Y' or ch == 'N': 63 | break 64 | else: 65 | print 'Please type Y or N' 66 | 67 | ## Step4. Download coco images 68 | while True: 69 | ch = raw_input('Do you want to download MS COCO train2014 images? (Y/N)') 70 | if ch == 'Y': 71 | # Download images 72 | os.system('mkdir ./external/coco/images') 73 | os.system('cd ./external/coco/images && wget http://msvocds.blob.core.windows.net/coco2014/train2014.zip && unzip train2014.zip && rm -f train2014.zip') 74 | print 'COCO train2014 images are downloaded at ./external/coco/images/train2014/' 75 | if ch == 'Y' or ch == 'N': 76 | break 77 | else: 78 | print 'Please type Y or N' 79 | 80 | ## Step5. Align our dataset with MS COCO 81 | # Check whether MS COCO toolbox are installed 82 | assert os.path.isdir('./external/coco/PythonAPI'), ('Please download MS COCO' 83 | 'first (run this script again and choose "Y" for MS COCO downloading)') 84 | assert os.path.isdir('./external/coco/PythonAPI/build'), ('Please install' 85 | 'MS COCO python API first (go to ./external/coco/PythonAPI/ and run setup.py)') 86 | assert os.path.isfile('./external/coco/annotations/instances_train2014.json'), ('Please' 87 | 'download MS COCO train2014 annotation first') 88 | 89 | # Create a symbolic link from pycocotools to google_refexp_lib 90 | pycoco_dir = os.path.join(os.getcwd(), 'external', 'coco', 'PythonAPI', 'pycocotools') 91 | os.system('cd google_refexp_py_lib && ln -sf %s pycocotools' % pycoco_dir) 92 | 93 | # Load MS COCO data 94 | logger.info('Start to load MS COCO file') 95 | sys.path.append('./external/coco/PythonAPI') 96 | from pycocotools.coco import COCO 97 | coco = COCO('./external/coco/annotations/instances_train2014.json') 98 | 99 | # Align our dataset with MS COCO 100 | logger.info('Start to align Google Refexp Data with ms coco annotations') 101 | dataset_template = './google_refexp_dataset_release/google_refexp_%s_201511.json' 102 | split_set_names = ('train', 'val') 103 | 104 | for set_name in split_set_names: 105 | logger.info('Start to align Google Refexp %s set with ms coco' % set_name) 106 | dataset_path_o = dataset_template % set_name 107 | dataset_path_n = dataset_path_o[:-5] + '_coco_aligned.json' 108 | with open(dataset_path_o) as fin: 109 | dataset = json.load(fin) 110 | 111 | # The new dataset that concatenates coco and refexp. 112 | coco_aligned_dataset = {} 113 | # Maps image id to image object that concatenates the image info from both coco and refexp. 114 | images = {} 115 | # Maps annotation id to annotation object that concatenates the annotation info from both coco and refexp. 116 | annotations = {} 117 | # Maps refering expression id to refexp object that contains refering expression info from Google refexp. 118 | refexps = {} 119 | 120 | # Annotations from GoogleRefexp. 121 | googleRefexp_annotations = dataset['annotations'] 122 | googleRefexp_refexps = dataset['refexps'] 123 | 124 | for ann in googleRefexp_annotations: 125 | if images.get(ann['image_id'], None) is None: 126 | # Take image object from coco. 127 | img_coco = copy.deepcopy(coco.loadImgs(ann['image_id'])[0]) 128 | # Extend with the region candidates info from GoogleRefexp. 129 | img_coco['region_candidates'] = ann['region_candidates'] 130 | # Change id to more specific image_id 131 | img_coco['image_id'] = img_coco['id'] 132 | img_coco.pop('id', None) 133 | # Add the new image object to the dictionary of image ids to image objects. 134 | images[ann['image_id']] = img_coco 135 | 136 | if annotations.get(ann['annotation_id'], None) is None: 137 | # Take annotation from coco. 138 | ann_coco = copy.deepcopy(coco.loadAnns(ann['annotation_id'])[0]) 139 | # Change id to more specific annotation_id 140 | ann_coco['annotation_id'] = ann_coco['id'] 141 | ann_coco.pop('id', None) 142 | # Extend annotation with the Google refering expression ids. 143 | ann_coco['refexp_ids'] = ann['refexp_ids'] 144 | # Add the new annotation object to the dictionary of annotation ids to annotation objects. 145 | annotations[ann['annotation_id']] = ann_coco 146 | else: 147 | anno_coco_o = annotations[ann['annotation_id']] 148 | anno_coco_o['refexp_ids'] += ann['refexp_ids'] 149 | 150 | # Set key in refering expression dictionary to be the refering expression id. 151 | for ref in googleRefexp_refexps: 152 | refexps[ref['refexp_id']] = ref 153 | 154 | # Set the images field to the dictionary of images created above. 155 | coco_aligned_dataset['images'] = images 156 | # Set the annotations field to the dictionary of annotations created above. 157 | coco_aligned_dataset['annotations'] = annotations 158 | # Set the refering expressions field in the dictionary of referering expressions created above. 159 | coco_aligned_dataset['refexps'] = refexps 160 | 161 | # Write data to file. 162 | with open(dataset_path_n, 'w') as fout: 163 | json.dump(coco_aligned_dataset, fout) 164 | -------------------------------------------------------------------------------- /google_refexp_py_lib/refexp.py: -------------------------------------------------------------------------------- 1 | # API for Google Refexp. 2 | 3 | from __future__ import print_function 4 | from pycocotools.coco import COCO 5 | import cPickle as pkl 6 | import os.path 7 | import json 8 | import warnings 9 | from common_utils import draw_bbox 10 | 11 | 12 | class Refexp: 13 | def __init__(self, dataset_filename, coco_filename, cache_filename=None): 14 | """ 15 | Constructor for Refexp class for reading and visualizing referent expressions dataset. 16 | :param dataset_filename (str): location of the dataset file 17 | :param coco_filename (str): location of the COCO instances JSON file (e.g. `instances_train2014.json`) 18 | :param cache_filename (str): if specified, writes/reads to this cache file. Should be specific to each (`dataset_filename`, `coco_filename`). 19 | :return: 20 | """ 21 | 22 | # Load from cache: 23 | if cache_filename is not None and os.path.isfile(cache_filename): 24 | print('Loading from cache...') 25 | with open(cache_filename, 'r') as f: 26 | obj = pkl.load(f) 27 | self.coco = obj['coco'] 28 | self.dataset = obj['dataset'] 29 | self.imgIds = obj['imgIds'] 30 | self.annIds = obj['annIds'] 31 | self.refexpIds = obj['refexpIds'] 32 | self.refexpToAnnId = obj['refexpToAnnId'] 33 | self.annToImgId = obj['annToImgId'] 34 | self.catIds = obj['catIds'] 35 | else: 36 | # Instantiate COCO class to use its methods. 37 | self.coco = COCO(coco_filename) 38 | 39 | # Load dataset form the _aligned_coco.json files generated by setup.py. 40 | with open(dataset_filename, 'r') as f: 41 | self.dataset = json.load(f) 42 | # Convert string keys to integers for images, annotations, refexps. 43 | def _convert(dataset_key): 44 | converted = {} 45 | for k, v in self.dataset[dataset_key].iteritems(): 46 | converted[int(k)] = v 47 | self.dataset[dataset_key] = converted 48 | _convert('images') 49 | _convert('annotations') 50 | _convert('refexps') 51 | print('Dataset loaded.') 52 | 53 | # List of all image ids in the dataset. 54 | self.imgIds = self.dataset['images'].keys() 55 | 56 | # List of all annotation ids in the dataset. 57 | self.annIds = self.dataset['annotations'].keys() 58 | 59 | # List of all referring expression ids in the dataset. 60 | self.refexpIds = self.dataset['refexps'].keys() 61 | 62 | # Map from referring expression id to corresponding annotation id (1:1). 63 | refexpToAnnId = {} 64 | # Map annotation id to image id (1:1). 65 | annToImgId = {} 66 | # Set of all category ids in the dataset. 67 | catIds = set([]) 68 | for ann_id, ann in self.dataset['annotations'].iteritems(): 69 | if ann['category_id'] not in catIds: 70 | catIds.add(ann['category_id']) 71 | # Check 1:1 relation. 72 | assert ann_id not in annToImgId 73 | annToImgId[ann_id] = ann['image_id'] 74 | for ref_id in ann['refexp_ids']: 75 | # Check 1:1 relation. 76 | assert ref_id not in refexpToAnnId 77 | refexpToAnnId[ref_id] = ann_id 78 | 79 | self.refexpToAnnId = refexpToAnnId 80 | self.annToImgId = annToImgId 81 | self.catIds = list(catIds) 82 | 83 | # And save to cache if cache_filename given: 84 | if cache_filename is not None: 85 | print('Writing to cache...') 86 | obj = {} 87 | obj['coco'] = self.coco 88 | obj['dataset'] = self.dataset 89 | obj['imgIds'] = self.imgIds 90 | obj['annIds'] = self.annIds 91 | obj['refexpIds'] = self.refexpIds 92 | obj['refexpToAnnId'] = self.refexpToAnnId 93 | obj['annToImgId'] = self.annToImgId 94 | obj['catIds'] = self.catIds 95 | with open(cache_filename, 'w') as f: 96 | pkl.dump(obj, f) 97 | 98 | # Helper functions for the rest of the COCO get functions. 99 | def _filterImgIds(self, imgIds): 100 | """ 101 | Sets a list of image ids to the intersection of itself with the list of GoogleRefexp image ids. 102 | Raises warning if the list contains images outside GoogleRefexp dataset. 103 | """ 104 | imgIds = imgIds if isinstance(imgIds, list) else [imgIds] 105 | if len(imgIds) == 0: 106 | return self.imgIds 107 | imgIdsSet = set(imgIds) 108 | refexpImgsSet = set(self.imgIds) 109 | notRefexpImages = imgIdsSet - refexpImgsSet 110 | if len(notRefexpImages) > 0: 111 | warnings.warn('Images ' + str(notRefexpImages) + ' are not part of the GoogleRefexp dataset and will be ignored from the answer.', RuntimeWarning) 112 | imgIds = imgIdsSet.intersection(refexpImgsSet) 113 | return list(imgIds) 114 | 115 | def _filterAnnIds(self, annIds, warn=True): 116 | """ 117 | Sets a list of annotation ids to the intersection of itself with the list of GoogleRefexp annotation ids. 118 | Raises warning if the input annotation id list contains ids outside GoogleRefexp dataset. 119 | """ 120 | annIds = annIds if isinstance(annIds, list) else [annIds] 121 | if len(annIds) == 0: 122 | return self.annIds 123 | annIdsSet = set(annIds) 124 | refexpAnnsSet = set(self.annIds) 125 | notRefexpAnns = annIdsSet - refexpAnnsSet 126 | if len(notRefexpAnns) > 0 and warn: 127 | warnings.warn('Annotations ' + str(notRefexpAnns) + ' are not part of the GoogleRefexp dataset and will be ignored from the answer.', RuntimeWarning) 128 | 129 | return list(annIdsSet.intersection(refexpAnnsSet)) 130 | 131 | def _filterCatIds(self, catIds): 132 | """ 133 | Sets a list of categories ids to the intersection of itself with the list of GoogleRefexp category ids. 134 | Raises warning if the category id list contains ids outside GoogleRefexp dataset. 135 | """ 136 | catIds = catIds if isinstance(catIds, list) else [catIds] 137 | if len(catIds) == 0: 138 | return self.catIds 139 | catIdsSet = set(catIds) 140 | refexpCatsSet = set(self.catIds) 141 | notRefexpCats = catIdsSet - refexpCatsSet 142 | if len(notRefexpCats) > 0: 143 | warnings.warn('Category ids ' + str(notRefexpCats) + ' are not part of the GoogleRefexp dataset and will be ignored from the answer.', RuntimeWarning) 144 | catIds = catIdsSet.intersection(refexpCatsSet) 145 | return list(catIds) 146 | 147 | # Below wrappers of the COCO toolkit methods, adding referring expression ids as additional filter. 148 | # Filters default to the set of GoogleRefexp images, annotations and categories(subset of COCO). 149 | def getAnnIds(self, imgIds=[], catIds=[], refexpIds=[], areaRng=[], iscrowd=None): 150 | """ 151 | Same as COCO.getAnnIds(). Gets only annotations that appear in GoogleRefexp. 152 | Adds filtering by referring expression ids. 153 | """ 154 | refexpIds = refexpIds if isinstance(refexpIds, list) else [refexpIds] 155 | imgIds = self._filterImgIds(imgIds) 156 | if len(refexpIds) == 0: 157 | return self._filterAnnIds(self.coco.getAnnIds(imgIds, catIds, areaRng, iscrowd), warn=False) 158 | annsForRefexps = set([]) 159 | for refexp_id in refexpIds: 160 | annsForRefexps.add(self.refexpToAnnId[refexp_id]) 161 | return list(annsForRefexps.intersection(set(self.coco.getAnnIds(imgIds, catIds, areaRng, iscrowd)))) 162 | 163 | def getCatIds(self, catNms=[], supNms=[], catIds=[]): 164 | """ 165 | Same as COCO.getAnnIds(). Gets only categories that appear in GoogleRefexp. 166 | """ 167 | catIds = self._filterCatIds(catIds) 168 | return self.coco.getCatIds(catNms, supNms, catIds) 169 | 170 | def getImgIds(self, imgIds=[], catIds=[], refexpIds=[]): 171 | """ 172 | Same as COCO.getImgIds(). Gets only images that appear in GoogleRefexp. 173 | Adds filtering by referring expression ids. 174 | """ 175 | refexpIds = refexpIds if isinstance(refexpIds, list) else [refexpIds] 176 | imgIds = self._filterImgIds(imgIds) 177 | if len(refexpIds) == 0: 178 | return self.coco.getImgIds(imgIds, catIds) 179 | imgsForRefexps = set([]) 180 | for refexp_id in refexpIds: 181 | annForRefexp = self.refexpToAnnId[refexp_id] 182 | imgsForRefexps.add(self.annToImgId[annForRefexp]) 183 | cocoImgs = set(self.coco.getImgIds(imgIds, catIds)) 184 | return list(cocoImgs.intersection(imgsForRefexps)) 185 | 186 | def loadAnns(self, ids=[]): 187 | """ 188 | Load annotations with the specified ids. 189 | :param ids (int array) : integer ids specifying anns 190 | :return: anns (object array) : loaded ann objects 191 | """ 192 | ids = self._filterAnnIds(ids) 193 | annotations = self.dataset['annotations'] 194 | if type(ids) == list: 195 | return [annotations[id] for id in ids] 196 | elif type(ids) == int: 197 | return [annotations[ids]] 198 | 199 | def loadCats(self, ids=[]): 200 | """ 201 | Load categories with the specified ids. 202 | :param ids (int array) : integer ids specifying categories 203 | :return: categories (object array) : loaded ann objects 204 | """ 205 | ids = self._filterCatIds(ids) 206 | return self.coco.loadCats(ids) 207 | 208 | def loadImgs(self, ids=[]): 209 | """ 210 | Load images with the specified ids. 211 | :param ids (int array) : integer ids specifying images 212 | :return: images (object array) : loaded images objects 213 | """ 214 | ids = ids if isinstance(ids, list) else [ids] 215 | ids = self._filterImgIds(ids) 216 | if len(ids) == 0: 217 | return [] 218 | images = self.dataset['images'] 219 | return [images[id] for id in ids] 220 | 221 | 222 | def loadRefexps(self, ids=[]): 223 | """ 224 | Load referring expressions with the specified ids. 225 | :param ids (int array) : integer ids specifying referring expressions 226 | :return: images (object array) : loaded referring expressions objects 227 | """ 228 | refexps = self.dataset['refexps'] 229 | if type(ids) == list: 230 | return [refexps[id] for id in ids] 231 | elif type(ids) == int: 232 | return [refexps[ids]] 233 | 234 | def download(self, tarDir=None, imgIds=[]): 235 | """ 236 | Wrapper for COCO.download(). Defaults to GoogleRefexp images only. 237 | """ 238 | imgIds = self._filterImgIds(imgIds) 239 | return self.coco.download(tarDir, imgIds) 240 | 241 | def showAnn(self, ann, ax=None, printRefexps=True): 242 | """ 243 | Display the bbox and referring expressions of the given annotation. 244 | For segmentation display refer to COCO toolkit. 245 | :param anns (array of object): annotations to display 246 | :return: None 247 | """ 248 | import matplotlib.pyplot as plt 249 | if ax is None: 250 | ax = plt.gca() 251 | bbox = ann['bbox'] 252 | draw_bbox(ax, bbox, edge_color='green') 253 | if printRefexps: 254 | print('Referring expressions for the object in the bounding box: ') 255 | for ref_id in ann['refexp_ids']: 256 | print(self.dataset['refexps'][ref_id]['raw']) 257 | 258 | def showRegionCandidates(self, image, ax=None): 259 | """ 260 | Display the region candidates for the given image. 261 | :param imgId (int): image id for which we display region candidates 262 | :param ax: matplotlib axes to draw on, or if unspecified, the current axis from `plt.gca()` 263 | :return: None 264 | """ 265 | import matplotlib.pyplot as plt 266 | if ax is None: 267 | ax = plt.gca() 268 | 269 | candidates = image['region_candidates'] 270 | for candidate in candidates: 271 | bbox = candidate['bounding_box'] 272 | draw_bbox(ax, bbox, edge_color='blue') 273 | 274 | # Methods specific to Refexp. 275 | def getRefexpIds(self, tokens=[], len_min=0, len_max=-1, referent=None, referent_has_attributes=False): 276 | """ 277 | Get description ids that contain all the tokens in the given list, 278 | legnth in the specified range and in which the referent has attributes. 279 | :param: contains_tokens (str array) : get refexps containing all given tokens. Defaults to all referring expressions. 280 | :param: len_min (int) : minimum number of tokens in the referring expression 281 | :param: len_max (int) : maximum number of tokens in the referring expression 282 | :param: referent (str) : the referent word in the referring expression 283 | :param: referent_has_attributes (bool) : if True return referring expressions in which the referent has attributes 284 | :return: ids (int array) : integer array of refexp 285 | """ 286 | tokens = tokens if isinstance(tokens, list) else [tokens] 287 | tokensSet = set(tokens) 288 | refexps_ids = set([]) 289 | for ref_id, ref in self.dataset['refexps'].iteritems(): 290 | if (set(ref['tokens']).issuperset(tokensSet) 291 | and len(ref['tokens']) >= len_min 292 | and (len_max == -1 or len(ref['tokens']) <= len_max) 293 | and (referent == None or ('word' in ref['parse']['referent'] and ref['parse']['referent']['word'] == referent)) 294 | and (not referent_has_attributes or 295 | ('amods' in ref['parse']['referent'] and len(ref['parse']['referent']['amods']) > 0))): 296 | refexps_ids.add(ref_id) 297 | return list(refexps_ids) 298 | -------------------------------------------------------------------------------- /google_refexp_py_lib/refexp_eval.py: -------------------------------------------------------------------------------- 1 | """Python class for the evaluation of Google Refexp dataset. 2 | 3 | This script contains two python classes: 4 | 1. GoogleRefexpEvalComprehension 5 | - Use precision@k score to evaluate comprehension task performance 6 | - Can evaluate generation task through an end-to-end way 7 | 2. GoogleRefexpEvalGeneration 8 | - Use Amazon Mechanical Turker (AMT) to compare generated refexps with GT 9 | with the following steps (step a, c, f covered by the class): 10 | a. Generate csv files for AMT 11 | b. Generate images and masked images 12 | c. Upload these images to a server (e.g. Amazon S3) so that the image are 13 | publicly accessible 14 | d. Create a AMT project with the interface at 15 | ./cache_evaluation/AMT_interface/AMT_template_generated_vs_GT.html 16 | e. Upload csv files and start AMT job 17 | f. Download annotated json file and calculate the score 18 | 19 | TO CHECK: 20 | GoogleRefexp.getAnnIds(): get COCO object ids 21 | GoogleRefexp.getRefexpIds(): get referring expression ids 22 | GoogleRefexp.getRefexpAnns(): get google refexp annotations for a list of annotation_id 23 | GoogleRefexp.getGtBoxes(): currently assume a dictionary with key of id, value of a list for bbox 24 | 25 | TODO: 26 | Comprehention: 27 | - A script that can visualize predicted bboxes whose iou satistied a constrain 28 | """ 29 | 30 | import json 31 | import os 32 | import copy 33 | import random 34 | import sys 35 | import numpy 36 | import csv 37 | from scipy import misc 38 | from PIL import Image, ImageDraw 39 | import matplotlib.pyplot as plt 40 | 41 | from refexp import Refexp # Need to check - change 42 | import common_utils as cu 43 | 44 | class RefexpEvalComprehension(object): 45 | def __init__(self, refexp_dataset_path, coco_data_path): 46 | """Constructor for GoogleRefexpEvalComprehension class for evaluation. 47 | 48 | Args: 49 | refexp_dataset_path: path for the Google Refexp dataset file 50 | coco_data_path: path for the original coco dataset file (e.g. 'instances_train2014.json') 51 | """ 52 | # handle refexp dataset file 53 | assert refexp_dataset_path, "Refexp dataset file missing!" 54 | self.refexp_dataset_path = refexp_dataset_path 55 | print 'Loading Google Refexp dataset file for the comprehension task.' 56 | self.refexp_dataset = Refexp(refexp_dataset_path, coco_data_path) # Need to check - change 57 | self.gt_ann_ids_set = frozenset(self.refexp_dataset.getAnnIds()) # Need to check - change 58 | self.gt_refexp_ids_set = frozenset(self.refexp_dataset.getRefexpIds()) # Need to check - change 59 | 60 | # reset evaluation state 61 | self.reset_eval_state() 62 | 63 | def reset_eval_state(self): 64 | """Reset evaluation state.""" 65 | self.pred_results_path = None 66 | self.pred_results = None 67 | self.flag_already_eval = False 68 | 69 | def evaluate(self, pred_results_path, 70 | thresh_iou=0.5, 71 | thresh_k=1, 72 | flag_ignore_non_existed_object=False, 73 | flag_ignore_non_existed_gt_refexp=False, 74 | flag_missing_objects_verbose=False, 75 | flag_missing_refexps_verbose=False): 76 | """Evaluate the predicted results for the comprehension task. 77 | 78 | Args: 79 | pred_results_path: path for the predicted results with the format 80 | described in ./cache_evaluation/format_comprehension_eval.md 81 | thresh_iou: threshold of the IoU ratio of the evaluation 82 | thresh_k: precision@k 83 | flag_ignore_non_existed_object: if set True, the evaluation process 84 | continues with an warning when encountered non existed objects in 85 | self.refexp_dataset. Otherwise stops. 86 | flag_ignore_non_existed_gt_refexp: if set True, the evaluation process 87 | continues when encountered non existed GT referring expressions. 88 | Otherwise stops. 89 | flag_missing_objects_verbose: if set true, will list the ids of all the 90 | missing objects in self.refexp_dataset 91 | flag_missing_refexps_verbose: if set true, will list the ids of all the 92 | missing referring expressions in self.refexp_dataset 93 | 94 | Returns: 95 | A two element tuple. The first element is precision@k. The second 96 | element is the predicted results (a dictionary) with an added field 97 | 'best_iou' of the best iou for the top k bounding boxes. 98 | """ 99 | # Load predicted results 100 | self.reset_eval_state() 101 | print 'Loading predicted result file for the comprehension task.' 102 | with open(pred_results_path) as fin: 103 | self.pred_results = json.load(fin) 104 | 105 | # evaluation 106 | pred_ann_ids_set = set() 107 | pred_refexp_ids_set = set() 108 | score = 0.0 109 | num_valid_pred = 0 110 | for pred_elem in self.pred_results: 111 | # validate the predicted results 112 | assert 'annotation_id' in pred_elem, 'Object annotation id missing!' 113 | assert 'predicted_bounding_boxes' in pred_elem, \ 114 | 'list of predicted bounding boxes missing!' 115 | ann_id = pred_elem['annotation_id'] 116 | gt_bbox = self._get_GT_bbox_with_annotation_id(ann_id) # Need to check - change 117 | if gt_bbox is None: 118 | if flag_ignore_non_existed_object: 119 | print ('Ignore COCO annotation id %d which does not exist in ' 120 | 'Refexp dataset file for evaluation' % ann_id) 121 | pred_elem['best_iou'] = 0.0 122 | continue 123 | else: 124 | print ('COCO annotation id %d does not exist in Refexp ' 125 | 'dataset file for evaluation!' % ann_id) 126 | raise 127 | if ('refexp_id' in pred_elem) and not(pred_elem['refexp_id'] in self.gt_refexp_ids_set): 128 | if flag_ignore_non_existed_gt_refexp: 129 | print ('Ignore refexp id %d which does not exist in ' 130 | 'Refexp dataset file for evaluation' % pred_elem['refexp_id']) 131 | pred_elem['best_iou'] = 0.0 132 | continue 133 | else: 134 | print ('refexp id %d does not exist in Refexp ' 135 | 'dataset file for evaluation!' % pred_elem['refexp_id']) 136 | raise 137 | pred_ann_ids_set.add(ann_id) 138 | if 'refexp_id' in pred_elem: 139 | pred_refexp_ids_set.add(pred_elem['refexp_id']) 140 | num_valid_pred += 1 141 | 142 | # check whether it is a correct prediction 143 | pred_bboxes = pred_elem['predicted_bounding_boxes'] 144 | best_iou = 0.0 145 | for k in xrange(min(thresh_k, len(pred_bboxes))): 146 | iou = cu.iou_bboxes(pred_bboxes[k], gt_bbox) 147 | best_iou = max(best_iou, iou) 148 | if best_iou >= thresh_iou: 149 | score += 1.0 150 | pred_elem['best_iou'] = best_iou 151 | score /= num_valid_pred 152 | 153 | # warning for missing objects and refexps 154 | gt_ann_ids_left_set = self.gt_ann_ids_set - pred_ann_ids_set 155 | gt_refexp_ids_left_set = self.gt_refexp_ids_set - pred_refexp_ids_set 156 | if gt_ann_ids_left_set: 157 | print ('Missing %d objects in the refexp dataset file in the predicted ' 158 | 'file' % len(gt_ann_ids_left_set)) 159 | if flag_missing_objects_verbose: 160 | print ('The missing object annotation ids are:') 161 | print gt_ann_ids_left_set # TODO pretty print format 162 | if gt_refexp_ids_left_set: 163 | print ('Missing %d refexps in the refexp dataset file in the predicted ' 164 | 'file' % len(gt_refexp_ids_left_set)) 165 | if flag_missing_refexps_verbose: 166 | print ('The missing refexp ids are:') 167 | print gt_refexp_ids_left_set # TODO pretty print format 168 | 169 | # summarize the results 170 | print 'The average prec@%d score is %.3f' % (thresh_k, score) 171 | return (score, self.pred_results) 172 | 173 | def _get_GT_bbox_with_annotation_id(self, ann_id): 174 | if not ann_id in self.gt_ann_ids_set: 175 | return None 176 | anns = self.refexp_dataset.loadAnns(ids = [ann_id]) 177 | if len(anns) == 0: 178 | return None 179 | assert len(anns) == 1 180 | return anns[0]['bbox'] 181 | 182 | def visualize_top_predicted_bbox(self, pred_sample, coco_image_dir): 183 | """Visualize the top predicted bounding box.""" 184 | assert 'annotation_id' in pred_sample, 'Object annotation id missing!' 185 | assert 'predicted_bounding_boxes' in pred_sample, \ 186 | 'list of predicted bounding boxes missing!' 187 | if not pred_sample['predicted_bounding_boxes']: 188 | print 'Empty predicted bounding boxes.' 189 | return 190 | 191 | bbox_pred_top = pred_sample['predicted_bounding_boxes'][0] 192 | ann_id = pred_sample['annotation_id'] 193 | ann = self.refexp_dataset.loadAnns(ids=[ann_id])[0] 194 | image_id = ann['image_id'] 195 | img_coco = self.refexp_dataset.loadImgs(ids=[image_id])[0] 196 | iou = cu.iou_bboxes(bbox_pred_top, ann['bbox']) 197 | 198 | if 'refexp' in pred_sample or 'refexp_id' in pred_sample: 199 | print 'The Referring expression input to the model is:' 200 | if 'refexp' in pred_sample: 201 | print ' ' + pred_sample['refexp'] 202 | else: 203 | refexp_tmp = self.refexp_dataset.loadRefexps(ids=pred_sample['refexp_id'])[0] 204 | print ' ' + refexp_tmp['raw'] 205 | 206 | I = misc.imread(os.path.join(coco_image_dir, (img_coco['file_name']))) 207 | ax = plt.imshow(I) 208 | ax = plt.axis('off') 209 | ax = plt.title('IoU: %.3f, green bbox: GT, red bbox: predicted' % iou) 210 | cu.draw_bbox(plt.gca(), ann['bbox'], edge_color='green') 211 | cu.draw_bbox(plt.gca(), bbox_pred_top, edge_color='red') 212 | 213 | 214 | class RefexpEvalGeneration(object): 215 | def __init__(self, refexp_dataset_path, coco_data_path): 216 | """Constructor for GoogleRefexpEvalGeneration class for evaluation. 217 | 218 | Args: 219 | refexp_dataset_path: path for the Google Refexp dataset file 220 | """ 221 | # handle refexp dataset file 222 | assert refexp_dataset_path, "Refexp dataset file missing!" 223 | self.refexp_dataset_path = refexp_dataset_path 224 | print 'Loading Google Refexp dataset file for the generation task.' 225 | self.refexp_dataset = Refexp(refexp_dataset_path, coco_data_path) # Need to check - change 226 | self.gt_ann_ids_set = frozenset(self.refexp_dataset.getAnnIds()) # Need to check - change 227 | 228 | def generate_AMT_csv_and_images(self, pred_results_path, 229 | public_image_url_prefix, 230 | AMT_csv_path, 231 | num_refexp_group=5, 232 | flag_generate_images=True, 233 | coco_image_dir=None, 234 | local_image_dir=None): 235 | """Generate a csv file and images for AMT evaluation. 236 | 237 | Args: 238 | pred_results_path: path for the predicted results with the format 239 | described in ./cache_evaluation/format_generation_eval.md 240 | public_image_url_prefix: image url prefix for the publicly accessible 241 | images. AMTurkers should be able to access images with this url prefix 242 | (see details in README.md, AMT section) 243 | AMT_csv_path: path for the generated csv file. 244 | num_refexp_group: the number of referring expressions that we plan to 245 | group as one HIT for AMT. default=5 (highly recommended, otherwise 246 | need to change AMT_interface) 247 | flag_generate_images: if set true, will generate images for AMT 248 | coco_image_dir: directory that coco images can be found, e.g. 249 | ./external/coco/images/train2014 250 | local_image_dir: directory to save the images locally. 251 | """ 252 | # Load predicted results 253 | print 'Loading predicted result file for the generation task.' 254 | with open(pred_results_path) as fin: 255 | self.pred_results = json.load(fin) 256 | assert len(self.pred_results) % num_refexp_group == 0, ('The number of ' 257 | 'generated sentences should be a multiple of num of images in the' 258 | 'AMT group (i.e. %d)' % num_refexp_group) 259 | 260 | # Generate csv file for AMT 261 | pred_ann_ids = self._generate_AMT_csv_file( 262 | AMT_csv_path, public_image_url_prefix, 263 | num_refexp_group=num_refexp_group) 264 | 265 | # Generate images for AMT if necessary 266 | if flag_generate_images: 267 | assert coco_image_dir, 'Missing the directory of original coco image' 268 | assert local_image_dir, 'Missing the local directory for storing images' 269 | self._generate_images_for_AMT(pred_ann_ids, 270 | coco_image_dir=coco_image_dir, local_image_dir=local_image_dir) 271 | 272 | def parse_AMT_results(self, csv_download_path, num_refexp_group=5): 273 | """Parse the AMT results from the downloaded csv file. 274 | 275 | Args: 276 | csv_download_path: the path of the downloaded csv result file from AMT. 277 | num_refexp_group: the number of the refexp grouped in a HIT. 278 | 279 | Return: 280 | A tuple with two numbers. They represent the ratio of the generated 281 | referring expressions are considered to be better and similar 282 | respectively. 283 | """ 284 | num_better = 0 285 | num_similar = 0 286 | num_row = 0 287 | with open(csv_download_path) as fin: 288 | reader = csv.DictReader(fin) 289 | for row in reader: 290 | for ind in xrange(num_refexp_group): 291 | key = 'Answer.choice_%d' % ind 292 | if row[key] == 'GEN': 293 | num_better += 1 294 | elif row[key] == 'similar': 295 | num_similar += 1 296 | num_row += 1 297 | ratio_better = num_better / float(num_row * num_refexp_group) 298 | ratio_similar = num_similar / float(num_row * num_refexp_group) 299 | print ('%.4f of the generated referring expressions are considered to be ' 300 | 'better than humans (groundtruth)' % ratio_better) 301 | print ('%.4f of the generated referring expressions are considered to be ' 302 | 'similar to humans (groundtruth)' % ratio_similar) 303 | return (ratio_better, ratio_similar) 304 | 305 | def _generate_AMT_csv_file(self, AMT_csv_path, public_image_url_prefix, 306 | num_refexp_group=5): 307 | """Private function to generate csv file for AMT.""" 308 | print 'Start to generate csv file to upload to AMT' 309 | fieldnames_template = ['image_url_o_%d', 'image_url_mask_%d', 310 | 'descrip_type_%d_0', 'descrip_type_%d_1', 311 | 'descrip_%d_0', 'descrip_%d_1'] 312 | 313 | pred_ann_ids = [] 314 | ind_cur = 0 315 | with open(AMT_csv_path, 'w') as fout: 316 | while ind_cur < len(self.pred_results): 317 | dct_row = {} 318 | fields_all = [] 319 | for ind_group in xrange(num_refexp_group): 320 | # check pred_result format 321 | pred_elem = self.pred_results[ind_cur] 322 | assert 'annotation_id' in pred_elem, 'Object annotation id missing!' 323 | assert 'generated_refexp' in pred_elem, 'Generated refexp missing!' 324 | pred_ann_id = pred_elem['annotation_id'] 325 | # load GT data 326 | assert pred_ann_id in self.gt_ann_ids_set, ('Cannot find object with' 327 | 'annotation id %d' % pred_ann_id) 328 | gt_data = self.refexp_dataset.loadAnns(ids = [pred_ann_id])[0] # Need to check - change 329 | gt_refexps = self.refexp_dataset.loadRefexps(ids = gt_data['refexp_ids']) # Need to check - change 330 | pred_ann_ids.append(pred_ann_id) 331 | # add fieldnames 332 | for field_template in fieldnames_template: 333 | fields_all.append(field_template % ind_group) 334 | # add image urls 335 | img_name = 'coco_%d.jpg' % gt_data['image_id'] 336 | img_mask_name = 'coco_%d_ann_%d_masked.jpg' % (gt_data['image_id'], pred_ann_id) 337 | dct_row['image_url_o_%d' % ind_group] = public_image_url_prefix + img_name 338 | dct_row['image_url_mask_%d' % ind_group] = public_image_url_prefix + img_mask_name 339 | # get refexp and type, shuffle them (refexp, type) 340 | descrip_gen = (pred_elem['generated_refexp'], 'GEN') 341 | descrip_gt = (' '.join(gt_refexps[0]['tokens']), 'GT') # Need to check - change 342 | list_descrip = [descrip_gen, descrip_gt] 343 | random.shuffle(list_descrip) 344 | for ind in xrange(2): 345 | dct_row['descrip_%d_%d' % (ind_group, ind)] = list_descrip[ind][0] 346 | dct_row['descrip_type_%d_%d' % (ind_group, ind)] = list_descrip[ind][1] 347 | ind_cur += 1 348 | 349 | # write row to csv files 350 | assert len(dct_row) == len(fields_all) 351 | if ind_cur == num_refexp_group: 352 | writer = csv.DictWriter(fout, fieldnames=fields_all) 353 | writer.writeheader() 354 | writer.writerow(dct_row) 355 | print 'Finished to generate the csv file' 356 | return pred_ann_ids 357 | 358 | def _generate_images_for_AMT(self, pred_ann_ids, 359 | coco_image_dir=None, local_image_dir=None): 360 | """Private function to generated images to upload to AMT.""" 361 | assert coco_image_dir and local_image_dir 362 | assert os.path.isdir(coco_image_dir) 363 | if not os.path.isdir(local_image_dir): 364 | print 'Input local image directory does not exist, create it' 365 | os.makedirs(local_image_dir) 366 | 367 | print 'Start to generate images for AMT in local hard disk' 368 | image_ids_saved = set() 369 | for (ind, pred_ann_id) in enumerate(pred_ann_ids): 370 | gt_data = self.refexp_dataset.loadAnns(ids = [pred_ann_id])[0] # Need to check - change 371 | img = self._read_image(coco_image_dir, gt_data) 372 | mask = self._load_mask(gt_data) 373 | masked_img = cu.apply_mask_to_image(img, mask) 374 | masked_img_path = os.path.join(local_image_dir, ('coco_%d_ann_%d' 375 | '_masked.jpg' % (gt_data['image_id'], pred_ann_id))) 376 | misc.imsave(masked_img_path, masked_img) 377 | if not gt_data['image_id'] in image_ids_saved: 378 | image_ids_saved.add(gt_data['image_id']) 379 | img_path = os.path.join(local_image_dir, 'coco_%d.jpg' % gt_data['image_id']) 380 | misc.imsave(img_path, img) 381 | print ('Images generated in local hard disk, please make sure to make them ' 382 | 'publicly available online.') 383 | 384 | def _read_image(self, coco_image_dir, gt_data): 385 | """Private function to read an original coco image.""" 386 | img_coco = self.refexp_dataset.loadImgs(ids=gt_data['image_id'])[0] 387 | return misc.imread(os.path.join(coco_image_dir, img_coco['file_name'])) 388 | 389 | def _load_mask(self, gt_data): 390 | """Private function to load the mask of a coco object.""" 391 | img_coco = self.refexp_dataset.loadImgs(ids=gt_data['image_id'])[0] 392 | mask = Image.new('L', (img_coco['width'], img_coco['height']), 0) 393 | for seg in gt_data['segmentation']: 394 | ImageDraw.Draw(mask).polygon(seg, outline='white', fill='white') 395 | return numpy.asarray(mask) 396 | -------------------------------------------------------------------------------- /evaluation/sample_results/sample_results_generation.json: -------------------------------------------------------------------------------- 1 | [{"annotation_id": 298801, "generated_refexp": "tie"}, {"annotation_id": 453172, "generated_refexp": "person"}, {"annotation_id": 401571, "generated_refexp": "truck"}, {"annotation_id": 584873, "generated_refexp": "elephant"}, {"annotation_id": 1154908, "generated_refexp": "vase"}, {"annotation_id": 381043, "generated_refexp": "chair"}, {"annotation_id": 100704, "generated_refexp": "chair"}, {"annotation_id": 1667788, "generated_refexp": "vase"}, {"annotation_id": 172040, "generated_refexp": "train"}, {"annotation_id": 29589, "generated_refexp": "tv"}, {"annotation_id": 1957469, "generated_refexp": "bed"}, {"annotation_id": 438539, "generated_refexp": "person"}, {"annotation_id": 432201, "generated_refexp": "person"}, {"annotation_id": 79063, "generated_refexp": "bottle"}, {"annotation_id": 418173, "generated_refexp": "parking meter"}, {"annotation_id": 1387963, "generated_refexp": "fire hydrant"}, {"annotation_id": 207320, "generated_refexp": "person"}, {"annotation_id": 157059, "generated_refexp": "airplane"}, {"annotation_id": 590216, "generated_refexp": "zebra"}, {"annotation_id": 522176, "generated_refexp": "person"}, {"annotation_id": 225363, "generated_refexp": "person"}, {"annotation_id": 453508, "generated_refexp": "person"}, {"annotation_id": 1468559, "generated_refexp": "kite"}, {"annotation_id": 2149160, "generated_refexp": "person"}, {"annotation_id": 213657, "generated_refexp": "person"}, {"annotation_id": 184108, "generated_refexp": "person"}, {"annotation_id": 1048737, "generated_refexp": "apple"}, {"annotation_id": 373989, "generated_refexp": "chair"}, {"annotation_id": 393345, "generated_refexp": "dining table"}, {"annotation_id": 2166892, "generated_refexp": "person"}, {"annotation_id": 416004, "generated_refexp": "fire hydrant"}, {"annotation_id": 184959, "generated_refexp": "person"}, {"annotation_id": 60468, "generated_refexp": "horse"}, {"annotation_id": 597582, "generated_refexp": "giraffe"}, {"annotation_id": 499287, "generated_refexp": "person"}, {"annotation_id": 26893, "generated_refexp": "potted plant"}, {"annotation_id": 1720455, "generated_refexp": "person"}, {"annotation_id": 108052, "generated_refexp": "chair"}, {"annotation_id": 115801, "generated_refexp": "couch"}, {"annotation_id": 209697, "generated_refexp": "person"}, {"annotation_id": 1445846, "generated_refexp": "suitcase"}, {"annotation_id": 473510, "generated_refexp": "person"}, {"annotation_id": 479787, "generated_refexp": "person"}, {"annotation_id": 381178, "generated_refexp": "chair"}, {"annotation_id": 455625, "generated_refexp": "person"}, {"annotation_id": 638442, "generated_refexp": "skateboard"}, {"annotation_id": 589064, "generated_refexp": "zebra"}, {"annotation_id": 1744784, "generated_refexp": "person"}, {"annotation_id": 1625095, "generated_refexp": "tv"}, {"annotation_id": 1085309, "generated_refexp": "cake"}, {"annotation_id": 443556, "generated_refexp": "person"}, {"annotation_id": 1069026, "generated_refexp": "hot dog"}, {"annotation_id": 205373, "generated_refexp": "person"}, {"annotation_id": 192463, "generated_refexp": "person"}, {"annotation_id": 1620331, "generated_refexp": "dining table"}, {"annotation_id": 474228, "generated_refexp": "person"}, {"annotation_id": 1932002, "generated_refexp": "chair"}, {"annotation_id": 365993, "generated_refexp": "bus"}, {"annotation_id": 522995, "generated_refexp": "person"}, {"annotation_id": 108494, "generated_refexp": "chair"}, {"annotation_id": 587642, "generated_refexp": "bear"}, {"annotation_id": 2061563, "generated_refexp": "bench"}, {"annotation_id": 1965216, "generated_refexp": "dining table"}, {"annotation_id": 1577240, "generated_refexp": "cake"}, {"annotation_id": 2174167, "generated_refexp": "bird"}, {"annotation_id": 1821404, "generated_refexp": "elephant"}, {"annotation_id": 379937, "generated_refexp": "chair"}, {"annotation_id": 1091513, "generated_refexp": "dining table"}, {"annotation_id": 1098398, "generated_refexp": "laptop"}, {"annotation_id": 397485, "generated_refexp": "truck"}, {"annotation_id": 196399, "generated_refexp": "person"}, {"annotation_id": 596130, "generated_refexp": "giraffe"}, {"annotation_id": 1956039, "generated_refexp": "potted plant"}, {"annotation_id": 1534158, "generated_refexp": "bowl"}, {"annotation_id": 1152377, "generated_refexp": "vase"}, {"annotation_id": 536186, "generated_refexp": "person"}, {"annotation_id": 1084979, "generated_refexp": "cake"}, {"annotation_id": 108076, "generated_refexp": "chair"}, {"annotation_id": 489574, "generated_refexp": "person"}, {"annotation_id": 1651788, "generated_refexp": "book"}, {"annotation_id": 567255, "generated_refexp": "person"}, {"annotation_id": 2165132, "generated_refexp": "person"}, {"annotation_id": 1081377, "generated_refexp": "donut"}, {"annotation_id": 467111, "generated_refexp": "person"}, {"annotation_id": 661161, "generated_refexp": "wine glass"}, {"annotation_id": 2160344, "generated_refexp": "person"}, {"annotation_id": 54905, "generated_refexp": "horse"}, {"annotation_id": 524467, "generated_refexp": "person"}, {"annotation_id": 643425, "generated_refexp": "skateboard"}, {"annotation_id": 2150776, "generated_refexp": "person"}, {"annotation_id": 1159899, "generated_refexp": "teddy bear"}, {"annotation_id": 385941, "generated_refexp": "chair"}, {"annotation_id": 581857, "generated_refexp": "elephant"}, {"annotation_id": 581384, "generated_refexp": "elephant"}, {"annotation_id": 573614, "generated_refexp": "bench"}, {"annotation_id": 1121234, "generated_refexp": "oven"}, {"annotation_id": 1730481, "generated_refexp": "person"}, {"annotation_id": 453326, "generated_refexp": "person"}, {"annotation_id": 537966, "generated_refexp": "person"}, {"annotation_id": 113428, "generated_refexp": "couch"}, {"annotation_id": 2229035, "generated_refexp": "person"}, {"annotation_id": 1075846, "generated_refexp": "pizza"}, {"annotation_id": 253561, "generated_refexp": "person"}, {"annotation_id": 1534318, "generated_refexp": "bowl"}, {"annotation_id": 583742, "generated_refexp": "elephant"}, {"annotation_id": 422638, "generated_refexp": "person"}, {"annotation_id": 426027, "generated_refexp": "person"}, {"annotation_id": 229403, "generated_refexp": "person"}, {"annotation_id": 1605998, "generated_refexp": "potted plant"}, {"annotation_id": 291163, "generated_refexp": "couch"}, {"annotation_id": 1073646, "generated_refexp": "pizza"}, {"annotation_id": 668759, "generated_refexp": "cup"}, {"annotation_id": 112603, "generated_refexp": "couch"}, {"annotation_id": 234346, "generated_refexp": "person"}, {"annotation_id": 508004, "generated_refexp": "person"}, {"annotation_id": 460577, "generated_refexp": "person"}, {"annotation_id": 613570, "generated_refexp": "skis"}, {"annotation_id": 1846358, "generated_refexp": "skis"}, {"annotation_id": 428800, "generated_refexp": "person"}, {"annotation_id": 2205589, "generated_refexp": "person"}, {"annotation_id": 507596, "generated_refexp": "person"}, {"annotation_id": 623716, "generated_refexp": "kite"}, {"annotation_id": 597566, "generated_refexp": "giraffe"}, {"annotation_id": 497692, "generated_refexp": "person"}, {"annotation_id": 1157545, "generated_refexp": "scissors"}, {"annotation_id": 1690688, "generated_refexp": "person"}, {"annotation_id": 418266, "generated_refexp": "parking meter"}, {"annotation_id": 1707669, "generated_refexp": "person"}, {"annotation_id": 605788, "generated_refexp": "skis"}, {"annotation_id": 454048, "generated_refexp": "person"}, {"annotation_id": 62422, "generated_refexp": "sheep"}, {"annotation_id": 694426, "generated_refexp": "knife"}, {"annotation_id": 1101202, "generated_refexp": "laptop"}, {"annotation_id": 588601, "generated_refexp": "zebra"}, {"annotation_id": 60141, "generated_refexp": "horse"}, {"annotation_id": 4095, "generated_refexp": "dog"}, {"annotation_id": 444265, "generated_refexp": "person"}, {"annotation_id": 1100285, "generated_refexp": "laptop"}, {"annotation_id": 63517, "generated_refexp": "sheep"}, {"annotation_id": 1938820, "generated_refexp": "chair"}, {"annotation_id": 1716310, "generated_refexp": "person"}, {"annotation_id": 1864523, "generated_refexp": "tennis racket"}, {"annotation_id": 366497, "generated_refexp": "bus"}, {"annotation_id": 1076634, "generated_refexp": "pizza"}, {"annotation_id": 216928, "generated_refexp": "person"}, {"annotation_id": 513763, "generated_refexp": "person"}, {"annotation_id": 1402086, "generated_refexp": "cat"}, {"annotation_id": 1051897, "generated_refexp": "orange"}, {"annotation_id": 1121366, "generated_refexp": "oven"}, {"annotation_id": 1080273, "generated_refexp": "donut"}, {"annotation_id": 309539, "generated_refexp": "sandwich"}, {"annotation_id": 30669, "generated_refexp": "tv"}, {"annotation_id": 460610, "generated_refexp": "person"}, {"annotation_id": 199253, "generated_refexp": "person"}, {"annotation_id": 1790270, "generated_refexp": "motorcycle"}, {"annotation_id": 280506, "generated_refexp": "umbrella"}, {"annotation_id": 43908, "generated_refexp": "bird"}, {"annotation_id": 661382, "generated_refexp": "wine glass"}, {"annotation_id": 1740443, "generated_refexp": "person"}, {"annotation_id": 1071030, "generated_refexp": "pizza"}, {"annotation_id": 649543, "generated_refexp": "surfboard"}, {"annotation_id": 284478, "generated_refexp": "umbrella"}, {"annotation_id": 219779, "generated_refexp": "person"}, {"annotation_id": 1044306, "generated_refexp": "banana"}, {"annotation_id": 33575, "generated_refexp": "tv"}, {"annotation_id": 515654, "generated_refexp": "person"}, {"annotation_id": 1080754, "generated_refexp": "donut"}, {"annotation_id": 582207, "generated_refexp": "elephant"}, {"annotation_id": 595253, "generated_refexp": "giraffe"}, {"annotation_id": 1214084, "generated_refexp": "person"}, {"annotation_id": 460731, "generated_refexp": "person"}, {"annotation_id": 373167, "generated_refexp": "chair"}, {"annotation_id": 1159325, "generated_refexp": "teddy bear"}, {"annotation_id": 1162625, "generated_refexp": "teddy bear"}, {"annotation_id": 448138, "generated_refexp": "person"}, {"annotation_id": 203163, "generated_refexp": "person"}, {"annotation_id": 2174556, "generated_refexp": "cat"}, {"annotation_id": 1117329, "generated_refexp": "keyboard"}, {"annotation_id": 1923538, "generated_refexp": "pizza"}, {"annotation_id": 1413929, "generated_refexp": "giraffe"}, {"annotation_id": 1202869, "generated_refexp": "person"}, {"annotation_id": 181899, "generated_refexp": "boat"}, {"annotation_id": 397758, "generated_refexp": "truck"}, {"annotation_id": 450193, "generated_refexp": "person"}, {"annotation_id": 73172, "generated_refexp": "cow"}, {"annotation_id": 471964, "generated_refexp": "person"}, {"annotation_id": 197753, "generated_refexp": "person"}, {"annotation_id": 581176, "generated_refexp": "elephant"}, {"annotation_id": 413149, "generated_refexp": "dining table"}, {"annotation_id": 459403, "generated_refexp": "person"}, {"annotation_id": 153273, "generated_refexp": "motorcycle"}, {"annotation_id": 309401, "generated_refexp": "sandwich"}, {"annotation_id": 1871616, "generated_refexp": "bottle"}, {"annotation_id": 120188, "generated_refexp": "dining table"}, {"annotation_id": 153062, "generated_refexp": "motorcycle"}, {"annotation_id": 628193, "generated_refexp": "baseball bat"}, {"annotation_id": 1716693, "generated_refexp": "person"}, {"annotation_id": 715930, "generated_refexp": "bowl"}, {"annotation_id": 1823626, "generated_refexp": "giraffe"}, {"annotation_id": 1237720, "generated_refexp": "person"}, {"annotation_id": 173106, "generated_refexp": "train"}, {"annotation_id": 485992, "generated_refexp": "person"}, {"annotation_id": 502345, "generated_refexp": "person"}, {"annotation_id": 201122, "generated_refexp": "person"}, {"annotation_id": 55719, "generated_refexp": "horse"}, {"annotation_id": 718077, "generated_refexp": "bowl"}, {"annotation_id": 104554, "generated_refexp": "chair"}, {"annotation_id": 474602, "generated_refexp": "person"}, {"annotation_id": 173000, "generated_refexp": "train"}, {"annotation_id": 81207, "generated_refexp": "bottle"}, {"annotation_id": 614745, "generated_refexp": "skis"}, {"annotation_id": 220478, "generated_refexp": "person"}, {"annotation_id": 421793, "generated_refexp": "person"}, {"annotation_id": 443396, "generated_refexp": "person"}, {"annotation_id": 1970686, "generated_refexp": "laptop"}, {"annotation_id": 447450, "generated_refexp": "person"}, {"annotation_id": 439918, "generated_refexp": "person"}, {"annotation_id": 312208, "generated_refexp": "sandwich"}, {"annotation_id": 381751, "generated_refexp": "chair"}, {"annotation_id": 581063, "generated_refexp": "elephant"}, {"annotation_id": 176572, "generated_refexp": "boat"}, {"annotation_id": 2157143, "generated_refexp": "person"}, {"annotation_id": 434919, "generated_refexp": "person"}, {"annotation_id": 252681, "generated_refexp": "person"}, {"annotation_id": 2168180, "generated_refexp": "bicycle"}, {"annotation_id": 472695, "generated_refexp": "person"}, {"annotation_id": 1075753, "generated_refexp": "pizza"}, {"annotation_id": 1816336, "generated_refexp": "cat"}, {"annotation_id": 476083, "generated_refexp": "person"}, {"annotation_id": 41510, "generated_refexp": "bird"}, {"annotation_id": 214939, "generated_refexp": "person"}, {"annotation_id": 2154138, "generated_refexp": "person"}, {"annotation_id": 1137080, "generated_refexp": "book"}, {"annotation_id": 589237, "generated_refexp": "zebra"}, {"annotation_id": 150536, "generated_refexp": "motorcycle"}, {"annotation_id": 713175, "generated_refexp": "bowl"}, {"annotation_id": 559743, "generated_refexp": "person"}, {"annotation_id": 483508, "generated_refexp": "person"}, {"annotation_id": 523595, "generated_refexp": "person"}, {"annotation_id": 444187, "generated_refexp": "person"}, {"annotation_id": 475601, "generated_refexp": "person"}, {"annotation_id": 469830, "generated_refexp": "person"}, {"annotation_id": 315605, "generated_refexp": "bed"}, {"annotation_id": 589476, "generated_refexp": "zebra"}, {"annotation_id": 351477, "generated_refexp": "car"}, {"annotation_id": 1160306, "generated_refexp": "teddy bear"}, {"annotation_id": 440573, "generated_refexp": "person"}, {"annotation_id": 502428, "generated_refexp": "person"}, {"annotation_id": 1072894, "generated_refexp": "pizza"}, {"annotation_id": 342042, "generated_refexp": "toothbrush"}, {"annotation_id": 497628, "generated_refexp": "person"}, {"annotation_id": 441432, "generated_refexp": "person"}, {"annotation_id": 1816611, "generated_refexp": "cat"}, {"annotation_id": 1933925, "generated_refexp": "chair"}, {"annotation_id": 51559, "generated_refexp": "cat"}, {"annotation_id": 1962763, "generated_refexp": "dining table"}, {"annotation_id": 1103404, "generated_refexp": "laptop"}, {"annotation_id": 2207664, "generated_refexp": "person"}, {"annotation_id": 599780, "generated_refexp": "giraffe"}, {"annotation_id": 2162637, "generated_refexp": "person"}, {"annotation_id": 1043529, "generated_refexp": "banana"}, {"annotation_id": 507197, "generated_refexp": "person"}, {"annotation_id": 1055064, "generated_refexp": "broccoli"}, {"annotation_id": 576654, "generated_refexp": "bench"}, {"annotation_id": 452027, "generated_refexp": "person"}, {"annotation_id": 1906694, "generated_refexp": "apple"}, {"annotation_id": 2225300, "generated_refexp": "couch"}, {"annotation_id": 1061932, "generated_refexp": "carrot"}, {"annotation_id": 1734041, "generated_refexp": "person"}, {"annotation_id": 33032, "generated_refexp": "tv"}, {"annotation_id": 507500, "generated_refexp": "person"}, {"annotation_id": 480467, "generated_refexp": "person"}, {"annotation_id": 1097790, "generated_refexp": "laptop"}, {"annotation_id": 494919, "generated_refexp": "person"}, {"annotation_id": 1074788, "generated_refexp": "pizza"}, {"annotation_id": 600764, "generated_refexp": "giraffe"}, {"annotation_id": 1361827, "generated_refexp": "motorcycle"}, {"annotation_id": 186901, "generated_refexp": "person"}, {"annotation_id": 444226, "generated_refexp": "person"}, {"annotation_id": 1710131, "generated_refexp": "person"}, {"annotation_id": 2214495, "generated_refexp": "cup"}, {"annotation_id": 130374, "generated_refexp": "bicycle"}, {"annotation_id": 1183982, "generated_refexp": "suitcase"}, {"annotation_id": 28390, "generated_refexp": "tv"}, {"annotation_id": 498436, "generated_refexp": "person"}, {"annotation_id": 1588209, "generated_refexp": "chair"}, {"annotation_id": 214754, "generated_refexp": "person"}, {"annotation_id": 203730, "generated_refexp": "person"}, {"annotation_id": 165363, "generated_refexp": "bus"}, {"annotation_id": 427459, "generated_refexp": "person"}, {"annotation_id": 106190, "generated_refexp": "chair"}, {"annotation_id": 1715849, "generated_refexp": "person"}, {"annotation_id": 483024, "generated_refexp": "person"}, {"annotation_id": 472187, "generated_refexp": "person"}, {"annotation_id": 177396, "generated_refexp": "boat"}, {"annotation_id": 587301, "generated_refexp": "bear"}, {"annotation_id": 1171642, "generated_refexp": "backpack"}, {"annotation_id": 1451356, "generated_refexp": "skis"}, {"annotation_id": 163864, "generated_refexp": "bus"}, {"annotation_id": 348609, "generated_refexp": "car"}, {"annotation_id": 451107, "generated_refexp": "person"}, {"annotation_id": 460237, "generated_refexp": "person"}, {"annotation_id": 1048960, "generated_refexp": "apple"}, {"annotation_id": 1899086, "generated_refexp": "bowl"}, {"annotation_id": 504101, "generated_refexp": "person"}, {"annotation_id": 419067, "generated_refexp": "parking meter"}, {"annotation_id": 1975473, "generated_refexp": "keyboard"}, {"annotation_id": 186177, "generated_refexp": "person"}, {"annotation_id": 1077854, "generated_refexp": "pizza"}, {"annotation_id": 605833, "generated_refexp": "skis"}, {"annotation_id": 1934938, "generated_refexp": "chair"}, {"annotation_id": 58485, "generated_refexp": "horse"}, {"annotation_id": 222929, "generated_refexp": "person"}, {"annotation_id": 1592539, "generated_refexp": "chair"}, {"annotation_id": 1162173, "generated_refexp": "teddy bear"}, {"annotation_id": 459247, "generated_refexp": "person"}, {"annotation_id": 439538, "generated_refexp": "person"}, {"annotation_id": 63686, "generated_refexp": "sheep"}, {"annotation_id": 195123, "generated_refexp": "person"}, {"annotation_id": 533343, "generated_refexp": "person"}, {"annotation_id": 1635087, "generated_refexp": "keyboard"}, {"annotation_id": 340251, "generated_refexp": "clock"}, {"annotation_id": 1052756, "generated_refexp": "orange"}, {"annotation_id": 696584, "generated_refexp": "knife"}, {"annotation_id": 505524, "generated_refexp": "person"}, {"annotation_id": 582808, "generated_refexp": "elephant"}, {"annotation_id": 2157739, "generated_refexp": "person"}, {"annotation_id": 26660, "generated_refexp": "potted plant"}, {"annotation_id": 557474, "generated_refexp": "person"}, {"annotation_id": 172160, "generated_refexp": "train"}, {"annotation_id": 524475, "generated_refexp": "person"}, {"annotation_id": 1767777, "generated_refexp": "bicycle"}, {"annotation_id": 1708923, "generated_refexp": "person"}, {"annotation_id": 1852179, "generated_refexp": "snowboard"}, {"annotation_id": 599465, "generated_refexp": "giraffe"}, {"annotation_id": 273457, "generated_refexp": "cat"}, {"annotation_id": 15860, "generated_refexp": "dog"}, {"annotation_id": 1226564, "generated_refexp": "person"}, {"annotation_id": 1642095, "generated_refexp": "oven"}, {"annotation_id": 191536, "generated_refexp": "person"}, {"annotation_id": 460330, "generated_refexp": "person"}, {"annotation_id": 519576, "generated_refexp": "person"}, {"annotation_id": 497079, "generated_refexp": "person"}, {"annotation_id": 442141, "generated_refexp": "person"}, {"annotation_id": 1045298, "generated_refexp": "banana"}, {"annotation_id": 42236, "generated_refexp": "bird"}, {"annotation_id": 1620652, "generated_refexp": "dining table"}, {"annotation_id": 231272, "generated_refexp": "person"}, {"annotation_id": 1100462, "generated_refexp": "laptop"}, {"annotation_id": 621590, "generated_refexp": "kite"}, {"annotation_id": 13854, "generated_refexp": "dog"}, {"annotation_id": 1127043, "generated_refexp": "oven"}, {"annotation_id": 1899636, "generated_refexp": "bowl"}, {"annotation_id": 2152606, "generated_refexp": "person"}, {"annotation_id": 116267, "generated_refexp": "couch"}, {"annotation_id": 1611312, "generated_refexp": "bed"}, {"annotation_id": 440215, "generated_refexp": "person"}, {"annotation_id": 57859, "generated_refexp": "horse"}, {"annotation_id": 203477, "generated_refexp": "person"}, {"annotation_id": 418856, "generated_refexp": "parking meter"}, {"annotation_id": 489852, "generated_refexp": "person"}, {"annotation_id": 395388, "generated_refexp": "truck"}, {"annotation_id": 212606, "generated_refexp": "person"}, {"annotation_id": 1237892, "generated_refexp": "person"}, {"annotation_id": 211890, "generated_refexp": "person"}, {"annotation_id": 599679, "generated_refexp": "giraffe"}, {"annotation_id": 447398, "generated_refexp": "person"}, {"annotation_id": 1396205, "generated_refexp": "bench"}, {"annotation_id": 1612967, "generated_refexp": "dining table"}, {"annotation_id": 382809, "generated_refexp": "chair"}, {"annotation_id": 343198, "generated_refexp": "toothbrush"}, {"annotation_id": 477387, "generated_refexp": "person"}, {"annotation_id": 468409, "generated_refexp": "person"}, {"annotation_id": 16191, "generated_refexp": "dog"}, {"annotation_id": 228181, "generated_refexp": "person"}, {"annotation_id": 498740, "generated_refexp": "person"}, {"annotation_id": 502911, "generated_refexp": "person"}, {"annotation_id": 1586963, "generated_refexp": "chair"}, {"annotation_id": 82427, "generated_refexp": "bottle"}, {"annotation_id": 232589, "generated_refexp": "person"}, {"annotation_id": 53671, "generated_refexp": "horse"}, {"annotation_id": 590411, "generated_refexp": "zebra"}, {"annotation_id": 157276, "generated_refexp": "airplane"}, {"annotation_id": 310956, "generated_refexp": "sandwich"}, {"annotation_id": 139781, "generated_refexp": "car"}, {"annotation_id": 343162, "generated_refexp": "toothbrush"}, {"annotation_id": 398610, "generated_refexp": "truck"}, {"annotation_id": 2204383, "generated_refexp": "person"}, {"annotation_id": 523513, "generated_refexp": "person"}, {"annotation_id": 434163, "generated_refexp": "person"}, {"annotation_id": 587498, "generated_refexp": "bear"}, {"annotation_id": 1102106, "generated_refexp": "laptop"}, {"annotation_id": 1738007, "generated_refexp": "person"}, {"annotation_id": 577270, "generated_refexp": "bench"}, {"annotation_id": 1734409, "generated_refexp": "person"}, {"annotation_id": 600318, "generated_refexp": "giraffe"}, {"annotation_id": 1891925, "generated_refexp": "knife"}, {"annotation_id": 585542, "generated_refexp": "elephant"}, {"annotation_id": 511142, "generated_refexp": "person"}, {"annotation_id": 556284, "generated_refexp": "person"}, {"annotation_id": 1133925, "generated_refexp": "sink"}, {"annotation_id": 427914, "generated_refexp": "person"}, {"annotation_id": 1936645, "generated_refexp": "chair"}, {"annotation_id": 592107, "generated_refexp": "zebra"}, {"annotation_id": 2159240, "generated_refexp": "person"}, {"annotation_id": 1730671, "generated_refexp": "person"}, {"annotation_id": 309513, "generated_refexp": "sandwich"}, {"annotation_id": 1607199, "generated_refexp": "potted plant"}, {"annotation_id": 500940, "generated_refexp": "person"}, {"annotation_id": 160676, "generated_refexp": "airplane"}, {"annotation_id": 1043239, "generated_refexp": "banana"}, {"annotation_id": 1049508, "generated_refexp": "apple"}, {"annotation_id": 64965, "generated_refexp": "sheep"}, {"annotation_id": 651710, "generated_refexp": "surfboard"}, {"annotation_id": 19024, "generated_refexp": "potted plant"}, {"annotation_id": 424164, "generated_refexp": "person"}, {"annotation_id": 56455, "generated_refexp": "horse"}, {"annotation_id": 1060588, "generated_refexp": "broccoli"}, {"annotation_id": 201232, "generated_refexp": "person"}, {"annotation_id": 186594, "generated_refexp": "person"}, {"annotation_id": 1096411, "generated_refexp": "toilet"}, {"annotation_id": 517371, "generated_refexp": "person"}, {"annotation_id": 492105, "generated_refexp": "person"}, {"annotation_id": 1170905, "generated_refexp": "backpack"}, {"annotation_id": 572264, "generated_refexp": "bench"}, {"annotation_id": 25658, "generated_refexp": "potted plant"}, {"annotation_id": 424779, "generated_refexp": "person"}, {"annotation_id": 1141815, "generated_refexp": "book"}, {"annotation_id": 461511, "generated_refexp": "person"}, {"annotation_id": 174649, "generated_refexp": "train"}, {"annotation_id": 1559925, "generated_refexp": "broccoli"}, {"annotation_id": 157307, "generated_refexp": "airplane"}, {"annotation_id": 225375, "generated_refexp": "person"}, {"annotation_id": 1559814, "generated_refexp": "broccoli"}, {"annotation_id": 234337, "generated_refexp": "person"}, {"annotation_id": 1819451, "generated_refexp": "cow"}, {"annotation_id": 1195963, "generated_refexp": "bus"}, {"annotation_id": 157102, "generated_refexp": "airplane"}, {"annotation_id": 194908, "generated_refexp": "person"}, {"annotation_id": 1184976, "generated_refexp": "suitcase"}, {"annotation_id": 1045176, "generated_refexp": "banana"}, {"annotation_id": 55947, "generated_refexp": "horse"}, {"annotation_id": 642690, "generated_refexp": "skateboard"}, {"annotation_id": 568889, "generated_refexp": "person"}, {"annotation_id": 513261, "generated_refexp": "person"}, {"annotation_id": 1823880, "generated_refexp": "backpack"}, {"annotation_id": 98501, "generated_refexp": "couch"}, {"annotation_id": 514928, "generated_refexp": "person"}, {"annotation_id": 398456, "generated_refexp": "truck"}, {"annotation_id": 2159291, "generated_refexp": "person"}, {"annotation_id": 513121, "generated_refexp": "person"}, {"annotation_id": 1156179, "generated_refexp": "vase"}, {"annotation_id": 1049773, "generated_refexp": "apple"}, {"annotation_id": 1883835, "generated_refexp": "cup"}, {"annotation_id": 2214685, "generated_refexp": "cup"}, {"annotation_id": 2204366, "generated_refexp": "person"}, {"annotation_id": 581347, "generated_refexp": "elephant"}, {"annotation_id": 395780, "generated_refexp": "truck"}, {"annotation_id": 375463, "generated_refexp": "chair"}, {"annotation_id": 14246, "generated_refexp": "dog"}, {"annotation_id": 279389, "generated_refexp": "umbrella"}, {"annotation_id": 1167087, "generated_refexp": "backpack"}, {"annotation_id": 1154190, "generated_refexp": "vase"}, {"annotation_id": 2224327, "generated_refexp": "couch"}, {"annotation_id": 667049, "generated_refexp": "wine glass"}, {"annotation_id": 1103622, "generated_refexp": "laptop"}, {"annotation_id": 597940, "generated_refexp": "giraffe"}, {"annotation_id": 1058074, "generated_refexp": "broccoli"}, {"annotation_id": 1621281, "generated_refexp": "dining table"}, {"annotation_id": 597093, "generated_refexp": "giraffe"}, {"annotation_id": 2222721, "generated_refexp": "chair"}, {"annotation_id": 1153960, "generated_refexp": "vase"}, {"annotation_id": 289428, "generated_refexp": "chair"}, {"annotation_id": 1038858, "generated_refexp": "bowl"}, {"annotation_id": 1151831, "generated_refexp": "vase"}, {"annotation_id": 115254, "generated_refexp": "couch"}, {"annotation_id": 1099755, "generated_refexp": "laptop"}, {"annotation_id": 1076331, "generated_refexp": "pizza"}, {"annotation_id": 283947, "generated_refexp": "umbrella"}, {"annotation_id": 421708, "generated_refexp": "person"}, {"annotation_id": 138568, "generated_refexp": "car"}, {"annotation_id": 218419, "generated_refexp": "person"}, {"annotation_id": 1242763, "generated_refexp": "person"}, {"annotation_id": 595818, "generated_refexp": "giraffe"}, {"annotation_id": 1364013, "generated_refexp": "airplane"}, {"annotation_id": 600795, "generated_refexp": "giraffe"}, {"annotation_id": 586861, "generated_refexp": "bear"}, {"annotation_id": 202432, "generated_refexp": "person"}, {"annotation_id": 1414653, "generated_refexp": "giraffe"}, {"annotation_id": 64564, "generated_refexp": "sheep"}, {"annotation_id": 375197, "generated_refexp": "chair"}, {"annotation_id": 471696, "generated_refexp": "person"}, {"annotation_id": 1412034, "generated_refexp": "zebra"}, {"annotation_id": 93830, "generated_refexp": "bottle"}, {"annotation_id": 377525, "generated_refexp": "chair"}, {"annotation_id": 100786, "generated_refexp": "chair"}, {"annotation_id": 1979942, "generated_refexp": "oven"}, {"annotation_id": 1652763, "generated_refexp": "book"}, {"annotation_id": 499008, "generated_refexp": "person"}, {"annotation_id": 1966672, "generated_refexp": "dining table"}, {"annotation_id": 471701, "generated_refexp": "person"}, {"annotation_id": 168169, "generated_refexp": "bus"}, {"annotation_id": 1059094, "generated_refexp": "broccoli"}, {"annotation_id": 2155943, "generated_refexp": "person"}, {"annotation_id": 210557, "generated_refexp": "person"}, {"annotation_id": 1151588, "generated_refexp": "vase"}, {"annotation_id": 148973, "generated_refexp": "motorcycle"}, {"annotation_id": 1220146, "generated_refexp": "person"}, {"annotation_id": 434129, "generated_refexp": "person"}, {"annotation_id": 1673119, "generated_refexp": "teddy bear"}, {"annotation_id": 598123, "generated_refexp": "giraffe"}, {"annotation_id": 471802, "generated_refexp": "person"}, {"annotation_id": 445170, "generated_refexp": "person"}, {"annotation_id": 428131, "generated_refexp": "person"}, {"annotation_id": 278385, "generated_refexp": "cow"}, {"annotation_id": 2193909, "generated_refexp": "bed"}, {"annotation_id": 60477, "generated_refexp": "horse"}, {"annotation_id": 154039, "generated_refexp": "motorcycle"}, {"annotation_id": 715075, "generated_refexp": "bowl"}, {"annotation_id": 381257, "generated_refexp": "chair"}, {"annotation_id": 100776, "generated_refexp": "chair"}, {"annotation_id": 2063199, "generated_refexp": "bench"}, {"annotation_id": 580527, "generated_refexp": "elephant"}, {"annotation_id": 134742, "generated_refexp": "car"}, {"annotation_id": 1100336, "generated_refexp": "laptop"}, {"annotation_id": 583926, "generated_refexp": "elephant"}, {"annotation_id": 1539240, "generated_refexp": "bowl"}, {"annotation_id": 382461, "generated_refexp": "chair"}, {"annotation_id": 2161685, "generated_refexp": "person"}, {"annotation_id": 443167, "generated_refexp": "person"}, {"annotation_id": 1758619, "generated_refexp": "person"}, {"annotation_id": 2193096, "generated_refexp": "chair"}, {"annotation_id": 161035, "generated_refexp": "airplane"}, {"annotation_id": 366096, "generated_refexp": "bus"}, {"annotation_id": 1712967, "generated_refexp": "person"}, {"annotation_id": 1931931, "generated_refexp": "chair"}, {"annotation_id": 1091105, "generated_refexp": "dining table"}, {"annotation_id": 557507, "generated_refexp": "person"}, {"annotation_id": 595532, "generated_refexp": "giraffe"}, {"annotation_id": 294685, "generated_refexp": "tie"}, {"annotation_id": 1745849, "generated_refexp": "person"}, {"annotation_id": 1963988, "generated_refexp": "dining table"}, {"annotation_id": 2162482, "generated_refexp": "person"}, {"annotation_id": 1995832, "generated_refexp": "teddy bear"}, {"annotation_id": 583172, "generated_refexp": "elephant"}, {"annotation_id": 127142, "generated_refexp": "bicycle"}, {"annotation_id": 691090, "generated_refexp": "fork"}, {"annotation_id": 538188, "generated_refexp": "person"}, {"annotation_id": 590068, "generated_refexp": "zebra"}, {"annotation_id": 2164198, "generated_refexp": "person"}, {"annotation_id": 424820, "generated_refexp": "person"}, {"annotation_id": 1067333, "generated_refexp": "hot dog"}, {"annotation_id": 434891, "generated_refexp": "person"}, {"annotation_id": 102395, "generated_refexp": "chair"}, {"annotation_id": 420402, "generated_refexp": "dining table"}, {"annotation_id": 559092, "generated_refexp": "person"}, {"annotation_id": 592124, "generated_refexp": "zebra"}, {"annotation_id": 19801, "generated_refexp": "potted plant"}, {"annotation_id": 508174, "generated_refexp": "person"}, {"annotation_id": 1153891, "generated_refexp": "vase"}, {"annotation_id": 562355, "generated_refexp": "person"}, {"annotation_id": 157035, "generated_refexp": "airplane"}, {"annotation_id": 35715, "generated_refexp": "tv"}, {"annotation_id": 584744, "generated_refexp": "elephant"}, {"annotation_id": 280255, "generated_refexp": "umbrella"}, {"annotation_id": 25573, "generated_refexp": "potted plant"}, {"annotation_id": 477878, "generated_refexp": "person"}, {"annotation_id": 1098301, "generated_refexp": "laptop"}, {"annotation_id": 1366469, "generated_refexp": "train"}, {"annotation_id": 506474, "generated_refexp": "person"}, {"annotation_id": 1042383, "generated_refexp": "banana"}, {"annotation_id": 1080321, "generated_refexp": "donut"}, {"annotation_id": 207519, "generated_refexp": "person"}, {"annotation_id": 1924141, "generated_refexp": "pizza"}, {"annotation_id": 75484, "generated_refexp": "cow"}, {"annotation_id": 1609964, "generated_refexp": "potted plant"}, {"annotation_id": 168342, "generated_refexp": "bus"}, {"annotation_id": 1568284, "generated_refexp": "hot dog"}, {"annotation_id": 328387, "generated_refexp": "cell phone"}, {"annotation_id": 383226, "generated_refexp": "chair"}, {"annotation_id": 453464, "generated_refexp": "person"}, {"annotation_id": 1443898, "generated_refexp": "suitcase"}, {"annotation_id": 595872, "generated_refexp": "giraffe"}, {"annotation_id": 1077491, "generated_refexp": "pizza"}, {"annotation_id": 564481, "generated_refexp": "person"}, {"annotation_id": 1039895, "generated_refexp": "bowl"}, {"annotation_id": 318139, "generated_refexp": "bed"}, {"annotation_id": 1065592, "generated_refexp": "carrot"}, {"annotation_id": 131654, "generated_refexp": "bicycle"}, {"annotation_id": 1413869, "generated_refexp": "giraffe"}, {"annotation_id": 442779, "generated_refexp": "person"}, {"annotation_id": 175140, "generated_refexp": "train"}, {"annotation_id": 1155249, "generated_refexp": "vase"}, {"annotation_id": 1131257, "generated_refexp": "sink"}, {"annotation_id": 1158590, "generated_refexp": "scissors"}, {"annotation_id": 1502208, "generated_refexp": "cup"}, {"annotation_id": 1762841, "generated_refexp": "person"}, {"annotation_id": 512541, "generated_refexp": "person"}, {"annotation_id": 449307, "generated_refexp": "person"}, {"annotation_id": 384572, "generated_refexp": "chair"}, {"annotation_id": 521357, "generated_refexp": "person"}, {"annotation_id": 542757, "generated_refexp": "person"}, {"annotation_id": 2134262, "generated_refexp": "tv"}, {"annotation_id": 207103, "generated_refexp": "person"}, {"annotation_id": 1563984, "generated_refexp": "carrot"}, {"annotation_id": 1228980, "generated_refexp": "person"}, {"annotation_id": 1143529, "generated_refexp": "book"}, {"annotation_id": 1087395, "generated_refexp": "cake"}, {"annotation_id": 316230, "generated_refexp": "bed"}, {"annotation_id": 662914, "generated_refexp": "wine glass"}, {"annotation_id": 285987, "generated_refexp": "umbrella"}, {"annotation_id": 1671537, "generated_refexp": "vase"}, {"annotation_id": 1605150, "generated_refexp": "couch"}, {"annotation_id": 1915101, "generated_refexp": "broccoli"}, {"annotation_id": 461331, "generated_refexp": "person"}, {"annotation_id": 1040751, "generated_refexp": "bowl"}, {"annotation_id": 1709086, "generated_refexp": "person"}, {"annotation_id": 588167, "generated_refexp": "zebra"}, {"annotation_id": 230824, "generated_refexp": "person"}, {"annotation_id": 356359, "generated_refexp": "car"}, {"annotation_id": 2166571, "generated_refexp": "person"}, {"annotation_id": 2231181, "generated_refexp": "hot dog"}, {"annotation_id": 471851, "generated_refexp": "person"}, {"annotation_id": 1709050, "generated_refexp": "person"}, {"annotation_id": 458374, "generated_refexp": "person"}, {"annotation_id": 469674, "generated_refexp": "person"}, {"annotation_id": 1039727, "generated_refexp": "bowl"}, {"annotation_id": 436922, "generated_refexp": "person"}, {"annotation_id": 1798333, "generated_refexp": "truck"}, {"annotation_id": 435886, "generated_refexp": "person"}, {"annotation_id": 2162390, "generated_refexp": "person"}, {"annotation_id": 99007, "generated_refexp": "couch"}, {"annotation_id": 51417, "generated_refexp": "cat"}, {"annotation_id": 678430, "generated_refexp": "cup"}, {"annotation_id": 99623, "generated_refexp": "couch"}, {"annotation_id": 712868, "generated_refexp": "bowl"}, {"annotation_id": 253996, "generated_refexp": "person"}, {"annotation_id": 1080644, "generated_refexp": "donut"}, {"annotation_id": 500239, "generated_refexp": "person"}, {"annotation_id": 153648, "generated_refexp": "motorcycle"}, {"annotation_id": 28835, "generated_refexp": "tv"}, {"annotation_id": 471067, "generated_refexp": "person"}, {"annotation_id": 160922, "generated_refexp": "airplane"}, {"annotation_id": 571871, "generated_refexp": "bench"}, {"annotation_id": 655443, "generated_refexp": "tennis racket"}, {"annotation_id": 464765, "generated_refexp": "person"}, {"annotation_id": 2161347, "generated_refexp": "person"}, {"annotation_id": 365937, "generated_refexp": "bus"}, {"annotation_id": 514092, "generated_refexp": "person"}, {"annotation_id": 672414, "generated_refexp": "cup"}, {"annotation_id": 108195, "generated_refexp": "chair"}, {"annotation_id": 673399, "generated_refexp": "cup"}, {"annotation_id": 582095, "generated_refexp": "elephant"}, {"annotation_id": 598886, "generated_refexp": "giraffe"}, {"annotation_id": 1604016, "generated_refexp": "couch"}, {"annotation_id": 554036, "generated_refexp": "person"}, {"annotation_id": 582314, "generated_refexp": "elephant"}, {"annotation_id": 213175, "generated_refexp": "person"}, {"annotation_id": 197424, "generated_refexp": "person"}, {"annotation_id": 1791564, "generated_refexp": "motorcycle"}, {"annotation_id": 588846, "generated_refexp": "zebra"}, {"annotation_id": 2200469, "generated_refexp": "skis"}, {"annotation_id": 1159243, "generated_refexp": "teddy bear"}, {"annotation_id": 1797216, "generated_refexp": "truck"}, {"annotation_id": 429546, "generated_refexp": "person"}, {"annotation_id": 350602, "generated_refexp": "car"}, {"annotation_id": 467796, "generated_refexp": "person"}, {"annotation_id": 1371461, "generated_refexp": "truck"}, {"annotation_id": 527546, "generated_refexp": "person"}, {"annotation_id": 396551, "generated_refexp": "truck"}, {"annotation_id": 158762, "generated_refexp": "airplane"}, {"annotation_id": 503803, "generated_refexp": "person"}, {"annotation_id": 556847, "generated_refexp": "person"}, {"annotation_id": 584430, "generated_refexp": "elephant"}, {"annotation_id": 449741, "generated_refexp": "person"}, {"annotation_id": 1940830, "generated_refexp": "chair"}, {"annotation_id": 515223, "generated_refexp": "person"}, {"annotation_id": 1939003, "generated_refexp": "chair"}, {"annotation_id": 226841, "generated_refexp": "person"}, {"annotation_id": 366514, "generated_refexp": "bus"}, {"annotation_id": 565174, "generated_refexp": "person"}, {"annotation_id": 2159648, "generated_refexp": "person"}, {"annotation_id": 161141, "generated_refexp": "airplane"}, {"annotation_id": 2093111, "generated_refexp": "bottle"}, {"annotation_id": 448811, "generated_refexp": "person"}, {"annotation_id": 173016, "generated_refexp": "train"}, {"annotation_id": 374598, "generated_refexp": "chair"}, {"annotation_id": 439171, "generated_refexp": "person"}, {"annotation_id": 483453, "generated_refexp": "person"}, {"annotation_id": 585036, "generated_refexp": "elephant"}, {"annotation_id": 505849, "generated_refexp": "person"}, {"annotation_id": 71514, "generated_refexp": "cow"}, {"annotation_id": 391871, "generated_refexp": "dining table"}, {"annotation_id": 212328, "generated_refexp": "person"}, {"annotation_id": 351876, "generated_refexp": "car"}, {"annotation_id": 474534, "generated_refexp": "person"}, {"annotation_id": 17364, "generated_refexp": "dog"}, {"annotation_id": 453168, "generated_refexp": "person"}, {"annotation_id": 1964840, "generated_refexp": "dining table"}, {"annotation_id": 1635093, "generated_refexp": "keyboard"}, {"annotation_id": 413248, "generated_refexp": "dining table"}, {"annotation_id": 428175, "generated_refexp": "person"}, {"annotation_id": 178673, "generated_refexp": "boat"}, {"annotation_id": 1075744, "generated_refexp": "pizza"}, {"annotation_id": 14800, "generated_refexp": "dog"}, {"annotation_id": 1240840, "generated_refexp": "person"}, {"annotation_id": 225453, "generated_refexp": "person"}, {"annotation_id": 308219, "generated_refexp": "sandwich"}, {"annotation_id": 1724551, "generated_refexp": "person"}, {"annotation_id": 710147, "generated_refexp": "bowl"}, {"annotation_id": 108716, "generated_refexp": "chair"}, {"annotation_id": 1708154, "generated_refexp": "person"}, {"annotation_id": 400274, "generated_refexp": "truck"}, {"annotation_id": 1834342, "generated_refexp": "handbag"}, {"annotation_id": 72547, "generated_refexp": "cow"}, {"annotation_id": 226698, "generated_refexp": "person"}, {"annotation_id": 348200, "generated_refexp": "car"}, {"annotation_id": 311035, "generated_refexp": "sandwich"}, {"annotation_id": 484507, "generated_refexp": "person"}, {"annotation_id": 1716048, "generated_refexp": "person"}, {"annotation_id": 2165660, "generated_refexp": "person"}, {"annotation_id": 453067, "generated_refexp": "person"}, {"annotation_id": 485117, "generated_refexp": "person"}, {"annotation_id": 61397, "generated_refexp": "sheep"}, {"annotation_id": 10845, "generated_refexp": "dog"}, {"annotation_id": 660551, "generated_refexp": "tennis racket"}, {"annotation_id": 214544, "generated_refexp": "person"}, {"annotation_id": 80143, "generated_refexp": "bottle"}, {"annotation_id": 342341, "generated_refexp": "toothbrush"}, {"annotation_id": 56155, "generated_refexp": "horse"}, {"annotation_id": 157972, "generated_refexp": "airplane"}, {"annotation_id": 1172671, "generated_refexp": "handbag"}, {"annotation_id": 1177974, "generated_refexp": "handbag"}, {"annotation_id": 473501, "generated_refexp": "person"}, {"annotation_id": 55916, "generated_refexp": "horse"}, {"annotation_id": 446241, "generated_refexp": "person"}, {"annotation_id": 441094, "generated_refexp": "person"}, {"annotation_id": 716247, "generated_refexp": "bowl"}, {"annotation_id": 422622, "generated_refexp": "person"}, {"annotation_id": 365304, "generated_refexp": "motorcycle"}, {"annotation_id": 2223576, "generated_refexp": "couch"}, {"annotation_id": 505413, "generated_refexp": "person"}, {"annotation_id": 1554857, "generated_refexp": "sandwich"}, {"annotation_id": 213872, "generated_refexp": "person"}, {"annotation_id": 1184254, "generated_refexp": "suitcase"}, {"annotation_id": 487048, "generated_refexp": "person"}, {"annotation_id": 190769, "generated_refexp": "person"}, {"annotation_id": 510671, "generated_refexp": "person"}, {"annotation_id": 594407, "generated_refexp": "zebra"}, {"annotation_id": 1817307, "generated_refexp": "dog"}, {"annotation_id": 283147, "generated_refexp": "umbrella"}, {"annotation_id": 423370, "generated_refexp": "person"}, {"annotation_id": 1099942, "generated_refexp": "laptop"}, {"annotation_id": 157010, "generated_refexp": "airplane"}, {"annotation_id": 358886, "generated_refexp": "car"}, {"annotation_id": 310373, "generated_refexp": "sandwich"}, {"annotation_id": 207930, "generated_refexp": "person"}, {"annotation_id": 464827, "generated_refexp": "person"}, {"annotation_id": 1041708, "generated_refexp": "banana"}, {"annotation_id": 57890, "generated_refexp": "horse"}, {"annotation_id": 2157972, "generated_refexp": "person"}, {"annotation_id": 203132, "generated_refexp": "person"}, {"annotation_id": 1397814, "generated_refexp": "bench"}, {"annotation_id": 1137708, "generated_refexp": "book"}, {"annotation_id": 1227378, "generated_refexp": "person"}, {"annotation_id": 462667, "generated_refexp": "person"}, {"annotation_id": 428786, "generated_refexp": "person"}, {"annotation_id": 471021, "generated_refexp": "person"}, {"annotation_id": 126629, "generated_refexp": "bicycle"}, {"annotation_id": 257628, "generated_refexp": "person"}, {"annotation_id": 121092, "generated_refexp": "dining table"}, {"annotation_id": 73466, "generated_refexp": "cow"}, {"annotation_id": 492843, "generated_refexp": "person"}, {"annotation_id": 470571, "generated_refexp": "person"}, {"annotation_id": 232583, "generated_refexp": "person"}, {"annotation_id": 591688, "generated_refexp": "zebra"}, {"annotation_id": 479624, "generated_refexp": "person"}, {"annotation_id": 428026, "generated_refexp": "person"}, {"annotation_id": 1263525, "generated_refexp": "person"}, {"annotation_id": 196849, "generated_refexp": "person"}, {"annotation_id": 206642, "generated_refexp": "person"}, {"annotation_id": 582924, "generated_refexp": "elephant"}, {"annotation_id": 5897, "generated_refexp": "dog"}, {"annotation_id": 1924031, "generated_refexp": "pizza"}, {"annotation_id": 2161826, "generated_refexp": "person"}, {"annotation_id": 465087, "generated_refexp": "person"}, {"annotation_id": 2197038, "generated_refexp": "clock"}, {"annotation_id": 61186, "generated_refexp": "sheep"}, {"annotation_id": 672779, "generated_refexp": "cup"}, {"annotation_id": 271003, "generated_refexp": "stop sign"}, {"annotation_id": 365918, "generated_refexp": "bus"}, {"annotation_id": 118124, "generated_refexp": "dining table"}, {"annotation_id": 431462, "generated_refexp": "person"}, {"annotation_id": 469460, "generated_refexp": "person"}, {"annotation_id": 470211, "generated_refexp": "person"}, {"annotation_id": 442068, "generated_refexp": "person"}, {"annotation_id": 99089, "generated_refexp": "couch"}, {"annotation_id": 1413948, "generated_refexp": "giraffe"}, {"annotation_id": 1184174, "generated_refexp": "suitcase"}, {"annotation_id": 598008, "generated_refexp": "giraffe"}, {"annotation_id": 1718080, "generated_refexp": "person"}, {"annotation_id": 1713881, "generated_refexp": "person"}, {"annotation_id": 212453, "generated_refexp": "person"}, {"annotation_id": 2201354, "generated_refexp": "skis"}, {"annotation_id": 155810, "generated_refexp": "motorcycle"}, {"annotation_id": 1104409, "generated_refexp": "laptop"}, {"annotation_id": 107362, "generated_refexp": "chair"}, {"annotation_id": 54470, "generated_refexp": "horse"}, {"annotation_id": 1101575, "generated_refexp": "laptop"}, {"annotation_id": 225209, "generated_refexp": "person"}, {"annotation_id": 1179610, "generated_refexp": "handbag"}, {"annotation_id": 57681, "generated_refexp": "horse"}, {"annotation_id": 217314, "generated_refexp": "person"}, {"annotation_id": 1570810, "generated_refexp": "pizza"}, {"annotation_id": 592662, "generated_refexp": "zebra"}, {"annotation_id": 608755, "generated_refexp": "skis"}, {"annotation_id": 1393021, "generated_refexp": "bench"}, {"annotation_id": 474305, "generated_refexp": "person"}, {"annotation_id": 1162122, "generated_refexp": "teddy bear"}, {"annotation_id": 1079330, "generated_refexp": "donut"}, {"annotation_id": 591664, "generated_refexp": "zebra"}, {"annotation_id": 151717, "generated_refexp": "motorcycle"}, {"annotation_id": 509810, "generated_refexp": "person"}, {"annotation_id": 448413, "generated_refexp": "person"}, {"annotation_id": 283898, "generated_refexp": "umbrella"}, {"annotation_id": 1553841, "generated_refexp": "sandwich"}, {"annotation_id": 1772320, "generated_refexp": "car"}, {"annotation_id": 1185965, "generated_refexp": "suitcase"}, {"annotation_id": 599153, "generated_refexp": "giraffe"}, {"annotation_id": 470184, "generated_refexp": "person"}, {"annotation_id": 115500, "generated_refexp": "couch"}, {"annotation_id": 2201727, "generated_refexp": "skis"}, {"annotation_id": 645218, "generated_refexp": "skateboard"}, {"annotation_id": 592267, "generated_refexp": "zebra"}, {"annotation_id": 597027, "generated_refexp": "giraffe"}, {"annotation_id": 716482, "generated_refexp": "bowl"}, {"annotation_id": 497322, "generated_refexp": "person"}, {"annotation_id": 432343, "generated_refexp": "person"}, {"annotation_id": 340072, "generated_refexp": "clock"}, {"annotation_id": 1726116, "generated_refexp": "person"}, {"annotation_id": 1163034, "generated_refexp": "teddy bear"}, {"annotation_id": 481879, "generated_refexp": "person"}, {"annotation_id": 152286, "generated_refexp": "motorcycle"}, {"annotation_id": 1414635, "generated_refexp": "giraffe"}, {"annotation_id": 718045, "generated_refexp": "bowl"}, {"annotation_id": 145240, "generated_refexp": "car"}, {"annotation_id": 665015, "generated_refexp": "wine glass"}, {"annotation_id": 596812, "generated_refexp": "giraffe"}, {"annotation_id": 1056839, "generated_refexp": "broccoli"}, {"annotation_id": 164921, "generated_refexp": "bus"}, {"annotation_id": 2158374, "generated_refexp": "person"}, {"annotation_id": 171692, "generated_refexp": "train"}, {"annotation_id": 14318, "generated_refexp": "dog"}, {"annotation_id": 454937, "generated_refexp": "person"}, {"annotation_id": 1077304, "generated_refexp": "pizza"}, {"annotation_id": 2198831, "generated_refexp": "skis"}, {"annotation_id": 1048484, "generated_refexp": "apple"}, {"annotation_id": 587134, "generated_refexp": "bear"}, {"annotation_id": 494521, "generated_refexp": "person"}, {"annotation_id": 1865573, "generated_refexp": "bottle"}, {"annotation_id": 99778, "generated_refexp": "couch"}, {"annotation_id": 580703, "generated_refexp": "elephant"}, {"annotation_id": 488246, "generated_refexp": "person"}, {"annotation_id": 171556, "generated_refexp": "train"}, {"annotation_id": 154845, "generated_refexp": "motorcycle"}, {"annotation_id": 137365, "generated_refexp": "car"}, {"annotation_id": 1048862, "generated_refexp": "apple"}, {"annotation_id": 103563, "generated_refexp": "chair"}, {"annotation_id": 490965, "generated_refexp": "person"}, {"annotation_id": 2151216, "generated_refexp": "person"}, {"annotation_id": 51790, "generated_refexp": "cat"}, {"annotation_id": 128905, "generated_refexp": "bicycle"}, {"annotation_id": 2095915, "generated_refexp": "bottle"}, {"annotation_id": 2207659, "generated_refexp": "person"}, {"annotation_id": 1504009, "generated_refexp": "cup"}, {"annotation_id": 1445610, "generated_refexp": "suitcase"}, {"annotation_id": 2177098, "generated_refexp": "cow"}, {"annotation_id": 353311, "generated_refexp": "car"}, {"annotation_id": 1088200, "generated_refexp": "cake"}, {"annotation_id": 114998, "generated_refexp": "couch"}, {"annotation_id": 442962, "generated_refexp": "person"}, {"annotation_id": 197389, "generated_refexp": "person"}, {"annotation_id": 101461, "generated_refexp": "chair"}, {"annotation_id": 546294, "generated_refexp": "person"}, {"annotation_id": 1790232, "generated_refexp": "motorcycle"}, {"annotation_id": 585129, "generated_refexp": "elephant"}, {"annotation_id": 599570, "generated_refexp": "giraffe"}, {"annotation_id": 1810863, "generated_refexp": "bench"}, {"annotation_id": 60011, "generated_refexp": "horse"}, {"annotation_id": 1076093, "generated_refexp": "pizza"}, {"annotation_id": 1058742, "generated_refexp": "broccoli"}, {"annotation_id": 481560, "generated_refexp": "person"}, {"annotation_id": 525379, "generated_refexp": "person"}, {"annotation_id": 519638, "generated_refexp": "person"}, {"annotation_id": 598990, "generated_refexp": "giraffe"}, {"annotation_id": 1165554, "generated_refexp": "backpack"}, {"annotation_id": 1241606, "generated_refexp": "person"}, {"annotation_id": 179628, "generated_refexp": "boat"}, {"annotation_id": 1097785, "generated_refexp": "laptop"}, {"annotation_id": 500620, "generated_refexp": "person"}, {"annotation_id": 64633, "generated_refexp": "sheep"}, {"annotation_id": 1400618, "generated_refexp": "bird"}, {"annotation_id": 1959095, "generated_refexp": "dining table"}, {"annotation_id": 160643, "generated_refexp": "airplane"}, {"annotation_id": 1817799, "generated_refexp": "sheep"}, {"annotation_id": 1969924, "generated_refexp": "tv"}, {"annotation_id": 468172, "generated_refexp": "person"}, {"annotation_id": 133056, "generated_refexp": "car"}, {"annotation_id": 484375, "generated_refexp": "person"}, {"annotation_id": 522626, "generated_refexp": "person"}, {"annotation_id": 101287, "generated_refexp": "chair"}, {"annotation_id": 2159528, "generated_refexp": "person"}, {"annotation_id": 590684, "generated_refexp": "zebra"}, {"annotation_id": 455140, "generated_refexp": "person"}, {"annotation_id": 98641, "generated_refexp": "couch"}, {"annotation_id": 589190, "generated_refexp": "zebra"}, {"annotation_id": 1089913, "generated_refexp": "cake"}, {"annotation_id": 64713, "generated_refexp": "sheep"}, {"annotation_id": 514218, "generated_refexp": "person"}, {"annotation_id": 163056, "generated_refexp": "bus"}, {"annotation_id": 392289, "generated_refexp": "dining table"}, {"annotation_id": 1089675, "generated_refexp": "cake"}, {"annotation_id": 1101044, "generated_refexp": "laptop"}, {"annotation_id": 430951, "generated_refexp": "person"}, {"annotation_id": 497288, "generated_refexp": "person"}, {"annotation_id": 55463, "generated_refexp": "horse"}, {"annotation_id": 51266, "generated_refexp": "cat"}, {"annotation_id": 144033, "generated_refexp": "car"}, {"annotation_id": 522567, "generated_refexp": "person"}, {"annotation_id": 2187836, "generated_refexp": "broccoli"}, {"annotation_id": 1646050, "generated_refexp": "sink"}, {"annotation_id": 497056, "generated_refexp": "person"}, {"annotation_id": 169660, "generated_refexp": "train"}, {"annotation_id": 51971, "generated_refexp": "cat"}, {"annotation_id": 473900, "generated_refexp": "person"}, {"annotation_id": 105901, "generated_refexp": "chair"}, {"annotation_id": 465228, "generated_refexp": "person"}, {"annotation_id": 1752074, "generated_refexp": "person"}, {"annotation_id": 471504, "generated_refexp": "person"}, {"annotation_id": 1091782, "generated_refexp": "dining table"}, {"annotation_id": 423521, "generated_refexp": "person"}, {"annotation_id": 326645, "generated_refexp": "cell phone"}, {"annotation_id": 49423, "generated_refexp": "cat"}, {"annotation_id": 480373, "generated_refexp": "person"}, {"annotation_id": 40662, "generated_refexp": "bird"}, {"annotation_id": 692593, "generated_refexp": "knife"}, {"annotation_id": 598524, "generated_refexp": "giraffe"}, {"annotation_id": 196515, "generated_refexp": "person"}, {"annotation_id": 505325, "generated_refexp": "person"}, {"annotation_id": 2225698, "generated_refexp": "dining table"}, {"annotation_id": 1115725, "generated_refexp": "keyboard"}, {"annotation_id": 117357, "generated_refexp": "couch"}, {"annotation_id": 597472, "generated_refexp": "giraffe"}, {"annotation_id": 521300, "generated_refexp": "person"}, {"annotation_id": 108940, "generated_refexp": "chair"}, {"annotation_id": 312052, "generated_refexp": "sandwich"}, {"annotation_id": 55462, "generated_refexp": "horse"}, {"annotation_id": 151038, "generated_refexp": "motorcycle"}, {"annotation_id": 1072408, "generated_refexp": "pizza"}, {"annotation_id": 1450412, "generated_refexp": "skis"}, {"annotation_id": 1728957, "generated_refexp": "person"}, {"annotation_id": 483055, "generated_refexp": "person"}, {"annotation_id": 71729, "generated_refexp": "cow"}, {"annotation_id": 520924, "generated_refexp": "person"}, {"annotation_id": 61928, "generated_refexp": "sheep"}, {"annotation_id": 167224, "generated_refexp": "bus"}, {"annotation_id": 581399, "generated_refexp": "elephant"}, {"annotation_id": 1154131, "generated_refexp": "vase"}, {"annotation_id": 153450, "generated_refexp": "motorcycle"}, {"annotation_id": 601267, "generated_refexp": "giraffe"}, {"annotation_id": 589947, "generated_refexp": "zebra"}, {"annotation_id": 488172, "generated_refexp": "person"}, {"annotation_id": 151976, "generated_refexp": "motorcycle"}, {"annotation_id": 181015, "generated_refexp": "boat"}, {"annotation_id": 366044, "generated_refexp": "bus"}, {"annotation_id": 1156021, "generated_refexp": "vase"}, {"annotation_id": 487838, "generated_refexp": "person"}, {"annotation_id": 516016, "generated_refexp": "person"}, {"annotation_id": 670496, "generated_refexp": "cup"}, {"annotation_id": 468240, "generated_refexp": "person"}, {"annotation_id": 574980, "generated_refexp": "bench"}, {"annotation_id": 598108, "generated_refexp": "giraffe"}, {"annotation_id": 1964198, "generated_refexp": "dining table"}, {"annotation_id": 505732, "generated_refexp": "person"}, {"annotation_id": 442342, "generated_refexp": "person"}, {"annotation_id": 1729111, "generated_refexp": "person"}, {"annotation_id": 1745688, "generated_refexp": "person"}, {"annotation_id": 209542, "generated_refexp": "person"}, {"annotation_id": 1796633, "generated_refexp": "truck"}, {"annotation_id": 513009, "generated_refexp": "person"}, {"annotation_id": 1425800, "generated_refexp": "umbrella"}, {"annotation_id": 140121, "generated_refexp": "car"}, {"annotation_id": 478047, "generated_refexp": "person"}, {"annotation_id": 473096, "generated_refexp": "person"}, {"annotation_id": 589814, "generated_refexp": "zebra"}, {"annotation_id": 19042, "generated_refexp": "potted plant"}, {"annotation_id": 461354, "generated_refexp": "person"}, {"annotation_id": 1052304, "generated_refexp": "orange"}, {"annotation_id": 1715093, "generated_refexp": "person"}, {"annotation_id": 470650, "generated_refexp": "person"}] --------------------------------------------------------------------------------