├── .gitignore ├── README.md ├── bleu ├── LICENSE ├── bleu.py └── bleu_scorer.py ├── cider ├── cider.py └── cider_scorer.py ├── eval.py ├── example ├── captions_val2014.json ├── captions_val2014_fakecap_results.json └── coco_eval_example.py ├── license.txt ├── meteor ├── data │ └── paraphrase-en.gz ├── meteor-1.5.jar └── meteor.py ├── rouge └── rouge.py ├── setup.py ├── spice ├── __init__.py ├── get_stanford_models.py ├── lib │ ├── Meteor-1.5.jar │ ├── SceneGraphParser-1.0.jar │ ├── ejml-0.23.jar │ ├── fst-2.47.jar │ ├── guava-19.0.jar │ ├── hamcrest-core-1.3.jar │ ├── jackson-core-2.5.3.jar │ ├── javassist-3.19.0-GA.jar │ ├── json-simple-1.1.1.jar │ ├── junit-4.12.jar │ ├── lmdbjni-0.4.6.jar │ ├── lmdbjni-linux64-0.4.6.jar │ ├── lmdbjni-osx64-0.4.6.jar │ ├── lmdbjni-win64-0.4.6.jar │ ├── objenesis-2.4.jar │ ├── slf4j-api-1.7.12.jar │ └── slf4j-simple-1.7.21.jar ├── spice-1.0.jar └── spice.py └── tokenizer ├── ptbtokenizer.py └── stanford-corenlp-3.4.1.jar /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Microsoft COCO Caption Evaluation 2 | =================== 3 | 4 | Evaluation codes for MS COCO caption generation. 5 | 6 | ## Description ## 7 | This repository provides Python 3 support for the caption evaluation metrics used for the MS COCO dataset. 8 | 9 | The code is derived from the original repository that supports Python 2.7: https://github.com/tylin/coco-caption. 10 | Caption evaluation depends on the COCO API that natively supports Python 3. 11 | 12 | ## Requirements ## 13 | - Java 1.8.0 14 | - Python 3 15 | 16 | ## Installation ## 17 | To install pycocoevalcap and the pycocotools dependency (https://github.com/cocodataset/cocoapi), run: 18 | ``` 19 | pip install pycocoevalcap 20 | ``` 21 | 22 | ## Usage ## 23 | See the example script: [example/coco_eval_example.py](example/coco_eval_example.py) 24 | 25 | ## Files ## 26 | ./ 27 | - eval.py: The file includes COCOEavlCap class that can be used to evaluate results on COCO. 28 | - tokenizer: Python wrapper of Stanford CoreNLP PTBTokenizer 29 | - bleu: Bleu evalutation codes 30 | - meteor: Meteor evaluation codes 31 | - rouge: Rouge-L evaluation codes 32 | - cider: CIDEr evaluation codes 33 | - spice: SPICE evaluation codes 34 | 35 | ## Setup ## 36 | 37 | - SPICE requires the download of [Stanford CoreNLP 3.6.0](http://stanfordnlp.github.io/CoreNLP/index.html) code and models. This will be done automatically the first time the SPICE evaluation is performed. 38 | - Note: SPICE will try to create a cache of parsed sentences in ./spice/cache/. This dramatically speeds up repeated evaluations. The cache directory can be moved by setting 'CACHE_DIR' in ./spice. In the same file, caching can be turned off by removing the '-cache' argument to 'spice_cmd'. 39 | 40 | ## References ## 41 | 42 | - [Microsoft COCO Captions: Data Collection and Evaluation Server](http://arxiv.org/abs/1504.00325) 43 | - PTBTokenizer: We use the [Stanford Tokenizer](http://nlp.stanford.edu/software/tokenizer.shtml) which is included in [Stanford CoreNLP 3.4.1](http://nlp.stanford.edu/software/corenlp.shtml). 44 | - BLEU: [BLEU: a Method for Automatic Evaluation of Machine Translation](http://www.aclweb.org/anthology/P02-1040.pdf) 45 | - Meteor: [Project page](http://www.cs.cmu.edu/~alavie/METEOR/) with related publications. We use the latest version (1.5) of the [Code](https://github.com/mjdenkowski/meteor). Changes have been made to the source code to properly aggreate the statistics for the entire corpus. 46 | - Rouge-L: [ROUGE: A Package for Automatic Evaluation of Summaries](http://anthology.aclweb.org/W/W04/W04-1013.pdf) 47 | - CIDEr: [CIDEr: Consensus-based Image Description Evaluation](http://arxiv.org/pdf/1411.5726.pdf) 48 | - SPICE: [SPICE: Semantic Propositional Image Caption Evaluation](https://arxiv.org/abs/1607.08822) 49 | 50 | ## Developers ## 51 | - Xinlei Chen (CMU) 52 | - Hao Fang (University of Washington) 53 | - Tsung-Yi Lin (Cornell) 54 | - Ramakrishna Vedantam (Virgina Tech) 55 | 56 | ## Acknowledgement ## 57 | - David Chiang (University of Norte Dame) 58 | - Michael Denkowski (CMU) 59 | - Alexander Rush (Harvard University) 60 | -------------------------------------------------------------------------------- /bleu/LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015 Xinlei Chen, Hao Fang, Tsung-Yi Lin, and Ramakrishna Vedantam 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy 4 | of this software and associated documentation files (the "Software"), to deal 5 | in the Software without restriction, including without limitation the rights 6 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 7 | copies of the Software, and to permit persons to whom the Software is 8 | furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in 11 | all copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 19 | THE SOFTWARE. 20 | -------------------------------------------------------------------------------- /bleu/bleu.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # 3 | # File Name : bleu.py 4 | # 5 | # Description : Wrapper for BLEU scorer. 6 | # 7 | # Creation Date : 06-01-2015 8 | # Last Modified : Thu 19 Mar 2015 09:13:28 PM PDT 9 | # Authors : Hao Fang and Tsung-Yi Lin 10 | 11 | from .bleu_scorer import BleuScorer 12 | 13 | 14 | class Bleu: 15 | def __init__(self, n=4): 16 | # default compute Blue score up to 4 17 | self._n = n 18 | self._hypo_for_image = {} 19 | self.ref_for_image = {} 20 | 21 | def compute_score(self, gts, res, verbose=1): 22 | 23 | assert(gts.keys() == res.keys()) 24 | imgIds = gts.keys() 25 | 26 | bleu_scorer = BleuScorer(n=self._n) 27 | for id in imgIds: 28 | hypo = res[id] 29 | ref = gts[id] 30 | 31 | # Sanity check. 32 | assert(type(hypo) is list) 33 | assert(len(hypo) == 1) 34 | assert(type(ref) is list) 35 | assert(len(ref) >= 1) 36 | 37 | bleu_scorer += (hypo[0], ref) 38 | 39 | #score, scores = bleu_scorer.compute_score(option='shortest') 40 | score, scores = bleu_scorer.compute_score(option='closest', verbose=verbose) 41 | #score, scores = bleu_scorer.compute_score(option='average', verbose=1) 42 | 43 | # return (bleu, bleu_info) 44 | return score, scores 45 | 46 | def method(self): 47 | return "Bleu" 48 | -------------------------------------------------------------------------------- /bleu/bleu_scorer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # bleu_scorer.py 4 | # David Chiang 5 | 6 | # Copyright (c) 2004-2006 University of Maryland. All rights 7 | # reserved. Do not redistribute without permission from the 8 | # author. Not for commercial use. 9 | 10 | # Modified by: 11 | # Hao Fang 12 | # Tsung-Yi Lin 13 | 14 | '''Provides: 15 | cook_refs(refs, n=4): Transform a list of reference sentences as strings into a form usable by cook_test(). 16 | cook_test(test, refs, n=4): Transform a test sentence as a string (together with the cooked reference sentences) into a form usable by score_cooked(). 17 | ''' 18 | 19 | import copy 20 | import sys, math, re 21 | from collections import defaultdict 22 | 23 | def precook(s, n=4, out=False): 24 | """Takes a string as input and returns an object that can be given to 25 | either cook_refs or cook_test. This is optional: cook_refs and cook_test 26 | can take string arguments as well.""" 27 | words = s.split() 28 | counts = defaultdict(int) 29 | for k in range(1,n+1): 30 | for i in range(len(words)-k+1): 31 | ngram = tuple(words[i:i+k]) 32 | counts[ngram] += 1 33 | return (len(words), counts) 34 | 35 | def cook_refs(refs, eff=None, n=4): ## lhuang: oracle will call with "average" 36 | '''Takes a list of reference sentences for a single segment 37 | and returns an object that encapsulates everything that BLEU 38 | needs to know about them.''' 39 | 40 | reflen = [] 41 | maxcounts = {} 42 | for ref in refs: 43 | rl, counts = precook(ref, n) 44 | reflen.append(rl) 45 | for (ngram,count) in counts.items(): 46 | maxcounts[ngram] = max(maxcounts.get(ngram,0), count) 47 | 48 | # Calculate effective reference sentence length. 49 | if eff == "shortest": 50 | reflen = min(reflen) 51 | elif eff == "average": 52 | reflen = float(sum(reflen))/len(reflen) 53 | 54 | ## lhuang: N.B.: leave reflen computaiton to the very end!! 55 | 56 | ## lhuang: N.B.: in case of "closest", keep a list of reflens!! (bad design) 57 | 58 | return (reflen, maxcounts) 59 | 60 | def cook_test(test, refs, eff=None, n=4): 61 | '''Takes a test sentence and returns an object that 62 | encapsulates everything that BLEU needs to know about it.''' 63 | 64 | reflen, refmaxcounts = refs 65 | testlen, counts = precook(test, n, True) 66 | 67 | result = {} 68 | 69 | # Calculate effective reference sentence length. 70 | 71 | if eff == "closest": 72 | result["reflen"] = min((abs(l-testlen), l) for l in reflen)[1] 73 | else: ## i.e., "average" or "shortest" or None 74 | result["reflen"] = reflen 75 | 76 | result["testlen"] = testlen 77 | 78 | result["guess"] = [max(0,testlen-k+1) for k in range(1,n+1)] 79 | 80 | result['correct'] = [0]*n 81 | for (ngram, count) in counts.items(): 82 | result["correct"][len(ngram)-1] += min(refmaxcounts.get(ngram,0), count) 83 | 84 | return result 85 | 86 | class BleuScorer(object): 87 | """Bleu scorer. 88 | """ 89 | 90 | __slots__ = "n", "crefs", "ctest", "_score", "_ratio", "_testlen", "_reflen", "special_reflen" 91 | # special_reflen is used in oracle (proportional effective ref len for a node). 92 | 93 | def copy(self): 94 | ''' copy the refs.''' 95 | new = BleuScorer(n=self.n) 96 | new.ctest = copy.copy(self.ctest) 97 | new.crefs = copy.copy(self.crefs) 98 | new._score = None 99 | return new 100 | 101 | def __init__(self, test=None, refs=None, n=4, special_reflen=None): 102 | ''' singular instance ''' 103 | 104 | self.n = n 105 | self.crefs = [] 106 | self.ctest = [] 107 | self.cook_append(test, refs) 108 | self.special_reflen = special_reflen 109 | 110 | def cook_append(self, test, refs): 111 | '''called by constructor and __iadd__ to avoid creating new instances.''' 112 | 113 | if refs is not None: 114 | self.crefs.append(cook_refs(refs)) 115 | if test is not None: 116 | cooked_test = cook_test(test, self.crefs[-1]) 117 | self.ctest.append(cooked_test) ## N.B.: -1 118 | else: 119 | self.ctest.append(None) # lens of crefs and ctest have to match 120 | 121 | self._score = None ## need to recompute 122 | 123 | def ratio(self, option=None): 124 | self.compute_score(option=option) 125 | return self._ratio 126 | 127 | def score_ratio(self, option=None): 128 | '''return (bleu, len_ratio) pair''' 129 | return (self.fscore(option=option), self.ratio(option=option)) 130 | 131 | def score_ratio_str(self, option=None): 132 | return "%.4f (%.2f)" % self.score_ratio(option) 133 | 134 | def reflen(self, option=None): 135 | self.compute_score(option=option) 136 | return self._reflen 137 | 138 | def testlen(self, option=None): 139 | self.compute_score(option=option) 140 | return self._testlen 141 | 142 | def retest(self, new_test): 143 | if type(new_test) is str: 144 | new_test = [new_test] 145 | assert len(new_test) == len(self.crefs), new_test 146 | self.ctest = [] 147 | for t, rs in zip(new_test, self.crefs): 148 | self.ctest.append(cook_test(t, rs)) 149 | self._score = None 150 | 151 | return self 152 | 153 | def rescore(self, new_test): 154 | ''' replace test(s) with new test(s), and returns the new score.''' 155 | 156 | return self.retest(new_test).compute_score() 157 | 158 | def size(self): 159 | assert len(self.crefs) == len(self.ctest), "refs/test mismatch! %d<>%d" % (len(self.crefs), len(self.ctest)) 160 | return len(self.crefs) 161 | 162 | def __iadd__(self, other): 163 | '''add an instance (e.g., from another sentence).''' 164 | 165 | if type(other) is tuple: 166 | ## avoid creating new BleuScorer instances 167 | self.cook_append(other[0], other[1]) 168 | else: 169 | assert self.compatible(other), "incompatible BLEUs." 170 | self.ctest.extend(other.ctest) 171 | self.crefs.extend(other.crefs) 172 | self._score = None ## need to recompute 173 | 174 | return self 175 | 176 | def compatible(self, other): 177 | return isinstance(other, BleuScorer) and self.n == other.n 178 | 179 | def single_reflen(self, option="average"): 180 | return self._single_reflen(self.crefs[0][0], option) 181 | 182 | def _single_reflen(self, reflens, option=None, testlen=None): 183 | 184 | if option == "shortest": 185 | reflen = min(reflens) 186 | elif option == "average": 187 | reflen = float(sum(reflens))/len(reflens) 188 | elif option == "closest": 189 | reflen = min((abs(l-testlen), l) for l in reflens)[1] 190 | else: 191 | assert False, "unsupported reflen option %s" % option 192 | 193 | return reflen 194 | 195 | def recompute_score(self, option=None, verbose=0): 196 | self._score = None 197 | return self.compute_score(option, verbose) 198 | 199 | def compute_score(self, option=None, verbose=0): 200 | n = self.n 201 | small = 1e-9 202 | tiny = 1e-15 ## so that if guess is 0 still return 0 203 | bleu_list = [[] for _ in range(n)] 204 | 205 | if self._score is not None: 206 | return self._score 207 | 208 | if option is None: 209 | option = "average" if len(self.crefs) == 1 else "closest" 210 | 211 | self._testlen = 0 212 | self._reflen = 0 213 | totalcomps = {'testlen':0, 'reflen':0, 'guess':[0]*n, 'correct':[0]*n} 214 | 215 | # for each sentence 216 | for comps in self.ctest: 217 | testlen = comps['testlen'] 218 | self._testlen += testlen 219 | 220 | if self.special_reflen is None: ## need computation 221 | reflen = self._single_reflen(comps['reflen'], option, testlen) 222 | else: 223 | reflen = self.special_reflen 224 | 225 | self._reflen += reflen 226 | 227 | for key in ['guess','correct']: 228 | for k in range(n): 229 | totalcomps[key][k] += comps[key][k] 230 | 231 | # append per image bleu score 232 | bleu = 1. 233 | for k in range(n): 234 | bleu *= (float(comps['correct'][k]) + tiny) \ 235 | /(float(comps['guess'][k]) + small) 236 | bleu_list[k].append(bleu ** (1./(k+1))) 237 | ratio = (testlen + tiny) / (reflen + small) ## N.B.: avoid zero division 238 | if ratio < 1: 239 | for k in range(n): 240 | bleu_list[k][-1] *= math.exp(1 - 1/ratio) 241 | 242 | if verbose > 1: 243 | print(comps, reflen) 244 | 245 | totalcomps['reflen'] = self._reflen 246 | totalcomps['testlen'] = self._testlen 247 | 248 | bleus = [] 249 | bleu = 1. 250 | for k in range(n): 251 | bleu *= float(totalcomps['correct'][k] + tiny) \ 252 | / (totalcomps['guess'][k] + small) 253 | bleus.append(bleu ** (1./(k+1))) 254 | ratio = (self._testlen + tiny) / (self._reflen + small) ## N.B.: avoid zero division 255 | if ratio < 1: 256 | for k in range(n): 257 | bleus[k] *= math.exp(1 - 1/ratio) 258 | 259 | if verbose > 0: 260 | print(totalcomps) 261 | print("ratio:", ratio) 262 | 263 | self._score = bleus 264 | return self._score, bleu_list 265 | -------------------------------------------------------------------------------- /cider/cider.py: -------------------------------------------------------------------------------- 1 | # Filename: cider.py 2 | # 3 | # Description: Describes the class to compute the CIDEr (Consensus-Based Image Description Evaluation) Metric 4 | # by Vedantam, Zitnick, and Parikh (http://arxiv.org/abs/1411.5726) 5 | # 6 | # Creation Date: Sun Feb 8 14:16:54 2015 7 | # 8 | # Authors: Ramakrishna Vedantam and Tsung-Yi Lin 9 | 10 | from .cider_scorer import CiderScorer 11 | import pdb 12 | 13 | class Cider: 14 | """ 15 | Main Class to compute the CIDEr metric 16 | 17 | """ 18 | def __init__(self, test=None, refs=None, n=4, sigma=6.0): 19 | # set cider to sum over 1 to 4-grams 20 | self._n = n 21 | # set the standard deviation parameter for gaussian penalty 22 | self._sigma = sigma 23 | 24 | def compute_score(self, gts, res): 25 | """ 26 | Main function to compute CIDEr score 27 | :param hypo_for_image (dict) : dictionary with key and value 28 | ref_for_image (dict) : dictionary with key and value 29 | :return: cider (float) : computed CIDEr score for the corpus 30 | """ 31 | 32 | assert(gts.keys() == res.keys()) 33 | imgIds = gts.keys() 34 | 35 | cider_scorer = CiderScorer(n=self._n, sigma=self._sigma) 36 | 37 | for id in imgIds: 38 | hypo = res[id] 39 | ref = gts[id] 40 | 41 | # Sanity check. 42 | assert(type(hypo) is list) 43 | assert(len(hypo) == 1) 44 | assert(type(ref) is list) 45 | assert(len(ref) > 0) 46 | 47 | cider_scorer += (hypo[0], ref) 48 | 49 | (score, scores) = cider_scorer.compute_score() 50 | 51 | return score, scores 52 | 53 | def method(self): 54 | return "CIDEr" 55 | -------------------------------------------------------------------------------- /cider/cider_scorer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Tsung-Yi Lin 3 | # Ramakrishna Vedantam 4 | 5 | import copy 6 | from collections import defaultdict 7 | import numpy as np 8 | import pdb 9 | import math 10 | 11 | def precook(s, n=4, out=False): 12 | """ 13 | Takes a string as input and returns an object that can be given to 14 | either cook_refs or cook_test. This is optional: cook_refs and cook_test 15 | can take string arguments as well. 16 | :param s: string : sentence to be converted into ngrams 17 | :param n: int : number of ngrams for which representation is calculated 18 | :return: term frequency vector for occuring ngrams 19 | """ 20 | words = s.split() 21 | counts = defaultdict(int) 22 | for k in range(1,n+1): 23 | for i in range(len(words)-k+1): 24 | ngram = tuple(words[i:i+k]) 25 | counts[ngram] += 1 26 | return counts 27 | 28 | def cook_refs(refs, n=4): ## lhuang: oracle will call with "average" 29 | '''Takes a list of reference sentences for a single segment 30 | and returns an object that encapsulates everything that BLEU 31 | needs to know about them. 32 | :param refs: list of string : reference sentences for some image 33 | :param n: int : number of ngrams for which (ngram) representation is calculated 34 | :return: result (list of dict) 35 | ''' 36 | return [precook(ref, n) for ref in refs] 37 | 38 | def cook_test(test, n=4): 39 | '''Takes a test sentence and returns an object that 40 | encapsulates everything that BLEU needs to know about it. 41 | :param test: list of string : hypothesis sentence for some image 42 | :param n: int : number of ngrams for which (ngram) representation is calculated 43 | :return: result (dict) 44 | ''' 45 | return precook(test, n, True) 46 | 47 | class CiderScorer(object): 48 | """CIDEr scorer. 49 | """ 50 | 51 | def copy(self): 52 | ''' copy the refs.''' 53 | new = CiderScorer(n=self.n) 54 | new.ctest = copy.copy(self.ctest) 55 | new.crefs = copy.copy(self.crefs) 56 | return new 57 | 58 | def __init__(self, test=None, refs=None, n=4, sigma=6.0): 59 | ''' singular instance ''' 60 | self.n = n 61 | self.sigma = sigma 62 | self.crefs = [] 63 | self.ctest = [] 64 | self.document_frequency = defaultdict(float) 65 | self.cook_append(test, refs) 66 | self.ref_len = None 67 | 68 | def cook_append(self, test, refs): 69 | '''called by constructor and __iadd__ to avoid creating new instances.''' 70 | 71 | if refs is not None: 72 | self.crefs.append(cook_refs(refs)) 73 | if test is not None: 74 | self.ctest.append(cook_test(test)) ## N.B.: -1 75 | else: 76 | self.ctest.append(None) # lens of crefs and ctest have to match 77 | 78 | def size(self): 79 | assert len(self.crefs) == len(self.ctest), "refs/test mismatch! %d<>%d" % (len(self.crefs), len(self.ctest)) 80 | return len(self.crefs) 81 | 82 | def __iadd__(self, other): 83 | '''add an instance (e.g., from another sentence).''' 84 | 85 | if type(other) is tuple: 86 | ## avoid creating new CiderScorer instances 87 | self.cook_append(other[0], other[1]) 88 | else: 89 | self.ctest.extend(other.ctest) 90 | self.crefs.extend(other.crefs) 91 | 92 | return self 93 | def compute_doc_freq(self): 94 | ''' 95 | Compute term frequency for reference data. 96 | This will be used to compute idf (inverse document frequency later) 97 | The term frequency is stored in the object 98 | :return: None 99 | ''' 100 | for refs in self.crefs: 101 | # refs, k ref captions of one image 102 | for ngram in set([ngram for ref in refs for (ngram,count) in ref.items()]): 103 | self.document_frequency[ngram] += 1 104 | # maxcounts[ngram] = max(maxcounts.get(ngram,0), count) 105 | 106 | def compute_cider(self): 107 | def counts2vec(cnts): 108 | """ 109 | Function maps counts of ngram to vector of tfidf weights. 110 | The function returns vec, an array of dictionary that store mapping of n-gram and tf-idf weights. 111 | The n-th entry of array denotes length of n-grams. 112 | :param cnts: 113 | :return: vec (array of dict), norm (array of float), length (int) 114 | """ 115 | vec = [defaultdict(float) for _ in range(self.n)] 116 | length = 0 117 | norm = [0.0 for _ in range(self.n)] 118 | for (ngram,term_freq) in cnts.items(): 119 | # give word count 1 if it doesn't appear in reference corpus 120 | df = np.log(max(1.0, self.document_frequency[ngram])) 121 | # ngram index 122 | n = len(ngram)-1 123 | # tf (term_freq) * idf (precomputed idf) for n-grams 124 | vec[n][ngram] = float(term_freq)*(self.ref_len - df) 125 | # compute norm for the vector. the norm will be used for computing similarity 126 | norm[n] += pow(vec[n][ngram], 2) 127 | 128 | if n == 1: 129 | length += term_freq 130 | norm = [np.sqrt(n) for n in norm] 131 | return vec, norm, length 132 | 133 | def sim(vec_hyp, vec_ref, norm_hyp, norm_ref, length_hyp, length_ref): 134 | ''' 135 | Compute the cosine similarity of two vectors. 136 | :param vec_hyp: array of dictionary for vector corresponding to hypothesis 137 | :param vec_ref: array of dictionary for vector corresponding to reference 138 | :param norm_hyp: array of float for vector corresponding to hypothesis 139 | :param norm_ref: array of float for vector corresponding to reference 140 | :param length_hyp: int containing length of hypothesis 141 | :param length_ref: int containing length of reference 142 | :return: array of score for each n-grams cosine similarity 143 | ''' 144 | delta = float(length_hyp - length_ref) 145 | # measure consine similarity 146 | val = np.array([0.0 for _ in range(self.n)]) 147 | for n in range(self.n): 148 | # ngram 149 | for (ngram,count) in vec_hyp[n].items(): 150 | # vrama91 : added clipping 151 | val[n] += min(vec_hyp[n][ngram], vec_ref[n][ngram]) * vec_ref[n][ngram] 152 | 153 | if (norm_hyp[n] != 0) and (norm_ref[n] != 0): 154 | val[n] /= (norm_hyp[n]*norm_ref[n]) 155 | 156 | assert(not math.isnan(val[n])) 157 | # vrama91: added a length based gaussian penalty 158 | val[n] *= np.e**(-(delta**2)/(2*self.sigma**2)) 159 | return val 160 | 161 | # compute log reference length 162 | self.ref_len = np.log(float(len(self.crefs))) 163 | 164 | scores = [] 165 | for test, refs in zip(self.ctest, self.crefs): 166 | # compute vector for test captions 167 | vec, norm, length = counts2vec(test) 168 | # compute vector for ref captions 169 | score = np.array([0.0 for _ in range(self.n)]) 170 | for ref in refs: 171 | vec_ref, norm_ref, length_ref = counts2vec(ref) 172 | score += sim(vec, vec_ref, norm, norm_ref, length, length_ref) 173 | # change by vrama91 - mean of ngram scores, instead of sum 174 | score_avg = np.mean(score) 175 | # divide by number of references 176 | score_avg /= len(refs) 177 | # multiply score by 10 178 | score_avg *= 10.0 179 | # append score of an image to the score list 180 | scores.append(score_avg) 181 | return scores 182 | 183 | def compute_score(self, option=None, verbose=0): 184 | # compute idf 185 | self.compute_doc_freq() 186 | # assert to check document frequency 187 | assert(len(self.ctest) >= max(self.document_frequency.values())) 188 | # compute cider score 189 | score = self.compute_cider() 190 | # debug 191 | # print score 192 | return np.mean(np.array(score)), np.array(score) 193 | -------------------------------------------------------------------------------- /eval.py: -------------------------------------------------------------------------------- 1 | __author__ = 'tylin' 2 | from .tokenizer.ptbtokenizer import PTBTokenizer 3 | from .bleu.bleu import Bleu 4 | from .meteor.meteor import Meteor 5 | from .rouge.rouge import Rouge 6 | from .cider.cider import Cider 7 | from .spice.spice import Spice 8 | 9 | 10 | class COCOEvalCap: 11 | def __init__(self, coco, cocoRes): 12 | self.evalImgs = [] 13 | self.eval = {} 14 | self.imgToEval = {} 15 | self.coco = coco 16 | self.cocoRes = cocoRes 17 | self.params = {'image_id': coco.getImgIds()} 18 | 19 | def evaluate(self): 20 | imgIds = self.params['image_id'] 21 | # imgIds = self.coco.getImgIds() 22 | gts = {} 23 | res = {} 24 | for imgId in imgIds: 25 | gts[imgId] = self.coco.imgToAnns[imgId] 26 | res[imgId] = self.cocoRes.imgToAnns[imgId] 27 | 28 | # ================================================= 29 | # Set up scorers 30 | # ================================================= 31 | print('tokenization...') 32 | tokenizer = PTBTokenizer() 33 | gts = tokenizer.tokenize(gts) 34 | res = tokenizer.tokenize(res) 35 | 36 | # ================================================= 37 | # Set up scorers 38 | # ================================================= 39 | print('setting up scorers...') 40 | scorers = [ 41 | (Bleu(4), ["Bleu_1", "Bleu_2", "Bleu_3", "Bleu_4"]), 42 | (Meteor(),"METEOR"), 43 | (Rouge(), "ROUGE_L"), 44 | (Cider(), "CIDEr"), 45 | (Spice(), "SPICE") 46 | ] 47 | 48 | # ================================================= 49 | # Compute scores 50 | # ================================================= 51 | for scorer, method in scorers: 52 | print('computing %s score...'%(scorer.method())) 53 | score, scores = scorer.compute_score(gts, res) 54 | if type(method) == list: 55 | for sc, scs, m in zip(score, scores, method): 56 | self.setEval(sc, m) 57 | self.setImgToEvalImgs(scs, gts.keys(), m) 58 | print("%s: %0.3f"%(m, sc)) 59 | else: 60 | self.setEval(score, method) 61 | self.setImgToEvalImgs(scores, gts.keys(), method) 62 | print("%s: %0.3f"%(method, score)) 63 | self.setEvalImgs() 64 | 65 | def setEval(self, score, method): 66 | self.eval[method] = score 67 | 68 | def setImgToEvalImgs(self, scores, imgIds, method): 69 | for imgId, score in zip(imgIds, scores): 70 | if not imgId in self.imgToEval: 71 | self.imgToEval[imgId] = {} 72 | self.imgToEval[imgId]["image_id"] = imgId 73 | self.imgToEval[imgId][method] = score 74 | 75 | def setEvalImgs(self): 76 | self.evalImgs = [eval for imgId, eval in self.imgToEval.items()] 77 | -------------------------------------------------------------------------------- /example/captions_val2014_fakecap_results.json: -------------------------------------------------------------------------------- 1 | [{"image_id": 404464, "caption": "black and white photo of a man standing in front of a building"}, {"image_id": 380932, "caption": "group of people are on the side of a snowy field"}, {"image_id": 565778, "caption": "train traveling down a train station"}, {"image_id": 431573, "caption": "red fire hydrant sitting on a park bench in front of a road"}, {"image_id": 322226, "caption": "black and white cat is sitting on top of a wooden bench"}, {"image_id": 237669, "caption": "baseball player swinging a bat at a game"}, {"image_id": 351053, "caption": "laptop computer sitting on top of a table"}, {"image_id": 344860, "caption": "zebra standing on top of a lush green field"}, {"image_id": 40102, "caption": "group of giraffes standing next to each other in a grassy field"}, {"image_id": 95427, "caption": "close up of a pile of oranges sitting on a table"}, {"image_id": 510755, "caption": "couple of a motorcycle parked in front of a lush green field"}, {"image_id": 399012, "caption": "close up of a donut sitting on top of a plate"}, {"image_id": 62151, "caption": "herd of cows are grazing on a lush green field"}, {"image_id": 314154, "caption": "man standing in a living room holding a wii game controller"}, {"image_id": 308026, "caption": "man riding a skateboard in front of a building"}, {"image_id": 207151, "caption": "pizza that is sitting on top of a plate"}, {"image_id": 227227, "caption": "man standing next to a dog"}, {"image_id": 521400, "caption": "woman on a tennis court holding a tennis racket in front of a tennis ball"}, {"image_id": 281878, "caption": "cat is sitting on top of a bed in front of a living room"}, {"image_id": 258516, "caption": "man is standing in front of a window"}, {"image_id": 491497, "caption": "bathroom that is sitting on top of a bed"}, {"image_id": 104612, "caption": "bowl of food on top of a plate"}, {"image_id": 369826, "caption": "laptop computer sitting on top of a street"}, {"image_id": 497006, "caption": "bunch of bananas in front of an outdoor market"}, {"image_id": 2240, "caption": "close up of a teddy bear sitting on a table"}, {"image_id": 350341, "caption": "close up of a pizza sitting on top of a table"}, {"image_id": 119181, "caption": "white plate that is sitting on top of a table"}, {"image_id": 564563, "caption": "black and white photo of a plane flying through the water"}, {"image_id": 397133, "caption": "group of people sitting at a table with a glass of wine"}, {"image_id": 280819, "caption": "group of people sitting at a table with plates of food"}, {"image_id": 349860, "caption": "man doing a trick on a skateboard in the air on top of a ramp"}, {"image_id": 143671, "caption": "close up of a close up of a plate of food on top of a plate"}, {"image_id": 132001, "caption": "black and white cat sitting on top of a computer keyboard"}, {"image_id": 193271, "caption": "kitchen with white cabinets and stainless steel appliances"}, {"image_id": 122934, "caption": "man riding a horse drawn carriage on the back of a motorcycle"}, {"image_id": 489023, "caption": "group of people playing a game of frisbee in a field"}, {"image_id": 535668, "caption": "baseball player swinging a bat at home plate holding a bat in a baseball game"}, {"image_id": 310177, "caption": "bowl of hot dogs are sitting on a table"}, {"image_id": 375521, "caption": "group of people standing on top of a snow covered slope"}, {"image_id": 223648, "caption": "wooden bench sitting on top of a table"}, {"image_id": 121041, "caption": "man riding a wave on a surfboard in the ocean"}, {"image_id": 129576, "caption": "living room with a flat screen tv"}, {"image_id": 156416, "caption": "man riding a wave on a surfboard in the ocean"}, {"image_id": 496768, "caption": "man flying a kite in a body of water"}, {"image_id": 86320, "caption": "bathroom with a sink and a mirror"}, {"image_id": 377715, "caption": "baseball player holding a baseball bat at a game"}, {"image_id": 191096, "caption": "baseball player swinging a bat at a baseball game"}, {"image_id": 328289, "caption": "group of people standing on top of a lush green field"}, {"image_id": 371250, "caption": "black and white cat sitting on top of a bed"}, {"image_id": 199602, "caption": "young woman holding a kite on the beach"}, {"image_id": 329486, "caption": "group of people standing in the middle of water"}, {"image_id": 398362, "caption": "black and white photo of a living room"}, {"image_id": 25560, "caption": "black and white cat sitting on top of a tv"}, {"image_id": 236412, "caption": "slice of pizza sitting on top of a plate"}, {"image_id": 438862, "caption": "group of young men playing a game of frisbee in a field"}, {"image_id": 450247, "caption": "black and white cat laying on top of a computer desk"}, {"image_id": 278166, "caption": "black and white cat sitting on top of a toilet"}, {"image_id": 480985, "caption": "group of people standing next to each other"}, {"image_id": 312421, "caption": "black and white cat sitting on top of a street"}, {"image_id": 473237, "caption": "man is holding a slice of pizza"}, {"image_id": 166323, "caption": "bathroom with a sink and mirror"}, {"image_id": 144200, "caption": "close up of a plate of food on a table"}, {"image_id": 484982, "caption": "living room with a clock tower in front of a window"}, {"image_id": 376549, "caption": "man riding a snowboard down a snow covered slope"}, {"image_id": 288955, "caption": "black and white cat laying on a table next to a bowl of food"}, {"image_id": 450263, "caption": "group of people sitting on a bench in front of a city street"}, {"image_id": 165547, "caption": "living room with a large window"}, {"image_id": 281533, "caption": "woman sitting on top of a living room"}, {"image_id": 521967, "caption": "stuffed teddy bear sitting on top of a bed"}, {"image_id": 434494, "caption": "train is parked in front of a city street"}, {"image_id": 421010, "caption": "group of people sitting on a bench in front of a park"}, {"image_id": 109005, "caption": "herd of cows are grazing on a lush green field"}, {"image_id": 138079, "caption": "bathroom with a sink and a mirror"}, {"image_id": 127451, "caption": "man riding a snowboard on top of a snow covered slope"}, {"image_id": 490081, "caption": "group of cows that are standing in a dirt field"}, {"image_id": 262873, "caption": "black and white cat sitting on top of a table"}, {"image_id": 309022, "caption": "kitchen counter in front of a table"}, {"image_id": 500663, "caption": "man flying a kite on top of a lush green field"}, {"image_id": 247984, "caption": "black and white cat sitting on top of a computer desk"}, {"image_id": 428454, "caption": "man riding a surf board on the beach flying a kite in the snow"}, {"image_id": 439969, "caption": "black and white fire hydrant sitting on top of a boat"}, {"image_id": 143975, "caption": "living room with a coffee table"}, {"image_id": 545959, "caption": "fire hydrant sitting on top of a lush green field"}, {"image_id": 12448, "caption": "little girl sitting on top of a teddy bear"}, {"image_id": 63154, "caption": "man riding a wave on a surfboard in the ocean"}, {"image_id": 546702, "caption": "black and white cat laying on top of a bed"}, {"image_id": 570579, "caption": "group of people sitting at a table with a plate of food"}, {"image_id": 295589, "caption": "baseball player holding a baseball bat on a baseball field"}, {"image_id": 26767, "caption": "couple of a bench in front of a street sign"}, {"image_id": 146723, "caption": "man playing a game of a soccer ball"}, {"image_id": 216841, "caption": "man sitting on top of a table"}, {"image_id": 122390, "caption": "black and white dog laying on a bed"}, {"image_id": 1146, "caption": "man wearing a suit and tie holding a cell phone"}, {"image_id": 511136, "caption": "giraffe standing in front of a building"}, {"image_id": 485709, "caption": "baseball player holding a baseball bat on a baseball field"}, {"image_id": 299319, "caption": "man holding a surfboard on the beach"}, {"image_id": 60760, "caption": "bathroom with a sink and white toilet and a mirror"}, {"image_id": 136501, "caption": "baseball player is holding a bat in front of a field"}, {"image_id": 188958, "caption": "group of people standing on a bench in front of a street"}, {"image_id": 462341, "caption": "clock tower with a clock on the side of a building"}, {"image_id": 564745, "caption": "man holding a tennis racket on a tennis court holding a tennis racquet on a skateboard"}, {"image_id": 86408, "caption": "kitchen with wood cabinets and stainless steel appliances"}, {"image_id": 122851, "caption": "group of people standing next to each other in the background"}, {"image_id": 75162, "caption": "zebra standing on top of a lush green field"}, {"image_id": 400538, "caption": "group of people sitting at a table with food in a living room"}, {"image_id": 268556, "caption": "man riding a motorcycle parked in front of a city street"}, {"image_id": 455859, "caption": "group of people playing a game of frisbee"}, {"image_id": 515289, "caption": "group of people riding a bike down a road in front of a street"}, {"image_id": 493174, "caption": "baseball player holding a baseball bat at a game"}, {"image_id": 545293, "caption": "park bench sitting on top of a lush green field"}, {"image_id": 352194, "caption": "man riding a motorcycle down a busy city street"}, {"image_id": 184321, "caption": "train traveling down a track in front of a road"}, {"image_id": 297343, "caption": "close up of a stop sign with a pair of scissors"}, {"image_id": 261061, "caption": "young girl holding a frisbee in a field"}, {"image_id": 67868, "caption": "black and white dog sitting on top of a bench"}, {"image_id": 39540, "caption": "pizza that is sitting on top of a plate"}, {"image_id": 78707, "caption": "baseball player swinging a bat at home plate during a game"}, {"image_id": 28758, "caption": "giraffe standing on top of a lush green field"}, {"image_id": 570465, "caption": "plate of food sitting on top of a table"}, {"image_id": 108094, "caption": "man riding a bike parked in front of a building with a clock on a city street"}, {"image_id": 201723, "caption": "cat sitting on top of a table with books"}, {"image_id": 356302, "caption": "black and white cat sitting on top of a desk"}, {"image_id": 513681, "caption": "large white airplane that is sitting on top of a runway"}, {"image_id": 27353, "caption": "close up of a plate of food on a table"}, {"image_id": 418281, "caption": "giraffe standing in front of a lush green field"}, {"image_id": 323682, "caption": "group of people standing in front of a building"}, {"image_id": 563964, "caption": "young boy holding a frisbee in the air"}, {"image_id": 386146, "caption": "giraffe standing on top of a lush green field"}, {"image_id": 555705, "caption": "cat sitting on top of a computer desk"}, {"image_id": 129159, "caption": "city street with a clock tower in front of a building"}, {"image_id": 93201, "caption": "black and white cat laying on top of a bed"}, {"image_id": 517822, "caption": "man holding a frisbee in a field with a baseball bat"}, {"image_id": 213687, "caption": "group of people walking down the street in front of a city street"}, {"image_id": 466787, "caption": "bowl of fruits and vegetables on it"}, {"image_id": 507065, "caption": "man sitting on top of a bench"}, {"image_id": 16356, "caption": "man flying a kite in the air"}, {"image_id": 307569, "caption": "group of people riding horses on the beach"}, {"image_id": 486438, "caption": "close up of donuts that are sitting on a table"}, {"image_id": 35197, "caption": "man riding a skateboard on top of a skate board"}, {"image_id": 65244, "caption": "man sitting on top of a wall"}, {"image_id": 322654, "caption": "street sign on a pole next to a street sign"}, {"image_id": 347170, "caption": "young boy riding a skateboard on a tennis court holding a baseball bat"}, {"image_id": 197254, "caption": "large truck parked in front of a building with a clock on top of a building"}, {"image_id": 180800, "caption": "bowl of fruit and vegetables on a table"}, {"image_id": 462565, "caption": "man riding a bike down a street"}, {"image_id": 14941, "caption": "cat laying on a bed next to each other"}, {"image_id": 516750, "caption": "black and white cat is sitting on top of a wall"}, {"image_id": 500062, "caption": "tennis player is playing tennis on a tennis court holding a tennis racket at a tennis ball"}, {"image_id": 170629, "caption": "double decker bus is parked in front of a city street"}, {"image_id": 310103, "caption": "pile of luggage sitting on a road"}, {"image_id": 143769, "caption": "baby elephant standing next to a fence"}, {"image_id": 427438, "caption": "group of zebras standing next to each other in a field"}, {"image_id": 168330, "caption": "city bus driving down the street at night"}, {"image_id": 477949, "caption": "vase filled with flowers sitting on a table"}, {"image_id": 534957, "caption": "living room with a white toilet sitting on top of a table"}, {"image_id": 560108, "caption": "young boy riding a skateboard down a baseball bat"}, {"image_id": 242499, "caption": "group of people standing next to a boat in the water"}, {"image_id": 551952, "caption": "truck is parked in front of a train station with a parking meter on a city street"}, {"image_id": 383406, "caption": "man sitting on top of a bed"}, {"image_id": 475779, "caption": "baby elephant standing in front of a tree"}, {"image_id": 504142, "caption": "cat laying on top of a bed"}, {"image_id": 466774, "caption": "baby elephant standing in front of a tree"}, {"image_id": 217285, "caption": "baseball player is holding a bat in a baseball game"}, {"image_id": 209692, "caption": "red fire truck parked in front of a building"}, {"image_id": 385861, "caption": "man sitting at a table eating a birthday cake with a hot dog"}, {"image_id": 251252, "caption": "man riding a motorcycle parked in front of a street"}, {"image_id": 9426, "caption": "large plane flying through the sky"}, {"image_id": 308394, "caption": "cat sitting on top of a bed next to a window"}, {"image_id": 185768, "caption": "close up of a teddy bear sitting on top of it"}, {"image_id": 572051, "caption": "close up of a plate of food on a table"}, {"image_id": 26942, "caption": "close up of a cat laying on top of a table"}, {"image_id": 335532, "caption": "man sitting on a piece of a plate of food"}, {"image_id": 524456, "caption": "cat sitting on top of a dog"}, {"image_id": 307936, "caption": "baby elephant standing in a dirt field"}, {"image_id": 365289, "caption": "little girl sitting on a bed with a teddy bear"}, {"image_id": 334321, "caption": "group of people sitting on a horse"}, {"image_id": 66412, "caption": "man standing in front of a bathroom"}, {"image_id": 144941, "caption": "bathroom with a toilet sitting on top of a bathroom"}, {"image_id": 191381, "caption": "bathroom with a white toilet sitting on top of a toilet"}, {"image_id": 284350, "caption": "man in a suit standing on a tennis court holding a tennis racket"}, {"image_id": 41888, "caption": "zebra standing on top of a lush green field"}, {"image_id": 579815, "caption": "man standing in front of a city street"}, {"image_id": 573206, "caption": "pizza that is sitting on top of a table"}, {"image_id": 198448, "caption": "woman sitting on a bench in front of a cell phone"}, {"image_id": 542145, "caption": "bathroom with a sink in front of a mirror"}, {"image_id": 573291, "caption": "giraffe standing on top of a lush green grass covered field"}, {"image_id": 77184, "caption": "couple of boats docked in the water"}, {"image_id": 142092, "caption": "close up of a pizza sitting on top of a table"}, {"image_id": 305821, "caption": "giraffe standing on top of a lush green field"}, {"image_id": 191270, "caption": "group of people sitting on a table"}, {"image_id": 32992, "caption": "black and white cat sitting on top of a wooden table"}, {"image_id": 270244, "caption": "zebra standing on top of a lush green field"}, {"image_id": 168706, "caption": "cow standing on top of a lush green field"}, {"image_id": 222304, "caption": "vase filled with flowers sitting on a table"}, {"image_id": 579056, "caption": "man standing on top of a tennis court"}, {"image_id": 476597, "caption": "group of people standing in a living room playing a video game"}, {"image_id": 495612, "caption": "street sign sitting on top of a building"}, {"image_id": 372938, "caption": "man riding a motorcycle down the road"}, {"image_id": 516316, "caption": "herd of zebra standing in a grassy field"}, {"image_id": 578292, "caption": "stop sign in front of a street"}, {"image_id": 288403, "caption": "group of people standing in front of a soccer ball"}, {"image_id": 258985, "caption": "white toilet sitting on top of a table"}, {"image_id": 420229, "caption": "close up of a bowl of scissors sitting on a table"}, {"image_id": 139530, "caption": "group of people standing next to each other"}, {"image_id": 463730, "caption": "group of people walking down a city street"}, {"image_id": 196311, "caption": "living room with lots of luggage sitting on top of a table"}, {"image_id": 568101, "caption": "group of people flying kites in the sky"}, {"image_id": 564659, "caption": "close up of a plate of food on a table"}, {"image_id": 460347, "caption": "double decker bus driving down the road next to a car"}, {"image_id": 53990, "caption": "man eating a hot dog sitting on a plate with a slice of cake"}, {"image_id": 557981, "caption": "young woman holding a cell phone"}, {"image_id": 400139, "caption": "plate of food on a table"}, {"image_id": 234572, "caption": "man doing a trick on a skateboard in the air on top of a ramp"}, {"image_id": 39760, "caption": "man flying a kite in the sky"}, {"image_id": 226111, "caption": "stop sign in front of a street sign"}, {"image_id": 540763, "caption": "bathroom with a white toilet sitting on top of a table"}, {"image_id": 550529, "caption": "red motorcycle parked in front of a building"}, {"image_id": 283210, "caption": "man riding a horse in the middle of an elephant"}, {"image_id": 403817, "caption": "man standing in front of a tv"}, {"image_id": 81922, "caption": "plane is flying through the sky"}, {"image_id": 303818, "caption": "group of people riding a motorcycle on a city street"}, {"image_id": 481635, "caption": "man sitting at a table with a plate of food"}, {"image_id": 17707, "caption": "black and white cat sitting on top of a beach"}, {"image_id": 224861, "caption": "black and white photo of a man holding a cell phone"}, {"image_id": 60623, "caption": "man sitting at a table with a plate of food"}, {"image_id": 511058, "caption": "close up of giraffe standing next to each other in the background"}, {"image_id": 453757, "caption": "man standing on a tennis court holding a baseball bat at a ball"}, {"image_id": 46859, "caption": "group of people playing a game of frisbee in a field"}, {"image_id": 208589, "caption": "close up of a pizza sitting on top of a table"}, {"image_id": 42069, "caption": "giraffe standing in front of a building"}, {"image_id": 73830, "caption": "giraffe is standing next to a fence"}, {"image_id": 212663, "caption": "close up of a bus stop sign on a city street"}, {"image_id": 84235, "caption": "hot dog sitting on a bun next to a plate of food"}, {"image_id": 232309, "caption": "man riding a kite on a surfboard in the ocean"}, {"image_id": 314294, "caption": "herd of elephants standing next to each other in the background"}, {"image_id": 156832, "caption": "bathroom with a toilet and sink and a mirror"}, {"image_id": 325690, "caption": "stop sign on the side of a street"}, {"image_id": 336777, "caption": "stop sign in front of a city street"}, {"image_id": 572517, "caption": "black and white polar bear standing in the water"}, {"image_id": 577539, "caption": "birthday cake sitting on top of a white plate"}, {"image_id": 14869, "caption": "giraffe standing next to a park bench in front of a lush green field"}, {"image_id": 226278, "caption": "group of people standing on a city street"}, {"image_id": 117407, "caption": "black and white cat sitting on top of a bed"}, {"image_id": 370423, "caption": "black and white cat sitting on top of a bed"}, {"image_id": 156397, "caption": "herd of sheep grazing on a lush green field"}, {"image_id": 125476, "caption": "black and white cat sitting on a park bench in front of a lush green field"}, {"image_id": 244339, "caption": "man sitting on top of a table"}, {"image_id": 157269, "caption": "group of people standing in front of a bus"}, {"image_id": 98416, "caption": "group of sheep standing next to a herd of sheep grazing on a lush green field"}, {"image_id": 313789, "caption": "group of fruit in front of an outdoor market"}, {"image_id": 7211, "caption": "train is parked in front of a train station"}, {"image_id": 540186, "caption": "bathroom with a sink in front of a kitchen"}, {"image_id": 453819, "caption": "boat that is sitting on the water"}, {"image_id": 15113, "caption": "close up of a bird perched on top of a tree"}, {"image_id": 493868, "caption": "bowl of vegetables sitting on top of a plate"}, {"image_id": 165029, "caption": "man sitting on top of a bed"}, {"image_id": 556616, "caption": "man standing next to a parking meter"}, {"image_id": 317560, "caption": "young boy sitting on a table holding a banana"}, {"image_id": 368402, "caption": "man standing in front of a kitchen"}, {"image_id": 292301, "caption": "white plate that is sitting on a table next to a cup of coffee"}, {"image_id": 436470, "caption": "living room with a chair in front of a living room"}, {"image_id": 448269, "caption": "group of people standing on a baseball field"}, {"image_id": 311913, "caption": "brown horse standing in front of a lush green field"}, {"image_id": 183666, "caption": "train traveling down the tracks in front of a train station"}, {"image_id": 436141, "caption": "bathroom with a toilet and a sink"}, {"image_id": 138975, "caption": "group of people standing on top of a snow covered slope"}, {"image_id": 66423, "caption": "group of people sitting on the beach"}, {"image_id": 34180, "caption": "young boy holding a banana"}, {"image_id": 138477, "caption": "group of people standing in a field flying a kite in a grassy field"}, {"image_id": 74711, "caption": "group of birds are standing in the water"}, {"image_id": 62060, "caption": "group of people standing in a field with a frisbee"}, {"image_id": 153343, "caption": "teddy bear sitting on top of a fire hydrant"}, {"image_id": 502136, "caption": "clock tower sitting on top of a fire hydrant in front of a building"}, {"image_id": 71466, "caption": "bed is sitting on top of a wall"}, {"image_id": 541010, "caption": "group of people standing in a field with a frisbee"}, {"image_id": 499940, "caption": "young boy standing in a field holding a kite in the air"}, {"image_id": 281455, "caption": "person riding a wave on top of a beach"}, {"image_id": 180490, "caption": "black and white photo of a living room with a refrigerator"}, {"image_id": 171062, "caption": "dog is sitting in a living room with a computer screen"}, {"image_id": 540264, "caption": "woman holding a cell phone in front of a man"}, {"image_id": 301102, "caption": "man sitting on top of a tennis court"}, {"image_id": 182245, "caption": "group of people sitting at a table with plates of food in front of a birthday cake"}, {"image_id": 190081, "caption": "woman is playing a game in a living room with a tennis racket"}, {"image_id": 109819, "caption": "group of people walking down a street"}, {"image_id": 212704, "caption": "group of people sitting on a couch playing a video game"}, {"image_id": 35474, "caption": "baby elephant standing in front of a tree"}, {"image_id": 515579, "caption": "young boy standing on a tennis court holding a baseball bat"}, {"image_id": 143541, "caption": "man riding a wave on the beach holding a surfboard in the ocean"}, {"image_id": 17756, "caption": "man sitting on a park bench in the water"}, {"image_id": 335472, "caption": "young man riding skis down a snow covered slope"}, {"image_id": 249720, "caption": "man riding a skateboard down a street"}, {"image_id": 171500, "caption": "truck driving down a road in front of a lush green field"}, {"image_id": 355257, "caption": "laptop computer sitting on top of a desk"}, {"image_id": 62790, "caption": "man sitting on top of a tv"}, {"image_id": 80172, "caption": "young boy holding a video game controller"}, {"image_id": 24430, "caption": "boat that is sitting on top of a city street"}, {"image_id": 182784, "caption": "group of people standing next to each other in the background"}, {"image_id": 72833, "caption": "man riding a wave on top of a surfboard"}, {"image_id": 31747, "caption": "boat that is on a body of boats in the water"}, {"image_id": 12543, "caption": "man flying a kite in front of a field"}, {"image_id": 129637, "caption": "group of horses standing next to each other"}, {"image_id": 568623, "caption": "group of people standing in front of a luggage bag"}, {"image_id": 527025, "caption": "group of people sitting on a table"}, {"image_id": 383470, "caption": "stop sign on the side of a snow covered slope"}, {"image_id": 248242, "caption": "man riding a snowboard down the side of a snow covered slope"}, {"image_id": 206027, "caption": "sandwich sitting on top of a table"}, {"image_id": 384204, "caption": "black and white cat laying on top of a couch next to a window"}, {"image_id": 190056, "caption": "group of zebras are standing next to each other in the background"}, {"image_id": 103579, "caption": "group of people standing on a lush green field holding a frisbee in the park"}, {"image_id": 238866, "caption": "black and white photo of a teddy bear sitting on top of a table"}, {"image_id": 57545, "caption": "man sitting on a park bench in front of a tree"}, {"image_id": 453756, "caption": "close up of a table that is sitting on top of a table"}, {"image_id": 446322, "caption": "group of donuts sitting on a table"}, {"image_id": 272110, "caption": "man on a skateboard doing a trick on a bench in front of a building"}, {"image_id": 536831, "caption": "man riding skis down a snow covered slope"}, {"image_id": 180560, "caption": "group of people standing next to each other"}, {"image_id": 364521, "caption": "close up of a banana sitting on top of a table"}, {"image_id": 403013, "caption": "bathroom with a sink in the kitchen"}, {"image_id": 318171, "caption": "giraffe standing in front of a lush green field"}, {"image_id": 574184, "caption": "black and white cat sitting on top of a computer desk"}, {"image_id": 345998, "caption": "man riding a horse in a dirt field"}, {"image_id": 508730, "caption": "woman holding a stuffed teddy bear sitting on a table"}, {"image_id": 189845, "caption": "young man holding a tennis racquet on a tennis court"}, {"image_id": 335833, "caption": "city street with a clock tower on the side of a building"}, {"image_id": 329717, "caption": "woman sitting on top of a bed"}, {"image_id": 5064, "caption": "baseball player swinging a bat at a ball during a game"}, {"image_id": 554625, "caption": "man sitting on top of a laptop computer"}, {"image_id": 562150, "caption": "woman sitting on top of a plate"}, {"image_id": 467477, "caption": "man riding a wave on a surfboard in the ocean"}, {"image_id": 112085, "caption": "herd of horses grazing on a lush green field"}, {"image_id": 410437, "caption": "red fire hydrant sitting on top of a building"}, {"image_id": 20965, "caption": "woman swinging a tennis ball on a tennis court holding a tennis racket on a tennis court"}, {"image_id": 558524, "caption": "white and white vase that is sitting on top of a table"}, {"image_id": 256091, "caption": "herd of zebra standing on a dirt road"}, {"image_id": 480076, "caption": "black and white cat sitting on top of a bed"}, {"image_id": 208663, "caption": "close up of a plate of food on a table"}, {"image_id": 451872, "caption": "group of elephants are standing in a dirt field"}, {"image_id": 272262, "caption": "man holding a tennis racket on a tennis court holding a tennis racquet on a baseball field"}, {"image_id": 324937, "caption": "cat laying on a bed with a teddy bear sitting on top of a table"}, {"image_id": 203849, "caption": "kitchen with white cabinets and stainless steel appliances"}, {"image_id": 403385, "caption": "bathroom with a toilet sitting next to a bathroom"}, {"image_id": 577858, "caption": "group of people sitting at a table with plates of food"}, {"image_id": 85329, "caption": "man wearing a hat and tie"}, {"image_id": 412151, "caption": "group of motorcycles parked in front of a street"}, {"image_id": 308599, "caption": "stop sign sitting on the side of a building"}, {"image_id": 104025, "caption": "black and white horse is riding a wave on the beach"}, {"image_id": 185250, "caption": "young boy holding a frisbee in a field"}, {"image_id": 278506, "caption": "group of people standing on a beach flying kites"}, {"image_id": 7682, "caption": "man wearing a suit and tie holding a cell phone"}, {"image_id": 15085, "caption": "couple of zebras are standing in the water"}, {"image_id": 216482, "caption": "giraffe standing next to a tree in the background"}, {"image_id": 143931, "caption": "close up of a bus stop sign on a city street"}, {"image_id": 60687, "caption": "herd of sheep standing on top of a lush green field"}, {"image_id": 413321, "caption": "man riding a wave on a surfboard in the ocean"}, {"image_id": 280607, "caption": "close up of a piece of pizza sitting on a table"}, {"image_id": 74331, "caption": "bunch of bananas on display in front of a street"}, {"image_id": 51191, "caption": "bathroom with a sink and a mirror"}, {"image_id": 313155, "caption": "hot dog sitting on top of a plate on a table"}, {"image_id": 375363, "caption": "bench sitting in front of a tree"}, {"image_id": 235302, "caption": "group of people sitting on a table"}, {"image_id": 497312, "caption": "man riding a horse in a field"}, {"image_id": 191501, "caption": "red and white photo of a street sign sitting on top of a building"}, {"image_id": 241453, "caption": "couple of cows are standing next to each other"}, {"image_id": 232262, "caption": "group of people walking down a busy city street"}, {"image_id": 113914, "caption": "woman sitting in front of a car"}, {"image_id": 502737, "caption": "woman holding a piece of cake on a table"}, {"image_id": 65415, "caption": "young boy riding skis down a snow covered slope"}, {"image_id": 102446, "caption": "living room with a television and a window"}, {"image_id": 368117, "caption": "traffic light on the side of a red stop sign"}, {"image_id": 162827, "caption": "black and white teddy bear sitting on a table"}, {"image_id": 41369, "caption": "black and white cat sitting on top of a laptop"}, {"image_id": 15690, "caption": "herd of elephants walking through the water"}, {"image_id": 122745, "caption": "stop sign on the side of a street sign"}, {"image_id": 458054, "caption": "bird that is sitting on top of a wooden table"}, {"image_id": 102912, "caption": "group of elephants standing next to a fire hydrant"}, {"image_id": 177941, "caption": "close up of a pizza that is sitting on a table"}, {"image_id": 321214, "caption": "little boy sitting at a table eating a slice of pizza"}, {"image_id": 137003, "caption": "young girl holding a kite in a field"}, {"image_id": 25202, "caption": "plate that is sitting on a table"}, {"image_id": 321674, "caption": "man sitting in front of a window"}, {"image_id": 129001, "caption": "bathroom with a toilet and a sink"}, {"image_id": 79841, "caption": "man riding a bike down a street in front of a city street"}, {"image_id": 521634, "caption": "street sign on a pole in front of a traffic light"}, {"image_id": 350694, "caption": "group of people sitting on a boat in the water"}, {"image_id": 529636, "caption": "large clock tower in front of a building"}, {"image_id": 415746, "caption": "display of donuts sitting on top of a table"}, {"image_id": 64822, "caption": "red and white cat sitting on top of a street sign"}, {"image_id": 162952, "caption": "woman standing in a living room holding a laptop computer"}, {"image_id": 368978, "caption": "blue and white airplane flying through the sky"}, {"image_id": 76460, "caption": "group of elephants standing in a grassy field"}, {"image_id": 81561, "caption": "traffic light with a clock tower in the air"}, {"image_id": 514508, "caption": "group of people sitting on a table"}, {"image_id": 269089, "caption": "hot dog sitting on top of a plate next to a cup of food on a table"}, {"image_id": 472621, "caption": "bathroom with a sink and a mirror"}, {"image_id": 222564, "caption": "man standing in front of a kitchen"}, {"image_id": 66523, "caption": "bathroom with a white bath tub"}, {"image_id": 289444, "caption": "group of young children playing a game of frisbee in a field"}, {"image_id": 16716, "caption": "group of zebras standing next to each other in a grassy field"}, {"image_id": 418949, "caption": "baseball player holding a bat on a field playing a game"}, {"image_id": 129699, "caption": "group of people standing in front of a train station"}, {"image_id": 153656, "caption": "white and white cat is sitting on top of a table"}, {"image_id": 234500, "caption": "man eating a hot dog on a plate with a sandwich on a table"}, {"image_id": 281221, "caption": "group of giraffes standing in a grassy field"}, {"image_id": 94194, "caption": "man sitting in front of a kitchen"}, {"image_id": 289393, "caption": "black and white cat sitting on top of a table"}, {"image_id": 398076, "caption": "man is holding a cell phone"}, {"image_id": 578210, "caption": "teddy bear sitting on a table next to a birthday cake"}, {"image_id": 167854, "caption": "woman holding a tennis racket on the beach"}, {"image_id": 362373, "caption": "laptop computer sitting on top of a table in front of a living room"}, {"image_id": 41945, "caption": "red and white photo of a boat that is parked in front of a building"}, {"image_id": 391895, "caption": "man holding a red umbrella in the rain"}, {"image_id": 432176, "caption": "black and white cat sitting in front of a window"}, {"image_id": 146973, "caption": "white plate that is sitting on a table"}, {"image_id": 381021, "caption": "man standing in front of a refrigerator"}, {"image_id": 434230, "caption": "group of people sitting on top of a table with a clock on it"}, {"image_id": 96493, "caption": "young girl laying on top of a couch holding a baby sitting on a bed"}, {"image_id": 360661, "caption": "group of people riding horses on a horse"}, {"image_id": 483108, "caption": "double decker bus is parked in front of a building"}, {"image_id": 87038, "caption": "group of people riding a skateboard down a street in front of a city street"}, {"image_id": 440575, "caption": "large airplane flying through the sky"}, {"image_id": 224736, "caption": "bathroom that is sitting on top of a kitchen counter"}, {"image_id": 247306, "caption": "group of boats docked in the water"}, {"image_id": 173350, "caption": "black and white dog laying on a bed next to each other"}, {"image_id": 240903, "caption": "group of sheep standing on top of a lush green field"}, {"image_id": 240028, "caption": "young boy standing on a tennis court holding a frisbee in a field"}, {"image_id": 413044, "caption": "little girl sitting on a bench in front of a fire hydrant"}, {"image_id": 201925, "caption": "close up of a pizza sitting on top of a table"}, {"image_id": 279305, "caption": "close up of a close up of a pair of scissors sitting on a table"}, {"image_id": 176649, "caption": "black and white photo of a street sign sitting on top of a table"}, {"image_id": 83915, "caption": "large clock tower with a clock tower in front of a building"}, {"image_id": 181449, "caption": "black and white cat sitting on top of a cell phone"}, {"image_id": 158015, "caption": "man standing on a tennis court holding a tennis racket in front of a tennis ball"}, {"image_id": 293802, "caption": "man on a skateboard doing a trick on a tennis court holding a tennis racket"}, {"image_id": 414679, "caption": "brown and white cow standing in a field"}, {"image_id": 514083, "caption": "man riding a skate board doing a trick on his skateboard"}, {"image_id": 153607, "caption": "vase that is sitting on top of a table"}, {"image_id": 262986, "caption": "black bear standing in the water"}, {"image_id": 142667, "caption": "group of people standing in front of a street"}, {"image_id": 6818, "caption": "bathroom with a toilet and sink"}, {"image_id": 143824, "caption": "man sitting in front of a dog"}, {"image_id": 28655, "caption": "street sign sitting on top of a pole in front of a building"}, {"image_id": 393258, "caption": "black and white cat sitting on top of a bed"}, {"image_id": 515779, "caption": "group of people sitting on a cell phone"}, {"image_id": 250108, "caption": "laptop computer sitting on top of a desk next to each other"}, {"image_id": 236182, "caption": "close up of a street sign sitting on top of a tree"}, {"image_id": 387362, "caption": "man sitting on top of a cell phone"}, {"image_id": 289423, "caption": "group of young men playing a game of frisbee in the air"}, {"image_id": 54277, "caption": "man standing on top of a wall"}, {"image_id": 318219, "caption": "black and white cat sitting on top of a computer desk"}, {"image_id": 559842, "caption": "group of people playing a game of soccer on a field"}, {"image_id": 419048, "caption": "woman holding a frisbee on a lush green grass covered field"}, {"image_id": 341041, "caption": "young man riding a skateboard on a tennis court holding a tennis racket at a skate park"}, {"image_id": 245201, "caption": "black and white cat sitting on top of a wall"}, {"image_id": 155877, "caption": "group of sheep standing in front of a fence"}, {"image_id": 10766, "caption": "young boy holding a cell phone"}, {"image_id": 69392, "caption": "man sitting on top of a tennis court"}, {"image_id": 108169, "caption": "black and white photo of a snow covered in the middle of water"}, {"image_id": 101622, "caption": "man holding a frisbee in the air"}, {"image_id": 551215, "caption": "man standing on a tennis court holding a frisbee on a tennis court holding a tennis racket"}, {"image_id": 155743, "caption": "zebra standing on top of a lush green field"}, {"image_id": 82327, "caption": "clock tower sitting in front of a building"}, {"image_id": 557360, "caption": "close up of a close up of a close up of a hand holding a tree"}, {"image_id": 295564, "caption": "bunch of vases sitting on display in the background"}, {"image_id": 173704, "caption": "man sitting at a table holding a cell phone"}, {"image_id": 173574, "caption": "herd of sheep are standing in the grass field"}, {"image_id": 321079, "caption": "chocolate cake sitting on top of a white plate"}, {"image_id": 295837, "caption": "man sitting at a table in front of a pizza"}, {"image_id": 527248, "caption": "group of people standing in front of a building"}, {"image_id": 61181, "caption": "group of cars are driving down a busy city street"}, {"image_id": 410632, "caption": "group of people standing on a tennis court holding a soccer ball"}, {"image_id": 421681, "caption": "laptop computer sitting on top of a wooden table"}, {"image_id": 37675, "caption": "group of cows are standing in a grassy field"}, {"image_id": 5802, "caption": "group of people sitting at a table with food in front of a kitchen"}, {"image_id": 522713, "caption": "boat that is sitting on top of a beach"}, {"image_id": 420532, "caption": "group of men standing next to a man and a woman wearing a suit and tie"}, {"image_id": 32965, "caption": "man in the water holding a kite on the beach"}, {"image_id": 409944, "caption": "man sitting on top of a laptop computer"}, {"image_id": 17627, "caption": "group of a bus that is driving down a street with a traffic light"}, {"image_id": 503707, "caption": "man riding a motorcycle down a road next to a car"}, {"image_id": 290570, "caption": "herd of sheep are standing on top of a lush green field"}, {"image_id": 458052, "caption": "black and white cat laying on top of a bed"}, {"image_id": 437609, "caption": "black and white photo of a herd of cows are standing next to each other"}, {"image_id": 146411, "caption": "cat laying on top of a bed"}, {"image_id": 207056, "caption": "group of elephants are standing in the water"}, {"image_id": 299116, "caption": "man sitting at a table with a plate of food"}, {"image_id": 18149, "caption": "man standing next to a building"}, {"image_id": 443303, "caption": "black and white cat sitting on top of a bed"}, {"image_id": 519706, "caption": "man standing on top of a tennis court"}, {"image_id": 248468, "caption": "black and white photo of a man riding a horse drawn carriage in the background"}, {"image_id": 166532, "caption": "bathroom with a sink and a mirror"}, {"image_id": 334405, "caption": "double decker bus driving down the street with a traffic light"}, {"image_id": 69946, "caption": "group of boats are docked in the water"}, {"image_id": 285291, "caption": "man is skiing down the side of a snow covered slope"}, {"image_id": 409009, "caption": "passenger train is parked in front of a train station"}, {"image_id": 579003, "caption": "black and white photo of a bathroom with a mirror"}, {"image_id": 216228, "caption": "man standing in front of a store"}, {"image_id": 426523, "caption": "group of people standing on a park bench holding a frisbee in a field"}, {"image_id": 209868, "caption": "man sitting on a cell phone"}, {"image_id": 324266, "caption": "bathroom with a toilet and a sink"}, {"image_id": 407368, "caption": "polar bear in a body of water"}, {"image_id": 183803, "caption": "living room with a couch next to a wooden table"}, {"image_id": 59202, "caption": "group of people doing a trick on a skateboard"}, {"image_id": 320039, "caption": "little girl sitting at a table with a plate of food"}, {"image_id": 219578, "caption": "close up of a dog sitting on a table"}, {"image_id": 533750, "caption": "close up of a donut sitting on a table"}, {"image_id": 118113, "caption": "cat sitting in front of a living room"}, {"image_id": 430961, "caption": "baseball player getting ready to hit a ball on a tennis court"}, {"image_id": 538859, "caption": "man sitting at a table with plates of food"}, {"image_id": 403065, "caption": "truck is parked in front of a road"}, {"image_id": 266409, "caption": "man riding skis down a snow covered slope"}, {"image_id": 152751, "caption": "laptop computer sitting in front of a living room"}, {"image_id": 409251, "caption": "group of people standing next to each other"}, {"image_id": 365426, "caption": "group of people playing tennis on a tennis court holding tennis rackets on a tennis court"}, {"image_id": 57387, "caption": "group of people standing on a tennis court holding a tennis racket at a ball"}, {"image_id": 85160, "caption": "group of people sitting in a living room"}, {"image_id": 207797, "caption": "giraffe standing in the water"}, {"image_id": 336102, "caption": "close up of a pizza sitting on top of a plate"}, {"image_id": 13383, "caption": "woman that is sitting on top of a plate on a table"}, {"image_id": 340420, "caption": "pizza that is sitting on top of a table"}, {"image_id": 557780, "caption": "black and white dog holding a frisbee in the background"}, {"image_id": 150410, "caption": "man sitting on top of a field"}, {"image_id": 184613, "caption": "group of cows are standing in front of a lush green field"}, {"image_id": 430359, "caption": "baseball player holding a bat in front of a field"}, {"image_id": 560323, "caption": "man standing on a tennis court holding a tennis racket at a tennis ball"}, {"image_id": 169891, "caption": "black and white cat is standing in front of a kitchen"}, {"image_id": 241876, "caption": "black and white cat laying on top of a bed"}, {"image_id": 146627, "caption": "group of people sitting on a table"}, {"image_id": 153299, "caption": "giraffe is standing in front of a building"}, {"image_id": 354533, "caption": "motorcycle parked in front of a lush green field"}, {"image_id": 232383, "caption": "man sitting on top of a laptop"}, {"image_id": 174482, "caption": "man riding a motorcycle down the side of a road"}, {"image_id": 483234, "caption": "man standing on top of a snow covered slope"}, {"image_id": 125211, "caption": "zebra standing on top of a lush green field"}, {"image_id": 557556, "caption": "man holding a tennis racquet on a tennis court holding a tennis racket"}, {"image_id": 461898, "caption": "woman is holding a cell phone"}, {"image_id": 181666, "caption": "group of cows are standing next to each other in the background"}, {"image_id": 38034, "caption": "laptop computer sitting on top of a desk next to a computer monitor"}, {"image_id": 158952, "caption": "stop sign on the side of a street sign"}, {"image_id": 446751, "caption": "herd of sheep standing next to each other in a field"}, {"image_id": 347693, "caption": "living room with chairs sitting on a bed in front of a window"}, {"image_id": 553667, "caption": "man holding a hot dog in his mouth"}, {"image_id": 140743, "caption": "woman holding a teddy bear sitting on a cell phone"}, {"image_id": 75083, "caption": "street sign in front of a city street"}, {"image_id": 82258, "caption": "clock sitting on top of books"}, {"image_id": 369370, "caption": "close up of a plate of food on a table"}, {"image_id": 124629, "caption": "group of young men playing a game of frisbee on a tennis court"}, {"image_id": 172327, "caption": "man riding a wave on a surfboard in the ocean"}, {"image_id": 339974, "caption": "traffic light on a city street at night"}, {"image_id": 24223, "caption": "pizza sitting on top of a plate"}, {"image_id": 527040, "caption": "man sitting on a bed with a cake on a table"}, {"image_id": 74369, "caption": "young boy playing a game of frisbee in a field"}, {"image_id": 252213, "caption": "cat laying on top of a white plate"}, {"image_id": 122203, "caption": "black and white photo of a herd of sheep grazing on a lush green field"}, {"image_id": 419856, "caption": "bunch of bananas on a table next to each other"}, {"image_id": 16574, "caption": "group of people riding skis down the side of a snow covered slope"}, {"image_id": 384012, "caption": "baseball player swinging a bat in front of a baseball game"}, {"image_id": 386164, "caption": "close up of a piece of scissors sitting on a table"}, {"image_id": 271032, "caption": "small boat in a body of water"}, {"image_id": 310227, "caption": "group of people sitting on a bed"}, {"image_id": 327807, "caption": "group of people standing in front of a train station"}, {"image_id": 433460, "caption": "group of zebras standing next to each other in a field"}, {"image_id": 505440, "caption": "group of zebras standing on top of a lush green field"}, {"image_id": 430125, "caption": "group of people standing in front of a store"}, {"image_id": 449981, "caption": "group of people standing next to each other in the background"}, {"image_id": 334686, "caption": "man eating a hot dog on a plate"}, {"image_id": 560623, "caption": "plane that is parked on a runway"}, {"image_id": 506552, "caption": "group of people are playing a video game"}, {"image_id": 469088, "caption": "brown bear standing next to each other"}, {"image_id": 46743, "caption": "young boy sitting on a bench in front of a fire hydrant"}, {"image_id": 271177, "caption": "man riding a motorcycle parked in front of a road"}, {"image_id": 577826, "caption": "group of people riding a skateboard down a street"}, {"image_id": 6005, "caption": "group of giraffes standing next to each other"}, {"image_id": 287570, "caption": "man riding skis down a snow covered slope"}, {"image_id": 356708, "caption": "group of people riding skis down a snowy slope"}, {"image_id": 23899, "caption": "man sitting on top of a bed"}, {"image_id": 294119, "caption": "close up of a stop sign on a cell phone"}, {"image_id": 94590, "caption": "cup of donuts are sitting on top of a plate"}, {"image_id": 577821, "caption": "large clock tower with a clock on top of a building"}, {"image_id": 142592, "caption": "double decker bus parked in front of a building"}, {"image_id": 402559, "caption": "giraffe standing in front of a tree"}, {"image_id": 154971, "caption": "group of people playing a game of frisbee in a field with a soccer ball"}, {"image_id": 138246, "caption": "man standing in the middle of a grassy field"}, {"image_id": 181714, "caption": "woman holding a red umbrella"}, {"image_id": 205378, "caption": "black and white cat sitting on top of a bed"}, {"image_id": 429580, "caption": "group of people standing in a field holding a frisbee in the grass"}, {"image_id": 247285, "caption": "woman sitting at a table with a cell phone"}, {"image_id": 269105, "caption": "white toilet sitting on top of a bathroom sink"}, {"image_id": 262284, "caption": "bathroom with a sink and a mirror"}, {"image_id": 273045, "caption": "close up of a cat sitting on top of a table"}, {"image_id": 89668, "caption": "herd of elephants standing on top of a grass field"}, {"image_id": 93725, "caption": "man swinging a tennis racket on a tennis court holding a tennis racquet on a baseball field"}, {"image_id": 71171, "caption": "close up of a pizza sitting on top of a plate"}, {"image_id": 43635, "caption": "group of people standing in front of a building"}, {"image_id": 405778, "caption": "group of zebras grazing on top of a lush green field"}, {"image_id": 358763, "caption": "laptop computer sitting on top of a desk next to each other"}, {"image_id": 28377, "caption": "large clock tower in front of a building"}, {"image_id": 64710, "caption": "man on a beach flying a kite in the snow"}, {"image_id": 472854, "caption": "group of people standing in front of a street"}, {"image_id": 291712, "caption": "blue and white photo of a plane flying through the water"}, {"image_id": 358342, "caption": "black and white photo of a building with a clock tower in front of a city street"}, {"image_id": 326781, "caption": "bathroom with a sink and a mirror"}, {"image_id": 1369, "caption": "man riding a wave in the water holding a surfboard in the ocean"}, {"image_id": 173383, "caption": "group of people sitting on a table"}, {"image_id": 386500, "caption": "cat laying on a bed with a teddy bear"}, {"image_id": 125059, "caption": "bathroom with a white toilet sitting on top of a table"}, {"image_id": 235006, "caption": "truck parked in front of a fire hydrant in front of a lush green field"}, {"image_id": 260141, "caption": "boat that is parked in the water"}, {"image_id": 235597, "caption": "group of people sitting in a living room with two laptops are playing a video game"}, {"image_id": 338948, "caption": "group of people standing in front of a building"}, {"image_id": 30255, "caption": "large black and white photo of people sitting on top of a boat"}, {"image_id": 551334, "caption": "cat sitting on top of a computer desk"}, {"image_id": 260166, "caption": "group of people standing on a dirt road"}, {"image_id": 58636, "caption": "street sign on a pole in front of a building"}, {"image_id": 377385, "caption": "baseball player is holding a tennis racket"}, {"image_id": 227879, "caption": "baseball player swinging a bat at a baseball game"}, {"image_id": 501762, "caption": "man in the water holding a frisbee in the ocean"}, {"image_id": 326541, "caption": "man holding a cell phone"}, {"image_id": 511117, "caption": "young girl holding a baseball bat in a field"}, {"image_id": 303590, "caption": "woman holding a hot dog standing in front of a hot dog"}, {"image_id": 331692, "caption": "pizza that is sitting on top of a plate"}, {"image_id": 191738, "caption": "black and white cat laying on top of a lush green field"}, {"image_id": 412914, "caption": "person riding a wave in the water"}, {"image_id": 114549, "caption": "bunch of bananas are standing in front of a tree"}, {"image_id": 520727, "caption": "group of zebra standing on top of a lush green field"}, {"image_id": 185036, "caption": "group of people standing on top of a snow covered slope"}, {"image_id": 438723, "caption": "man sitting on top of a motorcycle"}, {"image_id": 180135, "caption": "man is holding a slice of cake"}, {"image_id": 504304, "caption": "young girl holding a tennis racquet on a tennis court holding a tennis racket"}, {"image_id": 459374, "caption": "yellow fire hydrant sitting in front of a building"}, {"image_id": 328421, "caption": "herd of elephants walking through a field"}, {"image_id": 388325, "caption": "fire hydrant in front of a street sign in front of a building"}, {"image_id": 410337, "caption": "man holding a cell phone"}, {"image_id": 370043, "caption": "baby elephant standing next to each other in the background"}, {"image_id": 129942, "caption": "small bird sitting on top of a tree branch"}, {"image_id": 150675, "caption": "cat is sitting on top of a wall"}, {"image_id": 382088, "caption": "horse that is standing next to a fence"}, {"image_id": 331807, "caption": "bowl of bananas sitting on top of a wooden table"}, {"image_id": 38029, "caption": "group of people sitting on a city street"}, {"image_id": 231163, "caption": "train is parked in front of a train station"}, {"image_id": 578498, "caption": "man sitting on a bed in front of a living room"}, {"image_id": 403020, "caption": "zebra is standing next to a giraffe"}, {"image_id": 178592, "caption": "herd of elephants standing on top of a lush green field"}, {"image_id": 479057, "caption": "herd of sheep grazing on a lush green grass covered field"}, {"image_id": 330835, "caption": "man sitting on top of a skateboard"}, {"image_id": 394240, "caption": "man riding a motorcycle on a city street"}, {"image_id": 199551, "caption": "man sitting on top of a bed"}, {"image_id": 40881, "caption": "close up of a plate of food on a white plate"}, {"image_id": 399462, "caption": "young boy holding a frisbee in a field with a kite in a park"}, {"image_id": 527023, "caption": "man swinging a tennis racket on a tennis court holding a tennis racquet on a baseball field"}, {"image_id": 147980, "caption": "group of people playing a game of frisbee in a field"}, {"image_id": 10694, "caption": "black and white cat sitting on top of a table"}, {"image_id": 385633, "caption": "bathroom that is sitting on top of a kitchen"}, {"image_id": 10442, "caption": "group of baseball players are standing in a baseball game"}, {"image_id": 578500, "caption": "living room filled with lots of chairs in a living room"}, {"image_id": 523772, "caption": "plate of donuts sitting on top of a plate"}, {"image_id": 102843, "caption": "large boat that is on the beach"}, {"image_id": 309852, "caption": "cat sitting on top of a teddy bear"}, {"image_id": 209604, "caption": "group of people standing in front of a city street"}, {"image_id": 316795, "caption": "man riding a horse on a sunny day"}, {"image_id": 503311, "caption": "man flying a kite in the water holding a kite in the air"}, {"image_id": 256668, "caption": "white toilet sitting next to a small bathroom"}, {"image_id": 301867, "caption": "group of people standing in the rain holding umbrellas"}, {"image_id": 148358, "caption": "group of giraffes standing next to each other in a grassy field"}, {"image_id": 95786, "caption": "white toilet sitting on top of a table"}, {"image_id": 459912, "caption": "group of motorcycles are parked in front of a city street"}, {"image_id": 515445, "caption": "group of people standing on the beach"}, {"image_id": 291380, "caption": "man sitting on top of a bench in front of a building"}, {"image_id": 5754, "caption": "close up of donuts that are sitting on a table"}, {"image_id": 353136, "caption": "man sitting in front of a dog"}, {"image_id": 48636, "caption": "baseball player is holding a bat in a baseball game"}, {"image_id": 132615, "caption": "baseball player is holding a bat in a baseball game"}, {"image_id": 296098, "caption": "large clock tower on top of a building"}, {"image_id": 556462, "caption": "close up of a pizza sitting on top of a plate"}, {"image_id": 522778, "caption": "group of people standing on top of a skateboard"}, {"image_id": 366141, "caption": "living room with a flat screen tv"}, {"image_id": 78371, "caption": "man riding a skateboard on top of a baseball field"}, {"image_id": 322864, "caption": "black and white cat sitting on top of a computer desk"}, {"image_id": 105975, "caption": "baby elephant standing in the grass"}, {"image_id": 319865, "caption": "cat laying on top of a black and white dog sitting on top of a couch"}, {"image_id": 296303, "caption": "cat is sitting in front of a window"}, {"image_id": 187743, "caption": "little girl brushing her teeth"}, {"image_id": 179558, "caption": "giraffe standing in front of a fence"}, {"image_id": 216273, "caption": "teddy bear sitting on top of a bed"}, {"image_id": 109092, "caption": "man in a suit and tie"}, {"image_id": 410533, "caption": "black and white cat sitting on top of a bed"}, {"image_id": 224757, "caption": "group of people standing on a park bench"}, {"image_id": 319908, "caption": "group of people standing in front of a baseball game"}, {"image_id": 354540, "caption": "close up of a laptop computer sitting on top of a table"}, {"image_id": 153570, "caption": "man riding a wave on a surfboard in the ocean"}, {"image_id": 184791, "caption": "vase that is sitting on top of a table next to a glass of wine"}, {"image_id": 161470, "caption": "bird sitting on top of a lush green field"}, {"image_id": 329323, "caption": "group of people standing next to each other in the background"}, {"image_id": 365366, "caption": "man is holding a tennis racket"}, {"image_id": 3084, "caption": "clock tower with a clock on top of a building"}, {"image_id": 299716, "caption": "close up of a teddy bear sitting on top of it"}, {"image_id": 332654, "caption": "yellow fire hydrant in front of a parking meter on the side of a street"}, {"image_id": 476005, "caption": "double decker bus is parked in front of a building"}, {"image_id": 180447, "caption": "group of zebras grazing on a lush green field"}, {"image_id": 562614, "caption": "group of people playing a game of a soccer ball on a lush green field"}, {"image_id": 522418, "caption": "black and white photo of a bathroom with a mirror"}, {"image_id": 450500, "caption": "group of people walking down a street with an umbrella"}, {"image_id": 310302, "caption": "pizza sitting on top of a table"}, {"image_id": 84982, "caption": "stop sign on a pole next to a traffic light"}, {"image_id": 140007, "caption": "man standing on top of a tennis court"}, {"image_id": 369763, "caption": "group of people standing in front of a street"}, {"image_id": 78915, "caption": "group of people standing on a tennis court holding a tennis racket"}, {"image_id": 503005, "caption": "man standing on top of a lush green field"}, {"image_id": 529917, "caption": "group of people standing next to each other in the background"}, {"image_id": 456496, "caption": "man riding a skateboard on top of a bench"}, {"image_id": 440329, "caption": "group of people that are parked in front of a road"}, {"image_id": 448365, "caption": "man riding a skate board doing a trick on a skateboard in the air"}, {"image_id": 580041, "caption": "black and white photo of a street sign with a traffic light"}, {"image_id": 197461, "caption": "group of sheep standing on top of a lush green field"}, {"image_id": 310391, "caption": "man riding a motorcycle parked on the side of a truck"}, {"image_id": 395405, "caption": "living room with a couch next to a window in front of a wooden table"}, {"image_id": 210103, "caption": "plate of food on a table next to a birthday cake"}, {"image_id": 159537, "caption": "large clock tower with a clock on top of a building"}, {"image_id": 331352, "caption": "bathroom with a toilet and a sink"}, {"image_id": 310325, "caption": "black and white cat sitting on top of a toilet"}, {"image_id": 336182, "caption": "woman sitting at a table in a living room"}, {"image_id": 362368, "caption": "man sitting at a table with a plate of food"}, {"image_id": 239274, "caption": "couple of boats docked in the water"}, {"image_id": 336587, "caption": "stop sign sitting on a pole in front of a building"}, {"image_id": 19358, "caption": "close up of a pizza sitting on top of a plate"}, {"image_id": 82680, "caption": "dog is standing in the grass"}, {"image_id": 54796, "caption": "young girl playing tennis on a tennis court holding a tennis racket on a tennis court"}, {"image_id": 120935, "caption": "group of zebras are standing next to each other"}, {"image_id": 117125, "caption": "black and white photo of a train traveling down the tracks"}, {"image_id": 175417, "caption": "couple of a bench sitting on top of a lush green field"}, {"image_id": 289173, "caption": "plane that is parked on top of a runway"}, {"image_id": 16497, "caption": "man standing on top of a skateboard"}, {"image_id": 406932, "caption": "dog is standing in front of a car mirror"}, {"image_id": 15827, "caption": "group of motorcycles parked in front of a parking lot"}, {"image_id": 486203, "caption": "herd of horses standing in a grassy field"}, {"image_id": 383419, "caption": "young boy sitting on a table holding a hot dog"}, {"image_id": 125997, "caption": "stop sign in front of a street sign"}, {"image_id": 406404, "caption": "man riding a skateboard at a skate park bench in front of a snow covered slope"}, {"image_id": 400573, "caption": "man eating a hot dog on a cell phone"}, {"image_id": 289610, "caption": "group of people standing in front of a stuffed teddy bear sitting on top of a table"}, {"image_id": 65227, "caption": "man in a field holding a frisbee in a park"}, {"image_id": 181962, "caption": "group of people riding horses on the beach"}, {"image_id": 239448, "caption": "clock tower with a clock on the side of a building"}, {"image_id": 293554, "caption": "man is sitting on top of it"}, {"image_id": 308441, "caption": "man standing on a tennis court holding a tennis racket at a baseball game"}, {"image_id": 266041, "caption": "man wearing a tie standing in front of a cell phone"}, {"image_id": 381106, "caption": "man flying a kite on the beach playing a game of water"}, {"image_id": 58915, "caption": "red and white photo of a building with a clock tower in front of a city street"}, {"image_id": 195829, "caption": "group of people playing a game of frisbee in a field with a soccer ball"}, {"image_id": 502090, "caption": "blue airplane flying a kite in the air"}, {"image_id": 76292, "caption": "man holding a frisbee in the air"}, {"image_id": 338327, "caption": "close up of a clock sitting on top of a table"}, {"image_id": 236189, "caption": "man in a red shirt holding a frisbee in a field"}, {"image_id": 75051, "caption": "white toilet sitting next to a small bathroom"}, {"image_id": 503772, "caption": "kitchen with white cabinets and stainless steel appliances"}, {"image_id": 189550, "caption": "black and white photo of a person standing next to each other"}, {"image_id": 95022, "caption": "close up of a bird perched on top of a tree branch"}, {"image_id": 554114, "caption": "man standing in front of a street"}, {"image_id": 169802, "caption": "little girl sitting in front of a birthday cake"}, {"image_id": 241691, "caption": "black and white photo of a man standing next to each other"}, {"image_id": 348881, "caption": "white toilet sitting on top of a bathroom"}, {"image_id": 175831, "caption": "group of motorcycles parked in front of a building in front of a city street"}, {"image_id": 371999, "caption": "clock tower with a clock on the side of a building"}, {"image_id": 353968, "caption": "man sitting on top of a bed"}, {"image_id": 129739, "caption": "man sitting in front of a living room with a laptop computer"}, {"image_id": 565797, "caption": "black and white cat sitting on top of a car"}, {"image_id": 66800, "caption": "herd of elephants standing in front of a lush green field"}, {"image_id": 199951, "caption": "man riding a skateboard on a boat in the water"}, {"image_id": 426578, "caption": "man riding a wave on a surfboard in the ocean"}, {"image_id": 183757, "caption": "group of people standing next to each other"}, {"image_id": 217005, "caption": "bird perched on top of a tree branch"}, {"image_id": 39743, "caption": "woman holding a stuffed teddy bear sitting on a table"}, {"image_id": 261757, "caption": "pizza that is sitting on top of a wooden table"}, {"image_id": 190767, "caption": "herd of cows that are standing in a grassy field"}, {"image_id": 495107, "caption": "man eating a hot dog on a cell phone"}, {"image_id": 549399, "caption": "black and white motorcycle parked in front of a building"}, {"image_id": 505099, "caption": "baseball player swinging a bat at the ball during a baseball game"}, {"image_id": 204805, "caption": "black and white photo of people that are parked in front of a building"}, {"image_id": 128939, "caption": "traffic light on the side of a car"}, {"image_id": 420775, "caption": "man riding a snowboard down the side of a snow covered slope"}, {"image_id": 192440, "caption": "living room is sitting on top of a wooden table"}, {"image_id": 324383, "caption": "black and white cat laying on top of a bed"}, {"image_id": 296649, "caption": "group of people riding bikes down a city street"}, {"image_id": 397773, "caption": "close up of a plate of food on a table"}, {"image_id": 8418, "caption": "close up of a laptop computer sitting on a table"}, {"image_id": 154202, "caption": "laptop computer sitting on top of a desk"}, {"image_id": 10142, "caption": "man riding skis down a snow covered slope"}, {"image_id": 49559, "caption": "group of people standing in the air on the beach flying a kite in the ocean"}, {"image_id": 569768, "caption": "young man holding a frisbee in front of a baseball bat"}, {"image_id": 98590, "caption": "group of people standing on top of a snow covered slope"}, {"image_id": 356427, "caption": "man is riding a motorcycle down the road"}, {"image_id": 356298, "caption": "train is parked in front of a lush green field"}, {"image_id": 228144, "caption": "living room with a bed next to a window"}, {"image_id": 340559, "caption": "bathroom with a sink in front of a window"}, {"image_id": 153563, "caption": "man sitting on a cell phone in front of a laptop computer"}, {"image_id": 511321, "caption": "large body of boats docked in the water"}, {"image_id": 49633, "caption": "yellow and white train is parked in front of a lush green grass covered field"}, {"image_id": 461885, "caption": "man standing on a lush green field"}, {"image_id": 69911, "caption": "stop sign on a pole next to a traffic light"}, {"image_id": 579664, "caption": "bunch of oranges sitting on a table"}, {"image_id": 444444, "caption": "dog that is sitting on the back of a road"}, {"image_id": 530212, "caption": "red fire hydrant sitting on top of a table"}, {"image_id": 373591, "caption": "black and white bird sitting on top of a tree branch"}, {"image_id": 261779, "caption": "man holding a skateboard on top of a tennis court"}, {"image_id": 346207, "caption": "black and white cat laying on top of a desk in front of a computer desk"}, {"image_id": 52759, "caption": "plane that is parked on a runway"}, {"image_id": 384213, "caption": "man sitting on top of a table"}, {"image_id": 436795, "caption": "man holding a wii video game controller"}, {"image_id": 374628, "caption": "living room with a kitchen with a wooden table"}, {"image_id": 458153, "caption": "baby elephant standing on top of a lush green field"}, {"image_id": 244575, "caption": "brown bear standing next to a horse"}, {"image_id": 509822, "caption": "black and white bike parked in front of a car"}, {"image_id": 282225, "caption": "young man playing tennis on a tennis court holding a tennis racquet on a field"}, {"image_id": 348654, "caption": "close up of a laptop computer sitting on top of a cell phone"}, {"image_id": 562870, "caption": "person sitting on top of a cell phone"}, {"image_id": 328757, "caption": "young boy sitting at a table with a plate of food"}, {"image_id": 286994, "caption": "herd of elephants are standing in the water"}, {"image_id": 267571, "caption": "close up of giraffe standing next to each other"}, {"image_id": 490923, "caption": "man brushing his teeth on her cell phone"}, {"image_id": 457754, "caption": "group of men standing in front of wine"}, {"image_id": 204049, "caption": "man riding a snowboard down a hill in front of a snow covered slope"}, {"image_id": 573349, "caption": "group of people walking down a street with an umbrella"}, {"image_id": 233771, "caption": "man standing in front of a city street"}, {"image_id": 452684, "caption": "close up of a bowl of oranges sitting on a table"}, {"image_id": 227125, "caption": "street sign sitting in front of a traffic light"}, {"image_id": 289899, "caption": "group of people standing on top of a body of water"}, {"image_id": 72944, "caption": "black and white cat is sitting in front of a window"}, {"image_id": 144053, "caption": "street sign in front of a building"}, {"image_id": 476280, "caption": "man holding a surfboard on the beach"}, {"image_id": 146738, "caption": "living room with flowers in front of a wooden table"}, {"image_id": 468604, "caption": "baby elephant standing in front of a lush green field"}, {"image_id": 553442, "caption": "man sitting on top of a bed"}, {"image_id": 300138, "caption": "vase of flowers sitting on top of a table"}, {"image_id": 570045, "caption": "street sign on a pole next to a building"}, {"image_id": 353830, "caption": "pizza that is sitting on top of a table"}, {"image_id": 531854, "caption": "group of birds swimming in a body of water"}, {"image_id": 461063, "caption": "woman standing on a tennis court holding a tennis racket on a tennis court"}, {"image_id": 137658, "caption": "man sitting on a cell phone"}, {"image_id": 562261, "caption": "man holding a frisbee in a field"}, {"image_id": 340175, "caption": "living room with a lot of furniture"}, {"image_id": 404243, "caption": "bunch of donuts sitting on top of a grill in front of a display case"}, {"image_id": 426175, "caption": "man sitting on a bench in front of a building"}, {"image_id": 387173, "caption": "boat that is sitting on top of a body of water"}, {"image_id": 77123, "caption": "plane that is sitting on top of a runway"}, {"image_id": 335578, "caption": "young boy holding a tennis racquet on a tennis court holding a baseball bat at a ball"}, {"image_id": 385786, "caption": "living room with a couch sitting on a bed"}, {"image_id": 493652, "caption": "living room with a christmas tree in front of a living room"}, {"image_id": 476258, "caption": "man riding a skateboard down the side of a street"}, {"image_id": 146193, "caption": "man riding a wave on top of a beach"}, {"image_id": 284220, "caption": "woman holding a tennis racket on a tennis court holding a tennis racquet on a baseball field"}, {"image_id": 165133, "caption": "man standing in front of a street"}, {"image_id": 471009, "caption": "man flying a kite on the beach flying kites in the air"}, {"image_id": 379977, "caption": "black and white horse is standing on top of a lush green field"}, {"image_id": 133698, "caption": "pizza sitting on top of a pan on a table"}, {"image_id": 353807, "caption": "man riding a motorcycle down a street"}, {"image_id": 113588, "caption": "group of people sitting at a desk in front of a laptop computer"}, {"image_id": 193565, "caption": "man sitting at a table eating a hot dog"}, {"image_id": 372979, "caption": "black and white cat sitting in front of a window"}, {"image_id": 574769, "caption": "group of people standing in a living room"}, {"image_id": 576939, "caption": "pair of scissors hanging on the side of a stop sign"}, {"image_id": 235914, "caption": "group of zebras are standing in a dirt field"}, {"image_id": 37038, "caption": "group of people standing on a beach"}, {"image_id": 311553, "caption": "plate of food on a table"}, {"image_id": 399839, "caption": "man holding a teddy bear sitting on a table"}, {"image_id": 453926, "caption": "white plate topped with a piece of pizza sitting on top of a table"}, {"image_id": 87199, "caption": "man riding a horse in a field"}, {"image_id": 414560, "caption": "black and white cat sitting on top of a lush green field"}, {"image_id": 387431, "caption": "black and white photo of a city street in front of a building"}, {"image_id": 376959, "caption": "young boy brushing his teeth in front of a bed"}, {"image_id": 14990, "caption": "group of elephants walking down a dirt road"}, {"image_id": 163020, "caption": "flock of birds flying in the water"}, {"image_id": 39956, "caption": "living room with a chair in the living room"}, {"image_id": 318671, "caption": "man standing on top of a tennis court"}, {"image_id": 476975, "caption": "group of giraffes standing next to each other"}, {"image_id": 321107, "caption": "man riding a skateboard down a street"}, {"image_id": 451468, "caption": "cat sitting on top of a table in front of a living room"}, {"image_id": 571034, "caption": "group of people sitting on a lush green field"}, {"image_id": 550627, "caption": "plate of food that are sitting on a table"}, {"image_id": 127474, "caption": "group of people standing next to a woman holding a baseball bat"}, {"image_id": 8665, "caption": "bird sitting on top of a branch of water"}, {"image_id": 37907, "caption": "black and white cat sitting on top of a tree"}, {"image_id": 205103, "caption": "black and white cat sitting on top of a bed"}, {"image_id": 342006, "caption": "large clock tower on a cloudy day"}, {"image_id": 467522, "caption": "group of zebras grazing on a lush green field"}, {"image_id": 219514, "caption": "traffic light with a stop sign on the side of a street"}, {"image_id": 421478, "caption": "man in a suit and tie holding a cell phone"}, {"image_id": 472795, "caption": "cow standing next to a horse"}, {"image_id": 37777, "caption": "black and white photo of a living room"}, {"image_id": 386912, "caption": "group of people standing in front of a living room with a laptop computer"}, {"image_id": 309120, "caption": "man in a field holding a frisbee in the air"}, {"image_id": 466211, "caption": "baseball player is holding a bat at a baseball game"}, {"image_id": 80671, "caption": "person riding a snowboard down the side of a snow covered slope"}, {"image_id": 134778, "caption": "group of birds flying in the water"}, {"image_id": 391735, "caption": "yellow fire hydrant in front of a city street"}, {"image_id": 375840, "caption": "close up of a bowl of bananas sitting on a table"}, {"image_id": 29913, "caption": "red fire hydrant sitting on top of a motorcycle"}, {"image_id": 554348, "caption": "man standing on top of a bench in front of a street"}, {"image_id": 231339, "caption": "woman sitting at a table with food in a kitchen"}, {"image_id": 480726, "caption": "group of people standing on a tennis court with a tennis ball during a baseball game"}, {"image_id": 32907, "caption": "black and white fire hydrant sitting on top of a wooden bench"}, {"image_id": 45864, "caption": "group of kids playing a game of soccer on a lush green field"}, {"image_id": 41687, "caption": "man riding a snowboard down the side of a snow covered slope"}, {"image_id": 384553, "caption": "man riding a dog in the water"}, {"image_id": 79380, "caption": "group of people sitting on a park bench in front of a lush green field"}, {"image_id": 276580, "caption": "group of people standing in front of a baseball game"}, {"image_id": 360772, "caption": "bathroom that is sitting on top of a toilet"}, {"image_id": 463633, "caption": "stop sign with a stop sign on the street"}, {"image_id": 383137, "caption": "man riding a skateboard on a skate board"}, {"image_id": 97434, "caption": "group of motorcycles parked in front of a city street"}, {"image_id": 239347, "caption": "man sitting on top of a bed"}, {"image_id": 240727, "caption": "woman sitting on a bed"}, {"image_id": 110196, "caption": "group of people standing in front of a city street"}, {"image_id": 520433, "caption": "train is parked in front of a train station"}, {"image_id": 91227, "caption": "giraffe standing on top of a lush green field"}, {"image_id": 542510, "caption": "truck parked in front of a lush green field"}, {"image_id": 440646, "caption": "man sitting on a bed"}, {"image_id": 383384, "caption": "baseball player swinging a bat in front of a baseball game"}, {"image_id": 155885, "caption": "black and white boat is sitting on top of a beach"}, {"image_id": 550426, "caption": "vase of flowers sitting on top of a table"}, {"image_id": 341393, "caption": "dog laying on top of a lush green field"}, {"image_id": 57597, "caption": "group of men playing a game of soccer on a field with a baseball bat"}, {"image_id": 53015, "caption": "woman sitting at a table holding a pizza on a table"}, {"image_id": 410002, "caption": "pizza sitting on top of a table"}, {"image_id": 542959, "caption": "kitchen with a microwave oven"}, {"image_id": 75748, "caption": "train traveling down the tracks in the water"}, {"image_id": 270721, "caption": "bathroom with a white toilet and a sink and a mirror"}, {"image_id": 540174, "caption": "young boy eating a hot dog in his mouth"}, {"image_id": 317595, "caption": "group of people sitting on a table"}, {"image_id": 90935, "caption": "living room with a bed next to a window"}, {"image_id": 550444, "caption": "group of people playing a game of soccer on a field"}, {"image_id": 360926, "caption": "man holding a tennis racket on a tennis court holding a tennis racquet on a baseball field"}, {"image_id": 50125, "caption": "double decker bus driving down a parking lot"}, {"image_id": 132686, "caption": "man flying a kite in the background"}, {"image_id": 16228, "caption": "man riding a horse drawn carriage in front of a street"}, {"image_id": 41550, "caption": "man sitting on top of a computer desk"}, {"image_id": 383445, "caption": "close up of a close up of a donut sitting on a table"}, {"image_id": 246199, "caption": "bunch of fruits and vegetables on a table"}, {"image_id": 111076, "caption": "bathroom with a white toilet and a sink and a mirror"}, {"image_id": 242611, "caption": "bathroom with a sink and a mirror"}, {"image_id": 518586, "caption": "black and white photo of a train traveling down the tracks in front of a train station"}, {"image_id": 78522, "caption": "herd of sheep grazing on a lush green grass covered field"}, {"image_id": 315601, "caption": "group of people walking down a busy city street"}, {"image_id": 337264, "caption": "man standing in front of a kitchen"}, {"image_id": 331162, "caption": "laptop computer sitting on top of a table in front of a living room"}, {"image_id": 535292, "caption": "herd of sheep standing on top of a lush green field"}, {"image_id": 27235, "caption": "man flying a kite in the sky"}, {"image_id": 452224, "caption": "small bird perched on top of a tree branch"}, {"image_id": 207264, "caption": "herd of elephants are standing in a dirt field"}, {"image_id": 437370, "caption": "black and white airplane sitting on top of a lush green field"}, {"image_id": 427135, "caption": "group of people standing on a tennis court holding a baseball bat"}, {"image_id": 419281, "caption": "group of boats that are sitting on a boat in the water"}, {"image_id": 109798, "caption": "hot dog sitting on top of a plate of food"}, {"image_id": 349737, "caption": "desk with a laptop computer sitting on top of a table"}, {"image_id": 121745, "caption": "close up of a bowl of oranges sitting on a table"}, {"image_id": 229643, "caption": "black and white photo of a bathroom with a large window"}, {"image_id": 245453, "caption": "man flying a kite in the sky"}, {"image_id": 213086, "caption": "laptop computer sitting on top of a kitchen counter"}, {"image_id": 370677, "caption": "man standing in front of a store"}, {"image_id": 455483, "caption": "bathroom with a sink and a mirror"}, {"image_id": 52644, "caption": "parking meter sitting on top of a beach"}, {"image_id": 461953, "caption": "close up of a hot dog is sitting on a table"}, {"image_id": 31255, "caption": "black and white dog is standing in front of an elephant"}, {"image_id": 394892, "caption": "young boy riding a skateboard on a baseball field"}, {"image_id": 463836, "caption": "man riding a skateboard down a street"}, {"image_id": 336493, "caption": "man riding a bike on a baseball field"}, {"image_id": 484816, "caption": "yellow airplane flying through the sky"}, {"image_id": 269394, "caption": "close up of a bus that is parked in a parking lot"}, {"image_id": 77187, "caption": "brown bear standing in front of a tree"}, {"image_id": 511204, "caption": "man standing in front of an elephant"}, {"image_id": 562121, "caption": "giraffe standing on top of a lush green field"}, {"image_id": 457691, "caption": "street sign with a clock tower in front of a building"}, {"image_id": 79472, "caption": "laptop computer sitting on top of a desk next to a wooden table"}, {"image_id": 54088, "caption": "red motorcycle parked in front of a city street"}, {"image_id": 559665, "caption": "man riding a motorcycle down a street"}, {"image_id": 443591, "caption": "zebra standing on top of a lush green field"}, {"image_id": 477010, "caption": "baseball player swinging a bat on a tennis court holding a tennis racket"}, {"image_id": 474028, "caption": "group of young children playing a game of frisbee in a field"}, {"image_id": 294832, "caption": "bathroom with a sink and a mirror"}, {"image_id": 304305, "caption": "group of elephants standing next to each other in a grassy field"}, {"image_id": 400, "caption": "black and white dog sitting on top of a wall"}, {"image_id": 196053, "caption": "woman standing on a tennis court holding a tennis racket on a tennis court"}, {"image_id": 437218, "caption": "living room with a television sitting on top of a desk"}, {"image_id": 207826, "caption": "woman holding a laptop on her cell phone"}, {"image_id": 232894, "caption": "young boy sitting on a table holding a cell phone"}, {"image_id": 99734, "caption": "boat that is sitting on top of a body of water"}, {"image_id": 242139, "caption": "city street with a traffic light on the side of a building"}, {"image_id": 561100, "caption": "plane that is parked on top of a lush green field"}, {"image_id": 544519, "caption": "young boy brushing his teeth with a toothbrush in her hand"}, {"image_id": 263136, "caption": "man standing on top of a field"}, {"image_id": 300814, "caption": "group of people riding horses down a street"}, {"image_id": 2684, "caption": "group of zebras standing next to each other in a field"}, {"image_id": 65267, "caption": "living room with a television sitting on top of a table"}, {"image_id": 53542, "caption": "red and white photo of a building with a clock tower in front of a city street"}, {"image_id": 231408, "caption": "black and white bird sitting on top of a lush green field"}, {"image_id": 303298, "caption": "man standing on a tennis court holding a tennis racket at a tennis ball"}, {"image_id": 351683, "caption": "baseball player swinging a bat at a home plate"}, {"image_id": 472246, "caption": "close up of a bowl of oranges sitting on a table"}, {"image_id": 569839, "caption": "man sitting at a table with a plate of food"}, {"image_id": 246809, "caption": "boat that is sitting on top of a body of water"}, {"image_id": 252219, "caption": "group of people standing in front of a building"}, {"image_id": 553162, "caption": "black bear standing next to a tree in the woods"}] -------------------------------------------------------------------------------- /example/coco_eval_example.py: -------------------------------------------------------------------------------- 1 | from pycocotools.coco import COCO 2 | from pycocoevalcap.eval import COCOEvalCap 3 | 4 | annotation_file = 'captions_val2014.json' 5 | results_file = 'captions_val2014_fakecap_results.json' 6 | 7 | # create coco object and coco_result object 8 | coco = COCO(annotation_file) 9 | coco_result = coco.loadRes(results_file) 10 | 11 | # create coco_eval object by taking coco and coco_result 12 | coco_eval = COCOEvalCap(coco, coco_result) 13 | 14 | # evaluate on a subset of images by setting 15 | # coco_eval.params['image_id'] = coco_result.getImgIds() 16 | # please remove this line when evaluating the full validation set 17 | coco_eval.params['image_id'] = coco_result.getImgIds() 18 | 19 | # evaluate results 20 | # SPICE will take a few minutes the first time, but speeds up due to caching 21 | coco_eval.evaluate() 22 | 23 | # print output evaluation scores 24 | for metric, score in coco_eval.eval.items(): 25 | print(f'{metric}: {score:.3f}') 26 | -------------------------------------------------------------------------------- /license.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015, Xinlei Chen, Hao Fang, Tsung-Yi Lin, and Ramakrishna Vedantam 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | 1. Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 2. Redistributions in binary form must reproduce the above copyright notice, 10 | this list of conditions and the following disclaimer in the documentation 11 | and/or other materials provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 14 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 15 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 16 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR 17 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 18 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 19 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 20 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 21 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 22 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 23 | 24 | The views and conclusions contained in the software and documentation are those 25 | of the authors and should not be interpreted as representing official policies, 26 | either expressed or implied, of the FreeBSD Project. 27 | -------------------------------------------------------------------------------- /meteor/data/paraphrase-en.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/meteor/data/paraphrase-en.gz -------------------------------------------------------------------------------- /meteor/meteor-1.5.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/meteor/meteor-1.5.jar -------------------------------------------------------------------------------- /meteor/meteor.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Python wrapper for METEOR implementation, by Xinlei Chen 4 | # Acknowledge Michael Denkowski for the generous discussion and help 5 | 6 | import os 7 | import sys 8 | import subprocess 9 | import threading 10 | 11 | # Assumes meteor-1.5.jar is in the same directory as meteor.py. Change as needed. 12 | METEOR_JAR = 'meteor-1.5.jar' 13 | # print METEOR_JAR 14 | 15 | class Meteor: 16 | 17 | def __init__(self): 18 | self.meteor_cmd = ['java', '-jar', '-Xmx2G', METEOR_JAR, \ 19 | '-', '-', '-stdio', '-l', 'en', '-norm'] 20 | self.meteor_p = subprocess.Popen(self.meteor_cmd, \ 21 | cwd=os.path.dirname(os.path.abspath(__file__)), \ 22 | stdin=subprocess.PIPE, \ 23 | stdout=subprocess.PIPE, \ 24 | stderr=subprocess.PIPE) 25 | # Used to guarantee thread safety 26 | self.lock = threading.Lock() 27 | 28 | def compute_score(self, gts, res): 29 | assert(gts.keys() == res.keys()) 30 | imgIds = gts.keys() 31 | scores = [] 32 | 33 | eval_line = 'EVAL' 34 | self.lock.acquire() 35 | for i in imgIds: 36 | assert(len(res[i]) == 1) 37 | stat = self._stat(res[i][0], gts[i]) 38 | eval_line += ' ||| {}'.format(stat) 39 | 40 | self.meteor_p.stdin.write('{}\n'.format(eval_line).encode()) 41 | self.meteor_p.stdin.flush() 42 | for i in range(0,len(imgIds)): 43 | scores.append(float(self.meteor_p.stdout.readline().strip())) 44 | score = float(self.meteor_p.stdout.readline().strip()) 45 | self.lock.release() 46 | 47 | return score, scores 48 | 49 | def method(self): 50 | return "METEOR" 51 | 52 | def _stat(self, hypothesis_str, reference_list): 53 | # SCORE ||| reference 1 words ||| reference n words ||| hypothesis words 54 | hypothesis_str = hypothesis_str.replace('|||','').replace(' ',' ') 55 | score_line = ' ||| '.join(('SCORE', ' ||| '.join(reference_list), hypothesis_str)) 56 | self.meteor_p.stdin.write('{}\n'.format(score_line).encode()) 57 | self.meteor_p.stdin.flush() 58 | return self.meteor_p.stdout.readline().decode().strip() 59 | 60 | def _score(self, hypothesis_str, reference_list): 61 | self.lock.acquire() 62 | # SCORE ||| reference 1 words ||| reference n words ||| hypothesis words 63 | hypothesis_str = hypothesis_str.replace('|||','').replace(' ',' ') 64 | score_line = ' ||| '.join(('SCORE', ' ||| '.join(reference_list), hypothesis_str)) 65 | self.meteor_p.stdin.write('{}\n'.format(score_line)) 66 | stats = self.meteor_p.stdout.readline().strip() 67 | eval_line = 'EVAL ||| {}'.format(stats) 68 | # EVAL ||| stats 69 | self.meteor_p.stdin.write('{}\n'.format(eval_line)) 70 | score = float(self.meteor_p.stdout.readline().strip()) 71 | # bug fix: there are two values returned by the jar file, one average, and one all, so do it twice 72 | # thanks for Andrej for pointing this out 73 | score = float(self.meteor_p.stdout.readline().strip()) 74 | self.lock.release() 75 | return score 76 | 77 | def __del__(self): 78 | self.lock.acquire() 79 | self.meteor_p.stdin.close() 80 | self.meteor_p.kill() 81 | self.meteor_p.wait() 82 | self.lock.release() 83 | -------------------------------------------------------------------------------- /rouge/rouge.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # 3 | # File Name : rouge.py 4 | # 5 | # Description : Computes ROUGE-L metric as described by Lin and Hovey (2004) 6 | # 7 | # Creation Date : 2015-01-07 06:03 8 | # Author : Ramakrishna Vedantam 9 | 10 | import numpy as np 11 | import pdb 12 | 13 | def my_lcs(string, sub): 14 | """ 15 | Calculates longest common subsequence for a pair of tokenized strings 16 | :param string : list of str : tokens from a string split using whitespace 17 | :param sub : list of str : shorter string, also split using whitespace 18 | :returns: length (list of int): length of the longest common subsequence between the two strings 19 | 20 | Note: my_lcs only gives length of the longest common subsequence, not the actual LCS 21 | """ 22 | if(len(string)< len(sub)): 23 | sub, string = string, sub 24 | 25 | lengths = [[0 for i in range(0,len(sub)+1)] for j in range(0,len(string)+1)] 26 | 27 | for j in range(1,len(sub)+1): 28 | for i in range(1,len(string)+1): 29 | if(string[i-1] == sub[j-1]): 30 | lengths[i][j] = lengths[i-1][j-1] + 1 31 | else: 32 | lengths[i][j] = max(lengths[i-1][j] , lengths[i][j-1]) 33 | 34 | return lengths[len(string)][len(sub)] 35 | 36 | class Rouge(): 37 | ''' 38 | Class for computing ROUGE-L score for a set of candidate sentences for the MS COCO test set 39 | 40 | ''' 41 | def __init__(self): 42 | # vrama91: updated the value below based on discussion with Hovey 43 | self.beta = 1.2 44 | 45 | def calc_score(self, candidate, refs): 46 | """ 47 | Compute ROUGE-L score given one candidate and references for an image 48 | :param candidate: str : candidate sentence to be evaluated 49 | :param refs: list of str : COCO reference sentences for the particular image to be evaluated 50 | :returns score: int (ROUGE-L score for the candidate evaluated against references) 51 | """ 52 | assert(len(candidate)==1) 53 | assert(len(refs)>0) 54 | prec = [] 55 | rec = [] 56 | 57 | # split into tokens 58 | token_c = candidate[0].split(" ") 59 | 60 | for reference in refs: 61 | # split into tokens 62 | token_r = reference.split(" ") 63 | # compute the longest common subsequence 64 | lcs = my_lcs(token_r, token_c) 65 | prec.append(lcs/float(len(token_c))) 66 | rec.append(lcs/float(len(token_r))) 67 | 68 | prec_max = max(prec) 69 | rec_max = max(rec) 70 | 71 | if(prec_max!=0 and rec_max !=0): 72 | score = ((1 + self.beta**2)*prec_max*rec_max)/float(rec_max + self.beta**2*prec_max) 73 | else: 74 | score = 0.0 75 | return score 76 | 77 | def compute_score(self, gts, res): 78 | """ 79 | Computes Rouge-L score given a set of reference and candidate sentences for the dataset 80 | Invoked by evaluate_captions.py 81 | :param hypo_for_image: dict : candidate / test sentences with "image name" key and "tokenized sentences" as values 82 | :param ref_for_image: dict : reference MS-COCO sentences with "image name" key and "tokenized sentences" as values 83 | :returns: average_score: float (mean ROUGE-L score computed by averaging scores for all the images) 84 | """ 85 | assert(gts.keys() == res.keys()) 86 | imgIds = gts.keys() 87 | 88 | score = [] 89 | for id in imgIds: 90 | hypo = res[id] 91 | ref = gts[id] 92 | 93 | score.append(self.calc_score(hypo, ref)) 94 | 95 | # Sanity check. 96 | assert(type(hypo) is list) 97 | assert(len(hypo) == 1) 98 | assert(type(ref) is list) 99 | assert(len(ref) > 0) 100 | 101 | average_score = np.mean(np.array(score)) 102 | return average_score, np.array(score) 103 | 104 | def method(self): 105 | return "Rouge" 106 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_namespace_packages 2 | 3 | # Prepend pycocoevalcap to package names 4 | package_names = ['pycocoevalcap.'+p for p in find_namespace_packages()] 5 | 6 | with open("README.md", "r") as fh: 7 | readme = fh.read() 8 | 9 | setup( 10 | name='pycocoevalcap', 11 | version=1.2, 12 | maintainer='salaniz', 13 | description="MS-COCO Caption Evaluation for Python 3", 14 | long_description=readme, 15 | long_description_content_type="text/markdown", 16 | url="https://github.com/salaniz/pycocoevalcap", 17 | packages=['pycocoevalcap']+package_names, 18 | package_dir={'pycocoevalcap': '.'}, 19 | package_data={'': ['*.jar', '*.gz']}, 20 | install_requires=['pycocotools>=2.0.2'], 21 | python_requires='>=3' 22 | ) 23 | -------------------------------------------------------------------------------- /spice/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/__init__.py -------------------------------------------------------------------------------- /spice/get_stanford_models.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # This script downloads the Stanford CoreNLP models. 3 | import os 4 | from urllib.request import urlretrieve 5 | from zipfile import ZipFile 6 | 7 | CORENLP = 'stanford-corenlp-full-2015-12-09' 8 | SPICELIB = 'lib' 9 | JAR = 'stanford-corenlp-3.6.0' 10 | SPICEDIR = os.path.dirname(__file__) 11 | 12 | 13 | def print_progress(transferred_blocks, block_size, total_size): 14 | current_mb = transferred_blocks * block_size / 1024 / 1024 15 | total_mb = total_size / 1024 / 1024 16 | percent = current_mb / total_mb 17 | progress_str = "Progress: {:5.1f}M / {:5.1f}M ({:6.1%})" 18 | print(progress_str.format(current_mb, total_mb, percent), end='\r') 19 | 20 | 21 | def get_stanford_models(): 22 | jar_name = os.path.join(SPICEDIR, SPICELIB, '{}.jar'.format(JAR)) 23 | # Only download file if file does not yet exist. Else: do nothing 24 | if not os.path.exists(jar_name): 25 | print('Downloading {} for SPICE ...'.format(JAR)) 26 | url = 'http://nlp.stanford.edu/software/{}.zip'.format(CORENLP) 27 | zip_file, headers = urlretrieve(url, reporthook=print_progress) 28 | print() 29 | print('Extracting {} ...'.format(JAR)) 30 | file_name = os.path.join(CORENLP, JAR) 31 | # file names in zip use '/' separator regardless of OS 32 | zip_file_name = '/'.join([CORENLP, JAR]) 33 | target_name = os.path.join(SPICEDIR, SPICELIB, JAR) 34 | for filef in ['{}.jar', '{}-models.jar']: 35 | ZipFile(zip_file).extract(filef.format(zip_file_name), SPICEDIR) 36 | os.rename(os.path.join(SPICEDIR, filef.format(file_name)), 37 | filef.format(target_name)) 38 | 39 | os.rmdir(os.path.join(SPICEDIR, CORENLP)) 40 | os.remove(zip_file) 41 | print('Done.') 42 | 43 | 44 | if __name__ == '__main__': 45 | # If run as a script, excute inside the spice/ folder. 46 | get_stanford_models() 47 | -------------------------------------------------------------------------------- /spice/lib/Meteor-1.5.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/Meteor-1.5.jar -------------------------------------------------------------------------------- /spice/lib/SceneGraphParser-1.0.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/SceneGraphParser-1.0.jar -------------------------------------------------------------------------------- /spice/lib/ejml-0.23.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/ejml-0.23.jar -------------------------------------------------------------------------------- /spice/lib/fst-2.47.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/fst-2.47.jar -------------------------------------------------------------------------------- /spice/lib/guava-19.0.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/guava-19.0.jar -------------------------------------------------------------------------------- /spice/lib/hamcrest-core-1.3.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/hamcrest-core-1.3.jar -------------------------------------------------------------------------------- /spice/lib/jackson-core-2.5.3.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/jackson-core-2.5.3.jar -------------------------------------------------------------------------------- /spice/lib/javassist-3.19.0-GA.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/javassist-3.19.0-GA.jar -------------------------------------------------------------------------------- /spice/lib/json-simple-1.1.1.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/json-simple-1.1.1.jar -------------------------------------------------------------------------------- /spice/lib/junit-4.12.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/junit-4.12.jar -------------------------------------------------------------------------------- /spice/lib/lmdbjni-0.4.6.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/lmdbjni-0.4.6.jar -------------------------------------------------------------------------------- /spice/lib/lmdbjni-linux64-0.4.6.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/lmdbjni-linux64-0.4.6.jar -------------------------------------------------------------------------------- /spice/lib/lmdbjni-osx64-0.4.6.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/lmdbjni-osx64-0.4.6.jar -------------------------------------------------------------------------------- /spice/lib/lmdbjni-win64-0.4.6.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/lmdbjni-win64-0.4.6.jar -------------------------------------------------------------------------------- /spice/lib/objenesis-2.4.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/objenesis-2.4.jar -------------------------------------------------------------------------------- /spice/lib/slf4j-api-1.7.12.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/slf4j-api-1.7.12.jar -------------------------------------------------------------------------------- /spice/lib/slf4j-simple-1.7.21.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/lib/slf4j-simple-1.7.21.jar -------------------------------------------------------------------------------- /spice/spice-1.0.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/spice/spice-1.0.jar -------------------------------------------------------------------------------- /spice/spice.py: -------------------------------------------------------------------------------- 1 | from __future__ import division 2 | import os 3 | import sys 4 | import subprocess 5 | import threading 6 | import json 7 | import numpy as np 8 | import ast 9 | import tempfile 10 | 11 | from .get_stanford_models import get_stanford_models 12 | 13 | # Assumes spice.jar is in the same directory as spice.py. Change as needed. 14 | SPICE_JAR = 'spice-1.0.jar' 15 | TEMP_DIR = 'tmp' 16 | CACHE_DIR = 'cache' 17 | 18 | class Spice: 19 | """ 20 | Main Class to compute the SPICE metric 21 | """ 22 | 23 | def __init__(self): 24 | get_stanford_models() 25 | 26 | def float_convert(self, obj): 27 | try: 28 | return float(obj) 29 | except: 30 | return np.nan 31 | 32 | def compute_score(self, gts, res): 33 | assert(sorted(gts.keys()) == sorted(res.keys())) 34 | imgIds = sorted(gts.keys()) 35 | 36 | # Prepare temp input file for the SPICE scorer 37 | input_data = [] 38 | for id in imgIds: 39 | hypo = res[id] 40 | ref = gts[id] 41 | 42 | # Sanity check. 43 | assert(type(hypo) is list) 44 | assert(len(hypo) == 1) 45 | assert(type(ref) is list) 46 | assert(len(ref) >= 1) 47 | 48 | input_data.append({ 49 | "image_id" : id, 50 | "test" : hypo[0], 51 | "refs" : ref 52 | }) 53 | 54 | cwd = os.path.dirname(os.path.abspath(__file__)) 55 | temp_dir=os.path.join(cwd, TEMP_DIR) 56 | if not os.path.exists(temp_dir): 57 | os.makedirs(temp_dir) 58 | in_file = tempfile.NamedTemporaryFile(delete=False, dir=temp_dir, 59 | mode='w+') 60 | json.dump(input_data, in_file, indent=2) 61 | in_file.close() 62 | 63 | # Start job 64 | out_file = tempfile.NamedTemporaryFile(delete=False, dir=temp_dir) 65 | out_file.close() 66 | cache_dir=os.path.join(cwd, CACHE_DIR) 67 | if not os.path.exists(cache_dir): 68 | os.makedirs(cache_dir) 69 | spice_cmd = ['java', '-jar', '-Xmx8G', SPICE_JAR, in_file.name, 70 | '-cache', cache_dir, 71 | '-out', out_file.name, 72 | '-subset', 73 | '-silent' 74 | ] 75 | subprocess.check_call(spice_cmd, 76 | cwd=os.path.dirname(os.path.abspath(__file__))) 77 | 78 | # Read and process results 79 | with open(out_file.name) as data_file: 80 | results = json.load(data_file) 81 | os.remove(in_file.name) 82 | os.remove(out_file.name) 83 | 84 | imgId_to_scores = {} 85 | spice_scores = [] 86 | for item in results: 87 | imgId_to_scores[item['image_id']] = item['scores'] 88 | spice_scores.append(self.float_convert(item['scores']['All']['f'])) 89 | average_score = np.mean(np.array(spice_scores)) 90 | scores = [] 91 | for image_id in imgIds: 92 | # Convert none to NaN before saving scores over subcategories 93 | score_set = {} 94 | for category,score_tuple in imgId_to_scores[image_id].items(): 95 | score_set[category] = {k: self.float_convert(v) for k, v in score_tuple.items()} 96 | scores.append(score_set) 97 | return average_score, scores 98 | 99 | def method(self): 100 | return "SPICE" 101 | 102 | 103 | -------------------------------------------------------------------------------- /tokenizer/ptbtokenizer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # 3 | # File Name : ptbtokenizer.py 4 | # 5 | # Description : Do the PTB Tokenization and remove punctuations. 6 | # 7 | # Creation Date : 29-12-2014 8 | # Last Modified : Thu Mar 19 09:53:35 2015 9 | # Authors : Hao Fang and Tsung-Yi Lin 10 | 11 | import os 12 | import sys 13 | import subprocess 14 | import tempfile 15 | import itertools 16 | 17 | # path to the stanford corenlp jar 18 | STANFORD_CORENLP_3_4_1_JAR = 'stanford-corenlp-3.4.1.jar' 19 | 20 | # punctuations to be removed from the sentences 21 | PUNCTUATIONS = ["''", "'", "``", "`", "-LRB-", "-RRB-", "-LCB-", "-RCB-", \ 22 | ".", "?", "!", ",", ":", "-", "--", "...", ";"] 23 | 24 | class PTBTokenizer: 25 | """Python wrapper of Stanford PTBTokenizer""" 26 | def __init__(self, verbose=True): 27 | self.verbose = verbose 28 | 29 | def tokenize(self, captions_for_image): 30 | cmd = ['java', '-cp', STANFORD_CORENLP_3_4_1_JAR, \ 31 | 'edu.stanford.nlp.process.PTBTokenizer', \ 32 | '-preserveLines', '-lowerCase'] 33 | 34 | # ====================================================== 35 | # prepare data for PTB Tokenizer 36 | # ====================================================== 37 | final_tokenized_captions_for_image = {} 38 | image_id = [k for k, v in captions_for_image.items() for _ in range(len(v))] 39 | sentences = '\n'.join([c['caption'].replace('\n', ' ') for k, v in captions_for_image.items() for c in v]) 40 | 41 | # ====================================================== 42 | # save sentences to temporary file 43 | # ====================================================== 44 | path_to_jar_dirname=os.path.dirname(os.path.abspath(__file__)) 45 | tmp_file = tempfile.NamedTemporaryFile(delete=False, dir=path_to_jar_dirname) 46 | tmp_file.write(sentences.encode()) 47 | tmp_file.close() 48 | 49 | # ====================================================== 50 | # tokenize sentence 51 | # ====================================================== 52 | cmd.append(os.path.basename(tmp_file.name)) 53 | if self.verbose: 54 | p_tokenizer = subprocess.Popen(cmd, cwd=path_to_jar_dirname, \ 55 | stdout=subprocess.PIPE) 56 | else: 57 | p_tokenizer = subprocess.Popen(cmd, cwd=path_to_jar_dirname, \ 58 | stdout=subprocess.PIPE, stderr=subprocess.DEVNULL) 59 | token_lines = p_tokenizer.communicate(input=sentences.rstrip())[0] 60 | token_lines = token_lines.decode() 61 | lines = token_lines.split('\n') 62 | # remove temp file 63 | os.remove(tmp_file.name) 64 | 65 | # ====================================================== 66 | # create dictionary for tokenized captions 67 | # ====================================================== 68 | for k, line in zip(image_id, lines): 69 | if not k in final_tokenized_captions_for_image: 70 | final_tokenized_captions_for_image[k] = [] 71 | tokenized_caption = ' '.join([w for w in line.rstrip().split(' ') \ 72 | if w not in PUNCTUATIONS]) 73 | final_tokenized_captions_for_image[k].append(tokenized_caption) 74 | 75 | return final_tokenized_captions_for_image 76 | -------------------------------------------------------------------------------- /tokenizer/stanford-corenlp-3.4.1.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/salaniz/pycocoevalcap/a24f74c408c918f1f4ec34e9514bc8a76ce41ffd/tokenizer/stanford-corenlp-3.4.1.jar --------------------------------------------------------------------------------