├── .gitignore ├── requirements.txt ├── hearst_counts.txt.gz ├── CODE_OF_CONDUCT.md ├── hypernymysuite ├── __init__.py ├── base.py ├── reader.py ├── pattern.py ├── unsup.py └── evaluation.py ├── setup.py ├── CONTRIBUTING.md ├── generate_table.sh ├── main.py ├── README.md ├── download_data.sh ├── compile_table.py ├── LICENSE └── data ├── bibless.tsv └── wbless.tsv /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy 2 | scipy 3 | scikit-learn 4 | nltk 5 | pandas 6 | -------------------------------------------------------------------------------- /hearst_counts.txt.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/facebookresearch/hypernymysuite/HEAD/hearst_counts.txt.gz -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please read the [full text](https://code.fb.com/codeofconduct/)) so that you can understand what actions will and will not be tolerated. 4 | -------------------------------------------------------------------------------- /hypernymysuite/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2017-present, Facebook, Inc. 2 | # All rights reserved. 3 | # 4 | # This source code is licensed under the license found in the 5 | # LICENSE file in the root directory of this source tree. 6 | from __future__ import absolute_import, division, print_function, unicode_literals 7 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright (c) 2017-present, Facebook, Inc. 4 | # All rights reserved. 5 | # 6 | # This source code is licensed under the license found in the 7 | # LICENSE file in the root directory of this source tree. 8 | 9 | import os 10 | from setuptools import setup, find_packages, Extension 11 | import sys 12 | 13 | 14 | with open('README.md') as f: 15 | readme = f.read() 16 | 17 | 18 | setup( 19 | name='hypernymysuite', 20 | version='0.0.1', 21 | description='Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora', 22 | url='https://github.com/facebookresearch/hypernymysuite.git', 23 | classifiers=[ 24 | 'Intended Audience :: Science/Research', 25 | 'Topic :: Scientific/Engineering :: Artificial Intelligence', 26 | ], 27 | long_description=readme, 28 | long_description_content_type='text/markdown', 29 | setup_requires=[ 30 | 'setuptools>=18.0', 31 | ], 32 | install_requires=[ 33 | 'numpy', 34 | 'scipy', 35 | 'scikit-learn', 36 | 'nltk', 37 | 'pandas', 38 | ], 39 | packages=find_packages(), 40 | ) 41 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to HypernymySuite 2 | We want to make contributing to this project as easy and transparent as 3 | possible. 4 | 5 | ## Pull Requests 6 | We actively welcome your pull requests. 7 | 8 | 1. Fork the repo and create your branch from `main`. 9 | 2. If you've added code that should be tested, add tests. 10 | 3. If you've changed APIs, update the documentation. 11 | 4. Ensure the test suite passes. 12 | 5. Make sure your code lints. 13 | 6. If you haven't already, complete the Contributor License Agreement ("CLA"). 14 | 15 | ## Contributor License Agreement ("CLA") 16 | In order to accept your pull request, we need you to submit a CLA. You only need 17 | to do this once to work on any of Facebook's open source projects. 18 | 19 | Complete your CLA here: 20 | 21 | ## Issues 22 | We use GitHub issues to track public bugs. Please ensure your description is 23 | clear and has sufficient instructions to be able to reproduce the issue. 24 | 25 | Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe 26 | disclosure of security bugs. In those cases, please go through the process 27 | outlined on that page and do not file a public issue. 28 | 29 | ## Coding Style 30 | Please use [black](https://github.com/ambv/black) to autoformat all files. 31 | 32 | ## License 33 | By contributing to HypernymySuite, you agree that your contributions will be licensed 34 | under the LICENSE file in the root directory of this source tree. 35 | -------------------------------------------------------------------------------- /generate_table.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Copyright (c) 2017-present, Facebook, Inc. 4 | # All rights reserved. 5 | # 6 | # This source code is licensed under the license found in the 7 | # LICENSE file in the root directory of this source tree. 8 | 9 | # ------------------------------------------------------------------------------- 10 | # This shell script generates the table presented in the paper 11 | # ------------------------------------------------------------------------------- 12 | 13 | # Quit if there's any errors 14 | set -e 15 | 16 | # There must be exactly one argument 17 | if [[ "$#" != "1" ]] 18 | then 19 | echo "Usage: $0 output_filename.json" >&2 20 | echo "You must provide an output filename to store results." >&2 21 | exit 1 22 | fi 23 | 24 | RESULTS_FILE="$1" 25 | DSET="hearst_counts.txt.gz" 26 | 27 | if [[ -f "$RESULTS_FILE" ]] 28 | then 29 | echo "$RESULTS_FILE already exists. Refusing to overwrite it." >&2 30 | echo "Run 'rm $RESULTS_FILE' if this is intentional." >&2 31 | exit 1 32 | fi 33 | 34 | echo "Computing results. Please be very patient." >&2 35 | 36 | # p(x,y) model 37 | python main.py cnt --dset "$DSET" > "$RESULTS_FILE" 38 | # ppmi(x,y) model 39 | python main.py ppmi --dset "$DSET" >> "$RESULTS_FILE" 40 | 41 | for k in 5 10 15 20 25 50 100 150 200 250 300 500 1000 42 | do 43 | # sp(x, y) model 44 | python main.py svdcnt --dset "$DSET" --k "$k" >> "$RESULTS_FILE" 45 | # spmi(x, y) model 46 | python main.py svdppmi --dset "$DSET" --k "$k" >> "$RESULTS_FILE" 47 | done 48 | 49 | # Finally present results 50 | echo "Validation performance:" 51 | python compile_table.py -i "$RESULTS_FILE" 52 | 53 | echo 54 | echo 55 | 56 | echo "Test performance:" 57 | python compile_table.py -i "$RESULTS_FILE" --test 58 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright (c) 2017-present, Facebook, Inc. 4 | # All rights reserved. 5 | # 6 | # This source code is licensed under the license found in the 7 | # LICENSE file in the root directory of this source tree. 8 | 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | import sys 15 | import argparse 16 | import logging 17 | import json 18 | 19 | from hypernymysuite import base 20 | from hypernymysuite import pattern 21 | from hypernymysuite import evaluation 22 | from hypernymysuite import unsup 23 | 24 | 25 | def main(): 26 | logging.basicConfig(level=logging.INFO) 27 | parser = argparse.ArgumentParser() 28 | parser.add_argument("cmd", help="Baseline to run") 29 | parser.add_argument("--dset", help="Corpus of hearst patterns.") 30 | parser.add_argument("--k", default=None, type=int, help="Number of dimensions.") 31 | args = parser.parse_args() 32 | 33 | if args.cmd in {"svdppmi", "svdcnt", "random", "slqs", "slqscos"} and not args.k: 34 | raise parser.error("You must specify --k") 35 | 36 | if args.cmd == "cnt": 37 | model = pattern.RawCountModel(args.dset) 38 | elif args.cmd == "ppmi": 39 | model = pattern.PPMIModel(args.dset) 40 | elif args.cmd == "svdppmi": 41 | model = pattern.SvdPpmiModel(args.dset, k=args.k) 42 | elif args.cmd == "svdcnt": 43 | model = pattern.SvdRawModel(args.dset, k=args.k) 44 | elif args.cmd == "random": 45 | model = base.RandomBaseline(args.dset, k=args.k) 46 | elif args.cmd == "weeds": 47 | model = unsup.UnsupervisedBaseline(args.dset, unsup.weeds_prec) 48 | elif args.cmd == "invcl": 49 | model = unsup.UnsupervisedBaseline(args.dset, unsup.invCL) 50 | elif args.cmd == "slqs": 51 | model = unsup.SLQS(args.dset, args.k) 52 | elif args.cmd == "slqscos": 53 | model = unsup.SLQS_Cos(args.dset, args.k) 54 | elif args.cmd == "cosine": 55 | model = unsup.UnsupervisedBaseline(args.dset, unsup.cosine) 56 | elif args.cmd == "precomputed": 57 | model = unsup.Precomputed(args.dset) 58 | else: 59 | parser.print_help() 60 | sys.exit(1) 61 | 62 | result = evaluation.all_evaluations(model, args) 63 | result["name"] = args.cmd 64 | result["dset"] = args.dset 65 | result["k"] = args.k 66 | print(json.dumps(result)) 67 | 68 | 69 | if __name__ == "__main__": 70 | main() 71 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Hypernymy Suite 2 | 3 | HypernymySuite is a tool for evaluating some hypernymy detection modules. Its 4 | predominant focus is reproducing the results for the following paper. 5 | 6 | > Stephen Roller, Douwe Kiela, and Maximilian Nickel. 2018. Hearst Patterns 7 | > Revisited: Automatic Hypernym Detection from Large Text Corpora. ACL. 8 | > ([arXiv](https://arxiv.org/abs/1806.03191)) 9 | 10 | We hope that open sourcing our evaluation will help facilitate future research. 11 | 12 | ## Example 13 | 14 | You can produce results in a JSON format by calling main.py: 15 | 16 | python main.py cnt --dset hearst_counts.txt.gz 17 | 18 | These results can be made machine readable by piping them into `compile_table`: 19 | 20 | python main.py cnt --dset hearst_counts.txt.gz | python compile_table.py 21 | 22 | To generate the full table from the report, you may simply use `generate_table.sh`: 23 | 24 | bash generate_table.sh results.json 25 | 26 | Please note that due to licensing concerns, we were not able to release our 27 | train/validation/test folds from the paper, so results may differ slightly than 28 | those reported. 29 | 30 | ## Requirements 31 | 32 | The module was developed with python3 in mind, and is not tested for python2. 33 | Nonetheless, cross-platform compatibility may be possible. 34 | 35 | The suite requires several packages you probably already have installed: 36 | `numpy`, `scipy`, `pandas`, `scikit-learn` and `nltk`. These can be installed 37 | using pip: 38 | 39 | pip install -r requirements.txt 40 | 41 | If you've never used `nltk` before, you'll need to install the wordnet module. 42 | 43 | python -c "import nltk; nltk.download('wordnet')" 44 | 45 | On OS X, you may need to install `coreutils` and `gnu-sed` for the script `download_data.sh` to run correctly. These can be installed using brew: 46 | 47 | brew install coreutils gnu-sed 48 | 49 | After installation, you will either need to modify `download_data.sh` to run `gsort` and `gsed` instead of `sort` and `sed`, or alternatively add a "gnubin" directory to your PATH from your bashrc: 50 | 51 | PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH" 52 | 53 | For more information, see `brew info coreutils` or `brew info gnu-sed`. 54 | 55 | ## Evaluating your own model 56 | 57 | You can evaluate your own model in two separate ways. The simplest way is simply 58 | to create a copy of example.tsv, and fill in your model's predictions in the `sim` 59 | column. You must include a prediction for every pair, but you may set the `is_oov` 60 | column to `1` to ensure it is correctly calculated. 61 | 62 | You may then evaluate the model: 63 | 64 | python main.py precomputed --dset example.tsv 65 | 66 | You can also implement any model by extending the `base.HypernymySuiteModel` class 67 | and filling in your own implemenation for `predict` or `predict_many`. 68 | 69 | ## References 70 | 71 | If you find this code useful for your research, please cite the following paper: 72 | 73 | @inproceedings{roller2018hearst 74 | title = {Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora}, 75 | author = {Roller, Stephen and Kiela, Douwe and Nickel, Maximilian}, 76 | year = {2018}, 77 | booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics}, 78 | location = {Melbourne, Australia}, 79 | publisher = {Association for Computational Linguistics} 80 | } 81 | 82 | ## License 83 | 84 | This code is licensed under [CC-BY-NC4.0](https://creativecommons.org/licenses/by-nc/4.0/). 85 | 86 | The data contained in `hearst_counts.txt` was extracted from a combination of 87 | [Wikipedia](https://en.wikipedia.org/wiki/Wikipedia:Database_download) and Gigaword. 88 | Please see publication for details. 89 | -------------------------------------------------------------------------------- /hypernymysuite/base.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright (c) 2017-present, Facebook, Inc. 4 | # All rights reserved. 5 | # 6 | # This source code is licensed under the license found in the 7 | # LICENSE file in the root directory of this source tree. 8 | 9 | """ 10 | Abstract class module which defines the interface for our HypernymySuite model. 11 | """ 12 | 13 | from __future__ import absolute_import 14 | from __future__ import division 15 | from __future__ import print_function 16 | from __future__ import unicode_literals 17 | 18 | import numpy as np 19 | 20 | 21 | class HypernymySuiteModel(object): 22 | """ 23 | Base class for all hypernymy suite models. 24 | 25 | To use this, must implement these methods: 26 | 27 | predict(self, hypo: str, hyper: str): float, which makes a 28 | prediction about two words. 29 | vocab: dict[str, int], which tells if a word is in the 30 | vocabulary. 31 | 32 | Your predict method *must* be prepared to handle OOV terms, but it may 33 | returning any sentinel value you wish. 34 | 35 | You can optionally implement 36 | predict_many(hypo: list[str], hyper: list[str]: array[float] 37 | 38 | The skeleton method here will just call predict() in a for loop, but 39 | some methods can be vectorized for improved performance. This is the 40 | actual method called by the evaluation script. 41 | """ 42 | 43 | vocab = {} 44 | 45 | def __init__(self): 46 | raise NotImplementedError 47 | 48 | def predict(self, hypo, hyper): 49 | """ 50 | Core modeling procedure, estimating the degree to which hypo is_a hyper. 51 | 52 | This is an abstract method, describing the interface. 53 | 54 | Args: 55 | hypo: str. A hypothesized hyponym. 56 | hyper: str. A hypothesized hypernym. 57 | 58 | Returns: 59 | float. The score estimating the degree to which hypo is_a hyper. 60 | Higher values indicate a stronger degree. 61 | """ 62 | raise NotImplementedError 63 | 64 | def predict_many(self, hypos, hypers): 65 | """ 66 | Make predictions for many pairs at the same time. The default 67 | implementation just calls predict() many times, but many models 68 | benefit from vectorization. 69 | 70 | Args: 71 | hypos: list[str]. A list of hypothesized hyponyms. 72 | hypers: list[str]. A list of corresponding hypothesized hypernyms. 73 | """ 74 | result = [] 75 | for x, y in zip(hypos, hypers): 76 | result.append(self.predict(x, y)) 77 | return np.array(result, dtype=np.float32) 78 | 79 | 80 | class Precomputed(HypernymySuiteModel): 81 | """ 82 | A model which uses precomputed prediction, read from a TSV file. 83 | """ 84 | 85 | def __init__(self, precomputed): 86 | self.vocab = {"": 0} 87 | self.lookup = {} 88 | with open(precomputed) as f: 89 | for line in f: 90 | w1, w2, sim, is_oov = line.strip().split("\t") 91 | if w1 == "hypo" and w2 == "hyper": 92 | # header, ignore it 93 | continue 94 | if is_oov == "1" or is_oov.lower() in ("t", "true"): 95 | # Don't read in oov predictions 96 | continue 97 | if w1 not in self.vocab: 98 | self.vocab[w1] = len(self.vocab) 99 | if w2 not in self.vocab: 100 | self.vocab[w2] = len(self.vocab) 101 | sim = float(sim) 102 | self.lookup[(self.vocab[w1], self.vocab[w2])] = sim 103 | 104 | def predict(self, hypo, hyper): 105 | x = self.vocab.get(hypo, 0) 106 | y = self.vocab.get(hyper, 0) 107 | return self.lookup.get((x, y), 0.) 108 | -------------------------------------------------------------------------------- /hypernymysuite/reader.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright (c) 2017-present, Facebook, Inc. 4 | # All rights reserved. 5 | # 6 | # This source code is licensed under the license found in the 7 | # LICENSE file in the root directory of this source tree. 8 | 9 | """ 10 | Utility module for easily reading a sparse matrix. 11 | """ 12 | 13 | from __future__ import absolute_import 14 | from __future__ import division 15 | from __future__ import print_function 16 | from __future__ import unicode_literals 17 | 18 | import numpy as np 19 | import logging 20 | import os 21 | import gzip 22 | import scipy.sparse as sp 23 | 24 | try: 25 | import cPickle as pickle 26 | except ImportError: 27 | import pickle 28 | 29 | 30 | def __try_three_columns(string): 31 | fields = string.split("\t") 32 | if len(fields) > 3: 33 | fields = fields[:3] 34 | if len(fields) == 3: 35 | return fields[0], fields[1], float(fields[2]) 36 | if len(fields) == 2: 37 | return fields[0], fields[1], 1.0 38 | else: 39 | raise ValueError("Invalid number of fields {}".format(len(fields))) 40 | 41 | 42 | def __load_sparse_matrix(filename, same_vocab): 43 | """ 44 | Actual workhorse for loading a sparse matrix. See docstring for 45 | read_sparse_matrix. 46 | 47 | """ 48 | objects = [""] 49 | rowvocab = {"": 0} 50 | if same_vocab: 51 | colvocab = rowvocab 52 | else: 53 | colvocab = {} 54 | _is = [] 55 | _js = [] 56 | _vs = [] 57 | 58 | # Read gzip files 59 | if filename.endswith(".gz"): 60 | f = gzip.open(filename, "r") 61 | else: 62 | f = open(filename, "rb") 63 | 64 | for line in f: 65 | line = line.decode("utf-8") 66 | target, context, weight = __try_three_columns(line) 67 | if target not in rowvocab: 68 | rowvocab[target] = len(rowvocab) 69 | objects.append(target) 70 | if context not in colvocab: 71 | colvocab[context] = len(colvocab) 72 | if same_vocab: 73 | objects.append(context) 74 | 75 | _is.append(rowvocab[target]) 76 | _js.append(colvocab[context]) 77 | _vs.append(weight) 78 | 79 | # clean up 80 | f.close() 81 | 82 | _shape = (len(rowvocab), len(colvocab)) 83 | spmatrix = sp.csr_matrix((_vs, (_is, _js)), shape=_shape, dtype=np.float64) 84 | return spmatrix, objects, rowvocab, colvocab 85 | 86 | 87 | def read_sparse_matrix(filename, allow_binary_cache=False, same_vocab=False): 88 | """ 89 | Reads in a 3 column file as a sparse matrix, where each line (x, y, v) 90 | gives the name of the row x, column y, and the value z. 91 | 92 | If filename ends with .gz, will assume the file is gzip compressed. 93 | 94 | Args: 95 | filename: str. The filename containing sparse matrix in 3-col format. 96 | allow_binary_cache: bool. If true, caches the matrix in a pkl file with 97 | the same filename for faster reads. If cache doesn't exist, will 98 | create it. 99 | same_vocab: bool. Indicates whether rows and columns have the same vocab. 100 | 101 | Returns: 102 | A tuple containing (spmatrix, id2row, row2id, col2id): 103 | spmatrix: a scipy.sparse matrix with the entries 104 | id2row: a list[str] containing the names for the rows of the matrix 105 | row2id: a dict[str,int] mapping words to row indices 106 | col2id: a dict[str,int] mapping words to col indices. If same_vocab, 107 | this is identical to row2id. 108 | """ 109 | # make sure the cache is new enough 110 | cache_filename = filename + ".pkl" 111 | cache_exists = os.path.exists(cache_filename) 112 | cache_fresh = cache_exists and os.path.getmtime(filename) <= os.path.getmtime( 113 | cache_filename 114 | ) 115 | if allow_binary_cache and cache_fresh: 116 | logging.debug("Using space cache {}".format(cache_filename)) 117 | with open(cache_filename + ".pkl", "rb") as pklf: 118 | return pickle.load(pklf) 119 | else: 120 | # binary cache is not allowed, or it's stale 121 | result = __load_sparse_matrix(filename, same_vocab=same_vocab) 122 | if allow_binary_cache: 123 | logging.warning("Dumping the binary cache {}.pkl".format(filename)) 124 | with open(filename + ".pkl", "wb") as pklf: 125 | pickle.dump(result, pklf) 126 | return result 127 | -------------------------------------------------------------------------------- /hypernymysuite/pattern.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright (c) 2017-present, Facebook, Inc. 4 | # All rights reserved. 5 | # 6 | # This source code is licensed under the license found in the 7 | # LICENSE file in the root directory of this source tree. 8 | 9 | """ 10 | Hearst pattern based models. 11 | """ 12 | 13 | from __future__ import absolute_import 14 | from __future__ import division 15 | from __future__ import print_function 16 | from __future__ import unicode_literals 17 | 18 | import numpy as np 19 | import scipy.sparse as sparse 20 | 21 | from .base import HypernymySuiteModel 22 | from .reader import read_sparse_matrix 23 | 24 | 25 | class PatternBasedModel(HypernymySuiteModel): 26 | """ 27 | Basis class for all Hearst-pattern based approaches. 28 | """ 29 | 30 | def __init__(self, patterns_filename): 31 | csr_m, self.id2word, self.vocab, _ = read_sparse_matrix( 32 | patterns_filename, same_vocab=True 33 | ) 34 | self.p_w = csr_m.sum(axis=1).A[:, 0] 35 | self.p_c = csr_m.sum(axis=0).A[0, :] 36 | self.matrix = csr_m.todok() 37 | self.N = self.p_w.sum() 38 | 39 | def predict(self, hypo, hyper): 40 | raise NotImplementedError("Abstract class") 41 | 42 | def __str__(self): 43 | raise NotImplementedError("Abstract class") 44 | 45 | 46 | class RawCountModel(PatternBasedModel): 47 | """ 48 | P(x, y) model which uses raw counts. 49 | """ 50 | 51 | def predict(self, hypo, hyper): 52 | L = self.vocab.get(hypo, 0) 53 | R = self.vocab.get(hyper, 0) 54 | return self.matrix[(L, R)] 55 | 56 | def __str__(self): 57 | return "raw" 58 | 59 | 60 | class PPMIModel(RawCountModel): 61 | """ 62 | PPMI(x, y) model which uses a PPMI-transformed Hearst patterns for 63 | predictions. 64 | """ 65 | 66 | def __init__(self, patterns_filename): 67 | # first read in the normal stuff 68 | super(PPMIModel, self).__init__(patterns_filename) 69 | # now let's transform the matrix 70 | tr_matrix = sparse.dok_matrix(self.matrix.shape) 71 | # actually do the transformation 72 | for (l, r) in self.matrix.keys(): 73 | pmi_lr = ( 74 | np.log(self.N) 75 | + np.log(self.matrix[(l, r)]) 76 | - np.log(self.p_w[l]) 77 | - np.log(self.p_c[r]) 78 | ) 79 | # ensure it's /positive/ pmi 80 | ppmi_lr = np.clip(pmi_lr, 0.0, 1e12) 81 | tr_matrix[(l, r)] = ppmi_lr 82 | self.matrix = tr_matrix 83 | 84 | def __str__(self): 85 | return "ppmi" 86 | 87 | 88 | class _SvdMixIn(object): 89 | """ 90 | Abstract mixin, do not use directly. Computes the SVD on top of the matrix 91 | from the superclass (may only be mixed in with a PatternBasedModel). 92 | """ 93 | 94 | def __init__(self, pattern_filename, k): 95 | # First make sure the matrix is loaded 96 | super(_SvdMixIn, self).__init__(pattern_filename) 97 | self.k = k 98 | U, S, V = sparse.linalg.svds(self.matrix.tocsr(), k=k) 99 | self.U = U.dot(np.diag(S)) 100 | self.V = V.T 101 | 102 | def predict(self, hypo, hyper): 103 | L = self.vocab.get(hypo, 0) 104 | R = self.vocab.get(hyper, 0) 105 | return self.U[L].dot(self.V[R]) 106 | 107 | def predict_many(self, hypos, hypers): 108 | lhs = [self.vocab.get(x, 0) for x in hypos] 109 | rhs = [self.vocab.get(x, 0) for x in hypers] 110 | 111 | retval = np.sum(self.U[lhs] * self.V[rhs], axis=1) 112 | return retval 113 | 114 | def __str__(self): 115 | return "svd" + super(_SvdMixIn, self).__str__() 116 | 117 | 118 | class SvdRawModel(_SvdMixIn, RawCountModel): 119 | """ 120 | sp(x,y) model presented in the paper. This is an svd over the raw counts. 121 | """ 122 | 123 | pass 124 | 125 | 126 | class SvdPpmiModel(_SvdMixIn, PPMIModel): 127 | """ 128 | spmi(x,y) model presented in the paper. This is the svd over the ppmi matrix. 129 | """ 130 | 131 | pass 132 | 133 | 134 | class RandomBaseline(PatternBasedModel): 135 | """ 136 | Generates random, but consistent predictions. Essentially a random 137 | matrix factorization. 138 | """ 139 | 140 | def __init__(self, filename, k=10, seed=42): 141 | super(RandomBaseline, self).__init__(filename) 142 | np.random.seed(seed) 143 | self.L = np.random.rand(len(self.vocab), k) 144 | self.R = np.random.rand(len(self.vocab), k) 145 | 146 | def predict(self, hypo, hyper): 147 | lid = self.vocab.get(hypo, 0) 148 | rid = self.vocab.get(hyper, 0) 149 | return self.L[lid].dot(self.R[rid]) 150 | 151 | def __str__(self): 152 | return "random" 153 | -------------------------------------------------------------------------------- /download_data.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # Copyright (c) 2017-present, Facebook, Inc. 4 | # All rights reserved. 5 | # 6 | # This source code is licensed under the license found in the 7 | # LICENSE file in the root directory of this source tree. 8 | # 9 | # ------------------------------------------------------------------------------- 10 | # This shell script downloads and preprocesses all the datasets 11 | # ------------------------------------------------------------------------------- 12 | 13 | # Immediately quit on error 14 | set -e 15 | 16 | # if you have any proxies, etc., put them here 17 | CURL_OPTIONS="-s" 18 | 19 | 20 | # URLS of each of the different datasets 21 | OMER_URL="http://u.cs.biu.ac.il/~nlp/wp-content/uploads/lexical_inference.zip" 22 | SHWARTZ_URL="https://raw.githubusercontent.com/vered1986/HypeNET/v2/dataset/datasets.rar" 23 | VERED_REPO_URL="https://raw.githubusercontent.com/vered1986/UnsupervisedHypernymy/e3b22709365c7b3042126e5887c9baa03631354e/datasets" 24 | KIMANH_REPO_URL="https://raw.githubusercontent.com/nguyenkh/HyperVec/bd2cb15a6be2a4726ffbf9c0d7e742144790dee3/datasets_classification" 25 | HYPERLEX_URL="https://raw.githubusercontent.com/ivulic/hyperlex/master/hyperlex-data.zip" 26 | 27 | function warning () { 28 | echo "$1" >&2 29 | } 30 | 31 | get_seeded_random() 32 | { 33 | seed="$1" 34 | openssl enc -aes-256-ctr -pass pass:"$seed" -nosalt \ 35 | /dev/null 36 | } 37 | 38 | function deterministic_shuffle () { 39 | # sort randomly but with a predictable seed 40 | sort --random-sort --random-source=<(get_seeded_random 42) 41 | } 42 | 43 | function download_hyperlex () { 44 | TMPFILE="$(mktemp)" 45 | TMPDIRE="$(mktemp -d)" 46 | curl $CURL_OPTIONS "$HYPERLEX_URL" > "$TMPFILE" 47 | unzip "$TMPFILE" -d "$TMPDIRE" > /dev/null 48 | 49 | echo -e 'word1\tword2\tpos\tlabel\tscore\tfold' 50 | grep -v WORD1 "$TMPDIRE/splits/random/hyperlex_training_all_random.txt" | \ 51 | cut -d' ' -f1-5 | tr ' ' '\t' | \ 52 | awk -F'\t' '$0=$0"\ttrain"' 53 | grep -v WORD1 "$TMPDIRE/splits/random/hyperlex_dev_all_random.txt" | \ 54 | cut -d' ' -f1-5 | tr ' ' '\t' | \ 55 | awk -F'\t' '$0=$0"\tval"' 56 | grep -v WORD1 "$TMPDIRE/splits/random/hyperlex_test_all_random.txt" | \ 57 | cut -d' ' -f1-5 | tr ' ' '\t' | \ 58 | awk -F'\t' '$0=$0"\ttest"' 59 | 60 | rm -rf "$TMPFILE" "$TMPDIRE" 61 | } 62 | 63 | function download_bless () { 64 | TMPFILE="$(mktemp)" 65 | TMPDIRE="$(mktemp -d)" 66 | curl $CURL_OPTIONS "$OMER_URL" > "$TMPFILE" 67 | unzip "$TMPFILE" -d "$TMPDIRE" > /dev/null 68 | 69 | echo -e 'word1\tword2\tlabel\tfold' 70 | cat "${TMPDIRE}/lexical_entailment/bless2011/data_rnd_test.tsv" \ 71 | "${TMPDIRE}/lexical_entailment/bless2011/data_rnd_train.tsv" \ 72 | "${TMPDIRE}/lexical_entailment/bless2011/data_rnd_val.tsv" | \ 73 | tr -d '\15' | \ 74 | deterministic_shuffle | \ 75 | awk '{if (NR < 1454) {print $0 "\tval"} else {print $0 "\ttest"}}' 76 | 77 | rm -rf "$TMPFILE" "$TMPDIRE" 78 | } 79 | 80 | function download_leds () { 81 | TMPFILE="$(mktemp)" 82 | TMPDIRE="$(mktemp -d)" 83 | curl $CURL_OPTIONS "$OMER_URL" > "$TMPFILE" 84 | unzip "$TMPFILE" -d "$TMPDIRE" > /dev/null 85 | 86 | echo -e 'word1\tword2\tlabel\tfold' 87 | cat "${TMPDIRE}/lexical_entailment/baroni2012/data_rnd_test.tsv" \ 88 | "${TMPDIRE}/lexical_entailment/baroni2012/data_rnd_train.tsv" \ 89 | "${TMPDIRE}/lexical_entailment/baroni2012/data_rnd_val.tsv" | \ 90 | tr -d '\15' | \ 91 | deterministic_shuffle | \ 92 | awk '{if (NR < 276) {print $0 "\tval"} else {print $0 "\ttest"}}' 93 | 94 | rm -rf "$TMPFILE" "$TMPDIRE" 95 | } 96 | 97 | function download_shwartz () { 98 | TMPFILE="$(mktemp)" 99 | TMPDIRE="$(mktemp -d)" 100 | curl $CURL_OPTIONS "$SHWARTZ_URL" > "$TMPFILE" 101 | 102 | unrar x "$TMPFILE" "$TMPDIRE" >/dev/null 103 | echo -e 'word1\tword2\tlabel\tfold' 104 | cat "$TMPDIRE/dataset_rnd/train.tsv" \ 105 | "$TMPDIRE/dataset_rnd/test.tsv" \ 106 | "$TMPDIRE/dataset_rnd/val.tsv" | \ 107 | grep -v ' ' | \ 108 | deterministic_shuffle | \ 109 | awk '{if (NR < 5257) {print $0 "\tval"} else {print $0 "\ttest"}}' 110 | 111 | rm -rf "$TMPFILE" "$TMPDIRE" 112 | } 113 | 114 | function download_bibless () { 115 | echo -e 'word1\tword2\trelation\tlabel' 116 | curl $CURL_OPTIONS "$KIMANH_REPO_URL/ABIBLESS.txt" | \ 117 | cut -f1,2,4 | \ 118 | awk -F'\t' '{if ($3 == "hyper") {print $0 "\t1"} else if ($3 == "other") {print $0 "\t0"} else {print $0 "\t-1"}}' 119 | } 120 | 121 | function download_wbless () { 122 | echo -e 'word1\tword2\tlabel\trelation\tfold' 123 | curl $CURL_OPTIONS "$KIMANH_REPO_URL/AWBLESS.txt" | \ 124 | deterministic_shuffle | \ 125 | awk '{if (NR < 168) {print $0 "\tval"} else {print $0 "\ttest"}}' 126 | } 127 | 128 | function download_eval () { 129 | echo -e 'word1\tword2\tlabel\trelation\tfold' 130 | curl $CURL_OPTIONS "$VERED_REPO_URL/EVALution.val" "$VERED_REPO_URL/EVALution.test" | \ 131 | sort | uniq | sed 's/-[jvn]\t/\t/g' | \ 132 | deterministic_shuffle | \ 133 | awk '{if (NR < 737) {print $0 "\tval"} else {print $0 "\ttest"}}' 134 | } 135 | 136 | 137 | # Let the user specify output directory, default to `data` 138 | # Ex: `HYPERNYMY_DATA_OUTPUT=.my_data_dir bash download_data.sh` 139 | if [ -z $HYPERNYMY_DATA_OUTPUT ]; then 140 | HYPERNYMY_DATA_OUTPUT="data" 141 | fi 142 | 143 | 144 | echo "The data has been checked into script and you no longer need to run this. It is left for posterity." 145 | exit 146 | 147 | if [ -d "$HYPERNYMY_DATA_OUTPUT" ] 148 | then 149 | echo "Warning: Already found the data. Please run 'rm -rf $HYPERNYMY_DATA_OUTPUT'" >&2 150 | exit 1 151 | fi 152 | 153 | if [ ! -x "$(command -v unrar)" ] 154 | then 155 | warning "This script requires the 'unrar' tool. Please run" 156 | warning " brew install unrar" 157 | warning "or whatever your system's equivalent is." 158 | exit 1 159 | fi 160 | 161 | if [ ! -x "$(command -v openssl)" ] 162 | then 163 | warning "This script requires the 'openssl' tool. Please run" 164 | warning " brew install unrar" 165 | warning "or whatever your system's equivalent is." 166 | exit 1 167 | fi 168 | 169 | 170 | 171 | # prep the output folder 172 | mkdir -p "$HYPERNYMY_DATA_OUTPUT" 173 | 174 | 175 | warning "[1/7] Downloading BLESS" 176 | download_bless > "$HYPERNYMY_DATA_OUTPUT/bless.tsv" 177 | 178 | warning "[2/7] Downloading LEDS" 179 | download_leds > "$HYPERNYMY_DATA_OUTPUT/leds.tsv" 180 | 181 | warning "[3/7] Downloading EVAL" 182 | download_eval > "$HYPERNYMY_DATA_OUTPUT/eval.tsv" 183 | 184 | warning "[4/7] Downloading Shwartz" 185 | download_shwartz > "$HYPERNYMY_DATA_OUTPUT/shwartz.tsv" 186 | 187 | warning "[5/7] Downloading Hyperlex" 188 | download_hyperlex > "$HYPERNYMY_DATA_OUTPUT/hyperlex_rnd.tsv" 189 | 190 | warning "[6/7] Downloading WBLESS" 191 | download_wbless > "$HYPERNYMY_DATA_OUTPUT/wbless.tsv" 192 | 193 | warning "[7/7] Downloading BiBLESS" 194 | download_bibless > "$HYPERNYMY_DATA_OUTPUT/bibless.tsv" 195 | 196 | warning "All done." 197 | -------------------------------------------------------------------------------- /compile_table.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | # Copyright (c) 2017-present, Facebook, Inc. 4 | # All rights reserved. 5 | # 6 | # This source code is licensed under the license found in the 7 | # LICENSE file in the root directory of this source tree. 8 | 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | import sys 15 | import argparse 16 | import logging 17 | import pandas as pd 18 | import numpy as np 19 | import json 20 | 21 | logging.basicConfig(level=logging.INFO) 22 | 23 | REPORT_FIELDS = [ 24 | ("Siege", "BLESS", "siege_bless_other_ap_{fold}"), 25 | ("Siege", "EVAL", "siege_eval_other_ap_{fold}"), 26 | ("Siege", "LEDS", "siege_leds_other_ap_{fold}"), 27 | ("Siege", "Shwartz", "siege_shwartz_other_ap_{fold}"), 28 | ("Siege", "WBless", "siege_weeds_other_ap_{fold}"), 29 | ("Graded", "Hyperlex", "cor_hyperlex_rho_{fold}"), 30 | ("Direction", "BLESS", "dir_dbless_acc_{fold}"), 31 | ("Direction", "Wbless", "dir_wbless_acc_{fold}_inv"), 32 | ("Direction", "BiBless", "dir_bibless_acc_{fold}_inv"), 33 | ] 34 | 35 | order = {} 36 | for i, (_, metric, _) in enumerate(REPORT_FIELDS): 37 | if metric not in order: 38 | order[metric] = i 39 | 40 | 41 | def nice_grouping(df): 42 | r = df.pivot_table(index=["modeltype"], columns=["metric"], values="score") 43 | cols = sorted(r.columns, key=lambda x: order[x]) 44 | return r[cols] 45 | 46 | 47 | def fprint(x): 48 | if np.isnan(x): 49 | return "" 50 | if x <= 1: 51 | return ("%.2f" % x).replace("0.", ".") 52 | else: 53 | return "%d" % x 54 | 55 | 56 | def gather_metrics(results): 57 | """ 58 | Gathers up metrics across all the different report fields, and puts them into 59 | one nice groupable table. 60 | """ 61 | output = [] 62 | for modeltype, modelset in results.groupby("name"): 63 | for tablename, metricname, metrickey in REPORT_FIELDS: 64 | # gather up the best score by validation fold of this model group 65 | modelset = modelset.copy().reset_index(drop=True) 66 | valkey = metrickey.replace("{fold}", "val") 67 | testkey = metrickey.replace("{fold}", "test") 68 | 69 | if valkey not in results.columns: 70 | continue 71 | 72 | if valkey not in modelset.columns: 73 | modelset[valkey] = np.nan 74 | if testkey not in modelset.columns: 75 | modelset[testkey] = np.nan 76 | 77 | # find the best model on validation 78 | modelset = modelset.sort_values(valkey, ascending=(tablename == "Ranking")) 79 | best_on_val = modelset.head(1).iloc[0] 80 | val_score = best_on_val[valkey] 81 | test_score = best_on_val[testkey] 82 | 83 | # report results 84 | output.append( 85 | { 86 | "modeltype": modeltype, 87 | "tablename": tablename, 88 | "metric": metricname, 89 | "fold": "val", 90 | "score": val_score, 91 | } 92 | ) 93 | output.append( 94 | { 95 | "modeltype": modeltype, 96 | "tablename": tablename, 97 | "metric": metricname, 98 | "fold": "test", 99 | "score": test_score, 100 | } 101 | ) 102 | 103 | df = pd.DataFrame(output) 104 | return df 105 | 106 | 107 | def output_latex(nice_subset): 108 | return nice_subset.to_latex(sys.stdout, float_format=fprint) 109 | 110 | 111 | def output_html(nice_subset): 112 | return nice_subset.to_html( 113 | sys.stdout, col_space=100, border=0, float_format=fprint, justify="right" 114 | ) 115 | 116 | 117 | def output_console(subset): 118 | return subset.to_string(sys.stdout, float_format=fprint, justify="right") 119 | 120 | 121 | def __flatten_dict(d, joiner="_"): 122 | items = [] 123 | for k, v in d.items(): 124 | if type(v) is dict: 125 | for k2, v2 in __flatten_dict(v, joiner=joiner): 126 | items.append((k + joiner + k2, v2)) 127 | else: 128 | items.append((k, v)) 129 | return items 130 | 131 | 132 | def read_json_log(filename): 133 | """ 134 | Reads a json log as output from a given model run. Does simple filtering 135 | to prevent bad plots (e.g. drop NaN lines), and flattens the dictionary. 136 | """ 137 | output = [] 138 | if filename == "-" or "": 139 | f = sys.stdin 140 | else: 141 | f = open(filename) # noqa: P201 142 | 143 | for i, line in enumerate(f, 1): 144 | line = line.strip() 145 | if not line: 146 | continue 147 | 148 | line = line.replace("NaN", "null") 149 | line = line.replace("-Infinity", "null") 150 | line = line.replace("Infinity", "null") 151 | try: 152 | d = json.loads(line) 153 | output.append(dict(__flatten_dict(d))) 154 | except ValueError: 155 | logging.warning("Warning: Line {} of {} didn't parse: ".format(i, filename)) 156 | 157 | f.close() 158 | return pd.DataFrame(output) 159 | 160 | 161 | def main(): 162 | parser = argparse.ArgumentParser(description="Compiles results into a table.") 163 | parser.add_argument("--input", "-i", default="-", help="Input logs") 164 | parser.add_argument("--latex", action="store_true", help="Output latex") 165 | parser.add_argument("--html", action="store_true", help="Output html") 166 | parser.add_argument("--test", action="store_true", help="Display test results") 167 | args = parser.parse_args() 168 | 169 | # Rad in the log format 170 | results = read_json_log(args.input) 171 | # For output, we want to limit the number of decimal points and auto round 172 | pd.set_option("precision", 2) 173 | # And for the purpose of output, don't allow wrapping 174 | pd.set_option("display.width", 100000) 175 | 176 | # baselines replace "distfn" with the baseline name for simplicitly of grouping 177 | df = gather_metrics(results) 178 | for tablename, subset in df.groupby("tablename"): 179 | subset = subset.copy().reset_index(drop=False) 180 | nice_val = nice_grouping(subset[subset.fold == "val"]) 181 | nice_test = nice_grouping(subset[subset.fold == "test"]) 182 | if args.test: 183 | nice_subset = nice_test 184 | else: 185 | nice_subset = nice_val 186 | 187 | if args.latex: 188 | output_latex(nice_subset) 189 | elif args.html: 190 | sys.stdout.write( 191 | "\n" 194 | ) 195 | output_html(nice_subset) 196 | else: 197 | print(tablename) 198 | output_console(nice_subset) 199 | print() 200 | print() 201 | 202 | 203 | if __name__ == "__main__": 204 | main() 205 | -------------------------------------------------------------------------------- /hypernymysuite/unsup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright (c) 2017-present, Facebook, Inc. 4 | # All rights reserved. 5 | # 6 | # This source code is licensed under the license found in the 7 | # LICENSE file in the root directory of this source tree. 8 | 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | import numpy as np 15 | import logging 16 | 17 | from scipy.stats import entropy 18 | 19 | from .reader import read_sparse_matrix 20 | from .base import HypernymySuiteModel 21 | from .base import Precomputed 22 | 23 | 24 | def invCL(x_row, y_row): 25 | """ 26 | Computes invCL(x, y) 27 | 28 | Args: 29 | x_row, y_row: ndarray[float]. Vectors for x and y. 30 | 31 | Returns: 32 | float. Estimation of distributional inclusion. 33 | """ 34 | return np.sqrt(clarkeDE(x_row, y_row) * (1 - clarkeDE(y_row, x_row))) 35 | 36 | 37 | def clarkeDE(x_row, y_row): 38 | """ 39 | clarkeDE similarity 40 | 41 | Args: 42 | x_row, y_row: ndarray[float]. Vectors for x and y. 43 | 44 | Returns: 45 | float. Estimation of distributional inclusion. 46 | """ 47 | # Get the sum of the minimum for each context. Only the mutual contexts 48 | # will yield values > 0 49 | numerator = np.min([x_row, y_row], axis=0).sum(axis=1) 50 | # The sum of x's contexts (for ppmi) is the sum of x_row. 51 | denominator = x_row.sum(axis=1) 52 | return numerator / (denominator + 1e-12) 53 | 54 | 55 | def weeds_prec(x_row, y_row): 56 | """ 57 | WeedsPrec similarity 58 | 59 | Args: 60 | x_row, y_row: ndarray[float]. Vectors for x and y. 61 | 62 | Returns: 63 | float. Estimation of distributional inclusion. 64 | """ 65 | # Get the mutual contexts: use y as a binary vector and apply dot product 66 | # with x: If c is a mutual context, it is 1 in y_non_zero and the value 67 | # ppmi(x, c) is added to the sum Otherwise, if it is 0 in either x or y, it 68 | # adds 0 to the sum. 69 | numerator = np.sum(x_row * (y_row > 0), axis=1) 70 | # The sum of x's contexts (for ppmi) is the sum of x_row. 71 | denominator = x_row.sum(axis=1) 72 | return numerator / (denominator + 1e-12) 73 | 74 | 75 | def mdot(x, y): 76 | """ 77 | Inner product of x and y 78 | 79 | Args: 80 | x_row, y_row: ndarray[float]. Vectors for x and y. 81 | 82 | Returns: 83 | float. Estimation of distributional inclusion. 84 | """ 85 | return (x * y).sum(axis=1) 86 | 87 | 88 | def cosine(x, y): 89 | """ 90 | Cosine similarity 91 | 92 | Args: 93 | x_row, y_row: ndarray[float]. Vectors for x and y. 94 | 95 | Returns: 96 | float. 97 | """ 98 | return mdot(x, y) / np.sqrt(mdot(x, x) * mdot(y, y) + 1e-12) 99 | 100 | 101 | class POSSearchDict(object): 102 | """ 103 | Utility, hack-ish "dictionary" which automatically tries to find the 104 | most appropriate pos-tagged vector for a given non-tagged word. 105 | Simply prefers nouns over verbs over adjectives. 106 | """ 107 | 108 | def __init__(self, lookup): 109 | self.lookup = lookup 110 | self.cache = {"": self.lookup[""]} 111 | 112 | def __contains__(self, item): 113 | try: 114 | self[item] 115 | return True 116 | except KeyError: 117 | return False 118 | 119 | def get(self, item, default): 120 | try: 121 | return self[item] 122 | except KeyError: 123 | return default 124 | 125 | def __getitem__(self, item): 126 | if item in self.cache: 127 | return self.cache[item] 128 | 129 | for pos in ["-n", "-v", "-j"]: 130 | if (item + pos) in self.lookup: 131 | self.cache[item] = self.lookup[(item + pos)] 132 | return self.cache[item] 133 | 134 | raise KeyError(item) 135 | 136 | 137 | class SparseMatrixBaseline(HypernymySuiteModel): 138 | """ 139 | Abstract class based on sparse, distributional models. 140 | """ 141 | 142 | def __init__(self, space_filename, pos_tagged_space=None): 143 | logging.info("Prepping sparse matrix") 144 | 145 | self.space_filename = space_filename 146 | self.matrix, self.objects, full_vocab, col_vocab = read_sparse_matrix( 147 | space_filename 148 | ) 149 | 150 | if pos_tagged_space is None: 151 | logging.warning("Having to guess whether this is a POS tagged space.") 152 | pos_tagged_space = "animal-n" in full_vocab 153 | 154 | # hard hack to mark it as POS tagged or not 155 | if pos_tagged_space: 156 | logging.info("This is a POS tagged space.") 157 | self.vocab = POSSearchDict(full_vocab) 158 | else: 159 | logging.info("This is a non-tagged space.") 160 | self.vocab = full_vocab 161 | 162 | def forward(self, inputs): 163 | raise NotImplementedError("SparseMatrixBaseline is an abstract class") 164 | 165 | 166 | class UnsupervisedBaseline(SparseMatrixBaseline): 167 | """ 168 | Unsupervised distributional similarity model. Based on some similarity 169 | measure. 170 | 171 | Args: 172 | space_filename: filename of the 3-column sparse distributional space. 173 | measurefn: fn[v,v] -> s: a measure taking two (dense) vectors and returns 174 | their similarity. Can be an asymmetric measure. 175 | """ 176 | 177 | def __init__(self, space_filename, measurefn): 178 | super(UnsupervisedBaseline, self).__init__(space_filename) 179 | self.measure = measurefn 180 | 181 | def predict(self, hypo, hyper): 182 | lhs = self.matrix[self.vocab.get(hypo, 0)].todense().A 183 | rhs = self.matrix[self.vocab.get(hyper, 0)].todense().A 184 | result = self.measure(lhs, rhs) 185 | assert result.shape == (1,) 186 | return result[0] 187 | 188 | 189 | class SLQS(SparseMatrixBaseline): 190 | """ 191 | Core implementation of SLQS model. See Santus 2014. 192 | 193 | Args: 194 | space_filename: filename of sparse distributional space 195 | topk: The number of entropy items for each row. 196 | """ 197 | 198 | _row_entropy_cache = {} 199 | 200 | def __init__(self, space_filename, topk): 201 | super(SLQS, self).__init__(space_filename) 202 | self.topk = topk 203 | logging.info("Computing column entropies") 204 | tr = self.matrix.transpose().tocsr() 205 | 206 | entropies = [] 207 | # Minibatches for computation efficiency 208 | bs = 1024 209 | for idx_start in range(0, tr.shape[0], bs): 210 | idx_end = min(idx_start + bs, tr.shape[0]) 211 | v = tr[idx_start:idx_end].todense().A 212 | entropies += list(entropy(v.T)) 213 | self.colent = np.array(entropies) 214 | assert len(self.colent) == tr.shape[0] 215 | logging.info("Done computing entropies") 216 | # cleanup 217 | del tr 218 | logging.info("Done computing row entropies") 219 | 220 | def compute_row_entropy(self, i): 221 | if i in self._row_entropy_cache: 222 | return self._row_entropy_cache[i] 223 | row = self.matrix[i] 224 | data = row.data 225 | indx = row.indices 226 | if len(data) == 0: 227 | return 0 228 | k = min(self.topk, len(data)) 229 | ranked = np.argpartition(data, -k) 230 | sigdims = indx[ranked[-k:]] 231 | rowent = np.median(self.colent[sigdims]) 232 | self._row_entropy_cache[i] = rowent 233 | return rowent 234 | 235 | def predict(self, hypo, hyper): 236 | x = self.vocab.get(hypo, 0) 237 | y = self.vocab.get(hyper, 0) 238 | # Compute entropies for each individual word 239 | ent_x = self.compute_row_entropy(x) 240 | ent_y = self.compute_row_entropy(y) 241 | # Prevent divide by zero, which shouldn't happen but does when testing things 242 | # sometimes 243 | return 1 - ent_x / (ent_y + 1e-12) 244 | 245 | 246 | class SLQS_Cos(SLQS): 247 | """ 248 | Variant of SLQS which also includes the cosine similarity. 249 | """ 250 | 251 | def __init__(self, space_filename, topk): 252 | super(SLQS_Cos, self).__init__(space_filename, topk) 253 | self.measure = cosine 254 | 255 | def predict(self, hypo, hyper): 256 | entropy_measure = super(SLQS_Cos, self).predict(hypo, hyper) 257 | # Gross hack, inheritence is messed up but python DGAF 258 | cosines = UnsupervisedBaseline.predict(self, hypo, hyper) 259 | return entropy_measure * cosines 260 | -------------------------------------------------------------------------------- /hypernymysuite/evaluation.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright (c) 2017-present, Facebook, Inc. 4 | # All rights reserved. 5 | # 6 | # This source code is licensed under the license found in the 7 | # LICENSE file in the root directory of this source tree. 8 | 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | import os 15 | import numpy as np 16 | import scipy.stats 17 | from sklearn.metrics import average_precision_score, precision_recall_curve 18 | 19 | import pandas as pd 20 | 21 | from nltk.stem.wordnet import WordNetLemmatizer 22 | 23 | 24 | # Lemmatizer we need for the model 25 | lemmatizer = WordNetLemmatizer() 26 | 27 | DATA_DIR = os.environ.get('HYPERNYMY_DATA_DIR', 'data') 28 | 29 | # Datasets! 30 | CORRELATION_EVAL_DATASETS = [("hyperlex", os.path.join(DATA_DIR, "hyperlex_rnd.tsv"))] 31 | 32 | SIEGE_EVALUATIONS = [ 33 | ("bless", os.path.join(DATA_DIR, "bless.tsv")), 34 | ("leds", os.path.join(DATA_DIR, "leds.tsv")), 35 | ("eval", os.path.join(DATA_DIR, "eval.tsv")), 36 | ("weeds", os.path.join(DATA_DIR, "wbless.tsv")), 37 | ("shwartz", os.path.join(DATA_DIR, "shwartz.tsv")), 38 | ] 39 | 40 | 41 | class Dataset(object): 42 | """ 43 | Represents a hypernymy dataset, which contains a left hand side (LHS) of hyponyms, 44 | and right hand side (RHS) of hypernyms. 45 | 46 | Params: 47 | filename: str. Filename on disk corresponding to the TSV file 48 | vocabdict: dict[str,*]. Dictionary whose keys are the vocabulary of the 49 | model to test 50 | ycolumn: str. Optional name of the label column. 51 | """ 52 | 53 | def __init__(self, filename, vocabdict, ycolumn="label"): 54 | if "" not in vocabdict: 55 | raise ValueError("Reserved word must appear in vocabulary.") 56 | 57 | table = pd.read_table(filename) 58 | 59 | # some things require the part of speech, which may not be explicitly 60 | # given in the dataset. 61 | if "pos" not in table.columns: 62 | table["pos"] = "N" 63 | table = table[table.pos.str.lower() == "n"] 64 | 65 | # Handle MWEs by replacing the space 66 | table["word1"] = table.word1.apply(lambda x: x.replace(" ", "_").lower()) 67 | table["word2"] = table.word2.apply(lambda x: x.replace(" ", "_").lower()) 68 | 69 | if vocabdict: 70 | self.word1_inv = table.word1.apply(vocabdict.__contains__) 71 | self.word2_inv = table.word2.apply(vocabdict.__contains__) 72 | else: 73 | self.word1_inv = table.word1.apply(lambda x: True) 74 | self.word2_inv = table.word2.apply(lambda x: True) 75 | 76 | # Always evaluate on lemmas 77 | table["word1"] = table.word1.apply(lemmatizer.lemmatize) 78 | table["word2"] = table.word2.apply(lemmatizer.lemmatize) 79 | 80 | self.table = table 81 | self.labels = np.array(table[ycolumn]) 82 | if "fold" in table: 83 | self.folds = table["fold"] 84 | else: 85 | self.folds = np.array(["test"] * len(self.table)) 86 | 87 | self.table["is_oov"] = self.oov_mask 88 | 89 | def __len__(self): 90 | return len(self.table) 91 | 92 | @property 93 | def hypos(self): 94 | return np.array(self.table.word1) 95 | 96 | @property 97 | def hypers(self): 98 | return np.array(self.table.word2) 99 | 100 | @property 101 | def invocab_mask(self): 102 | return self.word1_inv & self.word2_inv 103 | 104 | @property 105 | def oov_mask(self): 106 | return ~self.invocab_mask 107 | 108 | @property 109 | def val_mask(self): 110 | return np.array(self.folds == "val") 111 | 112 | @property 113 | def test_mask(self): 114 | return np.array(self.folds == "test") 115 | 116 | @property 117 | def train_mask(self): 118 | return np.array(self.folds == "train") 119 | 120 | @property 121 | def train_inv_mask(self): 122 | return self.invocab_mask & self.train_mask 123 | 124 | @property 125 | def val_inv_mask(self): 126 | return self.invocab_mask & self.val_mask 127 | 128 | @property 129 | def test_inv_mask(self): 130 | return self.invocab_mask & self.test_mask 131 | 132 | @property 133 | def y(self): 134 | return self.labels 135 | 136 | 137 | def correlation_setup(filename, model): 138 | """ 139 | Computes a spearman's rho correlation between model and continuous value. 140 | """ 141 | ds = Dataset(filename, model.vocab, ycolumn="score") 142 | 143 | h = model.predict_many(ds.hypos, ds.hypers) 144 | # For OOV words, we should guess the median distance of all the pairs. 145 | # i.e. We're not committing to high or low similarity 146 | h[ds.oov_mask] = np.median(h[ds.train_inv_mask]) 147 | 148 | y = ds.labels 149 | mi = ds.invocab_mask 150 | m_train = ds.train_mask 151 | mi_train = ds.train_inv_mask 152 | m_val = ds.val_mask 153 | mi_val = ds.val_inv_mask 154 | m_test = ds.test_mask 155 | mi_test = ds.val_inv_mask 156 | 157 | return { 158 | "rho_train": scipy.stats.spearmanr(y[m_train], h[m_train])[0], 159 | "rho_val": scipy.stats.spearmanr(y[m_val], h[m_val])[0], 160 | "rho_test": scipy.stats.spearmanr(y[m_test], h[m_test])[0], 161 | "rho_all": scipy.stats.spearmanr(y, h)[0], 162 | "rho_train_inv": scipy.stats.spearmanr(y[mi_train], h[mi_train])[0], 163 | "rho_val_inv": scipy.stats.spearmanr(y[mi_val], h[mi_val])[0], 164 | "rho_test_inv": scipy.stats.spearmanr(y[mi_test], h[mi_test])[0], 165 | "rho_all_inv": scipy.stats.spearmanr(y[mi], h[mi])[0], 166 | "num_all": len(ds), 167 | "num_oov_all": int(sum(ds.oov_mask)), 168 | "pct_oov_all": np.mean(ds.oov_mask), 169 | } 170 | 171 | 172 | def bless_directionality_setup(model): 173 | """ 174 | Asks a model whether (x, y) > (y, x) for a number of positive hypernymy examples. 175 | """ 176 | # load up the data 177 | ds = Dataset(os.path.join(DATA_DIR, "bless.tsv"), model.vocab) 178 | 179 | # only keep the positive pairs 180 | hypos = ds.hypos[ds.y] 181 | hypers = ds.hypers[ds.y] 182 | 183 | forward_predictions = model.predict_many(hypos, hypers) 184 | reverse_predictions = model.predict_many(hypers, hypos) 185 | 186 | # Fold masks 187 | m_val = ds.val_mask[ds.y] 188 | m_test = ds.test_mask[ds.y] 189 | 190 | # Fold, in-vocab masks 191 | oov = ds.oov_mask[ds.y] 192 | mi_val = m_val & ~oov 193 | mi_test = m_test & ~oov 194 | 195 | # final input 196 | # Check that the original directionality is correct 197 | yhat = forward_predictions > reverse_predictions 198 | 199 | return { 200 | "acc_val": np.mean(yhat[m_val]), 201 | "acc_test": np.mean(yhat[m_test]), 202 | "acc_all": np.mean(yhat), 203 | "acc_val_inv": np.mean(yhat[mi_val]), 204 | "acc_test_inv": np.mean(yhat[mi_test]), 205 | "acc_all_inv": np.mean(yhat[~oov]), 206 | "num_val": int(sum(m_val)), 207 | "num_test": int(sum(m_test)), 208 | "num_oov_all": int(sum(oov)), 209 | "pct_oov_all": np.mean(oov), 210 | } 211 | 212 | 213 | def wbless_setup(model): 214 | """ 215 | Accuracy using a threshold, with a dataset that explicitly contains reverse pairs. 216 | """ 217 | ds = Dataset(os.path.join(DATA_DIR, "wbless.tsv"), model.vocab) 218 | 219 | # Ensure we always get the same results 220 | rng = np.random.RandomState(42) 221 | VAL_PROB = .02 222 | NUM_TRIALS = 1000 223 | 224 | # We have no way of handling oov 225 | h = model.predict_many(ds.hypos[ds.invocab_mask], ds.hypers[ds.invocab_mask]) 226 | y = ds.y[ds.invocab_mask] 227 | 228 | val_scores = [] 229 | test_scores = [] 230 | 231 | for _ in range(NUM_TRIALS): 232 | # Generate a new mask every time 233 | m_val = rng.rand(len(y)) < VAL_PROB 234 | # Test is everything except val 235 | m_test = ~m_val 236 | _, _, t = precision_recall_curve(y[m_val], h[m_val]) 237 | # pick the highest accuracy on the validation set 238 | thr_accs = np.mean((h[m_val, np.newaxis] >= t) == y[m_val, np.newaxis], axis=0) 239 | best_t = t[thr_accs.argmax()] 240 | preds_val = h[m_val] >= best_t 241 | preds_test = h[m_test] >= best_t 242 | # Evaluate 243 | val_scores.append(np.mean(preds_val == y[m_val])) 244 | test_scores.append(np.mean(preds_test == y[m_test])) 245 | # sanity check 246 | assert np.allclose(val_scores[-1], thr_accs.max()) 247 | 248 | # report average across many folds 249 | return {"acc_val_inv": np.mean(val_scores), "acc_test_inv": np.mean(test_scores)} 250 | 251 | 252 | def bibless_setup(model): 253 | """ 254 | Combined detection with a threshold, plus direction prediction. 255 | """ 256 | ds = Dataset(os.path.join(DATA_DIR, "bibless.tsv"), model.vocab) 257 | 258 | # Ensure we always get the same results 259 | rng = np.random.RandomState(42) 260 | VAL_PROB = .02 261 | NUM_TRIALS = 1000 262 | 263 | # We have no way of handling oov 264 | y = ds.y[ds.invocab_mask] 265 | 266 | # hypernymy could be either direction 267 | yh = y != 0 268 | 269 | # get forward and backward predictions 270 | hf = model.predict_many(ds.hypos[ds.invocab_mask], ds.hypers[ds.invocab_mask]) 271 | hr = model.predict_many(ds.hypers[ds.invocab_mask], ds.hypos[ds.invocab_mask]) 272 | h = np.max([hf, hr], axis=0) 273 | 274 | dir_pred = 2 * np.float32(hf >= hr) - 1 275 | 276 | val_scores = [] 277 | test_scores = [] 278 | for _ in range(NUM_TRIALS): 279 | # Generate a new mask every time 280 | m_val = rng.rand(len(y)) < VAL_PROB 281 | # Test is everything except val 282 | m_test = ~m_val 283 | 284 | # set the threshold based on the maximum score 285 | _, _, t = precision_recall_curve(yh[m_val], h[m_val]) 286 | thr_accs = np.mean((h[m_val, np.newaxis] >= t) == yh[m_val, np.newaxis], axis=0) 287 | best_t = t[thr_accs.argmax()] 288 | 289 | det_preds_val = h[m_val] >= best_t 290 | det_preds_test = h[m_test] >= best_t 291 | 292 | fin_preds_val = det_preds_val * dir_pred[m_val] 293 | fin_preds_test = det_preds_test * dir_pred[m_test] 294 | 295 | val_scores.append(np.mean(fin_preds_val == y[m_val])) 296 | test_scores.append(np.mean(fin_preds_test == y[m_test])) 297 | 298 | # report average across many folds 299 | return {"acc_val_inv": np.mean(val_scores), "acc_test_inv": np.mean(test_scores)} 300 | 301 | 302 | def ap_at_k(y_true, y_score, k): 303 | """ 304 | Computes AP@k, or AP of the model's top K predictions. Used in 305 | Shwartz, Santus and Schlectweg, EACL 2017. 306 | https://arxiv.org/abs/1612.04460 307 | """ 308 | argsort = np.argsort(y_score) 309 | score_srt = y_score[argsort[-k:]] 310 | label_srt = y_true[argsort[-k:]] 311 | return average_precision_score(label_srt, score_srt) 312 | 313 | 314 | def siege_setup(filename, model): 315 | """ 316 | Computes Average Precision for a binary dataset. 317 | """ 318 | ds = Dataset(filename, model.vocab) 319 | 320 | m_val = ds.val_mask 321 | mi_val = ds.val_inv_mask 322 | m_test = ds.test_mask 323 | mi_test = ds.test_inv_mask 324 | 325 | # we only need to compute forward on in-vocab words, speeds things up 326 | h_inv = model.predict_many(ds.hypos[ds.invocab_mask], ds.hypers[ds.invocab_mask]) 327 | 328 | # stub out for the entire data though 329 | h = np.zeros(len(ds)) 330 | # and fill with our predictions 331 | h[ds.invocab_mask] = h_inv 332 | # And OOV predictions should be our lowest validation prediction, essentially 333 | # always predicting false 334 | h[ds.oov_mask] = h[mi_val].min() 335 | y = ds.y 336 | results = {} 337 | 338 | # Across all relations 339 | results["other"] = { 340 | "ap_val": average_precision_score(y[m_val], h[m_val]), 341 | "ap_test": average_precision_score(y[m_test], h[m_test]), 342 | "ap100_val": ap_at_k(y[m_val], h[m_val], 100), 343 | "ap100_test": ap_at_k(y[m_test], h[m_test], 100), 344 | "ap_val_inv": average_precision_score(y[mi_val], h[mi_val]), 345 | "ap_test_inv": average_precision_score(y[mi_test], h[mi_test]), 346 | "ap100_val_inv": ap_at_k(y[mi_val], h[mi_val], 100), 347 | "ap100_test_inv": ap_at_k(y[mi_test], h[mi_test], 100), 348 | } 349 | results["pct_oov"] = np.mean(ds.oov_mask) 350 | return results 351 | 352 | 353 | def all_evaluations(model, extra_args=None): 354 | """ 355 | Utility method which performs all evaluations and unifies the results into 356 | a single (nested) dictionary. 357 | 358 | Args: 359 | model: HypernymySuiteModel. Model to be evaluated. 360 | 361 | Returns: 362 | A nested dictionary of results. Best combined with `compile_table` 363 | module. 364 | """ 365 | results = {} 366 | results["dir_wbless"] = wbless_setup(model) 367 | results["dir_bibless"] = bibless_setup(model) 368 | results["dir_dbless"] = bless_directionality_setup(model) 369 | 370 | for taskname, filename in CORRELATION_EVAL_DATASETS: 371 | result = correlation_setup(filename, model) 372 | results["cor_{}".format(taskname)] = result 373 | 374 | # siege evaluations 375 | for taskname, filename in SIEGE_EVALUATIONS: 376 | result = siege_setup(filename, model) 377 | results["siege_{}".format(taskname)] = result 378 | 379 | return results 380 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Attribution-NonCommercial 4.0 International 2 | 3 | ======================================================================= 4 | 5 | Creative Commons Corporation ("Creative Commons") is not a law firm and 6 | does not provide legal services or legal advice. Distribution of 7 | Creative Commons public licenses does not create a lawyer-client or 8 | other relationship. Creative Commons makes its licenses and related 9 | information available on an "as-is" basis. Creative Commons gives no 10 | warranties regarding its licenses, any material licensed under their 11 | terms and conditions, or any related information. Creative Commons 12 | disclaims all liability for damages resulting from their use to the 13 | fullest extent possible. 14 | 15 | Using Creative Commons Public Licenses 16 | 17 | Creative Commons public licenses provide a standard set of terms and 18 | conditions that creators and other rights holders may use to share 19 | original works of authorship and other material subject to copyright 20 | and certain other rights specified in the public license below. The 21 | following considerations are for informational purposes only, are not 22 | exhaustive, and do not form part of our licenses. 23 | 24 | Considerations for licensors: Our public licenses are 25 | intended for use by those authorized to give the public 26 | permission to use material in ways otherwise restricted by 27 | copyright and certain other rights. Our licenses are 28 | irrevocable. Licensors should read and understand the terms 29 | and conditions of the license they choose before applying it. 30 | Licensors should also secure all rights necessary before 31 | applying our licenses so that the public can reuse the 32 | material as expected. Licensors should clearly mark any 33 | material not subject to the license. This includes other CC- 34 | licensed material, or material used under an exception or 35 | limitation to copyright. More considerations for licensors: 36 | wiki.creativecommons.org/Considerations_for_licensors 37 | 38 | Considerations for the public: By using one of our public 39 | licenses, a licensor grants the public permission to use the 40 | licensed material under specified terms and conditions. If 41 | the licensor's permission is not necessary for any reason--for 42 | example, because of any applicable exception or limitation to 43 | copyright--then that use is not regulated by the license. Our 44 | licenses grant only permissions under copyright and certain 45 | other rights that a licensor has authority to grant. Use of 46 | the licensed material may still be restricted for other 47 | reasons, including because others have copyright or other 48 | rights in the material. A licensor may make special requests, 49 | such as asking that all changes be marked or described. 50 | Although not required by our licenses, you are encouraged to 51 | respect those requests where reasonable. More_considerations 52 | for the public: 53 | wiki.creativecommons.org/Considerations_for_licensees 54 | 55 | ======================================================================= 56 | 57 | Creative Commons Attribution-NonCommercial 4.0 International Public 58 | License 59 | 60 | By exercising the Licensed Rights (defined below), You accept and agree 61 | to be bound by the terms and conditions of this Creative Commons 62 | Attribution-NonCommercial 4.0 International Public License ("Public 63 | License"). To the extent this Public License may be interpreted as a 64 | contract, You are granted the Licensed Rights in consideration of Your 65 | acceptance of these terms and conditions, and the Licensor grants You 66 | such rights in consideration of benefits the Licensor receives from 67 | making the Licensed Material available under these terms and 68 | conditions. 69 | 70 | Section 1 -- Definitions. 71 | 72 | a. Adapted Material means material subject to Copyright and Similar 73 | Rights that is derived from or based upon the Licensed Material 74 | and in which the Licensed Material is translated, altered, 75 | arranged, transformed, or otherwise modified in a manner requiring 76 | permission under the Copyright and Similar Rights held by the 77 | Licensor. For purposes of this Public License, where the Licensed 78 | Material is a musical work, performance, or sound recording, 79 | Adapted Material is always produced where the Licensed Material is 80 | synched in timed relation with a moving image. 81 | 82 | b. Adapter's License means the license You apply to Your Copyright 83 | and Similar Rights in Your contributions to Adapted Material in 84 | accordance with the terms and conditions of this Public License. 85 | 86 | c. Copyright and Similar Rights means copyright and/or similar rights 87 | closely related to copyright including, without limitation, 88 | performance, broadcast, sound recording, and Sui Generis Database 89 | Rights, without regard to how the rights are labeled or 90 | categorized. For purposes of this Public License, the rights 91 | specified in Section 2(b)(1)-(2) are not Copyright and Similar 92 | Rights. 93 | d. Effective Technological Measures means those measures that, in the 94 | absence of proper authority, may not be circumvented under laws 95 | fulfilling obligations under Article 11 of the WIPO Copyright 96 | Treaty adopted on December 20, 1996, and/or similar international 97 | agreements. 98 | 99 | e. Exceptions and Limitations means fair use, fair dealing, and/or 100 | any other exception or limitation to Copyright and Similar Rights 101 | that applies to Your use of the Licensed Material. 102 | 103 | f. Licensed Material means the artistic or literary work, database, 104 | or other material to which the Licensor applied this Public 105 | License. 106 | 107 | g. Licensed Rights means the rights granted to You subject to the 108 | terms and conditions of this Public License, which are limited to 109 | all Copyright and Similar Rights that apply to Your use of the 110 | Licensed Material and that the Licensor has authority to license. 111 | 112 | h. Licensor means the individual(s) or entity(ies) granting rights 113 | under this Public License. 114 | 115 | i. NonCommercial means not primarily intended for or directed towards 116 | commercial advantage or monetary compensation. For purposes of 117 | this Public License, the exchange of the Licensed Material for 118 | other material subject to Copyright and Similar Rights by digital 119 | file-sharing or similar means is NonCommercial provided there is 120 | no payment of monetary compensation in connection with the 121 | exchange. 122 | 123 | j. Share means to provide material to the public by any means or 124 | process that requires permission under the Licensed Rights, such 125 | as reproduction, public display, public performance, distribution, 126 | dissemination, communication, or importation, and to make material 127 | available to the public including in ways that members of the 128 | public may access the material from a place and at a time 129 | individually chosen by them. 130 | 131 | k. Sui Generis Database Rights means rights other than copyright 132 | resulting from Directive 96/9/EC of the European Parliament and of 133 | the Council of 11 March 1996 on the legal protection of databases, 134 | as amended and/or succeeded, as well as other essentially 135 | equivalent rights anywhere in the world. 136 | 137 | l. You means the individual or entity exercising the Licensed Rights 138 | under this Public License. Your has a corresponding meaning. 139 | 140 | Section 2 -- Scope. 141 | 142 | a. License grant. 143 | 144 | 1. Subject to the terms and conditions of this Public License, 145 | the Licensor hereby grants You a worldwide, royalty-free, 146 | non-sublicensable, non-exclusive, irrevocable license to 147 | exercise the Licensed Rights in the Licensed Material to: 148 | 149 | a. reproduce and Share the Licensed Material, in whole or 150 | in part, for NonCommercial purposes only; and 151 | 152 | b. produce, reproduce, and Share Adapted Material for 153 | NonCommercial purposes only. 154 | 155 | 2. Exceptions and Limitations. For the avoidance of doubt, where 156 | Exceptions and Limitations apply to Your use, this Public 157 | License does not apply, and You do not need to comply with 158 | its terms and conditions. 159 | 160 | 3. Term. The term of this Public License is specified in Section 161 | 6(a). 162 | 163 | 4. Media and formats; technical modifications allowed. The 164 | Licensor authorizes You to exercise the Licensed Rights in 165 | all media and formats whether now known or hereafter created, 166 | and to make technical modifications necessary to do so. The 167 | Licensor waives and/or agrees not to assert any right or 168 | authority to forbid You from making technical modifications 169 | necessary to exercise the Licensed Rights, including 170 | technical modifications necessary to circumvent Effective 171 | Technological Measures. For purposes of this Public License, 172 | simply making modifications authorized by this Section 2(a) 173 | (4) never produces Adapted Material. 174 | 175 | 5. Downstream recipients. 176 | 177 | a. Offer from the Licensor -- Licensed Material. Every 178 | recipient of the Licensed Material automatically 179 | receives an offer from the Licensor to exercise the 180 | Licensed Rights under the terms and conditions of this 181 | Public License. 182 | 183 | b. No downstream restrictions. You may not offer or impose 184 | any additional or different terms or conditions on, or 185 | apply any Effective Technological Measures to, the 186 | Licensed Material if doing so restricts exercise of the 187 | Licensed Rights by any recipient of the Licensed 188 | Material. 189 | 190 | 6. No endorsement. Nothing in this Public License constitutes or 191 | may be construed as permission to assert or imply that You 192 | are, or that Your use of the Licensed Material is, connected 193 | with, or sponsored, endorsed, or granted official status by, 194 | the Licensor or others designated to receive attribution as 195 | provided in Section 3(a)(1)(A)(i). 196 | 197 | b. Other rights. 198 | 199 | 1. Moral rights, such as the right of integrity, are not 200 | licensed under this Public License, nor are publicity, 201 | privacy, and/or other similar personality rights; however, to 202 | the extent possible, the Licensor waives and/or agrees not to 203 | assert any such rights held by the Licensor to the limited 204 | extent necessary to allow You to exercise the Licensed 205 | Rights, but not otherwise. 206 | 207 | 2. Patent and trademark rights are not licensed under this 208 | Public License. 209 | 210 | 3. To the extent possible, the Licensor waives any right to 211 | collect royalties from You for the exercise of the Licensed 212 | Rights, whether directly or through a collecting society 213 | under any voluntary or waivable statutory or compulsory 214 | licensing scheme. In all other cases the Licensor expressly 215 | reserves any right to collect such royalties, including when 216 | the Licensed Material is used other than for NonCommercial 217 | purposes. 218 | 219 | Section 3 -- License Conditions. 220 | 221 | Your exercise of the Licensed Rights is expressly made subject to the 222 | following conditions. 223 | 224 | a. Attribution. 225 | 226 | 1. If You Share the Licensed Material (including in modified 227 | form), You must: 228 | 229 | a. retain the following if it is supplied by the Licensor 230 | with the Licensed Material: 231 | 232 | i. identification of the creator(s) of the Licensed 233 | Material and any others designated to receive 234 | attribution, in any reasonable manner requested by 235 | the Licensor (including by pseudonym if 236 | designated); 237 | 238 | ii. a copyright notice; 239 | 240 | iii. a notice that refers to this Public License; 241 | 242 | iv. a notice that refers to the disclaimer of 243 | warranties; 244 | 245 | v. a URI or hyperlink to the Licensed Material to the 246 | extent reasonably practicable; 247 | 248 | b. indicate if You modified the Licensed Material and 249 | retain an indication of any previous modifications; and 250 | 251 | c. indicate the Licensed Material is licensed under this 252 | Public License, and include the text of, or the URI or 253 | hyperlink to, this Public License. 254 | 255 | 2. You may satisfy the conditions in Section 3(a)(1) in any 256 | reasonable manner based on the medium, means, and context in 257 | which You Share the Licensed Material. For example, it may be 258 | reasonable to satisfy the conditions by providing a URI or 259 | hyperlink to a resource that includes the required 260 | information. 261 | 262 | 3. If requested by the Licensor, You must remove any of the 263 | information required by Section 3(a)(1)(A) to the extent 264 | reasonably practicable. 265 | 266 | 4. If You Share Adapted Material You produce, the Adapter's 267 | License You apply must not prevent recipients of the Adapted 268 | Material from complying with this Public License. 269 | 270 | Section 4 -- Sui Generis Database Rights. 271 | 272 | Where the Licensed Rights include Sui Generis Database Rights that 273 | apply to Your use of the Licensed Material: 274 | 275 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right 276 | to extract, reuse, reproduce, and Share all or a substantial 277 | portion of the contents of the database for NonCommercial purposes 278 | only; 279 | 280 | b. if You include all or a substantial portion of the database 281 | contents in a database in which You have Sui Generis Database 282 | Rights, then the database in which You have Sui Generis Database 283 | Rights (but not its individual contents) is Adapted Material; and 284 | 285 | c. You must comply with the conditions in Section 3(a) if You Share 286 | all or a substantial portion of the contents of the database. 287 | 288 | For the avoidance of doubt, this Section 4 supplements and does not 289 | replace Your obligations under this Public License where the Licensed 290 | Rights include other Copyright and Similar Rights. 291 | 292 | Section 5 -- Disclaimer of Warranties and Limitation of Liability. 293 | 294 | a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE 295 | EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS 296 | AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF 297 | ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, 298 | IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, 299 | WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR 300 | PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, 301 | ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT 302 | KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT 303 | ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. 304 | 305 | b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE 306 | TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, 307 | NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, 308 | INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, 309 | COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR 310 | USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN 311 | ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR 312 | DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR 313 | IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. 314 | 315 | c. The disclaimer of warranties and limitation of liability provided 316 | above shall be interpreted in a manner that, to the extent 317 | possible, most closely approximates an absolute disclaimer and 318 | waiver of all liability. 319 | 320 | Section 6 -- Term and Termination. 321 | 322 | a. This Public License applies for the term of the Copyright and 323 | Similar Rights licensed here. However, if You fail to comply with 324 | this Public License, then Your rights under this Public License 325 | terminate automatically. 326 | 327 | b. Where Your right to use the Licensed Material has terminated under 328 | Section 6(a), it reinstates: 329 | 330 | 1. automatically as of the date the violation is cured, provided 331 | it is cured within 30 days of Your discovery of the 332 | violation; or 333 | 334 | 2. upon express reinstatement by the Licensor. 335 | 336 | For the avoidance of doubt, this Section 6(b) does not affect any 337 | right the Licensor may have to seek remedies for Your violations 338 | of this Public License. 339 | 340 | c. For the avoidance of doubt, the Licensor may also offer the 341 | Licensed Material under separate terms or conditions or stop 342 | distributing the Licensed Material at any time; however, doing so 343 | will not terminate this Public License. 344 | 345 | d. Sections 1, 5, 6, 7, and 8 survive termination of this Public 346 | License. 347 | 348 | Section 7 -- Other Terms and Conditions. 349 | 350 | a. The Licensor shall not be bound by any additional or different 351 | terms or conditions communicated by You unless expressly agreed. 352 | 353 | b. Any arrangements, understandings, or agreements regarding the 354 | Licensed Material not stated herein are separate from and 355 | independent of the terms and conditions of this Public License. 356 | 357 | Section 8 -- Interpretation. 358 | 359 | a. For the avoidance of doubt, this Public License does not, and 360 | shall not be interpreted to, reduce, limit, restrict, or impose 361 | conditions on any use of the Licensed Material that could lawfully 362 | be made without permission under this Public License. 363 | 364 | b. To the extent possible, if any provision of this Public License is 365 | deemed unenforceable, it shall be automatically reformed to the 366 | minimum extent necessary to make it enforceable. If the provision 367 | cannot be reformed, it shall be severed from this Public License 368 | without affecting the enforceability of the remaining terms and 369 | conditions. 370 | 371 | c. No term or condition of this Public License will be waived and no 372 | failure to comply consented to unless expressly agreed to by the 373 | Licensor. 374 | 375 | d. Nothing in this Public License constitutes or may be interpreted 376 | as a limitation upon, or waiver of, any privileges and immunities 377 | that apply to the Licensor or You, including from the legal 378 | processes of any jurisdiction or authority. 379 | 380 | ======================================================================= 381 | 382 | Creative Commons is not a party to its public 383 | licenses. Notwithstanding, Creative Commons may elect to apply one of 384 | its public licenses to material it publishes and in those instances 385 | will be considered the “Licensor.” The text of the Creative Commons 386 | public licenses is dedicated to the public domain under the CC0 Public 387 | Domain Dedication. Except for the limited purpose of indicating that 388 | material is shared under a Creative Commons public license or as 389 | otherwise permitted by the Creative Commons policies published at 390 | creativecommons.org/policies, Creative Commons does not authorize the 391 | use of the trademark "Creative Commons" or any other trademark or logo 392 | of Creative Commons without its prior written consent including, 393 | without limitation, in connection with any unauthorized modifications 394 | to any of its public licenses or any other arrangements, 395 | understandings, or agreements concerning use of licensed material. For 396 | the avoidance of doubt, this paragraph does not form part of the 397 | public licenses. 398 | 399 | Creative Commons may be contacted at creativecommons.org. 400 | -------------------------------------------------------------------------------- /data/bibless.tsv: -------------------------------------------------------------------------------- 1 | word1 word2 relation label 2 | pheasant game hyper 1 3 | pheasant bird hyper 1 4 | pheasant vertebrate hyper 1 5 | pheasant creature hyper 1 6 | dove pheasant other 0 7 | animal pheasant rhyper -1 8 | beak pheasant other 0 9 | pheasant pakistani other 0 10 | sparrow creature hyper 1 11 | sparrow animal hyper 1 12 | sparrow vertebrate hyper 1 13 | sparrow football other 0 14 | pheasant sparrow other 0 15 | bird sparrow rhyper -1 16 | fox vertebrate hyper 1 17 | fox animal hyper 1 18 | fox mammal hyper 1 19 | fox predator hyper 1 20 | fox carnivore hyper 1 21 | fox canine hyper 1 22 | beast fox rhyper -1 23 | fox sub-committee other 0 24 | creature fox rhyper -1 25 | fox mouth other 0 26 | fox giraffe other 0 27 | tail fox other 0 28 | stove device hyper 1 29 | stove object hyper 1 30 | stove artifact hyper 1 31 | stove artefact hyper 1 32 | stove commodity hyper 1 33 | stove migraine other 0 34 | good stove rhyper -1 35 | appliance stove rhyper -1 36 | bookshop stove other 0 37 | telephone stove other 0 38 | rifle artifact hyper 1 39 | rifle weapon hyper 1 40 | rifle arm hyper 1 41 | rifle artefact hyper 1 42 | rifle gun hyper 1 43 | rifle firearm hyper 1 44 | kalashnikov rifle other 0 45 | rifle revolver other 0 46 | device rifle rhyper -1 47 | object rifle rhyper -1 48 | barrel rifle other 0 49 | rifle hockey other 0 50 | cottage construction hyper 1 51 | cottage dwelling hyper 1 52 | cottage housing hyper 1 53 | cottage structure hyper 1 54 | cottage building hyper 1 55 | cottage home hyper 1 56 | cottage habitation hyper 1 57 | cottage edifice hyper 1 58 | alcohol cottage other 0 59 | restaurant cottage other 0 60 | accommodation cottage rhyper -1 61 | owner cottage other 0 62 | ceiling cottage other 0 63 | lodging cottage rhyper -1 64 | cottage castle other 0 65 | cottage camera other 0 66 | radio artefact hyper 1 67 | radio good hyper 1 68 | radio object hyper 1 69 | radio system hyper 1 70 | radio device hyper 1 71 | radio appliance hyper 1 72 | radio commodity hyper 1 73 | washer radio other 0 74 | speaker radio other 0 75 | radio wire other 0 76 | artifact radio rhyper -1 77 | radio canvas other 0 78 | equipment radio rhyper -1 79 | radio stereo other 0 80 | pine tree hyper 1 81 | pine evergreen hyper 1 82 | pine tooling other 0 83 | stump pine other 0 84 | oven artifact hyper 1 85 | oven artefact hyper 1 86 | oven device hyper 1 87 | oven good hyper 1 88 | oven commodity hyper 1 89 | steel oven other 0 90 | appliance oven rhyper -1 91 | oven import other 0 92 | oven television other 0 93 | oven timer other 0 94 | bomb artefact hyper 1 95 | bomb device hyper 1 96 | bomb arm hyper 1 97 | bomb weapon hyper 1 98 | fuse bomb other 0 99 | artifact bomb rhyper -1 100 | dagger bomb other 0 101 | bomb thoroughfare other 0 102 | chair artifact hyper 1 103 | chair furniture hyper 1 104 | chair object hyper 1 105 | chair artefact hyper 1 106 | stool chair other 0 107 | chair scroll other 0 108 | furnishing chair rhyper -1 109 | cushion chair other 0 110 | bomber conveyance hyper 1 111 | bomber aircraft hyper 1 112 | bomber plane hyper 1 113 | bomber transport hyper 1 114 | bomber warplane hyper 1 115 | bomber aeroplane hyper 1 116 | bomber artifact hyper 1 117 | bomber airplane hyper 1 118 | bomber helicopter other 0 119 | vehicle bomber rhyper -1 120 | bomber fuel other 0 121 | taste bomber other 0 122 | windshield bomber other 0 123 | frigate bomber other 0 124 | artefact bomber rhyper -1 125 | bomber setting other 0 126 | grape berry hyper 1 127 | grape food hyper 1 128 | grape fruit hyper 1 129 | grape reef other 0 130 | cherry grape other 0 131 | produce grape rhyper -1 132 | elm tree hyper 1 133 | plant elm rhyper -1 134 | willow tree hyper 1 135 | flower willow other 0 136 | cherry produce hyper 1 137 | cherry food hyper 1 138 | stone cherry other 0 139 | fruit cherry rhyper -1 140 | scooter transport hyper 1 141 | scooter artefact hyper 1 142 | scooter vehicle hyper 1 143 | scooter artifact hyper 1 144 | scooter necessity other 0 145 | conveyance scooter rhyper -1 146 | moped scooter other 0 147 | fuel scooter other 0 148 | swan vertebrate hyper 1 149 | swan bird hyper 1 150 | swan waterfowl hyper 1 151 | swan creature hyper 1 152 | crow swan other 0 153 | swan overdraft other 0 154 | animal swan rhyper -1 155 | eye swan other 0 156 | jar vessel hyper 1 157 | jar artefact hyper 1 158 | jar container hyper 1 159 | jar artifact hyper 1 160 | jar object hyper 1 161 | kitchenware jar rhyper -1 162 | jam jar other 0 163 | jar cookie other 0 164 | respect jar other 0 165 | jar plate other 0 166 | cloak clothing hyper 1 167 | cloak artefact hyper 1 168 | cloak clothes hyper 1 169 | cloak covering hyper 1 170 | cloak commodity hyper 1 171 | cloak artifact hyper 1 172 | cloak apparel hyper 1 173 | cloak wear hyper 1 174 | back cloak other 0 175 | workload cloak other 0 176 | garment cloak rhyper -1 177 | jacket cloak other 0 178 | cloak greenbelt other 0 179 | cloak stigma other 0 180 | cloak trousers other 0 181 | good cloak rhyper -1 182 | glider artifact hyper 1 183 | glider aircraft hyper 1 184 | glider transport hyper 1 185 | glider craft hyper 1 186 | glider artefact hyper 1 187 | back glider other 0 188 | conveyance glider rhyper -1 189 | glider front other 0 190 | vehicle glider rhyper -1 191 | boy glider other 0 192 | grapefruit produce hyper 1 193 | grapefruit citrus hyper 1 194 | grapefruit fruit hyper 1 195 | food grapefruit rhyper -1 196 | cranberry grapefruit other 0 197 | grapefruit re-assessment other 0 198 | radish produce hyper 1 199 | radish food hyper 1 200 | radish vegetable hyper 1 201 | leaf radish other 0 202 | turnip radish other 0 203 | radish vinyl other 0 204 | robe garment hyper 1 205 | robe artefact hyper 1 206 | robe covering hyper 1 207 | robe commodity hyper 1 208 | robe apparel hyper 1 209 | robe clothing hyper 1 210 | robe good hyper 1 211 | robe artifact hyper 1 212 | hat robe other 0 213 | robe commentary other 0 214 | back robe other 0 215 | robe ruminant other 0 216 | object robe rhyper -1 217 | blouse robe other 0 218 | something robe other 0 219 | robe front other 0 220 | hat covering hyper 1 221 | hat apparel hyper 1 222 | hat wear hyper 1 223 | hat good hyper 1 224 | hat accessory hyper 1 225 | hat artefact hyper 1 226 | hat clothing hyper 1 227 | hat headgear hyper 1 228 | hat garment hyper 1 229 | hat headdress hyper 1 230 | hat object hyper 1 231 | commencement hat other 0 232 | hat virus other 0 233 | robe hat other 0 234 | commodity hat rhyper -1 235 | inside hat other 0 236 | hat italian other 0 237 | artifact hat rhyper -1 238 | jacket hat other 0 239 | outside hat other 0 240 | hat crown other 0 241 | clothes hat rhyper -1 242 | battleship artefact hyper 1 243 | battleship vehicle hyper 1 244 | battleship vessel hyper 1 245 | battleship warship hyper 1 246 | battleship craft hyper 1 247 | plantation battleship other 0 248 | battleship lack other 0 249 | battleship glider other 0 250 | topside battleship other 0 251 | ship battleship rhyper -1 252 | cannon artifact hyper 1 253 | cannon gun hyper 1 254 | cannon arm hyper 1 255 | cannon device hyper 1 256 | cannon object hyper 1 257 | trigger cannon other 0 258 | cannon chamber other 0 259 | achiever cannon other 0 260 | artefact cannon rhyper -1 261 | hatchet cannon other 0 262 | vulture raptor hyper 1 263 | vulture predator hyper 1 264 | vulture scavenger hyper 1 265 | vulture creature hyper 1 266 | vulture animal hyper 1 267 | vertebrate vulture rhyper -1 268 | vulture goose other 0 269 | beak vulture other 0 270 | dismay vulture other 0 271 | bird vulture rhyper -1 272 | fridge commodity hyper 1 273 | fridge object hyper 1 274 | fridge artifact hyper 1 275 | fridge device hyper 1 276 | fridge artefact hyper 1 277 | telephone fridge other 0 278 | fridge food other 0 279 | good fridge rhyper -1 280 | fridge test other 0 281 | learner fridge other 0 282 | bear mammal hyper 1 283 | bear creature hyper 1 284 | bear vertebrate hyper 1 285 | bear beast hyper 1 286 | dog bear other 0 287 | jaw bear other 0 288 | bear fur other 0 289 | animal bear rhyper -1 290 | spoon artefact hyper 1 291 | spoon cutlery hyper 1 292 | spoon artifact hyper 1 293 | spoon tool hyper 1 294 | spoon object hyper 1 295 | spoon tableware hyper 1 296 | spoon container hyper 1 297 | kitchenware spoon rhyper -1 298 | spoon empathy other 0 299 | lock-up spoon other 0 300 | shovel spoon other 0 301 | wood spoon other 0 302 | spoon head other 0 303 | spoon wrench other 0 304 | giraffe herbivore hyper 1 305 | giraffe beast hyper 1 306 | giraffe mammal hyper 1 307 | giraffe vertebrate hyper 1 308 | assessor giraffe other 0 309 | giraffe leg other 0 310 | giraffe violin other 0 311 | creature giraffe rhyper -1 312 | cedar plant hyper 1 313 | cedar evergreen hyper 1 314 | cedar sense other 0 315 | tree cedar rhyper -1 316 | snake creature hyper 1 317 | snake animal hyper 1 318 | snake reptile hyper 1 319 | snake beast hyper 1 320 | tooth snake other 0 321 | alligator snake other 0 322 | vertebrate snake rhyper -1 323 | snake establishment other 0 324 | desk artefact hyper 1 325 | desk furniture hyper 1 326 | desk furnishing hyper 1 327 | desk object hyper 1 328 | artifact desk rhyper -1 329 | wood desk other 0 330 | sofa desk other 0 331 | desk beating other 0 332 | falcon creature hyper 1 333 | falcon predator hyper 1 334 | falcon vertebrate hyper 1 335 | falcon bird hyper 1 336 | liberalization falcon other 0 337 | plumage falcon other 0 338 | falcon claw other 0 339 | falcon underdog other 0 340 | donkey herbivore hyper 1 341 | donkey creature hyper 1 342 | donkey vertebrate hyper 1 343 | donkey beast hyper 1 344 | donkey hair other 0 345 | mammal donkey rhyper -1 346 | snout donkey other 0 347 | artillery donkey other 0 348 | frigate vessel hyper 1 349 | frigate artifact hyper 1 350 | frigate vehicle hyper 1 351 | frigate ship hyper 1 352 | frigate craft hyper 1 353 | artefact frigate rhyper -1 354 | timetable frigate other 0 355 | starboard frigate other 0 356 | frigate raft other 0 357 | jet frigate other 0 358 | onion vegetable hyper 1 359 | onion produce hyper 1 360 | food onion rhyper -1 361 | onion dresser other 0 362 | sofa artefact hyper 1 363 | sofa artifact hyper 1 364 | sofa object hyper 1 365 | sofa furniture hyper 1 366 | furnishing sofa rhyper -1 367 | sofa declaration other 0 368 | wood sofa other 0 369 | bed sofa other 0 370 | gun object hyper 1 371 | gun arm hyper 1 372 | gun artifact hyper 1 373 | gun artefact hyper 1 374 | device gun rhyper -1 375 | bazooka gun other 0 376 | butt gun other 0 377 | gun shotgun other 0 378 | glove accessory hyper 1 379 | glove clothes hyper 1 380 | glove artifact hyper 1 381 | glove good hyper 1 382 | glove apparel hyper 1 383 | glove garment hyper 1 384 | glove artefact hyper 1 385 | glove wear hyper 1 386 | glove object hyper 1 387 | glove bonfire other 0 388 | jacket glove other 0 389 | covering glove rhyper -1 390 | clothing glove rhyper -1 391 | commodity glove rhyper -1 392 | crust glove other 0 393 | glove historian other 0 394 | glove sweatshirt other 0 395 | dress glove other 0 396 | bag object hyper 1 397 | bag luggage hyper 1 398 | bag artefact hyper 1 399 | bag accessory hyper 1 400 | bag incontinence other 0 401 | bottom bag other 0 402 | circumcision bag other 0 403 | artifact bag rhyper -1 404 | hatchet weapon hyper 1 405 | hatchet cutlery hyper 1 406 | hatchet utensil hyper 1 407 | hatchet artifact hyper 1 408 | hatchet artefact hyper 1 409 | tool hatchet rhyper -1 410 | hatchet lesson other 0 411 | hatchet iron other 0 412 | hatchet pincer other 0 413 | scissors hatchet other 0 414 | saxophone instrument hyper 1 415 | saxophone artefact hyper 1 416 | saxophone object hyper 1 417 | saxophone artifact hyper 1 418 | key saxophone other 0 419 | device saxophone rhyper -1 420 | drum saxophone other 0 421 | saxophone fare other 0 422 | car artifact hyper 1 423 | car conveyance hyper 1 424 | car vehicle hyper 1 425 | car artefact hyper 1 426 | car goody other 0 427 | trunk car other 0 428 | transport car rhyper -1 429 | bicycle car other 0 430 | tanker artefact hyper 1 431 | tanker craft hyper 1 432 | tanker artifact hyper 1 433 | tanker ship hyper 1 434 | van tanker other 0 435 | tanker hull other 0 436 | tanker ferry other 0 437 | racism tanker other 0 438 | motorcycle vehicle hyper 1 439 | motorcycle artefact hyper 1 440 | motorcycle transport hyper 1 441 | motorcycle conveyance hyper 1 442 | motorcycle sauce other 0 443 | artifact motorcycle rhyper -1 444 | ferry motorcycle other 0 445 | oil motorcycle other 0 446 | turtle amphibian hyper 1 447 | turtle animal hyper 1 448 | turtle vertebrate hyper 1 449 | turtle beast hyper 1 450 | turtle reptile hyper 1 451 | turtle creature hyper 1 452 | salary turtle other 0 453 | pet turtle rhyper -1 454 | frog turtle other 0 455 | turtle eye other 0 456 | scale turtle other 0 457 | food turtle rhyper -1 458 | villa construction hyper 1 459 | villa house hyper 1 460 | villa dwelling hyper 1 461 | villa edifice hyper 1 462 | villa lodging hyper 1 463 | villa habitation hyper 1 464 | villa site hyper 1 465 | villa accommodation hyper 1 466 | villa structure hyper 1 467 | inn villa other 0 468 | building villa rhyper -1 469 | villa speed other 0 470 | housing villa rhyper -1 471 | villa wood other 0 472 | warehouse villa other 0 473 | roof villa other 0 474 | villa pub other 0 475 | ceiling villa other 0 476 | crow bird hyper 1 477 | crow animal hyper 1 478 | crow creature hyper 1 479 | crow vertebrate hyper 1 480 | crow coward other 0 481 | scavenger crow rhyper -1 482 | swan crow other 0 483 | beak crow other 0 484 | acacia tree hyper 1 485 | root acacia other 0 486 | grasshopper insect hyper 1 487 | grasshopper creature hyper 1 488 | grasshopper bug hyper 1 489 | grasshopper childhood other 0 490 | leg grasshopper other 0 491 | moth grasshopper other 0 492 | library edifice hyper 1 493 | library site hyper 1 494 | library building hyper 1 495 | library construction hyper 1 496 | elevator library other 0 497 | institution library rhyper -1 498 | structure library rhyper -1 499 | problem library other 0 500 | goose waterfowl hyper 1 501 | goose bird hyper 1 502 | goose creature hyper 1 503 | goose animal hyper 1 504 | plumage goose other 0 505 | hawk goose other 0 506 | vertebrate goose rhyper -1 507 | goose continuum other 0 508 | lion vertebrate hyper 1 509 | lion predator hyper 1 510 | lion carnivore hyper 1 511 | lion creature hyper 1 512 | lion mammal hyper 1 513 | rat lion other 0 514 | lion mouth other 0 515 | implementer lion other 0 516 | beast lion rhyper -1 517 | paw lion other 0 518 | turnip root hyper 1 519 | turnip food hyper 1 520 | turnip produce hyper 1 521 | carrot turnip other 0 522 | bulb turnip other 0 523 | turnip dyslexia other 0 524 | beetle animal hyper 1 525 | beetle insect hyper 1 526 | beetle bug hyper 1 527 | head beetle other 0 528 | creature beetle rhyper -1 529 | wasp beetle other 0 530 | cathedral building hyper 1 531 | cathedral church hyper 1 532 | cathedral site hyper 1 533 | cathedral edifice hyper 1 534 | cathedral structure hyper 1 535 | cathedral pulse other 0 536 | cathedral pub other 0 537 | stair cathedral other 0 538 | construction cathedral rhyper -1 539 | cathedral candle other 0 540 | hammer tool hyper 1 541 | hammer artifact hyper 1 542 | hammer utensil hyper 1 543 | hammer object hyper 1 544 | scissors hammer other 0 545 | artefact hammer rhyper -1 546 | hammer nausea other 0 547 | wood hammer other 0 548 | blouse good hyper 1 549 | blouse wear hyper 1 550 | blouse object hyper 1 551 | blouse clothes hyper 1 552 | blouse garment hyper 1 553 | blouse apparel hyper 1 554 | blouse commodity hyper 1 555 | blouse clothing hyper 1 556 | artefact blouse rhyper -1 557 | button blouse other 0 558 | blouse spider other 0 559 | blouse arm other 0 560 | hat blouse other 0 561 | silk blouse other 0 562 | artifact blouse rhyper -1 563 | cloak blouse other 0 564 | saw object hyper 1 565 | saw utensil hyper 1 566 | saw tool hyper 1 567 | saw artifact hyper 1 568 | saw herat other 0 569 | steel saw other 0 570 | artefact saw rhyper -1 571 | fork saw other 0 572 | pigeon creature hyper 1 573 | pigeon animal hyper 1 574 | pigeon vertebrate hyper 1 575 | bird pigeon rhyper -1 576 | owl pigeon other 0 577 | plume pigeon other 0 578 | revolver object hyper 1 579 | revolver weapon hyper 1 580 | revolver arm hyper 1 581 | revolver pistol hyper 1 582 | revolver gun hyper 1 583 | revolver artifact hyper 1 584 | revolver device hyper 1 585 | muzzle revolver other 0 586 | revolver merchandising other 0 587 | revolver steel other 0 588 | kalashnikov revolver other 0 589 | artefact revolver rhyper -1 590 | revolver bazooka other 0 591 | firearm revolver rhyper -1 592 | mug artifact hyper 1 593 | mug artefact hyper 1 594 | mug kitchenware hyper 1 595 | mug container hyper 1 596 | mug object hyper 1 597 | mug whisky other 0 598 | exterior mug other 0 599 | box mug other 0 600 | mug bottle other 0 601 | vessel mug rhyper -1 602 | washer artefact hyper 1 603 | washer appliance hyper 1 604 | washer object hyper 1 605 | washer commodity hyper 1 606 | washer device hyper 1 607 | washer steel other 0 608 | washer refrigerator other 0 609 | good washer rhyper -1 610 | washer affluence other 0 611 | artifact washer rhyper -1 612 | pistol artifact hyper 1 613 | pistol arm hyper 1 614 | pistol artefact hyper 1 615 | pistol object hyper 1 616 | pistol weapon hyper 1 617 | pistol device hyper 1 618 | pistol initiative other 0 619 | gun pistol rhyper -1 620 | priest pistol other 0 621 | pistol cartridge other 0 622 | pistol hatchet other 0 623 | holder pistol other 0 624 | carp animal hyper 1 625 | carp seafood hyper 1 626 | carp vertebrate hyper 1 627 | carp creature hyper 1 628 | trouble carp other 0 629 | carp trespasser other 0 630 | cod carp other 0 631 | fish carp rhyper -1 632 | knife tool hyper 1 633 | knife kitchenware hyper 1 634 | knife weapon hyper 1 635 | knife utensil hyper 1 636 | knife artefact hyper 1 637 | knife tableware hyper 1 638 | knife fork other 0 639 | artifact knife rhyper -1 640 | ketchup knife other 0 641 | iron knife other 0 642 | cutlery knife rhyper -1 643 | knife foreboding other 0 644 | trout fish hyper 1 645 | trout vertebrate hyper 1 646 | trout food hyper 1 647 | trout creature hyper 1 648 | animal trout rhyper -1 649 | seafood trout rhyper -1 650 | trout phobia other 0 651 | whale trout other 0 652 | eagle vertebrate hyper 1 653 | eagle creature hyper 1 654 | eagle raptor hyper 1 655 | eagle predator hyper 1 656 | animal eagle rhyper -1 657 | bird eagle rhyper -1 658 | claw eagle other 0 659 | generation eagle other 0 660 | cauliflower food hyper 1 661 | cauliflower produce hyper 1 662 | spinach cauliflower other 0 663 | cauliflower rainbow other 0 664 | salmon fish hyper 1 665 | salmon animal hyper 1 666 | salmon food hyper 1 667 | salmon vertebrate hyper 1 668 | salmon head other 0 669 | cod salmon other 0 670 | seafood salmon rhyper -1 671 | demonstration salmon other 0 672 | corn vegetable hyper 1 673 | corn food hyper 1 674 | corn cereal hyper 1 675 | corn produce hyper 1 676 | ear corn other 0 677 | grain corn rhyper -1 678 | onion corn other 0 679 | corn infertility other 0 680 | pub institution hyper 1 681 | pub structure hyper 1 682 | pub edifice hyper 1 683 | pub construction hyper 1 684 | pub deity other 0 685 | inn pub other 0 686 | building pub rhyper -1 687 | brick pub other 0 688 | guitar instrument hyper 1 689 | guitar device hyper 1 690 | guitar artifact hyper 1 691 | guitar artefact hyper 1 692 | bottom guitar other 0 693 | object guitar rhyper -1 694 | guitar liturgy other 0 695 | bagpipe guitar other 0 696 | cockroach pest hyper 1 697 | cockroach bug hyper 1 698 | cockroach creature hyper 1 699 | cockroach insect hyper 1 700 | animal cockroach rhyper -1 701 | leg cockroach other 0 702 | cockroach bust other 0 703 | mosquito cockroach other 0 704 | axe tool hyper 1 705 | axe cutlery hyper 1 706 | axe artefact hyper 1 707 | axe artifact hyper 1 708 | axe utensil hyper 1 709 | axe clearance other 0 710 | spoon axe other 0 711 | launcher axe other 0 712 | weapon axe rhyper -1 713 | object axe rhyper -1 714 | goldfish vertebrate hyper 1 715 | goldfish pet hyper 1 716 | goldfish creature hyper 1 717 | goldfish animal hyper 1 718 | salmon goldfish other 0 719 | fish goldfish rhyper -1 720 | goldfish guitar other 0 721 | gill goldfish other 0 722 | sieve object hyper 1 723 | sieve kitchenware hyper 1 724 | sieve tool hyper 1 725 | sieve artifact hyper 1 726 | sieve filter hyper 1 727 | spade sieve other 0 728 | sieve glass other 0 729 | artefact sieve rhyper -1 730 | metal sieve other 0 731 | sieve frame other 0 732 | box artifact hyper 1 733 | box artefact hyper 1 734 | box container hyper 1 735 | wood box other 0 736 | box mortality other 0 737 | object box rhyper -1 738 | dishwasher good hyper 1 739 | dishwasher object hyper 1 740 | dishwasher appliance hyper 1 741 | dishwasher artefact hyper 1 742 | dishwasher device hyper 1 743 | dishwasher fridge other 0 744 | inside dishwasher other 0 745 | availability dishwasher other 0 746 | dishwasher masonry other 0 747 | washer dishwasher other 0 748 | cow beast hyper 1 749 | cow cattle hyper 1 750 | cow animal hyper 1 751 | cow mammal hyper 1 752 | cow creature hyper 1 753 | cow ruminant hyper 1 754 | eye cow other 0 755 | cow petition other 0 756 | herbivore cow rhyper -1 757 | vertebrate cow rhyper -1 758 | buffalo cow other 0 759 | cow hoof other 0 760 | restaurant construction hyper 1 761 | restaurant edifice hyper 1 762 | restaurant structure hyper 1 763 | inn restaurant other 0 764 | building restaurant rhyper -1 765 | restaurant battle other 0 766 | plum produce hyper 1 767 | plum fruit hyper 1 768 | plum gunman other 0 769 | pear plum other 0 770 | grenade object hyper 1 771 | grenade arm hyper 1 772 | grenade device hyper 1 773 | grenade weapon hyper 1 774 | artefact grenade rhyper -1 775 | grenade glucose other 0 776 | shell grenade other 0 777 | artifact grenade rhyper -1 778 | cod fish hyper 1 779 | cod vertebrate hyper 1 780 | cod food hyper 1 781 | cod seafood hyper 1 782 | cod gender other 0 783 | creature cod rhyper -1 784 | tail cod other 0 785 | whale cod other 0 786 | cranberry produce hyper 1 787 | cranberry berry hyper 1 788 | cranberry fruit hyper 1 789 | juice cranberry other 0 790 | food cranberry rhyper -1 791 | cranberry quarantine other 0 792 | garlic seasoning hyper 1 793 | garlic spice hyper 1 794 | garlic food hyper 1 795 | garlic produce hyper 1 796 | clove garlic other 0 797 | garlic hallway other 0 798 | cucumber garlic other 0 799 | vegetable garlic rhyper -1 800 | castle fortification hyper 1 801 | castle lodging hyper 1 802 | castle building hyper 1 803 | castle home hyper 1 804 | castle defence hyper 1 805 | castle habitation hyper 1 806 | castle edifice hyper 1 807 | castle site hyper 1 808 | castle structure hyper 1 809 | castle house hyper 1 810 | castle defense hyper 1 811 | castle housing hyper 1 812 | castle restaurant other 0 813 | villa castle other 0 814 | hotel castle other 0 815 | guy castle other 0 816 | castle season other 0 817 | construction castle rhyper -1 818 | gate castle other 0 819 | castle door other 0 820 | castle necessity other 0 821 | visitor castle other 0 822 | dwelling castle rhyper -1 823 | accommodation castle rhyper -1 824 | sweater clothing hyper 1 825 | sweater good hyper 1 826 | sweater covering hyper 1 827 | sweater apparel hyper 1 828 | sweater garment hyper 1 829 | sweater wear hyper 1 830 | sweater artefact hyper 1 831 | sweater object hyper 1 832 | pattern sweater other 0 833 | fold sweater other 0 834 | ridge sweater other 0 835 | stocking sweater other 0 836 | artifact sweater rhyper -1 837 | commodity sweater rhyper -1 838 | sweater underpants other 0 839 | sweater button other 0 840 | scarf artefact hyper 1 841 | scarf apparel hyper 1 842 | scarf object hyper 1 843 | scarf artifact hyper 1 844 | scarf accessory hyper 1 845 | scarf commodity hyper 1 846 | scarf covering hyper 1 847 | scarf garment hyper 1 848 | scarf good hyper 1 849 | scarf fabric other 0 850 | wear scarf rhyper -1 851 | clothing scarf rhyper -1 852 | fraction scarf other 0 853 | cashmere scarf other 0 854 | scarf poem other 0 855 | logo scarf other 0 856 | scarf trouser other 0 857 | scarf garrison other 0 858 | owl predator hyper 1 859 | owl vertebrate hyper 1 860 | owl raptor hyper 1 861 | owl creature hyper 1 862 | vulture owl other 0 863 | animal owl rhyper -1 864 | bird owl rhyper -1 865 | owl veracity other 0 866 | shovel artifact hyper 1 867 | shovel object hyper 1 868 | shovel utensil hyper 1 869 | shovel artefact hyper 1 870 | end shovel other 0 871 | tool shovel rhyper -1 872 | spade shovel other 0 873 | shovel smile other 0 874 | dagger object hyper 1 875 | dagger device hyper 1 876 | dagger artefact hyper 1 877 | dagger artifact hyper 1 878 | authorisation dagger other 0 879 | arm dagger rhyper -1 880 | dagger pommel other 0 881 | guard dagger other 0 882 | apple food hyper 1 883 | apple produce hyper 1 884 | plum apple other 0 885 | fruit apple rhyper -1 886 | spear arm hyper 1 887 | spear object hyper 1 888 | spear device hyper 1 889 | spear artefact hyper 1 890 | wizard spear other 0 891 | artifact spear rhyper -1 892 | dagger spear other 0 893 | end spear other 0 894 | wasp animal hyper 1 895 | wasp insect hyper 1 896 | wasp occupation other 0 897 | antenna wasp other 0 898 | coconut fruit hyper 1 899 | coconut food hyper 1 900 | produce coconut rhyper -1 901 | peach coconut other 0 902 | lettuce vegetable hyper 1 903 | lettuce produce hyper 1 904 | lettuce bond other 0 905 | food lettuce rhyper -1 906 | table artefact hyper 1 907 | table furniture hyper 1 908 | table artifact hyper 1 909 | table furnishing hyper 1 910 | table functioning other 0 911 | chair table other 0 912 | cloth table other 0 913 | object table rhyper -1 914 | herring creature hyper 1 915 | herring animal hyper 1 916 | herring food hyper 1 917 | herring seafood hyper 1 918 | londoner herring other 0 919 | fish herring rhyper -1 920 | vertebrate herring rhyper -1 921 | scale herring other 0 922 | tuna food hyper 1 923 | tuna seafood hyper 1 924 | tuna creature hyper 1 925 | tuna animal hyper 1 926 | tuna goldfish other 0 927 | language tuna other 0 928 | vertebrate tuna rhyper -1 929 | dolphin tuna other 0 930 | lime fruit hyper 1 931 | lime produce hyper 1 932 | lime food hyper 1 933 | lime amazement other 0 934 | citrus lime rhyper -1 935 | peach lime other 0 936 | sheep beast hyper 1 937 | sheep creature hyper 1 938 | sheep animal hyper 1 939 | sheep herbivore hyper 1 940 | sheep mammal hyper 1 941 | freight sheep other 0 942 | sheep snout other 0 943 | sheep cow other 0 944 | vertebrate sheep rhyper -1 945 | rabbit sheep other 0 946 | horse herbivore hyper 1 947 | horse beast hyper 1 948 | horse mammal hyper 1 949 | horse vertebrate hyper 1 950 | bit horse other 0 951 | horse head other 0 952 | horse circle other 0 953 | creature horse rhyper -1 954 | van conveyance hyper 1 955 | van transport hyper 1 956 | van vehicle hyper 1 957 | van artefact hyper 1 958 | artifact van rhyper -1 959 | van crime other 0 960 | windshield van other 0 961 | helicopter van other 0 962 | jet plane hyper 1 963 | jet artifact hyper 1 964 | jet aeroplane hyper 1 965 | jet vehicle hyper 1 966 | jet craft hyper 1 967 | jet conveyance hyper 1 968 | jet aircraft hyper 1 969 | jet artefact hyper 1 970 | transport jet rhyper -1 971 | car jet other 0 972 | airplane jet rhyper -1 973 | jet color other 0 974 | durability jet other 0 975 | jet window other 0 976 | windshield jet other 0 977 | jet ambulance other 0 978 | spinach food hyper 1 979 | spinach vegetable hyper 1 980 | produce spinach rhyper -1 981 | iron spinach other 0 982 | alligator reptile hyper 1 983 | alligator creature hyper 1 984 | alligator predator hyper 1 985 | alligator beast hyper 1 986 | alligator carnivore hyper 1 987 | turtle alligator other 0 988 | alligator eye other 0 989 | alligator frog other 0 990 | alligator courthouse other 0 991 | animal alligator rhyper -1 992 | hospital structure hyper 1 993 | hospital building hyper 1 994 | hospital institution hyper 1 995 | hospital edifice hyper 1 996 | construction hospital rhyper -1 997 | elephant hospital other 0 998 | site hospital rhyper -1 999 | hospital patient other 0 1000 | oak tree hyper 1 1001 | elm oak other 0 1002 | parsley food hyper 1 1003 | parsley vegetable hyper 1 1004 | parsley produce hyper 1 1005 | parsley seasoning hyper 1 1006 | herb parsley rhyper -1 1007 | carrot parsley other 0 1008 | parsley minibus other 0 1009 | leaf parsley other 0 1010 | frog animal hyper 1 1011 | frog beast hyper 1 1012 | frog creature hyper 1 1013 | frog amphibian hyper 1 1014 | frog force other 0 1015 | toad frog other 0 1016 | vertebrate frog rhyper -1 1017 | eye frog other 0 1018 | mackerel fish hyper 1 1019 | mackerel vertebrate hyper 1 1020 | mackerel seafood hyper 1 1021 | mackerel food hyper 1 1022 | pharaoh mackerel other 0 1023 | creature mackerel rhyper -1 1024 | animal mackerel rhyper -1 1025 | mackerel dolphin other 0 1026 | beet root hyper 1 1027 | beet produce hyper 1 1028 | beet vegetable hyper 1 1029 | beet clinician other 0 1030 | food beet rhyper -1 1031 | leaf beet other 0 1032 | helicopter artefact hyper 1 1033 | helicopter aircraft hyper 1 1034 | helicopter artifact hyper 1 1035 | helicopter craft hyper 1 1036 | helicopter conveyance hyper 1 1037 | helicopter top other 0 1038 | helicopter orientation other 0 1039 | vehicle helicopter rhyper -1 1040 | truck helicopter other 0 1041 | hymn helicopter other 0 1042 | banana fruit hyper 1 1043 | banana food hyper 1 1044 | peel banana other 0 1045 | pineapple banana other 0 1046 | fork cutlery hyper 1 1047 | fork artifact hyper 1 1048 | fork utensil hyper 1 1049 | fork artefact hyper 1 1050 | fork tool hyper 1 1051 | fork object hyper 1 1052 | kitchenware fork rhyper -1 1053 | hammer fork other 0 1054 | fork hatchet other 0 1055 | plastic fork other 0 1056 | fork turnkey other 0 1057 | fork tooth other 0 1058 | squirrel rodent hyper 1 1059 | squirrel vertebrate hyper 1 1060 | squirrel creature hyper 1 1061 | squirrel animal hyper 1 1062 | beast squirrel rhyper -1 1063 | rabbit squirrel other 0 1064 | ear squirrel other 0 1065 | squirrel goat other 0 1066 | bus artefact hyper 1 1067 | bus conveyance hyper 1 1068 | bus artifact hyper 1 1069 | bus vehicle hyper 1 1070 | transport bus rhyper -1 1071 | battleship bus other 0 1072 | metal bus other 0 1073 | bus contracting other 0 1074 | cypress plant hyper 1 1075 | cypress tree hyper 1 1076 | birch cypress other 0 1077 | evergreen cypress rhyper -1 1078 | hotel lodging hyper 1 1079 | hotel edifice hyper 1 1080 | hotel accommodation hyper 1 1081 | hotel building hyper 1 1082 | construction hotel rhyper -1 1083 | hotel pub other 0 1084 | store hotel other 0 1085 | tightrope hotel other 0 1086 | missile arm hyper 1 1087 | missile weapon hyper 1 1088 | missile artifact hyper 1 1089 | missile rocket hyper 1 1090 | missile artefact hyper 1 1091 | missile vehicle hyper 1 1092 | object missile rhyper -1 1093 | missile warhead other 0 1094 | missile grenade other 0 1095 | engine missile other 0 1096 | scholar missile other 0 1097 | dagger missile other 0 1098 | tiger beast hyper 1 1099 | tiger creature hyper 1 1100 | tiger carnivore hyper 1 1101 | tiger mammal hyper 1 1102 | tiger animal hyper 1 1103 | maneuver tiger other 0 1104 | vertebrate tiger rhyper -1 1105 | tiger jaw other 0 1106 | predator tiger rhyper -1 1107 | tiger homeland other 0 1108 | phone gadget hyper 1 1109 | phone commodity hyper 1 1110 | phone good hyper 1 1111 | phone system hyper 1 1112 | phone artefact hyper 1 1113 | phone artifact hyper 1 1114 | phone device hyper 1 1115 | phone equipment hyper 1 1116 | dial phone other 0 1117 | appliance phone rhyper -1 1118 | fridge phone other 0 1119 | object phone rhyper -1 1120 | sanctity phone other 0 1121 | phone washer other 0 1122 | phone cable other 0 1123 | phone artillery other 0 1124 | train vehicle hyper 1 1125 | train conveyance hyper 1 1126 | train artifact hyper 1 1127 | train artefact hyper 1 1128 | transport train rhyper -1 1129 | yacht train other 0 1130 | back train other 0 1131 | train anything other 0 1132 | rabbit animal hyper 1 1133 | rabbit creature hyper 1 1134 | rabbit mammal hyper 1 1135 | rabbit vertebrate hyper 1 1136 | rabbit herbivore hyper 1 1137 | rabbit beast hyper 1 1138 | licence rabbit other 0 1139 | deer rabbit other 0 1140 | rabbit leg other 0 1141 | rabbit rat other 0 1142 | rabbit audit other 0 1143 | food rabbit rhyper -1 1144 | bull vertebrate hyper 1 1145 | bull ruminant hyper 1 1146 | bull creature hyper 1 1147 | bull mammal hyper 1 1148 | bull animal hyper 1 1149 | bull tail other 0 1150 | beef bull other 0 1151 | bull cluster other 0 1152 | beast bull rhyper -1 1153 | tiger bull other 0 1154 | wardrobe object hyper 1 1155 | wardrobe artefact hyper 1 1156 | wardrobe furniture hyper 1 1157 | wardrobe artifact hyper 1 1158 | furnishing wardrobe rhyper -1 1159 | wardrobe characteristic other 0 1160 | chair wardrobe other 0 1161 | bottom wardrobe other 0 1162 | coyote creature hyper 1 1163 | coyote beast hyper 1 1164 | coyote mammal hyper 1 1165 | coyote carnivore hyper 1 1166 | coyote animal hyper 1 1167 | coyote predator hyper 1 1168 | vertebrate coyote rhyper -1 1169 | tail coyote other 0 1170 | canine coyote rhyper -1 1171 | coyote petal other 0 1172 | coyote head other 0 1173 | beaver coyote other 0 1174 | piano artefact hyper 1 1175 | piano furniture hyper 1 1176 | piano artifact hyper 1 1177 | piano device hyper 1 1178 | top piano other 0 1179 | trumpet piano other 0 1180 | piano investor other 0 1181 | instrument piano rhyper -1 1182 | penguin animal hyper 1 1183 | penguin creature hyper 1 1184 | penguin vertebrate hyper 1 1185 | penguin seabird hyper 1 1186 | plumage penguin other 0 1187 | bird penguin rhyper -1 1188 | penguin teddy other 0 1189 | crow penguin other 0 1190 | television equipment hyper 1 1191 | television object hyper 1 1192 | television commodity hyper 1 1193 | television system hyper 1 1194 | television good hyper 1 1195 | television artifact hyper 1 1196 | television artefact hyper 1 1197 | outing television other 0 1198 | circuit television other 0 1199 | device television rhyper -1 1200 | appliance television rhyper -1 1201 | television channel other 0 1202 | television shovel other 0 1203 | telephone television other 0 1204 | apricot produce hyper 1 1205 | apricot fruit hyper 1 1206 | apricot republican other 0 1207 | grape apricot other 0 1208 | bed artefact hyper 1 1209 | bed furniture hyper 1 1210 | bed artifact hyper 1 1211 | bed furnishing hyper 1 1212 | object bed rhyper -1 1213 | pillow bed other 0 1214 | bed peninsula other 0 1215 | couch bed other 0 1216 | cat carnivore hyper 1 1217 | cat animal hyper 1 1218 | cat mammal hyper 1 1219 | cat creature hyper 1 1220 | cat pet hyper 1 1221 | cat inception other 0 1222 | cat paw other 0 1223 | eye cat other 0 1224 | beast cat rhyper -1 1225 | bear cat other 0 1226 | strawberry fruit hyper 1 1227 | strawberry produce hyper 1 1228 | lime strawberry other 0 1229 | strawberry alibi other 0 1230 | lizard animal hyper 1 1231 | lizard beast hyper 1 1232 | lizard vertebrate hyper 1 1233 | lizard creature hyper 1 1234 | lizard alligator other 0 1235 | stripe lizard other 0 1236 | chameleon lizard other 0 1237 | lizard eye other 0 1238 | cello object hyper 1 1239 | cello artefact hyper 1 1240 | cello artifact hyper 1 1241 | cello device hyper 1 1242 | bout cello other 0 1243 | instrument cello rhyper -1 1244 | cello parking other 0 1245 | bass cello other 0 1246 | ferry artefact hyper 1 1247 | ferry vessel hyper 1 1248 | ferry vehicle hyper 1 1249 | ferry boat hyper 1 1250 | engine ferry other 0 1251 | ferry outfielder other 0 1252 | canoe ferry other 0 1253 | artifact ferry rhyper -1 1254 | robin animal hyper 1 1255 | robin vertebrate hyper 1 1256 | robin creature hyper 1 1257 | bird robin rhyper -1 1258 | foot robin other 0 1259 | swan robin other 0 1260 | hornet creature hyper 1 1261 | hornet insect hyper 1 1262 | hornet effectiveness other 0 1263 | animal hornet rhyper -1 1264 | whale creature hyper 1 1265 | whale mammal hyper 1 1266 | whale seafood hyper 1 1267 | whale food hyper 1 1268 | whale vertebrate hyper 1 1269 | whale donkey other 0 1270 | whale windshield other 0 1271 | cetacean whale rhyper -1 1272 | whale tooth other 0 1273 | liar whale other 0 1274 | vest covering hyper 1 1275 | vest garment hyper 1 1276 | vest clothing hyper 1 1277 | vest artefact hyper 1 1278 | vest clothes hyper 1 1279 | vest apparel hyper 1 1280 | vest commodity hyper 1 1281 | vest good hyper 1 1282 | stocking vest other 0 1283 | vest skin other 0 1284 | artifact vest rhyper -1 1285 | vest pc other 0 1286 | tobacco vest other 0 1287 | object vest rhyper -1 1288 | silk vest other 0 1289 | vest trousers other 0 1290 | moth insect hyper 1 1291 | moth animal hyper 1 1292 | moth dairy other 0 1293 | body moth other 0 1294 | dolphin mammal hyper 1 1295 | dolphin cetacean hyper 1 1296 | dolphin animal hyper 1 1297 | dolphin vertebrate hyper 1 1298 | skin dolphin other 0 1299 | dolphin parliament other 0 1300 | lion dolphin other 0 1301 | creature dolphin rhyper -1 1302 | deer vertebrate hyper 1 1303 | deer herbivore hyper 1 1304 | deer beast hyper 1 1305 | deer ruminant hyper 1 1306 | deer creature hyper 1 1307 | deer ear other 0 1308 | deer bear other 0 1309 | infiltration deer other 0 1310 | mammal deer rhyper -1 1311 | deer bullshit other 0 1312 | pig beast hyper 1 1313 | pig vertebrate hyper 1 1314 | pig animal hyper 1 1315 | pig mammal hyper 1 1316 | pig sheep other 0 1317 | pig skin other 0 1318 | pig counterpart other 0 1319 | creature pig rhyper -1 1320 | ant insect hyper 1 1321 | ant animal hyper 1 1322 | ant bug hyper 1 1323 | creature ant rhyper -1 1324 | leg ant other 0 1325 | bee ant other 0 1326 | rat beast hyper 1 1327 | rat mammal hyper 1 1328 | rat animal hyper 1 1329 | rat creature hyper 1 1330 | rat cow other 0 1331 | rat snout other 0 1332 | rodent rat rhyper -1 1333 | tooth rat other 0 1334 | carrot produce hyper 1 1335 | carrot root hyper 1 1336 | carrot food hyper 1 1337 | cucumber carrot other 0 1338 | vitamin carrot other 0 1339 | carrot ssa other 0 1340 | violin instrument hyper 1 1341 | violin artefact hyper 1 1342 | violin device hyper 1 1343 | violin object hyper 1 1344 | artifact violin rhyper -1 1345 | trumpet violin other 0 1346 | violin club other 0 1347 | body violin other 0 1348 | beaver creature hyper 1 1349 | beaver mammal hyper 1 1350 | beaver rodent hyper 1 1351 | beaver animal hyper 1 1352 | vertebrate beaver rhyper -1 1353 | fur beaver other 0 1354 | beaver eye other 0 1355 | rat beaver other 0 1356 | woodpecker animal hyper 1 1357 | woodpecker creature hyper 1 1358 | woodpecker vertebrate hyper 1 1359 | woodpecker wheel other 0 1360 | bird woodpecker rhyper -1 1361 | hawk woodpecker other 0 1362 | dress garment hyper 1 1363 | dress object hyper 1 1364 | dress wear hyper 1 1365 | dress apparel hyper 1 1366 | dress clothes hyper 1 1367 | dress artefact hyper 1 1368 | dress artifact hyper 1 1369 | dress clothing hyper 1 1370 | gia dress other 0 1371 | stocking dress other 0 1372 | dress shirt other 0 1373 | dress zone other 0 1374 | dress lace other 0 1375 | jacket dress other 0 1376 | commodity dress rhyper -1 1377 | bodice dress other 0 1378 | spade artefact hyper 1 1379 | spade utensil hyper 1 1380 | spade tool hyper 1 1381 | spade object hyper 1 1382 | artifact spade rhyper -1 1383 | spade submissiveness other 0 1384 | axe spade other 0 1385 | iron spade other 0 1386 | lemon food hyper 1 1387 | lemon produce hyper 1 1388 | lemon citrus hyper 1 1389 | vitamin lemon other 0 1390 | fruit lemon rhyper -1 1391 | lemon disguise other 0 1392 | shirt good hyper 1 1393 | shirt artifact hyper 1 1394 | shirt covering hyper 1 1395 | shirt artefact hyper 1 1396 | shirt garment hyper 1 1397 | shirt commodity hyper 1 1398 | shirt clothes hyper 1 1399 | shirt object hyper 1 1400 | collar shirt other 0 1401 | jacket shirt other 0 1402 | shirt hieroglyphic other 0 1403 | shirt pattern other 0 1404 | shirt legend other 0 1405 | shirt underpants other 0 1406 | apparel shirt rhyper -1 1407 | barrel shirt other 0 1408 | peach produce hyper 1 1409 | peach fruit hyper 1 1410 | juice peach other 0 1411 | apple peach other 0 1412 | trumpet instrument hyper 1 1413 | trumpet artifact hyper 1 1414 | trumpet device hyper 1 1415 | trumpet artefact hyper 1 1416 | object trumpet rhyper -1 1417 | guitar trumpet other 0 1418 | metal trumpet other 0 1419 | trumpet cyst other 0 1420 | bowl artifact hyper 1 1421 | bowl utensil hyper 1 1422 | bowl object hyper 1 1423 | bowl kitchenware hyper 1 1424 | bowl container hyper 1 1425 | trip bowl other 0 1426 | bowl jar other 0 1427 | bowl aim other 0 1428 | bowl inside other 0 1429 | artefact bowl rhyper -1 1430 | freezer good hyper 1 1431 | freezer appliance hyper 1 1432 | freezer artifact hyper 1 1433 | freezer artefact hyper 1 1434 | freezer device hyper 1 1435 | freezer television other 0 1436 | freezer directory other 0 1437 | commodity freezer rhyper -1 1438 | freezer plug other 0 1439 | supply freezer other 0 1440 | couch furniture hyper 1 1441 | couch furnishing hyper 1 1442 | couch object hyper 1 1443 | couch artefact hyper 1 1444 | artifact couch rhyper -1 1445 | table couch other 0 1446 | back couch other 0 1447 | couch snap other 0 1448 | catfish fish hyper 1 1449 | catfish vertebrate hyper 1 1450 | catfish seafood hyper 1 1451 | catfish food hyper 1 1452 | inflow catfish other 0 1453 | creature catfish rhyper -1 1454 | catfish gill other 0 1455 | trout catfish other 0 1456 | sword object hyper 1 1457 | sword artefact hyper 1 1458 | sword artifact hyper 1 1459 | sword arm hyper 1 1460 | pike sword other 0 1461 | advantage sword other 0 1462 | hilt sword other 0 1463 | sword magazine other 0 1464 | elephant animal hyper 1 1465 | elephant herbivore hyper 1 1466 | elephant vertebrate hyper 1 1467 | elephant creature hyper 1 1468 | elephant beaver other 0 1469 | coyote elephant other 0 1470 | elephant triumph other 0 1471 | mammal elephant rhyper -1 1472 | truck transport hyper 1 1473 | truck artifact hyper 1 1474 | truck artefact hyper 1 1475 | truck vehicle hyper 1 1476 | truck m other 0 1477 | ambulance truck other 0 1478 | inside truck other 0 1479 | conveyance truck rhyper -1 1480 | goat mammal hyper 1 1481 | goat food hyper 1 1482 | goat animal hyper 1 1483 | goat vertebrate hyper 1 1484 | goat herbivore hyper 1 1485 | goat beast hyper 1 1486 | goat antelope other 0 1487 | creature goat rhyper -1 1488 | goat ear other 0 1489 | overhaul goat other 0 1490 | ruminant goat rhyper -1 1491 | cat goat other 0 1492 | dresser furnishing hyper 1 1493 | dresser artefact hyper 1 1494 | dresser artifact hyper 1 1495 | dresser furniture hyper 1 1496 | clothes dresser other 0 1497 | dresser plating other 0 1498 | chair dresser other 0 1499 | object dresser rhyper -1 1500 | butterfly creature hyper 1 1501 | butterfly insect hyper 1 1502 | cricket butterfly other 0 1503 | butterfly affidavit other 0 1504 | flute device hyper 1 1505 | flute artefact hyper 1 1506 | flute instrument hyper 1 1507 | flute artifact hyper 1 1508 | object flute rhyper -1 1509 | banjo flute other 0 1510 | flute o other 0 1511 | copper flute other 0 1512 | pear fruit hyper 1 1513 | pear produce hyper 1 1514 | skin pear other 0 1515 | food pear rhyper -1 1516 | screwdriver artefact hyper 1 1517 | screwdriver tool hyper 1 1518 | screwdriver object hyper 1 1519 | screwdriver utensil hyper 1 1520 | point screwdriver other 0 1521 | artifact screwdriver rhyper -1 1522 | screwdriver drain other 0 1523 | spoon screwdriver other 0 1524 | cucumber food hyper 1 1525 | cucumber vegetable hyper 1 1526 | produce cucumber rhyper -1 1527 | spinach cucumber other 0 1528 | birch tree hyper 1 1529 | flower birch other 0 1530 | cabbage produce hyper 1 1531 | cabbage food hyper 1 1532 | head cabbage other 0 1533 | cabbage diy other 0 1534 | hawk predator hyper 1 1535 | hawk bird hyper 1 1536 | hawk creature hyper 1 1537 | hawk raptor hyper 1 1538 | animal hawk rhyper -1 1539 | eye hawk other 0 1540 | passenger hawk other 0 1541 | hawk head other 0 1542 | dove vertebrate hyper 1 1543 | dove bird hyper 1 1544 | dove animal hyper 1 1545 | foot dove other 0 1546 | penguin dove other 0 1547 | dove bounce other 0 1548 | broccoli produce hyper 1 1549 | broccoli food hyper 1 1550 | broccoli film other 0 1551 | stalk broccoli other 0 1552 | stereo device hyper 1 1553 | stereo commodity hyper 1 1554 | stereo system hyper 1 1555 | stereo appliance hyper 1 1556 | stereo object hyper 1 1557 | stereo equipment hyper 1 1558 | stereo artifact hyper 1 1559 | good stereo rhyper -1 1560 | ticket stereo other 0 1561 | antenna stereo other 0 1562 | artefact stereo rhyper -1 1563 | stereo reservation other 0 1564 | stereo amplifier other 0 1565 | computer stereo other 0 1566 | fighter airplane hyper 1 1567 | fighter transport hyper 1 1568 | fighter aeroplane hyper 1 1569 | fighter plane hyper 1 1570 | fighter artifact hyper 1 1571 | fighter conveyance hyper 1 1572 | fighter aircraft hyper 1 1573 | fighter warplane hyper 1 1574 | fighter lecturer other 0 1575 | interior fighter other 0 1576 | rocket fighter other 0 1577 | craft fighter rhyper -1 1578 | fighter truck other 0 1579 | volunteer fighter other 0 1580 | car fighter other 0 1581 | vehicle fighter rhyper -1 1582 | potato food hyper 1 1583 | potato produce hyper 1 1584 | potato vegetable hyper 1 1585 | potato stooge other 0 1586 | starch potato other 0 1587 | leek potato other 0 1588 | ambulance transport hyper 1 1589 | ambulance artefact hyper 1 1590 | ambulance conveyance hyper 1 1591 | ambulance artifact hyper 1 1592 | helicopter ambulance other 0 1593 | ambulance runner-up other 0 1594 | vehicle ambulance rhyper -1 1595 | metal ambulance other 0 1596 | coat garment hyper 1 1597 | coat artifact hyper 1 1598 | coat commodity hyper 1 1599 | coat artefact hyper 1 1600 | coat good hyper 1 1601 | coat clothes hyper 1 1602 | coat clothing hyper 1 1603 | coat covering hyper 1 1604 | fleece coat other 0 1605 | joke coat other 0 1606 | back coat other 0 1607 | coat hem other 0 1608 | wear coat rhyper -1 1609 | fabric coat other 0 1610 | object coat rhyper -1 1611 | coat opposition other 0 1612 | yacht vessel hyper 1 1613 | yacht artefact hyper 1 1614 | yacht artifact hyper 1 1615 | yacht craft hyper 1 1616 | yacht bus other 0 1617 | camcorder yacht other 0 1618 | boat yacht rhyper -1 1619 | car yacht other 0 1620 | jacket wear hyper 1 1621 | jacket good hyper 1 1622 | jacket clothing hyper 1 1623 | jacket garment hyper 1 1624 | jacket commodity hyper 1 1625 | jacket artifact hyper 1 1626 | jacket object hyper 1 1627 | jacket artefact hyper 1 1628 | arm jacket other 0 1629 | jacket scrutiny other 0 1630 | covering jacket rhyper -1 1631 | fleece jacket other 0 1632 | jacket raincoat other 0 1633 | justice jacket other 0 1634 | jacket pulse other 0 1635 | apparel jacket rhyper -1 1636 | gorilla animal hyper 1 1637 | gorilla vertebrate hyper 1 1638 | gorilla mammal hyper 1 1639 | gorilla beast hyper 1 1640 | gorilla primate hyper 1 1641 | ape gorilla rhyper -1 1642 | creature gorilla rhyper -1 1643 | gorilla elephant other 0 1644 | ear gorilla other 0 1645 | gorilla foot other 0 1646 | poplar plant hyper 1 1647 | poplar angiosperm hyper 1 1648 | tree poplar rhyper -1 1649 | oak poplar other 0 1650 | wrench utensil hyper 1 1651 | wrench object hyper 1 1652 | wrench artefact hyper 1 1653 | wrench tool hyper 1 1654 | steel wrench other 0 1655 | wrench croatia other 0 1656 | artifact wrench rhyper -1 1657 | spade wrench other 0 1658 | bottle artifact hyper 1 1659 | bottle artefact hyper 1 1660 | bottle container hyper 1 1661 | bottle object hyper 1 1662 | bottle bag other 0 1663 | bottle label other 0 1664 | microphone bottle other 0 1665 | exterior bottle other 0 1666 | pineapple food hyper 1 1667 | pineapple produce hyper 1 1668 | pineapple plan other 0 1669 | grapefruit pineapple other 0 1670 | -------------------------------------------------------------------------------- /data/wbless.tsv: -------------------------------------------------------------------------------- 1 | word1 word2 label relation fold 2 | frigate craft True hyper val 3 | trouble carp False other val 4 | fox mouth False other val 5 | foot robin False other val 6 | vest garment True hyper val 7 | beetle insect True hyper val 8 | axe utensil True hyper val 9 | chair scroll False other val 10 | end spear False other val 11 | assessor giraffe False other val 12 | conveyance scooter False other val 13 | car yacht False other val 14 | back robe False other val 15 | commencement hat False other val 16 | sheep cow False other val 17 | cloak greenbelt False other val 18 | eye cat False other val 19 | birch cypress False other val 20 | box container True hyper val 21 | trumpet artefact True hyper val 22 | frog creature True hyper val 23 | holder pistol False other val 24 | grapefruit re-assessment False other val 25 | pistol initiative False other val 26 | robin creature True hyper val 27 | sword artefact True hyper val 28 | box artifact True hyper val 29 | lemon food True hyper val 30 | coyote mammal True hyper val 31 | sword artifact True hyper val 32 | train artifact True hyper val 33 | claw eagle False other val 34 | commodity freezer False other val 35 | violin device True hyper val 36 | beast lion False other val 37 | body violin False other val 38 | pillow bed False other val 39 | artefact bowl False other val 40 | radio appliance True hyper val 41 | butt gun False other val 42 | lettuce bond False other val 43 | bottom wardrobe False other val 44 | wrench object True hyper val 45 | transport jet False other val 46 | stereo appliance True hyper val 47 | freezer plug False other val 48 | conveyance truck False other val 49 | tuna creature True hyper val 50 | object coat False other val 51 | onion produce True hyper val 52 | lizard animal True hyper val 53 | frog animal True hyper val 54 | radio wire False other val 55 | clothing scarf False other val 56 | hatchet iron False other val 57 | fish herring False other val 58 | mug artefact True hyper val 59 | saw herat False other val 60 | food cranberry False other val 61 | cannon chamber False other val 62 | steel oven False other val 63 | apparel jacket False other val 64 | snout donkey False other val 65 | donkey hair False other val 66 | glider front False other val 67 | artifact couch False other val 68 | cabbage food True hyper val 69 | bottom guitar False other val 70 | logo scarf False other val 71 | spoon container True hyper val 72 | cannon object True hyper val 73 | cello parking False other val 74 | overhaul goat False other val 75 | wasp animal True hyper val 76 | grapefruit produce True hyper val 77 | coat garment True hyper val 78 | fork utensil True hyper val 79 | dress shirt False other val 80 | dishwasher artefact True hyper val 81 | instrument piano False other val 82 | jacket shirt False other val 83 | cloak covering True hyper val 84 | missile arm True hyper val 85 | tuna food True hyper val 86 | saxophone instrument True hyper val 87 | building pub False other val 88 | availability dishwasher False other val 89 | windshield van False other val 90 | dishwasher good True hyper val 91 | cranberry berry True hyper val 92 | sieve glass False other val 93 | bomber transport True hyper val 94 | metal ambulance False other val 95 | plastic fork False other val 96 | point screwdriver False other val 97 | potato food True hyper val 98 | fur beaver False other val 99 | scooter necessity False other val 100 | motorcycle transport True hyper val 101 | helicopter conveyance True hyper val 102 | washer steel False other val 103 | jar artifact True hyper val 104 | garlic food True hyper val 105 | creature catfish False other val 106 | rabbit creature True hyper val 107 | tanker ship True hyper val 108 | plantation battleship False other val 109 | cathedral candle False other val 110 | scale turtle False other val 111 | hawk head False other val 112 | telephone fridge False other val 113 | bull mammal True hyper val 114 | train anything False other val 115 | creature pig False other val 116 | pike sword False other val 117 | toad frog False other val 118 | artifact spear False other val 119 | bus vehicle True hyper val 120 | glove wear True hyper val 121 | leaf beet False other val 122 | car fighter False other val 123 | metal trumpet False other val 124 | dishwasher fridge False other val 125 | inside hat False other val 126 | crow creature True hyper val 127 | pigeon animal True hyper val 128 | beetle animal True hyper val 129 | stove artefact True hyper val 130 | restaurant cottage False other val 131 | spoon object True hyper val 132 | building restaurant False other val 133 | antenna stereo False other val 134 | hat object True hyper val 135 | mug artifact True hyper val 136 | fuel scooter False other val 137 | trout creature True hyper val 138 | craft fighter False other val 139 | truck artefact True hyper val 140 | drum saxophone False other val 141 | hymn helicopter False other val 142 | dagger device True hyper val 143 | wasp occupation False other val 144 | bed sofa False other val 145 | peach produce True hyper val 146 | bookshop stove False other val 147 | frog amphibian True hyper val 148 | fork cutlery True hyper val 149 | antenna wasp False other val 150 | television artefact True hyper val 151 | vegetable garlic False other val 152 | coat clothes True hyper val 153 | bag accessory True hyper val 154 | animal alligator False other val 155 | swan vertebrate True hyper val 156 | chair table False other val 157 | cottage dwelling True hyper val 158 | good stereo False other val 159 | crow coward False other val 160 | table artifact True hyper val 161 | stair cathedral False other val 162 | chair dresser False other val 163 | goat herbivore True hyper val 164 | revolver arm True hyper val 165 | pistol cartridge False other val 166 | food pear False other val 167 | artifact hat False other val 168 | cherry food True hyper val 169 | castle necessity False other test 170 | fish carp False other test 171 | leg cockroach False other test 172 | starch potato False other test 173 | pear plum False other test 174 | creature fox False other test 175 | outing television False other test 176 | garlic spice True hyper test 177 | object bed False other test 178 | sieve artifact True hyper test 179 | rabbit beast True hyper test 180 | villa dwelling True hyper test 181 | weapon axe False other test 182 | phone equipment True hyper test 183 | jet conveyance True hyper test 184 | robe commentary False other test 185 | starboard frigate False other test 186 | cottage home True hyper test 187 | moped scooter False other test 188 | respect jar False other test 189 | television channel False other test 190 | food grapefruit False other test 191 | salmon animal True hyper test 192 | cod food True hyper test 193 | bear mammal True hyper test 194 | hat crown False other test 195 | butterfly insect True hyper test 196 | vertebrate snake False other test 197 | sparrow animal True hyper test 198 | gorilla primate True hyper test 199 | glider artefact True hyper test 200 | lettuce vegetable True hyper test 201 | penguin creature True hyper test 202 | carp vertebrate True hyper test 203 | bull animal True hyper test 204 | broccoli produce True hyper test 205 | cello artefact True hyper test 206 | owl vertebrate True hyper test 207 | frigate vessel True hyper test 208 | scarf apparel True hyper test 209 | rabbit sheep False other test 210 | bottle label False other test 211 | frog force False other test 212 | pharaoh mackerel False other test 213 | garlic hallway False other test 214 | glove sweatshirt False other test 215 | cedar sense False other test 216 | whale food True hyper test 217 | jar vessel True hyper test 218 | hotel castle False other test 219 | corn food True hyper test 220 | turtle eye False other test 221 | bird vulture False other test 222 | animal mackerel False other test 223 | washer device True hyper test 224 | library building True hyper test 225 | cathedral structure True hyper test 226 | bottle bag False other test 227 | stove migraine False other test 228 | penguin vertebrate True hyper test 229 | appliance phone False other test 230 | tuna animal True hyper test 231 | hatchet lesson False other test 232 | seafood salmon False other test 233 | lion creature True hyper test 234 | priest pistol False other test 235 | carp creature True hyper test 236 | problem library False other test 237 | yacht bus False other test 238 | lizard creature True hyper test 239 | robe artifact True hyper test 240 | helicopter artefact True hyper test 241 | creature beetle False other test 242 | shirt covering True hyper test 243 | wear coat False other test 244 | library construction True hyper test 245 | hatchet weapon True hyper test 246 | jacket object True hyper test 247 | artefact cannon False other test 248 | leg ant False other test 249 | fox animal True hyper test 250 | goose creature True hyper test 251 | interior fighter False other test 252 | bowl kitchenware True hyper test 253 | object rifle False other test 254 | tanker hull False other test 255 | flute artifact True hyper test 256 | scissors hatchet False other test 257 | hat accessory True hyper test 258 | grape reef False other test 259 | fox vertebrate True hyper test 260 | ant animal True hyper test 261 | vulture goose False other test 262 | beet root True hyper test 263 | sword object True hyper test 264 | good washer False other test 265 | washer dishwasher False other test 266 | fox predator True hyper test 267 | revolver pistol True hyper test 268 | flute artefact True hyper test 269 | butterfly creature True hyper test 270 | jet artifact True hyper test 271 | vertebrate cow False other test 272 | appliance oven False other test 273 | penguin seabird True hyper test 274 | kitchenware jar False other test 275 | vertebrate coyote False other test 276 | wrench tool True hyper test 277 | revolver weapon True hyper test 278 | spinach cucumber False other test 279 | scale herring False other test 280 | owl predator True hyper test 281 | knife tool True hyper test 282 | castle structure True hyper test 283 | car goody False other test 284 | lion mouth False other test 285 | hat headgear True hyper test 286 | knife utensil True hyper test 287 | axe clearance False other test 288 | strawberry produce True hyper test 289 | lion dolphin False other test 290 | artillery donkey False other test 291 | deer herbivore True hyper test 292 | cockroach creature True hyper test 293 | vertebrate vulture False other test 294 | cucumber garlic False other test 295 | cat carnivore True hyper test 296 | steel saw False other test 297 | banjo flute False other test 298 | cat creature True hyper test 299 | lodging cottage False other test 300 | frog beast True hyper test 301 | washer appliance True hyper test 302 | glove clothes True hyper test 303 | moth dairy False other test 304 | bag object True hyper test 305 | hotel edifice True hyper test 306 | chair artifact True hyper test 307 | owl raptor True hyper test 308 | device television False other test 309 | bus artifact True hyper test 310 | object table False other test 311 | fridge artifact True hyper test 312 | turtle animal True hyper test 313 | institution library False other test 314 | woodpecker vertebrate True hyper test 315 | object guitar False other test 316 | castle building True hyper test 317 | swan crow False other test 318 | rat creature True hyper test 319 | swan robin False other test 320 | pig sheep False other test 321 | button blouse False other test 322 | television object True hyper test 323 | corn cereal True hyper test 324 | artifact van False other test 325 | television system True hyper test 326 | bear cat False other test 327 | villa pub False other test 328 | freezer directory False other test 329 | tail cod False other test 330 | topside battleship False other test 331 | chameleon lizard False other test 332 | metal sieve False other test 333 | train artefact True hyper test 334 | something robe False other test 335 | oven good True hyper test 336 | apricot produce True hyper test 337 | goldfish creature True hyper test 338 | shirt underpants False other test 339 | couch furniture True hyper test 340 | herbivore cow False other test 341 | cloak apparel True hyper test 342 | bed furniture True hyper test 343 | cow hoof False other test 344 | shovel artefact True hyper test 345 | trumpet device True hyper test 346 | artefact sieve False other test 347 | pistol device True hyper test 348 | stocking sweater False other test 349 | oven import False other test 350 | bomber aeroplane True hyper test 351 | furnishing sofa False other test 352 | stocking vest False other test 353 | durability jet False other test 354 | goldfish animal True hyper test 355 | alligator frog False other test 356 | inside dishwasher False other test 357 | stool chair False other test 358 | deer beast True hyper test 359 | glider transport True hyper test 360 | dog bear False other test 361 | firearm revolver False other test 362 | hawk creature True hyper test 363 | strawberry fruit True hyper test 364 | falcon creature True hyper test 365 | goose continuum False other test 366 | catfish vertebrate True hyper test 367 | creature dolphin False other test 368 | appliance television False other test 369 | bird woodpecker False other test 370 | vessel mug False other test 371 | elephant beaver False other test 372 | grenade arm True hyper test 373 | yacht train False other test 374 | axe tool True hyper test 375 | radio artefact True hyper test 376 | truck transport True hyper test 377 | knife kitchenware True hyper test 378 | hat robe False other test 379 | spoon empathy False other test 380 | accommodation cottage False other test 381 | parsley produce True hyper test 382 | good cloak False other test 383 | swan waterfowl True hyper test 384 | copper flute False other test 385 | motorcycle artefact True hyper test 386 | vertebrate sheep False other test 387 | guitar artifact True hyper test 388 | vitamin lemon False other test 389 | glider aircraft True hyper test 390 | shovel utensil True hyper test 391 | blouse wear True hyper test 392 | airplane jet False other test 393 | grapefruit fruit True hyper test 394 | vertebrate frog False other test 395 | glove accessory True hyper test 396 | pet turtle False other test 397 | parsley vegetable True hyper test 398 | food lettuce False other test 399 | microphone bottle False other test 400 | radio canvas False other test 401 | scarf garrison False other test 402 | owner cottage False other test 403 | flute instrument True hyper test 404 | crow animal True hyper test 405 | radio stereo False other test 406 | shirt garment True hyper test 407 | whale creature True hyper test 408 | car artifact True hyper test 409 | transport train False other test 410 | shovel smile False other test 411 | cedar evergreen True hyper test 412 | villa wood False other test 413 | tiger bull False other test 414 | computer stereo False other test 415 | bear vertebrate True hyper test 416 | pub construction True hyper test 417 | swan overdraft False other test 418 | elephant animal True hyper test 419 | gun pistol False other test 420 | object robe False other test 421 | spoon axe False other test 422 | rabbit audit False other test 423 | salmon goldfish False other test 424 | battleship warship True hyper test 425 | truck artifact True hyper test 426 | grape food True hyper test 427 | garlic seasoning True hyper test 428 | dial phone False other test 429 | animal pheasant False other test 430 | sofa declaration False other test 431 | bottle object True hyper test 432 | ambulance runner-up False other test 433 | van conveyance True hyper test 434 | cloth table False other test 435 | pine evergreen True hyper test 436 | bazooka gun False other test 437 | herring creature True hyper test 438 | building villa False other test 439 | equipment radio False other test 440 | alligator eye False other test 441 | bomb weapon True hyper test 442 | wasp beetle False other test 443 | artifact radio False other test 444 | device saxophone False other test 445 | cauliflower rainbow False other test 446 | beak pheasant False other test 447 | motorcycle conveyance True hyper test 448 | castle home True hyper test 449 | tuna seafood True hyper test 450 | kitchenware spoon False other test 451 | food rabbit False other test 452 | animal trout False other test 453 | tanker craft True hyper test 454 | dresser artefact True hyper test 455 | bag luggage True hyper test 456 | revolver merchandising False other test 457 | train conveyance True hyper test 458 | horse vertebrate True hyper test 459 | ferry vessel True hyper test 460 | trip bowl False other test 461 | fleece coat False other test 462 | piano furniture True hyper test 463 | cottage housing True hyper test 464 | pineapple food True hyper test 465 | fork saw False other test 466 | fruit apple False other test 467 | cottage building True hyper test 468 | dishwasher device True hyper test 469 | warehouse villa False other test 470 | potato produce True hyper test 471 | lime fruit True hyper test 472 | fridge phone False other test 473 | covering glove False other test 474 | tanker ferry False other test 475 | elevator library False other test 476 | glove bonfire False other test 477 | scarf fabric False other test 478 | saxophone artifact True hyper test 479 | glove artifact True hyper test 480 | bull ruminant True hyper test 481 | missile rocket True hyper test 482 | vehicle bomber False other test 483 | pistol weapon True hyper test 484 | shirt clothes True hyper test 485 | goose bird True hyper test 486 | revolver artifact True hyper test 487 | stripe lizard False other test 488 | woodpecker creature True hyper test 489 | coat opposition False other test 490 | cannon device True hyper test 491 | fighter warplane True hyper test 492 | bird pigeon False other test 493 | radish vinyl False other test 494 | cow cattle True hyper test 495 | guitar trumpet False other test 496 | jacket garment True hyper test 497 | beetle bug True hyper test 498 | fighter conveyance True hyper test 499 | wardrobe characteristic False other test 500 | dolphin vertebrate True hyper test 501 | inside truck False other test 502 | blouse commodity True hyper test 503 | alligator reptile True hyper test 504 | pheasant game True hyper test 505 | owl pigeon False other test 506 | rifle firearm True hyper test 507 | good fridge False other test 508 | salmon fish True hyper test 509 | animal owl False other test 510 | bull creature True hyper test 511 | jet craft True hyper test 512 | carrot ssa False other test 513 | turnip dyslexia False other test 514 | cat pet True hyper test 515 | lion carnivore True hyper test 516 | onion dresser False other test 517 | bull tail False other test 518 | spinach food True hyper test 519 | cockroach bust False other test 520 | jet plane True hyper test 521 | mammal deer False other test 522 | spoon wrench False other test 523 | plum fruit True hyper test 524 | goldfish vertebrate True hyper test 525 | cat animal True hyper test 526 | trout fish True hyper test 527 | silk blouse False other test 528 | visitor castle False other test 529 | spoon screwdriver False other test 530 | inflow catfish False other test 531 | giraffe vertebrate True hyper test 532 | citrus lime False other test 533 | lemon citrus True hyper test 534 | advantage sword False other test 535 | dolphin tuna False other test 536 | turnip radish False other test 537 | spade submissiveness False other test 538 | cherry produce True hyper test 539 | falcon bird True hyper test 540 | battleship vehicle True hyper test 541 | fruit lemon False other test 542 | vehicle helicopter False other test 543 | herring seafood True hyper test 544 | villa site True hyper test 545 | yacht artefact True hyper test 546 | villa lodging True hyper test 547 | hornet insect True hyper test 548 | glove good True hyper test 549 | fighter aircraft True hyper test 550 | alcohol cottage False other test 551 | whale cod False other test 552 | robin animal True hyper test 553 | creature horse False other test 554 | fork tool True hyper test 555 | engine missile False other test 556 | villa structure True hyper test 557 | telephone television False other test 558 | mug bottle False other test 559 | potato vegetable True hyper test 560 | jacket raincoat False other test 561 | stalk broccoli False other test 562 | fridge device True hyper test 563 | vertebrate goose False other test 564 | carrot produce True hyper test 565 | vitamin carrot False other test 566 | glove historian False other test 567 | bull vertebrate True hyper test 568 | arm jacket False other test 569 | site hospital False other test 570 | jet vehicle True hyper test 571 | revolver device True hyper test 572 | hawk predator True hyper test 573 | chair wardrobe False other test 574 | sofa object True hyper test 575 | scholar missile False other test 576 | stereo equipment True hyper test 577 | ship battleship False other test 578 | robe ruminant False other test 579 | carrot root True hyper test 580 | hatchet cutlery True hyper test 581 | good stove False other test 582 | freezer artifact True hyper test 583 | screwdriver artefact True hyper test 584 | tiger homeland False other test 585 | giraffe violin False other test 586 | hammer utensil True hyper test 587 | garment cloak False other test 588 | stove object True hyper test 589 | canoe ferry False other test 590 | artifact motorcycle False other test 591 | dove bounce False other test 592 | fork tooth False other test 593 | cranberry quarantine False other test 594 | top piano False other test 595 | dress garment True hyper test 596 | shirt object True hyper test 597 | jar cookie False other test 598 | robe hat False other test 599 | sweater underpants False other test 600 | horse beast True hyper test 601 | fox mammal True hyper test 602 | volunteer fighter False other test 603 | pig animal True hyper test 604 | learner fridge False other test 605 | cottage habitation True hyper test 606 | jacket cloak False other test 607 | buffalo cow False other test 608 | turtle beast True hyper test 609 | cypress plant True hyper test 610 | fork hatchet False other test 611 | goat vertebrate True hyper test 612 | fruit cherry False other test 613 | fox giraffe False other test 614 | sparrow football False other test 615 | cloak artefact True hyper test 616 | plant elm False other test 617 | vertebrate beaver False other test 618 | oven device True hyper test 619 | freezer device True hyper test 620 | van crime False other test 621 | knife artefact True hyper test 622 | dress clothes True hyper test 623 | banana fruit True hyper test 624 | acacia tree True hyper test 625 | fraction scarf False other test 626 | sheep beast True hyper test 627 | artefact bomber False other test 628 | stocking dress False other test 629 | sweater wear True hyper test 630 | owl veracity False other test 631 | bee ant False other test 632 | tail fox False other test 633 | saw utensil True hyper test 634 | jacket scrutiny False other test 635 | vest artefact True hyper test 636 | bomber helicopter False other test 637 | chair object True hyper test 638 | robe covering True hyper test 639 | clothes hat False other test 640 | exterior bottle False other test 641 | beet produce True hyper test 642 | radish vegetable True hyper test 643 | circuit television False other test 644 | shovel artifact True hyper test 645 | hammer object True hyper test 646 | spade object True hyper test 647 | hornet effectiveness False other test 648 | jet artefact True hyper test 649 | authorisation dagger False other test 650 | snake reptile True hyper test 651 | villa accommodation True hyper test 652 | library edifice True hyper test 653 | coyote carnivore True hyper test 654 | lettuce produce True hyper test 655 | coat good True hyper test 656 | elephant herbivore True hyper test 657 | rifle weapon True hyper test 658 | phone gadget True hyper test 659 | creature giraffe False other test 660 | deer creature True hyper test 661 | scavenger crow False other test 662 | store hotel False other test 663 | carp trespasser False other test 664 | steel wrench False other test 665 | donkey beast True hyper test 666 | mackerel food True hyper test 667 | helicopter aircraft True hyper test 668 | falcon claw False other test 669 | whale windshield False other test 670 | bomber artifact True hyper test 671 | stereo commodity True hyper test 672 | cow ruminant True hyper test 673 | stove commodity True hyper test 674 | shirt artifact True hyper test 675 | animal hawk False other test 676 | cod vertebrate True hyper test 677 | dove pheasant False other test 678 | produce cucumber False other test 679 | wardrobe artefact True hyper test 680 | screwdriver utensil True hyper test 681 | blouse arm False other test 682 | stereo reservation False other test 683 | instrument cello False other test 684 | leek potato False other test 685 | artifact desk False other test 686 | artifact bomb False other test 687 | cloak trousers False other test 688 | hammer fork False other test 689 | bowl aim False other test 690 | transport bus False other test 691 | screwdriver drain False other test 692 | plum apple False other test 693 | tuna goldfish False other test 694 | bird sparrow False other test 695 | vest good True hyper test 696 | trout vertebrate True hyper test 697 | artifact screwdriver False other test 698 | pig counterpart False other test 699 | pineapple produce True hyper test 700 | guitar liturgy False other test 701 | apricot fruit True hyper test 702 | object dresser False other test 703 | sieve kitchenware True hyper test 704 | cod gender False other test 705 | jar plate False other test 706 | goat antelope False other test 707 | mosquito cockroach False other test 708 | back couch False other test 709 | object vest False other test 710 | bomber airplane True hyper test 711 | creature goat False other test 712 | boat yacht False other test 713 | coat artifact True hyper test 714 | iron spade False other test 715 | coat artefact True hyper test 716 | bird owl False other test 717 | robe garment True hyper test 718 | robe front False other test 719 | pine tooling False other test 720 | lime strawberry False other test 721 | artifact spade False other test 722 | revolver steel False other test 723 | animal eagle False other test 724 | produce grape False other test 725 | circumcision bag False other test 726 | grain corn False other test 727 | van vehicle True hyper test 728 | poplar plant True hyper test 729 | goose animal True hyper test 730 | ambulance truck False other test 731 | knife tableware True hyper test 732 | restaurant construction True hyper test 733 | saw artifact True hyper test 734 | cockroach insect True hyper test 735 | fridge test False other test 736 | eagle vertebrate True hyper test 737 | elephant creature True hyper test 738 | guitar instrument True hyper test 739 | desk furniture True hyper test 740 | maneuver tiger False other test 741 | shovel object True hyper test 742 | wood hammer False other test 743 | rat lion False other test 744 | pig beast True hyper test 745 | transport car False other test 746 | pheasant pakistani False other test 747 | eagle raptor True hyper test 748 | carp animal True hyper test 749 | television equipment True hyper test 750 | hospital edifice True hyper test 751 | dress zone False other test 752 | battleship lack False other test 753 | artifact sweater False other test 754 | eye cow False other test 755 | blouse apparel True hyper test 756 | dress lace False other test 757 | grape fruit True hyper test 758 | vertebrate tuna False other test 759 | cathedral edifice True hyper test 760 | herring animal True hyper test 761 | helicopter ambulance False other test 762 | horse mammal True hyper test 763 | accommodation castle False other test 764 | spear artefact True hyper test 765 | scooter vehicle True hyper test 766 | screwdriver tool True hyper test 767 | cabbage produce True hyper test 768 | cranberry grapefruit False other test 769 | alligator predator True hyper test 770 | bomb thoroughfare False other test 771 | grasshopper insect True hyper test 772 | couch furnishing True hyper test 773 | vertebrate herring False other test 774 | missile artifact True hyper test 775 | catfish seafood True hyper test 776 | glove apparel True hyper test 777 | cloak clothing True hyper test 778 | fish goldfish False other test 779 | blouse spider False other test 780 | tobacco vest False other test 781 | fuse bomb False other test 782 | car artefact True hyper test 783 | grenade glucose False other test 784 | produce spinach False other test 785 | goat beast True hyper test 786 | castle house True hyper test 787 | spear arm True hyper test 788 | device gun False other test 789 | phone artifact True hyper test 790 | peach coconut False other test 791 | bed peninsula False other test 792 | grenade device True hyper test 793 | jacket pulse False other test 794 | deer ruminant True hyper test 795 | ridge sweater False other test 796 | rat animal True hyper test 797 | beak crow False other test 798 | commodity dress False other test 799 | boy glider False other test 800 | hat garment True hyper test 801 | gun arm True hyper test 802 | dolphin parliament False other test 803 | carrot food True hyper test 804 | grapefruit pineapple False other test 805 | collar shirt False other test 806 | artifact bag False other test 807 | bird penguin False other test 808 | bomber setting False other test 809 | tightrope hotel False other test 810 | phone device True hyper test 811 | gun artifact True hyper test 812 | bear fur False other test 813 | root acacia False other test 814 | dress artefact True hyper test 815 | spoon head False other test 816 | donkey herbivore True hyper test 817 | end shovel False other test 818 | clove garlic False other test 819 | cat goat False other test 820 | couch object True hyper test 821 | glove artefact True hyper test 822 | robe artefact True hyper test 823 | ant insect True hyper test 824 | missile grenade False other test 825 | speaker radio False other test 826 | sheep animal True hyper test 827 | radish food True hyper test 828 | scarf good True hyper test 829 | battleship vessel True hyper test 830 | cloak clothes True hyper test 831 | deer bear False other test 832 | head beetle False other test 833 | squirrel rodent True hyper test 834 | elephant triumph False other test 835 | animal bear False other test 836 | bagpipe guitar False other test 837 | wizard spear False other test 838 | motorcycle sauce False other test 839 | car conveyance True hyper test 840 | food beet False other test 841 | jacket clothing True hyper test 842 | sweater object True hyper test 843 | grape apricot False other test 844 | squirrel animal True hyper test 845 | trumpet piano False other test 846 | mug kitchenware True hyper test 847 | saxophone artefact True hyper test 848 | snake establishment False other test 849 | stereo artifact True hyper test 850 | mug object True hyper test 851 | sword arm True hyper test 852 | washer refrigerator False other test 853 | whale vertebrate True hyper test 854 | hat blouse False other test 855 | moth grasshopper False other test 856 | joke coat False other test 857 | spoon artefact True hyper test 858 | grenade weapon True hyper test 859 | phone good True hyper test 860 | swan creature True hyper test 861 | villa habitation True hyper test 862 | television good True hyper test 863 | ferry boat True hyper test 864 | artifact violin False other test 865 | bird robin False other test 866 | dresser plating False other test 867 | frigate bomber False other test 868 | skin dolphin False other test 869 | juice cranberry False other test 870 | parsley minibus False other test 871 | elephant hospital False other test 872 | car vehicle True hyper test 873 | wear scarf False other test 874 | beast bull False other test 875 | dolphin cetacean True hyper test 876 | generation eagle False other test 877 | food turtle False other test 878 | bottle container True hyper test 879 | trout food True hyper test 880 | goat ear False other test 881 | beast fox False other test 882 | hospital building True hyper test 883 | cello device True hyper test 884 | alligator carnivore True hyper test 885 | pheasant bird True hyper test 886 | scarf covering True hyper test 887 | train vehicle True hyper test 888 | lion vertebrate True hyper test 889 | goat mammal True hyper test 890 | axe cutlery True hyper test 891 | oven timer False other test 892 | cod fish True hyper test 893 | fork object True hyper test 894 | paw lion False other test 895 | penguin teddy False other test 896 | jet color False other test 897 | coyote animal True hyper test 898 | object axe False other test 899 | bowl artifact True hyper test 900 | hammer tool True hyper test 901 | violin artefact True hyper test 902 | dove vertebrate True hyper test 903 | stove device True hyper test 904 | ferry motorcycle False other test 905 | hammer artifact True hyper test 906 | restaurant battle False other test 907 | falcon underdog False other test 908 | sparrow vertebrate True hyper test 909 | rabbit mammal True hyper test 910 | castle housing True hyper test 911 | radio object True hyper test 912 | iron knife False other test 913 | shirt legend False other test 914 | spoon cutlery True hyper test 915 | skin pear False other test 916 | saw object True hyper test 917 | knife foreboding False other test 918 | fox sub-committee False other test 919 | blouse clothing True hyper test 920 | outside hat False other test 921 | spade tool True hyper test 922 | grasshopper creature True hyper test 923 | sweater button False other test 924 | tooth snake False other test 925 | apple food True hyper test 926 | spear device True hyper test 927 | achiever cannon False other test 928 | violin instrument True hyper test 929 | grenade object True hyper test 930 | launcher axe False other test 931 | spade artefact True hyper test 932 | gate castle False other test 933 | construction cathedral False other test 934 | elm oak False other test 935 | tiger carnivore True hyper test 936 | rabbit squirrel False other test 937 | fighter truck False other test 938 | bear beast True hyper test 939 | swan bird True hyper test 940 | castle habitation True hyper test 941 | bus conveyance True hyper test 942 | radio system True hyper test 943 | stove artifact True hyper test 944 | dagger missile False other test 945 | cow beast True hyper test 946 | object phone False other test 947 | stereo object True hyper test 948 | goldfish pet True hyper test 949 | washer object True hyper test 950 | tail coyote False other test 951 | tree cedar False other test 952 | vulture scavenger True hyper test 953 | rabbit leg False other test 954 | language tuna False other test 955 | bag artefact True hyper test 956 | castle fortification True hyper test 957 | lizard eye False other test 958 | cushion chair False other test 959 | turtle vertebrate True hyper test 960 | liberalization falcon False other test 961 | spade shovel False other test 962 | butterfly affidavit False other test 963 | ketchup knife False other test 964 | bomber conveyance True hyper test 965 | strawberry alibi False other test 966 | bowl jar False other test 967 | lime produce True hyper test 968 | garlic produce True hyper test 969 | sieve tool True hyper test 970 | robin vertebrate True hyper test 971 | infiltration deer False other test 972 | deer bullshit False other test 973 | hat good True hyper test 974 | peel banana False other test 975 | artifact grenade False other test 976 | beaver creature True hyper test 977 | cod salmon False other test 978 | oven artefact True hyper test 979 | parsley seasoning True hyper test 980 | catfish gill False other test 981 | cypress tree True hyper test 982 | cockroach bug True hyper test 983 | alligator snake False other test 984 | robe clothing True hyper test 985 | roof villa False other test 986 | head cabbage False other test 987 | bomb arm True hyper test 988 | broccoli film False other test 989 | whale trout False other test 990 | library site True hyper test 991 | jacket artefact True hyper test 992 | scarf artifact True hyper test 993 | device rifle False other test 994 | spoon tool True hyper test 995 | bomber fuel False other test 996 | bulb turnip False other test 997 | axe artifact True hyper test 998 | chair furniture True hyper test 999 | bus contracting False other test 1000 | beet vegetable True hyper test 1001 | donkey vertebrate True hyper test 1002 | gorilla animal True hyper test 1003 | grasshopper childhood False other test 1004 | hatchet pincer False other test 1005 | dishwasher object True hyper test 1006 | cloak wear True hyper test 1007 | cloak blouse False other test 1008 | food onion False other test 1009 | rat beast True hyper test 1010 | hammer nausea False other test 1011 | battleship glider False other test 1012 | oil motorcycle False other test 1013 | frigate vehicle True hyper test 1014 | creature cod False other test 1015 | pineapple banana False other test 1016 | inn villa False other test 1017 | lion predator True hyper test 1018 | robe commodity True hyper test 1019 | pattern sweater False other test 1020 | construction castle False other test 1021 | hilt sword False other test 1022 | beast cat False other test 1023 | beaver coyote False other test 1024 | stone cherry False other test 1025 | snake animal True hyper test 1026 | sofa desk False other test 1027 | whale donkey False other test 1028 | radio device True hyper test 1029 | plum gunman False other test 1030 | lizard beast True hyper test 1031 | cathedral church True hyper test 1032 | furnishing chair False other test 1033 | scarf object True hyper test 1034 | sieve object True hyper test 1035 | fold sweater False other test 1036 | hotel lodging True hyper test 1037 | scarf accessory True hyper test 1038 | ambulance artifact True hyper test 1039 | hatchet cannon False other test 1040 | helicopter craft True hyper test 1041 | eagle predator True hyper test 1042 | birch tree True hyper test 1043 | sanctity phone False other test 1044 | pig vertebrate True hyper test 1045 | vehicle ambulance False other test 1046 | dove bird True hyper test 1047 | dress glove False other test 1048 | fridge object True hyper test 1049 | yacht vessel True hyper test 1050 | flower birch False other test 1051 | cow animal True hyper test 1052 | vest clothes True hyper test 1053 | shirt hieroglyphic False other test 1054 | bed artefact True hyper test 1055 | cedar plant True hyper test 1056 | willow tree True hyper test 1057 | potato stooge False other test 1058 | cabbage diy False other test 1059 | blouse robe False other test 1060 | gorilla foot False other test 1061 | cod carp False other test 1062 | mackerel dolphin False other test 1063 | animal swan False other test 1064 | mug whisky False other test 1065 | corn infertility False other test 1066 | cashmere scarf False other test 1067 | sword magazine False other test 1068 | beaver animal True hyper test 1069 | rat cow False other test 1070 | ear squirrel False other test 1071 | muzzle revolver False other test 1072 | flute device True hyper test 1073 | ceiling cottage False other test 1074 | hat artefact True hyper test 1075 | penguin dove False other test 1076 | shirt pattern False other test 1077 | londoner herring False other test 1078 | crust glove False other test 1079 | rifle hockey False other test 1080 | fork turnkey False other test 1081 | rat beaver False other test 1082 | giraffe beast True hyper test 1083 | cathedral site True hyper test 1084 | trumpet violin False other test 1085 | ferry artefact True hyper test 1086 | artefact grenade False other test 1087 | kitchenware fork False other test 1088 | ape gorilla False other test 1089 | inn restaurant False other test 1090 | trout phobia False other test 1091 | truck helicopter False other test 1092 | vest pc False other test 1093 | onion vegetable True hyper test 1094 | bicycle car False other test 1095 | catfish food True hyper test 1096 | axe spade False other test 1097 | cricket butterfly False other test 1098 | catfish fish True hyper test 1099 | vest commodity True hyper test 1100 | sweater artefact True hyper test 1101 | goose waterfowl True hyper test 1102 | beast squirrel False other test 1103 | truck m False other test 1104 | pistol hatchet False other test 1105 | scooter transport True hyper test 1106 | fridge commodity True hyper test 1107 | artifact knife False other test 1108 | coyote creature True hyper test 1109 | rabbit animal True hyper test 1110 | television commodity True hyper test 1111 | appliance stove False other test 1112 | piano device True hyper test 1113 | onion corn False other test 1114 | saxophone fare False other test 1115 | ear gorilla False other test 1116 | carrot turnip False other test 1117 | horse head False other test 1118 | rabbit vertebrate True hyper test 1119 | construction hospital False other test 1120 | lime amazement False other test 1121 | whale tooth False other test 1122 | taste bomber False other test 1123 | dove animal True hyper test 1124 | elm tree True hyper test 1125 | tree poplar False other test 1126 | cetacean whale False other test 1127 | cottage camera False other test 1128 | freezer good True hyper test 1129 | corn produce True hyper test 1130 | peach fruit True hyper test 1131 | crow penguin False other test 1132 | hornet creature True hyper test 1133 | dress wear True hyper test 1134 | battleship bus False other test 1135 | cod seafood True hyper test 1136 | artifact ferry False other test 1137 | trigger cannon False other test 1138 | lemon disguise False other test 1139 | cherry grape False other test 1140 | yacht craft True hyper test 1141 | pub deity False other test 1142 | licence rabbit False other test 1143 | wasp insect True hyper test 1144 | wood spoon False other test 1145 | pigeon creature True hyper test 1146 | leg grasshopper False other test 1147 | hospital structure True hyper test 1148 | violin object True hyper test 1149 | pheasant sparrow False other test 1150 | ferry outfielder False other test 1151 | seafood trout False other test 1152 | table couch False other test 1153 | fox canine True hyper test 1154 | phone artefact True hyper test 1155 | justice jacket False other test 1156 | jacket glove False other test 1157 | jacket dress False other test 1158 | artefact blouse False other test 1159 | bomber aircraft True hyper test 1160 | dolphin mammal True hyper test 1161 | chair artefact True hyper test 1162 | stereo device True hyper test 1163 | goldfish guitar False other test 1164 | hotel accommodation True hyper test 1165 | giraffe mammal True hyper test 1166 | mug container True hyper test 1167 | dagger pommel False other test 1168 | gorilla mammal True hyper test 1169 | frigate raft False other test 1170 | glider craft True hyper test 1171 | pigeon vertebrate True hyper test 1172 | furnishing wardrobe False other test 1173 | saw tool True hyper test 1174 | implementer lion False other test 1175 | shirt good True hyper test 1176 | giraffe leg False other test 1177 | trumpet instrument True hyper test 1178 | arm dagger False other test 1179 | eye frog False other test 1180 | scooter artefact True hyper test 1181 | cranberry produce True hyper test 1182 | clothing glove False other test 1183 | fighter artifact True hyper test 1184 | oven commodity True hyper test 1185 | jacket wear True hyper test 1186 | gia dress False other test 1187 | ambulance conveyance True hyper test 1188 | spade utensil True hyper test 1189 | object trumpet False other test 1190 | carrot parsley False other test 1191 | parsley food True hyper test 1192 | sweater clothing True hyper test 1193 | alligator creature True hyper test 1194 | helicopter artifact True hyper test 1195 | jacket hat False other test 1196 | radio commodity True hyper test 1197 | creature mackerel False other test 1198 | corn vegetable True hyper test 1199 | sieve frame False other test 1200 | castle season False other test 1201 | workload cloak False other test 1202 | robe good True hyper test 1203 | donkey creature True hyper test 1204 | telephone stove False other test 1205 | wood desk False other test 1206 | wrench artefact True hyper test 1207 | ant bug True hyper test 1208 | phone artillery False other test 1209 | helicopter top False other test 1210 | snake beast True hyper test 1211 | cucumber carrot False other test 1212 | dishwasher appliance True hyper test 1213 | coconut food True hyper test 1214 | hat clothing True hyper test 1215 | blouse garment True hyper test 1216 | missile warhead False other test 1217 | jacket good True hyper test 1218 | cathedral building True hyper test 1219 | cow petition False other test 1220 | artefact saw False other test 1221 | sofa artefact True hyper test 1222 | turtle reptile True hyper test 1223 | coat covering True hyper test 1224 | fox carnivore True hyper test 1225 | cannon arm True hyper test 1226 | cannon gun True hyper test 1227 | pistol arm True hyper test 1228 | gun shotgun False other test 1229 | goat food True hyper test 1230 | salmon head False other test 1231 | washer affluence False other test 1232 | tanker artefact True hyper test 1233 | vulture animal True hyper test 1234 | gorilla vertebrate True hyper test 1235 | jacket artifact True hyper test 1236 | pub institution True hyper test 1237 | creature ant False other test 1238 | tool hatchet False other test 1239 | silk vest False other test 1240 | ambulance transport True hyper test 1241 | jet frigate False other test 1242 | hotel pub False other test 1243 | dresser furniture True hyper test 1244 | wardrobe furniture True hyper test 1245 | vulture raptor True hyper test 1246 | canine coyote False other test 1247 | box mug False other test 1248 | object flute False other test 1249 | artifact vest False other test 1250 | turnip food True hyper test 1251 | phone cable False other test 1252 | salary turtle False other test 1253 | axe artefact True hyper test 1254 | radio good True hyper test 1255 | sweater good True hyper test 1256 | freezer artefact True hyper test 1257 | ruminant goat False other test 1258 | demonstration salmon False other test 1259 | jar artefact True hyper test 1260 | castle defence True hyper test 1261 | missile weapon True hyper test 1262 | wrench croatia False other test 1263 | dress clothing True hyper test 1264 | penguin animal True hyper test 1265 | flower willow False other test 1266 | vulture owl False other test 1267 | fridge artefact True hyper test 1268 | wood sofa False other test 1269 | passenger hawk False other test 1270 | hatchet artefact True hyper test 1271 | dress apparel True hyper test 1272 | sparrow creature True hyper test 1273 | creature gorilla False other test 1274 | beaver mammal True hyper test 1275 | jet window False other test 1276 | hawk goose False other test 1277 | helicopter van False other test 1278 | iron spinach False other test 1279 | battleship craft True hyper test 1280 | juice peach False other test 1281 | shovel spoon False other test 1282 | jar object True hyper test 1283 | hat apparel True hyper test 1284 | bomb artefact True hyper test 1285 | wrench utensil True hyper test 1286 | deer vertebrate True hyper test 1287 | spear object True hyper test 1288 | key saxophone False other test 1289 | bus artefact True hyper test 1290 | housing villa False other test 1291 | carp seafood True hyper test 1292 | radish produce True hyper test 1293 | box artefact True hyper test 1294 | dwelling castle False other test 1295 | pear fruit True hyper test 1296 | apple peach False other test 1297 | saxophone object True hyper test 1298 | trout catfish False other test 1299 | commodity hat False other test 1300 | glove garment True hyper test 1301 | jet aeroplane True hyper test 1302 | hat virus False other test 1303 | ferry vehicle True hyper test 1304 | lizard vertebrate True hyper test 1305 | piano investor False other test 1306 | table artefact True hyper test 1307 | artefact stereo False other test 1308 | stereo amplifier False other test 1309 | table furniture True hyper test 1310 | freezer television False other test 1311 | battleship artefact True hyper test 1312 | beet clinician False other test 1313 | plumage goose False other test 1314 | blouse clothes True hyper test 1315 | guy castle False other test 1316 | back train False other test 1317 | windshield jet False other test 1318 | castle lodging True hyper test 1319 | pine tree True hyper test 1320 | liar whale False other test 1321 | jet ambulance False other test 1322 | freezer appliance True hyper test 1323 | back glider False other test 1324 | bag incontinence False other test 1325 | dagger spear False other test 1326 | predator tiger False other test 1327 | beef bull False other test 1328 | pistol object True hyper test 1329 | spade wrench False other test 1330 | vest apparel True hyper test 1331 | fighter transport True hyper test 1332 | washer radio False other test 1333 | couch artefact True hyper test 1334 | engine ferry False other test 1335 | alligator courthouse False other test 1336 | alligator beast True hyper test 1337 | turtle amphibian True hyper test 1338 | trunk car False other test 1339 | coyote petal False other test 1340 | scarf trouser False other test 1341 | hat headdress True hyper test 1342 | frog turtle False other test 1343 | cathedral pub False other test 1344 | sweater covering True hyper test 1345 | pub structure True hyper test 1346 | gorilla elephant False other test 1347 | sweater garment True hyper test 1348 | bowl container True hyper test 1349 | cathedral pulse False other test 1350 | artifact washer False other test 1351 | dresser furnishing True hyper test 1352 | mackerel fish True hyper test 1353 | bomber plane True hyper test 1354 | knife weapon True hyper test 1355 | cloak commodity True hyper test 1356 | barrel shirt False other test 1357 | villa construction True hyper test 1358 | bomb device True hyper test 1359 | guitar artefact True hyper test 1360 | wood box False other test 1361 | frigate artifact True hyper test 1362 | gun object True hyper test 1363 | van tanker False other test 1364 | dresser artifact True hyper test 1365 | grapefruit citrus True hyper test 1366 | bodice dress False other test 1367 | coyote elephant False other test 1368 | supply freezer False other test 1369 | pistol artifact True hyper test 1370 | mackerel vertebrate True hyper test 1371 | sheep creature True hyper test 1372 | bird eagle False other test 1373 | truck vehicle True hyper test 1374 | dishwasher masonry False other test 1375 | tiger animal True hyper test 1376 | artifact blouse False other test 1377 | evergreen cypress False other test 1378 | back cloak False other test 1379 | gun artefact True hyper test 1380 | hotel building True hyper test 1381 | van transport True hyper test 1382 | fighter airplane True hyper test 1383 | robe apparel True hyper test 1384 | tiger creature True hyper test 1385 | beaver eye False other test 1386 | cow mammal True hyper test 1387 | jar container True hyper test 1388 | salmon food True hyper test 1389 | desk beating False other test 1390 | tiger jaw False other test 1391 | fighter plane True hyper test 1392 | cat mammal True hyper test 1393 | restaurant edifice True hyper test 1394 | box mortality False other test 1395 | mackerel seafood True hyper test 1396 | rifle artefact True hyper test 1397 | snake creature True hyper test 1398 | whale mammal True hyper test 1399 | hospital institution True hyper test 1400 | lock-up spoon False other test 1401 | stump pine False other test 1402 | knife fork False other test 1403 | pheasant vertebrate True hyper test 1404 | beak vulture False other test 1405 | beaver rodent True hyper test 1406 | elephant vertebrate True hyper test 1407 | sofa furniture True hyper test 1408 | screwdriver object True hyper test 1409 | bowl object True hyper test 1410 | castle restaurant False other test 1411 | cutlery knife False other test 1412 | rabbit herbivore True hyper test 1413 | spoon tableware True hyper test 1414 | coat hem False other test 1415 | commodity sweater False other test 1416 | table functioning False other test 1417 | tiger beast True hyper test 1418 | vest skin False other test 1419 | scarf commodity True hyper test 1420 | bottle artifact True hyper test 1421 | revolver object True hyper test 1422 | freight sheep False other test 1423 | dress object True hyper test 1424 | revolver gun True hyper test 1425 | dagger artefact True hyper test 1426 | cucumber vegetable True hyper test 1427 | cow creature True hyper test 1428 | spinach vegetable True hyper test 1429 | ear corn False other test 1430 | vertebrate tiger False other test 1431 | goat animal True hyper test 1432 | scarf garment True hyper test 1433 | shirt artefact True hyper test 1434 | inn pub False other test 1435 | rifle gun True hyper test 1436 | villa house True hyper test 1437 | hat wear True hyper test 1438 | animal hornet False other test 1439 | tooth rat False other test 1440 | cucumber food True hyper test 1441 | bed artifact True hyper test 1442 | hawk bird True hyper test 1443 | lemon produce True hyper test 1444 | cottage construction True hyper test 1445 | windshield bomber False other test 1446 | fighter aeroplane True hyper test 1447 | moth insect True hyper test 1448 | hawk raptor True hyper test 1449 | shell grenade False other test 1450 | lizard alligator False other test 1451 | rocket fighter False other test 1452 | guard dagger False other test 1453 | vulture creature True hyper test 1454 | conveyance glider False other test 1455 | covering jacket False other test 1456 | wardrobe object True hyper test 1457 | herb parsley False other test 1458 | piano artifact True hyper test 1459 | phone commodity True hyper test 1460 | frigate ship True hyper test 1461 | couch snap False other test 1462 | crow swan False other test 1463 | cranberry fruit True hyper test 1464 | kalashnikov rifle False other test 1465 | scissors hammer False other test 1466 | artefact frigate False other test 1467 | desk artefact True hyper test 1468 | bout cello False other test 1469 | oak tree True hyper test 1470 | trumpet artifact True hyper test 1471 | bear creature True hyper test 1472 | banana food True hyper test 1473 | timetable frigate False other test 1474 | castle door False other test 1475 | ticket stereo False other test 1476 | coyote predator True hyper test 1477 | coconut fruit True hyper test 1478 | fork artefact True hyper test 1479 | fighter lecturer False other test 1480 | yacht artifact True hyper test 1481 | falcon vertebrate True hyper test 1482 | coyote head False other test 1483 | cottage structure True hyper test 1484 | coyote beast True hyper test 1485 | vulture predator True hyper test 1486 | blouse object True hyper test 1487 | structure library False other test 1488 | squirrel vertebrate True hyper test 1489 | cauliflower food True hyper test 1490 | leaf radish False other test 1491 | pig skin False other test 1492 | apricot republican False other test 1493 | rifle revolver False other test 1494 | hawk woodpecker False other test 1495 | spade sieve False other test 1496 | cockroach pest True hyper test 1497 | hatchet utensil True hyper test 1498 | plumage penguin False other test 1499 | van artefact True hyper test 1500 | vest trousers False other test 1501 | ambulance artefact True hyper test 1502 | hatchet artifact True hyper test 1503 | brick pub False other test 1504 | turnip root True hyper test 1505 | artefact hammer False other test 1506 | wardrobe artifact True hyper test 1507 | bottom bag False other test 1508 | coat commodity True hyper test 1509 | object box False other test 1510 | sofa artifact True hyper test 1511 | owl creature True hyper test 1512 | sheep herbivore True hyper test 1513 | herring food True hyper test 1514 | desk furnishing True hyper test 1515 | crow vertebrate True hyper test 1516 | phone system True hyper test 1517 | artifact wrench False other test 1518 | helicopter orientation False other test 1519 | dress artifact True hyper test 1520 | produce coconut False other test 1521 | bit horse False other test 1522 | stereo system True hyper test 1523 | television artifact True hyper test 1524 | dagger bomb False other test 1525 | falcon predator True hyper test 1526 | horse circle False other test 1527 | trumpet cyst False other test 1528 | deer ear False other test 1529 | leaf parsley False other test 1530 | missile artefact True hyper test 1531 | bowl inside False other test 1532 | turtle alligator False other test 1533 | racism tanker False other test 1534 | dismay vulture False other test 1535 | fabric coat False other test 1536 | lime food True hyper test 1537 | fleece jacket False other test 1538 | table furnishing True hyper test 1539 | dagger object True hyper test 1540 | villa edifice True hyper test 1541 | crow bird True hyper test 1542 | pineapple plan False other test 1543 | cannon artifact True hyper test 1544 | sheep snout False other test 1545 | bomber warplane True hyper test 1546 | plumage falcon False other test 1547 | missile vehicle True hyper test 1548 | villa speed False other test 1549 | rat mammal True hyper test 1550 | barrel rifle False other test 1551 | glider artifact True hyper test 1552 | bed furnishing True hyper test 1553 | gorilla beast True hyper test 1554 | moth animal True hyper test 1555 | commodity glove False other test 1556 | cottage edifice True hyper test 1557 | giraffe herbivore True hyper test 1558 | bowl utensil True hyper test 1559 | castle defense True hyper test 1560 | cello artifact True hyper test 1561 | ceiling villa False other test 1562 | rat snout False other test 1563 | plume pigeon False other test 1564 | mammal donkey False other test 1565 | dolphin animal True hyper test 1566 | eagle creature True hyper test 1567 | cello object True hyper test 1568 | sweater apparel True hyper test 1569 | oak poplar False other test 1570 | jacket commodity True hyper test 1571 | phone washer False other test 1572 | deer rabbit False other test 1573 | pear produce True hyper test 1574 | blouse good True hyper test 1575 | hospital patient False other test 1576 | apparel shirt False other test 1577 | couch bed False other test 1578 | body moth False other test 1579 | grape berry True hyper test 1580 | glove object True hyper test 1581 | horse herbivore True hyper test 1582 | jam jar False other test 1583 | tanker artifact True hyper test 1584 | eye hawk False other test 1585 | squirrel goat False other test 1586 | washer artefact True hyper test 1587 | washer commodity True hyper test 1588 | jaw bear False other test 1589 | desk object True hyper test 1590 | poplar angiosperm True hyper test 1591 | vest covering True hyper test 1592 | cat paw False other test 1593 | violin club False other test 1594 | cloak artifact True hyper test 1595 | kalashnikov revolver False other test 1596 | squirrel creature True hyper test 1597 | rifle arm True hyper test 1598 | turnip produce True hyper test 1599 | camcorder yacht False other test 1600 | vest clothing True hyper test 1601 | cottage castle False other test 1602 | castle site True hyper test 1603 | mammal elephant False other test 1604 | woodpecker animal True hyper test 1605 | piano artefact True hyper test 1606 | restaurant structure True hyper test 1607 | pistol artefact True hyper test 1608 | guitar device True hyper test 1609 | hat covering True hyper test 1610 | artefact revolver False other test 1611 | pheasant creature True hyper test 1612 | villa castle False other test 1613 | hat italian False other test 1614 | peach lime False other test 1615 | cloak stigma False other test 1616 | whale seafood True hyper test 1617 | bull cluster False other test 1618 | television shovel False other test 1619 | oven television False other test 1620 | exterior mug False other test 1621 | grasshopper bug True hyper test 1622 | bass cello False other test 1623 | castle edifice True hyper test 1624 | foot dove False other test 1625 | lion mammal True hyper test 1626 | fridge food False other test 1627 | sieve filter True hyper test 1628 | jet aircraft True hyper test 1629 | scarf poem False other test 1630 | back coat False other test 1631 | cauliflower produce True hyper test 1632 | scooter artifact True hyper test 1633 | shirt commodity True hyper test 1634 | rabbit rat False other test 1635 | broccoli food True hyper test 1636 | apple produce True hyper test 1637 | animal cockroach False other test 1638 | plum produce True hyper test 1639 | tiger mammal True hyper test 1640 | turtle creature True hyper test 1641 | pub edifice True hyper test 1642 | gill goldfish False other test 1643 | flute o False other test 1644 | sheep mammal True hyper test 1645 | pig mammal True hyper test 1646 | fork artifact True hyper test 1647 | eye swan False other test 1648 | oven artifact True hyper test 1649 | vehicle glider False other test 1650 | tool shovel False other test 1651 | woodpecker wheel False other test 1652 | salmon vertebrate True hyper test 1653 | car jet False other test 1654 | construction hotel False other test 1655 | scarf artefact True hyper test 1656 | motorcycle vehicle True hyper test 1657 | clothes dresser False other test 1658 | object missile False other test 1659 | vehicle fighter False other test 1660 | spinach cauliflower False other test 1661 | metal bus False other test 1662 | cat inception False other test 1663 | bottle artefact True hyper test 1664 | coat clothing True hyper test 1665 | revolver bazooka False other test 1666 | spoon artifact True hyper test 1667 | rifle artifact True hyper test 1668 | dagger artifact True hyper test 1669 | rodent rat False other test 1670 | --------------------------------------------------------------------------------