├── LICENSE ├── README.md ├── data ├── ConEL_Concept_Named_Entity │ ├── ConEL_CNE.json │ └── runs │ │ ├── ConEL_CNE-REL_history.json │ │ ├── ConEL_CNE-REL_turn.json │ │ ├── ConEL_CNE-Tagme_history.json │ │ ├── ConEL_CNE-Tagme_turn.json │ │ ├── ConEL_CNE-WAT_history.json │ │ └── ConEL_CNE-WAT_turn.json ├── ConEL_Personal_Entity │ ├── ConEL_PE.json │ └── runs │ │ ├── ConEL_PE-REL_history.json │ │ ├── ConEL_PE-REL_turn.json │ │ ├── ConEL_PE-Tagme_history.json │ │ ├── ConEL_PE-Tagme_turn.json │ │ ├── ConEL_PE-WAT_history.json │ │ └── ConEL_PE-WAT_turn.json └── README.md └── mturk_interfaces ├── 01_stage1_annotations ├── html │ └── stage1.html └── screenshots │ ├── stage1_example1.png │ ├── stage1_example2.png │ └── stage1_overview.png ├── 02_stage2_annotations ├── html │ └── stage2.html └── screenshots │ ├── stage2_example1.png │ ├── stage2_example2.png │ └── stage2_overview.png └── 03_personal_entity_annotations ├── html └── personal_entity_annotations.html └── screenshots ├── personal_entity_annotations_examples.png └── personal_entity_annotations_overview.png /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Informagi 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ConEL: Conversational Entity Linking Datasets 2 | ============ 3 | 4 | This repository provides the resources related to entity linking annotations in conversational settings.\ 5 | These resources are created on the existing datasets: 6 | - [MultiWOZ](https://github.com/budzianowski/multiwoz) (MWOZ) 7 | - [Question Answering in Context](https://quac.ai/) (QuAC), 8 | - [Wizard-Of-Wikipedia](https://parl.ai/projects/wizard_of_wikipedia/) (WoW) 9 | - [TREC-CAsT 2020](https://github.com/daltonj/treccastweb) 10 | 11 | These resources are developed within the following paper: 12 | - *Hideaki Joko, Faegheh Hasibi, Krisztian Balog, and Arjen P. de Vries. “[Conversational Entity Linking: Problem Definition and Datasets](https://arxiv.org/abs/2105.04903)”.* 13 | 14 | 15 | The repository is structured as follows: 16 | - `./data`: MTurk entity annotations 17 | - `./mturk_interfaces`: MTurk interface used to collect the entity annotations 18 | 19 | # Data 20 | 21 | MTurk entity annotation data is stored in `./data`. 22 | - `./data/ConEL_Concept_Named_Entity/`: Stratified samples 23 | - `./data/ConEL_Concept_Named_Entity/ConEL_CNE.json`: Entity annotations from 25 dialogues from each dataset (i.e., MWOZ, QuAC, WoW, and TREC-CAST 2020). 24 | - `./data/ConEL_Personal_Entity/`: WoW with personal entities 25 | - `./data/ConEL_Personal_Entity/ConEL_PE.json`: 25 WoW dialogues which contains personal entities in each dialogue. 26 | - `run` folders contain EL tools' results 27 | 28 | ## Statistics 29 | 30 | | | Stratified samples
(`ConEL_CNE.json`) | WoW with personal entities
(`ConEL_PE.json`) | 31 | |-------------------|----------------------------------------|---------------------------------------------------------| 32 | | # dialogues | 100 | 25 | 33 | | # user utterances | 708 | 113 | 34 | 35 | 36 | ## Data Format 37 | This section explains ground truth files data format (`ConEL_CNE.json` and `ConEL_PE.json`)\ 38 | Each element in a list has a dict structure as follows: 39 | 40 | ```py 41 | { 42 | "dialogue_id": "10060", 43 | "dataset_name": "wow", # or "quac", "mwoz", "cast20raw", "cast20manu" 44 | "turns": [ 45 | { 46 | "speaker": "USER", # or "SYSTEM" 47 | "utterance": "Blue is my favorite color, by far. What's yours?", 48 | "turn_number": 0, 49 | "el_annotations": [ # Ground truth annotations 50 | { 51 | "mention": "Blue", 52 | "entity": "Blue", 53 | "entity_type": "concept", # or "named_entity" 54 | } 55 | ], 56 | "personal_entity_annotations": [ # Personal entity annotations 57 | { 58 | "personal_entity_mention": "my favorite color", 59 | "explicit_entity_mention": "Blue", 60 | "turn_number_of_explicit_entity_mention": 0, 61 | "entity": "Blue" 62 | } 63 | ] 64 | }, 65 | ] 66 | } 67 | ``` 68 | 69 | - `dialogue_id`: dialogue id provided by each original dataset (i.e., MWOZ, QuAC, WoW, and TREC-CAsT 2020). 70 | - `dataset_name`: The name of the dataset in which the conversations were used (cast20raw and cast20manu represent ) 71 | - `turns`: each element contains an user or system turns 72 | - `speaker`: USER or SYSTEM 73 | - `utterance`: utterance acquired from the dataset. (Note that for TREC-CAST 2020 system turns, only manual_canonical_result_id are shown) 74 | - `el_annotations`: annotations with MTurk workers 75 | - `personal_entity_annotations`: Personal entity annotations. Note that only `ConEL_PE.json` has this annotations. 76 | 77 | # MTurk Interfaces 78 | 79 | MTurk interface used to collect the entity annotations.\ 80 | Interfaces are Stored in `./mturk_interfaces` directory. 81 | 82 | # Conversational Dataset List 83 | 84 | [Conversational Dataset List](https://docs.google.com/spreadsheets/d/1N5_5gBKlGR-OrigRNct4jQ6iEqSycyqcoN61JpsHFDQ/edit?usp=sharing): A comprehensive list of around 130 conversational datasets released by different research communities 85 | 86 | 88 | 89 | # Cite 90 | 91 | ```bibtex 92 | @inproceedings{Joko:2021:CEL, 93 | author = {Joko, Hideaki and Hasibi, Faegheh and Balog, Krisztian and de Vries, Arjen P.}, 94 | title = {Conversational Entity Linking: Problem Definition and Datasets}, 95 | booktitle = {Proceedings of the 44rd International ACM SIGIR Conference on Research and Development in Information Retrieval}, 96 | series = {SIGIR '21}, 97 | year = {2021}, 98 | publisher = {ACM} 99 | } 100 | ``` 101 | 102 | # Contact 103 | 104 | If you have any questions, please contact Hideaki Joko at hideaki.joko@ru.nl 105 | -------------------------------------------------------------------------------- /data/ConEL_Personal_Entity/runs/ConEL_PE-REL_history.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "dialogue_id": "10060", 4 | "dataset_name": "wow", 5 | "turns": { 6 | "0": { 7 | "turn_number": 0, 8 | "el_tool_annotations": [] 9 | }, 10 | "2": { 11 | "turn_number": 2, 12 | "el_tool_annotations": [ 13 | { 14 | "mention": "ROYGBIV", 15 | "entity": "ROYGBIV" 16 | } 17 | ] 18 | }, 19 | "4": { 20 | "turn_number": 4, 21 | "el_tool_annotations": [] 22 | }, 23 | "6": { 24 | "turn_number": 6, 25 | "el_tool_annotations": [] 26 | } 27 | } 28 | }, 29 | { 30 | "dialogue_id": "1191", 31 | "dataset_name": "wow", 32 | "turns": { 33 | "1": { 34 | "turn_number": 1, 35 | "el_tool_annotations": [ 36 | { 37 | "mention": "Hard Rock", 38 | "entity": "Hard_rock" 39 | }, 40 | { 41 | "mention": "Punk Music", 42 | "entity": "Punk_rock" 43 | } 44 | ] 45 | }, 46 | "3": { 47 | "turn_number": 3, 48 | "el_tool_annotations": [ 49 | { 50 | "mention": "Iggy Pop", 51 | "entity": "Iggy_Pop" 52 | } 53 | ] 54 | }, 55 | "5": { 56 | "turn_number": 5, 57 | "el_tool_annotations": [ 58 | { 59 | "mention": "White Snake", 60 | "entity": "Whitesnake" 61 | } 62 | ] 63 | }, 64 | "7": { 65 | "turn_number": 7, 66 | "el_tool_annotations": [ 67 | { 68 | "mention": "Glen Danzig", 69 | "entity": "Glenn_Danzig" 70 | }, 71 | { 72 | "mention": "Misfit", 73 | "entity": "Misfits_(band)" 74 | }, 75 | { 76 | "mention": "Elvis", 77 | "entity": "Elvis_Presley" 78 | }, 79 | { 80 | "mention": "Johnny Cash", 81 | "entity": "Johnny_Cash" 82 | } 83 | ] 84 | } 85 | } 86 | }, 87 | { 88 | "dialogue_id": "17985", 89 | "dataset_name": "wow", 90 | "turns": { 91 | "0": { 92 | "turn_number": 0, 93 | "el_tool_annotations": [] 94 | }, 95 | "2": { 96 | "turn_number": 2, 97 | "el_tool_annotations": [] 98 | }, 99 | "4": { 100 | "turn_number": 4, 101 | "el_tool_annotations": [] 102 | }, 103 | "6": { 104 | "turn_number": 6, 105 | "el_tool_annotations": [] 106 | }, 107 | "8": { 108 | "turn_number": 8, 109 | "el_tool_annotations": [] 110 | } 111 | } 112 | }, 113 | { 114 | "dialogue_id": "16840", 115 | "dataset_name": "wow", 116 | "turns": { 117 | "0": { 118 | "turn_number": 0, 119 | "el_tool_annotations": [] 120 | }, 121 | "2": { 122 | "turn_number": 2, 123 | "el_tool_annotations": [] 124 | }, 125 | "4": { 126 | "turn_number": 4, 127 | "el_tool_annotations": [] 128 | }, 129 | "6": { 130 | "turn_number": 6, 131 | "el_tool_annotations": [] 132 | } 133 | } 134 | }, 135 | { 136 | "dialogue_id": "15141", 137 | "dataset_name": "wow", 138 | "turns": { 139 | "0": { 140 | "turn_number": 0, 141 | "el_tool_annotations": [] 142 | }, 143 | "2": { 144 | "turn_number": 2, 145 | "el_tool_annotations": [ 146 | { 147 | "mention": "New York", 148 | "entity": "New_York_City" 149 | } 150 | ] 151 | }, 152 | "4": { 153 | "turn_number": 4, 154 | "el_tool_annotations": [] 155 | }, 156 | "6": { 157 | "turn_number": 6, 158 | "el_tool_annotations": [] 159 | }, 160 | "8": { 161 | "turn_number": 8, 162 | "el_tool_annotations": [ 163 | { 164 | "mention": "Popeye", 165 | "entity": "Popeye" 166 | } 167 | ] 168 | } 169 | } 170 | }, 171 | { 172 | "dialogue_id": "5336", 173 | "dataset_name": "wow", 174 | "turns": { 175 | "0": { 176 | "turn_number": 0, 177 | "el_tool_annotations": [] 178 | }, 179 | "2": { 180 | "turn_number": 2, 181 | "el_tool_annotations": [] 182 | }, 183 | "4": { 184 | "turn_number": 4, 185 | "el_tool_annotations": [] 186 | }, 187 | "6": { 188 | "turn_number": 6, 189 | "el_tool_annotations": [ 190 | { 191 | "mention": "Reese", 192 | "entity": "Reese's_Peanut_Butter_Cups" 193 | } 194 | ] 195 | } 196 | } 197 | }, 198 | { 199 | "dialogue_id": "1383", 200 | "dataset_name": "wow", 201 | "turns": { 202 | "0": { 203 | "turn_number": 0, 204 | "el_tool_annotations": [ 205 | { 206 | "mention": "Mediterranean", 207 | "entity": "Mediterranean_cuisine" 208 | } 209 | ] 210 | }, 211 | "2": { 212 | "turn_number": 2, 213 | "el_tool_annotations": [ 214 | { 215 | "mention": "Mediterranean", 216 | "entity": "Mediterranean_cuisine" 217 | } 218 | ] 219 | }, 220 | "4": { 221 | "turn_number": 4, 222 | "el_tool_annotations": [] 223 | }, 224 | "6": { 225 | "turn_number": 6, 226 | "el_tool_annotations": [] 227 | } 228 | } 229 | }, 230 | { 231 | "dialogue_id": "15621", 232 | "dataset_name": "wow", 233 | "turns": { 234 | "0": { 235 | "turn_number": 0, 236 | "el_tool_annotations": [] 237 | }, 238 | "2": { 239 | "turn_number": 2, 240 | "el_tool_annotations": [] 241 | }, 242 | "4": { 243 | "turn_number": 4, 244 | "el_tool_annotations": [] 245 | }, 246 | "6": { 247 | "turn_number": 6, 248 | "el_tool_annotations": [] 249 | } 250 | } 251 | }, 252 | { 253 | "dialogue_id": "1161", 254 | "dataset_name": "wow", 255 | "turns": { 256 | "0": { 257 | "turn_number": 0, 258 | "el_tool_annotations": [] 259 | }, 260 | "2": { 261 | "turn_number": 2, 262 | "el_tool_annotations": [ 263 | { 264 | "mention": "Carvel", 265 | "entity": "Carvel_(franchise)" 266 | } 267 | ] 268 | }, 269 | "4": { 270 | "turn_number": 4, 271 | "el_tool_annotations": [] 272 | }, 273 | "6": { 274 | "turn_number": 6, 275 | "el_tool_annotations": [] 276 | } 277 | } 278 | }, 279 | { 280 | "dialogue_id": "7957", 281 | "dataset_name": "wow", 282 | "turns": { 283 | "1": { 284 | "turn_number": 1, 285 | "el_tool_annotations": [] 286 | }, 287 | "3": { 288 | "turn_number": 3, 289 | "el_tool_annotations": [ 290 | { 291 | "mention": "Oreo", 292 | "entity": "Oreo" 293 | } 294 | ] 295 | }, 296 | "5": { 297 | "turn_number": 5, 298 | "el_tool_annotations": [] 299 | }, 300 | "7": { 301 | "turn_number": 7, 302 | "el_tool_annotations": [] 303 | }, 304 | "9": { 305 | "turn_number": 9, 306 | "el_tool_annotations": [] 307 | } 308 | } 309 | }, 310 | { 311 | "dialogue_id": "2573", 312 | "dataset_name": "wow", 313 | "turns": { 314 | "1": { 315 | "turn_number": 1, 316 | "el_tool_annotations": [ 317 | { 318 | "mention": "Vanilla", 319 | "entity": "Vanilla" 320 | }, 321 | { 322 | "mention": "Strawberry", 323 | "entity": "Strawberry" 324 | } 325 | ] 326 | }, 327 | "3": { 328 | "turn_number": 3, 329 | "el_tool_annotations": [] 330 | }, 331 | "5": { 332 | "turn_number": 5, 333 | "el_tool_annotations": [] 334 | }, 335 | "7": { 336 | "turn_number": 7, 337 | "el_tool_annotations": [ 338 | { 339 | "mention": "A&W", 340 | "entity": "A&W_Restaurants" 341 | } 342 | ] 343 | }, 344 | "9": { 345 | "turn_number": 9, 346 | "el_tool_annotations": [] 347 | } 348 | } 349 | }, 350 | { 351 | "dialogue_id": "13604", 352 | "dataset_name": "wow", 353 | "turns": { 354 | "0": { 355 | "turn_number": 0, 356 | "el_tool_annotations": [ 357 | { 358 | "mention": "Pizza", 359 | "entity": "Pizza" 360 | } 361 | ] 362 | }, 363 | "2": { 364 | "turn_number": 2, 365 | "el_tool_annotations": [] 366 | }, 367 | "4": { 368 | "turn_number": 4, 369 | "el_tool_annotations": [] 370 | }, 371 | "6": { 372 | "turn_number": 6, 373 | "el_tool_annotations": [] 374 | }, 375 | "8": { 376 | "turn_number": 8, 377 | "el_tool_annotations": [ 378 | { 379 | "mention": "Italy", 380 | "entity": "Italy" 381 | } 382 | ] 383 | } 384 | } 385 | }, 386 | { 387 | "dialogue_id": "873", 388 | "dataset_name": "wow", 389 | "turns": { 390 | "0": { 391 | "turn_number": 0, 392 | "el_tool_annotations": [] 393 | }, 394 | "2": { 395 | "turn_number": 2, 396 | "el_tool_annotations": [] 397 | }, 398 | "4": { 399 | "turn_number": 4, 400 | "el_tool_annotations": [] 401 | }, 402 | "6": { 403 | "turn_number": 6, 404 | "el_tool_annotations": [] 405 | } 406 | } 407 | }, 408 | { 409 | "dialogue_id": "22270", 410 | "dataset_name": "wow", 411 | "turns": { 412 | "0": { 413 | "turn_number": 0, 414 | "el_tool_annotations": [] 415 | }, 416 | "2": { 417 | "turn_number": 2, 418 | "el_tool_annotations": [] 419 | }, 420 | "4": { 421 | "turn_number": 4, 422 | "el_tool_annotations": [] 423 | }, 424 | "6": { 425 | "turn_number": 6, 426 | "el_tool_annotations": [] 427 | }, 428 | "8": { 429 | "turn_number": 8, 430 | "el_tool_annotations": [] 431 | } 432 | } 433 | }, 434 | { 435 | "dialogue_id": "17206", 436 | "dataset_name": "wow", 437 | "turns": { 438 | "0": { 439 | "turn_number": 0, 440 | "el_tool_annotations": [ 441 | { 442 | "mention": "Mavericks", 443 | "entity": "Mavericks,_California" 444 | } 445 | ] 446 | }, 447 | "2": { 448 | "turn_number": 2, 449 | "el_tool_annotations": [] 450 | }, 451 | "4": { 452 | "turn_number": 4, 453 | "el_tool_annotations": [] 454 | }, 455 | "6": { 456 | "turn_number": 6, 457 | "el_tool_annotations": [ 458 | { 459 | "mention": "NBA", 460 | "entity": "National_Basketball_Association" 461 | } 462 | ] 463 | }, 464 | "8": { 465 | "turn_number": 8, 466 | "el_tool_annotations": [] 467 | } 468 | } 469 | }, 470 | { 471 | "dialogue_id": "16339", 472 | "dataset_name": "wow", 473 | "turns": { 474 | "0": { 475 | "turn_number": 0, 476 | "el_tool_annotations": [ 477 | { 478 | "mention": "American", 479 | "entity": "American_football" 480 | } 481 | ] 482 | }, 483 | "2": { 484 | "turn_number": 2, 485 | "el_tool_annotations": [ 486 | { 487 | "mention": "New England Patriots", 488 | "entity": "New_England_Patriots" 489 | }, 490 | { 491 | "mention": "New England", 492 | "entity": "New_England" 493 | } 494 | ] 495 | }, 496 | "4": { 497 | "turn_number": 4, 498 | "el_tool_annotations": [ 499 | { 500 | "mention": "Dolphins", 501 | "entity": "Miami_Dolphins" 502 | }, 503 | { 504 | "mention": "Marino", 505 | "entity": "Dan_Marino" 506 | } 507 | ] 508 | }, 509 | "6": { 510 | "turn_number": 6, 511 | "el_tool_annotations": [ 512 | { 513 | "mention": "NFL", 514 | "entity": "National_Football_League" 515 | }, 516 | { 517 | "mention": "Peyton Manning", 518 | "entity": "Peyton_Manning" 519 | } 520 | ] 521 | }, 522 | "8": { 523 | "turn_number": 8, 524 | "el_tool_annotations": [ 525 | { 526 | "mention": "NFL", 527 | "entity": "National_Football_League" 528 | }, 529 | { 530 | "mention": "Peyton Manning", 531 | "entity": "Peyton_Manning" 532 | } 533 | ] 534 | } 535 | } 536 | }, 537 | { 538 | "dialogue_id": "17006", 539 | "dataset_name": "wow", 540 | "turns": { 541 | "0": { 542 | "turn_number": 0, 543 | "el_tool_annotations": [] 544 | }, 545 | "2": { 546 | "turn_number": 2, 547 | "el_tool_annotations": [] 548 | }, 549 | "4": { 550 | "turn_number": 4, 551 | "el_tool_annotations": [ 552 | { 553 | "mention": "Violet", 554 | "entity": "Violet_(color)" 555 | } 556 | ] 557 | }, 558 | "6": { 559 | "turn_number": 6, 560 | "el_tool_annotations": [ 561 | { 562 | "mention": "Japan", 563 | "entity": "Japan" 564 | } 565 | ] 566 | } 567 | } 568 | }, 569 | { 570 | "dialogue_id": "1168", 571 | "dataset_name": "wow", 572 | "turns": { 573 | "0": { 574 | "turn_number": 0, 575 | "el_tool_annotations": [] 576 | }, 577 | "2": { 578 | "turn_number": 2, 579 | "el_tool_annotations": [] 580 | }, 581 | "4": { 582 | "turn_number": 4, 583 | "el_tool_annotations": [ 584 | { 585 | "mention": "italian", 586 | "entity": "Italy" 587 | } 588 | ] 589 | }, 590 | "6": { 591 | "turn_number": 6, 592 | "el_tool_annotations": [] 593 | }, 594 | "8": { 595 | "turn_number": 8, 596 | "el_tool_annotations": [] 597 | } 598 | } 599 | }, 600 | { 601 | "dialogue_id": "17067", 602 | "dataset_name": "wow", 603 | "turns": { 604 | "0": { 605 | "turn_number": 0, 606 | "el_tool_annotations": [ 607 | { 608 | "mention": "Channing Tatum", 609 | "entity": "Channing_Tatum" 610 | } 611 | ] 612 | }, 613 | "2": { 614 | "turn_number": 2, 615 | "el_tool_annotations": [] 616 | }, 617 | "4": { 618 | "turn_number": 4, 619 | "el_tool_annotations": [ 620 | { 621 | "mention": "Jump Street", 622 | "entity": "21_Jump_Street" 623 | }, 624 | { 625 | "mention": "Magic Mike", 626 | "entity": "Magic_Mike" 627 | } 628 | ] 629 | }, 630 | "6": { 631 | "turn_number": 6, 632 | "el_tool_annotations": [] 633 | } 634 | } 635 | }, 636 | { 637 | "dialogue_id": "4371", 638 | "dataset_name": "wow", 639 | "turns": { 640 | "0": { 641 | "turn_number": 0, 642 | "el_tool_annotations": [] 643 | }, 644 | "2": { 645 | "turn_number": 2, 646 | "el_tool_annotations": [] 647 | }, 648 | "4": { 649 | "turn_number": 4, 650 | "el_tool_annotations": [] 651 | }, 652 | "6": { 653 | "turn_number": 6, 654 | "el_tool_annotations": [] 655 | }, 656 | "8": { 657 | "turn_number": 8, 658 | "el_tool_annotations": [] 659 | } 660 | } 661 | }, 662 | { 663 | "dialogue_id": "12407", 664 | "dataset_name": "wow", 665 | "turns": { 666 | "1": { 667 | "turn_number": 1, 668 | "el_tool_annotations": [] 669 | }, 670 | "3": { 671 | "turn_number": 3, 672 | "el_tool_annotations": [] 673 | }, 674 | "5": { 675 | "turn_number": 5, 676 | "el_tool_annotations": [ 677 | { 678 | "mention": "Salmon", 679 | "entity": "Salmon" 680 | } 681 | ] 682 | }, 683 | "7": { 684 | "turn_number": 7, 685 | "el_tool_annotations": [] 686 | } 687 | } 688 | }, 689 | { 690 | "dialogue_id": "8214", 691 | "dataset_name": "wow", 692 | "turns": { 693 | "0": { 694 | "turn_number": 0, 695 | "el_tool_annotations": [ 696 | { 697 | "mention": "Pearl Jam", 698 | "entity": "Pearl_Jam" 699 | } 700 | ] 701 | }, 702 | "2": { 703 | "turn_number": 2, 704 | "el_tool_annotations": [ 705 | { 706 | "mention": "Kurt Cobain", 707 | "entity": "Kurt_Cobain" 708 | }, 709 | { 710 | "mention": "Dave Grohl", 711 | "entity": "Dave_Grohl" 712 | } 713 | ] 714 | }, 715 | "4": { 716 | "turn_number": 4, 717 | "el_tool_annotations": [ 718 | { 719 | "mention": "Kurt", 720 | "entity": "Kurt_Cobain" 721 | } 722 | ] 723 | }, 724 | "6": { 725 | "turn_number": 6, 726 | "el_tool_annotations": [ 727 | { 728 | "mention": "Matt Cameron", 729 | "entity": "Matt_Cameron" 730 | }, 731 | { 732 | "mention": "Soundgarden", 733 | "entity": "Soundgarden" 734 | } 735 | ] 736 | }, 737 | "8": { 738 | "turn_number": 8, 739 | "el_tool_annotations": [ 740 | { 741 | "mention": "Pearl Jam", 742 | "entity": "Pearl_Jam" 743 | } 744 | ] 745 | } 746 | } 747 | }, 748 | { 749 | "dialogue_id": "7051", 750 | "dataset_name": "wow", 751 | "turns": { 752 | "0": { 753 | "turn_number": 0, 754 | "el_tool_annotations": [] 755 | }, 756 | "2": { 757 | "turn_number": 2, 758 | "el_tool_annotations": [] 759 | }, 760 | "4": { 761 | "turn_number": 4, 762 | "el_tool_annotations": [] 763 | }, 764 | "6": { 765 | "turn_number": 6, 766 | "el_tool_annotations": [] 767 | }, 768 | "8": { 769 | "turn_number": 8, 770 | "el_tool_annotations": [] 771 | } 772 | } 773 | }, 774 | { 775 | "dialogue_id": "1266", 776 | "dataset_name": "wow", 777 | "turns": { 778 | "0": { 779 | "turn_number": 0, 780 | "el_tool_annotations": [] 781 | }, 782 | "2": { 783 | "turn_number": 2, 784 | "el_tool_annotations": [] 785 | }, 786 | "4": { 787 | "turn_number": 4, 788 | "el_tool_annotations": [] 789 | }, 790 | "6": { 791 | "turn_number": 6, 792 | "el_tool_annotations": [] 793 | }, 794 | "8": { 795 | "turn_number": 8, 796 | "el_tool_annotations": [] 797 | } 798 | } 799 | }, 800 | { 801 | "dialogue_id": "2040", 802 | "dataset_name": "wow", 803 | "turns": { 804 | "0": { 805 | "turn_number": 0, 806 | "el_tool_annotations": [ 807 | { 808 | "mention": "Beagle", 809 | "entity": "Beagle" 810 | }, 811 | { 812 | "mention": "Droopy", 813 | "entity": "Droopy" 814 | } 815 | ] 816 | }, 817 | "2": { 818 | "turn_number": 2, 819 | "el_tool_annotations": [] 820 | }, 821 | "4": { 822 | "turn_number": 4, 823 | "el_tool_annotations": [ 824 | { 825 | "mention": "Beagles", 826 | "entity": "Beagle" 827 | } 828 | ] 829 | }, 830 | "6": { 831 | "turn_number": 6, 832 | "el_tool_annotations": [ 833 | { 834 | "mention": "Beagle", 835 | "entity": "Beagle" 836 | } 837 | ] 838 | } 839 | } 840 | } 841 | ] -------------------------------------------------------------------------------- /data/ConEL_Personal_Entity/runs/ConEL_PE-REL_turn.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "dialogue_id": "10060", 4 | "dataset_name": "wow", 5 | "turns": { 6 | "0": { 7 | "turn_number": 0, 8 | "el_tool_annotations": [] 9 | }, 10 | "2": { 11 | "turn_number": 2, 12 | "el_tool_annotations": [ 13 | { 14 | "mention": "ROYGBIV", 15 | "entity": "ROYGBIV" 16 | } 17 | ] 18 | }, 19 | "4": { 20 | "turn_number": 4, 21 | "el_tool_annotations": [] 22 | }, 23 | "6": { 24 | "turn_number": 6, 25 | "el_tool_annotations": [] 26 | } 27 | } 28 | }, 29 | { 30 | "dialogue_id": "1191", 31 | "dataset_name": "wow", 32 | "turns": { 33 | "1": { 34 | "turn_number": 1, 35 | "el_tool_annotations": [ 36 | { 37 | "mention": "Hard Rock", 38 | "entity": "Hard_rock" 39 | }, 40 | { 41 | "mention": "Punk Music", 42 | "entity": "Punk_rock" 43 | } 44 | ] 45 | }, 46 | "3": { 47 | "turn_number": 3, 48 | "el_tool_annotations": [ 49 | { 50 | "mention": "Iggy Pop", 51 | "entity": "Iggy_Pop" 52 | } 53 | ] 54 | }, 55 | "5": { 56 | "turn_number": 5, 57 | "el_tool_annotations": [ 58 | { 59 | "mention": "White Snake", 60 | "entity": "Whitesnake" 61 | } 62 | ] 63 | }, 64 | "7": { 65 | "turn_number": 7, 66 | "el_tool_annotations": [ 67 | { 68 | "mention": "Glen Danzig", 69 | "entity": "Glenn_Danzig" 70 | }, 71 | { 72 | "mention": "Misfit", 73 | "entity": "The_Oblongs" 74 | }, 75 | { 76 | "mention": "Elvis", 77 | "entity": "Elvis_Presley" 78 | }, 79 | { 80 | "mention": "Johnny Cash", 81 | "entity": "Johnny_Cash" 82 | } 83 | ] 84 | } 85 | } 86 | }, 87 | { 88 | "dialogue_id": "17985", 89 | "dataset_name": "wow", 90 | "turns": { 91 | "0": { 92 | "turn_number": 0, 93 | "el_tool_annotations": [] 94 | }, 95 | "2": { 96 | "turn_number": 2, 97 | "el_tool_annotations": [] 98 | }, 99 | "4": { 100 | "turn_number": 4, 101 | "el_tool_annotations": [] 102 | }, 103 | "6": { 104 | "turn_number": 6, 105 | "el_tool_annotations": [] 106 | }, 107 | "8": { 108 | "turn_number": 8, 109 | "el_tool_annotations": [] 110 | } 111 | } 112 | }, 113 | { 114 | "dialogue_id": "16840", 115 | "dataset_name": "wow", 116 | "turns": { 117 | "0": { 118 | "turn_number": 0, 119 | "el_tool_annotations": [] 120 | }, 121 | "2": { 122 | "turn_number": 2, 123 | "el_tool_annotations": [] 124 | }, 125 | "4": { 126 | "turn_number": 4, 127 | "el_tool_annotations": [] 128 | }, 129 | "6": { 130 | "turn_number": 6, 131 | "el_tool_annotations": [] 132 | } 133 | } 134 | }, 135 | { 136 | "dialogue_id": "15141", 137 | "dataset_name": "wow", 138 | "turns": { 139 | "0": { 140 | "turn_number": 0, 141 | "el_tool_annotations": [] 142 | }, 143 | "2": { 144 | "turn_number": 2, 145 | "el_tool_annotations": [ 146 | { 147 | "mention": "New York", 148 | "entity": "New_York_(state)" 149 | } 150 | ] 151 | }, 152 | "4": { 153 | "turn_number": 4, 154 | "el_tool_annotations": [] 155 | }, 156 | "6": { 157 | "turn_number": 6, 158 | "el_tool_annotations": [] 159 | }, 160 | "8": { 161 | "turn_number": 8, 162 | "el_tool_annotations": [ 163 | { 164 | "mention": "Popeye", 165 | "entity": "Popeye" 166 | } 167 | ] 168 | } 169 | } 170 | }, 171 | { 172 | "dialogue_id": "5336", 173 | "dataset_name": "wow", 174 | "turns": { 175 | "0": { 176 | "turn_number": 0, 177 | "el_tool_annotations": [] 178 | }, 179 | "2": { 180 | "turn_number": 2, 181 | "el_tool_annotations": [] 182 | }, 183 | "4": { 184 | "turn_number": 4, 185 | "el_tool_annotations": [] 186 | }, 187 | "6": { 188 | "turn_number": 6, 189 | "el_tool_annotations": [ 190 | { 191 | "mention": "Reese", 192 | "entity": "Reese_Witherspoon" 193 | } 194 | ] 195 | } 196 | } 197 | }, 198 | { 199 | "dialogue_id": "1383", 200 | "dataset_name": "wow", 201 | "turns": { 202 | "0": { 203 | "turn_number": 0, 204 | "el_tool_annotations": [ 205 | { 206 | "mention": "Mediterranean", 207 | "entity": "Mediterranean_cuisine" 208 | } 209 | ] 210 | }, 211 | "2": { 212 | "turn_number": 2, 213 | "el_tool_annotations": [ 214 | { 215 | "mention": "Mediterranean", 216 | "entity": "Mediterranean_cuisine" 217 | } 218 | ] 219 | }, 220 | "4": { 221 | "turn_number": 4, 222 | "el_tool_annotations": [] 223 | }, 224 | "6": { 225 | "turn_number": 6, 226 | "el_tool_annotations": [] 227 | } 228 | } 229 | }, 230 | { 231 | "dialogue_id": "15621", 232 | "dataset_name": "wow", 233 | "turns": { 234 | "0": { 235 | "turn_number": 0, 236 | "el_tool_annotations": [] 237 | }, 238 | "2": { 239 | "turn_number": 2, 240 | "el_tool_annotations": [] 241 | }, 242 | "4": { 243 | "turn_number": 4, 244 | "el_tool_annotations": [] 245 | }, 246 | "6": { 247 | "turn_number": 6, 248 | "el_tool_annotations": [] 249 | } 250 | } 251 | }, 252 | { 253 | "dialogue_id": "1161", 254 | "dataset_name": "wow", 255 | "turns": { 256 | "0": { 257 | "turn_number": 0, 258 | "el_tool_annotations": [] 259 | }, 260 | "2": { 261 | "turn_number": 2, 262 | "el_tool_annotations": [ 263 | { 264 | "mention": "Carvel", 265 | "entity": "Carvel_(franchise)" 266 | } 267 | ] 268 | }, 269 | "4": { 270 | "turn_number": 4, 271 | "el_tool_annotations": [] 272 | }, 273 | "6": { 274 | "turn_number": 6, 275 | "el_tool_annotations": [] 276 | } 277 | } 278 | }, 279 | { 280 | "dialogue_id": "7957", 281 | "dataset_name": "wow", 282 | "turns": { 283 | "1": { 284 | "turn_number": 1, 285 | "el_tool_annotations": [] 286 | }, 287 | "3": { 288 | "turn_number": 3, 289 | "el_tool_annotations": [ 290 | { 291 | "mention": "Oreo", 292 | "entity": "Oreo" 293 | } 294 | ] 295 | }, 296 | "5": { 297 | "turn_number": 5, 298 | "el_tool_annotations": [] 299 | }, 300 | "7": { 301 | "turn_number": 7, 302 | "el_tool_annotations": [] 303 | }, 304 | "9": { 305 | "turn_number": 9, 306 | "el_tool_annotations": [] 307 | } 308 | } 309 | }, 310 | { 311 | "dialogue_id": "2573", 312 | "dataset_name": "wow", 313 | "turns": { 314 | "1": { 315 | "turn_number": 1, 316 | "el_tool_annotations": [ 317 | { 318 | "mention": "Vanilla", 319 | "entity": "Conventional_sex" 320 | }, 321 | { 322 | "mention": "Strawberry", 323 | "entity": "Darryl_Strawberry" 324 | } 325 | ] 326 | }, 327 | "3": { 328 | "turn_number": 3, 329 | "el_tool_annotations": [] 330 | }, 331 | "5": { 332 | "turn_number": 5, 333 | "el_tool_annotations": [] 334 | }, 335 | "7": { 336 | "turn_number": 7, 337 | "el_tool_annotations": [ 338 | { 339 | "mention": "A&W", 340 | "entity": "A&W_Restaurants" 341 | } 342 | ] 343 | }, 344 | "9": { 345 | "turn_number": 9, 346 | "el_tool_annotations": [] 347 | } 348 | } 349 | }, 350 | { 351 | "dialogue_id": "13604", 352 | "dataset_name": "wow", 353 | "turns": { 354 | "0": { 355 | "turn_number": 0, 356 | "el_tool_annotations": [ 357 | { 358 | "mention": "Pizza", 359 | "entity": "Pizza" 360 | } 361 | ] 362 | }, 363 | "2": { 364 | "turn_number": 2, 365 | "el_tool_annotations": [] 366 | }, 367 | "4": { 368 | "turn_number": 4, 369 | "el_tool_annotations": [] 370 | }, 371 | "6": { 372 | "turn_number": 6, 373 | "el_tool_annotations": [] 374 | }, 375 | "8": { 376 | "turn_number": 8, 377 | "el_tool_annotations": [ 378 | { 379 | "mention": "Italy", 380 | "entity": "Italy" 381 | } 382 | ] 383 | } 384 | } 385 | }, 386 | { 387 | "dialogue_id": "873", 388 | "dataset_name": "wow", 389 | "turns": { 390 | "0": { 391 | "turn_number": 0, 392 | "el_tool_annotations": [] 393 | }, 394 | "2": { 395 | "turn_number": 2, 396 | "el_tool_annotations": [] 397 | }, 398 | "4": { 399 | "turn_number": 4, 400 | "el_tool_annotations": [] 401 | }, 402 | "6": { 403 | "turn_number": 6, 404 | "el_tool_annotations": [] 405 | } 406 | } 407 | }, 408 | { 409 | "dialogue_id": "22270", 410 | "dataset_name": "wow", 411 | "turns": { 412 | "0": { 413 | "turn_number": 0, 414 | "el_tool_annotations": [] 415 | }, 416 | "2": { 417 | "turn_number": 2, 418 | "el_tool_annotations": [] 419 | }, 420 | "4": { 421 | "turn_number": 4, 422 | "el_tool_annotations": [] 423 | }, 424 | "6": { 425 | "turn_number": 6, 426 | "el_tool_annotations": [] 427 | }, 428 | "8": { 429 | "turn_number": 8, 430 | "el_tool_annotations": [] 431 | } 432 | } 433 | }, 434 | { 435 | "dialogue_id": "17206", 436 | "dataset_name": "wow", 437 | "turns": { 438 | "0": { 439 | "turn_number": 0, 440 | "el_tool_annotations": [ 441 | { 442 | "mention": "Mavericks", 443 | "entity": "Mavericks,_California" 444 | } 445 | ] 446 | }, 447 | "2": { 448 | "turn_number": 2, 449 | "el_tool_annotations": [] 450 | }, 451 | "4": { 452 | "turn_number": 4, 453 | "el_tool_annotations": [] 454 | }, 455 | "6": { 456 | "turn_number": 6, 457 | "el_tool_annotations": [ 458 | { 459 | "mention": "NBA", 460 | "entity": "National_Basketball_Association" 461 | } 462 | ] 463 | }, 464 | "8": { 465 | "turn_number": 8, 466 | "el_tool_annotations": [] 467 | } 468 | } 469 | }, 470 | { 471 | "dialogue_id": "16339", 472 | "dataset_name": "wow", 473 | "turns": { 474 | "0": { 475 | "turn_number": 0, 476 | "el_tool_annotations": [ 477 | { 478 | "mention": "American", 479 | "entity": "American_football" 480 | } 481 | ] 482 | }, 483 | "2": { 484 | "turn_number": 2, 485 | "el_tool_annotations": [ 486 | { 487 | "mention": "New England Patriots", 488 | "entity": "New_England_Patriots" 489 | }, 490 | { 491 | "mention": "New England", 492 | "entity": "New_England" 493 | } 494 | ] 495 | }, 496 | "4": { 497 | "turn_number": 4, 498 | "el_tool_annotations": [ 499 | { 500 | "mention": "Dolphins", 501 | "entity": "Miami_Dolphins" 502 | }, 503 | { 504 | "mention": "Marino", 505 | "entity": "Dan_Marino" 506 | } 507 | ] 508 | }, 509 | "6": { 510 | "turn_number": 6, 511 | "el_tool_annotations": [ 512 | { 513 | "mention": "NFL", 514 | "entity": "Monday_Night_Football" 515 | }, 516 | { 517 | "mention": "Peyton Manning", 518 | "entity": "Peyton_Manning" 519 | } 520 | ] 521 | }, 522 | "8": { 523 | "turn_number": 8, 524 | "el_tool_annotations": [ 525 | { 526 | "mention": "NFL", 527 | "entity": "Monday_Night_Football" 528 | }, 529 | { 530 | "mention": "Peyton Manning", 531 | "entity": "Peyton_Manning" 532 | } 533 | ] 534 | } 535 | } 536 | }, 537 | { 538 | "dialogue_id": "17006", 539 | "dataset_name": "wow", 540 | "turns": { 541 | "0": { 542 | "turn_number": 0, 543 | "el_tool_annotations": [] 544 | }, 545 | "2": { 546 | "turn_number": 2, 547 | "el_tool_annotations": [] 548 | }, 549 | "4": { 550 | "turn_number": 4, 551 | "el_tool_annotations": [ 552 | { 553 | "mention": "Violet", 554 | "entity": "Violet_(color)" 555 | } 556 | ] 557 | }, 558 | "6": { 559 | "turn_number": 6, 560 | "el_tool_annotations": [ 561 | { 562 | "mention": "Japan", 563 | "entity": "Japan" 564 | } 565 | ] 566 | } 567 | } 568 | }, 569 | { 570 | "dialogue_id": "1168", 571 | "dataset_name": "wow", 572 | "turns": { 573 | "0": { 574 | "turn_number": 0, 575 | "el_tool_annotations": [] 576 | }, 577 | "2": { 578 | "turn_number": 2, 579 | "el_tool_annotations": [] 580 | }, 581 | "4": { 582 | "turn_number": 4, 583 | "el_tool_annotations": [ 584 | { 585 | "mention": "italian", 586 | "entity": "Italian_cuisine" 587 | } 588 | ] 589 | }, 590 | "6": { 591 | "turn_number": 6, 592 | "el_tool_annotations": [] 593 | }, 594 | "8": { 595 | "turn_number": 8, 596 | "el_tool_annotations": [] 597 | } 598 | } 599 | }, 600 | { 601 | "dialogue_id": "17067", 602 | "dataset_name": "wow", 603 | "turns": { 604 | "0": { 605 | "turn_number": 0, 606 | "el_tool_annotations": [ 607 | { 608 | "mention": "Channing Tatum", 609 | "entity": "Channing_Tatum" 610 | } 611 | ] 612 | }, 613 | "2": { 614 | "turn_number": 2, 615 | "el_tool_annotations": [] 616 | }, 617 | "4": { 618 | "turn_number": 4, 619 | "el_tool_annotations": [ 620 | { 621 | "mention": "Jump Street", 622 | "entity": "21_Jump_Street" 623 | }, 624 | { 625 | "mention": "Magic Mike", 626 | "entity": "Magic_Mike" 627 | } 628 | ] 629 | }, 630 | "6": { 631 | "turn_number": 6, 632 | "el_tool_annotations": [] 633 | } 634 | } 635 | }, 636 | { 637 | "dialogue_id": "4371", 638 | "dataset_name": "wow", 639 | "turns": { 640 | "0": { 641 | "turn_number": 0, 642 | "el_tool_annotations": [] 643 | }, 644 | "2": { 645 | "turn_number": 2, 646 | "el_tool_annotations": [] 647 | }, 648 | "4": { 649 | "turn_number": 4, 650 | "el_tool_annotations": [] 651 | }, 652 | "6": { 653 | "turn_number": 6, 654 | "el_tool_annotations": [] 655 | }, 656 | "8": { 657 | "turn_number": 8, 658 | "el_tool_annotations": [] 659 | } 660 | } 661 | }, 662 | { 663 | "dialogue_id": "12407", 664 | "dataset_name": "wow", 665 | "turns": { 666 | "1": { 667 | "turn_number": 1, 668 | "el_tool_annotations": [] 669 | }, 670 | "3": { 671 | "turn_number": 3, 672 | "el_tool_annotations": [] 673 | }, 674 | "5": { 675 | "turn_number": 5, 676 | "el_tool_annotations": [ 677 | { 678 | "mention": "Salmon", 679 | "entity": "Salmon" 680 | } 681 | ] 682 | }, 683 | "7": { 684 | "turn_number": 7, 685 | "el_tool_annotations": [] 686 | } 687 | } 688 | }, 689 | { 690 | "dialogue_id": "8214", 691 | "dataset_name": "wow", 692 | "turns": { 693 | "0": { 694 | "turn_number": 0, 695 | "el_tool_annotations": [ 696 | { 697 | "mention": "Pearl Jam", 698 | "entity": "Pearl_Jam" 699 | } 700 | ] 701 | }, 702 | "2": { 703 | "turn_number": 2, 704 | "el_tool_annotations": [ 705 | { 706 | "mention": "Kurt Cobain", 707 | "entity": "Suicide_of_Kurt_Cobain" 708 | }, 709 | { 710 | "mention": "Dave Grohl", 711 | "entity": "Dave_Grohl" 712 | } 713 | ] 714 | }, 715 | "4": { 716 | "turn_number": 4, 717 | "el_tool_annotations": [ 718 | { 719 | "mention": "Kurt", 720 | "entity": "Kurt" 721 | } 722 | ] 723 | }, 724 | "6": { 725 | "turn_number": 6, 726 | "el_tool_annotations": [ 727 | { 728 | "mention": "Matt Cameron", 729 | "entity": "Matt_Cameron" 730 | }, 731 | { 732 | "mention": "Soundgarden", 733 | "entity": "Soundgarden" 734 | } 735 | ] 736 | }, 737 | "8": { 738 | "turn_number": 8, 739 | "el_tool_annotations": [ 740 | { 741 | "mention": "Pearl Jam", 742 | "entity": "Pearl_Jam" 743 | } 744 | ] 745 | } 746 | } 747 | }, 748 | { 749 | "dialogue_id": "7051", 750 | "dataset_name": "wow", 751 | "turns": { 752 | "0": { 753 | "turn_number": 0, 754 | "el_tool_annotations": [] 755 | }, 756 | "2": { 757 | "turn_number": 2, 758 | "el_tool_annotations": [] 759 | }, 760 | "4": { 761 | "turn_number": 4, 762 | "el_tool_annotations": [] 763 | }, 764 | "6": { 765 | "turn_number": 6, 766 | "el_tool_annotations": [] 767 | }, 768 | "8": { 769 | "turn_number": 8, 770 | "el_tool_annotations": [] 771 | } 772 | } 773 | }, 774 | { 775 | "dialogue_id": "1266", 776 | "dataset_name": "wow", 777 | "turns": { 778 | "0": { 779 | "turn_number": 0, 780 | "el_tool_annotations": [] 781 | }, 782 | "2": { 783 | "turn_number": 2, 784 | "el_tool_annotations": [] 785 | }, 786 | "4": { 787 | "turn_number": 4, 788 | "el_tool_annotations": [] 789 | }, 790 | "6": { 791 | "turn_number": 6, 792 | "el_tool_annotations": [] 793 | }, 794 | "8": { 795 | "turn_number": 8, 796 | "el_tool_annotations": [] 797 | } 798 | } 799 | }, 800 | { 801 | "dialogue_id": "2040", 802 | "dataset_name": "wow", 803 | "turns": { 804 | "0": { 805 | "turn_number": 0, 806 | "el_tool_annotations": [ 807 | { 808 | "mention": "Beagle", 809 | "entity": "Beagle" 810 | }, 811 | { 812 | "mention": "Droopy", 813 | "entity": "Droopy" 814 | } 815 | ] 816 | }, 817 | "2": { 818 | "turn_number": 2, 819 | "el_tool_annotations": [] 820 | }, 821 | "4": { 822 | "turn_number": 4, 823 | "el_tool_annotations": [ 824 | { 825 | "mention": "Beagles", 826 | "entity": "Animal_testing" 827 | } 828 | ] 829 | }, 830 | "6": { 831 | "turn_number": 6, 832 | "el_tool_annotations": [ 833 | { 834 | "mention": "Beagle", 835 | "entity": "Beagle" 836 | } 837 | ] 838 | } 839 | } 840 | } 841 | ] -------------------------------------------------------------------------------- /data/ConEL_Personal_Entity/runs/ConEL_PE-Tagme_history.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "dialogue_id": "10060", 4 | "dataset_name": "wow", 5 | "turns": { 6 | "0": { 7 | "turn_number": 0, 8 | "el_tool_annotations": [] 9 | }, 10 | "2": { 11 | "turn_number": 2, 12 | "el_tool_annotations": [ 13 | { 14 | "mention": "ROYGBIV", 15 | "entity": "ROYGBIV" 16 | } 17 | ] 18 | }, 19 | "4": { 20 | "turn_number": 4, 21 | "el_tool_annotations": [ 22 | { 23 | "mention": "when I start to", 24 | "entity": "When_I_Start_To_(Break_It_All_Down)" 25 | }, 26 | { 27 | "mention": "can", 28 | "entity": "Can_(band)" 29 | }, 30 | { 31 | "mention": "light", 32 | "entity": "Light" 33 | } 34 | ] 35 | }, 36 | "6": { 37 | "turn_number": 6, 38 | "el_tool_annotations": [ 39 | { 40 | "mention": "reflecting", 41 | "entity": "Reflection_(physics)" 42 | }, 43 | { 44 | "mention": "lol", 45 | "entity": "LOL" 46 | } 47 | ] 48 | } 49 | } 50 | }, 51 | { 52 | "dialogue_id": "1191", 53 | "dataset_name": "wow", 54 | "turns": { 55 | "1": { 56 | "turn_number": 1, 57 | "el_tool_annotations": [ 58 | { 59 | "mention": "Hard Rock", 60 | "entity": "Hard_rock" 61 | } 62 | ] 63 | }, 64 | "3": { 65 | "turn_number": 3, 66 | "el_tool_annotations": [ 67 | { 68 | "mention": "I like it", 69 | "entity": "I_Like_It_(Enrique_Iglesias_song)" 70 | }, 71 | { 72 | "mention": "Iggy Pop", 73 | "entity": "Iggy_Pop" 74 | }, 75 | { 76 | "mention": "time", 77 | "entity": "The_Original_7ven" 78 | } 79 | ] 80 | }, 81 | "5": { 82 | "turn_number": 5, 83 | "el_tool_annotations": [ 84 | { 85 | "mention": "my passion", 86 | "entity": "My_Passion" 87 | }, 88 | { 89 | "mention": "classic rock", 90 | "entity": "Rock_music" 91 | }, 92 | { 93 | "mention": "White Snake", 94 | "entity": "White_Snake_(album)" 95 | } 96 | ] 97 | }, 98 | "7": { 99 | "turn_number": 7, 100 | "el_tool_annotations": [ 101 | { 102 | "mention": "Glen Danzig", 103 | "entity": "Glenn_Danzig" 104 | }, 105 | { 106 | "mention": "Misfit", 107 | "entity": "Misfits_(band)" 108 | }, 109 | { 110 | "mention": "Elvis", 111 | "entity": "Elvis_Presley" 112 | }, 113 | { 114 | "mention": "Johnny Cash", 115 | "entity": "Johnny_Cash" 116 | } 117 | ] 118 | } 119 | } 120 | }, 121 | { 122 | "dialogue_id": "17985", 123 | "dataset_name": "wow", 124 | "turns": { 125 | "0": { 126 | "turn_number": 0, 127 | "el_tool_annotations": [] 128 | }, 129 | "2": { 130 | "turn_number": 2, 131 | "el_tool_annotations": [ 132 | { 133 | "mention": "primary color", 134 | "entity": "Primary_color" 135 | } 136 | ] 137 | }, 138 | "4": { 139 | "turn_number": 4, 140 | "el_tool_annotations": [] 141 | }, 142 | "6": { 143 | "turn_number": 6, 144 | "el_tool_annotations": [ 145 | { 146 | "mention": "Oh ok", 147 | "entity": "Oh-OK" 148 | }, 149 | { 150 | "mention": "red", 151 | "entity": "Red" 152 | }, 153 | { 154 | "mention": "yellow", 155 | "entity": "Yellow" 156 | }, 157 | { 158 | "mention": "orange", 159 | "entity": "Orange_(colour)" 160 | }, 161 | { 162 | "mention": "yellow", 163 | "entity": "Yellow" 164 | }, 165 | { 166 | "mention": "green", 167 | "entity": "Green" 168 | }, 169 | { 170 | "mention": "red", 171 | "entity": "Red" 172 | }, 173 | { 174 | "mention": "blue", 175 | "entity": "Blue" 176 | }, 177 | { 178 | "mention": "purple", 179 | "entity": "Purple" 180 | } 181 | ] 182 | }, 183 | "8": { 184 | "turn_number": 8, 185 | "el_tool_annotations": [] 186 | } 187 | } 188 | }, 189 | { 190 | "dialogue_id": "16840", 191 | "dataset_name": "wow", 192 | "turns": { 193 | "0": { 194 | "turn_number": 0, 195 | "el_tool_annotations": [ 196 | { 197 | "mention": "youtube", 198 | "entity": "YouTube" 199 | } 200 | ] 201 | }, 202 | "2": { 203 | "turn_number": 2, 204 | "el_tool_annotations": [ 205 | { 206 | "mention": "youtube", 207 | "entity": "YouTube" 208 | } 209 | ] 210 | }, 211 | "4": { 212 | "turn_number": 4, 213 | "el_tool_annotations": [ 214 | { 215 | "mention": "per-say", 216 | "entity": "PerSay" 217 | } 218 | ] 219 | }, 220 | "6": { 221 | "turn_number": 6, 222 | "el_tool_annotations": [ 223 | { 224 | "mention": "socially responsible", 225 | "entity": "Corporate_social_responsibility" 226 | } 227 | ] 228 | } 229 | } 230 | }, 231 | { 232 | "dialogue_id": "15141", 233 | "dataset_name": "wow", 234 | "turns": { 235 | "0": { 236 | "turn_number": 0, 237 | "el_tool_annotations": [] 238 | }, 239 | "2": { 240 | "turn_number": 2, 241 | "el_tool_annotations": [] 242 | }, 243 | "4": { 244 | "turn_number": 4, 245 | "el_tool_annotations": [ 246 | { 247 | "mention": "pepperoni", 248 | "entity": "Pepperoni" 249 | }, 250 | { 251 | "mention": "meats", 252 | "entity": "Meat" 253 | }, 254 | { 255 | "mention": "sausage", 256 | "entity": "Sausage" 257 | }, 258 | { 259 | "mention": "bell peppers", 260 | "entity": "Bell_pepper" 261 | }, 262 | { 263 | "mention": "pizza", 264 | "entity": "Pizza" 265 | } 266 | ] 267 | }, 268 | "6": { 269 | "turn_number": 6, 270 | "el_tool_annotations": [] 271 | }, 272 | "8": { 273 | "turn_number": 8, 274 | "el_tool_annotations": [ 275 | { 276 | "mention": "spinach", 277 | "entity": "Spinach" 278 | }, 279 | { 280 | "mention": "Popeye", 281 | "entity": "Popeye" 282 | }, 283 | { 284 | "mention": "spinach", 285 | "entity": "Spinach" 286 | } 287 | ] 288 | } 289 | } 290 | }, 291 | { 292 | "dialogue_id": "5336", 293 | "dataset_name": "wow", 294 | "turns": { 295 | "0": { 296 | "turn_number": 0, 297 | "el_tool_annotations": [ 298 | { 299 | "mention": "chocolate", 300 | "entity": "Chocolate" 301 | } 302 | ] 303 | }, 304 | "2": { 305 | "turn_number": 2, 306 | "el_tool_annotations": [ 307 | { 308 | "mention": "milk chocolate", 309 | "entity": "Types_of_chocolate" 310 | }, 311 | { 312 | "mention": "M&Ms", 313 | "entity": "M&M's" 314 | }, 315 | { 316 | "mention": "chocolate covered peanuts", 317 | "entity": "Chocolate-coated_peanut" 318 | } 319 | ] 320 | }, 321 | "4": { 322 | "turn_number": 4, 323 | "el_tool_annotations": [] 324 | }, 325 | "6": { 326 | "turn_number": 6, 327 | "el_tool_annotations": [ 328 | { 329 | "mention": "Reese's Pieces", 330 | "entity": "Reese's_Pieces" 331 | } 332 | ] 333 | } 334 | } 335 | }, 336 | { 337 | "dialogue_id": "1383", 338 | "dataset_name": "wow", 339 | "turns": { 340 | "0": { 341 | "turn_number": 0, 342 | "el_tool_annotations": [ 343 | { 344 | "mention": "Mediterranean cuisine", 345 | "entity": "Mediterranean_cuisine" 346 | }, 347 | { 348 | "mention": "love stuff", 349 | "entity": "Love_Stuff" 350 | }, 351 | { 352 | "mention": "pita", 353 | "entity": "Pita" 354 | }, 355 | { 356 | "mention": "hummus", 357 | "entity": "Hummus" 358 | } 359 | ] 360 | }, 361 | "2": { 362 | "turn_number": 2, 363 | "el_tool_annotations": [ 364 | { 365 | "mention": "Mediterranean cuisine", 366 | "entity": "Mediterranean_cuisine" 367 | } 368 | ] 369 | }, 370 | "4": { 371 | "turn_number": 4, 372 | "el_tool_annotations": [ 373 | { 374 | "mention": "wine", 375 | "entity": "Wine" 376 | }, 377 | { 378 | "mention": "olive oil", 379 | "entity": "Olive_oil" 380 | } 381 | ] 382 | }, 383 | "6": { 384 | "turn_number": 6, 385 | "el_tool_annotations": [] 386 | } 387 | } 388 | }, 389 | { 390 | "dialogue_id": "15621", 391 | "dataset_name": "wow", 392 | "turns": { 393 | "0": { 394 | "turn_number": 0, 395 | "el_tool_annotations": [ 396 | { 397 | "mention": "cheetos", 398 | "entity": "Cheetos" 399 | } 400 | ] 401 | }, 402 | "2": { 403 | "turn_number": 2, 404 | "el_tool_annotations": [ 405 | { 406 | "mention": "cheeto", 407 | "entity": "Cheetos" 408 | } 409 | ] 410 | }, 411 | "4": { 412 | "turn_number": 4, 413 | "el_tool_annotations": [ 414 | { 415 | "mention": "flaming hot cheetos", 416 | "entity": "Cheetos" 417 | }, 418 | { 419 | "mention": "cheetos", 420 | "entity": "Cheetos" 421 | } 422 | ] 423 | }, 424 | "6": { 425 | "turn_number": 6, 426 | "el_tool_annotations": [] 427 | } 428 | } 429 | }, 430 | { 431 | "dialogue_id": "1161", 432 | "dataset_name": "wow", 433 | "turns": { 434 | "0": { 435 | "turn_number": 0, 436 | "el_tool_annotations": [ 437 | { 438 | "mention": "Ice cream", 439 | "entity": "Ice_Cream_(Raekwon_song)" 440 | }, 441 | { 442 | "mention": "sliced bread", 443 | "entity": "Sliced_bread" 444 | } 445 | ] 446 | }, 447 | "2": { 448 | "turn_number": 2, 449 | "el_tool_annotations": [ 450 | { 451 | "mention": "cookies and cream", 452 | "entity": "Cookies_and_Cream" 453 | }, 454 | { 455 | "mention": "Carvel ice cream", 456 | "entity": "Carvel_(restaurant)" 457 | }, 458 | { 459 | "mention": "ice cream cake", 460 | "entity": "Ice_cream_cake" 461 | } 462 | ] 463 | }, 464 | "4": { 465 | "turn_number": 4, 466 | "el_tool_annotations": [ 467 | { 468 | "mention": "italy", 469 | "entity": "Italy" 470 | }, 471 | { 472 | "mention": "ice cream", 473 | "entity": "Ice_cream" 474 | }, 475 | { 476 | "mention": "gelato", 477 | "entity": "Gelato" 478 | }, 479 | { 480 | "mention": "ice cream", 481 | "entity": "Ice_cream" 482 | } 483 | ] 484 | }, 485 | "6": { 486 | "turn_number": 6, 487 | "el_tool_annotations": [ 488 | { 489 | "mention": "frozen yogurt", 490 | "entity": "Frozen_yogurt" 491 | }, 492 | { 493 | "mention": "pinkberry", 494 | "entity": "Pinkberry" 495 | } 496 | ] 497 | } 498 | } 499 | }, 500 | { 501 | "dialogue_id": "7957", 502 | "dataset_name": "wow", 503 | "turns": { 504 | "1": { 505 | "turn_number": 1, 506 | "el_tool_annotations": [ 507 | { 508 | "mention": "lactose intolerant", 509 | "entity": "Lactose_intolerance" 510 | } 511 | ] 512 | }, 513 | "3": { 514 | "turn_number": 3, 515 | "el_tool_annotations": [ 516 | { 517 | "mention": "Oreo", 518 | "entity": "Oreo" 519 | } 520 | ] 521 | }, 522 | "5": { 523 | "turn_number": 5, 524 | "el_tool_annotations": [ 525 | { 526 | "mention": "lactose", 527 | "entity": "Lactose" 528 | } 529 | ] 530 | }, 531 | "7": { 532 | "turn_number": 7, 533 | "el_tool_annotations": [] 534 | }, 535 | "9": { 536 | "turn_number": 9, 537 | "el_tool_annotations": [ 538 | { 539 | "mention": "organic molecule", 540 | "entity": "Organic_compound" 541 | }, 542 | { 543 | "mention": "fructose", 544 | "entity": "Fructose" 545 | }, 546 | { 547 | "mention": "lactose", 548 | "entity": "Lactose" 549 | } 550 | ] 551 | } 552 | } 553 | }, 554 | { 555 | "dialogue_id": "2573", 556 | "dataset_name": "wow", 557 | "turns": { 558 | "1": { 559 | "turn_number": 1, 560 | "el_tool_annotations": [ 561 | { 562 | "mention": "Vanilla", 563 | "entity": "Vanilla" 564 | } 565 | ] 566 | }, 567 | "3": { 568 | "turn_number": 3, 569 | "el_tool_annotations": [ 570 | { 571 | "mention": "cookies", 572 | "entity": "Cookie" 573 | } 574 | ] 575 | }, 576 | "5": { 577 | "turn_number": 5, 578 | "el_tool_annotations": [ 579 | { 580 | "mention": "mint chocolate chip", 581 | "entity": "Mint_chocolate_chip" 582 | } 583 | ] 584 | }, 585 | "7": { 586 | "turn_number": 7, 587 | "el_tool_annotations": [ 588 | { 589 | "mention": "Oh man", 590 | "entity": "Oh,_For_a_Man!" 591 | }, 592 | { 593 | "mention": "root beer float", 594 | "entity": "Ice_cream_float" 595 | } 596 | ] 597 | }, 598 | "9": { 599 | "turn_number": 9, 600 | "el_tool_annotations": [ 601 | { 602 | "mention": "lol", 603 | "entity": "LOL" 604 | } 605 | ] 606 | } 607 | } 608 | }, 609 | { 610 | "dialogue_id": "13604", 611 | "dataset_name": "wow", 612 | "turns": { 613 | "0": { 614 | "turn_number": 0, 615 | "el_tool_annotations": [] 616 | }, 617 | "2": { 618 | "turn_number": 2, 619 | "el_tool_annotations": [ 620 | { 621 | "mention": "meats", 622 | "entity": "Meat" 623 | }, 624 | { 625 | "mention": "vegetables", 626 | "entity": "Vegetable" 627 | } 628 | ] 629 | }, 630 | "4": { 631 | "turn_number": 4, 632 | "el_tool_annotations": [ 633 | { 634 | "mention": "pepperoni", 635 | "entity": "Pepperoni" 636 | } 637 | ] 638 | }, 639 | "6": { 640 | "turn_number": 6, 641 | "el_tool_annotations": [] 642 | }, 643 | "8": { 644 | "turn_number": 8, 645 | "el_tool_annotations": [ 646 | { 647 | "mention": "Italy", 648 | "entity": "Italy" 649 | } 650 | ] 651 | } 652 | } 653 | }, 654 | { 655 | "dialogue_id": "873", 656 | "dataset_name": "wow", 657 | "turns": { 658 | "0": { 659 | "turn_number": 0, 660 | "el_tool_annotations": [ 661 | { 662 | "mention": "I love to eat", 663 | "entity": "I_Love_to_Eat" 664 | }, 665 | { 666 | "mention": "sushi", 667 | "entity": "Sushi" 668 | }, 669 | { 670 | "mention": "fish", 671 | "entity": "Fish_as_food" 672 | } 673 | ] 674 | }, 675 | "2": { 676 | "turn_number": 2, 677 | "el_tool_annotations": [] 678 | }, 679 | "4": { 680 | "turn_number": 4, 681 | "el_tool_annotations": [ 682 | { 683 | "mention": "rice vinegar", 684 | "entity": "Rice_vinegar" 685 | }, 686 | { 687 | "mention": "it. It", 688 | "entity": "Laconic_phrase" 689 | } 690 | ] 691 | }, 692 | "6": { 693 | "turn_number": 6, 694 | "el_tool_annotations": [] 695 | } 696 | } 697 | }, 698 | { 699 | "dialogue_id": "22270", 700 | "dataset_name": "wow", 701 | "turns": { 702 | "0": { 703 | "turn_number": 0, 704 | "el_tool_annotations": [] 705 | }, 706 | "2": { 707 | "turn_number": 2, 708 | "el_tool_annotations": [] 709 | }, 710 | "4": { 711 | "turn_number": 4, 712 | "el_tool_annotations": [] 713 | }, 714 | "6": { 715 | "turn_number": 6, 716 | "el_tool_annotations": [ 717 | { 718 | "mention": "Oh ok", 719 | "entity": "Oh-OK" 720 | } 721 | ] 722 | }, 723 | "8": { 724 | "turn_number": 8, 725 | "el_tool_annotations": [] 726 | } 727 | } 728 | }, 729 | { 730 | "dialogue_id": "17206", 731 | "dataset_name": "wow", 732 | "turns": { 733 | "0": { 734 | "turn_number": 0, 735 | "el_tool_annotations": [ 736 | { 737 | "mention": "Basketball", 738 | "entity": "Basketball" 739 | } 740 | ] 741 | }, 742 | "2": { 743 | "turn_number": 2, 744 | "el_tool_annotations": [] 745 | }, 746 | "4": { 747 | "turn_number": 4, 748 | "el_tool_annotations": [ 749 | { 750 | "mention": "Simple", 751 | "entity": "Curve" 752 | } 753 | ] 754 | }, 755 | "6": { 756 | "turn_number": 6, 757 | "el_tool_annotations": [] 758 | }, 759 | "8": { 760 | "turn_number": 8, 761 | "el_tool_annotations": [] 762 | } 763 | } 764 | }, 765 | { 766 | "dialogue_id": "16339", 767 | "dataset_name": "wow", 768 | "turns": { 769 | "0": { 770 | "turn_number": 0, 771 | "el_tool_annotations": [ 772 | { 773 | "mention": "football", 774 | "entity": "Football" 775 | }, 776 | { 777 | "mention": "American pastime", 778 | "entity": "American_Pastime_(film)" 779 | } 780 | ] 781 | }, 782 | "2": { 783 | "turn_number": 2, 784 | "el_tool_annotations": [ 785 | { 786 | "mention": "New England Patriots", 787 | "entity": "New_England_Patriots" 788 | }, 789 | { 790 | "mention": "aquaintance", 791 | "entity": "Interpersonal_relationship" 792 | } 793 | ] 794 | }, 795 | "4": { 796 | "turn_number": 4, 797 | "el_tool_annotations": [ 798 | { 799 | "mention": "Dolphins", 800 | "entity": "Miami_Dolphins" 801 | }, 802 | { 803 | "mention": "Marino", 804 | "entity": "Dan_Marino" 805 | }, 806 | { 807 | "mention": "quarterback", 808 | "entity": "Quarterback" 809 | }, 810 | { 811 | "mention": "growing up", 812 | "entity": "Growing_Up_(Singaporean_TV_series)" 813 | } 814 | ] 815 | }, 816 | "6": { 817 | "turn_number": 6, 818 | "el_tool_annotations": [ 819 | { 820 | "mention": "commercials", 821 | "entity": "Super_Bowl_commercials" 822 | }, 823 | { 824 | "mention": "Peyton Manning", 825 | "entity": "Peyton_Manning" 826 | } 827 | ] 828 | }, 829 | "8": { 830 | "turn_number": 8, 831 | "el_tool_annotations": [ 832 | { 833 | "mention": "commercials", 834 | "entity": "Super_Bowl_commercials" 835 | }, 836 | { 837 | "mention": "Peyton Manning", 838 | "entity": "Peyton_Manning" 839 | } 840 | ] 841 | } 842 | } 843 | }, 844 | { 845 | "dialogue_id": "17006", 846 | "dataset_name": "wow", 847 | "turns": { 848 | "0": { 849 | "turn_number": 0, 850 | "el_tool_annotations": [] 851 | }, 852 | "2": { 853 | "turn_number": 2, 854 | "el_tool_annotations": [] 855 | }, 856 | "4": { 857 | "turn_number": 4, 858 | "el_tool_annotations": [ 859 | { 860 | "mention": "Violet", 861 | "entity": "Violet_(color)" 862 | } 863 | ] 864 | }, 865 | "6": { 866 | "turn_number": 6, 867 | "el_tool_annotations": [ 868 | { 869 | "mention": "Japan", 870 | "entity": "Empire_of_Japan" 871 | }, 872 | { 873 | "mention": "Emperor", 874 | "entity": "Emperor" 875 | }, 876 | { 877 | "mention": "aristocracy", 878 | "entity": "Aristocracy" 879 | } 880 | ] 881 | } 882 | } 883 | }, 884 | { 885 | "dialogue_id": "1168", 886 | "dataset_name": "wow", 887 | "turns": { 888 | "0": { 889 | "turn_number": 0, 890 | "el_tool_annotations": [] 891 | }, 892 | "2": { 893 | "turn_number": 2, 894 | "el_tool_annotations": [ 895 | { 896 | "mention": "tomato", 897 | "entity": "Tomato" 898 | }, 899 | { 900 | "mention": "tomato sauce", 901 | "entity": "Tomato_sauce" 902 | }, 903 | { 904 | "mention": "fresh", 905 | "entity": "Fruit" 906 | } 907 | ] 908 | }, 909 | "4": { 910 | "turn_number": 4, 911 | "el_tool_annotations": [ 912 | { 913 | "mention": "italian", 914 | "entity": "Italian_cuisine" 915 | } 916 | ] 917 | }, 918 | "6": { 919 | "turn_number": 6, 920 | "el_tool_annotations": [ 921 | { 922 | "mention": "fettucini alfredo", 923 | "entity": "Fettuccine_Alfredo" 924 | } 925 | ] 926 | }, 927 | "8": { 928 | "turn_number": 8, 929 | "el_tool_annotations": [ 930 | { 931 | "mention": "pizza hut", 932 | "entity": "Pizza_Hut" 933 | }, 934 | { 935 | "mention": "pan pizza", 936 | "entity": "Pizza" 937 | }, 938 | { 939 | "mention": "pizza crust", 940 | "entity": "Pizza" 941 | }, 942 | { 943 | "mention": "crust", 944 | "entity": "Bread" 945 | } 946 | ] 947 | } 948 | } 949 | }, 950 | { 951 | "dialogue_id": "17067", 952 | "dataset_name": "wow", 953 | "turns": { 954 | "0": { 955 | "turn_number": 0, 956 | "el_tool_annotations": [ 957 | { 958 | "mention": "Channing Tatum", 959 | "entity": "Channing_Tatum" 960 | } 961 | ] 962 | }, 963 | "2": { 964 | "turn_number": 2, 965 | "el_tool_annotations": [] 966 | }, 967 | "4": { 968 | "turn_number": 4, 969 | "el_tool_annotations": [ 970 | { 971 | "mention": "21 Jump Street", 972 | "entity": "21_Jump_Street" 973 | }, 974 | { 975 | "mention": "but not", 976 | "entity": "For_but_not_with" 977 | }, 978 | { 979 | "mention": "Magic Mike", 980 | "entity": "Magic_Mike" 981 | } 982 | ] 983 | }, 984 | "6": { 985 | "turn_number": 6, 986 | "el_tool_annotations": [] 987 | } 988 | } 989 | }, 990 | { 991 | "dialogue_id": "4371", 992 | "dataset_name": "wow", 993 | "turns": { 994 | "0": { 995 | "turn_number": 0, 996 | "el_tool_annotations": [ 997 | { 998 | "mention": "yellow", 999 | "entity": "Yellow" 1000 | }, 1001 | { 1002 | "mention": "how about you", 1003 | "entity": "How_About_You?" 1004 | } 1005 | ] 1006 | }, 1007 | "2": { 1008 | "turn_number": 2, 1009 | "el_tool_annotations": [] 1010 | }, 1011 | "4": { 1012 | "turn_number": 4, 1013 | "el_tool_annotations": [ 1014 | { 1015 | "mention": "thank you", 1016 | "entity": "Thank_You_(Dido_song)" 1017 | } 1018 | ] 1019 | }, 1020 | "6": { 1021 | "turn_number": 6, 1022 | "el_tool_annotations": [] 1023 | }, 1024 | "8": { 1025 | "turn_number": 8, 1026 | "el_tool_annotations": [ 1027 | { 1028 | "mention": "informative", 1029 | "entity": "Information" 1030 | }, 1031 | { 1032 | "mention": "information", 1033 | "entity": "Information" 1034 | } 1035 | ] 1036 | } 1037 | } 1038 | }, 1039 | { 1040 | "dialogue_id": "12407", 1041 | "dataset_name": "wow", 1042 | "turns": { 1043 | "1": { 1044 | "turn_number": 1, 1045 | "el_tool_annotations": [] 1046 | }, 1047 | "3": { 1048 | "turn_number": 3, 1049 | "el_tool_annotations": [] 1050 | }, 1051 | "5": { 1052 | "turn_number": 5, 1053 | "el_tool_annotations": [ 1054 | { 1055 | "mention": "Salmon", 1056 | "entity": "Salmon" 1057 | } 1058 | ] 1059 | }, 1060 | "7": { 1061 | "turn_number": 7, 1062 | "el_tool_annotations": [] 1063 | } 1064 | } 1065 | }, 1066 | { 1067 | "dialogue_id": "8214", 1068 | "dataset_name": "wow", 1069 | "turns": { 1070 | "0": { 1071 | "turn_number": 0, 1072 | "el_tool_annotations": [ 1073 | { 1074 | "mention": "Pearl Jam", 1075 | "entity": "Pearl_Jam" 1076 | } 1077 | ] 1078 | }, 1079 | "2": { 1080 | "turn_number": 2, 1081 | "el_tool_annotations": [ 1082 | { 1083 | "mention": "Kurt Cobain", 1084 | "entity": "Kurt_Cobain" 1085 | }, 1086 | { 1087 | "mention": "Dave Grohl", 1088 | "entity": "Dave_Grohl" 1089 | } 1090 | ] 1091 | }, 1092 | "4": { 1093 | "turn_number": 4, 1094 | "el_tool_annotations": [] 1095 | }, 1096 | "6": { 1097 | "turn_number": 6, 1098 | "el_tool_annotations": [ 1099 | { 1100 | "mention": "Matt Cameron", 1101 | "entity": "Matt_Cameron" 1102 | }, 1103 | { 1104 | "mention": "him", 1105 | "entity": "HIM_(Finnish_band)" 1106 | }, 1107 | { 1108 | "mention": "Soundgarden", 1109 | "entity": "Soundgarden" 1110 | }, 1111 | { 1112 | "mention": "big", 1113 | "entity": "The_Notorious_B.I.G." 1114 | } 1115 | ] 1116 | }, 1117 | "8": { 1118 | "turn_number": 8, 1119 | "el_tool_annotations": [ 1120 | { 1121 | "mention": "Pearl Jam", 1122 | "entity": "Pearl_Jam" 1123 | }, 1124 | { 1125 | "mention": "music", 1126 | "entity": "The_Music" 1127 | } 1128 | ] 1129 | } 1130 | } 1131 | }, 1132 | { 1133 | "dialogue_id": "7051", 1134 | "dataset_name": "wow", 1135 | "turns": { 1136 | "0": { 1137 | "turn_number": 0, 1138 | "el_tool_annotations": [] 1139 | }, 1140 | "2": { 1141 | "turn_number": 2, 1142 | "el_tool_annotations": [ 1143 | { 1144 | "mention": "cw", 1145 | "entity": "The_CW" 1146 | }, 1147 | { 1148 | "mention": "adult swim", 1149 | "entity": "Adult_Swim" 1150 | }, 1151 | { 1152 | "mention": "tbs", 1153 | "entity": "TBS_(U.S._TV_channel)" 1154 | } 1155 | ] 1156 | }, 1157 | "4": { 1158 | "turn_number": 4, 1159 | "el_tool_annotations": [] 1160 | }, 1161 | "6": { 1162 | "turn_number": 6, 1163 | "el_tool_annotations": [ 1164 | { 1165 | "mention": "you", 1166 | "entity": "With_You_(Chris_Brown_song)" 1167 | } 1168 | ] 1169 | }, 1170 | "8": { 1171 | "turn_number": 8, 1172 | "el_tool_annotations": [ 1173 | { 1174 | "mention": "stewie", 1175 | "entity": "Stewie_Griffin" 1176 | } 1177 | ] 1178 | } 1179 | } 1180 | }, 1181 | { 1182 | "dialogue_id": "1266", 1183 | "dataset_name": "wow", 1184 | "turns": { 1185 | "0": { 1186 | "turn_number": 0, 1187 | "el_tool_annotations": [ 1188 | { 1189 | "mention": "hamburgers", 1190 | "entity": "Hamburger" 1191 | } 1192 | ] 1193 | }, 1194 | "2": { 1195 | "turn_number": 2, 1196 | "el_tool_annotations": [] 1197 | }, 1198 | "4": { 1199 | "turn_number": 4, 1200 | "el_tool_annotations": [] 1201 | }, 1202 | "6": { 1203 | "turn_number": 6, 1204 | "el_tool_annotations": [ 1205 | { 1206 | "mention": "turkey", 1207 | "entity": "Turkey_(bird)" 1208 | }, 1209 | { 1210 | "mention": "turkey burger", 1211 | "entity": "Hamburger" 1212 | } 1213 | ] 1214 | }, 1215 | "8": { 1216 | "turn_number": 8, 1217 | "el_tool_annotations": [] 1218 | } 1219 | } 1220 | }, 1221 | { 1222 | "dialogue_id": "2040", 1223 | "dataset_name": "wow", 1224 | "turns": { 1225 | "0": { 1226 | "turn_number": 0, 1227 | "el_tool_annotations": [ 1228 | { 1229 | "mention": "Droopy", 1230 | "entity": "Droopy" 1231 | } 1232 | ] 1233 | }, 1234 | "2": { 1235 | "turn_number": 2, 1236 | "el_tool_annotations": [ 1237 | { 1238 | "mention": "foxhounds", 1239 | "entity": "English_Foxhound" 1240 | } 1241 | ] 1242 | }, 1243 | "4": { 1244 | "turn_number": 4, 1245 | "el_tool_annotations": [] 1246 | }, 1247 | "6": { 1248 | "turn_number": 6, 1249 | "el_tool_annotations": [ 1250 | { 1251 | "mention": "when we", 1252 | "entity": "When_We_On" 1253 | } 1254 | ] 1255 | } 1256 | } 1257 | } 1258 | ] -------------------------------------------------------------------------------- /data/ConEL_Personal_Entity/runs/ConEL_PE-Tagme_turn.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "dialogue_id": "10060", 4 | "dataset_name": "wow", 5 | "turns": { 6 | "0": { 7 | "turn_number": 0, 8 | "el_tool_annotations": [] 9 | }, 10 | "2": { 11 | "turn_number": 2, 12 | "el_tool_annotations": [ 13 | { 14 | "mention": "ROYGBIV", 15 | "entity": "ROYGBIV" 16 | } 17 | ] 18 | }, 19 | "4": { 20 | "turn_number": 4, 21 | "el_tool_annotations": [ 22 | { 23 | "mention": "when I start to", 24 | "entity": "When_I_Start_To_(Break_It_All_Down)" 25 | }, 26 | { 27 | "mention": "color", 28 | "entity": "Color" 29 | }, 30 | { 31 | "mention": "can", 32 | "entity": "Can_(band)" 33 | }, 34 | { 35 | "mention": "light", 36 | "entity": "Light" 37 | } 38 | ] 39 | }, 40 | "6": { 41 | "turn_number": 6, 42 | "el_tool_annotations": [ 43 | { 44 | "mention": "lol", 45 | "entity": "LOL" 46 | } 47 | ] 48 | } 49 | } 50 | }, 51 | { 52 | "dialogue_id": "1191", 53 | "dataset_name": "wow", 54 | "turns": { 55 | "1": { 56 | "turn_number": 1, 57 | "el_tool_annotations": [ 58 | { 59 | "mention": "Hard Rock", 60 | "entity": "Hard_rock" 61 | } 62 | ] 63 | }, 64 | "3": { 65 | "turn_number": 3, 66 | "el_tool_annotations": [ 67 | { 68 | "mention": "I like it", 69 | "entity": "I_Like_It_(Enrique_Iglesias_song)" 70 | }, 71 | { 72 | "mention": "Iggy Pop", 73 | "entity": "Iggy_Pop" 74 | }, 75 | { 76 | "mention": "time", 77 | "entity": "The_Original_7ven" 78 | } 79 | ] 80 | }, 81 | "5": { 82 | "turn_number": 5, 83 | "el_tool_annotations": [ 84 | { 85 | "mention": "classic rock", 86 | "entity": "Classic_Rock_(magazine)" 87 | }, 88 | { 89 | "mention": "White Snake", 90 | "entity": "White_Snake_(album)" 91 | } 92 | ] 93 | }, 94 | "7": { 95 | "turn_number": 7, 96 | "el_tool_annotations": [ 97 | { 98 | "mention": "Glen Danzig", 99 | "entity": "Glenn_Danzig" 100 | }, 101 | { 102 | "mention": "who", 103 | "entity": "The_Who" 104 | }, 105 | { 106 | "mention": "Misfit", 107 | "entity": "Misfits_(band)" 108 | }, 109 | { 110 | "mention": "Elvis", 111 | "entity": "Elvis_Presley" 112 | }, 113 | { 114 | "mention": "Johnny Cash", 115 | "entity": "Johnny_Cash" 116 | } 117 | ] 118 | } 119 | } 120 | }, 121 | { 122 | "dialogue_id": "17985", 123 | "dataset_name": "wow", 124 | "turns": { 125 | "0": { 126 | "turn_number": 0, 127 | "el_tool_annotations": [] 128 | }, 129 | "2": { 130 | "turn_number": 2, 131 | "el_tool_annotations": [ 132 | { 133 | "mention": "primary color", 134 | "entity": "Primary_color" 135 | } 136 | ] 137 | }, 138 | "4": { 139 | "turn_number": 4, 140 | "el_tool_annotations": [] 141 | }, 142 | "6": { 143 | "turn_number": 6, 144 | "el_tool_annotations": [ 145 | { 146 | "mention": "Oh ok", 147 | "entity": "Oh-OK" 148 | }, 149 | { 150 | "mention": "red", 151 | "entity": "Red" 152 | }, 153 | { 154 | "mention": "yellow", 155 | "entity": "Yellow" 156 | }, 157 | { 158 | "mention": "orange", 159 | "entity": "Orange_(colour)" 160 | }, 161 | { 162 | "mention": "yellow", 163 | "entity": "Yellow" 164 | }, 165 | { 166 | "mention": "green", 167 | "entity": "Green" 168 | }, 169 | { 170 | "mention": "red", 171 | "entity": "Red" 172 | }, 173 | { 174 | "mention": "blue", 175 | "entity": "Blue" 176 | }, 177 | { 178 | "mention": "purple", 179 | "entity": "Purple" 180 | } 181 | ] 182 | }, 183 | "8": { 184 | "turn_number": 8, 185 | "el_tool_annotations": [] 186 | } 187 | } 188 | }, 189 | { 190 | "dialogue_id": "16840", 191 | "dataset_name": "wow", 192 | "turns": { 193 | "0": { 194 | "turn_number": 0, 195 | "el_tool_annotations": [ 196 | { 197 | "mention": "youtube", 198 | "entity": "YouTube" 199 | } 200 | ] 201 | }, 202 | "2": { 203 | "turn_number": 2, 204 | "el_tool_annotations": [ 205 | { 206 | "mention": "youtube", 207 | "entity": "YouTube" 208 | } 209 | ] 210 | }, 211 | "4": { 212 | "turn_number": 4, 213 | "el_tool_annotations": [ 214 | { 215 | "mention": "per-say", 216 | "entity": "PerSay" 217 | } 218 | ] 219 | }, 220 | "6": { 221 | "turn_number": 6, 222 | "el_tool_annotations": [ 223 | { 224 | "mention": "socially responsible", 225 | "entity": "Corporate_social_responsibility" 226 | } 227 | ] 228 | } 229 | } 230 | }, 231 | { 232 | "dialogue_id": "15141", 233 | "dataset_name": "wow", 234 | "turns": { 235 | "0": { 236 | "turn_number": 0, 237 | "el_tool_annotations": [] 238 | }, 239 | "2": { 240 | "turn_number": 2, 241 | "el_tool_annotations": [] 242 | }, 243 | "4": { 244 | "turn_number": 4, 245 | "el_tool_annotations": [ 246 | { 247 | "mention": "pepperoni", 248 | "entity": "Pepperoni" 249 | }, 250 | { 251 | "mention": "meats", 252 | "entity": "Meat" 253 | }, 254 | { 255 | "mention": "sausage", 256 | "entity": "Sausage" 257 | }, 258 | { 259 | "mention": "bell peppers", 260 | "entity": "Bell_pepper" 261 | }, 262 | { 263 | "mention": "pizza", 264 | "entity": "Pizza" 265 | } 266 | ] 267 | }, 268 | "6": { 269 | "turn_number": 6, 270 | "el_tool_annotations": [ 271 | { 272 | "mention": "spinach", 273 | "entity": "Spinach" 274 | }, 275 | { 276 | "mention": "pineapple", 277 | "entity": "Pineapple" 278 | } 279 | ] 280 | }, 281 | "8": { 282 | "turn_number": 8, 283 | "el_tool_annotations": [ 284 | { 285 | "mention": "spinach", 286 | "entity": "Spinach" 287 | }, 288 | { 289 | "mention": "Popeye", 290 | "entity": "Popeye" 291 | }, 292 | { 293 | "mention": "spinach", 294 | "entity": "Spinach" 295 | } 296 | ] 297 | } 298 | } 299 | }, 300 | { 301 | "dialogue_id": "5336", 302 | "dataset_name": "wow", 303 | "turns": { 304 | "0": { 305 | "turn_number": 0, 306 | "el_tool_annotations": [ 307 | { 308 | "mention": "chocolate", 309 | "entity": "Chocolate" 310 | } 311 | ] 312 | }, 313 | "2": { 314 | "turn_number": 2, 315 | "el_tool_annotations": [ 316 | { 317 | "mention": "milk chocolate", 318 | "entity": "Types_of_chocolate" 319 | }, 320 | { 321 | "mention": "M&Ms", 322 | "entity": "M&M's" 323 | }, 324 | { 325 | "mention": "chocolate covered peanuts", 326 | "entity": "Chocolate-coated_peanut" 327 | } 328 | ] 329 | }, 330 | "4": { 331 | "turn_number": 4, 332 | "el_tool_annotations": [] 333 | }, 334 | "6": { 335 | "turn_number": 6, 336 | "el_tool_annotations": [ 337 | { 338 | "mention": "mine", 339 | "entity": "Mine_(Beyonc\u00e9_song)" 340 | }, 341 | { 342 | "mention": "Reese's Pieces", 343 | "entity": "Reese's_Pieces" 344 | } 345 | ] 346 | } 347 | } 348 | }, 349 | { 350 | "dialogue_id": "1383", 351 | "dataset_name": "wow", 352 | "turns": { 353 | "0": { 354 | "turn_number": 0, 355 | "el_tool_annotations": [ 356 | { 357 | "mention": "Mediterranean cuisine", 358 | "entity": "Mediterranean_cuisine" 359 | }, 360 | { 361 | "mention": "love stuff", 362 | "entity": "Love_Stuff" 363 | }, 364 | { 365 | "mention": "pita", 366 | "entity": "Pita" 367 | }, 368 | { 369 | "mention": "hummus", 370 | "entity": "Hummus" 371 | } 372 | ] 373 | }, 374 | "2": { 375 | "turn_number": 2, 376 | "el_tool_annotations": [ 377 | { 378 | "mention": "Mediterranean cuisine", 379 | "entity": "Mediterranean_cuisine" 380 | } 381 | ] 382 | }, 383 | "4": { 384 | "turn_number": 4, 385 | "el_tool_annotations": [ 386 | { 387 | "mention": "wine", 388 | "entity": "Wine" 389 | }, 390 | { 391 | "mention": "olive oil", 392 | "entity": "Olive_oil" 393 | } 394 | ] 395 | }, 396 | "6": { 397 | "turn_number": 6, 398 | "el_tool_annotations": [] 399 | } 400 | } 401 | }, 402 | { 403 | "dialogue_id": "15621", 404 | "dataset_name": "wow", 405 | "turns": { 406 | "0": { 407 | "turn_number": 0, 408 | "el_tool_annotations": [ 409 | { 410 | "mention": "cheetos", 411 | "entity": "Cheetos" 412 | } 413 | ] 414 | }, 415 | "2": { 416 | "turn_number": 2, 417 | "el_tool_annotations": [ 418 | { 419 | "mention": "cheeto", 420 | "entity": "Cheetos" 421 | } 422 | ] 423 | }, 424 | "4": { 425 | "turn_number": 4, 426 | "el_tool_annotations": [ 427 | { 428 | "mention": "flaming hot cheetos", 429 | "entity": "Cheetos" 430 | }, 431 | { 432 | "mention": "cheetos", 433 | "entity": "Cheetos" 434 | } 435 | ] 436 | }, 437 | "6": { 438 | "turn_number": 6, 439 | "el_tool_annotations": [] 440 | } 441 | } 442 | }, 443 | { 444 | "dialogue_id": "1161", 445 | "dataset_name": "wow", 446 | "turns": { 447 | "0": { 448 | "turn_number": 0, 449 | "el_tool_annotations": [ 450 | { 451 | "mention": "Ice cream", 452 | "entity": "Ice_Cream_(Raekwon_song)" 453 | }, 454 | { 455 | "mention": "sliced bread", 456 | "entity": "Sliced_bread" 457 | } 458 | ] 459 | }, 460 | "2": { 461 | "turn_number": 2, 462 | "el_tool_annotations": [ 463 | { 464 | "mention": "cookies and cream", 465 | "entity": "Cookies_and_Cream" 466 | }, 467 | { 468 | "mention": "Carvel ice cream", 469 | "entity": "Carvel_(restaurant)" 470 | }, 471 | { 472 | "mention": "ice cream cake", 473 | "entity": "Ice_cream_cake" 474 | } 475 | ] 476 | }, 477 | "4": { 478 | "turn_number": 4, 479 | "el_tool_annotations": [ 480 | { 481 | "mention": "italy", 482 | "entity": "Italy" 483 | }, 484 | { 485 | "mention": "ice cream", 486 | "entity": "Ice_cream" 487 | }, 488 | { 489 | "mention": "gelato", 490 | "entity": "Gelato" 491 | }, 492 | { 493 | "mention": "ice cream", 494 | "entity": "Ice_cream" 495 | } 496 | ] 497 | }, 498 | "6": { 499 | "turn_number": 6, 500 | "el_tool_annotations": [ 501 | { 502 | "mention": "frozen yogurt", 503 | "entity": "Frozen_yogurt" 504 | }, 505 | { 506 | "mention": "pinkberry", 507 | "entity": "Pinkberry" 508 | } 509 | ] 510 | } 511 | } 512 | }, 513 | { 514 | "dialogue_id": "7957", 515 | "dataset_name": "wow", 516 | "turns": { 517 | "1": { 518 | "turn_number": 1, 519 | "el_tool_annotations": [ 520 | { 521 | "mention": "lactose intolerant", 522 | "entity": "Lactose_intolerance" 523 | } 524 | ] 525 | }, 526 | "3": { 527 | "turn_number": 3, 528 | "el_tool_annotations": [ 529 | { 530 | "mention": "Oreo", 531 | "entity": "Oreo" 532 | } 533 | ] 534 | }, 535 | "5": { 536 | "turn_number": 5, 537 | "el_tool_annotations": [ 538 | { 539 | "mention": "lactose", 540 | "entity": "Lactose" 541 | }, 542 | { 543 | "mention": "good", 544 | "entity": "Value_theory" 545 | } 546 | ] 547 | }, 548 | "7": { 549 | "turn_number": 7, 550 | "el_tool_annotations": [] 551 | }, 552 | "9": { 553 | "turn_number": 9, 554 | "el_tool_annotations": [ 555 | { 556 | "mention": "Sucrose", 557 | "entity": "Sucrose" 558 | }, 559 | { 560 | "mention": "organic molecule", 561 | "entity": "Organic_compound" 562 | }, 563 | { 564 | "mention": "fructose", 565 | "entity": "Fructose" 566 | }, 567 | { 568 | "mention": "lactose", 569 | "entity": "Lactose" 570 | } 571 | ] 572 | } 573 | } 574 | }, 575 | { 576 | "dialogue_id": "2573", 577 | "dataset_name": "wow", 578 | "turns": { 579 | "1": { 580 | "turn_number": 1, 581 | "el_tool_annotations": [ 582 | { 583 | "mention": "Vanilla", 584 | "entity": "Vanilla" 585 | } 586 | ] 587 | }, 588 | "3": { 589 | "turn_number": 3, 590 | "el_tool_annotations": [ 591 | { 592 | "mention": "cookies", 593 | "entity": "Cookie" 594 | } 595 | ] 596 | }, 597 | "5": { 598 | "turn_number": 5, 599 | "el_tool_annotations": [ 600 | { 601 | "mention": "mint chocolate chip", 602 | "entity": "Mint_chocolate_chip" 603 | } 604 | ] 605 | }, 606 | "7": { 607 | "turn_number": 7, 608 | "el_tool_annotations": [ 609 | { 610 | "mention": "Oh man", 611 | "entity": "Oh,_For_a_Man!" 612 | }, 613 | { 614 | "mention": "root beer float", 615 | "entity": "Ice_cream_float" 616 | } 617 | ] 618 | }, 619 | "9": { 620 | "turn_number": 9, 621 | "el_tool_annotations": [ 622 | { 623 | "mention": "lol", 624 | "entity": "LOL" 625 | } 626 | ] 627 | } 628 | } 629 | }, 630 | { 631 | "dialogue_id": "13604", 632 | "dataset_name": "wow", 633 | "turns": { 634 | "0": { 635 | "turn_number": 0, 636 | "el_tool_annotations": [] 637 | }, 638 | "2": { 639 | "turn_number": 2, 640 | "el_tool_annotations": [] 641 | }, 642 | "4": { 643 | "turn_number": 4, 644 | "el_tool_annotations": [ 645 | { 646 | "mention": "pepperoni", 647 | "entity": "Pepperoni" 648 | } 649 | ] 650 | }, 651 | "6": { 652 | "turn_number": 6, 653 | "el_tool_annotations": [] 654 | }, 655 | "8": { 656 | "turn_number": 8, 657 | "el_tool_annotations": [ 658 | { 659 | "mention": "Italy", 660 | "entity": "Italy" 661 | } 662 | ] 663 | } 664 | } 665 | }, 666 | { 667 | "dialogue_id": "873", 668 | "dataset_name": "wow", 669 | "turns": { 670 | "0": { 671 | "turn_number": 0, 672 | "el_tool_annotations": [ 673 | { 674 | "mention": "I love to eat", 675 | "entity": "I_Love_to_Eat" 676 | }, 677 | { 678 | "mention": "sushi", 679 | "entity": "Sushi" 680 | }, 681 | { 682 | "mention": "fish", 683 | "entity": "Fish_as_food" 684 | } 685 | ] 686 | }, 687 | "2": { 688 | "turn_number": 2, 689 | "el_tool_annotations": [] 690 | }, 691 | "4": { 692 | "turn_number": 4, 693 | "el_tool_annotations": [ 694 | { 695 | "mention": "rice vinegar", 696 | "entity": "Rice_vinegar" 697 | }, 698 | { 699 | "mention": "it. It", 700 | "entity": "Laconic_phrase" 701 | } 702 | ] 703 | }, 704 | "6": { 705 | "turn_number": 6, 706 | "el_tool_annotations": [ 707 | { 708 | "mention": "seed", 709 | "entity": "Seed" 710 | } 711 | ] 712 | } 713 | } 714 | }, 715 | { 716 | "dialogue_id": "22270", 717 | "dataset_name": "wow", 718 | "turns": { 719 | "0": { 720 | "turn_number": 0, 721 | "el_tool_annotations": [] 722 | }, 723 | "2": { 724 | "turn_number": 2, 725 | "el_tool_annotations": [] 726 | }, 727 | "4": { 728 | "turn_number": 4, 729 | "el_tool_annotations": [] 730 | }, 731 | "6": { 732 | "turn_number": 6, 733 | "el_tool_annotations": [ 734 | { 735 | "mention": "Oh ok", 736 | "entity": "Oh-OK" 737 | } 738 | ] 739 | }, 740 | "8": { 741 | "turn_number": 8, 742 | "el_tool_annotations": [] 743 | } 744 | } 745 | }, 746 | { 747 | "dialogue_id": "17206", 748 | "dataset_name": "wow", 749 | "turns": { 750 | "0": { 751 | "turn_number": 0, 752 | "el_tool_annotations": [ 753 | { 754 | "mention": "Basketball", 755 | "entity": "Basketball" 756 | } 757 | ] 758 | }, 759 | "2": { 760 | "turn_number": 2, 761 | "el_tool_annotations": [] 762 | }, 763 | "4": { 764 | "turn_number": 4, 765 | "el_tool_annotations": [] 766 | }, 767 | "6": { 768 | "turn_number": 6, 769 | "el_tool_annotations": [] 770 | }, 771 | "8": { 772 | "turn_number": 8, 773 | "el_tool_annotations": [] 774 | } 775 | } 776 | }, 777 | { 778 | "dialogue_id": "16339", 779 | "dataset_name": "wow", 780 | "turns": { 781 | "0": { 782 | "turn_number": 0, 783 | "el_tool_annotations": [ 784 | { 785 | "mention": "football", 786 | "entity": "Football" 787 | }, 788 | { 789 | "mention": "American pastime", 790 | "entity": "American_Pastime_(film)" 791 | } 792 | ] 793 | }, 794 | "2": { 795 | "turn_number": 2, 796 | "el_tool_annotations": [ 797 | { 798 | "mention": "New England Patriots", 799 | "entity": "New_England_Patriots" 800 | }, 801 | { 802 | "mention": "aquaintance", 803 | "entity": "Interpersonal_relationship" 804 | } 805 | ] 806 | }, 807 | "4": { 808 | "turn_number": 4, 809 | "el_tool_annotations": [ 810 | { 811 | "mention": "Dolphins", 812 | "entity": "Miami_Dolphins" 813 | }, 814 | { 815 | "mention": "Marino", 816 | "entity": "Dan_Marino" 817 | }, 818 | { 819 | "mention": "quarterback", 820 | "entity": "Quarterback" 821 | }, 822 | { 823 | "mention": "growing up", 824 | "entity": "Growing_Up_(Singaporean_TV_series)" 825 | } 826 | ] 827 | }, 828 | "6": { 829 | "turn_number": 6, 830 | "el_tool_annotations": [ 831 | { 832 | "mention": "Peyton Manning", 833 | "entity": "Peyton_Manning" 834 | } 835 | ] 836 | }, 837 | "8": { 838 | "turn_number": 8, 839 | "el_tool_annotations": [ 840 | { 841 | "mention": "Peyton Manning", 842 | "entity": "Peyton_Manning" 843 | } 844 | ] 845 | } 846 | } 847 | }, 848 | { 849 | "dialogue_id": "17006", 850 | "dataset_name": "wow", 851 | "turns": { 852 | "0": { 853 | "turn_number": 0, 854 | "el_tool_annotations": [] 855 | }, 856 | "2": { 857 | "turn_number": 2, 858 | "el_tool_annotations": [] 859 | }, 860 | "4": { 861 | "turn_number": 4, 862 | "el_tool_annotations": [ 863 | { 864 | "mention": "Violet", 865 | "entity": "Violet_(color)" 866 | } 867 | ] 868 | }, 869 | "6": { 870 | "turn_number": 6, 871 | "el_tool_annotations": [ 872 | { 873 | "mention": "Japan", 874 | "entity": "Empire_of_Japan" 875 | }, 876 | { 877 | "mention": "Emperor", 878 | "entity": "Emperor" 879 | }, 880 | { 881 | "mention": "aristocracy", 882 | "entity": "Aristocracy" 883 | } 884 | ] 885 | } 886 | } 887 | }, 888 | { 889 | "dialogue_id": "1168", 890 | "dataset_name": "wow", 891 | "turns": { 892 | "0": { 893 | "turn_number": 0, 894 | "el_tool_annotations": [] 895 | }, 896 | "2": { 897 | "turn_number": 2, 898 | "el_tool_annotations": [ 899 | { 900 | "mention": "tomato", 901 | "entity": "Tomato" 902 | }, 903 | { 904 | "mention": "tomato sauce", 905 | "entity": "Tomato_sauce" 906 | }, 907 | { 908 | "mention": "fresh", 909 | "entity": "Fruit" 910 | } 911 | ] 912 | }, 913 | "4": { 914 | "turn_number": 4, 915 | "el_tool_annotations": [ 916 | { 917 | "mention": "italian", 918 | "entity": "Italian_cuisine" 919 | } 920 | ] 921 | }, 922 | "6": { 923 | "turn_number": 6, 924 | "el_tool_annotations": [ 925 | { 926 | "mention": "fettucini alfredo", 927 | "entity": "Fettuccine_Alfredo" 928 | } 929 | ] 930 | }, 931 | "8": { 932 | "turn_number": 8, 933 | "el_tool_annotations": [ 934 | { 935 | "mention": "pizza hut", 936 | "entity": "Pizza_Hut" 937 | }, 938 | { 939 | "mention": "pan pizza", 940 | "entity": "Pizza" 941 | }, 942 | { 943 | "mention": "pizza crust", 944 | "entity": "Pizza" 945 | }, 946 | { 947 | "mention": "crust", 948 | "entity": "Bread" 949 | } 950 | ] 951 | } 952 | } 953 | }, 954 | { 955 | "dialogue_id": "17067", 956 | "dataset_name": "wow", 957 | "turns": { 958 | "0": { 959 | "turn_number": 0, 960 | "el_tool_annotations": [ 961 | { 962 | "mention": "Channing Tatum", 963 | "entity": "Channing_Tatum" 964 | } 965 | ] 966 | }, 967 | "2": { 968 | "turn_number": 2, 969 | "el_tool_annotations": [] 970 | }, 971 | "4": { 972 | "turn_number": 4, 973 | "el_tool_annotations": [ 974 | { 975 | "mention": "21 Jump Street", 976 | "entity": "21_Jump_Street" 977 | }, 978 | { 979 | "mention": "but not", 980 | "entity": "For_but_not_with" 981 | }, 982 | { 983 | "mention": "Magic Mike", 984 | "entity": "Magic_Mike" 985 | } 986 | ] 987 | }, 988 | "6": { 989 | "turn_number": 6, 990 | "el_tool_annotations": [] 991 | } 992 | } 993 | }, 994 | { 995 | "dialogue_id": "4371", 996 | "dataset_name": "wow", 997 | "turns": { 998 | "0": { 999 | "turn_number": 0, 1000 | "el_tool_annotations": [ 1001 | { 1002 | "mention": "yellow", 1003 | "entity": "Yellow" 1004 | }, 1005 | { 1006 | "mention": "how about you", 1007 | "entity": "How_About_You?" 1008 | } 1009 | ] 1010 | }, 1011 | "2": { 1012 | "turn_number": 2, 1013 | "el_tool_annotations": [] 1014 | }, 1015 | "4": { 1016 | "turn_number": 4, 1017 | "el_tool_annotations": [ 1018 | { 1019 | "mention": "thank you", 1020 | "entity": "Thank_You_(Dido_song)" 1021 | } 1022 | ] 1023 | }, 1024 | "6": { 1025 | "turn_number": 6, 1026 | "el_tool_annotations": [] 1027 | }, 1028 | "8": { 1029 | "turn_number": 8, 1030 | "el_tool_annotations": [] 1031 | } 1032 | } 1033 | }, 1034 | { 1035 | "dialogue_id": "12407", 1036 | "dataset_name": "wow", 1037 | "turns": { 1038 | "1": { 1039 | "turn_number": 1, 1040 | "el_tool_annotations": [ 1041 | { 1042 | "mention": "Yes", 1043 | "entity": "Yes_(band)" 1044 | }, 1045 | { 1046 | "mention": "fish", 1047 | "entity": "Fish_(singer)" 1048 | } 1049 | ] 1050 | }, 1051 | "3": { 1052 | "turn_number": 3, 1053 | "el_tool_annotations": [] 1054 | }, 1055 | "5": { 1056 | "turn_number": 5, 1057 | "el_tool_annotations": [ 1058 | { 1059 | "mention": "Salmon", 1060 | "entity": "Salmon" 1061 | } 1062 | ] 1063 | }, 1064 | "7": { 1065 | "turn_number": 7, 1066 | "el_tool_annotations": [] 1067 | } 1068 | } 1069 | }, 1070 | { 1071 | "dialogue_id": "8214", 1072 | "dataset_name": "wow", 1073 | "turns": { 1074 | "0": { 1075 | "turn_number": 0, 1076 | "el_tool_annotations": [ 1077 | { 1078 | "mention": "Pearl Jam", 1079 | "entity": "Pearl_Jam" 1080 | } 1081 | ] 1082 | }, 1083 | "2": { 1084 | "turn_number": 2, 1085 | "el_tool_annotations": [ 1086 | { 1087 | "mention": "Kurt Cobain", 1088 | "entity": "Kurt_Cobain" 1089 | }, 1090 | { 1091 | "mention": "Dave Grohl", 1092 | "entity": "Dave_Grohl" 1093 | } 1094 | ] 1095 | }, 1096 | "4": { 1097 | "turn_number": 4, 1098 | "el_tool_annotations": [ 1099 | { 1100 | "mention": "I wish", 1101 | "entity": "I_Wish_(Stevie_Wonder_song)" 1102 | } 1103 | ] 1104 | }, 1105 | "6": { 1106 | "turn_number": 6, 1107 | "el_tool_annotations": [ 1108 | { 1109 | "mention": "Matt Cameron", 1110 | "entity": "Matt_Cameron" 1111 | }, 1112 | { 1113 | "mention": "him", 1114 | "entity": "HIM_(Finnish_band)" 1115 | }, 1116 | { 1117 | "mention": "Soundgarden", 1118 | "entity": "Soundgarden" 1119 | }, 1120 | { 1121 | "mention": "big", 1122 | "entity": "The_Notorious_B.I.G." 1123 | } 1124 | ] 1125 | }, 1126 | "8": { 1127 | "turn_number": 8, 1128 | "el_tool_annotations": [ 1129 | { 1130 | "mention": "Pearl Jam", 1131 | "entity": "Pearl_Jam" 1132 | }, 1133 | { 1134 | "mention": "music", 1135 | "entity": "The_Music" 1136 | } 1137 | ] 1138 | } 1139 | } 1140 | }, 1141 | { 1142 | "dialogue_id": "7051", 1143 | "dataset_name": "wow", 1144 | "turns": { 1145 | "0": { 1146 | "turn_number": 0, 1147 | "el_tool_annotations": [] 1148 | }, 1149 | "2": { 1150 | "turn_number": 2, 1151 | "el_tool_annotations": [ 1152 | { 1153 | "mention": "cw", 1154 | "entity": "The_CW" 1155 | }, 1156 | { 1157 | "mention": "adult swim", 1158 | "entity": "Adult_Swim" 1159 | }, 1160 | { 1161 | "mention": "tbs", 1162 | "entity": "TBS_(U.S._TV_channel)" 1163 | } 1164 | ] 1165 | }, 1166 | "4": { 1167 | "turn_number": 4, 1168 | "el_tool_annotations": [] 1169 | }, 1170 | "6": { 1171 | "turn_number": 6, 1172 | "el_tool_annotations": [] 1173 | }, 1174 | "8": { 1175 | "turn_number": 8, 1176 | "el_tool_annotations": [ 1177 | { 1178 | "mention": "stewie", 1179 | "entity": "Stewie_Griffin" 1180 | } 1181 | ] 1182 | } 1183 | } 1184 | }, 1185 | { 1186 | "dialogue_id": "1266", 1187 | "dataset_name": "wow", 1188 | "turns": { 1189 | "0": { 1190 | "turn_number": 0, 1191 | "el_tool_annotations": [ 1192 | { 1193 | "mention": "hamburgers", 1194 | "entity": "Hamburger" 1195 | } 1196 | ] 1197 | }, 1198 | "2": { 1199 | "turn_number": 2, 1200 | "el_tool_annotations": [] 1201 | }, 1202 | "4": { 1203 | "turn_number": 4, 1204 | "el_tool_annotations": [] 1205 | }, 1206 | "6": { 1207 | "turn_number": 6, 1208 | "el_tool_annotations": [ 1209 | { 1210 | "mention": "turkey", 1211 | "entity": "Turkey_(bird)" 1212 | }, 1213 | { 1214 | "mention": "turkey burger", 1215 | "entity": "Hamburger" 1216 | } 1217 | ] 1218 | }, 1219 | "8": { 1220 | "turn_number": 8, 1221 | "el_tool_annotations": [ 1222 | { 1223 | "mention": "thought", 1224 | "entity": "Thought" 1225 | } 1226 | ] 1227 | } 1228 | } 1229 | }, 1230 | { 1231 | "dialogue_id": "2040", 1232 | "dataset_name": "wow", 1233 | "turns": { 1234 | "0": { 1235 | "turn_number": 0, 1236 | "el_tool_annotations": [ 1237 | { 1238 | "mention": "Droopy", 1239 | "entity": "Droopy" 1240 | } 1241 | ] 1242 | }, 1243 | "2": { 1244 | "turn_number": 2, 1245 | "el_tool_annotations": [] 1246 | }, 1247 | "4": { 1248 | "turn_number": 4, 1249 | "el_tool_annotations": [ 1250 | { 1251 | "mention": "don't know much", 1252 | "entity": "Don't_Know_Much" 1253 | } 1254 | ] 1255 | }, 1256 | "6": { 1257 | "turn_number": 6, 1258 | "el_tool_annotations": [ 1259 | { 1260 | "mention": "when we", 1261 | "entity": "When_We_On" 1262 | } 1263 | ] 1264 | } 1265 | } 1266 | } 1267 | ] -------------------------------------------------------------------------------- /data/README.md: -------------------------------------------------------------------------------- 1 | ```sh 2 | . 3 | ├── ConEL_Concept_Named_Entity # Stratified samples 4 | │   ├── ConEL_CNE.json # Entity annotations from 25 dialogues from each dataset (i.e., MWOZ, QuAC, WoW, and TREC-CAST 2020). 5 | │   └── runs # Annotations by EL tools reported in our paper 6 | │   ├── ConEL_CNE-REL_history.json 7 | │   ├── ConEL_CNE-REL_turn.json 8 | │   ├── ConEL_CNE-Tagme_history.json 9 | │   ├── ConEL_CNE-Tagme_turn.json 10 | │   ├── ConEL_CNE-WAT_history.json 11 | │   └── ConEL_CNE-WAT_turn.json 12 | ├── ConEL_Personal_Entity # WoW with personal entities 13 | │   ├── ConEL_PE.json # 25 WoW dialogues which contains personal entities in each dialogue. 14 | │   └── runs # Annotations by EL tools reported in our paper 15 | │   ├── ConEL_PE-REL_history.json 16 | │   ├── ConEL_PE-REL_turn.json 17 | │   ├── ConEL_PE-Tagme_history.json 18 | │   ├── ConEL_PE-Tagme_turn.json 19 | │   ├── ConEL_PE-WAT_history.json 20 | │   └── ConEL_PE-WAT_turn.json 21 | └── README.md 22 | 23 | ``` -------------------------------------------------------------------------------- /mturk_interfaces/01_stage1_annotations/html/stage1.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 |
8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |

Instructions

16 | 17 |

Overview

18 | 19 | In this task, we give you a piece of dialogue between a user and a conversational assistant (e.g., Siri or the Google Assistant).
20 | Your task is to find the Wikipedia article that is mentioned in the final user utterance. 21 | 22 |

Guidelines

23 | 24 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 44 | 45 | 46 |

Example

47 |

48 | 49 | 50 | 51 | 52 | 53 |
54 | 55 |

Example 1

56 | 57 |
    58 |
  • See the example below. If you click the link to the Wikipedia article, you can find that the London in the given options represents London in the UK
    • 59 |
  • Based on the dialogue, it can be assumed that the user and system are talking about a train in the UK
    • 60 |
  • Thus, London should be chosen
    • 61 |
62 | 63 |
64 |

Step 1: Read this dialogue.

65 | 66 | 67 | 68 | 69 | 70 |
USER: I am looking for a train to University of Cambridge.
SYSTEM: Where do you plan to depart from?
USER: I am staying next to Waterloo Station in London.
71 |
72 | 73 |

Step 2: Which Wikipedia atricle does the phrase "London" refer to?


74 |
75 | 76 |
USER: I am staying next to Waterloo Station in London.

77 | London,_Ontario
78 | London
79 | London_Island
80 | None of the above
81 |
82 |
83 | 84 | 85 | 86 | 87 | 88 | 89 |
90 |

Example 2

91 | 92 | 93 |
    94 |
  • See the example below. If you click the link to the Wikipedia article, you can find that the The_Band represents the name of the rock group formed in Toronto.
    • 95 |
  • Based on the dialogue, you can infer that the user and system are talking about the band Grateful Dead not The_Band.
    • 96 |
  • Thus, None of the above should be chosen
    • 97 |
98 | 99 |
100 |

Step 1: Read this dialogue.

101 | 102 | 103 | 104 | 105 | 106 |
USER: Why were the Dead such an influential live band?
SYSTEM: The Grateful Dead formed during the era when bands such ...
USER: Why did the band break up?
107 |
108 | 109 |

Step 2: Which Wikipedia atricle does the phrase "band" refer to?


110 |
111 | 112 |
USER: Why did the band break up?

113 | The_Band
114 | None of the above
115 |
116 |
117 | 118 | 119 | 120 | 121 | 122 | 123 | 139 | 140 | 141 | 142 | 143 | 144 |

Dialogue

145 | 146 |
147 |

Step 1: Read this dialogue.

148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 |
USER: It's summertime and I know snakes will be more active.
SYSTEM: Snakes terrify me! I don't like their elongated and legless bodies. They freak me out!
... ...
USER: I always try to make sure in the spring that there isn't a food source for rodents in the summer. The website does have that recommendation.
SYSTEM: I will definitely keep that in mind. I live in a hot and humid location and I'm sure we have tons of snakes around here!
USER: The humidity here in SC has increased dramatically over the years and I'm seeing more Black King snakes
SYSTEM: I lived in Charleston, South Carolina for twenty years before moving here to Florida. I did see a lot of snakes there!
USER: I used to live in Tampa, Fla. and moved to Greenville, SC. Good luck keeping the snakes away! :)
160 |
161 | 162 | 163 |

Step 2: Which Wikipedia atricle does the phrase "Fla" refer to?


164 |
165 | 166 |
USER: I used to live in Tampa, Fla. and moved to Greenville, SC. Good luck keeping the snakes away! :)

167 | 168 | Florida
169 | J.Fla
170 | Miami
171 | Flå
172 | Tampa,_Florida
173 | Jacksonville,_Florida
174 | Pensacola,_Florida
175 | Palm_Beach,_Florida
176 | None of the above
177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 |
187 | -------------------------------------------------------------------------------- /mturk_interfaces/01_stage1_annotations/screenshots/stage1_example1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/informagi/conversational-entity-linking/d4d3d863060fa60f79c1c8c4012f85947ed6f33e/mturk_interfaces/01_stage1_annotations/screenshots/stage1_example1.png -------------------------------------------------------------------------------- /mturk_interfaces/01_stage1_annotations/screenshots/stage1_example2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/informagi/conversational-entity-linking/d4d3d863060fa60f79c1c8c4012f85947ed6f33e/mturk_interfaces/01_stage1_annotations/screenshots/stage1_example2.png -------------------------------------------------------------------------------- /mturk_interfaces/01_stage1_annotations/screenshots/stage1_overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/informagi/conversational-entity-linking/d4d3d863060fa60f79c1c8c4012f85947ed6f33e/mturk_interfaces/01_stage1_annotations/screenshots/stage1_overview.png -------------------------------------------------------------------------------- /mturk_interfaces/02_stage2_annotations/html/stage2.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 |
8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |

Instructions

16 | 17 |

Overview

18 | 19 | Imagine you are an AI agent (e.g., Siri or Google Now), having a dialogue with a person. You have access to Wikipedia articles (and some other information sources) to answer the person's questions. 20 | Select the Wikipedia articles that help you to find an answer to the person's question. 21 | 22 |

Guidelines

23 | 24 |
    25 |
  • Step 1: Read the dialogue history between the user and system.
    • 26 |
  • Step 2: We give you some phrases (all linked to Wikipedia articles) from the final user utterance. Select:
    • 27 |
      28 |
    • 2-1: Known entities, such as name of people, places, or organizations (e.g., "Sarah", "London", or "NASA"). 29 |
        30 |
      • Select known entities, even if they are abbreviations (e.g., "Fla."), acronyms (e.g., "SC"), or compound nouns (e.g., "Tempa, Florida").
        • 31 |
      32 |
    • 2-2: Phrases that help you (as a SYSTEM) to provide a good response to the user.
      • 33 |
34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 47 | 48 | 49 |

Example

50 |

51 | 52 | 53 | 54 | 55 | 56 |
57 | 58 |

Example 1

59 | 60 |
    61 |
  • Ignore the phrase that does NOT contain important information that helps you (as a SYSTEM) to provide a good response to the user. 62 |
      63 |
    • In the example below, "college" is not directly related to the user’s question.
      • 64 |
    65 |
  • Select the longest phrase "Anne Frank House", instead of "Anne Frank" or "House".
    • 66 |
67 | 68 |
69 |

Step 1: Read this dialogue.

70 | 71 | 72 | 73 | 74 | 75 |
USER: I would like to travel to Western Europe. Do you have any recommendations for places to visit?
SYSTEM: Have you ever been to Amsterdam?
USER: I have wanted to travel there since I was in college. What are the attractions of Amsterdam? I heard that the Anne Frank House is very attractive.
76 |
77 | 78 |

Step 2-2: Select the phrases that help you (as a SYSTEM) to provide a good response to the user.

79 |
80 | 81 |
USER: I have wanted to travel there since I was in college. What are the attractions of Amsterdam? I heard that the Anne Frank House is very attractive.

82 | college (College - Wikipedia)
83 | Amsterdam (Amsterdam - Wikipedia)
84 | Anne Frank (Anne Frank - Wikipedia)
85 | House (House - Wikipedia)
86 | Anne Frank House (Anne Frank House - Wikipedia)
87 | None of the above
88 | 89 |
90 |
91 | 92 | 93 | 94 | 95 | 96 |
97 | 98 |

Example 2

99 | 100 |
    101 |
  • See the example below. In this case phrase "Thank you" should NOT be chosen.
    • 102 |
  • This is because the Wikipedia article Thank You (Led Zeppelin song) - Wikipedia does not help you (as a SYSTEM) to provide a good response to the user.
    • 103 |
104 | 105 |
106 |

Step 1: Read this dialogue.

107 | 108 | 109 | 110 | 111 | 112 |
USER: I am looking for a restaurant in the west side of town
SYSTEM: I found saint johns chop house. The address is 21-24 Northampton Street and the phone number is 01223353110. Is there anything else I can do for you?
USER: No, you have been a great help. Thank you.
113 |
114 | 115 |

Step 2-2: Select the phrases that help you (as a SYSTEM) to provide a good response to the user.

116 |
117 | 118 |
USER: No, you have been a great help. Thank you.

119 | Thank you (Thank You (Led Zeppelin song) - Wikipedia)
120 | None of the above
121 | 122 |
123 |
124 | 125 | 126 | 127 | 128 | 129 | 145 | 146 | 147 | 148 | 149 |

Dialogue

150 | 151 |
152 |

Step 1: Read this dialogue.

153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 |
USER: I've caught the movie star bug! I was just an extra in a movie!
SYSTEM: That's great! I bet you'll have a lot of people referring to you as a background artist or cast member. Apparently on a TV set the term extra isn't used that often.
... ...
USER: Yeah, I think my next step is to be a "seat filler" at the Academy Awards.
SYSTEM: Oh, you mean a person who fills in an empty seat during an event? I bet that's how Trump gets people to go to his speeches!
USER: Yes that's exactly what I mean! I just need a NON Harvey Weinstein type person to discover me to be famous!
SYSTEM: Well, depending on the production the criteria for casting background actors is going to be different. I'm sure you'll find your niche role eventually!
USER: My parents say I shouldn't go to Hollywood. They want me to keep my day job as a Wal-Mart greeter, but I'm packing my car. California here I come!
165 |
166 | 167 |

Step 2-1: Select known entities, such as name of people, places, or organizations (e.g., "Sarah", "London", or "NASA")

168 |
169 | 170 |
USER: My parents say I shouldn't go to Hollywood. They want me to keep my day job as a Wal-Mart greeter, but I'm packing my car. California here I come!

171 | California (California - Wikipedia)
172 | Hollywood (Hollywood - Wikipedia)
173 | Wal-Mart (Walmart - Wikipedia)
174 | None of the above
175 | 176 |

Step 2-2: Select the phrases that help you (as a SYSTEM) to provide a good response to the user.

177 | California (California - Wikipedia)
178 | Hollywood (Hollywood - Wikipedia)
179 | Wal-Mart (Walmart - Wikipedia)
180 | None of the above
181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 |
190 | -------------------------------------------------------------------------------- /mturk_interfaces/02_stage2_annotations/screenshots/stage2_example1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/informagi/conversational-entity-linking/d4d3d863060fa60f79c1c8c4012f85947ed6f33e/mturk_interfaces/02_stage2_annotations/screenshots/stage2_example1.png -------------------------------------------------------------------------------- /mturk_interfaces/02_stage2_annotations/screenshots/stage2_example2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/informagi/conversational-entity-linking/d4d3d863060fa60f79c1c8c4012f85947ed6f33e/mturk_interfaces/02_stage2_annotations/screenshots/stage2_example2.png -------------------------------------------------------------------------------- /mturk_interfaces/02_stage2_annotations/screenshots/stage2_overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/informagi/conversational-entity-linking/d4d3d863060fa60f79c1c8c4012f85947ed6f33e/mturk_interfaces/02_stage2_annotations/screenshots/stage2_overview.png -------------------------------------------------------------------------------- /mturk_interfaces/03_personal_entity_annotations/html/personal_entity_annotations.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 |
8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |

Instructions

16 | 17 |

Overview

18 | 19 | In this task, we give you a piece of dialogue between a user and a conversational assistant (e.g., Siri or Google Assistant).
20 | Your task is to select the text span from the dialogue history that the given phrase (e.g., "my guitar") refers to. 21 | 22 | 23 | 24 |

Guidelines

25 | 26 |
    27 |
  • Step 1: Read the dialogue history between the user and system.
    • 28 |
  • Step 2: We give you a phrase (e.g., "my guitar"). Select the text span that from the dialouge history the given phrase refers to.
    • 29 |
      30 |
    • Select the option that you think is best.
      • 31 |
32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 44 | 45 | 46 |

Example

47 |

48 | 49 | 50 | 51 | 52 | 53 |
54 | 55 |

Example 1

56 | 57 |
    58 |
  • In this example, "my guitar" refers to "Gibson Les Paul". 59 |
  • Select the text span "Gibson Les Paul".
    • 60 |
61 | 62 |
63 |

Step 1: Read this dialogue.

64 | 65 | 66 | 67 | 68 | 69 |
USER: I am using Gibson Les Paul, but I am thinking about buying a new guitar. Do you have any recommendations?
SYSTEM: What are your preferences?
USER: Do you have something similar to my guitar?
70 |
71 | 72 |

Step 2: Select the text span that the phrase "my guitar" refers to.


73 | 74 | Gibson
75 | Les Paul
76 | Gibson Les Paul
77 | No information in the dialogue
78 | None of the above
79 | 80 | 81 |
82 |
83 | 84 | 85 | 86 | 87 | 88 | 89 |
90 | 91 |

Example 2

92 | 93 |
    94 |
  • In this example, there is no specific information about the phrase "my Nike" in the dialogue.
    • 95 |
  • Select the option "No information in the dialogue".
    • 96 |
97 | 98 |
99 |

Step 1: Read this dialogue.

100 | 101 | 102 | 103 | 104 | 105 |
USER: I'd like to buy new running shoes.
SYSTEM: What kind of shoes do you prefer?
USER: Do you have something similar to my Nike?
106 |
107 | 108 |

Step 2: Select the text span that the phrase "my Nike" refers to.


109 | 110 | running shoes
111 | something
112 | No information in the dialogue
113 | None of the above
114 | 115 |
116 |
117 | 118 | 119 | 120 | 121 | 122 |
123 | 124 |

Example 3

125 | 126 |
    127 |
  • The phrase "my reservation time" refers to 20:00 on Friday, but there are no options about it.
    • 128 |
  • In this case, select "None of the above".
    • 129 |
130 | 131 |
132 |

Step 1: Read this dialogue.

133 | 134 | 135 | 136 | 137 | 138 |
USER: Can I get reservations for 7 at 20:00 on Friday?
SYSTEM: No problem. I have you booked on Friday at 20:00 for 7. Can I help you with anything else?
USER: Yes. I would like a taxi to take me to the restaurant by my reservation time.
139 |
140 | 141 |

Step 2: Select the text span that the phrase "my reservation time" refers to.


142 | 143 | reservations
144 | No information in the dialogue
145 | None of the above
146 | 147 |
148 |
149 | 150 | 151 | 152 | 153 | 154 | 170 | 171 | 172 | 173 | 174 |

Dialogue

175 | 176 |
177 |

Step 1: Read this dialogue.

178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 |
USER: Have you volunteer in any humane service before?
SYSTEM: Yes, I did volunteer for a brief time, but I developed an allergy to cats. So I had to quit.
USER: oh thats nice, I worked at houston food bank as a volunteer
SYSTEM: The cat food always smelled really strongly. I think smelling so many food smells suppressed my appetite when I volunteered.
USER: yes thats true, since you will work in a warehouse, is something worth doing
SYSTEM: I do like helping the animals though. Perhaps I will try to foster dogs instead. They eat a lot more because they have a bigger appetite.
USER: yes dogs eat a lot, I love working with animals, am an animal scientist and is part of my job
189 |
190 | 191 |

Step 2: Select the text span that the phrase "my job" refers to.


192 | 193 | 194 | volunteer
195 | service
196 | houston food bank
197 | a volunteer
198 | houston
199 | dogs
200 | animals
201 | an animal scientist
202 | No information in the dialogue
203 | None of the above
204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | -------------------------------------------------------------------------------- /mturk_interfaces/03_personal_entity_annotations/screenshots/personal_entity_annotations_examples.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/informagi/conversational-entity-linking/d4d3d863060fa60f79c1c8c4012f85947ed6f33e/mturk_interfaces/03_personal_entity_annotations/screenshots/personal_entity_annotations_examples.png -------------------------------------------------------------------------------- /mturk_interfaces/03_personal_entity_annotations/screenshots/personal_entity_annotations_overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/informagi/conversational-entity-linking/d4d3d863060fa60f79c1c8c4012f85947ed6f33e/mturk_interfaces/03_personal_entity_annotations/screenshots/personal_entity_annotations_overview.png --------------------------------------------------------------------------------