├── .gitignore
├── error_analysis
    ├── rc_exp_1_stage_3_5_e60_drop
    │   ├── eval_reports_valid_set
    │   │   ├── performance_overall.csv
    │   │   ├── performance_category.csv
    │   │   ├── ea_date_full_86.csv
    │   │   ├── ea_date_sample_86.csv
    │   │   ├── ea_number_sample_100.csv
    │   │   ├── ea_sapn_sample_100.csv
    │   │   └── ea_spans_sample_100.csv
    │   └── submission_drop_leaderboard
    │   │   └── drop_submission_pickle_1122083041.P
    └── val_exp_2_stage_2_4_e60_drop
    │   └── eval_reports_valid_set
    │       ├── performance_overall.csv
    │       ├── performance_category.csv
    │       ├── ea_number_sample_100.csv
    │       ├── ea_date_full_90.csv
    │       ├── ea_date_sample_90.csv
    │       ├── ea_sapn_sample_100.csv
    │       └── ea_spans_sample_100.csv
├── data
    └── README.md
├── models
    └── README.md
├── LICENSE
├── README.md
└── tfrec
    ├── tfrec_synthetic_data.ipynb
    ├── tfrec_squad.ipynb
    ├── squad_parser.ipynb
    ├── tfrec_drop.ipynb
    ├── tfrec_drop_cat.ipynb
    └── tfrec_drop_num.ipynb


/.gitignore:
--------------------------------------------------------------------------------
1 | .DS_Store
2 | .ipynb_checkpoints
3 | 


--------------------------------------------------------------------------------
/error_analysis/rc_exp_1_stage_3_5_e60_drop/eval_reports_valid_set/performance_overall.csv:
--------------------------------------------------------------------------------
1 | ,exact_match,f1
2 | 0,0.6687290268456376,0.7031449244966449
3 | 


--------------------------------------------------------------------------------
/error_analysis/val_exp_2_stage_2_4_e60_drop/eval_reports_valid_set/performance_overall.csv:
--------------------------------------------------------------------------------
1 | ,exact_match,f1
2 | 0,0.6604446308724832,0.6959783976510072
3 | 


--------------------------------------------------------------------------------
/data/README.md:
--------------------------------------------------------------------------------
1 | # NT5?! Training T5 to Perform Numerical Reasoning
2 | 
3 | Please access NT5's training datasets using the following link: [Google Drive](https://drive.google.com/drive/folders/1x510WLbCDVU9pVuMII6nFsIxv62jnk9O?usp=sharing)
4 | 
5 | 


--------------------------------------------------------------------------------
/error_analysis/rc_exp_1_stage_3_5_e60_drop/submission_drop_leaderboard/drop_submission_pickle_1122083041.P:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/lesterpjy/numeric-t5/HEAD/error_analysis/rc_exp_1_stage_3_5_e60_drop/submission_drop_leaderboard/drop_submission_pickle_1122083041.P


--------------------------------------------------------------------------------
/error_analysis/rc_exp_1_stage_3_5_e60_drop/eval_reports_valid_set/performance_category.csv:
--------------------------------------------------------------------------------
1 | ,number,date,span,spans
2 | exact_match,0.7034188034188035,0.45222929936305734,0.6674544226873734,0.37742504409171074
3 | f1,0.7039162393162393,0.5384713375796178,0.7235280216070231,0.6343033509700174
4 | 


--------------------------------------------------------------------------------
/error_analysis/val_exp_2_stage_2_4_e60_drop/eval_reports_valid_set/performance_category.csv:
--------------------------------------------------------------------------------
1 | ,number,date,span,spans
2 | exact_match,0.6977777777777778,0.4267515923566879,0.6600270087778528,0.3421516754850088
3 | f1,0.6982581196581197,0.5122929936305733,0.7147400405131675,0.6253086419753083
4 | 


--------------------------------------------------------------------------------
/models/README.md:
--------------------------------------------------------------------------------
1 | # NT5?! Training T5 to Perform Numerical Reasoning
2 | 
3 | Two fully trained NT5 models (in h5 format) with their evaluation reports can be downloaded using the link, [Google Drive](https://drive.google.com/drive/folders/18K_HakMgY8e1R-OBLjrfREsacbf9FNTJ?usp=sharing). The two models included are our best and second best performing models: **RC Experiment 1** and **Validation Experiment 2**.   
4 | 
5 | <p align="center"><img src="https://www.dropbox.com/s/ujq8dc229nl79bh/schedule.jpeg?raw=1" width="800" /></p>
6 | 
7 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2021 Peng-Jian Yang
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/error_analysis/val_exp_2_stage_2_4_e60_drop/eval_reports_valid_set/ea_number_sample_100.csv:
--------------------------------------------------------------------------------
  1 | ,query_id,prediction,answer,validated_answers,f1,exact_match
  2 | 195,17169f44-cb53-4809-9dc7-19872cc70480,2,1,1<sv>1,0.0,0.0
  3 | 249,d3721a75-8d89-4afe-aa70-b6579b2ab3f0,85,7,7<sv>7,0.0,0.0
  4 | 1315,a8c4a16a-52ff-4def-a175-c723dcbd115e,36,2392,2392<sv>2392,0.0,0.0
  5 | 1177,cf084a28-2e17-4b61-a65b-95fb1afff749,30000,171728,9728<sv>171728<sv>171728,0.0,0.0
  6 | 316,20fedde5-fb10-44d4-8e22-472c062802cd,3,2,1<sv>1,0.0,0.0
  7 | 327,b2a864d6-52a7-44f7-bcdd-45b12daef5a2,4,3,3<sv>3,0.0,0.0
  8 | 396,ad19857f-cd76-4d01-ba29-1a589cfee053,2,1,4<sv>29,0.0,0.0
  9 | 1088,c69e1211-2b3b-4dd1-aaea-b3a867890df1,12.8,30.5,30.5<sv>30.5,0.0,0.0
 10 | 514,be8224f9-3a76-4929-8282-de34692901a9,5000,1400,1400<sv>1400<sv>1400,0.0,0.0
 11 | 865,9365453d-96f3-44e8-b2d8-8ae40d1c059e,2,1,1<sv>1,0.0,0.0
 12 | 1119,5d7817e1-605f-4993-8e9d-00f1d8963351,33,72,78<sv>78,0.0,0.0
 13 | 1100,c2770bd8-24e3-453d-ac39-38df971dcdeb,4426228,4426828,4426828<sv>4426828,0.0,0.0
 14 | 626,a8a67087-38a5-48e8-a4cd-66363c897deb,.4,0.7,0.7<sv>1.23,0.0,0.0
 15 | 373,a588aaaf-070a-4704-8a7d-11e03cc3ada1,3,2,2<sv>2,0.0,0.0
 16 | 1132,84414060-ac52-4b57-ac5f-8eff6e89e36d,34,7,32<sv>7,0.0,0.0
 17 | 1623,81d7244f-562a-4ff2-9e0f-5daabf9c2aa7,64,61,61<sv>61,0.0,0.0
 18 | 398,c9d15396-8d34-465b-8bba-502db7e63aea,134000,84000,42000<sv>42000,0.0,0.0
 19 | 1523,6e7ea881-daf6-4a8d-b21a-a85a16fb18e5,16.3,8,8<sv>8,0.0,0.0
 20 | 279,c584f463-800e-491e-9d80-b0c3a43059ca,4,12,12<sv>12,0.0,0.0
 21 | 86,b65417dd-5a7e-4605-aa85-780add03990e,23,53,53<sv>53<sv>53,0.0,0.0
 22 | 1635,9bb4fa02-f946-4586-8441-f333777a9386,1541290,19,19<sv>19,0.0,0.0
 23 | 819,f19dd4b6-3063-48c1-adb8-739df0bdba4f,6,7,7<sv>7,0.0,0.0
 24 | 1665,4029aace-b57a-4e15-9b00-d865bd94a0ec,20986,40245,40245<sv>40245,0.0,0.0
 25 | 1417,59fbae57-8f09-4637-8383-64659e27a801,96.6,97,97<sv>97,0.0,0.0
 26 | 135,9eb68f5c-0c59-4850-9f2d-e6bbb80cbfa0,1,2,2<sv>2,0.0,0.0
 27 | 181,c3674bd0-9069-4cdc-917e-717e62e34121,37,57,57<sv>56,0.0,0.0
 28 | 190,594810b2-fa72-46d1-a5e0-8ccdf097df8d,10,17,17<sv>17,0.0,0.0
 29 | 1370,8e073f31-04c3-4913-b873-0f32bee619db,43,76,76<sv>76,0.0,0.0
 30 | 503,ac8d3dbe-d2b9-4694-9985-1ac3dd118461,3,4,4<sv>4<sv>4,0.0,0.0
 31 | 547,d6eb8c17-da01-4bf1-bfcb-98a87c8d7bb2,5,4,4<sv>4,0.0,0.0
 32 | 1426,8e993148-e22d-4cb4-99e5-3351a79ba2b1,2,7,6<sv>7,0.0,0.0
 33 | 441,0c958c14-eb90-4734-9c56-c399a81f650c,3,2,2,0.0,0.0
 34 | 246,d3ae6931-3ff9-4266-8d32-bd9e0a303df9,85,7,7<sv>7,0.0,0.0
 35 | 440,b6d462f0-70ce-4f0f-98d8-65ca5767e583,1000,1300,1300,0.0,0.0
 36 | 1025,5c6d6980-d6c9-4265-b53a-2243cc4bd7ce,4,3,2<sv>3<sv>3<sv>3,0.0,0.0
 37 | 792,77a7bc22-eebc-478b-bb3b-32d7ae8d94c7,14,9,9<sv>9,0.0,0.0
 38 | 1345,cea24564-bfbe-4723-8089-6d11635e013a,119,198,198<sv>198,0.0,0.0
 39 | 37,99e4335f-8309-4524-abc3-9de16d46b0bb,1986,13986,12993<sv>13986,0.0,0.0
 40 | 1626,5938dd6c-bb84-4a63-9b98-f347568f1ff4,14075,6,6<sv>6,0.0,0.0
 41 | 555,b4230db9-a914-4172-9489-c0f6ea85d583,4,5,5<sv>5,0.0,0.0
 42 | 867,c1de63d9-180b-4ce0-afd1-432a90686e91,21,0,0<sv>0,0.0,0.0
 43 | 4,f38d7fa8-910c-44fd-b6ee-026fa5d585fd,3,2,2<sv>2,0.0,0.0
 44 | 770,d3e11015-0ff1-4496-82ff-34fd762d300f,21,376,376<sv>473,0.0,0.0
 45 | 183,87ca0aad-2581-4d09-80a4-e383f6edf1e7,15,1,1<sv>1,0.0,0.0
 46 | 741,0757df6a-6711-4ba5-8475-cf729b214fa5,17,58,413<sv>58,0.0,0.0
 47 | 658,e9ae0bd8-9a5d-4809-8ce8-ace215b971c2,5,3,4<sv>4<sv>4,0.0,0.0
 48 | 922,1dafb9b8-7519-4a87-981f-2a8afb52fc52,3,14,14<sv>17,0.0,0.0
 49 | 399,eff3e74a-3804-4aaf-8d34-82e5aeba6923,101000,42000,42000<sv>42000<sv>42000<sv>46000<sv>42000,0.0,0.0
 50 | 1144,5338c4a1-6366-4a41-9e7d-f6bc61038bb9,52,8,10<sv>8,0.0,0.0
 51 | 133,1346c438-4a3c-4d4f-85b2-cc60d8307160,5,9,9<sv>3<sv>9,0.0,0.0
 52 | 136,3c3706dc-9026-4cbd-8250-cdc61ed090f8,1,2,2<sv>2<sv>2<sv>2<sv>2,0.0,0.0
 53 | 1442,421d4353-efd6-4551-a065-f33149bf2751,5,6,6<sv>6,0.0,0.0
 54 | 880,685689ca-a548-47e3-bd56-ad13d8251b4c,2,3,3<sv>3<sv>3,0.0,0.0
 55 | 605,3961617c-675a-4c18-8209-5752c65b1e84,360,1060,1060<sv>1060<sv>1060,0.0,0.0
 56 | 1678,cee777be-77b8-4e26-9430-4989329179ac,2,3,3<sv>3,0.0,0.0
 57 | 41,59745eca-d6df-481b-9806-f86a3becee38,23078,8572,8572<sv>8572,0.0,0.0
 58 | 1508,1f6b2aa8-fd2b-4de2-99ad-6c5fdc4cd267,10,.5,1.5<sv>.5,0.0,0.0
 59 | 1162,58706586-1d76-41fe-b39d-7de89f80f8ce,3,2,2<sv>2,0.0,0.0
 60 | 1737,934eeaca-5d9d-4059-a962-51404f26e59a,3,2,2<sv>2,0.0,0.0
 61 | 791,620b97c3-ce1c-429f-ac08-cc0eb2ef93bf,112,126,126<sv>126,0.0,0.0
 62 | 362,d6ee5db6-76bb-431d-b610-4a6bd4cf318b,13000,28700,28700<sv>28700<sv>28700,0.0,0.0
 63 | 1444,25293bf7-6e8c-4362-8a37-1078142e5be1,71,55,55<sv>55,0.0,0.0
 64 | 1668,c270470f-54e7-406e-8aa8-c3c32424054a,7.98,7.99,7.99<sv>7.99,0.0,0.0
 65 | 1027,451fa52b-2739-4440-81a1-ce4cf1438ec9,14,1,1<sv>1<sv>1<sv>1<sv>1<sv>1,0.0,0.0
 66 | 1290,89d74a05-05c6-41c9-aaf1-4b6205a185cf,144811,25179,25179<sv>25179,0.0,0.0
 67 | 651,b5e978f9-b487-42e3-b0d2-f3f8f6f16cb7,44,4,4<sv>4<sv>4<sv>4<sv>4,0.0,0.0
 68 | 982,5ba22fe2-23cc-43f2-8bcd-bf3d8481cb17,36,26,59<sv>59,0.0,0.0
 69 | 171,2b920398-fa1e-400e-b2a7-56450e4d72bb,4,2,2<sv>2,0.0,0.0
 70 | 894,43d21cab-10a3-4cb5-8a2e-db6786914b03,6,5,5<sv>5,0.0,0.0
 71 | 1757,eb92ceb7-af99-4c2c-8ad6-91ba990d107e,23,9,9<sv>9,0.0,0.0
 72 | 1199,43ed87a0-d245-4b97-9783-1de7e1afbc7f,71834,81834,81834<sv>22250,0.0,0.0
 73 | 1546,9395771b-b86a-49ef-9dcd-83014a552977,127,827,827<sv>827,0.0,0.0
 74 | 1040,88633537-4589-4d6f-bd40-cbf09e57facb,2,16,16<sv>16<sv>16<sv>16<sv>16<sv>16,0.0,0.0
 75 | 442,72f13f2b-cb28-49c8-92f5-cd8a53968acb,39,29,29,0.0,0.0
 76 | 296,35094e25-c518-4e85-acca-456b83a88fad,10,11,11<sv>8<sv>11,0.0,0.0
 77 | 583,8ba4aa76-7aa9-4611-992a-7281e4813b12,80,75,75<sv>13<sv>75,0.0,0.0
 78 | 384,20e30994-6435-48e6-ad0e-8fe3abf415da,2,18000,many<sv>many<sv>20000<sv>18000,0.0,0.0
 79 | 1367,3b0f9180-c60c-4a19-acbb-c7c9b8d80cd8,43,76,76<sv>76,0.0,0.0
 80 | 988,a4e1c06a-f97a-48bf-aeb6-350b0b2fdb47,18,10,10<sv>10,0.0,0.0
 81 | 1563,f03d9c4f-518b-45d4-84d8-7482cf1f9484,.0,1,1.0<sv>1,0.0,0.0
 82 | 795,c8c3ea22-eefd-49e9-abb2-2a24e9c530c3,38.66,42,42<sv>42,0.0,0.0
 83 | 1299,b351b368-0cb6-4998-8e5d-0e9e0f127b1a,3.18,0.08,.08<sv>.08,0.0,0.0
 84 | 369,d57087ac-ab66-4d18-8f96-ee16db240696,86,3,3<sv>2,0.0,0.0
 85 | 1050,88707d47-9392-495d-8dcc-ef44c6609715,14,0,0,0.0,0.0
 86 | 1389,97322694-337b-413e-8fb4-e0b554214c62,89996,79996,79996<sv>79996,0.0,0.0
 87 | 1260,391e850b-dc79-4d7a-b4e9-180ca5a31c05,168,178,178<sv>178,0.0,0.0
 88 | 686,d1553e2a-f5d7-456a-86cb-a44c3bc10286,1,2,2<sv>40,0.0,0.0
 89 | 1262,2a51e061-a2a7-4b66-bd63-34b83c054947,1,10,10<sv>10,0.0,0.0
 90 | 319,8d1d1067-668e-47c9-9c68-4047b59b178b,6,9,3<sv>11,0.0,0.0
 91 | 1259,647e3a27-d46e-42f8-a82a-61a95e24e1ab,74,53,53<sv>53,0.0,0.0
 92 | 713,0e641c63-ee26-406b-a907-44447d254cb6,3,2,2<sv>21,0.0,0.0
 93 | 1059,6ec6ec48-d9b7-4581-995a-da8797475b82,2,1,1<sv>1,0.0,0.0
 94 | 271,a3192959-ef01-425d-8dfe-84678d68e362,15,10,10<sv>10,0.0,0.0
 95 | 124,7016d115-81df-4e47-b901-ba0d7af1b807,99593,89683,89593<sv>103891,0.0,0.0
 96 | 965,b0415f9b-cde5-4444-99a4-194e1dc0a95d,5,2,3<sv>3,0.0,0.0
 97 | 142,2e443bd5-af92-4b82-9a7b-e4f88af684cb,24296,25176,25176<sv>25176,0.0,0.0
 98 | 154,a4437091-36da-45ee-8623-87e106a0fa0a,56,28,38<sv>29<sv>38,0.0,0.0
 99 | 161,f466f6ff-52c1-41d2-aabf-45adc57527fa,127,1526,1526<sv>1526,0.0,0.0
100 | 1690,42c8925e-7e0a-41fc-a564-f98344c609b2,.8115,0.0752,.0539<sv>0.0752,0.0,0.0
101 | 1384,2480d9f5-3b28-4ab3-b3d2-44cd00e3288d,58.3,48.3,48.3<sv>48.3,0.0,0.0
102 | 


--------------------------------------------------------------------------------
/error_analysis/rc_exp_1_stage_3_5_e60_drop/eval_reports_valid_set/ea_date_full_86.csv:
--------------------------------------------------------------------------------
 1 | ,query_id,prediction,answer,validated_answers,f1,exact_match
 2 | 0,4ef57f34-8c7c-4500-bc1f-2d2512e35523,1944,1943,1943<sv>1943<sv>1943<sv>1943,0.0,0.0
 3 | 1,0903294b-08fc-4061-937a-44890a55ca78,1944,1943,1943<sv>1943<sv>1943,0.0,0.0
 4 | 2,7c72066b-f148-4554-8b81-4d74fbe61101,1944,1943,1942<sv>1943,0.0,0.0
 5 | 3,45c95f84-92e4-4d83-b426-894687613da0,1683,1698,1698<sv>1698,0.0,0.0
 6 | 4,98f0f04a-7dbc-4717-bf97-5c272321d930,1913,1923,1923<sv>1923,0.0,0.0
 7 | 5,a76d9d78-2e31-49d9-9562-2abc3f758943,2002,2000,2000<sv>2000<sv>2000,0.0,0.0
 8 | 6,9d5b3589-d92c-46d8-8a22-0412f7d2d306,23 April 1734,1734,1734<sv>1734<sv>1734<sv>1734<sv>1734,0.5,0.0
 9 | 7,381ef468-9cc8-4b70-98b2-eab5183952b3,May 1751,April 1752,May 1754<sv>1752<sv>May 1754,0.0,0.0
10 | 8,32fcdc18-7118-40b5-a22e-13e4ae90bc20,December 1769,1769,1767<sv>1769,0.67,0.0
11 | 9,8fd1e5f3-e0dc-4996-aeb4-63f90f76e771,13 August 1563,August 1563,August<sv>August 1563,0.8,0.0
12 | 10,cca408e3-6075-430b-9e58-b820bb6aca06,13 August 1563,August 1563,August 1563<sv>August 1563<sv>August 1563,0.8,0.0
13 | 11,b44a05fc-4696-402f-a313-fb5236ac0ab6,30 May 1563,1 September 1563,August 1563<sv>4 September,0.4,0.0
14 | 12,8cbc1e9a-7f0c-4a32-8089-658398a9a1f0,1442,1438,1438<sv>1438,0.0,0.0
15 | 13,59b2070a-4027-49d5-b797-2ec7befcf2c4,1440,1452,1452<sv>1452,0.0,0.0
16 | 14,9fdfd076-2163-448e-b67d-1ad392ebfc5d,1440,1452,1452<sv>1452<sv>1552,0.0,0.0
17 | 15,79bacdf4-b669-4e2a-9be3-e73ab82a9467,1980,1955,1955<sv>1955,0.0,0.0
18 | 16,2f2a8ca0-199f-433a-a64c-d65b03c7ddd5,1980,1955,1955<sv>1955,0.0,0.0
19 | 17,e7d28a55-3aab-4313-9b65-ca20c329e166,In retaliation,1277,1277<sv>1277,0.0,0.0
20 | 18,9b6d8136-2535-448c-97b2-708ebe73b75c,1276,1277,1277<sv>1277,0.0,0.0
21 | 19,167c68ad-6ecd-4915-a795-f5f91441c413,1990,2010,2010<sv>2010,0.0,0.0
22 | 20,6c6f5cbe-c01c-46f0-aad5-df16aa478aa5,1990,1981,1981<sv>1981<sv>1981<sv>1981,0.0,0.0
23 | 21,45be16ff-c43e-412b-a6f2-6439e38a12c8,1583,1593,1593<sv>1593,0.0,0.0
24 | 22,d7c261cc-b929-4a1c-81b8-55a4d5f6fdbb,1553,1525,1525<sv>1525<sv>1525,0.0,0.0
25 | 23,ff20ab78-6d2c-4238-919e-34c40b4c9194,10 September,10 October 1556,29 September 1556<sv>October 10,0.5,0.0
26 | 24,dc5a38ba-47c4-4ff3-882f-0854b724928e,1418,1488,1488<sv>1488,0.0,0.0
27 | 25,c01c2890-80fc-4dd3-9073-533f7d2de8a8,27 April 100,30 April,27 April<sv>27,0.8,0.0
28 | 26,686e10d5-95bd-4b50-b2a8-72dfe6439f2f,2009,2007,2007<sv>2007,0.0,0.0
29 | 27,d2bb1b96-b931-4e1a-ab99-092d3bd0af52,February 29,1897,1896<sv>1897,0.0,0.0
30 | 28,e444e95d-e3e0-47b0-a5a8-38f99430f3cc,May,June,June 1272<sv>June,0.0,0.0
31 | 29,d3af4f76-8f9e-4199-84da-6956b4443279,May,June 1272,June<sv>June 1272,0.0,0.0
32 | 30,fd2ead2e-e669-409a-9dbf-700c838d8c15,1683,1698,1698<sv>1698,0.0,0.0
33 | 31,6629ae4d-99a4-4786-a8e8-4c0d723dc231,1683,1698,1698<sv>1698,0.0,0.0
34 | 32,2d60e681-7ca9-41a2-956b-1233fde15719,2003,2002,2002<sv>2002,0.0,0.0
35 | 33,f47e8521-c8ad-42f8-b164-02c35afff6c5,33 August 2000,31 April 2000,2000<sv>2000,0.5,0.0
36 | 34,61de672b-6d8d-4b3e-8394-1b73c8fcbc4d,April 1606,April 1605,April 1605<sv>3 March 1606,0.4,0.0
37 | 35,db540314-ad8e-4e87-bed8-d57ae991d26a,April 1606,April 1605,April 1605<sv>1605,0.0,0.0
38 | 36,72407857-72ac-480e-bc2d-0e4ce3781f10,1854,1852,1852<sv>1852,0.0,0.0
39 | 37,1aed8cf7-96a2-4c0d-af61-ff771944db22,The Siege of Nice,1542,Polin succeeding Antoine de Rincon<sv>Polin succeeding Antoine de Rincon,0.0,0.0
40 | 38,af5b5adf-ed88-45a3-bd91-902078145db0,1464,1384,1384<sv>1384,0.0,0.0
41 | 39,8593bfa1-eb03-489e-aa64-1364739265b7,1236,1234,1234<sv>1234,0.0,0.0
42 | 40,acfd906d-5098-4038-a55c-2be27af298e6,6 December 1283,3 December 1283,3 December 1283,0.67,0.0
43 | 41,c352733d-975e-4dc7-9f28-695dcc6d4f31,6 February 1284,5 February 1284,5 February 1284<sv>5 February 1284,0.67,0.0
44 | 42,1173ca51-f21e-4d5a-9092-93caf53eedd4,9 December 1283,3 December 1283,3 December 1283<sv>3 December 1283<sv>3 December 1283,0.67,0.0
45 | 43,6806f0b2-3baa-4919-ac39-b47c407bdbb5,26 January 1285,3 December 1283,3 December 1283<sv>3 December 1283,0.0,0.0
46 | 44,fe6f6f4e-7654-4574-83e7-d32f81c3af13,26 January 1285,3 December 1283,1283<sv>3 December 1283,0.0,0.0
47 | 45,e50b593b-a77a-4b67-8833-2820eb6fc314,1284,1285,that same year<sv>1285,0.0,0.0
48 | 46,1748ad77-26e5-4a8b-9881-40b7151d99e0,1982,1983,1981<sv>1981<sv>1981,0.0,0.0
49 | 47,6cdb01b3-76f4-4d14-8911-0badd33ac25b,March 1742,February 1743,February 1743<sv>February 1743,0.0,0.0
50 | 48,1b9b6544-d73a-4150-a1b6-68a2ee20b8eb,1920,1919,1919<sv>1919,0.0,0.0
51 | 49,4f47dd36-456b-4991-b2b3-3b3208d176b2,4 October 1910,5 October 1910,5 October 1910<sv>5 October 1910,0.67,0.0
52 | 50,247acee1-f214-46d2-a33a-f55d2ff49b32,1067,1061,1061<sv>1068,0.0,0.0
53 | 51,c323389b-e07c-4235-9ce7-3c2e955397b0,24 April 1976,3 February,3 February<sv>3 February,0.0,0.0
54 | 52,72b358b9-8cae-45a2-9b5c-597ae8ad56bf,1614,1615,1615<sv>1615,0.0,0.0
55 | 53,b113d114-d358-464e-84ab-866901f1c07f,1976,2009,2009<sv>2009,0.0,0.0
56 | 54,1dd90b41-df53-40fb-9528-847a0f9474b7,1890,May 2016,1899<sv>May 2016<sv>May 2016,0.0,0.0
57 | 55,b13d08ae-53b0-47ce-ab02-954db5fd58b0,1350,1367,1367<sv>1367<sv>1367<sv>1367<sv>1367,0.0,0.0
58 | 56,89eb3b39-409c-48ee-8856-c16c1f89cc40,31 August 1782,31 August 1781,31 August 1781<sv>31 August 1781,0.67,0.0
59 | 57,46142269-3156-44c8-95f2-5df25abd420c,1990,2004,2004<sv>2004,0.0,0.0
60 | 58,f79aa0fb-2c48-4e9d-ae8a-985a28ebf63b,2004,2010,2010<sv>2010,0.0,0.0
61 | 59,cee9867a-2ed2-40fc-a2f9-aa43117ef764,1908,1904,1904<sv>1904,0.0,0.0
62 | 60,a26a97a8-8ca9-4c2c-b6e4-490d37826f09,1905 <ss> 1906 <ss> 1907,1904,1904<sv>1904,0.0,0.0
63 | 61,c035a076-35c0-440c-b39d-4951027a3cec,1906,1904,1904<sv>1904,0.0,0.0
64 | 62,75afb1a1-c59b-4b60-a7d2-2346f9f2b68a,2011,1990,1990<sv>1990,0.0,0.0
65 | 63,db1dd1a9-a580-4c43-8f12-19da771715c3,2016,2015,2015<sv>2015,0.0,0.0
66 | 64,abef9d7e-48fc-4f4a-a842-707e7be221f6,2004,12000,2000<sv>12000,0.0,0.0
67 | 65,4f02227d-6f5c-4501-8cdc-54a2fab27452,2017,2016,2016<sv>2016,0.0,0.0
68 | 66,6ce9b33b-b86e-4390-8d9d-c2e6e7b353fe,1996,1982,1996-97<sv>1982,0.67,0.0
69 | 67,fa4a1714-eac7-41c7-8377-d539ce068b05,2025,2015,2015<sv>2015,0.0,0.0
70 | 68,91c9f153-654d-4d84-aeb1-1c9ca2c549cd,1985,1984,January 1985<sv>1984,0.67,0.0
71 | 69,585f56ac-5a3e-498d-b21a-1ca8eda1f456,3 November 1989,8 November 1989,1994 November 1989<sv>8 November 1989,0.67,0.0
72 | 70,26a8d265-b7a9-4aa0-9882-0ec3fc146ba5,24 August 2005,14 August 2005,14 August 2005<sv>14 August 2005,0.67,0.0
73 | 71,60bbfb58-b839-4bce-9ea3-8c43c85e262c,2010,2017,last year<sv>2017,0.0,0.0
74 | 72,b419c1ce-abe0-4139-a83b-713cd89f54f9,31 July 1941,19 July 1941,July 1941<sv>19 July 1941,0.8,0.0
75 | 73,e4337ab8-36a3-4195-94f3-62d25300ccbb,31 July 1941,19 July 1941,19 July 1941<sv>19 July 1941,0.67,0.0
76 | 74,086318c9-aa91-42fc-b2e5-94fbfa028a0b,1697,1694,1694<sv>1694,0.0,0.0
77 | 75,3c45e42f-5a1e-414c-9692-9a10f713d541,201 AD,210,210 AD<sv>210,0.0,0.0
78 | 76,703eb661-5524-4d0e-867e-882d842e974c,273,210,210<sv>210,0.0,0.0
79 | 77,a32961ce-e927-4c16-b924-b52c5a7a45c2,1995,2002,2002<sv>2002,0.0,0.0
80 | 78,4789a862-3100-406b-857a-618879434699,1995,1996,1996<sv>1996,0.0,0.0
81 | 79,7ad86006-5910-4fbc-99c4-ce10a2c71a6e,2011-12,2005,2005<sv>2005,0.0,0.0
82 | 80,cd055ab9-6680-4af7-94d9-35835355f08c,1580,1590,1590,0.0,0.0
83 | 81,35473529-30cf-4aaa-9aa2-4e59bc30a970,20 November 1996,27 November 1996,27 November 1996<sv>27,0.67,0.0
84 | 82,494a271f-d9cb-4861-8b65-4c9411684b5b,1996,1997,1977<sv>1997,0.0,0.0
85 | 83,8b405cec-665e-4f11-b3b4-698e6ea3b69c,2008,1980,1980<sv>1980,0.0,0.0
86 | 84,923c1673-9a62-4bef-9c45-6048d8f5ad09,1900,1950,1950<sv>1950,0.0,0.0
87 | 85,fd9f4aa9-62e0-4896-a03a-ac452679fac7,2014,2012,2012<sv>2012,0.0,0.0
88 | 


--------------------------------------------------------------------------------
/error_analysis/rc_exp_1_stage_3_5_e60_drop/eval_reports_valid_set/ea_date_sample_86.csv:
--------------------------------------------------------------------------------
 1 | ,query_id,prediction,answer,validated_answers,f1,exact_match
 2 | 0,4ef57f34-8c7c-4500-bc1f-2d2512e35523,1944,1943,1943<sv>1943<sv>1943<sv>1943,0.0,0.0
 3 | 1,0903294b-08fc-4061-937a-44890a55ca78,1944,1943,1943<sv>1943<sv>1943,0.0,0.0
 4 | 2,7c72066b-f148-4554-8b81-4d74fbe61101,1944,1943,1942<sv>1943,0.0,0.0
 5 | 3,45c95f84-92e4-4d83-b426-894687613da0,1683,1698,1698<sv>1698,0.0,0.0
 6 | 4,98f0f04a-7dbc-4717-bf97-5c272321d930,1913,1923,1923<sv>1923,0.0,0.0
 7 | 5,a76d9d78-2e31-49d9-9562-2abc3f758943,2002,2000,2000<sv>2000<sv>2000,0.0,0.0
 8 | 6,9d5b3589-d92c-46d8-8a22-0412f7d2d306,23 April 1734,1734,1734<sv>1734<sv>1734<sv>1734<sv>1734,0.5,0.0
 9 | 7,381ef468-9cc8-4b70-98b2-eab5183952b3,May 1751,April 1752,May 1754<sv>1752<sv>May 1754,0.0,0.0
10 | 8,32fcdc18-7118-40b5-a22e-13e4ae90bc20,December 1769,1769,1767<sv>1769,0.67,0.0
11 | 9,8fd1e5f3-e0dc-4996-aeb4-63f90f76e771,13 August 1563,August 1563,August<sv>August 1563,0.8,0.0
12 | 10,cca408e3-6075-430b-9e58-b820bb6aca06,13 August 1563,August 1563,August 1563<sv>August 1563<sv>August 1563,0.8,0.0
13 | 11,b44a05fc-4696-402f-a313-fb5236ac0ab6,30 May 1563,1 September 1563,August 1563<sv>4 September,0.4,0.0
14 | 12,8cbc1e9a-7f0c-4a32-8089-658398a9a1f0,1442,1438,1438<sv>1438,0.0,0.0
15 | 13,59b2070a-4027-49d5-b797-2ec7befcf2c4,1440,1452,1452<sv>1452,0.0,0.0
16 | 14,9fdfd076-2163-448e-b67d-1ad392ebfc5d,1440,1452,1452<sv>1452<sv>1552,0.0,0.0
17 | 15,79bacdf4-b669-4e2a-9be3-e73ab82a9467,1980,1955,1955<sv>1955,0.0,0.0
18 | 16,2f2a8ca0-199f-433a-a64c-d65b03c7ddd5,1980,1955,1955<sv>1955,0.0,0.0
19 | 17,e7d28a55-3aab-4313-9b65-ca20c329e166,In retaliation,1277,1277<sv>1277,0.0,0.0
20 | 18,9b6d8136-2535-448c-97b2-708ebe73b75c,1276,1277,1277<sv>1277,0.0,0.0
21 | 19,167c68ad-6ecd-4915-a795-f5f91441c413,1990,2010,2010<sv>2010,0.0,0.0
22 | 20,6c6f5cbe-c01c-46f0-aad5-df16aa478aa5,1990,1981,1981<sv>1981<sv>1981<sv>1981,0.0,0.0
23 | 21,45be16ff-c43e-412b-a6f2-6439e38a12c8,1583,1593,1593<sv>1593,0.0,0.0
24 | 22,d7c261cc-b929-4a1c-81b8-55a4d5f6fdbb,1553,1525,1525<sv>1525<sv>1525,0.0,0.0
25 | 23,ff20ab78-6d2c-4238-919e-34c40b4c9194,10 September,10 October 1556,29 September 1556<sv>October 10,0.5,0.0
26 | 24,dc5a38ba-47c4-4ff3-882f-0854b724928e,1418,1488,1488<sv>1488,0.0,0.0
27 | 25,c01c2890-80fc-4dd3-9073-533f7d2de8a8,27 April 100,30 April,27 April<sv>27,0.8,0.0
28 | 26,686e10d5-95bd-4b50-b2a8-72dfe6439f2f,2009,2007,2007<sv>2007,0.0,0.0
29 | 27,d2bb1b96-b931-4e1a-ab99-092d3bd0af52,February 29,1897,1896<sv>1897,0.0,0.0
30 | 28,e444e95d-e3e0-47b0-a5a8-38f99430f3cc,May,June,June 1272<sv>June,0.0,0.0
31 | 29,d3af4f76-8f9e-4199-84da-6956b4443279,May,June 1272,June<sv>June 1272,0.0,0.0
32 | 30,fd2ead2e-e669-409a-9dbf-700c838d8c15,1683,1698,1698<sv>1698,0.0,0.0
33 | 31,6629ae4d-99a4-4786-a8e8-4c0d723dc231,1683,1698,1698<sv>1698,0.0,0.0
34 | 32,2d60e681-7ca9-41a2-956b-1233fde15719,2003,2002,2002<sv>2002,0.0,0.0
35 | 33,f47e8521-c8ad-42f8-b164-02c35afff6c5,33 August 2000,31 April 2000,2000<sv>2000,0.5,0.0
36 | 34,61de672b-6d8d-4b3e-8394-1b73c8fcbc4d,April 1606,April 1605,April 1605<sv>3 March 1606,0.4,0.0
37 | 35,db540314-ad8e-4e87-bed8-d57ae991d26a,April 1606,April 1605,April 1605<sv>1605,0.0,0.0
38 | 36,72407857-72ac-480e-bc2d-0e4ce3781f10,1854,1852,1852<sv>1852,0.0,0.0
39 | 37,1aed8cf7-96a2-4c0d-af61-ff771944db22,The Siege of Nice,1542,Polin succeeding Antoine de Rincon<sv>Polin succeeding Antoine de Rincon,0.0,0.0
40 | 38,af5b5adf-ed88-45a3-bd91-902078145db0,1464,1384,1384<sv>1384,0.0,0.0
41 | 39,8593bfa1-eb03-489e-aa64-1364739265b7,1236,1234,1234<sv>1234,0.0,0.0
42 | 40,acfd906d-5098-4038-a55c-2be27af298e6,6 December 1283,3 December 1283,3 December 1283,0.67,0.0
43 | 41,c352733d-975e-4dc7-9f28-695dcc6d4f31,6 February 1284,5 February 1284,5 February 1284<sv>5 February 1284,0.67,0.0
44 | 42,1173ca51-f21e-4d5a-9092-93caf53eedd4,9 December 1283,3 December 1283,3 December 1283<sv>3 December 1283<sv>3 December 1283,0.67,0.0
45 | 43,6806f0b2-3baa-4919-ac39-b47c407bdbb5,26 January 1285,3 December 1283,3 December 1283<sv>3 December 1283,0.0,0.0
46 | 44,fe6f6f4e-7654-4574-83e7-d32f81c3af13,26 January 1285,3 December 1283,1283<sv>3 December 1283,0.0,0.0
47 | 45,e50b593b-a77a-4b67-8833-2820eb6fc314,1284,1285,that same year<sv>1285,0.0,0.0
48 | 46,1748ad77-26e5-4a8b-9881-40b7151d99e0,1982,1983,1981<sv>1981<sv>1981,0.0,0.0
49 | 47,6cdb01b3-76f4-4d14-8911-0badd33ac25b,March 1742,February 1743,February 1743<sv>February 1743,0.0,0.0
50 | 48,1b9b6544-d73a-4150-a1b6-68a2ee20b8eb,1920,1919,1919<sv>1919,0.0,0.0
51 | 49,4f47dd36-456b-4991-b2b3-3b3208d176b2,4 October 1910,5 October 1910,5 October 1910<sv>5 October 1910,0.67,0.0
52 | 50,247acee1-f214-46d2-a33a-f55d2ff49b32,1067,1061,1061<sv>1068,0.0,0.0
53 | 51,c323389b-e07c-4235-9ce7-3c2e955397b0,24 April 1976,3 February,3 February<sv>3 February,0.0,0.0
54 | 52,72b358b9-8cae-45a2-9b5c-597ae8ad56bf,1614,1615,1615<sv>1615,0.0,0.0
55 | 53,b113d114-d358-464e-84ab-866901f1c07f,1976,2009,2009<sv>2009,0.0,0.0
56 | 54,1dd90b41-df53-40fb-9528-847a0f9474b7,1890,May 2016,1899<sv>May 2016<sv>May 2016,0.0,0.0
57 | 55,b13d08ae-53b0-47ce-ab02-954db5fd58b0,1350,1367,1367<sv>1367<sv>1367<sv>1367<sv>1367,0.0,0.0
58 | 56,89eb3b39-409c-48ee-8856-c16c1f89cc40,31 August 1782,31 August 1781,31 August 1781<sv>31 August 1781,0.67,0.0
59 | 57,46142269-3156-44c8-95f2-5df25abd420c,1990,2004,2004<sv>2004,0.0,0.0
60 | 58,f79aa0fb-2c48-4e9d-ae8a-985a28ebf63b,2004,2010,2010<sv>2010,0.0,0.0
61 | 59,cee9867a-2ed2-40fc-a2f9-aa43117ef764,1908,1904,1904<sv>1904,0.0,0.0
62 | 60,a26a97a8-8ca9-4c2c-b6e4-490d37826f09,1905 <ss> 1906 <ss> 1907,1904,1904<sv>1904,0.0,0.0
63 | 61,c035a076-35c0-440c-b39d-4951027a3cec,1906,1904,1904<sv>1904,0.0,0.0
64 | 62,75afb1a1-c59b-4b60-a7d2-2346f9f2b68a,2011,1990,1990<sv>1990,0.0,0.0
65 | 63,db1dd1a9-a580-4c43-8f12-19da771715c3,2016,2015,2015<sv>2015,0.0,0.0
66 | 64,abef9d7e-48fc-4f4a-a842-707e7be221f6,2004,12000,2000<sv>12000,0.0,0.0
67 | 65,4f02227d-6f5c-4501-8cdc-54a2fab27452,2017,2016,2016<sv>2016,0.0,0.0
68 | 66,6ce9b33b-b86e-4390-8d9d-c2e6e7b353fe,1996,1982,1996-97<sv>1982,0.67,0.0
69 | 67,fa4a1714-eac7-41c7-8377-d539ce068b05,2025,2015,2015<sv>2015,0.0,0.0
70 | 68,91c9f153-654d-4d84-aeb1-1c9ca2c549cd,1985,1984,January 1985<sv>1984,0.67,0.0
71 | 69,585f56ac-5a3e-498d-b21a-1ca8eda1f456,3 November 1989,8 November 1989,1994 November 1989<sv>8 November 1989,0.67,0.0
72 | 70,26a8d265-b7a9-4aa0-9882-0ec3fc146ba5,24 August 2005,14 August 2005,14 August 2005<sv>14 August 2005,0.67,0.0
73 | 71,60bbfb58-b839-4bce-9ea3-8c43c85e262c,2010,2017,last year<sv>2017,0.0,0.0
74 | 72,b419c1ce-abe0-4139-a83b-713cd89f54f9,31 July 1941,19 July 1941,July 1941<sv>19 July 1941,0.8,0.0
75 | 73,e4337ab8-36a3-4195-94f3-62d25300ccbb,31 July 1941,19 July 1941,19 July 1941<sv>19 July 1941,0.67,0.0
76 | 74,086318c9-aa91-42fc-b2e5-94fbfa028a0b,1697,1694,1694<sv>1694,0.0,0.0
77 | 75,3c45e42f-5a1e-414c-9692-9a10f713d541,201 AD,210,210 AD<sv>210,0.0,0.0
78 | 76,703eb661-5524-4d0e-867e-882d842e974c,273,210,210<sv>210,0.0,0.0
79 | 77,a32961ce-e927-4c16-b924-b52c5a7a45c2,1995,2002,2002<sv>2002,0.0,0.0
80 | 78,4789a862-3100-406b-857a-618879434699,1995,1996,1996<sv>1996,0.0,0.0
81 | 79,7ad86006-5910-4fbc-99c4-ce10a2c71a6e,2011-12,2005,2005<sv>2005,0.0,0.0
82 | 80,cd055ab9-6680-4af7-94d9-35835355f08c,1580,1590,1590,0.0,0.0
83 | 81,35473529-30cf-4aaa-9aa2-4e59bc30a970,20 November 1996,27 November 1996,27 November 1996<sv>27,0.67,0.0
84 | 82,494a271f-d9cb-4861-8b65-4c9411684b5b,1996,1997,1977<sv>1997,0.0,0.0
85 | 83,8b405cec-665e-4f11-b3b4-698e6ea3b69c,2008,1980,1980<sv>1980,0.0,0.0
86 | 84,923c1673-9a62-4bef-9c45-6048d8f5ad09,1900,1950,1950<sv>1950,0.0,0.0
87 | 85,fd9f4aa9-62e0-4896-a03a-ac452679fac7,2014,2012,2012<sv>2012,0.0,0.0
88 | 


--------------------------------------------------------------------------------
/error_analysis/rc_exp_1_stage_3_5_e60_drop/eval_reports_valid_set/ea_number_sample_100.csv:
--------------------------------------------------------------------------------
  1 | ,query_id,prediction,answer,validated_answers,f1,exact_match
  2 | 1539,deb4531f-3dfd-4f7d-8c05-c8e810a72d6c,14,2,3<sv>2,0.0,0.0
  3 | 463,6a8ad68c-a109-41c2-a803-0f574786355e,10.5,60.7,60.7<sv>60.7,0.0,0.0
  4 | 906,2e3f7398-c2c0-4a27-9e01-65564d7f9856,3,2,2<sv>2<sv>2,0.0,0.0
  5 | 1640,bd5d6cf0-8d41-426a-967f-bcfe5b9276e9,5000,50800,37200<sv>50800,0.0,0.0
  6 | 63,41f67f99-45a1-4708-8ab3-18f2b3d86ef8,1,2,2,0.0,0.0
  7 | 708,4ae5022a-aa54-4fd0-af92-50fab967378c,41,8,56<sv>56<sv>56<sv>56,0.0,0.0
  8 | 413,a42c6f15-e259-4367-916f-774631c5ae88,3,38,0,0.0,0.0
  9 | 1696,81e7db95-a6ae-45a4-b0a9-fe7aa52f8250,4.8,4.9,4.9<sv>4.9,0.0,0.0
 10 | 1193,5bc8b6b1-4f81-49ef-97f3-df0b7800df67,4641,6831,6831<sv>6831,0.0,0.0
 11 | 1313,67fe323f-0a36-488e-ae6f-6366a6ce3f14,6,2,Portuguese Colonial War<ss>Six-Day War by Israel<sv>3,0.0,0.0
 12 | 83,b68faa1d-deb1-491e-b12f-35c10e6bc0db,63,53,53<sv>53<sv>53,0.0,0.0
 13 | 1455,9d2d700b-6454-49b3-af70-97b8303572d2,400,233,233<sv>233,0.0,0.0
 14 | 465,13483b80-8b39-4ff9-98d1-5b892cd56b41,6.3,60.7,73.2<sv>60.7<sv>60.7,0.0,0.0
 15 | 215,ed5ea5d3-53f3-4c23-9338-101e96170dc2,3,2,2<sv>2,0.0,0.0
 16 | 374,30504e30-71aa-4643-98f3-8132de444b3b,6,3,2<sv>2<sv>2<sv>4,0.0,0.0
 17 | 533,25fcff18-0e99-4106-9108-4a808037fb91,5,1,1<sv>1,0.0,0.0
 18 | 466,b246ca25-9c62-481f-9257-412c1d022f2e,6.3,60.7,60.7<sv>60.7,0.0,0.0
 19 | 154,f466f6ff-52c1-41d2-aabf-45adc57527fa,31,1526,1526<sv>1526,0.0,0.0
 20 | 1246,365a6f48-4697-41b0-afd1-0711d1e930c7,138,1.81,1.81<sv>1.81,0.0,0.0
 21 | 861,7b97c202-4eee-4f9f-9ed2-ca04972bd115,1300,300,2000<sv>300,0.0,0.0
 22 | 1355,8615ad1c-7bbe-441a-bc31-89c11a31abd2,11437,2.73,2.73<sv>2.73,0.0,0.0
 23 | 1631,ef8e7317-1c63-45da-8d89-b17fe86f234c,4017,4117,4117<sv>4117,0.0,0.0
 24 | 1670,0d1f0c21-7a38-4a36-9827-d87e0a5669fa,4,3,Germany<ss>Italy<ss>France<sv>3,0.0,0.0
 25 | 1667,cccb4bc9-405c-4ff4-aafc-5c1dc2080f16,4.0,5,5<sv>5,0.0,0.0
 26 | 1147,bd473f9a-5643-4a8b-ae22-0431e3ece379,71,81,191<sv>81,0.0,0.0
 27 | 574,73bf60f1-dcb5-4bb0-9f67-3caae6e1c526,19,57,57<sv>57,0.0,0.0
 28 | 330,2b769309-c6b4-4106-af0d-46456ae6a1f9,22.9,21.9,21.9<sv>21.9<sv>21.9,0.0,0.0
 29 | 513,859719cd-e2b2-43cc-9edf-d33e6099cb3e,9.7,15.7,15.7<sv>15.7,0.0,0.0
 30 | 1023,4623b426-9754-4945-84af-d72db8563c9a,19,9,9<sv>9,0.0,0.0
 31 | 1196,1950b2e2-801c-4c06-9cba-25be7ca5eab4,1,2,2<sv>2,0.0,0.0
 32 | 1702,6731084b-fb60-4dbc-b254-a9180044e87a,4,21,2<sv>21,0.0,0.0
 33 | 1141,20c29368-7cc7-4a57-9fdb-e211a3af4e14,43,53,53<sv>10 August 1544,0.0,0.0
 34 | 1236,391e850b-dc79-4d7a-b4e9-180ca5a31c05,188,178,178<sv>178,0.0,0.0
 35 | 831,32531770-cac5-470b-b048-8f1e9c64156f,45,42,42<sv>42<sv>42,0.0,0.0
 36 | 1711,5deadbd9-2096-404f-a45a-a90baf8217ff,5,2,2<sv>2,0.0,0.0
 37 | 221,ff26d0d8-6ac6-4f10-ab2a-e1e4f47af971,4,3,3<sv>3,0.0,0.0
 38 | 1460,cd858d75-24f3-48b9-a1c6-94694632b593,2,9,9<sv>9,0.0,0.0
 39 | 1261,fd05a1e4-842c-4ae8-8b93-46eb3620837f,11,10,10<sv>10,0.0,0.0
 40 | 532,ef732aa1-1bdd-4183-9223-dc6452681ad9,38,138,138<sv>138,0.0,0.0
 41 | 386,ed464f2a-f347-4591-867c-b9c850cac964,3,6,"67,000 kilometers of roads<ss>7,500 kilometers of railways<ss>many large bridges<ss>modern irrigation systems<ss>several international harbors<ss>140 public drinking water systems<sv>6",0.0,0.0
 42 | 818,616f1936-bcb7-44d9-aa66-d34d920d9e69,6,11,8<sv>11,0.0,0.0
 43 | 886,3dc39720-264d-4235-b023-6aa4d480c5a9,3,4,4<sv>4,0.0,0.0
 44 | 1352,0616f167-5a33-4005-a0ca-6a785da63597,5.2,6,6,0.0,0.0
 45 | 355,4e78f07b-07f0-45fe-803a-ac2b01ecf1c9,4312,4187,4187<sv>160.000,0.0,0.0
 46 | 1576,642915d1-04b2-4851-8adb-220e9604aba9,20,11,11,0.0,0.0
 47 | 840,8bcf741b-787e-4d94-a052-54000f89a920,48,54,54<sv>49,0.0,0.0
 48 | 695,21115a01-56ba-4818-aff9-bb6cafea2a32,4,3,3<sv>3,0.0,0.0
 49 | 482,c9b47364-841e-49fc-9f77-5250b3cb7ddc,4,3,3<sv>3,0.0,0.0
 50 | 1143,e3301aaf-848e-44a1-b76a-20791ef6c55d,19,10,10<sv>33,0.0,0.0
 51 | 1443,c7322c2c-dcb3-42cb-b0ae-97021dd04561,52452,13144,13144<sv>13144,0.0,0.0
 52 | 1142,5a6b733e-5e61-402b-b97b-70d2ab7329bf,4,2,2<sv>2 October 1544,0.0,0.0
 53 | 1429,544591c9-0fb3-4c36-a3b6-69002fd90d28,13.11,7.04,7.04<sv>7.04,0.0,0.0
 54 | 1232,f23bab05-c051-43a5-a25c-1f869599fc7f,1428,2122,2125<sv>9.2,0.0,0.0
 55 | 1333,312d2f43-7b26-49ab-a1e6-8a0525a79d62,18.7,.5,.5<sv>.5,0.0,0.0
 56 | 584,9bf015d2-429d-4b2a-97dc-1b861a5c2e97,76,40,50<sv>52,0.0,0.0
 57 | 1061,c69e1211-2b3b-4dd1-aaea-b3a867890df1,21.7,30.5,30.5<sv>30.5,0.0,0.0
 58 | 292,00a87937-cf6b-487e-a5fd-fd93ab298c35,15399187,15499287,15499287<sv>15529287<sv>15499287<sv>15499287,0.0,0.0
 59 | 780,c8c3ea22-eefd-49e9-abb2-2a24e9c530c3,38.66,42,42<sv>42,0.0,0.0
 60 | 512,0def9c5e-066c-45a4-ab56-81c7c7c685f4,300,200,200<sv>200,0.0,0.0
 61 | 927,af260a26-836d-4204-9b59-fd97927117b2,17.88,8.88,8.88<sv>8.88<sv>8.88<sv>8.88,0.0,0.0
 62 | 1492,beb7e8bf-440e-4423-bf0a-0b188dbfb087,39,10,2<sv>10,0.0,0.0
 63 | 849,4f823e2e-bcf4-407e-ab19-66593b49f575,14,18,21<sv>21,0.0,0.0
 64 | 1069,a859e78c-f025-4a15-8f21-b9ccb5de8697,11.9,16.8,16.8,0.0,0.0
 65 | 1136,09b90451-e508-4b2e-9c96-948b18573bac,2,1,1<sv>1,0.0,0.0
 66 | 1051,35bd9df9-d1c7-4e3b-8ee1-98755f94d90b,1173000,173000,173000<sv>173000,0.0,0.0
 67 | 86,2f80b82f-a430-4832-85a5-16963487c4b7,70,10,36<sv>36,0.0,0.0
 68 | 1524,247333ad-d933-49ad-9d6c-ff9372d31c73,488267,9388267,9388807<sv>9388267,0.0,0.0
 69 | 470,891b6805-1952-481a-ba19-e53daa399a67,1,3,3<sv>3,0.0,0.0
 70 | 321,4bd0232f-ad7b-4140-b543-9bfa34d7caf5,18,1,1<sv>1,0.0,0.0
 71 | 1315,8f6813c3-3469-475b-82af-9309feac4410,6,2,1965<ss>1971<sv>2,0.0,0.0
 72 | 968,a4901d1d-6f6c-4f5c-a897-3af6d4bc5546,3,4,4<sv>4,0.0,0.0
 73 | 310,e164bf4e-fd24-4071-910b-446d3ab97644,41.25,40.75,41<sv>40.75,0.0,0.0
 74 | 377,69f0add7-09b8-4def-86f8-f196003f0159,3,32,25<sv>2042,0.0,0.0
 75 | 1457,278601c1-f1ca-43a4-8590-1da4254ff2f2,97,69,69<sv>69,0.0,0.0
 76 | 1341,3b0f9180-c60c-4a19-acbb-c7c9b8d80cd8,43,76,76<sv>76,0.0,0.0
 77 | 889,e1668e02-998b-4a36-bd31-41ace887db5b,5,3,3<sv>3,0.0,0.0
 78 | 647,e9ae0bd8-9a5d-4809-8ce8-ace215b971c2,5,3,4<sv>4<sv>4,0.0,0.0
 79 | 449,be8226d3-3893-4b94-9a46-34b9c8b3e9c7,1500,500,500<sv>500,0.0,0.0
 80 | 282,5ea8cd2e-b1c5-4901-a815-02f6e98d90fe,14,5,5<sv>5<sv>5,0.0,0.0
 81 | 415,d818f5ca-4d38-4b51-a750-0285fdb76802,6,5,5<sv>4,0.0,0.0
 82 | 350,b264b13e-f6a9-47ad-acc1-f987d93d7e5b,7628,628,628<sv>628,0.0,0.0
 83 | 1044,b0fe6f5f-9b1c-468f-b57a-7e646f489b4a,4,2,2<sv>2,0.0,0.0
 84 | 1726,eb92ceb7-af99-4c2c-8ad6-91ba990d107e,43,9,9<sv>9,0.0,0.0
 85 | 727,60e0ffa6-d4c1-4c26-aa08-23347dcb847c,2,1,1<sv>1<sv>9,0.0,0.0
 86 | 39,10d8063d-1a20-4cd5-819e-1a77f14afda9,13214,9117,9117<sv>9117<sv>9117<sv>9117,0.0,0.0
 87 | 1265,066aeaf6-10ff-4dcb-8a01-919fbeadf5ab,2.1,2,94.1<sv>2,0.0,0.0
 88 | 1317,cea24564-bfbe-4723-8089-6d11635e013a,119,198,198<sv>198,0.0,0.0
 89 | 1174,31081a5d-b5c6-483a-8ae5-b10335719b5c,3843306,3843316,3843316<sv>3843316,0.0,0.0
 90 | 1021,88707d47-9392-495d-8dcc-ef44c6609715,14,0,0,0.0,0.0
 91 | 1616,c3061d02-9549-4398-97bd-9d16cbfb8246,3,4,4<sv>4,0.0,0.0
 92 | 387,fc38f709-a5df-48b1-9d54-dfca095fae43,7500,74500,74500,0.0,0.0
 93 | 1607,54b3be33-59da-4860-86b5-5c5c17833ee6,1400000,7,7000000<sv>7,0.0,0.0
 94 | 955,53851614-70a8-4a1c-8a8d-4435c053fd45,3063,2240,11.6%<sv>2239,0.0,0.0
 95 | 1698,cad40e98-d5de-4522-9769-93c8dc4d400a,3.8,3.9,3.9<sv>3.9,0.0,0.0
 96 | 1658,1e507704-e02d-4e8a-9007-e51ed9f1a8dd,1525,1725,1725<sv>1725,0.0,0.0
 97 | 943,529f6ef7-6af9-4527-b25f-03a0edaff6cb,17,5,5<sv>5<sv>5,0.0,0.0
 98 | 1037,af1aa186-2eef-4e01-ac8c-c17091694d2a,20,9,9<sv>9,0.0,0.0
 99 | 1579,70bc3caa-6b10-427d-88c4-5190ce18e871,5,8,8<sv>8,0.0,0.0
100 | 1074,c86c075a-af4d-4655-9dcd-12ea39b7c8ef,482427,1633680,"1633680<sv>1,962,059",0.0,0.0
101 | 778,35340705-28a9-4165-a5c2-3cb3e3023e82,18,24,24<sv>24,0.0,0.0
102 | 


--------------------------------------------------------------------------------
/error_analysis/val_exp_2_stage_2_4_e60_drop/eval_reports_valid_set/ea_date_full_90.csv:
--------------------------------------------------------------------------------
 1 | ,query_id,prediction,answer,validated_answers,f1,exact_match
 2 | 0,4ef57f34-8c7c-4500-bc1f-2d2512e35523,1944,1943,1943<sv>1943<sv>1943<sv>1943,0.0,0.0
 3 | 1,0903294b-08fc-4061-937a-44890a55ca78,1944,1943,1943<sv>1943<sv>1943,0.0,0.0
 4 | 2,7c72066b-f148-4554-8b81-4d74fbe61101,1944,1943,1942<sv>1943,0.0,0.0
 5 | 3,45c95f84-92e4-4d83-b426-894687613da0,1683,1698,1698<sv>1698,0.0,0.0
 6 | 4,a76d9d78-2e31-49d9-9562-2abc3f758943,2002,2000,2000<sv>2000<sv>2000,0.0,0.0
 7 | 5,381ef468-9cc8-4b70-98b2-eab5183952b3,May 1752,April 1752,May 1754<sv>1752<sv>May 1754,0.67,0.0
 8 | 6,8fd1e5f3-e0dc-4996-aeb4-63f90f76e771,13 August 1563,August 1563,August<sv>August 1563,0.8,0.0
 9 | 7,cca408e3-6075-430b-9e58-b820bb6aca06,13 August 1563,August 1563,August 1563<sv>August 1563<sv>August 1563,0.8,0.0
10 | 8,b44a05fc-4696-402f-a313-fb5236ac0ab6,4 September 1563,1 September 1563,August 1563<sv>4 September,0.8,0.0
11 | 9,8cbc1e9a-7f0c-4a32-8089-658398a9a1f0,1442,1438,1438<sv>1438,0.0,0.0
12 | 10,59b2070a-4027-49d5-b797-2ec7befcf2c4,1440,1452,1452<sv>1452,0.0,0.0
13 | 11,9fdfd076-2163-448e-b67d-1ad392ebfc5d,1440,1452,1452<sv>1452<sv>1552,0.0,0.0
14 | 12,1aa99f9e-5291-4e8e-9be5-8016fee01767,1675,8 August 1675,August 1675<sv>August 1675,0.67,0.0
15 | 13,79bacdf4-b669-4e2a-9be3-e73ab82a9467,1980,1955,1955<sv>1955,0.0,0.0
16 | 14,2f2a8ca0-199f-433a-a64c-d65b03c7ddd5,1980,1955,1955<sv>1955,0.0,0.0
17 | 15,e7d28a55-3aab-4313-9b65-ca20c329e166,In the next year,1277,1277<sv>1277,0.0,0.0
18 | 16,9b6d8136-2535-448c-97b2-708ebe73b75c,1276,1277,1277<sv>1277,0.0,0.0
19 | 17,167c68ad-6ecd-4915-a795-f5f91441c413,1990,2010,2010<sv>2010,0.0,0.0
20 | 18,6c6f5cbe-c01c-46f0-aad5-df16aa478aa5,1990,1981,1981<sv>1981<sv>1981<sv>1981,0.0,0.0
21 | 19,c68e2b2d-a1da-41c6-9866-7cce53b70791,1882,1874,1874<sv>1874,0.0,0.0
22 | 20,6580ed72-f11b-45b5-a890-1e8e3860afc3,1699,1606,1606<sv>1606,0.0,0.0
23 | 21,45be16ff-c43e-412b-a6f2-6439e38a12c8,1583,1593,1593<sv>1593,0.0,0.0
24 | 22,d7c261cc-b929-4a1c-81b8-55a4d5f6fdbb,1553,1525,1525<sv>1525<sv>1525,0.0,0.0
25 | 23,ff20ab78-6d2c-4238-919e-34c40b4c9194,10 September,10 October 1556,29 September 1556<sv>October 10,0.5,0.0
26 | 24,3a2cd28a-4deb-4715-9225-ab53e78d11d7,1446,1445,1477<sv>1445<sv>1445,0.0,0.0
27 | 25,dc5a38ba-47c4-4ff3-882f-0854b724928e,1418,1488,1488<sv>1488,0.0,0.0
28 | 26,c01c2890-80fc-4dd3-9073-533f7d2de8a8,28 April,30 April,27 April<sv>27,0.0,0.0
29 | 27,686e10d5-95bd-4b50-b2a8-72dfe6439f2f,2009,2007,2007<sv>2007,0.0,0.0
30 | 28,6ca5606a-706a-4325-86f8-e6b69b0575b6,1724,1722,1729,0.0,0.0
31 | 29,d2bb1b96-b931-4e1a-ab99-092d3bd0af52,February,1897,1896<sv>1897,0.0,0.0
32 | 30,e444e95d-e3e0-47b0-a5a8-38f99430f3cc,May,June,June 1272<sv>June,0.0,0.0
33 | 31,d3af4f76-8f9e-4199-84da-6956b4443279,May,June 1272,June<sv>June 1272,0.0,0.0
34 | 32,fd2ead2e-e669-409a-9dbf-700c838d8c15,1683,1698,1698<sv>1698,0.0,0.0
35 | 33,6629ae4d-99a4-4786-a8e8-4c0d723dc231,1683,1698,1698<sv>1698,0.0,0.0
36 | 34,2d60e681-7ca9-41a2-956b-1233fde15719,2003,2002,2002<sv>2002,0.0,0.0
37 | 35,f47e8521-c8ad-42f8-b164-02c35afff6c5,32 August 2000,31 April 2000,2000<sv>2000,0.5,0.0
38 | 36,61de672b-6d8d-4b3e-8394-1b73c8fcbc4d,April 1606,April 1605,April 1605<sv>3 March 1606,0.4,0.0
39 | 37,db540314-ad8e-4e87-bed8-d57ae991d26a,April 1606,April 1605,April 1605<sv>1605,0.0,0.0
40 | 38,72407857-72ac-480e-bc2d-0e4ce3781f10,1854,1852,1852<sv>1852,0.0,0.0
41 | 39,1aed8cf7-96a2-4c0d-af61-ff771944db22,The Siege of Nice,1542,Polin succeeding Antoine de Rincon<sv>Polin succeeding Antoine de Rincon,0.0,0.0
42 | 40,af5b5adf-ed88-45a3-bd91-902078145db0,1424,1384,1384<sv>1384,0.0,0.0
43 | 41,8593bfa1-eb03-489e-aa64-1364739265b7,1236,1234,1234<sv>1234,0.0,0.0
44 | 42,acfd906d-5098-4038-a55c-2be27af298e6,9 December 1283,3 December 1283,3 December 1283,0.67,0.0
45 | 43,1173ca51-f21e-4d5a-9092-93caf53eedd4,9 December 1283,3 December 1283,3 December 1283<sv>3 December 1283<sv>3 December 1283,0.67,0.0
46 | 44,6806f0b2-3baa-4919-ac39-b47c407bdbb5,26 January 1285,3 December 1283,3 December 1283<sv>3 December 1283,0.0,0.0
47 | 45,fe6f6f4e-7654-4574-83e7-d32f81c3af13,26 January 1285,3 December 1283,1283<sv>3 December 1283,0.0,0.0
48 | 46,e50b593b-a77a-4b67-8833-2820eb6fc314,26 January 1285,1285,that same year<sv>1285,0.5,0.0
49 | 47,1d703a46-9934-4545-b550-2416997672d1,10,7 October 1580,7 October 1580<sv>10 October 1580<sv>7 October 1580,0.5,0.0
50 | 48,1748ad77-26e5-4a8b-9881-40b7151d99e0,1980,1983,1981<sv>1981<sv>1981,0.0,0.0
51 | 49,6cdb01b3-76f4-4d14-8911-0badd33ac25b,March 1742,February 1743,February 1743<sv>February 1743,0.0,0.0
52 | 50,1b9b6544-d73a-4150-a1b6-68a2ee20b8eb,1920,1919,1919<sv>1919,0.0,0.0
53 | 51,4f47dd36-456b-4991-b2b3-3b3208d176b2,4 October 1910,5 October 1910,5 October 1910<sv>5 October 1910,0.67,0.0
54 | 52,247acee1-f214-46d2-a33a-f55d2ff49b32,1055,1061,1061<sv>1068,0.0,0.0
55 | 53,c323389b-e07c-4235-9ce7-3c2e955397b0,1522,3 February,3 February<sv>3 February,0.0,0.0
56 | 54,72b358b9-8cae-45a2-9b5c-597ae8ad56bf,1614,1615,1615<sv>1615,0.0,0.0
57 | 55,1dd90b41-df53-40fb-9528-847a0f9474b7,1890,May 2016,1899<sv>May 2016<sv>May 2016,0.0,0.0
58 | 56,b13d08ae-53b0-47ce-ab02-954db5fd58b0,1350,1367,1367<sv>1367<sv>1367<sv>1367<sv>1367,0.0,0.0
59 | 57,0d5cfb06-b1e5-45d2-93e2-3742fcf59e80,1350,1367,1367<sv>1367,0.0,0.0
60 | 58,89eb3b39-409c-48ee-8856-c16c1f89cc40,31 August,31 August 1781,31 August 1781<sv>31 August 1781,0.8,0.0
61 | 59,f79aa0fb-2c48-4e9d-ae8a-985a28ebf63b,2004,2010,2010<sv>2010,0.0,0.0
62 | 60,cee9867a-2ed2-40fc-a2f9-aa43117ef764,1908,1904,1904<sv>1904,0.0,0.0
63 | 61,a26a97a8-8ca9-4c2c-b6e4-490d37826f09,1905 <ss> 1906 <ss> 1907,1904,1904<sv>1904,0.0,0.0
64 | 62,c035a076-35c0-440c-b39d-4951027a3cec,1906,1904,1904<sv>1904,0.0,0.0
65 | 63,75afb1a1-c59b-4b60-a7d2-2346f9f2b68a,2011,1990,1990<sv>1990,0.0,0.0
66 | 64,9a403ff7-e467-469b-a86d-d5f891bd8332,2000,1990,1990<sv>1990,0.0,0.0
67 | 65,15b6fad2-00df-41a0-b5e9-929120cfc43d,2016,2015,2015<sv>2015,0.0,0.0
68 | 66,db1dd1a9-a580-4c43-8f12-19da771715c3,2016,2015,2015<sv>2015,0.0,0.0
69 | 67,abef9d7e-48fc-4f4a-a842-707e7be221f6,2012,12000,2000<sv>12000,0.0,0.0
70 | 68,4f02227d-6f5c-4501-8cdc-54a2fab27452,2017,2016,2016<sv>2016,0.0,0.0
71 | 69,6ce9b33b-b86e-4390-8d9d-c2e6e7b353fe,1996,1982,1996-97<sv>1982,0.67,0.0
72 | 70,fa4a1714-eac7-41c7-8377-d539ce068b05,2025,2015,2015<sv>2015,0.0,0.0
73 | 71,585f56ac-5a3e-498d-b21a-1ca8eda1f456,14 November 1989,8 November 1989,1994 November 1989<sv>8 November 1989,0.67,0.0
74 | 72,26a8d265-b7a9-4aa0-9882-0ec3fc146ba5,24 August 2005,14 August 2005,14 August 2005<sv>14 August 2005,0.67,0.0
75 | 73,60bbfb58-b839-4bce-9ea3-8c43c85e262c,2019,2017,last year<sv>2017,0.0,0.0
76 | 74,b419c1ce-abe0-4139-a83b-713cd89f54f9,31 July 1941,19 July 1941,July 1941<sv>19 July 1941,0.8,0.0
77 | 75,e4337ab8-36a3-4195-94f3-62d25300ccbb,31 July 1941,19 July 1941,19 July 1941<sv>19 July 1941,0.67,0.0
78 | 76,3c45e42f-5a1e-414c-9692-9a10f713d541,273 AD,210,210 AD<sv>210,0.0,0.0
79 | 77,703eb661-5524-4d0e-867e-882d842e974c,273,210,210<sv>210,0.0,0.0
80 | 78,a32961ce-e927-4c16-b924-b52c5a7a45c2,1995,2002,2002<sv>2002,0.0,0.0
81 | 79,4789a862-3100-406b-857a-618879434699,1995,1996,1996<sv>1996,0.0,0.0
82 | 80,5abe52c8-45fb-4e3c-97f1-ad9255c8f209,2008,1997,70000<sv>1997,0.0,0.0
83 | 81,445f9cb5-b4b9-4b83-9dac-26ae4d4956dd,29 May 1991,25 April 1991,25 April 1991<sv>25 April 1991,0.33,0.0
84 | 82,7ad86006-5910-4fbc-99c4-ce10a2c71a6e,2011-12,2005,2005<sv>2005,0.0,0.0
85 | 83,cd055ab9-6680-4af7-94d9-35835355f08c,1580,1590,1590,0.0,0.0
86 | 84,35473529-30cf-4aaa-9aa2-4e59bc30a970,20 November 1996,27 November 1996,27 November 1996<sv>27,0.67,0.0
87 | 85,494a271f-d9cb-4861-8b65-4c9411684b5b,1996,1997,1977<sv>1997,0.0,0.0
88 | 86,8b405cec-665e-4f11-b3b4-698e6ea3b69c,2008,1980,1980<sv>1980,0.0,0.0
89 | 87,800d3b41-91bf-4af2-b460-7d92437433c2,2006,2000,2000<sv>2000,0.0,0.0
90 | 88,923c1673-9a62-4bef-9c45-6048d8f5ad09,1900,1950,1950<sv>1950,0.0,0.0
91 | 89,fd9f4aa9-62e0-4896-a03a-ac452679fac7,2014,2012,2012<sv>2012,0.0,0.0
92 | 


--------------------------------------------------------------------------------
/error_analysis/val_exp_2_stage_2_4_e60_drop/eval_reports_valid_set/ea_date_sample_90.csv:
--------------------------------------------------------------------------------
 1 | ,query_id,prediction,answer,validated_answers,f1,exact_match
 2 | 0,4ef57f34-8c7c-4500-bc1f-2d2512e35523,1944,1943,1943<sv>1943<sv>1943<sv>1943,0.0,0.0
 3 | 1,0903294b-08fc-4061-937a-44890a55ca78,1944,1943,1943<sv>1943<sv>1943,0.0,0.0
 4 | 2,7c72066b-f148-4554-8b81-4d74fbe61101,1944,1943,1942<sv>1943,0.0,0.0
 5 | 3,45c95f84-92e4-4d83-b426-894687613da0,1683,1698,1698<sv>1698,0.0,0.0
 6 | 4,a76d9d78-2e31-49d9-9562-2abc3f758943,2002,2000,2000<sv>2000<sv>2000,0.0,0.0
 7 | 5,381ef468-9cc8-4b70-98b2-eab5183952b3,May 1752,April 1752,May 1754<sv>1752<sv>May 1754,0.67,0.0
 8 | 6,8fd1e5f3-e0dc-4996-aeb4-63f90f76e771,13 August 1563,August 1563,August<sv>August 1563,0.8,0.0
 9 | 7,cca408e3-6075-430b-9e58-b820bb6aca06,13 August 1563,August 1563,August 1563<sv>August 1563<sv>August 1563,0.8,0.0
10 | 8,b44a05fc-4696-402f-a313-fb5236ac0ab6,4 September 1563,1 September 1563,August 1563<sv>4 September,0.8,0.0
11 | 9,8cbc1e9a-7f0c-4a32-8089-658398a9a1f0,1442,1438,1438<sv>1438,0.0,0.0
12 | 10,59b2070a-4027-49d5-b797-2ec7befcf2c4,1440,1452,1452<sv>1452,0.0,0.0
13 | 11,9fdfd076-2163-448e-b67d-1ad392ebfc5d,1440,1452,1452<sv>1452<sv>1552,0.0,0.0
14 | 12,1aa99f9e-5291-4e8e-9be5-8016fee01767,1675,8 August 1675,August 1675<sv>August 1675,0.67,0.0
15 | 13,79bacdf4-b669-4e2a-9be3-e73ab82a9467,1980,1955,1955<sv>1955,0.0,0.0
16 | 14,2f2a8ca0-199f-433a-a64c-d65b03c7ddd5,1980,1955,1955<sv>1955,0.0,0.0
17 | 15,e7d28a55-3aab-4313-9b65-ca20c329e166,In the next year,1277,1277<sv>1277,0.0,0.0
18 | 16,9b6d8136-2535-448c-97b2-708ebe73b75c,1276,1277,1277<sv>1277,0.0,0.0
19 | 17,167c68ad-6ecd-4915-a795-f5f91441c413,1990,2010,2010<sv>2010,0.0,0.0
20 | 18,6c6f5cbe-c01c-46f0-aad5-df16aa478aa5,1990,1981,1981<sv>1981<sv>1981<sv>1981,0.0,0.0
21 | 19,c68e2b2d-a1da-41c6-9866-7cce53b70791,1882,1874,1874<sv>1874,0.0,0.0
22 | 20,6580ed72-f11b-45b5-a890-1e8e3860afc3,1699,1606,1606<sv>1606,0.0,0.0
23 | 21,45be16ff-c43e-412b-a6f2-6439e38a12c8,1583,1593,1593<sv>1593,0.0,0.0
24 | 22,d7c261cc-b929-4a1c-81b8-55a4d5f6fdbb,1553,1525,1525<sv>1525<sv>1525,0.0,0.0
25 | 23,ff20ab78-6d2c-4238-919e-34c40b4c9194,10 September,10 October 1556,29 September 1556<sv>October 10,0.5,0.0
26 | 24,3a2cd28a-4deb-4715-9225-ab53e78d11d7,1446,1445,1477<sv>1445<sv>1445,0.0,0.0
27 | 25,dc5a38ba-47c4-4ff3-882f-0854b724928e,1418,1488,1488<sv>1488,0.0,0.0
28 | 26,c01c2890-80fc-4dd3-9073-533f7d2de8a8,28 April,30 April,27 April<sv>27,0.0,0.0
29 | 27,686e10d5-95bd-4b50-b2a8-72dfe6439f2f,2009,2007,2007<sv>2007,0.0,0.0
30 | 28,6ca5606a-706a-4325-86f8-e6b69b0575b6,1724,1722,1729,0.0,0.0
31 | 29,d2bb1b96-b931-4e1a-ab99-092d3bd0af52,February,1897,1896<sv>1897,0.0,0.0
32 | 30,e444e95d-e3e0-47b0-a5a8-38f99430f3cc,May,June,June 1272<sv>June,0.0,0.0
33 | 31,d3af4f76-8f9e-4199-84da-6956b4443279,May,June 1272,June<sv>June 1272,0.0,0.0
34 | 32,fd2ead2e-e669-409a-9dbf-700c838d8c15,1683,1698,1698<sv>1698,0.0,0.0
35 | 33,6629ae4d-99a4-4786-a8e8-4c0d723dc231,1683,1698,1698<sv>1698,0.0,0.0
36 | 34,2d60e681-7ca9-41a2-956b-1233fde15719,2003,2002,2002<sv>2002,0.0,0.0
37 | 35,f47e8521-c8ad-42f8-b164-02c35afff6c5,32 August 2000,31 April 2000,2000<sv>2000,0.5,0.0
38 | 36,61de672b-6d8d-4b3e-8394-1b73c8fcbc4d,April 1606,April 1605,April 1605<sv>3 March 1606,0.4,0.0
39 | 37,db540314-ad8e-4e87-bed8-d57ae991d26a,April 1606,April 1605,April 1605<sv>1605,0.0,0.0
40 | 38,72407857-72ac-480e-bc2d-0e4ce3781f10,1854,1852,1852<sv>1852,0.0,0.0
41 | 39,1aed8cf7-96a2-4c0d-af61-ff771944db22,The Siege of Nice,1542,Polin succeeding Antoine de Rincon<sv>Polin succeeding Antoine de Rincon,0.0,0.0
42 | 40,af5b5adf-ed88-45a3-bd91-902078145db0,1424,1384,1384<sv>1384,0.0,0.0
43 | 41,8593bfa1-eb03-489e-aa64-1364739265b7,1236,1234,1234<sv>1234,0.0,0.0
44 | 42,acfd906d-5098-4038-a55c-2be27af298e6,9 December 1283,3 December 1283,3 December 1283,0.67,0.0
45 | 43,1173ca51-f21e-4d5a-9092-93caf53eedd4,9 December 1283,3 December 1283,3 December 1283<sv>3 December 1283<sv>3 December 1283,0.67,0.0
46 | 44,6806f0b2-3baa-4919-ac39-b47c407bdbb5,26 January 1285,3 December 1283,3 December 1283<sv>3 December 1283,0.0,0.0
47 | 45,fe6f6f4e-7654-4574-83e7-d32f81c3af13,26 January 1285,3 December 1283,1283<sv>3 December 1283,0.0,0.0
48 | 46,e50b593b-a77a-4b67-8833-2820eb6fc314,26 January 1285,1285,that same year<sv>1285,0.5,0.0
49 | 47,1d703a46-9934-4545-b550-2416997672d1,10,7 October 1580,7 October 1580<sv>10 October 1580<sv>7 October 1580,0.5,0.0
50 | 48,1748ad77-26e5-4a8b-9881-40b7151d99e0,1980,1983,1981<sv>1981<sv>1981,0.0,0.0
51 | 49,6cdb01b3-76f4-4d14-8911-0badd33ac25b,March 1742,February 1743,February 1743<sv>February 1743,0.0,0.0
52 | 50,1b9b6544-d73a-4150-a1b6-68a2ee20b8eb,1920,1919,1919<sv>1919,0.0,0.0
53 | 51,4f47dd36-456b-4991-b2b3-3b3208d176b2,4 October 1910,5 October 1910,5 October 1910<sv>5 October 1910,0.67,0.0
54 | 52,247acee1-f214-46d2-a33a-f55d2ff49b32,1055,1061,1061<sv>1068,0.0,0.0
55 | 53,c323389b-e07c-4235-9ce7-3c2e955397b0,1522,3 February,3 February<sv>3 February,0.0,0.0
56 | 54,72b358b9-8cae-45a2-9b5c-597ae8ad56bf,1614,1615,1615<sv>1615,0.0,0.0
57 | 55,1dd90b41-df53-40fb-9528-847a0f9474b7,1890,May 2016,1899<sv>May 2016<sv>May 2016,0.0,0.0
58 | 56,b13d08ae-53b0-47ce-ab02-954db5fd58b0,1350,1367,1367<sv>1367<sv>1367<sv>1367<sv>1367,0.0,0.0
59 | 57,0d5cfb06-b1e5-45d2-93e2-3742fcf59e80,1350,1367,1367<sv>1367,0.0,0.0
60 | 58,89eb3b39-409c-48ee-8856-c16c1f89cc40,31 August,31 August 1781,31 August 1781<sv>31 August 1781,0.8,0.0
61 | 59,f79aa0fb-2c48-4e9d-ae8a-985a28ebf63b,2004,2010,2010<sv>2010,0.0,0.0
62 | 60,cee9867a-2ed2-40fc-a2f9-aa43117ef764,1908,1904,1904<sv>1904,0.0,0.0
63 | 61,a26a97a8-8ca9-4c2c-b6e4-490d37826f09,1905 <ss> 1906 <ss> 1907,1904,1904<sv>1904,0.0,0.0
64 | 62,c035a076-35c0-440c-b39d-4951027a3cec,1906,1904,1904<sv>1904,0.0,0.0
65 | 63,75afb1a1-c59b-4b60-a7d2-2346f9f2b68a,2011,1990,1990<sv>1990,0.0,0.0
66 | 64,9a403ff7-e467-469b-a86d-d5f891bd8332,2000,1990,1990<sv>1990,0.0,0.0
67 | 65,15b6fad2-00df-41a0-b5e9-929120cfc43d,2016,2015,2015<sv>2015,0.0,0.0
68 | 66,db1dd1a9-a580-4c43-8f12-19da771715c3,2016,2015,2015<sv>2015,0.0,0.0
69 | 67,abef9d7e-48fc-4f4a-a842-707e7be221f6,2012,12000,2000<sv>12000,0.0,0.0
70 | 68,4f02227d-6f5c-4501-8cdc-54a2fab27452,2017,2016,2016<sv>2016,0.0,0.0
71 | 69,6ce9b33b-b86e-4390-8d9d-c2e6e7b353fe,1996,1982,1996-97<sv>1982,0.67,0.0
72 | 70,fa4a1714-eac7-41c7-8377-d539ce068b05,2025,2015,2015<sv>2015,0.0,0.0
73 | 71,585f56ac-5a3e-498d-b21a-1ca8eda1f456,14 November 1989,8 November 1989,1994 November 1989<sv>8 November 1989,0.67,0.0
74 | 72,26a8d265-b7a9-4aa0-9882-0ec3fc146ba5,24 August 2005,14 August 2005,14 August 2005<sv>14 August 2005,0.67,0.0
75 | 73,60bbfb58-b839-4bce-9ea3-8c43c85e262c,2019,2017,last year<sv>2017,0.0,0.0
76 | 74,b419c1ce-abe0-4139-a83b-713cd89f54f9,31 July 1941,19 July 1941,July 1941<sv>19 July 1941,0.8,0.0
77 | 75,e4337ab8-36a3-4195-94f3-62d25300ccbb,31 July 1941,19 July 1941,19 July 1941<sv>19 July 1941,0.67,0.0
78 | 76,3c45e42f-5a1e-414c-9692-9a10f713d541,273 AD,210,210 AD<sv>210,0.0,0.0
79 | 77,703eb661-5524-4d0e-867e-882d842e974c,273,210,210<sv>210,0.0,0.0
80 | 78,a32961ce-e927-4c16-b924-b52c5a7a45c2,1995,2002,2002<sv>2002,0.0,0.0
81 | 79,4789a862-3100-406b-857a-618879434699,1995,1996,1996<sv>1996,0.0,0.0
82 | 80,5abe52c8-45fb-4e3c-97f1-ad9255c8f209,2008,1997,70000<sv>1997,0.0,0.0
83 | 81,445f9cb5-b4b9-4b83-9dac-26ae4d4956dd,29 May 1991,25 April 1991,25 April 1991<sv>25 April 1991,0.33,0.0
84 | 82,7ad86006-5910-4fbc-99c4-ce10a2c71a6e,2011-12,2005,2005<sv>2005,0.0,0.0
85 | 83,cd055ab9-6680-4af7-94d9-35835355f08c,1580,1590,1590,0.0,0.0
86 | 84,35473529-30cf-4aaa-9aa2-4e59bc30a970,20 November 1996,27 November 1996,27 November 1996<sv>27,0.67,0.0
87 | 85,494a271f-d9cb-4861-8b65-4c9411684b5b,1996,1997,1977<sv>1997,0.0,0.0
88 | 86,8b405cec-665e-4f11-b3b4-698e6ea3b69c,2008,1980,1980<sv>1980,0.0,0.0
89 | 87,800d3b41-91bf-4af2-b460-7d92437433c2,2006,2000,2000<sv>2000,0.0,0.0
90 | 88,923c1673-9a62-4bef-9c45-6048d8f5ad09,1900,1950,1950<sv>1950,0.0,0.0
91 | 89,fd9f4aa9-62e0-4896-a03a-ac452679fac7,2014,2012,2012<sv>2012,0.0,0.0
92 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # NT5?! Training T5 to Perform Numerical Reasoning
 2 | 
 3 | **Authors**: Peng-Jian Yang<sup>a</sup>, Ying Ting Chen<sup>a</sup>, Yuechan Chen<sup>a </sup>  
 4 | **Advisor**: Daniel Cer<sup>a, b</sup>   
 5 | <sup>a</sup>University of California Berkeley, <sup>b</sup>Google Research   
 6 | 
 7 | *NT5?! Training T5 to Perform Numerical Reasoning* is a NLP research project on training T5 to perform NRoT (numerical reasoning over text). Latest version of the paper [can be reviewed on ArXiv](https://arxiv.org/abs/2104.07307). All source codes and two fully trained NT5 models (**RC Experiment 1**, our best performing model, and **Validation Experiment 2**, our second best performing model) are included in the repository.  
 8 | 
 9 | ## Abstract
10 | 
11 | Numerical reasoning over text (NRoT) presents unique challenges that are not well addressed by existing pre-training objectives in NLP. We explore five sequential training schedules that adapt a pre-trained T5 model for NRoT. Our final model adapted from T5 but further pre-trained on three datasets designed to strengthen skills necessary for NRoT and general reading comprehension before being fine-tuned on Discrete Reasoning over Text (DROP) dataset. We show that our training improves DROP’s adjusted F1 performance (a numeracy-focused score) **from 45.90 to 70.83**. Our model outperforms the best model in the original DROP paper (47.01), and closes in on GenBERT (72.4), a custom BERT-Base model with significantly more parameters.
12 | 
13 | ## NRoT Challenges   
14 | 
15 | NRoT in NLP is unique in that answers require **numerical reasoning** in addition to the traditional NLP task, **reading comprehension (RC)**. Additionally, answers can demand the model to be both **generative** and **discriminative**, as demonstrated by the two examples extracted from DROP, our gold dataset:  
16 | 
17 | <p align="center"><img src="https://www.dropbox.com/s/abbjlaalmn00ozi/num_example.jpg?raw=1" width="800"/></p>
18 | 
19 | The answer for the **first question** is an extraction from the passage, and requires the model to compute the probability distribution across all words in the passage. In particular, our chosen model requires the following **three NLP skills in sequence**:  
20 | 
21 | <p align="center"><img src="https://www.dropbox.com/s/lq5ldksof5x4vu5/nrot.jpg?raw=1" width="650"/></p>  
22 | 
23 | The answer for the **second question**, on the other hand, cannot be extracted from either the passage or question. We need a **generative language model** to generate the string, "4300000." 
24 | 
25 | <p align="center"><img src="https://www.dropbox.com/s/xg3bhfoffw95lh1/nrot2.jpg?raw=1" width="650"/></p>  
26 | 
27 | Note that many NRoT models, including the current state of the art for solving DROP, **only** generates the mathematical equations required to calculate the final answer as the output. Our research aims to take it **one step further**: Our final model internalizes the equation, perform the calculation, and directly generate the final numerical answer, 4,300,000, as the output.  
28 | 
29 | ## Datasets
30 | 
31 | A total of **6 datasets** are explored during training. The splits and sizes for each dataset are summarized by the diagram below. 
32 | 
33 |   <p align="center"><img src="https://www.dropbox.com/s/w01jfmt0s15zco5/datasets.png?raw=1" width="650" /></p>
34 | 
35 | **DROP** (Discrete Reasoning Over Paragraphs), introduced by AllenNLP in 2019, includes 96k examples in a "Q&A with context" format similar to SQuAD. The benchmark includes four distinct types of questions, all of which require NRoT skills to solve. **DROP Class** is exactly the same as DROP, but with the labels changed to the four classes of questions found in DROP : numeric, date, single span, and multiple spans. The goal of DROP Class is to help T5 learn to classify the four types of questions that require different skillsets to solve in DROP.
36 | 
37 | **Synthetic Data** consists of two datasets: The Numeric dataset (NUM) with near 1M synthetically generated questions on seven types of numerical skills (e.g. addition, sorting, comparison, etc.). The Textual dataset (TXT) builds on NUM, and includes 2M+ synthetically generated examples in formats similar to DROP's Q&As. 
38 | 
39 | **SQuAD v1.1**, a benchmark dataset by Stanford with an emphasis on RC through Q&As, is included in training to strengthen the model's general RC capability. 
40 | 
41 | Unfortunately, we are unable to complete our multitask training with **C4EN** (used a part of T5's pre-training) due to limited resources, but we hypothesize that the inclusion of which would lead to an improved performance.  
42 | 
43 | ## Evaluation
44 | 
45 | We employ two evaluation metrics: **Exact-Match (EM)**, and an **adjusted F1** (macro-averaged, adjusted for numerical performance). EM uses that same criteria as SQuAD. The adjusted F1 has additional logic that invalidates all matching material within an answer when there is a numeric mismatch. In other words, the prediction receives an **F1 score of 0** if it gets the number wrong. In addition, F1 is computed using macro-averaging over individual answers. In the presence of **multiple ground truths**, both EM and F1 will take a max over all computed scores.  
46 | 
47 | ## Model Selection 
48 | 
49 | At the time of research, BERT with self-attention is becoming increasingly popular across a wide variety of NLP tasks. However, inspired by **WT5** ([Sharan et al., 2020](https://arxiv.org/abs/2004.14546) ) and **GenBERT** ([Geva et al., 2020](https://arxiv.org/abs/2004.04487)), we choose **T5** as our model specifically for its following strengths:
50 | 
51 | * **Multitasking, enabled by T5's Text-to-Text Framework** :
52 |   * T5 can concurrently train on multiple datasets with distinct label formats. 
53 |   * One single T5 model can be fine-tuned against multiple objectives and perform different types of predictions. This is a **strong contrast to BERT** and an important feature required by DROP, our gold dataset, as explained in the NRoT challenges section.      
54 | * **Strong RC from transfer learning**: 
55 |   * T5 is Google's attempt to take transfer learning to its limit across a wide verity of NLP tasks. It is pre-trained on Colossal Clean Crawled Corpus (C4).
56 |   * A short description of T5 can be found on the [Google AI Blog](https://tinyurl.com/yg6wsf38) and T5's [original paper](https://arxiv.org/abs/1910.10683). 
57 | 
58 | - **Parsimonious architecture & training process**:
59 |   - T5 allows us to complete the entire training schedule using its **out-of-box architecture**. BERT, on the other hand, requires additional feedforward neural networks for fine-tuning. 
60 |   - T5's multitasking allows us to fine-tune **one single model** for all the different NLP tasks demanded by DROP. In contrast, BERT requires multiple different models to solve DROP.  
61 |   - We hypothesize that T5's pre-training and encoder-decoder architectures would lead to a performance comparable to BERT but with a much **smaller model scale**.
62 | 
63 | ## Training Methodology
64 | 
65 | The parsimony of T5 allows us to **focus on refining our training methods** instead of the model architecture. Our training involves a series of experiments using both sequential and multitask trainings. The full schedule is summarized by the diagram below and a detailed description can be found in the paper. Our best model is **RC1**, the model pertained on the two synthetic datasets and SQuAD, and then fine-tuned on DROP and DROP Class. 
66 | 
67 | <p align="center"><img src="https://www.dropbox.com/s/ujq8dc229nl79bh/schedule.jpeg?raw=1" width="750" /></p>
68 | 
69 | 
70 | 
71 | ## Results
72 | 
73 | Our best model, **RC1**, using T5-Small (the smallest scale of T5) achieves an adjusted F1 performance of **70.83**. This is a considerable improvement over the performance achieved by the model proposed in the original DROP paper (47.01). Our model also closes in on GenBERT (72.4), a custom BERT-Base model pre-trained on our same synthetic data. In addition, our model is a lot more parsimonious: GenBERT's architecture includes 5 additional feedforward neural networks on top of the BERT-Base encoder and comes with **significantly more weights** (110 million from BERT-Base + additional weights from the 5 neural networks vs. 60 million from our T5-Small).    
74 | 
75 | ## Repository 
76 | - [`./nt5_multitask_training.ipynb`](./nt5_multitask_training.ipynb) is the notebook with all source codes for modeling and training.
77 | - [`./models`](./models) includes two fully trained models in h5 format: our best (**RC Experiment 1**) and second best (**Validation Experiment 2**) performing models. 
78 | - [`./error_analysis`](./error_analysis) includes the evaluation reports for the two models included in `./models`.
79 | - `./data` includes the datasets we used for training NT5.
80 | - `./tfrec` includes the source codes used to generate the TFRecord files used for data streaming during training.
81 | 
82 | 


--------------------------------------------------------------------------------
/error_analysis/val_exp_2_stage_2_4_e60_drop/eval_reports_valid_set/ea_sapn_sample_100.csv:
--------------------------------------------------------------------------------
  1 | ,query_id,prediction,answer,validated_answers,f1,exact_match
  2 | 37,7ef82663-f1d5-499f-b779-bb07bd04ed35,Todd Yoder,Michael Turner,Falcons<sv>Michael Turner<sv>Michael Turner<sv>Michael Turner,0.0,0.0
  3 | 223,706150a0-dd65-4495-bb32-e378d0e31411,Roger Clemens,Julio Franco,Julio Franco<sv>Julio Franco,0.0,0.0
  4 | 325,ef051abe-04c2-4953-a720-030c6be42664,gunmen raided a Malaysian fish farm,gunmen raided,Malaysian fish farm,0.75,0.0
  5 | 548,a73b4ca5-9da7-4e7e-a695-9cb86798561a,Browns,Chargers,0,0.0,0.0
  6 | 727,a7e9bf8d-b442-4851-bc9c-2d1edd3c4075,game 1,game 2,game 2<sv>Game 2,0.0,0.0
  7 | 386,7d97814e-34aa-45a1-8aac-cc5844c789e3,Sebastian Janikowski,Kris Brown,Kris Brown<sv>Kris Brown<sv>Kris Brown,0.0,0.0
  8 | 616,da76bb3f-7719-4719-b31b-a14f02761cf0,Yancey Thigpen,Erric Pegram,Neil ODonnell<sv>Neil ODonnell,0.0,0.0
  9 | 688,219964d9-a538-446f-baf4-7cbcdb82e64d,Robert Mathis,Peyton Manning,Peyton Manning<sv>Manning,0.0,0.0
 10 | 928,25cd7f66-4c81-47b0-ac8d-19456c6ac608,80 and older,10 to 29,10 to 29<sv>10 to 29,0.0,0.0
 11 | 844,cffe618e-4bd1-4420-9840-83158e2d202b,49 ers,Niners,Cowboys<sv>Dallas Cowboys,0.0,0.0
 12 | 823,2916cf1b-3a53-451e-9174-8b3fa5c0ab70,Rice,Craig,Craig<sv>Craig,0.0,0.0
 13 | 271,556d070b-5aac-4626-a76d-eb88c7b0f5c8,Tatars,Sheremetev,Sheremetev<sv>Sheremetev,0.0,0.0
 14 | 792,02d53885-3423-4417-a391-99ab78216cf9,Tim Carter,Benjamin Watson,Benjamin Watson<sv>Benjamin Watson,0.0,0.0
 15 | 664,0587a86c-2d15-4308-9804-69724144de13,Derek Anderson,Cleo Lemon,Lemon<sv>Lemon,0.0,0.0
 16 | 751,c78fec38-c81b-4775-a760-ad9cb32f4817,1512,five days,5<sv>five days<sv>five days,0.0,0.0
 17 | 558,2e8ed938-6129-461a-9fee-5e0f21b09f93,vice-president,treasurer,treasurer<sv>treasurer,0.0,0.0
 18 | 801,41184fc5-48b7-4198-9e95-a19d9c20db00,large rowing boats,smaller galleys,smaller galleys<sv>15,0.0,0.0
 19 | 889,5d8deba0-4978-4364-b717-dbf1fee1d16a,Democratic Party,Republican,Republican<sv>Republican,0.0,0.0
 20 | 718,2de31cfa-c0dc-42a6-bdc5-8d13d88bef54,Brett Favre,Matthew Stafford,Matthew Stafford<ss>Brett Favre<sv>Matthew Stafford<ss>Brett Favre,0.5,0.0
 21 | 774,ec87d200-5b4b-4a05-8904-23048c8aafb2,45 and 49 years,under age 18,age 65 years and over<sv>65 years and over<sv>under age 18<sv>65 years and over,0.0,0.0
 22 | 938,7b68915e-a17e-4bd8-b200-2777553a195c,Oaxaca,San Luis Potosí,San Luis Potosí<sv>San Luis Potosí,0.0,0.0
 23 | 878,e67e66c3-5fbe-4d0c-a349-89ecacaae3a8,10 to 14,80 to 89,80 to 89<sv>80 to 89,0.0,0.0
 24 | 594,08b1edc5-3c23-41d9-a07f-bbc6613b2ba1,League of Ireland,Dundalk,Dundalk<sv>Dundalk<sv>Dundalk,0.0,0.0
 25 | 498,c0905966-b3fc-4710-9309-afbc24d57c33,rifles,machine guns,30<sv>machine guns,0.0,0.0
 26 | 489,b79dbdf9-e95e-4626-8597-e52b1a416d95,killed,Somali Airforce,the Somali Airforce<sv>Somali Air-force destroyed,0.0,0.0
 27 | 8,c38c4097-07c4-43a1-9612-460c333860a1,the Parliament,Charles V,Charles V<sv>the Emperor<sv>Charles V<sv>Charles V,0.0,0.0
 28 | 149,1e615315-c426-4590-86e1-fc04c6aaf464,to raise taxes <ss> pay the army,unpaid soldiery,refused to raise taxes<sv>the Sejm again refused to raise taxes and pay the army,0.67,0.0
 29 | 868,3f52a04c-3638-4246-98d3-0221ebb50fe7,10 and 19,90 and older,90 and older<sv>90 and older,0.0,0.0
 30 | 406,a5e32736-1ee5-4257-8c32-c645b28a3836,Hernán Peraza The Elder <ss> Diego Garc ⁇ a de Herrera,Hernán Peraza The Elder,Hernán Peraza<sv>Hernán Peraza The Elder,0.5,0.0
 31 | 836,8a60f9fe-3f98-4a2a-a69e-7b4b3f6f3483,45 to 64,25 to 44,25 to 44<sv>25 to 44,0.0,0.0
 32 | 196,0ced29ed-fe16-4eb1-8bb2-1ca65038cf48,New York,Eagles,Eagles<sv>Eagles,0.0,0.0
 33 | 262,1be6d1bc-f309-4d2c-ba0a-a8f2af4b33d8,Poles <ss> Tatars,"14,000-15,000 Poles and 5,000-6,000 Tatars",Poles<sv>Poles<ss>Tatars<sv>Tatars,0.5,0.0
 34 | 702,e1aa449c-4683-439a-aa70-c9448e54d42a,Drew Brees,Alex Smith,Alex Smith<sv>Alex Smith,0.0,0.0
 35 | 584,acc895d6-b9c5-4a2b-a08c-c8d99532d521,Anthony Becht,Austin Collie,Austin Collie<sv>Austin Collie<sv>Austin Collie,0.0,0.0
 36 | 809,d9c97396-7e82-4bff-94bb-cdc0fd08cac1,Philip Rivers,Drew Brees,Drew Brees<sv>12,0.0,0.0
 37 | 689,8045bb56-015b-4adb-a401-4ffcacf75168,blocked a Seahawks punt,a safety,safety<sv>safety,0.0,0.0
 38 | 443,caaf041f-3c93-447c-a3e5-7ba0f2cc7e9a,Lawrence Tynes,Plaxico Burress,Plaxico Burress<sv>Plaxico Burress,0.0,0.0
 39 | 419,fb83d655-ec4d-4d72-b9ef-a53493ac71a4,Romanian,the Russian Army,Russia<sv>Bolshevik Russian,0.0,0.0
 40 | 829,dc647615-c287-4e86-b5a6-8e56248f5dfb,0.95 % African American,African American,African American<sv>African American,0.8,0.0
 41 | 244,e0803ff8-ea43-455e-8553-579d72efe9eb,45-64,45 to 64,45 to 64,0.8,0.0
 42 | 838,4bfdd7c8-5d9a-48d1-96b0-16c22dcab8b4,under age 18,45 to 64,age 45 to 64<sv>age 45 to 64,0.0,0.0
 43 | 375,30e5c3ce-1253-4c9e-aa35-276e7fe46130,Deputy,1st Earl of Tyrconnell,Earl of Tyrconnell<sv>1st Earl of Tyrconnell<sv>1st Earl of Tyrconnell,0.0,0.0
 44 | 886,6ba1c8f7-435b-419f-a8b6-0b7d11e158e7,Donald Trump,George W. Bush,George W. Bush<sv>George W. Bush,0.0,0.0
 45 | 235,d9665bea-ad89-46ab-9ea7-254d8cdc7195,under the age of 18,45 to 64,45 to 64<sv>45 to 64,0.0,0.0
 46 | 182,1cb7eb32-cfa4-4dc6-8426-1debcb862ea8,Male unemployment continued to be significantly higher <ss> of females,females,males<sv>females,0.18,0.0
 47 | 459,96de8779-c216-415a-8d63-b9cc8593108f,Mart&#237 ;n Gram&#225 ;tica <ss> 31-yard field goal.,Mart&#237;n Gram&#225;tica,Mart&#237;n Gram&#225;tica<sv>Martin Gramatica,0.0,0.0
 48 | 314,01521a1d-a116-4b99-90c0-913d2f7e86be,Bengals,Bears,the Bears<sv>Bears,0.0,0.0
 49 | 721,f984af3a-8871-4382-beb4-3ca08488b9bb,Alexander Hay,MacGill,MacGill<sv>MacGill<sv>MacGill<sv>MacGill<sv>MacGill<sv>MacGill,0.0,0.0
 50 | 739,b7f19a57-0a07-4749-8682-ee326d89a498,Buccaneers,Titans,Tennessee<sv>Titans<sv>Titans<sv>Tennessee<sv>Tennessee<sv>Titans<sv>Tennessee<sv>Titans,0.0,0.0
 51 | 273,238b868a-52a6-456d-aa71-ca4403c5027c,Wallachia <ss> Transylvania <ss> Moldavia,"Wallachia, Transylvania and Moldavia","Wallachia, Transylvania and Moldavia<sv>Wallachia, Transylvania and Moldavia<sv>Wallachia<ss>Transylvania<ss>Moldavia<sv>Wallachia<ss>Transylvania<ss>Moldavia",0.75,0.0
 52 | 388,75b44e05-52df-4116-ba96-474407d360ee,infantry battalions,Tank Battalions,Tank Battalions<sv>Tank Battalions,0.5,0.0
 53 | 474,3d281629-25e3-4ecc-8ff1-872449635f19,"45, 000 Bangladeshis",55%,55%<sv>more than half,0.0,0.0
 54 | 706,0f9b7c37-2dc6-4d39-90e8-c7ab6fab6036,Thailand,global,85 percent<sv>Indonesia,0.0,0.0
 55 | 304,c438c440-274e-4cf2-aafb-520bfb23f40b,Spanish,Acalan,native populations<sv>Acalan<sv>Acalan,0.0,0.0
 56 | 1001,58e7c549-9e67-4925-8431-b0d6c4d6aef2,Greenleaf,Notus,Notus<sv>Notus,0.0,0.0
 57 | 55,2c31b467-8347-440d-974c-173b8f21c6f9,wanted to annex Kuwait,Saud wanted to annex Kuwait,Saud wanted to annex Kuwait,0.89,0.0
 58 | 855,a6b91124-e534-42f3-83b7-400980cbbd97,Polish people,Polish,Polish<sv>Polish,0.67,0.0
 59 | 83,021777cc-48af-4379-9c44-f444dc80e73b,French,Puerto Rican,Puerto Rican<sv>Puerto Rican,0.0,0.0
 60 | 733,cdd6547f-d418-425c-b2cb-b87016e4297a,Marques Colston,Kellen Winslow,Kellen Winslow<sv>Kellen Winslow,0.0,0.0
 61 | 989,0ea99be0-9bc5-496d-8bc6-bb2b43ac949f,Matt Prater,Brown,Josh Brown<sv>Brown,0.0,0.0
 62 | 96,601d2132-a3b2-4178-b080-ec8f11a6283a,White,Race,Native American<sv>Race,0.0,0.0
 63 | 202,bc63c231-a342-48fc-9df2-f2019d271bc2,priests,churches and chapels,churches and chapels<sv>churches and chapels<sv>churches and chapels<sv>churches and chapels<sv>chapels,0.0,0.0
 64 | 358,64c289f7-6aa4-4cea-b286-a918496909b7,Evangelical Lutheran Church,rapid secularisation,rapid secularisation of the Netherlands,0.0,0.0
 65 | 492,68e62ff7-4b18-426e-a27d-4bf8f642a729,Somali National Army,Somali Airforce,Somali Airforce<sv>Somali Airforce,0.4,0.0
 66 | 233,ffaf5282-15b5-4c74-b274-c655096cfd05,65 and older,under the age of 18,under the age of 18<sv>under the age of 18,0.0,0.0
 67 | 9,7a21b6e3-3273-48c9-afc9-b5a968104828,in the town,Pamplona,Pamplona<sv>Pamplona,0.0,0.0
 68 | 613,3b669f59-b2b6-4fc5-ac77-1206cadd4ee5,Andre Hastings,Yancey Thigpen,Yancey Thigpen<sv>Yancey Thigpen,0.0,0.0
 69 | 608,4060a4d3-2bc9-4db8-b6e1-2e302624d529,Tim Tebow,Matt Moore,Matt Moore<sv>Matt Moore,0.0,0.0
 70 | 350,9b2b5519-a8dc-43f1-a4bd-ec3ed9adef46,immediately shock of unexpected humiliation,unexpected humiliation,discontent,0.57,0.0
 71 | 697,7f8ac59b-b589-43e0-ac3b-717c35f6f914,female householder with no husband,female householder,female householder with no husband present<sv>female householder,0.91,0.0
 72 | 898,becd0619-d572-48d5-b039-54f46181b368,49 ers,49ers,49ers<sv>49ers,0.0,0.0
 73 | 552,8203ffaf-cb55-4d28-a010-d933cb0e9c8e,Bangkok,Laos,Lao<sv>africa,0.0,0.0
 74 | 514,30e32471-a199-45a3-a340-d1b087234f2b,Noah Herron,Steven Jackson,Steven Jackson<sv>Steven Jackson,0.0,0.0
 75 | 245,e4d1e980-6b7e-4d2e-ab3b-6a7f63d073cf,males,male,male,0.0,0.0
 76 | 993,81a66eed-e2b3-4eb2-9060-4096ce02ec68,Clinical Dentistry,Physiotherapy,Physiotherapy<sv>Physiotherapy,0.0,0.0
 77 | 242,7b9d968f-cf07-4b56-a1df-4cc68fb9bea4,45 to 64,25 to 44,25 to 44<sv>25 to 44,0.0,0.0
 78 | 563,af628589-4332-40b0-98ac-f689aae03f40,Cardinal Guillaume Court,"Cardinal Guillaume Court, O.Cist.","Cardinal Guillaume Court, O.Cist.<sv>Cardinal Guillaume Court, O.Cist.",0.86,0.0
 79 | 20,003aa246-5f52-4917-8646-ab140934a2ae,Roddy White,Michael Turner,Michael Turner<sv>Michael Turner<sv>Michael Turner<sv>Michael Turner,0.0,0.0
 80 | 356,e29c5ecd-bb5e-4d68-a88d-dfe6d7c0d128,to face Germany,a reform,reform of the Russian military,0.0,0.0
 81 | 501,d99bc5ec-73f2-4fa0-a951-424cdf42b003,Hasselbeck,Matt Hasselbeck,Matt Hasselbeck<sv>Matt Hasselbeck,0.67,0.0
 82 | 796,a91b722b-f872-4027-9885-d39d48f7796c,the purchase of two Donnet-Leveque flying boats,purchase of two Donnet-Leveque,Donnet-Leveque flying boats<sv>purchase two Donnet-Leveque flying boats,0.92,0.0
 83 | 634,e927e3b5-3329-4e7c-96cd-5815a9d28c2c,Nazar Chisto,Leontii Pleshcheyev,Leontii Pleshcheyev<sv>Nazar Chistoy,0.5,0.0
 84 | 890,b7ad25d9-ef82-4a5a-91e8-b93f96659af1,2012,2016,2016<sv>2016,0.0,0.0
 85 | 623,5c0ad12f-3854-43d6-bd02-98d0a85bef0a,Latavius Murray,Matt Forte,Forte<sv>Matt Forte,0.0,0.0
 86 | 610,b7490b9a-3901-49e8-bcf2-0f0e56290740,Dermontti Dawson,Yancey Thigpen,Yancey Thigpen<sv>Yancey Thigpen<sv>Yancey Thigpen,0.0,0.0
 87 | 528,e2e4c788-b06a-4eca-9375-5d60a7f741ac,Titans,Jets,Jets<sv>Jets<sv>Jets<sv>Jets,0.0,0.0
 88 | 731,9fcab4cd-3f2c-4479-839b-a2cfbd691de9,141 all-purpose yards,20-yard,20-yard<sv>20-yard,0.0,0.0
 89 | 990,d729b650-0e09-4c6c-b534-89fbf546f08e,Josh Brown,Prater,Matt Prater<sv>Prater,0.0,0.0
 90 | 48,7f5841d1-74f0-47c7-a5f5-1f9c5ca12d09,Sweden,Lithuania,Lithuania<sv>Lithuania,0.0,0.0
 91 | 374,5351d216-8f79-4a0f-b7b3-fd4053a34d5b,Marion Barber,Barber,Cowboys<sv>Barber,0.67,0.0
 92 | 767,93de4485-9932-48d9-8855-74d8e5461a3c,2010,"952,000 in 1990",1990<sv>1990,0.0,0.0
 93 | 611,9432f147-fed2-4655-8448-11df40f82759,Yancey Thigpen,Neil ODonnell,Neil ODonnell<sv>Neil ODonnell,0.0,0.0
 94 | 143,048dbf55-d585-4f99-a911-781c43e38b92,Malayalam,Kodava Takk,Kodava Takk<sv>Kodava Takk,0.0,0.0
 95 | 118,0dd0cf11-2366-4844-9361-66e7daf258c6,the British,rebels,rebels<sv>the rebels,0.0,0.0
 96 | 812,8ce4996b-16ae-4ed6-b101-e2dcf69bc310,investment,enterprise restructuring,enterprise restructuring<sv>enterprise restructuring<ss>well-functioning capital market,0.0,0.0
 97 | 404,15e84b63-4fcb-4c0b-a7de-d1a130fce77d,60.9,39.1%,39.1<sv>39.1,0.0,0.0
 98 | 972,097d72ee-5615-4cdb-a324-dad475fca91d,1957 to 1961,1953 to 1957,1953 to 1957,0.67,0.0
 99 | 909,cca8a213-10a9-4452-81ff-4a9394d8a719,Ronnie Brown,Faulk,Faulk<sv>Faulk,0.0,0.0
100 | 31,41e34db9-7cc5-4385-8ac4-0f6725288063,1634,1633,4 July<sv>1633,0.0,0.0
101 | 566,630d7794-2524-4add-8849-6988f3425a4b,stickum-type substance,Gorilla Gold Grip Enhancer,Gorilla Gold Grip Enhancer<sv>Gorilla Gold Grip Enhancer,0.0,0.0
102 | 


--------------------------------------------------------------------------------
/error_analysis/rc_exp_1_stage_3_5_e60_drop/eval_reports_valid_set/ea_sapn_sample_100.csv:
--------------------------------------------------------------------------------
  1 | ,query_id,prediction,answer,validated_answers,f1,exact_match
  2 | 276,c665ee1c-197c-438b-9101-9487bb86916d,Asian,Hispanic or Latino origin,Hispanic or Latino<sv>Hispanic or Latino origin,0.0,0.0
  3 | 64,acc160c1-6a5c-4d63-a359-f5fb396a4f74,65 years of age,18 to 24,18 to 24<sv>18 to 24,0.0,0.0
  4 | 174,dee684a8-1c7e-40cf-a332-08608ece5bdd,Rian Lindell,J. P. Losman,Losman<sv>Losman<sv>J. P. Losman,0.0,0.0
  5 | 949,097d72ee-5615-4cdb-a324-dad475fca91d,1957 to 1961,1953 to 1957,1953 to 1957,0.67,0.0
  6 | 775,6911351b-3c3a-45d1-add2-363e45df147d,"Orban was named ""Basilica""",Basilica,cannon designed by Orban<sv>Basilica,0.4,0.0
  7 | 134,f8ba7241-0755-42f9-a5a9-789b138361af,he was assassinated,attempted to reconcile with Poland,reconcile with Poland<sv>reconcile,0.0,0.0
  8 | 504,f7fa0ed9-275a-4936-b379-bb369d4c1f97,Titans,Jets,Jets<sv>Jets<sv>Jets<sv>Baker,0.0,0.0
  9 | 632,9a6351a6-f479-4776-858e-a7a7f6ea8f53,campaigns of 1702 <ss> and 1703 <ss> he spoke with Envoy of Savoy <ss> about possible opportunities in their army,campaigns,Envoy of Savoy<sv>campaigns of 1702 and 1703,0.45,0.0
 10 | 906,ee8a5b27-a384-4f03-af0b-cedb03f95d29,Rye,Sheep,Sheep<sv>Sheep,0.0,0.0
 11 | 823,e15555fe-0a2b-4d0c-833e-886a1b5fa359,2000,2011,2011<sv>2011,0.0,0.0
 12 | 350,9c89d33b-bd0d-4021-8c63-ea86199c32af,45 to 64 year olds,45 to 64,45 to 64<sv>45 to 64,0.75,0.0
 13 | 656,b123f48b-91c8-4350-8a6b-fbf9b8e52ff2,65 years of age,female householder with no husband,female householder with no husband present,0.0,0.0
 14 | 410,64bd2eb7-aa7f-4791-b035-ce8a6135853f,withdrawal of Romanian forces,Romanian forces established control,Romanian forces established control,0.5,0.0
 15 | 576,3c3f3961-8122-4fe5-94e9-652843402f42,1-yard,29-yard,29-yard TD run<sv>29-yard,0.0,0.0
 16 | 265,74ac2f37-3000-4c8a-9112-aa1ef1ab7252,Great Turkish War,Cretan War,Great Turkish War of 1683-99<sv>Cretan War,0.4,0.0
 17 | 302,931a0838-5083-45ed-924f-ead8b6a0bab7,a Pittsburgh challenge,rushing touchdown,Jason Campbell<ss>rushing touchdown<sv>touchdown<sv>a rushing touchdown,0.0,0.0
 18 | 720,1a448acc-7738-4fbf-88c7-11bac242e17a,secondary class,primary classes,lower secondary classes<sv>primary class<sv>primary classes,0.5,0.0
 19 | 342,e29c5ecd-bb5e-4d68-a88d-dfe6d7c0d128,the revolts at home following the war planted seeds,a reform,reform of the Russian military,0.0,0.0
 20 | 588,da76bb3f-7719-4719-b31b-a14f02761cf0,Yancey Thigpen,Erric Pegram,Neil ODonnell<sv>Neil ODonnell,0.0,0.0
 21 | 824,b24bc125-850b-4d5b-ae8f-c768185f2460,public transportation,carpooled,carpool<sv>carpooled,0.0,0.0
 22 | 595,bea1a14c-cfcc-405c-b465-e8e0962ca148,third,second quarter,second<sv>2<sv>second<sv>2<sv>second,0.0,0.0
 23 | 9,7a21b6e3-3273-48c9-afc9-b5a968104828,in the town,Pamplona,Pamplona<sv>Pamplona,0.0,0.0
 24 | 522,88fc0951-1a7f-406f-ac93-14914a926f39,Chargers,Browns,Browns<sv>Browns kicker Phil Dawson<sv>Phil Dawson,0.0,0.0
 25 | 553,65c6c28f-20ef-401b-8fe6-da1935a4f07c,Colts,Texans,Texans<sv>David Car,0.0,0.0
 26 | 310,ef051abe-04c2-4953-a720-030c6be42664,gunmen raided a Malaysian fish farm,gunmen raided,Malaysian fish farm,0.75,0.0
 27 | 168,99024634-a781-4c5e-a0f2-baf70c5f1348,WR Justin Gage,Washington,Washington<sv>Washington,0.0,0.0
 28 | 137,b2a1b19a-68a6-4e16-a834-43b41005990a, ⁇ vitrigaila withdrew to Moldavia,Švitrigaila withdrew to Moldavia,Švitrigaila withdrew to Moldavia<sv>Švitrigaila withdrew to Moldavia,0.67,0.0
 29 | 958,9148f6ee-5e70-4cfe-9d97-fabb2579d80d,James Peele,George,George,0.0,0.0
 30 | 674,4cbba189-3b38-44ad-9456-f2fc33fbdb1c,65 years of age,someone living alone,someone living alone who was 65 years of age or older,0.53,0.0
 31 | 14,a4f5dd52-e312-4730-a248-223a9cf5a76b,Patriots,Bengals,Cincinnati<sv>Bengals<sv>Bengals<sv>Bengals,0.0,0.0
 32 | 108,0f63b69c-28c1-4d88-8e6b-fcff99740ca6,Russian market <ss> separation would disrupt the Finnish financial zone,the huge Russian market,huge Russian market<sv>the huge Russian market,0.33,0.0
 33 | 723,69738a63-9436-4228-ad0b-4c8cc09ca225,northern Albanian mal ⁇ sor ⁇ ,the northern Albanian malësorë,northern Albanian malësorë<sv>the northern Albanian malësorë,0.5,0.0
 34 | 266,0529c2d5-9bea-4643-b850-9292daf57b8f,Great Turkish War,Ottoman-Venetian War,Ottoman-Venetian War,0.33,0.0
 35 | 947,d070b0bb-cf79-4d15-af15-ea62c9fe6812,PV2,bulbar polio,bulbar polio<sv>bulbar polio,0.0,0.0
 36 | 148,f8594739-f74f-46f2-a1d7-77303e0460e5,Kannada language,Konkani,Konkani<sv>Konkani,0.0,0.0
 37 | 358,e0c33628-a347-44ed-a677-819adcd24033,45 to 64,under the age of 18,under the age of 18<sv>under the age of 18,0.0,0.0
 38 | 840,3f52a04c-3638-4246-98d3-0221ebb50fe7,10 and 19,90 and older,90 and older<sv>90 and older,0.0,0.0
 39 | 741,4f00a2b2-124c-4342-96e0-72ba4b36375d,Delhomme,Jake Delhomme,Jake Delhomme<sv>Jake Delhomme,0.67,0.0
 40 | 791,02644f23-a380-4920-8ec4-e15ad624ac34,Oldsmobile <ss> Ford Motor Company <ss> Maxwell automobile,Eugene Richard,Eugene Richard<sv>Eugene Richard,0.0,0.0
 41 | 437,f88ebc47-83bc-4759-bbcf-3ba383a91513,Packers,Lions,Lions<sv>Lions<sv>Lions,0.0,0.0
 42 | 128,20222a55-434f-4fc2-93c4-4b44f2fd228e,the Qing decided to impose a tighter control,a tighter control of Yunnan,tighter control of Yunnan<sv>control of Yunnan,0.4,0.0
 43 | 598,5a281e65-84cc-4d5c-9dfd-2282ac1b5d8d,German forces,a major Soviet offensive,Soviet<sv>Soviet,0.0,0.0
 44 | 263,0ae84168-e40b-4cc7-9860-cfefb02b1333,The Long Turkish War,Thirteen Years' War,The Thirteen Years' War<sv>Thirteen Years' War<sv>Thirteen Years' War<sv>Thirteen Years' War,0.33,0.0
 45 | 91,88b4c5b2-c9cc-46af-b79b-a888766c406f,Henderson,WR Devery Henderson,Devery Henderson<sv>Devery Henderson<sv>Devery Henderson<sv>Devery Henderson<sv>Devery Henderson,0.67,0.0
 46 | 126,5c1c7fd1-4b12-4cad-b2de-0dfacbfe6725,Manco,Atahualpa,Atahualpa<sv>Atahualpa<sv>Atahualpa,0.0,0.0
 47 | 185,ba141618-4caa-48bf-b53f-928dd4049b84,Louis II,Frederick,Frederick<sv>Frederick<sv>Matthias Corvinus,0.0,0.0
 48 | 821,5d68b7c8-d196-46a4-a89f-13b710fa7400,registered Democratic,Democratic,Democratic<sv>Democratic,0.67,0.0
 49 | 489,1ef0e140-8cf0-4213-82c1-7f9e745bb2bd,65 years of age,18 to 24,18 to 24<sv>18 to 24,0.0,0.0
 50 | 45,e9a17f71-94ee-4031-af88-e9a3e116b575,Josh Brown,Brown,23<sv>Robbie Gould,0.67,0.0
 51 | 296,48a7ff48-5178-431d-a826-7c8dd21362a4,trade unionists <ss> Armenian-Cypriots,Armenian-Cypriots,Armenian-Cypriots<sv>Armenian-Cypriots,0.57,0.0
 52 | 200,f220c2eb-dc1e-4327-bb2a-750bd1320b2f,Akishino,Prince Akishino,Prince Akishino<sv>Prince Akishino<sv>Prince Akishino,0.67,0.0
 53 | 835,6c979fd5-3f58-4181-95fc-7c2f4189cec6,45 to 64 <ss> 65 years of age,from 25 to 44,25 to 44<sv>25 to 44,0.0,0.0
 54 | 241,58b0b58a-e8e9-4528-a223-039f36f217a2,Rif,Rifians,Rifians<sv>Rifians<sv>Rifians<sv>Rifians,0.0,0.0
 55 | 224,d9665bea-ad89-46ab-9ea7-254d8cdc7195,under the age of 18,45 to 64,45 to 64<sv>45 to 64,0.0,0.0
 56 | 837,f9474436-2af8-4e3b-a280-64946420d76e,mobile games,retail games,retail games<sv>retail games,0.5,0.0
 57 | 5,9ea6e8ae-b36a-4b2e-9fee-5114fe3a4093,began a new stage in which they scattered even further,a museum for the remembrance,started the Museo<sv>a museum,0.0,0.0
 58 | 751,f90b9803-10f1-4c9f-a7ff-3dda8da5cf35,60 and 64 years,65 years and over,65 years and over<sv>65 years and over<sv>65 years and over<sv>65 years and over,0.0,0.0
 59 | 710,9fcab4cd-3f2c-4479-839b-a2cfbd691de9,141 all-purpose yards,20-yard,20-yard<sv>20-yard,0.0,0.0
 60 | 518,9576b510-783f-425f-bf8c-5d745e929413,Matthew Stafford,Kris Durham,Kris Durham<sv>6,0.0,0.0
 61 | 226,e09edd44-40fa-465a-a1bb-2482c6b66c77,under the age of 18,25 to 44,25 to 44<sv>25 to 44,0.0,0.0
 62 | 924,7f1783b1-a6b0-4620-8644-8bfdc7fa8221,2013,1985,1985<sv>1985,0.0,0.0
 63 | 733,55d313fa-cc3a-44fa-a8b7-353feb096a1e,Peyton Manning,Trevor Siemian,Trevor Siemian<sv>Trevor Siemian,0.0,0.0
 64 | 430,379a0f56-07c2-43a0-8be4-020e000c400b,QB Philip Rivers,Philip Rivers,Philip Rivers<sv>Philip Rivers<sv>Philip Rivers,0.8,0.0
 65 | 431,3ca3f0be-f34c-4c74-be52-d324d678daf8,fourth,second,second<sv>second,0.0,0.0
 66 | 894,aeff18e2-a939-49e2-8886-a1d551bd2dbb,Bulgaria,Turkey,turkey<sv>Turkey,0.0,0.0
 67 | 198,e116a51c-e22f-4f42-a37c-2019ae7d6bd6,his appointment as the archbishop,the Metropolitan of Helsinki,Metropolitan of Helsinki<sv>Metropolitan of Helsinki<sv>Metropolitan of Helsinki,0.0,0.0
 68 | 730,92dcbc50-27a4-4875-82c2-9ed4c57a5840,a pouring rain,a field of mud,The city of Brescia<sv>Brescia<sv>field of mud,0.0,0.0
 69 | 192,e5e11477-7b11-4f73-b8c8-2bb7da5c8cdd,Jason Avant,DeSean Jackson,DeSean Jackson<sv>DeSean Jackso,0.0,0.0
 70 | 691,da855b91-32a1-4511-94be-17d473edf712,Mansfeld,Mansfeld troops,6000 on foot<sv>6000,0.67,0.0
 71 | 501,e2e4c788-b06a-4eca-9375-5d60a7f741ac,Titans,Jets,Jets<sv>Jets<sv>Jets<sv>Jets,0.0,0.0
 72 | 538,bb8f794b-705b-4cab-a563-74262bc380bf,under 18 years,had individuals under 1,individuals under 18 years<sv>individuals under 18 years,0.86,0.0
 73 | 140,5cd1eb46-1788-4f87-899d-a2953fe19995,Rob Bironas <ss> 49-yard field goal,Bironas,Bironas<sv>Bironas<sv>Bironas<sv>Bironas<sv>Bironas<sv>Bironas,0.25,0.0
 74 | 982,79d4ed26-291a-4857-892f-67cf44abc16f,Multi-family residential homes,three family homes,three-family homes<sv>three family homes,0.57,0.0
 75 | 332,d139e832-8076-47cb-bbd7-6ae38b9efc73,2004-05 season,2005-06 season,2005-06<sv>2005–06,0.0,0.0
 76 | 322,044d83bf-2cd2-4285-b6a5-fcb78394c643,"the French, English, and Dutch",Spanish and Portuguese,Spanish and Portuguese<sv>Spanish<ss>Portuguese,0.29,0.0
 77 | 822,7d6b785e-12c8-4ebf-b1dd-a9b8bceebb27,registered Democratic,Democratic,Democratic<sv>Democratic,0.67,0.0
 78 | 281,3fad6fbf-f3a1-473c-924b-8de53b48a768,the patrol area,enforce an international arms embargo,enforcement of an arms embargo?<sv>enforcement of the arms embargo<sv>allotted the patrol area,0.8,0.0
 79 | 960,7c810e73-0089-45a3-aa84-fb94fca24439,Nintendos 36,Atari 7800,Atari<sv>Atari 7800,0.0,0.0
 80 | 753,37b30b0d-db14-40c6-80d4-626d03ec0d8b,20 to 29,10 to 19,10 to 19<sv>10 to 19,0.0,0.0
 81 | 543,ade55d7d-b1c4-4085-bc0c-0f81971f42ed,under 18 years,under the age of 18,had individuals under 18 years,0.75,0.0
 82 | 622,b24cbc34-ee42-41a9-b93a-30ae6db630f2,14-yard,28-yard,28<sv>28-yard<sv>28-yard<sv>28<sv>28,0.0,0.0
 83 | 360,5351d216-8f79-4a0f-b7b3-fd4053a34d5b,Marion Barber,Barber,Cowboys<sv>Barber,0.67,0.0
 84 | 684,0f9b7c37-2dc6-4d39-90e8-c7ab6fab6036,South Thailand,global,85 percent<sv>Indonesia,0.0,0.0
 85 | 336,3d39df5b-dc6e-4124-86f4-1cd72789f9f1,2008-08 FA Cup,2005-06,2006-07<sv>2004-05 season,0.0,0.0
 86 | 196,6a140a4e-a62b-4c84-b8bc-64337df1f262,Troop H/10 th Air Cavalry,Troop H/10th Air Cavalry,H Troop<sv>Troop H/10th Air Cavalry,0.67,0.0
 87 | 381,f1613fa6-cb64-4944-a9a4-de083b260362,2,22,22<sv>22<sv>22<sv>22,0.0,0.0
 88 | 328,688c05f6-5906-4a65-9c7a-9a510bf15408,2008-08 FA Cup,2005–06,2005–06<sv>2005–06<sv>2005–06,0.0,0.0
 89 | 552,acc895d6-b9c5-4a2b-a08c-c8d99532d521,Anthony Becht,Austin Collie,Austin Collie<sv>Austin Collie<sv>Austin Collie,0.0,0.0
 90 | 457,1bea634c-25de-4349-9a93-bb80f12416af,the Mujaheddin,the Hmong tribe,Hmong tribe<sv>Afghanistan,0.0,0.0
 91 | 243,4d66bab8-8b88-4ece-ab58-dd212efd7c07,Spanish,The French,French<sv>French,0.0,0.0
 92 | 511,10ae6922-1c8d-4a74-b236-6b42dd793489,Minkhaung II,lord of Yamethin,Toungoo<sv>lord of Yamethin<sv>lord of Yamethin,0.0,0.0
 93 | 416,0288f49d-4c5a-4df4-947b-fb97b047ae54,"Treaty of Karlowitz <ss> signed in January 1699 <ss> Treaty of Karlowitz <ss> The Treaty of Karlowitz, <ss> n 1715 a swift Ottoman campaign reclaimed the Morea.",Treaty of Karlowitz,Treaty of Karlowitz<sv>The Treaty of Karlowitz<sv>Treaty of Karlowitz,0.33,0.0
 94 | 300,679c015d-a136-487d-94e1-85410233ca5f,The imperial inspector,reiterated the emperor's wishes,Zhao Wenhua<sv>emperor,0.0,0.0
 95 | 179,eb29dd74-5bee-48e5-b49b-baf3eb79e3b3,hispanic,white,white<sv>white,0.0,0.0
 96 | 15,a5a47d9f-e175-485c-8cc2-b0665e05ec53,officers,non-commissioned officers,non-commissioned officers<sv>non-commissioned officers<sv>non-commissioned officers<sv>non-commissioned officers,0.5,0.0
 97 | 845,a98e674c-053e-4c52-905c-6b00ea2b9498,5,9,9,0.0,0.0
 98 | 400,6aa3b033-a793-449e-ba1c-a42ef921d62a,1950,1960,1960<sv>1960<sv>1960,0.0,0.0
 99 | 705,1f0a030e-25f0-4b0c-9fbc-f446569effd6,New England,Maine,Maine<sv>Maine<sv>Maine,0.0,0.0
100 | 953,f0eda175-7694-4013-a48e-d28478c5c5ee,information industry,he insurance or financial industry,insurance or financial industry<sv>he insurance or financial industry,0.33,0.0
101 | 662,6d29490d-6a58-437c-8b79-31aa555e3b12,Andrew Luck,Peyton Manning,Peyton Manning,0.0,0.0
102 | 


--------------------------------------------------------------------------------
/tfrec/tfrec_synthetic_data.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "id": "GBPhve_C3Hqj"
  7 |    },
  8 |    "source": [
  9 |     "# Preprocessing\n",
 10 |     "\n",
 11 |     "This notebook preprocesses the synthetic numerical and textual datasets in JSON format to a TFRecords dataset that can be fed into the model after batching for training."
 12 |    ]
 13 |   },
 14 |   {
 15 |    "cell_type": "code",
 16 |    "execution_count": 1,
 17 |    "metadata": {
 18 |     "id": "Nfp8x5hOmrPI"
 19 |    },
 20 |    "outputs": [],
 21 |    "source": [
 22 |     "if 'colab' in str(get_ipython()):\n",
 23 |     "    import google.colab as colab\n",
 24 |     "    colab.drive.mount('/content/gdrive') # mount google drive\n",
 25 |     "\n",
 26 |     "# install libraries not native to colab\n",
 27 |     "!pip install tensorflow-text\n",
 28 |     "!pip install transformers==3.3.1\n",
 29 |     "!pip install datasets==1.1.2\n",
 30 |     "!pip install tqdm\n",
 31 |     "!pip install tensorflow_datasets\n",
 32 |     "\n",
 33 |     "# remove pip install outputs\n",
 34 |     "from IPython.display import clear_output\n",
 35 |     "clear_output()"
 36 |    ]
 37 |   },
 38 |   {
 39 |    "cell_type": "code",
 40 |    "execution_count": 3,
 41 |    "metadata": {
 42 |     "id": "9PJz-2sqMIFR"
 43 |    },
 44 |    "outputs": [],
 45 |    "source": [
 46 |     "# # For accessing data from google storage (gs://)\n",
 47 |     "# from google.colab import auth\n",
 48 |     "# auth.authenticate_user()\n",
 49 |     "\n",
 50 |     "from IPython.display import clear_output\n",
 51 |     "clear_output()"
 52 |    ]
 53 |   },
 54 |   {
 55 |    "cell_type": "code",
 56 |    "execution_count": 4,
 57 |    "metadata": {
 58 |     "colab": {
 59 |      "base_uri": "https://localhost:8080/",
 60 |      "height": 35
 61 |     },
 62 |     "id": "Ls1fXWF3sLs1",
 63 |     "outputId": "0c8c0921-202d-42e7-e8d4-c95d238a5f8d"
 64 |    },
 65 |    "outputs": [
 66 |     {
 67 |      "name": "stdout",
 68 |      "output_type": "stream",
 69 |      "text": [
 70 |       "TensorFlow 2.3.1\n"
 71 |      ]
 72 |     }
 73 |    ],
 74 |    "source": [
 75 |     "# ml libraries\n",
 76 |     "import tensorflow as tf\n",
 77 |     "import tensorflow_text as tf_text\n",
 78 |     "# import tensorflow.keras as keras\n",
 79 |     "import tensorflow.keras.backend as K\n",
 80 |     "import tensorflow_datasets as tfds\n",
 81 |     "import transformers\n",
 82 |     "import datasets # https://huggingface.co/docs/datasets/\n",
 83 |     "\n",
 84 |     "# data processing libraries\n",
 85 |     "import numpy as np\n",
 86 |     "import pandas as pd\n",
 87 |     "import matplotlib.pyplot as plt\n",
 88 |     "from tqdm.auto import tqdm\n",
 89 |     "\n",
 90 |     "# other libraries\n",
 91 |     "import os\n",
 92 |     "import json\n",
 93 |     "import functools\n",
 94 |     "import time\n",
 95 |     "import warnings\n",
 96 |     "from datetime import datetime\n",
 97 |     "\n",
 98 |     "print(f'TensorFlow {tf.__version__}')"
 99 |    ]
100 |   },
101 |   {
102 |    "cell_type": "code",
103 |    "execution_count": 12,
104 |    "metadata": {
105 |     "id": "l1NeuclbwOB0"
106 |    },
107 |    "outputs": [],
108 |    "source": [
109 |     "#@title Constants\n",
110 |     "direc = '' # path on gs with parsed synthetic data\n",
111 |     "TRAIN_DATA = 'numeric_train_data_parsed.json' # or textual data: TEXTUAL_textual_train_data_parsed_v2.json\n",
112 |     "DEV_DATA = 'numeric_dev_data_parsed.json' # or textual data: TEXTUAL_textual_dev_data_parsed_v2.json\n",
113 |     "TRAIN_DATA_PATH = os.path.join(direc, TRAIN_DATA)\n",
114 |     "DEV_DATA_PATH = os.path.join(direc, DEV_DATA)\n",
115 |     "\n",
116 |     "T5_MODEL = 't5-small'\n",
117 |     "ENCODER_MAX_LEN = 512\n",
118 |     "DECODER_MAX_LEN = 54\n",
119 |     "TOKENIZER = transformers.AutoTokenizer.from_pretrained(T5_MODEL)"
120 |    ]
121 |   },
122 |   {
123 |    "cell_type": "code",
124 |    "execution_count": 14,
125 |    "metadata": {
126 |     "id": "_n2JVO7M7ZkI"
127 |    },
128 |    "outputs": [],
129 |    "source": [
130 |     "file_path = TRAIN_DATA\n",
131 |     "with open(file_path) as json_file:\n",
132 |     "    data_json_dict = json.load(json_file)  # dict of dict."
133 |    ]
134 |   },
135 |   {
136 |    "cell_type": "code",
137 |    "execution_count": 18,
138 |    "metadata": {
139 |     "cellView": "both",
140 |     "id": "HvtsIwtXnocn"
141 |    },
142 |    "outputs": [],
143 |    "source": [
144 |     "#@title Define functions: encode(), prep_tf_dataset()\n",
145 |     "\n",
146 |     "def encode(\n",
147 |     "        example: dict,\n",
148 |     "        encoder_max_len: int = ENCODER_MAX_LEN, \n",
149 |     "        decoder_max_len: int = DECODER_MAX_LEN,\n",
150 |     "        tokenizer: transformers.PreTrainedTokenizer = TOKENIZER) -> dict:\n",
151 |     "    \"\"\"Tokenize data.\n",
152 |     "    Args:\n",
153 |     "        example (dict): Raw dict parsed from DROP json:\n",
154 |     "                            example['context']\n",
155 |     "                            example['question']\n",
156 |     "                            example['answer']\n",
157 |     "    Returns: \n",
158 |     "        (dict) Dictionary with values tokenized:\n",
159 |     "                            return['input_ids']\n",
160 |     "                            return['attention_mask']\n",
161 |     "                            return['labels']\n",
162 |     "                            return['decoder_attention_mask']\n",
163 |     "    \"\"\"            \n",
164 |     "    context = example['context']\n",
165 |     "    question = example['question']\n",
166 |     "    answer = example['answer']\n",
167 |     "  \n",
168 |     "    # TODO: this representation is not good - better to separately embed\n",
169 |     "    # question and context.\n",
170 |     "    question_plus = f\"answer_me: {str(question)}\" #  context: {str(context)}\"\n",
171 |     "    answer_plus = str(answer)\n",
172 |     "\n",
173 |     "    encoder_inputs = tokenizer(\n",
174 |     "                            question_plus, \n",
175 |     "                            truncation=True, \n",
176 |     "                            return_tensors='tf', \n",
177 |     "                            max_length=encoder_max_len,\n",
178 |     "                            # padding='longest',\n",
179 |     "                            pad_to_max_length=True)\n",
180 |     "    decoder_inputs = tokenizer(\n",
181 |     "                            answer_plus, \n",
182 |     "                            truncation=True, \n",
183 |     "                            return_tensors='tf', \n",
184 |     "                            max_length=decoder_max_len,\n",
185 |     "                            # padding='longest',\n",
186 |     "                            pad_to_max_length=True)\n",
187 |     "    # https://tinyurl.com/y2yh56gp\n",
188 |     "    # input_ids – Token ids to be fed to the encoder.\n",
189 |     "    # attention_mask – Specifying which tokens are allowed to be attended by encoder.\n",
190 |     "    # decoder_input_ids – Token ids to be fed to the decoder.\n",
191 |     "    # decoder_attention_mask – Specifying which tokens are allowed to be attended by \n",
192 |     "    #      decoder. Note that this is NOT the mask for the transformer decoder's \n",
193 |     "    #      self-attention mechanism (i.e. not casual attention). The mask here is \n",
194 |     "    #      simply telling which tokens are allowed to be attended (e.g. after padding is added).\n",
195 |     "    # [0] convert to rank 1 array (e.g. (1, 250) => (250,))\n",
196 |     "    return {\n",
197 |     "        'input_ids': encoder_inputs['input_ids'][0], \n",
198 |     "        'attention_mask': encoder_inputs['attention_mask'][0], \n",
199 |     "        'labels': decoder_inputs['input_ids'][0], \n",
200 |     "        'decoder_attention_mask': decoder_inputs['attention_mask'][0]}\n"
201 |    ]
202 |   },
203 |   {
204 |    "cell_type": "code",
205 |    "execution_count": 19,
206 |    "metadata": {
207 |     "cellView": "both",
208 |     "id": "x9rc72mrwSVd"
209 |    },
210 |    "outputs": [],
211 |    "source": [
212 |     "#@title Define function: load_dataset()\n",
213 |     "# The following functions can be used to convert a value to a type compatible\n",
214 |     "# with tf.train.Example.\n",
215 |     "\n",
216 |     "def _bytes_feature(values):\n",
217 |     "  \"\"\"Returns a bytes_list from a list of string / byte.\"\"\"\n",
218 |     "  return tf.train.Feature(bytes_list=tf.train.BytesList(value=values))\n",
219 |     "\n",
220 |     "def _float_feature(values):\n",
221 |     "  \"\"\"Returns a float_list from a list of float / double.\"\"\"\n",
222 |     "  return tf.train.Feature(float_list=tf.train.FloatList(value=values))\n",
223 |     "\n",
224 |     "def _int64_feature(values):\n",
225 |     "    \"\"\"Returns an int64_list from a list of bool / enum / int / uint.\"\"\"\n",
226 |     "    try:\n",
227 |     "        return tf.train.Feature(int64_list=tf.train.Int64List(value=values))\n",
228 |     "    except Exception as e:\n",
229 |     "        print(f'failed at: {values}')\n",
230 |     "        raise e\n",
231 |     "\n",
232 |     "\n",
233 |     "def to_serialized_example(encoded_example):\n",
234 |     "    feature = {\n",
235 |     "        'input_ids': _int64_feature(encoded_example['input_ids']),\n",
236 |     "        'attention_mask': _int64_feature(encoded_example['attention_mask']),\n",
237 |     "        'labels': _int64_feature(encoded_example['labels']),\n",
238 |     "        'decoder_attention_mask': _int64_feature(encoded_example['decoder_attention_mask']),\n",
239 |     "    }\n",
240 |     "    example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n",
241 |     "    return example_proto.SerializeToString()\n",
242 |     "\n",
243 |     "\n",
244 |     "def make_data_generator(file_path):\n",
245 |     "    \"\"\"\n",
246 |     "    Returns a generator function.\n",
247 |     "    \"\"\"\n",
248 |     "    # with open(file_path) as json_file:\n",
249 |     "    with open(file_path) as json_file:\n",
250 |     "        data_json_dict = json.load(json_file)  # dict of list.\n",
251 |     "\n",
252 |     "        passages = data_json_dict['context']\n",
253 |     "        questions = data_json_dict['questions']\n",
254 |     "        answers = data_json_dict['answers']\n",
255 |     "\n",
256 |     "    def gen():\n",
257 |     "        for passage, question, answer in zip(passages, questions, answers):\n",
258 |     "            encoded_example = encode({\n",
259 |     "                'context': passage,\n",
260 |     "                'question': question,\n",
261 |     "                'answer': answer,\n",
262 |     "            })\n",
263 |     "            serialized_example = to_serialized_example(encoded_example)\n",
264 |     "            yield serialized_example\n",
265 |     "    \n",
266 |     "    return gen\n",
267 |     "\n",
268 |     "\n",
269 |     "def get_num_examples(file_path):\n",
270 |     "    with open(file_path) as json_file:\n",
271 |     "        data_json_dict = json.load(json_file)  # dict of dict.\n",
272 |     "        return len(data_json_dict)\n",
273 |     "\n",
274 |     "\n",
275 |     "def load_dataset(file_path):\n",
276 |     "    # num_examples = get_num_examples(file_path)\n",
277 |     "    dataset = tf.data.Dataset.from_generator(\n",
278 |     "        make_data_generator(file_path),\n",
279 |     "        output_types=tf.string\n",
280 |     "    )\n",
281 |     "    return dataset"
282 |    ]
283 |   },
284 |   {
285 |    "cell_type": "code",
286 |    "execution_count": null,
287 |    "metadata": {
288 |     "id": "hpkWCbQBaUwR"
289 |    },
290 |    "outputs": [],
291 |    "source": [
292 |     "def write_json_to_tf_record_file(file_path):\n",
293 |     "    dataset = load_dataset(file_path)\n",
294 |     "\n",
295 |     "    tf_record_file_path = os.path.join(\n",
296 |     "        os.path.dirname(file_path),\n",
297 |     "        os.path.basename(file_path).replace('.json', '.tfrecord')\n",
298 |     "    )\n",
299 |     "    print(f'Writing TF Record file to: {tf_record_file_path} ...')\n",
300 |     "    writer = tf.data.experimental.TFRecordWriter(tf_record_file_path)\n",
301 |     "    writer.write(dataset)\n",
302 |     "\n",
303 |     "\n",
304 |     "WRITE_JSON_TO_TF_RECORD_FILE = True  #@param { type: \"boolean\" }\n",
305 |     "if WRITE_JSON_TO_TF_RECORD_FILE:\n",
306 |     "    write_json_to_tf_record_file(DEV_DATA_PATH)\n",
307 |     "    write_json_to_tf_record_file(TRAIN_DATA)"
308 |    ]
309 |   }
310 |  ],
311 |  "metadata": {
312 |   "colab": {
313 |    "collapsed_sections": [],
314 |    "name": "synthetic-data-tfrecord.ipynb",
315 |    "provenance": []
316 |   },
317 |   "kernelspec": {
318 |    "display_name": "Python 3",
319 |    "language": "python",
320 |    "name": "python3"
321 |   },
322 |   "language_info": {
323 |    "codemirror_mode": {
324 |     "name": "ipython",
325 |     "version": 3
326 |    },
327 |    "file_extension": ".py",
328 |    "mimetype": "text/x-python",
329 |    "name": "python",
330 |    "nbconvert_exporter": "python",
331 |    "pygments_lexer": "ipython3",
332 |    "version": "3.9.2"
333 |   }
334 |  },
335 |  "nbformat": 4,
336 |  "nbformat_minor": 4
337 | }
338 | 


--------------------------------------------------------------------------------
/tfrec/tfrec_squad.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "id": "GBPhve_C3Hqj"
  7 |    },
  8 |    "source": [
  9 |     "# Preprocessing\n",
 10 |     "\n",
 11 |     "This notebook preprocesses the SQuAD v1.1 dataset in JSON format to a TFRecords dataset that can be fed into the model after batching for training. Refer to `squad_parser.ipynb` for how SQuAD datasets are parsed and saved on Google storage."
 12 |    ]
 13 |   },
 14 |   {
 15 |    "cell_type": "code",
 16 |    "execution_count": 28,
 17 |    "metadata": {
 18 |     "id": "Nfp8x5hOmrPI"
 19 |    },
 20 |    "outputs": [],
 21 |    "source": [
 22 |     "if 'colab' in str(get_ipython()):\n",
 23 |     "    import google.colab as colab\n",
 24 |     "    colab.drive.mount('/content/gdrive') # mount google drive\n",
 25 |     "\n",
 26 |     "# install libraries not native to colab\n",
 27 |     "!pip install tensorflow-text\n",
 28 |     "!pip install transformers==3.3.1\n",
 29 |     "!pip install datasets==1.1.2\n",
 30 |     "!pip install tqdm\n",
 31 |     "\n",
 32 |     "# remove pip install outputs\n",
 33 |     "from IPython.display import clear_output\n",
 34 |     "clear_output()"
 35 |    ]
 36 |   },
 37 |   {
 38 |    "cell_type": "code",
 39 |    "execution_count": 29,
 40 |    "metadata": {
 41 |     "id": "9PJz-2sqMIFR"
 42 |    },
 43 |    "outputs": [],
 44 |    "source": [
 45 |     "# # For accessing data from google storage (gs://)\n",
 46 |     "# from google.colab import auth\n",
 47 |     "# auth.authenticate_user()\n",
 48 |     "!pip install tensorflow_datasets\n",
 49 |     "from IPython.display import clear_output\n",
 50 |     "clear_output()"
 51 |    ]
 52 |   },
 53 |   {
 54 |    "cell_type": "code",
 55 |    "execution_count": 30,
 56 |    "metadata": {
 57 |     "colab": {
 58 |      "base_uri": "https://localhost:8080/"
 59 |     },
 60 |     "id": "Ls1fXWF3sLs1",
 61 |     "outputId": "b8da14d0-7722-4e20-fcd7-309b9d873270",
 62 |     "tags": []
 63 |    },
 64 |    "outputs": [
 65 |     {
 66 |      "name": "stdout",
 67 |      "output_type": "stream",
 68 |      "text": [
 69 |       "TensorFlow 2.3.0\n"
 70 |      ]
 71 |     }
 72 |    ],
 73 |    "source": [
 74 |     "# ml libraries\n",
 75 |     "import tensorflow as tf\n",
 76 |     "import tensorflow_text as tf_text\n",
 77 |     "# import tensorflow.keras as keras\n",
 78 |     "import tensorflow.keras.backend as K\n",
 79 |     "import tensorflow_datasets as tfds\n",
 80 |     "import transformers\n",
 81 |     "import datasets # https://huggingface.co/docs/datasets/\n",
 82 |     "\n",
 83 |     "# data processing libraries\n",
 84 |     "import numpy as np\n",
 85 |     "import pandas as pd\n",
 86 |     "import matplotlib.pyplot as plt\n",
 87 |     "from tqdm.auto import tqdm\n",
 88 |     "\n",
 89 |     "# other libraries\n",
 90 |     "import os\n",
 91 |     "import json\n",
 92 |     "import functools\n",
 93 |     "import time\n",
 94 |     "import warnings\n",
 95 |     "from datetime import datetime\n",
 96 |     "\n",
 97 |     "print(f'TensorFlow {tf.__version__}')"
 98 |    ]
 99 |   },
100 |   {
101 |    "cell_type": "code",
102 |    "execution_count": 31,
103 |    "metadata": {
104 |     "id": "l1NeuclbwOB0"
105 |    },
106 |    "outputs": [],
107 |    "source": [
108 |     "#@title Constants\n",
109 |     "\n",
110 |     "#@title define functin: write_to_gs\n",
111 |     "# authenticate\n",
112 |     "# For accessing data from google storage (gs://)\n",
113 |     "from google.colab import auth\n",
114 |     "auth.authenticate_user()\n",
115 |     "\n",
116 |     "direc = '' # directory on gs containing parsed squad json\n",
117 |     "TRAIN_DATA = \"squad1.1_train_data_parsed.json\"\n",
118 |     "DEV_DATA = \"squad1.1_dev_data_parsed.json\"\n",
119 |     "TRAIN_DATA_PATH = os.path.join(direc, TRAIN_DATA)\n",
120 |     "DEV_DATA_PATH = os.path.join(direc, DEV_DATA)\n",
121 |     "\n",
122 |     "\n",
123 |     "T5_MODEL = 't5-small'\n",
124 |     "ENCODER_MAX_LEN = 512\n",
125 |     "DECODER_MAX_LEN = 54\n",
126 |     "TOKENIZER = transformers.AutoTokenizer.from_pretrained(T5_MODEL)"
127 |    ]
128 |   },
129 |   {
130 |    "cell_type": "code",
131 |    "execution_count": 34,
132 |    "metadata": {
133 |     "id": "2i0IQHwZKhjr"
134 |    },
135 |    "outputs": [],
136 |    "source": [
137 |     "#@title Sample data to file: so it's easy to iterate\n",
138 |     "def sample_data_to_file(file_path, sample_count):\n",
139 |     "    with tf.io.gfile.GFile(file_path) as json_file:\n",
140 |     "        data_json_dict = json.load(json_file)  # dict of dict.\n",
141 |     "\n",
142 |     "        context = data_json_dict['context'][:sample_count]\n",
143 |     "        questions = data_json_dict['questions'][:sample_count]\n",
144 |     "        answers = data_json_dict['answers'][:sample_count]\n",
145 |     "        question_ids = data_json_dict[\"question_ids\"][:sample_count]\n",
146 |     "\n",
147 |     "\n",
148 |     "        sampled_data_json_dict =  {\n",
149 |     "            \"context\": context,\n",
150 |     "            \"questions\": questions,\n",
151 |     "            \"answers\": answers,\n",
152 |     "            \"question_ids\": question_ids,\n",
153 |     "        }\n",
154 |     "\n",
155 |     "    # sample_count = len(sampled_data_json_dict)  # Get the real sample count.\n",
156 |     "    sampled_output_path = os.path.join(\n",
157 |     "        os.path.dirname(file_path),\n",
158 |     "        os.path.basename(file_path).replace('.json', f'{sample_count}.json')\n",
159 |     "    )\n",
160 |     "    with tf.io.gfile.GFile(sampled_output_path, 'w') as fout:\n",
161 |     "        json.dump(sampled_data_json_dict, fout)\n",
162 |     "    print(f'Dumped {sample_count} samples to file: {sampled_output_path}')\n",
163 |     "\n",
164 |     "    return sampled_output_path\n",
165 |     "\n",
166 |     "\n",
167 |     "SAMPLE_DATA_TO_FILE = False  #@param { type: \"boolean\" }\n",
168 |     "SAMPLE_COUNT = 1  #@param { type: \"integer\" }\n",
169 |     "if SAMPLE_DATA_TO_FILE:\n",
170 |     "    TRAIN_DATA_PATH = sample_data_to_file(TRAIN_DATA_PATH, sample_count=SAMPLE_COUNT)\n",
171 |     "    DEV_DATA_PATH = sample_data_to_file(DEV_DATA_PATH, sample_count=SAMPLE_COUNT)\n",
172 |     "\n",
173 |     "    # if want to see the saved sampled_dict\n",
174 |     "    with tf.io.gfile.GFile(TRAIN_DATA_PATH) as json_file:\n",
175 |     "        check_sampled_dict = json.load(json_file)\n",
176 |     "\n",
177 |     "    print(check_sampled_dict)\n",
178 |     "     "
179 |    ]
180 |   },
181 |   {
182 |    "cell_type": "code",
183 |    "execution_count": 36,
184 |    "metadata": {
185 |     "cellView": "both",
186 |     "id": "HvtsIwtXnocn"
187 |    },
188 |    "outputs": [],
189 |    "source": [
190 |     "#@title Define functions: encode(), prep_tf_dataset()\n",
191 |     "def encode(\n",
192 |     "        example: dict,\n",
193 |     "        encoder_max_len: int = ENCODER_MAX_LEN, \n",
194 |     "        decoder_max_len: int = DECODER_MAX_LEN,\n",
195 |     "        tokenizer: transformers.PreTrainedTokenizer = TOKENIZER) -> dict:\n",
196 |     "    \"\"\"Tokenize data.\n",
197 |     "    Args:\n",
198 |     "        example (dict): Raw dict parsed from DROP json:\n",
199 |     "                            example['context']\n",
200 |     "                            example['question']\n",
201 |     "                            example['answer']\n",
202 |     "    Returns: \n",
203 |     "        (dict) Dictionary with values tokenized:\n",
204 |     "                            return['input_ids']\n",
205 |     "                            return['attention_mask']\n",
206 |     "                            return['labels']\n",
207 |     "                            return['decoder_attention_mask']\n",
208 |     "    \"\"\"            \n",
209 |     "    context = example['context']\n",
210 |     "    question = example['question']\n",
211 |     "    answer = example['answer']\n",
212 |     "  \n",
213 |     "    # TODO: this representation is not good - better to separately embed\n",
214 |     "    # question and context.\n",
215 |     "    question_plus = f\"answer_me: {str(question)} context: {str(context)}\"\n",
216 |     "    answer_plus = str(answer)\n",
217 |     "\n",
218 |     "    encoder_inputs = tokenizer(\n",
219 |     "                            question_plus, \n",
220 |     "                            truncation=True, \n",
221 |     "                            return_tensors='tf', \n",
222 |     "                            max_length=encoder_max_len,\n",
223 |     "                            # padding='longest',\n",
224 |     "                            pad_to_max_length=True)\n",
225 |     "    decoder_inputs = tokenizer(\n",
226 |     "                            answer_plus, \n",
227 |     "                            truncation=True, \n",
228 |     "                            return_tensors='tf', \n",
229 |     "                            max_length=decoder_max_len,\n",
230 |     "                            # padding='longest',\n",
231 |     "                            pad_to_max_length=True)\n",
232 |     "    # https://tinyurl.com/y2yh56gp\n",
233 |     "    # input_ids – Token ids to be fed to the encoder.\n",
234 |     "    # attention_mask – Specifying which tokens are allowed to be attended by encoder.\n",
235 |     "    # decoder_input_ids – Token ids to be fed to the decoder.\n",
236 |     "    # decoder_attention_mask – Specifying which tokens are allowed to be attended by \n",
237 |     "    #      decoder. Note that this is NOT the mask for the transformer decoder's \n",
238 |     "    #      self-attention mechanism (i.e. not casual attention). The mask here is \n",
239 |     "    #      simply telling which tokens are allowed to be attended (e.g. after padding is added).\n",
240 |     "    # [0] convert to rank 1 array (e.g. (1, 250) => (250,))\n",
241 |     "    return {\n",
242 |     "        'input_ids': encoder_inputs['input_ids'][0], \n",
243 |     "        'attention_mask': encoder_inputs['attention_mask'][0], \n",
244 |     "        'labels': decoder_inputs['input_ids'][0], \n",
245 |     "        'decoder_attention_mask': decoder_inputs['attention_mask'][0]}"
246 |    ]
247 |   },
248 |   {
249 |    "cell_type": "code",
250 |    "execution_count": 37,
251 |    "metadata": {
252 |     "cellView": "both",
253 |     "id": "x9rc72mrwSVd"
254 |    },
255 |    "outputs": [],
256 |    "source": [
257 |     "#@title Define function: load_dataset()\n",
258 |     "# The following functions can be used to convert a value to a type compatible\n",
259 |     "# with tf.train.Example.\n",
260 |     "\n",
261 |     "def _bytes_feature(values):\n",
262 |     "  \"\"\"Returns a bytes_list from a list of string / byte.\"\"\"\n",
263 |     "  return tf.train.Feature(bytes_list=tf.train.BytesList(value=values))\n",
264 |     "\n",
265 |     "def _float_feature(values):\n",
266 |     "  \"\"\"Returns a float_list from a list of float / double.\"\"\"\n",
267 |     "  return tf.train.Feature(float_list=tf.train.FloatList(value=values))\n",
268 |     "\n",
269 |     "def _int64_feature(values):\n",
270 |     "    \"\"\"Returns an int64_list from a list of bool / enum / int / uint.\"\"\"\n",
271 |     "    try:\n",
272 |     "        return tf.train.Feature(int64_list=tf.train.Int64List(value=values))\n",
273 |     "    except Exception as e:\n",
274 |     "        print(f'failed at: {values}')\n",
275 |     "        raise e\n",
276 |     "\n",
277 |     "def to_serialized_example(encoded_example):\n",
278 |     "    feature = {\n",
279 |     "        'input_ids': _int64_feature(encoded_example['input_ids']),\n",
280 |     "        'attention_mask': _int64_feature(encoded_example['attention_mask']),\n",
281 |     "        'labels': _int64_feature(encoded_example['labels']),\n",
282 |     "        'decoder_attention_mask': _int64_feature(encoded_example['decoder_attention_mask']),\n",
283 |     "    }\n",
284 |     "    example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n",
285 |     "    return example_proto.SerializeToString()\n",
286 |     "\n",
287 |     "def make_data_generator(file_path):\n",
288 |     "    \"\"\"\n",
289 |     "    Returns a generator function.\n",
290 |     "    \"\"\"\n",
291 |     "    \n",
292 |     "    # with open(file_path) as json_file:\n",
293 |     "    with tf.io.gfile.GFile(file_path) as json_file:\n",
294 |     "        data_json_dict = json.load(json_file)  # dict of dict.\n",
295 |     "\n",
296 |     "        passages = data_json_dict['context']\n",
297 |     "        questions = data_json_dict['questions']\n",
298 |     "        answers = data_json_dict['answers']\n",
299 |     "\n",
300 |     "    def gen():\n",
301 |     "        for passage, question, answer in zip(passages, questions, answers):\n",
302 |     "            encoded_example = encode({\n",
303 |     "                'context': passage,\n",
304 |     "                'question': question,\n",
305 |     "                'answer': answer,\n",
306 |     "            })\n",
307 |     "            serialized_example = to_serialized_example(encoded_example)\n",
308 |     "            yield serialized_example\n",
309 |     "    return gen\n",
310 |     "\n",
311 |     "def get_num_examples(file_path):\n",
312 |     "    with open(file_path) as json_file:\n",
313 |     "        data_json_dict = json.load(json_file)  # dict of dict.\n",
314 |     "        return len(data_json_dict)\n",
315 |     "\n",
316 |     "def load_dataset(file_path):\n",
317 |     "    # num_examples = get_num_examples(file_path)\n",
318 |     "    dataset = tf.data.Dataset.from_generator(\n",
319 |     "        make_data_generator(file_path),\n",
320 |     "        output_types=tf.string\n",
321 |     "    )\n",
322 |     "    return dataset"
323 |    ]
324 |   },
325 |   {
326 |    "cell_type": "code",
327 |    "execution_count": null,
328 |    "metadata": {},
329 |    "outputs": [],
330 |    "source": [
331 |     "def write_json_to_tf_record_file(file_path):\n",
332 |     "    dataset = load_dataset(file_path)\n",
333 |     "\n",
334 |     "    tf_record_file_path = os.path.join(\n",
335 |     "        os.path.dirname(file_path),\n",
336 |     "        os.path.basename(file_path).replace('.json', '.tfrecord')\n",
337 |     "    )\n",
338 |     "    print(f'Writing TF Record file to: {tf_record_file_path} ...')\n",
339 |     "    writer = tf.data.experimental.TFRecordWriter(tf_record_file_path)\n",
340 |     "    writer.write(dataset)\n",
341 |     "\n",
342 |     "\n",
343 |     "WRITE_JSON_TO_TF_RECORD_FILE = True  #@param { type: \"boolean\" }\n",
344 |     "if WRITE_JSON_TO_TF_RECORD_FILE:\n",
345 |     "    write_json_to_tf_record_file(DEV_DATA_PATH)\n",
346 |     "    write_json_to_tf_record_file(TRAIN_DATA_PATH)"
347 |    ]
348 |   },
349 |   {
350 |    "cell_type": "code",
351 |    "execution_count": null,
352 |    "metadata": {
353 |     "id": "owOKH0HDUqFI"
354 |    },
355 |    "outputs": [],
356 |    "source": [
357 |     "def print_tfrecord(tfrecrod_file_path, example_count=1):\n",
358 |     "    raw_dataset = tf.data.TFRecordDataset(tfrecord_file_path)\n",
359 |     "    for raw_record in raw_dataset.take(example_count):\n",
360 |     "        example_proto = tf.train.Example.FromString(raw_record.numpy())\n",
361 |     "        print(example_proto)\n",
362 |     "\n",
363 |     "PRINT_TF = False #@param { type: \"boolean\"}\n",
364 |     "tfrecord_file_path = # path to squad tfrecord on gs #@param { type: \"string\" }\n",
365 |     "\n",
366 |     "if PRINT_TF:\n",
367 |     "    print_tfrecord(tfrecord_file_path)"
368 |    ]
369 |   }
370 |  ],
371 |  "metadata": {
372 |   "colab": {
373 |    "collapsed_sections": [],
374 |    "name": "synthetic_data_tfrecord_squad.ipynb",
375 |    "provenance": []
376 |   },
377 |   "kernelspec": {
378 |    "display_name": "Python 3",
379 |    "language": "python",
380 |    "name": "python3"
381 |   },
382 |   "language_info": {
383 |    "codemirror_mode": {
384 |     "name": "ipython",
385 |     "version": 3
386 |    },
387 |    "file_extension": ".py",
388 |    "mimetype": "text/x-python",
389 |    "name": "python",
390 |    "nbconvert_exporter": "python",
391 |    "pygments_lexer": "ipython3",
392 |    "version": "3.9.2"
393 |   }
394 |  },
395 |  "nbformat": 4,
396 |  "nbformat_minor": 4
397 | }
398 | 


--------------------------------------------------------------------------------
/error_analysis/val_exp_2_stage_2_4_e60_drop/eval_reports_valid_set/ea_spans_sample_100.csv:
--------------------------------------------------------------------------------
  1 | ,query_id,prediction,answer,validated_answers,f1,exact_match
  2 | 68,9897b605-c973-4c1e-8a77-769d0ed75e68,Aigina <ss> Romania <ss> Navarino <ss> Achaea,Romania<ss>Laconia<ss>Messenia<ss>Achaea,Romania<ss>Laconia<ss>Messenia<ss>Achaea<sv>Romania<ss>Laconia<ss>Messenia<ss>Achaea,0.5,0.0
  3 | 256,cad81db5-2b73-459b-9cd2-afb1f45de23e,White <ss> African American,White<ss>Latinos,Latinos<ss>Germans,0.5,0.0
  4 | 291,4e1c200c-dd6b-4515-99a6-96e23453a1f4,Barack Obama <ss> John McCain,Donald Trump<ss>George W. Bush,Donald Trump<sv>Donald Trump<ss>George W. Bush,0.0,0.0
  5 | 191,9b00a7f1-c5f1-4c60-926e-d7df131b26e3,10 and 19 years <ss> 25 and 29 years <ss> 40 and 49 years <ss> 60 and 64 years <ss> 60 and 64 years,20 and 24 years<ss>25 and 29 years,20 and 24<ss>25 and 29,0.2,0.0
  6 | 90,23e5f993-8682-40dc-a61e-7d268890daa8,convergence of Protestant hostility to the Habsburgs,convergence of Protestant<ss>hostility to the Habsburgs,Protestant hostility to the Habsburgs<sv>Protestant hostility to the Habsburgs,0.8,0.0
  7 | 25,a2f25486-e034-4532-b28e-e80ce8555c0f,The Qing,The third invasion<ss>penetrating deep into central Burma,The third invasion<sv>The third invasion<sv>third invasion<sv>third<sv>The third<sv>The third invasion,0.0,0.0
  8 | 39,2ca4b84c-1253-4e85-8758-450a633de9b7,Mind Dynasty conquering Yunnan borderlands,"borderlands between 1380 and 1388,<ss>Shan States came in 1557",Ming dynasty,0.29,0.0
  9 | 69,78330072-09f3-4606-8590-ca1900b01093,"Romania <ss> Monemvasia, Laconia <ss> Achaea",Nafplion<ss>Monemvasia<ss>Navarino<ss>Patras,Romania<ss>Laconia<ss>Messenia<ss>Achaea<sv>Nafplion<ss>Monemvasia<ss>Navarino<ss>Patras,0.67,0.0
 10 | 16,db314f06-4dc0-4cbd-a1d7-db195c51977c,United States <ss> Italian,danish<ss>Italian,danish<ss>Italian<sv>danish<ss>Italian,0.5,0.0
 11 | 209,7f483afc-67be-4923-a3c0-711dc4e217a6,Mike Karney <ss> Deuce McAllister,Deuce McAllister<ss>Mark Campbell<ss>Mike Karney,Mark Campbell<ss>Deuce McAllister<ss>Mike Karney<sv>Deuce McAllister<ss>Mike Karney<ss>Mark Campbell,0.67,0.0
 12 | 41,086ef7c9-0cc8-43f0-a41e-e71c46c343ec,Ming dynasty <ss> King Bayinnaung,The Ming dynasty<ss>Toungoo dynasty,Ming dynasty<ss>Toungoo dynasty<sv>Ming<ss>Toungoo,0.5,0.0
 13 | 296,8d0ec4b4-f6b0-4fc2-95b1-402a851f812f,Puerto Rican <ss> Mexican,Puerto Rican<ss>Dominican,Puerto Rican<ss>Dominican<sv>Puerto Rican<ss>Dominican,0.5,0.0
 14 | 147,3a70b771-2f1f-4ba9-a2eb-ad3d21c8c560,Cambodians <ss> Laos <ss> Myanmar,Australia<ss>Africa<ss>Lao,Australia<ss>Africa<sv>Australia<ss>Africa<ss>Lao,0.0,0.0
 15 | 126,26c6ed1a-2b22-44f5-81a6-e7ee99d1e5fb,Indians <ss> Bangladeshis,Pakistanis<ss>Filipinos,Pakistanis<ss>Filipinos<sv>Pakistanis<ss>Filipinos<sv>Pakistani<ss>Filipino,0.0,0.0
 16 | 280,92ae64d4-f6c5-44d9-9fc7-16d39d9f239d,Six-Day War <ss> Nigeria,the Portuguese Colonial War<ss>the Six-Day War by Israel,Portuguese Colonial War<ss>Six-Day War by Israel<sv>Portuguese Colonial War<ss>Six-Day War by Israel,0.37,0.0
 17 | 172,1e221f63-9654-4ba4-98ed-355cca7c401b,Chicago Bears <ss> Minnesota Vikings <ss> Pittsburgh Steelers <ss> Green Bay Packers <ss> New England Patriots,49ers<ss>Bears<ss>New England Patriots,Bears<ss>Patriots<ss>49ers<sv>3,0.33,0.0
 18 | 350,1098b648-abb4-4ab4-be17-2584c2f832ea,2000 and 2002 <ss> 2004,2003<ss>2004<ss>2005,2003<ss>2004<ss>2005<sv>2003<ss>2004<ss>2005,0.33,0.0
 19 | 38,69fd9e90-4fc6-4908-8c99-ac7ba902f577,Italian <ss> united states <ss> Italian <ss> Italian,German<ss>Italian<ss>united states<ss>Italian<ss>Italian,French Canadian<ss>German people<ss>Italian<sv>German<ss>Italian<ss>united states<ss>Italian<ss>Italian,0.8,0.0
 20 | 84,98842c5f-aa44-4372-a501-948765e5a88d,Craig Terrill <ss> Patrick Willis,DT Craig Terrill<ss>LB Patrick Willis,Craig Terrill<sv>Craig Terrill,0.8,0.0
 21 | 76,bbdb1206-a342-4350-b660-c120fbc22c6d,broke the alliance with Moscow,"pay 300,000 rubles<ss>evacuate","pay 300,000 rubles<ss>evacuate Kiev, Pereyaslav, Chernigov<sv>pay 300,000 rubles<ss>evacuate Kiev, Pereyaslav, Chernigov",0.0,0.0
 22 | 269,208eff0e-b7f0-4796-8892-f528bbeb62a5,25 to 44 <ss> 45 to 64,65 years of age<ss>25 to 44<ss>25 to 44<ss>45 to 64,25 to 44<ss>65 years of age<sv>65 years of age<ss>25 to 44<ss>25 to 44<ss>45 to 64,0.5,0.0
 23 | 180,93381eda-fb0b-4a63-9e40-2f0d3ec1c8ba,Grand Prince Iziaslav of Kiev <ss> Princes Sviatoslav Yaroslavich of Chernigov <ss> Vsevolod,Grand Prince Iziaslav of Kiev<ss>Sviatoslav Yaroslavich of Chernigov<ss>Vsevolod,Grand Prince Iziaslav<ss>Sviatoslav Yaroslavich<ss>Vsevolod,0.96,0.0
 24 | 360,d1ef042c-4bf5-4877-b330-dce16939ca18,Melba <ss> Wilder,Notus<ss>Greenleaf<ss>Wilder,Wilder<ss>Star<ss>Melba<ss>Notus<ss>Greenleaf<sv>Notus<ss>Greenleaf<ss>Wilder,0.4,0.0
 25 | 226,e92d782f-9cb1-4cfe-8074-bebd50e9551f,Peyton Hillis,Jerricho Cotchery<ss>Peyton Hillis,Jerricho Cotchery<ss>Peyton Hillis<sv>Jerricho Cotchery<ss>Peyton Hillis,0.5,0.0
 26 | 309,207879d3-9b35-42cd-8de0-1dcb853d95e0,teenagers <ss> adult population,between 40 and 49<ss>between 50 and 59,40 and 49<ss>50 and 59<sv>between 40 and 49<ss>between 50 and 59,0.0,0.0
 27 | 341,96bd2e17-faec-40ee-8018-2d0747a17863,Arabic <ss> Japanese,Japanese<ss>African languages,Japanese<ss>African languages<sv>Japanese<ss>African languages,0.5,0.0
 28 | 142,f91804a4-cef8-4e2c-a589-e26fd45745b5,Chad Simpson getting a 31-yard touchdown run,Jacob Lacey returning an interception<ss>35 yards for a touchdown,45-yard<sv>45-yard field goal<sv>35 yards<sv>45-yard field goal,0.0,0.0
 29 | 53,817609c8-61c2-4bc0-a13a-886537dcf03a,Jaél de Peigne,"November 1566 Killigrew married<ss>Catherine,<ss>November 1590 he was married<ss>Jaél de Peigne,",Catherine<sv>Killigrew getting married to Catherine,0.25,0.0
 30 | 304,6639ffe0-acdd-40cc-a6f5-081c6e09df04,8-yard <ss> 1-yard,8-yard<ss>7-yard,7-yard<ss>8-yard<sv>8-yard<ss>7-yard,0.5,0.0
 31 | 207,09d40815-d551-42db-9096-59378b0190df,Devery Henderson <ss> Antonio Gates,Devery Henderson<ss>Antonio Gates<ss>LaDainian Tomlinson,Devery Henderson<ss>LaDainian Tomlinson<ss>Antonio Gates<sv>LaDainian Tomlinson<ss>Antonio Gates,0.67,0.0
 32 | 335,d2c73134-f543-4f5e-ad3a-2285b64d7155,1569,1553<ss>1569,1553<sv>1553<ss>1569,0.5,0.0
 33 | 121,ed1e3110-ac93-402a-82b4-fdbd574731bb,pro-Free State Blueshirts,IRA<ss>pro-Free State Blueshirts,IRA<sv>the IRA,0.5,0.0
 34 | 299,a308efbb-8383-49c1-ae58-d323c783c649,Cambodian <ss> Filipino <ss> Pakistani,Cambodian<ss>Filipino<ss>Korean<ss>Pakistani<ss>Laotian,0.1% Cambodian<ss>0.1% Filipino<ss>0.1% Korean<ss>0.1% Pakistani<ss>0.1% Laotian<sv>Cambodian<ss>Filipino<ss>Korean<ss>Pakistani<ss>Laotian,0.6,0.0
 35 | 240,0890b283-e99f-4e72-9f01-22e4e2ab6cce,African American <ss> Native American,Asian<ss>Native American,Native American<ss>Asian<sv>Native American<ss>Asian,0.5,0.0
 36 | 82,0edbf486-7037-490f-8e37-a0ec47589ad1,Wokingham <ss> Wokingham <ss> Bracknell Forest <ss> Surrey,Portsmouth<ss>Thanet,Thanet<ss>Portsmouth<sv>Portsmouth<ss>Thanet,0.0,0.0
 37 | 134,acc6fd38-cbe8-4d71-8d3f-abdda57415fc,under the age of 18 <ss> 25 to 44 <ss> 45 to 64,under the age of 18<ss>18 to 24<ss>25 to 44<ss>45 to 64,under the age of 18<ss>from 18 to 24<ss>from 25 to 44<ss>from 45 to 64,0.75,0.0
 38 | 342,ca1f3891-bb6a-48e9-9eb7-1112da79169c,Vietnamese <ss> English,German<ss>Vietnamese,German<ss>Vietnamese<sv>German<ss>Vietnamese,0.5,0.0
 39 | 330,defe22ce-e081-435e-afa6-f9fb8932fba2,manufacturing <ss> mining <ss> construction,manufacturing<ss>construction,manufacturing<ss>construction<sv>manufacturing<ss>construction,0.67,0.0
 40 | 185,92797363-3e73-4170-930d-c4efdfd96dda,ASEAN <ss> Malaysia <ss> China,Indonesia<ss>Malaysia,Malaysia<ss>Indonesia<sv>Indonesia<ss>Malaysia,0.33,0.0
 41 | 122,084d9915-4af4-429e-9b30-002ebc62cc58,White <ss> Hispanic,White<ss>Asian,White<ss>Asian<sv>White<ss>Asian,0.5,0.0
 42 | 65,73cbcf82-9292-4c30-9751-8a242d6d1a29,Atlanta Braves <ss> Astros,Astros<ss>Braves,The Astros<ss>St. Louis<sv>The Astros<ss>St. Louis,0.83,0.0
 43 | 141,a6117d57-8f80-40f0-9a02-0bd4778509fa,11 <ss> 2 <ss> 2,1-yard<ss>21-yarder<ss>2-yarder,11-yard<ss>21-yarder<ss>2-yarder,0.44,0.0
 44 | 277,ddb04730-1656-4d56-bc4b-b3be9b469baf,31,31<ss>30,61<sv>31<ss>30,0.5,0.0
 45 | 193,215d7499-3945-4577-bd8e-e2022dce832a,25 and 29 years <ss> 25 and 29 years,20 and 24 years<ss>25 and 29 years,between 20 and 24 years<ss>between 25 and 29 years,0.5,0.0
 46 | 61,2728b800-74a3-49cd-9ffb-eb358cc801b0,Duke Magnus Birgersson started a war,Duke Magnus Birgersson<ss>started a war,started a war<sv>he started a war,0.57,0.0
 47 | 247,2762894d-d640-4566-ab4c-30d87340a5f0,Germans <ss> English <ss> Irish <ss> United States <ss> Italian,Germans<ss>English<ss>Irish<ss>United States,Germans<ss>English<ss>Irish<ss>United States<sv>Germans<ss>English<ss>Irish<ss>United States,0.8,0.0
 48 | 156,d39fb084-0964-4702-9bca-c23cf07607f3,Cuban soldiers,Ethiopian<ss>soldiers,Ethiopian<sv>Ethiopians,0.33,0.0
 49 | 266,5cc93449-2d02-4e19-842e-8fa2baceb3a3,Irish <ss> German,Scottish<ss>Canadian<ss>Irish,Scottish people<ss>Canadian<ss>Irish people<sv>Scottish<ss>Canadian<ss>Irish,0.33,0.0
 50 | 348,39fc7ab6-0041-4d8b-9121-ca234dbb6bb2,28 <ss> 37 yards <ss> 26-yard,28 and 37 yards<ss>26-yard,28<ss>37<ss>26-yard<sv>28 and 37 yards<ss>26-yard,0.89,0.0
 51 | 46,9b2cfd65-8042-4085-9d3e-876428cb5158,Tamil <ss> Hindi,Tamil<ss>Marathi<ss>Hindi,3.45<sv>Tamil<ss>Marathi<ss>Hindi,0.67,0.0
 52 | 151,a250133b-9b85-4c0b-8496-fb1fd74bb274,May <ss> May <ss> Cardinal Pierre des Près <ss> Fasanier <ss> Fasanier,May<ss>July<ss>September,May<ss>September<sv>May<ss>July<ss>September,0.2,0.0
 53 | 35,2a2e9790-30a1-4663-a2ea-76fac5e788e4,English <ss> Irish <ss> French <ss> German <ss> Italian,English<ss>Irish<ss>polish<ss>French,English<ss>Irish<ss>polish<ss>French<sv>English<ss>Irish<ss>polish<ss>French,0.6,0.0
 54 | 235,6f579268-78fa-4887-84ea-2ee4ba68f8cf,Rice,Craig<ss>Rice<ss>Tom Rathman,Craig<sv>Craig,0.33,0.0
 55 | 168,712ba53b-d57b-4149-827f-83f90ba1dfe0,Wieli ⁇  <ss> Pu ⁇ sk <ss> Kupise <ss> Biel <ss> Markowa,Wieliż<ss>Puńsk<ss>Kupise<ss>Biel<ss>Markowa,Wieliż<ss>Puńsk<ss>Kupise<sv>Wieliż<ss>Puńsk<ss>Kupise<ss>Biel<ss>Markowa,0.6,0.0
 56 | 93,6cbe18a5-09e1-4eed-a56a-9cab4b240bea,Khitans <ss> Minister Seo Hui,Khitan<ss>Goryeo,Khitans<ss>Korea<sv>Khitans<ss>Korea,0.5,0.0
 57 | 183,3f6187dd-4a87-4bcb-ad46-94e3b0b809f6,drain on state budgets,helped elect their bosses<ss>overly generous pension systems,overly generous pension systems<sv>they helped elect their bosses,0.0,0.0
 58 | 200,1179a508-77e1-4816-a30e-3020b86b8dc7,heart disease <ss> hypertension,diabetes<ss>cancer,diabetes<ss>obesity-related cancer<sv>obesity-related cancer<ss>diabetes,0.0,0.0
 59 | 243,5d72920d-ed5f-4470-9e6a-b0313f31095b,under age 18 <ss> 25 to 44 <ss> 45 to 64,under age 18<ss>45 to 64,under age 18<ss>45 to 64<sv>under age 18<ss>45 to 64,0.67,0.0
 60 | 81,99675a67-efdc-415d-bc9a-7273ee03cb22,South Bucks <ss> Canterbury <ss> Portsmouth <ss> Thanet,South Bucks<ss>Canterbury,South Bucks<ss>Canterbury<sv>South Bucks<ss>Canterbury,0.5,0.0
 61 | 29,3322e208-39e5-4980-a219-70da8663629a,Native American <ss> Pacific Islander,Pacific Islander<ss>Asian,Pacific Islander<ss>Asian<sv>Asian<ss>Pacific Islander,0.5,0.0
 62 | 177,2c1435a5-47b5-46e9-9269-fce36f88e79f,second <ss> third,first<ss>second<ss>third<ss>fourth,first<ss>second<ss>third<ss>fourth<sv>first<ss>second<ss>third<ss>fourth,0.5,0.0
 63 | 2,1ffa94c5-05b2-4e78-9a98-a0ac56fbc5cc,irish <ss> english <ss> irish,english<ss>irish,english<ss>irish<sv>english<ss>irish,0.67,0.0
 64 | 49,fbddf60a-31e4-4850-be04-44c4871079a3,second <ss> third,second<ss>fourth,second quarter<ss>fourth quarter<sv>the second<ss>the fourth<sv>2<sv>second<ss>fourth<sv>2,0.5,0.0
 65 | 106,f47d9f89-3aa3-4499-bb6b-baf626a27636,first <ss> third,first quarter<ss>second quarter,first<ss>second<ss>third<ss>fourth<sv>first quarter<ss>second quarter<sv>second quarter<sv>first<ss>second<ss>third<ss>fourth,0.5,0.0
 66 | 85,86969fad-0eda-4854-83e4-e08014809588,Bryant Johnson <ss> T. J. Duckett,Frank Gore<ss>T. J. Duckett,Frank Gore<ss>T. J. Duckett,0.5,0.0
 67 | 233,2927f0d9-dd04-4f19-9856-27789269cf40,Rice <ss> Tom Rathman,Craig<ss>Montana,Montana<ss>Craig<sv>Montana<ss>Craig,0.0,0.0
 68 | 13,44a430a0-b710-4e85-a3c2-6ea8e6f03a4d,under the age of 18 <ss> 18 to 24 <ss> 25 to 44 <ss> 45 to 64,under the age of 18<ss>25 to 44<ss>45 to 64,"under the age of 18<ss>25 to 44<ss>45 to 64<sv>21.8% under the age of 18, 13.1% from 18 to 24, 31.7% from 25 to 44, 20.1% from 45 to 64,",0.75,0.0
 69 | 161,12d2bbe2-263a-4e90-9035-ec44a1d8e9d9,Saints,New Orleans<ss>Saints,New Orleans<sv>New Orleans<sv>New Orleans<sv>New Orleans,0.5,0.0
 70 | 195,3c7497ea-78a3-486b-8d45-909f6c0072ab,25 and 29 years,65 years and over<ss>older than 60<ss>under age 18,between 45 and 49 years<sv>45 and 49 years,0.0,0.0
 71 | 26,706d1e15-617f-44f6-bb45-c6b659c607ea,battleship Espana running aground,Spanish military suffered losses even<ss>at sea; in March,the Spanish military suffering losses even at sea<sv>Spanish military suffering losses<sv>Spanish military suffering losses<sv>suffered losses even at sea,0.0,0.0
 72 | 215,d92cb0da-5627-46a1-9749-fdbfbf3dc025,Hindu,Muslim<ss>Tamil<ss>Telugu<ss>Urdu,"Muslim<sv>Hindu, 9.5%",0.0,0.0
 73 | 127,ce8cdee7-0004-4084-8531-978e670062c8,John <ss> his father,his son<ss>his father,his son John<ss>his father<sv>his son John<ss>his father,0.75,0.0
 74 | 205,d7c74a2c-4789-4087-8aa4-3b637acb285e,French fleet,Jerôme Maurand<ss>Polin,Jerôme Maurand<ss>Polin<sv>1544,0.0,0.0
 75 | 358,6bf6e431-0cbe-4394-ac5d-bd714184c85c,Germany <ss> Romania <ss> France <ss> Italy <ss> Czech Republic,Romania<ss>Slovakia<ss>France<ss>Italy,Italy<ss>Slovakia<ss>Romania<ss>France<sv>Romania<ss>Slovakia<ss>France<ss>Italy,0.6,0.0
 76 | 227,ffcc13f6-5522-4474-88a1-5358fd6eb74d,Muslim adherents <ss> Buddhists,Buddhists<ss>Hindus,Buddhists<ss>Hindus<sv>Buddhists<ss>Hindus,0.5,0.0
 77 | 86,c6e29c82-dcc6-4feb-b966-2fb5e51748b8,first quarter,"first quarter, the Texans delivered<ss>for a touchdown<ss>Colts responded<ss>completing a 2-yard TD pass",first<sv>1<sv>1,0.67,0.0
 78 | 210,a74c8faa-ef32-4779-91d0-3dc7d4b1e7a4,Hindi <ss> Kannada,Urdu<ss>Malayalam<ss>Hindi<ss>Kannada,Urdu<ss>Malayalam<ss>Hindi<ss>Kannada,0.5,0.0
 79 | 58,02d84db7-34ac-4c30-87e0-8769ec27083d,Battle of Mohács,in 1491<ss>signed the Peace of Pressburg,Peace of Pressburg<sv>Peace of Pressburg,0.33,0.0
 80 | 150,ba04b5c7-b0ea-45c4-81c6-e32db9d323d8,May <ss> May <ss> May,June<ss>August,June<ss>August<sv>June<ss>August,0.0,0.0
 81 | 246,87c0bfbb-f591-4674-8b4d-7bbbc6d592ca,Germans <ss> English <ss> Irish <ss> United States <ss> Italian,Hispanics<ss>Germans<ss>English<ss>Irish<ss>United States<ss>Italian,Hispanics<ss>Germans<ss>English<ss>Irish<ss>Italian<ss>United States,0.83,0.0
 82 | 265,0ffb346d-c23c-4018-b68b-84b39eb99677,Polish <ss> English,Norwegian<ss>Polish,Norwegian people<ss>Polish people<sv>Norwegian<ss>Polish,0.5,0.0
 83 | 139,1f3ddaf6-f003-418b-9fa4-d73f6032028a,Scholar Flotation <ss> Alan Sugar,Irving Scholar<ss>Paul Bobroff,Paul Bobroff<ss>Irving Scholar<sv>Irving Scholar<ss>Paul Bobroff<sv>Irving Scholar<ss>Paul Bobroff,0.25,0.0
 84 | 52,f4b97a88-acb5-453f-aa82-b2b1fe1e9a00,Jaél de Peigne <ss> Jaél de Peigne,"Catherine<ss>Jaél de Peigne,",Jaél de Peigne<sv>Catherine<ss>Jaél de Peigne,0.5,0.0
 85 | 55,abbbabbd-878b-427c-aed5-521c71ce15e0,Cecil <ss> Jaél de Peigne <ss> Jaél de Peigne,Catherine<ss>Jaél de Peigne,Catherine<ss>Jaél de Peigne<sv>Catherine<ss>Jaél de Peigne,0.33,0.0
 86 | 11,25ed59a4-e726-47c5-bdb7-862e8baf1bae,the Uqair conference,Uqair conference in 1922<ss>from 1923,trade blockade<sv>trade blockade,0.0,0.0
 87 | 352,da82ac4b-8c49-4332-a6aa-5c7fca99f61e,Eagles <ss> Minnesota Vikings,Philadelphia Eagles<ss>New England Patriots,the Philadelphia Eagles<ss>the New England Patriots<sv>Philadelphia Eagles<ss>New England Patriots,0.33,0.0
 88 | 276,08b77000-467e-4ba1-a810-ca205cb81f6c,60 and 69 years old,teenagers<ss>adult,adults<sv>teenagers<ss>adult,0.0,0.0
 89 | 328,6c74d56c-721e-49a4-85a8-a0da63071b73,1990 <ss> 1994 <ss> 1998,1990<ss>1994<ss>1998<ss>2002,1990<ss>1994<ss>1998<ss>2002<sv>1990<ss>1994<ss>1998<ss>2002,0.75,0.0
 90 | 95,721a35e3-573f-45e7-a1fc-93dde3ff31cc,2006-08 FA Cup <ss> 2012-13 FA Cup <ss> 2014-15 FA Cup,2007-08 FA Cup<ss>2013-14 FA Cup<ss>2014-15 FA Cup,2014-15<sv>3,0.58,0.0
 91 | 275,fa562801-16fd-4ac5-91cd-c2590286f835,10 and 19 <ss> 50 and 59,between 70 and 79<ss>80 and 89<ss>90 and older,70 and 79<ss>80 and 89<ss>90 and older<sv>70 and 79<ss>80 and 89<ss>90 and older,0.0,0.0
 92 | 159,36606e9e-faa1-4b6a-b821-5e1dd4a122f1,WR,QB<ss>WR,QB<ss>WR,0.5,0.0
 93 | 0,8b7f7ee7-a1cf-4656-bb92-fcc74acdae82,"1, 5 and 10 Pesos",100 Peso note<ss>500 Pesos note,500 Pesos note<sv>100<ss>500<sv>500 Pesos note,0.0,0.0
 94 | 365,b8d54267-0930-4b88-97de-d1d742167c28,1987 s <ss> 1992 s,1999<ss>1987<ss>1992,1999<ss>1987<ss>1992<sv>1999<ss>1987<ss>1992,0.44,0.0
 95 | 294,fea934fd-f8ad-4d33-b1bc-7744f8ef10fb,18 to 24,18 to 24<ss>25 and older<ss>between 25 and 64,under the age of 18<sv>18 to 24<ss>25 and older<ss>between 25 and 64,0.33,0.0
 96 | 21,b8c3ef46-22c0-4e80-beee-03e9dc27274a,Pacific Islander <ss> English,Polish<ss>French,Polish<ss>French<sv>Polish<ss>French,0.0,0.0
 97 | 314,de295075-849a-4746-870c-e318f9fc9aaf,Brett Favre <ss> Robert Brooks,Antonio Freeman<ss>Robert Brooks,Favre<ss>Antonio Freeman<ss>Robert Brooks<ss>Dorsey Levens<sv>Antonio Freeman<ss>Robert Brooks,0.5,0.0
 98 | 355,62bf9785-191f-4f88-8738-6a7d2e7e2184,Libya <ss> U.S.A.,Germany<ss>Spain<ss>Libya<ss>U.S.A,Germany<ss>Spain<ss>Libya<ss>U.S.A<sv>Germany<ss>Spain<ss>Libya<ss>U.S.A,0.5,0.0
 99 | 176,a41f2022-868f-4dba-9ac5-f5710c5865b8,Mark Sanchez,Ryan Fitzpatrick<ss>Mark Sanchez,Mark Sanchez<ss>Ryan Fitzpatrick<sv>Mark Sanchez<ss>Ryan Fitzpatrick,0.5,0.0
100 | 202,ab248bfc-5853-464d-a561-fc430d40275e,heart disease <ss> hypertension,diabetes<ss>hypertension,"diabetes, hypertension<sv>diabetes<ss>hypertension",0.5,0.0
101 | 229,16662b13-8d04-4171-a0a9-0087fa2cc657,Muslim <ss> Buddhists,Buddhists<ss>Muslims,Muslims<ss>Buddhists<sv>Buddhists<ss>Muslims,0.5,0.0
102 | 


--------------------------------------------------------------------------------
/tfrec/squad_parser.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "id": "QvowpfKM7Qot"
  7 |    },
  8 |    "source": [
  9 |     "## Explore the Squad Dataset 2.0\n",
 10 |     "\n"
 11 |    ]
 12 |   },
 13 |   {
 14 |    "cell_type": "code",
 15 |    "execution_count": null,
 16 |    "metadata": {},
 17 |    "outputs": [],
 18 |    "source": [
 19 |     "#@title specify file_path for Squad Dataset\n",
 20 |     "from google.colab import drive\n",
 21 |     "import json\n",
 22 |     "import os\n",
 23 |     "\n",
 24 |     "drive.mount('/content/drive')\n",
 25 |     "\n",
 26 |     "squad_train_path = '' # path to train-v1.1.json on gdrive\n",
 27 |     "squad_dev_path = '' # path to dev-v1.1.json on gdrive"
 28 |    ]
 29 |   },
 30 |   {
 31 |    "cell_type": "code",
 32 |    "execution_count": 41,
 33 |    "metadata": {
 34 |     "id": "Sn3DeGwl5dAh"
 35 |    },
 36 |    "outputs": [],
 37 |    "source": [
 38 |     "with open(squad_train_path) as file_d:\n",
 39 |     "    squad_train_data = json.load(file_d)\n",
 40 |     "    \n",
 41 |     "with open(squad_dev_path) as file_d:\n",
 42 |     "    squad_dev_data = json.load(file_d)"
 43 |    ]
 44 |   },
 45 |   {
 46 |    "cell_type": "markdown",
 47 |    "metadata": {
 48 |     "id": "XRxmQNlHHwYb"
 49 |    },
 50 |    "source": [
 51 |     "## Main Parse Function for Squad Dataset"
 52 |    ]
 53 |   },
 54 |   {
 55 |    "cell_type": "code",
 56 |    "execution_count": 54,
 57 |    "metadata": {
 58 |     "id": "wL1KdYju9RVI"
 59 |    },
 60 |    "outputs": [],
 61 |    "source": [
 62 |     "def parse_squad_datset(squad_dataset: dict) -> dict:\n",
 63 |     "    \"\"\"\n",
 64 |     "    input: List of dict, each dict is an exmample.\n",
 65 |     "\n",
 66 |     "    ouput: dict with 4 item. Key: str; value: List[str]\n",
 67 |     "    {\n",
 68 |     "        contexts = [str1, ...] # note: contexts is optional\n",
 69 |     "        question_ids = [str1, ...] # from id\n",
 70 |     "        questions: [str1, ..] # from question\n",
 71 |     "        answers: [str1, ...]\n",
 72 |     "    }\n",
 73 |     "    \n",
 74 |     "    \"\"\"\n",
 75 |     "    contexts = []\n",
 76 |     "    question_ids = []\n",
 77 |     "    questions = []\n",
 78 |     "    answers = []\n",
 79 |     "\n",
 80 |     "    data = squad_dataset[\"data\"]\n",
 81 |     "    for example in data:\n",
 82 |     "        paragraphs = example[\"paragraphs\"]\n",
 83 |     "        for paragraph in paragraphs: # paragraphs: List[json]\n",
 84 |     "            context = paragraph[\"context\"]\n",
 85 |     "            qas = paragraph[\"qas\"]\n",
 86 |     "            for qa in qas:\n",
 87 |     "                question = qa[\"question\"]\n",
 88 |     "                question_id = qa[\"id\"]\n",
 89 |     "                answer = qa[\"answers\"][0][\"text\"]\n",
 90 |     "\n",
 91 |     "                # appending\n",
 92 |     "                contexts.append(context)\n",
 93 |     "                question_ids.append(question_id)\n",
 94 |     "                questions.append(question)\n",
 95 |     "                answers.append(answer)\n",
 96 |     "\n",
 97 |     "    return {\n",
 98 |     "        \"context\": contexts, \n",
 99 |     "        \"question_ids\": question_ids,\n",
100 |     "        \"questions\": questions,\n",
101 |     "        \"answers\": answers,\n",
102 |     "    }"
103 |    ]
104 |   },
105 |   {
106 |    "cell_type": "code",
107 |    "execution_count": 67,
108 |    "metadata": {
109 |     "id": "0QP2NI3YFY8V"
110 |    },
111 |    "outputs": [],
112 |    "source": [
113 |     "#@title define function: check_saved_dataset() { display-mode: \"both\" }\n",
114 |     "\n",
115 |     "def check_saved_dataset(gs_dataset, i = 5):\n",
116 |     "    j = 0 \n",
117 |     "    # print(f\"dataset: {string(gs_dataset)}\")\n",
118 |     "\n",
119 |     "    print(f\"\"\"\n",
120 |     "    len of context: {len(gs_dataset[\"context\"])}\n",
121 |     "    len of questions: {len(gs_dataset[\"questions\"])}\n",
122 |     "    len of ids: {len(gs_dataset[\"question_ids\"])}\n",
123 |     "    len of answers: {len(gs_dataset[\"answers\"])}\n",
124 |     "    \"\"\")\n",
125 |     "    \n",
126 |     "    for context, question, answer, id in zip(gs_dataset[\"context\"], gs_dataset[\"questions\"], gs_dataset[\"answers\"], gs_dataset[\"question_ids\"]):\n",
127 |     "        print(f\"context: {context}\")\n",
128 |     "        print(f\"question: {question}\")\n",
129 |     "        print(f\"answer: {answer}\")\n",
130 |     "        print(f\"id: {id}\")\n",
131 |     "        \n",
132 |     "        print(\"\\n\")\n",
133 |     "        j += 1\n",
134 |     "        if j == i:\n",
135 |     "            break\n"
136 |    ]
137 |   },
138 |   {
139 |    "cell_type": "code",
140 |    "execution_count": 68,
141 |    "metadata": {
142 |     "colab": {
143 |      "base_uri": "https://localhost:8080/"
144 |     },
145 |     "id": "9pVB88b5GaUI",
146 |     "outputId": "f2483e5e-3c1c-4f09-fd2d-2cb4e8c50c6e"
147 |    },
148 |    "outputs": [
149 |     {
150 |      "name": "stdout",
151 |      "output_type": "stream",
152 |      "text": [
153 |       "\n",
154 |       "    len of context: 10570\n",
155 |       "    len of questions: 10570\n",
156 |       "    len of ids: 10570\n",
157 |       "    len of answers: 10570\n",
158 |       "    \n",
159 |       "context: Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.\n",
160 |       "question: Which NFL team represented the AFC at Super Bowl 50?\n",
161 |       "answer: Denver Broncos\n",
162 |       "id: 56be4db0acb8001400a502ec\n",
163 |       "\n",
164 |       "\n",
165 |       "context: Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.\n",
166 |       "question: Which NFL team represented the NFC at Super Bowl 50?\n",
167 |       "answer: Carolina Panthers\n",
168 |       "id: 56be4db0acb8001400a502ed\n",
169 |       "\n",
170 |       "\n",
171 |       "context: Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.\n",
172 |       "question: Where did Super Bowl 50 take place?\n",
173 |       "answer: Santa Clara, California\n",
174 |       "id: 56be4db0acb8001400a502ee\n",
175 |       "\n",
176 |       "\n",
177 |       "context: Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.\n",
178 |       "question: Which NFL team won Super Bowl 50?\n",
179 |       "answer: Denver Broncos\n",
180 |       "id: 56be4db0acb8001400a502ef\n",
181 |       "\n",
182 |       "\n",
183 |       "context: Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.\n",
184 |       "question: What color was used to emphasize the 50th anniversary of the Super Bowl?\n",
185 |       "answer: gold\n",
186 |       "id: 56be4db0acb8001400a502f0\n",
187 |       "\n",
188 |       "\n"
189 |      ]
190 |     }
191 |    ],
192 |    "source": [
193 |     "check_saved_dataset(parsed_squad_dev_dataset)"
194 |    ]
195 |   },
196 |   {
197 |    "cell_type": "code",
198 |    "execution_count": 69,
199 |    "metadata": {
200 |     "colab": {
201 |      "base_uri": "https://localhost:8080/"
202 |     },
203 |     "id": "jTmIOf8IGc4N",
204 |     "outputId": "8de051eb-cd58-4698-bbd3-d50dde16e957"
205 |    },
206 |    "outputs": [
207 |     {
208 |      "name": "stdout",
209 |      "output_type": "stream",
210 |      "text": [
211 |       "\n",
212 |       "    len of context: 87599\n",
213 |       "    len of questions: 87599\n",
214 |       "    len of ids: 87599\n",
215 |       "    len of answers: 87599\n",
216 |       "    \n",
217 |       "context: Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\n",
218 |       "question: To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?\n",
219 |       "answer: Saint Bernadette Soubirous\n",
220 |       "id: 5733be284776f41900661182\n",
221 |       "\n",
222 |       "\n",
223 |       "context: Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\n",
224 |       "question: What is in front of the Notre Dame Main Building?\n",
225 |       "answer: a copper statue of Christ\n",
226 |       "id: 5733be284776f4190066117f\n",
227 |       "\n",
228 |       "\n",
229 |       "context: Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\n",
230 |       "question: The Basilica of the Sacred heart at Notre Dame is beside to which structure?\n",
231 |       "answer: the Main Building\n",
232 |       "id: 5733be284776f41900661180\n",
233 |       "\n",
234 |       "\n",
235 |       "context: Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\n",
236 |       "question: What is the Grotto at Notre Dame?\n",
237 |       "answer: a Marian place of prayer and reflection\n",
238 |       "id: 5733be284776f41900661181\n",
239 |       "\n",
240 |       "\n",
241 |       "context: Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\n",
242 |       "question: What sits on top of the Main Building at Notre Dame?\n",
243 |       "answer: a golden statue of the Virgin Mary\n",
244 |       "id: 5733be284776f4190066117e\n",
245 |       "\n",
246 |       "\n"
247 |      ]
248 |     }
249 |    ],
250 |    "source": [
251 |     "check_saved_dataset(parsed_squad_train_dataset)"
252 |    ]
253 |   },
254 |   {
255 |    "cell_type": "code",
256 |    "execution_count": 81,
257 |    "metadata": {
258 |     "id": "nBBBrPf3K8SZ"
259 |    },
260 |    "outputs": [],
261 |    "source": [
262 |     "#@title define functin: write_to_gs\n",
263 |     "import tensorflow as tf\n",
264 |     "# authenticate\n",
265 |     "# For accessing data from google storage (gs://)\n",
266 |     "from google.colab import auth\n",
267 |     "auth.authenticate_user()\n",
268 |     "\n",
269 |     "def write_to_gs(path, parsed_dataset):\n",
270 |     "    with tf.io.gfile.GFile(path, 'w') as json_file:\n",
271 |     "        json.dump(parsed_dataset, json_file)\n"
272 |    ]
273 |   },
274 |   {
275 |    "cell_type": "code",
276 |    "execution_count": 66,
277 |    "metadata": {
278 |     "id": "DmrOs6pqFPmw"
279 |    },
280 |    "outputs": [],
281 |    "source": [
282 |     "parsed_squad_train_dataset = parse_squad_datset(squad_train_data)\n",
283 |     "parsed_squad_dev_dataset = parse_squad_datset(squad_dev_data)"
284 |    ]
285 |   },
286 |   {
287 |    "cell_type": "code",
288 |    "execution_count": 82,
289 |    "metadata": {
290 |     "id": "2Q03_o1oIEdA"
291 |    },
292 |    "outputs": [],
293 |    "source": [
294 |     "#@title Write parsed_squad data to Google Storage\n",
295 |     "squad_data_path = '' # path on gs to save parsed data\n",
296 |     "squad_train_data_path = os.path.join(squad_data_path, \"squad1.1_train_data_parsed.json\")\n",
297 |     "squad_dev_data_path = os.path.join(squad_data_path, \"squad1.1_dev_data_parsed.json\")\n",
298 |     "\n",
299 |     "\n",
300 |     "write_to_gs(squad_train_data_path, parsed_squad_train_dataset)\n",
301 |     "write_to_gs(squad_dev_data_path, parsed_squad_dev_dataset)"
302 |    ]
303 |   },
304 |   {
305 |    "cell_type": "code",
306 |    "execution_count": null,
307 |    "metadata": {
308 |     "id": "Dm4NEeQwMoCJ"
309 |    },
310 |    "outputs": [],
311 |    "source": [
312 |     "# download the file from google storage\n",
313 |     "!gsutil cp # copy parsed file to local\n",
314 |     "with open(\"/tmp/squad_dev_data_parsed.json\") as a_file:\n",
315 |     "    check_squad_dev = json.load(a_file)\n",
316 |     "\n",
317 |     "check_saved_dataset(check_squad_dev)"
318 |    ]
319 |   }
320 |  ],
321 |  "metadata": {
322 |   "colab": {
323 |    "collapsed_sections": [],
324 |    "name": "parsing_squad_dataset.ipynb",
325 |    "provenance": []
326 |   },
327 |   "kernelspec": {
328 |    "display_name": "Python 3",
329 |    "language": "python",
330 |    "name": "python3"
331 |   },
332 |   "language_info": {
333 |    "codemirror_mode": {
334 |     "name": "ipython",
335 |     "version": 3
336 |    },
337 |    "file_extension": ".py",
338 |    "mimetype": "text/x-python",
339 |    "name": "python",
340 |    "nbconvert_exporter": "python",
341 |    "pygments_lexer": "ipython3",
342 |    "version": "3.9.2"
343 |   }
344 |  },
345 |  "nbformat": 4,
346 |  "nbformat_minor": 4
347 | }
348 | 


--------------------------------------------------------------------------------
/tfrec/tfrec_drop.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "id": "qp9-AReX2bt_"
  7 |    },
  8 |    "source": [
  9 |     "# T5 TFRecord Preprocessing\n",
 10 |     "\n",
 11 |     "This notebook preprocesses the DROP dataset in JSON format to a TFRecords dataset that can be fed into the model after batching for training."
 12 |    ]
 13 |   },
 14 |   {
 15 |    "cell_type": "code",
 16 |    "execution_count": null,
 17 |    "metadata": {
 18 |     "id": "Nfp8x5hOmrPI"
 19 |    },
 20 |    "outputs": [],
 21 |    "source": [
 22 |     "if 'colab' in str(get_ipython()):\n",
 23 |     "    import google.colab as colab\n",
 24 |     "    colab.auth.authenticate_user()\n",
 25 |     "    colab.drive.mount('/content/gdrive') # mount google drive\n",
 26 |     "\n",
 27 |     "    # install libraries not native to colab\n",
 28 |     "    !pip install tensorflow-text\n",
 29 |     "    !pip install transformers==3.3.1\n",
 30 |     "    !pip install datasets==1.1.2\n",
 31 |     "    !pip install tqdm\n",
 32 |     "\n",
 33 |     "# remove pip install outputs\n",
 34 |     "from IPython.display import clear_output\n",
 35 |     "clear_output()"
 36 |    ]
 37 |   },
 38 |   {
 39 |    "cell_type": "code",
 40 |    "execution_count": null,
 41 |    "metadata": {
 42 |     "colab": {
 43 |      "base_uri": "https://localhost:8080/"
 44 |     },
 45 |     "executionInfo": {
 46 |      "elapsed": 7649,
 47 |      "status": "ok",
 48 |      "timestamp": 1603953774465,
 49 |      "user": {
 50 |       "displayName": "Tim Chen",
 51 |       "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjCHzDArJyPPAHsYKioqVZeP1reZIsBL9RZDKwCOy8=s64",
 52 |       "userId": "17136390159455304905"
 53 |      },
 54 |      "user_tz": 420
 55 |     },
 56 |     "id": "Ls1fXWF3sLs1",
 57 |     "outputId": "6da152f6-0a57-4abf-fefe-7b06ea7c11e7"
 58 |    },
 59 |    "outputs": [
 60 |     {
 61 |      "name": "stdout",
 62 |      "output_type": "stream",
 63 |      "text": [
 64 |       "TensorFlow 2.2.0\n"
 65 |      ]
 66 |     }
 67 |    ],
 68 |    "source": [
 69 |     "# ml libraries\n",
 70 |     "import tensorflow as tf\n",
 71 |     "import tensorflow.keras as keras\n",
 72 |     "import tensorflow.keras.backend as K\n",
 73 |     "import transformers\n",
 74 |     "import datasets # https://huggingface.co/docs/datasets/\n",
 75 |     "\n",
 76 |     "# data processing libraries\n",
 77 |     "import numpy as np\n",
 78 |     "import pandas as pd\n",
 79 |     "import matplotlib.pyplot as plt\n",
 80 |     "from tqdm.auto import tqdm\n",
 81 |     "\n",
 82 |     "# other libraries\n",
 83 |     "import os\n",
 84 |     "import json\n",
 85 |     "import functools\n",
 86 |     "import time\n",
 87 |     "import warnings\n",
 88 |     "from datetime import datetime\n",
 89 |     "from typing import Generator\n",
 90 |     "\n",
 91 |     "print(f'TensorFlow {tf.__version__}')"
 92 |    ]
 93 |   },
 94 |   {
 95 |    "cell_type": "markdown",
 96 |    "metadata": {
 97 |     "id": "jY88Y3FzXUWN"
 98 |    },
 99 |    "source": [
100 |     "## Configurations"
101 |    ]
102 |   },
103 |   {
104 |    "cell_type": "code",
105 |    "execution_count": null,
106 |    "metadata": {
107 |     "executionInfo": {
108 |      "elapsed": 563,
109 |      "status": "ok",
110 |      "timestamp": 1603953774466,
111 |      "user": {
112 |       "displayName": "Tim Chen",
113 |       "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjCHzDArJyPPAHsYKioqVZeP1reZIsBL9RZDKwCOy8=s64",
114 |       "userId": "17136390159455304905"
115 |      },
116 |      "user_tz": 420
117 |     },
118 |     "id": "l1NeuclbwOB0",
119 |     "outputId": "603c57ff-7a3f-427c-9918-cbba06d7cd45"
120 |    },
121 |    "outputs": [
122 |     {
123 |      "name": "stdout",
124 |      "output_type": "stream",
125 |      "text": [
126 |       "New tokens added: 2\n"
127 |      ]
128 |     }
129 |    ],
130 |    "source": [
131 |     "#@title Constants\n",
132 |     "T5_MODEL                = 't5-small'  #@param { type: \"string\" }\n",
133 |     "ENCODER_MAX_LEN         = 512 #@param { type: \"integer\" }\n",
134 |     "DECODER_MAX_LEN         = 54 #@param { type: \"integer\" } \n",
135 |     "TOKENIZER               = transformers.AutoTokenizer.from_pretrained(T5_MODEL)\n",
136 |     "special_tokens_dict     = {'additional_special_tokens': ['0','1','2','3', '4', '5', '6', '7', '8', '9', '<ss>', '<sv>']}\n",
137 |     "num_added_toks          = TOKENIZER.add_special_tokens(special_tokens_dict)\n",
138 |     "print(f'New tokens added: {num_added_toks}')\n",
139 |     "\n",
140 |     "# tim's default paths\n",
141 |     "LOCAL_PATH              = '' # local folder containing DROP json\n",
142 |     "TRAIN_JSON              = 'drop_dataset_train.json' \n",
143 |     "DEV_JSON                = 'drop_dataset_dev.json' \n",
144 |     "DROP_JSON_DIR_LOCAL     = LOCAL_PATH + '/data/drop_dataset' #@param { type: \"string\" }\n",
145 |     "DROP_JSON_DIR_GCP       = '' # gcp storage path for saving tfrecords #@param { type: \"string\" }\n",
146 |     "EXPORT_DIR              = LOCAL_PATH + '/temp' #@param { type: \"string\" }\n",
147 |     "\n",
148 |     "TRAIN_DROP_JSON_GCP     = os.path.join(DROP_JSON_DIR_GCP, TRAIN_JSON)\n",
149 |     "DEV_DROP_JSON_GCP       = os.path.join(DROP_JSON_DIR_GCP, DEV_JSON) "
150 |    ]
151 |   },
152 |   {
153 |    "cell_type": "code",
154 |    "execution_count": null,
155 |    "metadata": {},
156 |    "outputs": [],
157 |    "source": [
158 |     "#@title Test Mode Setup\n",
159 |     "def sample_data_to_file(\n",
160 |     "                json_file: str,\n",
161 |     "                sample_count: int,\n",
162 |     "                source_dir: str, \n",
163 |     "                export_dir: str = EXPORT_DIR) -> str:\n",
164 |     "    \n",
165 |     "    # load source json\n",
166 |     "    source_file_path = os.path.join(source_dir, json_file)\n",
167 |     "    with tf.io.gfile.GFile(source_file_path) as f:\n",
168 |     "        data_json_dict = json.load(f)  # dict of dict\n",
169 |     "        sampled_data_json_dict = {\n",
170 |     "            key: value\n",
171 |     "            for i, (key, value) in enumerate(data_json_dict.items())\n",
172 |     "            if i < sample_count\n",
173 |     "        }\n",
174 |     "\n",
175 |     "    # prep export json file path\n",
176 |     "    sample_count = len(sampled_data_json_dict)  # get the real sample count\n",
177 |     "    sampled_output_path = os.path.join(export_dir, json_file)\n",
178 |     "    sampled_output_path = sampled_output_path.replace('.json', f'{sample_count}.json')\n",
179 |     "\n",
180 |     "    # export sample json\n",
181 |     "    with tf.io.gfile.GFile(sampled_output_path, 'w') as fout:\n",
182 |     "        json.dump(sampled_data_json_dict, fout)\n",
183 |     "\n",
184 |     "    print(f'Dumped {sample_count} samples to file: {sampled_output_path}')\n",
185 |     "    return sampled_output_path\n",
186 |     "\n",
187 |     "SAMPLE_DATA = True  #@param { type: \"boolean\" }\n",
188 |     "SAMPLE_COUNT = 3 #@param { type: \"integer\" }\n",
189 |     "if SAMPLE_DATA:\n",
190 |     "    train_json = sample_data_to_file(\n",
191 |     "                    json_file=TRAIN_JSON, \n",
192 |     "                    sample_count=SAMPLE_COUNT,\n",
193 |     "                    source_dir=DROP_JSON_DIR_LOCAL,\n",
194 |     "                    export_dir=EXPORT_DIR)\n",
195 |     "    dev_json = sample_data_to_file(\n",
196 |     "                    json_file=DEV_JSON, \n",
197 |     "                    sample_count=SAMPLE_COUNT,\n",
198 |     "                    source_dir=DROP_JSON_DIR_LOCAL,\n",
199 |     "                    export_dir=EXPORT_DIR)\n",
200 |     "else:\n",
201 |     "    train_json = os.path.join(DROP_JSON_DIR_LOCAL, TRAIN_JSON)\n",
202 |     "    dev_json = os.path.join(DROP_JSON_DIR_LOCAL, DEV_JSON)\n",
203 |     "    print(train_json)\n",
204 |     "    print(dev_json)"
205 |    ]
206 |   },
207 |   {
208 |    "cell_type": "markdown",
209 |    "metadata": {
210 |     "id": "dOBK366_aq6P"
211 |    },
212 |    "source": [
213 |     "## Encoding TFRecord"
214 |    ]
215 |   },
216 |   {
217 |    "cell_type": "code",
218 |    "execution_count": null,
219 |    "metadata": {
220 |     "cellView": "both",
221 |     "id": "x9rc72mrwSVd"
222 |    },
223 |    "outputs": [],
224 |    "source": [
225 |     "def encode(\n",
226 |     "        example: dict,\n",
227 |     "        encoder_max_len: int = ENCODER_MAX_LEN, \n",
228 |     "        decoder_max_len: int = DECODER_MAX_LEN,\n",
229 |     "        tokenizer: transformers.PreTrainedTokenizer = TOKENIZER) -> dict:\n",
230 |     "    \"\"\"Tokenize data.\n",
231 |     "    Args:\n",
232 |     "        example (dict): Raw dict parsed from DROP json:\n",
233 |     "                            example['context']\n",
234 |     "                            example['question']\n",
235 |     "                            example['answer']\n",
236 |     "    Returns: \n",
237 |     "        (dict) Dictionary with values tokenized:\n",
238 |     "                            return['input_ids']\n",
239 |     "                            return['attention_mask']\n",
240 |     "                            return['labels']\n",
241 |     "                            return['decoder_attention_mask']\n",
242 |     "    \"\"\"            \n",
243 |     "    context = example['context']\n",
244 |     "    question = example['question'] # '1+1+(1+(1+1))=10'\n",
245 |     "    answer = example['answer']\n",
246 |     "  \n",
247 |     "    question_plus = f\"answer_me: {str(question)} context: {str(context)}\"\n",
248 |     "    answer_plus = str(answer)\n",
249 |     "\n",
250 |     "    encoder_inputs = tokenizer(\n",
251 |     "                            question_plus, \n",
252 |     "                            truncation=True, \n",
253 |     "                            return_tensors='tf', \n",
254 |     "                            max_length=encoder_max_len,\n",
255 |     "                            padding='max_length')\n",
256 |     "    decoder_inputs = tokenizer(\n",
257 |     "                            answer_plus, \n",
258 |     "                            truncation=True, \n",
259 |     "                            return_tensors='tf', \n",
260 |     "                            max_length=decoder_max_len,\n",
261 |     "                            padding='max_length')\n",
262 |     "    \n",
263 |     "    return {\n",
264 |     "        'query_id'                  : [str.encode(example['query_id'])], \n",
265 |     "        'answer_type'               : [str.encode(example['answer_type'])], \n",
266 |     "        'validated_answers'         : [str.encode(example['validated_answers'])],\n",
267 |     "        'input_ids'                 : encoder_inputs['input_ids'][0], \n",
268 |     "        'attention_mask'            : encoder_inputs['attention_mask'][0], \n",
269 |     "        'labels'                    : decoder_inputs['input_ids'][0], \n",
270 |     "        'decoder_attention_mask'    : decoder_inputs['attention_mask'][0]}\n",
271 |     "\n",
272 |     "def _bytes_feature(values):\n",
273 |     "  \"\"\"Returns a bytes_list from a list of string / byte.\"\"\"\n",
274 |     "  return tf.train.Feature(bytes_list=tf.train.BytesList(value=values))\n",
275 |     "\n",
276 |     "def _float_feature(values):\n",
277 |     "  \"\"\"Returns a float_list from a list of float / double.\"\"\"\n",
278 |     "  return tf.train.Feature(float_list=tf.train.FloatList(value=values))\n",
279 |     "\n",
280 |     "def _int64_feature(values):\n",
281 |     "    \"\"\"Returns an int64_list from a list of bool / enum / int / uint.\"\"\"\n",
282 |     "    try:\n",
283 |     "        return tf.train.Feature(int64_list=tf.train.Int64List(value=values))\n",
284 |     "    except Exception as e:\n",
285 |     "        print(f'failed at: {values}')\n",
286 |     "        raise e\n",
287 |     "\n",
288 |     "def to_serialized_example(encoded_example):\n",
289 |     "    feature = {\n",
290 |     "        'query_id': _bytes_feature(encoded_example['query_id']),\n",
291 |     "        'answer_type': _bytes_feature(encoded_example['answer_type']),\n",
292 |     "        'validated_answers': _bytes_feature(encoded_example['validated_answers']),\n",
293 |     "        'input_ids': _int64_feature(encoded_example['input_ids']),\n",
294 |     "        'attention_mask': _int64_feature(encoded_example['attention_mask']),\n",
295 |     "        'labels': _int64_feature(encoded_example['labels']),\n",
296 |     "        'decoder_attention_mask': _int64_feature(encoded_example['decoder_attention_mask']),\n",
297 |     "    }\n",
298 |     "    example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n",
299 |     "    return example_proto.SerializeToString()\n",
300 |     "\n",
301 |     "def parse_validated_answers(validated_answers: list) -> str:\n",
302 |     "    answers = []\n",
303 |     "    for answer_dict in validated_answers:\n",
304 |     "        ans, _ = parse_answer(answer_dict)\n",
305 |     "        answers.append(ans)\n",
306 |     "    return '<sv>'.join([a.strip() for a in answers if a.strip()])\n",
307 |     "        \n",
308 |     "def parse_answer(answer_dict: dict) -> str:\n",
309 |     "    \"\"\"\n",
310 |     "    Example of answer_dict = { \"number\": \"\", \"date\": { \"day\": \"\", \"month\": \"\", \"year\": \"\" }, \"spans\": [ \"Chaz Schilens\" ] }\n",
311 |     "    Returns a string or None.\n",
312 |     "    \"\"\"\n",
313 |     "    number = answer_dict['number'].strip()\n",
314 |     "    if number:\n",
315 |     "        return number, 'n'\n",
316 |     "    \n",
317 |     "    spans = answer_dict['spans']\n",
318 |     "    spans_str = '<ss>'.join([span.strip() for span in spans if span.strip()])\n",
319 |     "    if spans_str:\n",
320 |     "        if len(spans) > 1:\n",
321 |     "            return spans_str, 's'\n",
322 |     "        else:\n",
323 |     "            return spans_str, 'ss'\n",
324 |     "    \n",
325 |     "    date = answer_dict['date']\n",
326 |     "    if len(date) != 3:\n",
327 |     "        return None, None\n",
328 |     "    date = ' '.join([d.strip() for d in [date['day'], date['month'], date['year']] if d.strip()])\n",
329 |     "    if date:\n",
330 |     "        return date, 'd'\n",
331 |     "    \n",
332 |     "    return None, None\n",
333 |     "\n",
334 |     "def make_data_generator(file_path: str) -> Generator:\n",
335 |     "    \"\"\"\n",
336 |     "    json_example: {\n",
337 |     "        \"passage\": \"foo bar\",\n",
338 |     "        \"qa_pairs\": [\n",
339 |     "            {\n",
340 |     "                \"question\": \"foo bar\",\n",
341 |     "                \"answer\": { \"number\": \"\", \"date\": { \"day\": \"\", \"month\": \"\", \"year\": \"\" }, \"spans\": [ \"Chaz Schilens\" ] },\n",
342 |     "                \"query_id\": \"f37e81fa-ef7b-4583-b671-762fc433faa9\",\n",
343 |     "            }\n",
344 |     "        ]\n",
345 |     "    }\n",
346 |     "\n",
347 |     "    Returns a generator function.\n",
348 |     "    \"\"\"\n",
349 |     "    with tf.io.gfile.GFile(file_path) as json_file:\n",
350 |     "        data_json_dict = json.load(json_file) \n",
351 |     "\n",
352 |     "    def gen():\n",
353 |     "        for key, json_example in data_json_dict.items():\n",
354 |     "            passage = json_example['passage']\n",
355 |     "            for qa_dict in json_example['qa_pairs']:\n",
356 |     "                \n",
357 |     "                question = qa_dict['question']\n",
358 |     "                answer, answer_type = parse_answer(qa_dict['answer'])\n",
359 |     "                query_id = qa_dict['query_id']\n",
360 |     "                if 'validated_answers' in qa_dict:\n",
361 |     "                    valid_ans = parse_validated_answers(qa_dict['validated_answers'])\n",
362 |     "                else:\n",
363 |     "                    valid_ans = ''                \n",
364 |     "                if answer is None:\n",
365 |     "                    continue\n",
366 |     "\n",
367 |     "                encoded_example = encode({\n",
368 |     "                    'context': passage,\n",
369 |     "                    'question': question,\n",
370 |     "                    'answer': answer,\n",
371 |     "                    'validated_answers': valid_ans,\n",
372 |     "                    'answer_type': answer_type,\n",
373 |     "                    'query_id': query_id\n",
374 |     "                })\n",
375 |     "                serialized_example = to_serialized_example(encoded_example)\n",
376 |     "                \n",
377 |     "                yield serialized_example\n",
378 |     "    \n",
379 |     "    return gen\n",
380 |     "\n",
381 |     "def get_num_examples(file_path):\n",
382 |     "    with tf.io.gfile.GFile(file_path) as json_file:\n",
383 |     "        data_json_dict = json.load(json_file) \n",
384 |     "        return len(data_json_dict)\n",
385 |     "\n",
386 |     "def load_dataset(file_path: str) -> tf.data.Dataset:\n",
387 |     "    return tf.data.Dataset.from_generator(\n",
388 |     "                        make_data_generator(file_path),\n",
389 |     "                        output_types=tf.string)"
390 |    ]
391 |   },
392 |   {
393 |    "cell_type": "code",
394 |    "execution_count": null,
395 |    "metadata": {},
396 |    "outputs": [],
397 |    "source": [
398 |     "SUFFIX = '_test'\n",
399 |     "def write_json_to_tf_record_file(json_path: str) -> str:\n",
400 |     "    dataset = load_dataset(json_path)\n",
401 |     "    tf_record_file_path = os.path.join(\n",
402 |     "        os.path.dirname(json_path),\n",
403 |     "        os.path.basename(json_path).replace('.json', f'{SUFFIX}.tfrecord')\n",
404 |     "    )\n",
405 |     "\n",
406 |     "    print(f'Writing TF Record file to: {tf_record_file_path} ...')\n",
407 |     "    writer = tf.data.experimental.TFRecordWriter(tf_record_file_path)\n",
408 |     "    writer.write(dataset)\n",
409 |     "\n",
410 |     "    return tf_record_file_path\n",
411 |     "\n",
412 |     "train_tfrec_path = write_json_to_tf_record_file(train_json)\n",
413 |     "dev_tfrec_path = write_json_to_tf_record_file(dev_json)"
414 |    ]
415 |   }
416 |  ],
417 |  "metadata": {
418 |   "colab": {
419 |    "collapsed_sections": [],
420 |    "name": "drop_tfrec_conv_local.ipynb",
421 |    "provenance": []
422 |   },
423 |   "kernelspec": {
424 |    "display_name": "Python 3",
425 |    "language": "python",
426 |    "name": "python3"
427 |   },
428 |   "language_info": {
429 |    "codemirror_mode": {
430 |     "name": "ipython",
431 |     "version": 3
432 |    },
433 |    "file_extension": ".py",
434 |    "mimetype": "text/x-python",
435 |    "name": "python",
436 |    "nbconvert_exporter": "python",
437 |    "pygments_lexer": "ipython3",
438 |    "version": "3.9.2"
439 |   }
440 |  },
441 |  "nbformat": 4,
442 |  "nbformat_minor": 4
443 | }
444 | 


--------------------------------------------------------------------------------
/tfrec/tfrec_drop_cat.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "id": "qp9-AReX2bt_"
  7 |    },
  8 |    "source": [
  9 |     "# T5 TFRecord Conversion\n",
 10 |     "\n",
 11 |     "This notebook preprocesses the DROP dataset in JSON format to a TFRecords dataset that can be streamed to the model for training."
 12 |    ]
 13 |   },
 14 |   {
 15 |    "cell_type": "code",
 16 |    "execution_count": null,
 17 |    "metadata": {
 18 |     "id": "Nfp8x5hOmrPI"
 19 |    },
 20 |    "outputs": [],
 21 |    "source": [
 22 |     "if 'colab' in str(get_ipython()):\n",
 23 |     "    import google.colab as colab\n",
 24 |     "    colab.auth.authenticate_user()\n",
 25 |     "    colab.drive.mount('/content/gdrive') # mount google drive\n",
 26 |     "\n",
 27 |     "    # install libraries not native to colab\n",
 28 |     "    !pip install tensorflow-text\n",
 29 |     "    !pip install transformers==3.3.1\n",
 30 |     "    !pip install datasets==1.1.2\n",
 31 |     "    !pip install tqdm\n",
 32 |     "\n",
 33 |     "# remove pip install outputs\n",
 34 |     "from IPython.display import clear_output\n",
 35 |     "clear_output()"
 36 |    ]
 37 |   },
 38 |   {
 39 |    "cell_type": "code",
 40 |    "execution_count": null,
 41 |    "metadata": {
 42 |     "colab": {
 43 |      "base_uri": "https://localhost:8080/"
 44 |     },
 45 |     "executionInfo": {
 46 |      "elapsed": 7649,
 47 |      "status": "ok",
 48 |      "timestamp": 1603953774465,
 49 |      "user": {
 50 |       "displayName": "Tim Chen",
 51 |       "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjCHzDArJyPPAHsYKioqVZeP1reZIsBL9RZDKwCOy8=s64",
 52 |       "userId": "17136390159455304905"
 53 |      },
 54 |      "user_tz": 420
 55 |     },
 56 |     "id": "Ls1fXWF3sLs1",
 57 |     "outputId": "6da152f6-0a57-4abf-fefe-7b06ea7c11e7"
 58 |    },
 59 |    "outputs": [
 60 |     {
 61 |      "name": "stdout",
 62 |      "output_type": "stream",
 63 |      "text": [
 64 |       "TensorFlow 2.2.0\n"
 65 |      ]
 66 |     }
 67 |    ],
 68 |    "source": [
 69 |     "# ml libraries\n",
 70 |     "import tensorflow as tf\n",
 71 |     "import tensorflow.keras as keras\n",
 72 |     "import tensorflow.keras.backend as K\n",
 73 |     "import transformers\n",
 74 |     "import datasets # https://huggingface.co/docs/datasets/\n",
 75 |     "\n",
 76 |     "# data processing libraries\n",
 77 |     "import numpy as np\n",
 78 |     "import pandas as pd\n",
 79 |     "import matplotlib.pyplot as plt\n",
 80 |     "from tqdm.auto import tqdm\n",
 81 |     "\n",
 82 |     "# other libraries\n",
 83 |     "import os\n",
 84 |     "import json\n",
 85 |     "import functools\n",
 86 |     "import time\n",
 87 |     "import warnings\n",
 88 |     "from datetime import datetime\n",
 89 |     "from typing import Generator\n",
 90 |     "\n",
 91 |     "print(f'TensorFlow {tf.__version__}')"
 92 |    ]
 93 |   },
 94 |   {
 95 |    "cell_type": "markdown",
 96 |    "metadata": {
 97 |     "id": "jY88Y3FzXUWN"
 98 |    },
 99 |    "source": [
100 |     "## Configurations"
101 |    ]
102 |   },
103 |   {
104 |    "cell_type": "code",
105 |    "execution_count": null,
106 |    "metadata": {
107 |     "executionInfo": {
108 |      "elapsed": 563,
109 |      "status": "ok",
110 |      "timestamp": 1603953774466,
111 |      "user": {
112 |       "displayName": "Tim Chen",
113 |       "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjCHzDArJyPPAHsYKioqVZeP1reZIsBL9RZDKwCOy8=s64",
114 |       "userId": "17136390159455304905"
115 |      },
116 |      "user_tz": 420
117 |     },
118 |     "id": "l1NeuclbwOB0",
119 |     "outputId": "bf98ad7d-cac0-4cb0-bf38-099a62b819c5"
120 |    },
121 |    "outputs": [
122 |     {
123 |      "name": "stdout",
124 |      "output_type": "stream",
125 |      "text": [
126 |       "New tokens added: 2\n"
127 |      ]
128 |     }
129 |    ],
130 |    "source": [
131 |     "#@title Constants\n",
132 |     "T5_MODEL                = 't5-small'  #@param { type: \"string\" }\n",
133 |     "ENCODER_MAX_LEN         = 512 #@param { type: \"integer\" }\n",
134 |     "DECODER_MAX_LEN         = 54 #@param { type: \"integer\" } \n",
135 |     "TOKENIZER               = transformers.AutoTokenizer.from_pretrained(T5_MODEL)\n",
136 |     "special_tokens_dict     = {'additional_special_tokens': ['0','1','2','3', '4', '5', '6', '7', '8', '9', '<ss>', '<sv>']}\n",
137 |     "num_added_toks          = TOKENIZER.add_special_tokens(special_tokens_dict)\n",
138 |     "print(f'New tokens added: {num_added_toks}')\n",
139 |     "\n",
140 |     "# tim's default paths\n",
141 |     "LOCAL_PATH              = '' # local folder containing DROP json\n",
142 |     "TRAIN_JSON              = 'drop_dataset_train.json' \n",
143 |     "DEV_JSON                = 'drop_dataset_dev.json' \n",
144 |     "DROP_JSON_DIR_LOCAL     = LOCAL_PATH + '/data/drop_dataset' #@param { type: \"string\" }\n",
145 |     "DROP_JSON_DIR_GCP       = '' # gcp storage path for saving tfrecords #@param { type: \"string\" }\n",
146 |     "EXPORT_DIR              = LOCAL_PATH + '/temp' #@param { type: \"string\" }\n",
147 |     "\n",
148 |     "TRAIN_DROP_JSON_GCP     = os.path.join(DROP_JSON_DIR_GCP, TRAIN_JSON)\n",
149 |     "DEV_DROP_JSON_GCP       = os.path.join(DROP_JSON_DIR_GCP, DEV_JSON) "
150 |    ]
151 |   },
152 |   {
153 |    "cell_type": "code",
154 |    "execution_count": null,
155 |    "metadata": {
156 |     "id": "LEERtyO0meh-"
157 |    },
158 |    "outputs": [],
159 |    "source": [
160 |     "#@title Test Mode Setup\n",
161 |     "def sample_data_to_file(\n",
162 |     "                json_file: str,\n",
163 |     "                sample_count: int,\n",
164 |     "                source_dir: str, \n",
165 |     "                export_dir: str = EXPORT_DIR) -> str:\n",
166 |     "    \n",
167 |     "    # load source json\n",
168 |     "    source_file_path = os.path.join(source_dir, json_file)\n",
169 |     "    with tf.io.gfile.GFile(source_file_path) as f:\n",
170 |     "        data_json_dict = json.load(f)  # dict of dict\n",
171 |     "        sampled_data_json_dict = {\n",
172 |     "            key: value\n",
173 |     "            for i, (key, value) in enumerate(data_json_dict.items())\n",
174 |     "            if i < sample_count\n",
175 |     "        }\n",
176 |     "\n",
177 |     "    # prep export json file path\n",
178 |     "    sample_count = len(sampled_data_json_dict)  # get the real sample count\n",
179 |     "    sampled_output_path = os.path.join(export_dir, json_file)\n",
180 |     "    sampled_output_path = sampled_output_path.replace('.json', f'{sample_count}.json')\n",
181 |     "\n",
182 |     "    # export sample json\n",
183 |     "    with tf.io.gfile.GFile(sampled_output_path, 'w') as fout:\n",
184 |     "        json.dump(sampled_data_json_dict, fout)\n",
185 |     "\n",
186 |     "    print(f'Dumped {sample_count} samples to file: {sampled_output_path}')\n",
187 |     "    return sampled_output_path\n",
188 |     "\n",
189 |     "SAMPLE_DATA = False  #@param { type: \"boolean\" }\n",
190 |     "SAMPLE_COUNT = 3 #@param { type: \"integer\" }\n",
191 |     "if SAMPLE_DATA:\n",
192 |     "    train_json = sample_data_to_file(\n",
193 |     "                    json_file=TRAIN_JSON, \n",
194 |     "                    sample_count=SAMPLE_COUNT,\n",
195 |     "                    source_dir=DROP_JSON_DIR_LOCAL,\n",
196 |     "                    export_dir=EXPORT_DIR)\n",
197 |     "    dev_json = sample_data_to_file(\n",
198 |     "                    json_file=DEV_JSON, \n",
199 |     "                    sample_count=SAMPLE_COUNT,\n",
200 |     "                    source_dir=DROP_JSON_DIR_LOCAL,\n",
201 |     "                    export_dir=EXPORT_DIR)\n",
202 |     "else:\n",
203 |     "    train_json = os.path.join(DROP_JSON_DIR_LOCAL, TRAIN_JSON)\n",
204 |     "    dev_json = os.path.join(DROP_JSON_DIR_LOCAL, DEV_JSON)\n",
205 |     "    print(train_json)\n",
206 |     "    print(dev_json)"
207 |    ]
208 |   },
209 |   {
210 |    "cell_type": "markdown",
211 |    "metadata": {
212 |     "id": "dOBK366_aq6P"
213 |    },
214 |    "source": [
215 |     "## Encoding TFRecord"
216 |    ]
217 |   },
218 |   {
219 |    "cell_type": "code",
220 |    "execution_count": null,
221 |    "metadata": {
222 |     "id": "NIHhIzcsF1N3"
223 |    },
224 |    "outputs": [],
225 |    "source": [
226 |     "def encode(\n",
227 |     "        example: dict,\n",
228 |     "        encoder_max_len: int = ENCODER_MAX_LEN, \n",
229 |     "        decoder_max_len: int = DECODER_MAX_LEN,\n",
230 |     "        tokenizer: transformers.PreTrainedTokenizer = TOKENIZER) -> dict:\n",
231 |     "    \"\"\"Tokenize data.\n",
232 |     "    Args:\n",
233 |     "        example (dict): Raw dict parsed from DROP json:\n",
234 |     "                            example['context']\n",
235 |     "                            example['question']\n",
236 |     "                            example['answer']\n",
237 |     "    Returns: \n",
238 |     "        (dict) Dictionary with values tokenized:\n",
239 |     "                            return['input_ids']\n",
240 |     "                            return['attention_mask']\n",
241 |     "                            return['labels']\n",
242 |     "                            return['decoder_attention_mask']\n",
243 |     "    \"\"\"            \n",
244 |     "    context = example['context']\n",
245 |     "    question = example['question']\n",
246 |     "    answer_type = example['answer_type']\n",
247 |     "  \n",
248 |     "    question_plus = f\"classify_me: {str(question)} context: {str(context)}\"\n",
249 |     "    answer_plus = answer_type\n",
250 |     "\n",
251 |     "    encoder_inputs = tokenizer(\n",
252 |     "                            question_plus, \n",
253 |     "                            truncation=True, \n",
254 |     "                            return_tensors='tf', \n",
255 |     "                            max_length=encoder_max_len,\n",
256 |     "                            padding='max_length')\n",
257 |     "    decoder_inputs = tokenizer(\n",
258 |     "                            answer_plus, \n",
259 |     "                            truncation=True, \n",
260 |     "                            return_tensors='tf', \n",
261 |     "                            max_length=decoder_max_len,\n",
262 |     "                            padding='max_length')\n",
263 |     "    \n",
264 |     "    return {\n",
265 |     "        'query_id'                  : [str.encode(example['query_id'])], \n",
266 |     "        'answer_type'               : [str.encode(example['answer_type'])], \n",
267 |     "        'validated_answers'         : [str.encode(example['validated_answers'])],\n",
268 |     "        'input_ids'                 : encoder_inputs['input_ids'][0], \n",
269 |     "        'attention_mask'            : encoder_inputs['attention_mask'][0], \n",
270 |     "        'labels'                    : decoder_inputs['input_ids'][0], \n",
271 |     "        'decoder_attention_mask'    : decoder_inputs['attention_mask'][0]}\n",
272 |     "\n",
273 |     "def _bytes_feature(values):\n",
274 |     "  \"\"\"Returns a bytes_list from a list of string / byte.\"\"\"\n",
275 |     "  return tf.train.Feature(bytes_list=tf.train.BytesList(value=values))\n",
276 |     "\n",
277 |     "def _float_feature(values):\n",
278 |     "  \"\"\"Returns a float_list from a list of float / double.\"\"\"\n",
279 |     "  return tf.train.Feature(float_list=tf.train.FloatList(value=values))\n",
280 |     "\n",
281 |     "def _int64_feature(values):\n",
282 |     "    \"\"\"Returns an int64_list from a list of bool / enum / int / uint.\"\"\"\n",
283 |     "    try:\n",
284 |     "        return tf.train.Feature(int64_list=tf.train.Int64List(value=values))\n",
285 |     "    except Exception as e:\n",
286 |     "        print(f'failed at: {values}')\n",
287 |     "        raise e\n",
288 |     "\n",
289 |     "def to_serialized_example(encoded_example):\n",
290 |     "    feature = {\n",
291 |     "        'query_id': _bytes_feature(encoded_example['query_id']),\n",
292 |     "        'answer_type': _bytes_feature(encoded_example['answer_type']),\n",
293 |     "        'validated_answers': _bytes_feature(encoded_example['validated_answers']),\n",
294 |     "        'input_ids': _int64_feature(encoded_example['input_ids']),\n",
295 |     "        'attention_mask': _int64_feature(encoded_example['attention_mask']),\n",
296 |     "        'labels': _int64_feature(encoded_example['labels']),\n",
297 |     "        'decoder_attention_mask': _int64_feature(encoded_example['decoder_attention_mask']),\n",
298 |     "    }\n",
299 |     "    example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n",
300 |     "    return example_proto.SerializeToString()\n",
301 |     "\n",
302 |     "def parse_validated_answers(validated_answers: list) -> str:\n",
303 |     "    answers = []\n",
304 |     "    for answer_dict in validated_answers:\n",
305 |     "        ans, _ = parse_answer(answer_dict)\n",
306 |     "        answers.append(ans)\n",
307 |     "    return '<sv>'.join([a.strip() for a in answers if a.strip()])\n",
308 |     "        \n",
309 |     "def parse_answer(answer_dict: dict) -> str:\n",
310 |     "    \"\"\"\n",
311 |     "    Example of answer_dict = { \"number\": \"\", \"date\": { \"day\": \"\", \"month\": \"\", \"year\": \"\" }, \"spans\": [ \"Chaz Schilens\" ] }\n",
312 |     "    Returns a string or None.\n",
313 |     "    \"\"\"\n",
314 |     "    number = answer_dict['number'].strip()\n",
315 |     "    if number:\n",
316 |     "        return number, 'n'\n",
317 |     "    \n",
318 |     "    spans = answer_dict['spans']\n",
319 |     "    spans_str = '<ss>'.join([span.strip() for span in spans if span.strip()])\n",
320 |     "    if spans_str:\n",
321 |     "        if len(spans) > 1:\n",
322 |     "            return spans_str, 's'\n",
323 |     "        else:\n",
324 |     "            return spans_str, 'ss'\n",
325 |     "    \n",
326 |     "    date = answer_dict['date']\n",
327 |     "    if len(date) != 3:\n",
328 |     "        return None, None\n",
329 |     "    date = ' '.join([d.strip() for d in [date['day'], date['month'], date['year']] if d.strip()])\n",
330 |     "    if date:\n",
331 |     "        return date, 'd'\n",
332 |     "    \n",
333 |     "    return None, None\n",
334 |     "\n",
335 |     "def make_data_generator(file_path: str) -> Generator:\n",
336 |     "    \"\"\"\n",
337 |     "    json_example: {\n",
338 |     "        \"passage\": \"foo bar\",\n",
339 |     "        \"qa_pairs\": [\n",
340 |     "            {\n",
341 |     "                \"question\": \"foo bar\",\n",
342 |     "                \"answer\": { \"number\": \"\", \"date\": { \"day\": \"\", \"month\": \"\", \"year\": \"\" }, \"spans\": [ \"Chaz Schilens\" ] },\n",
343 |     "                \"query_id\": \"f37e81fa-ef7b-4583-b671-762fc433faa9\",\n",
344 |     "            }\n",
345 |     "        ]\n",
346 |     "    }\n",
347 |     "\n",
348 |     "    Returns a generator function.\n",
349 |     "    \"\"\"\n",
350 |     "    with tf.io.gfile.GFile(file_path) as json_file:\n",
351 |     "        data_json_dict = json.load(json_file) \n",
352 |     "\n",
353 |     "    def gen():\n",
354 |     "        for key, json_example in data_json_dict.items():\n",
355 |     "            passage = json_example['passage']\n",
356 |     "            for qa_dict in json_example['qa_pairs']:\n",
357 |     "                \n",
358 |     "                question = qa_dict['question']\n",
359 |     "                answer, answer_type = parse_answer(qa_dict['answer'])\n",
360 |     "                query_id = qa_dict['query_id']\n",
361 |     "                if 'validated_answers' in qa_dict:\n",
362 |     "                    valid_ans = parse_validated_answers(qa_dict['validated_answers'])\n",
363 |     "                else:\n",
364 |     "                    valid_ans = ''                \n",
365 |     "                if answer is None:\n",
366 |     "                    continue\n",
367 |     "\n",
368 |     "                encoded_example = encode({\n",
369 |     "                    'context': passage,\n",
370 |     "                    'question': question,\n",
371 |     "                    'answer': answer,\n",
372 |     "                    'validated_answers': valid_ans,\n",
373 |     "                    'answer_type': answer_type,\n",
374 |     "                    'query_id': query_id\n",
375 |     "                })\n",
376 |     "                serialized_example = to_serialized_example(encoded_example)\n",
377 |     "                \n",
378 |     "                yield serialized_example\n",
379 |     "    \n",
380 |     "    return gen\n",
381 |     "\n",
382 |     "def get_num_examples(file_path):\n",
383 |     "    with tf.io.gfile.GFile(file_path) as json_file:\n",
384 |     "        data_json_dict = json.load(json_file) \n",
385 |     "        return len(data_json_dict)\n",
386 |     "\n",
387 |     "def load_dataset(file_path: str) -> tf.data.Dataset:\n",
388 |     "    return tf.data.Dataset.from_generator(\n",
389 |     "                        make_data_generator(file_path),\n",
390 |     "                        output_types=tf.string)"
391 |    ]
392 |   },
393 |   {
394 |    "cell_type": "code",
395 |    "execution_count": null,
396 |    "metadata": {
397 |     "id": "VGT2hPS9HJ8E"
398 |    },
399 |    "outputs": [],
400 |    "source": [
401 |     "SUFFIX = '_v4'\n",
402 |     "def write_json_to_tf_record_file(json_path: str) -> str:\n",
403 |     "    dataset = load_dataset(json_path)\n",
404 |     "    tf_record_file_path = os.path.join(\n",
405 |     "        os.path.dirname(json_path),\n",
406 |     "        os.path.basename(json_path).replace('.json', f'{SUFFIX}.tfrecord')\n",
407 |     "    )\n",
408 |     "\n",
409 |     "    print(f'Writing TF Record file to: {tf_record_file_path} ...')\n",
410 |     "    writer = tf.data.experimental.TFRecordWriter(tf_record_file_path)\n",
411 |     "    writer.write(dataset)\n",
412 |     "\n",
413 |     "    return tf_record_file_path\n",
414 |     "\n",
415 |     "train_tfrec_path = write_json_to_tf_record_file(train_json)\n",
416 |     "dev_tfrec_path = write_json_to_tf_record_file(dev_json)"
417 |    ]
418 |   }
419 |  ],
420 |  "metadata": {
421 |   "colab": {
422 |    "collapsed_sections": [],
423 |    "name": "tfrec_drop_cat.ipynb",
424 |    "provenance": []
425 |   },
426 |   "kernelspec": {
427 |    "display_name": "Python 3",
428 |    "language": "python",
429 |    "name": "python3"
430 |   },
431 |   "language_info": {
432 |    "codemirror_mode": {
433 |     "name": "ipython",
434 |     "version": 3
435 |    },
436 |    "file_extension": ".py",
437 |    "mimetype": "text/x-python",
438 |    "name": "python",
439 |    "nbconvert_exporter": "python",
440 |    "pygments_lexer": "ipython3",
441 |    "version": "3.9.2"
442 |   }
443 |  },
444 |  "nbformat": 4,
445 |  "nbformat_minor": 4
446 | }
447 | 


--------------------------------------------------------------------------------
/tfrec/tfrec_drop_num.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "id": "qp9-AReX2bt_"
  7 |    },
  8 |    "source": [
  9 |     "# T5 TFRecord Preprocessing\n",
 10 |     "\n",
 11 |     "This notebook preprocesses the DROP dataset in JSON format to a TFRecords dataset that can be fed into the model after batching for training."
 12 |    ]
 13 |   },
 14 |   {
 15 |    "cell_type": "code",
 16 |    "execution_count": null,
 17 |    "metadata": {
 18 |     "id": "Nfp8x5hOmrPI"
 19 |    },
 20 |    "outputs": [],
 21 |    "source": [
 22 |     "if 'colab' in str(get_ipython()):\n",
 23 |     "    import google.colab as colab\n",
 24 |     "    colab.auth.authenticate_user()\n",
 25 |     "    colab.drive.mount('/content/gdrive') # mount google drive\n",
 26 |     "\n",
 27 |     "    # install libraries not native to colab\n",
 28 |     "    !pip install tensorflow-text\n",
 29 |     "    !pip install transformers==3.3.1\n",
 30 |     "    !pip install datasets==1.1.2\n",
 31 |     "    !pip install tqdm\n",
 32 |     "\n",
 33 |     "# remove pip install outputs\n",
 34 |     "from IPython.display import clear_output\n",
 35 |     "clear_output()"
 36 |    ]
 37 |   },
 38 |   {
 39 |    "cell_type": "code",
 40 |    "execution_count": null,
 41 |    "metadata": {
 42 |     "colab": {
 43 |      "base_uri": "https://localhost:8080/"
 44 |     },
 45 |     "executionInfo": {
 46 |      "elapsed": 7649,
 47 |      "status": "ok",
 48 |      "timestamp": 1603953774465,
 49 |      "user": {
 50 |       "displayName": "Tim Chen",
 51 |       "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjCHzDArJyPPAHsYKioqVZeP1reZIsBL9RZDKwCOy8=s64",
 52 |       "userId": "17136390159455304905"
 53 |      },
 54 |      "user_tz": 420
 55 |     },
 56 |     "id": "Ls1fXWF3sLs1",
 57 |     "outputId": "6da152f6-0a57-4abf-fefe-7b06ea7c11e7"
 58 |    },
 59 |    "outputs": [
 60 |     {
 61 |      "name": "stdout",
 62 |      "output_type": "stream",
 63 |      "text": [
 64 |       "TensorFlow 2.2.0\n"
 65 |      ]
 66 |     }
 67 |    ],
 68 |    "source": [
 69 |     "# ml libraries\n",
 70 |     "import tensorflow as tf\n",
 71 |     "import tensorflow.keras as keras\n",
 72 |     "import tensorflow.keras.backend as K\n",
 73 |     "import transformers\n",
 74 |     "import datasets # https://huggingface.co/docs/datasets/\n",
 75 |     "\n",
 76 |     "# data processing libraries\n",
 77 |     "import numpy as np\n",
 78 |     "import pandas as pd\n",
 79 |     "import matplotlib.pyplot as plt\n",
 80 |     "from tqdm.auto import tqdm\n",
 81 |     "\n",
 82 |     "# other libraries\n",
 83 |     "import os\n",
 84 |     "import json\n",
 85 |     "import functools\n",
 86 |     "import time\n",
 87 |     "import warnings\n",
 88 |     "from datetime import datetime\n",
 89 |     "from typing import Generator\n",
 90 |     "\n",
 91 |     "print(f'TensorFlow {tf.__version__}')"
 92 |    ]
 93 |   },
 94 |   {
 95 |    "cell_type": "markdown",
 96 |    "metadata": {
 97 |     "id": "jY88Y3FzXUWN"
 98 |    },
 99 |    "source": [
100 |     "## Configurations"
101 |    ]
102 |   },
103 |   {
104 |    "cell_type": "code",
105 |    "execution_count": null,
106 |    "metadata": {
107 |     "executionInfo": {
108 |      "elapsed": 563,
109 |      "status": "ok",
110 |      "timestamp": 1603953774466,
111 |      "user": {
112 |       "displayName": "Tim Chen",
113 |       "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjCHzDArJyPPAHsYKioqVZeP1reZIsBL9RZDKwCOy8=s64",
114 |       "userId": "17136390159455304905"
115 |      },
116 |      "user_tz": 420
117 |     },
118 |     "id": "l1NeuclbwOB0",
119 |     "outputId": "d9983b62-439b-42ad-9807-746e37fab9e2"
120 |    },
121 |    "outputs": [
122 |     {
123 |      "name": "stdout",
124 |      "output_type": "stream",
125 |      "text": [
126 |       "New tokens added: 2\n"
127 |      ]
128 |     }
129 |    ],
130 |    "source": [
131 |     "#@title Constants\n",
132 |     "T5_MODEL                = 't5-small'  #@param { type: \"string\" }\n",
133 |     "ENCODER_MAX_LEN         = 512 #@param { type: \"integer\" }\n",
134 |     "DECODER_MAX_LEN         = 54 #@param { type: \"integer\" } \n",
135 |     "TOKENIZER               = transformers.AutoTokenizer.from_pretrained(T5_MODEL)\n",
136 |     "special_tokens_dict     = {'additional_special_tokens': ['0','1','2','3', '4', '5', '6', '7', '8', '9', '<ss>', '<sv>']}\n",
137 |     "num_added_toks          = TOKENIZER.add_special_tokens(special_tokens_dict)\n",
138 |     "print(f'New tokens added: {num_added_toks}')\n",
139 |     "\n",
140 |     "# tim's default paths\n",
141 |     "LOCAL_PATH              = '' # local folder containing DROP json\n",
142 |     "TRAIN_JSON              = 'drop_dataset_train.json' \n",
143 |     "DEV_JSON                = 'drop_dataset_dev.json' \n",
144 |     "DROP_JSON_DIR_LOCAL     = LOCAL_PATH + '/data/drop_dataset' #@param { type: \"string\" }\n",
145 |     "DROP_JSON_DIR_GCP       = '' # gcp storage path for saving tfrecords #@param { type: \"string\" }\n",
146 |     "EXPORT_DIR              = LOCAL_PATH + '/temp' #@param { type: \"string\" }\n",
147 |     "\n",
148 |     "TRAIN_DROP_JSON_GCP     = os.path.join(DROP_JSON_DIR_GCP, TRAIN_JSON)\n",
149 |     "DEV_DROP_JSON_GCP       = os.path.join(DROP_JSON_DIR_GCP, DEV_JSON) "
150 |    ]
151 |   },
152 |   {
153 |    "cell_type": "code",
154 |    "execution_count": null,
155 |    "metadata": {},
156 |    "outputs": [],
157 |    "source": [
158 |     "#@title Test Mode Setup\n",
159 |     "def sample_data_to_file(\n",
160 |     "                json_file: str,\n",
161 |     "                sample_count: int,\n",
162 |     "                source_dir: str, \n",
163 |     "                export_dir: str = EXPORT_DIR) -> str:\n",
164 |     "    \n",
165 |     "    # load source json\n",
166 |     "    source_file_path = os.path.join(source_dir, json_file)\n",
167 |     "    with tf.io.gfile.GFile(source_file_path) as f:\n",
168 |     "        data_json_dict = json.load(f)  # dict of dict\n",
169 |     "        sampled_data_json_dict = {\n",
170 |     "            key: value\n",
171 |     "            for i, (key, value) in enumerate(data_json_dict.items())\n",
172 |     "            if i < sample_count\n",
173 |     "        }\n",
174 |     "\n",
175 |     "    # prep export json file path\n",
176 |     "    sample_count = len(sampled_data_json_dict)  # get the real sample count\n",
177 |     "    sampled_output_path = os.path.join(export_dir, json_file)\n",
178 |     "    sampled_output_path = sampled_output_path.replace('.json', f'{sample_count}.json')\n",
179 |     "\n",
180 |     "    # export sample json\n",
181 |     "    with tf.io.gfile.GFile(sampled_output_path, 'w') as fout:\n",
182 |     "        json.dump(sampled_data_json_dict, fout)\n",
183 |     "\n",
184 |     "    print(f'Dumped {sample_count} samples to file: {sampled_output_path}')\n",
185 |     "    return sampled_output_path\n",
186 |     "\n",
187 |     "SAMPLE_DATA = False  #@param { type: \"boolean\" }\n",
188 |     "SAMPLE_COUNT = 3 #@param { type: \"integer\" }\n",
189 |     "if SAMPLE_DATA:\n",
190 |     "    train_json = sample_data_to_file(\n",
191 |     "                    json_file=TRAIN_JSON, \n",
192 |     "                    sample_count=SAMPLE_COUNT,\n",
193 |     "                    source_dir=DROP_JSON_DIR_LOCAL,\n",
194 |     "                    export_dir=EXPORT_DIR)\n",
195 |     "    dev_json = sample_data_to_file(\n",
196 |     "                    json_file=DEV_JSON, \n",
197 |     "                    sample_count=SAMPLE_COUNT,\n",
198 |     "                    source_dir=DROP_JSON_DIR_LOCAL,\n",
199 |     "                    export_dir=EXPORT_DIR)\n",
200 |     "else:\n",
201 |     "    train_json = os.path.join(DROP_JSON_DIR_LOCAL, TRAIN_JSON)\n",
202 |     "    dev_json = os.path.join(DROP_JSON_DIR_LOCAL, DEV_JSON)\n",
203 |     "    print(train_json)\n",
204 |     "    print(dev_json)"
205 |    ]
206 |   },
207 |   {
208 |    "cell_type": "markdown",
209 |    "metadata": {
210 |     "id": "dOBK366_aq6P"
211 |    },
212 |    "source": [
213 |     "## Encoding TFRecord"
214 |    ]
215 |   },
216 |   {
217 |    "cell_type": "code",
218 |    "execution_count": null,
219 |    "metadata": {
220 |     "id": "LbN9VKGSe8BG"
221 |    },
222 |    "outputs": [],
223 |    "source": [
224 |     "# define function: load_dataset()\n",
225 |     "# functions to convert values to types compatible with tf.train.Example\n",
226 |     "\n",
227 |     "def encode(\n",
228 |     "        example: dict,\n",
229 |     "        encoder_max_len: int = ENCODER_MAX_LEN, \n",
230 |     "        decoder_max_len: int = DECODER_MAX_LEN,\n",
231 |     "        tokenizer: transformers.PreTrainedTokenizer = TOKENIZER) -> dict:\n",
232 |     "    \"\"\"Tokenize data.\n",
233 |     "    Args:\n",
234 |     "        example (dict): Raw dict parsed from DROP json:\n",
235 |     "                            example['context']\n",
236 |     "                            example['question']\n",
237 |     "                            example['answer']\n",
238 |     "    Returns: \n",
239 |     "        (dict) Dictionary with values tokenized:\n",
240 |     "                            return['input_ids']\n",
241 |     "                            return['attention_mask']\n",
242 |     "                            return['labels']\n",
243 |     "                            return['decoder_attention_mask']\n",
244 |     "    \"\"\"            \n",
245 |     "    context = example['context']\n",
246 |     "    question = example['question']\n",
247 |     "    answer_type = example['answer_type']\n",
248 |     "  \n",
249 |     "    question_plus = f\"classify_math: {str(question)} context: {str(context)}\"\n",
250 |     "    answer_plus = answer_type\n",
251 |     "\n",
252 |     "    encoder_inputs = tokenizer(\n",
253 |     "                            question_plus, \n",
254 |     "                            truncation=True, \n",
255 |     "                            return_tensors='tf', \n",
256 |     "                            max_length=encoder_max_len,\n",
257 |     "                            padding='max_length')\n",
258 |     "    decoder_inputs = tokenizer(\n",
259 |     "                            answer_plus, \n",
260 |     "                            truncation=True, \n",
261 |     "                            return_tensors='tf', \n",
262 |     "                            max_length=decoder_max_len,\n",
263 |     "                            padding='max_length')\n",
264 |     "    \n",
265 |     "    return {\n",
266 |     "        'query_id'                  : [str.encode(example['query_id'])], \n",
267 |     "        'answer_type'               : [str.encode(example['answer_type'])], \n",
268 |     "        'validated_answers'         : [str.encode(example['validated_answers'])],\n",
269 |     "        'input_ids'                 : encoder_inputs['input_ids'][0], \n",
270 |     "        'attention_mask'            : encoder_inputs['attention_mask'][0], \n",
271 |     "        'labels'                    : decoder_inputs['input_ids'][0], \n",
272 |     "        'decoder_attention_mask'    : decoder_inputs['attention_mask'][0]}\n",
273 |     "\n",
274 |     "def _bytes_feature(values):\n",
275 |     "  \"\"\"Returns a bytes_list from a list of string / byte.\"\"\"\n",
276 |     "  return tf.train.Feature(bytes_list=tf.train.BytesList(value=values))\n",
277 |     "\n",
278 |     "def _float_feature(values):\n",
279 |     "  \"\"\"Returns a float_list from a list of float / double.\"\"\"\n",
280 |     "  return tf.train.Feature(float_list=tf.train.FloatList(value=values))\n",
281 |     "\n",
282 |     "def _int64_feature(values):\n",
283 |     "    \"\"\"Returns an int64_list from a list of bool / enum / int / uint.\"\"\"\n",
284 |     "    try:\n",
285 |     "        return tf.train.Feature(int64_list=tf.train.Int64List(value=values))\n",
286 |     "    except Exception as e:\n",
287 |     "        print(f'failed at: {values}')\n",
288 |     "        raise e\n",
289 |     "\n",
290 |     "def to_serialized_example(encoded_example):\n",
291 |     "    feature = {\n",
292 |     "        'query_id': _bytes_feature(encoded_example['query_id']),\n",
293 |     "        'answer_type': _bytes_feature(encoded_example['answer_type']),\n",
294 |     "        'validated_answers': _bytes_feature(encoded_example['validated_answers']),\n",
295 |     "        'input_ids': _int64_feature(encoded_example['input_ids']),\n",
296 |     "        'attention_mask': _int64_feature(encoded_example['attention_mask']),\n",
297 |     "        'labels': _int64_feature(encoded_example['labels']),\n",
298 |     "        'decoder_attention_mask': _int64_feature(encoded_example['decoder_attention_mask']),\n",
299 |     "    }\n",
300 |     "    example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n",
301 |     "    return example_proto.SerializeToString()\n",
302 |     "\n",
303 |     "def parse_validated_answers(validated_answers: list) -> str:\n",
304 |     "    answers = []\n",
305 |     "    for answer_dict in validated_answers:\n",
306 |     "        ans, _ = parse_answer(answer_dict)\n",
307 |     "        answers.append(ans)\n",
308 |     "    return '<sv>'.join([a.strip() for a in answers if a.strip()])\n",
309 |     "        \n",
310 |     "def parse_answer(answer_dict: dict) -> str:\n",
311 |     "    \"\"\"\n",
312 |     "    Example of answer_dict = { \"number\": \"\", \"date\": { \"day\": \"\", \"month\": \"\", \"year\": \"\" }, \"spans\": [ \"Chaz Schilens\" ] }\n",
313 |     "    Returns a string or None.\n",
314 |     "    \"\"\"\n",
315 |     "    number = answer_dict['number'].strip()\n",
316 |     "    if number:\n",
317 |     "        return number, 'math'\n",
318 |     "    \n",
319 |     "    spans = answer_dict['spans']\n",
320 |     "    spans_str = '<ss>'.join([span.strip() for span in spans if span.strip()])\n",
321 |     "    if spans_str:\n",
322 |     "        if len(spans) > 1:\n",
323 |     "            return spans_str, 'no_math'\n",
324 |     "        else:\n",
325 |     "            return spans_str, 'no_math'\n",
326 |     "    \n",
327 |     "    date = answer_dict['date']\n",
328 |     "    if len(date) != 3:\n",
329 |     "        return None, None\n",
330 |     "    date = ' '.join([d.strip() for d in [date['day'], date['month'], date['year']] if d.strip()])\n",
331 |     "    if date:\n",
332 |     "        return date, 'no_math'\n",
333 |     "    \n",
334 |     "    return None, None\n",
335 |     "\n",
336 |     "def make_data_generator(file_path: str) -> Generator:\n",
337 |     "    \"\"\"\n",
338 |     "    json_example: {\n",
339 |     "        \"passage\": \"foo bar\",\n",
340 |     "        \"qa_pairs\": [\n",
341 |     "            {\n",
342 |     "                \"question\": \"foo bar\",\n",
343 |     "                \"answer\": { \"number\": \"\", \"date\": { \"day\": \"\", \"month\": \"\", \"year\": \"\" }, \"spans\": [ \"Chaz Schilens\" ] },\n",
344 |     "                \"query_id\": \"f37e81fa-ef7b-4583-b671-762fc433faa9\",\n",
345 |     "            }\n",
346 |     "        ]\n",
347 |     "    }\n",
348 |     "\n",
349 |     "    Returns a generator function.\n",
350 |     "    \"\"\"\n",
351 |     "    with tf.io.gfile.GFile(file_path) as json_file:\n",
352 |     "        data_json_dict = json.load(json_file) \n",
353 |     "\n",
354 |     "    def gen():\n",
355 |     "        for key, json_example in data_json_dict.items():\n",
356 |     "            passage = json_example['passage']\n",
357 |     "            for qa_dict in json_example['qa_pairs']:\n",
358 |     "                \n",
359 |     "                question = qa_dict['question']\n",
360 |     "                answer, answer_type = parse_answer(qa_dict['answer'])\n",
361 |     "                query_id = qa_dict['query_id']\n",
362 |     "                if 'validated_answers' in qa_dict:\n",
363 |     "                    valid_ans = parse_validated_answers(qa_dict['validated_answers'])\n",
364 |     "                else:\n",
365 |     "                    valid_ans = ''                \n",
366 |     "                if answer is None:\n",
367 |     "                    continue\n",
368 |     "\n",
369 |     "                encoded_example = encode({\n",
370 |     "                    'context': passage,\n",
371 |     "                    'question': question,\n",
372 |     "                    'answer': answer,\n",
373 |     "                    'validated_answers': valid_ans,\n",
374 |     "                    'answer_type': answer_type,\n",
375 |     "                    'query_id': query_id\n",
376 |     "                })\n",
377 |     "                serialized_example = to_serialized_example(encoded_example)\n",
378 |     "                \n",
379 |     "                yield serialized_example\n",
380 |     "    \n",
381 |     "    return gen\n",
382 |     "\n",
383 |     "def get_num_examples(file_path):\n",
384 |     "    with tf.io.gfile.GFile(file_path) as json_file:\n",
385 |     "        data_json_dict = json.load(json_file) \n",
386 |     "        return len(data_json_dict)\n",
387 |     "\n",
388 |     "def load_dataset(file_path: str) -> tf.data.Dataset:\n",
389 |     "    return tf.data.Dataset.from_generator(\n",
390 |     "                        make_data_generator(file_path),\n",
391 |     "                        output_types=tf.string)"
392 |    ]
393 |   },
394 |   {
395 |    "cell_type": "code",
396 |    "execution_count": null,
397 |    "metadata": {
398 |     "id": "VGT2hPS9HJ8E"
399 |    },
400 |    "outputs": [],
401 |    "source": [
402 |     "SUFFIX = '_v4_catmath'\n",
403 |     "def write_json_to_tf_record_file(json_path: str) -> str:\n",
404 |     "    dataset = load_dataset(json_path)\n",
405 |     "    tf_record_file_path = os.path.join(\n",
406 |     "        os.path.dirname(json_path),\n",
407 |     "        os.path.basename(json_path).replace('.json', f'{SUFFIX}.tfrecord')\n",
408 |     "    )\n",
409 |     "\n",
410 |     "    print(f'Writing TF Record file to: {tf_record_file_path} ...')\n",
411 |     "    writer = tf.data.experimental.TFRecordWriter(tf_record_file_path)\n",
412 |     "    writer.write(dataset)\n",
413 |     "\n",
414 |     "    return tf_record_file_path\n",
415 |     "\n",
416 |     "train_tfrec_path = write_json_to_tf_record_file(train_json)\n",
417 |     "dev_tfrec_path = write_json_to_tf_record_file(dev_json)"
418 |    ]
419 |   }
420 |  ],
421 |  "metadata": {
422 |   "colab": {
423 |    "collapsed_sections": [],
424 |    "name": "drop_num_tfrec_conv_local.ipynb",
425 |    "provenance": []
426 |   },
427 |   "kernelspec": {
428 |    "display_name": "Python 3",
429 |    "language": "python",
430 |    "name": "python3"
431 |   },
432 |   "language_info": {
433 |    "codemirror_mode": {
434 |     "name": "ipython",
435 |     "version": 3
436 |    },
437 |    "file_extension": ".py",
438 |    "mimetype": "text/x-python",
439 |    "name": "python",
440 |    "nbconvert_exporter": "python",
441 |    "pygments_lexer": "ipython3",
442 |    "version": "3.9.2"
443 |   }
444 |  },
445 |  "nbformat": 4,
446 |  "nbformat_minor": 4
447 | }
448 | 


--------------------------------------------------------------------------------
/error_analysis/rc_exp_1_stage_3_5_e60_drop/eval_reports_valid_set/ea_spans_sample_100.csv:
--------------------------------------------------------------------------------
  1 | ,query_id,prediction,answer,validated_answers,f1,exact_match
  2 | 191,c9f24c25-5211-4027-9fa6-28a0e9b61b55,heart disease <ss> hypertension,diabetes<ss>hypertension,"diabetes<ss>hypertension<sv>4,300,252 cases of hypertension",0.5,0.0
  3 | 143,ba04b5c7-b0ea-45c4-81c6-e32db9d323d8,May <ss> May <ss> May,June<ss>August,June<ss>August<sv>June<ss>August,0.0,0.0
  4 | 337,401be1e9-c029-4d74-b2d5-b992da8eb935,France <ss> Italy <ss> Germany <ss> Spain <ss> U.S.A.4,France<ss>Italy,France<ss>Italy<sv>France<ss>Italy,0.4,0.0
  5 | 291,eb1cf33e-6dba-447f-946c-7de2e73e0be4,pigs <ss> pigs,Cattle<ss>wild boar,pigs<sv>Cattle<ss>wild boar,0.5,0.0
  6 | 96,63e0012e-22e6-4ff5-9ee9-67201e73af3a,Dutch Reformed Church <ss> Protestant Church,Evangelical Lutheran Church<ss>The Dutch Reformed Church<ss>the Roman Catholic Church,Evangelical Lutheran Church<ss>Dutch Reformed Church<sv>Evangelical Lutheran Church<ss>Dutch Reformed Church<ss>Protestant Church,0.7,0.0
  7 | 243,4462cafc-a479-4df2-ab37-83a25c0746c0,Asian <ss> Native American <ss> Pacific Islander,African American<ss>Asian<ss>Native American,Native American<ss>African American<ss>Asian<sv>African American<ss>Asian<ss>Native American,0.67,0.0
  8 | 20,c32a9237-0bec-4929-a36b-35be825970ab,White,African American<ss>English<ss>Polish<ss>French<ss>Puerto Rican,African American<sv>African American,0.0,0.0
  9 | 10,968990b2-1b7c-49f0-a691-a994a155769c,to create a freely available developers' kit,create a freely available<ss>developers' kit,developers' kit,0.5,0.0
 10 | 340,d1ef042c-4bf5-4877-b330-dce16939ca18,Melba <ss> Wilder,Notus<ss>Greenleaf<ss>Wilder,Wilder<ss>Star<ss>Melba<ss>Notus<ss>Greenleaf<sv>Notus<ss>Greenleaf<ss>Wilder,0.4,0.0
 11 | 111,10a76bc7-5704-4e74-b84e-c5a40ca2bdda,Somali Youth League <ss> Somali National League,Somali Youth League<ss>Hizbia Digil Mirifle Somali<ss>Somali National League,Hizbia Digil Mirifle Somali<ss>Somali National League<sv>Somali Youth League<ss>Hizbia Digil Mirifle Somali<ss>Somali National League<sv>Hizbia Digil Mirifle Somali<ss>Somali National League,0.67,0.0
 12 | 175,3f6187dd-4a87-4bcb-ad46-94e3b0b809f6,drain on state budgets,helped elect their bosses<ss>overly generous pension systems,overly generous pension systems<sv>they helped elect their bosses,0.0,0.0
 13 | 273,58d1fc14-a962-45aa-a490-bec00bd52089,John McCain <ss> Al Gore,Hillary Clinton<ss>John Kerry<ss>Al Gore,Hillary Clinton<ss>John Kerry<ss>Al Gore,0.5,0.0
 14 | 342,41ed8005-c4f9-4380-9f05-65eac015ba1b,Evansville Triplets <ss> Evansville River Rats,Evansville River Rats<ss>Evas/Pocketeers/Hubs,The Evansville River Rats<ss>Evas/Pocketeers/Hubs<sv>Evansville River Rats<ss>Evas/Pocketeers/Hubs,0.5,0.0
 15 | 153,9d44cd37-98ef-4e76-b266-79d66b9315a0,18-yard <ss> 43 yards,18-yard<ss>19-yard,18-yard<ss>19-yard<sv>18-yard<ss>19-yard,0.5,0.0
 16 | 55,b18bffdc-4d74-4dd2-92c6-b5e45b8b83cb,end of 1754 an armistice <ss> was signed between the Guarani rebels <ss> under Sepé Tiaraju,"leader of the Guarani rebels,<ss>Sepé Tiaraju, was killed",Armistice,0.34,0.0
 17 | 15,d05bbfcc-13c8-472e-819f-b25ad827e990,Greg Olsen,1-yard touchdown pass<ss>tight end Greg Olsen,Will Heller<sv>Kellen Davis<ss>Greg Olsen,0.5,0.0
 18 | 1,a6f9671e-a7d2-467c-b062-448c416aef4f,english <ss> irish <ss> english,united states<ss>germans<ss>english<ss>irish,english<ss>irish<ss>germans<ss>united states<sv>united states<ss>germans<ss>english<ss>irish,0.5,0.0
 19 | 28,cfc6b601-783a-4bf4-8b89-bf201a156456,Native American <ss> Pacific Islander,Asian<ss>Pacific Islander,Asian<ss>Pacific Islander<sv>Asian<ss>Pacific Islander,0.5,0.0
 20 | 2,1ffa94c5-05b2-4e78-9a98-a0ac56fbc5cc,english <ss> irish <ss> english <ss> irish,english<ss>irish,english<ss>irish<sv>english<ss>irish,0.5,0.0
 21 | 270,fa10c341-efbd-46b1-b1e2-a555244e68c3,Barack Obama <ss> Al Gore,Barack Obama<ss>George W. Bush,Barack Obama<ss>George W. Bush<sv>Barack Obama<ss>George W. Bush,0.5,0.0
 22 | 24,706d1e15-617f-44f6-bb45-c6b659c607ea,battleship Espana running aground,Spanish military suffered losses even<ss>at sea; in March,the Spanish military suffering losses even at sea<sv>Spanish military suffering losses<sv>Spanish military suffering losses<sv>suffered losses even at sea,0.0,0.0
 23 | 138,38b18b97-bb2f-4175-a39e-29252d72fb0c,Asian <ss> Japanese <ss> Chinese <ss> the Americas,Asian<ss>Chinese<ss>Japanese,Asian<ss>Chinese<ss>Japanese,0.75,0.0
 24 | 279,287502fc-e97f-41e3-ac1b-d17ab1981e5b,White American,White Americans<ss>Dominicans<ss>Mexicans,51.8% White American<sv>White Americans<ss>Dominicans<ss>Mexicans,0.17,0.0
 25 | 324,31f27624-14d1-44fe-9892-b260cbacd971,Maj. Gen. Robert W. Grow <ss> General Carlos Brewer,Brigadier General Carlos Brewer<ss>Maj. Gen. Robert W. Grow,Brigadier General Carlos Brewer<ss>Maj. Gen. Robert W. Grow<sv>Brigadier General Carlos Brewer<ss>Maj. Gen. Robert W. Grow,0.93,0.0
 26 | 36,086ef7c9-0cc8-43f0-a41e-e71c46c343ec,Ming <ss> King Bayinnaung,The Ming dynasty<ss>Toungoo dynasty,Ming dynasty<ss>Toungoo dynasty<sv>Ming<ss>Toungoo,0.5,0.0
 27 | 61,6fc56ebd-64f9-48ef-a336-174e465fe48e,burglary rate <ss> crime complaints,murder<ss>burglary,murder rate<ss>burglary<sv>murder<ss>burglary,0.33,0.0
 28 | 22,4519b9fa-011e-4f81-abe4-4711f7929b26,siege and all economic restrictions <ss> lifted in the past two years,suffered a great deal<ss>because of lack of trade.,Gdansk merchants suffering<sv>Gdansk merchants suffering<sv>Gdansk merchants<sv>Gdansk merchants suffering<sv>blockade,0.0,0.0
 29 | 173,a1807e6c-741e-4a74-8600-4b1a960ab619,"April 23, 1976 <ss> April 23",April 23<ss>February 3,"April 23, February 3<sv>April 23<ss>February 3",0.5,0.0
 30 | 213,e92d782f-9cb1-4cfe-8074-bebd50e9551f,Peyton Hillis,Jerricho Cotchery<ss>Peyton Hillis,Jerricho Cotchery<ss>Peyton Hillis<sv>Jerricho Cotchery<ss>Peyton Hillis,0.5,0.0
 31 | 90,29a767d2-5c86-46c1-9001-ee6870a5a93c,Khitans <ss> General Xiao Sunning,General Xiao Sunning<ss>Seo Hui,Seo Hui<ss>Xiao Sunning<sv>General Xiao Sunning<ss>Seo Hui of Goryeo,0.5,0.0
 32 | 139,3a70b771-2f1f-4ba9-a2eb-ad3d21c8c560,Asia <ss> Americas,Australia<ss>Africa<ss>Lao,Australia<ss>Africa<sv>Australia<ss>Africa<ss>Lao,0.0,0.0
 33 | 76,acf5ca68-bab4-451e-91dd-2be660deb92a,American Indian <ss> Pacific islander <ss> other races <ss> people <ss> households,American Indian<ss>Pacific islander<ss>other races<ss>Irish<ss>English,American Indian<ss>Pacific islander<ss>other<sv>0.3% American Indian<ss>0.1% Pacific islander<ss>0.9% from other races,0.6,0.0
 34 | 310,a1bf88c4-bbf9-4835-8d45-2b437c925a3b,339 <ss> World Cups <ss> Champions Trophies <ss> Asian Games,World Cup<ss>Asian Games<ss>Champions Trophies,World Cups<ss>Champions Trophies<ss>Asian Games<sv>World Cup<ss>Asian Games<ss>Champions Trophies,0.75,0.0
 35 | 86,23e5f993-8682-40dc-a61e-7d268890daa8,convergence of Protestant hostility to the Habsburgs,convergence of Protestant<ss>hostility to the Habsburgs,Protestant hostility to the Habsburgs<sv>Protestant hostility to the Habsburgs,0.8,0.0
 36 | 19,ee68cc80-e09a-4177-a3dc-5da3f116cc02,Italian <ss> Italian,English<ss>German,German<ss>English<ss>Polish<ss>French,0.0,0.0
 37 | 249,5cc93449-2d02-4e19-842e-8fa2baceb3a3,Irish <ss> French,Scottish<ss>Canadian<ss>Irish,Scottish people<ss>Canadian<ss>Irish people<sv>Scottish<ss>Canadian<ss>Irish,0.33,0.0
 38 | 27,3322e208-39e5-4980-a219-70da8663629a,Native American <ss> Pacific Islander,Pacific Islander<ss>Asian,Pacific Islander<ss>Asian<sv>Asian<ss>Pacific Islander,0.5,0.0
 39 | 187,f1391a97-ab51-48f0-a8b6-3096d67df321,5 and 9 years <ss> 60 and 64 years,between 20 and 24 years<ss>between 25 and 29 years,20 and 24<ss>25 and 29<sv>between 20 and 24 years<ss>between 25 and 29 years,0.0,0.0
 40 | 157,04f796e2-1860-4455-905e-7479f0465000,Longwell <ss> John Kasay,Kasay<ss>Karney,Karney<ss>Kasay<sv>John Kasay<ss>Karney,0.5,0.0
 41 | 199,7f483afc-67be-4923-a3c0-711dc4e217a6,Mike Karney <ss> Mark Campbell,Deuce McAllister<ss>Mark Campbell<ss>Mike Karney,Mark Campbell<ss>Deuce McAllister<ss>Mike Karney<sv>Deuce McAllister<ss>Mike Karney<ss>Mark Campbell,0.67,0.0
 42 | 162,333025d2-3ce9-405d-b47e-f5f4c660dbf4,Chicago Bears <ss> Minnesota Vikings <ss> Pittsburgh Steelers <ss> Green Bay Packers,Pittsburgh Steelers<ss>New England Patriots,New England Patriots<sv>3,0.25,0.0
 43 | 87,0f2c1862-c1eb-44c4-852e-357cc0ece6f6,François de Bonne <ss> Charles de Blanchefort <ss> Marquis de Créquy,"François de Bonne,<ss>Duke of Lesdiguières<ss>Charles de Blanchefort<ss>Marquis de Créquy,",François de Bonne<ss>Duke of Lesdiguières<ss>Charles de Blanchefort<ss>Marquis de Créquy<sv>François de Bonne<ss>Charles de Blanchefort,0.75,0.0
 44 | 226,152795cb-158a-42b3-aa02-78568a398749,African American <ss> Native American <ss> Asian,Native American<ss>Asian,Native American<ss>Asian<sv>Native American<ss>Asian,0.67,0.0
 45 | 56,2728b800-74a3-49cd-9ffb-eb358cc801b0,Duke Magnus Birgersson started a war,Duke Magnus Birgersson<ss>started a war,started a war<sv>he started a war,0.57,0.0
 46 | 271,113961e7-8809-4612-97fc-f2d5f39697fd,Barack Obama,Barack Obama<ss>George W. Bush,Barack Obama<ss>George W. Bush<sv>Barack Obama<ss>George W. Bush,0.5,0.0
 47 | 145,dff115fd-b081-470d-a217-628ed9ede17e,economically active,economically active<ss>students,less economically active unemployed<sv>unemployed,0.67,0.0
 48 | 307,51c1b08e-63fe-46bc-b232-29f8ca1e9372,Bulgarian citizens <ss> Pakistani citizens,Romanian<ss>Pakistani,Pakistani citizens<ss>Romanian citizenship<sv>Romanian<ss>Pakistani,0.5,0.0
 49 | 296,de295075-849a-4746-870c-e318f9fc9aaf,Brett Favre <ss> Robert Brooks,Antonio Freeman<ss>Robert Brooks,Favre<ss>Antonio Freeman<ss>Robert Brooks<ss>Dorsey Levens<sv>Antonio Freeman<ss>Robert Brooks,0.5,0.0
 50 | 40,9ce0c031-ead7-46e7-b5c9-4cbb5a090d19,Irish <ss> Americans,Irish American<ss>Americans ancestry<ss>households<ss>families,Irish American<sv>6.7% American,0.33,0.0
 51 | 163,1e221f63-9654-4ba4-98ed-355cca7c401b,Chicago Bears <ss> Minnesota Vikings <ss> Pittsburgh Steelers <ss> Green Bay Packers <ss> New England Patriots,49ers<ss>Bears<ss>New England Patriots,Bears<ss>Patriots<ss>49ers<sv>3,0.33,0.0
 52 | 189,08bc6401-934a-4703-ae77-37fca8fa8339,heart disease <ss> stroke <ss> cancer,type 2 diabetes<ss>arteriosclerosis<ss>hypertension,diabetes<ss>arteriosclerosis<ss>hypertension<sv>type 2 diabetes<ss>arteriosclerosis<ss>hypertension,0.0,0.0
 53 | 222,6f579268-78fa-4887-84ea-2ee4ba68f8cf,Rice,Craig<ss>Rice<ss>Tom Rathman,Craig<sv>Craig,0.33,0.0
 54 | 346,cf56b518-e56b-4a82-b477-ef397ccda620,Holly Hallstrom <ss> Rachel Reynolds,Pennington<ss>Bradley,Janice Pennington<ss>Kathleen Bradley<sv>Pennington<ss>Bradley,0.0,0.0
 55 | 159,712ba53b-d57b-4149-827f-83f90ba1dfe0,Wieli ⁇  <ss> Pu ⁇ sk <ss> Kupise <ss> Biel <ss> Markowa,Wieliż<ss>Puńsk<ss>Kupise<ss>Biel<ss>Markowa,Wieliż<ss>Puńsk<ss>Kupise<sv>Wieliż<ss>Puńsk<ss>Kupise<ss>Biel<ss>Markowa,0.6,0.0
 56 | 142,267b402c-4a04-4ccc-a725-23e827562724,Waldemar II <ss> the Joint Masters of the Order,joint Masters of the Order<ss>the Swedes,Danish<ss>Swedes<ss>joint Masters of the Order<sv>Danish king Waldemar II<ss>joint Masters of the Order<sv>Danish<ss>Swedes,0.83,0.0
 57 | 218,3b1d2481-ea0f-410d-a818-8c234af697b2,Punjabi <ss> French <ss> Vietnamese,Cantonese<ss>French<ss>Vietnamese,Cantonese<ss>French<ss>Vietnamese<sv>Cantonese<ss>French<ss>Vietnamese,0.67,0.0
 58 | 3,8630522f-4059-4127-bfc1-6ede5e94762b,Bears,Atlanta would regain the lead<ss>to lock up the victory,Atlanta<sv>Falcons,0.0,0.0
 59 | 53,02d84db7-34ac-4c30-87e0-8769ec27083d,Battle of Mohács,in 1491<ss>signed the Peace of Pressburg,Peace of Pressburg<sv>Peace of Pressburg,0.33,0.0
 60 | 320,6d4578ce-4d7c-435d-9328-05e10b91b3be,Vietnamese <ss> French,German<ss>Vietnamese,German<ss>Vietnamese<sv>German<ss>Vietnamese,0.5,0.0
 61 | 81,86969fad-0eda-4854-83e4-e08014809588,Bryant Johnson <ss> T. J. Duckett,Frank Gore<ss>T. J. Duckett,Frank Gore<ss>T. J. Duckett,0.5,0.0
 62 | 158,09af6e09-38f2-47f7-a3d5-055a779204a2,King Amangkurat I <ss> brother Pangeran Puger,Amangkurat II<ss>Pangeran Puger,Amangkurat II<ss>Pangeran Puger<sv>Amangkurat II<ss>Pangeran Puger<sv>Amangkurat II<ss>Pangeran Puger<sv>Amangkurat II<sv>Amangkurat II<ss>Pangeran Puger<sv>Amangkurat II<ss>Pangeran Puger,0.6,0.0
 63 | 45,fbddf60a-31e4-4850-be04-44c4871079a3,second <ss> third,second<ss>fourth,second quarter<ss>fourth quarter<sv>the second<ss>the fourth<sv>2<sv>second<ss>fourth<sv>2,0.5,0.0
 64 | 239,98f8ed5f-7505-4f7f-9fcf-5d627d6a0e1b,Hollister <ss> San Juan Bautista <ss> American Independent,American Independent<ss>Green Party,American Independent<ss>Green Party<sv>American Independent<ss>Green Party,0.33,0.0
 65 | 32,26c48624-0dec-4bf1-b4c9-e29d842415a9,Italian <ss> united states <ss> Italian <ss> German <ss> Italian,French Canadian<ss>German<ss>Italian<ss>united states<ss>Italian<ss>English,French Canadian<ss>German<ss>Italian<ss>united states<sv>French Canadian<ss>German<ss>Italian<ss>united states<ss>Italian<ss>English,0.67,0.0
 66 | 167,2c4c5003-e35a-45fa-bd57-2bec7a019c6c,4-yard <ss> 14-yard <ss> Derek Anderson <ss> Cleo Lemon,4-yard<ss>5-yard,5-yard<sv>4-yard<ss>5-yard,0.25,0.0
 67 | 75,865f86c6-5caf-4762-a3fd-912a2cf70852,American Indian <ss> Pacific islander <ss> other races <ss> two or more races,American Indian<ss>Pacific islander<ss>other races,Asian<ss>American Indian<ss>Pacific islander,0.75,0.0
 68 | 43,1b49e087-f1f0-46aa-b946-745ace013f96,Urdu <ss> Telugu <ss> Tamil,Urdu<ss>Kannada,10.83<sv>Urdu<ss>Kannada,0.33,0.0
 69 | 225,6ec2f511-4f46-47b3-8c99-301ba90ccf7d,White <ss> Asian,Native American<ss>Asian,Asian<ss>Native American<sv>Native American<ss>Asian,0.5,0.0
 70 | 44,f9e3a54c-cea9-47d6-9c95-852a443017d4,first quarter,first quarter<ss>second<ss>fourth,first<sv>first,0.67,0.0
 71 | 166,6bce16f6-eb56-4b08-99b5-538d28930847,Jason Wright <ss> Derek Anderson,Jason Wright<ss>Derek Anderson<ss>Cleo Lemon,Jason Wright<ss>Derek Anderson<ss>Cleo Lemon<sv>Jason Wright<ss>Derek Anderson<ss>Cleo Lemon,0.67,0.0
 72 | 169,2c1435a5-47b5-46e9-9269-fce36f88e79f,second <ss> third,first<ss>second<ss>third<ss>fourth,first<ss>second<ss>third<ss>fourth<sv>first<ss>second<ss>third<ss>fourth,0.5,0.0
 73 | 212,431d7974-400f-45f4-b5bf-4581cb938ae4,Jerricho Cotchery <ss> Mohamed Massaquoi,Jerricho Cotchery<ss>Mohamed Massaquoi<ss>Santonio Holmes,Colt McCoy<ss>Santonio Holmes<ss>Jerricho Cotchery<sv>Jerricho Cotchery<ss>Mohamed Massaquoi<ss>Santonio Holmes,0.67,0.0
 74 | 63,9897b605-c973-4c1e-8a77-769d0ed75e68,Aigina <ss> Romania <ss> Nafplion <ss> Laconia <ss> Navarino <ss> Achaea,Romania<ss>Laconia<ss>Messenia<ss>Achaea,Romania<ss>Laconia<ss>Messenia<ss>Achaea<sv>Romania<ss>Laconia<ss>Messenia<ss>Achaea,0.5,0.0
 75 | 245,664a70b4-cb10-481d-b16d-c1e322d3385f,taxicab <ss> motorcycle <ss> bicycle,driving alone<ss>carpooled<ss>public transportation<ss>walked<ss>all other forms of transportation,driving alone<ss>carpooled<ss>public transportation<ss>walked<ss>other forms of transportation<sv>driving alone<ss>carpooled<ss>public transportation<ss>walked<ss>all other forms of transportation,0.0,0.0
 76 | 46,305e4fc0-2a6c-4104-9337-d78c43924db8,end the war <ss> have a social revolution <ss> a social revolution,Peace<ss>Land<ss>Bread,to end the war<ss>have a social revolution,0.6,0.0
 77 | 203,4da64f03-53fb-4656-9522-2e38be1df8f1,Hindu <ss> Muslim <ss> Kannada,Sikh<ss>Buddhist,Sikh<ss>Buddhist<ss>other religions<sv>Sikh<ss>Buddhist<ss>other religions,0.0,0.0
 78 | 146,0d2f8c43-f67d-48fd-a63d-c451d29484f1,Homemaking <ss> unable to work,"economically active,<ss>Homemaking",economically active<ss>Homemaking<sv>economically active<ss>Homemaking,0.5,0.0
 79 | 49,817609c8-61c2-4bc0-a13a-886537dcf03a,Jaél de Peigne,"November 1566 Killigrew married<ss>Catherine,<ss>November 1590 he was married<ss>Jaél de Peigne,",Catherine<sv>Killigrew getting married to Catherine,0.25,0.0
 80 | 65,0bb6ba0f-807f-44da-aa0d-bb21e7b48fe0,king Chulalongkorn enacted two decrees banning <ss> the capture and sale of Kha slaves,king Chulalongkorn enacted two decrees<ss>banning the capture and sale<ss>of Kha slaves,king Chulalongkorn enacted two decrees banning the capture and sale of Kha slaves<sv>two decrees<sv>king Chulalongkorn enacted two decrees,0.53,0.0
 81 | 220,2927f0d9-dd04-4f19-9856-27789269cf40,Rice <ss> John Taylor,Craig<ss>Montana,Montana<ss>Craig<sv>Montana<ss>Craig,0.0,0.0
 82 | 280,a308efbb-8383-49c1-ae58-d323c783c649,Cambodian <ss> Filipino <ss> Pakistani,Cambodian<ss>Filipino<ss>Korean<ss>Pakistani<ss>Laotian,0.1% Cambodian<ss>0.1% Filipino<ss>0.1% Korean<ss>0.1% Pakistani<ss>0.1% Laotian<sv>Cambodian<ss>Filipino<ss>Korean<ss>Pakistani<ss>Laotian,0.6,0.0
 83 | 332,da82ac4b-8c49-4332-a6aa-5c7fca99f61e,Eagles <ss> New England Patriots,Philadelphia Eagles<ss>New England Patriots,the Philadelphia Eagles<ss>the New England Patriots<sv>Philadelphia Eagles<ss>New England Patriots,0.83,0.0
 84 | 178,54a3ca15-6a70-49c4-a6cb-2c41113e6473,"Agnes' cousin, Karl von Mansfeld",Tercios<ss>Italian<ss>Spanish<ss>German,"Mansfeld's 2000<ss>foot and Tercios<ss>Italian, Spanish and German soldiers<sv>Tercios<ss>Italian<ss>Spanish<ss>German",0.0,0.0
 85 | 130,6b2fe198-c401-42aa-8c47-5175d0f04e8e,France <ss> Italy,Ottoman Empire<ss>France,Ottoman Empire<ss>France<sv>france ottoman,0.5,0.0
 86 | 334,b6d652ec-8b25-446e-b1dd-e7da20a22a74,agriculture <ss> services,industry<ss>services,industry<ss>services<sv>industry<ss>services,0.5,0.0
 87 | 78,0edbf486-7037-490f-8e37-a0ec47589ad1,Wokingham <ss> Wokingham <ss> Wokingham <ss> Surrey <ss> Tandridge <ss> Windsor & Maidenhead,Portsmouth<ss>Thanet,Thanet<ss>Portsmouth<sv>Portsmouth<ss>Thanet,0.0,0.0
 88 | 85,3a628138-e841-47e9-89cc-2781f9a3b3d0,destroyed the remaining reinforcements,defense preparations had assured<ss>no boats were available,launched a successful surprise attack<sv>surprise attack<sv>surprise attack<sv>surprise attack,0.0,0.0
 89 | 155,eb29b7d7-abe6-437d-b4e8-c708df616c3e,Yancey Thigpen <ss> Ernie Mills,Yancey Thigpen<ss>Erric Pegram,Yancey Thigpen<ss>Erric Pegram<sv>Yancey Thigpen<ss>Erric Pegram,0.5,0.0
 90 | 150,98160156-f6ac-4953-98b2-3515a9467666,demoralized troops,constant Ethiopian air attacks<ss>on their supply lines,constant Ethiopian air attacks<sv>constant Ethiopian air attacks,0.0,0.0
 91 | 52,e78bd046-1df8-4191-8645-1baf57255d48,Louis II,Matthias Corvinus<ss>his son King Maximilian I,Frederick<sv>Matthias Corvinus,0.0,0.0
 92 | 117,ed1e3110-ac93-402a-82b4-fdbd574731bb,pro-Free State Blueshirts,IRA<ss>pro-Free State Blueshirts,IRA<sv>the IRA,0.5,0.0
 93 | 232,87c0bfbb-f591-4674-8b4d-7bbbc6d592ca,Germans <ss> English <ss> Irish <ss> United States <ss> Italian <ss> English,Hispanics<ss>Germans<ss>English<ss>Irish<ss>United States<ss>Italian,Hispanics<ss>Germans<ss>English<ss>Irish<ss>Italian<ss>United States,0.83,0.0
 94 | 97,95717e8a-3e01-44ac-9ad8-ebdc8746f3c0,2004,same-sex unions<ss>ordination of women,ordination of women<ss>blessing of same-sex unions<sv>ordination of women<ss>blessing of same-sex unions,0.0,0.0
 95 | 110,44296346-ad69-4303-8cb3-f7f9fc277b46,Diego Garc ⁇ a de Herrera,until 1477 when they ceded<ss>to their son Hernán Peraza,Diego García de Herrera<sv>Hernán Peraza The Younger,0.67,0.0
 96 | 207,894b5cac-0c9c-421e-bda9-f0b1e49541d3,Hindu <ss> Hindu,Hindu<ss>Muslim,"hennais population was 80.7% Hindu, 9.5% Muslim,<sv>Hindu<ss>Muslim",0.5,0.0
 97 | 319,bbf5bfef-2cb9-4c7d-b7a9-dd73e5807753,Arabic <ss> Japanese,African<ss>Japanese,Japanese<ss>African languages<sv>African<ss>Japanese,0.5,0.0
 98 | 171,69f93831-4081-44ea-b212-31e475092207,France <ss> United Kingdom <ss> Spain <ss> Italy <ss> Netherlands,Spain<ss>Italy<ss>the Netherlands,Germany<ss>France<ss>United Kingdom<sv>Spain<ss>Italy<ss>Netherlands,0.6,0.0
 99 | 183,9b00a7f1-c5f1-4c60-926e-d7df131b26e3,60 and 64 years <ss> 60 and 64 years,20 and 24 years<ss>25 and 29 years,20 and 24<ss>25 and 29,0.0,0.0
100 | 246,063a4c5c-9dd4-4b31-b3ea-bf91b418e84f,taxicab <ss> motorcycle <ss> bicycle,driving alone<ss>carpooled<ss>walked,driving alone<ss>carpooled<ss>walked<sv>driving alone<ss>carpooled<ss>walked,0.0,0.0
101 | 221,909dba4e-87da-4dcb-97b4-31356b108fee,Rice,Montana<ss>Craig,Montana<ss>Rice<ss>Craig<sv>Montana<ss>Rice<ss>Craig,0.33,0.0
102 | 


--------------------------------------------------------------------------------