├── CODE_OF_CONDUCT.md ├── LICENSE ├── README.md ├── SECURITY.md ├── data └── Mind │ ├── test.tsv │ ├── test_key_text.json │ ├── test_nn.json │ ├── test_query_text.json │ ├── train.tsv │ └── valid.tsv ├── dataloader.py ├── models ├── Differentiable_PQ.py ├── MoPQ.py └── TextEncoder.py ├── parameters.py ├── preprocess.py ├── run.py ├── test.py └── utils ├── recall.py ├── setup_utils.py └── utils.py /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Microsoft Open Source Code of Conduct 2 | 3 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 4 | 5 | Resources: 6 | 7 | - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) 8 | - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) 9 | - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns 10 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Matching-oriented Product Quantization For Ad-hoc Retrieval 2 | Repo for EMNLP 2021 paper: Matching-oriented Product Quantization For Ad-hoc Retrieval. 3 | 4 | ## Introduction 5 | In this work, we identify the limitation of using reconstruction loss minimization in supervised PQ methods, 6 | and propose MCL as the new training objective, where the model can be learned to maximize the query-key matching 7 | probability to achieve the optimal retrieval accuracy. We further leverage DCS for contrastive sample argumentation, which ensures the effective minimization of MCL. 8 | 9 | ## Dataset Format 10 | 11 | File Name | Description | Format 12 | ------------- | ------------- | -------------- 13 | train.tsv/valid.tsv/test.tsv | pairs of query and key | two columns and splited by "\t" 14 | test_keys.josn | keys' text and its id | {k_id : "key_text"} 15 | test_queries.json | queries' text and its id | {q_id : "query_text"} 16 | test_ann.json | queries' positive neighbors | {q_id : [k_id, ...]} 17 | 18 | 19 | ## Preprocess 20 | - **Jointly optimize the codebook and embeddings** 21 | Here are the command to for tokenization: 22 | ``` 23 | python preprocess.py --dataset Mind --bert_model bert-base-uncased 24 | ``` 25 | This command will create two files: `./data/MIND/preprocessed_train.tsv` and `./data/MIND/preprocessed_valid.tsv`, where each line is the tokenization results of query and key: 26 | ``` 27 | {"query_tokens":List[int], "key_tokens":List[int]} 28 | {"query_tokens":List[int], "key_tokens":List[int]} 29 | ... 30 | ``` 31 | 32 | - **Optimize the codebooks based on the fixed embeddings** 33 | If you want to trian the MoPQ based on the existing embeddings, you should generate the preprocessed files in the following format: 34 | ``` 35 | {"query_vec":List[float], "key_vec":List[float]} 36 | {"query_vec":List[float], "key_vec":List[float]} 37 | ... 38 | ``` 39 | You also can storage the "query_vec" and "key_tokens" in the preprocessed files to only fix queries' embeddings. 40 | Besides, you need to provide the embeddings of queries/keys for testdata, i.e., `test_queries.json`/`test_keys.json` 41 | ``` 42 | {"id":List[float], ...} 43 | ``` 44 | You can download the Mind_with_emb dataset from [here](https://microsoft-my.sharepoint.com/:f:/p/t-shxiao/Eq6352Kvvu1BtMAuSMKMIvwBBFkrWEdqFa6HA1RKN6zp8w?e=m6tP24). 45 | 46 | ## Train 47 | Use the following command to train MoPQ. And it will automatically select the best model to test. 48 | ``` 49 | python run.py \ 50 | --mode train \ 51 | --dataset {dataset: Mind or Mind_with_emb} \ 52 | --bert_model bert-base-uncased \ 53 | --model_type MoPQ \ 54 | --savename MoPQ_Mind \ 55 | --cross_device True \ 56 | --world_size {the number of your GPUs} 57 | ``` 58 | If use more than one GPU, `--cross_device` should be True to activate the Differentiable Cross-device in-batch Sampling. 59 | You can change the `model_type` to `TextEncoder` to train the encoder without PQ module: 60 | ``` 61 | python run.py \ 62 | --mode train \ 63 | --dataset Mind \ 64 | --bert_model bert-base-uncased \ 65 | --model_type TextEncoder \ 66 | --savename MoPQ_Mind \ 67 | --cross_device True \ 68 | --world_size {the number of your GPUs} 69 | ``` 70 | 71 | ## Test 72 | You can also start the test process manually using following command: 73 | ``` 74 | python run.py \ 75 | --mode test \ 76 | --dataset {dataset: Mind or Mind_with_emb} \ 77 | --model_type MoPQ \ 78 | --load_ckpt_name ./model/MoPQ_Mind-best.pt 79 | ``` 80 | 81 | 82 | ## Contributing 83 | 84 | This project welcomes contributions and suggestions. Most contributions require you to agree to a 85 | Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us 86 | the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. 87 | 88 | When you submit a pull request, a CLA bot will automatically determine whether you need to provide 89 | a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions 90 | provided by the bot. You will only need to do this once across all repos using our CLA. 91 | 92 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 93 | For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or 94 | contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. 95 | 96 | ## Trademarks 97 | 98 | This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft 99 | trademarks or logos is subject to and must follow 100 | [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). 101 | Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. 102 | Any use of third-party trademarks or logos are subject to those third-party's policies. 103 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). 6 | 7 | If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below. 8 | 9 | ## Reporting Security Issues 10 | 11 | **Please do not report security vulnerabilities through public GitHub issues.** 12 | 13 | Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report). 14 | 15 | If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc). 16 | 17 | You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). 18 | 19 | Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: 20 | 21 | * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) 22 | * Full paths of source file(s) related to the manifestation of the issue 23 | * The location of the affected source code (tag/branch/commit or direct URL) 24 | * Any special configuration required to reproduce the issue 25 | * Step-by-step instructions to reproduce the issue 26 | * Proof-of-concept or exploit code (if possible) 27 | * Impact of the issue, including how an attacker might exploit the issue 28 | 29 | This information will help us triage your report more quickly. 30 | 31 | If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://microsoft.com/msrc/bounty) page for more details about our active programs. 32 | 33 | ## Preferred Languages 34 | 35 | We prefer all communications to be in English. 36 | 37 | ## Policy 38 | 39 | Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd). 40 | 41 | -------------------------------------------------------------------------------- /data/Mind/test_nn.json: -------------------------------------------------------------------------------- 1 | {"0": [79122], "1": [79123], "2": [79124], "3": [79125], "4": [79126], "5": [79127], "6": [79128], "7": [79129], "8": [79130], "9": [79131], "10": [79132], "11": [79133], "12": [79134], "13": [79135], "14": [79136], "15": [79137], "16": [79138], "17": [79139], "18": [79140], "19": [79141], "20": [79142], "21": [79143], "22": [79144], "23": [79145], "24": [79146], "25": [79147], "26": [79148], "27": [79149], "28": [79150], "29": [79151], "30": [79152], "31": [79153], "32": [79154], "33": [79155], "34": [79156], "35": [79157], "36": [79158], "37": [79159], "38": [79160], "39": [79161], "40": [79162], "41": [79163], "42": [79164], "43": [79165], "44": [79166], "45": [79167], "46": [79168], "47": [79169], "48": [79170], "49": [79171], "50": [79172], "51": [79173], "52": [79174], "53": [79175], "54": [79176], "55": [79177], "56": [79178], "57": [79179], "58": [79180], "59": [79181], "60": [79182], "61": [79183], "62": [79184], "63": [79185], "64": [79186], "65": [79187], "66": [79188], "67": [79189], "68": [79190], "69": [79191], "70": [79192], "71": [79193], "72": [37856], "73": [79194], "74": [79195], "75": [79196], "76": [50197], "77": [79197], "78": [79198], "79": [79199], "80": [79200], "81": [79201], "82": [79202], "83": [79203], "84": [79204], "85": [79205], "86": [79206], "87": [79207], "88": [79208], "89": [79209], "90": [79210], "91": [79211], "92": [79212], "93": [79213], "94": [79214], "95": [79215], "96": [79216], "97": [79217], "98": [79218], "99": [79219], "100": [79220], "101": [79221], "102": [79222], "103": [79223], "104": [79224], "105": [79225], "106": [79226], "107": [79227], "108": [79228], "109": [79229], "110": [79230], "111": [79231], "112": [79232], "113": [79233], "114": [79234], "115": [79235], "116": [79236], "117": [79237], "118": [79238], "119": [79239], "120": [79240], "121": [79241], "122": [79242], "123": [79243], "124": [79244], "125": [79245], "126": [51208], "127": [79246], "128": [79247], "129": [79248], "130": [79249], "131": [79250], "132": [79251], "133": [79252], "134": [79253], "135": [79254], "136": [79255], "137": [79256], "138": [79257], "139": [79258], "140": [79259], "141": [79260], "142": [79261], "143": [79262], "144": [79263], "145": [79264], "146": [79265], "147": [79266], "148": [79267], "149": [79268], "150": [79269], "151": [79270], "152": [79271], "153": [79272], "154": [79273], "155": [79274], "156": [79275], "157": [79276], "158": [79277], "159": [79278], "160": [79279], "161": [79280], "162": [79281], "163": [79282], "164": [79283], "165": [79284], "166": [79285], "167": [79286], "168": [79287], "169": [79288], "170": [79289], "171": [79290], "172": [79291], "173": [79292], "174": [79293], "175": [79294], "176": [79295], "177": [79296], "178": [79297], "179": [79298], "180": [79299], "181": [79300], "182": [79301], "183": [79302], "184": [79303], "185": [79304], "186": [79305], "187": [79306], "188": [79307], "189": [79308], "190": [79309], "191": [79310], "192": [79311], "193": [79312], "194": [79313], "195": [79314], "196": [79315], "197": [79316], "198": [79317], "199": [79318], "200": [79319], "201": [79320], "202": [79321], "203": [79322], "204": [79323], "205": [79324], "206": [79325], "207": [79326], "208": [79327], "209": [79328], "210": [79329], "211": [79330], "212": [2147], "213": [79331], "214": [79332], "215": [79333], "216": [79334], "217": [79335], "218": [79336], "219": [79337], "220": [79338], "221": [79339], "222": [79340], "223": [79341], "224": [79342], "225": [79343], "226": [79344], "227": [79345], "228": [79346], "229": [79347], "230": [79348], "231": [79349], "232": [79350], "233": [79351], "234": [79352], "235": [79353], "236": [79354], "237": [79355], "238": [79356], "239": [79357], "240": [79358], "241": [79359], "242": [79360], "243": [79361], "244": [79362], "245": [79363], "246": [79364], "247": [79365], "248": [79366], "249": [79367], "250": [79368], "251": [79369], "252": [79370], "253": [79371], "254": [79372], "255": [79373], "256": [79374], "257": [79375], "258": [79376], "259": [79377], "260": [79378], "261": [79379], "262": [79380], "263": [79381], "264": [79382], "265": [79383], "266": [79384], "267": [3461], "268": [79385], "269": [79386], "270": [79387], "271": [79388], "272": [79389], "273": [79390], "274": [79391], "275": [79392], "276": [79393], "277": [79394], "278": [79395], "279": [79396], "280": [79397], "281": [79398], "282": [79399], "283": [79400], "284": [79401], "285": [79402], "286": [79403], "287": [79404], "288": [79405], "289": [79406], "290": [1050], "291": [79407], "292": [79408], "293": [79409], "294": [79410], "295": [79411], "296": [79412], "297": [79413], "298": [79414], "299": [79415], "300": [79416], "301": [79417], "302": [79418], "303": [79419], "304": [79420], "305": [79421], "306": [79422], "307": [79423], "308": [79424], "309": [79425], "310": [79426], "311": [79427], "312": [79428], "313": [79429], "314": [79430], "315": [79431], "316": [79432], "317": [79433], "318": [79434], "319": [79435], "320": [79436], "321": [79437], "322": [79438], "323": [79439], "324": [79440], "325": [79441], "326": [79442], "327": [79443], "328": [79444], "329": [79445], "330": [79446], "331": [79447], "332": [79448], "333": [79449], "334": [79450], "335": [79451], "336": [79452], "337": [79453], "338": [79454], "339": [79455], "340": [79456], "341": [79457], "342": [79458], "343": [79459], "344": [79460], "345": [79461], "346": [79462], "347": [79463], "348": [79464], "349": [79465], "350": [79466], "351": [79467], "352": [87003], "353": [79469], "354": [79470], "355": [79471], "356": [79472], "357": [79473], "358": [79474], "359": [79475], "360": [79476], "361": [79477], "362": [79478], "363": [11347], "364": [79479], "365": [79480], "366": [79481], "367": [79482], "368": [79483], "369": [79484], "370": [79485], "371": [79486], "372": [79487], "373": [79488], "374": [79489], "375": [79490], "376": [79491], "377": [79492], "378": [79493], "379": [79494], "380": [79495], "381": [79496], "382": [79497], "383": [79498], "384": [79499], "385": [79500], "386": [79501], "387": [79502], "388": [79503], "389": [79504], "390": [79505], "391": [79506], "392": [79507], "393": [79508], "394": [79509], "395": [79510], "396": [48348], "397": [79511], "398": [79512], "399": [79513], "400": [79514], "401": [79515], "402": [79516], "403": [79517], "404": [8916], "405": [79518], "406": [79519], "407": [79520], "408": [79521], "409": [79522], "410": [79523], "411": [79524], "412": [79525], "413": [79526], "414": [79527], "415": [79528], "416": [79529], "417": [79530], "418": [79531], "419": [79532], "420": [79533], "421": [79534], "422": [79535], "423": [79536], "424": [36486], "425": [6806], "426": [79537], "427": [79538], "428": [79539], "429": [79540], "430": [79541], "431": [79542], "432": [79543], "433": [79544], "434": [79545], "435": [79546], "436": [79547], "437": [79548], "438": [79549], "439": [79550], "440": [79551], "441": [79552], "442": [79553], "443": [79554], "444": [79555], "445": [79556], "446": [79557], "447": [79558], "448": [79559], "449": [79560], "450": [79561], "451": [79562], "452": [79563], "453": [79564], "454": [79565], "455": [79566], "456": [79567], "457": [79568], "458": [79569], "459": [79570], "460": [79571], "461": [79572], "462": [79573], "463": [79574], "464": [79575], "465": [79576], "466": [79577], "467": [79578], "468": [79579], "469": [79580], "470": [79581], "471": [79582], "472": [79583], "473": [79584], "474": [79585], "475": [79586], "476": [79587], "477": [79588], "478": [81805], "479": [79590], "480": [79591], "481": [79592], "482": [79593], "483": [79594], "484": [79595], "485": [79596], "486": [79597], "487": [79598], "488": [79599], "489": [79600], "490": [79601], "491": [79602], "492": [79603], "493": [79604], "494": [79605], "495": [79606], "496": [79607], "497": [79608], "498": [79609], "499": [79610], "500": [79611], "501": [79612], "502": [79613], "503": [79614], "504": [79615], "505": [79616], "506": [79617], "507": [79618], "508": [79619], "509": [79620], "510": [79621], "511": [79622], "512": [79623], "513": [79624], "514": [79625], "515": [14726], "516": [79626], "517": [79627], "518": [79628], "519": [79629], "520": [79630], "521": [79631], "522": [79632], "523": [79633], "524": [79634], "525": [79635], "526": [79636], "527": [79637], "528": [23952], "529": [79638], "530": [79639], "531": [79640], "532": [79641], "533": [79642], "534": [79643], "535": [79644], "536": [79645], "537": [79646], "538": [79647], "539": [79648], "540": [79649], "541": [79650], "542": [79651], "543": [79652], "544": [79653], "545": [79654], "546": [79655], "547": [79656], "548": [79657], "549": [79658], "550": [79659], "551": [79660], "552": [79661], "553": [79662], "554": [79663], "555": [79664], "556": [79665], "557": [79666], "558": [79667], "559": [79668], "560": [79669], "561": [79670], "562": [79671], "563": [79672], "564": [79673], "565": [79674], "566": [79675], "567": [79676], "568": [79677], "569": [79678], "570": [79679], "571": [79680], "572": [79681], "573": [79682], "574": [79683], "575": [79684], "576": [79685], "577": [79686], "578": [79687], "579": [79688], "580": [79689], "581": [79690], "582": [79691], "583": [79692], "584": [79693], "585": [79694], "586": [79695], "587": [79696], "588": [79697], "589": [79698], "590": [79699], "591": [79700], "592": [79701], "593": [79702], "594": [79703], "595": [79704], "596": [79705], "597": [79706], "598": [79707], "599": [79708], "600": [79709], "601": [79710], "602": [79711], "603": [79712], "604": [79713], "605": [79714], "606": [79715], "607": [79716], "608": [79717], "609": [79718], "610": [79719], "611": [79720], "612": [79721], "613": [79722], "614": [79723], "615": [79724], "616": [79725], "617": [79726], "618": [79727], "619": [79728], "620": [79729], "621": [79730], "622": [79731], "623": [79732], "624": [12679], "625": [79733], "626": [79734], "627": [79735], "628": [79736], "629": [79737], "630": [79738], "631": [79739], "632": [79740], "633": [79741], "634": [79742], "635": [79743], "636": [79744], "637": [79745], "638": [79746], "639": [79747], "640": [79748], "641": [79749], "642": [79750], "643": [79751], "644": [79752], "645": [79753], "646": [79754], "647": [79755], "648": [79756], "649": [79757], "650": [79758], "651": [79759], "652": [79760], "653": [79761], "654": [79762], "655": [79763], "656": [79764], "657": [79765], "658": [79766], "659": [79767], "660": [79768], "661": [79769], "662": [79770], "663": [79771], "664": [79772], "665": [79773], "666": [79774], "667": [79775], "668": [79776], "669": [79777], "670": [48153], "671": [79778], "672": [79779], "673": [79780], "674": [79781], "675": [79782], "676": [79783], "677": [79784], "678": [79785], "679": [79786], "680": [15695], "681": [79787], "682": [79788], "683": [79789], "684": [79790], "685": [79791], "686": [79792], "687": [79793], "688": [79794], "689": [79795], "690": [79796], "691": [79797], "692": [79798], "693": [79799], "694": [52351], "695": [79800], "696": [79801], "697": [79802], "698": [79803], "699": [79804], "700": [79805], "701": [79806], "702": [79807], "703": [79808], "704": [79809], "705": [79810], "706": [79811], "707": [79812], "708": [79813], "709": [79814], "710": [79815], "711": [79816], "712": [79817], "713": [79818], "714": [79819], "715": [79820], "716": [79821], "717": [79822], "718": [79823], "719": [79824], "720": [79825], "721": [79826], "722": [79827], "723": [9179], "724": [79828], "725": [79829], "726": [79830], "727": [79831], "728": [79832], "729": [79833], "730": [79834], "731": [79835], "732": [79836], "733": [79837], "734": [79838], "735": [79839], "736": [79840], "737": [79841], "738": [79842], "739": [79843], "740": [79844], "741": [79845], "742": [79846], "743": [79847], "744": [79848], "745": [79849], "746": [79850], "747": [79851], "748": [79852], "749": [79853], "750": [79854], "751": [79855], "752": [79856], "753": [79857], "754": [79858], "755": [79859], "756": [79860], "757": [79861], "758": [79862], "759": [79863], "760": [79864], "761": [79865], "762": [79866], "763": [79867], "764": [79868], "765": [79869], "766": [79870], "767": [79871], "768": [79872], "769": [79873], "770": [79874], "771": [79875], "772": [79876], "773": [79877], "774": [79878], "775": [79879], "776": [79880], "777": [79881], "778": [79882], "779": [79883], "780": [79884], "781": [79885], "782": [79886], "783": [79887], "784": [79888], "785": [79889], "786": [79890], "787": [79891], "788": [79892], "789": [79893], "790": [79894], "791": [79895], "792": [79896], "793": [79897], "794": [79898], "795": [79899], "796": [79900], "797": [79901], "798": [79902], "799": [79903], "800": [79904], "801": [79905], "802": [79906], "803": [79907], "804": [79908], "805": [79909], "806": [79910], "807": [79911], "808": [79912], "809": [79913], "810": [79914], "811": [79915], "812": [79916], "813": [79917], "814": [79918], "815": [79919], "816": [79920], "817": [79921], "818": [79922], "819": [79923], "820": [79924], "821": [79925], "822": [79926], "823": [79927], "824": [79928], "825": [79929], "826": [79930], "827": [79931], "828": [79932], "829": [79933], "830": [79934], "831": [79935], "832": [79936], "833": [79937], "834": [79938], "835": [79939], "836": [79940], "837": [79941], "838": [79942], "839": [79943], "840": [79944], "841": [79945], "842": [79946], "843": [79947], "844": [79948], "845": [79949], "846": [79950], "847": [79951], "848": [79952], "849": [79953], "850": [79954], "851": [79955], "852": [79956], "853": [79957], "854": [79958], "855": [79959], "856": [79960], "857": [79961], "858": [79962], "859": [79963], "860": [79964], "861": [79965], "862": [79966], "863": [79967], "864": [79968], "865": [79969], "866": [79970], "867": [14956], "868": [79971], "869": [79972], "870": [79973], "871": [79974], "872": [79975], "873": [79976], "874": [79977], "875": [79978], "876": [79979], "877": [79980], "878": [79981], "879": [79982], "880": [79983], "881": [79984], "882": [79985], "883": [79986], "884": [79987], "885": [79988], "886": [79989], "887": [79990], "888": [79991], "889": [79992], "890": [79993], "891": [79994], "892": [79995], "893": [79996], "894": [79997], "895": [79998], "896": [79999], "897": [80000], "898": [80001], "899": [80002], "900": [80003], "901": [80004], "902": [80005], "903": [80006], "904": [80007], "905": [80008], "906": [80009], "907": [80010], "908": [80011], "909": [80012], "910": [80013], "911": [80014], "912": [80015], "913": [80016], "914": [80017], "915": [80018], "916": [80019], "917": [80020], "918": [80021], "919": [80022], "920": [80023], "921": [80024], "922": [80025], "923": [80026], "924": [80027], "925": [80028], "926": [80029], "927": [80030], "928": [80031], "929": [80032], "930": [80033], "931": [80034], "932": [80035], "933": [80036], "934": [80037], "935": [80038], "936": [80039], "937": [80040], "938": [80041], "939": [80042], "940": [80043], "941": [80044], "942": [80045], "943": [80046], "944": [80047], "945": [80048], "946": [80049], "947": [80050], "948": [80051], "949": [80052], "950": [80053], "951": [80054], "952": [80055], "953": [80056], "954": [80057], "955": [80058], "956": [80059], "957": [80060], "958": [80061], "959": [80062], "960": [80063], "961": [80064], "962": [80065], "963": [80066], "964": [80067], "965": [80068], "966": [80069], "967": [80070], "968": [80071], "969": [80072], "970": [80073], "971": [80074], "972": [80075], "973": [80076], "974": [80077], "975": [80078], "976": [80079], "977": [80080], "978": [80081], "979": [80082], "980": [80083], "981": [80084], "982": [80085], "983": [80086], "984": [80087], "985": [80088], "986": [9179], "987": [80089], "988": [80090], "989": [80091], "990": [80092], "991": [80093], "992": [80094], "993": [80095], "994": [304], "995": [80096], "996": [80097], "997": [80098], "998": [80099], "999": [80100], "1000": [80101], "1001": [80102], "1002": [80103], "1003": [80104], "1004": [80105], "1005": [80106], "1006": [80107], "1007": [80108], "1008": [80109], "1009": [80110], "1010": [80111], "1011": [80112], "1012": [80113], "1013": [80114], "1014": [80115], "1015": [80116], "1016": [80117], "1017": [80118], "1018": [80119], "1019": [80120], "1020": [80121], "1021": [80122], "1022": [80123], "1023": [80124], "1024": [80125], "1025": [80126], "1026": [80127], "1027": [80128], "1028": [80129], "1029": [80130], "1030": [80131], "1031": [80132], "1032": [80133], "1033": [80134], "1034": [80135], "1035": [80136], "1036": [80137], "1037": [80138], "1038": [80139], "1039": [80140], "1040": [80141], "1041": [80142], "1042": [80143], "1043": [80144], "1044": [80145], "1045": [80146], "1046": [80147], "1047": [80148], "1048": [80149], "1049": [80150], "1050": [23861], "1051": [80151], "1052": [80152], "1053": [80153], "1054": [80154], "1055": [80155], "1056": [80156], "1057": [80157], "1058": [80158], "1059": [80159], "1060": [80160], "1061": [80161], "1062": [80162], "1063": [80163], "1064": [80164], "1065": [80165], "1066": [80166], "1067": [80167], "1068": [80168], "1069": [80169], "1070": [80170], "1071": [80171], "1072": [80172], "1073": [80173], "1074": [80174], "1075": [80175], "1076": [80176], "1077": [80177], "1078": [80178], "1079": [80179], "1080": [80180], "1081": [80181], "1082": [80182], "1083": [80183], "1084": [80184], "1085": [80185], "1086": [80186], "1087": [80187], "1088": [80188], "1089": [80189], "1090": [80190], "1091": [80191], "1092": [80192], "1093": [80193], "1094": [80194], "1095": [80195], "1096": [80196], "1097": [80197], "1098": [80198], "1099": [80199], "1100": [80200], "1101": [85414], "1102": [80202], "1103": [80203], "1104": [80204], "1105": [80205], "1106": [80206], "1107": [80207], "1108": [80208], "1109": [50054], "1110": [80209], "1111": [80210], "1112": [80211], "1113": [80212], "1114": [80213], "1115": [80214], "1116": [80215], "1117": [80216], "1118": [80217], "1119": [80218], "1120": [80219], "1121": [80220], "1122": [80221], "1123": [80222], "1124": [80223], "1125": [80224], "1126": [80225], "1127": [80226], "1128": [80227], "1129": [80228], "1130": [80229], "1131": [80230], "1132": [80231], "1133": [80232], "1134": [80233], "1135": [80234], "1136": [80235], "1137": [80236], "1138": [80237], "1139": [80238], "1140": [80239], "1141": [80240], "1142": [80241], "1143": [80242], "1144": [80243], "1145": [10255], "1146": [80244], "1147": [80245], "1148": [80246], "1149": [80247], "1150": [80248], "1151": [80249], "1152": [80250], "1153": [80251], "1154": [80252], "1155": [80253], "1156": [80254], "1157": [80255], "1158": [80256], "1159": [80257], "1160": [80258], "1161": [80259], "1162": [80260], "1163": [80261], "1164": [80262], "1165": [80263], "1166": [80264], "1167": [80265], "1168": [80266], "1169": [80267], "1170": [80268], "1171": [80269], "1172": [80270], "1173": [80271], "1174": [80272], "1175": [80273], "1176": [80274], "1177": [80275], "1178": [80276], "1179": [80277], "1180": [80278], "1181": [80279], "1182": [80280], "1183": [80281], "1184": [80282], "1185": [80283], "1186": [80284], "1187": [80285], "1188": [46320], "1189": [80286], "1190": [20109], "1191": [80287], "1192": [80288], "1193": [80289], "1194": [80290], "1195": [80291], "1196": [80292], "1197": [80293], "1198": [80294], "1199": [80295], "1200": [80296], "1201": [80297], "1202": [80298], "1203": [80299], "1204": [80300], "1205": [80301], "1206": [80302], "1207": [80303], "1208": [80304], "1209": [80305], "1210": [80306], "1211": [80307], "1212": [80308], "1213": [80309], "1214": [80310], "1215": [80311], "1216": [23435], "1217": [80312], "1218": [80313], "1219": [80314], "1220": [80315], "1221": [80316], "1222": [80317], "1223": [80318], "1224": [80319], "1225": [80320], "1226": [80321], "1227": [80322], "1228": [80323], "1229": [80324], "1230": [80325], "1231": [80326], "1232": [80327], "1233": [80328], "1234": [80329], "1235": [80330], "1236": [80331], "1237": [80332], "1238": [80333], "1239": [80334], "1240": [80335], "1241": [80336], "1242": [80337], "1243": [80338], "1244": [80339], "1245": [80340], "1246": [80341], "1247": [80342], "1248": [80343], "1249": [80344], "1250": [80345], "1251": [80346], "1252": [80347], "1253": [80348], "1254": [80349], "1255": [80350], "1256": [80351], "1257": [80352], "1258": [80353], "1259": [80354], "1260": [80355], "1261": [80356], "1262": [80357], "1263": [80358], "1264": [80359], "1265": [80360], "1266": [80361], "1267": [80362], "1268": [80363], "1269": [80364], "1270": [80365], "1271": [80366], "1272": [80367], "1273": [80368], "1274": [80369], "1275": [80370], "1276": [80371], "1277": [80372], "1278": [80373], "1279": [80374], "1280": [80375], "1281": [80376], "1282": [80377], "1283": [80378], "1284": [80379], "1285": [80380], "1286": [80381], "1287": [80382], "1288": [80383], "1289": [80384], "1290": [80385], "1291": [80386], "1292": [80387], "1293": [80388], "1294": [80389], "1295": [80390], "1296": [80391], "1297": [80392], "1298": [80393], "1299": [80394], "1300": [80395], "1301": [80396], "1302": [80397], "1303": [80398], "1304": [80399], "1305": [80400], "1306": [80401], "1307": [80402], "1308": [80403], "1309": [80404], "1310": [80405], "1311": [80406], "1312": [80407], "1313": [80408], "1314": [80409], "1315": [80410], "1316": [80411], "1317": [80412], "1318": [15095], "1319": [80413], "1320": [80414], "1321": [80415], "1322": [80416], "1323": [80417], "1324": [80418], "1325": [80419], "1326": [80420], "1327": [80421], "1328": [80422], "1329": [80423], "1330": [80424], "1331": [80425], "1332": [80426], "1333": [80427], "1334": [80428], "1335": [80429], "1336": [80430], "1337": [80431], "1338": [80432], "1339": [80433], "1340": [80434], "1341": [80435], "1342": [80436], "1343": [80437], "1344": [80438], "1345": [80439], "1346": [80440], "1347": [80441], "1348": [80442], "1349": [80443], "1350": [80444], "1351": [80445], "1352": [80446], "1353": [80447], "1354": [80448], "1355": [80449], "1356": [80450], "1357": [80451], "1358": [80452], "1359": [80453], "1360": [80454], "1361": [80455], "1362": [80456], "1363": [80457], "1364": [50951], "1365": [80458], "1366": [80459], "1367": [80460], "1368": [80461], "1369": [80462], "1370": [80463], "1371": [80464], "1372": [80465], "1373": [52656], "1374": [80466], "1375": [80467], "1376": [80468], "1377": [80469], "1378": [80470], "1379": [80471], "1380": [80472], "1381": [80473], "1382": [80474], "1383": [80475], "1384": [80476], "1385": [80477], "1386": [80478], "1387": [80479], "1388": [80480], "1389": [80481], "1390": [80482], "1391": [80483], "1392": [80484], "1393": [80485], "1394": [80486], "1395": [80487], "1396": [80488], "1397": [80489], "1398": [80490], "1399": [80491], "1400": [80492], "1401": [80493], "1402": [80494], "1403": [80495], "1404": [80496], "1405": [80497], "1406": [80498], "1407": [80499], "1408": [80500], "1409": [80501], "1410": [80502], "1411": [80503], "1412": [80504], "1413": [21788], "1414": [80505], "1415": [80506], "1416": [80507], "1417": [80508], "1418": [80509], "1419": [80510], "1420": [80511], "1421": [80512], "1422": [80513], "1423": [80514], "1424": [80515], "1425": [80516], "1426": [80517], "1427": [80518], "1428": [80519], "1429": [80520], "1430": [80521], "1431": [80522], "1432": [80523], "1433": [80524], "1434": [80525], "1435": [80526], "1436": [80527], "1437": [80528], "1438": [80529], "1439": [80530], "1440": [80531], "1441": [80532], "1442": [80533], "1443": [80534], "1444": [80535], "1445": [80536], "1446": [80537], "1447": [80538], "1448": [80539], "1449": [80540], "1450": [80541], "1451": [80542], "1452": [80543], "1453": [80544], "1454": [80545], "1455": [80546], "1456": [80547], "1457": [80548], "1458": [80549], "1459": [80550], "1460": [86651], "1461": [80552], "1462": [80553], "1463": [80554], "1464": [80555], "1465": [80556], "1466": [80557], "1467": [80558], "1468": [80559], "1469": [32882], "1470": [80560], "1471": [80561], "1472": [80562], "1473": [80563], "1474": [80564], "1475": [80565], "1476": [80566], "1477": [80567], "1478": [80568], "1479": [80569], "1480": [80570], "1481": [80571], "1482": [80572], "1483": [80573], "1484": [80574], "1485": [80575], "1486": [80576], "1487": [80577], "1488": [80578], "1489": [80579], "1490": [80580], "1491": [80581], "1492": [80582], "1493": [80583], "1494": [80584], "1495": [80585], "1496": [80586], "1497": [80587], "1498": [80588], "1499": [80589], "1500": [80590], "1501": [80591], "1502": [80592], "1503": [80593], "1504": [80594], "1505": [80595], "1506": [80596], "1507": [80597], "1508": [80598], "1509": [80599], "1510": [80600], "1511": [80601], "1512": [80602], "1513": [80603], "1514": [80604], "1515": [80605], "1516": [80606], "1517": [80607], "1518": [80608], "1519": [80609], "1520": [80610], "1521": [80611], "1522": [80612], "1523": [80613], "1524": [80614], "1525": [80615], "1526": [80616], "1527": [80617], "1528": [80618], "1529": [80619], "1530": [80620], "1531": [80621], "1532": [80622], "1533": [80623], "1534": [80624], "1535": [80625], "1536": [80626], "1537": [80627], "1538": [80628], "1539": [80629], "1540": [80630], "1541": [80631], "1542": [80632], "1543": [80633], "1544": [80634], "1545": [80635], "1546": [80636], "1547": [80637], "1548": [80638], "1549": [80639], "1550": [80640], "1551": [19300], "1552": [80641], "1553": [80642], "1554": [80643], "1555": [80644], "1556": [80645], "1557": [80646], "1558": [80647], "1559": [80648], "1560": [80649], "1561": [80650], "1562": [80651], "1563": [80652], "1564": [80653], "1565": [80654], "1566": [80655], "1567": [80656], "1568": [80657], "1569": [80658], "1570": [80659], "1571": [80660], "1572": [80661], "1573": [80662], "1574": [80663], "1575": [80664], "1576": [80665], "1577": [80666], "1578": [80667], "1579": [80668], "1580": [80669], "1581": [80670], "1582": [80671], "1583": [80672], "1584": [80673], "1585": [80674], "1586": [80675], "1587": [80676], "1588": [80677], "1589": [80678], "1590": [80679], "1591": [80680], "1592": [80681], "1593": [80682], "1594": [80683], "1595": [80684], "1596": [80685], "1597": [80686], "1598": [80687], "1599": [80688], "1600": [80689], "1601": [80690], "1602": [80691], "1603": [80692], "1604": [80693], "1605": [80694], "1606": [80695], "1607": [80696], "1608": [34773], "1609": [80697], "1610": [80698], "1611": [80699], "1612": [80700], "1613": [80701], "1614": [80702], "1615": [80703], "1616": [80704], "1617": [80705], "1618": [80706], "1619": [80707], "1620": [80708], "1621": [80709], "1622": [80710], "1623": [80711], "1624": [80712], "1625": [80713], "1626": [10255], "1627": [80714], "1628": [80715], "1629": [80716], "1630": [80717], "1631": [80718], "1632": [80719], "1633": [80720], "1634": [80721], "1635": [80722], "1636": [80723], "1637": [80724], "1638": [80725], "1639": [80726], "1640": [80727], "1641": [80728], "1642": [80729], "1643": [80730], "1644": [80731], "1645": [80732], "1646": [80733], "1647": [80734], "1648": [80735], "1649": [80736], "1650": [80737], "1651": [80738], "1652": [80739], "1653": [80740], "1654": [80741], "1655": [80742], "1656": [80743], "1657": [80744], "1658": [80745], "1659": [80746], "1660": [80747], "1661": [80748], "1662": [80749], "1663": [80750], "1664": [80751], "1665": [80752], "1666": [80753], "1667": [80754], "1668": [80755], "1669": [80756], "1670": [80757], "1671": [80758], "1672": [80759], "1673": [80760], "1674": [80761], "1675": [80762], "1676": [80763], "1677": [80764], "1678": [80765], "1679": [80766], "1680": [80767], "1681": [80768], "1682": [80769], "1683": [80770], "1684": [80771], "1685": [80772], "1686": [80773], "1687": [80774], "1688": [80775], "1689": [80776], "1690": [80777], "1691": [80778], "1692": [80779], "1693": [80780], "1694": [80781], "1695": [80782], "1696": [80783], "1697": [80784], "1698": [80785], "1699": [80786], "1700": [80787], "1701": [80788], "1702": [80789], "1703": [80790], "1704": [80791], "1705": [80792], "1706": [80793], "1707": [80794], "1708": [80795], "1709": [80796], "1710": [80797], "1711": [80798], "1712": [80799], "1713": [80800], "1714": [80801], "1715": [80802], "1716": [80803], "1717": [80804], "1718": [80805], "1719": [80806], "1720": [80807], "1721": [80808], "1722": [80809], "1723": [80810], "1724": [80811], "1725": [80812], "1726": [80813], "1727": [80814], "1728": [80815], "1729": [80816], "1730": [80817], "1731": [80818], "1732": [80819], "1733": [80820], "1734": [80821], "1735": [80822], "1736": [80823], "1737": [80824], "1738": [80825], "1739": [80826], "1740": [80827], "1741": [80828], "1742": [80829], "1743": [80830], "1744": [80831], "1745": [80832], "1746": [80833], "1747": [80834], "1748": [80835], "1749": [80836], "1750": [80837], "1751": [80838], "1752": [80839], "1753": [80840], "1754": [80841], "1755": [80842], "1756": [80843], "1757": [80844], "1758": [80845], "1759": [80846], "1760": [80847], "1761": [80848], "1762": [80849], "1763": [80850], "1764": [80851], "1765": [80852], "1766": [80853], "1767": [81387], "1768": [80855], "1769": [80856], "1770": [80857], "1771": [80858], "1772": [80859], "1773": [80860], "1774": [80861], "1775": [80862], "1776": [80863], "1777": [80864], "1778": [80865], "1779": [80866], "1780": [80867], "1781": [80868], "1782": [80869], "1783": [80870], "1784": [80871], "1785": [80872], "1786": [80873], "1787": [80874], "1788": [80875], "1789": [80876], "1790": [80877], "1791": [80878], "1792": [80879], "1793": [80880], "1794": [5458], "1795": [80881], "1796": [80882], "1797": [80883], "1798": [80884], "1799": [80885], "1800": [80886], "1801": [80887], "1802": [80888], "1803": [80889], "1804": [80890], "1805": [80891], "1806": [80892], "1807": [80893], "1808": [80894], "1809": [80895], "1810": [80896], "1811": [80897], "1812": [80898], "1813": [80899], "1814": [80900], "1815": [80901], "1816": [80902], "1817": [80903], "1818": [80904], "1819": [80905], "1820": [80906], "1821": [80907], "1822": [80908], "1823": [80909], "1824": [80910], "1825": [80911], "1826": [80912], "1827": [80913], "1828": [80914], "1829": [80915], "1830": [80916], "1831": [80917], "1832": [80918], "1833": [80919], "1834": [80920], "1835": [80921], "1836": [80922], "1837": [80923], "1838": [80924], "1839": [80925], "1840": [80926], "1841": [80927], "1842": [80928], "1843": [80929], "1844": [80930], "1845": [80931], "1846": [80932], "1847": [80933], "1848": [80934], "1849": [80935], "1850": [80936], "1851": [80937], "1852": [80938], "1853": [80939], "1854": [80940], "1855": [80941], "1856": [80942], "1857": [80943], "1858": [80944], "1859": [80945], "1860": [80946], "1861": [80947], "1862": [80948], "1863": [80949], "1864": [80950], "1865": [80951], "1866": [80952], "1867": [80953], "1868": [80954], "1869": [80955], "1870": [80956], "1871": [80957], "1872": [80958], "1873": [80959], "1874": [80960], "1875": [80961], "1876": [80962], "1877": [80963], "1878": [80964], "1879": [80965], "1880": [80966], "1881": [80967], "1882": [80968], "1883": [80969], "1884": [80970], "1885": [80971], "1886": [80972], "1887": [80973], "1888": [80974], "1889": [80975], "1890": [80976], "1891": [80977], "1892": [80978], "1893": [80979], "1894": [80980], "1895": [80981], "1896": [80982], "1897": [80983], "1898": [80984], "1899": [80985], "1900": [80986], "1901": [80987], "1902": [80988], "1903": [80989], "1904": [80990], "1905": [80991], "1906": [80992], "1907": [80993], "1908": [80994], "1909": [80995], "1910": [80996], "1911": [80997], "1912": [80998], "1913": [80999], "1914": [81000], "1915": [81001], "1916": [81002], "1917": [81003], "1918": [81004], "1919": [81005], "1920": [81006], "1921": [81007], "1922": [81008], "1923": [81009], "1924": [81010], "1925": [81011], "1926": [81012], "1927": [2983], "1928": [81013], "1929": [81014], "1930": [81015], "1931": [81016], "1932": [81017], "1933": [81018], "1934": [81019], "1935": [81020], "1936": [81021], "1937": [81022], "1938": [2147], "1939": [81023], "1940": [81024], "1941": [81025], "1942": [81026], "1943": [81027], "1944": [81028], "1945": [81029], "1946": [81030], "1947": [81031], "1948": [81032], "1949": [81033], "1950": [81034], "1951": [81035], "1952": [81036], "1953": [81037], "1954": [81038], "1955": [81039], "1956": [81040], "1957": [81041], "1958": [81042], "1959": [81043], "1960": [81044], "1961": [81045], "1962": [81046], "1963": [81047], "1964": [81048], "1965": [81049], "1966": [81050], "1967": [81051], "1968": [81052], "1969": [81053], "1970": [81054], "1971": [81055], "1972": [81056], "1973": [16610], "1974": [81057], "1975": [81058], "1976": [81059], "1977": [81060], "1978": [81061], "1979": [81062], "1980": [81063], "1981": [81064], "1982": [81065], "1983": [81066], "1984": [81067], "1985": [81068], "1986": [81069], "1987": [81070], "1988": [81071], "1989": [81072], "1990": [81073], "1991": [48748], "1992": [81074], "1993": [81075], "1994": [81076], "1995": [81077], "1996": [81078], "1997": [81079], "1998": [81080], "1999": [81081], "2000": [81082], "2001": [49246], "2002": [81083], "2003": [81084], "2004": [81085], "2005": [81086], "2006": [81087], "2007": [81088], "2008": [81089], "2009": [81090], "2010": [81091], "2011": [81092], "2012": [81093], "2013": [81094], "2014": [81095], "2015": [81096], "2016": [81097], "2017": [81098], "2018": [81099], "2019": [81100], "2020": [81101], "2021": [81102], "2022": [81103], "2023": [10255], "2024": [81104], "2025": [81105], "2026": [81106], "2027": [81107], "2028": [81108], "2029": [81109], "2030": [81110], "2031": [81111], "2032": [81112], "2033": [81113], "2034": [81114], "2035": [81115], "2036": [81116], "2037": [81117], "2038": [81118], "2039": [81119], "2040": [81120], "2041": [81121], "2042": [81122], "2043": [81123], "2044": [81124], "2045": [81125], "2046": [81126], "2047": [81127], "2048": [81128], "2049": [81129], "2050": [81130], "2051": [81131], "2052": [81132], "2053": [81133], "2054": [81134], "2055": [81135], "2056": [81136], "2057": [81137], "2058": [81138], "2059": [81139], "2060": [81140], "2061": [81141], "2062": [81142], "2063": [81143], "2064": [81144], "2065": [81145], "2066": [81146], "2067": [81147], "2068": [81148], "2069": [81149], "2070": [81150], "2071": [81151], "2072": [81152], "2073": [81153], "2074": [81154], "2075": [81155], "2076": [81156], "2077": [81157], "2078": [81158], "2079": [81159], "2080": [81160], "2081": [81161], "2082": [81162], "2083": [81163], "2084": [81164], "2085": [81165], "2086": [81166], "2087": [81167], "2088": [81168], "2089": [81169], "2090": [81170], "2091": [81171], "2092": [80082], "2093": [81172], "2094": [81173], "2095": [81174], "2096": [81175], "2097": [81176], "2098": [81177], "2099": [81178], "2100": [81179], "2101": [81180], "2102": [81181], "2103": [81182], "2104": [81183], "2105": [81184], "2106": [81185], "2107": [81186], "2108": [81187], "2109": [81188], "2110": [81189], "2111": [81190], "2112": [81191], "2113": [81192], "2114": [81193], "2115": [81194], "2116": [81195], "2117": [81196], "2118": [81197], "2119": [81198], "2120": [81199], "2121": [8489], "2122": [81200], "2123": [81201], "2124": [81202], "2125": [81203], "2126": [81204], "2127": [81205], "2128": [28850], "2129": [81206], "2130": [81207], "2131": [81208], "2132": [43456], "2133": [81209], "2134": [81210], "2135": [81211], "2136": [81212], "2137": [81213], "2138": [6806], "2139": [81214], "2140": [81215], "2141": [81216], "2142": [81217], "2143": [81218], "2144": [81219], "2145": [81220], "2146": [81221], "2147": [81222], "2148": [81223], "2149": [81224], "2150": [81225], "2151": [81226], "2152": [81227], "2153": [81228], "2154": [81229], "2155": [81230], "2156": [81231], "2157": [81232], "2158": [81233], "2159": [81234], "2160": [81235], "2161": [81236], "2162": [81237], "2163": [81238], "2164": [81239], "2165": [81240], "2166": [81241], "2167": [81242], "2168": [81243], "2169": [81244], "2170": [81245], "2171": [81246], "2172": [81247], "2173": [81248], "2174": [81249], "2175": [81250], "2176": [81251], "2177": [81252], "2178": [81253], "2179": [81254], "2180": [81255], "2181": [81256], "2182": [81257], "2183": [81258], "2184": [81259], "2185": [81260], "2186": [81261], "2187": [81262], "2188": [81263], "2189": [81264], "2190": [81265], "2191": [81266], "2192": [81267], "2193": [81268], "2194": [81269], "2195": [81270], "2196": [81271], "2197": [81272], "2198": [81273], "2199": [81274], "2200": [81275], "2201": [81276], "2202": [81277], "2203": [81278], "2204": [81279], "2205": [81280], "2206": [81281], "2207": [81282], "2208": [81283], "2209": [81284], "2210": [81285], "2211": [81286], "2212": [81287], "2213": [81288], "2214": [81289], "2215": [81290], "2216": [81291], "2217": [81292], "2218": [81293], "2219": [81294], "2220": [81295], "2221": [81296], "2222": [81297], "2223": [81298], "2224": [81299], "2225": [81300], "2226": [81301], "2227": [81302], "2228": [81303], "2229": [81304], "2230": [81305], "2231": [81306], "2232": [81307], "2233": [81308], "2234": [81309], "2235": [81310], "2236": [81311], "2237": [81312], "2238": [81313], "2239": [81314], "2240": [81315], "2241": [81316], "2242": [81317], "2243": [81318], "2244": [81319], "2245": [81320], "2246": [81321], "2247": [81322], "2248": [81323], "2249": [81324], "2250": [81325], "2251": [81326], "2252": [81327], "2253": [81328], "2254": [81329], "2255": [81330], "2256": [81331], "2257": [81332], "2258": [81333], "2259": [81334], "2260": [81335], "2261": [81336], "2262": [81337], "2263": [81338], "2264": [54373], "2265": [81339], "2266": [81340], "2267": [81341], "2268": [81342], "2269": [81343], "2270": [81344], "2271": [81345], "2272": [81346], "2273": [81347], "2274": [81348], "2275": [81349], "2276": [81350], "2277": [45500], "2278": [81351], "2279": [81352], "2280": [81353], "2281": [81354], "2282": [81355], "2283": [81356], "2284": [81357], "2285": [81358], "2286": [81359], "2287": [81360], "2288": [81361], "2289": [81362], "2290": [81363], "2291": [81364], "2292": [81365], "2293": [81366], "2294": [81367], "2295": [81368], "2296": [81369], "2297": [81370], "2298": [81371], "2299": [81372], "2300": [81373], "2301": [81374], "2302": [81375], "2303": [81376], "2304": [81377], "2305": [81378], "2306": [81379], "2307": [81380], "2308": [81381], "2309": [81382], "2310": [86115], "2311": [81384], "2312": [81385], "2313": [81386], "2314": [81388], "2315": [81389], "2316": [81390], "2317": [81391], "2318": [81392], "2319": [81393], "2320": [81394], "2321": [81395], "2322": [81396], "2323": [81397], "2324": [81398], "2325": [81399], "2326": [81400], "2327": [81401], "2328": [81402], "2329": [81403], "2330": [81404], "2331": [81405], "2332": [81406], "2333": [81407], "2334": [81408], "2335": [81409], "2336": [17024], "2337": [81410], "2338": [81411], "2339": [81412], "2340": [81413], "2341": [81414], "2342": [81415], "2343": [81416], "2344": [81417], "2345": [81418], "2346": [81419], "2347": [81420], "2348": [81421], "2349": [81422], "2350": [81423], "2351": [81424], "2352": [81425], "2353": [81426], "2354": [81427], "2355": [81428], "2356": [81429], "2357": [81430], "2358": [81431], "2359": [81432], "2360": [81433], "2361": [81434], "2362": [2199], "2363": [81435], "2364": [81436], "2365": [81437], "2366": [81438], "2367": [81439], "2368": [81440], "2369": [81441], "2370": [81442], "2371": [48296], "2372": [81443], "2373": [81444], "2374": [81445], "2375": [81446], "2376": [81447], "2377": [81448], "2378": [81449], "2379": [81450], "2380": [81451], "2381": [81452], "2382": [81453], "2383": [81454], "2384": [81455], "2385": [81456], "2386": [81457], "2387": [81458], "2388": [81459], "2389": [81460], "2390": [81461], "2391": [81462], "2392": [81463], "2393": [81464], "2394": [81465], "2395": [81466], "2396": [81467], "2397": [81468], "2398": [81469], "2399": [81470], "2400": [81471], "2401": [81472], "2402": [81473], "2403": [81474], "2404": [81475], "2405": [81476], "2406": [81477], "2407": [81478], "2408": [81479], "2409": [81480], "2410": [81481], "2411": [81482], "2412": [81483], "2413": [81484], "2414": [81485], "2415": [81486], "2416": [81487], "2417": [81488], "2418": [81489], "2419": [81490], "2420": [81491], "2421": [81492], "2422": [81493], "2423": [81494], "2424": [81495], "2425": [81496], "2426": [81497], "2427": [81498], "2428": [81499], "2429": [81500], "2430": [81501], "2431": [81502], "2432": [81503], "2433": [81504], "2434": [81505], "2435": [81506], "2436": [81507], "2437": [81508], "2438": [81509], "2439": [81510], "2440": [81511], "2441": [81512], "2442": [81513], "2443": [81514], "2444": [81515], "2445": [81516], "2446": [81517], "2447": [81518], "2448": [81519], "2449": [10255], "2450": [81520], "2451": [81521], "2452": [81522], "2453": [81523], "2454": [81524], "2455": [81525], "2456": [81526], "2457": [81527], "2458": [81528], "2459": [81529], "2460": [81530], "2461": [81531], "2462": [81532], "2463": [81533], "2464": [81534], "2465": [81535], "2466": [81536], "2467": [81537], "2468": [81538], "2469": [81539], "2470": [81540], "2471": [81541], "2472": [81542], "2473": [81543], "2474": [81544], "2475": [81545], "2476": [81546], "2477": [81547], "2478": [81548], "2479": [81549], "2480": [81550], "2481": [81551], "2482": [81552], "2483": [81553], "2484": [81554], "2485": [81555], "2486": [81556], "2487": [81557], "2488": [81558], "2489": [81559], "2490": [81560], "2491": [81561], "2492": [81562], "2493": [81563], "2494": [81564], "2495": [81565], "2496": [81566], "2497": [81567], "2498": [81568], "2499": [81569], "2500": [81570], "2501": [81571], "2502": [81572], "2503": [81573], "2504": [81574], "2505": [81575], "2506": [81576], "2507": [81577], "2508": [81578], "2509": [81579], "2510": [81580], "2511": [81581], "2512": [81582], "2513": [81583], "2514": [81584], "2515": [81585], "2516": [81586], "2517": [81587], "2518": [81588], "2519": [81589], "2520": [81590], "2521": [81591], "2522": [81592], "2523": [81593], "2524": [81594], "2525": [81595], "2526": [81596], "2527": [81597], "2528": [81598], "2529": [81599], "2530": [81600], "2531": [81601], "2532": [81602], "2533": [81603], "2534": [81604], "2535": [81605], "2536": [13903], "2537": [81606], "2538": [81607], "2539": [81608], "2540": [81609], "2541": [81610], "2542": [81611], "2543": [81612], "2544": [81613], "2545": [81614], "2546": [81615], "2547": [81616], "2548": [81617], "2549": [81618], "2550": [81619], "2551": [81620], "2552": [81621], "2553": [81622], "2554": [81623], "2555": [81624], "2556": [81625], "2557": [81626], "2558": [81627], "2559": [81628], "2560": [81629], "2561": [81630], "2562": [81631], "2563": [81632], "2564": [81633], "2565": [81634], "2566": [81635], "2567": [81636], "2568": [81637], "2569": [81638], "2570": [81639], "2571": [81640], "2572": [81641], "2573": [81642], "2574": [81643], "2575": [81644], "2576": [81645], "2577": [81646], "2578": [81647], "2579": [81648], "2580": [81649], "2581": [81650], "2582": [81651], "2583": [81652], "2584": [81653], "2585": [81654], "2586": [81655], "2587": [81656], "2588": [81657], "2589": [81658], "2590": [81659], "2591": [81660], "2592": [81661], "2593": [81662], "2594": [81663], "2595": [81664], "2596": [81665], "2597": [81666], "2598": [81667], "2599": [81668], "2600": [81669], "2601": [81670], "2602": [81671], "2603": [81672], "2604": [81673], "2605": [81674], "2606": [81675], "2607": [81676], "2608": [81677], "2609": [81678], "2610": [81679], "2611": [81680], "2612": [81681], "2613": [49317], "2614": [81682], "2615": [81683], "2616": [81684], "2617": [81685], "2618": [81686], "2619": [81687], "2620": [851], "2621": [81688], "2622": [50336], "2623": [81689], "2624": [81690], "2625": [81691], "2626": [81692], "2627": [5614], "2628": [81693], "2629": [81694], "2630": [81695], "2631": [81696], "2632": [81697], "2633": [81698], "2634": [81699], "2635": [81700], "2636": [81701], "2637": [81702], "2638": [81703], "2639": [81704], "2640": [81705], "2641": [81706], "2642": [81707], "2643": [81708], "2644": [81709], "2645": [81710], "2646": [81711], "2647": [81712], "2648": [81713], "2649": [81714], "2650": [81715], "2651": [81716], "2652": [81717], "2653": [81718], "2654": [81719], "2655": [81720], "2656": [81721], "2657": [81722], "2658": [81723], "2659": [81724], "2660": [81725], "2661": [2147], "2662": [81726], "2663": [81727], "2664": [81728], "2665": [81729], "2666": [81730], "2667": [81731], "2668": [81732], "2669": [81733], "2670": [81734], "2671": [81735], "2672": [81736], "2673": [81737], "2674": [81738], "2675": [81739], "2676": [81740], "2677": [81741], "2678": [81742], "2679": [81743], "2680": [81744], "2681": [81745], "2682": [81746], "2683": [81747], "2684": [81748], "2685": [81749], "2686": [81750], "2687": [81751], "2688": [81752], "2689": [81753], "2690": [81754], "2691": [81755], "2692": [81756], "2693": [81757], "2694": [41162], "2695": [81758], "2696": [81759], "2697": [81760], "2698": [81761], "2699": [81762], "2700": [81763], "2701": [81764], "2702": [81765], "2703": [81766], "2704": [81767], "2705": [81768], "2706": [81769], "2707": [81770], "2708": [81771], "2709": [81772], "2710": [81773], "2711": [81774], "2712": [81775], "2713": [81776], "2714": [81777], "2715": [81778], "2716": [81779], "2717": [81780], "2718": [81781], "2719": [81782], "2720": [81783], "2721": [81784], "2722": [81785], "2723": [81786], "2724": [81787], "2725": [81788], "2726": [81789], "2727": [81790], "2728": [81791], "2729": [81792], "2730": [81793], "2731": [81794], "2732": [81795], "2733": [81796], "2734": [81797], "2735": [81798], "2736": [81799], "2737": [81800], "2738": [81801], "2739": [81802], "2740": [81803], "2741": [81804], "2742": [81806], "2743": [81807], "2744": [81808], "2745": [81809], "2746": [81810], "2747": [81811], "2748": [81812], "2749": [5762], "2750": [81767], "2751": [81813], "2752": [81814], "2753": [81815], "2754": [81816], "2755": [81817], "2756": [81818], "2757": [81819], "2758": [81820], "2759": [81821], "2760": [81822], "2761": [81823], "2762": [81824], "2763": [81825], "2764": [81826], "2765": [81827], "2766": [81828], "2767": [81829], "2768": [81830], "2769": [81831], "2770": [81832], "2771": [81833], "2772": [81834], "2773": [81835], "2774": [45722], "2775": [81836], "2776": [81837], "2777": [81838], "2778": [81839], "2779": [81840], "2780": [81841], "2781": [81842], "2782": [81843], "2783": [81844], "2784": [81845], "2785": [81846], "2786": [81847], "2787": [81848], "2788": [81849], "2789": [81850], "2790": [81851], "2791": [81852], "2792": [81853], "2793": [81854], "2794": [81855], "2795": [81856], "2796": [81857], "2797": [81858], "2798": [81859], "2799": [81860], "2800": [81861], "2801": [81862], "2802": [81863], "2803": [81864], "2804": [81865], "2805": [81866], "2806": [81867], "2807": [81868], "2808": [81869], "2809": [81870], "2810": [81871], "2811": [81872], "2812": [81873], "2813": [81874], "2814": [81875], "2815": [81876], "2816": [81877], "2817": [81878], "2818": [81879], "2819": [81880], "2820": [81881], "2821": [81882], "2822": [81883], "2823": [81884], "2824": [81885], "2825": [81886], "2826": [81887], "2827": [81888], "2828": [81889], "2829": [81890], "2830": [81891], "2831": [81892], "2832": [81893], "2833": [81894], "2834": [2596], "2835": [81895], "2836": [81896], "2837": [47488], "2838": [81897], "2839": [81898], "2840": [81899], "2841": [81900], "2842": [81901], "2843": [32758], "2844": [81902], "2845": [81903], "2846": [81904], "2847": [81905], "2848": [81906], "2849": [81907], "2850": [81908], "2851": [81909], "2852": [81910], "2853": [81911], "2854": [81912], "2855": [81913], "2856": [81914], "2857": [81915], "2858": [81916], "2859": [81917], "2860": [81918], "2861": [81919], "2862": [81920], "2863": [81921], "2864": [81922], "2865": [81923], "2866": [81924], "2867": [81925], "2868": [81926], "2869": [81927], "2870": [81928], "2871": [81929], "2872": [81930], "2873": [81931], "2874": [81932], "2875": [81933], "2876": [81934], "2877": [81935], "2878": [81936], "2879": [81937], "2880": [81938], "2881": [81939], "2882": [81940], "2883": [81941], "2884": [81942], "2885": [81943], "2886": [81944], "2887": [81945], "2888": [81946], "2889": [81947], "2890": [81948], "2891": [81949], "2892": [81950], "2893": [81951], "2894": [81952], "2895": [81953], "2896": [81954], "2897": [81955], "2898": [81956], "2899": [81957], "2900": [81958], "2901": [81959], "2902": [81960], "2903": [81961], "2904": [81962], "2905": [81963], "2906": [81964], "2907": [81965], "2908": [81966], "2909": [81967], "2910": [81968], "2911": [81969], "2912": [30665], "2913": [81970], "2914": [81971], "2915": [37810], "2916": [81972], "2917": [81973], "2918": [81974], "2919": [81975], "2920": [81976], "2921": [81977], "2922": [81978], "2923": [81979], "2924": [81980], "2925": [81981], "2926": [81982], "2927": [81983], "2928": [81984], "2929": [81985], "2930": [81986], "2931": [81987], "2932": [81988], "2933": [81989], "2934": [81990], "2935": [81991], "2936": [81992], "2937": [81993], "2938": [81994], "2939": [81995], "2940": [81996], "2941": [81997], "2942": [81998], "2943": [81999], "2944": [82000], "2945": [82001], "2946": [82002], "2947": [82003], "2948": [82004], "2949": [82005], "2950": [82006], "2951": [82007], "2952": [82008], "2953": [82009], "2954": [82010], "2955": [82011], "2956": [82012], "2957": [84784], "2958": [82014], "2959": [82015], "2960": [82016], "2961": [82017], "2962": [82018], "2963": [82019], "2964": [82020], "2965": [82021], "2966": [82022], "2967": [82023], "2968": [82024], "2969": [82025], "2970": [82026], "2971": [82027], "2972": [82028], "2973": [82029], "2974": [82030], "2975": [82031], "2976": [82032], "2977": [82033], "2978": [82034], "2979": [82035], "2980": [82036], "2981": [82037], "2982": [82038], "2983": [20838], "2984": [82039], "2985": [82040], "2986": [82041], "2987": [82042], "2988": [82043], "2989": [82044], "2990": [82045], "2991": [82046], "2992": [82047], "2993": [82048], "2994": [82049], "2995": [26761], "2996": [82050], "2997": [82051], "2998": [82052], "2999": [82053], "3000": [82054], "3001": [82055], "3002": [82056], "3003": [82057], "3004": [82058], "3005": [82059], "3006": [82060], "3007": [82061], "3008": [82062], "3009": [82063], "3010": [82064], "3011": [82065], "3012": [82066], "3013": [82067], "3014": [82068], "3015": [82069], "3016": [82070], "3017": [82071], "3018": [82072], "3019": [82073], "3020": [82074], "3021": [82075], "3022": [82076], "3023": [82077], "3024": [82078], "3025": [82079], "3026": [82080], "3027": [82081], "3028": [82082], "3029": [82083], "3030": [82084], "3031": [82085], "3032": [82086], "3033": [82087], "3034": [82088], "3035": [82089], "3036": [82090], "3037": [82091], "3038": [82092], "3039": [82093], "3040": [82094], "3041": [82095], "3042": [82096], "3043": [82097], "3044": [82098], "3045": [82099], "3046": [82100], "3047": [82101], "3048": [82102], "3049": [82103], "3050": [82104], "3051": [82105], "3052": [82106], "3053": [49482], "3054": [82107], "3055": [82108], "3056": [82109], "3057": [82110], "3058": [82111], "3059": [82112], "3060": [82113], "3061": [82114], "3062": [82115], "3063": [82116], "3064": [82117], "3065": [82118], "3066": [82119], "3067": [82120], "3068": [82121], "3069": [82122], "3070": [82123], "3071": [82124], "3072": [82125], "3073": [82126], "3074": [82127], "3075": [82128], "3076": [82129], "3077": [82130], "3078": [82131], "3079": [82132], "3080": [82133], "3081": [82134], "3082": [82135], "3083": [82136], "3084": [82137], "3085": [82138], "3086": [82139], "3087": [82140], "3088": [82141], "3089": [82142], "3090": [82143], "3091": [82144], "3092": [82145], "3093": [82146], "3094": [82147], "3095": [82148], "3096": [82149], "3097": [82150], "3098": [82151], "3099": [82152], "3100": [82153], "3101": [82154], "3102": [82155], "3103": [82156], "3104": [82157], "3105": [82158], "3106": [82159], "3107": [82160], "3108": [82161], "3109": [82162], "3110": [82163], "3111": [82164], "3112": [82165], "3113": [82166], "3114": [82167], "3115": [82168], "3116": [82169], "3117": [82170], "3118": [82171], "3119": [82172], "3120": [82173], "3121": [82174], "3122": [82175], "3123": [82176], "3124": [82177], "3125": [82178], "3126": [82179], "3127": [82180], "3128": [82181], "3129": [82182], "3130": [82183], "3131": [82184], "3132": [82185], "3133": [82186], "3134": [82187], "3135": [82188], "3136": [82189], "3137": [82190], "3138": [82191], "3139": [82192], "3140": [82193], "3141": [82194], "3142": [82195], "3143": [82196], "3144": [82197], "3145": [82198], "3146": [82199], "3147": [82200], "3148": [82201], "3149": [86526], "3150": [82202], "3151": [82203], "3152": [82204], "3153": [82205], "3154": [82206], "3155": [82207], "3156": [82208], "3157": [82209], "3158": [82210], "3159": [82211], "3160": [82212], "3161": [82213], "3162": [82214], "3163": [16759], "3164": [82215], "3165": [82216], "3166": [82217], "3167": [82218], "3168": [82219], "3169": [82220], "3170": [82221], "3171": [10898], "3172": [82222], "3173": [82223], "3174": [82224], "3175": [82225], "3176": [82226], "3177": [82227], "3178": [82033], "3179": [82228], "3180": [82229], "3181": [82230], "3182": [82231], "3183": [82232], "3184": [82233], "3185": [82234], "3186": [82235], "3187": [82236], "3188": [82237], "3189": [82238], "3190": [82239], "3191": [82240], "3192": [82241], "3193": [82242], "3194": [82243], "3195": [82244], "3196": [82245], "3197": [82246], "3198": [82247], "3199": [82248], "3200": [82249], "3201": [82250], "3202": [82251], "3203": [82252], "3204": [82253], "3205": [82254], "3206": [82255], "3207": [82256], "3208": [82257], "3209": [82258], "3210": [82259], "3211": [82260], "3212": [82261], "3213": [82262], "3214": [10255], "3215": [82263], "3216": [82264], "3217": [82265], "3218": [82266], "3219": [82267], "3220": [82268], "3221": [82269], "3222": [82270], "3223": [82271], "3224": [82272], "3225": [82273], "3226": [82274], "3227": [82275], "3228": [82276], "3229": [82277], "3230": [82278], "3231": [82279], "3232": [82280], "3233": [82281], "3234": [82282], "3235": [82283], "3236": [82284], "3237": [82285], "3238": [82286], "3239": [82287], "3240": [82288], "3241": [2020], "3242": [82289], "3243": [48415], "3244": [82290], "3245": [82291], "3246": [82292], "3247": [82293], "3248": [82294], "3249": [82295], "3250": [82296], "3251": [82297], "3252": [82298], "3253": [82299], "3254": [82300], "3255": [82301], "3256": [82302], "3257": [82303], "3258": [82304], "3259": [82305], "3260": [82306], "3261": [82307], "3262": [82308], "3263": [82309], "3264": [82310], "3265": [82311], "3266": [82312], "3267": [82313], "3268": [82314], "3269": [82315], "3270": [82316], "3271": [82317], "3272": [82318], "3273": [82319], "3274": [82320], "3275": [82321], "3276": [82322], "3277": [82323], "3278": [82324], "3279": [82325], "3280": [82326], "3281": [82327], "3282": [82328], "3283": [82329], "3284": [82330], "3285": [82331], "3286": [82332], "3287": [82333], "3288": [82334], "3289": [82335], "3290": [82336], "3291": [82337], "3292": [82338], "3293": [82339], "3294": [82340], "3295": [82341], "3296": [82342], "3297": [82343], "3298": [82344], "3299": [82345], "3300": [82346], "3301": [82347], "3302": [82348], "3303": [82349], "3304": [82350], "3305": [82351], "3306": [82352], "3307": [82353], "3308": [82354], "3309": [82355], "3310": [82356], "3311": [82357], "3312": [82358], "3313": [82359], "3314": [82360], "3315": [82361], "3316": [82362], "3317": [82363], "3318": [82364], "3319": [82365], "3320": [82366], "3321": [82367], "3322": [82368], "3323": [82369], "3324": [82370], "3325": [82371], "3326": [82372], "3327": [82373], "3328": [82374], "3329": [82375], "3330": [82376], "3331": [82377], "3332": [82378], "3333": [82379], "3334": [82380], "3335": [82381], "3336": [84559], "3337": [82383], "3338": [82384], "3339": [82385], "3340": [82386], "3341": [82387], "3342": [82388], "3343": [85043], "3344": [82390], "3345": [82391], "3346": [82392], "3347": [82393], "3348": [82394], "3349": [82395], "3350": [82396], "3351": [82397], "3352": [82398], "3353": [82399], "3354": [82400], "3355": [82401], "3356": [82402], "3357": [82403], "3358": [82404], "3359": [82405], "3360": [82406], "3361": [82407], "3362": [82408], "3363": [82409], "3364": [82410], "3365": [82411], "3366": [82412], "3367": [82413], "3368": [82414], "3369": [82415], "3370": [82416], "3371": [82417], "3372": [82418], "3373": [82419], "3374": [82420], "3375": [82421], "3376": [82422], "3377": [82423], "3378": [82424], "3379": [82425], "3380": [82426], "3381": [82427], "3382": [82428], "3383": [56187], "3384": [82429], "3385": [82430], "3386": [82431], "3387": [82432], "3388": [82433], "3389": [82434], "3390": [82435], "3391": [82436], "3392": [82437], "3393": [82438], "3394": [82439], "3395": [82440], "3396": [82441], "3397": [82442], "3398": [82443], "3399": [82444], "3400": [82445], "3401": [82446], "3402": [82447], "3403": [82448], "3404": [82449], "3405": [82450], "3406": [82451], "3407": [82452], "3408": [82453], "3409": [4021], "3410": [82454], "3411": [82455], "3412": [82456], "3413": [82457], "3414": [82458], "3415": [82459], "3416": [82460], "3417": [82461], "3418": [82462], "3419": [82463], "3420": [82464], "3421": [82465], "3422": [82466], "3423": [82467], "3424": [82468], "3425": [82469], "3426": [82470], "3427": [82471], "3428": [82472], "3429": [82473], "3430": [82474], "3431": [82475], "3432": [82476], "3433": [82477], "3434": [82478], "3435": [82479], "3436": [82480], "3437": [82481], "3438": [82482], "3439": [82483], "3440": [82484], "3441": [82485], "3442": [82486], "3443": [82487], "3444": [82488], "3445": [82489], "3446": [82490], "3447": [82491], "3448": [82492], "3449": [82493], "3450": [82494], "3451": [82495], "3452": [82496], "3453": [82497], "3454": [8409], "3455": [82498], "3456": [82499], "3457": [82500], "3458": [82501], "3459": [82502], "3460": [82503], "3461": [82504], "3462": [82505], "3463": [82506], "3464": [82507], "3465": [82508], "3466": [82509], "3467": [82510], "3468": [82511], "3469": [82512], "3470": [82513], "3471": [82514], "3472": [82515], "3473": [82516], "3474": [82517], "3475": [82518], "3476": [82519], "3477": [82520], "3478": [9199], "3479": [82521], "3480": [82522], "3481": [82523], "3482": [82524], "3483": [82525], "3484": [82526], "3485": [82527], "3486": [82528], "3487": [82529], "3488": [16121], "3489": [82530], "3490": [82531], "3491": [82532], "3492": [82533], "3493": [82534], "3494": [82535], "3495": [82536], "3496": [82537], "3497": [82538], "3498": [82539], "3499": [82540], "3500": [82541], "3501": [82542], "3502": [82543], "3503": [82544], "3504": [82545], "3505": [82546], "3506": [82547], "3507": [82548], "3508": [82549], "3509": [4920], "3510": [82550], "3511": [82551], "3512": [82552], "3513": [82553], "3514": [82554], "3515": [82555], "3516": [82556], "3517": [82557], "3518": [82558], "3519": [82559], "3520": [82560], "3521": [82561], "3522": [82562], "3523": [82563], "3524": [82564], "3525": [82565], "3526": [82566], "3527": [82567], "3528": [82568], "3529": [82569], "3530": [82570], "3531": [82571], "3532": [82572], "3533": [82573], "3534": [82574], "3535": [82575], "3536": [82576], "3537": [82577], "3538": [82578], "3539": [82579], "3540": [82580], "3541": [82581], "3542": [82582], "3543": [82583], "3544": [82584], "3545": [82585], "3546": [82586], "3547": [82587], "3548": [82588], "3549": [82589], "3550": [82590], "3551": [82591], "3552": [82592], "3553": [82593], "3554": [82594], "3555": [82595], "3556": [82596], "3557": [82597], "3558": [82598], "3559": [82599], "3560": [82600], "3561": [82601], "3562": [82602], "3563": [82603], "3564": [82604], "3565": [82605], "3566": [82606], "3567": [18521], "3568": [82607], "3569": [82608], "3570": [82609], "3571": [28386], "3572": [82610], "3573": [82611], "3574": [82612], "3575": [82613], "3576": [82614], "3577": [82615], "3578": [82616], "3579": [82617], "3580": [82618], "3581": [82619], "3582": [82620], "3583": [82621], "3584": [82622], "3585": [82623], "3586": [82624], "3587": [82625], "3588": [82626], "3589": [82627], "3590": [82628], "3591": [82629], "3592": [82630], "3593": [82631], "3594": [82632], "3595": [82633], "3596": [82634], "3597": [82635], "3598": [82636], "3599": [82637], "3600": [82638], "3601": [82639], "3602": [82640], "3603": [82641], "3604": [82642], "3605": [82643], "3606": [82644], "3607": [82645], "3608": [82646], "3609": [82647], "3610": [82648], "3611": [82649], "3612": [82650], "3613": [82651], "3614": [82652], "3615": [82653], "3616": [82654], "3617": [82655], "3618": [82656], "3619": [82657], "3620": [82658], "3621": [82659], "3622": [82660], "3623": [82661], "3624": [82662], "3625": [82663], "3626": [85943], "3627": [82664], "3628": [82665], "3629": [82666], "3630": [49988], "3631": [82667], "3632": [82668], "3633": [82669], "3634": [82670], "3635": [82671], "3636": [82672], "3637": [82673], "3638": [82674], "3639": [82675], "3640": [82676], "3641": [82677], "3642": [82678], "3643": [82679], "3644": [82680], "3645": [82681], "3646": [82682], "3647": [82683], "3648": [6806], "3649": [82684], "3650": [82685], "3651": [82686], "3652": [1267], "3653": [82687], "3654": [82688], "3655": [82689], "3656": [82690], "3657": [82691], "3658": [82692], "3659": [82693], "3660": [82694], "3661": [82695], "3662": [82696], "3663": [82697], "3664": [82698], "3665": [82699], "3666": [82700], "3667": [82701], "3668": [82702], "3669": [82703], "3670": [1192], "3671": [82704], "3672": [82705], "3673": [82706], "3674": [82707], "3675": [82708], "3676": [82709], "3677": [82710], "3678": [82711], "3679": [82712], "3680": [82713], "3681": [82714], "3682": [82715], "3683": [82716], "3684": [82717], "3685": [82718], "3686": [82719], "3687": [82720], "3688": [82721], "3689": [82722], "3690": [82723], "3691": [82724], "3692": [82725], "3693": [82726], "3694": [82727], "3695": [82728], "3696": [82729], "3697": [82730], "3698": [82731], "3699": [82732], "3700": [82733], "3701": [82734], "3702": [82735], "3703": [82736], "3704": [82737], "3705": [82738], "3706": [82739], "3707": [82740], "3708": [82741], "3709": [82742], "3710": [82743], "3711": [82744], "3712": [82745], "3713": [82746], "3714": [82747], "3715": [82748], "3716": [82749], "3717": [82750], "3718": [82751], "3719": [82752], "3720": [82753], "3721": [82754], "3722": [82755], "3723": [82756], "3724": [82757], "3725": [82758], "3726": [82759], "3727": [82760], "3728": [82761], "3729": [82762], "3730": [82763], "3731": [82764], "3732": [82765], "3733": [82766], "3734": [82767], "3735": [82768], "3736": [82769], "3737": [82770], "3738": [82771], "3739": [82772], "3740": [82773], "3741": [82774], "3742": [82775], "3743": [82776], "3744": [82777], "3745": [82778], "3746": [82779], "3747": [82780], "3748": [82781], "3749": [82782], "3750": [82783], "3751": [82784], "3752": [82785], "3753": [82786], "3754": [82787], "3755": [82788], "3756": [82789], "3757": [82790], "3758": [82791], "3759": [82792], "3760": [82793], "3761": [82794], "3762": [82795], "3763": [82796], "3764": [82797], "3765": [82798], "3766": [82799], "3767": [82800], "3768": [82801], "3769": [82802], "3770": [82803], "3771": [82804], "3772": [82805], "3773": [82806], "3774": [8096], "3775": [82807], "3776": [82808], "3777": [82809], "3778": [82810], "3779": [82811], "3780": [82812], "3781": [82813], "3782": [82814], "3783": [82815], "3784": [82816], "3785": [82817], "3786": [82818], "3787": [82819], "3788": [82820], "3789": [82821], "3790": [82822], "3791": [82823], "3792": [82824], "3793": [82825], "3794": [82826], "3795": [82827], "3796": [82828], "3797": [82829], "3798": [82830], "3799": [82831], "3800": [82832], "3801": [82833], "3802": [82834], "3803": [82835], "3804": [82836], "3805": [82837], "3806": [82838], "3807": [82839], "3808": [82840], "3809": [82841], "3810": [82842], "3811": [82843], "3812": [82844], "3813": [82845], "3814": [82846], "3815": [82847], "3816": [82848], "3817": [82849], "3818": [82850], "3819": [82851], "3820": [82852], "3821": [82853], "3822": [82854], "3823": [82855], "3824": [82856], "3825": [82857], "3826": [82858], "3827": [82859], "3828": [82860], "3829": [82861], "3830": [82862], "3831": [82863], "3832": [82864], "3833": [3603], "3834": [82865], "3835": [82866], "3836": [82867], "3837": [82868], "3838": [82869], "3839": [82870], "3840": [82871], "3841": [1731], "3842": [82872], "3843": [82873], "3844": [82874], "3845": [82875], "3846": [82876], "3847": [82877], "3848": [82878], "3849": [82879], "3850": [82880], "3851": [82881], "3852": [82882], "3853": [82883], "3854": [41109], "3855": [82884], "3856": [82885], "3857": [82886], "3858": [82887], "3859": [82888], "3860": [82889], "3861": [82890], "3862": [82891], "3863": [82892], "3864": [82893], "3865": [82894], "3866": [82895], "3867": [82896], "3868": [82897], "3869": [82898], "3870": [82899], "3871": [82900], "3872": [82901], "3873": [82902], "3874": [82903], "3875": [82904], "3876": [82905], "3877": [82906], "3878": [82907], "3879": [82908], "3880": [82909], "3881": [82910], "3882": [82911], "3883": [82912], "3884": [82913], "3885": [82914], "3886": [82915], "3887": [82916], "3888": [82917], "3889": [82918], "3890": [82919], "3891": [82920], "3892": [82921], "3893": [82922], "3894": [82923], "3895": [82924], "3896": [82925], "3897": [82926], "3898": [82927], "3899": [82928], "3900": [82929], "3901": [82930], "3902": [82931], "3903": [82932], "3904": [82933], "3905": [82934], "3906": [82935], "3907": [82936], "3908": [82937], "3909": [82938], "3910": [82939], "3911": [82940], "3912": [82941], "3913": [82942], "3914": [82943], "3915": [82944], "3916": [82945], "3917": [82946], "3918": [82947], "3919": [82948], "3920": [82949], "3921": [82950], "3922": [82951], "3923": [82952], "3924": [82953], "3925": [82954], "3926": [82955], "3927": [82956], "3928": [82957], "3929": [82958], "3930": [82959], "3931": [82960], "3932": [82961], "3933": [82962], "3934": [82963], "3935": [82964], "3936": [82965], "3937": [82966], "3938": [82967], "3939": [82968], "3940": [82969], "3941": [82970], "3942": [82971], "3943": [82972], "3944": [82973], "3945": [82974], "3946": [82975], "3947": [82976], "3948": [82977], "3949": [82978], "3950": [82979], "3951": [82980], "3952": [82981], "3953": [82982], "3954": [82983], "3955": [82984], "3956": [82985], "3957": [82986], "3958": [82987], "3959": [82988], "3960": [82989], "3961": [82990], "3962": [82991], "3963": [82992], "3964": [82993], "3965": [82994], "3966": [82995], "3967": [82996], "3968": [82997], "3969": [82998], "3970": [82999], "3971": [83000], "3972": [83001], "3973": [83002], "3974": [83003], "3975": [83004], "3976": [83005], "3977": [83006], "3978": [83007], "3979": [83008], "3980": [83009], "3981": [83010], "3982": [83011], "3983": [83012], "3984": [83013], "3985": [83014], "3986": [83015], "3987": [83016], "3988": [83017], "3989": [83018], "3990": [83019], "3991": [83020], "3992": [83021], "3993": [83022], "3994": [83023], "3995": [83024], "3996": [83025], "3997": [83026], "3998": [83027], "3999": [83028], "4000": [83029], "4001": [83030], "4002": [83031], "4003": [83032], "4004": [83033], "4005": [83034], "4006": [83035], "4007": [83036], "4008": [83037], "4009": [83038], "4010": [83039], "4011": [83040], "4012": [83041], "4013": [83042], "4014": [83043], "4015": [83044], "4016": [83045], "4017": [83046], "4018": [83047], "4019": [83048], "4020": [83049], "4021": [83050], "4022": [83051], "4023": [83052], "4024": [83053], "4025": [83054], "4026": [83055], "4027": [83056], "4028": [83057], "4029": [83058], "4030": [83059], "4031": [83060], "4032": [83061], "4033": [83062], "4034": [83063], "4035": [83064], "4036": [83065], "4037": [83066], "4038": [83067], "4039": [83068], "4040": [83069], "4041": [83070], "4042": [83071], "4043": [83072], "4044": [83073], "4045": [83074], "4046": [83075], "4047": [83076], "4048": [83077], "4049": [83078], "4050": [83079], "4051": [83080], "4052": [83081], "4053": [83082], "4054": [83083], "4055": [83084], "4056": [83085], "4057": [83086], "4058": [83087], "4059": [83088], "4060": [83089], "4061": [83090], "4062": [83091], "4063": [83092], "4064": [83093], "4065": [83094], "4066": [83095], "4067": [83096], "4068": [83097], "4069": [83098], "4070": [83099], "4071": [83100], "4072": [83101], "4073": [83102], "4074": [83103], "4075": [83104], "4076": [83105], "4077": [83106], "4078": [83107], "4079": [83108], "4080": [3933], "4081": [83109], "4082": [83110], "4083": [83111], "4084": [83112], "4085": [83113], "4086": [83114], "4087": [83115], "4088": [83116], "4089": [83117], "4090": [83118], "4091": [83119], "4092": [83120], "4093": [83121], "4094": [83122], "4095": [83123], "4096": [83124], "4097": [83125], "4098": [83126], "4099": [83127], "4100": [83128], "4101": [83129], "4102": [83130], "4103": [83131], "4104": [83132], "4105": [83133], "4106": [83134], "4107": [83135], "4108": [83136], "4109": [83137], "4110": [83138], "4111": [83139], "4112": [83140], "4113": [83141], "4114": [83142], "4115": [83143], "4116": [83144], "4117": [83145], "4118": [83146], "4119": [39058], "4120": [83147], "4121": [83148], "4122": [83149], "4123": [83150], "4124": [83151], "4125": [83152], "4126": [83153], "4127": [83154], "4128": [83155], "4129": [83156], "4130": [83157], "4131": [83158], "4132": [83159], "4133": [83160], "4134": [83161], "4135": [83162], "4136": [83163], "4137": [83164], "4138": [83165], "4139": [83166], "4140": [83167], "4141": [83168], "4142": [83169], "4143": [83170], "4144": [83171], "4145": [83172], "4146": [83173], "4147": [83174], "4148": [83175], "4149": [83176], "4150": [83177], "4151": [83178], "4152": [83179], "4153": [83180], "4154": [83181], "4155": [83182], "4156": [83183], "4157": [83184], "4158": [83185], "4159": [83186], "4160": [83187], "4161": [83188], "4162": [83189], "4163": [83190], "4164": [83191], "4165": [83192], "4166": [83193], "4167": [83194], "4168": [83195], "4169": [83196], "4170": [83197], "4171": [83198], "4172": [83199], "4173": [83200], "4174": [83201], "4175": [83202], "4176": [83203], "4177": [83204], "4178": [83205], "4179": [83206], "4180": [83207], "4181": [83208], "4182": [83209], "4183": [83210], "4184": [83211], "4185": [3461], "4186": [83212], "4187": [83213], "4188": [83214], "4189": [83215], "4190": [83216], "4191": [83217], "4192": [83218], "4193": [83219], "4194": [83220], "4195": [83221], "4196": [83222], "4197": [83223], "4198": [83224], "4199": [83225], "4200": [83226], "4201": [83227], "4202": [83228], "4203": [83229], "4204": [83230], "4205": [83231], "4206": [83232], "4207": [83233], "4208": [83234], "4209": [83235], "4210": [83236], "4211": [83237], "4212": [83238], "4213": [83239], "4214": [83240], "4215": [83241], "4216": [83242], "4217": [83243], "4218": [83244], "4219": [83245], "4220": [3461], "4221": [83246], "4222": [83247], "4223": [83248], "4224": [83249], "4225": [83250], "4226": [83251], "4227": [83252], "4228": [83253], "4229": [83254], "4230": [83255], "4231": [83256], "4232": [83257], "4233": [83258], "4234": [83259], "4235": [83260], "4236": [83261], "4237": [83262], "4238": [83263], "4239": [83264], "4240": [83265], "4241": [83266], "4242": [83267], "4243": [83268], "4244": [83269], "4245": [83270], "4246": [83271], "4247": [83787], "4248": [83273], "4249": [83274], "4250": [83275], "4251": [83276], "4252": [83277], "4253": [83278], "4254": [83279], "4255": [83280], "4256": [83281], "4257": [83282], "4258": [83283], "4259": [83284], "4260": [83285], "4261": [83286], "4262": [83287], "4263": [83288], "4264": [83289], "4265": [83290], "4266": [83291], "4267": [83292], "4268": [83293], "4269": [83294], "4270": [83295], "4271": [83296], "4272": [83297], "4273": [83298], "4274": [83299], "4275": [83300], "4276": [83301], "4277": [83302], "4278": [83303], "4279": [83304], "4280": [83305], "4281": [83306], "4282": [83307], "4283": [83308], "4284": [83309], "4285": [83310], "4286": [10255], "4287": [83311], "4288": [83312], "4289": [83313], "4290": [83314], "4291": [83315], "4292": [83316], "4293": [83317], "4294": [83318], "4295": [83319], "4296": [83320], "4297": [83321], "4298": [83322], "4299": [83323], "4300": [83324], "4301": [83325], "4302": [83326], "4303": [83327], "4304": [83328], "4305": [83329], "4306": [83330], "4307": [83331], "4308": [83332], "4309": [83333], "4310": [83334], "4311": [83335], "4312": [83336], "4313": [83337], "4314": [83338], "4315": [83339], "4316": [9549], "4317": [83340], "4318": [83341], "4319": [83342], "4320": [83343], "4321": [83344], "4322": [83345], "4323": [83346], "4324": [83347], "4325": [83348], "4326": [83349], "4327": [83350], "4328": [83351], "4329": [83352], "4330": [83353], "4331": [83354], "4332": [83355], "4333": [83356], "4334": [83357], "4335": [83358], "4336": [83359], "4337": [83360], "4338": [83361], "4339": [83362], "4340": [83363], "4341": [83364], "4342": [83365], "4343": [83366], "4344": [83367], "4345": [83368], "4346": [83369], "4347": [83370], "4348": [83371], "4349": [83372], "4350": [83373], "4351": [83374], "4352": [2584], "4353": [83375], "4354": [83376], "4355": [83377], "4356": [83378], "4357": [83379], "4358": [83380], "4359": [83381], "4360": [83382], "4361": [83383], "4362": [83384], "4363": [83385], "4364": [83386], "4365": [83387], "4366": [83388], "4367": [83389], "4368": [40348], "4369": [83390], "4370": [83391], "4371": [83392], "4372": [83393], "4373": [83394], "4374": [83395], "4375": [23198], "4376": [83396], "4377": [83397], "4378": [83398], "4379": [83399], "4380": [83400], "4381": [83401], "4382": [83402], "4383": [83403], "4384": [83404], "4385": [83405], "4386": [83406], "4387": [83407], "4388": [83408], "4389": [83409], "4390": [83410], "4391": [83411], "4392": [83412], "4393": [83413], "4394": [83414], "4395": [83415], "4396": [83416], "4397": [83417], "4398": [83418], "4399": [83419], "4400": [83420], "4401": [83421], "4402": [83422], "4403": [83423], "4404": [83424], "4405": [83425], "4406": [83426], "4407": [83427], "4408": [83428], "4409": [83429], "4410": [83430], "4411": [83431], "4412": [83432], "4413": [83433], "4414": [83434], "4415": [83435], "4416": [6354], "4417": [83436], "4418": [83437], "4419": [83438], "4420": [83439], "4421": [83440], "4422": [83441], "4423": [83442], "4424": [83443], "4425": [83444], "4426": [83445], "4427": [83446], "4428": [83447], "4429": [83448], "4430": [83449], "4431": [83450], "4432": [83451], "4433": [83452], "4434": [83453], "4435": [83454], "4436": [83455], "4437": [83456], "4438": [83457], "4439": [83458], "4440": [83459], "4441": [83460], "4442": [83461], "4443": [3067], "4444": [83462], "4445": [83463], "4446": [83464], "4447": [83465], "4448": [83466], "4449": [83467], "4450": [83468], "4451": [83469], "4452": [83470], "4453": [83471], "4454": [83472], "4455": [83473], "4456": [83474], "4457": [83475], "4458": [83476], "4459": [83477], "4460": [83478], "4461": [83479], "4462": [83480], "4463": [83481], "4464": [83482], "4465": [83483], "4466": [83484], "4467": [83485], "4468": [83486], "4469": [83487], "4470": [83488], "4471": [83489], "4472": [83490], "4473": [44127], "4474": [83491], "4475": [83492], "4476": [83493], "4477": [83494], "4478": [83495], "4479": [83496], "4480": [83497], "4481": [83498], "4482": [83499], "4483": [83500], "4484": [83501], "4485": [83502], "4486": [83503], "4487": [83504], "4488": [83505], "4489": [83506], "4490": [83507], "4491": [84815], "4492": [83509], "4493": [83510], "4494": [83511], "4495": [83512], "4496": [83513], "4497": [83514], "4498": [83515], "4499": [83516], "4500": [83517], "4501": [83518], "4502": [10255], "4503": [54373], "4504": [83519], "4505": [83520], "4506": [83521], "4507": [83522], "4508": [83523], "4509": [83524], "4510": [83525], "4511": [83526], "4512": [83527], "4513": [83528], "4514": [83529], "4515": [83530], "4516": [83531], "4517": [83532], "4518": [83533], "4519": [83534], "4520": [83535], "4521": [83536], "4522": [83537], "4523": [83538], "4524": [83539], "4525": [83540], "4526": [83541], "4527": [83542], "4528": [83543], "4529": [83544], "4530": [83545], "4531": [83546], "4532": [83547], "4533": [83548], "4534": [83549], "4535": [83550], "4536": [83551], "4537": [83552], "4538": [83553], "4539": [83554], "4540": [83555], "4541": [83556], "4542": [83557], "4543": [83558], "4544": [83559], "4545": [83560], "4546": [83561], "4547": [83562], "4548": [83563], "4549": [83564], "4550": [83565], "4551": [83566], "4552": [83567], "4553": [83568], "4554": [83569], "4555": [83570], "4556": [83571], "4557": [83572], "4558": [83573], "4559": [83574], "4560": [83575], "4561": [83576], "4562": [83577], "4563": [83578], "4564": [83579], "4565": [83580], "4566": [83581], "4567": [83582], "4568": [83583], "4569": [83584], "4570": [83585], "4571": [83586], "4572": [83587], "4573": [83588], "4574": [83589], "4575": [83590], "4576": [83591], "4577": [83592], "4578": [83593], "4579": [83594], "4580": [83595], "4581": [83596], "4582": [83597], "4583": [83598], "4584": [83599], "4585": [83600], "4586": [83601], "4587": [83602], "4588": [9969], "4589": [83603], "4590": [83604], "4591": [83605], "4592": [83606], "4593": [83607], "4594": [83608], "4595": [83609], "4596": [83610], "4597": [83611], "4598": [83612], "4599": [83613], "4600": [83614], "4601": [83615], "4602": [83616], "4603": [83617], "4604": [83618], "4605": [83619], "4606": [83620], "4607": [83621], "4608": [83622], "4609": [11887], "4610": [83623], "4611": [83624], "4612": [83625], "4613": [83626], "4614": [83627], "4615": [83628], "4616": [83629], "4617": [83630], "4618": [83631], "4619": [83632], "4620": [83633], "4621": [83634], "4622": [83635], "4623": [83636], "4624": [83637], "4625": [83638], "4626": [83639], "4627": [83640], "4628": [83641], "4629": [83642], "4630": [83643], "4631": [83644], "4632": [83645], "4633": [83646], "4634": [83647], "4635": [83648], "4636": [83649], "4637": [83650], "4638": [83651], "4639": [83652], "4640": [83653], "4641": [8259], "4642": [83654], "4643": [83655], "4644": [83656], "4645": [83657], "4646": [83658], "4647": [83659], "4648": [83660], "4649": [83661], "4650": [83662], "4651": [83663], "4652": [83664], "4653": [83665], "4654": [83666], "4655": [83667], "4656": [83668], "4657": [83669], "4658": [83670], "4659": [83671], "4660": [10255], "4661": [83672], "4662": [83673], "4663": [83674], "4664": [83675], "4665": [83676], "4666": [31623], "4667": [83677], "4668": [83678], "4669": [83679], "4670": [83680], "4671": [83681], "4672": [83682], "4673": [83683], "4674": [83684], "4675": [83685], "4676": [83686], "4677": [83687], "4678": [83688], "4679": [83689], "4680": [83690], "4681": [83691], "4682": [83692], "4683": [83693], "4684": [83694], "4685": [83695], "4686": [83696], "4687": [83697], "4688": [83698], "4689": [83699], "4690": [83700], "4691": [83701], "4692": [6486], "4693": [83702], "4694": [83703], "4695": [83704], "4696": [83705], "4697": [1900], "4698": [83706], "4699": [83707], "4700": [83708], "4701": [83709], "4702": [83710], "4703": [83711], "4704": [6196], "4705": [83712], "4706": [83713], "4707": [83714], "4708": [83715], "4709": [83888], "4710": [83717], "4711": [37837], "4712": [83718], "4713": [83719], "4714": [83720], "4715": [83721], "4716": [83722], "4717": [83723], "4718": [83724], "4719": [83725], "4720": [83726], "4721": [83727], "4722": [83728], "4723": [83729], "4724": [83730], "4725": [83731], "4726": [83732], "4727": [83733], "4728": [83734], "4729": [83735], "4730": [83736], "4731": [83737], "4732": [83738], "4733": [83739], "4734": [83740], "4735": [83741], "4736": [83742], "4737": [83743], "4738": [83744], "4739": [83745], "4740": [83746], "4741": [83747], "4742": [83748], "4743": [83749], "4744": [83750], "4745": [83751], "4746": [83752], "4747": [83753], "4748": [83754], "4749": [83755], "4750": [83756], "4751": [83757], "4752": [83758], "4753": [83759], "4754": [83760], "4755": [83761], "4756": [83762], "4757": [83763], "4758": [83764], "4759": [83765], "4760": [83766], "4761": [2879], "4762": [83767], "4763": [83768], "4764": [83769], "4765": [83770], "4766": [83771], "4767": [83772], "4768": [83773], "4769": [83774], "4770": [83775], "4771": [83776], "4772": [83777], "4773": [83778], "4774": [83779], "4775": [83780], "4776": [83781], "4777": [83782], "4778": [83783], "4779": [83784], "4780": [83785], "4781": [83786], "4782": [83788], "4783": [83789], "4784": [83790], "4785": [83791], "4786": [83792], "4787": [83793], "4788": [83794], "4789": [83795], "4790": [83796], "4791": [83797], "4792": [83798], "4793": [83799], "4794": [83800], "4795": [49968], "4796": [83801], "4797": [83802], "4798": [83803], "4799": [83804], "4800": [83805], "4801": [83806], "4802": [83807], "4803": [83808], "4804": [83809], "4805": [83810], "4806": [83811], "4807": [83812], "4808": [2927], "4809": [83813], "4810": [83814], "4811": [83815], "4812": [83816], "4813": [83817], "4814": [83818], "4815": [83819], "4816": [83820], "4817": [83821], "4818": [83822], "4819": [83823], "4820": [83824], "4821": [83825], "4822": [83826], "4823": [83827], "4824": [83828], "4825": [83829], "4826": [83830], "4827": [83831], "4828": [83832], "4829": [83833], "4830": [83834], "4831": [83835], "4832": [83836], "4833": [83837], "4834": [83838], "4835": [83839], "4836": [83840], "4837": [83841], "4838": [83842], "4839": [83843], "4840": [83844], "4841": [51015], "4842": [83845], "4843": [83846], "4844": [83847], "4845": [83848], "4846": [83849], "4847": [83850], "4848": [83851], "4849": [83852], "4850": [83853], "4851": [83854], "4852": [83855], "4853": [83856], "4854": [83857], "4855": [83858], "4856": [83859], "4857": [83860], "4858": [83861], "4859": [83862], "4860": [83863], "4861": [83864], "4862": [83865], "4863": [83866], "4864": [83867], "4865": [83868], "4866": [83869], "4867": [83870], "4868": [83871], "4869": [83872], "4870": [83873], "4871": [83874], "4872": [83875], "4873": [83876], "4874": [83877], "4875": [83878], "4876": [83879], "4877": [83880], "4878": [83881], "4879": [83882], "4880": [83883], "4881": [83884], "4882": [83885], "4883": [83886], "4884": [83887], "4885": [83889], "4886": [83890], "4887": [83891], "4888": [83892], "4889": [83893], "4890": [83894], "4891": [83895], "4892": [83896], "4893": [83897], "4894": [83898], "4895": [83899], "4896": [83900], "4897": [83901], "4898": [83902], "4899": [83903], "4900": [83904], "4901": [83905], "4902": [83906], "4903": [83907], "4904": [83908], "4905": [83909], "4906": [83910], "4907": [83911], "4908": [83912], "4909": [83913], "4910": [83914], "4911": [83915], "4912": [83916], "4913": [16977], "4914": [83917], "4915": [83918], "4916": [83919], "4917": [29986], "4918": [83920], "4919": [83921], "4920": [83922], "4921": [83923], "4922": [83924], "4923": [83925], "4924": [83926], "4925": [46625], "4926": [83927], "4927": [83928], "4928": [83929], "4929": [83930], "4930": [83931], "4931": [83932], "4932": [83933], "4933": [83934], "4934": [83935], "4935": [83936], "4936": [83937], "4937": [83938], "4938": [83939], "4939": [83940], "4940": [83941], "4941": [83942], "4942": [83943], "4943": [83944], "4944": [83945], "4945": [83946], "4946": [83947], "4947": [83948], "4948": [83949], "4949": [83950], "4950": [83951], "4951": [83952], "4952": [83953], "4953": [83954], "4954": [83955], "4955": [83956], "4956": [83957], "4957": [83958], "4958": [83959], "4959": [83960], "4960": [83961], "4961": [83962], "4962": [83963], "4963": [83964], "4964": [1365], "4965": [83965], "4966": [83966], "4967": [83967], "4968": [83968], "4969": [83969], "4970": [83970], "4971": [83971], "4972": [83972], "4973": [83973], "4974": [83974], "4975": [83975], "4976": [83976], "4977": [83977], "4978": [83978], "4979": [83979], "4980": [83980], "4981": [83981], "4982": [83982], "4983": [83983], "4984": [83984], "4985": [83985], "4986": [83986], "4987": [83987], "4988": [83988], "4989": [83989], "4990": [83990], "4991": [83991], "4992": [83992], "4993": [83993], "4994": [83994], "4995": [83995], "4996": [83996], "4997": [83997], "4998": [83998], "4999": [83999], "5000": [84000], "5001": [84001], "5002": [84002], "5003": [84003], "5004": [84004], "5005": [84005], "5006": [84006], "5007": [84007], "5008": [84008], "5009": [84009], "5010": [84010], "5011": [84011], "5012": [84012], "5013": [84013], "5014": [84014], "5015": [84015], "5016": [84016], "5017": [88646], "5018": [84018], "5019": [84019], "5020": [84020], "5021": [84021], "5022": [84022], "5023": [84023], "5024": [84024], "5025": [35877], "5026": [84025], "5027": [84026], "5028": [84027], "5029": [84028], "5030": [84029], "5031": [84030], "5032": [84031], "5033": [84032], "5034": [84033], "5035": [84034], "5036": [84035], "5037": [84036], "5038": [84037], "5039": [84038], "5040": [84039], "5041": [84040], "5042": [84041], "5043": [84042], "5044": [84043], "5045": [84044], "5046": [84045], "5047": [84046], "5048": [84047], "5049": [84048], "5050": [84049], "5051": [84050], "5052": [84051], "5053": [84052], "5054": [84053], "5055": [84054], "5056": [84055], "5057": [84056], "5058": [84057], "5059": [84058], "5060": [84059], "5061": [84060], "5062": [84061], "5063": [84062], "5064": [84063], "5065": [84064], "5066": [84065], "5067": [84066], "5068": [84067], "5069": [84068], "5070": [84069], "5071": [84070], "5072": [84071], "5073": [84072], "5074": [84073], "5075": [84074], "5076": [84075], "5077": [84076], "5078": [84077], "5079": [84078], "5080": [84079], "5081": [84080], "5082": [84081], "5083": [84082], "5084": [84083], "5085": [84084], "5086": [84085], "5087": [84086], "5088": [84087], "5089": [84088], "5090": [84089], "5091": [84090], "5092": [84091], "5093": [84092], "5094": [84093], "5095": [84094], "5096": [84095], "5097": [84096], "5098": [84097], "5099": [84098], "5100": [84099], "5101": [84100], "5102": [84101], "5103": [84102], "5104": [84103], "5105": [84104], "5106": [84105], "5107": [84106], "5108": [84107], "5109": [84108], "5110": [84109], "5111": [84110], "5112": [84111], "5113": [84112], "5114": [414], "5115": [84113], "5116": [84114], "5117": [84115], "5118": [84116], "5119": [84117], "5120": [84118], "5121": [84119], "5122": [84120], "5123": [84121], "5124": [84122], "5125": [84123], "5126": [84124], "5127": [84125], "5128": [84126], "5129": [84127], "5130": [84128], "5131": [84129], "5132": [84130], "5133": [84131], "5134": [84132], "5135": [84133], "5136": [84134], "5137": [84135], "5138": [84136], "5139": [84137], "5140": [84138], "5141": [84139], "5142": [84140], "5143": [84141], "5144": [84142], "5145": [84143], "5146": [84144], "5147": [84145], "5148": [84146], "5149": [84147], "5150": [84148], "5151": [84149], "5152": [84150], "5153": [84151], "5154": [84152], "5155": [84153], "5156": [84154], "5157": [84155], "5158": [84156], "5159": [84157], "5160": [84158], "5161": [84159], "5162": [84160], "5163": [84161], "5164": [84162], "5165": [84163], "5166": [84164], "5167": [84165], "5168": [84166], "5169": [84167], "5170": [84168], "5171": [84169], "5172": [84170], "5173": [84171], "5174": [84172], "5175": [84173], "5176": [84174], "5177": [84175], "5178": [84176], "5179": [84177], "5180": [84178], "5181": [84179], "5182": [84180], "5183": [84181], "5184": [84182], "5185": [84183], "5186": [84184], "5187": [84185], "5188": [84186], "5189": [84187], "5190": [84188], "5191": [84189], "5192": [84190], "5193": [84191], "5194": [84192], "5195": [84193], "5196": [84194], "5197": [84195], "5198": [84196], "5199": [84197], "5200": [84198], "5201": [84199], "5202": [84200], "5203": [84201], "5204": [84202], "5205": [84203], "5206": [84204], "5207": [84205], "5208": [84206], "5209": [84207], "5210": [84208], "5211": [84209], "5212": [84210], "5213": [84211], "5214": [84212], "5215": [84213], "5216": [84214], "5217": [84215], "5218": [84216], "5219": [84217], "5220": [84218], "5221": [84219], "5222": [84220], "5223": [84221], "5224": [84222], "5225": [84223], "5226": [84224], "5227": [84225], "5228": [84226], "5229": [84227], "5230": [84228], "5231": [84229], "5232": [84230], "5233": [84231], "5234": [84232], "5235": [84233], "5236": [84234], "5237": [84235], "5238": [84236], "5239": [84237], "5240": [84238], "5241": [84239], "5242": [84240], "5243": [84241], "5244": [84242], "5245": [84243], "5246": [3619], "5247": [84244], "5248": [84245], "5249": [84246], "5250": [84247], "5251": [84248], "5252": [84249], "5253": [84250], "5254": [84251], "5255": [84252], "5256": [84253], "5257": [84254], "5258": [84255], "5259": [84256], "5260": [84257], "5261": [84258], "5262": [84259], "5263": [84260], "5264": [84261], "5265": [84262], "5266": [84263], "5267": [84264], "5268": [84265], "5269": [84266], "5270": [84267], "5271": [84268], "5272": [84269], "5273": [84270], "5274": [84271], "5275": [84272], "5276": [84273], "5277": [84274], "5278": [84275], "5279": [84276], "5280": [84277], "5281": [84278], "5282": [84279], "5283": [84280], "5284": [84281], "5285": [84282], "5286": [84283], "5287": [84284], "5288": [84285], "5289": [84286], "5290": [84287], "5291": [84288], "5292": [84289], "5293": [84290], "5294": [84291], "5295": [84292], "5296": [84293], "5297": [84294], "5298": [84295], "5299": [84296], "5300": [84297], "5301": [84298], "5302": [84299], "5303": [84300], "5304": [84301], "5305": [84302], "5306": [84303], "5307": [84304], "5308": [84305], "5309": [84306], "5310": [84307], "5311": [84308], "5312": [84309], "5313": [84310], "5314": [84311], "5315": [84312], "5316": [84313], "5317": [84314], "5318": [84315], "5319": [84316], "5320": [18983], "5321": [84317], "5322": [84318], "5323": [84319], "5324": [84320], "5325": [84321], "5326": [84322], "5327": [84323], "5328": [84324], "5329": [84325], "5330": [84326], "5331": [84327], "5332": [84328], "5333": [84329], "5334": [84330], "5335": [84331], "5336": [84332], "5337": [84333], "5338": [84334], "5339": [84335], "5340": [84336], "5341": [84337], "5342": [84338], "5343": [84339], "5344": [7846], "5345": [84340], "5346": [84341], "5347": [84342], "5348": [84343], "5349": [84344], "5350": [84345], "5351": [84346], "5352": [84347], "5353": [84348], "5354": [84349], "5355": [1731], "5356": [84350], "5357": [84351], "5358": [84352], "5359": [84353], "5360": [84354], "5361": [84355], "5362": [84356], "5363": [84357], "5364": [84358], "5365": [84359], "5366": [84360], "5367": [84361], "5368": [84362], "5369": [84363], "5370": [84364], "5371": [84365], "5372": [84366], "5373": [35877], "5374": [84367], "5375": [84368], "5376": [84369], "5377": [84370], "5378": [84371], "5379": [84372], "5380": [84373], "5381": [84374], "5382": [84375], "5383": [84376], "5384": [84377], "5385": [84378], "5386": [84379], "5387": [84380], "5388": [84381], "5389": [84382], "5390": [84383], "5391": [84384], "5392": [84385], "5393": [84386], "5394": [84387], "5395": [84388], "5396": [84389], "5397": [84390], "5398": [84391], "5399": [84392], "5400": [84393], "5401": [84394], "5402": [84395], "5403": [84396], "5404": [84397], "5405": [84398], "5406": [84399], "5407": [84400], "5408": [84401], "5409": [84402], "5410": [84403], "5411": [84404], "5412": [84405], "5413": [84406], "5414": [84407], "5415": [84408], "5416": [84409], "5417": [84410], "5418": [84411], "5419": [84412], "5420": [84413], "5421": [84414], "5422": [85208], "5423": [84416], "5424": [84417], "5425": [84418], "5426": [84419], "5427": [84420], "5428": [84421], "5429": [84422], "5430": [84423], "5431": [84424], "5432": [84425], "5433": [84426], "5434": [84427], "5435": [84428], "5436": [84429], "5437": [84430], "5438": [84431], "5439": [84432], "5440": [84433], "5441": [84434], "5442": [84435], "5443": [84436], "5444": [84437], "5445": [84438], "5446": [84439], "5447": [84440], "5448": [84441], "5449": [84442], "5450": [84443], "5451": [84444], "5452": [84445], "5453": [84446], "5454": [84447], "5455": [84448], "5456": [84449], "5457": [84450], "5458": [84451], "5459": [84452], "5460": [84453], "5461": [84454], "5462": [84455], "5463": [84456], "5464": [84457], "5465": [84458], "5466": [84459], "5467": [84460], "5468": [84461], "5469": [84462], "5470": [84463], "5471": [84464], "5472": [84465], "5473": [84466], "5474": [84467], "5475": [84468], "5476": [84469], "5477": [84470], "5478": [29241], "5479": [84471], "5480": [84472], "5481": [84473], "5482": [84474], "5483": [84475], "5484": [84476], "5485": [84477], "5486": [84478], "5487": [84479], "5488": [84480], "5489": [84481], "5490": [84482], "5491": [84483], "5492": [84484], "5493": [20901], "5494": [84485], "5495": [84486], "5496": [84487], "5497": [84488], "5498": [84489], "5499": [84490], "5500": [84491], "5501": [84492], "5502": [84493], "5503": [84494], "5504": [84495], "5505": [84496], "5506": [84497], "5507": [84498], "5508": [84499], "5509": [84500], "5510": [84501], "5511": [84502], "5512": [84503], "5513": [84504], "5514": [22598], "5515": [84505], "5516": [84506], "5517": [84507], "5518": [84508], "5519": [84509], "5520": [84510], "5521": [84511], "5522": [84512], "5523": [84513], "5524": [84514], "5525": [84515], "5526": [84516], "5527": [84517], "5528": [84518], "5529": [84519], "5530": [84520], "5531": [84521], "5532": [84522], "5533": [84523], "5534": [84524], "5535": [84525], "5536": [84526], "5537": [84527], "5538": [84528], "5539": [84529], "5540": [84530], "5541": [84531], "5542": [84532], "5543": [84533], "5544": [84534], "5545": [84535], "5546": [8274], "5547": [84536], "5548": [84537], "5549": [84538], "5550": [84539], "5551": [84540], "5552": [84541], "5553": [84542], "5554": [84543], "5555": [84544], "5556": [84545], "5557": [84546], "5558": [84547], "5559": [84548], "5560": [84549], "5561": [84550], "5562": [84551], "5563": [84552], "5564": [84553], "5565": [84554], "5566": [84555], "5567": [84556], "5568": [84557], "5569": [84558], "5570": [84560], "5571": [84561], "5572": [84562], "5573": [84563], "5574": [84564], "5575": [84565], "5576": [84566], "5577": [84567], "5578": [84568], "5579": [84569], "5580": [84570], "5581": [84571], "5582": [84572], "5583": [84573], "5584": [84574], "5585": [84575], "5586": [84576], "5587": [84577], "5588": [84578], "5589": [84579], "5590": [84580], "5591": [84581], "5592": [84582], "5593": [84583], "5594": [84584], "5595": [84585], "5596": [84586], "5597": [84587], "5598": [84588], "5599": [6345], "5600": [84589], "5601": [84590], "5602": [84591], "5603": [84592], "5604": [84593], "5605": [84594], "5606": [84595], "5607": [84596], "5608": [84597], "5609": [84598], "5610": [84599], "5611": [84600], "5612": [84601], "5613": [84602], "5614": [84603], "5615": [84604], "5616": [84605], "5617": [84606], "5618": [84607], "5619": [84608], "5620": [84609], "5621": [84610], "5622": [84611], "5623": [84612], "5624": [84613], "5625": [84614], "5626": [84615], "5627": [84616], "5628": [84617], "5629": [84618], "5630": [84619], "5631": [84620], "5632": [84621], "5633": [84622], "5634": [84623], "5635": [84624], "5636": [84625], "5637": [84626], "5638": [84627], "5639": [84628], "5640": [84629], "5641": [84630], "5642": [84631], "5643": [84632], "5644": [84633], "5645": [84634], "5646": [84635], "5647": [84636], "5648": [84637], "5649": [84638], "5650": [84639], "5651": [84640], "5652": [84641], "5653": [84642], "5654": [84643], "5655": [84644], "5656": [84645], "5657": [84646], "5658": [84647], "5659": [84648], "5660": [84649], "5661": [84650], "5662": [84651], "5663": [84652], "5664": [84653], "5665": [84654], "5666": [84655], "5667": [84656], "5668": [84657], "5669": [84658], "5670": [84659], "5671": [84660], "5672": [84661], "5673": [84662], "5674": [84663], "5675": [84664], "5676": [84665], "5677": [39731], "5678": [84666], "5679": [84667], "5680": [84668], "5681": [42132], "5682": [84669], "5683": [84670], "5684": [84671], "5685": [84672], "5686": [84673], "5687": [84674], "5688": [84675], "5689": [84676], "5690": [84677], "5691": [84678], "5692": [84679], "5693": [84680], "5694": [84681], "5695": [84682], "5696": [84683], "5697": [84684], "5698": [84685], "5699": [84686], "5700": [84687], "5701": [84688], "5702": [84689], "5703": [84690], "5704": [84691], "5705": [84692], "5706": [84693], "5707": [84694], "5708": [84695], "5709": [84696], "5710": [84697], "5711": [84698], "5712": [84699], "5713": [84700], "5714": [84701], "5715": [84702], "5716": [84703], "5717": [84704], "5718": [84705], "5719": [84706], "5720": [84707], "5721": [84708], "5722": [84709], "5723": [84710], "5724": [84711], "5725": [84712], "5726": [84713], "5727": [84714], "5728": [84715], "5729": [84716], "5730": [84717], "5731": [84718], "5732": [84719], "5733": [84720], "5734": [84721], "5735": [84722], "5736": [84723], "5737": [84724], "5738": [84725], "5739": [84726], "5740": [84727], "5741": [84728], "5742": [84729], "5743": [84730], "5744": [84731], "5745": [84732], "5746": [84733], "5747": [84734], "5748": [84735], "5749": [84736], "5750": [84737], "5751": [84738], "5752": [84739], "5753": [84740], "5754": [84741], "5755": [84742], "5756": [84743], "5757": [84744], "5758": [84745], "5759": [32338], "5760": [84746], "5761": [84747], "5762": [84748], "5763": [84749], "5764": [84750], "5765": [84751], "5766": [84752], "5767": [84753], "5768": [84754], "5769": [84755], "5770": [84756], "5771": [84757], "5772": [84758], "5773": [84759], "5774": [7911], "5775": [84760], "5776": [84761], "5777": [84762], "5778": [84763], "5779": [84764], "5780": [84765], "5781": [84766], "5782": [84767], "5783": [84768], "5784": [84769], "5785": [84770], "5786": [84771], "5787": [84772], "5788": [84773], "5789": [84774], "5790": [84775], "5791": [84776], "5792": [84777], "5793": [84778], "5794": [84779], "5795": [84780], "5796": [84781], "5797": [84782], "5798": [84783], "5799": [84785], "5800": [84786], "5801": [84787], "5802": [84788], "5803": [84789], "5804": [84790], "5805": [84791], "5806": [84792], "5807": [84793], "5808": [84794], "5809": [84795], "5810": [84796], "5811": [84797], "5812": [84798], "5813": [84799], "5814": [84800], "5815": [84801], "5816": [84802], "5817": [84803], "5818": [84804], "5819": [84805], "5820": [84806], "5821": [84807], "5822": [84808], "5823": [84809], "5824": [84810], "5825": [84811], "5826": [84812], "5827": [84813], "5828": [84814], "5829": [84816], "5830": [84817], "5831": [84818], "5832": [84819], "5833": [84820], "5834": [84821], "5835": [84822], "5836": [84823], "5837": [84824], "5838": [84825], "5839": [84826], "5840": [84827], "5841": [84828], "5842": [84829], "5843": [84830], "5844": [84831], "5845": [84832], "5846": [84833], "5847": [84834], "5848": [84835], "5849": [84836], "5850": [84837], "5851": [84838], "5852": [84839], "5853": [84840], "5854": [84841], "5855": [84842], "5856": [84843], "5857": [84844], "5858": [84845], "5859": [84846], "5860": [84847], "5861": [84848], "5862": [84849], "5863": [84850], "5864": [84851], "5865": [84852], "5866": [84853], "5867": [84854], "5868": [84855], "5869": [84856], "5870": [84857], "5871": [84858], "5872": [84859], "5873": [84860], "5874": [84861], "5875": [84862], "5876": [84863], "5877": [84864], "5878": [84865], "5879": [84866], "5880": [84867], "5881": [84868], "5882": [84869], "5883": [84870], "5884": [84871], "5885": [84872], "5886": [84873], "5887": [84874], "5888": [84875], "5889": [84876], "5890": [84877], "5891": [84878], "5892": [84879], "5893": [84880], "5894": [84881], "5895": [84882], "5896": [84883], "5897": [84884], "5898": [84885], "5899": [84886], "5900": [84887], "5901": [9179], "5902": [84888], "5903": [84889], "5904": [84890], "5905": [84891], "5906": [84892], "5907": [84893], "5908": [84894], "5909": [84895], "5910": [84896], "5911": [84897], "5912": [84898], "5913": [84899], "5914": [27566], "5915": [84900], "5916": [84901], "5917": [84902], "5918": [84903], "5919": [84904], "5920": [2255], "5921": [84905], "5922": [84906], "5923": [84907], "5924": [84908], "5925": [84909], "5926": [84910], "5927": [84911], "5928": [84912], "5929": [84913], "5930": [84914], "5931": [84915], "5932": [84916], "5933": [84917], "5934": [84918], "5935": [84919], "5936": [84920], "5937": [84921], "5938": [84922], "5939": [84923], "5940": [84924], "5941": [84925], "5942": [84926], "5943": [84927], "5944": [84928], "5945": [84929], "5946": [84930], "5947": [84931], "5948": [84932], "5949": [84933], "5950": [84934], "5951": [84935], "5952": [84936], "5953": [84937], "5954": [84938], "5955": [4963], "5956": [84939], "5957": [84940], "5958": [84941], "5959": [84942], "5960": [84943], "5961": [84944], "5962": [84945], "5963": [84946], "5964": [84947], "5965": [84948], "5966": [84949], "5967": [84950], "5968": [84951], "5969": [84952], "5970": [84953], "5971": [84954], "5972": [84955], "5973": [84956], "5974": [84957], "5975": [84958], "5976": [84959], "5977": [84960], "5978": [84961], "5979": [84962], "5980": [84963], "5981": [84964], "5982": [84965], "5983": [84966], "5984": [84967], "5985": [84968], "5986": [84969], "5987": [84970], "5988": [84971], "5989": [84972], "5990": [84973], "5991": [84974], "5992": [84975], "5993": [84976], "5994": [84977], "5995": [84978], "5996": [84979], "5997": [84980], "5998": [84981], "5999": [84982], "6000": [84983], "6001": [84984], "6002": [84985], "6003": [84986], "6004": [84987], "6005": [84988], "6006": [84989], "6007": [84990], "6008": [84991], "6009": [84992], "6010": [84993], "6011": [237], "6012": [84994], "6013": [84995], "6014": [84996], "6015": [84997], "6016": [84998], "6017": [84999], "6018": [85000], "6019": [85001], "6020": [85002], "6021": [85003], "6022": [85004], "6023": [85005], "6024": [85006], "6025": [85007], "6026": [85008], "6027": [85009], "6028": [85010], "6029": [85011], "6030": [85012], "6031": [85013], "6032": [85014], "6033": [85015], "6034": [85016], "6035": [85017], "6036": [85018], "6037": [85019], "6038": [85020], "6039": [85021], "6040": [85022], "6041": [85023], "6042": [85024], "6043": [85025], "6044": [85026], "6045": [85027], "6046": [85028], "6047": [85029], "6048": [85030], "6049": [85031], "6050": [85032], "6051": [85033], "6052": [52195], "6053": [85034], "6054": [85035], "6055": [85036], "6056": [85037], "6057": [85038], "6058": [85039], "6059": [85040], "6060": [85041], "6061": [85042], "6062": [85044], "6063": [85045], "6064": [85046], "6065": [85047], "6066": [85048], "6067": [85049], "6068": [85050], "6069": [85051], "6070": [85052], "6071": [85053], "6072": [85054], "6073": [85055], "6074": [85056], "6075": [85057], "6076": [85058], "6077": [85059], "6078": [85060], "6079": [85061], "6080": [85062], "6081": [85063], "6082": [85064], "6083": [85065], "6084": [85066], "6085": [85067], "6086": [85068], "6087": [85069], "6088": [85070], "6089": [85071], "6090": [85072], "6091": [85073], "6092": [85074], "6093": [85075], "6094": [85076], "6095": [85077], "6096": [85078], "6097": [85079], "6098": [85080], "6099": [85081], "6100": [85082], "6101": [85083], "6102": [85084], "6103": [21824], "6104": [85085], "6105": [85086], "6106": [85087], "6107": [85088], "6108": [85089], "6109": [85090], "6110": [85091], "6111": [85092], "6112": [85093], "6113": [85094], "6114": [85095], "6115": [85096], "6116": [85097], "6117": [85098], "6118": [85099], "6119": [85100], "6120": [85101], "6121": [85102], "6122": [85103], "6123": [10097], "6124": [85104], "6125": [85105], "6126": [85106], "6127": [85107], "6128": [85108], "6129": [85109], "6130": [85110], "6131": [85111], "6132": [85112], "6133": [85113], "6134": [85114], "6135": [85115], "6136": [85116], "6137": [85117], "6138": [85118], "6139": [85119], "6140": [85120], "6141": [85121], "6142": [85122], "6143": [85123], "6144": [85124], "6145": [85125], "6146": [85126], "6147": [85127], "6148": [85128], "6149": [85129], "6150": [85130], "6151": [85131], "6152": [85132], "6153": [85133], "6154": [54373], "6155": [85134], "6156": [85135], "6157": [85136], "6158": [85137], "6159": [85138], "6160": [39903], "6161": [85139], "6162": [85140], "6163": [85141], "6164": [85142], "6165": [85143], "6166": [85144], "6167": [85145], "6168": [85146], "6169": [85147], "6170": [85148], "6171": [85149], "6172": [85150], "6173": [85151], "6174": [85152], "6175": [85153], "6176": [85154], "6177": [85155], "6178": [85156], "6179": [85157], "6180": [85158], "6181": [85159], "6182": [85160], "6183": [85161], "6184": [85162], "6185": [85163], "6186": [85164], "6187": [85165], "6188": [85166], "6189": [85167], "6190": [85168], "6191": [85169], "6192": [85170], "6193": [85171], "6194": [85172], "6195": [85173], "6196": [85174], "6197": [85175], "6198": [85176], "6199": [85177], "6200": [85178], "6201": [85179], "6202": [85180], "6203": [85181], "6204": [85182], "6205": [85183], "6206": [85184], "6207": [85185], "6208": [85186], "6209": [85187], "6210": [85188], "6211": [85189], "6212": [85190], "6213": [85191], "6214": [85192], "6215": [85193], "6216": [44293], "6217": [85194], "6218": [85195], "6219": [85196], "6220": [85197], "6221": [85198], "6222": [85199], "6223": [85200], "6224": [85201], "6225": [85202], "6226": [85203], "6227": [85204], "6228": [85205], "6229": [85206], "6230": [85207], "6231": [85209], "6232": [85210], "6233": [85211], "6234": [85212], "6235": [85213], "6236": [85214], "6237": [85215], "6238": [85216], "6239": [85217], "6240": [85218], "6241": [85219], "6242": [85220], "6243": [85221], "6244": [85222], "6245": [85223], "6246": [85224], "6247": [85225], "6248": [85226], "6249": [85227], "6250": [85228], "6251": [85229], "6252": [85230], "6253": [85231], "6254": [85232], "6255": [85233], "6256": [85234], "6257": [85235], "6258": [87013], "6259": [85237], "6260": [85238], "6261": [85239], "6262": [85240], "6263": [85241], "6264": [85242], "6265": [85243], "6266": [85244], "6267": [85245], "6268": [85246], "6269": [85247], "6270": [85248], "6271": [85249], "6272": [85250], "6273": [3617], "6274": [85251], "6275": [85252], "6276": [85253], "6277": [85254], "6278": [85255], "6279": [85256], "6280": [85257], "6281": [85258], "6282": [85259], "6283": [85260], "6284": [85261], "6285": [85262], "6286": [85263], "6287": [85264], "6288": [85265], "6289": [85266], "6290": [85267], "6291": [85268], "6292": [85269], "6293": [85270], "6294": [85271], "6295": [85272], "6296": [85273], "6297": [85274], "6298": [12144], "6299": [85275], "6300": [85276], "6301": [85277], "6302": [85278], "6303": [85279], "6304": [85280], "6305": [85281], "6306": [85282], "6307": [85283], "6308": [85284], "6309": [85285], "6310": [85286], "6311": [85287], "6312": [85288], "6313": [85289], "6314": [85290], "6315": [85291], "6316": [85292], "6317": [85293], "6318": [85294], "6319": [85295], "6320": [85296], "6321": [85297], "6322": [85298], "6323": [85299], "6324": [85300], "6325": [85301], "6326": [85302], "6327": [85303], "6328": [85304], "6329": [85305], "6330": [85306], "6331": [85307], "6332": [85308], "6333": [85309], "6334": [85310], "6335": [85311], "6336": [85312], "6337": [85313], "6338": [85314], "6339": [85315], "6340": [85316], "6341": [85317], "6342": [85318], "6343": [85319], "6344": [85320], "6345": [85321], "6346": [85322], "6347": [85323], "6348": [85324], "6349": [85325], "6350": [85326], "6351": [85327], "6352": [85328], "6353": [85329], "6354": [85330], "6355": [85331], "6356": [85332], "6357": [85333], "6358": [85334], "6359": [85335], "6360": [85336], "6361": [85337], "6362": [85338], "6363": [85339], "6364": [85340], "6365": [85341], "6366": [85342], "6367": [85343], "6368": [85344], "6369": [85345], "6370": [85346], "6371": [85347], "6372": [85348], "6373": [85349], "6374": [85350], "6375": [85351], "6376": [85352], "6377": [85353], "6378": [85354], "6379": [85355], "6380": [85356], "6381": [4963], "6382": [85357], "6383": [85358], "6384": [85359], "6385": [85360], "6386": [85361], "6387": [85362], "6388": [85363], "6389": [85364], "6390": [85365], "6391": [85366], "6392": [85367], "6393": [85368], "6394": [85369], "6395": [85370], "6396": [85371], "6397": [85372], "6398": [85373], "6399": [85374], "6400": [85375], "6401": [85376], "6402": [85377], "6403": [85378], "6404": [85379], "6405": [85380], "6406": [85381], "6407": [85382], "6408": [85383], "6409": [85384], "6410": [85385], "6411": [85386], "6412": [85387], "6413": [85388], "6414": [85389], "6415": [85390], "6416": [85391], "6417": [85392], "6418": [85393], "6419": [85394], "6420": [85395], "6421": [85396], "6422": [85397], "6423": [8195], "6424": [85398], "6425": [85399], "6426": [85400], "6427": [85401], "6428": [85402], "6429": [85403], "6430": [85404], "6431": [85405], "6432": [85406], "6433": [85407], "6434": [85408], "6435": [85409], "6436": [1019], "6437": [85410], "6438": [85411], "6439": [85412], "6440": [85413], "6441": [17443], "6442": [85415], "6443": [85416], "6444": [85417], "6445": [85418], "6446": [3955], "6447": [85419], "6448": [85420], "6449": [85421], "6450": [85422], "6451": [85423], "6452": [85424], "6453": [85425], "6454": [85426], "6455": [85427], "6456": [85428], "6457": [85429], "6458": [85430], "6459": [85431], "6460": [85432], "6461": [85433], "6462": [85434], "6463": [85435], "6464": [85436], "6465": [85437], "6466": [85438], "6467": [85439], "6468": [85440], "6469": [85441], "6470": [85442], "6471": [85443], "6472": [85444], "6473": [85445], "6474": [85446], "6475": [85447], "6476": [85448], "6477": [85449], "6478": [85450], "6479": [85451], "6480": [85452], "6481": [85453], "6482": [85454], "6483": [85455], "6484": [85456], "6485": [85457], "6486": [85458], "6487": [85459], "6488": [85460], "6489": [85461], "6490": [85462], "6491": [85463], "6492": [85464], "6493": [85465], "6494": [85466], "6495": [85467], "6496": [85468], "6497": [85469], "6498": [85470], "6499": [85471], "6500": [85472], "6501": [85473], "6502": [85474], "6503": [85475], "6504": [85476], "6505": [85477], "6506": [85478], "6507": [85479], "6508": [85480], "6509": [85481], "6510": [85482], "6511": [85483], "6512": [85484], "6513": [85485], "6514": [85486], "6515": [85487], "6516": [85488], "6517": [85489], "6518": [85490], "6519": [85491], "6520": [85492], "6521": [85493], "6522": [85494], "6523": [85495], "6524": [85496], "6525": [85497], "6526": [85498], "6527": [85499], "6528": [85500], "6529": [85501], "6530": [85502], "6531": [85503], "6532": [85504], "6533": [85505], "6534": [85506], "6535": [85507], "6536": [85508], "6537": [85509], "6538": [85510], "6539": [85511], "6540": [85512], "6541": [85513], "6542": [85514], "6543": [85515], "6544": [85516], "6545": [85517], "6546": [85518], "6547": [85519], "6548": [85520], "6549": [85521], "6550": [85522], "6551": [85523], "6552": [85524], "6553": [85525], "6554": [85526], "6555": [85527], "6556": [85528], "6557": [85529], "6558": [85530], "6559": [85531], "6560": [85532], "6561": [85533], "6562": [85534], "6563": [85535], "6564": [85536], "6565": [83871], "6566": [85537], "6567": [85538], "6568": [85539], "6569": [85540], "6570": [85541], "6571": [85542], "6572": [85543], "6573": [1603], "6574": [85544], "6575": [85545], "6576": [85546], "6577": [85547], "6578": [85548], "6579": [85549], "6580": [85550], "6581": [85551], "6582": [85552], "6583": [85553], "6584": [85554], "6585": [85555], "6586": [85556], "6587": [85557], "6588": [85558], "6589": [40511], "6590": [85559], "6591": [49427], "6592": [85560], "6593": [85561], "6594": [85562], "6595": [85563], "6596": [85564], "6597": [85565], "6598": [85566], "6599": [85567], "6600": [85568], "6601": [85569], "6602": [85570], "6603": [85571], "6604": [85572], "6605": [85573], "6606": [85574], "6607": [85575], "6608": [85576], "6609": [85577], "6610": [85578], "6611": [85579], "6612": [85580], "6613": [85581], "6614": [85582], "6615": [85583], "6616": [85584], "6617": [85585], "6618": [85586], "6619": [85587], "6620": [85588], "6621": [85589], "6622": [85590], "6623": [85591], "6624": [85592], "6625": [85593], "6626": [85594], "6627": [37894], "6628": [85595], "6629": [85596], "6630": [85597], "6631": [85598], "6632": [85599], "6633": [85600], "6634": [85601], "6635": [85602], "6636": [85603], "6637": [85604], "6638": [85605], "6639": [85606], "6640": [85607], "6641": [15712], "6642": [85608], "6643": [85609], "6644": [85610], "6645": [85611], "6646": [85612], "6647": [85613], "6648": [85614], "6649": [85615], "6650": [85616], "6651": [85617], "6652": [85618], "6653": [85619], "6654": [85620], "6655": [85621], "6656": [85622], "6657": [85623], "6658": [85624], "6659": [85625], "6660": [85626], "6661": [85627], "6662": [87581], "6663": [85629], "6664": [85630], "6665": [85631], "6666": [85632], "6667": [85633], "6668": [85634], "6669": [85635], "6670": [85636], "6671": [85637], "6672": [85638], "6673": [85639], "6674": [85640], "6675": [85641], "6676": [85642], "6677": [85643], "6678": [85644], "6679": [85645], "6680": [85646], "6681": [85647], "6682": [85648], "6683": [85649], "6684": [85650], "6685": [85651], "6686": [85652], "6687": [85653], "6688": [85654], "6689": [85655], "6690": [85656], "6691": [85657], "6692": [85658], "6693": [85659], "6694": [85660], "6695": [85661], "6696": [85662], "6697": [85663], "6698": [85664], "6699": [85665], "6700": [85666], "6701": [85667], "6702": [85668], "6703": [85669], "6704": [85670], "6705": [85671], "6706": [85672], "6707": [85673], "6708": [85674], "6709": [85675], "6710": [85676], "6711": [85677], "6712": [85678], "6713": [85679], "6714": [85680], "6715": [85681], "6716": [85682], "6717": [85683], "6718": [85684], "6719": [85685], "6720": [85686], "6721": [85687], "6722": [85688], "6723": [85689], "6724": [85690], "6725": [85691], "6726": [85692], "6727": [85693], "6728": [85694], "6729": [85695], "6730": [85696], "6731": [85697], "6732": [85698], "6733": [85699], "6734": [85700], "6735": [85701], "6736": [10803], "6737": [85702], "6738": [85703], "6739": [85653], "6740": [85704], "6741": [85705], "6742": [85706], "6743": [85707], "6744": [85708], "6745": [85709], "6746": [85710], "6747": [85711], "6748": [85712], "6749": [85713], "6750": [85714], "6751": [85715], "6752": [85716], "6753": [85717], "6754": [85718], "6755": [85719], "6756": [85720], "6757": [85721], "6758": [85722], "6759": [85723], "6760": [85724], "6761": [85725], "6762": [85726], "6763": [85727], "6764": [85728], "6765": [85729], "6766": [85730], "6767": [85731], "6768": [85732], "6769": [85733], "6770": [85734], "6771": [85735], "6772": [85736], "6773": [85737], "6774": [85738], "6775": [85739], "6776": [85740], "6777": [85741], "6778": [85742], "6779": [85743], "6780": [85744], "6781": [85745], "6782": [8504], "6783": [85746], "6784": [85747], "6785": [85748], "6786": [85749], "6787": [85750], "6788": [85751], "6789": [85752], "6790": [85753], "6791": [85754], "6792": [85755], "6793": [85756], "6794": [85757], "6795": [85758], "6796": [85759], "6797": [85760], "6798": [85761], "6799": [85762], "6800": [85763], "6801": [85764], "6802": [85765], "6803": [85766], "6804": [85767], "6805": [85768], "6806": [85769], "6807": [85770], "6808": [85771], "6809": [85772], "6810": [85773], "6811": [85774], "6812": [83413], "6813": [85775], "6814": [85776], "6815": [13357], "6816": [85777], "6817": [85778], "6818": [85779], "6819": [85780], "6820": [85781], "6821": [85782], "6822": [85783], "6823": [85784], "6824": [85785], "6825": [85786], "6826": [85787], "6827": [85788], "6828": [85789], "6829": [85790], "6830": [85791], "6831": [85792], "6832": [85793], "6833": [85794], "6834": [85795], "6835": [85796], "6836": [85797], "6837": [85798], "6838": [85799], "6839": [85800], "6840": [85801], "6841": [85802], "6842": [85803], "6843": [85804], "6844": [85805], "6845": [85806], "6846": [85807], "6847": [85808], "6848": [85809], "6849": [85810], "6850": [85811], "6851": [85812], "6852": [85813], "6853": [85814], "6854": [85815], "6855": [85816], "6856": [85817], "6857": [85818], "6858": [85819], "6859": [85820], "6860": [85821], "6861": [85822], "6862": [85823], "6863": [85824], "6864": [85825], "6865": [85826], "6866": [85827], "6867": [85828], "6868": [85829], "6869": [85830], "6870": [85831], "6871": [10178], "6872": [85832], "6873": [85833], "6874": [85834], "6875": [85835], "6876": [85836], "6877": [85837], "6878": [85838], "6879": [85839], "6880": [85840], "6881": [85841], "6882": [1550], "6883": [85842], "6884": [85843], "6885": [85844], "6886": [85845], "6887": [85846], "6888": [85847], "6889": [85848], "6890": [85849], "6891": [85850], "6892": [39171], "6893": [85851], "6894": [85852], "6895": [85853], "6896": [85854], "6897": [85855], "6898": [85856], "6899": [85857], "6900": [85858], "6901": [85859], "6902": [85860], "6903": [85861], "6904": [85862], "6905": [85863], "6906": [85864], "6907": [85865], "6908": [85866], "6909": [85867], "6910": [85868], "6911": [85869], "6912": [85870], "6913": [85871], "6914": [85872], "6915": [85873], "6916": [85874], "6917": [85875], "6918": [85876], "6919": [85877], "6920": [85878], "6921": [85879], "6922": [85880], "6923": [85881], "6924": [85882], "6925": [85883], "6926": [85884], "6927": [85885], "6928": [85886], "6929": [85887], "6930": [85888], "6931": [85889], "6932": [85890], "6933": [85891], "6934": [85892], "6935": [85893], "6936": [85894], "6937": [85895], "6938": [85896], "6939": [85897], "6940": [85898], "6941": [85899], "6942": [85900], "6943": [85901], "6944": [85902], "6945": [85903], "6946": [85904], "6947": [85905], "6948": [85906], "6949": [85907], "6950": [85908], "6951": [85909], "6952": [85910], "6953": [85911], "6954": [85912], "6955": [85913], "6956": [85914], "6957": [85915], "6958": [85916], "6959": [85917], "6960": [85918], "6961": [85919], "6962": [85920], "6963": [85921], "6964": [85922], "6965": [85923], "6966": [85924], "6967": [85925], "6968": [85926], "6969": [85927], "6970": [85928], "6971": [85929], "6972": [85930], "6973": [85931], "6974": [85932], "6975": [85933], "6976": [85934], "6977": [85935], "6978": [85936], "6979": [85937], "6980": [85938], "6981": [85939], "6982": [85940], "6983": [85941], "6984": [85942], "6985": [85943], "6986": [85944], "6987": [85945], "6988": [85946], "6989": [1170], "6990": [85947], "6991": [85948], "6992": [85949], "6993": [85950], "6994": [85951], "6995": [85952], "6996": [85953], "6997": [85954], "6998": [85955], "6999": [85956], "7000": [85957], "7001": [85958], "7002": [85959], "7003": [85960], "7004": [85961], "7005": [85962], "7006": [85963], "7007": [85964], "7008": [85965], "7009": [85966], "7010": [85967], "7011": [85968], "7012": [85969], "7013": [85970], "7014": [85971], "7015": [85972], "7016": [85973], "7017": [85974], "7018": [85975], "7019": [85976], "7020": [85977], "7021": [85978], "7022": [85979], "7023": [85980], "7024": [85981], "7025": [85982], "7026": [85983], "7027": [85984], "7028": [85985], "7029": [85986], "7030": [85987], "7031": [85988], "7032": [85989], "7033": [85990], "7034": [85991], "7035": [85992], "7036": [85993], "7037": [85994], "7038": [85995], "7039": [85996], "7040": [85997], "7041": [85998], "7042": [85999], "7043": [86000], "7044": [86001], "7045": [86002], "7046": [86003], "7047": [86004], "7048": [86005], "7049": [86006], "7050": [86007], "7051": [86008], "7052": [86009], "7053": [86010], "7054": [86011], "7055": [86012], "7056": [86013], "7057": [86014], "7058": [86015], "7059": [86016], "7060": [86017], "7061": [86018], "7062": [86019], "7063": [86020], "7064": [86021], "7065": [86022], "7066": [86023], "7067": [86024], "7068": [86025], "7069": [86026], "7070": [86027], "7071": [86028], "7072": [86029], "7073": [86030], "7074": [86031], "7075": [86032], "7076": [86033], "7077": [86034], "7078": [86035], "7079": [86036], "7080": [86037], "7081": [86038], "7082": [86039], "7083": [86040], "7084": [86041], "7085": [86042], "7086": [86043], "7087": [86044], "7088": [86045], "7089": [86046], "7090": [86047], "7091": [86048], "7092": [86049], "7093": [86050], "7094": [86051], "7095": [86052], "7096": [86053], "7097": [86054], "7098": [86055], "7099": [86056], "7100": [86057], "7101": [86058], "7102": [86059], "7103": [86060], "7104": [86061], "7105": [86062], "7106": [86063], "7107": [86064], "7108": [86065], "7109": [86066], "7110": [86067], "7111": [86068], "7112": [86069], "7113": [86070], "7114": [86071], "7115": [86072], "7116": [86073], "7117": [86074], "7118": [86075], "7119": [86076], "7120": [86077], "7121": [86078], "7122": [86079], "7123": [86080], "7124": [86081], "7125": [86082], "7126": [86083], "7127": [86084], "7128": [86085], "7129": [86086], "7130": [86087], "7131": [86088], "7132": [86089], "7133": [86090], "7134": [86091], "7135": [86092], "7136": [86093], "7137": [86094], "7138": [86095], "7139": [86096], "7140": [86097], "7141": [86098], "7142": [86099], "7143": [86100], "7144": [86101], "7145": [86102], "7146": [86103], "7147": [86104], "7148": [86105], "7149": [86106], "7150": [86107], "7151": [86108], "7152": [86109], "7153": [86110], "7154": [86111], "7155": [86112], "7156": [86113], "7157": [86114], "7158": [86116], "7159": [86117], "7160": [86118], "7161": [86119], "7162": [86120], "7163": [86121], "7164": [86122], "7165": [86123], "7166": [86124], "7167": [86125], "7168": [86126], "7169": [86127], "7170": [86128], "7171": [86129], "7172": [86130], "7173": [86131], "7174": [86132], "7175": [86133], "7176": [86134], "7177": [86135], "7178": [86136], "7179": [86137], "7180": [86138], "7181": [86139], "7182": [86140], "7183": [86141], "7184": [86142], "7185": [86143], "7186": [86144], "7187": [86145], "7188": [86146], "7189": [86147], "7190": [86148], "7191": [86149], "7192": [86150], "7193": [86151], "7194": [86152], "7195": [86153], "7196": [86154], "7197": [86155], "7198": [86156], "7199": [86157], "7200": [3094], "7201": [86158], "7202": [86159], "7203": [86160], "7204": [86161], "7205": [86162], "7206": [86163], "7207": [86164], "7208": [86165], "7209": [86166], "7210": [86167], "7211": [86168], "7212": [86169], "7213": [86170], "7214": [86171], "7215": [86172], "7216": [86173], "7217": [86174], "7218": [86175], "7219": [86176], "7220": [86177], "7221": [86178], "7222": [86179], "7223": [86180], "7224": [86181], "7225": [86182], "7226": [86183], "7227": [86184], "7228": [86185], "7229": [86186], "7230": [86187], "7231": [86188], "7232": [86189], "7233": [86190], "7234": [86191], "7235": [86192], "7236": [86193], "7237": [86194], "7238": [86195], "7239": [86196], "7240": [86197], "7241": [86198], "7242": [86199], "7243": [86200], "7244": [86201], "7245": [86202], "7246": [86203], "7247": [86204], "7248": [86205], "7249": [86206], "7250": [86207], "7251": [86208], "7252": [86209], "7253": [86210], "7254": [86211], "7255": [86212], "7256": [86213], "7257": [86214], "7258": [86215], "7259": [86216], "7260": [86217], "7261": [86218], "7262": [86219], "7263": [86220], "7264": [86221], "7265": [86222], "7266": [86223], "7267": [86224], "7268": [86225], "7269": [86226], "7270": [86227], "7271": [86228], "7272": [86229], "7273": [86230], "7274": [86231], "7275": [86232], "7276": [86233], "7277": [86234], "7278": [86235], "7279": [86236], "7280": [86237], "7281": [86238], "7282": [86239], "7283": [86240], "7284": [86241], "7285": [86242], "7286": [86243], "7287": [86244], "7288": [86245], "7289": [86246], "7290": [86247], "7291": [86248], "7292": [86249], "7293": [86250], "7294": [86251], "7295": [86252], "7296": [86253], "7297": [86254], "7298": [86255], "7299": [86256], "7300": [86257], "7301": [86258], "7302": [86259], "7303": [86260], "7304": [86261], "7305": [86262], "7306": [86263], "7307": [86264], "7308": [86265], "7309": [86266], "7310": [86267], "7311": [86268], "7312": [83765], "7313": [86269], "7314": [86270], "7315": [86271], "7316": [86272], "7317": [86273], "7318": [86274], "7319": [86275], "7320": [86276], "7321": [86277], "7322": [86278], "7323": [86279], "7324": [86280], "7325": [86281], "7326": [86282], "7327": [86283], "7328": [86284], "7329": [86285], "7330": [86286], "7331": [86287], "7332": [86288], "7333": [86289], "7334": [86290], "7335": [86291], "7336": [86292], "7337": [86293], "7338": [82033], "7339": [86294], "7340": [86295], "7341": [86296], "7342": [86297], "7343": [86298], "7344": [86299], "7345": [86300], "7346": [86301], "7347": [86302], "7348": [86303], "7349": [86304], "7350": [86305], "7351": [86306], "7352": [86307], "7353": [86308], "7354": [86309], "7355": [86310], "7356": [86311], "7357": [86312], "7358": [86313], "7359": [86314], "7360": [86315], "7361": [86316], "7362": [86317], "7363": [86318], "7364": [86319], "7365": [86320], "7366": [86321], "7367": [50919], "7368": [86322], "7369": [86323], "7370": [86324], "7371": [86325], "7372": [86326], "7373": [86327], "7374": [86328], "7375": [86329], "7376": [86330], "7377": [86331], "7378": [86332], "7379": [86333], "7380": [86334], "7381": [86335], "7382": [86336], "7383": [86337], "7384": [86338], "7385": [86339], "7386": [86340], "7387": [86341], "7388": [86342], "7389": [86343], "7390": [86344], "7391": [86345], "7392": [86346], "7393": [86347], "7394": [86348], "7395": [86349], "7396": [86350], "7397": [86351], "7398": [35406], "7399": [86352], "7400": [86353], "7401": [86354], "7402": [86355], "7403": [86356], "7404": [86357], "7405": [86358], "7406": [86359], "7407": [86360], "7408": [86361], "7409": [51746], "7410": [86362], "7411": [18390], "7412": [86363], "7413": [86364], "7414": [86365], "7415": [86366], "7416": [86367], "7417": [86368], "7418": [86369], "7419": [86370], "7420": [86371], "7421": [86372], "7422": [86373], "7423": [86374], "7424": [86375], "7425": [86376], "7426": [86377], "7427": [86378], "7428": [86379], "7429": [86380], "7430": [86381], "7431": [86382], "7432": [86383], "7433": [86384], "7434": [86385], "7435": [86386], "7436": [86387], "7437": [86388], "7438": [86389], "7439": [86390], "7440": [86391], "7441": [86392], "7442": [86393], "7443": [86394], "7444": [86395], "7445": [86396], "7446": [86397], "7447": [86398], "7448": [86399], "7449": [86400], "7450": [86401], "7451": [86402], "7452": [86403], "7453": [86404], "7454": [86405], "7455": [86406], "7456": [16029], "7457": [86407], "7458": [86408], "7459": [86409], "7460": [86410], "7461": [86411], "7462": [86412], "7463": [86413], "7464": [86414], "7465": [86415], "7466": [86416], "7467": [86417], "7468": [86418], "7469": [86419], "7470": [86420], "7471": [86421], "7472": [86422], "7473": [86423], "7474": [86424], "7475": [86425], "7476": [86426], "7477": [86427], "7478": [86428], "7479": [86429], "7480": [86430], "7481": [86431], "7482": [86432], "7483": [9179], "7484": [86433], "7485": [86434], "7486": [86435], "7487": [86436], "7488": [86437], "7489": [86438], "7490": [86439], "7491": [86440], "7492": [86441], "7493": [86442], "7494": [86443], "7495": [86444], "7496": [86445], "7497": [86446], "7498": [86447], "7499": [86448], "7500": [3056], "7501": [41979], "7502": [86449], "7503": [86450], "7504": [86451], "7505": [86452], "7506": [86453], "7507": [86454], "7508": [86455], "7509": [86456], "7510": [86457], "7511": [86458], "7512": [86459], "7513": [86460], "7514": [86461], "7515": [86462], "7516": [86463], "7517": [86464], "7518": [86465], "7519": [86466], "7520": [86467], "7521": [86468], "7522": [86469], "7523": [86470], "7524": [86471], "7525": [86472], "7526": [86473], "7527": [86474], "7528": [86475], "7529": [86476], "7530": [86477], "7531": [86478], "7532": [86479], "7533": [86480], "7534": [86481], "7535": [86482], "7536": [86483], "7537": [86484], "7538": [86485], "7539": [86486], "7540": [86487], "7541": [86488], "7542": [86489], "7543": [86490], "7544": [86491], "7545": [86492], "7546": [86493], "7547": [86494], "7548": [86495], "7549": [86496], "7550": [86497], "7551": [86498], "7552": [86499], "7553": [86500], "7554": [86501], "7555": [86502], "7556": [86503], "7557": [86504], "7558": [86505], "7559": [86506], "7560": [86507], "7561": [86508], "7562": [86509], "7563": [86510], "7564": [86511], "7565": [1258], "7566": [51746], "7567": [86512], "7568": [86513], "7569": [86514], "7570": [86515], "7571": [86516], "7572": [86517], "7573": [86518], "7574": [86519], "7575": [86520], "7576": [86521], "7577": [86522], "7578": [86523], "7579": [86524], "7580": [86525], "7581": [86527], "7582": [86528], "7583": [86529], "7584": [86530], "7585": [86531], "7586": [86532], "7587": [86533], "7588": [86534], "7589": [86535], "7590": [86536], "7591": [86537], "7592": [86538], "7593": [86539], "7594": [86540], "7595": [86541], "7596": [86542], "7597": [86543], "7598": [86544], "7599": [86545], "7600": [86546], "7601": [86547], "7602": [86548], "7603": [86549], "7604": [6027], "7605": [86550], "7606": [86551], "7607": [86552], "7608": [86553], "7609": [86554], "7610": [86555], "7611": [86556], "7612": [86557], "7613": [86558], "7614": [86559], "7615": [86560], "7616": [86561], "7617": [86562], "7618": [86563], "7619": [86564], "7620": [86565], "7621": [86566], "7622": [86567], "7623": [86568], "7624": [86569], "7625": [86570], "7626": [86571], "7627": [86572], "7628": [86573], "7629": [86574], "7630": [86575], "7631": [86576], "7632": [86577], "7633": [86578], "7634": [86579], "7635": [86580], "7636": [86581], "7637": [15317], "7638": [86582], "7639": [86583], "7640": [86584], "7641": [86585], "7642": [86586], "7643": [86587], "7644": [86588], "7645": [86589], "7646": [86590], "7647": [86591], "7648": [86592], "7649": [86593], "7650": [86594], "7651": [86595], "7652": [86596], "7653": [86597], "7654": [86598], "7655": [86599], "7656": [86600], "7657": [86601], "7658": [86602], "7659": [86603], "7660": [86604], "7661": [86605], "7662": [86606], "7663": [86607], "7664": [86608], "7665": [86609], "7666": [86610], "7667": [86611], "7668": [86612], "7669": [86613], "7670": [86614], "7671": [86615], "7672": [86616], "7673": [86617], "7674": [86618], "7675": [86619], "7676": [86620], "7677": [86621], "7678": [86622], "7679": [86623], "7680": [86624], "7681": [86625], "7682": [86626], "7683": [86627], "7684": [86628], "7685": [86629], "7686": [86630], "7687": [86631], "7688": [86632], "7689": [86633], "7690": [86634], "7691": [86635], "7692": [86636], "7693": [49779], "7694": [86637], "7695": [86638], "7696": [86639], "7697": [86640], "7698": [86641], "7699": [86642], "7700": [86643], "7701": [86644], "7702": [86645], "7703": [86646], "7704": [86647], "7705": [86648], "7706": [86649], "7707": [86650], "7708": [86652], "7709": [21867], "7710": [86653], "7711": [86654], "7712": [86655], "7713": [86656], "7714": [86657], "7715": [86658], "7716": [86659], "7717": [86660], "7718": [86661], "7719": [86662], "7720": [86663], "7721": [86664], "7722": [86665], "7723": [86666], "7724": [86667], "7725": [86668], "7726": [86669], "7727": [86670], "7728": [86671], "7729": [86672], "7730": [86673], "7731": [86674], "7732": [86675], "7733": [86676], "7734": [86677], "7735": [86678], "7736": [86679], "7737": [86680], "7738": [86681], "7739": [86682], "7740": [86683], "7741": [86684], "7742": [86685], "7743": [86686], "7744": [86687], "7745": [86713], "7746": [86689], "7747": [86690], "7748": [86691], "7749": [86692], "7750": [86693], "7751": [86694], "7752": [86695], "7753": [86696], "7754": [86697], "7755": [86698], "7756": [86699], "7757": [86700], "7758": [86701], "7759": [86702], "7760": [86703], "7761": [86704], "7762": [86705], "7763": [86706], "7764": [86707], "7765": [86708], "7766": [86709], "7767": [316], "7768": [86710], "7769": [86711], "7770": [86712], "7771": [86714], "7772": [86715], "7773": [86716], "7774": [86717], "7775": [86718], "7776": [86719], "7777": [86720], "7778": [86721], "7779": [86722], "7780": [86723], "7781": [86724], "7782": [86725], "7783": [86726], "7784": [86727], "7785": [86728], "7786": [86729], "7787": [86730], "7788": [86731], "7789": [86732], "7790": [1279], "7791": [86733], "7792": [86734], "7793": [86735], "7794": [86736], "7795": [88095], "7796": [86738], "7797": [86739], "7798": [86740], "7799": [86741], "7800": [86742], "7801": [86743], "7802": [86744], "7803": [86745], "7804": [86746], "7805": [86747], "7806": [86748], "7807": [86749], "7808": [86750], "7809": [86751], "7810": [86752], "7811": [86753], "7812": [86754], "7813": [86755], "7814": [86756], "7815": [86757], "7816": [86758], "7817": [86759], "7818": [86760], "7819": [86761], "7820": [86762], "7821": [86763], "7822": [86764], "7823": [86765], "7824": [86766], "7825": [86767], "7826": [86768], "7827": [86769], "7828": [86770], "7829": [86771], "7830": [86772], "7831": [86773], "7832": [86774], "7833": [86775], "7834": [86776], "7835": [86777], "7836": [86778], "7837": [86779], "7838": [86780], "7839": [86781], "7840": [86782], "7841": [86783], "7842": [86784], "7843": [86785], "7844": [86786], "7845": [86787], "7846": [86788], "7847": [86789], "7848": [86790], "7849": [86791], "7850": [86792], "7851": [86793], "7852": [86794], "7853": [86795], "7854": [86796], "7855": [86797], "7856": [86798], "7857": [86799], "7858": [86800], "7859": [86801], "7860": [86802], "7861": [40552], "7862": [86803], "7863": [86804], "7864": [86805], "7865": [86806], "7866": [86807], "7867": [86808], "7868": [86809], "7869": [86810], "7870": [86811], "7871": [86812], "7872": [86813], "7873": [86814], "7874": [86815], "7875": [86816], "7876": [86817], "7877": [86818], "7878": [86819], "7879": [86820], "7880": [86821], "7881": [86822], "7882": [86823], "7883": [86824], "7884": [86825], "7885": [86826], "7886": [86827], "7887": [86828], "7888": [86829], "7889": [86830], "7890": [86831], "7891": [86832], "7892": [86833], "7893": [86834], "7894": [86835], "7895": [86836], "7896": [86837], "7897": [86838], "7898": [86839], "7899": [86840], "7900": [86841], "7901": [86842], "7902": [86843], "7903": [86844], "7904": [86845], "7905": [86846], "7906": [86847], "7907": [86848], "7908": [86849], "7909": [86850], "7910": [86851], "7911": [86852], "7912": [86853], "7913": [86854], "7914": [86855], "7915": [86856], "7916": [86857], "7917": [86858], "7918": [86859], "7919": [86860], "7920": [86861], "7921": [86862], "7922": [86863], "7923": [86864], "7924": [86865], "7925": [86866], "7926": [86867], "7927": [86868], "7928": [86869], "7929": [86870], "7930": [80768], "7931": [86871], "7932": [86872], "7933": [86873], "7934": [86874], "7935": [86875], "7936": [86876], "7937": [86877], "7938": [86878], "7939": [86879], "7940": [86880], "7941": [86881], "7942": [86882], "7943": [86883], "7944": [86884], "7945": [86885], "7946": [86886], "7947": [86887], "7948": [86888], "7949": [86889], "7950": [86890], "7951": [86891], "7952": [86892], "7953": [50660], "7954": [86893], "7955": [86894], "7956": [86895], "7957": [86896], "7958": [86897], "7959": [86898], "7960": [86899], "7961": [86900], "7962": [86901], "7963": [86902], "7964": [86903], "7965": [86904], "7966": [86905], "7967": [86906], "7968": [86907], "7969": [86908], "7970": [86909], "7971": [86910], "7972": [86911], "7973": [86912], "7974": [86913], "7975": [86914], "7976": [86915], "7977": [86916], "7978": [1731], "7979": [86917], "7980": [86918], "7981": [86919], "7982": [86920], "7983": [86921], "7984": [86922], "7985": [86923], "7986": [86924], "7987": [86925], "7988": [86926], "7989": [86927], "7990": [86928], "7991": [86929], "7992": [86930], "7993": [86931], "7994": [86932], "7995": [86933], "7996": [86934], "7997": [86935], "7998": [86936], "7999": [86937], "8000": [10255], "8001": [86938], "8002": [86939], "8003": [86940], "8004": [86941], "8005": [86942], "8006": [86943], "8007": [86944], "8008": [86945], "8009": [86946], "8010": [86947], "8011": [86948], "8012": [86949], "8013": [86950], "8014": [86951], "8015": [86952], "8016": [86953], "8017": [86954], "8018": [86955], "8019": [86956], "8020": [86957], "8021": [86958], "8022": [86959], "8023": [86960], "8024": [86961], "8025": [86962], "8026": [86963], "8027": [86964], "8028": [86965], "8029": [86966], "8030": [86967], "8031": [86968], "8032": [86969], "8033": [86970], "8034": [86971], "8035": [86972], "8036": [86973], "8037": [86974], "8038": [86975], "8039": [86976], "8040": [86977], "8041": [86978], "8042": [86979], "8043": [86980], "8044": [86981], "8045": [86982], "8046": [86983], "8047": [86984], "8048": [86985], "8049": [86986], "8050": [86987], "8051": [86988], "8052": [86989], "8053": [86990], "8054": [86991], "8055": [86992], "8056": [86993], "8057": [86994], "8058": [86995], "8059": [86996], "8060": [86997], "8061": [86998], "8062": [86999], "8063": [49382], "8064": [87000], "8065": [87001], "8066": [87002], "8067": [87004], "8068": [87005], "8069": [87006], "8070": [87007], "8071": [87008], "8072": [87009], "8073": [87010], "8074": [87011], "8075": [87012], "8076": [87014], "8077": [87015], "8078": [87016], "8079": [87017], "8080": [87018], "8081": [87019], "8082": [87020], "8083": [87021], "8084": [87022], "8085": [87023], "8086": [87024], "8087": [87025], "8088": [87026], "8089": [5486], "8090": [87027], "8091": [87028], "8092": [87029], "8093": [87030], "8094": [87031], "8095": [87032], "8096": [87033], "8097": [87034], "8098": [87035], "8099": [87036], "8100": [87037], "8101": [87038], "8102": [87039], "8103": [87040], "8104": [87041], "8105": [87042], "8106": [87043], "8107": [87044], "8108": [87045], "8109": [87046], "8110": [87047], "8111": [87048], "8112": [87049], "8113": [87050], "8114": [87051], "8115": [87052], "8116": [87053], "8117": [87054], "8118": [87055], "8119": [87056], "8120": [87057], "8121": [87058], "8122": [87059], "8123": [10435], "8124": [87060], "8125": [87061], "8126": [87062], "8127": [87063], "8128": [87064], "8129": [87065], "8130": [87066], "8131": [87067], "8132": [87068], "8133": [87069], "8134": [87070], "8135": [87071], "8136": [87072], "8137": [87073], "8138": [87074], "8139": [87075], "8140": [87076], "8141": [87077], "8142": [87078], "8143": [87079], "8144": [87080], "8145": [87081], "8146": [87082], "8147": [87083], "8148": [87084], "8149": [87085], "8150": [87086], "8151": [87087], "8152": [87088], "8153": [87089], "8154": [87090], "8155": [87091], "8156": [87092], "8157": [87093], "8158": [87094], "8159": [87095], "8160": [87096], "8161": [87097], "8162": [87098], "8163": [87099], "8164": [87100], "8165": [87101], "8166": [87102], "8167": [87103], "8168": [87104], "8169": [87105], "8170": [87106], "8171": [87107], "8172": [87108], "8173": [87109], "8174": [87110], "8175": [87111], "8176": [87112], "8177": [87113], "8178": [87114], "8179": [87115], "8180": [87116], "8181": [87117], "8182": [87118], "8183": [87119], "8184": [87120], "8185": [87121], "8186": [87122], "8187": [87123], "8188": [87124], "8189": [87125], "8190": [87126], "8191": [87127], "8192": [87128], "8193": [87129], "8194": [87130], "8195": [87131], "8196": [87132], "8197": [87133], "8198": [87134], "8199": [87135], "8200": [2089], "8201": [87136], "8202": [87137], "8203": [87138], "8204": [87139], "8205": [87140], "8206": [87141], "8207": [87142], "8208": [87143], "8209": [87144], "8210": [87145], "8211": [87146], "8212": [87147], "8213": [87148], "8214": [87149], "8215": [87150], "8216": [87151], "8217": [87152], "8218": [87153], "8219": [87154], "8220": [87155], "8221": [87156], "8222": [87157], "8223": [87158], "8224": [87159], "8225": [87160], "8226": [87161], "8227": [87162], "8228": [87163], "8229": [87164], "8230": [87165], "8231": [87166], "8232": [87167], "8233": [87168], "8234": [87169], "8235": [87170], "8236": [87171], "8237": [87172], "8238": [87173], "8239": [87174], "8240": [87175], "8241": [87176], "8242": [87177], "8243": [87178], "8244": [87179], "8245": [87180], "8246": [87181], "8247": [87182], "8248": [87183], "8249": [87184], "8250": [87185], "8251": [87186], "8252": [87187], "8253": [87188], "8254": [87189], "8255": [87190], "8256": [87191], "8257": [87192], "8258": [87193], "8259": [87194], "8260": [87195], "8261": [87196], "8262": [87197], "8263": [87198], "8264": [87199], "8265": [87200], "8266": [87201], "8267": [87202], "8268": [87203], "8269": [87204], "8270": [87205], "8271": [87206], "8272": [87207], "8273": [87208], "8274": [87209], "8275": [87210], "8276": [87211], "8277": [87212], "8278": [87213], "8279": [87214], "8280": [87215], "8281": [87216], "8282": [87217], "8283": [87218], "8284": [87219], "8285": [87220], "8286": [87221], "8287": [87222], "8288": [87223], "8289": [87224], "8290": [87225], "8291": [87226], "8292": [87227], "8293": [87228], "8294": [87229], "8295": [87230], "8296": [87231], "8297": [87232], "8298": [87233], "8299": [87234], "8300": [87235], "8301": [87236], "8302": [87237], "8303": [87238], "8304": [87239], "8305": [86487], "8306": [87240], "8307": [87241], "8308": [87242], "8309": [87243], "8310": [87244], "8311": [87245], "8312": [87246], "8313": [87247], "8314": [87248], "8315": [87249], "8316": [87250], "8317": [87251], "8318": [87252], "8319": [87253], "8320": [87254], "8321": [87255], "8322": [87256], "8323": [87257], "8324": [87258], "8325": [87259], "8326": [87260], "8327": [87261], "8328": [87262], "8329": [87263], "8330": [87264], "8331": [87265], "8332": [87266], "8333": [87267], "8334": [87268], "8335": [87269], "8336": [87270], "8337": [87271], "8338": [87272], "8339": [87273], "8340": [87274], "8341": [87275], "8342": [87276], "8343": [87277], "8344": [87278], "8345": [87279], "8346": [87280], "8347": [87281], "8348": [87282], "8349": [87283], "8350": [87284], "8351": [87285], "8352": [87286], "8353": [87287], "8354": [87288], "8355": [87289], "8356": [87290], "8357": [87291], "8358": [87292], "8359": [87293], "8360": [87294], "8361": [3767], "8362": [87295], "8363": [87296], "8364": [87297], "8365": [87298], "8366": [87299], "8367": [87300], "8368": [87301], "8369": [87302], "8370": [87303], "8371": [87304], "8372": [87305], "8373": [87306], "8374": [87307], "8375": [87308], "8376": [87309], "8377": [87310], "8378": [87311], "8379": [87312], "8380": [87313], "8381": [87314], "8382": [6027], "8383": [87315], "8384": [87316], "8385": [49939], "8386": [87317], "8387": [87318], "8388": [87319], "8389": [87320], "8390": [87321], "8391": [87322], "8392": [87323], "8393": [87324], "8394": [87325], "8395": [87326], "8396": [87327], "8397": [87328], "8398": [87329], "8399": [87330], "8400": [87331], "8401": [87332], "8402": [87333], "8403": [87334], "8404": [87335], "8405": [87336], "8406": [87337], "8407": [87338], "8408": [87339], "8409": [87340], "8410": [87341], "8411": [87342], "8412": [87343], "8413": [87344], "8414": [87345], "8415": [87346], "8416": [87347], "8417": [87348], "8418": [87349], "8419": [87350], "8420": [87351], "8421": [87352], "8422": [87353], "8423": [87354], "8424": [87355], "8425": [87356], "8426": [87357], "8427": [87358], "8428": [87359], "8429": [87360], "8430": [87361], "8431": [87362], "8432": [87363], "8433": [87364], "8434": [87365], "8435": [87366], "8436": [87367], "8437": [87368], "8438": [87369], "8439": [87370], "8440": [87371], "8441": [87372], "8442": [87373], "8443": [87374], "8444": [87375], "8445": [87376], "8446": [87377], "8447": [87378], "8448": [87379], "8449": [87380], "8450": [49], "8451": [87381], "8452": [87382], "8453": [87383], "8454": [87384], "8455": [87385], "8456": [87386], "8457": [87387], "8458": [87388], "8459": [3828], "8460": [87389], "8461": [87390], "8462": [87391], "8463": [87392], "8464": [87393], "8465": [87394], "8466": [87395], "8467": [87396], "8468": [87397], "8469": [87398], "8470": [87399], "8471": [87400], "8472": [87401], "8473": [87402], "8474": [87403], "8475": [87404], "8476": [87405], "8477": [87406], "8478": [87407], "8479": [87408], "8480": [87409], "8481": [87410], "8482": [87411], "8483": [87412], "8484": [87413], "8485": [87414], "8486": [87415], "8487": [87416], "8488": [87417], "8489": [87418], "8490": [87419], "8491": [87420], "8492": [87421], "8493": [87422], "8494": [87423], "8495": [87424], "8496": [87425], "8497": [87426], "8498": [87427], "8499": [19960], "8500": [37909], "8501": [87428], "8502": [87429], "8503": [87430], "8504": [87431], "8505": [87432], "8506": [87433], "8507": [87434], "8508": [87435], "8509": [87436], "8510": [87437], "8511": [87438], "8512": [87439], "8513": [87440], "8514": [87441], "8515": [87442], "8516": [87443], "8517": [87444], "8518": [87445], "8519": [87446], "8520": [87447], "8521": [87448], "8522": [87449], "8523": [87450], "8524": [87451], "8525": [87452], "8526": [87453], "8527": [87454], "8528": [87455], "8529": [87456], "8530": [87457], "8531": [87458], "8532": [87459], "8533": [87460], "8534": [87461], "8535": [87462], "8536": [87463], "8537": [87464], "8538": [87465], "8539": [87466], "8540": [87467], "8541": [87468], "8542": [87469], "8543": [87470], "8544": [87471], "8545": [87472], "8546": [87473], "8547": [87474], "8548": [87475], "8549": [87476], "8550": [87477], "8551": [87478], "8552": [87479], "8553": [87480], "8554": [87481], "8555": [87482], "8556": [87483], "8557": [87484], "8558": [87485], "8559": [87486], "8560": [87487], "8561": [87488], "8562": [87489], "8563": [87490], "8564": [87491], "8565": [87492], "8566": [87493], "8567": [87494], "8568": [87495], "8569": [87496], "8570": [87497], "8571": [87498], "8572": [87499], "8573": [87500], "8574": [87501], "8575": [87502], "8576": [87503], "8577": [31238], "8578": [87504], "8579": [87505], "8580": [87506], "8581": [87507], "8582": [87508], "8583": [87509], "8584": [87510], "8585": [87511], "8586": [87512], "8587": [87513], "8588": [87514], "8589": [87515], "8590": [87516], "8591": [87517], "8592": [87518], "8593": [87519], "8594": [87520], "8595": [87521], "8596": [87522], "8597": [87523], "8598": [87524], "8599": [87525], "8600": [87526], "8601": [87527], "8602": [87528], "8603": [87529], "8604": [87530], "8605": [87531], "8606": [87532], "8607": [87533], "8608": [87534], "8609": [87535], "8610": [87536], "8611": [87537], "8612": [87538], "8613": [87539], "8614": [87540], "8615": [87541], "8616": [875], "8617": [87542], "8618": [87543], "8619": [87544], "8620": [87545], "8621": [87546], "8622": [87547], "8623": [87548], "8624": [87549], "8625": [14877], "8626": [87550], "8627": [87551], "8628": [87552], "8629": [87553], "8630": [87554], "8631": [87555], "8632": [87556], "8633": [87557], "8634": [87558], "8635": [87559], "8636": [87560], "8637": [87561], "8638": [87562], "8639": [87563], "8640": [87564], "8641": [87565], "8642": [87566], "8643": [6741], "8644": [87567], "8645": [87568], "8646": [87569], "8647": [87570], "8648": [87571], "8649": [87572], "8650": [87573], "8651": [87574], "8652": [87575], "8653": [87576], "8654": [87577], "8655": [87578], "8656": [87579], "8657": [87580], "8658": [87582], "8659": [87583], "8660": [87584], "8661": [87585], "8662": [87586], "8663": [87587], "8664": [87588], "8665": [87589], "8666": [87590], "8667": [87591], "8668": [87592], "8669": [87593], "8670": [87594], "8671": [87595], "8672": [87596], "8673": [87597], "8674": [87598], "8675": [87599], "8676": [87600], "8677": [87601], "8678": [87602], "8679": [87603], "8680": [87604], "8681": [87605], "8682": [87606], "8683": [87607], "8684": [87608], "8685": [87609], "8686": [87610], "8687": [87611], "8688": [87612], "8689": [87613], "8690": [87614], "8691": [87615], "8692": [87616], "8693": [87617], "8694": [87618], "8695": [87619], "8696": [87620], "8697": [87621], "8698": [753], "8699": [87622], "8700": [87623], "8701": [87624], "8702": [87625], "8703": [87626], "8704": [87627], "8705": [87628], "8706": [9804], "8707": [87629], "8708": [87630], "8709": [87631], "8710": [87632], "8711": [87633], "8712": [87634], "8713": [87635], "8714": [87636], "8715": [87637], "8716": [87638], "8717": [87639], "8718": [87640], "8719": [87641], "8720": [87642], "8721": [87643], "8722": [87644], "8723": [87645], "8724": [87646], "8725": [87647], "8726": [87648], "8727": [87649], "8728": [87650], "8729": [87651], "8730": [87652], "8731": [87653], "8732": [87654], "8733": [18736], "8734": [87655], "8735": [87656], "8736": [87657], "8737": [87658], "8738": [87659], "8739": [83927], "8740": [87660], "8741": [87661], "8742": [37312], "8743": [87662], "8744": [87663], "8745": [49317], "8746": [87664], "8747": [87665], "8748": [87666], "8749": [87667], "8750": [87668], "8751": [87669], "8752": [87670], "8753": [9294], "8754": [87671], "8755": [87672], "8756": [87673], "8757": [87674], "8758": [87675], "8759": [87676], "8760": [87677], "8761": [87678], "8762": [87679], "8763": [87680], "8764": [87681], "8765": [87682], "8766": [87683], "8767": [87684], "8768": [87685], "8769": [87686], "8770": [87687], "8771": [87688], "8772": [87689], "8773": [87690], "8774": [87691], "8775": [87692], "8776": [87693], "8777": [87694], "8778": [87695], "8779": [87696], "8780": [87697], "8781": [87698], "8782": [87699], "8783": [87700], "8784": [87701], "8785": [87702], "8786": [87703], "8787": [87704], "8788": [87705], "8789": [87706], "8790": [87707], "8791": [87708], "8792": [87709], "8793": [87710], "8794": [87711], "8795": [87712], "8796": [87713], "8797": [87714], "8798": [87715], "8799": [87716], "8800": [87717], "8801": [87718], "8802": [87719], "8803": [87720], "8804": [87721], "8805": [87722], "8806": [87723], "8807": [87724], "8808": [87725], "8809": [87726], "8810": [87727], "8811": [87728], "8812": [87729], "8813": [87730], "8814": [87731], "8815": [87732], "8816": [87733], "8817": [87734], "8818": [87735], "8819": [823], "8820": [87736], "8821": [87737], "8822": [87738], "8823": [87739], "8824": [87740], "8825": [87741], "8826": [87742], "8827": [87743], "8828": [87744], "8829": [87745], "8830": [87746], "8831": [87747], "8832": [87748], "8833": [87749], "8834": [87750], "8835": [87751], "8836": [87752], "8837": [87753], "8838": [87754], "8839": [87755], "8840": [87756], "8841": [87757], "8842": [87758], "8843": [87759], "8844": [87760], "8845": [87761], "8846": [87762], "8847": [87763], "8848": [87764], "8849": [87765], "8850": [87766], "8851": [87767], "8852": [87768], "8853": [87769], "8854": [87770], "8855": [87771], "8856": [87772], "8857": [87773], "8858": [87774], "8859": [87775], "8860": [87776], "8861": [87777], "8862": [87778], "8863": [87779], "8864": [87780], "8865": [87781], "8866": [87782], "8867": [49740], "8868": [87783], "8869": [87784], "8870": [87785], "8871": [87786], "8872": [87787], "8873": [87788], "8874": [87789], "8875": [87790], "8876": [87791], "8877": [87792], "8878": [87793], "8879": [87794], "8880": [87795], "8881": [3647], "8882": [87796], "8883": [87797], "8884": [87798], "8885": [87799], "8886": [87800], "8887": [33838], "8888": [87801], "8889": [15093], "8890": [4366], "8891": [87802], "8892": [87803], "8893": [87804], "8894": [87805], "8895": [87806], "8896": [87807], "8897": [87808], "8898": [87809], "8899": [87810], "8900": [87811], "8901": [87812], "8902": [87813], "8903": [87814], "8904": [87815], "8905": [87816], "8906": [87817], "8907": [87818], "8908": [87819], "8909": [87820], "8910": [87821], "8911": [87822], "8912": [87823], "8913": [87824], "8914": [87825], "8915": [87826], "8916": [87827], "8917": [87828], "8918": [87829], "8919": [87830], "8920": [87831], "8921": [87832], "8922": [87833], "8923": [87834], "8924": [87835], "8925": [6510], "8926": [87836], "8927": [87837], "8928": [87838], "8929": [87839], "8930": [87840], "8931": [87841], "8932": [87842], "8933": [87843], "8934": [87844], "8935": [87845], "8936": [87846], "8937": [87847], "8938": [87848], "8939": [87849], "8940": [87850], "8941": [87851], "8942": [87852], "8943": [11037], "8944": [49683], "8945": [87853], "8946": [87854], "8947": [87855], "8948": [87856], "8949": [49290], "8950": [87857], "8951": [87858], "8952": [87859], "8953": [87860], "8954": [87861], "8955": [87862], "8956": [87863], "8957": [87864], "8958": [87865], "8959": [87866], "8960": [87867], "8961": [87868], "8962": [87869], "8963": [87870], "8964": [87871], "8965": [87872], "8966": [87873], "8967": [87874], "8968": [87875], "8969": [87876], "8970": [9408], "8971": [87877], "8972": [262], "8973": [37302], "8974": [87878], "8975": [87879], "8976": [87880], "8977": [87881], "8978": [87882], "8979": [87883], "8980": [87884], "8981": [87885], "8982": [87886], "8983": [87887], "8984": [87888], "8985": [87889], "8986": [87890], "8987": [87891], "8988": [87892], "8989": [87893], "8990": [87894], "8991": [87895], "8992": [87896], "8993": [87897], "8994": [87898], "8995": [87899], "8996": [87900], "8997": [87901], "8998": [87902], "8999": [87903], "9000": [87904], "9001": [87905], "9002": [87906], "9003": [87907], "9004": [87908], "9005": [87909], "9006": [87910], "9007": [87911], "9008": [87912], "9009": [87913], "9010": [87914], "9011": [87915], "9012": [87916], "9013": [87917], "9014": [87918], "9015": [87919], "9016": [87920], "9017": [87921], "9018": [87922], "9019": [87923], "9020": [87924], "9021": [87925], "9022": [87926], "9023": [87927], "9024": [87928], "9025": [87929], "9026": [87930], "9027": [87931], "9028": [87932], "9029": [87933], "9030": [87934], "9031": [87935], "9032": [87936], "9033": [87937], "9034": [87938], "9035": [87939], "9036": [87940], "9037": [87941], "9038": [87942], "9039": [87943], "9040": [87944], "9041": [87945], "9042": [1799], "9043": [87946], "9044": [87947], "9045": [87948], "9046": [87949], "9047": [87950], "9048": [87951], "9049": [87952], "9050": [87953], "9051": [87954], "9052": [87955], "9053": [87956], "9054": [87957], "9055": [87958], "9056": [87959], "9057": [13477], "9058": [10953], "9059": [87960], "9060": [87961], "9061": [87962], "9062": [87963], "9063": [87964], "9064": [6741], "9065": [87965], "9066": [87966], "9067": [87967], "9068": [87968], "9069": [87969], "9070": [87970], "9071": [87971], "9072": [87972], "9073": [87973], "9074": [87974], "9075": [87975], "9076": [87976], "9077": [87977], "9078": [87978], "9079": [87979], "9080": [87980], "9081": [87981], "9082": [87982], "9083": [87983], "9084": [87984], "9085": [87985], "9086": [87986], "9087": [87987], "9088": [87988], "9089": [9003], "9090": [87989], "9091": [87990], "9092": [87991], "9093": [87992], "9094": [87993], "9095": [87994], "9096": [87995], "9097": [87996], "9098": [87997], "9099": [87998], "9100": [87999], "9101": [88000], "9102": [88001], "9103": [88002], "9104": [88003], "9105": [88004], "9106": [88005], "9107": [88006], "9108": [88007], "9109": [88008], "9110": [88009], "9111": [88010], "9112": [88011], "9113": [88012], "9114": [88013], "9115": [88014], "9116": [88015], "9117": [88016], "9118": [88017], "9119": [88018], "9120": [88019], "9121": [88020], "9122": [88021], "9123": [88022], "9124": [88023], "9125": [88024], "9126": [88025], "9127": [88026], "9128": [88027], "9129": [88028], "9130": [88029], "9131": [88030], "9132": [12154], "9133": [88031], "9134": [88032], "9135": [88033], "9136": [88034], "9137": [88035], "9138": [88036], "9139": [88037], "9140": [88038], "9141": [88039], "9142": [88040], "9143": [88041], "9144": [88042], "9145": [88043], "9146": [88044], "9147": [88045], "9148": [88046], "9149": [88047], "9150": [88048], "9151": [88049], "9152": [88050], "9153": [88051], "9154": [88052], "9155": [88053], "9156": [88054], "9157": [88055], "9158": [88056], "9159": [88057], "9160": [88058], "9161": [88059], "9162": [88060], "9163": [88061], "9164": [88062], "9165": [88063], "9166": [88064], "9167": [88065], "9168": [88066], "9169": [88067], "9170": [88068], "9171": [88069], "9172": [88070], "9173": [88071], "9174": [88072], "9175": [88073], "9176": [88074], "9177": [88075], "9178": [88076], "9179": [88077], "9180": [88078], "9181": [88079], "9182": [88080], "9183": [88081], "9184": [88082], "9185": [88083], "9186": [88084], "9187": [88085], "9188": [88086], "9189": [88087], "9190": [88088], "9191": [88089], "9192": [88090], "9193": [88091], "9194": [88092], "9195": [88093], "9196": [88094], "9197": [88096], "9198": [88097], "9199": [88098], "9200": [88099], "9201": [88100], "9202": [88101], "9203": [88102], "9204": [88103], "9205": [88104], "9206": [88105], "9207": [88106], "9208": [88107], "9209": [88108], "9210": [88109], "9211": [88110], "9212": [88111], "9213": [88112], "9214": [88113], "9215": [88114], "9216": [25654], "9217": [88115], "9218": [88116], "9219": [88117], "9220": [88118], "9221": [88119], "9222": [88120], "9223": [88121], "9224": [88122], "9225": [88123], "9226": [88124], "9227": [88125], "9228": [88126], "9229": [88127], "9230": [88128], "9231": [88129], "9232": [88130], "9233": [88131], "9234": [88132], "9235": [88133], "9236": [88134], "9237": [88135], "9238": [88136], "9239": [88137], "9240": [88138], "9241": [88139], "9242": [88140], "9243": [88141], "9244": [88142], "9245": [88143], "9246": [88144], "9247": [88145], "9248": [88146], "9249": [88147], "9250": [88148], "9251": [88149], "9252": [88150], "9253": [88151], "9254": [88152], "9255": [2511], "9256": [88153], "9257": [88154], "9258": [88155], "9259": [88156], "9260": [88157], "9261": [88158], "9262": [88159], "9263": [88160], "9264": [88161], "9265": [88162], "9266": [88163], "9267": [88164], "9268": [88165], "9269": [88166], "9270": [88167], "9271": [88168], "9272": [88169], "9273": [88170], "9274": [88171], "9275": [88172], "9276": [88173], "9277": [88174], "9278": [88175], "9279": [88176], "9280": [88177], "9281": [88178], "9282": [88179], "9283": [88180], "9284": [88181], "9285": [88182], "9286": [88183], "9287": [88184], "9288": [88185], "9289": [88186], "9290": [88187], "9291": [88188], "9292": [88189], "9293": [88190], "9294": [88191], "9295": [88192], "9296": [88193], "9297": [88194], "9298": [88195], "9299": [88196], "9300": [88197], "9301": [88198], "9302": [88199], "9303": [88200], "9304": [88201], "9305": [88202], "9306": [88203], "9307": [88204], "9308": [88205], "9309": [88206], "9310": [88207], "9311": [88208], "9312": [88209], "9313": [88210], "9314": [88211], "9315": [88212], "9316": [88213], "9317": [88214], "9318": [88215], "9319": [88216], "9320": [88217], "9321": [88218], "9322": [88219], "9323": [88220], "9324": [88221], "9325": [88222], "9326": [88223], "9327": [88224], "9328": [88225], "9329": [88226], "9330": [88227], "9331": [88228], "9332": [88229], "9333": [88230], "9334": [88231], "9335": [88232], "9336": [88233], "9337": [88234], "9338": [88235], "9339": [88236], "9340": [88237], "9341": [88238], "9342": [88239], "9343": [88240], "9344": [88241], "9345": [88242], "9346": [88243], "9347": [88244], "9348": [88245], "9349": [88246], "9350": [88247], "9351": [88248], "9352": [88249], "9353": [88250], "9354": [88251], "9355": [88252], "9356": [88253], "9357": [88254], "9358": [88255], "9359": [88256], "9360": [81767], "9361": [88257], "9362": [88258], "9363": [88259], "9364": [88260], "9365": [88261], "9366": [88262], "9367": [88263], "9368": [88264], "9369": [88265], "9370": [88266], "9371": [88267], "9372": [88268], "9373": [88269], "9374": [88270], "9375": [88271], "9376": [88272], "9377": [88273], "9378": [88274], "9379": [88275], "9380": [88276], "9381": [88277], "9382": [88278], "9383": [88279], "9384": [88280], "9385": [88281], "9386": [88282], "9387": [88283], "9388": [88284], "9389": [88285], "9390": [88286], "9391": [88287], "9392": [88288], "9393": [88289], "9394": [88290], "9395": [88291], "9396": [88292], "9397": [88293], "9398": [88294], "9399": [88295], "9400": [88296], "9401": [88297], "9402": [88298], "9403": [88299], "9404": [88300], "9405": [88301], "9406": [88302], "9407": [88303], "9408": [88304], "9409": [88305], "9410": [88306], "9411": [88307], "9412": [88308], "9413": [88309], "9414": [88310], "9415": [88311], "9416": [88312], "9417": [88313], "9418": [88314], "9419": [88315], "9420": [88316], "9421": [88317], "9422": [88318], "9423": [88319], "9424": [88320], "9425": [88321], "9426": [88322], "9427": [88323], "9428": [88324], "9429": [88325], "9430": [88326], "9431": [1435], "9432": [88327], "9433": [88328], "9434": [88329], "9435": [88330], "9436": [88331], "9437": [88332], "9438": [88333], "9439": [88334], "9440": [88335], "9441": [88336], "9442": [88337], "9443": [88338], "9444": [88339], "9445": [88340], "9446": [88341], "9447": [88342], "9448": [88343], "9449": [88344], "9450": [88345], "9451": [88346], "9452": [88347], "9453": [88348], "9454": [88349], "9455": [88350], "9456": [88351], "9457": [88352], "9458": [88353], "9459": [88354], "9460": [88355], "9461": [88356], "9462": [88357], "9463": [88358], "9464": [88359], "9465": [88360], "9466": [88361], "9467": [88362], "9468": [88363], "9469": [88364], "9470": [88365], "9471": [88366], "9472": [88367], "9473": [88368], "9474": [88369], "9475": [88370], "9476": [88371], "9477": [88372], "9478": [88373], "9479": [88374], "9480": [88375], "9481": [88376], "9482": [88377], "9483": [88378], "9484": [88379], "9485": [88380], "9486": [88381], "9487": [88382], "9488": [88383], "9489": [88384], "9490": [88385], "9491": [88386], "9492": [88387], "9493": [88388], "9494": [88389], "9495": [88390], "9496": [88391], "9497": [88392], "9498": [88393], "9499": [88394], "9500": [88395], "9501": [88396], "9502": [88397], "9503": [88398], "9504": [88399], "9505": [88400], "9506": [88401], "9507": [88402], "9508": [88403], "9509": [88404], "9510": [88405], "9511": [88406], "9512": [88407], "9513": [88408], "9514": [88409], "9515": [88410], "9516": [88411], "9517": [88412], "9518": [88413], "9519": [88414], "9520": [88415], "9521": [88416], "9522": [88417], "9523": [88418], "9524": [88419], "9525": [88420], "9526": [88421], "9527": [88422], "9528": [88423], "9529": [88424], "9530": [88425], "9531": [88426], "9532": [88427], "9533": [88428], "9534": [88429], "9535": [88430], "9536": [88431], "9537": [88432], "9538": [88433], "9539": [88434], "9540": [88435], "9541": [88436], "9542": [12553], "9543": [88437], "9544": [88438], "9545": [88439], "9546": [88440], "9547": [88441], "9548": [88442], "9549": [88443], "9550": [88444], "9551": [88445], "9552": [88446], "9553": [88447], "9554": [88448], "9555": [88449], "9556": [88450], "9557": [88451], "9558": [88452], "9559": [88453], "9560": [88454], "9561": [88455], "9562": [88456], "9563": [88457], "9564": [88458], "9565": [88459], "9566": [88460], "9567": [88461], "9568": [88462], "9569": [88463], "9570": [88464], "9571": [88465], "9572": [88466], "9573": [88467], "9574": [88468], "9575": [88469], "9576": [88470], "9577": [88471], "9578": [411], "9579": [88472], "9580": [88473], "9581": [88474], "9582": [88475], "9583": [88476], "9584": [88477], "9585": [88478], "9586": [88479], "9587": [88480], "9588": [88481], "9589": [88482], "9590": [88483], "9591": [88484], "9592": [88485], "9593": [88486], "9594": [88487], "9595": [88488], "9596": [88489], "9597": [88490], "9598": [88491], "9599": [88492], "9600": [88493], "9601": [88494], "9602": [88495], "9603": [173], "9604": [88496], "9605": [88497], "9606": [88498], "9607": [88499], "9608": [753], "9609": [88500], "9610": [88501], "9611": [88502], "9612": [88503], "9613": [88504], "9614": [88505], "9615": [88506], "9616": [88507], "9617": [88508], "9618": [88509], "9619": [88510], "9620": [88511], "9621": [88512], "9622": [88513], "9623": [88514], "9624": [88515], "9625": [88516], "9626": [88517], "9627": [88518], "9628": [88519], "9629": [88520], "9630": [25654], "9631": [88521], "9632": [88522], "9633": [88523], "9634": [88524], "9635": [88525], "9636": [88526], "9637": [88527], "9638": [88528], "9639": [88529], "9640": [88530], "9641": [88531], "9642": [88532], "9643": [88533], "9644": [88534], "9645": [88535], "9646": [88536], "9647": [88537], "9648": [88538], "9649": [88539], "9650": [88540], "9651": [88541], "9652": [88542], "9653": [88543], "9654": [88544], "9655": [88545], "9656": [88546], "9657": [88547], "9658": [88548], "9659": [88549], "9660": [88550], "9661": [88551], "9662": [88552], "9663": [88553], "9664": [88554], "9665": [88555], "9666": [88556], "9667": [88557], "9668": [88558], "9669": [88559], "9670": [88560], "9671": [88561], "9672": [88562], "9673": [88563], "9674": [88564], "9675": [88565], "9676": [88566], "9677": [88567], "9678": [88568], "9679": [88569], "9680": [88570], "9681": [1381], "9682": [88571], "9683": [88572], "9684": [88573], "9685": [88574], "9686": [88575], "9687": [3032], "9688": [4692], "9689": [88576], "9690": [88577], "9691": [88578], "9692": [88579], "9693": [88580], "9694": [88581], "9695": [88582], "9696": [88583], "9697": [88584], "9698": [88585], "9699": [88586], "9700": [88587], "9701": [88588], "9702": [88589], "9703": [88590], "9704": [88591], "9705": [88592], "9706": [17773], "9707": [88593], "9708": [88594], "9709": [88595], "9710": [88596], "9711": [88597], "9712": [88598], "9713": [88599], "9714": [88600], "9715": [88601], "9716": [88602], "9717": [88603], "9718": [88604], "9719": [88605], "9720": [88606], "9721": [88607], "9722": [88608], "9723": [88609], "9724": [88610], "9725": [88611], "9726": [88612], "9727": [88613], "9728": [88614], "9729": [88615], "9730": [88616], "9731": [88617], "9732": [88618], "9733": [88619], "9734": [88620], "9735": [88621], "9736": [88622], "9737": [88623], "9738": [88624], "9739": [88625], "9740": [88626], "9741": [88627], "9742": [88628], "9743": [88629], "9744": [88630], "9745": [88631], "9746": [88632], "9747": [88633], "9748": [88634], "9749": [88635], "9750": [86487], "9751": [88636], "9752": [88637], "9753": [88638], "9754": [88639], "9755": [88640], "9756": [88641], "9757": [88642], "9758": [88643], "9759": [88644], "9760": [88645], "9761": [88647], "9762": [88648], "9763": [88649], "9764": [88650], "9765": [88651], "9766": [88652], "9767": [88653], "9768": [88654], "9769": [88655], "9770": [88656], "9771": [88657], "9772": [88658], "9773": [88659], "9774": [88660], "9775": [88661], "9776": [88662], "9777": [88663], "9778": [88664], "9779": [88665], "9780": [88666], "9781": [88667], "9782": [88668], "9783": [88669], "9784": [88670], "9785": [88671], "9786": [88672], "9787": [88673], "9788": [88674], "9789": [88675], "9790": [88676], "9791": [88677], "9792": [88678], "9793": [88679], "9794": [88680], "9795": [88681], "9796": [1965], "9797": [88682], "9798": [88683], "9799": [88684], "9800": [88685], "9801": [88686], "9802": [88687], "9803": [88688], "9804": [88689], "9805": [88690], "9806": [88691], "9807": [88692], "9808": [88693], "9809": [88694], "9810": [88695], "9811": [88696], "9812": [88697], "9813": [88698], "9814": [88699], "9815": [88700], "9816": [42291], "9817": [88701], "9818": [88702], "9819": [88703], "9820": [88704], "9821": [88705], "9822": [88706], "9823": [88707], "9824": [88708], "9825": [88709], "9826": [88710], "9827": [88711], "9828": [88712], "9829": [88713], "9830": [84216], "9831": [88714], "9832": [88715], "9833": [88716], "9834": [88717], "9835": [88718], "9836": [88719], "9837": [88720], "9838": [88721], "9839": [88722], "9840": [88723], "9841": [88724], "9842": [88725], "9843": [88726], "9844": [88727], "9845": [88728], "9846": [88729], "9847": [88730], "9848": [88731], "9849": [88732], "9850": [88733], "9851": [88734], "9852": [88735], "9853": [88736], "9854": [88737], "9855": [88738], "9856": [88739], "9857": [88740], "9858": [88741], "9859": [88742], "9860": [88743], "9861": [88744], "9862": [88745], "9863": [88746], "9864": [88747], "9865": [88748], "9866": [88749], "9867": [88750], "9868": [88751], "9869": [88752], "9870": [88753], "9871": [88754], "9872": [88755], "9873": [88756], "9874": [88757], "9875": [88758], "9876": [88759], "9877": [88760], "9878": [88761], "9879": [88762], "9880": [88763], "9881": [88764], "9882": [88765], "9883": [88766], "9884": [88767], "9885": [88768], "9886": [88769], "9887": [88770], "9888": [88771], "9889": [88772], "9890": [88773]} -------------------------------------------------------------------------------- /dataloader.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | import json 5 | import logging 6 | import os 7 | import random 8 | from concurrent.futures import ThreadPoolExecutor 9 | from queue import Queue 10 | from typing import Any, Callable, Dict, List, Tuple 11 | from dataclasses import dataclass 12 | 13 | import numpy as np 14 | import torch 15 | from filelock import FileLock 16 | from argparse import Namespace 17 | from torch.nn.utils.rnn import pad_sequence 18 | from torch.utils.data.dataset import IterableDataset 19 | import torch.distributed as dist 20 | from transformers import BertTokenizerFast 21 | from transformers.utils import logging 22 | from tqdm import tqdm 23 | from random import shuffle 24 | 25 | from utils.utils import setuplogging 26 | 27 | logger = logging.get_logger(__name__) 28 | 29 | 30 | class DatasetForMatching(IterableDataset): 31 | def __init__(self, 32 | file_path: str): 33 | self.data_file = open(file_path, 34 | "r", 35 | encoding="utf-8") 36 | 37 | def __iter__(self): 38 | for line in self.data_file: 39 | tokens_and_vecs = json.loads(line) 40 | yield tokens_and_vecs 41 | 42 | 43 | # @dataclass 44 | class DataCollatorForMatching: 45 | def __init__(self, tokenizer: BertTokenizerFast, 46 | args: Namespace): 47 | self.tokenizer = tokenizer 48 | self.args = args 49 | 50 | def __call__( 51 | self, 52 | samples: List[Dict]) -> Dict[str, torch.Tensor]: 53 | input_id_queries = [] 54 | attention_mask_queries = [] 55 | input_id_keys = [] 56 | attention_mask_keys = [] 57 | queries_vec = [] 58 | keys_vec = [] 59 | for i, sample in (enumerate(samples)): 60 | (input_id_query, 61 | attention_mask_query, 62 | input_id_key, 63 | attention_mask_key, 64 | query_vec, 65 | key_vec) = self.create_training_sample(sample) 66 | 67 | if input_id_query is not None: 68 | input_id_queries.append(input_id_query) 69 | attention_mask_queries.append(attention_mask_query) 70 | else: 71 | queries_vec.append(query_vec) 72 | 73 | if input_id_key is not None: 74 | input_id_keys.append(input_id_key) 75 | attention_mask_keys.append(attention_mask_key) 76 | else: 77 | keys_vec.append(key_vec) 78 | 79 | if len(queries_vec) == 0: 80 | input_id_queries = self._tensorize_batch( 81 | input_id_queries, self.tokenizer.pad_token_id).long() 82 | attention_mask_queries = self._tensorize_batch(attention_mask_queries, 0).float() 83 | queries_vec = None 84 | else: 85 | # print(queries_vec) 86 | queries_vec = torch.FloatTensor(queries_vec) 87 | input_id_queries, attention_mask_queries = None, None 88 | 89 | if len(keys_vec) == 0: 90 | input_id_keys = self._tensorize_batch(input_id_keys, 91 | self.tokenizer.pad_token_id).long() 92 | attention_mask_keys = self._tensorize_batch(attention_mask_keys, 0).float() 93 | keys_vec = None 94 | else: 95 | keys_vec = torch.FloatTensor(keys_vec) 96 | input_id_keys, attention_mask_keys = None, None 97 | 98 | return { 99 | "input_id_query": 100 | input_id_queries, 101 | "attention_mask_query": 102 | attention_mask_queries, 103 | "input_id_key": 104 | input_id_keys, 105 | "attention_mask_key": 106 | attention_mask_keys, 107 | "queries_vec": 108 | queries_vec, 109 | "keys_vec": 110 | keys_vec 111 | } 112 | 113 | def _tensorize_batch(self, examples: List[torch.Tensor], 114 | padding_value) -> torch.Tensor: 115 | length_of_first = examples[0].size(0) 116 | are_tensors_same_length = all( 117 | x.size(0) == length_of_first for x in examples) 118 | if are_tensors_same_length: 119 | return torch.stack(examples, dim=0) 120 | else: 121 | return pad_sequence(examples, 122 | batch_first=True, 123 | padding_value=padding_value) 124 | 125 | def create_training_sample(self, sample: Dict): 126 | """Turn sample into train pair 127 | 128 | Args: 129 | sample (Dict): {"query_tokens":List[int], "query_vec":List[float], "key_tokens":List[int], "key_vec":List[float]} 130 | 131 | """ 132 | input_id_queries, attention_mask_queries, input_id_keys, attention_mask_keys, query_vec, key_vec = None, None, None, None, None, None 133 | if self.args.self_supervised: 134 | assert 'key_tokens' in sample 135 | input_id_keys, attention_mask_keys = self.creat_tokens_sample(sample['key_tokens'], self.args.key_max_token) 136 | token_queries = self.creat_query_from_key(sample['key_tokens']) 137 | input_id_queries, attention_mask_queries = self.creat_tokens_sample(token_queries, self.args.query_max_token) 138 | return input_id_queries, attention_mask_queries, input_id_keys, attention_mask_keys, query_vec, key_vec 139 | 140 | if 'query_tokens' in sample: 141 | input_id_queries, attention_mask_queries = self.creat_tokens_sample(sample['query_tokens'], 142 | self.args.query_max_token) 143 | else: 144 | query_vec = self.creat_vecs_sample(sample['query_vec']) 145 | 146 | if 'key_tokens' in sample: 147 | input_id_keys, attention_mask_keys = self.creat_tokens_sample(sample['key_tokens'], 148 | self.args.query_max_token) 149 | else: 150 | key_vec = self.creat_vecs_sample(sample['key_vec']) 151 | 152 | return input_id_queries, attention_mask_queries, input_id_keys, attention_mask_keys, query_vec, key_vec 153 | 154 | def creat_query_from_key(self, token_keys): 155 | if len(token_keys) > 1: 156 | if self.args.augment_method == 'drop': 157 | token_queries = self.drop_tokens(token_keys) 158 | elif self.args.augment_method == 'replace': 159 | token_queries = self.replace_tokens(token_keys) 160 | elif self.args.augment_method == 'add': 161 | token_queries = self.add_tokens(token_keys) 162 | elif self.args.augment_method == 'all': 163 | rand = random.random() 164 | if rand < 0.33: 165 | token_queries = self.add_tokens(token_keys) 166 | elif rand < 0.66: 167 | token_queries = self.replace_tokens(token_keys) 168 | else: 169 | token_queries = self.drop_tokens(token_keys) 170 | return token_queries 171 | 172 | def creat_tokens_sample(self, tokens_list, max_tokens_num): 173 | input_ids= torch.tensor( 174 | self.tokenizer.build_inputs_with_special_tokens( 175 | tokens_list[:max_tokens_num])) 176 | attention_mask = torch.tensor([1] * len(input_ids)) 177 | return input_ids, attention_mask 178 | 179 | 180 | def creat_vecs_sample(self, vector): 181 | return vector 182 | 183 | 184 | def drop_tokens(self, input_ids): 185 | drop_num = 1 186 | 187 | l = len(input_ids) 188 | final_tokens = random.sample(range(l),l-drop_num) 189 | temp = [] 190 | for inx, t in enumerate(input_ids): 191 | if inx in final_tokens: 192 | temp.append(t) 193 | return temp 194 | 195 | def replace_tokens(self, input_ids): 196 | replace_num = 1 197 | 198 | raplace_inx = set() 199 | for i in range(replace_num): 200 | inx = np.random.randint(0, len(input_ids)-1) 201 | while inx in raplace_inx: 202 | inx = np.random.randint(0, len(input_ids) - 1) 203 | raplace_inx.add(inx) 204 | input_ids[inx] = np.random.randint(0, self.tokenizer.vocab_size - 1) 205 | return input_ids 206 | 207 | def add_tokens(self, input_ids): 208 | final_tokens = random.sample(range(self.tokenizer.vocab_size), 1) 209 | rand = random.random() 210 | if rand<0.5: 211 | final_tokens.extend(input_ids) 212 | return final_tokens 213 | else: 214 | input_ids.extend(final_tokens) 215 | return input_ids 216 | 217 | 218 | 219 | 220 | # @dataclass 221 | class SingleProcessDataLoaderForMatching: 222 | # dataset: IterableDataset 223 | # batch_size: int 224 | # collate_fn: Callable 225 | # drop_last: bool = True 226 | def __init__(self, 227 | dataset: IterableDataset, 228 | batch_size: int, 229 | collate_fn: Callable, 230 | drop_last: bool = True, 231 | debug = False): 232 | self.dataset = dataset 233 | self.batch_size = batch_size 234 | self.collate_fn = collate_fn 235 | self.drop_last = drop_last 236 | self.debug = debug 237 | 238 | def set_end(self, value): 239 | self.end = value 240 | 241 | def if_end(self,): 242 | return self.end 243 | 244 | def skip_sample(self, i, sample): 245 | return False 246 | 247 | def _start(self): 248 | self.set_end(False) 249 | self.aval_count = 0 250 | self.outputs = Queue(10) 251 | self.pool = ThreadPoolExecutor(1) 252 | self.pool.submit(self._produce) 253 | 254 | def _produce(self): 255 | for batch in self._generate_batch(): 256 | if self.if_end(): 257 | break 258 | self.outputs.put(batch) 259 | self.aval_count += 1 260 | self.pool.shutdown(wait=False) 261 | raise 262 | 263 | def _generate_batch(self): 264 | batch = [] 265 | for i, sample in enumerate(self.dataset): 266 | if self.skip_sample(i, sample): continue 267 | batch.append(sample) 268 | if len(batch) >= self.batch_size: 269 | yield self.collate_fn(batch[:self.batch_size]) 270 | batch = batch[self.batch_size:] 271 | else: 272 | if len(batch) > 0 and not self.drop_last: 273 | yield self.collate_fn(batch) 274 | batch = [] 275 | self.set_end(True) 276 | 277 | def __iter__(self): 278 | if self.debug: 279 | return self._generate_batch() 280 | self._start() 281 | return self 282 | 283 | def __next__(self): 284 | if self.aval_count == 0 and self.if_end(): 285 | raise StopIteration 286 | next_batch = self.outputs.get() 287 | self.outputs.task_done() 288 | self.aval_count -= 1 289 | return next_batch 290 | 291 | 292 | @dataclass 293 | class MultiProcessDataLoaderForMatching: 294 | dataset: IterableDataset 295 | batch_size: int 296 | collate_fn: Callable 297 | local_rank: int 298 | world_size: int 299 | global_end: Any 300 | buffer_num:int 301 | blocking: bool=False 302 | drop_last: bool = True 303 | 304 | def _start(self): 305 | self.local_end=False 306 | self.global_end.value = False 307 | self.aval_count = 0 308 | self.outputs = Queue(10) 309 | self.pool = ThreadPoolExecutor(1) 310 | self.pool.submit(self._produce) 311 | 312 | def _produce(self): 313 | try: 314 | for batch in self._generate_buffer_batch(): 315 | self.outputs.put(batch) 316 | self.aval_count += 1 317 | self.pool.shutdown(wait=False) 318 | except: 319 | import sys 320 | import traceback 321 | import logging 322 | error_type, error_value, error_trace = sys.exc_info() 323 | traceback.print_tb(error_trace) 324 | logging.info(error_value) 325 | self.pool.shutdown(wait=False) 326 | raise 327 | 328 | def _generate_buffer_batch(self): 329 | """buffer batches, and shuffling insider batchs to produce next. 330 | """ 331 | buffer = [] 332 | for i, sample in enumerate(self.dataset): 333 | if i % self.world_size != self.local_rank: continue 334 | buffer.append(sample) 335 | if len(buffer) >= self.batch_size * self.buffer_num: 336 | buffer, next_batch = self.buffer_shuffle(buffer) 337 | yield self.collate_fn(next_batch) 338 | else: 339 | if len(buffer) > 0: 340 | while len(buffer) >= self.batch_size: 341 | buffer, next_batch = self.buffer_shuffle(buffer) 342 | yield self.collate_fn(next_batch) 343 | if len(buffer) > 0 and not self.drop_last: 344 | yield self.collate_fn(buffer) 345 | self.local_end=True 346 | 347 | def buffer_shuffle(self, batchs): 348 | shuffle(batchs) 349 | return batchs[:-self.batch_size], batchs[-self.batch_size:] 350 | 351 | def __iter__(self): 352 | if self.blocking: 353 | return self._generate_buffer_batch() 354 | self._start() 355 | return self 356 | 357 | def __next__(self): 358 | dist.barrier() 359 | while self.aval_count == 0: 360 | if self.local_end or self.global_end.value: 361 | self.global_end.value=True 362 | break 363 | dist.barrier() 364 | if self.global_end.value: 365 | raise StopIteration 366 | next_batch = self.outputs.get() 367 | self.aval_count -= 1 368 | return next_batch 369 | 370 | 371 | # 372 | -------------------------------------------------------------------------------- /models/Differentiable_PQ.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | import torch 5 | import torch.nn as nn 6 | import torch.nn.functional as F 7 | from models.TextEncoder import TextEncoder 8 | 9 | class DPQ(TextEncoder): 10 | ''' 11 | This is the implementation for differentiable product quantization models (e.g., SPQ, DQN, DVSQ) 12 | ''' 13 | def __init__(self, args, query_bert, key_bert=None, hidden_size=768): 14 | super().__init__(query_bert, key_bert, hidden_size, args.output_hidden_size) 15 | 16 | self.partition = args.partition_num 17 | self.centroids = args.centroids_num 18 | self.select_mode = args.select_mode 19 | self.output_hidden_size = args.output_hidden_size 20 | self.sub_dim = args.output_hidden_size//self.partition 21 | self.quantization_loss_weight = args.quantization_loss_weight 22 | self.quantization_loss_list = args.quantization_loss 23 | self.model_type = args.model_type 24 | self.cross_device = args.cross_device 25 | 26 | if self.select_mode == 'mlp': 27 | self.prob_w = nn.Linear(args.output_hidden_size, 28 | self.partition*self.centroids) 29 | elif self.select_mode == 'sub_mlp': 30 | self.prob_w2 = nn.Linear(args.output_hidden_size//self.partition, 31 | self.centroids) 32 | elif self.select_mode == 'mix_w': 33 | self.prob_w3 = nn.Linear(self.sub_dim*2, 1, bias=False) 34 | elif self.select_mode == 'dpq': 35 | self.matrix_k = nn.Parameter(torch.empty(self.partition, self.centroids, self.sub_dim).uniform_(-1, 1)).type(torch.FloatTensor) #P K D 36 | 37 | self.codebook = nn.Parameter(torch.empty(self.partition, self.centroids, self.sub_dim).uniform_(-1, 1)).type(torch.FloatTensor) #nn.Parameter default requires_grad=True 38 | 39 | 40 | def select_codeword(self, vecs, softmax=True): 41 | if self.select_mode == 'mlp': 42 | vecs = F.relu(self.prob_w(vecs)) 43 | assignment = vecs.view(vecs.size(0), self.partition, -1) # B P K 44 | 45 | elif self.select_mode == 'sub_mlp': 46 | vecs = vecs.view(vecs.size(0), self.partition, -1) # B P D 47 | assignment = F.relu(self.prob_w2(vecs)) # B P K 48 | 49 | elif self.select_mode == 'dpq': 50 | #the selection is used by DPQ model 51 | batch_size = vecs.size(0) 52 | vecs = vecs.view(batch_size, self.partition, -1).unsqueeze(-1) #B P D 1 53 | reshape_k = self.matrix_k.unsqueeze(0).expand(batch_size, -1, -1, -1) # B P K D 54 | assignment = torch.matmul(reshape_k, vecs).squeeze(-1) # B P K 55 | 56 | elif self.select_mode == 'ip': 57 | vecs = vecs.view(vecs.size(0), self.partition, -1).unsqueeze(-1) # B P D 1 58 | codebook = self.codebook.unsqueeze(0).expand(vecs.size(0), -1, -1, -1) # B P K D 59 | assignment = torch.matmul(codebook, vecs).squeeze(-1) # B P K 60 | 61 | elif self.select_mode == 'l2': 62 | vecs = vecs.view(vecs.size(0), self.partition, -1) #B P D 63 | codebook = self.codebook.unsqueeze(0).expand(vecs.size(0), -1, -1, -1) # B P K D 64 | assignment = - torch.sum((vecs.unsqueeze(-2)-codebook)**2,-1) #B P K 65 | 66 | elif self.select_mode == 'mix_w': 67 | vecs = vecs.view(vecs.size(0), self.partition, -1) # B P D 68 | codebook = self.codebook.unsqueeze(0).expand(vecs.size(0), -1, -1, -1) # B P K D 69 | l2 = (vecs.unsqueeze(-2) - codebook) ** 2 # B P K D 70 | 71 | codebook = self.codebook.unsqueeze(0).expand(vecs.size(0), -1, -1, -1) # B P K D 72 | ip = torch.mul(vecs.unsqueeze(-2), codebook) #B P K D 73 | 74 | ip_l2 = torch.cat([ip,l2],-1) #B P K 2D 75 | assignment = self.prob_w3(ip_l2).squeeze(-1) #B P K 76 | 77 | elif self.select_mode == 'cosine': 78 | vecs = vecs.view(vecs.size(0), self.partition, -1) 79 | vecs = vecs.unsqueeze(-2).expand(-1,-1,self.centroids,-1) #B P K D 80 | codebook = self.codebook.unsqueeze(0).expand(vecs.size(0), -1, -1, -1) # B P K D 81 | assignment = torch.cosine_similarity(vecs,codebook,dim=-1) #B P K 82 | 83 | else: 84 | raise NotImplementedError 85 | 86 | if softmax: 87 | assignment = F.softmax(assignment, -1) 88 | return assignment 89 | 90 | def soft_vecs(self, assignment): 91 | # self.codebook #P K D 92 | assignment = assignment.unsqueeze(2) #B P 1 K 93 | codebook = self.codebook.unsqueeze(0).expand(assignment.size(0),-1,-1,-1) #B P K D 94 | soft_vecs = torch.matmul(assignment,codebook).squeeze(2) #B P D 95 | soft_vecs = soft_vecs.view(assignment.size(0),-1) #B L 96 | return soft_vecs 97 | 98 | def STEstimator(self, assignment): 99 | index = assignment.max(dim=-1, keepdim=True)[1] 100 | assignment_hard = torch.zeros_like(assignment,device=assignment.device,dtype=assignment.dtype).scatter_(-1, index, 1.0) 101 | return assignment_hard.detach() - assignment.detach() + assignment 102 | 103 | def hard_vecs(self, assignment): 104 | assignment = self.STEstimator(assignment) #B P K 105 | assignment = assignment.unsqueeze(2) # B P 1 K 106 | codebook = self.codebook.unsqueeze(0).expand(assignment.size(0), -1, -1, -1) # B P K D 107 | hard_vecs = torch.matmul(assignment, codebook).squeeze(2) # B P D 108 | hard_vecs = hard_vecs.view(assignment.size(0), -1) # B L 109 | return hard_vecs 110 | 111 | def get_soft_and_hard(self, vecs): 112 | prob = self.select_codeword(vecs) 113 | soft_vecs = self.soft_vecs(prob) 114 | hard_vecs = self.hard_vecs(prob) 115 | return prob,soft_vecs,hard_vecs 116 | 117 | def quant(self, vecs): 118 | assignment = self.select_codeword(vecs) 119 | hard_vecs = self.hard_vecs(assignment) 120 | return hard_vecs 121 | 122 | def sub_scores(self,q,k): 123 | batch_size = q.size(0) 124 | q = q.view(batch_size,self.partition,-1) #B P K 125 | k = k.view(batch_size,self.partition,-1) #B P K 126 | q = q.unsqueeze(1).expand(-1,batch_size,-1,-1) #B B P K 127 | k = k.unsqueeze(0).expand(batch_size,-1,-1,-1) #B B P K 128 | sub_score = torch.sum(torch.mul(q,k),dim=-1) #B B P 129 | return sub_score 130 | 131 | def JointCentralLoss(self,soft_vecs, hard_vecs): 132 | return torch.mean(torch.norm(soft_vecs-hard_vecs,dim=-1)) 133 | 134 | def quantization_loss(self, q, k, hard_k): 135 | # 'dot', 'subdot', 'l2' 136 | sub_score = self.sub_scores(q, k) 137 | score = torch.sum(sub_score, dim=-1) 138 | 139 | sub_qscore = self.sub_scores(q, hard_k) 140 | qscore = torch.sum(sub_qscore, dim=-1) 141 | 142 | loss = 0.0 143 | if 'dot' in self.quantization_loss_list: 144 | loss = loss + self.quantization_loss_weight*torch.mean((score-qscore)**2) 145 | if 'subdot' in self.quantization_loss_list: 146 | loss = loss + self.quantization_loss_weight*torch.mean((sub_score-sub_qscore)**2) 147 | if 'l2' in self.quantization_loss_list: 148 | loss = loss + self.quantization_loss_weight*torch.mean(torch.sum((k-hard_k)**2,dim=-1)) 149 | return loss 150 | 151 | def encode(self, input=None, mask=None, mode='hard', vecs=None): 152 | if mode == 'hard': 153 | if vecs is None: vecs = self.infer_k(input, mask) 154 | prob = self.select_codeword(vecs) 155 | codes = prob.max(dim=-1, keepdim=True)[1].squeeze(-1) 156 | return codes 157 | else: 158 | if vecs is None: vecs = self.infer_q(input, mask) 159 | if mode == 'soft': # This mode is used by SPQ 160 | prob = self.select_codeword(vecs) 161 | vecs = self.soft_vecs(prob) 162 | 163 | vecs_reshape = vecs.view(vecs.size(0), self.partition, -1) # B P D 164 | codebook = self.codebook.unsqueeze(0).expand(vecs_reshape.size(0), -1, -1, -1) # B P K D 165 | dtable = torch.matmul(codebook, vecs_reshape.unsqueeze(-1)).squeeze(-1) # B P K 166 | return dtable 167 | 168 | def forward(self, 169 | input_id_query, attention_mask_query, 170 | input_id_key, attention_mask_key): 171 | q = self.infer_q(input_id_query, attention_mask_query) 172 | k = self.infer_k(input_id_key, attention_mask_key) 173 | 174 | if self.model_type == 'spq': 175 | batch = input_id_query.size(0) 176 | vecs = torch.cat([q,k],0) 177 | prob, soft_vecs, hard_vecs = self.get_soft_and_hard(vecs) 178 | 179 | loss = self.retrieve_loss(soft_vecs[:batch], soft_vecs[batch:]) 180 | loss = loss + self.retrieve_loss(soft_vecs[:batch], hard_vecs[batch:]) 181 | loss = loss + self.quantization_loss_weight*self.JointCentralLoss(soft_vecs, hard_vecs) 182 | return loss 183 | 184 | else: 185 | loss = self.retrieve_loss(q,k) 186 | hard_k = self.quant(k) 187 | loss = loss + self.quantization_loss(q, k, hard_k) 188 | 189 | return loss 190 | 191 | -------------------------------------------------------------------------------- /models/MoPQ.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | from models.Differentiable_PQ import * 5 | import torch.distributed as dist 6 | 7 | class MoPQ(DPQ): 8 | def __init__(self, args, query_bert, key_bert=None, hidden_size=768, local_rank=-1): 9 | super().__init__(args, query_bert, key_bert, hidden_size) 10 | self.cross_device = args.cross_device 11 | self.world_size = args.world_size 12 | self.local_rank = local_rank 13 | 14 | def forward(self, 15 | input_id_query=None, attention_mask_query=None, 16 | input_id_key=None, attention_mask_key=None, 17 | queries_vec=None, keys_vec=None): 18 | 19 | q, k, hard_k = self.infer(input_id_query, attention_mask_query, 20 | input_id_key, attention_mask_key, 21 | queries_vec, keys_vec) 22 | if self.cross_device: 23 | q = self.gather_tensor(q) 24 | k = self.gather_tensor(k) 25 | hard_k = self.gather_tensor(hard_k) 26 | 27 | loss = self.retrieve_loss(q, k) 28 | loss = loss + self.retrieve_loss(q, hard_k) 29 | loss = loss + self.quantization_loss(q, k, hard_k) 30 | return loss 31 | 32 | def infer(self, input_id_query=None, attention_mask_query=None, 33 | input_id_key=None, attention_mask_key=None, 34 | queries_vec=None, keys_vec=None): 35 | if input_id_query is not None: 36 | q = self.infer_q(input_id_query, attention_mask_query) 37 | else: 38 | q = queries_vec 39 | 40 | if input_id_key is not None: 41 | k = self.infer_k(input_id_key, attention_mask_key) 42 | else: 43 | k = keys_vec 44 | hard_k = self.quant(k) 45 | return q,k,hard_k 46 | 47 | def gather_tensor(self, vecs): 48 | ''' 49 | broadcast vectors among all devices 50 | ''' 51 | all_tensors = [torch.empty_like(vecs) for _ in range(self.world_size)] 52 | dist.all_gather(all_tensors, vecs) 53 | all_tensors[self.local_rank] = vecs 54 | all_tensors = torch.cat(all_tensors, dim=0) 55 | return all_tensors 56 | 57 | 58 | 59 | -------------------------------------------------------------------------------- /models/TextEncoder.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | import torch 5 | import torch.nn as nn 6 | import torch.nn.functional as F 7 | 8 | class TextEncoder(nn.Module): 9 | def __init__(self, query_bert, key_bert=None, hidden_size=768, output_hidden_size=128): 10 | super().__init__() 11 | 12 | self.bert = query_bert 13 | self.last_linear = nn.Linear(hidden_size, output_hidden_size) 14 | self.last_f = nn.ELU() 15 | 16 | self.key_bert = None 17 | if key_bert is not None: 18 | self.key_bert = key_bert 19 | self.key_last_linear = nn.Linear(hidden_size, output_hidden_size) 20 | 21 | def retrieve_loss(self, q, k): 22 | score = torch.matmul(q, k.transpose(0, 1)) 23 | labels = torch.arange(start=0, end=score.shape[0], 24 | dtype=torch.long, device=score.device) 25 | loss = F.cross_entropy(score,labels) 26 | return loss 27 | 28 | def pooling(self, ents, masks): 29 | valid_len = torch.sum(masks, dim=1, keepdim=True) 30 | new_masks = masks[..., None] 31 | masks_dim = new_masks.repeat(1, 1, ents.shape[-1]) 32 | ents = ents*masks_dim 33 | pool_ents = torch.sum(ents, dim=1) 34 | averag_ents = pool_ents/valid_len 35 | return averag_ents 36 | 37 | def infer_q( 38 | self, 39 | input_id_query, attention_mask_query): 40 | outputs_query = self.bert( 41 | input_id_query, 42 | attention_mask=attention_mask_query, 43 | )[0] 44 | q = self.pooling(outputs_query, attention_mask_query) 45 | q = self.last_f(self.last_linear(q)) 46 | return q 47 | 48 | def infer_k(self, 49 | input_id_key, attention_mask_key): 50 | if self.key_bert is not None: 51 | outputs_key = self.key_bert( 52 | input_id_key, 53 | attention_mask=attention_mask_key, 54 | )[0] 55 | k = self.pooling(outputs_key, attention_mask_key) 56 | k = self.last_f(self.key_last_linear(k)) 57 | else: 58 | k = self.infer_q(input_id_key, attention_mask_key) 59 | return k 60 | 61 | def forward(self, 62 | input_id_query,attention_mask_query, 63 | input_id_key, attention_mask_key): 64 | q = self.infer_q(input_id_query, attention_mask_query) 65 | k = self.infer_k(input_id_key, attention_mask_key) 66 | return self.retrieve_loss(q, k) 67 | 68 | 69 | -------------------------------------------------------------------------------- /parameters.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | import argparse 5 | import logging 6 | 7 | def str2bool(v): 8 | if isinstance(v, bool): 9 | return v 10 | if v.lower() in ("yes", "true", "t", "y", "1"): 11 | return True 12 | elif v.lower() in ("no", "false", "f", "n", "0"): 13 | return False 14 | else: 15 | raise argparse.ArgumentTypeError("Boolean value expected.") 16 | 17 | 18 | def parse_args(): 19 | parser = argparse.ArgumentParser() 20 | 21 | parser.add_argument("--mode", type=str, default='train') 22 | parser.add_argument("--train_batch_size", type=int, default=500) 23 | parser.add_argument("--valid_batch_size", type=int, default=500) 24 | parser.add_argument("--test_batch_size", type=int, default=500) 25 | 26 | parser.add_argument("--model_dir", type=str, default='./model') # path to save 27 | parser.add_argument("--enable_gpu", type=str2bool, default=True) 28 | 29 | parser.add_argument("--query_max_token", type=int, default=32) 30 | parser.add_argument("--key_max_token", type=int, default=50) 31 | 32 | parser.add_argument("--savename", type=str, default='model') 33 | parser.add_argument("--world_size", type=int, default=8) 34 | 35 | parser.add_argument("--epochs", type=int, default=50) 36 | parser.add_argument("--save_steps", type=int, default=3000) 37 | parser.add_argument("--log_steps", type=int, default=10) 38 | 39 | parser.add_argument("--model_type", default="MoPQ", choices=['MoPQ', 'TextEncoder', 'DPQ', 'SPQ', 'DQN', 'DVSQ'], type=str) 40 | parser.add_argument("--bert_model", default="bert-base-uncased", type=str) 41 | parser.add_argument( 42 | "--load_ckpt_name", 43 | type=str, 44 | default = 'model/savename-epoch-1.pt', 45 | help="choose which ckpt to load and test" 46 | ) 47 | parser.add_argument("--ckpt_for_infer", default="bert-base-uncased", type=str) 48 | 49 | 50 | # lr schedule 51 | parser.add_argument("--lr", type=float, default=0.001) 52 | parser.add_argument("--pretrain_lr", type=float, default=0.0001) 53 | 54 | # model 55 | parser.add_argument("--layers", type=int, default=4) 56 | parser.add_argument("--partition_num", type=int, default=8) 57 | parser.add_argument("--centroids_num", type=int, default=256) 58 | parser.add_argument("--output_hidden_size", type=int, default=128) 59 | parser.add_argument("--select_mode", type=str, default='l2', help='codeword selection function', choices=['ip','mlp','sub_mlp','l2','mix','mix_w','cosine','dpq']) 60 | parser.add_argument("--recall_method", type=str, default='ip') 61 | parser.add_argument("--load", type=str2bool, default=False) 62 | parser.add_argument("--start_test_epoch", type=int, default=5) 63 | parser.add_argument("--blocking", type=str2bool, default=True) 64 | parser.add_argument("--dataset", type=str, default='Mind') 65 | parser.add_argument("--early_stop_epoch", type=int, default=3) 66 | parser.add_argument("--fix_embedding", type=str2bool, default=False) 67 | parser.add_argument("--self_supervised", type=str2bool, 68 | help='if use self-superived methods for unlabeled data, please load a pretrained TextEncoder and freeze it (fix_embedding=True) during training', 69 | default=False) 70 | parser.add_argument("--augment_method", type=str, default='all',choices=['drop','delete','add','all']) 71 | parser.add_argument("--cross_device", type=str2bool, default=True) 72 | 73 | parser.add_argument("--input_query_vec", type=str2bool, default=False) 74 | parser.add_argument("--input_key_vec", type=str2bool, default=False) 75 | 76 | parser.add_argument("--quantization_loss", type=str, 77 | nargs='+', 78 | default=['l2'], 79 | choices=['dot', 'subdot', 'l2']) 80 | parser.add_argument("--quantization_loss_weight", type=float, default=1e-8) 81 | 82 | 83 | 84 | args = parser.parse_args() 85 | logging.info(args) 86 | return args 87 | 88 | 89 | if __name__ == "__main__": 90 | args = parse_args() 91 | -------------------------------------------------------------------------------- /preprocess.py: -------------------------------------------------------------------------------- 1 | import os 2 | import logging 3 | import json 4 | import torch 5 | from tqdm import tqdm 6 | from transformers import AutoTokenizer 7 | from utils.setup_utils import setup_model 8 | from parameters import parse_args 9 | from utils.utils import setuplogging 10 | 11 | 12 | def preprocess(args, tokenizing_batch_size=2048): 13 | tokenizer = AutoTokenizer.from_pretrained(args.bert_model) 14 | args.train_data_path = './data/{}/train.tsv'.format(args.dataset) 15 | args.valid_data_path = './data/{}/valid.tsv'.format(args.dataset) 16 | args.test_data_path = './data/{}/test.tsv'.format(args.dataset) 17 | for file_path in [args.train_data_path, args.valid_data_path]: 18 | logging.info(f"processing {file_path}") 19 | 20 | directory, filename = os.path.split(file_path) 21 | cached_features_file = os.path.join( 22 | directory, 23 | "preprocessed_{}".format(filename), 24 | ) 25 | 26 | model_for_infer = None 27 | if args.input_query_vec or args.input_key_vec: 28 | model_for_infer = setup_model(args, model_type='TextEncoder') 29 | model_for_infer.load_state_dict(torch.load(args.ckpt_for_infer)['model_state_dict']) 30 | model_for_infer.cuda() 31 | 32 | batch_query, batch_key = [], [] 33 | with open(file_path, encoding="utf-8") as f, open( 34 | cached_features_file, "w", encoding="utf-8") as fout: 35 | for lineNum, line in tqdm(enumerate(f)): 36 | line = line.strip() 37 | if not line: continue 38 | try: 39 | query, key = line.strip('\n').split( '\t') 40 | except ValueError: 41 | logging.error("line {}: {}".format( 42 | lineNum, line.replace("\t", "|"))) 43 | continue 44 | batch_query.append(query.strip()) 45 | batch_key.append(key.strip()) 46 | 47 | if len(batch_query) >= tokenizing_batch_size: 48 | input_ids_query, input_ids_key, query_vecs, key_vecs = tokenize_and_infer(batch_query, 49 | batch_key, 50 | input_query_vec=args.input_query_vec, 51 | input_key_vec=args.input_key_vec, 52 | max_query_length=args.query_max_token, 53 | max_key_length=args.key_max_token, 54 | tokenizer=tokenizer, 55 | model=model_for_infer) 56 | save_data(input_ids_query, input_ids_key, query_vecs, key_vecs, output_f=fout) 57 | batch_query, batch_key = [], [] 58 | 59 | if len(batch_query) > 0: 60 | input_ids_query, input_ids_key, query_vecs, key_vecs = tokenize_and_infer(batch_query, 61 | batch_key, 62 | input_query_vec=args.input_query_vec, 63 | input_key_vec=args.input_key_vec, 64 | max_query_length=args.query_max_token, 65 | max_key_length=args.key_max_token, 66 | tokenizer=tokenizer, 67 | model=model_for_infer) 68 | save_data(input_ids_query, input_ids_key, query_vecs, key_vecs, output_f=fout) 69 | batch_query, batch_key = [], [] 70 | 71 | logging.info(f"Finish creating") 72 | 73 | 74 | 75 | def tokenize_and_infer(batch_query, batch_key, input_query_vec, input_key_vec, max_query_length, max_key_length, tokenizer, model=None): 76 | input_ids_query, input_ids_key, q_vecs, k_vecs = None, None, None, None 77 | if input_query_vec: 78 | tokenized_result_query = tokenizer.batch_encode_plus( 79 | batch_query, add_special_tokens=True, padding=True, truncation=True, max_length=max_query_length) 80 | input_ids = torch.LongTensor(tokenized_result_query['input_ids']).cuda() 81 | attention_mask = torch.FloatTensor(tokenized_result_query['attention_mask']).cuda() 82 | q_vecs = model.infer_q(input_ids, attention_mask).detach().cpu().numpy() 83 | else: 84 | tokenized_result_query = tokenizer.batch_encode_plus( 85 | batch_query, add_special_tokens=False) 86 | input_ids_query = tokenized_result_query['input_ids'] 87 | 88 | if input_key_vec: 89 | tokenized_result_key = tokenizer.batch_encode_plus( 90 | batch_key, add_special_tokens=True, padding=True, truncation=True, max_length=max_key_length) 91 | input_ids = torch.LongTensor(tokenized_result_key['input_ids']).cuda() 92 | attention_mask = torch.FloatTensor(tokenized_result_key['attention_mask']).cuda() 93 | k_vecs = model.infer_k(input_ids, attention_mask).detach().cpu().numpy() 94 | else: 95 | tokenized_result_key = tokenizer.batch_encode_plus( 96 | batch_key, add_special_tokens=False) 97 | input_ids_key = tokenized_result_key['input_ids'] 98 | 99 | return input_ids_query, input_ids_key, q_vecs, k_vecs 100 | 101 | def save_data(input_ids_query, input_ids_key, query_vecs, key_vecs, output_f): 102 | if query_vecs is not None: 103 | query_name = 'query_vec' 104 | query_data = query_vecs 105 | else: 106 | query_name = 'query_tokens' 107 | query_data = input_ids_query 108 | 109 | if key_vecs is not None: 110 | key_name = 'key_vec' 111 | key_data = key_vecs 112 | else: 113 | key_name = 'key_tokens' 114 | key_data = input_ids_key 115 | 116 | for j, (q, k) in enumerate( 117 | zip(query_data, key_data)): 118 | output_f.write(json.dumps({query_name:[float(x) for x in q], key_name:[float(x) for x in k]}) + '\n') 119 | 120 | 121 | if __name__ == '__main__': 122 | setuplogging() 123 | args = parse_args() 124 | preprocess(args, tokenizing_batch_size=2048) -------------------------------------------------------------------------------- /run.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | import copy 5 | import logging 6 | import os 7 | import time 8 | from pathlib import Path 9 | import sys, traceback 10 | from argparse import Namespace 11 | import torch 12 | import torch.optim as optim 13 | from torch.nn.parallel import DistributedDataParallel as DDP 14 | import torch.multiprocessing as mp 15 | 16 | from parameters import parse_args 17 | from utils.setup_utils import (cleanup, setup_worker, setup_loader, setup_manager, setup_model) 18 | from utils.utils import (compute_retrive_acc, dump_args, 19 | get_barrier, save_model, setuplogging) 20 | 21 | from test import test_single_process, test, test_recall 22 | 23 | def main(): 24 | setuplogging() 25 | os.environ['MASTER_ADDR'] = 'localhost' 26 | os.environ['MASTER_PORT'] = '12355' 27 | args = parse_args() 28 | 29 | args.train_data_path = './data/{}/preprocessed_train.tsv'.format(args.dataset) 30 | args.valid_data_path = './data/{}/preprocessed_valid.tsv'.format(args.dataset) 31 | args.test_data_path = './data/{}/preprocessed_test.tsv'.format(args.dataset) 32 | dump_args(args) 33 | 34 | Path(args.model_dir).mkdir(parents=True, exist_ok=True) 35 | 36 | if 'train' in args.mode: 37 | if args.cross_device: assert args.world_size>1 38 | 39 | mgr, end, end_train = setup_manager() 40 | mp.spawn(train, 41 | args=(args, end, end_train), 42 | nprocs=args.world_size, 43 | join=True) 44 | 45 | elif 'test' in args.mode: 46 | test(args) 47 | 48 | def train(local_rank:int, 49 | args: Namespace, 50 | end_dataloder, 51 | end_train 52 | ): 53 | ''' 54 | Args: 55 | local_rank: the rank of GPU 56 | args: Namespace 57 | end_dataloader: mp.Manager().Value(), exit dataloader 58 | end_train: mp.Manager().Value(), exit training 59 | ''' 60 | setuplogging() 61 | try: 62 | if args.world_size > 1: 63 | os.environ["RANK"] = str(local_rank) 64 | setup_worker(local_rank, args.world_size) 65 | device = torch.device("cuda" if torch.cuda.is_available() else 'cpu', 66 | local_rank) 67 | barrier = get_barrier(args) 68 | 69 | model = setup_model(args, args.model_type, local_rank) 70 | model = model.to(device) 71 | 72 | # if use self-superived methods for unlabeled data, please load a pretrained TextEncoder and freeze it during training 73 | if args.load: 74 | checkpoint = torch.load(args.load_ckpt_name, map_location="cpu") 75 | model.load_state_dict(checkpoint['model_state_dict'],strict=False) 76 | logging.info('load ckpt:{}'.format(args.load_ckpt_name)) 77 | if args.fix_embedding: 78 | model.bert = model.bert.requires_grad_(False) 79 | model.last_linear = model.last_linear.requires_grad_(False) 80 | 81 | if args.world_size > 1: 82 | ddp_model = DDP(model, 83 | device_ids=[local_rank], 84 | output_device=local_rank, 85 | find_unused_parameters=True) 86 | logging.info("[{}] Done copying model".format(local_rank)) 87 | else: 88 | logging.info("Not using ddp model") 89 | ddp_model = model 90 | 91 | rest_param = filter( 92 | lambda x: id(x) not in list(map(id, model.bert.parameters())), 93 | ddp_model.parameters()) 94 | 95 | optimizer = optim.Adam([{ 96 | 'params': model.bert.parameters(), 97 | 'lr': args.pretrain_lr 98 | }, { 99 | 'params': rest_param, 100 | 'lr': args.lr}]) 101 | 102 | loss = 0.0 103 | global_step = 0 104 | best_acc, best_count = 0.0, 0 105 | best_ep = 0 106 | best_model = copy.deepcopy(model) 107 | 108 | for ep in range(args.epochs): 109 | start_time = time.time() 110 | dataloader = setup_loader(args, 111 | local_rank=local_rank, 112 | end=end_dataloder, 113 | mode='train', 114 | buffer_batchs=50, 115 | blocking=False) 116 | 117 | ddp_model.train() 118 | for step, batch_data in enumerate(dataloader): 119 | if args.enable_gpu: 120 | batch_data = { 121 | k: v.cuda(non_blocking=True) 122 | for k, v in batch_data.items() if v is not None 123 | } 124 | 125 | batch_loss = ddp_model(**batch_data) 126 | 127 | loss += batch_loss.item() 128 | optimizer.zero_grad() 129 | batch_loss.backward() 130 | optimizer.step() 131 | global_step += 1 132 | 133 | if global_step % args.log_steps == 0: 134 | logging.info('[{}] step:{}, train_loss: {:.5f}'. 135 | format(local_rank, 136 | global_step * args.train_batch_size, 137 | loss / args.log_steps)) 138 | loss = 0.0 139 | 140 | logging.info("train time:{}".format(time.time() - start_time)) 141 | 142 | if local_rank == 0 and ep >= args.start_test_epoch: 143 | logging.info("Star validation for epoch-{}".format(ep + 1)) 144 | acc = test_single_process(model, args, "valid", args.model_type) 145 | logging.info("validation time:{}".format(time.time() - start_time)) 146 | if acc > best_acc: 147 | best_model = copy.deepcopy(model) 148 | best_acc = acc 149 | best_count = 0 150 | save_model(best_model, optimizer, args, name='best') 151 | best_ep = ep + 1 152 | else: 153 | best_count += 1 154 | if best_count >= args.early_stop_epoch: 155 | start_time = time.time() 156 | save_model(best_model, optimizer, args, name='best') 157 | logging.info('Best_vali_ACC-- soft/ori-hard_qk_acc:{}'.format(best_acc)) 158 | logging.info("Star testing for best") 159 | res = test_recall(best_model, args, mode='test', model_type=args.model_type) 160 | logging.info("best epoch:{}".format(best_ep)) 161 | logging.info("test_recall:{},{},{},{},{},{}".format( 162 | res[0], res[1], res[2], res[3], 163 | res[4], res[5])) 164 | logging.info("test time:{}".format(time.time() - 165 | start_time)) 166 | 167 | end_train.value = True 168 | barrier() 169 | if end_train.value: 170 | break 171 | 172 | if local_rank == 0 and not end_train.value: 173 | start_time = time.time() 174 | save_model(best_model, optimizer, args, name='best') 175 | logging.info('Best_vali_ACC-- soft/ori-hard_qk_acc:{}'.format(best_acc)) 176 | logging.info("Star testing for best") 177 | res = test_recall(model, args, mode='test', model_type=args.model_type) 178 | logging.info("best epoch:{}".format(best_ep)) 179 | logging.info("test_recall:{},{},{},{},{},{}".format( 180 | res[0], res[1], res[2], res[3], 181 | res[4], res[5])) 182 | logging.info("test time:{}".format(time.time() - 183 | start_time)) 184 | 185 | cleanup() 186 | except: 187 | error_type, error_value, error_trace = sys.exc_info() 188 | traceback.print_tb(error_trace) 189 | logging.info(error_value) 190 | 191 | 192 | if __name__ == "__main__": 193 | main() 194 | -------------------------------------------------------------------------------- /test.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | import logging 5 | import time 6 | from argparse import Namespace 7 | import torch 8 | 9 | from utils.setup_utils import (setup_loader, setup_model) 10 | from utils.utils import (compute_retrive_acc) 11 | from utils.recall import recall_by_dtable, encode 12 | 13 | def test_single_process(model: torch.nn.Module, 14 | args: Namespace, 15 | mode: str, 16 | model_type:str): 17 | 18 | assert mode in {"valid", "test"} 19 | model.eval() 20 | 21 | with torch.no_grad(): 22 | dataloader = setup_loader(args, mode=mode, multi_process=False) 23 | 24 | retrive_acc = [0, 0] 25 | for step, batch in enumerate(dataloader): 26 | if args.enable_gpu: 27 | batch = { 28 | k: v.cuda(non_blocking=True) 29 | for k, v in batch.items() if v is not None 30 | } 31 | 32 | if 'queries_vec' not in batch: 33 | input_id_query = batch['input_id_query'] 34 | attention_masks_query = batch['attention_mask_query'] 35 | q = model.infer_q( 36 | input_id_query, attention_masks_query) 37 | else: q = batch['queries_vec'] 38 | 39 | if 'keys_vec' not in batch: 40 | input_id_key = batch['input_id_key'] 41 | attention_masks_key = batch['attention_mask_key'] 42 | k = model.infer_k( 43 | input_id_key, attention_masks_key) 44 | else: k = batch['keys_vec'] 45 | 46 | if model_type == 'spq': 47 | _, q, _ = model.get_soft_and_hard(q) 48 | _, _, k = model.get_soft_and_hard(k) 49 | elif model_type == 'TextEncoder': 50 | pass 51 | else: 52 | k = model.quant(k) 53 | 54 | hit_num, all_num = compute_retrive_acc(q,k) 55 | retrive_acc[0] += hit_num.item() 56 | retrive_acc[1] += all_num.item() 57 | 58 | logging.info('Final-- qk_acc:{}'.format(retrive_acc[0] / 59 | retrive_acc[1])) 60 | return retrive_acc[0] / retrive_acc[1] 61 | 62 | 63 | def test_recall(model: torch.nn.Module, 64 | args: Namespace, 65 | mode:str ='test', 66 | model_type: str ='MoPQ'): 67 | if model_type == 'TextEncoder': 68 | logging.info('PQ not used in this model. You cannot use the PQ to do ANN search!') 69 | raise 70 | nn_file = './data/{}/{}_nn.json'.format(args.dataset, mode) 71 | topk = [1, 5, 10, 50, 100, 500] 72 | 73 | codes, dtable, encode_time = encode(args, model, mode=mode, model_type=model_type) 74 | logging.info('compute dtable costs:{}s---------------------------------------'.format(encode_time)) 75 | stime = time.time() 76 | res = recall_by_dtable(codes, 77 | dtable, 78 | topk=topk, 79 | enable_mulprocess=True, 80 | nn_file=nn_file) 81 | search_time = time.time() - stime 82 | logging.info('search topk costs:{}s---------------------------------------'.format(search_time)) 83 | logging.info('All Time:{}s [{} + {}]---------------------------------------'.format(encode_time+search_time, encode_time, search_time)) 84 | return res 85 | 86 | def test(args): 87 | device = torch.device("cuda") 88 | model = setup_model(args, args.model_type, local_rank=-1) 89 | model = model.to(device) 90 | 91 | checkpoint = torch.load(args.load_ckpt_name, map_location="cpu") 92 | model.load_state_dict(checkpoint['model_state_dict'], strict=False) 93 | logging.info('load ckpt:{}'.format(args.load_ckpt_name)) 94 | 95 | res = test_recall(model, args, mode='test', model_type=args.model_type) 96 | logging.info("test_recall:{},{},{},{},{},{}".format( 97 | res[0], res[1], res[2], res[3], 98 | res[4], res[5])) -------------------------------------------------------------------------------- /utils/recall.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | from utils.setup_utils import * 5 | import json 6 | from typing import Any, Callable, Dict, List, Tuple 7 | from tqdm import tqdm 8 | import heapq 9 | from multiprocessing import Process,Manager,cpu_count,Pool 10 | import time 11 | import os 12 | os.environ["TOKENIZERS_PARALLELISM"] = "false" 13 | 14 | def generate_batch(texts, tokenizer, batch_size, max_l): 15 | tokens = [] 16 | attention_mask = [] 17 | vecs = [] 18 | for i in range(len(texts)): 19 | content = texts[str(i)] 20 | if isinstance(content, str): 21 | temp = tokenizer(content, max_length=max_l, pad_to_max_length=True, truncation=True, add_special_tokens=False) 22 | tokens.append(temp['input_ids']) 23 | attention_mask.append(temp['attention_mask']) 24 | else: 25 | vecs.append(content) 26 | if len(tokens)==batch_size or len(vecs)==batch_size: 27 | if len(vecs) == 0: 28 | tokens = torch.LongTensor(tokens).cuda() 29 | attention_mask = torch.FloatTensor(attention_mask).cuda() 30 | vecs = None 31 | else: 32 | tokens = None 33 | attention_mask = None 34 | vecs = torch.FloatTensor(vecs).cuda() 35 | yield tokens,attention_mask, vecs 36 | tokens = []; attention_mask = []; vecs = [] 37 | if len(tokens)>0 or len(vecs)>0: 38 | if len(vecs) == 0: 39 | tokens = torch.LongTensor(tokens).cuda() 40 | attention_mask = torch.FloatTensor(attention_mask).cuda() 41 | vecs = None 42 | else: 43 | tokens = None 44 | attention_mask = None 45 | vecs = torch.FloatTensor(vecs).cuda() 46 | yield tokens, attention_mask, vecs 47 | tokens = []; attention_mask = []; vecs = [] 48 | 49 | 50 | def encode(args, model=None, mode='test', model_type='MoPQ'): 51 | tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased") 52 | device = torch.device("cuda" if torch.cuda.is_available() else 'cpu') 53 | model = model.to(device) 54 | model.eval() 55 | 56 | query_file = './data/{}/{}_queries.json'.format(args.dataset, mode) 57 | key_file = './data/{}/{}_keys.json'.format(args.dataset, mode) 58 | 59 | key_codes = [] 60 | query_dtable = [] 61 | batch_size = 2048 62 | with torch.no_grad(): 63 | mode = 'soft' if model_type == 'spq' else 'ori' 64 | query = json.load(open(query_file)) 65 | # stime = time.time() 66 | for data in tqdm(generate_batch(query, tokenizer, batch_size=batch_size, max_l=args.query_max_token), 67 | total=(len(query) // batch_size)): 68 | input, mask, vecs = data 69 | dtable = model.encode(input=input, mask=mask, vecs=vecs, mode=mode) 70 | dtable = dtable.detach().cpu().numpy() 71 | query_dtable.extend(dtable) 72 | 73 | key = json.load(open(key_file)) 74 | encode_time = 0 75 | for data in tqdm(generate_batch(key,tokenizer,batch_size=batch_size,max_l=args.key_max_token), 76 | total=(len(key) // batch_size)): 77 | input, mask, vecs = data 78 | start_time = time.time() 79 | codes = model.encode(input=input, mask=mask, vecs=vecs, mode='hard') 80 | codes = codes.detach().cpu().numpy() 81 | key_codes.extend(codes) 82 | encode_time += time.time() - start_time 83 | return key_codes, query_dtable, encode_time 84 | 85 | 86 | 87 | def get_result_by_dtable(dtable, codes, truth_nn, topk): 88 | res = [0]*len(topk) 89 | maxk = max(topk) 90 | 91 | for qi,table in enumerate(dtable): 92 | dists = np.sum(table[range(len(table)),codes],axis=-1) 93 | ans = heapq.nlargest(maxk, range(len(dists)), dists.__getitem__) 94 | 95 | for ki,k in enumerate(topk): 96 | ansk = set(ans[:k]) 97 | res[ki] += len(set.intersection(truth_nn[qi], ansk)) / len(truth_nn[qi]) 98 | res = np.array(res) 99 | return res 100 | 101 | def recall_by_dtable(key_codes, query_dtable, topk, nn_file=None, enable_mulprocess=True, ): 102 | query_nn = json.load(open(nn_file)) 103 | 104 | dtable = [] 105 | truth_nn = [] 106 | for k, v in query_nn.items(): 107 | dtable.append(query_dtable[int(k)]) 108 | truth_nn.append(set(v)) 109 | 110 | if not enable_mulprocess: 111 | res = get_result_by_dtable(dtable, key_codes, truth_nn, topk) 112 | return res 113 | else: 114 | # mul process 115 | world_size = max(cpu_count() - 3,1) 116 | pool = Pool(processes=world_size) 117 | results = [] 118 | per_process = len(truth_nn) // world_size 119 | for i in range(world_size): 120 | start = per_process * i 121 | end = min(len(truth_nn), per_process * (i + 1)) 122 | per_dtable = dtable[start:end] 123 | per_truth_nn = truth_nn[start:end] 124 | 125 | result = pool.apply_async(get_result_by_dtable, args=(per_dtable, key_codes, per_truth_nn, topk)) 126 | results.append(result) 127 | logging.info('Waiting for all subprocesses done...') 128 | pool.close() 129 | pool.join() 130 | logging.info('All subprocesses done.') 131 | 132 | res = np.array([0.0] * len(topk)) 133 | for result in results: 134 | res += result.get() 135 | return res / len(truth_nn) 136 | 137 | 138 | -------------------------------------------------------------------------------- /utils/setup_utils.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | import random 5 | import numpy as np 6 | import torch 7 | import torch.multiprocessing as mp 8 | import torch.distributed as dist 9 | 10 | from dataloader import ( 11 | DataCollatorForMatching, DatasetForMatching, 12 | MultiProcessDataLoaderForMatching, SingleProcessDataLoaderForMatching) 13 | import logging 14 | from transformers import BertTokenizerFast 15 | from models.MoPQ import MoPQ 16 | from models.TextEncoder import TextEncoder 17 | from models.Differentiable_PQ import DPQ 18 | from transformers import AutoConfig, AutoModel, AutoTokenizer 19 | 20 | def setup_worker(rank, world_size): 21 | # initialize the process group 22 | dist.init_process_group( 23 | "nccl", 24 | rank=rank, 25 | world_size=world_size, 26 | ) 27 | torch.cuda.set_device(rank) 28 | # Explicitly setting seed 29 | torch.manual_seed(42) 30 | np.random.seed(42) 31 | random.seed(42) 32 | 33 | def cleanup(): 34 | dist.destroy_process_group() 35 | 36 | 37 | def setup_loader(args, 38 | local_rank=0, 39 | end=None, 40 | blocking=False, 41 | multi_process=True, 42 | tokenizer=None, 43 | mode="train", 44 | buffer_batchs=None): 45 | datapath = {'train': args.train_data_path, 46 | 'valid': args.valid_data_path, 47 | 'test': args.test_data_path}[mode] 48 | 49 | batch_size = {'train': args.train_batch_size, 50 | 'valid': args.valid_batch_size, 51 | 'test': args.test_batch_size}[mode] 52 | 53 | if tokenizer is None: 54 | tokenizer = AutoTokenizer.from_pretrained(args.bert_model) 55 | 56 | data_collator = DataCollatorForMatching(tokenizer=tokenizer, 57 | args=args) 58 | dataset = DatasetForMatching(file_path=datapath) 59 | 60 | if multi_process: 61 | dataloader = MultiProcessDataLoaderForMatching( 62 | dataset, 63 | batch_size=batch_size, 64 | collate_fn=data_collator, 65 | local_rank=local_rank, 66 | world_size=args.world_size, 67 | global_end=end, 68 | buffer_num=buffer_batchs, 69 | blocking=blocking) 70 | else: 71 | # mostly for testing 72 | dataloader = SingleProcessDataLoaderForMatching( 73 | dataset, batch_size=batch_size, collate_fn=data_collator) 74 | return dataloader 75 | 76 | 77 | def setup_manager(): 78 | mgr = mp.Manager() 79 | end = mgr.Value('b', False) 80 | end_train = mgr.Value('b', False) 81 | return mgr, end, end_train 82 | 83 | 84 | def setup_model(args, model_type, local_rank=-1): 85 | config = AutoConfig.from_pretrained(args.bert_model) 86 | config.num_hidden_layers = args.layers 87 | bert = AutoModel.from_pretrained(args.bert_model, config=config) 88 | if model_type == 'MoPQ': 89 | model = MoPQ(args, bert, key_bert=None, hidden_size=config.hidden_size, local_rank=local_rank) 90 | elif model_type == 'TextEncoder': 91 | model = TextEncoder(bert, key_bert=None, hidden_size=config.hidden_size, output_hidden_size=args.output_hidden_size) 92 | else: 93 | model = DPQ(args, bert, key_bert=None, hidden_size=config.hidden_size) 94 | return model -------------------------------------------------------------------------------- /utils/utils.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | import logging 5 | import os 6 | import sys 7 | import torch 8 | import torch.nn.functional as F 9 | import numpy as np 10 | from sklearn.metrics import roc_auc_score 11 | 12 | 13 | def warmup_linear(args, step): 14 | if step <= args.warmup_step: 15 | return max(step, 1) / args.warmup_step 16 | return max(1e-4, (args.schedule_step - step) / 17 | (args.schedule_step - args.warmup_step)) 18 | 19 | 20 | def dummy(*args, **kwargs): 21 | pass 22 | 23 | def compute_acc(scores, labels): 24 | #hit num 25 | prediction = torch.argmax(scores, dim=-1) # N L 26 | hit = (prediction == labels).float() # N　L 27 | hit = torch.sum(hit) 28 | 29 | #all num 30 | labels = labels.masked_fill(labels >= 0, 1) 31 | labels = labels.masked_fill(labels < 0, 0) 32 | labels = torch.sum(labels) 33 | 34 | return hit, labels 35 | 36 | 37 | def compute_retrive_acc(q, k, mask_q=None, mask_k=None): 38 | score = torch.matmul(q, k.transpose(0, 1)) #N N 39 | labels = torch.arange(start=0, 40 | end=score.shape[0], 41 | dtype=torch.long, 42 | device=score.device) #N 43 | if mask_q is not None and mask_k is not None: 44 | mask = mask_q * mask_k 45 | elif mask_q is not None: 46 | mask = mask_q 47 | elif mask_k is not None: 48 | mask = mask_k 49 | else: 50 | mask = None 51 | 52 | if mask is not None: 53 | score = score.masked_fill(mask.unsqueeze(0) == 0, float("-inf")) #N N 54 | labels = labels.masked_fill(mask == 0, -100) 55 | 56 | return compute_acc(score, labels) 57 | 58 | 59 | def setuplogging(level=logging.INFO): 60 | # silent transformers 61 | logging.getLogger("transformers").setLevel(logging.ERROR) 62 | 63 | root = logging.getLogger() 64 | root.setLevel(logging.INFO) 65 | handler = logging.StreamHandler(sys.stdout) 66 | handler.setLevel(logging.INFO) 67 | formatter = logging.Formatter("[%(levelname)s %(asctime)s] %(message)s") 68 | handler.setFormatter(formatter) 69 | if (root.hasHandlers()): 70 | root.handlers.clear() # otherwise logging have multi output 71 | root.addHandler(handler) 72 | 73 | 74 | def dump_args(args): 75 | for arg in dir(args): 76 | if not arg.startswith("_"): 77 | logging.info(f"args[{arg}]={getattr(args, arg)}") 78 | 79 | 80 | def save_model(model, optim, args, 81 | epoch=None, steps=None, name=None): 82 | if name is None: 83 | ckpt_path = os.path.join(args.model_dir, 84 | f'{args.savename}-{steps*args.train_batch_size}.pt') 85 | else: 86 | ckpt_path = os.path.join(args.model_dir, 87 | f'{args.savename}-{name}.pt') 88 | torch.save( 89 | { 90 | 'model_state_dict': model.state_dict(), 91 | 'optimizer': optim.state_dict() 92 | }, ckpt_path) 93 | logging.info(f"Model saved to {ckpt_path}") 94 | 95 | 96 | def acc(y_true, y_hat): 97 | y_hat = torch.argmax(y_hat, dim=-1) 98 | tot = y_true.shape[0] 99 | hit = torch.sum(y_true == y_hat) 100 | return hit.data.float() * 1.0 / tot 101 | 102 | 103 | def dcg_score(y_true, y_score, k=10): 104 | order = np.argsort(y_score)[::-1] 105 | y_true = np.take(y_true, order[:k]) 106 | gains = 2**y_true - 1 107 | discounts = np.log2(np.arange(len(y_true)) + 2) 108 | return np.sum(gains / discounts) 109 | 110 | 111 | def ndcg_score(y_true, y_score, k=10): 112 | best = dcg_score(y_true, y_true, k) 113 | actual = dcg_score(y_true, y_score, k) 114 | return actual / best 115 | 116 | 117 | def mrr_score(y_true, y_score): 118 | order = np.argsort(y_score)[::-1] 119 | y_true = np.take(y_true, order) 120 | rr_score = y_true / (np.arange(len(y_true)) + 1) 121 | return np.sum(rr_score) / np.sum(y_true) 122 | 123 | 124 | def get_barrier(args): 125 | import torch.distributed as dist 126 | 127 | def nothing(): 128 | pass 129 | 130 | if args.world_size > 1: 131 | return dist.barrier 132 | return nothing 133 | 134 | --------------------------------------------------------------------------------