├── .idea ├── Cail2019_track2.iml ├── misc.xml ├── modules.xml └── workspace.xml ├── README.md ├── bert ├── .gitignore ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── __init__.py ├── create_pretraining_data.py ├── extract_features.py ├── modeling.py ├── modeling_test.py ├── multilingual.md ├── optimization.py ├── optimization_test.py ├── predicting_movie_reviews_with_bert_on_tf_hub.ipynb ├── requirements.txt ├── run_classifier.py ├── run_classifier_with_tfhub.py ├── run_pretraining.py ├── run_squad.py ├── sample_text.txt ├── tokenization.py └── tokenization_test.py ├── convert.py ├── createPretrainData.py ├── data ├── divorce │ ├── data_small_selected.json │ ├── tags.txt │ └── train_selected.json ├── labor │ ├── data_small_selected.json │ ├── tags.txt │ └── train_selected.json └── loan │ ├── data_small_selected.json │ ├── tags.txt │ └── train_selected.json ├── evaluation.py ├── genPretrainData.py ├── run_pretrain.py ├── search_threshold.py ├── train.py └── utils ├── __init__.py ├── ckpt2pb.py ├── evaluate.py ├── models.py └── predict.py /.idea/Cail2019_track2.iml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 11 | -------------------------------------------------------------------------------- /.idea/misc.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 6 | 7 | -------------------------------------------------------------------------------- /.idea/modules.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # cail2019_track2 2 | 中国法研杯CAIL2019要素抽取任务第三名方案分享 3 | ==== 4 | 欢迎大家使用[tensorflow1.x的bert系列模型库，支持单机多卡，梯度累积，自动导出pb部署](https://github.com/huanghuidmml/textToy) 5 | 6 | （修改了一下readme，之前那一版感觉写的太水了。） 7 | 8 | 这次比赛和前两名差距很大，但是也在此给大家分享一下我所用的方案。 9 | 10 | 主要的trick包括领域预训练、focal loss、阈值移动、规则匹配以及模型优化、调参。 11 | 12 | 没有使用模型融合。 13 | 14 | ### **效果对比** 15 | 16 | 由于是第一次参赛，很多比赛细节没有做记录，效果对比的分数是我从凭印象在上传历史记录里边找的，可能分数不一致，但是大概就在那个范围，还请见谅。 17 | 18 | | Model | 详情 | 线上评分 | 19 | | :------: | :------: | :------: | 20 | | BERT | 使用bert_base做多标签分类 | 69.553 | 21 | | BERT+RCNN+ATT | 在BERT后增加RCNN层，并把最大池化换成Attention | 70.143 | 22 | | BERT+RCNN+ATT | 增加阈值移动 | 70.809 | 23 | | BERT+RCNN+ATT | 增加focal loss | 71.126 | 24 | | BERT+RCNN+ATT | 增加规则 | 72.2 | 25 | | BERT+RCNN+ATT | 使用比赛数据预训练BERT | 72.526 | 26 | | BERT+RCNN+ATT | copy样本正例少的数据（divorce loan有效） | 72.909 | 27 | | BERT+RCNN+ATT | 在比赛数据基础上增加裁判文书（1000篇）做预训练（labor有效） | 73.483 | 28 | | BERT+RCNN+ATT | 增加否定词规则 | 73.533 | 29 | 30 | ### **主要参数** 31 | 32 | | 参数名 | 参数值 | 33 | | :------: | :------: | 34 | | 预训练模型 | [BERT_Base_Chinese](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip) | 35 | | max_length(divorce) | 128 | 36 | | max_length(labor) | 150 | 37 | | max_length(loan) | 200 | 38 | | batch_size | 32 | 39 | | learning_rate | 2e-5 | 40 | | num_train_epochs | 30 | 41 | | alpha(focal loss) | 0.25 | 42 | | gamma(focal loss) | 2 | 43 | | hidden_dim(lstm) | 200 | 44 | 45 | **方案介绍** 46 | ------ 47 | ### **任务简介** 48 | 根据给定司法文书中的相关段落，识别相应的关键案情要素，其中每个句子对应的类别标签个数不定，属于多标签问题。任务共涉及三个领域，包括婚姻家庭、劳动争议、借款合同。 49 | 例如： 50 | 51 | | 例句 | 标签 | 52 | | :------: | :------: | 53 | | 高雷红提出诉讼请求：1、判令被告支付原告的工资1630×0.8×4＝5216元； | ["LB2"] | 54 | | 原告范合诉称：原告系被告处职工。 | [] | 55 | | 5、判令被告某甲公司支付2011年9月8日至2012年8月7日未签订劳动合同的二倍工资差额16，298.37元； | ["LB9", "LB6"] | 56 | 57 | 根据数据集，我选定的方案就是传统的多标签分类方法。bert预训练模型使用的是google开源的bert_base_chinese. 58 | 59 | ### **任务难点** 60 | 61 | * **正负例样本不均衡** 62 | * **有的要素标签正例仅有几条，模型无法学习** 63 | 64 | ### **解决方案** 65 | 66 | #### **focal loss** 67 | 减少易分类样本的权重，增加难分类样本的损失贡献值，参数见上表的alpha，gamma 68 | 69 | #### **阈值移动** 70 | 将比赛的数据集切分为训练集和测试集。先用训练集去训练模型， 71 | 然后使用测试集去测试模型，筛选阈值；最后把所有数据拿去训练最后的提交模型， 72 | 预测阈值就采用之前筛选出来的阈值。 73 | 74 | #### **copy少量数据** 75 | 数据增强我尝试过eda，但是效果不行，不如不用，后来使用copy的方法做数据增强， 76 | 将正例少的样本copy一定的数量，但是不能copy太多，否则会严重破坏分布。 77 | 而且这个方法我只在divorce和loan两种领域有提升，labor上下降了， 78 | 可能是copy量不合理，大家可以下去尝试修改一下，看下会不会提升。 79 | 80 | #### **模型优化** 81 | 最后使用的模型是BERT + RCNN，并且RCNN部分的最大池化修改为Attention。 82 | 主要方法就是将BERT的输出向量X输入BiLstm，得到一个特征向量H，最后将X和H 83 | 拼接送入Attention。 84 | 85 | #### **规则** 86 | 规则主要是为了修正模型无法学习的要素标签，使用的方式：首先通过 87 | 标签的解释说明和包含标签的样本确定规则，规则在python中使用的是正则 88 | 表达式；然后针对需要预测的文本，我们先使用正则表达式去匹配，若是 89 | 匹配成功，则说明文本包含该规则对应的标签；最后把规则匹配出来的标签与 90 | 模型预测的标签取并集，得到最终预测要素集。 91 | 92 | 规则举例： 93 | > ['.(保证合同|抵押合同|借款合同).(无效|不发生效力).*'] 94 | ，对应的要素是LN12。 95 | 96 | **否定词规则** 97 | 98 | 否定词规则的意思是：在采用规则修正的时候，若是句子以一些否定词结尾，规则将不生效。 99 | 100 | 举例： 101 | 102 | > 被告五金公司辩称本案借款合同和保证合同均无效，缺乏法律依据，本院**不予采纳**。 103 | 104 | > 实际标签: LN13 LN10 105 | 106 | 这个句子可以匹配到我们写的LN12的规则：‘.*(保证合同|抵押合同|借款合同).*(无效|不发生效力).*‘ 107 | 108 | 但是因为末尾出现了不予采纳，所以该标签规则不生效，没有LN12。 109 | 110 | #### **领域预训练** 111 | 112 | bert模型采用的是bert_base_chinese，如果使用徐亮大佬的roberta应该还会有提升。 113 | 114 | 司法领域属于特殊领域，所以使用比赛数据先做了一次预训练，在三种领域都有一定的提升， 115 | 后边我爬取一些裁判文书来做预训练，可能是因为数据量小和质量不够，只在labor上得到了 116 | 提升，如果保证数据量和质量，应该会有提升。 117 | 118 | **代码说明** 119 | ------- 120 | 121 | #### **基本代码** 122 | CUDA_VISIBLE_DEVICES=1是指定第一块显卡，根据具体情况自己改， 123 | 如果CPU的话就不用了。 124 | 125 | **训练** 126 | 127 | > CUDA_VISIBLE_DEVICES=1 python train.py 128 | 129 | **将ckpt转为pb** 130 | 131 | > CUDA_VISIBLE_DEVICES=1 python convert.py 132 | 133 | **线下测试** 134 | 135 | > CUDA_VISIBLE_DEVICES=1 python evaluation.py 136 | 137 | #### **如果需要额外预训练的话，使用以下代码** 138 | 139 | **创建预训练数据txt** 140 | 141 | > python genPretrainData.py 142 | 143 | **创建预训练数据的tfrecord文件** 144 | 145 | > python createPretrainData.py 146 | 147 | **预训练** 148 | 149 | > CUDA_VISIBLE_DEVICES=1 python run_pretrain.py 150 | 151 | **Reference** 152 | ----- 153 | 1. [TensorFlow code and pre-trained models for BERT](https://github.com/google-research/bert) 154 | 2. [The implementation of focal loss proposed on "Focal Loss for Dense Object Detection" by KM He and support for multi-label dataset.](https://github.com/ailias/Focal-Loss-implement-on-Tensorflow) 155 | 156 | **感谢** 157 | ----- 158 | 感谢队友牧笛的帮助 159 | -------------------------------------------------------------------------------- /bert/.gitignore: -------------------------------------------------------------------------------- 1 | # Initially taken from Github's Python gitignore file 2 | 3 | # Byte-compiled / optimized / DLL files 4 | __pycache__/ 5 | *.py[cod] 6 | *$py.class 7 | 8 | # C extensions 9 | *.so 10 | 11 | # Distribution / packaging 12 | .Python 13 | build/ 14 | develop-eggs/ 15 | dist/ 16 | downloads/ 17 | eggs/ 18 | .eggs/ 19 | lib/ 20 | lib64/ 21 | parts/ 22 | sdist/ 23 | var/ 24 | wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | .hypothesis/ 51 | .pytest_cache/ 52 | 53 | # Translations 54 | *.mo 55 | *.pot 56 | 57 | # Django stuff: 58 | *.log 59 | local_settings.py 60 | db.sqlite3 61 | 62 | # Flask stuff: 63 | instance/ 64 | .webassets-cache 65 | 66 | # Scrapy stuff: 67 | .scrapy 68 | 69 | # Sphinx documentation 70 | docs/_build/ 71 | 72 | # PyBuilder 73 | target/ 74 | 75 | # Jupyter Notebook 76 | .ipynb_checkpoints 77 | 78 | # IPython 79 | profile_default/ 80 | ipython_config.py 81 | 82 | # pyenv 83 | .python-version 84 | 85 | # celery beat schedule file 86 | celerybeat-schedule 87 | 88 | # SageMath parsed files 89 | *.sage.py 90 | 91 | # Environments 92 | .env 93 | .venv 94 | env/ 95 | venv/ 96 | ENV/ 97 | env.bak/ 98 | venv.bak/ 99 | 100 | # Spyder project settings 101 | .spyderproject 102 | .spyproject 103 | 104 | # Rope project settings 105 | .ropeproject 106 | 107 | # mkdocs documentation 108 | /site 109 | 110 | # mypy 111 | .mypy_cache/ 112 | .dmypy.json 113 | dmypy.json 114 | 115 | # Pyre type checker 116 | .pyre/ 117 | -------------------------------------------------------------------------------- /bert/CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # How to Contribute 2 | 3 | BERT needs to maintain permanent compatibility with the pre-trained model files, 4 | so we do not plan to make any major changes to this library (other than what was 5 | promised in the README). However, we can accept small patches related to 6 | re-factoring and documentation. To submit contributes, there are just a few 7 | small guidelines you need to follow. 8 | 9 | ## Contributor License Agreement 10 | 11 | Contributions to this project must be accompanied by a Contributor License 12 | Agreement. You (or your employer) retain the copyright to your contribution; 13 | this simply gives us permission to use and redistribute your contributions as 14 | part of the project. Head over to to see 15 | your current agreements on file or to sign a new one. 16 | 17 | You generally only need to submit a CLA once, so if you've already submitted one 18 | (even if it was for a different project), you probably don't need to do it 19 | again. 20 | 21 | ## Code reviews 22 | 23 | All submissions, including submissions by project members, require review. We 24 | use GitHub pull requests for this purpose. Consult 25 | [GitHub Help](https://help.github.com/articles/about-pull-requests/) for more 26 | information on using pull requests. 27 | 28 | ## Community Guidelines 29 | 30 | This project follows 31 | [Google's Open Source Community Guidelines](https://opensource.google.com/conduct/). 32 | -------------------------------------------------------------------------------- /bert/LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /bert/__init__.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | -------------------------------------------------------------------------------- /bert/create_pretraining_data.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Create masked LM/next sentence masked_lm TF examples for BERT.""" 16 | 17 | from __future__ import absolute_import 18 | from __future__ import division 19 | from __future__ import print_function 20 | 21 | import collections 22 | import random 23 | from bert import tokenization 24 | import tensorflow as tf 25 | 26 | flags = tf.flags 27 | 28 | FLAGS = flags.FLAGS 29 | 30 | flags.DEFINE_string("input_file", None, 31 | "Input raw text file (or comma-separated list of files).") 32 | 33 | flags.DEFINE_string( 34 | "output_file", None, 35 | "Output TF example file (or comma-separated list of files).") 36 | 37 | flags.DEFINE_string("vocab_file", None, 38 | "The vocabulary file that the BERT model was trained on.") 39 | 40 | flags.DEFINE_bool( 41 | "do_lower_case", True, 42 | "Whether to lower case the input text. Should be True for uncased " 43 | "models and False for cased models.") 44 | 45 | flags.DEFINE_integer("max_seq_length", 128, "Maximum sequence length.") 46 | 47 | flags.DEFINE_integer("max_predictions_per_seq", 20, 48 | "Maximum number of masked LM predictions per sequence.") 49 | 50 | flags.DEFINE_integer("random_seed", 12345, "Random seed for data generation.") 51 | 52 | flags.DEFINE_integer( 53 | "dupe_factor", 10, 54 | "Number of times to duplicate the input data (with different masks).") 55 | 56 | flags.DEFINE_float("masked_lm_prob", 0.15, "Masked LM probability.") 57 | 58 | flags.DEFINE_float( 59 | "short_seq_prob", 0.1, 60 | "Probability of creating sequences which are shorter than the " 61 | "maximum length.") 62 | 63 | 64 | class TrainingInstance(object): 65 | """A single training instance (sentence pair).""" 66 | 67 | def __init__(self, tokens, segment_ids, masked_lm_positions, masked_lm_labels, 68 | is_random_next): 69 | self.tokens = tokens 70 | self.segment_ids = segment_ids 71 | self.is_random_next = is_random_next 72 | self.masked_lm_positions = masked_lm_positions 73 | self.masked_lm_labels = masked_lm_labels 74 | 75 | def __str__(self): 76 | s = "" 77 | s += "tokens: %s\n" % (" ".join( 78 | [tokenization.printable_text(x) for x in self.tokens])) 79 | s += "segment_ids: %s\n" % (" ".join([str(x) for x in self.segment_ids])) 80 | s += "is_random_next: %s\n" % self.is_random_next 81 | s += "masked_lm_positions: %s\n" % (" ".join( 82 | [str(x) for x in self.masked_lm_positions])) 83 | s += "masked_lm_labels: %s\n" % (" ".join( 84 | [tokenization.printable_text(x) for x in self.masked_lm_labels])) 85 | s += "\n" 86 | return s 87 | 88 | def __repr__(self): 89 | return self.__str__() 90 | 91 | 92 | def write_instance_to_example_files(instances, tokenizer, max_seq_length, 93 | max_predictions_per_seq, output_files): 94 | """Create TF example files from `TrainingInstance`s.""" 95 | writers = [] 96 | for output_file in output_files: 97 | writers.append(tf.python_io.TFRecordWriter(output_file)) 98 | 99 | writer_index = 0 100 | 101 | total_written = 0 102 | for (inst_index, instance) in enumerate(instances): 103 | input_ids = tokenizer.convert_tokens_to_ids(instance.tokens) 104 | input_mask = [1] * len(input_ids) 105 | segment_ids = list(instance.segment_ids) 106 | assert len(input_ids) <= max_seq_length 107 | 108 | while len(input_ids) < max_seq_length: 109 | input_ids.append(0) 110 | input_mask.append(0) 111 | segment_ids.append(0) 112 | 113 | assert len(input_ids) == max_seq_length 114 | assert len(input_mask) == max_seq_length 115 | assert len(segment_ids) == max_seq_length 116 | 117 | masked_lm_positions = list(instance.masked_lm_positions) 118 | masked_lm_ids = tokenizer.convert_tokens_to_ids(instance.masked_lm_labels) 119 | masked_lm_weights = [1.0] * len(masked_lm_ids) 120 | 121 | while len(masked_lm_positions) < max_predictions_per_seq: 122 | masked_lm_positions.append(0) 123 | masked_lm_ids.append(0) 124 | masked_lm_weights.append(0.0) 125 | 126 | next_sentence_label = 1 if instance.is_random_next else 0 127 | 128 | features = collections.OrderedDict() 129 | features["input_ids"] = create_int_feature(input_ids) 130 | features["input_mask"] = create_int_feature(input_mask) 131 | features["segment_ids"] = create_int_feature(segment_ids) 132 | features["masked_lm_positions"] = create_int_feature(masked_lm_positions) 133 | features["masked_lm_ids"] = create_int_feature(masked_lm_ids) 134 | features["masked_lm_weights"] = create_float_feature(masked_lm_weights) 135 | features["next_sentence_labels"] = create_int_feature([next_sentence_label]) 136 | 137 | tf_example = tf.train.Example(features=tf.train.Features(feature=features)) 138 | 139 | writers[writer_index].write(tf_example.SerializeToString()) 140 | writer_index = (writer_index + 1) % len(writers) 141 | 142 | total_written += 1 143 | 144 | if inst_index < 20: 145 | tf.logging.info("*** Example ***") 146 | tf.logging.info("tokens: %s" % " ".join( 147 | [tokenization.printable_text(x) for x in instance.tokens])) 148 | 149 | for feature_name in features.keys(): 150 | feature = features[feature_name] 151 | values = [] 152 | if feature.int64_list.value: 153 | values = feature.int64_list.value 154 | elif feature.float_list.value: 155 | values = feature.float_list.value 156 | tf.logging.info( 157 | "%s: %s" % (feature_name, " ".join([str(x) for x in values]))) 158 | 159 | for writer in writers: 160 | writer.close() 161 | 162 | tf.logging.info("Wrote %d total instances", total_written) 163 | 164 | 165 | def create_int_feature(values): 166 | feature = tf.train.Feature(int64_list=tf.train.Int64List(value=list(values))) 167 | return feature 168 | 169 | 170 | def create_float_feature(values): 171 | feature = tf.train.Feature(float_list=tf.train.FloatList(value=list(values))) 172 | return feature 173 | 174 | 175 | def create_training_instances(input_files, tokenizer, max_seq_length, 176 | dupe_factor, short_seq_prob, masked_lm_prob, 177 | max_predictions_per_seq, rng): 178 | """Create `TrainingInstance`s from raw text.""" 179 | all_documents = [[]] 180 | 181 | # Input file format: 182 | # (1) One sentence per line. These should ideally be actual sentences, not 183 | # entire paragraphs or arbitrary spans of text. (Because we use the 184 | # sentence boundaries for the "next sentence prediction" task). 185 | # (2) Blank lines between documents. Document boundaries are needed so 186 | # that the "next sentence prediction" task doesn't span between documents. 187 | for input_file in input_files: 188 | with tf.gfile.GFile(input_file, "r") as reader: 189 | while True: 190 | line = tokenization.convert_to_unicode(reader.readline()) 191 | if not line: 192 | break 193 | line = line.strip() 194 | 195 | # Empty lines are used as document delimiters 196 | if not line: 197 | all_documents.append([]) 198 | tokens = tokenizer.tokenize(line) 199 | if tokens: 200 | all_documents[-1].append(tokens) 201 | 202 | # Remove empty documents 203 | all_documents = [x for x in all_documents if x] 204 | rng.shuffle(all_documents) 205 | 206 | vocab_words = list(tokenizer.vocab.keys()) 207 | instances = [] 208 | for _ in range(dupe_factor): 209 | for document_index in range(len(all_documents)): 210 | instances.extend( 211 | create_instances_from_document( 212 | all_documents, document_index, max_seq_length, short_seq_prob, 213 | masked_lm_prob, max_predictions_per_seq, vocab_words, rng)) 214 | 215 | rng.shuffle(instances) 216 | return instances 217 | 218 | 219 | def create_instances_from_document( 220 | all_documents, document_index, max_seq_length, short_seq_prob, 221 | masked_lm_prob, max_predictions_per_seq, vocab_words, rng): 222 | """Creates `TrainingInstance`s for a single document.""" 223 | document = all_documents[document_index] 224 | 225 | # Account for [CLS], [SEP], [SEP] 226 | max_num_tokens = max_seq_length - 3 227 | 228 | # We *usually* want to fill up the entire sequence since we are padding 229 | # to `max_seq_length` anyways, so short sequences are generally wasted 230 | # computation. However, we *sometimes* 231 | # (i.e., short_seq_prob == 0.1 == 10% of the time) want to use shorter 232 | # sequences to minimize the mismatch between pre-training and fine-tuning. 233 | # The `target_seq_length` is just a rough target however, whereas 234 | # `max_seq_length` is a hard limit. 235 | target_seq_length = max_num_tokens 236 | if rng.random() < short_seq_prob: 237 | target_seq_length = rng.randint(2, max_num_tokens) 238 | 239 | # We DON'T just concatenate all of the tokens from a document into a long 240 | # sequence and choose an arbitrary split point because this would make the 241 | # next sentence prediction task too easy. Instead, we split the input into 242 | # segments "A" and "B" based on the actual "sentences" provided by the user 243 | # input. 244 | instances = [] 245 | current_chunk = [] 246 | current_length = 0 247 | i = 0 248 | while i < len(document): 249 | segment = document[i] 250 | current_chunk.append(segment) 251 | current_length += len(segment) 252 | if i == len(document) - 1 or current_length >= target_seq_length: 253 | if current_chunk: 254 | # `a_end` is how many segments from `current_chunk` go into the `A` 255 | # (first) sentence. 256 | a_end = 1 257 | if len(current_chunk) >= 2: 258 | a_end = rng.randint(1, len(current_chunk) - 1) 259 | 260 | tokens_a = [] 261 | for j in range(a_end): 262 | tokens_a.extend(current_chunk[j]) 263 | 264 | tokens_b = [] 265 | # Random next 266 | is_random_next = False 267 | if len(current_chunk) == 1 or rng.random() < 0.5: 268 | is_random_next = True 269 | target_b_length = target_seq_length - len(tokens_a) 270 | 271 | # This should rarely go for more than one iteration for large 272 | # corpora. However, just to be careful, we try to make sure that 273 | # the random document is not the same as the document 274 | # we're processing. 275 | for _ in range(10): 276 | random_document_index = rng.randint(0, len(all_documents) - 1) 277 | if random_document_index != document_index: 278 | break 279 | 280 | random_document = all_documents[random_document_index] 281 | random_start = rng.randint(0, len(random_document) - 1) 282 | for j in range(random_start, len(random_document)): 283 | tokens_b.extend(random_document[j]) 284 | if len(tokens_b) >= target_b_length: 285 | break 286 | # We didn't actually use these segments so we "put them back" so 287 | # they don't go to waste. 288 | num_unused_segments = len(current_chunk) - a_end 289 | i -= num_unused_segments 290 | # Actual next 291 | else: 292 | is_random_next = False 293 | for j in range(a_end, len(current_chunk)): 294 | tokens_b.extend(current_chunk[j]) 295 | truncate_seq_pair(tokens_a, tokens_b, max_num_tokens, rng) 296 | 297 | assert len(tokens_a) >= 1 298 | assert len(tokens_b) >= 1 299 | 300 | tokens = [] 301 | segment_ids = [] 302 | tokens.append("[CLS]") 303 | segment_ids.append(0) 304 | for token in tokens_a: 305 | tokens.append(token) 306 | segment_ids.append(0) 307 | 308 | tokens.append("[SEP]") 309 | segment_ids.append(0) 310 | 311 | for token in tokens_b: 312 | tokens.append(token) 313 | segment_ids.append(1) 314 | tokens.append("[SEP]") 315 | segment_ids.append(1) 316 | 317 | (tokens, masked_lm_positions, 318 | masked_lm_labels) = create_masked_lm_predictions( 319 | tokens, masked_lm_prob, max_predictions_per_seq, vocab_words, rng) 320 | instance = TrainingInstance( 321 | tokens=tokens, 322 | segment_ids=segment_ids, 323 | is_random_next=is_random_next, 324 | masked_lm_positions=masked_lm_positions, 325 | masked_lm_labels=masked_lm_labels) 326 | instances.append(instance) 327 | current_chunk = [] 328 | current_length = 0 329 | i += 1 330 | 331 | return instances 332 | 333 | 334 | MaskedLmInstance = collections.namedtuple("MaskedLmInstance", 335 | ["index", "label"]) 336 | 337 | 338 | def create_masked_lm_predictions(tokens, masked_lm_prob, 339 | max_predictions_per_seq, vocab_words, rng): 340 | """Creates the predictions for the masked LM objective.""" 341 | 342 | cand_indexes = [] 343 | for (i, token) in enumerate(tokens): 344 | if token == "[CLS]" or token == "[SEP]": 345 | continue 346 | cand_indexes.append(i) 347 | 348 | rng.shuffle(cand_indexes) 349 | 350 | output_tokens = list(tokens) 351 | 352 | num_to_predict = min(max_predictions_per_seq, 353 | max(1, int(round(len(tokens) * masked_lm_prob)))) 354 | 355 | masked_lms = [] 356 | covered_indexes = set() 357 | for index in cand_indexes: 358 | if len(masked_lms) >= num_to_predict: 359 | break 360 | if index in covered_indexes: 361 | continue 362 | covered_indexes.add(index) 363 | 364 | masked_token = None 365 | # 80% of the time, replace with [MASK] 366 | if rng.random() < 0.8: 367 | masked_token = "[MASK]" 368 | else: 369 | # 10% of the time, keep original 370 | if rng.random() < 0.5: 371 | masked_token = tokens[index] 372 | # 10% of the time, replace with random word 373 | else: 374 | masked_token = vocab_words[rng.randint(0, len(vocab_words) - 1)] 375 | 376 | output_tokens[index] = masked_token 377 | 378 | masked_lms.append(MaskedLmInstance(index=index, label=tokens[index])) 379 | 380 | masked_lms = sorted(masked_lms, key=lambda x: x.index) 381 | 382 | masked_lm_positions = [] 383 | masked_lm_labels = [] 384 | for p in masked_lms: 385 | masked_lm_positions.append(p.index) 386 | masked_lm_labels.append(p.label) 387 | 388 | return (output_tokens, masked_lm_positions, masked_lm_labels) 389 | 390 | 391 | def truncate_seq_pair(tokens_a, tokens_b, max_num_tokens, rng): 392 | """Truncates a pair of sequences to a maximum sequence length.""" 393 | while True: 394 | total_length = len(tokens_a) + len(tokens_b) 395 | if total_length <= max_num_tokens: 396 | break 397 | 398 | trunc_tokens = tokens_a if len(tokens_a) > len(tokens_b) else tokens_b 399 | assert len(trunc_tokens) >= 1 400 | 401 | # We want to sometimes truncate from the front and sometimes from the 402 | # back to add more randomness and avoid biases. 403 | if rng.random() < 0.5: 404 | del trunc_tokens[0] 405 | else: 406 | trunc_tokens.pop() 407 | 408 | 409 | def main(_): 410 | tf.logging.set_verbosity(tf.logging.INFO) 411 | 412 | tokenizer = tokenization.FullTokenizer( 413 | vocab_file=FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case) 414 | 415 | input_files = [] 416 | for input_pattern in FLAGS.input_file.split(","): 417 | input_files.extend(tf.gfile.Glob(input_pattern)) 418 | 419 | tf.logging.info("*** Reading from input files ***") 420 | for input_file in input_files: 421 | tf.logging.info(" %s", input_file) 422 | 423 | rng = random.Random(FLAGS.random_seed) 424 | instances = create_training_instances( 425 | input_files, tokenizer, FLAGS.max_seq_length, FLAGS.dupe_factor, 426 | FLAGS.short_seq_prob, FLAGS.masked_lm_prob, FLAGS.max_predictions_per_seq, 427 | rng) 428 | 429 | output_files = FLAGS.output_file.split(",") 430 | tf.logging.info("*** Writing to output files ***") 431 | for output_file in output_files: 432 | tf.logging.info(" %s", output_file) 433 | 434 | write_instance_to_example_files(instances, tokenizer, FLAGS.max_seq_length, 435 | FLAGS.max_predictions_per_seq, output_files) 436 | 437 | 438 | if __name__ == "__main__": 439 | flags.mark_flag_as_required("input_file") 440 | flags.mark_flag_as_required("output_file") 441 | flags.mark_flag_as_required("vocab_file") 442 | tf.app.run() 443 | -------------------------------------------------------------------------------- /bert/extract_features.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Extract pre-computed feature vectors from BERT.""" 16 | 17 | from __future__ import absolute_import 18 | from __future__ import division 19 | from __future__ import print_function 20 | 21 | import codecs 22 | import collections 23 | import json 24 | import re 25 | 26 | from bert import modeling 27 | from bert import tokenization 28 | import tensorflow as tf 29 | 30 | flags = tf.flags 31 | 32 | FLAGS = flags.FLAGS 33 | 34 | flags.DEFINE_string("input_file", None, "") 35 | 36 | flags.DEFINE_string("output_file", None, "") 37 | 38 | flags.DEFINE_string("layers", "-1,-2,-3,-4", "") 39 | 40 | flags.DEFINE_string( 41 | "bert_config_file", None, 42 | "The config json file corresponding to the pre-trained BERT model. " 43 | "This specifies the model architecture.") 44 | 45 | flags.DEFINE_integer( 46 | "max_seq_length", 128, 47 | "The maximum total input sequence length after WordPiece tokenization. " 48 | "Sequences longer than this will be truncated, and sequences shorter " 49 | "than this will be padded.") 50 | 51 | flags.DEFINE_string( 52 | "init_checkpoint", None, 53 | "Initial checkpoint (usually from a pre-trained BERT model).") 54 | 55 | flags.DEFINE_string("vocab_file", None, 56 | "The vocabulary file that the BERT model was trained on.") 57 | 58 | flags.DEFINE_bool( 59 | "do_lower_case", True, 60 | "Whether to lower case the input text. Should be True for uncased " 61 | "models and False for cased models.") 62 | 63 | flags.DEFINE_integer("batch_size", 32, "Batch size for predictions.") 64 | 65 | flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.") 66 | 67 | flags.DEFINE_string("master", None, 68 | "If using a TPU, the address of the master.") 69 | 70 | flags.DEFINE_integer( 71 | "num_tpu_cores", 8, 72 | "Only used if `use_tpu` is True. Total number of TPU cores to use.") 73 | 74 | flags.DEFINE_bool( 75 | "use_one_hot_embeddings", False, 76 | "If True, tf.one_hot will be used for embedding lookups, otherwise " 77 | "tf.nn.embedding_lookup will be used. On TPUs, this should be True " 78 | "since it is much faster.") 79 | 80 | 81 | class InputExample(object): 82 | 83 | def __init__(self, unique_id, text_a, text_b): 84 | self.unique_id = unique_id 85 | self.text_a = text_a 86 | self.text_b = text_b 87 | 88 | 89 | class InputFeatures(object): 90 | """A single set of features of data.""" 91 | 92 | def __init__(self, unique_id, tokens, input_ids, input_mask, input_type_ids): 93 | self.unique_id = unique_id 94 | self.tokens = tokens 95 | self.input_ids = input_ids 96 | self.input_mask = input_mask 97 | self.input_type_ids = input_type_ids 98 | 99 | 100 | def input_fn_builder(features, seq_length): 101 | """Creates an `input_fn` closure to be passed to TPUEstimator.""" 102 | 103 | all_unique_ids = [] 104 | all_input_ids = [] 105 | all_input_mask = [] 106 | all_input_type_ids = [] 107 | 108 | for feature in features: 109 | all_unique_ids.append(feature.unique_id) 110 | all_input_ids.append(feature.input_ids) 111 | all_input_mask.append(feature.input_mask) 112 | all_input_type_ids.append(feature.input_type_ids) 113 | 114 | def input_fn(params): 115 | """The actual input function.""" 116 | batch_size = params["batch_size"] 117 | 118 | num_examples = len(features) 119 | 120 | # This is for demo purposes and does NOT scale to large data sets. We do 121 | # not use Dataset.from_generator() because that uses tf.py_func which is 122 | # not TPU compatible. The right way to load data is with TFRecordReader. 123 | d = tf.data.Dataset.from_tensor_slices({ 124 | "unique_ids": 125 | tf.constant(all_unique_ids, shape=[num_examples], dtype=tf.int32), 126 | "input_ids": 127 | tf.constant( 128 | all_input_ids, shape=[num_examples, seq_length], 129 | dtype=tf.int32), 130 | "input_mask": 131 | tf.constant( 132 | all_input_mask, 133 | shape=[num_examples, seq_length], 134 | dtype=tf.int32), 135 | "input_type_ids": 136 | tf.constant( 137 | all_input_type_ids, 138 | shape=[num_examples, seq_length], 139 | dtype=tf.int32), 140 | }) 141 | 142 | d = d.batch(batch_size=batch_size, drop_remainder=False) 143 | return d 144 | 145 | return input_fn 146 | 147 | 148 | def model_fn_builder(bert_config, init_checkpoint, layer_indexes, use_tpu, 149 | use_one_hot_embeddings): 150 | """Returns `model_fn` closure for TPUEstimator.""" 151 | 152 | def model_fn(features, labels, mode, params): # pylint: disable=unused-argument 153 | """The `model_fn` for TPUEstimator.""" 154 | 155 | unique_ids = features["unique_ids"] 156 | input_ids = features["input_ids"] 157 | input_mask = features["input_mask"] 158 | input_type_ids = features["input_type_ids"] 159 | 160 | model = modeling.BertModel( 161 | config=bert_config, 162 | is_training=False, 163 | input_ids=input_ids, 164 | input_mask=input_mask, 165 | token_type_ids=input_type_ids, 166 | use_one_hot_embeddings=use_one_hot_embeddings) 167 | 168 | if mode != tf.estimator.ModeKeys.PREDICT: 169 | raise ValueError("Only PREDICT modes are supported: %s" % (mode)) 170 | 171 | tvars = tf.trainable_variables() 172 | scaffold_fn = None 173 | (assignment_map, 174 | initialized_variable_names) = modeling.get_assignment_map_from_checkpoint( 175 | tvars, init_checkpoint) 176 | if use_tpu: 177 | 178 | def tpu_scaffold(): 179 | tf.train.init_from_checkpoint(init_checkpoint, assignment_map) 180 | return tf.train.Scaffold() 181 | 182 | scaffold_fn = tpu_scaffold 183 | else: 184 | tf.train.init_from_checkpoint(init_checkpoint, assignment_map) 185 | 186 | tf.logging.info("**** Trainable Variables ****") 187 | for var in tvars: 188 | init_string = "" 189 | if var.name in initialized_variable_names: 190 | init_string = ", *INIT_FROM_CKPT*" 191 | tf.logging.info(" name = %s, shape = %s%s", var.name, var.shape, 192 | init_string) 193 | 194 | all_layers = model.get_all_encoder_layers() 195 | 196 | predictions = { 197 | "unique_id": unique_ids, 198 | } 199 | 200 | for (i, layer_index) in enumerate(layer_indexes): 201 | predictions["layer_output_%d" % i] = all_layers[layer_index] 202 | 203 | output_spec = tf.contrib.tpu.TPUEstimatorSpec( 204 | mode=mode, predictions=predictions, scaffold_fn=scaffold_fn) 205 | return output_spec 206 | 207 | return model_fn 208 | 209 | 210 | def convert_examples_to_features(examples, seq_length, tokenizer): 211 | """Loads a data file into a list of `InputBatch`s.""" 212 | 213 | features = [] 214 | for (ex_index, example) in enumerate(examples): 215 | tokens_a = tokenizer.tokenize(example.text_a) 216 | 217 | tokens_b = None 218 | if example.text_b: 219 | tokens_b = tokenizer.tokenize(example.text_b) 220 | 221 | if tokens_b: 222 | # Modifies `tokens_a` and `tokens_b` in place so that the total 223 | # length is less than the specified length. 224 | # Account for [CLS], [SEP], [SEP] with "- 3" 225 | _truncate_seq_pair(tokens_a, tokens_b, seq_length - 3) 226 | else: 227 | # Account for [CLS] and [SEP] with "- 2" 228 | if len(tokens_a) > seq_length - 2: 229 | tokens_a = tokens_a[0:(seq_length - 2)] 230 | 231 | # The convention in BERT is: 232 | # (a) For sequence pairs: 233 | # tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] 234 | # type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 235 | # (b) For single sequences: 236 | # tokens: [CLS] the dog is hairy . [SEP] 237 | # type_ids: 0 0 0 0 0 0 0 238 | # 239 | # Where "type_ids" are used to indicate whether this is the first 240 | # sequence or the second sequence. The embedding vectors for `type=0` and 241 | # `type=1` were learned during pre-training and are added to the wordpiece 242 | # embedding vector (and position vector). This is not *strictly* necessary 243 | # since the [SEP] token unambiguously separates the sequences, but it makes 244 | # it easier for the model to learn the concept of sequences. 245 | # 246 | # For classification tasks, the first vector (corresponding to [CLS]) is 247 | # used as as the "sentence vector". Note that this only makes sense because 248 | # the entire model is fine-tuned. 249 | tokens = [] 250 | input_type_ids = [] 251 | tokens.append("[CLS]") 252 | input_type_ids.append(0) 253 | for token in tokens_a: 254 | tokens.append(token) 255 | input_type_ids.append(0) 256 | tokens.append("[SEP]") 257 | input_type_ids.append(0) 258 | 259 | if tokens_b: 260 | for token in tokens_b: 261 | tokens.append(token) 262 | input_type_ids.append(1) 263 | tokens.append("[SEP]") 264 | input_type_ids.append(1) 265 | 266 | input_ids = tokenizer.convert_tokens_to_ids(tokens) 267 | 268 | # The mask has 1 for real tokens and 0 for padding tokens. Only real 269 | # tokens are attended to. 270 | input_mask = [1] * len(input_ids) 271 | 272 | # Zero-pad up to the sequence length. 273 | while len(input_ids) < seq_length: 274 | input_ids.append(0) 275 | input_mask.append(0) 276 | input_type_ids.append(0) 277 | 278 | assert len(input_ids) == seq_length 279 | assert len(input_mask) == seq_length 280 | assert len(input_type_ids) == seq_length 281 | 282 | if ex_index < 5: 283 | tf.logging.info("*** Example ***") 284 | tf.logging.info("unique_id: %s" % (example.unique_id)) 285 | tf.logging.info("tokens: %s" % " ".join( 286 | [tokenization.printable_text(x) for x in tokens])) 287 | tf.logging.info("input_ids: %s" % " ".join([str(x) for x in input_ids])) 288 | tf.logging.info("input_mask: %s" % " ".join([str(x) for x in input_mask])) 289 | tf.logging.info( 290 | "input_type_ids: %s" % " ".join([str(x) for x in input_type_ids])) 291 | 292 | features.append( 293 | InputFeatures( 294 | unique_id=example.unique_id, 295 | tokens=tokens, 296 | input_ids=input_ids, 297 | input_mask=input_mask, 298 | input_type_ids=input_type_ids)) 299 | return features 300 | 301 | 302 | def _truncate_seq_pair(tokens_a, tokens_b, max_length): 303 | """Truncates a sequence pair in place to the maximum length.""" 304 | 305 | # This is a simple heuristic which will always truncate the longer sequence 306 | # one token at a time. This makes more sense than truncating an equal percent 307 | # of tokens from each, since if one sequence is very short then each token 308 | # that's truncated likely contains more information than a longer sequence. 309 | while True: 310 | total_length = len(tokens_a) + len(tokens_b) 311 | if total_length <= max_length: 312 | break 313 | if len(tokens_a) > len(tokens_b): 314 | tokens_a.pop() 315 | else: 316 | tokens_b.pop() 317 | 318 | 319 | def read_examples(input_file): 320 | """Read a list of `InputExample`s from an input file.""" 321 | examples = [] 322 | unique_id = 0 323 | with tf.gfile.GFile(input_file, "r") as reader: 324 | while True: 325 | line = tokenization.convert_to_unicode(reader.readline()) 326 | if not line: 327 | break 328 | line = line.strip() 329 | text_a = None 330 | text_b = None 331 | m = re.match(r"^(.*) \|\|\| (.*)$", line) 332 | if m is None: 333 | text_a = line 334 | else: 335 | text_a = m.group(1) 336 | text_b = m.group(2) 337 | examples.append( 338 | InputExample(unique_id=unique_id, text_a=text_a, text_b=text_b)) 339 | unique_id += 1 340 | return examples 341 | 342 | 343 | def main(_): 344 | tf.logging.set_verbosity(tf.logging.INFO) 345 | 346 | layer_indexes = [int(x) for x in FLAGS.layers.split(",")] 347 | 348 | bert_config = modeling.BertConfig.from_json_file(FLAGS.bert_config_file) 349 | 350 | tokenizer = tokenization.FullTokenizer( 351 | vocab_file=FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case) 352 | 353 | is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2 354 | run_config = tf.contrib.tpu.RunConfig( 355 | master=FLAGS.master, 356 | tpu_config=tf.contrib.tpu.TPUConfig( 357 | num_shards=FLAGS.num_tpu_cores, 358 | per_host_input_for_training=is_per_host)) 359 | 360 | examples = read_examples(FLAGS.input_file) 361 | 362 | features = convert_examples_to_features( 363 | examples=examples, seq_length=FLAGS.max_seq_length, tokenizer=tokenizer) 364 | 365 | unique_id_to_feature = {} 366 | for feature in features: 367 | unique_id_to_feature[feature.unique_id] = feature 368 | 369 | model_fn = model_fn_builder( 370 | bert_config=bert_config, 371 | init_checkpoint=FLAGS.init_checkpoint, 372 | layer_indexes=layer_indexes, 373 | use_tpu=FLAGS.use_tpu, 374 | use_one_hot_embeddings=FLAGS.use_one_hot_embeddings) 375 | 376 | # If TPU is not available, this will fall back to normal Estimator on CPU 377 | # or GPU. 378 | estimator = tf.contrib.tpu.TPUEstimator( 379 | use_tpu=FLAGS.use_tpu, 380 | model_fn=model_fn, 381 | config=run_config, 382 | predict_batch_size=FLAGS.batch_size) 383 | 384 | input_fn = input_fn_builder( 385 | features=features, seq_length=FLAGS.max_seq_length) 386 | 387 | with codecs.getwriter("utf-8")(tf.gfile.Open(FLAGS.output_file, 388 | "w")) as writer: 389 | for result in estimator.predict(input_fn, yield_single_examples=True): 390 | unique_id = int(result["unique_id"]) 391 | feature = unique_id_to_feature[unique_id] 392 | output_json = collections.OrderedDict() 393 | output_json["linex_index"] = unique_id 394 | all_features = [] 395 | for (i, token) in enumerate(feature.tokens): 396 | all_layers = [] 397 | for (j, layer_index) in enumerate(layer_indexes): 398 | layer_output = result["layer_output_%d" % j] 399 | layers = collections.OrderedDict() 400 | layers["index"] = layer_index 401 | layers["values"] = [ 402 | round(float(x), 6) for x in layer_output[i:(i + 1)].flat 403 | ] 404 | all_layers.append(layers) 405 | features = collections.OrderedDict() 406 | features["token"] = token 407 | features["layers"] = all_layers 408 | all_features.append(features) 409 | output_json["features"] = all_features 410 | writer.write(json.dumps(output_json) + "\n") 411 | 412 | 413 | if __name__ == "__main__": 414 | flags.mark_flag_as_required("input_file") 415 | flags.mark_flag_as_required("vocab_file") 416 | flags.mark_flag_as_required("bert_config_file") 417 | flags.mark_flag_as_required("init_checkpoint") 418 | flags.mark_flag_as_required("output_file") 419 | tf.app.run() 420 | -------------------------------------------------------------------------------- /bert/modeling_test.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | from __future__ import absolute_import 16 | from __future__ import division 17 | from __future__ import print_function 18 | 19 | import collections 20 | import json 21 | import random 22 | import re 23 | 24 | from bert import modeling 25 | import six 26 | import tensorflow as tf 27 | 28 | 29 | class BertModelTest(tf.test.TestCase): 30 | 31 | class BertModelTester(object): 32 | 33 | def __init__(self, 34 | parent, 35 | batch_size=13, 36 | seq_length=7, 37 | is_training=True, 38 | use_input_mask=True, 39 | use_token_type_ids=True, 40 | vocab_size=99, 41 | hidden_size=32, 42 | num_hidden_layers=5, 43 | num_attention_heads=4, 44 | intermediate_size=37, 45 | hidden_act="gelu", 46 | hidden_dropout_prob=0.1, 47 | attention_probs_dropout_prob=0.1, 48 | max_position_embeddings=512, 49 | type_vocab_size=16, 50 | initializer_range=0.02, 51 | scope=None): 52 | self.parent = parent 53 | self.batch_size = batch_size 54 | self.seq_length = seq_length 55 | self.is_training = is_training 56 | self.use_input_mask = use_input_mask 57 | self.use_token_type_ids = use_token_type_ids 58 | self.vocab_size = vocab_size 59 | self.hidden_size = hidden_size 60 | self.num_hidden_layers = num_hidden_layers 61 | self.num_attention_heads = num_attention_heads 62 | self.intermediate_size = intermediate_size 63 | self.hidden_act = hidden_act 64 | self.hidden_dropout_prob = hidden_dropout_prob 65 | self.attention_probs_dropout_prob = attention_probs_dropout_prob 66 | self.max_position_embeddings = max_position_embeddings 67 | self.type_vocab_size = type_vocab_size 68 | self.initializer_range = initializer_range 69 | self.scope = scope 70 | 71 | def create_model(self): 72 | input_ids = BertModelTest.ids_tensor([self.batch_size, self.seq_length], 73 | self.vocab_size) 74 | 75 | input_mask = None 76 | if self.use_input_mask: 77 | input_mask = BertModelTest.ids_tensor( 78 | [self.batch_size, self.seq_length], vocab_size=2) 79 | 80 | token_type_ids = None 81 | if self.use_token_type_ids: 82 | token_type_ids = BertModelTest.ids_tensor( 83 | [self.batch_size, self.seq_length], self.type_vocab_size) 84 | 85 | config = modeling.BertConfig( 86 | vocab_size=self.vocab_size, 87 | hidden_size=self.hidden_size, 88 | num_hidden_layers=self.num_hidden_layers, 89 | num_attention_heads=self.num_attention_heads, 90 | intermediate_size=self.intermediate_size, 91 | hidden_act=self.hidden_act, 92 | hidden_dropout_prob=self.hidden_dropout_prob, 93 | attention_probs_dropout_prob=self.attention_probs_dropout_prob, 94 | max_position_embeddings=self.max_position_embeddings, 95 | type_vocab_size=self.type_vocab_size, 96 | initializer_range=self.initializer_range) 97 | 98 | model = modeling.BertModel( 99 | config=config, 100 | is_training=self.is_training, 101 | input_ids=input_ids, 102 | input_mask=input_mask, 103 | token_type_ids=token_type_ids, 104 | scope=self.scope) 105 | 106 | outputs = { 107 | "embedding_output": model.get_embedding_output(), 108 | "sequence_output": model.get_sequence_output(), 109 | "pooled_output": model.get_pooled_output(), 110 | "all_encoder_layers": model.get_all_encoder_layers(), 111 | } 112 | return outputs 113 | 114 | def check_output(self, result): 115 | self.parent.assertAllEqual( 116 | result["embedding_output"].shape, 117 | [self.batch_size, self.seq_length, self.hidden_size]) 118 | 119 | self.parent.assertAllEqual( 120 | result["sequence_output"].shape, 121 | [self.batch_size, self.seq_length, self.hidden_size]) 122 | 123 | self.parent.assertAllEqual(result["pooled_output"].shape, 124 | [self.batch_size, self.hidden_size]) 125 | 126 | def test_default(self): 127 | self.run_tester(BertModelTest.BertModelTester(self)) 128 | 129 | def test_config_to_json_string(self): 130 | config = modeling.BertConfig(vocab_size=99, hidden_size=37) 131 | obj = json.loads(config.to_json_string()) 132 | self.assertEqual(obj["vocab_size"], 99) 133 | self.assertEqual(obj["hidden_size"], 37) 134 | 135 | def run_tester(self, tester): 136 | with self.test_session() as sess: 137 | ops = tester.create_model() 138 | init_op = tf.group(tf.global_variables_initializer(), 139 | tf.local_variables_initializer()) 140 | sess.run(init_op) 141 | output_result = sess.run(ops) 142 | tester.check_output(output_result) 143 | 144 | self.assert_all_tensors_reachable(sess, [init_op, ops]) 145 | 146 | @classmethod 147 | def ids_tensor(cls, shape, vocab_size, rng=None, name=None): 148 | """Creates a random int32 tensor of the shape within the vocab size.""" 149 | if rng is None: 150 | rng = random.Random() 151 | 152 | total_dims = 1 153 | for dim in shape: 154 | total_dims *= dim 155 | 156 | values = [] 157 | for _ in range(total_dims): 158 | values.append(rng.randint(0, vocab_size - 1)) 159 | 160 | return tf.constant(value=values, dtype=tf.int32, shape=shape, name=name) 161 | 162 | def assert_all_tensors_reachable(self, sess, outputs): 163 | """Checks that all the tensors in the graph are reachable from outputs.""" 164 | graph = sess.graph 165 | 166 | ignore_strings = [ 167 | "^.*/assert_less_equal/.*$", 168 | "^.*/dilation_rate$", 169 | "^.*/Tensordot/concat$", 170 | "^.*/Tensordot/concat/axis$", 171 | "^testing/.*$", 172 | ] 173 | 174 | ignore_regexes = [re.compile(x) for x in ignore_strings] 175 | 176 | unreachable = self.get_unreachable_ops(graph, outputs) 177 | filtered_unreachable = [] 178 | for x in unreachable: 179 | do_ignore = False 180 | for r in ignore_regexes: 181 | m = r.match(x.name) 182 | if m is not None: 183 | do_ignore = True 184 | if do_ignore: 185 | continue 186 | filtered_unreachable.append(x) 187 | unreachable = filtered_unreachable 188 | 189 | self.assertEqual( 190 | len(unreachable), 0, "The following ops are unreachable: %s" % 191 | (" ".join([x.name for x in unreachable]))) 192 | 193 | @classmethod 194 | def get_unreachable_ops(cls, graph, outputs): 195 | """Finds all of the tensors in graph that are unreachable from outputs.""" 196 | outputs = cls.flatten_recursive(outputs) 197 | output_to_op = collections.defaultdict(list) 198 | op_to_all = collections.defaultdict(list) 199 | assign_out_to_in = collections.defaultdict(list) 200 | 201 | for op in graph.get_operations(): 202 | for x in op.inputs: 203 | op_to_all[op.name].append(x.name) 204 | for y in op.outputs: 205 | output_to_op[y.name].append(op.name) 206 | op_to_all[op.name].append(y.name) 207 | if str(op.type) == "Assign": 208 | for y in op.outputs: 209 | for x in op.inputs: 210 | assign_out_to_in[y.name].append(x.name) 211 | 212 | assign_groups = collections.defaultdict(list) 213 | for out_name in assign_out_to_in.keys(): 214 | name_group = assign_out_to_in[out_name] 215 | for n1 in name_group: 216 | assign_groups[n1].append(out_name) 217 | for n2 in name_group: 218 | if n1 != n2: 219 | assign_groups[n1].append(n2) 220 | 221 | seen_tensors = {} 222 | stack = [x.name for x in outputs] 223 | while stack: 224 | name = stack.pop() 225 | if name in seen_tensors: 226 | continue 227 | seen_tensors[name] = True 228 | 229 | if name in output_to_op: 230 | for op_name in output_to_op[name]: 231 | if op_name in op_to_all: 232 | for input_name in op_to_all[op_name]: 233 | if input_name not in stack: 234 | stack.append(input_name) 235 | 236 | expanded_names = [] 237 | if name in assign_groups: 238 | for assign_name in assign_groups[name]: 239 | expanded_names.append(assign_name) 240 | 241 | for expanded_name in expanded_names: 242 | if expanded_name not in stack: 243 | stack.append(expanded_name) 244 | 245 | unreachable_ops = [] 246 | for op in graph.get_operations(): 247 | is_unreachable = False 248 | all_names = [x.name for x in op.inputs] + [x.name for x in op.outputs] 249 | for name in all_names: 250 | if name not in seen_tensors: 251 | is_unreachable = True 252 | if is_unreachable: 253 | unreachable_ops.append(op) 254 | return unreachable_ops 255 | 256 | @classmethod 257 | def flatten_recursive(cls, item): 258 | """Flattens (potentially nested) a tuple/dictionary/list to a list.""" 259 | output = [] 260 | if isinstance(item, list): 261 | output.extend(item) 262 | elif isinstance(item, tuple): 263 | output.extend(list(item)) 264 | elif isinstance(item, dict): 265 | for (_, v) in six.iteritems(item): 266 | output.append(v) 267 | else: 268 | return [item] 269 | 270 | flat_output = [] 271 | for x in output: 272 | flat_output.extend(cls.flatten_recursive(x)) 273 | return flat_output 274 | 275 | 276 | if __name__ == "__main__": 277 | tf.test.main() 278 | -------------------------------------------------------------------------------- /bert/multilingual.md: -------------------------------------------------------------------------------- 1 | ## Models 2 | 3 | There are two multilingual models currently available. We do not plan to release 4 | more single-language models, but we may release `BERT-Large` versions of these 5 | two in the future: 6 | 7 | * **[`BERT-Base, Multilingual Cased (New, recommended)`](https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip)**: 8 | 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters 9 | * **[`BERT-Base, Multilingual Uncased (Orig, not recommended)`](https://storage.googleapis.com/bert_models/2018_11_03/multilingual_L-12_H-768_A-12.zip)**: 10 | 102 languages, 12-layer, 768-hidden, 12-heads, 110M parameters 11 | * **[`BERT-Base, Chinese`](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip)**: 12 | Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M 13 | parameters 14 | 15 | **The `Multilingual Cased (New)` model also fixes normalization issues in many 16 | languages, so it is recommended in languages with non-Latin alphabets (and is 17 | often better for most languages with Latin alphabets). When using this model, 18 | make sure to pass `--do_lower_case=false` to `run_pretraining.py` and other 19 | scripts.** 20 | 21 | See the [list of languages](#list-of-languages) that the Multilingual model 22 | supports. The Multilingual model does include Chinese (and English), but if your 23 | fine-tuning data is Chinese-only, then the Chinese model will likely produce 24 | better results. 25 | 26 | ## Results 27 | 28 | To evaluate these systems, we use the 29 | [XNLI dataset](https://github.com/facebookresearch/XNLI) dataset, which is a 30 | version of [MultiNLI](https://www.nyu.edu/projects/bowman/multinli/) where the 31 | dev and test sets have been translated (by humans) into 15 languages. Note that 32 | the training set was *machine* translated (we used the translations provided by 33 | XNLI, not Google NMT). For clarity, we only report on 6 languages below: 34 | 35 | 36 | 37 | | System | English | Chinese | Spanish | German | Arabic | Urdu | 38 | | --------------------------------- | -------- | -------- | -------- | -------- | -------- | -------- | 39 | | XNLI Baseline - Translate Train | 73.7 | 67.0 | 68.8 | 66.5 | 65.8 | 56.6 | 40 | | XNLI Baseline - Translate Test | 73.7 | 68.3 | 70.7 | 68.7 | 66.8 | 59.3 | 41 | | BERT - Translate Train Cased | **81.9** | **76.6** | **77.8** | **75.9** | **70.7** | 61.6 | 42 | | BERT - Translate Train Uncased | 81.4 | 74.2 | 77.3 | 75.2 | 70.5 | 61.7 | 43 | | BERT - Translate Test Uncased | 81.4 | 70.1 | 74.9 | 74.4 | 70.4 | **62.1** | 44 | | BERT - Zero Shot Uncased | 81.4 | 63.8 | 74.3 | 70.5 | 62.1 | 58.3 | 45 | 46 | 47 | 48 | The first two rows are baselines from the XNLI paper and the last three rows are 49 | our results with BERT. 50 | 51 | **Translate Train** means that the MultiNLI training set was machine translated 52 | from English into the foreign language. So training and evaluation were both 53 | done in the foreign language. Unfortunately, training was done on 54 | machine-translated data, so it is impossible to quantify how much of the lower 55 | accuracy (compared to English) is due to the quality of the machine translation 56 | vs. the quality of the pre-trained model. 57 | 58 | **Translate Test** means that the XNLI test set was machine translated from the 59 | foreign language into English. So training and evaluation were both done on 60 | English. However, test evaluation was done on machine-translated English, so the 61 | accuracy depends on the quality of the machine translation system. 62 | 63 | **Zero Shot** means that the Multilingual BERT system was fine-tuned on English 64 | MultiNLI, and then evaluated on the foreign language XNLI test. In this case, 65 | machine translation was not involved at all in either the pre-training or 66 | fine-tuning. 67 | 68 | Note that the English result is worse than the 84.2 MultiNLI baseline because 69 | this training used Multilingual BERT rather than English-only BERT. This implies 70 | that for high-resource languages, the Multilingual model is somewhat worse than 71 | a single-language model. However, it is not feasible for us to train and 72 | maintain dozens of single-language model. Therefore, if your goal is to maximize 73 | performance with a language other than English or Chinese, you might find it 74 | beneficial to run pre-training for additional steps starting from our 75 | Multilingual model on data from your language of interest. 76 | 77 | Here is a comparison of training Chinese models with the Multilingual 78 | `BERT-Base` and Chinese-only `BERT-Base`: 79 | 80 | System | Chinese 81 | ----------------------- | ------- 82 | XNLI Baseline | 67.0 83 | BERT Multilingual Model | 74.2 84 | BERT Chinese-only Model | 77.2 85 | 86 | Similar to English, the single-language model does 3% better than the 87 | Multilingual model. 88 | 89 | ## Fine-tuning Example 90 | 91 | The multilingual model does **not** require any special consideration or API 92 | changes. We did update the implementation of `BasicTokenizer` in 93 | `tokenization.py` to support Chinese character tokenization, so please update if 94 | you forked it. However, we did not change the tokenization API. 95 | 96 | To test the new models, we did modify `run_classifier.py` to add support for the 97 | [XNLI dataset](https://github.com/facebookresearch/XNLI). This is a 15-language 98 | version of MultiNLI where the dev/test sets have been human-translated, and the 99 | training set has been machine-translated. 100 | 101 | To run the fine-tuning code, please download the 102 | [XNLI dev/test set](https://s3.amazonaws.com/xnli/XNLI-1.0.zip) and the 103 | [XNLI machine-translated training set](https://s3.amazonaws.com/xnli/XNLI-MT-1.0.zip) 104 | and then unpack both .zip files into some directory `$XNLI_DIR`. 105 | 106 | To run fine-tuning on XNLI. The language is hard-coded into `run_classifier.py` 107 | (Chinese by default), so please modify `XnliProcessor` if you want to run on 108 | another language. 109 | 110 | This is a large dataset, so this will training will take a few hours on a GPU 111 | (or about 30 minutes on a Cloud TPU). To run an experiment quickly for 112 | debugging, just set `num_train_epochs` to a small value like `0.1`. 113 | 114 | ```shell 115 | export BERT_BASE_DIR=/path/to/bert/chinese_L-12_H-768_A-12 # or multilingual_L-12_H-768_A-12 116 | export XNLI_DIR=/path/to/xnli 117 | 118 | python run_classifier.py \ 119 | --task_name=XNLI \ 120 | --do_train=true \ 121 | --do_eval=true \ 122 | --data_dir=$XNLI_DIR \ 123 | --vocab_file=$BERT_BASE_DIR/vocab.txt \ 124 | --bert_config_file=$BERT_BASE_DIR/bert_config.json \ 125 | --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \ 126 | --max_seq_length=128 \ 127 | --train_batch_size=32 \ 128 | --learning_rate=5e-5 \ 129 | --num_train_epochs=2.0 \ 130 | --output_dir=/tmp/xnli_output/ 131 | ``` 132 | 133 | With the Chinese-only model, the results should look something like this: 134 | 135 | ``` 136 | ***** Eval results ***** 137 | eval_accuracy = 0.774116 138 | eval_loss = 0.83554 139 | global_step = 24543 140 | loss = 0.74603 141 | ``` 142 | 143 | ## Details 144 | 145 | ### Data Source and Sampling 146 | 147 | The languages chosen were the 148 | [top 100 languages with the largest Wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias). 149 | The entire Wikipedia dump for each language (excluding user and talk pages) was 150 | taken as the training data for each language 151 | 152 | However, the size of the Wikipedia for a given language varies greatly, and 153 | therefore low-resource languages may be "under-represented" in terms of the 154 | neural network model (under the assumption that languages are "competing" for 155 | limited model capacity to some extent). 156 | 157 | However, the size of a Wikipedia also correlates with the number of speakers of 158 | a language, and we also don't want to overfit the model by performing thousands 159 | of epochs over a tiny Wikipedia for a particular language. 160 | 161 | To balance these two factors, we performed exponentially smoothed weighting of 162 | the data during pre-training data creation (and WordPiece vocab creation). In 163 | other words, let's say that the probability of a language is *P(L)*, e.g., 164 | *P(English) = 0.21* means that after concatenating all of the Wikipedias 165 | together, 21% of our data is English. We exponentiate each probability by some 166 | factor *S* and then re-normalize, and sample from that distribution. In our case 167 | we use *S=0.7*. So, high-resource languages like English will be under-sampled, 168 | and low-resource languages like Icelandic will be over-sampled. E.g., in the 169 | original distribution English would be sampled 1000x more than Icelandic, but 170 | after smoothing it's only sampled 100x more. 171 | 172 | ### Tokenization 173 | 174 | For tokenization, we use a 110k shared WordPiece vocabulary. The word counts are 175 | weighted the same way as the data, so low-resource languages are upweighted by 176 | some factor. We intentionally do *not* use any marker to denote the input 177 | language (so that zero-shot training can work). 178 | 179 | Because Chinese (and Japanese Kanji and Korean Hanja) does not have whitespace 180 | characters, we add spaces around every character in the 181 | [CJK Unicode range](https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_$Unicode_block$) 182 | before applying WordPiece. This means that Chinese is effectively 183 | character-tokenized. Note that the CJK Unicode block only includes 184 | Chinese-origin characters and does *not* include Hangul Korean or 185 | Katakana/Hiragana Japanese, which are tokenized with whitespace+WordPiece like 186 | all other languages. 187 | 188 | For all other languages, we apply the 189 | [same recipe as English](https://github.com/google-research/bert#tokenization): 190 | (a) lower casing+accent removal, (b) punctuation splitting, (c) whitespace 191 | tokenization. We understand that accent markers have substantial meaning in some 192 | languages, but felt that the benefits of reducing the effective vocabulary make 193 | up for this. Generally the strong contextual models of BERT should make up for 194 | any ambiguity introduced by stripping accent markers. 195 | 196 | ### List of Languages 197 | 198 | The multilingual model supports the following languages. These languages were 199 | chosen because they are the top 100 languages with the largest Wikipedias: 200 | 201 | * Afrikaans 202 | * Albanian 203 | * Arabic 204 | * Aragonese 205 | * Armenian 206 | * Asturian 207 | * Azerbaijani 208 | * Bashkir 209 | * Basque 210 | * Bavarian 211 | * Belarusian 212 | * Bengali 213 | * Bishnupriya Manipuri 214 | * Bosnian 215 | * Breton 216 | * Bulgarian 217 | * Burmese 218 | * Catalan 219 | * Cebuano 220 | * Chechen 221 | * Chinese (Simplified) 222 | * Chinese (Traditional) 223 | * Chuvash 224 | * Croatian 225 | * Czech 226 | * Danish 227 | * Dutch 228 | * English 229 | * Estonian 230 | * Finnish 231 | * French 232 | * Galician 233 | * Georgian 234 | * German 235 | * Greek 236 | * Gujarati 237 | * Haitian 238 | * Hebrew 239 | * Hindi 240 | * Hungarian 241 | * Icelandic 242 | * Ido 243 | * Indonesian 244 | * Irish 245 | * Italian 246 | * Japanese 247 | * Javanese 248 | * Kannada 249 | * Kazakh 250 | * Kirghiz 251 | * Korean 252 | * Latin 253 | * Latvian 254 | * Lithuanian 255 | * Lombard 256 | * Low Saxon 257 | * Luxembourgish 258 | * Macedonian 259 | * Malagasy 260 | * Malay 261 | * Malayalam 262 | * Marathi 263 | * Minangkabau 264 | * Nepali 265 | * Newar 266 | * Norwegian (Bokmal) 267 | * Norwegian (Nynorsk) 268 | * Occitan 269 | * Persian (Farsi) 270 | * Piedmontese 271 | * Polish 272 | * Portuguese 273 | * Punjabi 274 | * Romanian 275 | * Russian 276 | * Scots 277 | * Serbian 278 | * Serbo-Croatian 279 | * Sicilian 280 | * Slovak 281 | * Slovenian 282 | * South Azerbaijani 283 | * Spanish 284 | * Sundanese 285 | * Swahili 286 | * Swedish 287 | * Tagalog 288 | * Tajik 289 | * Tamil 290 | * Tatar 291 | * Telugu 292 | * Turkish 293 | * Ukrainian 294 | * Urdu 295 | * Uzbek 296 | * Vietnamese 297 | * Volapük 298 | * Waray-Waray 299 | * Welsh 300 | * West Frisian 301 | * Western Punjabi 302 | * Yoruba 303 | 304 | The **Multilingual Cased (New)** release contains additionally **Thai** and 305 | **Mongolian**, which were not included in the original release. 306 | -------------------------------------------------------------------------------- /bert/optimization.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Functions and classes related to optimization (weight updates).""" 16 | 17 | from __future__ import absolute_import 18 | from __future__ import division 19 | from __future__ import print_function 20 | 21 | import re 22 | import tensorflow as tf 23 | 24 | 25 | def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, use_tpu): 26 | """Creates an optimizer training op.""" 27 | global_step = tf.train.get_or_create_global_step() 28 | 29 | learning_rate = tf.constant(value=init_lr, shape=[], dtype=tf.float32) 30 | 31 | # Implements linear decay of the learning rate. 32 | learning_rate = tf.train.polynomial_decay( 33 | learning_rate, 34 | global_step, 35 | num_train_steps, 36 | end_learning_rate=0.0, 37 | power=1.0, 38 | cycle=False) 39 | 40 | # Implements linear warmup. I.e., if global_step < num_warmup_steps, the 41 | # learning rate will be `global_step/num_warmup_steps * init_lr`. 42 | if num_warmup_steps: 43 | global_steps_int = tf.cast(global_step, tf.int32) 44 | warmup_steps_int = tf.constant(num_warmup_steps, dtype=tf.int32) 45 | 46 | global_steps_float = tf.cast(global_steps_int, tf.float32) 47 | warmup_steps_float = tf.cast(warmup_steps_int, tf.float32) 48 | 49 | warmup_percent_done = global_steps_float / warmup_steps_float 50 | warmup_learning_rate = init_lr * warmup_percent_done 51 | 52 | is_warmup = tf.cast(global_steps_int < warmup_steps_int, tf.float32) 53 | learning_rate = ( 54 | (1.0 - is_warmup) * learning_rate + is_warmup * warmup_learning_rate) 55 | 56 | # It is recommended that you use this optimizer for fine tuning, since this 57 | # is how the model was trained (note that the Adam m/v variables are NOT 58 | # loaded from init_checkpoint.) 59 | optimizer = AdamWeightDecayOptimizer( 60 | learning_rate=learning_rate, 61 | weight_decay_rate=0.01, 62 | beta_1=0.9, 63 | beta_2=0.999, 64 | epsilon=1e-6, 65 | exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"]) 66 | 67 | if use_tpu: 68 | optimizer = tf.contrib.tpu.CrossShardOptimizer(optimizer) 69 | 70 | tvars = tf.trainable_variables() 71 | grads = tf.gradients(loss, tvars) 72 | 73 | # This is how the model was pre-trained. 74 | (grads, _) = tf.clip_by_global_norm(grads, clip_norm=1.0) 75 | 76 | train_op = optimizer.apply_gradients( 77 | zip(grads, tvars), global_step=global_step) 78 | 79 | # Normally the global step update is done inside of `apply_gradients`. 80 | # However, `AdamWeightDecayOptimizer` doesn't do this. But if you use 81 | # a different optimizer, you should probably take this line out. 82 | new_global_step = global_step + 1 83 | train_op = tf.group(train_op, [global_step.assign(new_global_step)]) 84 | return train_op 85 | 86 | 87 | class AdamWeightDecayOptimizer(tf.train.Optimizer): 88 | """A basic Adam optimizer that includes "correct" L2 weight decay.""" 89 | 90 | def __init__(self, 91 | learning_rate, 92 | weight_decay_rate=0.0, 93 | beta_1=0.9, 94 | beta_2=0.999, 95 | epsilon=1e-6, 96 | exclude_from_weight_decay=None, 97 | name="AdamWeightDecayOptimizer"): 98 | """Constructs a AdamWeightDecayOptimizer.""" 99 | super(AdamWeightDecayOptimizer, self).__init__(False, name) 100 | 101 | self.learning_rate = learning_rate 102 | self.weight_decay_rate = weight_decay_rate 103 | self.beta_1 = beta_1 104 | self.beta_2 = beta_2 105 | self.epsilon = epsilon 106 | self.exclude_from_weight_decay = exclude_from_weight_decay 107 | 108 | def apply_gradients(self, grads_and_vars, global_step=None, name=None): 109 | """See base class.""" 110 | assignments = [] 111 | for (grad, param) in grads_and_vars: 112 | if grad is None or param is None: 113 | continue 114 | 115 | param_name = self._get_variable_name(param.name) 116 | 117 | m = tf.get_variable( 118 | name=param_name + "/adam_m", 119 | shape=param.shape.as_list(), 120 | dtype=tf.float32, 121 | trainable=False, 122 | initializer=tf.zeros_initializer()) 123 | v = tf.get_variable( 124 | name=param_name + "/adam_v", 125 | shape=param.shape.as_list(), 126 | dtype=tf.float32, 127 | trainable=False, 128 | initializer=tf.zeros_initializer()) 129 | 130 | # Standard Adam update. 131 | next_m = ( 132 | tf.multiply(self.beta_1, m) + tf.multiply(1.0 - self.beta_1, grad)) 133 | next_v = ( 134 | tf.multiply(self.beta_2, v) + tf.multiply(1.0 - self.beta_2, 135 | tf.square(grad))) 136 | 137 | update = next_m / (tf.sqrt(next_v) + self.epsilon) 138 | 139 | # Just adding the square of the weights to the loss function is *not* 140 | # the correct way of using L2 regularization/weight decay with Adam, 141 | # since that will interact with the m and v parameters in strange ways. 142 | # 143 | # Instead we want ot decay the weights in a manner that doesn't interact 144 | # with the m/v parameters. This is equivalent to adding the square 145 | # of the weights to the loss with plain (non-momentum) SGD. 146 | if self._do_use_weight_decay(param_name): 147 | update += self.weight_decay_rate * param 148 | 149 | update_with_lr = self.learning_rate * update 150 | 151 | next_param = param - update_with_lr 152 | 153 | assignments.extend( 154 | [param.assign(next_param), 155 | m.assign(next_m), 156 | v.assign(next_v)]) 157 | return tf.group(*assignments, name=name) 158 | 159 | def _do_use_weight_decay(self, param_name): 160 | """Whether to use L2 weight decay for `param_name`.""" 161 | if not self.weight_decay_rate: 162 | return False 163 | if self.exclude_from_weight_decay: 164 | for r in self.exclude_from_weight_decay: 165 | if re.search(r, param_name) is not None: 166 | return False 167 | return True 168 | 169 | def _get_variable_name(self, param_name): 170 | """Get the variable name from the tensor name.""" 171 | m = re.match("^(.*):\\d+$", param_name) 172 | if m is not None: 173 | param_name = m.group(1) 174 | return param_name 175 | -------------------------------------------------------------------------------- /bert/optimization_test.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | from __future__ import absolute_import 16 | from __future__ import division 17 | from __future__ import print_function 18 | 19 | from bert import optimization 20 | import tensorflow as tf 21 | 22 | 23 | class OptimizationTest(tf.test.TestCase): 24 | 25 | def test_adam(self): 26 | with self.test_session() as sess: 27 | w = tf.get_variable( 28 | "w", 29 | shape=[3], 30 | initializer=tf.constant_initializer([0.1, -0.2, -0.1])) 31 | x = tf.constant([0.4, 0.2, -0.5]) 32 | loss = tf.reduce_mean(tf.square(x - w)) 33 | tvars = tf.trainable_variables() 34 | grads = tf.gradients(loss, tvars) 35 | global_step = tf.train.get_or_create_global_step() 36 | optimizer = optimization.AdamWeightDecayOptimizer(learning_rate=0.2) 37 | train_op = optimizer.apply_gradients(zip(grads, tvars), global_step) 38 | init_op = tf.group(tf.global_variables_initializer(), 39 | tf.local_variables_initializer()) 40 | sess.run(init_op) 41 | for _ in range(100): 42 | sess.run(train_op) 43 | w_np = sess.run(w) 44 | self.assertAllClose(w_np.flat, [0.4, 0.2, -0.5], rtol=1e-2, atol=1e-2) 45 | 46 | 47 | if __name__ == "__main__": 48 | tf.test.main() 49 | -------------------------------------------------------------------------------- /bert/requirements.txt: -------------------------------------------------------------------------------- 1 | tensorflow >= 1.11.0 # CPU Version of TensorFlow. 2 | # tensorflow-gpu >= 1.11.0 # GPU version of TensorFlow. 3 | -------------------------------------------------------------------------------- /bert/run_classifier_with_tfhub.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """BERT finetuning runner with TF-Hub.""" 16 | 17 | from __future__ import absolute_import 18 | from __future__ import division 19 | from __future__ import print_function 20 | 21 | import os 22 | from bert import optimization 23 | from bert import run_classifier 24 | from bert import tokenization 25 | import tensorflow as tf 26 | import tensorflow_hub as hub 27 | 28 | flags = tf.flags 29 | 30 | FLAGS = flags.FLAGS 31 | 32 | flags.DEFINE_string( 33 | "bert_hub_module_handle", None, 34 | "Handle for the BERT TF-Hub module.") 35 | 36 | 37 | def create_model(is_training, input_ids, input_mask, segment_ids, labels, 38 | num_labels, bert_hub_module_handle): 39 | """Creates a classification model.""" 40 | tags = set() 41 | if is_training: 42 | tags.add("train") 43 | bert_module = hub.Module(bert_hub_module_handle, tags=tags, trainable=True) 44 | bert_inputs = dict( 45 | input_ids=input_ids, 46 | input_mask=input_mask, 47 | segment_ids=segment_ids) 48 | bert_outputs = bert_module( 49 | inputs=bert_inputs, 50 | signature="tokens", 51 | as_dict=True) 52 | 53 | # In the demo, we are doing a simple classification task on the entire 54 | # segment. 55 | # 56 | # If you want to use the token-level output, use 57 | # bert_outputs["sequence_output"] instead. 58 | output_layer = bert_outputs["pooled_output"] 59 | 60 | hidden_size = output_layer.shape[-1].value 61 | 62 | output_weights = tf.get_variable( 63 | "output_weights", [num_labels, hidden_size], 64 | initializer=tf.truncated_normal_initializer(stddev=0.02)) 65 | 66 | output_bias = tf.get_variable( 67 | "output_bias", [num_labels], initializer=tf.zeros_initializer()) 68 | 69 | with tf.variable_scope("loss"): 70 | if is_training: 71 | # I.e., 0.1 dropout 72 | output_layer = tf.nn.dropout(output_layer, keep_prob=0.9) 73 | 74 | logits = tf.matmul(output_layer, output_weights, transpose_b=True) 75 | logits = tf.nn.bias_add(logits, output_bias) 76 | probabilities = tf.nn.softmax(logits, axis=-1) 77 | log_probs = tf.nn.log_softmax(logits, axis=-1) 78 | 79 | one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32) 80 | 81 | per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1) 82 | loss = tf.reduce_mean(per_example_loss) 83 | 84 | return (loss, per_example_loss, logits, probabilities) 85 | 86 | 87 | def model_fn_builder(num_labels, learning_rate, num_train_steps, 88 | num_warmup_steps, use_tpu, bert_hub_module_handle): 89 | """Returns `model_fn` closure for TPUEstimator.""" 90 | 91 | def model_fn(features, labels, mode, params): # pylint: disable=unused-argument 92 | """The `model_fn` for TPUEstimator.""" 93 | 94 | tf.logging.info("*** Features ***") 95 | for name in sorted(features.keys()): 96 | tf.logging.info(" name = %s, shape = %s" % (name, features[name].shape)) 97 | 98 | input_ids = features["input_ids"] 99 | input_mask = features["input_mask"] 100 | segment_ids = features["segment_ids"] 101 | label_ids = features["label_ids"] 102 | 103 | is_training = (mode == tf.estimator.ModeKeys.TRAIN) 104 | 105 | (total_loss, per_example_loss, logits, probabilities) = create_model( 106 | is_training, input_ids, input_mask, segment_ids, label_ids, num_labels, 107 | bert_hub_module_handle) 108 | 109 | output_spec = None 110 | if mode == tf.estimator.ModeKeys.TRAIN: 111 | train_op = optimization.create_optimizer( 112 | total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu) 113 | 114 | output_spec = tf.contrib.tpu.TPUEstimatorSpec( 115 | mode=mode, 116 | loss=total_loss, 117 | train_op=train_op) 118 | elif mode == tf.estimator.ModeKeys.EVAL: 119 | 120 | def metric_fn(per_example_loss, label_ids, logits): 121 | predictions = tf.argmax(logits, axis=-1, output_type=tf.int32) 122 | accuracy = tf.metrics.accuracy(label_ids, predictions) 123 | loss = tf.metrics.mean(per_example_loss) 124 | return { 125 | "eval_accuracy": accuracy, 126 | "eval_loss": loss, 127 | } 128 | 129 | eval_metrics = (metric_fn, [per_example_loss, label_ids, logits]) 130 | output_spec = tf.contrib.tpu.TPUEstimatorSpec( 131 | mode=mode, 132 | loss=total_loss, 133 | eval_metrics=eval_metrics) 134 | elif mode == tf.estimator.ModeKeys.PREDICT: 135 | output_spec = tf.contrib.tpu.TPUEstimatorSpec( 136 | mode=mode, predictions={"probabilities": probabilities}) 137 | else: 138 | raise ValueError( 139 | "Only TRAIN, EVAL and PREDICT modes are supported: %s" % (mode)) 140 | 141 | return output_spec 142 | 143 | return model_fn 144 | 145 | 146 | def create_tokenizer_from_hub_module(bert_hub_module_handle): 147 | """Get the vocab file and casing info from the Hub module.""" 148 | with tf.Graph().as_default(): 149 | bert_module = hub.Module(bert_hub_module_handle) 150 | tokenization_info = bert_module(signature="tokenization_info", as_dict=True) 151 | with tf.Session() as sess: 152 | vocab_file, do_lower_case = sess.run([tokenization_info["vocab_file"], 153 | tokenization_info["do_lower_case"]]) 154 | return tokenization.FullTokenizer( 155 | vocab_file=vocab_file, do_lower_case=do_lower_case) 156 | 157 | 158 | def main(_): 159 | tf.logging.set_verbosity(tf.logging.INFO) 160 | 161 | processors = { 162 | "cola": run_classifier.ColaProcessor, 163 | "mnli": run_classifier.MnliProcessor, 164 | "mrpc": run_classifier.MrpcProcessor, 165 | } 166 | 167 | if not FLAGS.do_train and not FLAGS.do_eval: 168 | raise ValueError("At least one of `do_train` or `do_eval` must be True.") 169 | 170 | tf.gfile.MakeDirs(FLAGS.output_dir) 171 | 172 | task_name = FLAGS.task_name.lower() 173 | 174 | if task_name not in processors: 175 | raise ValueError("Task not found: %s" % (task_name)) 176 | 177 | processor = processors[task_name]() 178 | 179 | label_list = processor.get_labels() 180 | 181 | tokenizer = create_tokenizer_from_hub_module(FLAGS.bert_hub_module_handle) 182 | 183 | tpu_cluster_resolver = None 184 | if FLAGS.use_tpu and FLAGS.tpu_name: 185 | tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver( 186 | FLAGS.tpu_name, zone=FLAGS.tpu_zone, project=FLAGS.gcp_project) 187 | 188 | is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2 189 | run_config = tf.contrib.tpu.RunConfig( 190 | cluster=tpu_cluster_resolver, 191 | master=FLAGS.master, 192 | model_dir=FLAGS.output_dir, 193 | save_checkpoints_steps=FLAGS.save_checkpoints_steps, 194 | tpu_config=tf.contrib.tpu.TPUConfig( 195 | iterations_per_loop=FLAGS.iterations_per_loop, 196 | num_shards=FLAGS.num_tpu_cores, 197 | per_host_input_for_training=is_per_host)) 198 | 199 | train_examples = None 200 | num_train_steps = None 201 | num_warmup_steps = None 202 | if FLAGS.do_train: 203 | train_examples = processor.get_train_examples(FLAGS.data_dir) 204 | num_train_steps = int( 205 | len(train_examples) / FLAGS.train_batch_size * FLAGS.num_train_epochs) 206 | num_warmup_steps = int(num_train_steps * FLAGS.warmup_proportion) 207 | 208 | model_fn = model_fn_builder( 209 | num_labels=len(label_list), 210 | learning_rate=FLAGS.learning_rate, 211 | num_train_steps=num_train_steps, 212 | num_warmup_steps=num_warmup_steps, 213 | use_tpu=FLAGS.use_tpu, 214 | bert_hub_module_handle=FLAGS.bert_hub_module_handle) 215 | 216 | # If TPU is not available, this will fall back to normal Estimator on CPU 217 | # or GPU. 218 | estimator = tf.contrib.tpu.TPUEstimator( 219 | use_tpu=FLAGS.use_tpu, 220 | model_fn=model_fn, 221 | config=run_config, 222 | train_batch_size=FLAGS.train_batch_size, 223 | eval_batch_size=FLAGS.eval_batch_size, 224 | predict_batch_size=FLAGS.predict_batch_size) 225 | 226 | if FLAGS.do_train: 227 | train_features = run_classifier.convert_examples_to_features( 228 | train_examples, label_list, FLAGS.max_seq_length, tokenizer) 229 | tf.logging.info("***** Running training *****") 230 | tf.logging.info(" Num examples = %d", len(train_examples)) 231 | tf.logging.info(" Batch size = %d", FLAGS.train_batch_size) 232 | tf.logging.info(" Num steps = %d", num_train_steps) 233 | train_input_fn = run_classifier.input_fn_builder( 234 | features=train_features, 235 | seq_length=FLAGS.max_seq_length, 236 | is_training=True, 237 | drop_remainder=True) 238 | estimator.train(input_fn=train_input_fn, max_steps=num_train_steps) 239 | 240 | if FLAGS.do_eval: 241 | eval_examples = processor.get_dev_examples(FLAGS.data_dir) 242 | eval_features = run_classifier.convert_examples_to_features( 243 | eval_examples, label_list, FLAGS.max_seq_length, tokenizer) 244 | 245 | tf.logging.info("***** Running evaluation *****") 246 | tf.logging.info(" Num examples = %d", len(eval_examples)) 247 | tf.logging.info(" Batch size = %d", FLAGS.eval_batch_size) 248 | 249 | # This tells the estimator to run through the entire set. 250 | eval_steps = None 251 | # However, if running eval on the TPU, you will need to specify the 252 | # number of steps. 253 | if FLAGS.use_tpu: 254 | # Eval will be slightly WRONG on the TPU because it will truncate 255 | # the last batch. 256 | eval_steps = int(len(eval_examples) / FLAGS.eval_batch_size) 257 | 258 | eval_drop_remainder = True if FLAGS.use_tpu else False 259 | eval_input_fn = run_classifier.input_fn_builder( 260 | features=eval_features, 261 | seq_length=FLAGS.max_seq_length, 262 | is_training=False, 263 | drop_remainder=eval_drop_remainder) 264 | 265 | result = estimator.evaluate(input_fn=eval_input_fn, steps=eval_steps) 266 | 267 | output_eval_file = os.path.join(FLAGS.output_dir, "eval_results.txt") 268 | with tf.gfile.GFile(output_eval_file, "w") as writer: 269 | tf.logging.info("***** Eval results *****") 270 | for key in sorted(result.keys()): 271 | tf.logging.info(" %s = %s", key, str(result[key])) 272 | writer.write("%s = %s\n" % (key, str(result[key]))) 273 | 274 | if FLAGS.do_predict: 275 | predict_examples = processor.get_test_examples(FLAGS.data_dir) 276 | if FLAGS.use_tpu: 277 | # Discard batch remainder if running on TPU 278 | n = len(predict_examples) 279 | predict_examples = predict_examples[:(n - n % FLAGS.predict_batch_size)] 280 | 281 | predict_file = os.path.join(FLAGS.output_dir, "predict.tf_record") 282 | run_classifier.file_based_convert_examples_to_features( 283 | predict_examples, label_list, FLAGS.max_seq_length, tokenizer, 284 | predict_file) 285 | 286 | tf.logging.info("***** Running prediction*****") 287 | tf.logging.info(" Num examples = %d", len(predict_examples)) 288 | tf.logging.info(" Batch size = %d", FLAGS.predict_batch_size) 289 | 290 | predict_input_fn = run_classifier.file_based_input_fn_builder( 291 | input_file=predict_file, 292 | seq_length=FLAGS.max_seq_length, 293 | is_training=False, 294 | drop_remainder=FLAGS.use_tpu) 295 | 296 | result = estimator.predict(input_fn=predict_input_fn) 297 | 298 | output_predict_file = os.path.join(FLAGS.output_dir, "test_results.tsv") 299 | with tf.gfile.GFile(output_predict_file, "w") as writer: 300 | tf.logging.info("***** Predict results *****") 301 | for prediction in result: 302 | probabilities = prediction["probabilities"] 303 | output_line = "\t".join( 304 | str(class_probability) 305 | for class_probability in probabilities) + "\n" 306 | writer.write(output_line) 307 | 308 | 309 | if __name__ == "__main__": 310 | flags.mark_flag_as_required("data_dir") 311 | flags.mark_flag_as_required("task_name") 312 | flags.mark_flag_as_required("bert_hub_module_handle") 313 | flags.mark_flag_as_required("output_dir") 314 | tf.app.run() 315 | -------------------------------------------------------------------------------- /bert/run_pretraining.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Run masked LM/next sentence masked_lm pre-training for BERT.""" 16 | 17 | from __future__ import absolute_import 18 | from __future__ import division 19 | from __future__ import print_function 20 | 21 | import os 22 | from bert import modeling 23 | from bert import optimization 24 | import tensorflow as tf 25 | 26 | flags = tf.flags 27 | 28 | FLAGS = flags.FLAGS 29 | 30 | ## Required parameters 31 | flags.DEFINE_string( 32 | "bert_config_file", None, 33 | "The config json file corresponding to the pre-trained BERT model. " 34 | "This specifies the model architecture.") 35 | 36 | flags.DEFINE_string( 37 | "input_file", None, 38 | "Input TF example files (can be a glob or comma separated).") 39 | 40 | flags.DEFINE_string( 41 | "output_dir", None, 42 | "The output directory where the model checkpoints will be written.") 43 | 44 | ## Other parameters 45 | flags.DEFINE_string( 46 | "init_checkpoint", None, 47 | "Initial checkpoint (usually from a pre-trained BERT model).") 48 | 49 | flags.DEFINE_integer( 50 | "max_seq_length", 128, 51 | "The maximum total input sequence length after WordPiece tokenization. " 52 | "Sequences longer than this will be truncated, and sequences shorter " 53 | "than this will be padded. Must match data generation.") 54 | 55 | flags.DEFINE_integer( 56 | "max_predictions_per_seq", 20, 57 | "Maximum number of masked LM predictions per sequence. " 58 | "Must match data generation.") 59 | 60 | flags.DEFINE_bool("do_train", False, "Whether to run training.") 61 | 62 | flags.DEFINE_bool("do_eval", False, "Whether to run eval on the dev set.") 63 | 64 | flags.DEFINE_integer("train_batch_size", 32, "Total batch size for training.") 65 | 66 | flags.DEFINE_integer("eval_batch_size", 8, "Total batch size for eval.") 67 | 68 | flags.DEFINE_float("learning_rate", 5e-5, "The initial learning rate for Adam.") 69 | 70 | flags.DEFINE_integer("num_train_steps", 100000, "Number of training steps.") 71 | 72 | flags.DEFINE_integer("num_warmup_steps", 10000, "Number of warmup steps.") 73 | 74 | flags.DEFINE_integer("save_checkpoints_steps", 1000, 75 | "How often to save the model checkpoint.") 76 | 77 | flags.DEFINE_integer("iterations_per_loop", 1000, 78 | "How many steps to make in each estimator call.") 79 | 80 | flags.DEFINE_integer("max_eval_steps", 100, "Maximum number of eval steps.") 81 | 82 | flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.") 83 | 84 | tf.flags.DEFINE_string( 85 | "tpu_name", None, 86 | "The Cloud TPU to use for training. This should be either the name " 87 | "used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 " 88 | "url.") 89 | 90 | tf.flags.DEFINE_string( 91 | "tpu_zone", None, 92 | "[Optional] GCE zone where the Cloud TPU is located in. If not " 93 | "specified, we will attempt to automatically detect the GCE project from " 94 | "metadata.") 95 | 96 | tf.flags.DEFINE_string( 97 | "gcp_project", None, 98 | "[Optional] Project name for the Cloud TPU-enabled project. If not " 99 | "specified, we will attempt to automatically detect the GCE project from " 100 | "metadata.") 101 | 102 | tf.flags.DEFINE_string("master", None, "[Optional] TensorFlow master URL.") 103 | 104 | flags.DEFINE_integer( 105 | "num_tpu_cores", 8, 106 | "Only used if `use_tpu` is True. Total number of TPU cores to use.") 107 | 108 | 109 | def model_fn_builder(bert_config, init_checkpoint, learning_rate, 110 | num_train_steps, num_warmup_steps, use_tpu, 111 | use_one_hot_embeddings): 112 | """Returns `model_fn` closure for TPUEstimator.""" 113 | 114 | def model_fn(features, labels, mode, params): # pylint: disable=unused-argument 115 | """The `model_fn` for TPUEstimator.""" 116 | 117 | tf.logging.info("*** Features ***") 118 | for name in sorted(features.keys()): 119 | tf.logging.info(" name = %s, shape = %s" % (name, features[name].shape)) 120 | 121 | input_ids = features["input_ids"] 122 | input_mask = features["input_mask"] 123 | segment_ids = features["segment_ids"] 124 | masked_lm_positions = features["masked_lm_positions"] 125 | masked_lm_ids = features["masked_lm_ids"] 126 | masked_lm_weights = features["masked_lm_weights"] 127 | next_sentence_labels = features["next_sentence_labels"] 128 | 129 | is_training = (mode == tf.estimator.ModeKeys.TRAIN) 130 | 131 | model = modeling.BertModel( 132 | config=bert_config, 133 | is_training=is_training, 134 | input_ids=input_ids, 135 | input_mask=input_mask, 136 | token_type_ids=segment_ids, 137 | use_one_hot_embeddings=use_one_hot_embeddings) 138 | 139 | (masked_lm_loss, 140 | masked_lm_example_loss, masked_lm_log_probs) = get_masked_lm_output( 141 | bert_config, model.get_sequence_output(), model.get_embedding_table(), 142 | masked_lm_positions, masked_lm_ids, masked_lm_weights) 143 | 144 | (next_sentence_loss, next_sentence_example_loss, 145 | next_sentence_log_probs) = get_next_sentence_output( 146 | bert_config, model.get_pooled_output(), next_sentence_labels) 147 | 148 | total_loss = masked_lm_loss + next_sentence_loss 149 | 150 | tvars = tf.trainable_variables() 151 | 152 | initialized_variable_names = {} 153 | scaffold_fn = None 154 | if init_checkpoint: 155 | (assignment_map, initialized_variable_names 156 | ) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint) 157 | if use_tpu: 158 | 159 | def tpu_scaffold(): 160 | tf.train.init_from_checkpoint(init_checkpoint, assignment_map) 161 | return tf.train.Scaffold() 162 | 163 | scaffold_fn = tpu_scaffold 164 | else: 165 | tf.train.init_from_checkpoint(init_checkpoint, assignment_map) 166 | 167 | tf.logging.info("**** Trainable Variables ****") 168 | for var in tvars: 169 | init_string = "" 170 | if var.name in initialized_variable_names: 171 | init_string = ", *INIT_FROM_CKPT*" 172 | tf.logging.info(" name = %s, shape = %s%s", var.name, var.shape, 173 | init_string) 174 | 175 | output_spec = None 176 | if mode == tf.estimator.ModeKeys.TRAIN: 177 | train_op = optimization.create_optimizer( 178 | total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu) 179 | 180 | output_spec = tf.contrib.tpu.TPUEstimatorSpec( 181 | mode=mode, 182 | loss=total_loss, 183 | train_op=train_op, 184 | scaffold_fn=scaffold_fn) 185 | elif mode == tf.estimator.ModeKeys.EVAL: 186 | 187 | def metric_fn(masked_lm_example_loss, masked_lm_log_probs, masked_lm_ids, 188 | masked_lm_weights, next_sentence_example_loss, 189 | next_sentence_log_probs, next_sentence_labels): 190 | """Computes the loss and accuracy of the model.""" 191 | masked_lm_log_probs = tf.reshape(masked_lm_log_probs, 192 | [-1, masked_lm_log_probs.shape[-1]]) 193 | masked_lm_predictions = tf.argmax( 194 | masked_lm_log_probs, axis=-1, output_type=tf.int32) 195 | masked_lm_example_loss = tf.reshape(masked_lm_example_loss, [-1]) 196 | masked_lm_ids = tf.reshape(masked_lm_ids, [-1]) 197 | masked_lm_weights = tf.reshape(masked_lm_weights, [-1]) 198 | masked_lm_accuracy = tf.metrics.accuracy( 199 | labels=masked_lm_ids, 200 | predictions=masked_lm_predictions, 201 | weights=masked_lm_weights) 202 | masked_lm_mean_loss = tf.metrics.mean( 203 | values=masked_lm_example_loss, weights=masked_lm_weights) 204 | 205 | next_sentence_log_probs = tf.reshape( 206 | next_sentence_log_probs, [-1, next_sentence_log_probs.shape[-1]]) 207 | next_sentence_predictions = tf.argmax( 208 | next_sentence_log_probs, axis=-1, output_type=tf.int32) 209 | next_sentence_labels = tf.reshape(next_sentence_labels, [-1]) 210 | next_sentence_accuracy = tf.metrics.accuracy( 211 | labels=next_sentence_labels, predictions=next_sentence_predictions) 212 | next_sentence_mean_loss = tf.metrics.mean( 213 | values=next_sentence_example_loss) 214 | 215 | return { 216 | "masked_lm_accuracy": masked_lm_accuracy, 217 | "masked_lm_loss": masked_lm_mean_loss, 218 | "next_sentence_accuracy": next_sentence_accuracy, 219 | "next_sentence_loss": next_sentence_mean_loss, 220 | } 221 | 222 | eval_metrics = (metric_fn, [ 223 | masked_lm_example_loss, masked_lm_log_probs, masked_lm_ids, 224 | masked_lm_weights, next_sentence_example_loss, 225 | next_sentence_log_probs, next_sentence_labels 226 | ]) 227 | output_spec = tf.contrib.tpu.TPUEstimatorSpec( 228 | mode=mode, 229 | loss=total_loss, 230 | eval_metrics=eval_metrics, 231 | scaffold_fn=scaffold_fn) 232 | else: 233 | raise ValueError("Only TRAIN and EVAL modes are supported: %s" % (mode)) 234 | 235 | return output_spec 236 | 237 | return model_fn 238 | 239 | 240 | def get_masked_lm_output(bert_config, input_tensor, output_weights, positions, 241 | label_ids, label_weights): 242 | """Get loss and log probs for the masked LM.""" 243 | input_tensor = gather_indexes(input_tensor, positions) 244 | 245 | with tf.variable_scope("cls/predictions"): 246 | # We apply one more non-linear transformation before the output layer. 247 | # This matrix is not used after pre-training. 248 | with tf.variable_scope("transform"): 249 | input_tensor = tf.layers.dense( 250 | input_tensor, 251 | units=bert_config.hidden_size, 252 | activation=modeling.get_activation(bert_config.hidden_act), 253 | kernel_initializer=modeling.create_initializer( 254 | bert_config.initializer_range)) 255 | input_tensor = modeling.layer_norm(input_tensor) 256 | 257 | # The output weights are the same as the input embeddings, but there is 258 | # an output-only bias for each token. 259 | output_bias = tf.get_variable( 260 | "output_bias", 261 | shape=[bert_config.vocab_size], 262 | initializer=tf.zeros_initializer()) 263 | logits = tf.matmul(input_tensor, output_weights, transpose_b=True) 264 | logits = tf.nn.bias_add(logits, output_bias) 265 | log_probs = tf.nn.log_softmax(logits, axis=-1) 266 | 267 | label_ids = tf.reshape(label_ids, [-1]) 268 | label_weights = tf.reshape(label_weights, [-1]) 269 | 270 | one_hot_labels = tf.one_hot( 271 | label_ids, depth=bert_config.vocab_size, dtype=tf.float32) 272 | 273 | # The `positions` tensor might be zero-padded (if the sequence is too 274 | # short to have the maximum number of predictions). The `label_weights` 275 | # tensor has a value of 1.0 for every real prediction and 0.0 for the 276 | # padding predictions. 277 | per_example_loss = -tf.reduce_sum(log_probs * one_hot_labels, axis=[-1]) 278 | numerator = tf.reduce_sum(label_weights * per_example_loss) 279 | denominator = tf.reduce_sum(label_weights) + 1e-5 280 | loss = numerator / denominator 281 | 282 | return (loss, per_example_loss, log_probs) 283 | 284 | 285 | def get_next_sentence_output(bert_config, input_tensor, labels): 286 | """Get loss and log probs for the next sentence prediction.""" 287 | 288 | # Simple binary classification. Note that 0 is "next sentence" and 1 is 289 | # "random sentence". This weight matrix is not used after pre-training. 290 | with tf.variable_scope("cls/seq_relationship"): 291 | output_weights = tf.get_variable( 292 | "output_weights", 293 | shape=[2, bert_config.hidden_size], 294 | initializer=modeling.create_initializer(bert_config.initializer_range)) 295 | output_bias = tf.get_variable( 296 | "output_bias", shape=[2], initializer=tf.zeros_initializer()) 297 | 298 | logits = tf.matmul(input_tensor, output_weights, transpose_b=True) 299 | logits = tf.nn.bias_add(logits, output_bias) 300 | log_probs = tf.nn.log_softmax(logits, axis=-1) 301 | labels = tf.reshape(labels, [-1]) 302 | one_hot_labels = tf.one_hot(labels, depth=2, dtype=tf.float32) 303 | per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1) 304 | loss = tf.reduce_mean(per_example_loss) 305 | return (loss, per_example_loss, log_probs) 306 | 307 | 308 | def gather_indexes(sequence_tensor, positions): 309 | """Gathers the vectors at the specific positions over a minibatch.""" 310 | sequence_shape = modeling.get_shape_list(sequence_tensor, expected_rank=3) 311 | batch_size = sequence_shape[0] 312 | seq_length = sequence_shape[1] 313 | width = sequence_shape[2] 314 | 315 | flat_offsets = tf.reshape( 316 | tf.range(0, batch_size, dtype=tf.int32) * seq_length, [-1, 1]) 317 | flat_positions = tf.reshape(positions + flat_offsets, [-1]) 318 | flat_sequence_tensor = tf.reshape(sequence_tensor, 319 | [batch_size * seq_length, width]) 320 | output_tensor = tf.gather(flat_sequence_tensor, flat_positions) 321 | return output_tensor 322 | 323 | 324 | def input_fn_builder(input_files, 325 | max_seq_length, 326 | max_predictions_per_seq, 327 | is_training, 328 | num_cpu_threads=4): 329 | """Creates an `input_fn` closure to be passed to TPUEstimator.""" 330 | 331 | def input_fn(params): 332 | """The actual input function.""" 333 | batch_size = params["batch_size"] 334 | 335 | name_to_features = { 336 | "input_ids": 337 | tf.FixedLenFeature([max_seq_length], tf.int64), 338 | "input_mask": 339 | tf.FixedLenFeature([max_seq_length], tf.int64), 340 | "segment_ids": 341 | tf.FixedLenFeature([max_seq_length], tf.int64), 342 | "masked_lm_positions": 343 | tf.FixedLenFeature([max_predictions_per_seq], tf.int64), 344 | "masked_lm_ids": 345 | tf.FixedLenFeature([max_predictions_per_seq], tf.int64), 346 | "masked_lm_weights": 347 | tf.FixedLenFeature([max_predictions_per_seq], tf.float32), 348 | "next_sentence_labels": 349 | tf.FixedLenFeature([1], tf.int64), 350 | } 351 | 352 | # For training, we want a lot of parallel reading and shuffling. 353 | # For eval, we want no shuffling and parallel reading doesn't matter. 354 | if is_training: 355 | d = tf.data.Dataset.from_tensor_slices(tf.constant(input_files)) 356 | d = d.repeat() 357 | d = d.shuffle(buffer_size=len(input_files)) 358 | 359 | # `cycle_length` is the number of parallel files that get read. 360 | cycle_length = min(num_cpu_threads, len(input_files)) 361 | 362 | # `sloppy` mode means that the interleaving is not exact. This adds 363 | # even more randomness to the training pipeline. 364 | d = d.apply( 365 | tf.contrib.data.parallel_interleave( 366 | tf.data.TFRecordDataset, 367 | sloppy=is_training, 368 | cycle_length=cycle_length)) 369 | d = d.shuffle(buffer_size=100) 370 | else: 371 | d = tf.data.TFRecordDataset(input_files) 372 | # Since we evaluate for a fixed number of steps we don't want to encounter 373 | # out-of-range exceptions. 374 | d = d.repeat() 375 | 376 | # We must `drop_remainder` on training because the TPU requires fixed 377 | # size dimensions. For eval, we assume we are evaluating on the CPU or GPU 378 | # and we *don't* want to drop the remainder, otherwise we wont cover 379 | # every sample. 380 | d = d.apply( 381 | tf.contrib.data.map_and_batch( 382 | lambda record: _decode_record(record, name_to_features), 383 | batch_size=batch_size, 384 | num_parallel_batches=num_cpu_threads, 385 | drop_remainder=True)) 386 | return d 387 | 388 | return input_fn 389 | 390 | 391 | def _decode_record(record, name_to_features): 392 | """Decodes a record to a TensorFlow example.""" 393 | example = tf.parse_single_example(record, name_to_features) 394 | 395 | # tf.Example only supports tf.int64, but the TPU only supports tf.int32. 396 | # So cast all int64 to int32. 397 | for name in list(example.keys()): 398 | t = example[name] 399 | if t.dtype == tf.int64: 400 | t = tf.to_int32(t) 401 | example[name] = t 402 | 403 | return example 404 | 405 | 406 | def main(_): 407 | tf.logging.set_verbosity(tf.logging.INFO) 408 | 409 | if not FLAGS.do_train and not FLAGS.do_eval: 410 | raise ValueError("At least one of `do_train` or `do_eval` must be True.") 411 | 412 | bert_config = modeling.BertConfig.from_json_file(FLAGS.bert_config_file) 413 | 414 | tf.gfile.MakeDirs(FLAGS.output_dir) 415 | 416 | input_files = [] 417 | for input_pattern in FLAGS.input_file.split(","): 418 | input_files.extend(tf.gfile.Glob(input_pattern)) 419 | 420 | tf.logging.info("*** Input Files ***") 421 | for input_file in input_files: 422 | tf.logging.info(" %s" % input_file) 423 | 424 | tpu_cluster_resolver = None 425 | if FLAGS.use_tpu and FLAGS.tpu_name: 426 | tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver( 427 | FLAGS.tpu_name, zone=FLAGS.tpu_zone, project=FLAGS.gcp_project) 428 | 429 | is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2 430 | run_config = tf.contrib.tpu.RunConfig( 431 | cluster=tpu_cluster_resolver, 432 | master=FLAGS.master, 433 | model_dir=FLAGS.output_dir, 434 | save_checkpoints_steps=FLAGS.save_checkpoints_steps, 435 | tpu_config=tf.contrib.tpu.TPUConfig( 436 | iterations_per_loop=FLAGS.iterations_per_loop, 437 | num_shards=FLAGS.num_tpu_cores, 438 | per_host_input_for_training=is_per_host)) 439 | 440 | model_fn = model_fn_builder( 441 | bert_config=bert_config, 442 | init_checkpoint=FLAGS.init_checkpoint, 443 | learning_rate=FLAGS.learning_rate, 444 | num_train_steps=FLAGS.num_train_steps, 445 | num_warmup_steps=FLAGS.num_warmup_steps, 446 | use_tpu=FLAGS.use_tpu, 447 | use_one_hot_embeddings=FLAGS.use_tpu) 448 | 449 | # If TPU is not available, this will fall back to normal Estimator on CPU 450 | # or GPU. 451 | estimator = tf.contrib.tpu.TPUEstimator( 452 | use_tpu=FLAGS.use_tpu, 453 | model_fn=model_fn, 454 | config=run_config, 455 | train_batch_size=FLAGS.train_batch_size, 456 | eval_batch_size=FLAGS.eval_batch_size) 457 | 458 | if FLAGS.do_train: 459 | tf.logging.info("***** Running training *****") 460 | tf.logging.info(" Batch size = %d", FLAGS.train_batch_size) 461 | train_input_fn = input_fn_builder( 462 | input_files=input_files, 463 | max_seq_length=FLAGS.max_seq_length, 464 | max_predictions_per_seq=FLAGS.max_predictions_per_seq, 465 | is_training=True) 466 | estimator.train(input_fn=train_input_fn, max_steps=FLAGS.num_train_steps) 467 | 468 | if FLAGS.do_eval: 469 | tf.logging.info("***** Running evaluation *****") 470 | tf.logging.info(" Batch size = %d", FLAGS.eval_batch_size) 471 | 472 | eval_input_fn = input_fn_builder( 473 | input_files=input_files, 474 | max_seq_length=FLAGS.max_seq_length, 475 | max_predictions_per_seq=FLAGS.max_predictions_per_seq, 476 | is_training=False) 477 | 478 | result = estimator.evaluate( 479 | input_fn=eval_input_fn, steps=FLAGS.max_eval_steps) 480 | 481 | output_eval_file = os.path.join(FLAGS.output_dir, "eval_results.txt") 482 | with tf.gfile.GFile(output_eval_file, "w") as writer: 483 | tf.logging.info("***** Eval results *****") 484 | for key in sorted(result.keys()): 485 | tf.logging.info(" %s = %s", key, str(result[key])) 486 | writer.write("%s = %s\n" % (key, str(result[key]))) 487 | 488 | 489 | if __name__ == "__main__": 490 | flags.mark_flag_as_required("input_file") 491 | flags.mark_flag_as_required("bert_config_file") 492 | flags.mark_flag_as_required("output_dir") 493 | tf.app.run() 494 | -------------------------------------------------------------------------------- /bert/sample_text.txt: -------------------------------------------------------------------------------- 1 | This text is included to make sure Unicode is handled properly: 力加勝北区ᴵᴺᵀᵃছজটডণত 2 | Text should be one-sentence-per-line, with empty lines between documents. 3 | This sample text is public domain and was randomly selected from Project Guttenberg. 4 | 5 | The rain had only ceased with the gray streaks of morning at Blazing Star, and the settlement awoke to a moral sense of cleanliness, and the finding of forgotten knives, tin cups, and smaller camp utensils, where the heavy showers had washed away the debris and dust heaps before the cabin doors. 6 | Indeed, it was recorded in Blazing Star that a fortunate early riser had once picked up on the highway a solid chunk of gold quartz which the rain had freed from its incumbering soil, and washed into immediate and glittering popularity. 7 | Possibly this may have been the reason why early risers in that locality, during the rainy season, adopted a thoughtful habit of body, and seldom lifted their eyes to the rifted or india-ink washed skies above them. 8 | "Cass" Beard had risen early that morning, but not with a view to discovery. 9 | A leak in his cabin roof,--quite consistent with his careless, improvident habits,--had roused him at 4 A. M., with a flooded "bunk" and wet blankets. 10 | The chips from his wood pile refused to kindle a fire to dry his bed-clothes, and he had recourse to a more provident neighbor's to supply the deficiency. 11 | This was nearly opposite. 12 | Mr. Cassius crossed the highway, and stopped suddenly. 13 | Something glittered in the nearest red pool before him. 14 | Gold, surely! 15 | But, wonderful to relate, not an irregular, shapeless fragment of crude ore, fresh from Nature's crucible, but a bit of jeweler's handicraft in the form of a plain gold ring. 16 | Looking at it more attentively, he saw that it bore the inscription, "May to Cass." 17 | Like most of his fellow gold-seekers, Cass was superstitious. 18 | 19 | The fountain of classic wisdom, Hypatia herself. 20 | As the ancient sage--the name is unimportant to a monk--pumped water nightly that he might study by day, so I, the guardian of cloaks and parasols, at the sacred doors of her lecture-room, imbibe celestial knowledge. 21 | From my youth I felt in me a soul above the matter-entangled herd. 22 | She revealed to me the glorious fact, that I am a spark of Divinity itself. 23 | A fallen star, I am, sir!' continued he, pensively, stroking his lean stomach--'a fallen star!--fallen, if the dignity of philosophy will allow of the simile, among the hogs of the lower world--indeed, even into the hog-bucket itself. Well, after all, I will show you the way to the Archbishop's. 24 | There is a philosophic pleasure in opening one's treasures to the modest young. 25 | Perhaps you will assist me by carrying this basket of fruit?' And the little man jumped up, put his basket on Philammon's head, and trotted off up a neighbouring street. 26 | Philammon followed, half contemptuous, half wondering at what this philosophy might be, which could feed the self-conceit of anything so abject as his ragged little apish guide; 27 | but the novel roar and whirl of the street, the perpetual stream of busy faces, the line of curricles, palanquins, laden asses, camels, elephants, which met and passed him, and squeezed him up steps and into doorways, as they threaded their way through the great Moon-gate into the ample street beyond, drove everything from his mind but wondering curiosity, and a vague, helpless dread of that great living wilderness, more terrible than any dead wilderness of sand which he had left behind. 28 | Already he longed for the repose, the silence of the Laura--for faces which knew him and smiled upon him; but it was too late to turn back now. 29 | His guide held on for more than a mile up the great main street, crossed in the centre of the city, at right angles, by one equally magnificent, at each end of which, miles away, appeared, dim and distant over the heads of the living stream of passengers, the yellow sand-hills of the desert; 30 | while at the end of the vista in front of them gleamed the blue harbour, through a network of countless masts. 31 | At last they reached the quay at the opposite end of the street; 32 | and there burst on Philammon's astonished eyes a vast semicircle of blue sea, ringed with palaces and towers. 33 | He stopped involuntarily; and his little guide stopped also, and looked askance at the young monk, to watch the effect which that grand panorama should produce on him. 34 | -------------------------------------------------------------------------------- /bert/tokenization.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Tokenization classes.""" 16 | 17 | from __future__ import absolute_import 18 | from __future__ import division 19 | from __future__ import print_function 20 | 21 | import collections 22 | import re 23 | import unicodedata 24 | import six 25 | import tensorflow as tf 26 | 27 | 28 | def validate_case_matches_checkpoint(do_lower_case, init_checkpoint): 29 | """Checks whether the casing config is consistent with the checkpoint name.""" 30 | 31 | # The casing has to be passed in by the user and there is no explicit check 32 | # as to whether it matches the checkpoint. The casing information probably 33 | # should have been stored in the bert_config.json file, but it's not, so 34 | # we have to heuristically detect it to validate. 35 | 36 | if not init_checkpoint: 37 | return 38 | 39 | m = re.match("^.*?([A-Za-z0-9_-]+)/bert_model.ckpt", init_checkpoint) 40 | if m is None: 41 | return 42 | 43 | model_name = m.group(1) 44 | 45 | lower_models = [ 46 | "uncased_L-24_H-1024_A-16", "uncased_L-12_H-768_A-12", 47 | "multilingual_L-12_H-768_A-12", "chinese_L-12_H-768_A-12" 48 | ] 49 | 50 | cased_models = [ 51 | "cased_L-12_H-768_A-12", "cased_L-24_H-1024_A-16", 52 | "multi_cased_L-12_H-768_A-12" 53 | ] 54 | 55 | is_bad_config = False 56 | if model_name in lower_models and not do_lower_case: 57 | is_bad_config = True 58 | actual_flag = "False" 59 | case_name = "lowercased" 60 | opposite_flag = "True" 61 | 62 | if model_name in cased_models and do_lower_case: 63 | is_bad_config = True 64 | actual_flag = "True" 65 | case_name = "cased" 66 | opposite_flag = "False" 67 | 68 | if is_bad_config: 69 | raise ValueError( 70 | "You passed in `--do_lower_case=%s` with `--init_checkpoint=%s`. " 71 | "However, `%s` seems to be a %s model, so you " 72 | "should pass in `--do_lower_case=%s` so that the fine-tuning matches " 73 | "how the model was pre-training. If this error is wrong, please " 74 | "just comment out this check." % (actual_flag, init_checkpoint, 75 | model_name, case_name, opposite_flag)) 76 | 77 | 78 | def convert_to_unicode(text): 79 | """Converts `text` to Unicode (if it's not already), assuming utf-8 input.""" 80 | if six.PY3: 81 | if isinstance(text, str): 82 | return text 83 | elif isinstance(text, bytes): 84 | return text.decode("utf-8", "ignore") 85 | else: 86 | raise ValueError("Unsupported string type: %s" % (type(text))) 87 | elif six.PY2: 88 | if isinstance(text, str): 89 | return text.decode("utf-8", "ignore") 90 | elif isinstance(text, unicode): 91 | return text 92 | else: 93 | raise ValueError("Unsupported string type: %s" % (type(text))) 94 | else: 95 | raise ValueError("Not running on Python2 or Python 3?") 96 | 97 | 98 | def printable_text(text): 99 | """Returns text encoded in a way suitable for print or `tf.logging`.""" 100 | 101 | # These functions want `str` for both Python2 and Python3, but in one case 102 | # it's a Unicode string and in the other it's a byte string. 103 | if six.PY3: 104 | if isinstance(text, str): 105 | return text 106 | elif isinstance(text, bytes): 107 | return text.decode("utf-8", "ignore") 108 | else: 109 | raise ValueError("Unsupported string type: %s" % (type(text))) 110 | elif six.PY2: 111 | if isinstance(text, str): 112 | return text 113 | elif isinstance(text, unicode): 114 | return text.encode("utf-8") 115 | else: 116 | raise ValueError("Unsupported string type: %s" % (type(text))) 117 | else: 118 | raise ValueError("Not running on Python2 or Python 3?") 119 | 120 | 121 | def load_vocab(vocab_file): 122 | """Loads a vocabulary file into a dictionary.""" 123 | vocab = collections.OrderedDict() 124 | index = 0 125 | with tf.gfile.GFile(vocab_file, "r") as reader: 126 | while True: 127 | token = convert_to_unicode(reader.readline()) 128 | if not token: 129 | break 130 | token = token.strip() 131 | vocab[token] = index 132 | index += 1 133 | return vocab 134 | 135 | 136 | def convert_by_vocab(vocab, items): 137 | """Converts a sequence of [tokens|ids] using the vocab.""" 138 | output = [] 139 | for item in items: 140 | output.append(vocab[item]) 141 | return output 142 | 143 | 144 | def convert_tokens_to_ids(vocab, tokens): 145 | return convert_by_vocab(vocab, tokens) 146 | 147 | 148 | def convert_ids_to_tokens(inv_vocab, ids): 149 | return convert_by_vocab(inv_vocab, ids) 150 | 151 | 152 | def whitespace_tokenize(text): 153 | """Runs basic whitespace cleaning and splitting on a piece of text.""" 154 | text = text.strip() 155 | if not text: 156 | return [] 157 | tokens = text.split() 158 | return tokens 159 | 160 | 161 | class FullTokenizer(object): 162 | """Runs end-to-end tokenziation.""" 163 | 164 | def __init__(self, vocab_file, do_lower_case=True): 165 | self.vocab = load_vocab(vocab_file) 166 | self.inv_vocab = {v: k for k, v in self.vocab.items()} 167 | self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case) 168 | self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab) 169 | 170 | def tokenize(self, text): 171 | split_tokens = [] 172 | for token in self.basic_tokenizer.tokenize(text): 173 | for sub_token in self.wordpiece_tokenizer.tokenize(token): 174 | split_tokens.append(sub_token) 175 | 176 | return split_tokens 177 | 178 | def convert_tokens_to_ids(self, tokens): 179 | return convert_by_vocab(self.vocab, tokens) 180 | 181 | def convert_ids_to_tokens(self, ids): 182 | return convert_by_vocab(self.inv_vocab, ids) 183 | 184 | 185 | class BasicTokenizer(object): 186 | """Runs basic tokenization (punctuation splitting, lower casing, etc.).""" 187 | 188 | def __init__(self, do_lower_case=True): 189 | """Constructs a BasicTokenizer. 190 | 191 | Args: 192 | do_lower_case: Whether to lower case the input. 193 | """ 194 | self.do_lower_case = do_lower_case 195 | 196 | def tokenize(self, text): 197 | """Tokenizes a piece of text.""" 198 | text = convert_to_unicode(text) 199 | text = self._clean_text(text) 200 | 201 | # This was added on November 1st, 2018 for the multilingual and Chinese 202 | # models. This is also applied to the English models now, but it doesn't 203 | # matter since the English models were not trained on any Chinese data 204 | # and generally don't have any Chinese data in them (there are Chinese 205 | # characters in the vocabulary because Wikipedia does have some Chinese 206 | # words in the English Wikipedia.). 207 | text = self._tokenize_chinese_chars(text) 208 | 209 | orig_tokens = whitespace_tokenize(text) 210 | split_tokens = [] 211 | for token in orig_tokens: 212 | if self.do_lower_case: 213 | token = token.lower() 214 | token = self._run_strip_accents(token) 215 | split_tokens.extend(self._run_split_on_punc(token)) 216 | 217 | output_tokens = whitespace_tokenize(" ".join(split_tokens)) 218 | return output_tokens 219 | 220 | def _run_strip_accents(self, text): 221 | """Strips accents from a piece of text.""" 222 | text = unicodedata.normalize("NFD", text) 223 | output = [] 224 | for char in text: 225 | cat = unicodedata.category(char) 226 | if cat == "Mn": 227 | continue 228 | output.append(char) 229 | return "".join(output) 230 | 231 | def _run_split_on_punc(self, text): 232 | """Splits punctuation on a piece of text.""" 233 | chars = list(text) 234 | i = 0 235 | start_new_word = True 236 | output = [] 237 | while i < len(chars): 238 | char = chars[i] 239 | if _is_punctuation(char): 240 | output.append([char]) 241 | start_new_word = True 242 | else: 243 | if start_new_word: 244 | output.append([]) 245 | start_new_word = False 246 | output[-1].append(char) 247 | i += 1 248 | 249 | return ["".join(x) for x in output] 250 | 251 | def _tokenize_chinese_chars(self, text): 252 | """Adds whitespace around any CJK character.""" 253 | output = [] 254 | for char in text: 255 | cp = ord(char) 256 | if self._is_chinese_char(cp): 257 | output.append(" ") 258 | output.append(char) 259 | output.append(" ") 260 | else: 261 | output.append(char) 262 | return "".join(output) 263 | 264 | def _is_chinese_char(self, cp): 265 | """Checks whether CP is the codepoint of a CJK character.""" 266 | # This defines a "chinese character" as anything in the CJK Unicode block: 267 | # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block) 268 | # 269 | # Note that the CJK Unicode block is NOT all Japanese and Korean characters, 270 | # despite its name. The modern Korean Hangul alphabet is a different block, 271 | # as is Japanese Hiragana and Katakana. Those alphabets are used to write 272 | # space-separated words, so they are not treated specially and handled 273 | # like the all of the other languages. 274 | if ((cp >= 0x4E00 and cp <= 0x9FFF) or # 275 | (cp >= 0x3400 and cp <= 0x4DBF) or # 276 | (cp >= 0x20000 and cp <= 0x2A6DF) or # 277 | (cp >= 0x2A700 and cp <= 0x2B73F) or # 278 | (cp >= 0x2B740 and cp <= 0x2B81F) or # 279 | (cp >= 0x2B820 and cp <= 0x2CEAF) or 280 | (cp >= 0xF900 and cp <= 0xFAFF) or # 281 | (cp >= 0x2F800 and cp <= 0x2FA1F)): # 282 | return True 283 | 284 | return False 285 | 286 | def _clean_text(self, text): 287 | """Performs invalid character removal and whitespace cleanup on text.""" 288 | output = [] 289 | for char in text: 290 | cp = ord(char) 291 | if cp == 0 or cp == 0xfffd or _is_control(char): 292 | continue 293 | if _is_whitespace(char): 294 | output.append(" ") 295 | else: 296 | output.append(char) 297 | return "".join(output) 298 | 299 | 300 | class WordpieceTokenizer(object): 301 | """Runs WordPiece tokenziation.""" 302 | 303 | def __init__(self, vocab, unk_token="[UNK]", max_input_chars_per_word=200): 304 | self.vocab = vocab 305 | self.unk_token = unk_token 306 | self.max_input_chars_per_word = max_input_chars_per_word 307 | 308 | def tokenize(self, text): 309 | """Tokenizes a piece of text into its word pieces. 310 | 311 | This uses a greedy longest-match-first algorithm to perform tokenization 312 | using the given vocabulary. 313 | 314 | For example: 315 | input = "unaffable" 316 | output = ["un", "##aff", "##able"] 317 | 318 | Args: 319 | text: A single token or whitespace separated tokens. This should have 320 | already been passed through `BasicTokenizer. 321 | 322 | Returns: 323 | A list of wordpiece tokens. 324 | """ 325 | 326 | text = convert_to_unicode(text) 327 | 328 | output_tokens = [] 329 | for token in whitespace_tokenize(text): 330 | chars = list(token) 331 | if len(chars) > self.max_input_chars_per_word: 332 | output_tokens.append(self.unk_token) 333 | continue 334 | 335 | is_bad = False 336 | start = 0 337 | sub_tokens = [] 338 | while start < len(chars): 339 | end = len(chars) 340 | cur_substr = None 341 | while start < end: 342 | substr = "".join(chars[start:end]) 343 | if start > 0: 344 | substr = "##" + substr 345 | if substr in self.vocab: 346 | cur_substr = substr 347 | break 348 | end -= 1 349 | if cur_substr is None: 350 | is_bad = True 351 | break 352 | sub_tokens.append(cur_substr) 353 | start = end 354 | 355 | if is_bad: 356 | output_tokens.append(self.unk_token) 357 | else: 358 | output_tokens.extend(sub_tokens) 359 | return output_tokens 360 | 361 | 362 | def _is_whitespace(char): 363 | """Checks whether `chars` is a whitespace character.""" 364 | # \t, \n, and \r are technically contorl characters but we treat them 365 | # as whitespace since they are generally considered as such. 366 | if char == " " or char == "\t" or char == "\n" or char == "\r": 367 | return True 368 | cat = unicodedata.category(char) 369 | if cat == "Zs": 370 | return True 371 | return False 372 | 373 | 374 | def _is_control(char): 375 | """Checks whether `chars` is a control character.""" 376 | # These are technically control characters but we count them as whitespace 377 | # characters. 378 | if char == "\t" or char == "\n" or char == "\r": 379 | return False 380 | cat = unicodedata.category(char) 381 | if cat in ("Cc", "Cf"): 382 | return True 383 | return False 384 | 385 | 386 | def _is_punctuation(char): 387 | """Checks whether `chars` is a punctuation character.""" 388 | cp = ord(char) 389 | # We treat all non-letter/number ASCII as punctuation. 390 | # Characters such as "^", "$", and "`" are not in the Unicode 391 | # Punctuation class but we treat them as punctuation anyways, for 392 | # consistency. 393 | if ((cp >= 33 and cp <= 47) or (cp >= 58 and cp <= 64) or 394 | (cp >= 91 and cp <= 96) or (cp >= 123 and cp <= 126)): 395 | return True 396 | cat = unicodedata.category(char) 397 | if cat.startswith("P"): 398 | return True 399 | return False 400 | -------------------------------------------------------------------------------- /bert/tokenization_test.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | from __future__ import absolute_import 16 | from __future__ import division 17 | from __future__ import print_function 18 | 19 | import os 20 | import tempfile 21 | from bert import tokenization 22 | import six 23 | import tensorflow as tf 24 | 25 | 26 | class TokenizationTest(tf.test.TestCase): 27 | 28 | def test_full_tokenizer(self): 29 | vocab_tokens = [ 30 | "[UNK]", "[CLS]", "[SEP]", "want", "##want", "##ed", "wa", "un", "runn", 31 | "##ing", "," 32 | ] 33 | with tempfile.NamedTemporaryFile(delete=False) as vocab_writer: 34 | if six.PY2: 35 | vocab_writer.write("".join([x + "\n" for x in vocab_tokens])) 36 | else: 37 | vocab_writer.write("".join( 38 | [x + "\n" for x in vocab_tokens]).encode("utf-8")) 39 | 40 | vocab_file = vocab_writer.name 41 | 42 | tokenizer = tokenization.FullTokenizer(vocab_file) 43 | os.unlink(vocab_file) 44 | 45 | tokens = tokenizer.tokenize(u"UNwant\u00E9d,running") 46 | self.assertAllEqual(tokens, ["un", "##want", "##ed", ",", "runn", "##ing"]) 47 | 48 | self.assertAllEqual( 49 | tokenizer.convert_tokens_to_ids(tokens), [7, 4, 5, 10, 8, 9]) 50 | 51 | def test_chinese(self): 52 | tokenizer = tokenization.BasicTokenizer() 53 | 54 | self.assertAllEqual( 55 | tokenizer.tokenize(u"ah\u535A\u63A8zz"), 56 | [u"ah", u"\u535A", u"\u63A8", u"zz"]) 57 | 58 | def test_basic_tokenizer_lower(self): 59 | tokenizer = tokenization.BasicTokenizer(do_lower_case=True) 60 | 61 | self.assertAllEqual( 62 | tokenizer.tokenize(u" \tHeLLo!how \n Are yoU? "), 63 | ["hello", "!", "how", "are", "you", "?"]) 64 | self.assertAllEqual(tokenizer.tokenize(u"H\u00E9llo"), ["hello"]) 65 | 66 | def test_basic_tokenizer_no_lower(self): 67 | tokenizer = tokenization.BasicTokenizer(do_lower_case=False) 68 | 69 | self.assertAllEqual( 70 | tokenizer.tokenize(u" \tHeLLo!how \n Are yoU? "), 71 | ["HeLLo", "!", "how", "Are", "yoU", "?"]) 72 | 73 | def test_wordpiece_tokenizer(self): 74 | vocab_tokens = [ 75 | "[UNK]", "[CLS]", "[SEP]", "want", "##want", "##ed", "wa", "un", "runn", 76 | "##ing" 77 | ] 78 | 79 | vocab = {} 80 | for (i, token) in enumerate(vocab_tokens): 81 | vocab[token] = i 82 | tokenizer = tokenization.WordpieceTokenizer(vocab=vocab) 83 | 84 | self.assertAllEqual(tokenizer.tokenize(""), []) 85 | 86 | self.assertAllEqual( 87 | tokenizer.tokenize("unwanted running"), 88 | ["un", "##want", "##ed", "runn", "##ing"]) 89 | 90 | self.assertAllEqual( 91 | tokenizer.tokenize("unwantedX running"), ["[UNK]", "runn", "##ing"]) 92 | 93 | def test_convert_tokens_to_ids(self): 94 | vocab_tokens = [ 95 | "[UNK]", "[CLS]", "[SEP]", "want", "##want", "##ed", "wa", "un", "runn", 96 | "##ing" 97 | ] 98 | 99 | vocab = {} 100 | for (i, token) in enumerate(vocab_tokens): 101 | vocab[token] = i 102 | 103 | self.assertAllEqual( 104 | tokenization.convert_tokens_to_ids( 105 | vocab, ["un", "##want", "##ed", "runn", "##ing"]), [7, 4, 5, 8, 9]) 106 | 107 | def test_is_whitespace(self): 108 | self.assertTrue(tokenization._is_whitespace(u" ")) 109 | self.assertTrue(tokenization._is_whitespace(u"\t")) 110 | self.assertTrue(tokenization._is_whitespace(u"\r")) 111 | self.assertTrue(tokenization._is_whitespace(u"\n")) 112 | self.assertTrue(tokenization._is_whitespace(u"\u00A0")) 113 | 114 | self.assertFalse(tokenization._is_whitespace(u"A")) 115 | self.assertFalse(tokenization._is_whitespace(u"-")) 116 | 117 | def test_is_control(self): 118 | self.assertTrue(tokenization._is_control(u"\u0005")) 119 | 120 | self.assertFalse(tokenization._is_control(u"A")) 121 | self.assertFalse(tokenization._is_control(u" ")) 122 | self.assertFalse(tokenization._is_control(u"\t")) 123 | self.assertFalse(tokenization._is_control(u"\r")) 124 | self.assertFalse(tokenization._is_control(u"\U0001F4A9")) 125 | 126 | def test_is_punctuation(self): 127 | self.assertTrue(tokenization._is_punctuation(u"-")) 128 | self.assertTrue(tokenization._is_punctuation(u"$")) 129 | self.assertTrue(tokenization._is_punctuation(u"`")) 130 | self.assertTrue(tokenization._is_punctuation(u".")) 131 | 132 | self.assertFalse(tokenization._is_punctuation(u"A")) 133 | self.assertFalse(tokenization._is_punctuation(u" ")) 134 | 135 | 136 | if __name__ == "__main__": 137 | tf.test.main() 138 | -------------------------------------------------------------------------------- /convert.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | #-*- coding:utf-8 -*- 3 | # author:huanghui 4 | # datetime:2019/9/30 10:12 5 | from utils.ckpt2pb import convert 6 | 7 | """ 8 | convert调用参数说明 9 | :param task: 任务名，divorce，labor， loan 10 | :param tagDir: 任务标签文件，tags.txt 11 | :param originDir: 若是文件夹，则选择最后一个模型，若是文件名，则选择该模型文件。 12 | :param convertDir: 生成的pb模型名叫model.pb，在该目录下 13 | :param model_type: bert后接的模型类型，rcnnatt，如果为None直接接全连接层 14 | :param bert_dir: bert预训练模型文件夹，下边只需要包含配置文件和词典 15 | """ 16 | convert(task="divorce", tagDir="data/divorce/tags.txt", originDir="ckpt/divorce", 17 | convertDir="pb/divorce", model_type="rcnnatt", bert_dir="/home/huanghui/data/chinese_L-12_H-768_A-12") 18 | -------------------------------------------------------------------------------- /createPretrainData.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Create masked LM/next sentence masked_lm TF examples for BERT.""" 16 | 17 | from __future__ import absolute_import 18 | from __future__ import division 19 | from __future__ import print_function 20 | 21 | import collections 22 | import random 23 | import os 24 | from bert import tokenization 25 | import tensorflow as tf 26 | 27 | flags = tf.flags 28 | 29 | FLAGS = flags.FLAGS 30 | 31 | domain2len = {"divorce": 128, "labor": 150, "loan": 200} 32 | domain2per_seq = {"divorce": 20, "labor": 25, "loan": 30} 33 | 34 | flags.DEFINE_string( 35 | "domain", "divorce", 36 | "The name of the task to train selected in the list: divorce, labor, loan") 37 | 38 | flags.DEFINE_string( 39 | "bert_path", "/home/huanghui/data/chinese_L-12_H-768_A-12", 40 | "The dir corresponding to the pre-trained BERT model. ") 41 | 42 | ###输入的预训练数据 43 | flags.DEFINE_string("input_file", "data/" + FLAGS.domain + "/pretrain.txt", 44 | "Input raw text file (or comma-separated list of files).") 45 | 46 | ##输出的tfrecord文件 47 | flags.DEFINE_string( 48 | "output_file", "pretrain_data/" + FLAGS.domain + "/tf_examples.tfrecord", 49 | "Output TF example file (or comma-separated list of files).") 50 | 51 | flags.DEFINE_string("vocab_file", os.path.join(FLAGS.bert_path, 'vocab.txt'), 52 | "The vocabulary file that the BERT model was trained on.") 53 | 54 | flags.DEFINE_bool( 55 | "do_lower_case", True, 56 | "Whether to lower case the input text. Should be True for uncased " 57 | "models and False for cased models.") 58 | 59 | flags.DEFINE_integer("max_seq_length", domain2len[FLAGS.domain], "Maximum sequence length.") 60 | 61 | flags.DEFINE_integer("max_predictions_per_seq", domain2per_seq[FLAGS.domain], 62 | "Maximum number of masked LM predictions per sequence.") 63 | 64 | flags.DEFINE_integer("random_seed", 12345, "Random seed for data generation.") 65 | 66 | flags.DEFINE_integer( 67 | "dupe_factor", 5, 68 | "Number of times to duplicate the input data (with different masks).") 69 | 70 | flags.DEFINE_float("masked_lm_prob", 0.15, "Masked LM probability.") 71 | 72 | flags.DEFINE_float( 73 | "short_seq_prob", 0.1, 74 | "Probability of creating sequences which are shorter than the " 75 | "maximum length.") 76 | 77 | 78 | class TrainingInstance(object): 79 | """A single training instance (sentence pair).""" 80 | 81 | def __init__(self, tokens, segment_ids, masked_lm_positions, masked_lm_labels, 82 | is_random_next): 83 | self.tokens = tokens 84 | self.segment_ids = segment_ids 85 | self.is_random_next = is_random_next 86 | self.masked_lm_positions = masked_lm_positions 87 | self.masked_lm_labels = masked_lm_labels 88 | 89 | def __str__(self): 90 | s = "" 91 | s += "tokens: %s\n" % (" ".join( 92 | [tokenization.printable_text(x) for x in self.tokens])) 93 | s += "segment_ids: %s\n" % (" ".join([str(x) for x in self.segment_ids])) 94 | s += "is_random_next: %s\n" % self.is_random_next 95 | s += "masked_lm_positions: %s\n" % (" ".join( 96 | [str(x) for x in self.masked_lm_positions])) 97 | s += "masked_lm_labels: %s\n" % (" ".join( 98 | [tokenization.printable_text(x) for x in self.masked_lm_labels])) 99 | s += "\n" 100 | return s 101 | 102 | def __repr__(self): 103 | return self.__str__() 104 | 105 | 106 | def write_instance_to_example_files(instances, tokenizer, max_seq_length, 107 | max_predictions_per_seq, output_files): 108 | """Create TF example files from `TrainingInstance`s.""" 109 | writers = [] 110 | for output_file in output_files: 111 | writers.append(tf.python_io.TFRecordWriter(output_file)) 112 | 113 | writer_index = 0 114 | 115 | total_written = 0 116 | for (inst_index, instance) in enumerate(instances): 117 | input_ids = tokenizer.convert_tokens_to_ids(instance.tokens) 118 | input_mask = [1] * len(input_ids) 119 | segment_ids = list(instance.segment_ids) 120 | assert len(input_ids) <= max_seq_length 121 | 122 | while len(input_ids) < max_seq_length: 123 | input_ids.append(0) 124 | input_mask.append(0) 125 | segment_ids.append(0) 126 | 127 | assert len(input_ids) == max_seq_length 128 | assert len(input_mask) == max_seq_length 129 | assert len(segment_ids) == max_seq_length 130 | 131 | masked_lm_positions = list(instance.masked_lm_positions) 132 | masked_lm_ids = tokenizer.convert_tokens_to_ids(instance.masked_lm_labels) 133 | masked_lm_weights = [1.0] * len(masked_lm_ids) 134 | 135 | while len(masked_lm_positions) < max_predictions_per_seq: 136 | masked_lm_positions.append(0) 137 | masked_lm_ids.append(0) 138 | masked_lm_weights.append(0.0) 139 | 140 | next_sentence_label = 1 if instance.is_random_next else 0 141 | 142 | features = collections.OrderedDict() 143 | features["input_ids"] = create_int_feature(input_ids) 144 | features["input_mask"] = create_int_feature(input_mask) 145 | features["segment_ids"] = create_int_feature(segment_ids) 146 | features["masked_lm_positions"] = create_int_feature(masked_lm_positions) 147 | features["masked_lm_ids"] = create_int_feature(masked_lm_ids) 148 | features["masked_lm_weights"] = create_float_feature(masked_lm_weights) 149 | features["next_sentence_labels"] = create_int_feature([next_sentence_label]) 150 | 151 | tf_example = tf.train.Example(features=tf.train.Features(feature=features)) 152 | 153 | writers[writer_index].write(tf_example.SerializeToString()) 154 | writer_index = (writer_index + 1) % len(writers) 155 | 156 | total_written += 1 157 | 158 | if inst_index < 20: 159 | tf.logging.info("*** Example ***") 160 | tf.logging.info("tokens: %s" % " ".join( 161 | [tokenization.printable_text(x) for x in instance.tokens])) 162 | 163 | for feature_name in features.keys(): 164 | feature = features[feature_name] 165 | values = [] 166 | if feature.int64_list.value: 167 | values = feature.int64_list.value 168 | elif feature.float_list.value: 169 | values = feature.float_list.value 170 | tf.logging.info( 171 | "%s: %s" % (feature_name, " ".join([str(x) for x in values]))) 172 | 173 | for writer in writers: 174 | writer.close() 175 | 176 | tf.logging.info("Wrote %d total instances", total_written) 177 | 178 | 179 | def create_int_feature(values): 180 | feature = tf.train.Feature(int64_list=tf.train.Int64List(value=list(values))) 181 | return feature 182 | 183 | 184 | def create_float_feature(values): 185 | feature = tf.train.Feature(float_list=tf.train.FloatList(value=list(values))) 186 | return feature 187 | 188 | 189 | def create_training_instances(input_files, tokenizer, max_seq_length, 190 | dupe_factor, short_seq_prob, masked_lm_prob, 191 | max_predictions_per_seq, rng): 192 | """Create `TrainingInstance`s from raw text.""" 193 | all_documents = [[]] 194 | 195 | # Input file format: 196 | # (1) One sentence per line. These should ideally be actual sentences, not 197 | # entire paragraphs or arbitrary spans of text. (Because we use the 198 | # sentence boundaries for the "next sentence prediction" task). 199 | # (2) Blank lines between documents. Document boundaries are needed so 200 | # that the "next sentence prediction" task doesn't span between documents. 201 | for input_file in input_files: 202 | with tf.gfile.GFile(input_file, "r") as reader: 203 | while True: 204 | line = tokenization.convert_to_unicode(reader.readline()) 205 | if not line: 206 | break 207 | line = line.strip() 208 | 209 | # Empty lines are used as document delimiters 210 | if not line: 211 | all_documents.append([]) 212 | tokens = tokenizer.tokenize(line) 213 | if tokens: 214 | all_documents[-1].append(tokens) 215 | 216 | # Remove empty documents 217 | all_documents = [x for x in all_documents if x] 218 | rng.shuffle(all_documents) 219 | 220 | vocab_words = list(tokenizer.vocab.keys()) 221 | instances = [] 222 | for _ in range(dupe_factor): 223 | for document_index in range(len(all_documents)): 224 | instances.extend( 225 | create_instances_from_document( 226 | all_documents, document_index, max_seq_length, short_seq_prob, 227 | masked_lm_prob, max_predictions_per_seq, vocab_words, rng)) 228 | 229 | rng.shuffle(instances) 230 | return instances 231 | 232 | 233 | def create_instances_from_document( 234 | all_documents, document_index, max_seq_length, short_seq_prob, 235 | masked_lm_prob, max_predictions_per_seq, vocab_words, rng): 236 | """Creates `TrainingInstance`s for a single document.""" 237 | document = all_documents[document_index] 238 | 239 | # Account for [CLS], [SEP], [SEP] 240 | max_num_tokens = max_seq_length - 3 241 | 242 | # We *usually* want to fill up the entire sequence since we are padding 243 | # to `max_seq_length` anyways, so short sequences are generally wasted 244 | # computation. However, we *sometimes* 245 | # (i.e., short_seq_prob == 0.1 == 10% of the time) want to use shorter 246 | # sequences to minimize the mismatch between pre-training and fine-tuning. 247 | # The `target_seq_length` is just a rough target however, whereas 248 | # `max_seq_length` is a hard limit. 249 | target_seq_length = max_num_tokens 250 | if rng.random() < short_seq_prob: 251 | target_seq_length = rng.randint(2, max_num_tokens) 252 | 253 | # We DON'T just concatenate all of the tokens from a document into a long 254 | # sequence and choose an arbitrary split point because this would make the 255 | # next sentence prediction task too easy. Instead, we split the input into 256 | # segments "A" and "B" based on the actual "sentences" provided by the user 257 | # input. 258 | instances = [] 259 | current_chunk = [] 260 | current_length = 0 261 | i = 0 262 | while i < len(document): 263 | segment = document[i] 264 | current_chunk.append(segment) 265 | current_length += len(segment) 266 | if i == len(document) - 1 or current_length >= target_seq_length: 267 | if current_chunk: 268 | # `a_end` is how many segments from `current_chunk` go into the `A` 269 | # (first) sentence. 270 | a_end = 1 271 | if len(current_chunk) >= 2: 272 | a_end = rng.randint(1, len(current_chunk) - 1) 273 | 274 | tokens_a = [] 275 | for j in range(a_end): 276 | tokens_a.extend(current_chunk[j]) 277 | 278 | tokens_b = [] 279 | # Random next 280 | is_random_next = False 281 | if len(current_chunk) == 1 or rng.random() < 0.5: 282 | is_random_next = True 283 | target_b_length = target_seq_length - len(tokens_a) 284 | 285 | # This should rarely go for more than one iteration for large 286 | # corpora. However, just to be careful, we try to make sure that 287 | # the random document is not the same as the document 288 | # we're processing. 289 | for _ in range(10): 290 | random_document_index = rng.randint(0, len(all_documents) - 1) 291 | if random_document_index != document_index: 292 | break 293 | 294 | random_document = all_documents[random_document_index] 295 | random_start = rng.randint(0, len(random_document) - 1) 296 | for j in range(random_start, len(random_document)): 297 | tokens_b.extend(random_document[j]) 298 | if len(tokens_b) >= target_b_length: 299 | break 300 | # We didn't actually use these segments so we "put them back" so 301 | # they don't go to waste. 302 | num_unused_segments = len(current_chunk) - a_end 303 | i -= num_unused_segments 304 | # Actual next 305 | else: 306 | is_random_next = False 307 | for j in range(a_end, len(current_chunk)): 308 | tokens_b.extend(current_chunk[j]) 309 | truncate_seq_pair(tokens_a, tokens_b, max_num_tokens, rng) 310 | 311 | assert len(tokens_a) >= 1 312 | assert len(tokens_b) >= 1 313 | 314 | tokens = [] 315 | segment_ids = [] 316 | tokens.append("[CLS]") 317 | segment_ids.append(0) 318 | for token in tokens_a: 319 | tokens.append(token) 320 | segment_ids.append(0) 321 | 322 | tokens.append("[SEP]") 323 | segment_ids.append(0) 324 | 325 | for token in tokens_b: 326 | tokens.append(token) 327 | segment_ids.append(1) 328 | tokens.append("[SEP]") 329 | segment_ids.append(1) 330 | 331 | (tokens, masked_lm_positions, 332 | masked_lm_labels) = create_masked_lm_predictions( 333 | tokens, masked_lm_prob, max_predictions_per_seq, vocab_words, rng) 334 | instance = TrainingInstance( 335 | tokens=tokens, 336 | segment_ids=segment_ids, 337 | is_random_next=is_random_next, 338 | masked_lm_positions=masked_lm_positions, 339 | masked_lm_labels=masked_lm_labels) 340 | instances.append(instance) 341 | current_chunk = [] 342 | current_length = 0 343 | i += 1 344 | 345 | return instances 346 | 347 | 348 | MaskedLmInstance = collections.namedtuple("MaskedLmInstance", 349 | ["index", "label"]) 350 | 351 | 352 | def create_masked_lm_predictions(tokens, masked_lm_prob, 353 | max_predictions_per_seq, vocab_words, rng): 354 | """Creates the predictions for the masked LM objective.""" 355 | 356 | cand_indexes = [] 357 | for (i, token) in enumerate(tokens): 358 | if token == "[CLS]" or token == "[SEP]": 359 | continue 360 | cand_indexes.append(i) 361 | 362 | rng.shuffle(cand_indexes) 363 | 364 | output_tokens = list(tokens) 365 | 366 | num_to_predict = min(max_predictions_per_seq, 367 | max(1, int(round(len(tokens) * masked_lm_prob)))) 368 | 369 | masked_lms = [] 370 | covered_indexes = set() 371 | for index in cand_indexes: 372 | if len(masked_lms) >= num_to_predict: 373 | break 374 | if index in covered_indexes: 375 | continue 376 | covered_indexes.add(index) 377 | 378 | masked_token = None 379 | # 80% of the time, replace with [MASK] 380 | if rng.random() < 0.8: 381 | masked_token = "[MASK]" 382 | else: 383 | # 10% of the time, keep original 384 | if rng.random() < 0.5: 385 | masked_token = tokens[index] 386 | # 10% of the time, replace with random word 387 | else: 388 | masked_token = vocab_words[rng.randint(0, len(vocab_words) - 1)] 389 | 390 | output_tokens[index] = masked_token 391 | 392 | masked_lms.append(MaskedLmInstance(index=index, label=tokens[index])) 393 | 394 | masked_lms = sorted(masked_lms, key=lambda x: x.index) 395 | 396 | masked_lm_positions = [] 397 | masked_lm_labels = [] 398 | for p in masked_lms: 399 | masked_lm_positions.append(p.index) 400 | masked_lm_labels.append(p.label) 401 | 402 | return (output_tokens, masked_lm_positions, masked_lm_labels) 403 | 404 | 405 | def truncate_seq_pair(tokens_a, tokens_b, max_num_tokens, rng): 406 | """Truncates a pair of sequences to a maximum sequence length.""" 407 | while True: 408 | total_length = len(tokens_a) + len(tokens_b) 409 | if total_length <= max_num_tokens: 410 | break 411 | 412 | trunc_tokens = tokens_a if len(tokens_a) > len(tokens_b) else tokens_b 413 | assert len(trunc_tokens) >= 1 414 | 415 | # We want to sometimes truncate from the front and sometimes from the 416 | # back to add more randomness and avoid biases. 417 | if rng.random() < 0.5: 418 | del trunc_tokens[0] 419 | else: 420 | trunc_tokens.pop() 421 | 422 | 423 | def main(_): 424 | tf.logging.set_verbosity(tf.logging.INFO) 425 | 426 | tokenizer = tokenization.FullTokenizer( 427 | vocab_file=FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case) 428 | 429 | input_files = [] 430 | for input_pattern in FLAGS.input_file.split(","): 431 | input_files.extend(tf.gfile.Glob(input_pattern)) 432 | 433 | tf.logging.info("*** Reading from input files ***") 434 | for input_file in input_files: 435 | tf.logging.info(" %s", input_file) 436 | 437 | rng = random.Random(FLAGS.random_seed) 438 | instances = create_training_instances( 439 | input_files, tokenizer, FLAGS.max_seq_length, FLAGS.dupe_factor, 440 | FLAGS.short_seq_prob, FLAGS.masked_lm_prob, FLAGS.max_predictions_per_seq, 441 | rng) 442 | 443 | output_files = FLAGS.output_file.split(",") 444 | tf.logging.info("*** Writing to output files ***") 445 | for output_file in output_files: 446 | output_dir = os.path.dirname(output_file) 447 | if not os.path.exists(output_dir): 448 | os.makedirs(output_dir) 449 | tf.logging.info(" %s", output_file) 450 | 451 | write_instance_to_example_files(instances, tokenizer, FLAGS.max_seq_length, 452 | FLAGS.max_predictions_per_seq, output_files) 453 | 454 | 455 | if __name__ == "__main__": 456 | 457 | tf.app.run() 458 | -------------------------------------------------------------------------------- /data/divorce/tags.txt: -------------------------------------------------------------------------------- 1 | DV1 2 | DV2 3 | DV3 4 | DV4 5 | DV5 6 | DV6 7 | DV7 8 | DV8 9 | DV9 10 | DV10 11 | DV11 12 | DV12 13 | DV13 14 | DV14 15 | DV15 16 | DV16 17 | DV17 18 | DV18 19 | DV19 20 | DV20 21 | -------------------------------------------------------------------------------- /data/labor/tags.txt: -------------------------------------------------------------------------------- 1 | LB1 2 | LB2 3 | LB3 4 | LB4 5 | LB5 6 | LB6 7 | LB7 8 | LB8 9 | LB9 10 | LB10 11 | LB11 12 | LB12 13 | LB13 14 | LB14 15 | LB15 16 | LB16 17 | LB17 18 | LB18 19 | LB19 20 | LB20 21 | -------------------------------------------------------------------------------- /data/loan/tags.txt: -------------------------------------------------------------------------------- 1 | LN1 2 | LN2 3 | LN3 4 | LN4 5 | LN5 6 | LN6 7 | LN7 8 | LN8 9 | LN9 10 | LN10 11 | LN11 12 | LN12 13 | LN13 14 | LN14 15 | LN15 16 | LN16 17 | LN17 18 | LN18 19 | LN19 20 | LN20 21 | -------------------------------------------------------------------------------- /evaluation.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | #-*- coding:utf-8 -*- 3 | # author:huanghui 4 | # datetime:2019/9/30 10:28 5 | import json 6 | from utils.evaluate import evaluate 7 | from utils.predict import BERTModel 8 | import logging 9 | import numpy as np 10 | import re 11 | 12 | logging.basicConfig(level=logging.INFO) 13 | 14 | def readThreshold(fname): 15 | """读取json阈值文件""" 16 | with open(fname, "r", encoding="utf-8") as f: 17 | dic = json.loads(f.readline()) 18 | return dic 19 | 20 | def add_(arr1, arr2): 21 | for i in range(len(arr1)): 22 | arr1[i].extend(arr2[i]) 23 | arr1[i] = list(set(arr1[i])) 24 | return arr1 25 | 26 | def re_match(text, feature): 27 | pred = np.zeros(20, dtype=np.int32) 28 | if feature is None: 29 | return pred 30 | for i in feature.keys(): 31 | pred[i-1] = any([re.match(key, text) is not None for key in feature[i]]) 32 | return pred 33 | 34 | def getMatch(feature, sentences, model): 35 | """添加规则测试，采用正则表达式匹配，最后与模型结果取并集""" 36 | re_pred = np.zeros((len(sentences), 20), dtype=np.int32) 37 | for idx, sent in enumerate(sentences): 38 | re_pred[idx] = re_match(str(sent), feature) 39 | re_pre = model.rematch(re_pred) 40 | return re_pre 41 | 42 | def load_file(filename): 43 | f = open(filename, "r", encoding='utf-8') 44 | all_sentence = [] 45 | all_label = [] 46 | for line in f: 47 | pre_doc = json.loads(line) 48 | for sent in pre_doc: 49 | all_sentence.append(sent["sentence"]) 50 | all_label.append(sent["labels"]) 51 | f.close() 52 | return all_sentence, all_label 53 | 54 | if __name__ == '__main__': 55 | 56 | task = "divorce" 57 | 58 | ##这里传入切分好的测试数据，这里由于是整理代码做测试，随便导入训练数据集测试下 59 | sentences, labels = load_file("data/divorce/train_selected.json") 60 | 61 | logging.info("开始载入bert模型") 62 | model_1 = BERTModel(task=task, pb_model="pb/model.pb", 63 | tagDir="data/divorce/tags.txt", threshold=[0.5] * 20, 64 | vocab_file="/home/huanghui/data/chinese_L-12_H-768_A-12/vocab.txt") 65 | 66 | logging.info("bert模型载入完毕，开始进行预测！！！\n") 67 | logging.info("模型开始预测\n") 68 | predicts_1 = model_1.getAllResult(sentences) 69 | logging.info("模型预测结束\n") 70 | 71 | logging.info("模型每个类别f值计算如下：\n") 72 | score_1, f1_1 = evaluate(predict_labels=predicts_1, target_labels=labels, tag_dir="data/divorce/tags.txt") 73 | logging.info(f1_1) 74 | logging.info("总评分如下： {}".format(score_1)) 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | -------------------------------------------------------------------------------- /genPretrainData.py: -------------------------------------------------------------------------------- 1 | import json 2 | 3 | def getSentences(filename): 4 | sentences = [] 5 | with open(filename, "r", encoding="utf-8") as f: 6 | for line in f: 7 | temp = [] 8 | documents = json.loads(line) 9 | for content in documents: 10 | sentence = content["sentence"] 11 | temp.append(sentence) 12 | sentences.append(temp) 13 | return sentences 14 | 15 | def work(task): 16 | small = "data/" + task + "/data_small_selected.json" 17 | large = "data/" + task + "/train_selected.json" 18 | oufname = "data/" + task + "/pretrain.txt" 19 | all_sentences = [] 20 | small_sentences = getSentences(small) 21 | large_sentences = getSentences(large) 22 | all_sentences.extend(small_sentences) 23 | all_sentences.extend(large_sentences) 24 | ouf = open(oufname, "w", encoding="utf-8") 25 | for document in all_sentences: 26 | for sentence in document: 27 | ouf.write(str(sentence) + "\n") 28 | ouf.write("\n") 29 | ouf.close() 30 | 31 | if __name__ == '__main__': 32 | tasks = ["divorce", "labor", "loan"] 33 | for task in tasks: 34 | work(task) 35 | -------------------------------------------------------------------------------- /run_pretrain.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Language Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Run masked LM/next sentence masked_lm pre-training for BERT.""" 16 | 17 | from __future__ import absolute_import 18 | from __future__ import division 19 | from __future__ import print_function 20 | 21 | import os 22 | from bert import modeling 23 | from bert import optimization 24 | import tensorflow as tf 25 | 26 | flags = tf.flags 27 | 28 | FLAGS = flags.FLAGS 29 | 30 | domain2len = {"divorce": 128, "labor": 150, "loan": 200} 31 | 32 | domain2per_seq = {"divorce": 20, "labor": 25, "loan": 30} 33 | 34 | ## Required parameters 35 | flags.DEFINE_string( 36 | "domain", "divorce", 37 | "The name of the task to train selected in the list: divorce, labor, loan") 38 | flags.DEFINE_string( 39 | "bert_path", "/home/huanghui/data/chinese_L-12_H-768_A-12", 40 | "The dir corresponding to the pre-trained BERT model. ") 41 | flags.DEFINE_string( 42 | "bert_config_file", os.path.join(FLAGS.bert_path, "bert_config.json"), 43 | "The config json file corresponding to the pre-trained BERT model. " 44 | "This specifies the model architecture.") 45 | 46 | flags.DEFINE_string( 47 | "input_file", "pretrain_data/" + FLAGS.domain + "/tf_examples.tfrecord", 48 | "Input TF example files (can be a glob or comma separated).") 49 | 50 | flags.DEFINE_string( 51 | "output_dir", "ckpt_pretrain/"+FLAGS.domain, 52 | "The output directory where the model checkpoints will be written.") 53 | 54 | ## Other parameters 55 | flags.DEFINE_string( 56 | "init_checkpoint", os.path.join(FLAGS.bert_path, "bert_model.ckpt"), 57 | "Initial checkpoint (usually from a pre-trained BERT model).") 58 | 59 | flags.DEFINE_integer( 60 | "max_seq_length", domain2len[FLAGS.domain], 61 | "The maximum total input sequence length after WordPiece tokenization. " 62 | "Sequences longer than this will be truncated, and sequences shorter " 63 | "than this will be padded. Must match data generation.") 64 | 65 | flags.DEFINE_integer( 66 | "max_predictions_per_seq", domain2per_seq[FLAGS.domain], 67 | "Maximum number of masked LM predictions per sequence. " 68 | "Must match data generation.") 69 | 70 | flags.DEFINE_bool("do_train", True, "Whether to run training.") 71 | 72 | flags.DEFINE_bool("do_eval", True, "Whether to run eval on the dev set.") 73 | 74 | flags.DEFINE_integer("train_batch_size", 32, "Total batch size for training.") 75 | 76 | flags.DEFINE_integer("eval_batch_size", 8, "Total batch size for eval.") 77 | 78 | flags.DEFINE_float("learning_rate", 3e-5, "The initial learning rate for Adam.") 79 | 80 | flags.DEFINE_integer("num_train_steps", 50000, "Number of training steps.") 81 | 82 | flags.DEFINE_integer("num_warmup_steps", 5000, "Number of warmup steps.") 83 | 84 | flags.DEFINE_integer("save_checkpoints_steps", 5000, 85 | "How often to save the model checkpoint.") 86 | 87 | flags.DEFINE_integer("iterations_per_loop", 1000, 88 | "How many steps to make in each estimator call.") 89 | 90 | flags.DEFINE_integer("max_eval_steps", 100, "Maximum number of eval steps.") 91 | 92 | flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.") 93 | 94 | tf.flags.DEFINE_string( 95 | "tpu_name", None, 96 | "The Cloud TPU to use for training. This should be either the name " 97 | "used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 " 98 | "url.") 99 | 100 | tf.flags.DEFINE_string( 101 | "tpu_zone", None, 102 | "[Optional] GCE zone where the Cloud TPU is located in. If not " 103 | "specified, we will attempt to automatically detect the GCE project from " 104 | "metadata.") 105 | 106 | tf.flags.DEFINE_string( 107 | "gcp_project", None, 108 | "[Optional] Project name for the Cloud TPU-enabled project. If not " 109 | "specified, we will attempt to automatically detect the GCE project from " 110 | "metadata.") 111 | 112 | tf.flags.DEFINE_string("master", None, "[Optional] TensorFlow master URL.") 113 | 114 | flags.DEFINE_integer( 115 | "num_tpu_cores", 8, 116 | "Only used if `use_tpu` is True. Total number of TPU cores to use.") 117 | 118 | 119 | def model_fn_builder(bert_config, init_checkpoint, learning_rate, 120 | num_train_steps, num_warmup_steps, use_tpu, 121 | use_one_hot_embeddings): 122 | """Returns `model_fn` closure for TPUEstimator.""" 123 | 124 | def model_fn(features, labels, mode, params): # pylint: disable=unused-argument 125 | """The `model_fn` for TPUEstimator.""" 126 | 127 | tf.logging.info("*** Features ***") 128 | for name in sorted(features.keys()): 129 | tf.logging.info(" name = %s, shape = %s" % (name, features[name].shape)) 130 | 131 | input_ids = features["input_ids"] 132 | input_mask = features["input_mask"] 133 | segment_ids = features["segment_ids"] 134 | masked_lm_positions = features["masked_lm_positions"] 135 | masked_lm_ids = features["masked_lm_ids"] 136 | masked_lm_weights = features["masked_lm_weights"] 137 | next_sentence_labels = features["next_sentence_labels"] 138 | 139 | is_training = (mode == tf.estimator.ModeKeys.TRAIN) 140 | 141 | model = modeling.BertModel( 142 | config=bert_config, 143 | is_training=is_training, 144 | input_ids=input_ids, 145 | input_mask=input_mask, 146 | token_type_ids=segment_ids, 147 | use_one_hot_embeddings=use_one_hot_embeddings) 148 | 149 | (masked_lm_loss, 150 | masked_lm_example_loss, masked_lm_log_probs) = get_masked_lm_output( 151 | bert_config, model.get_sequence_output(), model.get_embedding_table(), 152 | masked_lm_positions, masked_lm_ids, masked_lm_weights) 153 | 154 | (next_sentence_loss, next_sentence_example_loss, 155 | next_sentence_log_probs) = get_next_sentence_output( 156 | bert_config, model.get_pooled_output(), next_sentence_labels) 157 | 158 | total_loss = masked_lm_loss + next_sentence_loss 159 | 160 | tvars = tf.trainable_variables() 161 | 162 | initialized_variable_names = {} 163 | scaffold_fn = None 164 | if init_checkpoint: 165 | (assignment_map, initialized_variable_names 166 | ) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint) 167 | if use_tpu: 168 | 169 | def tpu_scaffold(): 170 | tf.train.init_from_checkpoint(init_checkpoint, assignment_map) 171 | return tf.train.Scaffold() 172 | 173 | scaffold_fn = tpu_scaffold 174 | else: 175 | tf.train.init_from_checkpoint(init_checkpoint, assignment_map) 176 | 177 | tf.logging.info("**** Trainable Variables ****") 178 | for var in tvars: 179 | init_string = "" 180 | if var.name in initialized_variable_names: 181 | init_string = ", *INIT_FROM_CKPT*" 182 | tf.logging.info(" name = %s, shape = %s%s", var.name, var.shape, 183 | init_string) 184 | 185 | output_spec = None 186 | if mode == tf.estimator.ModeKeys.TRAIN: 187 | train_op = optimization.create_optimizer( 188 | total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu) 189 | 190 | output_spec = tf.contrib.tpu.TPUEstimatorSpec( 191 | mode=mode, 192 | loss=total_loss, 193 | train_op=train_op, 194 | scaffold_fn=scaffold_fn) 195 | elif mode == tf.estimator.ModeKeys.EVAL: 196 | 197 | def metric_fn(masked_lm_example_loss, masked_lm_log_probs, masked_lm_ids, 198 | masked_lm_weights, next_sentence_example_loss, 199 | next_sentence_log_probs, next_sentence_labels): 200 | """Computes the loss and accuracy of the model.""" 201 | masked_lm_log_probs = tf.reshape(masked_lm_log_probs, 202 | [-1, masked_lm_log_probs.shape[-1]]) 203 | masked_lm_predictions = tf.argmax( 204 | masked_lm_log_probs, axis=-1, output_type=tf.int32) 205 | masked_lm_example_loss = tf.reshape(masked_lm_example_loss, [-1]) 206 | masked_lm_ids = tf.reshape(masked_lm_ids, [-1]) 207 | masked_lm_weights = tf.reshape(masked_lm_weights, [-1]) 208 | masked_lm_accuracy = tf.metrics.accuracy( 209 | labels=masked_lm_ids, 210 | predictions=masked_lm_predictions, 211 | weights=masked_lm_weights) 212 | masked_lm_mean_loss = tf.metrics.mean( 213 | values=masked_lm_example_loss, weights=masked_lm_weights) 214 | 215 | next_sentence_log_probs = tf.reshape( 216 | next_sentence_log_probs, [-1, next_sentence_log_probs.shape[-1]]) 217 | next_sentence_predictions = tf.argmax( 218 | next_sentence_log_probs, axis=-1, output_type=tf.int32) 219 | next_sentence_labels = tf.reshape(next_sentence_labels, [-1]) 220 | next_sentence_accuracy = tf.metrics.accuracy( 221 | labels=next_sentence_labels, predictions=next_sentence_predictions) 222 | next_sentence_mean_loss = tf.metrics.mean( 223 | values=next_sentence_example_loss) 224 | 225 | return { 226 | "masked_lm_accuracy": masked_lm_accuracy, 227 | "masked_lm_loss": masked_lm_mean_loss, 228 | "next_sentence_accuracy": next_sentence_accuracy, 229 | "next_sentence_loss": next_sentence_mean_loss, 230 | } 231 | 232 | eval_metrics = (metric_fn, [ 233 | masked_lm_example_loss, masked_lm_log_probs, masked_lm_ids, 234 | masked_lm_weights, next_sentence_example_loss, 235 | next_sentence_log_probs, next_sentence_labels 236 | ]) 237 | output_spec = tf.contrib.tpu.TPUEstimatorSpec( 238 | mode=mode, 239 | loss=total_loss, 240 | eval_metrics=eval_metrics, 241 | scaffold_fn=scaffold_fn) 242 | else: 243 | raise ValueError("Only TRAIN and EVAL modes are supported: %s" % (mode)) 244 | 245 | return output_spec 246 | 247 | return model_fn 248 | 249 | 250 | def get_masked_lm_output(bert_config, input_tensor, output_weights, positions, 251 | label_ids, label_weights): 252 | """Get loss and log probs for the masked LM.""" 253 | input_tensor = gather_indexes(input_tensor, positions) 254 | 255 | with tf.variable_scope("cls/predictions"): 256 | # We apply one more non-linear transformation before the output layer. 257 | # This matrix is not used after pre-training. 258 | with tf.variable_scope("transform"): 259 | input_tensor = tf.layers.dense( 260 | input_tensor, 261 | units=bert_config.hidden_size, 262 | activation=modeling.get_activation(bert_config.hidden_act), 263 | kernel_initializer=modeling.create_initializer( 264 | bert_config.initializer_range)) 265 | input_tensor = modeling.layer_norm(input_tensor) 266 | 267 | # The output weights are the same as the input embeddings, but there is 268 | # an output-only bias for each token. 269 | output_bias = tf.get_variable( 270 | "output_bias", 271 | shape=[bert_config.vocab_size], 272 | initializer=tf.zeros_initializer()) 273 | logits = tf.matmul(input_tensor, output_weights, transpose_b=True) 274 | logits = tf.nn.bias_add(logits, output_bias) 275 | log_probs = tf.nn.log_softmax(logits, axis=-1) 276 | 277 | label_ids = tf.reshape(label_ids, [-1]) 278 | label_weights = tf.reshape(label_weights, [-1]) 279 | 280 | one_hot_labels = tf.one_hot( 281 | label_ids, depth=bert_config.vocab_size, dtype=tf.float32) 282 | 283 | # The `positions` tensor might be zero-padded (if the sequence is too 284 | # short to have the maximum number of predictions). The `label_weights` 285 | # tensor has a value of 1.0 for every real prediction and 0.0 for the 286 | # padding predictions. 287 | per_example_loss = -tf.reduce_sum(log_probs * one_hot_labels, axis=[-1]) 288 | numerator = tf.reduce_sum(label_weights * per_example_loss) 289 | denominator = tf.reduce_sum(label_weights) + 1e-5 290 | loss = numerator / denominator 291 | 292 | return (loss, per_example_loss, log_probs) 293 | 294 | 295 | def get_next_sentence_output(bert_config, input_tensor, labels): 296 | """Get loss and log probs for the next sentence prediction.""" 297 | 298 | # Simple binary classification. Note that 0 is "next sentence" and 1 is 299 | # "random sentence". This weight matrix is not used after pre-training. 300 | with tf.variable_scope("cls/seq_relationship"): 301 | output_weights = tf.get_variable( 302 | "output_weights", 303 | shape=[2, bert_config.hidden_size], 304 | initializer=modeling.create_initializer(bert_config.initializer_range)) 305 | output_bias = tf.get_variable( 306 | "output_bias", shape=[2], initializer=tf.zeros_initializer()) 307 | 308 | logits = tf.matmul(input_tensor, output_weights, transpose_b=True) 309 | logits = tf.nn.bias_add(logits, output_bias) 310 | log_probs = tf.nn.log_softmax(logits, axis=-1) 311 | labels = tf.reshape(labels, [-1]) 312 | one_hot_labels = tf.one_hot(labels, depth=2, dtype=tf.float32) 313 | per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1) 314 | loss = tf.reduce_mean(per_example_loss) 315 | return (loss, per_example_loss, log_probs) 316 | 317 | 318 | def gather_indexes(sequence_tensor, positions): 319 | """Gathers the vectors at the specific positions over a minibatch.""" 320 | sequence_shape = modeling.get_shape_list(sequence_tensor, expected_rank=3) 321 | batch_size = sequence_shape[0] 322 | seq_length = sequence_shape[1] 323 | width = sequence_shape[2] 324 | 325 | flat_offsets = tf.reshape( 326 | tf.range(0, batch_size, dtype=tf.int32) * seq_length, [-1, 1]) 327 | flat_positions = tf.reshape(positions + flat_offsets, [-1]) 328 | flat_sequence_tensor = tf.reshape(sequence_tensor, 329 | [batch_size * seq_length, width]) 330 | output_tensor = tf.gather(flat_sequence_tensor, flat_positions) 331 | return output_tensor 332 | 333 | 334 | def input_fn_builder(input_files, 335 | max_seq_length, 336 | max_predictions_per_seq, 337 | is_training, 338 | num_cpu_threads=4): 339 | """Creates an `input_fn` closure to be passed to TPUEstimator.""" 340 | 341 | def input_fn(params): 342 | """The actual input function.""" 343 | batch_size = params["batch_size"] 344 | 345 | name_to_features = { 346 | "input_ids": 347 | tf.FixedLenFeature([max_seq_length], tf.int64), 348 | "input_mask": 349 | tf.FixedLenFeature([max_seq_length], tf.int64), 350 | "segment_ids": 351 | tf.FixedLenFeature([max_seq_length], tf.int64), 352 | "masked_lm_positions": 353 | tf.FixedLenFeature([max_predictions_per_seq], tf.int64), 354 | "masked_lm_ids": 355 | tf.FixedLenFeature([max_predictions_per_seq], tf.int64), 356 | "masked_lm_weights": 357 | tf.FixedLenFeature([max_predictions_per_seq], tf.float32), 358 | "next_sentence_labels": 359 | tf.FixedLenFeature([1], tf.int64), 360 | } 361 | 362 | # For training, we want a lot of parallel reading and shuffling. 363 | # For eval, we want no shuffling and parallel reading doesn't matter. 364 | if is_training: 365 | d = tf.data.Dataset.from_tensor_slices(tf.constant(input_files)) 366 | d = d.repeat() 367 | d = d.shuffle(buffer_size=len(input_files)) 368 | 369 | # `cycle_length` is the number of parallel files that get read. 370 | cycle_length = min(num_cpu_threads, len(input_files)) 371 | 372 | # `sloppy` mode means that the interleaving is not exact. This adds 373 | # even more randomness to the training pipeline. 374 | d = d.apply( 375 | tf.contrib.data.parallel_interleave( 376 | tf.data.TFRecordDataset, 377 | sloppy=is_training, 378 | cycle_length=cycle_length)) 379 | d = d.shuffle(buffer_size=100) 380 | else: 381 | d = tf.data.TFRecordDataset(input_files) 382 | # Since we evaluate for a fixed number of steps we don't want to encounter 383 | # out-of-range exceptions. 384 | d = d.repeat() 385 | 386 | # We must `drop_remainder` on training because the TPU requires fixed 387 | # size dimensions. For eval, we assume we are evaluating on the CPU or GPU 388 | # and we *don't* want to drop the remainder, otherwise we wont cover 389 | # every sample. 390 | d = d.apply( 391 | tf.contrib.data.map_and_batch( 392 | lambda record: _decode_record(record, name_to_features), 393 | batch_size=batch_size, 394 | num_parallel_batches=num_cpu_threads, 395 | drop_remainder=True)) 396 | return d 397 | 398 | return input_fn 399 | 400 | 401 | def _decode_record(record, name_to_features): 402 | """Decodes a record to a TensorFlow example.""" 403 | example = tf.parse_single_example(record, name_to_features) 404 | 405 | # tf.Example only supports tf.int64, but the TPU only supports tf.int32. 406 | # So cast all int64 to int32. 407 | for name in list(example.keys()): 408 | t = example[name] 409 | if t.dtype == tf.int64: 410 | t = tf.to_int32(t) 411 | example[name] = t 412 | 413 | return example 414 | 415 | 416 | def main(_): 417 | tf.logging.set_verbosity(tf.logging.INFO) 418 | 419 | if not FLAGS.do_train and not FLAGS.do_eval: 420 | raise ValueError("At least one of `do_train` or `do_eval` must be True.") 421 | 422 | bert_config = modeling.BertConfig.from_json_file(FLAGS.bert_config_file) 423 | 424 | tf.gfile.MakeDirs(FLAGS.output_dir) 425 | 426 | input_files = [] 427 | for input_pattern in FLAGS.input_file.split(","): 428 | input_files.extend(tf.gfile.Glob(input_pattern)) 429 | 430 | tf.logging.info("*** Input Files ***") 431 | for input_file in input_files: 432 | tf.logging.info(" %s" % input_file) 433 | 434 | tpu_cluster_resolver = None 435 | if FLAGS.use_tpu and FLAGS.tpu_name: 436 | tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver( 437 | FLAGS.tpu_name, zone=FLAGS.tpu_zone, project=FLAGS.gcp_project) 438 | 439 | is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2 440 | run_config = tf.contrib.tpu.RunConfig( 441 | cluster=tpu_cluster_resolver, 442 | master=FLAGS.master, 443 | model_dir=FLAGS.output_dir, 444 | keep_checkpoint_max=2, 445 | save_checkpoints_steps=FLAGS.save_checkpoints_steps, 446 | tpu_config=tf.contrib.tpu.TPUConfig( 447 | iterations_per_loop=FLAGS.iterations_per_loop, 448 | num_shards=FLAGS.num_tpu_cores, 449 | per_host_input_for_training=is_per_host)) 450 | 451 | model_fn = model_fn_builder( 452 | bert_config=bert_config, 453 | init_checkpoint=FLAGS.init_checkpoint, 454 | learning_rate=FLAGS.learning_rate, 455 | num_train_steps=FLAGS.num_train_steps, 456 | num_warmup_steps=FLAGS.num_warmup_steps, 457 | use_tpu=FLAGS.use_tpu, 458 | use_one_hot_embeddings=FLAGS.use_tpu) 459 | 460 | # If TPU is not available, this will fall back to normal Estimator on CPU 461 | # or GPU. 462 | estimator = tf.contrib.tpu.TPUEstimator( 463 | use_tpu=FLAGS.use_tpu, 464 | model_fn=model_fn, 465 | config=run_config, 466 | train_batch_size=FLAGS.train_batch_size, 467 | eval_batch_size=FLAGS.eval_batch_size) 468 | 469 | if FLAGS.do_train: 470 | tf.logging.info("***** Running training *****") 471 | tf.logging.info(" Batch size = %d", FLAGS.train_batch_size) 472 | train_input_fn = input_fn_builder( 473 | input_files=input_files, 474 | max_seq_length=FLAGS.max_seq_length, 475 | max_predictions_per_seq=FLAGS.max_predictions_per_seq, 476 | is_training=True) 477 | estimator.train(input_fn=train_input_fn, max_steps=FLAGS.num_train_steps) 478 | 479 | if FLAGS.do_eval: 480 | tf.logging.info("***** Running evaluation *****") 481 | tf.logging.info(" Batch size = %d", FLAGS.eval_batch_size) 482 | 483 | eval_input_fn = input_fn_builder( 484 | input_files=input_files, 485 | max_seq_length=FLAGS.max_seq_length, 486 | max_predictions_per_seq=FLAGS.max_predictions_per_seq, 487 | is_training=False) 488 | 489 | result = estimator.evaluate( 490 | input_fn=eval_input_fn, steps=FLAGS.max_eval_steps) 491 | 492 | output_eval_file = os.path.join(FLAGS.output_dir, "eval_results.txt") 493 | with tf.gfile.GFile(output_eval_file, "w") as writer: 494 | tf.logging.info("***** Eval results *****") 495 | for key in sorted(result.keys()): 496 | tf.logging.info(" %s = %s", key, str(result[key])) 497 | writer.write("%s = %s\n" % (key, str(result[key]))) 498 | 499 | 500 | if __name__ == "__main__": 501 | 502 | tf.app.run() 503 | -------------------------------------------------------------------------------- /search_threshold.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | #-*- coding:utf-8 -*- 3 | # author:huanghui 4 | # datetime:2019/9/30 10:55 5 | 6 | from utils.predict import BERTModel 7 | import json 8 | import os 9 | from utils.evaluate import evaluate 10 | import logging 11 | 12 | logging.basicConfig(level=logging.INFO) 13 | 14 | def getLab(probs, id2label, threshold): 15 | predict_list = [] 16 | for i in range(len(probs)): 17 | if probs[i] > threshold[i]: 18 | predict_list.append(id2label[i]) 19 | return predict_list 20 | 21 | def getPreLab(array, id2label, threshold): 22 | result = [] 23 | for p in array: 24 | result.append(getLab(p, id2label, threshold)) 25 | return result 26 | 27 | def load_file(filename): 28 | f = open(filename, "r", encoding='utf-8') 29 | all_sentence = [] 30 | all_label = [] 31 | for line in f: 32 | pre_doc = json.loads(line) 33 | for sent in pre_doc: 34 | all_sentence.append(sent["sentence"]) 35 | all_label.append(sent["labels"]) 36 | f.close() 37 | return all_sentence, all_label 38 | 39 | def searchThreshold(domain, model_pb, threshold_dir, 40 | test_file, tag_file, vocab_file): 41 | """ 42 | 用划分好的测试集取搜索最优的阈值，精度0.1,再低会过拟合，最好使用交叉验证来做 43 | 由于交叉验证bert代价很大，就没做 44 | :param domain: 数据集类别，divorce、labor、loan 45 | :param model_pb: pb模型文件 46 | :param threshold_dir: 阈值搜索结果json文件存放地址 47 | :param test_file: 用来搜索阈值的测试文件 48 | :param tag_file: 标签tags文件 49 | :param vocab_file: bert模型词典文件 50 | :return: 将搜索的阈值存入threshold_dir，命名为threshold.json 51 | 将搜索过程记录在search.json 52 | """ 53 | thresholds = [] 54 | for i in range(1, 10): 55 | thresholds.append(round(i * 0.1, 1)) 56 | 57 | all_sentences, all_labels = load_file(test_file) 58 | 59 | logging.info("———— 开始加载模型 ————\n") 60 | model = BERTModel(task=domain, pb_model=model_pb, tagDir=tag_file, threshold=None, vocab_file=vocab_file) 61 | logging.info("———— 模型加载结束 ————\n") 62 | logging.info("———— 开始生成预测概率metric ————\n") 63 | probas = model.getProbs(all_sentences) 64 | logging.info("———— 预测概率metric生成结束 ————\n") 65 | 66 | result = {} 67 | result["domain"] = domain 68 | result["label_score"] = [] 69 | logging.info("———— 开始搜索 %s 的最优阈值 ————\n" % domain) 70 | best_threshold = [0.5] * 20 71 | threshold_init = [0.5] * 20 72 | for i in range(20): 73 | best_score = 0 74 | label_result = {} 75 | scoreOfthreshold = {} 76 | label_result["label"] = i 77 | for j in range(len(best_threshold)): 78 | threshold_init[j] = best_threshold[j] 79 | ##遍历一开始初始化的候选阈值列表，0.1--0.9的九个候选阈值 80 | for threshold in thresholds: 81 | threshold_init[i] = threshold 82 | predicts = getPreLab(probas, model.id2label, threshold_init) 83 | score, f1 = evaluate(predict_labels=predicts, target_labels=all_labels, tag_dir=tag_file) 84 | scoreOfthreshold[threshold] = score 85 | if score > best_score: 86 | best_threshold[i] = threshold 87 | best_score = score 88 | label_result["score"] = scoreOfthreshold 89 | result["label_score"].append(label_result) 90 | logging.info(best_threshold) 91 | logging.info(label_result) 92 | logging.info("\n") 93 | result["best_threshold"] = best_threshold 94 | logging.info("搜索出来的阈值： %s \n" % best_threshold) 95 | logging.info("————开始将结果写入文件————\n") 96 | if not os.path.exists(threshold_dir): 97 | os.makedirs(threshold_dir) 98 | threshold_file = os.path.join(threshold_dir, "threshold.json") 99 | search_file = os.path.join(threshold_dir, "search.json") 100 | 101 | ouf_t = open(threshold_file, "w", encoding="utf-8") 102 | ouf_s = open(search_file, "w", encoding="utf-8") 103 | json.dump(best_threshold, ouf_t, ensure_ascii=False) 104 | json.dump(result, ouf_s, ensure_ascii=False) 105 | ouf_s.close() 106 | ouf_t.close() 107 | 108 | if __name__ == '__main__': 109 | task = "divorce" 110 | 111 | """整理代码测试，测试文件test_file就用的训练文件，正式使用需要改为切分的测试数据集""" 112 | 113 | searchThreshold(domain=task, model_pb="pb/model.pb", threshold_dir="threshold", 114 | test_file="data/divorce/train_selected.json", tag_file="data/divorce/tags.txt", 115 | vocab_file="/home/huanghui/data/chinese_L-12_H-768_A-12/vocab.txt") 116 | -------------------------------------------------------------------------------- /utils/__init__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | #-*- coding:utf-8 -*- 3 | # author:huanghui 4 | # datetime:2019/9/30 8:51 -------------------------------------------------------------------------------- /utils/ckpt2pb.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | #-*- coding:utf-8 -*- 3 | # author:huanghui 4 | # datetime:2019/9/30 9:58 5 | 6 | from bert import modeling 7 | import os 8 | from tensorflow.python.framework import graph_util 9 | from utils.models import * 10 | 11 | modelMap = {"rcnnatt": RCNNATT, "rcnn": RCNN} 12 | 13 | def create_model(bert_config, input_ids, input_mask, segment_ids, 14 | num_labels, model_type=None): 15 | """ 16 | :param bert_config: 17 | :param input_ids: 18 | :param input_mask: 19 | :param segment_ids: 20 | :param num_labels: 类别数 21 | :param model_type: bert后接的模型类型，rcnn，rcnnatt 22 | :return: sigmoid后的结果 23 | """ 24 | model = modeling.BertModel( 25 | config=bert_config, 26 | is_training=False, 27 | input_ids=input_ids, 28 | input_mask=input_mask, 29 | token_type_ids=segment_ids, 30 | use_one_hot_embeddings=False) 31 | 32 | if model_type: 33 | embedding = model.get_sequence_output() 34 | model_layer = modelMap[model_type]( 35 | embedding=embedding, context_dim=200, hidden_dim=200, dropout_keep_prob=1.0 36 | ) 37 | output_layer = model_layer.getLogits() 38 | else: 39 | output_layer = model.get_pooled_output() 40 | hidden_size = output_layer.shape[-1].value 41 | output_weights = tf.get_variable( 42 | "output_weights", [num_labels, hidden_size], 43 | initializer=tf.truncated_normal_initializer(stddev=0.02)) 44 | 45 | output_bias = tf.get_variable( 46 | "output_bias", [num_labels], initializer=tf.zeros_initializer()) 47 | 48 | with tf.variable_scope("loss"): 49 | output_layer = tf.nn.dropout(output_layer, keep_prob=1.0) 50 | logits = tf.matmul(output_layer, output_weights, transpose_b=True) 51 | logits = tf.nn.bias_add(logits, output_bias) 52 | 53 | probabilities = tf.nn.sigmoid(logits) 54 | return probabilities 55 | 56 | 57 | def convert(task, tagDir, originDir, convertDir, model_type, bert_dir): 58 | """ 59 | :param task: 任务名，divorce，labor， loan 60 | :param tagDir: 任务标签文件，tags.txt 61 | :param originDir: 若是文件夹，则选择最后一个模型，若是文件名，则选择该模型文件。 62 | :param convertDir: 生成的pb模型名叫model.pb，在该目录下 63 | :param model_type: bert后接的模型类型，rcnn，orgin,模型类型都为小写 64 | :param bert_dir: bert预训练模型文件夹，下边只需要包含配置文件和词典 65 | """ 66 | tf.reset_default_graph() 67 | domain2len = {"divorce": 128, "labor": 150, "loan": 200} 68 | max_seg_length = domain2len[task] 69 | 70 | if not os.path.exists(convertDir): 71 | os.makedirs(convertDir) 72 | f = open(tagDir, 'r', encoding='utf-8') 73 | lines = f.readlines() 74 | label = [] 75 | for line in lines: 76 | label.append(line.strip()) 77 | f.close() 78 | num_labels = len(label) 79 | 80 | gpu_config = tf.ConfigProto() 81 | gpu_config.gpu_options.allow_growth = True 82 | sess = tf.Session(config=gpu_config) 83 | graph = tf.get_default_graph() 84 | with graph.as_default(): 85 | input_ids_p = tf.placeholder(tf.int32, [None, max_seg_length], name="input_ids") 86 | input_mask_p = tf.placeholder(tf.int32, [None, max_seg_length], name="input_mask") 87 | segment_ids_p = tf.placeholder(tf.int32, [None, max_seg_length], name="segment_ids") 88 | bert_config = modeling.BertConfig.from_json_file(os.path.join(bert_dir, 'bert_config.json')) 89 | probabilities = create_model( 90 | bert_config=bert_config, input_ids=input_ids_p, input_mask=input_mask_p, 91 | segment_ids=segment_ids_p, num_labels=num_labels, model_type=model_type 92 | ) 93 | probabilities = tf.identity(probabilities, 'pred_prob') 94 | saver = tf.train.Saver() 95 | if os.path.isdir(originDir): 96 | saver.restore(sess, tf.train.latest_checkpoint(originDir)) 97 | else: 98 | saver.restore(sess, originDir) 99 | tmp_g = graph_util.convert_variables_to_constants(sess, graph.as_graph_def(), ['pred_prob']) 100 | with tf.gfile.GFile(os.path.join(convertDir, "model.pb"), 'wb') as f: 101 | f.write(tmp_g.SerializeToString()) 102 | 103 | -------------------------------------------------------------------------------- /utils/evaluate.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | #-*- coding:utf-8 -*- 3 | # author:huanghui 4 | # datetime:2019/9/30 10:28 5 | 6 | class Judger: 7 | # Initialize Judger, with the path of tag list 8 | def __init__(self, tag_path): 9 | self.tag_dic = {} 10 | f = open(tag_path, "r", encoding='utf-8') 11 | self.task_cnt = 0 12 | for line in f: 13 | self.task_cnt += 1 14 | self.tag_dic[line.strip()] = self.task_cnt 15 | 16 | 17 | # Format the result generated by the Predictor class 18 | @staticmethod 19 | def format_result(result): 20 | rex = {"tags": []} 21 | res_art = [] 22 | for x in result["tags"]: 23 | if not (x is None): 24 | res_art.append(int(x)) 25 | rex["tags"] = res_art 26 | 27 | return rex 28 | 29 | # Gen new results according to the truth and users output 30 | def gen_new_result(self, result, truth, label): 31 | 32 | s1 = set() 33 | for tag in label: 34 | s1.add(self.tag_dic[tag.replace(' ', '')]) 35 | s2 = set() 36 | for name in truth: 37 | 38 | s2.add(self.tag_dic[name.replace(' ', '')]) 39 | 40 | for a in range(0, self.task_cnt): 41 | in1 = (a + 1) in s1 42 | in2 = (a + 1) in s2 43 | if in1: 44 | if in2: 45 | result[0][a]["TP"] += 1 46 | else: 47 | result[0][a]["FP"] += 1 48 | else: 49 | if in2: 50 | result[0][a]["FN"] += 1 51 | else: 52 | result[0][a]["TN"] += 1 53 | 54 | return result 55 | 56 | # Calculate precision, recall and f1 value 57 | # According to https://github.com/dice-group/gerbil/wiki/Precision,-Recall-and-F1-measure 58 | @staticmethod 59 | def get_value(res): 60 | if res["TP"] == 0: 61 | if res["FP"] == 0 and res["FN"] == 0: 62 | precision = 1.0 63 | recall = 1.0 64 | f1 = 1.0 65 | else: 66 | precision = 0.0 67 | recall = 0.0 68 | f1 = 0.0 69 | else: 70 | precision = 1.0 * res["TP"] / (res["TP"] + res["FP"]) 71 | recall = 1.0 * res["TP"] / (res["TP"] + res["FN"]) 72 | f1 = 2 * precision * recall / (precision + recall) 73 | 74 | return precision, recall, f1 75 | 76 | # Generate score 77 | def gen_score(self, arr): 78 | sumf = 0 79 | f1 = {} 80 | y = {"TP": 0, "FP": 0, "FN": 0, "TN": 0} 81 | i = 0 82 | for x in arr[0]: 83 | i += 1 84 | p, r, f = self.get_value(x) 85 | f1[str(i)] = round(f, 2) 86 | sumf += f 87 | for z in x.keys(): 88 | y[z] += x[z] 89 | 90 | _, __, f_ = self.get_value(y) 91 | 92 | 93 | 94 | return (f_ + sumf * 1.0 / len(arr[0])) / 2.0, f1 95 | 96 | # Test with ground truth path and the user's output path 97 | def test(self, truth_label, pre_label): 98 | cnt = 0 99 | result = [[]] 100 | for a in range(0, self.task_cnt): 101 | result[0].append({"TP": 0, "FP": 0, "TN": 0, "FN": 0}) 102 | 103 | for i in range(len(truth_label)): 104 | cnt += 1 105 | result = self.gen_new_result(result, truth_label[i], pre_label[i]) 106 | return result 107 | 108 | def evaluate(predict_labels, target_labels, tag_dir): 109 | """传入预测标签，目标标签，tags地址。""" 110 | """标签需要真实标签，而不是id，例如["dv1","dv2"]""" 111 | judger = Judger(tag_dir) 112 | reslt = judger.test(target_labels, predict_labels) 113 | 114 | score, f1 = judger.gen_score(reslt) 115 | return score, f1 -------------------------------------------------------------------------------- /utils/models.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | class RCNN(): 4 | def __init__(self, embedding, context_dim, hidden_dim, dropout_keep_prob): 5 | """ 6 | :param embedding: bert生成的embedding 7 | :param context_dim: lstm隐藏层维度 8 | :param hidden_dim:全连接层隐藏层维度 9 | :param dropout_keep_prob:lstm keep_prob 10 | :return 调用getLogits返回logits 11 | """ 12 | self.embedding_dim = embedding.shape[-1].value 13 | self.embedding = embedding 14 | self.dropout_keep_prob = dropout_keep_prob 15 | self.context_dim = context_dim 16 | self.hidden_dim = hidden_dim 17 | 18 | with tf.name_scope("bi_rnn"): 19 | fw_cell = tf.nn.rnn_cell.BasicLSTMCell(self.context_dim) 20 | fw_cell = tf.nn.rnn_cell.DropoutWrapper(fw_cell, self.dropout_keep_prob) 21 | bw_cell = tf.nn.rnn_cell.BasicLSTMCell(self.context_dim) 22 | bw_cell = tf.nn.rnn_cell.DropoutWrapper(bw_cell, self.dropout_keep_prob) 23 | (output_fw, output_bw), states = tf.nn.bidirectional_dynamic_rnn(cell_fw=fw_cell, 24 | cell_bw=bw_cell, 25 | inputs=self.embedding, 26 | dtype=tf.float32) 27 | with tf.name_scope("context"): 28 | shape = [tf.shape(output_fw)[0], 1, tf.shape(output_fw)[2]] 29 | c_left = tf.concat([tf.zeros(shape), output_fw[:, :-1]], axis=1, name="context_left") 30 | c_right = tf.concat([output_bw[:, 1:], tf.zeros(shape)], axis=1, name="context_right") 31 | 32 | with tf.name_scope("word_representation"): 33 | y2 = tf.concat([c_left, self.embedding, c_right], axis=2, name="word_representation") 34 | 35 | # max_pooling层 36 | with tf.name_scope("max_pooling"): 37 | fc = tf.layers.dense(y2, self.hidden_dim, activation=tf.nn.relu, name='fc1') 38 | self.output = tf.reduce_max(fc, axis=1) 39 | 40 | def getLogits(self): 41 | return self.output 42 | 43 | class RCNNATT(): 44 | def __init__(self, embedding, context_dim, hidden_dim, dropout_keep_prob): 45 | """ 46 | :param embedding: bert生成的embedding 47 | :param context_dim: lstm隐藏层维度 48 | :param hidden_dim:全连接层隐藏层维度 49 | :param dropout_keep_prob:lstm keep_prob 50 | :return 调用getLogits返回logits 51 | """ 52 | self.embedding_dim = embedding.shape[-1].value 53 | self.embedding = embedding 54 | self.dropout_keep_prob = dropout_keep_prob 55 | self.context_dim = context_dim 56 | self.hidden_dim = hidden_dim 57 | 58 | with tf.name_scope("bi_rnn"): 59 | fw_cell = tf.nn.rnn_cell.BasicLSTMCell(self.context_dim) 60 | fw_cell = tf.nn.rnn_cell.DropoutWrapper(fw_cell, self.dropout_keep_prob) 61 | bw_cell = tf.nn.rnn_cell.BasicLSTMCell(self.context_dim) 62 | bw_cell = tf.nn.rnn_cell.DropoutWrapper(bw_cell, self.dropout_keep_prob) 63 | (output_fw, output_bw), states = tf.nn.bidirectional_dynamic_rnn(cell_fw=fw_cell, 64 | cell_bw=bw_cell, 65 | inputs=self.embedding, 66 | dtype=tf.float32) 67 | with tf.name_scope("context"): 68 | shape = [tf.shape(output_fw)[0], 1, tf.shape(output_fw)[2]] 69 | c_left = tf.concat([tf.zeros(shape), output_fw[:, :-1]], axis=1, name="context_left") 70 | c_right = tf.concat([output_bw[:, 1:], tf.zeros(shape)], axis=1, name="context_right") 71 | 72 | with tf.name_scope("word_representation"): 73 | y2 = tf.concat([c_left, self.embedding, c_right], axis=2, name="word_representation") 74 | 75 | 76 | # # max_pooling层 77 | # with tf.name_scope("max_pooling"): 78 | # fc = tf.layers.dense(y2, self.hidden_dim, activation=tf.nn.relu, name='fc1') 79 | # fc_pool = tf.reduce_max(fc, axis=1) 80 | 81 | with tf.name_scope("attention"): 82 | hidden_size = y2.shape[2].value 83 | u_omega = tf.get_variable("u_omega", [hidden_size]) 84 | with tf.name_scope('v'): 85 | v = tf.tanh(y2) 86 | vu = tf.tensordot(v, u_omega, axes=1, name='vu') 87 | alphas = tf.nn.softmax(vu, name='alphas') 88 | output = tf.reduce_sum(y2 * tf.expand_dims(alphas, -1), 1) 89 | self.output = tf.tanh(output) 90 | 91 | def getLogits(self): 92 | return self.output 93 | 94 | 95 | -------------------------------------------------------------------------------- /utils/predict.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | #-*- coding:utf-8 -*- 3 | # author:huanghui 4 | # datetime:2019/9/30 9:42 5 | 6 | import tensorflow as tf 7 | from bert import tokenization 8 | from tensorflow.python.platform import gfile 9 | import numpy as np 10 | from tqdm import tqdm 11 | 12 | pre_batch_size = 100 13 | 14 | class InputFeatures(object): 15 | """A single set of features of data.""" 16 | def __init__(self, 17 | input_ids, 18 | input_mask, 19 | segment_ids, 20 | label_id, 21 | is_real_example=True): 22 | self.input_ids = input_ids 23 | self.input_mask = input_mask 24 | self.segment_ids = segment_ids 25 | self.label_id = label_id 26 | self.is_real_example = is_real_example 27 | 28 | def convert_single_example(sent, label_list, max_seq_length, 29 | tokenizer): 30 | label_map = {} 31 | for (i, label) in enumerate(label_list): 32 | label_map[label] = i 33 | 34 | tokens_a = sent 35 | 36 | if len(tokens_a) > max_seq_length - 2: 37 | tokens_a = tokens_a[0:(max_seq_length - 2)] 38 | # bound = int((max_seq_length-2)/2) 39 | # tokens_a = tokens_a[0:bound] + tokens_a[(len(tokens_a)-bound):] 40 | tokens = [] 41 | segment_ids = [] 42 | tokens.append("[CLS]") 43 | segment_ids.append(0) 44 | for token in tokens_a: 45 | tokens.append(token) 46 | segment_ids.append(0) 47 | tokens.append("[SEP]") 48 | segment_ids.append(0) 49 | 50 | input_ids = tokenizer.convert_tokens_to_ids(tokens) 51 | 52 | input_mask = [1] * len(input_ids) 53 | 54 | while len(input_ids) < max_seq_length: 55 | input_ids.append(0) 56 | input_mask.append(0) 57 | segment_ids.append(0) 58 | 59 | assert len(input_ids) == max_seq_length 60 | assert len(input_mask) == max_seq_length 61 | assert len(segment_ids) == max_seq_length 62 | ##label是一个列表 63 | # label_id = label_map[example.label] 64 | 65 | label_id = [0]*len(label_map) 66 | feature = InputFeatures( 67 | input_ids=input_ids, 68 | input_mask=input_mask, 69 | segment_ids=segment_ids, 70 | label_id=label_id, 71 | is_real_example=True) 72 | return feature 73 | 74 | class BERTModel: 75 | 76 | def __init__(self, task, pb_model, tagDir, threshold, vocab_file): 77 | """ 78 | :param task: 任务类型，包括divorce，labor，loan 79 | :param pb_model: pb模型文件名 80 | :param tagDir: 任务标签tags文件 81 | :param threshold: 多标签分类的阈值列表 82 | :param vocab_file: bert词典文件 vocab.txt 83 | """ 84 | tf.reset_default_graph() 85 | domain2len = {"divorce": 128, "labor": 150, "loan": 200} 86 | self.max_seg_length = domain2len[task] 87 | self.pb_model = pb_model 88 | self.vocab_file = vocab_file 89 | self.label_dir = tagDir 90 | 91 | gpu_config = tf.ConfigProto() 92 | gpu_config.gpu_options.allow_growth = True 93 | self.sess = tf.Session(config=gpu_config) 94 | ##加载阈值列表 95 | self.threshold = threshold 96 | 97 | f = open(self.label_dir, 'r', encoding='utf-8') 98 | lines = f.readlines() 99 | self.label = [] 100 | for line in lines: 101 | self.label.append(line.strip()) 102 | f.close() 103 | # 生成label和id转换字典 104 | self.label2id = {} 105 | for (i, label) in enumerate(self.label): 106 | self.label2id[label] = i 107 | self.id2label = {value: key for key, value in self.label2id.items()} 108 | self.tokenizer = tokenization.FullTokenizer( 109 | vocab_file=self.vocab_file, do_lower_case=True) 110 | self.sess = tf.Session() 111 | with gfile.FastGFile(pb_model, 'rb') as f: 112 | self.graph = tf.GraphDef() 113 | self.graph.ParseFromString(f.read()) 114 | self.sess.graph.as_default() 115 | tf.import_graph_def(self.graph, name='') 116 | self.sess.run(tf.global_variables_initializer()) 117 | self.input_ids_p = self.sess.graph.get_tensor_by_name('input_ids:0') 118 | self.input_mask_p = self.sess.graph.get_tensor_by_name('input_mask:0') 119 | self.segment_ids_p = self.sess.graph.get_tensor_by_name('segment_ids:0') 120 | self.probabilities = self.sess.graph.get_tensor_by_name('pred_prob:0') 121 | 122 | def convert(self, line): 123 | feature = convert_single_example(line, self.label, self.max_seg_length, self.tokenizer) 124 | input_ids = feature.input_ids 125 | input_mask = feature.input_mask 126 | segment_ids = feature.segment_ids 127 | label_ids = feature.label_id 128 | return input_ids, input_mask, segment_ids, label_ids 129 | 130 | def getAllResult(self, sentences): 131 | """一次性预测所有句子""" 132 | step = int(len(sentences) / pre_batch_size) 133 | all_result = [] 134 | for i in tqdm(range(step)): 135 | result = self.predict(sentences[i * pre_batch_size:(i + 1) * pre_batch_size]) 136 | all_result.extend(result) 137 | if len(all_result) < len(sentences): 138 | result = self.predict(sentences[len(all_result):]) 139 | all_result.extend(result) 140 | return all_result 141 | 142 | def rematch(self, arrays): 143 | predict_list = [] 144 | for array in arrays: 145 | temp = [] 146 | for i in range(len(array)): 147 | if array[i] == 1: 148 | temp.append(self.id2label[i]) 149 | predict_list.append(temp) 150 | return predict_list 151 | 152 | """predict返回的是一个二维列表，存储预测结果[[], ['DV1', 'DV2']]""" 153 | def predict(self, sentences): 154 | """预测小批量句子""" 155 | def getPre(arr, id2label): 156 | predict_list = [] 157 | for i in range(len(arr)): 158 | if arr[i] > self.threshold[i]: 159 | predict_list.append(id2label[i]) 160 | return predict_list 161 | 162 | def getPredictLabel(array, id2label): 163 | proba = array[0] 164 | result = [] 165 | for p in proba: 166 | result.append(getPre(p, id2label)) 167 | 168 | return result 169 | 170 | input_ids_list = [] 171 | input_mask_list = [] 172 | segment_ids_list = [] 173 | 174 | for sentence in sentences: 175 | sentence = self.tokenizer.tokenize(sentence) 176 | input_ids, input_mask, segment_ids, label_ids = self.convert(sentence) 177 | input_ids_list.append(input_ids) 178 | input_mask_list.append(input_mask) 179 | segment_ids_list.append(segment_ids) 180 | 181 | feed_dict = {self.input_ids_p: input_ids_list, 182 | self.input_mask_p: input_mask_list, 183 | self.segment_ids_p: segment_ids_list} 184 | probabilities_ = self.sess.run([self.probabilities], feed_dict) 185 | 186 | result = getPredictLabel(probabilities_, self.id2label) 187 | return result 188 | 189 | def getProb(self, sentences): 190 | input_ids_list = [] 191 | input_mask_list = [] 192 | segment_ids_list = [] 193 | 194 | for sentence in sentences: 195 | sentence = self.tokenizer.tokenize(sentence) 196 | input_ids, input_mask, segment_ids, label_ids = self.convert(sentence) 197 | input_ids_list.append(input_ids) 198 | input_mask_list.append(input_mask) 199 | segment_ids_list.append(segment_ids) 200 | 201 | feed_dict = {self.input_ids_p: input_ids_list, 202 | self.input_mask_p: input_mask_list, 203 | self.segment_ids_p: segment_ids_list} 204 | probabilities_ = self.sess.run([self.probabilities], feed_dict) 205 | 206 | return probabilities_[0] 207 | 208 | def getProbs(self, sentences): 209 | step = int(len(sentences) / pre_batch_size) 210 | all_result = [] 211 | for i in tqdm(range(step)): 212 | probs = self.getProb(sentences[i * pre_batch_size:(i + 1) * pre_batch_size]) 213 | all_result.extend(probs) 214 | if len(all_result) < len(sentences): 215 | probs = self.getProb(sentences[len(all_result):]) 216 | all_result.extend(probs) 217 | all_result = np.asarray(all_result) 218 | return all_result --------------------------------------------------------------------------------