├── README.md ├── classifier_multi_label ├── README.md ├── __init__.py ├── albert_small_zh_google │ ├── albert_config.json │ └── vocab_chinese.txt ├── classifier_utils.py ├── data │ ├── test.csv │ ├── test_onehot.csv │ ├── train.csv │ ├── train_onehot.csv │ └── vocabulary_label.txt ├── eval.py ├── hyperparameters.py ├── image │ └── graph.png ├── lamb_optimizer.py ├── logdir │ └── model_01 │ │ └── events.out.tfevents.1595940723.DESKTOP-QC1A83I ├── model │ ├── model_load │ │ └── README.md │ └── model_save │ │ └── READEME.md ├── modeling.py ├── modules.py ├── networks.py ├── optimization.py ├── predict.py ├── tokenization.py ├── train.py └── utils.py ├── classifier_multi_label_denses ├── README.md ├── __init__.py ├── albert_small_zh_google │ └── albert_config.json ├── classifier_utils.py ├── data │ ├── test.csv │ ├── test_onehot.csv │ ├── train.csv │ ├── train_onehot.csv │ └── vocabulary_label.txt ├── eval.py ├── hyperparameters.py ├── lamb_optimizer.py ├── logdir │ └── CML_Denses │ │ └── events.out.tfevents.1602241580.DESKTOP-QC1A83I ├── model │ ├── model_load │ │ └── README.md │ └── model_save │ │ └── README.md ├── modeling.py ├── networks.py ├── optimization.py ├── predict.py ├── tokenization.py ├── train.py └── utils.py ├── classifier_multi_label_seq2seq ├── README.md ├── __pycache__ │ ├── classifier_utils.cpython-37.pyc │ ├── hyperparameters.cpython-37.pyc │ ├── lamb_optimizer.cpython-37.pyc │ ├── load.cpython-37.pyc │ ├── modeling.cpython-37.pyc │ ├── modules.cpython-37.pyc │ ├── networks.cpython-37.pyc │ ├── optimization.cpython-37.pyc │ ├── predict.cpython-37.pyc │ ├── tokenization.cpython-37.pyc │ └── utils.cpython-37.pyc ├── albert_small_zh_google │ ├── checkpoint │ └── vocab_chinese.txt ├── classifier_utils.py ├── data │ ├── test.csv │ ├── train.csv │ └── vocabulary_label.txt ├── eval.py ├── hyperparameters.py ├── lamb_optimizer.py ├── load.py ├── logdir │ └── CML_Seq2Seq │ │ └── Readme.txt ├── model │ ├── model_load │ │ └── README.md │ └── model_save │ │ └── README.md ├── modeling.py ├── modules.py ├── networks.py ├── optimization.py ├── predict.py ├── tokenization.py ├── train.py └── utils.py ├── classifier_multi_label_textcnn ├── README.md ├── __init__.py ├── albert_small_zh_google │ └── albert_config.json ├── classifier_utils.py ├── data │ ├── test.csv │ ├── test_onehot.csv │ ├── train.csv │ ├── train_onehot.csv │ └── vocabulary_label.txt ├── eval.py ├── hyperparameters.py ├── image │ ├── graph.png │ └── loss.png ├── lamb_optimizer.py ├── model │ ├── model_load │ │ └── README.md │ └── model_save │ │ └── README.md ├── modeling.py ├── modules.py ├── networks.py ├── optimization.py ├── predict.py ├── tokenization.py ├── train.py └── utils.py ├── evaluation ├── README.md └── eval_multi_label.py └── imgs ├── 01.png ├── 01b.png ├── 02.png ├── 09.png ├── 09b.jpg ├── 10.png ├── HELLONLP.png ├── base.png ├── denses.jpg ├── denses01.png ├── denses02.jpg ├── denses02.png ├── seq2seq.png └── textcnn.png /README.md: -------------------------------------------------------------------------------- 1 | # Text Classification Multi-Label: 多标签文本分类 2 | [![Python](https://img.shields.io/badge/python-3.7.6-blue?logo=python&logoColor=FED643)](https://www.python.org/downloads/release/python-376/) 3 | [![Pytorch](https://img.shields.io/badge/tensorflow-1.15.0-red?logo=tensorflow)](https://www.tensorflow.org/versions/) 4 | 5 | 6 | ## 一、简介 7 | ### 1. 多元分类 8 | 多分类任务中一条数据只有一个标签，但这个标签可能有多种类别。比如判定某个人的性别，只能归类为"男性"、"女性"其中一个。再比如判断一个文本的情感只能归类为"正面"、"中面"或者"负面"其中一个。 9 | ### 2. 多标签分类 10 | 多标签分类任务中一条数据可能有多个标签，每个标签可能有两个或者多个类别。例如，一篇新闻可能同时归类为"娱乐"和"运动"，也可能只属于"娱乐"或者其它类别。 11 | 12 |
13 | 14 | 15 | 16 |
17 | 18 | 19 | ## 二、算法 20 | 21 | **4种实现方法** 22 | ``` 23 | ├── classifier_multi_label 24 | └── classifier_multi_label 25 | └── classifier_multi_label_textcnn 26 | └── classifier_multi_label_denses 27 | └── classifier_multi_label_seq2seq 28 | ``` 29 | 30 | ### 1. classifier_multi_label 31 | 32 | 33 | - 使用BERT第一个token[CLS]的向量，维度为(batch_size,hidden_size)。 34 | - 使用了tf.nn.sigmoid_cross_entropy_with_logits的损失函数。 35 | - 使用了tf.where函数来选择概率小于0.5的对应id。 36 | 37 | ### 2. classifier_multi_label_textcnn 38 | 39 | 40 | - 使用BERT输出的三维向量，维度为(batch_size,sequence_length,hidden_size)，然后做为输入进入TextCNN层。 41 | - 使用了tf.nn.sigmoid_cross_entropy_with_logits的损失函数。 42 | - 使用了tf.where函数来选择概率小于0.5的对应id。 43 | 44 | ### 3. classifier_multi_label_denses 45 | 46 | 47 | 48 | - 使用BERT第一个token[CLS]的向量，维度为(batch_size,hidden_size)，然后通过多个二分类(全连接层)来解决多标签分类问题。 49 | - 使用了tf.nn.softmax_cross_entropy_with_logits的损失函数。 50 | - 使用了tf.argmax函数来选择输出最高概率。 51 | 52 | ### 4. classifier_multi_label_seq2seq 53 | 54 | 55 | - 使用BERT输出的三维向量，维度为(batch_size,sequence_length,hidden_size)，然后做为输入进入seq2seq+attention层。 56 | - 使用了tf.nn.softmax_cross_entropy_with_logits的损失函数。 57 | - 使用了beam search 来解码输出概率。 58 | 59 |
60 | 61 | ## 三、实验 62 | ### 1. 训练过程 63 | 64 | 65 | ### 2. 实验结果 66 | 67 | 68 | 69 | ### 3. 实验结论 70 | - 如果对推理速度的要求不是非常高，基于ALBERT+Seq2Seq_Attention框架的多标签文本分类效果最好。 71 | - 如果对推理速度和模型效果要求都非常高，基于ALBERT+TextCNN会是一个不错的选择。 72 | 73 |
74 | 75 | ## 参考 76 | [多标签文本分类介绍，以及对比训练](https://zhuanlan.zhihu.com/p/152140983) 77 | [多标签文本分类 [ALBERT]](https://zhuanlan.zhihu.com/p/164873441) 78 | [多标签文本分类 [ALBERT+TextCNN]](https://zhuanlan.zhihu.com/p/158622992) 79 | [多标签文本分类 [ALBERT+Multi_Denses]](https://zhuanlan.zhihu.com/p/263573628) 80 | [多标签文本分类 [ALBERT+Seq2Seq+Attention]](https://zhuanlan.zhihu.com/p/260743336) 81 | 82 | 83 | 84 | 85 | -------------------------------------------------------------------------------- /classifier_multi_label/README.md: -------------------------------------------------------------------------------- 1 | # 简介 2 | 1、本项目是在tensorflow版本1.15.0的基础上做的训练和测试。 3 | 2、本项目为中文的多标签文本分类。 4 | 3、欢迎大家联系我 www.hellonlp.com 5 | 4、albert_small_zh_google对应的百度云下载地址： 6 | 链接：https://pan.baidu.com/s/1RKzGJTazlZ7y12YRbAWvyA 7 | 提取码：wuxw 8 | 9 | 10 | # 使用方法 11 | 1、准备数据 12 | 数据格式为：classifier_multi_label/data/test_onehot.csv 13 | 2、参数设置 14 | 参考脚本 hyperparameters.py，直接修改里面的数值即可。 15 | 3、训练 16 | ``` 17 | python train.py 18 | ``` 19 | 4、预测 20 | ``` 21 | python predict.py 22 | ``` 23 | 注意：预测时需要把model/save中的模型复制到model/load中，并修改model/load中的checkpoint文件的内容为当前模型名称，例如：model_checkpoint_path: "model_xx_xx.ckpt"。 24 | 。 25 | 26 | 27 | 28 | # 知乎解读 29 | https://zhuanlan.zhihu.com/p/164873441 30 | 31 | -------------------------------------------------------------------------------- /classifier_multi_label/__init__.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | -------------------------------------------------------------------------------- /classifier_multi_label/albert_small_zh_google/albert_config.json: -------------------------------------------------------------------------------- 1 | { 2 | "attention_probs_dropout_prob": 0.0, 3 | "hidden_act": "gelu", 4 | "hidden_dropout_prob": 0.0, 5 | "embedding_size": 128, 6 | "hidden_size": 384, 7 | "initializer_range": 0.02, 8 | "intermediate_size": 1536, 9 | "max_position_embeddings": 512, 10 | "num_attention_heads": 12, 11 | "num_hidden_layers": 6, 12 | "num_hidden_groups": 1, 13 | "net_structure_type": 0, 14 | "gap_size": 0, 15 | "num_memory_blocks": 0, 16 | "inner_group_num": 1, 17 | "down_scale_factor": 1, 18 | "type_vocab_size": 2, 19 | "vocab_size": 21128 20 | } -------------------------------------------------------------------------------- /classifier_multi_label/data/test.csv: -------------------------------------------------------------------------------- 1 | content,label 2 | 也没有烧伤,| 3 | 主要是之前在北方呆过几几年,| 4 | 温暖过冬,| 5 | 开箱试了一下看着很不错,| 6 | 还可以给宝宝凉衣服,| 7 | 快递师傅送货迅速,| 8 | 快递员差评,| 9 | 沒有什么噪音,| 10 | 就感觉有一部分是强行捏到一起去的,| 11 | 今天刚到货还没有使用,| 12 | 后面还有个设计,| 13 | 非常好用的宝贝,| 14 | 这个开关我是一直没搞懂,| 15 | 该有的功能都有了,| 16 | 我只说一次,| 17 | 收到l等了15天,| 18 | 所以系统默认好评,| 19 | 但是卖家很快就补发了,| 20 | 现在过了二十天了也没退,| 21 | 冬天有她很暖和,制热效果 22 | 收到第一时间体验,| 23 | 刮花了,| 24 | 也不知道耗不好点,| 25 | 表扬一下物流服务,| 26 | 外观设计：不站地方,| 27 | 换了小米,| 28 | 总之我自己用下来,| 29 | 价格实惠量又足,| 30 | 耗电情况：整体来说耗电不是特别严重,产品功耗 31 | 小兔兔外型还不错了,| 32 | 其他特色：利用小爱或者APP可以智能设定,| 33 | 京东的服务超好,| 34 | 这款取暖器已经买两个了,| 35 | 吹到的风才暖,| 36 | 来的及时,| 37 | 全网最便宜,| 38 | 错过了7天试用,| 39 | 连几十块钱的取暖器都不如,| 40 | 家里有小朋友的建议还是使用空调,| 41 | 定时升温控制都很简单,| 42 | 用了近一月,| 43 | 南方湿冷的冬天是很不错的选择,| 44 | 但是我刚开始使用确实有味道,味道 45 | 说是也转到人工客服,| 46 | 只有在旁边能暖一点,| 47 | 2、结构简单,| 48 | 是想象的模样,| 49 | 1档热风和2档热风,| 50 | 除了物流慢一点点,| 51 | 5分钟屋里就暖和了,制热效果 52 | 碰到长期下雨阴雨的天气,| 53 | 颜色比图片还好看,| 54 | 用起来还可以吧,| 55 | 就是感觉大了一点,| 56 | 安全设计：反正加热中我是一点不敢碰的,| 57 | 尤其自营商品,| 58 | 实在是感人,| 59 | 到现在按钮也开不起,| 60 | 开关,| 61 | 放在客房足够用,| 62 | 一会就暖和,制热效果 63 | 回来赶快拆了包装试下效果,| 64 | 性价比还好吧,| 65 | 使用过后一段时间感觉还不错,| 66 | 但是售后服务太差,| 67 | *元多一点,| 68 | 一如既往地又快又好呀,| 69 | 给东北老家买的,| 70 | 再过几分钟就感觉到开始散热了,| 71 | 整个屋子都很暖静音效果：完全没有声音,声音 72 | 比老款要美观,| 73 | 是划线价的一半多一丢丢,| 74 | 这次再买个送婆婆,| 75 | 小太阳完胜,| 76 | 最重要的是,| 77 | 功能强大发热快,制热效果 78 | 塑料味道太大,味道 79 | 格力原装产品,| 80 | 取暖效果：取暖好,制热效果 81 | 是我们买的第二个了,| 82 | 需要个人安装的也不多,| 83 | 没有任何破损的地方,| 84 | 自营售后服务品质有保障,| 85 | 南方冬天暖气是刚需啊,| 86 | 用了两次我和孩子都喉咙发炎了,| 87 | 十几平方的房间开一夜都没觉得有效果,| 88 | 概念&rdquo,| 89 | 外观设计：很不错取暖效果：很好耗电情况：蛮高的静音效果：很不错安全设计：安全其他特色：没有了,产品功耗/制热效果 90 | 我觉得效果比浴霸好,制热效果 91 | 可根据需要选择功率,| 92 | 身边的朋友都有介绍此款,| 93 | 安全设计：设计独特完美,| 94 | 总体来说物美价廉,| 95 | 放书桌旁干活用很合适,| 96 | 如果属于南方的没暖气喜欢的宝宝可以放心购买,| 97 | 还赠送了晾衣架,| 98 | 贴墙放置,| 99 | 非常的美观还耐用,| 100 | 买了你就后悔,| 101 | 说24小时回电话,| 102 | 家里离不开了,| 103 | 估计到冬天没太阳的时候烘衣服方便,| 104 | 十天用了一百六的电费,产品功耗 105 | 有遥控装置,| 106 | 体积虽小,| 107 | 不把手凑上去都感觉不到他的温度,| 108 | 用了两天的,| 109 | 小房间很快就能加热,| 110 | 当天下单第二天就到,| 111 | 加热很快??,| 112 | 静音效果：二档声音不小,| 113 | 也没有长螺丝刀,| 114 | 平均每天就开两小时,| 115 | 但声音还能接受,| 116 | 静音效果：试用了几天,| 117 | 取暖效果：开到二挡很暖和,制热效果 118 | 今年给老人又买一个,| 119 | 看着小,| 120 | 外形取暖效果：超级棒耗电情况：没在意静音效果：安静舒适安全设计：童锁设计省心,产品功耗 121 | 不知道能不能优化,| 122 | 货到手没有一点损坏,| 123 | 今天就到货了,| 124 | 希望要买好好选别家的,| 125 | 利用热空气上升的自然流动,| 126 | 长的差不多也是200左右,| 127 | 现在穿着睡衣,| 128 | 产品的质量不错,| 129 | 希望能耐用,| 130 | 再也不用担心受凉感冒了,| 131 | 一晚耗电30不是梦,产品功耗 132 | 还会有电火花闪现,| 133 | 有一定的防水,| 134 | 双十二特价很便宜,| 135 | 但是一个小时两度电也是真的,| 136 | 在广州用很合适,| 137 | 大人小朋友不容易烫伤,| 138 | 选择小米唯一吸引我的是支持小爱和手机,| 139 | 着实让我觉得有点生气,| 140 | 主要是家里没有安装暖气片,| 141 | 好多人都和我一样的情况,| 142 | 有风扇设计所致,| 143 | 外观设计：很好看取暖效果：取暖很快效果很好,制热效果 144 | 强卖,| 145 | 还危险了,| 146 | 必须要贴着机器才有点温度,| 147 | 耐用也就有安全保障其他特色：冬天衣服难干,| 148 | 时不时的会咔的一声响,| 149 | 就算开低档温度也不错,| 150 | 挺适合南方的冬天,| 151 | 这东西买的真好,| 152 | -------------------------------------------------------------------------------- /classifier_multi_label/data/test_onehot.csv: -------------------------------------------------------------------------------- 1 | content,|,互联互通,产品功耗,滑轮提手,声音,APP操控性,呼吸灯,外观,底座,制热范围,遥控器电池,味道,制热效果,衣物烘干,体积大小 2 | 也没有烧伤,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 3 | 主要是之前在北方呆过几几年,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 4 | 温暖过冬,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 5 | 开箱试了一下看着很不错,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 6 | 还可以给宝宝凉衣服,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 7 | 快递师傅送货迅速,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 8 | 快递员差评,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 9 | 沒有什么噪音,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 10 | 就感觉有一部分是强行捏到一起去的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 11 | 今天刚到货还没有使用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 12 | 后面还有个设计,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 13 | 非常好用的宝贝,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 14 | 这个开关我是一直没搞懂,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 15 | 该有的功能都有了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 16 | 我只说一次,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 17 | 收到l等了15天,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 18 | 所以系统默认好评,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 19 | 但是卖家很快就补发了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 20 | 现在过了二十天了也没退,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 21 | 冬天有她很暖和,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 22 | 收到第一时间体验,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 23 | 刮花了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 24 | 也不知道耗不好点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 25 | 表扬一下物流服务,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 26 | 外观设计：不站地方,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 27 | 换了小米,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 28 | 总之我自己用下来,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 29 | 价格实惠量又足,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 30 | 耗电情况：整体来说耗电不是特别严重,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 31 | 小兔兔外型还不错了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 32 | 其他特色：利用小爱或者APP可以智能设定,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 33 | 京东的服务超好,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 34 | 这款取暖器已经买两个了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 35 | 吹到的风才暖,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 36 | 来的及时,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 37 | 全网最便宜,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 38 | 错过了7天试用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 39 | 连几十块钱的取暖器都不如,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 40 | 家里有小朋友的建议还是使用空调,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 41 | 定时升温控制都很简单,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 42 | 用了近一月,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 43 | 南方湿冷的冬天是很不错的选择,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 44 | 但是我刚开始使用确实有味道,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0 45 | 说是也转到人工客服,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 46 | 只有在旁边能暖一点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 47 | 2、结构简单,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 48 | 是想象的模样,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 49 | 1档热风和2档热风,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 50 | 除了物流慢一点点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 51 | 5分钟屋里就暖和了,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 52 | 碰到长期下雨阴雨的天气,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 53 | 颜色比图片还好看,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 54 | 用起来还可以吧,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 55 | 就是感觉大了一点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 56 | 安全设计：反正加热中我是一点不敢碰的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 57 | 尤其自营商品,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 58 | 实在是感人,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 59 | 到现在按钮也开不起,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 60 | 开关,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 61 | 放在客房足够用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 62 | 一会就暖和,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 63 | 回来赶快拆了包装试下效果,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 64 | 性价比还好吧,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 65 | 使用过后一段时间感觉还不错,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 66 | 但是售后服务太差,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 67 | *元多一点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 68 | 一如既往地又快又好呀,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 69 | 给东北老家买的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 70 | 再过几分钟就感觉到开始散热了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 71 | 整个屋子都很暖静音效果：完全没有声音,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0 72 | 比老款要美观,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 73 | 是划线价的一半多一丢丢,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 74 | 这次再买个送婆婆,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 75 | 小太阳完胜,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 76 | 最重要的是,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 77 | 功能强大发热快,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 78 | 塑料味道太大,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0 79 | 格力原装产品,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 80 | 取暖效果：取暖好,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 81 | 是我们买的第二个了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 82 | 需要个人安装的也不多,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 83 | 没有任何破损的地方,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 84 | 自营售后服务品质有保障,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 85 | 南方冬天暖气是刚需啊,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 86 | 用了两次我和孩子都喉咙发炎了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 87 | 十几平方的房间开一夜都没觉得有效果,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 88 | 概念&rdquo,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 89 | 外观设计：很不错取暖效果：很好耗电情况：蛮高的静音效果：很不错安全设计：安全其他特色：没有了,0,0,1,0,0,0,0,0,0,0,0,0,1,0,0 90 | 我觉得效果比浴霸好,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 91 | 可根据需要选择功率,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 92 | 身边的朋友都有介绍此款,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 93 | 安全设计：设计独特完美,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 94 | 总体来说物美价廉,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 95 | 放书桌旁干活用很合适,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 96 | 如果属于南方的没暖气喜欢的宝宝可以放心购买,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 97 | 还赠送了晾衣架,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 98 | 贴墙放置,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 99 | 非常的美观还耐用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 100 | 买了你就后悔,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 101 | 说24小时回电话,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 102 | 家里离不开了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 103 | 估计到冬天没太阳的时候烘衣服方便,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 104 | 十天用了一百六的电费,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 105 | 有遥控装置,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 106 | 体积虽小,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 107 | 不把手凑上去都感觉不到他的温度,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 108 | 用了两天的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 109 | 小房间很快就能加热,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 110 | 当天下单第二天就到,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 111 | 加热很快??,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 112 | 静音效果：二档声音不小,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 113 | 也没有长螺丝刀,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 114 | 平均每天就开两小时,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 115 | 但声音还能接受,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 116 | 静音效果：试用了几天,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 117 | 取暖效果：开到二挡很暖和,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 118 | 今年给老人又买一个,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 119 | 看着小,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 120 | 外形取暖效果：超级棒耗电情况：没在意静音效果：安静舒适安全设计：童锁设计省心,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 121 | 不知道能不能优化,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 122 | 货到手没有一点损坏,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 123 | 今天就到货了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 124 | 希望要买好好选别家的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 125 | 利用热空气上升的自然流动,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 126 | 长的差不多也是200左右,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 127 | 现在穿着睡衣,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 128 | 产品的质量不错,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 129 | 希望能耐用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 130 | 再也不用担心受凉感冒了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 131 | 一晚耗电30不是梦,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 132 | 还会有电火花闪现,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 133 | 有一定的防水,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 134 | 双十二特价很便宜,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 135 | 但是一个小时两度电也是真的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 136 | 在广州用很合适,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 137 | 大人小朋友不容易烫伤,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 138 | 选择小米唯一吸引我的是支持小爱和手机,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 139 | 着实让我觉得有点生气,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 140 | 主要是家里没有安装暖气片,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 141 | 好多人都和我一样的情况,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 142 | 有风扇设计所致,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 143 | 外观设计：很好看取暖效果：取暖很快效果很好,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 144 | 强卖,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 145 | 还危险了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 146 | 必须要贴着机器才有点温度,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 147 | 耐用也就有安全保障其他特色：冬天衣服难干,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 148 | 时不时的会咔的一声响,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 149 | 就算开低档温度也不错,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 150 | 挺适合南方的冬天,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 151 | 这东西买的真好,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 152 | -------------------------------------------------------------------------------- /classifier_multi_label/data/vocabulary_label.txt: -------------------------------------------------------------------------------- 1 | | 2 | 互联互通 3 | 产品功耗 4 | 滑轮提手 5 | 声音 6 | APP操控性 7 | 呼吸灯 8 | 外观 9 | 底座 10 | 制热范围 11 | 遥控器电池 12 | 味道 13 | 制热效果 14 | 衣物烘干 15 | 体积大小 16 | -------------------------------------------------------------------------------- /classifier_multi_label/eval.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu Apr 9 10:56:43 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | 10 | # 参考： https://github.com/hellonlp/classifier-multi-label/blob/master/evaluation/eval_multi_label.py 11 | -------------------------------------------------------------------------------- /classifier_multi_label/hyperparameters.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Mon Nov 12 14:23:12 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import os 10 | pwd = os.path.dirname(os.path.abspath(__file__)) 11 | from classifier_multi_label.utils import load_vocabulary 12 | 13 | 14 | class Hyperparamters: 15 | # Train parameters 16 | num_train_epochs = 40 17 | print_step = 10 18 | batch_size = 32 19 | summary_step = 10 20 | num_saved_per_epoch = 3 21 | max_to_keep = 100 22 | logdir = 'logdir/model_01' 23 | 24 | # Model file 25 | file_model_load = 'model/model_load' 26 | file_model_save = 'model/model_save' 27 | 28 | # Train/Test data 29 | data_dir = os.path.join(pwd,'data') 30 | train_data = 'train_onehot.csv' 31 | test_data = 'test_onehot.csv' 32 | 33 | # Load vocabulcary dict 34 | dict_id2label,dict_label2id = load_vocabulary(os.path.join(pwd,'data','vocabulary_label.txt')) 35 | label_vocabulary = list(dict_id2label.values()) 36 | 37 | # Optimization parameters 38 | warmup_proportion = 0.1 39 | use_tpu = None 40 | do_lower_case = True 41 | learning_rate = 5e-5 42 | 43 | # TextCNN parameters 44 | num_filters = 128 45 | filter_sizes = [2,3,4,5,6,7] 46 | embedding_size = 384 47 | keep_prob = 0.5 48 | 49 | # Sequence and Label 50 | sequence_length = 60 51 | num_labels = len(list(dict_id2label)) 52 | 53 | # ALBERT 54 | model = 'albert_small_zh_google' 55 | bert_path = os.path.join(pwd,model) 56 | vocab_file = os.path.join(pwd,model,'vocab_chinese.txt') 57 | init_checkpoint = os.path.join(pwd,model,'albert_model.ckpt') 58 | saved_model_path = os.path.join(pwd,'model') 59 | 60 | 61 | 62 | 63 | 64 | 65 | if __name__ == '__main__': 66 | hp = Hyperparamters() 67 | 68 | 69 | 70 | 71 | 72 | -------------------------------------------------------------------------------- /classifier_multi_label/image/graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label/image/graph.png -------------------------------------------------------------------------------- /classifier_multi_label/lamb_optimizer.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | # Lint as: python2, python3 16 | """Functions and classes related to optimization (weight updates).""" 17 | 18 | from __future__ import absolute_import 19 | from __future__ import division 20 | from __future__ import print_function 21 | 22 | import re 23 | import six 24 | import tensorflow.compat.v1 as tf 25 | 26 | # pylint: disable=g-direct-tensorflow-import 27 | from tensorflow.python.ops import array_ops 28 | from tensorflow.python.ops import linalg_ops 29 | from tensorflow.python.ops import math_ops 30 | # pylint: enable=g-direct-tensorflow-import 31 | 32 | 33 | class LAMBOptimizer(tf.train.Optimizer): 34 | """LAMB (Layer-wise Adaptive Moments optimizer for Batch training).""" 35 | # A new optimizer that includes correct L2 weight decay, adaptive 36 | # element-wise updating, and layer-wise justification. The LAMB optimizer 37 | # was proposed by Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, 38 | # James Demmel, and Cho-Jui Hsieh in a paper titled as Reducing BERT 39 | # Pre-Training Time from 3 Days to 76 Minutes (arxiv.org/abs/1904.00962) 40 | 41 | def __init__(self, 42 | learning_rate, 43 | weight_decay_rate=0.0, 44 | beta_1=0.9, 45 | beta_2=0.999, 46 | epsilon=1e-6, 47 | exclude_from_weight_decay=None, 48 | exclude_from_layer_adaptation=None, 49 | name="LAMBOptimizer"): 50 | """Constructs a LAMBOptimizer.""" 51 | super(LAMBOptimizer, self).__init__(False, name) 52 | 53 | self.learning_rate = learning_rate 54 | self.weight_decay_rate = weight_decay_rate 55 | self.beta_1 = beta_1 56 | self.beta_2 = beta_2 57 | self.epsilon = epsilon 58 | self.exclude_from_weight_decay = exclude_from_weight_decay 59 | # exclude_from_layer_adaptation is set to exclude_from_weight_decay if the 60 | # arg is None. 61 | # TODO(jingli): validate if exclude_from_layer_adaptation is necessary. 62 | if exclude_from_layer_adaptation: 63 | self.exclude_from_layer_adaptation = exclude_from_layer_adaptation 64 | else: 65 | self.exclude_from_layer_adaptation = exclude_from_weight_decay 66 | 67 | def apply_gradients(self, grads_and_vars, global_step=None, name=None): 68 | """See base class.""" 69 | assignments = [] 70 | for (grad, param) in grads_and_vars: 71 | if grad is None or param is None: 72 | continue 73 | 74 | param_name = self._get_variable_name(param.name) 75 | 76 | m = tf.get_variable( 77 | name=six.ensure_str(param_name) + "/adam_m", 78 | shape=param.shape.as_list(), 79 | dtype=tf.float32, 80 | trainable=False, 81 | initializer=tf.zeros_initializer()) 82 | v = tf.get_variable( 83 | name=six.ensure_str(param_name) + "/adam_v", 84 | shape=param.shape.as_list(), 85 | dtype=tf.float32, 86 | trainable=False, 87 | initializer=tf.zeros_initializer()) 88 | 89 | # Standard Adam update. 90 | next_m = ( 91 | tf.multiply(self.beta_1, m) + tf.multiply(1.0 - self.beta_1, grad)) 92 | next_v = ( 93 | tf.multiply(self.beta_2, v) + tf.multiply(1.0 - self.beta_2, 94 | tf.square(grad))) 95 | 96 | update = next_m / (tf.sqrt(next_v) + self.epsilon) 97 | 98 | # Just adding the square of the weights to the loss function is *not* 99 | # the correct way of using L2 regularization/weight decay with Adam, 100 | # since that will interact with the m and v parameters in strange ways. 101 | # 102 | # Instead we want ot decay the weights in a manner that doesn't interact 103 | # with the m/v parameters. This is equivalent to adding the square 104 | # of the weights to the loss with plain (non-momentum) SGD. 105 | if self._do_use_weight_decay(param_name): 106 | update += self.weight_decay_rate * param 107 | 108 | ratio = 1.0 109 | if self._do_layer_adaptation(param_name): 110 | w_norm = linalg_ops.norm(param, ord=2) 111 | g_norm = linalg_ops.norm(update, ord=2) 112 | ratio = array_ops.where(math_ops.greater(w_norm, 0), array_ops.where( 113 | math_ops.greater(g_norm, 0), (w_norm / g_norm), 1.0), 1.0) 114 | 115 | update_with_lr = ratio * self.learning_rate * update 116 | 117 | next_param = param - update_with_lr 118 | 119 | assignments.extend( 120 | [param.assign(next_param), 121 | m.assign(next_m), 122 | v.assign(next_v)]) 123 | return tf.group(*assignments, name=name) 124 | 125 | def _do_use_weight_decay(self, param_name): 126 | """Whether to use L2 weight decay for `param_name`.""" 127 | if not self.weight_decay_rate: 128 | return False 129 | if self.exclude_from_weight_decay: 130 | for r in self.exclude_from_weight_decay: 131 | if re.search(r, param_name) is not None: 132 | return False 133 | return True 134 | 135 | def _do_layer_adaptation(self, param_name): 136 | """Whether to do layer-wise learning rate adaptation for `param_name`.""" 137 | if self.exclude_from_layer_adaptation: 138 | for r in self.exclude_from_layer_adaptation: 139 | if re.search(r, param_name) is not None: 140 | return False 141 | return True 142 | 143 | def _get_variable_name(self, param_name): 144 | """Get the variable name from the tensor name.""" 145 | m = re.match("^(.*):\\d+$", six.ensure_str(param_name)) 146 | if m is not None: 147 | param_name = m.group(1) 148 | return param_name 149 | -------------------------------------------------------------------------------- /classifier_multi_label/logdir/model_01/events.out.tfevents.1595940723.DESKTOP-QC1A83I: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label/logdir/model_01/events.out.tfevents.1595940723.DESKTOP-QC1A83I -------------------------------------------------------------------------------- /classifier_multi_label/model/model_load/README.md: -------------------------------------------------------------------------------- 1 | # 训练过程保存的模型 2 | 3 | ## 模型路径 4 | - model_save/checkpoit 5 | - model_save/model_xx_0.ckpt.data-00000-of-00001 6 | - model_save/model_xx_0.ckpt.index 7 | - model_save/model_xx_0.ckpt.meta 8 | 9 | 10 | ## checkpoint内容 11 | ``` 12 | model_checkpoint_path: "model_xx_0.ckpt" 13 | ``` 14 | -------------------------------------------------------------------------------- /classifier_multi_label/model/model_save/READEME.md: -------------------------------------------------------------------------------- 1 | # 推理所用模型 2 | -------------------------------------------------------------------------------- /classifier_multi_label/modules.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu May 30 21:01:45 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import tensorflow as tf 10 | 11 | from classifier_multi_label.hyperparameters import Hyperparamters as hp 12 | 13 | 14 | def cell_textcnn(inputs,is_training): 15 | # Add a dimension in final shape 16 | inputs_expand = tf.expand_dims(inputs, -1) 17 | # Create a convolution + maxpool layer for each filter size 18 | pooled_outputs = [] 19 | with tf.name_scope("TextCNN"): 20 | for i, filter_size in enumerate(hp.filter_sizes): 21 | with tf.name_scope("conv-maxpool-%s" % filter_size): 22 | # Convolution Layer 23 | filter_shape = [filter_size, hp.embedding_size, 1, hp.num_filters] 24 | W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1),dtype=tf.float32, name="W") 25 | b = tf.Variable(tf.constant(0.1, shape=[hp.num_filters]),dtype=tf.float32, name="b") 26 | conv = tf.nn.conv2d( 27 | inputs_expand, 28 | W, 29 | strides=[1, 1, 1, 1], 30 | padding="VALID", 31 | name="conv") 32 | # Apply nonlinearity 33 | h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu") 34 | # Maxpooling over the outputs 35 | pooled = tf.nn.max_pool( 36 | h, 37 | ksize=[1, hp.sequence_length - filter_size + 1, 1, 1], 38 | strides=[1, 1, 1, 1], 39 | padding='VALID', 40 | name="pool") 41 | pooled_outputs.append(pooled) 42 | # Combine all the pooled features 43 | num_filters_total = hp.num_filters * len(hp.filter_sizes) 44 | h_pool = tf.concat(pooled_outputs, 3) 45 | h_pool_flat = tf.reshape(h_pool, [-1, num_filters_total]) 46 | # Dropout 47 | h_pool_flat_dropout = tf.nn.dropout(h_pool_flat, keep_prob=hp.keep_prob if is_training else 1) 48 | return h_pool_flat_dropout 49 | 50 | 51 | -------------------------------------------------------------------------------- /classifier_multi_label/networks.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu May 30 20:44:42 2019 4 | 5 | @author: cm 6 | """ 7 | 8 | import os 9 | import tensorflow as tf 10 | from classifier_multi_label import modeling 11 | from classifier_multi_label import optimization 12 | from classifier_multi_label.utils import time_now_string 13 | from classifier_multi_label.hyperparameters import Hyperparamters as hp 14 | from classifier_multi_label.classifier_utils import ClassifyProcessor 15 | 16 | 17 | num_labels = hp.num_labels 18 | processor = ClassifyProcessor() 19 | bert_config_file = os.path.join(hp.bert_path,'albert_config.json') 20 | bert_config = modeling.AlbertConfig.from_json_file(bert_config_file) 21 | 22 | 23 | 24 | class NetworkAlbert(object): 25 | def __init__(self,is_training): 26 | # Training or not 27 | self.is_training = is_training 28 | 29 | # Placeholder 30 | self.input_ids = tf.placeholder(tf.int32, shape=[None, hp.sequence_length], name='input_ids') 31 | self.input_masks = tf.placeholder(tf.int32, shape=[None, hp.sequence_length], name='input_masks') 32 | self.segment_ids = tf.placeholder(tf.int32, shape=[None, hp.sequence_length], name='segment_ids') 33 | self.label_ids = tf.placeholder(tf.float32, shape=[None,hp.num_labels], name='label_ids') 34 | 35 | # Load BERT model 36 | self.model = modeling.AlbertModel( 37 | config=bert_config, 38 | is_training=self.is_training, 39 | input_ids=self.input_ids, 40 | input_mask=self.input_masks, 41 | token_type_ids=self.segment_ids, 42 | use_one_hot_embeddings=False) 43 | 44 | # Get the feature vector by BERT 45 | output_layer = self.model.get_pooled_output() 46 | 47 | # Hidden size 48 | hidden_size = output_layer.shape[-1].value 49 | 50 | with tf.name_scope("Full-connection"): 51 | output_weights = tf.get_variable( 52 | "output_weights", [num_labels, hidden_size], 53 | initializer=tf.truncated_normal_initializer(stddev=0.02)) 54 | output_bias = tf.get_variable( 55 | "output_bias", [num_labels], initializer=tf.zeros_initializer()) 56 | logits = tf.nn.bias_add(tf.matmul(output_layer, output_weights, transpose_b=True), output_bias) 57 | # Prediction sigmoid(Multi-label) 58 | self.probabilities = tf.nn.sigmoid(logits) 59 | 60 | 61 | with tf.variable_scope("Prediction"): 62 | # Prediction 63 | zero = tf.zeros_like(self.probabilities) 64 | one = tf.ones_like(self.probabilities) 65 | self.predictions = tf.where(self.probabilities < 0.5, x=zero, y=one) 66 | 67 | with tf.variable_scope("loss"): 68 | # Summary for tensorboard 69 | if self.is_training: 70 | self.accuracy = tf.reduce_mean(tf.to_float(tf.equal(self.predictions, self.label_ids))) 71 | tf.summary.scalar('accuracy', self.accuracy) 72 | 73 | # Initial embedding by BERT 74 | ckpt = tf.train.get_checkpoint_state(hp.saved_model_path) 75 | checkpoint_suffix = ".index" 76 | if ckpt and tf.gfile.Exists(ckpt.model_checkpoint_path + checkpoint_suffix): 77 | print('='*10,'Restoring model from checkpoint!','='*10) 78 | print("%s - Restoring model from checkpoint ~%s" % (time_now_string(), 79 | ckpt.model_checkpoint_path)) 80 | else: 81 | print('='*10,'First time load BERT model!','='*10) 82 | tvars = tf.trainable_variables() 83 | if hp.init_checkpoint: 84 | (assignment_map, initialized_variable_names) = \ 85 | modeling.get_assignment_map_from_checkpoint(tvars, 86 | hp.init_checkpoint) 87 | tf.train.init_from_checkpoint(hp.init_checkpoint, assignment_map) 88 | 89 | # Loss and Optimizer 90 | if self.is_training: 91 | # Global_step 92 | self.global_step = tf.Variable(0, name='global_step', trainable=False) 93 | per_example_loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=self.label_ids,logits=logits) 94 | self.loss = tf.reduce_mean(per_example_loss) 95 | 96 | # Optimizer BERT 97 | train_examples = processor.get_train_examples(hp.data_dir) 98 | num_train_steps = int( 99 | len(train_examples) / hp.batch_size * hp.num_train_epochs) 100 | #num_train_steps = 10000 101 | num_warmup_steps = int(num_train_steps * hp.warmup_proportion) 102 | print('num_train_steps',num_train_steps) 103 | self.optimizer = optimization.create_optimizer(self.loss, 104 | hp.learning_rate, 105 | num_train_steps, 106 | num_warmup_steps, 107 | hp.use_tpu, 108 | Global_step=self.global_step) 109 | 110 | # Summary for tensorboard 111 | tf.summary.scalar('loss', self.loss) 112 | self.merged = tf.summary.merge_all() 113 | 114 | 115 | 116 | if __name__ == '__main__': 117 | # Load model 118 | albert = NetworkAlbert(is_training=True) 119 | 120 | 121 | 122 | -------------------------------------------------------------------------------- /classifier_multi_label/optimization.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | """Functions and classes related to optimization (weight updates).""" 3 | 4 | from __future__ import absolute_import 5 | from __future__ import division 6 | from __future__ import print_function 7 | import re 8 | import six 9 | from six.moves import zip 10 | import tensorflow.compat.v1 as tf 11 | from tensorflow.contrib import tpu as contrib_tpu 12 | from classifier_multi_label import lamb_optimizer 13 | 14 | 15 | def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, use_tpu,Global_step, 16 | optimizer="adamw", poly_power=1.0, start_warmup_step=0): 17 | """Creates an optimizer training op.""" 18 | #global_step = tf.train.get_or_create_global_step() 19 | 20 | # by chenming 21 | if Global_step: 22 | global_step = Global_step 23 | else: 24 | global_step = tf.train.get_or_create_global_step() 25 | 26 | learning_rate = tf.constant(value=init_lr, shape=[], dtype=tf.float32) 27 | 28 | # Implements linear decay of the learning rate. 29 | learning_rate = tf.train.polynomial_decay( 30 | learning_rate, 31 | global_step, 32 | num_train_steps, 33 | end_learning_rate=0.0, 34 | power=poly_power, 35 | cycle=False) 36 | 37 | # Implements linear warmup. I.e., if global_step - start_warmup_step < 38 | # num_warmup_steps, the learning rate will be 39 | # `(global_step - start_warmup_step)/num_warmup_steps * init_lr`. 40 | if num_warmup_steps: 41 | tf.logging.info("++++++ warmup starts at step " + str(start_warmup_step) 42 | + ", for " + str(num_warmup_steps) + " steps ++++++") 43 | global_steps_int = tf.cast(global_step, tf.int32) 44 | start_warm_int = tf.constant(start_warmup_step, dtype=tf.int32) 45 | global_steps_int = global_steps_int - start_warm_int 46 | warmup_steps_int = tf.constant(num_warmup_steps, dtype=tf.int32) 47 | 48 | global_steps_float = tf.cast(global_steps_int, tf.float32) 49 | warmup_steps_float = tf.cast(warmup_steps_int, tf.float32) 50 | 51 | warmup_percent_done = global_steps_float / warmup_steps_float 52 | warmup_learning_rate = init_lr * warmup_percent_done 53 | 54 | is_warmup = tf.cast(global_steps_int < warmup_steps_int, tf.float32) 55 | learning_rate = ( 56 | (1.0 - is_warmup) * learning_rate + is_warmup * warmup_learning_rate) 57 | 58 | # It is OK that you use this optimizer for finetuning, since this 59 | # is how the model was trained (note that the Adam m/v variables are NOT 60 | # loaded from init_checkpoint.) 61 | # It is OK to use AdamW in the finetuning even the model is trained by LAMB. 62 | # As report in the Bert pulic github, the learning rate for SQuAD 1.1 finetune 63 | # is 3e-5, 4e-5 or 5e-5. For LAMB, the users can use 3e-4, 4e-4,or 5e-4 for a 64 | # batch size of 64 in the finetune. 65 | if optimizer == "adamw": 66 | tf.logging.info("using adamw") 67 | optimizer = AdamWeightDecayOptimizer( 68 | learning_rate=learning_rate, 69 | weight_decay_rate=0.01, 70 | beta_1=0.9, 71 | beta_2=0.999, 72 | epsilon=1e-6, 73 | exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"]) 74 | elif optimizer == "lamb": 75 | tf.logging.info("using lamb") 76 | optimizer = lamb_optimizer.LAMBOptimizer( 77 | learning_rate=learning_rate, 78 | weight_decay_rate=0.01, 79 | beta_1=0.9, 80 | beta_2=0.999, 81 | epsilon=1e-6, 82 | exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"]) 83 | else: 84 | raise ValueError("Not supported optimizer: ", optimizer) 85 | 86 | if use_tpu: 87 | optimizer = contrib_tpu.CrossShardOptimizer(optimizer) 88 | 89 | tvars = tf.trainable_variables() 90 | grads = tf.gradients(loss, tvars) 91 | 92 | # This is how the model was pre-trained. 93 | (grads, _) = tf.clip_by_global_norm(grads, clip_norm=1.0) 94 | 95 | train_op = optimizer.apply_gradients( 96 | list(zip(grads, tvars)), global_step=global_step) 97 | 98 | # Normally the global step update is done inside of `apply_gradients`. 99 | # However, neither `AdamWeightDecayOptimizer` nor `LAMBOptimizer` do this. 100 | # But if you use a different optimizer, you should probably take this line 101 | # out. 102 | new_global_step = global_step + 1 103 | train_op = tf.group(train_op, [global_step.assign(new_global_step)]) 104 | return train_op 105 | 106 | 107 | class AdamWeightDecayOptimizer(tf.train.Optimizer): 108 | """A basic Adam optimizer that includes "correct" L2 weight decay.""" 109 | 110 | def __init__(self, 111 | learning_rate, 112 | weight_decay_rate=0.0, 113 | beta_1=0.9, 114 | beta_2=0.999, 115 | epsilon=1e-6, 116 | exclude_from_weight_decay=None, 117 | name="AdamWeightDecayOptimizer"): 118 | """Constructs a AdamWeightDecayOptimizer.""" 119 | super(AdamWeightDecayOptimizer, self).__init__(False, name) 120 | 121 | self.learning_rate = learning_rate 122 | self.weight_decay_rate = weight_decay_rate 123 | self.beta_1 = beta_1 124 | self.beta_2 = beta_2 125 | self.epsilon = epsilon 126 | self.exclude_from_weight_decay = exclude_from_weight_decay 127 | 128 | def apply_gradients(self, grads_and_vars, global_step=None, name=None): 129 | """See base class.""" 130 | assignments = [] 131 | for (grad, param) in grads_and_vars: 132 | if grad is None or param is None: 133 | continue 134 | 135 | param_name = self._get_variable_name(param.name) 136 | 137 | m = tf.get_variable( 138 | name=six.ensure_str(param_name) + "/adam_m", 139 | shape=param.shape.as_list(), 140 | dtype=tf.float32, 141 | trainable=False, 142 | initializer=tf.zeros_initializer()) 143 | v = tf.get_variable( 144 | name=six.ensure_str(param_name) + "/adam_v", 145 | shape=param.shape.as_list(), 146 | dtype=tf.float32, 147 | trainable=False, 148 | initializer=tf.zeros_initializer()) 149 | 150 | # Standard Adam update. 151 | next_m = ( 152 | tf.multiply(self.beta_1, m) + tf.multiply(1.0 - self.beta_1, grad)) 153 | next_v = ( 154 | tf.multiply(self.beta_2, v) + tf.multiply(1.0 - self.beta_2, 155 | tf.square(grad))) 156 | 157 | update = next_m / (tf.sqrt(next_v) + self.epsilon) 158 | 159 | # Just adding the square of the weights to the loss function is *not* 160 | # the correct way of using L2 regularization/weight decay with Adam, 161 | # since that will interact with the m and v parameters in strange ways. 162 | # 163 | # Instead we want ot decay the weights in a manner that doesn't interact 164 | # with the m/v parameters. This is equivalent to adding the square 165 | # of the weights to the loss with plain (non-momentum) SGD. 166 | if self._do_use_weight_decay(param_name): 167 | update += self.weight_decay_rate * param 168 | 169 | update_with_lr = self.learning_rate * update 170 | 171 | next_param = param - update_with_lr 172 | 173 | assignments.extend( 174 | [param.assign(next_param), 175 | m.assign(next_m), 176 | v.assign(next_v)]) 177 | return tf.group(*assignments, name=name) 178 | 179 | def _do_use_weight_decay(self, param_name): 180 | """Whether to use L2 weight decay for `param_name`.""" 181 | if not self.weight_decay_rate: 182 | return False 183 | if self.exclude_from_weight_decay: 184 | for r in self.exclude_from_weight_decay: 185 | if re.search(r, param_name) is not None: 186 | return False 187 | return True 188 | 189 | def _get_variable_name(self, param_name): 190 | """Get the variable name from the tensor name.""" 191 | m = re.match("^(.*):\\d+$", six.ensure_str(param_name)) 192 | if m is not None: 193 | param_name = m.group(1) 194 | return param_name 195 | -------------------------------------------------------------------------------- /classifier_multi_label/predict.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu May 30 17:12:37 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import os 10 | #os.environ["CUDA_VISIBLE_DEVICES"] = '-1' 11 | pwd = os.path.dirname(os.path.abspath(__file__)) 12 | import sys 13 | sys.path.append(os.path.dirname(os.path.dirname(__file__))) 14 | import numpy as np 15 | import tensorflow as tf 16 | from classifier_multi_label.networks import NetworkAlbert 17 | from classifier_multi_label.classifier_utils import get_feature_test,id2label 18 | from classifier_multi_label.hyperparameters import Hyperparamters as hp 19 | 20 | 21 | class Model(object,): 22 | """ 23 | Load NetworkAlbert model 24 | """ 25 | def __init__(self): 26 | self.albert, self.sess = self.load_model() 27 | @staticmethod 28 | def load_model(): 29 | with tf.Graph().as_default(): 30 | sess = tf.Session() 31 | with sess.as_default(): 32 | albert = NetworkAlbert(is_training=False) 33 | saver = tf.train.Saver() 34 | sess.run(tf.global_variables_initializer()) 35 | checkpoint_dir = os.path.abspath(os.path.join(pwd,hp.file_model_load)) 36 | print (checkpoint_dir) 37 | ckpt = tf.train.get_checkpoint_state(checkpoint_dir) 38 | saver.restore(sess, ckpt.model_checkpoint_path) 39 | return albert,sess 40 | 41 | MODEL = Model() 42 | print('Load model finished!') 43 | 44 | 45 | def get_label(sentence): 46 | """ 47 | Prediction of the sentence's label. 48 | """ 49 | feature = get_feature_test(sentence) 50 | fd = {MODEL.albert.input_ids: [feature[0]], 51 | MODEL.albert.input_masks: [feature[1]], 52 | MODEL.albert.segment_ids:[feature[2]], 53 | } 54 | prediction = MODEL.sess.run(MODEL.albert.predictions, feed_dict=fd)[0] 55 | return [id2label(l) for l in np.where(prediction==1)[0] if l!=0] 56 | 57 | 58 | if __name__ == '__main__': 59 | # Test 60 | sentences = ['耗电情况：整体来说耗电不是特别严重', 61 | '取暖效果：取暖效果好', 62 | '取暖效果：开到二挡很暖和', 63 | '一个小时房间仍然没暖和', 64 | '开着坐旁边才能暖和'] 65 | for sentence in sentences: 66 | print(sentence,get_label(sentence)) 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | -------------------------------------------------------------------------------- /classifier_multi_label/train.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu May 30 21:42:07 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | import os 9 | #os.environ["CUDA_VISIBLE_DEVICES"] = '-1' 10 | import numpy as np 11 | import tensorflow as tf 12 | from classifier_multi_label.networks import NetworkAlbert 13 | from classifier_multi_label.classifier_utils import get_features 14 | from classifier_multi_label.hyperparameters import Hyperparamters as hp 15 | from classifier_multi_label.utils import select,shuffle_one,time_now_string 16 | 17 | 18 | pwd = os.path.dirname(os.path.abspath(__file__)) 19 | MODEL = NetworkAlbert(is_training=True) 20 | 21 | 22 | # Get data features 23 | input_ids,input_masks,segment_ids,label_ids = get_features() 24 | num_train_samples = len(input_ids) 25 | indexs = np.arange(num_train_samples) 26 | num_batchs = int((num_train_samples - 1) /hp.batch_size) + 1 27 | print('Number of batch:',num_batchs) 28 | 29 | # Set up the graph 30 | saver = tf.train.Saver(max_to_keep=hp.max_to_keep) 31 | sess = tf.Session() 32 | sess.run(tf.global_variables_initializer()) 33 | 34 | # Load model saved before 35 | MODEL_SAVE_PATH = os.path.join(pwd, hp.file_model_save) 36 | ckpt = tf.train.get_checkpoint_state(MODEL_SAVE_PATH) 37 | if ckpt and ckpt.model_checkpoint_path: 38 | saver.restore(sess, ckpt.model_checkpoint_path) 39 | print('Restored model!') 40 | 41 | 42 | with sess.as_default(): 43 | # Tensorboard writer 44 | writer = tf.summary.FileWriter(hp.logdir, sess.graph) 45 | for i in range(hp.num_train_epochs): 46 | np.random.shuffle(indexs) 47 | for j in range(num_batchs-1): 48 | # Get ids selected 49 | i1 = indexs[j * hp.batch_size:min((j + 1) * hp.batch_size, num_train_samples)] 50 | 51 | # Get features 52 | input_id_ = select(input_ids,i1) 53 | input_mask_ = select(input_masks,i1) 54 | segment_id_ = select(segment_ids,i1) 55 | label_id_ = select(label_ids,i1) 56 | 57 | # Feed dict 58 | fd = {MODEL.input_ids: input_id_, 59 | MODEL.input_masks: input_mask_, 60 | MODEL.segment_ids:segment_id_, 61 | MODEL.label_ids:label_id_} 62 | 63 | # Optimizer 64 | sess.run(MODEL.optimizer, feed_dict = fd) 65 | 66 | # Tensorboard 67 | if j%hp.summary_step==0: 68 | summary,glolal_step = sess.run([MODEL.merged,MODEL.global_step], feed_dict = fd) 69 | writer.add_summary(summary, glolal_step) 70 | 71 | # Save Model 72 | if j%(num_batchs//hp.num_saved_per_epoch)==0: 73 | if not os.path.exists(os.path.join(pwd, hp.file_model_save)): 74 | os.makedirs(os.path.join(pwd, hp.file_model_save)) 75 | saver.save(sess, os.path.join(pwd, hp.file_model_save, 'model'+'_%s_%s.ckpt'%(str(i),str(j)))) 76 | 77 | # Log 78 | if j % hp.print_step == 0: 79 | fd = {MODEL.input_ids: input_id_, 80 | MODEL.input_masks: input_mask_, 81 | MODEL.segment_ids:segment_id_, 82 | MODEL.label_ids:label_id_} 83 | loss = sess.run(MODEL.loss, feed_dict = fd) 84 | print('Time:%s, Epoch:%s, Batch number:%s/%s, Loss:%s'%(time_now_string(),str(i),str(j),str(num_batchs),str(loss))) 85 | print('Train finished') 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | -------------------------------------------------------------------------------- /classifier_multi_label/utils.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Fri May 25 23:43:39 2021 4 | @author: cm 5 | """ 6 | 7 | 8 | import time 9 | import numpy as np 10 | import pandas as pd 11 | 12 | 13 | 14 | def time_now_string(): 15 | return time.strftime("%Y-%m-%d %H:%M:%S",time.localtime( time.time() )) 16 | 17 | 18 | def select(data,ids): 19 | return [data[i] for i in ids] 20 | 21 | 22 | def shuffle_one(a1): 23 | ran = np.arange(len(a1)) 24 | np.random.shuffle(ran) 25 | return np.array(a1)[ran].tolist() 26 | 27 | 28 | def shuffle_two(a1,a2): 29 | ran = np.arange(len(a1)) 30 | np.random.shuffle(ran) 31 | return [a1[l] for l in ran], [a2[l] for l in ran] 32 | 33 | 34 | def load_txt(file): 35 | with open(file,encoding='utf-8',errors='ignore') as fp: 36 | lines = fp.readlines() 37 | lines = [l.strip() for l in lines] 38 | print("Load data from file (%s) finished !"%file) 39 | return lines 40 | 41 | 42 | def save_txt(file,lines): 43 | lines = [l+'\n' for l in lines] 44 | with open(file,'w+',encoding='utf-8') as fp:#a+添加 45 | fp.writelines(lines) 46 | return "Write data to txt finished !" 47 | 48 | 49 | def load_csv(file,encoding='utf-8',header=0): 50 | return pd.read_csv(file,encoding=encoding,header=header,error_bad_lines=False)#,encoding='gbk' 51 | 52 | 53 | def save_csv(dataframe,file,header=True,index=None,encoding="utf-8-sig"): 54 | return dataframe.to_csv(file, 55 | mode='w+', 56 | header=header, 57 | index=index, 58 | encoding=encoding) 59 | 60 | 61 | def save_excel(dataframe,file,header=True,sheetname='Sheet1'): 62 | return dataframe.to_excel(file, 63 | header=header, 64 | sheet_name=sheetname) 65 | 66 | 67 | def load_excel(file,header=0): 68 | return pd.read_excel(file, 69 | header=header, 70 | ) 71 | 72 | 73 | def cut_list(data,size): 74 | """ 75 | data: a list 76 | size: the size of cut 77 | """ 78 | return [data[i * size:min((i + 1) * size, len(data))] for i in range(int(len(data)-1)//size + 1)] 79 | 80 | 81 | 82 | def load_vocabulary(file_vocabulary_label): 83 | """ 84 | Load vocabulary to dict 85 | """ 86 | vocabulary = load_txt(file_vocabulary_label) 87 | dict_id2label,dict_label2id = {},{} 88 | for i,l in enumerate(vocabulary): 89 | dict_id2label[str(i)] = str(l) 90 | dict_label2id[str(l)] = str(i) 91 | return dict_id2label,dict_label2id 92 | 93 | 94 | 95 | if __name__ == '__main__': 96 | print('') 97 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/README.md: -------------------------------------------------------------------------------- 1 | # 简介 2 | 1、本项目是在tensorflow版本1.15.0的基础上做的训练和测试。 3 | 2、本项目为中文的多标签文本分类。 4 | 3、欢迎大家联系我 www.hellonlp.com 5 | 4、albert_small_zh_google对应的百度云下载地址： 6 | 链接：https://pan.baidu.com/s/1RKzGJTazlZ7y12YRbAWvyA 7 | 提取码：wuxw 8 | 9 | # 使用方法 10 | 1、准备数据 11 | 数据格式为：classifier_multi_label_denses/data/test_onehot.csv 12 | 2、参数设置 13 | 参考脚本 hyperparameters.py，直接修改里面的数值即可。 14 | 3、训练 15 | ``` 16 | python train.py 17 | ``` 18 | 4、预测 19 | ``` 20 | python predict.py 21 | ``` 22 | 注意：推理时需要把model/save中的模型复制到model/load中，并修改model/load中的checkpoint文件的内容为当前模型名称，例如：model_checkpoint_path: "model_xx_xx.ckpt"。 23 | 。 24 | 25 | 26 | # 知乎解读 27 | https://zhuanlan.zhihu.com/p/263573628 28 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/__init__.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | # Copyright 2018 The Google AI Team Authors. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/albert_small_zh_google/albert_config.json: -------------------------------------------------------------------------------- 1 | { 2 | "attention_probs_dropout_prob": 0.0, 3 | "hidden_act": "gelu", 4 | "hidden_dropout_prob": 0.0, 5 | "embedding_size": 128, 6 | "hidden_size": 384, 7 | "initializer_range": 0.02, 8 | "intermediate_size": 1536, 9 | "max_position_embeddings": 512, 10 | "num_attention_heads": 12, 11 | "num_hidden_layers": 6, 12 | "num_hidden_groups": 1, 13 | "net_structure_type": 0, 14 | "gap_size": 0, 15 | "num_memory_blocks": 0, 16 | "inner_group_num": 1, 17 | "down_scale_factor": 1, 18 | "type_vocab_size": 2, 19 | "vocab_size": 21128 20 | } -------------------------------------------------------------------------------- /classifier_multi_label_denses/data/test.csv: -------------------------------------------------------------------------------- 1 | content,label 2 | 也没有烧伤,| 3 | 主要是之前在北方呆过几几年,| 4 | 温暖过冬,| 5 | 开箱试了一下看着很不错,| 6 | 还可以给宝宝凉衣服,| 7 | 快递师傅送货迅速,| 8 | 快递员差评,| 9 | 沒有什么噪音,| 10 | 就感觉有一部分是强行捏到一起去的,| 11 | 今天刚到货还没有使用,| 12 | 后面还有个设计,| 13 | 非常好用的宝贝,| 14 | 这个开关我是一直没搞懂,| 15 | 该有的功能都有了,| 16 | 我只说一次,| 17 | 收到l等了15天,| 18 | 所以系统默认好评,| 19 | 但是卖家很快就补发了,| 20 | 现在过了二十天了也没退,| 21 | 冬天有她很暖和,制热效果 22 | 收到第一时间体验,| 23 | 刮花了,| 24 | 也不知道耗不好点,| 25 | 表扬一下物流服务,| 26 | 外观设计：不站地方,| 27 | 换了小米,| 28 | 总之我自己用下来,| 29 | 价格实惠量又足,| 30 | 耗电情况：整体来说耗电不是特别严重,产品功耗 31 | 小兔兔外型还不错了,| 32 | 其他特色：利用小爱或者APP可以智能设定,| 33 | 京东的服务超好,| 34 | 这款取暖器已经买两个了,| 35 | 吹到的风才暖,| 36 | 来的及时,| 37 | 全网最便宜,| 38 | 错过了7天试用,| 39 | 连几十块钱的取暖器都不如,| 40 | 家里有小朋友的建议还是使用空调,| 41 | 定时升温控制都很简单,| 42 | 用了近一月,| 43 | 南方湿冷的冬天是很不错的选择,| 44 | 但是我刚开始使用确实有味道,味道 45 | 说是也转到人工客服,| 46 | 只有在旁边能暖一点,| 47 | 2、结构简单,| 48 | 是想象的模样,| 49 | 1档热风和2档热风,| 50 | 除了物流慢一点点,| 51 | 5分钟屋里就暖和了,制热效果 52 | 碰到长期下雨阴雨的天气,| 53 | 颜色比图片还好看,| 54 | 用起来还可以吧,| 55 | 就是感觉大了一点,| 56 | 安全设计：反正加热中我是一点不敢碰的,| 57 | 尤其自营商品,| 58 | 实在是感人,| 59 | 到现在按钮也开不起,| 60 | 开关,| 61 | 放在客房足够用,| 62 | 一会就暖和,制热效果 63 | 回来赶快拆了包装试下效果,| 64 | 性价比还好吧,| 65 | 使用过后一段时间感觉还不错,| 66 | 但是售后服务太差,| 67 | *元多一点,| 68 | 一如既往地又快又好呀,| 69 | 给东北老家买的,| 70 | 再过几分钟就感觉到开始散热了,| 71 | 整个屋子都很暖静音效果：完全没有声音,声音 72 | 比老款要美观,| 73 | 是划线价的一半多一丢丢,| 74 | 这次再买个送婆婆,| 75 | 小太阳完胜,| 76 | 最重要的是,| 77 | 功能强大发热快,制热效果 78 | 塑料味道太大,味道 79 | 格力原装产品,| 80 | 取暖效果：取暖好,制热效果 81 | 是我们买的第二个了,| 82 | 需要个人安装的也不多,| 83 | 没有任何破损的地方,| 84 | 自营售后服务品质有保障,| 85 | 南方冬天暖气是刚需啊,| 86 | 用了两次我和孩子都喉咙发炎了,| 87 | 十几平方的房间开一夜都没觉得有效果,| 88 | 概念&rdquo,| 89 | 外观设计：很不错取暖效果：很好耗电情况：蛮高的静音效果：很不错安全设计：安全其他特色：没有了,产品功耗/制热效果 90 | 我觉得效果比浴霸好,制热效果 91 | 可根据需要选择功率,| 92 | 身边的朋友都有介绍此款,| 93 | 安全设计：设计独特完美,| 94 | 总体来说物美价廉,| 95 | 放书桌旁干活用很合适,| 96 | 如果属于南方的没暖气喜欢的宝宝可以放心购买,| 97 | 还赠送了晾衣架,| 98 | 贴墙放置,| 99 | 非常的美观还耐用,| 100 | 买了你就后悔,| 101 | 说24小时回电话,| 102 | 家里离不开了,| 103 | 估计到冬天没太阳的时候烘衣服方便,| 104 | 十天用了一百六的电费,产品功耗 105 | 有遥控装置,| 106 | 体积虽小,| 107 | 不把手凑上去都感觉不到他的温度,| 108 | 用了两天的,| 109 | 小房间很快就能加热,| 110 | 当天下单第二天就到,| 111 | 加热很快??,| 112 | 静音效果：二档声音不小,| 113 | 也没有长螺丝刀,| 114 | 平均每天就开两小时,| 115 | 但声音还能接受,| 116 | 静音效果：试用了几天,| 117 | 取暖效果：开到二挡很暖和,制热效果 118 | 今年给老人又买一个,| 119 | 看着小,| 120 | 外形取暖效果：超级棒耗电情况：没在意静音效果：安静舒适安全设计：童锁设计省心,产品功耗 121 | 不知道能不能优化,| 122 | 货到手没有一点损坏,| 123 | 今天就到货了,| 124 | 希望要买好好选别家的,| 125 | 利用热空气上升的自然流动,| 126 | 长的差不多也是200左右,| 127 | 现在穿着睡衣,| 128 | 产品的质量不错,| 129 | 希望能耐用,| 130 | 再也不用担心受凉感冒了,| 131 | 一晚耗电30不是梦,产品功耗 132 | 还会有电火花闪现,| 133 | 有一定的防水,| 134 | 双十二特价很便宜,| 135 | 但是一个小时两度电也是真的,| 136 | 在广州用很合适,| 137 | 大人小朋友不容易烫伤,| 138 | 选择小米唯一吸引我的是支持小爱和手机,| 139 | 着实让我觉得有点生气,| 140 | 主要是家里没有安装暖气片,| 141 | 好多人都和我一样的情况,| 142 | 有风扇设计所致,| 143 | 外观设计：很好看取暖效果：取暖很快效果很好,制热效果 144 | 强卖,| 145 | 还危险了,| 146 | 必须要贴着机器才有点温度,| 147 | 耐用也就有安全保障其他特色：冬天衣服难干,| 148 | 时不时的会咔的一声响,| 149 | 就算开低档温度也不错,| 150 | 挺适合南方的冬天,| 151 | 这东西买的真好,| 152 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/data/test_onehot.csv: -------------------------------------------------------------------------------- 1 | content,|,互联互通,产品功耗,滑轮提手,声音,APP操控性,呼吸灯,外观,底座,制热范围,遥控器电池,味道,制热效果,衣物烘干,体积大小 2 | 也没有烧伤,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 3 | 主要是之前在北方呆过几几年,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 4 | 温暖过冬,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 5 | 开箱试了一下看着很不错,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 6 | 还可以给宝宝凉衣服,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 7 | 快递师傅送货迅速,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 8 | 快递员差评,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 9 | 沒有什么噪音,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 10 | 就感觉有一部分是强行捏到一起去的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 11 | 今天刚到货还没有使用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 12 | 后面还有个设计,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 13 | 非常好用的宝贝,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 14 | 这个开关我是一直没搞懂,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 15 | 该有的功能都有了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 16 | 我只说一次,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 17 | 收到l等了15天,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 18 | 所以系统默认好评,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 19 | 但是卖家很快就补发了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 20 | 现在过了二十天了也没退,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 21 | 冬天有她很暖和,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 22 | 收到第一时间体验,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 23 | 刮花了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 24 | 也不知道耗不好点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 25 | 表扬一下物流服务,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 26 | 外观设计：不站地方,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 27 | 换了小米,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 28 | 总之我自己用下来,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 29 | 价格实惠量又足,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 30 | 耗电情况：整体来说耗电不是特别严重,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 31 | 小兔兔外型还不错了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 32 | 其他特色：利用小爱或者APP可以智能设定,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 33 | 京东的服务超好,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 34 | 这款取暖器已经买两个了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 35 | 吹到的风才暖,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 36 | 来的及时,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 37 | 全网最便宜,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 38 | 错过了7天试用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 39 | 连几十块钱的取暖器都不如,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 40 | 家里有小朋友的建议还是使用空调,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 41 | 定时升温控制都很简单,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 42 | 用了近一月,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 43 | 南方湿冷的冬天是很不错的选择,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 44 | 但是我刚开始使用确实有味道,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0 45 | 说是也转到人工客服,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 46 | 只有在旁边能暖一点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 47 | 2、结构简单,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 48 | 是想象的模样,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 49 | 1档热风和2档热风,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 50 | 除了物流慢一点点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 51 | 5分钟屋里就暖和了,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 52 | 碰到长期下雨阴雨的天气,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 53 | 颜色比图片还好看,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 54 | 用起来还可以吧,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 55 | 就是感觉大了一点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 56 | 安全设计：反正加热中我是一点不敢碰的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 57 | 尤其自营商品,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 58 | 实在是感人,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 59 | 到现在按钮也开不起,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 60 | 开关,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 61 | 放在客房足够用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 62 | 一会就暖和,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 63 | 回来赶快拆了包装试下效果,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 64 | 性价比还好吧,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 65 | 使用过后一段时间感觉还不错,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 66 | 但是售后服务太差,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 67 | *元多一点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 68 | 一如既往地又快又好呀,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 69 | 给东北老家买的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 70 | 再过几分钟就感觉到开始散热了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 71 | 整个屋子都很暖静音效果：完全没有声音,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0 72 | 比老款要美观,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 73 | 是划线价的一半多一丢丢,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 74 | 这次再买个送婆婆,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 75 | 小太阳完胜,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 76 | 最重要的是,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 77 | 功能强大发热快,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 78 | 塑料味道太大,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0 79 | 格力原装产品,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 80 | 取暖效果：取暖好,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 81 | 是我们买的第二个了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 82 | 需要个人安装的也不多,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 83 | 没有任何破损的地方,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 84 | 自营售后服务品质有保障,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 85 | 南方冬天暖气是刚需啊,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 86 | 用了两次我和孩子都喉咙发炎了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 87 | 十几平方的房间开一夜都没觉得有效果,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 88 | 概念&rdquo,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 89 | 外观设计：很不错取暖效果：很好耗电情况：蛮高的静音效果：很不错安全设计：安全其他特色：没有了,0,0,1,0,0,0,0,0,0,0,0,0,1,0,0 90 | 我觉得效果比浴霸好,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 91 | 可根据需要选择功率,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 92 | 身边的朋友都有介绍此款,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 93 | 安全设计：设计独特完美,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 94 | 总体来说物美价廉,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 95 | 放书桌旁干活用很合适,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 96 | 如果属于南方的没暖气喜欢的宝宝可以放心购买,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 97 | 还赠送了晾衣架,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 98 | 贴墙放置,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 99 | 非常的美观还耐用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 100 | 买了你就后悔,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 101 | 说24小时回电话,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 102 | 家里离不开了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 103 | 估计到冬天没太阳的时候烘衣服方便,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 104 | 十天用了一百六的电费,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 105 | 有遥控装置,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 106 | 体积虽小,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 107 | 不把手凑上去都感觉不到他的温度,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 108 | 用了两天的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 109 | 小房间很快就能加热,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 110 | 当天下单第二天就到,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 111 | 加热很快??,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 112 | 静音效果：二档声音不小,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 113 | 也没有长螺丝刀,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 114 | 平均每天就开两小时,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 115 | 但声音还能接受,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 116 | 静音效果：试用了几天,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 117 | 取暖效果：开到二挡很暖和,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 118 | 今年给老人又买一个,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 119 | 看着小,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 120 | 外形取暖效果：超级棒耗电情况：没在意静音效果：安静舒适安全设计：童锁设计省心,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 121 | 不知道能不能优化,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 122 | 货到手没有一点损坏,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 123 | 今天就到货了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 124 | 希望要买好好选别家的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 125 | 利用热空气上升的自然流动,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 126 | 长的差不多也是200左右,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 127 | 现在穿着睡衣,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 128 | 产品的质量不错,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 129 | 希望能耐用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 130 | 再也不用担心受凉感冒了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 131 | 一晚耗电30不是梦,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 132 | 还会有电火花闪现,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 133 | 有一定的防水,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 134 | 双十二特价很便宜,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 135 | 但是一个小时两度电也是真的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 136 | 在广州用很合适,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 137 | 大人小朋友不容易烫伤,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 138 | 选择小米唯一吸引我的是支持小爱和手机,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 139 | 着实让我觉得有点生气,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 140 | 主要是家里没有安装暖气片,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 141 | 好多人都和我一样的情况,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 142 | 有风扇设计所致,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 143 | 外观设计：很好看取暖效果：取暖很快效果很好,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 144 | 强卖,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 145 | 还危险了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 146 | 必须要贴着机器才有点温度,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 147 | 耐用也就有安全保障其他特色：冬天衣服难干,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 148 | 时不时的会咔的一声响,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 149 | 就算开低档温度也不错,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 150 | 挺适合南方的冬天,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 151 | 这东西买的真好,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 152 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/data/vocabulary_label.txt: -------------------------------------------------------------------------------- 1 | | 2 | 互联互通 3 | 产品功耗 4 | 滑轮提手 5 | 声音 6 | APP操控性 7 | 呼吸灯 8 | 外观 9 | 底座 10 | 制热范围 11 | 遥控器电池 12 | 味道 13 | 制热效果 14 | 衣物烘干 15 | 体积大小 16 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/eval.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu Apr 9 10:56:43 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | 10 | # 参考： https://github.com/hellonlp/classifier-multi-label/blob/master/evaluation/eval_multi_label.py 11 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/hyperparameters.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Mon Nov 12 14:23:12 2020 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import os 10 | import sys 11 | pwd = os.path.dirname(os.path.abspath(__file__)) 12 | sys.path.append(pwd) 13 | from classifier_multi_label_denses.utils import load_vocabulary 14 | 15 | 16 | class Hyperparamters: 17 | # Train parameters 18 | num_train_epochs = 5 19 | print_step = 100 20 | batch_size = 8#64 21 | summary_step = 10 22 | num_saved_per_epoch = 3 23 | max_to_keep = 100 24 | logdir = 'logdir/CML_Denses' 25 | 26 | # Model paths 27 | file_model_save = 'model/model_save' 28 | file_model_load = 'model/model_load' 29 | 30 | # Train/Test data 31 | data_dir = os.path.join(pwd,'data') 32 | train_data = 'train_onehot.csv' 33 | test_data = 'test_onehot.csv' 34 | 35 | # Load vocabulcary dict 36 | dict_id2label,dict_label2id = load_vocabulary(os.path.join(pwd,'data','vocabulary_label.txt') ) 37 | label_vocabulary = list(dict_id2label.values()) 38 | 39 | # Optimization parameters 40 | warmup_proportion = 0.1 41 | use_tpu = None 42 | do_lower_case = True 43 | learning_rate = 5e-5 44 | 45 | 46 | # Sequence and Label 47 | sequence_length = 60 48 | num_labels = len(list(dict_id2label)) 49 | 50 | # ALBERT 51 | model = 'albert_small_zh_google' 52 | bert_path = os.path.join(pwd,model) 53 | vocab_file = os.path.join(pwd,model,'vocab_chinese.txt') 54 | init_checkpoint = os.path.join(pwd,model,'albert_model.ckpt') 55 | saved_model_path = os.path.join(pwd,'model') 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/lamb_optimizer.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | 3 | """Functions and classes related to optimization (weight updates).""" 4 | 5 | from __future__ import absolute_import 6 | from __future__ import division 7 | from __future__ import print_function 8 | 9 | import re 10 | import six 11 | import tensorflow.compat.v1 as tf 12 | 13 | # pylint: disable=g-direct-tensorflow-import 14 | from tensorflow.python.ops import array_ops 15 | from tensorflow.python.ops import linalg_ops 16 | from tensorflow.python.ops import math_ops 17 | # pylint: enable=g-direct-tensorflow-import 18 | 19 | 20 | class LAMBOptimizer(tf.train.Optimizer): 21 | """LAMB (Layer-wise Adaptive Moments optimizer for Batch training).""" 22 | # A new optimizer that includes correct L2 weight decay, adaptive 23 | # element-wise updating, and layer-wise justification. The LAMB optimizer 24 | # was proposed by Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, 25 | # James Demmel, and Cho-Jui Hsieh in a paper titled as Reducing BERT 26 | # Pre-Training Time from 3 Days to 76 Minutes (arxiv.org/abs/1904.00962) 27 | 28 | def __init__(self, 29 | learning_rate, 30 | weight_decay_rate=0.0, 31 | beta_1=0.9, 32 | beta_2=0.999, 33 | epsilon=1e-6, 34 | exclude_from_weight_decay=None, 35 | exclude_from_layer_adaptation=None, 36 | name="LAMBOptimizer"): 37 | """Constructs a LAMBOptimizer.""" 38 | super(LAMBOptimizer, self).__init__(False, name) 39 | 40 | self.learning_rate = learning_rate 41 | self.weight_decay_rate = weight_decay_rate 42 | self.beta_1 = beta_1 43 | self.beta_2 = beta_2 44 | self.epsilon = epsilon 45 | self.exclude_from_weight_decay = exclude_from_weight_decay 46 | # exclude_from_layer_adaptation is set to exclude_from_weight_decay if the 47 | # arg is None. 48 | # TODO(jingli): validate if exclude_from_layer_adaptation is necessary. 49 | if exclude_from_layer_adaptation: 50 | self.exclude_from_layer_adaptation = exclude_from_layer_adaptation 51 | else: 52 | self.exclude_from_layer_adaptation = exclude_from_weight_decay 53 | 54 | def apply_gradients(self, grads_and_vars, global_step=None, name=None): 55 | """See base class.""" 56 | assignments = [] 57 | for (grad, param) in grads_and_vars: 58 | if grad is None or param is None: 59 | continue 60 | 61 | param_name = self._get_variable_name(param.name) 62 | 63 | m = tf.get_variable( 64 | name=six.ensure_str(param_name) + "/adam_m", 65 | shape=param.shape.as_list(), 66 | dtype=tf.float32, 67 | trainable=False, 68 | initializer=tf.zeros_initializer()) 69 | v = tf.get_variable( 70 | name=six.ensure_str(param_name) + "/adam_v", 71 | shape=param.shape.as_list(), 72 | dtype=tf.float32, 73 | trainable=False, 74 | initializer=tf.zeros_initializer()) 75 | 76 | # Standard Adam update. 77 | next_m = ( 78 | tf.multiply(self.beta_1, m) + tf.multiply(1.0 - self.beta_1, grad)) 79 | next_v = ( 80 | tf.multiply(self.beta_2, v) + tf.multiply(1.0 - self.beta_2, 81 | tf.square(grad))) 82 | 83 | update = next_m / (tf.sqrt(next_v) + self.epsilon) 84 | 85 | # Just adding the square of the weights to the loss function is *not* 86 | # the correct way of using L2 regularization/weight decay with Adam, 87 | # since that will interact with the m and v parameters in strange ways. 88 | # 89 | # Instead we want ot decay the weights in a manner that doesn't interact 90 | # with the m/v parameters. This is equivalent to adding the square 91 | # of the weights to the loss with plain (non-momentum) SGD. 92 | if self._do_use_weight_decay(param_name): 93 | update += self.weight_decay_rate * param 94 | 95 | ratio = 1.0 96 | if self._do_layer_adaptation(param_name): 97 | w_norm = linalg_ops.norm(param, ord=2) 98 | g_norm = linalg_ops.norm(update, ord=2) 99 | ratio = array_ops.where(math_ops.greater(w_norm, 0), array_ops.where( 100 | math_ops.greater(g_norm, 0), (w_norm / g_norm), 1.0), 1.0) 101 | 102 | update_with_lr = ratio * self.learning_rate * update 103 | 104 | next_param = param - update_with_lr 105 | 106 | assignments.extend( 107 | [param.assign(next_param), 108 | m.assign(next_m), 109 | v.assign(next_v)]) 110 | return tf.group(*assignments, name=name) 111 | 112 | def _do_use_weight_decay(self, param_name): 113 | """Whether to use L2 weight decay for `param_name`.""" 114 | if not self.weight_decay_rate: 115 | return False 116 | if self.exclude_from_weight_decay: 117 | for r in self.exclude_from_weight_decay: 118 | if re.search(r, param_name) is not None: 119 | return False 120 | return True 121 | 122 | def _do_layer_adaptation(self, param_name): 123 | """Whether to do layer-wise learning rate adaptation for `param_name`.""" 124 | if self.exclude_from_layer_adaptation: 125 | for r in self.exclude_from_layer_adaptation: 126 | if re.search(r, param_name) is not None: 127 | return False 128 | return True 129 | 130 | def _get_variable_name(self, param_name): 131 | """Get the variable name from the tensor name.""" 132 | m = re.match("^(.*):\\d+$", six.ensure_str(param_name)) 133 | if m is not None: 134 | param_name = m.group(1) 135 | return param_name 136 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/logdir/CML_Denses/events.out.tfevents.1602241580.DESKTOP-QC1A83I: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_denses/logdir/CML_Denses/events.out.tfevents.1602241580.DESKTOP-QC1A83I -------------------------------------------------------------------------------- /classifier_multi_label_denses/model/model_load/README.md: -------------------------------------------------------------------------------- 1 | # 推理所用的模型 2 | 3 | ## 模型路径 4 | - model_load/checkpoit 5 | - model_load/model_xx_0.ckpt.data-00000-of-00001 6 | - model_load/model_xx_0.ckpt.index 7 | - model_load/model_xx_0.ckpt.meta 8 | 9 | ## checkpoint内容 10 | ``` 11 | model_checkpoint_path: "model_xx_0.ckpt" 12 | ``` 13 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/model/model_save/README.md: -------------------------------------------------------------------------------- 1 | # 训练过程所得模型 2 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/networks.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu May 30 20:44:42 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | import os 9 | import tensorflow as tf 10 | 11 | from classifier_multi_label_denses import modeling 12 | from classifier_multi_label_denses import optimization 13 | from classifier_multi_label_denses.utils import time_now_string 14 | from classifier_multi_label_denses.hyperparameters import Hyperparamters as hp 15 | from classifier_multi_label_denses.classifier_utils import ClassifyProcessor 16 | 17 | 18 | num_labels = hp.num_labels 19 | processor = ClassifyProcessor() 20 | bert_config_file = os.path.join(hp.bert_path,'albert_config.json') 21 | bert_config = modeling.AlbertConfig.from_json_file(bert_config_file) 22 | 23 | 24 | class NetworkAlbert(object): 25 | def __init__(self,is_training): 26 | # Training or not 27 | self.is_training = is_training 28 | 29 | # Placeholder 30 | self.input_ids = tf.placeholder(tf.int32, shape=[None, hp.sequence_length], name='input_ids') 31 | self.input_masks = tf.placeholder(tf.int32, shape=[None, hp.sequence_length], name='input_masks') 32 | self.segment_ids = tf.placeholder(tf.int32, shape=[None, hp.sequence_length], name='segment_ids') 33 | self.label_ids = tf.placeholder(tf.int32, shape=[None,hp.num_labels], name='label_ids') 34 | 35 | # Load BERT model 36 | self.model = modeling.AlbertModel( 37 | config=bert_config, 38 | is_training=self.is_training, 39 | input_ids=self.input_ids, 40 | input_mask=self.input_masks, 41 | token_type_ids=self.segment_ids, 42 | use_one_hot_embeddings=False) 43 | 44 | 45 | # Get the feature vector by BERT 46 | output_layer = self.model.get_pooled_output() 47 | print('output_layer',output_layer)#(?, 384) 48 | 49 | # Hidden size 50 | hidden_size = output_layer.shape[-1].value 51 | 52 | with tf.name_scope("Full-connection"): 53 | loss_num_label = [] 54 | logits_num_label = [] 55 | for i in range(hp.num_labels): 56 | output_weights = tf.get_variable( 57 | "output_weights%s"%str(i), [2, hidden_size], 58 | initializer=tf.truncated_normal_initializer(stddev=0.02)) 59 | output_bias = tf.get_variable( 60 | "output_bias%s"%str(i), [2], initializer=tf.zeros_initializer()) 61 | logits = tf.matmul(output_layer, output_weights, transpose_b=True) 62 | logits = tf.nn.bias_add(logits, output_bias) 63 | logits_num_label.append(logits) 64 | one_hot_labels = tf.one_hot(self.label_ids[:,i], depth=2, dtype=tf.int32) 65 | per_example_loss = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_labels,logits=logits) 66 | loss_num_label.append(tf.reduce_mean(per_example_loss)) 67 | self.logits_num_label = tf.transpose(tf.stack(logits_num_label, 0),[1,0,2]) 68 | self.loss_num_label = tf.stack(loss_num_label, 0) 69 | self.probabilities = tf.nn.sigmoid(self.logits_num_label) 70 | 71 | with tf.variable_scope("Prediction"): 72 | # Prediction 73 | self.predictions = tf.to_int32(tf.argmax(self.probabilities,2)) 74 | 75 | with tf.variable_scope("loss"): 76 | # Summary for tensorboard 77 | if self.is_training: 78 | self.accuracy = tf.reduce_mean(tf.to_float(tf.equal(self.predictions, self.label_ids))) 79 | tf.summary.scalar('accuracy', self.accuracy) 80 | 81 | # Initial embedding by BERT 82 | ckpt = tf.train.get_checkpoint_state(hp.saved_model_path) 83 | checkpoint_suffix = ".index" 84 | if ckpt and tf.gfile.Exists(ckpt.model_checkpoint_path + checkpoint_suffix): 85 | print('='*10,'Restoring model from checkpoint!','='*10) 86 | print("%s - Restoring model from checkpoint ~%s" % (time_now_string(), 87 | ckpt.model_checkpoint_path)) 88 | else: 89 | print('='*10,'First time load BERT model!','='*10) 90 | tvars = tf.trainable_variables() 91 | if hp.init_checkpoint: 92 | (assignment_map, initialized_variable_names) = \ 93 | modeling.get_assignment_map_from_checkpoint(tvars, 94 | hp.init_checkpoint) 95 | tf.train.init_from_checkpoint(hp.init_checkpoint, assignment_map) 96 | 97 | # Loss and Optimizer 98 | if self.is_training: 99 | # Global_step 100 | self.global_step = tf.Variable(0, name='global_step', trainable=False) 101 | self.loss = tf.reduce_mean(self.loss_num_label) 102 | 103 | # Optimizer BERT 104 | train_examples = processor.get_train_examples(hp.data_dir) 105 | num_train_steps = int( 106 | len(train_examples) / hp.batch_size * hp.num_train_epochs) 107 | num_warmup_steps = int(num_train_steps * hp.warmup_proportion) 108 | print('num_train_steps',num_train_steps) 109 | self.optimizer = optimization.create_optimizer(self.loss, 110 | hp.learning_rate, 111 | num_train_steps, 112 | num_warmup_steps, 113 | hp.use_tpu, 114 | Global_step=self.global_step) 115 | 116 | # Summary for tensorboard 117 | tf.summary.scalar('loss', self.loss) 118 | self.merged = tf.summary.merge_all() 119 | 120 | 121 | if __name__ == '__main__': 122 | # Load model 123 | albert = NetworkAlbert(is_training=True) 124 | 125 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/optimization.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | 3 | """Functions and classes related to optimization (weight updates).""" 4 | 5 | from __future__ import absolute_import 6 | from __future__ import division 7 | from __future__ import print_function 8 | import re 9 | import six 10 | from six.moves import zip 11 | import tensorflow.compat.v1 as tf 12 | from tensorflow.contrib import tpu as contrib_tpu 13 | from classifier_multi_label_denses import lamb_optimizer 14 | 15 | 16 | 17 | def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, use_tpu,Global_step, 18 | optimizer="adamw", poly_power=1.0, start_warmup_step=0): 19 | """Creates an optimizer training op.""" 20 | 21 | if Global_step: 22 | global_step = Global_step 23 | else: 24 | global_step = tf.train.get_or_create_global_step() 25 | #global_step = tf.train.get_or_create_global_step() 26 | 27 | learning_rate = tf.constant(value=init_lr, shape=[], dtype=tf.float32) 28 | 29 | # Implements linear decay of the learning rate. 30 | learning_rate = tf.train.polynomial_decay( 31 | learning_rate, 32 | global_step, 33 | num_train_steps, 34 | end_learning_rate=0.0, 35 | power=poly_power, 36 | cycle=False) 37 | 38 | # Implements linear warmup. I.e., if global_step - start_warmup_step < 39 | # num_warmup_steps, the learning rate will be 40 | # `(global_step - start_warmup_step)/num_warmup_steps * init_lr`. 41 | if num_warmup_steps: 42 | tf.logging.info("++++++ warmup starts at step " + str(start_warmup_step) 43 | + ", for " + str(num_warmup_steps) + " steps ++++++") 44 | global_steps_int = tf.cast(global_step, tf.int32) 45 | start_warm_int = tf.constant(start_warmup_step, dtype=tf.int32) 46 | global_steps_int = global_steps_int - start_warm_int 47 | warmup_steps_int = tf.constant(num_warmup_steps, dtype=tf.int32) 48 | 49 | global_steps_float = tf.cast(global_steps_int, tf.float32) 50 | warmup_steps_float = tf.cast(warmup_steps_int, tf.float32) 51 | 52 | warmup_percent_done = global_steps_float / warmup_steps_float 53 | warmup_learning_rate = init_lr * warmup_percent_done 54 | 55 | is_warmup = tf.cast(global_steps_int < warmup_steps_int, tf.float32) 56 | learning_rate = ( 57 | (1.0 - is_warmup) * learning_rate + is_warmup * warmup_learning_rate) 58 | 59 | # It is OK that you use this optimizer for finetuning, since this 60 | # is how the model was trained (note that the Adam m/v variables are NOT 61 | # loaded from init_checkpoint.) 62 | # It is OK to use AdamW in the finetuning even the model is trained by LAMB. 63 | # As report in the Bert pulic github, the learning rate for SQuAD 1.1 finetune 64 | # is 3e-5, 4e-5 or 5e-5. For LAMB, the users can use 3e-4, 4e-4,or 5e-4 for a 65 | # batch size of 64 in the finetune. 66 | if optimizer == "adamw": 67 | tf.logging.info("using adamw") 68 | optimizer = AdamWeightDecayOptimizer( 69 | learning_rate=learning_rate, 70 | weight_decay_rate=0.01, 71 | beta_1=0.9, 72 | beta_2=0.999, 73 | epsilon=1e-6, 74 | exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"]) 75 | elif optimizer == "lamb": 76 | tf.logging.info("using lamb") 77 | optimizer = lamb_optimizer.LAMBOptimizer( 78 | learning_rate=learning_rate, 79 | weight_decay_rate=0.01, 80 | beta_1=0.9, 81 | beta_2=0.999, 82 | epsilon=1e-6, 83 | exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"]) 84 | else: 85 | raise ValueError("Not supported optimizer: ", optimizer) 86 | 87 | if use_tpu: 88 | optimizer = contrib_tpu.CrossShardOptimizer(optimizer) 89 | 90 | tvars = tf.trainable_variables() 91 | grads = tf.gradients(loss, tvars) 92 | 93 | # This is how the model was pre-trained. 94 | (grads, _) = tf.clip_by_global_norm(grads, clip_norm=1.0) 95 | 96 | train_op = optimizer.apply_gradients( 97 | list(zip(grads, tvars)), global_step=global_step) 98 | 99 | # Normally the global step update is done inside of `apply_gradients`. 100 | # However, neither `AdamWeightDecayOptimizer` nor `LAMBOptimizer` do this. 101 | # But if you use a different optimizer, you should probably take this line 102 | # out. 103 | new_global_step = global_step + 1 104 | train_op = tf.group(train_op, [global_step.assign(new_global_step)]) 105 | return train_op 106 | 107 | 108 | class AdamWeightDecayOptimizer(tf.train.Optimizer): 109 | """A basic Adam optimizer that includes "correct" L2 weight decay.""" 110 | 111 | def __init__(self, 112 | learning_rate, 113 | weight_decay_rate=0.0, 114 | beta_1=0.9, 115 | beta_2=0.999, 116 | epsilon=1e-6, 117 | exclude_from_weight_decay=None, 118 | name="AdamWeightDecayOptimizer"): 119 | """Constructs a AdamWeightDecayOptimizer.""" 120 | super(AdamWeightDecayOptimizer, self).__init__(False, name) 121 | 122 | self.learning_rate = learning_rate 123 | self.weight_decay_rate = weight_decay_rate 124 | self.beta_1 = beta_1 125 | self.beta_2 = beta_2 126 | self.epsilon = epsilon 127 | self.exclude_from_weight_decay = exclude_from_weight_decay 128 | 129 | def apply_gradients(self, grads_and_vars, global_step=None, name=None): 130 | """See base class.""" 131 | assignments = [] 132 | for (grad, param) in grads_and_vars: 133 | if grad is None or param is None: 134 | continue 135 | 136 | param_name = self._get_variable_name(param.name) 137 | 138 | m = tf.get_variable( 139 | name=six.ensure_str(param_name) + "/adam_m", 140 | shape=param.shape.as_list(), 141 | dtype=tf.float32, 142 | trainable=False, 143 | initializer=tf.zeros_initializer()) 144 | v = tf.get_variable( 145 | name=six.ensure_str(param_name) + "/adam_v", 146 | shape=param.shape.as_list(), 147 | dtype=tf.float32, 148 | trainable=False, 149 | initializer=tf.zeros_initializer()) 150 | 151 | # Standard Adam update. 152 | next_m = ( 153 | tf.multiply(self.beta_1, m) + tf.multiply(1.0 - self.beta_1, grad)) 154 | next_v = ( 155 | tf.multiply(self.beta_2, v) + tf.multiply(1.0 - self.beta_2, 156 | tf.square(grad))) 157 | 158 | update = next_m / (tf.sqrt(next_v) + self.epsilon) 159 | 160 | # Just adding the square of the weights to the loss function is *not* 161 | # the correct way of using L2 regularization/weight decay with Adam, 162 | # since that will interact with the m and v parameters in strange ways. 163 | # 164 | # Instead we want ot decay the weights in a manner that doesn't interact 165 | # with the m/v parameters. This is equivalent to adding the square 166 | # of the weights to the loss with plain (non-momentum) SGD. 167 | if self._do_use_weight_decay(param_name): 168 | update += self.weight_decay_rate * param 169 | 170 | update_with_lr = self.learning_rate * update 171 | 172 | next_param = param - update_with_lr 173 | 174 | assignments.extend( 175 | [param.assign(next_param), 176 | m.assign(next_m), 177 | v.assign(next_v)]) 178 | return tf.group(*assignments, name=name) 179 | 180 | def _do_use_weight_decay(self, param_name): 181 | """Whether to use L2 weight decay for `param_name`.""" 182 | if not self.weight_decay_rate: 183 | return False 184 | if self.exclude_from_weight_decay: 185 | for r in self.exclude_from_weight_decay: 186 | if re.search(r, param_name) is not None: 187 | return False 188 | return True 189 | 190 | def _get_variable_name(self, param_name): 191 | """Get the variable name from the tensor name.""" 192 | m = re.match("^(.*):\\d+$", six.ensure_str(param_name)) 193 | if m is not None: 194 | param_name = m.group(1) 195 | return param_name 196 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/predict.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu May 30 17:12:37 2019 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import os 10 | import sys 11 | pwd = os.path.dirname(os.path.abspath(__file__)) 12 | sys.path.append(os.path.dirname(os.path.dirname(__file__))) 13 | import numpy as np 14 | import tensorflow as tf 15 | from classifier_multi_label_denses.networks import NetworkAlbert 16 | from classifier_multi_label_denses.classifier_utils import get_feature_test,id2label 17 | from classifier_multi_label_denses.hyperparameters import Hyperparamters as hp 18 | 19 | 20 | 21 | class ModelAlbertDenses(object,): 22 | """ 23 | Load NetworkAlbertDenses 24 | """ 25 | def __init__(self): 26 | self.albert, self.sess = self.load_model() 27 | @staticmethod 28 | def load_model(): 29 | with tf.Graph().as_default(): 30 | sess = tf.Session() 31 | out_dir = os.path.join(pwd, "model") 32 | with sess.as_default(): 33 | albert = NetworkAlbert(is_training=False) 34 | saver = tf.train.Saver() 35 | sess.run(tf.global_variables_initializer()) 36 | checkpoint_dir = os.path.abspath(os.path.join(out_dir,hp.file_model_load)) 37 | print (checkpoint_dir) 38 | ckpt = tf.train.get_checkpoint_state(checkpoint_dir) 39 | saver.restore(sess, ckpt.model_checkpoint_path) 40 | return albert,sess 41 | 42 | MODEL = ModelAlbertDenses() 43 | print('Load model finished!') 44 | 45 | 46 | 47 | 48 | def get_label(sentence): 49 | """ 50 | Prediction of the sentence's label. 51 | """ 52 | feature = get_feature_test(sentence) 53 | fd = {MODEL.albert.input_ids: [feature[0]], 54 | MODEL.albert.input_masks: [feature[1]], 55 | MODEL.albert.segment_ids:[feature[2]], 56 | } 57 | prediction = MODEL.sess.run(MODEL.albert.predictions, feed_dict=fd)[0] 58 | return [id2label(l) for l in np.where(prediction==1)[0] if l!=0] 59 | 60 | 61 | 62 | if __name__ == '__main__': 63 | ## 64 | sentences = ['外形外观：好看', 65 | '有股难闻的怪味', 66 | '制热效果很好', 67 | '开一晚上也感觉不到热'] 68 | for sentence in sentences: 69 | print(get_label(sentence)) 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/tokenization.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | 3 | """Tokenization classes.""" 4 | 5 | from __future__ import absolute_import 6 | from __future__ import division 7 | from __future__ import print_function 8 | 9 | import collections 10 | import re 11 | import unicodedata 12 | import six 13 | from six.moves import range 14 | import tensorflow.compat.v1 as tf 15 | #import tensorflow_hub as hub 16 | import sentencepiece as spm 17 | 18 | SPIECE_UNDERLINE = u"▁".encode("utf-8") 19 | 20 | 21 | def validate_case_matches_checkpoint(do_lower_case, init_checkpoint): 22 | """Checks whether the casing config is consistent with the checkpoint name.""" 23 | 24 | # The casing has to be passed in by the user and there is no explicit check 25 | # as to whether it matches the checkpoint. The casing information probably 26 | # should have been stored in the bert_config.json file, but it's not, so 27 | # we have to heuristically detect it to validate. 28 | 29 | if not init_checkpoint: 30 | return 31 | 32 | m = re.match("^.*?([A-Za-z0-9_-]+)/bert_model.ckpt", 33 | six.ensure_str(init_checkpoint)) 34 | if m is None: 35 | return 36 | 37 | model_name = m.group(1) 38 | 39 | lower_models = [ 40 | "uncased_L-24_H-1024_A-16", "uncased_L-12_H-768_A-12", 41 | "multilingual_L-12_H-768_A-12", "chinese_L-12_H-768_A-12" 42 | ] 43 | 44 | cased_models = [ 45 | "cased_L-12_H-768_A-12", "cased_L-24_H-1024_A-16", 46 | "multi_cased_L-12_H-768_A-12" 47 | ] 48 | 49 | is_bad_config = False 50 | if model_name in lower_models and not do_lower_case: 51 | is_bad_config = True 52 | actual_flag = "False" 53 | case_name = "lowercased" 54 | opposite_flag = "True" 55 | 56 | if model_name in cased_models and do_lower_case: 57 | is_bad_config = True 58 | actual_flag = "True" 59 | case_name = "cased" 60 | opposite_flag = "False" 61 | 62 | if is_bad_config: 63 | raise ValueError( 64 | "You passed in `--do_lower_case=%s` with `--init_checkpoint=%s`. " 65 | "However, `%s` seems to be a %s model, so you " 66 | "should pass in `--do_lower_case=%s` so that the fine-tuning matches " 67 | "how the model was pre-training. If this error is wrong, please " 68 | "just comment out this check." % (actual_flag, init_checkpoint, 69 | model_name, case_name, opposite_flag)) 70 | 71 | 72 | def preprocess_text(inputs, remove_space=True, lower=False): 73 | """preprocess data by removing extra space and normalize data.""" 74 | outputs = inputs 75 | if remove_space: 76 | outputs = " ".join(inputs.strip().split()) 77 | 78 | if six.PY2 and isinstance(outputs, str): 79 | try: 80 | outputs = six.ensure_text(outputs, "utf-8") 81 | except UnicodeDecodeError: 82 | outputs = six.ensure_text(outputs, "latin-1") 83 | 84 | outputs = unicodedata.normalize("NFKD", outputs) 85 | outputs = "".join([c for c in outputs if not unicodedata.combining(c)]) 86 | if lower: 87 | outputs = outputs.lower() 88 | 89 | return outputs 90 | 91 | 92 | def encode_pieces(sp_model, text, return_unicode=True, sample=False): 93 | """turn sentences into word pieces.""" 94 | 95 | if six.PY2 and isinstance(text, six.text_type): 96 | text = six.ensure_binary(text, "utf-8") 97 | 98 | if not sample: 99 | pieces = sp_model.EncodeAsPieces(text) 100 | else: 101 | pieces = sp_model.SampleEncodeAsPieces(text, 64, 0.1) 102 | new_pieces = [] 103 | for piece in pieces: 104 | piece = printable_text(piece) 105 | if len(piece) > 1 and piece[-1] == "," and piece[-2].isdigit(): 106 | cur_pieces = sp_model.EncodeAsPieces( 107 | six.ensure_binary(piece[:-1]).replace(SPIECE_UNDERLINE, b"")) 108 | if piece[0] != SPIECE_UNDERLINE and cur_pieces[0][0] == SPIECE_UNDERLINE: 109 | if len(cur_pieces[0]) == 1: 110 | cur_pieces = cur_pieces[1:] 111 | else: 112 | cur_pieces[0] = cur_pieces[0][1:] 113 | cur_pieces.append(piece[-1]) 114 | new_pieces.extend(cur_pieces) 115 | else: 116 | new_pieces.append(piece) 117 | 118 | # note(zhiliny): convert back to unicode for py2 119 | if six.PY2 and return_unicode: 120 | ret_pieces = [] 121 | for piece in new_pieces: 122 | if isinstance(piece, str): 123 | piece = six.ensure_text(piece, "utf-8") 124 | ret_pieces.append(piece) 125 | new_pieces = ret_pieces 126 | 127 | return new_pieces 128 | 129 | 130 | def encode_ids(sp_model, text, sample=False): 131 | pieces = encode_pieces(sp_model, text, return_unicode=False, sample=sample) 132 | ids = [sp_model.PieceToId(piece) for piece in pieces] 133 | return ids 134 | 135 | 136 | def convert_to_unicode(text): 137 | """Converts `text` to Unicode (if it's not already), assuming utf-8 input.""" 138 | if six.PY3: 139 | if type(text) in [str,bytes]: 140 | if isinstance(text, str): 141 | return text 142 | elif isinstance(text, bytes): 143 | return six.ensure_text(text, "utf-8", "ignore") 144 | else: 145 | raise ValueError("Unsupported string type: %s" % (type(text))) 146 | # by chenming 147 | else: 148 | return text 149 | elif six.PY2: 150 | if isinstance(text, str): 151 | return six.ensure_text(text, "utf-8", "ignore") 152 | elif isinstance(text, six.text_type): 153 | return text 154 | else: 155 | raise ValueError("Unsupported string type: %s" % (type(text))) 156 | else: 157 | raise ValueError("Not running on Python2 or Python 3?") 158 | 159 | 160 | 161 | def printale_text(text): 162 | """Returnsb text encoded in a way suitable for print or `tf.logging`.""" 163 | 164 | # These functions want `str` for both Python2 and Python3, but in one case 165 | # it's a Unicode string and in the other it's a byte string. 166 | if six.PY3: 167 | if isinstance(text, str): 168 | return text 169 | elif isinstance(text, bytes): 170 | return six.ensure_text(text, "utf-8", "ignore") 171 | else: 172 | raise ValueError("Unsupported string type: %s" % (type(text))) 173 | elif six.PY2: 174 | if isinstance(text, str): 175 | return text 176 | elif isinstance(text, six.text_type): 177 | return six.ensure_binary(text, "utf-8") 178 | else: 179 | raise ValueError("Unsupported string type: %s" % (type(text))) 180 | else: 181 | raise ValueError("Not running on Python2 or Python 3?") 182 | 183 | 184 | def load_vocab(vocab_file): 185 | """Loads a vocabulary file into a dictionary.""" 186 | vocab = collections.OrderedDict() 187 | with tf.gfile.GFile(vocab_file, "r") as reader: 188 | while True: 189 | token = convert_to_unicode(reader.readline()) 190 | if not token: 191 | break 192 | token = token.strip()#.split()[0] 193 | if token not in vocab: 194 | vocab[token] = len(vocab) 195 | return vocab 196 | 197 | 198 | 199 | 200 | def convert_by_vocab(vocab, items): 201 | """Converts a sequence of [tokens|ids] using the vocab.""" 202 | output = [] 203 | for item in items: 204 | output.append(vocab[item]) 205 | return output 206 | 207 | 208 | def convert_tokens_to_ids(vocab, tokens): 209 | return convert_by_vocab(vocab, tokens) 210 | 211 | 212 | def convert_ids_to_tokens(inv_vocab, ids): 213 | return convert_by_vocab(inv_vocab, ids) 214 | 215 | 216 | def whitespace_tokenize(text): 217 | """Runs basic whitespace cleaning and splitting on a piece of text.""" 218 | text = text.strip() 219 | if not text: 220 | return [] 221 | tokens = text.split() 222 | return tokens 223 | 224 | 225 | class FullTokenizer(object): 226 | """Runs end-to-end tokenziation.""" 227 | 228 | def __init__(self, vocab_file, do_lower_case=True, spm_model_file=None): 229 | self.vocab = None 230 | self.sp_model = None 231 | if spm_model_file: 232 | self.sp_model = spm.SentencePieceProcessor() 233 | tf.logging.info("loading sentence piece model") 234 | self.sp_model.Load(spm_model_file) 235 | # Note(mingdachen): For the purpose of consisent API, we are 236 | # generating a vocabulary for the sentence piece tokenizer. 237 | self.vocab = {self.sp_model.IdToPiece(i): i for i 238 | in range(self.sp_model.GetPieceSize())} 239 | else: 240 | self.vocab = load_vocab(vocab_file) 241 | self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case) 242 | self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab) 243 | self.inv_vocab = {v: k for k, v in self.vocab.items()} 244 | 245 | @classmethod 246 | def from_scratch(cls, vocab_file, do_lower_case, spm_model_file): 247 | return FullTokenizer(vocab_file, do_lower_case, spm_model_file) 248 | 249 | # @classmethod 250 | # def from_hub_module(cls, hub_module, spm_model_file): 251 | # """Get the vocab file and casing info from the Hub module.""" 252 | # with tf.Graph().as_default(): 253 | # albert_module = hub.Module(hub_module) 254 | # tokenization_info = albert_module(signature="tokenization_info", 255 | # as_dict=True) 256 | # with tf.Session() as sess: 257 | # vocab_file, do_lower_case = sess.run( 258 | # [tokenization_info["vocab_file"], 259 | # tokenization_info["do_lower_case"]]) 260 | # return FullTokenizer( 261 | # vocab_file=vocab_file, do_lower_case=do_lower_case, 262 | # spm_model_file=spm_model_file) 263 | 264 | def tokenize(self, text): 265 | if self.sp_model: 266 | split_tokens = encode_pieces(self.sp_model, text, return_unicode=False) 267 | else: 268 | split_tokens = [] 269 | for token in self.basic_tokenizer.tokenize(text): 270 | for sub_token in self.wordpiece_tokenizer.tokenize(token): 271 | split_tokens.append(sub_token) 272 | 273 | return split_tokens 274 | 275 | def convert_tokens_to_ids(self, tokens): 276 | if self.sp_model: 277 | tf.logging.info("using sentence piece tokenzier.") 278 | return [self.sp_model.PieceToId( 279 | printable_text(token)) for token in tokens] 280 | else: 281 | return convert_by_vocab(self.vocab, tokens) 282 | 283 | def convert_ids_to_tokens(self, ids): 284 | if self.sp_model: 285 | tf.logging.info("using sentence piece tokenzier.") 286 | return [self.sp_model.IdToPiece(id_) for id_ in ids] 287 | else: 288 | return convert_by_vocab(self.inv_vocab, ids) 289 | 290 | 291 | class BasicTokenizer(object): 292 | """Runs basic tokenization (punctuation splitting, lower casing, etc.).""" 293 | 294 | def __init__(self, do_lower_case=True): 295 | """Constructs a BasicTokenizer. 296 | 297 | Args: 298 | do_lower_case: Whether to lower case the input. 299 | """ 300 | self.do_lower_case = do_lower_case 301 | 302 | def tokenize(self, text): 303 | """Tokenizes a piece of text.""" 304 | text = convert_to_unicode(text) 305 | text = self._clean_text(text) 306 | 307 | # This was added on November 1st, 2018 for the multilingual and Chinese 308 | # models. This is also applied to the English models now, but it doesn't 309 | # matter since the English models were not trained on any Chinese data 310 | # and generally don't have any Chinese data in them (there are Chinese 311 | # characters in the vocabulary because Wikipedia does have some Chinese 312 | # words in the English Wikipedia.). 313 | text = self._tokenize_chinese_chars(text) 314 | 315 | orig_tokens = whitespace_tokenize(text) 316 | split_tokens = [] 317 | for token in orig_tokens: 318 | if self.do_lower_case: 319 | token = token.lower() 320 | token = self._run_strip_accents(token) 321 | split_tokens.extend(self._run_split_on_punc(token)) 322 | 323 | output_tokens = whitespace_tokenize(" ".join(split_tokens)) 324 | return output_tokens 325 | 326 | def _run_strip_accents(self, text): 327 | """Strips accents from a piece of text.""" 328 | text = unicodedata.normalize("NFD", text) 329 | output = [] 330 | for char in text: 331 | cat = unicodedata.category(char) 332 | if cat == "Mn": 333 | continue 334 | output.append(char) 335 | return "".join(output) 336 | 337 | def _run_split_on_punc(self, text): 338 | """Splits punctuation on a piece of text.""" 339 | chars = list(text) 340 | i = 0 341 | start_new_word = True 342 | output = [] 343 | while i < len(chars): 344 | char = chars[i] 345 | if _is_punctuation(char): 346 | output.append([char]) 347 | start_new_word = True 348 | else: 349 | if start_new_word: 350 | output.append([]) 351 | start_new_word = False 352 | output[-1].append(char) 353 | i += 1 354 | 355 | return ["".join(x) for x in output] 356 | 357 | def _tokenize_chinese_chars(self, text): 358 | """Adds whitespace around any CJK character.""" 359 | output = [] 360 | for char in text: 361 | cp = ord(char) 362 | if self._is_chinese_char(cp): 363 | output.append(" ") 364 | output.append(char) 365 | output.append(" ") 366 | else: 367 | output.append(char) 368 | return "".join(output) 369 | 370 | def _is_chinese_char(self, cp): 371 | """Checks whether CP is the codepoint of a CJK character.""" 372 | # This defines a "chinese character" as anything in the CJK Unicode block: 373 | # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block) 374 | # 375 | # Note that the CJK Unicode block is NOT all Japanese and Korean characters, 376 | # despite its name. The modern Korean Hangul alphabet is a different block, 377 | # as is Japanese Hiragana and Katakana. Those alphabets are used to write 378 | # space-separated words, so they are not treated specially and handled 379 | # like the all of the other languages. 380 | if ((cp >= 0x4E00 and cp <= 0x9FFF) or # 381 | (cp >= 0x3400 and cp <= 0x4DBF) or # 382 | (cp >= 0x20000 and cp <= 0x2A6DF) or # 383 | (cp >= 0x2A700 and cp <= 0x2B73F) or # 384 | (cp >= 0x2B740 and cp <= 0x2B81F) or # 385 | (cp >= 0x2B820 and cp <= 0x2CEAF) or 386 | (cp >= 0xF900 and cp <= 0xFAFF) or # 387 | (cp >= 0x2F800 and cp <= 0x2FA1F)): # 388 | return True 389 | 390 | return False 391 | 392 | def _clean_text(self, text): 393 | """Performs invalid character removal and whitespace cleanup on text.""" 394 | output = [] 395 | for char in text: 396 | cp = ord(char) 397 | if cp == 0 or cp == 0xfffd or _is_control(char): 398 | continue 399 | if _is_whitespace(char): 400 | output.append(" ") 401 | else: 402 | output.append(char) 403 | return "".join(output) 404 | 405 | 406 | class WordpieceTokenizer(object): 407 | """Runs WordPiece tokenziation.""" 408 | 409 | def __init__(self, vocab, unk_token="[UNK]", max_input_chars_per_word=200): 410 | self.vocab = vocab 411 | self.unk_token = unk_token 412 | self.max_input_chars_per_word = max_input_chars_per_word 413 | 414 | def tokenize(self, text): 415 | """Tokenizes a piece of text into its word pieces. 416 | 417 | This uses a greedy longest-match-first algorithm to perform tokenization 418 | using the given vocabulary. 419 | 420 | For example: 421 | input = "unaffable" 422 | output = ["un", "##aff", "##able"] 423 | 424 | Args: 425 | text: A single token or whitespace separated tokens. This should have 426 | already been passed through `BasicTokenizer. 427 | 428 | Returns: 429 | A list of wordpiece tokens. 430 | """ 431 | 432 | text = convert_to_unicode(text) 433 | 434 | output_tokens = [] 435 | for token in whitespace_tokenize(text): 436 | chars = list(token) 437 | if len(chars) > self.max_input_chars_per_word: 438 | output_tokens.append(self.unk_token) 439 | continue 440 | 441 | is_bad = False 442 | start = 0 443 | sub_tokens = [] 444 | while start < len(chars): 445 | end = len(chars) 446 | cur_substr = None 447 | while start < end: 448 | substr = "".join(chars[start:end]) 449 | if start > 0: 450 | substr = "##" + six.ensure_str(substr) 451 | if substr in self.vocab: 452 | cur_substr = substr 453 | break 454 | end -= 1 455 | if cur_substr is None: 456 | is_bad = True 457 | break 458 | sub_tokens.append(cur_substr) 459 | start = end 460 | 461 | if is_bad: 462 | output_tokens.append(self.unk_token) 463 | else: 464 | output_tokens.extend(sub_tokens) 465 | return output_tokens 466 | 467 | 468 | def _is_whitespace(char): 469 | """Checks whether `chars` is a whitespace character.""" 470 | # \t, \n, and \r are technically control characters but we treat them 471 | # as whitespace since they are generally considered as such. 472 | if char == " " or char == "\t" or char == "\n" or char == "\r": 473 | return True 474 | cat = unicodedata.category(char) 475 | if cat == "Zs": 476 | return True 477 | return False 478 | 479 | 480 | def _is_control(char): 481 | """Checks whether `chars` is a control character.""" 482 | # These are technically control characters but we count them as whitespace 483 | # characters. 484 | if char == "\t" or char == "\n" or char == "\r": 485 | return False 486 | cat = unicodedata.category(char) 487 | if cat in ("Cc", "Cf"): 488 | return True 489 | return False 490 | 491 | 492 | def _is_punctuation(char): 493 | """Checks whether `chars` is a punctuation character.""" 494 | cp = ord(char) 495 | # We treat all non-letter/number ASCII as punctuation. 496 | # Characters such as "^", "$", and "`" are not in the Unicode 497 | # Punctuation class but we treat them as punctuation anyways, for 498 | # consistency. 499 | if ((cp >= 33 and cp <= 47) or (cp >= 58 and cp <= 64) or 500 | (cp >= 91 and cp <= 96) or (cp >= 123 and cp <= 126)): 501 | return True 502 | cat = unicodedata.category(char) 503 | if cat.startswith("P"): 504 | return True 505 | return False 506 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/train.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu May 30 21:42:07 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | import os 9 | #os.environ["CUDA_VISIBLE_DEVICES"] = '-1' 10 | import numpy as np 11 | import tensorflow as tf 12 | from classifier_multi_label_denses.networks import NetworkAlbert 13 | from classifier_multi_label_denses.classifier_utils import get_features 14 | from classifier_multi_label_denses.hyperparameters import Hyperparamters as hp 15 | from classifier_multi_label_denses.utils import select,time_now_string 16 | 17 | 18 | 19 | pwd = os.path.dirname(os.path.abspath(__file__)) 20 | MODEL = NetworkAlbert(is_training=True) 21 | 22 | 23 | 24 | 25 | # Get data features 26 | input_ids,input_masks,segment_ids,label_ids = get_features() 27 | num_train_samples = len(input_ids) 28 | indexs = np.arange(num_train_samples) 29 | num_batchs = int((num_train_samples - 1) /hp.batch_size) + 1 30 | print('Number of batch:',num_batchs) 31 | 32 | # Set up the graph 33 | saver = tf.train.Saver(max_to_keep=hp.max_to_keep) 34 | sess = tf.Session() 35 | sess.run(tf.global_variables_initializer()) 36 | 37 | # Load model saved before 38 | MODEL_SAVE_PATH = os.path.join(pwd, hp.file_model_save) 39 | ckpt = tf.train.get_checkpoint_state(MODEL_SAVE_PATH) 40 | if ckpt and ckpt.model_checkpoint_path: 41 | saver.restore(sess, ckpt.model_checkpoint_path) 42 | print('Restored model!') 43 | 44 | 45 | with sess.as_default(): 46 | # Tensorboard writer 47 | writer = tf.summary.FileWriter(hp.logdir, sess.graph) 48 | for i in range(hp.num_train_epochs): 49 | np.random.shuffle(indexs) 50 | for j in range(num_batchs-1): 51 | # Get ids selected 52 | i1 = indexs[j * hp.batch_size:min((j + 1) * hp.batch_size, num_train_samples)] 53 | 54 | # Get features 55 | input_id_ = select(input_ids,i1) 56 | input_mask_ = select(input_masks,i1) 57 | segment_id_ = select(segment_ids,i1) 58 | label_id_ = select(label_ids,i1) 59 | 60 | # Feed dict 61 | fd = {MODEL.input_ids: input_id_, 62 | MODEL.input_masks: input_mask_, 63 | MODEL.segment_ids:segment_id_, 64 | MODEL.label_ids:label_id_} 65 | 66 | # Optimizer 67 | sess.run(MODEL.optimizer, feed_dict = fd) 68 | 69 | # Tensorboard 70 | if j%hp.summary_step==0: 71 | summary,glolal_step = sess.run([MODEL.merged,MODEL.global_step], feed_dict = fd) 72 | writer.add_summary(summary, glolal_step) 73 | 74 | # Save Model 75 | if j%(num_batchs//hp.num_saved_per_epoch)==0: 76 | if not os.path.exists(os.path.join(pwd, hp.file_model_save)): 77 | os.makedirs(os.path.join(pwd, hp.file_model_save)) 78 | saver.save(sess, os.path.join(pwd, hp.file_model_save, 'model'+'_%s_%s.ckpt'%(str(i),str(j)))) 79 | 80 | # Log 81 | if j % hp.print_step == 0: 82 | fd = {MODEL.input_ids: input_id_, 83 | MODEL.input_masks: input_mask_, 84 | MODEL.segment_ids:segment_id_, 85 | MODEL.label_ids:label_id_} 86 | loss = sess.run(MODEL.loss, feed_dict = fd) 87 | print('Time:%s, Epoch:%s, Batch number:%s/%s, Loss:%s'%(time_now_string(),str(i),str(j),str(num_batchs),str(loss))) 88 | print('Train finished') 89 | 90 | 91 | -------------------------------------------------------------------------------- /classifier_multi_label_denses/utils.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Fri May 25 23:43:39 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import time 10 | import numpy as np 11 | import pandas as pd 12 | 13 | 14 | def time_now_string(): 15 | return time.strftime("%Y-%m-%d %H:%M:%S",time.localtime( time.time() )) 16 | 17 | 18 | def select(data,ids): 19 | return [data[i] for i in ids] 20 | 21 | 22 | def load_txt(file): 23 | with open(file,encoding='utf-8',errors='ignore') as fp: 24 | lines = fp.readlines() 25 | lines = [l.strip() for l in lines] 26 | print("Load data from file (%s) finished !"%file) 27 | return lines 28 | 29 | 30 | def save_txt(file,lines): 31 | lines = [l+'\n' for l in lines] 32 | with open(file,'w+',encoding='utf-8') as fp: 33 | fp.writelines(lines) 34 | return "Write data to txt finished !" 35 | 36 | 37 | def load_csv(file,encoding='utf-8',header=0): 38 | return pd.read_csv(file, 39 | encoding=encoding, 40 | header=header, 41 | error_bad_lines=False) 42 | 43 | 44 | def save_csv(dataframe,file,header=True,index=None,encoding="gbk"): 45 | return dataframe.to_csv(file, 46 | mode='w+', 47 | header=header, 48 | index=index, 49 | encoding=encoding) 50 | 51 | 52 | 53 | def save_excel(dataframe,file,header=True,sheetname='Sheet1'): 54 | return dataframe.to_excel(file, 55 | header=header, 56 | sheet_name=sheetname) 57 | 58 | 59 | def load_excel(file,header=0): 60 | return pd.read_excel(file, 61 | header=header, 62 | ) 63 | 64 | def load_vocabulary(file_vocabulary_label): 65 | """ 66 | Load vocabulary to dict 67 | """ 68 | vocabulary = load_txt(file_vocabulary_label) 69 | dict_id2label,dict_label2id = {},{} 70 | for i,l in enumerate(vocabulary): 71 | dict_id2label[str(i)] = str(l) 72 | dict_label2id[str(l)] = str(i) 73 | return dict_id2label,dict_label2id 74 | 75 | 76 | def shuffle_two(a1,a2): 77 | ran = np.arange(len(a1)) 78 | np.random.shuffle(ran) 79 | a1_ = [a1[l] for l in ran] 80 | a2_ = [a2[l] for l in ran] 81 | return a1_, a2_ 82 | 83 | 84 | if __name__ == '__main__': 85 | print('') 86 | 87 | 88 | 89 | 90 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/README.md: -------------------------------------------------------------------------------- 1 | # 简介 2 | 1、本项目是在tensorflow版本1.15.0的基础上做的训练和测试。 3 | 2、本项目为中文的多标签文本分类。 4 | 3、欢迎大家联系我 www.hellonlp.com 5 | 4、albert_small_zh_google对应的百度云下载地址： 6 | 链接：https://pan.baidu.com/s/1RKzGJTazlZ7y12YRbAWvyA 7 | 提取码：wuxw 8 | 9 | # 使用方法 10 | 1、准备数据 11 | 数据格式为：classifier_multi_label_seq2seq_attention/data/test.csv 12 | 2、参数设置 13 | 参考脚本 hyperparameters.py，直接修改里面的数值即可。 14 | 3、训练 15 | ``` 16 | python train.py 17 | ``` 18 | 4、预测 19 | ``` 20 | python predict.py 21 | ``` 22 | 注意：推理时需要把model/save中的模型复制到model/load中，并修改model/load中的checkpoint文件的内容为当前模型名称，例如：model_checkpoint_path: "model_xx_xx.ckpt"。 23 | 。 24 | 25 | 26 | # 知乎解读 27 | https://zhuanlan.zhihu.com/p/260743336 28 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/__pycache__/classifier_utils.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_seq2seq/__pycache__/classifier_utils.cpython-37.pyc -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/__pycache__/hyperparameters.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_seq2seq/__pycache__/hyperparameters.cpython-37.pyc -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/__pycache__/lamb_optimizer.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_seq2seq/__pycache__/lamb_optimizer.cpython-37.pyc -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/__pycache__/load.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_seq2seq/__pycache__/load.cpython-37.pyc -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/__pycache__/modeling.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_seq2seq/__pycache__/modeling.cpython-37.pyc -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/__pycache__/modules.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_seq2seq/__pycache__/modules.cpython-37.pyc -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/__pycache__/networks.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_seq2seq/__pycache__/networks.cpython-37.pyc -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/__pycache__/optimization.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_seq2seq/__pycache__/optimization.cpython-37.pyc -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/__pycache__/predict.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_seq2seq/__pycache__/predict.cpython-37.pyc -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/__pycache__/tokenization.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_seq2seq/__pycache__/tokenization.cpython-37.pyc -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/__pycache__/utils.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_seq2seq/__pycache__/utils.cpython-37.pyc -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/albert_small_zh_google/checkpoint: -------------------------------------------------------------------------------- 1 | model_checkpoint_path: "albert_model.ckpt" 2 | all_model_checkpoint_paths: "albert_model.ckpt" 3 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/data/test.csv: -------------------------------------------------------------------------------- 1 | content,label 2 | 这肯定是个假店, 3 | 我是把他们寄给我的所有东西算寄给他们了, 4 | 也就偶尔做个表, 5 | 运行速度：开机不到十秒钟, 6 | 散热性能：散热真垃圾, 7 | 就是系统不怎么好用, 8 | 不会自动启动, 9 | 客服小可的服务也很好, 10 | 否则开机3分钟就发热了, 11 | 充电时完全不能用,产品整体评价 12 | 不能差太多, 13 | 而且我用的是华为手机, 14 | 发货的时候漏发鼠标, 15 | 办公用运行速度还可以, 16 | 触摸屏很省力, 17 | 电脑风扇确实不是太静音, 18 | 没想到真挺好用,产品整体评价 19 | 建议直接送小米游戏耳机, 20 | 可以说这有点霸王条款玩的文字游戏, 21 | 不知道后期会不会再出现, 22 | 不支持Windows系统, 23 | 所以又买了戴尔, 24 | 焚香, 25 | 京东快递还在坚持送货, 26 | 单位够买, 27 | 其他特色：掀开盖子自动开机, 28 | 可以玩LOL没有问题, 29 | 游戏本很可以啦,产品整体评价 30 | 先用试试水哦嘻嘻, 31 | 驱动精灵不得, 32 | 所以最好是装好系统激活了再发货, 33 | 除摄像头位置不好, 34 | 劝告各位 如果是买来作为游戏机, 35 | 所以买了华为电脑, 36 | 我觉得这不是正版, 37 | 物流也是超级给力 赞👍, 38 | 还是感觉电脑操作一般,产品整体评价 39 | 找他们也没用, 40 | 上身效果会好看一些, 41 | 网络就连接不上, 42 | 简约外观,外观设计 43 | 就转圈圈, 44 | 所以就优先考虑买华为的笔记本, 45 | 其他特色：手纹快速开机又安全, 46 | 就是我手机热点要感应很久才可以, 47 | 散热性能：暂时发热不明显, 48 | logo周围的槽打磨不平整,外壳做工 49 | 大爱必须五星呀,产品整体评价 50 | 外形外观：机器外观不错,外观设计 51 | 运行速度秒级别快, 52 | 一切满意, 53 | 从手机型号4-8, 54 | 我这是二手货, 55 | 总体就是不好,产品整体评价 56 | 包括后来咨询补发是也很好, 57 | 快递效率也高, 58 | 外观材质：我买的亮银色的超级好看,机身材质/机身颜色 59 | 华为笔记本配华为手机, 60 | 用的时间长了风扇声音会有点明显, 61 | 电脑排风扇声音太大, 62 | 因为后盖有点翘着,外壳做工 63 | 这个电脑收到两天了, 64 | 画面品质：还没玩游戏应该还可以吧, 65 | 欢迎大家花钱买真货, 66 | 屏幕效果：屏幕看起来很细腻, 67 | 平常状态应该好一些,产品整体评价 68 | 客服潇洒也真的是潇洒😂, 69 | 优点：屏幕占比大, 70 | 轻薄程度：很轻薄很漂亮,重量尺寸 71 | 虽说有时候风扇有点响, 72 | 就是电量只能支撑4个多小时的视频, 73 | 外形外观：外观整体银灰色非常美观,机身颜色 74 | 纠结了好久才决定买银蓝色, 75 | 我开着ps和ai都不卡, 76 | 但就是店家发货太慢了啊啊啊啊, 77 | 电脑卡慢是怎么回事, 78 | 用着特别快, 79 | 整体来说电脑除了灵敏度不行其他都可以,产品整体评价 80 | 轻薄本的通病,重量尺寸 81 | 无语了有点后悔, 82 | 粉色少女心爆棚,机身颜色 83 | 值得购买的,产品整体评价 84 | 以后再碰戴尔的东西我**, 85 | 连基本的办公软件用着也卡, 86 | 用了几天电脑质量还不错,产品整体评价 87 | 显卡效果：超级流畅, 88 | 现在商家真是扣到家了, 89 | 声音也响, 90 | 低调蓝,机身颜色 91 | 送了安装纯净版windows的u盘, 92 | 才用10, 93 | 运行速度：这个价位, 94 | 可以和华为手机共享, 95 | 期待了很久终于到手了, 96 | 挺好的搭配华为手机使用更加👍, 97 | 也真正实现了在手机和笔记本之间的投屏, 98 | 我手机用的是华为P9到P30pro一直忠于他的性能与质量, 99 | 指纹解锁很方便除了电源, 100 | 唯一让我犹豫了一下的是, 101 | 公司批量采购的笔记本, 102 | 运行时基本没声音, 103 | 比预期的快了两天, 104 | 快递员送货立马就说了, 105 | 外形外观：5分,外观设计 106 | 屏幕的彩色非常鲜艳, 107 | 要不左边真的烫, 108 | 日常办公足矣,产品整体评价 109 | 不知道是不是兼容性不太好, 110 | 频繁出现我才要求换货的, 111 | 好用 🉑️ 真的不错,产品整体评价 112 | 非常有质感,机身材质 113 | 偶尔会想几下风扇的声音噪音很小, 114 | 驱动升级, 115 | 客服真心不错, 116 | 屏幕看上去也很舒服, 117 | 会一直支持华为做大做强👍, 118 | 挺好看的 就是激活windows网一直出错 半天才搞好,外观设计 119 | 就是没有送包感觉有点不好, 120 | 客服态度也还蛮好, 121 | 还要一会儿, 122 | 隐蔽式摄像头很创新, 123 | 买给妹妹当学习本的, 124 | 外形外观：外形设计有心了,外观设计 125 | 有谁见过这样的电脑 真的垃圾,产品整体评价 126 | 第一台10年了, 127 | 我都尽量给孩子们严格把关, 128 | 孩子说比原来那个好,产品整体评价 129 | 之前一直也有点担心, 130 | 触屏功能很新颖, 131 | 学会了如何分盘,产品整体评价 132 | 集显完全够用了, 133 | 绑定小米真的非常方便, 134 | 所以果断换了这家店, 135 | 我主要是办公软件还有PS, 136 | 各方面非常到位,产品整体评价 137 | 所以买了的兄弟姐妹就自求多福吧,产品整体评价 138 | 刚开机不久只是注册了一下就开始发热了, 139 | 我是为了无线鼠标才来评论的, 140 | 自己选择很久了, 141 | 电脑总体来说还是满意的,产品整体评价 142 | 主要是买的东西太多, 143 | 也没有出现你说的那种情况啊,产品整体评价 144 | 散热器声音响起来确实挺大, 145 | 怀疑他们家是不是只有一个客服, 146 | 打游戏看着要舒服很多, 147 | 后续再评价性能感受, 148 | 买来就是做做课件, 149 | 买了个绿联的扩展坞, 150 | 刚用不好评论, 151 | 劝大家谨慎购买吧,产品整体评价 152 | 为什么我的跑分怎么低, 153 | 有前置摄像头, 154 | 就是价位高了点, 155 | 一点都不输苹果呀, 156 | 应该是1920*1080P的分辨率, 157 | 轻薄程度：我觉得可以接受,重量尺寸 158 | 重量大小出差也很OK,重量尺寸 159 | 拿起来好轻,重量尺寸 160 | 图一和图二是一个颜色,机身颜色 161 | 屏幕效果：分辨率很高的, 162 | 电脑整体做工还可以, 163 | 感谢为我补发的鼠标, 164 | 最终定位华为品牌, 165 | 三星512G M2高速硬盘, 166 | 同行的话不会买这个啦,产品整体评价 167 | 本来担心几千块东西走物流行不行, 168 | 而且色域不错, 169 | 一天用个几分钟, 170 | 1、开箱后感觉外形做工非常漂亮,外观设计 171 | 虽然有让我产生一些不愉快, 172 | 考虑入手的大家还是换一家吧,产品整体评价 173 | 选择, 174 | cpu基本够用, 175 | 之前那台Thinkpad很老旧了, 176 | 自然我喜欢它就有它的特点,产品整体评价 177 | 运行速度：运行速度：换了Windows系统（换系统各种问题, 178 | 有电源时候, 179 | 买完第二天就降价200, 180 | 买个性能强的笔记本主要为了敲代码, 181 | 出门带着也很方便客服服务态度也非常好很耐心的解答,重量尺寸 182 | 色彩很鲜艳真实, 183 | 女孩子选电脑好看就行了,外观设计 184 | 开机也有点慢, 185 | 另外再告诉大家一个问题, 186 | 非常高清的画面, 187 | 缺少的东西也安排了补发, 188 | 屏幕效果：打开之后特别清晰, 189 | 你的人生态度就是这样一件事情就是这样一件小事就可以了, 190 | 电源键都有问题, 191 | 屏幕效果：清晰度也可以, 192 | 没有鼠标操作有点不方便, 193 | 秒回消息, 194 | 散热性能：暂未高强度使用, 195 | 出乎意料的散热差, 196 | 还, 197 | 感觉跟4千多的没什么区别, 198 | 我的反正是起飞了, 199 | 一个下午出现2次电脑卡死, 200 | 华为电脑做工很不错, 201 | 其他的物流方面, 202 | 还想买一台, 203 | 就是是从武汉发过来的, 204 | 已经给小孩了, 205 | 这个是语音操作设置, 206 | 但感觉不是很轻,重量尺寸 207 | 菜单什么都点不开, 208 | 没有特别值得夸奖,产品整体评价 209 | 没有想象中特别高清, 210 | 办公不会再卡啦, 211 | 外观好看简约大方,外观设计 212 | 让笔记本显得很小巧,重量尺寸 213 | 感觉目前对于我这种大学生来说相当适合,产品整体评价 214 | 打电话想求助, 215 | 可是角度大 性能也够用, 216 | 台式准备卖了, 217 | 觉得Dell很不错, 218 | 容易**, 219 | 不错很好用起来很方便宜点的没有找到,产品整体评价 220 | 512GB PCIe SSD, 221 | 小米的还是不行, 222 | 找苏宁易购的客服解决, 223 | 拿到电脑用了一周多了, 224 | 一直使用联想笔记本、上一个用了10年, 225 | 没送鼠标就算了吧, 226 | 不重操作还算简单,重量尺寸/产品整体评价 227 | 后面收到货后说要退货又给重新发了一个, 228 | 那个包没带子不能背, 229 | 买几千元的东西用一个月, 230 | 好歹几千块的东西, 231 | 不得不说京东服务不错, 232 | 明白吧, 233 | 无需输入密码, 234 | 散热性能：还没有玩过游戏, 235 | 联系工程师说硬件问题, 236 | 特别是电脑背面的大logo以及页面介绍的那个排风系统的外观,外观设计 237 | 联想小新pro13酷睿系列真的, 238 | 这是第一次在京东买东西, 239 | 画面品质：看起来很舒服, 240 | 京东买东西就是好, 241 | 我价保一过, 242 | 然后觉得声音有点大, 243 | 包装也只是加了个塑料袋, 244 | 很值呀, 245 | 而且占屏率高, 246 | 这样用起来真的很不方便,产品整体评价 247 | 外形外观：我选的皓月银很大气,机身颜色 248 | 如后续仍问题不断, 249 | 这个价格相当实惠, 250 | 难道是C盘东西多了吗, 251 | 关注了很多天, 252 | 不建议玩游戏有点卡, 253 | 看起来就是舒服,产品整体评价 254 | 充着这个品牌去, 255 | 居然一个鼠标都没有, 256 | 评论完了告诉我要加3张图才能领E卡, 257 | 简直严重差评,产品整体评价 258 | 也非常灵敏, 259 | 有点提心吊胆, 260 | 真烦人, 261 | 可以补回来, 262 | 运行速度：性能好, 263 | 怪自己买的急了, 264 | 已经到了两三天了, 265 | 比较认可,产品整体评价 266 | 运行cad好快, 267 | 就是玩游戏有点小卡, 268 | 卖家也很好👍👍👍, 269 | 唯一的吐槽点就是没USB接口要额外转接器比较麻烦点, 270 | 鼠标不能一起送货吗, 271 | 超级喜欢奥,产品整体评价 272 | 重启才会好, 273 | 感觉电一下就没有了, 274 | 公司通知在家办公毫不犹豫下单了, 275 | 办公外出携带完美,重量尺寸 276 | 运行速度：速度点击很快, 277 | 也按客服说的又是修复又是卸载又是重装, 278 | 期待已久了, 279 | office 无法注册使用, 280 | 并且还有滋滋的声响, 281 | 所谓的性能猛兽完全就是骗人的, 282 | 太爱这个电脑了,产品整体评价 283 | 第一次在天猫旗舰店购买, 284 | 网络顺畅, 285 | 系统在关机和开机的时候都自动更新系统, 286 | 轻薄程度：比预想的轻多了,重量尺寸 287 | 运行速度：直接慢, 288 | 在直播间下单的, 289 | 发现使用两三天以后就开始卡了, 290 | 按我要求, 291 | 运行速度：16g内存保证了运行流畅, 292 | 指纹等功能太强大了, 293 | 放音乐乐质也不错, 294 | 与台式机比起来反应略迟钝, 295 | 风机开起来声音好大, 296 | 散热性能：也就那样, 297 | 外形外观：小新风格带指纹,外观设计 298 | 官方客服电话打不通, 299 | 简直突破我的想象,产品整体评价 300 | 我当时也是没货客服说核对就上了一个, 301 | 相信京东的品控才在京东买电子产品的, 302 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/data/vocabulary_label.txt: -------------------------------------------------------------------------------- 1 | S 2 | E 3 | | 4 | 产品整体评价 5 | 外观设计 6 | 机身颜色 7 | 重量尺寸 8 | 机身材质 9 | 外壳做工 10 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/eval.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu Apr 9 10:56:43 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | 10 | # 参考： https://github.com/hellonlp/classifier-multi-label/blob/master/evaluation/eval_multi_label.py 11 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/hyperparameters.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Mon Nov 12 14:23:12 2018 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import os 10 | import sys 11 | pwd = os.path.dirname(os.path.abspath(__file__)) 12 | sys.path.append(pwd) 13 | from classifier_multi_label_seq2seq.utils import load_vocabulary 14 | 15 | 16 | class Hyperparamters: 17 | # Train parameters 18 | num_train_epochs = 20 19 | print_step = 100 20 | batch_size = 4 # 128 21 | summary_step = 10 22 | num_saved_per_epoch = 3 23 | max_to_keep = 100 24 | logdir = 'logdir/CML_Seq2Seq' 25 | 26 | # Model paths 27 | file_model_save = 'model/model_save' 28 | file_model_load = 'model/model_load' 29 | 30 | # Train/Test data 31 | data_dir = os.path.join(pwd, 'data') 32 | train_data = 'train.csv' 33 | test_data = 'test.csv' 34 | 35 | # Load vocabulcary dict 36 | dict_id2label, dict_label2id = load_vocabulary(os.path.join(pwd, 'data', 'vocabulary_label.txt')) 37 | label_vocabulary = list(dict_id2label.values()) 38 | 39 | # Optimization parameters 40 | warmup_proportion = 0.1 41 | use_tpu = None 42 | do_lower_case = True 43 | learning_rate = 5e-5 44 | 45 | # BiLSTM parameters 46 | num_layer_lstm_encode = 3 47 | lstm_hidden_size = 768 48 | decoder_embedding_size = 768 49 | 50 | # Beam search 51 | is_beam_search = True 52 | beam_size = 5 53 | max_length = 5 54 | 55 | # Sequence and Label 56 | sequence_length = 60 57 | num_labels = len(list(dict_id2label)) 58 | 59 | # ALBERT 60 | model = 'albert_small_zh_google' 61 | bert_path = os.path.join(pwd, model) 62 | vocab_file = os.path.join(pwd, model, 'vocab_chinese.txt') 63 | init_checkpoint = os.path.join(pwd, model, 'albert_model.ckpt') 64 | saved_model_path = os.path.join(pwd, 'model') 65 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/lamb_optimizer.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | 3 | 4 | import re 5 | import six 6 | import tensorflow.compat.v1 as tf 7 | from tensorflow.python.ops import array_ops 8 | from tensorflow.python.ops import linalg_ops 9 | from tensorflow.python.ops import math_ops 10 | 11 | 12 | class LAMBOptimizer(tf.train.Optimizer): 13 | """LAMB (Layer-wise Adaptive Moments optimizer for Batch training).""" 14 | 15 | # A new optimizer that includes correct L2 weight decay, adaptive 16 | # element-wise updating, and layer-wise justification. The LAMB optimizer 17 | # was proposed by Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, 18 | # James Demmel, and Cho-Jui Hsieh in a paper titled as Reducing BERT 19 | # Pre-Training Time from 3 Days to 76 Minutes (arxiv.org/abs/1904.00962) 20 | 21 | def __init__(self, 22 | learning_rate, 23 | weight_decay_rate=0.0, 24 | beta_1=0.9, 25 | beta_2=0.999, 26 | epsilon=1e-6, 27 | exclude_from_weight_decay=None, 28 | exclude_from_layer_adaptation=None, 29 | name="LAMBOptimizer"): 30 | """Constructs a LAMBOptimizer.""" 31 | super(LAMBOptimizer, self).__init__(False, name) 32 | 33 | self.learning_rate = learning_rate 34 | self.weight_decay_rate = weight_decay_rate 35 | self.beta_1 = beta_1 36 | self.beta_2 = beta_2 37 | self.epsilon = epsilon 38 | self.exclude_from_weight_decay = exclude_from_weight_decay 39 | # exclude_from_layer_adaptation is set to exclude_from_weight_decay if the 40 | # arg is None. 41 | # TODO(jingli): validate if exclude_from_layer_adaptation is necessary. 42 | if exclude_from_layer_adaptation: 43 | self.exclude_from_layer_adaptation = exclude_from_layer_adaptation 44 | else: 45 | self.exclude_from_layer_adaptation = exclude_from_weight_decay 46 | 47 | def apply_gradients(self, grads_and_vars, global_step=None, name=None): 48 | """See base class.""" 49 | assignments = [] 50 | for (grad, param) in grads_and_vars: 51 | if grad is None or param is None: 52 | continue 53 | 54 | param_name = self._get_variable_name(param.name) 55 | 56 | m = tf.get_variable( 57 | name=six.ensure_str(param_name) + "/adam_m", 58 | shape=param.shape.as_list(), 59 | dtype=tf.float32, 60 | trainable=False, 61 | initializer=tf.zeros_initializer()) 62 | v = tf.get_variable( 63 | name=six.ensure_str(param_name) + "/adam_v", 64 | shape=param.shape.as_list(), 65 | dtype=tf.float32, 66 | trainable=False, 67 | initializer=tf.zeros_initializer()) 68 | 69 | # Standard Adam update. 70 | next_m = ( 71 | tf.multiply(self.beta_1, m) + tf.multiply(1.0 - self.beta_1, grad)) 72 | next_v = ( 73 | tf.multiply(self.beta_2, v) + tf.multiply(1.0 - self.beta_2, 74 | tf.square(grad))) 75 | 76 | update = next_m / (tf.sqrt(next_v) + self.epsilon) 77 | 78 | # Just adding the square of the weights to the loss function is *not* 79 | # the correct way of using L2 regularization/weight decay with Adam, 80 | # since that will interact with the m and v parameters in strange ways. 81 | # 82 | # Instead we want ot decay the weights in a manner that doesn't interact 83 | # with the m/v parameters. This is equivalent to adding the square 84 | # of the weights to the loss with plain (non-momentum) SGD. 85 | if self._do_use_weight_decay(param_name): 86 | update += self.weight_decay_rate * param 87 | 88 | ratio = 1.0 89 | if self._do_layer_adaptation(param_name): 90 | w_norm = linalg_ops.norm(param, ord=2) 91 | g_norm = linalg_ops.norm(update, ord=2) 92 | ratio = array_ops.where(math_ops.greater(w_norm, 0), array_ops.where( 93 | math_ops.greater(g_norm, 0), (w_norm / g_norm), 1.0), 1.0) 94 | 95 | update_with_lr = ratio * self.learning_rate * update 96 | 97 | next_param = param - update_with_lr 98 | 99 | assignments.extend( 100 | [param.assign(next_param), 101 | m.assign(next_m), 102 | v.assign(next_v)]) 103 | return tf.group(*assignments, name=name) 104 | 105 | def _do_use_weight_decay(self, param_name): 106 | """Whether to use L2 weight decay for `param_name`.""" 107 | if not self.weight_decay_rate: 108 | return False 109 | if self.exclude_from_weight_decay: 110 | for r in self.exclude_from_weight_decay: 111 | if re.search(r, param_name) is not None: 112 | return False 113 | return True 114 | 115 | def _do_layer_adaptation(self, param_name): 116 | """Whether to do layer-wise learning rate adaptation for `param_name`.""" 117 | if self.exclude_from_layer_adaptation: 118 | for r in self.exclude_from_layer_adaptation: 119 | if re.search(r, param_name) is not None: 120 | return False 121 | return True 122 | 123 | def _get_variable_name(self, param_name): 124 | """Get the variable name from the tensor name.""" 125 | m = re.match("^(.*):\\d+$", six.ensure_str(param_name)) 126 | if m is not None: 127 | param_name = m.group(1) 128 | return param_name 129 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/load.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Wed Jul 8 20:00:59 2020 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | from classifier_multi_label_seq2seq.hyperparameters import Hyperparamters as hp 10 | 11 | 12 | def label2onehot(string): 13 | string = '|' if string == '' else string 14 | string_list = list(str(string).split('/')) + ['E'] 15 | return [int(hp.dict_label2id.get(l)) for l in string_list] 16 | 17 | 18 | def normalization_label(label_ids): 19 | max_length = max([len(l) for l in label_ids]) 20 | return [l + [0] * (max_length - len(l)) if len(l) < max_length else l for i, l in enumerate(label_ids)] 21 | 22 | 23 | if __name__ == '__main__': 24 | # Test 25 | label_ids = [[1, 2, 3], [1, 2], [2, 3, 4, 5, 6]] 26 | print(normalization_label(label_ids)) 27 | # 28 | # from classifier_multi_label_seq2seq.utils import load_csv,save_csv 29 | # f = 'classifier_multi_label_seq2seq/data/test.csv' 30 | # df = load_csv(f) 31 | # contents = df['content'].tolist() 32 | # labels = df['label'].tolist() 33 | # ls = ['产品整体评价','机身颜色','外观设计','重量尺寸','机身材质','外壳做工'] 34 | # labels_new = [] 35 | # for l in labels: 36 | # l1 = str(l).split('/') 37 | # l1_new = [] 38 | # for li in l1: 39 | # if li in ls: 40 | # l1_new.append(li) 41 | # labels_new.append(l1_new) 42 | # # 43 | # import pandas as pd 44 | # df = pd.DataFrame(columns=['content','label']) 45 | # df['content'] = contents 46 | # df['label'] = ['/'.join(l) for l in labels_new] 47 | # file_csv = f = 'classifier_multi_label_seq2seq/data/test0.csv' 48 | # save_csv(df,file_csv,encoding='utf-8-sig') 49 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/logdir/CML_Seq2Seq/Readme.txt: -------------------------------------------------------------------------------- 1 | Readme -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/model/model_load/README.md: -------------------------------------------------------------------------------- 1 | # 推理所用的模型 2 | 3 | ## 模型路径 4 | - model_load/checkpoit 5 | - model_load/model_xx_0.ckpt.data-00000-of-00001 6 | - model_load/model_xx_0.ckpt.index 7 | - model_load/model_xx_0.ckpt.meta 8 | 9 | ## checkpoint内容 10 | ``` 11 | model_checkpoint_path: "model_xx_0.ckpt" 12 | ``` 13 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/model/model_save/README.md: -------------------------------------------------------------------------------- 1 | # 训练过程所得模型 2 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/modules.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Wed Jul 8 19:55:52 2020 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import tensorflow as tf 10 | from tensorflow.python.util import nest 11 | from classifier_multi_label_seq2seq.hyperparameters import Hyperparamters as hp 12 | 13 | dict_id2label, dict_label2id = hp.dict_id2label, hp.dict_label2id 14 | 15 | 16 | def shift_by_one(inputs): 17 | '''Shifts the content of `inputs` to the right by one 18 | so that it becomes the decoder inputs. 19 | Args: 20 | inputs: A 3d tensor with shape of [N, T, C] 21 | Returns: 22 | A 3d tensor with the same shape and dtype as `inputs`. 23 | ''' 24 | return tf.concat((tf.zeros_like(inputs[:, :1]), inputs[:, :-1]), 1) 25 | 26 | 27 | def cell_lstm(lstm_hidden_size, is_training, scope='lstm', reuse=None): 28 | """ 29 | A cell of LSTM 30 | """ 31 | with tf.variable_scope(scope, reuse=reuse): 32 | lstm_cell = tf.contrib.rnn.BasicLSTMCell(lstm_hidden_size) 33 | lstm_cell_drop = tf.contrib.rnn.DropoutWrapper(cell=lstm_cell, output_keep_prob=0.5 if is_training else 1) 34 | return lstm_cell_drop 35 | 36 | 37 | def cell_attention_lstm(units, input_, _is_training): 38 | """ 39 | A cell of attention 40 | """ 41 | attention_mechanism = tf.contrib.seq2seq.BahdanauAttention(num_units=units, 42 | memory=input_) 43 | lstm_cell_ = tf.contrib.rnn.BasicLSTMCell(units) 44 | lstm_cell_drop = tf.contrib.rnn.DropoutWrapper(cell=lstm_cell_, 45 | output_keep_prob=0.5 if _is_training else 1) 46 | cell_with_attetion = tf.contrib.seq2seq.AttentionWrapper(lstm_cell_drop, 47 | attention_mechanism, 48 | hp.lstm_hidden_size) 49 | return cell_with_attetion 50 | 51 | 52 | def encoder(inputs, hidden_size, encoder_inputs_length, _is_training=True, bi_direction=True, scope="Encoder", 53 | reuse=None): 54 | ''' 55 | Args: 56 | inputs: A 2d tensor with shape of [N, T], dtype of int32. 57 | is_training: Whether or not the layer is in training mode. 58 | scope: Optional scope for `variable_scope` 59 | reuse: Boolean, whether to reuse the weights of a previous layer 60 | by the same name. 61 | 62 | Returns: 63 | A collection of Hidden vectors, whose shape is (N, T, E). 64 | ''' 65 | with tf.variable_scope(scope, reuse=reuse): 66 | # Encoder 67 | num_units = hidden_size 68 | if bi_direction: 69 | cell_forward = tf.contrib.rnn.MultiRNNCell( 70 | [cell_lstm(num_units, _is_training) for i in range(hp.num_layer_lstm_encode)]) 71 | cell_backward = tf.contrib.rnn.MultiRNNCell( 72 | [cell_lstm(num_units, _is_training) for i in range(hp.num_layer_lstm_encode)]) 73 | (output_forward, output_backword), (state_forward, state_backward) = tf.nn.bidirectional_dynamic_rnn( 74 | cell_forward, 75 | cell_backward, 76 | inputs, 77 | sequence_length=encoder_inputs_length, 78 | dtype=tf.float32) 79 | 80 | memory = tf.concat([output_forward, output_backword], 2) 81 | state_c = tf.concat([state_forward[2].c, state_backward[2].c], 1) 82 | state_h = tf.concat([state_forward[2].h, state_backward[2].h], 1) 83 | state = tf.contrib.rnn.LSTMStateTuple(state_c, state_h) 84 | 85 | 86 | else: 87 | cell = tf.contrib.rnn.MultiRNNCell(cell_lstm(num_units) * hp.num_encode_lstm) 88 | memory, state = tf.nn.bidirectional_dynamic_rnn(cell, 89 | inputs, 90 | dtype=tf.float32) 91 | return memory, state 92 | 93 | 94 | def decoder(inputs, memory, encode_state, _is_training=True, scope="Decoder", reuse=None): 95 | with tf.variable_scope(scope, reuse=reuse): 96 | # Decoder 97 | vocab_size = len(dict_label2id) 98 | if _is_training: 99 | memory_ = memory 100 | encode_state = encode_state 101 | batch_size = hp.batch_size 102 | else: 103 | if hp.is_beam_search: 104 | memory_ = tf.contrib.seq2seq.tile_batch(memory, multiplier=hp.beam_size) 105 | encode_state = nest.map_structure(lambda s: tf.contrib.seq2seq.tile_batch(s, hp.beam_size), 106 | encode_state) 107 | batch_size = tf.shape(memory)[0] * hp.beam_size 108 | else: 109 | memory_ = memory 110 | encode_state = encode_state 111 | batch_size = tf.shape(memory)[0] 112 | 113 | cell_with_attention = cell_attention_lstm(units=hp.lstm_hidden_size, 114 | input_=memory_, 115 | _is_training=_is_training) 116 | h_decode_initial = cell_with_attention.zero_state(batch_size=batch_size, dtype=tf.float32).clone( 117 | cell_state=encode_state) 118 | output_layer = tf.layers.Dense(vocab_size, 119 | kernel_initializer=tf.truncated_normal_initializer(mean=0.0, stddev=0.1)) 120 | 121 | embedding = tf.get_variable('decoder_embedding', [vocab_size, hp.decoder_embedding_size]) 122 | embedding = tf.concat((tf.zeros(shape=[1, hp.decoder_embedding_size]), embedding[1:, :]), 0) 123 | 124 | if _is_training: 125 | 126 | decoder_inputs = tf.nn.embedding_lookup(embedding, shift_by_one(inputs)) 127 | 128 | targets_length = tf.count_nonzero(inputs, axis=1, dtype=tf.int32) 129 | 130 | max_target_sequence_length = tf.reduce_max(targets_length, name='max_target_len') 131 | mask = tf.sequence_mask(targets_length, max_target_sequence_length, dtype=tf.float32, name='masks') 132 | 133 | training_helper = tf.contrib.seq2seq.TrainingHelper(inputs=decoder_inputs, 134 | sequence_length=targets_length, 135 | time_major=False, 136 | name='training_helper') 137 | 138 | training_decoder = tf.contrib.seq2seq.BasicDecoder(cell=cell_with_attention, 139 | helper=training_helper, 140 | initial_state=h_decode_initial, 141 | output_layer=output_layer) 142 | 143 | outputs, final_state, final_sequence_length = tf.contrib.seq2seq.dynamic_decode(decoder=training_decoder, 144 | impute_finished=True, 145 | maximum_iterations=hp.num_labels) ##解码token的长度 146 | 147 | else: 148 | mask = tf.zeros(shape=[tf.shape(memory)[0], hp.decoder_embedding_size]) 149 | start_tokens = tf.fill([tf.shape(memory)[0]], dict_label2id['E']) 150 | end_token = dict_label2id['S'] 151 | if hp.is_beam_search: 152 | 153 | inference_decoder = tf.contrib.seq2seq.BeamSearchDecoder(cell=cell_with_attention, 154 | embedding=embedding, 155 | start_tokens=start_tokens, 156 | end_token=end_token, 157 | initial_state=h_decode_initial, 158 | beam_width=hp.beam_size, 159 | output_layer=output_layer) 160 | 161 | else: 162 | 163 | decoding_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(embedding=embedding, 164 | start_tokens=start_tokens, 165 | end_token=end_token) 166 | 167 | inference_decoder = tf.contrib.seq2seq.BasicDecoder(cell=cell_with_attention, 168 | helper=decoding_helper, 169 | initial_state=h_decode_initial, 170 | output_layer=output_layer) 171 | 172 | outputs, final_state, final_sequence_length = tf.contrib.seq2seq.dynamic_decode(decoder=inference_decoder, 173 | maximum_iterations=hp.max_length) 174 | return outputs, final_state, mask, final_sequence_length 175 | 176 | 177 | def embed(inputs, vocab_size, num_units, zero_pad=True, scope="embedding", reuse=None): 178 | '''Embeds a given tensor. 179 | 180 | Args: 181 | inputs: A `Tensor` with type `int32` or `int64` containing the ids 182 | to be looked up in `lookup table`. 183 | vocab_size: An int. Vocabulary size. 184 | num_units: An int. Number of embedding hidden units. 185 | zero_pad: A boolean. If True, all the values of the fist row (id 0) 186 | should be constant zeros. 187 | scope: Optional scope for `variable_scope`. 188 | reuse: Boolean, whether to reuse the weights of a previous layer 189 | by the same name. 190 | 191 | Returns: 192 | A `Tensor` with one more rank than inputs's. The last dimesionality 193 | should be `num_units`. 194 | ''' 195 | with tf.variable_scope(scope, reuse=reuse): 196 | lookup_table = tf.get_variable('lookup_table', 197 | dtype=tf.float32, 198 | shape=[vocab_size, num_units], 199 | initializer=tf.truncated_normal_initializer(mean=0.0, stddev=0.01)) 200 | if zero_pad: 201 | lookup_table = tf.concat((tf.zeros(shape=[1, num_units]), 202 | lookup_table[1:, :]), 0) 203 | return tf.nn.embedding_lookup(lookup_table, inputs) 204 | 205 | 206 | if __name__ == '__main__': 207 | print('Done') 208 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/networks.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Wed Jul 8 20:03:12 2020 4 | 5 | @author: cm 6 | """ 7 | 8 | import os 9 | import tensorflow as tf 10 | 11 | from classifier_multi_label_seq2seq import modeling 12 | from classifier_multi_label_seq2seq import optimization 13 | from classifier_multi_label_seq2seq.modules import encoder, decoder 14 | from classifier_multi_label_seq2seq.hyperparameters import Hyperparamters as hp 15 | from classifier_multi_label_seq2seq.utils import time_now_string 16 | from classifier_multi_label_seq2seq.classifier_utils import ClassifyProcessor 17 | 18 | 19 | num_labels = hp.num_labels 20 | processor = ClassifyProcessor() 21 | bert_config_file = os.path.join(hp.bert_path, 'albert_config.json') 22 | bert_config = modeling.AlbertConfig.from_json_file(bert_config_file) 23 | 24 | 25 | class NetworkAlbertSeq2Seq(object): 26 | def __init__(self, is_training): 27 | # Training or not 28 | self.is_training = is_training 29 | 30 | # Placeholder 31 | self.input_ids = tf.placeholder(tf.int32, shape=[None, hp.sequence_length], name='input_ids') 32 | self.input_masks = tf.placeholder(tf.int32, shape=[None, hp.sequence_length], name='input_masks') 33 | self.segment_ids = tf.placeholder(tf.int32, shape=[None, hp.sequence_length], name='segment_ids') 34 | self.label_ids = tf.placeholder(tf.int32, shape=[None, None], name='label_ids') 35 | 36 | # Load BERT token features 37 | self.model = modeling.AlbertModel( 38 | config=bert_config, 39 | is_training=self.is_training, 40 | input_ids=self.input_ids, 41 | input_mask=self.input_masks, 42 | token_type_ids=self.segment_ids, 43 | use_one_hot_embeddings=False) 44 | 45 | # Get tensor BERT 46 | self.output_layer_initial = self.model.get_sequence_output() 47 | 48 | # Hidden_size 49 | self.hidden_size = self.output_layer_initial.shape[-1].value 50 | 51 | with tf.name_scope("Encoder"): 52 | # Get input length of encoder 53 | self.input_ids_sequence_length = tf.count_nonzero(self.output_layer_initial, axis=2, dtype=tf.int32) 54 | self.input_ids_length = tf.count_nonzero(self.input_ids_sequence_length, axis=1, dtype=tf.int32) 55 | 56 | # Encoder 57 | self.memory, self.encode_state = encoder(self.output_layer_initial, 58 | self.hidden_size, 59 | self.input_ids_length, 60 | _is_training=self.is_training) 61 | with tf.name_scope("Decoder"): 62 | # Decoder 63 | self.outputs, self.alignments, self.mask, self.final_sequence_length = decoder(self.label_ids, 64 | self.memory, 65 | self.encode_state, 66 | _is_training=is_training) 67 | 68 | # Initial embedding by BERT 69 | ckpt = tf.train.get_checkpoint_state(hp.saved_model_path) 70 | checkpoint_suffix = ".index" 71 | if ckpt and tf.gfile.Exists(ckpt.model_checkpoint_path + checkpoint_suffix): 72 | print('=' * 10, 'Restoring model from checkpoint!', '=' * 10) 73 | print("%s - Restoring model from checkpoint ~%s" % (time_now_string(), 74 | ckpt.model_checkpoint_path)) 75 | else: 76 | tvars = tf.trainable_variables() 77 | if hp.init_checkpoint: 78 | (assignment_map, initialized_variable_names) = \ 79 | modeling.get_assignment_map_from_checkpoint(tvars, 80 | hp.init_checkpoint) 81 | tf.train.init_from_checkpoint(hp.init_checkpoint, assignment_map) 82 | 83 | # Loss and Optimizer 84 | if self.is_training: 85 | with tf.name_scope("loss"): 86 | # Global step 87 | self.global_step = tf.Variable(0, name='global_step', trainable=False) 88 | 89 | # Prediction 90 | self.predictions = self.outputs.sample_id 91 | 92 | # Loss 93 | per_example_loss = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=self.label_ids, 94 | logits=self.outputs.rnn_output) 95 | 96 | self.istarget = tf.to_float(tf.not_equal(self.label_ids, 0)) 97 | self.loss = tf.reduce_sum(per_example_loss * self.istarget) / (tf.reduce_sum(self.istarget) + 1e-7) 98 | 99 | # Accuracy 100 | self.accuracy = tf.reduce_mean(tf.cast(tf.equal(self.predictions, self.label_ids), tf.float32)) 101 | 102 | # Summary for tensorboard 103 | tf.summary.scalar('accuracy', self.accuracy) 104 | tf.summary.scalar('loss', self.loss) 105 | self.merged = tf.summary.merge_all() 106 | 107 | # Optimizer BERT 108 | num_train_examples = len(processor.get_train_examples(hp.data_dir)) 109 | num_train_steps = int( 110 | num_train_examples / hp.batch_size * hp.num_train_epochs) 111 | num_warmup_steps = int(num_train_steps * hp.warmup_proportion) 112 | self.optimizer = optimization.create_optimizer(self.loss, 113 | hp.learning_rate, 114 | num_train_steps, 115 | num_warmup_steps, 116 | hp.use_tpu, 117 | Global_step=self.global_step) 118 | 119 | else: 120 | if hp.is_beam_search: 121 | self.predictions = self.outputs.predicted_ids 122 | self.predictions_prob = self.outputs.beam_search_decoder_output[0] 123 | self.predictions_all = self.outputs.beam_search_decoder_output 124 | else: 125 | self.predictions = tf.expand_dims(self.outputs.sample_id, -1) 126 | self.probs = tf.expand_dims(self.outputs.rnn_output, -1) 127 | 128 | 129 | if __name__ == '__main__': 130 | # Load model 131 | albert = NetworkAlbertSeq2Seq(is_training=True) 132 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/optimization.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | 3 | 4 | import re 5 | import six 6 | from six.moves import zip 7 | import tensorflow.compat.v1 as tf 8 | from tensorflow.contrib import tpu as contrib_tpu 9 | from classifier_multi_label_seq2seq import lamb_optimizer 10 | 11 | 12 | def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, use_tpu, Global_step, 13 | optimizer="adamw", poly_power=1.0, start_warmup_step=0): 14 | """Creates an optimizer training op.""" 15 | 16 | if Global_step: 17 | global_step = Global_step 18 | else: 19 | global_step = tf.train.get_or_create_global_step() 20 | 21 | learning_rate = tf.constant(value=init_lr, shape=[], dtype=tf.float32) 22 | 23 | # Implements linear decay of the learning rate. 24 | learning_rate = tf.train.polynomial_decay( 25 | learning_rate, 26 | global_step, 27 | num_train_steps, 28 | end_learning_rate=0.0, 29 | power=poly_power, 30 | cycle=False) 31 | 32 | # Implements linear warmup. I.e., if global_step - start_warmup_step < 33 | # num_warmup_steps, the learning rate will be 34 | # `(global_step - start_warmup_step)/num_warmup_steps * init_lr`. 35 | if num_warmup_steps: 36 | tf.logging.info("++++++ warmup starts at step " + str(start_warmup_step) 37 | + ", for " + str(num_warmup_steps) + " steps ++++++") 38 | global_steps_int = tf.cast(global_step, tf.int32) 39 | start_warm_int = tf.constant(start_warmup_step, dtype=tf.int32) 40 | global_steps_int = global_steps_int - start_warm_int 41 | warmup_steps_int = tf.constant(num_warmup_steps, dtype=tf.int32) 42 | 43 | global_steps_float = tf.cast(global_steps_int, tf.float32) 44 | warmup_steps_float = tf.cast(warmup_steps_int, tf.float32) 45 | 46 | warmup_percent_done = global_steps_float / warmup_steps_float 47 | warmup_learning_rate = init_lr * warmup_percent_done 48 | 49 | is_warmup = tf.cast(global_steps_int < warmup_steps_int, tf.float32) 50 | learning_rate = ( 51 | (1.0 - is_warmup) * learning_rate + is_warmup * warmup_learning_rate) 52 | 53 | # It is OK that you use this optimizer for finetuning, since this 54 | # is how the model was trained (note that the Adam m/v variables are NOT 55 | # loaded from init_checkpoint.) 56 | # It is OK to use AdamW in the finetuning even the model is trained by LAMB. 57 | # As report in the Bert pulic github, the learning rate for SQuAD 1.1 finetune 58 | # is 3e-5, 4e-5 or 5e-5. For LAMB, the users can use 3e-4, 4e-4,or 5e-4 for a 59 | # batch size of 64 in the finetune. 60 | if optimizer == "adamw": 61 | tf.logging.info("using adamw") 62 | optimizer = AdamWeightDecayOptimizer( 63 | learning_rate=learning_rate, 64 | weight_decay_rate=0.01, 65 | beta_1=0.9, 66 | beta_2=0.999, 67 | epsilon=1e-6, 68 | exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"]) 69 | elif optimizer == "lamb": 70 | tf.logging.info("using lamb") 71 | optimizer = lamb_optimizer.LAMBOptimizer( 72 | learning_rate=learning_rate, 73 | weight_decay_rate=0.01, 74 | beta_1=0.9, 75 | beta_2=0.999, 76 | epsilon=1e-6, 77 | exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"]) 78 | else: 79 | raise ValueError("Not supported optimizer: ", optimizer) 80 | 81 | if use_tpu: 82 | optimizer = contrib_tpu.CrossShardOptimizer(optimizer) 83 | 84 | tvars = tf.trainable_variables() 85 | grads = tf.gradients(loss, tvars) 86 | 87 | # This is how the model was pre-trained. 88 | (grads, _) = tf.clip_by_global_norm(grads, clip_norm=1.0) 89 | 90 | train_op = optimizer.apply_gradients( 91 | list(zip(grads, tvars)), global_step=global_step) 92 | 93 | # Normally the global step update is done inside of `apply_gradients`. 94 | # However, neither `AdamWeightDecayOptimizer` nor `LAMBOptimizer` do this. 95 | # But if you use a different optimizer, you should probably take this line 96 | # out. 97 | new_global_step = global_step + 1 98 | train_op = tf.group(train_op, [global_step.assign(new_global_step)]) 99 | return train_op 100 | 101 | 102 | class AdamWeightDecayOptimizer(tf.train.Optimizer): 103 | """A basic Adam optimizer that includes "correct" L2 weight decay.""" 104 | 105 | def __init__(self, 106 | learning_rate, 107 | weight_decay_rate=0.0, 108 | beta_1=0.9, 109 | beta_2=0.999, 110 | epsilon=1e-6, 111 | exclude_from_weight_decay=None, 112 | name="AdamWeightDecayOptimizer"): 113 | """Constructs a AdamWeightDecayOptimizer.""" 114 | super(AdamWeightDecayOptimizer, self).__init__(False, name) 115 | 116 | self.learning_rate = learning_rate 117 | self.weight_decay_rate = weight_decay_rate 118 | self.beta_1 = beta_1 119 | self.beta_2 = beta_2 120 | self.epsilon = epsilon 121 | self.exclude_from_weight_decay = exclude_from_weight_decay 122 | 123 | def apply_gradients(self, grads_and_vars, global_step=None, name=None): 124 | """See base class.""" 125 | assignments = [] 126 | for (grad, param) in grads_and_vars: 127 | if grad is None or param is None: 128 | continue 129 | 130 | param_name = self._get_variable_name(param.name) 131 | 132 | m = tf.get_variable( 133 | name=six.ensure_str(param_name) + "/adam_m", 134 | shape=param.shape.as_list(), 135 | dtype=tf.float32, 136 | trainable=False, 137 | initializer=tf.zeros_initializer()) 138 | v = tf.get_variable( 139 | name=six.ensure_str(param_name) + "/adam_v", 140 | shape=param.shape.as_list(), 141 | dtype=tf.float32, 142 | trainable=False, 143 | initializer=tf.zeros_initializer()) 144 | 145 | # Standard Adam update. 146 | next_m = ( 147 | tf.multiply(self.beta_1, m) + tf.multiply(1.0 - self.beta_1, grad)) 148 | next_v = ( 149 | tf.multiply(self.beta_2, v) + tf.multiply(1.0 - self.beta_2, 150 | tf.square(grad))) 151 | 152 | update = next_m / (tf.sqrt(next_v) + self.epsilon) 153 | 154 | # Just adding the square of the weights to the loss function is *not* 155 | # the correct way of using L2 regularization/weight decay with Adam, 156 | # since that will interact with the m and v parameters in strange ways. 157 | # 158 | # Instead we want ot decay the weights in a manner that doesn't interact 159 | # with the m/v parameters. This is equivalent to adding the square 160 | # of the weights to the loss with plain (non-momentum) SGD. 161 | if self._do_use_weight_decay(param_name): 162 | update += self.weight_decay_rate * param 163 | 164 | update_with_lr = self.learning_rate * update 165 | 166 | next_param = param - update_with_lr 167 | 168 | assignments.extend( 169 | [param.assign(next_param), 170 | m.assign(next_m), 171 | v.assign(next_v)]) 172 | return tf.group(*assignments, name=name) 173 | 174 | def _do_use_weight_decay(self, param_name): 175 | """Whether to use L2 weight decay for `param_name`.""" 176 | if not self.weight_decay_rate: 177 | return False 178 | if self.exclude_from_weight_decay: 179 | for r in self.exclude_from_weight_decay: 180 | if re.search(r, param_name) is not None: 181 | return False 182 | return True 183 | 184 | def _get_variable_name(self, param_name): 185 | """Get the variable name from the tensor name.""" 186 | m = re.match("^(.*):\\d+$", six.ensure_str(param_name)) 187 | if m is not None: 188 | param_name = m.group(1) 189 | return param_name 190 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/predict.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu May 30 17:12:37 2019 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import os 10 | # os.environ["CUDA_VISIBLE_DEVICES"] = '-1' 11 | pwd = os.path.dirname(os.path.abspath(__file__)) 12 | import sys 13 | sys.path.append(os.path.dirname(os.path.dirname(__file__))) 14 | import tensorflow as tf 15 | 16 | from classifier_multi_label_seq2seq.networks import NetworkAlbertSeq2Seq 17 | from classifier_multi_label_seq2seq.hyperparameters import Hyperparamters as hp 18 | from classifier_multi_label_seq2seq.classifier_utils import get_feature_test 19 | 20 | 21 | class ModelAlbertSeq2seq(object, ): 22 | """ 23 | Load Network Albert Seq2seq model 24 | """ 25 | 26 | def __init__(self): 27 | self.albert, self.sess = self.load_model() 28 | 29 | @staticmethod 30 | def load_model(): 31 | with tf.Graph().as_default(): 32 | sess = tf.Session() 33 | with sess.as_default(): 34 | albert = NetworkAlbertSeq2Seq(is_training=False) 35 | saver = tf.train.Saver() 36 | sess.run(tf.global_variables_initializer()) 37 | checkpoint_dir = os.path.abspath(os.path.join(pwd, hp.file_model_load)) 38 | ckpt = tf.train.get_checkpoint_state(checkpoint_dir) 39 | saver.restore(sess, ckpt.model_checkpoint_path) 40 | return albert, sess 41 | 42 | 43 | MODEL = ModelAlbertSeq2seq() 44 | print('Load model finished!') 45 | 46 | 47 | def get_label(sentence): 48 | """ 49 | Prediction of the sentence's sentiment. 50 | """ 51 | 52 | feature = get_feature_test(sentence) 53 | fd = {MODEL.albert.input_ids: [feature[0]], 54 | MODEL.albert.input_masks: [feature[1]], 55 | MODEL.albert.segment_ids: [feature[2]], 56 | } 57 | output = MODEL.sess.run(MODEL.albert.predictions, feed_dict=fd) 58 | return sorted([hp.dict_id2label[i] for i in output[0][:, 0] if i != 1]) 59 | 60 | 61 | def get_labels(sentences): 62 | """ 63 | Prediction of some sentences's sentiment. 64 | """ 65 | features = [get_feature_test(str(sentence)) for sentence in sentences] 66 | fd = {MODEL.albert.input_ids: [feature[0] for feature in features], 67 | MODEL.albert.input_masks: [feature[1] for feature in features], 68 | MODEL.albert.segment_ids: [feature[2] for feature in features]} 69 | outputs = MODEL.sess.run(MODEL.albert.predictions, feed_dict=fd) 70 | return [sorted([hp.dict_id2label[i] for i in output[:, 0] if i != 1]) for output in outputs] 71 | 72 | 73 | if __name__ == '__main__': 74 | # Test 75 | sentences = ['重量大小出差也很OK', 76 | '轻薄本的通病', 77 | 'logo周围的槽打磨不平整', 78 | '简约外观'] 79 | for sentence in sentences: 80 | print(get_label(sentence)) 81 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/train.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Fri Jul 10 19:25:55 2020 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | 10 | import os 11 | # os.environ["CUDA_VISIBLE_DEVICES"] = '-1' 12 | import numpy as np 13 | import tensorflow as tf 14 | 15 | from classifier_multi_label_seq2seq.networks import NetworkAlbertSeq2Seq 16 | from classifier_multi_label_seq2seq.classifier_utils import get_features 17 | from classifier_multi_label_seq2seq.hyperparameters import Hyperparamters as hp 18 | from classifier_multi_label_seq2seq.utils import select, shuffle_one, time_now_string 19 | from classifier_multi_label_seq2seq.load import normalization_label 20 | 21 | 22 | pwd = os.path.dirname(os.path.abspath(__file__)) 23 | MODEL = NetworkAlbertSeq2Seq(is_training=True) 24 | 25 | 26 | # Get data features 27 | input_ids, input_masks, segment_ids, label_ids = get_features() 28 | num_train_samples = len(input_ids) 29 | arr = np.arange(num_train_samples) 30 | num_batchs = int((num_train_samples - 1) / hp.batch_size) + 1 31 | print('Number of batch:', num_batchs) 32 | 33 | # Set up the graph 34 | saver = tf.train.Saver(max_to_keep=hp.max_to_keep) 35 | sess = tf.Session() 36 | sess.run(tf.global_variables_initializer()) 37 | 38 | # Load model saved before 39 | MODEL_SAVE_PATH = os.path.join(pwd, hp.file_model_save) 40 | ckpt = tf.train.get_checkpoint_state(MODEL_SAVE_PATH) 41 | if ckpt and ckpt.model_checkpoint_path: 42 | saver.restore(sess, ckpt.model_checkpoint_path) 43 | print('Restored model!') 44 | 45 | with sess.as_default(): 46 | # Tensorboard writer 47 | writer = tf.summary.FileWriter(hp.logdir, sess.graph) 48 | for i in range(hp.num_train_epochs): 49 | indexs = shuffle_one(arr) 50 | for j in range(num_batchs - 1): 51 | i1 = indexs[j * hp.batch_size:min((j + 1) * hp.batch_size, num_train_samples)] 52 | 53 | # Get features 54 | input_id_ = select(input_ids, i1) 55 | input_mask_ = select(input_masks, i1) 56 | segment_id_ = select(segment_ids, i1) 57 | label_id_ = normalization_label(select(label_ids, i1)) 58 | 59 | # Feed dict 60 | fd = {MODEL.input_ids: input_id_, 61 | MODEL.input_masks: input_mask_, 62 | MODEL.segment_ids: segment_id_, 63 | MODEL.label_ids: label_id_} 64 | 65 | # Optimizer 66 | sess.run(MODEL.optimizer, feed_dict=fd) 67 | 68 | # Tensorboard 69 | if j % hp.summary_step == 0: 70 | summary, glolal_step = sess.run([MODEL.merged, MODEL.global_step], feed_dict=fd) 71 | writer.add_summary(summary, glolal_step) 72 | 73 | # Save Model 74 | if j % (num_batchs // hp.num_saved_per_epoch) == 0: 75 | if not os.path.exists(os.path.join(pwd, hp.file_model_save)): 76 | os.makedirs(os.path.join(pwd, hp.file_model_save)) 77 | saver.save(sess, os.path.join(pwd, hp.file_model_save, 'model' + '_%s_%s.ckpt' % (str(i), str(j)))) 78 | 79 | # Log 80 | if j % hp.print_step == 0: 81 | fd = {MODEL.input_ids: input_id_, 82 | MODEL.input_masks: input_mask_, 83 | MODEL.segment_ids: segment_id_, 84 | MODEL.label_ids: label_id_} 85 | loss = sess.run(MODEL.loss, feed_dict=fd) 86 | print('Time:%s, Epoch:%s, Batch number:%s/%s, Loss:%s' % ( 87 | time_now_string(), str(i), str(j), str(num_batchs), str(loss))) 88 | print('Train finished') 89 | -------------------------------------------------------------------------------- /classifier_multi_label_seq2seq/utils.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Wed Jul 8 19:57:44 2020 4 | 5 | @author: cm 6 | """ 7 | 8 | import time 9 | import pandas as pd 10 | import numpy as np 11 | 12 | 13 | def time_now_string(): 14 | return time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(time.time())) 15 | 16 | 17 | def cut_list(data, size): 18 | """ 19 | data: a list 20 | size: the size of cut 21 | """ 22 | return [data[i * size:min((i + 1) * size, len(data))] for i in range(int(len(data) - 1) // size + 1)] 23 | 24 | 25 | def select(data, ids): 26 | return [data[i] for i in ids] 27 | 28 | 29 | def shuffle_one(a1): 30 | ran = np.arange(len(a1)) 31 | np.random.shuffle(ran) 32 | return a1[ran] 33 | 34 | 35 | def load_txt(file): 36 | with open(file, encoding='utf-8', errors='ignore') as fp: 37 | lines = fp.readlines() 38 | lines = [l.strip() for l in lines] 39 | print("Load data from file (%s) finished !" % file) 40 | return lines 41 | 42 | 43 | def save_txt(file, lines): 44 | lines = [l + '\n' for l in lines] 45 | with open(file, 'w+', encoding='utf-8') as fp: 46 | fp.writelines(lines) 47 | return "Write data to txt finished !" 48 | 49 | 50 | def load_csv(file, header=0, encoding="utf-8"): 51 | return pd.read_csv(file, 52 | encoding=encoding, 53 | header=header, 54 | error_bad_lines=False) 55 | 56 | 57 | def save_csv(dataframe, file, header=True, index=None, encoding="utf-8"): 58 | return dataframe.to_csv(file, 59 | mode='w+', 60 | header=header, 61 | index=index, 62 | encoding=encoding) 63 | 64 | 65 | def save_excel(dataframe, file, header=True, sheetname='Sheet1'): 66 | return dataframe.to_excel(file, 67 | header=header, 68 | sheet_name=sheetname) 69 | 70 | 71 | def load_excel(file, header=0): 72 | return pd.read_excel(file, 73 | header=header, 74 | ) 75 | 76 | 77 | def load_vocabulary(file_vocabulary_label): 78 | """ 79 | Load vocabulary to dict 80 | """ 81 | vocabulary = load_txt(file_vocabulary_label) 82 | dict_id2label, dict_label2id = {}, {} 83 | for i, l in enumerate(vocabulary): 84 | dict_id2label[i] = str(l) 85 | dict_label2id[str(l)] = i 86 | return dict_id2label, dict_label2id 87 | 88 | 89 | def shuffle_two(a1, a2): 90 | ran = np.arange(len(a1)) 91 | np.random.shuffle(ran) 92 | a1_ = [a1[l] for l in ran] 93 | a2_ = [a2[l] for l in ran] 94 | return a1_, a2_ 95 | 96 | 97 | if __name__ == '__main__': 98 | print('') 99 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/README.md: -------------------------------------------------------------------------------- 1 | # 简介 2 | 1、本项目是在tensorflow版本1.15.0的基础上做的训练和测试。 3 | 2、本项目为中文的多标签文本分类。 4 | 3、欢迎大家联系 www.hellonlp.com 5 | 4、albert_small_zh_google对应的百度云下载地址： 6 | 链接：https://pan.baidu.com/s/1RKzGJTazlZ7y12YRbAWvyA 7 | 提取码：wuxw 8 | 9 | 10 | # 使用方法 11 | 1、准备数据 12 | 数据格式为：classifier_multi_label_textcnn/data/test_onehot.csv 13 | 2、参数设置 14 | 参考脚本 hyperparameters.py，直接修改里面的数值即可。 15 | 3、训练 16 | ``` 17 | python train.py 18 | ``` 19 | 4、预测 20 | ``` 21 | python predict.py 22 | ``` 23 | 注意：推理时需要把model/save中的模型复制到model/load中，并修改model/load中的checkpoint文件的内容为当前模型名称，例如：model_checkpoint_path: "model_xx_xx.ckpt"。 24 | 。 25 | 26 | 27 | 28 | # 知乎解读 29 | https://zhuanlan.zhihu.com/p/158622992 30 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/__init__.py: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/albert_small_zh_google/albert_config.json: -------------------------------------------------------------------------------- 1 | { 2 | "attention_probs_dropout_prob": 0.0, 3 | "hidden_act": "gelu", 4 | "hidden_dropout_prob": 0.0, 5 | "embedding_size": 128, 6 | "hidden_size": 384, 7 | "initializer_range": 0.02, 8 | "intermediate_size": 1536, 9 | "max_position_embeddings": 512, 10 | "num_attention_heads": 12, 11 | "num_hidden_layers": 6, 12 | "num_hidden_groups": 1, 13 | "net_structure_type": 0, 14 | "gap_size": 0, 15 | "num_memory_blocks": 0, 16 | "inner_group_num": 1, 17 | "down_scale_factor": 1, 18 | "type_vocab_size": 2, 19 | "vocab_size": 21128 20 | } -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/data/test.csv: -------------------------------------------------------------------------------- 1 | content,label 2 | 也没有烧伤,| 3 | 主要是之前在北方呆过几几年,| 4 | 温暖过冬,| 5 | 开箱试了一下看着很不错,| 6 | 还可以给宝宝凉衣服,| 7 | 快递师傅送货迅速,| 8 | 快递员差评,| 9 | 沒有什么噪音,| 10 | 就感觉有一部分是强行捏到一起去的,| 11 | 今天刚到货还没有使用,| 12 | 后面还有个设计,| 13 | 非常好用的宝贝,| 14 | 这个开关我是一直没搞懂,| 15 | 该有的功能都有了,| 16 | 我只说一次,| 17 | 收到l等了15天,| 18 | 所以系统默认好评,| 19 | 但是卖家很快就补发了,| 20 | 现在过了二十天了也没退,| 21 | 冬天有她很暖和,制热效果 22 | 收到第一时间体验,| 23 | 刮花了,| 24 | 也不知道耗不好点,| 25 | 表扬一下物流服务,| 26 | 外观设计：不站地方,| 27 | 换了小米,| 28 | 总之我自己用下来,| 29 | 价格实惠量又足,| 30 | 耗电情况：整体来说耗电不是特别严重,产品功耗 31 | 小兔兔外型还不错了,| 32 | 其他特色：利用小爱或者APP可以智能设定,| 33 | 京东的服务超好,| 34 | 这款取暖器已经买两个了,| 35 | 吹到的风才暖,| 36 | 来的及时,| 37 | 全网最便宜,| 38 | 错过了7天试用,| 39 | 连几十块钱的取暖器都不如,| 40 | 家里有小朋友的建议还是使用空调,| 41 | 定时升温控制都很简单,| 42 | 用了近一月,| 43 | 南方湿冷的冬天是很不错的选择,| 44 | 但是我刚开始使用确实有味道,味道 45 | 说是也转到人工客服,| 46 | 只有在旁边能暖一点,| 47 | 2、结构简单,| 48 | 是想象的模样,| 49 | 1档热风和2档热风,| 50 | 除了物流慢一点点,| 51 | 5分钟屋里就暖和了,制热效果 52 | 碰到长期下雨阴雨的天气,| 53 | 颜色比图片还好看,| 54 | 用起来还可以吧,| 55 | 就是感觉大了一点,| 56 | 安全设计：反正加热中我是一点不敢碰的,| 57 | 尤其自营商品,| 58 | 实在是感人,| 59 | 到现在按钮也开不起,| 60 | 开关,| 61 | 放在客房足够用,| 62 | 一会就暖和,制热效果 63 | 回来赶快拆了包装试下效果,| 64 | 性价比还好吧,| 65 | 使用过后一段时间感觉还不错,| 66 | 但是售后服务太差,| 67 | *元多一点,| 68 | 一如既往地又快又好呀,| 69 | 给东北老家买的,| 70 | 再过几分钟就感觉到开始散热了,| 71 | 整个屋子都很暖静音效果：完全没有声音,声音 72 | 比老款要美观,| 73 | 是划线价的一半多一丢丢,| 74 | 这次再买个送婆婆,| 75 | 小太阳完胜,| 76 | 最重要的是,| 77 | 功能强大发热快,制热效果 78 | 塑料味道太大,味道 79 | 格力原装产品,| 80 | 取暖效果：取暖好,制热效果 81 | 是我们买的第二个了,| 82 | 需要个人安装的也不多,| 83 | 没有任何破损的地方,| 84 | 自营售后服务品质有保障,| 85 | 南方冬天暖气是刚需啊,| 86 | 用了两次我和孩子都喉咙发炎了,| 87 | 十几平方的房间开一夜都没觉得有效果,| 88 | 概念&rdquo,| 89 | 外观设计：很不错取暖效果：很好耗电情况：蛮高的静音效果：很不错安全设计：安全其他特色：没有了,产品功耗/制热效果 90 | 我觉得效果比浴霸好,制热效果 91 | 可根据需要选择功率,| 92 | 身边的朋友都有介绍此款,| 93 | 安全设计：设计独特完美,| 94 | 总体来说物美价廉,| 95 | 放书桌旁干活用很合适,| 96 | 如果属于南方的没暖气喜欢的宝宝可以放心购买,| 97 | 还赠送了晾衣架,| 98 | 贴墙放置,| 99 | 非常的美观还耐用,| 100 | 买了你就后悔,| 101 | 说24小时回电话,| 102 | 家里离不开了,| 103 | 估计到冬天没太阳的时候烘衣服方便,| 104 | 十天用了一百六的电费,产品功耗 105 | 有遥控装置,| 106 | 体积虽小,| 107 | 不把手凑上去都感觉不到他的温度,| 108 | 用了两天的,| 109 | 小房间很快就能加热,| 110 | 当天下单第二天就到,| 111 | 加热很快??,| 112 | 静音效果：二档声音不小,| 113 | 也没有长螺丝刀,| 114 | 平均每天就开两小时,| 115 | 但声音还能接受,| 116 | 静音效果：试用了几天,| 117 | 取暖效果：开到二挡很暖和,制热效果 118 | 今年给老人又买一个,| 119 | 看着小,| 120 | 外形取暖效果：超级棒耗电情况：没在意静音效果：安静舒适安全设计：童锁设计省心,产品功耗 121 | 不知道能不能优化,| 122 | 货到手没有一点损坏,| 123 | 今天就到货了,| 124 | 希望要买好好选别家的,| 125 | 利用热空气上升的自然流动,| 126 | 长的差不多也是200左右,| 127 | 现在穿着睡衣,| 128 | 产品的质量不错,| 129 | 希望能耐用,| 130 | 再也不用担心受凉感冒了,| 131 | 一晚耗电30不是梦,产品功耗 132 | 还会有电火花闪现,| 133 | 有一定的防水,| 134 | 双十二特价很便宜,| 135 | 但是一个小时两度电也是真的,| 136 | 在广州用很合适,| 137 | 大人小朋友不容易烫伤,| 138 | 选择小米唯一吸引我的是支持小爱和手机,| 139 | 着实让我觉得有点生气,| 140 | 主要是家里没有安装暖气片,| 141 | 好多人都和我一样的情况,| 142 | 有风扇设计所致,| 143 | 外观设计：很好看取暖效果：取暖很快效果很好,制热效果 144 | 强卖,| 145 | 还危险了,| 146 | 必须要贴着机器才有点温度,| 147 | 耐用也就有安全保障其他特色：冬天衣服难干,| 148 | 时不时的会咔的一声响,| 149 | 就算开低档温度也不错,| 150 | 挺适合南方的冬天,| 151 | 这东西买的真好,| 152 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/data/test_onehot.csv: -------------------------------------------------------------------------------- 1 | content,|,互联互通,产品功耗,滑轮提手,声音,APP操控性,呼吸灯,外观,底座,制热范围,遥控器电池,味道,制热效果,衣物烘干,体积大小 2 | 也没有烧伤,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 3 | 主要是之前在北方呆过几几年,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 4 | 温暖过冬,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 5 | 开箱试了一下看着很不错,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 6 | 还可以给宝宝凉衣服,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 7 | 快递师傅送货迅速,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 8 | 快递员差评,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 9 | 沒有什么噪音,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 10 | 就感觉有一部分是强行捏到一起去的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 11 | 今天刚到货还没有使用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 12 | 后面还有个设计,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 13 | 非常好用的宝贝,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 14 | 这个开关我是一直没搞懂,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 15 | 该有的功能都有了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 16 | 我只说一次,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 17 | 收到l等了15天,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 18 | 所以系统默认好评,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 19 | 但是卖家很快就补发了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 20 | 现在过了二十天了也没退,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 21 | 冬天有她很暖和,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 22 | 收到第一时间体验,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 23 | 刮花了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 24 | 也不知道耗不好点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 25 | 表扬一下物流服务,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 26 | 外观设计：不站地方,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 27 | 换了小米,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 28 | 总之我自己用下来,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 29 | 价格实惠量又足,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 30 | 耗电情况：整体来说耗电不是特别严重,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 31 | 小兔兔外型还不错了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 32 | 其他特色：利用小爱或者APP可以智能设定,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 33 | 京东的服务超好,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 34 | 这款取暖器已经买两个了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 35 | 吹到的风才暖,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 36 | 来的及时,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 37 | 全网最便宜,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 38 | 错过了7天试用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 39 | 连几十块钱的取暖器都不如,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 40 | 家里有小朋友的建议还是使用空调,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 41 | 定时升温控制都很简单,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 42 | 用了近一月,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 43 | 南方湿冷的冬天是很不错的选择,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 44 | 但是我刚开始使用确实有味道,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0 45 | 说是也转到人工客服,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 46 | 只有在旁边能暖一点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 47 | 2、结构简单,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 48 | 是想象的模样,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 49 | 1档热风和2档热风,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 50 | 除了物流慢一点点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 51 | 5分钟屋里就暖和了,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 52 | 碰到长期下雨阴雨的天气,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 53 | 颜色比图片还好看,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 54 | 用起来还可以吧,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 55 | 就是感觉大了一点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 56 | 安全设计：反正加热中我是一点不敢碰的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 57 | 尤其自营商品,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 58 | 实在是感人,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 59 | 到现在按钮也开不起,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 60 | 开关,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 61 | 放在客房足够用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 62 | 一会就暖和,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 63 | 回来赶快拆了包装试下效果,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 64 | 性价比还好吧,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 65 | 使用过后一段时间感觉还不错,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 66 | 但是售后服务太差,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 67 | *元多一点,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 68 | 一如既往地又快又好呀,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 69 | 给东北老家买的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 70 | 再过几分钟就感觉到开始散热了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 71 | 整个屋子都很暖静音效果：完全没有声音,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0 72 | 比老款要美观,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 73 | 是划线价的一半多一丢丢,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 74 | 这次再买个送婆婆,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 75 | 小太阳完胜,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 76 | 最重要的是,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 77 | 功能强大发热快,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 78 | 塑料味道太大,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0 79 | 格力原装产品,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 80 | 取暖效果：取暖好,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 81 | 是我们买的第二个了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 82 | 需要个人安装的也不多,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 83 | 没有任何破损的地方,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 84 | 自营售后服务品质有保障,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 85 | 南方冬天暖气是刚需啊,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 86 | 用了两次我和孩子都喉咙发炎了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 87 | 十几平方的房间开一夜都没觉得有效果,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 88 | 概念&rdquo,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 89 | 外观设计：很不错取暖效果：很好耗电情况：蛮高的静音效果：很不错安全设计：安全其他特色：没有了,0,0,1,0,0,0,0,0,0,0,0,0,1,0,0 90 | 我觉得效果比浴霸好,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 91 | 可根据需要选择功率,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 92 | 身边的朋友都有介绍此款,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 93 | 安全设计：设计独特完美,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 94 | 总体来说物美价廉,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 95 | 放书桌旁干活用很合适,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 96 | 如果属于南方的没暖气喜欢的宝宝可以放心购买,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 97 | 还赠送了晾衣架,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 98 | 贴墙放置,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 99 | 非常的美观还耐用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 100 | 买了你就后悔,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 101 | 说24小时回电话,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 102 | 家里离不开了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 103 | 估计到冬天没太阳的时候烘衣服方便,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 104 | 十天用了一百六的电费,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 105 | 有遥控装置,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 106 | 体积虽小,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 107 | 不把手凑上去都感觉不到他的温度,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 108 | 用了两天的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 109 | 小房间很快就能加热,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 110 | 当天下单第二天就到,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 111 | 加热很快??,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 112 | 静音效果：二档声音不小,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 113 | 也没有长螺丝刀,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 114 | 平均每天就开两小时,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 115 | 但声音还能接受,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 116 | 静音效果：试用了几天,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 117 | 取暖效果：开到二挡很暖和,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 118 | 今年给老人又买一个,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 119 | 看着小,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 120 | 外形取暖效果：超级棒耗电情况：没在意静音效果：安静舒适安全设计：童锁设计省心,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 121 | 不知道能不能优化,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 122 | 货到手没有一点损坏,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 123 | 今天就到货了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 124 | 希望要买好好选别家的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 125 | 利用热空气上升的自然流动,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 126 | 长的差不多也是200左右,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 127 | 现在穿着睡衣,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 128 | 产品的质量不错,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 129 | 希望能耐用,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 130 | 再也不用担心受凉感冒了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 131 | 一晚耗电30不是梦,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 132 | 还会有电火花闪现,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 133 | 有一定的防水,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 134 | 双十二特价很便宜,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 135 | 但是一个小时两度电也是真的,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 136 | 在广州用很合适,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 137 | 大人小朋友不容易烫伤,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 138 | 选择小米唯一吸引我的是支持小爱和手机,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 139 | 着实让我觉得有点生气,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 140 | 主要是家里没有安装暖气片,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 141 | 好多人都和我一样的情况,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 142 | 有风扇设计所致,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 143 | 外观设计：很好看取暖效果：取暖很快效果很好,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0 144 | 强卖,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 145 | 还危险了,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 146 | 必须要贴着机器才有点温度,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 147 | 耐用也就有安全保障其他特色：冬天衣服难干,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 148 | 时不时的会咔的一声响,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 149 | 就算开低档温度也不错,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 150 | 挺适合南方的冬天,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 151 | 这东西买的真好,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0 152 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/data/vocabulary_label.txt: -------------------------------------------------------------------------------- 1 | | 2 | 互联互通 3 | 产品功耗 4 | 滑轮提手 5 | 声音 6 | APP操控性 7 | 呼吸灯 8 | 外观 9 | 底座 10 | 制热范围 11 | 遥控器电池 12 | 味道 13 | 制热效果 14 | 衣物烘干 15 | 体积大小 16 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/eval.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu Apr 9 10:56:43 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import pandas as pd 10 | from classifier_multi_label_textcnn.utils import load_csv,load_excel,save_csv,shuffle_two,save_txt 11 | from classifier_multi_label_textcnn.predict import get_label_multi 12 | from classifier_multi_label_textcnn.utils import cut_list 13 | 14 | 15 | if __name__ == '__main__': 16 | print("参考下面的方法") 17 | ## 参考 https://github.com/hellonlp/classifier-multi-label/blob/master/evaluation/eval_multi_label.py 18 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/hyperparameters.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Mon Nov 12 14:23:12 2018 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import os 10 | import sys 11 | pwd = os.path.dirname(os.path.abspath(__file__)) 12 | sys.path.append(pwd) 13 | from classifier_multi_label_textcnn.utils import load_vocabulary 14 | 15 | 16 | class Hyperparamters: 17 | # Train parameters 18 | num_train_epochs = 50 # 由于训练集较少，所以模型拟合需要更多是epoch。 19 | print_step = 100 20 | batch_size = 64 21 | summary_step = 10 22 | num_saved_per_epoch = 3 23 | max_to_keep = 100 24 | logdir = 'logdir/model_01' 25 | 26 | # Model paths 27 | file_model_save = 'model/model_save' 28 | file_model_load = 'model/model_load' 29 | 30 | # Train/Test data 31 | data_dir = os.path.join(pwd,'data') 32 | train_data = 'train_onehot.csv' 33 | test_data = 'test_onehot.csv' 34 | 35 | # Load vocabulcary dict 36 | dict_id2label,dict_label2id = load_vocabulary(os.path.join(pwd,'data','vocabulary_label.txt') ) 37 | label_vocabulary = list(dict_id2label.values()) 38 | 39 | # Optimization parameters 40 | warmup_proportion = 0.1 41 | use_tpu = None 42 | do_lower_case = True 43 | learning_rate = 5e-5 44 | 45 | # TextCNN parameters 46 | num_filters = 128 47 | filter_sizes = [2,3,4,5,6,7] 48 | embedding_size = 384 49 | keep_prob = 0.5 50 | 51 | # Sequence and Label 52 | sequence_length = 60 53 | num_labels = len(list(dict_id2label)) 54 | 55 | # ALBERT 56 | model = 'albert_small_zh_google' 57 | bert_path = os.path.join(pwd,model) 58 | vocab_file = os.path.join(pwd,model,'vocab_chinese.txt') 59 | init_checkpoint = os.path.join(pwd,model,'albert_model.ckpt') 60 | saved_model_path = os.path.join(pwd,'model') 61 | 62 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/image/graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_textcnn/image/graph.png -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/image/loss.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/classifier_multi_label_textcnn/image/loss.png -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/lamb_optimizer.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | 3 | """Functions and classes related to optimization (weight updates).""" 4 | 5 | from __future__ import absolute_import 6 | from __future__ import division 7 | from __future__ import print_function 8 | 9 | import re 10 | import six 11 | import tensorflow.compat.v1 as tf 12 | 13 | # pylint: disable=g-direct-tensorflow-import 14 | from tensorflow.python.ops import array_ops 15 | from tensorflow.python.ops import linalg_ops 16 | from tensorflow.python.ops import math_ops 17 | # pylint: enable=g-direct-tensorflow-import 18 | 19 | 20 | class LAMBOptimizer(tf.train.Optimizer): 21 | """LAMB (Layer-wise Adaptive Moments optimizer for Batch training).""" 22 | # A new optimizer that includes correct L2 weight decay, adaptive 23 | # element-wise updating, and layer-wise justification. The LAMB optimizer 24 | # was proposed by Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, 25 | # James Demmel, and Cho-Jui Hsieh in a paper titled as Reducing BERT 26 | # Pre-Training Time from 3 Days to 76 Minutes (arxiv.org/abs/1904.00962) 27 | 28 | def __init__(self, 29 | learning_rate, 30 | weight_decay_rate=0.0, 31 | beta_1=0.9, 32 | beta_2=0.999, 33 | epsilon=1e-6, 34 | exclude_from_weight_decay=None, 35 | exclude_from_layer_adaptation=None, 36 | name="LAMBOptimizer"): 37 | """Constructs a LAMBOptimizer.""" 38 | super(LAMBOptimizer, self).__init__(False, name) 39 | 40 | self.learning_rate = learning_rate 41 | self.weight_decay_rate = weight_decay_rate 42 | self.beta_1 = beta_1 43 | self.beta_2 = beta_2 44 | self.epsilon = epsilon 45 | self.exclude_from_weight_decay = exclude_from_weight_decay 46 | # exclude_from_layer_adaptation is set to exclude_from_weight_decay if the 47 | # arg is None. 48 | # TODO(jingli): validate if exclude_from_layer_adaptation is necessary. 49 | if exclude_from_layer_adaptation: 50 | self.exclude_from_layer_adaptation = exclude_from_layer_adaptation 51 | else: 52 | self.exclude_from_layer_adaptation = exclude_from_weight_decay 53 | 54 | def apply_gradients(self, grads_and_vars, global_step=None, name=None): 55 | """See base class.""" 56 | assignments = [] 57 | for (grad, param) in grads_and_vars: 58 | if grad is None or param is None: 59 | continue 60 | 61 | param_name = self._get_variable_name(param.name) 62 | 63 | m = tf.get_variable( 64 | name=six.ensure_str(param_name) + "/adam_m", 65 | shape=param.shape.as_list(), 66 | dtype=tf.float32, 67 | trainable=False, 68 | initializer=tf.zeros_initializer()) 69 | v = tf.get_variable( 70 | name=six.ensure_str(param_name) + "/adam_v", 71 | shape=param.shape.as_list(), 72 | dtype=tf.float32, 73 | trainable=False, 74 | initializer=tf.zeros_initializer()) 75 | 76 | # Standard Adam update. 77 | next_m = ( 78 | tf.multiply(self.beta_1, m) + tf.multiply(1.0 - self.beta_1, grad)) 79 | next_v = ( 80 | tf.multiply(self.beta_2, v) + tf.multiply(1.0 - self.beta_2, 81 | tf.square(grad))) 82 | 83 | update = next_m / (tf.sqrt(next_v) + self.epsilon) 84 | 85 | # Just adding the square of the weights to the loss function is *not* 86 | # the correct way of using L2 regularization/weight decay with Adam, 87 | # since that will interact with the m and v parameters in strange ways. 88 | # 89 | # Instead we want ot decay the weights in a manner that doesn't interact 90 | # with the m/v parameters. This is equivalent to adding the square 91 | # of the weights to the loss with plain (non-momentum) SGD. 92 | if self._do_use_weight_decay(param_name): 93 | update += self.weight_decay_rate * param 94 | 95 | ratio = 1.0 96 | if self._do_layer_adaptation(param_name): 97 | w_norm = linalg_ops.norm(param, ord=2) 98 | g_norm = linalg_ops.norm(update, ord=2) 99 | ratio = array_ops.where(math_ops.greater(w_norm, 0), array_ops.where( 100 | math_ops.greater(g_norm, 0), (w_norm / g_norm), 1.0), 1.0) 101 | 102 | update_with_lr = ratio * self.learning_rate * update 103 | 104 | next_param = param - update_with_lr 105 | 106 | assignments.extend( 107 | [param.assign(next_param), 108 | m.assign(next_m), 109 | v.assign(next_v)]) 110 | return tf.group(*assignments, name=name) 111 | 112 | def _do_use_weight_decay(self, param_name): 113 | """Whether to use L2 weight decay for `param_name`.""" 114 | if not self.weight_decay_rate: 115 | return False 116 | if self.exclude_from_weight_decay: 117 | for r in self.exclude_from_weight_decay: 118 | if re.search(r, param_name) is not None: 119 | return False 120 | return True 121 | 122 | def _do_layer_adaptation(self, param_name): 123 | """Whether to do layer-wise learning rate adaptation for `param_name`.""" 124 | if self.exclude_from_layer_adaptation: 125 | for r in self.exclude_from_layer_adaptation: 126 | if re.search(r, param_name) is not None: 127 | return False 128 | return True 129 | 130 | def _get_variable_name(self, param_name): 131 | """Get the variable name from the tensor name.""" 132 | m = re.match("^(.*):\\d+$", six.ensure_str(param_name)) 133 | if m is not None: 134 | param_name = m.group(1) 135 | return param_name 136 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/model/model_load/README.md: -------------------------------------------------------------------------------- 1 | # 训练过程所得模型 2 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/model/model_save/README.md: -------------------------------------------------------------------------------- 1 | # 推理所用的模型 2 | 3 | ## 模型路径 4 | - model_load/checkpoit 5 | - model_load/model_xx_0.ckpt.data-00000-of-00001 6 | - model_load/model_xx_0.ckpt.index 7 | - model_load/model_xx_0.ckpt.meta 8 | 9 | ## checkpoint内容 10 | ``` 11 | model_checkpoint_path: "model_xx_0.ckpt" 12 | ``` 13 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/modules.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu May 30 21:01:45 2019 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import tensorflow as tf 10 | from tensorflow.contrib.rnn import DropoutWrapper 11 | from classifier_multi_label_textcnn.hyperparameters import Hyperparamters as hp 12 | 13 | 14 | def cell_textcnn(inputs,is_training): 15 | # Add a dimension in final shape 16 | inputs_expand = tf.expand_dims(inputs, -1) 17 | # Create a convolution + maxpool layer for each filter size 18 | pooled_outputs = [] 19 | with tf.name_scope("TextCNN"): 20 | for i, filter_size in enumerate(hp.filter_sizes): 21 | with tf.name_scope("conv-maxpool-%s" % filter_size): 22 | # Convolution Layer 23 | filter_shape = [filter_size, hp.embedding_size, 1, hp.num_filters] 24 | W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1),dtype=tf.float32, name="W") 25 | b = tf.Variable(tf.constant(0.1, shape=[hp.num_filters]),dtype=tf.float32, name="b") 26 | conv = tf.nn.conv2d( 27 | inputs_expand, 28 | W, 29 | strides=[1, 1, 1, 1], 30 | padding="VALID", 31 | name="conv") 32 | # Apply nonlinearity 33 | h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu") 34 | # Maxpooling over the outputs 35 | pooled = tf.nn.max_pool( 36 | h, 37 | ksize=[1, hp.sequence_length - filter_size + 1, 1, 1], 38 | strides=[1, 1, 1, 1], 39 | padding='VALID', 40 | name="pool") 41 | pooled_outputs.append(pooled) 42 | # Combine all the pooled features 43 | num_filters_total = hp.num_filters * len(hp.filter_sizes) 44 | h_pool = tf.concat(pooled_outputs, 3) 45 | h_pool_flat = tf.reshape(h_pool, [-1, num_filters_total]) 46 | # Dropout 47 | h_pool_flat_dropout = tf.nn.dropout(h_pool_flat, keep_prob=hp.keep_prob if is_training else 1) 48 | return h_pool_flat_dropout 49 | 50 | 51 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/networks.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu May 30 20:44:42 2019 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import os 10 | import tensorflow as tf 11 | from classifier_multi_label_textcnn import modeling 12 | from classifier_multi_label_textcnn import optimization 13 | from classifier_multi_label_textcnn.modules import cell_textcnn 14 | from classifier_multi_label_textcnn.utils import time_now_string 15 | from classifier_multi_label_textcnn.hyperparameters import Hyperparamters as hp 16 | from classifier_multi_label_textcnn.classifier_utils import ClassifyProcessor 17 | 18 | 19 | num_labels = hp.num_labels 20 | processor = ClassifyProcessor() 21 | bert_config_file = os.path.join(hp.bert_path,'albert_config.json') 22 | bert_config = modeling.AlbertConfig.from_json_file(bert_config_file) 23 | 24 | 25 | class NetworkAlbertTextCNN(object): 26 | def __init__(self,is_training): 27 | # Training or not 28 | self.is_training = is_training 29 | # Placeholder 30 | self.input_ids = tf.placeholder(tf.int32, shape=[None, hp.sequence_length], name='input_ids') 31 | self.input_masks = tf.placeholder(tf.int32, shape=[None, hp.sequence_length], name='input_masks') 32 | self.segment_ids = tf.placeholder(tf.int32, shape=[None, hp.sequence_length], name='segment_ids') 33 | self.label_ids = tf.placeholder(tf.float32, shape=[None,hp.num_labels], name='label_ids') 34 | # Load BERT model 35 | self.model = modeling.AlbertModel( 36 | config=bert_config, 37 | is_training=self.is_training, 38 | input_ids=self.input_ids, 39 | input_mask=self.input_masks, 40 | token_type_ids=self.segment_ids, 41 | use_one_hot_embeddings=False) 42 | # Get the feature vector by BERT 43 | output_layer_init = self.model.get_sequence_output() 44 | # Cell TextCNN 45 | output_layer = cell_textcnn(output_layer_init,self.is_training) 46 | # Hidden size 47 | hidden_size = output_layer.shape[-1].value 48 | # Full-connection 49 | with tf.name_scope("Full-connection"): 50 | output_weights = tf.get_variable( 51 | "output_weights", [num_labels, hidden_size], 52 | initializer=tf.truncated_normal_initializer(stddev=0.02)) 53 | output_bias = tf.get_variable( 54 | "output_bias", [num_labels], initializer=tf.zeros_initializer()) 55 | logits = tf.nn.bias_add(tf.matmul(output_layer, output_weights, transpose_b=True), output_bias) 56 | # Prediction sigmoid(Multi-label) 57 | self.probabilities = tf.nn.sigmoid(logits) 58 | with tf.variable_scope("Prediction"): 59 | # Prediction 60 | zero = tf.zeros_like(self.probabilities) 61 | one = tf.ones_like(self.probabilities) 62 | self.predictions = tf.where(self.probabilities < 0.5, x=zero, y=one) 63 | with tf.variable_scope("loss"): 64 | # Summary for tensorboard 65 | if self.is_training: 66 | self.accuracy = tf.reduce_mean(tf.to_float(tf.equal(self.predictions, self.label_ids))) 67 | tf.summary.scalar('accuracy', self.accuracy) 68 | 69 | # Initial embedding by BERT 70 | ckpt = tf.train.get_checkpoint_state(hp.saved_model_path) 71 | checkpoint_suffix = ".index" 72 | if ckpt and tf.gfile.Exists(ckpt.model_checkpoint_path + checkpoint_suffix): 73 | print('='*10,'Restoring model from checkpoint!','='*10) 74 | print("%s - Restoring model from checkpoint ~%s" % (time_now_string(), 75 | ckpt.model_checkpoint_path)) 76 | else: 77 | print('='*10,'First time load BERT model!','='*10) 78 | tvars = tf.trainable_variables() 79 | if hp.init_checkpoint: 80 | (assignment_map, initialized_variable_names) = \ 81 | modeling.get_assignment_map_from_checkpoint(tvars, 82 | hp.init_checkpoint) 83 | tf.train.init_from_checkpoint(hp.init_checkpoint, assignment_map) 84 | 85 | # Loss and Optimizer 86 | if self.is_training: 87 | # Global_step 88 | self.global_step = tf.Variable(0, name='global_step', trainable=False) 89 | per_example_loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=self.label_ids,logits=logits) 90 | self.loss = tf.reduce_mean(per_example_loss) 91 | # Optimizer BERT 92 | train_examples = processor.get_train_examples(hp.data_dir) 93 | num_train_steps = int( 94 | len(train_examples) / hp.batch_size * hp.num_train_epochs) 95 | num_warmup_steps = int(num_train_steps * hp.warmup_proportion) 96 | print('num_train_steps',num_train_steps) 97 | self.optimizer = optimization.create_optimizer(self.loss, 98 | hp.learning_rate, 99 | num_train_steps, 100 | num_warmup_steps, 101 | hp.use_tpu, 102 | Global_step=self.global_step) 103 | 104 | # Summary for tensorboard 105 | tf.summary.scalar('loss', self.loss) 106 | self.merged = tf.summary.merge_all() 107 | 108 | 109 | 110 | if __name__ == '__main__': 111 | # Load model 112 | albert = NetworkAlbertTextCNN(is_training=True) 113 | 114 | 115 | 116 | 117 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/optimization.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | """ 3 | Created on Wed Aug 10 19:35:56 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | import re 9 | import six 10 | from six.moves import zip 11 | import tensorflow.compat.v1 as tf 12 | from tensorflow.contrib import tpu as contrib_tpu 13 | from classifier_multi_label_textcnn import lamb_optimizer 14 | 15 | 16 | def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, use_tpu,Global_step, 17 | optimizer="adamw", poly_power=1.0, start_warmup_step=0): 18 | """Creates an optimizer training op.""" 19 | 20 | if Global_step: 21 | global_step = Global_step 22 | else: 23 | global_step = tf.train.get_or_create_global_step() 24 | #global_step = tf.train.get_or_create_global_step() 25 | 26 | learning_rate = tf.constant(value=init_lr, shape=[], dtype=tf.float32) 27 | 28 | # Implements linear decay of the learning rate. 29 | learning_rate = tf.train.polynomial_decay( 30 | learning_rate, 31 | global_step, 32 | num_train_steps, 33 | end_learning_rate=0.0, 34 | power=poly_power, 35 | cycle=False) 36 | 37 | # Implements linear warmup. I.e., if global_step - start_warmup_step < 38 | # num_warmup_steps, the learning rate will be 39 | # `(global_step - start_warmup_step)/num_warmup_steps * init_lr`. 40 | if num_warmup_steps: 41 | tf.logging.info("++++++ warmup starts at step " + str(start_warmup_step) 42 | + ", for " + str(num_warmup_steps) + " steps ++++++") 43 | global_steps_int = tf.cast(global_step, tf.int32) 44 | start_warm_int = tf.constant(start_warmup_step, dtype=tf.int32) 45 | global_steps_int = global_steps_int - start_warm_int 46 | warmup_steps_int = tf.constant(num_warmup_steps, dtype=tf.int32) 47 | 48 | global_steps_float = tf.cast(global_steps_int, tf.float32) 49 | warmup_steps_float = tf.cast(warmup_steps_int, tf.float32) 50 | 51 | warmup_percent_done = global_steps_float / warmup_steps_float 52 | warmup_learning_rate = init_lr * warmup_percent_done 53 | 54 | is_warmup = tf.cast(global_steps_int < warmup_steps_int, tf.float32) 55 | learning_rate = ( 56 | (1.0 - is_warmup) * learning_rate + is_warmup * warmup_learning_rate) 57 | 58 | # It is OK that you use this optimizer for finetuning, since this 59 | # is how the model was trained (note that the Adam m/v variables are NOT 60 | # loaded from init_checkpoint.) 61 | # It is OK to use AdamW in the finetuning even the model is trained by LAMB. 62 | # As report in the Bert pulic github, the learning rate for SQuAD 1.1 finetune 63 | # is 3e-5, 4e-5 or 5e-5. For LAMB, the users can use 3e-4, 4e-4,or 5e-4 for a 64 | # batch size of 64 in the finetune. 65 | if optimizer == "adamw": 66 | tf.logging.info("using adamw") 67 | optimizer = AdamWeightDecayOptimizer( 68 | learning_rate=learning_rate, 69 | weight_decay_rate=0.01, 70 | beta_1=0.9, 71 | beta_2=0.999, 72 | epsilon=1e-6, 73 | exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"]) 74 | elif optimizer == "lamb": 75 | tf.logging.info("using lamb") 76 | optimizer = lamb_optimizer.LAMBOptimizer( 77 | learning_rate=learning_rate, 78 | weight_decay_rate=0.01, 79 | beta_1=0.9, 80 | beta_2=0.999, 81 | epsilon=1e-6, 82 | exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"]) 83 | else: 84 | raise ValueError("Not supported optimizer: ", optimizer) 85 | 86 | if use_tpu: 87 | optimizer = contrib_tpu.CrossShardOptimizer(optimizer) 88 | 89 | tvars = tf.trainable_variables() 90 | grads = tf.gradients(loss, tvars) 91 | 92 | # This is how the model was pre-trained. 93 | (grads, _) = tf.clip_by_global_norm(grads, clip_norm=1.0) 94 | 95 | train_op = optimizer.apply_gradients( 96 | list(zip(grads, tvars)), global_step=global_step) 97 | 98 | # Normally the global step update is done inside of `apply_gradients`. 99 | # However, neither `AdamWeightDecayOptimizer` nor `LAMBOptimizer` do this. 100 | # But if you use a different optimizer, you should probably take this line 101 | # out. 102 | new_global_step = global_step + 1 103 | train_op = tf.group(train_op, [global_step.assign(new_global_step)]) 104 | return train_op 105 | 106 | 107 | class AdamWeightDecayOptimizer(tf.train.Optimizer): 108 | """A basic Adam optimizer that includes "correct" L2 weight decay.""" 109 | 110 | def __init__(self, 111 | learning_rate, 112 | weight_decay_rate=0.0, 113 | beta_1=0.9, 114 | beta_2=0.999, 115 | epsilon=1e-6, 116 | exclude_from_weight_decay=None, 117 | name="AdamWeightDecayOptimizer"): 118 | """Constructs a AdamWeightDecayOptimizer.""" 119 | super(AdamWeightDecayOptimizer, self).__init__(False, name) 120 | 121 | self.learning_rate = learning_rate 122 | self.weight_decay_rate = weight_decay_rate 123 | self.beta_1 = beta_1 124 | self.beta_2 = beta_2 125 | self.epsilon = epsilon 126 | self.exclude_from_weight_decay = exclude_from_weight_decay 127 | 128 | def apply_gradients(self, grads_and_vars, global_step=None, name=None): 129 | """See base class.""" 130 | assignments = [] 131 | for (grad, param) in grads_and_vars: 132 | if grad is None or param is None: 133 | continue 134 | 135 | param_name = self._get_variable_name(param.name) 136 | 137 | m = tf.get_variable( 138 | name=six.ensure_str(param_name) + "/adam_m", 139 | shape=param.shape.as_list(), 140 | dtype=tf.float32, 141 | trainable=False, 142 | initializer=tf.zeros_initializer()) 143 | v = tf.get_variable( 144 | name=six.ensure_str(param_name) + "/adam_v", 145 | shape=param.shape.as_list(), 146 | dtype=tf.float32, 147 | trainable=False, 148 | initializer=tf.zeros_initializer()) 149 | 150 | # Standard Adam update. 151 | next_m = ( 152 | tf.multiply(self.beta_1, m) + tf.multiply(1.0 - self.beta_1, grad)) 153 | next_v = ( 154 | tf.multiply(self.beta_2, v) + tf.multiply(1.0 - self.beta_2, 155 | tf.square(grad))) 156 | 157 | update = next_m / (tf.sqrt(next_v) + self.epsilon) 158 | 159 | # Just adding the square of the weights to the loss function is *not* 160 | # the correct way of using L2 regularization/weight decay with Adam, 161 | # since that will interact with the m and v parameters in strange ways. 162 | # 163 | # Instead we want ot decay the weights in a manner that doesn't interact 164 | # with the m/v parameters. This is equivalent to adding the square 165 | # of the weights to the loss with plain (non-momentum) SGD. 166 | if self._do_use_weight_decay(param_name): 167 | update += self.weight_decay_rate * param 168 | 169 | update_with_lr = self.learning_rate * update 170 | 171 | next_param = param - update_with_lr 172 | 173 | assignments.extend( 174 | [param.assign(next_param), 175 | m.assign(next_m), 176 | v.assign(next_v)]) 177 | return tf.group(*assignments, name=name) 178 | 179 | def _do_use_weight_decay(self, param_name): 180 | """Whether to use L2 weight decay for `param_name`.""" 181 | if not self.weight_decay_rate: 182 | return False 183 | if self.exclude_from_weight_decay: 184 | for r in self.exclude_from_weight_decay: 185 | if re.search(r, param_name) is not None: 186 | return False 187 | return True 188 | 189 | def _get_variable_name(self, param_name): 190 | """Get the variable name from the tensor name.""" 191 | m = re.match("^(.*):\\d+$", six.ensure_str(param_name)) 192 | if m is not None: 193 | param_name = m.group(1) 194 | return param_name 195 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/predict.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu May 30 17:12:37 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import os 10 | import sys 11 | pwd = os.path.dirname(os.path.abspath(__file__)) 12 | sys.path.append(os.path.dirname(os.path.dirname(__file__))) 13 | import numpy as np 14 | import tensorflow as tf 15 | 16 | from classifier_multi_label_textcnn.networks import NetworkAlbertTextCNN 17 | from classifier_multi_label_textcnn.classifier_utils import id2label 18 | from classifier_multi_label_textcnn.classifier_utils import get_feature_test 19 | from classifier_multi_label_textcnn.hyperparameters import Hyperparamters as hp 20 | 21 | 22 | class ModelAlbertTextCNN(object,): 23 | """ 24 | Load NetworkAlbert TextCNN model 25 | """ 26 | def __init__(self): 27 | self.albert, self.sess = self.load_model() 28 | @staticmethod 29 | def load_model(): 30 | with tf.Graph().as_default(): 31 | sess = tf.Session() 32 | with sess.as_default(): 33 | albert = NetworkAlbertTextCNN(is_training=False) 34 | saver = tf.train.Saver() 35 | sess.run(tf.global_variables_initializer()) 36 | checkpoint_dir = os.path.abspath(os.path.join(pwd,hp.file_model_load)) 37 | print (checkpoint_dir) 38 | ckpt = tf.train.get_checkpoint_state(checkpoint_dir) 39 | saver.restore(sess, ckpt.model_checkpoint_path) 40 | return albert,sess 41 | 42 | MODEL = ModelAlbertTextCNN() 43 | print('Load model finished!') 44 | 45 | 46 | def get_label(sentence): 47 | """ 48 | Prediction of the sentence's label. 49 | """ 50 | feature = get_feature_test(sentence) 51 | fd = {MODEL.albert.input_ids: [feature[0]], 52 | MODEL.albert.input_masks: [feature[1]], 53 | MODEL.albert.segment_ids:[feature[2]], 54 | } 55 | prediction = MODEL.sess.run(MODEL.albert.predictions, feed_dict=fd)[0] 56 | return [id2label(l) for l in np.where(prediction==1)[0] if l!=0] 57 | 58 | 59 | 60 | if __name__ == '__main__': 61 | # Test 62 | sentences = ['耗电不是特别严重', 63 | '取暖效果：取暖效果好'] 64 | for sentence in sentences: 65 | print(sentence,get_label(sentence)) 66 | 67 | 68 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/train.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu May 30 21:42:07 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | 10 | import os 11 | #os.environ["CUDA_VISIBLE_DEVICES"] = '-1' 12 | import numpy as np 13 | import tensorflow as tf 14 | from classifier_multi_label_textcnn.networks import NetworkAlbertTextCNN 15 | from classifier_multi_label_textcnn.classifier_utils import get_features 16 | from classifier_multi_label_textcnn.hyperparameters import Hyperparamters as hp 17 | from classifier_multi_label_textcnn.utils import select,time_now_string 18 | 19 | 20 | pwd = os.path.dirname(os.path.abspath(__file__)) 21 | MODEL = NetworkAlbertTextCNN(is_training=True) 22 | 23 | 24 | # Get data features 25 | input_ids,input_masks,segment_ids,label_ids = get_features() 26 | num_train_samples = len(input_ids) 27 | indexs = np.arange(num_train_samples) 28 | num_batchs = int((num_train_samples - 1) /hp.batch_size) + 1 29 | 30 | # Set up the graph 31 | saver = tf.train.Saver(max_to_keep=hp.max_to_keep) 32 | sess = tf.Session() 33 | sess.run(tf.global_variables_initializer()) 34 | 35 | # Load model saved before 36 | MODEL_SAVE_PATH = os.path.join(pwd, hp.file_model_save) 37 | ckpt = tf.train.get_checkpoint_state(MODEL_SAVE_PATH) 38 | if ckpt and ckpt.model_checkpoint_path: 39 | saver.restore(sess, ckpt.model_checkpoint_path) 40 | print('Restored model!') 41 | 42 | 43 | with sess.as_default(): 44 | # Tensorboard writer 45 | writer = tf.summary.FileWriter(hp.logdir, sess.graph) 46 | for i in range(hp.num_train_epochs): 47 | np.random.shuffle(indexs) 48 | for j in range(num_batchs-1): 49 | # Get ids selected 50 | i1 = indexs[j * hp.batch_size:min((j + 1) * hp.batch_size, num_train_samples)] 51 | 52 | # Get features 53 | input_id_ = select(input_ids,i1) 54 | input_mask_ = select(input_masks,i1) 55 | segment_id_ = select(segment_ids,i1) 56 | label_id_ = select(label_ids,i1) 57 | 58 | # Feed dict 59 | fd = {MODEL.input_ids: input_id_, 60 | MODEL.input_masks: input_mask_, 61 | MODEL.segment_ids:segment_id_, 62 | MODEL.label_ids:label_id_} 63 | 64 | # Optimizer 65 | sess.run(MODEL.optimizer, feed_dict = fd) 66 | 67 | # Tensorboard 68 | if j%hp.summary_step==0: 69 | summary,glolal_step = sess.run([MODEL.merged,MODEL.global_step], feed_dict = fd) 70 | writer.add_summary(summary, glolal_step) 71 | 72 | # Save Model 73 | if j%(num_batchs//hp.num_saved_per_epoch)==0: 74 | if not os.path.exists(os.path.join(pwd, hp.file_model_save)): 75 | os.makedirs(os.path.join(pwd, hp.file_model_save)) 76 | saver.save(sess, os.path.join(pwd, hp.file_model_save, 'model'+'_%s_%s.ckpt'%(str(i),str(j)))) 77 | 78 | # Log 79 | if j % hp.print_step == 0: 80 | fd = {MODEL.input_ids: input_id_, 81 | MODEL.input_masks: input_mask_, 82 | MODEL.segment_ids:segment_id_, 83 | MODEL.label_ids:label_id_} 84 | accuracy,loss = sess.run([MODEL.accuracy,MODEL.loss], feed_dict = fd) 85 | print('Time:%s, Epoch:%s, Batch number:%s/%s, Loss:%s, Accuracy:%s'%(time_now_string(),str(i),str(j),str(num_batchs),str(loss),str(accuracy))) 86 | print('Train finished') 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | -------------------------------------------------------------------------------- /classifier_multi_label_textcnn/utils.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Fri May 25 23:43:39 2021 4 | 5 | @author: cm 6 | """ 7 | 8 | 9 | import time 10 | import numpy as np 11 | import pandas as pd 12 | 13 | 14 | def cut_list(data,size): 15 | """ 16 | data: a list 17 | size: the size of cut 18 | """ 19 | return [data[i * size:min((i + 1) * size, len(data))] for i in range(int(len(data)-1)//size + 1)] 20 | 21 | 22 | def time_now_string(): 23 | return time.strftime("%Y-%m-%d %H:%M:%S",time.localtime( time.time() )) 24 | 25 | 26 | def select(data,ids): 27 | return [data[i] for i in ids] 28 | 29 | 30 | def load_txt(file): 31 | with open(file,encoding='utf-8',errors='ignore') as fp: 32 | lines = fp.readlines() 33 | lines = [l.strip() for l in lines] 34 | print("Load data from file (%s) finished !"%file) 35 | return lines 36 | 37 | 38 | def save_txt(file,lines): 39 | lines = [l+'\n' for l in lines] 40 | with open(file,'w+',encoding='utf-8') as fp:#a+添加 41 | fp.writelines(lines) 42 | return "Write data to txt finished !" 43 | 44 | 45 | def load_csv(file,header=None): 46 | return pd.read_csv(file,encoding='utf-8',header=header,error_bad_lines=False)#,encoding='gbk' 47 | 48 | 49 | def save_csv(dataframe,file,header=True,index=None,encoding="gbk"): 50 | return dataframe.to_csv(file, 51 | mode='w+', 52 | header=header, 53 | index=index, 54 | encoding=encoding) 55 | 56 | 57 | 58 | def save_excel(dataframe,file,header=True,sheetname='Sheet1'): 59 | return dataframe.to_excel(file, 60 | header=header, 61 | sheet_name=sheetname) 62 | 63 | 64 | def load_excel(file,header=0): 65 | return pd.read_excel(file, 66 | header=header, 67 | ) 68 | 69 | def load_vocabulary(file_vocabulary_label): 70 | """ 71 | Load vocabulary to dict 72 | """ 73 | vocabulary = load_txt(file_vocabulary_label) 74 | dict_id2label,dict_label2id = {},{} 75 | for i,l in enumerate(vocabulary): 76 | dict_id2label[str(i)] = str(l) 77 | dict_label2id[str(l)] = str(i) 78 | return dict_id2label,dict_label2id 79 | 80 | 81 | def shuffle_two(a1,a2): 82 | """ 83 | Shuffle two list 84 | """ 85 | ran = np.arange(len(a1)) 86 | np.random.shuffle(ran) 87 | a1_ = [a1[l] for l in ran] 88 | a2_ = [a2[l] for l in ran] 89 | return a1_, a2_ 90 | 91 | 92 | if __name__ == '__main__': 93 | print('') 94 | 95 | 96 | 97 | 98 | -------------------------------------------------------------------------------- /evaluation/README.md: -------------------------------------------------------------------------------- 1 | # 评估函数 2 | - 详情参考：https://zhuanlan.zhihu.com/p/397934047 3 | -------------------------------------------------------------------------------- /evaluation/eval_multi_label.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Wed Nov 1 15:26:11 2023 4 | 5 | @author: Chen Ming 6 | """ 7 | 8 | 9 | class EvalMultiLabel(object): 10 | """ 评估多标签的精确率、召回率和F1值 """ 11 | 12 | def __init__(self, y_trues, y_preds): 13 | self.y_trues = y_trues 14 | self.y_preds = y_preds 15 | 16 | def some_samples(y_trues,y_preds): 17 | """ 评估多个样本的TP、FN、FP、TN """ 18 | if len(y_trues) == len(y_preds): 19 | tp = 0 20 | fn = 0 21 | fp = 0 22 | tn = 0 23 | for i in range(len(y_trues)): 24 | y_true = y_trues[i] 25 | y_pred = y_preds[i] 26 | tpi,fni,fpi,tni = single_sample(y_true,y_pred) 27 | tp = tp + tpi 28 | fn = fn + fni 29 | fp = fp + fpi 30 | tn = tn + tni 31 | return tp,fn,fp,tn 32 | else: 33 | print('Different length between y_trues and y_preds!') 34 | return 0,0,0,0 35 | 36 | def single_sample(y_true,y_pred): 37 | """ 评估单个样本的TP、FN、FP、TN """ 38 | y_true = list(set(y_true)) 39 | y_pred = list(set(y_pred)) 40 | y_ = list(set(y_true) | set(y_pred)) 41 | K = len(y_) 42 | tp1 = 0 43 | fn1 = 0 44 | fp1 = 0 45 | tn1 = 0 46 | for i in range(len(y_)): 47 | if y_[i] in y_true and y_[i] in y_pred: 48 | tp1 = tp1 + 1/K 49 | elif y_[i] in y_true and y_[i] not in y_pred: 50 | fn1 = fn1 + 1/K 51 | elif y_[i] not in y_true and y_[i] in y_pred: 52 | fp1 = fp1 + 1/K 53 | elif y_[i] not in y_true and y_[i] not in y_pred: 54 | tn1 = tn1 + 1/K 55 | return tp1,fn1,fp1,tn1 56 | 57 | self.tp,self.fn,self.fp,self.tn = some_samples(self.y_trues,self.y_preds) 58 | self.recall = self.tp/(self.tp+self.fn) 59 | self.precision = self.tp/(self.tp+self.fp) 60 | self.f1 = 2*self.recall*self.precision/(self.precision+self.recall) 61 | 62 | 63 | if __name__ == '__main__': 64 | y_trues = [['a','b','c'],['a','f','c','h']] #真实标签 65 | y_preds = [['a','d'],['a','d']] #预测标签 66 | EML = EvalMultiLabel(y_trues,y_preds) 67 | print(EML.precision) 68 | print(EML.recall) 69 | print(EML.f1) 70 | 71 | -------------------------------------------------------------------------------- /imgs/01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/01.png -------------------------------------------------------------------------------- /imgs/01b.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/01b.png -------------------------------------------------------------------------------- /imgs/02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/02.png -------------------------------------------------------------------------------- /imgs/09.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/09.png -------------------------------------------------------------------------------- /imgs/09b.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/09b.jpg -------------------------------------------------------------------------------- /imgs/10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/10.png -------------------------------------------------------------------------------- /imgs/HELLONLP.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/HELLONLP.png -------------------------------------------------------------------------------- /imgs/base.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/base.png -------------------------------------------------------------------------------- /imgs/denses.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/denses.jpg -------------------------------------------------------------------------------- /imgs/denses01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/denses01.png -------------------------------------------------------------------------------- /imgs/denses02.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/denses02.jpg -------------------------------------------------------------------------------- /imgs/denses02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/denses02.png -------------------------------------------------------------------------------- /imgs/seq2seq.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/seq2seq.png -------------------------------------------------------------------------------- /imgs/textcnn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hellonlp/classifier-multi-label/7beb1c60a1c3720b175c584f5cbea4bc0c427333/imgs/textcnn.png --------------------------------------------------------------------------------