├── README.md
├── datasets
    ├── ChnSentiCorp_htl_all
    │   ├── ChnSentiCorp_htl_all.csv
    │   └── intro.ipynb
    ├── anhuidianxinzhidao
    │   └── intro.ipynb
    ├── baoxianzhidao
    │   └── intro.ipynb
    ├── dh_msra
    │   ├── dh_msra.zip
    │   └── intro.ipynb
    ├── dmsc_v2
    │   └── intro.ipynb
    ├── ez_douban
    │   └── intro.ipynb
    ├── financezhidao
    │   └── intro.ipynb
    ├── lawzhidao
    │   └── intro.ipynb
    ├── liantongzhidao
    │   └── intro.ipynb
    ├── nonghangzhidao
    │   └── intro.ipynb
    ├── online_shopping_10_cats
    │   ├── intro.ipynb
    │   └── online_shopping_10_cats.zip
    ├── simplifyweibo_4_moods
    │   └── intro.ipynb
    ├── touzizhidao
    │   └── intro.ipynb
    ├── waimai_10k
    │   ├── intro.ipynb
    │   └── waimai_10k.csv
    ├── weibo_senti_100k
    │   └── intro.ipynb
    ├── yf_amazon
    │   └── intro.ipynb
    └── yf_dianping
    │   └── intro.ipynb
└── docs
    ├── images
        └── recruit
        │   ├── jd_header.png
        │   └── recruit_banner.png
    └── recruit
        ├── architect.md
        ├── engineer.md
        └── researcher.md


/README.md:
--------------------------------------------------------------------------------
 1 | # ChineseNlpCorpus
 2 | 搜集、整理、发布 中文 自然语言处理 语料/数据集，与 有志之士 共同 促进 中文 自然语言处理 的 发展。
 3 | 
 4 | ## 情感/观点/评论 倾向性分析
 5 | 
 6 | | 数据集 | 数据概览 | 下载地址 |
 7 | | ----- | -------- | ------- |
 8 | | ChnSentiCorp_htl_all | 7000 多条酒店评论数据，5000 多条正向评论，2000 多条负向评论 | [点击查看](./datasets/ChnSentiCorp_htl_all/intro.ipynb) |
 9 | | waimai_10k | 某外卖平台收集的用户评价，正向 4000 条，负向 约 8000 条 | [点击查看](./datasets/waimai_10k/intro.ipynb) |
10 | | online_shopping_10_cats | 10 个类别，共 6 万多条评论数据，正、负向评论各约 3 万条，<br /> 包括书籍、平板、手机、水果、洗发水、热水器、蒙牛、衣服、计算机、酒店 | [点击查看](./datasets/online_shopping_10_cats/intro.ipynb) |
11 | | weibo_senti_100k | 10 万多条，带情感标注 新浪微博，正负向评论约各 5 万条 | [点击查看](./datasets/weibo_senti_100k/intro.ipynb) |
12 | | simplifyweibo_4_moods | 36 万多条，带情感标注 新浪微博，包含 4 种情感，<br /> 其中喜悦约 20 万条，愤怒、厌恶、低落各约 5 万条 | [点击查看](./datasets/simplifyweibo_4_moods/intro.ipynb) |
13 | | dmsc_v2 | 28 部电影，超 70 万 用户，超 200 万条 评分/评论 数据 | [点击查看](./datasets/dmsc_v2/intro.ipynb) |
14 | | yf_dianping | 24 万家餐馆，54 万用户，440 万条评论/评分数据 | [点击查看](./datasets/yf_dianping/intro.ipynb) |
15 | | yf_amazon | 52 万件商品，1100 多个类目，142 万用户，720 万条评论/评分数据 | [点击查看](./datasets/yf_amazon/intro.ipynb) |
16 | 
17 | ## 中文命名实体识别
18 | 
19 | | 数据集 | 数据概览 | 下载地址 |
20 | | ----- | -------- | ------- |
21 | | dh_msra | 5 万多条中文命名实体识别标注数据（包括地点、机构、人物） | [点击查看](./datasets/dh_msra/intro.ipynb) |
22 | 
23 | ## 推荐系统
24 | 
25 | | 数据集 | 数据概览 | 下载地址 |
26 | | ----- | -------- | ------- |
27 | | ez_douban | 5 万多部电影（3 万多有电影名称，2 万多没有电影名称），2.8 万 用户，280 万条评分数据 | [点击查看](./datasets/ez_douban/intro.ipynb) |
28 | | dmsc_v2 | 28 部电影，超 70 万 用户，超 200 万条 评分/评论 数据 | [点击查看](./datasets/dmsc_v2/intro.ipynb) |
29 | | yf_dianping | 24 万家餐馆，54 万用户，440 万条评论/评分数据 | [点击查看](./datasets/yf_dianping/intro.ipynb) |
30 | | yf_amazon | 52 万件商品，1100 多个类目，142 万用户，720 万条评论/评分数据 | [点击查看](./datasets/yf_amazon/intro.ipynb) |
31 | 
32 | ## FAQ 问答系统
33 | 
34 | | 数据集 | 数据概览 | 下载地址 |
35 | | ----- | -------- | ------- |
36 | | 保险知道 | 8000 多条保险行业问答数据，包括用户提问、网友回答、最佳回答 | [点击查看](./datasets/baoxianzhidao/intro.ipynb) |
37 | | 安徽电信知道 | 15.6 万条电信问答数据，包括用户提问、网友回答、最佳回答 | [点击查看](./datasets/anhuidianxinzhidao/intro.ipynb) |
38 | | 金融知道 | 77 万条金融行业问答数据，包括用户提问、网友回答、最佳回答 | [点击查看](./datasets/financezhidao/intro.ipynb) |
39 | | 法律知道 | 3.6 万条法律问答数据，包括用户提问、网友回答、最佳回答 | [点击查看](./datasets/lawzhidao/intro.ipynb) |
40 | | 联通知道 | 20.3 万条联通问答数据，包括用户提问、网友回答、最佳回答 | [点击查看](./datasets/liantongzhidao/intro.ipynb) |
41 | | 农行知道 | 4 万条农业银行问答数据，包括用户提问、网友回答、最佳回答 | [点击查看](./datasets/nonghangzhidao/intro.ipynb) |
42 | | 保险知道 | 58.8 万条保险行业问答数据，包括用户提问、网友回答、最佳回答 | [点击查看](./datasets/baoxianzhidao/intro.ipynb) |
43 | 
44 | 
45 | 
46 | ## 加入我们
47 | 
48 | - 愿景：以人工智能产品和技术服务 30 亿人
49 | - 团队：极客精神、技术驱动，做有温度的技术，让世界更美好
50 | - 产品：面向细分行业领域的自动营销机器人，客户需求旺盛，产品前景无限
51 | - 职位：[自然语言人机交互应用研究](./docs/recruit/researcher.md)、[自然语言处理算法工程师](./docs/recruit/engineer.md)、[系统架构师（人工智能产品）](./docs/recruit/architect.md)
52 | 
53 | ![](./docs/images/recruit/recruit_banner.png)


--------------------------------------------------------------------------------
/datasets/ChnSentiCorp_htl_all/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# ChnSentiCorp_htl_all 说明\n",
  8 |     "0. **下载地址：** [Github](https://github.com/SophonPlus/ChineseNlpCorpus/raw/master/datasets/ChnSentiCorp_htl_all/ChnSentiCorp_htl_all.csv)\n",
  9 |     "1. **数据概览：** 7000 多条酒店评论数据，5000 多条正向评论，2000 多条负向评论\n",
 10 |     "2. **推荐实验：** 情感/观点/评论 倾向性分析\n",
 11 |     "2. **数据来源：**[携程网](http://www.ctrip.com/)\n",
 12 |     "3. **原数据集：** ChnSentiCorp_htl，由 [谭松波](http://people.ucas.ac.cn/~0012244) 老师整理的一份数据集\n",
 13 |     "4. **加工处理：**\n",
 14 |     "    1. 将原来 1 万个离散的文件整合到 1 个文件中\n",
 15 |     "    2. 将负向评论的 label 从 -1 改成 0\n",
 16 |     "    3. 去重"
 17 |    ]
 18 |   },
 19 |   {
 20 |    "cell_type": "code",
 21 |    "execution_count": 50,
 22 |    "metadata": {},
 23 |    "outputs": [],
 24 |    "source": [
 25 |     "import pandas as pd"
 26 |    ]
 27 |   },
 28 |   {
 29 |    "cell_type": "code",
 30 |    "execution_count": 51,
 31 |    "metadata": {},
 32 |    "outputs": [],
 33 |    "source": [
 34 |     "path = 'ChnSentiCorp_htl_all_文件夹_所在_路径'"
 35 |    ]
 36 |   },
 37 |   {
 38 |    "cell_type": "markdown",
 39 |    "metadata": {},
 40 |    "source": [
 41 |     "# 1. ChnSentiCorp_htl_all.csv"
 42 |    ]
 43 |   },
 44 |   {
 45 |    "cell_type": "markdown",
 46 |    "metadata": {},
 47 |    "source": [
 48 |     "## 加载数据"
 49 |    ]
 50 |   },
 51 |   {
 52 |    "cell_type": "code",
 53 |    "execution_count": 52,
 54 |    "metadata": {},
 55 |    "outputs": [
 56 |     {
 57 |      "name": "stdout",
 58 |      "output_type": "stream",
 59 |      "text": [
 60 |       "评论数目（总体）：7766\n",
 61 |       "评论数目（正向）：5322\n",
 62 |       "评论数目（负向）：2444\n"
 63 |      ]
 64 |     }
 65 |    ],
 66 |    "source": [
 67 |     "pd_all = pd.read_csv(path + 'ChnSentiCorp_htl_all.csv')\n",
 68 |     "\n",
 69 |     "print('评论数目（总体）：%d' % pd_all.shape[0])\n",
 70 |     "print('评论数目（正向）：%d' % pd_all[pd_all.label==1].shape[0])\n",
 71 |     "print('评论数目（负向）：%d' % pd_all[pd_all.label==0].shape[0])"
 72 |    ]
 73 |   },
 74 |   {
 75 |    "cell_type": "markdown",
 76 |    "metadata": {},
 77 |    "source": [
 78 |     "## 字段说明\n",
 79 |     "\n",
 80 |     "| 字段 | 说明 |\n",
 81 |     "| ---- | ---- |\n",
 82 |     "| label | 1 表示正向评论，0 表示负向评论 |\n",
 83 |     "| review | 评论内容 |"
 84 |    ]
 85 |   },
 86 |   {
 87 |    "cell_type": "code",
 88 |    "execution_count": 53,
 89 |    "metadata": {},
 90 |    "outputs": [
 91 |     {
 92 |      "data": {
 93 |       "text/html": [
 94 |        "<div>\n",
 95 |        "<style scoped>\n",
 96 |        "    .dataframe tbody tr th:only-of-type {\n",
 97 |        "        vertical-align: middle;\n",
 98 |        "    }\n",
 99 |        "\n",
100 |        "    .dataframe tbody tr th {\n",
101 |        "        vertical-align: top;\n",
102 |        "    }\n",
103 |        "\n",
104 |        "    .dataframe thead th {\n",
105 |        "        text-align: right;\n",
106 |        "    }\n",
107 |        "</style>\n",
108 |        "<table border=\"1\" class=\"dataframe\">\n",
109 |        "  <thead>\n",
110 |        "    <tr style=\"text-align: right;\">\n",
111 |        "      <th></th>\n",
112 |        "      <th>label</th>\n",
113 |        "      <th>review</th>\n",
114 |        "    </tr>\n",
115 |        "  </thead>\n",
116 |        "  <tbody>\n",
117 |        "    <tr>\n",
118 |        "      <th>5612</th>\n",
119 |        "      <td>0</td>\n",
120 |        "      <td>房间小得无法想象,建议个子大的不要选择,一般的睡觉脚也伸不直.房间不超过10平方,彩电是14...</td>\n",
121 |        "    </tr>\n",
122 |        "    <tr>\n",
123 |        "      <th>7321</th>\n",
124 |        "      <td>0</td>\n",
125 |        "      <td>我们一家人带孩子去过“五.一”，在协程网上挑了半天才选中的酒店，但看来还是错了。1.酒店除了...</td>\n",
126 |        "    </tr>\n",
127 |        "    <tr>\n",
128 |        "      <th>3870</th>\n",
129 |        "      <td>1</td>\n",
130 |        "      <td>周六到西山去采橘子,路过这家酒店的时候就觉得应该不错的,采好橘子回来天也晚了,就临时决定住在...</td>\n",
131 |        "    </tr>\n",
132 |        "    <tr>\n",
133 |        "      <th>4057</th>\n",
134 |        "      <td>1</td>\n",
135 |        "      <td>交通很便利,到渔人码头和港澳码头都在步行的范围之内.CHECKIN和CHECKOUT的速度都...</td>\n",
136 |        "    </tr>\n",
137 |        "    <tr>\n",
138 |        "      <th>1452</th>\n",
139 |        "      <td>1</td>\n",
140 |        "      <td>很不错的一个酒店,床很大,很舒服.酒店员工的服务态度很亲切.</td>\n",
141 |        "    </tr>\n",
142 |        "    <tr>\n",
143 |        "      <th>4805</th>\n",
144 |        "      <td>1</td>\n",
145 |        "      <td>酒店环境和服务都还不错，地理位置也不错，尤其是酒店北面的川北凉粉确实好吃，不过就是隔音效果不...</td>\n",
146 |        "    </tr>\n",
147 |        "    <tr>\n",
148 |        "      <th>6868</th>\n",
149 |        "      <td>0</td>\n",
150 |        "      <td>旧楼改建的酒店，期望不要太高。酒店经理的态度很好，会帮助解决问题。有一位前台小姐的态度实在是...</td>\n",
151 |        "    </tr>\n",
152 |        "    <tr>\n",
153 |        "      <th>1345</th>\n",
154 |        "      <td>1</td>\n",
155 |        "      <td>经常去海口出差,但从没住过该酒店.看外表感觉一般吧其实酒店里面还真不错,房间是新装修的(我住...</td>\n",
156 |        "    </tr>\n",
157 |        "    <tr>\n",
158 |        "      <th>2026</th>\n",
159 |        "      <td>1</td>\n",
160 |        "      <td>算是海口市比较好的酒店了。处于市中心，购物方便。服务态度好。保险柜出问题了叫人来开，打个电话...</td>\n",
161 |        "    </tr>\n",
162 |        "    <tr>\n",
163 |        "      <th>2805</th>\n",
164 |        "      <td>1</td>\n",
165 |        "      <td>感受的是热情的服务！从入门开始，一直很愉快！房间硬件只是准2星的吧，卫生间淋浴头在马桶上方，...</td>\n",
166 |        "    </tr>\n",
167 |        "    <tr>\n",
168 |        "      <th>2915</th>\n",
169 |        "      <td>1</td>\n",
170 |        "      <td>房间很整洁，尤其是床上的哪个靠枕是我以前所住过宾馆没有的，红色的很喜庆。虽然是在当地比较繁华...</td>\n",
171 |        "    </tr>\n",
172 |        "    <tr>\n",
173 |        "      <th>1803</th>\n",
174 |        "      <td>1</td>\n",
175 |        "      <td>准确的说，酒店的环境很漂亮，房间设施也还行，可以算4星标准。但是，卫生间下水道的气味实在是让...</td>\n",
176 |        "    </tr>\n",
177 |        "    <tr>\n",
178 |        "      <th>4729</th>\n",
179 |        "      <td>1</td>\n",
180 |        "      <td>价格越来越高了,周遍不方便,去哪里都需要打车.不过装修风格很时尚舒适.服务态度不错.</td>\n",
181 |        "    </tr>\n",
182 |        "    <tr>\n",
183 |        "      <th>1913</th>\n",
184 |        "      <td>1</td>\n",
185 |        "      <td>地理位置不错。但好像人气不太旺。不过下次也会考虑住这的。</td>\n",
186 |        "    </tr>\n",
187 |        "    <tr>\n",
188 |        "      <th>7159</th>\n",
189 |        "      <td>0</td>\n",
190 |        "      <td>设施老化，紧靠马路噪音太大。晚上楼上卫生间的水流声和空调噪音非常大，无法入眠，跟总台反映后，...</td>\n",
191 |        "    </tr>\n",
192 |        "    <tr>\n",
193 |        "      <th>1119</th>\n",
194 |        "      <td>1</td>\n",
195 |        "      <td>11月份住了一次。1.服务方面还不错，门童挺积极。2.感觉房间略有陈旧。3.早餐品种还算丰富...</td>\n",
196 |        "    </tr>\n",
197 |        "    <tr>\n",
198 |        "      <th>2170</th>\n",
199 |        "      <td>1</td>\n",
200 |        "      <td>总的来说，酒店还不错。比较安静，地理位置比较好，服务也不错，包括入住和结账。不太好的地方，7...</td>\n",
201 |        "    </tr>\n",
202 |        "    <tr>\n",
203 |        "      <th>2793</th>\n",
204 |        "      <td>1</td>\n",
205 |        "      <td>我喜欢那里,性价比很高地.去太原90%都住在那里的.服务员的服务很不错</td>\n",
206 |        "    </tr>\n",
207 |        "    <tr>\n",
208 |        "      <th>5895</th>\n",
209 |        "      <td>0</td>\n",
210 |        "      <td>非常糟糕！1。我们通过其商务中心包了一辆车游西湖，该车拉我们去不正规景点买茶叶（我们买了），...</td>\n",
211 |        "    </tr>\n",
212 |        "    <tr>\n",
213 |        "      <th>4089</th>\n",
214 |        "      <td>1</td>\n",
215 |        "      <td>我是7月9号晚10点多的时候入住的，房间很新，据说是跟格林豪泰是同一公司的，可能是是新开业的...</td>\n",
216 |        "    </tr>\n",
217 |        "  </tbody>\n",
218 |        "</table>\n",
219 |        "</div>"
220 |       ],
221 |       "text/plain": [
222 |        "      label                                             review\n",
223 |        "5612      0  房间小得无法想象,建议个子大的不要选择,一般的睡觉脚也伸不直.房间不超过10平方,彩电是14...\n",
224 |        "7321      0  我们一家人带孩子去过“五.一”，在协程网上挑了半天才选中的酒店，但看来还是错了。1.酒店除了...\n",
225 |        "3870      1  周六到西山去采橘子,路过这家酒店的时候就觉得应该不错的,采好橘子回来天也晚了,就临时决定住在...\n",
226 |        "4057      1  交通很便利,到渔人码头和港澳码头都在步行的范围之内.CHECKIN和CHECKOUT的速度都...\n",
227 |        "1452      1                     很不错的一个酒店,床很大,很舒服.酒店员工的服务态度很亲切.\n",
228 |        "4805      1  酒店环境和服务都还不错，地理位置也不错，尤其是酒店北面的川北凉粉确实好吃，不过就是隔音效果不...\n",
229 |        "6868      0  旧楼改建的酒店，期望不要太高。酒店经理的态度很好，会帮助解决问题。有一位前台小姐的态度实在是...\n",
230 |        "1345      1  经常去海口出差,但从没住过该酒店.看外表感觉一般吧其实酒店里面还真不错,房间是新装修的(我住...\n",
231 |        "2026      1  算是海口市比较好的酒店了。处于市中心，购物方便。服务态度好。保险柜出问题了叫人来开，打个电话...\n",
232 |        "2805      1  感受的是热情的服务！从入门开始，一直很愉快！房间硬件只是准2星的吧，卫生间淋浴头在马桶上方，...\n",
233 |        "2915      1  房间很整洁，尤其是床上的哪个靠枕是我以前所住过宾馆没有的，红色的很喜庆。虽然是在当地比较繁华...\n",
234 |        "1803      1  准确的说，酒店的环境很漂亮，房间设施也还行，可以算4星标准。但是，卫生间下水道的气味实在是让...\n",
235 |        "4729      1         价格越来越高了,周遍不方便,去哪里都需要打车.不过装修风格很时尚舒适.服务态度不错.\n",
236 |        "1913      1                       地理位置不错。但好像人气不太旺。不过下次也会考虑住这的。\n",
237 |        "7159      0  设施老化，紧靠马路噪音太大。晚上楼上卫生间的水流声和空调噪音非常大，无法入眠，跟总台反映后，...\n",
238 |        "1119      1  11月份住了一次。1.服务方面还不错，门童挺积极。2.感觉房间略有陈旧。3.早餐品种还算丰富...\n",
239 |        "2170      1  总的来说，酒店还不错。比较安静，地理位置比较好，服务也不错，包括入住和结账。不太好的地方，7...\n",
240 |        "2793      1                我喜欢那里,性价比很高地.去太原90%都住在那里的.服务员的服务很不错\n",
241 |        "5895      0  非常糟糕！1。我们通过其商务中心包了一辆车游西湖，该车拉我们去不正规景点买茶叶（我们买了），...\n",
242 |        "4089      1  我是7月9号晚10点多的时候入住的，房间很新，据说是跟格林豪泰是同一公司的，可能是是新开业的..."
243 |       ]
244 |      },
245 |      "execution_count": 53,
246 |      "metadata": {},
247 |      "output_type": "execute_result"
248 |     }
249 |    ],
250 |    "source": [
251 |     "pd_all.sample(20)"
252 |    ]
253 |   },
254 |   {
255 |    "cell_type": "markdown",
256 |    "metadata": {},
257 |    "source": [
258 |     "# 2. 构造平衡语料\n",
259 |     "\n",
260 |     "- 原数据集中还包含了 3 份平衡语料：ChnSentiCorp_htl_ba_2000, ChnSentiCorp_htl_ba_4000, ChnSentiCorp_htl_ba_6000\n",
261 |     "- 用随机抽样的方法，很容易构造出类似的平衡语料"
262 |    ]
263 |   },
264 |   {
265 |    "cell_type": "code",
266 |    "execution_count": 54,
267 |    "metadata": {},
268 |    "outputs": [],
269 |    "source": [
270 |     "pd_positive = pd_all[pd_all.label==1]\n",
271 |     "pd_negative = pd_all[pd_all.label==0]\n",
272 |     "\n",
273 |     "def get_balance_corpus(corpus_size, corpus_pos, corpus_neg):\n",
274 |     "    sample_size = corpus_size // 2\n",
275 |     "    pd_corpus_balance = pd.concat([corpus_pos.sample(sample_size, replace=corpus_pos.shape[0]<sample_size), \\\n",
276 |     "                                   corpus_neg.sample(sample_size, replace=corpus_neg.shape[0]<sample_size)])\n",
277 |     "    \n",
278 |     "    print('评论数目（总体）：%d' % pd_corpus_balance.shape[0])\n",
279 |     "    print('评论数目（正向）：%d' % pd_corpus_balance[pd_corpus_balance.label==1].shape[0])\n",
280 |     "    print('评论数目（负向）：%d' % pd_corpus_balance[pd_corpus_balance.label==0].shape[0])    \n",
281 |     "    \n",
282 |     "    return pd_corpus_balance"
283 |    ]
284 |   },
285 |   {
286 |    "cell_type": "code",
287 |    "execution_count": 55,
288 |    "metadata": {},
289 |    "outputs": [
290 |     {
291 |      "name": "stdout",
292 |      "output_type": "stream",
293 |      "text": [
294 |       "评论数目（总体）：2000\n",
295 |       "评论数目（正向）：1000\n",
296 |       "评论数目（负向）：1000\n"
297 |      ]
298 |     },
299 |     {
300 |      "data": {
301 |       "text/html": [
302 |        "<div>\n",
303 |        "<style scoped>\n",
304 |        "    .dataframe tbody tr th:only-of-type {\n",
305 |        "        vertical-align: middle;\n",
306 |        "    }\n",
307 |        "\n",
308 |        "    .dataframe tbody tr th {\n",
309 |        "        vertical-align: top;\n",
310 |        "    }\n",
311 |        "\n",
312 |        "    .dataframe thead th {\n",
313 |        "        text-align: right;\n",
314 |        "    }\n",
315 |        "</style>\n",
316 |        "<table border=\"1\" class=\"dataframe\">\n",
317 |        "  <thead>\n",
318 |        "    <tr style=\"text-align: right;\">\n",
319 |        "      <th></th>\n",
320 |        "      <th>label</th>\n",
321 |        "      <th>review</th>\n",
322 |        "    </tr>\n",
323 |        "  </thead>\n",
324 |        "  <tbody>\n",
325 |        "    <tr>\n",
326 |        "      <th>5536</th>\n",
327 |        "      <td>0</td>\n",
328 |        "      <td>建议携程不要和这家酒店合作,名曰三星,要我看准星级都勉强!首先不在市区里面(去涵江区打车还要...</td>\n",
329 |        "    </tr>\n",
330 |        "    <tr>\n",
331 |        "      <th>4086</th>\n",
332 |        "      <td>1</td>\n",
333 |        "      <td>感觉比老街口客栈舒适，很中规中矩的3星级，推荐大家住主楼的豪华间，设施比较好，前台和大堂的服...</td>\n",
334 |        "    </tr>\n",
335 |        "    <tr>\n",
336 |        "      <th>6112</th>\n",
337 |        "      <td>0</td>\n",
338 |        "      <td>是我遇到的最差的4星酒店，进门没人管，进去要我和大堂打招呼，退房也很慢，不会再去住了</td>\n",
339 |        "    </tr>\n",
340 |        "    <tr>\n",
341 |        "      <th>4440</th>\n",
342 |        "      <td>1</td>\n",
343 |        "      <td>房间的设施不错，由于武夷山市是个小地方，酒店离景区有一定距离，如果没有自己开车就不太方便，但...</td>\n",
344 |        "    </tr>\n",
345 |        "    <tr>\n",
346 |        "      <th>2706</th>\n",
347 |        "      <td>1</td>\n",
348 |        "      <td>首次入住该酒店,环境雅致,服务非常不错,很多笑脸,感觉热情,早餐可以接受,有送餐服务以后去徐...</td>\n",
349 |        "    </tr>\n",
350 |        "    <tr>\n",
351 |        "      <th>1770</th>\n",
352 |        "      <td>1</td>\n",
353 |        "      <td>不错!就是洗澡的地方小点~~下回去还住这家~~</td>\n",
354 |        "    </tr>\n",
355 |        "    <tr>\n",
356 |        "      <th>4306</th>\n",
357 |        "      <td>1</td>\n",
358 |        "      <td>环境位置很好,房间情况尚可,早餐一般般,价格偏高了一些.</td>\n",
359 |        "    </tr>\n",
360 |        "    <tr>\n",
361 |        "      <th>2161</th>\n",
362 |        "      <td>1</td>\n",
363 |        "      <td>位置优越，出行方便。就是房间较小，床位较小，房间装修较旧，其他方面都不错。</td>\n",
364 |        "    </tr>\n",
365 |        "    <tr>\n",
366 |        "      <th>7667</th>\n",
367 |        "      <td>0</td>\n",
368 |        "      <td>酒店周围环境差，内部也很旧，卫生不好，很脏，总之没什么好的，下次决不住这。</td>\n",
369 |        "    </tr>\n",
370 |        "    <tr>\n",
371 |        "      <th>4419</th>\n",
372 |        "      <td>1</td>\n",
373 |        "      <td>我7月24号入住瑞豪酒店，开始有些不顺利，但是那里的管理还是非常好的，有位姓赵的经理发现问题...</td>\n",
374 |        "    </tr>\n",
375 |        "  </tbody>\n",
376 |        "</table>\n",
377 |        "</div>"
378 |       ],
379 |       "text/plain": [
380 |        "      label                                             review\n",
381 |        "5536      0  建议携程不要和这家酒店合作,名曰三星,要我看准星级都勉强!首先不在市区里面(去涵江区打车还要...\n",
382 |        "4086      1  感觉比老街口客栈舒适，很中规中矩的3星级，推荐大家住主楼的豪华间，设施比较好，前台和大堂的服...\n",
383 |        "6112      0         是我遇到的最差的4星酒店，进门没人管，进去要我和大堂打招呼，退房也很慢，不会再去住了\n",
384 |        "4440      1  房间的设施不错，由于武夷山市是个小地方，酒店离景区有一定距离，如果没有自己开车就不太方便，但...\n",
385 |        "2706      1  首次入住该酒店,环境雅致,服务非常不错,很多笑脸,感觉热情,早餐可以接受,有送餐服务以后去徐...\n",
386 |        "1770      1                            不错!就是洗澡的地方小点~~下回去还住这家~~\n",
387 |        "4306      1                       环境位置很好,房间情况尚可,早餐一般般,价格偏高了一些.\n",
388 |        "2161      1              位置优越，出行方便。就是房间较小，床位较小，房间装修较旧，其他方面都不错。\n",
389 |        "7667      0              酒店周围环境差，内部也很旧，卫生不好，很脏，总之没什么好的，下次决不住这。\n",
390 |        "4419      1  我7月24号入住瑞豪酒店，开始有些不顺利，但是那里的管理还是非常好的，有位姓赵的经理发现问题..."
391 |       ]
392 |      },
393 |      "execution_count": 55,
394 |      "metadata": {},
395 |      "output_type": "execute_result"
396 |     }
397 |    ],
398 |    "source": [
399 |     "ChnSentiCorp_htl_ba_2000 = get_balance_corpus(2000, pd_positive, pd_negative)\n",
400 |     "\n",
401 |     "ChnSentiCorp_htl_ba_2000.sample(10)"
402 |    ]
403 |   },
404 |   {
405 |    "cell_type": "code",
406 |    "execution_count": 56,
407 |    "metadata": {},
408 |    "outputs": [
409 |     {
410 |      "name": "stdout",
411 |      "output_type": "stream",
412 |      "text": [
413 |       "评论数目（总体）：4000\n",
414 |       "评论数目（正向）：2000\n",
415 |       "评论数目（负向）：2000\n"
416 |      ]
417 |     },
418 |     {
419 |      "data": {
420 |       "text/html": [
421 |        "<div>\n",
422 |        "<style scoped>\n",
423 |        "    .dataframe tbody tr th:only-of-type {\n",
424 |        "        vertical-align: middle;\n",
425 |        "    }\n",
426 |        "\n",
427 |        "    .dataframe tbody tr th {\n",
428 |        "        vertical-align: top;\n",
429 |        "    }\n",
430 |        "\n",
431 |        "    .dataframe thead th {\n",
432 |        "        text-align: right;\n",
433 |        "    }\n",
434 |        "</style>\n",
435 |        "<table border=\"1\" class=\"dataframe\">\n",
436 |        "  <thead>\n",
437 |        "    <tr style=\"text-align: right;\">\n",
438 |        "      <th></th>\n",
439 |        "      <th>label</th>\n",
440 |        "      <th>review</th>\n",
441 |        "    </tr>\n",
442 |        "  </thead>\n",
443 |        "  <tbody>\n",
444 |        "    <tr>\n",
445 |        "      <th>3605</th>\n",
446 |        "      <td>1</td>\n",
447 |        "      <td>酒店就在海水浴场旁边，出门到接触到海水两分钟，如果要和海水亲近的朋友，极力推荐。这样游泳换衣...</td>\n",
448 |        "    </tr>\n",
449 |        "    <tr>\n",
450 |        "      <th>7260</th>\n",
451 |        "      <td>0</td>\n",
452 |        "      <td>TheWorsehotelinChengdurightnow,checkoutat12.30...</td>\n",
453 |        "    </tr>\n",
454 |        "    <tr>\n",
455 |        "      <th>5762</th>\n",
456 |        "      <td>0</td>\n",
457 |        "      <td>房间还算可以，不过前台服务人员的态度，受不了，我晚上11点多到酒店CHEKIN第二天退房的时...</td>\n",
458 |        "    </tr>\n",
459 |        "    <tr>\n",
460 |        "      <th>5790</th>\n",
461 |        "      <td>0</td>\n",
462 |        "      <td>酒店设施陈旧，浴缸排水不畅，入住无房，一间16：00，一间22：00，早餐差</td>\n",
463 |        "    </tr>\n",
464 |        "    <tr>\n",
465 |        "      <th>4504</th>\n",
466 |        "      <td>1</td>\n",
467 |        "      <td>虽是公寓式酒店，但其房间整洁程度、全方位的服务都给我留下了很好的印象。丝丝不完善之处在于很多...</td>\n",
468 |        "    </tr>\n",
469 |        "    <tr>\n",
470 |        "      <th>5246</th>\n",
471 |        "      <td>1</td>\n",
472 |        "      <td>很好的酒店，很喜欢，房间很干净很漂亮，从房间的窗口看出去，超美的，在市中心区域，出行也非常的...</td>\n",
473 |        "    </tr>\n",
474 |        "    <tr>\n",
475 |        "      <th>624</th>\n",
476 |        "      <td>1</td>\n",
477 |        "      <td>在临沂，这个酒店算是比较有档次的了，给外国客人的服务也比较合格。可惜电视内容比较单调，国外的...</td>\n",
478 |        "    </tr>\n",
479 |        "    <tr>\n",
480 |        "      <th>1382</th>\n",
481 |        "      <td>1</td>\n",
482 |        "      <td>4年前住过，我和德国同事都觉得很不错。今年我又选了豪门，还是觉得很好。自助餐品种丰富，房间宽...</td>\n",
483 |        "    </tr>\n",
484 |        "    <tr>\n",
485 |        "      <th>3723</th>\n",
486 |        "      <td>1</td>\n",
487 |        "      <td>价格不高,比较实惠,服务也不错,离闹市区不远.交通也比较方便.</td>\n",
488 |        "    </tr>\n",
489 |        "    <tr>\n",
490 |        "      <th>3328</th>\n",
491 |        "      <td>1</td>\n",
492 |        "      <td>房间：建筑风格比较独特。木屋矗立在随潮汐涨落的水中，围廊象迷宫一样。看着自己的小屋，却没有直...</td>\n",
493 |        "    </tr>\n",
494 |        "  </tbody>\n",
495 |        "</table>\n",
496 |        "</div>"
497 |       ],
498 |       "text/plain": [
499 |        "      label                                             review\n",
500 |        "3605      1  酒店就在海水浴场旁边，出门到接触到海水两分钟，如果要和海水亲近的朋友，极力推荐。这样游泳换衣...\n",
501 |        "7260      0  TheWorsehotelinChengdurightnow,checkoutat12.30...\n",
502 |        "5762      0  房间还算可以，不过前台服务人员的态度，受不了，我晚上11点多到酒店CHEKIN第二天退房的时...\n",
503 |        "5790      0             酒店设施陈旧，浴缸排水不畅，入住无房，一间16：00，一间22：00，早餐差\n",
504 |        "4504      1  虽是公寓式酒店，但其房间整洁程度、全方位的服务都给我留下了很好的印象。丝丝不完善之处在于很多...\n",
505 |        "5246      1  很好的酒店，很喜欢，房间很干净很漂亮，从房间的窗口看出去，超美的，在市中心区域，出行也非常的...\n",
506 |        "624       1  在临沂，这个酒店算是比较有档次的了，给外国客人的服务也比较合格。可惜电视内容比较单调，国外的...\n",
507 |        "1382      1  4年前住过，我和德国同事都觉得很不错。今年我又选了豪门，还是觉得很好。自助餐品种丰富，房间宽...\n",
508 |        "3723      1                    价格不高,比较实惠,服务也不错,离闹市区不远.交通也比较方便.\n",
509 |        "3328      1  房间：建筑风格比较独特。木屋矗立在随潮汐涨落的水中，围廊象迷宫一样。看着自己的小屋，却没有直..."
510 |       ]
511 |      },
512 |      "execution_count": 56,
513 |      "metadata": {},
514 |      "output_type": "execute_result"
515 |     }
516 |    ],
517 |    "source": [
518 |     "ChnSentiCorp_htl_ba_4000 = get_balance_corpus(4000, pd_positive, pd_negative)\n",
519 |     "\n",
520 |     "ChnSentiCorp_htl_ba_4000.sample(10)"
521 |    ]
522 |   },
523 |   {
524 |    "cell_type": "code",
525 |    "execution_count": 57,
526 |    "metadata": {},
527 |    "outputs": [
528 |     {
529 |      "name": "stdout",
530 |      "output_type": "stream",
531 |      "text": [
532 |       "评论数目（总体）：6000\n",
533 |       "评论数目（正向）：3000\n",
534 |       "评论数目（负向）：3000\n"
535 |      ]
536 |     },
537 |     {
538 |      "data": {
539 |       "text/html": [
540 |        "<div>\n",
541 |        "<style scoped>\n",
542 |        "    .dataframe tbody tr th:only-of-type {\n",
543 |        "        vertical-align: middle;\n",
544 |        "    }\n",
545 |        "\n",
546 |        "    .dataframe tbody tr th {\n",
547 |        "        vertical-align: top;\n",
548 |        "    }\n",
549 |        "\n",
550 |        "    .dataframe thead th {\n",
551 |        "        text-align: right;\n",
552 |        "    }\n",
553 |        "</style>\n",
554 |        "<table border=\"1\" class=\"dataframe\">\n",
555 |        "  <thead>\n",
556 |        "    <tr style=\"text-align: right;\">\n",
557 |        "      <th></th>\n",
558 |        "      <th>label</th>\n",
559 |        "      <th>review</th>\n",
560 |        "    </tr>\n",
561 |        "  </thead>\n",
562 |        "  <tbody>\n",
563 |        "    <tr>\n",
564 |        "      <th>4817</th>\n",
565 |        "      <td>1</td>\n",
566 |        "      <td>入住的是260元的迷你标准间。感觉比想象的要好很多，房间如果住一个人很合适的，洗手间很大，很...</td>\n",
567 |        "    </tr>\n",
568 |        "    <tr>\n",
569 |        "      <th>7021</th>\n",
570 |        "      <td>0</td>\n",
571 |        "      <td>7点到了酒店前台打电话问了楼层说房间可以入住，上楼竟然房间的垃圾成堆根本就没有打扫，下楼要求...</td>\n",
572 |        "    </tr>\n",
573 |        "    <tr>\n",
574 |        "      <th>6484</th>\n",
575 |        "      <td>0</td>\n",
576 |        "      <td>又要对他进行点评了，呜呜。。。说什么好呢</td>\n",
577 |        "    </tr>\n",
578 |        "    <tr>\n",
579 |        "      <th>6715</th>\n",
580 |        "      <td>0</td>\n",
581 |        "      <td>看了前面介绍的推荐去入住的，结果很失望，酒店的淋浴居然没有维护设施，洗个澡弄得整个洗手间都淋...</td>\n",
582 |        "    </tr>\n",
583 |        "    <tr>\n",
584 |        "      <th>6775</th>\n",
585 |        "      <td>0</td>\n",
586 |        "      <td>酒店的设施太差了，估计连1星级都没有，房间空调都不开的，简直就是一塌糊涂。建议大家不要去预订该酒店</td>\n",
587 |        "    </tr>\n",
588 |        "    <tr>\n",
589 |        "      <th>7575</th>\n",
590 |        "      <td>0</td>\n",
591 |        "      <td>真的差得没话说，但说起来又有一堆。住进去的时候发现没有浴巾，第二天却一直打电话说我们拿了那两...</td>\n",
592 |        "    </tr>\n",
593 |        "    <tr>\n",
594 |        "      <th>1615</th>\n",
595 |        "      <td>1</td>\n",
596 |        "      <td>酒店非常好，距离高速出口很近，服务也很到位，值得推荐的酒店，到泰山应该是最好的酒店了．</td>\n",
597 |        "    </tr>\n",
598 |        "    <tr>\n",
599 |        "      <th>6466</th>\n",
600 |        "      <td>0</td>\n",
601 |        "      <td>携城预定员极力推荐这家酒店，相信她才入住了这家，结果到了酒店才发现，连一星级都不如，前台的小...</td>\n",
602 |        "    </tr>\n",
603 |        "    <tr>\n",
604 |        "      <th>1392</th>\n",
605 |        "      <td>1</td>\n",
606 |        "      <td>酒店很大，服务太差，Ａ楼房间也老，下次再也不住了。环境很好，打高尔夫的或许可以忍忍吧。</td>\n",
607 |        "    </tr>\n",
608 |        "    <tr>\n",
609 |        "      <th>4408</th>\n",
610 |        "      <td>1</td>\n",
611 |        "      <td>房间很大，大的让我去其他宾馆都感觉性价比不高！服务也不错，值得一住！！</td>\n",
612 |        "    </tr>\n",
613 |        "  </tbody>\n",
614 |        "</table>\n",
615 |        "</div>"
616 |       ],
617 |       "text/plain": [
618 |        "      label                                             review\n",
619 |        "4817      1  入住的是260元的迷你标准间。感觉比想象的要好很多，房间如果住一个人很合适的，洗手间很大，很...\n",
620 |        "7021      0  7点到了酒店前台打电话问了楼层说房间可以入住，上楼竟然房间的垃圾成堆根本就没有打扫，下楼要求...\n",
621 |        "6484      0                               又要对他进行点评了，呜呜。。。说什么好呢\n",
622 |        "6715      0  看了前面介绍的推荐去入住的，结果很失望，酒店的淋浴居然没有维护设施，洗个澡弄得整个洗手间都淋...\n",
623 |        "6775      0  酒店的设施太差了，估计连1星级都没有，房间空调都不开的，简直就是一塌糊涂。建议大家不要去预订该酒店\n",
624 |        "7575      0  真的差得没话说，但说起来又有一堆。住进去的时候发现没有浴巾，第二天却一直打电话说我们拿了那两...\n",
625 |        "1615      1        酒店非常好，距离高速出口很近，服务也很到位，值得推荐的酒店，到泰山应该是最好的酒店了．\n",
626 |        "6466      0  携城预定员极力推荐这家酒店，相信她才入住了这家，结果到了酒店才发现，连一星级都不如，前台的小...\n",
627 |        "1392      1        酒店很大，服务太差，Ａ楼房间也老，下次再也不住了。环境很好，打高尔夫的或许可以忍忍吧。\n",
628 |        "4408      1                房间很大，大的让我去其他宾馆都感觉性价比不高！服务也不错，值得一住！！"
629 |       ]
630 |      },
631 |      "execution_count": 57,
632 |      "metadata": {},
633 |      "output_type": "execute_result"
634 |     }
635 |    ],
636 |    "source": [
637 |     "ChnSentiCorp_htl_ba_6000 = get_balance_corpus(6000, pd_positive, pd_negative)\n",
638 |     "\n",
639 |     "ChnSentiCorp_htl_ba_6000.sample(10)"
640 |    ]
641 |   }
642 |  ],
643 |  "metadata": {
644 |   "kernelspec": {
645 |    "display_name": "Python 3",
646 |    "language": "python",
647 |    "name": "python3"
648 |   },
649 |   "language_info": {
650 |    "codemirror_mode": {
651 |     "name": "ipython",
652 |     "version": 3
653 |    },
654 |    "file_extension": ".py",
655 |    "mimetype": "text/x-python",
656 |    "name": "python",
657 |    "nbconvert_exporter": "python",
658 |    "pygments_lexer": "ipython3",
659 |    "version": "3.6.4"
660 |   },
661 |   "widgets": {
662 |    "state": {},
663 |    "version": "1.1.2"
664 |   }
665 |  },
666 |  "nbformat": 4,
667 |  "nbformat_minor": 2
668 | }
669 | 


--------------------------------------------------------------------------------
/datasets/anhuidianxinzhidao/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# anhuidianxinzhidao 说明\n",
  8 |     "0. **下载地址：** [百度网盘](https://pan.baidu.com/s/1nrg5SRU3Xy1VN85dd85-vg)\n",
  9 |     "1. **数据概览：** 15.6 万条电信问答数据\n",
 10 |     "2. **推荐实验：** FAQ 问答系统\n",
 11 |     "3. **数据来源：** 百度知道\n",
 12 |     "4. **加工处理：**\n",
 13 |     "    1. 过滤了id、url、qid、reply_t、user字段\n",
 14 |     "    2. 对question、reply做了脱敏处理"
 15 |    ]
 16 |   },
 17 |   {
 18 |    "cell_type": "code",
 19 |    "execution_count": 1,
 20 |    "metadata": {},
 21 |    "outputs": [],
 22 |    "source": [
 23 |     "import pandas as pd"
 24 |    ]
 25 |   },
 26 |   {
 27 |    "cell_type": "code",
 28 |    "execution_count": 2,
 29 |    "metadata": {},
 30 |    "outputs": [],
 31 |    "source": [
 32 |     "path = 'anhuidianxinzhidao_文件夹_所在_路径'"
 33 |    ]
 34 |   },
 35 |   {
 36 |    "cell_type": "markdown",
 37 |    "metadata": {},
 38 |    "source": [
 39 |     "# 1.anhuidianxinzhidao_filter.csv"
 40 |    ]
 41 |   },
 42 |   {
 43 |    "cell_type": "markdown",
 44 |    "metadata": {},
 45 |    "source": [
 46 |     "## 加载数据"
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "code",
 51 |    "execution_count": 3,
 52 |    "metadata": {},
 53 |    "outputs": [],
 54 |    "source": [
 55 |     "pd_all = pd.read_csv(path + 'anhuidianxinzhidao_filter.csv')"
 56 |    ]
 57 |   },
 58 |   {
 59 |    "cell_type": "markdown",
 60 |    "metadata": {},
 61 |    "source": [
 62 |     "## 字段说明\n",
 63 |     "\n",
 64 |     "| 字段 | 说明 |\n",
 65 |     "| ---- | ---- |\n",
 66 |     "| title | 标题 |\n",
 67 |     "| question | 问题（可为空） |\n",
 68 |     "| reply| 每个问题的内容 |\n",
 69 |     "| is_best| 是否是最佳答案 |"
 70 |    ]
 71 |   },
 72 |   {
 73 |    "cell_type": "code",
 74 |    "execution_count": 4,
 75 |    "metadata": {},
 76 |    "outputs": [
 77 |     {
 78 |      "data": {
 79 |       "text/html": [
 80 |        "<div>\n",
 81 |        "<style scoped>\n",
 82 |        "    .dataframe tbody tr th:only-of-type {\n",
 83 |        "        vertical-align: middle;\n",
 84 |        "    }\n",
 85 |        "\n",
 86 |        "    .dataframe tbody tr th {\n",
 87 |        "        vertical-align: top;\n",
 88 |        "    }\n",
 89 |        "\n",
 90 |        "    .dataframe thead th {\n",
 91 |        "        text-align: right;\n",
 92 |        "    }\n",
 93 |        "</style>\n",
 94 |        "<table border=\"1\" class=\"dataframe\">\n",
 95 |        "  <thead>\n",
 96 |        "    <tr style=\"text-align: right;\">\n",
 97 |        "      <th></th>\n",
 98 |        "      <th>title</th>\n",
 99 |        "      <th>question</th>\n",
100 |        "      <th>reply</th>\n",
101 |        "      <th>is_best</th>\n",
102 |        "    </tr>\n",
103 |        "  </thead>\n",
104 |        "  <tbody>\n",
105 |        "    <tr>\n",
106 |        "      <th>129754</th>\n",
107 |        "      <td>红米no##4x</td>\n",
108 |        "      <td>NaN</td>\n",
109 |        "      <td>可以，</td>\n",
110 |        "      <td>0</td>\n",
111 |        "    </tr>\n",
112 |        "    <tr>\n",
113 |        "      <th>15843</th>\n",
114 |        "      <td>为什么不能同时用两个电信卡</td>\n",
115 |        "      <td>NaN</td>\n",
116 |        "      <td>您好不可以的，目前推出的手机都是不能同时支持两张电信手机卡的，即使是全网通手机也只能在其中的...</td>\n",
117 |        "      <td>1</td>\n",
118 |        "    </tr>\n",
119 |        "    <tr>\n",
120 |        "      <th>23985</th>\n",
121 |        "      <td>电信181、177、133哪个号段好？</td>\n",
122 |        "      <td>NaN</td>\n",
123 |        "      <td>133的</td>\n",
124 |        "      <td>0</td>\n",
125 |        "    </tr>\n",
126 |        "    <tr>\n",
127 |        "      <th>72065</th>\n",
128 |        "      <td>华*荣耀7x和魅蓝note6哪个好</td>\n",
129 |        "      <td>NaN</td>\n",
130 |        "      <td>荣耀畅玩7X很不错，性价比很高，以下是手机的配置：1、外观方面：荣耀畅玩7X采用5.93英寸...</td>\n",
131 |        "      <td>1</td>\n",
132 |        "    </tr>\n",
133 |        "    <tr>\n",
134 |        "      <th>11843</th>\n",
135 |        "      <td>p8青春版电信版多少钱</td>\n",
136 |        "      <td>NaN</td>\n",
137 |        "      <td>您好，这款手机价格参考如下</td>\n",
138 |        "      <td>1</td>\n",
139 |        "    </tr>\n",
140 |        "    <tr>\n",
141 |        "      <th>3280</th>\n",
142 |        "      <td>华为di####00叫什么</td>\n",
143 |        "      <td>华为di####00叫什么</td>\n",
144 |        "      <td>DI####00是华为畅享6S全网通版。华为畅享6S性价比高,是一款很不错的手机。电信新出流...</td>\n",
145 |        "      <td>1</td>\n",
146 |        "    </tr>\n",
147 |        "    <tr>\n",
148 |        "      <th>143200</th>\n",
149 |        "      <td>电信版酷派9190L双卡双通可以用移动网络吗</td>\n",
150 |        "      <td>NaN</td>\n",
151 |        "      <td>您好电信版双卡双待手机只能使用电信手机卡上网，卡槽2的移动或联通手机卡只能支持2G网络，一般...</td>\n",
152 |        "      <td>1</td>\n",
153 |        "    </tr>\n",
154 |        "    <tr>\n",
155 |        "      <th>120692</th>\n",
156 |        "      <td>苹果微信载图怎么载图</td>\n",
157 |        "      <td>苹果微信载图怎么载图</td>\n",
158 |        "      <td>您说的应该是截图吧。您可以直接通过苹果手机截图组合按键进行截图操作。直接同时安装电源键和ho...</td>\n",
159 |        "      <td>1</td>\n",
160 |        "    </tr>\n",
161 |        "    <tr>\n",
162 |        "      <th>109786</th>\n",
163 |        "      <td>天翼网关的wifi被我关了又没有邦定客户端怎么办想再连wifi该怎么办</td>\n",
164 |        "      <td>NaN</td>\n",
165 |        "      <td>您好电信光纤猫的无线网络一般需要破解才能使用的，但破解可能会到帐宽带不稳定或不能正常上网，建...</td>\n",
166 |        "      <td>1</td>\n",
167 |        "    </tr>\n",
168 |        "    <tr>\n",
169 |        "      <th>29030</th>\n",
170 |        "      <td>v*v*x21是不是全网通</td>\n",
171 |        "      <td>v*v*x21是不是全网通</td>\n",
172 |        "      <td>vi###21系列是有vi###21A全网通版本与vi###21移动全网通版本的；此两款机型...</td>\n",
173 |        "      <td>0</td>\n",
174 |        "    </tr>\n",
175 |        "    <tr>\n",
176 |        "      <th>72603</th>\n",
177 |        "      <td>电信网上营业厅手机卡办理步骤</td>\n",
178 |        "      <td>NaN</td>\n",
179 |        "      <td>中*电信目前是支持网上办理手机号的，下面分享下网上营业厅办理号卡的步骤：1、首先打开浏览器，...</td>\n",
180 |        "      <td>1</td>\n",
181 |        "    </tr>\n",
182 |        "    <tr>\n",
183 |        "      <th>103229</th>\n",
184 |        "      <td>花呗可以充话费吗</td>\n",
185 |        "      <td>NaN</td>\n",
186 |        "      <td>您好，是可以的，目前花呗进行充值话费,每个月只能使用花呗一次,最高不超过500元,如果您已经...</td>\n",
187 |        "      <td>0</td>\n",
188 |        "    </tr>\n",
189 |        "    <tr>\n",
190 |        "      <th>91507</th>\n",
191 |        "      <td>荣耀8好还是三星noT4好</td>\n",
192 |        "      <td>NaN</td>\n",
193 |        "      <td>如果我选择三星，华为去论坛发个意见都很尴尬。</td>\n",
194 |        "      <td>0</td>\n",
195 |        "    </tr>\n",
196 |        "    <tr>\n",
197 |        "      <th>143504</th>\n",
198 |        "      <td>ios10.2.1能降级吗ios10.2.1怎么降级</td>\n",
199 |        "      <td>NaN</td>\n",
200 |        "      <td>IOS设备一旦升级IOS系统就无法降级了，因为：1、IOS采用推荐升级、强制保持最新的升级策...</td>\n",
201 |        "      <td>1</td>\n",
202 |        "    </tr>\n",
203 |        "    <tr>\n",
204 |        "      <th>21999</th>\n",
205 |        "      <td>电信校园网宽带超一分钟多少钱</td>\n",
206 |        "      <td>NaN</td>\n",
207 |        "      <td>由于各地业务情况不同，建议用户通过当地的电信网是营业厅或者手机营业厅了解，也可以直接到附近的...</td>\n",
208 |        "      <td>1</td>\n",
209 |        "    </tr>\n",
210 |        "    <tr>\n",
211 |        "      <th>7644</th>\n",
212 |        "      <td>有没有人办过开发区的电信卡</td>\n",
213 |        "      <td>NaN</td>\n",
214 |        "      <td>您好目前使用电信手机卡的用户非常多，电信手机卡资费更优惠、网络更稳定、网速更快，请放心办理使...</td>\n",
215 |        "      <td>1</td>\n",
216 |        "    </tr>\n",
217 |        "    <tr>\n",
218 |        "      <th>76835</th>\n",
219 |        "      <td>请问67###18这个电话号码是哪里的</td>\n",
220 |        "      <td>NaN</td>\n",
221 |        "      <td>查吧</td>\n",
222 |        "      <td>0</td>\n",
223 |        "    </tr>\n",
224 |        "    <tr>\n",
225 |        "      <th>76752</th>\n",
226 |        "      <td>电信，铁通，移动，广电。那个网速好呢？</td>\n",
227 |        "      <td>NaN</td>\n",
228 |        "      <td>办理宽带推荐您办理电信宽带使用。由于中*电信的服务器、网络架设等较完善，且每年都在不断完善和...</td>\n",
229 |        "      <td>1</td>\n",
230 |        "    </tr>\n",
231 |        "    <tr>\n",
232 |        "      <th>94290</th>\n",
233 |        "      <td>三星s8+好用不</td>\n",
234 |        "      <td>NaN</td>\n",
235 |        "      <td>S8+的主要特征：1.全视曲面屏:超窄边框、沉浸感视效、双曲面侧屏的显示屏，为您带来更纯粹的...</td>\n",
236 |        "      <td>1</td>\n",
237 |        "    </tr>\n",
238 |        "    <tr>\n",
239 |        "      <th>79345</th>\n",
240 |        "      <td>一加手机5玩王者会卡吗？</td>\n",
241 |        "      <td>NaN</td>\n",
242 |        "      <td>不会卡，我也推荐你买一加5，它运行内存有8G，玩游戏的时候就能感受到性能有多好，手机不卡，丢...</td>\n",
243 |        "      <td>1</td>\n",
244 |        "    </tr>\n",
245 |        "  </tbody>\n",
246 |        "</table>\n",
247 |        "</div>"
248 |       ],
249 |       "text/plain": [
250 |        "                                      title       question  \\\n",
251 |        "129754                             红米no##4x            NaN   \n",
252 |        "15843                         为什么不能同时用两个电信卡            NaN   \n",
253 |        "23985                   电信181、177、133哪个号段好？            NaN   \n",
254 |        "72065                     华*荣耀7x和魅蓝note6哪个好            NaN   \n",
255 |        "11843                           p8青春版电信版多少钱            NaN   \n",
256 |        "3280                          华为di####00叫什么  华为di####00叫什么   \n",
257 |        "143200               电信版酷派9190L双卡双通可以用移动网络吗            NaN   \n",
258 |        "120692                           苹果微信载图怎么载图     苹果微信载图怎么载图   \n",
259 |        "109786  天翼网关的wifi被我关了又没有邦定客户端怎么办想再连wifi该怎么办            NaN   \n",
260 |        "29030                         v*v*x21是不是全网通  v*v*x21是不是全网通   \n",
261 |        "72603                        电信网上营业厅手机卡办理步骤            NaN   \n",
262 |        "103229                             花呗可以充话费吗            NaN   \n",
263 |        "91507                         荣耀8好还是三星noT4好            NaN   \n",
264 |        "143504           ios10.2.1能降级吗ios10.2.1怎么降级            NaN   \n",
265 |        "21999                        电信校园网宽带超一分钟多少钱            NaN   \n",
266 |        "7644                          有没有人办过开发区的电信卡            NaN   \n",
267 |        "76835                   请问67###18这个电话号码是哪里的            NaN   \n",
268 |        "76752                   电信，铁通，移动，广电。那个网速好呢？            NaN   \n",
269 |        "94290                              三星s8+好用不            NaN   \n",
270 |        "79345                          一加手机5玩王者会卡吗？            NaN   \n",
271 |        "\n",
272 |        "                                                    reply  is_best  \n",
273 |        "129754                                                可以，        0  \n",
274 |        "15843   您好不可以的，目前推出的手机都是不能同时支持两张电信手机卡的，即使是全网通手机也只能在其中的...        1  \n",
275 |        "23985                                                133的        0  \n",
276 |        "72065   荣耀畅玩7X很不错，性价比很高，以下是手机的配置：1、外观方面：荣耀畅玩7X采用5.93英寸...        1  \n",
277 |        "11843                                       您好，这款手机价格参考如下        1  \n",
278 |        "3280    DI####00是华为畅享6S全网通版。华为畅享6S性价比高,是一款很不错的手机。电信新出流...        1  \n",
279 |        "143200  您好电信版双卡双待手机只能使用电信手机卡上网，卡槽2的移动或联通手机卡只能支持2G网络，一般...        1  \n",
280 |        "120692  您说的应该是截图吧。您可以直接通过苹果手机截图组合按键进行截图操作。直接同时安装电源键和ho...        1  \n",
281 |        "109786  您好电信光纤猫的无线网络一般需要破解才能使用的，但破解可能会到帐宽带不稳定或不能正常上网，建...        1  \n",
282 |        "29030   vi###21系列是有vi###21A全网通版本与vi###21移动全网通版本的；此两款机型...        0  \n",
283 |        "72603   中*电信目前是支持网上办理手机号的，下面分享下网上营业厅办理号卡的步骤：1、首先打开浏览器，...        1  \n",
284 |        "103229  您好，是可以的，目前花呗进行充值话费,每个月只能使用花呗一次,最高不超过500元,如果您已经...        0  \n",
285 |        "91507                              如果我选择三星，华为去论坛发个意见都很尴尬。        0  \n",
286 |        "143504  IOS设备一旦升级IOS系统就无法降级了，因为：1、IOS采用推荐升级、强制保持最新的升级策...        1  \n",
287 |        "21999   由于各地业务情况不同，建议用户通过当地的电信网是营业厅或者手机营业厅了解，也可以直接到附近的...        1  \n",
288 |        "7644    您好目前使用电信手机卡的用户非常多，电信手机卡资费更优惠、网络更稳定、网速更快，请放心办理使...        1  \n",
289 |        "76835                                                  查吧        0  \n",
290 |        "76752   办理宽带推荐您办理电信宽带使用。由于中*电信的服务器、网络架设等较完善，且每年都在不断完善和...        1  \n",
291 |        "94290   S8+的主要特征：1.全视曲面屏:超窄边框、沉浸感视效、双曲面侧屏的显示屏，为您带来更纯粹的...        1  \n",
292 |        "79345   不会卡，我也推荐你买一加5，它运行内存有8G，玩游戏的时候就能感受到性能有多好，手机不卡，丢...        1  "
293 |       ]
294 |      },
295 |      "execution_count": 4,
296 |      "metadata": {},
297 |      "output_type": "execute_result"
298 |     }
299 |    ],
300 |    "source": [
301 |     "pd_all.sample(n=20)"
302 |    ]
303 |   },
304 |   {
305 |    "cell_type": "code",
306 |    "execution_count": null,
307 |    "metadata": {},
308 |    "outputs": [],
309 |    "source": []
310 |   }
311 |  ],
312 |  "metadata": {
313 |   "kernelspec": {
314 |    "display_name": "Python 3",
315 |    "language": "python",
316 |    "name": "python3"
317 |   },
318 |   "language_info": {
319 |    "codemirror_mode": {
320 |     "name": "ipython",
321 |     "version": 3
322 |    },
323 |    "file_extension": ".py",
324 |    "mimetype": "text/x-python",
325 |    "name": "python",
326 |    "nbconvert_exporter": "python",
327 |    "pygments_lexer": "ipython3",
328 |    "version": "3.7.0"
329 |   }
330 |  },
331 |  "nbformat": 4,
332 |  "nbformat_minor": 2
333 | }
334 | 


--------------------------------------------------------------------------------
/datasets/baoxianzhidao/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# baoxianzhidao_filter 说明\n",
  8 |     "0. **下载地址：** [百度网盘](https://pan.baidu.com/s/1cgYeIrJHAgb8D33H09Zc5w)\n",
  9 |     "1. **数据概览：** 8000 多条保险行业问答数据\n",
 10 |     "2. **推荐实验：** FAQ 问答系统\n",
 11 |     "3. **数据来源：** 百度知道\n",
 12 |     "4. **加工处理：**\n",
 13 |     "    1. 过滤了id、url、qid、reply_t、user字段\n",
 14 |     "    2. 对question、reply做了脱敏处理"
 15 |    ]
 16 |   },
 17 |   {
 18 |    "cell_type": "code",
 19 |    "execution_count": 1,
 20 |    "metadata": {},
 21 |    "outputs": [],
 22 |    "source": [
 23 |     "import pandas as pd"
 24 |    ]
 25 |   },
 26 |   {
 27 |    "cell_type": "code",
 28 |    "execution_count": 2,
 29 |    "metadata": {},
 30 |    "outputs": [],
 31 |    "source": [
 32 |     "path = 'baoxianzhidao_文件夹_所在_路径'"
 33 |    ]
 34 |   },
 35 |   {
 36 |    "cell_type": "markdown",
 37 |    "metadata": {},
 38 |    "source": [
 39 |     "# 1. baoxianzhidao_filter.csv"
 40 |    ]
 41 |   },
 42 |   {
 43 |    "cell_type": "markdown",
 44 |    "metadata": {},
 45 |    "source": [
 46 |     "## 加载数据"
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "code",
 51 |    "execution_count": 4,
 52 |    "metadata": {},
 53 |    "outputs": [],
 54 |    "source": [
 55 |     "pd_all = pd.read_csv(path + 'baoxianzhidao_filter.csv')"
 56 |    ]
 57 |   },
 58 |   {
 59 |    "cell_type": "markdown",
 60 |    "metadata": {},
 61 |    "source": [
 62 |     "## 字段说明\n",
 63 |     "\n",
 64 |     "| 字段 | 说明 |\n",
 65 |     "| ---- | ---- |\n",
 66 |     "| title | 问题的标题 |\n",
 67 |     "| question | 问题内容（可为空） |\n",
 68 |     "| reply| 回复内容 |\n",
 69 |     "| is_best| 是否为页面上显示的最佳回答 |"
 70 |    ]
 71 |   },
 72 |   {
 73 |    "cell_type": "code",
 74 |    "execution_count": 5,
 75 |    "metadata": {},
 76 |    "outputs": [
 77 |     {
 78 |      "data": {
 79 |       "text/html": [
 80 |        "<div>\n",
 81 |        "<style scoped>\n",
 82 |        "    .dataframe tbody tr th:only-of-type {\n",
 83 |        "        vertical-align: middle;\n",
 84 |        "    }\n",
 85 |        "\n",
 86 |        "    .dataframe tbody tr th {\n",
 87 |        "        vertical-align: top;\n",
 88 |        "    }\n",
 89 |        "\n",
 90 |        "    .dataframe thead th {\n",
 91 |        "        text-align: right;\n",
 92 |        "    }\n",
 93 |        "</style>\n",
 94 |        "<table border=\"1\" class=\"dataframe\">\n",
 95 |        "  <thead>\n",
 96 |        "    <tr style=\"text-align: right;\">\n",
 97 |        "      <th></th>\n",
 98 |        "      <th>title</th>\n",
 99 |        "      <th>question</th>\n",
100 |        "      <th>reply</th>\n",
101 |        "      <th>is_best</th>\n",
102 |        "    </tr>\n",
103 |        "  </thead>\n",
104 |        "  <tbody>\n",
105 |        "    <tr>\n",
106 |        "      <th>6733</th>\n",
107 |        "      <td>五险两金和五险一金有什么区别</td>\n",
108 |        "      <td>单位招聘，独立待遇中有一项是五险两金。有些单位是五险一金，还有些五险两金。然而我刚毕业小白，...</td>\n",
109 |        "      <td>五险一金是指：医疗保险，生育保险，工伤保险，失业保险和养老保险，还有住房公积金。五险两金指的...</td>\n",
110 |        "      <td>0</td>\n",
111 |        "    </tr>\n",
112 |        "    <tr>\n",
113 |        "      <th>7580</th>\n",
114 |        "      <td>户口不在本地如何办医疗保险</td>\n",
115 |        "      <td>户口不在本地如何办医疗保险</td>\n",
116 |        "      <td>户口不在本地可以办理医保，通常都是以单位名义进行办理。医疗保险分两种办理方式，一种是单位办理...</td>\n",
117 |        "      <td>1</td>\n",
118 |        "    </tr>\n",
119 |        "    <tr>\n",
120 |        "      <th>6310</th>\n",
121 |        "      <td>酒精含量百分之二十八保险公司理赔吗？</td>\n",
122 |        "      <td>NaN</td>\n",
123 |        "      <td>不会赔</td>\n",
124 |        "      <td>0</td>\n",
125 |        "    </tr>\n",
126 |        "    <tr>\n",
127 |        "      <th>5843</th>\n",
128 |        "      <td>我买的二手车，车险都没过户，怎么交保险</td>\n",
129 |        "      <td>NaN</td>\n",
130 |        "      <td>要看保险合同了，有的是指定被保险人的，如果你出了险，保险公司是不理赔的。建议尽快去过户，或者...</td>\n",
131 |        "      <td>0</td>\n",
132 |        "    </tr>\n",
133 |        "    <tr>\n",
134 |        "      <th>2790</th>\n",
135 |        "      <td>保险买交强险后可加其他险种吗</td>\n",
136 |        "      <td>NaN</td>\n",
137 |        "      <td>可以的。车险种类包括：1.交强险，交强险[全称机动车交通事故责任强制保险]是我国首个由国家法...</td>\n",
138 |        "      <td>1</td>\n",
139 |        "    </tr>\n",
140 |        "    <tr>\n",
141 |        "      <th>4301</th>\n",
142 |        "      <td>农村九级伤残赔偿标准我父亲.因矿采煤塌陷至伤残九级应赔多少钱</td>\n",
143 |        "      <td>农村九级伤残赔偿标准我父亲.因矿采煤塌陷至伤残九级应赔多少钱</td>\n",
144 |        "      <td>发生九级伤残的赔偿标准主要包括医疗费用、一次性补偿金等等，具体包括这些：医疗费：以医院发票金...</td>\n",
145 |        "      <td>1</td>\n",
146 |        "    </tr>\n",
147 |        "    <tr>\n",
148 |        "      <th>4685</th>\n",
149 |        "      <td>领着失业金还可以交失业险吗</td>\n",
150 |        "      <td>NaN</td>\n",
151 |        "      <td>可以。领取失业金只是说明目前是离职状态，但仍可以居民形式参加保险，但缴纳的只能是医疗保险和养...</td>\n",
152 |        "      <td>1</td>\n",
153 |        "    </tr>\n",
154 |        "    <tr>\n",
155 |        "      <th>7350</th>\n",
156 |        "      <td>车辆上牌照必须在当地上保险吗</td>\n",
157 |        "      <td>车辆上牌照必须在当地上保险吗</td>\n",
158 |        "      <td>不是必须在当地买保险，也可以异地投保，现在很多保险公司开发了异地买汽车保险的购买渠道。但是保...</td>\n",
159 |        "      <td>0</td>\n",
160 |        "    </tr>\n",
161 |        "    <tr>\n",
162 |        "      <th>1611</th>\n",
163 |        "      <td>泰康人寿保险官网产品多不多，能直接在网上买吗</td>\n",
164 |        "      <td>NaN</td>\n",
165 |        "      <td>你想买哪方面保险呢，主要是看给你的服务，国寿现在新*市一款你可以考虑下</td>\n",
166 |        "      <td>0</td>\n",
167 |        "    </tr>\n",
168 |        "    <tr>\n",
169 |        "      <th>5127</th>\n",
170 |        "      <td>车出事故对方全责第三者受伤对方保险应怎样理赔?</td>\n",
171 |        "      <td>NaN</td>\n",
172 |        "      <td>对方的交强险和第三者责任险可以对第三者的伤害进行赔偿。第三者责任险是保险车辆因意外事故致使第...</td>\n",
173 |        "      <td>1</td>\n",
174 |        "    </tr>\n",
175 |        "    <tr>\n",
176 |        "      <th>4743</th>\n",
177 |        "      <td>我主责对方次责,对方摩托车无保险怎么赔付？</td>\n",
178 |        "      <td>我汽车全险有不计免赔，对方摩托车什么都没有。他的车辆损失和医药费是不是由我保险公司出？那我的...</td>\n",
179 |        "      <td>对方无保险需要自费赔付损失。一般在机动车与机动车之间发生交通事故，由保险公司在机动车第三者责...</td>\n",
180 |        "      <td>1</td>\n",
181 |        "    </tr>\n",
182 |        "    <tr>\n",
183 |        "      <th>5729</th>\n",
184 |        "      <td>网上买健康保险不用检查身体吗</td>\n",
185 |        "      <td>我想在慧择网买一款险种，保大病的，但是有个疑问就是，如果不用确认我身体健康就能入保的话，这样...</td>\n",
186 |        "      <td>通常普通的健康保险是不需要体检的，不过如果年龄、保额超过保险公司规定的限度，就一定需要体检。...</td>\n",
187 |        "      <td>1</td>\n",
188 |        "    </tr>\n",
189 |        "    <tr>\n",
190 |        "      <th>3564</th>\n",
191 |        "      <td>招商信诺儿童险如何投保？</td>\n",
192 |        "      <td>NaN</td>\n",
193 |        "      <td>儿童保险是指用于解决其成长过程中所需要的教育、创业、婚嫁等费用，以及应付孩子可能面临的疾病、...</td>\n",
194 |        "      <td>1</td>\n",
195 |        "    </tr>\n",
196 |        "    <tr>\n",
197 |        "      <th>824</th>\n",
198 |        "      <td>医疗保险请问单位交的医疗保险到底有啥用–手机爱问</td>\n",
199 |        "      <td>NaN</td>\n",
200 |        "      <td>直接到当地社保处办理就可以了</td>\n",
201 |        "      <td>0</td>\n",
202 |        "    </tr>\n",
203 |        "    <tr>\n",
204 |        "      <th>4856</th>\n",
205 |        "      <td>以前办理过养老金，在交要身份证吗</td>\n",
206 |        "      <td>NaN</td>\n",
207 |        "      <td>第二次办理养老保险需要的资料1.本地人才市场《劳动保障事物代理委托协议书》2.身份正原件及复...</td>\n",
208 |        "      <td>1</td>\n",
209 |        "    </tr>\n",
210 |        "    <tr>\n",
211 |        "      <th>2054</th>\n",
212 |        "      <td>江*车可以在防城港买保险吗?</td>\n",
213 |        "      <td>江*车可以在防城港买保险吗?</td>\n",
214 |        "      <td>理论上说是可行的。具体要看各地的政策和监管要求是如何运行，不同的城市对异地投保的情况的规定是...</td>\n",
215 |        "      <td>1</td>\n",
216 |        "    </tr>\n",
217 |        "    <tr>\n",
218 |        "      <th>1415</th>\n",
219 |        "      <td>中英人寿户外保险好吗？有什么好处</td>\n",
220 |        "      <td>NaN</td>\n",
221 |        "      <td>建议直接拨打人寿客服电话咨询</td>\n",
222 |        "      <td>0</td>\n",
223 |        "    </tr>\n",
224 |        "    <tr>\n",
225 |        "      <th>5225</th>\n",
226 |        "      <td>机动车保险到期多少日内免于处罚</td>\n",
227 |        "      <td>NaN</td>\n",
228 |        "      <td>机动车保险到期就等于无保险，机动车交通事故责任强制保险条例第三十九条：机动车所有人、管理人未...</td>\n",
229 |        "      <td>1</td>\n",
230 |        "    </tr>\n",
231 |        "    <tr>\n",
232 |        "      <th>5596</th>\n",
233 |        "      <td>上学放学途中发生意外，学校购买的意外保险，可以理赔吗</td>\n",
234 |        "      <td>NaN</td>\n",
235 |        "      <td>那要看你们学校买的意外保险的条款中有没有限定只负责理赔在校园中发生的意外伤害，如果没有这样的...</td>\n",
236 |        "      <td>0</td>\n",
237 |        "    </tr>\n",
238 |        "    <tr>\n",
239 |        "      <th>7390</th>\n",
240 |        "      <td>办建筑工人意外险需要交什么证件</td>\n",
241 |        "      <td>NaN</td>\n",
242 |        "      <td>需要提供工人的身份证号需要提供建筑公司的组织机构代码证团体意外险投保书填写及盖章一、企业施工...</td>\n",
243 |        "      <td>0</td>\n",
244 |        "    </tr>\n",
245 |        "  </tbody>\n",
246 |        "</table>\n",
247 |        "</div>"
248 |       ],
249 |       "text/plain": [
250 |        "                               title  \\\n",
251 |        "6733                  五险两金和五险一金有什么区别   \n",
252 |        "7580                   户口不在本地如何办医疗保险   \n",
253 |        "6310              酒精含量百分之二十八保险公司理赔吗？   \n",
254 |        "5843             我买的二手车，车险都没过户，怎么交保险   \n",
255 |        "2790                  保险买交强险后可加其他险种吗   \n",
256 |        "4301  农村九级伤残赔偿标准我父亲.因矿采煤塌陷至伤残九级应赔多少钱   \n",
257 |        "4685                   领着失业金还可以交失业险吗   \n",
258 |        "7350                  车辆上牌照必须在当地上保险吗   \n",
259 |        "1611          泰康人寿保险官网产品多不多，能直接在网上买吗   \n",
260 |        "5127         车出事故对方全责第三者受伤对方保险应怎样理赔?   \n",
261 |        "4743           我主责对方次责,对方摩托车无保险怎么赔付？   \n",
262 |        "5729                  网上买健康保险不用检查身体吗   \n",
263 |        "3564                    招商信诺儿童险如何投保？   \n",
264 |        "824         医疗保险请问单位交的医疗保险到底有啥用–手机爱问   \n",
265 |        "4856                以前办理过养老金，在交要身份证吗   \n",
266 |        "2054                  江*车可以在防城港买保险吗?   \n",
267 |        "1415                中英人寿户外保险好吗？有什么好处   \n",
268 |        "5225                 机动车保险到期多少日内免于处罚   \n",
269 |        "5596      上学放学途中发生意外，学校购买的意外保险，可以理赔吗   \n",
270 |        "7390                 办建筑工人意外险需要交什么证件   \n",
271 |        "\n",
272 |        "                                               question  \\\n",
273 |        "6733  单位招聘，独立待遇中有一项是五险两金。有些单位是五险一金，还有些五险两金。然而我刚毕业小白，...   \n",
274 |        "7580                                      户口不在本地如何办医疗保险   \n",
275 |        "6310                                                NaN   \n",
276 |        "5843                                                NaN   \n",
277 |        "2790                                                NaN   \n",
278 |        "4301                     农村九级伤残赔偿标准我父亲.因矿采煤塌陷至伤残九级应赔多少钱   \n",
279 |        "4685                                                NaN   \n",
280 |        "7350                                     车辆上牌照必须在当地上保险吗   \n",
281 |        "1611                                                NaN   \n",
282 |        "5127                                                NaN   \n",
283 |        "4743  我汽车全险有不计免赔，对方摩托车什么都没有。他的车辆损失和医药费是不是由我保险公司出？那我的...   \n",
284 |        "5729  我想在慧择网买一款险种，保大病的，但是有个疑问就是，如果不用确认我身体健康就能入保的话，这样...   \n",
285 |        "3564                                                NaN   \n",
286 |        "824                                                 NaN   \n",
287 |        "4856                                                NaN   \n",
288 |        "2054                                     江*车可以在防城港买保险吗?   \n",
289 |        "1415                                                NaN   \n",
290 |        "5225                                                NaN   \n",
291 |        "5596                                                NaN   \n",
292 |        "7390                                                NaN   \n",
293 |        "\n",
294 |        "                                                  reply  is_best  \n",
295 |        "6733  五险一金是指：医疗保险，生育保险，工伤保险，失业保险和养老保险，还有住房公积金。五险两金指的...        0  \n",
296 |        "7580  户口不在本地可以办理医保，通常都是以单位名义进行办理。医疗保险分两种办理方式，一种是单位办理...        1  \n",
297 |        "6310                                                不会赔        0  \n",
298 |        "5843  要看保险合同了，有的是指定被保险人的，如果你出了险，保险公司是不理赔的。建议尽快去过户，或者...        0  \n",
299 |        "2790  可以的。车险种类包括：1.交强险，交强险[全称机动车交通事故责任强制保险]是我国首个由国家法...        1  \n",
300 |        "4301  发生九级伤残的赔偿标准主要包括医疗费用、一次性补偿金等等，具体包括这些：医疗费：以医院发票金...        1  \n",
301 |        "4685  可以。领取失业金只是说明目前是离职状态，但仍可以居民形式参加保险，但缴纳的只能是医疗保险和养...        1  \n",
302 |        "7350  不是必须在当地买保险，也可以异地投保，现在很多保险公司开发了异地买汽车保险的购买渠道。但是保...        0  \n",
303 |        "1611                你想买哪方面保险呢，主要是看给你的服务，国寿现在新*市一款你可以考虑下        0  \n",
304 |        "5127  对方的交强险和第三者责任险可以对第三者的伤害进行赔偿。第三者责任险是保险车辆因意外事故致使第...        1  \n",
305 |        "4743  对方无保险需要自费赔付损失。一般在机动车与机动车之间发生交通事故，由保险公司在机动车第三者责...        1  \n",
306 |        "5729  通常普通的健康保险是不需要体检的，不过如果年龄、保额超过保险公司规定的限度，就一定需要体检。...        1  \n",
307 |        "3564  儿童保险是指用于解决其成长过程中所需要的教育、创业、婚嫁等费用，以及应付孩子可能面临的疾病、...        1  \n",
308 |        "824                                      直接到当地社保处办理就可以了        0  \n",
309 |        "4856  第二次办理养老保险需要的资料1.本地人才市场《劳动保障事物代理委托协议书》2.身份正原件及复...        1  \n",
310 |        "2054  理论上说是可行的。具体要看各地的政策和监管要求是如何运行，不同的城市对异地投保的情况的规定是...        1  \n",
311 |        "1415                                     建议直接拨打人寿客服电话咨询        0  \n",
312 |        "5225  机动车保险到期就等于无保险，机动车交通事故责任强制保险条例第三十九条：机动车所有人、管理人未...        1  \n",
313 |        "5596  那要看你们学校买的意外保险的条款中有没有限定只负责理赔在校园中发生的意外伤害，如果没有这样的...        0  \n",
314 |        "7390  需要提供工人的身份证号需要提供建筑公司的组织机构代码证团体意外险投保书填写及盖章一、企业施工...        0  "
315 |       ]
316 |      },
317 |      "execution_count": 5,
318 |      "metadata": {},
319 |      "output_type": "execute_result"
320 |     }
321 |    ],
322 |    "source": [
323 |     "pd_all.sample(n=20)"
324 |    ]
325 |   },
326 |   {
327 |    "cell_type": "code",
328 |    "execution_count": null,
329 |    "metadata": {},
330 |    "outputs": [],
331 |    "source": []
332 |   }
333 |  ],
334 |  "metadata": {
335 |   "kernelspec": {
336 |    "display_name": "Python 3",
337 |    "language": "python",
338 |    "name": "python3"
339 |   },
340 |   "language_info": {
341 |    "codemirror_mode": {
342 |     "name": "ipython",
343 |     "version": 3
344 |    },
345 |    "file_extension": ".py",
346 |    "mimetype": "text/x-python",
347 |    "name": "python",
348 |    "nbconvert_exporter": "python",
349 |    "pygments_lexer": "ipython3",
350 |    "version": "3.7.0"
351 |   }
352 |  },
353 |  "nbformat": 4,
354 |  "nbformat_minor": 2
355 | }
356 | 


--------------------------------------------------------------------------------
/datasets/dh_msra/dh_msra.zip:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/SophonPlus/ChineseNlpCorpus/a0b781d72158bde242c0626c9e354d2b328865ff/datasets/dh_msra/dh_msra.zip


--------------------------------------------------------------------------------
/datasets/dh_msra/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# dh_msra 说明\n",
  8 |     "0. **下载地址：** [Github](https://github.com/SophonPlus/ChineseNlpCorpus/raw/master/datasets/dh_msra/dh_msra.zip)\n",
  9 |     "1. **数据概览：** 5 万多条中文命名实体识别标注数据（[IOB2](https://dl.acm.org/citation.cfm?id=977059) 格式，符合 [CoNLL 2002](https://www.clips.uantwerpen.be/conll2002/ner/) 和 [CRF++](https://taku910.github.io/crfpp/#format) 标准）\n",
 10 |     "2. **推荐实验：** 中文命名实体识别\n",
 11 |     "2. **数据来源：** 不详\n",
 12 |     "3. **原数据集：** [zh-NER-TF](https://github.com/Determined22/zh-NER-TF)，网上搜集，具体作者、来源不详，可能是来自于 MSRA 的语料\n",
 13 |     "4. **加工处理：**\n",
 14 |     "    1. 将原来 2 个文件 (train 和 test) 整合到 1 个文件中"
 15 |    ]
 16 |   },
 17 |   {
 18 |    "cell_type": "code",
 19 |    "execution_count": 1,
 20 |    "metadata": {},
 21 |    "outputs": [],
 22 |    "source": [
 23 |     "import codecs\n",
 24 |     "import random\n",
 25 |     "\n",
 26 |     "import numpy as np"
 27 |    ]
 28 |   },
 29 |   {
 30 |    "cell_type": "code",
 31 |    "execution_count": 2,
 32 |    "metadata": {},
 33 |    "outputs": [],
 34 |    "source": [
 35 |     "path = 'dh_msra_文件夹_所在_路径'"
 36 |    ]
 37 |   },
 38 |   {
 39 |    "cell_type": "markdown",
 40 |    "metadata": {},
 41 |    "source": [
 42 |     "# 1. dh_msra.txt"
 43 |    ]
 44 |   },
 45 |   {
 46 |    "cell_type": "markdown",
 47 |    "metadata": {},
 48 |    "source": [
 49 |     "## 加载数据"
 50 |    ]
 51 |   },
 52 |   {
 53 |    "cell_type": "code",
 54 |    "execution_count": 3,
 55 |    "metadata": {},
 56 |    "outputs": [],
 57 |    "source": [
 58 |     "def load_iob2(file_path):\n",
 59 |     "    '''加载 IOB2 格式的数据'''\n",
 60 |     "    token_seqs = []\n",
 61 |     "    label_seqs = []\n",
 62 |     "    tokens = []\n",
 63 |     "    labels = []\n",
 64 |     "    with codecs.open(file_path) as f:\n",
 65 |     "        for index, line in enumerate(f):\n",
 66 |     "            items = line.strip().split()\n",
 67 |     "            if len(items) == 2:\n",
 68 |     "                token, label = items\n",
 69 |     "                tokens.append(token)\n",
 70 |     "                labels.append(label)\n",
 71 |     "            elif len(items) == 0:\n",
 72 |     "                if tokens:\n",
 73 |     "                    token_seqs.append(tokens)\n",
 74 |     "                    label_seqs.append(labels)\n",
 75 |     "                    tokens = []\n",
 76 |     "                    labels = []\n",
 77 |     "            else:\n",
 78 |     "                print('格式错误。行号：{} 内容：{}'.format(index, line))\n",
 79 |     "                continue\n",
 80 |     "                \n",
 81 |     "    if tokens: # 如果文件末尾没有空行，手动将最后一条数据加入序列的列表中\n",
 82 |     "        token_seqs.append(tokens)\n",
 83 |     "        label_seqs.append(labels)    \n",
 84 |     "        \n",
 85 |     "    return np.array(token_seqs), np.array(label_seqs)\n",
 86 |     "\n",
 87 |     "\n",
 88 |     "def show_iob2(token_seqs, label_seqs, num=5, shuffle=True):\n",
 89 |     "    '''显示 IOB2 格式数据'''\n",
 90 |     "    if shuffle:\n",
 91 |     "        length = len(token_seqs)\n",
 92 |     "        indexes = [random.randrange(0, length) for i in range(num)] \n",
 93 |     "        zip_seqs = zip(token_seqs[indexes], label_seqs[indexes])\n",
 94 |     "    else:\n",
 95 |     "        zip_seqs = zip(token_seqs[0:num], label_seqs[0:num])\n",
 96 |     "        \n",
 97 |     "    for tokens, labels in zip_seqs:\n",
 98 |     "        for token, label in zip(tokens, labels):\n",
 99 |     "            print('{}/{} '.format(token, label), end='')\n",
100 |     "        print('\\n')"
101 |    ]
102 |   },
103 |   {
104 |    "cell_type": "code",
105 |    "execution_count": 4,
106 |    "metadata": {},
107 |    "outputs": [
108 |     {
109 |      "name": "stdout",
110 |      "output_type": "stream",
111 |      "text": [
112 |       "55289 55289\n",
113 |       "\n",
114 |       "目/O 前/O “/O 继/B-PER 生/I-PER ”/O 共/O 产/O 仔/O 5/O 胎/O ，/O 产/O 下/O 小/O 老/O 虎/O 1/O 8/O 只/O ，/O 堪/O 称/O 虎/O 妈/O 妈/O 中/O 的/O 英/O 雄/O 。/O \n",
115 |       "\n",
116 |       "历/O 史/O 的/O 内/O 涵/O 是/O 很/O 丰/O 富/O 的/O ，/O 经/O 典/O 作/O 家/O 的/O 论/O 断/O 固/O 然/O 有/O 其/O 权/O 威/O 性/O 和/O 合/O 理/O 性/O ，/O 但/O 历/O 史/O 学/O 家/O 显/O 然/O 不/O 能/O 局/O 限/O 于/O 此/O 。/O \n",
117 |       "\n",
118 |       "5/O 月/O 3/O 0/O 日/O 在/O 中/B-LOC 国/I-LOC 革/I-LOC 命/I-LOC 军/I-LOC 事/I-LOC 博/I-LOC 物/I-LOC 馆/I-LOC 开/O 幕/O 的/O 全/O 国/O 禁/O 毒/O 展/O 览/O ，/O 在/O 社/O 会/O 上/O 引/O 起/O 了/O 强/O 烈/O 的/O 反/O 响/O 。/O \n",
119 |       "\n",
120 |       "另/O 外/O ，/O 还/O 有/O 一/O 个/O 惊/O 人/O 的/O 发/O 现/O ：/O 有/O 的/O 发/O 展/O 中/O 国/O 家/O 人/O 均/O 国/O 民/O 资/O 源/O 非/O 常/O 丰/O 富/O ，/O 但/O 发/O 展/O 不/O 起/O 来/O 的/O 原/O 因/O 在/O 于/O 教/O 育/O 水/O 平/O 太/O 低/O 、/O 对/O 技/O 术/O 的/O 理/O 解/O 和/O 把/O 握/O 太/O 低/O 、/O 管/O 理/O 水/O 平/O 太/O 低/O 等/O 等/O ，/O 一/O 句/O 话/O ，/O 智/O 力/O 资/O 本/O 太/O 贫/O 乏/O 。/O \n",
121 |       "\n",
122 |       "这/O 还/O 要/O 看/O 进/O 一/O 步/O 深/O 入/O 调/O 查/O 的/O 结/O 果/O 。/O \n",
123 |       "\n"
124 |      ]
125 |     }
126 |    ],
127 |    "source": [
128 |     "token_seqs, label_seqs = load_iob2(path+'dh_msra.txt')\n",
129 |     "\n",
130 |     "print(len(token_seqs), len(label_seqs))\n",
131 |     "print()    \n",
132 |     "show_iob2(token_seqs, label_seqs)"
133 |    ]
134 |   },
135 |   {
136 |    "cell_type": "markdown",
137 |    "metadata": {},
138 |    "source": [
139 |     "## 标签说明\n",
140 |     "\n",
141 |     "| 标签 | 说明 |\n",
142 |     "| ---- | ---- |\n",
143 |     "| LOC | 地点 (LOCATION) |\n",
144 |     "| ORG | 机构 (ORGANIZATION) |\n",
145 |     "| PER | 人物 (PERSON) |"
146 |    ]
147 |   },
148 |   {
149 |    "cell_type": "code",
150 |    "execution_count": 5,
151 |    "metadata": {},
152 |    "outputs": [
153 |     {
154 |      "data": {
155 |       "text/plain": [
156 |        "{'B-LOC', 'B-ORG', 'B-PER', 'I-LOC', 'I-ORG', 'I-PER', 'O'}"
157 |       ]
158 |      },
159 |      "execution_count": 5,
160 |      "metadata": {},
161 |      "output_type": "execute_result"
162 |     }
163 |    ],
164 |    "source": [
165 |     "set([label for labels in label_seqs for label in labels])"
166 |    ]
167 |   }
168 |  ],
169 |  "metadata": {
170 |   "kernelspec": {
171 |    "display_name": "Python 3",
172 |    "language": "python",
173 |    "name": "python3"
174 |   },
175 |   "language_info": {
176 |    "codemirror_mode": {
177 |     "name": "ipython",
178 |     "version": 3
179 |    },
180 |    "file_extension": ".py",
181 |    "mimetype": "text/x-python",
182 |    "name": "python",
183 |    "nbconvert_exporter": "python",
184 |    "pygments_lexer": "ipython3",
185 |    "version": "3.6.5"
186 |   },
187 |   "widgets": {
188 |    "state": {},
189 |    "version": "1.1.2"
190 |   }
191 |  },
192 |  "nbformat": 4,
193 |  "nbformat_minor": 2
194 | }
195 | 


--------------------------------------------------------------------------------
/datasets/financezhidao/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# financezhidao 说明\n",
  8 |     "0. **下载地址：** [百度知道](https://pan.baidu.com/s/1z1Rnnk-ubRSvzDu4UvLlIw)\n",
  9 |     "1. **数据概览：** 77万 条金融行业问答数据\n",
 10 |     "2. **推荐实验：** FAQ 问答系统\n",
 11 |     "3. **数据来源：** 百度知道\n",
 12 |     "4. **加工处理：**\n",
 13 |     "    1. 过滤了id、url、qid、reply_t、user字段\n",
 14 |     "    2. 对question、reply做了脱敏处理"
 15 |    ]
 16 |   },
 17 |   {
 18 |    "cell_type": "code",
 19 |    "execution_count": 1,
 20 |    "metadata": {},
 21 |    "outputs": [],
 22 |    "source": [
 23 |     "import pandas as pd"
 24 |    ]
 25 |   },
 26 |   {
 27 |    "cell_type": "code",
 28 |    "execution_count": 2,
 29 |    "metadata": {},
 30 |    "outputs": [],
 31 |    "source": [
 32 |     "path = 'financezhidao_文件夹_所在_路径'"
 33 |    ]
 34 |   },
 35 |   {
 36 |    "cell_type": "markdown",
 37 |    "metadata": {},
 38 |    "source": [
 39 |     "# 1. financezhidao_filter.csv"
 40 |    ]
 41 |   },
 42 |   {
 43 |    "cell_type": "markdown",
 44 |    "metadata": {},
 45 |    "source": [
 46 |     "## 加载数据"
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "code",
 51 |    "execution_count": 3,
 52 |    "metadata": {},
 53 |    "outputs": [],
 54 |    "source": [
 55 |     "pd_all = pd.read_csv(path + 'financezhidao_filter.csv')"
 56 |    ]
 57 |   },
 58 |   {
 59 |    "cell_type": "markdown",
 60 |    "metadata": {},
 61 |    "source": [
 62 |     "## 字段说明\n",
 63 |     "\n",
 64 |     "| 字段 | 说明 |\n",
 65 |     "| ---- | ---- |\n",
 66 |     "| title | 标题 |\n",
 67 |     "| question | 问题（可为空） |\n",
 68 |     "| reply| 每个问题的内容 |\n",
 69 |     "| is_best| 是否是最佳答案 |"
 70 |    ]
 71 |   },
 72 |   {
 73 |    "cell_type": "code",
 74 |    "execution_count": 4,
 75 |    "metadata": {
 76 |     "scrolled": true
 77 |    },
 78 |    "outputs": [
 79 |     {
 80 |      "data": {
 81 |       "text/html": [
 82 |        "<div>\n",
 83 |        "<style scoped>\n",
 84 |        "    .dataframe tbody tr th:only-of-type {\n",
 85 |        "        vertical-align: middle;\n",
 86 |        "    }\n",
 87 |        "\n",
 88 |        "    .dataframe tbody tr th {\n",
 89 |        "        vertical-align: top;\n",
 90 |        "    }\n",
 91 |        "\n",
 92 |        "    .dataframe thead th {\n",
 93 |        "        text-align: right;\n",
 94 |        "    }\n",
 95 |        "</style>\n",
 96 |        "<table border=\"1\" class=\"dataframe\">\n",
 97 |        "  <thead>\n",
 98 |        "    <tr style=\"text-align: right;\">\n",
 99 |        "      <th></th>\n",
100 |        "      <th>title</th>\n",
101 |        "      <th>question</th>\n",
102 |        "      <th>reply</th>\n",
103 |        "      <th>is_best</th>\n",
104 |        "    </tr>\n",
105 |        "  </thead>\n",
106 |        "  <tbody>\n",
107 |        "    <tr>\n",
108 |        "      <th>678109</th>\n",
109 |        "      <td>大家好，请问信用卡怎么分期，分期有什么用处呢</td>\n",
110 |        "      <td>NaN</td>\n",
111 |        "      <td>分期好提额，但是有利息</td>\n",
112 |        "      <td>0</td>\n",
113 |        "    </tr>\n",
114 |        "    <tr>\n",
115 |        "      <th>534025</th>\n",
116 |        "      <td>本人在银行的存款，别人带本人的身份证可以取出来吗</td>\n",
117 |        "      <td>NaN</td>\n",
118 |        "      <td>若使用的是招商银行储蓄卡，在网点取款可代办，取款金额在1万元以上，需出示双人身份证原件和银行...</td>\n",
119 |        "      <td>1</td>\n",
120 |        "    </tr>\n",
121 |        "    <tr>\n",
122 |        "      <th>501941</th>\n",
123 |        "      <td>向银行贷款30万一个月要多少利息</td>\n",
124 |        "      <td>NaN</td>\n",
125 |        "      <td>1000万</td>\n",
126 |        "      <td>0</td>\n",
127 |        "    </tr>\n",
128 |        "    <tr>\n",
129 |        "      <th>734438</th>\n",
130 |        "      <td>招商信用卡还款怎么还，是每个月固定还多少钱，还是按照我们用款额度来算每个月还多少钱？</td>\n",
131 |        "      <td>NaN</td>\n",
132 |        "      <td>消费多少还多少，还款期内免利息。账单出来会提示你全额还多少，最低还多少的。</td>\n",
133 |        "      <td>1</td>\n",
134 |        "    </tr>\n",
135 |        "    <tr>\n",
136 |        "      <th>448905</th>\n",
137 |        "      <td>年利率6每月多少钱</td>\n",
138 |        "      <td>NaN</td>\n",
139 |        "      <td>一年按12个月算的</td>\n",
140 |        "      <td>0</td>\n",
141 |        "    </tr>\n",
142 |        "    <tr>\n",
143 |        "      <th>521387</th>\n",
144 |        "      <td>以卡办卡查卡里余额吗</td>\n",
145 |        "      <td>NaN</td>\n",
146 |        "      <td>若需查询招行一卡通余额，可通过电话银行，手机银行，网上银行（大众版和专业版），自助设备等渠道...</td>\n",
147 |        "      <td>1</td>\n",
148 |        "    </tr>\n",
149 |        "    <tr>\n",
150 |        "      <th>758812</th>\n",
151 |        "      <td>2016年调整金融机构各个银行人民币存贷款基准利率是多少</td>\n",
152 |        "      <td>NaN</td>\n",
153 |        "      <td>这个问题的话，本金*利率*时间就可以算出来了总的存款利率的话一般都是有央*规定的，怕出现什么...</td>\n",
154 |        "      <td>0</td>\n",
155 |        "    </tr>\n",
156 |        "    <tr>\n",
157 |        "      <th>220626</th>\n",
158 |        "      <td>请问一下，广信贷怎么样？这个理财真的可以赚？</td>\n",
159 |        "      <td>NaN</td>\n",
160 |        "      <td>所在城市若有招商银行，也可以了解下招行发售的理财产品，您可以进入招行主页，点击“理财产品”-...</td>\n",
161 |        "      <td>1</td>\n",
162 |        "    </tr>\n",
163 |        "    <tr>\n",
164 |        "      <th>86984</th>\n",
165 |        "      <td>公积金断交后，补上可以申请公积金贷款吗</td>\n",
166 |        "      <td>公积金交了一年了，但是断了大概5个月了，现在想申请公积金贷款，请问补上可以吗</td>\n",
167 |        "      <td>住房公积金断了，需要当事人准备相应的补交材料给单位经办人，由单位的经办人去有关部门办理补缴手...</td>\n",
168 |        "      <td>1</td>\n",
169 |        "    </tr>\n",
170 |        "    <tr>\n",
171 |        "      <th>20026</th>\n",
172 |        "      <td>在哪里能借到钱</td>\n",
173 |        "      <td>NaN</td>\n",
174 |        "      <td>你要借多少</td>\n",
175 |        "      <td>0</td>\n",
176 |        "    </tr>\n",
177 |        "    <tr>\n",
178 |        "      <th>121538</th>\n",
179 |        "      <td>哪有办理个人信用卡pos机</td>\n",
180 |        "      <td>NaN</td>\n",
181 |        "      <td>很多都可以办理</td>\n",
182 |        "      <td>0</td>\n",
183 |        "    </tr>\n",
184 |        "    <tr>\n",
185 |        "      <th>467245</th>\n",
186 |        "      <td>身份证消磁了就不能办银行卡了吗</td>\n",
187 |        "      <td>NaN</td>\n",
188 |        "      <td>身份证读不出信息就是无效证件是没法去银行办理业务的目前部分银行支持临时身份证+辅助证明的方式...</td>\n",
189 |        "      <td>0</td>\n",
190 |        "    </tr>\n",
191 |        "    <tr>\n",
192 |        "      <th>730725</th>\n",
193 |        "      <td>年薪20万，招行信用卡标准金卡额度能有多少？</td>\n",
194 |        "      <td>NaN</td>\n",
195 |        "      <td>正常来说一般是一万，要看你个人的信用度。这个情况要去银行问。</td>\n",
196 |        "      <td>0</td>\n",
197 |        "    </tr>\n",
198 |        "    <tr>\n",
199 |        "      <th>517301</th>\n",
200 |        "      <td>自己可以拿家长的身份证办银行卡么吗</td>\n",
201 |        "      <td>NaN</td>\n",
202 |        "      <td>必须本人办理</td>\n",
203 |        "      <td>0</td>\n",
204 |        "    </tr>\n",
205 |        "    <tr>\n",
206 |        "      <th>255614</th>\n",
207 |        "      <td>有没有人现在能借一千以内给我，急需，无前期，走今借到</td>\n",
208 |        "      <td>NaN</td>\n",
209 |        "      <td>那么晚了还出来诈骗</td>\n",
210 |        "      <td>0</td>\n",
211 |        "    </tr>\n",
212 |        "    <tr>\n",
213 |        "      <th>545539</th>\n",
214 |        "      <td>招行信用卡查询密码怎么修改有多种方式</td>\n",
215 |        "      <td>NaN</td>\n",
216 |        "      <td>可以通过网银大众版、专业版、手机银行、掌上生活客户端、电话银行等渠道修改。</td>\n",
217 |        "      <td>1</td>\n",
218 |        "    </tr>\n",
219 |        "    <tr>\n",
220 |        "      <th>747413</th>\n",
221 |        "      <td>信用卡面签被拒的原因是什么？</td>\n",
222 |        "      <td>信用卡面签被拒的原因是什么？</td>\n",
223 |        "      <td>若申请的是招行信用卡，最主要的条件是有稳定的工作和收入，必备申请文件为身份证明复印件和工作证...</td>\n",
224 |        "      <td>1</td>\n",
225 |        "    </tr>\n",
226 |        "    <tr>\n",
227 |        "      <th>669087</th>\n",
228 |        "      <td>信用卡还款日提前一天会黑名单</td>\n",
229 |        "      <td>NaN</td>\n",
230 |        "      <td>你好，这个是不会的，信用卡还款日是指免息期的最后一天，在这个时间之前全额还款都是没有问题的。...</td>\n",
231 |        "      <td>0</td>\n",
232 |        "    </tr>\n",
233 |        "    <tr>\n",
234 |        "      <th>237058</th>\n",
235 |        "      <td>求一个农村人可以借钱的软件，就几百块急用，在网上找了十几个认证了</td>\n",
236 |        "      <td>求一个农村人可以借钱的软件，就几百块急用，在网上找了十几个认证了半天都不给借，求一个靠谱的</td>\n",
237 |        "      <td>你好很高兴为您解答:qq现金贷不错</td>\n",
238 |        "      <td>0</td>\n",
239 |        "    </tr>\n",
240 |        "    <tr>\n",
241 |        "      <th>131290</th>\n",
242 |        "      <td>我们办理房贷合同时银行工作人员给信用卡申请来填，那个信用卡的核实信息我答不上会影响放款吗</td>\n",
243 |        "      <td>我们办理房贷合同时银行工作人员给信用卡申请来填，那个信用卡的核实信息我答不上会影响放款吗急用</td>\n",
244 |        "      <td>若是在招行申请的个人住房贷款，信用卡的核发情况不影响贷款放款。贷款的最终审核是否能够通过，是...</td>\n",
245 |        "      <td>1</td>\n",
246 |        "    </tr>\n",
247 |        "  </tbody>\n",
248 |        "</table>\n",
249 |        "</div>"
250 |       ],
251 |       "text/plain": [
252 |        "                                               title  \\\n",
253 |        "678109                        大家好，请问信用卡怎么分期，分期有什么用处呢   \n",
254 |        "534025                      本人在银行的存款，别人带本人的身份证可以取出来吗   \n",
255 |        "501941                              向银行贷款30万一个月要多少利息   \n",
256 |        "734438    招商信用卡还款怎么还，是每个月固定还多少钱，还是按照我们用款额度来算每个月还多少钱？   \n",
257 |        "448905                                     年利率6每月多少钱   \n",
258 |        "521387                                    以卡办卡查卡里余额吗   \n",
259 |        "758812                  2016年调整金融机构各个银行人民币存贷款基准利率是多少   \n",
260 |        "220626                        请问一下，广信贷怎么样？这个理财真的可以赚？   \n",
261 |        "86984                            公积金断交后，补上可以申请公积金贷款吗   \n",
262 |        "20026                                        在哪里能借到钱   \n",
263 |        "121538                                 哪有办理个人信用卡pos机   \n",
264 |        "467245                               身份证消磁了就不能办银行卡了吗   \n",
265 |        "730725                        年薪20万，招行信用卡标准金卡额度能有多少？   \n",
266 |        "517301                             自己可以拿家长的身份证办银行卡么吗   \n",
267 |        "255614                    有没有人现在能借一千以内给我，急需，无前期，走今借到   \n",
268 |        "545539                            招行信用卡查询密码怎么修改有多种方式   \n",
269 |        "747413                                信用卡面签被拒的原因是什么？   \n",
270 |        "669087                                信用卡还款日提前一天会黑名单   \n",
271 |        "237058              求一个农村人可以借钱的软件，就几百块急用，在网上找了十几个认证了   \n",
272 |        "131290  我们办理房贷合同时银行工作人员给信用卡申请来填，那个信用卡的核实信息我答不上会影响放款吗   \n",
273 |        "\n",
274 |        "                                              question  \\\n",
275 |        "678109                                             NaN   \n",
276 |        "534025                                             NaN   \n",
277 |        "501941                                             NaN   \n",
278 |        "734438                                             NaN   \n",
279 |        "448905                                             NaN   \n",
280 |        "521387                                             NaN   \n",
281 |        "758812                                             NaN   \n",
282 |        "220626                                             NaN   \n",
283 |        "86984           公积金交了一年了，但是断了大概5个月了，现在想申请公积金贷款，请问补上可以吗   \n",
284 |        "20026                                              NaN   \n",
285 |        "121538                                             NaN   \n",
286 |        "467245                                             NaN   \n",
287 |        "730725                                             NaN   \n",
288 |        "517301                                             NaN   \n",
289 |        "255614                                             NaN   \n",
290 |        "545539                                             NaN   \n",
291 |        "747413                                  信用卡面签被拒的原因是什么？   \n",
292 |        "669087                                             NaN   \n",
293 |        "237058   求一个农村人可以借钱的软件，就几百块急用，在网上找了十几个认证了半天都不给借，求一个靠谱的   \n",
294 |        "131290  我们办理房贷合同时银行工作人员给信用卡申请来填，那个信用卡的核实信息我答不上会影响放款吗急用   \n",
295 |        "\n",
296 |        "                                                    reply  is_best  \n",
297 |        "678109                                        分期好提额，但是有利息        0  \n",
298 |        "534025  若使用的是招商银行储蓄卡，在网点取款可代办，取款金额在1万元以上，需出示双人身份证原件和银行...        1  \n",
299 |        "501941                                              1000万        0  \n",
300 |        "734438              消费多少还多少，还款期内免利息。账单出来会提示你全额还多少，最低还多少的。        1  \n",
301 |        "448905                                          一年按12个月算的        0  \n",
302 |        "521387  若需查询招行一卡通余额，可通过电话银行，手机银行，网上银行（大众版和专业版），自助设备等渠道...        1  \n",
303 |        "758812  这个问题的话，本金*利率*时间就可以算出来了总的存款利率的话一般都是有央*规定的，怕出现什么...        0  \n",
304 |        "220626  所在城市若有招商银行，也可以了解下招行发售的理财产品，您可以进入招行主页，点击“理财产品”-...        1  \n",
305 |        "86984   住房公积金断了，需要当事人准备相应的补交材料给单位经办人，由单位的经办人去有关部门办理补缴手...        1  \n",
306 |        "20026                                               你要借多少        0  \n",
307 |        "121538                                            很多都可以办理        0  \n",
308 |        "467245  身份证读不出信息就是无效证件是没法去银行办理业务的目前部分银行支持临时身份证+辅助证明的方式...        0  \n",
309 |        "730725                     正常来说一般是一万，要看你个人的信用度。这个情况要去银行问。        0  \n",
310 |        "517301                                             必须本人办理        0  \n",
311 |        "255614                                          那么晚了还出来诈骗        0  \n",
312 |        "545539              可以通过网银大众版、专业版、手机银行、掌上生活客户端、电话银行等渠道修改。        1  \n",
313 |        "747413  若申请的是招行信用卡，最主要的条件是有稳定的工作和收入，必备申请文件为身份证明复印件和工作证...        1  \n",
314 |        "669087  你好，这个是不会的，信用卡还款日是指免息期的最后一天，在这个时间之前全额还款都是没有问题的。...        0  \n",
315 |        "237058                                  你好很高兴为您解答:qq现金贷不错        0  \n",
316 |        "131290  若是在招行申请的个人住房贷款，信用卡的核发情况不影响贷款放款。贷款的最终审核是否能够通过，是...        1  "
317 |       ]
318 |      },
319 |      "execution_count": 4,
320 |      "metadata": {},
321 |      "output_type": "execute_result"
322 |     }
323 |    ],
324 |    "source": [
325 |     "pd_all.sample(n=20)"
326 |    ]
327 |   },
328 |   {
329 |    "cell_type": "code",
330 |    "execution_count": null,
331 |    "metadata": {},
332 |    "outputs": [],
333 |    "source": []
334 |   }
335 |  ],
336 |  "metadata": {
337 |   "kernelspec": {
338 |    "display_name": "Python 3",
339 |    "language": "python",
340 |    "name": "python3"
341 |   },
342 |   "language_info": {
343 |    "codemirror_mode": {
344 |     "name": "ipython",
345 |     "version": 3
346 |    },
347 |    "file_extension": ".py",
348 |    "mimetype": "text/x-python",
349 |    "name": "python",
350 |    "nbconvert_exporter": "python",
351 |    "pygments_lexer": "ipython3",
352 |    "version": "3.7.0"
353 |   }
354 |  },
355 |  "nbformat": 4,
356 |  "nbformat_minor": 2
357 | }
358 | 


--------------------------------------------------------------------------------
/datasets/lawzhidao/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# lawzhidao_filter 说明\n",
  8 |     "0. **下载地址：** [百度知道](https://pan.baidu.com/s/18Lwq16VBo6wBD_qLb3i33g)\n",
  9 |     "1. **数据概览：** 3.6 万条法律问答数据\n",
 10 |     "2. **推荐实验：** FAQ 问答系统\n",
 11 |     "3. **数据来源：** 百度知道\n",
 12 |     "4. **加工处理：**\n",
 13 |     "    1. 过滤了id、url、qid、reply_t、user字段\n",
 14 |     "    2. 对question、reply做了脱敏处理"
 15 |    ]
 16 |   },
 17 |   {
 18 |    "cell_type": "code",
 19 |    "execution_count": 1,
 20 |    "metadata": {},
 21 |    "outputs": [],
 22 |    "source": [
 23 |     "import pandas as pd"
 24 |    ]
 25 |   },
 26 |   {
 27 |    "cell_type": "code",
 28 |    "execution_count": 2,
 29 |    "metadata": {},
 30 |    "outputs": [],
 31 |    "source": [
 32 |     "path = 'lawzhidao_文件夹_所在_路径'"
 33 |    ]
 34 |   },
 35 |   {
 36 |    "cell_type": "markdown",
 37 |    "metadata": {},
 38 |    "source": [
 39 |     "# 1. lawzhidao_filter.csv"
 40 |    ]
 41 |   },
 42 |   {
 43 |    "cell_type": "markdown",
 44 |    "metadata": {},
 45 |    "source": [
 46 |     "## 加载数据"
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "code",
 51 |    "execution_count": 3,
 52 |    "metadata": {},
 53 |    "outputs": [],
 54 |    "source": [
 55 |     "pd_all = pd.read_csv(path + 'baoxianzhidao_filter.csv')"
 56 |    ]
 57 |   },
 58 |   {
 59 |    "cell_type": "markdown",
 60 |    "metadata": {},
 61 |    "source": [
 62 |     "## 字段说明\n",
 63 |     "\n",
 64 |     "| 字段 | 说明 |\n",
 65 |     "| ---- | ---- |\n",
 66 |     "| title | 问题的标题 |\n",
 67 |     "| question | 问题内容（可为空） |\n",
 68 |     "| reply| 回复内容 |\n",
 69 |     "| is_best| 是否为页面上显示的最佳回答 |"
 70 |    ]
 71 |   },
 72 |   {
 73 |    "cell_type": "code",
 74 |    "execution_count": 4,
 75 |    "metadata": {
 76 |     "scrolled": false
 77 |    },
 78 |    "outputs": [
 79 |     {
 80 |      "data": {
 81 |       "text/html": [
 82 |        "<div>\n",
 83 |        "<style scoped>\n",
 84 |        "    .dataframe tbody tr th:only-of-type {\n",
 85 |        "        vertical-align: middle;\n",
 86 |        "    }\n",
 87 |        "\n",
 88 |        "    .dataframe tbody tr th {\n",
 89 |        "        vertical-align: top;\n",
 90 |        "    }\n",
 91 |        "\n",
 92 |        "    .dataframe thead th {\n",
 93 |        "        text-align: right;\n",
 94 |        "    }\n",
 95 |        "</style>\n",
 96 |        "<table border=\"1\" class=\"dataframe\">\n",
 97 |        "  <thead>\n",
 98 |        "    <tr style=\"text-align: right;\">\n",
 99 |        "      <th></th>\n",
100 |        "      <th>title</th>\n",
101 |        "      <th>question</th>\n",
102 |        "      <th>reply</th>\n",
103 |        "      <th>is_best</th>\n",
104 |        "    </tr>\n",
105 |        "  </thead>\n",
106 |        "  <tbody>\n",
107 |        "    <tr>\n",
108 |        "      <th>6725</th>\n",
109 |        "      <td>请问车险理赔时，全责一方和无责任一方收到待遇的区别</td>\n",
110 |        "      <td>NaN</td>\n",
111 |        "      <td>这位朋友提问的有些过于笼统了不是很详细，理论上来讲，从商业险的角度分析，有责任，保险公司才会...</td>\n",
112 |        "      <td>0</td>\n",
113 |        "    </tr>\n",
114 |        "    <tr>\n",
115 |        "      <th>6399</th>\n",
116 |        "      <td>买保险,一定要找代理人吗,直接去保险公司买不可以吗?</td>\n",
117 |        "      <td>买保险,一定要找代理人吗,直接去保险公司买不可以吗?</td>\n",
118 |        "      <td>可以的。可以自行去保险公司进行投保，也可以选择在网上投保。不过有代理人的好处在于可以为被保险...</td>\n",
119 |        "      <td>1</td>\n",
120 |        "    </tr>\n",
121 |        "    <tr>\n",
122 |        "      <th>4242</th>\n",
123 |        "      <td>机动车撞伤人至骨折保险公司该怎么赔偿</td>\n",
124 |        "      <td>NaN</td>\n",
125 |        "      <td>交通事故赔偿是有标准的，因交通事故造成损失，肇事者向受害者、保险公司对承保车辆造成的损失进行...</td>\n",
126 |        "      <td>1</td>\n",
127 |        "    </tr>\n",
128 |        "    <tr>\n",
129 |        "      <th>7481</th>\n",
130 |        "      <td>贷款买养老保险如何办理？</td>\n",
131 |        "      <td>贷款买养老保险如何办理？</td>\n",
132 |        "      <td>助保贷款主要是针对中断缴纳基本养老保险费的接近退休年龄无力续保的困难*员，通过政府担保贴息、...</td>\n",
133 |        "      <td>0</td>\n",
134 |        "    </tr>\n",
135 |        "    <tr>\n",
136 |        "      <th>5674</th>\n",
137 |        "      <td>摩托车行车证年审应交哪些保险?一定要交驾驶员个人险吗?</td>\n",
138 |        "      <td>NaN</td>\n",
139 |        "      <td>摩托车买保险最应该买的就是交强险，一般根据排量的不同共分为三个类别，其中50CC及以下的排量...</td>\n",
140 |        "      <td>1</td>\n",
141 |        "    </tr>\n",
142 |        "    <tr>\n",
143 |        "      <th>1122</th>\n",
144 |        "      <td>惠*安保费贵不贵？一年需要多少钱？</td>\n",
145 |        "      <td>NaN</td>\n",
146 |        "      <td>年缴保费500元，缴费20年，保障30年。</td>\n",
147 |        "      <td>1</td>\n",
148 |        "    </tr>\n",
149 |        "    <tr>\n",
150 |        "      <th>5511</th>\n",
151 |        "      <td>农村医保没有交,会把户口注销了吗?本人现不在家无法交医保，乡镇通知我，他说我不交医保就把我的户口</td>\n",
152 |        "      <td>销了。是真的吗？</td>\n",
153 |        "      <td>不会的，这是不合法的，新农合是指由政府组织、引导、支持，农民自愿参加，个人、集体和政府多方筹...</td>\n",
154 |        "      <td>1</td>\n",
155 |        "    </tr>\n",
156 |        "    <tr>\n",
157 |        "      <th>7338</th>\n",
158 |        "      <td>新华保险的保单贷款是怎样还的?</td>\n",
159 |        "      <td>NaN</td>\n",
160 |        "      <td>半年要去签一次息，具体情况，可以直接咨询新华人寿保险公司，新华客服热线9##67</td>\n",
161 |        "      <td>0</td>\n",
162 |        "    </tr>\n",
163 |        "    <tr>\n",
164 |        "      <th>1280</th>\n",
165 |        "      <td>一起慧99到底有什么优惠相比其他的保险</td>\n",
166 |        "      <td>NaN</td>\n",
167 |        "      <td>您好！一起慧99</td>\n",
168 |        "      <td>0</td>\n",
169 |        "    </tr>\n",
170 |        "    <tr>\n",
171 |        "      <th>6388</th>\n",
172 |        "      <td>辞职后，养老保险如果不转移会怎么样</td>\n",
173 |        "      <td>我2010年2月在原公司辞职后，养老保险没有转移。如果不转移，我这部分养老保险会怎么处理？</td>\n",
174 |        "      <td>会被封存，所以要及时转移。养老保险转移和接续手续：一、申请出具《基本养老保险参保缴费凭证》职...</td>\n",
175 |        "      <td>1</td>\n",
176 |        "    </tr>\n",
177 |        "    <tr>\n",
178 |        "      <th>7920</th>\n",
179 |        "      <td>慧*安*儿定期重疾是怎么理赔的</td>\n",
180 |        "      <td>NaN</td>\n",
181 |        "      <td>首先是报案您或被保险人应在知道保险事故发生之日起10日内通知本公司。如果您或受益人故意或者因...</td>\n",
182 |        "      <td>1</td>\n",
183 |        "    </tr>\n",
184 |        "    <tr>\n",
185 |        "      <th>3134</th>\n",
186 |        "      <td>构不成住院条件的车祸需要赔付精神损失费误工费营养费护理费吗</td>\n",
187 |        "      <td>NaN</td>\n",
188 |        "      <td>只要存在精神损失、误工、需要增加营养、护理的费用，就可以向侵权人主张赔偿责任。</td>\n",
189 |        "      <td>0</td>\n",
190 |        "    </tr>\n",
191 |        "    <tr>\n",
192 |        "      <th>4015</th>\n",
193 |        "      <td>基本保险金额是什么意思</td>\n",
194 |        "      <td>基本保险金额是什么意思</td>\n",
195 |        "      <td>基本保险金额是保单上明确标注的金额，保险金额是能拿到的保险赔付金额，有些保险条款的基本保险金...</td>\n",
196 |        "      <td>1</td>\n",
197 |        "    </tr>\n",
198 |        "    <tr>\n",
199 |        "      <th>6848</th>\n",
200 |        "      <td>重大疾病保险有必要买吗？</td>\n",
201 |        "      <td>我今年25岁，身体很健康，我去买保险，保险公司的人给我的计划里有重大疾病保险的项目，但是我只...</td>\n",
202 |        "      <td>重大疾病保险还是很有必要买的。我国的医疗保障体系是由基本医保和商业健康保险组成。如果发生重大...</td>\n",
203 |        "      <td>1</td>\n",
204 |        "    </tr>\n",
205 |        "    <tr>\n",
206 |        "      <th>2494</th>\n",
207 |        "      <td>库*勒妇科商业医保报销范围有哪些？</td>\n",
208 |        "      <td>库*勒妇科商业医保报销范围有哪些？</td>\n",
209 |        "      <td>你好，商业医保报销范围比医疗保险报销更广。基本都是能报销的。报销分农村居民和城镇职工：1、居...</td>\n",
210 |        "      <td>0</td>\n",
211 |        "    </tr>\n",
212 |        "    <tr>\n",
213 |        "      <th>7341</th>\n",
214 |        "      <td>第三者保险营运与非营运什么区别</td>\n",
215 |        "      <td>第三者保险营运与非营运什么区别</td>\n",
216 |        "      <td>车辆行驶证的“使用性质“一个是营运，一个是非营运。营运需要在运输管理部门办理车辆的道路运输许...</td>\n",
217 |        "      <td>1</td>\n",
218 |        "    </tr>\n",
219 |        "    <tr>\n",
220 |        "      <th>4997</th>\n",
221 |        "      <td>犹豫期内退保一定要去原来办理的地点吗?</td>\n",
222 |        "      <td>犹豫期内退保一定要去原来办理的地点吗?</td>\n",
223 |        "      <td>要退保必须去保险公司退，在银行的柜台上是没办法退的，而且退保必须由投保人本人持其身份证去退，...</td>\n",
224 |        "      <td>1</td>\n",
225 |        "    </tr>\n",
226 |        "    <tr>\n",
227 |        "      <th>5694</th>\n",
228 |        "      <td>保险法的构成主要包括</td>\n",
229 |        "      <td>NaN</td>\n",
230 |        "      <td>保险法的构成主要包括保险业法、保险合同法*保险特别法。1.保险业法又称保险事业法、保险事业监...</td>\n",
231 |        "      <td>0</td>\n",
232 |        "    </tr>\n",
233 |        "    <tr>\n",
234 |        "      <th>1604</th>\n",
235 |        "      <td>适合中老年的保险多不多，能买哪些保险？</td>\n",
236 |        "      <td>NaN</td>\n",
237 |        "      <td>年龄多大呢？保费预算多少？</td>\n",
238 |        "      <td>0</td>\n",
239 |        "    </tr>\n",
240 |        "    <tr>\n",
241 |        "      <th>3098</th>\n",
242 |        "      <td>汽车购置税属于机动车第三者责任险赔偿范围内吗？</td>\n",
243 |        "      <td>NaN</td>\n",
244 |        "      <td>购置税你是你购置车辆的时候上牌还需要交的费用。跟保险不是一个范围。</td>\n",
245 |        "      <td>0</td>\n",
246 |        "    </tr>\n",
247 |        "  </tbody>\n",
248 |        "</table>\n",
249 |        "</div>"
250 |       ],
251 |       "text/plain": [
252 |        "                                                 title  \\\n",
253 |        "6725                         请问车险理赔时，全责一方和无责任一方收到待遇的区别   \n",
254 |        "6399                        买保险,一定要找代理人吗,直接去保险公司买不可以吗?   \n",
255 |        "4242                                机动车撞伤人至骨折保险公司该怎么赔偿   \n",
256 |        "7481                                      贷款买养老保险如何办理？   \n",
257 |        "5674                       摩托车行车证年审应交哪些保险?一定要交驾驶员个人险吗?   \n",
258 |        "1122                                 惠*安保费贵不贵？一年需要多少钱？   \n",
259 |        "5511  农村医保没有交,会把户口注销了吗?本人现不在家无法交医保，乡镇通知我，他说我不交医保就把我的户口   \n",
260 |        "7338                                   新华保险的保单贷款是怎样还的?   \n",
261 |        "1280                               一起慧99到底有什么优惠相比其他的保险   \n",
262 |        "6388                                 辞职后，养老保险如果不转移会怎么样   \n",
263 |        "7920                                   慧*安*儿定期重疾是怎么理赔的   \n",
264 |        "3134                     构不成住院条件的车祸需要赔付精神损失费误工费营养费护理费吗   \n",
265 |        "4015                                       基本保险金额是什么意思   \n",
266 |        "6848                                      重大疾病保险有必要买吗？   \n",
267 |        "2494                                 库*勒妇科商业医保报销范围有哪些？   \n",
268 |        "7341                                   第三者保险营运与非营运什么区别   \n",
269 |        "4997                               犹豫期内退保一定要去原来办理的地点吗?   \n",
270 |        "5694                                        保险法的构成主要包括   \n",
271 |        "1604                               适合中老年的保险多不多，能买哪些保险？   \n",
272 |        "3098                           汽车购置税属于机动车第三者责任险赔偿范围内吗？   \n",
273 |        "\n",
274 |        "                                               question  \\\n",
275 |        "6725                                                NaN   \n",
276 |        "6399                         买保险,一定要找代理人吗,直接去保险公司买不可以吗?   \n",
277 |        "4242                                                NaN   \n",
278 |        "7481                                       贷款买养老保险如何办理？   \n",
279 |        "5674                                                NaN   \n",
280 |        "1122                                                NaN   \n",
281 |        "5511                                           销了。是真的吗？   \n",
282 |        "7338                                                NaN   \n",
283 |        "1280                                                NaN   \n",
284 |        "6388      我2010年2月在原公司辞职后，养老保险没有转移。如果不转移，我这部分养老保险会怎么处理？   \n",
285 |        "7920                                                NaN   \n",
286 |        "3134                                                NaN   \n",
287 |        "4015                                        基本保险金额是什么意思   \n",
288 |        "6848  我今年25岁，身体很健康，我去买保险，保险公司的人给我的计划里有重大疾病保险的项目，但是我只...   \n",
289 |        "2494                                  库*勒妇科商业医保报销范围有哪些？   \n",
290 |        "7341                                    第三者保险营运与非营运什么区别   \n",
291 |        "4997                                犹豫期内退保一定要去原来办理的地点吗?   \n",
292 |        "5694                                                NaN   \n",
293 |        "1604                                                NaN   \n",
294 |        "3098                                                NaN   \n",
295 |        "\n",
296 |        "                                                  reply  is_best  \n",
297 |        "6725  这位朋友提问的有些过于笼统了不是很详细，理论上来讲，从商业险的角度分析，有责任，保险公司才会...        0  \n",
298 |        "6399  可以的。可以自行去保险公司进行投保，也可以选择在网上投保。不过有代理人的好处在于可以为被保险...        1  \n",
299 |        "4242  交通事故赔偿是有标准的，因交通事故造成损失，肇事者向受害者、保险公司对承保车辆造成的损失进行...        1  \n",
300 |        "7481  助保贷款主要是针对中断缴纳基本养老保险费的接近退休年龄无力续保的困难*员，通过政府担保贴息、...        0  \n",
301 |        "5674  摩托车买保险最应该买的就是交强险，一般根据排量的不同共分为三个类别，其中50CC及以下的排量...        1  \n",
302 |        "1122                              年缴保费500元，缴费20年，保障30年。        1  \n",
303 |        "5511  不会的，这是不合法的，新农合是指由政府组织、引导、支持，农民自愿参加，个人、集体和政府多方筹...        1  \n",
304 |        "7338           半年要去签一次息，具体情况，可以直接咨询新华人寿保险公司，新华客服热线9##67        0  \n",
305 |        "1280                                           您好！一起慧99        0  \n",
306 |        "6388  会被封存，所以要及时转移。养老保险转移和接续手续：一、申请出具《基本养老保险参保缴费凭证》职...        1  \n",
307 |        "7920  首先是报案您或被保险人应在知道保险事故发生之日起10日内通知本公司。如果您或受益人故意或者因...        1  \n",
308 |        "3134            只要存在精神损失、误工、需要增加营养、护理的费用，就可以向侵权人主张赔偿责任。        0  \n",
309 |        "4015  基本保险金额是保单上明确标注的金额，保险金额是能拿到的保险赔付金额，有些保险条款的基本保险金...        1  \n",
310 |        "6848  重大疾病保险还是很有必要买的。我国的医疗保障体系是由基本医保和商业健康保险组成。如果发生重大...        1  \n",
311 |        "2494  你好，商业医保报销范围比医疗保险报销更广。基本都是能报销的。报销分农村居民和城镇职工：1、居...        0  \n",
312 |        "7341  车辆行驶证的“使用性质“一个是营运，一个是非营运。营运需要在运输管理部门办理车辆的道路运输许...        1  \n",
313 |        "4997  要退保必须去保险公司退，在银行的柜台上是没办法退的，而且退保必须由投保人本人持其身份证去退，...        1  \n",
314 |        "5694  保险法的构成主要包括保险业法、保险合同法*保险特别法。1.保险业法又称保险事业法、保险事业监...        0  \n",
315 |        "1604                                      年龄多大呢？保费预算多少？        0  \n",
316 |        "3098                  购置税你是你购置车辆的时候上牌还需要交的费用。跟保险不是一个范围。        0  "
317 |       ]
318 |      },
319 |      "execution_count": 4,
320 |      "metadata": {},
321 |      "output_type": "execute_result"
322 |     }
323 |    ],
324 |    "source": [
325 |     "pd_all.sample(n=20)"
326 |    ]
327 |   },
328 |   {
329 |    "cell_type": "code",
330 |    "execution_count": null,
331 |    "metadata": {},
332 |    "outputs": [],
333 |    "source": []
334 |   }
335 |  ],
336 |  "metadata": {
337 |   "kernelspec": {
338 |    "display_name": "Python 3",
339 |    "language": "python",
340 |    "name": "python3"
341 |   },
342 |   "language_info": {
343 |    "codemirror_mode": {
344 |     "name": "ipython",
345 |     "version": 3
346 |    },
347 |    "file_extension": ".py",
348 |    "mimetype": "text/x-python",
349 |    "name": "python",
350 |    "nbconvert_exporter": "python",
351 |    "pygments_lexer": "ipython3",
352 |    "version": "3.7.0"
353 |   }
354 |  },
355 |  "nbformat": 4,
356 |  "nbformat_minor": 2
357 | }
358 | 


--------------------------------------------------------------------------------
/datasets/liantongzhidao/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# liantongzhidao_filter 说明\n",
  8 |     "0. **下载地址：** [百度网盘](https://pan.baidu.com/s/1oYi9SfbXpnvreJYGV837Nw)\n",
  9 |     "1. **数据概览：** 20.3万 条联通问答数据\n",
 10 |     "2. **推荐实验：** FAQ 问答系统\n",
 11 |     "3. **数据来源：** 百度知道\n",
 12 |     "4. **加工处理：**\n",
 13 |     "    1. 过滤了id、url、qid、reply_t、user字段\n",
 14 |     "    2. 对question、reply做了脱敏处理"
 15 |    ]
 16 |   },
 17 |   {
 18 |    "cell_type": "code",
 19 |    "execution_count": 1,
 20 |    "metadata": {},
 21 |    "outputs": [],
 22 |    "source": [
 23 |     "import pandas as pd"
 24 |    ]
 25 |   },
 26 |   {
 27 |    "cell_type": "code",
 28 |    "execution_count": 2,
 29 |    "metadata": {},
 30 |    "outputs": [],
 31 |    "source": [
 32 |     "path = 'liantongzhidao_文件夹_所在_路径'"
 33 |    ]
 34 |   },
 35 |   {
 36 |    "cell_type": "markdown",
 37 |    "metadata": {},
 38 |    "source": [
 39 |     "# 1. liantongzhidao_filter.csv"
 40 |    ]
 41 |   },
 42 |   {
 43 |    "cell_type": "markdown",
 44 |    "metadata": {},
 45 |    "source": [
 46 |     "## 加载数据"
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "code",
 51 |    "execution_count": 3,
 52 |    "metadata": {},
 53 |    "outputs": [],
 54 |    "source": [
 55 |     "pd_all = pd.read_csv(path + 'liantongzhidao_filter.csv')"
 56 |    ]
 57 |   },
 58 |   {
 59 |    "cell_type": "markdown",
 60 |    "metadata": {},
 61 |    "source": [
 62 |     "## 字段说明\n",
 63 |     "\n",
 64 |     "| 字段 | 说明 |\n",
 65 |     "| ---- | ---- |\n",
 66 |     "| title | 问题的标题 |\n",
 67 |     "| question | 问题内容（可为空） |\n",
 68 |     "| reply| 回复内容 |\n",
 69 |     "| is_best| 是否为页面上显示的最佳回答 |"
 70 |    ]
 71 |   },
 72 |   {
 73 |    "cell_type": "code",
 74 |    "execution_count": 4,
 75 |    "metadata": {},
 76 |    "outputs": [
 77 |     {
 78 |      "data": {
 79 |       "text/html": [
 80 |        "<div>\n",
 81 |        "<style scoped>\n",
 82 |        "    .dataframe tbody tr th:only-of-type {\n",
 83 |        "        vertical-align: middle;\n",
 84 |        "    }\n",
 85 |        "\n",
 86 |        "    .dataframe tbody tr th {\n",
 87 |        "        vertical-align: top;\n",
 88 |        "    }\n",
 89 |        "\n",
 90 |        "    .dataframe thead th {\n",
 91 |        "        text-align: right;\n",
 92 |        "    }\n",
 93 |        "</style>\n",
 94 |        "<table border=\"1\" class=\"dataframe\">\n",
 95 |        "  <thead>\n",
 96 |        "    <tr style=\"text-align: right;\">\n",
 97 |        "      <th></th>\n",
 98 |        "      <th>title</th>\n",
 99 |        "      <th>question</th>\n",
100 |        "      <th>reply</th>\n",
101 |        "      <th>is_best</th>\n",
102 |        "    </tr>\n",
103 |        "  </thead>\n",
104 |        "  <tbody>\n",
105 |        "    <tr>\n",
106 |        "      <th>104525</th>\n",
107 |        "      <td>拖欠联通话费会有利息出现吗？</td>\n",
108 |        "      <td>NaN</td>\n",
109 |        "      <td>应该没有</td>\n",
110 |        "      <td>0</td>\n",
111 |        "    </tr>\n",
112 |        "    <tr>\n",
113 |        "      <th>116168</th>\n",
114 |        "      <td>5S日版为什么插移动卡可以用.联通卡就不读卡</td>\n",
115 |        "      <td>NaN</td>\n",
116 |        "      <td>苹果手机卡贴分为移动和联通的，说明卡贴支持移动卡，不支持联通卡，主要是网络制式决定的。联通网...</td>\n",
117 |        "      <td>1</td>\n",
118 |        "    </tr>\n",
119 |        "    <tr>\n",
120 |        "      <th>154475</th>\n",
121 |        "      <td>联通空中号激活了也不能打电话是怎么回事</td>\n",
122 |        "      <td>联通空中号激活了也不能打电话是怎么回事</td>\n",
123 |        "      <td>手机已激活却无法接打电话的常见原因及解决方法如下：【1】检查手机是否欠费停机，建议缴费充值；...</td>\n",
124 |        "      <td>1</td>\n",
125 |        "    </tr>\n",
126 |        "    <tr>\n",
127 |        "      <th>153069</th>\n",
128 |        "      <td>联通48元送2g活动本月月租到底算不算进去？</td>\n",
129 |        "      <td>NaN</td>\n",
130 |        "      <td>算</td>\n",
131 |        "      <td>1</td>\n",
132 |        "    </tr>\n",
133 |        "    <tr>\n",
134 |        "      <th>195043</th>\n",
135 |        "      <td>VI###13是不是不支持联通上网卡</td>\n",
136 |        "      <td>NaN</td>\n",
137 |        "      <td>VI###13支持联通上网卡。网络参考：主屏尺寸：4.5英寸主屏分辨率：854x480像素后...</td>\n",
138 |        "      <td>1</td>\n",
139 |        "    </tr>\n",
140 |        "    <tr>\n",
141 |        "      <th>5235</th>\n",
142 |        "      <td>电话号码能定位是真吗</td>\n",
143 |        "      <td>电话号码能定位是真吗</td>\n",
144 |        "      <td>当然了</td>\n",
145 |        "      <td>0</td>\n",
146 |        "    </tr>\n",
147 |        "    <tr>\n",
148 |        "      <th>10472</th>\n",
149 |        "      <td>索尼LT22i可以刷机到4.1吗</td>\n",
150 |        "      <td>NaN</td>\n",
151 |        "      <td>由于手机所支持的网络是由硬件所确定的，无法通过破解软件或者升级软件系统让手机支持其他运营商的...</td>\n",
152 |        "      <td>1</td>\n",
153 |        "    </tr>\n",
154 |        "    <tr>\n",
155 |        "      <th>86083</th>\n",
156 |        "      <td>苹果ip##ne手机的个人热点怎么设置使用</td>\n",
157 |        "      <td>NaN</td>\n",
158 |        "      <td>1、点击“设置”选项;2、在“设置”界面中找到“个人热点”;3、然后我们可以看到“个人热点”...</td>\n",
159 |        "      <td>0</td>\n",
160 |        "    </tr>\n",
161 |        "    <tr>\n",
162 |        "      <th>150247</th>\n",
163 |        "      <td>我用的联通的号码，信号一会有一会没有，请问到底是怎什么回事</td>\n",
164 |        "      <td>NaN</td>\n",
165 |        "      <td>信号不好，手机因素，运营商问题，手机卡问题，很多因素你可以到当地联通营业厅寻求帮助</td>\n",
166 |        "      <td>0</td>\n",
167 |        "    </tr>\n",
168 |        "    <tr>\n",
169 |        "      <th>202724</th>\n",
170 |        "      <td>流量畅享包订购生效时间</td>\n",
171 |        "      <td>NaN</td>\n",
172 |        "      <td>您订购沃商店/沃游戏流量畅享包后，订购当月立即生效，按月自动续订；退订月底生效，当月可继续使...</td>\n",
173 |        "      <td>1</td>\n",
174 |        "    </tr>\n",
175 |        "    <tr>\n",
176 |        "      <th>44450</th>\n",
177 |        "      <td>办了腾讯大王卡，激活后，身份证是不是就剩下半俩张卡的机会了</td>\n",
178 |        "      <td>办了腾讯大王卡，激活后，身份证是不是就剩下半俩张卡的机会了</td>\n",
179 |        "      <td>每人仅可预约一张音乐小*卡或视频小*卡或腾讯大*卡或腾讯天*卡（一共仅1张）（识别条件为：联...</td>\n",
180 |        "      <td>1</td>\n",
181 |        "    </tr>\n",
182 |        "    <tr>\n",
183 |        "      <th>65875</th>\n",
184 |        "      <td>现在有联通的合约机吗</td>\n",
185 |        "      <td>NaN</td>\n",
186 |        "      <td>联通有合约机。合约种类大致有存话费送手机、买手机送话费、合约惠机等，具体合约种类可登录联通网...</td>\n",
187 |        "      <td>0</td>\n",
188 |        "    </tr>\n",
189 |        "    <tr>\n",
190 |        "      <th>51934</th>\n",
191 |        "      <td>联通卡那个腾讯应用省内定向流量免费是什么意思啊</td>\n",
192 |        "      <td>NaN</td>\n",
193 |        "      <td>大王卡，对腾讯的应用，都免流量！</td>\n",
194 |        "      <td>0</td>\n",
195 |        "    </tr>\n",
196 |        "    <tr>\n",
197 |        "      <th>155866</th>\n",
198 |        "      <td>怎么设置电信手机彩铃？</td>\n",
199 |        "      <td>NaN</td>\n",
200 |        "      <td>设置中*电信的彩铃可以自己在网上操作的，前提是先开通中*电信的彩铃业务，可以直接致电电信客服...</td>\n",
201 |        "      <td>1</td>\n",
202 |        "    </tr>\n",
203 |        "    <tr>\n",
204 |        "      <th>118696</th>\n",
205 |        "      <td>联通手机号挂失还能交费吗</td>\n",
206 |        "      <td>NaN</td>\n",
207 |        "      <td>1、挂失状态下可以交费。交费渠道与手机正常状态下是一样的。2、温馨提示：如果号码有套餐，挂失...</td>\n",
208 |        "      <td>1</td>\n",
209 |        "    </tr>\n",
210 |        "    <tr>\n",
211 |        "      <th>115890</th>\n",
212 |        "      <td>我刚买了一张联通卡，过了几天我怎么收到达飞即有分期让我还款的信息</td>\n",
213 |        "      <td>我刚买了一张联通卡，过了几天我怎么收到达飞即有分期让我还款的信息，我又没有借过，该怎么办，打...</td>\n",
214 |        "      <td>出现此情况一般是有以下几种情况：1、信息可能发错接收人了。2、此卡为二次放号的手机卡，前一个...</td>\n",
215 |        "      <td>1</td>\n",
216 |        "    </tr>\n",
217 |        "    <tr>\n",
218 |        "      <th>35555</th>\n",
219 |        "      <td>就是联通网用不了。</td>\n",
220 |        "      <td>NaN</td>\n",
221 |        "      <td>如使用联通手机无法上网，可做以下排查：1、升级为4G套餐后如不重启手机则无法正常使用上网功能...</td>\n",
222 |        "      <td>1</td>\n",
223 |        "    </tr>\n",
224 |        "    <tr>\n",
225 |        "      <th>73496</th>\n",
226 |        "      <td>生份证复印件被公司拿去开联通号码了怎么办</td>\n",
227 |        "      <td>生份证复印件被公司拿去开联通号码了怎么办</td>\n",
228 |        "      <td>你再用原件去注销</td>\n",
229 |        "      <td>0</td>\n",
230 |        "    </tr>\n",
231 |        "    <tr>\n",
232 |        "      <th>114899</th>\n",
233 |        "      <td>手机有4g网络，可是却显示无法上网</td>\n",
234 |        "      <td>NaN</td>\n",
235 |        "      <td>1、检查信号是否正常；2、号卡是否欠费；3、如上面2项都正常，可重新开关机、换机换卡测试；4...</td>\n",
236 |        "      <td>1</td>\n",
237 |        "    </tr>\n",
238 |        "    <tr>\n",
239 |        "      <th>47230</th>\n",
240 |        "      <td>移动，联通无限打到底是怎么回事</td>\n",
241 |        "      <td>NaN</td>\n",
242 |        "      <td>您好！现运营商均有推出各种语音、流量优惠套餐，具体情况建议您可咨询当地客服热线、实体营业厅、...</td>\n",
243 |        "      <td>1</td>\n",
244 |        "    </tr>\n",
245 |        "  </tbody>\n",
246 |        "</table>\n",
247 |        "</div>"
248 |       ],
249 |       "text/plain": [
250 |        "                                   title  \\\n",
251 |        "104525                    拖欠联通话费会有利息出现吗？   \n",
252 |        "116168            5S日版为什么插移动卡可以用.联通卡就不读卡   \n",
253 |        "154475               联通空中号激活了也不能打电话是怎么回事   \n",
254 |        "153069            联通48元送2g活动本月月租到底算不算进去？   \n",
255 |        "195043                VI###13是不是不支持联通上网卡   \n",
256 |        "5235                          电话号码能定位是真吗   \n",
257 |        "10472                   索尼LT22i可以刷机到4.1吗   \n",
258 |        "86083              苹果ip##ne手机的个人热点怎么设置使用   \n",
259 |        "150247     我用的联通的号码，信号一会有一会没有，请问到底是怎什么回事   \n",
260 |        "202724                       流量畅享包订购生效时间   \n",
261 |        "44450      办了腾讯大王卡，激活后，身份证是不是就剩下半俩张卡的机会了   \n",
262 |        "65875                         现在有联通的合约机吗   \n",
263 |        "51934            联通卡那个腾讯应用省内定向流量免费是什么意思啊   \n",
264 |        "155866                       怎么设置电信手机彩铃？   \n",
265 |        "118696                      联通手机号挂失还能交费吗   \n",
266 |        "115890  我刚买了一张联通卡，过了几天我怎么收到达飞即有分期让我还款的信息   \n",
267 |        "35555                          就是联通网用不了。   \n",
268 |        "73496               生份证复印件被公司拿去开联通号码了怎么办   \n",
269 |        "114899                 手机有4g网络，可是却显示无法上网   \n",
270 |        "47230                    移动，联通无限打到底是怎么回事   \n",
271 |        "\n",
272 |        "                                                 question  \\\n",
273 |        "104525                                                NaN   \n",
274 |        "116168                                                NaN   \n",
275 |        "154475                                联通空中号激活了也不能打电话是怎么回事   \n",
276 |        "153069                                                NaN   \n",
277 |        "195043                                                NaN   \n",
278 |        "5235                                           电话号码能定位是真吗   \n",
279 |        "10472                                                 NaN   \n",
280 |        "86083                                                 NaN   \n",
281 |        "150247                                                NaN   \n",
282 |        "202724                                                NaN   \n",
283 |        "44450                       办了腾讯大王卡，激活后，身份证是不是就剩下半俩张卡的机会了   \n",
284 |        "65875                                                 NaN   \n",
285 |        "51934                                                 NaN   \n",
286 |        "155866                                                NaN   \n",
287 |        "118696                                                NaN   \n",
288 |        "115890  我刚买了一张联通卡，过了几天我怎么收到达飞即有分期让我还款的信息，我又没有借过，该怎么办，打...   \n",
289 |        "35555                                                 NaN   \n",
290 |        "73496                                生份证复印件被公司拿去开联通号码了怎么办   \n",
291 |        "114899                                                NaN   \n",
292 |        "47230                                                 NaN   \n",
293 |        "\n",
294 |        "                                                    reply  is_best  \n",
295 |        "104525                                               应该没有        0  \n",
296 |        "116168  苹果手机卡贴分为移动和联通的，说明卡贴支持移动卡，不支持联通卡，主要是网络制式决定的。联通网...        1  \n",
297 |        "154475  手机已激活却无法接打电话的常见原因及解决方法如下：【1】检查手机是否欠费停机，建议缴费充值；...        1  \n",
298 |        "153069                                                  算        1  \n",
299 |        "195043  VI###13支持联通上网卡。网络参考：主屏尺寸：4.5英寸主屏分辨率：854x480像素后...        1  \n",
300 |        "5235                                                  当然了        0  \n",
301 |        "10472   由于手机所支持的网络是由硬件所确定的，无法通过破解软件或者升级软件系统让手机支持其他运营商的...        1  \n",
302 |        "86083   1、点击“设置”选项;2、在“设置”界面中找到“个人热点”;3、然后我们可以看到“个人热点”...        0  \n",
303 |        "150247          信号不好，手机因素，运营商问题，手机卡问题，很多因素你可以到当地联通营业厅寻求帮助        0  \n",
304 |        "202724  您订购沃商店/沃游戏流量畅享包后，订购当月立即生效，按月自动续订；退订月底生效，当月可继续使...        1  \n",
305 |        "44450   每人仅可预约一张音乐小*卡或视频小*卡或腾讯大*卡或腾讯天*卡（一共仅1张）（识别条件为：联...        1  \n",
306 |        "65875   联通有合约机。合约种类大致有存话费送手机、买手机送话费、合约惠机等，具体合约种类可登录联通网...        0  \n",
307 |        "51934                                    大王卡，对腾讯的应用，都免流量！        0  \n",
308 |        "155866  设置中*电信的彩铃可以自己在网上操作的，前提是先开通中*电信的彩铃业务，可以直接致电电信客服...        1  \n",
309 |        "118696  1、挂失状态下可以交费。交费渠道与手机正常状态下是一样的。2、温馨提示：如果号码有套餐，挂失...        1  \n",
310 |        "115890  出现此情况一般是有以下几种情况：1、信息可能发错接收人了。2、此卡为二次放号的手机卡，前一个...        1  \n",
311 |        "35555   如使用联通手机无法上网，可做以下排查：1、升级为4G套餐后如不重启手机则无法正常使用上网功能...        1  \n",
312 |        "73496                                            你再用原件去注销        0  \n",
313 |        "114899  1、检查信号是否正常；2、号卡是否欠费；3、如上面2项都正常，可重新开关机、换机换卡测试；4...        1  \n",
314 |        "47230   您好！现运营商均有推出各种语音、流量优惠套餐，具体情况建议您可咨询当地客服热线、实体营业厅、...        1  "
315 |       ]
316 |      },
317 |      "execution_count": 4,
318 |      "metadata": {},
319 |      "output_type": "execute_result"
320 |     }
321 |    ],
322 |    "source": [
323 |     "pd_all.sample(n=20)"
324 |    ]
325 |   },
326 |   {
327 |    "cell_type": "code",
328 |    "execution_count": null,
329 |    "metadata": {},
330 |    "outputs": [],
331 |    "source": []
332 |   }
333 |  ],
334 |  "metadata": {
335 |   "kernelspec": {
336 |    "display_name": "Python 3",
337 |    "language": "python",
338 |    "name": "python3"
339 |   },
340 |   "language_info": {
341 |    "codemirror_mode": {
342 |     "name": "ipython",
343 |     "version": 3
344 |    },
345 |    "file_extension": ".py",
346 |    "mimetype": "text/x-python",
347 |    "name": "python",
348 |    "nbconvert_exporter": "python",
349 |    "pygments_lexer": "ipython3",
350 |    "version": "3.7.0"
351 |   }
352 |  },
353 |  "nbformat": 4,
354 |  "nbformat_minor": 2
355 | }
356 | 


--------------------------------------------------------------------------------
/datasets/nonghangzhidao/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# nonghangzhidao_filter 说明\n",
  8 |     "0. **下载地址：** [百度网盘](https://pan.baidu.com/s/1n-jT9SKkt6cwI_PjCd7i_g)\n",
  9 |     "1. **数据概览：** 4万 条农业银行问答数据\n",
 10 |     "2. **推荐实验：** FAQ 问答系统\n",
 11 |     "3. **数据来源：** 百度知道\n",
 12 |     "4. **加工处理：**\n",
 13 |     "    1. 过滤了id、url、qid、reply_t、user字段\n",
 14 |     "    2. 对question、reply做了脱敏处理"
 15 |    ]
 16 |   },
 17 |   {
 18 |    "cell_type": "code",
 19 |    "execution_count": 1,
 20 |    "metadata": {},
 21 |    "outputs": [],
 22 |    "source": [
 23 |     "import pandas as pd"
 24 |    ]
 25 |   },
 26 |   {
 27 |    "cell_type": "code",
 28 |    "execution_count": 2,
 29 |    "metadata": {},
 30 |    "outputs": [],
 31 |    "source": [
 32 |     "path = 'nonghangzhidao_文件夹_所在_路径'"
 33 |    ]
 34 |   },
 35 |   {
 36 |    "cell_type": "markdown",
 37 |    "metadata": {},
 38 |    "source": [
 39 |     "# 1. nonghangzhidao_filter.csv"
 40 |    ]
 41 |   },
 42 |   {
 43 |    "cell_type": "markdown",
 44 |    "metadata": {},
 45 |    "source": [
 46 |     "## 加载数据"
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "code",
 51 |    "execution_count": 3,
 52 |    "metadata": {},
 53 |    "outputs": [],
 54 |    "source": [
 55 |     "pd_all = pd.read_csv(path + 'nonghangzhidao_filter.csv')"
 56 |    ]
 57 |   },
 58 |   {
 59 |    "cell_type": "markdown",
 60 |    "metadata": {},
 61 |    "source": [
 62 |     "## 字段说明\n",
 63 |     "\n",
 64 |     "| 字段 | 说明 |\n",
 65 |     "| ---- | ---- |\n",
 66 |     "| title | 问题的标题 |\n",
 67 |     "| question | 问题内容（可为空） |\n",
 68 |     "| reply| 回复内容 |\n",
 69 |     "| is_best| 是否为页面上显示的最佳回答 |"
 70 |    ]
 71 |   },
 72 |   {
 73 |    "cell_type": "code",
 74 |    "execution_count": 4,
 75 |    "metadata": {},
 76 |    "outputs": [
 77 |     {
 78 |      "data": {
 79 |       "text/html": [
 80 |        "<div>\n",
 81 |        "<style scoped>\n",
 82 |        "    .dataframe tbody tr th:only-of-type {\n",
 83 |        "        vertical-align: middle;\n",
 84 |        "    }\n",
 85 |        "\n",
 86 |        "    .dataframe tbody tr th {\n",
 87 |        "        vertical-align: top;\n",
 88 |        "    }\n",
 89 |        "\n",
 90 |        "    .dataframe thead th {\n",
 91 |        "        text-align: right;\n",
 92 |        "    }\n",
 93 |        "</style>\n",
 94 |        "<table border=\"1\" class=\"dataframe\">\n",
 95 |        "  <thead>\n",
 96 |        "    <tr style=\"text-align: right;\">\n",
 97 |        "      <th></th>\n",
 98 |        "      <th>title</th>\n",
 99 |        "      <th>question</th>\n",
100 |        "      <th>reply</th>\n",
101 |        "      <th>is_best</th>\n",
102 |        "    </tr>\n",
103 |        "  </thead>\n",
104 |        "  <tbody>\n",
105 |        "    <tr>\n",
106 |        "      <th>31655</th>\n",
107 |        "      <td>广东农行转账到江苏农行，几天可以到账？1月4日晚上10点多转的！</td>\n",
108 |        "      <td>NaN</td>\n",
109 |        "      <td>这么久还没有到账的话，建议查询一下是否被退回了，如果未退回的话，需要联系银行查询原因。</td>\n",
110 |        "      <td>0</td>\n",
111 |        "    </tr>\n",
112 |        "    <tr>\n",
113 |        "      <th>20349</th>\n",
114 |        "      <td>惠水哪里有小额贷款的，而且抵押的东西能方</td>\n",
115 |        "      <td>NaN</td>\n",
116 |        "      <td>留vx..</td>\n",
117 |        "      <td>0</td>\n",
118 |        "    </tr>\n",
119 |        "    <tr>\n",
120 |        "      <th>20303</th>\n",
121 |        "      <td>想问一下重庆分行的体检通知还有第二批吗</td>\n",
122 |        "      <td>NaN</td>\n",
123 |        "      <td>若客户申请的是农行招聘，则可以参考以下信息：1、请登录农行官网，在“关于农行”栏目下选择点击...</td>\n",
124 |        "      <td>1</td>\n",
125 |        "    </tr>\n",
126 |        "    <tr>\n",
127 |        "      <th>18420</th>\n",
128 |        "      <td>现在有什么软件借钱可以秒过的没。江湖救急</td>\n",
129 |        "      <td>NaN</td>\n",
130 |        "      <td>资料真实有效二十分钟放款</td>\n",
131 |        "      <td>0</td>\n",
132 |        "    </tr>\n",
133 |        "    <tr>\n",
134 |        "      <th>39804</th>\n",
135 |        "      <td>想找高利贷怎么找?</td>\n",
136 |        "      <td>武*那里有高利贷啊？接个几千块就行年后还？有吗</td>\n",
137 |        "      <td>留你联系方式</td>\n",
138 |        "      <td>0</td>\n",
139 |        "    </tr>\n",
140 |        "    <tr>\n",
141 |        "      <th>23242</th>\n",
142 |        "      <td>别人用建行卡往我农行卡转了20万，一天了怎么还不到账？</td>\n",
143 |        "      <td>别人用建行卡往我农行卡转了20万，一天了怎么还不到账？肯定是</td>\n",
144 |        "      <td>如果是昨天下午五点后就要等到中午以后</td>\n",
145 |        "      <td>0</td>\n",
146 |        "    </tr>\n",
147 |        "    <tr>\n",
148 |        "      <th>30656</th>\n",
149 |        "      <td>我问银行了，说消不了户、只能刷掉</td>\n",
150 |        "      <td>NaN</td>\n",
151 |        "      <td>如果使用的是农行信用卡，可以致电信用卡客服40######99反馈核实一下。</td>\n",
152 |        "      <td>0</td>\n",
153 |        "    </tr>\n",
154 |        "    <tr>\n",
155 |        "      <th>8</th>\n",
156 |        "      <td>贷的太少了，可以提前还清贷款，然后多贷点他用吗</td>\n",
157 |        "      <td>NaN</td>\n",
158 |        "      <td>建议客户选择正规渠道申请贷款，例如农行“网捷贷”。网捷贷是指农业银行向符合特定条件的农业银行...</td>\n",
159 |        "      <td>0</td>\n",
160 |        "    </tr>\n",
161 |        "    <tr>\n",
162 |        "      <th>2961</th>\n",
163 |        "      <td>银行理财和证券公司理财一样吗</td>\n",
164 |        "      <td>NaN</td>\n",
165 |        "      <td>不太一样，产品的种类风险不同</td>\n",
166 |        "      <td>0</td>\n",
167 |        "    </tr>\n",
168 |        "    <tr>\n",
169 |        "      <th>10837</th>\n",
170 |        "      <td>农行提额喜欢刷大额是还是小额</td>\n",
171 |        "      <td>NaN</td>\n",
172 |        "      <td>老农现在是印头与时俱进哦比其他银行都大方。</td>\n",
173 |        "      <td>0</td>\n",
174 |        "    </tr>\n",
175 |        "    <tr>\n",
176 |        "      <th>34726</th>\n",
177 |        "      <td>存折可以异地取款吗存折取钱一定要本人吗</td>\n",
178 |        "      <td>NaN</td>\n",
179 |        "      <td>农行个人活期存折支取方式里如果有凭证件支取，此类存折必须户主本人办理；没有密码的存折只能到开...</td>\n",
180 |        "      <td>1</td>\n",
181 |        "    </tr>\n",
182 |        "    <tr>\n",
183 |        "      <th>2381</th>\n",
184 |        "      <td>住房公积金可以首付么</td>\n",
185 |        "      <td>NaN</td>\n",
186 |        "      <td>不能用公积金来付首付。这个贷款是在购房付了首付款后才能给贷的，也就是说公积金使用只能是与房屋...</td>\n",
187 |        "      <td>1</td>\n",
188 |        "    </tr>\n",
189 |        "    <tr>\n",
190 |        "      <th>21717</th>\n",
191 |        "      <td>农行有银联标识的社会保障卡能开通网银吗？</td>\n",
192 |        "      <td>如题</td>\n",
193 |        "      <td>由农业银行发行的有银联标识的社会保障卡，上面如果有农业银行卡号的话，是可以用本人由身份证和银...</td>\n",
194 |        "      <td>1</td>\n",
195 |        "    </tr>\n",
196 |        "    <tr>\n",
197 |        "      <th>35871</th>\n",
198 |        "      <td>求告知宁*装修贷款条件有哪些</td>\n",
199 |        "      <td>NaN</td>\n",
200 |        "      <td>以建行家装贷为例：“家装贷”是建设银行所有具有装修融资服务功能的个人贷款产品，包括个人住房抵...</td>\n",
201 |        "      <td>1</td>\n",
202 |        "    </tr>\n",
203 |        "    <tr>\n",
204 |        "      <th>3417</th>\n",
205 |        "      <td>重*公积金中介能代取的吗</td>\n",
206 |        "      <td>重*公积金中介能代取的吗个人公积金代取</td>\n",
207 |        "      <td>一般来说，公积金套现主要存在几个方面的风险：一、中介机构提取完公积金后，有可能会携款潜逃，竹...</td>\n",
208 |        "      <td>1</td>\n",
209 |        "    </tr>\n",
210 |        "    <tr>\n",
211 |        "      <th>25763</th>\n",
212 |        "      <td>农行的理财产品能买吗</td>\n",
213 |        "      <td>NaN</td>\n",
214 |        "      <td>农行理财业务与国内同业同步，迄今为止，已经形成了制度体系较为完善、系统开发不断前进、产品系列...</td>\n",
215 |        "      <td>1</td>\n",
216 |        "    </tr>\n",
217 |        "    <tr>\n",
218 |        "      <th>13381</th>\n",
219 |        "      <td>出大事了，出大事了，急用钱，请问我</td>\n",
220 |        "      <td>NaN</td>\n",
221 |        "      <td>需要多少呢</td>\n",
222 |        "      <td>0</td>\n",
223 |        "    </tr>\n",
224 |        "    <tr>\n",
225 |        "      <th>7448</th>\n",
226 |        "      <td>向钱贷会跑路吗？</td>\n",
227 |        "      <td>NaN</td>\n",
228 |        "      <td>不会，放心。</td>\n",
229 |        "      <td>0</td>\n",
230 |        "    </tr>\n",
231 |        "    <tr>\n",
232 |        "      <th>2617</th>\n",
233 |        "      <td>我想开通农业掌上银行提示要开通短信，可以先开通短信把掌上银行开通后，取消短信服务吗？对其他有影响吗</td>\n",
234 |        "      <td>我想开通农业掌上银行提示要开通短信，可以先开通短信把掌上银行开通后，取消短信服务吗？对其他有...</td>\n",
235 |        "      <td>我想开通农业掌上银行提示要开通短信，可以先开通短信把掌上银行开通后，取消短信服务吗？对其他有...</td>\n",
236 |        "      <td>0</td>\n",
237 |        "    </tr>\n",
238 |        "    <tr>\n",
239 |        "      <th>24669</th>\n",
240 |        "      <td>朋友晚上8点半转账给我到现在还没到帐</td>\n",
241 |        "      <td>NaN</td>\n",
242 |        "      <td>现在外面ATM机都是24小时才到帐</td>\n",
243 |        "      <td>0</td>\n",
244 |        "    </tr>\n",
245 |        "  </tbody>\n",
246 |        "</table>\n",
247 |        "</div>"
248 |       ],
249 |       "text/plain": [
250 |        "                                                   title  \\\n",
251 |        "31655                   广东农行转账到江苏农行，几天可以到账？1月4日晚上10点多转的！   \n",
252 |        "20349                               惠水哪里有小额贷款的，而且抵押的东西能方   \n",
253 |        "20303                                想问一下重庆分行的体检通知还有第二批吗   \n",
254 |        "18420                               现在有什么软件借钱可以秒过的没。江湖救急   \n",
255 |        "39804                                          想找高利贷怎么找?   \n",
256 |        "23242                        别人用建行卡往我农行卡转了20万，一天了怎么还不到账？   \n",
257 |        "30656                                   我问银行了，说消不了户、只能刷掉   \n",
258 |        "8                                贷的太少了，可以提前还清贷款，然后多贷点他用吗   \n",
259 |        "2961                                      银行理财和证券公司理财一样吗   \n",
260 |        "10837                                     农行提额喜欢刷大额是还是小额   \n",
261 |        "34726                                存折可以异地取款吗存折取钱一定要本人吗   \n",
262 |        "2381                                          住房公积金可以首付么   \n",
263 |        "21717                               农行有银联标识的社会保障卡能开通网银吗？   \n",
264 |        "35871                                     求告知宁*装修贷款条件有哪些   \n",
265 |        "3417                                        重*公积金中介能代取的吗   \n",
266 |        "25763                                         农行的理财产品能买吗   \n",
267 |        "13381                                  出大事了，出大事了，急用钱，请问我   \n",
268 |        "7448                                            向钱贷会跑路吗？   \n",
269 |        "2617   我想开通农业掌上银行提示要开通短信，可以先开通短信把掌上银行开通后，取消短信服务吗？对其他有影响吗   \n",
270 |        "24669                                 朋友晚上8点半转账给我到现在还没到帐   \n",
271 |        "\n",
272 |        "                                                question  \\\n",
273 |        "31655                                                NaN   \n",
274 |        "20349                                                NaN   \n",
275 |        "20303                                                NaN   \n",
276 |        "18420                                                NaN   \n",
277 |        "39804                            武*那里有高利贷啊？接个几千块就行年后还？有吗   \n",
278 |        "23242                     别人用建行卡往我农行卡转了20万，一天了怎么还不到账？肯定是   \n",
279 |        "30656                                                NaN   \n",
280 |        "8                                                    NaN   \n",
281 |        "2961                                                 NaN   \n",
282 |        "10837                                                NaN   \n",
283 |        "34726                                                NaN   \n",
284 |        "2381                                                 NaN   \n",
285 |        "21717                                                 如题   \n",
286 |        "35871                                                NaN   \n",
287 |        "3417                                 重*公积金中介能代取的吗个人公积金代取   \n",
288 |        "25763                                                NaN   \n",
289 |        "13381                                                NaN   \n",
290 |        "7448                                                 NaN   \n",
291 |        "2617   我想开通农业掌上银行提示要开通短信，可以先开通短信把掌上银行开通后，取消短信服务吗？对其他有...   \n",
292 |        "24669                                                NaN   \n",
293 |        "\n",
294 |        "                                                   reply  is_best  \n",
295 |        "31655        这么久还没有到账的话，建议查询一下是否被退回了，如果未退回的话，需要联系银行查询原因。        0  \n",
296 |        "20349                                              留vx..        0  \n",
297 |        "20303  若客户申请的是农行招聘，则可以参考以下信息：1、请登录农行官网，在“关于农行”栏目下选择点击...        1  \n",
298 |        "18420                                       资料真实有效二十分钟放款        0  \n",
299 |        "39804                                             留你联系方式        0  \n",
300 |        "23242                                 如果是昨天下午五点后就要等到中午以后        0  \n",
301 |        "30656             如果使用的是农行信用卡，可以致电信用卡客服40######99反馈核实一下。        0  \n",
302 |        "8      建议客户选择正规渠道申请贷款，例如农行“网捷贷”。网捷贷是指农业银行向符合特定条件的农业银行...        0  \n",
303 |        "2961                                      不太一样，产品的种类风险不同        0  \n",
304 |        "10837                              老农现在是印头与时俱进哦比其他银行都大方。        0  \n",
305 |        "34726  农行个人活期存折支取方式里如果有凭证件支取，此类存折必须户主本人办理；没有密码的存折只能到开...        1  \n",
306 |        "2381   不能用公积金来付首付。这个贷款是在购房付了首付款后才能给贷的，也就是说公积金使用只能是与房屋...        1  \n",
307 |        "21717  由农业银行发行的有银联标识的社会保障卡，上面如果有农业银行卡号的话，是可以用本人由身份证和银...        1  \n",
308 |        "35871  以建行家装贷为例：“家装贷”是建设银行所有具有装修融资服务功能的个人贷款产品，包括个人住房抵...        1  \n",
309 |        "3417   一般来说，公积金套现主要存在几个方面的风险：一、中介机构提取完公积金后，有可能会携款潜逃，竹...        1  \n",
310 |        "25763  农行理财业务与国内同业同步，迄今为止，已经形成了制度体系较为完善、系统开发不断前进、产品系列...        1  \n",
311 |        "13381                                              需要多少呢        0  \n",
312 |        "7448                                              不会，放心。        0  \n",
313 |        "2617   我想开通农业掌上银行提示要开通短信，可以先开通短信把掌上银行开通后，取消短信服务吗？对其他有...        0  \n",
314 |        "24669                                  现在外面ATM机都是24小时才到帐        0  "
315 |       ]
316 |      },
317 |      "execution_count": 4,
318 |      "metadata": {},
319 |      "output_type": "execute_result"
320 |     }
321 |    ],
322 |    "source": [
323 |     "pd_all.sample(n=20)"
324 |    ]
325 |   },
326 |   {
327 |    "cell_type": "code",
328 |    "execution_count": null,
329 |    "metadata": {},
330 |    "outputs": [],
331 |    "source": []
332 |   }
333 |  ],
334 |  "metadata": {
335 |   "kernelspec": {
336 |    "display_name": "Python 3",
337 |    "language": "python",
338 |    "name": "python3"
339 |   },
340 |   "language_info": {
341 |    "codemirror_mode": {
342 |     "name": "ipython",
343 |     "version": 3
344 |    },
345 |    "file_extension": ".py",
346 |    "mimetype": "text/x-python",
347 |    "name": "python",
348 |    "nbconvert_exporter": "python",
349 |    "pygments_lexer": "ipython3",
350 |    "version": "3.7.0"
351 |   }
352 |  },
353 |  "nbformat": 4,
354 |  "nbformat_minor": 2
355 | }
356 | 


--------------------------------------------------------------------------------
/datasets/online_shopping_10_cats/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# online_shopping_10_cats 说明\n",
  8 |     "0. **下载地址：** [Github](https://github.com/SophonPlus/ChineseNlpCorpus/raw/master/datasets/online_shopping_10_cats/online_shopping_10_cats.zip)\n",
  9 |     "1. **数据概览：** 10 个类别（书籍、平板、手机、水果、洗发水、热水器、蒙牛、衣服、计算机、酒店），共 6 万多条评论数据，正、负向评论各约 3 万条\n",
 10 |     "2. **推荐实验：** 情感/观点/评论 倾向性分析\n",
 11 |     "2. **数据来源：** 各电商平台，具体不详\n",
 12 |     "3. **原数据集：** [中文情感分析语料](https://download.csdn.net/download/weixin_38395744/10231401)、[中文情感分析语料库](https://download.csdn.net/download/u010097581/9919245)，网上搜集，具体作者、来源不详\n",
 13 |     "4. **加工处理：**\n",
 14 |     "    1. 将 2 份语料整合成 1 份语料\n",
 15 |     "    2. 将原来零散的 excel, txt 文档，整合成 1 个 统一的 csv 文档\n",
 16 |     "    3. 去重"
 17 |    ]
 18 |   },
 19 |   {
 20 |    "cell_type": "code",
 21 |    "execution_count": 24,
 22 |    "metadata": {},
 23 |    "outputs": [],
 24 |    "source": [
 25 |     "import pandas as pd"
 26 |    ]
 27 |   },
 28 |   {
 29 |    "cell_type": "code",
 30 |    "execution_count": 25,
 31 |    "metadata": {},
 32 |    "outputs": [],
 33 |    "source": [
 34 |     "path = 'online_shopping_10_cats_文件夹_所在_路径'"
 35 |    ]
 36 |   },
 37 |   {
 38 |    "cell_type": "markdown",
 39 |    "metadata": {},
 40 |    "source": [
 41 |     "# 1. online_shopping_10_cats.csv"
 42 |    ]
 43 |   },
 44 |   {
 45 |    "cell_type": "markdown",
 46 |    "metadata": {},
 47 |    "source": [
 48 |     "## 加载数据"
 49 |    ]
 50 |   },
 51 |   {
 52 |    "cell_type": "code",
 53 |    "execution_count": 26,
 54 |    "metadata": {},
 55 |    "outputs": [
 56 |     {
 57 |      "name": "stdout",
 58 |      "output_type": "stream",
 59 |      "text": [
 60 |       "评论数目（总体）：62774\n",
 61 |       "评论数目（正向）：31728\n",
 62 |       "评论数目（负向）：31046\n"
 63 |      ]
 64 |     }
 65 |    ],
 66 |    "source": [
 67 |     "pd_all = pd.read_csv(path + 'online_shopping_10_cats.csv')\n",
 68 |     "\n",
 69 |     "print('评论数目（总体）：%d' % pd_all.shape[0])\n",
 70 |     "print('评论数目（正向）：%d' % pd_all[pd_all.label==1].shape[0])\n",
 71 |     "print('评论数目（负向）：%d' % pd_all[pd_all.label==0].shape[0])"
 72 |    ]
 73 |   },
 74 |   {
 75 |    "cell_type": "markdown",
 76 |    "metadata": {},
 77 |    "source": [
 78 |     "## 字段说明\n",
 79 |     "\n",
 80 |     "| 字段 | 说明 |\n",
 81 |     "| ---- | ---- |\n",
 82 |     "| cat | 类别：包括 书籍、平板、手机、水果、洗发水、热水器、蒙牛、衣服、计算机、酒店 |\n",
 83 |     "| label | 1 表示正向评论，0 表示负向评论 |\n",
 84 |     "| review | 评论内容 |"
 85 |    ]
 86 |   },
 87 |   {
 88 |    "cell_type": "code",
 89 |    "execution_count": 27,
 90 |    "metadata": {},
 91 |    "outputs": [
 92 |     {
 93 |      "data": {
 94 |       "text/html": [
 95 |        "<div>\n",
 96 |        "<style scoped>\n",
 97 |        "    .dataframe tbody tr th:only-of-type {\n",
 98 |        "        vertical-align: middle;\n",
 99 |        "    }\n",
100 |        "\n",
101 |        "    .dataframe tbody tr th {\n",
102 |        "        vertical-align: top;\n",
103 |        "    }\n",
104 |        "\n",
105 |        "    .dataframe thead th {\n",
106 |        "        text-align: right;\n",
107 |        "    }\n",
108 |        "</style>\n",
109 |        "<table border=\"1\" class=\"dataframe\">\n",
110 |        "  <thead>\n",
111 |        "    <tr style=\"text-align: right;\">\n",
112 |        "      <th></th>\n",
113 |        "      <th>cat</th>\n",
114 |        "      <th>label</th>\n",
115 |        "      <th>review</th>\n",
116 |        "    </tr>\n",
117 |        "  </thead>\n",
118 |        "  <tbody>\n",
119 |        "    <tr>\n",
120 |        "      <th>11194</th>\n",
121 |        "      <td>平板</td>\n",
122 |        "      <td>0</td>\n",
123 |        "      <td>什么玩意。刚用一天，就充不上电，开不开机，返厂老麻烦，</td>\n",
124 |        "    </tr>\n",
125 |        "    <tr>\n",
126 |        "      <th>17794</th>\n",
127 |        "      <td>水果</td>\n",
128 |        "      <td>1</td>\n",
129 |        "      <td>买了几次了，价格实惠，口感不错，保鲜好！</td>\n",
130 |        "    </tr>\n",
131 |        "    <tr>\n",
132 |        "      <th>29529</th>\n",
133 |        "      <td>洗发水</td>\n",
134 |        "      <td>1</td>\n",
135 |        "      <td>挺值得购买的，有包装买回去送家人，毛巾质量不错。小块的可以拿来当擦手帕。</td>\n",
136 |        "    </tr>\n",
137 |        "    <tr>\n",
138 |        "      <th>24976</th>\n",
139 |        "      <td>水果</td>\n",
140 |        "      <td>0</td>\n",
141 |        "      <td>真的就算后悔了。两天才拿到货。还不如水果店买！还都发霉不新鲜了！以后不买了</td>\n",
142 |        "    </tr>\n",
143 |        "    <tr>\n",
144 |        "      <th>28447</th>\n",
145 |        "      <td>洗发水</td>\n",
146 |        "      <td>1</td>\n",
147 |        "      <td>一般般，薄荷洗发水没想象中的凉快</td>\n",
148 |        "    </tr>\n",
149 |        "    <tr>\n",
150 |        "      <th>264</th>\n",
151 |        "      <td>书籍</td>\n",
152 |        "      <td>1</td>\n",
153 |        "      <td>这本书有别于以往看过的早教书籍，结合了说明文的写实，散文的情致和图册的一目了然。特别是读过几...</td>\n",
154 |        "    </tr>\n",
155 |        "    <tr>\n",
156 |        "      <th>53035</th>\n",
157 |        "      <td>酒店</td>\n",
158 |        "      <td>1</td>\n",
159 |        "      <td>酒店的大堂很漂亮,房间不算小,设施还可以也很干净,离码头很近,而且又有车接送,很方便.晚上2...</td>\n",
160 |        "    </tr>\n",
161 |        "    <tr>\n",
162 |        "      <th>50250</th>\n",
163 |        "      <td>计算机</td>\n",
164 |        "      <td>1</td>\n",
165 |        "      <td>做工不错，外壳也很漂亮。测试了一下还行！~中通很快啊，13号下午的订单，今天早上就收到了。</td>\n",
166 |        "    </tr>\n",
167 |        "    <tr>\n",
168 |        "      <th>62461</th>\n",
169 |        "      <td>酒店</td>\n",
170 |        "      <td>0</td>\n",
171 |        "      <td>房间空间比较小， 环境比较吵。特别半夜被窗户外面的空调外机的声音吵醒（因为窗外一条巷子之隔，...</td>\n",
172 |        "    </tr>\n",
173 |        "    <tr>\n",
174 |        "      <th>52888</th>\n",
175 |        "      <td>酒店</td>\n",
176 |        "      <td>1</td>\n",
177 |        "      <td>清明节入住两天.从进入酒店就感受到无处不在的服务,非常周到,又很得体.从大堂,商务中心,到前...</td>\n",
178 |        "    </tr>\n",
179 |        "    <tr>\n",
180 |        "      <th>31429</th>\n",
181 |        "      <td>洗发水</td>\n",
182 |        "      <td>0</td>\n",
183 |        "      <td>感觉不怎么样，刚刚洗完头发又感觉头发干枯枯的而且还是好油</td>\n",
184 |        "    </tr>\n",
185 |        "    <tr>\n",
186 |        "      <th>21443</th>\n",
187 |        "      <td>水果</td>\n",
188 |        "      <td>0</td>\n",
189 |        "      <td>算了，不要买了，先不说个头小，就味道难吃的要死，还没有路边摊卖的好吃，硬，涩，根本就没有苹果...</td>\n",
190 |        "    </tr>\n",
191 |        "    <tr>\n",
192 |        "      <th>19374</th>\n",
193 |        "      <td>水果</td>\n",
194 |        "      <td>1</td>\n",
195 |        "      <td>快递神速，品种与描述一样，比上次买的好吃！</td>\n",
196 |        "    </tr>\n",
197 |        "    <tr>\n",
198 |        "      <th>28188</th>\n",
199 |        "      <td>洗发水</td>\n",
200 |        "      <td>1</td>\n",
201 |        "      <td>还没有用，不过感觉和实体店买的差不多，等用过之后再追加评价吧</td>\n",
202 |        "    </tr>\n",
203 |        "    <tr>\n",
204 |        "      <th>46182</th>\n",
205 |        "      <td>衣服</td>\n",
206 |        "      <td>0</td>\n",
207 |        "      <td>裤子又大又长，那里像休闲裤，妈的，还修身呢，真是够了</td>\n",
208 |        "    </tr>\n",
209 |        "    <tr>\n",
210 |        "      <th>62616</th>\n",
211 |        "      <td>酒店</td>\n",
212 |        "      <td>0</td>\n",
213 |        "      <td>奇葩的酒店。在一个办公楼里，自己开车去酒店，很难找到，等到了酒店地下停车场，不知道应该坐那部...</td>\n",
214 |        "    </tr>\n",
215 |        "    <tr>\n",
216 |        "      <th>44044</th>\n",
217 |        "      <td>衣服</td>\n",
218 |        "      <td>0</td>\n",
219 |        "      <td>我要晕死得节奏，买回来就没穿过，真的是霉！</td>\n",
220 |        "    </tr>\n",
221 |        "    <tr>\n",
222 |        "      <th>19456</th>\n",
223 |        "      <td>水果</td>\n",
224 |        "      <td>1</td>\n",
225 |        "      <td>苹果不大，但很脆甜。检查了一下，48个没有烂的，有个别难看的。总体上质量不错</td>\n",
226 |        "    </tr>\n",
227 |        "    <tr>\n",
228 |        "      <th>10562</th>\n",
229 |        "      <td>平板</td>\n",
230 |        "      <td>0</td>\n",
231 |        "      <td>差差差真卡渣渣品牌以后在也不相信大品牌了坑是了</td>\n",
232 |        "    </tr>\n",
233 |        "    <tr>\n",
234 |        "      <th>34199</th>\n",
235 |        "      <td>洗发水</td>\n",
236 |        "      <td>0</td>\n",
237 |        "      <td>这个是6月18当天买的，只有半瓶。购物太差劲了</td>\n",
238 |        "    </tr>\n",
239 |        "  </tbody>\n",
240 |        "</table>\n",
241 |        "</div>"
242 |       ],
243 |       "text/plain": [
244 |        "       cat  label                                             review\n",
245 |        "11194   平板      0                        什么玩意。刚用一天，就充不上电，开不开机，返厂老麻烦，\n",
246 |        "17794   水果      1                               买了几次了，价格实惠，口感不错，保鲜好！\n",
247 |        "29529  洗发水      1               挺值得购买的，有包装买回去送家人，毛巾质量不错。小块的可以拿来当擦手帕。\n",
248 |        "24976   水果      0              真的就算后悔了。两天才拿到货。还不如水果店买！还都发霉不新鲜了！以后不买了\n",
249 |        "28447  洗发水      1                                   一般般，薄荷洗发水没想象中的凉快\n",
250 |        "264     书籍      1  这本书有别于以往看过的早教书籍，结合了说明文的写实，散文的情致和图册的一目了然。特别是读过几...\n",
251 |        "53035   酒店      1  酒店的大堂很漂亮,房间不算小,设施还可以也很干净,离码头很近,而且又有车接送,很方便.晚上2...\n",
252 |        "50250  计算机      1      做工不错，外壳也很漂亮。测试了一下还行！~中通很快啊，13号下午的订单，今天早上就收到了。\n",
253 |        "62461   酒店      0  房间空间比较小， 环境比较吵。特别半夜被窗户外面的空调外机的声音吵醒（因为窗外一条巷子之隔，...\n",
254 |        "52888   酒店      1  清明节入住两天.从进入酒店就感受到无处不在的服务,非常周到,又很得体.从大堂,商务中心,到前...\n",
255 |        "31429  洗发水      0                       感觉不怎么样，刚刚洗完头发又感觉头发干枯枯的而且还是好油\n",
256 |        "21443   水果      0  算了，不要买了，先不说个头小，就味道难吃的要死，还没有路边摊卖的好吃，硬，涩，根本就没有苹果...\n",
257 |        "19374   水果      1                              快递神速，品种与描述一样，比上次买的好吃！\n",
258 |        "28188  洗发水      1                     还没有用，不过感觉和实体店买的差不多，等用过之后再追加评价吧\n",
259 |        "46182   衣服      0                         裤子又大又长，那里像休闲裤，妈的，还修身呢，真是够了\n",
260 |        "62616   酒店      0  奇葩的酒店。在一个办公楼里，自己开车去酒店，很难找到，等到了酒店地下停车场，不知道应该坐那部...\n",
261 |        "44044   衣服      0                              我要晕死得节奏，买回来就没穿过，真的是霉！\n",
262 |        "19456   水果      1             苹果不大，但很脆甜。检查了一下，48个没有烂的，有个别难看的。总体上质量不错\n",
263 |        "10562   平板      0                            差差差真卡渣渣品牌以后在也不相信大品牌了坑是了\n",
264 |        "34199  洗发水      0                            这个是6月18当天买的，只有半瓶。购物太差劲了"
265 |       ]
266 |      },
267 |      "execution_count": 27,
268 |      "metadata": {},
269 |      "output_type": "execute_result"
270 |     }
271 |    ],
272 |    "source": [
273 |     "pd_all.sample(20)"
274 |    ]
275 |   },
276 |   {
277 |    "cell_type": "markdown",
278 |    "metadata": {},
279 |    "source": [
280 |     "# 2. 统计各类别语料的规模"
281 |    ]
282 |   },
283 |   {
284 |    "cell_type": "code",
285 |    "execution_count": 28,
286 |    "metadata": {},
287 |    "outputs": [
288 |     {
289 |      "name": "stdout",
290 |      "output_type": "stream",
291 |      "text": [
292 |       "书籍: 3851 (总体), 2100 (正例), 1751 (负例)\n",
293 |       "平板: 10000 (总体), 5000 (正例), 5000 (负例)\n",
294 |       "手机: 2323 (总体), 1165 (正例), 1158 (负例)\n",
295 |       "水果: 10000 (总体), 5000 (正例), 5000 (负例)\n",
296 |       "洗发水: 10000 (总体), 5000 (正例), 5000 (负例)\n",
297 |       "热水器: 575 (总体), 475 (正例), 100 (负例)\n",
298 |       "蒙牛: 2033 (总体), 992 (正例), 1041 (负例)\n",
299 |       "衣服: 10000 (总体), 5000 (正例), 5000 (负例)\n",
300 |       "计算机: 3992 (总体), 1996 (正例), 1996 (负例)\n",
301 |       "酒店: 10000 (总体), 5000 (正例), 5000 (负例)\n"
302 |      ]
303 |     }
304 |    ],
305 |    "source": [
306 |     "all_cats = ['书籍', '平板', '手机', '水果', '洗发水', '热水器', '蒙牛', '衣服', '计算机', '酒店'] # 全部类别\n",
307 |     "\n",
308 |     "for cat in all_cats:\n",
309 |     "    pd_data = pd_all[pd_all.cat==cat]\n",
310 |     "    print('{}: {} (总体), {} (正例), {} (负例)'.format(cat, pd_data.shape[0], \n",
311 |     "                                                 pd_data[pd_data.label==1].shape[0], pd_data[pd_data.label==0].shape[0]))"
312 |    ]
313 |   },
314 |   {
315 |    "cell_type": "markdown",
316 |    "metadata": {},
317 |    "source": [
318 |     "# 3. 加载指定类别的语料"
319 |    ]
320 |   },
321 |   {
322 |    "cell_type": "code",
323 |    "execution_count": 29,
324 |    "metadata": {},
325 |    "outputs": [
326 |     {
327 |      "name": "stdout",
328 |      "output_type": "stream",
329 |      "text": [
330 |       "评论数目（总体）：17843\n",
331 |       "评论数目（正向）：9096\n",
332 |       "评论数目（负向）：8747\n"
333 |      ]
334 |     },
335 |     {
336 |      "data": {
337 |       "text/html": [
338 |        "<div>\n",
339 |        "<style scoped>\n",
340 |        "    .dataframe tbody tr th:only-of-type {\n",
341 |        "        vertical-align: middle;\n",
342 |        "    }\n",
343 |        "\n",
344 |        "    .dataframe tbody tr th {\n",
345 |        "        vertical-align: top;\n",
346 |        "    }\n",
347 |        "\n",
348 |        "    .dataframe thead th {\n",
349 |        "        text-align: right;\n",
350 |        "    }\n",
351 |        "</style>\n",
352 |        "<table border=\"1\" class=\"dataframe\">\n",
353 |        "  <thead>\n",
354 |        "    <tr style=\"text-align: right;\">\n",
355 |        "      <th></th>\n",
356 |        "      <th>cat</th>\n",
357 |        "      <th>label</th>\n",
358 |        "      <th>review</th>\n",
359 |        "    </tr>\n",
360 |        "  </thead>\n",
361 |        "  <tbody>\n",
362 |        "    <tr>\n",
363 |        "      <th>1620</th>\n",
364 |        "      <td>书籍</td>\n",
365 |        "      <td>1</td>\n",
366 |        "      <td>符弦歌&amp;凌悠扬，一个背负着道义和家族荣誉，一个洒脱且桀骜不羁，两个完全不相同的人却因为千丝万...</td>\n",
367 |        "    </tr>\n",
368 |        "    <tr>\n",
369 |        "      <th>18872</th>\n",
370 |        "      <td>水果</td>\n",
371 |        "      <td>1</td>\n",
372 |        "      <td>一直在吃，烟台苹果，味道不错，物流快</td>\n",
373 |        "    </tr>\n",
374 |        "    <tr>\n",
375 |        "      <th>443</th>\n",
376 |        "      <td>书籍</td>\n",
377 |        "      <td>1</td>\n",
378 |        "      <td>仔细回想这本文集，发现自己喜欢的只是写《教室朝南，没有风筝》的麻宁，不知道是她成长了还是自己...</td>\n",
379 |        "    </tr>\n",
380 |        "    <tr>\n",
381 |        "      <th>21437</th>\n",
382 |        "      <td>水果</td>\n",
383 |        "      <td>0</td>\n",
384 |        "      <td>最差的一次购物体验，干瘪，坏心，糟糕透顶</td>\n",
385 |        "    </tr>\n",
386 |        "    <tr>\n",
387 |        "      <th>18321</th>\n",
388 |        "      <td>水果</td>\n",
389 |        "      <td>1</td>\n",
390 |        "      <td>多次购买新鲜爽甜，80个头大大个，物流超快，上午9点前下单，下午16点收货</td>\n",
391 |        "    </tr>\n",
392 |        "    <tr>\n",
393 |        "      <th>568</th>\n",
394 |        "      <td>书籍</td>\n",
395 |        "      <td>1</td>\n",
396 |        "      <td>一开始我是看了当当上的推荐，说不一样的卡梅拉这套书是亚马逊的五星级图书，大家的评论也非常好。...</td>\n",
397 |        "    </tr>\n",
398 |        "    <tr>\n",
399 |        "      <th>23927</th>\n",
400 |        "      <td>水果</td>\n",
401 |        "      <td>0</td>\n",
402 |        "      <td>垃圾啊，以后再也不 会买了啊 ，好几个坏的，还有好多歪头歪闹的</td>\n",
403 |        "    </tr>\n",
404 |        "    <tr>\n",
405 |        "      <th>19244</th>\n",
406 |        "      <td>水果</td>\n",
407 |        "      <td>1</td>\n",
408 |        "      <td>包装完好，没有烂果，就是比较小粒，卖相不好。</td>\n",
409 |        "    </tr>\n",
410 |        "    <tr>\n",
411 |        "      <th>20643</th>\n",
412 |        "      <td>水果</td>\n",
413 |        "      <td>1</td>\n",
414 |        "      <td>不错不错特别好吃，甜甜的水分还足而且还很脆，第一次在京东买苹果，果然没让我失望，</td>\n",
415 |        "    </tr>\n",
416 |        "    <tr>\n",
417 |        "      <th>22330</th>\n",
418 |        "      <td>水果</td>\n",
419 |        "      <td>0</td>\n",
420 |        "      <td>第一次给差评，刚拿上打开第一个就黑心。差评。</td>\n",
421 |        "    </tr>\n",
422 |        "    <tr>\n",
423 |        "      <th>17905</th>\n",
424 |        "      <td>水果</td>\n",
425 |        "      <td>1</td>\n",
426 |        "      <td>妈妈说非常好，谢谢店家，会继续支持</td>\n",
427 |        "    </tr>\n",
428 |        "    <tr>\n",
429 |        "      <th>19439</th>\n",
430 |        "      <td>水果</td>\n",
431 |        "      <td>1</td>\n",
432 |        "      <td>不错不错挺甜的。 收到还凉凉的。</td>\n",
433 |        "    </tr>\n",
434 |        "    <tr>\n",
435 |        "      <th>23419</th>\n",
436 |        "      <td>水果</td>\n",
437 |        "      <td>0</td>\n",
438 |        "      <td>吃第一个就是烂的，而且是烂透了的。认栽，图都难得传了！</td>\n",
439 |        "    </tr>\n",
440 |        "    <tr>\n",
441 |        "      <th>355</th>\n",
442 |        "      <td>书籍</td>\n",
443 |        "      <td>1</td>\n",
444 |        "      <td>这本书从男性的视觉诠释了承诺和责任的关系。从达菲一开始的茫然到最后勇敢面对自己的真心，以及对...</td>\n",
445 |        "    </tr>\n",
446 |        "    <tr>\n",
447 |        "      <th>24028</th>\n",
448 |        "      <td>水果</td>\n",
449 |        "      <td>0</td>\n",
450 |        "      <td>味同嚼蜡，水泥地里长出来的吗？一点味道都没有还硬的很，颜色很红，个头很小，口感特别差，真后悔</td>\n",
451 |        "    </tr>\n",
452 |        "    <tr>\n",
453 |        "      <th>497</th>\n",
454 |        "      <td>书籍</td>\n",
455 |        "      <td>1</td>\n",
456 |        "      <td>因为众所周知的原因，我一直在内心深处比较抵制日本文化，我们接受的教育也是负面的信息多于正面的...</td>\n",
457 |        "    </tr>\n",
458 |        "    <tr>\n",
459 |        "      <th>52307</th>\n",
460 |        "      <td>计算机</td>\n",
461 |        "      <td>0</td>\n",
462 |        "      <td>噪音稍大，再就是装XP系统确实蓝屏的几率比较大，装VISTA算了，别的缺点暂时真没发觉，水平有限</td>\n",
463 |        "    </tr>\n",
464 |        "    <tr>\n",
465 |        "      <th>51268</th>\n",
466 |        "      <td>计算机</td>\n",
467 |        "      <td>0</td>\n",
468 |        "      <td>可能是主板比较特殊，很多Ghost启动光盘不能识别光驱，不过好像萝卜花园的可以识别。</td>\n",
469 |        "    </tr>\n",
470 |        "    <tr>\n",
471 |        "      <th>21227</th>\n",
472 |        "      <td>水果</td>\n",
473 |        "      <td>0</td>\n",
474 |        "      <td>好小一个，根本不是进口的。包装好看而已！</td>\n",
475 |        "    </tr>\n",
476 |        "    <tr>\n",
477 |        "      <th>18596</th>\n",
478 |        "      <td>水果</td>\n",
479 |        "      <td>1</td>\n",
480 |        "      <td>好吃真心的好吃赞了，快递特快，继续关注，会回购的</td>\n",
481 |        "    </tr>\n",
482 |        "  </tbody>\n",
483 |        "</table>\n",
484 |        "</div>"
485 |       ],
486 |       "text/plain": [
487 |        "       cat  label                                             review\n",
488 |        "1620    书籍      1  符弦歌&凌悠扬，一个背负着道义和家族荣誉，一个洒脱且桀骜不羁，两个完全不相同的人却因为千丝万...\n",
489 |        "18872   水果      1                                 一直在吃，烟台苹果，味道不错，物流快\n",
490 |        "443     书籍      1  仔细回想这本文集，发现自己喜欢的只是写《教室朝南，没有风筝》的麻宁，不知道是她成长了还是自己...\n",
491 |        "21437   水果      0                               最差的一次购物体验，干瘪，坏心，糟糕透顶\n",
492 |        "18321   水果      1              多次购买新鲜爽甜，80个头大大个，物流超快，上午9点前下单，下午16点收货\n",
493 |        "568     书籍      1  一开始我是看了当当上的推荐，说不一样的卡梅拉这套书是亚马逊的五星级图书，大家的评论也非常好。...\n",
494 |        "23927   水果      0                    垃圾啊，以后再也不 会买了啊 ，好几个坏的，还有好多歪头歪闹的\n",
495 |        "19244   水果      1                             包装完好，没有烂果，就是比较小粒，卖相不好。\n",
496 |        "20643   水果      1           不错不错特别好吃，甜甜的水分还足而且还很脆，第一次在京东买苹果，果然没让我失望，\n",
497 |        "22330   水果      0                             第一次给差评，刚拿上打开第一个就黑心。差评。\n",
498 |        "17905   水果      1                                  妈妈说非常好，谢谢店家，会继续支持\n",
499 |        "19439   水果      1                                   不错不错挺甜的。 收到还凉凉的。\n",
500 |        "23419   水果      0                        吃第一个就是烂的，而且是烂透了的。认栽，图都难得传了！\n",
501 |        "355     书籍      1  这本书从男性的视觉诠释了承诺和责任的关系。从达菲一开始的茫然到最后勇敢面对自己的真心，以及对...\n",
502 |        "24028   水果      0     味同嚼蜡，水泥地里长出来的吗？一点味道都没有还硬的很，颜色很红，个头很小，口感特别差，真后悔\n",
503 |        "497     书籍      1  因为众所周知的原因，我一直在内心深处比较抵制日本文化，我们接受的教育也是负面的信息多于正面的...\n",
504 |        "52307  计算机      0   噪音稍大，再就是装XP系统确实蓝屏的几率比较大，装VISTA算了，别的缺点暂时真没发觉，水平有限\n",
505 |        "51268  计算机      0         可能是主板比较特殊，很多Ghost启动光盘不能识别光驱，不过好像萝卜花园的可以识别。\n",
506 |        "21227   水果      0                               好小一个，根本不是进口的。包装好看而已！\n",
507 |        "18596   水果      1                           好吃真心的好吃赞了，快递特快，继续关注，会回购的"
508 |       ]
509 |      },
510 |      "execution_count": 29,
511 |      "metadata": {},
512 |      "output_type": "execute_result"
513 |     }
514 |    ],
515 |    "source": [
516 |     "target_cats = ['书籍', '水果', '计算机'] # 假定只需要 书籍、水果、计算机 3 个 类别的数据\n",
517 |     "\n",
518 |     "pd_data = pd_all[pd_all.cat.isin(target_cats)]\n",
519 |     "\n",
520 |     "print('评论数目（总体）：%d' % pd_data.shape[0])\n",
521 |     "print('评论数目（正向）：%d' % pd_data[pd_data.label==1].shape[0])\n",
522 |     "print('评论数目（负向）：%d' % pd_data[pd_data.label==0].shape[0])\n",
523 |     "\n",
524 |     "pd_data.sample(20)"
525 |    ]
526 |   }
527 |  ],
528 |  "metadata": {
529 |   "kernelspec": {
530 |    "display_name": "Python 3",
531 |    "language": "python",
532 |    "name": "python3"
533 |   },
534 |   "language_info": {
535 |    "codemirror_mode": {
536 |     "name": "ipython",
537 |     "version": 3
538 |    },
539 |    "file_extension": ".py",
540 |    "mimetype": "text/x-python",
541 |    "name": "python",
542 |    "nbconvert_exporter": "python",
543 |    "pygments_lexer": "ipython3",
544 |    "version": "3.6.4"
545 |   },
546 |   "widgets": {
547 |    "state": {},
548 |    "version": "1.1.2"
549 |   }
550 |  },
551 |  "nbformat": 4,
552 |  "nbformat_minor": 2
553 | }
554 | 


--------------------------------------------------------------------------------
/datasets/online_shopping_10_cats/online_shopping_10_cats.zip:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/SophonPlus/ChineseNlpCorpus/a0b781d72158bde242c0626c9e354d2b328865ff/datasets/online_shopping_10_cats/online_shopping_10_cats.zip


--------------------------------------------------------------------------------
/datasets/simplifyweibo_4_moods/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# simplifyweibo_4_moods 说明\n",
  8 |     "0. **下载地址：** [百度网盘](https://pan.baidu.com/s/16c93E5x373nsGozyWevITg)\n",
  9 |     "1. **数据概览：** 36 万多条，带情感标注 新浪微博，包含 4 种情感，其中喜悦约 20 万条，愤怒、厌恶、低落各约 5 万条\n",
 10 |     "2. **推荐实验：** 情感/观点/评论 倾向性分析\n",
 11 |     "2. **数据来源：** [新浪微博](https://weibo.com/)\n",
 12 |     "3. **原数据集：** [微博情感分析数据集](https://download.csdn.net/download/turkan/9181661)，网上搜集，具体作者、来源不详\n",
 13 |     "4. **加工处理：**\n",
 14 |     "    1. 将原来的 4 份文档，整合成 1 份 csv 文件\n",
 15 |     "    2. 原始语料进行了分词处理，我们重新将其还原为未分词的语料\n",
 16 |     "    3. 编码统一为 UTF-8\n",
 17 |     "    4. 去重"
 18 |    ]
 19 |   },
 20 |   {
 21 |    "cell_type": "code",
 22 |    "execution_count": 10,
 23 |    "metadata": {},
 24 |    "outputs": [],
 25 |    "source": [
 26 |     "import pandas as pd"
 27 |    ]
 28 |   },
 29 |   {
 30 |    "cell_type": "code",
 31 |    "execution_count": 1,
 32 |    "metadata": {},
 33 |    "outputs": [],
 34 |    "source": [
 35 |     "path = 'simplifyweibo_4_moods_文件夹_所在_路径'"
 36 |    ]
 37 |   },
 38 |   {
 39 |    "cell_type": "markdown",
 40 |    "metadata": {},
 41 |    "source": [
 42 |     "# 1. simplifyweibo_4_moods.csv"
 43 |    ]
 44 |   },
 45 |   {
 46 |    "cell_type": "markdown",
 47 |    "metadata": {},
 48 |    "source": [
 49 |     "## 加载数据"
 50 |    ]
 51 |   },
 52 |   {
 53 |    "cell_type": "code",
 54 |    "execution_count": 12,
 55 |    "metadata": {},
 56 |    "outputs": [
 57 |     {
 58 |      "name": "stdout",
 59 |      "output_type": "stream",
 60 |      "text": [
 61 |       "微博数目（总体）：361744\n",
 62 |       "微博数目（喜悦）：199496\n",
 63 |       "微博数目（愤怒）：51714\n",
 64 |       "微博数目（厌恶）：55267\n",
 65 |       "微博数目（低落）：55267\n"
 66 |      ]
 67 |     }
 68 |    ],
 69 |    "source": [
 70 |     "pd_all = pd.read_csv(path + 'simplifyweibo_4_moods.csv')\n",
 71 |     "moods = {0: '喜悦', 1: '愤怒', 2: '厌恶', 3: '低落'}\n",
 72 |     "\n",
 73 |     "print('微博数目（总体）：%d' % pd_all.shape[0])\n",
 74 |     "\n",
 75 |     "for label, mood in moods.items(): \n",
 76 |     "    print('微博数目（{}）：{}'.format(mood,  pd_all[pd_all.label==label].shape[0]))"
 77 |    ]
 78 |   },
 79 |   {
 80 |    "cell_type": "markdown",
 81 |    "metadata": {},
 82 |    "source": [
 83 |     "## 字段说明\n",
 84 |     "\n",
 85 |     "| 字段 | 说明 |\n",
 86 |     "| ---- | ---- |\n",
 87 |     "| label | 0 喜悦，1 愤怒，2 厌恶，3 低落 |\n",
 88 |     "| review | 微博内容 |"
 89 |    ]
 90 |   },
 91 |   {
 92 |    "cell_type": "code",
 93 |    "execution_count": 13,
 94 |    "metadata": {
 95 |     "scrolled": false
 96 |    },
 97 |    "outputs": [
 98 |     {
 99 |      "data": {
100 |       "text/html": [
101 |        "<div>\n",
102 |        "<style scoped>\n",
103 |        "    .dataframe tbody tr th:only-of-type {\n",
104 |        "        vertical-align: middle;\n",
105 |        "    }\n",
106 |        "\n",
107 |        "    .dataframe tbody tr th {\n",
108 |        "        vertical-align: top;\n",
109 |        "    }\n",
110 |        "\n",
111 |        "    .dataframe thead th {\n",
112 |        "        text-align: right;\n",
113 |        "    }\n",
114 |        "</style>\n",
115 |        "<table border=\"1\" class=\"dataframe\">\n",
116 |        "  <thead>\n",
117 |        "    <tr style=\"text-align: right;\">\n",
118 |        "      <th></th>\n",
119 |        "      <th>label</th>\n",
120 |        "      <th>review</th>\n",
121 |        "    </tr>\n",
122 |        "  </thead>\n",
123 |        "  <tbody>\n",
124 |        "    <tr>\n",
125 |        "      <th>307114</th>\n",
126 |        "      <td>3</td>\n",
127 |        "      <td>回复美国看起来很美，对别人比较狠！对付哪国人，就用哪国人做他的腿，简称狗腿落后的祖宗挨过打！...</td>\n",
128 |        "    </tr>\n",
129 |        "    <tr>\n",
130 |        "      <th>258815</th>\n",
131 |        "      <td>2</td>\n",
132 |        "      <td>我表示压力狠大。!哇。犀利妹！偶尔街拍，其实姐只是一个你永远无法超越的传说。</td>\n",
133 |        "    </tr>\n",
134 |        "    <tr>\n",
135 |        "      <th>249801</th>\n",
136 |        "      <td>1</td>\n",
137 |        "      <td>可怜，帮这孩子转下，希望不会因为涉嫌联系业务负什么责任啊…………是想粉丝想疯了什么情况啊？想...</td>\n",
138 |        "    </tr>\n",
139 |        "    <tr>\n",
140 |        "      <th>165587</th>\n",
141 |        "      <td>0</td>\n",
142 |        "      <td>哦也~ ~ ~ ！得瑟哈哈哈耶~ ~ ~ ！新logo 。。。。我们的logo 会不会抢了的...</td>\n",
143 |        "    </tr>\n",
144 |        "    <tr>\n",
145 |        "      <th>351395</th>\n",
146 |        "      <td>3</td>\n",
147 |        "      <td>我发现真的是最齐全的一张。这是去看北方儿子的时候啊。怀念。对了，我怎么穿那件破衬衫。。好难看...</td>\n",
148 |        "    </tr>\n",
149 |        "    <tr>\n",
150 |        "      <th>339894</th>\n",
151 |        "      <td>3</td>\n",
152 |        "      <td>看你那个享受的表情nuna 很感动~</td>\n",
153 |        "    </tr>\n",
154 |        "    <tr>\n",
155 |        "      <th>307523</th>\n",
156 |        "      <td>3</td>\n",
157 |        "      <td>不得不轉 ！大家淚 奔吧哈哈27開 始,短短8秒,我咽哽了</td>\n",
158 |        "    </tr>\n",
159 |        "    <tr>\n",
160 |        "      <th>124636</th>\n",
161 |        "      <td>0</td>\n",
162 |        "      <td>早看到了，再看到还是想笑，好可爱啊</td>\n",
163 |        "    </tr>\n",
164 |        "    <tr>\n",
165 |        "      <th>56901</th>\n",
166 |        "      <td>0</td>\n",
167 |        "      <td>快来围观我的小丸子模板~ ~ 哇咔咔~ 得瑟~ ~ ~</td>\n",
168 |        "    </tr>\n",
169 |        "    <tr>\n",
170 |        "      <th>106905</th>\n",
171 |        "      <td>0</td>\n",
172 |        "      <td>也未免太厉害了吧.......观看完此视频之后，我终于明白了香港歌星GEM—— 邓紫棋走红的...</td>\n",
173 |        "    </tr>\n",
174 |        "    <tr>\n",
175 |        "      <th>291966</th>\n",
176 |        "      <td>2</td>\n",
177 |        "      <td>天啊…是住家发生爆炸了，天热，各位注意安全。一朋友开化工厂的。唉。注意安全。真难以想像，不知...</td>\n",
178 |        "    </tr>\n",
179 |        "    <tr>\n",
180 |        "      <th>321489</th>\n",
181 |        "      <td>3</td>\n",
182 |        "      <td>肯德基你就不会带个头，做件好事可爱的脖子们，帮她圆了梦吧~ ~ 小时候来北京，吃过一种小糕点...</td>\n",
183 |        "    </tr>\n",
184 |        "    <tr>\n",
185 |        "      <th>188566</th>\n",
186 |        "      <td>0</td>\n",
187 |        "      <td>想去桂林，上学时候就学到一课文说桂林山水甲天下，一直想去看看品橙网国庆旅游胜地创意评奖活动开...</td>\n",
188 |        "    </tr>\n",
189 |        "    <tr>\n",
190 |        "      <th>15444</th>\n",
191 |        "      <td>0</td>\n",
192 |        "      <td>晃姐姐口才真不是一般人的高，这大概就是文凭带来的区别吧。拿着真文凭的人总会觉得那是自己的底线...</td>\n",
193 |        "    </tr>\n",
194 |        "    <tr>\n",
195 |        "      <th>56820</th>\n",
196 |        "      <td>0</td>\n",
197 |        "      <td>火火happy birthday 天蝎座的人虽然喜欢隐藏自己，但是他喜欢掌握每天生活当中与他...</td>\n",
198 |        "    </tr>\n",
199 |        "    <tr>\n",
200 |        "      <th>257031</th>\n",
201 |        "      <td>2</td>\n",
202 |        "      <td>好久没看了。。。还是那么的感动~ ~ ~ ~</td>\n",
203 |        "    </tr>\n",
204 |        "    <tr>\n",
205 |        "      <th>144782</th>\n",
206 |        "      <td>0</td>\n",
207 |        "      <td>你看他像几岁？关键是牛尔多大?【分享图片】现场挑战高难度抗衰老奇迹~ 看看他都使用倩碧什么产品~</td>\n",
208 |        "    </tr>\n",
209 |        "    <tr>\n",
210 |        "      <th>130776</th>\n",
211 |        "      <td>0</td>\n",
212 |        "      <td>比江苏台的好玩这个真的很搞笑，再次推荐！哈哈，这个绝对值得一看，搞笑死了。当然其中的讽刺意味...</td>\n",
213 |        "    </tr>\n",
214 |        "    <tr>\n",
215 |        "      <th>59158</th>\n",
216 |        "      <td>0</td>\n",
217 |        "      <td>【YMG 推荐】来，哥让你见识下，什么是真正的招财猫！要发财的童鞋抱走~ ~ 在海味舖 買 ?</td>\n",
218 |        "    </tr>\n",
219 |        "    <tr>\n",
220 |        "      <th>240262</th>\n",
221 |        "      <td>1</td>\n",
222 |        "      <td>该带套的时候要带上。大哥，你就得瑟吧和吃饭。美女很美很火。因为吃香辣小龙虾，我的衬衫歇火了。...</td>\n",
223 |        "    </tr>\n",
224 |        "  </tbody>\n",
225 |        "</table>\n",
226 |        "</div>"
227 |       ],
228 |       "text/plain": [
229 |        "        label                                             review\n",
230 |        "307114      3  回复美国看起来很美，对别人比较狠！对付哪国人，就用哪国人做他的腿，简称狗腿落后的祖宗挨过打！...\n",
231 |        "258815      2             我表示压力狠大。!哇。犀利妹！偶尔街拍，其实姐只是一个你永远无法超越的传说。\n",
232 |        "249801      1  可怜，帮这孩子转下，希望不会因为涉嫌联系业务负什么责任啊…………是想粉丝想疯了什么情况啊？想...\n",
233 |        "165587      0  哦也~ ~ ~ ！得瑟哈哈哈耶~ ~ ~ ！新logo 。。。。我们的logo 会不会抢了的...\n",
234 |        "351395      3  我发现真的是最齐全的一张。这是去看北方儿子的时候啊。怀念。对了，我怎么穿那件破衬衫。。好难看...\n",
235 |        "339894      3                                 看你那个享受的表情nuna 很感动~\n",
236 |        "307523      3                      不得不轉 ！大家淚 奔吧哈哈27開 始,短短8秒,我咽哽了\n",
237 |        "124636      0                                  早看到了，再看到还是想笑，好可爱啊\n",
238 |        "56901       0                        快来围观我的小丸子模板~ ~ 哇咔咔~ 得瑟~ ~ ~\n",
239 |        "106905      0  也未免太厉害了吧.......观看完此视频之后，我终于明白了香港歌星GEM—— 邓紫棋走红的...\n",
240 |        "291966      2  天啊…是住家发生爆炸了，天热，各位注意安全。一朋友开化工厂的。唉。注意安全。真难以想像，不知...\n",
241 |        "321489      3  肯德基你就不会带个头，做件好事可爱的脖子们，帮她圆了梦吧~ ~ 小时候来北京，吃过一种小糕点...\n",
242 |        "188566      0  想去桂林，上学时候就学到一课文说桂林山水甲天下，一直想去看看品橙网国庆旅游胜地创意评奖活动开...\n",
243 |        "15444       0  晃姐姐口才真不是一般人的高，这大概就是文凭带来的区别吧。拿着真文凭的人总会觉得那是自己的底线...\n",
244 |        "56820       0  火火happy birthday 天蝎座的人虽然喜欢隐藏自己，但是他喜欢掌握每天生活当中与他...\n",
245 |        "257031      2                             好久没看了。。。还是那么的感动~ ~ ~ ~\n",
246 |        "144782      0   你看他像几岁？关键是牛尔多大?【分享图片】现场挑战高难度抗衰老奇迹~ 看看他都使用倩碧什么产品~\n",
247 |        "130776      0  比江苏台的好玩这个真的很搞笑，再次推荐！哈哈，这个绝对值得一看，搞笑死了。当然其中的讽刺意味...\n",
248 |        "59158       0    【YMG 推荐】来，哥让你见识下，什么是真正的招财猫！要发财的童鞋抱走~ ~ 在海味舖 買 ?\n",
249 |        "240262      1  该带套的时候要带上。大哥，你就得瑟吧和吃饭。美女很美很火。因为吃香辣小龙虾，我的衬衫歇火了。..."
250 |       ]
251 |      },
252 |      "execution_count": 13,
253 |      "metadata": {},
254 |      "output_type": "execute_result"
255 |     }
256 |    ],
257 |    "source": [
258 |     "pd_all.sample(20)"
259 |    ]
260 |   }
261 |  ],
262 |  "metadata": {
263 |   "kernelspec": {
264 |    "display_name": "Python 3",
265 |    "language": "python",
266 |    "name": "python3"
267 |   },
268 |   "language_info": {
269 |    "codemirror_mode": {
270 |     "name": "ipython",
271 |     "version": 3
272 |    },
273 |    "file_extension": ".py",
274 |    "mimetype": "text/x-python",
275 |    "name": "python",
276 |    "nbconvert_exporter": "python",
277 |    "pygments_lexer": "ipython3",
278 |    "version": "3.6.4"
279 |   },
280 |   "widgets": {
281 |    "state": {},
282 |    "version": "1.1.2"
283 |   }
284 |  },
285 |  "nbformat": 4,
286 |  "nbformat_minor": 2
287 | }
288 | 


--------------------------------------------------------------------------------
/datasets/touzizhidao/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# touzizhidao_filter 说明\n",
  8 |     "0. **下载地址：** [百度网盘](https://pan.baidu.com/s/1SR5d20DPpU7F1h_OVf64GA)\n",
  9 |     "1. **数据概览：** 58.8 万条投资行业问答数据\n",
 10 |     "2. **推荐实验：** FAQ 问答系统\n",
 11 |     "3. **数据来源：** 百度知道\n",
 12 |     "4. **加工处理：**\n",
 13 |     "    1. 过滤了id、url、qid、reply_t、user字段\n",
 14 |     "    2. 对question、reply做了脱敏处理"
 15 |    ]
 16 |   },
 17 |   {
 18 |    "cell_type": "code",
 19 |    "execution_count": 1,
 20 |    "metadata": {},
 21 |    "outputs": [],
 22 |    "source": [
 23 |     "import pandas as pd"
 24 |    ]
 25 |   },
 26 |   {
 27 |    "cell_type": "code",
 28 |    "execution_count": 2,
 29 |    "metadata": {},
 30 |    "outputs": [],
 31 |    "source": [
 32 |     "path = 'touzizhidao_文件夹_所在_路径'"
 33 |    ]
 34 |   },
 35 |   {
 36 |    "cell_type": "markdown",
 37 |    "metadata": {},
 38 |    "source": [
 39 |     "# 1. touzizhidao_filter.csv"
 40 |    ]
 41 |   },
 42 |   {
 43 |    "cell_type": "code",
 44 |    "execution_count": 3,
 45 |    "metadata": {},
 46 |    "outputs": [],
 47 |    "source": [
 48 |     "pd_all = pd.read_csv(path + 'touzizhidao_filter.csv')"
 49 |    ]
 50 |   },
 51 |   {
 52 |    "cell_type": "markdown",
 53 |    "metadata": {},
 54 |    "source": [
 55 |     "## 字段说明\n",
 56 |     "\n",
 57 |     "| 字段 | 说明 |\n",
 58 |     "| ---- | ---- |\n",
 59 |     "| title | 问题的标题 |\n",
 60 |     "| question | 问题内容（可为空） |\n",
 61 |     "| reply| 回复内容 |\n",
 62 |     "| is_best| 是否为页面上显示的最佳回答 |"
 63 |    ]
 64 |   },
 65 |   {
 66 |    "cell_type": "code",
 67 |    "execution_count": 4,
 68 |    "metadata": {},
 69 |    "outputs": [
 70 |     {
 71 |      "data": {
 72 |       "text/html": [
 73 |        "<div>\n",
 74 |        "<style scoped>\n",
 75 |        "    .dataframe tbody tr th:only-of-type {\n",
 76 |        "        vertical-align: middle;\n",
 77 |        "    }\n",
 78 |        "\n",
 79 |        "    .dataframe tbody tr th {\n",
 80 |        "        vertical-align: top;\n",
 81 |        "    }\n",
 82 |        "\n",
 83 |        "    .dataframe thead th {\n",
 84 |        "        text-align: right;\n",
 85 |        "    }\n",
 86 |        "</style>\n",
 87 |        "<table border=\"1\" class=\"dataframe\">\n",
 88 |        "  <thead>\n",
 89 |        "    <tr style=\"text-align: right;\">\n",
 90 |        "      <th></th>\n",
 91 |        "      <th>title</th>\n",
 92 |        "      <th>question</th>\n",
 93 |        "      <th>reply</th>\n",
 94 |        "      <th>is_best</th>\n",
 95 |        "    </tr>\n",
 96 |        "  </thead>\n",
 97 |        "  <tbody>\n",
 98 |        "    <tr>\n",
 99 |        "      <th>133637</th>\n",
100 |        "      <td>华夏银行信用卡怎么查询申请进度</td>\n",
101 |        "      <td>NaN</td>\n",
102 |        "      <td>信用卡申请进度查询：查询步骤：一、网银查询：1、登录银行信用卡中心页面，然后点击“办卡进度查...</td>\n",
103 |        "      <td>1</td>\n",
104 |        "    </tr>\n",
105 |        "    <tr>\n",
106 |        "      <th>295236</th>\n",
107 |        "      <td>我向上海复星投资创业有限公司申请贷款要交1000元保险开户费，交了</td>\n",
108 |        "      <td>我向上海复星投资创业有限公司申请贷款要交1000元保险开户费，交了过后又说我银行卡不行还要交...</td>\n",
109 |        "      <td>我的不用</td>\n",
110 |        "      <td>0</td>\n",
111 |        "    </tr>\n",
112 |        "    <tr>\n",
113 |        "      <th>329332</th>\n",
114 |        "      <td>二手房买卖中介收费是多少二手房买卖中介如何收费</td>\n",
115 |        "      <td>NaN</td>\n",
116 |        "      <td>二手房交易流程(1)买方咨询买卖双方建立信息沟通渠道，买方了解房屋整体现状及产权状况，要求卖...</td>\n",
117 |        "      <td>1</td>\n",
118 |        "    </tr>\n",
119 |        "    <tr>\n",
120 |        "      <th>176871</th>\n",
121 |        "      <td>单位给职工办的社保卡买药里面资金不足怎么办</td>\n",
122 |        "      <td>单位给职工办的社保卡买药里面资金不足怎么办</td>\n",
123 |        "      <td>不足的部分需要自己支付医保卡的使用范围主要有以下三个方面：1、用于购药：参保人员在定点药店买...</td>\n",
124 |        "      <td>1</td>\n",
125 |        "    </tr>\n",
126 |        "    <tr>\n",
127 |        "      <th>485667</th>\n",
128 |        "      <td>银联医保卡去哪家银行激活。</td>\n",
129 |        "      <td>NaN</td>\n",
130 |        "      <td>医保卡上面的银行医保卡激活的步骤：1、带着老卡和新卡到建设银行办理；2、新医保卡的密码是身份...</td>\n",
131 |        "      <td>1</td>\n",
132 |        "    </tr>\n",
133 |        "    <tr>\n",
134 |        "      <th>5012</th>\n",
135 |        "      <td>买一套大概150万的二手门面房大概要交多少钱</td>\n",
136 |        "      <td>NaN</td>\n",
137 |        "      <td>如果购买的是非普通住宅，除了缴纳房屋费用，还需要按以下规定缴纳相关税费：（1）增值税：非住宅...</td>\n",
138 |        "      <td>1</td>\n",
139 |        "    </tr>\n",
140 |        "    <tr>\n",
141 |        "      <th>475672</th>\n",
142 |        "      <td>二手房买卖，公共维修基金应怎么处理？是需要下家支付给上家账面余额，还是无偿顺延呢？</td>\n",
143 |        "      <td>买卖合同中是这样写的：“出卖人同意其缴纳的该房屋专项维修资金（公共维修基金）的账面余额转移至...</td>\n",
144 |        "      <td>需要办理维修基金过户。无偿顺延就可以。维修基金使用条件：1、维修基金只有在保修期满后，对物业...</td>\n",
145 |        "      <td>0</td>\n",
146 |        "    </tr>\n",
147 |        "    <tr>\n",
148 |        "      <th>199291</th>\n",
149 |        "      <td>信用卡全额还款好还是最低还款好</td>\n",
150 |        "      <td>NaN</td>\n",
151 |        "      <td>如果条件可以，当然是全额还款好，最低还款是要付利息的，而且还有点高，银行当然希望是最低还款，...</td>\n",
152 |        "      <td>1</td>\n",
153 |        "    </tr>\n",
154 |        "    <tr>\n",
155 |        "      <th>265499</th>\n",
156 |        "      <td>花呗如何才能提额</td>\n",
157 |        "      <td>花呗怎样才能提高额度</td>\n",
158 |        "      <td>花呗额度取决于芝麻信用分，若要提升额度，需要先提升芝麻信用分，提升芝麻信用分小技巧：1、多在...</td>\n",
159 |        "      <td>0</td>\n",
160 |        "    </tr>\n",
161 |        "    <tr>\n",
162 |        "      <th>224237</th>\n",
163 |        "      <td>工行信用卡逾期两个月，没有90天！！</td>\n",
164 |        "      <td>工行信用卡逾期两个月，没有90天！！银行把卡冻结了，欠款7000，全部还清以后打电话解冻，客...</td>\n",
165 |        "      <td>可以用，但额度只有2000元，且征信上有逾期记录注销吧</td>\n",
166 |        "      <td>0</td>\n",
167 |        "    </tr>\n",
168 |        "    <tr>\n",
169 |        "      <th>271023</th>\n",
170 |        "      <td>中*房价什么时候会大跌</td>\n",
171 |        "      <td>NaN</td>\n",
172 |        "      <td>我感觉房价下降的几率比较小，现在啥都涨价，国家再调控，也不可能让我这月收入几千块钱的人买得起...</td>\n",
173 |        "      <td>0</td>\n",
174 |        "    </tr>\n",
175 |        "    <tr>\n",
176 |        "      <th>14097</th>\n",
177 |        "      <td>在四*地*，个人所得税达到多少金额</td>\n",
178 |        "      <td>NaN</td>\n",
179 |        "      <td>个人所得税征税内容工资、薪金所得,个体工商户的生产、经营所得,他有偿服务活动取得的所得。经营...</td>\n",
180 |        "      <td>1</td>\n",
181 |        "    </tr>\n",
182 |        "    <tr>\n",
183 |        "      <th>363978</th>\n",
184 |        "      <td>关于贷款的</td>\n",
185 |        "      <td>关于贷款的有没有什么借款途径</td>\n",
186 |        "      <td>有口子。</td>\n",
187 |        "      <td>0</td>\n",
188 |        "    </tr>\n",
189 |        "    <tr>\n",
190 |        "      <th>517939</th>\n",
191 |        "      <td>农村建房可以贷款吗</td>\n",
192 |        "      <td>NaN</td>\n",
193 |        "      <td>不可以，银行贷款一般是能够上市交易的房子。贷款需要准备四大类资料：1、个人身份证明：身份证、...</td>\n",
194 |        "      <td>1</td>\n",
195 |        "    </tr>\n",
196 |        "    <tr>\n",
197 |        "      <th>505671</th>\n",
198 |        "      <td>2017年甘*个人医保卡能让别人用吗</td>\n",
199 |        "      <td>NaN</td>\n",
200 |        "      <td>个人医保卡是不能让别人使用的。医保卡（社保卡）只限本人就医时使用，不能出借给他人。参保人如把...</td>\n",
201 |        "      <td>1</td>\n",
202 |        "    </tr>\n",
203 |        "    <tr>\n",
204 |        "      <th>117318</th>\n",
205 |        "      <td>别墅好还是高层好</td>\n",
206 |        "      <td>别墅好还是高层好</td>\n",
207 |        "      <td>别墅。还是看你自己的需要还有经济能力了不是房子建的好看就算是别墅的。别墅即别野，讲究的是周围...</td>\n",
208 |        "      <td>0</td>\n",
209 |        "    </tr>\n",
210 |        "    <tr>\n",
211 |        "      <th>376669</th>\n",
212 |        "      <td>新车什么时候算上户了也就说法律上属于自己的财产</td>\n",
213 |        "      <td>NaN</td>\n",
214 |        "      <td>购房合同签订完了车子就属于个人财产了。中*人*共*国*法通则第七十五条规定：个人财产所有权包...</td>\n",
215 |        "      <td>1</td>\n",
216 |        "    </tr>\n",
217 |        "    <tr>\n",
218 |        "      <th>179097</th>\n",
219 |        "      <td>农民59岁买什么养老</td>\n",
220 |        "      <td>农民59岁买什么养老</td>\n",
221 |        "      <td>多存点钱。</td>\n",
222 |        "      <td>0</td>\n",
223 |        "    </tr>\n",
224 |        "    <tr>\n",
225 |        "      <th>77847</th>\n",
226 |        "      <td>支付宝转账手续费怎么收的</td>\n",
227 |        "      <td>NaN</td>\n",
228 |        "      <td>好想是一个月内不能超过5万没有手续费你好，每个支付宝账户有两万元的免费提现和转账额度，提现和...</td>\n",
229 |        "      <td>0</td>\n",
230 |        "    </tr>\n",
231 |        "    <tr>\n",
232 |        "      <th>319220</th>\n",
233 |        "      <td>甘*省企业退休人员养老金怎么调整</td>\n",
234 |        "      <td>NaN</td>\n",
235 |        "      <td>2016年，我国实现了企业和机关事业单位养老金待遇同步调整，按6.5%左右提高企业和机关事业...</td>\n",
236 |        "      <td>1</td>\n",
237 |        "    </tr>\n",
238 |        "  </tbody>\n",
239 |        "</table>\n",
240 |        "</div>"
241 |       ],
242 |       "text/plain": [
243 |        "                                            title  \\\n",
244 |        "133637                            华夏银行信用卡怎么查询申请进度   \n",
245 |        "295236          我向上海复星投资创业有限公司申请贷款要交1000元保险开户费，交了   \n",
246 |        "329332                    二手房买卖中介收费是多少二手房买卖中介如何收费   \n",
247 |        "176871                      单位给职工办的社保卡买药里面资金不足怎么办   \n",
248 |        "485667                              银联医保卡去哪家银行激活。   \n",
249 |        "5012                       买一套大概150万的二手门面房大概要交多少钱   \n",
250 |        "475672  二手房买卖，公共维修基金应怎么处理？是需要下家支付给上家账面余额，还是无偿顺延呢？   \n",
251 |        "199291                            信用卡全额还款好还是最低还款好   \n",
252 |        "265499                                   花呗如何才能提额   \n",
253 |        "224237                         工行信用卡逾期两个月，没有90天！！   \n",
254 |        "271023                                中*房价什么时候会大跌   \n",
255 |        "14097                           在四*地*，个人所得税达到多少金额   \n",
256 |        "363978                                      关于贷款的   \n",
257 |        "517939                                  农村建房可以贷款吗   \n",
258 |        "505671                         2017年甘*个人医保卡能让别人用吗   \n",
259 |        "117318                                   别墅好还是高层好   \n",
260 |        "376669                    新车什么时候算上户了也就说法律上属于自己的财产   \n",
261 |        "179097                                 农民59岁买什么养老   \n",
262 |        "77847                                支付宝转账手续费怎么收的   \n",
263 |        "319220                           甘*省企业退休人员养老金怎么调整   \n",
264 |        "\n",
265 |        "                                                 question  \\\n",
266 |        "133637                                                NaN   \n",
267 |        "295236  我向上海复星投资创业有限公司申请贷款要交1000元保险开户费，交了过后又说我银行卡不行还要交...   \n",
268 |        "329332                                                NaN   \n",
269 |        "176871                              单位给职工办的社保卡买药里面资金不足怎么办   \n",
270 |        "485667                                                NaN   \n",
271 |        "5012                                                  NaN   \n",
272 |        "475672  买卖合同中是这样写的：“出卖人同意其缴纳的该房屋专项维修资金（公共维修基金）的账面余额转移至...   \n",
273 |        "199291                                                NaN   \n",
274 |        "265499                                         花呗怎样才能提高额度   \n",
275 |        "224237  工行信用卡逾期两个月，没有90天！！银行把卡冻结了，欠款7000，全部还清以后打电话解冻，客...   \n",
276 |        "271023                                                NaN   \n",
277 |        "14097                                                 NaN   \n",
278 |        "363978                                     关于贷款的有没有什么借款途径   \n",
279 |        "517939                                                NaN   \n",
280 |        "505671                                                NaN   \n",
281 |        "117318                                           别墅好还是高层好   \n",
282 |        "376669                                                NaN   \n",
283 |        "179097                                         农民59岁买什么养老   \n",
284 |        "77847                                                 NaN   \n",
285 |        "319220                                                NaN   \n",
286 |        "\n",
287 |        "                                                    reply  is_best  \n",
288 |        "133637  信用卡申请进度查询：查询步骤：一、网银查询：1、登录银行信用卡中心页面，然后点击“办卡进度查...        1  \n",
289 |        "295236                                               我的不用        0  \n",
290 |        "329332  二手房交易流程(1)买方咨询买卖双方建立信息沟通渠道，买方了解房屋整体现状及产权状况，要求卖...        1  \n",
291 |        "176871  不足的部分需要自己支付医保卡的使用范围主要有以下三个方面：1、用于购药：参保人员在定点药店买...        1  \n",
292 |        "485667  医保卡上面的银行医保卡激活的步骤：1、带着老卡和新卡到建设银行办理；2、新医保卡的密码是身份...        1  \n",
293 |        "5012    如果购买的是非普通住宅，除了缴纳房屋费用，还需要按以下规定缴纳相关税费：（1）增值税：非住宅...        1  \n",
294 |        "475672  需要办理维修基金过户。无偿顺延就可以。维修基金使用条件：1、维修基金只有在保修期满后，对物业...        0  \n",
295 |        "199291  如果条件可以，当然是全额还款好，最低还款是要付利息的，而且还有点高，银行当然希望是最低还款，...        1  \n",
296 |        "265499  花呗额度取决于芝麻信用分，若要提升额度，需要先提升芝麻信用分，提升芝麻信用分小技巧：1、多在...        0  \n",
297 |        "224237                        可以用，但额度只有2000元，且征信上有逾期记录注销吧        0  \n",
298 |        "271023  我感觉房价下降的几率比较小，现在啥都涨价，国家再调控，也不可能让我这月收入几千块钱的人买得起...        0  \n",
299 |        "14097   个人所得税征税内容工资、薪金所得,个体工商户的生产、经营所得,他有偿服务活动取得的所得。经营...        1  \n",
300 |        "363978                                               有口子。        0  \n",
301 |        "517939  不可以，银行贷款一般是能够上市交易的房子。贷款需要准备四大类资料：1、个人身份证明：身份证、...        1  \n",
302 |        "505671  个人医保卡是不能让别人使用的。医保卡（社保卡）只限本人就医时使用，不能出借给他人。参保人如把...        1  \n",
303 |        "117318  别墅。还是看你自己的需要还有经济能力了不是房子建的好看就算是别墅的。别墅即别野，讲究的是周围...        0  \n",
304 |        "376669  购房合同签订完了车子就属于个人财产了。中*人*共*国*法通则第七十五条规定：个人财产所有权包...        1  \n",
305 |        "179097                                              多存点钱。        0  \n",
306 |        "77847   好想是一个月内不能超过5万没有手续费你好，每个支付宝账户有两万元的免费提现和转账额度，提现和...        0  \n",
307 |        "319220  2016年，我国实现了企业和机关事业单位养老金待遇同步调整，按6.5%左右提高企业和机关事业...        1  "
308 |       ]
309 |      },
310 |      "execution_count": 4,
311 |      "metadata": {},
312 |      "output_type": "execute_result"
313 |     }
314 |    ],
315 |    "source": [
316 |     "pd_all.sample(n=20)"
317 |    ]
318 |   }
319 |  ],
320 |  "metadata": {
321 |   "kernelspec": {
322 |    "display_name": "Python 3",
323 |    "language": "python",
324 |    "name": "python3"
325 |   },
326 |   "language_info": {
327 |    "codemirror_mode": {
328 |     "name": "ipython",
329 |     "version": 3
330 |    },
331 |    "file_extension": ".py",
332 |    "mimetype": "text/x-python",
333 |    "name": "python",
334 |    "nbconvert_exporter": "python",
335 |    "pygments_lexer": "ipython3",
336 |    "version": "3.7.0"
337 |   }
338 |  },
339 |  "nbformat": 4,
340 |  "nbformat_minor": 2
341 | }
342 | 


--------------------------------------------------------------------------------
/datasets/waimai_10k/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# waimai_10k 说明\n",
  8 |     "0. **下载地址：** [Github](https://github.com/SophonPlus/ChineseNlpCorpus/raw/master/datasets/waimai_10k/waimai_10k.csv)\n",
  9 |     "1. **数据概览：** 某外卖平台收集的用户评价，正向 4000 条，负向 约 8000 条\n",
 10 |     "2. **推荐实验：** 情感/观点/评论 倾向性分析\n",
 11 |     "2. **数据来源：** 某外卖平台\n",
 12 |     "3. **原数据集：** [中文短文本情感分析语料 外卖评价](https://download.csdn.net/download/cstkl/10236683)，网上搜集，具体作者、来源不详\n",
 13 |     "4. **加工处理：**\n",
 14 |     "    1. 将原来 2 个文件整合到 1 个文件中\n",
 15 |     "    2. 去重"
 16 |    ]
 17 |   },
 18 |   {
 19 |    "cell_type": "code",
 20 |    "execution_count": 17,
 21 |    "metadata": {},
 22 |    "outputs": [],
 23 |    "source": [
 24 |     "import pandas as pd"
 25 |    ]
 26 |   },
 27 |   {
 28 |    "cell_type": "code",
 29 |    "execution_count": 18,
 30 |    "metadata": {},
 31 |    "outputs": [],
 32 |    "source": [
 33 |     "path = 'waimai_10k_文件夹_所在_路径'"
 34 |    ]
 35 |   },
 36 |   {
 37 |    "cell_type": "markdown",
 38 |    "metadata": {},
 39 |    "source": [
 40 |     "# 1. waimai_10k.csv"
 41 |    ]
 42 |   },
 43 |   {
 44 |    "cell_type": "markdown",
 45 |    "metadata": {},
 46 |    "source": [
 47 |     "## 加载数据"
 48 |    ]
 49 |   },
 50 |   {
 51 |    "cell_type": "code",
 52 |    "execution_count": 19,
 53 |    "metadata": {},
 54 |    "outputs": [
 55 |     {
 56 |      "name": "stdout",
 57 |      "output_type": "stream",
 58 |      "text": [
 59 |       "评论数目（总体）：11987\n",
 60 |       "评论数目（正向）：4000\n",
 61 |       "评论数目（负向）：7987\n"
 62 |      ]
 63 |     }
 64 |    ],
 65 |    "source": [
 66 |     "pd_all = pd.read_csv(path + 'waimai_10k.csv')\n",
 67 |     "\n",
 68 |     "print('评论数目（总体）：%d' % pd_all.shape[0])\n",
 69 |     "print('评论数目（正向）：%d' % pd_all[pd_all.label==1].shape[0])\n",
 70 |     "print('评论数目（负向）：%d' % pd_all[pd_all.label==0].shape[0])"
 71 |    ]
 72 |   },
 73 |   {
 74 |    "cell_type": "markdown",
 75 |    "metadata": {},
 76 |    "source": [
 77 |     "## 字段说明\n",
 78 |     "\n",
 79 |     "| 字段 | 说明 |\n",
 80 |     "| ---- | ---- |\n",
 81 |     "| label | 1 表示正向评论，0 表示负向评论 |\n",
 82 |     "| review | 评论内容 |"
 83 |    ]
 84 |   },
 85 |   {
 86 |    "cell_type": "code",
 87 |    "execution_count": 20,
 88 |    "metadata": {},
 89 |    "outputs": [
 90 |     {
 91 |      "data": {
 92 |       "text/html": [
 93 |        "<div>\n",
 94 |        "<style scoped>\n",
 95 |        "    .dataframe tbody tr th:only-of-type {\n",
 96 |        "        vertical-align: middle;\n",
 97 |        "    }\n",
 98 |        "\n",
 99 |        "    .dataframe tbody tr th {\n",
100 |        "        vertical-align: top;\n",
101 |        "    }\n",
102 |        "\n",
103 |        "    .dataframe thead th {\n",
104 |        "        text-align: right;\n",
105 |        "    }\n",
106 |        "</style>\n",
107 |        "<table border=\"1\" class=\"dataframe\">\n",
108 |        "  <thead>\n",
109 |        "    <tr style=\"text-align: right;\">\n",
110 |        "      <th></th>\n",
111 |        "      <th>label</th>\n",
112 |        "      <th>review</th>\n",
113 |        "    </tr>\n",
114 |        "  </thead>\n",
115 |        "  <tbody>\n",
116 |        "    <tr>\n",
117 |        "      <th>25</th>\n",
118 |        "      <td>1</td>\n",
119 |        "      <td>送餐特别快,态度也好,辛苦啦</td>\n",
120 |        "    </tr>\n",
121 |        "    <tr>\n",
122 |        "      <th>6632</th>\n",
123 |        "      <td>0</td>\n",
124 |        "      <td>点了热带雨林披萨+饮料，和BBQ鸡肉披萨+饮料，送来的是两个奥尔良披萨+两个银耳冰粥，冰凉冰...</td>\n",
125 |        "    </tr>\n",
126 |        "    <tr>\n",
127 |        "      <th>8849</th>\n",
128 |        "      <td>0</td>\n",
129 |        "      <td>难吃!!!油死了，味道烂</td>\n",
130 |        "    </tr>\n",
131 |        "    <tr>\n",
132 |        "      <th>11114</th>\n",
133 |        "      <td>0</td>\n",
134 |        "      <td>今天菜太咸，连着定了3天吃，一天比一天难吃。</td>\n",
135 |        "    </tr>\n",
136 |        "    <tr>\n",
137 |        "      <th>11661</th>\n",
138 |        "      <td>0</td>\n",
139 |        "      <td>送的太慢了，菜都凉了。</td>\n",
140 |        "    </tr>\n",
141 |        "    <tr>\n",
142 |        "      <th>9571</th>\n",
143 |        "      <td>0</td>\n",
144 |        "      <td>没有满减！</td>\n",
145 |        "    </tr>\n",
146 |        "    <tr>\n",
147 |        "      <th>10614</th>\n",
148 |        "      <td>0</td>\n",
149 |        "      <td>差评！定的时间是12点一刻，结果刚11点就送来了！果断退单。送餐前不看时间吗？</td>\n",
150 |        "    </tr>\n",
151 |        "    <tr>\n",
152 |        "      <th>7585</th>\n",
153 |        "      <td>0</td>\n",
154 |        "      <td>羊肉串太咸，还有些不新鲜。鸡心和鸡胗烤的太老</td>\n",
155 |        "    </tr>\n",
156 |        "    <tr>\n",
157 |        "      <th>6919</th>\n",
158 |        "      <td>0</td>\n",
159 |        "      <td>快递员挺好，速度挺快</td>\n",
160 |        "    </tr>\n",
161 |        "    <tr>\n",
162 |        "      <th>3192</th>\n",
163 |        "      <td>1</td>\n",
164 |        "      <td>小炒肉卷饼好辣~</td>\n",
165 |        "    </tr>\n",
166 |        "    <tr>\n",
167 |        "      <th>10224</th>\n",
168 |        "      <td>0</td>\n",
169 |        "      <td>送来的时候都凉了,味道一般,鲜果西米露就两口的量,鲜果就是一块西瓜一个西瓜籽</td>\n",
170 |        "    </tr>\n",
171 |        "    <tr>\n",
172 |        "      <th>7295</th>\n",
173 |        "      <td>0</td>\n",
174 |        "      <td>没放糖，没放奶油，好难喝</td>\n",
175 |        "    </tr>\n",
176 |        "    <tr>\n",
177 |        "      <th>275</th>\n",
178 |        "      <td>1</td>\n",
179 |        "      <td>他家的奶茶超级好喝。。。</td>\n",
180 |        "    </tr>\n",
181 |        "    <tr>\n",
182 |        "      <th>8378</th>\n",
183 |        "      <td>0</td>\n",
184 |        "      <td>黑椒牛柳饭送成大排饭</td>\n",
185 |        "    </tr>\n",
186 |        "    <tr>\n",
187 |        "      <th>5879</th>\n",
188 |        "      <td>0</td>\n",
189 |        "      <td>一个半小时，可以</td>\n",
190 |        "    </tr>\n",
191 |        "    <tr>\n",
192 |        "      <th>7523</th>\n",
193 |        "      <td>0</td>\n",
194 |        "      <td>订单满减后应该是24，送过来要收我原价39？你搞笑呐，还少听加多宝！我管你什么美食送的还是你...</td>\n",
195 |        "    </tr>\n",
196 |        "    <tr>\n",
197 |        "      <th>6590</th>\n",
198 |        "      <td>0</td>\n",
199 |        "      <td>真心也忒慢了，其他都还成</td>\n",
200 |        "    </tr>\n",
201 |        "    <tr>\n",
202 |        "      <th>1703</th>\n",
203 |        "      <td>1</td>\n",
204 |        "      <td>非常划算，很好</td>\n",
205 |        "    </tr>\n",
206 |        "    <tr>\n",
207 |        "      <th>5345</th>\n",
208 |        "      <td>0</td>\n",
209 |        "      <td>首选是得吐槽一下这家的速度,一个半小时起,然后卷饼包装很不错,酱香鸡肉的比较赞,飘香肘子一般...</td>\n",
210 |        "    </tr>\n",
211 |        "    <tr>\n",
212 |        "      <th>1674</th>\n",
213 |        "      <td>1</td>\n",
214 |        "      <td>离我们远点55分钟送到的，可以理解，饼和粥都不错</td>\n",
215 |        "    </tr>\n",
216 |        "  </tbody>\n",
217 |        "</table>\n",
218 |        "</div>"
219 |       ],
220 |       "text/plain": [
221 |        "       label                                             review\n",
222 |        "25         1                                     送餐特别快,态度也好,辛苦啦\n",
223 |        "6632       0  点了热带雨林披萨+饮料，和BBQ鸡肉披萨+饮料，送来的是两个奥尔良披萨+两个银耳冰粥，冰凉冰...\n",
224 |        "8849       0                                       难吃!!!油死了，味道烂\n",
225 |        "11114      0                             今天菜太咸，连着定了3天吃，一天比一天难吃。\n",
226 |        "11661      0                                        送的太慢了，菜都凉了。\n",
227 |        "9571       0                                              没有满减！\n",
228 |        "10614      0            差评！定的时间是12点一刻，结果刚11点就送来了！果断退单。送餐前不看时间吗？\n",
229 |        "7585       0                             羊肉串太咸，还有些不新鲜。鸡心和鸡胗烤的太老\n",
230 |        "6919       0                                         快递员挺好，速度挺快\n",
231 |        "3192       1                                           小炒肉卷饼好辣~\n",
232 |        "10224      0             送来的时候都凉了,味道一般,鲜果西米露就两口的量,鲜果就是一块西瓜一个西瓜籽\n",
233 |        "7295       0                                       没放糖，没放奶油，好难喝\n",
234 |        "275        1                                       他家的奶茶超级好喝。。。\n",
235 |        "8378       0                                         黑椒牛柳饭送成大排饭\n",
236 |        "5879       0                                           一个半小时，可以\n",
237 |        "7523       0  订单满减后应该是24，送过来要收我原价39？你搞笑呐，还少听加多宝！我管你什么美食送的还是你...\n",
238 |        "6590       0                                       真心也忒慢了，其他都还成\n",
239 |        "1703       1                                            非常划算，很好\n",
240 |        "5345       0  首选是得吐槽一下这家的速度,一个半小时起,然后卷饼包装很不错,酱香鸡肉的比较赞,飘香肘子一般...\n",
241 |        "1674       1                           离我们远点55分钟送到的，可以理解，饼和粥都不错"
242 |       ]
243 |      },
244 |      "execution_count": 20,
245 |      "metadata": {},
246 |      "output_type": "execute_result"
247 |     }
248 |    ],
249 |    "source": [
250 |     "pd_all.sample(20)"
251 |    ]
252 |   },
253 |   {
254 |    "cell_type": "markdown",
255 |    "metadata": {},
256 |    "source": [
257 |     "# 2. 构造平衡语料"
258 |    ]
259 |   },
260 |   {
261 |    "cell_type": "code",
262 |    "execution_count": 21,
263 |    "metadata": {},
264 |    "outputs": [],
265 |    "source": [
266 |     "pd_positive = pd_all[pd_all.label==1]\n",
267 |     "pd_negative = pd_all[pd_all.label==0]\n",
268 |     "\n",
269 |     "def get_balance_corpus(corpus_size, corpus_pos, corpus_neg):\n",
270 |     "    sample_size = corpus_size // 2\n",
271 |     "    pd_corpus_balance = pd.concat([corpus_pos.sample(sample_size, replace=corpus_pos.shape[0]<sample_size), \\\n",
272 |     "                                   corpus_neg.sample(sample_size, replace=corpus_neg.shape[0]<sample_size)])\n",
273 |     "    \n",
274 |     "    print('评论数目（总体）：%d' % pd_corpus_balance.shape[0])\n",
275 |     "    print('评论数目（正向）：%d' % pd_corpus_balance[pd_corpus_balance.label==1].shape[0])\n",
276 |     "    print('评论数目（负向）：%d' % pd_corpus_balance[pd_corpus_balance.label==0].shape[0])    \n",
277 |     "    \n",
278 |     "    return pd_corpus_balance"
279 |    ]
280 |   },
281 |   {
282 |    "cell_type": "code",
283 |    "execution_count": 22,
284 |    "metadata": {},
285 |    "outputs": [
286 |     {
287 |      "name": "stdout",
288 |      "output_type": "stream",
289 |      "text": [
290 |       "评论数目（总体）：4000\n",
291 |       "评论数目（正向）：2000\n",
292 |       "评论数目（负向）：2000\n"
293 |      ]
294 |     },
295 |     {
296 |      "data": {
297 |       "text/html": [
298 |        "<div>\n",
299 |        "<style scoped>\n",
300 |        "    .dataframe tbody tr th:only-of-type {\n",
301 |        "        vertical-align: middle;\n",
302 |        "    }\n",
303 |        "\n",
304 |        "    .dataframe tbody tr th {\n",
305 |        "        vertical-align: top;\n",
306 |        "    }\n",
307 |        "\n",
308 |        "    .dataframe thead th {\n",
309 |        "        text-align: right;\n",
310 |        "    }\n",
311 |        "</style>\n",
312 |        "<table border=\"1\" class=\"dataframe\">\n",
313 |        "  <thead>\n",
314 |        "    <tr style=\"text-align: right;\">\n",
315 |        "      <th></th>\n",
316 |        "      <th>label</th>\n",
317 |        "      <th>review</th>\n",
318 |        "    </tr>\n",
319 |        "  </thead>\n",
320 |        "  <tbody>\n",
321 |        "    <tr>\n",
322 |        "      <th>10436</th>\n",
323 |        "      <td>0</td>\n",
324 |        "      <td>难吃～石锅拌饭居然没酱～而且刚好晚了29分钟</td>\n",
325 |        "    </tr>\n",
326 |        "    <tr>\n",
327 |        "      <th>10468</th>\n",
328 |        "      <td>0</td>\n",
329 |        "      <td>等了很久，没关系，毕竟还在约定时间内，可是最让我忍不了的是真的很一般，个人口味吧，反正不和我...</td>\n",
330 |        "    </tr>\n",
331 |        "    <tr>\n",
332 |        "      <th>1643</th>\n",
333 |        "      <td>1</td>\n",
334 |        "      <td>嗯，纸袋比较高大上</td>\n",
335 |        "    </tr>\n",
336 |        "    <tr>\n",
337 |        "      <th>8723</th>\n",
338 |        "      <td>0</td>\n",
339 |        "      <td>海参怎么是生的，没法吃，郁闷</td>\n",
340 |        "    </tr>\n",
341 |        "    <tr>\n",
342 |        "      <th>2431</th>\n",
343 |        "      <td>1</td>\n",
344 |        "      <td>送餐很快，送餐人员很热情！～</td>\n",
345 |        "    </tr>\n",
346 |        "    <tr>\n",
347 |        "      <th>5121</th>\n",
348 |        "      <td>0</td>\n",
349 |        "      <td>不如以前好吃，肘子都有味儿了！哎！</td>\n",
350 |        "    </tr>\n",
351 |        "    <tr>\n",
352 |        "      <th>10565</th>\n",
353 |        "      <td>0</td>\n",
354 |        "      <td>东西有些小贵。</td>\n",
355 |        "    </tr>\n",
356 |        "    <tr>\n",
357 |        "      <th>2413</th>\n",
358 |        "      <td>1</td>\n",
359 |        "      <td>虽然时间长了些但是很准时。下次记得给些番茄酱就更好了。,一个人吃足够了。好好吃</td>\n",
360 |        "    </tr>\n",
361 |        "    <tr>\n",
362 |        "      <th>11937</th>\n",
363 |        "      <td>0</td>\n",
364 |        "      <td>11点以前就定的餐，做了1小时48分钟，呵呵，我只想说：拜拜！！！</td>\n",
365 |        "    </tr>\n",
366 |        "    <tr>\n",
367 |        "      <th>1024</th>\n",
368 |        "      <td>1</td>\n",
369 |        "      <td>很好吃，面皮特别有嚼劲儿，酱料也很好吃</td>\n",
370 |        "    </tr>\n",
371 |        "  </tbody>\n",
372 |        "</table>\n",
373 |        "</div>"
374 |       ],
375 |       "text/plain": [
376 |        "       label                                             review\n",
377 |        "10436      0                             难吃～石锅拌饭居然没酱～而且刚好晚了29分钟\n",
378 |        "10468      0  等了很久，没关系，毕竟还在约定时间内，可是最让我忍不了的是真的很一般，个人口味吧，反正不和我...\n",
379 |        "1643       1                                          嗯，纸袋比较高大上\n",
380 |        "8723       0                                     海参怎么是生的，没法吃，郁闷\n",
381 |        "2431       1                                     送餐很快，送餐人员很热情！～\n",
382 |        "5121       0                                  不如以前好吃，肘子都有味儿了！哎！\n",
383 |        "10565      0                                            东西有些小贵。\n",
384 |        "2413       1            虽然时间长了些但是很准时。下次记得给些番茄酱就更好了。,一个人吃足够了。好好吃\n",
385 |        "11937      0                  11点以前就定的餐，做了1小时48分钟，呵呵，我只想说：拜拜！！！\n",
386 |        "1024       1                                很好吃，面皮特别有嚼劲儿，酱料也很好吃"
387 |       ]
388 |      },
389 |      "execution_count": 22,
390 |      "metadata": {},
391 |      "output_type": "execute_result"
392 |     }
393 |    ],
394 |    "source": [
395 |     "waimai_10k_ba_4000 = get_balance_corpus(4000, pd_positive, pd_negative)\n",
396 |     "\n",
397 |     "waimai_10k_ba_4000.sample(10)"
398 |    ]
399 |   }
400 |  ],
401 |  "metadata": {
402 |   "kernelspec": {
403 |    "display_name": "Python 3",
404 |    "language": "python",
405 |    "name": "python3"
406 |   },
407 |   "language_info": {
408 |    "codemirror_mode": {
409 |     "name": "ipython",
410 |     "version": 3
411 |    },
412 |    "file_extension": ".py",
413 |    "mimetype": "text/x-python",
414 |    "name": "python",
415 |    "nbconvert_exporter": "python",
416 |    "pygments_lexer": "ipython3",
417 |    "version": "3.6.4"
418 |   },
419 |   "widgets": {
420 |    "state": {},
421 |    "version": "1.1.2"
422 |   }
423 |  },
424 |  "nbformat": 4,
425 |  "nbformat_minor": 2
426 | }
427 | 


--------------------------------------------------------------------------------
/datasets/weibo_senti_100k/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# weibo_senti_100k 说明\n",
  8 |     "0. **下载地址：** [百度网盘](https://pan.baidu.com/s/1DoQbki3YwqkuwQUOj64R_g)\n",
  9 |     "1. **数据概览：** 10 万多条，带情感标注 新浪微博，正负向评论约各 5 万条\n",
 10 |     "2. **推荐实验：** 情感/观点/评论 倾向性分析\n",
 11 |     "2. **数据来源：** [新浪微博](https://weibo.com/)\n",
 12 |     "3. **原数据集：** [新浪微博，情感分析标记语料共12万条](https://download.csdn.net/download/weixin_38442818/10214750)，网上搜集，具体作者、来源不详\n",
 13 |     "4. **加工处理：**\n",
 14 |     "    1. 将原来的 2 份文档，整合成 1 份 csv 文件\n",
 15 |     "    2. 编码统一为 UTF-8\n",
 16 |     "    3. 去重"
 17 |    ]
 18 |   },
 19 |   {
 20 |    "cell_type": "code",
 21 |    "execution_count": 13,
 22 |    "metadata": {},
 23 |    "outputs": [],
 24 |    "source": [
 25 |     "import pandas as pd"
 26 |    ]
 27 |   },
 28 |   {
 29 |    "cell_type": "code",
 30 |    "execution_count": 1,
 31 |    "metadata": {},
 32 |    "outputs": [],
 33 |    "source": [
 34 |     "path = 'weibo_senti_100k_文件夹_所在_路径'"
 35 |    ]
 36 |   },
 37 |   {
 38 |    "cell_type": "markdown",
 39 |    "metadata": {},
 40 |    "source": [
 41 |     "# 1. weibo_senti_100k.csv"
 42 |    ]
 43 |   },
 44 |   {
 45 |    "cell_type": "markdown",
 46 |    "metadata": {},
 47 |    "source": [
 48 |     "## 加载数据"
 49 |    ]
 50 |   },
 51 |   {
 52 |    "cell_type": "code",
 53 |    "execution_count": 15,
 54 |    "metadata": {},
 55 |    "outputs": [
 56 |     {
 57 |      "name": "stdout",
 58 |      "output_type": "stream",
 59 |      "text": [
 60 |       "评论数目（总体）：119988\n",
 61 |       "评论数目（正向）：59993\n",
 62 |       "评论数目（负向）：59995\n"
 63 |      ]
 64 |     }
 65 |    ],
 66 |    "source": [
 67 |     "pd_all = pd.read_csv(path + 'weibo_senti_100k.csv')\n",
 68 |     "\n",
 69 |     "print('评论数目（总体）：%d' % pd_all.shape[0])\n",
 70 |     "print('评论数目（正向）：%d' % pd_all[pd_all.label==1].shape[0])\n",
 71 |     "print('评论数目（负向）：%d' % pd_all[pd_all.label==0].shape[0])"
 72 |    ]
 73 |   },
 74 |   {
 75 |    "cell_type": "markdown",
 76 |    "metadata": {},
 77 |    "source": [
 78 |     "## 字段说明\n",
 79 |     "\n",
 80 |     "| 字段 | 说明 |\n",
 81 |     "| ---- | ---- |\n",
 82 |     "| label | 1 表示正向评论，0 表示负向评论 |\n",
 83 |     "| review | 微博内容 |"
 84 |    ]
 85 |   },
 86 |   {
 87 |    "cell_type": "code",
 88 |    "execution_count": 16,
 89 |    "metadata": {},
 90 |    "outputs": [
 91 |     {
 92 |      "data": {
 93 |       "text/html": [
 94 |        "<div>\n",
 95 |        "<style scoped>\n",
 96 |        "    .dataframe tbody tr th:only-of-type {\n",
 97 |        "        vertical-align: middle;\n",
 98 |        "    }\n",
 99 |        "\n",
100 |        "    .dataframe tbody tr th {\n",
101 |        "        vertical-align: top;\n",
102 |        "    }\n",
103 |        "\n",
104 |        "    .dataframe thead th {\n",
105 |        "        text-align: right;\n",
106 |        "    }\n",
107 |        "</style>\n",
108 |        "<table border=\"1\" class=\"dataframe\">\n",
109 |        "  <thead>\n",
110 |        "    <tr style=\"text-align: right;\">\n",
111 |        "      <th></th>\n",
112 |        "      <th>label</th>\n",
113 |        "      <th>review</th>\n",
114 |        "    </tr>\n",
115 |        "  </thead>\n",
116 |        "  <tbody>\n",
117 |        "    <tr>\n",
118 |        "      <th>62050</th>\n",
119 |        "      <td>0</td>\n",
120 |        "      <td>太过分了@Rexzhenghao  //@Janie_Zhang:招行最近负面新闻越来越多呀...</td>\n",
121 |        "    </tr>\n",
122 |        "    <tr>\n",
123 |        "      <th>68263</th>\n",
124 |        "      <td>0</td>\n",
125 |        "      <td>希望你?得好?我本＂?肥血?史＂[晕][哈哈]@Pete三姑父</td>\n",
126 |        "    </tr>\n",
127 |        "    <tr>\n",
128 |        "      <th>81472</th>\n",
129 |        "      <td>0</td>\n",
130 |        "      <td>有点想参加????[偷?]想安排下时间再决定[抓狂]//@黑晶晶crystal: @细腿大羽...</td>\n",
131 |        "    </tr>\n",
132 |        "    <tr>\n",
133 |        "      <th>42021</th>\n",
134 |        "      <td>1</td>\n",
135 |        "      <td>[给力]感谢所有支持雯婕的芝麻！[爱你]</td>\n",
136 |        "    </tr>\n",
137 |        "    <tr>\n",
138 |        "      <th>7777</th>\n",
139 |        "      <td>1</td>\n",
140 |        "      <td>2013最后一天，在新加坡开心度过，向所有的朋友们问声：新年快乐！2014年，我们会更好[调...</td>\n",
141 |        "    </tr>\n",
142 |        "    <tr>\n",
143 |        "      <th>100399</th>\n",
144 |        "      <td>0</td>\n",
145 |        "      <td>大中午出门办事找错路，曝晒中。要多杯具有多杯具。[泪][泪][汗]</td>\n",
146 |        "    </tr>\n",
147 |        "    <tr>\n",
148 |        "      <th>82398</th>\n",
149 |        "      <td>0</td>\n",
150 |        "      <td>马航还会否认吗？到底在隐瞒啥呢？[抓狂]//@头条新闻: 转发微博</td>\n",
151 |        "    </tr>\n",
152 |        "    <tr>\n",
153 |        "      <th>106423</th>\n",
154 |        "      <td>0</td>\n",
155 |        "      <td>克罗地亚球迷很爱放烟火！球又没进，就硝烟四起。[晕]</td>\n",
156 |        "    </tr>\n",
157 |        "    <tr>\n",
158 |        "      <th>24798</th>\n",
159 |        "      <td>1</td>\n",
160 |        "      <td>[抱抱]福芦 TangRoulou 吉祥书 8.8折优惠 &gt;&gt;&gt; http://t.cn/z...</td>\n",
161 |        "    </tr>\n",
162 |        "    <tr>\n",
163 |        "      <th>6598</th>\n",
164 |        "      <td>1</td>\n",
165 |        "      <td>回复@钱旭明QXM:[嘻嘻][嘻嘻] //@钱旭明QXM:杨大哥[good][good][g...</td>\n",
166 |        "    </tr>\n",
167 |        "    <tr>\n",
168 |        "      <th>53920</th>\n",
169 |        "      <td>1</td>\n",
170 |        "      <td>人家这脸长的!!!!!![哈哈]</td>\n",
171 |        "    </tr>\n",
172 |        "    <tr>\n",
173 |        "      <th>15587</th>\n",
174 |        "      <td>1</td>\n",
175 |        "      <td>这个价不算高，和一天内训相比相差无几。。[哈哈]//@博通传媒v: 6个月！一个月工资1万，...</td>\n",
176 |        "    </tr>\n",
177 |        "    <tr>\n",
178 |        "      <th>101237</th>\n",
179 |        "      <td>0</td>\n",
180 |        "      <td>终于收工啦，脚丫子快冻掉了[泪][泪][泪]</td>\n",
181 |        "    </tr>\n",
182 |        "    <tr>\n",
183 |        "      <th>82449</th>\n",
184 |        "      <td>0</td>\n",
185 |        "      <td>我决定从今天开始我想吃什么就去吃什么，一个人吃也无所谓，重点是不要因为别人的意见委屈了自己[...</td>\n",
186 |        "    </tr>\n",
187 |        "    <tr>\n",
188 |        "      <th>32537</th>\n",
189 |        "      <td>1</td>\n",
190 |        "      <td>飘雪的北京 需要双份早餐.......//@美食天下: [哈哈]//@王淼Margay: 屁...</td>\n",
191 |        "    </tr>\n",
192 |        "    <tr>\n",
193 |        "      <th>10630</th>\n",
194 |        "      <td>1</td>\n",
195 |        "      <td>[耶]，这个太赞了，生活大爆炸第六季马上要出啦[鼓掌] //@-郑瑜-:这个不错 //@经典...</td>\n",
196 |        "    </tr>\n",
197 |        "    <tr>\n",
198 |        "      <th>85130</th>\n",
199 |        "      <td>0</td>\n",
200 |        "      <td>刚追完#倾世皇妃#，#千山暮雪#又紧随其后，网速和更新速度都太不给力，尽管我看过原著，还是焦...</td>\n",
201 |        "    </tr>\n",
202 |        "    <tr>\n",
203 |        "      <th>105956</th>\n",
204 |        "      <td>0</td>\n",
205 |        "      <td>晚上看金二胖?察前?，推出的火炮基座?糟了，可以PK了[泪] //@艾米粒er: //@wi...</td>\n",
206 |        "    </tr>\n",
207 |        "    <tr>\n",
208 |        "      <th>72391</th>\n",
209 |        "      <td>0</td>\n",
210 |        "      <td>必须把中国足球的伟大，用我的职业演说出来 //@袁腾飞:[泪]</td>\n",
211 |        "    </tr>\n",
212 |        "    <tr>\n",
213 |        "      <th>10761</th>\n",
214 |        "      <td>1</td>\n",
215 |        "      <td>[鼓掌] //@宁波香格里拉大酒店: 小编来答疑，周五晚惊艳全场的树根蛋糕到底有多长？蛋糕全...</td>\n",
216 |        "    </tr>\n",
217 |        "  </tbody>\n",
218 |        "</table>\n",
219 |        "</div>"
220 |       ],
221 |       "text/plain": [
222 |        "        label                                             review\n",
223 |        "62050       0  太过分了@Rexzhenghao  //@Janie_Zhang:招行最近负面新闻越来越多呀...\n",
224 |        "68263       0                    希望你?得好?我本＂?肥血?史＂[晕][哈哈]@Pete三姑父\n",
225 |        "81472       0  有点想参加????[偷?]想安排下时间再决定[抓狂]//@黑晶晶crystal: @细腿大羽...\n",
226 |        "42021       1                               [给力]感谢所有支持雯婕的芝麻！[爱你]\n",
227 |        "7777        1  2013最后一天，在新加坡开心度过，向所有的朋友们问声：新年快乐！2014年，我们会更好[调...\n",
228 |        "100399      0                  大中午出门办事找错路，曝晒中。要多杯具有多杯具。[泪][泪][汗]\n",
229 |        "82398       0                  马航还会否认吗？到底在隐瞒啥呢？[抓狂]//@头条新闻: 转发微博\n",
230 |        "106423      0                         克罗地亚球迷很爱放烟火！球又没进，就硝烟四起。[晕]\n",
231 |        "24798       1  [抱抱]福芦 TangRoulou 吉祥书 8.8折优惠 >>> http://t.cn/z...\n",
232 |        "6598        1  回复@钱旭明QXM:[嘻嘻][嘻嘻] //@钱旭明QXM:杨大哥[good][good][g...\n",
233 |        "53920       1                                   人家这脸长的!!!!!![哈哈]\n",
234 |        "15587       1  这个价不算高，和一天内训相比相差无几。。[哈哈]//@博通传媒v: 6个月！一个月工资1万，...\n",
235 |        "101237      0                             终于收工啦，脚丫子快冻掉了[泪][泪][泪]\n",
236 |        "82449       0  我决定从今天开始我想吃什么就去吃什么，一个人吃也无所谓，重点是不要因为别人的意见委屈了自己[...\n",
237 |        "32537       1  飘雪的北京 需要双份早餐.......//@美食天下: [哈哈]//@王淼Margay: 屁...\n",
238 |        "10630       1  [耶]，这个太赞了，生活大爆炸第六季马上要出啦[鼓掌] //@-郑瑜-:这个不错 //@经典...\n",
239 |        "85130       0  刚追完#倾世皇妃#，#千山暮雪#又紧随其后，网速和更新速度都太不给力，尽管我看过原著，还是焦...\n",
240 |        "105956      0  晚上看金二胖?察前?，推出的火炮基座?糟了，可以PK了[泪] //@艾米粒er: //@wi...\n",
241 |        "72391       0                    必须把中国足球的伟大，用我的职业演说出来 //@袁腾飞:[泪]\n",
242 |        "10761       1  [鼓掌] //@宁波香格里拉大酒店: 小编来答疑，周五晚惊艳全场的树根蛋糕到底有多长？蛋糕全..."
243 |       ]
244 |      },
245 |      "execution_count": 16,
246 |      "metadata": {},
247 |      "output_type": "execute_result"
248 |     }
249 |    ],
250 |    "source": [
251 |     "pd_all.sample(20)"
252 |    ]
253 |   }
254 |  ],
255 |  "metadata": {
256 |   "kernelspec": {
257 |    "display_name": "Python 3",
258 |    "language": "python",
259 |    "name": "python3"
260 |   },
261 |   "language_info": {
262 |    "codemirror_mode": {
263 |     "name": "ipython",
264 |     "version": 3
265 |    },
266 |    "file_extension": ".py",
267 |    "mimetype": "text/x-python",
268 |    "name": "python",
269 |    "nbconvert_exporter": "python",
270 |    "pygments_lexer": "ipython3",
271 |    "version": "3.6.4"
272 |   },
273 |   "widgets": {
274 |    "state": {},
275 |    "version": "1.1.2"
276 |   }
277 |  },
278 |  "nbformat": 4,
279 |  "nbformat_minor": 2
280 | }
281 | 


--------------------------------------------------------------------------------
/datasets/yf_dianping/intro.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# yf_dianping 说明\n",
  8 |     "0. **下载地址：** [百度网盘](https://pan.baidu.com/s/1yMNvHLl6QYsGbjT7u51Nfg)\n",
  9 |     "1. **数据概览：** 24 万家餐馆，54 万用户，440 万条评论/评分数据\n",
 10 |     "2. **推荐实验：** 推荐系统、情感/观点/评论 倾向性分析\n",
 11 |     "2. **数据来源：** [大众点评](http://www.dianping.com/)\n",
 12 |     "3. **原数据集：** [Dianping Review Dataset](http://yongfeng.me/dataset/)，Yongfeng Zhang 教授为 WWW 2013, SIGIR 2013, SIGIR 2014 会议论文而搜集的数据\n",
 13 |     "4. **加工处理：**\n",
 14 |     "    1. 只保留原数据集中的评论、评分等信息，去除其他无用信息\n",
 15 |     "    2. 整理成与 [MovieLens](https://grouplens.org/datasets/movielens/) 兼容的格式\n",
 16 |     "    3. 进行脱敏操作，以保护用户隐私"
 17 |    ]
 18 |   },
 19 |   {
 20 |    "cell_type": "code",
 21 |    "execution_count": 79,
 22 |    "metadata": {},
 23 |    "outputs": [],
 24 |    "source": [
 25 |     "import pandas as pd"
 26 |    ]
 27 |   },
 28 |   {
 29 |    "cell_type": "code",
 30 |    "execution_count": 80,
 31 |    "metadata": {},
 32 |    "outputs": [],
 33 |    "source": [
 34 |     "path = 'yf_dianping_文件夹_所在_路径'"
 35 |    ]
 36 |   },
 37 |   {
 38 |    "cell_type": "markdown",
 39 |    "metadata": {},
 40 |    "source": [
 41 |     "# 1. restaurants.csv"
 42 |    ]
 43 |   },
 44 |   {
 45 |    "cell_type": "markdown",
 46 |    "metadata": {},
 47 |    "source": [
 48 |     "## 加载数据"
 49 |    ]
 50 |   },
 51 |   {
 52 |    "cell_type": "code",
 53 |    "execution_count": 81,
 54 |    "metadata": {},
 55 |    "outputs": [
 56 |     {
 57 |      "name": "stdout",
 58 |      "output_type": "stream",
 59 |      "text": [
 60 |       "餐馆数目（有名称）：209132\n",
 61 |       "餐馆数目（没有名称）：34115\n",
 62 |       "餐馆数目（总计）：243247\n"
 63 |      ]
 64 |     }
 65 |    ],
 66 |    "source": [
 67 |     "restaurants = pd.read_csv(path + 'restaurants.csv')\n",
 68 |     "\n",
 69 |     "print('餐馆数目（有名称）：%d' % restaurants[~pd.isnull(restaurants.name)].shape[0])\n",
 70 |     "print('餐馆数目（没有名称）：%d' % restaurants[pd.isnull(restaurants.name)].shape[0])\n",
 71 |     "print('餐馆数目（总计）：%d' % restaurants.shape[0])"
 72 |    ]
 73 |   },
 74 |   {
 75 |    "cell_type": "markdown",
 76 |    "metadata": {},
 77 |    "source": [
 78 |     "## 字段说明\n",
 79 |     "\n",
 80 |     "| 字段 | 说明 |\n",
 81 |     "| ---- | ---- |\n",
 82 |     "| restId | 餐馆 id (从 0 开始，连续编号) |\n",
 83 |     "| name | 餐馆名称 |"
 84 |    ]
 85 |   },
 86 |   {
 87 |    "cell_type": "code",
 88 |    "execution_count": 82,
 89 |    "metadata": {},
 90 |    "outputs": [
 91 |     {
 92 |      "data": {
 93 |       "text/html": [
 94 |        "<div>\n",
 95 |        "<style scoped>\n",
 96 |        "    .dataframe tbody tr th:only-of-type {\n",
 97 |        "        vertical-align: middle;\n",
 98 |        "    }\n",
 99 |        "\n",
100 |        "    .dataframe tbody tr th {\n",
101 |        "        vertical-align: top;\n",
102 |        "    }\n",
103 |        "\n",
104 |        "    .dataframe thead th {\n",
105 |        "        text-align: right;\n",
106 |        "    }\n",
107 |        "</style>\n",
108 |        "<table border=\"1\" class=\"dataframe\">\n",
109 |        "  <thead>\n",
110 |        "    <tr style=\"text-align: right;\">\n",
111 |        "      <th></th>\n",
112 |        "      <th>restId</th>\n",
113 |        "      <th>name</th>\n",
114 |        "    </tr>\n",
115 |        "  </thead>\n",
116 |        "  <tbody>\n",
117 |        "    <tr>\n",
118 |        "      <th>210902</th>\n",
119 |        "      <td>210902</td>\n",
120 |        "      <td>NaN</td>\n",
121 |        "    </tr>\n",
122 |        "    <tr>\n",
123 |        "      <th>124832</th>\n",
124 |        "      <td>124832</td>\n",
125 |        "      <td>NaN</td>\n",
126 |        "    </tr>\n",
127 |        "    <tr>\n",
128 |        "      <th>26766</th>\n",
129 |        "      <td>26766</td>\n",
130 |        "      <td>香锅制造(新苏天地店)</td>\n",
131 |        "    </tr>\n",
132 |        "    <tr>\n",
133 |        "      <th>91754</th>\n",
134 |        "      <td>91754</td>\n",
135 |        "      <td>NaN</td>\n",
136 |        "    </tr>\n",
137 |        "    <tr>\n",
138 |        "      <th>204465</th>\n",
139 |        "      <td>204465</td>\n",
140 |        "      <td>西部牛扒城(湖塘店)</td>\n",
141 |        "    </tr>\n",
142 |        "    <tr>\n",
143 |        "      <th>36475</th>\n",
144 |        "      <td>36475</td>\n",
145 |        "      <td>NaN</td>\n",
146 |        "    </tr>\n",
147 |        "    <tr>\n",
148 |        "      <th>231861</th>\n",
149 |        "      <td>231861</td>\n",
150 |        "      <td>四季火锅</td>\n",
151 |        "    </tr>\n",
152 |        "    <tr>\n",
153 |        "      <th>79816</th>\n",
154 |        "      <td>79816</td>\n",
155 |        "      <td>NaN</td>\n",
156 |        "    </tr>\n",
157 |        "    <tr>\n",
158 |        "      <th>140694</th>\n",
159 |        "      <td>140694</td>\n",
160 |        "      <td>彝家牛汤锅</td>\n",
161 |        "    </tr>\n",
162 |        "    <tr>\n",
163 |        "      <th>169641</th>\n",
164 |        "      <td>169641</td>\n",
165 |        "      <td>春秋</td>\n",
166 |        "    </tr>\n",
167 |        "    <tr>\n",
168 |        "      <th>33809</th>\n",
169 |        "      <td>33809</td>\n",
170 |        "      <td>九头鸟酒家(永定门店)</td>\n",
171 |        "    </tr>\n",
172 |        "    <tr>\n",
173 |        "      <th>236919</th>\n",
174 |        "      <td>236919</td>\n",
175 |        "      <td>老上海城隍庙小吃(人民大学店)</td>\n",
176 |        "    </tr>\n",
177 |        "    <tr>\n",
178 |        "      <th>182387</th>\n",
179 |        "      <td>182387</td>\n",
180 |        "      <td>河源三家村酒楼</td>\n",
181 |        "    </tr>\n",
182 |        "    <tr>\n",
183 |        "      <th>140475</th>\n",
184 |        "      <td>140475</td>\n",
185 |        "      <td>荣记麻辣烫</td>\n",
186 |        "    </tr>\n",
187 |        "    <tr>\n",
188 |        "      <th>194224</th>\n",
189 |        "      <td>194224</td>\n",
190 |        "      <td>NaN</td>\n",
191 |        "    </tr>\n",
192 |        "    <tr>\n",
193 |        "      <th>152406</th>\n",
194 |        "      <td>152406</td>\n",
195 |        "      <td>鼎丰真(东四马路店)</td>\n",
196 |        "    </tr>\n",
197 |        "    <tr>\n",
198 |        "      <th>11701</th>\n",
199 |        "      <td>11701</td>\n",
200 |        "      <td>南亚餐厅</td>\n",
201 |        "    </tr>\n",
202 |        "    <tr>\n",
203 |        "      <th>58805</th>\n",
204 |        "      <td>58805</td>\n",
205 |        "      <td>益丰坊(虎泉店)</td>\n",
206 |        "    </tr>\n",
207 |        "    <tr>\n",
208 |        "      <th>15641</th>\n",
209 |        "      <td>15641</td>\n",
210 |        "      <td>万达艾美酒店大堂吧</td>\n",
211 |        "    </tr>\n",
212 |        "    <tr>\n",
213 |        "      <th>43424</th>\n",
214 |        "      <td>43424</td>\n",
215 |        "      <td>新美心绿姿生活</td>\n",
216 |        "    </tr>\n",
217 |        "  </tbody>\n",
218 |        "</table>\n",
219 |        "</div>"
220 |       ],
221 |       "text/plain": [
222 |        "        restId             name\n",
223 |        "210902  210902              NaN\n",
224 |        "124832  124832              NaN\n",
225 |        "26766    26766      香锅制造(新苏天地店)\n",
226 |        "91754    91754              NaN\n",
227 |        "204465  204465       西部牛扒城(湖塘店)\n",
228 |        "36475    36475              NaN\n",
229 |        "231861  231861             四季火锅\n",
230 |        "79816    79816              NaN\n",
231 |        "140694  140694            彝家牛汤锅\n",
232 |        "169641  169641               春秋\n",
233 |        "33809    33809      九头鸟酒家(永定门店)\n",
234 |        "236919  236919  老上海城隍庙小吃(人民大学店)\n",
235 |        "182387  182387          河源三家村酒楼\n",
236 |        "140475  140475            荣记麻辣烫\n",
237 |        "194224  194224              NaN\n",
238 |        "152406  152406       鼎丰真(东四马路店)\n",
239 |        "11701    11701             南亚餐厅\n",
240 |        "58805    58805         益丰坊(虎泉店)\n",
241 |        "15641    15641        万达艾美酒店大堂吧\n",
242 |        "43424    43424          新美心绿姿生活"
243 |       ]
244 |      },
245 |      "execution_count": 82,
246 |      "metadata": {},
247 |      "output_type": "execute_result"
248 |     }
249 |    ],
250 |    "source": [
251 |     "restaurants.sample(20)"
252 |    ]
253 |   },
254 |   {
255 |    "cell_type": "markdown",
256 |    "metadata": {},
257 |    "source": [
258 |     "# 2. ratings.csv"
259 |    ]
260 |   },
261 |   {
262 |    "cell_type": "markdown",
263 |    "metadata": {},
264 |    "source": [
265 |     "## 加载数据"
266 |    ]
267 |   },
268 |   {
269 |    "cell_type": "code",
270 |    "execution_count": 89,
271 |    "metadata": {},
272 |    "outputs": [
273 |     {
274 |      "name": "stdout",
275 |      "output_type": "stream",
276 |      "text": [
277 |       "用户 数目：542706\n",
278 |       "评分/评论 数目（总计）：4422473\n",
279 |       "\n",
280 |       "总体 评分 数目（[1,5]）：3293878\n",
281 |       "环境 评分 数目（[1,5]）：4076220\n",
282 |       "口味 评分 数目（[1,5]）：4093819\n",
283 |       "服务 评分 数目（[1,5]）：4076220\n",
284 |       "评论 数目：4107409\n"
285 |      ]
286 |     }
287 |    ],
288 |    "source": [
289 |     "pd_ratings = pd.read_csv(path+'ratings.csv')\n",
290 |     "\n",
291 |     "print('用户 数目：%d' % pd_ratings.userId.unique().shape[0])\n",
292 |     "print('评分/评论 数目（总计）：%d\\n' % pd_ratings.shape[0])\n",
293 |     "\n",
294 |     "print('总体 评分 数目（[1,5]）：%d' % pd_ratings[(pd_ratings.rating>=1) & (pd_ratings.rating<=5)].shape[0])\n",
295 |     "print('环境 评分 数目（[1,5]）：%d' % pd_ratings[(pd_ratings.rating_env>=1) & (pd_ratings.rating_env<=5)].shape[0])\n",
296 |     "print('口味 评分 数目（[1,5]）：%d' % pd_ratings[(pd_ratings.rating_flavor>=1) & (pd_ratings.rating_flavor<=5)].shape[0])\n",
297 |     "print('服务 评分 数目（[1,5]）：%d' % pd_ratings[(pd_ratings.rating_service>=1) & (pd_ratings.rating_service<=5)].shape[0])\n",
298 |     "print('评论 数目：%d' % pd_ratings[~pd_ratings.comment.isna()].shape[0])"
299 |    ]
300 |   },
301 |   {
302 |    "cell_type": "markdown",
303 |    "metadata": {},
304 |    "source": [
305 |     "## 字段说明\n",
306 |     "\n",
307 |     "| 字段 | 说明 |\n",
308 |     "| ---- | ---- |\n",
309 |     "| userId | 用户 id (从 0 开始，连续编号) |\n",
310 |     "| restId | 即 restaurants.csv 中的 restId |\n",
311 |     "| rating | 总体评分，[0,5] 之间的整数 |\n",
312 |     "| rating_env | 环境评分，[1,5] 之间的整数 |\n",
313 |     "| rating_flavor | 口味评分，[1,5] 之间的整数 |\n",
314 |     "| rating_service | 服务评分，[1,5] 之间的整数 |\n",
315 |     "| timestamp | 评分时间戳 |\n",
316 |     "| comment |  评论内容 |"
317 |    ]
318 |   },
319 |   {
320 |    "cell_type": "code",
321 |    "execution_count": 84,
322 |    "metadata": {
323 |     "scrolled": false
324 |    },
325 |    "outputs": [
326 |     {
327 |      "data": {
328 |       "text/html": [
329 |        "<div>\n",
330 |        "<style scoped>\n",
331 |        "    .dataframe tbody tr th:only-of-type {\n",
332 |        "        vertical-align: middle;\n",
333 |        "    }\n",
334 |        "\n",
335 |        "    .dataframe tbody tr th {\n",
336 |        "        vertical-align: top;\n",
337 |        "    }\n",
338 |        "\n",
339 |        "    .dataframe thead th {\n",
340 |        "        text-align: right;\n",
341 |        "    }\n",
342 |        "</style>\n",
343 |        "<table border=\"1\" class=\"dataframe\">\n",
344 |        "  <thead>\n",
345 |        "    <tr style=\"text-align: right;\">\n",
346 |        "      <th></th>\n",
347 |        "      <th>userId</th>\n",
348 |        "      <th>restId</th>\n",
349 |        "      <th>rating</th>\n",
350 |        "      <th>rating_env</th>\n",
351 |        "      <th>rating_flavor</th>\n",
352 |        "      <th>rating_service</th>\n",
353 |        "      <th>timestamp</th>\n",
354 |        "      <th>comment</th>\n",
355 |        "    </tr>\n",
356 |        "  </thead>\n",
357 |        "  <tbody>\n",
358 |        "    <tr>\n",
359 |        "      <th>3331708</th>\n",
360 |        "      <td>6802</td>\n",
361 |        "      <td>183728</td>\n",
362 |        "      <td>3.0</td>\n",
363 |        "      <td>3.0</td>\n",
364 |        "      <td>4.0</td>\n",
365 |        "      <td>3.0</td>\n",
366 |        "      <td>1315673880000</td>\n",
367 |        "      <td>环境不错，停车方便，交通也比较方便，东西齐全，应有尽有，吃、喝、玩、乐样样齐全，还有个五星级...</td>\n",
368 |        "    </tr>\n",
369 |        "    <tr>\n",
370 |        "      <th>3332473</th>\n",
371 |        "      <td>3106</td>\n",
372 |        "      <td>183750</td>\n",
373 |        "      <td>5.0</td>\n",
374 |        "      <td>4.0</td>\n",
375 |        "      <td>4.0</td>\n",
376 |        "      <td>4.0</td>\n",
377 |        "      <td>1260155880000</td>\n",
378 |        "      <td>去过两次，都是由日本朋友带着去的，很喜欢那种在小巷子深处的店，总觉得那样的店料理会很好吃。最...</td>\n",
379 |        "    </tr>\n",
380 |        "    <tr>\n",
381 |        "      <th>291609</th>\n",
382 |        "      <td>39590</td>\n",
383 |        "      <td>13570</td>\n",
384 |        "      <td>3.0</td>\n",
385 |        "      <td>3.0</td>\n",
386 |        "      <td>2.0</td>\n",
387 |        "      <td>3.0</td>\n",
388 |        "      <td>1324792500000</td>\n",
389 |        "      <td>朋友请客，两个人中午去吃的，虽然不是节假日，但人还是非常的多，等了很长时间才上餐，价位偏高，...</td>\n",
390 |        "    </tr>\n",
391 |        "    <tr>\n",
392 |        "      <th>749582</th>\n",
393 |        "      <td>59192</td>\n",
394 |        "      <td>38519</td>\n",
395 |        "      <td>4.0</td>\n",
396 |        "      <td>2.0</td>\n",
397 |        "      <td>3.0</td>\n",
398 |        "      <td>2.0</td>\n",
399 |        "      <td>1321430760000</td>\n",
400 |        "      <td>十一长假之前，我们的房子终于有了好消息，这个月底就可以拿到钥匙，真是不容易，盼星星盼月亮的，...</td>\n",
401 |        "    </tr>\n",
402 |        "    <tr>\n",
403 |        "      <th>719908</th>\n",
404 |        "      <td>241643</td>\n",
405 |        "      <td>36382</td>\n",
406 |        "      <td>1.0</td>\n",
407 |        "      <td>2.0</td>\n",
408 |        "      <td>1.0</td>\n",
409 |        "      <td>1.0</td>\n",
410 |        "      <td>1271862180000</td>\n",
411 |        "      <td>很差的一家店！公司聚餐居然选在这里，真是个大大的失策！\\n点的菜迟迟不上，不知道是故意不上还...</td>\n",
412 |        "    </tr>\n",
413 |        "    <tr>\n",
414 |        "      <th>3127953</th>\n",
415 |        "      <td>12481</td>\n",
416 |        "      <td>173459</td>\n",
417 |        "      <td>4.0</td>\n",
418 |        "      <td>3.0</td>\n",
419 |        "      <td>3.0</td>\n",
420 |        "      <td>3.0</td>\n",
421 |        "      <td>1300407540000</td>\n",
422 |        "      <td>这家是离家最近的一家城市超市了，所以自然要进去随便逛逛啦。\\n因为附近是居民区，自然光顾的主...</td>\n",
423 |        "    </tr>\n",
424 |        "    <tr>\n",
425 |        "      <th>2068253</th>\n",
426 |        "      <td>13070</td>\n",
427 |        "      <td>115853</td>\n",
428 |        "      <td>3.0</td>\n",
429 |        "      <td>3.0</td>\n",
430 |        "      <td>3.0</td>\n",
431 |        "      <td>2.0</td>\n",
432 |        "      <td>1308671820000</td>\n",
433 |        "      <td>以前觉得还行，但有了85度之后就不行了。要了个提拉米苏，不行，太甜了。\\n辣松的味道倒不错，...</td>\n",
434 |        "    </tr>\n",
435 |        "    <tr>\n",
436 |        "      <th>640356</th>\n",
437 |        "      <td>168006</td>\n",
438 |        "      <td>33263</td>\n",
439 |        "      <td>NaN</td>\n",
440 |        "      <td>3.0</td>\n",
441 |        "      <td>5.0</td>\n",
442 |        "      <td>3.0</td>\n",
443 |        "      <td>1224868560000</td>\n",
444 |        "      <td>算比较地道的川菜了 味道辣的很正 强力推荐 据说还是标点美食的... 香辣鸡翅每去必点~！不...</td>\n",
445 |        "    </tr>\n",
446 |        "    <tr>\n",
447 |        "      <th>1222261</th>\n",
448 |        "      <td>76280</td>\n",
449 |        "      <td>65171</td>\n",
450 |        "      <td>3.0</td>\n",
451 |        "      <td>2.0</td>\n",
452 |        "      <td>2.0</td>\n",
453 |        "      <td>2.0</td>\n",
454 |        "      <td>1302136740000</td>\n",
455 |        "      <td>为什么这么多人说好吃啊？为什么这么多人说肉多啊？难道是我人品有问题？\\n这个也是慕名而去的~...</td>\n",
456 |        "    </tr>\n",
457 |        "    <tr>\n",
458 |        "      <th>101366</th>\n",
459 |        "      <td>67372</td>\n",
460 |        "      <td>2853</td>\n",
461 |        "      <td>1.0</td>\n",
462 |        "      <td>1.0</td>\n",
463 |        "      <td>1.0</td>\n",
464 |        "      <td>1.0</td>\n",
465 |        "      <td>1283741400000</td>\n",
466 |        "      <td>两年前经常去这家吃卤煮，感觉特别好吃，可是最近吃了一次，让我大失所望。。。\\n卤煮的汤和食材...</td>\n",
467 |        "    </tr>\n",
468 |        "  </tbody>\n",
469 |        "</table>\n",
470 |        "</div>"
471 |       ],
472 |       "text/plain": [
473 |        "         userId  restId  rating  rating_env  rating_flavor  rating_service  \\\n",
474 |        "3331708    6802  183728     3.0         3.0            4.0             3.0   \n",
475 |        "3332473    3106  183750     5.0         4.0            4.0             4.0   \n",
476 |        "291609    39590   13570     3.0         3.0            2.0             3.0   \n",
477 |        "749582    59192   38519     4.0         2.0            3.0             2.0   \n",
478 |        "719908   241643   36382     1.0         2.0            1.0             1.0   \n",
479 |        "3127953   12481  173459     4.0         3.0            3.0             3.0   \n",
480 |        "2068253   13070  115853     3.0         3.0            3.0             2.0   \n",
481 |        "640356   168006   33263     NaN         3.0            5.0             3.0   \n",
482 |        "1222261   76280   65171     3.0         2.0            2.0             2.0   \n",
483 |        "101366    67372    2853     1.0         1.0            1.0             1.0   \n",
484 |        "\n",
485 |        "             timestamp                                            comment  \n",
486 |        "3331708  1315673880000  环境不错，停车方便，交通也比较方便，东西齐全，应有尽有，吃、喝、玩、乐样样齐全，还有个五星级...  \n",
487 |        "3332473  1260155880000  去过两次，都是由日本朋友带着去的，很喜欢那种在小巷子深处的店，总觉得那样的店料理会很好吃。最...  \n",
488 |        "291609   1324792500000  朋友请客，两个人中午去吃的，虽然不是节假日，但人还是非常的多，等了很长时间才上餐，价位偏高，...  \n",
489 |        "749582   1321430760000  十一长假之前，我们的房子终于有了好消息，这个月底就可以拿到钥匙，真是不容易，盼星星盼月亮的，...  \n",
490 |        "719908   1271862180000  很差的一家店！公司聚餐居然选在这里，真是个大大的失策！\\n点的菜迟迟不上，不知道是故意不上还...  \n",
491 |        "3127953  1300407540000  这家是离家最近的一家城市超市了，所以自然要进去随便逛逛啦。\\n因为附近是居民区，自然光顾的主...  \n",
492 |        "2068253  1308671820000  以前觉得还行，但有了85度之后就不行了。要了个提拉米苏，不行，太甜了。\\n辣松的味道倒不错，...  \n",
493 |        "640356   1224868560000  算比较地道的川菜了 味道辣的很正 强力推荐 据说还是标点美食的... 香辣鸡翅每去必点~！不...  \n",
494 |        "1222261  1302136740000  为什么这么多人说好吃啊？为什么这么多人说肉多啊？难道是我人品有问题？\\n这个也是慕名而去的~...  \n",
495 |        "101366   1283741400000  两年前经常去这家吃卤煮，感觉特别好吃，可是最近吃了一次，让我大失所望。。。\\n卤煮的汤和食材...  "
496 |       ]
497 |      },
498 |      "execution_count": 84,
499 |      "metadata": {},
500 |      "output_type": "execute_result"
501 |     }
502 |    ],
503 |    "source": [
504 |     "pd_ratings.sample(10)"
505 |    ]
506 |   },
507 |   {
508 |    "cell_type": "markdown",
509 |    "metadata": {},
510 |    "source": [
511 |     "# 3. links.csv"
512 |    ]
513 |   },
514 |   {
515 |    "cell_type": "markdown",
516 |    "metadata": {},
517 |    "source": [
518 |     "## 加载数据"
519 |    ]
520 |   },
521 |   {
522 |    "cell_type": "code",
523 |    "execution_count": 85,
524 |    "metadata": {},
525 |    "outputs": [],
526 |    "source": [
527 |     "links = pd.read_csv(path + 'links.csv')"
528 |    ]
529 |   },
530 |   {
531 |    "cell_type": "markdown",
532 |    "metadata": {},
533 |    "source": [
534 |     "## 字段说明\n",
535 |     "\n",
536 |     "| 字段 | 说明 |\n",
537 |     "| ---- | ---- |\n",
538 |     "| restId | 即 restaurants.csv 和 ratings.csv 中的 restId |\n",
539 |     "| dianpingId | 大众点评网的餐馆编号 |"
540 |    ]
541 |   },
542 |   {
543 |    "cell_type": "code",
544 |    "execution_count": 86,
545 |    "metadata": {},
546 |    "outputs": [
547 |     {
548 |      "data": {
549 |       "text/html": [
550 |        "<div>\n",
551 |        "<style scoped>\n",
552 |        "    .dataframe tbody tr th:only-of-type {\n",
553 |        "        vertical-align: middle;\n",
554 |        "    }\n",
555 |        "\n",
556 |        "    .dataframe tbody tr th {\n",
557 |        "        vertical-align: top;\n",
558 |        "    }\n",
559 |        "\n",
560 |        "    .dataframe thead th {\n",
561 |        "        text-align: right;\n",
562 |        "    }\n",
563 |        "</style>\n",
564 |        "<table border=\"1\" class=\"dataframe\">\n",
565 |        "  <thead>\n",
566 |        "    <tr style=\"text-align: right;\">\n",
567 |        "      <th></th>\n",
568 |        "      <th>restId</th>\n",
569 |        "      <th>dianpingId</th>\n",
570 |        "    </tr>\n",
571 |        "  </thead>\n",
572 |        "  <tbody>\n",
573 |        "    <tr>\n",
574 |        "      <th>138492</th>\n",
575 |        "      <td>138492</td>\n",
576 |        "      <td>3566359</td>\n",
577 |        "    </tr>\n",
578 |        "    <tr>\n",
579 |        "      <th>158007</th>\n",
580 |        "      <td>158007</td>\n",
581 |        "      <td>2484433</td>\n",
582 |        "    </tr>\n",
583 |        "    <tr>\n",
584 |        "      <th>16170</th>\n",
585 |        "      <td>16170</td>\n",
586 |        "      <td>3651451</td>\n",
587 |        "    </tr>\n",
588 |        "    <tr>\n",
589 |        "      <th>116637</th>\n",
590 |        "      <td>116637</td>\n",
591 |        "      <td>5143029</td>\n",
592 |        "    </tr>\n",
593 |        "    <tr>\n",
594 |        "      <th>191554</th>\n",
595 |        "      <td>191554</td>\n",
596 |        "      <td>2734621</td>\n",
597 |        "    </tr>\n",
598 |        "    <tr>\n",
599 |        "      <th>192481</th>\n",
600 |        "      <td>192481</td>\n",
601 |        "      <td>3000367</td>\n",
602 |        "    </tr>\n",
603 |        "    <tr>\n",
604 |        "      <th>40978</th>\n",
605 |        "      <td>40978</td>\n",
606 |        "      <td>3168181</td>\n",
607 |        "    </tr>\n",
608 |        "    <tr>\n",
609 |        "      <th>196832</th>\n",
610 |        "      <td>196832</td>\n",
611 |        "      <td>3523291</td>\n",
612 |        "    </tr>\n",
613 |        "    <tr>\n",
614 |        "      <th>6048</th>\n",
615 |        "      <td>6048</td>\n",
616 |        "      <td>2435827</td>\n",
617 |        "    </tr>\n",
618 |        "    <tr>\n",
619 |        "      <th>200405</th>\n",
620 |        "      <td>200405</td>\n",
621 |        "      <td>4130573</td>\n",
622 |        "    </tr>\n",
623 |        "    <tr>\n",
624 |        "      <th>69792</th>\n",
625 |        "      <td>69792</td>\n",
626 |        "      <td>2853502</td>\n",
627 |        "    </tr>\n",
628 |        "    <tr>\n",
629 |        "      <th>153075</th>\n",
630 |        "      <td>153075</td>\n",
631 |        "      <td>2000257</td>\n",
632 |        "    </tr>\n",
633 |        "    <tr>\n",
634 |        "      <th>8528</th>\n",
635 |        "      <td>8528</td>\n",
636 |        "      <td>2651221</td>\n",
637 |        "    </tr>\n",
638 |        "    <tr>\n",
639 |        "      <th>196930</th>\n",
640 |        "      <td>196930</td>\n",
641 |        "      <td>3534673</td>\n",
642 |        "    </tr>\n",
643 |        "    <tr>\n",
644 |        "      <th>224063</th>\n",
645 |        "      <td>224063</td>\n",
646 |        "      <td>3138160</td>\n",
647 |        "    </tr>\n",
648 |        "    <tr>\n",
649 |        "      <th>3434</th>\n",
650 |        "      <td>3434</td>\n",
651 |        "      <td>2185753</td>\n",
652 |        "    </tr>\n",
653 |        "    <tr>\n",
654 |        "      <th>125490</th>\n",
655 |        "      <td>125490</td>\n",
656 |        "      <td>2112511</td>\n",
657 |        "    </tr>\n",
658 |        "    <tr>\n",
659 |        "      <th>230533</th>\n",
660 |        "      <td>230533</td>\n",
661 |        "      <td>4122445</td>\n",
662 |        "    </tr>\n",
663 |        "    <tr>\n",
664 |        "      <th>130597</th>\n",
665 |        "      <td>130597</td>\n",
666 |        "      <td>2632129</td>\n",
667 |        "    </tr>\n",
668 |        "    <tr>\n",
669 |        "      <th>186956</th>\n",
670 |        "      <td>186956</td>\n",
671 |        "      <td>2233513</td>\n",
672 |        "    </tr>\n",
673 |        "  </tbody>\n",
674 |        "</table>\n",
675 |        "</div>"
676 |       ],
677 |       "text/plain": [
678 |        "        restId  dianpingId\n",
679 |        "138492  138492     3566359\n",
680 |        "158007  158007     2484433\n",
681 |        "16170    16170     3651451\n",
682 |        "116637  116637     5143029\n",
683 |        "191554  191554     2734621\n",
684 |        "192481  192481     3000367\n",
685 |        "40978    40978     3168181\n",
686 |        "196832  196832     3523291\n",
687 |        "6048      6048     2435827\n",
688 |        "200405  200405     4130573\n",
689 |        "69792    69792     2853502\n",
690 |        "153075  153075     2000257\n",
691 |        "8528      8528     2651221\n",
692 |        "196930  196930     3534673\n",
693 |        "224063  224063     3138160\n",
694 |        "3434      3434     2185753\n",
695 |        "125490  125490     2112511\n",
696 |        "230533  230533     4122445\n",
697 |        "130597  130597     2632129\n",
698 |        "186956  186956     2233513"
699 |       ]
700 |      },
701 |      "execution_count": 86,
702 |      "metadata": {},
703 |      "output_type": "execute_result"
704 |     }
705 |    ],
706 |    "source": [
707 |     "links.sample(20)"
708 |    ]
709 |   }
710 |  ],
711 |  "metadata": {
712 |   "kernelspec": {
713 |    "display_name": "Python 3",
714 |    "language": "python",
715 |    "name": "python3"
716 |   },
717 |   "language_info": {
718 |    "codemirror_mode": {
719 |     "name": "ipython",
720 |     "version": 3
721 |    },
722 |    "file_extension": ".py",
723 |    "mimetype": "text/x-python",
724 |    "name": "python",
725 |    "nbconvert_exporter": "python",
726 |    "pygments_lexer": "ipython3",
727 |    "version": "3.6.4"
728 |   },
729 |   "widgets": {
730 |    "state": {},
731 |    "version": "1.1.2"
732 |   }
733 |  },
734 |  "nbformat": 4,
735 |  "nbformat_minor": 2
736 | }
737 | 


--------------------------------------------------------------------------------
/docs/images/recruit/jd_header.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/SophonPlus/ChineseNlpCorpus/a0b781d72158bde242c0626c9e354d2b328865ff/docs/images/recruit/jd_header.png


--------------------------------------------------------------------------------
/docs/images/recruit/recruit_banner.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/SophonPlus/ChineseNlpCorpus/a0b781d72158bde242c0626c9e354d2b328865ff/docs/images/recruit/recruit_banner.png


--------------------------------------------------------------------------------
/docs/recruit/architect.md:
--------------------------------------------------------------------------------
 1 | ![](../images/recruit/jd_header.png)
 2 | 
 3 | 【岗位名称】系统架构师（人工智能产品）
 4 | 
 5 | 厦门 / 5-10 年 / 本科 / 15k-25k
 6 | 
 7 | ---
 8 | 
 9 | 【岗位职责】
10 | 
11 | 1. 架构设计机器人客服软件系统
12 | 2. 组织重构现有机器人客服软件系统
13 | 3. 制定并优化机器人客服软件定制开发流程，提升定制开发的效率
14 | 4. 制定内部技术标准，优化软件开发、测试、部署全流程，提升研发效率
15 | 5. 对研发/测试人员进行技术培训，并指导其日常工作，打造科学、严谨、高效的技术团队
16 | 
17 | 【任职要求】
18 | 
19 | 1. 本科（或以上）学历，计算机、软件工程、自动化等相关专业毕业
20 | 2. 5 年以上软件开发/系统架构相关工作经验
21 | 3. 精通微服务系统架构、设计模式、常见数据结构及相关算法
22 | 4. 具备优异的逻辑思维、系统抽象能力，强悍的工程实现能力
23 | 
24 | 【优先录用】
25 | 
26 | 1. 具备 1 年以上大型、复杂系统（尤其是人工智能或 SAAS 产品）软件开发或系统架构经验
27 | 2. 了解自然语言处理（或机器学习、人工智能）基础知识，能迅速理解机器人客服软件各功能模块
28 | 3. 熟悉 Python 开发
29 | 
30 | 【成长路径】
31 | 
32 | 1. 纵向发展：公司完善的职级职等体系，配合 OKR 工具，帮助你在成为技术专家的道路上不断突破进取
33 | 2. 横向发展：扁平化的组织架构，大量的技术分享、交流活动，有机会接触到人工智能前沿相关产品、技术岗位，根据个人意愿，考核通过后，可申请调岗
34 | 
35 | 【团队介绍】
36 | 
37 | 1. 以人工智能技术服务全球 30 亿用户
38 | 2. 人工智能朝阳产业，风口中的风口，期待你的加盟
39 | 3. 极客精神、技术驱动，做有温度的技术，让世界更美好
40 | 4. 每月不定期小组及大部门分享、交流活动，一同领略人工智能前沿的无限魅力……
41 | 
42 | 【公司福利、交通】
43 | 
44 | 1. 每周工作 5 天（双休），劳逸结合，高效执行
45 | 2. 每天半小时弹性工作时间，可自由申请调休（不影响工作条件下）
46 | 3. 交通便利，公司楼下即为莲花路口地铁站，如风一般快捷
47 | 4. 市中心，一站公交直达莲坂、明发商业广场，吃喝不愁
48 | 5. 公司有用餐区、咖啡厅、按摩椅
49 | 6. 下午茶，节日礼物
50 | 7. 各种团建活动
51 | 8. 工作满 3 年以上，绩效/价值观优秀，有机会申请期权奖励
52 | 
53 | ---
54 | 
55 | 【联系方式】
56 | 
57 | - 蔡先生, jinhua@kuaishang.com.cn
58 | - 蓝先生, lanzl@kuaishang.com.cn, 180-3025-1206
59 | - 叶女士, yeyp@kuaishang.com.cn, 0592-5380356
60 | 


--------------------------------------------------------------------------------
/docs/recruit/engineer.md:
--------------------------------------------------------------------------------
 1 | ![](../images/recruit/jd_header.png)
 2 | 
 3 | 【岗位名称】自然语言处理算法工程师
 4 | 
 5 | 厦门 / 3-5 年 / 硕士 / 15k-25k
 6 | 
 7 | ---
 8 | 
 9 | 【岗位职责】
10 | 
11 | 1. 参与自动营销机器人客服软件核心模块的设计与研发，满足客户需求
12 | 2. 专注对话系统的若干研究/应用领域/关键技术，展开深入研究，保持技术领先优势
13 | 
14 | 【任职要求】
15 | 
16 | 1. 硕士（或以上）学历
17 | 2. 3 年以上对话/问答系统相关研究或开发经验
18 | 3. 对话/问答系统核心技术骨干，了解各模块的设计与构造，并深入掌握其中的若干模块或关键技术
19 | 4. 优秀的工程实现能力，能快速实现各种创新技术构想，编码和文档规范
20 | 5. 英文阅读理解能力优秀，具有良好的英文技术文献阅读和理解能力
21 | 
22 | 【优先录用】
23 | 
24 | 1. 具有对话/问答系统相关产品研发成功经验者优先录用
25 | 
26 | 【团队介绍】
27 | 
28 | 1. 以人工智能技术服务全球 30 亿用户
29 | 2. 专注面向行业细分领域的自动营销机器人，客户需求旺盛，产品前景无限
30 | 3. 极客精神、技术驱动，做有温度的技术，让世界更美好
31 | 4. 每月不定期小组及大部门分享、交流活动，团队氛围燃爆……
32 | 
33 | 【公司福利、交通】
34 | 
35 | 1. 每周工作 5 天（双休），劳逸结合，高效执行
36 | 2. 每天半小时弹性工作时间，可自由申请调休（不影响工作条件下）
37 | 3. 交通便利，公司楼下即为莲花路口地铁站，如风一般快捷
38 | 4. 市中心，一站公交直达莲坂、明发商业广场，吃喝不愁
39 | 5. 公司有用餐区、咖啡厅、按摩椅
40 | 6. 下午茶，节日礼物
41 | 7. 各种团建活动
42 | 8. 工作满 3 年以上，绩效/价值观优秀，有机会申请期权奖励
43 | 
44 | ---
45 | 
46 | 【联系方式】
47 | 
48 | - 蔡先生, jinhua@kuaishang.com.cn
49 | - 蓝先生, lanzl@kuaishang.com.cn, 180-3025-1206
50 | - 叶女士, yeyp@kuaishang.com.cn, 0592-5380356
51 | 


--------------------------------------------------------------------------------
/docs/recruit/researcher.md:
--------------------------------------------------------------------------------
 1 | ![](../images/recruit/jd_header.png)
 2 | 
 3 | 【岗位名称】自然语言人机交互应用研究
 4 | 
 5 | 厦门 / 5-10 年 / 硕士 / 20k-35k
 6 | 
 7 | ---
 8 | 
 9 | 【岗位职责】
10 | 
11 | 1. 主持设计并组织研发面向行业细分领域的自动营销机器人客服软件
12 | 2. 制定并优化机器人客服软件定制开发流程，显著提升定制开发的效率
13 | 3. 洞察前沿技术发展趋势，帮助提升团队整体技术水平
14 | 
15 | 【任职要求】
16 | 
17 | 1. 硕士（或以上）学历
18 | 2. 5 年以上对话/问答系统相关研究或开发经验
19 | 3. 对话/问答系统核心技术骨干，熟悉各模块的设计与构造，尤其精通对话流程管理与控制（即中控系统）的研发
20 | 4. 优秀的工程实现能力，能快速实现各种创新技术构想，编码和文档规范
21 | 5. 优异的英文文献阅读能力，时刻把握前沿技术发展趋势
22 | 
23 | 【优先录用】
24 | 
25 | 1. 具有对话/问答系统相关产品研发成功经验者优先录用
26 | 
27 | 【团队介绍】
28 | 
29 | 1. 以人工智能技术服务全球 30 亿用户
30 | 2. 专注面向行业细分领域的自动营销机器人，客户需求旺盛，产品前景无限
31 | 3. 极客精神、技术驱动，做有温度的技术，让世界更美好
32 | 4. 每月不定期小组及大部门分享、交流活动，团队氛围燃爆……
33 | 
34 | 【公司福利、交通】
35 | 
36 | 1. 每周工作 5 天（双休），劳逸结合，高效执行
37 | 2. 每天半小时弹性工作时间，可自由申请调休（不影响工作条件下）
38 | 3. 交通便利，公司楼下即为莲花路口地铁站，如风一般快捷
39 | 4. 市中心，一站公交直达莲坂、明发商业广场，吃喝不愁
40 | 5. 公司有用餐区、咖啡厅、按摩椅
41 | 6. 下午茶，节日礼物
42 | 7. 各种团建活动
43 | 8. 工作满 3 年以上，绩效/价值观优秀，有机会申请期权奖励
44 | 
45 | ---
46 | 
47 | 【联系方式】
48 | 
49 | - 蔡先生, jinhua@kuaishang.com.cn
50 | - 蓝先生, lanzl@kuaishang.com.cn, 180-3025-1206
51 | - 叶女士, yeyp@kuaishang.com.cn, 0592-5380356
52 | 


--------------------------------------------------------------------------------