├── .gitignore
├── README.md
├── README_en.md
├── config.json
├── config.py
├── emotion.json
├── filter.py
├── scrapy.cfg
└── tieba
    ├── __init__.py
    ├── commands
        ├── __init__.py
        └── run.py
    ├── items.py
    ├── middlewares.py
    ├── pipelines.py
    ├── settings.py
    └── spiders
        ├── __init__.py
        ├── emotion.py
        ├── helper.py
        └── tieba_spider.py


/.gitignore:
--------------------------------------------------------------------------------
1 | __pycache__/
2 | *.py[cod]
3 | *$py.class
4 | *.log


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | # Tieba_Spider
  2 | [Readme(EN)](README_en.md)
  3 | 
  4 | 贴吧爬虫。
  5 | 
  6 | 
  7 | ## 依赖参考
  8 | Python >= 3.6
  9 | 
 10 | mysql >= 5.5
 11 | 
 12 | beautifulsoup4 >= 4.6.0
 13 | 
 14 | **scrapy == 2.4.1**
 15 | 
 16 | mysqlclient >= 1.3.10
 17 | 
 18 | ## 使用方法
 19 | 先打开config.json文件，在其中配置好数据库的域名、用户名和密码。接着直接运行命令即可：
 20 | ```
 21 | scrapy run <贴吧名> <数据库名> <选项>
 22 | ```
 23 | 其中贴吧名不含末尾的“吧”字，而数据库名则是要存入的数据库名字，数据库在爬取前会被创建。例如
 24 | ```
 25 | scrapy run 仙五前修改 Pal5Q_Diy
 26 | ```
 27 | 但若要在控制台输入中文(非ASCII字符)，请确保控制台编码为UTF8。
 28 | 
 29 | 若在config.json里面已经配置好贴吧名和对应数据库名，则可以忽略数据库名。若忽略贴吧名，则爬取config.json里面DEFAULT的数据库。
 30 | 
 31 | **特别提醒** 任务一旦断开，不可继续进行。因此SSH打开任务时，请保证不要断开连接，或者考虑使用后台任务或者screen命令等。
 32 | 
 33 | ## 选项说明
 34 | 
 35 | |短形式|长形式     |参数个数|作用                              |举例                           |
 36 | |------|-----------|--------|----------------------------------|-------------------------------|
 37 | |-p    |--pages    |2       |设定爬取帖子的开始页和结束页      |scrapy run ... -p 2 5          |
 38 | |-g    |--good_only|0       |只爬精品帖                        |scrapy run ... -g              |
 39 | |-s    |--see_lz   |0       |只看楼主，即不爬非楼主的楼层      |scrapy run ... -s              |
 40 | |-f    |--filter   |1       |设定帖子过滤函数名(见`filter.py`) |scrapy run ... -f thread_filter| 
 41 | 
 42 | 举例：
 43 | ```
 44 | scrapy run 仙剑五外传 -gs -p 5 12 -f thread_filter
 45 | ```
 46 | 使用只看楼主模式爬仙剑五外传吧精品帖中第5页到第12页的帖子，其中能通过过滤器`filter.py`中的`thread_filter`函数的帖子及其内容会被存入数据库。
 47 | 
 48 | ## 数据处理
 49 | 对爬取的数据并非原样入库，会进行一些处理。
 50 | 
 51 | 1. 广告楼层会被去掉(右下角有“广告”两字的楼层)。
 52 | 2. 加粗和红字效果丢失为纯文本(beautifulsoup的get_text功能)。
 53 | 3. 常用表情会转换为文字表达(emotion.json，欢迎补充)。
 54 | 4. 图片和视频会变成对应链接(要获取视频链接需要拿到一个302响应)。
 55 | 
 56 | ## 数据保存结构
 57 |  - thread
 58 |  
 59 | 为各帖子的一些基本信息。
 60 | 
 61 | |属性     |类型        |备注                                                    |
 62 | |---------|------------|--------------------------------------------------------|
 63 | |id       |BIGINT(12)  |"http://tieba.baidu.com/p/4778655068" 的ID就是4778655068|
 64 | |title    |VARCHAR(100)|                                                        |
 65 | |author   |VARCHAR(30) |                                                        |
 66 | |reply_num|INT(4)      |回复数量(含楼中楼, 不含1楼)                             |
 67 | |good     |BOOL        |是否为精品帖                                            |
 68 | 
 69 | 
 70 |  - post
 71 | 
 72 | 为各楼层的一些基本信息，包括1楼。
 73 | 
 74 | |属性       |类型       |备注                  |
 75 | |-----------|-----------|----------------------|
 76 | |id         |BIGINT(12) |楼层也有对应ID        |
 77 | |floor      |INT(4)     |楼层编号              |
 78 | |author     |VARCHAR(30)|                      |
 79 | |content    |TEXT       |楼层内容              |
 80 | |time       |DATETIME   |发布时间              |
 81 | |comment_num|INT(4)     |楼中楼回复数量        |
 82 | |thread_id  |BIGINT(12) |楼层的主体帖子ID，外键|
 83 | 
 84 | 
 85 |  - comment
 86 |  
 87 | 楼中楼的一些信息。
 88 | 
 89 | |属性   |类型       |备注                      |
 90 | |-------|-----------|--------------------------|
 91 | |id     |BIGINT(12) |楼中楼也有ID，且和楼层共用|
 92 | |author |VARCHAR(30)|                          |
 93 | |content|TEXT       |楼中楼内容                |
 94 | |time   |DATETIME   |发布时间                  |
 95 | |post_id|BIGINT(12) |楼中楼的主体楼层ID，外键  |
 96 | 
 97 | 爬取方式决定了comment有可能先于对应的post被爬取，从而外键错误。因此任务开始阶段数据库的外键检测会被关闭。
 98 | 
 99 | ## 耗时参考
100 | 耗时和服务器带宽以及爬取时段有关，下面是我的阿里云服务器对几个贴吧的爬取用时，仅供参考。
101 | 
102 | |贴吧名    |帖子数|回复数 |楼中楼数|用时(秒)|
103 | |----------|------|-------|--------|--------|
104 | |pandakill |3638  |41221  |50206   |222.2   |
105 | |lyingman  |11290 |122662 |126670  |718.9   |
106 | |仙剑五外传|67356 |1262705|807435  |7188    |
107 | 
108 | 下面几个吧是同一时刻爬取的：
109 | 
110 | |贴吧名     |帖子数|回复数|楼中楼数|用时(秒)|
111 | |-----------|------|------|--------|--------|
112 | |仙五前修改 |530   |3518  |7045    |79.02   |
113 | |仙剑3高难度|2080  |21293 |16185   |274.6   |
114 | |古剑高难度 |1703  |26086 |32941   |254.0   |
115 | 
116 | **特别提醒** 请注意下爬取数据的占用空间，别把磁盘占满了。
117 | 
118 | ## 更新日志
119 | 更新后请先删除原有的日志`spider.log`。
120 | 
121 | 2020-08-09更新：解决了只爬楼中楼前10层的问题。注：由于python官方已放弃对python 2的支持，此后版本将不再保证python 2能正常运行。
122 | 
123 | 2020-02-23更新：解决了被百度识别为爬虫返回403的问题。
124 | 
125 | 2018-06-13更新：新增支持python 3。请卸载原来的python库`mysql-python`，改为使用`mysqlclient`。
126 | 
127 | 2017-03-23更新：修改了页选项参数形式，增加了只看楼主、只爬精品和自定义过滤帖子功能。
128 | 
129 | ## 参考文献
130 | [Scrapy 1.0 文档][1]
131 | 
132 | [Scrapy 源代码][2]
133 | 
134 | [Beautiful Soup的用法][3]
135 | 
136 | [Ubuntu/Debian 安装lxml的正确方式][4]
137 | 
138 | [Twisted adbapi 源代码][5]
139 | 
140 | [mysql升级8.0后遇到的坑][6]
141 | 
142 | 有什么问题或建议欢迎到[我的主页][7]留言~
143 | 
144 |   [1]: http://scrapy-chs.readthedocs.io/zh_CN/1.0/
145 |   [2]: https://coding.net/u/fmyl/p/scrapy
146 |   [3]: https://cuiqingcai.com/1319.html
147 |   [4]: http://www.cnblogs.com/numbbbbb/p/3434519.html
148 |   [5]: https://github.com/twisted/twisted/blob/twisted-16.5.0/src/twisted/enterprise/adbapi.py
149 |   [6]: https://www.shiqidu.com/d/358
150 |   [7]: https://aqua.hk.cn/Programming/oew0fr/
151 | 


--------------------------------------------------------------------------------
/README_en.md:
--------------------------------------------------------------------------------
  1 | # Tieba_Spider
  2 | [Readme(CN)](README.md)
  3 | 
  4 | Spider for [Baidu Tieba](https://tieba.baidu.com). Crawl threads, posts and comments, and store them into MySQL database. 
  5 | 
  6 | ## Dependency
  7 | Python >= 3.6
  8 | 
  9 | mysql >= 5.5
 10 | 
 11 | beautifulsoup4 >= 4.6.0
 12 | 
 13 | scrapy >= 2.4
 14 | 
 15 | mysqlclient >= 1.3.10
 16 | 
 17 | ## Usage
 18 | First time: Edit `config.json`. Fill in database host, username as well as password.
 19 | 
 20 | Command:
 21 | ```
 22 | scrapy run <Tieba_Name> <Database_Name> <Option>
 23 | ```
 24 | You should not include the extra tail character `吧` in tieba name. Database will be created automatically if not exists. 
 25 | 
 26 | Example:
 27 | ```
 28 | scrapy run 仙五前修改 Pal5Q_Diy
 29 | ```
 30 | Attention: If you type in non-ASCII character, please make sure that the encoding of console is `UTF8`.
 31 | 
 32 | You can omit database name if is has been specified in `config.json`. Tieba name can also be ignored if you would like to crawl the **DEFAULT_TIEBA**.
 33 | 
 34 | **Attention** Pause is not supported. Once the task is interrupted, you have to restart it.
 35 | 
 36 | ## Option
 37 | 
 38 | |Short |Long       |Args    |Function                              |Example                        |
 39 | |------|-----------|--------|--------------------------------------|-------------------------------|
 40 | |-p    |--pages    |2       |The start and end page to be crawled  |scrapy run ... -p 2 5          |
 41 | |-g    |--good_only|0       |Only crawl good threads               |scrapy run ... -g              |
 42 | |-s    |--see_lz   |0       |Only crawl posts from original poster |scrapy run ... -s              |
 43 | |-f    |--filter   |1       |Name of filter (See `filter.py`)      |scrapy run ... -f thread_filter| 
 44 | 
 45 | Example：
 46 | ```
 47 | scrapy run 仙剑五外传 -gs -p 5 12 -f thread_filter
 48 | ```
 49 | Crawl tieba `仙剑五外传` of good threads from page 5 to 12. Only the threads that can pass `thread_filter` function would be recorded. For each thread, only posts from original poster would be recorded.
 50 | 
 51 | ## Text Cleaning
 52 | The crawled message would be modified as follows:
 53 | 
 54 | 1. Advertisements floor (with `广告` on the bottom right) would be discarded。
 55 | 2. Bold or red text effects would be discarded.
 56 | 3. Frequent used emtion picture would be converted to text (by emotion.json).
 57 | 4. Pictures and videos would be stored as their link.
 58 | 
 59 | ## Database Structure
 60 |  - thread
 61 |  
 62 | Base information of each thread.
 63 | 
 64 | |Attribute|Type        |Remark                                                  |
 65 | |---------|------------|--------------------------------------------------------|
 66 | |id       |BIGINT(12)  |"http://tieba.baidu.com/p/4778655068".ID = 4778655068   |
 67 | |title    |VARCHAR(100)|                                                        |
 68 | |author   |VARCHAR(30) |                                                        |
 69 | |reply_num|INT(4)      |Including comment; excluding the first floor            |
 70 | |good     |BOOL        |                                                        |
 71 | 
 72 | 
 73 |  - post
 74 | 
 75 | Information of each post (including the first floor).
 76 | 
 77 | |Attribute  |Type       |Remark                               |
 78 | |-----------|-----------|-------------------------------------|
 79 | |id         |BIGINT(12) |                                     |
 80 | |floor      |INT(4)     |                                     |
 81 | |author     |VARCHAR(30)|                                     |
 82 | |content    |TEXT       |                                     |
 83 | |time       |DATETIME   |Post time                            |
 84 | |comment_num|INT(4)     |                                     |
 85 | |thread_id  |BIGINT(12) |Corresponding thread id (foreign key)|
 86 | 
 87 | 
 88 |  - comment
 89 |  
 90 | Information of each comment.
 91 | 
 92 | |Attribute|Type       |Remark                             |
 93 | |---------|-----------|-----------------------------------|
 94 | |id       |BIGINT(12) |                                   |
 95 | |author   |VARCHAR(30)|                                   |
 96 | |content  |TEXT       |                                   |
 97 | |time     |DATETIME   |Post time                          |
 98 | |post_id  |BIGINT(12) |Corresponding post id (foreign key)|
 99 | 
100 | It is possible that comment is crawled before its corresponding post. Therefore, to avoid foreign key error, constraints detection would be stopped during crawling.
101 | 
102 | ## Time
103 | Crawling time is mainly decided by the network. Belowing data are for your reference:
104 | 
105 | |Tieba Name|Thread Num|Reply Num|Comment Num|Time(s) |
106 | |----------|----------|---------|-----------|--------|
107 | |pandakill |3638      |41221    |50206      |222.2   |
108 | |lyingman  |11290     |122662   |126670     |718.9   |
109 | |仙剑五外传|67356     |1262705  |807435     |7188    |
110 | 
111 | ## References
112 | [Scrapy 1.0 文档][1]
113 | 
114 | [Scrapy 源代码][2]
115 | 
116 | [Beautiful Soup的用法][3]
117 | 
118 | [Ubuntu/Debian 安装lxml的正确方式][4]
119 | 
120 | [Twisted adbapi 源代码][5]
121 | 
122 | [mysql升级8.0后遇到的坑][6]
123 | 
124 | 
125 |   [1]: http://scrapy-chs.readthedocs.io/zh_CN/1.0/
126 |   [2]: https://coding.net/u/fmyl/p/scrapy
127 |   [3]: https://cuiqingcai.com/1319.html
128 |   [4]: http://www.cnblogs.com/numbbbbb/p/3434519.html
129 |   [5]: https://github.com/twisted/twisted/blob/twisted-16.5.0/src/twisted/enterprise/adbapi.py
130 |   [6]: https://www.shiqidu.com/d/358
131 | 


--------------------------------------------------------------------------------
/config.json:
--------------------------------------------------------------------------------
 1 | {
 2 |     "DEFAULT_TIEBA": "仙五前修改",
 3 |     "MYSQL_PASSWD": "123456",
 4 |     "MYSQL_DBNAME": {
 5 |         "仙五前修改": "Pal5Q_Diy",
 6 |         "风靡义磊": "fmyl"
 7 |     },
 8 |     "MYSQL_USER": "root",
 9 |     "MYSQL_HOST": "localhost",
10 |     "MYSQL_PORT": 3306
11 | }


--------------------------------------------------------------------------------
/config.py:
--------------------------------------------------------------------------------
 1 | # -*- coding: utf-8 -*-
 2 | 
 3 | import json
 4 | import os
 5 | import sys
 6 | import MySQLdb
 7 | import warnings
 8 | import time
 9 | import csv
10 | from io import open
11 | 
12 | class config:
13 |     config_path = 'config.json'
14 |     config = None
15 | 
16 |     def __init__(self):
17 |         with open(self.config_path, 'r', encoding='utf8') as f:
18 |             self.config = json.loads(f.read())
19 |             # loads后若有中文 为unicode
20 |     def save(self):
21 |         with open(self.config_path, 'wb') as f:
22 |             s = json.dumps(self.config, indent=4, ensure_ascii=False).encode('utf8')
23 |             f.write(s)
24 | 
25 | class log:
26 |     log_path = 'spider.log'
27 |     
28 |     def __init__(self, tbname, dbname, begin_page, good_only, see_lz):
29 |         if not os.path.isfile(self.log_path):
30 |             with open(self.log_path, 'wb') as f:
31 |                 row = ['start_time','end_time','elapsed_time','tieba_name','database_name', 'pages', 'etc']
32 |                 s = '\t'.join(row) + '\n'
33 |                 if not isinstance(s, bytes):
34 |                     s = s.encode('utf8')
35 |                 f.write(s)
36 |         self.tbname = tbname
37 |         self.dbname = dbname
38 |         self.begin_page = begin_page
39 |         etc = []
40 |         if good_only:
41 |             etc.append('good_only')
42 |         if see_lz:
43 |             etc.append('see_lz')
44 |         self.etc = '&'.join(etc)
45 |         if not self.etc:
46 |             self.etc = "None"
47 |         self.start_time = time.time()
48 |         
49 |     def log(self, end_page):
50 |         end_time = time.time()
51 |         elapsed_time = '%.4g' % (end_time - self.start_time)
52 |         start_time = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(self.start_time))
53 |         end_time = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(end_time))
54 |         tbname = self.tbname
55 | 
56 |         pages = '%d~%d'%(self.begin_page, end_page) if end_page >= self.begin_page else 'None'
57 |         with open(self.log_path, 'ab') as f:
58 |             row = [start_time, end_time, elapsed_time, tbname, self.dbname, pages, self.etc]
59 |             s = '\t'.join(row) + '\n'
60 |             if not isinstance(s, bytes):
61 |                 s = s.encode('utf8')
62 |             f.write(s)
63 |         
64 |         
65 | def init_database(host, user, passwd, port, dbname):
66 |     warnings.filterwarnings('ignore', message = ".*exists.*")  
67 |     warnings.filterwarnings('ignore', message = ".*looks like a.*") 
68 |     db = MySQLdb.connect(host=host, user=user, passwd=passwd, port=port)
69 |     tx = db.cursor()
70 |     tx.execute('set names utf8mb4')
71 |     tx.execute('create database if not exists `%s`default charset utf8mb4\
72 |     default collate utf8mb4_general_ci;' % MySQLdb.escape_string(dbname).decode("utf8"))
73 |     #要用斜引号不然报错
74 |     #万恶的MySQLdb会自动加上单引号 结果导致错误
75 |     db.select_db(dbname)
76 |     tx.execute("create table if not exists thread(\
77 |         id BIGINT(12), title VARCHAR(100), author VARCHAR(30), reply_num INT(4),\
78 |         good BOOL, PRIMARY KEY (id)) CHARSET=utf8mb4;")
79 |     tx.execute("create table if not exists post(\
80 |         id BIGINT(12), floor INT(4), author VARCHAR(30), content TEXT,\
81 |         time DATETIME, comment_num INT(4), thread_id BIGINT(12),PRIMARY KEY (id),\
82 |         FOREIGN KEY (thread_id) REFERENCES thread(id)) CHARSET=utf8mb4;")
83 |     tx.execute("create table if not exists comment(id BIGINT(12),\
84 |         author VARCHAR(30), content TEXT, time DATETIME, post_id BIGINT(12),\
85 |         PRIMARY KEY (id), FOREIGN KEY (post_id) REFERENCES post(id)) CHARSET=utf8mb4;")
86 |     db.commit()
87 |     db.close()


--------------------------------------------------------------------------------
/emotion.json:
--------------------------------------------------------------------------------
   1 | [
   2 |     {
   3 |         "regex": "static.tieba.baidu.com/tb/editor/images/client/image_emoticon([0-9]+)",
   4 |         "emotion_list": [
   5 |             {
   6 |                 "text": [
   7 |                     "呵呵",
   8 |                     "哈哈",
   9 |                     "吐舌",
  10 |                     "啊",
  11 |                     "酷",
  12 |                     "怒",
  13 |                     "开心",
  14 |                     "汗",
  15 |                     "泪",
  16 |                     "黑线",
  17 |                     "鄙视",
  18 |                     "不高兴",
  19 |                     "真棒",
  20 |                     "钱",
  21 |                     "疑问",
  22 |                     "阴险",
  23 |                     "吐",
  24 |                     "咦",
  25 |                     "委屈",
  26 |                     "花心",
  27 |                     "呼~",
  28 |                     "笑眼",
  29 |                     "冷",
  30 |                     "太开心",
  31 |                     "滑稽",
  32 |                     "勉强",
  33 |                     "狂汗",
  34 |                     "乖",
  35 |                     "睡觉",
  36 |                     "惊哭",
  37 |                     "升起",
  38 |                     "惊讶",
  39 |                     "喷",
  40 |                     "爱心",
  41 |                     "心碎",
  42 |                     "玫瑰",
  43 |                     "礼物",
  44 |                     "彩虹",
  45 |                     "星星月亮",
  46 |                     "太阳",
  47 |                     "钱币",
  48 |                     "灯泡",
  49 |                     "茶杯",
  50 |                     "蛋糕",
  51 |                     "音乐",
  52 |                     "haha",
  53 |                     "胜利",
  54 |                     "大拇指",
  55 |                     "弱",
  56 |                     "OK"
  57 |                 ],
  58 |                 "pattern": [
  59 |                     "__index__"
  60 |                 ]
  61 |             }
  62 |         ]
  63 |     },
  64 |     {
  65 |         "regex": "tb2.bdstatic.com/tb/editor/images/(.*?)/.*?([0-9]+)",
  66 |         "emotion_list": [
  67 |             {
  68 |                 "text": [
  69 |                     "呵呵",
  70 |                     "哈哈",
  71 |                     "吐舌",
  72 |                     "啊",
  73 |                     "酷",
  74 |                     "怒",
  75 |                     "开心",
  76 |                     "汗",
  77 |                     "泪",
  78 |                     "黑线",
  79 |                     "鄙视",
  80 |                     "不高兴",
  81 |                     "真棒",
  82 |                     "钱",
  83 |                     "疑问",
  84 |                     "阴险",
  85 |                     "吐",
  86 |                     "咦",
  87 |                     "委屈",
  88 |                     "花心",
  89 |                     "呼~",
  90 |                     "笑眼",
  91 |                     "冷",
  92 |                     "太开心",
  93 |                     "滑稽",
  94 |                     "勉强",
  95 |                     "狂汗",
  96 |                     "乖",
  97 |                     "睡觉",
  98 |                     "惊哭",
  99 |                     "升起",
 100 |                     "惊讶",
 101 |                     "喷",
 102 |                     "爱心",
 103 |                     "心碎",
 104 |                     "玫瑰",
 105 |                     "礼物",
 106 |                     "彩虹",
 107 |                     "星星月亮",
 108 |                     "太阳",
 109 |                     "钱币",
 110 |                     "灯泡",
 111 |                     "茶杯",
 112 |                     "蛋糕",
 113 |                     "音乐",
 114 |                     "haha",
 115 |                     "胜利",
 116 |                     "大拇指",
 117 |                     "弱",
 118 |                     "OK",
 119 |                     "伤心",
 120 |                     "加油",
 121 |                     "必胜",
 122 |                     "期待",
 123 |                     "牛逼",
 124 |                     "胜利",
 125 |                     "跟丫死磕",
 126 |                     "踢球",
 127 |                     "面壁",
 128 |                     "顶",
 129 |                     "巴西怒",
 130 |                     "伴舞",
 131 |                     "奔跑",
 132 |                     "点赞手",
 133 |                     "加油",
 134 |                     "哭泣",
 135 |                     "亮红牌",
 136 |                     "球迷",
 137 |                     "耶",
 138 |                     "转屁股"
 139 |                 ],
 140 |                 "pattern": [
 141 |                     "face",
 142 |                     "__index__"
 143 |                 ]
 144 |             },
 145 |             {
 146 |                 "text": [
 147 |                     "孤独寂寞冷",
 148 |                     "光棍吃根油条",
 149 |                     "可怜没人爱",
 150 |                     "去tmd光棍节",
 151 |                     "来捡肥皂啊",
 152 |                     "给跪了",
 153 |                     "惊呆了",
 154 |                     "拿去撸",
 155 |                     "哦女神",
 156 |                     "楼主好人"
 157 |                 ],
 158 |                 "pattern": [
 159 |                     "yxj",
 160 |                     "__index__"
 161 |                 ]
 162 |             },
 163 |             {
 164 |                 "text": [
 165 |                     "蹦蹦跳跳",
 166 |                     "晃悠",
 167 |                     "摇摇摆摆",
 168 |                     "大撒花",
 169 |                     "高高兴兴",
 170 |                     "弹吉他",
 171 |                     "魔法棒",
 172 |                     "扛大旗",
 173 |                     "点蜡烛",
 174 |                     "大秧歌",
 175 |                     "贴吧十周年",
 176 |                     "十周年"
 177 |                 ],
 178 |                 "pattern": [
 179 |                     "10th",
 180 |                     "__index__"
 181 |                 ]
 182 |             },
 183 |             {
 184 |                 "text": [
 185 |                     "一楼喂熊",
 186 |                     "俯卧撑",
 187 |                     "打酱油",
 188 |                     "囧",
 189 |                     "大笑",
 190 |                     "瀑布汗~",
 191 |                     "惊讶",
 192 |                     "臭美",
 193 |                     "傻笑",
 194 |                     "抛媚眼",
 195 |                     "发怒",
 196 |                     "我错了",
 197 |                     "money",
 198 |                     "气愤",
 199 |                     "挑逗",
 200 |                     "吻",
 201 |                     "怒",
 202 |                     "胜利",
 203 |                     "委屈",
 204 |                     "受伤",
 205 |                     "说啥呢？",
 206 |                     "闭嘴",
 207 |                     "不",
 208 |                     "逗你玩儿",
 209 |                     "飞吻",
 210 |                     "眩晕",
 211 |                     "魔法",
 212 |                     "我来了",
 213 |                     "睡了",
 214 |                     "我打",
 215 |                     "闭嘴",
 216 |                     "打",
 217 |                     "打晕了",
 218 |                     "刷牙",
 219 |                     "爆揍",
 220 |                     "炸弹",
 221 |                     "倒立",
 222 |                     "刮胡子",
 223 |                     "邪恶的笑",
 224 |                     "不要不要",
 225 |                     "爱恋中",
 226 |                     "放大仔细看",
 227 |                     "偷窥",
 228 |                     "超高兴",
 229 |                     "晕",
 230 |                     "松口气",
 231 |                     "我跑",
 232 |                     "享受",
 233 |                     "修养",
 234 |                     "哭",
 235 |                     "汗",
 236 |                     "啊~",
 237 |                     "热烈欢迎"
 238 |                 ],
 239 |                 "pattern": [
 240 |                     "ldw",
 241 |                     "__index__"
 242 |                 ]
 243 |             },
 244 |             {
 245 |                 "text": [
 246 |                     "lxs不想动",
 247 |                     "lxs吃惊",
 248 |                     "lxs吹口哨",
 249 |                     "lxs打气",
 250 |                     "lxs打坐",
 251 |                     "lxs等待",
 252 |                     "lxs愤怒",
 253 |                     "lxs幻象",
 254 |                     "lxs惊喜",
 255 |                     "lxs开心",
 256 |                     "lxs哭啼",
 257 |                     "lxs哭着跑",
 258 |                     "lxs没钱",
 259 |                     "lxs怕鬼",
 260 |                     "lxs抛媚眼",
 261 |                     "lxs七夕",
 262 |                     "lxs骑马舞",
 263 |                     "lxs升天",
 264 |                     "lxs石化",
 265 |                     "lxs数钱",
 266 |                     "lxs偷吃",
 267 |                     "lxs偷笑",
 268 |                     "lxs万圣节",
 269 |                     "lxs无聊",
 270 |                     "lxs无奈",
 271 |                     "lxs无语",
 272 |                     "lxs妩媚",
 273 |                     "lxs捉急",
 274 |                     "lxs转晕",
 275 |                     "lxs撞豆腐",
 276 |                     "lxs赘肉"
 277 |                 ],
 278 |                 "pattern": [
 279 |                     "lxs",
 280 |                     "__index__"
 281 |                 ]
 282 |             },
 283 |             {
 284 |                 "text": [
 285 |                     "疯啦",
 286 |                     "抱头跑",
 287 |                     "不想睡",
 288 |                     "不要呀",
 289 |                     "苍蝇",
 290 |                     "蹭痒",
 291 |                     "到碗里来",
 292 |                     "队长",
 293 |                     "发财啦",
 294 |                     "非洲舞",
 295 |                     "鼓掌",
 296 |                     "广场舞",
 297 |                     "滚筒",
 298 |                     "横行",
 299 |                     "挥手",
 300 |                     "火舞鸡",
 301 |                     "寂寞呀",
 302 |                     "加油",
 303 |                     "江南style",
 304 |                     "娇羞",
 305 |                     "惊慌失措",
 306 |                     "纠结",
 307 |                     "开饭啦",
 308 |                     "雷神",
 309 |                     "泪奔",
 310 |                     "聊QQ",
 311 |                     "麦霸",
 312 |                     "猫女",
 313 |                     "魔咒",
 314 |                     "尿急",
 315 |                     "扭屁股",
 316 |                     "瀑布汗",
 317 |                     "求饶",
 318 |                     "撒娇",
 319 |                     "少女",
 320 |                     "水冰鸡",
 321 |                     "天然呆",
 322 |                     "吐血",
 323 |                     "玩手机",
 324 |                     "威武",
 325 |                     "五道杠",
 326 |                     "舞动",
 327 |                     "洗洗",
 328 |                     "咸蛋鸡",
 329 |                     "向日葵",
 330 |                     "鸭子舞",
 331 |                     "游泳",
 332 |                     "招财鸡",
 333 |                     "照镜子",
 334 |                     "蜘蛛鸡"
 335 |                 ],
 336 |                 "pattern": [
 337 |                     "xyj",
 338 |                     "__index__"
 339 |                 ]
 340 |             },
 341 |             {
 342 |                 "text": [
 343 |                     "赖皮",
 344 |                     "感动",
 345 |                     "惊讶",
 346 |                     "怒气",
 347 |                     "哭泣",
 348 |                     "吃惊",
 349 |                     "嘲弄",
 350 |                     "飘过",
 351 |                     "转圈哭",
 352 |                     "神经病",
 353 |                     "揪耳朵",
 354 |                     "惊汗",
 355 |                     "隐身",
 356 |                     "不要",
 357 |                     "遁",
 358 |                     "不公平",
 359 |                     "爬来了",
 360 |                     "蛋花哭",
 361 |                     "温暖",
 362 |                     "点头",
 363 |                     "撒钱",
 364 |                     "献花",
 365 |                     "寒",
 366 |                     "傻笑",
 367 |                     "扭扭",
 368 |                     "疯",
 369 |                     "抓狂",
 370 |                     "抓",
 371 |                     "蜷",
 372 |                     "挠墙",
 373 |                     "狂笑",
 374 |                     "抱枕",
 375 |                     "吼叫",
 376 |                     "嚷",
 377 |                     "唠叨",
 378 |                     "捏脸",
 379 |                     "狂笑",
 380 |                     "郁闷",
 381 |                     "潜水",
 382 |                     "开心",
 383 |                     "冷笑话",
 384 |                     "顶",
 385 |                     "潜",
 386 |                     "画圈圈",
 387 |                     "玩电脑",
 388 |                     "吐",
 389 |                     "哭着跑",
 390 |                     "阿狸侠",
 391 |                     "冷死了",
 392 |                     "惆怅~",
 393 |                     "摸头",
 394 |                     "蹭",
 395 |                     "打滚",
 396 |                     "叩拜",
 397 |                     "摸",
 398 |                     "数钱",
 399 |                     "拖走",
 400 |                     "热",
 401 |                     "加1",
 402 |                     "压力",
 403 |                     "表逼我",
 404 |                     "人呢",
 405 |                     "摇晃",
 406 |                     "打地鼠",
 407 |                     "这个屌",
 408 |                     "恐慌",
 409 |                     "晕乎乎",
 410 |                     "浮云",
 411 |                     "给力",
 412 |                     "杯具了"
 413 |                 ],
 414 |                 "pattern": [
 415 |                     "ali",
 416 |                     "__index__"
 417 |                 ]
 418 |             },
 419 |             {
 420 |                 "text": [
 421 |                     "hold住",
 422 |                     "阿门",
 423 |                     "哎哟",
 424 |                     "唉～",
 425 |                     "不好意思",
 426 |                     "不要嘛！",
 427 |                     "唱歌",
 428 |                     "扯肚皮",
 429 |                     "扯脸",
 430 |                     "搓脸",
 431 |                     "搓手",
 432 |                     "打灯笼",
 433 |                     "打起精神",
 434 |                     "点头",
 435 |                     "点头2",
 436 |                     "动动脚",
 437 |                     "动动脚2",
 438 |                     "抖脚",
 439 |                     "抖脚2",
 440 |                     "嗯～",
 441 |                     "发呆",
 442 |                     "鼓掌",
 443 |                     "滚桌子",
 444 |                     "好闲阿～",
 445 |                     "喝茶发呆",
 446 |                     "哼！",
 447 |                     "晃",
 448 |                     "哭",
 449 |                     "困了",
 450 |                     "扩胸运动",
 451 |                     "来来来",
 452 |                     "来嘛～",
 453 |                     "冷静冷静",
 454 |                     "罗爸爸扭扭",
 455 |                     "罗妈妈扭扭",
 456 |                     "麋鹿罗罗布",
 457 |                     "扭pp",
 458 |                     "扭pp2",
 459 |                     "扭扭",
 460 |                     "欧也",
 461 |                     "趴着",
 462 |                     "切！",
 463 |                     "去去",
 464 |                     "撒花",
 465 |                     "上WC",
 466 |                     "圣诞老人",
 467 |                     "天使罗罗布",
 468 |                     "跳呀跳",
 469 |                     "跳呀跳2",
 470 |                     "玩个球",
 471 |                     "我来了",
 472 |                     "嘻嘻",
 473 |                     "向前进",
 474 |                     "亚克西",
 475 |                     "摇啊摇",
 476 |                     "一边呆着去！",
 477 |                     "一起扭",
 478 |                     "有礼了",
 479 |                     "这厢有礼了",
 480 |                     "走呀走"
 481 |                 ],
 482 |                 "pattern": [
 483 |                     "luoluobu",
 484 |                     "__index__"
 485 |                 ]
 486 |             },
 487 |             {
 488 |                 "text": [
 489 |                     "微笑",
 490 |                     "帅哥",
 491 |                     "美女",
 492 |                     "老大",
 493 |                     "哈哈哈",
 494 |                     "奸笑",
 495 |                     "傻乐",
 496 |                     "飞吻",
 497 |                     "害羞",
 498 |                     "花痴",
 499 |                     "憧憬",
 500 |                     "你牛",
 501 |                     "鼓掌",
 502 |                     "可爱",
 503 |                     "委屈",
 504 |                     "大哭",
 505 |                     "泪奔",
 506 |                     "寻死",
 507 |                     "惊讶",
 508 |                     "疑问",
 509 |                     "鄙视",
 510 |                     "大骂",
 511 |                     "狂怒",
 512 |                     "囧",
 513 |                     "晕",
 514 |                     "呕吐",
 515 |                     "财迷",
 516 |                     "睡了",
 517 |                     "思考",
 518 |                     "汗",
 519 |                     "无语",
 520 |                     "拜",
 521 |                     "鞠躬",
 522 |                     "累了",
 523 |                     "病了",
 524 |                     "墙角",
 525 |                     "超人",
 526 |                     "戳",
 527 |                     "跳舞",
 528 |                     "鬼脸",
 529 |                     "无聊",
 530 |                     "撞墙",
 531 |                     "顶",
 532 |                     "HI",
 533 |                     "饭",
 534 |                     "蛋糕",
 535 |                     "礼物",
 536 |                     "心",
 537 |                     "心碎",
 538 |                     "玫瑰",
 539 |                     "炸弹",
 540 |                     "集体舞",
 541 |                     "打人",
 542 |                     "拍拍",
 543 |                     "亲亲",
 544 |                     "握手",
 545 |                     "安慰",
 546 |                     "石头",
 547 |                     "剪子",
 548 |                     "布",
 549 |                     "恭喜啦",
 550 |                     "啵啵"
 551 |                 ],
 552 |                 "pattern": [
 553 |                     "qpx_n",
 554 |                     "__index__"
 555 |                 ]
 556 |             },
 557 |             {
 558 |                 "text": [
 559 |                     "+1",
 560 |                     "哀伤",
 561 |                     "饱",
 562 |                     "濒死",
 563 |                     "病",
 564 |                     "吃货",
 565 |                     "不满",
 566 |                     "呆",
 567 |                     "倒吊",
 568 |                     "点子",
 569 |                     "顶",
 570 |                     "鼓掌",
 571 |                     "害羞",
 572 |                     "汗",
 573 |                     "呵呵呵",
 574 |                     "加油",
 575 |                     "奸笑",
 576 |                     "奸诈",
 577 |                     "惊讶",
 578 |                     "囧",
 579 |                     "看",
 580 |                     "抠鼻子",
 581 |                     "哭",
 582 |                     "困",
 583 |                     "乐",
 584 |                     "溜走",
 585 |                     "路过",
 586 |                     "买你妹的房纸",
 587 |                     "敲锣",
 588 |                     "亲吻",
 589 |                     "去你妹的工作",
 590 |                     "扇风",
 591 |                     "失落",
 592 |                     "刷牙",
 593 |                     "逃命",
 594 |                     "跳舞",
 595 |                     "听音乐",
 596 |                     "偷笑",
 597 |                     "吐",
 598 |                     "无语",
 599 |                     "喜",
 600 |                     "笑",
 601 |                     "兴奋",
 602 |                     "鸭子跳",
 603 |                     "咬",
 604 |                     "咬",
 605 |                     "礼物",
 606 |                     "赞",
 607 |                     "震惊",
 608 |                     "追钱",
 609 |                     "做鬼脸"
 610 |                 ],
 611 |                 "pattern": [
 612 |                     "pczxh",
 613 |                     "__index__"
 614 |                 ]
 615 |             },
 616 |             {
 617 |                 "text": [
 618 |                     "Kiss",
 619 |                     "Love",
 620 |                     "Yeah",
 621 |                     "啊！",
 622 |                     "背扭",
 623 |                     "顶",
 624 |                     "抖胸",
 625 |                     "88",
 626 |                     "汗",
 627 |                     "瞌睡",
 628 |                     "鲁拉",
 629 |                     "拍砖",
 630 |                     "揉脸",
 631 |                     "生日快乐",
 632 |                     "摊手",
 633 |                     "睡觉",
 634 |                     "瘫坐",
 635 |                     "无聊",
 636 |                     "星星闪",
 637 |                     "旋转",
 638 |                     "也不行",
 639 |                     "郁闷",
 640 |                     "正Music",
 641 |                     "抓墙",
 642 |                     "撞墙至死",
 643 |                     "歪头",
 644 |                     "戳眼",
 645 |                     "飘过",
 646 |                     "互相拍砖",
 647 |                     "砍死你",
 648 |                     "扔桌子",
 649 |                     "少林寺",
 650 |                     "什么？",
 651 |                     "转头",
 652 |                     "我爱牛奶",
 653 |                     "我踢",
 654 |                     "摇晃",
 655 |                     "晕厥",
 656 |                     "在笼子里",
 657 |                     "震荡"
 658 |                 ],
 659 |                 "pattern": [
 660 |                     "tsj",
 661 |                     "__index__"
 662 |                 ]
 663 |             },
 664 |             {
 665 |                 "text": [
 666 |                     "心照不宣",
 667 |                     "蓝脸",
 668 |                     "困惑",
 669 |                     "够拽",
 670 |                     "瞪眼",
 671 |                     "藐视",
 672 |                     "思考",
 673 |                     "呵呵",
 674 |                     "想要妹子",
 675 |                     "幻想",
 676 |                     "无奈的笑",
 677 |                     "为什么",
 678 |                     "黑人歪嘴",
 679 |                     "歪嘴笑",
 680 |                     "苦逼",
 681 |                     "面瘫",
 682 |                     "咯咯咯",
 683 |                     "生气",
 684 |                     "高兴",
 685 |                     "喜极而泣",
 686 |                     "告诉我",
 687 |                     "有了",
 688 |                     "好吧",
 689 |                     "孤独一生",
 690 |                     "嘎嘎",
 691 |                     "暴躁",
 692 |                     "紫歪嘴",
 693 |                     "红歪嘴",
 694 |                     "这不科学",
 695 |                     "笑而不语",
 696 |                     "围观",
 697 |                     "牛仔歪嘴",
 698 |                     "方肘子",
 699 |                     "震惊",
 700 |                     "惊讶",
 701 |                     "颠覆三观",
 702 |                     "尼美藐视",
 703 |                     "尼美呵呵",
 704 |                     "尼美发呆",
 705 |                     "尼美有了",
 706 |                     "猫嘴笑",
 707 |                     "思春",
 708 |                     "尼美苦逼",
 709 |                     "笑尿",
 710 |                     "尼美咯咯",
 711 |                     "尼美歪嘴",
 712 |                     "尼美姚",
 713 |                     "暴走尼美",
 714 |                     "娇羞",
 715 |                     "尼美震惊",
 716 |                     "孤独尼美",
 717 |                     "尼美大哭",
 718 |                     "尼美愤怒",
 719 |                     "尼美嘎嘎",
 720 |                     "尼美愤懑",
 721 |                     "尼美不解"
 722 |                 ],
 723 |                 "pattern": [
 724 |                     "baodong",
 725 |                     "__index__"
 726 |                 ]
 727 |             },
 728 |             {
 729 |                 "text": [
 730 |                     "坏淫",
 731 |                     "抓狂",
 732 |                     "悲戚",
 733 |                     "痛哭",
 734 |                     "犯贱",
 735 |                     "奸笑",
 736 |                     "屌丝",
 737 |                     "高呼",
 738 |                     "感叹",
 739 |                     "感动",
 740 |                     "憋屈",
 741 |                     "舒爽",
 742 |                     "坏笑",
 743 |                     "屁颠",
 744 |                     "瞳术",
 745 |                     "杂耍",
 746 |                     "姚脸",
 747 |                     "孤独",
 748 |                     "暴走",
 749 |                     "潇洒",
 750 |                     "失望",
 751 |                     "同桌",
 752 |                     "变脸",
 753 |                     "得瑟",
 754 |                     "腹黑男",
 755 |                     "腹黑女",
 756 |                     "够拽",
 757 |                     "白领",
 758 |                     "喵尼玛",
 759 |                     "汪尼玛",
 760 |                     "嘿嘿尼玛",
 761 |                     "勾引",
 762 |                     "发泄",
 763 |                     "操操操"
 764 |                 ],
 765 |                 "pattern": [
 766 |                     "baodong_d",
 767 |                     "__index__"
 768 |                 ]
 769 |             },
 770 |             {
 771 |                 "text": [
 772 |                     "HI",
 773 |                     "KISS",
 774 |                     "不说",
 775 |                     "不要",
 776 |                     "扯花",
 777 |                     "大心",
 778 |                     "顶",
 779 |                     "大惊",
 780 |                     "飞吻",
 781 |                     "鬼脸",
 782 |                     "害羞",
 783 |                     "口水",
 784 |                     "狂哭",
 785 |                     "来",
 786 |                     "泪眼",
 787 |                     "流泪",
 788 |                     "生气",
 789 |                     "吐舌",
 790 |                     "喜欢",
 791 |                     "旋转",
 792 |                     "再见",
 793 |                     "抓狂",
 794 |                     "汗",
 795 |                     "鄙视",
 796 |                     "拜",
 797 |                     "吐血",
 798 |                     "嘘",
 799 |                     "打人",
 800 |                     "蹦跳",
 801 |                     "变脸",
 802 |                     "扯肉",
 803 |                     "吃To",
 804 |                     "吃花",
 805 |                     "吹泡泡糖",
 806 |                     "大变身",
 807 |                     "飞天舞",
 808 |                     "回眸",
 809 |                     "可怜",
 810 |                     "猛抽",
 811 |                     "泡泡",
 812 |                     "苹果",
 813 |                     "亲",
 814 |                     "",
 815 |                     "骚舞",
 816 |                     "烧香",
 817 |                     "睡",
 818 |                     "套娃娃",
 819 |                     "捅捅",
 820 |                     "舞倒",
 821 |                     "西红柿",
 822 |                     "爱慕",
 823 |                     "摇",
 824 |                     "摇摆",
 825 |                     "杂耍",
 826 |                     "招财",
 827 |                     "被殴",
 828 |                     "被球闷",
 829 |                     "大惊",
 830 |                     "理想",
 831 |                     "欧打",
 832 |                     "呕吐",
 833 |                     "碎",
 834 |                     "吐痰"
 835 |                 ],
 836 |                 "pattern": [
 837 |                     "bobo",
 838 |                     "__index__"
 839 |                 ]
 840 |             },
 841 |             {
 842 |                 "text": [
 843 |                     "焦糖舞",
 844 |                     "翻滚",
 845 |                     "拍屁股做鬼脸",
 846 |                     "不",
 847 |                     "河蟹掉啦",
 848 |                     "哦耶",
 849 |                     "我倒",
 850 |                     "投降",
 851 |                     "听音乐",
 852 |                     "被砸",
 853 |                     "吐舌头",
 854 |                     "太好啦",
 855 |                     "财源滚滚",
 856 |                     "人呢",
 857 |                     "挥手帕",
 858 |                     "偷笑",
 859 |                     "卷被",
 860 |                     "看楼上",
 861 |                     "我抽",
 862 |                     "有木有",
 863 |                     "哭",
 864 |                     "欢呼",
 865 |                     "傻兮兮",
 866 |                     "扇扇子",
 867 |                     "拍手",
 868 |                     "紧张",
 869 |                     "洗澡",
 870 |                     "耶",
 871 |                     "吃西瓜",
 872 |                     "纳尼",
 873 |                     "没有钱",
 874 |                     "拿玩偶",
 875 |                     "来看看",
 876 |                     "哇哇叫",
 877 |                     "招财猫",
 878 |                     "放屁",
 879 |                     "隆重介绍",
 880 |                     "求你了",
 881 |                     "撞玻璃",
 882 |                     "眺望",
 883 |                     "撒花",
 884 |                     "玩铃铛",
 885 |                     "思考者",
 886 |                     "抽风",
 887 |                     "吃苹果",
 888 |                     "光棍"
 889 |                 ],
 890 |                 "pattern": [
 891 |                     "shadow",
 892 |                     "__index__"
 893 |                 ]
 894 |             },
 895 |             {
 896 |                 "text": [
 897 |                     "中箭",
 898 |                     "LS去死",
 899 |                     "LS去死子弹",
 900 |                     "LS去死增强",
 901 |                     "LS去死炮弹",
 902 |                     "板凳",
 903 |                     "不爽",
 904 |                     "不要",
 905 |                     "超人",
 906 |                     "吃零食",
 907 |                     "抽烟",
 908 |                     "串",
 909 |                     "打狗",
 910 |                     "打酱油",
 911 |                     "打你",
 912 |                     "戴墨镜",
 913 |                     "蛋",
 914 |                     "荡漾",
 915 |                     "顶破楼上",
 916 |                     "发懵",
 917 |                     "放屁",
 918 |                     "负分滚粗",
 919 |                     "喝大了",
 920 |                     "呼啦圈儿",
 921 |                     "护",
 922 |                     "滑冰",
 923 |                     "画圈圈",
 924 |                     "火钳刘",
 925 |                     "贱贱的",
 926 |                     "惊吓",
 927 |                     "抠脚",
 928 |                     "lt哭",
 929 |                     "拉粑粑",
 930 |                     "撸",
 931 |                     "挠PP",
 932 |                     "闹",
 933 |                     "排球",
 934 |                     "前排",
 935 |                     "敲手鼓",
 936 |                     "沙发",
 937 |                     "剔牙",
 938 |                     "兔子蹦起来",
 939 |                     "挖鼻屎",
 940 |                     "无语",
 941 |                     "瞎了",
 942 |                     "秀肌肉",
 943 |                     "摇滚",
 944 |                     "意外惊喜",
 945 |                     "震惊",
 946 |                     "正步"
 947 |                 ],
 948 |                 "pattern": [
 949 |                     "lt",
 950 |                     "__index__"
 951 |                 ]
 952 |             },
 953 |             {
 954 |                 "text": [
 955 |                     "爱你",
 956 |                     "傲娇",
 957 |                     "打你",
 958 |                     "带回去压寨",
 959 |                     "给脸不要脸",
 960 |                     "滚（百度独家）",
 961 |                     "滚",
 962 |                     "好喜欢",
 963 |                     "泪奔",
 964 |                     "女屌丝",
 965 |                     "深井冰",
 966 |                     "算你狠",
 967 |                     "我美",
 968 |                     "虾米",
 969 |                     "不要",
 970 |                     "好崇拜",
 971 |                     "泪奔",
 972 |                     "你妹啊",
 973 |                     "你造么",
 974 |                     "求交往",
 975 |                     "骚年",
 976 |                     "土豪做朋友（百度独家）",
 977 |                     "围观",
 978 |                     "无药可医",
 979 |                     "再见",
 980 |                     "在么",
 981 |                     "call-me",
 982 |                     "no-zuo-no-die"
 983 |                 ],
 984 |                 "pattern": [
 985 |                     "bfmn",
 986 |                     "__index__"
 987 |                 ]
 988 |             },
 989 |             {
 990 |                 "text": [
 991 |                     "88",
 992 |                     "HI",
 993 |                     "人艰不拆",
 994 |                     "啥",
 995 |                     "大哭",
 996 |                     "失落",
 997 |                     "怒赞",
 998 |                     "惊呆了",
 999 |                     "李菊福",
1000 |                     "来信砍",
1001 |                     "欢迎入群",
1002 |                     "牛闪闪",
1003 |                     "生日快乐",
1004 |                     "石化",
1005 |                     "羞羞哒",
1006 |                     "肥皂必杀",
1007 |                     "谢谢你",
1008 |                     "跳舞",
1009 |                     "霹雳舞",
1010 |                     "鼓掌",
1011 |                     "偷内裤",
1012 |                     "哦耶",
1013 |                     "哭",
1014 |                     "喜欢",
1015 |                     "好棒",
1016 |                     "摔",
1017 |                     "求签名",
1018 |                     "菊花",
1019 |                     "屁",
1020 |                     "哭",
1021 |                     "顶",
1022 |                     "+1",
1023 |                     "biu",
1024 |                     "亲亲",
1025 |                     "鞭刑",
1026 |                     "胡子",
1027 |                     "僵尸",
1028 |                     "捉急",
1029 |                     "扭扭",
1030 |                     "插楼上",
1031 |                     "喜欢",
1032 |                     "跪求",
1033 |                     "羞羞",
1034 |                     "抠鼻子",
1035 |                     "休息一下",
1036 |                     "已死",
1037 |                     "忙着呢",
1038 |                     "汗",
1039 |                     "涨姿势",
1040 |                     "糊你熊脸",
1041 |                     "谜团",
1042 |                     "飞吻",
1043 |                     "love",
1044 |                     "哦耶",
1045 |                     "哭",
1046 |                     "崩溃",
1047 |                     "嘚瑟",
1048 |                     "断网",
1049 |                     "洗澡",
1050 |                     "购物"
1051 |                 ],
1052 |                 "pattern": [
1053 |                     "bearchildren",
1054 |                     "__index__"
1055 |                 ]
1056 |             },
1057 |             {
1058 |                 "text": [
1059 |                     "ILOVEU",
1060 |                     "上班中",
1061 |                     "不要嘛",
1062 |                     "中箭",
1063 |                     "仰卧起坐",
1064 |                     "体操",
1065 |                     "哈哈",
1066 |                     "哭",
1067 |                     "好基友",
1068 |                     "寂寞",
1069 |                     "嘚瑟",
1070 |                     "惊呆了",
1071 |                     "挑衅",
1072 |                     "搓澡",
1073 |                     "撞头",
1074 |                     "无聊",
1075 |                     "洗澡",
1076 |                     "洗衣服",
1077 |                     "点手指",
1078 |                     "牛逼",
1079 |                     "玩水果",
1080 |                     "生病",
1081 |                     "皮鞭伺候",
1082 |                     "睡觉",
1083 |                     "示爱",
1084 |                     "给跪了",
1085 |                     "胜利",
1086 |                     "讨厌",
1087 |                     "跟丫死磕",
1088 |                     "踢球",
1089 |                     "躺枪",
1090 |                     "送纸",
1091 |                     "逃跑",
1092 |                     "顶",
1093 |                     "马萨吉",
1094 |                     "不服",
1095 |                     "亲吻",
1096 |                     "伤心",
1097 |                     "切克闹",
1098 |                     "加油",
1099 |                     "勾引",
1100 |                     "千年杀",
1101 |                     "去tmd",
1102 |                     "发抖",
1103 |                     "吃",
1104 |                     "呼哈",
1105 |                     "哭",
1106 |                     "失落",
1107 |                     "孤独",
1108 |                     "庆祝",
1109 |                     "嘚瑟",
1110 |                     "必胜",
1111 |                     "惊",
1112 |                     "抽",
1113 |                     "抽丫的",
1114 |                     "挑衅",
1115 |                     "捡肥皂",
1116 |                     "撒花",
1117 |                     "撞墙",
1118 |                     "期待",
1119 |                     "板凳",
1120 |                     "犯贱",
1121 |                     "笑",
1122 |                     "路过",
1123 |                     "鄙视",
1124 |                     "面壁",
1125 |                     "顶",
1126 |                     "顶楼上",
1127 |                     "鲁",
1128 |                     "鼓掌"
1129 |                 ],
1130 |                 "pattern": [
1131 |                     "tiexing",
1132 |                     "__index__"
1133 |                 ]
1134 |             }
1135 |         ]
1136 |     }
1137 | ]


--------------------------------------------------------------------------------
/filter.py:
--------------------------------------------------------------------------------
 1 | # filter function.
 2 | # the function must accept 5 args(id, title, author, reply_num, good)
 3 | # and return True(if you would like to store this thread..) or False (or not)
 4 | 
 5 | 
 6 | # ex:
 7 | 
 8 | def thread_filter(id, title, author, reply_num, good):
 9 |     return reply_num > 30
10 |     
11 |     
12 | # > scrapy run hadoop hadoop -f thread_filter
13 | # only threads which have more than 30 replies will be stored.


--------------------------------------------------------------------------------
/scrapy.cfg:
--------------------------------------------------------------------------------
 1 | # Tieba spider
 2 | # By Aqua Dream
 3 | 
 4 | [settings]
 5 | default = tieba.settings
 6 | 
 7 | [deploy]
 8 | #url = http://localhost:6800/
 9 | project = tieba
10 | 


--------------------------------------------------------------------------------
/tieba/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Aqua-Dream/Tieba_Spider/25ffe2c6f5f2546db47d8cf46b64de117a45700d/tieba/__init__.py


--------------------------------------------------------------------------------
/tieba/commands/__init__.py:
--------------------------------------------------------------------------------
1 | # My commands.
2 | 


--------------------------------------------------------------------------------
/tieba/commands/run.py:
--------------------------------------------------------------------------------
 1 | from scrapy.exceptions import UsageError
 2 | from scrapy.commands import ScrapyCommand
 3 | import config
 4 | import filter
 5 | from scrapy.utils.conf import arglist_to_dict
 6 | 
 7 | class Command(ScrapyCommand):
 8 |     def syntax(self):
 9 |         return "<tieba_name> <database_name>"
10 | 
11 |     def short_desc(self):
12 |         return "Crawl tieba"
13 |         
14 |     def long_desc(self):
15 |         return "Crawl baidu tieba data to a MySQL database."
16 |         
17 |     def add_options(self, parser):
18 |         ScrapyCommand.add_options(self, parser)
19 |         parser.add_option("-a", dest="spargs", action="append", default=[], metavar="NAME=VALUE",
20 |                           help="set spider argument (may be repeated)")
21 |         parser.add_option("-p", "--pages", nargs = 2, type="int", dest="pages", default=[],
22 |                           help="set the range of pages you want to crawl")  
23 |         parser.add_option("-g", "--good", action="store_true", dest="good_only", default=False,
24 |                           help="only crawl good threads and their posts and comments")
25 |         parser.add_option("-f", "--filter", type="str", dest="filter", default="",
26 |                           help='set function name in "filter.py" to filter threads')
27 |         parser.add_option("-s", "--see_lz", action="store_true", dest="see_lz", default=False,
28 |                           help='enable "only see lz" mode')              
29 |                           
30 |     def set_pages(self, pages):
31 |         if len(pages) == 0:
32 |             begin_page = 1
33 |             end_page = 999999
34 |         else:
35 |             begin_page = pages[0]
36 |             end_page = pages[1]
37 |         if begin_page <= 0:
38 |             raise UsageError("The number of begin page must not be less than 1!")
39 |         if begin_page > end_page:
40 |             raise UsageError("The number of end page must not be less than that of begin page!")
41 |         self.settings.set('BEGIN_PAGE', begin_page, priority='cmdline')
42 |         self.settings.set('END_PAGE', end_page, priority='cmdline')
43 | 
44 |     def run(self, args, opts):
45 |         self.set_pages(opts.pages)
46 |         try:
47 |             opts.spargs = arglist_to_dict(opts.spargs)
48 |         except ValueError:
49 |             raise UsageError("Invalid -a value, use -a NAME=VALUE", print_help=False)
50 |         self.settings.set('GOOD_ONLY', opts.good_only)
51 |         self.settings.set('SEE_LZ', opts.see_lz)
52 |         if opts.filter:
53 |             try:
54 |                 opts.filter = eval('filter.' + opts.filter)
55 |             except:
56 |                 raise UsageError("Invalid filter function name!")
57 |         self.settings.set("FILTER", opts.filter)
58 |         cfg = config.config()
59 |         if len(args) >= 3:
60 |             raise UsageError("Too many arguments!")
61 |             
62 |         for i in range(len(args)):
63 |             if isinstance(args[i], bytes):
64 |                 args[i] = args[i].decode("utf8")
65 |         
66 |         if not 'MYSQL_PORT' in cfg.config.keys():
67 |             cfg.config['MYSQL_PORT'] = 3306
68 |         
69 |         self.settings.set('MYSQL_HOST', cfg.config['MYSQL_HOST'])
70 |         self.settings.set('MYSQL_USER', cfg.config['MYSQL_USER'])
71 |         self.settings.set('MYSQL_PASSWD', cfg.config['MYSQL_PASSWD'])
72 |         self.settings.set('MYSQL_PORT', cfg.config['MYSQL_PORT'])
73 | 
74 |         tbname = cfg.config['DEFAULT_TIEBA']
75 |         if len(args) >= 1:
76 |             tbname = args[0]
77 |             
78 |         dbname = None    
79 |         if tbname in cfg.config['MYSQL_DBNAME'].keys():
80 |             dbname = cfg.config['MYSQL_DBNAME'][tbname]
81 |         if len(args) >= 2:
82 |             dbname = args[1]
83 |             cfg.config['MYSQL_DBNAME'][tbname] = dbname
84 |         if not dbname:
85 |             raise UsageError("Please input database name!")
86 |             
87 |         self.settings.set('TIEBA_NAME', tbname, priority='cmdline')
88 |         self.settings.set('MYSQL_DBNAME', dbname, priority='cmdline')
89 |         
90 |         config.init_database(cfg.config['MYSQL_HOST'], cfg.config['MYSQL_USER'], cfg.config['MYSQL_PASSWD'],cfg.config['MYSQL_PORT'], dbname)
91 |         
92 |         log = config.log(tbname, dbname, self.settings['BEGIN_PAGE'], opts.good_only, opts.see_lz)
93 |         self.settings.set('SIMPLE_LOG', log)
94 |         self.crawler_process.crawl('tieba', **opts.spargs)
95 |         self.crawler_process.start()
96 |         
97 |         cfg.save()
98 | 


--------------------------------------------------------------------------------
/tieba/items.py:
--------------------------------------------------------------------------------
 1 | # -*- coding: utf-8 -*-
 2 | 
 3 | # Define here the models for your scraped items
 4 | #
 5 | # See documentation in:
 6 | # http://doc.scrapy.org/en/latest/topics/items.html
 7 | 
 8 | import scrapy
 9 | 
10 | 
11 | class ThreadItem(scrapy.Item):
12 |     name = 'thread'
13 |     id = scrapy.Field()
14 |     title = scrapy.Field()
15 |     author = scrapy.Field()
16 |     reply_num = scrapy.Field()
17 |     good = scrapy.Field()
18 |     
19 | class PostItem(scrapy.Item):
20 |     name = 'post'
21 |     id = scrapy.Field()
22 |     floor = scrapy.Field()
23 |     author = scrapy.Field()
24 |     content = scrapy.Field()
25 |     time = scrapy.Field()
26 |     comment_num = scrapy.Field()
27 |     thread_id = scrapy.Field()
28 | 
29 | class CommentItem(scrapy.Item):
30 |     name = 'comment'
31 |     id = scrapy.Field()
32 |     author = scrapy.Field()
33 |     content = scrapy.Field()
34 |     time = scrapy.Field()
35 |     post_id = scrapy.Field()
36 | 


--------------------------------------------------------------------------------
/tieba/middlewares.py:
--------------------------------------------------------------------------------
 1 | # -*- coding: utf-8 -*-
 2 | 
 3 | # Define here the models for your spider middleware
 4 | #
 5 | # See documentation in:
 6 | # http://doc.scrapy.org/en/latest/topics/spider-middleware.html
 7 | 
 8 | from scrapy import signals
 9 | 
10 | 
11 | class TiebaSpiderMiddleware(object):
12 |     # Not all methods need to be defined. If a method is not defined,
13 |     # scrapy acts as if the spider middleware does not modify the
14 |     # passed objects.
15 | 
16 |     @classmethod
17 |     def from_crawler(cls, crawler):
18 |         # This method is used by Scrapy to create your spiders.
19 |         s = cls()
20 |         crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
21 |         return s
22 | 
23 |     def process_spider_input(response, spider):
24 |         # Called for each response that goes through the spider
25 |         # middleware and into the spider.
26 | 
27 |         # Should return None or raise an exception.
28 |         return None
29 | 
30 |     def process_spider_output(response, result, spider):
31 |         # Called with the results returned from the Spider, after
32 |         # it has processed the response.
33 | 
34 |         # Must return an iterable of Request, dict or Item objects.
35 |         for i in result:
36 |             yield i
37 | 
38 |     def process_spider_exception(response, exception, spider):
39 |         # Called when a spider or process_spider_input() method
40 |         # (from other spider middleware) raises an exception.
41 | 
42 |         # Should return either None or an iterable of Response, dict
43 |         # or Item objects.
44 |         pass
45 | 
46 |     def process_start_requests(start_requests, spider):
47 |         # Called with the start requests of the spider, and works
48 |         # similarly to the process_spider_output() method, except
49 |         # that it doesn’t have a response associated.
50 | 
51 |         # Must return only requests (not items).
52 |         for r in start_requests:
53 |             yield r
54 | 
55 |     def spider_opened(self, spider):
56 |         spider.logger.info('Spider opened: %s' % spider.name)
57 | 


--------------------------------------------------------------------------------
/tieba/pipelines.py:
--------------------------------------------------------------------------------
 1 | # -*- coding: utf-8 -*-
 2 | 
 3 | # Define your item pipelines here
 4 | #
 5 | # Don't forget to add your pipeline to the ITEM_PIPELINES setting
 6 | # See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
 7 | 
 8 | from twisted.enterprise import adbapi
 9 | import MySQLdb
10 | import MySQLdb.cursors
11 | from six.moves.urllib.parse import quote
12 | from tieba.items import ThreadItem, PostItem, CommentItem
13 | 
14 | class TiebaPipeline(object):
15 |     @classmethod
16 |     def from_settings(cls, settings):
17 |         return cls(settings)
18 | 
19 |     def __init__(self, settings):
20 |         dbname = settings['MYSQL_DBNAME']
21 |         tbname = settings['TIEBA_NAME']
22 |         if not dbname.strip():
23 |             raise ValueError("No database name!")
24 |         if not tbname.strip():
25 |             raise ValueError("No tieba name!")    
26 | 
27 |         self.settings = settings
28 |         
29 |         self.dbpool = adbapi.ConnectionPool('MySQLdb',
30 |             host=settings['MYSQL_HOST'],
31 |             db=settings['MYSQL_DBNAME'],
32 |             user=settings['MYSQL_USER'],
33 |             passwd=settings['MYSQL_PASSWD'],
34 |             port=settings['MYSQL_PORT'],
35 |             charset='utf8mb4',
36 |             cursorclass = MySQLdb.cursors.DictCursor,
37 |             init_command = 'set foreign_key_checks=0' #异步容易冲突
38 |         )
39 |         
40 |     def open_spider(self, spider):
41 |         spider.cur_page = begin_page = self.settings['BEGIN_PAGE']
42 |         spider.end_page = self.settings['END_PAGE']
43 |         spider.filter = self.settings['FILTER']
44 |         spider.see_lz = self.settings['SEE_LZ']
45 |         tbname = self.settings['TIEBA_NAME']
46 |         if not isinstance(tbname, bytes):
47 |             tbname = tbname.encode('utf8')
48 |         start_url = "http://tieba.baidu.com/f?kw=%s&pn=%d" \
49 |                 %(quote(tbname), 50 * (begin_page - 1))
50 |         if self.settings['GOOD_ONLY']:
51 |             start_url += '&tab=good'
52 |         
53 |         spider.start_urls = [start_url]
54 |         
55 |     def close_spider(self, spider):
56 |         self.settings['SIMPLE_LOG'].log(spider.cur_page - 1)
57 |     
58 |     def process_item(self, item, spider):
59 |         _conditional_insert = {
60 |             'thread': self.insert_thread, 
61 |             'post': self.insert_post, 
62 |             'comment': self.insert_comment
63 |         }
64 |         query = self.dbpool.runInteraction(_conditional_insert[item.name], item)
65 |         query.addErrback(self._handle_error, item, spider)
66 |         return item
67 |         
68 |     def insert_thread(self, tx, item):
69 |         sql = "replace into thread values(%s, %s, %s, %s, %s)"
70 |         params = (item["id"], item["title"], item['author'], item['reply_num'], item['good'])
71 |         tx.execute(sql, params)     
72 |         
73 |     def insert_post(self, tx, item):
74 |         sql = "replace into post values(%s, %s, %s, %s, %s, %s, %s)"
75 |         params = (item["id"], item["floor"], item['author'], item['content'], 
76 |             item['time'], item['comment_num'], item['thread_id'])
77 |         tx.execute(sql, params)
78 |         
79 |     def insert_comment(self, tx, item):
80 |         #tx.execute('set names utf8mb4') # seems to be redundant
81 |         sql = "replace into comment values(%s, %s, %s, %s, %s)"
82 |         params = (item["id"], item['author'], item['content'], item['time'], item['post_id'])
83 |         tx.execute(sql, params)
84 |         
85 |     #错误处理方法
86 |     def _handle_error(self, fail, item, spider):
87 |         spider.logger.error('Insert to database error: %s \
88 |         when dealing with item: %s' %(fail, item))
89 | 


--------------------------------------------------------------------------------
/tieba/settings.py:
--------------------------------------------------------------------------------
 1 | # -*- coding: utf-8 -*-
 2 | 
 3 | BOT_NAME = 'tieba'
 4 | 
 5 | SPIDER_MODULES = ['tieba.spiders']
 6 | NEWSPIDER_MODULE = 'tieba.spiders'
 7 | 
 8 | # Obey robots.txt rules
 9 | ROBOTSTXT_OBEY = True
10 | 
11 | # Configure item pipelines
12 | # See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
13 | 
14 | ITEM_PIPELINES = {
15 |     'tieba.pipelines.TiebaPipeline': 300,
16 | }
17 | 
18 | LOG_LEVEL = 'WARNING'
19 | #LOG_LEVEL = 'INFO'
20 | 
21 | COMMANDS_MODULE = 'tieba.commands'
22 | 
23 | COOKIES_ENABLED = False
24 | 


--------------------------------------------------------------------------------
/tieba/spiders/__init__.py:
--------------------------------------------------------------------------------
1 | # This package will contain the spiders of your Scrapy project
2 | #
3 | # Please refer to the documentation for information on how to create and manage
4 | # your spiders.
5 | 


--------------------------------------------------------------------------------
/tieba/spiders/emotion.py:
--------------------------------------------------------------------------------
 1 | import re
 2 | import json
 3 | from io import open
 4 | emotion_data = json.loads(open('emotion.json','r',encoding='utf8').read())
 5 | 
 6 | def get_text(url):
 7 |     for data in emotion_data:
 8 |         match = re.findall(data['regex'], url)
 9 |         if not match:
10 |             continue
11 |         if isinstance(match[0], tuple):
12 |             match = match[0]
13 |         for emotion in data['emotion_list']:
14 |             flag = True
15 |             for i in range(len(match)):
16 |                 if emotion['pattern'][i] == '__index__':
17 |                     index = int(match[i]) - 1
18 |                     if index >= len(emotion['text']):
19 |                         return ' ' + url + ' '
20 |                 elif emotion['pattern'][i] != match[i]:
21 |                     flag = False
22 |                     break
23 |             if flag:
24 |                 return '[' + emotion['text'][index] + ']'
25 |     return ' ' + url + ' '
26 | 


--------------------------------------------------------------------------------
/tieba/spiders/helper.py:
--------------------------------------------------------------------------------
 1 | # -*- coding: utf-8 -*-
 2 | 
 3 | import re
 4 | from six.moves.urllib import request
 5 | from bs4 import BeautifulSoup
 6 | from . import emotion
 7 | 
 8 | 
 9 | def is_ad(s): #判断楼层是否为广告
10 |     ad = s.xpath(u".//span[contains(text(), '广告')]")
11 |     # 广告楼层中间有个span含有广告俩字
12 |     return ad
13 | 
14 | def parse_content(content):
15 |     if not content or not content.strip():
16 |         return None
17 |     content = content.replace('\r', '\n') #古老的帖子会出现奇怪的\r
18 |     s = BeautifulSoup(content, 'lxml')
19 |     if s.html:
20 |         s=s.html
21 |     if s.body:
22 |         s=s.body
23 |     if s.div:
24 |         s = s.div  #post 外层有个div
25 |     if s.p:
26 |         s = s.p
27 |     l = list(s.children)
28 |     for i in range(len(l)):
29 |         parse_func = (is_str, is_br, is_img, is_video, other_case)
30 |         for func in parse_func:
31 |             try:
32 |                 ret = func(l[i])
33 |             except:
34 |                 continue
35 |             if ret is not False: 
36 |                 l[i] = ret
37 |                 break
38 |     return strip_blank(''.join(l))
39 | 
40 | def strip_blank(s): #按个人喜好去掉空白字符
41 |     s = re.sub(r'\n[ \t]+\n', '\n', s)
42 |     s = re.sub(r'  +', ' ', s) #去掉多余的空格
43 |     s = re.sub(r'\n\n\n+', '\n\n', s) #去掉过多的连续换行
44 |     return s.strip()
45 | 
46 | def is_str(s): 
47 |     if s.name: 
48 |         return False
49 |     #NavigableString类型需要手动转换下
50 |     return str(s)
51 | 
52 | def is_br(s):
53 |     if s.name == 'br':
54 |         return '\n'
55 |     return False
56 | 
57 | def is_img(s):
58 |     # 处理了部分表情
59 |     if s.name == 'img':
60 |         src = str(s.get('src'))
61 |         return emotion.get_text(src)
62 |     return False
63 | 
64 | def is_video(s):
65 |     t = str(s.get('class'))
66 |     if 'video' in t:
67 |         url = s.find('a').get('href')
68 |         return ' ' + getJumpUrl(url) + ' '
69 |     return False
70 | 
71 | #bs带的get_text功能，很好很强大
72 | #粗体红字之类的都一句话搞定了
73 | def other_case(s): 
74 |     return s.get_text()
75 | 
76 | 
77 | 
78 | # 发送请求到 jump.bdimg.com/.. 来获取真实链接
79 | # 阻止302跳转后可以大大节省时间 
80 | 
81 | class RedirctHandler(request.HTTPRedirectHandler):      
82 |     def http_error_302(self, req, fp, code, msg, headers):  
83 |         raise Exception(headers.get('location'))
84 | 
85 | def getJumpUrl(url):    
86 |     req = request.Request(url)  
87 |     opener = request.build_opener(RedirctHandler())      
88 |     try:    
89 |         opener.open(url)
90 |     except Exception as e:
91 |         return str(e)
92 | 


--------------------------------------------------------------------------------
/tieba/spiders/tieba_spider.py:
--------------------------------------------------------------------------------
  1 | # -*- coding: utf-8 -*-
  2 | 
  3 | import scrapy
  4 | import json
  5 | from tieba.items import ThreadItem, PostItem, CommentItem
  6 | from . import helper
  7 | import time
  8 | from math import ceil
  9 | 
 10 | class TiebaSpider(scrapy.Spider):
 11 |     name = "tieba"
 12 |     cur_page = 1    #modified by pipelines (open_spider)
 13 |     end_page = 9999
 14 |     filter = None
 15 |     see_lz = False
 16 |     my_headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'}
 17 |     
 18 |     def parse(self, response): #forum parser
 19 |         print("Crawling page %d..." % self.cur_page)
 20 |         for sel in response.xpath('//li[contains(@class, "j_thread_list")]'):
 21 |             data = json.loads(sel.xpath('@data-field').extract_first())
 22 |             if data['id'] == 1: # 去掉"本吧吧主火热招募"
 23 |                 continue
 24 |             item = ThreadItem()
 25 |             item['id'] = data['id']
 26 |             item['author'] = data['author_name']
 27 |             item['reply_num'] = data['reply_num']
 28 |             item['good'] = data['is_good']
 29 |             if not item['good']:
 30 |                 item['good'] = False
 31 |             item['title'] = sel.xpath('.//div[contains(@class, "threadlist_title")]/a/@title').extract_first()
 32 |             if self.filter and not self.filter(item["id"], item["title"], item['author'], item['reply_num'], item['good']):
 33 |                 continue
 34 |             #filter过滤掉的帖子及其回复均不存入数据库
 35 |                 
 36 |             yield item
 37 |             meta = {'thread_id': data['id'], 'page': 1}
 38 |             url = 'http://tieba.baidu.com/p/%d' % data['id']
 39 |             if self.see_lz:
 40 |                 url += '?see_lz=1'
 41 |             yield scrapy.Request(url, callback = self.parse_post,  meta = meta, 
 42 |                 headers=self.my_headers)
 43 |         next_page = response.xpath('//a[@class="next pagination-item "]/@href')
 44 |         self.cur_page += 1
 45 |         if next_page:
 46 |             if self.cur_page <= self.end_page:
 47 |                 yield scrapy.Request('http:'+next_page.extract_first())
 48 |             
 49 |     def parse_post(self, response): 
 50 |         meta = response.meta
 51 |         has_comment = False
 52 |         for floor in response.xpath("//div[contains(@class, 'l_post')]"):
 53 |             if not helper.is_ad(floor):
 54 |                 data = json.loads(floor.xpath("@data-field").extract_first())
 55 |                 item = PostItem()
 56 |                 item['id'] = data['content']['post_id']
 57 |                 item['author'] = data['author']['user_name']
 58 |                 item['comment_num'] = data['content']['comment_num']
 59 |                 if item['comment_num'] > 0:
 60 |                     has_comment = True
 61 |                 content = floor.xpath(".//div[contains(@class,'j_d_post_content')]").extract_first()
 62 |                 #以前的帖子, data-field里面没有content
 63 |                 item['content'] = helper.parse_content(content)
 64 |                 #以前的帖子, data-field里面没有thread_id
 65 |                 item['thread_id'] = meta['thread_id']
 66 |                 item['floor'] = data['content']['post_no']
 67 |                 #只有以前的帖子, data-field里面才有date
 68 |                 if 'date' in data['content'].keys():
 69 |                     item['time'] = data['content']['date']
 70 |                     #只有以前的帖子, data-field里面才有date
 71 |                 else:
 72 |                     item['time'] = floor.xpath(".//span[@class='tail-info']")\
 73 |                     .re_first(r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}')
 74 |                 yield item
 75 | 
 76 |         if has_comment:
 77 |             url = "http://tieba.baidu.com/p/totalComment?tid=%d&fid=1&pn=%d" % (meta['thread_id'], meta['page'])
 78 |             if self.see_lz:
 79 |                 url += '&see_lz=1'
 80 |             yield scrapy.Request(url, callback = self.parse_totalComment, meta = meta, 
 81 |                 headers=self.my_headers)
 82 |         next_page = response.xpath(u".//ul[@class='l_posts_num']//a[text()='下一页']/@href")
 83 |         if next_page:
 84 |             meta['page'] += 1
 85 |             url = response.urljoin(next_page.extract_first())
 86 |             yield scrapy.Request(url, callback = self.parse_post, meta = meta, 
 87 |                 headers=self.my_headers)
 88 | 
 89 |     def parse_totalComment(self, response):
 90 |         meta = response.meta.copy()
 91 |         comment_list = json.loads(response.text)['data']['comment_list']
 92 |         if not comment_list:
 93 |             return
 94 |         for value in comment_list.values():
 95 |             comments = value['comment_info']
 96 |             for comment in comments:
 97 |                 item = CommentItem()
 98 |                 item['id'] = comment['comment_id']
 99 |                 item['author'] = comment['username']
100 |                 item['post_id'] = meta['post_id'] = comment['post_id']
101 |                 item['content'] = helper.parse_content(comment['content'])
102 |                 item['time'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(comment['now_time']))
103 |                 yield item
104 |             comment_pages = ceil(value['comment_num'] / 10.0)
105 |             for i in range(1, comment_pages): # other pages
106 |                 url = "https://tieba.baidu.com/p/comment?tid=%d&pid=%d&pn=%d" % (meta['thread_id'], item['post_id'], i+1) 
107 |                 yield scrapy.Request(url, callback = self.parse_comment, meta = meta)
108 |          
109 |     def parse_comment(self, response):
110 |         comment_list = response.xpath('body/li')
111 |         for i in range(len(comment_list)-1):
112 |             comment = comment_list[i]
113 |             data = json.loads(comment.attrib['data-field'])
114 |             item = CommentItem()
115 |             item['id'] = data['spid']
116 |             item['author'] = data['user_name']
117 |             item['post_id'] = response.meta['post_id']
118 |             item['content'] = helper.parse_content(comment.css('.lzl_content_main').get())
119 |             item['time'] = comment.css('.lzl_time').xpath('./text()').get()
120 |             yield item
121 | 


--------------------------------------------------------------------------------