├── AppCommentsSpider ├── AppCommentsSpider │ ├── items.py │ ├── launch.py │ ├── log │ │ └── spider_log.log │ ├── middlewares.py │ ├── pipelines.py │ ├── scrapy_redis │ │ ├── __init__.py │ │ ├── connection.py │ │ ├── defaults.py │ │ ├── dupefilter.py │ │ ├── picklecompat.py │ │ ├── pipelines.py │ │ ├── queue.py │ │ ├── scheduler.py │ │ ├── spiders.py │ │ └── utils.py │ ├── settings.py │ ├── spiders │ │ ├── __init__.py │ │ ├── appstore.py │ │ ├── taptap.py │ │ ├── tieba.py │ │ └── weibo.py │ └── utils │ │ ├── __init__.py │ │ └── bloomfilter.py ├── requirements └── scrapy.cfg └── README.md /AppCommentsSpider/AppCommentsSpider/items.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/items.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/launch.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/launch.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/log/spider_log.log: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/middlewares.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/middlewares.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/pipelines.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/pipelines.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/scrapy_redis/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/scrapy_redis/__init__.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/scrapy_redis/connection.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/scrapy_redis/connection.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/scrapy_redis/defaults.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/scrapy_redis/defaults.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/scrapy_redis/dupefilter.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/scrapy_redis/dupefilter.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/scrapy_redis/picklecompat.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/scrapy_redis/picklecompat.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/scrapy_redis/pipelines.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/scrapy_redis/pipelines.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/scrapy_redis/queue.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/scrapy_redis/queue.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/scrapy_redis/scheduler.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/scrapy_redis/scheduler.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/scrapy_redis/spiders.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/scrapy_redis/spiders.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/scrapy_redis/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/scrapy_redis/utils.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/settings.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/settings.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/spiders/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/spiders/__init__.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/spiders/appstore.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/spiders/appstore.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/spiders/taptap.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/spiders/taptap.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/spiders/tieba.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/spiders/tieba.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/spiders/weibo.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/spiders/weibo.py -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/utils/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /AppCommentsSpider/AppCommentsSpider/utils/bloomfilter.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/AppCommentsSpider/utils/bloomfilter.py -------------------------------------------------------------------------------- /AppCommentsSpider/requirements: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/requirements -------------------------------------------------------------------------------- /AppCommentsSpider/scrapy.cfg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/AppCommentsSpider/scrapy.cfg -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stanleylsx/app_comments_spider/HEAD/README.md --------------------------------------------------------------------------------