├── .gitignore ├── LICENSE ├── README.md ├── WeiboCrawler ├── __init__.py ├── items.py ├── middlewares.py ├── pipelines.py ├── settings.py └── spiders │ ├── __init__.py │ ├── comment.py │ ├── mblog.py │ ├── repost.py │ ├── user.py │ └── utils.py ├── requirements.txt └── scrapy.cfg /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/.gitignore -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/README.md -------------------------------------------------------------------------------- /WeiboCrawler/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /WeiboCrawler/items.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/WeiboCrawler/items.py -------------------------------------------------------------------------------- /WeiboCrawler/middlewares.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/WeiboCrawler/middlewares.py -------------------------------------------------------------------------------- /WeiboCrawler/pipelines.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/WeiboCrawler/pipelines.py -------------------------------------------------------------------------------- /WeiboCrawler/settings.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/WeiboCrawler/settings.py -------------------------------------------------------------------------------- /WeiboCrawler/spiders/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/WeiboCrawler/spiders/__init__.py -------------------------------------------------------------------------------- /WeiboCrawler/spiders/comment.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/WeiboCrawler/spiders/comment.py -------------------------------------------------------------------------------- /WeiboCrawler/spiders/mblog.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/WeiboCrawler/spiders/mblog.py -------------------------------------------------------------------------------- /WeiboCrawler/spiders/repost.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/WeiboCrawler/spiders/repost.py -------------------------------------------------------------------------------- /WeiboCrawler/spiders/user.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/WeiboCrawler/spiders/user.py -------------------------------------------------------------------------------- /WeiboCrawler/spiders/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/WeiboCrawler/spiders/utils.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | pymongo==3.10.1 2 | Scrapy==1.5.1 3 | Pillow==8.2.0 -------------------------------------------------------------------------------- /scrapy.cfg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/XWang20/WeiboCrawler/HEAD/scrapy.cfg --------------------------------------------------------------------------------