├── .gitignore ├── Hupu ├── HupuSpider.py ├── MongoAPI.py └── hupu_tools.py ├── LianJia ├── tongbu.py └── yibu.py ├── README.md ├── Wechat ├── WechatSpider │ ├── __init__.py │ ├── config.py │ ├── db.py │ ├── mysqlapi.py │ ├── request.py │ └── spider.py ├── __init__.py ├── create_table.py └── run.py ├── Wikipedia ├── Breadth_First(257s).txt ├── Breadth_First.py ├── Depth_First(2539s).txt └── Depth_First.py ├── baidumapapi ├── cities.txt ├── cities1.txt ├── city_crawler.py ├── creat_table_park.py ├── create_table_city.py ├── fake_headers.py ├── filter.py ├── get_ip.py ├── get_province.py └── park_crawler.py ├── bilibili_video ├── crawler.py └── imgs │ └── b_video.jpg ├── fangtianxia_scrapy_redis ├── create_table.py ├── fangtianxia_scrapy │ ├── __init__.py │ ├── items.py │ ├── middlewares.py │ ├── pipelines.py │ ├── settings.py │ └── spiders │ │ ├── __init__.py │ │ └── fang.py ├── imgs │ ├── fang.jpg │ ├── linux_error.jpg │ └── settings.jpg ├── requirements.txt └── scrapy.cfg ├── html2pdf ├── crawler.py └── liaoxuefeng_Python3_tutorial.pdf ├── lianjia_scrapy ├── create_table.py ├── get_url_list.py ├── imgs │ ├── item_pipeline.jpg │ ├── mysql.jpg │ ├── settings.jpg │ ├── 公寓.jpg │ ├── 普通.png │ └── 高德api.jpg └── lianjiaSpider │ ├── lianjiaSpider │ ├── __init__.py │ ├── items.py │ ├── middlewares.py │ ├── pipelines.py │ ├── settings.py │ └── spiders │ │ ├── __init__.py │ │ └── lianjiazufang.py │ └── scrapy.cfg ├── requirements.txt └── 微信公众号文章获取 ├── README.md └── image ├── fiddler.jpg ├── fiddler01.jpg └── html.jpg /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/.gitignore -------------------------------------------------------------------------------- /Hupu/HupuSpider.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Hupu/HupuSpider.py -------------------------------------------------------------------------------- /Hupu/MongoAPI.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Hupu/MongoAPI.py -------------------------------------------------------------------------------- /Hupu/hupu_tools.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Hupu/hupu_tools.py -------------------------------------------------------------------------------- /LianJia/tongbu.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/LianJia/tongbu.py -------------------------------------------------------------------------------- /LianJia/yibu.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/LianJia/yibu.py -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/README.md -------------------------------------------------------------------------------- /Wechat/WechatSpider/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /Wechat/WechatSpider/config.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Wechat/WechatSpider/config.py -------------------------------------------------------------------------------- /Wechat/WechatSpider/db.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Wechat/WechatSpider/db.py -------------------------------------------------------------------------------- /Wechat/WechatSpider/mysqlapi.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Wechat/WechatSpider/mysqlapi.py -------------------------------------------------------------------------------- /Wechat/WechatSpider/request.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Wechat/WechatSpider/request.py -------------------------------------------------------------------------------- /Wechat/WechatSpider/spider.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Wechat/WechatSpider/spider.py -------------------------------------------------------------------------------- /Wechat/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /Wechat/create_table.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Wechat/create_table.py -------------------------------------------------------------------------------- /Wechat/run.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Wechat/run.py -------------------------------------------------------------------------------- /Wikipedia/Breadth_First(257s).txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Wikipedia/Breadth_First(257s).txt -------------------------------------------------------------------------------- /Wikipedia/Breadth_First.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Wikipedia/Breadth_First.py -------------------------------------------------------------------------------- /Wikipedia/Depth_First(2539s).txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Wikipedia/Depth_First(2539s).txt -------------------------------------------------------------------------------- /Wikipedia/Depth_First.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/Wikipedia/Depth_First.py -------------------------------------------------------------------------------- /baidumapapi/cities.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/baidumapapi/cities.txt -------------------------------------------------------------------------------- /baidumapapi/cities1.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/baidumapapi/cities1.txt -------------------------------------------------------------------------------- /baidumapapi/city_crawler.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/baidumapapi/city_crawler.py -------------------------------------------------------------------------------- /baidumapapi/creat_table_park.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/baidumapapi/creat_table_park.py -------------------------------------------------------------------------------- /baidumapapi/create_table_city.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/baidumapapi/create_table_city.py -------------------------------------------------------------------------------- /baidumapapi/fake_headers.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/baidumapapi/fake_headers.py -------------------------------------------------------------------------------- /baidumapapi/filter.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/baidumapapi/filter.py -------------------------------------------------------------------------------- /baidumapapi/get_ip.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/baidumapapi/get_ip.py -------------------------------------------------------------------------------- /baidumapapi/get_province.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/baidumapapi/get_province.py -------------------------------------------------------------------------------- /baidumapapi/park_crawler.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/baidumapapi/park_crawler.py -------------------------------------------------------------------------------- /bilibili_video/crawler.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/bilibili_video/crawler.py -------------------------------------------------------------------------------- /bilibili_video/imgs/b_video.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/bilibili_video/imgs/b_video.jpg -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/create_table.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/fangtianxia_scrapy_redis/create_table.py -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/fangtianxia_scrapy/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/fangtianxia_scrapy/items.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/fangtianxia_scrapy_redis/fangtianxia_scrapy/items.py -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/fangtianxia_scrapy/middlewares.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/fangtianxia_scrapy_redis/fangtianxia_scrapy/middlewares.py -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/fangtianxia_scrapy/pipelines.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/fangtianxia_scrapy_redis/fangtianxia_scrapy/pipelines.py -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/fangtianxia_scrapy/settings.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/fangtianxia_scrapy_redis/fangtianxia_scrapy/settings.py -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/fangtianxia_scrapy/spiders/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/fangtianxia_scrapy_redis/fangtianxia_scrapy/spiders/__init__.py -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/fangtianxia_scrapy/spiders/fang.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/fangtianxia_scrapy_redis/fangtianxia_scrapy/spiders/fang.py -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/imgs/fang.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/fangtianxia_scrapy_redis/imgs/fang.jpg -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/imgs/linux_error.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/fangtianxia_scrapy_redis/imgs/linux_error.jpg -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/imgs/settings.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/fangtianxia_scrapy_redis/imgs/settings.jpg -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/fangtianxia_scrapy_redis/requirements.txt -------------------------------------------------------------------------------- /fangtianxia_scrapy_redis/scrapy.cfg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/fangtianxia_scrapy_redis/scrapy.cfg -------------------------------------------------------------------------------- /html2pdf/crawler.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/html2pdf/crawler.py -------------------------------------------------------------------------------- /html2pdf/liaoxuefeng_Python3_tutorial.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/html2pdf/liaoxuefeng_Python3_tutorial.pdf -------------------------------------------------------------------------------- /lianjia_scrapy/create_table.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/create_table.py -------------------------------------------------------------------------------- /lianjia_scrapy/get_url_list.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/get_url_list.py -------------------------------------------------------------------------------- /lianjia_scrapy/imgs/item_pipeline.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/imgs/item_pipeline.jpg -------------------------------------------------------------------------------- /lianjia_scrapy/imgs/mysql.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/imgs/mysql.jpg -------------------------------------------------------------------------------- /lianjia_scrapy/imgs/settings.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/imgs/settings.jpg -------------------------------------------------------------------------------- /lianjia_scrapy/imgs/公寓.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/imgs/公寓.jpg -------------------------------------------------------------------------------- /lianjia_scrapy/imgs/普通.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/imgs/普通.png -------------------------------------------------------------------------------- /lianjia_scrapy/imgs/高德api.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/imgs/高德api.jpg -------------------------------------------------------------------------------- /lianjia_scrapy/lianjiaSpider/lianjiaSpider/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /lianjia_scrapy/lianjiaSpider/lianjiaSpider/items.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/lianjiaSpider/lianjiaSpider/items.py -------------------------------------------------------------------------------- /lianjia_scrapy/lianjiaSpider/lianjiaSpider/middlewares.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/lianjiaSpider/lianjiaSpider/middlewares.py -------------------------------------------------------------------------------- /lianjia_scrapy/lianjiaSpider/lianjiaSpider/pipelines.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/lianjiaSpider/lianjiaSpider/pipelines.py -------------------------------------------------------------------------------- /lianjia_scrapy/lianjiaSpider/lianjiaSpider/settings.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/lianjiaSpider/lianjiaSpider/settings.py -------------------------------------------------------------------------------- /lianjia_scrapy/lianjiaSpider/lianjiaSpider/spiders/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/lianjiaSpider/lianjiaSpider/spiders/__init__.py -------------------------------------------------------------------------------- /lianjia_scrapy/lianjiaSpider/lianjiaSpider/spiders/lianjiazufang.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/lianjiaSpider/lianjiaSpider/spiders/lianjiazufang.py -------------------------------------------------------------------------------- /lianjia_scrapy/lianjiaSpider/scrapy.cfg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/lianjia_scrapy/lianjiaSpider/scrapy.cfg -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/requirements.txt -------------------------------------------------------------------------------- /微信公众号文章获取/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/微信公众号文章获取/README.md -------------------------------------------------------------------------------- /微信公众号文章获取/image/fiddler.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/微信公众号文章获取/image/fiddler.jpg -------------------------------------------------------------------------------- /微信公众号文章获取/image/fiddler01.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/微信公众号文章获取/image/fiddler01.jpg -------------------------------------------------------------------------------- /微信公众号文章获取/image/html.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LMFrank/CrawlerProject/HEAD/微信公众号文章获取/image/html.jpg --------------------------------------------------------------------------------