├── .gitignore ├── README.md ├── doc ├── queues.png ├── repos.png ├── routing.png ├── user.png ├── 爬虫流程.graffle ├── 爬虫流程.png ├── 程序流程.graffle └── 程序流程.png ├── github_spider ├── __init__.py ├── const.py ├── extensions.py ├── proxy │ ├── __init__.py │ └── extract.py ├── queue │ ├── __init__.py │ ├── consumer.py │ ├── main.py │ └── producer.py ├── recursion │ ├── __init__.py │ ├── flow.py │ ├── main.py │ └── request.py ├── settings.py ├── utils.py └── worker.py └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | .idea/* 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/README.md -------------------------------------------------------------------------------- /doc/queues.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/doc/queues.png -------------------------------------------------------------------------------- /doc/repos.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/doc/repos.png -------------------------------------------------------------------------------- /doc/routing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/doc/routing.png -------------------------------------------------------------------------------- /doc/user.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/doc/user.png -------------------------------------------------------------------------------- /doc/爬虫流程.graffle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/doc/爬虫流程.graffle -------------------------------------------------------------------------------- /doc/爬虫流程.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/doc/爬虫流程.png -------------------------------------------------------------------------------- /doc/程序流程.graffle: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/doc/程序流程.graffle -------------------------------------------------------------------------------- /doc/程序流程.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/doc/程序流程.png -------------------------------------------------------------------------------- /github_spider/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/__init__.py -------------------------------------------------------------------------------- /github_spider/const.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/const.py -------------------------------------------------------------------------------- /github_spider/extensions.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/extensions.py -------------------------------------------------------------------------------- /github_spider/proxy/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding=utf8 -*_ 2 | -------------------------------------------------------------------------------- /github_spider/proxy/extract.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/proxy/extract.py -------------------------------------------------------------------------------- /github_spider/queue/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/queue/__init__.py -------------------------------------------------------------------------------- /github_spider/queue/consumer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/queue/consumer.py -------------------------------------------------------------------------------- /github_spider/queue/main.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/queue/main.py -------------------------------------------------------------------------------- /github_spider/queue/producer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/queue/producer.py -------------------------------------------------------------------------------- /github_spider/recursion/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/recursion/__init__.py -------------------------------------------------------------------------------- /github_spider/recursion/flow.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/recursion/flow.py -------------------------------------------------------------------------------- /github_spider/recursion/main.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/recursion/main.py -------------------------------------------------------------------------------- /github_spider/recursion/request.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/recursion/request.py -------------------------------------------------------------------------------- /github_spider/settings.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/settings.py -------------------------------------------------------------------------------- /github_spider/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/utils.py -------------------------------------------------------------------------------- /github_spider/worker.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/github_spider/worker.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LiuRoy/github_spider/HEAD/requirements.txt --------------------------------------------------------------------------------