├── .gitignore ├── .travis.yml ├── README.md ├── docs ├── item.md ├── process.png └── spider.md ├── examples ├── bd_img_demo.py ├── douban250.txt ├── douban_page_by_item.py ├── douban_page_by_spider.py ├── qidian_details_by_item.py ├── qidian_details_by_spider.py ├── qidian_ranking.txt └── qidian_ranking_demo.py ├── requirements.txt ├── setup.py ├── talospider ├── __init__.py ├── downloader.py ├── field.py ├── item.py ├── spider.py └── utils │ ├── __init__.py │ ├── function.py │ ├── log.py │ └── user_agents.txt └── tests ├── test_fields.py ├── test_item.py └── test_request.py /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/.gitignore -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/.travis.yml -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/README.md -------------------------------------------------------------------------------- /docs/item.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/docs/item.md -------------------------------------------------------------------------------- /docs/process.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/docs/process.png -------------------------------------------------------------------------------- /docs/spider.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/docs/spider.md -------------------------------------------------------------------------------- /examples/bd_img_demo.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/examples/bd_img_demo.py -------------------------------------------------------------------------------- /examples/douban250.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/examples/douban250.txt -------------------------------------------------------------------------------- /examples/douban_page_by_item.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/examples/douban_page_by_item.py -------------------------------------------------------------------------------- /examples/douban_page_by_spider.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/examples/douban_page_by_spider.py -------------------------------------------------------------------------------- /examples/qidian_details_by_item.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/examples/qidian_details_by_item.py -------------------------------------------------------------------------------- /examples/qidian_details_by_spider.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/examples/qidian_details_by_spider.py -------------------------------------------------------------------------------- /examples/qidian_ranking.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/examples/qidian_ranking.txt -------------------------------------------------------------------------------- /examples/qidian_ranking_demo.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/examples/qidian_ranking_demo.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | requests 2 | lxml 3 | cchardet 4 | cssselect -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/setup.py -------------------------------------------------------------------------------- /talospider/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/talospider/__init__.py -------------------------------------------------------------------------------- /talospider/downloader.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/talospider/downloader.py -------------------------------------------------------------------------------- /talospider/field.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/talospider/field.py -------------------------------------------------------------------------------- /talospider/item.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/talospider/item.py -------------------------------------------------------------------------------- /talospider/spider.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/talospider/spider.py -------------------------------------------------------------------------------- /talospider/utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/talospider/utils/__init__.py -------------------------------------------------------------------------------- /talospider/utils/function.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/talospider/utils/function.py -------------------------------------------------------------------------------- /talospider/utils/log.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/talospider/utils/log.py -------------------------------------------------------------------------------- /talospider/utils/user_agents.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/talospider/utils/user_agents.txt -------------------------------------------------------------------------------- /tests/test_fields.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/tests/test_fields.py -------------------------------------------------------------------------------- /tests/test_item.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/tests/test_item.py -------------------------------------------------------------------------------- /tests/test_request.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/howie6879/talospider/HEAD/tests/test_request.py --------------------------------------------------------------------------------