├── .gitignore ├── LICENSE ├── README.md └── scrapy_scrapers ├── environment.yml └── src ├── __init__.py ├── items.py ├── pipelines.py ├── scrapy.cfg ├── settings.py ├── spiders ├── __init__.py └── scrapers.py └── tests ├── test_elasticsearch.py ├── test_pages └── dmoz_index.html ├── test_pipeline.py └── test_scraper.py /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/.gitignore -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/README.md -------------------------------------------------------------------------------- /scrapy_scrapers/environment.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/scrapy_scrapers/environment.yml -------------------------------------------------------------------------------- /scrapy_scrapers/src/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /scrapy_scrapers/src/items.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/scrapy_scrapers/src/items.py -------------------------------------------------------------------------------- /scrapy_scrapers/src/pipelines.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/scrapy_scrapers/src/pipelines.py -------------------------------------------------------------------------------- /scrapy_scrapers/src/scrapy.cfg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/scrapy_scrapers/src/scrapy.cfg -------------------------------------------------------------------------------- /scrapy_scrapers/src/settings.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/scrapy_scrapers/src/settings.py -------------------------------------------------------------------------------- /scrapy_scrapers/src/spiders/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /scrapy_scrapers/src/spiders/scrapers.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/scrapy_scrapers/src/spiders/scrapers.py -------------------------------------------------------------------------------- /scrapy_scrapers/src/tests/test_elasticsearch.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/scrapy_scrapers/src/tests/test_elasticsearch.py -------------------------------------------------------------------------------- /scrapy_scrapers/src/tests/test_pages/dmoz_index.html: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/scrapy_scrapers/src/tests/test_pages/dmoz_index.html -------------------------------------------------------------------------------- /scrapy_scrapers/src/tests/test_pipeline.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/scrapy_scrapers/src/tests/test_pipeline.py -------------------------------------------------------------------------------- /scrapy_scrapers/src/tests/test_scraper.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ContinuumIO/scrapy_scrapers/HEAD/scrapy_scrapers/src/tests/test_scraper.py --------------------------------------------------------------------------------