├── .gitignore ├── README.md ├── bin ├── index_lookup_local ├── index_lookup_remote └── remote_copy ├── docs ├── data_block.png ├── file_format.graffle │ ├── data.plist │ └── image1.pdf ├── file_overview.png ├── header.png ├── index_block.png ├── project_header.png └── tree.png └── lib ├── __init__.py ├── adaptor.py ├── pbtree.py ├── prefix.py ├── sorted_urls ├── test.py ├── test_map.py └── test_pbtree.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[co] 2 | .DS_Store 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/README.md -------------------------------------------------------------------------------- /bin/index_lookup_local: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/bin/index_lookup_local -------------------------------------------------------------------------------- /bin/index_lookup_remote: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/bin/index_lookup_remote -------------------------------------------------------------------------------- /bin/remote_copy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/bin/remote_copy -------------------------------------------------------------------------------- /docs/data_block.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/docs/data_block.png -------------------------------------------------------------------------------- /docs/file_format.graffle/data.plist: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/docs/file_format.graffle/data.plist -------------------------------------------------------------------------------- /docs/file_format.graffle/image1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/docs/file_format.graffle/image1.pdf -------------------------------------------------------------------------------- /docs/file_overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/docs/file_overview.png -------------------------------------------------------------------------------- /docs/header.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/docs/header.png -------------------------------------------------------------------------------- /docs/index_block.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/docs/index_block.png -------------------------------------------------------------------------------- /docs/project_header.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/docs/project_header.png -------------------------------------------------------------------------------- /docs/tree.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/docs/tree.png -------------------------------------------------------------------------------- /lib/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/lib/__init__.py -------------------------------------------------------------------------------- /lib/adaptor.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/lib/adaptor.py -------------------------------------------------------------------------------- /lib/pbtree.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/lib/pbtree.py -------------------------------------------------------------------------------- /lib/prefix.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/lib/prefix.py -------------------------------------------------------------------------------- /lib/sorted_urls: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/lib/sorted_urls -------------------------------------------------------------------------------- /lib/test.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/lib/test.py -------------------------------------------------------------------------------- /lib/test_map.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/lib/test_map.py -------------------------------------------------------------------------------- /lib/test_pbtree.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/trivio/common_crawl_index/HEAD/lib/test_pbtree.py --------------------------------------------------------------------------------