├── .gitignore ├── .travis.yml ├── LICENSE ├── README.md ├── lm_dataformat └── __init__.py ├── setup.py └── test ├── blns.jsonl.zst.tar ├── blns.txt ├── blns.txt.tar.gz ├── blns.txt.zip ├── test_dat_archive_reader.py └── testtarfile.tar /.gitignore: -------------------------------------------------------------------------------- 1 | env 2 | *.pyc 3 | *.egg-info 4 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leogao2/lm_dataformat/HEAD/.travis.yml -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leogao2/lm_dataformat/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leogao2/lm_dataformat/HEAD/README.md -------------------------------------------------------------------------------- /lm_dataformat/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leogao2/lm_dataformat/HEAD/lm_dataformat/__init__.py -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leogao2/lm_dataformat/HEAD/setup.py -------------------------------------------------------------------------------- /test/blns.jsonl.zst.tar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leogao2/lm_dataformat/HEAD/test/blns.jsonl.zst.tar -------------------------------------------------------------------------------- /test/blns.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leogao2/lm_dataformat/HEAD/test/blns.txt -------------------------------------------------------------------------------- /test/blns.txt.tar.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leogao2/lm_dataformat/HEAD/test/blns.txt.tar.gz -------------------------------------------------------------------------------- /test/blns.txt.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leogao2/lm_dataformat/HEAD/test/blns.txt.zip -------------------------------------------------------------------------------- /test/test_dat_archive_reader.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leogao2/lm_dataformat/HEAD/test/test_dat_archive_reader.py -------------------------------------------------------------------------------- /test/testtarfile.tar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leogao2/lm_dataformat/HEAD/test/testtarfile.tar --------------------------------------------------------------------------------