├── .gitignore ├── .pre-commit-config.yaml ├── .pylintrc ├── LICENSE ├── README.md ├── figures ├── parallel_tokenizer.gif └── tokenizer_speed.png ├── parallel_tokenizer ├── __init__.py ├── logger.py ├── parallel_tokenizer.py ├── sp_tokenizer.py ├── special_cases.py └── utils.py ├── setup.py ├── tests ├── __init__.py ├── test_interface.py ├── test_parallel_tokenizer.py └── utils.py └── version.txt /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/.gitignore -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/.pre-commit-config.yaml -------------------------------------------------------------------------------- /.pylintrc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/.pylintrc -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/README.md -------------------------------------------------------------------------------- /figures/parallel_tokenizer.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/figures/parallel_tokenizer.gif -------------------------------------------------------------------------------- /figures/tokenizer_speed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/figures/tokenizer_speed.png -------------------------------------------------------------------------------- /parallel_tokenizer/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/parallel_tokenizer/__init__.py -------------------------------------------------------------------------------- /parallel_tokenizer/logger.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/parallel_tokenizer/logger.py -------------------------------------------------------------------------------- /parallel_tokenizer/parallel_tokenizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/parallel_tokenizer/parallel_tokenizer.py -------------------------------------------------------------------------------- /parallel_tokenizer/sp_tokenizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/parallel_tokenizer/sp_tokenizer.py -------------------------------------------------------------------------------- /parallel_tokenizer/special_cases.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/parallel_tokenizer/special_cases.py -------------------------------------------------------------------------------- /parallel_tokenizer/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/parallel_tokenizer/utils.py -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/setup.py -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/test_interface.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/tests/test_interface.py -------------------------------------------------------------------------------- /tests/test_parallel_tokenizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/tests/test_parallel_tokenizer.py -------------------------------------------------------------------------------- /tests/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OpenLMLab/ParallelTokenizer/HEAD/tests/utils.py -------------------------------------------------------------------------------- /version.txt: -------------------------------------------------------------------------------- 1 | 0.1.0 2 | --------------------------------------------------------------------------------