├── .gitignore ├── LICENSE ├── README.md ├── add_tokens.py ├── etc └── tokenizer.model ├── merge_tokenizer.py ├── requirements.txt ├── sputil ├── sentencepiece_model.proto └── sentencepiece_model_pb2.py ├── test_list.txt └── tokenizer_info.py /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__/ 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Ronsor/llama-tools/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Ronsor/llama-tools/HEAD/README.md -------------------------------------------------------------------------------- /add_tokens.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Ronsor/llama-tools/HEAD/add_tokens.py -------------------------------------------------------------------------------- /etc/tokenizer.model: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Ronsor/llama-tools/HEAD/etc/tokenizer.model -------------------------------------------------------------------------------- /merge_tokenizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Ronsor/llama-tools/HEAD/merge_tokenizer.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | protobuf==4.22.1 2 | -------------------------------------------------------------------------------- /sputil/sentencepiece_model.proto: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Ronsor/llama-tools/HEAD/sputil/sentencepiece_model.proto -------------------------------------------------------------------------------- /sputil/sentencepiece_model_pb2.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Ronsor/llama-tools/HEAD/sputil/sentencepiece_model_pb2.py -------------------------------------------------------------------------------- /test_list.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Ronsor/llama-tools/HEAD/test_list.txt -------------------------------------------------------------------------------- /tokenizer_info.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Ronsor/llama-tools/HEAD/tokenizer_info.py --------------------------------------------------------------------------------