├── .github └── workflows │ └── build_wheels.yml ├── .gitignore ├── CHANGELOG.md ├── Cargo.toml ├── LICENSE ├── MANIFEST.in ├── README.md ├── perf.svg ├── pyproject.toml ├── scripts ├── benchmark.py ├── redact.py └── wheel_download.py ├── setup.py ├── src ├── lib.rs └── py.rs ├── tests ├── __init__.py ├── test_encoding.py ├── test_helpers.py ├── test_misc.py ├── test_offsets.py ├── test_pickle.py └── test_simple_public.py ├── tiktoken ├── __init__.py ├── _educational.py ├── core.py ├── load.py ├── model.py ├── py.typed └── registry.py └── tiktoken_ext └── openai_public.py /.github/workflows/build_wheels.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/.github/workflows/build_wheels.yml -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/.gitignore -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/CHANGELOG.md -------------------------------------------------------------------------------- /Cargo.toml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/Cargo.toml -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/LICENSE -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/MANIFEST.in -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/README.md -------------------------------------------------------------------------------- /perf.svg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/perf.svg -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/pyproject.toml -------------------------------------------------------------------------------- /scripts/benchmark.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/scripts/benchmark.py -------------------------------------------------------------------------------- /scripts/redact.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/scripts/redact.py -------------------------------------------------------------------------------- /scripts/wheel_download.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/scripts/wheel_download.py -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/setup.py -------------------------------------------------------------------------------- /src/lib.rs: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/src/lib.rs -------------------------------------------------------------------------------- /src/py.rs: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/src/py.rs -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/test_encoding.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tests/test_encoding.py -------------------------------------------------------------------------------- /tests/test_helpers.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tests/test_helpers.py -------------------------------------------------------------------------------- /tests/test_misc.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tests/test_misc.py -------------------------------------------------------------------------------- /tests/test_offsets.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tests/test_offsets.py -------------------------------------------------------------------------------- /tests/test_pickle.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tests/test_pickle.py -------------------------------------------------------------------------------- /tests/test_simple_public.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tests/test_simple_public.py -------------------------------------------------------------------------------- /tiktoken/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tiktoken/__init__.py -------------------------------------------------------------------------------- /tiktoken/_educational.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tiktoken/_educational.py -------------------------------------------------------------------------------- /tiktoken/core.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tiktoken/core.py -------------------------------------------------------------------------------- /tiktoken/load.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tiktoken/load.py -------------------------------------------------------------------------------- /tiktoken/model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tiktoken/model.py -------------------------------------------------------------------------------- /tiktoken/py.typed: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tiktoken/registry.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tiktoken/registry.py -------------------------------------------------------------------------------- /tiktoken_ext/openai_public.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/openai/tiktoken/HEAD/tiktoken_ext/openai_public.py --------------------------------------------------------------------------------