├── LICENSE.txt ├── README.md ├── bbpe.py ├── models ├── llama3 │ ├── merges.txt │ └── vocab.json ├── qwen │ ├── merges.txt │ └── vocab.json └── 斗破苍穹 │ ├── merges.txt │ └── vocab.json ├── test.py ├── train.py └── train_data └── 斗破苍穹.txt /LICENSE.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OctopusMind/BBPE/HEAD/LICENSE.txt -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OctopusMind/BBPE/HEAD/README.md -------------------------------------------------------------------------------- /bbpe.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OctopusMind/BBPE/HEAD/bbpe.py -------------------------------------------------------------------------------- /models/llama3/merges.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OctopusMind/BBPE/HEAD/models/llama3/merges.txt -------------------------------------------------------------------------------- /models/llama3/vocab.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OctopusMind/BBPE/HEAD/models/llama3/vocab.json -------------------------------------------------------------------------------- /models/qwen/merges.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OctopusMind/BBPE/HEAD/models/qwen/merges.txt -------------------------------------------------------------------------------- /models/qwen/vocab.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OctopusMind/BBPE/HEAD/models/qwen/vocab.json -------------------------------------------------------------------------------- /models/斗破苍穹/merges.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OctopusMind/BBPE/HEAD/models/斗破苍穹/merges.txt -------------------------------------------------------------------------------- /models/斗破苍穹/vocab.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OctopusMind/BBPE/HEAD/models/斗破苍穹/vocab.json -------------------------------------------------------------------------------- /test.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OctopusMind/BBPE/HEAD/test.py -------------------------------------------------------------------------------- /train.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OctopusMind/BBPE/HEAD/train.py -------------------------------------------------------------------------------- /train_data/斗破苍穹.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/OctopusMind/BBPE/HEAD/train_data/斗破苍穹.txt --------------------------------------------------------------------------------