├── .gitignore ├── README.md ├── README_zh.md ├── check_lambda ├── lambda_hf.py └── lambda_pt.py ├── run_hf_world_model_on_cpu.py ├── run_hf_world_model_on_gpu.py ├── rwkv5_model ├── README.md ├── config.json ├── configuration_rwkv5.py ├── generation_config.json ├── modeling_rwkv5.py ├── special_tokens_map.json ├── wkv5_cuda.cu └── wkv5_op.cpp ├── rwkv5_world_tokenizer ├── config.json ├── tokenization_rwkv5.py ├── tokenizer_config.json └── vocab.txt ├── rwkv6_model ├── README.md ├── config.json ├── configuration_rwkv6.py ├── generation_config.json ├── modeling_rwkv6.py └── special_tokens_map.json ├── rwkv_tokenizer.py ├── rwkv_vocab_v20230424.txt └── scripts ├── configuration_rwkv5.py ├── configuration_rwkv6.py ├── convert5to6.py ├── convert_rwkv5_checkpoint_to_hf.py ├── convert_rwkv5_world_model_to_hf.sh ├── convert_rwkv6_checkpoint_to_hf.py └── convert_rwkv6_world_model_to_hf.sh /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/.gitignore -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/README.md -------------------------------------------------------------------------------- /README_zh.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/README_zh.md -------------------------------------------------------------------------------- /check_lambda/lambda_hf.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/check_lambda/lambda_hf.py -------------------------------------------------------------------------------- /check_lambda/lambda_pt.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/check_lambda/lambda_pt.py -------------------------------------------------------------------------------- /run_hf_world_model_on_cpu.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/run_hf_world_model_on_cpu.py -------------------------------------------------------------------------------- /run_hf_world_model_on_gpu.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/run_hf_world_model_on_gpu.py -------------------------------------------------------------------------------- /rwkv5_model/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv5_model/README.md -------------------------------------------------------------------------------- /rwkv5_model/config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv5_model/config.json -------------------------------------------------------------------------------- /rwkv5_model/configuration_rwkv5.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv5_model/configuration_rwkv5.py -------------------------------------------------------------------------------- /rwkv5_model/generation_config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv5_model/generation_config.json -------------------------------------------------------------------------------- /rwkv5_model/modeling_rwkv5.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv5_model/modeling_rwkv5.py -------------------------------------------------------------------------------- /rwkv5_model/special_tokens_map.json: -------------------------------------------------------------------------------- 1 | {} 2 | -------------------------------------------------------------------------------- /rwkv5_model/wkv5_cuda.cu: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv5_model/wkv5_cuda.cu -------------------------------------------------------------------------------- /rwkv5_model/wkv5_op.cpp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv5_model/wkv5_op.cpp -------------------------------------------------------------------------------- /rwkv5_world_tokenizer/config.json: -------------------------------------------------------------------------------- 1 | { 2 | } -------------------------------------------------------------------------------- /rwkv5_world_tokenizer/tokenization_rwkv5.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv5_world_tokenizer/tokenization_rwkv5.py -------------------------------------------------------------------------------- /rwkv5_world_tokenizer/tokenizer_config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv5_world_tokenizer/tokenizer_config.json -------------------------------------------------------------------------------- /rwkv5_world_tokenizer/vocab.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv5_world_tokenizer/vocab.txt -------------------------------------------------------------------------------- /rwkv6_model/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv6_model/README.md -------------------------------------------------------------------------------- /rwkv6_model/config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv6_model/config.json -------------------------------------------------------------------------------- /rwkv6_model/configuration_rwkv6.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv6_model/configuration_rwkv6.py -------------------------------------------------------------------------------- /rwkv6_model/generation_config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv6_model/generation_config.json -------------------------------------------------------------------------------- /rwkv6_model/modeling_rwkv6.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv6_model/modeling_rwkv6.py -------------------------------------------------------------------------------- /rwkv6_model/special_tokens_map.json: -------------------------------------------------------------------------------- 1 | {} 2 | -------------------------------------------------------------------------------- /rwkv_tokenizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv_tokenizer.py -------------------------------------------------------------------------------- /rwkv_vocab_v20230424.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/rwkv_vocab_v20230424.txt -------------------------------------------------------------------------------- /scripts/configuration_rwkv5.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/scripts/configuration_rwkv5.py -------------------------------------------------------------------------------- /scripts/configuration_rwkv6.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/scripts/configuration_rwkv6.py -------------------------------------------------------------------------------- /scripts/convert5to6.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/scripts/convert5to6.py -------------------------------------------------------------------------------- /scripts/convert_rwkv5_checkpoint_to_hf.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/scripts/convert_rwkv5_checkpoint_to_hf.py -------------------------------------------------------------------------------- /scripts/convert_rwkv5_world_model_to_hf.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/scripts/convert_rwkv5_world_model_to_hf.sh -------------------------------------------------------------------------------- /scripts/convert_rwkv6_checkpoint_to_hf.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/scripts/convert_rwkv6_checkpoint_to_hf.py -------------------------------------------------------------------------------- /scripts/convert_rwkv6_world_model_to_hf.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/BBuf/RWKV-World-HF-Tokenizer/HEAD/scripts/convert_rwkv6_world_model_to_hf.sh --------------------------------------------------------------------------------