├── .gitignore ├── LICENSE ├── README.MD ├── args.MD ├── chat.py ├── config ├── __init__.py ├── colossalai_strategy.yaml ├── deepspeed.yaml ├── deepspeed_offload.yaml ├── global.yaml ├── main.py ├── petl.yaml ├── train_ac.yaml ├── train_cl.yaml ├── train_hf.yaml └── train_pl.yaml ├── data ├── finetune_train_conversations.json ├── finetune_train_paragraph.json └── make_data_example.py ├── data_processer.py ├── data_utils.py ├── infer ├── __init__.py ├── infer.py ├── infer_finetuning.py ├── infer_lora_finetuning.py └── infer_muti_lora_finetuning.py ├── requirements.txt ├── scripts ├── train_full.sh ├── train_lora.sh ├── train_lora_int4.sh └── train_lora_int8.sh ├── tools ├── 1.png ├── 2.png ├── 20B_tokenizer.json ├── __init__.py ├── convert_rwkv4.py ├── rwkv4_infer │ ├── 20B_tokenizer.json │ ├── __init__.py │ ├── chat.py │ ├── model_run.py │ └── utils.py └── rwkv_vocab_v20230424.txt ├── train.py └── training ├── __init__.py ├── train_ac.py ├── train_cl.py ├── train_hf.py └── train_pl.py /.gitignore: -------------------------------------------------------------------------------- 1 | /.idea 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/LICENSE -------------------------------------------------------------------------------- /README.MD: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/README.MD -------------------------------------------------------------------------------- /args.MD: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/args.MD -------------------------------------------------------------------------------- /chat.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/chat.py -------------------------------------------------------------------------------- /config/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/config/__init__.py -------------------------------------------------------------------------------- /config/colossalai_strategy.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/config/colossalai_strategy.yaml -------------------------------------------------------------------------------- /config/deepspeed.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/config/deepspeed.yaml -------------------------------------------------------------------------------- /config/deepspeed_offload.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/config/deepspeed_offload.yaml -------------------------------------------------------------------------------- /config/global.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/config/global.yaml -------------------------------------------------------------------------------- /config/main.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/config/main.py -------------------------------------------------------------------------------- /config/petl.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/config/petl.yaml -------------------------------------------------------------------------------- /config/train_ac.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/config/train_ac.yaml -------------------------------------------------------------------------------- /config/train_cl.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/config/train_cl.yaml -------------------------------------------------------------------------------- /config/train_hf.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/config/train_hf.yaml -------------------------------------------------------------------------------- /config/train_pl.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/config/train_pl.yaml -------------------------------------------------------------------------------- /data/finetune_train_conversations.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/data/finetune_train_conversations.json -------------------------------------------------------------------------------- /data/finetune_train_paragraph.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/data/finetune_train_paragraph.json -------------------------------------------------------------------------------- /data/make_data_example.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/data/make_data_example.py -------------------------------------------------------------------------------- /data_processer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/data_processer.py -------------------------------------------------------------------------------- /data_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/data_utils.py -------------------------------------------------------------------------------- /infer/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/infer/__init__.py -------------------------------------------------------------------------------- /infer/infer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/infer/infer.py -------------------------------------------------------------------------------- /infer/infer_finetuning.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/infer/infer_finetuning.py -------------------------------------------------------------------------------- /infer/infer_lora_finetuning.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/infer/infer_lora_finetuning.py -------------------------------------------------------------------------------- /infer/infer_muti_lora_finetuning.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/infer/infer_muti_lora_finetuning.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/requirements.txt -------------------------------------------------------------------------------- /scripts/train_full.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/scripts/train_full.sh -------------------------------------------------------------------------------- /scripts/train_lora.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/scripts/train_lora.sh -------------------------------------------------------------------------------- /scripts/train_lora_int4.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/scripts/train_lora_int4.sh -------------------------------------------------------------------------------- /scripts/train_lora_int8.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/scripts/train_lora_int8.sh -------------------------------------------------------------------------------- /tools/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/tools/1.png -------------------------------------------------------------------------------- /tools/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/tools/2.png -------------------------------------------------------------------------------- /tools/20B_tokenizer.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/tools/20B_tokenizer.json -------------------------------------------------------------------------------- /tools/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/tools/__init__.py -------------------------------------------------------------------------------- /tools/convert_rwkv4.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/tools/convert_rwkv4.py -------------------------------------------------------------------------------- /tools/rwkv4_infer/20B_tokenizer.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/tools/rwkv4_infer/20B_tokenizer.json -------------------------------------------------------------------------------- /tools/rwkv4_infer/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/tools/rwkv4_infer/__init__.py -------------------------------------------------------------------------------- /tools/rwkv4_infer/chat.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/tools/rwkv4_infer/chat.py -------------------------------------------------------------------------------- /tools/rwkv4_infer/model_run.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/tools/rwkv4_infer/model_run.py -------------------------------------------------------------------------------- /tools/rwkv4_infer/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/tools/rwkv4_infer/utils.py -------------------------------------------------------------------------------- /tools/rwkv_vocab_v20230424.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/tools/rwkv_vocab_v20230424.txt -------------------------------------------------------------------------------- /train.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/train.py -------------------------------------------------------------------------------- /training/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/training/__init__.py -------------------------------------------------------------------------------- /training/train_ac.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/training/train_ac.py -------------------------------------------------------------------------------- /training/train_cl.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/training/train_cl.py -------------------------------------------------------------------------------- /training/train_hf.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/training/train_hf.py -------------------------------------------------------------------------------- /training/train_pl.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbuild/rwkv_finetuning/HEAD/training/train_pl.py --------------------------------------------------------------------------------