├── LICENSE ├── LLMLingua.py ├── Readme.md ├── assets ├── Q&A.md ├── logo.png ├── results.png └── tokenskip.png ├── configs ├── examples │ └── train_lora │ │ ├── myllama3_lora_sft_compressed_gsm8k_llmlingua2_qwen_14B.yaml │ │ ├── myllama3_lora_sft_compressed_gsm8k_llmlingua2_qwen_3B.yaml │ │ └── myllama3_lora_sft_compressed_gsm8k_llmlingua2_qwen_7B.yaml ├── gsm8k_test.json ├── gsm8k_train.json ├── math_test.json └── math_train.json ├── data_processing ├── answer_extraction.py └── process_utils.py ├── datasets ├── gsm8k │ ├── llamafactory_inputs │ │ ├── mydataset_compressed_gsm8k_llmlingua2_qwen_14B.json │ │ ├── mydataset_compressed_gsm8k_llmlingua2_qwen_3B.json │ │ └── mydataset_compressed_gsm8k_llmlingua2_qwen_7B.json │ ├── test.jsonl │ └── train.jsonl └── math-500 │ ├── test.jsonl │ └── train.jsonl ├── eval.sh ├── eval ├── eval_script.py ├── eval_utils.py └── utils.py ├── evaluation.py ├── get_llamafactory_input.py └── requirements.txt /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/LICENSE -------------------------------------------------------------------------------- /LLMLingua.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/LLMLingua.py -------------------------------------------------------------------------------- /Readme.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/Readme.md -------------------------------------------------------------------------------- /assets/Q&A.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/assets/Q&A.md -------------------------------------------------------------------------------- /assets/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/assets/logo.png -------------------------------------------------------------------------------- /assets/results.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/assets/results.png -------------------------------------------------------------------------------- /assets/tokenskip.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/assets/tokenskip.png -------------------------------------------------------------------------------- /configs/examples/train_lora/myllama3_lora_sft_compressed_gsm8k_llmlingua2_qwen_14B.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/configs/examples/train_lora/myllama3_lora_sft_compressed_gsm8k_llmlingua2_qwen_14B.yaml -------------------------------------------------------------------------------- /configs/examples/train_lora/myllama3_lora_sft_compressed_gsm8k_llmlingua2_qwen_3B.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/configs/examples/train_lora/myllama3_lora_sft_compressed_gsm8k_llmlingua2_qwen_3B.yaml -------------------------------------------------------------------------------- /configs/examples/train_lora/myllama3_lora_sft_compressed_gsm8k_llmlingua2_qwen_7B.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/configs/examples/train_lora/myllama3_lora_sft_compressed_gsm8k_llmlingua2_qwen_7B.yaml -------------------------------------------------------------------------------- /configs/gsm8k_test.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/configs/gsm8k_test.json -------------------------------------------------------------------------------- /configs/gsm8k_train.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/configs/gsm8k_train.json -------------------------------------------------------------------------------- /configs/math_test.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/configs/math_test.json -------------------------------------------------------------------------------- /configs/math_train.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/configs/math_train.json -------------------------------------------------------------------------------- /data_processing/answer_extraction.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/data_processing/answer_extraction.py -------------------------------------------------------------------------------- /data_processing/process_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/data_processing/process_utils.py -------------------------------------------------------------------------------- /datasets/gsm8k/llamafactory_inputs/mydataset_compressed_gsm8k_llmlingua2_qwen_14B.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/datasets/gsm8k/llamafactory_inputs/mydataset_compressed_gsm8k_llmlingua2_qwen_14B.json -------------------------------------------------------------------------------- /datasets/gsm8k/llamafactory_inputs/mydataset_compressed_gsm8k_llmlingua2_qwen_3B.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/datasets/gsm8k/llamafactory_inputs/mydataset_compressed_gsm8k_llmlingua2_qwen_3B.json -------------------------------------------------------------------------------- /datasets/gsm8k/llamafactory_inputs/mydataset_compressed_gsm8k_llmlingua2_qwen_7B.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/datasets/gsm8k/llamafactory_inputs/mydataset_compressed_gsm8k_llmlingua2_qwen_7B.json -------------------------------------------------------------------------------- /datasets/gsm8k/test.jsonl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/datasets/gsm8k/test.jsonl -------------------------------------------------------------------------------- /datasets/gsm8k/train.jsonl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/datasets/gsm8k/train.jsonl -------------------------------------------------------------------------------- /datasets/math-500/test.jsonl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/datasets/math-500/test.jsonl -------------------------------------------------------------------------------- /datasets/math-500/train.jsonl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/datasets/math-500/train.jsonl -------------------------------------------------------------------------------- /eval.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/eval.sh -------------------------------------------------------------------------------- /eval/eval_script.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/eval/eval_script.py -------------------------------------------------------------------------------- /eval/eval_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/eval/eval_utils.py -------------------------------------------------------------------------------- /eval/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/eval/utils.py -------------------------------------------------------------------------------- /evaluation.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/evaluation.py -------------------------------------------------------------------------------- /get_llamafactory_input.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/get_llamafactory_input.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hemingkx/TokenSkip/HEAD/requirements.txt --------------------------------------------------------------------------------