├── README.md ├── example.py ├── figures ├── longheads.png └── passkey_128k.png ├── modeling_longheads ├── __init__.py ├── configuration_llama.py ├── convert_llama_weights_to_hf.py ├── modeling_llama.py ├── tokenization_llama.py └── tokenization_llama_fast.py ├── passkey_retrieval ├── passkey_retrieval.py └── passkey_retrieval_script.sh └── requirements.txt /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/README.md -------------------------------------------------------------------------------- /example.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/example.py -------------------------------------------------------------------------------- /figures/longheads.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/figures/longheads.png -------------------------------------------------------------------------------- /figures/passkey_128k.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/figures/passkey_128k.png -------------------------------------------------------------------------------- /modeling_longheads/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/modeling_longheads/__init__.py -------------------------------------------------------------------------------- /modeling_longheads/configuration_llama.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/modeling_longheads/configuration_llama.py -------------------------------------------------------------------------------- /modeling_longheads/convert_llama_weights_to_hf.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/modeling_longheads/convert_llama_weights_to_hf.py -------------------------------------------------------------------------------- /modeling_longheads/modeling_llama.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/modeling_longheads/modeling_llama.py -------------------------------------------------------------------------------- /modeling_longheads/tokenization_llama.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/modeling_longheads/tokenization_llama.py -------------------------------------------------------------------------------- /modeling_longheads/tokenization_llama_fast.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/modeling_longheads/tokenization_llama_fast.py -------------------------------------------------------------------------------- /passkey_retrieval/passkey_retrieval.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/passkey_retrieval/passkey_retrieval.py -------------------------------------------------------------------------------- /passkey_retrieval/passkey_retrieval_script.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/passkey_retrieval/passkey_retrieval_script.sh -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/LuLuLuyi/LongHeads/HEAD/requirements.txt --------------------------------------------------------------------------------