├── README.md └── LICENSE /README.md: -------------------------------------------------------------------------------- 1 | # llama2-Chinese-chat 2 | 更新记录: 3 | - 2023-07-19 全网首个中文对话llama2 13b版本发布，实现可流畅中文对话 4 | - 2023-07-23 完成第2个epoch训练放出，有更好的对话体验 5 | - 2023-08-03 分支版本：bimoGPT放出，拥有自我身份认知、不错的代码问答能力，权重下载地址：https://huggingface.co/shareAI/bimoGPT-llama2-13b 6 | - 2023-08-21 更新世界模型排名榜，超越某号称“中文Llama2官方”社区的收费模型十多个名次。 7 | 8 | ![image](https://github.com/CrazyBoyM/llama2-Chinese-chat/assets/35400185/1aefbbf4-cd54-4ea2-a07a-c15f73a829ef) 9 | 10 | 11 | llama2 Chinese chat - 本项目是一个教程记录整理的repo，旨在提供给新手的参照价值和开箱即用的中文LLaMa2对话体验。包含训练过程记录，各种主要量化方式，部署后端api的推荐方案，以及在一个具体的前端网页上实现开箱即用的流畅对话体验。 12 | 也是当前第一个实际可用的中文13b llama2对话（已实现且放出实际文件）。 13 | - 中文llama2对话模型下载（仅差分部分）：https://huggingface.co/shareAI/llama2-13b-Chinese-chat 14 | - 完整权重下载（最新的练到了第2个epoch）：https://www.codewithgpu.com/m/file/llama2-13b-Chinese-chat 15 | - 训练用数据集：https://huggingface.co/datasets/shareAI/ShareGPT-Chinese-English-90k 16 | - llama2训练交流QQ群：443064756 17 | 18 | 更多实测问答记录：https://gist.github.com/goog/a3276016bd17eb401d6cc74d1f96d2e0 感谢@goog 提供 19 | image 20 | image 21 | 22 | 23 | --- 24 | library_name: peft 25 | datasets: 26 | - shareAI/shareGPT_cn 27 | - FreedomIntelligence/ShareGPT-CN 28 | language: 29 | - zh 30 | pipeline_tag: question-answering 31 | tags: 32 | - chat 33 | - llm 34 | - llama2 35 | - chatgpt 36 | --- 37 | tip: 优秀对话llm的训练离不开高质量的多轮对话数据集，如果你也想成为志愿者 38 | 欢迎加入QQ群：130920969，共同进行优质数据集的交流、收集和建设工作 39 | 40 | 项目在中文sharegpt数据集上训练得到的llama2 Chinese chat 13b，为减轻文件大小负担这里只放出了adapter的权重 41 | 请拉取https://huggingface.co/TheBloke/Llama-2-13B-fp16 作为基础权重，使用如下脚步执行合并得到可工作的总权重： 42 | 43 | ```python 44 | from peft import PeftModel 45 | from transformers import AutoModelForCausalLM, AutoTokenizer 46 | import torch 47 | 48 | model_name_or_path = '/data/TheBloke/Llama-2-13B-fp16' 49 | adapter_name_or_path = '/data/llama2-13b-Chinese-chat' 50 | save_path = '/data/llama2-13b-Chinese-chat_v1' 51 | 52 | tokenizer = AutoTokenizer.from_pretrained( 53 | model_name_or_path, 54 | trust_remote_code=True 55 | ) 56 | model = AutoModelForCausalLM.from_pretrained( 57 | model_name_or_path, 58 | trust_remote_code=True, 59 | low_cpu_mem_usage=True, 60 | torch_dtype=torch.float16, 61 | device_map='auto' 62 | ) 63 | print("load model success") 64 | model = PeftModel.from_pretrained(model, adapter_name_or_path) 65 | print("load adapter success") 66 | model = model.merge_and_unload() 67 | print("merge success") 68 | 69 | tokenizer.save_pretrained(save_path) 70 | model.save_pretrained(save_path) 71 | print("save done.") 72 | ``` 73 | 合并后，体验对话： 74 | ```python 75 | from transformers import AutoModelForCausalLM, AutoTokenizer 76 | import torch 77 | 78 | 79 | def main(): 80 | model_name = '/data/llama2-13b-Chinese-chat_v1' 81 | 82 | device = 'cuda' 83 | max_new_tokens = 500 # 每轮对话最多生成多少个token 84 | history_max_len = 2000 # 模型记忆的最大token长度 85 | top_p = 0.9 86 | temperature = 0.35 # 越大模型越浪 87 | repetition_penalty = 1.2 # 如果模型出现重复说话可以调节该系数 88 | 89 | # 加载模型 90 | model = AutoModelForCausalLM.from_pretrained( 91 | model_name, 92 | trust_remote_code=True, 93 | low_cpu_mem_usage=True, 94 | torch_dtype=torch.float16, 95 | device_map='auto' 96 | ).to(device).eval() 97 | tokenizer = AutoTokenizer.from_pretrained( 98 | model_name, 99 | trust_remote_code=True, 100 | # llama不支持fast 101 | use_fast=False if model.config.model_type == 'llama' else True 102 | ) 103 | # 记录所有历史记录 104 | history_token_ids = tokenizer('', return_tensors="pt").input_ids 105 | 106 | # 开始对话 107 | user_input = input('User：') 108 | while True: 109 | user_input = '{}'.format(user_input) 110 | user_input_ids = tokenizer(user_input, return_tensors="pt", add_special_tokens=False).input_ids 111 | history_token_ids = torch.concat((history_token_ids, user_input_ids), dim=1) 112 | model_input_ids = history_token_ids[:, -history_max_len:].to(device) 113 | with torch.no_grad(): 114 | outputs = model.generate( 115 | input_ids=model_input_ids, max_new_tokens=max_new_tokens, do_sample=True, top_p=top_p, 116 | temperature=temperature, repetition_penalty=repetition_penalty, eos_token_id=tokenizer.eos_token_id 117 | ) 118 | model_input_ids_len = model_input_ids.size(1) 119 | response_ids = outputs[:, model_input_ids_len:] 120 | history_token_ids = torch.concat((history_token_ids, response_ids.cpu()), dim=1) 121 | response = tokenizer.batch_decode(response_ids) 122 | print("Bot：" + response[0].strip().replace('', "")) 123 | user_input = input('User：') 124 | 125 | 126 | if __name__ == '__main__': 127 | main() 128 | 129 | ``` 130 | 推荐继续二次训练以针对性调优对话效果~ 131 | ## Training procedure 132 | 133 | 134 | The following `bitsandbytes` quantization config was used during training: 135 | - load_in_8bit: False 136 | - load_in_4bit: True 137 | - llm_int8_threshold: 6.0 138 | - llm_int8_skip_modules: None 139 | - llm_int8_enable_fp32_cpu_offload: False 140 | - llm_int8_has_fp16_weight: False 141 | - bnb_4bit_quant_type: nf4 142 | - bnb_4bit_use_double_quant: True 143 | - bnb_4bit_compute_dtype: float16 144 | ### Framework versions 145 | 146 | 147 | - PEFT 0.4.0.dev0 148 | 训练1个epoch，loss 0.9，实测用中文对话体验优于baichuan13b(仅主观感受)。还有很大潜力，建议作为底座把文件拉回去继续调优。 149 | 150 | 感谢： 151 | - LLaMA2 152 | - Firefly项目 153 | - shareGPT中文数据集的建设者们 154 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | --------------------------------------------------------------------------------