├── LICENSE
├── README.md
├── chat_templates
├── alpaca.jinja
├── amberchat.jinja
├── chatml.jinja
├── chatqa.jinja
├── falcon-instruct.jinja
├── gemma-it.jinja
├── granite-3.0-instruct.jinja
├── llama-2-chat.jinja
├── llama-3-instruct.jinja
├── mistral-instruct.jinja
├── openchat-3.5.jinja
├── phi-3-small.jinja
├── phi-3.jinja
├── qwen2.5-instruct.jinja
├── saiga.jinja
├── solar-instruct.jinja
├── vicuna.jinja
└── zephyr.jinja
└── generation_configs
├── alpaca.json
├── amberchat.json
├── chatqa.json
├── gemma-it.json
├── granite-3.0-instruct.json
├── llama-2-chat.json
├── llama-3-instruct.json
├── llama-3.1-instruct.json
├── mistral-instruct.json
├── openchat-3.5.json
├── orca-2.json
├── phi-3-small.json
├── phi-3.json
├── qwen2-instruct.json
├── saiga.json
├── solar-instruct.json
├── vicuna.json
├── yi-chat.json
└── zephyr.json
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2024 Chujie Zheng
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # chat_templates
2 |
3 | **NOTICE: I may not update or maintain this repository actively, as I noticed that recent HF LLMs have usually implemented their chat templates in the `tokenizer_config.json` file. I am glad about this common practice and it will definitely facilitate out-of-box use of these powerful models.**
4 |
5 | This is a repository that includes proper chat templates (or input formats) for instruction-tuned large language models (LLMs), to support `transformers`'s `chat_template` [feature](https://huggingface.co/docs/transformers/chat_templating). If you are interested to include more chat templates, feel free to open a pull request
6 |
7 | If you find this repo useful, please kindly cite it:
8 | ```tex
9 | @misc{zheng-2024-chat-templates,
10 | author = {Zheng, Chujie},
11 | title = {Chat Templates for HuggingFace Large Language Models},
12 | year = {2024},
13 | howpublished = {\url{https://github.com/chujiezheng/chat_templates}}
14 | }
15 | ```
16 |
17 | ## Updates
18 |
19 | * **[11/2024]** Added support for Meta's **Llama-3.2** models
20 | * **[10/2024]** Added support for IBM's **Granite-3.0** models
21 | * **[07/2024]** Added support for Meta's **Llama-3.1** models
22 | * **[06/2024]** Added support for Google's **Gemma-2** models
23 | * **[05/2024]** Added support for Nvidia's **ChatQA** models
24 | * **[04/2024]** Added support for Microsoft's **Phi-3** models
25 | * **[04/2024]** Added support for Meta's **Llama-3** models
26 | * **[02/2024]** Added support for Google's **Gemma** models
27 | * **[02/2024]** Added usage explanation for **generation_configs**
28 | * **[01/2024]** Added support for Alibaba's **Qwen2** models
29 |
30 | ## What are Contained in This Repo?
31 |
32 | - [`chat_templates`](/chat_templates/) contains the jinja files of collected chat templates, which can be directly replaced in the Huggingface tokenizers
33 |
34 | - [`generation_configs`](/generation_configs/) contains the corresponding json configs used for controlling the ending of response generations. Specially, **the `stop_token_ids` should be directly passed into the `generate` method by the `eos_token_id` argument**
35 |
36 |
37 | ## Usage Examples
38 |
39 | **Important NOTE:** As mentioned in [this issue](https://github.com/chujiezheng/chat_templates/issues/15), the `messages` should contain **at least one user message**. It is strongly not recommented to pass only the system message, as there may result in unexpected outputs (because the models are not trained in this way).
40 |
41 |
42 |
43 | Example 1: Meta-Llama-3-8B-Instruct
44 |
45 | This example may check if the jinja file is correctly implemented.
46 |
47 | ```python
48 | from transformers import AutoTokenizer
49 |
50 | toker = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", token="YOUR_OWN_TOKEN")
51 | messages = [
52 | {'role': 'system', 'content': 'This is a system prompt.'},
53 | {'role': 'user', 'content': 'This is the first user input.'},
54 | {'role': 'assistant', 'content': 'This is the first assistant response.'},
55 | {'role': 'user', 'content': 'This is the second user input.'},
56 | ]
57 | print('###### Default (yet Correct) Chat Template ######')
58 | print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
59 | print('###### Corrected Chat Template ######')
60 | chat_template = open('./chat_templates/llama-3-instruct.jinja').read()
61 | chat_template = chat_template.replace(' ', '').replace('\n', '')
62 | toker.chat_template = chat_template
63 | print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
64 | ```
65 |
66 | Expected output:
67 |
68 | ```
69 | ###### Default (yet Correct) Chat Template ######
70 | <|begin_of_text|><|start_header_id|>system<|end_header_id|>
71 |
72 | This is a system prompt.<|eot_id|><|start_header_id|>user<|end_header_id|>
73 |
74 | This is the first user input.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
75 |
76 | This is the first assistant response.<|eot_id|><|start_header_id|>user<|end_header_id|>
77 |
78 | This is the second user input.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
79 |
80 |
81 | ###### Corrected Chat Template ######
82 | <|begin_of_text|><|start_header_id|>system<|end_header_id|>
83 |
84 | This is a system prompt.<|eot_id|><|start_header_id|>user<|end_header_id|>
85 |
86 | This is the first user input.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
87 |
88 | This is the first assistant response.<|eot_id|><|start_header_id|>user<|end_header_id|>
89 |
90 | This is the second user input.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
91 | ```
92 |
93 |
94 |
95 |
96 |
97 | Example 2: Mistral-7B-Instruct-v0.2
98 |
99 | For `mistral-instruct` (also `gemma-it`), it does not natively support the `system` message, so passing the `system` message would raise error.
100 |
101 | ```python
102 | from transformers import AutoTokenizer
103 |
104 | toker = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
105 | messages = [
106 | {'role': 'system', 'content': 'This is a system prompt.'},
107 | {'role': 'user', 'content': 'This is the first user input.'},
108 | {'role': 'assistant', 'content': 'This is the first assistant response.'},
109 | {'role': 'user', 'content': 'This is the second user input.'},
110 | ]
111 | print('###### Default (but Improper) Chat Template ######')
112 | # raising error
113 | #print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
114 | print('###### Corrected Chat Template ######')
115 | chat_template = open('./chat_templates/mistral-instruct.jinja').read()
116 | chat_template = chat_template.replace(' ', '').replace('\n', '')
117 | toker.chat_template = chat_template
118 | print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
119 | ```
120 |
121 | Expected output:
122 |
123 | ```
124 | ###### Default (but Error-Raising) Chat Template ######
125 | jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/...
126 | ###### Corrected Chat Template ######
127 | [INST] This is a system prompt.
128 |
129 | This is the first user input. [/INST] This is the first assistant response. [INST] This is the second user input. [/INST]
130 | ```
131 |
132 |
133 |
134 |
135 |
136 | Example 3: vicuna-7b-v1.5
137 |
138 | **NOTE:** In [fast-chat](https://github.com/lm-sys/FastChat/blob/d578599c69d060e6d40943f1b5b72af98956092a/fastchat/conversation.py#L287C3-L287C3), `vicuna` does not add linebreaks between roles' messages. But I found that adding linebreaks leads to a bit better performance (especially for the v1.5 version).
139 |
140 | Also, I found `vicuna-7/13/33b-v1.3` may not work well when given a system message different from its default one. So I would recommend to use `vicuna-7/13b-v1.5` instead.
141 |
142 | ```python
143 | from transformers import AutoTokenizer
144 |
145 | toker = AutoTokenizer.from_pretrained("lmsys/vicuna-7b-v1.5")
146 | messages = [
147 | {'role': 'system', 'content': 'This is a system prompt.'},
148 | {'role': 'user', 'content': 'This is the first user input.'},
149 | {'role': 'assistant', 'content': 'This is the first assistant response.'},
150 | {'role': 'user', 'content': 'This is the second user input.'},
151 | ]
152 | print('###### Default (but Improper) Chat Template ######')
153 | print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
154 | print('###### Corrected Chat Template ######')
155 | chat_template = open('./chat_templates/vicuna.jinja').read()
156 | chat_template = chat_template.replace(' ', '').replace('\n', '')
157 | toker.chat_template = chat_template
158 | print(toker.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
159 | ```
160 |
161 | Expected output:
162 |
163 | ```
164 | ###### Default (but Improper) Chat Template ######
165 | [INST] <>
166 | This is a system prompt.
167 | <>
168 |
169 | This is the first user input. [/INST] This is the first assistant response. [INST] This is the second user input. [/INST]
170 | ###### Corrected Chat Template ######
171 | This is a system prompt.
172 |
173 | USER: This is the first user input.
174 | ASSISTANT: This is the first assistant response.
175 | USER: This is the second user input.
176 | ASSISTANT:
177 | ```
178 |
179 |
180 |
181 |
182 | ## Supported Models
183 |
184 | **NOTE:** The listed models are not inclusive and also include other-sized ones in the same model family
185 |
186 |
187 | granite-3.0-instruct
188 |
189 | - Models: `ibm-granite/granite-3.0-2b-instruct`, `ibm-granite/granite-3.0-8b-instruct`, `ibm-granite/granite-3.0-1b-a400m-instruct`, `ibm-granite/granite-3.0-3b-a800m-instruct`
190 | - Chat template: `chat_templates/granite-3.0-instruct.jinja`
191 | - Generation config: `generation_configs/granite-3.0-instruct.json`
192 | - Reference: https://huggingface.co/ibm-granite/granite-3.0-8b-instruct/blob/main/tokenizer_config.json#L188
193 |
194 |
195 |
196 |
197 |
198 | Llama-3.2-Instruct
199 |
200 | - Models: `meta-llama/Meta-Llama-3.2-1B-Instruct`, `meta-llama/Meta-Llama-3.2-3B-Instruct`
201 | - Chat template: `chat_templates/llama-3-instruct.jinja`
202 | - Generation config: `generation_configs/llama-3.1-instruct.json`
203 | - Reference: https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct/blob/main/tokenizer_config.json#L2053
204 |
205 |
206 |
207 |
208 |
209 | Llama-3.1-Instruct
210 |
211 | - Models: `meta-llama/Meta-Llama-3.1-8B-Instruct`, `meta-llama/Meta-Llama-3.1-405B-Instruct-FP8`
212 | - Chat template: `chat_templates/llama-3-instruct.jinja`
213 | - Generation config: `generation_configs/llama-3.1-instruct.json`
214 | - Reference: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/tokenizer_config.json#L2053
215 |
216 |
217 |
218 |
219 |
220 | Llama-3-Instruct
221 |
222 | - Models: `meta-llama/Meta-Llama-3-8B-Instruct`
223 | - Chat template: `chat_templates/llama-3-instruct.jinja`
224 | - Generation config: `generation_configs/llama-3-instruct.json`
225 | - Reference: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/blob/main/tokenizer_config.json#L2053
226 |
227 |
228 |
229 |
230 |
231 | Llama-2-Chat, CodeLlama-Instruct
232 |
233 | - Models: `meta-llama/Llama-2-7b-chat-hf`, `meta-llama/CodeLlama-7b-Instruct-hf`
234 | - Chat template: `chat_templates/llama-2-chat.jinja`
235 | - Generation config: `generation_configs/llama-2-chat.json`
236 | - Reference: https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/blob/main/tokenizer_config.json#L12
237 |
238 |
239 |
240 |
241 |
242 | Qwen2-Instruct, Qwen1.5-Chat
243 |
244 | - Models: `Qwen/Qwen2-7B-Instruct`, `Qwen/Qwen1.5-7B-Chat`
245 | - Chat template: `chat_templates/chatml.jinja`
246 | - Generation config: `generation_configs/qwen2-instruct.json`
247 | - Reference: https://huggingface.co/Qwen/Qwen2-72B-Instruct/blob/main/tokenizer_config.json#L31
248 |
249 |
250 |
251 |
252 |
253 | Mistral-Instruct
254 |
255 | - Models: `mistralai/Mistral-7B-Instruct-v0.3`, `mistralai/Mixtral-8x7B-Instruct-v0.1`
256 | - Chat template: `chat_templates/mistral-instruct.jinja`
257 | - Generation config: `generation_configs/mistral-instruct.json`
258 | - Reference: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/blob/main/tokenizer_config.json#L42
259 | - Comment: **System message is acceptable** by prepending it before the first user input
260 |
261 |
262 |
263 |
264 |
265 | Phi-3-Instruct
266 |
267 | - Models: `microsoft/Phi-3-mini-4k-instruct`, `microsoft/Phi-3-small-8k-instruct`
268 | - Chat template: `chat_templates/phi-3.jinja`, `chat_templates/phi-3-small.jinja`
269 | - Generation config: `generation_configs/phi-3.json`, `generation_configs/phi-3-small.json`
270 | - Reference: https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/blob/main/tokenizer_config.json#L338
271 | - **Note:** `Phi-3-mini/medium` and `Phi-3-small` adopt different configs
272 |
273 |
274 |
275 |
276 |
277 | Yi-1.5-Chat, Yi-Chat
278 |
279 | - Models: `01-ai/Yi-1.5-6B-Chat`, `01-ai/Yi-6B-Chat`
280 | - Chat template: `chat_templates/chatml.jinja`
281 | - Generation config: `generation_configs/yi-chat.json`
282 | - Reference: https://huggingface.co/01-ai/Yi-6B-Chat/blob/main/tokenizer_config.json#L60
283 |
284 |
285 |
286 |
287 |
288 | gemma-it, gemma-2-it
289 |
290 | - Models: `google/gemma-7b-it`, `google/gemma-2-9b-it`
291 | - Chat template: `chat_templates/gemma-it.jinja`
292 | - Generation config: `generation_configs/gemma-it.json`
293 | - Reference: https://huggingface.co/google/gemma-7b-it/blob/main/tokenizer_config.json#L1507
294 | - Comment: **System message is acceptable**
295 |
296 |
297 |
298 |
299 |
300 | Llama3-ChatQA-1.5
301 |
302 | - Models: `nvidia/Llama3-ChatQA-1.5-8B`
303 | - Chat template: `chat_templates/chatqa.jinja`
304 | - Generation config: `generation_configs/chatqa.json`
305 | - Reference: https://huggingface.co/nvidia/Llama3-ChatQA-1.5-8B#when-context-is-available
306 | - Comment: Context message is acceptable
307 |
308 |
309 |
310 |
311 |
312 | openchat-3.5, Starling-LM
313 |
314 | - Models: `openchat/openchat_3.5`, `berkeley-nest/Starling-LM-7B-alpha`
315 | - Chat template: `chat_templates/openchat-3.5.jinja`
316 | - Generation config: `generation_configs/openchat-3.5.json`
317 | - Reference: https://huggingface.co/openchat/openchat_3.5/blob/main/tokenizer_config.json#L51
318 |
319 |
320 |
321 |
322 |
323 | zephyr
324 |
325 | - Models: `zephyr-7b-alpha`
326 | - Chat template: `chat_templates/zephyr.jinja`
327 | - Generation config: `generation_configs/zephyr.json`
328 | - Reference: https://huggingface.co/HuggingFaceH4/zephyr-7b-beta/blob/main/tokenizer_config.json#L34
329 |
330 |
331 |
332 |
333 |
334 | vicuna
335 |
336 | - Models: `vicuna-7b-v1.5`, `vicuna-7b-v1.3`
337 | - Chat template: `chat_templates/vicuna.jinja`
338 | - Generation config: `generation_configs/vicuna.json`
339 | - Reference: https://github.com/lm-sys/FastChat/blob/main/docs/vicuna_weights_version.md#prompt-template
340 |
341 |
342 |
343 |
344 |
345 | Orca-2
346 |
347 | - Models: `microsoft/Orca-2-7b`
348 | - Chat template: `chat_templates/chatml.jinja`
349 | - Generation config: `generation_configs/orca-2.json`
350 | - Reference: https://huggingface.co/microsoft/Orca-2-7b
351 |
352 |
353 |
354 |
355 |
356 | falcon-instruct
357 |
358 | - Models: `tiiuae/falcon-7b-instruct`
359 | - Chat template: `chat_templates/falcon-instruct.jinja`
360 | - Reference: https://github.com/lm-sys/FastChat/blob/main/docs/vicuna_weights_version.md#prompt-template
361 |
362 |
363 |
364 |
365 |
366 | SOLAR-Instruct
367 |
368 | - Models: `upstage/SOLAR-10.7B-Instruct-v1.0`
369 | - Chat template: `chat_templates/solar-instruct.jinja`
370 | - Generation config: `generation_configs/solar-instruct.json`
371 | - Reference: https://huggingface.co/upstage/SOLAR-10.7B-Instruct-v1.0/blob/main/tokenizer_config.json#L31
372 |
373 |
374 |
375 |
376 |
377 | Alpaca
378 |
379 | - Models: `tatsu-lab/alpaca-7b-wdiff`
380 | - Chat template: `chat_templates/alpaca.jinja`
381 | - Generation config: `generation_configs/alpaca.json`
382 | - Reference: https://github.com/tatsu-lab/stanford_alpaca
383 |
384 |
385 |
386 |
387 |
388 | AmberChat
389 |
390 | - Models: `LLM360/AmberChat`, `LLM360/AmberSafe`
391 | - Chat template: `chat_templates/amberchat.jinja`
392 | - Generation config: `generation_configs/amberchat.json`
393 | - Reference: https://huggingface.co/LLM360/AmberChat
394 |
395 |
396 |
397 |
398 |
399 | saiga
400 |
401 | - Models: `IlyaGusev/saiga_mistral_7b_lora`
402 | - Chat template: `chat_templates/saiga.jinja`
403 | - Generation config: `generation_configs/saiga.json`
404 | - Reference: https://huggingface.co/IlyaGusev/saiga_mistral_7b_lora#saigamistral-7b-russian-mistral-based-chatbot
405 | - Comment: A series of Russian models
406 |
407 |
408 |
409 | ## Star History
410 |
411 | [](https://star-history.com/#chujiezheng/chat_templates&Date)
412 |
--------------------------------------------------------------------------------
/chat_templates/alpaca.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set system_message = messages[0]['content'] | trim + '\n\n' %}
3 | {% set messages = messages[1:] %}
4 | {% else %}
5 | {% set system_message = '' %}
6 | {% endif %}
7 |
8 | {{ bos_token + system_message }}
9 | {% for message in messages %}
10 | {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
11 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
12 | {% endif %}
13 |
14 | {% if message['role'] == 'user' %}
15 | {{ '### Instruction:\n' + message['content'] | trim + '\n\n' }}
16 | {% elif message['role'] == 'assistant' %}
17 | {{ '### Response:\n' + message['content'] | trim + eos_token + '\n\n' }}
18 | {% endif %}
19 | {% endfor %}
20 |
21 | {% if add_generation_prompt %}
22 | {{ '### Response:\n' }}
23 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/amberchat.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set system_message = messages[0]['content'] | trim + '\n' %}
3 | {% set messages = messages[1:] %}
4 | {% else %}
5 | {% set system_message = '' %}
6 | {% endif %}
7 |
8 | {{ bos_token + system_message }}
9 | {% for message in messages %}
10 | {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
11 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
12 | {% endif %}
13 |
14 | {% if message['role'] == 'user' %}
15 | {{ '###Human: ' + message['content'] | trim + '\n' }}
16 | {% elif message['role'] == 'assistant' %}
17 | {{ '###Assistant: ' + message['content'] | trim + '\n' }}
18 | {% endif %}
19 | {% endfor %}
20 |
21 | {% if add_generation_prompt %}
22 | {{ '###Assistant:' }}
23 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/chatml.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set offset = 1 %}
3 | {% else %}
4 | {% set offset = 0 %}
5 | {% endif %}
6 |
7 | {{ bos_token }}
8 | {% for message in messages %}
9 | {% if (message['role'] == 'user') != (loop.index0 % 2 == offset) %}
10 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
11 | {% endif %}
12 |
13 | {{ '<|im_start|>' + message['role'] + '\n' + message['content'] | trim + '<|im_end|>\n' }}
14 | {% endfor %}
15 |
16 | {% if add_generation_prompt %}
17 | {{ '<|im_start|>assistant\n' }}
18 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/chatqa.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set system_message = 'System: ' + messages[0]['content'] | trim %}
3 | {% set messages = messages[1:] %}
4 | {% else %}
5 | {% set system_message = '' %}
6 | {% endif %}
7 |
8 | {% if messages[0]['role'] == 'context' %}
9 | {% set context_message = '\n\n' + messages[0]['content'] | trim %}
10 | {% set messages = messages[1:] %}
11 | {% else %}
12 | {% set context_message = '' %}
13 | {% endif %}
14 |
15 | {{ bos_token + system_message + context_message}}
16 | {% for message in messages %}
17 | {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
18 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
19 | {% endif %}
20 |
21 | {% if message['role'] == 'user' %}
22 | {{ '\n\nUser: ' + message['content'] | trim }}
23 | {% elif message['role'] == 'assistant' %}
24 | {{ '\n\nAssistant: ' + message['content'] | trim }}
25 | {% endif %}
26 | {% endfor %}
27 |
28 | {% if add_generation_prompt %}
29 | {{ '\n\nAssistant:' }}
30 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/falcon-instruct.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set system_message = messages[0]['content'] %}
3 | {% set messages = messages[1:] %}
4 | {% else %}
5 | {% set system_message = '' %}
6 | {% endif %}
7 |
8 | {{ system_message | trim }}
9 | {% for message in messages %}
10 | {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
11 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
12 | {% endif %}
13 |
14 | {% set content = message['content'].replace('\r\n', '\n').replace('\n\n', '\n') %}
15 | {{ '\n\n' + message['role'] | capitalize + ': ' + content | trim }}
16 | {% endfor %}
17 |
18 | {% if add_generation_prompt %}
19 | {{ '\n\nAssistant:' }}
20 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/gemma-it.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set system_message = messages[0]['content'] | trim + '\n\n' %}
3 | {% set messages = messages[1:] %}
4 | {% else %}
5 | {% set system_message = '' %}
6 | {% endif %}
7 |
8 | {% for message in messages %}
9 | {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
10 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
11 | {% endif %}
12 |
13 | {% if loop.index0 == 0 %}
14 | {% set content = system_message + message['content'] %}
15 | {% else %}
16 | {% set content = message['content'] %}
17 | {% endif %}
18 |
19 | {% if (message['role'] == 'assistant') %}
20 | {% set role = 'model' %}
21 | {% else %}
22 | {% set role = message['role'] %}
23 | {% endif %}
24 |
25 | {{ '' + role + '\n' + content | trim + '\n' }}
26 | {% endfor %}
27 |
28 | {% if add_generation_prompt %}
29 | {{'model\n'}}
30 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/granite-3.0-instruct.jinja:
--------------------------------------------------------------------------------
1 | {%- if tools %}
2 | {{- '<|start_of_role|>available_tools<|end_of_role|>\n' }}
3 | {%- for tool in tools %}
4 | {{- tool | tojson(indent=4) }}
5 | {%- if not loop.last %}
6 | {{- '\n\n' }}
7 | {%- endif %}
8 | {%- endfor %}
9 | {{- '<|end_of_text|>\n' }}
10 | {%- endif %}
11 |
12 | {%- for message in messages %}
13 | {%- if message['role'] == 'system' %}
14 | {{- '<|start_of_role|>system<|end_of_role|>' + message['content'] + '<|end_of_text|>\n' }}
15 | {%- elif message['role'] == 'user' %}
16 | {{- '<|start_of_role|>user<|end_of_role|>' + message['content'] + '<|end_of_text|>\n' }}
17 | {%- elif message['role'] == 'assistant' %}
18 | {{- '<|start_of_role|>assistant<|end_of_role|>' + message['content'] + '<|end_of_text|>\n' }}
19 | {%- elif message['role'] == 'assistant_tool_call' %}
20 | {{- '<|start_of_role|>assistant<|end_of_role|><|tool_call|>' + message['content'] + '<|end_of_text|>\n' }}
21 | {%- elif message['role'] == 'tool_response' %}
22 | {{- '<|start_of_role|>tool_response<|end_of_role|>' + message['content'] + '<|end_of_text|>\n' }}
23 | {%- endif %}
24 |
25 | {%- if loop.last and add_generation_prompt %}
26 | {{- '<|start_of_role|>assistant<|end_of_role|>' }}
27 | {%- endif %}
28 | {%- endfor %}
--------------------------------------------------------------------------------
/chat_templates/llama-2-chat.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set system_message = '<>\n' + messages[0]['content'] | trim + '\n<>\n\n' %}
3 | {% set messages = messages[1:] %}
4 | {% else %}
5 | {% set system_message = '' %}
6 | {% endif %}
7 |
8 | {% for message in messages %}
9 | {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
10 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
11 | {% endif %}
12 |
13 | {% if loop.index0 == 0 %}
14 | {% set content = system_message + message['content'] %}
15 | {% else %}
16 | {% set content = message['content'] %}
17 | {% endif %}
18 |
19 | {% if message['role'] == 'user' %}
20 | {{ bos_token + '[INST] ' + content | trim + ' [/INST]' }}
21 | {% elif message['role'] == 'assistant' %}
22 | {{ ' ' + content | trim + ' ' + eos_token }}
23 | {% endif %}
24 | {% endfor %}
--------------------------------------------------------------------------------
/chat_templates/llama-3-instruct.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set offset = 1 %}
3 | {% else %}
4 | {% set offset = 0 %}
5 | {% endif %}
6 |
7 | {{ bos_token }}
8 | {% for message in messages %}
9 | {% if (message['role'] == 'user') != (loop.index0 % 2 == offset) %}
10 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
11 | {% endif %}
12 |
13 | {{ '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' + message['content'] | trim + '<|eot_id|>' }}
14 | {% endfor %}
15 |
16 | {% if add_generation_prompt %}
17 | {{ '<|start_header_id|>' + 'assistant' + '<|end_header_id|>\n\n' }}
18 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/mistral-instruct.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set system_message = messages[0]['content'] | trim + '\n\n' %}
3 | {% set messages = messages[1:] %}
4 | {% else %}
5 | {% set system_message = '' %}
6 | {% endif %}
7 |
8 | {{ bos_token + system_message}}
9 | {% for message in messages %}
10 | {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
11 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
12 | {% endif %}
13 |
14 | {% if message['role'] == 'user' %}
15 | {{ '[INST] ' + message['content'] | trim + ' [/INST]' }}
16 | {% elif message['role'] == 'assistant' %}
17 | {{ ' ' + message['content'] | trim + eos_token }}
18 | {% endif %}
19 | {% endfor %}
--------------------------------------------------------------------------------
/chat_templates/openchat-3.5.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set system_message = messages[0]['content'] | trim + '<|end_of_turn|>' %}
3 | {% set messages = messages[1:] %}
4 | {% else %}
5 | {% set system_message = '' %}
6 | {% endif %}
7 |
8 | {{ bos_token + system_message }}
9 | {% for message in messages %}
10 | {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
11 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
12 | {% endif %}
13 |
14 | {{ 'GPT4 Correct ' + message['role'] | capitalize + ': ' + message['content'] + '<|end_of_turn|>' }}
15 | {% endfor %}
16 |
17 | {% if add_generation_prompt %}
18 | {{ 'GPT4 Correct Assistant:' }}
19 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/phi-3-small.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set offset = 1 %}
3 | {% else %}
4 | {% set offset = 0 %}
5 | {% endif %}
6 |
7 | {{ bos_token }}
8 | {% for message in messages %}
9 | {% if (message['role'] == 'user') != (loop.index0 % 2 == offset) %}
10 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
11 | {% endif %}
12 |
13 | {{ '<|' + message['role'] + '|>\n' + message['content'] | trim + '<|end|>' + '\n' }}
14 | {% endfor %}
15 |
16 | {% if add_generation_prompt %}
17 | {{ '<|assistant|>\n' }}
18 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/phi-3.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set offset = 1 %}
3 | {% else %}
4 | {% set offset = 0 %}
5 | {% endif %}
6 |
7 | {% for message in messages %}
8 | {% if (message['role'] == 'user') != (loop.index0 % 2 == offset) %}
9 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
10 | {% endif %}
11 |
12 | {{ '<|' + message['role'] + '|>\n' + message['content'] | trim + '<|end|>' + '\n' }}
13 | {% endfor %}
14 |
15 | {% if add_generation_prompt %}
16 | {{ '<|assistant|>\n' }}
17 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/qwen2.5-instruct.jinja:
--------------------------------------------------------------------------------
1 | {%- if tools %}
2 | {{- '<|im_start|>system
3 | ' }}
4 | {%- if messages[0]['role'] == 'system' %}
5 | {{- messages[0]['content'] }}
6 | {%- else %}
7 | {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
8 | {%- endif %}
9 | {{- "
10 |
11 | # Tools
12 |
13 | You may call one or more functions to assist with the user query.
14 |
15 | You are provided with function signatures within XML tags:
16 | " }}
17 | {%- for tool in tools %}
18 | {{- "
19 | " }}
20 | {{- tool | tojson }}
21 | {%- endfor %}
22 | {{- "
23 |
24 |
25 | For each function call, return a json object with function name and arguments within XML tags:
26 |
27 | {\"name\": , \"arguments\": }
28 | <|im_end|>
29 | " }}
30 | {%- else %}
31 | {%- if messages[0]['role'] == 'system' %}
32 | {{- '<|im_start|>system
33 | ' + messages[0]['content'] + '<|im_end|>
34 | ' }}
35 | {%- else %}
36 | {{- '<|im_start|>system
37 | You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
38 | ' }}
39 | {%- endif %}
40 | {%- endif %}
41 | {%- for message in messages %}
42 | {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
43 | {{- '<|im_start|>' + message.role + '
44 | ' + message.content + '<|im_end|>' + '
45 | ' }}
46 | {%- elif message.role == "assistant" %}
47 | {{- '<|im_start|>' + message.role }}
48 | {%- if message.content %}
49 | {{- '
50 | ' + message.content }}
51 | {%- endif %}
52 | {%- for tool_call in message.tool_calls %}
53 | {%- if tool_call.function is defined %}
54 | {%- set tool_call = tool_call.function %}
55 | {%- endif %}
56 | {{- '
57 |
58 | {"name": "' }}
59 | {{- tool_call.name }}
60 | {{- '", "arguments": ' }}
61 | {{- tool_call.arguments | tojson }}
62 | {{- '}
63 | ' }}
64 | {%- endfor %}
65 | {{- '<|im_end|>
66 | ' }}
67 | {%- elif message.role == "tool" %}
68 | {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
69 | {{- '<|im_start|>user' }}
70 | {%- endif %}
71 | {{- '
72 |
73 | ' }}
74 | {{- message.content }}
75 | {{- '
76 | ' }}
77 | {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
78 | {{- '<|im_end|>
79 | ' }}
80 | {%- endif %}
81 | {%- endif %}
82 | {%- endfor %}
83 | {%- if add_generation_prompt %}
84 | {{- '<|im_start|>assistant
85 | ' }}
86 | {%- endif %}
--------------------------------------------------------------------------------
/chat_templates/saiga.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set offset = 1 %}
3 | {% else %}
4 | {% set offset = 0 %}
5 | {% endif %}
6 |
7 | {% for message in messages %}
8 | {% if (message['role'] == 'user') != (loop.index0 % 2 == offset) %}
9 | {{ raise_exception('Conversation roles must alternate user/bot/user/bot/...') }}
10 | {% endif %}
11 |
12 | {% if message['role'] == 'assistant' %}
13 | {% set role = 'bot' %}
14 | {% else %}
15 | {% set role = message['role'] %}
16 | {% endif %}
17 |
18 | {{ bos_token + role + '\n' + message['content'] | trim + eos_token }}
19 | {% endfor %}
20 |
21 | {% if add_generation_prompt %}
22 | {{ bos_token + 'bot\n' }}
23 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/solar-instruct.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set offset = 1 %}
3 | {% else %}
4 | {% set offset = 0 %}
5 | {% endif %}
6 |
7 | {{ bos_token }}
8 | {% for message in messages %}
9 | {% if (message['role'] == 'user') != (loop.index0 % 2 == offset) %}
10 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
11 | {% endif %}
12 |
13 | {{ '### ' + message['role'] | capitalize + ':\n' + message['content'] | trim + '\n\n' }}
14 | {% endfor %}
15 |
16 | {% if add_generation_prompt %}
17 | {{ '### Assistant:\n' }}
18 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/vicuna.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set system_message = messages[0]['content'] | trim + '\n\n' %}
3 | {% set messages = messages[1:] %}
4 | {% else %}
5 | {% set system_message = '' %}
6 | {% endif %}
7 |
8 | {{ bos_token + system_message }}
9 | {% for message in messages %}
10 | {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
11 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
12 | {% endif %}
13 |
14 | {% if message['role'] == 'user' %}
15 | {{ 'USER: ' + message['content'] | trim + '\n' }}
16 | {% elif message['role'] == 'assistant' %}
17 | {{ 'ASSISTANT: ' + message['content'] | trim + eos_token + '\n' }}
18 | {% endif %}
19 | {% endfor %}
20 |
21 | {% if add_generation_prompt %}
22 | {{ 'ASSISTANT:' }}
23 | {% endif %}
--------------------------------------------------------------------------------
/chat_templates/zephyr.jinja:
--------------------------------------------------------------------------------
1 | {% if messages[0]['role'] == 'system' %}
2 | {% set offset = 1 %}
3 | {% else %}
4 | {% set offset = 0 %}
5 | {% endif %}
6 |
7 | {% for message in messages %}
8 | {% if (message['role'] == 'user') != (loop.index0 % 2 == offset) %}
9 | {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
10 | {% endif %}
11 |
12 | {{ '<|' + message['role'] + '|>\n' + message['content'] | trim + eos_token + '\n' }}
13 | {% endfor %}
14 |
15 | {% if add_generation_prompt %}
16 | {{ '<|assistant|>\n' }}
17 | {% endif %}
--------------------------------------------------------------------------------
/generation_configs/alpaca.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/alpaca.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [2],
5 | "system_prompt": "Below is an instruction that describes a task. Write a response that appropriately completes the request."
6 | }
--------------------------------------------------------------------------------
/generation_configs/amberchat.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/amberchat.jinja",
3 | "stop_str": "\n###Human",
4 | "stop_token_ids": [2],
5 | "system_prompt": null
6 | }
--------------------------------------------------------------------------------
/generation_configs/chatqa.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/chatqa.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [128001, 128009],
5 | "system_prompt": "This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found in the context."
6 | }
--------------------------------------------------------------------------------
/generation_configs/gemma-it.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/gemma-it.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [1, 107],
5 | "system_prompt": null
6 | }
--------------------------------------------------------------------------------
/generation_configs/granite-3.0-instruct.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/granite-3.0-instruct.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [0, 49153],
5 | "system_prompt": null
6 | }
--------------------------------------------------------------------------------
/generation_configs/llama-2-chat.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/llama-2-chat.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [2],
5 | "system_prompt": "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information."
6 | }
--------------------------------------------------------------------------------
/generation_configs/llama-3-instruct.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/llama-3-instruct.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [128001, 128009],
5 | "system_prompt": null
6 | }
--------------------------------------------------------------------------------
/generation_configs/llama-3.1-instruct.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/llama-3-instruct.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [128001, 128008, 128009],
5 | "system_prompt": null
6 | }
--------------------------------------------------------------------------------
/generation_configs/mistral-instruct.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/mistral-instruct.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [2],
5 | "system_prompt": "Always assist with care, respect, and truth. Respond with utmost utility yet securely. Avoid harmful, unethical, prejudiced, or negative content. Ensure replies promote fairness and positivity."
6 | }
--------------------------------------------------------------------------------
/generation_configs/openchat-3.5.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/openchat-3.5.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [2, 32000],
5 | "system_prompt": null
6 | }
--------------------------------------------------------------------------------
/generation_configs/orca-2.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/chatml.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [2, 32002],
5 | "system_prompt": "You are Orca, an AI language model created by Microsoft. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior."
6 | }
--------------------------------------------------------------------------------
/generation_configs/phi-3-small.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/phi-3-small.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [100257, 100266],
5 | "system_prompt": null
6 | }
--------------------------------------------------------------------------------
/generation_configs/phi-3.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/phi-3.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [2, 32000, 32001, 32007],
5 | "system_prompt": null
6 | }
--------------------------------------------------------------------------------
/generation_configs/qwen2-instruct.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/chatml.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [151643, 151645],
5 | "system_prompt": null
6 | }
--------------------------------------------------------------------------------
/generation_configs/saiga.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/saiga.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [2],
5 | "system_prompt": "Ты — Сайга, русскоязычный автоматический ассистент. Ты разговариваешь с людьми и помогаешь им."
6 | }
7 |
--------------------------------------------------------------------------------
/generation_configs/solar-instruct.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/solar-instruct.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [2],
5 | "system_prompt": null
6 | }
--------------------------------------------------------------------------------
/generation_configs/vicuna.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/vicuna.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [2],
5 | "system_prompt": "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions."
6 | }
--------------------------------------------------------------------------------
/generation_configs/yi-chat.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/chatml.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [2, 7],
5 | "system_prompt": null
6 | }
--------------------------------------------------------------------------------
/generation_configs/zephyr.json:
--------------------------------------------------------------------------------
1 | {
2 | "chat_template": "chat_templates/zephyr.jinja",
3 | "stop_str": null,
4 | "stop_token_ids": [2],
5 | "system_prompt": null
6 | }
--------------------------------------------------------------------------------