├── attach └── mllm.png ├── readinglist ├── data.md ├── security.md └── inference.md ├── LICENSE └── README.md /attach/mllm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kaiyuhwang/MLLM-Survey/HEAD/attach/mllm.png -------------------------------------------------------------------------------- /readinglist/data.md: -------------------------------------------------------------------------------- 1 | # Data Resource and Benchmarks 2 | 3 | ## Data Resource 4 | * Massive Text Embedding Benchmark (MTEB). 2022. Amazon intent. [Link](https://huggingface.co/datasets/mteb/amazon_massive_intent) 5 | ## Benchmark 6 | * Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, Nedjma Ousidhoum, David Ifeoluwa Adelani, Seid Muhie Yimam, Ibrahim Sa'id Ahmad, Meriem Beloucif, Saif M. Mohammad, Sebastian Ruder, Oumaima Hourrane, Pavel Brazdil, Felermino Dário Mário António Ali, Davis David, Salomey Osei, Bello Shehu Bello, Falalu Ibrahim, Tajuddeen Gwadabe, Samuel Rutunda, Tadesse Belay, Wendimu Baye Messelle, Hailu Beshada Balcha, Sisay Adugna Chala, Hagos Tesfahun Gebremichael, Bernard Opoku, Steven Arthur. 2023. [AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages](https://arxiv.org/abs/2302.08956). *arXiv:2302.08956*. 7 | 8 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2024, Kaiyu Huang 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | 8 | 1. Redistributions of source code must retain the above copyright notice, this 9 | list of conditions and the following disclaimer. 10 | 11 | 2. Redistributions in binary form must reproduce the above copyright notice, 12 | this list of conditions and the following disclaimer in the documentation 13 | and/or other materials provided with the distribution. 14 | 15 | 3. Neither the name of the copyright holder nor the names of its 16 | contributors may be used to endorse or promote products derived from 17 | this software without specific prior written permission. 18 | 19 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 20 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 21 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 22 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 23 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 24 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 25 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 26 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 27 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 28 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 29 | -------------------------------------------------------------------------------- /readinglist/security.md: -------------------------------------------------------------------------------- 1 | # Security 2 | 3 | ## Attack 4 | ### Greedy Coordinate Gradient Series Attack Methods 5 | * Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J.Zico Kolter, Matt Fredrikon. 2023. [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://arxiv.org/abs/2307.15043). *arXiv:2307.15043*. 6 | * Chawin Sitawarin, Norman Mu, David Wagner, Alexandre Araujo. 2024. [PAL: Proxy-Guided Black-Box Attack on Large Language Models](https://arxiv.org/abs/2402.09674). *arXiv:2402.09674*. 7 | 8 | ### Prompt-based Attack Methods 9 | * Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024. [Jailbroken: How does llm safety training fail?](https://proceedings.neurips.cc/paper_files/paper/2023/file/fd6613131889a4b656206c50a8bd7790-Paper-Conference.pdf) *Advances in Neural Information Processing Systems, 36*. 10 | * Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023. [Jailbreaking chatgpt via prompt engineering: An empirical study.](https://arxiv.org/pdf/2305.13860). *arXiv:2305.13860*. 11 | * Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. ["do anything now": 12 | Characterizing and evaluating in-the-wild jailbreak prompts on large language models](https://arxiv.org/pdf/2308.03825). *arXiv:2308.03825.* 13 | * Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2024. [Masterkey: Automated jailbreaking of large language model 14 | chatbots.](https://www.ndss-symposium.org/wp-content/uploads/2024-188-paper.pdf) *In Proc. ISOC NDSS*. 15 | * Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, 16 | and Jing Shao. 2024. [Salad-bench: A hierarchical and comprehensive safety benchmark for large 17 | language models](https://arxiv.org/pdf/2402.05044). *arXiv:2402.05044*. 18 | * Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023. [Autodan: Generating stealthy 19 | jailbreak prompts on aligned large language models.](https://arxiv.org/pdf/2310.04451) *arXiv:2310.04451*. 20 | * Haibo Jin, Ruoxi Chen, Andy Zhou, Jinyin Chen, Yang Zhang, and Haohan Wang. 2024. [Guard: 21 | Role-playing to generate natural-language jailbreakings to test guideline adherence of large 22 | language models.](https://arxiv.org/pdf/2402.03299) *arXiv:2402.03299*. 23 | 24 | ### Multilingual Attack Methods 25 | * Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan 26 | Zheng, Philipp Koehn, and Daniel Khashabi. 2024. [The language barrier: Dissecting safety challenges of llms in multilingual contexts](https://arxiv.org/pdf/2401.13136). 27 | * Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023. [Multilingual jailbreak challenges in large language models](https://arxiv.org/pdf/2310.06474). *arXiv:2310.06474*. 28 | * Poorna Chander Reddy Puttaparthi, Soham Sanjay Deo, Hakan Gul, Yiming Tang, Weiyi 29 | Shang, and Zhe Yu. 2023. [Comprehensive evaluation of chatgpt reliability through multilingual 30 | inquiries](https://arxiv.org/pdf/2312.10524). 31 | * Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023. [Low-resource languages jailbreak 32 | gpt-4](https://arxiv.org/pdf/2310.02446). *arXiv:2310.02446*. 33 | * Nan Xu, Fei Wang, Ben Zhou, Bang Zheng Li, Chaowei Xiao, and Muhao Chen. 2023. [Cognitive overload: Jailbreaking large language models with overloaded logical thinking](https://arxiv.org/pdf/2311.09827). *arXiv:2311.09827*. 34 | * Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, and 35 | Yinxing Xue. 2024. [A cross-language investigation into jailbreak attacks in large language models.](https://arxiv.org/pdf/2401.16765). 36 | *arXiv:2401.16765*. 37 | * Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and 38 | Zhaopeng Tu. 2024. [Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher](https://arxiv.org/pdf/2308.06463). 39 | * Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. 2023. [Catastrophic 40 | jailbreak of open-source llms via exploiting generation](https://arxiv.org/pdf/2310.06987). *arXiv:2310.06987*. 41 | 42 | 43 | 44 | ## Defense 45 | We are investigating more defense methods... 46 | ### For Open-Source Models 47 | * Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023. [Multilingual jailbreak challenges in large language models](https://arxiv.org/pdf/2310.06474). *arXiv:2310.06474*. 48 | * Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, and 49 | Yinxing Xue. 2024. [A cross-language investigation into jailbreak attacks in large language models.](https://arxiv.org/pdf/2401.16765). 50 | * Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023. [Smoothllm: Defending 51 | large language models against jailbreaking attacks](https://arxiv.org/pdf/2310.03684). *arXiv:2310.03684*. 52 | * Andy Zhou, Bo Li, and Haohan Wang. 2024. [Robust prompt optimization for defending language 53 | models against jailbreaking attacks](https://arxiv.org/pdf/2401.17263). *arXiv:2401.17263*. 54 | 55 | 56 | 57 | ### For Close-Source Models 58 | P.S. These Defense Methods are also suitable for open-source models. 59 | * Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Pingyeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. [Baseline defenses for adversarial attacks against aligned language models.](https://arxiv.org/pdf/2309.00614) *arXiv:2309.00614*. 60 | * Daoyuan Wu, Shuai Wang, Yang Liu, and Ning Liu. 2024. [Llms can defend themselves against 61 | jailbreaking in a practical manner: A vision paper](https://arxiv.org/pdf/2402.15727). 62 | * Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. 2023. [Rain: Your language models can align themselves without finetuning](https://arxiv.org/pdf/2309.07124). *In The Twelfth International Conference on Learning Representations*. 63 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Multilingual Pre-trained Models Tutorial 2 | 3 | 4 | Considering the rapid growth of the research of multilingual NLP, we have established this repository to gather relevant literature in this specific multilingual domain. *(As a contribution of the survey paper "[A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers](https://arxiv.org/abs/2405.10936)")* 5 | 6 | This is also a tutorial of multilingual pre-trained models maintained by the Beijing Jiaotong University (BJTU) NLP Group (Continual Updated). 7 | 8 | The past five years have witnessed the rapid development of multilingual pre-trained models, especially for data-driven large language models (LLMs). Due to the dominance of multilingual NLP at the present time, priority is given to collecting important, up-to-date multilingual pre-trained models papers and their performance. As one of the contributions of the survey, we continuously update and expand the content according to the chapters in the survey. Our list is still incomplete and the categorization might be inappropriate. We will keep adding papers and improving the list. [Any suggestions are welcome!](#cus) 9 | 10 | ## LLMs with Multilingualism 11 | We only present an overview of **representative** LLMs (most of trainable parameters greater than ***7B***) that have certain multilingual capabilities, including their release time and details. The latest models that achieve good performance on the leaderboard will be updated in a timely manner, or contact us for updates and promotion. 12 | 13 | ### General Multilingual Leaderboard 14 | 15 | We investigate the LLMs with multilingualism in our reconstructed benchmarks. *(If there are many versions of a model, we only choose the version that perform the best.)* 16 | 17 | In this leaderboard we use a unified *prompt* for each task to explore the multilingual capabilities of the model. The potential enhancement capabilities of the model are explored in the next chapter "**[Multilingual Inference Strategies](#mis)**". 18 | 19 | > 🎈 A suite for calling LLMs is coming soon! The benchmark is under built. 20 | 21 | ### Open-Source Models 22 | 23 | > All models are available on the Internet. The link of paper or Github is given. 24 | 25 | - LLaMA, Meta AI 26 | - LLaMA-1, [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971), 2023.02.27 27 | - LLaMA-2, [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/abs/2307.09288), 2023.07.18 28 | - LLaMA-3, [Meta Llama 3](https://github.com/meta-llama/llama3/tree/main), 2024.04.18 29 | - GLM, ZHIPU, [ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B), 2023.05.13, The other versions are released at their Github as well. 30 | - Baichuan, Baichuan AI 31 | - Baichuan-1, [Technical Report](https://huggingface.co/baichuan-inc/Baichuan-7B), 2023.06.15 32 | - Baichuan-2, [Baichuan 2: Open Large-scale Language Models](https://arxiv.org/abs/2309.10305), 2023.09.06 33 | - Baichuan-3, [Chat Platform](https://www.baichuan-ai.com/home), 2024.01.29 34 | - Qwen, Alibaba, [Qwen Technical Report](https://arxiv.org/abs/2309.16609) 35 | - Qwen, [Technical Report](https://github.com/QwenLM/Qwen/blob/main/README.md), 2023.08.03 36 | - Qwen-1.5, [Technical Report](https://github.com/QwenLM/Qwen1.5?tab=readme-ov-file), 2024.02.05 37 | - Phi, Microsoft 38 | - Phi-1, [Textbooks Are All You Need](https://arxiv.org/abs/2306.11644), 2023.06.20 39 | - Phi-1.5, [Textbooks Are All You Need II: phi-1.5 technical report](https://www.microsoft.com/en-us/research/publication/textbooks-are-all-you-need-ii-phi-1-5-technical-report/), 2023.09.11 40 | - Phi-2, [Phi-2: The surprising power of small language models](https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/), 2023.12.12 41 | - Phi-3, [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://export.arxiv.org/abs/2404.14219), 2024.04.23 42 | - Mistral, [Mistral 7B](https://arxiv.org/abs/2310.06825) 43 | - Mistral 7B, 2023.10.10 44 | - Mixtral 8x7B, 2023.12.11 45 | - Mixtral 8x22B, 2024.04.17 46 | - OpenChat, Tsinghua University, [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://arxiv.org/abs/2309.11235), 2023.09.20 47 | - Deepseek, DeepSeek AI, [DeepSeek LLM Scaling Open-Source Language Models with Longtermism](https://arxiv.org/abs/2401.02954), 2024.01.05 48 | - InternLM, Shanghai AI Laboratory 49 | - InternLM, [Repo](https://huggingface.co/internlm/internlm-7b), 2023.09.20 50 | - InternLM2, [InternLM2 Technical Report](https://arxiv.org/abs/2403.17297), 2024.03.26 51 | - BLOOM, Big Science, [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://arxiv.org/abs/2211.05100), 2022.11.09 52 | - BLOOMZ-7b1, Hugging Face, [Crosslingual Generalization through Multitask Finetuning](https://arxiv.org/abs/2211.01786), 2023.05.29 53 | - Bayling, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS), [BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models](https://arxiv.org/abs/2306.10968), 2023.06.19 54 | 55 | ### Closed-Source Systems 56 | 57 | We only investigate a few representative closed-source LLMs because most of these commercial systems are expensive to invoke. We hope to get sponsors or voluntary enterprise responses to compare closed-source systems, otherwise our goal is to discuss the future potential of LLMs within the open-source community. 58 | 59 | - GPT, [OpenAI](https://openai.com/) 60 | - ChatGPT (GPT-3.5-turbo) 61 | - GPT-4, [GPT-4 Technical Report](https://arxiv.org/abs/2303.08774), 2023.05.15 62 | - PaLM, Google 63 | - [PaLM-2](https://ai.google/discover/palm2/) 64 | - Claude, [Anthropic](https://claude.ai/) 65 | - Gemini, DeepMind, [Chat with Gemini](https://gemini.google.com/?hl=zh) 66 | 67 | ## Multilingual Inference Strategies 68 | We investigate several inference strategies for LLMs to explore the potential enhancement capabilities with multilingualism in the related benchmarks. *(The multilingual inference strategies are to act on prompt with external knowledge, and LLMs are frozen.)* 69 | 70 | ### Reasoning Leaderboard 71 | 72 | | Model | Method | MGSM | XCOPA | XNLI | PAWS-X | MKQA | Avg | 73 | | ------------------- | ------------ | ---- | ----- | ---- | ------ | ---- | ---- | 74 | | GPT-3.5 | Basic | 34.4 | 72.3 | 52.2 | 49.7 | 35.4 | 48.8 | 75 | | | En-Basic | 41.1 | 76.1 | 63.0 | 62.0 | 36.5 | 55.7 | 76 | | | CoT | 49.9 | 72.4 | 50.6 | 50.3 | 35.4 | 51.7 | 77 | | | En-CoT | 61.1 | 78.6 | 56.4 | 61.6 | 42.9 | 60.1 | 78 | | | XLT | 62.3 | 79.3 | 59.2 | 59.4 | 37.6 | 59.6 | 79 | | | Trans-Google | 73.9 | 84.5 | 60.5 | 67.2 | 43.8 | 66.0 | 80 | | | Trans-NLLB | 61.0 | 79.7 | 59.2 | 67.5 | 37.2 | 60.9 | 81 | | BLOOMZ-7b1 | Basic | 1.3 | 21.4 | 8.3 | - | 7.8 | 9.7 | 82 | | | En-Basic | 2.0 | 56.5 | 43.9 | - | 10.6 | 28.3 | 83 | | | CoT | 1.2 | 20.9 | 8.2 | - | 6.5 | 9.2 | 84 | | | En-CoT | 1.7 | 53.9 | 35.9 | - | 9.3 | 25.2 | 85 | | | XLT | 1.7 | 50.5 | 35.4 | - | 8.0 | 23.9 | 86 | | | Trans-Google | 2.7 | 63.7 | 44.3 | - | 17.2 | 32.0 | 87 | | | Trans-NLLB | 2.4 | 61.8 | 43.8 | - | 14.7 | 30.7 | 88 | | Mistral-7B-Instruct | Basic | 11.2 | 54.2 | 42.8 | 44.6 | 7.8 | 32.1 | 89 | | | En-Basic | 23.8 | 34.9 | 50.2 | 46.9 | 7.0 | 32.6 | 90 | | | CoT | 17.0 | 53.8 | 43.4 | 44.3 | 7.8 | 33.3 | 91 | | | En-CoT | 27.6 | 40.8 | 50.0 | 46.6 | 11.5 | 35.3 | 92 | | | XLT | 31.8 | 61.5 | 46.0 | 47.8 | 9.6 | 39.3 | 93 | | | Trans-Google | 41.3 | 59.2 | 55.0 | 51.5 | 17.0 | 44.8 | 94 | | | Trans-NLLB | 31.7 | 54.4 | 53.0 | 52.4 | 15.5 | 41.4 | 95 | | Llama-2-7B-Chat | Basic | 8.4 | 46.5 | 34.6 | 48.1 | 14.4 | 30.4 | 96 | | | En-Basic | 9.3 | 49.7 | 39.0 | 48.8 | 16.1 | 32.6 | 97 | | | CoT | 10.9 | 46.3 | 35.6 | 48.3 | 13.3 | 30.9 | 98 | | | En-CoT | 13.6 | 54.9 | 41.0 | 48.7 | 13.8 | 34.4 | 99 | | | XLT | 10.4 | 50.8 | 44.8 | 44.5 | 14.6 | 33.0 | 100 | | | Trans-Google | 28.6 | 67.7 | 45.5 | 57.5 | 19.8 | 43.8 | 101 | | | Trans-NLLB | 24.8 | 64.6 | 44.1 | 56.2 | 17.4 | 41.4 | 102 | | Llama-2-13B-Chat | Basic | 15.6 | 50.1 | 36.4 | 54.0 | 18.3 | 34.9 | 103 | | | En-Basic | 19.0 | 54.3 | 43.4 | 59.1 | 20.2 | 39.2 | 104 | | | CoT | 18.1 | 50.9 | 35.7 | 54.8 | 15.7 | 35.0 | 105 | | | En-CoT | 19.9 | 54.5 | 43.7 | 57.6 | 19.8 | 39.1 | 106 | | | XLT | 22.3 | 56.0 | 51.4 | 55.7 | 19.0 | 40.9 | 107 | | | Trans-Google | 39.1 | 71.9 | 46.1 | 58.4 | 33.8 | 49.9 | 108 | | | Trans-NLLB | 31.8 | 68.2 | 45.4 | 57.8 | 28.4 | 46.3 | 109 | | Llama-2-70B-Chat | Basic | 23.6 | 51.5 | 39.0 | 52.8 | 24.8 | 38.3 | 110 | | | En-Basic | 28.6 | 55.3 | 46.5 | 60.4 | 24.7 | 43.1 | 111 | | | CoT | 23.5 | 50.5 | 37.9 | 54.9 | 21.9 | 37.7 | 112 | | | En-CoT | 30.2 | 61.2 | 45.9 | 64.9 | 31.1 | 46.7 | 113 | | | XLT | 32.8 | 58.7 | 52.2 | 55.7 | 26.6 | 45.2 | 114 | | | Trans-Google | 53.3 | 80.9 | 54.0 | 68.5 | 39.7 | 59.3 | 115 | | | Trans-NLLB | 43.8 | 77.1 | 52.2 | 69.2 | 19.4 | 52.3 | 116 | 117 | *Note: This leaderboard is followed by [Liu et al.](https://arxiv.org/abs/2403.10258), we will update it in the next version when the evaluation suite for calling LLMs is built. 118 | 119 | ### Baseline Details 120 | 121 | ``` 122 | [Question]: "制作一件袍子需要2匹蓝色纤维布料和这个数量一半的白色纤维布料。它一共需要用掉多少匹布料" 123 | Basic: [Query]=[Question]+[Prompt: 您的最终答案的格式应为："答案: <阿拉伯数字>".] 124 | En-Basic: [Query]=[Question]+[Prompt -> English Prompt: You should format your final answer as "Answer: ".] 125 | CoT: [Query]=[Question]+[Prompt -> CoT: 让我们一步步思考。您的最终答案的格式应为："答案: <阿拉伯数字>".] 126 | En-CoT: [Query]=[Question]+[Prompt -> English CoT: Let’s think step by step in English. You should format your final answer as "Answer: ".] 127 | XLT: [Query]=[Prefix: I want you to act as an arithmetic reasoning expert for Chinese.Request: ]+[Question]+[Complex Prompt: You should retell the request in English. You should do step-by-step answer to obtain a number answer. You should step-by-step answer the request. You should tell me the answer in this format "Answer:".] 128 | Trans-X: [Query]=[Question -> English Question by X]+[Prompt -> English CoT: Let’s think step by step in English. You should format your final answer as "Answer: ".] 129 | ``` 130 | 131 | ### Reading List 132 | 133 | We provide a [reading list](./readinglist/inference.md) (Continual Updated) for this chapter corresponding to the section 4 in the survey. 134 | 135 | ## Security 136 | 137 | 138 | ### Jailbreaking Leaderboard 139 | 140 | This leaderboard is built by the [EasyJailbreak](https://github.com/EasyJailbreak/EasyJailbreak) framework on the AdvBench. 141 | 142 | | Method | GPT-3.5 | GPT-4 | Llama-2-7B-Chat | Llama-2-13B-Chat | Vicuna-7B-v1.5 | Vicuna-13B-v1.5 | ChatGLM | Qwen-7B-Chat | InterLM-7B | Mistral-7B | 143 | | ------------- | ------- | ----- | --------------- | ---------------- | -------------- | --------------- | ------- | ------------ | ---------- | ---------- | 144 | | GCG | 12% | 0% | 46% | 46% | 94% | 94% | 34% | 48% | 10% | 82% | 145 | | JailBroken | 100% | 58% | 6% | 4% | 100% | 100% | 95% | 100% | 100% | 100% | 146 | | GPTFUZZER | 35% | 0% | 31% | 41% | 93% | 94% | 85% | 82% | 92% | 99% | 147 | | AutoDAN | 45% | 2% | 51% | 72% | 100% | 97% | 89% | 99% | 98% | 98% | 148 | | DeepInception | 66% | 35% | 8% | 0% | 29% | 17% | 33% | 58% | 36% | 40% | 149 | | ICA | 0% | 1% | 0% | 0% | 51% | 81% | 54% | 36% | 23% | 75% | 150 | | PAIR | 19% | 20% | 27% | 13% | 99% | 95% | 96% | 77% | 86% | 95% | 151 | | ReNeLLM | 87% | 38% | 31% | 69% | 77% | 87% | 86% | 70% | 67% | 90% | 152 | | Multilingual | 12% | 0% | 46% | 46% | 94% | 94% | 34% | 48% | 10% | 82% | 153 | | Cipher | 100% | 58% | 6% | 4% | 100% | 100% | 95% | 100% | 100% | 100% | 154 | | CodeChameleon | 35% | 0% | 31% | 41% | 93% | 94% | 85% | 82% | 92% | 99% | 155 | 156 | ### Reading List 157 | 158 | We provide a [reading list](./readinglist/security.md) of jailbreaking and defense methods (Continual Updated) for this chapter corresponding to the section 5 in the survey. 159 | 160 | ## Multidomain 161 | 162 | ### Legal 163 | 164 | > 🎈 The leaderboard of legal benchmark is under built. 165 | 166 | ### Medical 167 | 168 | > 🎈 The leaderboard of medical benchmark is under built. 169 | 170 | ### Finance 171 | 172 | > Coming soon! This domain is under updated. 173 | 174 | ## Data Resource & Evaluation 175 | The data resource and popular benchmarks are listed in the [reading list](./readinglist/data.md) in details. 176 | 177 | ## Contact Us 178 | 179 | **Project Lead:** 180 | 181 | - Kaiyu Huang, kyhuang@bjtu.edu.cn 182 | - Fengran Mo, fengran.mo@umontreal.ca 183 | 184 | **Section Contributors:** 185 | 186 | - Inference: Yulong Mao 187 | - Security: Hongliang Li 188 | - Multidomain: You Li 189 | 190 | **Special Thanks:** 191 | 192 | - Chaoqun Liu (Nanyang Technological University, Singapore) provides valuable thoughts and contributes part of the implementation of the multilingual inference strategies. 193 | 194 | ## Citation 195 | 196 | ```bib 197 | @misc{huang2024survey, 198 | title={A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers}, 199 | author={Kaiyu Huang and Fengran Mo and Hongliang Li and You Li and Yuanchi Zhang and Weijian Yi and Yulong Mao and Jinchen Liu and Yuzhuang Xu and Jinan Xu and Jian-Yun Nie and Yang Liu}, 200 | year={2024}, 201 | eprint={2405.10936}, 202 | archivePrefix={arXiv}, 203 | primaryClass={cs.CL} 204 | } 205 | ``` 206 | 207 | 208 | 209 | 210 | 211 | ​ 212 | 213 | -------------------------------------------------------------------------------- /readinglist/inference.md: -------------------------------------------------------------------------------- 1 | # Multilingual Inference Strategies 2 | 3 | ## Translation-Related 4 | * Yotam Intrator, Matan Halfon, Roman Goldenberg, Reut Tsarfaty, Matan Eyal, Ehud Rivlin, Yossi Matias, Natalia Aizenberg. 2024. [Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications?](https://arxiv.org/abs/2403.04792). *arXiv:2403.04792*. 5 | * Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, Mikel Artetxe. 2023. [Do Multilingual Language Models Think Better in English?](https://arxiv.org/abs/2308.01223). *arXiv:2308.01223*. 6 | * Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, Lidong Bing. 2024. [Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models](https://arxiv.org/abs/2403.10258). *arXiv:2403.10258*. 7 | * Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, Sunayana Sitaram. 2023 [MEGA: Multilingual Evaluation of Generative AI](https://arxiv.org/abs/2303.12528). In *Proceedings of EMNLP 2023*. 8 | * Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, Jason Wei. 2022. [Language Models are Multilingual Chain-of-Thought Reasoners](https://arxiv.org/abs/2210.03057). *arXiv:2210.03057*. 9 | * Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, Anna Korhonen. 2020. [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://arxiv.org/abs/2005.00333). *arXiv:2005.00333*. 10 | * Edoardo Maria Ponti, Julia Kreutzer, Ivan Vulić, Siva Reddy. 2021. [Modelling Latent Translations for Cross-Lingual Transfer](https://arxiv.org/abs/2107.11353). *arXiv:2107.11353*. 11 | * Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale, Angela Fan, Luke Zettlemoyer. 2023. [Revisiting Machine Translation for Cross-lingual Classification](https://arxiv.org/abs/2305.14240). *arXiv:2305.14240*. 12 | * David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, George Foster. 2023. [Prompting PaLM for Translation: Assessing Strategies and Performance](https://arxiv.org/abs/2211.09102). In *Proceedings of ACL 2023* 13 | * Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, Furu Wei. 2023. [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://arxiv.org/abs/2305.07004). In *Findings of EMNLP 2023*. 14 | * Mengjie Zhao, Hinrich Schütze. 2021. [Discrete and Soft Prompting for Multilingual Models](https://arxiv.org/abs/2109.03630). In *Proceedings of EMNLP 2021*. 15 | * Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, Pascale Fung. 2021. [Language Models are Few-shot Multilingual Learners](https://arxiv.org/abs/2109.07684). *arXiv:2109.07684*. 16 | * Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li. 2021. [Few-shot Learning with Multilingual Language Models](https://arxiv.org/abs/2112.10668). In *Proceedings of EMNLP 2022*. 17 | 18 | ## Chain-of-Thought (CoT) 19 | * Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, Lidong Bing. 2024. [Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models](https://arxiv.org/abs/2403.10258). *arXiv:2403.10258*. 20 | * Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, Furu Wei. 2023. [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://arxiv.org/abs/2305.07004). In *Findings of EMNLP 2023*. 21 | * Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, Jason Wei. 2022. [Language Models are Multilingual Chain-of-Thought Reasoners](https://arxiv.org/abs/2210.03057). *arXiv:2210.03057*. 22 | * Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, Minjoon Seo. 2023. [The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning](https://arxiv.org/abs/2305.14045). In *Proceedings of EMNLP 2023*. 23 | * Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, Jason Wei. 2022. [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://arxiv.org/abs/2210.09261). *arXiv:2210.09261*. 24 | * Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xiannian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, Zhoujun Li. 2024. [xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning](https://arxiv.org/abs/2401.07037). *arXiv:2401.07037*. 25 | * Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, Wanxiang Che. 2023. [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://arxiv.org/abs/2310.14799). In *Proceedings of EMNLP 2023*. 26 | * Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou. 2022. [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html). In *Proceedings of NeurIPS 2022*. 27 | * Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, Denny Zhou. 2023. [Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models](https://arxiv.org/abs/2310.06117). In *Proceedings of ICLR 2024*. 28 | * Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, Denny Zhou. 2022. [Recitation-Augmented Language Models](https://arxiv.org/abs/2210.01296). In *Proceedings of ICLR 2023*. 29 | * Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, Liwei Wang. 2023. [Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective](https://proceedings.neurips.cc/paper_files/paper/2023/hash/dfc310e81992d2e4cedc09ac47eff13e-Abstract-Conference.html). In *Proceedings of NeurIPS 2023*. 30 | * Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou. 2023. [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://arxiv.org/abs/2203.11171). In *Proceedings of ICLR 2023*. 31 | * Aman Madaan, Amir Yazdanbakhsh. 2022. [Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango](https://arxiv.org/abs/2209.07686). *arXiv:2209.07686*. 32 | * Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, Alex Smola. 2023. [Multimodal Chain-of-Thought Reasoning in Language Models](https://arxiv.org/abs/2302.00923). In *Transactions on Machine Learning Research*. 33 | * Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, Ting Liu. 2023. [Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future](https://arxiv.org/abs/2309.15402). In *Proceedings of ACL 2024*. 34 | * Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, Chris Callison-Burch. 2023. [Faithful Chain-of-Thought Reasoning](https://arxiv.org/abs/2301.13379). In *Proceedings of IJCNLP-AACL 2023*. 35 | * Zhuosheng Zhang, Aston Zhang, Mu Li, Alex Smola. 2022. [Automatic Chain of Thought Prompting in Large Language Models](https://arxiv.org/abs/2210.03493). *arXiv:2210.03493*. 36 | * Jiuhai Chen, Lichang Chen, Heng Huang, Tianyi Zhou. 2023. [When do you need Chain-of-Thought Prompting for ChatGPT?](https://arxiv.org/abs/2304.03262). *arXiv:2304.03262*. 37 | * Shizhe Diao, Pengcheng Wang, Yong Lin, Tong Zhang. 2023. [Active Prompting with Chain-of-Thought for Large Language Models](https://arxiv.org/abs/2302.12246). In *Proceedings of ACL 2024*. 38 | * Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, Ethan Perez. 2023. [Measuring Faithfulness in Chain-of-Thought Reasoning](https://arxiv.org/abs/2307.13702). *arXiv:2307.13702*. 39 | * Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, Sibei Yang. 2023. [DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models](https://proceedings.neurips.cc/paper_files/paper/2023/hash/108030643e640ac050e0ed5e6aace48f-Abstract-Conference.html). In *Proceedings of NeurIPS 2023*. 40 | * Leonardo Ranaldi, Andre Freitas. 2024. [Aligning Large and Small Language Models via Chain-of-Thought Reasoning](https://aclanthology.org/2024.eacl-long.109/). In *Proceedings of EACL 2024*. 41 | * Miles Turpin, Julian Michael, Ethan Perez, Samuel Bowman. 2023. [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://proceedings.neurips.cc/paper_files/paper/2023/hash/ed3fea9033a80fea1376299fa7863f4a-Abstract-Conference.html). In *Proceedings of NeurIPS 2023*. 42 | * Abulhair Saparov, He He. 2023. [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://arxiv.org/abs/2210.01240). In *Proceedings of ICLR 2023*. 43 | * Chancharik Mitra, Brandon Huang, Trevor Darrell, Roei Herzig. 2024. [Compositional Chain-of-Thought Prompting for Large Multimodal Models](https://openaccess.thecvf.com/content/CVPR2024/html/Mitra_Compositional_Chain-of-Thought_Prompting_for_Large_Multimodal_Models_CVPR_2024_paper.html). In *Proceedings of CVPR 2024*. 44 | * Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, Hao Su. 2023. [Deductive Verification of Chain-of-Thought Reasoning](https://proceedings.neurips.cc/paper_files/paper/2023/hash/72393bd47a35f5b3bee4c609e7bba733-Abstract-Conference.html). In *Proceedings of NeurIPS 2023*. 45 | 46 | ## Retrieval-Augmented Generation (RAG) 47 | * Peng Shi, Rui Zhang, He Bai, Jimmy Lin. 2022. [XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsing](https://arxiv.org/abs/2210.13693). In *Findings of EMNLP 2022. 48 | * Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, Nan Duan. 2023. [Query Rewriting for Retrieval-Augmented Large Language Models](https://arxiv.org/abs/2305.14283). *arXiv:2305.14283*. 49 | * Luyu Gao, Xueguang Ma, Jimmy Lin, Jamie Callan. 2022. [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://arxiv.org/abs/2212.10496). *arXiv:2212.10496*. 50 | * Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, Meng Jiang. 2022. [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://arxiv.org/abs/2209.10063). In *Proceedings of ICLR 2023*. 51 | * Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, Weizhu Chen. 2023. [Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy](https://arxiv.org/abs/2305.15294). *arXiv:2305.15294*. 52 | * Xintao Wang, Qianwen Yang, Yongting Qiu, Jiaqing Liang, Qianyu He, Zhouhong Gu, Yanghua Xiao, Wei Wang. 2023. [KnowledGPT: Enhancing Large Language Models with Retrieval and Storage Access on Knowledge Bases](https://arxiv.org/abs/2308.11761). *arXiv:2308.11761*. 53 | * Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, Rui Yan. 2023. [Lift Yourself Up: Retrieval-augmented Text Generation with Self-Memory](https://proceedings.neurips.cc/paper_files/paper/2023/hash/887262aeb3eafb01ef0fd0e3a87a8831-Abstract-Conference.html). In *Proceedings of NeurIPS 2023*. 54 | * Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, Michael Zeng. 2022. [Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data](https://arxiv.org/abs/2203.08773). In *Proceedings of ACL 2022*. 55 | * Xiaoqian Li, Ercong Nie, Sheng Liang. 2023. [From Classification to Generation: Insights into Crosslingual Retrieval Augmented ICL](https://arxiv.org/abs/2311.06595). In *The Workshop on Instruction Tuning and Instruction Following*. 56 | * Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Denvy Deng, Qi Zhang. 2023. [UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation](https://arxiv.org/abs/2303.08518). In *Proceedings of EMNLP 2023*. 57 | * Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, Ming-Wei Chang. 2022. [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://arxiv.org/abs/2209.11755). *arXiv:2209.11755*. 58 | * Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, Matei Zaharia. 2022. [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://arxiv.org/abs/2212.14024). *arXiv:2212.14024*. 59 | * Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, Graham Neubig. 2023. [Active Retrieval Augmented Generation](https://arxiv.org/abs/2305.06983). In *Proceedings of EMNLP 2023*. 60 | * Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, Hannaneh Hajishirzi. 2023. [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://arxiv.org/abs/2310.11511). *arXiv:2310.11511*. 61 | * Zixuan Ke, Weize Kong, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky. 2024. [Bridging the Preference Gap between Retrievers and LLMs](https://arxiv.org/abs/2401.06954). *arXiv:2401.06954*. 62 | * Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, Scott Yih. 2023. [RA-DIT: Retrieval-Augmented Dual Instruction Tuning](https://arxiv.org/abs/2310.01352). In *Proceedings of ICLR 2024*. 63 | * Oded Ovadia, Menachem Brief, Moshik Mishaeli, Oren Elisha. 2023. [Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs](https://arxiv.org/abs/2312.05934). *arXiv:2312.05934*. 64 | * Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, Dong Yu. 2023. [Dense X Retrieval: What Retrieval Granularity Should We Use?](https://arxiv.org/abs/2312.06648). *arXiv:2312.06648*. 65 | * Fan Luo, Mihai Surdeanu. 2023. [Divide & Conquer for Entailment-aware Multi-hop Evidence Retrieval](https://arxiv.org/abs/2311.02616). In *Proceedings of NAACL-HLT SRW 2022*. 66 | * Qi Gou, Zehua Xia, Bowen Yu, Haiyang Yu, Fei Huang, Yongbin Li, Nguyen Cam-Tu. 2023. [Diversify Question Generation with Retrieval-Augmented Style Transfer](https://arxiv.org/abs/2310.14503). In *Proceedings of EMNLP 2023*. 67 | * Zhicheng Guo, Sijie Cheng, Yile Wang, Peng Li, Yang Liu. 2023. [Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Tasks](https://arxiv.org/abs/2305.17653). *arXiv:2305.17653*. 68 | * Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, Graham Neubig. 2023. [Learning to Filter Context for Retrieval-Augmented Generation](https://arxiv.org/abs/2311.08377). *arXiv:2311.08377*. 69 | * Xinya Du, Heng Ji. 2022. [Retrieval-Augmented Generative Question Answering for Event Argument Extraction](https://arxiv.org/abs/2211.07067). In *Proceedings of EMNLP 2022*. 70 | * Liang Wang, Nan Yang, Furu Wei. 2023. [Learning to Retrieve In-Context Examples for Large Language Models](https://arxiv.org/abs/2307.07164). In *Proceedings of EACL 2024*. 71 | * Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, Maciej Kula, Ed Chi, Maheswaran Sathiamoorthy. 2023. [Recommender Systems with Generative Retrieval](https://proceedings.neurips.cc/paper_files/paper/2023/hash/20dcab0f14046a5c6b02b61da9f13229-Abstract-Conference.html). In *Proceedings of NeurIPS 2023*. 72 | 73 | ## Code-Switching 74 | * Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Indra Winata, Alham Fikri Aji. 2023. [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://arxiv.org/abs/2305.14235). In *Proceedings of EMNLP 2023*. 75 | * Fajri Koto, Tilman Beck, Zeerak Talat, Iryna Gurevych, Timothy Baldwin. 2024. [Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon](https://arxiv.org/abs/2402.02113). In *Proceedings of EACL 2024*. 76 | * Navya Jose, Bharathi Raja Chakravarthi, Shardul Suryawanshi, Elizabeth Sherly, John P. McCrae. 2020. [A Survey of Current Datasets for Code-Switching Research](https://ieeexplore.ieee.org/abstract/document/9074205). In *Proceedings of ICACCS 2020*. 77 | * Richeek Das, Sahasra Ranjan, Shreya Pathak, Preethi Jyothi. 2023. [Improving Pretraining Techniques for Code-Switched NLP](https://aclanthology.org/2023.acl-long.66/). In *Proceedings of ACL 2023*. 78 | * Himashi Rathnayake, Author PictureJanani Sumanapala, Author PictureRaveesha Rukshani, Author PictureSurangika Ranathunga. 2022. [Adapter-based fine-tuning of pre-trained multilingual language models for code-mixed and code-switched text classification](https://dl.acm.org/doi/abs/10.1007/s10115-022-01698-1). In *Knowledge and Information Systems*. 79 | * Ronny Mabokela, Tim Schlippe. 2022. [A Sentiment Corpus for South African Under-Resourced Languages in a Multilingual Context](https://aclanthology.org/2022.sigul-1.9/). In *Proceedings of SIGUL 2022*. 80 | * Frances A. Laureano De Leon, Harish Tayyar Madabushi, Mark Lee. 2024. [Code-Mixed Probes Show How Pre-Trained Models Generalise On Code-Switched Text](https://arxiv.org/abs/2403.04872). In *Proceedings of LREC-COLING 2024*. 81 | * Hariram Veeramani, Surendrabikram Thapa, Usman Naseem. 2024. [MLInitiative@WILDRE7: Hybrid Approaches with Large Language Models for Enhanced Sentiment Analysis in Code-Switched and Code-Mixed Texts](https://aclanthology.org/2024.wildre-1.10/). In *Proceedings of WILDRE 2024*. 82 | * Anjali Yadav, Tanya Garg, Matej Klemen, Matej Ulcar, Basant Agarwal, Marko Robnik Sikonja. 2024. [Code-mixed Sentiment and Hate-speech Prediction](https://arxiv.org/abs/2405.12929). *arXiv:2405.12929*. 83 | * Sargam Yadav, Abhishek Kaushik, Kevin McDaid. 2024. [Leveraging Weakly Annotated Data for Hate Speech Detection in Code-Mixed Hinglish: A Feasibility-Driven Transfer Learning Approach with Large Language Models](https://arxiv.org/abs/2403.02121). In *Proceedings of ISDSI-Global 2023*. 84 | * Gaurish Thakkar, Marko Tadić, Nives Mikelic Preradovic. 2024. [FZZG at WILDRE-7: Fine-tuning Pre-trained Models for Code-mixed, Less-resourced Sentiment Analysis](https://aclanthology.org/2024.wildre-1.9/). In *Proceedings of WILDRE 2024*. 85 | * Akash Ghosh, Arkadeep Acharya, Prince Jha, Sriparna Saha, Aniket Gaudgaul, Rajdeep Majumdar, Aman Chadha, Raghav Jain, Setu Sinha, Shivani Agarwal. 2024. [MedSumm: A Multimodal Approach to Summarizing Code-Mixed Hindi-English Clinical Queries](https://link.springer.com/chapter/10.1007/978-3-031-56069-9_8). In *Advances in Information Retrieval 2024*. 86 | --------------------------------------------------------------------------------