31 |
32 |
33 | ## Overview
34 | This is the official repository for PandaLM: Re**P**roducible **and** **A**utomated **L**anguage **M**odel Assessment.
35 |
36 | Paper: [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://arxiv.org/abs/2306.05087)
37 |
38 | PandaLM aims to provide reproducible and automated comparisons between different large language models (LLMs). By giving PandaLM the same context, it can compare the responses of different LLMs and provide a reason for the decision, along with a reference answer. The target audience for PandaLM may be organizations that have confidential data and research labs with limited funds that seek reproducibility. These organizations may not want to disclose their data to third parties or may not be able to afford the high costs of secret data leakage using third-party APIs or hiring human annotators. With PandaLM, they can perform evaluations without compromising data security or incurring high costs, and obtain reproducible results. To demonstrate the reliability and consistency of our tool, we have created a diverse human-annotated test dataset of approximately 1,000 samples, where the contexts and the labels are all created by humans. **Our results indicate that PandaLM-7B achieves 93.75% of GPT-3.5's evaluation ability and 88.28% of GPT-4's in terms of F1-score on our test dataset.**. **More papers and features are coming soon.**
39 |
40 | 
41 |
42 | This repository contains:
43 |
44 | - The codes for training PandaLM
45 | - The human-annotated test dataset with ~1k samples for validating PandaLM's ability to evaluate LLMs
46 | - The model weights of PandaLM
47 | - The codes and configs for instruction tuning other foundation models such as Bloom, OPT and LLaMA, etc.
48 |
49 | ## News
50 | - [2024/05/21] Fix the training data 404 problem.
51 | - [2024/01/16] Pandalm is accepted by ICLR 2024!
52 | - [2023/07/25] We share our tuned Alpaca and PandaLM-Alpaca at [Huggingface](https://huggingface.co/WeOpenML). Besides, we uploaded our tuned Alpaca and PandaLM-Alpaca to [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), the results prove that using PandaLM can boost the instruction tuning performance.
53 | - [2023/04/30] We are pleased to announce the release of PandaLM 1.0 as an open-source tool for evaluating LLMs with reliability. To further demonstrate the effectiveness of PandaLM, we are also sharing a human-annotated test dataset.
54 | - [2023/06/08] We release our paper on arxiv: [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://arxiv.org/abs/2306.05087)
55 |
56 | ## **Table of Contents**
57 |
58 | - [PandaLM: Panda Language Models](#pandalm-panda-language-models)
59 | - [News](#News)
60 | - [Table of Contents](#table-of-contents)
61 | - [Motivation](#motivation)
62 | - [Installation](#installation)
63 | - [Usage](#usage)
64 | - [Data](#data)
65 | - [Train data](#train-data)
66 | - [Test data](#test-data)
67 | - [Experimental Results](#experimental-results)
68 | - [Conrtibuting](#conrtibuting)
69 | - [Citation](#citation)
70 | - [License](#license)
71 |
72 | ## Motivation
73 |
74 | 
75 |
76 | As shown above, an iteration of the instruction tuning of LLMs includes training and evaluation. Each iteration refers to changes in hyper-paramters or fine tuning algorithms. The instruction tuning of LLMs can be done within a few GPU hours even using a consumer-grade GPU, thanks to parameter-efficient-tuning methods. However, the human-based and API-based evaluations can be more expensive and time consuming. Furthermore, they can be inconsistent and unreproducible due to a lack of transparency regarding LLM changelogs and subjectivity in human annotations. Moreover, the use of API-based evaluations can result in potential high costs of remediation after [secret data leakage](https://mashable.com/article/samsung-chatgpt-leak-details). Formally, the total cost of instruction tuning LLMs is $N \times (C_{train}+C_{cost})$, where $C_{train}$ can be significantly smaller than $C_{cost}$ if human-based or API-based evaluations are adopted.
77 |
78 | To address these challenges, we propose an evaluation model named PandaLM, which can ensure reproducibility, safety, and efficiency in evaluation. By automating the evaluation process, our model can achieve efficient and consistent evaluations while maintaining high evaluation ability.
79 |
80 | ## **Installation**
81 |
82 | To install PandaLM, follow these steps:
83 |
84 | 1. Clone the repository: `git clone https://github.com/WeOpenML/PandaLM.git`
85 | 2. Navigate to the project directory: `cd PandaLM`
86 | 3. Install the required dependencies: `pip install -r requirements.txt` or use `conda env create -f conda-env.yml` if you prefer conda. Note that it's required to modify `prefix` in `conda-env.yml` to your conda path.
87 |
88 | To instruction a foundation model, follow these steps:
89 | 1. Install PandaLM.
90 | 2. Navigate to the project directory: `cd PandaLM/pandalm`
91 | 3. Run the demo scripts: `bash scripts/inst-tune.sh`
92 |
93 | Due to concerns about copyright issues, we do not provided the instruction tuned model. The instruction tuned model can be easily reproduced in PandaLM/pandalm/scripts/inst-tune.sh.
94 |
95 | ## **Usage**
96 |
97 | We have uploaded PandaLM-7B to HuggingFace, you can simply initialize the model and tokenizer with:
98 |
99 | ```python
100 | from transformers import AutoTokenizer, AutoModelForCausalLM
101 |
102 | tokenizer = AutoTokenizer.from_pretrained("WeOpenML/PandaLM-7B-v1",use_fast=False)
103 |
104 | model = AutoModelForCausalLM.from_pretrained("WeOpenML/PandaLM-7B-v1")
105 | ```
106 |
107 |
108 | We offer several choices for experiencing our PandaLM. (Preparing codes..Please be patient.)
109 | 1. Try PandaLM on your local machine (with a GPU having at least 24G VRAM) using a Web UI:
110 | ```shell
111 | cd PandaLM/pandalm/
112 | CUDA_VISIBLE_DEVICES=0 python3 run-gradio.py --base_model=WeOpenML/PandaLM-7B-v1 --server_port=
6 |
7 |
8 | Yidong Wang*1,2
9 | Zhuohao Yu*1
10 |
11 | Zhengran Zeng1
12 | Linyi Yang2
13 | Qiang Heng3
14 | Cunxiang Wang2
15 | Hao Chen4
16 | Chaoya Jiang1
17 | Rui Xie1
18 |
19 | Jindong Wang5
20 | Xing Xie5
21 | Wei Ye†1
22 | Shikun Zhang†1
23 | Yue Zhang†2
24 | 
118 |