├── LICENSE
├── README.md
├── README_zh.md
├── dataset_example.md
├── examples
    ├── result_extract_custom_example.py
    └── result_extract_standard_example.py
├── resources
    ├── logo.jpg
    ├── logo.png
    ├── overview.jpg
    └── overview.png
└── submission_example.json


/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2024 CS-EVAL
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | <p align="center"> <img src="resources/logo.jpg" style="width: 45%;" id="title-icon">       </p>
  2 | 
  3 | <p align="center">
  4 |    🌐 <a href="https://cs-eval.com/" target="_blank">Website</a> ｜ 🤗 <a href="https://huggingface.co/datasets/cseval/cs-eval" target="_blank">Hugging Face</a> • 🤖️ <a href="https://modelscope.cn/datasets/cseval/cs-eval/summary" target="_blank">ModelScope</a><br> <a href="https://github.com/CS-EVAL/CS-Eval/blob/main/README.md">English | <a href="https://github.com/CS-EVAL/CS-Eval/blob/main/README_zh.md">中文</a>
  5 | </p>
  6 | CS-Eval is a comprehensive evaluation toolkit for fundamental cybersecurity models or large language models' cybersecurity ability, encompassing 11 major cybersecurity categories, 42 subdomains, featuring 4,369 assessment items across multiple-choice, true/false, and knowledge extraction questions. It delivers a balanced mix of knowledge-oriented and practice-focused evaluation tasks. The platform empowers users to conduct self-assessments and offers leaderboards across various subdomains, fostering competitive benchmarking and performance insights.
  7 | 
  8 | <img src="resources/overview.jpg" style="zoom: 80%;" >
  9 | 
 10 | 
 11 | 
 12 | ## News
 13 | 
 14 | - **[2024.05.31]** CS-Eval has been released, and users are now able to submit evaluations on the website independently. 🎉🎉🎉
 15 | - **[2024.03.29]** The CS-Eval dataset has been jointly constructed and completed. ✅✅✅
 16 | 
 17 | 
 18 | 
 19 | ## Table of Contents
 20 | 
 21 | - [Leaderboard](#leaderboard)
 22 | 
 23 | - [Data](#data)
 24 | - [CS-Eval Usage Steps](#cs-eval-usage-steps)
 25 | - [How to Submit](#how-to-submit)
 26 | - [Licenses](#licenses)
 27 | - [Citation](#citation)
 28 | - [Disclaimer](#disclaimer)
 29 | 
 30 | 
 31 | 
 32 | ## Leaderboard
 33 | 
 34 | Here are the accuracies obtained when evaluating industry-leading models upon our initial release. Please refer to our official platform's Leaderboard for the latest community rankings and also pay attention to rankings within different subdomains. Note that subtle differences may exist in the results for the same model due to variations in its generation config.
 35 | 
 36 | | Model                    | Overall Score | AI & Cybersecurity | Business Continuity & Emergency Response & Recovery | Supply Chain Security | Cryptography Techniques & Key Management | Infrastructure Security | Threat Detection & Prevention | Secure Architecture Design | Data Security & Privacy Protection | Vulnerability Management & Penetration Testing | System Security & Software Security Fundamentals | Access Control & Identity Management | Chinese Questions | English Questions |
 37 | | ------------------------ | ------------- | ------------------ | --------------------------------------------------- | --------------------- | ---------------------------------------- | ----------------------- | ----------------------------- | -------------------------- | ---------------------------------- | ---------------------------------------------- | ------------------------------------------------ | ------------------------------------ | ----------------- | ----------------- |
 38 | | GPT4-8K                  | 87.57         | 91.58              | 84.28                                               | 89.30                 | 86.51                                    | 88.83                   | 85.21                         | 83.90                      | 86.90                              | 89.63                                          | 90.00                                            | 86.56                                | 87.96             | 82.19             |
 39 | | GPT3.5-Turbo-16K         | 80.59         | 80.69              | 81.27                                               | 88.96                 | 69.59                                    | 83.17                   | 79.52                         | 76.59                      | 82.14                              | 80.71                                          | 80.00                                            | 78.31                                | 80.62             | 80.14             |
 40 | | Qwen-14B-Chat            | 79.04         | 87.13              | 78.60                                               | 87.63                 | 68.49                                    | 81.33                   | 79.67                         | 74.15                      | 76.68                              | 77.80                                          | 77.00                                            | 78.89                                | 79.99             | 65.41             |
 41 | | Qwen1.5-14B-Chat         | 76.66         | 78.71              | 70.23                                               | 81.27                 | 76.13                                    | 78.00                   | 77.53                         | 70.73                      | 77.58                              | 75.77                                          | 75.33                                            | 77.59                                | 76.68             | 75.68             |
 42 | | Qwen1.5-MoE-A2.7B-Chat   | 74.63         | 74.75              | 72.24                                               | 81.94                 | 73.50                                    | 71.88                   | 76.61                         | 68.78                      | 70.24                              | 74.80                                          | 74.33                                            | 79.50                                | 75.99             | 55.14             |
 43 | | Baichuan2-13B-Chat       | 73.92         | 76.24              | 73.91                                               | 80.27                 | 60.09                                    | 76.50                   | 76.94                         | 71.71                      | 75.69                              | 70.55                                          | 70.67                                            | 73.90                                | 73.79             | 75.34             |
 44 | | 360Zhinao-7B-Chat-4K     | 66.37         | 71.29              | 66.33                                               | 70.00                 | 51.04                                    | 66.78                   | 68.63                         | 68.78                      | 65.02                              | 64.78                                          | 67.67                                            | 68.14                                | 66.68             | 61.99             |
 45 | | Mistral-7B-Instruct-v0.2 | 65.93         | 69.31              | 63.67                                               | 72.76                 | 57.78                                    | 70.43                   | 64.40                         | 62.44                      | 63.44                              | 63.71                                          | 63.67                                            | 69.54                                | 66.01             | 63.36             |
 46 | | Yi-6B-Chat               | 65.27         | 65.84              | 59.67                                               | 72.76                 | 68.80                                    | 64.84                   | 63.85                         | 60.00                      | 62.85                              | 64.68                                          | 63.00                                            | 69.98                                | 65.58             | 59.93             |
 47 | | ChatGLM3-6B              | 57.33         | 65.35              | 56.67                                               | 68.44                 | 47.78                                    | 59.87                   | 61.47                         | 61.46                      | 57.71                              | 50.81                                          | 50.33                                            | 55.26                                | 57.14             | 59.25             |
 48 | | SecGPT-13B               | 47.34         | 40.59              | 45.33                                               | 59.14                 | 41.54                                    | 47.60                   | 47.34                         | 45.85                      | 43.08                              | 46.77                                          | 46.00                                            | 53.15                                | 48.45             | 31.85             |
 49 | | Llama-2-13b-chat-hf      | 38.08         | 38.12              | 39.13                                               | 30.43                 | 34.11                                    | 37.67                   | 39.00                         | 37.07                      | 35.52                              | 38.57                                          | 33.33                                            | 47.60                                | 38.40             | 32.88             |
 50 | 
 51 | 
 52 | 
 53 | ## Data
 54 | 
 55 | #### Download
 56 | 
 57 | - Method 1: Download or load on [Hugging Face](https://huggingface.co/datasets/cseval/cs-eval):
 58 | 
 59 |   - Download data directly:
 60 |   
 61 |   ```
 62 |     wget https://huggingface.co/datasets/cseval/cs-eval/resolve/main/cs-eval-questions.zip
 63 |   ```
 64 |   
 65 |   - Load dataset using Hugging Face datasets:
 66 |   
 67 |   ```
 68 |   from datasets import load_dataset
 69 |   dataset=load_dataset(r"cseval/cs-eval")
 70 |   
 71 |   print(dataset['{test}'][0])
 72 |   ```
 73 | 
 74 | - Method 2: Download on [ModelScope](https://modelscope.cn/datasets/cseval/cs-eval/summary):
 75 | 
 76 |   - Download data directly:
 77 | 
 78 |   ```
 79 |   git clone https://www.modelscope.cn/datasets/cseval/cs-eval.git
 80 |   ```
 81 | 
 82 |   - Load dataset using Model Scope SDK:
 83 | 
 84 |   ```
 85 |   from modelscope.msdatasets import MsDataset
 86 |   ds =  MsDataset.load('cseval/cs-eval')
 87 |   ```
 88 | 
 89 | 
 90 | 
 91 | 
 92 | ## CS-Eval Usage Steps
 93 | 
 94 | 1. Download the CS-Eval evaluation data from either Hugging Face or ModelScope.
 95 | 2. Adapt the model inference format.
 96 | 3. Conduct model inference on the CS-Eval evaluation dataset.
 97 | 4. Format the inference results according to the specified submission guidelines.
 98 | 5. Submit the model results to the CS-Eval platform.
 99 | 6. Obtain the evaluation results on the platform.
100 | 7. Decide whether to opt into public leaderboard.
101 | 
102 | 
103 | 
104 | ## How to Submit
105 | 
106 | You need to convert the organized model inference results into a JSON file encoded in UTF-8 and format it according to the following example.
107 | ```
108 | ## Example
109 | [
110 |     {
111 |       "question_id": "1",
112 |       "answer": "A"
113 |     },
114 |     {
115 |       "question_id": "123",
116 |       "answer": "对"
117 |     },
118 |     {
119 |       "question_id": "1234",
120 |       "answer": "是否涉及漏洞：是\n漏洞号：CVE-2024-22891\n影响的产品及版本：Nteract v.0.28.0"
121 |     }
122 | ]
123 | ```
124 | 
125 | In this example, question_id refers to the question number, and answer contains the processed model output.
126 | 
127 | 
128 | 
129 | Please note:
130 | 
131 | - For multiple-choice questions, the correct answer option(s) can typically be extracted directly from the model's generated result using regular expressions.
132 | - In the case of multiple-answer questions, regular expressions can similarly be used to extract multiple correct answer options from the model's output.
133 | - For true/false questions, if the question instructions require a specific answer format, the judgment result is usually taken from the beginning or end of the model's output.
134 | - For knowledge extraction tasks, where the question specifies a particular response format, use the raw text output from the model inference directly.
135 | 
136 | 
137 | 
138 | When you regularize multiple-choice questions, you can quickly locate multiple-choice questions by filtering the following keywords in the dataset prompt.
139 | 
140 | ```
141 | "单选题："
142 | "多选题："
143 | "Single-choice question:"
144 | ```
145 | 
146 | 
147 | 
148 | ## Licenses
149 | 
150 | [![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](https://lbesson.mit-license.org/)
151 | 
152 | This project adheres to the [MIT License](https://lbesson.mit-license.org/).
153 | 
154 | [![CC BY-NC-SA 4.0](https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg)](http://creativecommons.org/licenses/by-nc-sa/4.0/)
155 | 
156 | The CS-Eval dataset adheres to the [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-nc-sa/4.0/).
157 | 
158 | 
159 | 
160 | ## Citation
161 | 
162 | If you utilize our dataset in your research or technical reports, please ensure proper citation.
163 | 
164 | ```bibtex
165 | @inproceedings{Yu2024CSEvalAC,
166 |   title={CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity},
167 |   author={Zhengmin Yu and Jiutian Zeng and Siyi Chen and Wenhan Xu and Dandan Xu and Xiangyu Liu and Zonghao Ying and Nan Wang and Yuan Zhang and Min Yang},
168 |   year={2024},
169 |   url={https://api.semanticscholar.org/CorpusID:274234403}
170 | }
171 | ```
172 | 
173 | 
174 | 
175 | ## Disclaimer
176 | 
177 | Our platform and its affiliated entities consistently adhere to principles of legality, compliance, positivity, and health, dedicated to promoting the research and application of large language models in the field of cybersecurity to enhance protective capabilities. To prevent any potential misunderstanding of the content on this platform, we hereby issue the following statement:
178 | 
179 | 1. Legitimate Purposes: All information, resources, tools, and services we provide are intended to facilitate lawful and beneficial activities in the cybersecurity sector, including scientific research, technological innovation, risk assessment, and the formulation of defensive strategies using large models. We firmly oppose any utilization of large language models for illegal activities, infringement, or compromising cybersecurity.
180 | 1. Non-Inducement: This platform strictly prohibits any content that incites or encourages others to engage in cyberattacks, intrusions, disruptions, or unauthorized data acquisition. We emphasize that all content related to large language model cybersecurity evaluation sets aims to advance industry development, provide cybersecurity system evaluations, and facilitate technical exchanges, devoid of any elements that induce, encourage, or imply malicious attacks.
181 | 1. Non-Malicious Attack Education: We explicitly state that our provided content does not involve teaching, demonstrating, or guiding techniques for malicious cyberattacks. All discussions involving offensive actions are strictly confined within the realms of legitimate cybersecurity drills, vulnerability research, and risk assessments, aimed at enhancing defensive capabilities rather than offensive purposes.
182 | 1. User Responsibility: Users must strictly abide by relevant laws and regulations when using our platform's services and are prohibited from utilizing platform resources for any illegal, infringing, or cybersecurity-compromising activities. In the event of user violations, our platform reserves the right to take measures including but not limited to warnings, service suspension, account banning, and the pursuit of legal responsibility.
183 | 1. Disclaimer: While we strive to ensure the accuracy, legality, and appropriateness of our platform's content, users are solely responsible for any direct or indirect losses incurred from their own actions during usage, including legal disputes, property loss, data breaches, reputational harm, etc. Neither our platform nor its affiliated entities assume any legal liability. Users should assess and bear all risks associated with using platform resources themselves.
184 | 
185 | 
186 | 
187 | We sincerely call upon all users to jointly maintain a sound order in the cybersecurity domain, employing large model technologies and related resources legally, rationally, and responsibly. The final interpretation right of this disclaimer resides with our platform, and changes, if any, will not be separately notified.
188 | 
189 | 


--------------------------------------------------------------------------------
/README_zh.md:
--------------------------------------------------------------------------------
  1 | <p align="center"> <img src="resources/logo.jpg" style="width: 45%;" id="title-icon">       </p>
  2 | 
  3 | <p align="center">
  4 |    🌐 <a href="https://cs-eval.com/" target="_blank">评测网站</a> ｜ 🤗 <a href="https://huggingface.co/datasets/cseval/cs-eval" target="_blank">Hugging Face</a> • 🤖️ <a href="https://modelscope.cn/datasets/cseval/cs-eval/summary" target="_blank">ModelScope</a><br> <a href="https://github.com/CS-EVAL/CS-Eval/blob/main/README.md">English | <a href="https://github.com/CS-EVAL/CS-Eval/blob/main/README_zh.md">中文</a>
  5 | </p>
  6 | CS-Eval 是网络安全基础模型或大模型网络安全能力的综合评测套件，覆盖了11个网络安全大类领域、42个子类领域的4369个多项选择题、判断题、知识抽取题，提供偏知识型和偏实战型的综合评估任务。平台支持用户自主评测，并提供不同子领域的排行。
  7 | 
  8 | <img src="resources/overview.jpg" style="zoom: 80%;" >
  9 | 
 10 | ## 更新
 11 | 
 12 | - **[2024.05.31]** CS-Eval已正式发布，用户可自主在网站提交评估。🎉🎉🎉 
 13 | 
 14 | - **[2024.03.29]** CS-Eval数据集已联合构建完成。✅✅✅
 15 | 
 16 | 
 17 | 
 18 | ## 目录
 19 | 
 20 | - [排行榜](#排行榜)
 21 | 
 22 | - [数据](#数据)
 23 | - [CS-Eval使用步骤](#CS-Eval使用步骤)
 24 | - [如何提交](#如何提交)
 25 | - [Licenses](#licenses)
 26 | - [引用](#引用)
 27 | - [免责声明](#免责声明)
 28 | 
 29 | 
 30 | 
 31 | ## 排行榜
 32 | 
 33 | 以下是我们在初次公开发布之际，对业界一系列前沿模型所进行的精度评估数据。请持续关注CS-Eval官方平台Leaderboard，以便及时获取社区内最新的排名动态。同时，我们也详尽提供了针对各细分应用领域的专项排行榜。鉴于模型推理过程中生成参数设定的多样性，即使是同一模型，在实际运行后所产生的最终表现亦可存在微小的差异。
 34 | 
 35 | | 模型名称                 | 综合得分 | AI与网络安全 | 业务连续性与应急响应恢复 | 供应链安全 | 加密技术与密钥管理 | 基础设施安全 | 威胁检测与预防 | 安全架构设计 | 数据安全和隐私保护 | 漏洞管理与渗透测试 | 系统安全及软件安全基础 | 访问控制与身份管理 | 中文题 | 英文题 |
 36 | | ------------------------ | -------- | ------------ | ------------------------ | ---------- | ------------------ | ------------ | -------------- | ------------ | ------------------ | ------------------ | ---------------------- | ------------------ | ------ | ------ |
 37 | | GPT4-8K                  | 87.57    | 91.58        | 84.28                    | 89.30      | 86.51              | 88.83        | 85.21          | 83.90        | 86.90              | 89.63              | 90.00                  | 86.56              | 87.96  | 82.19  |
 38 | | GPT3.5-Turbo-16K         | 80.59    | 80.69        | 81.27                    | 88.96      | 69.59              | 83.17        | 79.52          | 76.59        | 82.14              | 80.71              | 80.00                  | 78.31              | 80.62  | 80.14  |
 39 | | Qwen-14B-Chat            | 79.04    | 87.13        | 78.60                    | 87.63      | 68.49              | 81.33        | 79.67          | 74.15        | 76.68              | 77.80              | 77.00                  | 78.89              | 79.99  | 65.41  |
 40 | | Qwen1.5-14B-Chat         | 76.66    | 78.71        | 70.23                    | 81.27      | 76.13              | 78.00        | 77.53          | 70.73        | 77.58              | 75.77              | 75.33                  | 77.59              | 76.68  | 75.68  |
 41 | | Qwen1.5-MoE-A2.7B-Chat   | 74.63    | 74.75        | 72.24                    | 81.94      | 73.50              | 71.88        | 76.61          | 68.78        | 70.24              | 74.80              | 74.33                  | 79.50              | 75.99  | 55.14  |
 42 | | Baichuan2-13B-Chat       | 73.92    | 76.24        | 73.91                    | 80.27      | 60.09              | 76.50        | 76.94          | 71.71        | 75.69              | 70.55              | 70.67                  | 73.90              | 73.79  | 75.34  |
 43 | | 360Zhinao-7B-Chat-4K     | 66.37    | 71.29        | 66.33                    | 70.00      | 51.04              | 66.78        | 68.63          | 68.78        | 65.02              | 64.78              | 67.67                  | 68.14              | 66.68  | 61.99  |
 44 | | Mistral-7B-Instruct-v0.2 | 65.93    | 69.31        | 63.67                    | 72.76      | 57.78              | 70.43        | 64.40          | 62.44        | 63.44              | 63.71              | 63.67                  | 69.54              | 66.01  | 63.36  |
 45 | | Yi-6B-Chat               | 65.27    | 65.84        | 59.67                    | 72.76      | 68.80              | 64.84        | 63.85          | 60.00        | 62.85              | 64.68              | 63.00                  | 69.98              | 65.58  | 59.93  |
 46 | | ChatGLM3-6B              | 57.33    | 65.35        | 56.67                    | 68.44      | 47.78              | 59.87        | 61.47          | 61.46        | 57.71              | 50.81              | 50.33                  | 55.26              | 57.14  | 59.25  |
 47 | | SecGPT-13B               | 47.34    | 40.59        | 45.33                    | 59.14      | 41.54              | 47.60        | 47.34          | 45.85        | 43.08              | 46.77              | 46.00                  | 53.15              | 48.45  | 31.85  |
 48 | | Llama-2-13b-chat-hf      | 38.08    | 38.12        | 39.13                    | 30.43      | 34.11              | 37.67        | 39.00          | 37.07        | 35.52              | 38.57              | 33.33                  | 47.60              | 38.40  | 32.88  |
 49 | 
 50 | 
 51 | 
 52 | ## 数据
 53 | 
 54 | #### 下载
 55 | 
 56 | - 方法一：在[Hugging Face](https://huggingface.co/datasets/cseval/cs-eval)直接下载或使用接口加载数据集：
 57 | 
 58 |   - 直接下载：
 59 |   
 60 |   ```
 61 |     wget https://huggingface.co/datasets/cseval/cs-eval/resolve/main/cs-eval-questions.zip
 62 |   ```
 63 |   
 64 |   - 使用Hugging Face datasets加载数据集。示例如下:
 65 |   
 66 |   ```
 67 |   from datasets import load_dataset
 68 |   dataset=load_dataset(r"cseval/cs-eval")
 69 |   
 70 |   print(dataset['{test}'][0])
 71 |   ```
 72 |   
 73 | - 方法二：在[Model Scope](https://modelscope.cn/datasets/cseval/cs-eval/summary)上下载：
 74 | 
 75 |   - 直接下载：
 76 | 
 77 |   ```
 78 |   git clone https://www.modelscope.cn/datasets/cseval/cs-eval.git
 79 |   ```
 80 | 
 81 |   - 使用Model Scope SDK来加载数据：
 82 | 
 83 |   ```
 84 |   from modelscope.msdatasets import MsDataset
 85 |   ds =  MsDataset.load('cseval/cs-eval')
 86 |   ```
 87 | 
 88 | 
 89 | 
 90 | ## CS-Eval使用步骤
 91 | 
 92 | 1. 在Hugging Face或ModelScope上下载CS-Eval评估数据
 93 | 2. 适配模型推理格式
 94 | 3. 模型推理
 95 | 4. 整理推理结果格式
 96 | 5. 在CS-Eval平台上提交模型结果
 97 | 6. 获取评测结果
 98 | 7. 选择是否参与公开排行
 99 | 
100 | 
101 | 
102 | ## 如何提交
103 | 
104 | 您需要将整理后的模型推理结果转换成UTF-8编码的JSON文件，并按照以下格式编写。
105 | 
106 | ```
107 | ## 示例
108 | [
109 |     {
110 |       "question_id": "1",
111 |       "answer": "A"
112 |     },
113 |     {
114 |       "question_id": "123",
115 |       "answer": "对"
116 |     },
117 |     {
118 |       "question_id": "1234",
119 |       "answer": "是否涉及漏洞：是\n漏洞号：CVE-2024-22891\n影响的产品及版本：Nteract v.0.28.0"
120 |     }
121 | ]
122 | ```
123 | 
124 | 其中question_id是题号，answer是整理后的模型结果。
125 | 
126 | 请注意：
127 | 
128 | - 对于单项选择题，通常可以直接从模型的生成结果中使用正则表达式提取出正确答案选项。
129 | - 对于多项选择题，通常可以直接从模型的生成结果中使用正则表达式提取出多个正确答案选项。
130 | - 对于判断题，题目指令中有要求模型回答指定答案，通常可以取模型的首部或尾部的判断结果。
131 | - 对于知识抽取题，题目有要求按照指定格式回答，直接使用模型推理的原始结果文本。
132 | 
133 | 您在对选择题进行正则化时，可以通过筛选数据集prompt中的以下关键词来快速定位。
134 | 
135 | ```
136 | "单选题："
137 | "多选题："
138 | "Single-choice question:"
139 | ```
140 | 
141 | ## Licenses
142 | 
143 | [![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](https://lbesson.mit-license.org/)
144 | 
145 | 本项目遵循 [MIT License](https://lbesson.mit-license.org/).
146 | 
147 | [![CC BY-NC-SA 4.0](https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg)](http://creativecommons.org/licenses/by-nc-sa/4.0/)
148 | 
149 | CS-Eval数据集遵循 [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-nc-sa/4.0/).
150 | 
151 | 
152 | 
153 | ## 引用
154 | 
155 | 如果您在研究或技术报告等场景使用了我们的数据集，请标明引用。
156 | 
157 | ```bibtex
158 | @inproceedings{Yu2024CSEvalAC,
159 |   title={CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity},
160 |   author={Zhengmin Yu and Jiutian Zeng and Siyi Chen and Wenhan Xu and Dandan Xu and Xiangyu Liu and Zonghao Ying and Nan Wang and Yuan Zhang and Min Yang},
161 |   year={2024},
162 |   url={https://api.semanticscholar.org/CorpusID:274234403}
163 | }
164 | ```
165 | 
166 | 
167 | 
168 | ## 免责声明
169 | 
170 | 本平台及关联主体始终坚持合法、合规、积极、健康的宗旨，致力于推动大模型在网络安全领域的研究与应用，提升安全防护能力。为避免公众对本平台内容产生误导，特就此声明如下：
171 | 
172 | 1. 合法目的：我们提供的所有信息、资源、工具及服务，旨在促进大模型技术在网络安全领域的科学研究、技术创新、风险评估、防御策略制定等合法、有益活动，坚决反对任何利用大模型从事违法、侵权或破坏网络安全的行为。
173 | 2. 非诱导性：本平台严格禁止任何诱导、教唆他人实施网络攻击、入侵、破坏、非法获取数据等非法行为的内容。我们强调，所有与网络安全大模型评测集相关的内容均以促进行业发展、提供网络安全系统评测和促进技术交流为目的，不含有任何诱导、鼓励或暗示实施恶意攻击的成分。
174 | 3. 非恶意攻击教学：本平台明确声明，我们提供的所有内容均不涉及教授、演示或指导实施恶意网络攻击的技术手段，所有涉及攻击性的讨论均严格限定在合法的网络安全攻防演练、漏洞研究、风险评估等范畴内，旨在提高防御能力，而非用于攻击性目的。
175 | 4. 用户责任：用户在使用本平台服务过程中，必须严格遵守相关法律法规，不得利用平台资源进行任何违法、侵权或破坏网络安全的活动。对于用户违反本声明的行为，本平台有权采取包括但不限于警告、暂停服务、封禁账号等措施，并保留依法追究其法律责任的权利。
176 | 5. 免责声明：尽管我们努力确保平台内容的准确、合法、适宜，但用户在使用过程中，因自身行为导致的任何直接或间接损失，包括但不限于法律纠纷、财产损失、数据泄露、声誉损害等，本平台及关联主体不承担任何法律责任。用户应自行判断并承担使用平台资源可能产生的全部风险。
177 | 
178 | 
179 | 
180 | 我们真诚呼吁广大用户共同维护网络安全领域的良好秩序，合法、理性、负责任地使用大模型技术及相关资源。本免责声明的最终解释权归本平台所有，如有变动，恕不另行通知。
181 | 


--------------------------------------------------------------------------------
/dataset_example.md:
--------------------------------------------------------------------------------
  1 | ```
  2 | 单选题：
  3 | 
  4 | 假设我们有一个用户Alice，以下是她的行为时间线和活动：
  5 | 
  6 | 时间线：
  7 | - 9:00 AM: 使用了她未曾使用过的设备登录公司网络
  8 | - 9:30 AM: 大量的文件夹被下载到她的个人文件夹，包括一些敏感文件
  9 | - 10:00 AM: 在系统上执行了一些未授权的系统命令
 10 | - 11:00 AM: 尝试多次登录失败，同时账号被锁定
 11 | - 11:30 AM: 解锁后立即进行了大量的文件传输操作
 12 | 
 13 | Alice的账号出现异常行为的第一个迹象是什么？
 14 | A. 在系统上执行了一些未授权的系统命令
 15 | B. 多次登录失败，账号被锁定
 16 | C. 使用了她未曾使用过的设备登录公司网络
 17 | D. 解锁后立即进行了大量的文件传输操作
 18 | 
 19 | 请给出符合题意的唯一选项。
 20 | ```
 21 | 
 22 | ```
 23 | 单选题：
 24 | 
 25 | 下面是一个网络行为列表，我们逐个进行分析：
 26 | 
 27 | | 用户/ IP 地址   | 行为                                       | 类型    |
 28 | |-----------------|--------------------------------------------|---------|
 29 | | Alice           | 文件访问和编辑                             | xx    |
 30 | | 192.168.1.10    | 大量的文件下载                              | xx    |
 31 | | Bob             | 访问公司内部资源             | xx    |
 32 | | 192.168.1.20    | 非工作时间登录并进行大量数据传输         | xx    |
 33 | | Charlie         | 网站浏览行为                          | xx    |
 34 | | David           | 多次尝试使用错误的凭证登录               | xx    |
 35 | | 192.168.1.30    | 1分钟内5次数据传输                              | xx    |
 36 | 
 37 | 
 38 | 根据网络行为进行分析，哪个用户或IP地址的行为是异常的？
 39 | A) Alice
 40 | B) 192.168.1.20
 41 | C) Charlie
 42 | D) 192.168.1.30
 43 | 
 44 | 请给出符合题意的唯一选项。
 45 | ```
 46 | 
 47 | ````
 48 | 单选题：
 49 | 
 50 | 日志：
 51 | 
 52 | ```
 53 | 2023-09-15 12:01:00 INFO [User: Alice] Login successful
 54 | 2023-09-15 12:05:00 WARNING [User: Alice] Failed to access sensitive data
 55 | 2023-09-15 12:10:00 INFO [User: Bob] Login successful
 56 | 2023-09-15 12:15:00 INFO [User: Bob] Accessed salary.docx
 57 | 2023-09-15 12:20:00 INFO [User: Charlie] Login successful
 58 | 2023-09-15 12:25:00 WARNING [User: Charlie] Multiple failed login attempts
 59 | 2023-09-15 12:30:00 INFO [User: Alice] Accessed confidential.docx
 60 | 2023-09-15 12:35:00 INFO [User: Bob] Login successful
 61 | 2023-09-15 12:40:00 INFO [User: Bob] Accessed project_plan.docx
 62 | 2023-09-15 12:45:00 WARNING [User: Alice] Unauthorized attempt to access sensitive data
 63 | 2023-09-15 12:45:05 INFO [User: Alice] Accessed home page
 64 | 2023-09-15 12:45:08 INFO [User: Alice] Accessed "Friends" page
 65 | 2023-09-15 12:45:08 INFO [User: Alice] called sendMessage API
 66 | 2023-09-15 12:46:00 INFO [User: Bob] called sendMessage API
 67 | 2023-09-15 12:46:00 INFO [User: Bob] called sendMessage API
 68 | 2023-09-15 12:46:00 INFO [User: Bob] called sendMessage API
 69 | 2023-09-15 12:46:00 INFO [User: Bob] called sendMessage API
 70 | 2023-09-15 12:46:00 INFO [User: Bob] called sendMessage API
 71 | 2023-09-15 12:46:00 INFO [User: Bob] called sendMessage API
 72 | 2023-09-15 12:46:00 INFO [User: Bob] called sendMessage API
 73 | 2023-09-15 12:46:01 INFO [User: Bob] called sendMessage API
 74 | 2023-09-15 12:46:01 INFO [User: Bob] called sendMessage API
 75 | 2023-09-15 12:46:01 INFO [User: Bob] called sendMessage API
 76 | 2023-09-15 12:46:02 INFO [User: Bob] called sendMessage API
 77 | ```
 78 | 
 79 | 根据日志记录，哪位用户的行为最可能需要进一步重点调查和审查，对应的原因是什么？
 80 | 
 81 | A) Alice，尝试未经授权地访问敏感数据
 82 | B) Bob，没有走正常的好友页面发消息流程，直接调用了大量的消息发送接口，可能存在数据滥用或者恶意行为，需要进一步重点调查和审查其行为
 83 | C) Charlie，存在多次失败的登录尝试，可能是自动化测试工具在登录界面遇到阻碍，这种登录工具可能是被恶意团伙操控的
 84 | D) None of the above
 85 | 
 86 | 请给出符合题意的唯一选项。
 87 | ````
 88 | 
 89 | ```
 90 | 单选题：
 91 | 
 92 | $ ROPgadget --binary testexp --only “pop|ret"
 93 | Gadgets information
 94 | ============================================================
 95 | 0x000000000040062c : pop r12 ; pop r13 ; pop r14 ; pop r15 ; ret
 96 | 0x000000000040062e : pop r13 ; pop r14 ; pop r15 ; ret
 97 | 0x0000000000400630 : pop r14 ; pop r15 ; ret
 98 | 0x0000000000400632 : pop r15 ; ret
 99 | 0x000000000040062b : pop rbp ; pop r12 ; pop r13 ; pop r14 ; pop r15 ; ret
100 | 0x000000000040062f : pop rbp ; pop r14 ; pop r15 ; ret
101 | 0x00000000004004d5 : pop rbp ; ret
102 | 0x0000000000400633 : pop rdi ; ret
103 | 0x0000000000400631 : pop rsi ; pop r15 ; ret
104 | 0x000000000040062d : pop rsp ; pop r13 ; pop r14 ; pop r15 ; ret
105 | 0x0000000000400431 : ret
106 | 0x0000000000400442 : ret 0x200b
107 | 0x0000000000400505 : ret 0xc148
108 | 
109 | 如上述命令执行结果所示，我们只能通过“0x0000000000400633 : pop rdi ; ret”和“0x0000000000400631 : pop rsi ; pop r15 ; ret”对rdi和rsi传递前两个参数，第三个参数需要用rdx传递，但是没有这样的gadget。 
110 | 
111 | 其实在x64下有一些万能的gadgets可以利用。比如说我们用objdump -d ./testexp观察一下________这个函数。一般来说，只要程序调用了libc.so，程序都会有这个函数用来对libc进行初始化操作。 
112 | 
113 | 上述在执行ROP攻击时，若需要传递第三个参数至rdx寄存器，但是在当前gadget列表中没有直接的gadget可用，我们通常会利用哪个函数中的gadget来完成这一操作？
114 | 
115 | A) __libc_start_main
116 | B) __libc_csu_init
117 | C) __libc_csu_fini
118 | D) __libc_malloc
119 | 
120 | 请给出符合题意的唯一选项。
121 | ```
122 | 
123 | ````
124 | 单选题：
125 | 
126 | <|context|>
127 | ```
128 | type InfoResp struct {
129 |     GroupId         int
130 |     InfoId         int
131 |     InfoStatus     int
132 |     InfoContent    string
133 | }
134 | ```
135 | <|function|>
136 | ```
137 | func GetInfo(ctx *gin.Context, req *InfoReq) (*InfoResp, error) {
138 |    if req.InfoId == nil {
139 |       logs.CtxError(ctx, "InfoId is nil")
140 |       return nil, common.NewError(common.ReqError, "InfoId is nil")
141 |    }
142 |    [A]
143 |    infoResp, err := info.GetInfo(req.InfoId)
144 |    [B]
145 |    if err != nil {
146 |       logs.CtxError(ctx, "Get Info error")
147 |       return nil, err
148 |    }
149 |    [C]
150 |    if infoResp == nil {
151 |       logs.CtxError(ctx, "Info is nil")
152 |       return nil, common.NewError(common.RespError, "Info is nil")
153 |    }
154 |    [D]
155 |    return infoResp, nil
156 | }
157 | ```
158 | 修复代码块：
159 | ```
160 | if infoResp.GroupId != common.GetGroupID(ctx) {
161 |   logs.CtxError(ctx, "GroupId not match, req:%v, infoResp:%v", req, infoResp)
162 |   return nil, common.NewError(common.RespError, "GroupId not match")
163 | }
164 | ```
165 | 请针对上述函数和上下文，选择最恰当的使用修复代码块进行越权漏洞修复的位置。
166 | 
167 | A: [A]
168 | B: [B]
169 | C: [C]
170 | D: [D]
171 | 
172 | 请给出符合题意的唯一选项。
173 | ````
174 | 
175 | ````
176 | 单选题：
177 | 
178 | log4j-core 目前存在命令执行漏洞，具有非常高的风险。若您目前使用的是 log4j 日志框架，或您的项目内有 log4j-core 依赖，且版本在 2.0.0 ~ 2.14.1，建议您及时修复该漏洞。
179 | 查看方式
180 | 您可以通过以下方式确认您的环境是否需要修复 log4j 漏洞：
181 | - 查看源码
因为 log4j 可能存在间接依赖的情况，无法从 pom 中确定使用版本，您可以在项目根目录执行如下命令查看 log4j 版本：
```
mvn compile dependency:tree -Dincludes=org.apache.logging.log4j
```
查看输出中 log4j 的版本。例如，如下版本就是 2.12.1：

```
[INFO] org.example:Demo1:jar:1.0-SNAPSHOT
182 | [INFO] \- org.apache.logging.log4j:log4j-core:jar:2.12.1:compile
183 | [INFO]    \- org.apache.logging.log4j:log4j-api:jar:2.12.1:compile
```

184 | - 查看服务器上的 Jar 包
您可以查看项目服务器的 Jar 包，确认是否依赖 log4j-core：

```
ls /home/admin/release/run/target/boot/*/BOOT-INF/lib/ | grep log4j-core
```
185 | /home/admin/release/run/target/boot/* 为 Jar 包的路径，您需要根据自己项目修改。
186 | 
187 | # 修复方案
188 | 您可以通过以下任意方式修复 log4j 漏洞，时间充足的情况下，建议您通过升级 log4j-core 版本的方式进行修复。
189 | 
190 | - 升级 log4j-core 版本
191 | 您可以在最外层 pom 的 dependencyManagement 中增加以下内容：
192 |  
193 | <dependency>
194 |   <groupId>org.apache.logging.log4j</groupId>
195 |   <artifactId>log4j-core</artifactId>
196 |   <version>${version}</version>
197 | </dependency>
198 | 已修复漏洞的版本如下：
199 | 2.6.2_nonelookup2
200 | 2.7_nonelookup2
201 | 2.8_nonelookup2
202 | 2.8.1_nonelookup2
203 | 2.8.2_nonelookup2
204 | 2.9.0_nonelookup2
205 | 2.9.1_nonelookup2
206 | 2.10.0_nonelookup2
207 | 2.11.0_nonelookup2
208 | 2.11.1_nonelookup2
209 | 2.11.2_nonelookup2
210 | 2.12.0_nonelookup2
211 | 2.12.1_nonelookup2
212 | 2.13.0_nonelookup2
213 | 2.13.1_nonelookup2
214 | 2.13.2_nonelookup2
215 | 2.13.3_nonelookup2
216 | 2.14.1_nonelookup2
217 | 示例如下：
218 | ```
219 | <dependency>
220 |   <groupId>org.apache.logging.log4j</groupId>
221 |   <artifactId>log4j-core</artifactId>
222 |   <version>2.6.2_nonelookup2</version>
223 | </dependency>
224 | ```
225 | 
226 | - 排除 log4j-core 依赖
227 | 如果您没有使用 log4j 日志框架，但依赖了 log4j-core。您可以在查出依赖路径后，在依赖 log4j-core 的 dependency 中增加以下内容：
228 | ```
229 | <exclusions>
230 |      <exclusion>
231 |         <groupId>org.apache.logging.log4j</groupId>
232 |         <artifactId>log4j-core</artifactId>
233 |      </exclusion>
234 | </exclusions>
235 | ```
236 | 
237 | - 屏蔽出问题的 lookup 功能
如果您使用了 log4j-core 2.10.0 及以上版本，您可以在 Java 启动参数增加 -Dlog4j2.formatMsgNoLookups=true 配置，以屏蔽出问题的 lookup 功能。
238 | 
239 | 请阅读上述文档，并回答以下哪个版本是已修复漏洞的 log4j-core 版本？
240 | 
241 | A) 2.6.2_nonelookup1
242 | B) 2.12.1_nonelookup2
243 | C) 2.13.3_nonelookup1
244 | D) 2.14.0_nonelookup2
245 | 
246 | 请给出符合题意的唯一选项。
247 | ````
248 | 
249 | ```
250 | 单选题：
251 | 
252 | <?php
$name = $_GET['name'];
echo "Welcome $name<br>";
echo “<a href=“http://www.test.com/testdom/">Click to Download</a>";
?> 
253 | 该页面显示两行信息，从URI获取'name'参数，并在页面显示，同时显 示跳转到一条URL的链接。如果攻击者在代码中嵌入攻击脚本内容，输入时采用如下的参数: 
254 | index.php?name=guest<script>alert('attacked')</script> 
255 | 当用户点击该链接时，攻击者提交的脚本内容会被执行，带'attacked'的 告警提示框弹出。更进一步，如果攻击者提交一个URL实现修改链接。 用户将可能会跳转至攻击者提供的链接。 
256 | index.php?name=<script>window.onload=function(){
257 | var link=document.getElementsByTagName(“a");
258 | link[0].href=“http://attacker-site.com/";}</script> 
259 | 
260 | 在上述PHP代码中，存在一种常见的网络安全漏洞。这种漏洞是什么？
261 | A) SQL注入
262 | B) 跨站请求伪造 (CSRF)
263 | C) 分布式拒绝服务 (DDoS)
264 | D) 跨站脚本攻击 (XSS)
265 | 
266 | 请给出符合题意的唯一选项。
267 | ```
268 | 
269 | ```
270 | 单选题：
271 | 
272 | 某个网站的登录验证的SQL查询代码为: 
273 | strSQL = "SELECT * FROM users WHERE (name = ‘’’ + userName + ‘’’) and (pw = ‘’’+ passWord +’’’);” 
274 | 恶意填入
______
275 | 也就是实际上运行的SQL命令会变成下面这样的 
276 | strSQL = "SELECT * FROM users;" 
277 | 最终SQL注入漏洞导致无账号密码，也能够登录网站。 
278 | 
279 | 请问，恶意用户输入的是哪一对userName和passWord值？
280 | 
281 | A. userName = "admin' --"; passWord = "admin' --"
282 | B. userName = "1' DROP TABLE users; --"; passWord = "1' DROP TABLE users; --"
283 | C. userName = "1' OR '1'='1"; passWord = "1' OR '1'='1"
284 | D. userName = "'; SELECT * FROM users; --"; passWord = "'; SELECT * FROM users; --"
285 | 
286 | 请给出符合题意的唯一选项。
287 | ```
288 | 
289 | ````
290 | Single-choice question:
291 | 
292 | ```
293 | from datetime import date
294 | 
295 | today = date.today()
296 | target = date(today.year, 12, 25)
297 | 
298 | def show_message():
299 |     theTree = [0, 0, 1, 1, 3, 5, 7, 9, 13, 7,
300 |               11, 15, 19, 11, 15, 19, 11, 15,
301 |               19, 23, 27, 6, 6, 6, 0]
302 | 
303 |     for row in theTree:
304 |         gap_size = int((14 - (0.5 * (row + 1))))
305 |         print(" " * gap_size + "*" * row)
306 |     print(">>>>> MERRY CHRISTMAS <<<<<")
307 |     print(" ")
308 |     exit()
309 | 
310 | 
311 | def bomb():
312 |     if (today == target):
313 |         show_message()
314 | 
315 | 
316 | print("Running program as normal...")
317 | bomb()
318 | print("Nothing to see here…")
319 | ```
320 | 
321 | What is the condition that triggers the "logic bomb" in the provided Python code to display a festive greeting?
322 | 
323 | A) today = date.today()
324 | B) today == date(today.year, 12, 25)
325 | C) gap_size = int((14 - (0.5 * (row + 1))))
326 | D) print(">>>>> MERRY CHRISTMAS <<<<<")
327 | 
328 | Please provide the only option that meets the requirements of the question.
329 | ````
330 | 
331 | ```
332 | 单选题：
333 | 
334 | 以下是一个简单的 Python 代码示例，演示如何使用 VirusTotal API 进行文件扫描：
335 | 
336 | import requests
337 | 
338 | # 您的 VirusTotal API Key
339 | api_key = 'YOUR_API_KEY'
340 | 
341 | # 要扫描的文件的路径
342 | file_path = 'path_to_your_file'
343 | 
344 | # 将文件上传到 VirusTotal 进行扫描
345 | url = 'https://www.virustotal.com/vtapi/v2/file/scan'
346 | files = {'file': (file_path, open(file_path, 'rb'))}
347 | params = {'apikey': api_key}
348 | response = requests.post(url, files=files, params=params)
349 | 
350 | # 获取扫描报告
351 | resource = response.json()['resource']
352 | url = 'https://www.virustotal.com/vtapi/v2/file/report'
353 | params = {'apikey': api_key, 'resource': resource}
354 | response = requests.get(url, params=params)
355 | 
356 | # 输出扫描结果
357 | print(response.json())
358 | 
359 | 在上述 Python 代码示例中，要进行文件扫描操作，需要将文件上传到 VirusTotal。请问在代码中，哪一行负责将文件上传到 VirusTotal 进行扫描？
360 | 
361 | A. `api_key = 'YOUR_API_KEY'`
362 | B. `file_path = 'path_to_your_file'`
363 | C. `response = requests.post(url, files=files, params=params)`
364 | D. `response = requests.get(url, params=params)`
365 | 
366 | 请给出符合题意的唯一选项。
367 | ```
368 | 
369 | ```
370 | 单选题：
371 | 
372 | Zygote进程在启动的过程中，除了会创建一个虚拟机实例之外还会将Java _______ 加载到进程中并注册一些Android核心类的JNI方法。一个App进程被Zygote进程孵化出来的时候，不仅会获得Zygote进程中的虚拟机实例拷贝，还会与Zygote进程一起共享Java _______，也就是可以将XposedBridge.jar这个Jar包加载到每一个Android App进程中去。
373 | 
374 | A. Runtime
375 | B. Compiler
376 | C. SDK
377 | D. API
378 | 
379 | 请给出符合题意的唯一选项。
380 | ```
381 | 
382 | ```
383 | 单选题：
384 | 
385 | import "androguard"
386 | rule main
387 | {
388 | meta:
389 | description = "Identify Agent Smith core app"
390 | sample_analysis = "https://www.hybrid-analysis.com/sample/a3e95b5774c3f4d0f742fbc61ec0b3536deba4388840a398a8ec9c3eb351a177"
391 | 
392 | 
393 | strings:
394 | $a1 = "adsdk.zip"
395 | $a2 = "boot.zip"
396 | $a3 = "patch.zip"
397 | 
398 | $b1 = "com.infectionAds.AdsManagement"
399 | $b2 = "com.infectionAds.AdmobPulic"
400 | $b3 = "com.infectionapk.patchMain"
401 | 
402 | $c1 = /assets\/fonts\/DIsplay[0-9]*\.jpg/  //Encrypted malware
403 | condition:
404 | 2 of ($a*) and (any of ($b*) or any of ($c*))
405 | 
406 | }
407 | rule dropper
408 | {
409 | meta:
410 | description = "Identifies a few known dropper apps"
411 | sample_analysis = "https://www.hybrid-analysis.com/sample/850253669b80ea2bf3ab02b3035ee330a8b718d7690f3fc0bf5d11b29e71b6ca/5d262933038838e412e9d9d1"
412 | 
413 | 
414 | //strings:
415 | //$b1 = "androVM.vbox_dpi"
416 | //$b2 = "qemu.sf.fake_camera"
417 | 
418 | 
419 | condition:
420 | androguard.certificate.sha1("895d1abd26aaf7da4e52d37fa37d4e4a08bd5ca2") and
421 | (androguard.package_name("com.cool.temple007") or
422 | androguard.package_name("com.cool.rabbit.temple"))
423 | 
424 | }
425 | rule JaguarKillSwitch : dropper_variant
426 | {
427 | meta:
428 | description = "Identify (currently) dormant variants of Agent Smith droppers containing the 'Jaguar Kill Switch'"
429 | 
430 | 
431 | strings:
432 | $a1 = /com[\.\/]jaguar/
433 | $a2 = "hippo-sdk"
434 | 
435 | $b1 = /tt.androidcloud.net/
436 | $b2 = /sdk.ihippogame.com/
437 | $b3 = /sdk.soonistudio.com/
438 | 
439 | condition:
440 | all of ($a*) and any of ($b*)
441 | }
442 | 
443 | 这段代码是______语言编写的，通过______、______、______来检测恶意软件的特征。是针对名为“______”的恶意软件家族设计的。
444 | 
445 | A. GROOVY，特征码，正则表达式，逻辑条件，Jaguar Kill Switch
446 | B. YARA，字符串，正则表达式，逻辑条件，Jaguar Kill Switch
447 | C. PHP，特征码，正则表达式，逻辑条件，Agent Smith
448 | D. YARA，字符串，正则表达式，逻辑条件，Agent Smith
449 | 
450 | 请给出符合题意的唯一选项。
451 | ```
452 | 
453 | ```
454 | 判断题：
455 | 
456 | ABC科技公司
457 | 日期：  2030.6.6
458 | |  资产       |      金额    |  负债和股东权益  |      金额    |
459 | |------------|------------|----------------|------------|
460 | |流动资产    |100,000,00|流动负债         |xxx,xxx|
461 | |现金及现金等价物|       |应付账款         |            |
462 | |应收账款    |            |短期借款         |            |
463 | |存货        |            |应付利息         |            |
464 | |其他流动资产 |           |其他流动负债     |            |
465 | |合计流动资产 |           |合计流动负债     |            |
466 | |固定资产    |            |长期负债         |            |
467 | |设备        |            |长期借款         |            |
468 | |房地产      |            |应付债券         |            |
469 | |其他固定资产 |           |其他长期负债     |            |
470 | |合计固定资产 |           |合计长期负债     |            |
471 | |其他资产    |            |股东权益         |            |
472 | |            |            |普通股          |            |
473 | |合计资产    |            |留存收益         |            |
474 | |            |            |其他股东权益     |            |
475 | |            |            |合计股东权益     |            |
476 | |            |            |总负债和股东权益 |            |
477 | 
478 | 要求：请判断以上内容是否涉及公司内部数据或敏感信息？给出是或否
479 | ```
480 | 
481 | ````
482 | 单选题：
483 | 
484 | ```
485 | if (role == UserRoleEnum.CITY_MANAGER) {
486 |     permission = AclConstant.CITY_MANAGER_DATA_PERMISSION;
487 | } else if (role == UserRoleEnum.REGION_MANAGER) {
488 |     permission = AclConstant.REGION_MANAGER;
489 | } else {
490 |     throw new AssetsException("This role does not have data configuration permission:" + role);
491 | }
492 | ```
493 | 这段代码从访问控制和鉴权的角度来看，通过______和______来实现了基于角色的权限管理。
494 | 
495 | A. ACL, DAC
496 | B. ACL, MAC
497 | C. ACL, ABAC
498 | D. ACL, RBAC
499 | 
500 | 请给出符合题意的唯一选项。
501 | ````
502 | 
503 | ```
504 | 单选题：
505 | 
506 | 通过水平越权到权限更高的用户，水平权限提升攻击可以转变为垂直权限提升。例如，通过水平越权，攻击者可能可以得到一个可以重置或捕获属于其他用户的密码的账号。如果攻击者以管理用户为重置、捕获密码的目标并入侵其帐户，则他们可以获得管理访问权限，从而执行垂直权限提升。攻击者也可能在一个水平越权点造成垂直越权效果，比如：
507 | https://benchmark.com/myaccount?id=456
508 | 这是一个水平越权点，但如果管理员用户id为0或者1，此时id=0则会造成______，攻击者将获得对管理帐户页面的访问权限。此时页面可能会泄露管理员的密码或提供更改密码的方法，或者可能提供对特权功能的直接访问。
509 | 
510 | 空白处填入哪一项最恰当？
511 | 
512 | A. 水平越权
513 | B. 垂直越权
514 | C. 水平和垂直越权
515 | D. SQL注入
516 | 
517 | 请给出符合题意的唯一选项。
518 | ```
519 | 
520 | ````
521 | 单选题：
522 | 
523 | 在Spring Security中，有时候一个角色可能涵盖了多种其他角色。例如：admin 可能包含 user 的权限，为实现这一功能，我们可能会为 admin 用户增加 user 角色。
524 | 要实现这种配置，我们需要用到 RoleVoter 的扩展实现类，RoleHierarchyVoter。其配置方式：
525 | ```
526 | <bean id="roleVoter" class="org.springframework.security.access.vote.RoleHierarchyVoter">
527 |     <constructor-arg ref="roleHierarchy" />
528 | </bean>
529 | <bean id="roleHierarchy"
530 |         class="org.springframework.security.access.hierarchicalroles.RoleHierarchyImpl">
531 |     <property name="hierarchy">
532 |         <value>
533 |             ROLE_ADMIN > ROLE_STAFF
534 |             ROLE_STAFF > ROLE_USER
535 |             ROLE_USER > ROLE_GUEST
536 |         </value>
537 |     </property>
538 | </bean>
539 | ```
540 | 配置中出现了四种角色，ADMIN、STAFF、USER、GUEST，而这四种又有明显的包含于被包含关系，如此便可以使角色投票机制根据角色的层级进行决策。
541 | 
542 | 阅读上述材料，分析在Spring Security的配置中，如果要设置角色的层级关系，应该使用哪个类的实例？
543 | A. RoleHierarchyImpl
544 | B. RoleVoterImpl
545 | C. RoleHierarchyVoter
546 | D. RoleVoter
547 | 
548 | 请给出符合题意的唯一选项。
549 | ````
550 | 
551 | 


--------------------------------------------------------------------------------
/examples/result_extract_custom_example.py:
--------------------------------------------------------------------------------
  1 | import re
  2 | import json
  3 | 
  4 | def extract_singal_choice(gen):
  5 |     gen = gen.replace('a', 'A').replace('b', 'B').replace('c', 'C').replace('d', 'D')
  6 |     
  7 |     # 答案是A | 选项是A | 应该选A选项
  8 |     res = re.search(
  9 |         r"(?:(?:选|选择|选定)[：:]?\s*|(?:(?:答案|选项)(?![^ABCD]{0,10}?(?:不|非)[^ABCD]{0,10}?(?:是|选|为|：|:|】))[^ABCD]{0,10}?(?:是|选|为|：|:|】))[^ABCD]{0,10}?)(A|B|C|D)(?:选项)?(?:\)|。|\.|，|,|．|、|A|B|C|D|$|：|:|\)|）)",
 10 |         gen,
 11 |     )
 12 | 
 13 |     # A选项正确 | A选项符合题意
 14 |     if res is None:
 15 |         res = re.search(
 16 |             r"(A|B|C|D)(?:选?项)?(?![^ABCD]{0,4}?(?:不|非)[^ABCD]{0,4}?(?:正确|对[的，。：]|符合))[^ABCD]{0,4}?(?:正确|对[的，。：]|符合)",
 17 |             gen,
 18 |         )
 19 | 
 20 |     # 直接输出 A
 21 |     if res is None:
 22 |         res = re.search(r"^[\(（]?(A|B|C|D)(?:。|\)|）|\.|，|,|．|：|:|$)", gen)
 23 | 
 24 |     # 获取第一个出现的字母
 25 |     if res is None:
 26 |         res = re.search(r"(?<![a-zA-Z])(A|B|C|D)(?![a-zA-Z=])", gen)
 27 |     
 28 |     try:
 29 |         return res.group(1)
 30 |     except:
 31 |         print(f"res is {res}")
 32 |         return None
 33 | 
 34 | def extract_judge(gen):
 35 |     if gen[0] in ['对', '错', '是', '否']:
 36 |         return gen[0]
 37 |     else:
 38 |         for label in ['对', '错', '是', '否']:
 39 |             if label in gen:
 40 |                 return label
 41 |         return None
 42 | 
 43 | def extract_yes_or_no(gen):
 44 |     gen = gen.lower()
 45 |     if gen.startswith("yes"):
 46 |         return "Yes"
 47 |     elif gen.startswith("no"):
 48 |         return "No"
 49 |     else:
 50 |         if gen.find('yes') != -1:
 51 |             return "Yes"
 52 |         elif gen.find('no') != -1:
 53 |             return "No"
 54 |         else:
 55 |             return None
 56 | 
 57 | def extract_multiple_choices(gen):
 58 |     gen = gen.replace('a', 'A').replace('b', 'B').replace('c', 'C').replace('d', 'D')
 59 |     gen = gen.replace('、', '')
 60 |     gen = gen.replace(',', '')
 61 |     gen = gen.replace('，', '')
 62 |     gen = gen.replace(' ', '')
 63 | 
 64 |     choices = []
 65 |     if 'A.' in gen or 'A)' in gen:
 66 |         choices.append("A")
 67 |     if 'B.' in gen or 'B)' in gen:
 68 |         choices.append("B")
 69 |     if 'C.' in gen or 'C)' in gen:
 70 |         choices.append("C")
 71 |     if 'D.' in gen or 'D)' in gen:
 72 |         choices.append("D")
 73 | 
 74 |     res = ''.join(choices)
 75 | 
 76 |     if res == '':
 77 |         res = None
 78 |     else:
 79 |         return res
 80 | 
 81 |     res = None
 82 | 
 83 |     if res is None:
 84 |         pattern_list = [
 85 |             # 答案是A|B|C 选项是A|B|C 应该选A|B|C选项
 86 |             r"(?:(?:选|选择|选定)[：:]?\s*|(?:(?:答案|选项)(?![^ABCD\uFF21\uFF22\uFF23\uFF24]{0,10}?(?:不|非)[^ABCD\uFF21\uFF22\uFF23\uFF24]{0,10}?(?:是|选|为|：|:|】))[^ABCD\uFF21\uFF22\uFF23\uFF24]{0,10}?(?:是|选|为|：|:|】))[^ABCD\uFF21\uFF22\uFF23\uFF24]{0,10}?)([ABCD\uFF21\uFF22\uFF23\uFF24]+)(?:选项)?(?:\)|。|\.|，|,|．|、|$|：|:|\)|）)",
 87 |             # A|B|C选项正确 A|B|C选项符合题意
 88 |             r"([ABCD\uFF21\uFF22\uFF23\uFF24]+)(?:选?项)?(?![^ABCD\uFF21\uFF22\uFF23\uFF24]{0,4}?(?:不|非)[^ABCD\uFF21\uFF22\uFF23\uFF24]{0,4}?(?:正确|对[的，。：]|符合))[^ABCD\uFF21\uFF22\uFF23\uFF24]{0,4}?(?:正确|对[的，。：]|符合)",
 89 |             # 获取所有出现的字母
 90 |             r"(?<![a-zA-Z])([ABCD\uFF21\uFF22\uFF23\uFF24]+)(?![a-zA-Z=])"
 91 |         ]
 92 |     
 93 |         for pattern in pattern_list:
 94 |             res = re.search(pattern, gen)
 95 |             if res:
 96 |                 choices = res.group(1)
 97 |                 # 去重
 98 |                 choices = list(set(choices))
 99 |                 choices.sort()
100 |                 return ''.join(choices)
101 |     
102 |     return res
103 | 
104 | def extract_choice_list(l):
105 |     choice_list = []
106 |     for span in l:
107 |         if span.lower().startswith('a.') or span.lower().startswith('b.') or span.lower().startswith('b.') or span.lower().startswith('d.'):
108 |             choice_list.append(span)
109 |         if span.lower().startswith('a)') or span.lower().startswith('b)') or span.lower().startswith('b)') or span.lower().startswith('d)'):
110 |             choice_list.append(span)
111 | 
112 |     return choice_list
113 | 
114 | 
115 | # 读取原始答案文件
116 | input_file = 'cs-eval-questions_qwen2-72b_result.json'
117 | # 例如文件格式为：
118 | # [
119 | # {"id":xxx, "prompt":xxx, "response":xxx},
120 | # {"id":xxx, "prompt":xxx, "response":xxx}
121 | # ]
122 | 
123 | with open(input_file, 'r', encoding='utf-8') as f:
124 |     answers = json.load(f)
125 | 
126 | # 正则化答案
127 | normalized_answers = []
128 | 
129 | for answer in answers:
130 |     if answer["prompt"].split('\n')[0] in ['单选题：', 'Single-choice question:']:
131 |         temp_json = {}
132 |         temp_json["question_id"] = answer["id"]
133 |         temp_json["answer"] = extract_singal_choice(answer["response"])
134 |         normalized_answers.append(temp_json)
135 |     elif answer["prompt"].split('\n')[0] in ['多选题：']:
136 |         temp_json = {}
137 |         temp_json["question_id"] = answer["id"]
138 |         temp_json["answer"] = extract_multiple_choices(answer["response"])
139 |         normalized_answers.append(temp_json)
140 |     elif answer["prompt"].split('\n')[0] in ['判断题：']:
141 |         temp_json = {}
142 |         temp_json["question_id"] = answer["id"]
143 |         temp_json["answer"] = extract_judge(answer["response"])
144 |         normalized_answers.append(temp_json)
145 |     elif answer["prompt"].split('\n')[0] in ['Yes or No question:']:
146 |         temp_json = {}
147 |         temp_json["question_id"] = answer["id"]
148 |         temp_json["answer"] = extract_yes_or_no(answer["response"])
149 |         normalized_answers.append(temp_json)
150 |     elif answer["prompt"].split('\n')[0] in ['知识抽取题：', 'Knowledge extraction question:']:
151 |         temp_json = {}
152 |         temp_json["question_id"] = answer["id"]
153 |         temp_json["answer"] = answer["response"]
154 |         normalized_answers.append(temp_json)
155 | 
156 | # 将结果写入result.json文件
157 | with open(input_file+'_format.json', 'w', encoding='utf-8') as f:
158 |     json.dump(normalized_answers, f, ensure_ascii=False, indent=4)
159 | 


--------------------------------------------------------------------------------
/examples/result_extract_standard_example.py:
--------------------------------------------------------------------------------
  1 | import re
  2 | import json
  3 | 
  4 | def extract_singal_choice(gen):
  5 |     gen = gen.replace('a', 'A').replace('b', 'B').replace('c', 'C').replace('d', 'D')
  6 |     
  7 |     # 答案是A | 选项是A | 应该选A选项
  8 |     res = re.search(
  9 |         r"(?:(?:选|选择|选定)[：:]?\s*|(?:(?:答案|选项)(?![^ABCD]{0,10}?(?:不|非)[^ABCD]{0,10}?(?:是|选|为|：|:|】))[^ABCD]{0,10}?(?:是|选|为|：|:|】))[^ABCD]{0,10}?)(A|B|C|D)(?:选项)?(?:\)|。|\.|，|,|．|、|A|B|C|D|$|：|:|\)|）)",
 10 |         gen,
 11 |     )
 12 | 
 13 |     # A选项正确 | A选项符合题意
 14 |     if res is None:
 15 |         res = re.search(
 16 |             r"(A|B|C|D)(?:选?项)?(?![^ABCD]{0,4}?(?:不|非)[^ABCD]{0,4}?(?:正确|对[的，。：]|符合))[^ABCD]{0,4}?(?:正确|对[的，。：]|符合)",
 17 |             gen,
 18 |         )
 19 | 
 20 |     # 直接输出 A
 21 |     if res is None:
 22 |         res = re.search(r"^[\(（]?(A|B|C|D)(?:。|\)|）|\.|，|,|．|：|:|$)", gen)
 23 | 
 24 |     # 获取第一个出现的字母
 25 |     if res is None:
 26 |         res = re.search(r"(?<![a-zA-Z])(A|B|C|D)(?![a-zA-Z=])", gen)
 27 |     
 28 |     try:
 29 |         return res.group(1)
 30 |     except:
 31 |         print(f"res is {res}")
 32 |         return None
 33 | 
 34 | def extract_judge(gen):
 35 |     if gen[0] in ['对', '错', '是', '否']:
 36 |         return gen[0]
 37 |     else:
 38 |         for label in ['对', '错', '是', '否']:
 39 |             if label in gen:
 40 |                 return label
 41 |         return None
 42 | 
 43 | def extract_yes_or_no(gen):
 44 |     gen = gen.lower()
 45 |     if gen.startswith("yes"):
 46 |         return "Yes"
 47 |     elif gen.startswith("no"):
 48 |         return "No"
 49 |     else:
 50 |         if gen.find('yes') != -1:
 51 |             return "Yes"
 52 |         elif gen.find('no') != -1:
 53 |             return "No"
 54 |         else:
 55 |             return None
 56 | 
 57 | def extract_multiple_choices(gen):
 58 |     gen = gen.replace('a', 'A').replace('b', 'B').replace('c', 'C').replace('d', 'D')
 59 |     gen = gen.replace('、', '')
 60 |     gen = gen.replace(',', '')
 61 |     gen = gen.replace('，', '')
 62 |     gen = gen.replace(' ', '')
 63 | 
 64 |     choices = []
 65 |     if 'A.' in gen or 'A)' in gen:
 66 |         choices.append("A")
 67 |     if 'B.' in gen or 'B)' in gen:
 68 |         choices.append("B")
 69 |     if 'C.' in gen or 'C)' in gen:
 70 |         choices.append("C")
 71 |     if 'D.' in gen or 'D)' in gen:
 72 |         choices.append("D")
 73 | 
 74 |     res = ''.join(choices)
 75 | 
 76 |     if res == '':
 77 |         res = None
 78 |     else:
 79 |         return res
 80 | 
 81 |     res = None
 82 | 
 83 |     if res is None:
 84 |         pattern_list = [
 85 |             # 答案是A|B|C 选项是A|B|C 应该选A|B|C选项
 86 |             r"(?:(?:选|选择|选定)[：:]?\s*|(?:(?:答案|选项)(?![^ABCD\uFF21\uFF22\uFF23\uFF24]{0,10}?(?:不|非)[^ABCD\uFF21\uFF22\uFF23\uFF24]{0,10}?(?:是|选|为|：|:|】))[^ABCD\uFF21\uFF22\uFF23\uFF24]{0,10}?(?:是|选|为|：|:|】))[^ABCD\uFF21\uFF22\uFF23\uFF24]{0,10}?)([ABCD\uFF21\uFF22\uFF23\uFF24]+)(?:选项)?(?:\)|。|\.|，|,|．|、|$|：|:|\)|）)",
 87 |             # A|B|C选项正确 A|B|C选项符合题意
 88 |             r"([ABCD\uFF21\uFF22\uFF23\uFF24]+)(?:选?项)?(?![^ABCD\uFF21\uFF22\uFF23\uFF24]{0,4}?(?:不|非)[^ABCD\uFF21\uFF22\uFF23\uFF24]{0,4}?(?:正确|对[的，。：]|符合))[^ABCD\uFF21\uFF22\uFF23\uFF24]{0,4}?(?:正确|对[的，。：]|符合)",
 89 |             # 获取所有出现的字母
 90 |             r"(?<![a-zA-Z])([ABCD\uFF21\uFF22\uFF23\uFF24]+)(?![a-zA-Z=])"
 91 |         ]
 92 |     
 93 |         for pattern in pattern_list:
 94 |             res = re.search(pattern, gen)
 95 |             if res:
 96 |                 choices = res.group(1)
 97 |                 # 去重
 98 |                 choices = list(set(choices))
 99 |                 choices.sort()
100 |                 return ''.join(choices)
101 |     
102 |     return res
103 | 
104 | def extract_choice_list(l):
105 |     choice_list = []
106 |     for span in l:
107 |         if span.lower().startswith('a.') or span.lower().startswith('b.') or span.lower().startswith('b.') or span.lower().startswith('d.'):
108 |             choice_list.append(span)
109 |         if span.lower().startswith('a)') or span.lower().startswith('b)') or span.lower().startswith('b)') or span.lower().startswith('d)'):
110 |             choice_list.append(span)
111 | 
112 |     return choice_list
113 | 
114 | # 不同题型列表
115 | single_choice_id_list = [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 58, 72, 75, 76, 77, 78, 84, 85, 86, 87, 88, 101, 102, 103, 106, 109, 111, 112, 113, 119, 120, 121, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 246, 249, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 278, 281, 285, 288, 291, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 320, 321, 322, 323, 324, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 406, 409, 411, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 469, 471, 474, 476, 479, 480, 483, 485, 488, 490, 493, 496, 499, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 526, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 556, 559, 560, 562, 563, 564, 565, 566, 567, 568, 569, 570, 573, 574, 575, 576, 578, 579, 581, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 604, 606, 609, 610, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 666, 669, 671, 674, 676, 677, 679, 681, 682, 683, 684, 685, 687, 688, 689, 690, 691, 693, 695, 698, 700, 701, 702, 704, 705, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 767, 768, 769, 770, 774, 775, 777, 778, 781, 784, 785, 791, 793, 795, 796, 797, 800, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 1014, 1015, 1016, 1017, 1018, 1019, 1020, 1021, 1023, 1024, 1025, 1026, 1027, 1028, 1029, 1030, 1031, 1032, 1033, 1034, 1035, 1036, 1037, 1038, 1039, 1040, 1041, 1042, 1043, 1044, 1045, 1046, 1047, 1048, 1049, 1050, 1051, 1052, 1053, 1054, 1055, 1056, 1057, 1058, 1059, 1060, 1061, 1062, 1063, 1064, 1065, 1066, 1067, 1068, 1069, 1070, 1071, 1072, 1073, 1074, 1075, 1076, 1077, 1083, 1084, 1085, 1086, 1087, 1088, 1089, 1090, 1091, 1092, 1093, 1094, 1095, 1096, 1097, 1098, 1099, 1100, 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111, 1112, 1113, 1114, 1115, 1116, 1117, 1118, 1119, 1120, 1121, 1122, 1123, 1124, 1125, 1126, 1127, 1128, 1129, 1130, 1131, 1132, 1133, 1134, 1135, 1136, 1137, 1138, 1139, 1140, 1141, 1142, 1143, 1144, 1145, 1146, 1147, 1149, 1150, 1151, 1152, 1153, 1154, 1155, 1156, 1157, 1158, 1159, 1160, 1161, 1162, 1163, 1164, 1165, 1166, 1167, 1168, 1173, 1174, 1175, 1176, 1178, 1179, 1180, 1181, 1182, 1183, 1184, 1185, 1186, 1187, 1188, 1189, 1190, 1191, 1192, 1193, 1194, 1195, 1196, 1197, 1198, 1199, 1200, 1201, 1202, 1203, 1204, 1205, 1206, 1207, 1208, 1209, 1210, 1211, 1212, 1213, 1214, 1215, 1216, 1217, 1218, 1219, 1220, 1221, 1222, 1223, 1224, 1225, 1226, 1227, 1228, 1229, 1230, 1231, 1232, 1233, 1234, 1235, 1236, 1237, 1238, 1239, 1240, 1241, 1242, 1243, 1244, 1245, 1246, 1247, 1248, 1249, 1250, 1251, 1252, 1253, 1254, 1255, 1256, 1257, 1258, 1259, 1260, 1261, 1262, 1263, 1264, 1265, 1266, 1267, 1268, 1269, 1270, 1271, 1272, 1273, 1274, 1275, 1276, 1277, 1278, 1279, 1280, 1281, 1282, 1283, 1284, 1285, 1286, 1287, 1288, 1289, 1290, 1291, 1292, 1293, 1294, 1295, 1296, 1297, 1298, 1299, 1300, 1301, 1302, 1303, 1304, 1305, 1306, 1307, 1308, 1309, 1310, 1316, 1317, 1318, 1324, 1325, 1326, 1329, 1330, 1331, 1332, 1340, 1341, 1342, 1343, 1344, 1350, 1351, 1352, 1356, 1357, 1358, 1359, 1360, 1361, 1362, 1363, 1364, 1365, 1366, 1367, 1368, 1369, 1370, 1371, 1372, 1373, 1374, 1375, 1376, 1377, 1378, 1380, 1383, 1384, 1387, 1389, 1393, 1396, 1399, 1402, 1405, 1409, 1412, 1415, 1417, 1420, 1422, 1425, 1427, 1428, 1429, 1430, 1431, 1432, 1433, 1434, 1435, 1436, 1437, 1438, 1439, 1440, 1441, 1442, 1443, 1444, 1445, 1446, 1447, 1448, 1449, 1450, 1451, 1452, 1453, 1454, 1455, 1456, 1457, 1458, 1459, 1460, 1461, 1462, 1463, 1464, 1465, 1466, 1467, 1468, 1469, 1470, 1471, 1472, 1473, 1474, 1475, 1476, 1477, 1478, 1479, 1480, 1481, 1482, 1483, 1484, 1485, 1486, 1487, 1488, 1489, 1490, 1491, 1492, 1493, 1494, 1495, 1496, 1497, 1498, 1499, 1500, 1501, 1502, 1503, 1504, 1505, 1506, 1507, 1508, 1509, 1510, 1511, 1512, 1513, 1514, 1515, 1516, 1517, 1518, 1519, 1520, 1521, 1522, 1523, 1524, 1525, 1526, 1527, 1528, 1529, 1530, 1531, 1532, 1533, 1534, 1535, 1536, 1537, 1538, 1539, 1540, 1541, 1542, 1543, 1544, 1545, 1546, 1547, 1548, 1549, 1550, 1551, 1552, 1553, 1554, 1555, 1556, 1557, 1558, 1559, 1560, 1561, 1562, 1563, 1564, 1565, 1566, 1567, 1568, 1569, 1570, 1571, 1572, 1573, 1574, 1575, 1576, 1577, 1578, 1579, 1580, 1581, 1582, 1583, 1584, 1587, 1588, 1589, 1592, 1593, 1594, 1598, 1601, 1602, 1603, 1610, 1611, 1612, 1615, 1618, 1621, 1624, 1625, 1628, 1631, 1635, 1636, 1637, 1638, 1639, 1640, 1641, 1642, 1643, 1644, 1645, 1646, 1647, 1648, 1649, 1650, 1651, 1652, 1653, 1654, 1655, 1656, 1657, 1658, 1659, 1660, 1661, 1662, 1663, 1664, 1665, 1666, 1667, 1668, 1669, 1670, 1671, 1672, 1673, 1674, 1675, 1676, 1677, 1678, 1679, 1680, 1681, 1682, 1683, 1684, 1685, 1686, 1687, 1688, 1689, 1690, 1691, 1692, 1693, 1694, 1695, 1696, 1697, 1698, 1699, 1700, 1701, 1702, 1703, 1704, 1705, 1706, 1707, 1708, 1709, 1710, 1711, 1712, 1713, 1714, 1715, 1716, 1717, 1718, 1719, 1720, 1721, 1722, 1723, 1724, 1725, 1726, 1727, 1728, 1729, 1730, 1731, 1732, 1733, 1734, 1735, 1736, 1737, 1738, 1739, 1740, 1741, 1742, 1743, 1744, 1745, 1746, 1747, 1748, 1749, 1750, 1752, 1753, 1754, 1755, 1756, 1757, 1758, 1759, 1760, 1765, 1768, 1771, 1772, 1773, 1774, 1775, 1776, 1777, 1778, 1779, 1780, 1781, 1782, 1783, 1784, 1785, 1786, 1787, 1788, 1789, 1790, 1791, 1792, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1808, 1809, 1810, 1821, 1822, 1823, 1824, 1825, 1836, 1837, 1838, 1839, 1840, 1842, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1944, 1945, 1946, 1950, 1952, 1955, 1959, 1962, 1965, 1968, 1971, 1974, 1976, 1978, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2044, 2045, 2046, 2047, 2048, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2119, 2122, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2192, 2195, 2198, 2200, 2202, 2203, 2205, 2207, 2208, 2210, 2212, 2213, 2215, 2217, 2219, 2220, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 2304, 2305, 2306, 2307, 2308, 2309, 2326, 2327, 2328, 2329, 2330, 2331, 2332, 2333, 2334, 2335, 2336, 2337, 2338, 2339, 2340, 2341, 2342, 2343, 2344, 2345, 2346, 2347, 2348, 2349, 2350, 2351, 2352, 2353, 2354, 2355, 2356, 2357, 2358, 2359, 2360, 2361, 2362, 2363, 2364, 2365, 2366, 2367, 2368, 2369, 2370, 2371, 2372, 2373, 2374, 2375, 2376, 2377, 2378, 2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2409, 2410, 2411, 2412, 2413, 2414, 2416, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2437, 2438, 2439, 2440, 2441, 2442, 2443, 2444, 2445, 2446, 2447, 2448, 2449, 2450, 2451, 2452, 2453, 2454, 2455, 2456, 2457, 2458, 2459, 2460, 2461, 2462, 2463, 2464, 2465, 2466, 2467, 2468, 2469, 2470, 2471, 2472, 2473, 2474, 2475, 2476, 2477, 2478, 2479, 2480, 2481, 2482, 2483, 2484, 2485, 2486, 2487, 2488, 2489, 2490, 2491, 2492, 2493, 2494, 2495, 2496, 2497, 2498, 2499, 2500, 2501, 2502, 2503, 2504, 2505, 2506, 2507, 2508, 2509, 2510, 2511, 2512, 2513, 2514, 2515, 2516, 2517, 2518, 2519, 2520, 2521, 2522, 2523, 2524, 2525, 2526, 2527, 2528, 2529, 2530, 2531, 2532, 2533, 2534, 2535, 2536, 2537, 2538, 2539, 2540, 2541, 2542, 2543, 2544, 2545, 2546, 2547, 2548, 2549, 2550, 2551, 2552, 2553, 2554, 2555, 2556, 2557, 2558, 2559, 2560, 2561, 2562, 2563, 2564, 2565, 2566, 2567, 2568, 2569, 2570, 2571, 2572, 2573, 2574, 2575, 2576, 2577, 2578, 2579, 2580, 2581, 2582, 2583, 2584, 2585, 2586, 2587, 2588, 2589, 2590, 2591, 2592, 2593, 2594, 2595, 2596, 2597, 2598, 2599, 2600, 2601, 2602, 2603, 2604, 2605, 2606, 2607, 2608, 2609, 2610, 2611, 2612, 2613, 2614, 2615, 2616, 2617, 2618, 2619, 2620, 2621, 2622, 2623, 2624, 2625, 2626, 2627, 2628, 2629, 2631, 2633, 2634, 2635, 2636, 2637, 2638, 2639, 2640, 2641, 2642, 2643, 2644, 2645, 2646, 2647, 2648, 2649, 2650, 2651, 2652, 2653, 2654, 2655, 2656, 2657, 2658, 2659, 2660, 2661, 2662, 2663, 2664, 2665, 2666, 2667, 2668, 2669, 2670, 2671, 2672, 2673, 2674, 2675, 2676, 2677, 2678, 2679, 2680, 2681, 2682, 2683, 2684, 2685, 2686, 2687, 2690, 2691, 2692, 2693, 2694, 2695, 2700, 2703, 2704, 2705, 2706, 2707, 2708, 2709, 2710, 2711, 2712, 2713, 2714, 2715, 2716, 2717, 2718, 2719, 2720, 2722, 2723, 2724, 2725, 2727, 2728, 2729, 2730, 2731, 2732, 2733, 2734, 2735, 2736, 2737, 2738, 2739, 2740, 2741, 2742, 2743, 2744, 2745, 2746, 2747, 2748, 2749, 2750, 2751, 2752, 2757, 2758, 2759, 2760, 2761, 2762, 2763, 2764, 2765, 2766, 2767, 2768, 2769, 2770, 2771, 2772, 2794, 2795, 2796, 2797, 2798, 2799, 2800, 2801, 2802, 2803, 2814, 2815, 2816, 2817, 2818, 2819, 2820, 2821, 2822, 2823, 2824, 2825, 2826, 2827, 2828, 2829, 2830, 2831, 2832, 2833, 2834, 2835, 2836, 2837, 2838, 2839, 2840, 2841, 2842, 2843, 2844, 2845, 2846, 2847, 2848, 2849, 2850, 2851, 2862, 2863, 2864, 2865, 2867, 2868, 2869, 2870, 2871, 2872, 2873, 2874, 2875, 2876, 2877, 2878, 2879, 2880, 2881, 2882, 2883, 2884, 2885, 2886, 2887, 2888, 2889, 2890, 2891, 2892, 2893, 2894, 2895, 2896, 2897, 2898, 2899, 2900, 2901, 2902, 2903, 2904, 2905, 2906, 2907, 2908, 2909, 2910, 2911, 2912, 2913, 2914, 2915, 2916, 2917, 2918, 2919, 2920, 2921, 2922, 2923, 2924, 2925, 2926, 2927, 2928, 2929, 2930, 2931, 2932, 2933, 2934, 2935, 2936, 2937, 2938, 2939, 2940, 2941, 2942, 2943, 2944, 2945, 2946, 2947, 2948, 2949, 2950, 2951, 2952, 2953, 2954, 2955, 2956, 2957, 2958, 2959, 2960, 2961, 2962, 2963, 2964, 2965, 2966, 2967, 2968, 2969, 2970, 2971, 2972, 2973, 2974, 2975, 2976, 2977, 2978, 2979, 2980, 2981, 2982, 2983, 2984, 2985, 2986, 2987, 2988, 2989, 2990, 2991, 2992, 2993, 2994, 2995, 2996, 2997, 2998, 2999, 3000, 3001, 3002, 3003, 3004, 3005, 3006, 3007, 3008, 3009, 3010, 3012, 3013, 3014, 3015, 3016, 3017, 3018, 3019, 3020, 3021, 3022, 3023, 3024, 3025, 3026, 3027, 3028, 3029, 3030, 3031, 3032, 3033, 3038, 3039, 3040, 3041, 3042, 3043, 3044, 3050, 3051, 3052, 3053, 3054, 3055, 3056, 3057, 3059, 3065, 3066, 3067, 3073, 3074, 3075, 3076, 3077, 3078, 3079, 3080, 3081, 3082, 3083, 3084, 3085, 3086, 3087, 3088, 3089, 3090, 3091, 3092, 3093, 3094, 3095, 3096, 3097, 3098, 3099, 3100, 3101, 3102, 3103, 3104, 3105, 3106, 3107, 3108, 3109, 3110, 3111, 3112, 3113, 3114, 3115, 3116, 3117, 3118, 3119, 3120, 3121, 3122, 3123, 3124, 3125, 3128, 3135, 3136, 3137, 3138, 3141, 3142, 3143, 3144, 3145, 3146, 3147, 3148, 3149, 3150, 3151, 3152, 3153, 3154, 3155, 3156, 3157, 3158, 3159, 3160, 3161, 3162, 3163, 3164, 3165, 3166, 3167, 3168, 3169, 3170, 3171, 3172, 3173, 3174, 3175, 3176, 3177, 3178, 3179, 3180, 3181, 3182, 3183, 3184, 3185, 3186, 3187, 3188, 3189, 3190, 3191, 3192, 3193, 3194, 3195, 3196, 3197, 3198, 3199, 3200, 3201, 3202, 3203, 3204, 3205, 3206, 3207, 3208, 3209, 3210, 3211, 3212, 3213, 3214, 3215, 3216, 3217, 3218, 3219, 3227, 3228, 3229, 3230, 3231, 3238, 3239, 3240, 3241, 3242, 3243, 3244, 3245, 3246, 3247, 3248, 3249, 3250, 3251, 3252, 3253, 3254, 3255, 3256, 3257, 3258, 3259, 3260, 3261, 3262, 3263, 3264, 3265, 3266, 3267, 3268, 3269, 3270, 3271, 3272, 3273, 3274, 3275, 3276, 3277, 3278, 3279, 3280, 3281, 3282, 3283, 3284, 3285, 3286, 3287, 3288, 3289, 3290, 3296, 3297, 3301, 3302, 3303, 3316, 3317, 3318, 3319, 3320, 3321, 3322, 3323, 3324, 3325, 3326, 3327, 3328, 3329, 3330, 3331, 3332, 3333, 3334, 3335, 3336, 3337, 3338, 3339, 3340, 3341, 3342, 3343, 3344, 3345, 3346, 3347, 3348, 3349, 3350, 3351, 3352, 3353, 3354, 3355, 3356, 3357, 3358, 3359, 3360, 3361, 3362, 3363, 3364, 3365, 3366, 3367, 3368, 3369, 3370, 3371, 3372, 3373, 3374, 3375, 3376, 3377, 3378, 3379, 3380, 3381, 3382, 3383, 3384, 3385, 3386, 3391, 3392, 3396, 3397, 3398, 3399, 3400, 3401, 3402, 3403, 3404, 3405, 3406, 3407, 3408, 3409, 3410, 3411, 3412, 3413, 3414, 3415, 3416, 3417, 3418, 3419, 3420, 3421, 3422, 3423, 3424, 3425, 3426, 3427, 3428, 3429, 3430, 3431, 3432, 3433, 3434, 3435, 3436, 3437, 3438, 3439, 3440, 3441, 3442, 3443, 3444, 3445, 3446, 3447, 3448, 3449, 3450, 3451, 3452, 3453, 3454, 3455, 3456, 3457, 3458, 3459, 3460, 3461, 3462, 3463, 3464, 3465, 3466, 3467, 3468, 3469, 3470, 3471, 3472, 3473, 3474, 3475, 3476, 3477, 3478, 3479, 3480, 3481, 3482, 3483, 3484, 3485, 3486, 3487, 3488, 3489, 3490, 3491, 3492, 3493, 3494, 3495, 3496, 3497, 3498, 3499, 3504, 3505, 3506, 3507, 3508, 3509, 3510, 3511, 3512, 3513, 3514, 3515, 3516, 3517, 3518, 3519, 3520, 3521, 3522, 3523, 3524, 3525, 3526, 3527, 3528, 3529, 3530, 3531, 3532, 3533, 3534, 3535, 3536, 3537, 3538, 3539, 3540, 3541, 3542, 3543, 3544, 3545, 3546, 3547, 3550, 3558, 3560, 3563, 3564, 3565, 3566, 3567, 3568, 3569, 3571, 3572, 3573, 3574, 3575, 3576, 3577, 3578, 3579, 3580, 3581, 3582, 3583, 3584, 3585, 3586, 3587, 3588, 3589, 3590, 3591, 3592, 3593, 3594, 3595, 3596, 3597, 3598, 3599, 3600, 3601, 3602, 3603, 3604, 3605, 3606, 3607, 3608, 3609, 3610, 3611, 3612, 3613, 3614, 3615, 3616, 3617, 3618, 3619, 3620, 3621, 3622, 3629, 3635, 3636, 3637, 3638, 3639, 3640, 3641, 3642, 3643, 3648, 3649, 3650, 3651, 3652, 3653, 3654, 3655, 3656, 3657, 3658, 3659, 3660, 3661, 3662, 3663, 3664, 3665, 3666, 3667, 3668, 3669, 3670, 3671, 3672, 3673, 3674, 3675, 3676, 3678, 3680, 3682, 3683, 3684, 3685, 3686, 3687, 3689, 3690, 3691, 3692, 3693, 3694, 3695, 3696, 3697, 3698, 3699, 3700, 3701, 3702, 3703, 3704, 3705, 3706, 3707, 3708, 3709, 3710, 3711, 3712, 3713, 3714, 3715, 3716, 3717, 3718, 3719, 3720, 3721, 3722, 3723, 3724, 3725, 3726, 3727, 3728, 3729, 3730, 3731, 3732, 3733, 3734, 3735, 3736, 3737, 3738, 3739, 3740, 3741, 3742, 3743, 3744, 3745, 3746, 3747, 3748, 3750, 3751, 3752, 3753, 3754, 3755, 3756, 3757, 3758, 3759, 3760, 3761, 3762, 3763, 3765, 3766, 3767, 3770, 3771, 3772, 3773, 3774, 3775, 3776, 3777, 3778, 3779, 3780, 3781, 3782, 3783, 3784, 3785, 3786, 3787, 3788, 3789, 3790, 3791, 3792, 3793, 3795, 3796, 3797, 3798, 3799, 3800, 3801, 3802, 3804, 3805, 3806, 3809, 3810, 3811, 3814, 3815, 3816, 3817, 3818, 3819, 3820, 3821, 3822, 3823, 3824, 3825, 3826, 3827, 3828, 3829, 3830, 3831, 3832, 3833, 3834, 3835, 3836, 3837, 3838, 3839, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094, 4095, 4096, 4097, 4098, 4099, 4100, 4101, 4102, 4103, 4104, 4105, 4106, 4107, 4108, 4109, 4110, 4111, 4112, 4113, 4114, 4115, 4116, 4117, 4118, 4119, 4120, 4121, 4122, 4123, 4124, 4125, 4126, 4127, 4128, 4129, 4130, 4131, 4132, 4133, 4134, 4135, 4136, 4137, 4138, 4139, 4140, 4141, 4142, 4143, 4144, 4145, 4146, 4147, 4148, 4149, 4150, 4151, 4152, 4153, 4154, 4155, 4156, 4157, 4158, 4159, 4160, 4161, 4162, 4163, 4164, 4165, 4166, 4167, 4168, 4169, 4170, 4171, 4172, 4173, 4174, 4175, 4176, 4177, 4178, 4179, 4180, 4181, 4182, 4183, 4184, 4185, 4186, 4187, 4188, 4189, 4190, 4191, 4192, 4193, 4194, 4195, 4196, 4197, 4198, 4199, 4200, 4201, 4202, 4203, 4204, 4205, 4206, 4207, 4208, 4209, 4210, 4211, 4212, 4213, 4214, 4215, 4216, 4217, 4218, 4219, 4220, 4221, 4222, 4223, 4224, 4225, 4226, 4227, 4228, 4229, 4230, 4231, 4232, 4233, 4234, 4235, 4236, 4237, 4238, 4239, 4240, 4241, 4242, 4243, 4244, 4245, 4246, 4247, 4248, 4249, 4250, 4251, 4252, 4253, 4254, 4255, 4256, 4257, 4258, 4259, 4260, 4261, 4262, 4263, 4264, 4265, 4266, 4267, 4268, 4269, 4270, 4271, 4272, 4273, 4274, 4275, 4276, 4277, 4278, 4279, 4280, 4281, 4282, 4283, 4284, 4285, 4286, 4287, 4288, 4289, 4290, 4291, 4292, 4293, 4294, 4295, 4296, 4297, 4298, 4299, 4300, 4301, 4302, 4303, 4304, 4305, 4306, 4307, 4308, 4309, 4310, 4311, 4312, 4313, 4314, 4315, 4316, 4317, 4318, 4319, 4320, 4321, 4322, 4323, 4324, 4325, 4326, 4327, 4328, 4329, 4330, 4331, 4332, 4333, 4334, 4335, 4336, 4337, 4338, 4339, 4340, 4341, 4342, 4343, 4344, 4345, 4346, 4347, 4348, 4349, 4350, 4351, 4352, 4353, 4354, 4355, 4356, 4357, 4358, 4359, 4360, 4361, 4362, 4363, 4364, 4365, 4366, 4367, 4368, 4369]
116 | multi_choice_id_list = [1, 2, 59, 60, 61, 62, 63, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 105, 108, 110, 114, 115, 122, 123, 124, 245, 248, 251, 277, 280, 284, 287, 290, 293, 317, 318, 319, 335, 336, 337, 338, 339, 404, 405, 408, 412, 467, 472, 475, 478, 481, 486, 492, 495, 498, 501, 524, 527, 555, 558, 561, 571, 582, 607, 611, 668, 673, 680, 697, 703, 706, 708, 709, 762, 763, 764, 765, 766, 771, 772, 773, 782, 783, 792, 794, 798, 1022, 1078, 1079, 1148, 1169, 1170, 1171, 1177, 1314, 1315, 1322, 1323, 1328, 1335, 1336, 1337, 1338, 1339, 1345, 1346, 1349, 1353, 1354, 1355, 1379, 1381, 1386, 1388, 1390, 1391, 1394, 1397, 1400, 1403, 1406, 1408, 1410, 1413, 1416, 1418, 1421, 1423, 1426, 1585, 1586, 1590, 1591, 1599, 1600, 1607, 1608, 1609, 1613, 1616, 1619, 1622, 1626, 1629, 1806, 1807, 1817, 1818, 1819, 1820, 1833, 1834, 1835, 1843, 1937, 1938, 1939, 1940, 1943, 1948, 1951, 1954, 1957, 1960, 1963, 1966, 1969, 1972, 1975, 1977, 1979, 2039, 2040, 2049, 2050, 2051, 2052, 2117, 2120, 2123, 2191, 2193, 2196, 2223, 2224, 2225, 2226, 2314, 2315, 2316, 2317, 2318, 2319, 2321, 2322, 2323, 2389, 2390, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2415, 2417, 2773, 2774, 2782, 2783, 2784, 2785, 2786, 2787, 2788, 2789, 2790, 2791, 2792, 2793, 2805, 3011, 3063, 3064, 3071, 3072, 3139, 3140, 3220, 3221, 3232, 3233, 3291, 3292, 3298, 3299, 3304, 3305, 3306, 3312, 3387, 3388, 3393, 3394, 3395, 3500, 3501, 3551, 3552, 3554, 3555, 3556, 3559, 3561, 3570, 3623, 3644, 3645]
117 | judge_id_list = [57, 64, 65, 66, 67, 68, 69, 70, 71, 73, 74, 104, 107, 116, 117, 118, 125, 126, 127, 128, 247, 250, 276, 279, 282, 283, 286, 289, 292, 315, 316, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 403, 407, 410, 466, 468, 470, 473, 477, 482, 484, 487, 489, 491, 494, 497, 500, 525, 557, 572, 577, 580, 583, 603, 605, 608, 665, 667, 670, 672, 675, 678, 686, 692, 694, 696, 699, 707, 710, 776, 779, 780, 786, 787, 788, 789, 790, 799, 801, 1080, 1081, 1082, 1172, 1311, 1312, 1313, 1319, 1320, 1321, 1327, 1333, 1334, 1347, 1348, 1382, 1385, 1392, 1395, 1398, 1401, 1404, 1407, 1411, 1414, 1419, 1424, 1595, 1596, 1597, 1604, 1605, 1606, 1614, 1617, 1620, 1623, 1627, 1630, 1632, 1633, 1634, 1811, 1812, 1813, 1814, 1815, 1816, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1841, 1844, 1941, 1942, 1947, 1949, 1953, 1956, 1958, 1961, 1964, 1967, 1970, 1973, 1980, 2041, 2042, 2043, 2053, 2054, 2055, 2056, 2118, 2121, 2124, 2194, 2197, 2199, 2201, 2204, 2206, 2209, 2211, 2214, 2216, 2218, 2221, 2222, 2310, 2311, 2312, 2313, 2320, 2324, 2325, 2391, 2392, 2393, 2394, 2395, 2403, 2404, 2405, 2406, 2407, 2408, 2630, 2632, 2688, 2689, 2696, 2697, 2698, 2699, 2701, 2702, 2721, 2726, 2753, 2754, 2755, 2756, 2775, 2776, 2777, 2778, 2779, 2780, 2781, 2804, 2806, 2807, 2808, 2809, 2810, 2811, 2812, 2813, 2853, 2854, 2855, 2856, 2857, 2858, 2859, 2860, 2861, 2866, 3034, 3035, 3036, 3037, 3045, 3046, 3047, 3048, 3049, 3058, 3060, 3061, 3062, 3068, 3069, 3070, 3126, 3130, 3131, 3132, 3133, 3134, 3222, 3223, 3224, 3225, 3226, 3234, 3235, 3236, 3237, 3293, 3294, 3295, 3300, 3307, 3308, 3309, 3310, 3311, 3313, 3314, 3315, 3389, 3390, 3502, 3503, 3548, 3549, 3553, 3557, 3562, 3624, 3625, 3626, 3627, 3628, 3630, 3631, 3632, 3633, 3634, 3646, 3647, 3677, 3679, 3681, 3688, 3749, 3764, 3768, 3769, 3794, 3803, 3807, 3808, 3812, 3813, 3847]
118 | en_judge_id_list = [2852, 3127, 3129]
119 | knowledge_extract_id_list = [1751, 1761, 1762, 1763, 1764, 1766, 1767, 1769, 1770]
120 | 
121 | # 读取原始答案文件
122 | input_file = 'qwen2-7b-cs-eval.json'
123 | # 例如文件格式为：
124 | # [
125 | # {"question_id":xxx, "answer":xxx},
126 | # {"question_id":xxx, "answer":xxx}
127 | # ]
128 | 
129 | with open(input_file, 'r', encoding='utf-8') as f:
130 |     answers = json.load(f)
131 | 
132 | # 正则化答案
133 | normalized_answers = []
134 | for answer in answers:
135 |     if answer["question_id"] in single_choice_id_list:
136 |         temp_json = {}
137 |         temp_json["question_id"] = answer["question_id"]
138 |         temp_json["answer"] = extract_singal_choice(answer["answer"])
139 |         normalized_answers.append(temp_json)
140 |     elif answer["question_id"] in multi_choice_id_list:
141 |         temp_json = {}
142 |         temp_json["question_id"] = answer["question_id"]
143 |         temp_json["answer"] = extract_multiple_choices(answer["answer"])
144 |         normalized_answers.append(temp_json)
145 |     elif answer["question_id"] in judge_id_list:
146 |         temp_json = {}
147 |         temp_json["question_id"] = answer["question_id"]
148 |         temp_json["answer"] = extract_judge(answer["answer"])
149 |         normalized_answers.append(temp_json)
150 |     elif answer["question_id"] in en_judge_id_list:
151 |         temp_json = {}
152 |         temp_json["question_id"] = answer["question_id"]
153 |         temp_json["answer"] = extract_yes_or_no(answer["answer"])
154 |         normalized_answers.append(temp_json)
155 |     elif answer["question_id"] in knowledge_extract_id_list:
156 |         temp_json = {}
157 |         temp_json["question_id"] = answer["question_id"]
158 |         temp_json["answer"] = answer["answer"]
159 |         normalized_answers.append(temp_json)
160 | 
161 | # 将结果写入result.json文件
162 | with open(input_file+'_format.json', 'w', encoding='utf-8') as f:
163 |     json.dump(normalized_answers, f, ensure_ascii=False, indent=4)
164 | 


--------------------------------------------------------------------------------
/resources/logo.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/CS-EVAL/CS-Eval/622da7e646ac00184a7bcd9f5d07619bc8d0900a/resources/logo.jpg


--------------------------------------------------------------------------------
/resources/logo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/CS-EVAL/CS-Eval/622da7e646ac00184a7bcd9f5d07619bc8d0900a/resources/logo.png


--------------------------------------------------------------------------------
/resources/overview.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/CS-EVAL/CS-Eval/622da7e646ac00184a7bcd9f5d07619bc8d0900a/resources/overview.jpg


--------------------------------------------------------------------------------
/resources/overview.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/CS-EVAL/CS-Eval/622da7e646ac00184a7bcd9f5d07619bc8d0900a/resources/overview.png


--------------------------------------------------------------------------------
/submission_example.json:
--------------------------------------------------------------------------------
 1 | [
 2 |     {
 3 |       "question_id": "1",
 4 |       "answer": "A"
 5 |     },
 6 |     {
 7 |       "question_id": "123",
 8 |       "answer": "对"
 9 |     },
10 |     {
11 |       "question_id": "235",
12 |       "answer": "ABD"
13 |     },
14 |     {
15 |       "question_id": "1234",
16 |       "answer": "是否涉及漏洞：是\n漏洞号：CVE-2024-22891\n影响的产品及版本：Nteract v.0.28.0"
17 |     }
18 | ]
19 | 


--------------------------------------------------------------------------------