.
29 | pad_id (int or None): The word id used to pad the sequences. If None, it means there is no padding. Default: None.
30 | device (str, optional): The device used for running `paddle_model`, options: ``cpu``, ``gpu``, ``gpu:0``, ``gpu:1`` etc. Default: None.
31 | batch_size (int, optional): Number of samples to forward each time. Default: 50
32 | """
33 | TokenInterpreter.__init__(self, paddle_model, device)
34 |
35 | # build predict function
36 | self.normlime = self._build_predict_fn(paddle_model, device)
37 |
38 | self.batch_size = batch_size
39 | self.preprocess_fn = preprocess_fn
40 | self.unk_id = unk_id
41 | self.pad_id = pad_id
42 |
43 | def interpret(self, data, num_samples=500, temp_data_file='all_lime_weights.npz', save_path='normlime_weights.npy'):
44 | """Main function of the interpreter.
45 | Args:
46 | data ([type]): The inputs of the paddle_model.
47 | labels ([type], optional): The target label to analyze. If None, the most likely label will be used. Default: None.
48 | num_samples (int, optional): LIME sampling numbers. Larger number of samples usually gives more accurate interpretation. Default: 1000
49 | temp_data_file (str, optinal): The .npz file to save/load the dictionary where key is word ids joined by '-' and value is another dictionary with lime weights. Default: 'all_lime_weights.npz'
50 | save_path (str, optional): The .npy path to save the normlime weights. It is a dictionary where the key is label and value is segmentation ids with their importance. Default: 'normlime_weights.npy'
51 |
52 | Returns:
53 | [NormLIMEResult] NormLIME weights: {label_i: weights on features}
54 |
55 | """
56 |
57 | normlime_weights = self.normlime.interpret(data,
58 | self.preprocess_fn,
59 | unk_id=self.unk_id,
60 | pad_id=self.pad_id,
61 | num_samples=num_samples,
62 | batch_size=self.batch_size,
63 | temp_data_file=temp_data_file,
64 | save_path=save_path)
65 |
66 | normresult = NormLIMEResult(attributions=normlime_weights)
67 | return normresult
68 |
69 | def _build_predict_fn(self, paddle_model, device='gpu'):
70 | try:
71 | from interpretdl import NormLIMENLPInterpreter
72 | except ImportError as e:
73 | import sys
74 | sys.stderr.write(
75 | '''Warning with import interpretdl: please install interpretdl firstly. cmd: pip install -U interpretdl'''
76 | )
77 | raise e
78 | return NormLIMENLPInterpreter(paddle_model, device)
79 |
--------------------------------------------------------------------------------
/tutorials/README.md:
--------------------------------------------------------------------------------
1 | # 应用案例
2 | 基于对模型预测依赖证据的分析，TrustAI提供了模型缺陷识别方案及对应的优化方案。在该目录，我们通过具体实例介绍几种方案，一是希望开发者可直接利用这些方案对其任务模型进行优化，二是希望启发研究者探索证据分析的更多价值。
3 | * 训练数据缺陷识别及针对性优化策略
4 | * [训练数据覆盖不足识别及有效数据增强](#训练数据覆盖不足识别及有效数据增强)
5 | * [训练数据中脏数据识别及标注修正](#训练数据中脏数据识别及标注修正)
6 | * [训练数据分布偏置识别及偏置消除](#训练数据偏置识别及偏置消除)
7 | * [数据权重修正](#数据权重修正)
8 | * [数据分布修正](#数据分布修正)
9 | * 基于证据指导的预测机制优化
10 | * [证据识别及基于证据的预测](#证据识别及基于证据的预测)
11 |
12 | 同样地，我们也探索了证据分析的其他价值，如：
13 | * [基于证据指导的模型增强方案](#基于证据指导的模型增强方案)
14 | * [基于证据指导的预测错误数据识别](#基于证据指导的预测错误数据识别)
15 |
16 |
17 | ## 训练数据覆盖不足识别及有效数据增强
18 | ### 方法介绍
19 | 训练数据覆盖不足会导致模型在对应的测试数据上表现不好。数据扩充是提升模型效果首选方法，然而数据标注是一个费时费力的工作，如何标注更少的数据带来更大的效果提升是大多数NLP开发者面临的难题。
20 |
21 | TrustAI可识别因训练数据覆盖不足而导致的预测效果差的测试样本（这些样本构成的集合称为目标集），并能帮助开发者从未标注数据中选择有效数据进行标注，提高训练数据对目标集的覆盖度，进而提升模型效果。
22 |
23 | ### 方法效果
24 |
25 | 由于标注数据成本高昂，下表给出了基于相似度计算任务LCQMC数据集进行的模拟实验效果。实验基于ERNIE-3.0-base-zh在LCQMC训练数据上微调得到模型，在LCQMC测试集和DuQM鲁棒性数据集上进行效果评估，评估指标为准确率。
26 |
27 |
28 | | 数据集 | 数据量 | LCQMCdev | LCQMCtest | DuQM | 目标集 |
29 | | :-------: | :-------: | :-----: | :-----: |:-----: |:-----: |
30 | | 基线 | 5000 | 86.42% | 84.49% | 69.17% | 55.19% |
31 | | 基线 + 随机1000条 | 6000 | 86.76% | 85.05% | 69.23% | 55.20% |
32 | | 基线 + 策略1000条 | 6000 | 87.04% | 85.58% | 70.20% | 69.60% |
33 |
34 | 实验结论：增加20%有效训练数据，该方案将目标集效果提升14.40%（随机选择同等规模数据加入训练数据，效果仅提升0.01%）；同时在整个测试集上，该方案将效果提升1.03%（随机选择方案仅提升0.06%）。
35 |
36 | 同时，该策略接入了PaddleNLP的分类系统，在多分类、多标签分类、及层次分类任务上完成了效果验证，效果如图1所示：通过TrustAI提供的有效数据选择策略，增加10%训练数据带来的效果提升大于随机增加20%训练数据的效果，也就是说，该策略能够**节省一半标注成本**。
37 |
38 | 
39 | 图1 在三个常见分类任务上应用“数据覆盖不足识别及有效数据增强”策略的效果
40 |
41 |
42 |
43 | 详细方案和实验介绍见应用示例[训练数据覆盖不足识别及有效数据增强](./sparse_data_identification)。
44 |
45 |
46 |
47 | ## 训练数据中脏数据识别及标注修正
48 | ### 方法介绍
49 | 训练数据标注质量对模型效果有较大影响，往往会成为模型效果提升的瓶颈。但当标注数据规模较大时，数据检查就成为一个难题。
50 |
51 | TrustAI提供了脏数据（即标注质量差的数据）自动识别功能，降低人工检查数据成本。如图2所示，在三个公开数据集上，TrustAI提供的脏数据识别策略，其识别的脏数据比例远高于随机选择策略。
52 |
53 |
54 |
55 | 
56 | 图2 在3个数据集上，不同策略识别的脏数据效果
57 |
58 |
59 | ### 方法效果
60 |
61 | 下表给出了基于相似度计算任务LCQMC数据集上进行的实验效果。实验基于ERNIE-3.0-base-zh在LCQMC训练数据上微调得到模型，并在LCQMC测试集和DuQM鲁棒性数据集上评估效果，评估指标为准确率。
62 |
63 |
64 | | 数据集 | LCQMCdev | LCQMCtest | DuQM |
65 | | :-------: | :-----: | :-----: |:-----: |
66 | | 基线 | 86.42% | 84.49% | 69.17% |
67 | | 数据修正 | 87.76% | 86.62% | 73.18% |
68 |
69 | 结果说明：对候选脏数据（规模为原始训练集的10%）进行人工标注修正，数据修正后重新训练模型，在LCQMC测试集上效果提升2.13%，在DuQM数据集上效果提升4.01%。
70 |
71 | 同时，该策略接入了PaddleNLP的分类系统，在多分类、多标签分类、及层次分类任务上完成了效果验证，效果如图3所示。
72 |
73 | 
74 | 图3 在三个常见分类任务上应用“脏数据识别及标注修正”策略的效果
75 |
76 |
77 | 详细方案和实验介绍见应用示例[训练数据中脏数据识别](./dirty_data_identification)。
78 |
79 |
80 |
81 |
82 | ## 训练数据偏置识别及偏置消除
83 | ### 方法介绍
84 | 研究表明，神经网络模型会利用数据集中的偏置作为预测捷径，如在情感分析任务中，遇到否定词模型会倾向预测为“负向”情感。这种偏置会导致模型没有真正理解语言，导致模型的鲁棒性降低。
85 |
86 | TrustAI提供了数据权重修正和数据分布修正两种优化策略，在不需要人工介入的条件下，缓解训练数据偏置对模型训练的影响，提升模型的语义理解能力，进而提升模型的鲁棒性。
87 | * 数据权重修正：降低偏置样本对训练loss的影响，即减少模型从偏置样本中学习。具体方案详见[Du, Yanrui, et al. 2022](https://arxiv.org/abs/2205.12593)，其提供了`lls_d`和`lls_d_f`两种样本偏置度计算策略，前者考虑了词的有偏性，而后者同时考虑词的有偏性和频次。
88 | * 数据分布修正：通过对非偏置数据多次重复采样，使训练数据分布尽量均衡。
89 |
90 | ### 方法效果 - 数据权重修正
91 |
92 | 实验基于ERNIE-3.0-base-zh在相似度计算任务LCQMC数据集上微调得到基线模型，在LCQMC测试集和DuQM鲁棒性数据集上评估效果，评估指标为准确率。
93 |
94 | 效果如下表所示：相比于基线，数据权重修正后，模型在鲁棒性数据集DuQM上准确率提升0.94%。
95 |
96 | | 数据集 | LCQMCdev | LCQMCtest | DuQM |
97 | | :-------: | :-------: | :-------: | :-------: |
98 | | 基线 | 90.93% | 87.06% | 73.82% |
99 | | lls_d | 90.76% | 87.58% | 74.76% |
100 | | lls_d_f | 90.80% | 87.22% | 74.44% |
101 |
102 | 详细见应用示例[数据权重修正](./data_bias_identification/data_distribution_correction)。
103 |
104 |
105 |
106 | ### 方法效果 - 数据分布修正
107 |
108 | 实验基于ERNIE-3.0-base-zh在情感分析任务ChnsentiCorp数据集上微调得到基线模型，在情感分析鲁棒性数据集上评估效果，评估指标为准确率。
109 |
110 | 效果如下表所示：相比于基线，数据分布修正后，模型在鲁棒性数据集上准确率提升1.41%。
111 | | 数据集 | 鲁棒性数据集 |
112 | | :-------: | :-------: |
113 | | 基线 | 69.97 |
114 | | 分布修正 | 71.38 |
115 |
116 | 详细见应用示例[数据分布修正](./data_bias_identification/data_distribution_correction)。
117 |
118 |
119 |
120 | ## 证据识别及基于证据的预测
121 |
122 | ### 方法介绍
123 |
124 | 在长文本理解问题上，输入中的冗余信息往往会干扰模型预测，导致模型鲁棒性差。如在机器阅读理解(MRC)任务中，模型容易受到输入中扰动信息干扰，即输入中加入一些与答案生成无关的信息，模型生成的答案却可能发生改变。
125 |
126 | 为了降低模型受无关信息干扰带来的影响，TrustAI构建“证据识别-基于证据的预测”二阶段流程。首先，通过证据抽取识别输入中有效信息，排除冗余数据；然后基于识别的有效信息进行最终答案生成，提高模型鲁棒性。
127 |
128 | ### 方法效果
129 | 我们在MRC任务上做了验证，基于ERNIE-3.0-base-zh在DuReader-robust训练数据上微调了基线模型，在DuReader-robust的验证集合、测试集合和Challenge Test（DuReader-chechlist）集合上做了效果验证评估，评估指标为答案的EM（exactly match）。
130 |
131 |
132 | | 模型 | DuReader-robust dev EM | DuReader-robust Test EM | **DuReader-checklist dev EM** |
133 | | :----------------: | ---------------------- | ----------------------- | :---------------------------: |
134 | | roberta-base | 73.18 | 45.97 | 27.56 |
135 | | Selector-Predictor | 74.31 | 50.91 | 31.04 |
136 |
137 | 实验结论：“证据识别-基于证据的预测”二阶段方案，将模型在测试集上的效果提升4.94%，同时将训练的模型直接在DuReader Checklist数据集评估，相较于官方基线汇报结果，EM提升3.48%。
138 |
139 | 详细见应用示例[解决文本冗余导致精度下降的问题](./redundancy_removal)。
140 |
141 |
142 | ## 基于证据指导的模型增强方案
143 | ### 方法介绍
144 | 经过对多个模型预测依赖证据的评估，发现NN模型提供证据的合理性偏弱。为进一步提高证据的合理性，TrustAI提供了基于证据指导的模型增强方案（[Jayaram etc. 2021](https://aclanthology.org/2021.emnlp-main.450/)），即标注少量证据数据，通过联合学习原始任务和证据学习任务，用证据学习目标指导模型依赖合理的证据进行预测，提升模型可解释性。
145 |
146 |
147 | 
148 |
149 |
150 | ### 方法效果
151 |
152 | 实验基于ERNIE-2.0-EN-Base在英文情感分析SST数据集上微调得到基线模型，然后选择1000条训练数据进行证据标注，在这些数据上进行证据学习。最终，在500条标有证据的验证数据上进行了效果评估。评估指标除了模型预测准确率外，还包括可解释评估指标，即证据的合理性、充分性和完备性。
153 |
154 | 实验结果如下表：在加入证据指导后，模型预测效果略有提升，准确率提升0.5%；模型可解释性提升明显：证据合理性提升5.0%、充分性降低0.185（该指标越低越好）、完备性提升0.044。
155 |
156 | | 数据集 | 准确率 | 合理性 | 充分性 | 完备性 |
157 | | :-------: | :-----: | :-----: | :-----: | :-----: |
158 | | base | 93.5% | 26.1% | 0.367 | 0.118 |
159 | | base + maw loss | 94.0% | 31.1% | 0.182 | 0.162 |
160 |
161 | 应用的详细示例见[基于证据指导的模型增强方案](./enhanced_by_rationale)。
162 |
163 |
164 |
165 | ## 基于证据指导的预测错误数据识别
166 | ### 方法介绍
167 | TrustAI可信分析方法提供了模型预测依赖的证据，用来解释模型为什么会做这个预测。当预测依赖证据合理性弱，是否说明预测结果不可信呢？基于这个假设，我们做了一些探索，发现可以基于对模型预测依赖证据的分析，识别潜在预测错误的数据。
168 |
169 |
170 | ### 方法效果
171 | 实验基于ERNIE-1.0-base在相似度计算任务LCQMC数据集上微调得到相似度计算模型，然后在LCQMC验证集合上进行预测，并利用TrustAI提供的特征级证据分析方法识别模型预测依赖证据。
172 |
173 | 在我们的实验中，选择模型判断为语义相似但MAP（评估两个输入文本证据的一致性）低于指定阈值（0.3）的数据，其占全部测试数据比例为3.4%。相比于全量测试数据，模型在该类数据上效果下降了9.67%。
174 |
175 | | 数据集 | acc |
176 | | :-------: | :-----: |
177 | | 全部数据 | 89.53% |
178 | | 候选预测错误数据 | 79.86% |
179 |
180 | 基于候选预测错误数据，通过数据自动增强方法生成相关数据，作为强正负例加入到原始训练数据中，重训模型后，模型在该类数据上准确率提升7%。
181 |
182 | 详细见应用示例[相似度计算任务的预测错误数据识别](./map_analysis/zh-similarity-application.ipynb)。
183 |
184 |
185 |
186 |
187 |
--------------------------------------------------------------------------------
/tutorials/data_bias_identification/data_distribution_correction/README.md:
--------------------------------------------------------------------------------
1 | # 训练数据偏置识别及偏置消除 - 数据分布修正
2 |
3 | ## 方法介绍
4 | 受限于数据集收集方法、标注人员经验等影响，构建的训练数据集中往往存在偏置现象。模型会利用数据集偏置作为预测捷径，如在情感分析任务中，遇到否定词或描述直接给出“负向”情感预测。这种偏置会导致模型没有学会真正的理解和推理能力，在与训练数据分布一致的测试数据上表现很好，但在与训练数据分布不一致的测试数据上往往会表现较差。
5 |
6 | TrustAI提供了数据集偏置识别及基于分布修正的偏置缓解策略。
7 | * 偏置识别：统计训练数据中词与标注标签的分布，在分布上不均衡的词可能是偏置词，这里需要使用任务相关词典对候选偏置词过滤，得到真正的偏置词。包含偏置词的样本为偏置样本。
8 | * 分布修正：对非偏置样本进行重复采样。
9 |
10 |
11 | 注：开发者可访问[ AI Studio示例 ](https://aistudio.baidu.com/aistudio/projectdetail/4434652)快速体验本案例。
12 |
13 | ## 实验步骤
14 | 实验基于ERNIE-3.0-base-zh在情感分析任务ChnsentiCorp数据集上微调得到基线模型，在情感分析鲁棒性数据集上评估效果，评估指标为准确率。
15 |
16 |
17 | **Step 1**：识别偏置词。基于特征级证据可信分析方法（`IntGradInterpreter`）获取训练数据预测依赖的证据，然后统计各证据频次信息。
18 | ```shell
19 | # 下载数据
20 | wget --no-check-certificate https://trustai.bj.bcebos.com/application_data/distribution_data.tar && tar xf distribution_data.tar && rm distribution_data.tar
21 | # 训练基线模型
22 | python -u train.py --dataset_dir ./data --train_file train.tsv --dev_file robust.tsv --num_classes 2 --save_dir ./checkpoint
23 |
24 | # 统计重要证据和频次
25 | python -u get_rationale_importance.py --dataset_dir ./data --input_file train.tsv --num_classes 2 --rationale_path ./data/rationale_importance.txt --init_from_ckpt ./checkpoint/model_state.pdparams
26 | # rationale_path为证据及其频次保存的地址
27 | ```
28 |
29 | **Step 2**：识别偏置样本，及对偏置样本重复采样以达到均衡。
30 |
31 | ```shell
32 | # 生成均衡训练数据
33 | python -u balance_train_data.py --input_path ./data/train.tsv --rationale_path ./data/rationale_importance.txt --output_path ./data/balanced_train.tsv
34 | ```
35 |
36 | 基于生成的均衡数据`balanced_train.tsv`训练模型。
37 |
38 | ```shell
39 | python -u train.py --dataset_dir ./data --train_file balanced_train.tsv --dev_file robust.tsv --num_classes 2 --save_dir ./checkpoint
40 | ```
41 | 实验效果如下表所示：
42 | | 数据集 | 鲁棒性数据集 |
43 | | :-------: | :-------: |
44 | | 基线 | 69.97 |
45 | | 分布修正 | 71.38 |
46 |
47 | 注：以上结果均为10次实验的平均值。
48 |
--------------------------------------------------------------------------------
/tutorials/data_bias_identification/data_distribution_correction/balance_train_data.py:
--------------------------------------------------------------------------------
1 | import re
2 | import json
3 | import collections
4 | import random
5 | import time
6 | import os
7 | import argparse
8 | from collections import defaultdict
9 |
10 | import numpy as np
11 | import paddle
12 | import jieba
13 | from paddle.io import DataLoader, BatchSampler
14 | from paddlenlp.data import DataCollatorWithPadding
15 | from paddlenlp.datasets import load_dataset
16 | from paddlenlp.transformers import AutoModelForSequenceClassification, AutoTokenizer
17 | from paddlenlp.utils.log import logger
18 | from trustai.interpretation import get_word_offset
19 | from trustai.interpretation import IntGradInterpreter
20 | from LAC import LAC
21 | from tqdm import tqdm
22 |
23 | from utils import evaluate, preprocess_function
24 |
25 | parser = argparse.ArgumentParser()
26 | parser.add_argument("--input_path", type=str, default=None, help="file path of input data.")
27 | parser.add_argument("--output_path", type=str, default=None, help="file path of output data.")
28 |
29 | parser.add_argument("--seed", type=int, default=3, help="random seed for initialization")
30 | parser.add_argument("--rationale_path",
31 | type=str,
32 | default="./data/rationale_importance.txt",
33 | help="Path to save rationale importance data.")
34 |
35 | args = parser.parse_args()
36 |
37 |
38 | def set_seed(seed):
39 | """
40 | Sets random seed
41 | """
42 | random.seed(seed)
43 | np.random.seed(seed)
44 | paddle.seed(seed)
45 | os.environ['PYTHONHASHSEED'] = str(seed)
46 |
47 |
48 | def run():
49 | """
50 | Get rationale importance
51 | """
52 | set_seed(args.seed)
53 |
54 | # init lexical analyzer of chinese
55 | lac = LAC(mode='lac')
56 |
57 | # load ratioanle importance
58 | with open(args.rationale_path, 'r') as f:
59 | tokens = []
60 | for line in f:
61 | if line.strip():
62 | token, frequency = line.split('\t')
63 | frequency = int(frequency)
64 | if frequency > 2:
65 | tokens.append(token)
66 | # load ChnSentiCorp train data
67 | with open(args.input_path, 'r') as f:
68 | examples = []
69 | for i, line in enumerate(tqdm(list(f))):
70 | label, text = line.strip().split('\t')
71 | examples.append((i, int(label), text, list(jieba.cut(text))))
72 |
73 | # Statistics rationale index in positive and negative examples respectively
74 | pos_dict = collections.defaultdict(list)
75 | neg_dict = collections.defaultdict(list)
76 | rate_dict = {}
77 | for i, token in enumerate(tqdm(tokens[::-1])):
78 | for example in examples:
79 | if token in example[3]:
80 | if example[1] == 1:
81 | pos_dict[token].append(example[0])
82 | else:
83 | neg_dict[token].append(example[0])
84 |
85 | # filter rationale by postag and positive negative ratio
86 | for token in sorted(list(set(pos_dict.keys()) & set(neg_dict.keys()))):
87 | pos_list = pos_dict[token]
88 | neg_list = neg_dict[token]
89 | pos_ratio = len(pos_list) / (len(pos_list) + len(neg_list))
90 | postags = lac.run(token)[1]
91 | if (pos_ratio <= 0.15 or pos_ratio >= 0.85) and not (set(['c', 'r', 'w', 'm']) & set(postags)):
92 | rate_dict[token] = [pos_ratio if pos_ratio < 0.5 else 1 - pos_ratio, len(pos_list), len(neg_list), postags]
93 | for k, v in rate_dict.items():
94 | print(k, v, len(pos_dict[k]), len(neg_dict[k]))
95 | # sampling the data that will be added to the training set
96 | add_dict = defaultdict(int)
97 | add_list = []
98 | for token in rate_dict:
99 | pos_num = len(pos_dict[token])
100 | neg_num = len(neg_dict[token])
101 | tmp_dict = defaultdict(int)
102 | if pos_num > neg_num:
103 | for idx in random.choices(neg_dict[token], k=min(pos_num - neg_num, neg_num * 2)):
104 | tmp_dict[idx] += 1
105 | else:
106 | for idx in random.choices(pos_dict[token], k=min(neg_num - pos_num, pos_num * 2)):
107 | tmp_dict[idx] += 1
108 | for idx, count in tmp_dict.items():
109 | add_dict[idx] = max(add_dict[idx], count)
110 | for idx, count in add_dict.items():
111 | add_list.extend([idx] * count)
112 | print(add_dict)
113 | random.shuffle(add_list)
114 | # write data to train data
115 | logger.info(f"add number: {len(add_list)}")
116 | with open(args.output_path, 'w') as f:
117 | for example in examples:
118 | f.write(str(example[1]) + '\t' + example[2] + '\n')
119 | for idx in add_list:
120 | example = examples[idx]
121 | f.write(str(example[1]) + '\t' + example[2] + '\n')
122 |
123 |
124 | if __name__ == "__main__":
125 | run()
126 |
--------------------------------------------------------------------------------
/tutorials/data_bias_identification/data_distribution_correction/utils.py:
--------------------------------------------------------------------------------
1 | # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | import numpy as np
16 |
17 | import paddle
18 | import paddle.nn.functional as F
19 | from paddlenlp.utils.log import logger
20 |
21 |
22 | @paddle.no_grad()
23 | def evaluate(model, criterion, metric, data_loader, name=''):
24 | """
25 | Given a dataset, it evaluates model and computes the metric.
26 | Args:
27 | model(obj:`paddle.nn.Layer`): A model to classify texts.
28 | criterion(obj:`paddle.nn.Layer`): It can compute the loss.
29 | metric(obj:`paddle.metric.Metric`): The evaluation metric.
30 | data_loader(obj:`paddle.io.DataLoader`): The dataset loader which generates batches.
31 | """
32 |
33 | model.eval()
34 | metric.reset()
35 | losses = []
36 | for batch in data_loader:
37 | input_ids, token_type_ids, labels = batch['input_ids'], batch[
38 | 'token_type_ids'], batch['labels']
39 | logits = model(input_ids, token_type_ids)
40 | loss = criterion(logits, labels)
41 | losses.append(loss.numpy())
42 | correct = metric.compute(logits, labels)
43 | metric.update(correct)
44 |
45 | acc = metric.accumulate()
46 | logger.info("%s: eval loss: %.5f, acc: %.5f" % (name, np.mean(losses), acc))
47 | model.train()
48 | metric.reset()
49 |
50 | return acc
51 |
52 |
53 | def preprocess_function(example, tokenizer, max_seq_length, is_test=False):
54 | """
55 | Builds model inputs from a sequence for sequence classification tasks
56 | by concatenating and adding special tokens.
57 |
58 | Args:
59 | example(obj:`list[str]`): input data, containing text and label if it have label.
60 | tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer`
61 | which contains most of the methods. Users should refer to the superclass for more information regarding methods.
62 | max_seq_length(obj:`int`): The maximum total input sequence length after tokenization.
63 | Sequences longer than this will be truncated, sequences shorter will be padded.
64 | label_nums(obj:`int`): The number of the labels.
65 | Returns:
66 | result(obj:`dict`): The preprocessed data including input_ids, token_type_ids, labels.
67 | """
68 | if 'text_b' not in example:
69 | result = tokenizer(text=example["text_a"], max_seq_len=max_seq_length)
70 | else:
71 | result = tokenizer(text=example["text_a"], text_pair=example['text_b'], max_seq_len=max_seq_length)
72 |
73 | if not is_test:
74 | result["labels"] = np.array([example['label']], dtype='int64')
75 | return result
76 |
--------------------------------------------------------------------------------
/tutorials/data_bias_identification/less_learn_shortcut/README.md:
--------------------------------------------------------------------------------
1 | # 解决训练数据分布偏置的问题 - 数据权重修正方案
2 | ## 方法介绍
3 | 受限于数据集收集方法、标注人员经验等影响，构建的训练数据集中往往存在偏置现象。模型会利用数据集偏置作为预测捷径，如在情感分析任务中，遇到否定词或描述直接给出“负向”情感预测。这种偏置会导致模型没有学会真正的理解和推理能力，在与训练数据分布一致的测试数据上表现很好，但在与训练数据分布不一致的测试数据上往往会表现较差。
4 |
5 | TrustAI提供了数据集偏置识别及基于权重修正的偏置缓解策略。
6 | * 偏置识别：统计训练数据中词与标注标签的分布，在分布上不均衡的词可能是偏置词，包含偏置词的样本为偏置样本。
7 | * 权重修正：降低偏置样本对训练loss的影响，即针对每一条样本计算一个偏置度，在训练loss计算时通过偏置度降低偏置样本影响，具体见[Du, Yanrui, et al. 2022](https://arxiv.org/abs/2205.12593)。
8 |
9 | 注：开发者可访问[ AI Studio示例 ](https://aistudio.baidu.com/aistudio/projectdetail/4434616)快速体验本案例。
10 |
11 | ## 实验步骤
12 | 实验基于ERNIE-3.0-base-zh在情感分析任务ChnsentiCorp数据集上微调得到基线模型，在情感分析鲁棒性数据集上评估效果，评估指标为准确率。
13 |
14 |
15 | **Step 1**：识别训练数据中的偏置词。在训练数据中，统计每个词在不同类别上的分布，对于频次大于`cnt_threshold`、且最少在一个类别上出现比例大于`p_threshold`的词视为偏置词。
16 |
17 | ```shell
18 | # 下载数据
19 | wget --no-check-certificate https://trustai.bj.bcebos.com/application_data/lls_data.tar && tar xf lls_data.tar && rm lls_data.tar
20 | # 统计偏置词
21 | python -u find_bias_word.py --output_dir output --input_path ./data/train.tsv --num_classes 2 --cnt_threshold 3 --p_threshold 0.90 --output_dir output
22 | # cnt_threshold表示为偏置词最少需要出现的频次
23 | # p_threshold表示偏置比例的阈值，偏置词至少需要在一个类别上大于此阈值
24 | # output_dir表示统计结果的存储路径
25 | ```
26 |
27 | **Step 2**：基于偏置词的统计结果，针对每一训练样本，计算偏置度，作为样本对训练loss的影响权重。
28 |
29 | 当前方案提供了`lls_d`和`lls_d_f`两种计算样本偏置度的策略，前者考虑词的有偏性，后者同时考虑词的有偏性和频次。
30 |
31 | ```shell
32 | # 基于`lls_d`策略计算样本偏置度
33 | python -u lls.py --input_path ./data/train.tsv --bias_dir ./output --stopwords_path ./data/stop_words.txt --num_classes 2 --mode lls_d --output_path ./data/train_lls_d.tsv
34 | # 基于`lls_d_f`策略计算样本偏置度
35 | python -u lls.py --input_path ./data/train.tsv --bias_dir ./output --stopwords_path ./data/stop_words.txt --num_classes 2 --mode lls_d_f --output_path ./data/train_lls_d_f.tsv
36 | # mode表示计算样本偏置度的策略，当前有`lls_d`和`lls_d_f`两种策略
37 | # output_path表示为生成带偏置度训练集的存储路径
38 | ```
39 |
40 | **Step 3**：用带偏置度的训练数据训练模型，偏置度作用于loss计算。
41 | ```shell
42 | # 基于`lls_d`策略产生的数据训练模型
43 | python -u train.py --dataset_dir ./data --train_file train_lls_d.tsv --dev_file dev.tsv --test_files test.tsv DuQM --num_classes 2 --save_dir ./lls_d_checkpoint
44 | # 基于`lls_d_f`策略产生的数据训练模型
45 | python -u train.py --dataset_dir ./data --train_file train_lls_d_f.tsv --dev_file dev.tsv --test_files test.tsv DuQM --num_classes 2 --save_dir ./lls_d_f_checkpoint
46 | ```
47 |
48 | 实验结果如下表所示：相比于基线，权重修正后，模型在鲁棒性数据集DuQM上准确率提升0.94%。
49 |
50 | | 数据集 | LCQMCdev | LCQMCtest | DuQM |
51 | | :-------: | :-------: | :-------: | :-------: |
52 | | 基线 | 90.93 | 87.06 | 73.82 |
53 | | lls_d | 90.76 | 87.58 | 74.76 |
54 | | lls_d_f | 90.80 | 87.22 | 74.44 |
55 |
56 | 注：以上结果均为3次实验的平均值。
57 |
--------------------------------------------------------------------------------
/tutorials/data_bias_identification/less_learn_shortcut/find_bias_word.py:
--------------------------------------------------------------------------------
1 | import os
2 | import json
3 | import collections
4 | import argparse
5 |
6 | from LAC import LAC
7 | from tqdm import tqdm
8 |
9 | parser = argparse.ArgumentParser()
10 | parser.add_argument("--output_dir",
11 | default="./output",
12 | type=str,
13 | help="The output directory where the result will be written.")
14 | parser.add_argument("--input_path", type=str, default=None, help="train data file path")
15 | parser.add_argument('--num_classes', type=int, default=2, help='Number of classification.')
16 | parser.add_argument('--cnt_threshold', type=int, default=3, help='Count threshold of statistical biased words')
17 | parser.add_argument('--p_threshold', type=float, default=0.85, help='Probability threshold of statistical biased words')
18 |
19 | args = parser.parse_args()
20 |
21 |
22 | class BiasWord(object):
23 | """
24 | Statistic the biased words in the dataset
25 | """
26 |
27 | def __init__(self, segments, labels, num_classes=2, cnt_threshold=3, p_threshold=0.85):
28 | self.cnt_threshold = cnt_threshold
29 | self.p_threshold = p_threshold
30 | self.num_classes = num_classes
31 | self.segments = segments
32 | self.labels = labels
33 |
34 | def process(self):
35 | """
36 | process function
37 | """
38 | self._get_dict()
39 | self._search_bias_word()
40 | print("number of bias_words:", len(self.bias_words))
41 | return self.bias_words, self.bias_word_cnt, self.id2words
42 |
43 | def _get_dict(self):
44 | self.word2ids = collections.defaultdict(set)
45 | self.id2words = collections.defaultdict(set)
46 | for n, segs in enumerate(self.segments):
47 | for seg in segs:
48 | self.word2ids[seg].add(n)
49 | self.id2words[n] = set(segs)
50 |
51 | def _search_bias_word(self):
52 | self.bias_words = {}
53 | self.bias_word_cnt = {}
54 | for word, sentids in self.word2ids.items():
55 | if len(sentids) >= self.cnt_threshold:
56 | cnts = [0] * self.num_classes
57 |
58 | for sentid in sentids:
59 | label = self.labels[sentid]
60 | cnts[label] += 1
61 | assert sum(cnts) != 0
62 | max_cnt = max(cnts)
63 | p = max_cnt / sum(cnts)
64 | if p >= self.p_threshold:
65 | self.bias_words[word] = p
66 | self.bias_word_cnt[word] = len(sentids)
67 |
68 |
69 | if __name__ == "__main__":
70 | # initialize tokenizer
71 | lac = LAC(mode='rank')
72 |
73 | # preprocess data, get segments、labels and lines
74 | segments = []
75 | labels = []
76 | lines = []
77 | with open(args.input_path, 'r') as f:
78 | for line in tqdm(list(f)):
79 | lines.append(line)
80 | query, title, label = line.strip().split('\t')
81 | seg_res = lac.run([query, title])
82 | query_segs = seg_res[0][0]
83 | title_segs = seg_res[1][0]
84 | segments.append(query_segs + title_segs)
85 | labels.append(int(label))
86 |
87 | # get bias_words
88 | biasword = BiasWord(segments, labels, num_classes=2, cnt_threshold=args.cnt_threshold, p_threshold=args.p_threshold)
89 | # b_words: biased words, dict
90 | # b_word_cnt: count of biased words, dict
91 | # id2words: sentence index to words, dict
92 | b_words, b_word_cnt, id2words = biasword.process()
93 |
94 | # save result to output_dir
95 | if not os.path.exists(args.output_dir):
96 | os.makedirs(args.output_dir)
97 | with open(os.path.join(args.output_dir, "bias_word.json"), 'w') as f:
98 | json.dump(b_words, f, ensure_ascii=False)
99 | with open(os.path.join(args.output_dir, "bias_word_cnt.json"), 'w') as f:
100 | json.dump(b_word_cnt, f, ensure_ascii=False)
101 | with open(os.path.join(args.output_dir, "id2words.json"), 'w') as f:
102 | for k, v in id2words.items():
103 | id2words[k] = list(v)
104 | json.dump(id2words, f, ensure_ascii=False)
105 |
--------------------------------------------------------------------------------
/tutorials/data_bias_identification/less_learn_shortcut/lls.py:
--------------------------------------------------------------------------------
1 | import os
2 | import json
3 | import argparse
4 |
5 | parser = argparse.ArgumentParser()
6 | parser.add_argument("--input_path", type=str, default=None, help="input data file path")
7 | parser.add_argument("--output_path", type=str, default=None, help="output data file path")
8 | parser.add_argument("--bias_dir", type=str, default=None, help="bias data directory.")
9 | parser.add_argument("--stopwords_path", type=str, default=None, help="stopwords data file path")
10 | parser.add_argument('--num_classes', type=int, default=2, help='Number of classification.')
11 | parser.add_argument('--alpha',
12 | type=float,
13 | default=0.01,
14 | help='Hyperparameters for frequency of words when mode is lls_d_f.')
15 | parser.add_argument('--mode',
16 | type=str,
17 | default='lls_d',
18 | choices=['lls_d', 'lls_d_f'],
19 | help='Hyperparameters for frequency of words.')
20 |
21 | args = parser.parse_args()
22 |
23 |
24 | def filter_stopwords(score_w, stop_words):
25 | for word in list(score_w.keys()):
26 | if word in stop_words:
27 | del score_w[word]
28 | return score_w
29 |
30 |
31 | def word_score(d, num_classes):
32 | score_w = {}
33 | for k in d.keys():
34 | score_w[k] = abs(d[k] - 1 / num_classes)
35 | return score_w
36 |
37 |
38 | def word_score_freq(d, d_cnt, num_classes, alpha):
39 | score_w = {}
40 | for k in d.keys():
41 | score_w[k] = abs(d[k] - 1 / num_classes) + alpha * d_cnt[k]
42 | return score_w
43 |
44 |
45 | def lls_basic(score_w, id2words):
46 | sample_bias = {}
47 | for n in range(len(id2words)):
48 |
49 | sample_score = 0
50 | cnt = 0
51 | for word in id2words[str(n)]:
52 | if word in score_w:
53 | sample_score += score_w[word]
54 | cnt += 1
55 | if cnt != 0:
56 | sample_bias[n] = sample_score / cnt
57 | return sample_bias
58 |
59 |
60 | def softxmax(sample_bias, a=0, b=0.15):
61 | """
62 | Score normalization
63 | """
64 | scores = []
65 | for k, v in sample_bias.items():
66 | scores.append(v)
67 | maxn, minn = max(scores), min(scores)
68 | sample_bias_norm = {}
69 | for k, sc in sample_bias.items():
70 | sc_softmax = a + (b - a) / (maxn - minn) * (sc - minn)
71 | sample_bias_norm[k] = (1 - sc_softmax)
72 | return sample_bias_norm
73 |
74 |
75 | if __name__ == "__main__":
76 |
77 | # load data
78 | with open(args.stopwords_path, 'r') as f:
79 | stop_words = []
80 | for line in f.readlines():
81 | stop_words.append(line.strip())
82 | with open(os.path.join(args.bias_dir, 'id2words.json'), 'r') as f:
83 | id2words = json.load(f)
84 | with open(os.path.join(args.bias_dir, 'bias_word.json'), 'r') as f:
85 | d = json.load(f)
86 | with open(os.path.join(args.bias_dir, 'bias_word_cnt.json'), 'r') as f:
87 | d_cnt = json.load(f)
88 | with open(args.input_path, 'r') as f:
89 | lines = list(f)
90 |
91 | # get bias degree for example
92 | mode = args.mode
93 | if mode == 'lls_d':
94 | score_w = word_score(d, num_classes=2)
95 | score_w = filter_stopwords(score_w, stop_words)
96 | sample_bias = lls_basic(score_w, id2words)
97 | sample_bias_norm = softxmax(sample_bias)
98 | elif mode == 'lls_d_f':
99 | score_w = word_score_freq(d, d_cnt, num_classes=args.num_classes, alpha=args.alpha)
100 | score_w = filter_stopwords(score_w, stop_words)
101 | sample_bias = lls_basic(score_w, id2words)
102 | sample_bias_norm = softxmax(sample_bias)
103 | else:
104 | raise KeyError(f"Unknown mode: {mode}, mode should be chosen from [lls_d, lls_d_f].")
105 |
106 | # save result
107 | with open(args.output_path, 'w', encoding='utf-8') as f:
108 | for n, line in enumerate(lines):
109 | if n in sample_bias_norm:
110 | f.write(line.strip() + '\t' + str(sample_bias_norm[n]) + '\n')
111 | else:
112 | f.write(line.strip() + '\t' + str(1) + '\n')
--------------------------------------------------------------------------------
/tutorials/data_bias_identification/less_learn_shortcut/utils.py:
--------------------------------------------------------------------------------
1 | # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | import numpy as np
16 |
17 | import paddle
18 | import paddle.nn.functional as F
19 | from paddlenlp.utils.log import logger
20 |
21 |
22 | @paddle.no_grad()
23 | def evaluate(model, criterion, metric, data_loader, name=''):
24 | """
25 | Given a dataset, it evaluates model and computes the metric.
26 | Args:
27 | model(obj:`paddle.nn.Layer`): A model to classify texts.
28 | criterion(obj:`paddle.nn.Layer`): It can compute the loss.
29 | metric(obj:`paddle.metric.Metric`): The evaluation metric.
30 | data_loader(obj:`paddle.io.DataLoader`): The dataset loader which generates batches.
31 | """
32 |
33 | model.eval()
34 | metric.reset()
35 | losses = []
36 | for batch in data_loader:
37 | input_ids, token_type_ids, labels = batch['input_ids'], batch['token_type_ids'], batch['labels']
38 | logits = model(input_ids, token_type_ids)
39 | loss = criterion(logits, labels)
40 | loss = loss.mean()
41 | losses.append(loss.numpy())
42 | correct = metric.compute(logits, labels)
43 | metric.update(correct)
44 |
45 | acc = metric.accumulate()
46 | logger.info("%s: eval loss: %.5f, acc: %.5f" % (name, np.mean(losses), acc))
47 | model.train()
48 | metric.reset()
49 |
50 | return acc
51 |
52 |
53 | def preprocess_function(example, tokenizer, max_seq_length, is_test=False):
54 | """
55 | Builds model inputs from a sequence for sequence classification tasks
56 | by concatenating and adding special tokens.
57 |
58 | Args:
59 | example(obj:`list[str]`): input data, containing text and label if it have label.
60 | tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer`
61 | which contains most of the methods. Users should refer to the superclass for more information regarding methods.
62 | max_seq_length(obj:`int`): The maximum total input sequence length after tokenization.
63 | Sequences longer than this will be truncated, sequences shorter will be padded.
64 | label_nums(obj:`int`): The number of the labels.
65 | Returns:
66 | result(obj:`dict`): The preprocessed data including input_ids, token_type_ids, labels.
67 | """
68 | if 'text_b' not in example:
69 | result = tokenizer(text=example["text_a"], max_seq_len=max_seq_length)
70 | else:
71 | result = tokenizer(text=example["text_a"], text_pair=example['text_b'], max_seq_len=max_seq_length)
72 |
73 | if not is_test:
74 | result["labels"] = np.array([example['label']], dtype='int64')
75 | result["weights"] = np.array([example['weight']], dtype='float32')
76 | return result
77 |
--------------------------------------------------------------------------------
/tutorials/data_map/README.md:
--------------------------------------------------------------------------------
1 | # 基于训练信号的数据地图绘制 Dataset Cartography with Training Dynamics
2 |
3 | ## 方法介绍
4 | 现有工作表明，可以使用训练数据在训练过程中的信号绘制数据地图；根据信号特征划分数据，不同数据区域具有不同特点，如难学、标注错误等。通过绘制数据地图，可以帮助开发者更好地了解训练数据。
5 |
6 | TrustAI提供了"训练信号收集 -> 数据地图绘制"方案。首先，收集每条训练数据在训练过程中不同step下的训练信号；然后，根据得到的统计信号，基于指定的信号维度绘制数据地图。
7 |
8 | 注：开发者可访问[ AI Studio示例 ](https://aistudio.baidu.com/aistudio/projectdetail/5307701)快速体验本案例。
9 |
10 | ## 实验步骤
11 | 我们以相似度计算任务LCQMC数据集上的模拟实验为例，介绍该方案实现步骤和效果。
12 |
13 |
14 | **Step 1**：从LCQMC训练集中随机抽取100条数据作为训练数据样例。训练集文件为.tsv格式，内容如下
15 |
16 | ```shell
17 | # tsv格式数据，一行一条训练数据，不同字段用tab分隔，各字段如下介绍:
18 | [
19 | "text_a" : xxx, // 训练数据文本a
20 | "text_b" : xxx, // 训练数据文本b
21 | "label" : xxx, // 训练数据对应的label
22 | "s_label" : xxx, // 训练数据的构造label （可省略）
23 | ]
24 | ```
25 | 注: ``s_label``可省，为用户构造的label，如构造脏数据。
26 |
27 | 基于ERNIE-3.0-base-zh在新训练集`sample_100.tsv`上微调得到基线模型，运行命令如下所示：
28 |
29 | ```shell
30 | # 训练模型并收集训练信号
31 | sh run_train_pointwise.sh
32 | ```
33 | 所有训练数据的训练信号按训练step保存在`outputs`路径下。
34 |
35 | 注: 训练信号的收集代码可参考代码`train_pointwise.py`中Lines 199-218，用户可根据自己模型代码进行修改。收集的训练信号如下（用户可自行设计更多信号进行收集）：
36 |
37 | ```shell
38 | # .jsonl 格式数据
39 | [
40 | {'id' : xxx, // 训练数据的id
41 | 'label' : xxx, // 训练数据对应的label
42 | 'pred_label' : xxx, // 训练数据的预测label
43 | 'correct' : xxx, // 训练数据是否被预测正确
44 | 'loss' : xxx, // 训练数据当前的loss
45 | 'probs' : [xxx, xxx], // 训练数据在当前每个类下的预测概率(one-hot形式)
46 | 'label_probs' : xxx // 训练数据在label类别下的预测概率
47 | }
48 | ... ...
49 | ]
50 | ```
51 |
52 | **Step 2**：训练信号处理，即基于不同训练steps收集到的信号计算整体信号，如基于不同steps得到的预测概率计算整体平均预测概率。
53 |
54 | ```shell
55 | # 训练信号处理
56 | python -u sample_stat_summary.py
57 | ```
58 | 产出数据保存在`outputs`路径下。
59 |
60 |
61 | 训练信号详细信息
62 |
63 | ```shell
64 | # tsv 格式数据，一行保存一条训练数据的所有训练信号，信号之间用tab进行分隔，各信号如下表示:
65 | [
66 | "id" : xxx, // 训练数据的id
67 | "label" : xxx, // 训练数据对应的label
68 | "s_label" : xxx, // 训练数据的构造label，数据地图绘制允许标记困难数据(s_label = 1)和构造脏数据(s_label = 2)
69 | "correct_times" : xxx, // 总共预测正确的次数
70 | "correct_ratio" : xxx, // 预测正确次数占比
71 | "avg_probs" : xxx, // 多次预测的置信度的平均数
72 | "label_var" : xxx, // 多次预测的置信度的方差
73 | "max_label_probs" : xxx, // 多次预测的置信度的最大值
74 | "min_label_probs" : xxx, // 多次预测的置信度的最小值
75 | "forgetting_times" : xxx, // 多次预测反映出的，模型对本数据的遗忘次数（之前预测对了，后来又错了）
76 | "learnt_times" : xxx, // 多次预测反映出的，模型对本数据的学会次数（之前预测错了，后来又对了）
77 | "first_forget" : xxx, // 多次预测中，第一次遗忘本数据
78 | "first_learn" : xxx, // 多次预测中，第一次学会本数据
79 | ]
80 | ```
81 |
82 |
83 | **Step 3**：基于产出的训练信号，选择两个信号作为数据地图的主要维度（默认为平均置信度与置信方差），并选择其他信号（如正确比例、正确次数、遗忘次数、学习次数等）以颜色、形状等进行区别绘制数据地图。
84 |
85 | ```shell
86 | # 数据地图绘制
87 | python -u plot_map.py
88 |
89 | # 参数选择
90 | attr1: str类型， 默认值为"avg_probs"，选择一个信号作为数据地图的纵轴
91 | attr2: str类型， 默认值为"label_var"，选择一个信号作为数据地图的横轴
92 | criterion: str类型，默认值为空，选择一个训练信号作为数据筛选依据，仅满足条件的数据会被绘制在地图上
93 | threshold: float类型，默认值0，与criterion一同使用，为选择的训练信号设置阈值，筛选数据
94 | use_f_times: float类型，默认值-1，使用forgotten_times并选择所有遗忘次数不小于use_f_times的样本
95 | use_l_times: float类型，默认值-1，使用learnt_times并选择所有遗忘次数不小于use_l_times的样本
96 |
97 | # 数据地图样例
98 | python -u plot_map.py # 图1左
99 | python -u plot_map.py --criterion forgetting_times --threshold 1 # 图1中
100 | python -u plot_map.py --use_l_times 0 # 图1右
101 | ```
102 |
103 |
104 | 
105 | 
106 | 
107 | 图1 数据地图样例。左: 默认参数设置的数据地图；中：指定criterion为forgetting_times，threshold为1的数据绘制地图；右：使用learnt_times(use_l_times=0)区分数据的颜色，其中左上和左下角数据分别使用correct_times做进一步区分。此数据地图根据全量LCQMC数据的训练信号绘制，而非提供的100条样例。
108 |
109 |
--------------------------------------------------------------------------------
/tutorials/data_map/data.py:
--------------------------------------------------------------------------------
1 | # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | import paddle
16 | import numpy as np
17 |
18 | from paddlenlp.datasets import MapDataset
19 |
20 |
21 | def create_dataloader(dataset,
22 | mode='train',
23 | batch_size=1,
24 | batchify_fn=None,
25 | trans_fn=None):
26 | if trans_fn:
27 | dataset = dataset.map(trans_fn)
28 |
29 | shuffle = True if mode == 'train' else False
30 | if mode == 'train':
31 | batch_sampler = paddle.io.DistributedBatchSampler(
32 | dataset, batch_size=batch_size, shuffle=shuffle)
33 | else:
34 | batch_sampler = paddle.io.BatchSampler(
35 | dataset, batch_size=batch_size, shuffle=shuffle)
36 |
37 | return paddle.io.DataLoader(
38 | dataset=dataset,
39 | batch_sampler=batch_sampler,
40 | collate_fn=batchify_fn,
41 | return_list=True)
42 |
43 |
44 | def read_text_pair(data_path):
45 | """Reads data."""
46 | with open(data_path, 'r', encoding='utf-8') as f:
47 | for line in f:
48 | data = line.rstrip().split("\t")
49 | if len(data) != 2:
50 | continue
51 | yield {'query': data[0], 'title': data[1]}
52 |
53 |
54 | def convert_pointwise_example(example,
55 | tokenizer,
56 | max_seq_length=512,
57 | is_test=False,
58 | language='en'):
59 | if language == 'ch':
60 | q_name = "text_t"
61 | t_name = "text_q"
62 | s_name = "s_label"
63 | #t_name = "title"
64 | l_name = "label"
65 | else:
66 | q_name = "text_t"
67 | t_name = "text_q"
68 | l_name = "label"
69 | s_name = "s_label"
70 | #q_name = "sentence1"
71 | #t_name = "sentence2"
72 | #l_name = "labels"
73 |
74 | query, title = example[q_name], example[t_name]
75 |
76 | encoded_inputs = tokenizer(
77 | text=query, text_pair=title, max_seq_len=max_seq_length)
78 |
79 | input_ids = encoded_inputs["input_ids"]
80 | token_type_ids = encoded_inputs["token_type_ids"]
81 | sep_ids = len(input_ids) - 1
82 |
83 | #print("input_ids:",input_ids)
84 | if not is_test:
85 | label = np.array([example[l_name]], dtype="int64")
86 | s_label = np.array([example[s_name]],dtype="int64")
87 | sep_ids = np.array([sep_ids], dtype="int64")
88 | #print('label', label)
89 | return input_ids, token_type_ids, label, s_label, sep_ids
90 | else:
91 | return input_ids, token_type_ids
92 |
93 |
94 | def convert_pairwise_example(example,
95 | tokenizer,
96 | max_seq_length=512,
97 | phase="train"):
98 |
99 | if phase == "train":
100 | query, pos_title, neg_title = example["query"], example[
101 | "title"], example["neg_title"]
102 |
103 | pos_inputs = tokenizer(
104 | text=query, text_pair=pos_title, max_seq_len=max_seq_length)
105 | neg_inputs = tokenizer(
106 | text=query, text_pair=neg_title, max_seq_len=max_seq_length)
107 |
108 | pos_input_ids = pos_inputs["input_ids"]
109 | pos_token_type_ids = pos_inputs["token_type_ids"]
110 | neg_input_ids = neg_inputs["input_ids"]
111 | neg_token_type_ids = neg_inputs["token_type_ids"]
112 |
113 | return (pos_input_ids, pos_token_type_ids, neg_input_ids,
114 | neg_token_type_ids)
115 |
116 | else:
117 | query, title = example["query"], example["title"]
118 |
119 | inputs = tokenizer(
120 | text=query, text_pair=title, max_seq_len=max_seq_length)
121 |
122 | input_ids = inputs["input_ids"]
123 | token_type_ids = inputs["token_type_ids"]
124 | if phase == "eval":
125 | return input_ids, token_type_ids, example["label"]
126 | elif phase == "predict":
127 | return input_ids, token_type_ids
128 | else:
129 | raise ValueError("not supported phase:{}".format(phase))
130 |
131 |
132 | def gen_pair(dataset, pool_size=100):
133 | """
134 | Generate triplet randomly based on dataset
135 |
136 | Args:
137 | dataset: A `MapDataset` or `IterDataset` or a tuple of those.
138 | Each example is composed of 2 texts: exampe["query"], example["title"]
139 | pool_size: the number of example to sample negative example randomly
140 |
141 | Return:
142 | dataset: A `MapDataset` or `IterDataset` or a tuple of those.
143 | Each example is composed of 2 texts: exampe["query"], example["pos_title"]、example["neg_title"]
144 | """
145 |
146 | if len(dataset) < pool_size:
147 | pool_size = len(dataset)
148 |
149 | new_examples = []
150 | pool = []
151 | tmp_exmaples = []
152 |
153 | for example in dataset:
154 | label = example["label"]
155 |
156 | # Filter negative example
157 | if label == 0:
158 | continue
159 |
160 | tmp_exmaples.append(example)
161 | pool.append(example["title"])
162 |
163 | if len(pool) >= pool_size:
164 | np.random.shuffle(pool)
165 | for idx, example in enumerate(tmp_exmaples):
166 | example["neg_title"] = pool[idx]
167 | new_examples.append(example)
168 | tmp_exmaples = []
169 | pool = []
170 | else:
171 | continue
172 | return MapDataset(new_examples)
173 |
--------------------------------------------------------------------------------
/tutorials/data_map/run_train_pointwise.sh:
--------------------------------------------------------------------------------
1 | ###
2 | # This script is used to finetune pretrained models
3 | ###
4 |
5 | export CUDA_VISIBLE_DEVICES=3
6 | LANGUAGE="ch"
7 | timestamp=`date +"%Y%m%d_%H%M%S"`
8 | data_dir='./'
9 | LEARNING_RATE=3e-5
10 | MAX_SEQ_LENGTH=256
11 |
12 | [ -d "logs" ] || mkdir -p "logs"
13 | [ -d "outputs" ] || mkdir -p "outputs"
14 | set -x
15 |
16 | train_file=sample_100.tsv
17 | dev_file=$train_file
18 | train_size=100
19 |
20 | batch_size=32
21 | epoch=5
22 | save_model_num=5
23 | epoch_steps=$[$train_size/$batch_size]
24 | save_steps=$[$epoch_steps*$epoch/${save_model_num}]
25 |
26 | python3 ./train_pointwise.py \
27 | --learning_rate $LEARNING_RATE \
28 | --max_seq_length $MAX_SEQ_LENGTH \
29 | --batch_size ${batch_size} \
30 | --epochs ${epoch} \
31 | --data_dir $data_dir \
32 | --train_set ${train_file} \
33 | --dev_set ${dev_file} \
34 | --eval_step ${save_steps} \
35 | --warmup_proportion 0.1 \
36 | --save_dir saved_model/${timestamp} >> logs/log_${timestamp}
37 |
38 |
--------------------------------------------------------------------------------
/tutorials/data_map/sample_100.tsv:
--------------------------------------------------------------------------------
1 | text_a text_b label type
2 | 御龙在天小还丹怎么做 御龙在天怎么上不去 0 0
3 | 东风日产轩逸怎么样？ 东风日产新轩逸怎么样 1 0
4 | 穿越火线有哪些小说 穿越火线小说。 1 0
5 | 我的爸爸作文 爸爸的手作文 1 0
6 | 小米华为魅族哪个好？ 酷派小米华为魅族哪个好 1 0
7 | 为什么苹果信号这么差？ 为什么苹果6手机信号这么差 1 0
8 | 微信朋友圈能看到访客吗 微信朋友圈都能看到吗 0 0
9 | 魔兽世界，猎人宏 魔兽世界猎人弓 1 0
10 | 怎么查别人微信的聊天记录 微信聊天记录怎么查 1 0
11 | 列方程解应用题， 列方程解应用题。 1 0
12 | 支付宝里的钱怎么转到银行卡 支付宝的钱怎么转到银行卡 1 0
13 | 北京夜店有哪些 北京有哪些夜店 1 0
14 | 这是什么颜色啊？ 这是什么颜色阿！ 1 0
15 | 优酷视频为什么不能下载 优酷为什么不能下载视频了 1 0
16 | 怎么样才能做好网店！ 怎么做好网店？ 1 0
17 | 这是哪个明星小时候 这是哪个明星的小时候 1 0
18 | 天天飞车哪个车手刷高分多 天天飞车高分用哪个车手？ 1 0
19 | 不客气的，为防止掉线，请您在3-4分钟内回复我一下就好 为防止掉线，请您在3分钟内回复我一下，谢谢您的配合。您好，在否？谢谢 1 0
20 | 我的特一营全集观看那有了吗 电视剧我的特一营全集观看哪里有 1 0
21 | 如何从网上查询个人征信 在网上如何查个人征信，要明细版的 1 0
22 | 在家赚钱有些什么方法？ 在家带孩子有什么赚钱的方法 1 0
23 | 现在有什么好看的连续剧或者电影？ 有什么好看的电视剧，或者电影 1 0
24 | 天龙八部问题 新天龙八部问题 0 0
25 | 这个里番是什么类型的？ 是什么类型 0 0
26 | 婴儿理发器哪个牌子好 婴儿理发器哪个牌子好？ 1 0
27 | 手机版的百度知道可以签到嘛？ 百度知道怎么看还有多少升级啊？ 0 0
28 | 怎样跟自己喜欢的人表白？ 怎么和自己喜欢的人表白？ 1 0
29 | 发什么成绩？ 发什么成绩 1 0
30 | 几岁才可以办银行卡啊？ 办银行卡要几岁才可以办？ 1 0
31 | 您的是人工审核的方式吗？ 只有人工审核的方式吗 0 0
32 | 怎么让刘海紧贴着额头 怎么使刘海不贴额头 0 0
33 | 无聊的时候你们都在干嘛呢？ 你们无聊的时候都干嘛呢？ 1 0
34 | 我爱你用韩语怎么写？ 我爱你用韩语怎么说？ 0 0
35 | 世界上有没有外星人 世界有没有外星人 1 0
36 | 这爱已打烊是什么意思 打烊是什么意思 0 0
37 | 硅是由什么构成的 硅由什么构成 1 0
38 | 二本中电子信息工程专业哪个学校比较好 电子信息工程专业分流哪个方向比较好 0 0
39 | 男孩子名字，哪个字好 带氵，钅字旁的男孩名字有哪些？ 0 0
40 | 天天酷跑怎么刷金币和钻石 天天酷跑怎么刷金币钻石 1 0
41 | 入团申请书500字左右 入团申请书600字 0 0
42 | 从马鞍山火车站怎么到博望汽车站？ 从马鞍山博望到黄池怎么走 0 0
43 | 大写的我怎么写 给的大写怎么写 0 0
44 | 戴耳机的男生头像 头像耳机男生 1 0
45 | 纪念碑谷这关怎么过？ 史上最坑爹的游戏，咋过， 0 0
46 | 这个是您转账的 这应该是您查询错误账户了 0 0
47 | 平板电脑哪个品牌好呢 国产哪个牌子的平板电脑好 1 0
48 | 小和尚念经下一句是什么 小和尚念经，下一句是什么？ 1 0
49 | 男票，是什么意思？ 男票、女票什么意思？ 1 0
50 | 怎样摆脱手机依赖症 如何摆脱手机依赖症 1 0
51 | 世界上最大的岛屿是 世界最大的岛屿？ 1 0
52 | 过年手机一般会降价么？ 过年手机会降价吗 1 0
53 | 您好，您是本人操作的吗 请您让本人操作 0 0
54 | 想找个情侣头像 谁帮我找个情侣头像吖 1 0
55 | 您好您的情况小二已经帮您反馈您后续关注一下您的手机和邮箱的信息. 您好.您的情况已经帮您反馈您可以后续关注一下您的手机和邮箱信息. 1 0
56 | 西游记是什么小说 西游记是什么体的小说 1 0
57 | 求！这张图片的高清大图！ 求这张图片的高清大图。 1 0
58 | 吃榴莲不能吃什么 榴莲吃了不能吃什么 1 0
59 | 苹果手机铃声叫什么啊 苹果手机是什么铃声 1 0
60 | 眼皮老是跳怎么回事 左眼皮老是跳是怎么回事 1 0
61 | 灵魂是什么？ 灵魂是什么？由来是什么？ 1 0
62 | 请问QQ游戏的欢乐斗地主比斗地主好玩很多吗？ 欢乐斗地主里面的斗牛游戏好玩吗？可以下载吗 0 0
63 | 薏米红豆水一天喝多少 红豆薏米一天喝多少 1 0
64 | 都在说大老虎什么意思 老虎油，什么意思？ 0 0
65 | 天津艺术职业学院有普通类专业 天津城市职业学院电脑艺术设计专业要考专业吗？ 0 0
66 | 传说中的黑洞是怎样形成的？ 斑竹是怎么形成的？ 0 0
67 | 什么意思？翻译一下。 翻译一下什么意思？ 1 0
68 | 想给我儿子起个名字： 我想给儿子起个名字 1 0
69 | 数学几何证明题 几何数学证明题 1 0
70 | 疯狂猜成语的答案是什么？ 疯狂猜成语关于嘴的成语有哪些 0 0
71 | 什么办法才能让鼻梁长高 有什么让鼻梁变高的方法 1 0
72 | 室内设计和计算机信息管理哪个专业好 计算机信息管理和电子商务哪个专业更好 0 0
73 | 暗黑魔法师崛起大理石手 暗黑魔法师：崛起缺少文件 0 0
74 | 为什么女人喜欢男人吃下面 为什么男人喜欢亲女人下面？ 0 0
75 | 蓝颜红颜是什么 红颜和蓝颜分别指什么？ 1 0
76 | 如何坚持马克思主义社会科学方法论的指导地位 如何理解马克思主义社会科学方法论的革命性变革 0 0
77 | 魏晨你喜欢吗 那么爱你为什么的吉他谱简单吗 0 0
78 | 璀璨人生全集在哪里可以看？ 哪里有璀璨人生的全集？ 1 0
79 | 这是什么动漫？叫什么？ 这是什么动漫叫什么名字？ 0 0
80 | 我什么时候放假？ 你什么时候放假？ 0 0
81 | 武林外传好看吗 谁有武林外传体验区激活码啊 0 0
82 | wd做个假体丰胸手术多少钱？最近有人做了吗 怎样才能有效丰胸？假如做隆胸手术得多少钱？ 0 0
83 | 这个头像的大图 求这个头像大图！ 1 0
84 | 十全十美是什么动物？ 十全十美的动物是什么 1 0
85 | 《步步惊情》什么时候上映 步步惊情什么时候上映？ 1 0
86 | 什么是婚育证明怎么写 村里开一胎生育证明怎么写 0 0
87 | 和您购买这个手机号的时间，麻烦您了。 手机号的购买的时间麻烦您您提供一下 1 0
88 | 双子座与狮子座配吗 双子座和狮子座和吗 1 0
89 | 该不该请领导吃饭 怎么暗示请领导吃饭 0 0
90 | 请问这是什么字体？ 请问大家这是什么字体？ 1 0
91 | 为什么天会下雨了 过年了为什么反而不放假了 0 0
92 | 为什么奥比岛打不开？ 为什么奥比岛打不开 1 0
93 | 梦见自己生了个男孩子。 我梦见自己生了男孩子 1 0
94 | 正确的反义词是什么？ 正确的反义词是什么 1 0
95 | 珠海有什么好玩的地方 珠海有什么好玩的地方？ 1 0
96 | 孕妇可以吃黄瓜吗加了醋的 孕妇能吃黄瓜吗 1 0
97 | 儿童音乐乐园雅马哈音乐中心怎么样 儿童钢琴专业课程雅马哈音乐中心怎么样 0 0
98 | 淘宝换货怎么换？ 淘宝怎么换货 1 0
99 | 想学摄影应该买什么书比较好？ 摄影技术去哪里学比较好。 0 0
100 | 传奇登陆器怎么下载 怎么下载传奇登陆器 1 0
101 | 男的做什么工作挣钱 做什么工作挣钱？ 1 0
102 |
--------------------------------------------------------------------------------
/tutorials/data_map/sample_stat_summary.py:
--------------------------------------------------------------------------------
1 | #coding=utf-8
2 |
3 | import sys
4 | import json
5 | import numpy as np
6 |
7 | file_name = "output_data.json"
8 | output_file_names = ["correct_times", "correct_ratio", "avg_probs", "label_var",
9 | "max_label_probs", "min_label_probs", "forgetting_times", "learnt_times", "first_forget", "first_learn", "pred_label", "pred_dist"]
10 | num_samples = 100
11 |
12 | _input_path = "./outputs/"
13 | _output = open(_input_path + file_name + ".result", "w")
14 |
15 | def list_concat(score_dict, input_file, input_path = "./data/", sample_size=1, pred_idx=-2, label_idx=-1, score_idx=-3, get_max_probs=False):
16 | """
17 | add info of each epoch (or given steps) into a dict of lists
18 | """
19 | _input = open(input_path + input_file, "r")
20 | for i, line in enumerate(_input):
21 | info = json.loads(line.strip().replace("\'", "\""))
22 | sid = int(info["id"])
23 | label = int(info["label"]) # [1:-1] to avoid "[]"
24 | if "noisy_label" in info:
25 | s_label = int(info["noisy_label"])
26 | else:
27 | s_label = 0
28 | # score = float(info["probs"])
29 | label_probs = float(info["label_probs"]) # the score under GL
30 | pred_correctness = info["correct"]
31 | if get_max_probs:
32 | all_probs = [eval(j) for j in info["probs"]]
33 | max_probs = np.max(all_probs)
34 | else:
35 | max_probs = 1.0
36 |
37 | score_info = [] # list of scores under different class
38 | for score in info["probs"]: # the number of classes here
39 | score_float = float(score)
40 | score_info.append(score_float)
41 |
42 | if not score_dict["id"][sid]:
43 | score_dict["id"][sid] = sid
44 | score_dict["label"][sid] = label
45 | score_dict["s_label"][sid] = s_label
46 | score_dict["label_probs"][sid].append(label_probs)
47 | score_dict["max_probs"][sid].append(max_probs)
48 | score_dict["pred_info"][sid].append(pred_correctness)
49 | score_dict["pred_label"][sid].append(str(np.argmax(score_info)))
50 |
51 | # add forget info
52 | list_length = len(score_dict["pred_info"][sid])
53 | if list_length > 1:
54 | if score_dict["pred_info"][sid][list_length - 1] == score_dict["pred_info"][sid][list_length - 2]:
55 | score_dict["forget_info"][sid].append("None")
56 | elif score_dict["pred_info"][sid][list_length - 1] == "true":
57 | score_dict["forget_info"][sid].append("Learn")
58 | else:
59 | score_dict["forget_info"][sid].append("Forget")
60 | else:
61 | score_dict["forget_info"][sid].append("None")
62 | #if sid == 1:
63 | # print(score_dict["forget_info"][sid])
64 |
65 | # if i >= sample_size:
66 | # break
67 |
68 | _input.close()
69 |
70 | def check_correct_ratio(correct_lists):
71 | """
72 | ratio that a model predict classes correctly in different epochs
73 | """
74 | if len(correct_lists) == 0 or len(correct_lists[0]) == 0:
75 | return [0], [0]
76 | ratio_list = []
77 | pos_list = []
78 | for c_list in correct_lists:
79 | pos_cnt = 0
80 | for info in c_list:
81 | if info == "true":
82 | pos_cnt += 1
83 | ratio_list.append(float(pos_cnt)/len(c_list) if len(c_list)!=0 else 0)
84 | pos_list.append(pos_cnt)
85 | return pos_list, ratio_list
86 |
87 | def check_forget_time(forget_lists):
88 | if len(forget_lists) == 0 or len(forget_lists[0]) == 0:
89 | return [0], [0], 0, 0
90 | forgetting_list = []
91 | learnt_list = []
92 | first_forgetting_time = []
93 | first_learnt_time = []
94 | for f_list in forget_lists:
95 | forgetting_cnt = 0
96 | learnt_cnt = 0
97 | first_f_time = 0
98 | first_l_time = 0
99 | for i, info in enumerate(f_list):
100 | if info == "Forget":
101 | forgetting_cnt += 1
102 | if first_f_time == 0:
103 | first_f_time = i
104 | elif info == "Learn":
105 | learnt_cnt += 1
106 | if first_l_time == 0:
107 | first_l_time = i
108 | forgetting_list.append(forgetting_cnt)
109 | learnt_list.append(learnt_cnt)
110 | first_forgetting_time.append(first_f_time)
111 | first_learnt_time.append(first_l_time)
112 |
113 | return forgetting_list, learnt_list, first_forgetting_time, first_learnt_time
114 |
115 | def check_pred_distribution(pred_lists):
116 | pred_list = []
117 | for scores in pred_lists:
118 | score_dist_dict = {"0": 0, "1": 0, "2": 0, "3": 0, "4": 0}
119 | for score in scores:
120 | score_dist_dict[score] += 1
121 | pred_list.append(score_dist_dict)
122 | return pred_list
123 |
124 |
125 | info_dict = {"id": [[] for i in range(num_samples)], "label": [[] for i in range(num_samples)],
126 | "s_label": [[] for i in range(num_samples)], "label_probs": [[] for i in range(num_samples)],
127 | "max_probs": [[] for i in range(num_samples)], "pred_info": [[] for i in range(num_samples)],
128 | "forget_info": [[] for i in range(num_samples)], "pred_label": [[] for i in range(num_samples)]}
129 | list_concat(info_dict, file_name, _input_path, sample_size=num_samples)
130 |
131 | print(len(info_dict["label_probs"]), len(info_dict["label_probs"][0]))
132 |
133 | info_dict["correct_times"], info_dict["correct_ratio"] = check_correct_ratio(info_dict["pred_info"])
134 | info_dict["label_var"] = np.var(info_dict["label_probs"], axis=1)
135 | info_dict["max_var"] = np.var(info_dict["max_probs"], axis=1)
136 | info_dict["avg_probs"] = np.mean(info_dict["label_probs"], axis=1)
137 | info_dict["max_label_probs"] = np.max(info_dict["label_probs"], axis=1)
138 | info_dict["min_label_probs"] = np.min(info_dict["label_probs"], axis=1)
139 | info_dict["forgetting_times"], info_dict["learnt_times"], info_dict["first_forget"], info_dict["first_learn"] = check_forget_time(info_dict["forget_info"])
140 | info_dict["pred_dist"] = check_pred_distribution(info_dict["pred_label"])
141 | output_file_names = ["id", "label", "s_label"] + output_file_names
142 |
143 | _output.write("\t".join(output_file_names) + "\n")
144 | for i in range(num_samples):
145 | info_list = []
146 | for name in output_file_names:
147 | info_list.append(str(info_dict[name][i]))
148 | _output.write("\t".join(info_list) + "\n")
149 |
150 | _output.close()
--------------------------------------------------------------------------------
/tutorials/dirty_data_identification/README.md:
--------------------------------------------------------------------------------
1 | # 训练数据中脏数据识别
2 |
3 | ### 方法介绍
4 | 训练数据标注质量对模型效果有较大影响，但受限于标注人员水平、标注任务难易程度等影响，训练数据中都存在一定比例的标注错误的数据（称为**脏数据**）。当标注数据规模较大时，数据标注检查就成为一个难题。
5 |
6 |
7 | 
8 | 图1 脏数据识别及标注修正策略流程
9 |
10 |
11 | TrustAI提供了"脏数据识别 -> 修正"方案，如图1所示。首先，基于实例级证据分析方法识别候选脏数据（对模型训练影响较大的数据）。然后，对候选脏数据进行标注修正，使用修正后的数据重新训练模型，可显著提升模型效果。
12 |
13 | 注：开发者可访问[ AI Studio示例 ](https://aistudio.baidu.com/aistudio/projectdetail/4434058)快速体验本案例。
14 |
15 | ## 实验步骤
16 | 我们以基于相似度计算任务LCQMC数据集上的模拟实验为例，介绍该方案实现步骤和效果。
17 |
18 |
19 | **Step 1**：从LCQMC训练集中随机抽取5000条数据作为新训练集。基于ERNIE-3.0-base-zh在新训练集`train_5000.tsv`微调得到基线模型，运行命令如下所示：
20 |
21 | ```shell
22 | # 下载数据
23 | wget --no-check-certificate https://trustai.bj.bcebos.com/application_data/dirty_data.tar && tar xf dirty_data.tar && rm dirty_data.tar
24 | # 训练基线模型
25 | python -u train.py --dataset_dir ./data --train_file train_5000.tsv --dev_file dev.tsv --test_files test.tsv --num_classes 2 --save_dir ./checkpoint
26 | ```
27 | 训练的基线模型保存在`checkpoint`路径下。
28 |
29 |
30 | **Step 2**：识别训练集中的脏数据。
31 | 脏数据选择方法：基于TrustAI提供的实例级可信分析方法`RepresenterPointModel`，计算每一条训练样本对模型loss的影响分数，一般该分数表明了样本作为脏数据的可能性。我们使用这个分数识别脏数据。
32 |
33 | ```shell
34 | # 从训练集中识别候选脏数据
35 | python -u find_dirty_data.py --dataset_dir ./data --train_file train_5000.tsv --num_classes 2 --rest_path ./data/rest_train.tsv --init_from_ckpt ./checkpoint/model_state.pdparams --dirty_path ./data/dirty_train.tsv --dirty_num 500
36 | # dirty_num表示选取候选脏数据的数量
37 | # dirty_path表示候选脏数据的存储路径
38 | ```
39 |
40 |
41 | **Step 3**：对候选脏数据（在我们的实验中，其占比为全部训练集10%）进行标注修正，修正后的数据保存在`correction_data.tsv`（数据修正比例为**38.4%**，随机选择数据其需要修正的数据比例为**5.0%**）。
42 |
43 |
44 | **Step 4**：使用修正后的新训练集`train_5000_correction.tsv`重新训练模型，并评估模型效果。
45 | ```shell
46 | # 下载数据：含train_5000_correction.tsv文件
47 | python -u train.py --dataset_dir ./data --train_file train_5000_correction.tsv --dev_file dev.tsv --test_files test.tsv DuQM --num_classes 2 --save_dir ./new_checkpoint
48 | ```
49 |
50 | 由下表可知，候选脏数据修正（规模为原始训练集的10%）后，模型在LCQMC测试集上提升2.13%，在DuQM数据集上提升4.01%。
51 |
52 |
53 | | 数据集 | LCQMCdev | LCQMCtest | DuQM |
54 | | :-------: | :-----: | :-----: |:-----: |
55 | | 基线 | 86.42% | 84.87% | 69.51% |
56 | | 数据修正 | 87.76% | 86.62% | 73.18% |
57 |
58 | 注：以上结果均为10次实验的平均值。
59 |
--------------------------------------------------------------------------------
/tutorials/dirty_data_identification/utils.py:
--------------------------------------------------------------------------------
1 | # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | import numpy as np
16 |
17 | import paddle
18 | import paddle.nn.functional as F
19 | from paddlenlp.utils.log import logger
20 |
21 |
22 | @paddle.no_grad()
23 | def evaluate(model, criterion, metric, data_loader, name=''):
24 | """
25 | Given a dataset, it evaluates model and computes the metric.
26 | Args:
27 | model(obj:`paddle.nn.Layer`): A model to classify texts.
28 | criterion(obj:`paddle.nn.Layer`): It can compute the loss.
29 | metric(obj:`paddle.metric.Metric`): The evaluation metric.
30 | data_loader(obj:`paddle.io.DataLoader`): The dataset loader which generates batches.
31 | """
32 |
33 | model.eval()
34 | metric.reset()
35 | losses = []
36 | for batch in data_loader:
37 | input_ids, token_type_ids, labels = batch['input_ids'], batch[
38 | 'token_type_ids'], batch['labels']
39 | logits = model(input_ids, token_type_ids)
40 | loss = criterion(logits, labels)
41 | losses.append(loss.numpy())
42 | correct = metric.compute(logits, labels)
43 | metric.update(correct)
44 |
45 | acc = metric.accumulate()
46 | logger.info("%s: eval loss: %.5f, acc: %.5f" % (name, np.mean(losses), acc))
47 | model.train()
48 | metric.reset()
49 |
50 | return acc
51 |
52 |
53 | def preprocess_function(example, tokenizer, max_seq_length, is_test=False):
54 | """
55 | Builds model inputs from a sequence for sequence classification tasks
56 | by concatenating and adding special tokens.
57 |
58 | Args:
59 | example(obj:`list[str]`): input data, containing text and label if it have label.
60 | tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer`
61 | which contains most of the methods. Users should refer to the superclass for more information regarding methods.
62 | max_seq_length(obj:`int`): The maximum total input sequence length after tokenization.
63 | Sequences longer than this will be truncated, sequences shorter will be padded.
64 | label_nums(obj:`int`): The number of the labels.
65 | Returns:
66 | result(obj:`dict`): The preprocessed data including input_ids, token_type_ids, labels.
67 | """
68 | if 'text_b' not in example:
69 | result = tokenizer(text=example["text_a"], max_seq_len=max_seq_length)
70 | else:
71 | result = tokenizer(text=example["text_a"], text_pair=example['text_b'], max_seq_len=max_seq_length)
72 |
73 | if not is_test:
74 | result["labels"] = np.array([example['label']], dtype='int64')
75 | return result
76 |
--------------------------------------------------------------------------------
/tutorials/enhanced_by_rationale/README.md:
--------------------------------------------------------------------------------
1 | # 基于证据指导的模型增强方案
2 | ## 方法介绍
3 |
4 | 经过对多个模型预测依赖证据的评估，发现深度学习模型提供的证据合理性偏弱。为进一步提高证据的合理性，TrustAI提供了基于证据指导的模型增强方案（[Jayaram etc. 2021](https://aclanthology.org/2021.emnlp-main.450/)），即标注少量证据数据，通过联合学习原始任务和证据学习任务，用证据学习目标指导模型依赖合理的证据进行预测，提升模型可解释性。
5 |
6 |
7 | ## 实验步骤
8 |
9 | 实验基于ERNIE-2.0-EN-Base在英文情感分析SST数据集上微调得到基线模型，然后选择1000条训练数据进行证据人工标注，并基于这1000条数据进行证据学习。最终，在500条标有证据的验证数据上进行效果评估。评估指标除了模型预测准确率外，还包括可解释评估指标，即证据的合理性、充分性和完备性。
10 |
11 |
12 | 我们实验中使用数据可通过如下命令下载：
13 | ```shell
14 | # 下载样例数据，每个文件仅包含两条样例数据，开发者可根据样例数据的格式自行标注证据数据
15 | wget --no-check-certificate https://trustai.bj.bcebos.com/application_data/rationale_data.tar && tar xf rationale_data.ta && rm rationale_data.ta
16 | ```
17 |
18 | 数据下载后，通过`train.py`进行模型训练，该训练过程引入了证据学习目标，指导模型依赖合理的证据进行预测。
19 | ```shell
20 | python -u train.py --dataset_dir ./data --train_file train.tsv --dev_file dev.tsv --num_classes 2 --save_dir ./maw --use_maw
21 | # user_maw表示是否使用证据增强模型效果
22 | ```
23 |
24 | 实验结果如下表，在加入证据指导后，模型预测效果略有提升，准确率提升0.5%；模型可解释性提升明显：证据合理性提升5.0%、充分性降低0.185（该指标越低越好）、完备性提升0.044。
25 |
26 | | 数据集 | 准确率 | 合理性 | 充分性 | 完备性 |
27 | | :-------: | :-----: | :-----: | :-----: | :-----: |
28 | | base | 93.5% | 26.1% | 0.367 | 0.118 |
29 | | base + maw loss | 94.0% | 31.1% | 0.182 | 0.162 |
30 |
31 | 注：以上结果均为3次实验的平均值。
32 |
--------------------------------------------------------------------------------
/tutorials/enhanced_by_rationale/utils.py:
--------------------------------------------------------------------------------
1 | # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | import numpy as np
16 | import itertools
17 |
18 | import paddle
19 | import paddle.nn.functional as F
20 | from paddlenlp.utils.log import logger
21 |
22 |
23 | @paddle.no_grad()
24 | def evaluate(model, criterion, metric, data_loader, name=''):
25 | """
26 | Given a dataset, it evaluates model and computes the metric.
27 | Args:
28 | model(obj:`paddle.nn.Layer`): A model to classify texts.
29 | criterion(obj:`paddle.nn.Layer`): It can compute the loss.
30 | metric(obj:`paddle.metric.Metric`): The evaluation metric.
31 | data_loader(obj:`paddle.io.DataLoader`): The dataset loader which generates batches.
32 | """
33 |
34 | model.eval()
35 | metric.reset()
36 | losses = []
37 | for batch in data_loader:
38 | input_ids, token_type_ids, labels, _, _ = batch
39 | logits = model(input_ids, token_type_ids)
40 | loss = criterion(logits, labels)
41 | losses.append(loss.numpy())
42 | correct = metric.compute(logits, labels)
43 | metric.update(correct)
44 |
45 | acc = metric.accumulate()
46 | logger.info("%s: eval loss: %.5f, acc: %.5f" % (name, np.mean(losses), acc))
47 | model.train()
48 | metric.reset()
49 |
50 | return acc
51 |
52 |
53 | def preprocess_function(example, tokenizer, max_seq_length, is_test=False):
54 | """
55 | Builds model inputs from a sequence for sequence classification tasks
56 | by concatenating and adding special tokens.
57 |
58 | Args:
59 | example(obj:`list[str]`): input data, containing text and label if it have label.
60 | tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer`
61 | which contains most of the methods. Users should refer to the superclass for more information regarding methods.
62 | max_seq_length(obj:`int`): The maximum total input sequence length after tokenization.
63 | Sequences longer than this will be truncated, sequences shorter will be padded.
64 | label_nums(obj:`int`): The number of the labels.
65 | Returns:
66 | result(obj:`dict`): The preprocessed data including input_ids, token_type_ids, labels.
67 | """
68 | if is_test:
69 | result = tokenizer(text=example['text_a'], max_seq_len=max_seq_length, return_attention_mask=True)
70 | return result['input_ids'], result['token_type_ids']
71 | else:
72 | tokens = example['tokens']
73 | rationales = example['rationales']
74 | tokens = [tokenizer._tokenize(token) for token in tokens]
75 | assert len(tokens) == len(rationales)
76 | rationales = list(
77 | itertools.chain(*[[rationale] * len(sub_tokens) for sub_tokens, rationale in zip(tokens, rationales)]))
78 | tokens = list(itertools.chain(*tokens))
79 | result = tokenizer(text=tokens,
80 | max_seq_len=max_seq_length,
81 | is_split_into_words=True,
82 | return_attention_mask=True)
83 | input_ids = result["input_ids"]
84 | token_type_ids = result["token_type_ids"]
85 | attention_mask = result["attention_mask"]
86 | seq_len = len(input_ids)
87 | rationales = [0] + rationales[:seq_len - 2] + [0]
88 | assert len(rationales) == seq_len
89 | label = np.array([example['label']], dtype="int64")
90 | return input_ids, token_type_ids, label, rationales, attention_mask
91 |
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/args.py:
--------------------------------------------------------------------------------
1 | # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | import argparse
16 |
17 |
18 | def parse_args():
19 | parser = argparse.ArgumentParser(description=__doc__)
20 | parser.add_argument("--model_name", type=str, required=True, help="Name of pre-trained model.")
21 | parser.add_argument("--output_dir",
22 | type=str,
23 | required=True,
24 | help="The output directory where the model predictions and checkpoints will be written.")
25 | parser.add_argument(
26 | "--data_dir",
27 | type=str,
28 | required=True,
29 | help="The data directory should include `train` and `dev` set to train model and `test` set to test model.")
30 | parser.add_argument("--max_seq_length",
31 | default=512,
32 | type=int,
33 | help="The maximum total input sequence length after tokenization.")
34 | parser.add_argument("--batch_size", default=24, type=int, help="Batch size per GPU/CPU for training.")
35 | parser.add_argument("--learning_rate", default=7e-5, type=float, help="The initial learning rate for Adam.")
36 | parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.")
37 | parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
38 | parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.")
39 | parser.add_argument("--num_train_epochs", default=20, type=int, help="Total number of train epochs to perform.")
40 | parser.add_argument("--max_steps",
41 | default=-1,
42 | type=int,
43 | help="If > 0: set total number of training steps to perform. Override num_train_epochs.")
44 | parser.add_argument("--warmup_proportion",
45 | default=0.0,
46 | type=float,
47 | help="Proportion of training steps to perform linear learning rate warmup for.")
48 | parser.add_argument("--logging_steps", type=int, default=10, help="Log every X updates steps.")
49 | parser.add_argument("--save_steps", type=int, default=200, help="Save checkpoint every X updates steps.")
50 | parser.add_argument("--load_model_path", type=str, default=None, help="The checkpoint directory where the model")
51 | parser.add_argument("--get_k_sentences", type=int, default=0, help="load checkpoint path")
52 | parser.add_argument("--set_k_sentences_ground_true", type=int, default=0, help="set k sentences ground true")
53 | parser.add_argument("--early_stop_nums", type=int, default=5, help="probability threshold for selecting sentences")
54 | parser.add_argument("--one_alpha", type=float, default=0.4, help="probability threshold for selecting sentences")
55 | parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")
56 | parser.add_argument('--device',
57 | choices=['cpu', 'gpu'],
58 | default="gpu",
59 | help="Select which device to train model, defaults to gpu.")
60 | parser.add_argument("--doc_stride",
61 | type=int,
62 | default=128,
63 | help="When splitting up a long document into chunks, how much stride to take between chunks.")
64 | parser.add_argument(
65 | "--n_best_size",
66 | type=int,
67 | default=20,
68 | help="The total number of n-best predictions to generate in the nbest_predictions.json output file.")
69 | parser.add_argument("--max_query_length", type=int, default=64, help="Max query length.")
70 | parser.add_argument("--max_answer_length", type=int, default=30, help="Max answer length.")
71 | parser.add_argument("--hidden_size", type=int, default=768, help="hidden size")
72 | parser.add_argument("--verbose", action='store_true', help="Whether to output verbose log.")
73 | parser.add_argument("--do_train", action='store_true', help="Whether to train the model.")
74 | parser.add_argument("--do_predict", action='store_true', help="Whether to predict.")
75 | parser.add_argument("--use_loose_metric", action='store_true', help="whether to use loose metric to choose model.")
76 | parser.add_argument("--use_similarity", action='store_true', help="whether to use similarity to choose sentence.")
77 | parser.add_argument("--early_stop", action='store_true', help="whether to use early stop.")
78 | args = parser.parse_args()
79 | return args
80 |
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/download.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | # Download dataset and model parameters
3 | set -e
4 |
5 | if [ ! -d "./data" ]; then
6 | mkdir "./data"
7 | fi
8 |
9 | echo "Download DuReader-robust dataset"
10 | wget --no-check-certificate https://dataset-bj.cdn.bcebos.com/dureader_robust/data/dureader_robust-data.tar.gz
11 | tar -zxvf dureader_robust-data.tar.gz
12 | mv dureader_robust-data data/robust
13 | rm dureader_robust-data.tar.gz
14 |
15 | echo "Download DuReader-checklist dataset"
16 | wget --no-check-certificate https://dataset-bj.cdn.bcebos.com/lic2021/dureader_checklist.dataset.tar.gz
17 | tar -zxvf dureader_checklist.dataset.tar.gz
18 | mv dataset data/checklist
19 | rm dureader_checklist.dataset.tar.gz
20 | mkdir ./data/checklist_wo_no_answer
21 | python ./utils/checklist_process.py --input_data_dir ./data/checklist --output_data_dir ./data/checklist_wo_no_answer
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/predictor/model.py:
--------------------------------------------------------------------------------
1 | import paddle
2 | from paddlenlp.transformers import AutoModelForQuestionAnswering
3 |
4 |
5 | class Predictor(paddle.nn.Layer):
6 |
7 | def __init__(self, args):
8 | super(Predictor, self).__init__()
9 | self.model = AutoModelForQuestionAnswering.from_pretrained(args.model_name)
10 |
11 | def forward(self, x, mode="train"):
12 | data = {
13 | "input_ids": x[0],
14 | "token_type_ids": x[1],
15 | }
16 | if mode == "train":
17 | data["start_positions"] = x[2]
18 | data["end_positions"] = x[3]
19 |
20 | logits = self.model(input_ids=data["input_ids"], token_type_ids=data["token_type_ids"])
21 |
22 | if mode == "dev" or mode == "test":
23 | return logits
24 |
25 | # Compute loss
26 | start_logits, end_logits = logits
27 | start_position = paddle.unsqueeze(data["start_positions"], axis=-1)
28 | end_position = paddle.unsqueeze(data["end_positions"], axis=-1)
29 | start_loss = paddle.nn.functional.cross_entropy(input=start_logits, label=start_position)
30 | end_loss = paddle.nn.functional.cross_entropy(input=end_logits, label=end_position)
31 | loss = (start_loss + end_loss) / 2
32 | return loss, logits
33 |
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/requirements.txt:
--------------------------------------------------------------------------------
1 | numpy
2 | paddlenlp
3 | paddlepaddle-gpu >= 2.0.0
4 | scikit-learn
5 | tqdm
6 | matplotlib
7 | IPython
8 | pre-commit
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/run_predict.py:
--------------------------------------------------------------------------------
1 | # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | import random
16 |
17 | import numpy as np
18 | import paddle
19 |
20 | from args import parse_args
21 | from predictor import model, model_manager
22 |
23 |
24 | def set_seed(args):
25 | random.seed(args.seed)
26 | np.random.seed(args.seed)
27 | paddle.seed(args.seed)
28 |
29 |
30 | def run(args):
31 | paddle.set_device(args.device)
32 | if paddle.distributed.get_world_size() > 1:
33 | paddle.distributed.init_parallel_env()
34 | rank = paddle.distributed.get_rank()
35 |
36 | set_seed(args)
37 |
38 | predictor = model.Predictor(args)
39 | if paddle.distributed.get_world_size() > 1:
40 | predictor = paddle.DataParallel(predictor)
41 |
42 | # Prepare model manager
43 | manager = model_manager.ModelManager(args, predictor)
44 |
45 | if args.do_train:
46 | manager.train(rank)
47 | if args.do_predict and rank == 0:
48 | manager.test()
49 |
50 |
51 | if __name__ == "__main__":
52 | args = parse_args()
53 | run(args)
54 |
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/run_select.py:
--------------------------------------------------------------------------------
1 | # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | import random
16 |
17 | import numpy as np
18 | import paddle
19 |
20 | from args import parse_args
21 | from selector import model, model_manager
22 |
23 |
24 | def set_seed(args):
25 | random.seed(args.seed)
26 | np.random.seed(args.seed)
27 | paddle.seed(args.seed)
28 |
29 |
30 | def run(args):
31 | set_seed(args)
32 |
33 | # Prepare device and model
34 | paddle.set_device(args.device)
35 | if paddle.distributed.get_world_size() > 1:
36 | paddle.distributed.init_parallel_env()
37 | rank = paddle.distributed.get_rank()
38 |
39 | selector = model.Selector(args)
40 | if paddle.distributed.get_world_size() > 1:
41 | selector = paddle.DataParallel(selector)
42 |
43 | # Prepare model manager
44 | manager = model_manager.ModelManager(args, selector)
45 |
46 | if args.do_train:
47 | manager.train(rank)
48 |
49 | if args.do_predict and rank == 0:
50 | manager.test()
51 |
52 |
53 | if __name__ == "__main__":
54 | args = parse_args()
55 | run(args)
56 |
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/test.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | data_dir="./data/robust"
3 | output_dir="./output"
4 | load_selector_model_dir="./"
5 | load_predictor_model_dir="./"
6 | split="dev"
7 |
8 | ARGS=`getopt -a -o :d:o:s:p:S:h -l data_dir:,output_dir:,selector_model_dir:,predictor_model_dir:,split:,help -- "$@"`
9 | eval set -- "${ARGS}"
10 | while true
11 | do
12 | case "$1" in
13 | -d|--data_dir)
14 | data_dir="$2"
15 | shift
16 | ;;
17 | -o|--output_dir)
18 | output_dir="$2"
19 | shift
20 | ;;
21 | -s|--selector_model_dir)
22 | load_selector_model_dir="$2"
23 | shift
24 | ;;
25 | -p|--predictor_model_dir)
26 | load_predictor_model_dir="$2"
27 | shift
28 | ;;
29 | -S|--split)
30 | split="$2"
31 | shift
32 | ;;
33 | -h|--help)
34 | echo "help"
35 | ;;
36 | --)
37 | shift
38 | break
39 | ;;
40 | esac
41 | shift
42 | done
43 |
44 | if [ ! -d "./cache" ]; then
45 | mkdir "./cache"
46 | fi
47 | if [ ! -d "${output_dir}" ]; then
48 | mkdir "${output_dir}"
49 | fi
50 | if [ ! -d "${output_dir}/selected-test-data" ]; then
51 | mkdir "${output_dir}/selected-test-data"
52 | fi
53 | if [ $split = "dev" ]; then
54 | if [ ! -d "${output_dir}/tmp" ]; then
55 | mkdir "${output_dir}/tmp"
56 | fi
57 | cp "${data_dir}/dev.json" "${output_dir}/tmp/test.json"
58 | echo "Data path: ${data_dir}/dev.json"
59 | data_dir="${output_dir}/tmp"
60 | else
61 | echo "Data path: ${data_dir}/test.json"
62 | fi
63 |
64 | echo "Output path: ${output_dir}/selected-test-data"
65 |
66 | python -u run_select.py \
67 | --model_name hfl/roberta-wwm-ext \
68 | --do_predict \
69 | --batch_size 24 \
70 | --data_dir $data_dir \
71 | --load_model_path "${load_selector_model_dir}/model_state.pdparams" \
72 | --one_alpha 0.1 \
73 | --output_dir "${output_dir}/selected-test-data/" \
74 | --device gpu
75 |
76 | mv "${output_dir}/selected-test-data/test_prediction.json" "${output_dir}/selected-test-data/test.json"
77 |
78 | python -u run_predict.py \
79 | --model_name hfl/roberta-wwm-ext \
80 | --do_predict \
81 | --batch_size 24 \
82 | --max_seq_length 384 \
83 | --data_dir "${output_dir}/selected-test-data/" \
84 | --load_model_path "${load_predictor_model_dir}/model_state.pdparams" \
85 | --output_dir "${output_dir}/predict-result/" \
86 | --device gpu
87 |
88 |
89 |
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/train.sh:
--------------------------------------------------------------------------------
1 |
2 | data_dir="./data/robust"
3 | output_dir="./output"
4 |
5 | # Read Parameter
6 | # -d data dir path
7 | # -o output_dir
8 | while getopts ":d:o:" optname
9 | do
10 | case "$optname" in
11 | "d")
12 | data_dir=$OPTARG
13 | ;;
14 | "o")
15 | output_dir=$OPTARG
16 | ;;
17 | ":")
18 | echo "No argument value for option $OPTARG"
19 | ;;
20 | "?")
21 | echo "Unknown option $OPTARG"
22 | ;;
23 | *)
24 | echo "Unknown error while processing options"
25 | ;;
26 | esac
27 | done
28 |
29 | echo "Data path: ${data_dir}"
30 | echo "Selector Output path: ${output_dir}/selector/"
31 |
32 | # prepare dir
33 |
34 | # clean cache
35 | if [ ! -d "./cache" ]; then
36 | mkdir "./cache"
37 | else
38 | rm -rf "./cache"
39 | mkdir "./cache"
40 | fi
41 | if [ ! -d "${output_dir}" ]; then
42 | mkdir "${output_dir}"
43 | fi
44 | if [ ! -d "${output_dir}/selector" ]; then
45 | mkdir "${output_dir}/selector"
46 | fi
47 | if [ ! -d "${output_dir}/selected-data" ]; then
48 | mkdir "${output_dir}/selected-data"
49 | fi
50 | if [ ! -d "${output_dir}/predictor" ]; then
51 | mkdir "${output_dir}/predictor"
52 | fi
53 | if [ ! -d "${output_dir}/tmp" ]; then
54 | mkdir "${output_dir}/tmp"
55 | fi
56 |
57 | echo "########## Selector Training #############"
58 |
59 | # train selector
60 | python -u ./run_select.py \
61 | --model_name hfl/roberta-wwm-ext \
62 | --max_seq_length 512 \
63 | --batch_size 24 \
64 | --learning_rate 8e-5 \
65 | --num_train_epochs 100 \
66 | --logging_steps 10 \
67 | --save_steps 200 \
68 | --warmup_proportion 0.1 \
69 | --weight_decay 0.01 \
70 | --output_dir "${output_dir}/selector/" \
71 | --data_dir ${data_dir} \
72 | --set_k_sentences_ground_true 0 \
73 | --early_stop_nums 5 \
74 | --one_alpha -1 \
75 | --do_train \
76 | --use_loose_metric \
77 | --early_stop \
78 | --device gpu
79 |
80 |
81 | echo "########## Dev Processing #############"
82 | cp "${data_dir}/dev.json" "${output_dir}/tmp/test.json"
83 |
84 | # predict selector
85 | python -u ./run_select.py \
86 | --model_name hfl/roberta-wwm-ext \
87 | --max_seq_length 512 \
88 | --batch_size 24 \
89 | --load_model_path "${output_dir}/selector/best_model/model_state.pdparams" \
90 | --data_dir "${output_dir}/tmp/" \
91 | --output_dir "${output_dir}/selector/" \
92 | --set_k_sentences_ground_true 0 \
93 | --one_alpha 0.1 \
94 | --do_predict \
95 | --use_loose_metric \
96 | --device gpu
97 |
98 | rm -rf "${output_dir}/tmp"
99 | # Postprocess selected data
100 | temp_dir="${data_dir}/*"
101 | cp -f ${temp_dir} "${output_dir}/selected-data"
102 | rm -f "${output_dir}/selected-data/dev.json"
103 | origin_dev_path="${output_dir}/selector/test_prediction.json"
104 | mv ${origin_dev_path} "${output_dir}/selected-data/dev.json"
105 |
106 |
107 | rm -rf "${output_dir}/tmp"
108 | # Postprocess selected data
109 | temp_dir="${data_dir}/*"
110 | cp -f ${temp_dir} "${output_dir}/selected-data"
111 | rm -f "${output_dir}/selected-data/dev.json"
112 | origin_dev_path="${output_dir}/selector/test_prediction.json"
113 | mv ${origin_dev_path} "${output_dir}/selected-data/dev.json"
114 |
115 | echo "########## Predictor Training #############"
116 |
117 | # Train predictor
118 | python -u run_predict.py \
119 | --model_name hfl/roberta-wwm-ext \
120 | --max_seq_length 384 \
121 | --batch_size 24 \
122 | --learning_rate 5e-5 \
123 | --num_train_epochs 8 \
124 | --logging_steps 10 \
125 | --save_steps 100 \
126 | --warmup_proportion 0.1 \
127 | --weight_decay 0.01 \
128 | --output_dir "$output_dir/predictor/" \
129 | --data_dir "$output_dir/selected-data/" \
130 | --do_train \
131 | --device gpu
132 |
133 | python -u run_predict.py \
134 | --model_name hfl/roberta-wwm-ext \
135 | --max_seq_length 384 \
136 | --batch_size 24 \
137 | --learning_rate 5e-5 \
138 | --num_train_epochs 8 \
139 | --logging_steps 10 \
140 | --save_steps 200 \
141 | --warmup_proportion 0.1 \
142 | --weight_decay 0.01 \
143 | --output_dir "$output_dir/predictor/" \
144 | --data_dir "$output_dir/selected-data/" \
145 | --do_predict \
146 | --device gpu
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/train_predictor.sh:
--------------------------------------------------------------------------------
1 | data_dir="./data/robust"
2 | output_dir="./output"
3 |
4 | # Read Parameter
5 | # -d data dir path
6 | # -o output_dir
7 | while getopts ":d:o:" optname
8 | do
9 | case "$optname" in
10 | "d")
11 | data_dir=$OPTARG
12 | ;;
13 | "o")
14 | output_dir=$OPTARG
15 | ;;
16 | ":")
17 | echo "No argument value for option $OPTARG"
18 | ;;
19 | "?")
20 | echo "Unknown option $OPTARG"
21 | ;;
22 | *)
23 | echo "Unknown error while processing options"
24 | ;;
25 | esac
26 | done
27 |
28 | echo "Data path: ${data_dir}"
29 |
30 | # prepare dir
31 |
32 | # clean cache
33 | if [ ! -d "./cache" ]; then
34 | mkdir "./cache"
35 | else
36 | rm -rf "./cache"
37 | mkdir "./cache"
38 | fi
39 | if [ ! -d "${output_dir}" ]; then
40 | mkdir "${output_dir}"
41 | fi
42 | if [ ! -d "${output_dir}/selector" ]; then
43 | mkdir "${output_dir}/selector"
44 | fi
45 | if [ ! -d "${output_dir}/selected-data" ]; then
46 | mkdir "${output_dir}/selected-data"
47 | fi
48 | if [ ! -d "${output_dir}/predictor" ]; then
49 | mkdir "${output_dir}/predictor"
50 | fi
51 | if [ ! -d "${output_dir}/tmp" ]; then
52 | mkdir "${output_dir}/tmp"
53 | fi
54 |
55 | echo "########## Predictor Training #############"
56 |
57 | # Train predictor
58 | python -u run_predict.py \
59 | --model_name hfl/roberta-wwm-ext \
60 | --max_seq_length 384 \
61 | --batch_size 24 \
62 | --learning_rate 5e-5 \
63 | --num_train_epochs 8 \
64 | --logging_steps 10 \
65 | --save_steps 100 \
66 | --warmup_proportion 0.1 \
67 | --weight_decay 0.01 \
68 | --output_dir "$output_dir/predictor/" \
69 | --data_dir "$output_dir/selected-data/" \
70 | --do_train \
71 | --device gpu
72 |
73 | python -u run_predict.py \
74 | --model_name hfl/roberta-wwm-ext \
75 | --max_seq_length 384 \
76 | --batch_size 24 \
77 | --learning_rate 5e-5 \
78 | --num_train_epochs 8 \
79 | --logging_steps 10 \
80 | --save_steps 200 \
81 | --warmup_proportion 0.1 \
82 | --weight_decay 0.01 \
83 | --output_dir "$output_dir/predictor/" \
84 | --data_dir "$output_dir/selected-data/" \
85 | --do_predict \
86 | --device gpu
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/train_select_data.sh:
--------------------------------------------------------------------------------
1 |
2 | data_dir="./data/robust"
3 | output_dir="./output"
4 |
5 | # Read Parameter
6 | # -d data dir path
7 | # -o output_dir
8 | while getopts ":d:o:" optname
9 | do
10 | case "$optname" in
11 | "d")
12 | data_dir=$OPTARG
13 | ;;
14 | "o")
15 | output_dir=$OPTARG
16 | ;;
17 | ":")
18 | echo "No argument value for option $OPTARG"
19 | ;;
20 | "?")
21 | echo "Unknown option $OPTARG"
22 | ;;
23 | *)
24 | echo "Unknown error while processing options"
25 | ;;
26 | esac
27 | done
28 |
29 | echo "Data path: ${data_dir}"
30 | echo "Selector Output path: ${output_dir}/selector/"
31 |
32 | # prepare dir
33 |
34 | # clean cache
35 | if [ ! -d "./cache" ]; then
36 | mkdir "./cache"
37 | else
38 | rm -rf "./cache"
39 | mkdir "./cache"
40 | fi
41 | if [ ! -d "${output_dir}" ]; then
42 | mkdir "${output_dir}"
43 | fi
44 | if [ ! -d "${output_dir}/selector" ]; then
45 | mkdir "${output_dir}/selector"
46 | fi
47 | if [ ! -d "${output_dir}/selected-data" ]; then
48 | mkdir "${output_dir}/selected-data"
49 | fi
50 | if [ ! -d "${output_dir}/predictor" ]; then
51 | mkdir "${output_dir}/predictor"
52 | fi
53 | if [ ! -d "${output_dir}/tmp" ]; then
54 | mkdir "${output_dir}/tmp"
55 | fi
56 |
57 |
58 | echo "########## Dev Processing #############"
59 | cp "${data_dir}/dev.json" "${output_dir}/tmp/test.json"
60 |
61 | # predict selector
62 | python -u ./run_select.py \
63 | --model_name hfl/roberta-wwm-ext \
64 | --max_seq_length 512 \
65 | --batch_size 24 \
66 | --load_model_path "${output_dir}/selector/best_model/model_state.pdparams" \
67 | --data_dir "${output_dir}/tmp/" \
68 | --output_dir "${output_dir}/selector/" \
69 | --set_k_sentences_ground_true 0 \
70 | --one_alpha 0.1 \
71 | --do_predict \
72 | --use_loose_metric \
73 | --device gpu
74 |
75 | rm -rf "${output_dir}/tmp"
76 | # Postprocess selected data
77 | temp_dir="${data_dir}/*"
78 | cp -f ${temp_dir} "${output_dir}/selected-data"
79 | rm -f "${output_dir}/selected-data/dev.json"
80 | origin_dev_path="${output_dir}/selector/test_prediction.json"
81 | mv ${origin_dev_path} "${output_dir}/selected-data/dev.json"
82 |
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/train_selector.sh:
--------------------------------------------------------------------------------
1 |
2 | data_dir="./data/robust"
3 | output_dir="./output"
4 |
5 | # Read Parameter
6 | # -d data dir path
7 | # -o output_dir
8 | while getopts ":d:o:" optname
9 | do
10 | case "$optname" in
11 | "d")
12 | data_dir=$OPTARG
13 | ;;
14 | "o")
15 | output_dir=$OPTARG
16 | ;;
17 | ":")
18 | echo "No argument value for option $OPTARG"
19 | ;;
20 | "?")
21 | echo "Unknown option $OPTARG"
22 | ;;
23 | *)
24 | echo "Unknown error while processing options"
25 | ;;
26 | esac
27 | done
28 |
29 | echo "Data path: ${data_dir}"
30 | echo "Selector Output path: ${output_dir}/selector/"
31 |
32 | # prepare dir
33 |
34 | # clean cache
35 | if [ ! -d "./cache" ]; then
36 | mkdir "./cache"
37 | else
38 | rm -rf "./cache"
39 | mkdir "./cache"
40 | fi
41 | if [ ! -d "${output_dir}" ]; then
42 | mkdir "${output_dir}"
43 | fi
44 | if [ ! -d "${output_dir}/selector" ]; then
45 | mkdir "${output_dir}/selector"
46 | fi
47 | if [ ! -d "${output_dir}/selected-data" ]; then
48 | mkdir "${output_dir}/selected-data"
49 | fi
50 | if [ ! -d "${output_dir}/predictor" ]; then
51 | mkdir "${output_dir}/predictor"
52 | fi
53 | if [ ! -d "${output_dir}/tmp" ]; then
54 | mkdir "${output_dir}/tmp"
55 | fi
56 |
57 | echo "########## Selector Training #############"
58 |
59 | # train selector
60 | python -u ./run_select.py \
61 | --model_name hfl/roberta-wwm-ext \
62 | --max_seq_length 512 \
63 | --batch_size 24 \
64 | --learning_rate 8e-5 \
65 | --num_train_epochs 100 \
66 | --logging_steps 10 \
67 | --save_steps 200 \
68 | --warmup_proportion 0.1 \
69 | --weight_decay 0.01 \
70 | --output_dir "${output_dir}/selector/" \
71 | --data_dir ${data_dir} \
72 | --set_k_sentences_ground_true 0 \
73 | --early_stop_nums 5 \
74 | --one_alpha -1 \
75 | --do_train \
76 | --use_loose_metric \
77 | --early_stop \
78 | --device gpu
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/utils/checklist_process.py:
--------------------------------------------------------------------------------
1 | import json
2 | import tools
3 | import argparse
4 | import os
5 |
6 |
7 | def process(input_data_path, output_data_path):
8 | with open(input_data_path, 'r', encoding='utf-8') as f:
9 | obj = json.load(f)
10 | for i, data in enumerate(obj['data']):
11 | x = []
12 | for j, paragraphs in enumerate(data["paragraphs"]):
13 | for k, ans in enumerate(paragraphs["qas"][0]["answers"]):
14 | answer = ans["text"]
15 | if answer != "" and len(tools.split_sentence(answer)) == 1:
16 | x.append(paragraphs)
17 | else:
18 | break
19 | obj["data"][i]["paragraphs"] = x
20 |
21 | with open(output_data_path, 'w+', encoding="utf8") as outfile:
22 | json.dump(obj, outfile, ensure_ascii=False)
23 |
24 |
25 | if __name__ == "__main__":
26 | parser = argparse.ArgumentParser(description=__doc__)
27 | parser.add_argument("--input_data_dir", type=str, required=True, help="checklist input data dir")
28 | parser.add_argument("--output_data_dir", type=str, required=True, help="checklist output data dir")
29 | args = parser.parse_args()
30 | process(os.path.join(args.input_data_dir, 'train.json'), os.path.join(args.output_data_dir, 'train.json'))
31 | process(os.path.join(args.input_data_dir, 'dev.json'), os.path.join(args.output_data_dir, 'dev.json'))
32 |
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/utils/dureader_robust.py:
--------------------------------------------------------------------------------
1 | # coding=utf-8
2 | # Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
3 | # Copyright 2020 The TensorFlow Datasets Authors and the HuggingFace Datasets Authors.
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | # Lint as: python3
18 |
19 | import json
20 | import os
21 |
22 | import datasets
23 | from datasets.tasks import QuestionAnsweringExtractive
24 |
25 | logger = datasets.logging.get_logger(__name__)
26 |
27 | _DESCRIPTION = """\
28 | DureaderRobust is a chinese reading comprehension \
29 | dataset, designed to evaluate the MRC models from \
30 | three aspects: over-sensitivity, over-stability \
31 | and generalization.
32 | """
33 |
34 | _URL = "https://bj.bcebos.com/paddlenlp/datasets/dureader_robust-data.tar.gz"
35 |
36 |
37 | class DureaderRobustConfig(datasets.BuilderConfig):
38 | """BuilderConfig for DureaderRobust."""
39 | data_dir: str = None
40 | do_train: bool = True
41 | do_predict: bool = True
42 |
43 | def __init__(self, **kwargs):
44 | """BuilderConfig for DureaderRobust.
45 |
46 | Args:
47 | **kwargs: keyword arguments forwarded to super.
48 | """
49 | super(DureaderRobustConfig, self).__init__(**kwargs)
50 |
51 |
52 | class DureaderRobust(datasets.GeneratorBasedBuilder):
53 | BUILDER_CONFIGS = [
54 | DureaderRobustConfig(
55 | name="plain_text",
56 | version=datasets.Version("1.0.0", ""),
57 | description="Plain text",
58 | ),
59 | ]
60 |
61 | def _info(self):
62 | return datasets.DatasetInfo(
63 | description=_DESCRIPTION,
64 | features=datasets.Features({
65 | "id":
66 | datasets.Value("string"),
67 | "title":
68 | datasets.Value("string"),
69 | "context":
70 | datasets.Value("string"),
71 | "question":
72 | datasets.Value("string"),
73 | "answers":
74 | datasets.features.Sequence({
75 | "text": datasets.Value("string"),
76 | "answer_start": datasets.Value("int32"),
77 | }),
78 | }),
79 | # No default supervised_keys (as we have to pass both question
80 | # and context as input).
81 | supervised_keys=None,
82 | homepage="https://arxiv.org/abs/2004.11142",
83 | task_templates=[
84 | QuestionAnsweringExtractive(question_column="question",
85 | context_column="context",
86 | answers_column="answers")
87 | ],
88 | )
89 |
90 | def _split_generators(self, dl_manager):
91 | # dl_dir = dl_manager.download_and_extract(_URL)
92 | result = []
93 | if self.config.do_train:
94 | result += [
95 | datasets.SplitGenerator(name=datasets.Split.TRAIN,
96 | gen_kwargs={"filepath": os.path.join(self.config.data_dir, "train.json")}),
97 | datasets.SplitGenerator(name=datasets.Split.VALIDATION,
98 | gen_kwargs={"filepath": os.path.join(self.config.data_dir, "dev.json")}),
99 | ]
100 | if self.config.do_predict:
101 | result.append(
102 | datasets.SplitGenerator(name=datasets.Split.TEST,
103 | gen_kwargs={"filepath": os.path.join(self.config.data_dir, 'test.json')}))
104 | return result
105 |
106 | def _generate_examples(self, filepath):
107 | """This function returns the examples in the raw (text) form."""
108 | logger.info("generating examples from = %s", filepath)
109 | key = 0
110 | with open(filepath, encoding="utf-8") as f:
111 | durobust = json.load(f)
112 | for article in durobust["data"]:
113 | title = article.get("title", "")
114 | for paragraph in article["paragraphs"]:
115 | context = paragraph["context"] # do not strip leading blank spaces GH-2585
116 | for qa in paragraph["qas"]:
117 | answer_starts = [answer["answer_start"] for answer in qa.get("answers", '')]
118 | answers = [answer["text"] for answer in qa.get("answers", '')]
119 | # Features currently used are "context", "question", and "answers".
120 | # Others are extracted here for the ease of future expansions.
121 | yield key, {
122 | "title": title,
123 | "context": context,
124 | "question": qa["question"],
125 | "id": qa["id"],
126 | "answers": {
127 | "answer_start": answer_starts,
128 | "text": answers,
129 | },
130 | }
131 | key += 1
132 |
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/utils/logger.py:
--------------------------------------------------------------------------------
1 | # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | import json
16 | import os
17 | import time
18 |
19 |
20 | class Logger():
21 |
22 | def __init__(self, args):
23 | self.args = args
24 | self.tic_train = time.time()
25 | self.performance_list = []
26 | self.mode_dict = {
27 | "ERROR": "31",
28 | "INFO": "32",
29 | "WARNING": "33",
30 | }
31 |
32 | def __get_time_str(self):
33 | return time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()) + "," + str(
34 | int(round(time.time() * 1000)) % 1000).zfill(3)
35 |
36 | def __log(self, log_str, mode="INFO"):
37 | print("\033[" + self.mode_dict[mode] + "m[" + self.__get_time_str() + "] [ " + mode + "]\033[0m - " +
38 | log_str)
39 |
40 | def info(self, info_str: str):
41 | self.__log(info_str, mode="INFO")
42 |
43 | def error(self, info_str: str):
44 | self.__log(info_str, mode="ERROR")
45 |
46 | def warn(self, info_str: str):
47 | self.__log(info_str, mode="WARNING")
48 |
49 | def logging_step(self, global_step, epoch, step, loss):
50 | if global_step % self.args.logging_steps == 0:
51 | self.info("Global step %d, epoch: %d, batch: %d, loss: %f, speed: %.2f step/s" %
52 | (global_step, epoch + 1, step + 1, loss, self.args.logging_steps /
53 | (time.time() - self.tic_train)))
54 | self.tic_train = time.time()
55 |
56 | def add_performance(self, performance):
57 | self.performance_list.append(performance)
58 |
59 | def save_performance(self):
60 | output_dir = os.path.join(self.args.output_dir, "logging.json")
61 | with open(output_dir, "w", encoding="utf8") as f:
62 | json.dump(self.performance_list, f)
63 |
64 | def logging_result(self, x):
65 | print(x)
66 |
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/utils/predict.py:
--------------------------------------------------------------------------------
1 | import argparse
2 | import json
3 |
4 | from paddlenlp.metrics.squad import squad_evaluate
5 |
6 |
7 | def get_data(filepath):
8 | with open(filepath, encoding="utf-8") as f:
9 | durobust = json.load(f)
10 | data = []
11 | for article in durobust["data"]:
12 | title = article.get("title", "")
13 | for paragraph in article["paragraphs"]:
14 | context = paragraph["context"] # do not strip leading blank spaces GH-2585
15 | for qa in paragraph["qas"]:
16 | answer_starts = [answer["answer_start"] for answer in qa.get("answers", '')]
17 | answers = [answer["text"] for answer in qa.get("answers", '')]
18 | # Features currently used are "context", "question", and "answers".
19 | # Others are extracted here for the ease of future expansions.
20 | data.append({
21 | "title": title,
22 | "context": context,
23 | "question": qa["question"],
24 | "id": qa["id"],
25 | "answers": {
26 | "answer_start": answer_starts,
27 | "text": answers,
28 | },
29 | })
30 | return data
31 |
32 |
33 | if __name__ == "__main__":
34 | parser = argparse.ArgumentParser(description=__doc__)
35 | parser.add_argument("--test_data_dir", type=str, required=True, help="test data dir")
36 | parser.add_argument("--pred_data_dir", type=str, required=True, help="prediction data dir")
37 | args = parser.parse_args()
38 | raw_datasets = get_data(args.test_data_dir)
39 |
40 | with open(args.pred_data_dir, encoding="utf-8") as f:
41 | all_predictions = json.load(f)
42 |
43 | result = squad_evaluate(examples=raw_datasets, preds=all_predictions, is_whitespace_splited=False)
44 |
--------------------------------------------------------------------------------
/tutorials/redundancy_removal/utils/tools.py:
--------------------------------------------------------------------------------
1 | # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 | import re
15 |
16 |
17 | def remove_None(str_list: list) -> list:
18 | return [t for t in str_list if t is not None]
19 |
20 |
21 | def remove_blank_str(str_list: list) -> list:
22 | return [t for t in str_list if t != ""]
23 |
24 |
25 | def process_single_sign(str_list: list) -> list:
26 | x = []
27 | for i in str_list:
28 | if i in ["|", "!", "。", "！", "？", "\\x0d", ";", "；", "?", "!"]:
29 | continue
30 | x.append(i)
31 | return x
32 |
33 |
34 | def strip_str(s: str) -> str:
35 | return s.strip().strip("\n").strip("\\x0d")
36 |
37 |
38 | def print_red(s: str, flag=True):
39 | if flag:
40 | print('\033[31m' + s + '\033[0m')
41 |
42 |
43 | def split_sentence(s: str, remove_illegal_sign=True) -> list:
44 | # s.replace(" ","。")
45 | c_list = re.split('(。|！|？|\\x0d|;|；|\?|!|\||\.{2,}|[\u4E00-\u9FA5]\.{1,} *|\. )', s)
46 | c_list.append("")
47 | c_list = ["".join(i) for i in zip(c_list[0::2], c_list[1::2])]
48 | if remove_illegal_sign:
49 | c_list = remove_None(c_list)
50 | c_list = process_single_sign(c_list)
51 | c_list = remove_blank_str(c_list)
52 | return [strip_str(c) for c in c_list]
53 | else:
54 | return c_list
55 |
56 |
57 | def batchify(dataset, batch_size):
58 | batch = []
59 | for i, data in enumerate(dataset):
60 | if (i % batch_size == 0):
61 | batch.append([data])
62 | else:
63 | batch[int(i / batch_size)].append(data)
64 | return batch
65 |
66 |
67 | def padding_sentence(data, padding_value=0):
68 | max_x_len = 0
69 | max_y_len = 0
70 | for x in data:
71 | if len(x) > max_x_len:
72 | max_x_len = len(x)
73 | for y in x:
74 | if len(y) > max_y_len:
75 | max_y_len = len(y)
76 | for x in data:
77 | x += [[0] * max_y_len] * (max_x_len - len(x))
78 | for y in x:
79 | y += [0] * (max_y_len - len(y))
80 | return data
81 |
82 |
83 | def padding_batch(data, padding_value=0):
84 | for x in data:
85 | max_y_len = 0
86 | for y in x:
87 | if len(y) > max_y_len:
88 | max_y_len = len(y)
89 | for y in x:
90 | y += [0] * (max_y_len - len(y))
91 | return data
92 |
--------------------------------------------------------------------------------
/tutorials/sparse_data_identification/README.md:
--------------------------------------------------------------------------------
1 | # 训练数据覆盖不足识别及有效数据增强
2 |
3 | ## 方法介绍
4 |
5 | 训练数据覆盖不足会导致模型在对应的测试数据上表现不好。数据扩充是提升模型效果首选方法，然而数据标注是一个费时费力的工作，如何标注更少的数据带来更大的效果提升是大多数NLP开发者面临的难题。
6 |
7 |
8 | 
9 | 图1 数据覆盖不足识别及有效数据增强策略流程
10 |
11 |
12 | TrustAI提供图1流程，用尽量少的标注数据提升模型效果。首先，基于可信分析中实例级证据分析方法，从测试数据中识别因训练数据覆盖不足而导致预测效果差的测试样本，称作目标集（即证据不足测试数据）。然后，在未标注数据中，选择可作为目标集预测证据的数据进行标注。最后，将新标注的数据加入到训练数据中重训模型。
13 |
14 | 注：开发者可访问[ AI Studio示例 ](https://aistudio.baidu.com/aistudio/projectdetail/4434403)快速体验本案例。
15 |
16 | ## 实验步骤
17 |
18 | 我们以基于相似度计算任务LCQMC数据集进行的模拟实验为例介绍该方案实现步骤和效果。
19 |
20 | **Step 1**：从LCQMC训练数据中随机抽取5000条作为新训练集，剩余数据作为未标注数据。基于ERNIE-3.0-base-zh在新训练集`train_5000.tsv`微调得到相似度计算基线模型，运行命令如下所示：
21 |
22 | ```shell
23 | # 下载数据
24 | wget --no-check-certificate https://trustai.bj.bcebos.com/application_data/sparse_data.tar && tar xf sparse_data.tar && rm sparse_data.tar
25 | # 训练基线模型
26 | python -u train.py --dataset_dir ./data --train_file train_5000.tsv --dev_file dev.tsv --test_files test.tsv DuQM --num_classes 2 --save_dir ./checkpoint
27 | ```
28 | 训练的基线模型保存在`checkpoint`目录中。
29 |
30 |
31 | **Step 2**：基于基线模型从验证集中选择目标数据，即**目标集**。
32 |
33 | 目标集选择方法：针对验证集中每一条样本，基于TrustAI提供的实例级证据分析方法`FeatureSimilarityModel`，计算每一训练样本作为其正支持证据的分数；然后计算每一验证样本在所有训练数据上得到的平均证据支持分数。分数较低的样本表明其证据不足，将其加入到目标集中。
34 |
35 | ```shell
36 | # 选取目标数据
37 | python -u find_sparse_data.py --dataset_dir ./data --train_file train_5000.tsv --dev_file dev.tsv --num_classes 2 --init_from_ckpt ./checkpoint/model_state.pdparams --sparse_num 50 --sparse_path ./data/sparse_data.tsv
38 | # sparse_num表示选择的目标数据的数量
39 | # sparse_path表示目标集存储的路径
40 | ```
41 |
42 | 如图2所示，模型在目标集上的效果降低非常明显。
43 |
44 | 
45 | 图2 模型在整个测试集和目标集上的表现
46 |
47 |
48 |
49 | **Step 3**：针对目标集中数据，再次利用`FeatureSimilarityModel`方法从未标注数据`rest_train.tsv`中选择可支持它们预测的数据，作为增强数据进行人工标注。
50 |
51 | 注：此处为模拟实验，`rest_train.tsv`的数据已被标注
52 |
53 | ```shell
54 | # 选取有效数据
55 | python -u find_valid_data.py --dataset_dir ./data --unlabeled_file rest_train.tsv --target_file sparse_data.tsv --num_classes 2 --init_from_ckpt ./checkpoint/model_state.pdparams --valid_threshold 0.7 --valid_num 1000 --valid_path ./data/valid_data.tsv
56 | # valid_threshold表示目标集证据的分数阈值，开发者可根据自己数据自主调整，默认为0.7
57 | # valid_num表示抽取有效数据的数量
58 | # valid_path表示有效数据的存储路径
59 | ```
60 |
61 | **Step 4**：完成增强数据的标注后，将其加入到原训练数据集合中，重新训练模型及评估模型效果。
62 |
63 | ```shell
64 | # 将标注过的有效数据加入到原始训练数据中
65 | cat ./data/train_5000.tsv ./data/valid_data.tsv > ./data/merge_valid.tsv
66 | # 基于增强后的数据训练模型
67 | python -u train.py --dataset_dir ./data --train_file merge_valid.tsv --dev_file dev.tsv --test_files test.tsv DuQM sparse_data.tsv --num_classes 2 --save_dir ./valid_checkpoint
68 | ```
69 | 实验结果如下表所示（对比基线：随机选择相同规模的数据进行标注，作为增强数据）：
70 |
71 | | 数据集 | 数据量 | LCQMCdev | LCQMCtest | DuQM | 目标集 |
72 | | :-------: | :-------: | :-----: | :-----: |:-----: |:-----: |
73 | | 基线 | 5000 | 86.31% | 84.49% | 69.17% | 55.20% |
74 | | 基线 + 随机1000条 | 6000 | 86.76% | 85.05% | 69.23% | 55.20% |
75 | | 基线 + 策略1000条 | 6000 | 87.04% | 85.58% | 70.20% | 69.60% |
76 |
77 | 注：以上结果均为10次实验的平均值。
78 |
--------------------------------------------------------------------------------
/tutorials/sparse_data_identification/utils.py:
--------------------------------------------------------------------------------
1 | # Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | import numpy as np
16 |
17 | import paddle
18 | import paddle.nn.functional as F
19 | from paddlenlp.utils.log import logger
20 |
21 |
22 | @paddle.no_grad()
23 | def evaluate(model, criterion, metric, data_loader, name=''):
24 | """
25 | Given a dataset, it evaluates model and computes the metric.
26 | Args:
27 | model(obj:`paddle.nn.Layer`): A model to classify texts.
28 | criterion(obj:`paddle.nn.Layer`): It can compute the loss.
29 | metric(obj:`paddle.metric.Metric`): The evaluation metric.
30 | data_loader(obj:`paddle.io.DataLoader`): The dataset loader which generates batches.
31 | """
32 |
33 | model.eval()
34 | metric.reset()
35 | losses = []
36 | for batch in data_loader:
37 | input_ids, token_type_ids, labels = batch['input_ids'], batch['token_type_ids'], batch['labels']
38 | logits = model(input_ids, token_type_ids)
39 | loss = criterion(logits, labels)
40 | losses.append(loss.numpy())
41 | correct = metric.compute(logits, labels)
42 | metric.update(correct)
43 |
44 | acc = metric.accumulate()
45 | logger.info("%s: eval loss: %.5f, acc: %.5f" % (name, np.mean(losses), acc))
46 | model.train()
47 | metric.reset()
48 |
49 | return acc
50 |
51 |
52 | def preprocess_function(example, tokenizer, max_seq_length, is_test=False):
53 | """
54 | Builds model inputs from a sequence for sequence classification tasks
55 | by concatenating and adding special tokens.
56 |
57 | Args:
58 | example(obj:`list[str]`): input data, containing text and label if it have label.
59 | tokenizer(obj:`PretrainedTokenizer`): This tokenizer inherits from :class:`~paddlenlp.transformers.PretrainedTokenizer`
60 | which contains most of the methods. Users should refer to the superclass for more information regarding methods.
61 | max_seq_length(obj:`int`): The maximum total input sequence length after tokenization.
62 | Sequences longer than this will be truncated, sequences shorter will be padded.
63 | label_nums(obj:`int`): The number of the labels.
64 | Returns:
65 | result(obj:`dict`): The preprocessed data including input_ids, token_type_ids, labels.
66 | """
67 | if 'text_b' not in example:
68 | result = tokenizer(text=example["text_a"], max_seq_len=max_seq_length)
69 | else:
70 | result = tokenizer(text=example["text_a"], text_pair=example['text_b'], max_seq_len=max_seq_length)
71 |
72 | if not is_test:
73 | result["labels"] = np.array([example['label']], dtype='int64')
74 | return result
75 |
--------------------------------------------------------------------------------
