├── .gitignore ├── README.md ├── architecture.png ├── check ├── bad_case.json ├── bad_case_extracted.py ├── bad_question_ids.txt ├── check_baidu_preprocess_ceil_metric.py ├── check_ceil_rouge_badcase.py ├── check_mrc_ceil_metric.py ├── metric │ ├── MRC Metrics.docx │ ├── README.md │ ├── bleu.py │ ├── common.py │ ├── demo_data │ │ ├── pred.json │ │ └── ref.json │ ├── mrc_eval.py │ └── rouge.py └── run_check_baidu.sh ├── data_augment ├── 1.create_es_index.py ├── 2.insert_train_documents.py ├── 3.augment_extracted_trainset.py ├── README.md ├── run_augment_extracted.sh ├── run_gen_augment_mrc_dataset.sh └── run_remove_low_ceil_metric.sh ├── ensemble ├── README.md ├── __init__.py ├── answer_text_norm.py ├── ensemble_dataset.py ├── final_postprocess_test1.py ├── final_postprocess_test2.py └── run_ensemble.py ├── experiment.png ├── mrc_model.png ├── postprocess ├── postprocess_test1.py └── postprocess_test2.py ├── preprocess ├── 0.fetch_and_map_urls.py ├── 0.fetch_and_map_urls_test2.py ├── 1.text_cleaning.py ├── 2.remove_not_related_paras.py ├── 3.extract_paragraph.py ├── 4.gen_mrc_dataset.py ├── 5.remove_low_ceil_metric.py ├── 6.generate_para_match_score_feature.py ├── README.md ├── distance_util.py ├── run_1.text_cleaning.sh ├── run_2.remove_not_related_paras.sh ├── run_3.extract_paragraph.sh ├── run_4.gen_mrc_dataset.sh ├── run_5.remove_low_ceil_metric.sh ├── run_6.generate_para_match_score_feature.sh └── 数据处理-操作记录.xlsx ├── tfmrc ├── README.md ├── layers │ ├── __init__.py │ ├── basic_rnn.py │ ├── loss_func.py │ ├── match_layer.py │ ├── optimizer.py │ ├── pointer_net.py │ └── rnet_modules │ │ ├── __init__.py │ │ ├── encoder.py │ │ ├── layers.py │ │ └── recurrent.py ├── model │ ├── __init__.py │ ├── baidu_baseline_model.py │ └── multi_task_mrc.py ├── run.py └── util │ ├── __init__.py │ ├── dataset │ ├── __init__.py │ ├── optimized_dataset.py │ └── optimized_vocab.py │ ├── fine_classify.py │ └── metric │ ├── __init__.py │ ├── bleu_metric │ ├── LICENSE │ ├── __init__.py │ ├── bleu.py │ └── bleu_scorer.py │ ├── dureader_eval.py │ └── rouge_metric │ ├── __init__.py │ └── rouge.py └── utils ├── __init__.py ├── config_util.py ├── jieba_util.py └── metric_util.py /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/.gitignore -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/README.md -------------------------------------------------------------------------------- /architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/architecture.png -------------------------------------------------------------------------------- /check/bad_case.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/bad_case.json -------------------------------------------------------------------------------- /check/bad_case_extracted.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/bad_case_extracted.py -------------------------------------------------------------------------------- /check/bad_question_ids.txt: -------------------------------------------------------------------------------- 1 | 有问题的question_id： 2 | 982 3 | 918 4 | 6384 5 | 167987 6 | 144655 7 | 360893 8 | -------------------------------------------------------------------------------- /check/check_baidu_preprocess_ceil_metric.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/check_baidu_preprocess_ceil_metric.py -------------------------------------------------------------------------------- /check/check_ceil_rouge_badcase.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/check_ceil_rouge_badcase.py -------------------------------------------------------------------------------- /check/check_mrc_ceil_metric.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/check_mrc_ceil_metric.py -------------------------------------------------------------------------------- /check/metric/MRC Metrics.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/metric/MRC Metrics.docx -------------------------------------------------------------------------------- /check/metric/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/metric/README.md -------------------------------------------------------------------------------- /check/metric/bleu.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/metric/bleu.py -------------------------------------------------------------------------------- /check/metric/common.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/metric/common.py -------------------------------------------------------------------------------- /check/metric/demo_data/pred.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/metric/demo_data/pred.json -------------------------------------------------------------------------------- /check/metric/demo_data/ref.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/metric/demo_data/ref.json -------------------------------------------------------------------------------- /check/metric/mrc_eval.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/metric/mrc_eval.py -------------------------------------------------------------------------------- /check/metric/rouge.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/metric/rouge.py -------------------------------------------------------------------------------- /check/run_check_baidu.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/check/run_check_baidu.sh -------------------------------------------------------------------------------- /data_augment/1.create_es_index.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/data_augment/1.create_es_index.py -------------------------------------------------------------------------------- /data_augment/2.insert_train_documents.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/data_augment/2.insert_train_documents.py -------------------------------------------------------------------------------- /data_augment/3.augment_extracted_trainset.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/data_augment/3.augment_extracted_trainset.py -------------------------------------------------------------------------------- /data_augment/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/data_augment/README.md -------------------------------------------------------------------------------- /data_augment/run_augment_extracted.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/data_augment/run_augment_extracted.sh -------------------------------------------------------------------------------- /data_augment/run_gen_augment_mrc_dataset.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/data_augment/run_gen_augment_mrc_dataset.sh -------------------------------------------------------------------------------- /data_augment/run_remove_low_ceil_metric.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/data_augment/run_remove_low_ceil_metric.sh -------------------------------------------------------------------------------- /ensemble/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/ensemble/README.md -------------------------------------------------------------------------------- /ensemble/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/ensemble/__init__.py -------------------------------------------------------------------------------- /ensemble/answer_text_norm.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/ensemble/answer_text_norm.py -------------------------------------------------------------------------------- /ensemble/ensemble_dataset.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/ensemble/ensemble_dataset.py -------------------------------------------------------------------------------- /ensemble/final_postprocess_test1.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/ensemble/final_postprocess_test1.py -------------------------------------------------------------------------------- /ensemble/final_postprocess_test2.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/ensemble/final_postprocess_test2.py -------------------------------------------------------------------------------- /ensemble/run_ensemble.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/ensemble/run_ensemble.py -------------------------------------------------------------------------------- /experiment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/experiment.png -------------------------------------------------------------------------------- /mrc_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/mrc_model.png -------------------------------------------------------------------------------- /postprocess/postprocess_test1.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/postprocess/postprocess_test1.py -------------------------------------------------------------------------------- /postprocess/postprocess_test2.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/postprocess/postprocess_test2.py -------------------------------------------------------------------------------- /preprocess/0.fetch_and_map_urls.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/0.fetch_and_map_urls.py -------------------------------------------------------------------------------- /preprocess/0.fetch_and_map_urls_test2.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/0.fetch_and_map_urls_test2.py -------------------------------------------------------------------------------- /preprocess/1.text_cleaning.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/1.text_cleaning.py -------------------------------------------------------------------------------- /preprocess/2.remove_not_related_paras.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/2.remove_not_related_paras.py -------------------------------------------------------------------------------- /preprocess/3.extract_paragraph.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/3.extract_paragraph.py -------------------------------------------------------------------------------- /preprocess/4.gen_mrc_dataset.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/4.gen_mrc_dataset.py -------------------------------------------------------------------------------- /preprocess/5.remove_low_ceil_metric.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/5.remove_low_ceil_metric.py -------------------------------------------------------------------------------- /preprocess/6.generate_para_match_score_feature.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/6.generate_para_match_score_feature.py -------------------------------------------------------------------------------- /preprocess/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/README.md -------------------------------------------------------------------------------- /preprocess/distance_util.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/distance_util.py -------------------------------------------------------------------------------- /preprocess/run_1.text_cleaning.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/run_1.text_cleaning.sh -------------------------------------------------------------------------------- /preprocess/run_2.remove_not_related_paras.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/run_2.remove_not_related_paras.sh -------------------------------------------------------------------------------- /preprocess/run_3.extract_paragraph.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/run_3.extract_paragraph.sh -------------------------------------------------------------------------------- /preprocess/run_4.gen_mrc_dataset.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/run_4.gen_mrc_dataset.sh -------------------------------------------------------------------------------- /preprocess/run_5.remove_low_ceil_metric.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/run_5.remove_low_ceil_metric.sh -------------------------------------------------------------------------------- /preprocess/run_6.generate_para_match_score_feature.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/run_6.generate_para_match_score_feature.sh -------------------------------------------------------------------------------- /preprocess/数据处理-操作记录.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/preprocess/数据处理-操作记录.xlsx -------------------------------------------------------------------------------- /tfmrc/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/README.md -------------------------------------------------------------------------------- /tfmrc/layers/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/layers/__init__.py -------------------------------------------------------------------------------- /tfmrc/layers/basic_rnn.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/layers/basic_rnn.py -------------------------------------------------------------------------------- /tfmrc/layers/loss_func.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/layers/loss_func.py -------------------------------------------------------------------------------- /tfmrc/layers/match_layer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/layers/match_layer.py -------------------------------------------------------------------------------- /tfmrc/layers/optimizer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/layers/optimizer.py -------------------------------------------------------------------------------- /tfmrc/layers/pointer_net.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/layers/pointer_net.py -------------------------------------------------------------------------------- /tfmrc/layers/rnet_modules/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/layers/rnet_modules/__init__.py -------------------------------------------------------------------------------- /tfmrc/layers/rnet_modules/encoder.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/layers/rnet_modules/encoder.py -------------------------------------------------------------------------------- /tfmrc/layers/rnet_modules/layers.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/layers/rnet_modules/layers.py -------------------------------------------------------------------------------- /tfmrc/layers/rnet_modules/recurrent.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/layers/rnet_modules/recurrent.py -------------------------------------------------------------------------------- /tfmrc/model/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/model/__init__.py -------------------------------------------------------------------------------- /tfmrc/model/baidu_baseline_model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/model/baidu_baseline_model.py -------------------------------------------------------------------------------- /tfmrc/model/multi_task_mrc.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/model/multi_task_mrc.py -------------------------------------------------------------------------------- /tfmrc/run.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/run.py -------------------------------------------------------------------------------- /tfmrc/util/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/util/__init__.py -------------------------------------------------------------------------------- /tfmrc/util/dataset/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/util/dataset/__init__.py -------------------------------------------------------------------------------- /tfmrc/util/dataset/optimized_dataset.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/util/dataset/optimized_dataset.py -------------------------------------------------------------------------------- /tfmrc/util/dataset/optimized_vocab.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/util/dataset/optimized_vocab.py -------------------------------------------------------------------------------- /tfmrc/util/fine_classify.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/util/fine_classify.py -------------------------------------------------------------------------------- /tfmrc/util/metric/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/util/metric/__init__.py -------------------------------------------------------------------------------- /tfmrc/util/metric/bleu_metric/LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/util/metric/bleu_metric/LICENSE -------------------------------------------------------------------------------- /tfmrc/util/metric/bleu_metric/__init__.py: -------------------------------------------------------------------------------- 1 | __author__ = 'tylin' 2 | -------------------------------------------------------------------------------- /tfmrc/util/metric/bleu_metric/bleu.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/util/metric/bleu_metric/bleu.py -------------------------------------------------------------------------------- /tfmrc/util/metric/bleu_metric/bleu_scorer.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/util/metric/bleu_metric/bleu_scorer.py -------------------------------------------------------------------------------- /tfmrc/util/metric/dureader_eval.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/util/metric/dureader_eval.py -------------------------------------------------------------------------------- /tfmrc/util/metric/rouge_metric/__init__.py: -------------------------------------------------------------------------------- 1 | __author__ = 'vrama91' 2 | -------------------------------------------------------------------------------- /tfmrc/util/metric/rouge_metric/rouge.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/tfmrc/util/metric/rouge_metric/rouge.py -------------------------------------------------------------------------------- /utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/utils/__init__.py -------------------------------------------------------------------------------- /utils/config_util.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/utils/config_util.py -------------------------------------------------------------------------------- /utils/jieba_util.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/utils/jieba_util.py -------------------------------------------------------------------------------- /utils/metric_util.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SunnyMarkLiu/lic2019-dureader2.0-rank2/HEAD/utils/metric_util.py --------------------------------------------------------------------------------