├── .gitignore ├── LICENSE ├── README.md ├── baselines ├── README.md ├── configs │ ├── datasets │ │ ├── contract_nli.json │ │ ├── gov_report.json │ │ ├── narrative_qa.json │ │ ├── qasper.json │ │ ├── qmsum.json │ │ ├── quality.json │ │ └── summ_screen_fd.json │ ├── no_metrics.json │ └── train.json ├── requirements.txt ├── scripts │ ├── __init__.py │ ├── commands │ │ ├── __init__.py │ │ ├── consts.py │ │ ├── finetune.py │ │ └── generate.py │ ├── execute.py │ └── utils.py └── src │ ├── __init__.py │ ├── metrics │ ├── __init__.py │ └── metrics.py │ ├── run.py │ └── utils │ ├── __init__.py │ ├── config.py │ ├── duplicates.py │ ├── namegenerator.py │ ├── output_dir.py │ └── reproducibility.py ├── evaluator ├── EVALUATE_BENCHMARK.md ├── EVALUATE_DATASET.md ├── PREPARE_SUBMISSION_FILE.md ├── README.md ├── VERIFY_SUBMISSION_FILE.md ├── benchmark_evaluator.py ├── dataset_evaluator.py ├── prepare_submission.py └── requirements.txt ├── pyproject.toml └── scrolls_datasets.bib /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/.gitignore -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/README.md -------------------------------------------------------------------------------- /baselines/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/README.md -------------------------------------------------------------------------------- /baselines/configs/datasets/contract_nli.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/configs/datasets/contract_nli.json -------------------------------------------------------------------------------- /baselines/configs/datasets/gov_report.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/configs/datasets/gov_report.json -------------------------------------------------------------------------------- /baselines/configs/datasets/narrative_qa.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/configs/datasets/narrative_qa.json -------------------------------------------------------------------------------- /baselines/configs/datasets/qasper.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/configs/datasets/qasper.json -------------------------------------------------------------------------------- /baselines/configs/datasets/qmsum.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/configs/datasets/qmsum.json -------------------------------------------------------------------------------- /baselines/configs/datasets/quality.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/configs/datasets/quality.json -------------------------------------------------------------------------------- /baselines/configs/datasets/summ_screen_fd.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/configs/datasets/summ_screen_fd.json -------------------------------------------------------------------------------- /baselines/configs/no_metrics.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/configs/no_metrics.json -------------------------------------------------------------------------------- /baselines/configs/train.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/configs/train.json -------------------------------------------------------------------------------- /baselines/requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/requirements.txt -------------------------------------------------------------------------------- /baselines/scripts/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /baselines/scripts/commands/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /baselines/scripts/commands/consts.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/scripts/commands/consts.py -------------------------------------------------------------------------------- /baselines/scripts/commands/finetune.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/scripts/commands/finetune.py -------------------------------------------------------------------------------- /baselines/scripts/commands/generate.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/scripts/commands/generate.py -------------------------------------------------------------------------------- /baselines/scripts/execute.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/scripts/execute.py -------------------------------------------------------------------------------- /baselines/scripts/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/scripts/utils.py -------------------------------------------------------------------------------- /baselines/src/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /baselines/src/metrics/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/src/metrics/__init__.py -------------------------------------------------------------------------------- /baselines/src/metrics/metrics.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/src/metrics/metrics.py -------------------------------------------------------------------------------- /baselines/src/run.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/src/run.py -------------------------------------------------------------------------------- /baselines/src/utils/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /baselines/src/utils/config.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/src/utils/config.py -------------------------------------------------------------------------------- /baselines/src/utils/duplicates.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/src/utils/duplicates.py -------------------------------------------------------------------------------- /baselines/src/utils/namegenerator.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/src/utils/namegenerator.py -------------------------------------------------------------------------------- /baselines/src/utils/output_dir.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/src/utils/output_dir.py -------------------------------------------------------------------------------- /baselines/src/utils/reproducibility.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/baselines/src/utils/reproducibility.py -------------------------------------------------------------------------------- /evaluator/EVALUATE_BENCHMARK.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/evaluator/EVALUATE_BENCHMARK.md -------------------------------------------------------------------------------- /evaluator/EVALUATE_DATASET.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/evaluator/EVALUATE_DATASET.md -------------------------------------------------------------------------------- /evaluator/PREPARE_SUBMISSION_FILE.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/evaluator/PREPARE_SUBMISSION_FILE.md -------------------------------------------------------------------------------- /evaluator/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/evaluator/README.md -------------------------------------------------------------------------------- /evaluator/VERIFY_SUBMISSION_FILE.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/evaluator/VERIFY_SUBMISSION_FILE.md -------------------------------------------------------------------------------- /evaluator/benchmark_evaluator.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/evaluator/benchmark_evaluator.py -------------------------------------------------------------------------------- /evaluator/dataset_evaluator.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/evaluator/dataset_evaluator.py -------------------------------------------------------------------------------- /evaluator/prepare_submission.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/evaluator/prepare_submission.py -------------------------------------------------------------------------------- /evaluator/requirements.txt: -------------------------------------------------------------------------------- 1 | datasets==1.17.0 2 | pandas 3 | 4 | # METRICS: 5 | # ROUGE 6 | nltk 7 | absl-py 8 | rouge_score 9 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [tool.black] 2 | line-length = 119 3 | target-version = ['py35'] 4 | -------------------------------------------------------------------------------- /scrolls_datasets.bib: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tau-nlp/scrolls/HEAD/scrolls_datasets.bib --------------------------------------------------------------------------------