SCROLLS

This repository contains the official code of the paper: "SCROLLS: Standardized CompaRison Over Long Language Sequences".

Links

• Official Website
• Baselines
• Evaluator

Citation

@misc{shaham2022scrolls,
      title={SCROLLS: Standardized CompaRison Over Long Language Sequences}, 
      author={Uri Shaham and Elad Segal and Maor Ivgi and Avia Efrat and Ori Yoran and Adi Haviv and Ankit Gupta and Wenhan Xiong and Mor Geva and Jonathan Berant and Omer Levy},
      year={2022},
      eprint={2201.03533},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Loading the SCROLLS Benchmark Datasets

• via 🤗 Datasets (huggingface/datasets) library (recommended):

  1. Installation

  2. Usage:

     from datasets import load_dataset
     
     qasper_dataset = load_dataset("tau/scrolls", "qasper")
     """
     Options are: ["gov_report", "summ_screen_fd", "qmsum", "narrative_qa", "qasper", "quality", "contract_nli"]
     """

• via ZIP files, where each split is in a JSONL file:

  • GovReport
  • SummScreenFD
  • QMSum
  • NarrativeQA
  • Qasper
  • QuALITY
  • ContractNLI