INSTRUCTIR: A Benchmark for Instruction Following of Information Retrieval Models (2402.14334v1)
Abstract: Despite the critical need to align search targets with users' intention, retrievers often only prioritize query information without delving into the users' intended search context. Enhancing the capability of retrievers to understand intentions and preferences of users, akin to LLM instructions, has the potential to yield more aligned search targets. Prior studies restrict the application of instructions in information retrieval to a task description format, neglecting the broader context of diverse and evolving search scenarios. Furthermore, the prevailing benchmarks utilized for evaluation lack explicit tailoring to assess instruction-following ability, thereby hindering progress in this field. In response to these limitations, we propose a novel benchmark,INSTRUCTIR, specifically designed to evaluate instruction-following ability in information retrieval tasks. Our approach focuses on user-aligned instructions tailored to each query instance, reflecting the diverse characteristics inherent in real-world search scenarios. Through experimental analysis, we observe that retrievers fine-tuned to follow task-style instructions, such as INSTRUCTOR, can underperform compared to their non-instruction-tuned counterparts. This underscores potential overfitting issues inherent in constructing retrievers trained on existing instruction-aware retrieval datasets.
- Task-aware retrieval with instructions. In Findings of the ACL.
- Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement.
- Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv.
- Lmentry: A language model benchmark of elementary language tasks. arXiv.
- Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res.
- Followbench: A multi-level fine-grained constraints following benchmark for large language models. arXiv.
- J Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics.
- Holistic evaluation of language models. arXiv.
- Query rewriting for retrieval-augmented large language models. arXiv.
- Fine-tuning llama for multi-stage text retrieval. arXiv.
- Mteb: Massive text embedding benchmark. In EACL.
- Ms marco: A human-generated machine reading comprehension dataset.
- Large dual encoders are generalizable retrievers. In EMNLP.
- Ktrl+ f: Knowledge-augmented in-document search. arXiv.
- OpenAI. 2023. Gpt-4 technical report.
- Training language models to follow instructions with human feedback. ArXiv.
- Justus J. Randolph. 2005. Free-marginal multirater kappa (multirater k[free]): An alternative to fleiss’ fixed-marginal multirater kappa.
- The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.
- Colbertv2: Effective and efficient retrieval via lightweight late interaction. In NAACL.
- One embedder, any task: Instruction-finetuned text embeddings. ArXiv.
- Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. ArXiv.
- Improving text embeddings with large language models. arXiv.
- Self-instruct: Aligning language model with self generated instructions. arXiv.
- Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv.
- Uniir: Training and benchmarking universal multimodal information retrievers. ArXiv.
- Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv.
- Instruction tuning for large language models: A survey. ArXiv.
- Instruction tuning for large language models: A survey. arXiv.
- Romqa: A benchmark for robust, multi-evidence, multi-answer question answering. arXiv.
- Instruction-following evaluation for large language models. arXiv.
- Hanseok Oh (8 papers)
- Hyunji Lee (19 papers)
- Seonghyeon Ye (25 papers)
- Haebin Shin (6 papers)
- Hansol Jang (5 papers)
- Changwook Jun (4 papers)
- Minjoon Seo (82 papers)