Assess LLM2Vec on Benchmarks Free of Pre-Training Contamination

Investigate the performance of LLM2Vec-transformed decoder-only large language models on newly designed evaluation benchmarks that are guaranteed not to overlap with the models’ pre-training corpora, in order to quantify and mitigate potential test set contamination effects.

Background

The authors note the possibility that publicly available evaluation datasets could overlap with the pre-training data of models like LLaMA-2-7B and Mistral-7B, which could confound evaluation results.

Because complete details of the pre-training data are not public, the extent of potential contamination is uncertain. The paper explicitly calls for evaluation on newly designed benchmarks excluded from pre-training corpora.