Benchmarking insight internalization against insight retrieval

Benchmark the performance of training insights into model parameters with the supervised insight-internalization loss against databases and retrieval systems that insert relevant insights into prompts at inference time.

Background

The paper contrasts parameter-based internalization of insights with external databases and retrieval systems that search for similar prior insights and add them to the prompt. The authors argue that internalization may be simpler because it can generalize insights automatically and apply them when needed.

However, the relative performance of these approaches is not established, leaving an empirical comparison as an explicit unresolved task.