No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow

Abstract: Given a large corpus, the questions one might ask can vary -- from "When was the first human heart transplant?" to "What are all the contradictory claims in this literature?" -- but what makes some questions more challenging than others? In this work, we define a notion of Corpus Task Complexity (CTC) that characterizes tasks by how their difficulty grows with corpus size; for instance, a retrieval query only requires a single linear pass over a corpus, while finding contradictions requires checking a quadratically growing set of claim pairs. Observing that prior work has largely only studied tasks whose difficulty grows linearly with corpus size, which we call low CTC tasks, we introduce 10 new tasks belonging to a class of high CTC whose difficulty grows quadratically or more in corpus size. We find that high-CTC tasks not only grow much more challenging on average at longer contexts for LCLMs, they reverse many modeling conclusions drawn solely from low-CTC evaluations. For instance, efficient block-sparse and hybrid attention approaches consistently match full attention performance on low-CTC tasks, but degrade much more on high-CTC tasks. Large-corpus high-CTC reasoning thus remains an open challenge as full attention is too costly to scale, motivating future research on these tasks. We release our code, data, and 22-task suite (CTC-Bench), to facilitate future research in this area.

• The paper introduces Corpus Task Complexity (CTC), a measure defining how computational requirements scale with corpus size, using oracle calls to characterize task complexity with asymptotic scaling analysis.
• Experimental results show that high-CTC tasks, such as contradiction detection and query-document matching, degrade more rapidly with increasing context length than low-CTC tasks like retrieval, highlighting the importance of task complexity in model performance.
• The study finds that block-sparse attention, which is effective for low-CTC tasks, fails to maintain performance on high-CTC tasks, indicating that task-specific attention mechanisms are crucial for handling complex corpus interactions.

Conceptual contribution

“No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow” (2609.29245) introduces Corpus Task Complexity (CTC), a complexity measure for corpus reasoning tasks based on how the number of required inference operations scales with corpus size. The paper’s central claim is that long-context evaluations have systematically overrepresented tasks whose computational requirements grow linearly with the number of documents, thereby obscuring a qualitatively different class of tasks requiring higher-order interactions among corpus elements.

The distinction is operationalized through oracle calls. Given a corpus of $N$ documents, an oracle can perform a constant-size operation, such as determining whether a document is relevant to a query or whether two claims contradict one another. The CTC of a task is then characterized by the asymptotic number of oracle calls required by the best available algorithm over an unstructured corpus. A conventional retrieval task can be solved with one pass over the documents and is therefore classified as $O_T(N)$. By contrast, identifying all contradictory claim pairs requires considering document pairs and is classified as $O_T(N^2)$. Finding triples satisfying a joint constraint provides an $O_T(N^3)$ example.

This definition concerns the scaling of the task-level computation, not the intrinsic difficulty of each elementary operation. Two tasks may both be $O_T(N)$ while differing substantially in the semantic difficulty of their per-document judgments. Conversely, two $O_T(N^2)$ tasks may differ in the difficulty of the pairwise relation being evaluated. The paper explicitly treats these factors as distinct, and later experiments show that they interact: higher base-operation difficulty amplifies the performance penalty associated with higher CTC.

The CTC framework is intended as a structural property of corpus tasks rather than a complete theory of model difficulty. The authors acknowledge that proving an optimal algorithm is generally difficult, so their empirical classifications use plausible best-known algorithms for unstructured corpora. This leaves open whether some tasks classified as quadratic admit practical subquadratic algorithms under realistic assumptions about document structure, metadata, semantic locality, or indexability.

CTC-BENCH and task taxonomy

The paper evaluates the framework through CTC-BENCH, a 22-task suite comprising 12 low-CTC tasks drawn from common long-context evaluations and 10 newly constructed higher-CTC tasks. The benchmark spans retrieval, aggregation, outlier detection, cross-corpus comparison, clustering, pair discovery, reordering, contradiction detection, and higher-order combinatorial search.

CTC class	Representative tasks	Required operation pattern
$O_T(N)$	NQ, HotpotQA, SciFact, FiQA, MS MARCO, OOLONG	One corpus pass or a constant number of passes
$O_T(NM)$	Wikipedia outlier detection, OpenAlex grouping	Compare documents against a growing set of categories or groups
$O_T(N^2)$	Contradiction, X-Absence, QDmatch, Strmatch, Reorder	Search over document or cross-corpus pairs
$O_T(N^3)$	Textgroups	Search over document triples satisfying a joint constraint

The low-CTC category includes standard retrieval tasks such as NQ, HotpotQA, SciFact, FiQA, MS MARCO, and OBLIQ. The authors also classify multi-hop HotpotQA as $O_T(N)$0 because the task requires only a fixed number of retrieval passes, despite requiring composition across two documents. Similarly, OOLONG and fixed-label outlier detection remain linear because each item can be classified independently before a lightweight aggregation step.

The benchmark’s higher-CTC tasks are designed to isolate the cost of cross-document interaction. Contradiction requires finding all contradictory claim pairs in a PubMed-derived corpus. X-Absence compares two shuffled, nearly identical corpora and identifies unmatched elements, removing the positional alignment that makes ordinary absence detection linear. Query-document matching constructs separate query and document pools and asks the model to identify a sparse set of relevant pairs among $O_T(N)$1 possible combinations. Strmatch is a synthetic control in which pairwise matching depends only on shared contiguous word sequences, allowing the effect of comparison count to be studied independently of semantic reasoning.

Other tasks test growing category structure. Wikipedia outlier detection has a fixed-$O_T(N)$2 condition, which remains $O_T(N)$3, and a scale-$O_T(N)$4 condition in which the number of source topics grows with corpus size, yielding $O_T(N)$5. OpenAlex grouping similarly requires assigning documents to a growing number of latent topic groups. Reorder requires recovering the original sequence of shuffled text segments; absent exploitable positional structure, the proposed oracle formulation requires pairwise comparisons. Textgroups extends the suite beyond quadratic complexity by asking for all document triples whose lexical feature counts sum to a target.

This construction is an important methodological contribution because it distinguishes corpus length from corpus interaction structure. A long retrieval context may contain many documents but still require only independent document-query judgments. A shorter corpus can be computationally more demanding if its answer depends on examining a large set of pairwise or higher-order relations. The benchmark therefore tests whether models can maintain relational coverage as the number of candidate interactions increases, rather than merely whether they can locate a salient item in a long sequence.

Experimental design

The principal experiments fine-tune Qwen3.5-4B separately on the 22 tasks, using context lengths from 2K through 32K tokens. The models are trained in-domain to reduce the confounding effects of task unfamiliarity and distribution shift. Except for task-specific deviations, each model receives 20,000 examples over one epoch, distributed evenly across the five context budgets. Full attention, block-sparse attention, and mask-mixed block-sparse attention are compared under matched training conditions.

The setup is deliberately designed to test architectural scaling rather than zero-shot competence. Nevertheless, the in-domain design imposes an important interpretation constraint: the reported degradation reflects limitations in representing and computing the required corpus interactions, but not necessarily the performance of a model trained with substantially more data, more sophisticated curricula, or task-specific decomposition strategies.

Evaluation uses task-appropriate metrics, including set-F1, pair-F1, gold-document F1, Kendall’s tau, MRR, and partial-credit aggregation scores. The benchmark includes both real and synthetic or semi-synthetic data. This increases experimental control, particularly for exact CTC assignments, but also means that some tasks may not capture the distributional complexity of naturally occurring corpus analysis.

Performance scaling with corpus size

The central empirical result is that high-CTC tasks deteriorate substantially faster than low-CTC tasks as context length increases, even when models are trained at the evaluated context lengths. Under full attention, the average low-CTC score falls from 0.910 at 2K tokens to 0.788 at 32K, a 13% relative decrease. In contrast, the average high-CTC score falls from 0.851 to 0.511, a 40% relative decrease. Ten low-CTC tasks are reported to degrade by less than 0.1, whereas seven of the ten high-CTC tasks degrade by more than 0.1.

These results support the paper’s primary claim that context length alone is an inadequate predictor of long-context difficulty. The relevant variable is the number and structure of corpus-level interactions that must be resolved. A model can preserve retrieval performance while failing to maintain reliable coverage over a quadratic candidate space. The implication is direct: evaluations restricted to linear-scan tasks can substantially overestimate the scalability of long-context models for corpus analysis.

The contrast is also visible in matched task pairs. NIAH-contradiction is constructed as a linear-time control for contradiction search: the query directly specifies one claim, and the model must retrieve its contradictory document. The corresponding contradiction task requires finding all contradictory pairs among the corpus. Similarly, ordinary HotpotQA retrieval is compared with QDmatch constructed from HotpotQA questions and documents. These pairings isolate the effect of all-pairs search from the underlying semantic operation. The high-CTC versions degrade more rapidly, indicating that the deterioration cannot be attributed solely to harder document content.

The paper further finds that CTC interacts with oracle difficulty. Among matched $O_T(N)$6 and $O_T(N)$7 tasks, the relative performance gap grows more quickly when the underlying operation is semantically harder. Thus, CTC is not merely an additive computational burden. As the candidate interaction space expands, errors in the elementary relation judgment become increasingly consequential, and the model must sustain precision over a larger number of potential comparisons.

Block-sparse attention and the failure of the apparent free lunch

The strongest architectural result concerns block-sparse attention. The evaluated block-sparse scheme processes documents independently and allows only designated query and answer tokens to attend globally. This reduces prefill computation relative to full attention by suppressing token-level interactions across documents. Such a design is well matched to retrieval tasks in which documents can be independently scored against a query.

On low-CTC tasks, block-sparse attention is effectively lossless relative to full attention across the evaluated context lengths. This reproduces the favorable conclusions of prior long-context work conducted primarily on retrieval, question answering, and related linear-time settings. The paper reports that mask-mixed training, in which batches probabilistically use full attention before annealing toward the block-sparse mask, improves block-sparse performance, particularly on aggregation and high-CTC tasks. Averaged across the 10-task comparison subset, the reported mask-mixing gains are larger for high-CTC tasks than for low-CTC tasks.

The result reverses on high-CTC tasks. Relative to full attention, block-sparse degradation averages approximately 26.9% at 2K tokens, 33.0% at 4K, 43.7% at 8K, 57.4% at 16K, and 65.9% at 32K. The degradation is therefore not a fixed penalty incurred at all context lengths; it increases as the corpus grows. On tasks such as contradiction, QDmatch, X-Absence, and scale-$O_T(N)$8 outlier detection, suppressing direct cross-document interactions removes precisely the computational pathway required to compare candidate elements.

This finding qualifies the broad claim that block sparsity provides a “free lunch” for corpus reasoning. It provides a favorable computation-performance tradeoff for tasks whose solution decomposes into independent document processing followed by limited global aggregation. It does not preserve performance when the answer depends on dense or extensive relational interaction among documents. The implication is that attention sparsity must be evaluated against the interaction graph induced by the task, not merely against the input format of a corpus.

Mask mixing improves this tradeoff but does not eliminate it. The method appears to distill some full-attention behavior into representations usable under block-sparse inference, yet evaluation still applies the sparse mask exclusively. The remaining high-CTC gap indicates that training-time exposure to global interactions cannot fully compensate for their absence at inference time under the tested model and data regime.

Hybrid architectures

The paper extends the comparison to hybrid architectures that combine full-attention layers with recurrent or state-space-like components. It compares a full-attention version of OLMo-3-7B with OLMo-3-7B-Hybrid, which uses a 3:1 mixture of gated delta-net and full-attention layers. The full-attention baseline is adapted from the model’s default sliding-window configuration through continued pretraining.

The two architectures perform similarly on low-CTC tasks, with the hybrid sometimes performing better at 32K. On high-CTC tasks, however, the hybrid exhibits consistent performance gaps relative to full attention. The reported comparison therefore mirrors the block-sparse result: architectures that preserve sufficient information for independent retrieval and aggregation may fail when the task requires repeated, distributed comparison across the corpus.

The result should be interpreted cautiously because the compared OLMo models are generally weaker than the Qwen3.5 models used elsewhere, and the full-attention baseline requires architectural adaptation and continued pretraining. The paper accordingly treats the result as evidence for a robust trend rather than as a definitive estimate of the optimal attention-to-recurrent layer ratio. The specific question left open is whether a hybrid architecture with selectively placed or content-adaptive full-attention layers can retain high-CTC performance while reducing the cost of indiscriminate all-to-all attention.

Length generalization

The experiments also challenge the assumption that long-context capability learned at shorter lengths transfers uniformly across task types. Models trained through 32K tokens are evaluated at 64K and 128K using full attention. Performance is normalized by the corresponding 32K score.

At 128K, low-CTC tasks retain between 28% and 100% of their 32K performance, whereas high-CTC tasks retain only 4% to 14%; every reported high-CTC point falls below every low-CTC point in the comparison. This is a particularly strong separation because the models are evaluated within their native context capacity, rather than being subjected to an external context-extension method.

The implication is that length generalization is task-structural. Training on shorter contexts may teach a model to perform a local or fixed-number retrieval procedure that transfers to larger corpora. It does not necessarily teach the model how to maintain reliable coverage over an interaction space whose size grows quadratically or faster. Consequently, context-length training schedules should be stratified by CTC rather than evaluated solely by token length.

Robustness across model families and scales

The full-versus-block-sparse pattern appears across Qwen3.5, OLMo-3-7B, and Llama-3.2-3B. On HotpotQA, a representative low-CTC task, block-sparse attention causes little degradation. On contradiction, a representative high-CTC task, the degradation is substantial and increases with context length across model families.

Within the Qwen3.5 family, smaller models degrade faster on larger corpora. Additional full-attention experiments show that some high-CTC tasks have minimum scale requirements: Qwen3.5-0.8B and 2B models can fail to learn tasks such as QDmatch or reordering even before the longest context lengths, while the 4B model achieves substantially higher performance. This suggests that CTC does not only govern asymptotic degradation; it also changes the minimum representational and algorithmic capacity needed to acquire the task.

The scale findings qualify any interpretation of CTC as a model-independent difficulty law. The benchmark identifies a structural source of difficulty, but observed performance depends on model size, pretraining, instruction tuning, data construction, and the quality of the elementary operation. CTC predicts a pressure on computation and coverage; it does not determine the absolute score.

Limitations and open questions

The empirical evidence is concentrated at 2K–32K tokens for the main in-domain experiments and extends to 128K only for length generalization. The paper does not directly test million-token corpora, despite arguing that high-CTC penalties may become more pronounced at that scale. The asymptotic interpretation is therefore supported by finite-range trends rather than by measurements over a sufficiently broad scaling regime.

CTC-BENCH is diverse but not exhaustive. Several high-CTC tasks are synthetic or semi-synthetic, and some use simplifying assumptions that may make the required relation easier than in natural corpora. In particular, contradiction pairs include one LLM-generated claim, and the authors note that subtle generation artifacts may remain. X-Absence includes exact copies, making identity matching easier than semantic cross-corpus alignment. Strmatch deliberately removes semantic complexity, while Textgroups uses controlled lexical features. These designs are useful for isolating computational structure, but performance on them should not be equated with performance on unconstrained scientific, legal, or web corpora.

The CTC classification itself depends on the assumed oracle model and corpus representation. Indexing, metadata, embeddings, hierarchical clustering, approximate nearest-neighbor search, or domain-specific structure could reduce the effective cost of some nominally quadratic tasks. The framework correctly distinguishes online and offline costs, but the experiments do not evaluate retrieval indexes, agentic decomposition, recursive processing, or specialized pair-generation algorithms as alternatives to end-to-end LCLM inference.

The architectural comparisons also have scope limitations. Block-sparse attention is implemented through a particular document-level mask, and hybrid results depend on specific OLMo configurations. The models are fine-tuned independently per task, with one training epoch and a fixed data budget. It remains unresolved whether more extensive training, intermediate supervision over pairwise relations, iterative refinement, or learned sparse interaction patterns can close the high-CTC gap without restoring full attention.

Finally, the benchmark evaluates exact or near-exact output structures, such as exhaustive pair lists and complete group assignments. This is appropriate for measuring coverage over candidate interactions, but real applications may permit ranking, calibrated abstention, approximate discovery, or human-in-the-loop verification. Whether high-CTC tasks remain as difficult under these alternative utility functions is an open empirical question.

Conclusion

The paper establishes CTC as a useful organizing principle for long-context corpus reasoning. Its main empirical result is that tasks requiring quadratic or higher-order corpus interactions degrade substantially faster with context length than conventional linear-scan tasks, even under in-domain training and full attention. Block-sparse and hybrid architectures that appear competitive on retrieval and aggregation can therefore incur severe losses on high-CTC tasks, while short-context length generalization is markedly less reliable for the same class.

CTC-BENCH provides a concrete basis for evaluating these distinctions. The paper’s principal methodological conclusion is that long-context evaluations should report not only context length and task accuracy, but also the scaling structure of the required corpus interactions. The specific unresolved problem is how to construct inference procedures that achieve reliable high-CTC reasoning at corpus scales where unrestricted full attention is computationally infeasible.