Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction

Abstract: LLMs are increasingly deployed in settings that require simultaneous adherence to multiple explicit constraints - reasoning structure, safety boundaries, output schemas. Individual constraints are handled proficiently, but the compositional regime, where many must hold jointly, remains poorly characterized: how rapidly does performance degrade, what governs the degradation, and can the collapse be mitigated? We introduce Constraint Saturation Evaluation (CSE), a procedurally generated benchmark that systematically varies the number of simultaneous constraints (k), with every constraint scored by a deterministic, rule-based verifier and zero LLM-judge involvement: 15 models, 36 constraint types, 369,753 checks at k=1-12. Three findings emerge. First, per-constraint pass rate decays gradually and predictably, while the chance of satisfying all k constraints collapses - a model passing individual constraints at ~41% at k=8 succeeds on all eight just 5.7% of the time. Second, constraints do not degrade equally: structural constraints lose 2x more baseline capability per added constraint than lexical ones, ordered by a comprehension-maintenance gap that separates constraints requiring sustained tracking from binary decisions immune to composition. Third, failures are nearly independent, which is what makes the accumulation multiplicative; the residual coupling that does exist tracks shared output features rather than pairwise interference - a wrong sentence count fails every constraint that reads it. Reliable instruction following breaks down beyond 5-6 simultaneous constraints: probe-level success falls below 50% at 7 constraints for the strongest model, and at 3 or fewer for 12 of 15.

• The paper introduces Constraint Saturation Evaluation (CSE), testing 15 language models across up to 12 simultaneous constraints with 36 deterministic verifiers and showing strict compliance falls from 70.7% at one constraint to near zero by 10 or more.
• The paper finds that failures accumulate multiplicatively rather than through strong pairwise interference: models may satisfy about 41% of individual constraints at eight constraints but only 5.7% of probes completely.
• The paper shows that structural and sustained-tracking constraints degrade about twice as quickly as lexical constraints, while planning, self-correction, and retries provide limited headroom compared with improving per-constraint reliability.

Motivation and scope

LLMs are routinely asked to satisfy many explicit constraints in a single response—output schemas, length limits, lexical bans, structural requirements. Existing benchmarks either cap the number of simultaneous constraints (IFEval at $k \leq 3$), use fixed compositions (COLLIE), or rely on LLM judges whose own compositional failures contaminate measurement precisely where models fail most. The paper introduces Constraint Saturation Evaluation (CSE), a procedurally generated benchmark that varies constraint count $k$ from 1 to 12 across 36 deterministically verifiable constraint types, evaluated on 15 models from 8 families over 4,527 probes and 369,753 constraint checks, with zero LLM-judge involvement. The central questions are the functional form of degradation under composition, whether constraints degrade uniformly, whether collapse arises from pairwise interference or simple accumulation, and whether inference-time interventions can mitigate it.

Benchmark design

CSE's design choices target measurement validity. All 36 verifiers return binary pass/fail plus a continuous partial-compliance score; probes are generated procedurally with a compatibility checker enforcing satisfiability via hard pair blocks, parameter-dependent arithmetic checks, and cross-constraint propagation. Rejection rates rise from ~32% at $k{=}4$ to ~98% at $k{=}12$, so high-$k$ probes draw from a narrowed compatible subset—a composition bias the authors control for explicitly. Constraint parameters are held constant across $k$ (Kruskal-Wallis $H{=}6.23$, $p{=}0.96$), presentation order is randomized (position explains <0.1% of variance), and IRT calibration preserves model rankings exactly (Spearman $\rho = 1.000$). Two metrics are reported: marginal CSR (per-constraint rate) and strict CSR (all-$k$-pass probe rate), whose divergence defines the phase transition. A compositional half-life $k$0 marks where sCSR falls below 50%.

The benchmark also includes 444 deliberately impossible probes with proofs of unsatisfiability, testing prioritization rather than capacity.

The phase transition is multiplicative accumulation

All 15 models show the same qualitative pattern: aggregate sCSR starts at 70.7% at $k$1 and approaches zero by $k$2. Per-constraint decay is gentle and predictable—$k$3 with held-out MAE of 0.2pp on $k$4—but probe-level success collapses because it approximates the product of $k$5 marginal rates: models still pass individual constraints ~41% of the time at $k$6 yet satisfy all eight simultaneously only 5.7% of the time. Probe-level success drops below 50% at 7 constraints for the strongest model (GPT-5.5) and at 3 or fewer for 12 of 15 models. This is the paper's strongest quantitative claim: reliable instruction following breaks down beyond roughly 5–6 simultaneous constraints, and the breakdown requires no interaction structure—it is pure multiplicative accumulation, mechanistically simpler than $k$7-SAT-style thresholds.

The co-failure analysis explains why the decay is predictable. Pairwise $k$8 coefficients across 601 constraint pairs yield a mean of +0.067, with 54% of pairs within $k$9 and only one negative pair. Elevated correlations track shared output features rather than cognitive interference: grouping by output feature yields within-cluster $k{=}4$0 vs. across-cluster 0.058 ($k{=}4$1), whereas grouping by linguistic dimension does not reach significance ($k{=}4$2). A wrong sentence count fails every sentence-dependent constraint simultaneously. The practical consequence is stark: since there is essentially no synergistic pairing to exploit, no selection or arrangement of constraints can mitigate the collapse—the only lever is per-constraint reliability.

Difficulty confounds are ruled out through three controls: intrinsic difficulty shows no variation across $k{=}4$3 strata, all difficulty terciles converge to the same floor by $k{=}4$4 even after excluding compositionally immune constraints, and ordering effects are negligible. An unexpected positive synergy at low $k{=}4$5 (+26pp above independence at $k{=}4$6) decays concavely to ~1pp by $k{=}4$7 and is shown to be an arithmetic artifact of higher marginals producing more pessimistic product baselines, not a behavioral "compliance mode."

A hierarchy of constraint fragility

Constraints do not degrade equally. Structural and ordering constraints lose $k{=}4$8 more baseline capability per added constraint than lexical ones (95% CI [1.9, 2.3]). The deeper predictor is the comprehension-maintenance gap $k{=}4$9—the difference between continuous compliance score and strict pass rate—which correlates with degradation slope at $k{=}12$0 ($k{=}12$1), stronger than baseline difficulty. Constraints requiring sustained tracking during generation (word counting, letter avoidance, word-length ranges, gaps of ~50pp) degrade fastest; binary decisions requiring no maintenance (mandatory words, JSON structure, gaps near zero) are immune. Three constraints retain >80% performance at $k{=}12$2: lipogram, forbidden word, and no-repeated-bigrams—with the lipogram's immunity traced to violations concentrated in the first 20% of responses, a vocabulary collision at onset rather than eroding capacity.

The impossible-probe experiments add a nuanced result: sacrifice behavior follows two regimes. Under binary contradiction, models preserve whichever constraint required greater planning investment—inverting the degradation hierarchy (relational R2 survives at 33% while lexically immune L1 is sacrificed at 7%, though this rests on only $k{=}12$3 probes per case). Under multi-constraint triage, the degradation hierarchy reasserts itself, with sustained-tracking constraints dropped first.

Interventions have limited reach

Three ablations test mitigation. Pre-generation planning does nothing: decay factors are unchanged and $k{=}12$4 is inconsistent in sign, despite 100% scaffold compliance for two of three models—notably, the token-matched control alone moved DeepSeek's $k{=}12$5 from 3 to 5, implicating output budget rather than planning structure. Post-hoc self-correction and best-of-5 retries each buy one to two constraints of headroom. Best-of-5 delivers only about two-fifths of the lift independent resampling predicts, even though draws are near-fully distinct: whether a given probe fails is largely deterministic across samples, even as which constraints fail within a response remains near-independent. Both senses of independence hold simultaneously, and neither removes the collapse. Only raising per-constraint reliability shifts the threshold materially.

Limitations and open questions

The paper concedes several boundaries plainly. CSE evaluates purely conjunctive composition; sequential chaining, conditional branching, and nested dependencies may exhibit different decay profiles. Deterministic verifiability systematically excludes semantic, pragmatic, and discourse-level constraints—the tradeoff that buys measurement invariance. The $k{=}12$6 regime is a stress test, though the transition at $k{=}12$7 falls within realistic deployment ranges. Verifier audits identified residual issues: an LLM-judge spot-check found four cross-cutting parsing problems, three of which were fixed and rescored, while R2 list-parsing gaps (~27% of sampled R2 failures potentially false negatives) remain known limitations. Specification-gap exploitation exists but is quantitatively bounded (empty code blocks at 82.4% of F3 passes being the largest), and robustness analyses confirm no single constraint drives any headline result. Interventions were run on only three models, and GPT-5.5's temperature rejection excludes it from retry experiments. Open questions left specific: whether constrained decoding or chain-of-thought scaffolding delays the transition's onset or merely raises the floor; whether the 5–6 constraint ceiling reflects a substrate-independent information-theoretic bound or an architectural coincidence; and how constraint load degrades primary-task accuracy.

Conclusion

CSE establishes that compositional instruction following degrades through predictable multiplicative accumulation of near-independent constraint failures, with reliable performance ending beyond 5–6 simultaneous constraints across all tested models. A comprehension-maintenance gap governs which constraints fail first, structural constraints degrading twice as fast as lexical ones. Because failures are weakly coupled through shared output features rather than pairwise interference, prompt restructuring cannot help, retries and self-correction buy only marginal headroom, and per-constraint reliability remains the operative lever. For deployment, systems assuming linear degradation will encounter sharp failures between 3 and 5 simultaneous constraints, and continuous compliance metrics overestimate true satisfaction by up to 50pp near the transition boundary.