Certification from Examples is Hard for Circuits and Transformers under Minimal Overparametrization

Abstract: As state-of-the-art neural networks are deployed on reasoning and algorithmic tasks, exactness guarantees become increasingly important. However, high average-case accuracy can still mask inconsistent behaviors. This motivates exact certification, which asks for the smallest set of labeled examples needed to certify that a learned hypothesis equals the target. We show that while some hypotheses are easy to certify, even minimal overparametrization can make certification exponentially hard across several hypothesis classes. For threshold circuits of depth $\ge 2$, adding a single extra gate can force certificate sizes exponential in the input dimension. We show an analogous hardness result for log-precision Transformers with only constant architectural overhead. We also characterize approximate certification, showing that allowing only polynomially many mistakes still requires exponentially large certificates, whereas constant relative-error guarantees can hide exponentially many mistakes. Empirically, we study certification for constructed circuits and trained Transformers for recognizing binary addition. While the constructed circuits instantiate the exponential barrier for certification, the trained Transformer analysis shows that imperfect models can evade detection by large uniformly sampled certificate candidates.

• The paper demonstrates that minimal overparametrization can exponentially increase the certificate complexity in both circuits and Transformer models.
• It leverages the trigger-deceiver construction to show that even one additional component forces an exponential gap in labeled sample requirements.
• Empirical experiments on binary addition confirm that standard certification methods may fail to distinguish non-exact models from true targets.

Certification Complexity under Minimal Overparametrization in Circuits and Transformers

Overview

The paper "Certification from Examples is Hard for Circuits and Transformers under Minimal Overparametrization" (2605.22964) investigates the complexity of certifying the correctness of hypotheses in algorithmic domains, specifically examining threshold circuits ($TC$), other circuit classes ($AC$, $NC$), and log-precision Transformer architectures. The focus is on the number of labeled input-output examples required to certify that a learned function matches a given target, in both exact and approximate settings.

The central result is that even minimal overparametrization—such as adding a single gate to a circuit or a single head with limited width to a Transformer—can provably escalate the certificate complexity from polynomial to exponential in the size of the input domain, for all targets in the base class. Theoretical arguments are complemented by empirical investigations using constructed threshold circuits and trained Transformers on binary addition.

Formal Problem: Certification from Labeled Examples

Let $\mathcal{H}$ be a hypothesis class (e.g., Boolean circuits or Transformer models), and let $f^*$ be a target function (i.e., the intended computation). A certificate is a set $S$ of labeled examples (input-output pairs) such that the only $h \in \mathcal{H}$ consistent with all these examples is $f^*$. The minimal certificate size for $(f^*,\mathcal{H})$ quantifies the data requirement to guarantee exactitude.

The principal concern is the effect of overparametrization: e.g., moving from a class $\mathcal{H}$ to a slightly larger class $AC$0 (e.g., by permitting one additional threshold gate), and how this affects the certificate size for targets $AC$1 considered within $AC$2.

Exponential Certification Hardness via Minimal Class Extensions

The backbone of the hardness results is a combinatorial construction, originated from the trigger-deceiver paradigm. This is illustrated structurally as follows:

Figure 1: Same-depth deceiver construction in $AC$3. A “trigger” threshold gate detects membership in a trigger block $AC$4, feeding the result to the top gate, thereby forcing the output on that block while leaving the rest of the function unchanged.

• Trigger blocks: For a subset $AC$5, each pattern $AC$6 defines a "trigger block" $AC$7—inputs where exactly $AC$8.
• Deceivers: For each block, construct a new function $AC$9 that outputs $NC$0 outside $NC$1 and a fixed, possibly incorrect, value within. By careful engineering (adding just one threshold or logic gate), the class extension accommodates all such deceivers, and these modifications are disjoint across blocks.
• Combinatorial lower bound: Any certificate must intersect all such blocks, so certification requires at least as many labels as there are disjoint blocks.

Sharpness in $NC$2

For threshold circuits $NC$3, adding a single extra threshold gate cannot be reliably tested by any certificate with fewer than $NC$4 examples, for trigger length $NC$5, potentially $NC$6. This yields an exponential gap between the best-case certificate size for some targets within $NC$7 (which can be polylogarithmic or polynomial, see the counting arguments in the appendix) and the worst-case for certification inside the minimally enlarged $NC$8.

Analogous Barriers for $NC$9 and $\mathcal{H}$0

In $\mathcal{H}$1, creation of deceivers is possible by adding one detection gate and one combining gate (usually requiring a depth increment). For bounded-fan-in circuits ($\mathcal{H}$2), the cost is linear in trigger set size $\mathcal{H}$3, but with a constant overhead, the exponential-in-$\mathcal{H}$4 lower bound holds for reasonable $\mathcal{H}$5.

Extension to Transformers

Central to the empirical relevance is the extension of the deceiver argument to Transformers, leveraging recent results on their circuit representational power. The authors provide an explicit block-override construction within log-precision projected-pre-norm AHAT Transformers: adding one head and six auxiliary coordinates suffices to implement the deceiver.

Figure 2: Surviving deceivers under uniformly sampled certificate candidates. (Left) The threshold-circuit construction reveals that the expected number of surviving deceivers declines only exponentially when the certificate size grows with the input dimensionality. (Right) Among trained Transformers, even large certificate subsets often fail to exclude all non-exact models, indicating empirical alignment with the combinatorial barrier.

Limits of Approximate Certification

The block-deceiver construction extends to approximate settings: to guarantee exclusion of all hypotheses with even polynomially many errors, one requires exponentially many labels. Only when tolerating a constant relative test error can certificates shrink to sizes that are subexponential (e.g., $\mathcal{H}$6 in tolerance $\mathcal{H}$7), but this permits exponentially many absolute mistakes in the domain.

Empirical Analysis: Addition Task

Two empirical studies complement the theory:

• Threshold Circuit Construction: On recognizing binary addition, the authors explicitly instantiate the trigger-deceiver construction, demonstrating that even a small gate increment results in $\mathcal{H}$8 indistinguishable functions outside any certificate that fails to sample the corresponding block.
• Trained Transformer Models: On held-out sets for binary addition, validated Transformer models with high accuracy nonetheless admit nontrivial residual error sets. Certificate candidates generated by uniform random sampling (over all test points or targeted subsets) rarely intersect all such "deceivers" unless their size is nearly as large as the test set itself.

  Figure 3: Distribution of validation and test errors across accepted models. While some models are test-exact, others pass all validation but admit nontrivial errors on held-out data, reinforcing the existence of deceivers among practical models.

These experiments highlight that even robust validation protocols—sampling up to hundreds of thousands of points—can fail to eliminate all candidates, dovetailing with the combinatorial lower bounds.

Theoretical Implications and Future Directions

The results establish that passive certification (via labeled input-output data) is fundamentally limited in overparametrized models, even if the overparametrization is minimal relative to the original class. The phenomenon is not an artifact of a few adversarially chosen targets; it occurs uniformly for all functions in the base class.

Practical consequences: In realistic settings where model architectures allow minor overparametrization (a nearly universal scenario in modern machine learning), no feasible-sized certificate can guarantee the absence of "deceivers"—functions that behave identically on all provided data yet differ elsewhere.

Theoretical implications: The strong dependence of certificate size on the hypothesis class (and even its minimal extensions) constrains the power of specification-by-example paradigms and highlights a major gap between learnability and certifiability.

Avenues for future work include:

• Exploring active certification protocols (e.g., queries, counterexample synthesis),
• Certification in settings with privileged access (e.g., model structure, intermediate traces),
• Quantification of hardness in neural architectures with richer interaction models,
• Designing architectural restrictions amenable to efficient certification.

  Figure 4: Easy-versus-hard certification in finite circuit classes. Certification can transition abruptly from tractable to exponentially hard with minimal overparametrization, demonstrated by a polynomially certifiable target in the base class whose certification becomes exponentially hard in the enlarged class.

Conclusion

This work rigorously quantifies the exponential vulnerability of certification to small increases in model expressivity, applicable not only to classical Boolean circuits but also directly to modern Transformer architectures. The results challenge the practical adequacy of output-only certification for verifying complex learned models in reasoning and algorithmic tasks. Consequently, a paradigm shift towards richer verification protocols or more constrained hypothesis classes may be necessary to achieve reliable certifiability in future AI systems.