Determine whether transformer models can learn good error-correcting codes

Determine whether transformer language models can be designed or trained to implement distributed good error-correcting codes whose redundancy remains a fixed fraction of the model at arbitrarily large scales, rather than relying only on repetition-based error correction whose useful-capacity fraction vanishes with model size.

Background

The paper contrasts repetition-based protection, for which redundancy grows until useful capacity vanishes, with good distributed codes, for which the coding rate remains finite as system size increases. The authors argue that asymptotically fault-tolerant AI would require the latter type of representation.

Although the experiments show a recovery of effective capacity at larger tested scales, they do not establish whether transformer architectures can reliably construct or learn good codes by design or through training.

References

It is not obvious a priori that good codes can be built into transformer models by design or learned by training.

— Fault-tolerant foundation models  (2610.10311 - McCourt et al., 7 Oct 2026) in Main text, paragraph beginning “Coding theory suggests that the scaling behavior of fault-tolerance in AI systems ought to be sharply defined”