---
title: Systematic Generalization in AI
url: https://www.emergentmind.com/topics/systematic-generalization
type: topic
---

# Systematic Generalization in AI

Systematic generalization refers to the capacity of a learning system to correctly and robustly process novel combinations of components, rules, or concepts—generalizing to configurations never seen during training, so long as their parts and their modes of combination were sufficiently covered. This capability, which underpins human compositional reasoning, is widely regarded as essential for flexible intelligence and remains a core challenge for neural models across domains including language, vision, grounded reasoning, reinforcement learning, and combinatorial optimization. Recent progress has illuminated both architectural and data-driven inductive biases that support or hinder systematic generalization, though even state-of-the-art deep networks often fall short in compositional OOD settings.

## 1. Foundational Definitions and Theoretical Scope

Systematic generalization, sometimes termed compositional generalization, is distinguished from i.i.d. generalization by its focus on structured out-of-distribution (OOD) extrapolation: a model, trained on a subset of all possible compositions of primitives, must generate correct outputs for previously unseen but permissible recombinations [2211.11956], [2505.13089]. Precise definitions vary per domain, but canonical instances include:

- **Language**: After seeing verbs {walk, jump} and adverbs {cautiously, quickly} in some combinations, the learner must predict the meaning of "jump cautiously" having never seen it during training [2003.05161], [2504.01445].
- **Vision and VQA**: The ability to answer about new pairs "red cylinder" or "left_of blue triangle" when all constituent attributes/relationships occurred in isolation or different pairings at train time [1811.12889], [1912.05783], [2309.08798].
- **Reinforcement Learning**: Solving planning tasks in unseen environments whose transition dynamics factor into shared and environment-specific causal mechanisms [2202.06545].

Mathematically, one typical formalization is: for input space $X$, output space $Y$, and some compositional splitting of $X$, a model $f: X \to Y$ achieves systematic generalization if, trained on $(x,y)$ pairs from a restricted $p_\text{train}(x,y)$, it outputs correct $y$ for all $x$ in a test set that comprises novel combinations of primitive factors seen during training [2505.13089]. Quantitatively, the *systematic generalization gap* is often measured as $\Delta = \text{Acc}_\text{train} - \text{Acc}_\text{ood}$ [2309.08798].

## 2. Inductive Biases and Architectural Mechanisms

Systematic generalization critically depends on the presence of correct inductive biases in the learner. Several mechanisms and classes of architectures have been studied:

### Symbolic and Hybrid (Neural-Symbolic) Approaches

Symbolic methods and hybrid systems—such as Neural-Symbolic Recursive Machines (NSR) [2210.01603]—hardwire or induce compositional structure explicitly. NSR constructs a latent Grounded Symbol System: a directed tree whose nodes encode symbols with semantic values, learned jointly with neural perception, syntactic parsing, and semantic program induction modules. Its deduction–abduction learning algorithm iteratively samples and refines tree-structured symbolic representations, introducing inductive biases of equivariance and recursive compositionality via pointwise, factorized likelihoods.

Other neural-symbolic frameworks include Neural Module Networks (NMN), with explicit program-defined module layouts whose degree and location of modularity is a key determinant of systematic generalization [2106.08170], [1811.12889].

### Attention and Relational Mechanisms

Edge Transformers [2112.00578] augment the Transformer paradigm with edge-centric representations and triangular attention, enabling explicit compositional chaining over relations (edges) in a discrete or relational structure—addressing the shortfalls of standard self-attention and GNN message-passing for recombining relational primitives.

### Modular and Compositional Models

Chart-based parsers [2007.00266], meta-learned compositional architectures [2504.01445], and tree-structured NMNs [1811.12889] introduce explicit or induced intermediate representations corresponding to compositional structure (e.g., constituent parse trees, functional programs, or modular reasoning layouts), substantially boosting systematicity in both language and visual grounding domains.

## 3. Data Distributional Properties as Inductive Bias

Inductive bias is also imparted by data properties. Systematic generalization can be heavily influenced—even enabled—by specific properties of the training distribution [2502.20499], [2309.08798]. Key observed factors:

- **Diversity**: Expanding the support size of latent factors (e.g., increasing the number of unique colors or adverbs in the training data) reduces reliance on spurious statistical associations and forces the learner to disentangle attributes, yielding large gains (up to +89% OOD accuracy) [2502.20499], [2309.08798].
- **Burstiness and Interventions**: Controlled within-context attribute diversity (burstiness) and random per-sample interventions break co-occurrence biases, further encouraging factorized representations [2502.20499].
- **Mutual Information and Parallelism**: Systematic generalization is strongly predicted by normalized mutual information (NMI) between latent factors in the data; lower NMI prompts more parallel, analogy-friendly neural geometries [2502.20499].

Empirically, mixing simple but diverse compositions into training (D3 principle) provides dramatic systematicity improvements regardless of the similarity between train and test, and is more data-efficient in modular than monolithic architectures [2309.08798].

## 4. Benchmarking and Quantitative Evaluation

Systematic generalization is probed with compositional and OOD splits designed to require robust recombination:

- **SCAN and gSCAN**: Sequence-to-sequence mappings in synthetic language-to-action or grounded command settings, with compositional splits holding out verb-adverb, color-shape, or directional combinations [2003.05161], [2009.05552].
- **CLOSURE and CLEVR-CoGenT**: Visual reasoning and VQA tasks with novel attribute-relation or program-template combinations [1912.05783], [2106.08170].
- **CFQ and SyGNS**: Semantic parsing benchmarks controlling for graph-structural or syntactic novelty [2106.01077], [2112.00578].
- **Meta-learning for Compositionality**: SYGAR demonstrates systematicity in spatial reasoning with dynamically composed transformation grammars [2504.01445].

Metrics are typically Exact Match accuracy, OOD accuracy, and the systematicity gap $\Delta$. Notable results include NSR achieving 100% accuracy on all SCAN and PCFG splits (far exceeding conventional and prior hybrid models) [2210.01603], and GLT's latent CKY compositionality achieving 96.1% accuracy on CLOSURE, outperforming both end-to-end and neuro-symbolic baselines [2007.00266].

## 5. Limitations, Ablations, and Open Challenges

Despite progress, several limitations remain:

- **Architectural fragility**: Many mechanisms for modularity, compositionality, or variable binding are sensitive to layout, parametrization, or hard-coded priors. NMNs require carefully chosen module layouts; end-to-end approaches often converge to configuration with poor systematic generalization unless strongly regularized [1811.12889], [2106.08170].
- **Data-hungriness**: Vanilla seq2seq architectures can achieve perfect systematicity *if* data support is information-rich (high entropy), but performance collapses at low entropy—suggesting the impossibility of data-only solutions in structured, sparse regimes [2505.13089].
- **Interpretability and reasoning transparency**: Models capable of implicit generalization (e.g., "no-proof" Transformer models in theorem-proving) may succeed in OOD tasks but render their computation opaque [2009.14786].
- **Generalization beyond limited compositionality**: Even compositional or modular models often fail on deeper recursive structures, variable binding, or higher-arity compositional splits if the requisite inductive bias is absent or data is insufficiently rich [2106.01077], [1912.05783].

Ablation studies consistently confirm the necessity of recursive, pointwise likelihoods, explicit modularity, and compositional interpretations at all levels of the network—removing these (e.g., abducing, parsing, enforcing equivariance, or modularization) sharply degrades systematicity [2210.01603], [2106.08170], [1811.12889].

## 6. Unifying Mechanisms and Future Directions

Synthesizing across domains, inductive bias supporting systematic generalization arises from the factorization of computation—embodied in module architectures, neural-symbolic hybrids, or meta-learning regimes—and is strengthened by designed data distributions to encourage independent, analogical, or algebraically structured representations [2210.01603], [2504.01445], [2502.20499]. Key principles with strong empirical and sometimes provable support include:

- **Equivariance and Compositionality**: Recursive, pointwise processing (as in NSR) guarantees permutation equivariance and compositionality by construction [2210.01603], [2504.01445].
- **Causality and Factorization**: Factoring models per the causal graph of the domain—e.g., per-variable in time series [2102.05602], or via shared causal transition models in reinforcement learning [2202.06545]—provably supports transfer and systematic extrapolation.
- **Data-centric approaches**: Manipulation of data entropy and mutual independence of latent factors provides a powerful and easily deployable lever for making systematic generalization possible even for generic neural nets [2505.13089], [2309.08798], [2502.20499].

Challenges remain in producing scalable, flexible architectures that retain interpretability and modularity while handling the intricacies of real-world, high-dimensional data. Key open directions are the unification of symbolic and distributed methods, meta-learning of structural priors, data-efficient curriculum design for compositionality, extension to cross-domain, temporally coupled, and multi-agent settings, and a principled mathematical characterization of systematic generalization's necessary and sufficient conditions [2211.11956], [2605.02026].

---

References:
- [2210.01603]: "Neural-Symbolic Recursive Machine for Systematic Generalization"
- [2112.00578]: "Systematic Generalization with Edge Transformers"
- [2505.13089]: "Systematic Generalization in Language Models Scales with Information Entropy"
- [2502.20499]: "Data Distributional Properties As Inductive Bias for Systematic Generalization"
- [2309.08798]: "D3: Data Diversity Design for Systematic Generalization in Visual Question Answering"
- [2003.05161]: "A Benchmark for Systematic Generalization in Grounded Language Understanding"
- [1912.05783]: "CLOSURE: Assessing Systematic Generalization of CLEVR Models"
- [2106.08170]: "How Modular Should Neural Module Networks Be for Systematic Generalization?"
- [1811.12889]: "Systematic Generalization: What Is Required and Can It Be Learned?"
- [2106.01077]: "SyGNS: A Systematic Generalization Testbed Based on Natural Language Semantics"
- [2202.06545]: "Provably Efficient Causal Model-Based Reinforcement Learning for Systematic Generalization"
- [2504.01445]: "Enabling Systematic Generalization in Abstract Spatial Reasoning through Meta-Learning for Compositionality"
- [2211.11956]: "A Short Survey of Systematic Generalization"
- [2102.05602]: "Systematic Generalization in Neural Networks-based Multivariate Time Series Forecasting Models"

Source: https://www.emergentmind.com/topics/systematic-generalization