Generation is Required for Data-Efficient Perception (2512.08854v1)

Abstract: It has been hypothesized that human-level visual perception requires a generative approach in which internal representations result from inverting a decoder. Yet today's most successful vision models are non-generative, relying on an encoder that maps images to representations without decoder inversion. This raises the question of whether generation is, in fact, necessary for machines to achieve human-level visual perception. To address this, we study whether generative and non-generative methods can achieve compositional generalization, a hallmark of human perception. Under a compositional data generating process, we formalize the inductive biases required to guarantee compositional generalization in decoder-based (generative) and encoder-based (non-generative) methods. We then show theoretically that enforcing these inductive biases on encoders is generally infeasible using regularization or architectural constraints. In contrast, for generative methods, the inductive biases can be enforced straightforwardly, thereby enabling compositional generalization by constraining a decoder and inverting it. We highlight how this inversion can be performed efficiently, either online through gradient-based search or offline through generative replay. We examine the empirical implications of our theory by training a range of generative and non-generative methods on photorealistic image datasets. We find that, without the necessary inductive biases, non-generative methods often fail to generalize compositionally and require large-scale pretraining or added supervision to improve generalization. By comparison, generative methods yield significant improvements in compositional generalization, without requiring additional data, by leveraging suitable inductive biases on a decoder along with search and replay.

• The paper reveals that generative decoding secures robust compositional generalization by explicitly inverting latent variable generators.
• It demonstrates that constrained decoders with techniques like gradient-based search and replay outperform encoder-only models, especially in OOD settings.
• Empirical results on synthetic datasets validate that integrating generative approaches yields marked improvements in data efficiency and compositional accuracy.

Generation is Required for Data-Efficient Perception: Theory and Empirics

Introduction and Motivation

This work presents a rigorous investigation into whether generative modeling is necessary for achieving data-efficient perception with strong compositional generalization. The core hypothesis scrutinized is whether machine perception models that explicitly invert a generative process—recovering latent causes rather than merely encoding observed data—are uniquely equipped to generalize compositionally and efficiently, as is observed in human perception. The dominant paradigm in present computer vision is encoder-based (non-generative); the paper challenges the assumption that this paradigm suffices for robust generalization, especially out-of-domain (OOD), in data-limited regimes.

Formal Framework for Compositional Generalization

Under this framework, images are assumed to arise from a latent variable model $x = f(z)$, where $f$ is a diffeomorphic generator mapping compositional latents $z$ (factored into slots like object, background, etc.) to high-dimensional observations. Compositional generalization is formally defined as reliable inverse inference of unseen combinations of latent slots in OOD regions of the latent manifold, even when only a subset of combinations is observed in-domain.

The distinction between paradigms is made precise: generative methods learn a decoder $\hat{f}$ and compute representations by inverting it; non-generative methods map inputs $x$ directly to representations via an encoder $\hat{g}$, independently of any decoder. Achieving the desired OOD guarantees depends on whether inductive biases in $\hat{f}$ and $\hat{g}$ can reliably enforce such recovery.

Figure 1: Generative versus non-generative compositional generalization; generative methods invert a decoder whereas non-generative methods map images to representations directly.

Theoretical Analysis: Constraints and Inductive Biases

A central theoretical result is established: for meaningful compositional generalization, inductive biases must restrict the function class of the generator (and its inverse) such that identification is possible both in-domain and OOD. For generative approaches, constraints on $\hat{f}$ can be enforced globally via architectural choices (e.g., additive slot-wise neural networks, sparsity of higher-order derivatives) or regularization, directly reflecting the generative process's structure. For instance, decoders can be regularized such that their Hessians are block-diagonal (for additive or low-degree interaction models).

In contrast, any attempt to enforce the necessary structure on a non-generative encoder $\hat{g}$ depends on properties of the data manifold in the ambient space, which is unknowable in OOD regions. The paper proves that, except in highly restrictive cases (such as $n = 0$ interaction functions with known sparsity), the encoder function class $\mathcal{G}_\textnormal{int}$ cannot be reliably constrained by architectural or regularization biases. This exposes a fundamental asymmetry: generative models can embed compositional generalization guarantees by design, but non-generative methods cannot.

Inversion Strategies: Gradient-Based Search and Generative Replay

Given the challenge of explicit inversion in high-dimensional generative models, the paper evaluates practical inference mechanisms:

• Gradient-based search: Given an unseen image $x$, initialize with the encoder output $\hat{g}(x)$, then optimize $z$ to minimize $\|x - \hat{f}(z)\|^2$.
• Generative replay: Sample novel OOD slot combinations, generate corresponding images with $\hat{f}$, and train encoders on these compositions purely synthetically.

  Figure 2: Left, gradient search for OOD inversion; right, generative replay, training the encoder on synthetic OOD samples.

Empirical Evaluation: Compositional Generalization in Practice

Experiments were conducted using the PUG dataset of photorealistic synthetic images, partitioned into ID and OOD by holding out various slot combinations. Both non-generative and generative approaches were evaluated for their ability to infer correct representations on OOD samples.

• Non-generative results: Encoder-only models, including unsupervised and supervised variants using recent vision transformers (ViTs), exhibit poor OOD generalization unless pre-trained at large data scales or using added supervision. OOD generalization emerges reliably only for encoders pretrained using SigLIP2-scale corpora. Notably, in split regimes where slot interactions are minimal ($n=0$), non-generative models generalize well, highlighting the influence of underlying slot interaction structure.

  Figure 3: OOD performance of non-generative models; strong generalization only with large-scale pretraining or simple slot structure.

• Generative results: Generative models leveraging constrained decoders, gradient search, and replay yield markedly higher OOD accuracy, even when based on ViTs trained from scratch or with limited data. These results confirm the theoretical prediction that generative methods can enable compositional generalization efficiently and with minimal supervision.

  Figure 4: Generative methods (VAE + replay/search) exhibit large improvements in OOD slot decoding accuracy on challenging compositions.

Implications, Related Work, and Future Directions

Contradictory Findings

The paper makes a bold claim: "Compositional generalization cannot be reliably induced in non-generative models via inductive bias, but can be straightforwardly embedded in generative models." This directly contradicts the prevailing belief that data scale and encoder tuning alone can suffice for robust generalization.

Theoretical Connections

The formal analysis relates to central principles of causal inference, specifically the asymmetry between explaining effects from causes (generative inference) and inferring causes from effects (anti-causal inference). This provides a theory-driven justification for the empirical advantage observed for generative approaches in compositional settings.

Practical Impact and Speculation

Theoretical and experimental evidence suggests that generative mechanisms—explicit decoder inversion, online search, and generative replay—will be central for scaling data-efficient, robust perception in AI. Future directions may focus on integrating generative modeling within large-scale multimodal pretraining, leveraging synthetic composition for efficient learning, and developing new benchmarks targeting compositional generalization beyond entangled, "bag-of-words" style datasets.

Conclusion

This work formally and empirically demonstrates that generative strategies—those that learn to construct and invert decoders—uniquely enable compositional generalization and data efficiency, in contrast to non-generative encoder-based approaches that lack enforceable inductive bias outside the observed data manifold. For progress towards human-level AI perception, future research must prioritize generative mechanisms for representation and reasoning, possibly integrating these methods with synthetic data scaling and causal abstractions.