Compositional Modular Networks (CMNs)

Updated 25 June 2026

CMNs are neural architectures that decompose complex tasks into reusable modules, enabling explicit compositional generalization and interpretable multi-task learning.
They dynamically assemble specialized units—such as attention, re-attend, and classification modules—to systematically parse and process structured inputs.
CMNs enhance transferability and sample efficiency by sharing parameters across modules, improving performance in zero-shot, visual grounding, and language tasks.

Compositional Modular Networks (CMNs) constitute a class of neural architectures designed to achieve compositional generalization, transferability, and sample efficiency through the systematic reuse, permutation, and modular assembly of small, specialized computational units (“modules”). CMNs decompose complex tasks—such as visual grounding, question answering, zero-shot classification, and multi-task learning—into simpler, reusable primitives, allowing for an explicit modeling of syntax, structure, and relationships in both inputs and outputs. This modular paradigm supports both data-driven soft compositionality (via learned attention or gating) and explicit, mathematically characterized compositionality (via structural gluing rules or dynamic neural network assembly).

1. Core Principles and Definitional Scope

CMNs are architectures wherein a task is decomposed into sub-tasks solved by distinct, reusable neural modules, whose outputs are composed via learned or structured mechanisms. The foundational characteristics are:

Explicit compositionality: Input structures (e.g., referential expressions, visual questions, semantic pairs) are decomposed into components (e.g., subject/object/relation, concept-attribute, task-specific operators), each assigned to specific modules (Hu et al., 2016, Andreas et al., 2015, Purushwalkam et al., 2019).
Module reusability and parameter sharing: Modules are trained jointly across multiple dynamic network instantiations, enabling transfer across tasks and sample-efficient learning (Andreas et al., 2015, Zhmoginov et al., 2021).
End-to-end learnability: Both module parameters and the rules for composing/wiring modules are differentiably trained for optimal performance in inference tasks (Hu et al., 2016, Tian et al., 2020, Assouel et al., 2023).
Functional separation: Modules operate on homogeneous (e.g., residual blocks (Zhmoginov et al., 2021)) or semantically partitioned (e.g., attribute detectors, attention units) subspaces, each specializing in a sub-task or relation.

The term ‘compositional modular network’ encompasses architectures from referential expression grounding (Hu et al., 2016), visual question answering (Andreas et al., 2015), zero-shot concept learning (Purushwalkam et al., 2019), program-like visual reasoning (Assouel et al., 2023), to theoretically principled constructions in neural dynamical systems (Alvarez, 5 Jun 2026).

2. Neural Module Network Architectures

The prototypical Neural Module Network (NMN) assembles task-specific neural circuits from a general-purpose module inventory, dynamically wiring together atomic units for each instance:

Module types: Attend, re-attend, combine, classify, measure modules, each parameterized per semantic ‘instance’ (e.g., attend[cat], re-attend[above]) (Andreas et al., 2015).
Data flow: Modules process canonical types (image features, attention maps, label logits), outputting intermediate tensors recursively combined according to a layout parsed from the input (e.g., dependency tree of a question).
Dynamic assembly: For each example, a question or referential phrase is syntactically parsed into an s-expression or sequence, which specifies the instantiated module graph (Andreas et al., 2015); grounding tasks decompose phrases into subject, relation, and object via soft-attention (Hu et al., 2016).
End-to-end training: All module weights are trained jointly via back-propagation through the dynamically constructed computation graph, with shared parameters encouraging multi-use and sample efficiency (Andreas et al., 2015).
Parameter sharing: Modules of identical semantic type and instance share parameters across all network instantiations, while different types are architecturally homogeneous but do not share weights (Andreas et al., 2015).

Empirically, NMNs demonstrate improved compositional reasoning and out-of-distribution generalization on tasks requiring hierarchical structure understanding, such as SHAPES and VQA datasets (Andreas et al., 2015).

3. Compositionality in Vision and Language Grounding

In visual grounding and compositional captioning, CMNs parse linguistic inputs and allocate modular networks for explicit substructure alignment:

Referential Expressions: CMNs parse natural-language referential phrases via a bi-LSTM with learned soft-attention, producing three contextual vectors (subject, relation, object) (Hu et al., 2016).
- Localization (unary) modules: Score candidate regions against subject/object vectors.
- Relationship (pairwise) modules: Score spatial/visual relations between region pairs given the relation vector.
- Joint scoring and marginalization: For subject region prediction, CMNs maximize over possible corresponding object boxes, yielding a final subject score (Hu et al., 2016).
Caption Generation: Hierarchical CMNs for image captioning employ attribute modules (color, count, size, spatial, semantic), stacking module-attended word vectors for more specific, grounded descriptions. A multi-head LSTM stack integrates region attention and sequential/natural language modeling (Tian et al., 2020).
Zero-shot Learning: Task-driven modular networks construct concepts as compositions of pre-trained modules, with task-conditioned gating learning to combine modules corresponding to objects and attributes for unseen combinations (Purushwalkam et al., 2019).

Performance improvements over monolithic baselines are observed in both accuracy and detail specificity, notably in the color and counting subcategories of the SPICE metric for captioning (Tian et al., 2020).

4. Architectural Strategies for Multi-task, Transfer, and Domain Adaptation

Compositional modularity extends to backbone architectures and multi-task regimes:

Isometric Modular Backbones: CMNs can be implemented atop isometric ResNet-style backbones in which all residual blocks have identical shape, allowing arbitrary reordering or repetition (Zhmoginov et al., 2021).
- Template bank and mixture modules: Instead of unique weights per block, a small template bank of modules is composed at each layer via learnable mixture weights; reordering and reuse of modules supports parameter-efficient adaptation (Zhmoginov et al., 2021).
- Soft weight sharing: Each task is assigned its own mixture-weight signature over the template bank, supporting per-task specialization and interpolation (Zhmoginov et al., 2021).
Knowledge transfer and domain adaptation: For novel tasks or domains, mixture weights can be reinitialized or interpolated, fine-tuning only the mixture coefficients with the module bank frozen. Domain adaptation can be performed by adversarial or moment-matching criteria, updating only a small set of parameters ( $\sim$ 100s) rather than all backbone weights (Zhmoginov et al., 2021).
Empirical evidence: CMN-based architectures exhibit gains in single-task, multi-task, and transfer learning benchmarks without a substantial parameter overhead, demonstrating the utility of plug-and-play compositionality (Zhmoginov et al., 2021).

5. Theoretical Formalizations and Mathematical Compositionality

Recent advances characterize compositionality in modular neural systems with mathematical precision:

Low-rank gluing in threshold-linear networks (TLNs): In the context of inhibition-dominated TLNs, compositional modular networks are constructed by low-rank gluing of subnetworks (modules), each with arbitrary recurrent connectivity (Alvarez, 5 Jun 2026).
- Fixed-point compositionality theorem: For any such low-rank gluing, every global fixed-point support is a union of local module fixed-point supports; rank-1 gluings yield a complete combinatorial rule—“all survive vs. all die”—determining the surviving global attractors (Alvarez, 5 Jun 2026).
- Application to graph-based networks (CTLNs and gCTLNs): Cyclic, clique, and disjoint unions correspond to rank-1 gluings; the number of global fixed points is then the product of the local module attractors, enabling exponential growth in the attractor count (Alvarez, 5 Jun 2026).
Design of combinatorially rich attractor landscapes: By arranging modules with known local dynamics and coupling them via prescribed low-rank patterns, engineers can construct networks supporting predictable repertoires of fixed points and limit cycles (Alvarez, 5 Jun 2026).

This theoretical work provides foundational recipes for scalable, interpretable, and compositional neural architectures.

6. Applications, Empirical Insights, and Generalization

CMNs are empirically validated across broad domains:

Visual Reasoning and VQA: NMNs and their descendants match the compositional linguistic structure of questions, yielding improved generalization, especially to deeper, unseen compositions (Andreas et al., 2015, Assouel et al., 2023).
Visual Grounding and Captioning: Fine-grained parsing into semantic modules produces more accurate and detailed referent localization and sentence generation, with robust gains in SPICE subcategories (Hu et al., 2016, Tian et al., 2020).
Zero-shot and Generalized Composition: Task-driven CMNs achieve superior trade-offs between seen and unseen concept pairs on benchmark datasets (e.g., MIT-States, UT-Zappos50k), aided by module gating, concept dropout, and compositional wiring (Purushwalkam et al., 2019).
Multi-task/Transfer/Domain Adaptation: Parameter-efficient module reordering enables flexible switching across tasks and domains, with competitive or improved performance and interpretability of sharing patterns (Zhmoginov et al., 2021).
Synthetic and Generative Tasks: OC-NMN extends compositional modularity to generative and analogical reasoning, reliably parsing I/O examples into module programs, and leveraging compositional “imagination” (synthetic program generation) for improved out-of-distribution generalization (Assouel et al., 2023).

A recurring insight is that soft, end-to-end learned parsing consistently outperforms externally imposed rigid decompositions, as evidenced by empirical drops when using external parsers for referential expressions (Hu et al., 2016).

7. Limitations, Challenges, and Outlook

While CMNs deliver explicit compositionality and modular transfer, several challenges remain:

Generalization to novel primitives: Most current CMNs assume a fixed vocabulary of modules or concepts. Handling truly novel entities may require meta-learning or additional language priors (Purushwalkam et al., 2019).
Scaling to deeper compositions: Empirical results show diminishing performance as the composition depth or structural complexity increases, indicating an ongoing need for research on deep compositional generalization (Andreas et al., 2015, Assouel et al., 2023).
Module bank optimization: The size and diversity of the module bank must be sufficient to cover the targeted compositional space; too few modules can cause collapse and poor extrapolation (Assouel et al., 2023).
Inductive bias vs flexibility: Explicit program structure (e.g., NMN parse trees) offers interpretability but may limit adaptation to ambiguous or ungrammatical inputs; soft attention mechanisms are empirically favored for robust parsing (Hu et al., 2016).
Biological validity and theoretical characterization: The connection between network modularity and functional compositionality is mathematically characterized in TLNs, suggesting further avenues for principled network design and neuroscientific modeling (Alvarez, 5 Jun 2026).

A plausible implication is that advances in low-rank modular assembly, program induction, and soft compositional parsers may enable scalable CMNs with systematic, interpretable, and biologically plausible compositional capabilities.

References:

Modeling Relationships in Referential Expressions with Compositional Modular Networks (Hu et al., 2016) Compositional Models: Multi-Task Learning and Knowledge Transfer with Modular Networks (Zhmoginov et al., 2021) Neural Module Networks (Andreas et al., 2015) Task-Driven Modular Networks for Zero-Shot Compositional Learning (Purushwalkam et al., 2019) Image Captioning with Compositional Neural Module Networks (Tian et al., 2020) OC-NMN: Object-centric Compositional Neural Module Network for Generative Visual Analogical Reasoning (Assouel et al., 2023) Fixed point compositionality via low-rank gluing rules in inhibition-dominated threshold-linear networks (Alvarez, 5 Jun 2026)