Penguin-Encoder: Multimodal and Partial Equivariance

• Penguin-Encoder is a novel encoder module that redefines both vision-language and MARL pipelines by departing from traditional contrastive pretraining and strict equivariance.
• It combines text-based Transformer layers with 2D positional embeddings and a two-stage pretraining approach to capture fine-grained spatial and temporal features.
• In multi-agent reinforcement learning, its partial equivariance mechanism adaptively blends symmetric and non-equivariant message passing, leading to improved efficiency and sample performance.

Penguin-Encoder refers to two distinct neural network architectures introduced in recent research for multimodal vision-language modeling and for partially equivariant message-passing in multi-agent reinforcement learning, respectively. In both contexts, the Penguin-Encoder designates an encoder module that departs from standard practices (contrastive pretraining in VLMs, or strict equivariant GNNs in MARL) to enable greater flexibility, efficiency, and fidelity in complex downstream tasks (Zhang et al., 6 Mar 2026, McClellan et al., 19 Mar 2025).

1. Penguin-Encoder in Vision-LLMs: Architecture and Initialization

The Penguin-Encoder for VLMs, as introduced within the Penguin-VL framework (Zhang et al., 6 Mar 2026), leverages the Transformer layers of a text-only LLM (specifically Qwen3-0.6B) to serve as the vision encoder. This encoder replaces the LLM’s causal (autoregressive) self-attention with bidirectional self-attention and augments it with a linear patch embedding front end and two-dimensional rotary positional embeddings (2D-RoPE).

Architectural Schematic

$$Q_i, K_i, V_i = z_{\ell-1} W_q,\, z_{\ell-1} W_k,\, z_{\ell-1} W_v$$3

Key implementation details:

• Patch embeddings map flattened image patches via a linear projection $E_p$.
• Each patch embedding receives a unique 2D-RoPE, encoding spatial information.
• Layer normalization is applied at input and after each attention block.
• The encoder reuses QK-Normalization as implemented in Qwen3.

Input representation for an image patch $i$:

• Patch embedding: $x_{p_i} = E_p(I_{\text{patch}_i})$
• 2D positional embedding and normalization: $$z_{0_i} = \text{LayerNorm}(x_{p_i} + \text{PE}_{2D}(i_x, i_y))$$
• Self-attention per Transformer layer:

$$Q_i, K_i, V_i = z_{\ell-1} W_q,\, z_{\ell-1} W_k,\, z_{\ell-1} W_v$$

$$A_i = \text{Softmax}\left( \frac{Q_i K_i^T}{\sqrt{d}} \right) V_i$$

$$z_\ell = \text{FFN}(\text{LayerNorm}(A + z_{\ell-1}))$$

2. Pretraining Objective and Fine-Grained Supervision

Penguin-Encoder departs from contrastive pretraining, instead adopting a two-stage procedure:

1. Stage 1 (Low-res, 2048 tokens): Mixed caption alignment with feature reconstruction; the latter is distillation from a SigLIP-trained teacher.
2. Stage 2 (High-res, 10,240 tokens): Pure caption alignment using cross-entropy against re-captioned ground truth.

The initialization loss combines:

• Amplitude loss: $$L_A = \frac{1}{N} \sum_n \| F_s^{(n)} - F_t^{(n)} \|_1$$
• Direction (cosine) loss: $$L_D = \frac{1}{N} \sum_n \mathrm{tr}\left(F_s F_t^T /(\|F_s\|_2 \cdot \|F_t\|_2)\right)$$
• Relation loss: $$L_R = \frac{1}{N} \sum_n \| (F_s F_s^T / \|F_s\|_2^2) - (F_t F_t^T / \|F_t\|_2^2) \|_1$$

Full objective:

$i$0

This configuration preserves both global and fine-grained spatial/temporal information, unlike contrastive approaches that operate only on global summaries (e.g., [CLS] tokens).

3. Integration in Vision-Language Modeling Pipelines

Penguin-Encoder outputs a sequence of visual features projected via a 2-layer MLP (with GELU activation) to the LLM's hidden size. The resulting tokens are merged—by concatenation or interleaving—with text tokens and fed into the decoder’s cross-attention layers.

At each cross-attention layer, the operation is:

• $i$1
• $i$2
• $i$3
• $i$4

with $i$5 (visual), $i$6 (textual+visual).

4. Benchmarking, Ablation, and Efficiency Analysis

Penguin-VL employing the Penguin-Encoder consistently matches or surpasses leading VLMs (e.g., Qwen3-VL, InternVL-3.5, GPT-5 nano) despite utilizing a far smaller dataset and encoder parameter budget. Notably, Penguin-VL 2B outperforms Qwen3-2B, InternVL-2B, Gemma3n-2B, and SmolVLM2 on chart/OCR/document and dense reasoning benchmarks by 5–15 points in accuracy. At 8B scale, Penguin-Encoder surpasses or ties state-of-the-art across DocVQA, ChartQA, AI2D, LongVideoBench, and NextQA with only 240 M pretraining samples—versus >40 B for SigLIP2-based pipelines.

Ablation studies reveal:

• Omission of relation loss ($i$7) lowers performance by 1.3 points vs. full objective.
• Removing the reconstruction loss further degrades accuracy.
• Replacing contrastive-pretrained encoders with Penguin-Encoder leads to consistent improvements in tasks requiring spatial and temporal fine granularity.

Its lightweight structure (400 M vision encoder params; compatible with 2B–8B decoders) enables deployment on mobile and edge hardware with reduced latency.

5. Penguin-Encoder in Partially Equivariant Graph Neural Networks

The Penguin-Encoder is also central in the PEnGUiN framework (McClellan et al., 19 Mar 2025) for Multi-Agent Reinforcement Learning, where strict global equivariance is rarely attainable. The architecture formally interpolates between symmetric (EGNN/E2GN2) and standard non-equivariant GNNs, allowing for partial equivariance at various granularities:

• Subgroup Equivariance: Equivariance is enforced for a subgroup $i$8; $i$9 is $x_{p_i} = E_p(I_{\text{patch}_i})$0-equivariant iff $x_{p_i} = E_p(I_{\text{patch}_i})$1.
• Feature-wise Equivariance: Only specific input/output coordinates transform under the symmetry group.
• Regional Equivariance: Equivariance holds only for a subset $x_{p_i} = E_p(I_{\text{patch}_i})$2 of the input space, formalized via an error $x_{p_i} = E_p(I_{\text{patch}_i})$3.
• Approximate Equivariance: $x_{p_i} = E_p(I_{\text{patch}_i})$4 is $x_{p_i} = E_p(I_{\text{patch}_i})$5-approximately equivariant if deviations are bounded by $x_{p_i} = E_p(I_{\text{patch}_i})$6 for all group actions and inputs.

Layer Formulation and Forward Pass

At each layer $x_{p_i} = E_p(I_{\text{patch}_i})$7:

• Each node $x_{p_i} = E_p(I_{\text{patch}_i})$8 carries invariant features $x_{p_i} = E_p(I_{\text{patch}_i})$9 and equivariant coordinates $$z_{0_i} = \text{LayerNorm}(x_{p_i} + \text{PE}_{2D}(i_x, i_y))$$0.
• Equivariant and non-equivariant message branches are computed:
  • $$z_{0_i} = \text{LayerNorm}(x_{p_i} + \text{PE}_{2D}(i_x, i_y))$$1
  • $$z_{0_i} = \text{LayerNorm}(x_{p_i} + \text{PE}_{2D}(i_x, i_y))$$2
• Messages are aggregated and blended using a learnable symmetry score $$z_{0_i} = \text{LayerNorm}(x_{p_i} + \text{PE}_{2D}(i_x, i_y))$$3:

$$z_{0_i} = \text{LayerNorm}(x_{p_i} + \text{PE}_{2D}(i_x, i_y))$$4

• The invariant feature and a non-equivariant coordinate increment are computed:

$$z_{0_i} = \text{LayerNorm}(x_{p_i} + \text{PE}_{2D}(i_x, i_y))$$5

• The EGNN-style coordinate update:

$$z_{0_i} = \text{LayerNorm}(x_{p_i} + \text{PE}_{2D}(i_x, i_y))$$6

• The new coordinates are then:

$$z_{0_i} = \text{LayerNorm}(x_{p_i} + \text{PE}_{2D}(i_x, i_y))$$7

Special Cases: Setting $$z_{0_i} = \text{LayerNorm}(x_{p_i} + \text{PE}_{2D}(i_x, i_y))$$8 recovers standard EGNN/E2GN2 equivariant layers; setting $$z_{0_i} = \text{LayerNorm}(x_{p_i} + \text{PE}_{2D}(i_x, i_y))$$9 yields an ordinary GNN layer.

High-level forward-pass pseudocode:

$$Q_i, K_i, V_i = z_{\ell-1} W_q,\, z_{\ell-1} W_k,\, z_{\ell-1} W_v$$4

6. Empirical Performance and Integration

Experiments in (McClellan et al., 19 Mar 2025) demonstrate that PPO+Penguin-Encoder outperforms standard EGNNs and GNNs across partially equivariant multi-agent environments (MPE, Highway-Env). On “Spread + Bias”, PEnGUiN achieves 80% of asymptotic performance in 50k samples, while EGNN requires 200k and GNN never exceeds 60%. The symmetry score $$Q_i, K_i, V_i = z_{\ell-1} W_q,\, z_{\ell-1} W_k,\, z_{\ell-1} W_v$$0 adapts regionally, reducing in high-asymmetry zones and increasing in symmetric regions. Integration is straightforward for MARL pipelines: replacing the encoder with PEnGUiN and attaching policy/value heads to the outputs.

Computational complexity per layer is $$Q_i, K_i, V_i = z_{\ell-1} W_q,\, z_{\ell-1} W_k,\, z_{\ell-1} W_v$$1, with real-time performance for moderate agent counts ($$Q_i, K_i, V_i = z_{\ell-1} W_q,\, z_{\ell-1} W_k,\, z_{\ell-1} W_v$$2). Extensions include hierarchical coarsening and group-generalization.

7. Significance and Impact

The Penguin-Encoder in vision-language modeling (Zhang et al., 6 Mar 2026) demonstrates that initializing vision encoders from text-only LLMs, together with fine-grained reconstruction alignment objectives, allows compact VLMs (2B–8B) to match or surpass the performance of much larger, contrastively pretrained encoders. This addresses the data inefficiency and lack of dense spatial supervision prevalent in global contrastive methods.

In MARL, PEnGUiN (McClellan et al., 19 Mar 2025) establishes a mathematically principled, sample-efficient architecture that learns to interpolate between full equivariance and unconstrained representation, achieving improved robustness and generalization under real-world asymmetries. The introduction of adjustable partial equivariance provides the flexibility required for challenging, non-ideal environments encountered in practical multi-agent systems.

References:

• "Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders" (Zhang et al., 6 Mar 2026)
• "PEnGUiN: Partially Equivariant Graph NeUral Networks for Sample Efficient MARL" (McClellan et al., 19 Mar 2025)