--- title: 'Penguin-Encoder: Multimodal and Partial Equivariance' url: https://www.emergentmind.com/topics/penguin-encoder type: topic --- # Penguin-Encoder: Multimodal and Partial Equivariance Penguin-Encoder refers to two distinct neural network architectures introduced in recent research for multimodal vision-language modeling and for partially equivariant message-passing in multi-agent reinforcement learning, respectively. In both contexts, the Penguin-Encoder designates an encoder module that departs from standard practices (contrastive pretraining in VLMs, or strict equivariant GNNs in MARL) to enable greater flexibility, efficiency, and fidelity in complex downstream tasks [2603.06569][2503.15615]. ## 1. Penguin-Encoder in Vision-Language Models: Architecture and Initialization The Penguin-Encoder for VLMs, as introduced within the Penguin-VL framework [2603.06569], leverages the Transformer layers of a text-only LLM (specifically Qwen3-0.6B) to serve as the vision encoder. This encoder replaces the LLM’s causal (autoregressive) self-attention with bidirectional self-attention and augments it with a linear patch embedding front end and two-dimensional rotary positional embeddings (2D-RoPE). ### Architectural Schematic ``` Input image I → patch partition → linear patch embed Eₚ → visual token sequence Xᵥ → Transformer Blocks (Qwen3-0.6B, causal→bidirectional, +2D-RoPE) → visual feature sequence Fₛ → lightweight MLP projector → multimodal merger → LLM decoder ``` Key implementation details: - Patch embeddings map flattened image patches via a linear projection $E_p$. - Each patch embedding receives a unique 2D-RoPE, encoding spatial information. - Layer normalization is applied at input and after each attention block. - The encoder reuses QK-Normalization as implemented in Qwen3. Input representation for an image patch $i$: - Patch embedding: $x_{p_i} = E_p(I_{\text{patch}_i})$ - 2D positional embedding and normalization: $z_{0_i} = \text{LayerNorm}(x_{p_i} + \text{PE}_{2D}(i_x, i_y))$ - Self-attention per Transformer layer: $$ Q_i, K_i, V_i = z_{\ell-1} W_q,\, z_{\ell-1} W_k,\, z_{\ell-1} W_v $$ $$ A_i = \text{Softmax}\left( \frac{Q_i K_i^T}{\sqrt{d}} \right) V_i $$ $$ z_\ell = \text{FFN}(\text{LayerNorm}(A + z_{\ell-1})) $$ ## 2. Pretraining Objective and Fine-Grained Supervision Penguin-Encoder departs from contrastive pretraining, instead adopting a two-stage procedure: 1. **Stage 1 (Low-res, 2048 tokens):** Mixed caption alignment with feature reconstruction; the latter is distillation from a SigLIP-trained teacher. 2. **Stage 2 (High-res, 10,240 tokens):** Pure caption alignment using cross-entropy against re-captioned ground truth. The initialization loss combines: - **Amplitude loss:** $L_A = \frac{1}{N} \sum_n \| F_s^{(n)} - F_t^{(n)} \|_1$ - **Direction (cosine) loss:** $L_D = \frac{1}{N} \sum_n \mathrm{tr}\left(F_s F_t^T /(\|F_s\|_2 \cdot \|F_t\|_2)\right)$ - **Relation loss:** $L_R = \frac{1}{N} \sum_n \| (F_s F_s^T / \|F_s\|_2^2) - (F_t F_t^T / \|F_t\|_2^2) \|_1$ Full objective: $$ L_\text{stage1} = L_{\text{CE}}(\text{text-caption}) + \lambda_A L_A + \lambda_D L_D + \lambda_R L_R $$ This configuration preserves both global and fine-grained spatial/temporal information, unlike contrastive approaches that operate only on global summaries (e.g., [CLS] tokens). ## 3. Integration in Vision-Language Modeling Pipelines Penguin-Encoder outputs a sequence of visual features projected via a 2-layer MLP (with GELU activation) to the LLM's hidden size. The resulting tokens are merged—by concatenation or interleaving—with text tokens and fed into the decoder’s cross-attention layers. At each cross-attention layer, the operation is: - $Q = X_v W_q$ - $K = X_t W_k$ - $V = X_t W_v$ - $\text{Attention}(X_v, X_t) = \text{Softmax}\left(\frac{Q K^T}{\sqrt{d}}\right) V$ with $X_v \in \mathbb{R}^{T_v \times d}$ (visual), $X_t \in \mathbb{R}^{T_t \times d}$ (textual+visual). ## 4. Benchmarking, Ablation, and Efficiency Analysis Penguin-VL employing the Penguin-Encoder consistently matches or surpasses leading VLMs (e.g., Qwen3-VL, InternVL-3.5, GPT-5 nano) despite utilizing a far smaller dataset and encoder parameter budget. Notably, Penguin-VL 2B outperforms Qwen3-2B, InternVL-2B, Gemma3n-2B, and SmolVLM2 on chart/OCR/document and dense reasoning benchmarks by 5–15 points in accuracy. At 8B scale, Penguin-Encoder surpasses or ties state-of-the-art across DocVQA, ChartQA, AI2D, LongVideoBench, and NextQA with only 240 M pretraining samples—versus >40 B for SigLIP2-based pipelines. Ablation studies reveal: - Omission of relation loss ($L_R$) lowers performance by 1.3 points vs. full objective. - Removing the reconstruction loss further degrades accuracy. - Replacing contrastive-pretrained encoders with Penguin-Encoder leads to consistent improvements in tasks requiring spatial and temporal fine granularity. Its lightweight structure (400 M vision encoder params; compatible with 2B–8B decoders) enables deployment on mobile and edge hardware with reduced latency. ## 5. Penguin-Encoder in Partially Equivariant Graph Neural Networks The Penguin-Encoder is also central in the PEnGUiN framework [2503.15615] for Multi-Agent Reinforcement Learning, where strict global equivariance is rarely attainable. The architecture formally interpolates between symmetric (EGNN/E2GN2) and standard non-equivariant GNNs, allowing for partial equivariance at various granularities: - **Subgroup Equivariance:** Equivariance is enforced for a subgroup $H \subseteq G$; $f$ is $H$-equivariant iff $\forall h \in H: f(T_h x) = L_h f(x)$. - **Feature-wise Equivariance:** Only specific input/output coordinates transform under the symmetry group. - **Regional Equivariance:** Equivariance holds only for a subset $S$ of the input space, formalized via an error $\epsilon(x)$. - **Approximate Equivariance:** $f$ is $\epsilon$-approximately equivariant if deviations are bounded by $\epsilon$ for all group actions and inputs. ### Layer Formulation and Forward Pass At each layer $l$: - Each node $i$ carries invariant features $h_i^l$ and equivariant coordinates $x_i^l$. - Equivariant and non-equivariant message branches are computed: - $m_{ij}^{(eq)} = \varphi_m(h_i^l, h_j^l, \|x_i^l - x_j^l\|^2)$ - $m_{ij}^{(ne)} = \varphi_n(h_i^l, h_j^l, x_i^l, x_j^l)$ - Messages are aggregated and blended using a learnable symmetry score $\alpha_i^l \in [0, 1]$: $M_i^l = \alpha_i^l M_i^{(eq)} + (1 - \alpha_i^l) M_i^{(ne)}$ - The invariant feature and a non-equivariant coordinate increment are computed: $[h_i^{l+1} \| x_i^p] = \varphi_h(h_i^l, M_i^l)$ - The EGNN-style coordinate update: $x_{i,eq}^{l+1} = x_i^l + x_i^l \cdot \varphi_e(M_i^l) + \sum_{j \neq i}(x_i^l - x_j^l)\cdot \varphi_u(m_{ij}^{(eq)})$ - The new coordinates are then: $x_i^{l+1} = \alpha_i^l x_{i,eq}^{l+1} + (1 - \alpha_i^l)x_i^p$ **Special Cases:** Setting $\alpha_i^l\equiv 1$ recovers standard EGNN/E2GN2 equivariant layers; setting $\alpha_i^l\equiv 0$ yields an ordinary GNN layer. High-level forward-pass pseudocode: ```python Input: Graph G=(V,E) with (h_i^0,x_i^0) for l in 0..L-1: for each directed edge i→j: m_eq[i,j] = φ_m(h_i^l, h_j^l, ||x_i^l - x_j^l||^2) m_ne[i,j] = φ_n(h_i^l, h_j^l, x_i^l, x_j^l) for each node i: M_eq = sum_j≠i m_eq[i,j] M_ne = sum_j≠i m_ne[i,j] α_i = sigmoid(φ_α(h_i^l, x_i^l)) M_i = α_i * M_eq + (1-α_i) * M_ne [h_i^{l+1} || x_i^p] = φ_h(h_i^l, M_i) x_{i,eq} = x_i^l + x_i^l * φ_e(M_i) + sum_j≠i (x_i^l - x_j^l) * φ_u(m_eq[i,j]) x_i^{l+1} = α_i * x_{i,eq} + (1-α_i) * x_i^p ``` ## 6. Empirical Performance and Integration Experiments in [2503.15615] demonstrate that PPO+Penguin-Encoder outperforms standard EGNNs and GNNs across partially equivariant multi-agent environments (MPE, Highway-Env). On “Spread + Bias”, PEnGUiN achieves 80% of asymptotic performance in 50k samples, while EGNN requires 200k and GNN never exceeds 60%. The symmetry score $\alpha$ adapts regionally, reducing in high-asymmetry zones and increasing in symmetric regions. Integration is straightforward for MARL pipelines: replacing the encoder with PEnGUiN and attaching policy/value heads to the outputs. Computational complexity per layer is $O(N^2 \cdot \max\{h, m, n\})$, with real-time performance for moderate agent counts ($N \lesssim 20$). Extensions include hierarchical coarsening and group-generalization. ## 7. Significance and Impact The Penguin-Encoder in vision-language modeling [2603.06569] demonstrates that initializing vision encoders from text-only LLMs, together with fine-grained reconstruction alignment objectives, allows compact VLMs (2B–8B) to match or surpass the performance of much larger, contrastively pretrained encoders. This addresses the data inefficiency and lack of dense spatial supervision prevalent in global contrastive methods. In MARL, PEnGUiN [2503.15615] establishes a mathematically principled, sample-efficient architecture that learns to interpolate between full equivariance and unconstrained representation, achieving improved robustness and generalization under real-world asymmetries. The introduction of adjustable partial equivariance provides the flexibility required for challenging, non-ideal environments encountered in practical multi-agent systems. **References:** - "Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders" [2603.06569] - "PEnGUiN: Partially Equivariant Graph NeUral Networks for Sample Efficient MARL" [2503.15615] Source: https://www.emergentmind.com/topics/penguin-encoder