DualFormer Architectures

• DualFormer is a family of transformer architectures that integrates dual reasoning, domain representations, and stratified processing to enhance efficiency and accuracy in ML tasks.
• It employs structured strategies like randomized trace dropping, hierarchical frequency sampling, and dual-path attention to flexibly control model behavior across applications.
• Demonstrated in areas from maze navigation and time series forecasting to computer vision and astrophysics, DualFormer models deliver significant speedups and performance gains.

DualFormer refers to a set of architectures and frameworks across multiple subfields of machine learning that share a common motif: the integration or stratification of dual forms of reasoning, attention, or domain representations within a transformer or transformer-like backbone. This article surveys the principal instantiations and technical foundations of "DualFormer" models for controllable fast and slow reasoning, time-frequency dual-domain sequence modeling, dual-path vision backbones, efficient video recognition, and multimodal astrophysical data integration, as established in recent arXiv literature.

1. Controllable Fast and Slow Reasoning with Dualformer

Originating from human dual-process cognitive theory (System 1 vs. System 2), "Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Traces" designs a transformer that can emulate both rapid, intuitive (System 1) and slow, deliberative (System 2) reasoning within a single neural network (Su et al., 2024).

Technical Implementation

• Architecture: Encoder-decoder transformer. Encoder: T5-style with RoPE; decoder: GPT-style causal transformer. For the 30×30 Maze, 6 encoder and 6 decoder layers, hidden size 64, 3 heads (~15M parameters); no auxiliary modules.
• Training via Randomized Trace Dropping:
  • Inputs: prompt $p$, complete search trace $\tau$ (from A*), solution $s$.
  • Five trace dropping levels $\ell$ define stochastic operators $D_\ell(\cdot)$ on traces, ranging from "no drop" to total drop (solution only), controlling which parts of the reasoning chain are visible at each training step.
  • At each SGD step, $\ell \sim \operatorname{Cat}(p_0,\dots,p_4)$; input is $[\langle bos \rangle, p, D_\ell(\tau)]$; target: generate any remaining trace tokens $D_\ell(\tau)$ then the solution $s$.
  • Trained via autoregressive log-likelihood:

  $$L(\theta) = \mathbb{E}_{(p, \tau, s), \ell}[ -\sum_{t=1}^T \log P_\theta(x_t | x_{<t}) ]$$

• Inference Modes:

1. Fast mode: Prefix prompt with $\tau$0 plan to produce only the solution (System 1).
2. Slow mode: Prefix with $\tau$1 create for full or partial trace + solution (System 2).
3. Auto mode: No explicit prefix; model stochastically selects fast or slow at generation.
4. Partially-dropped traces: Even "slow" outputs can be more concise than full A* replay, improving efficiency.

Empirical Findings

• Maze Navigation (30×30):

  • Slow: 97.6% optimal (vs. Searchformer 93.3%) with 44.5% fewer tokens.
  • Fast: 80% optimal (vs. Solution-Only 30%).
  • Auto: 96.6% optimal; 59.9% fewer tokens on average.
• Sokoban: Slow: 94.5% optimal (vs. 92.9%); Fast: 97.3% (vs. 86.8%).
• Math reasoning (LLMs): Mistral-7B Greedy@1, slow: 16.9%→18.6%; Llama-3-8B: 19.7%→20.5%.

Comparative Results and Insights

• Only the structured randomized-dropping recipe produces true fast/slow trade-offs within a single model.
• Mixing solution-only and complete-trace data without structure fails in either mode.
• Using only partial drops enables trace shortening but not solution-only mode; full-drop is essential for System 1 behavior.

Generalization and Impact

• The randomized dropping strategy can be applied to any reasoning trace with substructure (e.g., algebraic derivations, code execution, logic tableaux).
• Reduces computational cost (up to 60% fewer tokens at inference) without sacrificing answer quality, aiding deployment in latency or compute-constrained scenarios (Su et al., 2024).

2. Dualformer for Time-Frequency Dual Domain Learning in Time Series

"Dualformer: Time-Frequency Dual Domain Learning for Long-term Time Series Forecasting" addresses the limitations of standard transformers in handling high-frequency information, introducing a principled dual-branch framework (Bai et al., 22 Jan 2026).

Core Architecture

• Dual-branch Encoder Layers:
  • Time-domain branch: Input sequence normalized (RevIN), then embedded. Processes local, fine-grained patterns through standard self-attention after selective inverse FFT of frequency slices.
  • Frequency-domain branch: Processes representations in the frequency domain (via FFT), computes autocorrelation (Wiener–Khinchin theorem), aggregates via lags detected as salient.
  • Hierarchical Frequency Sampling (HFS): Each encoder layer receives a distinct frequency band, with high frequencies in shallow layers and low frequencies in deeper ones; sampling ratio $\tau$2 governs overlap/coverage.
• Periodicity-Aware Weighting:
  • For input $\tau$3 with FFT $\tau$4, harmonic energy ratio $\tau$5 adjusts the balance between time and frequency branches:

  $\tau$6 - Theoretical lower bound links higher periodicity to larger $\tau$7, justifying increased frequency emphasis.

Theoretical Foundation

• Deep self-attention exhibits a low-pass bias. HFS ensures high frequencies are preserved, and periodicity-aware weighting dynamically adapts representation emphasis.

Experimental Evaluation

• Datasets: ETTh1/2, ETTm1/2, Electricity, Solar, Traffic, Weather.

• Metrics: Dualformer ranks 1st in 13/16 average MSE cases, 44/64 individual outcomes, improving error by 5–15% over baselines.

• Ablations: Both branches are required for optimality. HFS outperforms fixed-band or random allocations. Weighting improves performance on non-periodic series.

Implementation

• PyTorch, A100 GPU; Adam optimizer, 3 encoder layers, $\tau$8, $\tau$9 heads, $s$0. RevIN used before/after encoder.

Significance

• Substantially improves accuracy and robustness for long-horizon forecasting, especially for heterogeneous and weakly-periodic time series (Bai et al., 22 Jan 2026).

3. Dual-Path and Local-Global Stratified Vision Transformers

Several DualFormer variants have been developed for computer vision, emphasizing duality between local and long-range interactions:

Dual Path Transformer with Partition Attention

"Dual Path Transformer with Partition Attention" presents a vision backbone that processes each feature set in two parallel paths (Jiang et al., 2023):

• MBConv branch: Standard convolutional local pathway for fine-grained, high-frequency features.

• Multi-Head Partition-wise Attention (MHPA): Efficient global context via approximate clustering (LSH), with separate intra- and inter-partition computations for scalable, long-range dependency modeling.

• Dataflow: Both branches run in parallel per block, outputs concatenated and fused.

• Empirical Results: On ImageNet-1K, DualFormer-XS achieves 81.5% top-1 accuracy with 2.3 GFLOPs, outperforming competing architectures of similar size. Similar improvements noted for COCO detection and ADE20K segmentation.

Ablation and Analysis

• Parallel (not serial) fusion of MBConv and partition attention is optimal.

• Including both intra- and inter-partition attention increases accuracy; LSH clustering improves throughput with negligible loss.

Local-Global Stratified Video Transformer

"DualFormer: Local-Global Stratified Transformer for Efficient Video Recognition" factors 3D space-time attention into:

• Local-Window MSA (LW-MSA): Captures local, fine-grained interactions within small, 3D non-overlapping windows.

• Global-Pyramid MSA (GP-MSA): Aggregates multi-scale, global spatiotemporal context using depthwise pooling and dramatically reduced key/value sets.

• Block Complexity: Sequential LW-MSA and GP-MSA yield full receptive field at a fraction of standard self-attention cost.

• Results: On Kinetics-400, DualFormer-B (IN-21K) achieves 82.9% top-1 with 1,072 GFLOPs (at least 3.2× fewer FLOPs than alternatives with similar accuracy) (Liang et al., 2021).

Comparative Table: Vision DualFormer Variants

Model/Domain	Key Dual Mechanism	Accuracy/Leaderboard Highlights
Dual Path (Jiang et al., 2023)	Parallel local MBConv + global MHPA	81.5% top-1 (XS, ImageNet-1K); efficient COCO, ADE20K
Local-Global (Liang et al., 2021)	Cascade of LW-MSA + GP-MSA	82.9% top-1 (K400, IN-21K, Base), low FLOPs

4. DualFormer for Cross-Modal Integration in Astrophysics

Within the DESA (Dual Embedding model for Stellar Astrophysics) framework (Kamai et al., 14 Jul 2025), DualFormer serves as a transformer-based module to align embeddings from photometric light curves and spectroscopic data.

Architectural Features

• Stage 1: Modality-specific Encoders

  • Spectra: CNN backbone + supervised and self-supervised heads (Conformer; MSE, CQR).
  • Light-curves: CNN and Conformer streams; SimSiam-style SSL plus CQR regression.
• Stage 2: DualFormer Alignment
  • Combined self- and cross-attention blocks process pooled encoder outputs from both modalities.
  • Dual linear projections $s$1, $s$2 align the two branches; final embedding in the eigenspace of $s$3.
• Losses
  • Covariance decorrelation (VicReg-style) ensures feature diversity.
  • Alignment via quadratic-form matching (relaxes pointwise invariance).

Empirical Results

• CMD/HR Recovery: Zero/few-shot neural CMD with $s$4 on color-magnitude regression; excellent recovery of HR diagrams and clusters.
• Binary Classification & Age Prediction: AUC = 0.99, AP = 1.00 for binaries; RMSE = 0.94 Gyr for age (baselines yield significantly worse metrics).
• Interpretability: Physically meaningful stellar population clusters emerge in projection eigenspace; separate rotationally synchronized binaries from young stars purely by embedding.

Significance

• Provides a framework for fully integrated, physical latent space representation across stellar photometric and spectroscopic modalities, enabling both high predictive accuracy and unsupervised scientific discovery (Kamai et al., 14 Jul 2025).

5. Comparative Principles and Broader Impacts

A unifying theme across all DualFormer variants is the explicit structuring of model inductive bias to capture complementary patterns—whether that involves reasoning speeds, temporal and frequency domains, spatial context scales, or cross-modal signal structures.

• Structured Data/Cognitive Stratification: Randomized trace dropping (Su et al., 2024) and dual architectural paths (Bai et al., 22 Jan 2026, Jiang et al., 2023) enable models to flexibly "dial up" different reasoning, frequency, or spatial regimes as dictated by the task or input.
• Efficiency and Generalization: DualFormer approaches consistently reduce computational costs compared to baselines, either by fewer tokens/steps (reasoning), reduced complexity (vision), or adaptive feature selection (time series/astrophysics).
• General Applicability: The dualization motif extends to any setting with distinct, complementary substructures, not restricted to the domains described above.

A plausible implication is that the core principles underpinning "DualFormer" architectures—randomized partial information, dual-branch feature pooling, hierarchical stratification, and cross-modal projections—will continue to inform the design of scalable, efficient, and controllable models for hybrid reasoning, perception, and scientific discovery tasks.