PaPaformer: Efficient Parallel Transformer

• PaPaformer is a decoder-only Transformer variant that decomposes high-dimensional inputs into lower-dimensional parallel substreams, enhancing training efficiency.
• The architecture integrates Share-Linear and Gumbel MoE routing strategies to aggregate independent subpath outputs while maintaining strong benchmark performance.
• Its two-phase training process, with independent pretraining on distinct datasets, reduces wall-clock time by 25–35% compared to traditional dense models.

PaPaformer is a decoder-only Transformer architecture variant that enables the rapid and efficient training of LLMs by assembling independently trained, lower-dimensional parallel paths into a single composite model. Its architecture and training methodology are specifically designed to reduce computational overhead and wall-clock time compared to traditional dense models, while maintaining competitive performance on established NLP benchmarks (Tapaninaho et al., 1 Aug 2025).

1. Architectural Design

PaPaformer modifies standard LLaMA-style decoder-only Transformer architectures by introducing "Parallel Layers," in which a high-dimensional input vector $x\in\mathbb{R}^d$ is first projected into $k$ smaller streams $x'\in\mathbb{R}^{d'}$, where $d'=d/k$. Each of these $k$ streams is processed independently by identical subblocks $f_i:\mathbb{R}^{d'}\to\mathbb{R}^{d'}$. The outputs are subsequently recombined into the original dimensionality via either concatenation followed by a single linear projection (Share-Linear) or soft selection through a Gumbel-Softmax router (Gumbel MoE variants).

The Parallel Layer can be formally described as:

• Connection Block: Projects $x$ to $[x_1', ..., x_k']\in\mathbb{R}^{k d'}$.
• Parallel Processing: Applies $f_i$ to each $x_i'$, yielding $k$0 processed substreams.
• Combine/Router Block:
  • Share-Linear: Aggregation via $k$1, where $k$2.
  • Gumbel MoE: Aggregation via $k$3, where $k$4 are soft weights obtained from a Gumbel-Softmax layer.

These Parallel Layers are interleaved with standard Layer Blocks comprising full-dimensional self-attention and feedforward networks, resulting in a stack pattern: Layer Block – Connection – [Parallel Layer]×N – Connection – Layer Block.

2. Two-Phase Training Procedure

PaPaformer is trained in a two-stage process designed to maximize modularity and computational efficiency:

Phase 1 (Path Pretraining):

Each parallel path is trained independently as a small decoder-only model (3 Layer Blocks, hidden size $k$5) on distinct datasets and from random initialization:

• Path_1: Trained on 60% of TinyStories tokens (≈286M tokens), with cross-entropy loss $k$6.
• Path_2: Trained on 60% of OpenMathInstruct-1 (≈124M tokens), same loss/objective.

Both use AdamW (LR $k$7), batch size 32, 2 epochs, and a cosine schedule. Training times are 3.5 h (Path_1) and 1.5 h (Path_2) on a single GPU.

Phase 2 (Composite Pretraining):

The fully parallelized PaPaformer is constructed by:

• Concatenating the embedding matrices $k$8 and final linear outputs.
• Initializing each subpath with corresponding pretrained QKV and FFN weights.
• Jointly training on the remaining 40% of both datasets (477M TinyStories tokens + 206M math tokens).
• Employing a composite objective:

$k$9

with $x'\in\mathbb{R}^{d'}$0.

Phase 2 runs for 4.35 h.

3. Computational Efficiency

PaPaformer leverages its parallel structure to achieve considerable reductions in training time and computational resource usage relative to conventional decoder-only Transformer models. The independent pretraining of each path (either in parallel or staggered) permits greater GPU utilization flexibility and reduces the overall training wall-clock time.

Training Efficiency Table

Model	Stage 1 (h)	Stage 2 (h)	Total (h)	Params (M)
LLaMA_256	–	12.5	12.5	32.0
LLaMA_192	–	11.0	11.0	22.5
Path_1	3.5	–	3.5	13.5
Path_2	1.5	–	1.5	13.5
PaPaformer (k=2)	3.5+1.5	4.35	9.35	28.5
PaPaformer² (staggered)	3.5	4.35	7.85	28.5

Significant findings include that PaPaformer ($x'\in\mathbb{R}^{d'}$1, 28.5M params) completes full training in under 10 hours—a ∼25–35% wall-clock reduction versus the best dense baseline (LLaMA_256, 32M params, 12.5h) for the same token count (Tapaninaho et al., 1 Aug 2025).

4. Empirical Performance on Downstream Tasks

PaPaformer was evaluated using fine-tuning (LoRA rank = 8, α=16, dropout=0.1, LR=1e-3, batch 64, up to 32 epochs) on BLiMP, GLUE, and SuperGLUE benchmarks. Macro-averaged accuracy and task results demonstrate the competitiveness of PaPaformer, especially in the Share-Linear configuration.

Macro-Average Results

Training Data	Model	BLiMP ↑	GLUE ↑	SuperGLUE ↑	Macro-avg ↑
TinyStories only	LLaMA_256	64.30	59.40	55.90	59.85
	LLaMA_192	64.75	59.10	55.20	59.70
	PaPaformer (Gumbel v1)	62.30	57.65	54.50	58.15
	PaPaformer (Gumbel v2)	61.85	58.80	54.75	58.45
	PaPaformer (ShareLin.)	64.15	58.65	56.70	59.85
TinyStories+OpenMath	LLaMA_256	62.55	59.85	55.30	59.25
	LLaMA_192	61.90	58.95	54.70	58.50
	Path_1	60.80	56.95	52.95	56.90
	Path_2	57.80	54.25	53.90	55.30
	PaPaformer (Gumbel v1)	62.75	57.35	54.70	58.25
	PaPaformer (Gumbel v2)	63.60	57.95	55.10	58.90
	PaPaformer (ShareLin.)	63.20	58.60	56.40	59.40

Key observations:

• PaPaformer (Share-Linear) attains macro-averages equivalent to LLaMA_256 despite using ∼10% fewer parameters and ∼25% less training time.
• Both Gumbel MoE variants underperform the Share-Linear approach, indicating that increased routing complexity does not yield superior specialization at this scale.
• Pretraining paths on distinct domains enables the composite model to outperform its components, suggesting effective representation reuse and complementary learning.

5. Modular Customization Capabilities

PaPaformer's architecture readily admits architectural and training customizations, including:

• Path Count ($x'\in\mathbb{R}^{d'}$2): Increasing the number of parallel subpaths to model multimodal or domain-specialized knowledge.
• Subpath Architecture: Divergent $x'\in\mathbb{R}^{d'}$3 blocks employing different attention mechanisms or hidden sizes, facilitating functional specialization.
• Routing Mechanisms: Beyond Gumbel-Softmax, potential alternatives include conditional masking, hard routers, or reinforcement learning-based expert selection.
• Residual and Normalization Schemes: Employing skip-connections and layer normalization across paths to improve gradient flow and subpath specialization.

Empirical evidence indicates that the Share-Linear configuration yields the most robust and stable path utilization at small scales. This suggests that more elaborate routing architectures require substantially greater capacity or tailored regularization to avoid over- or under-specialization.

6. Impact and Future Directions

PaPaformer demonstrates that assembling decoder-only Transformers from lower-dimensional, independently pretrained subpaths achieves substantial gains in computational efficiency and parameter economy while maintaining, or even surpassing, the performance of dense baselines on complex text and reasoning benchmarks. Future explorations include scaling to hundreds of millions or billions of parameters, introducing richer path configurations or routing objectives, and evaluating the architecture on a wider range of tasks and modalities (Tapaninaho et al., 1 Aug 2025). A plausible implication is that as model scales increase, the optimal balance between path specialization and integration complexity may shift, further motivating research into path-wise architectural heterogeneity and advanced routing schemes.