Parcae: Looped ML & Categorical Semantics

• Parcae is a family of systems that spans looped language models, categorical automata semantics, and liveput-optimized distributed training.
• It implements a recurrent block within Transformer layers to scale computational use without increasing memory or parameter overhead.
• It leverages formal stability constraints and proactive scheduling to achieve significant performance gains in neural network training.

Parcae denotes a family of modern concepts and systems distinguished in machine learning, categorical semantics, and distributed deep learning: (1) stable looped language architectures enabling efficient scaling of neural networks; (2) categorical models encoding parity acceptance conditions via alternating fixed points; and (3) a system for proactively optimizing DNN training throughput on preemptible cloud instances. Each instantiation advances its respective domain through novel algorithmic strategies, mathematical formalisms, and empirical performance gains.

1. Parcae: Stable Looped LLM Architecture

Parcae redefines the scaling laws of LLMs by employing looped architectures, in contrast to conventional fixed-depth Transformers. Instead of linearly scaling depth (and parameter count) to increase training FLOPs, Parcae “loops” a sub-stack of Transformer layers (the recurrent block $R$) for $T$ iterations. This approach scales computational usage without a commensurate increase in memory or parameter footprint (Prairie et al., 14 Apr 2026).

Architectural Organization

• Prelude $P$: Embeds input sequence $s \in V^n$ via $\mathcal{P}(s)\in\mathbb{R}^{n\times d_h}$.
• Recurrent Block $R$: $L_\mathcal{R}$ Transformer layers executed $T$ times, updating hidden state $h_t$.
• Coda $C$: Projects the terminal state $T$0 to vocabulary logits.
• Forward Pass:

1. $T$1
2. $T$2
3. Loop: $T$3
4. $T$4

Residual Stream and Dynamical System

The update reduces to a nonlinear, time-variant dynamical system:

$T$5

bridging a formal link between looped neural operations and iterative dynamical systems.

2. Stability Analysis of Looped Architectures

A key challenge in looped architectures is stability: unconstrained input injection matrices $T$6 can yield spectral norms $T$7, causing residual explosion and divergence.

Negative-Diagonal Parameterization

Parcae enforces stability by parameterizing $T$8 as a negative diagonal:

$T$9

Discretization yields:

$P$0

ensuring $P$1 and contractive updates at each loop iteration.

Empirical Stability

Empirical results demonstrate up to $P$2 lower validation perplexity relative to previous looped models. Normalizing the injected embedding $P$3 via LayerNorm is critical to prevent loss spikes during late training stages. Sampling per-sequence loop depths from a Poisson distribution further stabilizes training by reducing variance and eliminating spikes.

3. Scaling Laws for Training and Inference

Parcae introduces looping as an independent scaling axis $P$4 alongside parameter count $P$5 and data size $P$6, enabling predictable, power-law scaling at fixed FLOP budget:

$P$7

Optimal training, given fixed compute $P$8, is achieved by increasing loop count and data size in tandem. Validation loss at inference decays exponentially with loop count $P$9:

$s \in V^n$0

Establishing a strict compute–quality tradeoff curve for test-time scaling.

Empirical Quality

At 1.3B parameters, Parcae exceeds a parameter- and data-matched Transformer baseline by $s \in V^n$1 (CORE) and $s \in V^n$2 (CORE-Extended), reaching up to $s \in V^n$3 of the performance of a Transformer double its size.

4. Parcae in Category Theory: Alternating Fixed Points and Parity Conditions

In categorical semantics, “Parcae” formalizes parity acceptance via alternating fixed points of functors (Urabe et al., 2018). This construction enables parity-decorated data types and a coalgebraic trace semantics encoding acceptance conditions for infinite-state systems.

Alternating Fixed Points

Given a functor $s \in V^n$4 with both initial algebras and final coalgebras:

• $s \in V^n$5 (inductive step): Carrier of the initial algebra $s \in V^n$6.
• $s \in V^n$7 (coinductive step): Carrier of the final coalgebra $s \in V^n$8. Layers are built by alternately composing $s \in V^n$9 and $\mathcal{P}(s)\in\mathbb{R}^{n\times d_h}$0, yielding functors $\mathcal{P}(s)\in\mathbb{R}^{n\times d_h}$1 for $\mathcal{P}(s)\in\mathbb{R}^{n\times d_h}$2 that switch between induction and coinduction by parity.

Parity Trace Semantics

The main theorem asserts that flattening the decorated trace semantics $\mathcal{P}(s)\in\mathbb{R}^{n\times d_h}$3 recovers the standard coalgebraic trace semantics via canonical projection:

$\mathcal{P}(s)\in\mathbb{R}^{n\times d_h}$4

This construction substantiates the separation of system “behavior” from state-level automaton definitions in parity acceptance.

5. Parcae for Liveput-Optimized Distributed DNN Training

Parcae also denotes a system for robust DNN training on preemptible cloud instances, maximizing expected throughput—termed liveput—under arbitrary preemption patterns (Duan et al., 2024). In contrast to traditional reactive methods, Parcae proactively adapts training parallelism ahead of predicted preemptions.

System Overview

• ParcaeScheduler: Runs on a stable node; collects availability signals, forecasts future GPU counts via ARIMA, and solves the liveput optimization problem.
• ParcaeAgent: Executes model training, migration, and reorganizations as instructed.
• ParcaePS: A checkpointing fallback, only used for full pipeline stage loss.

Liveput Metric

Liveput quantifies the expected throughput of a (D,P)-parallel configuration given the preemption pattern distribution $\mathcal{P}(s)\in\mathbb{R}^{n\times d_h}$5. The liveput optimizer selects sequences of training configurations over planning windows to maximize cumulative liveput, accounting for migration overheads.

Lightweight Live Migration

Migration is structured in three levels:

• Intra-stage: Rewires communication with zero parameter movement if replicated.
• Inter-stage: Transfers stage parameters via GPU peer-to-peer.
• Pipeline migration: Full rebalance with model partition broadcasts, amortized by infrequency.

Empirical Performance

In dense preemption scenarios (e.g., on GPT-3), Parcae surpasses Varuna by up to $\mathcal{P}(s)\in\mathbb{R}^{n\times d_h}$6 and Bamboo by $\mathcal{P}(s)\in\mathbb{R}^{n\times d_h}$7, delivers $\mathcal{P}(s)\in\mathbb{R}^{n\times d_h}$8 average improvement in throughput over Varuna, and achieves $\mathcal{P}(s)\in\mathbb{R}^{n\times d_h}$9 of clairvoyant-oracle liveput. Spot-instance costs under Parcae average $R$0 lower than on-demand training.

6. Implementation Guidelines and Best Practices

For the looped architecture:

• Choose loop count $R$1 according to $R$2 for training FLOP budget $R$3.
• Normalize input injection $R$4 using LayerNorm.
• Constrain injection matrices to ensure $R$5 through negative-diagonal parameterization.
• Sample per-sequence loop depths to reduce gradient variance and loss spikes.
• For inference, apply the exponential law to select $R$6 for compute–quality balancing.

For the liveput-optimized training system:

• Employ ARIMA models for availability prediction.
• Solve the DP-based liveput maximization problem, considering both performance and migration cost.
• Apply migration strategies proactively based on predictive scheduling.

7. Summary and Cross-Disciplinary Connections

Parcae, across its architectural, categorical, and systems-level instantiations, embodies advances in stability theory for neural network architectures, categorical formalization of automata-theoretic concepts, and robust large-scale model training under resource uncertainty. The common thread is a principled approach to optimization—whether spectral norm constraints, functorial alternation, or liveput maximization—anchored in formal analysis and empirical validation (Prairie et al., 14 Apr 2026, Urabe et al., 2018, Duan et al., 2024).