The Geometry of Sequential Learning: Lie-Bracket Prediction of Transfer Order

Abstract: Sequential learning is order-dependent: from Pile-style next-token domain adaptation to instruction-SFT and DPO, N candidate sources induce N! possible curricula. We show that the local order effect is governed by a computable geometric quantity, the Lie-bracket commutator of gradient update fields, yielding a pairwise score for whether A->B or B->A is better for a target domain. The pairwise bracket primitive also defines a Lie-Bracket Tournament: with a shared theta_0 target-gradient reference, Hessian symmetry gives Borda/row-sum scores from one Hessian-vector product per source, O(N) dot products, and an O(N log N) sort, without materializing the O(N^2) edge matrix. Empirically, the planner reaches 98.1%/98.9% pairwise accuracy at k=1 for instruction-SFT/DPO, remains at 73.1%/72.2% at k=20, and preserves the original pretraining-domain evidence with 82.4-92.0% accuracy across four LLMs and 91.1% on diffusion. At curriculum scale, it recovers the best of all 3! schedules in 87.5% of trials, ranks 85 Stack programming-language source domains for a Python target in the 99th sampled percentile, and reaches the 99.0-99.6th sampled percentile on 56 MMLU subjects, sharply above the reported descending gradient-norm baseline. These results reframe sequential learning as a geometric tournament problem: commutators provide both local pairwise order information and a scalable primitive for many-domain schedules.

• The paper introduces a Lie-bracket geometric framework that predicts transfer order in sequential domain adaptation by linking gradient commutators to downstream loss reduction.
• The authors validate their approach with strong empirical results across large language models and diffusion models, achieving high sign accuracy and significant regret reduction.
• The study proposes scalable pairwise and tournament scheduling algorithms that efficiently optimize curriculum learning without brute-force iterations.

Geometric Structure of Sequential Learning: Lie-Bracket-Based Transfer-Order Prediction

Introduction

The paper "The Geometry of Sequential Learning: Lie-Bracket Prediction of Transfer Order" (2606.24993) introduces a geometric formalism for predicting and optimizing the order of sequential domain adaptation in gradient-based learning pipelines. The core contribution is an actionable, theoretical framework based on Lie-bracket commutators of gradient update fields, connecting operator non-commutativity in parameter space to order-sensitive transfer performance. This approach yields a scalable, computable pairwise primitive that generalizes to many-domain curriculum scheduling, empirically validated across LLMs in instruction-SFT, DPO, next-token pretraining-domain adaptation, and diffusion models.

Commutator Theory and Order Dependence

At the heart of the analysis is the observation that gradient update operators rarely commute across domains; sequential adaptation on domains $A$ then $B$ is not equivalent to $B$ then $A$, leading to distinct downstream losses on a target domain $E$. For two domains $A$ and $B$, the paper shows that the order-induced parameter displacement is dominated by a second-order commutator term:

$$\theta_{AB} - \theta_{BA} = \eta^2 (H_B g_A - H_A g_B) + O(\eta^3)$$

where $g_D$ and $H_D$ denote the gradient and Hessian of the loss with respect to domain $B$0, and $B$1 is the learning rate. The Lie-bracket vector $B$2 quantifies the non-commutativity between $B$3 and $B$4.

Projecting this displacement onto the target gradient $B$5 yields the directional score $B$6, whose sign predicts which domain order minimizes target loss. The paper also defines a scale-free cosine score for confidence gating and effect magnitude estimation.

Figure 1: $B$7 scaling linearly with $B$8 on Llama-3.2-1B, validating the theoretical second-order expansion.

Practical Algorithm: Pairwise and Tournament Scheduling

The authors formalize a pairwise order-selection algorithm leveraging the bracket primitive and introduce deployable estimators based on both the initial and drift-matched reference points in parameter space. This yields accurate, cost-effective order prediction without materializing full training for both orders, relying on gradient and Hessian-vector products computed in a trainable parameter subspace.

Extending beyond the pairwise case, the framework defines a Lie-Bracket Tournament: for $B$9 candidate source domains, each pairwise bracket forms a directed edge weighted by the bracket projection onto the target. Aggregating these edges via Borda/row-sum scoring enables scalable curriculum ranking with computation linear in $B$0—one HVP per source and an $B$1 sort—without evaluating all $B$2 orders. The shared-gradient reference point for large-$B$3 settings ensures independence from specific pairs, facilitating efficient scheduling.

Empirical Results and Numerical Claims

Strong empirical evidence is provided for both the bracket-based pairwise planner and tournament scheduler across diverse architectures and adaptation regimes.

• Instruction-SFT (Qwen2.5-1.5B): 98.1% sign accuracy at $B$4 steps, with accuracy remaining at 73.1% for $B$5. Spearman correlation of order ranking is 0.940; regret reduction exceeds 58%.
• DPO/Offline Preference Tuning: 98.9% sign accuracy at $B$6 and 72.2% at $B$7; 97.8% schedule recovery.
• Next-token Pretraining-Domain Adaptation (LLMs): Bracket-based prediction achieves 82.4%–92.0% sign accuracy across models, with accuracy on high-impact decisions exceeding 95%.
• Diffusion Adaptation (DDPM UNet): 91.1% accuracy, 100% on top-quartile decision impact.
• Large-$B$8 Lie-Bracket Tournament: Recovers best of all $B$9 schedules in 87.5% of trials; achieves 99th percentile for 85 Stack programming-language domains targeting Python, and 99.0–99.6th percentile on 56 MMLU subjects, dramatically outperforming gradient-norm baselines (which fall below the first percentile).

Additional results demonstrate the robustness of the approach:

• Long-horizon validation: With predictor fixed at $A$0, accuracy remains 81.5% at $A$1 and 65.3% at $A$2 for Llama-3.2-1B.
• Optimizer scope: The bracket signal persists with memoryless SGD; for AdamW, bracket-based prediction remains modestly informative but future extensions must augment the dynamical state.
• Subspace selection and HVP precision: Restricting to final-layer or convolutional subsets maintains accuracy, validating parameter-efficient tuning; finite-difference HVPs degrade accuracy by 4–12pp, confirming the necessity of precise curvature.

Theoretical Implications

The commutator perspective reframes sequential learning as a geometric tournament problem. The Baker–Campbell–Hausdorff expansion's leading-order term governs order dependence, and the Lie bracket projects this geometry onto downstream task loss. This deterministic relationship between operator composition and transfer efficacy provides a principled foundation for curriculum learning and transfer scheduling in deep learning.

The pairwise geometric primitive composes into scalable sequence-level optimization. The independence of tournament edges from $A$3 ensures that order selection remains tractable as domain cardinality grows, unlike combinatorial search.

Practical Impact and Future Directions

The planner reduces compute by obviating the need for brute-force curriculum ablations—at practical horizons ($A$4), planning is already faster than testing both orders, yielding significant savings in resource-constrained environments. Stakes/confidence scores allow for principled deployment and gating of the planner.

Capability preservation is inherently handled: assigning a protected capability as the target allows the same method to estimate loss impact and prioritize rehearsal or safety data.

The paper suggests further directions:

• Augmented-state commutators: Extending the theory to stateful optimizers (e.g., AdamW, RMSprop) via commutators on $A$5.
• Joint optimization paths: Exploring higher-order trajectory corrections in long-horizon adaptation.
• Generalization to broader classes of update operators: Including adapters, LoRA, and architectures beyond transformers and diffusion models.

Conclusion

The Lie-bracket geometric framework operationalizes sequential learning order selection with strong empirical and theoretical backing. The pairwise commutator serves as a robust, scalable predictor of transfer order, while tournament aggregation enables curriculum learning over many domains. This approach advances both the foundation and practical methodology of sequential adaptation, providing a principled path away from combinatorial search towards geometric understanding and optimization.