Git-Theta: Fine-Grained Model Versioning

• Git-Theta is an extension of Git that decomposes neural network checkpoints into named parameter groups, enabling fine-grained version control.
• It uses delta-encoding and custom Git hooks (clean/smudge, diff, merge) to efficiently track, compare, and merge model updates at a granular level.
• Empirical workflows show significant storage savings and faster collaboration, underpinned by a modular plug-in system and scalable conflict resolution.

Git-Theta is an extension of Git designed for collaborative and continual development of machine learning models, allowing fine-grained version control at the parameter tensor level. Unlike conventional workflows where model checkpoints are stored and tracked as opaque blobs, Git-Theta decomposes checkpoints into named parameter groups, employs communication- and storage-efficient delta-encoding, and supports sophisticated comparison, merging, and reporting mechanisms directly integrated into the Git ecosystem (Kandpal et al., 2023).

1. Motivation and Conceptual Overview

Traditional machine learning model development is typically managed by centralized teams with infrequent updates to pre-trained models. In contrast, open-source software leverages distributed, iterative contributions via version control systems such as Git. Git-Theta enables an analogous paradigm for model artifacts by extending Git’s primitives to the structured, high-dimensional data encountered in neural network checkpoints. This is achieved by organizing checkpoints into parameter groups (e.g., weight matrices or bias vectors), and tracking modifications at this fine granularity. The result is a system that supports:

• Delta-encoding for efficient storage and data transfer
• Parameter-level merging with domain-specific conflict resolution
• Informative change reporting for model introspection
• Extension via a plug-in architecture for new checkpoint formats and update schemes (Kandpal et al., 2023)

2. Architecture and System Integration

Git-Theta integrates with the Git workflow through custom drivers and hooks registered in .gitattributes and Git’s hook system:

• Clean/Smudge Filters (filter=theta): On file addition, the clean filter loads a checkpoint, decomposes it via a Checkpoint plug-in into parametrized groups, and computes update objects using Update plug-ins, serializing and delegating storage of deltas to Git LFS. A .theta metadata file records per-group metadata, hashes, update types, and LFS pointers.
• Diff Driver (diff=theta): Provides checkpoint-aware diffs by comparing metadata files, highlighting added, removed, or modified parameter groups with quantitative statistics.
• Merge Driver (merge=theta): Enables intelligent, parameter-level three-way merges with multiple conflict resolution strategies.
• Custom Hooks: Repository-level pre-push and post-commit hooks handle LFS object tracking and push optimization.

Upon checkout, merge, or fetch, the smudge filter reconstructs full checkpoints from deltas and metadata. This workflow allows efficient versioning and merging, while maintaining workflow compatibility for code and other artifacts (Kandpal et al., 2023).

3. Checkpoint Decomposition and Update Encodings

• Parameter Groups: Each checkpoint is decomposed into named tensors $p_1, \dots, p_m$ (parameter groups).
• Change Detection: Briefly, equality of versions $p_\text{old}$ and $p_\text{new}$ is determined via locality-sensitive hashing (LSH), with guarantees tuned to $L_2$-close parameters: if $LSH(p_\text{old}) = LSH(p_\text{new})$, then $\|p_\text{old} - p_\text{new}\|_2 < \epsilon$.
• Update Objects: For each detected change, different delta encodings are available:
  • Dense update: $\Delta = p_\text{new}$ (full tensor)
  • Sparse update: Encodes index set $I = \{i : (p_\text{new} - p_\text{old})_i \neq 0\}$ and corresponding values $V = (p_\text{new} - p_\text{old})_I$
  • Low-rank (LoRA) update: $\Delta \approx U V^\top$ with $\mathrm{rank}\,r \ll \min\dim$
  • IA³ update: Per-layer scaling vectors

The choice among update types can be guided by $L_0$ or $L_1$ diff metrics, e.g., $$\Delta_{\text{size}}(p) = \|(p_\text{new} - p_\text{old})\|_0$$ and $$\Delta_{\text{size},1}(p) = \|(p_\text{new} - p_\text{old})\|_1$$ (Kandpal et al., 2023).

4. Parameter-level Merge Algorithm

Git-Theta employs a parameter-level three-way merge for each parameter group across the base (B), "ours" (L), and "theirs" (R) versions. The merge logic is as follows:

for each p in union(names(B), names(L), names(R)): load hashes h_B, h_L, h_R from metadata if h_L == h_B and h_R == h_B: choose p = p_B # unchanged elif h_L == h_B and h_R != h_B: choose p = p_R # only theirs changed elif h_L != h_B and h_R == h_B: choose p = p_L # only ours changed else: # conflict present merge-strategy menu: - "ours": p = p_L - "theirs": p = p_R - "base": p = p_B - "average": p = (p_L + p_R)/2 load the chosen p assemble merged checkpoint

This algorithm is $O(n)$ in the parameter block size for equality checks and arithmetic, with total complexity $O(\sum_i \dim(p_i))$ per merge (Kandpal et al., 2023).

5. Reporting, Diff, and Introspection Tools

The git diff operation is extended to operate over checkpoint metadata. Output includes:

• Added groups: Present in new, absent in old
• Removed groups: Present in old, absent in new
• Modified groups: $LSH(p_\text{old}) \neq LSH(p_\text{new})$

For each modified parameter, summary statistics are provided:

• $$Δ_{\text{norm},1}(p) = \|p_\text{new} - p_\text{old}\|_1$$
• $$Δ_{\text{norm},2}(p) = \|p_\text{new} - p_\text{old}\|_2$$
• $$Δ_{\text{frac}}(p) = \|p_\text{new} - p_\text{old}\|_0 / \dim(p)$$

These tools enable detailed layer-wise change analysis and facilitate model evolution tracking (Kandpal et al., 2023).

6. Plug-in System and Extensibility

Git-Theta implements an inversion-of-control plug-in mechanism where the core system determines when plug-ins are invoked and plug-ins define how functionality is performed. Registration uses Python package entry points. Major interfaces:

Plug-in Type	Methods	Description
Checkpoint	load(path) → dict; assemble(dict, path_out)	Manipulate checkpoint files and convert to/from tensors
Update	compute(old, new) → UpdateObject; apply(base, upd) → tensor	Compute/apply parameter deltas
Serializer	serialize(UpdateObject) → bytes; deserialize(bytes) → UpdateObject	Storage serialization/deserialization
Merge	merge(base, ours, theirs) → merged tensor; metadata	Custom merge logic and metadata for resolution

Supported update plug-ins include LoRA (computing a rank-$r$ factorization for efficient updates) and IA³, among others. This architecture facilitates rapid adaptation to novel checkpoint formats and parameter update schemes (Kandpal et al., 2023).

7. Empirical Workflow and Performance

A representative multi-branch collaborative workflow with the T0 3B model illustrates Git-Theta’s benefits:

1. Track and commit model: git theta track t0_3b.pt
2. Branch for LoRA training: commit LoRA-updated checkpoint
3. Branch for separate fine-tuning tasks: commit updated checkpoints
4. Mainline fine-tuning: merge updates
5. Merge with parameter averaging (user-selectable strategy)
6. Further modifications and commits

Empirical results demonstrate:

• Storage Efficiency: After LoRA training, Git LFS required 11.4 GB versus 0.27 GB for Git-Theta (97.6% reduction). Final history totaled 57.0 GB (Git LFS) versus 41.5 GB (Git-Theta).
• Performance Tracking: On the RTE task, model accuracy progressed from ~76.4% (base) to 75.9% (after LoRA-CB), and 77.3% (after merging ANLI+RTE branches).
• Scalability: Checkout and addition times remain practical due to internal parallelism; space savings peak for sparse/low-rank deltas but remain positive for dense updates because of compression.

These results indicate substantial improvements in both disk utilization and collaborative workflow flexibility compared to treating checkpoints as undifferentiated blobs (Kandpal et al., 2023).

---

Git-Theta operationalizes proven version control principles for machine learning checkpoints, exposing the full structure of modern models to collaborative, distributed development. The system leverages modularity, extensibility, and computationally efficient mechanisms to enable open participation and continual improvement in model repositories (Kandpal et al., 2023).