GitFarm: Git Execution and Provenance at Scale

• GitFarm is a platform that centralizes remote Git execution for large monorepos by utilizing pre-warmed repositories and secure, ephemeral sandboxes.
• It eliminates the local-clone model, reducing latency from minutes to seconds and substantially cutting resource usage across services.
• The system also supports global repository mining and provenance tracking, enabling scalable, cross-project analysis and deduplication.

GitFarm is a platform for Git as a stateful, identity-scoped, repository-centric execution service that addresses the failure of the traditional “Git on every box” model at large monorepo scale. In the system described by Uber, Git operations are executed remotely through a gRPC API inside secure, ephemeral sandboxes backed by pre-warmed repositories, so clients no longer maintain local clones and instead obtain a ready-to-use checkout in less than a second (Dwivedi et al., 13 Apr 2026). In a broader research usage, “GitFarm” also denotes an architectural pattern for organizing, deduplicating, mining, and versioning large collections of Git repositories and branchable data, including global shared-commit provenance maps, graph-based repository mining, and immutable storage substrates for Git-like data collaboration (Mockus et al., 2020).

1. Definition and historical setting

GitFarm, in its narrow and most explicit sense, is a Git as a Service platform for large-scale monorepos. Uber’s formulation is repository-centric rather than host-centric: backend nodes maintain authoritative, pre-warmed clones of large monorepos, clients invoke Git through RPC, and the service returns command results while preserving the flexibility of native Git semantics (Dwivedi et al., 13 Apr 2026). This model arose from a setting in which large internal automation systems and developer services needed to observe repository changes, inspect commits and code ownership, compute merge bases and rebases, push derived refs or branches, and validate changes for code review and merge queues, yet historically each such system maintained its own full clone and periodically ran git fetch.

A broader, earlier usage treats GitFarm as a research pattern for Internet-scale repository organization. In that sense, the central task is not remote execution over monorepos, but identifying which repositories belong to the same software project, deduplicating clones and forks, and supporting cross-project analysis over infrastructures such as World of Code. Shared commits, graph clustering, graph databases, and immutable versioned substrates become the key ingredients in that interpretation (Mockus et al., 2020). This suggests that the term spans two related but distinct problem domains: operational Git execution at monorepo scale and global-scale Git provenance and mining.

The common denominator is the displacement of raw repository cloning as the primary unit of computation. In the monorepo setting, GitFarm centralizes clones and executes commands remotely. In the mining and provenance setting, GitFarm centralizes repository relationships, object histories, and derived analytics. A plausible implication is that both uses converge on the same architectural thesis: repository state, history, and identity should be maintained as shared infrastructure rather than replicated independently by every client.

2. Monorepo-scale motivation and failure of the local-clone model

Uber’s motivation is explicitly tied to the scale of its monorepos: Go, Java, Web, Python, Android, and iOS repositories are described as multi-GB repositories with millions of files and commits (Dwivedi et al., 13 Apr 2026). In that environment, cloning and maintaining local checkouts became a systems bottleneck rather than a mere developer convenience. The paper gives a concrete example for the Go monorepo: a full clone required ~15 minutes, ~4 CPU cores, ~16 GB RAM, and > 40 GB disk. A single service node cloning all six major monorepos could require 12 CPU cores, 48 GB RAM, and 64 GB disk.

The difficulty was not limited to initial clone latency. The traditional model imposed continuous cost through repeated git clone and git fetch operations, heavy I/O on each host, and large aggregate CPU, memory, and disk consumption across service fleets. On the server side, upstream Git hosts were burdened by object enumeration and pack generation for each client, causing slowdowns and scaling challenges (Dwivedi et al., 13 Apr 2026). The paper characterizes this as the breakdown of the classic “Git on every box” model.

CI systems such as Jenkins and Buildkite partially shifted this burden by acting as execution sites with long-lived checkouts and caches. However, for workflows that were primarily “Git-only,” CI job startup dominated the useful work. Scheduling an agent, preparing a workspace, and syncing a repository often produced 1–3 minute (or more) latency even when the actual Git logic was trivial. Cache misses and cold starts could still exceed 10 minutes, and the underlying repository state remained fragmented across many agents (Dwivedi et al., 13 Apr 2026).

The paper also reviews several known Git scaling techniques and explains why they were insufficient in this setting. Shallow clones reduce initial transfer but break operations that assume full history. Partial clones, sparse checkouts, and VFS for Git (GVFS/Scalar) optimize client throughput and working tree I/O, but each client still independently talks to the Git server and still maintains local repository state. CI caching improves per-agent performance, but state remains per-agent and tied to the job lifecycle (Dwivedi et al., 13 Apr 2026). The resulting design goal for GitFarm was therefore not merely faster cloning, but elimination of local clones for automation clients altogether.

3. Service model, execution API, and backend architecture

GitFarm exposes Git through a gRPC API that executes real git binaries/commands rather than a reduced hosting API (Dwivedi et al., 13 Apr 2026). The core abstraction is a repository-centric session: a client specifies repo_id, workspace_type, and an ordered list of commands, and GitFarm runs those commands sequentially in the same sandbox and checkout. The paper gives the pseudocode API:

Exec(repo_id, workspace_type, commands) → CommandResults Commands: list<Command> Command: alias: string binary: string arguments: list<string> stdin: string (optional) environment: map<string, string> CommandResults: list<CommandResult> CommandResult: alias: string exit_code: int stdout: string stderr: string

This API embodies three properties emphasized by the paper. First, execution is stateful within a session: commands share the same repository state unless the client explicitly runs git fetch. Second, execution is identity-scoped: each request is associated with a client identity, the gateway authorizes access to repo_id, and sandboxes execute with credentials and permissions matching that identity. Third, execution is isolated: each session receives a dedicated sandbox and checkout, preventing cross-tenant leakage (Dwivedi et al., 13 Apr 2026).

The architecture is divided into clients, a GitFarm Gateway, and GitFarm Backend clusters. The gateway authenticates callers, authorizes repository access, and routes requests to an appropriate backend node while maintaining state in Redis. For a given request, it chooses the backend node with the most available warm checkouts for the requested repository and marks a sandbox and checkout as occupied until completion (Dwivedi et al., 13 Apr 2026).

Each backend node maintains one bare clone per monorepo, a pool of working checkouts derived from that bare clone, and a pool of pre-initialized sandbox containers. Bare clones are kept current through event-triggered fetches and a periodic git fetch every 5 minutes as a safety net. Working checkouts are full working trees, kept synchronized with the bare clone, so a request acquires a ready repository rather than materializing one on demand (Dwivedi et al., 13 Apr 2026).

Pooling is the central latency optimization. Without pooling, container startup takes 1–2 seconds and materializing a working checkout from a bare clone can take up to 3 minutes for large monorepos. GitFarm moves both operations off the critical path by maintaining pools of sandboxes and checkouts. In the reported evaluation, each backend node kept 100 sandbox containers, 30 working checkouts for the Go monorepo, and 10 each for Java, Android, Web, Python, iOS (Dwivedi et al., 13 Apr 2026). The system also supports multiple backend clusters, including a shared cluster and specialized clusters, to isolate workloads with different throughput, latency, and reliability requirements.

The session model supports multi-command workflows that are awkward under job-oriented CI systems. The paper’s examples include git fetch, git merge-base, and git push in a single execution context. Because all commands run in one sandbox bound to one checkout, there is no need to re-checkout or re-authenticate between steps (Dwivedi et al., 13 Apr 2026).

4. Performance characteristics and operational effects

The principal performance claim is that GitFarm provides a ready-to-use checkout in less than a second by acquiring a warm sandbox and mounting a pre-existing checkout from the pool (Dwivedi et al., 13 Apr 2026). In the production evaluation on a shared GitFarm cluster with 4 backend nodes, each node had 48 CPU cores, 250 GB RAM, and 1.5 TB local SSD, and the P95 Acquire Sandbox Latency was reported as < 1 second.

The system’s impact is clearest in the case studies. In the compliance auditing for bypassed development workflows use case, the workload was 10k–20k deployment/push events per hour over ~9,000 repositories. The baseline Buildkite implementation had p50 ~110–160 seconds end-to-end latency, dominated by job provisioning, workspace checkout, and repository sync. The GitFarm implementation reduced this to p50 ~20–30 seconds, representing > 80% reduction in latency (Dwivedi et al., 13 Apr 2026).

In the pull request base branch update workflow, the paper reports p50 end-to-end latency: ~25 seconds, with the major components being git fetch: p50 ~5 seconds and git push: p50 ~12 seconds. The conclusion drawn is that GitFarm’s additional layers add negligible overhead relative to the Git operations themselves; the remaining latency is dominated by upstream Git behavior rather than by the service (Dwivedi et al., 13 Apr 2026).

The most substantial resource savings appear in the read-only inspection of code ownership files use case. Before migration, each of 6 hosts maintained full clones of 6 monorepos, with per-host allocations of 12 cores, 48GB RAM, 64GB disk. After migration to GitFarm, hosts no longer kept large Git checkouts and instead used 2 cores, 4GB RAM, 2GB disk. Fleet-wide utilization dropped from ~90.1 cores to 16 cores average (82% reduction) and from 472 GB to 32 GB (93% reduction). Startup time per node improved from 15–20 minutes to < 1 minute (Dwivedi et al., 13 Apr 2026).

The paper also emphasizes operational trade-offs. GitFarm provides eventual consistency with respect to upstream: clients may observe slightly stale state unless they explicitly run git fetch at session start. Sessions are typically short-lived (capped at 5 minutes), and the gateway applies explicit backpressure by rejecting or throttling requests when no node has capacity for a requested repository (Dwivedi et al., 13 Apr 2026). This suggests a deliberate separation between low-latency execution and freshness guarantees: repository warmness is always available, but exact recency is opt-in.

5. Storage, versioning, and branchable-data substrates

Although Uber’s GitFarm paper is focused on monorepo execution, related work on ForkBase describes a storage substrate that closely matches the needs of branchable applications and data-centric Git-like systems (Lin et al., 2020). ForkBase is a distributed, immutable storage engine explicitly designed to be “Git for data”. Its core properties are immutable storage, Git-like branching and forking, tamper-evidence, and fine-grained data collaboration across structured and unstructured data.

ForkBase organizes versions as a Merkle DAG and stores values in POS-Trees (Pattern-Oriented-Split Trees), which function simultaneously as search indexes and Merkle trees. The version derivation graph uses f-nodes that reference value roots, parent versions, and metadata, with version identifiers encoding both content and derivation history. This makes content and history tamper-evident and yields Git-like operations such as Put, Get, Select, Diff, Branch, Merge, Head, and Latest (Lin et al., 2020).

The relevance to GitFarm lies in the shift from file-oriented version control to a storage engine that natively supports structured diffs, structured merges, and fine-grained deduplication. ForkBase’s deduplication occurs at the page/chunk level, not at file level, and the paper’s example reports that for two nearly identical CSV datasets the first dataset consumed +338.54 KB while the second, differing by a single word, consumed only +0.04 KB (Lin et al., 2020). The POS-Tree also supports Diff with complexity $O(D \log N)$, where $N$ is the number of data entries and $D$ the number of differing leaf nodes.

This is not the same system as Uber’s GitFarm, but it illuminates a deeper design axis. GitFarm centralizes Git execution over existing monorepos; ForkBase pushes Git semantics into the storage layer for branchable applications. A plausible implication is that future GitFarm-like platforms may combine both patterns: remote execution for operational Git workflows and immutable, Merkle-indexed storage for structured artifacts, model outputs, or application state (Lin et al., 2020).

A related but object-centric direction appears in GoT, or “Git, but for Objects”, where distributed applications synchronize mutable, long-lived, replicated objects using a version graph, snapshots, diffs, commit, checkout, fetch, push, and merge (Achar et al., 2019). GoT formalizes causal consistency, application-level three-way merges, and version-graph garbage collection. This suggests that the GitFarm idea can be generalized beyond source repositories to replicated object state, provided that history growth, transport cost, and merge semantics are constrained appropriately.

6. Repository provenance, graph mining, and global “GitFarm” infrastructures

A second research lineage treats GitFarm as infrastructure for organizing and analyzing the world of repositories rather than serving Git commands for monorepos. In World of Code, shared commits are used to construct a global map of related repositories because Git commits are based on a Merkle tree and accidental reuse across unrelated projects is negligible (Mockus et al., 2020). Using WoC version Q, the study worked over 1,868,632,121 commits, 116,265,607 projects (repositories), and 99,154,451,345 commit-to-project links.

The initial project graph connected repositories that shared at least one commit and produced 61,921,909 distinct connected components, but also a megacluster of 13,912,612 projects caused by backup repositories, meta-history repositories, and arbitrary pushes or pulls between unrelated repositories (Mockus et al., 2020). To address this, the work applied the Louvain community detection algorithm after collapsing GitHub forks to ultimate parents and collapsing commits with identical project sets into shared hyperedges. The resulting largest groups were dramatically smaller, with the largest listed community containing 354,920 repositories. Evaluation against withheld GitHub fork metadata reported that 98.1% of forks ended up in the same community as their ultimate parent (Mockus et al., 2020).

A later refinement, “Deforking the World of Code”, scales this idea further by building a project-provenance map over WoC V2604, which includes about 5.87B commits and ~269M repositories (Mockus, 28 Jun 2026). Its released p2PFull.V2604.s maps 268,855,224 repositories into 190,273,566 deforked projects using the global shared-commit relation and parallel Louvain clustering over a hub-node star encoding. The paper shows that naive shared-history union still over-merges, producing an uncapped largest cluster of 861,948 repositories, and introduces capped variants such as cap250 and cap500 to remove very large shared-commit groups before clustering.

The cap is not applied to cluster size, but to the size of the evidence group. With C = 250, the largest community falls to 183,654 repos, the number of communities of size ≥100,000 falls from 13 to 1, and the number of families remains essentially unchanged (Mockus, 28 Jun 2026). Validation against GitHub’s declared fork graph reconstructed from GHArchive found 99.01% edge agreement conditional on both repositories being in WoC, and the map also exposed 775,420 families (5.41%) spanning more than one forge and 216,013 families (1.51%) whose fork root is not on GitHub (Mockus, 28 Jun 2026). These results show that commit-based provenance captures relationships that platform-local fork graphs cannot represent.

Graph-based storage and querying complete this picture. GraphRepo stores Git-derived entities in Neo4j, using a generic schema over Developer, Commit, File, Method, and Branch nodes, with relationships such as AUTHOR, PARENT, UPDATE_FILE, UPDATE_METHOD, and METHOD (Serban et al., 2020). Its architecture is modular, with drillers, miners, and mappers, and it supports a single-tenant database with project_id properties for multi-repository analysis. Benchmarks on Hadoop, Jax, Kibana, and Tensorflow show that, once extracted and indexed, common queries execute in tens of milliseconds to a few seconds (Serban et al., 2020).

Taken together, these systems outline a broader encyclopedia-level meaning of GitFarm. Uber’s GitFarm centralizes Git execution for very large monorepos (Dwivedi et al., 13 Apr 2026). ForkBase provides an immutable, tamper-evident storage substrate for branchable data (Lin et al., 2020). World of Code and later deforking work provide the project-provenance layer needed to reason about repository families rather than raw clones (Mockus et al., 2020, Mockus, 28 Jun 2026). GraphRepo provides a graph-native analysis layer for repository mining (Serban et al., 2020). The combined picture is that GitFarm names not only a specific 2026 service, but also a family of infrastructures in which Git history, Git execution, project identity, and Git-derived analytics are elevated to shared systems primitives.