RepoMark: Code Usage Audit Framework

• RepoMark is a data usage auditing framework that embeds semantically invariant marks in open-source code to verify its use in LLM training.
• It employs an m-variant marking method through controlled variable renaming, ensuring imperceptibility, semantic preservation, and sample efficiency.
• The detection algorithm uses ranking-based hypothesis testing with provable false detection rates, achieving high sensitivity even with few marked files.

RepoMark is a data usage auditing framework designed to enable repository owners to verify—post hoc—whether their open-source code has been used in the training of code LLMs. It addresses critical legal and ethical concerns related to data authorization and open-source license compliance in the era of widespread, large-scale code corpus collection. RepoMark operates by imperceptibly embedding marks into code repositories through semantically invariant transformations, providing rigorous statistical guarantees against false accusations through a ranking-based hypothesis test. Its framework demonstrates high detection power, sample efficiency, semantic preservation, and provable bounds on the false detection rate, marking a substantial advance over previous code auditing approaches (Qu et al., 29 Aug 2025).

1. Motivation, Threat Model, and Auditing Requirements

With the proliferation of code LLMs (e.g., Copilot, CodeWhisperer, StarCoder), vast collections of publicly hosted code—often from platforms such as GitHub—are routinely scraped and used for model training. Developers and rights-holders express significant concern over transparency, data authorization, and compliance with open-source licenses (GPL, MIT, Apache, etc.), especially since model trainers rarely disclose precise data-collection manifests. Under regulatory frameworks analogous to the GDPR, data owners have an entitled interest in verifying whether their data has been used.

RepoMark establishes a threat model in which a repository owner wishes to detect, after the fact, whether their specific code repository was included in an LLM's training set—despite having no a priori information about the model's data sources. The auditing solution must satisfy four core requirements:

1. Semantic Preservation: Marked code variants exactly preserve original behavior and compileability.
2. Imperceptibility: Markings remain undetectable both by human inspection and common automated analyses.
3. Sample Efficiency: Auditing remains effective with as few as ten to fifty code files.
4. Rigorous False Detection Rate (FDR) Guarantee: The probability of false accusation is mathematically bounded (e.g., ≤5%).

2. Data Marking Methodology

RepoMark implements an "m-variant marking" paradigm specialized for source code via controlled variable renaming. For each code file $x$, $m$ semantically equivalent variants $\{x^1, \ldots, x^m\}$ are generated, differing only in locally renamed single-token variable names that preserve semantic behavior. A single index $v \in \{1, \ldots, m\}$ is drawn uniformly at random; only $x^v$ is published, while the others remain private.

The marking workflow proceeds as follows:

• Variable Selection: Parse the code into its abstract syntax tree (AST), then collect all single-token local variables and their positions.
• Predictability Filtering: With access to an "oracle" LLM, compute the logits over the vocabulary at each candidate position. Retain variables whose real name ranks above a threshold $R$ (e.g., $R=500$) in this distribution, favoring positions with many plausible alternatives.
• Mark Embedding: For each retained variable position $i$, construct a candidate window $B_i$ of $m$ possible variable names from the ranked list. Each variant renames all occurrences of the targeted variable to a candidate from $m$0, yielding $m$1 code variants per mark.
• Sparsity Control: Enforce at most one mark per $m$2 lines to limit code surface change.
• Mark Publication: Randomly select one of the variant combinations for public release; the remaining variants are kept for audit detection.

This procedure is formalized in Algorithm 1 of the original work, supporting AST-guided, minimally invasive renamings that are both semantically safe and nearly invisible at the surface-form level.

3. Detection Algorithm and Statistical Foundations

Detection leverages black-box querying of the target LLM, using either logit access or per-variant cross-entropy loss monitoring. At audit time:

• For each file and mark, all $m$3 variant forms (as previously constructed) are submitted as probes.
• For the marked variable position, the cross-entropy loss $m$4 is measured for all $m$5.
• The rank $m$6 of the actually published variant’s loss among its $m$7 alternatives (ascending order) is computed.

Under the null hypothesis $m$8 ("the model was not trained on the marked code"), ranks $m$9 are distributed as $\{x^1, \ldots, x^m\}$0. Under the alternative ($\{x^1, \ldots, x^m\}$1), memorization by the model causes the marked variant to yield a lower loss, biasing ranks toward $\{x^1, \ldots, x^m\}$2. Aggregating across all marks yields a statistic:

$\{x^1, \ldots, x^m\}$3

with $\{x^1, \ldots, x^m\}$4 files and total $\{x^1, \ldots, x^m\}$5 marks. The null distribution of $\{x^1, \ldots, x^m\}$6 has generating function:

$\{x^1, \ldots, x^m\}$7

A decision threshold $\{x^1, \ldots, x^m\}$8 is set to guarantee $\{x^1, \ldots, x^m\}$9 for the desired FDR $v \in \{1, \ldots, m\}$0 (e.g., $v \in \{1, \ldots, m\}$1). Efficient calculation of the null CDF leverages FFT-based convolutions. The detection rule rejects $v \in \{1, \ldots, m\}$2 (declaring “model used marked code”) if $v \in \{1, \ldots, m\}$3.

4. Sample Efficiency and Comparative Performance

The m-variant approach, through statistical aggregation, achieves high sensitivity even with a small number of code files. Because each mark supplies an i.i.d. $v \in \{1, \ldots, m\}$4 signal under the null, the sum $v \in \{1, \ldots, m\}$5 rapidly concentrates around its expectation by the central limit theorem. As little as $v \in \{1, \ldots, m\}$6 marks (e.g., spread across $v \in \{1, \ldots, m\}$7 files) afford strong separation between the null and alternative hypotheses; significant detection power (DSR ≥ 90%) is achieved even for small repositories.

Contrastingly, prior code marking techniques such as CodeMark and standardized membership-inference techniques underperform due to low-power signals, often requiring much larger fractions of marked data or yielding low DSR, especially at modest FDR levels (Qu et al., 29 Aug 2025).

5. Experimental Evaluation

RepoMark’s methodology was evaluated on the following targets:

• Models: CodeParrot-1.5B, StarCoder2-3B, InCoder-6B.
• Datasets: CodeParrot (5.1M Python files), CodeSearchNet (Python/Java), CodeNet (4M+ multi-language submissions).
• Oracle Model: Yi-Coder-1.5B (for mark construction).

Baseline methods included loss-based membership inference, min-k attack, zlib extraction, dataset inference, CodeMark, and a generic LLM marking scheme (Huang et al. CCS '24) employing RepoMark-style marking but differing in detection.

Key metrics included:

• Detection Success Rate (DSR): Proportion of correctly flagged truly-trained repos.
• Theoretical FDR: Provable for RepoMark; empirical for some baselines.
• Code Quality: CodeBLEU, token-level edit distance, perplexity with Qwen2.5-Coder-7B.
• Model Utility Impact: HumanEval Pass@k for models trained on marked vs. unmarked code.

Summary of primary findings (at FDR = 5%):

Method	DSR (N~20 files)	FDR Guarantee	Code Quality Impact
RepoMark	≥90%	Provable	CodeBLEU 0.963; ≥0.99 unmarked; edit distance ≈3.6/100 lines; negligible perplexity change
Huang et al. + RepoMark mark	~53%	Provable	Slight
CodeMark	1–3%	None	Slight
Membership Inference	<20%	None	N/A

RepoMark maintained high DSR and negligible impact on downstream model utility or code quality, demonstrating robustness even on small-scale repositories.

6. Advances, Limitations, and Open Challenges

RepoMark’s principal advances include rigorous FDR control under black-box access, high sample efficiency enabling auditing for individual repositories, and imperceptible marks through minimal variable renaming. Unlike watermark/backdoor marks or prior code marking approaches, RepoMark is robust to most plausible surface-level detection and removal strategies.

However, several practical limitations remain:

• Variable Renaming Attacks: Systematic, aggressive renaming (exceeding 75% of candidate positions) can reduce marks’ detectability, but excessive renaming adversely affects code quality and LLM training utility. Empirically, detection remains robust (>75% DSR) up to this threshold.
• API Access Constraints: Under restrictive inference APIs (e.g., only top-20 tokens accessible), practical deployment requires adjustment (e.g., $v \in \{1, \ldots, m\}$8), modestly increasing detection cost.
• Repository Evolution: Ongoing repository changes require re-marking and version tracking; seamless solutions for dynamic, "online" marking and detection are not yet implemented.
• Very Small Repositories: For repositories with under ten files, DSR drops to ~70%, indicating lower power in the ultra-small regime.

RepoMark constitutes a semantically safe, imperceptible, efficient, and statistically robust framework for code LLM training audit, substantially advancing transparency and rights protection in large-scale code corpus usage (Qu et al., 29 Aug 2025).