RepoPeftBench: Benchmark for PEFT in Code Repos

• RepoPeftBench is a benchmark that evaluates parameter-efficient fine-tuning methods on entire code repositories.
• It releases full Python repositories with complete Git histories to enable both inference-time context injection and parametric adaptation.
• The evaluation uses metrics like Exact Match, Edit Similarity, and CodeBLEU to assess performance across static and evolution tracks.

RepoPeftBench is a code-repository-level benchmark for evaluating parameter-efficient fine-tuning methods for code LLMs under both stable and evolving software conditions. It was introduced in conjunction with Code2LoRA as a benchmark of 604 public, permissively licensed Python repositories, with two tracks: a static track based on single-snapshot assertion completion and an evolution track based on commit-derived assertion tasks (Hotsko et al., 4 Jun 2026). Its central design choice is to release entire repositories, including non-test source files and full Git histories, so that methods can either inject repository context at inference time or encode repository knowledge parametrically through adaptation mechanisms. This places RepoPeftBench at the intersection of repository-level code understanding, PEFT evaluation, and temporal generalization.

1. Scope, object of evaluation, and nomenclature

RepoPeftBench is designed to evaluate PEFT methods for code LLMs at repository granularity rather than at the level of isolated files or short retrieval snippets. The benchmark explicitly targets settings in which project-specific APIs, imports, naming conventions, and software evolution matter, and it does so by shipping entire Python repositories together with full commit diffs (Hotsko et al., 4 Jun 2026).

The benchmark comprises 604 public, permissively licensed Python repositories, of which 512 are in-distribution and 92 are temporal out-of-distribution. Repository selection is reported as being based on quality criteria including pytest or unittest usage, at least 300 stars, and recent activity. The resulting design probes whether a model can use repository context either through input-side mechanisms such as retrieval and dependency resolution, or through parameter updates such as LoRA and related PEFT strategies.

A recurring source of ambiguity is its name. RepoPeftBench is distinct from PEFT-Bench, the autoregressive-LLM benchmark implemented in the PEFT-Factory repository. PEFT-Bench evaluates diverse PEFT methods across 27 NLP datasets and 6 PEFT methods and introduces the PSCP metric, whereas RepoPeftBench targets repository-level code tasks with Exact Match, Edit Similarity, and CodeBLEU as the reported metrics (Belanec et al., 26 Nov 2025). This suggests that the two efforts are complementary rather than redundant: one is centered on general NLP PEFT benchmarking, the other on repository-conditioned code modeling.

2. Task formulation and corpus design

The core task in RepoPeftBench is assertion completion. Given a test prefix extracted from a test file, the model must predict the assertion’s right-hand side, or the last argument of an assertion call. The benchmark states that the input includes imports, the enclosing class, helper functions such as fixtures, and the test body up to the assertion site, requiring multi-step reasoning over imported definitions and project-specific abstractions (Hotsko et al., 4 Jun 2026).

For the static track, the input is formalized as

$$x = \operatorname{concatenate}(\text{imports};\ \text{enclosing class};\ \text{helpers};\ \text{test body up to “assert …”}),$$

truncated to a maximum length of 2 K tokens in PEFT training, or 8 K tokens if dependency-resolved context is prepended. The output $y$ is the expected literal or expression, specifically the right-hand side of == or the last argument of an assertion call.

Two input variants are defined. The first is the plain snapshot prefix, with an approximately 224-token median. The second is Dependency-Resolved Context, or DRC, which statically resolves imports in the test prefix, extracts reachable function and class definitions, and compresses them under a 4 K-token budget; this is reported as being available for approximately 64% of pairs. The benchmark therefore does not treat repository context as a monolithic object: it exposes both raw test-prefix inputs and structured, dependency-guided context augmentation.

Because every non-test source file is released for repository context, and full commit diffs are released for temporal evaluation, RepoPeftBench supports both inference-time context injection and parameteric knowledge incorporation. Its design explicitly contrasts these approaches rather than assuming one dominates.

3. Static track: single-snapshot assertion completion

The static track evaluates assertion completion on a single repository snapshot. Within the 512 in-distribution repositories, 409 are used for training and in-repo splits, while 103 held-out repositories are reserved for cross-repo validation and test. The benchmark reports 39,612 training instances from the 409 training repositories, approximately 97 tasks per repository. Cross-repo validation and test contain 6,213 and 6,414 tasks from 51 and 52 held-out repositories, respectively, approximately 121 tasks per repository. In-repo validation and test contain 4,833 and 5,222 tasks from the same 409 repositories using an 8:1:1 split of their test tasks, approximately 12 tasks per repository (Hotsko et al., 4 Jun 2026).

At the dataset-wide level, the static benchmark contains 62,294 total question-answer pairs, decomposed into 39,612 training, 11,046 validation, and 11,636 test instances. A temporal out-of-distribution set of 92 repositories and 9,942 tasks is also reported, although it is not used in the static track itself.

This split structure separates two generalization regimes. Cross-repo evaluation asks whether methods transfer to previously unseen repositories, while in-repo evaluation measures adaptation when training and test tasks come from the same repository family. A plausible implication is that the benchmark is not merely measuring memorization of project-local conventions; it is also measuring how well repository-level adaptation mechanisms extrapolate across codebases.

4. Evolution track: commit-derived evaluation under software change

The evolution track replays each repository’s Git history and emits a training or evaluation instance whenever a commit adds or modifies an assertion in a test file. Each instance is a triple $(\Delta_t, x_t, y_t)$, where $\Delta_t$ is the production-code diff up to that commit, $x_t$ is the test prefix in the post-commit snapshot, and $y_t$ is the new assertion target (Hotsko et al., 4 Jun 2026).

For in-repo splits, commits are chronologically partitioned so that training commits precede validation and test commits. For cross-repo splits, entire held-out repositories contribute all of their commit-derived tasks for evaluation. The training portion uses 400 repositories, since 9 of the 409 had too few commits, and includes 45,516 diffs that generate 215,129 training tasks. During training, tasks are capped at 4 tasks per file and 8 tasks per commit, yielding 44,149 “anchor” tasks for non-recurrent baselines. Cross-repo validation and test contain 58,944 and 44,732 tasks from 49 and 51 repositories, respectively. In-repo validation and test contain 38,783 and 42,061 tasks from 389 repositories. The benchmark also provides an out-of-distribution commit-derived test set of 92 repositories, 1,950 commits, and 14,813 tasks.

Across the full benchmark, the evolution track contains 399,649 total question-answer pairs, with 215,129 training, 97,727 validation, and 86,793 test instances. This is described as the first commit-derived QnA benchmark for code LLMs, specifically evaluating adapter refresh under real software evolution patterns (Hotsko et al., 4 Jun 2026).

The significance of the evolution track lies in its temporal structure. Repository knowledge is not assumed to be static, and the benchmark therefore exposes whether PEFT methods can track codebase drift, newly introduced APIs, and changing assertions as development proceeds. That emphasis differentiates RepoPeftBench from snapshot-only repository benchmarks.

5. Metrics and benchmarking protocol

RepoPeftBench reports Exact Match, Edit Similarity, and CodeBLEU. Exact Match is defined as

$$\mathrm{EM} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat y_i = y_i\},$$

with whitespace and casing normalized. The benchmark further distinguishes Cross-Repo EM, computed over the cross-repo test set, from In-Repo EM, which is computed per repository on its test tasks and then averaged across repositories (Hotsko et al., 4 Jun 2026).

The benchmarking protocol includes three method families. First are context-injection baselines: a pretrained Qwen2.5-Coder-1.5B model with no adaptation, RAG using Qwen3-Embedding-0.6B to embed 512-token chunks of non-test source and retrieve the top-$k$ chunks with $k=3$, and DRC, which statically resolves imports and prepends reachable definitions under an adaptive 8 K-token window. Second are fine-tuning baselines: FFT, sLoRA, and pLoRA. FFT makes all language-model parameters trainable. sLoRA uses one shared rank-16 LoRA across all repositories. pLoRA uses a separate rank-16 LoRA per repository and is characterized as an upper bound for in-repo splits, not applicable to cross-repo evaluation. Third are hypernetwork-generated adapters in the form of Code2LoRA, with static and evolutionary variants (Hotsko et al., 4 Jun 2026).

The benchmark therefore compares input-side context acquisition against parametric adaptation under a unified task. This is methodologically important because repository-level performance can improve either by exposing more context at inference time or by modifying the model’s parameters so that repository knowledge is internalized. RepoPeftBench is constructed to measure both strategies under identical assertion-completion objectives.

6. Reported findings, distinctive contributions, and relation to broader PEFT benchmarking

On the static track, Code2LoRA-Static is reported to achieve 63.8% cross-repo exact match and 66.2% in-repo exact match, matching the per-repository LoRA upper bound. On the evolution track, Code2LoRA-Evo achieves 60.3% cross-repo exact match, which is reported as a gain of 5.2 percentage points over a single shared LoRA (Hotsko et al., 4 Jun 2026). The paper further states that the temporal out-of-distribution holdout reveals that hypernetwork adapters retain cross-repo signal beyond in-distribution evolution.

RepoPeftBench’s stated contributions relative to prior benchmarks are threefold. First, it provides a full-repository release of every non-test source file and every first-parent commit diff, whereas RepoBench, CrossCodeEval, RepoHyper, and R2C2-Coder release only task-specific slices or retrieval snippets. Second, it introduces an evolution track centered on commit-derived QnAs. Third, by supporting both static and evolutionary tracks, it quantifies trade-offs among retrieval-based context injection, full-model fine-tuning, and hypernetwork-generated LoRA adapters that evolve with the code (Hotsko et al., 4 Jun 2026).

In the broader PEFT landscape, RepoPeftBench can be read alongside PEFT-Bench, which evaluates PEFT methods on autoregressive LLMs across 27 NLP datasets and introduces the PEFT Soft Score Penalties metric,

$$\text{PSCP} = P_t \times \left(1 + \tfrac{M_p}{C_p}\right)^{-\beta_p} \times \left(1 + \tfrac{M_f}{C_f}\right)^{-\beta_f} \times \left(1 + \tfrac{M_m}{C_m}\right)^{-\beta_m},$$

to incorporate trainable parameters, inference speed, and training memory usage (Belanec et al., 26 Nov 2025). RepoPeftBench does not adopt PSCP; instead, it evaluates repository-aware code prediction using task-level generation metrics. This suggests a division of labor between general PEFT efficiency benchmarking and repository-conditioned code adaptation benchmarking.

RepoPeftBench is therefore best understood as a benchmark for measuring how PEFT methods ingest and update repository knowledge under realistic software-engineering conditions. Its emphasis on whole repositories, Git history, cross-repo and in-repo regimes, and temporal out-of-distribution testing makes it a specialized instrument for studying PEFT in code LLMs rather than a generic benchmark for code completion.