---
title: 'LeaFBench: Benchmark for LLM Fingerprinting'
url: https://www.emergentmind.com/topics/leafbench
type: topic
---

# LeaFBench: Benchmark for LLM Fingerprinting

LeaFBench is an end-to-end, systematic benchmark for evaluating the reliability and robustness of large language model (LLM) fingerprinting under deployment-realistic conditions. Developed to standardize and rigorously compare LLM copyright auditing solutions, LeaFBench explicitly targets the most challenging adversarial and post-development settings, spanning both parameter-altering and parameter-independent model modifications. Its framework, dataset construction, evaluation metrics, and empirical findings jointly establish LeaFBench as the canonical reference point for fingerprinting research in generative AI contexts [2508.19843].

## 1. Motivation and Problem Definition

LLMs represent high-value intellectual property, yet they are susceptible to copyright infringement and model theft. Fingerprinting provides a non-intrusive means of asserting model provenance by extracting and later verifying distinctive signatures (or "fingerprints") from a model. The core open question addressed by LeaFBench is: **how robust are LLM fingerprinting methods to the broad spectrum of post-development model adaptations encountered in practical deployments?**

LeaFBench was designed to overcome two fundamental challenges:

- **Non-determinism:** Stochastic generation in LLMs creates instability in output- or activation-based fingerprints, necessitating robustness to sampling randomness.

- **Deployment-time modifications:** Modern LLMs are frequently subject to fine-tuning, quantization, instruction tuning, distillation, prompt-based steering, retrieval-augmentation, and adversarial paraphrasing—any of which may invalidate fingerprints crafted on the original base weights.

A systematic and extensible benchmark was needed to unify the evaluation protocols and put all fingerprinting schemes on equal empirical footing [2508.19843].

## 2. LeaFBench Framework and Taxonomy

Building on the formal framework of Shao et al., LeaFBench models all LLM fingerprinting schemes as a two-stage process:

- $\operatorname{Extract}(M) \rightarrow F$ extracts a compressed fingerprint $F$ from model $M$.
- $\operatorname{Verify}(M', F) \rightarrow \{0,1\}$ tests whether model $M'$ preserves fingerprint $F$.

The formal goal: An ideal auditor $\mathcal{A}$ satisfies $\mathcal{A}(M_s, M_o) = 1$ if $M_s$ is derived from $M_o$, and 0 otherwise.

Fingerprinting methods are categorized by access level and feature source:

- **White-box**: Full access to model internals.
  - *Static methods*: Direct analysis of model weights $W_1, ..., W_K$.
  - *Forward-pass methods*: Extraction from activations $R_1, ..., R_K$ generated by selected prompts.
  - *Backward-pass methods*: Analysis of gradients $G_1, ..., G_K$ after perturbation.

- **Black-box**: Only API-level querying is permitted.
  - *Untargeted*: Similarity comparison of model outputs over a query set $Q$.
  - *Targeted*: Use of optimized query-response pairs $(q_i, r_i)$ such that only the progenitor model (and direct derivatives) will match reliably.

This taxonomy enables the comparison of fundamentally diverse approaches within a unified experimental protocol.

## 3. Dataset Construction: Models and Transformations

LeaFBench comprises 149 distinct model instances, constructed by systematically applying thirteen canonical post-development adaptation techniques to seven mainstream foundation models (parameter size 1.1B–14B):

- **Models:**
  - Qwen-2.5-7B, Qwen-2.5-14B
  - Llama-3.1-8B, Mistral-7B-v0.3
  - Gemma-2-2B, TinyLlama-1.1B-v1.0
  - Llama-2-7B

- **Transformations:**
  - *Parameter-altering* (applied to each base model):
    1. Instruction Tuning (IT)
    2. General-purpose Fine-tuning (FT)
    3. Parameter-Efficient Fine-Tuning (PEFT)
    4. Quantization (QZ)
    5. Model Merging (MM)
    6. Distillation (DT)
  - *Parameter-independent* (applied to each instruction-tuned model):
    7. System Prompts (GP)
    8. Role-Playing Prompts (RP)
    9. Chain-of-Thought Prompts (CoT)
    10. Sampling Strategies (SS; various temperatures and greedy)
    11. Retrieval-Augmented Generation (RAG)
    12. Query Paraphrasing (ADV-input)
    13. Logit Perturbation (ADV-output)

The following table summarizes the distribution of model instances per technique:

| Technique | Number of Models |
|-----------|------------------|
| PT        | 6                |
| IT        | 7                |
| FT        | 18               |
| PEFT      | 9                |
| QZ        | 9                |
| MM        | 9                |
| DT        | 7                |
| GP        | 14               |
| RP        | 14               |
| CoT       | 14               |
| SS        | 21               |
| RAG       | 7                |
| ADV       | 14               |

LeaFBench thus emulates an extensive variety of real-world LLM deployments, ensuring comprehensive evaluation of fingerprinting robustness [2508.19843].

## 4. Evaluation Methodology and Metrics

LeaFBench employs a suite of quantitative metrics explicitly chosen to capture the key desiderata for copyright auditing:

- **Effectiveness:** Area Under ROC Curve (AUC). Differentiates true derivatives from independent models.
- **Uniqueness:** Partial AUC (pAUC) over FPR $\in [0,0.05]$, quantifying performance in the low-false-positive regime. Normalized to $[0,1]$.
- **Discriminability:** Mahalanobis Distance (MD) between mean fingerprint vectors of derivative and independent models:
  $$
  \mathrm{MD} = \sqrt{ (\mu_1 - \mu_2)^{\top} \Sigma^{-1} (\mu_1 - \mu_2) }
  $$
- **Efficiency:** End-to-end runtime for extraction and verification.

Additional reference points include absolute accuracy (ACC) at fixed thresholds and TPR at 1% FPR. This multidimensional metric design provides detailed diagnostic insight into both the power and limitations of candidate fingerprinting schemes [2508.19843].

## 5. Experimental Findings

Large-scale empirical evaluation on LeaFBench, spanning eight representative fingerprinting techniques (four white-box, four black-box), yielded several critical insights:

1. **White-box methods dominate.** Static methods (HuRef, PDF) achieve near-perfect AUCs (0.994, 0.995) and Mahalanobis Distances >1.7. Black-box methods reach at most AUC ~0.71 (TRAP) and MD ~1.38. Forward- and backward-pass white-box approaches are intermediate in performance.

2. **Static > dynamic:** Fingerprinting raw parameters offers maximum discriminative power; tying features to activations or gradients risks spurious "alignment" across distinct models.

3. **Beyond AUC:** Metrics such as pAUC and MD provide more granular assessment of uniqueness and class separation, revealing, for instance, that untargeted black-box methods (e.g., MET) can outperform targeted methods (TRAP) on low-FPR operation even when raw AUC trails.

4. **Model provenance matters:** Black-box performance collapses on vanilla pre-trained (PT) sources, in contrast to instruction-tuned (IT) sources, confirming behavioral fingerprints are weak except where strong conditioning yields distinctive outputs.

5. **Robustness analysis:** White-box methods exhibit only minor degradation under heavy parameter modification (FT, PEFT, QZ, MM, DT) and retain AUC >0.98. In contrast, black-box fingerprints are fragile to both parameter-altering and nonparameter transformations (system prompts, sampling strategies, or RAG).

6. **Efficiency trade-offs:** White-box static methods operate in under a minute, whereas advanced black-box (targeted) approaches may require hours of adversarial optimization, challenging scalability in forensic applications.

## 6. Limitations, Open Challenges, and Future Directions

LeaFBench highlights that white-box static fingerprinting—direct exploitation of the high-dimensional geometry of LLM weight spaces—is already highly effective for copyright auditing. Black-box methods, reliant on behavioral signals, remain inadequate for deployment settings characterized by extensive model editing and environmental entropy.

Key open challenges identified using LeaFBench include:

- **Inferring white-box features via black-box queries:** Attempting to invert or approximate internal model characteristics from API outputs.
- **Dynamic/adaptive querying:** Engaging in interactive or context-adaptive API interrogation to elicit more distinguishing signals.
- **Hybrid approaches:** Composing multiple behavioral features to increase reliability of black-box fingerprinting.
- **Efficiency improvements:** Reducing the computational cost of targeted adversarial prompt generation.
- **Multi-model attribution:** Distilling provenance in ensemble and multi-agent LLM systems, where attribution can be ambiguous.
- **Side-channel analysis:** Exploring operational telemetry (e.g., cache/timing/memory footprints) as a supplementary fingerprinting vector.
- **Extension to model honesty/auditing beyond lineage:** Including service transparency and detection of hidden/modified deployment tokens.
- **Dual-use risks:** Acknowledging that fingerprinting, if weaponized, may assist in model theft or adversarial probing.

*A plausible implication is* that systematic benchmarking, such as provided by LeaFBench, will be instrumental for closing the performance gap between white- and black-box techniques, and for anticipating new attacker-defender dynamics in LLM usage.

## 7. Impact and Availability

By rigorously standardizing datasets, transformations, and metrics, LeaFBench establishes the first common ground for reproducible, scalable, and deployment-relevant evaluation of LLM fingerprinting methodologies. Its results underscore the current inadequacy of black-box behavioral approaches for production-grade copyright auditing, and guide both method development and risk assessment for next-generation generative language systems.

LeaFBench is openly available at https://github.com/shaoshuo-ss/LeaFBench, enabling ongoing research and critical comparison across a rapidly evolving landscape [2508.19843].

Source: https://www.emergentmind.com/topics/leafbench