LeaFBench: Benchmark for LLM Fingerprinting

• LeaFBench is an end-to-end systematic benchmark designed to assess LLM fingerprinting under realistic, deployment-oriented adversarial conditions.
• It categorizes fingerprinting methods into white-box and black-box schemes, unifying diverse approaches and evaluation metrics for standardized comparison.
• Empirical findings highlight that white-box static methods outperform behavioral black-box approaches in accuracy, efficiency, and robustness against model modifications.

LeaFBench is an end-to-end, systematic benchmark for evaluating the reliability and robustness of LLM fingerprinting under deployment-realistic conditions. Developed to standardize and rigorously compare LLM copyright auditing solutions, LeaFBench explicitly targets the most challenging adversarial and post-development settings, spanning both parameter-altering and parameter-independent model modifications. Its framework, dataset construction, evaluation metrics, and empirical findings jointly establish LeaFBench as the canonical reference point for fingerprinting research in generative AI contexts (Shao et al., 27 Aug 2025).

1. Motivation and Problem Definition

LLMs represent high-value intellectual property, yet they are susceptible to copyright infringement and model theft. Fingerprinting provides a non-intrusive means of asserting model provenance by extracting and later verifying distinctive signatures (or "fingerprints") from a model. The core open question addressed by LeaFBench is: how robust are LLM fingerprinting methods to the broad spectrum of post-development model adaptations encountered in practical deployments?

LeaFBench was designed to overcome two fundamental challenges:

• Non-determinism: Stochastic generation in LLMs creates instability in output- or activation-based fingerprints, necessitating robustness to sampling randomness.
• Deployment-time modifications: Modern LLMs are frequently subject to fine-tuning, quantization, instruction tuning, distillation, prompt-based steering, retrieval-augmentation, and adversarial paraphrasing—any of which may invalidate fingerprints crafted on the original base weights.

A systematic and extensible benchmark was needed to unify the evaluation protocols and put all fingerprinting schemes on equal empirical footing (Shao et al., 27 Aug 2025).

2. LeaFBench Framework and Taxonomy

Building on the formal framework of Shao et al., LeaFBench models all LLM fingerprinting schemes as a two-stage process:

• $\operatorname{Extract}(M) \rightarrow F$ extracts a compressed fingerprint $F$ from model $M$.
• $\operatorname{Verify}(M', F) \rightarrow \{0,1\}$ tests whether model $M'$ preserves fingerprint $F$.

The formal goal: An ideal auditor $\mathcal{A}$ satisfies $\mathcal{A}(M_s, M_o) = 1$ if $M_s$ is derived from $M_o$, and 0 otherwise.

Fingerprinting methods are categorized by access level and feature source:

• White-box: Full access to model internals.
  • Static methods: Direct analysis of model weights $F$0.
  • Forward-pass methods: Extraction from activations $F$1 generated by selected prompts.
  • Backward-pass methods: Analysis of gradients $F$2 after perturbation.
• Black-box: Only API-level querying is permitted.
  • Untargeted: Similarity comparison of model outputs over a query set $F$3.
  • Targeted: Use of optimized query-response pairs $F$4 such that only the progenitor model (and direct derivatives) will match reliably.

This taxonomy enables the comparison of fundamentally diverse approaches within a unified experimental protocol.

3. Dataset Construction: Models and Transformations

LeaFBench comprises 149 distinct model instances, constructed by systematically applying thirteen canonical post-development adaptation techniques to seven mainstream foundation models (parameter size 1.1B–14B):

• Models:
  • Qwen-2.5-7B, Qwen-2.5-14B
  • Llama-3.1-8B, Mistral-7B-v0.3
  • Gemma-2-2B, TinyLlama-1.1B-v1.0
  • Llama-2-7B
• Transformations:
  • Parameter-altering (applied to each base model):
  • 1. Instruction Tuning (IT)
  • 2. General-purpose Fine-tuning (FT)
  • 3. Parameter-Efficient Fine-Tuning (PEFT)
  • 4. Quantization (QZ)
  • 5. Model Merging (MM)
  • 6. Distillation (DT)
  • Parameter-independent (applied to each instruction-tuned model):
  • 7. System Prompts (GP)
  • 8. Role-Playing Prompts (RP)
  • 9. Chain-of-Thought Prompts (CoT)
  • 10. Sampling Strategies (SS; various temperatures and greedy)
  • 11. Retrieval-Augmented Generation (RAG)
  • 12. Query Paraphrasing (ADV-input)
  • 13. Logit Perturbation (ADV-output)

The following table summarizes the distribution of model instances per technique:

Technique	Number of Models
PT	6
IT	7
FT	18
PEFT	9
QZ	9
MM	9
DT	7
GP	14
RP	14
CoT	14
SS	21
RAG	7
ADV	14

LeaFBench thus emulates an extensive variety of real-world LLM deployments, ensuring comprehensive evaluation of fingerprinting robustness (Shao et al., 27 Aug 2025).

4. Evaluation Methodology and Metrics

LeaFBench employs a suite of quantitative metrics explicitly chosen to capture the key desiderata for copyright auditing:

• Effectiveness: Area Under ROC Curve (AUC). Differentiates true derivatives from independent models.
• Uniqueness: Partial AUC (pAUC) over FPR $F$5, quantifying performance in the low-false-positive regime. Normalized to $F$6.
• Discriminability: Mahalanobis Distance (MD) between mean fingerprint vectors of derivative and independent models:

$F$7

• Efficiency: End-to-end runtime for extraction and verification.

Additional reference points include absolute accuracy (ACC) at fixed thresholds and TPR at 1% FPR. This multidimensional metric design provides detailed diagnostic insight into both the power and limitations of candidate fingerprinting schemes (Shao et al., 27 Aug 2025).

5. Experimental Findings

Large-scale empirical evaluation on LeaFBench, spanning eight representative fingerprinting techniques (four white-box, four black-box), yielded several critical insights:

1. White-box methods dominate. Static methods (HuRef, PDF) achieve near-perfect AUCs (0.994, 0.995) and Mahalanobis Distances >1.7. Black-box methods reach at most AUC ~0.71 (TRAP) and MD ~1.38. Forward- and backward-pass white-box approaches are intermediate in performance.
2. Static > dynamic: Fingerprinting raw parameters offers maximum discriminative power; tying features to activations or gradients risks spurious "alignment" across distinct models.
3. Beyond AUC: Metrics such as pAUC and MD provide more granular assessment of uniqueness and class separation, revealing, for instance, that untargeted black-box methods (e.g., MET) can outperform targeted methods (TRAP) on low-FPR operation even when raw AUC trails.
4. Model provenance matters: Black-box performance collapses on vanilla pre-trained (PT) sources, in contrast to instruction-tuned (IT) sources, confirming behavioral fingerprints are weak except where strong conditioning yields distinctive outputs.
5. Robustness analysis: White-box methods exhibit only minor degradation under heavy parameter modification (FT, PEFT, QZ, MM, DT) and retain AUC >0.98. In contrast, black-box fingerprints are fragile to both parameter-altering and nonparameter transformations (system prompts, sampling strategies, or RAG).
6. Efficiency trade-offs: White-box static methods operate in under a minute, whereas advanced black-box (targeted) approaches may require hours of adversarial optimization, challenging scalability in forensic applications.

6. Limitations, Open Challenges, and Future Directions

LeaFBench highlights that white-box static fingerprinting—direct exploitation of the high-dimensional geometry of LLM weight spaces—is already highly effective for copyright auditing. Black-box methods, reliant on behavioral signals, remain inadequate for deployment settings characterized by extensive model editing and environmental entropy.

Key open challenges identified using LeaFBench include:

• Inferring white-box features via black-box queries: Attempting to invert or approximate internal model characteristics from API outputs.
• Dynamic/adaptive querying: Engaging in interactive or context-adaptive API interrogation to elicit more distinguishing signals.
• Hybrid approaches: Composing multiple behavioral features to increase reliability of black-box fingerprinting.
• Efficiency improvements: Reducing the computational cost of targeted adversarial prompt generation.
• Multi-model attribution: Distilling provenance in ensemble and multi-agent LLM systems, where attribution can be ambiguous.
• Side-channel analysis: Exploring operational telemetry (e.g., cache/timing/memory footprints) as a supplementary fingerprinting vector.
• Extension to model honesty/auditing beyond lineage: Including service transparency and detection of hidden/modified deployment tokens.
• Dual-use risks: Acknowledging that fingerprinting, if weaponized, may assist in model theft or adversarial probing.

A plausible implication is that systematic benchmarking, such as provided by LeaFBench, will be instrumental for closing the performance gap between white- and black-box techniques, and for anticipating new attacker-defender dynamics in LLM usage.

7. Impact and Availability

By rigorously standardizing datasets, transformations, and metrics, LeaFBench establishes the first common ground for reproducible, scalable, and deployment-relevant evaluation of LLM fingerprinting methodologies. Its results underscore the current inadequacy of black-box behavioral approaches for production-grade copyright auditing, and guide both method development and risk assessment for next-generation generative language systems.

LeaFBench is openly available at https://github.com/shaoshuo-ss/LeaFBench, enabling ongoing research and critical comparison across a rapidly evolving landscape (Shao et al., 27 Aug 2025).