CARLoS: Behavioral Representation of LoRAs

• The paper introduces CARLoS, a framework that measures the behavioral impact of LoRA adapters by quantifying semantic shifts in generated images.
• It employs paired image comparisons with a CLIP encoder to isolate the LoRA's effect, enabling metadata-free and scalable indexing of SDXL LoRAs.
• Empirical results show CARLoS outperforms text-based retrieval baselines by offering improved relevance, diversity, and legal risk screening for LoRAs.

Searching arXiv for the exact topic name and closely related papers. CARLoS—“Retrieval via Concise Assessment Representation of LoRAs at Scale”—is a retrieval and analysis framework for text-to-image LoRAs that characterizes an adapter by its measured effect on generated images rather than by user-provided descriptions, tags, examples, or popularity signals. In the formulation introduced for SDXL LoRAs, CARLoS treats a LoRA as a behavioral object, measures the semantic shift it induces relative to a base model across many prompts and seeds, and compresses that behavior into three quantities: Direction, Strength, and Consistency. The resulting representation is metadata-free, prompt-independent in intent, and designed for scalable retrieval, screening, and comparative analysis (Sarfaty et al., 9 Dec 2025).

1. Problem setting and conceptual basis

CARLoS is motivated by the rapid proliferation of community-made LoRA adapters and the corresponding absence of reliable methods for understanding what a given LoRA actually does. Existing discovery workflows are described as depending on names, brief descriptions, tags, curated examples, or download counts, all of which can be sparse, misleading, socially biased, or entirely unavailable. The framework therefore replaces metadata-centric discovery with direct behavioral characterization: a LoRA is indexed through the change it causes in generated outputs under controlled paired comparisons with a fixed base model (Sarfaty et al., 9 Dec 2025).

The target domain is the modern modular image-generation stack, particularly SDXL LoRAs hosted in public repositories. In that setting, LoRAs may affect style, content, mood, identity, composition, or other output attributes, but those effects are often entangled with prompts and random seeds. CARLoS isolates the incremental effect attributable to the LoRA itself by contrasting generations with and without the adapter under identical conditions. A plausible implication is that the framework is best understood not as a general image retriever, but as a standardized behavioral indexing scheme for generative components.

The paper positions CARLoS as both a retrieval system and an analysis tool. Retrieval is the primary application: textual queries are mapped into the same semantic-shift space used to represent LoRAs, and matching is then performed by alignment in that space. The same representation is also used for downstream analyses of LoRA behavior, including discussions of strength, stability, diversity of retrieved items, and legal-risk screening.

2. Representation: Direction, Strength, and Consistency

CARLoS constructs its representation from paired image generations. Let $x^{(0)}_{p,s}$ denote the vanilla base-model output for prompt $p$ and seed $s$, and $x^{(l)}_{p,s}$ the corresponding output with LoRA $l$ applied. With $v(\cdot)\in \mathbb{R}^d$ the CLIP image encoder, the LoRA’s prompt-seed-specific effect is represented by the CLIP-difference vector

$$V^l = \left\{ v\!\big(x^{(l)}_{p,s}\big) - v\!\big(x^{(0)}_{p,s}\big)\right\}_{p\in\mathcal{P},\, s\in\mathcal{S}}.$$

In the implementation reported in the paper, the encoder is CLIP ViT-B/32, specifically openai/clip-vit-base-patch32, producing non-normalized 512-dimensional embeddings (Sarfaty et al., 9 Dec 2025).

The three components of CARLoS are defined as follows.

Component	Definition
Direction	$\text{SD}(l) = \frac{1}{|V^l|}\sum_{v \in V^l} v$
Strength	$$\text{Str}(l) = \frac{1}{|V^l|}\sum_{v \in V^l} \left\lVert v \right\rVert_2$$
Consistency	Average pairwise cosine similarity among the vectors in $V^l$

Direction is the LoRA’s average semantic shift in CLIP space. Strength is the average magnitude of that shift and quantifies how strongly the LoRA changes outputs relative to the base model. Consistency measures how stable that effect is across prompts and seeds. The paper prints the Consistency formula with a likely typesetting omission in the denominator, but its intended meaning is explicitly average pairwise cosine similarity among all CLIP-diff vectors for a LoRA. High Consistency indicates that the average direction is behaviorally meaningful; low Consistency indicates that the effect varies unpredictably.

The reliance on base-model subtraction is structurally central. Without the difference

$p$0

the embedding would largely encode prompt content rather than the adapter’s incremental effect. CARLoS therefore uses paired comparison to transform a generic CLIP representation into a behavioral representation of a LoRA.

3. Indexing pipeline and corpus construction

The corpus used to instantiate CARLoS was curated from CivitAI and restricted to SDXL LoRAs. The validated corpus contains 656 LoRAs. The supplementary description states that metadata were initially retrieved for the first 10,000 reachable SDXL LoRAs via the CivitAI API, after which the corpus was filtered to exclude LoRAs less than 100 days old, files larger than 10 GB, NSFW modules, corrupted files, and invalid modules that could not be loaded. From the filtered metadata, the top 1,875 by downloads were prioritized, downloaded where possible, and validated by loading them into a standard diffusers SDXL pipeline; only $p$1 were successfully validated (Sarfaty et al., 9 Dec 2025).

The indexing prompt set $p$2 contains $p$3 English prompts, and the retrieval/query-construction prompt set $p$4 contains another disjoint $p$5 prompts. Both were created with an LLM under human guidance and cover $p$6 semantic categories, including portraits, animals, fantasy, landscapes, vehicles, fashion, artistic styles, cinematic scenes, and conceptual art. The seed set has $p$7 values. For each LoRA, prompt, and seed, CARLoS generates paired base-model and LoRA outputs using Stable Diffusion XL 1.0 base with LoRA scale $p$8, CFG $p$9, Euler scheduler, and fixed defaults otherwise. The supplementary notes that generation begins with seed 42 and increments sequentially over the 16 samples.

The total indexing volume is approximately

$s$0

for $s$1, $s$2, and $s$3. The paper reports a one-time indexing cost of about 7 NVIDIA A6000 GPU-hours per LoRA. After embeddings were computed, original 1024$s$41024 images were replaced with 64$s$564 thumbnails for storage efficiency. A plausible implication is that CARLoS is operationally most attractive for centralized indexing on hubs or institutional libraries rather than for ad hoc local indexing of rapidly changing corpora.

4. Query construction and retrieval procedure

At query time, CARLoS maps a text query into the same semantic-shift space used to represent LoRAs. Let $s$6 be the CLIP text encoder. For a query $s$7 and a retrieval prompt $s$8, the framework forms a suffixed prompt $s$9 and defines

$x^{(l)}_{p,s}$0

Averaging over the retrieval prompt set gives the query effect vector

$x^{(l)}_{p,s}$1

The paper refers to this as the reciprocal textual CLIP-diff. Retrieval is then driven by cosine similarity between $x^{(l)}_{p,s}$2 and the LoRA direction vector $x^{(l)}_{p,s}$3 (Sarfaty et al., 9 Dec 2025).

The ranking score is reconstructed in the paper’s details as

$x^{(l)}_{p,s}$4

After ranking by Direction, CARLoS filters candidates using thresholds on Strength and Consistency. The reported thresholds are

$x^{(l)}_{p,s}$5

The intended filtered set is therefore the LoRAs satisfying $x^{(l)}_{p,s}$6 and $x^{(l)}_{p,s}$7, with ranking determined by semantic alignment to the query vector. This filtering suppresses LoRAs that are too overpowering or too unstable even when their average direction is query-relevant.

The retrieval design is explicitly training-free: the paper does not train a new retriever, fine-tune CLIP, or learn a projection between text and LoRA representations. Query-time computation of $x^{(l)}_{p,s}$8 takes about 5 seconds on one NVIDIA A5000 GPU, and ranking over all 656 indexed LoRAs takes about 0.09 seconds.

5. Empirical evaluation and retrieval performance

CARLoS is evaluated against four metadata-based multilingual text retrieval baselines operating on concatenated LoRA names and descriptions: Qwen3, Multilingual-E5-instruct, BGE-M3-reranker, and mGTE-reranker. The benchmark uses over 700 representative text queries generated using GPT, Grok, and Gemini. For each query, the system retrieves top-$x^{(l)}_{p,s}$9 LoRAs, generates images with a small fixed prompt set, and scores relevance using Qwen2.5-VL, SigLIP2, ImageReward, and HPS v2. Because evaluator ranges differ, scores are linearly min-max normalized to $l$0 across all queries and retrievers and then averaged (Sarfaty et al., 9 Dec 2025).

In the main top-3 evaluation, CARLoS outperforms all baselines on all four evaluators.

Retriever	SigLIP2	Qwen2.5	IR	HPS
E5	0.289	0.480	0.449	0.565
GTE	0.258	0.461	0.439	0.556
BGE	0.199	0.429	0.387	0.543
Qwen3	0.307	0.495	0.491	0.590
CARLoS	0.350	0.532	0.505	0.596

Relative to the strongest textual baseline, Qwen3, the gains are $l$1 on SigLIP2, $l$2 on Qwen2.5, $l$3 on ImageReward, and $l$4 on HPS. Supplementary results show that CARLoS remains best from top-1 through top-7. The user study includes 36 participants, around 100 unique A/B questions, at least 6 answers per question, and more than 1800 answered sub-questions in total; the reported qualitative outcome is a consistent human preference for CARLoS over all four baselines on image quality, relevance to the query, and overall preference.

The ablation study attributes the performance to all three components of the system. Removing Strength filtering or Consistency filtering degrades several metrics, and removing both degrades performance further. Query construction also matters: appending the query as a suffix performs best, while using only the query is substantially worse. The paper further reports diversity statistics showing that CARLoS retrieves from a broader portion of the corpus than text baselines. For top-3 retrieval distribution, CARLoS achieves normalized entropy $l$5, Gini coefficient $l$6, and effective LoRA count $l$7, compared with Qwen3 at $l$8, $l$9, and $v(\cdot)\in \mathbb{R}^d$0, respectively.

6. Broader relevance, legal interpretation, and limitations

Beyond retrieval, CARLoS is presented as a compact behavioral descriptor for LoRAs that may support platform search, recommendation, quality filtering, creator-facing summaries, and ecosystem analysis. The paper also proposes a legal and governance interpretation of its metrics. Strength is linked to legal notions of substantiality: stronger LoRAs are described as more likely to impose distinctive material into outputs. Consistency is linked to volition and predictability: highly consistent behavior is more foreseeable and therefore more easily attributable to deliberate use or design. The paper is explicit, however, that these are only proxies and do not determine whether protected expression is reproduced (Sarfaty et al., 9 Dec 2025).

The limitations are substantial and clearly stated. First, CARLoS inherits the strengths and weaknesses of CLIP, including weaker handling of spatial composition, fine-grained texture, and some multimodal biases. Second, despite averaging over many prompts and seeds, the representation remains dependent on the chosen prompt distribution; LoRAs specialized for underrepresented niches may be mischaracterized. Third, all indexing uses a single operating point, particularly LoRA scale $v(\cdot)\in \mathbb{R}^d$1, even though the paper notes that the relation between scale and measured Strength is nontrivial and varies across LoRAs. Fourth, the one-time indexing cost is high. Fifth, the experiments are limited to SDXL LoRAs and do not establish transfer to other backbones or adapter families.

Taken together, these features make CARLoS a behavioral indexing framework rather than a universal theory of adapter semantics. Its central contribution is the demonstration that LoRAs can be represented at scale by measured semantic direction, effect magnitude, and effect stability, and that such a representation can outperform metadata-based discovery in both automated and human evaluations.