Bridging the Modality Gap in Forensic Image Retrieval

Abstract: Automated image retrieval plays an increasingly critical role in modern forensic analysis, supporting investigative workflows that rely on efficient comparison of visual evidence. While prior work has focused primarily on developing and optimizing multimodal retrieval systems, limited attention has been paid to evaluating the forensic applicability of these technologies across diverse real-world scenarios. In this study, we present a unified retrieval framework adapted to four key forensic tasks: (1) tattoo image retrieval given a tattoo query image; (2) tattoo retrieval guided by human-expert textual descriptions, modelling the common situation where a witness verbally describes a tattoo; (3) tattoo retrieval from hand-drawn sketches; and (4) face retrieval from forensic face sketches. Our system leverages a multimodal LLM (MLLM) to automatically generate structured textual descriptions for all queries and gallery images, followed by sentence-transformer embedding for text-based comparison. We evaluate retrieval using visual-only embeddings, text-only embeddings and a multimodal fusion strategy that combines text- and image-based similarity scores derived from state-of-the-art visual feature extractors relevant to each task. The fusion of modalities consistently improves retrieval precision and robustness, especially in scenarios where visual information is limited or noisy (e.g., sketches, partial tattoos, or fragmented witness statements). This work highlights the forensic value of a unified multimodal retrieval pipeline and demonstrates how modern MLLMs can operationalize challenging forensic tasks that traditionally rely on manual expert analysis. Our results position multimodal retrieval as a promising tool for supporting investigative workflows involving tattoos, facial composites, and witness descriptions.

• The paper introduces a training-free pipeline that combines MLLM-generated captions, MPNet embeddings, forensic visual features, and multiplicative score fusion across tattoo and face retrieval tasks.
• The paper achieves mAP scores of up to 0.963 for tattoo photo retrieval, 0.898 for tattoo sketch retrieval, and 0.745 for text-based tattoo queries, while face retrieval reaches only about 0.384.
• The paper shows that compact MLLMs can outperform larger models, annotator variability can exceed algorithmic effects, and language-based retrieval works best for semantically rich evidence such as tattoos.

Overview and motivation

This paper presents a unified multimodal retrieval framework for forensic image search, evaluated across four operational scenarios: tattoo-to-tattoo image retrieval, tattoo retrieval from expert textual descriptions, tattoo retrieval from hand-drawn sketches, and face retrieval from forensic facial composites. The central mechanism is the conversion of all visual evidence—photographs, sketches, and even text queries—into a shared semantic space via MLLM-generated captions, encoded with a sentence transformer, and optionally fused with task-specific visual embeddings through a parameter-free multiplicative score combination. The authors' stated emphasis is on forensic applicability rather than architectural novelty, positioning the work as an assessment of how far zero-shot, training-free multimodal pipelines can support real investigative workflows.

Methodology

The pipeline has two branches. In the textual branch, every query and gallery item is captioned by one of five MLLMs: DeepSeek-VL2-tiny (1B), Qwen2-VL-2B, Bunny-v1.1-4B, Qwen2.5-VL-7B, and LLaVA-1.6-7B. Captions are produced under three prompt regimes of increasing specificity (generic description; element-focused; domain-oriented with forensic attributes such as colors, styles, text, or facial morphology). Captions are embedded with MPNet (all-mpnet-base-v2), chosen for its tolerance of long inputs relative to CLIP's text encoder. For human-written queries in Spanish, descriptions are machine-translated to English via Helsinki-NLP opus-mt-es-en; notably, this translate-then-encode strategy consistently outperformed a multilingual MPNet variant.

In the visual branch, the framework deliberately reuses validated extractors rather than proposing new ones: MobileNetV2, Weighted Average Pooling (WAP) features, CLIP, and TattTRN for tattoos; MobileFaceNet, ShuffleFaceNet, and ResNet50-ArcFace for faces. Fusion is multiplicative:

$S^i = \cos(q_t, g_t^i)\cdot\cos(q_v, g_v^i)$

The product formulation enforces cross-modal agreement and penalizes candidates strong in only one modality—an intentional contrast with weighted-additive fusion used in prior zero-shot CIR work. The design is entirely training-free, which the authors argue is essential given data scarcity and ethical constraints on supervised learning in forensics. Unimodal operation degrades gracefully to cosine ranking when only one modality is available.

Evaluation uses Rank@$K$ and mAP with bootstrap hypothesis testing, percentile confidence intervals, and Benjamini–Hochberg correction at $\alpha = 0.05$.

Experimental setup

Three public datasets are used: BIVTatt (210 images, 159 identities, 16 augmentation types), WebTattoo (1,029 images from 399 individuals for photo tasks; 300 sketch/photo pairs drawn by volunteers after a one-minute viewing and next-day recall), and UoM-SGFS (300 EFIT-V composites against Color-FERET photos). Human description queries were authored by three forensic experts in Spanish.

Results

Photo-to-photo tattoo retrieval. The strongest configurations reach mAP 0.963 on WebTattoo (DeepSeek-VL2-tiny + TattTRN), raising Rank@1 from roughly 0.82 (visual-only) to 0.87. A notable finding is that generic CLIP visual features achieve strong standalone performance (mAP 0.932), exceeding TattTRN (0.918); however, CLIP gains little from added text, whereas domain-specific backbones (TattTRN, MobileNetV2) show statistically significant improvements across most MLLM pairings. This indicates that textual descriptions supply complementary semantics precisely where specialized visual models lack language alignment. Pairwise analysis confirms TattTRN dominates CLIP and WAP in all 10 comparisons once augmented with text.

Human description-to-photo retrieval. Without any visual query, the best configuration achieves mAP 0.745 but Rank@1 of only about 0.43 (Rank@10 ≈ 0.69). The authors are explicit that this gap reflects linguistic ambiguity and that the system should be treated as an investigative lead generator requiring manual review of top candidates, not an automatic matcher. Two findings here carry substantial operational weight. First, annotator identity matters more than algorithmic choices: Expert 1 loses all 15 pairwise comparisons against Expert 2 and 13 against Expert 3, implying that human variability can outweigh prompt and model selection—a result that motivates standardized description protocols. Second, smaller MLLMs again dominate: DeepSeek-VL2-tiny wins 9 of 9 tests against LLaVA-1.6-7B and Qwen2.5-VL-7B.

Sketch-to-photo tattoo retrieval. This scenario exposes the modality gap most sharply. Generic features collapse (CLIP mAP 0.597, MobileNetV2 ≈ 0.54), while TattTRN alone reaches 0.816—an effect the authors attribute to its template-reconstruction training on semi-synthetic sketch-like overlays. Multimodal fusion lifts TattTRN to 0.888–0.898 mAP and Rank@1 to 0.649, with significant gains also for weak backbones (CLIP rises from 0.597 to 0.789). All top-10 configurations use TattTRN, confirming its necessity here. An important caveat is acknowledged: evaluation sketches were drawn by experts viewing the source image, so reported visual-feature performance may be optimistic relative to memory-based witness sketches, where the semantic branch would likely assume greater importance.

Face sketch-to-photo retrieval. Results diverge markedly. ResNet50-ArcFace dominates the visual baselines (18 of 20 pairwise wins; mAP 0.353), and only one fusion configuration (ResNet50 + Bunny-v1.1-4B, Prompt 3) yields a statistically significant improvement (mAP 0.384, Rank@10 from 0.615 to 0.660). The best Rank@1 is 0.253—far below the 0.649 achieved for tattoo sketches. The authors attribute this asymmetry to semantic richness rather than backbone quality: tattoos carry highly discriminative symbolic content readily articulated in language, whereas facial geometry is anatomically standardized and resistant to natural-language discrimination. This contrast functions as a boundary condition on the method: language-mediated retrieval delivers its largest benefits in semantically diverse domains, and gains should be expected to diminish for semantically narrow biometric traits.

Cross-cutting findings

Two results recur across scenarios and carry practical implications. First, model scale inversely correlates with retrieval quality: DeepSeek-VL2-tiny (1B) and Bunny-v1.1-4B consistently outperform 7B models, consistent with the authors' earlier observation of a negative correlation between output verbosity and retrieval accuracy—concise, attribute-focused captions embed more discriminatively than narrative ones. Second, prompt effects are task-dependent: Prompt 2 (element-focused) wins decisively for photographic tattoo retrieval, Prompt 1 slightly leads for human-description queries, and Prompt 3 (forensic-detail-oriented) is marginally preferred for faces, though with narrower margins.

Limitations and open questions

The paper concedes several constraints plainly. Evaluation relies on controlled datasets; operational environments introduce greater linguistic variability, noisier memory-based sketches, and partially incorrect descriptions not represented in these benchmarks. The system depends on caption fidelity—MLLM hallucinations or over-specification could inject systematic indexing errors, and increased descriptive detail may amplify biased content, raising concerns the authors themselves flag regarding perceptual bias in soft-biometric descriptions and legal admissibility. The study is restricted to English prompts, a limited set of forensic traits, and single-dataset evaluations per scenario; the face results rest on computer-generated composites (EFIT-V) rather than artist-drawn sketches, leaving hand-drawn composite performance untested. Open questions include robustness to cross-lingual queries, adaptive prompt strategies, uncertainty-aware ranking, and behavior under real casework conditions.

Conclusion

This paper demonstrates that a single, training-free pipeline—MLLM captioning, sentence-transformer embedding, and multiplicative score fusion—can serve heterogeneous forensic retrieval tasks with strong results in semantically rich domains (mAP up to 0.963 for tattoo photos, 0.898 for tattoo sketches, 0.745 for text-only queries) while delineating its own limits in semantically constrained ones (face composites, mAP ≈ 0.38). The findings that compact MLLMs outperform larger ones and that human annotator variability can dominate system-level choices are directly actionable for deployment. The work positions multimodal semantic retrieval as a complementary investigative tool whose value scales with the semantic diversity of the evidence, contingent on caption reliability, standardized human input protocols, and validation beyond controlled benchmarks.