LangIR: Multimodal IR Small Target Dataset

• LangIR is a multimodal infrared small target detection dataset that combines 8-bit grayscale images, pixel-level binary masks, and concise natural language descriptions.
• It extends NUAA-SIRST and IRSTD-1k datasets by integrating automatically generated text annotations using large vision-language models for enhanced segmentation.
• The dataset provides reproducible splits, standardized file structures, and benchmark metrics like IoU and nIoU to facilitate language-guided detection studies.

The LangIR dataset is a multimodal benchmark for infrared small target detection (IRSTD) that augments established single-modality datasets (NUAA-SIRST and IRSTD-1k) with a standardized text modality. Each sample is represented as a triplet of grayscale infrared image, pixel-level binary mask, and a natural language region descriptor generated via large vision-LLMs (VLMs). LangIR is designed to enable research on language-guided IRSTD and to study the synergy between visual and textual priors for small-object segmentation tasks in adverse imaging conditions (Singh et al., 17 Jul 2025).

1. Composition and Structure

LangIR is constructed by extending two widely used single-frame IR-small-target datasets:

• NUAA-SIRST subset: 427 real IR images
• IRSTD-1k subset: 1,001 real IR images

The total corpus comprises 1,428 unique triplets $(I, M, T)$, where $I$ is the infrared image, $M$ is the binary ground-truth mask, and $T$ is the text description. Directory structure and naming conventions are strictly maintained:

• Image files (PNG):
  • IRSTD-1k: images/XDU<id>.png
  • NUAA-SIRST: images/Misc_<id>.png
• Mask files (PNG):
  • IRSTD-1k: masks/XDU<id>_mask0.png
  • NUAA-SIRST: masks/Misc_<id>_pixels0.png
• Text descriptions (UTF-8, ≤50 words):
  • IRSTD-1k: descriptions/XDU<id>_description.txt
  • NUAA-SIRST: descriptions/Misc_<id>_description.txt

Official split files (trainval/test) are distributed for both parent datasets; no further official val/test split is defined beyond the original authors' divisions.

2. Modalities and Annotation Protocols

LangIR includes three aligned modalities per sample:

1. Infrared Image: 8-bit grayscale, depicting challenging small-object scenarios in complex backgrounds (man-made and natural scenes, varying contrast, day/night captured).
2. Ground-truth Mask: Binary segmentation mask, precisely co-registered with $I$.
3. Textual Region Description: Each plain-text .txt file encodes the approximate location of the target, typically using relative spatial terms ("upper left quadrant", "center") but deliberately omits explicit pixel coordinates or bounding-boxes.

Text descriptions are provided for all images and were generated using automatic large multimodal model prompting protocols (see §3), not via human annotation, to ensure consistency and scalability.

3. Language Annotation Workflow

Text descriptions were generated by an automated pipeline leveraging GPT-4 Vision, with Claude 3.5 Sonnet cross-validation. Key aspects:

• Prompt format: System-role instruction, concise spatial localization, task length limit (≤50 words), e.g., "You are an expert who can locate the small target in the infrared image. Locate the small target within the infrared image and respond succinctly within 50 words, detailing the region where the target is situated."
• Input: Each IR image is included as a Base64-encoded string.
• Prompt-style optimization: Zero-shot prompts were unreliable; few-shot over-generated; system-role prompts with explicit length restrictions yielded correct and concise outputs.
• Example outputs:
  • “A small bright spot appears near the lower right quadrant of the image, just above the tree line.”
  • “A tiny heat signature is located in the upper left corner of the frame, appearing as a dim circular blob.”
• Text-visual alignment: Descriptions are mapped to a small finite set of position words, empirically: {upper-left, upper-right, center, lower-left, lower-right}; actual word distributions are reported per subset and split (see §4).

During model training, the text $T$ is encoded via a CLIP text encoder, and the image $I$ via a CLIP image encoder, producing vector embeddings $T_e$ and $I_e$. The fused Target Descriptor $TD = T_e + I_e$ is input to the LGNet architecture's language-fusion block.

4. Dataset Statistics and Properties

• Per-split sample counts:
  • NUAA-SIRST: 427 images
  • IRSTD-1k: 1,001 images
• Positional-word count (partial excerpt):

Subset	Split	left	right	center	lower	upper
NUAA-SIRST	train	157	87	102	64	111
	test	37	30	24	12	28
IRSTD-1k	train	340	193	229	146	247
	test	81	61	53	41	61

• Target size and signal constraints (SPIE definition):
  • Target area: $I$00.15% of image
  • Contrast: $I$115%
  • SNR: $I$21.5
• NUAA-SIRST stats (from source): 55% of targets occupy $I$30.02% image area; ~90% of images have a single target.
• Scene diversity: Images cover both man-made clutter and natural scenes, textureless and high-texture backgrounds, and illumination variations.

5. Benchmark Evaluation Metrics

The dataset adopts standard IRSTD segmentation metrics:

• Intersection over Union (IoU):

$I$4

where $I$5 is predicted-target pixels, $I$6 is ground-truth pixels, and $I$7 is true positives.

• Normalized IoU (nIoU):

$I$8

• Probability of Detection $I$9:

$M$0

• False Alarm Rate $M$1:

$M$2

Reported model improvements on the LangIR-augmented benchmarks include relative gains in IoU (4.41%–9.74% depending on subset), nIoU, $M$3, and $M$4 when the language prior is incorporated in the training regime.

6. Key Methodological Insights and Usage Recommendations

• The language prior acts as auxiliary supervision and is only required during training. At inference time, image-only pipelines attain virtually equivalent performance (IoU: 73.30 with language-prior at training only, 69.99 without), indicating the text's primary value is in constraining attention during representation learning.
• Prompt engineering is critical: Failing to enforce singular "target" can cause VLMs to hallucinate multiple objects; verbosity control (≤50 words) is required for relevant positional output.
• Reproducibility is facilitated by releasing split-index files, standardized directory layout, and specifying exact toolchains: GPT-4 Vision or equivalent API, CLIP ViT-B/16, and PyTorch for LGNet (4M parameters, ≈25ms/img inference).
• Direct VLM-generated bounding boxes are unreliable for small targets (occupying ~0.02% image area); semantic language cues ("quadrant", "corner") outperform bounding box extraction from VLMs in these scenarios.
• Benchmark design: LangIR sets a concrete foundation for future multimodal IRSTD research, providing both visual data and a rigorously engineered text modality to enable and evaluate language-guided detection architectures (Singh et al., 17 Jul 2025).

7. Applications and Limitations

Applications of LangIR include training, fine-tuning, and benchmarking of multimodal and language-guided IRSTD models. It supports investigations into visual-textual fusion, robustness to ambiguous supervision, and serves as a testbed for prompt engineering in weakly supervised learning scenarios. Notable limitations include reliance on automatically generated (rather than expert-annotated) descriptions, constrained positional vocabularies, and the inability of the language prior to encode precise object boundaries. LangIR does not natively support video or temporal IRSTD, nor does it cover all possible real-world deployment scenarios.

LangIR is the first multimodal standard in IR small target detection, operationalizing vision-language fusion strategies for challenging sparse small-object settings and providing reproducibility and ablation tracking for emerging IRSTD architectures (Singh et al., 17 Jul 2025).