PLaMo 2.1-VL Technical Report

Abstract: We introduce PLaMo 2.1-VL, a lightweight Vision LLM (VLM) for autonomous devices, available in 8B and 2B variants and designed for local and edge deployment with Japanese-language operation. Focusing on Visual Question Answering (VQA) and Visual Grounding as its core capabilities, we develop and evaluate the models for two real-world application scenarios: factory task analysis via tool recognition, and infrastructure anomaly detection. We also develop a large-scale synthetic data generation pipeline and comprehensive Japanese training and evaluation resources. PLaMo 2.1-VL outperforms comparable open models on Japanese and English benchmarks, achieving 61.5 ROUGE-L on JA-VG-VQA-500 and 85.2% accuracy on Japanese Ref-L4. For the two application scenarios, it achieves 53.9% zero-shot accuracy on factory task analysis, and fine-tuning on power plant data improves anomaly detection bbox + label F1-score from 39.7 to 64.9.

• The paper introduces a bilingual VLM, PLaMo 2.1-VL, emphasizing Japanese-centric operation with an instruction-tuned LLM and vision-language MLP adapter.
• The paper details a two-stage training protocol using synthetic data and LoRA, which enhances performance in tasks like VQA, visual grounding, and anomaly detection.
• The paper demonstrates practical impact in factory task analysis and infrastructure anomaly detection, showcasing robust zero-shot results and effective in-domain finetuning.

Technical Summary of "PLaMo 2.1-VL Technical Report" (2604.19324)

Model Design and Training Paradigm

PLaMo 2.1-VL is a bilingual, lightweight VLM targeting low-latency, high-accuracy deployment on edge devices, with a core emphasis on Japanese-language operation. It is released in 8B and 2B variants, both built upon the instruction-tuned PLaMo 2.1 LLM backbone. The model architecture adheres to the LLaVA paradigm: a frozen LLM, an image encoder (SigLIP2), and a parameter-efficient MLP adapter for vision–language alignment.

The rationale for component selection heavily prioritizes edge deployment constraints: SigLIP2 yields superior localization and patch-wise spatial correspondence, which is critical for VQA and Visual Grounding/REC. The MLP adapter is memory-efficient and enables larger effective batch sizes during training versus alternatives like Q-Former. For visual input representation, dynamic tiling optimizes support for variable image resolutions and aspect ratios.

Training follows a two-stage protocol. Stage 1.0 freezes the LLM and image encoder, updating only the adapter using bilingual web-scale caption data (Japanese:English = 75:25). Stage 1.5 unfreezes all modules using LoRA with a diverse, instruction-tuned multimodal dataset. Synthetic data generation is central, employing large-scale pipelines for VQA, Visual Grounding, spatial/relational reasoning, counting, tool recognition, and difference detection. Japanese translation procedures leverage a custom translation model, context-aware example-driven translation, and post-filtering to maintain semantic and compositional fidelity.

Application Scenarios and Task Formulation

Two primary domains motivate evaluation: factory task analysis (via tool recognition for work inference) and infrastructure anomaly detection (comparing time-separated drone/camera imagery). The VLM is explicitly engineered to produce not just natural-language answers (VQA), but also visual evidence and grounded predictions, addressing safety and traceability requirements in industrial settings.

• Factory Task Analysis: The model is tasked with classifying factory task images into ten categories by merging visual tool cues with detailed natural language prompts emphasizing visual descriptors.
• Infrastructure Anomaly Detection: The pipeline constructs paired images (reference/target), aligns them via homography, and tasks the model with multi-label anomaly detection—requiring spatially localizing and semantically classifying anomalies such as foreign object presence or state changes.

Data Synthesis Pipelines

Synthetic data is generated with emphasis on multi-instance support and open-vocabulary distribution. For visual grounding, referring expressions are produced using a strong VLM and aligned with regions extracted by SAM3. Multi-instance expressions are emphasized to avoid single-bbox output bias. VQA data generation separates fact extraction, relation triplet enumeration, and QA pair construction for higher annotation reliability, especially for spatial and counting queries.

Synthetic difference detection data is constructed by overlaying object masks from Open Images onto destination backgrounds, then simulating geometric perturbations to the reference image. Instruction prompts enumerate candidate classes explicitly, with “dummy” categories included to control for hallucination and teach precise task following.

Experimental Results

VQA

On the Japanese JA-VG-VQA-500 benchmark, PLaMo 2.1-8B-VL achieves 61.5 ROUGE-L, 72.4 LLM-as-a-judge, and 4.37 Likert—surpassing strong baselines including Asagi-14B and Qwen3-VL-8B-Instruct. The 2B variant also outperforms contemporaries, indicating that data pipelines and Japanese-centric tuning yield substantial improvements in answer grounding and instruction adherence.

Visual Grounding

On the challenging Ref-L4 (English and PFN-constructed Japanese variants), PLaMo 2.1-8B-VL achieves 85.2% accuracy (Japanese) and 86.8% (English), outperforming the latest Qwen models, including the Qwen3-VL-235B-A22B-Instruct. The multi-instance visual grounding emphasis, together with synthetically constructed translated resources, results in robust generalization.

Factory Task Analysis

PLaMo 2.1-8B-VL attains 53.9% zero-shot accuracy for 10-way classification on challenging factory images (baselines: Qwen3-VL-235B at 45.8%, Qwen3-VL-8B at 38.3%). The approach of leveraging contextualized tool recognition, combined with detailed prompt engineering, enables substantial gains in operational task inference under zero-shot transfer.

Figure 1: PLaMo 2.1-8B-VL prediction examples for factory task analysis: two accurate predictions and one tool-induced misclassification.

Anomaly Detection

Zero-shot performance on power plant anomaly detection reaches 58.9 (bbox-only F1) and 39.3 (bbox+label F1), with the 8B and 2B models outperforming all strong open competitors. Fine-tuning on in-domain data increases bbox+label F1 dramatically (from 39.7 to 64.9), showing clear transfer and adaptation benefits. Analysis reveals that object size is strongly correlated with error rate; raising minimum target size thresholds elevates F1 substantially (e.g., 88.0 for objects with ≥200px geometric mean). State-driven anomalies remain the most challenging.

Figure 2: The two-pass anomaly detection pipeline: broad localization (pass 1), followed by label refinement on cropped regions (pass 2).

Figure 3: Synthetic anomaly data generated via object mask overlay for robust difference detection supervision.

Figure 4: Reference images perturbed with geometric transformations to simulate real-world misalignment.

Figure 5: PLaMo 2.1-8B-VL anomaly detection examples: top—successful localization/labeling, bottom—misses and state-driven errors.

Figure 6: Bounding box size quantile distribution illustrating the wide variation in anomaly target scales.

Figure 7: Agreement rates as a function of size quantile and anomaly type, indicating lower detection/labeling performance for smaller targets and state-driven anomalies.

Translation Pipeline and Benchmark Construction

Translation of publicly available and synthetic datasets is conducted with a hybrid in-house model that balances literal fidelity, compositional structure, and the preservation of quoted strings for referential integrity. The Japanese Ref-L4 benchmark is carefully built to preserve multi-clause referential structure and challenge linguistic understanding at parity with the original English version. Multiple candidates for translation are generated with varying context, and the highest embedding similarity is selected to maximize semantic equivalence.

Figure 8: Sample image highlighting text region for translation fidelity evaluation.

Limitations

Significant constraints include lack of OCR/document task coverage, untested performance on text-only or image-only queries, minimal domain expert knowledge, and incomplete coverage for inputs falling outside anticipated industrial task and anomaly distributions. The system is not intended for ungrounded or expert reasoning scenarios and assumes operational context and prompt engineering are aligned with intended real-world tasks.

Implications and Broader Impact

PLaMo 2.1-VL demonstrates that careful architecture, instruction tuning, comprehensive and multi-stage synthetic data generation (including robust translation support), and explicit multi-instance, open-vocabulary supervision can result in mid-scale VLMs that strongly outperform larger and more resource-intensive generalist models (such as Qwen3-VL-235B-A22B-Instruct) in targeted tasks, especially for Japanese language field operation. For practical deployment, especially in high-stakes industrial infrastructure and factory contexts, the results indicate robust zero-shot baseline performance and the feasibility of quick adaptation via moderate-scale in-domain finetuning.

The data and pipeline modularity demonstrated here are likely to influence future domain-centric VLM development, especially for non-English markets and resource-constrained deployment. However, state-driven anomaly detection and extremely small object recognition still represent open challenges; progress will depend on higher-resolution input pipelines, temporal aggregation (for occlusion mitigation), and further dataset expansion.

Conclusion

PLaMo 2.1-VL establishes a new standard for Japanese-language and edge-centric VLMs, offering high accuracy in VQA and Visual Grounding with strong transfer to complex industrial applications such as task analysis and anomaly detection. The technical approach—spanning architecture, training, synthetic and translation-augmented data, and application-driven benchmarks—provides a practical blueprint for future multilingual, domain-targeted VLM work. Continued research will need to address state-driven anomaly granularity, extensibility to document/OCR tasks, and robust zero-shot capabilities under less-controlled input distributions.