AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Model

Abstract: Large Vision and LLMs (LVLMs) have advanced rapidly, yet European Portuguese (pt-PT) remains systematically underserved by existing open-source multimodal models, which either conflate it with Brazilian Portuguese or severely under-represent it in their training data mixes. We introduce AMALIA-VL, the first open-source instruction-tuned LVLM built natively for pt-PT, pairing a high-resolution vision encoder with dynamic image tiling and a fully open pt-PT-optimized LLM via a learned connector. We contribute with a purposefully designed three-stage training process - vision-language alignment, general visual instruction tuning, and preference optimization - together with a pt-PT-centric multimodal data mix combining curated and translated public datasets with novel datasets that address the near-total absence of European Portuguese multimodal resources. Our evaluation shows that AMALIA-VL establishes a strong baseline for open-source pt-PT LVLMs.We will release model weights, training data, and construction pipelines along with machine-translated pt-PT evaluation benchmarks to help democratize pt-PT LVLM development.

• The paper introduces AMALIA-VL, a native Portuguese LVLM leveraging synthetic dataset engineering to fill a gap in pt-PT multimodal tasks.
• It employs a multistage training protocol combining vision-language alignment, visual instruction tuning, and DPO for enhanced performance.
• Empirical results show competitive scores in captioning, spatial grounding, and OCR, outperforming baseline European-centric models.

AMALIA-VL: A Native European Portuguese Open-Source Vision and LLM

Motivation and Context

The AMALIA-VL model addresses a longstanding gap in the landscape of open-source LVLMs: the absence of robust multimodal models natively supporting European Portuguese (pt-PT). Existing architectures, largely trained on web-scale corpora, conflate pt-PT with Brazilian Portuguese (pt-BR) or sideline it entirely—resulting in poor lexical, syntactic, and cultural grounding for pt-PT tasks. Prior initiatives such as V-Glória and TowerVision incorporate Portuguese or a broad European language mix, but lack native pt-PT centricity and instruction-following capabilities. AMALIA-VL establishes a transparent, fully open-cycle baseline specifically for pt-PT, leveraging custom data engineering and multimodal training strategies.

Figure 1: AMALIA-VL is natively European Portuguese grounding its answers in Portuguese visual culture, whereas general LVLMs hallucinate or fall back to Brazilian Portuguese.

Model Architecture

AMALIA-VL adopts a modular LVLM architecture inspired by LLaVA-NeXT, comprising a SigLip2-SO400M-patch16-384 vision encoder, a dedicated modality connector, and a pt-PT-tuned LLM (AMALIA DPO variant). Dynamic image tiling enables efficient high-resolution vision input, partitioning images into aspect-ratio matched tiles augmented with downsampled thumbnails for global context encoding. The vision encoder and language decoder remain frozen during initial connector alignment, optimizing the connector solely for robust vision-language grounding. The connector utilizes a two-layer MLP with GELU activation, outperforming linear and Q-former alternatives.

Multistage Training Protocol

AMALIA-VL's multistage training pipeline comprises:

• Stage 1: Vision-Language Alignment Connector warmup is accomplished with 500k PD12M image-caption pairs, freezing vision and language modules, disabled tiling.
• Stage 2: Visual Instruction Tuning This stage leverages a diverse mixture ($\sim$4.7M samples, 2B tokens) spanning grounding, VQA, naive OCR, captioning, chart/table comprehension, code reasoning, mathematical tasks, document understanding, and science domains. Coverage of pt-PT is increased via machine translation (Gemma3, TranslateGemma) and synthetic dataset generation, targeting "monolingual islands" to ensure transfer.
• Stage 3: Preference Optimization DPO training is employed, utilizing synthetic preference annotations based on Stage 2 data, supplemented via answer rewriting and automated scoring (Qwen3-30B). The lack of public multimodal preference data is mitigated by native synthetic pipelines, covering all task types for preference optimization.

Synthetic Dataset Engineering

Given the near-total absence of open pt-PT multimodal resources, this research builds a suite of synthetic datasets, constructed using open models and public-domain image collections. These datasets are tailored to support native pt-PT instruction tuning, OCR, document understanding, code parsing, and chart comprehension.

Figure 2: Samples from several of our pt-PT focused synthetic datasets.

For instance, PT-OCR leverages template-based dialogues and image perturbations, InvoiceQA combines field extraction and bounding box prediction with negative sampling, PT-Caps and PT-Caps-Fusion generate bilingual captions with controlled verbosity, and PT-VQA-Gen applies a multi-stage LVLM pipeline for high-quality pt-PT VQA pairs. Code datasets render executable Python snippets, pairing visual code images with semantic parsing challenges. InfographicSynth tackles chart reasoning with content generation and panel concatenation.

Evaluation Protocol and Benchmarks

Evaluation leverages the lmms-eval framework, covering General VQA, OCR/Document, Chart/Diagram, Spatial, Captioning, and Math tasks. All assessments are conducted in pt-PT, using both manually validated and machine-translated benchmarks covering 18 SoTA multimodal datasets. Deterministic inferencing is enforced (temperature=0) with default model configurations.

Empirical Performance and Results

AMALIA-VL achieves a competitive average score (54.4) among fully open models, excelling in captioning (50.8), spatial grounding (RefCOCO: 80.0), and OCR tasks (e.g., DocVQA: 69.1, TxtVQA: 69.2). Notably, it outperforms baseline European-centric models (TowerVision, EuroVLM), refuting the assumption that broad European language coverage is sufficient for pt-PT multimodal proficiency.

In captioning and spatial tasks, AMALIA-VL demonstrates superior long-form pt-PT generation and bounding box format adaptability—key requirements neglected by generalist models. In OCR, intentional translation of training data optimizes pt-PT extraction, leading to best-in-class performance in five out of eight OCR/document benchmarks. DPO preference optimization yields further gains, particularly in spatial and OCR benchmarks. Conversely, mathematical reasoning is constrained by the scarcity of native pt-PT long-form data, with low scores observed across all baselines.

Implications and Future Directions

AMALIA-VL's contributions—model weights, training pipelines, synthetic dataset engineering, and benchmark translation—fully operationalize pt-PT LVLM development for the community. By systematically mitigating cross-dialect contamination and leveraging advanced synthetic data generation, AMALIA-VL provides a native pt-PT multimodal baseline, enabling rigorous downstream evaluation and improved didactic applications in Portuguese visual culture.

Future research may pursue data augmentation strategies via scalable synthetic data generation, integration of multilingual benchmarks for contrastive evaluation, and targeted improvement of complex reasoning domains (e.g., MathVision) via native pt-PT task synthesis. The demonstrated efficacy of dynamic tiling and multistage preference optimization may inform broader architectures for other under-resourced language variants.

Conclusion

AMALIA-VL establishes a rigorous, native, open-source LVLM for European Portuguese, overcoming systematic representation and evaluation challenges via tailored synthetic datasets, translation pipelines, and multistage training. The architecture and datasets enable strong performance in tasks requiring fluent pt-PT comprehension and cultural grounding. These resources remove barriers for future pt-PT multimodal models, rendering the full development cycle auditable and reproducible, and setting the stage for further advances in multilingual, instruction-tuned LVLMs (2606.19100).