BabyVLM-V2: Infant-Inspired Vision–Language Model

• BabyVLM-V2 is a developmentally grounded framework that uses infant-inspired, ecologically valid data for efficient vision–language pretraining.
• It features a compact multimodal architecture with a staged training procedure incorporating egocentric video, image–utterance pairs, and dialogic sequences.
• The framework employs the DevCV Toolbox for rigorous cognitive benchmarking, revealing its sample efficiency and nuanced performance across varied tasks.

BabyVLM-V2 is a developmentally grounded framework for vision–language modeling that leverages infant-inspired data and cognitive benchmarking to achieve sample-efficient pretraining and robust evaluation of vision foundation models. By directly aligning pretraining data and evaluation protocols with early childhood sensory and cognitive trajectories, BabyVLM-V2 departs from conventional data-intensive paradigms, introducing an end-to-end recipe involving longitudinal, egocentric audiovisual corpora, a compact multimodal model architecture, and the DevCV Toolbox—a benchmark suite adapted from standardized infant neurocognitive assessments. The approach provides empirical evidence that vision–LLMs can acquire non-trivial visuocognitive competencies with training data volumes and content closely matching those encountered by human infants, in contrast to the orders-of-magnitude larger corpora characteristic of state-of-the-art (SOTA) large-scale models (Wang et al., 11 Dec 2025).

1. Developmentally Plausible Pretraining Corpus

Central to BabyVLM-V2 is the use of ecologically valid training data modeled after the experiences of infants. The core corpus, SayCam, comprises 478 hours of head-mounted, egocentric video from three children aged 6–32 months, with minimal processing to preserve sensory fidelity. Three primary modalities are constructed:

• Video–Utterance Pairs: 181,000 clips totaling approximately 138 hours, derived by segmenting continuous video at transcript boundaries (using Azure Speech Recognition) and filtering for temporal length, transcript reliability, and visual–textual similarity (X-CLIP threshold > 0.1). Clips are uniformly padded for temporal consistency.
• Image–Utterance Pairs: 768,000 image–transcript pairs, sampled at 1 FPS with relevant frame selection via CLIP similarity thresholding (> 0.2) to align images with utterances.
• Interleaved Multi-Turn Sequences: 63,000 sequences constructed via a sliding window over 4–8 consecutive image–utterance pairs with overlap, simulating dialogic exchanges reminiscent of infant–caregiver interaction.

This dataset totals under one million examples, spanning single images, short videos, and conversational multi-turns—mirroring the heterogeneity of infant sensory environments and representing a departure from the heavily curated, trillion-token-scale datasets typical in foundation model training.

2. Model Architecture and Training Procedure

BabyVLM-V2 utilizes a compact, modular architecture, building upon the BabyLLaVA infrastructure:

• Language Backbone: LLaMA-1.1B (1.1 billion parameters)
• Vision Encoder: ViT-L-16 (300 million parameters)
• MLP Connector: Projects visual features into the language embedding space

The resultant model contains approximately 1.4 billion parameters, supporting multimodal inputs (text, images, videos) and producing text outputs for downstream tasks.

Training is staged as follows:

1. Unimodal Warm‐Up (Stage 0):
   • Language-only autoregressive loss

   $$\mathcal{L}_{\text{lang}} = -\sum_t \log P(w_t | w_{<t})$$

   on 283,000 caregiver utterances. - Vision-only DINOv2 self-supervised loss on 1.08M SAYCam frames.

2. Connector Alignment (Stage 1):

   • MLP connector trained (backbones frozen) to autoregressively predict captions from visual features on 768,000 image–utterance pairs.
3. Joint Multimodal Pretraining (Stage 2):
   • Unfreeze language and connector, with ViT frozen; train on mixed video–utterance, image–utterance, and conversational sequences with combined autoregressive loss

   $$\mathcal{L}_\text{pre} = \alpha\,\mathcal{L}_\text{IU} + \beta\,\mathcal{L}_\text{VU} + \gamma\,\mathcal{L}_\text{MC}$$

4. Instruction Tuning (Stage 3):

   • All parameters unfrozen; finetuned on 150,000 curated instruction–response pairs that sample across DevCV tasks, again with next-token autoregressive loss.

This multistage regimen enables robust, tightly integrated multimodal learning using developmentally aligned, data-efficient supervision.

3. DevCV Toolbox: Cognitive Benchmarking

For evaluation, BabyVLM-V2 introduces the DevCV Toolbox, a suite of ten multimodal tasks adapted from the NIH Baby Toolbox, targeting competencies in Language (L), Executive Function/Memory (EF/M), and Math. Each task is formatted for either multiple-choice or free-response, addressed via a single inference pass through the model’s text-generation interface. Accuracy is the principal metric.

Task Name	Subdomain	Test Set Size
Looking While Listening	Language (L)	1,200
Picture Vocabulary	Language (L)	1,200 (SayCam) / 1,200 (Ego4D)
Localization	Language (L)	2,100
Left/Right	EF/M	2,300
Spatial Details	EF/M	1,200
Visual Delayed Response	EF/M	900 (Binary), 1,200 & 900 (Exact/Adjacent)
Memory	EF/M	500 test, 10,000 trials
Who Has More	Math	3,100 (Synthetic), 2,200 (Naturalistic)
Subitizing	Math	1,900 (SayCam), 200 (Ego4D)
Object Counting	Math	3,000

Notably, Visual Delayed Response employs both “exact” and “adjacent” correctness bands, and Memory accuracy is defined as the fraction of items for which both test-phase trials are answered correctly:

$$\mathrm{Acc_{mem}} = \frac{1}{k}\sum_{i=1}^k \mathbf{1}\{r_i=1\},\quad r_i = \begin{cases} 1, & \text{both trials for item }i\text{ correct} \ 0, & \text{otherwise} \end{cases}$$

4. Empirical Performance Against Baselines

On the SAYCam-driven DevCV benchmark, BabyVLM-V2 attains an overall accuracy of 55.2%, substantially exceeding the random-guess baseline (31.8%), though below proprietary large-scale models such as GPT-4o (74.6%), Gemini 2.5 Pro (82.5%), and GPT-5 (87.6%). Task-specific results include:

• Language: 27.4% on Picture Vocabulary (cf. GPT-4o 93.7%), 38.8% on Localization.
• Executive Function/Memory: 42.3% on Left/Right, 91.3% on Spatial Details (SOTA parity), 57.6% on Visual Delayed Response (binary), 75.3% on Memory.
• Math: 98.4% (synthetic) and 52.8% (naturalistic) on Who Has More; 44.6% on Object Counting (GPT-4o: 39.0%).

The model surpasses GPT-4o in Counting and Who Has More, matches on Spatial Details, and underperforms on language and multi-frame reasoning. Generalization is modest (41.1% on zero-shot Ego4D suite), and original NIH Baby Toolbox cartoons reveal pronounced domain gaps (2/6 Counting, 3/12 Mullen VR, 13/24 Who Has More).

Ablations confirm pretraining's necessity: models fine-tuned only on instructions without infant-inspired pretraining underperform by approximately 20 percentage points across data fractions. Replacing natural transcripts with GPT-4o-generated captions yields a minor overall improvement, notably a three-point gain in Picture Vocabulary. Additionally, instruction-tuned BabyVLM-V2 data yields 10–30 point improvements when transferred to open-source models (LLaVA-OneVision, Qwen2.5-VL).

5. Analysis and Broader Implications

Findings from BabyVLM-V2 reveal that sample-efficient, developmentally grounded pretraining—scaling to less than 400 hours of data—facilitates the emergence of advanced visuocognitive competencies. Key observations are as follows:

• Sample Efficiency and Transfer: Pretraining on minimally curated, ecologically aligned corpora induces robust transfer to vision–language tasks with minimal instruction following data.
• Instruction Tuning: Mixed-domain tuning on instruction–response data induces broad instruction-following ability, outperforming task-specific tuning of larger-scale models.
• Domain Specificity: SAYCam-trained models exhibit constrained generalization when evaluated on adult egocentric (Ego4D) or abstract carton domains, underscoring the need for broader, more heterogeneous developmental corpora for future research.
• Benchmark Discriminability: The DevCV Toolbox demonstrates high discriminative value, with human survey baselines near ceiling and clear differentiation among SOTA systems.
• Utility of Pretraining: Instruction tuning alone is insufficient; developmentally grounded pretraining remains vital for unlocking core visuocognitive abilities.

A plausible implication is that more extensive, diverse infant-centric datasets (e.g., BabyView), enhanced video self-supervised objectives, and richer naturalistic annotations may further close the performance gap to proprietary models and yield models with broader generalization and domain robustness.

6. Prospective Directions and Limitations

Looking forward, the BabyVLM-V2 protocol establishes a rigorous foundation for future work in developmentally plausible vision–language modeling. Prospective research directions include:

• Scaling Corpus Size and Diversity: Integrating larger or more varied developmental corpora to enhance generalization and domain adaptation.
• Advanced Self-Supervision: Employing more sophisticated video objectives (e.g., masked prediction, contrastive temporal alignment).
• Enhanced Language Annotations: Incorporating more naturalistic, child-directed speech in annotation pipelines.
• Expanded Benchmarks: Extending DevCV to cover a broader age range and introducing richer, interactive cognitive tasks.

Ultimately, the integration of brain- and child-inspired constraints on data and objectives may provide a pathway to more data-efficient, robust, and cognitively aligned multimodal models, advancing understanding of both artificial and natural developmental intelligence (Wang et al., 11 Dec 2025).