---
title: 'MiMo-VL-Miloco-7B: Home-Centric Vision-Language Model'
url: https://www.emergentmind.com/topics/mimo-vl-miloco-7b
type: topic
---

# MiMo-VL-Miloco-7B: Home-Centric Vision-Language Model

MiMo-VL-Miloco-7B is an open-source vision-language model tailored for home-centric multimodal understanding, built on the MiMo-VL-7B backbone, with adaptations for scenario-specific perception, chain-of-thought (CoT) reasoning, and efficient inference. It leverages a two-stage pipeline combining supervised fine-tuning (SFT) and reinforcement learning via Group Relative Policy Optimization (GRPO), enabling state-of-the-art performance in both dedicated home-domain tasks and broad multimodal benchmarks. A quantized version, MiMo-VL-Miloco-7B-GGUF, supports efficient edge deployment with minimal loss in accuracy. Model checkpoints and an evaluation toolkit are publicly available to support reproducibility and further research [2512.17436].

## 1. Model Architecture and Home-Centric Adaptations

MiMo-VL-Miloco-7B comprises a Vision Transformer (ViT) encoder processing raw video frames, an MLP projector aligning visual embeddings with the language model (LLM) input space, and a 7B-parameter LLM backbone initialized from MiMo-VL-7B. Home-centric modifications include the use of a native-resolution ViT for detailed gesture capture, integration of CoT heads in the LLM for explicit intermediate reasoning about home-related activities, and specialized token-budget prompts to facilitate concise, latency-efficient on-device inference.

The multi-modal input pipeline interleaves visual and text modalities as follows:
- Video input is sampled at 2 FPS (up to 256 frames per sample), with each frame split into non-overlapping 28×28 patches (maximum 4096 patches per image), encoded by ViT. Resulting visual embeddings are projected into the token space via MLP.
- Textual data, such as instructions or queries, are tokenized with byte-pair encoding (BPE), optionally augmented with special CoT guide markers.
- The final input sequence concatenates visual and text tokens for LLM consumption.

## 2. Two-Stage Training Pipeline: SFT and GRPO

The training regimen balances home-domain specialization with general multimodal competence through sequential SFT followed by GRPO-based reinforcement learning:

**Stage 1: Supervised Fine-Tuning (SFT)**
- Training utilizes proprietary datasets of annotated home-scenario videos (five daily activities and five gestures), with each sample labeled and provided with stepwise CoT rationales.
- Token-budget aware prompts and supervision incentivize concise output.
- Additional data from the general MiMo-VL corpus covers VQA, visual grounding, OCR, video QA, and multimodal reasoning.
- Loss functions consist of cross-entropy on all generated tokens (CoT and answer), with an extra penalty for excessive token generation in token-budget tasks.
- Major hyperparameters: AdamW optimizer, batch size 128, learning rate \( 1 \times 10^{-5} \), and maximum sequence length of 32,768 tokens.

**Stage 2: Reinforcement Learning (GRPO)**
- GRPO restores and augments general multimodal capacities post-SFT, employing a filtered dataset blend emphasizing temporal grounding, GUI prediction, and STEM-type reasoning.
- The GRPO objective is:
  $$
  J_{\rm GRPO}(\theta)
  =
  \mathbb{E}_{q\sim P(Q),\,\{o_i\}_{i=1}^G\sim\pi_{\theta_{\rm old}}}
  \left[
      \frac{1}{G}\sum_{i=1}^G
        \mathrm{clip}\left(r_i(\theta)\hat A_i,\,1-\epsilon,\,1+\epsilon\right)
  \right]
  -\beta\,\mathbb{D}_{\mathrm{KL}}(\pi_\theta\|\pi_{\rm ref})
  $$
  with group-normalized advantage
  $$
  \hat A_i = \frac{r_i^{\rm raw}-\overline r}{\mathrm{std}(r_1^{\rm raw},\dots,r_G^{\rm raw})}.
  $$
  Rewards mix accuracy (e.g., IoU or exact match) and format compliance (\( r = 0.9\,r_{\rm acc} + 0.1\,r_{\rm fmt} \)).

## 3. Specialized Reasoning: Chain-of-Thought and Token-Budget Awareness

- **Chain-of-Thought (CoT):** Home-domain data is systematically annotated with explicit, stepwise rationales. Training examples are prompted by phrases such as “Think step by step:” and supervised via cross-entropy on both reasoning tokens and the answer. This enhances the model’s interpretability and intermediate reasoning for fine-grained activity and gesture classification.
- **Token-Budget Awareness:** Select tasks employ token-efficient prompts (“Answer concisely:”) and loss penalties for verbosity, directly optimizing for concise outputs conducive to edge-device inference. At inference, this reduces average latency by approximately 30%, with the model defaulting to concise responses unless richer CoT is explicitly requested.

## 4. Quantization and Edge Deployment: GGUF Variant

To enable efficient edge deployment, MiMo-VL-Miloco-7B is quantized to the GGUF (Generic GGML Unified Format), converting transformer weights to 4-bit integers (LSQ). Attention and MLP layers employ NF4 quantization; layernorm and embeddings retain 16-bit float precision for stability. This reduces memory footprint from ~28 GB (fp16) to ~7 GB (4-bit), with minimal accuracy loss (≤ 0.5% absolute), and yields a ~20% speed increase on a single edge GPU due to improved memory bandwidth. The GGUF variant retains more than 99% of the performance of the full-precision model on both domain-specific and general tasks.

## 5. Empirical Results and Benchmark Performance

MiMo-VL-Miloco-7B demonstrates leading performance across home-scenario and general vision-language benchmarks. The following tables summarize select results [2512.17436]:

### Home-Scenario Activity and Gesture Recognition (per-category F1)

| Model             | Watch TV | Reading | Phone | Esports | Workout | OK   | Thumbs Up | V Sign | Shaka | Open Palm |
|-------------------|----------|---------|-------|---------|---------|------|-----------|--------|-------|-----------|
| Qwen2.5-VL-7B     |   81.2   |  78.7   | 88.4  |  95.2   | 88.9    | 73.8 |   78.6    |  67.0  | 33.3  |   53.4    |
| Gemini-2.5-Pro    |   93.8   |  88.9   | 81.9  |  95.9   | 91.9    | 81.8 |   86.9    |  86.7  | 70.1  |   87.4    |
| Miloco-7B         |   98.3   |  90.8   | 90.5  |  99.2   | 96.7    | 86.3 |   90.5    |  90.3  | 88.3  |   87.5    |
| Miloco-7B-GGUF    |   97.9   |  90.4   | 90.0  |  98.8   | 96.2    | 85.9 |   90.1    |  89.9  | 87.8  |   87.1    |

### Multimodal Video and Language Understanding

| Model        | Video-MME | Video-MMMU | Charades-STA | MMMU-Pro(std) | MMMU-Pro(vision) | MMLU-Pro (EM) |
|--------------|-----------|------------|--------------|---------------|------------------|---------------|
| Miloco-7B    | 68.0      | 63.6       | 46.6         | 55.7          | 47.2             | 68.5          |
| GGUF         | 67.7      | 63.2       | 46.2         | 55.3          | 46.9             | 68.1          |

Miloco-7B consistently surpasses open-source (Qwen2.5-VL, InternVL3-8B) and closed-source (Gemini-2.5-Pro) baselines on home-scenario tasks (by 4–18 F1 points), and sets new open-source records on generalized multimodal benchmarks (e.g., +6 points on Video-MMMU compared to the prior best). The GGUF variant matches >99% of full-precision accuracy on all tasks.

## 6. Analysis, Limitations, and Future Directions

SFT yields substantial improvements in household activity and gesture detection, with only minor catastrophic forgetting (<3 points) on long-document OCR and some GUI grounding tasks. The subsequent reinforcement learning stage largely restores general capabilities, minimizing this trade-off.

Minor regressions remain on document-centric tasks (e.g., 1-point reduction on OCRBench). Future research directions include multi-objective tuning for domain balancing, integration of additional modalities (audio, motion sensors), and further quantization/pruning to target sub-5 GB memory footprints on resource-constrained devices.

A plausible implication is that targeted scenario-specialization, when combined with robust recovery via reinforcement learning, allows for high-impact application of vision-language models in real-world settings such as the smart home domain, without significant sacrifice to generalization [2512.17436].

## 7. Relationship to MiMo-VL-7B Family and Broader Research Context

MiMo-VL-Miloco-7B extends the MiMo-VL-7B-RL model family [2506.03569], which pioneered a four-stage, 2.4-trillion token pre-training architecture combining massive scale with CoT-focused data mixing and a Mixed On-policy Reinforcement Learning (MORL) regime. MiMo-VL-Miloco-7B adapts this backbone for home environments, refining CoT annotation/methodology and introducing explicit token-budget calibration.

Both MiMo-VL-Miloco-7B and MiMo-VL-7B models adopt group-normalized or on-policy RL objectives, employ verifiable and human preference rewards, and leverage extensive, diverse multi-domain datasets. The research outputs highlight the utility of scenario-specific adaptations, explicit reasoning supervision, and efficient quantization for practical, deployable vision-language models [2506.03569] [2512.17436].

Source: https://www.emergentmind.com/topics/mimo-vl-miloco-7b