---
title: 'Meow-Omni 1: Feline Intent Inference'
url: https://www.emergentmind.com/topics/meow-omni-1
type: topic
---

# Meow-Omni 1: Feline Intent Inference

Meow-Omni 1 is a multimodal large language model for computational ethology whose stated objective is feline intent inference rather than generic recognition, captioning, or action labeling. It is presented as the first open-source, quad-modal MLLM in this domain, combining video, audio, physiological time-series, and text in a single reasoning stack. The model is motivated by “semantic aliasing,” the claim that visually or acoustically similar behaviors may correspond to different internal states depending on physiological context; in the reported evaluation on MeowBench, Meow-Omni 1 achieves 71.16% Top-1 accuracy and is described as state of the art for this benchmarked setting [2605.09152].

## 1. Problem definition and conceptual motivation

The central scientific problem addressed by Meow-Omni 1 is the inference of a cat’s underlying intent or internal state from multimodal observations. In the paper’s framing, observable behavior is not a sufficient proxy for latent state because identical external signals may arise from distinct causes. The canonical example is purring: a purr may indicate contentment, but it may also function as self-soothing during pain or respiratory distress. The same logic is extended to contrasts such as playful aggression versus territorial defense, where similar motion patterns may conceal different underlying drives [2605.09152].

This motivates the paper’s use of the term **semantic aliasing**: identical or near-identical external behavior can map to different internal states depending on context, especially physiological context. A core implication is that video and audio are treated as expressive but not causally sufficient modalities. Physiological information is introduced as a disambiguating signal, intended to ground inference in latent state rather than surface pattern matching.

The observation history up to time \(t\) is written as

\[
\mathcal{H}_t = \{V_{1:t}, A_{1:t}, B_{1:t}\},
\]

where \(V\), \(A\), and \(B\) denote vision, audio, and biometric time-series. The target mapping is

\[
f_\theta: \mathcal{H}_t \to \mathbb{I},
\]

with \(\mathbb{I}\) a discrete set of intent categories. The paper further defines intention as the latent biological or cognitive drive that would maximize the probability of the animal’s next action in an unconstrained environment. As printed in the manuscript, the causal definition is given with a bracket omission, but the intended expression is that intention is the category \(i \in \mathbb{I}\) maximizing the probability of the optimal next action \(a^*_{t+1}\) given observed history \(\mathcal{H}_t\) under \(do(\mathcal{E}_{free})\).

The label space is standardized into a unified 30-class intention taxonomy. Examples listed in the paper include **Feed**, **Groom**, **Rest**, **Run**, **Shake**, **Trot**, **Walk**, **active_climbing**, **active_jumping.horizontal**, **active_playfight.fighting**, **inactive_lying.resting**, **maintenance_nutrition.eating**, **maintenance_scratching**, and **other_social.allogrooming**. This taxonomy mixes coarse ethological categories with more granular behavior-intention labels, reflecting the paper’s emphasis on latent-state semantics rather than only motor primitives.

## 2. Architecture and multimodal integration

Meow-Omni 1 is built by modifying the MiniCPM-o multimodal backbone and incorporating a time-series branch adapted from Intern-S1-Pro. The model description is explicitly surgical rather than from-scratch: modality-specific encoders produce representations for video, audio, and time-series; the time-series features are projected into the language-model hidden space; modality embeddings are interleaved with text embeddings into a single multimodal token sequence; and a causal LLM performs unified reasoning and generation [2605.09152].

Across the main text and appendix, the backbone is described as based on **MiniCPM-o 4.5** or the **MiniCPM-V backbone**. The consistent architectural point is that a MiniCPM multimodal causal LLM functions as the shared reasoning engine. The paper does not introduce new custom vision or audio encoders; those are inherited from the MiniCPM stack. The main novelty is the native inclusion of physiological time-series.

The time-series branch is adapted from **Intern-S1-Pro**. Its encoder is dedicated to continuous scientific time-series processing, and its outputs remain frozen during training. A **linear projector** maps TS hidden states into the LLM hidden dimension. The tokenizer is expanded with three TS-specific control tokens:

- `<|ts_start|>`
- `<|ts_unit|>`
- `<|ts_end|>`

A custom **MeowOmniProcessor** dynamically expands `<|ts_unit|>` so that the number of TS placeholder positions matches the number of hidden states emitted by the TS encoder. TS is therefore represented as a variable-length token span bracketed by start and end delimiters.

Fusion is intentionally simple. The model accepts \(N\)-dimensional TS tensors in addition to normal multimodal inputs, projects TS embeddings to the LLM hidden size, interleaves them with visual and linguistic embeddings, and passes the resulting sequence through the causal transformer. The paper does not describe a separate cross-attention stack, an explicit alignment head, or a dedicated fusion module. A plausible implication is that the model’s multimodal integration strategy is closer to token-space co-embedding than to specialized cross-modal routing.

Handling of incomplete sensing is also explicit. The model can process **V+A+TS**, **V+A**, **V+TS**, **A+TS**, or single-modality inputs. If a modality is missing, it is simply omitted from the token sequence; no placeholder tokens are inserted for absent modalities. This design is presented as a mechanism for robustness under heterogeneous real-world sensing and mixed-modality training.

## 3. Data resources, preprocessing, and benchmark design

The primary training dataset is **Meow-10K**, reported as containing **10,831 samples**. It is intentionally heterogeneous rather than restricted to perfectly synchronized quad-modal recordings. The dataset includes **TS-only samples** from accelerometer datasets, **video-only clips**, **standalone audio clips**, **synchronized audio-video clips**, and a **small number of synthetic quad-modal samples**. After balancing, the abundant TS-only pool is subsampled to **2,000** examples so that it does not dominate the training distribution [2605.09152].

The TS pipeline begins from two peer-reviewed accelerometer datasets. Original signals are sampled at **30 Hz or 60 Hz**, then aggregated into second-level signals by averaging within each second. Windows of **5 s**, **7 s**, **10 s**, and **15 s** are used, with prediction targets taken **1**, **2**, **3**, or **5 seconds ahead** to form a **Next-Behaviour Prediction (NBP)** dataset. The full TS pool used in stage-1 alignment contains **383,853 labeled TS samples**.

Video preprocessing uses an open-source dataset pipeline cited to Bain et al. and employs **Qwen2-VL-72B-Instruct** for coarse action detection, temporal localization, verification, clip extraction, and sequence caption/action-label generation. These action labels are mapped by **Qwen3.5-35B** into the unified 30-class taxonomy and reviewed by experts.

Audio comes from two sources. First, synchronized AudioSet-derived A/V clips are filtered with an **AST classifier** for cat-related sounds. The retention policy is explicit: retain if score \(> 0.10\), discard if \(< 0.03\), and for scores between \(0.03\) and \(0.10\), apply stationary noise reduction with suppression factor **0.85** and recheck. Those clips are then processed by **Qwen2.5-VL-7B** to generate audio-focused captions and a 30-class score distribution. Second, standalone audio clips from **Freesound** and **taozi555/cat_class** are caption-expanded by **Qwen3.5-35B-A3B-GPTQ-Int4**; these do not receive direct intention labels and instead use descriptive targets.

Each training sample carries a natural-language query-response pair and, where available, an intention label from the 30-class taxonomy. For TS prompts, the paper reports prompt diversification with **DeepSeek**, **GPT**, and **Gemini**.

Evaluation is conducted on **MeowBench**, a held-out benchmark for multimodal intent reasoning. Because naturally synchronized quad-modal feline datasets do not exist, MeowBench is constructed by **intent-matched synthesis**: a synchronized video-audio clip from one session is paired with a TS sample from another session, provided that both share the same intention label. This is synthetic at the cross-session level, but intended to preserve semantic consistency. Candidate items were verified by **eight Professional Feline Ethologists**, divided into groups of **3**, **3**, and **2**. From **645 initial candidates**, the process yielded **527 expert-verified samples**. Each item is cast as a multiple-choice question with **1 correct intention** and **3 distractors**, and the principal metric is **Top-1 Accuracy**.

## 4. Training procedure and formal objectives

Training is explicitly two-stage. **Stage 1** is a projector-alignment phase intended to align TS encoder outputs with the LLM latent space. **Stage 2** is a multimodal-specialization phase intended to adapt the LLM backbone to feline multimodal intent reasoning [2605.09152].

In **Stage 1**, only the **TS projector** is trainable, while both the LLM backbone and TS encoder are frozen. The data source is the **383,853 labeled TS samples**. The objective is **cross-entropy on predicted next behavior class**. Optimization uses **AdamW** with learning rate \(10^{-4}\), weight decay \(0.0\), batch size **1 per device**, gradient accumulation **2**, cosine annealing with **3% linear warmup**, maximum **10 epochs**, and early stopping with patience **1 epoch**. The best checkpoint is reported at **epoch 2** and named **Meow-Omni 1 Aligned**.

In **Stage 2**, the trainable component is the **LLM backbone only**; all encoders and projectors are frozen. Training uses **Meow-10K** with **10,831 samples**. The objective is **next-token prediction loss over the multimodal sequence, including special tokens**. Optimization again uses **AdamW**, with learning rate \(2\times10^{-5}\), weight decay \(0.0\), batch size **1 per device**, gradient accumulation **2**, cosine decay with **3% warmup**, maximum **5 epochs**, and early stopping with patience **1 epoch**. The appendix identifies the final checkpoint at **epoch 1**, whereas the main text describes it as obtained after **2 epochs**.

The hardware configuration is reported as **1 node** with **8 × NVIDIA H200 80 GB GPUs**, interconnected by NVLink. Missing modalities are omitted entirely from the sequence; the paper does not propose a specialized asynchronous alignment algorithm. A plausible implication is that mixed-modality robustness is achieved primarily through omission-aware serialization and heterogeneous training, not through explicit temporal synchronization machinery.

The manuscript provides a high-level supervised objective,

\[
\mathcal{L}(\theta) = \sum_{(\mathcal{H}_t, \mathcal{I}) \in \mathcal{D}} \log P( \mathcal{I} \mid \text{Encoder}(\mathcal{H}_t); \theta),
\]

which expresses likelihood maximization of the correct intent label given encoded multimodal history. The paper also defines predictive entropy for uncertainty quantification under repeated stochastic decoding:

\[
H(\mathcal{H}_t) = - \sum_{c \in C} \hat{p}(c) \log \hat{p}(c),
\]

where \(C\) is the set of distinct generated intention categories and \(\hat{p}(c)\) is the empirical class probability. In the reported uncertainty experiment, the model uses **\(N = 10\)** stochastic forward passes at generation temperature **\(T = 0.7\)**.

The paper is mathematically light beyond these expressions. It does not provide explicit formulas for contrastive alignment, fusion, cross-attention, auxiliary physiological grounding, or detailed temporal modeling.

## 5. Empirical results, ablations, and uncertainty analysis

The headline result is **71.16% Top-1 accuracy** on **MeowBench**. The main baseline comparison includes unimodal, multimodal, and omni-modal systems. Reported unimodal baselines are **Audio SOTA** at **36.86%**, **Video SOTA (Qwen3.5-122B)** at **61.95%**, and **TS SOTA** at **48.98%**. Using **Qwen3.5-Omni-Plus** as an omni-modal baseline, the reported accuracies are **65.36%** for **V+A**, **66.21%** for **V+TS**, **42.15%** for **TS+A**, and **66.89%** for **V+A+TS**. Meow-Omni 1 reaches **71.16%** with **V+A+TS**, a gain of **4.27 percentage points** over the strongest omni baseline, described in the paper as **4.3 points** [2605.09152].

Ablation by modality masking clarifies the contribution of each signal. Within the Meow-Omni 1 architecture, **Audio only** yields **51.88%**, **Vision only** **69.97%**, and **TS only** **55.63%**. For bimodal settings, **V+A** gives **68.43%**, **V+TS** **70.82%**, and **TS+A** **54.95%**. The full **V+A+TS** model reaches **71.16%**. The most consequential finding is that time-series contributes more useful disambiguation than audio when added to vision: vision alone is **69.97%**, vision plus audio drops slightly to **68.43%**, while vision plus TS increases to **70.82%**.

This pattern is the paper’s main empirical support for physiological grounding. It suggests that, on this benchmark, vision carries most of the raw discriminative signal, TS supplies the critical disambiguating increment, and audio contributes primarily in the fully integrated setting rather than as a strong standalone complement to vision.

The uncertainty analysis evaluates **50 congruent samples** and **50 conflict samples**. Conflict cases deliberately mismatch the intention indicated by video-audio with that indicated by TS. Using **10 stochastic samples** at temperature **0.7**, the model’s predictive entropy is **1.28 bits** on congruent inputs and **3.15 bits** on conflict inputs. This is interpreted in the paper as evidence that the model registers multimodal inconsistency rather than collapsing deterministically onto one modality.

The paper does not report component-level ablations for alternative TS encoders, projector designs, or fusion modules. Consequently, most empirical evidence is modality-level rather than mechanism-level.

## 6. Scope, limitations, practical significance, and relation to “omni” models

Despite the term “Omni” in its name, Meow-Omni 1 is not presented as a generic any-to-any assistant. It is a domain-specific quad-modal MLLM for feline ethology whose sensory inputs are video, audio, and physiological time-series, and whose output space is intent reasoning in text. This distinguishes it sharply from general-purpose omni systems such as **HyperCLOVA X 8B Omni**, which supports text, audio, and vision as both inputs and outputs [2601.01792], **Baichuan-Omni-1.5**, which supports text, image, video, and audio input with text and audio output [2501.15368], and **M2-omni**, which takes arbitrary combinations of audio, video, image, and text as input and can generate interleaved audio, image, or text outputs [2502.18778]. Meow-Omni 1’s contribution is therefore not any-to-any generation, but native physiological grounding for intent disambiguation in a specific animal domain [2605.09152].

The paper explicitly identifies several limitations. First, **NBP is a proxy for intent**: the framework assumes internal intent will manifest in near-future behavior, which may fail for long-duration intentions or latent states that do not immediately produce action. Second, **MeowBench is synthetic** at the cross-session level, even though it is expert-verified; validation on native real-world quad-modal recordings remains open. Third, the system is a **passive observer**, not an interactive or real-time full-duplex agent. Fourth, **cross-species generalization** is unresolved.

The ethical appendix raises risks of continuous behavioral surveillance, profiling of pet health or owner behavior by employers or insurers, and clinical misinterpretation of ambiguous states. Proposed mitigations are to keep humans in the loop, use uncertainty flags for ambiguous cases, and restrict high-stakes decisions to certified veterinary professionals.

The practical significance claimed in the paper spans **veterinary diagnostics**, **animal welfare**, **wildlife conservation**, and **computational ethology**. In this framing, Meow-Omni 1 is less a general multimodal foundation model than a domain-specialized research instrument: a MiniCPM-based quad-modal MLLM designed to move from action recognition toward latent-state reasoning grounded in physiology. The authors report release of **model weights**, the **training framework**, **Meow-10K**, and **MeowBench**, positioning the project as an open-source pipeline for inter-species intent understanding [2605.09152].

Source: https://www.emergentmind.com/topics/meow-omni-1