Meow-Omni 1: Feline Intent Inference

• Meow-Omni 1 is a quad-modal model integrating video, audio, and physiological time-series to infer a cat’s internal intent.
• It uses a MiniCPM-based LLM and TS-specific encoding to overcome semantic aliasing, achieving 71.16% Top-1 accuracy on MeowBench.
• The model’s multimodal fusion highlights physiological grounding as essential for disambiguating visually similar but contextually different behaviors.

Meow-Omni 1 is a multimodal LLM for computational ethology whose stated objective is feline intent inference rather than generic recognition, captioning, or action labeling. It is presented as the first open-source, quad-modal MLLM in this domain, combining video, audio, physiological time-series, and text in a single reasoning stack. The model is motivated by “semantic aliasing,” the claim that visually or acoustically similar behaviors may correspond to different internal states depending on physiological context; in the reported evaluation on MeowBench, Meow-Omni 1 achieves 71.16% Top-1 accuracy and is described as state of the art for this benchmarked setting (Hu et al., 9 May 2026).

1. Problem definition and conceptual motivation

The central scientific problem addressed by Meow-Omni 1 is the inference of a cat’s underlying intent or internal state from multimodal observations. In the paper’s framing, observable behavior is not a sufficient proxy for latent state because identical external signals may arise from distinct causes. The canonical example is purring: a purr may indicate contentment, but it may also function as self-soothing during pain or respiratory distress. The same logic is extended to contrasts such as playful aggression versus territorial defense, where similar motion patterns may conceal different underlying drives (Hu et al., 9 May 2026).

This motivates the paper’s use of the term semantic aliasing: identical or near-identical external behavior can map to different internal states depending on context, especially physiological context. A core implication is that video and audio are treated as expressive but not causally sufficient modalities. Physiological information is introduced as a disambiguating signal, intended to ground inference in latent state rather than surface pattern matching.

The observation history up to time $t$ is written as

$\mathcal{H}_t = \{V_{1:t}, A_{1:t}, B_{1:t}\},$

where $V$, $A$, and $B$ denote vision, audio, and biometric time-series. The target mapping is

$f_\theta: \mathcal{H}_t \to \mathbb{I},$

with $\mathbb{I}$ a discrete set of intent categories. The paper further defines intention as the latent biological or cognitive drive that would maximize the probability of the animal’s next action in an unconstrained environment. As printed in the manuscript, the causal definition is given with a bracket omission, but the intended expression is that intention is the category $i \in \mathbb{I}$ maximizing the probability of the optimal next action $a^*_{t+1}$ given observed history $\mathcal{H}_t$ under $\mathcal{H}_t = \{V_{1:t}, A_{1:t}, B_{1:t}\},$0.

The label space is standardized into a unified 30-class intention taxonomy. Examples listed in the paper include Feed, Groom, Rest, Run, Shake, Trot, Walk, active_climbing, active_jumping.horizontal, active_playfight.fighting, inactive_lying.resting, maintenance_nutrition.eating, maintenance_scratching, and other_social.allogrooming. This taxonomy mixes coarse ethological categories with more granular behavior-intention labels, reflecting the paper’s emphasis on latent-state semantics rather than only motor primitives.

2. Architecture and multimodal integration

Meow-Omni 1 is built by modifying the MiniCPM-o multimodal backbone and incorporating a time-series branch adapted from Intern-S1-Pro. The model description is explicitly surgical rather than from-scratch: modality-specific encoders produce representations for video, audio, and time-series; the time-series features are projected into the language-model hidden space; modality embeddings are interleaved with text embeddings into a single multimodal token sequence; and a causal LLM performs unified reasoning and generation (Hu et al., 9 May 2026).

Across the main text and appendix, the backbone is described as based on MiniCPM-o 4.5 or the MiniCPM-V backbone. The consistent architectural point is that a MiniCPM multimodal causal LLM functions as the shared reasoning engine. The paper does not introduce new custom vision or audio encoders; those are inherited from the MiniCPM stack. The main novelty is the native inclusion of physiological time-series.

The time-series branch is adapted from Intern-S1-Pro. Its encoder is dedicated to continuous scientific time-series processing, and its outputs remain frozen during training. A linear projector maps TS hidden states into the LLM hidden dimension. The tokenizer is expanded with three TS-specific control tokens:

• <|ts_start|>
• <|ts_unit|>
• <|ts_end|>

A custom MeowOmniProcessor dynamically expands <|ts_unit|> so that the number of TS placeholder positions matches the number of hidden states emitted by the TS encoder. TS is therefore represented as a variable-length token span bracketed by start and end delimiters.

Fusion is intentionally simple. The model accepts $\mathcal{H}_t = \{V_{1:t}, A_{1:t}, B_{1:t}\},$1-dimensional TS tensors in addition to normal multimodal inputs, projects TS embeddings to the LLM hidden size, interleaves them with visual and linguistic embeddings, and passes the resulting sequence through the causal transformer. The paper does not describe a separate cross-attention stack, an explicit alignment head, or a dedicated fusion module. A plausible implication is that the model’s multimodal integration strategy is closer to token-space co-embedding than to specialized cross-modal routing.

Handling of incomplete sensing is also explicit. The model can process V+A+TS, V+A, V+TS, A+TS, or single-modality inputs. If a modality is missing, it is simply omitted from the token sequence; no placeholder tokens are inserted for absent modalities. This design is presented as a mechanism for robustness under heterogeneous real-world sensing and mixed-modality training.

3. Data resources, preprocessing, and benchmark design

The primary training dataset is Meow-10K, reported as containing 10,831 samples. It is intentionally heterogeneous rather than restricted to perfectly synchronized quad-modal recordings. The dataset includes TS-only samples from accelerometer datasets, video-only clips, standalone audio clips, synchronized audio-video clips, and a small number of synthetic quad-modal samples. After balancing, the abundant TS-only pool is subsampled to 2,000 examples so that it does not dominate the training distribution (Hu et al., 9 May 2026).

The TS pipeline begins from two peer-reviewed accelerometer datasets. Original signals are sampled at 30 Hz or 60 Hz, then aggregated into second-level signals by averaging within each second. Windows of 5 s, 7 s, 10 s, and 15 s are used, with prediction targets taken 1, 2, 3, or 5 seconds ahead to form a Next-Behaviour Prediction (NBP) dataset. The full TS pool used in stage-1 alignment contains 383,853 labeled TS samples.

Video preprocessing uses an open-source dataset pipeline cited to Bain et al. and employs Qwen2-VL-72B-Instruct for coarse action detection, temporal localization, verification, clip extraction, and sequence caption/action-label generation. These action labels are mapped by Qwen3.5-35B into the unified 30-class taxonomy and reviewed by experts.

Audio comes from two sources. First, synchronized AudioSet-derived A/V clips are filtered with an AST classifier for cat-related sounds. The retention policy is explicit: retain if score $\mathcal{H}_t = \{V_{1:t}, A_{1:t}, B_{1:t}\},$2, discard if $\mathcal{H}_t = \{V_{1:t}, A_{1:t}, B_{1:t}\},$3, and for scores between $\mathcal{H}_t = \{V_{1:t}, A_{1:t}, B_{1:t}\},$4 and $\mathcal{H}_t = \{V_{1:t}, A_{1:t}, B_{1:t}\},$5, apply stationary noise reduction with suppression factor 0.85 and recheck. Those clips are then processed by Qwen2.5-VL-7B to generate audio-focused captions and a 30-class score distribution. Second, standalone audio clips from Freesound and taozi555/cat_class are caption-expanded by Qwen3.5-35B-A3B-GPTQ-Int4; these do not receive direct intention labels and instead use descriptive targets.

Each training sample carries a natural-language query-response pair and, where available, an intention label from the 30-class taxonomy. For TS prompts, the paper reports prompt diversification with DeepSeek, GPT, and Gemini.

Evaluation is conducted on MeowBench, a held-out benchmark for multimodal intent reasoning. Because naturally synchronized quad-modal feline datasets do not exist, MeowBench is constructed by intent-matched synthesis: a synchronized video-audio clip from one session is paired with a TS sample from another session, provided that both share the same intention label. This is synthetic at the cross-session level, but intended to preserve semantic consistency. Candidate items were verified by eight Professional Feline Ethologists, divided into groups of 3, 3, and 2. From 645 initial candidates, the process yielded 527 expert-verified samples. Each item is cast as a multiple-choice question with 1 correct intention and 3 distractors, and the principal metric is Top-1 Accuracy.

4. Training procedure and formal objectives

Training is explicitly two-stage. Stage 1 is a projector-alignment phase intended to align TS encoder outputs with the LLM latent space. Stage 2 is a multimodal-specialization phase intended to adapt the LLM backbone to feline multimodal intent reasoning (Hu et al., 9 May 2026).

In Stage 1, only the TS projector is trainable, while both the LLM backbone and TS encoder are frozen. The data source is the 383,853 labeled TS samples. The objective is cross-entropy on predicted next behavior class. Optimization uses AdamW with learning rate $\mathcal{H}_t = \{V_{1:t}, A_{1:t}, B_{1:t}\},$6, weight decay $\mathcal{H}_t = \{V_{1:t}, A_{1:t}, B_{1:t}\},$7, batch size 1 per device, gradient accumulation 2, cosine annealing with 3% linear warmup, maximum 10 epochs, and early stopping with patience 1 epoch. The best checkpoint is reported at epoch 2 and named Meow-Omni 1 Aligned.

In Stage 2, the trainable component is the LLM backbone only; all encoders and projectors are frozen. Training uses Meow-10K with 10,831 samples. The objective is next-token prediction loss over the multimodal sequence, including special tokens. Optimization again uses AdamW, with learning rate $\mathcal{H}_t = \{V_{1:t}, A_{1:t}, B_{1:t}\},$8, weight decay $\mathcal{H}_t = \{V_{1:t}, A_{1:t}, B_{1:t}\},$9, batch size 1 per device, gradient accumulation 2, cosine decay with 3% warmup, maximum 5 epochs, and early stopping with patience 1 epoch. The appendix identifies the final checkpoint at epoch 1, whereas the main text describes it as obtained after 2 epochs.

The hardware configuration is reported as 1 node with 8 × NVIDIA H200 80 GB GPUs, interconnected by NVLink. Missing modalities are omitted entirely from the sequence; the paper does not propose a specialized asynchronous alignment algorithm. A plausible implication is that mixed-modality robustness is achieved primarily through omission-aware serialization and heterogeneous training, not through explicit temporal synchronization machinery.

The manuscript provides a high-level supervised objective,

$V$0

which expresses likelihood maximization of the correct intent label given encoded multimodal history. The paper also defines predictive entropy for uncertainty quantification under repeated stochastic decoding:

$V$1

where $V$2 is the set of distinct generated intention categories and $V$3 is the empirical class probability. In the reported uncertainty experiment, the model uses $V$4 stochastic forward passes at generation temperature $V$5.

The paper is mathematically light beyond these expressions. It does not provide explicit formulas for contrastive alignment, fusion, cross-attention, auxiliary physiological grounding, or detailed temporal modeling.

5. Empirical results, ablations, and uncertainty analysis

The headline result is 71.16% Top-1 accuracy on MeowBench. The main baseline comparison includes unimodal, multimodal, and omni-modal systems. Reported unimodal baselines are Audio SOTA at 36.86%, Video SOTA (Qwen3.5-122B) at 61.95%, and TS SOTA at 48.98%. Using Qwen3.5-Omni-Plus as an omni-modal baseline, the reported accuracies are 65.36% for V+A, 66.21% for V+TS, 42.15% for TS+A, and 66.89% for V+A+TS. Meow-Omni 1 reaches 71.16% with V+A+TS, a gain of 4.27 percentage points over the strongest omni baseline, described in the paper as 4.3 points (Hu et al., 9 May 2026).

Ablation by modality masking clarifies the contribution of each signal. Within the Meow-Omni 1 architecture, Audio only yields 51.88%, Vision only 69.97%, and TS only 55.63%. For bimodal settings, V+A gives 68.43%, V+TS 70.82%, and TS+A 54.95%. The full V+A+TS model reaches 71.16%. The most consequential finding is that time-series contributes more useful disambiguation than audio when added to vision: vision alone is 69.97%, vision plus audio drops slightly to 68.43%, while vision plus TS increases to 70.82%.

This pattern is the paper’s main empirical support for physiological grounding. It suggests that, on this benchmark, vision carries most of the raw discriminative signal, TS supplies the critical disambiguating increment, and audio contributes primarily in the fully integrated setting rather than as a strong standalone complement to vision.

The uncertainty analysis evaluates 50 congruent samples and 50 conflict samples. Conflict cases deliberately mismatch the intention indicated by video-audio with that indicated by TS. Using 10 stochastic samples at temperature 0.7, the model’s predictive entropy is 1.28 bits on congruent inputs and 3.15 bits on conflict inputs. This is interpreted in the paper as evidence that the model registers multimodal inconsistency rather than collapsing deterministically onto one modality.

The paper does not report component-level ablations for alternative TS encoders, projector designs, or fusion modules. Consequently, most empirical evidence is modality-level rather than mechanism-level.

6. Scope, limitations, practical significance, and relation to “omni” models

Despite the term “Omni” in its name, Meow-Omni 1 is not presented as a generic any-to-any assistant. It is a domain-specific quad-modal MLLM for feline ethology whose sensory inputs are video, audio, and physiological time-series, and whose output space is intent reasoning in text. This distinguishes it sharply from general-purpose omni systems such as HyperCLOVA X 8B Omni, which supports text, audio, and vision as both inputs and outputs (Team, 5 Jan 2026), Baichuan-Omni-1.5, which supports text, image, video, and audio input with text and audio output (Li et al., 26 Jan 2025), and M2-omni, which takes arbitrary combinations of audio, video, image, and text as input and can generate interleaved audio, image, or text outputs (Guo et al., 26 Feb 2025). Meow-Omni 1’s contribution is therefore not any-to-any generation, but native physiological grounding for intent disambiguation in a specific animal domain (Hu et al., 9 May 2026).

The paper explicitly identifies several limitations. First, NBP is a proxy for intent: the framework assumes internal intent will manifest in near-future behavior, which may fail for long-duration intentions or latent states that do not immediately produce action. Second, MeowBench is synthetic at the cross-session level, even though it is expert-verified; validation on native real-world quad-modal recordings remains open. Third, the system is a passive observer, not an interactive or real-time full-duplex agent. Fourth, cross-species generalization is unresolved.

The ethical appendix raises risks of continuous behavioral surveillance, profiling of pet health or owner behavior by employers or insurers, and clinical misinterpretation of ambiguous states. Proposed mitigations are to keep humans in the loop, use uncertainty flags for ambiguous cases, and restrict high-stakes decisions to certified veterinary professionals.

The practical significance claimed in the paper spans veterinary diagnostics, animal welfare, wildlife conservation, and computational ethology. In this framing, Meow-Omni 1 is less a general multimodal foundation model than a domain-specialized research instrument: a MiniCPM-based quad-modal MLLM designed to move from action recognition toward latent-state reasoning grounded in physiology. The authors report release of model weights, the training framework, Meow-10K, and MeowBench, positioning the project as an open-source pipeline for inter-species intent understanding (Hu et al., 9 May 2026).