Multi-Granular Perception Agent (MGPA)

• MGPA is a perception framework that unifies heterogeneous granularities by integrating differing instruction types and prediction scales across image, video, and multi-agent settings.
• It leverages dynamic query generation, multi-scale transformer decoders, and cross-modal attention to support tasks like detection, segmentation, and interactive video understanding.
• Empirical results demonstrate competitive performance in segmentation, detection, and collaborative 3D tasks while addressing challenges in communication efficiency and feature alignment.

Searching arXiv for the cited MGPA-related papers and closely related terminology. Multi-Granular Perception Agent (MGPA) denotes a class of perception systems that organizes sensing, representation, and action across more than one perceptual granularity. In the cited literature, the term is instantiated in three closely related ways: as a VLLM-centric image perception framework that unifies word-based and sentence-based instructions with box-level and mask-level predictions; as the perception module of an interactive long-video understanding system that alternates broad search, temporal refinement, and spatial questioning; and as a communication-aware collaborative perception design in which multiple agents fuse aligned intermediate features under global and local mutual-information constraints (Xiang et al., 22 Jul 2025, Li et al., 29 Sep 2025, Su et al., 2024). This suggests that MGPA is best understood as an architectural principle for unifying heterogeneous perceptual granularities rather than as a single fixed model family.

1. Taxonomy of granularity

In image-domain MGPA, multi-granular perception is defined by two axes: prediction granularity and instruction granularity. Prediction granularity distinguishes box-level outputs from mask-level outputs. Instruction granularity distinguishes word-based inputs, such as a list of category names, from sentence-based inputs, such as a full referring expression. Their Cartesian product yields four quadrants: Word × Box, Word × Mask, Sentence × Box, and Sentence × Mask. Representative tasks are object detection, open-vocabulary detection, grounding of category names, panoptic/semantic/instance segmentation, referring expression comprehension (REC), and referring expression segmentation (RES) (Xiang et al., 22 Jul 2025).

In long-video MGPA, granularity is temporal and spatial rather than limited to output type. The cited design uses a video-level overview, frame-level selection, segment/clip-level refinement, and region/object-level extraction. The corresponding operations are Quick Preview, Divergent Search, Temporal Focus, and Spatial Focus. The Working Memory stores actions and observations across iterations, making granularity an explicit control variable in an interactive policy rather than a static architectural feature (Li et al., 29 Sep 2025).

In multi-agent MGPA, granularity is formulated in terms of information preservation across collaborative representations. Local granularity corresponds to voxel/cell-level correlations; global granularity corresponds to scene-level dependencies; and the CMiMC/CMiMNet blueprint further describes an intermediate region-level extension. In this setting, multi-granularity concerns how collaborative views preserve discriminative information from pre-collaboration features at several spatial scopes while remaining effective for downstream 3D detection (Su et al., 2024).

Setting	Granularity axes	Representative functions
Image VLLM	word vs. sentence; box vs. mask	detection, grounding, panoptic segmentation, REC, RES
Long video	overview, frame, clip, region	Quick Preview, Divergent Search, Temporal Focus, Spatial Focus
Multi-agent collaboration	local, global, and region-level extension	aligned fusion, MVMI supervision, 3D detection

A common misconception is to equate “multi-granular” with ordinary multi-scale feature pyramids. The cited work uses the term more broadly: it covers instruction granularity and output granularity in image perception, temporal and spatial clue exploration in video understanding, and local/global mutual-information preservation in collaborative perception.

2. MVP-LM as an image-domain MGPA

MVP-LM realizes MGPA in a single architecture composed of a Swin-Base vision encoder, a connector that aligns image features to the LLM embedding space, a Phi-1.5 backbone, and a multi-granularity decoder described as an OpenSeeD-style transformer decoder with shared heads for boxes and masks (Xiang et al., 22 Jul 2025). The vision encoder extracts multi-scale image features $F = \{F_1, F_2, \dots, F_L\}$. The unified prompt, consisting of a task description and an instruction, is processed together with connector-projected visual tokens. The LLM then generates a response in the format "[image caption]. The perception result is <PER>", and the hidden state of the special summary token <[PER](https://www.emergentmind.com/topics/preference-informed-edge-reweighting-per)> serves as the base summary for query construction.

Instruction embeddings differ by granularity. For word-based inputs, the model passes the comma-separated category sentence through the LLM and average-pools the output embeddings corresponding to each category token to form per-word embeddings $t_k$. For sentence-based inputs, it average-pools the output embeddings of the full referring sentence into a single vector $t$. This dual instruction support is central to the model’s ability to cover all four quadrants in one system.

The dynamic query generation mechanism, termed query enhancement, expands the hidden state $h_{\text{PER}}$ into $N$ base queries and then augments them with language-guided residuals selected from image features. For the sentence case, the reported formulation is

$$Q_{\text{base}} = \text{MLP}(h_{\text{PER}}) \in \mathbb{R}^{N \times d},$$

$$t' = \text{MLP}_{\text{text}}(t), \qquad v_i' = \text{MLP}_{\text{vis}}(v_i),$$

$$\text{sim}_i = \frac{v_i' \cdot t'}{\|v_i'\| \cdot \|t'\|},$$

$I_N = \text{argsort}_i(\text{sim}_i)[:N],$

$$Q_{\text{final}}(j) = Q_{\text{base}}(j) + r_j, \qquad r_j = v'_{I_N(j)}.$$

The multi-granularity decoder applies MSDeformable attention over multi-scale features. Shared heads then output cross-modal affinity, bounding box coordinates, and mask logits. Hungarian bipartite matching assigns predictions to targets at each layer, denoising queries are used to stabilize training, and inference uses only the final decoder layer. For panoptic segmentation, the implementation adds 100 learnable “stuff” queries to the default 100 queries.

This architecture is positioned against specialist systems such as OWL, GLIP, Grounding DINO, X-Decoder, OpenSeeD, MaskDINO, and VLT/LAVT, as well as VLLM generalists such as QwenVL, InternVL, LISA, PixelLM, Glamm, OMG-LLaVA, PSALM, and HyperSeg. The stated distinction is that those systems cover subsets of the four quadrants, whereas MVP-LM covers all four in one model (Xiang et al., 22 Jul 2025).

3. Data unification, supervision, and optimization

MVP-LM trains the MGPA formulation through a CoT-inspired dataset unification strategy. Each supervised fine-tuning sample is converted into a QA pair with three components: a task description, an instruction, and the response "[image caption]. The perception result is <PER>". Word-based task descriptions use the template “Please identify all objects according to the given phrase list. This is all the candidate phrases.” Sentence-based task descriptions use “Please identify the target according to the following instruction.” The hidden states of <PER> provide the base queries for downstream perception (Xiang et al., 22 Jul 2025).

The data pipeline further uses multi-caption auto-labeling. A diverse set of open-source VLLMs—VILA-3B/13B and InternVL2-8B/26B—pre-generates multiple captions per image. Refinement filters captions by length and entropy and strips uncertain verbal hedges such as “indicate”, “might”, “may”, and “imply” to reduce hallucinations. During training, one of four captions is randomly selected for each sample, word order is shuffled each iteration to break positional biases, and a subset of negative category names not present in the image is dynamically added within sequence-length constraints.

The total loss is reported as

$t_k$0

For the experimental stage-2 setting, the paper uses

$t_k$1

The language-modeling term is next-token prediction cross-entropy on the response sequence. The word-based term is per-query cross-entropy over the category vocabulary, and the sentence-based affinity term is binary cross-entropy indicating whether a query corresponds to the described instance. The mask loss combines BCE and Dice, while the box loss combines $t_k$2 and GIoU. The reported expressions are

$t_k$3

$t_k$4

$t_k$5

$t_k$6

$t_k$7

$t_k$8

$t_k$9

Training is two-stage. Stage 1 trains only the connector on CC3M with AdamW, learning rate $t$0, batch size 128, 4650 steps, cosine decay, warmup ratio 0.03, and weight decay 0.0. Stage 2 trains the connector, LLM, and multi-granularity decoder with the vision encoder frozen, using AdamW, learning rate $t$1, batch size 64, 80k steps, cosine decay, warmup ratio 0.03, and weight decay 0.05. The Stage 2 data mixture is COCO Panoptic 33.3%, RefCOCO 33.3%, Objects365 16.7%, and GoldG 16.7%, and images are resized so that the longer side is 1024 and the shorter side is padded to 1024 (Xiang et al., 22 Jul 2025).

4. Long-video MGPA in the Perception–Reflection loop

In CogniGPT, MGPA is the perception engine in an interactive loop with the Verification-Enhanced Reflection Agent (VERA). Its objective is to progressively explore a minimal set of informative and reliable task-related clues sufficient to answer a query $t$2, where task-related clues include frames, clips, and localized visual facts such as objects, attributes, relations, events, and timestamps (Li et al., 29 Sep 2025).

The architecture uses EVA-CLIP-8B for text-video embedding, retrieval, and clustering. Captioning and VQA are dataset-dependent: LaViLa for EgoSchema, CogAgent for NExT-QA, and LLaVA-NeXT for Video-MME and MovieChat. The toolkit has three core perception actions. Divergent Search performs broad temporal exploration and selects diverse, high-relevance frames. Temporal Focus clusters clips within a selected interval via K-means into $t$3 semantic clusters and captions representative clips. Spatial Focus performs VQA on specific frames to recover fine-grained facts not captured in captions. Working Memory is initialized by Quick Preview, which clusters video features into $t$4 groups, captions cluster centroids, records timestamps, and stores the resulting overview.

The policy for action selection is

$t$5

where $t$6 is the selected tool, $t$7 is current memory, and $t$8 is VERA’s guidance. Divergent Search computes cosine similarities between a sub-query $t$9 and sampled frames $h_{\text{PER}}$0:

$h_{\text{PER}}$1

These similarities are smoothed by sliding-window averaging,

$h_{\text{PER}}$2

and thresholded by their mean,

$h_{\text{PER}}$3

The algorithm partitions time into peak and valley regions, chooses the frame with highest $h_{\text{PER}}$4 in each peak region, and retains the overall top-$h_{\text{PER}}$5 frames. This differs from plain top-$h_{\text{PER}}$6 retrieval by explicitly favoring diverse, causally relevant moments. Memory is updated as

$h_{\text{PER}}$7

VERA adds two functions. In Key Observation Verification, it detects whether the latest observation contains critical facts for $h_{\text{PER}}$8, generates 2–3 verification questions at relevant timestamps, queries the VLM, and cross-verifies consistency between answers and the observed fact. Only reliable facts are retained. In Evaluation and Planning, it performs chain-of-thought assessment of information sufficiency, identifies gaps, decides whether to continue or terminate, and produces guidance for the next MGPA step. The maximum iteration cap is $h_{\text{PER}}$9. No fixed numeric threshold is mandated for verification; trustworthiness is judged by consistency under cross-verification.

The implementation uses GPT-4 for the main results, with ablations on Mistral-8x7B, GPT-3.5, and DeepSeek-V3. Hyperparameters are dataset-specific: for EgoSchema, NExT-QA, and MovieChat, $N$0, $N$1, and $N$2; for Video-MME, $N$3, $N$4, and $N$5. The FPS settings are 1 for EgoSchema and NExT-QA, 0.5 for MovieChat, and 0.125 for Video-MME. Hardware is reported as up to 4×NVIDIA A6000 (Li et al., 29 Sep 2025).

5. Collaborative MGPA and mutual-information-guided fusion

The CMiMC/CMiMNet framework provides a principled blueprint for an MGPA in multi-agent collaborative perception. Each agent $N$6 encodes raw sensor measurements into a pre-collaboration feature $N$7 using a backbone $N$8. For the LiDAR setting in V2X-Sim 1.0, the point cloud is cropped to $N$9 meters and voxelized into a BEV map of size $$Q_{\text{base}} = \text{MLP}(h_{\text{PER}}) \in \mathbb{R}^{N \times d},$$0 with voxel dimensions $$Q_{\text{base}} = \text{MLP}(h_{\text{PER}}) \in \mathbb{R}^{N \times d},$$1 m. Features from agent $$Q_{\text{base}} = \text{MLP}(h_{\text{PER}}) \in \mathbb{R}^{N \times d},$$2 are aligned into agent $$Q_{\text{base}} = \text{MLP}(h_{\text{PER}}) \in \mathbb{R}^{N \times d},$$3’s local frame via $$Q_{\text{base}} = \text{MLP}(h_{\text{PER}}) \in \mathbb{R}^{N \times d},$$4, where $$Q_{\text{base}} = \text{MLP}(h_{\text{PER}}) \in \mathbb{R}^{N \times d},$$5 denotes pose information (Su et al., 2024).

The collaboration encoder $$Q_{\text{base}} = \text{MLP}(h_{\text{PER}}) \in \mathbb{R}^{N \times d},$$6 produces a spatial weight matrix $$Q_{\text{base}} = \text{MLP}(h_{\text{PER}}) \in \mathbb{R}^{N \times d},$$7 and fuses aligned evidence at voxel resolution:

$$Q_{\text{base}} = \text{MLP}(h_{\text{PER}}) \in \mathbb{R}^{N \times d},$$8

where $$Q_{\text{base}} = \text{MLP}(h_{\text{PER}}) \in \mathbb{R}^{N \times d},$$9 denotes element-wise multiplication. This formulation is explicitly fine-grained: it permits different agents to contribute differently at different spatial indices.

The core supervisory idea is Multi-View Mutual Information (MVMI), which ties the collaborative view $$t' = \text{MLP}_{\text{text}}(t), \qquad v_i' = \text{MLP}_{\text{vis}}(v_i),$$0 to all aligned individual views. The global term is

$$t' = \text{MLP}_{\text{text}}(t), \qquad v_i' = \text{MLP}_{\text{vis}}(v_i),$$1

and the local term averages mutual information between the collaborative view and spatial patches:

$$t' = \text{MLP}_{\text{text}}(t), \qquad v_i' = \text{MLP}_{\text{vis}}(v_i),$$2

For agent $$t' = \text{MLP}_{\text{text}}(t), \qquad v_i' = \text{MLP}_{\text{vis}}(v_i),$$3, MVMI is

$$t' = \text{MLP}_{\text{text}}(t), \qquad v_i' = \text{MLP}_{\text{vis}}(v_i),$$4

CMiMNet estimates these terms with JS-divergence-based discriminators. The estimator is

$$t' = \text{MLP}_{\text{text}}(t), \qquad v_i' = \text{MLP}_{\text{vis}}(v_i),$$5

with $$t' = \text{MLP}_{\text{text}}(t), \qquad v_i' = \text{MLP}_{\text{vis}}(v_i),$$6 as softplus. Global and local estimators reuse this form, and the network maximizes the MVMI lower bound over global and local discriminators. The overall objective combines task losses and MI supervision:

$$t' = \text{MLP}_{\text{text}}(t), \qquad v_i' = \text{MLP}_{\text{vis}}(v_i),$$7

$$t' = \text{MLP}_{\text{text}}(t), \qquad v_i' = \text{MLP}_{\text{vis}}(v_i),$$8

The detailed blueprint extends this design to tri-granular MGPA by adding a mid-level branch with region-level patches, hierarchical fusion modules, and multi-scale detection heads. The resulting loss is

$$t' = \text{MLP}_{\text{text}}(t), \qquad v_i' = \text{MLP}_{\text{vis}}(v_i),$$9

Communication efficiency is treated as part of the MGPA design. With compression ratio $$\text{sim}_i = \frac{v_i' \cdot t'}{\|v_i'\| \cdot \|t'\|},$$0, communication volume is reported to drop from roughly $$\text{sim}_i = \frac{v_i' \cdot t'}{\|v_i'\| \cdot \|t'\|},$$1 KB to $$\text{sim}_i = \frac{v_i' \cdot t'}{\|v_i'\| \cdot \|t'\|},$$2 KB while maintaining competitive AP. The blueprint also describes spatial masking, channel compression, and multi-scale selective transmission. This makes MGPA not only a perception architecture but also a bandwidth-aware coordination mechanism in distributed settings (Su et al., 2024).

6. Empirical profile, trade-offs, and limitations

The reported results show that MGPA-style systems are evaluated not by a single universal metric but by task-dependent metrics tied to the relevant granularity. In image perception, the reported metrics include PQ and mIoU for segmentation, cIoU for referring segmentation, and standard REC accuracy. In collaborative 3D detection, the relevant metrics are AP at IoU thresholds. In long-video understanding, efficiency is measured jointly by accuracy and the average number of frames requiring captioning or QA (Xiang et al., 22 Jul 2025, Li et al., 29 Sep 2025, Su et al., 2024).

Realization	Benchmark / setting	Reported result
MVP-LM	COCO Panoptic (Swin-B)	PQ 56.1, mIoU 66.8
MVP-LM	RefCOCO REC (1.3B)	93.5/94.5/91.6 on val/testA/testB
CogniGPT MGPA+VERA	EgoSchema	69.2% with 11.2 frames
CogniGPT MGPA+VERA	NExT-QA	76.8% with 10.9 frames
CMiMC-based MGPA	V2X-Sim 1.0	AP 61.58% at IoU=0.5 and 55.80% at IoU=0.7
CMiMC-based MGPA	V2X-Sim 1.0 with 1/32 compression	AP 59.78% at IoU=0.5 and 54.04% at IoU=0.7

For MVP-LM, representative image-domain results include COCO Panoptic PQ 56.1 and mIoU 66.8, open-set segmentation results of ADE-OV PQ 19.4 and mIoU 20.5, Citys-OV PQ 35.3 and mIoU 44.1, and PAS20-OV mIoU 85.7. In sentence-based RES, the reported scores are RefCOCO 83.6/85.1/82.5, RefCOCO+ 73.9/76.8/65.4, RefCOCOg (U) 75.1/75.6, and gRefCOCO 37.6/50.4/49.9. In sentence-based REC, MVP-LM (1.3B) reports RefCOCO 93.5/94.5/91.6, RefCOCO+ 84.9/88.8/79.3, and RefCOCOg 86.7/87.4, and is described as second only to DeepSeek-VL2 (200B+) on REC. The ablations show that adding O365 and then GoldG improves RefCOCO val cIoU from 77.6 to 81.8 to 83.6, that generated image captions before <PER> improve both RefCOCO and COCO, that RefCOCO prefers fewer queries while COCO prefers 100 queries, and that fixed learnable queries help RefCOCO but hurt COCO (Xiang et al., 22 Jul 2025).

For CogniGPT, the full MGPA+VERA system reports 69.2% with 11.2 frames on EgoSchema, 54.7% with 18.3 frames on Video-MME, 76.8% with 10.9 frames on NExT-QA, and 95.3% with 10.5 frames on MovieChat. On EgoSchema, the runtime analysis reports Embedding 4.0s, Retrieval 0.1s, Captioning 10.3s, QA 1.5s, LLM response time 23.6s with 6.8 calls and 392.9 tokens, for a Total of 39.5s. The ablation on NExT-QA shows DS only at 72.0 average accuracy, DS+TF at 74.3, TF+SF at 71.8, and DS+TF+SF at 76.8. The agentic ablation on EgoSchema reports ReAct at 63.2 with 14.7 frames, “Ours without Verification” at 64.8 with 8.1 frames, and full MGPA+VERA at 69.2 with 11.2 frames, so verification adds +4.4% accuracy with approximately 3.1 more frames. Failure cases include missed clues outside narrowly selected spans and inconclusive verification when VQA or captions are weak (Li et al., 29 Sep 2025).

For CMiMC-based collaborative MGPA, the reported V2X-Sim 1.0 results are AP 61.58% at IoU=0.5 and 55.80% at IoU=0.7, improving over DiscoNet by 3.08% and 4.44%. With 1/32 compression, AP is 59.78% and 54.04%. Under pose noise, [email protected] is 58.80% for standard deviation 0.2 m and 58.02% for 0.4 m. The ablations indicate that removing local MI harms detection, that global-only optimization tends to overfit scene-level statistics and underperform in crowded or occluded zones, and that AP degrades smoothly as compression increases. Reported limitations include sensitivity to misalignment, distribution shift from simulation to real-world domains, extreme sparsity and high agent count, and latency from alignment and multi-branch fusion (Su et al., 2024).

The future directions described across the three strands are consistent. For MVP-LM, proposed extensions include temporal/video perception, 3D perception, interactive agent behaviors, and reasoning-enhanced post-training. For CMiMC-based MGPA, the cited directions are adaptive granularity selection, dynamic agent weighting, curriculum on MI terms, uncertainty-aware fusion, occupancy/semantic maps, and temporal negatives/positives. For long-video MGPA, proposed extensions include learning or RL-based action policies, learned uncertainty/calibration for selective verification, and integration with stronger VLMs. Taken together, these proposals indicate that MGPA remains an active design space centered on how perception should be decomposed, verified, and coordinated under practical compute, bandwidth, and reliability constraints.