Tri-MARF: Tri-Modal Multi-Agent Framework

• Tri-MARF is a tri-modal system that integrates multi-view 2D images, text, and 3D point clouds to deliver scalable and precise 3D object annotation.
• It employs a four-stage pipeline with specialized agents for vision-language annotation, semantic aggregation using clustering and UCB bandit selection, and gating via geometry-text alignment.
• The framework achieves state-of-the-art performance on benchmarks like Objaverse and ABO, improving metrics such as CLIPScore and throughput while ensuring robustness to occlusion and noise.

Tri-MARF (Tri-Modal Multi-Agent Response Framework) is an integrated multi-agent system for scalable 3D object annotation utilizing tri-modal data inputs: multi-view 2D images, textual descriptions, and 3D point clouds. The framework is designed to address the spatial complexity, occlusion, and viewpoint inconsistency inherent to 3D annotation tasks, surpassing the typical limitations of single-model approaches by coordinating three specialized AI agents in a modular pipeline. Tri-MARF achieves state-of-the-art performance in both annotation quality and system throughput on established benchmarks including Objaverse-LVIS, Objaverse-XL, and ABO, as measured by metrics such as CLIPScore and ViLT retrieval accuracy, and has demonstrable robustness to occlusion and input noise (Zhang et al., 7 Jan 2026).

1. System Architecture

Tri-MARF employs a four-stage architecture, each managed by functionally distinct agents:

1. Data Preparation:
   • From each 3D mesh or point cloud, six canonical 2D views ($I_v$, $v\in${front, back, left, right, top, bottom}) at $512\times512$ pixels are rendered via Open3D. Correspondingly, the raw point cloud is down-sampled to 10,000 points.
2. Agent 1: Vision-LLM (VLM) Annotation Agent
   • Qwen2.5-VL-72B-Instruct is queried for each view in three conversational turns (viewpoint identification, attribute elicitation, context integration) to generate $M=5$ candidate captions $\{C_{v,i}\}_{i=1}^M$.
   • Each caption receives a confidence score:

   $$\mathrm{Conf}(C_{v,i}) = \frac{1}{N}\sum_{t=1}^N |\log P(t \mid \text{prefix})|$$

   where $N$ denotes caption token count.

3. Agent 2: Information Aggregation Agent
   • Deduplication is performed by semantic clustering (BERT-embedding, DBSCAN).
   • Relevance weighting uses CLIP image–text cosine similarity:

   $$w_{v,i} = \frac{\exp(\cos(f_{\rm CLIP}^{\rm img}(I_v), f_{\rm CLIP}^{\rm text}(C_{v,i})))}{\sum_{k=1}^M \exp(\cos(\dots))}$$

• Composite scoring blends VLM confidence with CLIP weighting:

  $$s_{v,i} = (1-\alpha)\mathrm{Conf}(C_{v,i}) + \alpha w_{v,i},\ \alpha \in [0,1]$$

• Selection: UCB1 multi-armed bandit selects $\hat C_v$:

  $v\in$0

• Cross-view fusion prioritizes front/back for salient global caption assembly.

1. Agent 3: Gating Agent
   • Encodes the selected caption and 3D point cloud via BERT/CLIP-text and Uni3D, computes:

   $v\in$1

   and applies a threshold $v\in$2 for acceptance or rejection.

Overall, the data flow is: data preparation → VLM annotation → semantic aggregation/fusion → point-cloud gating → final caption (Zhang et al., 7 Jan 2026).

2. Tri-Modal Input Processing and Representation

Tri-MARF explicitly integrates three input modalities:

• 2D Multi-View Images: Rendered at fixed canonical viewpoints to maximize visual coverage and minimize occlusion effects. These are consumed by the VLM for localized description generation.
• Textual Descriptions: Produced by the VLM, refined through redundancy reduction and semantic selection, with CLIP-based scoring for multimodal grounding.
• 3D Point Clouds: Down-sampled representations serve as geometric priors for the gating mechanism, ensuring semantic alignment between language and 3D shape.

Fusion mechanisms operate at two levels: image–text fusion for optimal caption selection (Agent 2), and text–geometry alignment for final output validation (Agent 3).

3. Algorithmic Workflow and Pseudocode

Tri-MARF’s full workflow consists of sequential operations for each object:

$v\in$4

Inference disables reward updating in the bandit logic, while training may incorporate iterative reinforcement based on CLIPScore and VLM confidence (Zhang et al., 7 Jan 2026).

4. Dataset Benchmarks and Experimental Setup

Tri-MARF has been evaluated on the following benchmarks:

Dataset	Objects	Modality Coverage	Usage
Objaverse-LVIS	1,000	3D, 2D views, text	Core comparison (Table 1/2)
Objaverse-XL	5,000	3D, 2D views, text	Scalability, robustness
ABO	6,400	3D, 2D views, text	System stress-testing

Experimental hardware consists of a single NVIDIA A100 GPU; annotation runs with batch size 1 (one object/6 views). Key timing metrics are: rendering 0.075s/object, VLM 6–18s/object (via remote API), aggregation 0.8s, final gating 0.15s, yielding effective throughput of ~12,000 objects/hour under typical networking conditions (Zhang et al., 7 Jan 2026).

5. Performance Evaluation

Tri-MARF achieves superior results in both annotation quality and throughput:

Method	CLIPScore	ViLT R@5 (I→T / T→I)	Throughput (objs/h)
Human	82.4	40.0 / 38.5	120
Cap3D	78.6	35.2 / 33.4	8,000
ScoreAgg	80.1	37.8 / 36.0	9,000
3D-LLM	77.4	34.9 / 33.3	6,500
Tri-MARF	88.7	45.2 / 43.8	12,000

These trends generalize across datasets (Objaverse-XL, ABO), with Tri-MARF outperforming SOTA by 4–10 points in CLIPScore and 5–10 points in retrieval accuracy. Single-agent baselines (e.g., Qwen2.5 single view or Uni3D only) deliver lower performance, demonstrating the multi-agent, tri-modal formulation's benefit (Zhang et al., 7 Jan 2026).

6. Ablation Studies and Design Insights

Key ablation studies reveal:

• Agent Contributions: Multi-agent pipeline outperforms any single modality/agent (e.g., Uni3D alone yields CLIPScore 58.3, markedly below Tri-MARF).
• View Number Effect: Six rendered 2D views optimally balance annotation coverage and computational efficiency; CLIPScore peaks at this number for all compared methods.
• RL Policy Effectiveness: UCB1 outperforms Thompson Sampling, PPO, A3C, SAC, Epsilon-Greedy, MCTS in balanced trade-off between CLIPScore, retrieval accuracy, qualitative ratings, and runtime.
• Gating Threshold Setting: The gate at $v\in$3 in cosine similarity minimizes combined type I/II error, determined via truncated-Gaussian fit to 10k validation pairs.
• Occlusion Robustness: Random black-plane occlusions result in merely a 4.2% CLIPScore drop, indicating resilience provided by multi-view and gating mechanisms.

A plausible implication is that tri-modal, multi-agent systems are inherently robust to lossy or partially corrupted signals in any single modality.

7. Context, Significance, and Future Directions

Tri-MARF establishes a quantitative advance in scalable, high-fidelity 3D annotation for computer vision and robotics. Its tri-modal architecture and reinforcement learning-driven aggregation set a reference design for workflows requiring robust semantic grounding across geometric and textual representations. The multi-agent model provides intrinsic extensibility to new input modalities or output objectives. Given the reliance on large-scale VLMs and robust multi-modal encoders, future directions may involve adaptive agent specialization, more efficient cross-modal fusion algorithms, or alternative gating strategies for higher semantic reliability under adversarial or domain-shifted scenarios (Zhang et al., 7 Jan 2026).