Tri-task & Chain of Modality Methods

• Tri-task learning and chain-of-modality methods are unified frameworks that integrate multiple tasks and modalities through shared backbones and specialized branches.
• They employ transformer decoders and fusion blocks to align heterogeneous data (e.g., LiDAR, image, text) and achieve improved metrics in tasks like 3D perception and translation.
• Adaptive weighting and multi-level gradient calibration mitigate task conflicts and modality bias, leading to measurable gains in mAP, mIoU, and BLEU scores.

Tri-task learning and chain of modality methods refer to approaches that enable simultaneous optimization and effective interaction across three or more tasks and modalities within a unified machine learning framework. These paradigms address the complexities of multi-modal, multi-task problems, often in areas such as 3D perception, language-vision-audio integration, translation, and retrieval. The following sections synthesize key principles, architectures, optimization strategies, and empirical findings from recent arXiv literature, in particular “FULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level Gradient Calibration” (Huang et al., 2023), supplemented by findings in tri-modal translation, retrieval, and fusion.

1. Architectures for Tri-Task and Multi-Modality Joint Learning

Tri-task learning frameworks integrate three distinct task heads above a shared backbone designed to support multiple data modalities. A canonical backbone for 3D perception tasks deploys modality-specific branches (e.g., LiDAR: VoxelNet; image: Swin-T), projecting outputs into a unified space—commonly a bird’s-eye view (BEV) feature plane. Concatenated modality features are refined by compact fusion blocks (such as two-layer FPNs). Each downstream task (e.g., object detection, static map segmentation, foreground segmentation) is managed by dedicated transformer-based decoders attached atop the fused feature (Huang et al., 2023). The ability to generalize these architectures to tri-task or higher regimes is contingent on suitable embedding design and capacity balancing for each modality.

Similar principles are observed in tri-modal translation and retrieval, with shared encoders handling tokenized inputs for speech, image, and text (Kim et al., 2024), or specialized sub-networks (text/video/motion encoders for motion retrieval (Yin et al., 2024), image/text encoders for hybrid retrieval (Zhao et al., 2022)). Synergy-CLIP (Cho et al., 30 Apr 2025) demonstrates equal-scale tri-modal data alignment, leveraging modality-specific encoders and a joint contrastive projection space for vision, text, and audio.

2. Multi-Level Gradient Calibration and Chain-of-Modality Optimization

FULLER introduces a robust multi-level gradient calibration protocol, with inter-task and intra-modality balancing during back-propagation. After loss computation for each task ($L_i$), gradients on the shared backbone’s final layer ($g_i = \nabla_{shared_L} L_i$) are normalized using solvers such as IMTL_G, yielding weights $\alpha_i$ for constructing a composite loss $L_{total} = \sum_i \alpha_i L_i$. This regularizes task dominance and resolves task conflict (Huang et al., 2023).

Intra-gradient calibration proceeds at the fusion block’s first convolution, isolating LiDAR and image branch parameters ($\theta^{(F)}_{lid}$, $\theta^{(F)}_{img}$), and computing per-branch gradient magnitudes. Gates $w_{lid}, w_{img}$ derived from ratio $\gamma_{mod}$ and smoothed via momentum, are used to scale back-propagated gradients, equalizing optimization pressure across modalities. The calibration chain—inter-task at the backbone, intra-modality at the fusion interface—ensures neither task nor modality overwrites the others.

This “chain-of-modality” perspective is broadly extensible: in TMT (Kim et al., 2024), joint optimization over six translation directions is performed by uniform summing of losses, achieving balanced improvements across all modalities. Synergy-CLIP (Cho et al., 30 Apr 2025) aligns three modalities via symmetric NT-Xent objectives, with downstream tasks benefiting from equal-pair weighting. Hybrid-modality retrieval (Zhao et al., 2022) leverages three-stage progression, propagating feature learning and adaptive weighting to dynamically allocate attention per query instance.

3. Task and Modality Aggregation Techniques

Task and modality aggregation encompasses both architectural and optimization-level strategies:

• Multi-head aggregation: Each modality branches to its own head for specific prediction tasks (e.g., detection, segmentation), with decoders tailored to output queries or masks in transformer variants (Huang et al., 2023).
• Gradient aggregators: Solvers such as IMTL_G or GradNorm compute weights or scaling factors so that gradient magnitudes are balanced, handling tri-task (or $T$-task) scenarios with pairwise norm/direction comparisons.
• Adaptive weighting: Hybrid retrieval models learn adaptive per-query weights via softmax over joint/query features, guided by pseudo-label supervision from auxiliary retrieval models (Zhao et al., 2022).
• Contrastive alignment: Embedding learning via contrastive InfoNCE or KL losses across modality pairs (text-motion, motion-video, etc.), reinforced by negative filtering for hard negative exclusion (Yin et al., 2024).
• Shared and disentangled experts: DiME architecture explicitly learns three experts—textual, visual, and alignment—each with tailored objectives (triplet-margin, cosine consistency), fused via a gating network calibrated over encoder outputs (Xie et al., 29 Jan 2026).

4. Empirical Outcomes and Cross-Task/Modality Transfer

Reported results empirically validate the superiority of tri-task and chain-of-modality calibration frameworks over naïve or manually coordinated baselines:

Method/Domain	Task(s)	Key Metric(s) (Best Model)	Absolute Improvement	Source
FULLER, NuScenes	Detection/Seg	mAP 60.5, mIoU 58.4	+1.4 mAP, +14.4 mIoU	(Huang et al., 2023)
FULLER, NuScenes	Det/Seg/FG-Seg	mAP 58.6, map-mIoU 57.1	+10.2 map-mIoU	(Huang et al., 2023)
TMT, COCO	i→t, t→i, s→i, etc.	BLEU-4, CIDEr, CLIP-score	+2.5 BLEU-4, +1.6 CLIP	(Kim et al., 2024)
Hybrid Retrieval	Fashion-IQ, Shoes	Rmean 58.68	+14.1 Absolute	(Zhao et al., 2022)
LAVIMO	HumanML3D, KIT-ML	R@1 (Text→Motion) 10.16	+2.93 R@1	(Yin et al., 2024)
Synergy-CLIP	CIFAR-10 (ZS)	Top-1 Acc 86.2%	+2.2%	(Cho et al., 30 Apr 2025)

Qualitative analysis in FULLER reveals that intra-modal calibration remedies sensor bias (e.g., map segmentation improves dramatically when gradient pressure is balanced toward image inputs), and that balancing across both tasks and modalities mitigates shortcut learning or suboptimal task focus. Ablation studies consistently show that joint calibration yields larger reductions in multi-task loss deltas and closer approach of all imbalance ratios ($\gamma_{task}, \gamma_{mod}$) to unity.

5. Design Considerations and Practical Implications

• Loss Balancing: Calibration frameworks remove the requirement for manual loss weighting; adaptive gradient normalization achieves “fair” updates organically.
• Data Curation: Balanced tri-modal datasets (e.g., VGG-sound+ (Cho et al., 30 Apr 2025)) are critical; model-generated captions substantially improve alignment and downstream metrics.
• Modularization: Encoder-decoder separation (with frozen base encoders during missing modality reconstruction) increases model robustness under partial modality scenarios.
• Chain-of-Modality Inference: Chaining tasks at inference (e.g., s→t→i in TMT) is supported by architectural design but direct mappings usually yield higher end-task performance, indicating compounded error risk in long chains.
• Compute Scaling: Multi-modal contrastive training is quadratic in modality count; extending frameworks to $N>3$ modalities requires careful system and batch size design.

6. Limitations, Challenges, and Future Directions

Despite advances in chain-of-modality tri-task learning, several issues persist:

• Modality bias and task conflict: While gradient calibration suppresses bias, downstream tasks may still exhibit modal over-reliance—addressed only partially by current methods (Huang et al., 2023).
• Discrete error compounding: Chaining tokenized translation steps (TMT) incurs quality degradation compared to direct mapping (Kim et al., 2024).
• Data scarcity: High-quality, equal-scale tri-modal data remains rare, limiting generalization (Cho et al., 30 Apr 2025).
• Scalability: Algorithmic and hardware limits on cross-modality pairwise training pose hurdles for $N$-modality extension.

A plausible implication is the need for scalable calibration and balancing methods for future multi-modal, multi-task systems as modality count and task complexity increase. Continued work is called for in non-gradient balancing optimizers, dynamic attention gating, and domain-specific tri-modal dataset construction to fully realize the promise of chain-of-modality and tri-task learning.