Task-Instructed Causal Routing of Vision Foundation Models for Multi-Task Learning

Abstract: Vision foundation models (VFMs) have demonstrated strong robustness and transferability across a wide range of visual tasks. However, each model typically encodes strong inductive biases shaped by its pre-training objective and data domain, resulting in fragmented yet complementary visual knowledge. As a result, a single model often struggles to capture the diverse visual representations required across multiple dense prediction tasks. To address this limitation, we propose TIGER (Task-Instruction-Guided Expert Routing), a framework that coordinates multiple heterogeneous VFMs for multi-task dense prediction. Instead of naively aggregating expert features, TIGER leverages natural-language task instructions to guide a routing network that assigns token-level expert weights conditioned on task semantics, enabling adaptive integration of complementary expert features. TIGER further introduces a counterfactual loss that aligns routing decisions with each expert's causal contribution by measuring prediction changes when experts are excluded, encouraging more reliable and interpretable routing. We evaluate TIGER on two multi-task dense prediction benchmarks, NYUD-v2 and Pascal Context, where it consistently outperforms recent multi-task learning baselines while keeping all VFMs frozen. These results demonstrate that combining instruction-guided expert routing with counterfactual causal alignment enables effective coordination of heterogeneous vision foundation models.

• The paper introduces TIGER, a lightweight framework that routes frozen CLIP, DINOv2, SAM, and OWLv2 experts using natural-language task instructions and counterfactual contribution estimates.
• TIGER achieves state-of-the-art or competitive results across PASCAL-Context and NYUD-v2, including 84.58 mIoU for semantic segmentation and 0.4115 RMSE for depth estimation.
• The counterfactual loss produces the largest ablation gains and interpretable specialization, although masking-based training adds computational cost and may confuse redundancy with causal importance.

Overview

TIGER (Task-Instruction-Guided Expert Routing) addresses a structural limitation of multi-task dense prediction: no single vision foundation model (VFM) spans the semantic, geometric, and boundary-level representations required across tasks such as segmentation, depth, normals, saliency, and boundary detection. Rather than distilling heterogeneous experts into one backbone (as in AM-RADIO or SAK) or fine-tuning large routing components (as in MoVA), TIGER keeps all VFMs frozen and learns lightweight, instruction-conditioned coordination modules on top (2606.15765). The framework rests on two design principles: expert fusion should be task-aware, driven by natural-language task instructions rather than learned task tokens alone; and it should be causally grounded, so that routing weights reflect each expert's counterfactual contribution rather than mere feature dominance.

Architecture

TIGER treats CLIP, DINOv2, SAM, and (in the large variant) OWLv2 as frozen experts. Because experts use different patch sizes, inputs are resized per-expert to produce a common token grid $H \times W$, with positional embeddings interpolated where needed. Expert features are projected into a shared embedding space via LayerNorm plus an FFN.

Task semantics are encoded by Sentence-BERT into a vector that is normalized and fused with a learnable task prompt. This representation conditions the router through a low-rank hyper-network that modulates the query projection ($W_Q(t) = W_0 + U\,\mathrm{diag}(g(t))\,V^\top$), after which token-wise expert weights are computed via softmax attention between task-conditioned queries and expert token features. Two lightweight multi-head attention modules — Expert-Interactive Attention before routing and Task-Interactive Attention after — allow cross-expert cue exchange and inter-task dependency modeling at minimal overhead. A shared decoder with task-specific heads produces dense predictions.

The most distinctive component is the counterfactual loss. For each expert, the paper estimates causal contribution as the increase in task loss when that expert's routed features are masked: $$\Delta_{k,n} = \mathcal{L}_{task}^{(-k)} - \mathcal{L}_{task}$$. These per-token differences are normalized across experts, converted into a causal importance distribution via temperature-scaled softmax, and aligned with the router's actual weights $\alpha_{k,n}(t)$ by minimizing their Jensen–Shannon divergence. The total objective sums weighted task losses plus this counterfactual regularizer. The motivation is explicit: attention-based routing scores can be inflated by feature scale differences and do not guarantee that emphasized experts causally improve predictions.

Results

On PASCAL-Context, TIGER-L achieves state-of-the-art results on four of five tasks: 84.58 mIoU in semantic segmentation, 77.56 mIoU in human parsing, 12.46 mErr in surface normals, and 84.05 ODS-F in boundary detection. Notably, even TIGER-B — using only ViT-B experts — surpasses SAK on semseg and normals despite a comparatively modest budget (579.35 GFLOPs, 362.84M parameters), and excluding frozen experts requires only 103.15M trainable parameters. On NYUD-v2, TIGER-L reaches 63.55 mIoU (semseg), 0.4115 RMSE (depth), 16.80 mErr (normals), and 80.31 ODS-F (boundary), outperforming all conventional MTL methods and single-VFM baselines. A consistent pattern in both tables is that individual VFMs exhibit sharply uneven strengths — SAM leads boundary detection but collapses on semantics (54.88 mIoU on PASCAL-Context), while DINOv2 provides strong geometry but weak saliency — which TIGER's orchestration converts into balanced gains.

Ablations on NYUD-v2 show monotonic improvement from base routing (59.43 mIoU) through instruction encoding, expert- and task-interaction attention, to the counterfactual loss, which yields the largest single gain (61.44 → 63.16 mIoU, with depth RMSE dropping from 0.4774 to 0.4386). Routing visualizations confirm interpretable specialization: SAM dominates edges and contours, DINOv2 covers object interiors and global structure, and CLIP/OWLv2 activate selectively on semantically meaningful regions. Replacing language instructions with randomly initialized learnable task embeddings degrades all four NYUD-v2 metrics (e.g., 63.16 → 62.32 mIoU), supporting the claim that language embeddings supply a structured relational prior over task similarity available from the start of training.

Limitations and open questions

Several caveats bear directly on the reported results. First, the counterfactual loss requires recomputing predictions with each expert masked during training, a cost the paper does not quantify against its favorable inference-time FLOP counts. Second, the causal contribution estimate is defined purely through task-loss deltas under masking — a reasonable but approximate notion of causality that may conflate redundancy among correlated experts with low contribution. Third, the evaluation uses fixed expert sets (three for TIGER-B, four for TIGER-L); how performance scales with more experts, or how sensitive routing is to the choice of Sentence-BERT as the instruction encoder, remains unexamined. Finally, the strongest comparisons (SAK, RADIO) involve different compute profiles, and detailed efficiency analysis is deferred to supplementary material, so the parameter-efficiency claim should be read with that caveat.

Conclusion

TIGER demonstrates that frozen, heterogeneous VFMs can be coordinated for multi-task dense prediction through two mechanisms: natural-language task instructions conditioning token-wise soft routing, and a counterfactual objective aligning routing weights with measured expert influence. The framework achieves state-of-the-art results on PASCAL-Context and NYUD-v2 while training only lightweight modules, and its ablations attribute the largest gains to the causal alignment term. The main open questions concern the computational overhead of counterfactual estimation, scaling behavior beyond four experts, and whether loss-delta-based contribution measures remain reliable as expert sets grow more redundant.