HiLa: Hierarchical Vision-Language Collaboration

• Hierarchical Vision-Language Collaboration (HiLa) is a design principle that enables multi-level alignment between vision and language through semantic, architectural, and structural hierarchies.
• It incorporates various mechanisms such as hierarchy-aware attention, multi-level fusion, and graph-structured alignment to effectively bridge local details and global context.
• Empirical studies demonstrate that multi-level alignment improves performance in tasks like medical imaging, retrieval, and robotics by enforcing consistent cross-modal interactions.

Searching arXiv for the cited HiLa-related papers to ground the article with current records. In the literature surveyed here, Hierarchical Vision-Language Collaboration (HiLa) denotes multimodal modeling in which visual and linguistic representations interact across multiple semantic, spatial, or control levels rather than through a single flat image-text match. The hierarchy may be semantic rather than purely architectural, as in global image, local region, and cross-category alignment; architectural, as in hierarchy-aware attention over image patches and text tokens; report-structural, as in Findings-versus-Impressions alignment; graph-structured, as in object-region-zone or coarse-to-fine pathology graphs; or policy-structural, as in high-level vision-language planning over low-level motor execution (Fuller et al., 16 Jan 2025, Geng et al., 2023, Liu et al., 2023, Fang et al., 1 Jun 2026, Yang et al., 15 Apr 2026).

1. Conceptual scope

A central property of HiLa is that “hierarchy” is not reducible to multiscale vision alone. In some systems, the hierarchy is semantic: HiCA aligns whole-image disease descriptors, region-of-interest descriptors, and inter-class repulsion constraints in a shared embedding space, while explicitly noting that its hierarchy is semantic rather than purely architectural (Fuller et al., 16 Jan 2025). In other systems, the hierarchy is linguistic or document-structural: IMITATE separates radiology reports into Findings and Impressions, then aligns multi-level visual features to those two textual levels rather than treating the report as a flat token bag (Liu et al., 2023). In pathology, HiLa and HiVE-MIL treat patch-level and region-level or 5× and 20× representations as distinct levels that must collaborate, and in navigation or manipulation the hierarchy shifts again, from representational scale to planner–controller decomposition (Cui et al., 7 Jul 2025, Wong et al., 23 May 2025, Schakkal et al., 28 Jun 2025, Yue et al., 23 Apr 2025).

This suggests that HiLa is best understood as a design principle rather than a single model family. Across domains, the recurring claim is that performance degrades when one relies only on a global image-text score, a single prompt, or a single control layer. The alternative is to preserve multiple levels of meaning simultaneously: local textures and global semantics, phrase-level and sentence-level language, object-level and scene-level structure, or high-level subtask descriptions and low-level actions.

A common misconception is that HiLa necessarily implies explicit heavy cross-attention between the two modalities. That is not borne out by the published systems. HiCLIP retains CLIP-style global contrastive coupling and places hierarchy inside the vision and text encoders through hierarchy-aware attention, whereas HiCA, IMITATE, HierVL, and HiVLA use explicit multi-level alignment, prompt routing, or cross-attention modules (Geng et al., 2023, Fuller et al., 16 Jan 2025, Liu et al., 2023, Nadeem et al., 16 Jun 2025, Yang et al., 15 Apr 2026).

2. Architectural patterns

Several architectural motifs recur across HiLa instantiations. One is the shared latent space of dual-tower vision-LLMs. In HiCA, a visual encoder $f_{\text{img}}$ and text encoder $f_{\text{text}}$ map images and class descriptions into a common embedding space, after which hierarchical constraints are imposed at the global, local, and cross-category levels (Fuller et al., 16 Jan 2025). HiCLIP keeps the same CLIP-like dual-tower template but augments each tower with hierarchy-aware attention, introducing a multiplicative mask $C$ into self-attention:

$$\mathrm{Hierarchy\_Attention}= \left( C \odot \operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_h}}\right) \right) V.$$

Here the hierarchy is discovered layer by layer from neighboring words or patches, under monotone “non-splittable” affinity updates (Geng et al., 2023).

A second motif is explicit multi-level fusion. The 2018 multi-task model of hierarchical vision-language representation stacks Dense Co-attention layers and attaches different decoders at different depths: grounding at a shallower level, retrieval at an intermediate level, and VQA at a deeper level (Nguyen et al., 2018). HierVL adopts a related principle for dense prediction: frozen CLIP text embeddings are expanded into hierarchical semantic queries, aligned with multi-scale pixel features, and then fused with instance queries in a dual-query decoder (Nadeem et al., 16 Jun 2025). HiCroPL moves this idea into prompt tuning: prompts are inserted in every CLIP layer, then a hierarchical knowledge mapper routes text-to-vision information in early layers and vision-to-text information in later layers, specifically to counter modality isolation and hierarchical semantic decay (Zheng et al., 20 Jul 2025).

A third motif is graph-structured hierarchy. HiVE-MIL constructs a unified heterogeneous graph whose node types are low-scale image, high-scale image, low-scale text, and high-scale text, with intra-scale vision–text edges and cross-scale same-modality edges (Wong et al., 23 May 2025). HGCLIP likewise turns the class taxonomy into a graph and runs graph encoders on both textual class features and visual prototypes, so that hierarchy is injected into both branches before similarity computation (Xia et al., 2023). In navigation, HSAN elevates the graph itself to the core world model, organizing objects, regions, and zones into a dynamic semantic graph used for planning (Fang et al., 1 Jun 2026).

3. Learning objectives and alignment strategies

HiLa systems differ most sharply in how they operationalize collaboration. HiCA uses an explicitly additive hierarchical objective:

$$\mathcal{L}_{\text{HiCA}} = \mathcal{L}_{\text{global}} + \lambda_1 \mathcal{L}_{\text{local}} + \lambda_2 \mathcal{L}_{\text{cross}},$$

where $\mathcal{L}_{\text{global}}$ is CLIP-style image–text contrast, $\mathcal{L}_{\text{local}}$ aligns ROIs to fine-grained descriptors, and $\mathcal{L}_{\text{cross}}$ imposes a margin against mismatched image–text pairs (Fuller et al., 16 Jan 2025). The important point is that local grounding is not auxiliary explanation added after classification; it is part of the training signal.

By contrast, HiCLIP shows that hierarchy can be architectural rather than loss-based. It trains with the standard symmetric CLIP InfoNCE objective, but modifies how token interactions occur inside each tower. The hierarchy is induced by local affinities, monotone accumulation across layers, and path-product masks over sequences or patch grids, not by extra cross-modal supervision (Geng et al., 2023). This demonstrates that HiLa need not entail a more complicated top-level objective if hierarchy is already built into the representation dynamics.

Medical VLP introduces yet another strategy. IMITATE’s Clinical-Informed Contrastive Loss replaces the binary positive-versus-negative view of contrastive learning with a soft affinity matrix derived from correlations between report embeddings. High-level visual features align with Impressions, multi-level visual features align with Findings, and image–image alignment is imposed at both levels (Liu et al., 2023). In pathology survival modeling, HiLa uses Optimal Prompt Learning to solve an optimal-transport matching problem between visual tokens and prompt sets, then adds Cross-Level Propagation and Mutual Contrastive Learning to enforce cooperation and consistency between patch-level and region-level representations (Cui et al., 7 Jul 2025). HiVE-MIL similarly uses a two-stage Text-Guided Dynamic Filtering mechanism to prune weak patch–text relations and a Hierarchical Text Contrastive Loss to align coarse and fine text semantics across scales (Wong et al., 23 May 2025).

In dense prediction, HierVL keeps the CLIP text encoder frozen but regularizes the adapted system with a prompt-topology loss, a masked vision–language consistency objective, and a query-to-text alignment term, so that segmentation queries remain close to the class semantics of the language space even under extremely sparse supervision (Nadeem et al., 16 Jun 2025). The broader pattern is that HiLa objectives usually do at least one of three things: align different levels, preserve level-consistent semantics, or penalize collapse between levels.

4. Representative instantiations

The concept is realized differently across application domains (Fuller et al., 16 Jan 2025, Geng et al., 2023, Liu et al., 2023, Cui et al., 7 Jul 2025, Wong et al., 23 May 2025, Nadeem et al., 16 Jun 2025).

Framework	Hierarchy	Collaboration mechanism
HiCA	global image, ROI, cross-category	domain-specific pretraining plus hierarchical contrastive alignment
HiCLIP	image patch groups, text token trees	hierarchy-aware attention with CLIP loss
IMITATE	multi-level CXR features; Findings and Impressions	hierarchical report alignment plus CICL
HiLa	patch-level and region-level WSI prompts	OPL, CLP, and MCL
HiVE-MIL	5× and 20× image-text graph	TGDF, HHGNN, and HTCL
HierVL	hierarchical semantic queries and multi-scale pixels	HSQG, CMSAM, and DQTD

In classification and retrieval, HiCLIP and HGCLIP illustrate two different routes to hierarchical collaboration. HiCLIP discovers unsupervised hierarchies in both branches through attention masks over neighboring patches and tokens, whereas HGCLIP injects explicit class-taxonomy structure into graph encoders over text embeddings and visual prototypes (Geng et al., 2023, Xia et al., 2023). HiCroPL extends this logic to adaptation, arguing that prompt tuning should be bidirectional and depth-aware rather than isolated by modality (Zheng et al., 20 Jul 2025).

In medical imaging, HiCA and IMITATE show complementary forms of hierarchy. HiCA is label- and region-centric: whole-image class descriptors, lesion-level descriptors, and cross-class margins (Fuller et al., 16 Jan 2025). IMITATE is report-centric: descriptive Findings and conclusive Impressions are treated as distinct supervision levels aligned with different visual abstractions (Liu et al., 2023).

In pathology, HiLa and HiVE-MIL represent two graphically different but conceptually related systems. HiLa for cancer survival prediction builds hierarchical prompt sets at patch and region levels, aligns them by optimal transport, and then propagates information across levels before survival prediction (Cui et al., 7 Jul 2025). HiVE-MIL instead constructs a single heterogeneous graph over low- and high-scale visual and textual nodes, with explicit parent–child relations and scale-aware message passing (Wong et al., 23 May 2025).

In embodied AI, hierarchy often migrates from representation to decision structure. “Hierarchical Vision-Language Planning for Multi-Step Humanoid Manipulation” uses a three-layer stack: low-level RL whole-body tracking, mid-level imitation-learned skills, and high-level VLM planning plus VLM monitoring (Schakkal et al., 28 Jun 2025). MFRA for vision-and-language navigation uses multi-level fusion over visual observations, language instructions, and navigation history (Yue et al., 23 Apr 2025). HSAN organizes the environment itself hierarchically as objects, regions, and zones for OT-based planning (Fang et al., 1 Jun 2026). HiVLA decouples a VLM planner that outputs subtask language and a grounded bounding box from a low-level DiT action expert with cascaded cross-attention over global images, local grounded crops, and skill language (Yang et al., 15 Apr 2026). A related line in hierarchical VLA emphasizes explicit language–action alignment, using a contrastive grounding model and preference learning so that generated subtask language remains faithful to executed trajectories (Wulff et al., 7 Apr 2026).

5. Empirical profile

The empirical record is consistently favorable, though heterogeneous in evaluation protocol. In few-shot and zero-shot medical image analysis, HiCA reports 86.3 accuracy and 92.8 AUC on Chest X-ray, 83.4 accuracy and 92.0 AUC on Breast Ultrasound, and 75.6 accuracy and 86.2 AUC on unseen Chest X-ray categories; its ablation shows that removing local alignment reduces Chest X-ray performance to 83.5 accuracy and 91.2 AUC, the largest drop among the three components (Fuller et al., 16 Jan 2025). This strongly supports the claim that local semantic grounding is not ornamental.

In CLIP-style pretraining, HiCLIP shows that architectural hierarchy alone can yield large cross-modal gains. On MSCOCO retrieval with 15M pretraining pairs and ViT-B/32, CLIP’s RSum rises from 211.5 to 285.1 under HiCLIP, while ImageNet zero-shot accuracy rises from 32.8 to 40.5; component analysis further shows that using both the Group Transformer and Tree Transformer outperforms using either one alone (Geng et al., 2023). In the older multi-task hierarchical representation model, joint training on VQA, image caption retrieval, and visual grounding improves all three tasks over their single-task counterparts, with the full three-task configuration achieving 66.35 VQA accuracy, 70.43 caption-retrieval annotation R@1, 57.50 retrieval R@1, and 58.26 grounding R@1 (Nguyen et al., 2018).

Pathology and dense prediction show similar behavior under weak supervision. HiLa for survival prediction reaches an overall C-index of 0.671 across BRCA, LUAD, and UCEC, compared with 0.641 for VLSA, and its ablation progresses from 0.612 for the ABMIL baseline to 0.671 after adding multiple prompts, OPL, region-level collaboration, CLP, and MCL (Cui et al., 7 Jul 2025). HiVE-MIL reports gains of up to 4.1% in macro F1 under 16-shot settings (Wong et al., 23 May 2025). HierVL reports a +4.4% mean improvement of the intersection over the union on COCO with 232 labeled images, +3.1% on Pascal VOC with 92 labels, +5.9% on ADE20K with 158 labels, and +1.8% on Cityscapes with 100 labels (Nadeem et al., 16 Jun 2025).

Interpretability and robustness are also recurring themes. HiCA reports a human evaluation in which radiologists rate Transfer Learning at 3.8 for interpretability and 3.6 for clinical validity, versus 4.5 and 4.7 for HiCA; it also remains strong under noisy text, dropping from 86.3/92.8 to 82.1/90.5 on Chest X-ray (Fuller et al., 16 Jan 2025). HiCLIP visualizes bottom-up grouping of patches into objects and scene regions, and constituent-like trees over text, though it also notes typical unsupervised grammar-induction errors (Geng et al., 2023). This suggests that one empirical hallmark of HiLa is not merely higher task accuracy, but more structured intermediate evidence.

6. Limitations, misconceptions, and research directions

One misconception is that hierarchy must be explicit, supervised, and tree-structured. The surveyed systems contradict all three conditions. HiCLIP induces hierarchies without parse trees, boxes, or region labels; HiCA defines hierarchy semantically through loss terms; HiVE-MIL uses a heterogeneous graph rather than a simple tree; and robotic systems often define hierarchy as planner–skill–controller decomposition rather than representation depth (Geng et al., 2023, Fuller et al., 16 Jan 2025, Wong et al., 23 May 2025, Schakkal et al., 28 Jun 2025).

Another misconception is that adding more levels automatically improves reasoning. Several papers identify real costs. HiCLIP’s hierarchy visualization depends on heuristic thresholds, its hierarchy depth is tied to Transformer depth, and it incurs computational overhead (Geng et al., 2023). IMITATE depends on the existence of well-structured reports with Findings and Impressions and is not validated outside chest radiography (Liu et al., 2023). HiLa for survival prediction is prompt-dependent, uses OT whose computation can be non-trivial, and is evaluated on three TCGA cohorts rather than broader multi-center data (Cui et al., 7 Jul 2025). HierVL adds nontrivial attention overhead and still evaluates in closed-vocabulary dataset settings, even though its design points toward open-vocabulary segmentation (Nadeem et al., 16 Jun 2025).

The main future directions are already visible in the literature. Several works call for more explicit multi-level cross-modal supervision, better multimodal fusion guided by learned hierarchies, adaptive parsing or threshold selection, dynamic layer routing, and extension to richer modalities such as omics, clinical variables, video, or interactive control (Geng et al., 2023, Liu et al., 2023, Zheng et al., 20 Jul 2025). In robotics, the natural next step is to combine hierarchical planning with explicit failure detection, re-planning, and grounded language–action consistency checks, so that intermediate language remains both useful and faithful during long-horizon execution (Wulff et al., 7 Apr 2026, Yang et al., 15 Apr 2026, Schakkal et al., 28 Jun 2025). Taken together, these directions suggest that HiLa is evolving from a collection of domain-specific design tricks into a broader recipe for multi-level multimodal reasoning: preserve structure, align like with like, and expose the hierarchy to both learning and inference.