---
title: Hierarchical Adaptive Projector
url: https://www.emergentmind.com/topics/hierarchical-adaptive-projector-hap
type: topic
---

# Hierarchical Adaptive Projector

“Hierarchical Adaptive Projector” is not a standardized term with a single canonical definition in the cited literature. Instead, closely related mechanisms appear across several technical lineages: instruction-conditioned multi-projector fusion in video multimodal large language models, hierarchical Bayesian transfer models followed by adaptive posterior projection, and geometric projector/backprojector constructions designed for hierarchical reconstruction. At the same time, the acronym **HAP** is heavily overloaded: in other papers it denotes **Hierarchical Adaptive Pooling**, **Human structure-Aware Pre-training**, or **High Altitude Platform** rather than any projector architecture. For that reason, the most precise treatment is to regard Hierarchical Adaptive Projector as an umbrella concept whose constituent properties—hierarchy, adaptivity, and projection/projector operations—are realized explicitly only in parts across different works [2501.05067][2606.08691][2110.09841].

## 1. Terminological status and acronym ambiguity

The literature does not present a single method formally named “Hierarchical Adaptive Projector.” What it does contain are several nearby constructs that each realize part of that description. In video understanding, LLaVA-Octopus introduces an **instruction-driven adaptive projector** but explicitly organizes its three projectors as **parallel branches**, not a hierarchy. In transfer learning, ProjectionTL couples a **source-guided hierarchical prior** with an **adaptive projection** step. In cone-beam CT, the cutting voxel projector is described as particularly suited for **hierarchical reconstruction approaches with nonuniform voxel grids** and local refinement. These are adjacent, but not identical, uses of the term space surrounding a Hierarchical Adaptive Projector [2501.05067][2606.08691][2110.09841].

| Literature line | HAP-related meaning | Relation to “Hierarchical Adaptive Projector” |
|---|---|---|
| Video MLLMs | Instruction-conditioned adaptive projector fusion | Adaptive, but not hierarchical |
| Transfer learning | Hierarchical prior plus adaptive posterior projection | Closest conceptual match |
| Cone-beam CT | Projector suited for hierarchical reconstruction | Strong projector component |
| Other HAP usages | Pooling, pre-training, or high-altitude platforms | Terminologically unrelated |

The acronym collision is substantial. In graph representation learning, HAP means **Hierarchical Adaptive Pooling** rather than projection [2104.05960]. In human-centric vision, HAP denotes **Human structure-Aware Pre-training with MIM** [2310.20695]. In non-terrestrial networking, multiple papers use HAP for **High Altitude Platform** or **HAP-LAP** architectures, including satellite-to-HAP links, RIS-assisted HAP networks, UAV/HAP-assisted vehicular edge computing, and hierarchical HAP-LAP RSMA systems [2310.01143][2411.03886][2409.10040][2202.10953][2312.04081]. A common misconception is therefore that “HAP” names a single architecture family; the record instead shows a shared acronym attached to distinct technical objects.

## 2. Constituent properties of a HAP-style architecture

A useful synthesis is to treat **“HAP-style”** *(Editor's term)* as a design pattern defined by three ingredients. The first is **hierarchy**, meaning that adaptation occurs at more than one level. The clearest explicit instance is ProjectionTL, where transfer is “decouple[d] … at two levels”: source-level borrowing through a hierarchical prior and feature-level borrowing through posterior projection [2606.08691]. The second is **adaptivity**, meaning that the projection or projector changes with task, prompt, source relevance, or local agreement. LLaVA-Octopus realizes this through instruction-conditioned projector weights, while ProjectionTL realizes it through source weights \(\boldsymbol{\eta}\) and coordinate weights \(w_j\) [2501.05067][2606.08691]. The third is **projection/projector functionality** itself: either a learned feature projector, an optimization-based projection step, or a geometric forward operator, as in the cutting voxel projector [2110.09841].

These three ingredients do not always co-occur. LLaVA-Octopus has adaptivity and projectors, but not strict hierarchy. ProjectionTL has hierarchy and adaptive projection, but its “projection” is a posterior-refinement optimization rather than a linear projector in the strict algebraic sense. The cutting voxel projector is a genuine projector/backprojector pair and is explicitly linked to hierarchical reconstruction, but it is not a complete adaptive hierarchy by itself. This suggests that Hierarchical Adaptive Projector is best understood as a compositional concept rather than the proper name of one established architecture.

Two equations capture the most central mechanisms. In LLaVA-Octopus, projector outputs are fused by
\[
F = w_1 \cdot F_1 + w_2 \cdot F_2 + w_3 \cdot F_3.
\]
In ProjectionTL, posterior draws are sparsified by the weighted projection
\[
\boldsymbol{\beta}^\ast \;=\; \arg\min_{\boldsymbol{u}\in\mathbb{R}^p} \Big\{ \|\boldsymbol{X}\boldsymbol{\beta}-\boldsymbol{X}\boldsymbol{u}\|_2^2 \;+\; \lambda \sum_{j=1}^p \frac{|u_j|}{w_j} \Big\}.
\]
The first is an adaptive fusion rule over specialized projectors; the second is an adaptive projection operator acting on posterior samples [2501.05067][2606.08691].

## 3. Instruction-conditioned adaptive projectors in video multimodal models

LLaVA-Octopus is the clearest example of an **adaptive projector** in the contemporary multimodal literature, even though the paper explicitly states that it is “better described as a parallel multi-projector system with instruction-driven expert weighting, rather than a truly hierarchical projector stack” [2501.05067]. The model consists of four components: a visual encoder, a text encoder, an instruction-driven adaptive projector, and a large language model decoder. The instantiated backbones are **SigLIP (so400m-patch14-384)**, **BERT**, and **Qwen2.5-7B-Instruct**. The adaptive projector contains three branches: an image-based projector implemented as **MLP2x_GELU**, a spatial-temporal projector implemented with the **STC module from VideoLLaMA2**, and a token-compress projector implemented with the **LLaMA-VID projector**.

The branches are functionally specialized. The image-based projector is intended to capture **static scene details / appearance**; the spatial-temporal projector captures **inter-frame relationships / temporal dynamics**; and the token-compress projector preserves **temporal coherence over many frames**, especially for long videos and tasks such as **dynamic counting**. The fusion controller is instruction-conditioned: BERT encodes the user instruction, the **[CLS] token** is passed through **two MLPs**, and the resulting three weights are applied to the full projector outputs through the fusion equation above. The weighting is therefore sample-level, projector-level, and sequence-wide rather than token-level or frame-level.

A practical constraint of this design is output-shape compatibility. LLaVA-Octopus modifies each projector so that all three produce **1568 visual tokens**. The MLP projector is reduced from \(14 \times 14 \times 8 + 8 = 1576\) to **1568** by removing separators between images; the STC projector is reparameterized with stride \((2,2,2)\) and \((1,2,2)\) and padding \((1,1,1)\) to also reach **1568**; and the LLaMA-VID projector uses **128 frames**, **6 context tokens + 6 content tokens** per frame, and a separator every 4 frames so that \(49 \times 32 = 1568\). This token alignment is essential because fusion is additive.

Empirically, the adaptive gate outperforms naive alternatives. On the fusion-strategy ablation, **Projector Fusion Gate** reaches **51.7 / 55.7** on **MVBench / VideoMME**, compared with **50.4 / 53.6** for **Average**, **51.2 / 54.8** for **Concat**, **50.1 / 52.9** for **Random weights**, and **50.9 / 53.4** for **Random choose**. On projector-combination ablations, the best single projector reaches **49.1**, while all three together reach **51.7**. The model also reports full-setting results such as **59.2** on **EgoSchema**, **57.5** on **MLVU**, **54.7** on **VideoMME**, and **66.9** on **MVBench**. The central significance is not hierarchy but **instruction-conditioned adaptive projector fusion** over specialized branches [2501.05067].

## 4. Hierarchical adaptive projection in selective knowledge transfer

ProjectionTL is the closest direct realization of a Hierarchical Adaptive Projector in the strict compositional sense because it is explicitly organized into two transfer levels: **source selection** and **feature selection** [2606.08691]. Its first stage builds a **source-guided hierarchical prior**. Let \(\widehat{\boldsymbol\omega}_k\) be source coefficient summaries and \(\widehat{\boldsymbol\beta}\) a target-only pilot estimate. Then the prior mean is assembled coordinatewise as
\[
\mu_j \;=\; \sum_{k=1}^K \eta_k \cdot \left(\frac{\|\hat{\boldsymbol\beta}\|}{\|\hat{\boldsymbol\omega}_k\|}\right)\cdot\widehat{\omega}_{kj} + \eta_{K+1}\cdot 0,
\]
or in matrix form \(\boldsymbol\mu = \mathbf W \boldsymbol\eta\). The mixing vector lies on the simplex,
\[
\boldsymbol{\eta} \sim \mathrm{Dirichlet}(\theta_1,\ldots,\theta_K,\zeta),
\]
with \(\zeta=1\), and each \(\theta_k\) is a cosine-similarity score,
\[
\theta_k \;=\; \frac{\widehat{\boldsymbol \omega}_k^\top \widehat{\boldsymbol\beta}}{\|\widehat{\boldsymbol \omega}_k\|\,\|\widehat{\boldsymbol\beta}\|}.
\]
The \((K+1)\)th component is a **dummy no-transfer source**, allowing the model to fall back to a zero-centered prior when external sources are not useful.

Conditionally, the target coefficients follow
\[
\beta_j \mid \mu_j,\sigma^2 \sim \mathcal{N}(\mu_j, a_n^{-1}\sigma^2), \qquad \sigma^2 \sim \mathrm{IG}(\nu,\nu).
\]
Given target data, the conditional posterior is Gaussian:
\[
\boldsymbol{\beta} \mid \mathbf y,\mathbf X,\boldsymbol\eta,\sigma^2 \sim N_p\!\left(\boldsymbol\mu_{\eta},\sigma^2 \mathbf V_n\right),
\]
with
\[
\mathbf V_n = \left(\mathbf X^\top \mathbf X + a_n \mathbf I_p\right)^{-1}, \qquad
\boldsymbol\mu_{\eta} = \mathbf V_n \left(\mathbf X^\top \mathbf y + a_n \mathbf W\boldsymbol\eta\right).
\]
This posterior is typically dense, which motivates the second stage: a **posterior-projection step** that acts at the feature level.

That projection uses coordinate-specific adaptive weights
\[
w_j \;=\; \sum_{k=1}^K \exp\!\left\{-\big(\widehat{\beta}_j - \widehat{\omega}_{kj}\big)^2\right\},
\]
so coordinates locally aligned with at least one source are penalized less heavily. The projected posterior is then summarized by a **Median Probability Model**, retaining coordinates that are nonzero in more than \(50\%\) of projected draws. This gives ProjectionTL both global and local adaptivity: “where to borrow” is handled through \(\boldsymbol\eta\), while “what to borrow” is handled through \(w_j\) and the projection step.

Theoretical analysis establishes posterior contraction at the sparse high-dimensional rate. With \(\lambda_n = \mathcal O(\sqrt{(\log p)/n_0})\), the paper proves
\[
\Pi_{\mathrm{ProjTL}}\!\left[ \boldsymbol\beta:\left\| \boldsymbol\beta-\boldsymbol\beta^0 \right\|_2 > M \sqrt{\frac{s_0\log p}{n_0}} \,\middle|\, \mathcal D_0,\mathcal D_1,\ldots,\mathcal D_K \right] \longrightarrow 0,
\]
and gives an analogous predictive contraction result. Empirically, on ADNI data, ProjectionTL reports test MSE **3.9524** in a heterogeneous single-model-source study and **3.3196** in a multimodal block-missingness study, outperforming **TRADER**, **TransGLM**, **CONCERT**, and **BR2** in the reported comparisons. Relative to a Hierarchical Adaptive Projector concept, this is the most explicit example of a method that is simultaneously hierarchical, adaptive, and projection-based [2606.08691].

## 5. Geometric projectors for hierarchical reconstruction

In cone-beam CT, the paper “Cutting Voxel Projector” provides a different but highly relevant meaning of projector: a forward and backward operator grounded in the **volume of the portion of a voxel that is cut out by all rays reaching a detector pixel** [2110.09841]. This is not a learned feature projector but a geometric operator designed for algebraic reconstruction. Its importance for a HAP-like concept lies in three explicit properties: it is described as a **near-exact projector and backprojector**, it is “particularly suited for algebraic reconstruction techniques and hierarchical reconstruction approaches with nonuniform voxel grids,” and it “enables local refinement of voxels.”

The core forward model replaces standard ray-length approximations with an integral over the cut volume \(V^{(i,j,k)}_{(m,n)}\). In its unit-sphere form,
\[
\bar{e}_{i,j,k}^{(m,n)} =  \int_{V^{(i,j,k)}_{(m,n)}}  \frac{1}{r^2}  \,\mathrm{d}V,
\]
and the pixel response aggregates these contributions as
\[
\bar{E}^{(m,n)} = \frac{1}{\bar{a}^{(m,n)}} \sum_{i,j,k} \mu^{i,j,k} \int_{V^{(i,j,k)}_{(m,n)}}  \frac{1}{r^2} \, d x.
\]
Operationally, the implementation approximates each cut volume by a planar cut area times a vertical cut length,
\[
|V^{(i,j,k)}_{(m,n)}|=A_{n}^{i,j}*d^{(i,j,k)}_{(m,n)}.
\]
This construction is voxel-centric rather than ray-centric, which is why it aligns naturally with local refinement and nonuniform discretization.

The paper reports that the projector/backprojector pair passes the discrete adjointness test
\[
\vec{b} \cdot (\vec{A} \vec{x}) = \vec{x} \cdot (\vec{A}^\top \vec{b}),
\]
and benchmarks both a standard and a relaxed GPU implementation against Siddon ray tracing and TT footprint projection. In the first benchmark on an AMD Radeon VII, **CVP relaxed** reports projection/backprojection times of **12.3 s / 5.2 s**, versus **101.1 s / 18.8 s** for TT and **117.8 s / 244.2 s** for Siddon8. In the second benchmark, **CVP relaxed** reports **7.5 s / 4.1 s**, versus **27.6 s / 11.5 s** for TT and **282.2 s / 684.5 s** for Siddon8. The paper also states that the cutting voxel projector achieves higher accuracy than the TT projector, especially for large cone-beam angles.

The limitations are equally important. The work does not implement a full hierarchical voxel data structure, adaptive mesh refinement policy, or octree traversal. A plausible implication is that it should be regarded as a **projector foundation** for a Hierarchical Adaptive Projector in reconstruction, not as a complete adaptive hierarchy by itself. Even so, among projector-centric papers it is the strongest explicit link between projector design and hierarchical reconstruction [2110.09841].

## 6. Misconceptions, limits, and likely directions of convergence

Several misconceptions recur when the term Hierarchical Adaptive Projector is applied loosely. The first is that any adaptive projector fusion model is hierarchical. LLaVA-Octopus explicitly is not: it has **parallel experts** with a **single global decision over three projector outputs**, no stage-wise gating, no token-level adaptive selection, and no coarse-to-fine projector stack [2501.05067]. The second is that ProjectionTL uses “projection” in the sense of a linear orthogonal projector. It does not; its projection is a **weighted \(\ell_1\)-penalized predictive projection** applied to posterior draws [2606.08691]. The third is that a high-fidelity projector suitable for hierarchical reconstruction is already a full adaptive framework. The cutting voxel projector is not: it lacks explicit parent-child hierarchy management, refinement rules, and multilevel solver logic [2110.09841].

A second class of misconception is purely terminological. The literature contains major, unrelated HAP expansions: **Hierarchical Adaptive Pooling** in graph learning, **Human structure-Aware Pre-training** in masked image modeling, and **High Altitude Platform** in multiple communications papers [2104.05960][2310.20695][2310.01143]. Consequently, the phrase “HAP” by itself is not semantically stable across fields. Any encyclopedia treatment of Hierarchical Adaptive Projector therefore has to separate the conceptual family from the acronym.

The present record suggests, but does not yet standardize, a more integrated future architecture. A plausible implication is that a canonical Hierarchical Adaptive Projector would combine the multilevel control of ProjectionTL, the expert specialization and task conditioning of LLaVA-Octopus, and the operator consistency of projector/backprojector pairs such as the cutting voxel projector. The missing pieces are already visible in the cited limitations: explicit multi-level routing rather than single-shot global weighting, richer uncertainty treatment after projection, and true hierarchical data structures with refinement/coarsening rules [2501.05067][2606.08691][2110.09841]. Until such a synthesis is formalized, Hierarchical Adaptive Projector remains best understood as a technical umbrella concept rather than the settled name of a single method class.

Source: https://www.emergentmind.com/topics/hierarchical-adaptive-projector-hap