HiLo: Decomposition, Hierarchy & Optimization

• HiLo is a versatile concept that denotes the separation of high-frequency local details from low-frequency global structures as well as human-in-the-loop optimization, depending on context.
• In computer vision and graphics, HiLo methods decompose information to accelerate networks by using local self-attention for detailed regions and global pooling for smoother areas.
• Beyond vision, HiLo extends to hierarchical learning and control systems, integrating adaptive token compression and human-in-the-loop strategies to enhance efficiency and robustness.

HiLo is a reused research name rather than a single standardized acronym. In contemporary arXiv usage, it denotes several distinct families of methods across computer vision, graphics, communications, control, robotics, and microscopy. Two meanings recur especially often: explicit separation of “high” and “low” information—most commonly high- and low-frequency content, but also high- and low-frequency relation classes or high-level and low-level features—and human-in-the-loop optimization and control. Representative instances include HiLo attention for Vision Transformers (Pan et al., 2022), HiLo-Token for Diffusion Transformer image editing (You et al., 11 Jun 2026), HiLo for unbiased panoptic scene graph generation (Zhou et al., 2023), HILO-MPC for machine-learning-supported control and estimation (Pohlodek et al., 2022), and HiLo microscopy (Jiao et al., 2023).

1. Terminological scope and recurring meanings

In the vision literature, HiLo most often refers to a decomposition between high-frequency local detail and low-frequency global structure. “Fast Vision Transformers with HiLo Attention” introduces HiLo as a self-attention mechanism that assigns local window attention to high frequencies and pooled global attention to low frequencies (Pan et al., 2022). “HiLo-Token” extends the same general intuition from attention to token compression in image editing, keeping full token density in and around the user mask while compressing smoother regions (You et al., 11 Jun 2026). “HiLo: Detailed and Robust 3D Clothed Human Reconstruction with High-and Low-Frequency Information of Parametric Models” uses high-frequency signed distance information for detail and low-frequency voxel information for robustness (Yang et al., 2024). “Se-HiLo” applies a transformer-based high-and-low frequency decomposition to semantic communication (Xi et al., 10 Mar 2025).

The same label also appears in settings where “high” and “low” do not refer to signal frequency. In panoptic scene graph generation, HiLo separates high-frequency and low-frequency relation categories to mitigate long-tail bias (Zhou et al., 2023). In generalized category discovery under domain shift, HiLo refers to High-level semantic and Low-level domain features, which are disentangled by minimizing mutual information between the two representations (Wang et al., 2024).

A different tradition uses HILO to mean human-in-the-loop optimization or related human-in-the-loop systems. HILO-MPC is “macHIne Learning and Optimization for Modeling, Prediction and Control,” a Python toolbox for machine-learning-supported optimization, MPC, and estimation (Pohlodek et al., 2022). Human-in-the-loop optimization also appears in robot-assisted gait training (Christou et al., 7 Oct 2025) and retinal implant calibration (Schoinas et al., 31 Jan 2025). This terminological spread means that the interpretation of “HiLo” is context dependent.

2. Frequency-aware transformers and token compression

The most influential machine-learning use of HiLo is the attention mechanism introduced in LITv2. HiLo attention splits the heads in a multi-head self-attention layer into two groups: Hi-Fi heads encode high frequencies via self-attention within each local window, while Lo-Fi heads encode low frequencies by performing global attention between average-pooled low-frequency keys and values from each window and each query position in the input feature map (Pan et al., 2022). The resulting layer concatenates the outputs of the two branches. The paper emphasizes direct speed evaluation on the target platform rather than FLOPs alone, and reports that HiLo is 1.4x faster than spatial reduction attention and 1.6x faster than local window attention on CPUs (Pan et al., 2022).

HiLo-Token applies the same high/low-frequency logic to DiT inference for production image editing. The method is input-adaptive and organized around the user mask. All tokens within the mask and a dilated region are fully retained, preserving locality and contextual relevance. Outside the editing region, a normalized spatial-frequency map is computed with Sobel edge detection,

$F(p) = \sqrt{(S_x * x)^2 + (S_y * x)^2},$

and tokens are preserved where $F(p)$ exceeds a threshold such as 0.1, with 16× spatial pooling and regionalization to align the selection to the token grid (You et al., 11 Jun 2026). Low-frequency regions outside the mask are represented by tokens from a 16x downsampled image. The final token set is the union of retained mask tokens, selected high-frequency tokens, and low-frequency tokens from the aggressively downsampled image. On production-level evaluation data, the method yields 3.13x, 2.59x, and 1.67x DiT speedups on A100-80GB across small, medium, and large mask-ratio categories, with average ratios of 6.38%, 15.92%, and 35.36%, and without regression in generation quality (You et al., 11 Jun 2026). The paper also states that token selection overhead is about 10 ms and that the approach is compatible with timestep distillation, FP8 quantization, and other pipeline optimizations (You et al., 11 Jun 2026).

HiLo attention has also been transplanted into other architectures. In unified face recognition and physical-digital spoof detection, a Swin Transformer backbone feeds a Unified Attack Detection module with HiLo attention attached at block 5 in Stage 3, where the intermediate representation has dimensions $14 \times 14 \times 512$ (Kunwar et al., 16 Jan 2025). The Hi branch uses local window self-attention with 4 attention heads over $2 \times 2$ windows, while the Lo branch uses average pooling to capture broader contextual relationships. The reported best placement achieves 97.2% accuracy on FF++ for digital attack detection, 86.8% on SiW-Mv2 for physical attack detection, and approximately 99.4% on FF++ for face recognition (Kunwar et al., 16 Jan 2025).

3. Reconstruction, discovery, and structured prediction

Several HiLo methods use high/low decomposition to separate detail from robustness, semantics from nuisance variation, or head classes from tail classes. In clothed human reconstruction from a single RGB image, HiLo combines a progressive high-frequency signed distance encoding with low-frequency information from a $32\times32\times32$ voxel grid of the parametric model (Yang et al., 2024). The progressive high-frequency transformation

$$\mathcal{H}_k(s; \beta) = \omega_k(\beta)[\sin(2^k\pi s), \cos(2^k\pi s)]$$

is designed to alleviate large gradients that hinder convergence, while a spatial interaction implicit function fuses the progressive SDF, low-resolution voxel features, and normal features (Yang et al., 2024). The paper reports improvements of 10.43% and 9.54% in Chamfer distance on Thuman2.0 and CAPE, respectively, and emphasizes robustness to noise from the parametric model, challenging poses, and various clothing styles (Yang et al., 2024).

In generalized category discovery with domain shifts, HiLo explicitly separates low-level domain features from high-level semantic features using a Vision Transformer’s first and last layers, then minimizes their mutual information (Wang et al., 2024). The framework is extended with PatchMix, which mixes patch embeddings across images, and with a curriculum that gradually introduces harder domain-shifted samples (Wang et al., 2024). On corrupted fine-grained benchmarks and on DomainNet, the paper reports large margins over prior GCD baselines; for example, on Painting in the Real+Painting setting, HiLo reaches 42.1% “All” accuracy versus 34.5% for SimGCD, and on CUB-C it reaches 52.0% “All” accuracy versus 28.8% for SimGCD (Wang et al., 2024).

In panoptic scene graph generation, HiLo addresses long-tail bias by decoupling learning for frequent and rare relations. A dual-branch relation head contains a Hi-branch trained on high-frequency relations and a Lo-branch trained on low-frequency relations, followed by fusion and a consistency regularizer,

$$\mathcal{L} = \mathcal{L}_{\text{Hi}} + \mathcal{L}_{\text{Lo}} + \lambda_{\mathrm{Con}} \mathcal{L}_{\mathrm{Con}},$$

with $\mathcal{L}_{\mathrm{Con}}$ defined on the difference between the branches’ softmax outputs (Zhou et al., 2023). The paper describes this as the first explicitly unbiased PSG method and reports state-of-the-art results on PSG and improvements on box-based scene graph generation on Visual Genome (Zhou et al., 2023). Here, “high” and “low” refer to relation frequency in the dataset rather than spectral frequency.

Se-HiLo extends the same separation principle to semantic communication. The method combines Finite Scalar Quantization with a transformer-based high-and-low frequency decomposition, mapping the two components into separate FSQ representation spaces (Xi et al., 10 Mar 2025). The stated objective is to improve robustness to semantic noise without adversarial training while preserving representational diversity that would otherwise be reduced by quantization alone (Xi et al., 10 Mar 2025).

4. Hierarchical learning, adapter configuration, and low-rank routing

A second major meaning of HiLo is hierarchical organization. In HPFF, HiLo denotes Hierarchical Locally Supervised Learning, which divides a network into independent local modules and cascade local modules (Su et al., 2024). The independent level gives each module its own auxiliary supervision; the cascade level spans adjacent modules and introduces information exchange between them. Patch Feature Fusion then computes auxiliary predictions patchwise and averages them. The paper reports that on CIFAR-10 with ResNet-110 and $K=55$ modules, DGL+HPFF uses 2.44 GB versus 9.26 GB for BP, a 73.7% reduction, while also improving classification error over prior local-learning baselines (Su et al., 2024).

In large-language-model fine-tuning, HILO stands for a hierarchical scheme for expert allocation and rank configuration in mixture-of-adapter-expert systems (Cong et al., 6 Feb 2025). The method extends LoRA-style updates by jointly varying both the number of experts and the rank of each expert across layers. The layerwise rank schedule is explicitly parameterized, and the paper argues that prior approaches focus on the number of experts while neglecting the adapter rank (Cong et al., 6 Feb 2025). On Llama 2-7B, HILO is reported to reduce active parameters by 37.5% relative to MoLA or AlphaLoRA while achieving 82.96% average accuracy versus 82.34% and 82.22%, respectively (Cong et al., 6 Feb 2025).

Two later methods generalize the same hierarchical logic to LoRA reuse. HiLoRA for training-free domain generalization treats each LoRA rank as a rank-one component, represents each LoRA by a Gaussian in an embedding space, performs sequence-level selection by Gaussian likelihood, and then refines routing at the token level by activating informative rank-one components only (Han et al., 14 Oct 2025). The paper gives theoretical error bounds for the probability of missing the correct LoRA and reports accuracy gains of up to 55% over baselines while maintaining comparable inference throughput (Han et al., 14 Oct 2025). HiLoRA for personalized federated learning instead organizes LoRA adapters into root, cluster, and leaf levels to capture global, subgroup, and client-specific knowledge, with LoRA-Subspace Adaptive Clustering used to infer latent client groups and cross-tier orthogonality used to separate update subspaces (Peng et al., 3 Mar 2026). The paper reports consistent improvements in both personalization and generalization on ViT backbones with CIFAR-100 and DomainNet (Peng et al., 3 Mar 2026).

5. Human-in-the-loop optimization and control

In control, robotics, and neuroprosthetics, HILO usually denotes human-in-the-loop optimization rather than high/low decomposition. HILO-MPC is an open-source Python toolbox for model predictive control, moving horizon estimation, Kalman filtering, and machine-learning-supported optimal control and estimation (Pohlodek et al., 2022). Its five integrated modules—dynamic model, control, estimator, machine learning, and embedded—are built around CasADi and interface with PyTorch and TensorFlow, enabling machine learning components to appear as dynamics models, objectives, constraints, reference generators, or control-law approximators (Pohlodek et al., 2022).

E2HiL is a human-in-the-loop reinforcement learning framework that uses entropy-guided sample selection. It estimates the influence of each sample on policy entropy using the covariance of action probabilities and soft advantages, prunes both shortcut samples and noisy samples, and updates the actor using only the retained samples (Deng et al., 27 Jan 2026). On four real-world manipulation tasks on the Lerobot SO-101 platform, the method reports a 42.1\% higher success rate and 10.1\% fewer human interventions than HIL-SERL (Deng et al., 27 Jan 2026).

Retinal-implant HILO uses preference-based optimization with human feedback to personalize a deep stimulus encoder. Seventeen sighted participants viewed simulated prosthetic vision, completed 60 duels for optimization and 39 duels for evaluation, and selected the phosphene pattern that better matched the target image (Schoinas et al., 31 Jan 2025). HILO-generated stimuli were preferred over both a naïve encoder and the DSE alone in the main, threshold-misspecified, and out-of-distribution conditions, with preference counts of 16/17, 17/17, and 14/17 participants, respectively (Schoinas et al., 31 Jan 2025).

By contrast, human-in-the-loop optimization in robot-assisted gait training produced a more mixed outcome. In a two-day experiment with six healthy individuals, CMA-ES optimized stiffness parameters of an assist-as-needed lower-limb exoskeleton controller and appeared to converge to a unique set of stiffnesses for each individual, but no measurable impact on performance was observed during validation trials (Christou et al., 7 Oct 2025). This result is presented as evidence that human-robot co-adaptation and human behaviour variability may outweigh the benefits of personalizing a rule-based assistive controller (Christou et al., 7 Oct 2025).

A common ambiguity arises here: HiLo in “Learning Whole-Body Human-like Locomotion with Motion Tracking Controller” does not mean human-in-the-loop at all. It expands to human-like locomotion with motion tracking and combines open-loop reference tracking with a residual RL policy, simple domain randomization through random force injection and action delay, and a distributional value function; the paper reports convergence 2.5 times faster than a standard value-function variant and zero-shot sim-to-real transfer on the GR1 humanoid (Zhang et al., 5 Feb 2025).

6. Other specialized usages and broader significance

Outside mainstream machine learning, HiLo is also an established term in optical imaging. HiLo microscopy is an optical sectioning technique for widefield fluorescence imaging. In an MLA-based incoherent implementation, structured illumination patterns are generated using microlens arrays and LED light sources, and the reconstructed image is formed as

$$I_{\textrm{HiLo}}(u, v) = I_{\textrm{Hi}}(u, v) + n I_{\textrm{Lo}}(u, v).$$

The numerical study reports that MLA numerical aperture between 0.006 and 0.01 does not significantly affect HiLo image contrast, that larger microlens pitch improves contrast up to an optimum, and that excessive pitch causes ring-like artefacts (Jiao et al., 2023). The best overall results are reported for a cross-type MLA with NA 0.01 and pitch 120 μm, corresponding to approximately 12 μm period at the sample plane (Jiao et al., 2023).

In autonomous driving, HiLO has yet another expansion: High-Level Object Fusion. The method adapts DETR-style transformer fusion to object lists from heterogeneous sensors, using self-attention for multi-modal association and decoder queries for fused object outputs (Osterburg et al., 3 Jun 2025). On a large-scale real-world dataset, the paper reports improvements of 25.9 percentage points in $F(p)$0 score and 6.1 percentage points in mean IoU over adapted Kalman-filter baselines, with a compact model of approximately 186k parameters and about 3.4 ms inference per sample on an Intel i7 CPU (Osterburg et al., 3 Jun 2025).

Taken together, these papers show that “HiLo” functions less as a single doctrine than as a compact naming pattern for decomposition, hierarchy, and selective information routing. In one group of works it separates fine detail from coarse structure, local content from global context, or tail classes from head classes; in another it explicitly keeps a person in the optimization loop; and in still others it abbreviates task-specific phrases such as High-Level Object Fusion or human-like locomotion. This suggests that, in technical reading, the meaning of HiLo should always be resolved from the paper’s expansion and operational definition rather than from the name alone.