HOICL: Human-Object Interaction Contrastive Learning

• HOICL is a supervised contrastive learning method that explicitly encodes interaction structures in LiDAR data to disentangle human and object features.
• It employs a multi-objective loss combining global, frequently interacting, and contact-region contrasts to sculpt feature geometry for precise keypoint detection.
• Empirical evaluations on datasets like Waymo show that HOICL reduces MPJPE and boosts PCK scores compared to conventional contrastive objectives.

Human-Object Interaction-Aware Contrastive Learning (HOICL) is a supervised contrastive learning module operating on point features to explicitly encode human–object interaction structure. In its canonical formulation, it is the core representation-learning component inside the HOIL framework for 3D human pose estimation from LiDAR point clouds, where it is designed specifically to resolve spatial ambiguity between human and object points, particularly in interaction regions (Jung et al., 17 Mar 2026). In a broader literature usage, closely related methods apply interaction-aware contrastive learning, interaction-weighted resampling, or contrastive distillation to HOI detection, video HOI localization, and manipulation, but the common design principle remains the same: interaction structure, contact, or HOI triplets are used to organize feature geometry more explicitly than generic class-based contrastive objectives (Shen et al., 10 Jun 2026).

1. Problem setting and motivating challenges

The immediate problem addressed by HOICL in HOIL is 3D human pose estimation from raw LiDAR point clouds in autonomous driving scenarios. A LiDAR frame contains points from humans, interacting objects such as bicycles, umbrellas, scooters, and walls, and background surfaces such as ground, buildings, and cars. Compared to RGB, LiDAR provides sparse geometry and very weak appearance cues, which makes human–object separation hard (Jung et al., 17 Mar 2026).

Two challenges motivate the introduction of interaction-aware contrastive learning in this setting. The first is spatial ambiguity in interaction regions: around contact, human and object surfaces are very close, LiDAR returns from human skin and the object can lie in nearly identical 3D locations and with similar local geometry, and the network tends to confuse human and object points, leading to mislocalized keypoints. The second is class imbalance between interacting and non-interacting body parts: interaction-frequent regions such as hands and feet occupy tiny surface area relative to torso or background, contribute very few points, and are both under-represented in the point set and most affected by spatial ambiguity. Standard purely geometric encoders trained only with pose regression or segmentation losses do not enforce strong separation between human vs. object features, especially at contacts (Jung et al., 17 Mar 2026).

A related motivation appears in goal-conditioned manipulation. There, object-centric interaction such as contact, grasping, collisions, or hitting induces distinct changes in the underlying dynamic modes, and standard contrastive reinforcement learning often struggles because interaction-induced mode changes create piecewise nonlinear reachability structures that are difficult for standard CRL energy functions to represent and plan over. This suggests a more general interpretation of HOICL: interaction-aware representation learning is needed when contact or triplet structure creates discontinuities that generic contrastive objectives tend to smooth away (Shen et al., 10 Jun 2026).

2. Canonical HOICL formulation inside HOIL

The HOIL framework follows a pretrain–finetune pipeline. A Point Transformer V3 encoder–decoder processes the LiDAR point cloud $\mathbf{P} \in \mathbb{R}^{N \times 3}$ and outputs point-wise features at original resolution,

$$\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}} \in \mathbb{R}^{N \times C}, \quad C=256.$$

Learnable keypoint queries $\mathbf{Q} \in \mathbb{R}^{N_k \times C}$, one per human or object keypoint, attend to these point features via cross-attention and produce 3D keypoints $\mathbf{K} \in \mathbb{R}^{N_k \times 3}$ and keypoint-level contact $\mathbf{C}_K \in \mathbb{R}^{N_k \times 1}$. In parallel, point-level heads predict human–object part segmentation $\mathbf{S}_p \in \mathbb{R}^{N \times K}$ and point-level contact $\mathbf{C}_p \in \mathbb{R}^{N \times 1}$ (Jung et al., 17 Mar 2026).

Within this pipeline, HOICL is applied on point embeddings derived from $\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}}$. Its role is not to replace pose regression or segmentation, but to sculpt the feature geometry so that human and object points are clearly separated, especially in frequently interacting regions and contact regions. During pre-training on synthetic LiDAR from 5 HOI datasets, HOICL is active together with the other HOIL heads and with contact-aware part-guided pooling. During fine-tuning on real LiDAR, only pose or heatmap and limb losses are used; HOICL acts as a prior via the pre-trained weights (Jung et al., 17 Mar 2026).

HOICL first projects decoder features $$\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}} = [\mathbf{f}_1,\dots,\mathbf{f}_N]$$ into a $D$-dimensional normalized embedding space,

$$\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}} \in \mathbb{R}^{N \times C}, \quad C=256.$$0

where $$\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}} \in \mathbb{R}^{N \times C}, \quad C=256.$$1 is a small MLP. These embeddings are trained with ground-truth part labels, contact labels, and region memberships derived from the underlying human and object meshes during LiDAR simulation (Jung et al., 17 Mar 2026).

3. Objective design and mathematical structure

The overall HOICL loss is

$$\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}} \in \mathbb{R}^{N \times C}, \quad C=256.$$2

It combines three supervised contrastive objectives defined over different subsets of points and labels (Jung et al., 17 Mar 2026).

The first term, $$\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}} \in \mathbb{R}^{N \times C}, \quad C=256.$$3, enforces global part separability across all 24 SMPL body parts together with object and background. It combines Hierarchical Multi-Label Contrastive loss and Targeted Supervised Contrastive loss,

$$\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}} \in \mathbb{R}^{N \times C}, \quad C=256.$$4

In this construction, HMLC encodes hierarchical relationships among body parts, while TSC pulls features of each class toward pre-defined class prototypes on a hypersphere (Jung et al., 17 Mar 2026).

The second term, $$\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}} \in \mathbb{R}^{N \times C}, \quad C=256.$$5, targets frequently interacting regions. Let $$\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}} \in \mathbb{R}^{N \times C}, \quad C=256.$$6 denote hand and foot points and $$\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}} \in \mathbb{R}^{N \times C}, \quad C=256.$$7 denote object points. A binary mask $$\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}} \in \mathbb{R}^{N \times C}, \quad C=256.$$8 marks positive pairs when both points are FIR or both are object points, and a supervised contrastive loss is applied over $$\mathbf{F}_{\mathrm{p},\mathrm{dec}^{(0)}} \in \mathbb{R}^{N \times C}, \quad C=256.$$9. This encourages FIR points to form compact clusters distinct from object points, and object points to cluster separately from FIR (Jung et al., 17 Mar 2026).

The third term, $\mathbf{Q} \in \mathbb{R}^{N_k \times C}$0, targets contact regions. Let $\mathbf{Q} \in \mathbb{R}^{N_k \times C}$1 be human-contact points and $\mathbf{Q} \in \mathbb{R}^{N_k \times C}$2 be object-contact points. A second binary mask $\mathbf{Q} \in \mathbb{R}^{N_k \times C}$3 marks positive pairs when both points are human-contact or both are object-contact, and supervised contrastive learning is applied over $\mathbf{Q} \in \mathbb{R}^{N_k \times C}$4. The explicit purpose is to push human-contact embeddings to be compact and separate from object-contact embeddings, even when those points occupy overlapping 3D regions (Jung et al., 17 Mar 2026).

During HOI pre-training, HOICL is added to the full supervision objective,

$\mathbf{Q} \in \mathbb{R}^{N_k \times C}$5

Here $\mathbf{Q} \in \mathbb{R}^{N_k \times C}$6, $\mathbf{Q} \in \mathbb{R}^{N_k \times C}$7, and $\mathbf{Q} \in \mathbb{R}^{N_k \times C}$8 are cross-entropy losses, $\mathbf{Q} \in \mathbb{R}^{N_k \times C}$9 is MSE on 3D keypoints, and $\mathbf{K} \in \mathbb{R}^{N_k \times 3}$0 supervises CPPool’s internal predictors. HOICL’s gradients flow back through the projection MLP and the PTv3 decoder and encoder, reshaping the backbone’s feature space to be interaction-aware (Jung et al., 17 Mar 2026).

4. Interaction awareness, contact labels, and integration with HOIL components

In HOIL, interaction awareness is grounded in explicit contact annotation during synthetic pre-training. Contact labels are extracted by computing distances from human vertices to the object surface and vice versa, marking vertices with distance smaller than $\mathbf{K} \in \mathbb{R}^{N_k \times 3}$1 cm as contact, propagating vertex contact to faces, and assigning contact labels to LiDAR points based on their originating mesh face. Each LiDAR point therefore has a body-part or object label, a binary contact indicator, and, if its part is hand or foot, membership in frequently interacting regions (Jung et al., 17 Mar 2026).

This labeling makes it possible for HOICL to decouple feature similarity from spatial proximity. In contact regions, human-contact and object-contact are explicitly treated as different classes in contrastive training; in frequently interacting regions, hand and foot points are treated as mutually exclusive from nearby object points. The intended effect is that the model learns to encode subtle geometric and contextual cues that distinguish human from object, rather than allowing nearby points to collapse into a shared embedding neighborhood (Jung et al., 17 Mar 2026).

HOICL is paired with contact-aware part-guided pooling (CPPool) and optionally with contact-based temporal refinement (CTRefine). CPPool replaces PTv3’s max pooling in each encoder stage. It computes a part score, a contact score, and an importance logit, then uses softmax weights within each grid cell to pool features. Dense non-contact torso or background points are down-weighted, while sparse interacting parts such as hands, feet, and contact regions are up-weighted. In this division of labor, CPPool addresses class imbalance structurally, whereas HOICL addresses feature ambiguity (Jung et al., 17 Mar 2026).

CTRefine operates at the keypoint level. A temporal self-attention Transformer processes sequences of keypoint coordinates, and a cross-attention module lets keypoint queries attend to joint and contact signals to produce refined keypoints. CTRefine does not directly use HOICL embeddings, but it uses contact predictions and initial keypoints from the HOIL backbone trained with HOICL. A plausible implication is that better human–object separation at the point level improves temporal correction indirectly through better keypoint contact predictions (Jung et al., 17 Mar 2026).

Training details further delimit HOICL’s scope. Pre-training uses synthetic LiDAR from BEHAVE, CHAIRS, HODome, OMOMO, and InterCap; objects are randomly dropped with probability $\mathbf{K} \in \mathbb{R}^{N_k \times 3}$2 to allow the model to handle human-only scenes; the supplementary reports $\mathbf{K} \in \mathbb{R}^{N_k \times 3}$3, $\mathbf{K} \in \mathbb{R}^{N_k \times 3}$4, $\mathbf{K} \in \mathbb{R}^{N_k \times 3}$5, and $\mathbf{K} \in \mathbb{R}^{N_k \times 3}$6; optimization uses AdamW with learning rate $\mathbf{K} \in \mathbb{R}^{N_k \times 3}$7, cosine schedule, and batch size $\mathbf{K} \in \mathbb{R}^{N_k \times 3}$8. HOICL is only used in pre-training, because real-world LiDAR lacks dense part and contact labels (Jung et al., 17 Mar 2026).

5. Related forms of HOI-aware contrastive learning

Although the name HOICL is tied most directly to HOIL, related work shows that interaction-aware contrastive learning has several distinct realizations. In manipulation, Interaction-weighted Resampling modifies only how positive goals are sampled for each anchor while leaving the InfoNCE objective and model architectures unchanged. Futures just before, during, and just after contact are upweighted with

$\mathbf{K} \in \mathbb{R}^{N_k \times 3}$9

so that the learned representation preserves the mode boundaries that determine future reachability. This formulation is interaction-aware because it concentrates contrastive supervision on interaction phases rather than changing the loss itself (Shen et al., 10 Jun 2026).

In image-based HOI detection, VLM-HOI uses BLIP as a frozen objective function. Predicted HOI triplets are converted into phrases such as “A person” + verb + “a” + object, and the Image–Text Matching score is used in a margin-based contrastive loss,

$\mathbf{C}_K \in \mathbb{R}^{N_k \times 1}$0

Positive and negative pairs are defined at the level of image–HOI triplet text rather than global image–caption alignment, and the VLM is removed at inference time (Kang et al., 2024).

A second distillation-oriented line appears in CL-HOI. There, a VLLM generates captions and HOI triplets at image level, a Visual Linguistic Translator distills context, an Interaction Cognition Network reasons about spatial, visual, and context relations for detected human–object pairs, and contrastive distillation losses transfer image-level context and interaction knowledge into instance-level HOI detection. The losses include context alignment, image-to-text interaction alignment, text-to-image interaction alignment, and a Soft-Relation pseudo-label loss on union regions (Gao et al., 2024).

Video HOI localization under weak supervision provides another variant. “Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions” aligns attended human features to verb embeddings, attended object features to object embeddings, and uses a temporal contrastive loss to keep the selected interacted object consistent across frames. The construction is HOI-aware because the positives and negatives are defined over attended human and object regions conditioned on a verb–object query, not over generic video augmentations (Li et al., 2021).

Other image HOI detectors use contrastive pretrained spaces without always introducing an explicit contrastive loss over HOI triplets. HOICLIP retrieves localized interaction features from CLIP’s spatial feature map by a query-based interaction decoder, uses CLIP text embeddings as HOI classifiers, derives verb embeddings by visual semantic arithmetic, and adds a training-free global HOI prior at inference (Ning et al., 2023). SCTC distills interaction features from CLIP text embeddings and then propagates self-triplet and cross-triplet correlations through graph structures, which is close in spirit to HOI-aware metric learning even though its explicit loss is not formulated as InfoNCE (Jiang et al., 2024).

Two additional frameworks extend the scope of the concept. Self-Compositional Learning maintains an online updated concept confidence matrix over verb–object pairs and uses pseudo-labels for all composite HOI instances, enabling learning on both known and unknown HOI concepts; this matrix can be read as a compatibility model over HOI concepts (Hou et al., 2022). Uni-HOI is not a contrastive-learning paper, but it learns the joint distribution among text, human motion, and object motion using an LLM and two motion-specific VQ-VAEs, and its authors explicitly describe it as exactly the type of structure on top of which a Human-Object Interaction-Aware Contrastive Learning approach could be built (Zhang et al., 30 Apr 2026).

6. Empirical evidence, common distinctions, and limitations

For the canonical LiDAR formulation, ablation evidence on Waymo shows a monotonic gain as HOICL becomes more interaction-specific. Without global, FIR, or contact contrastive losses, MPJPE is $\mathbf{C}_K \in \mathbb{R}^{N_k \times 1}$1, PCK-3 is $\mathbf{C}_K \in \mathbb{R}^{N_k \times 1}$2, and PCK-5 is $\mathbf{C}_K \in \mathbb{R}^{N_k \times 1}$3. Adding global contrastive learning yields $\mathbf{C}_K \in \mathbb{R}^{N_k \times 1}$4, $\mathbf{C}_K \in \mathbb{R}^{N_k \times 1}$5, and $\mathbf{C}_K \in \mathbb{R}^{N_k \times 1}$6. Adding FIR contrastive learning yields $\mathbf{C}_K \in \mathbb{R}^{N_k \times 1}$7, $\mathbf{C}_K \in \mathbb{R}^{N_k \times 1}$8, and $\mathbf{C}_K \in \mathbb{R}^{N_k \times 1}$9. Adding contact contrastive learning yields the best reported result, with MPJPE $\mathbf{S}_p \in \mathbb{R}^{N \times K}$0, PCK-3 $\mathbf{S}_p \in \mathbb{R}^{N \times K}$1, and PCK-5 $\mathbf{S}_p \in \mathbb{R}^{N \times K}$2. In a direct comparison against SupCon, KCL, BCL, CBL, HiMulConE, and TSC, the same HOICL configuration remains best on Waymo (Jung et al., 17 Mar 2026).

Related literatures report analogous gains when the contrastive signal is made interaction-specific. Interaction-weighted Resampling improves over the best prior CRL baseline by about $\mathbf{S}_p \in \mathbb{R}^{N \times K}$3 on average across $\mathbf{S}_p \in \mathbb{R}^{N \times K}$4 simulation tasks, and in real robot air hockey improves success from $\mathbf{S}_p \in \mathbb{R}^{N \times K}$5 to $\mathbf{S}_p \in \mathbb{R}^{N \times K}$6 with $\mathbf{S}_p \in \mathbb{R}^{N \times K}$7 successes versus $\mathbf{S}_p \in \mathbb{R}^{N \times K}$8 for SGCRL and $\mathbf{S}_p \in \mathbb{R}^{N \times K}$9 for CRTR (Shen et al., 10 Jun 2026). VLM-HOI reaches $\mathbf{C}_p \in \mathbb{R}^{N \times 1}$0 default full mAP on HICO-DET and $\mathbf{C}_p \in \mathbb{R}^{N \times 1}$1 Role AP on V-COCO, with especially notable gains on rare categories relative to CLIP-based baselines (Kang et al., 2024). CL-HOI reaches $\mathbf{C}_p \in \mathbb{R}^{N \times 1}$2 mAP on HICO-DET and $\mathbf{C}_p \in \mathbb{R}^{N \times 1}$3 Role AP on V-COCO without manual HOI labels, and ablations show consistent gains from adding context, image-to-text, text-to-image, and soft-relation losses (Gao et al., 2024). In weakly supervised video HOI detection, phrase mAP in the known-object setting rises to $\mathbf{C}_p \in \mathbb{R}^{N \times 1}$4 and relation mAP to $\mathbf{C}_p \in \mathbb{R}^{N \times 1}$5, substantially above adapted weakly supervised baselines (Li et al., 2021).

A common misconception is to treat HOICL as a single loss template. The literature instead spans supervised point-level contrastive learning, margin-based Image–Text Matching losses, CLIP- or VLLM-based distillation losses, weakly supervised spatiotemporal region contrast, and interaction-aware positive resampling. This suggests that the defining property is not a fixed objective form, but the explicit use of interaction structure—contact regions, human–object triplets, or interaction phases—to choose anchors, positives, negatives, or teacher signals.

The main limitations are likewise method-specific. In HOIL, HOICL depends on dense contact and part supervision from meshes, is confined to synthetic pre-training, is limited by sparse LiDAR at long range, and does not fully cover all road-relevant interactions (Jung et al., 17 Mar 2026). In interaction-aware CRL, the energy remains fundamentally unimodal, while factorized state and heuristic interaction indicators are assumed (Shen et al., 10 Jun 2026). In VLM- or VLLM-based HOI detection, performance is sensitive to prompt design, margin calibration, and teacher quality, and training cost can increase substantially even if inference cost remains unchanged after the teacher is removed (Kang et al., 2024). CL-HOI additionally inherits image-level teacher ambiguities because the teacher does not specify which person interacts with which object (Gao et al., 2024).

Taken together, these results establish HOICL as a family of representation-learning strategies in which contrastive structure is aligned with human–object interaction itself. In the narrow, canonical sense, HOICL denotes the point-level supervised contrastive module that enables HOIL to separate human and object features in LiDAR contact regions. In the broader literature, it denotes a design principle: representation learning becomes more robust when contact, interaction phase, or HOI triplet semantics are treated as first-class structure rather than incidental context.