Social-pose: Socially-Aware Pose Representation

• Social-pose is a representation paradigm that encodes explicit pose-derived social cues, including body configuration, interaction, and intent.
• It leverages varied formulations—from coupled local-global motion to tokenized joint trajectories and minimal 3D descriptors—to model multi-person social dynamics.
• Applications span forecasting, action recognition, and human–robot interaction, where integrating social-pose enhances safety and predictive performance in crowded environments.

Searching arXiv for recent and foundational papers on “social-pose” and closely related multi-person pose/social-motion methods. arxiv_search(query="social pose multi-person motion pose forecasting social-aware pose estimation", max_results=10) Social-pose denotes the use of pose-derived variables as explicit carriers of social information in visual computing. In the recent literature, the term is used in several related but non-identical senses: unified forecasting of body skeleton pose and global motion, pose-conditioned trajectory prediction, compact 3D descriptors of interpersonal geometry such as head position and facing direction, and multi-person pose representations optimized for social grouping, action understanding, contact reconstruction, or socially aware planning (Adeli et al., 2020, Gao et al., 30 Jul 2025, Qin et al., 6 Nov 2025, Ehsanpour et al., 2024). What unifies these formulations is the premise that body configuration is not merely kinematic state; it is also a socially informative signal that encodes interaction, attention, coordination, avoidance, grouping, and intent.

1. Conceptual scope

Social-pose is distinct from generic pose estimation. Conventional pose pipelines primarily localize joints, recover skeletons, or track identities. Social-pose methods instead treat pose as an intermediate or final representation for higher-level reasoning about multi-person scenes. This orientation is explicit in work arguing that complete comprehension of multi-person scenes requires moving beyond detection and tracking toward interactions and social activities, and in datasets built for crowded human-robot settings where pose estimation and tracking are prerequisites for understanding human motion and body dynamics over time (Ehsanpour et al., 2024, Vendrow et al., 2022).

Across the literature, social-pose operates at several scales. At the fine scale, it may refer to face crops used for joint head-pose regression and eye-contact detection, as in the Pose-Implicit CNN, which jointly predicts Euler angles and eye contact from a single face crop (Chong et al., 2019). At the body scale, it may denote full local joint configurations plus global motion, forecast jointly in a recurrent pipeline (Adeli et al., 2020). At the group scale, it can denote representations that explicitly encode inter-person relations, as in transformer or diffusion models that operate on all people in a scene and model social dependencies through attention, pooling, or conditioning (Ehsanpour et al., 2024, Markhorst et al., 8 Apr 2026).

A central consequence is that social-pose is not a single canonical representation. It is a family of pose-centered formulations whose defining property is relationality: the representation is designed to preserve socially salient structure rather than only anatomical correctness.

2. Representational forms

One major formulation couples local skeleton motion and global trajectory. In “Socially and Contextually Aware Human Motion and Pose Forecasting,” each person’s history consists of global motions, represented as body-center displacements, and local joint offsets relative to the torso, concatenated and fed to a shared GRU encoder. Forecasting is trained with a joint loss,

$$L_{joint} = \alpha \sum_{\tau=t+1}^T d_{traj}(\hat x_\tau, x_\tau) + (1-\alpha) \sum_{\tau=t+1}^T d_{pose}(\hat p_\tau, p_\tau),$$

with squared-error trajectory and pose terms (Adeli et al., 2020). In this formulation, social-pose is already a coupled state: where a person is going and how the body is configured are learned together.

A second formulation represents multi-person pose as tokenized joint trajectories. Social-MAE preprocesses each joint trajectory by subtracting each person’s global translation, applying a DCT, retaining the first $K$ low-frequency coefficients, and augmenting each token with learned embeddings for joint-type, person-ID, and global offset. The encoder reconstructs masked joint trajectories in the frequency domain, using tube masking over entire joint histories and a reconstruction loss over masked entries (Ehsanpour et al., 2024). SoMoFormer similarly reformulates motion input as a set of joint trajectories rather than a time sequence of full skeletons, adding learned embeddings for joint type, person identity, and global position so that attention can model both intra-body and inter-person dependencies (Vendrow et al., 2022).

A third formulation compresses social-pose to a minimal geometric descriptor. Qin and Isik define a 12-dimensional dyadic “social-pose” feature by assigning each person a 6-dimensional vector consisting of 3D head-center position and head-orientation unit vector:

$[x_i, y_i, z_i, d_i^x, d_i^y, d_i^z].$

The head center is computed from the eye joints, and the facing direction from the nose, eye center, and neck (Qin et al., 6 Nov 2025). This representation omits most of the skeleton while preserving explicit 3D visuospatial cues that are strongly predictive of social judgments.

A fourth formulation uses pose as an auxiliary modality for trajectory prediction. In “Social-Pose: Enhancing Trajectory Prediction with Human Body Pose,” each agent has global Cartesian coordinates and pelvis-centered local keypoints, with 2D or 3D pose encoded by a Transformer and concatenated with trajectory embeddings before social interaction modeling (Gao et al., 30 Jul 2025). Here social-pose is not the output target; it is the conditioning signal that enriches trajectory prediction.

These formulations differ in dimensionality and inductive bias, but all attempt to retain socially relevant structure: body-part coordination, interpersonal geometry, or interaction-aware motion regularities.

3. Modeling paradigms

The earliest social-pose architectures in this corpus are recurrent and pooling-based. The 2020 Social-Pose framework uses a shared GRU encoder-decoder, an I3D-based spatio-temporal scene context encoder, and a permutation-invariant social pooling layer over person embeddings, with max-pooling reported as best in practice (Adeli et al., 2020). The model’s defining idea is that local pose, global motion, scene context, and other people’s motion should be encoded jointly rather than in separate pipelines.

Transformer-based models generalize this idea by replacing pooled social context with token-level attention. SoMoFormer predicts multi-person 3D pose non-autoregressively from joint-sequence tokens and learns that attention weights between two people’s joints decay as their ground-plane distance grows (Vendrow et al., 2022). Social-MAE adds self-supervised masked modeling: the encoder learns to fill in missing joint trajectories from the context of other joints and other people’s motion, and the masked autoencoder is then fine-tuned for downstream social tasks (Ehsanpour et al., 2024). MuPPet extends this line to 2D-to-3D lifting with explicit Person Encoding, Permutation Augmentation, and Dynamic Multi-Person Attention, addressing varying group size and inter-person correlations in social scenes (Markhorst et al., 8 Apr 2026).

Graph-based reasoning remains important when the target is explicitly relational rather than predictive. The Multi-Granularity Reasoning framework for social relation recognition constructs a pose-guided Person-Object Graph and Person-Pose Graph, then applies GCNs to reason over interactions between persons and objects and between paired persons (Zhang et al., 2019). This places pose at the center of a structured relational graph rather than a purely sequential model.

Diffusion methods introduce generative social-pose modeling. A diffusion-based imitation-learning system for social pose generation learns facilitator behavior from raw or plotted image observations and evaluates the resulting motion with MPJPE, training time, and inference time (Martin-Ozimek et al., 18 Jan 2025). SEE-ME performs egocentric mesh estimation by latent diffusion conditioned on scene structure and an interactee’s mesh sequence, showing that social cues can regularize pose estimation even when the wearer’s body is out of view (Scofano et al., 2024). ProsePose uses a large multimodal model to extract coarse contact descriptions such as “hand touching back,” converts them into differentiable distance penalties over SMPL-X regions, and optimizes 3D social pose reconstruction without task-specific contact supervision (Subramanian et al., 2024).

These modeling families differ substantially, but they share an architectural pattern: social-pose is rarely treated as isolated per-person geometry; it is learned through interaction with scene context, other people, or semantically structured constraints.

4. Downstream tasks and empirical results

The most established social-pose task is forecasting. On PoseTrack, the Social-Pose GRU framework reports an average MSE of $43.0$ pixels at the $80$ ms horizon for the final Social+Context model, compared with approximately $72.4$ for a local-pose-only plus zero-motion baseline. On NTU RGB+D, the same framework reports $13.1$ cm MSE for joint learning and $12.8$ cm for the Social-only variant at the $80$ ms horizon, outperforming the $14.1$ cm zero-velocity baseline (Adeli et al., 2020).

Transformer pretraining further improves high-level social tasks. Social-MAE reports state-of-the-art results on multi-person pose forecasting, social grouping, and social action understanding. On the SoMoF benchmark, it achieves $K$0 overall VIM versus $K$1 for the previous best; on CMU-Mocap and MuPoTS-3D it reports MPJPE of $K$2 m at $K$3 s versus a $K$4 baseline. On JRDB-Act, grouping mAP on validation rises to $K$5 from $K$6, and action mAP on validation rises to $K$7 from $K$8 (Ehsanpour et al., 2024).

Pose as conditioning also improves pure trajectory prediction. Across LSTM-, GAN-, MLP-, and Transformer-based backbones, the 2025 Social-Pose encoder improves ADE/FDE on synthetic and real datasets. A notable result is Autobots on JTA, which improves from $K$9 m to $[x_i, y_i, z_i, d_i^x, d_i^y, d_i^z].$0 m with 3D pose; on JRDB, Autobots improves from $[x_i, y_i, z_i, d_i^x, d_i^y, d_i^z].$1 to $[x_i, y_i, z_i, d_i^x, d_i^y, d_i^z].$2; and 3D pose outperforms 2D pose on the same framework (Gao et al., 30 Jul 2025).

The empirical record also shows that social-pose need not be high-dimensional. Qin and Isik report that a 12-dimensional 3D social-pose descriptor matches the predictive strength of full 270-dimensional 3D joints for five human social judgment dimensions, and that concatenating these features to off-the-shelf AI vision model embeddings improves nearly all models, with especially large gains for “agents facing” (Qin et al., 6 Nov 2025). This suggests that a substantial fraction of socially relevant pose information may be linearly accessible from explicit 3D geometry even when it is poorly represented in generic vision embeddings.

Outside forecasting, social-pose is effective in reconstruction and egocentric estimation. ProsePose reduces PA-MPJPE on Hi4D from $[x_i, y_i, z_i, d_i^x, d_i^y, d_i^z].$3 mm to $[x_i, y_i, z_i, d_i^x, d_i^y, d_i^z].$4 mm and improves PCC on multiple datasets using language-derived contact priors (Subramanian et al., 2024). SEE-ME reduces EgoBody MPJPE from $[x_i, y_i, z_i, d_i^x, d_i^y, d_i^z].$5 mm for EgoEgo to $[x_i, y_i, z_i, d_i^x, d_i^y, d_i^z].$6 mm when conditioned on both scene and interactee cues, with the largest gains when people are within $[x_i, y_i, z_i, d_i^x, d_i^y, d_i^z].$7 m or are mutually looking at each other (Scofano et al., 2024).

5. Benchmarks and evaluation regimes

Social-pose research spans heterogeneous benchmarks. PoseTrack provides 2D multi-person sequences with $[x_i, y_i, z_i, d_i^x, d_i^y, d_i^z].$8 keypoints per person and a $[x_i, y_i, z_i, d_i^x, d_i^y, d_i^z].$9-frame observation to $43.0$0-frame prediction setup, while NTU RGB+D 60 provides a 3D multi-person subset with $43.0$1 body joints per person and a $43.0$2-frame observation to $43.0$3-frame prediction setup (Adeli et al., 2020). SoMoF, 3DPW, AMASS, CMU-Mocap, and MuPoTS-3D are standard for 3D multi-person forecasting, typically evaluated with VIM or MPJPE (Ehsanpour et al., 2024, Vendrow et al., 2022).

JRDB-Pose is a central benchmark for crowded robotic settings. It contains $43.0$4 sequences over $43.0$5 minutes, $43.0$6 annotated panoramic frames, approximately $43.0$7 pose instances, $43.0$8 head-box annotations, and $43.0$9 unique identities. Each person has $80$0 body keypoints, per-keypoint occlusion labels in $80$1, a 2D head box, and a consistent track ID shared across multiple annotation modalities (Vendrow et al., 2022). Its evaluation protocol combines COCO-style pose AP based on OKS with tracking metrics such as MOTA and MOTP, reflecting the fact that social-pose in robotics is inseparable from persistent identity under occlusion.

Additional benchmarks evaluate specialized aspects of social-pose. JRDB-Act targets grouping and action understanding from 2D pose (Ehsanpour et al., 2024). HHCD evaluates multimodal social signal prediction with body pose, head pose, gaze, and speech tokenization in triadic dining conversations (Tang et al., 23 Jan 2025). Hi4D, FlickrCI3D, CHI3D, and MOYO test contact-aware reconstruction (Subramanian et al., 2024). EgoBody and GIMO test egocentric mesh estimation with scene and interactee cues (Scofano et al., 2024). In animal behavior, SemiMultiPose evaluates honeybees, mouse pups, and flies under sparse-label regimes, while BigMaQ and BigMaQ500 provide mesh-based 3D social-pose for interacting rhesus macaques (Blau et al., 2022, Martini et al., 23 Feb 2026).

The evaluation landscape is therefore fragmented by task: MPJPE and VIM for motion, ADE/FDE for trajectories, mAP for grouping and action, PCC for contact recovery, and OKS/AP or MOTA for perception and tracking. A plausible implication is that “social-pose” is better understood as a cross-cutting representational paradigm than as a single benchmark task.

6. Applications, limitations, and research directions

Social-pose has direct relevance to human-robot interaction. JRDB-Pose was designed around robot-height egocentric perception in crowded indoor and outdoor environments (Vendrow et al., 2022). SAP-CoPE uses cooperative 3D human pose estimation and a personal-space field inside an MPC controller; in its reported experiments, the planner achieved zero collisions and a $80$2–$80$3 reduction in comfort-zone penetration metrics while respecting human personal space (Ning et al., 8 Apr 2025). In a related direction, Social-Pose-enhanced trajectory prediction improves a rule-based Social-Force navigator, reducing completion time from $80$4 s to $80$5 s and collision rate from $80$6 to $80$7 in approximately $80$8 JTA scenarios (Gao et al., 30 Jul 2025).

Social-pose is also central to social perception. Pose-Implicit CNN jointly predicts head pose and eye contact, reporting precision $80$9, recall $72.4$0, F1 $72.4$1, and AUC-PR $72.4$2 over $72.4$3K annotated test frames (Chong et al., 2019). M3PT uses quantized pose and head-pose tokens with person-aware blockwise attention to improve speaking-status and bite-timing prediction in multi-party dining interactions (Tang et al., 23 Jan 2025). In social inference, “Spot The Ball” shows that humans outperform current VLMs by a factor of roughly $72.4$4–$72.4$5 on hidden-ball localization, with humans at $72.4$6–$72.4$7 accuracy and models at $72.4$8; the analysis attributes the gap to models relying on near-player and center biases rather than structured gaze and body-pose cues (Balamurugan et al., 31 Oct 2025).

The same ideas generalize beyond humans. SemiMultiPose leverages unlabeled frames for multi-animal pose estimation and improves AP in sparse-label regimes, particularly for social-behavior videos (Blau et al., 2022). BigMaQ extends social-pose to subject-specific 3D macaque avatars and reports that adding pose descriptors to visual models consistently improves action-recognition mAP, with especially marked gains for social interaction categories (Martini et al., 23 Feb 2026).

Several limitations recur across the literature: scarcity of annotations for high-level social tasks, heavy occlusion in crowded scenes, variable group size, viewpoint sensitivity, and the difficulty of extracting robust social cues from generic vision backbones (Ehsanpour et al., 2024, Vendrow et al., 2022, Markhorst et al., 8 Apr 2026, Subramanian et al., 2024). Another recurring theme is that current end-to-end vision models often fail to expose the simple geometric primitives that humans exploit, even when those primitives can be computed explicitly from pose (Qin et al., 6 Nov 2025, Balamurugan et al., 31 Oct 2025). This suggests that future progress may depend less on ever-larger generic embeddings than on architectures that preserve structured relational geometry—joint trajectories, contact regions, head orientation, social distance, and person identity—as first-class variables in perception, forecasting, and decision-making.