Sound-based Multi-Person 3D Pose Estimation

Abstract: Can we recover the 3D poses of multiple people using only sound? This paper presents the first attempt to estimate multi-person 3D poses solely from acoustic signals. Estimating the poses of multiple individuals using acoustic signals is inherently challenging due to the superposition of motion-dependent signal variations. Unlike single-person scenarios, the presence of multiple subjects leads to overlapping acoustic signatures, making it difficult to attribute specific signal changes to an individual's pose. Furthermore, the complexity is compounded by inter-person reflections, which introduce intricate propagation delays that obscure the temporal motion-acoustic relationship. To address these issues, we propose SoundMHPE (Sound-based Multi-person Human Pose Estimator), a novel encoder-decoder framework consisting of two key components. First, the Acoustic Multi-scale Encoder captures diverse temporal and fine-grained frequency features to isolate subtle acoustic signatures from complex, overlapping signals. Second, the Temporal Pose Decoder employs an attention mechanism to disentangle multi-person information across successive frames. By jointly accounting for temporal dynamics and inter-person dependencies, this component precisely reconstructs frame-wise individual poses. To validate our approach, we constructed the 6-hour Acoustic Multi-person Pose (AMP) dataset consisting of 432K synchronized frames of multi-person pose and acoustic data, and demonstrated that our SoundMHPE outperforms baseline models. Project page: https://oumi03.github.io/sound-mhpe/

• The paper proposes a novel architecture, SoundMHPE, specifically for multi-person 3D pose estimation from active acoustic sensing.
• SoundMHPE uses Acoustic Multi-scale Encoder and Temporal Pose Decoder modules to accurately reconstruct poses for up to three people, achieving a significant 11.2% reduction in MPJPE (mean perimeter joint position error) and a 0.43 PCKh (percentage of correct keypoints) compared to baseline methods.
• SoundMHPE demonstrates adaptability in varying acoustic settings but highlights the need for data and environment-specific robustness for full deployment.

Problem formulation and contribution

“Sound-based Multi-Person 3D Pose Estimation” (2609.04902) addresses multi-person 3D human pose estimation from active acoustic sensing alone. The task is substantially more difficult than single-person acoustic pose estimation because the received signal is a superposition of motion-dependent perturbations generated by multiple bodies. Subject-specific acoustic signatures are therefore not directly separable, and reflections between bodies introduce propagation delays that disrupt the temporal correspondence between a physical motion and its measured signal.

The paper makes four principal contributions. First, it formulates multi-person 3D pose estimation as an acoustic sensing problem and presents SoundMHPE, an encoder-decoder architecture designed for multiple interacting subjects. Second, it introduces an Acoustic Multi-scale Encoder (AME), which processes spectrograms computed at multiple temporal-frequency resolutions. Third, it proposes a Temporal Pose Decoder (TPD), which assigns temporally indexed pose queries to each candidate person and explicitly models both intra-person motion and inter-person acoustic interactions. Finally, it constructs the Acoustic Multi-person Pose (AMP) dataset, comprising six hours of synchronized acoustic and motion-capture data, approximately 432,000 frames, and single-, double-, and triple-person sequences.

The sensing system uses two loudspeakers to emit a time-stretched pulse (TSP) and a four-channel ambisonics microphone to record the reflected signal. Unlike vision-based systems, the method does not depend on illumination or line of sight. However, the experimental setting remains controlled: the data are collected indoors, and the paper characterizes the method primarily in relation to reverberation and reflective partitions rather than uncontrolled acoustic scenes.

Dataset and experimental protocol

AMP contains 15 participants—12 male and 3 female—with heights between 150 and 181 cm. Participants were divided into three groups, and the collection protocol included randomized subject pairings and positions. Each group contributed 72 minutes of single-person data, 24 minutes of double-person data, and 24 minutes of triple-person data. The subjects performed walking, twisting, and arm-raising motions at randomized speeds. Ground-truth skeletons contain 21 joints and were recorded at 20 frames per second using a 16-camera OptiTrack motion-capture system.

Figure 1: AMP acquisition using active acoustic sensing and synchronized motion capture, with single-, double-, and triple-person recordings across 15 participants.

The primary evaluation uses cross-subject generalization. Two groups, comprising ten subjects, are used for training and the remaining group, comprising five unseen subjects, is used for testing. The held-out group is rotated in a three-fold cross-validation procedure. This protocol is more informative than a random frame split because it evaluates whether the acoustic-to-pose mapping transfers across identities, although it does not independently test transfer across rooms, microphone placements, speaker configurations, or substantially different motion distributions.

Performance is reported using MPJPE, PA-MPJPE, and [email protected]. MPJPE retains errors in global translation, rotation, and scale, whereas PA-MPJPE removes these factors through Procrustes alignment. Consequently, the two metrics distinguish absolute pose reconstruction from articulated configuration accuracy.

SoundMHPE architecture

SoundMHPE consists of the AME followed by the TPD. The input is a sequence of four-channel acoustic measurements. The system predicts eight consecutive pose frames using 16 preceding frames as temporal context. The model is trained with a pose MSE loss and a binary cross-entropy loss for instance confidence, with Hungarian matching used to associate predicted subjects with ground-truth subjects.

Figure 2: SoundMHPE combines the Acoustic Multi-scale Encoder with the Temporal Pose Decoder and structured attention over temporal, frequency, motion, and interaction dimensions.

Acoustic Multi-scale Encoder

The AME begins by computing log-Mel spectrograms with three STFT window sizes: $L$, $2L$, and $4L$. The corresponding representations have progressively lower temporal resolution and higher frequency resolution. The $L$ representation preserves frame-aligned temporal dynamics, while the longer windows provide finer spectral resolution and a larger temporal context.

Figure 3: Multi-scale STFT produces high-temporal, intermediate, and high-frequency spectrogram representations from the same acoustic sequence.

The central design choice is not merely to concatenate these spectrograms, but to impose structured attention over them. Temporal Self-Attention (TSA) operates within each STFT resolution and models temporal dependencies at a fixed time-frequency scale. Frequency Self-Attention (FSA) links representations generated from different window sizes but corresponding to the same underlying acoustic interval. This separation is intended to prevent temporal evolution and cross-resolution spectral correspondence from being indiscriminately mixed by global self-attention.

The architectural rationale is technically appropriate for acoustic pose estimation. Short STFT windows can preserve transient motion cues but provide limited frequency discrimination; long windows improve frequency localization but blur temporal events. The AME therefore treats the time-frequency trade-off as a representation problem rather than relying on a single spectrogram resolution.

Temporal Pose Decoder

The TPD adapts the query-based structure of DETR-like multi-person estimators to acoustic sequences. A conventional person query that predicts an entire multi-frame sequence must compress all temporal information into one latent representation. SoundMHPE instead allocates $N_{\mathrm{out}}$ pose queries to each candidate subject, with one query associated with each predicted frame. Each query is conditioned on a temporal positional embedding and a subject-specific embedding.

Figure 4: Frame-indexed queries allow different decoder queries to attend to acoustic evidence associated with different temporal locations.

This design gives cross-attention access to frame-specific acoustic features. It is particularly relevant under multipath propagation, where the acoustic evidence for a motion may be temporally displaced or distributed across several frames. The decoder additionally uses one instance query per candidate subject to estimate an identity-independent confidence score. With 15 candidate subjects and eight output frames, the implementation uses 135 learnable queries.

The TPD decomposes decoder self-attention into Motion Self-Attention (MSA) and Interaction Self-Attention (ISA). MSA is restricted to queries belonging to the same subject and models intra-person temporal continuity. ISA connects a subject’s queries to those belonging to other subjects and models acoustic interference and inter-person dependencies.

Figure 5: Structured attention separates temporal and frequency dependencies in the encoder from motion and interaction dependencies in the decoder.

This decomposition is more than an architectural convenience. In the acoustic setting, interactions between people are not simply visual occlusions; they alter the propagation path and the observed reflected waveform. Explicitly modeling these dependencies gives the decoder a mechanism for using one person’s inferred temporal structure when resolving another person’s ambiguous acoustic evidence.

Quantitative results

SoundMHPE achieves the best results among the evaluated baselines on all three primary metrics.

Method	MPJPE (mm)	PA-MPJPE (mm)	[email protected]
Adapted Shibata et al.	121.7	71.5	0.36
Adapted Yan et al.	119.9	69.7	0.36
SoundMHPE	106.5	65.0	0.43

Relative to the strongest baseline, SoundMHPE reduces MPJPE by 13.4 mm, or approximately 11.2%, and increases [email protected] from 0.36 to 0.43. PA-MPJPE decreases by 4.7 mm. These results support the claim that the proposed architecture improves both absolute reconstruction and articulated pose accuracy, although the absolute MPJPE remains over 10 cm on the cross-subject benchmark.

The baselines are adapted rather than originally designed for this task. The first extends a single-person acoustic pose estimator with a multi-person regression head, while the second modifies a WiFi multi-person pose model to accept acoustic spectrograms. Thus, the comparison establishes the value of the proposed design relative to plausible implementations, but it does not constitute comparison with a large set of independently optimized acoustic multi-person models.

The qualitative results show that SoundMHPE handles twisting and simultaneous arm raising more effectively than the baselines. These motions produce relatively subtle or spatially localized acoustic perturbations, so the reported advantage is consistent with the AME’s emphasis on fine-grained frequency structure and the TPD’s subject- and frame-specific cross-attention.

Figure 6: Qualitative predictions for double- and triple-person sequences; the marked failure illustrates that subject separation remains imperfect.

Component analysis

The ablation results isolate substantial contributions from both principal modules.

Configuration	MPJPE (mm)	PA-MPJPE (mm)	[email protected]
Without AME	115.2	67.5	0.38
Without TPD	116.5	69.0	0.38
Full SoundMHPE	106.5	65.0	0.43

Removing AME increases MPJPE by 8.7 mm, while removing TPD increases it by 10.0 mm. The larger degradation without TPD indicates that frame-specific query allocation and temporal cross-attention are especially important. This supports the paper’s central claim that multi-person acoustic pose estimation cannot be adequately handled by adding a multi-person regression head to a single-person architecture.

The attention ablations provide a more granular result. Using TSA and FSA in the encoder together with MSA and ISA in the decoder yields 106.5 mm MPJPE and 0.43 PCKh. Replacing the encoder’s structured attention with standard attention raises MPJPE to 111.7 mm, while replacing the decoder’s structured attention raises it to 114.4 mm. The decoder-specific result again suggests that separating intra-person temporal modeling from inter-person interaction modeling is particularly consequential.

The selected STFT resolutions also matter. The $(L, 2L, 4L)$ configuration obtains 106.5 mm MPJPE, compared with 114.5 mm for $(L, 2L)$, 117.0 mm for $(L/2, L, 2L)$, and 118.5 mm for $(L/4, L/2, L)$. The poorer performance of configurations emphasizing shorter windows contradicts the intuitive expectation that increasingly fine temporal resolution is always beneficial. In this setting, spectral resolution from longer windows appears necessary for resolving subtle motion-dependent acoustic variations.

Figure 7: The proposed $(L, 2L, 4L)$ STFT configuration outperforms alternatives that omit long windows or emphasize only temporal resolution.

Scaling with the number of subjects

The authors compare single-person and triple-person settings to quantify the cost of acoustic superposition.

Method	Single-person MPJPE	Triple-person MPJPE	Single-person PCKh	Triple-person PCKh
Adapted Shibata et al.	111.3	124.5	0.39	0.39
Adapted Yan et al.	108.7	122.4	0.40	0.38
SoundMHPE	95.0	111.2	0.47	0.44

SoundMHPE degrades by 16.2 mm in MPJPE from one to three subjects, while the strongest baseline degrades by 13.7 mm. The proposed model remains more accurate in both regimes, but the larger absolute error in the triple-person condition shows that the architecture does not eliminate the intrinsic ambiguity caused by signal superposition. Notably, its PCKh decreases only from 0.47 to 0.44, indicating that many joints remain within the relatively coarse PCKh threshold despite increased metric error.

The paper’s claim that multi-person estimation can retain “competitive accuracy” is therefore supported in relative terms, but should not be interpreted as parity between single- and triple-person reconstruction. The triple-person MPJPE of 111.2 mm remains materially higher than the 95.0 mm single-person result.

Generalization and cross-modal transfer

The environmental generalization experiment introduces black partitions that alter acoustic reflections. SoundMHPE continues to recover coarse poses in the modified environment.

Figure 7: Qualitative performance in an unseen acoustically reflective environment created by black partitions.

This result indicates some robustness to reflection changes, but the experiment is limited in scope. It uses a single indoor configuration with added partitions rather than a systematic evaluation across room geometry, reverberation time, background noise, speaker-microphone displacement, or unseen sensing hardware. The paper appropriately presents real-world deployment as unresolved rather than claiming environment-independent acoustic pose estimation.

A separate experiment tests the architecture on the Person-in-WiFi 3D dataset. The waveform-specific multi-scale STFT is omitted, while the structured attention and TPD are applied to CSI features.

Method	MPJPE (mm)	PA-MPJPE (mm)	[email protected]
PiW (Single)	127.4	71.5	0.13
PiW (Multi)	161.0	82.7	0.06
SoundMHPE architecture	122.6	69.1	0.31

The proposed architecture improves MPJPE over PiW (Single) by 4.8 mm and increases PCKh from 0.13 to 0.31. More importantly, the diagnostic PiW (Multi) model performs substantially worse than the single-frame baseline, with MPJPE increasing to 161.0 mm. This supports the paper’s contradictory but important observation that simply extending a single-frame estimator to sequence prediction can damage performance. Frame-specific correspondence and structured temporal modeling are required; sequence length alone is not sufficient.

The cross-modal result strengthens the architectural interpretation of SoundMHPE, but it should not be read as evidence that the acoustic and WiFi sensing problems are interchangeable. The experiment transfers the attention and decoder principles, not the complete input representation or sensing pipeline.

Limitations and open questions

The strongest limitation is dataset and environmental scope. AMP is substantial for a newly defined task, but it contains only 15 subjects and is collected in one indoor room with a fixed speaker-microphone arrangement and a fixed sensing protocol. The cross-subject split tests identity generalization, but not broad domain generalization. The reflective-partition experiment provides useful evidence of robustness while remaining a limited perturbation study.

The method also assumes a fixed maximum number of candidate subjects, $2L$0, and relies on confidence thresholding to suppress unused instance queries. The paper does not establish how performance changes when the number of people exceeds the training range, when subjects enter or leave the sensing region, or when person count is unknown and highly variable. Similarly, the evaluation focuses on up to three simultaneous people even though the decoder is parameterized for more candidates.

The acoustic setup uses active TSP transmission and a four-channel ambisonics microphone. This gives the model controlled excitation and directional information, but raises practical questions about audibility, hardware synchronization, speaker placement, and interference from ordinary environmental sounds. The reported results do not isolate the contribution of ambisonics from that of the active waveform, nor do they evaluate robustness to competing sound sources or nonstationary noise.

Finally, the model’s association is established through Hungarian matching during training, but the paper does not provide a dedicated identity-switch analysis over longer sequences. Since subject-specific queries are learned representations rather than externally tracked identities, it remains open whether the method maintains temporally consistent subject assignments under crossing trajectories, prolonged occlusion-equivalent acoustic ambiguity, or substantial inter-person interaction.

Conclusion

SoundMHPE presents a coherent formulation of multi-person 3D pose estimation from active acoustic signals. Its principal technical contribution is the joint treatment of multi-resolution acoustic structure, frame-specific temporal queries, intra-person motion, and inter-person acoustic interaction. On the AMP benchmark, it reduces MPJPE to 106.5 mm, improves PA-MPJPE to 65.0 mm, and raises [email protected] to 0.43, outperforming the adapted baselines and remaining effective in triple-person scenes.

The results establish that multi-person acoustic pose estimation benefits from architectural mechanisms designed around the propagation and superposition properties of sound. They do not yet establish broad deployment robustness: the central open question is whether the observed gains persist across substantially different rooms, sensing geometries, acoustic conditions, subject counts, and uncontrolled background audio.