SAVN-CE: Semantic Audio-Visual Navigation

• SAVN-CE is a continuous 3D semantic audio-visual navigation task in which agents deduce both spatial layout and semantic identity of transient, sound-emitting targets.
• The approach leverages multimodal encoders and a memory-augmented goal descriptor network (MAGNet) to integrate RGB-D, binaural audio, and self-motion data for refined goal reasoning.
• Empirical evaluations demonstrate that MAGNet enhances success rates, SPL, and precise goal localization in challenging continuous and distracted indoor environments.

Searching arXiv for the main paper and closely related works mentioned in the provided data. Semantic Audio-Visual Navigation in Continuous Environments (SAVN-CE) is an embodied navigation task in which an agent must navigate in a continuous 3D indoor environment to reach a semantically defined, sound-emitting goal object, without access to goal coordinates or ground-truth maps. The agent perceives egocentric RGB-D images, binaural audio, and its own relative pose, while the goal does not emit sound at the beginning of the episode, starts after a random delay, lasts for a short, random duration, and may be intermittent. The task therefore requires joint inference of the spatial location and semantic category of the sound-emitting object, together with maintenance of that goal representation after the audio disappears. SAVN-CE was introduced with the multimodal transformer-based model MAGNet, which jointly encodes spatial and semantic goal representations and integrates historical context with self-motion cues for memory-augmented goal reasoning (Zeng et al., 20 Mar 2026).

1. Task definition and operating regime

SAVN-CE defines a semantic audio-visual navigation problem in a continuous 3D indoor environment. The goal is a visible object in the scene that emits a semantically meaningful sound, such as a creaking chair. The agent must reach that object rather than an arbitrary acoustic point source. It is not given goal coordinates or a ground-truth map, and must rely on multimodal observations to infer both what the goal is and where it is located (Zeng et al., 20 Mar 2026).

The observation at step $t$ is

$$\mathbf{O}_t = \{\mathbf{I}_t, a_{t-1}, \mathbf{p}_t, \mathbf{B}_t\},$$

where $\mathbf{I}_t$ denotes RGB-D images, $a_{t-1}$ the previous action, $\mathbf{p}_t = [x_t, y_t, \theta_t, t]$ the pose relative to the initial pose, and $\mathbf{B}_t$ the binaural audio waveform. RGB-D images are rendered at $128 \times 128$ resolution with $90^\circ$ field of view, and binaural audio is sampled at 16 kHz with 4,000 samples per simulation step. Time is discretized at 0.25 s per step (Zeng et al., 20 Mar 2026).

Although the physical state space is continuous, control is effected through discrete low-level actions with fine continuous effects: MoveForward translates the agent by 0.25 m, TurnLeft rotates it by $15^\circ$, TurnRight rotates it by $15^\circ$, and Stop terminates the episode. This design follows the continuous-navigation style of VLN-CE while avoiding the discrete grid constraint of earlier audio-visual navigation formulations (Zeng et al., 20 Mar 2026).

A central difficulty is that the goal sound is absent at the beginning of the episode, may be audible only briefly or intermittently, and then becomes completely silent for the remainder of the episode. This makes the task inherently partially observable. The agent must explore before any goal cue is available, exploit the limited sound-emitting interval to infer semantic and geometric goal information, and then continue navigation while the goal is silent (Zeng et al., 20 Mar 2026).

2. Continuous rendering and semantic grounding

SAVN-CE is built on SoundSpaces 2.0 and Habitat using Matterport3D scenes. Its principal environmental distinction is the abandonment of precomputed binaural Room Impulse Responses at discrete grid positions in favor of continuous, on-the-fly binaural rendering. Prior AVN and SAVN settings, including SoundSpaces-style and SAVi-style formulations, used precomputed binaural RIRs on discrete grids with limited orientations, which induced spatial discontinuity in audio, imposed substantial storage requirements, and restricted navigation to graph-like motion. SAVN-CE instead permits free movement in continuous 3D space with realistic odometry and temporally coherent observations (Zeng et al., 20 Mar 2026).

At each step, the binaural signal is constructed by computing binaural RIRs for the current pose for each sound source, convolving the source waveform with those RIRs, adding reverberation-tail contributions from previous steps, and summing across active sources into a two-channel binaural signal. Because reverberation decay may exceed the 0.25 s simulation step, retaining only current and previous-step RIR responses would truncate the reverberation trail. SAVN-CE therefore accumulates residual responses from all previous steps so that reverberation decays coherently over time as the agent moves. This yields temporally coherent and spatially smooth binaural observations (Zeng et al., 20 Mar 2026).

Audio features are derived using a 512-point FFT with hop length 160 samples. The extracted multi-channel representation comprises magnitude mean, sine and cosine of inter-channel phase difference, and inter-channel level difference. These four channels are passed to an audio encoder consisting of convolutional layers followed by a fully connected projection to produce an audio embedding $$\mathbf{O}_t = \{\mathbf{I}_t, a_{t-1}, \mathbf{p}_t, \mathbf{B}_t\},$$0 (Zeng et al., 20 Mar 2026).

The task is semantic because the sound source is attached to an actual visible object in the 3D scene, and goal semantics span 21 object categories adapted from SAVi. The agent is not explicitly provided a symbolic category label; it must infer the semantic category and spatial location from multimodal evidence. Distractors, when present, are drawn from 102 periodic sounds whose categories are disjoint from the goal object categories, introducing acoustic ambiguity without collapsing the semantic distinction between goal and distractor sets (Zeng et al., 20 Mar 2026).

3. Goal specification, formalization, and evaluation

Each episode is defined by a Matterport3D scene, the agent’s initial pose, the goal location and semantic category, and the onset time and duration of the goal sound. In distracted variants, a distractor with its own location and category is also present, sharing the same temporal boundaries as the goal sound. Goal sound onset is sampled uniformly from $$\mathbf{O}_t = \{\mathbf{I}_t, a_{t-1}, \mathbf{p}_t, \mathbf{B}_t\},$$1 seconds, and duration is sampled from $$\mathbf{O}_t = \{\mathbf{I}_t, a_{t-1}, \mathbf{p}_t, \mathbf{B}_t\},$$2. Many goal sounds are intermittent within their emitting window, and after the window ends the goal becomes completely silent (Zeng et al., 20 Mar 2026).

The paper does not present a full POMDP derivation, but it explicitly characterizes the formulation as effectively a partially observable sequential decision problem with hidden state $$\mathbf{O}_t = \{\mathbf{I}_t, a_{t-1}, \mathbf{p}_t, \mathbf{B}_t\},$$3, observation $$\mathbf{O}_t = \{\mathbf{I}_t, a_{t-1}, \mathbf{p}_t, \mathbf{B}_t\},$$4, and actions $$\mathbf{O}_t = \{\mathbf{I}_t, a_{t-1}, \mathbf{p}_t, \mathbf{B}_t\},$$5. Reward at time step $$\mathbf{O}_t = \{\mathbf{I}_t, a_{t-1}, \mathbf{p}_t, \mathbf{B}_t\},$$6 includes a success reward of +10 when Stop is issued within 1 m geodesic distance of the goal and not near a distractor or another same-category instance, a shaping reward proportional to change in geodesic distance to the goal, and a time penalty of $$\mathbf{O}_t = \{\mathbf{I}_t, a_{t-1}, \mathbf{p}_t, \mathbf{B}_t\},$$7 per step. Episodes terminate when Stop is issued or when 500 steps are reached (Zeng et al., 20 Mar 2026).

Success is defined as stopping with geodesic distance to goal at most 1 m. Stopping near the distractor or a different instance of the same category counts as failure. Evaluation uses Success Rate (SR), SPL, SNA, Distance To Goal (DTG), Success While Silent (SWS), and Distractor Success Rate (DSR). SPL is the standard success-weighted path-length metric, SNA weights success by the number of executed actions relative to oracle actions, SWS measures the fraction of episodes in which the goal is reached after it has gone silent, and DSR measures episodes in which the agent incorrectly stops near the distractor (Zeng et al., 20 Mar 2026).

The goal representation used inside MAGNet’s Goal Descriptor Network is expressed in ACCDDOA form: $$\mathbf{O}_t = \{\mathbf{I}_t, a_{t-1}, \mathbf{p}_t, \mathbf{B}_t\},$$8 where $$\mathbf{O}_t = \{\mathbf{I}_t, a_{t-1}, \mathbf{p}_t, \mathbf{B}_t\},$$9 indexes semantic category, $\mathbf{I}_t$0 is the unit direction-of-arrival vector, $\mathbf{I}_t$1 is activity, and $\mathbf{I}_t$2 is normalized distance. This representation jointly encodes detection, localization, and semantics (Zeng et al., 20 Mar 2026).

4. MAGNet architecture

MAGNet, expanded as Memory-Augmented Goal descriptor Network, comprises three major components: a multimodal observation encoder, a Memory-Augmented Goal Descriptor Network (GDN), and a context-aware policy network. Its central design objective is to maintain a stable joint spatial-semantic representation of the goal even when audio is intermittent or absent (Zeng et al., 20 Mar 2026).

The multimodal observation encoder processes RGB and depth with two independent ResNet-18 backbones and concatenates their outputs into $\mathbf{I}_t$3. The previous action $\mathbf{I}_t$4 is mapped to a learnable embedding $\mathbf{I}_t$5. Pose is normalized as

$\mathbf{I}_t$6

with $\mathbf{I}_t$7, and projected to $\mathbf{I}_t$8. The audio encoder produces $\mathbf{I}_t$9, and the GDN contributes a goal embedding $a_{t-1}$0. These are concatenated into the observation embedding

$a_{t-1}$1

The most recent $a_{t-1}$2 observation embeddings are stored as scene memory,

$a_{t-1}$3

with $a_{t-1}$4 (Zeng et al., 20 Mar 2026).

The GDN receives audio, self-motion, and action history. Specifically, $a_{t-1}$5, $a_{t-1}$6, and $a_{t-1}$7 are fused via an MLP: $a_{t-1}$8 These fused vectors populate episodic goal memory,

$a_{t-1}$9

with capacity $\mathbf{p}_t = [x_t, y_t, \theta_t, t]$0. The sequence is augmented with positional encodings and passed through a transformer encoder with causal attention,

$\mathbf{p}_t = [x_t, y_t, \theta_t, t]$1

One branch projects the current hidden state to a dense goal embedding,

$\mathbf{p}_t = [x_t, y_t, \theta_t, t]$2

while a second branch predicts ACCDDOA outputs for supervised training (Zeng et al., 20 Mar 2026).

The context-aware policy network is a transformer encoder-decoder operating over scene memory. The encoder maps $\mathbf{p}_t = [x_t, y_t, \theta_t, t]$3 to encoded memory $\mathbf{p}_t = [x_t, y_t, \theta_t, t]$4, and the decoder combines $\mathbf{p}_t = [x_t, y_t, \theta_t, t]$5 with an MLP-transformed current observation embedding to form a context-aware latent state

$\mathbf{p}_t = [x_t, y_t, \theta_t, t]$6

Actor and critic heads project $\mathbf{p}_t = [x_t, y_t, \theta_t, t]$7 to a policy distribution $\mathbf{p}_t = [x_t, y_t, \theta_t, t]$8 and value estimate $\mathbf{p}_t = [x_t, y_t, \theta_t, t]$9, respectively (Zeng et al., 20 Mar 2026).

The rationale for the design is explicit. Self-motion cues are needed because turns alter azimuth and forward motion alters both distance and relative direction, so interpreting binaural cues over time depends on egomotion. Episodic memory is needed because short and intermittent audio segments may be semantically ambiguous, and the agent must preserve goal information once the source becomes silent. The causal transformer prohibits access to future observations during training (Zeng et al., 20 Mar 2026).

5. Training protocol and computational setting

Training alternates between policy rollouts and parameter updates. In each iteration, the current policy is rolled out for 150 steps per environment. After rollout, the GDN is trained in a supervised manner and the policy network is trained with reinforcement learning. Unfinished episodes are merged across iterations to form full trajectories for GDN supervision (Zeng et al., 20 Mar 2026).

The GDN is trained with oracle ACCDDOA labels using mean squared error between predicted and oracle outputs. Episodes shorter than 30 steps are discarded to ensure that the goal has emitted sound. Optimization uses Adam with learning rate $\mathbf{B}_t$0. This setup is characterized as SELD-style supervision, since the target jointly encodes sound event activity, localization, and distance (Zeng et al., 20 Mar 2026).

The policy is trained with Decentralized Distributed Proximal Policy Optimization (DD-PPO), using the standard PPO objective with clipped surrogate policy loss, value loss, and entropy regularization, optimized by Adam with learning rate $\mathbf{B}_t$1. The overall learning process follows a two-stage paradigm similar to SAVi, consisting of pretraining followed by fine-tuning. All learning-based methods are trained up to 240M steps, with early stopping if validation does not improve for 24M steps (Zeng et al., 20 Mar 2026).

SAVN-CE uses 0.5M training episodes, 500 validation episodes, and 1,000 test episodes. The splits are scene-disjoint and sound-disjoint: different Matterport3D scenes and different source sounds are used across train, validation, and test. In the test split, the average oracle number of actions is 78.49, compared with 26.52 in discrete SAVN, indicating substantially longer and harder tasks under continuous navigation (Zeng et al., 20 Mar 2026).

The computational cost is nontrivial. Reported training uses 128 CPU threads and 4 NVIDIA A800 GPUs, and 240M-step training with binaural audio takes approximately 14 days. A plausible implication is that the shift from discrete RIR lookup to continuous runtime rendering materially increases the systems burden of experimentation (Zeng et al., 20 Mar 2026).

6. Empirical performance, baselines, and ablations

SAVN-CE evaluates MAGNet against Random, ObjectGoal, AV-Nav, SMT + Audio, SAVi, and two oracle upper bounds. Random samples actions according to the training-set distribution. ObjectGoal receives RGB-D and ground-truth goal category but no audio, and must still learn when to stop. AV-Nav is a GRU-based audio-visual navigation model. SMT + Audio extends Scene Memory Transformer with audio but without an explicit goal inference module. SAVi is the original semantic AVN model with a heuristic temporal aggregation factor $\mathbf{B}_t$2. Oracle1 and Oracle2 replace the audio encoder and GDN with oracle ACCDDOA labels, available only during sound-active periods in Oracle1 and throughout the entire episode in Oracle2 (Zeng et al., 20 Mar 2026).

On the test split in clean environments, MAGNet attains an SR of 37.7%, compared with 25.6% for SAVi, 24.8% for SMT + Audio, 21.3% for AV-Nav, 0.8% for ObjectGoal, and 0.3% for Random. The paper reports this as a $\mathbf{B}_t$3 absolute improvement in success rate over SAVi. MAGNet also improves SPL from 21.2% to 32.9%, SNA from 17.3% to 27.4%, DTG from 10.1 m to 8.0 m, and SWS from 6.0% to 10.6%. Oracle1 reaches 41.4% SR, while Oracle2 reaches 75.0%, leaving a substantial gap between learned and perfect goal representations, especially when goal information remains available after silence (Zeng et al., 20 Mar 2026).

In distracted environments containing both goal and distractor, all methods degrade. SAVi achieves 18.5% SR and MAGNet 19.3% SR; SPL rises from 14.9% to 16.5% under MAGNet, but SWS declines slightly from 5.4% to 4.8%. DSR is 6.7% for AV-Nav, 6.4% for SAVi, and 7.8% for MAGNet. The reported interpretation is that MAGNet maintains modest gains in the presence of distractors but remains vulnerable to highly similar distractor sounds (Zeng et al., 20 Mar 2026).

The paper further analyzes performance as a function of action ratio and geodesic distance. Action ratio is defined as the oracle number of actions divided by the number of actions available during the sound-on period, so higher ratios indicate that a larger fraction of navigation must be completed without audio. Success saturates at lower cumulative rates as either action ratio or geodesic distance increases, but MAGNet retains a consistently higher cumulative SR than the baselines across these axes. This suggests robustness to short sound durations and long-distance navigation scenarios, matching the claim made in the abstract (Zeng et al., 20 Mar 2026).

Ablation results isolate the contributions of explicit goal inference, episodic memory, and self-motion. A version with no GDN reaches 32.4% SR, 27.9% SPL, and 6.3% SWS. Adding a GDN without memory ($\mathbf{B}_t$4) yields 33.9% SR, 29.8% SPL, and 8.9% SWS. Using a GDN with memory but without self-motion produces 34.3% SR, 30.4% SPL, and 7.8% SWS. Full MAGNet achieves 37.7% SR, 32.9% SPL, and 10.6% SWS. The reported conclusion is that episodic memory and self-motion are complementary, and that improvements in SELD metrics correlate with improvements in navigation performance (Zeng et al., 20 Mar 2026).

7. Interpretation, limitations, and research significance

The GDN is evaluated with standard SELD metrics including $\mathbf{B}_t$5, $\mathbf{B}_t$6, $\mathbf{B}_t$7, $\mathbf{B}_t$8, and RDE. During sounding periods in clean environments, MAGNet improves over SAVi from 0.882 to 0.762 on $\mathbf{B}_t$9, from 0.148 to 0.290 on $128 \times 128$0, from $128 \times 128$1 to $128 \times 128$2 on $128 \times 128$3, from 0.499 to 0.601 on $128 \times 128$4, and from 0.270 to 0.117 on RDE. During silent periods, SAVi’s $128 \times 128$5 drops to extremely low values, whereas MAGNet preserves substantially higher $128 \times 128$6 and $128 \times 128$7, reported as 0.140 and 0.368 in clean environments. The paper explicitly links these SELD gains to improved navigation (Zeng et al., 20 Mar 2026).

The significance of SAVN-CE lies in the conjunction of three properties: continuous navigation, semantically grounded sound sources, and temporally limited goal audibility. Earlier benchmark formulations could often rely on graph-constrained movement and continuously available acoustic cues. SAVN-CE instead forces the agent to reason over time about a goal that may only be partially observed acoustically and must later be pursued in silence. This suggests a closer alignment with embodied settings in which acoustic evidence is transient, egocentric motion changes cue geometry continuously, and semantic grounding cannot be reduced to localization alone (Zeng et al., 20 Mar 2026).

The paper identifies several limitations. First, MAGNet remains below Oracle1 and far below Oracle2, indicating persistent error in goal estimation, especially after the goal becomes silent. Second, all methods struggle in distracted environments, and MAGNet’s DSR indicates susceptibility to distractor confusion. Third, continuous binaural rendering makes training computationally expensive. Fourth, the benchmark focuses on a single stationary sound-emitting goal plus possible distractor, leaving moving sources and multiple goals unaddressed (Zeng et al., 20 Mar 2026).

Future directions named in the work include extending the framework to multiple goal objects, moving sound sources, greater robustness to distractors and ambiguous semantics, and improved architectures for long-horizon goal memory and reasoning. Within the scope of the reported results, the principal conclusion is that memory-augmented multimodal goal reasoning is critical when semantic audio-visual navigation is performed in continuous environments and the audio cue is intermittent or absent for much of the episode (Zeng et al., 20 Mar 2026).