MEAN-RIR: Multi-Modal Environment-Aware Network

• MEAN-RIR is a multi-modal neural network that fuses audio, visual, and textual cues to accurately synthesize realistic room impulse responses.
• It employs dedicated encoders and hierarchical cross-modal attention to extract spatial and acoustic features from diverse environmental data.
• Its dual-decoder architecture separates early reflections from late reverberation, leading to improved metrics such as T60, DRR, and enhanced ASR performance.

The Multi-Modal Environment-Aware Network for Robust Room Impulse Response Estimation (MEAN-RIR) is a neural framework designed to synthesize accurate room impulse responses (RIRs) by leveraging environmental cues extracted from audio, visual, and textual modalities. MEAN-RIR employs an encoder-decoder architecture enhanced by cross-modal attention modules to capture salient spatial and acoustic properties of real environments, providing high-fidelity RIRs crucial for applications in speech processing, virtual/augmented reality, and robust automatic speech recognition.

1. Architecture Overview: Multi-Modal Encoder-Decoder Paradigm

MEAN-RIR utilizes a dedicated encoder for each of the audio, visual, and textual input streams. This tripartite design allows the system to extract complementary features vital for nuanced environment modeling:

• Audio Encoder: Utilizing a residual structure with 13 time-convolutional blocks (kernel size 15×1) and skip connections (1×1), inspired by FiNS, the encoder captures long-term dependencies in reverberant speech. The output is a 128-dimensional embedding, $A \in \mathbb{R}^{1\times 128}$, summarizing the temporal spectral information characteristic of room acoustics.
• Visual Encoder: A pre-trained ResNet18 processes panoramic RGB images, extracting features related to room geometry, layout, and object arrangements. This encoder yields a 512-dimensional image embedding, $V \in \mathbb{R}^{1\times 512}$, characterizing the spatial scene.
• Text Encoder: Employing a pre-trained BERT model, scene text descriptions (generated by a vision-language pipeline) are converted into a 768-dimensional embedding, $T \in \mathbb{R}^{1\times 768}$, encoding semantic information such as material types and room function.

The outputs from these encoders facilitate downstream cross-modal fusion, providing a rich composite environmental context for RIR synthesis.

2. Cross-Modal Attention Fusion

Interacting across modalities is achieved through a hierarchy of cross-attention modules:

• Pairwise Fusion: Audio-visual and audio-textual relationships are modeled separately via cross-attention, allowing the dominant audio embedding to query the complementary semantic and spatial representations:
  • Audio-Text: $F_{AT} = \mathcal{A}(A, T, T)$
  • Audio-Visual: $F_{AV} = \mathcal{A}(A, V, V)$

Here, $\mathcal{A}(\cdot)$ denotes the cross-modal attention operation, with queries vs. keys/values as indicated.

• Final Fusion: The two pairwise fused representations are integrated with a final cross-attention:
  • $F = \mathcal{A}(F_{AV}, F_{AT}, F_{AT})$, yielding a unified $F \in \mathbb{R}^{1\times 128}$ feature embedding.

This cross-attention hierarchy enables effective multi-level interaction among the distinct modalities, ensuring the decoder receives a contextually integrated representation reflective of both physical and semantic environmental characteristics.

3. RIR Component Synthesis and Decoder Design

The MEAN-RIR decoder reconstructs the RIR as the sum of two signal components that correspond to the physically distinct early and late portions of a room response:

• Early Component ($E$): Modeled as a learnable 0.05-second segment, this portion encodes the direct sound and early reflections that are critically dependent on room geometry and source-microphone configuration.
• Late Component ($M \odot N$): The late reverberation is synthesized by modulating a filtered noise signal $N$ by a decoder-generated mask $M$ (element-wise multiplication), reflecting the stochastic, noise-like properties of diffuse reverberation tail.
• Output Fusion: The early ($E$) and late ($M \odot N$) components are concatenated along the time axis, then passed through a final 1D convolution to yield the full RIR:

$$\widehat{\mathrm{RIR}} = \mathrm{Conv1D}(E \oplus (M \odot N))$$

The output is a 44,160-sample sequence, corresponding to 1 second at 44.1 kHz sampling.

This composite synthesis approach permits MEAN-RIR to flexibly model both deterministic (geometry-driven) and stochastic (energy decay) aspects of real-world reverberation, leading to more perceptually and physically accurate RIRs.

4. Quantitative Evaluation and Comparative Results

MEAN-RIR has been evaluated on the PanoIR dataset using multiple acoustic and ASR-relevant metrics:

Model	T₆₀ (ms)↓	DRR (dB)↑	EDT (ms)↓	ASR WER (%)↓
Audio-only	76.3	4.1	25.0	—
Audio-visual	54.2	5.2	19.4	—
MEAN-RIR	39.6	6.1	17.2	Lower

• T₆₀: Reverberation time reduction indicates improved modeling of energy decay.
• DRR: Increased direct-to-reverberant ratio implies better distinction of direct path vs. reverberation.
• EDT: Lower early decay time further confirms improved fidelity in modeling reflection patterns.
• ASR WER: When MEAN-RIR RIRs are used to synthesize reverberant speech for training, downstream speech recognizers exhibit lower word error rates than when trained without such augmentation.

These metrics demonstrate that the multi-modal, cross-attentive fusion in MEAN-RIR yields substantial gains over unimodal and bimodal baselines in both physical and perceptual room acoustics fidelity.

5. Comparative Context and Novelty

MEAN-RIR advances over preceding models such as AV-RIR (Ratnarajah et al., 2023), which employs a neural codec-based architecture and Geo-Mat visual features for RIR and dereverberation joint learning. While AV-RIR integrates RGB imagery and depth/material information with audio, MEAN-RIR incorporates not only panoramic imagery but also free-form textual descriptions processed via a dedicated BERT encoder, establishing a three-modality regime. The two-step cross-modal fusion is unique, and the explicit dual-branch decoder design structurally demarcates early/late reverberation, offering more granular physical interpretability and synthesis control.

This multi-encoder, cross-attentive configuration is a defining feature distinguishing MEAN-RIR from architectures where modality fusion is either early or naively concatenated.

6. Practical Applications and Broader Implications

MEAN-RIR is well suited for generating realistic reverberation signatures in virtual and augmented reality, immersive telepresence, and audio forensics. Its ability to synthesize highly plausible RIRs from sparse real-world cues has implications for:

• Speech enhancement: Facilitating dereverberation and noise reduction pipelines.
• Automatic speech recognition: Enhancing robustness under far-field and noisy conditions through data augmentation.
• Spatial audio rendering: Advancing the realism of audio in gaming, VR, and architectural simulation via accurate synthetic reverberation.
• Room acoustics design: Enabling rapid prototyping and optimization of built environments using predicted RIRs under various scene and material configurations.

A plausible implication is that the model’s three-way modality integration could form the basis for future context-aware spatial audio synthesis systems that require minimal annotated measurement data but produce physically grounded environmental acoustics.

7. Limitations and Prospective Extensions

While MEAN-RIR demonstrates quantitative and qualitative improvements, its design presumes access to reasonably informative visual and textual environmental data. For severely occluded or visually impoverished scenes, or environments lacking structured textual descriptions, the effectiveness of the visual/textual encoders as supplementary information channels may be diminished. Additionally, the system is presently oriented toward static environments; extension to dynamic or time-varying scenes would require further architectural innovation.

Future enhancements may include adaptation for time-varying environments, integration with depth/range sensors, or incorporation of explicit surface material classification pipelines, potentially yielding further improvements in RIR synthesis fidelity and domain robustness.