Robustness to Visual Input Corruption

Investigate the robustness of AV-STE to visual corruption, including occlusion and missing video, in full-duplex spoken dialogue systems.

Background

AV-STE enhances corrupted semantic speech tokens by combining noisy audio-token representations with synchronized lip video, while the downstream Moshi speech dialogue model remains frozen. The evaluation demonstrates robustness to acoustic noise and interfering speakers, but it assumes that the visual input is reliable.

The paper explicitly identifies robustness to visual corruption, occlusion, and missing video as unexplored. Resolving this problem would clarify whether AV-STE can maintain reliable semantic-token enhancement when the target speaker’s facial video is degraded or unavailable.