J-Moshi: Japanese Full-Duplex Dialogue System

• J-Moshi is a full-duplex dialogue system that extends the Moshi architecture to Japanese, accurately handling overlaps and backchannels in conversation.
• It combines a neural Mimi codec with a retrained 7B-parameter RQ-Transformer, leveraging large-scale Japanese speech data and synthetic dialogue augmentation.
• Evaluation shows J-Moshi outperforms from-scratch models in naturalness and turn-taking, effectively capturing Japanese conversational dynamics.

J-Moshi is a full-duplex spoken dialogue system for Japanese, developed as a direct extension of the English Moshi architecture. It is the first publicly available model designed to support the simultaneous bidirectional conversational phenomena native to human dialogue—such as overlaps and backchannels—in Japanese. The system employs a modular framework that combines a neural audio codec and a large transformer-based “speech-language” model. J-Moshi achieves high performance by leveraging large-scale Japanese speech data, targeted architectural modifications, and synthetic data augmentation via text-to-speech synthesis. Its design, training, and evaluation explicitly target the naturalness, meaningfulness, and turn-taking properties characteristic of Japanese conversations (Ohashi et al., 3 Jun 2025).

1. Architectural Framework

J-Moshi inherits its architecture from the original English Moshi (Défossez et al., 2024), comprising two main components: the Mimi codec and the RQ-Transformer.

• Mimi Neural Codec: An encoder/decoder built on SEANet with a residual vector quantizer. It transforms 24 kHz audio waveforms into discrete token streams at 12.5 Hz. Each frame encodes 8 quantized layers: layer 1 carries “semantic” information, while layers 2–8 encode “acoustic” details.
• RQ-Transformer: A 7-billion-parameter model with two intertwined transformer modules:
  • Temporal Transformer processes time sequences by ingesting 17 parallel token streams: 1 for text, 8 for “self” audio, and 8 for “user” audio.
  • Depth Transformer operates at each timestep to autoregressively predict all 16 audio tokens, introducing a one-frame delay for acoustic stability.
  • Alignment of text and audio streams uses Whisper time-stamps, and PAD tokens fill unaligned timesteps.

For the Japanese adaptation, the Mimi codec weights are frozen (as they proved effective for Japanese), while the RQ-Transformer undergoes full re-training. The English SentencePiece tokenizer (32k vocab) is substituted with a Japanese GPT-2 SentencePiece model (32k vocab). Token embeddings and the final text-linear layer are randomly re-initialized to accommodate language-specific features.

2. Training Protocols and Data Utilization

J-Moshi’s training follows a two-stage process:

a) Pre-training:

• Corpus: J-CHAT (Nakata et al., 2024), ≈69,000 hours of monophonic YouTube/Podcast Japanese speech.
• Preprocessing: Diarization produces pseudo-stereo (Moshi/user channels). ASR transcriptions are timestamped via WhisperX; PAD tokens inserted so that 88% of 3B text tokens are PAD.
• Training Setup: 128×V100 GPUs, DeepSpeed ZeRO-3, mixed-precision float16, batch size 512 (up to 2,048 frames each, ≈2.7 minutes).
• Optimization: AdamW (β₁=0.9, β₂=0.95, ε=1e-5, weight decay=0.1), peak LR 3×10⁻⁵, 500-step linear warmup, total 8,880 steps (≈36 h).
• Loss Weighting: PAD tokens down-weighted by 0.5; semantic tokens prioritized at a 100:1 ratio to acoustic tokens.

b) Fine-tuning:

• Dialogues (344 h total): Japanese-CallHome (16 h), CSJ dialogue subset (12 h), Travel-Agency (115 h), in-house Casual (148 h), in-house Consultation (53 h).
• Protocol: 94/3/3 train/val/test split; training over 3 epochs (1,423 steps, ≈2 h) on 16×V100, batch size 16. Distinct LRs for Temporal (2×10⁻⁶) and Depth (4×10⁻⁶) transformers.

This two-stage process yields the base “J-Moshi” model.

3. Synthetic Dialogue Augmentation

J-Moshi is further enhanced through large-scale synthetic stereo dialogue generation:

• Text Sources: JapanesePersonaChat, JapaneseEmpatheticDialogues, Japanese Daily Dialogue, RealPersonaChat (≈43,739 written dialogues, ~602 h if spoken).
• Colloquial Transformation: Gemma-2-27B-IT LLM rewrites scripted chats into spoken style.
• Multi-stream TTS: Pre-trained on J-CHAT, fine-tuned on 344 h of stereo data; semantic and acoustic delays of 25 and 27 frames, respectively.
• Sample Selection: For each rewritten dialogue, 10 TTS variants are generated (distinct seeds); WER via ASR used to select the lowest-error sample.
• Augmented Training: The synthetic 602 h are combined with the real 344 h of dialogue data for a final fine-tuning step (2,401 steps on 16 GPUs), producing the “J-Moshi-ext” variant.

This synthetic data approach increases diversity and volume in training data to mitigate data scarcity in high-quality Japanese stereo recordings.

4. Loss Formulations and Objective Design

The RQ-Transformer’s training objective is a cross-entropy loss over frame-timesteps and token layers. Let $s$ index timesteps (total $S$), $l$ token layers (1 to 17), and $t_{s,l}$ the ground-truth token. The per-step loss is

$L_{s,l} = -\log p(t_{s,l}|\text{history})$

Key adjustments:

• PAD Weighting: $L_{s,l}$ is multiplied by $\gamma=0.5$ if $t_{s,l}$ is a PAD token, 1 otherwise.
• Semantic vs. Acoustic Loss Balancing: Let $\mathcal{L}_{sem}$ sum over text-plus-layer-1 (“semantic”) tokens, and $\mathcal{L}_{ac}$ over layers 2–8 (“acoustic”).

  $S$0

• Weight Collation:

  $S$1

  with

  $S$2

These adjustments emphasize learning semantics and downweight large fractions of the training data that are padded.

5. Evaluation Strategy

J-Moshi is evaluated on the prompted dialogue continuation task. Test splits from the 344 h corpus are divided into 30-second segments; the first 10 seconds are used as a prompt, and the model must generate the subsequent 20 seconds. Automatic and human evaluation metrics are both employed:

• Baselines: dGSLM (full-duplex transformer, trained from scratch), Re-synthesis (Mimi codec resynthesizing true target audio), and Ground-truth audio.
• Automatic Metric: Perplexity (PPL) from the external Japanese LM (LLM-jp-3-3.7b) on ASR transcripts; temperatures $S$3.
• Human Judgment: 50 samples per model, assessed by 125 crowdworkers (each rates 10 samples) on 5-point Likert scales for:
  • Naturalness: “How natural does it sound?”
  • Meaningfulness: “How understandable is it?”

J-Moshi and J-Moshi-ext are evaluated at $S$4.

Turn-taking behavior is quantified via averages of IPU (inter-pausal unit), pause, gap, and overlap durations (per minute), directly reflecting the capacity for full-duplex, overlap-heavy Japanese conversational rhythm.

6. Performance Outcomes and Comparative Analysis

Automatic and Human Evaluation Metrics

At $S$5:

• Perplexity: J-Moshi’s PPL is approximately 100 points lower than dGSLM, though both remain behind the resynthesis and ground-truth baselines, suggesting that RQ-Transformer is the primary performance bottleneck.
• Human Ratings (Mean ±95% CI):
  • dGSLM: Naturalness $S$6, Meaningfulness $S$7
  • J-Moshi: Naturalness $S$8, Meaningfulness $S$9
  • J-Moshi-ext: Naturalness $l$0, Meaningfulness $l$1
  • Resynthesis: $l$2
  • Ground-truth: $l$3

J-Moshi significantly outperforms dGSLM in both naturalness and meaningfulness. Synthetic data augmentation boosts meaningfulness further (J-Moshi-ext).

Turn-taking Measurements (per-minute, $l$4):

Model	IPU (s)	Pause (s)	Gap (s)	Overlap (s)
J-Moshi	53.2	6.3	4.5	5.0
J-Moshi-ext	50.9	7.0	4.6	4.2
Japanese GT	59.7	3.5	4.0	8.1
English Moshi†	35.1	13.2	12.5	1.2
English GT†	51.1	6.4	4.2	3.3

†English model results from Défossez et al., 2024.

J-Moshi’s output demonstrates turn-taking dynamics closer to Japanese conversational ground-truth than to English Moshi, most notably higher overlaps and shorter gaps. This suggests acquisition of language-specific communicative timing via model adaptation.

7. Significance, Emerging Directions, and Applications

J-Moshi demonstrates that full-duplex neural dialogue architectures such as Moshi can be ported to typologically distinct languages using targeted modifications—primarily tokenizer replacement, RQ-Transformer retraining (pre-training plus fine-tuning), and robust synthetic data augmentation. It empirically outperforms a from-scratch high-capacity baseline in both naturalness and meaningfulness, while capturing Japanese-specific conversational phenomena.

The remaining performance gap relative to ground-truth points to future directions: adapting the Mimi codec for Japanese, recalibrating loss weightings (e.g., PAD/semantic ratios), and expanding Japanese stereo dialogue corpora. Potential applications include real-time bilingual or language-specific voice assistants, research tools for cross-lingual dialogue modeling, and scalable deployment of multi-language full-duplex systems (Ohashi et al., 3 Jun 2025).