Papers
Topics
Authors
Recent
Search
2000 character limit reached

Fish Audio Drama 3 Technical Report

Published 13 Oct 2026 in cs.SD and cs.CL | (2610.18010v2)

Abstract: We introduce Fish Audio Drama 3, an open-source speech generation model that turns text into directed performances, built on a 35B-parameter mixture-of-experts backbone that activates 3B parameters per token. It clones voices from multiple references, follows natural-language direction, revises a take when the next turn gives a note, voices multi-character scenes, edits existing speech by infilling, and synthesizes long-form and streaming audio. Multi-turn revision, scene dubbing, editing by infilling, and streaming text input are new in Drama 3, all in one model. Drama 3 pairs the hybrid linear-attention backbone with a new codebook multi-token-prediction audio decoder that predicts the acoustic codebooks of each frame in dedicated layers that attend over the entire sequence. The model is trained with a three-stage pre-training recipe, instruction fine-tuning, and large-batch reinforcement learning whose only reward is a single learned model of human preference. Its inference engine decodes a single stream at 271 frames per second on one H200, against 177 for Fish Audio S2, although Drama 3 has more than seven times as many parameters. The model weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/drama-3), and inference is available through SGLang-Omni (https://github.com/sgl-project/sglang-omni). We highly encourage readers to visit https://fish.audio to direct their own performances.

Citations (2)

Summary

  • The paper presents the Fish Audio Drama 3 model, which integrates multiple speech generation capabilities into a single framework using a discrete speech language model.
  • The model uses a unique Codebook-MTP decoder that ensures better reference-timbre preservation and efficient speech synthesis across various settings, with notable improvements in voice cloning, long-form speech synthesis, and inference efficiency, with increases of 36.2%–74% heightened optimization over the previous S2 model.
  • The architecture overcomes computational constraints in full-attention neural networks and employs a hybrid attention mechanism with Gated DeltaNet layers, enabling improved efficiency in handling long-directed spoken content.

Fish Audio Drama 3 presents an open-weight speech LLM designed for directed performance rather than conventional text-to-speech alone. The model supports zero-shot voice cloning, natural-language direction, multi-turn revision, multi-character dialogue, speech infilling, long-form synthesis, and streaming text input within a single token-based framework. Its central design decision is to represent these tasks as conversations over interleaved text and discrete audio tokens, allowing references, instructions, previous takes, and target speech to remain in one autoregressive context (2610.18010).

Problem formulation and system scope

The paper identifies four limitations in the preceding Fish Audio S2 system. S2 was not designed for long directed sessions because its full-attention backbone scales poorly with accumulated reference audio and previous takes. Its direction mechanism was effectively single-shot: a new instruction could influence a generation, but the model could not explicitly revise an existing take in response to a subsequent note. Speech editing was also all-or-nothing, requiring regeneration of an entire utterance when only one word was incorrect. Finally, S2 used transcription, quality, and speaker-similarity rewards that conflict with expressive delivery: laughter, whispering, prolonged pauses, and other paralinguistic events can lower ASR fidelity or speaker similarity while satisfying the intended direction.

Drama 3 reformulates the task around a sequence of conversational interactions. The system turn contains task instructions and voice references; the user turn supplies text, speaker identities, controls, or revision notes; and the assistant turn contains generated audio. Loss is applied only to assistant turns. This format permits reference-based cloning, directing, scene dubbing, revision, editing, and streaming synthesis to share one model and one inference interface.

The intended functionality is summarized below.

Capability Conditioning mechanism
Voice cloning One or more indexed reference blocks
Directed delivery Point tags, scoped tags, style descriptions, and director’s notes
Revision A later turn containing a relative note such as “softer and warmer versus previous”
Multi-speaker synthesis Speaker tokens and line separators
Scene dubbing Character descriptions and a complete line sequence
Speech editing Prefix–suffix infilling with a target transcript
Streaming Incremental text turns with look-ahead
Long-form synthesis Whole-recording or line-separated generation

The model therefore treats speech generation as conditional sequence modeling over both linguistic and acoustic context, rather than as isolated sentence synthesis.

Figure 1

Figure 1: Drama 3 combines voice casting, natural-language direction, revision, scene synthesis, infilling, long-form generation, and streaming in a unified interaction format.

Architecture

Drama 3 retains the discrete audio tokenizer used by S2. The tokenizer encodes 44.1 kHz audio into approximately 21.5 frames per second, with ten residual-vector-quantization codebooks per frame. Codebook 0 carries semantic information, while codebooks 1 through 9 progressively encode acoustic detail. The tokenizer is causal, enabling both encoding and decoding to operate incrementally.

The backbone is initialized from Qwen3.5-35B-A3B and consists of a 40-layer hybrid architecture combining Gated DeltaNet linear-attention blocks with periodic gated full-attention layers. The linear-attention components maintain a constant-size recurrent state, while one layer in every four retains full KV attention for exact retrieval. This division is important for long-form speech: recurrent state limits the cost of extending context, whereas full attention preserves access to specific reference-audio details, prior lines, or distant script content.

The backbone is a sparse MoE network with 256 experts. Each token is routed to eight experts plus a shared expert, activating approximately 3B of the 35B parameters per token. The architecture therefore separates total capacity from per-token computation, although inference still requires managing the memory footprint and routing overhead of the complete parameter set.

Drama 3 feeds only codebooks 0 and 1 back into the backbone. These semantic and coarse-acoustic codes are fused into one input embedding. Codebooks 2 through 9 bypass the backbone and are injected into the audio decoder. The partial fusion strategy reduces exposure to errors in the model’s own sampled acoustic codes while retaining enough acoustic information to model timbre and coarse prosody.

Figure 2

Figure 2: The backbone consumes text and fused semantic/coarse-acoustic frame embeddings, while the codebook-MTP decoder predicts the remaining acoustic codebooks with access to full causal context.

Codebook-MTP decoding

The main architectural contribution is a codebook multi-token-prediction decoder that replaces S2’s per-frame Fast AR module. At each temporal position, the backbone predicts the semantic code of the next audio frame. A 13-layer decoder then predicts the nine remaining acoustic codes: four shared layers first construct a representation, followed by nine codebook-specific layers that sequentially emit codes 1 through 9.

Unlike a conventional per-frame acoustic decoder, each codebook-specific layer attends causally across the entire sequence. Consequently, fine acoustic predictions can directly use the residual codes of reference audio and all preceding frames rather than relying on a single projected backbone state. The decoder also passes its hidden state from one codebook-specific layer to the next, so each prediction is conditioned on the code selected immediately before it.

Training is parallel over time under teacher forcing. At inference, however, each frame requires one backbone step followed by 13 decoder steps, with sampling between successive codebook layers. The design trades additional decoder depth for a shorter weight-read path than S2’s Fast AR implementation.

The paper reports three primary differences from S2:

  • Context: the decoder directly observes fine acoustic detail from references and prior frames.
  • Training layout: all codebooks are trained over the backbone’s temporal sequence in one additional causal pass.
  • Capacity allocation: every acoustic codebook receives its own full-width layer, embedding, output head, and KV cache.

These properties are intended to improve reference-timbre preservation and temporal acoustic consistency. The report does not provide a controlled ablation isolating each architectural component, so the contribution of full-sequence codebook attention relative to the hybrid backbone, training recipe, and preference RL remains unresolved.

Figure 3

Figure 3: S2 performs 40 sequential Fast AR layer passes after each backbone step, whereas Drama 3 performs 13 decoder-layer passes once per frame.

Training procedure

Drama 3 uses three pre-training stages, instruction SFT, and preference-based RL. The training pipeline preserves the base LLM’s text capability while progressively introducing speech, long-context, streaming, and editing formats.

Figure 4

Figure 4: The training recipe progresses from Qwen3.5 initialization through three pre-training stages, instruction SFT, and preference RL.

The first two pre-training stages use sequences up to 8K tokens and combine text documents with multi-turn TTS conversations. Stage 2 doubles the batch size while delaying learning-rate decay until Stage 3. Stage 3 increases the maximum sequence length to 16K tokens and introduces whole-recording synthesis, line-separated generation, streaming interleave, and speech infilling. Samples are packed into 128K-token rows with document masking, and optimization uses Muon.

The loss contains text-token prediction, semantic-code prediction, and acoustic-code prediction. The text term maintains language modeling ability; the semantic term trains the backbone’s speech planning; and the codebook term trains the decoder while also backpropagating into the backbone. The semantic and codebook losses are each weighted by 0.5 relative to the text loss. An auxiliary load-balancing loss prevents pathological MoE routing.

SFT uses sequences up to 32K tokens and an indexed reference layout. The model is trained on point tags such as [laughs] and [pause], scoped controls such as <[whisper](https://www.emergentmind.com/topics/whisper)>...</whisper>, line-level style descriptions, free-form director’s notes, speaker switches, multi-character scenes, infilling, and multi-turn revision. The SFT configuration is selected using held-out semantic-token NLL rather than training loss, which provides a direct measure of speech modeling quality but does not itself evaluate controllability or perceptual preference.

Preference RL without rule-based rewards

Drama 3 makes a deliberately strong claim: expressive speech should not be optimized with the ASR and speaker-similarity rewards commonly used in TTS RL. The authors replace these objectives with a single learned reward model trained on human pairwise preferences.

The reward model is a smaller speech LLM that reads the complete prompt, references, script, and generated audio. It produces a scalar at the take’s closing token. Unlike an ASR-based reward, it can evaluate delivery in the context of the requested direction; unlike a speaker-similarity model, it can judge whether departures from neutral timbre are appropriate.

The reward is calibrated against a frozen anchor, clipped to three standard deviations, and converted into a within-prompt expected win rate. GRPO uses eight sampled takes per prompt, mean-centered group advantages, no KL penalty, and no reference model. Groups with reward standard deviation below 0.05 are discarded. Each rollout batch contains 1,536 prompts, with one optimizer step per batch. The codebook-MTP decoder remains frozen during RL, so policy optimization updates the backbone’s semantic generation but not the acoustic decoder.

On EXPRESSO-style data, rule-based metrics exhibit the expected conflict with expressive delivery. Relative to default speech, whispering lowers speaker similarity by 0.310 and laughing lowers it by 0.193. A similarity reward consequently prefers the default reading in 99.7% of whisper pairs and 93.9% of laughing pairs. Laughter also increases WER by 0.79 percentage points. By contrast, when the requested style is explicitly included in the prompt, the learned reward model prefers the styled reading in 62.2% of whisper pairs and 64.4% of laughing pairs.

The implication is not that the learned reward is universally reliable, but that conditioning the reward model on the instruction changes the evaluation target from acoustic proximity to contextual preference. The study also reports that RL reduces signal-check failures without an explicit transcription reward: flagged first takes decrease from 2.82% to 1.24% on Chinese Seed-TTS-Eval and from 3.40% to 1.84% on English. First-take error rates fall from 1.66 to 1.00 CER and from 3.49 to 2.20 WER. After regeneration of flagged outputs, final error rates are 0.83 versus 0.84 CER in Chinese and 1.54 versus 1.60 WER in English for Drama 3 and the SFT checkpoint, respectively.

These results support the paper’s claim that rule-based rewards are not necessary for maintaining intelligibility in this setting. However, they do not establish that a single learned reward model is sufficient across all languages, speakers, editing conditions, or pathological failure modes. The reward model and preference dataset are not characterized in enough detail to assess annotation scale, inter-rater agreement, demographic coverage, or susceptibility to reward-model bias.

Inference efficiency and serving design

Drama 3’s total parameter count is substantially larger than S2’s, but its serving path is shorter and more bandwidth-efficient. S2’s four-layer Fast AR executes ten passes per frame, producing 40 sequential layer passes and repeatedly rereading its weights. Drama 3 executes its 13 decoder layers once per frame. In addition, the routed experts are stored in block-scaled FP8.

At batch size one, the reported weight traffic per frame is approximately 6.0 GB for Drama 3 versus 15.6 GB for S2. On an H200, Drama 3 reaches 271 frames per second for a single stream, compared with 177 frames per second for S2. On a B300, it reaches 295 frames per second. At concurrency 32 on H200, FP8 experts increase per-stream decode rate by 36% relative to BF16 experts.

Metric Drama 3 Fish Audio S2
Total backbone parameters 35B Approximately 4.6B
Active parameters per token Approximately 3B —
Sequential decoder passes after backbone 13 40
Weight traffic per frame Approximately 6.0 GB Approximately 15.6 GB
H200 single-stream decode rate 271 fps 177 fps
B300 single-stream decode rate 295 fps —

The paper attributes the serving gains not only to the architecture but also to extensive systems work. CUDA graphs and programmatic dependent launch reduce host-side launch overhead but cannot eliminate the GPU cost of starting and draining thousands of small kernels. In one measurement, a frame took 4.0 ms and launched 1,171 kernels; approximately 830 auxiliary kernels moved little data but consumed about 2.7 ms. The authors therefore remove or fuse framework-generated operations such as mask writes, tensor copies, layout transformations, and type conversions.

A Gated DeltaNet decode fusion removes approximately 100 kernels per step and raises single-stream rate by 4.2% on B300. Prefill launch merging improves rate by 4.4% at concurrency 128. Capturing first-frame sampling and related prefill work into graphs reduces host time per round from 30.9 to 14.7 ms and improves rate by 15.3% at concurrency 32 in the reported development configuration.

The shape-driven kernel workflow is particularly notable because it evaluates kernels inside the complete serving engine rather than relying on isolated operator benchmarks. A candidate Gated DeltaNet prefill kernel was 3.4 times faster in isolation but initially lost in serving because of graph capture, state movement, and runtime compilation. After those issues were resolved, the optimized kernel improved throughput by 11.1% at concurrency 256. This demonstrates that operator-level speedups are not reliable proxies for end-to-end serving gains.

Figure 5

Figure 5: Kernel optimization is driven by production serving shapes and validated at the operator, engine, and end-to-end serving levels.

The system also disaggregates prefill and decode across GPUs. On a single GPU, a long prefill for a cold request pauses all active streams. With prefill–decode disaggregation, the prefill GPU writes KV entries and recurrent states into the decode GPU, allowing active streams to continue without waiting.

On one RTX PRO 6000 node, one prefill GPU and one decode GPU support 24 streams per GPU under a first-percentile rate requirement of at least 35 frames per second, compared with approximately 10 streams per GPU on a single-GPU configuration. In a production canary, disaggregated decode GPUs carry roughly twice as many streams as single-GPU workers. Their first-percentile per-stream rate is 55 fps versus 45–47 fps, and no disaggregated stream falls below the 42 fps service floor. CER differs by only 0.02 points, suggesting that the serving transformation improves continuity without materially changing generated content.

Figure 6

Figure 6: Prefill–decode disaggregation prevents cold-request prefills from interrupting streams already being played.

Empirical results

Voice cloning

On a three-reference extension of LibriSpeech-PC, Drama 3 achieves 1.85% WER and 4.31 UTMOS with either one or three references. Adding two additional references increases speaker similarity from 0.605 to 0.633 without changing WER or UTMOS. The multi-reference improvement supports the architecture’s indexed reference-block formulation, although the comparison is not against competing systems under the same three-reference condition.

On the 24-language MiniMax Multilingual Testset, Drama 3 has the lowest error rate among all listed systems in Chinese, Dutch, English, German, and Turkish. It outperforms OmniVoice in 15 languages and FireRedTTS3-Base in 17. It is lower than OmniVoice, FireRedTTS3-Base, and Fish Audio S2 in eleven languages, including Arabic, Chinese, English, French, German, Portuguese, Turkish, and Vietnamese.

The results are not uniformly dominant. Drama 3 is substantially weaker in Cantonese, with 22.70% CER compared with OmniVoice’s 17.71%, and underperforms several systems in Greek, Hindi, Indonesian, Japanese, Romanian, and Ukrainian. Thus the multilingual claim is best interpreted as broad competitiveness with strong performance in several major languages, not language-independent superiority.

Speech editing

Drama 3 performs infilling by decoding the entire sequence, including unedited context, rather than generating only a replacement segment and pasting it into the original waveform. This enables the decoder to reconstruct the full output while conditioning on the source prefix, source suffix, and target transcript.

Against Cartesia Sonic-3’s infill API, Drama 3 obtains lower average WER in both English and Chinese on both the basic and full Ming-Freeform splits. The Chinese gap is particularly large: 3.69% versus 8.64% on the basic split and 2.35% versus 6.53% on the full split. On the basic split under the CosyEdit protocol, Drama 3 achieves the highest Chinese speaker similarity for insertion and substitution, 0.909 and 0.898, respectively.

The comparison requires caution. Cartesia returns only the generated edit window, whereas Drama 3 decodes the complete sequence; the evaluation pastes Cartesia’s output into the source recording with loudness matching and crossfades. The systems therefore differ in reconstruction mechanism and potentially in how much unedited audio is regenerated.

Multi-turn revision

On Step-Audio-Edit, every revision round improves Drama 3’s measured emotion and style accuracy. Emotion accuracy increases from 55.5% at iteration 0 to 58.2% after three revisions; style accuracy increases from 76.5% to 81.8%. Paralinguistic-event scores rise from 1.32 to 2.32 after adding a tagged event through infilling.

The trajectory is consistent with the model’s conversational revision formulation: prior takes remain in context, and the next instruction refers to the immediately preceding performance. Nevertheless, Step-Audio-EditX achieves higher emotion accuracy after revision, reaching 69.3% after three iterations, while Drama 3 achieves higher style accuracy at iteration 0 and iteration 3. The comparison relies on Gemini-2.5-Pro as an external judge for both systems, and the paper does not provide human validation of judge agreement for these particular outputs.

Streaming quality

Streaming synthesis does not materially degrade the reported objective metrics. On 1,088 English Seed-TTS-Eval sentences, streamed outputs achieve 1.86% WER and 0.640 speaker similarity, compared with 1.83% WER and 0.640 similarity for non-streamed outputs. This result supports the causal tokenizer and incremental conversation format, although the evaluation measures final audio quality rather than latency, chunk-boundary artifacts, or listener-perceived interruption quality.

Limitations and open questions

The paper’s results depend on several assumptions and evaluation choices. First, the model is initialized from Qwen3.5-35B-A3B, but the report does not isolate the contribution of the backbone initialization, hybrid attention, MoE scaling, codebook-MTP decoder, SFT data mixture, and RL. Consequently, the causal explanation for the observed gains is incomplete.

Second, the preference-RL conclusions depend on one learned reward model whose training data and annotation protocol are only briefly described. A single reward may reduce conflicts among ASR, similarity, and expressiveness objectives, but it can also encode systematic annotator or model biases. The no-KL, few-step training strategy is reported as stable in development, yet no broad sensitivity analysis is provided for batch size, group size, clipping, reward calibration, or the frozen acoustic decoder.

Third, most quality comparisons use automatic metrics or model-based judges. Speaker similarity, WER, CER, UTMOS, DNSMOS, and Gemini-based emotion/style judgments capture different aspects of speech and can disagree with perceptual quality. The paper explicitly argues that proxy metrics can penalize valid expressiveness, but its principal benchmark results still rely heavily on such proxies.

Fourth, the strongest serving results are hardware- and workload-specific. The throughput claims depend on H200, B300, and RTX PRO 6000 systems, FP8 expert storage, fixed serving shapes, graph capture policies, and a particular concurrency distribution. The paper does not report energy consumption, cost per generated hour, performance under substantially longer contexts, or behavior under workloads with highly heterogeneous references and sequence lengths.

Finally, the paper leaves open how reliably the system preserves speaker identity and acoustic continuity during long multi-turn revision sessions, multi-character scenes, and repeated infilling. The reported revision metrics show improvement in categorical style and emotion judgments, but do not quantify cumulative drift across many turns or the perceptual integrity of unedited spans over repeated edits.

Conclusion

Fish Audio Drama 3 unifies directed synthesis, revision, scene dubbing, infilling, long-form generation, and streaming within a discrete speech LLM. Its principal technical contribution is the codebook-MTP decoder, which gives acoustic codebooks full causal access to temporal and reference-audio context while reducing the sequential weight-read cost of S2’s Fast AR decoder. The training strategy extends the model from generic TTS to conversational control, and preference RL replaces rule-based rewards that penalize expressive delivery.

The empirical results show strong multilingual cloning, competitive speech editing, measurable improvement across revision rounds, unchanged streaming quality, and substantially higher reported decode rates despite a more than sevenfold increase in total parameter count over S2. The paper’s main unresolved questions concern attribution among its many architectural and training changes, the robustness of a single learned preference reward, and the generality of its serving and perceptual results beyond the evaluated hardware, languages, and benchmarks.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Explain it Like I'm 14

1. ¿De qué trata el artículo?

El artículo presenta Fish Audio Drama 3, un sistema de inteligencia artificial que convierte texto en conversaciones y actuaciones de voz.

No está diseñado solo para leer palabras en voz alta. También intenta actuar como una persona o un personaje. Por ejemplo, puede:

  • Hablar con diferentes emociones, como alegría, tristeza o enfado.
  • Copiar la voz de una persona usando pequeños audios de referencia.
  • Interpretar conversaciones entre varios personajes.
  • Cambiar una grabación siguiendo instrucciones de un “director”.
  • Corregir solo una palabra o frase sin volver a crear toda la grabación.
  • Crear audiolibros, doblajes y conversaciones largas.
  • Empezar a hablar mientras el texto todavía está llegando, como en una conversación en tiempo real.

El modelo es de código abierto, lo que significa que sus creadores publicaron sus componentes principales para que otras personas puedan estudiarlo y utilizarlo.

2. ¿Qué preguntas intenta responder?

El equipo de investigación quería mejorar varios problemas de los sistemas anteriores de texto a voz. Sus preguntas principales eran:

  1. ¿Puede una IA actuar y no solo leer? Querían que siguiera instrucciones como “habla más suavemente” o “di esta frase con sorpresa”.
  2. ¿Puede cambiar una actuación después de recibir comentarios? Por ejemplo, después de producir una frase, el director podría decir: “Hazla más cálida y un poco más lenta”.
  3. ¿Puede representar una escena completa? El sistema debería utilizar varias voces y mantener claro qué personaje está hablando.
  4. ¿Puede editar una grabación sin destruir el resto? Si una palabra está mal, la IA debería reemplazar solo esa palabra, como editar una parte de una película sin volver a grabarla entera.
  5. ¿Puede trabajar con textos y grabaciones largas? Esto es importante para audiolibros, películas y videojuegos.
  6. ¿Puede producir el audio suficientemente rápido para una conversación en directo?

3. ¿Cómo hicieron la investigación?

Un modelo grande, pero con solo una parte activa

Drama 3 utiliza una arquitectura llamada mezcla de expertos (Mixture of Experts, o MoE). Podemos imaginarla como una escuela con 256 profesores especializados. Para cada palabra o fragmento de audio, el sistema elige solo unos pocos profesores adecuados, en vez de usar a todos.

El modelo completo tiene aproximadamente 35.000 millones de parámetros. Los parámetros son valores internos que la IA aprende durante el entrenamiento. Sin embargo, para cada fragmento solo activa unos 3.000 millones. Esto ayuda a que el sistema sea más rápido y necesite menos memoria durante cada paso.

El audio se divide en pequeñas piezas

La IA no trabaja directamente con la onda sonora completa. Primero convierte el audio en pequeños códigos digitales, parecidos a piezas de LEGO. Cada instante de voz se representa con diez grupos de códigos:

  • Uno contiene información más relacionada con el significado y el ritmo.
  • Los otros nueve describen detalles del sonido, como el tono, la pronunciación y el timbre de la voz.

Después, el sistema predice esos códigos y los vuelve a convertir en sonido.

Un nuevo decodificador de audio

Drama 3 utiliza un componente llamado codebook-MTP decoder. El nombre es técnico, pero la idea es sencilla: el sistema crea primero una representación general de cada parte del audio y luego añade detalles cada vez más precisos.

Es parecido a dibujar una imagen:

  1. Primero se hace un boceto.
  2. Después se añaden las formas principales.
  3. Finalmente se incorporan los colores y los detalles pequeños.

Este nuevo método permite que la IA tenga en cuenta no solo el texto, sino también todos los detalles de las voces de referencia y de los sonidos anteriores.

Entrenamiento en varias etapas

El modelo se entrenó siguiendo varias fases:

  1. Preentrenamiento: aprendió con grandes cantidades de texto y grabaciones de voz en unos 80 idiomas y dialectos.
  2. Entrenamiento con instrucciones: aprendió a responder a indicaciones como “susurra”, “ríe”, “haz una pausa” o “habla con más energía”.
  3. Aprendizaje con preferencias humanas: varias versiones de una misma actuación fueron comparadas por personas. La IA aprendió qué versiones preferían los oyentes.

Este último proceso se parece a entrenar a un estudiante con comentarios. En lugar de decirle solo si una respuesta está “bien” o “mal”, las personas indican cuál de dos actuaciones suena mejor.

Los investigadores usaron un único modelo aprendido de las preferencias humanas para valorar las actuaciones. No se basaron únicamente en comprobar si cada palabra coincidía exactamente con el texto, porque una buena actuación puede incluir risas, susurros, pausas o sonidos que no aparecen escritos.

Pruebas de velocidad

También crearon un sistema especializado para ejecutar el modelo en tarjetas gráficas. Una tarjeta gráfica es un procesador especialmente bueno para hacer muchos cálculos a la vez.

En estas pruebas midieron cuántos fragmentos de audio podía generar el sistema por segundo y cuántas conversaciones podía atender al mismo tiempo.

4. ¿Cuáles fueron los principales resultados?

Mejor control sobre la actuación

Según el artículo, Drama 3 puede seguir diferentes tipos de instrucciones:

  • Instrucciones para una palabra o frase concreta.
  • Instrucciones para toda una línea.
  • Indicaciones sobre emoción, velocidad, intención o énfasis.
  • Notas sobre una actuación anterior.

Por ejemplo, una persona puede pedir que una frase sea “más suave y cercana que la anterior”. El sistema conserva la actuación anterior en el contexto y produce una nueva versión.

En las pruebas de actuación dirigida, cada ronda de correcciones mejoró la precisión con la que seguía el estilo y la emoción solicitados.

Clonación de voz en varios idiomas

El modelo puede producir una voz parecida a la de una grabación de referencia. En una prueba multilingüe, obtuvo la tasa de errores más baja entre los sistemas comparados en chino, neerlandés, inglés, alemán y turco.

Esto significa que, en esas pruebas, cometió menos errores al producir el contenido hablado. Sin embargo, una tasa de error más baja no garantiza que todas las actuaciones suenen perfectas o que sean indistinguibles de una persona real.

Edición de audio

Drama 3 puede reemplazar una palabra o una frase dentro de una grabación manteniendo el resto del audio. Esto se llama infilling, o “rellenado”.

Es parecido a corregir una palabra en una película sin volver a grabar toda la escena. Las pruebas indicaron que tuvo menos errores que un servicio comparado en inglés y chino. En algunas pruebas en chino también consiguió una similitud de voz mayor que otros modelos especializados.

Escenas con varios personajes

El sistema puede utilizar varias grabaciones de referencia y representar conversaciones entre diferentes personajes. También puede cambiar de idioma para un mismo personaje conservando una voz parecida.

Esto podría ser útil para:

  • Doblaje de películas.
  • Videojuegos.
  • Audiolibros.
  • Historias de audio.
  • Personajes interactivos.

Generación rápida

En una tarjeta gráfica NVIDIA H200, Drama 3 generó aproximadamente 271 fragmentos de audio por segundo, frente a los 177 de un modelo anterior llamado Fish Audio S2.

Aunque Drama 3 tiene más de siete veces más parámetros que S2, fue más rápido en esa medición. Esto se debe a que su arquitectura y su programa de ejecución fueron diseñados para evitar cálculos y movimientos de memoria innecesarios.

La velocidad es importante porque el audio debe generarse más rápido de lo que se reproduce. Si el sistema produce sonido demasiado lentamente, la conversación tendría pausas incómodas.

Modelo abierto

Los autores publicaron los pesos del modelo y herramientas para ejecutarlo en plataformas como GitHub y Hugging Face. Esto permite que investigadores y desarrolladores puedan estudiar el sistema y crear nuevas aplicaciones.

5. ¿Por qué son importantes estos resultados?

La mayoría de los sistemas de texto a voz se concentran en pronunciar correctamente las palabras. Drama 3 intenta resolver un problema más difícil: hacer que la voz parezca una actuación.

Esto es importante porque una actuación real no depende solo de las palabras. También incluye:

  • El tono.
  • La emoción.
  • La velocidad.
  • Las pausas.
  • Las risas.
  • Los susurros.
  • La relación entre varios personajes.
  • Las correcciones que da un director.

El artículo también muestra que evaluar una actuación no siempre consiste en comprobar si coincide exactamente con el texto. Una risa o un susurro podrían hacer que un programa de reconocimiento de voz cometa un error, aunque la actuación sea mejor y más natural.

6. Posibles efectos y limitaciones

Drama 3 podría hacer más fácil y barato crear audiolibros, doblajes, videojuegos y asistentes de voz. Una persona podría escribir una escena, indicar cómo quiere que suene y corregirla con instrucciones sencillas, sin contratar inmediatamente a muchos actores o volver a grabar todo.

También podría ayudar a producir audio en varios idiomas y crear personajes capaces de responder rápidamente.

Sin embargo, el sistema tiene riesgos y limitaciones. La clonación de voces podría utilizarse sin permiso para imitar a personas reales, crear engaños o fabricar grabaciones falsas. Por eso sería importante obtener autorización, identificar el audio generado por IA y proteger la identidad de los hablantes.

Además, las pruebas descritas en el artículo no significan que el modelo sea perfecto. Puede equivocarse, interpretar mal una instrucción o producir una voz que no suene completamente natural. Los resultados dependen de los idiomas, las grabaciones de referencia y las situaciones evaluadas.

Conclusión

Fish Audio Drama 3 intenta transformar la IA de voz en una especie de actor digital. No solo lee un guion: puede recibir dirección, cambiar su actuación, representar a varios personajes, editar partes concretas y producir audio rápidamente.

La investigación es importante porque acerca la tecnología de voz a usos más creativos y naturales. Al mismo tiempo, su desarrollo hace necesario hablar de permisos, privacidad y seguridad para evitar que las voces clonadas se utilicen para engañar a otras personas.

Practical Applications

Immediate Applications

  • Audiobook and audio-drama production — media and publishing. Producers can use Drama 3 to generate directed performances from scripts, clone approved character voices from reference recordings, and revise takes through natural-language notes such as “slower,” “warmer,” or “more suspenseful.” This can reduce the number of manual recording cycles for narration, podcasts, and audio drama.
    • Potential workflow: script segmentation → voice-reference upload → first generated take → director feedback → regenerated take → human editorial review.
    • Dependencies: valid voice permissions, high-quality reference audio, human quality control, and sufficient GPU capacity. The reported throughput is measured on specialized NVIDIA hardware and should not be assumed for ordinary consumer devices.
  • Localized dubbing of films, games, and online video — entertainment and localization. The multi-speaker scene capability can generate dialogue for several characters in one pass, including characters speaking different languages while retaining their assigned voices. It can support first-pass dubbing, localization previews, animatics, and production drafts.
    • Potential products: a script-to-dubbing pipeline with speaker assignment, multilingual line generation, timing controls, and export to standard audio-editing tools.
    • Dependencies: translation quality, cultural adaptation, lip-sync and timing alignment, actor or rights-holder consent, and professional review before release. The paper evaluates multilingual voice cloning but does not establish reliable lip synchronization.
  • Voice editing and patch recording — post-production and broadcasting. Editors can replace a word, phrase, or sentence inside an existing recording through speech infilling rather than rerecording the entire passage. This is useful for correcting names, factual errors, pronunciation, or compliance issues in podcasts, commercials, lectures, and voice-over work.
    • Potential workflow: select erroneous span → provide replacement text and optional direction → generate localized edit → compare acoustic continuity with surrounding speech → approve or reject.
    • Dependencies: the surrounding recording must provide adequate context; edits may still require manual crossfades, noise matching, and timing adjustment. The system should not be used to alter legally sensitive recordings without disclosure and authorization.
  • Interactive character voices — games, virtual worlds, and conversational applications. Streaming text input and directed performance can support non-player characters, virtual assistants, educational characters, and interactive fiction that speak as text becomes available rather than waiting for a complete response.
    • Potential tools: real-time character dialogue engines with persistent voice references, emotion or style tags, and per-turn performance direction.
    • Dependencies: end-to-end latency, moderation, predictable behavior, and compute availability. The reported 271 frames per second on one H200 exceeds the approximately 21.5 frames per second playback requirement, but this does not by itself guarantee low conversational latency under many simultaneous users.
  • Accessible content generation — education, public services, and assistive technology. Organizations can produce expressive spoken versions of articles, course materials, public announcements, and user-generated text. Directions such as whispering, pausing, or emphasizing a phrase may improve comprehension and engagement compared with neutral TTS.
    • Potential workflow: accessible-text conversion → reading-level and pronunciation review → expressive speech generation → human or automated accessibility checks.
    • Dependencies: language and dialect coverage, pronunciation dictionaries, accessibility testing, and safeguards against inappropriate or stigmatizing voice characterization.
  • Rapid prototyping for researchers and developers — academia and open-source software. The publicly released weights, Hugging Face distribution, GitHub code, and SGLang-Omni inference support make it possible to study expressive TTS, voice cloning, speech editing, multi-speaker generation, and preference-based alignment without building a model from scratch.
    • Research uses: benchmarking controllability, testing new reward models, evaluating speech infilling, studying long-context audio generation, and developing specialized adapters.
    • Dependencies: hardware capable of hosting a large sparse model, compatible inference software, licensing compliance, reproducible evaluation sets, and careful handling of personally identifiable voice data.
  • Preference-based quality evaluation — media tooling and AI engineering. The paper’s learned human-preference reward model provides a practical template for ranking alternative speech takes when conventional ASR or speaker-similarity metrics penalize expressive delivery. Teams can use a similar approach to select among generated takes for internal review.
    • Potential workflow: sample multiple takes → rank or score them with human-preference data → present the strongest candidates to an editor.
    • Dependencies: representative preference annotations, calibration against different languages and performance styles, and monitoring for reward-model bias or reward hacking. The paper’s reward model should not be treated as a universally reliable quality or safety judge.
  • Government and institutional communications — public information. Agencies and organizations can prepare multilingual announcement drafts, emergency-information prototypes, and accessible audio versions of written notices. Directed pauses and emphasis may make key instructions clearer.
    • Dependencies: authorized voices, verified source text, review by communication professionals, and explicit labeling when speech is synthetic. Emergency use requires conservative validation because generation errors could have material consequences.

Long-Term Applications

  • End-to-end professional dubbing and localization platforms — entertainment technology. A future system could combine Drama 3-style scene generation with translation, phoneme-level timing, lip synchronization, ambience preservation, actor approval, and automated quality assurance. This could enable scalable localization for large catalogs while retaining human control over final performances.
    • Research required: robust cross-lingual prosody transfer, duration control, lip-sync integration, dialogue mixing, and evaluation by professional dubbing actors.
    • Key assumptions: voice rights and performer compensation models are resolved, generated voices can meet contractual quality requirements, and localization remains culturally appropriate.
  • Human-in-the-loop virtual production — film, television, and games. Multi-turn revision could become a conversational directing interface in which a director gives notes, compares several takes, locks selected phrases, and requests localized edits without losing the broader scene context.
    • Potential product: a digital audio workstation plug-in or cloud production environment with versioned takes, scene memory, speaker libraries, and reversible edits.
    • Dependencies: stable long-context behavior, deterministic regeneration controls, integration with existing editing software, and provenance tracking for every generated segment.
  • Personalized tutoring and language learning — education. Future systems could generate dialogues with controllable pacing, emotion, pronunciation, and character roles; revise delivery based on learner feedback; and provide the same lesson in multiple languages or reading styles.
    • Potential applications: role-play exercises, pronunciation demonstrations, accessible reading tutors, and adaptive listening assessments.
    • Research required: pedagogical validation, age-appropriate interaction, reliable pronunciation feedback, and safeguards against reinforcing accent or cultural stereotypes. Generated speech should supplement rather than replace qualified instruction.
  • Assistive and rehabilitative speech systems — healthcare and accessibility. With additional validation, expressive voice cloning and localized infilling could help people with speech loss maintain a personalized synthetic voice, or allow communication devices to express urgency, emotion, or conversational nuance.
    • Potential workflow: consented voice capture before or during illness → personalized voice model → clinician- or user-controlled text generation → speech output.
    • Dependencies: clinical trials, privacy-preserving training, durable identity and consent controls, intelligibility under medical conditions, and strict limits on who may authorize speech. The paper provides no clinical evidence, so this remains a research direction rather than a medical product.
  • Robotics and embodied agents — robotics and human-computer interaction. Robots could use directed, context-aware speech to coordinate with people, distinguish speakers, and adapt delivery to operational conditions—for example, using concise and emphatic speech during an alert and calmer speech during routine interaction.
    • Research required: grounding speech generation in sensor and task state, guaranteed latency, interruption handling, safety policies, and integration with robot motion and dialogue planners.
    • Dependencies: the model must avoid generating misleading confidence or unauthorized instructions; expressive speech alone does not provide reliable task planning or situational awareness.
  • Low-latency, high-concurrency speech services — cloud infrastructure. The inference innovations—sparse MoE activation, FP8 expert weights, fused kernels, prefill/decode separation, and shape-driven scheduling—could support scalable APIs for streaming narration, call-center agents, games, and interactive media.
    • Potential products: hosted streaming TTS, batch audiobook generation, and multi-tenant dubbing services with GPU-aware admission control.
    • Research and engineering required: performance on varied GPUs, quantization-quality trade-offs, fault tolerance, autoscaling, cost analysis, and privacy-preserving deployment. The paper reports results primarily on H200 and B300 hardware, so general cloud economics remain unverified.
  • Synthetic media provenance and consent infrastructure — policy, law, and platform governance. Because the model can clone voices, revise speech, and generate convincing multi-character scenes, future deployments should include consent records, watermarking or provenance metadata, audit logs, and detection or disclosure mechanisms.
    • Potential policy tools: voice-licensing registries, revocation systems, mandatory synthetic-audio labels, and provenance-aware media platforms.
    • Dependencies: watermark robustness after editing or compression, interoperability among platforms, enforceable voice rights, and balanced regulation that protects performers without blocking legitimate accessibility and creative uses.
  • Personalized financial, legal, and customer-service communication — finance and enterprise software. A mature system could generate approved, personalized explanations of account activity, contracts, or support procedures in a consistent organizational voice, with controlled emphasis and multilingual delivery.
    • Research required: verifiable grounding, exact-number preservation, auditability, and strict separation between generated explanation and authorized decision-making.
    • Dependencies: the current model’s expressive generation must be paired with deterministic retrieval and validation. It should not independently produce legally binding advice, financial decisions, or compliance-sensitive statements.
  • Large-scale speech dataset creation and model training — academia and industry research. Directed generation could produce controlled speech variations for testing emotion recognition, dialogue systems, accessibility interfaces, and multilingual speech technologies.
    • Dependencies: synthetic data may reproduce model biases and should not be assumed equivalent to diverse human recordings. Datasets require documentation, consent analysis, quality filtering, and evaluation for distributional artifacts.
  • Advanced audio editing as a general-purpose authoring medium — creative software. Longer term, speech editors could expose text-based operations such as “make this sentence more reassuring,” “replace the name but preserve timing,” or “match the following character’s energy,” combining semantic editing with acoustic continuity.
    • Research required: precise span control, preservation of background sound and room acoustics, speaker identity consistency, temporal constraints, and transparent version comparison.
    • Dependencies: reliable separation between intended edits and unintended changes; human review remains necessary for professional, contractual, and reputationally sensitive content.

Glossary

  • Autoregressive (AR): A generation process in which each prediction depends on previously generated outputs. “per-frame fast autoregressive (AR) module”
  • Bfloat16 (BF16): A 16-bit floating-point format commonly used for deep-learning computation. “the trainer runs in bfloat16 (BF16)”
  • Bradley–Terry loss: A probabilistic loss for learning preferences from pairwise comparisons. “It is trained on human preference judgments over pairs of takes with the Bradley--Terry loss”
  • Causal attention: Attention restricted to the current position and preceding positions, preventing access to future tokens. “each layer attends causally over the whole sequence”
  • Codebook: A discrete collection of vector-quantized representations used to encode audio information. “The first codebook is semantic”
  • Codebook-MTP decoder: A decoder that predicts multiple audio codebooks through sequential, dedicated prediction layers. “Drama~3 pairs the hybrid linear-attention backbone with a new codebook multi-token-prediction audio decoder”
  • Conservative direct preference optimization (DPO): A preference-optimization method that uses smoothed preference labels to reduce overconfident updates. “as in conservative direct preference optimization (DPO)”
  • CUDA graphs: A CUDA mechanism that records and replays a sequence of GPU operations to reduce CPU launch overhead. “CUDA graphs record a whole step once and replay it”
  • Curriculum-free teacher forcing: Training in which the model receives ground-truth previous outputs rather than its own predictions. “every code is teacher-forced from the ground truth”
  • Delta rule: An update mechanism for maintaining a recurrent state, used here in the Gated DeltaNet architecture. “applies the gated delta rule”
  • Descript Audio Codec: An audio codec that converts waveforms into discrete code sequences and reconstructs waveforms from them. “Built on the Descript Audio Codec”
  • Document masking: A training technique that prevents attention or loss computation from crossing document boundaries in packed sequences. “samples are packed with document masking”
  • Exposure bias: The discrepancy between training, when ground-truth history is available, and inference, when the model conditions on its own generated outputs. “to limit exposure bias”
  • FP8: An 8-bit floating-point representation used to reduce memory use and accelerate neural-network computation. “the experts use the microscaling FP8 format”
  • Gated DeltaNet: A recurrent or linear-attention architecture that uses gating and delta-rule updates to maintain sequence information efficiently. “a hybrid that stacks blocks of three Gated DeltaNet layers”
  • Gated full attention: A full-attention mechanism augmented with learned gating operations. “one gated full-attention layer”
  • Gated RMSNorm: A gated form of root-mean-square normalization applied within a neural-network layer. “a gated RMSNorm per layer”
  • General matrix multiplication (GEMM): A highly optimized operation that multiplies matrices and optionally adds a scaled matrix. “the router's general matrix multiplication (GEMM), softmax, and top-8 selection”
  • Group Relative Policy Optimization (GRPO): A reinforcement-learning method that computes relative advantages among multiple sampled outputs for the same prompt. “We use Group Relative Policy Optimization (GRPO)”
  • Hidden state: An internal vector representation produced by a neural-network layer for a sequence position. “the backbone's final hidden state”
  • Infilling: Generating missing or replaced content within an existing sequence while preserving surrounding context. “edits speech by infilling”
  • Instruction fine-tuning: Supervised training that teaches a model to follow natural-language instructions. “instruction fine-tuning, and large-batch reinforcement learning”
  • Key–value (KV) cache: Stored attention keys and values reused during autoregressive generation to avoid recomputation. “only one layer in four keeps a key--value (KV) cache”
  • Kullback–Leibler (KL) penalty: A regularization term that discourages a policy from diverging too far from a reference distribution. “without a Kullback--Leibler (KL) penalty”
  • Linear attention: An attention formulation whose computation scales more favorably with sequence length than standard full attention. “a hybrid linear-attention LLM”
  • Load-balancing loss: An auxiliary objective that encourages tokens to be distributed evenly among mixture-of-experts routes. “an auxiliary load-balancing loss”
  • Mixture of experts (MoE): A neural-network architecture that routes each token to a subset of specialized feed-forward networks. “a sparse mixture-of-experts feed-forward network”
  • Multi-token prediction (MTP): A method in which a model predicts multiple future or related tokens through additional prediction modules. “with a Codebook-MTP Decoder”
  • Muon: An optimizer designed for training large neural-network models, used here during pre-training and reinforcement learning. “the model is optimized with Muon”
  • Negative log-likelihood (NLL): The negative logarithm of the probability assigned to the observed target, commonly used as a training objective. “by the teacher-forced negative log-likelihood (NLL)”
  • Proximal policy optimization (PPO): A policy-gradient reinforcement-learning algorithm that limits updates using a clipped probability ratio. “the proximal policy optimization (PPO) ratio”
  • Quantization: Representing numerical model parameters with lower-precision values to reduce memory use and computation cost. “so serving adds no quantization of its own”
  • Reinforcement learning from human preference: Training a model to optimize a reward derived from human judgments of generated outputs. “Reinforcement Learning from Human Preference”
  • Residual vector quantizer: A sequence of quantizers that progressively encode the residual error left by earlier quantization stages. “the remaining nine are residual vector quantizers”
  • Rollout: A sampled sequence generated by a policy during reinforcement-learning training. “the rollout engines run the routed experts”
  • Rotary position embeddings: A positional-encoding method that incorporates token positions by rotating attention representations. “reusing its rotary position embeddings”
  • Root-mean-square normalization (RMSNorm): A normalization method that rescales activations using their root-mean-square magnitude. “Here RMSNorm⁡\operatorname{RMSNorm} is root-mean-square normalization”
  • Reward hacking: Optimizing a proxy reward in unintended ways that do not correspond to the desired behavior. “whose calibrated and bounded reward keeps reward hacking observable”
  • Sparse mixture-of-experts: An MoE configuration in which only a small number of experts are activated for each token. “with a sparse mixture-of-experts feed-forward network”
  • Streaming interleave: Alternating text input and generated audio so speech can be produced while text continues to arrive. “streaming interleave”
  • Supervised fine-tuning (SFT): Training a pretrained model on labeled examples to improve task-specific behavior. “then instruction SFT and preference RL”
  • Teacher forcing: Training a sequence model by supplying the correct previous tokens as inputs when predicting the next token. “every code is teacher-forced from the ground truth”
  • Tensor: A multidimensional numerical array used to represent data and parameters in neural networks. “the projection tensor by stride”
  • Top-pp sampling: A stochastic decoding method that samples from the smallest set of tokens whose cumulative probability exceeds a chosen threshold. “with no top-pp or top-kk truncation”
  • Top-kk sampling: A decoding method that restricts sampling to the kk most probable tokens. “with no top-pp or top-kk truncation”
  • Transcription-error reward: A reward based on disagreement between generated speech and its intended transcript. “The transcription-error reward compares what an automatic speech recognition (ASR) system hears with the script”
  • Vector quantization: Mapping continuous vectors to entries in a finite learned codebook. “residual vector quantizers that add progressively finer acoustic detail”
  • Zero-shot voice cloning: Generating speech in a target speaker’s voice using reference audio without speaker-specific training. “voice references for zero-shot cloning”

Open Problems

We found no open problems mentioned in this paper.

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

HackerNews