Fish Audio Drama 3 Technical Report
Abstract: We introduce Fish Audio Drama 3, an open-source speech generation model that turns text into directed performances, built on a 35B-parameter mixture-of-experts backbone that activates 3B parameters per token. It clones voices from multiple references, follows natural-language direction, revises a take when the next turn gives a note, voices multi-character scenes, edits existing speech by infilling, and synthesizes long-form and streaming audio. Multi-turn revision, scene dubbing, editing by infilling, and streaming text input are new in Drama 3, all in one model. Drama 3 pairs the hybrid linear-attention backbone with a new codebook multi-token-prediction audio decoder that predicts the acoustic codebooks of each frame in dedicated layers that attend over the entire sequence. The model is trained with a three-stage pre-training recipe, instruction fine-tuning, and large-batch reinforcement learning whose only reward is a single learned model of human preference. Its inference engine decodes a single stream at 271 frames per second on one H200, against 177 for Fish Audio S2, although Drama 3 has more than seven times as many parameters. The model weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/drama-3), and inference is available through SGLang-Omni (https://github.com/sgl-project/sglang-omni). We highly encourage readers to visit https://fish.audio to direct their own performances.
First 10 authors:
Paper Prompts
Sign up for free to create and run prompts on this paper.
Explain it Like I'm 14
1. ¿De qué trata el artículo?
El artículo presenta Fish Audio Drama 3, un sistema de inteligencia artificial que convierte texto en conversaciones y actuaciones de voz.
No está diseñado solo para leer palabras en voz alta. También intenta actuar como una persona o un personaje. Por ejemplo, puede:
- Hablar con diferentes emociones, como alegría, tristeza o enfado.
- Copiar la voz de una persona usando pequeños audios de referencia.
- Interpretar conversaciones entre varios personajes.
- Cambiar una grabación siguiendo instrucciones de un “director”.
- Corregir solo una palabra o frase sin volver a crear toda la grabación.
- Crear audiolibros, doblajes y conversaciones largas.
- Empezar a hablar mientras el texto todavía está llegando, como en una conversación en tiempo real.
El modelo es de código abierto, lo que significa que sus creadores publicaron sus componentes principales para que otras personas puedan estudiarlo y utilizarlo.
2. ¿Qué preguntas intenta responder?
El equipo de investigación quería mejorar varios problemas de los sistemas anteriores de texto a voz. Sus preguntas principales eran:
- ¿Puede una IA actuar y no solo leer? Querían que siguiera instrucciones como “habla más suavemente” o “di esta frase con sorpresa”.
- ¿Puede cambiar una actuación después de recibir comentarios? Por ejemplo, después de producir una frase, el director podría decir: “Hazla más cálida y un poco más lenta”.
- ¿Puede representar una escena completa? El sistema debería utilizar varias voces y mantener claro qué personaje está hablando.
- ¿Puede editar una grabación sin destruir el resto? Si una palabra está mal, la IA debería reemplazar solo esa palabra, como editar una parte de una película sin volver a grabarla entera.
- ¿Puede trabajar con textos y grabaciones largas? Esto es importante para audiolibros, películas y videojuegos.
- ¿Puede producir el audio suficientemente rápido para una conversación en directo?
3. ¿Cómo hicieron la investigación?
Un modelo grande, pero con solo una parte activa
Drama 3 utiliza una arquitectura llamada mezcla de expertos (Mixture of Experts, o MoE). Podemos imaginarla como una escuela con 256 profesores especializados. Para cada palabra o fragmento de audio, el sistema elige solo unos pocos profesores adecuados, en vez de usar a todos.
El modelo completo tiene aproximadamente 35.000 millones de parámetros. Los parámetros son valores internos que la IA aprende durante el entrenamiento. Sin embargo, para cada fragmento solo activa unos 3.000 millones. Esto ayuda a que el sistema sea más rápido y necesite menos memoria durante cada paso.
El audio se divide en pequeñas piezas
La IA no trabaja directamente con la onda sonora completa. Primero convierte el audio en pequeños códigos digitales, parecidos a piezas de LEGO. Cada instante de voz se representa con diez grupos de códigos:
- Uno contiene información más relacionada con el significado y el ritmo.
- Los otros nueve describen detalles del sonido, como el tono, la pronunciación y el timbre de la voz.
Después, el sistema predice esos códigos y los vuelve a convertir en sonido.
Un nuevo decodificador de audio
Drama 3 utiliza un componente llamado codebook-MTP decoder. El nombre es técnico, pero la idea es sencilla: el sistema crea primero una representación general de cada parte del audio y luego añade detalles cada vez más precisos.
Es parecido a dibujar una imagen:
- Primero se hace un boceto.
- Después se añaden las formas principales.
- Finalmente se incorporan los colores y los detalles pequeños.
Este nuevo método permite que la IA tenga en cuenta no solo el texto, sino también todos los detalles de las voces de referencia y de los sonidos anteriores.
Entrenamiento en varias etapas
El modelo se entrenó siguiendo varias fases:
- Preentrenamiento: aprendió con grandes cantidades de texto y grabaciones de voz en unos 80 idiomas y dialectos.
- Entrenamiento con instrucciones: aprendió a responder a indicaciones como “susurra”, “ríe”, “haz una pausa” o “habla con más energía”.
- Aprendizaje con preferencias humanas: varias versiones de una misma actuación fueron comparadas por personas. La IA aprendió qué versiones preferían los oyentes.
Este último proceso se parece a entrenar a un estudiante con comentarios. En lugar de decirle solo si una respuesta está “bien” o “mal”, las personas indican cuál de dos actuaciones suena mejor.
Los investigadores usaron un único modelo aprendido de las preferencias humanas para valorar las actuaciones. No se basaron únicamente en comprobar si cada palabra coincidía exactamente con el texto, porque una buena actuación puede incluir risas, susurros, pausas o sonidos que no aparecen escritos.
Pruebas de velocidad
También crearon un sistema especializado para ejecutar el modelo en tarjetas gráficas. Una tarjeta gráfica es un procesador especialmente bueno para hacer muchos cálculos a la vez.
En estas pruebas midieron cuántos fragmentos de audio podía generar el sistema por segundo y cuántas conversaciones podía atender al mismo tiempo.
4. ¿Cuáles fueron los principales resultados?
Mejor control sobre la actuación
Según el artículo, Drama 3 puede seguir diferentes tipos de instrucciones:
- Instrucciones para una palabra o frase concreta.
- Instrucciones para toda una línea.
- Indicaciones sobre emoción, velocidad, intención o énfasis.
- Notas sobre una actuación anterior.
Por ejemplo, una persona puede pedir que una frase sea “más suave y cercana que la anterior”. El sistema conserva la actuación anterior en el contexto y produce una nueva versión.
En las pruebas de actuación dirigida, cada ronda de correcciones mejoró la precisión con la que seguía el estilo y la emoción solicitados.
Clonación de voz en varios idiomas
El modelo puede producir una voz parecida a la de una grabación de referencia. En una prueba multilingüe, obtuvo la tasa de errores más baja entre los sistemas comparados en chino, neerlandés, inglés, alemán y turco.
Esto significa que, en esas pruebas, cometió menos errores al producir el contenido hablado. Sin embargo, una tasa de error más baja no garantiza que todas las actuaciones suenen perfectas o que sean indistinguibles de una persona real.
Edición de audio
Drama 3 puede reemplazar una palabra o una frase dentro de una grabación manteniendo el resto del audio. Esto se llama infilling, o “rellenado”.
Es parecido a corregir una palabra en una película sin volver a grabar toda la escena. Las pruebas indicaron que tuvo menos errores que un servicio comparado en inglés y chino. En algunas pruebas en chino también consiguió una similitud de voz mayor que otros modelos especializados.
Escenas con varios personajes
El sistema puede utilizar varias grabaciones de referencia y representar conversaciones entre diferentes personajes. También puede cambiar de idioma para un mismo personaje conservando una voz parecida.
Esto podría ser útil para:
- Doblaje de películas.
- Videojuegos.
- Audiolibros.
- Historias de audio.
- Personajes interactivos.
Generación rápida
En una tarjeta gráfica NVIDIA H200, Drama 3 generó aproximadamente 271 fragmentos de audio por segundo, frente a los 177 de un modelo anterior llamado Fish Audio S2.
Aunque Drama 3 tiene más de siete veces más parámetros que S2, fue más rápido en esa medición. Esto se debe a que su arquitectura y su programa de ejecución fueron diseñados para evitar cálculos y movimientos de memoria innecesarios.
La velocidad es importante porque el audio debe generarse más rápido de lo que se reproduce. Si el sistema produce sonido demasiado lentamente, la conversación tendría pausas incómodas.
Modelo abierto
Los autores publicaron los pesos del modelo y herramientas para ejecutarlo en plataformas como GitHub y Hugging Face. Esto permite que investigadores y desarrolladores puedan estudiar el sistema y crear nuevas aplicaciones.
5. ¿Por qué son importantes estos resultados?
La mayoría de los sistemas de texto a voz se concentran en pronunciar correctamente las palabras. Drama 3 intenta resolver un problema más difícil: hacer que la voz parezca una actuación.
Esto es importante porque una actuación real no depende solo de las palabras. También incluye:
- El tono.
- La emoción.
- La velocidad.
- Las pausas.
- Las risas.
- Los susurros.
- La relación entre varios personajes.
- Las correcciones que da un director.
El artículo también muestra que evaluar una actuación no siempre consiste en comprobar si coincide exactamente con el texto. Una risa o un susurro podrían hacer que un programa de reconocimiento de voz cometa un error, aunque la actuación sea mejor y más natural.
6. Posibles efectos y limitaciones
Drama 3 podría hacer más fácil y barato crear audiolibros, doblajes, videojuegos y asistentes de voz. Una persona podría escribir una escena, indicar cómo quiere que suene y corregirla con instrucciones sencillas, sin contratar inmediatamente a muchos actores o volver a grabar todo.
También podría ayudar a producir audio en varios idiomas y crear personajes capaces de responder rápidamente.
Sin embargo, el sistema tiene riesgos y limitaciones. La clonación de voces podría utilizarse sin permiso para imitar a personas reales, crear engaños o fabricar grabaciones falsas. Por eso sería importante obtener autorización, identificar el audio generado por IA y proteger la identidad de los hablantes.
Además, las pruebas descritas en el artículo no significan que el modelo sea perfecto. Puede equivocarse, interpretar mal una instrucción o producir una voz que no suene completamente natural. Los resultados dependen de los idiomas, las grabaciones de referencia y las situaciones evaluadas.
Conclusión
Fish Audio Drama 3 intenta transformar la IA de voz en una especie de actor digital. No solo lee un guion: puede recibir dirección, cambiar su actuación, representar a varios personajes, editar partes concretas y producir audio rápidamente.
La investigación es importante porque acerca la tecnología de voz a usos más creativos y naturales. Al mismo tiempo, su desarrollo hace necesario hablar de permisos, privacidad y seguridad para evitar que las voces clonadas se utilicen para engañar a otras personas.
Practical Applications
Immediate Applications
- Audiobook and audio-drama production — media and publishing. Producers can use Drama 3 to generate directed performances from scripts, clone approved character voices from reference recordings, and revise takes through natural-language notes such as “slower,” “warmer,” or “more suspenseful.” This can reduce the number of manual recording cycles for narration, podcasts, and audio drama.
- Potential workflow: script segmentation → voice-reference upload → first generated take → director feedback → regenerated take → human editorial review.
- Dependencies: valid voice permissions, high-quality reference audio, human quality control, and sufficient GPU capacity. The reported throughput is measured on specialized NVIDIA hardware and should not be assumed for ordinary consumer devices.
- Localized dubbing of films, games, and online video — entertainment and localization. The multi-speaker scene capability can generate dialogue for several characters in one pass, including characters speaking different languages while retaining their assigned voices. It can support first-pass dubbing, localization previews, animatics, and production drafts.
- Potential products: a script-to-dubbing pipeline with speaker assignment, multilingual line generation, timing controls, and export to standard audio-editing tools.
- Dependencies: translation quality, cultural adaptation, lip-sync and timing alignment, actor or rights-holder consent, and professional review before release. The paper evaluates multilingual voice cloning but does not establish reliable lip synchronization.
- Voice editing and patch recording — post-production and broadcasting. Editors can replace a word, phrase, or sentence inside an existing recording through speech infilling rather than rerecording the entire passage. This is useful for correcting names, factual errors, pronunciation, or compliance issues in podcasts, commercials, lectures, and voice-over work.
- Potential workflow: select erroneous span → provide replacement text and optional direction → generate localized edit → compare acoustic continuity with surrounding speech → approve or reject.
- Dependencies: the surrounding recording must provide adequate context; edits may still require manual crossfades, noise matching, and timing adjustment. The system should not be used to alter legally sensitive recordings without disclosure and authorization.
- Interactive character voices — games, virtual worlds, and conversational applications. Streaming text input and directed performance can support non-player characters, virtual assistants, educational characters, and interactive fiction that speak as text becomes available rather than waiting for a complete response.
- Potential tools: real-time character dialogue engines with persistent voice references, emotion or style tags, and per-turn performance direction.
- Dependencies: end-to-end latency, moderation, predictable behavior, and compute availability. The reported 271 frames per second on one H200 exceeds the approximately 21.5 frames per second playback requirement, but this does not by itself guarantee low conversational latency under many simultaneous users.
- Accessible content generation — education, public services, and assistive technology. Organizations can produce expressive spoken versions of articles, course materials, public announcements, and user-generated text. Directions such as whispering, pausing, or emphasizing a phrase may improve comprehension and engagement compared with neutral TTS.
- Potential workflow: accessible-text conversion → reading-level and pronunciation review → expressive speech generation → human or automated accessibility checks.
- Dependencies: language and dialect coverage, pronunciation dictionaries, accessibility testing, and safeguards against inappropriate or stigmatizing voice characterization.
- Rapid prototyping for researchers and developers — academia and open-source software. The publicly released weights, Hugging Face distribution, GitHub code, and SGLang-Omni inference support make it possible to study expressive TTS, voice cloning, speech editing, multi-speaker generation, and preference-based alignment without building a model from scratch.
- Research uses: benchmarking controllability, testing new reward models, evaluating speech infilling, studying long-context audio generation, and developing specialized adapters.
- Dependencies: hardware capable of hosting a large sparse model, compatible inference software, licensing compliance, reproducible evaluation sets, and careful handling of personally identifiable voice data.
- Preference-based quality evaluation — media tooling and AI engineering. The paper’s learned human-preference reward model provides a practical template for ranking alternative speech takes when conventional ASR or speaker-similarity metrics penalize expressive delivery. Teams can use a similar approach to select among generated takes for internal review.
- Potential workflow: sample multiple takes → rank or score them with human-preference data → present the strongest candidates to an editor.
- Dependencies: representative preference annotations, calibration against different languages and performance styles, and monitoring for reward-model bias or reward hacking. The paper’s reward model should not be treated as a universally reliable quality or safety judge.
- Government and institutional communications — public information. Agencies and organizations can prepare multilingual announcement drafts, emergency-information prototypes, and accessible audio versions of written notices. Directed pauses and emphasis may make key instructions clearer.
- Dependencies: authorized voices, verified source text, review by communication professionals, and explicit labeling when speech is synthetic. Emergency use requires conservative validation because generation errors could have material consequences.
Long-Term Applications
- End-to-end professional dubbing and localization platforms — entertainment technology. A future system could combine Drama 3-style scene generation with translation, phoneme-level timing, lip synchronization, ambience preservation, actor approval, and automated quality assurance. This could enable scalable localization for large catalogs while retaining human control over final performances.
- Research required: robust cross-lingual prosody transfer, duration control, lip-sync integration, dialogue mixing, and evaluation by professional dubbing actors.
- Key assumptions: voice rights and performer compensation models are resolved, generated voices can meet contractual quality requirements, and localization remains culturally appropriate.
- Human-in-the-loop virtual production — film, television, and games. Multi-turn revision could become a conversational directing interface in which a director gives notes, compares several takes, locks selected phrases, and requests localized edits without losing the broader scene context.
- Potential product: a digital audio workstation plug-in or cloud production environment with versioned takes, scene memory, speaker libraries, and reversible edits.
- Dependencies: stable long-context behavior, deterministic regeneration controls, integration with existing editing software, and provenance tracking for every generated segment.
- Personalized tutoring and language learning — education. Future systems could generate dialogues with controllable pacing, emotion, pronunciation, and character roles; revise delivery based on learner feedback; and provide the same lesson in multiple languages or reading styles.
- Potential applications: role-play exercises, pronunciation demonstrations, accessible reading tutors, and adaptive listening assessments.
- Research required: pedagogical validation, age-appropriate interaction, reliable pronunciation feedback, and safeguards against reinforcing accent or cultural stereotypes. Generated speech should supplement rather than replace qualified instruction.
- Assistive and rehabilitative speech systems — healthcare and accessibility. With additional validation, expressive voice cloning and localized infilling could help people with speech loss maintain a personalized synthetic voice, or allow communication devices to express urgency, emotion, or conversational nuance.
- Potential workflow: consented voice capture before or during illness → personalized voice model → clinician- or user-controlled text generation → speech output.
- Dependencies: clinical trials, privacy-preserving training, durable identity and consent controls, intelligibility under medical conditions, and strict limits on who may authorize speech. The paper provides no clinical evidence, so this remains a research direction rather than a medical product.
- Robotics and embodied agents — robotics and human-computer interaction. Robots could use directed, context-aware speech to coordinate with people, distinguish speakers, and adapt delivery to operational conditions—for example, using concise and emphatic speech during an alert and calmer speech during routine interaction.
- Research required: grounding speech generation in sensor and task state, guaranteed latency, interruption handling, safety policies, and integration with robot motion and dialogue planners.
- Dependencies: the model must avoid generating misleading confidence or unauthorized instructions; expressive speech alone does not provide reliable task planning or situational awareness.
- Low-latency, high-concurrency speech services — cloud infrastructure. The inference innovations—sparse MoE activation, FP8 expert weights, fused kernels, prefill/decode separation, and shape-driven scheduling—could support scalable APIs for streaming narration, call-center agents, games, and interactive media.
- Potential products: hosted streaming TTS, batch audiobook generation, and multi-tenant dubbing services with GPU-aware admission control.
- Research and engineering required: performance on varied GPUs, quantization-quality trade-offs, fault tolerance, autoscaling, cost analysis, and privacy-preserving deployment. The paper reports results primarily on H200 and B300 hardware, so general cloud economics remain unverified.
- Synthetic media provenance and consent infrastructure — policy, law, and platform governance. Because the model can clone voices, revise speech, and generate convincing multi-character scenes, future deployments should include consent records, watermarking or provenance metadata, audit logs, and detection or disclosure mechanisms.
- Potential policy tools: voice-licensing registries, revocation systems, mandatory synthetic-audio labels, and provenance-aware media platforms.
- Dependencies: watermark robustness after editing or compression, interoperability among platforms, enforceable voice rights, and balanced regulation that protects performers without blocking legitimate accessibility and creative uses.
- Personalized financial, legal, and customer-service communication — finance and enterprise software. A mature system could generate approved, personalized explanations of account activity, contracts, or support procedures in a consistent organizational voice, with controlled emphasis and multilingual delivery.
- Research required: verifiable grounding, exact-number preservation, auditability, and strict separation between generated explanation and authorized decision-making.
- Dependencies: the current model’s expressive generation must be paired with deterministic retrieval and validation. It should not independently produce legally binding advice, financial decisions, or compliance-sensitive statements.
- Large-scale speech dataset creation and model training — academia and industry research. Directed generation could produce controlled speech variations for testing emotion recognition, dialogue systems, accessibility interfaces, and multilingual speech technologies.
- Dependencies: synthetic data may reproduce model biases and should not be assumed equivalent to diverse human recordings. Datasets require documentation, consent analysis, quality filtering, and evaluation for distributional artifacts.
- Advanced audio editing as a general-purpose authoring medium — creative software. Longer term, speech editors could expose text-based operations such as “make this sentence more reassuring,” “replace the name but preserve timing,” or “match the following character’s energy,” combining semantic editing with acoustic continuity.
- Research required: precise span control, preservation of background sound and room acoustics, speaker identity consistency, temporal constraints, and transparent version comparison.
- Dependencies: reliable separation between intended edits and unintended changes; human review remains necessary for professional, contractual, and reputationally sensitive content.
Glossary
- Autoregressive (AR): A generation process in which each prediction depends on previously generated outputs. “per-frame fast autoregressive (AR) module”
- Bfloat16 (BF16): A 16-bit floating-point format commonly used for deep-learning computation. “the trainer runs in bfloat16 (BF16)”
- Bradley–Terry loss: A probabilistic loss for learning preferences from pairwise comparisons. “It is trained on human preference judgments over pairs of takes with the Bradley--Terry loss”
- Causal attention: Attention restricted to the current position and preceding positions, preventing access to future tokens. “each layer attends causally over the whole sequence”
- Codebook: A discrete collection of vector-quantized representations used to encode audio information. “The first codebook is semantic”
- Codebook-MTP decoder: A decoder that predicts multiple audio codebooks through sequential, dedicated prediction layers. “Drama~3 pairs the hybrid linear-attention backbone with a new codebook multi-token-prediction audio decoder”
- Conservative direct preference optimization (DPO): A preference-optimization method that uses smoothed preference labels to reduce overconfident updates. “as in conservative direct preference optimization (DPO)”
- CUDA graphs: A CUDA mechanism that records and replays a sequence of GPU operations to reduce CPU launch overhead. “CUDA graphs record a whole step once and replay it”
- Curriculum-free teacher forcing: Training in which the model receives ground-truth previous outputs rather than its own predictions. “every code is teacher-forced from the ground truth”
- Delta rule: An update mechanism for maintaining a recurrent state, used here in the Gated DeltaNet architecture. “applies the gated delta rule”
- Descript Audio Codec: An audio codec that converts waveforms into discrete code sequences and reconstructs waveforms from them. “Built on the Descript Audio Codec”
- Document masking: A training technique that prevents attention or loss computation from crossing document boundaries in packed sequences. “samples are packed with document masking”
- Exposure bias: The discrepancy between training, when ground-truth history is available, and inference, when the model conditions on its own generated outputs. “to limit exposure bias”
- FP8: An 8-bit floating-point representation used to reduce memory use and accelerate neural-network computation. “the experts use the microscaling FP8 format”
- Gated DeltaNet: A recurrent or linear-attention architecture that uses gating and delta-rule updates to maintain sequence information efficiently. “a hybrid that stacks blocks of three Gated DeltaNet layers”
- Gated full attention: A full-attention mechanism augmented with learned gating operations. “one gated full-attention layer”
- Gated RMSNorm: A gated form of root-mean-square normalization applied within a neural-network layer. “a gated RMSNorm per layer”
- General matrix multiplication (GEMM): A highly optimized operation that multiplies matrices and optionally adds a scaled matrix. “the router's general matrix multiplication (GEMM), softmax, and top-8 selection”
- Group Relative Policy Optimization (GRPO): A reinforcement-learning method that computes relative advantages among multiple sampled outputs for the same prompt. “We use Group Relative Policy Optimization (GRPO)”
- Hidden state: An internal vector representation produced by a neural-network layer for a sequence position. “the backbone's final hidden state”
- Infilling: Generating missing or replaced content within an existing sequence while preserving surrounding context. “edits speech by infilling”
- Instruction fine-tuning: Supervised training that teaches a model to follow natural-language instructions. “instruction fine-tuning, and large-batch reinforcement learning”
- Key–value (KV) cache: Stored attention keys and values reused during autoregressive generation to avoid recomputation. “only one layer in four keeps a key--value (KV) cache”
- Kullback–Leibler (KL) penalty: A regularization term that discourages a policy from diverging too far from a reference distribution. “without a Kullback--Leibler (KL) penalty”
- Linear attention: An attention formulation whose computation scales more favorably with sequence length than standard full attention. “a hybrid linear-attention LLM”
- Load-balancing loss: An auxiliary objective that encourages tokens to be distributed evenly among mixture-of-experts routes. “an auxiliary load-balancing loss”
- Mixture of experts (MoE): A neural-network architecture that routes each token to a subset of specialized feed-forward networks. “a sparse mixture-of-experts feed-forward network”
- Multi-token prediction (MTP): A method in which a model predicts multiple future or related tokens through additional prediction modules. “with a Codebook-MTP Decoder”
- Muon: An optimizer designed for training large neural-network models, used here during pre-training and reinforcement learning. “the model is optimized with Muon”
- Negative log-likelihood (NLL): The negative logarithm of the probability assigned to the observed target, commonly used as a training objective. “by the teacher-forced negative log-likelihood (NLL)”
- Proximal policy optimization (PPO): A policy-gradient reinforcement-learning algorithm that limits updates using a clipped probability ratio. “the proximal policy optimization (PPO) ratio”
- Quantization: Representing numerical model parameters with lower-precision values to reduce memory use and computation cost. “so serving adds no quantization of its own”
- Reinforcement learning from human preference: Training a model to optimize a reward derived from human judgments of generated outputs. “Reinforcement Learning from Human Preference”
- Residual vector quantizer: A sequence of quantizers that progressively encode the residual error left by earlier quantization stages. “the remaining nine are residual vector quantizers”
- Rollout: A sampled sequence generated by a policy during reinforcement-learning training. “the rollout engines run the routed experts”
- Rotary position embeddings: A positional-encoding method that incorporates token positions by rotating attention representations. “reusing its rotary position embeddings”
- Root-mean-square normalization (RMSNorm): A normalization method that rescales activations using their root-mean-square magnitude. “Here is root-mean-square normalization”
- Reward hacking: Optimizing a proxy reward in unintended ways that do not correspond to the desired behavior. “whose calibrated and bounded reward keeps reward hacking observable”
- Sparse mixture-of-experts: An MoE configuration in which only a small number of experts are activated for each token. “with a sparse mixture-of-experts feed-forward network”
- Streaming interleave: Alternating text input and generated audio so speech can be produced while text continues to arrive. “streaming interleave”
- Supervised fine-tuning (SFT): Training a pretrained model on labeled examples to improve task-specific behavior. “then instruction SFT and preference RL”
- Teacher forcing: Training a sequence model by supplying the correct previous tokens as inputs when predicting the next token. “every code is teacher-forced from the ground truth”
- Tensor: A multidimensional numerical array used to represent data and parameters in neural networks. “the projection tensor by stride”
- Top- sampling: A stochastic decoding method that samples from the smallest set of tokens whose cumulative probability exceeds a chosen threshold. “with no top- or top- truncation”
- Top- sampling: A decoding method that restricts sampling to the most probable tokens. “with no top- or top- truncation”
- Transcription-error reward: A reward based on disagreement between generated speech and its intended transcript. “The transcription-error reward compares what an automatic speech recognition (ASR) system hears with the script”
- Vector quantization: Mapping continuous vectors to entries in a finite learned codebook. “residual vector quantizers that add progressively finer acoustic detail”
- Zero-shot voice cloning: Generating speech in a target speaker’s voice using reference audio without speaker-specific training. “voice references for zero-shot cloning”





