--- title: FT-GRPO for Audio Deepfake Detection url: https://www.emergentmind.com/topics/frequency-time-group-relative-policy-optimization-ft-grpo type: topic --- # FT-GRPO for Audio Deepfake Detection Frequency Time-Group Relative Policy Optimization (FT-GRPO) is a structured two-stage training paradigm developed for interpretable all-type audio deepfake detection (ADD) using Audio Large Language Models (ALLMs). FT-GRPO addresses deficiencies of standard supervised and reinforcement fine-tuning approaches by training ALLMs to produce not only accurate real/fake classifications, but also frequency-time grounded, chain-of-thought (CoT) rationales. The method exploits high-quality demonstrations and enforces explicit rule-based frequency-time constraints during reinforcement learning, thereby mitigating common issues such as interpretability collapse and reward hacking in sparse supervision regimes [2601.02983]. ## 1. Motivation and Limitations of Standard Methods ALLMs applied to ADD face two primary methodological challenges: - **Supervised Fine-Tuning (SFT) Limitation**: Conventional SFT on binary real/fake labels reduces model behavior to a black-box classifier, precluding transparent or granular interpretability of model decisions. - **Vanilla Reinforcement Fine-Tuning (RFT) Limitation**: RFT under sparse reward structures—typically rewarding only superficial format compliance or final prediction—enables the model to generate ungrounded or hallucinated rationales simply to maximize reward, a phenomenon termed reward hacking. These limitations undermine both the trustworthiness and actionable value of ALLM-based ADD systems, particularly for real-world deployment where interpretability and generalization across heterogeneous audio types (speech, singing, environmental sounds, music) are critical. ## 2. Two-Stage Training Framework FT-GRPO is organized into two sequential stages: 1. **Supervised Fine-Tuning (Cold Start)**: ALLMs are initialized with high-quality, frequency-time structured chain-of-thought (FT-CoT) rationales, where demonstration data include explicit analysis in both frequency and time domains for each audio clip. The model architecture comprises an audio encoder, aligner, and large language model, with parameter-efficient LoRA adapters injected at all stages (rank $r=64$, $\alpha=16$). Training is performed on the “think” dataset subset, optimizing conditional log-likelihood over demonstration sequences. 2. **Frequency Time-Group Relative Policy Optimization (FT-GRPO)**: After SFT, group relative policy optimization is conducted under frequency-time constraints. FT-GRPO optimizes a composite reward structure, explicitly encouraging (i) prediction accuracy, (ii) output format compliance (wrapping rationales/answers in specified tags), and (iii) complete, grounded FT reasoning by requiring inclusion of both frequency and time domain arguments in each rationale. The process includes group-based sampling and advantage estimation for stable reinforcement learning dynamics. ## 3. FT-GRPO Methodology ### 3.1 Markov Decision Process Formulation FT-GRPO is framed as a single-step episodic Markov decision process (MDP): - **State $s$**: Input audio $a$ and prompt prefix. - **Action $a_t$**: Emission of structured output $o = (\tau, y_\mathrm{pred})$, where $\tau$ is the FT-CoT rationale and $y_\mathrm{pred} \in \{\text{real, fake}\}$. - **Transition**: The episode ends after output, with no further state transitions. - **Policy $\pi_\theta$**: Parameterizes $P(o|a)$. - **Reference policy $\pi_\mathrm{ref}$**: Fixed snapshot post-SFT, enabling KL regularization. ### 3.2 Frequency-Time Constraints and Reward Composition Two explicit meta-tags are defined: $\langle$Frequency Domain$\rangle$ and $\langle$Time Domain$\rangle$. Each CoT rationale must include both tags, each followed by at least one grammatically complete sentence, enforced via a completeness check function $g(a_i)$. The composite reward for output $o = (\tau, y_\mathrm{pred})$ on $(a, y)$ is: - **Accuracy reward**: $r_\mathrm{acc} = 1$ if $y_\mathrm{pred} = y$, else $0$. - **Format reward**: $r_\mathrm{fmt} = \mathbb{I}(\mathcal{F}(o))$, where $\mathcal{F}(o)$ checks structural tag compliance. - **FT reasoning reward**: $r_\mathrm{ft} = \tfrac{1}{2} \sum_{i\in\{\text{FD, TD}\}} \mathbb{I}(a_i \in \tau \land g(a_i) = \text{True})$, yielding total reward values in $\{0, 0.5, 1.0\}$. - **Total reward**: $r = r_\mathrm{acc} + \alpha \, r_\mathrm{fmt} + \beta \, r_\mathrm{ft}$, with $\alpha = \beta = 0.1$. ### 3.3 Group Relative Policy Optimization For each audio instance, a group of $G$ outputs $\{o_i\}_{i=1}^G$ is sampled: - **Group mean**: $\bar r = \frac{1}{G} \sum_i r_i$ - **Group std**: $\sigma_r = \mathrm{std}(\{r_i\}) + \epsilon$ - **Advantage per sample**: $A_i = \frac{r_i - \bar r}{\sigma_r}$ The optimization objective per batch is: $$ \mathcal{L}_\mathrm{FT-GRPO}(\theta) = -\mathbb{E}_{a}\biggl[\sum_{i=1}^G A_i \log \pi_\theta(o_i \mid a) \biggr] + \lambda\,\mathbb{E}_a\left[ D_\mathrm{KL}(\pi_\mathrm{ref}(\cdot \mid a) \| \pi_\theta(\cdot \mid a)) \right] $$ where $\lambda$ is the KL penalty strength (set via warmup ratio $0.05$). ### 3.4 Algorithmic and Implementation Notes - **LoRA adapters**: rank $r=64$, $\alpha=16$, dropout $0.05$. - **Learning rate**: $1 \times 10^{-5}$ for both stages. - **Batch size**: $16$ (SFT), $32$ (GRPO). - **Group size**: $G=8$, sampling temperature $0.9$. - **Epochs**: SFT ($2$–$3$), GRPO ($2$). - **Framework**: ms-swift + DeepSpeed ZeRO-2, bfloat16 precision. ## 4. Experimental Evaluation FT-GRPO demonstrates consistent improvements over both SFT and prior state-of-the-art ADD techniques. For a speech-trained Qwen2.5-Omni-3B ALLM, the following accuracy (ACC) improvements were observed after FT-GRPO compared to SFT: | Audio Type | SFT ACC (%) | FT-GRPO ACC (%) | Δ | |--------------|-------------|-----------------|-----| | Speech | 99.04 | 99.75 | +0.71 | | Sound | 73.31 | 75.05 | +1.74 | | Singing | 66.29 | 84.26 | +17.97| | Music | 91.16 | 90.44 | -0.72 | | Average | 82.45 | 87.38 | +4.93 | Additionally, FT-GRPO surpasses alternative models, including W2V2-AASIST (avg. 63.5%), WPT-W2V2-AASIST (69.2%), and ALLM4ADD (85.8%). In all-type co-training, FT-GRPO yields an average ACC of 90.10% (+5.15 over SFT) [2601.02983]. ## 5. Interpretability via Frequency-Time Grounded Chain-of-Thought By combining rule-based FT constraints with explicit reward structure during GRPO, FT-GRPO ensures that every output rationale contains: - At least one frequency-domain analysis, explicitly tagged and evidenced (e.g., “The spectrogram shows elevated energy around 8–10 kHz with uniform harmonics—an indicator of neural vocoder aliasing”). - At least one time-domain analysis, explicitly tagged and evidenced (e.g., “There is a sudden amplitude jump at 1.2 s without natural fade-in/out, revealing a generated segment boundary”). This FT-grounded CoT structure renders the decision process transparent. Practitioners can verify rationale validity alongside the real/fake output, substantially increasing trust and verifiability for critical MOVE domains. ## 6. Implications and Significance FT-GRPO establishes a robust paradigm for interpretable, all-type ADD by integrating strong SFT initialization with structured, constraint-driven reinforcement learning. The methodology mitigates black-box behavior and reward hacking, and enforces output rationales traceable to observable acoustic phenomena in both frequency and time domains. *A plausible implication is that FT-GRPO's constraint-based rationale structuring may generalize to other domains requiring transparent multi-modal LLM predictions. This suggests broader adoption in interpretable AI for safety-critical applications.* ## 7. Example FT-CoT Output An indicative post-GRPO rationale for a fake audio instance: ``` 〈think〉 ⟨Frequency Domain⟩ The spectrogram shows elevated energy around 8–10 kHz with uniform harmonics—an indicator of neural vocoder aliasing. ⟨Time Domain⟩ There is a sudden amplitude jump at 1.2 s without natural fade-in/out, revealing a generated segment boundary. 〈/think〉 〈answer〉fake〈/answer〉 ``` This output exemplifies the completeness and interpretability guaranteed by FT-GRPO for each ADD decision [2601.02983]. Source: https://www.emergentmind.com/topics/frequency-time-group-relative-policy-optimization-ft-grpo