---
title: DeepSeekR1 Professor Agent
url: https://www.emergentmind.com/topics/deepseekr1-professor-agent
type: topic
---

# DeepSeekR1 Professor Agent

The DeepSeekR1 Professor Agent is a search-integrated reasoning agent built on the DeepSeekR1 framework, enhanced by the SRR-Judge system for step-level assessment and refinement. It exemplifies an advanced approach to open-domain question answering and tool-augmented reasoning, incorporating explicit evaluation and correction at each decision step to produce robust, high-quality responses in complex multi-hop QA scenarios. The DeepSeekR1 Professor Agent leverages a modular architecture—Planner, Search Actor, and Reasoner—and adopts an SRR-Judge-mediated "rate-and-refine" workflow, aligning policy to fine-grained expert feedback and facilitating substantial improvement in pass@1 performance across challenging QA benchmarks [2602.07773].

## 1. Architecture and Workflow

The DeepSeekR1 Professor Agent operates via a modified ReAct loop—a stepwise interaction paradigm in which at each step $j$, the agent accumulates a history $h_j=(t_0,\text{act}_0,o_0,\ldots,t_{j-1},\text{act}_{j-1},o_{j-1})$, generates a "thought" $t_j$, selects an action $\text{act}_j\in\{\text{search},\text{answer}\}$, receives observation $o_j$, and proceeds iteratively to completion. The SRR-Judge model, a 32B-parameter LLM fine-tuned for step-level evaluation, interposes at each decision point to score and optionally refine ($t_j,\text{act}_j$) pairs based on their context-conditioned quality.

The inference protocol is organized as follows:
- Each step $j$, the agent proposes $N$ candidate thought–action pairs $\{(t_j^i, \text{act}_j^i)\}_{i=1}^N$.
- SRR-Judge evaluates these, outputting $(e_j^i, r_j^i, \tilde t_j^i, \tilde{\text{act}}_j^i)$ where $r_j^i\in\{1,\ldots,5\}$ denotes a step-level quality rating, $e_j^i$ is a short explanation, and $(\tilde t_j^i, \tilde{\text{act}}_j^i)$ provide optional refinements.
- The highest-rated candidate ($r_j^{i*} = \max_i r_j^i$) is selected if $r_j^{i*}\geq\tau$; otherwise, its refinement $(\tilde t_j^{i*}, \tilde{\text{act}}_j^{i*})$ is used.
- This process continues until an answer action is selected or the predefined maximum number of steps ($K=20$) is reached.

Thresholds and candidate counts are set as $N=1,\tau=4$ for online inference and $N=5,\tau=4$ for offline alignment trace generation.

## 2. Step-Level Rating, Training, and Calibration

SRR-Judge assigns ratings $r_j\in\{1,\ldots,5\}$ to each step considering four criteria: clarity/conciseness, logical structure, query appropriateness, and coverage/improvement potential. The model is fine-tuned from QwQ-32B on $(h_j,t_j,\text{act}_j)\to(e_j,r_j,\tilde t_j,\tilde{\text{act}}_j)$ supervision using the standard cross-entropy loss:
\[
\mathcal{L}_{SFT} = -\sum_{(h,t,\text{act})\in\mathcal{D}_{judge}} \log P_{\theta}(e, r, \tilde t, \tilde{\text{act}} \mid h, t, \text{act})
\]
Calibration is quantified by the point-biserial correlation $\rho_{pb}$ between step ratings and final-answer correctness $y\in\{0,1\}$:
\[
\rho_{pb} = \frac{\overline{r}_1 - \overline{r}_0}{\sigma_r} \sqrt{p (1-p)}
\]
where $p = P(y=1)$, $\overline{r}_1 = E[r|y=1]$, $\overline{r}_0 = E[r|y=0}$, and $\sigma_r$ denotes the standard deviation of $r$.

## 3. Iterative Rejection-Sampling Fine-Tuning (RFT)

The iterative RFT procedure aligns the DeepSeekR1 policy to high-quality, SRR-Judge-rated trajectories:
- For each RFT iteration, the current policy generates best-of-$N$ (typically $N=5$) trajectories per question using SRR-Judge scoring.
- Only trajectories for which $\min_j r_j\geq\tau_{accept}$ (with $\tau_{accept}=4$) are retained.
- The aggregate set of accepted trajectories is used to further fine-tune the policy via supervised loss.
- This loop is repeated for $T=2$ rounds or as needed for further improvement.

The following pseudocode encapsulates the RFT procedure:
``` 
procedure RFT_ITERATION(current_policy M_cur, judge F, data Q, iterations T)
  D_all ← ∅
  for it in 1..T do
    D_new ← ∅
    for q in Q do
      trajs ← INFER_WITH_SRR(M_cur, F, q, K, N=5, τ=4)
      if min_j r_j ≥ τ_accept then
        D_new ← D_new ∪ {trajs}
    D_all ← D_all ∪ D_new
    M_cur ← SFT_FINE_TUNE(M_cur, D_all)
  return M_cur
end procedure
```
Acceptance is predicated strictly on all step ratings meeting or exceeding the threshold.

## 4. Data Annotation, Filtering, and Trajectory Generation

SRR-Judge annotation relies on QA pairs drawn from a mixture of open QA benchmarks (e.g., InfoSeek-Hard, DuetQA-Verified, ASearcher-LRM), with approximately $6\,000$ pairs used for judge training and $8\,000$ for SFT in the described experiments.

The annotation pipeline entails:
1. Generating search-integrated trajectories using a strong teacher (DeepSeek-V3.1) under vanilla ReAct.
2. Performing 5 independent SRR-Judge annotations per step, computing the majority-vote for $r_j$.
3. Extracting one run’s $e_j$, $\tilde t_j$, $\tilde{\text{act}}_j$ as representative annotation.
4. Discarding trajectories whose average step rating correlates with final correctness below the point-biserial cutoff ($<0.7$).

Curation of the SRR-annotated pool involves balancing the distribution of step ratings by upsampling rare cases and synthesizing negative examples (e.g., $r=1$).

## 5. Retrofitting SRR-Judge to DeepSeekR1 Professor Agent

DeepSeekR1 is organized as three coupled modules: Planner (thought generation), Search Actor (tool invocation), and Reasoner (answer synthesis). Integration of SRR-Judge proceeds as follows:

- **Step Definition**: Treat each planner, search actor, and reasoner invocation as a step $(t_j,\text{act}_j)$.
- **Inference Pipeline**:
  1. Upon Planner emission of $t_j$ and Search Actor selection of $\text{act}_j$, invoke SRR-Judge $F_\theta(h_j, t_j, \text{act}_j)\to(e_j,r_j,...)$.
  2. If $r_j<4$, replace $t_j\leftarrow\tilde{t}_j$ via Reasoner logic and re-present to Search Actor.
  3. Proceed if rating passes threshold.
- **Training Pipeline**:
  1. Deploy DeepSeekR1 (current policy) on QA pool.
  2. Annotate trajectories with SRR-Judge; filter for $\min_j r_j\geq4$.
  3. Augment with baseline SFT data.
  4. Fine-tune all modules end-to-end to replicate filtered high-quality trajectories.
  5. Repeat RFT as detailed above.

Standard evaluation utilizes the same tool-budgeted benchmarks as the original experiments: BrowseComp-En, BrowseComp-Zh, and Xbench-DeepSearch, with $K=20$ step limit.

## 6. Empirical Evaluation

The efficacy of the SRR-Judge-augmented DeepSeekR1 Professor Agent is demonstrated by both improved calibration of step ratings and substantial performance gains under standard metrics.

- **Step Rating–Correctness Correlation**: SRR-Judge (QwQ-32B) achieves first-step $\rho=0.43$, last-step $\rho=0.495$, and average-step $\rho=0.479$, surpassing the larger DeepSeek-V3.1's average-step $\rho\approx0.45$.
- **Inference-Time Refinement Gains**: Across BrowseComp, BrowseComp-ZH, and Xbench-DeepSearch, DeepSeek-R1 + SRR refine achieves 14.6±3.2, 37.8±0.7, and 55.3±2.5 pass@1 respectively, improvements over QwQ-32B + SRR refine and much higher than vanilla QwQ-32B.
- **Alignment (RFT) Gains**:
  - RFT with SRR-Judge yields 16.2±0.8 (BrowseComp), 38.3±2.4 (BrowseComp-Zh), and 61.3±1.5 (Xbench) after two iterations, representing ∼10 percent absolute pass@1 increase over direct SFT, with all improvements statistically significant ($p<0.01$, paired bootstrap).

## 7. Design Choices and Hyperparameters

Key design parameters include:
- Maximum steps $K=20$.
- Candidate beams $N=1$ online, $N=5$ offline.
- Rating and refinement threshold $\tau=4$.
- Judge training: 1 epoch SFT on ≈40,000 step examples; upsampling for $r=2\times3$, 10,000 samples for $r\in\{3,4,5\}$, and synthetic 10,000 $r=1$ negatives.
- RFT rounds $T=2$, each using ≈6,000 QA instances.
- Trajectory filtering: point-biserial ≥ 0.7.

These choices optimize the balance between computational cost and the reliability of both ratings and policy alignment to high-quality intermediate reasoning steps.

---

By incorporating the SRR-Judge framework into the DeepSeekR1 Professor Agent, search-integrated reasoning agents can move beyond outcome-based supervision, achieving fine-grained control, traceable step-quality measurement, and significantly stronger benchmark performance, as confirmed by independent evaluations and controlled ablation studies [2602.07773].

Source: https://www.emergentmind.com/topics/deepseekr1-professor-agent