Synergistic Editor-Evaluator Policy Optimization (SEPO)

• The paper introduces SEPO, a dual-loop RL framework that integrates an editor and evaluator to optimize on-policy photo editing via intrinsic pairwise reward signals.
• It employs an interleaved multimodal chain-of-thought and Adobe Lightroom tool execution to achieve stable self-improvement and enhanced content fidelity with an 18.95% average improvement over baselines.
• The evaluator is continuously calibrated using human annotations and selective loss masking to mitigate reward hacking and ensure alignment with human judgments.

Synergistic Editor–Evaluator Policy Optimization (SEPO) is the reinforcement learning core of JarvisEvo, a unified image editing agent that turns a single vision–LLM into both a self-improving photo editor and a trustworthy evaluator. Its stated goal is to enable on-policy, multimodal, instruction-following photo retouching that improves without external reward models, while curbing reward hacking through a co-adaptive optimization of the editor and evaluator. In JarvisEvo, SEPO is coupled to an interleaved multimodal chain-of-thought (iMCoT) loop and Adobe Lightroom tool execution, and is reported to yield stable self-improvement, strong content fidelity, and human-aligned instruction following; on ArtEdit-Bench, JarvisEvo outperforms Nano-Banana by an average of 18.95% on preservative editing metrics, including a 44.96% improvement in pixel-level content fidelity (Lin et al., 28 Nov 2025).

1. Definition and distinguishing characteristics

SEPO comprises two interlocking RL loops. The first is an editor-policy optimization loop in which the model performs iMCoT editing, executes Lightroom tools, and generates a self-evaluation score. These scores are converted into a pairwise win-rate reward among $G$ sampled trajectories per instruction, and the editor is optimized by GRPO using these intrinsic rewards. The second is an evaluator-policy optimization loop in which the same model is trained to produce calibrated self-evaluation scores given the full editing context, using human-annotated target scores; an alignment reward encourages consistency with human judgments (Lin et al., 28 Nov 2025).

The framework is described as marrying internal feedback (RLIF) with verifiable reinforcement (RLVR). The editor is optimized using intrinsic self-evaluation signals and pairwise preferences, and the evaluator is simultaneously calibrated on human-annotated assessment data. This two-loop design makes the reward model dynamic and grounded, reducing the exploitability associated with static reward pipelines.

SEPO is explicitly contrasted with three prior paradigms. Relative to conventional policy gradients or actor–critic methods, it replaces a fixed reward function or learned critic with a co-adaptive evaluator and prefers relative pairwise rewards to absolute scores. Relative to preference learning approaches such as DPO and Bradley–Terry, it uses online, group-relative preferences computed across the current policy’s $G$ candidates rather than offline human comparisons and static reward surrogates. Relative to RLHF with a frozen reward model, it continuously updates the evaluator with human-labeled data, while using selective loss masking (SLM) and pairwise rewards to mitigate self-deception. The JarvisEvo ablations report that removing the evaluator loop or SLM causes rising self-scores but falling actual performance, which the paper identifies as classic reward hacking (Lin et al., 28 Nov 2025).

2. Formal problem formulation in multimodal photo editing

JarvisEvo models instruction-based photo editing as a partially observable, multimodal MDP with tool execution. The source image and instruction are

$$x_0 = I \in \mathcal{X}, \qquad c = Q \in \mathcal{C}.$$

At step $t$, the state aggregates the evolving visual and textual context:

$s_t = (x_t, c, H_t),$

where

$$H_t = \{[C_0, T_0, O_0], \ldots, [C_{t-1}, T_{t-1}, O_{t-1}], C_t\}$$

contains text reasoning $C$, tool calls $T$, and intermediate images $O$ (Lin et al., 28 Nov 2025).

The action is

$a_t = T_t,$

which encodes Lightroom tool selection and parameter settings from a catalog of more than 200 tools via the A2L protocol. The action space is discrete–continuous hybrid: the model chooses tool name(s) and predicts numeric or string parameters. The transition is executed by a sandboxed Lightroom engine:

$G$0

where $G$1 is deterministic. The edited image $G$2 is fed back into the state and into iMCoT. After at most $G$3 steps, the model outputs a self-evaluation $G$4 consisting of a rationale and a score $G$5, and the episode ends.

The trajectory-level return is defined as

$G$6

Here $G$7 is a format reward enforcing designated parsable tags, $G$8 is a tool accuracy reward checking tool names and parameters, and $G$9 is a pairwise preference reward. The editor policy is

$$x_0 = I \in \mathcal{X}, \qquad c = Q \in \mathcal{C}.$$0

which generates both iMCoT textual tokens and tool-call tokens, while the evaluator is

$$x_0 = I \in \mathcal{X}, \qquad c = Q \in \mathcal{C}.$$1

or equivalently a reward model $$x_0 = I \in \mathcal{X}, \qquad c = Q \in \mathcal{C}.$$2 used as an alignment signal during evaluator optimization. The editing and evaluation trajectories are

$$x_0 = I \in \mathcal{X}, \qquad c = Q \in \mathcal{C}.$$3

and

$$x_0 = I \in \mathcal{X}, \qquad c = Q \in \mathcal{C}.$$4

with $$x_0 = I \in \mathcal{X}, \qquad c = Q \in \mathcal{C}.$$5 (Lin et al., 28 Nov 2025).

3. Objectives, rewards, and alternating optimization

SEPO’s editor reward is built from formatting, tool correctness, and group-relative preference. For each instruction $$x_0 = I \in \mathcal{X}, \qquad c = Q \in \mathcal{C}.$$6, the model samples $$x_0 = I \in \mathcal{X}, \qquad c = Q \in \mathcal{C}.$$7 rollouts $$x_0 = I \in \mathcal{X}, \qquad c = Q \in \mathcal{C}.$$8 with self-evaluation scores $$x_0 = I \in \mathcal{X}, \qquad c = Q \in \mathcal{C}.$$9. The pairwise preference reward is defined as the win rate

$t$0

which converts raw scalar self-scores into relative preferences in order to stabilize learning and reduce threshold gaming (Lin et al., 28 Nov 2025).

The editor is optimized with GRPO. If $t$1, then standardized advantages per token are

$t$2

The editor objective is

$t$3

with the gradient estimator corresponding to REINFORCE with group-standardized advantages:

$t$4

A central stabilizer is SLM, which excludes the tokens in $t$5 from the loss. The stated reason is to prevent leakage of reward-target information into the policy logits, which otherwise produces overfitting and collapse. JarvisEvo also omits the KL penalty against a reference model, unlike common GRPO implementations, to encourage faster adaptation to the custom response format and reward signals. No entropy bonuses or trust-region constraints are reported; stability is attributed to group-standardization, pairwise rewards, SLM, and evaluator calibration (Lin et al., 28 Nov 2025).

The evaluator loop has its own reward and GRPO objective. In addition to the same format reward $t$6, it uses score alignment

$t$7

with $t$8 and $t$9 small for non-zero gradients. The evaluator objective mirrors the editor’s:

$s_t = (x_t, c, H_t),$0

The losses are

$s_t = (x_t, c, H_t),$1

and training alternates between maximizing $s_t = (x_t, c, H_t),$2 and $s_t = (x_t, c, H_t),$3 on separate mini-batches. Conceptually, the paper presents SEPO as a dual-loop alternating optimization,

$s_t = (x_t, c, H_t),$4

in which $s_t = (x_t, c, H_t),$5 affects how self-scores are produced and thus how pairwise rewards behave, while human labels prevent evaluator drift (Lin et al., 28 Nov 2025).

4. Edit–evaluate–reflect–refine synergy

SEPO unfolds within JarvisEvo’s iMCoT loop as an edit–evaluate–reflect–refine process. During editing, the agent writes multimodal reasoning in >, calls Lightroom tools in <tool_call>, receives the edited image $s_t = (x_t, c, H_t),$6, and updates the state. During evaluation, it generates $s_t = (x_t, c, H_t),$7 with an overall score covering aesthetic quality and instruction adherence. During training, evaluator GRPO aligns those scores with human annotations; during editor GRPO, the same scores feed the pairwise preference reward. This coupling is the mechanism by which the editor’s self-reward remains meaningful while the evaluator’s scoring behavior remains grounded (Lin et al., 28 Nov 2025).

Reflection is introduced on-policy. When one trajectory, such as $s_t = (x_t, c, H_t),$8, wins over another, such as $s_t = (x_t, c, H_t),$9, meaning $$H_t = \{[C_0, T_0, O_0], \ldots, [C_{t-1}, T_{t-1}, O_{t-1}], C_t\}$$0, a role-play prompt asks Gemini-2.5-Pro to analyze $$H_t = \{[C_0, T_0, O_0], \ldots, [C_{t-1}, T_{t-1}, O_{t-1}], C_t\}$$1 and produce a reflection rationale $$H_t = \{[C_0, T_0, O_0], \ldots, [C_{t-1}, T_{t-1}, O_{t-1}], C_t\}$$2 explaining the error and the correct path. Refinement then takes place through Stage 3 reflection fine-tuning (RFT), where the agent is trained on

$$H_t = \{[C_0, T_0, O_0], \ldots, [C_{t-1}, T_{t-1}, O_{t-1}], C_t\}$$3

with the stated purpose of strengthening error detection and self-correction (Lin et al., 28 Nov 2025).

The implementation details reported for this pipeline are specific. The base architecture is Qwen3-VL-8B-Instruct. Stage 1 SFT uses 150K examples, consisting of 110K ArtEdit-Lr and 40K ArtEdit-Eval, for 2 epochs with batch size 2 and learning rate $$H_t = \{[C_0, T_0, O_0], \ldots, [C_{t-1}, T_{t-1}, O_{t-1}], C_t\}$$4 via Llama-Factory. Stage 2 SEPO trains the editor loop on 10K ArtEdit-Lr and the evaluator loop on 10K ArtEdit-Eval for 1 epoch with batch size 1, learning rate $$H_t = \{[C_0, T_0, O_0], \ldots, [C_{t-1}, T_{t-1}, O_{t-1}], C_t\}$$5, and 4 responses per query via vlm-r1. Stage 3 RFT uses 5K reflection samples collected on-policy for 1 epoch with learning rate $$H_t = \{[C_0, T_0, O_0], \ldots, [C_{t-1}, T_{t-1}, O_{t-1}], C_t\}$$6. The reported compute budget is 32 A100 GPUs. Tool integration includes more than 200 Adobe Lightroom tools via A2L, supporting both global and localized masks across tone, HSL, split toning, color grading, detail, effects, calibration, lens blur, advanced point colors, look presets, and localized corrections (Lin et al., 28 Nov 2025).

5. Empirical evaluation, ablations, and reward-hacking mitigation

The main editing benchmark is ArtEdit-Bench-Lr, with 800 samples split into 400 English and 400 Chinese instances. The reported metrics are pixel fidelity $$H_t = \{[C_0, T_0, O_0], \ldots, [C_{t-1}, T_{t-1}, O_{t-1}], C_t\}$$7, $$H_t = \{[C_0, T_0, O_0], \ldots, [C_{t-1}, T_{t-1}, O_{t-1}], C_t\}$$8, instruction alignment $$H_t = \{[C_0, T_0, O_0], \ldots, [C_{t-1}, T_{t-1}, O_{t-1}], C_t\}$$9, perceptual quality $C$0, and overall $C$1, where $C$2 is the geometric mean of $C$3 and $C$4. Evaluator assessment uses ArtEdit-Bench-Eval with 200 English samples and reports $C$5 and $C$6 versus human ratings (Lin et al., 28 Nov 2025).

For JarvisEvo, the reported English results are $C$7, $C$8, $C$9, $T$0, and $T$1. The reported Chinese results are $T$2, $T$3, $T$4, $T$5, and $T$6. Relative to Nano-Banana, the average improvement is 18.95% across $T$7 and $T$8, and for content fidelity $T$9 the gain is 44.96% on average. Evaluator alignment reaches $O$0 and $O$1, surpassing task-specific IQA MLLMs and general MLLMs including Gemini-2.5-Flash and Qwen3-VL variants. In the human preference study, JarvisEvo obtains a 49% win rate versus Nano-Banana’s 28%, and is reported as better than GPT-Image, Bagel, UniWorld-v1, and Step1X-Edit (Lin et al., 28 Nov 2025).

The ablations isolate the contribution of SEPO’s safeguards:

Variant	Reported metrics
SFT only	$O$2, $O$3, $O$4, $O$5, $O$6
+ SEPO without evaluator loop	$O$7, $O$8, $O$9, $a_t = T_t,$0, $a_t = T_t,$1
+ SEPO without SLM	$a_t = T_t,$2, $a_t = T_t,$3, $a_t = T_t,$4, $a_t = T_t,$5, $a_t = T_t,$6
+ SEPO without both	$a_t = T_t,$7, $a_t = T_t,$8, $a_t = T_t,$9, $G$00, $G$01
Full SEPO	$G$02, $G$03, $G$04, $G$05, $G$06
Full SEPO + RFT	$G$07, $G$08, $G$09, $G$10, $G$11

These results are interpreted in the paper as evidence that removing the evaluator loop or SLM raises self-predicted scores while degrading $G$12 and $G$13, indicating reward hacking and training collapse. A related comparison replaces intrinsic co-adaptive scoring with Gemini-2.5-Pro as a static external evaluator, yielding $G$14, $G$15, $G$16, $G$17, and $G$18; the paper states that this performs similarly early on but diverges later, with self-scores increasing while editing metrics degrade. Another comparison replaces pairwise preference reward with absolute score reward, yielding $G$19, $G$20, $G$21, $G$22, and $G$23, whereas SEPO’s pairwise reward gives $G$24, $G$25, $G$26, $G$27, and $G$28; the paper attributes the difference to larger reward variance, win-rate polarization, and improved stability (Lin et al., 28 Nov 2025).

6. Relation to adjacent frameworks, acronym ambiguity, and limitations

SEPO belongs to a broader family of co-evolving generator–judge or editor–evaluator schemes, but the exact acronym is not unique in the literature. JarvisEvo uses “Synergistic Editor-Evaluator Policy Optimization” for multimodal photo editing (Lin et al., 28 Nov 2025). By contrast, “SePO: Self-Evolving Prompt Optimization” uses the same letters for a self-referential, archive-based evolutionary framework in which a prompt agent edits both task agents’ system prompts and its own system prompt; that work maps its components to an Editor and an Evaluator, with the Evaluator implemented as scoring, selection, and archive management rather than a GRPO-trained scorer (Tao et al., 3 Jun 2026). “Safe Equilibrium Policy Optimization” is a different SEPO altogether, defined for strategic agent policies and based on a safety-aware objective

$G$29

not on self-evaluated editing trajectories (Arumugam et al., 29 May 2026).

Two other 2025 frameworks are explicitly presented through a similar editor–evaluator lens. URPO, “Unified Reward & Policy Optimization,” merges instruction-following and reward modeling inside a single model and a single GRPO loop; it is described as training a player and referee concurrently and as eliminating the need for a separate reward model, with improvements from 42.24 to 44.84 on AlpacaEval and from 32.66 to 35.66 on the composite reasoning average, while its internal evaluator reaches 85.15 on RewardBench versus 83.55 for the dedicated reward model it replaces (Lu et al., 23 Jul 2025). SPARK, “Synergistic Policy And Reward Co-Evolving Framework,” similarly recycles RLVR rollouts and correctness signals so that the same model acts as both Editor and Evaluator; it reports, for example, that SPARK-VL-7B achieves an average 9.7% gain on 7 reasoning benchmarks, 12.1% on 2 reward benchmarks, and 1.5% on 8 general benchmarks over the baselines (Liu et al., 26 Sep 2025). These parallels suggest a broader methodological pattern in which a policy and its evaluator are co-trained rather than separated into fixed actor and reward-model components.

JarvisEvo’s limitations are stated narrowly and concretely. The current iMCoT horizon is capped around four steps, and scaling to workflows longer than ten steps is said to require more data and compute. Evaluator bias can persist from annotation guidelines and dataset composition, even though SEPO reduces it. Training stability depends on SLM, group size $G$30—set to 4 in the reported experiments—and $G$31 in $G$32, so hyperparameter mis-tuning can degrade performance. The method targets Lightroom-style preservative editing; cross-domain operations such as generative composition may require richer evaluators and tools. The two RL loops and reflection fine-tuning also add computational overhead. The future directions named in the paper include stronger evaluators such as ensembles and multi-dimensional criteria, multi-agent variants with separate editor and evaluator models, human-in-the-loop calibration, broader task coverage in code, math, and long-horizon planning, and longer iMCoT horizons (Lin et al., 28 Nov 2025).