Personalized In-Context Operator (PICO)

• PICO is a framework that learns image-to-image transformations from a single exemplar pair, enabling personalized vision tasks like segmentation and stylization.
• It uses a four-panel grid with a diffusion transformer backbone to transfer visual relations from an exemplar to a new query image in one forward pass.
• Empirical results demonstrate robust generalization across tasks, outperforming specialized models by leveraging many-tasks–few-shots learning on the VisRel dataset.

Personalized In-Context Operator (PICO) denotes a visual in-context learning framework in which a model infers an image-to-image transformation from a single exemplar pair $(A \to A')$ and applies the same transformation to a new image $B$, producing $B'$ without retraining. In its introduced form, PICO repurposes a diffusion transformer as a visual in-context learner, uses a four-panel grid to represent exemplar, query, and target, and is trained to generalize across both recognition and generation tasks (Jiang et al., 29 Sep 2025). More broadly, related operator-learning work describes a common pattern in which a single model conditions on examples, prompts, or histories to instantiate task-specific operators at inference time rather than via per-task weight updates (Yang et al., 2023).

1. Definition and conceptual scope

PICO was introduced for personalized vision, meaning vision tasks defined by a user at test time and grounded in a user’s objects, styles, or goals. The task specification is a single visual demonstration: an input-output pair $(A, A')$. Given a new image $B$, the system must infer the underlying transformation $r$ from the exemplar and produce $B' \approx r(B)$. The framework is designed to cover both predefined and novel operators, including composite tasks and user-invented transformations, rather than only fixed task formats such as segmentation or retrieval (Jiang et al., 29 Sep 2025).

The central operator is written as

$\mathcal{O}_{\Theta} : (A, A', B) \longmapsto B',$

where $A \to A'$ defines the relation to be transferred and $B$ is the query image. Because outputs are represented as RGB images, the same operator can express segmentation masks, dense prediction maps, stylized or edited images, and other image-to-image transformations. This unification is important: PICO does not introduce separate heads for segmentation, depth, stylization, or inpainting, but instead treats them as different regions of a common visual relation space (Jiang et al., 29 Sep 2025).

Within the broader literature, this formulation aligns with in-context operator learning: a single model learns to infer an operator from contextual examples and apply it in one forward pass. In ICON, for example, a single encoder-decoder Transformer learns operators for ODEs, PDEs, and mean-field control from prompted demos without weight updates (Yang et al., 2023). A plausible implication is that PICO is a vision-specific instantiation of the same operator-learning principle.

2. Four-panel operator and diffusion architecture

PICO structures its input as a $B$0 quad-grid,

$B$1

with $B$2 in the top-left, $B$3 in the top-right, $B$4 in the bottom-left, and $B$5 in the bottom-right. During training all four panels are observed; at test time the bottom-right panel is replaced by a placeholder initialized with Gaussian noise in latent space, while the other three panels remain clean and serve as context. This arrangement forces the model to read the relation in the exemplar pair and apply a consistent relation from $B$6 to the target quadrant (Jiang et al., 29 Sep 2025).

The backbone is FLUX.1-dev, a DiT-based rectified flow model. Let $B$7 and $B$8 denote the VAE encoder and decoder. The target latent is $B$9, the visual prompt is $B'$0, and text conditioning is $B'$1. At time $B'$2, the noised target latent is

$B'$3

and the model processes

$B'$4

through multi-modal attention. A key design choice is that only the target latent is noised; the visual and textual conditions remain clean. The paper states that, unlike In-Context LoRA, this prevents corruption of exemplar information and yields stable relation transfer (Jiang et al., 29 Sep 2025).

Training uses conditional flow matching on the target quadrant,

$B'$5

and inference integrates

$B'$6

from $B'$7 to $B'$8, then decodes $B'$9. Mechanistically, early denoising uses cross-attention to read the relation $(A, A')$0 and the content of $(A, A')$1, while the clean-noise split enforces that the remaining quadrants function as fixed in-context exemplars rather than co-denoised inputs (Jiang et al., 29 Sep 2025).

3. VisRel and the learned visual relation space

PICO is trained on VisRel, a compact but highly diverse dataset of 27 visual tasks represented as exemplar pairs $(A, A')$2. The full training set contains 315 samples across those tasks, including only 40 segmentation samples. Tasks are organized into four families: restoration/enhancement, physical/geometric perception, semantic perception, and generative manipulation. The dataset is explicitly designed to span a “visual relation space” along both semantic complexity and spatial locality (Jiang et al., 29 Sep 2025).

The training construction is relation-centric. A relation $(A, A')$3 is sampled, then two independent inputs $(A, A')$4 and $(A, A')$5 are drawn, $(A, A')$6 and $(A, A')$7 are computed, and the result is placed into the quad-grid. The paper emphasizes two design choices: intra-task diversity, such as multiple noise levels, mask shapes, and style variants, and minimal text labels, such as “Segment,” “Estimate depth,” or “Edit,” which disambiguate visually ambiguous relations (Jiang et al., 29 Sep 2025).

A central empirical claim is that task diversity, rather than scale, drives robust generalization. Under fixed model capacity, increasing shots per task helps in-domain performance, but out-of-domain performance improves monotonically with more task types and can decline when shots per task become too large. Under a fixed total image budget, the paper reports that a many-tasks–few-shots regime outperforms a few-tasks–many-shots regime for out-of-domain generalization. This positions VisRel less as a large corpus and more as a basis for learning reusable visual relations (Jiang et al., 29 Sep 2025).

Training uses LoRA with rank 256 on FLUX.1-dev, a 1024×1024 grid with 512×512 per cell, the Prodigy optimizer with weight decay 0.01, and 30,000 steps on a single H100 GPU. The minimal scale of the tuning set relative to the backbone is part of the intended argument: the model relies on pretrained diffusion-transformer structure, while VisRel teaches it how to read and transfer relations in context (Jiang et al., 29 Sep 2025).

4. Inference dynamics and attention-guided seed scoring

Because diffusion sampling is stochastic, different random seeds can produce materially different outputs, especially on local or deterministic tasks such as segmentation or edge prediction. PICO addresses this with an attention-guided seed scorer that probes several candidate seeds using only a few early solver steps and selects the most promising one before committing to full denoising (Jiang et al., 29 Sep 2025).

The diagnostic signal comes from attention flow in the bottom-right quadrant. Good seeds show increasing self-attention from target queries to target keys and decreasing reliance on visual context during early denoising. Let $(A, A')$8 denote attention mass from bottom-right queries to bottom-right keys, and $(A, A')$9 the mass from bottom-right queries to visual prompt keys, for seed $B$0, block $B$1, and probe step $B$2. Over critical blocks $B$3, the pivot statistics are

$B$4

$B$5

and the seed score is

$B$6

The deployed procedure probes 10 seeds, runs only 3 early steps, evaluates blocks $B$7, and continues only the best seed (Jiang et al., 29 Sep 2025).

This inference scaling method is called PICO+TTS. The paper reports that it improves personalized segmentation metrics on all datasets and qualitatively produces outputs more closely aligned with exemplars. The interpretation is narrowly operational: PICO is not only a learned operator but also an operator whose reliability depends on managing the stochasticity of diffusion inference (Jiang et al., 29 Sep 2025).

5. Empirical scope, benchmarks, and capabilities

PICO is evaluated on personalized segmentation benchmarks PerSeg, DOGS, PODS, and PerMIS, using mIoU, bIoU, and [email protected]. It is reported to strongly outperform generalist ICL models such as Visual Prompting, Painter, LVM, and OmniGen, and to outperform personalized representation methods such as PDM, PDM+PerSAM, and PRPG on difficult datasets such as PODS and PerMIS. Relative to large-scale segmentors, PICO is competitive with SegGPT and PerSAM, and is often second-best or best depending on metric, while using 40 segmentation examples within 315 total VisRel samples, versus 11M labeled segmentation masks for PerSAM and 138k–254k segmentation data for SegGPT and Painter (Jiang et al., 29 Sep 2025).

Test-time seed scoring yields measurable gains. On PODS, mIoU improves from 68.72 to 69.90, and bIoU improves from 60.26 to 63.60 under PICO+TTS. The paper presents this as a reliability improvement rather than a change in training objective (Jiang et al., 29 Sep 2025).

The operator is also evaluated on composite user-defined tasks. For deraining with inpainting, PICO achieves PSNR 22.24 dB and SSIM 0.67, outperforming OmniGen, LVM, Visual Prompting, Analogist, PromptDiffusion, InstaManip, and GPT-4o. For inpainting with stylization, it achieves Gram 21.27, LPIPS 0.52, and ArtFID 4.38, with the latter matching the “reference” baseline. Qualitatively, the paper highlights watermark removal with style transfer, background-only stylization, contour-only edge detection, and sticker insertion at correct semantic locations and scale (Jiang et al., 29 Sep 2025).

A recurring misconception is that PICO is only a segmentation method or only a generative editing method. The evidence in the paper runs against both reductions: the same model is used for personalized instance segmentation, dense prediction, inpainting, style transfer, relighting, sticker insertion, and composite operations, all expressed as image-to-image transfer from a single exemplar pair (Jiang et al., 29 Sep 2025).

6. Broader operator-learning lineage, adjacent personalization methods, and limitations

Related work situates PICO within a larger family of in-context operator systems. ICON trains a single Transformer to infer and apply new differential-equation operators from data prompts; across 19 problems, average relative errors are <6% with 1 demo and are typically around ~2% with 5 demos, without any weight update at inference (Yang et al., 2023). GICON extends the same principle to graph-structured spatiotemporal forecasting and shows that multi-operator in-context learning can outperform classical single-operator learning on more complex horizons, while scaling from few training examples to 100 at inference (Wu et al., 13 Mar 2026). On probability measures, in-context operator learning for optimal transport formalizes a single solution operator that maps prompt samples from source and target distributions to an OT map, with nonparametric generalization bounds and Gaussian-family finite-sample excess-risk results (Cole et al., 15 Jan 2026).

Personalization-oriented language-model work points to analogous constructions. Preference Pretrained Transformer (PPT) uses an offline phase with a history-dependent loss and an online phase in which the model adapts to user preferences through in-context learning in a contextual bandit setting (Lau et al., 2024). POPI distills heterogeneous user signals into concise natural-language summaries that condition a shared generation model, enabling plug-and-play personalization without weight updates (Chen et al., 17 Oct 2025). PICLe selects persona examples by a likelihood-ratio criterion for persona in-context learning (Choi et al., 2024), while PICACO optimizes a pluralistic value meta-instruction by maximizing total correlation between target values and model responses without fine-tuning (Jiang et al., 22 Jul 2025). This suggests a broader interpretation in which PICO is one member of a larger class of prompt-conditioned operators for personalization.

The visual formulation of PICO nevertheless has explicit limits. The paper identifies limited visual-relation coverage, a four-panel context that supports only one exemplar pair and one query per grid, dependence on a powerful diffusion transformer prior, and the computational cost of diffusion sampling. Proposed extensions include richer contexts, multiple exemplars, video sequences, long-context models, expansion of VisRel, and exploration of non-diffusion backbones (Jiang et al., 29 Sep 2025).

The acronym also requires disambiguation. In clinical NLP, PICO commonly denotes Population, Intervention, Comparator, and Outcome; AlpaPICO is an LLM framework for extracting those PICO frames from clinical trial documents, which is unrelated to the personalized visual operator defined here (Ghosh et al., 2024).