VisRel: Tuning Dataset for Personalized Vision

• VisRel is a compact dataset for personalized vision that defines tasks as image-to-image transformations using a single annotated exemplar.
• It comprises 27 diverse vision tasks and 315 exemplar pairs across restoration, geometric, semantic, and generative manipulation, organized in a quad-grid format.
• Empirical findings indicate that diverse, few-shot tasks boost out-of-domain generalization, achieving state-of-the-art performance with far fewer training examples.

{"query":"arXiv VisRel Personalized Vision via Visual In-Context Learning (Jiang et al., 29 Sep 2025)", "max_results": 5} VisRel is a compact yet diverse tuning dataset introduced in “Personalized Vision via Visual In-Context Learning” for personalized vision via visual in-context learning. It is designed around the premise that a single annotated exemplar can specify a user-defined visual transformation at test time, and it operationalizes that premise through exemplar image-to-image relations rather than fixed label spaces. In the reported formulation, VisRel comprises $N_{\text{tasks}}=27$ distinct vision tasks and $S_{\text{total}}=315$ exemplar pairs, organized to support PICO, a four-panel framework that repurposes diffusion transformers as visual in-context learners (Jiang et al., 29 Sep 2025).

1. Definition and research setting

VisRel is situated in the personalized vision setting described in the PICO work: modern vision models, trained on large-scale annotated datasets, excel at predefined tasks but struggle with tasks defined at test time by users with customized objects or novel objectives. Existing personalization approaches are characterized there as relying on costly fine-tuning or synthetic data pipelines, and prior visual in-context learning methods are described as confined to narrow, in-domain tasks. VisRel is introduced as the tuning dataset that supports a different regime: given a single annotated exemplar, PICO infers the underlying transformation and applies it to new inputs without retraining (Jiang et al., 29 Sep 2025).

A central property of VisRel is that it treats “relation” as an image-to-image transformation demonstrated by example. This differs from conventional supervised task definitions in which the model is trained for a fixed output ontology such as segmentation masks, depth maps, or object categories. In VisRel, the operative signal is the exemplar relation $A\rightarrow A'$, which is then transferred to a new query $B$. This suggests a shift from task-specific supervision toward task specification by demonstration.

2. Composition, scale, and task taxonomy

VisRel comprises $27$ distinct vision tasks and $315$ exemplar pairs $(A\rightarrow A')$, averaging $\sim 12$ pairs per task. In training, each exemplar pair is paired with one query image $B$ and its ground-truth output $B'$, yielding $S_{\text{total}}=315$0 quad-grid training samples. The dataset is organized into four semantic groups spanning low-level restoration through generative manipulation (Jiang et al., 29 Sep 2025).

Category	Tasks	Sample tasks
Restoration / Enhancement	5	Deblurring, Deraining
Physical / Geometric Perception	9	SurfaceNormal, 2DKeypoints
Semantic Perception	7	InstanceSegmentation, ObjectDetection
Generative Manipulation	6	Inpainting, StyleTransfer

The category structure is explicitly intended to cover a spectrum from low-level pixel edits up to high-level object- and scene-based transformations. The dataset therefore mixes tasks that differ not only in semantics but also in output type: clean-image restoration, geometric map estimation, discrete structured outputs such as masks or boxes, and open-ended image generation. A plausible implication is that the benchmark is less a collection of homogeneous tasks than a deliberately heterogeneous task family for studying cross-task transfer in visual in-context learning.

3. Task formulation as image-to-image relations

Every VisRel task is cast as an exemplar image-to-image transformation $S_{\text{total}}=315$1. At training time, PICO sees a $S_{\text{total}}=315$2 quad-grid

$S_{\text{total}}=315$3

where $S_{\text{total}}=315$4 is the input example, $S_{\text{total}}=315$5 is the desired output for $S_{\text{total}}=315$6, $S_{\text{total}}=315$7 is a new query input, and $S_{\text{total}}=315$8 is the ground-truth output for $S_{\text{total}}=315$9 (Jiang et al., 29 Sep 2025).

This formulation makes the dataset task-agnostic at the interface level: the same four-panel structure is used whether the underlying operation is deblurring, surface-normal estimation, instance segmentation, object detection, inpainting, or style transfer. The primary learning signal is the visual exemplar pair. A single word prompt such as “enhance,” “estimate,” or “segment” is used only to disambiguate among visually similar operations, and the paper states that text is minimally relied on. In that sense, VisRel is neither a purely textual instruction-tuning resource nor a conventional per-task dataset; it is a collection of visually specified input–output correspondences.

4. Construction principles, annotation procedure, and data sources

Three principles guided the construction of VisRel. First, task taxonomy: the dataset spans semantic complexity from low-level to mid-level to high-level and spans spatial locality from local pixel to patch to full-image. Second, intra-task diversity: each task uses varied exemplars, with examples in the paper summary including inpainting masks of different shapes, colors, and transparencies; denoising with multiple noise levels; and edge detection under different styles. Third, minimal text labels: a single word prompt disambiguates among visually similar operations, while the primary learning signal remains the visual exemplar pair (Jiang et al., 29 Sep 2025).

VisRel is assembled entirely from public datasets, with no large-scale new annotation cost. The reported sources include DIV2K for deblurring and denoising, Dense-Haze and LOL for enhancement, Taskonomy for geometric maps such as normals, depth, curvature, and edges, MS-COCO for instance, panoptic, and semantic segmentation, and PhotoDoodle and LineArt Colorization for stylization and manipulation. Human-labeled lightweight annotations are used for a small fraction of cases, specifically CORe50 images for bounding-box detection and sticker-placement exemplars positioned by hand. In total, the $A\rightarrow A'$0 paired exemplars are sampled uniformly across the $A\rightarrow A'$1 tasks, approximately $A\rightarrow A'$2–$A\rightarrow A'$3 examples per task.

The construction procedure is therefore notable less for scale than for curation. The paper’s emphasis on uniform sampling across a diverse task set indicates that VisRel is intended as a balanced tuning resource rather than a maximal-coverage benchmark.

5. Role in PICO and empirical findings

Once VisRel is constructed, PICO is finetuned with LoRA on the $A\rightarrow A'$4 pairs. Evaluation is reported along two fronts. For personalized recognition, the benchmarks are PerSeg, DOGS, PODS, and PerMIS, with metrics mIoU, boundary-IoU, and [email protected]. For novel test-time tasks, the reported composites include derain + inpaint, inpaint + stylize, background-only style, contour-only edge, and sticker insertion; the metrics include PSNR and SSIM for deraining + inpainting, and Gram distance, FID, LPIPS, and ArtFID for inpainting + stylization. The reported outcome is that PICO matches or exceeds state-of-the-art personalized and generalist segmentors while using $A\rightarrow A'$5–$A\rightarrow A'$6 orders of magnitude fewer training examples $A\rightarrow A'$7, and is uniformly best or second-best across all five metrics on the novel composite tasks (Jiang et al., 29 Sep 2025).

A central empirical claim associated with VisRel is that “task diversity > data scale” for out-of-domain generalization. The paper studies three regimes: varying shots per task $A\rightarrow A'$8 at fixed $A\rightarrow A'$9 tasks, varying number of tasks $B$0 at fixed $B$1 shots, and a budgeted regime with total images $B$2 fixed. Let $B$3 denote performance on unseen tasks. The reported findings are that $B$4 grows monotonically with $B$5 at fixed $B$6, peaks around moderate $B$7 and then declines if $B$8 grows too large, and under fixed $B$9, the many-tasks–few-shots setting outperforms few-tasks–many-shots. This positions VisRel not merely as a training set but as empirical evidence for a particular scaling law in visual in-context learning.

6. Relation to other “visual relation” literatures

Despite its name, VisRel is not a classical visual relationship detection or visual relationship recognition dataset. In the earlier visual relationship literature, a relation is typically a triplet $27$0, with work focusing on pairwise relation classification, spatial encoding, weak supervision, long-tail recognition, or future interaction prediction (Peyre et al., 2017). Representative directions include relative-location mining for predicate recognition (Zhou et al., 2019), Transformer-based long-tail visual relationship recognition (Chen et al., 2021), visual relationship forecasting in videos (Mi et al., 2021), unsupervised visual relations discovery with graph-level analogy under the name ViRel (Zeng et al., 2022), and the visually-relevant relationship curation strategy of VrR-VG (Liang et al., 2019).

VisRel departs from that triplet-based ontology. Its unit of supervision is the exemplar relation $27$1, not a predicate over a subject–object pair. Accordingly, its tasks include restoration, estimation, segmentation, detection, inpainting, stylization, and other manipulations that are not naturally represented as a single predicate label. This distinction matters because the term “visual relation” can otherwise suggest scene-graph or predicate-learning benchmarks. In current usage, VisRel refers specifically to a compact tuning dataset for visual in-context learning in personalized vision, whereas nearby names such as ViRel and VrR-VG belong to different research programs with different formal objects, supervision regimes, and evaluation protocols (Jiang et al., 29 Sep 2025).