Instruction-Visual Complexity

• IV-Complexity is defined by the joint challenges of visual clutter (e.g., multiple objects, non-dominant scenes) and complex instructions (e.g., multi-object references, indirect descriptions).
• It is operationalized using domain-specific measures such as atomic edit counts, hierarchical task taxonomies, and event-based proxies in dynamic graphs to capture reasoning and binding difficulties.
• Measurement frameworks range from simple count proxies to advanced instruction-conditioned visual feature models, emphasizing the need for scalable and interpretable complexity assessments.

Instruction–Visual Complexity (IV-Complexity) denotes the intrinsic challenge that arises from the interplay between visual complexity and instructional complexity, especially when instructions must be grounded in cluttered scenes, multiple similar objects, indirect references, or knowledge-intensive and compositional tasks (Qu et al., 18 Dec 2025). Although the term is explicit only in some recent work, related research converges on a broader view in which complexity is jointly shaped by visual structure, reasoning depth, temporal composition, response space, layout organization, and the amount of guidance needed to preserve interpretability across sequential or interactive views (Lei et al., 2024, Zhang et al., 2 Feb 2026, Yang et al., 17 Apr 2025, Windhager et al., 2024).

1. Conceptual scope

IV-Complexity is most explicitly framed as the interaction between two dimensions. On the visual side, the relevant factors include cluttered layouts, many objects, multiple instances of the same category, and non–subject-dominated scenes. On the instructional side, the relevant factors include multi-object references, indirect or implicit descriptions, world-knowledge requirements, causal or temporal reasoning, and compositional instructions (Qu et al., 18 Dec 2025). This framing emphasizes that difficulty is not reducible to either the image or the instruction alone; it emerges when the two must be aligned precisely.

Several adjacent literatures instantiate the same idea with different terminology. In iWISDM, complexity is structured through logical compositionality, temporal compositionality, operator counts, graph depth, response-space size, and visual attribute load, even though the paper does not name this IV-Complexity (Lei et al., 2024). In VIBE, visual instruction following is organized into a three-level hierarchy—deictic grounding, morphological manipulation, and causal reasoning—where each level adds a qualitatively new reasoning requirement (Zhang et al., 2 Feb 2026). In Complex-Edit, complexity is operationalized by compounding multiple atomic image-editing instructions into a single coherent instruction, with the complexity level $C_i$ determined by the number of atomic instructions merged (Yang et al., 17 Apr 2025).

The concept also extends beyond image editing or multimodal reasoning. In dynamic graph visualization, visual complexity is operationalized as the number of edge events projected into a timeslice, and the central design problem becomes how to distribute that complexity across views (Wang et al., 2019). In search-as-learning, page-level visual complexity and aesthetics are studied as predictors of knowledge gain, with layout and aesthetic order emerging as more consequential than simple surface counts such as the number of images (Gritz et al., 9 Jan 2025). In information design more broadly, complexity is treated not merely as visual clutter but as a phenomenon distributed across initiation, datafication, transformation, visualization, interaction, interpretation, and communication (Windhager et al., 2024).

2. Major dimensions and operationalizations

Across the literature, IV-Complexity is not represented by a single universal scalar. Instead, it is decomposed into domain-specific factors that can be controlled, measured, or balanced.

Setting	Operationalization of complexity	Source
Dynamic graphs	Number of events or edges per timeslice	(Wang et al., 2019)
iWISDM	Logical joiners, Switch operators, frame horizon, output space, visual attribute load	(Lei et al., 2024)
Complex-Edit	Number of atomic edits merged into one instruction	(Yang et al., 17 Apr 2025)
VIBE	Deictic, morphological, and causal instruction levels	(Zhang et al., 2 Feb 2026)
RePlan / IV-Edit	Cluttered scenes plus intricate, knowledge-intensive, multi-region instructions	(Qu et al., 18 Dec 2025)

The dynamic-graph formulation is deliberately minimal: visual complexity is proxied by $|E_l|$, the number of edge events in a slice $G_l$, and the design goal is to produce timeslices with approximately equal numbers of events (Wang et al., 2019). This is a count-based view of visual burden. By contrast, iWISDM operationalizes complexity through task graphs defined over operators such as Select, Get*, Switch, IsSame, NotSame, And, and Or, with benchmark levels controlled by the number of logical joiners, the presence or absence of a Switch operator, the number of frames, and whether the output space is Boolean-only or includes category and location tokens (Lei et al., 2024).

Complex-Edit uses a simpler but highly controllable scale. A complex instruction is formed by compounding multiple atomic editing tasks, and the complexity level $C_i$ is the number of atomic instructions integrated into the compound instruction. This single control variable indirectly increases semantic diversity, compositional dependencies, spatial coverage, and transformation difficulty (Yang et al., 17 Apr 2025). VIBE uses a hierarchical task taxonomy instead: deictic tasks require grounding boxes or arrows to local operators; morphological tasks require interpreting sparse structural blueprints such as skeletons, drafts, or view frustums; causal tasks require inferring consequences from visual causes such as light direction, wind vectors, or billiard-force arrows (Zhang et al., 2 Feb 2026).

A broader decomposition, proposed in the iWISDM synthesis as a practical IV-Complexity formulation, makes the structure explicit. For a task instance $x = (I, V, G)$, a plausible aggregate is:

$$C_{\text{IV}}(x) = w_1 C_{\text{inst}}(I) + w_2 C_{\text{vis}}(V) + w_3 C_{\text{bind}}(G) + w_4 C_{\text{reason}}(G) + w_5 C_{\text{out}},$$

where the terms capture instruction structure, visual scene complexity, instruction–visual binding, reasoning depth and breadth, and response-space complexity (Lei et al., 2024). This suggests that IV-Complexity is best understood as a composite of partially separable burdens rather than a single monolithic property.

3. Measurement frameworks and proxies

The measurement literature shows a sharp contrast between simple operational proxies and richer structural measures. In dynamic graphs, the proxy is intentionally coarse: equalizing the number of projected edge events per snapshot is taken as a way to equalize visual complexity across small multiples (Wang et al., 2019). The advantage is direct controllability; the limitation is that the proxy ignores node count, crossings, layout entropy, and perceptual factors.

For visual materials more generally, several objective families of metrics have been proposed. In search-as-learning, visual complexity and aesthetics are operationalized through HTML features, screenshot-level visual features, VIPS-based layout features, and 14 Gestalt-inspired aesthetic features such as Balance, Equilibrium, Symmetry, Sequence, Cohesion, Unity, Proportion, Simplicity, Density, Regularity, Economy, Homogeneity, Rhythm, and Order and complexity (Gritz et al., 9 Jan 2025). That work reports that content relevance is the strongest predictor of knowledge gain, but that sessions characterized by less visually complex and more aesthetically ordered pages are associated with higher knowledge gain, especially through layout and aesthetic structure rather than simple counts (Gritz et al., 9 Jan 2025).

A different line of work treats visual complexity as a structural property of images across scales. The multi-scale structural complexity (MSSC) measure defines total complexity as

$\mathcal{C} = \sum_k \mathcal{C}_k,$

where $\mathcal{C}_k$ is the partial complexity contributed by differences between successive coarse-grained representations of the same image (Kravchenko et al., 2024). MSSC performs on par with or better than several traditional image-complexity measures on multiple categories, is easier to compute, and provides scale-wise analysis, but it also underperforms on symbolic categories such as art and infographics, where semantics and interpretation diverge from structural complexity (Kravchenko et al., 2024). This makes MSSC a strong candidate for the structural backbone of IV-Complexity, but not a full account of instructional difficulty.

Segmentation-based approaches move closer to object-level interpretation. One model explains perceived visual complexity with a simple linear function of two features derived from SAM and FC-CLIP: the square root of the number of segments and the square root of the number of semantic class instances, with patch symmetry added for datasets where repeated, structured elements would otherwise be over-predicted as complex (Shen et al., 2024). Visualization-specific work reaches a similar conclusion from another direction: across 1,800 visualization images, the number of corners and distinct colors are robust metrics across visualizations, feature congestion is strongest for continuous color- and texture-rich views, edge density effectively explains perceived complexity in node-link diagrams, and text-to-ink ratio exhibits a bell-curve effect in which moderate annotation reduces perceived complexity before excessive text increases it again (Chu et al., 9 Oct 2025).

Recent predictive models reinforce the same split. DReX shows that a vision-only fusion of DINOv3 and ResNet-50 can reach state-of-the-art performance on IC9600, with Pearson $r = 0.9581$, indicating that visual features alone can be sufficient for human-aligned complexity prediction on the image side (Skaza et al., 21 Nov 2025). By contrast, ScalSelect suggests an instruction-conditioned route: extract the visual features most attended by instruction tokens in a target VLM, build instruction-conditioned sample representations, and score their global informativeness through dominant-subspace leverage (Wu et al., 12 Feb 2026). This suggests that IV-Complexity can also be approached as a model-relative property of instruction-conditioned representational geometry.

4. Benchmarks and empirical regularities

Benchmark evidence is consistent on one central point: as compositional, branching, or causally grounded demands increase, current multimodal systems degrade markedly. In iWISDM, the Low, Medium, and High benchmarks progressively vary logical joiners, Switch operators, frame horizon, and output space. GPT-4V and Gemini-Pro show a clear inverse relationship between complexity level and accuracy, while human participants maintain high accuracy across all levels, between approximately $0.78$ and $|E_l|$0 (Lei et al., 2024). The same study reports that location-only tasks are often the most difficult for models, additional Boolean operators and Switch operators reduce accuracy, and delay frames alone have little effect, suggesting that compositional reasoning and binding, rather than raw short-term memory, are the main source of difficulty (Lei et al., 2024).

VIBE shows the same pattern in visual instruction-driven image editing. Across 10 tasks and 1,034 samples, top proprietary models perform best on deictic tasks, worse on morphological tasks, and worst on causal tasks. Nano Banana Pro, for example, averages 84.83 on Deictic, 65.46 on Morphological, and 45.17 on Causal, with especially poor performance on Billiards, where multi-bounce causal reasoning is required (Zhang et al., 2 Feb 2026). The benchmark’s hierarchy therefore functions as an empirical scale of increasing IV-Complexity, and the drop is monotonic even for the strongest proprietary systems (Zhang et al., 2 Feb 2026).

Complex-Edit finds a parallel trend when complexity is defined by the number of atomic edit operations merged into a single instruction. From $|E_l|$1 to $|E_l|$2, identity preservation consistently drops sharply across models, perceptual quality generally declines, and open-source models underperform proprietary models with the gap widening at higher complexity levels (Yang et al., 17 Apr 2025). The same study reports that decomposing a complex instruction into a step-by-step sequential editing process substantially degrades performance across multiple metrics, whereas a straightforward Best-of-$|E_l|$3 selection strategy improves both direct and sequential editing, although sequential editing rarely surpasses direct editing with $|E_l|$4 (Yang et al., 17 Apr 2025).

Instruction tuning results point in the same direction from the data side. ComVint argues that good visual instructions are those that emphasize complex visual reasoning rather than captioning or generic VQA. Fine-tuning on ComVint improves all compared MLLMs, including a 27.86% improvement for LLaVA on MME-Perception and a 27.60% improvement on MME-Cognition, and the empirical study finds that increasing instruction complexity is more useful than enhancing task diversity or adding fine-grained spatial annotations (Du et al., 2023).

RePlan and IV-Edit make the interactional definition explicit. IV-Edit targets cluttered real-world scenes and text-related image editing, with ~800 instruction–image pairs, 182 multi-region edits, and a taxonomy spanning feature-, spatial-, knowledge-, and understanding-based referring expressions together with 16 task types (Qu et al., 18 Dec 2025). On this benchmark, RePlan improves regional precision and consistency on top of strong MMDiT editors; for example, Flux.1 Kontext dev rises from Consistency 2.88 to 3.64 and Overall 3.22 to 3.46, while Qwen-Image-Edit rises from Consistency 1.79 to 3.24 and Weighted score 2.62 to 2.91 (Qu et al., 18 Dec 2025). These gains concentrate exactly where IV-Complexity is highest: disambiguation, multi-region grounding, and spillover control.

5. Design and algorithmic responses

A major theme of the literature is that IV-Complexity can be redistributed, scaffolded, or made explicit rather than merely reduced. In dynamic graphs, nonuniform timeslicing via histogram equalization “warps” time so that bursty periods are expanded and quiet periods are compressed, thereby balancing the visual information carried by each small multiple (Wang et al., 2019). The design principle is to allocate more representational bandwidth to periods with richer activity while explicitly encoding variable interval lengths so interpretability is not lost (Wang et al., 2019).

In instructional or search settings, the same logic appears as layout control. Search-as-learning results suggest “relevance first, aesthetics second”: content relevance is the strongest predictor of knowledge gain, but among pages of comparable relevance, simpler and more ordered aesthetic layouts are associated with higher learning success (Gritz et al., 9 Jan 2025). This treats visual complexity not as a binary good or bad but as a factor to be tuned in relation to the learning goal.

Image-editing systems increasingly address IV-Complexity through decomposition and explicit grounding. RePlan represents the editing task as a set of region–hint pairs $|E_l|$5, where $|E_l|$6 is the whole image with a global hint and $|E_l|$7 for $|E_l|$8 are local regions with local edit hints (Qu et al., 18 Dec 2025). A vision–language planner produces these region-aligned plans through chain-of-thought reasoning; a diffusion editor then applies a training-free attention-region injection mechanism so each region primarily attends to its own hint and the global hint (Qu et al., 18 Dec 2025). The planner is improved with a two-stage GRPO-based reinforcement-learning procedure, first for valid format and reasoning, then for image-level rewards over Target, Effect, and Consistency (Qu et al., 18 Dec 2025).

VIBE addresses the same problem on the evaluation side. Its metric design decomposes performance into interpretable subcriteria such as Instruction Adherence, Contextual Preservation, Visual Coherence, Pose Consistency, Orientation Alignment, Lighting Direction Consistency, and Path Correctness, with geometric means and hard dependency rules enforcing holistic correctness (Zhang et al., 2 Feb 2026). This does not reduce complexity; it structures its measurement so that partial failure modes remain visible.

On the instruction-synthesis side, TAG-INSTRUCT provides a complementary strategy. It compresses instructions into a small semantic tag set, expands complexity through DPO-trained tag generation, and reconstructs more difficult instructions from the expanded tags (Zhu et al., 24 May 2025). Although the framework is text-only, it offers a direct method for controlling the instruction side of IV-Complexity by operating in a compact tag space rather than raw token space (Zhu et al., 24 May 2025). A plausible implication is that multimodal IV-Complexity could be managed similarly by extending the tag vocabulary to visual grounding, OCR, spatial reasoning, and multi-image comparison.

At the design-theory level, “Complexity as Design Material” argues against treating complexity as a pure defect. Complexity can be shifted across initiation, datafication, transformation, visualization, interaction, interpretation, and communication, and useful design often depends on placing complexity where it supports engagement, nuance, or truthful representation rather than simply minimizing it (Windhager et al., 2024). This perspective is compatible with the empirical results above: the goal is not zero complexity, but calibrated and interpretable complexity.

6. Limitations, controversies, and open problems

The literature remains fragmented in both terminology and measurement. Some works define visual complexity through simple count proxies, such as the number of edge events in a dynamic-graph timeslice (Wang et al., 2019). Others rely on human ratings and rich image-based metrics, as in DReX, segmentation-based models, or visualization-image studies (Skaza et al., 21 Nov 2025, Shen et al., 2024, Chu et al., 9 Oct 2025). RePlan explicitly defines IV-Complexity but does not provide a numeric scalar for it, instead operationalizing it through benchmark design and failure modes (Qu et al., 18 Dec 2025). This leaves the field without a single accepted formalism.

A second limitation is that many strong structural metrics are weak on semantics. MSSC captures hierarchical structural load and is consistent across several image categories, but it underperforms on symbolic categories such as art and infographics, where “complexity arises in the space of interpretations, not visual composition” (Kravchenko et al., 2024). Visualization-image metrics capture perceived visual complexity in a task-free setting, but they do not measure conceptual difficulty or instructional semantics (Chu et al., 9 Oct 2025). DReX shows that pure vision can be sufficient for the visual side, but it also leaves open how instructions alter which visual cues matter (Skaza et al., 21 Nov 2025).

Benchmark realism is another recurring issue. iWISDM uses synthetic ShapeNet objects, templated instructions, and a limited horizon of 6–9 frames (Lei et al., 2024). VIBE covers multiple visual instruction modalities but still operates in curated 2D image-editing settings (Zhang et al., 2 Feb 2026). Complex-Edit controls compositional instruction difficulty through atomic edit counts, but its main scalar of complexity is one-dimensional and does not disentangle object count, spatial relation complexity, or logical depth (Yang et al., 17 Apr 2025). TAG-INSTRUCT offers a powerful instruction-side augmentation method, but it is currently text-only and therefore incomplete for full IV-Complexity (Zhu et al., 24 May 2025).

A broader controversy concerns whether complexity should always be minimized. Empirical work on learning and visualization suggests that moderate annotation can reduce perceived complexity and that ordered layouts support knowledge gain (Chu et al., 9 Oct 2025, Gritz et al., 9 Jan 2025). At the same time, the design-theory literature argues that complexity is often irreducible and sometimes beneficial, especially when a phenomenon itself is complex or when slower, more reflective interpretation is desirable (Windhager et al., 2024). This suggests that future IV-Complexity research should distinguish harmful overload from necessary or even productive complexity.

Promising directions are already visible. ScalSelect suggests a model-relative approach based on instruction-conditioned attention and dominant-subspace leverage, which could yield instance-level IV metrics grounded in a target VLM’s own representations (Wu et al., 12 Feb 2026). DReX suggests that the visual component can be modeled robustly with pure vision features, while iWISDM offers a template for decomposing total difficulty into instruction structure, visual structure, binding, reasoning, and output-space terms (Skaza et al., 21 Nov 2025, Lei et al., 2024). Taken together, these lines of work indicate that IV-Complexity is unlikely to collapse into a single universal score; it is more plausibly a family of measurable, interacting dimensions whose relative importance depends on task, modality, and design objective.