Chain of Manipulations in Vision Models

• Chain of Manipulations (CoM) is a reasoning framework that performs sequential visual operations to gather and trace evidence.
• It interleaves explicit visual actions—such as cropping, OCR, and frame selection—with linguistic reasoning to enhance accuracy.
• Implementations like CogCoM and Video-CoM demonstrate improved image and video task performance through interpretable, inspectable evidence chains.

to=arxiv_search 天天彩票中大奖.multipart/form-data ￣奇米影视json {"query":"id:(Rasheed et al., 28 Nov 2025) OR id:(Qi et al., 2024)", "max_results": 5} to=arxiv_search 天天中彩票网站.multipart/form-data qarputjson {"query":"\"Chain of Manipulations\" multimodal reasoning", "max_results": 10} Chain of Manipulations (CoM) is a reasoning mechanism in which a Vision-LLM or multimodal LLM solves a visual question by executing a sequence of visual operations on the input in order to acquire evidence before producing a final answer. In this formulation, the model’s solution is not only a conclusive linguistic response, but a chain of manipulation-execution descriptions and their returns, so that intermediate evidence remains visually grounded and traceable (Qi et al., 2024). The concept was introduced for image-grounded reasoning in CogCoM and later extended to interactive video reasoning in Video-CoM, where CoM turns a video from a static context into an active cognitive workspace and interleaves language reasoning with visual actions that rewatch, refocus, and verify visual evidence (Rasheed et al., 28 Nov 2025).

1. Conceptual basis and historical development

CoM emerged from the observation that conclusive alignment training encourages models to map an image and question directly to an answer, thereby ignoring essential visual reasoning and relying on linguistic priors. CogCoM frames this as a failure of faithfulness in meticulous visual problems: a model may answer from scene priors rather than from the visual evidence itself. CoM addresses this by making visual manipulation explicit, so that evidence acquisition becomes part of inference rather than a hidden by-product of representation learning (Qi et al., 2024).

In CogCoM, a manipulation is an operation applied to the visual input that yields structured visual content such as bounding boxes, text, numbers, or a new image crop or zoom. The model is trained so that it can elicit such manipulations intrinsically at inference without external calls; external tools are used only in data generation to provide supervision for manipulation returns (Qi et al., 2024). This establishes CoM as a middle position between purely linguistic Chain-of-Thought and tool-augmented VLMs: unlike Chain-of-Thought, CoM collects evidence in the visual domain; unlike tool-use systems, CogCoM internalizes the manipulations at inference time (Qi et al., 2024).

Video-CoM generalizes the same idea from images to videos. Its stated contrast is between models that “think about videos” and models that “think with videos.” The passive paradigm encodes a video once and reasons entirely in text, creating a semantic bottleneck: the model cannot rewatch fleeting events, isolate critical frames, or zoom into tiny text. CoM removes this bottleneck by allowing the model to rewatch temporally localized segments, freeze key frames, and zoom into small regions, thereby improving fine-grained spatio-temporal reasoning and the alignment between intermediate reasoning and final answers (Rasheed et al., 28 Nov 2025).

A plausible implication is that CoM is best understood not as a single operator library, but as a design principle: reasoning steps should be perception-action units whose outputs are concrete, inspectable evidence. That principle is constant across the image and video formulations, even though the implemented manipulation sets differ (Qi et al., 2024, Rasheed et al., 28 Nov 2025).

2. Formalization and manipulation operators

CogCoM formalizes a CoM chain for an image-question pair $(I_0, Q)$ as

$$\sigma = (step_1, step_2, \ldots), \qquad step_i = (f_i, desc_i), \qquad f_i \in M$$

where $f_i$ is an instantiated manipulation and $desc_i$ is a natural language execution description containing inputs and intended returns (Qi et al., 2024). Image-returning manipulations partition the chain into multi-turn tuples,

$$\sigma \rightarrow [(I_0, Q), (I_1, A_1), (I_2, A_2), \ldots]$$

so that cropped images can be passed into subsequent turns while preserving textual history (Qi et al., 2024).

Video-CoM instead casts CoM as an iterative interaction loop over textual context $T_i$ and visual input $V_i$. At step $i$,

$O_i = f_{\mathrm{MLLM}}(T_i, V_i),$

where $O_i$ contains both reasoning tokens and a predicted manipulation $$\sigma = (step_1, step_2, \ldots), \qquad step_i = (f_i, desc_i), \qquad f_i \in M$$0, and the manipulation updates the visual workspace through

$$\sigma = (step_1, step_2, \ldots), \qquad step_i = (f_i, desc_i), \qquad f_i \in M$$1

The resulting trajectory is

$$\sigma = (step_1, step_2, \ldots), \qquad step_i = (f_i, desc_i), \qquad f_i \in M$$2

with termination at a final answer or at a step limit $$\sigma = (step_1, step_2, \ldots), \qquad step_i = (f_i, desc_i), \qquad f_i \in M$$3; in training and evaluation, $$\sigma = (step_1, step_2, \ldots), \qquad step_i = (f_i, desc_i), \qquad f_i \in M$$4 (Rasheed et al., 28 Nov 2025).

The operator sets implemented in the two systems differ substantially.

System	Implemented manipulations	Returns
CogCoM	Grounding, OCR, Calculate, Counting, CropZoomIn	boxes, text, numbers, cropped image
Video-CoM	Find-Segment, Find-Frame, Spatial-Zoom	segment frames, single frame, cropped region

CogCoM predefines a flexible set $$\sigma = (step_1, step_2, \ldots), \qquad step_i = (f_i, desc_i), \qquad f_i \in M$$5 while also allowing the model to devise new manipulations in context. Its predefined operators are Grounding(tgt) → bbx, OCR(tgt) → txt, Calculate(tgt) → num, Counting(tgt) → num, and CropZoomIn(bbx, x) → img (Qi et al., 2024). Bounding boxes are serialized as [[x0,y0,x1,y1], ...] with normalized integer coordinates in [000, 999], and cropped images are produced by bicubic interpolation (Qi et al., 2024).

Video-CoM implements exactly three atomic manipulations: find-segment(s), find-frame(f), and spatial-zoom(f,b), together with a terminal answer action. Videos are pre-segmented, and both segment and frame indices are overlaid onto frames so that temporal references are explicit and timestamp regression is avoided (Rasheed et al., 28 Nov 2025). The intended semantics of the action set are sharply delimited: find-segment revisits rapid actions and event order, find-frame isolates a precise moment for spatial details, and spatial-zoom magnifies small regions such as text on signs, prices, license plates, and small objects (Rasheed et al., 28 Nov 2025).

These operator libraries show that CoM is not identical to generic frame resampling. CogCoM’s chain can traverse symbolic manipulations such as OCR and Calculate, whereas Video-CoM composes temporal and spatial operations over the video buffer itself. This suggests that the defining property of CoM is the explicit coupling of reasoning with evidence-producing manipulations, not any particular modality-specific operator inventory.

3. Architectures and interaction mechanics

CogCoM is a 17B VLM composed of an EVA2-CLIP-E vision encoder, a Vicuna-7B-v1.5 LLM backbone, a two-layer MLP adapter with SwiGLU, and a visual expert module that injects vision-specific weights into every Transformer block’s attention and feed-forward layers. The vision encoder is frozen, whereas the visual experts are trainable (Qi et al., 2024). Its multi-turn, multi-image design accumulates per-layer key-value memories across turns, with attention computed over concatenated and truncated histories:

$$\sigma = (step_1, step_2, \ldots), \qquad step_i = (f_i, desc_i), \qquad f_i \in M$$6

where $$\sigma = (step_1, step_2, \ldots), \qquad step_i = (f_i, desc_i), \qquad f_i \in M$$7 and $$\sigma = (step_1, step_2, \ldots), \qquad step_i = (f_i, desc_i), \qquad f_i \in M$$8 (Qi et al., 2024). This memory mechanism allows later reasoning steps to attend jointly to the current crop and to earlier context.

Video-CoM reuses Qwen2.5-VL-7B-Instruct as its backbone and reuses Qwen2.5-VL’s visual encoder and multimodal fusion; no new backbone layers are introduced (Rasheed et al., 28 Nov 2025). Instead of architectural novelty in the encoder, its novelty lies in the interaction loop. The initial visual buffer is a uniform sample of 32 frames at 2 FPS and resolution 360×420 px. Subsequent manipulations alter which frames or crops enter the encoder at the next turn: find-segment samples up to 8 frames from the selected segment, find-frame supplies a single high-resolution frame, and spatial-zoom crops a region from the current frame (Rasheed et al., 28 Nov 2025).

Video-CoM explicitly frames the reasoning loop as an MDP with states $$\sigma = (step_1, step_2, \ldots), \qquad step_i = (f_i, desc_i), \qquad f_i \in M$$9, actions $f_i$0 or Answer, transition $f_i$1, and episode termination at Answer or $f_i$2 (Rasheed et al., 28 Nov 2025). CogCoM does not present an explicit policy $f_i$3; inference is treated as next-token prediction conditioned on the dialogue context, optionally aided by launching prompts that enumerate manipulation options (Qi et al., 2024). The distinction is significant: CogCoM integrates CoM into a generative multi-turn VLM interface, whereas Video-CoM makes action selection itself the target of policy optimization.

In both systems, interpretability is not an auxiliary visualization layer but a structural property of inference. Each step produces intermediate artifacts—boxes, text strings, zoomed crops, frame selections, or segment retrievals—that can be inspected to trace both successful and failed reasoning (Qi et al., 2024, Rasheed et al., 28 Nov 2025).

4. Data construction and training methodology

CogCoM realizes CoM through a four-part roadmap: a flexible manipulation design, an automated data generation pipeline, a multi-turn multi-image architecture, and a training process for versatile capabilities (Qi et al., 2024). Its CoM training data comprise 70K synthesized CoM chains from TextVQA, ST-VQA, and TDIUC-count, plus 11,677 implicit chains for TDIUC-absurd without manipulations to resist hallucinations (Qi et al., 2024). The pipeline uses GPT-4 as a linguistic annotator, GroundingDINO for Grounding, PaddleOCR for OCR, and DFS over execution trees to identify positive paths that terminate at the correct answer (Qi et al., 2024). Heuristic CropZoomIn insertion is triggered when a step requires a function of a bounding box and the area ratio is large, for example $f_i$4 (Qi et al., 2024).

CogCoM training proceeds in two stages. Stage 1 pretrains visual experts on 1.5B image-text pairs from LAION-2B and COYO-700M, then on 40M grounded image-QA triples from LAION-115M. Stage 2 aligns the model using 570K samples combining MultiInstruct, LLaVAR, ShareGPT4V, and CoM multi-turn chains; launching prompts $f_i$5 are randomly prepended to encourage explicit elicitation of manipulations (Qi et al., 2024).

Video-CoM adopts a different training stack centered on manipulation supervision and reinforcement learning. Its Video-CoM-Instruct dataset contains 18K manipulation-driven instruction-tuning samples designed for multi-step CoM reasoning, derived from approximately 9K source videos selected from large existing corpora using an information-density score, plus 70 manually curated high-complexity videos from YouTube for RL (Rasheed et al., 28 Nov 2025). Gemini-2.5-Pro generates 3–5 manipulation-targeted QAs per video, yielding approximately 25K candidates, and dual-format verification prunes this to 18K with correct step-by-step traces; spatial tubes for zoom are re-annotated with high-agreement filtering using Gemini + InternVL3-78B (Rasheed et al., 28 Nov 2025). The split is 15K for SFT and 3K for GRPO with dense step-level spatio-temporal annotations $f_i$6, $f_i$7, and $f_i$8 (Rasheed et al., 28 Nov 2025).

Stage I of Video-CoM uses supervised fine-tuning on 25K videos + 180K images: 15K Video-CoM-Instruct SFT, 9K ActivityNet, and 180K Visual-CoT. Stage II applies reinforcement learning with reasoning-aware GRPO (RA-GRPO) (Rasheed et al., 28 Nov 2025). Its central addition is a step-level reasoning reward:

$f_i$9

where $desc_i$0 if a predicted manipulation matches a valid segment, valid frame, or a spatial box with $desc_i$1, and $desc_i$2 otherwise; the paper sets $desc_i$3 (Rasheed et al., 28 Nov 2025). This is combined with an answer reward

$desc_i$4

into a total reward

$desc_i$5

For each input, a group of $desc_i$6 trajectories is sampled, and the group-relative advantage is

$desc_i$7

The optimization objective includes a KL anchor to the SFT policy with $desc_i$8, and the formulation introduces no clipping or entropy bonuses (Rasheed et al., 28 Nov 2025).

This contrast is instructive. CogCoM learns CoM primarily through large-scale synthetic supervision and alignment data, whereas Video-CoM further operationalizes CoM as a policy problem over explicit manipulations. A plausible implication is that video CoM requires stronger trajectory-level supervision because temporal mislocalization can corrupt all later evidence-gathering steps.

5. Empirical performance and characteristic trajectories

CogCoM reports state-of-the-art performance across 9 benchmarks from 4 categories with a 17B model (Qi et al., 2024). On detailed VQA, it achieves 71.7 on GQA, 84.0 on TallyVQA simple and 70.1 on TallyVQA complex, 71.1 on TextVQA, and 70.0 on ST-VQA. On visual grounding, it reports 92.34/94.57/89.15 on RefCOCO val/test-A/test-B, 88.19/92.80/82.08 on RefCOCO+ val/test-A/test-B, and 89.32/90.45 on RefCOCOg val/test. On POPE adversarial hallucination, it reports 87.8 F1 (Qi et al., 2024). The paper also introduces AutoCoM-test, built from TextVQA validation, where training on CoM data alone raises zero-shot accuracy from 2.15 to 48.41 after 2k steps and to 55.59 after 8k steps; explanation scores rise proportionally with answer accuracy (Qi et al., 2024).

Video-CoM evaluates on nine video reasoning benchmarks and reports an average performance gain of +3.6% over recent SOTA across nine benchmarks, while training on only 25K SFT and 3K GRPO video samples (Rasheed et al., 28 Nov 2025). On the new Video-CoM-Bench, it attains 68.7, compared with 60.2 for VideoRFT, 59.9 for VideoChat-R1.5, and 57.1 for Video-R1. On VideoMMMU/Perc it reports 70.0, on VideoMMMU/Avg 50.2, on ScienceVideoBench 27.6, on VideoMathQA 27.8, on MMVU-Val 65.4, and on Minerva 31.7 (Rasheed et al., 28 Nov 2025). Its reasoning average over five reasoning-oriented benchmarks is 40.5, compared with 39.3 for VideoChat-R1.5, 38.8 for VideoChat-R1 and VideoRFT, and 37.9 for Video-R1 (Rasheed et al., 28 Nov 2025).

Ablations in Video-CoM isolate the role of reasoning-aware rewards and of the manipulation set. On Video-CoM-Bench, performance improves from 64.0 under SFT to 66.7 under answer-only GRPO and to 68.7 under RA-GRPO; average reasoning improves from 37.5 to 38.9 to 40.5 (Rasheed et al., 28 Nov 2025). Enabling find-segment yields the largest single gain, and adding find-frame and then spatial-zoom progressively improves results; all three together perform best (Rasheed et al., 28 Nov 2025). Reasoning accuracy and joint Acc@IoU also improve monotonically from SFT to RA-GRPO, for example Reasoning 49.2 → 51.0 → 53.8 and Acc@IoU 38.1 → 37.6 → 39.2 (Rasheed et al., 28 Nov 2025).

The qualitative trajectories reported in both papers clarify what CoM is meant to solve. In CogCoM, a typical sequence is Grounding("the pillar") → bbx; CropZoomIn(bbx,2) → img1; OCR(img1) → txt ("STAFF ONLY"); Answer: "STAFF ONLY" (Qi et al., 2024). In Video-CoM, a representative trajectory for reading a brief on-screen timing graphic is Find-Segment, then Find-Frame, then Spatial-Zoom, so that the final answer quotes split times read from the zoomed overlay (Rasheed et al., 28 Nov 2025). Another example revisits the moment with transition titles, isolates a frame reading “Narrated by Brian Pederson,” and grounds the answer in that frame rather than in world knowledge (Rasheed et al., 28 Nov 2025).

These cases underscore that CoM is especially advantageous when evidence is fleeting, small, or compositionally distributed across time and space: reading fine text, counting localized entities, tracing event order, or distinguishing short-lived visual states (Qi et al., 2024, Rasheed et al., 28 Nov 2025).

6. Interpretive scope, related paradigms, and limitations

CoM is often compared with Chain-of-Thought, but the distinction is formal rather than merely stylistic. Chain-of-Thought decomposes a task into linguistic reasoning steps; CoM decomposes it into evidence-bearing visual steps. In CogCoM, the chain consists of executable manipulations over images; in Video-CoM, the steps are explicit perception-action units over the video workspace (Qi et al., 2024, Rasheed et al., 28 Nov 2025). Compared with tool-use agents that only re-sample frames, Video-CoM composes temporal and spatial operations into richer, multi-level reasoning chains (Rasheed et al., 28 Nov 2025).

A common source of confusion is terminological rather than methodological. In arXiv usage, “CoM” is overloaded. In the present topic it denotes Chain of Manipulations, as in CogCoM and Video-CoM (Qi et al., 2024, Rasheed et al., 28 Nov 2025). Other contemporaneous papers use CoM for Chain-of-Memory, Chain of Modality, or Chain of Merges, and explicitly note that these are not Chain of Manipulations (Xu et al., 14 Jan 2026, Luo et al., 16 Apr 2026, Buzzega et al., 29 Aug 2025). Any encyclopedic treatment therefore needs to separate the manipulation-based visual reasoning literature from unrelated CoM acronyms.

The limitations reported in the two principal CoM papers are also complementary. CogCoM identifies weaknesses in data generation, including limited linguistic step diversity from 5-shot prompting, coarse boxes from GroundingDINO, and OCR failures on slanted text and long strings; it also notes inference overhead from re-inputting manipulated images with hard prompts (Qi et al., 2024). Video-CoM emphasizes that spatial localization is hard, especially for text regions in videos; bounding boxes for text are less reliable than for objects and required manual annotation in the GRPO subset. It also notes that long, homogeneous videos reduce the need for manipulations, error cascades can arise when an early segment choice is wrong, and interactive steps add compute and latency. On VCoM-Bench, average per-iteration latency is 25.90s for Video-CoM, compared with 20.27s for VideoChat-R1.5 and 19.04s for Video-R1 at the same 32-frame setting (Rasheed et al., 28 Nov 2025).

The future directions named in the video work indicate where the framework may expand next: richer action spaces such as speed changes, object tracking, motion magnification, and temporal differencing; better grounding metrics and automatic tubelet annotation; hierarchical CoM for very long videos; and joint audio-visual manipulations when audio is available (Rasheed et al., 28 Nov 2025). CogCoM, by contrast, points toward in-vector manipulations that would avoid repeated image encoding, as well as more diverse manipulations and supervision for greater robustness (Qi et al., 2024).

Taken together, these papers define Chain of Manipulations as a general paradigm for grounded multimodal reasoning in which the model does not merely condition on visual input, but actively restructures that input through a sequence of evidence-producing operations. In the image case, this yields multi-turn chains over grounding, OCR, counting, calculation, and zoom. In the video case, it yields interactive trajectories over segment retrieval, frame selection, and spatial zoom. The unifying claim is that faithful reasoning improves when intermediate steps are themselves visual actions whose returns can be inspected, verified, and optimized (Qi et al., 2024, Rasheed et al., 28 Nov 2025).