---
title: Chain of Manipulations in Vision Models
url: https://www.emergentmind.com/topics/chain-of-manipulations-com
type: topic
---

# Chain of Manipulations in Vision Models

to=arxiv_search  天天彩票中大奖.multipart/form-data ￣奇米影视json
{"query":"id:2511.23477 OR id:2402.04236", "max_results": 5}
to=arxiv_search  天天中彩票网站.multipart/form-data qarputjson
{"query":"\"Chain of Manipulations\" multimodal reasoning", "max_results": 10}
Chain of Manipulations (CoM) is a reasoning mechanism in which a Vision-Language Model or multimodal large language model solves a visual question by executing a sequence of visual operations on the input in order to acquire evidence before producing a final answer. In this formulation, the model’s solution is not only a conclusive linguistic response, but a chain of manipulation-execution descriptions and their returns, so that intermediate evidence remains visually grounded and traceable [2402.04236]. The concept was introduced for image-grounded reasoning in CogCoM and later extended to interactive video reasoning in Video-CoM, where CoM turns a video from a static context into an active cognitive workspace and interleaves language reasoning with visual actions that rewatch, refocus, and verify visual evidence [2511.23477].

## 1. Conceptual basis and historical development

CoM emerged from the observation that conclusive alignment training encourages models to map an image and question directly to an answer, thereby ignoring essential visual reasoning and relying on linguistic priors. CogCoM frames this as a failure of faithfulness in meticulous visual problems: a model may answer from scene priors rather than from the visual evidence itself. CoM addresses this by making visual manipulation explicit, so that evidence acquisition becomes part of inference rather than a hidden by-product of representation learning [2402.04236].

In CogCoM, a manipulation is an operation applied to the visual input that yields structured visual content such as bounding boxes, text, numbers, or a new image crop or zoom. The model is trained so that it can elicit such manipulations intrinsically at inference without external calls; external tools are used only in data generation to provide supervision for manipulation returns [2402.04236]. This establishes CoM as a middle position between purely linguistic Chain-of-Thought and tool-augmented VLMs: unlike Chain-of-Thought, CoM collects evidence in the visual domain; unlike tool-use systems, CogCoM internalizes the manipulations at inference time [2402.04236].

Video-CoM generalizes the same idea from images to videos. Its stated contrast is between models that “think about videos” and models that “think with videos.” The passive paradigm encodes a video once and reasons entirely in text, creating a semantic bottleneck: the model cannot rewatch fleeting events, isolate critical frames, or zoom into tiny text. CoM removes this bottleneck by allowing the model to rewatch temporally localized segments, freeze key frames, and zoom into small regions, thereby improving fine-grained spatio-temporal reasoning and the alignment between intermediate reasoning and final answers [2511.23477].

A plausible implication is that CoM is best understood not as a single operator library, but as a design principle: reasoning steps should be perception-action units whose outputs are concrete, inspectable evidence. That principle is constant across the image and video formulations, even though the implemented manipulation sets differ [2402.04236][2511.23477].

## 2. Formalization and manipulation operators

CogCoM formalizes a CoM chain for an image-question pair $(I_0, Q)$ as
$$
\sigma = (step_1, step_2, \ldots), \qquad step_i = (f_i, desc_i), \qquad f_i \in M
$$
where $f_i$ is an instantiated manipulation and $desc_i$ is a natural language execution description containing inputs and intended returns [2402.04236]. Image-returning manipulations partition the chain into multi-turn tuples,
$$
\sigma \rightarrow [(I_0, Q), (I_1, A_1), (I_2, A_2), \ldots]
$$
so that cropped images can be passed into subsequent turns while preserving textual history [2402.04236].

Video-CoM instead casts CoM as an iterative interaction loop over textual context $T_i$ and visual input $V_i$. At step $i$,
$$
O_i = f_{\mathrm{MLLM}}(T_i, V_i),
$$
where $O_i$ contains both reasoning tokens and a predicted manipulation $m_i$, and the manipulation updates the visual workspace through
$$
V_{i+1} = g(V_i, m_i).
$$
The resulting trajectory is
$$
\tau = \{(T_1, V_1, m_1), \ldots, (T_N, V_N, m_N)\},
$$
with termination at a final answer or at a step limit $N$; in training and evaluation, $N_{\max}=5$ [2511.23477].

The operator sets implemented in the two systems differ substantially.

| System | Implemented manipulations | Returns |
|---|---|---|
| CogCoM | Grounding, OCR, Calculate, Counting, CropZoomIn | boxes, text, numbers, cropped image |
| Video-CoM | Find-Segment, Find-Frame, Spatial-Zoom | segment frames, single frame, cropped region |

CogCoM predefines a flexible set $M$ while also allowing the model to devise new manipulations in context. Its predefined operators are `Grounding(tgt) → bbx`, `OCR(tgt) → txt`, `Calculate(tgt) → num`, `Counting(tgt) → num`, and `CropZoomIn(bbx, x) → img` [2402.04236]. Bounding boxes are serialized as `[[x0,y0,x1,y1], ...]` with normalized integer coordinates in `[000, 999]`, and cropped images are produced by bicubic interpolation [2402.04236].

Video-CoM implements exactly three atomic manipulations: `find-segment(s)`, `find-frame(f)`, and `spatial-zoom(f,b)`, together with a terminal `answer` action. Videos are pre-segmented, and both segment and frame indices are overlaid onto frames so that temporal references are explicit and timestamp regression is avoided [2511.23477]. The intended semantics of the action set are sharply delimited: `find-segment` revisits rapid actions and event order, `find-frame` isolates a precise moment for spatial details, and `spatial-zoom` magnifies small regions such as text on signs, prices, license plates, and small objects [2511.23477].

These operator libraries show that CoM is not identical to generic frame resampling. CogCoM’s chain can traverse symbolic manipulations such as OCR and Calculate, whereas Video-CoM composes temporal and spatial operations over the video buffer itself. This suggests that the defining property of CoM is the explicit coupling of reasoning with evidence-producing manipulations, not any particular modality-specific operator inventory.

## 3. Architectures and interaction mechanics

CogCoM is a 17B VLM composed of an EVA2-CLIP-E vision encoder, a Vicuna-7B-v1.5 LLM backbone, a two-layer MLP adapter with SwiGLU, and a visual expert module that injects vision-specific weights into every Transformer block’s attention and feed-forward layers. The vision encoder is frozen, whereas the visual experts are trainable [2402.04236]. Its multi-turn, multi-image design accumulates per-layer key-value memories across turns, with attention computed over concatenated and truncated histories:
$$
att(X) = \mathrm{softmax}(Q_t K_t^{\prime \top} / \sqrt{d}) V_t^\prime,
$$
where $K_t^\prime = trunc(concat(K_0, K_1, \ldots, K_t))$ and $V_t^\prime = trunc(concat(V_0, V_1, \ldots, V_t))$ [2402.04236]. This memory mechanism allows later reasoning steps to attend jointly to the current crop and to earlier context.

Video-CoM reuses Qwen2.5-VL-7B-Instruct as its backbone and reuses Qwen2.5-VL’s visual encoder and multimodal fusion; no new backbone layers are introduced [2511.23477]. Instead of architectural novelty in the encoder, its novelty lies in the interaction loop. The initial visual buffer is a uniform sample of 32 frames at 2 FPS and resolution 360×420 px. Subsequent manipulations alter which frames or crops enter the encoder at the next turn: `find-segment` samples up to 8 frames from the selected segment, `find-frame` supplies a single high-resolution frame, and `spatial-zoom` crops a region from the current frame [2511.23477].

Video-CoM explicitly frames the reasoning loop as an MDP with states $s_i = (T_i, V_i)$, actions $a_i = m_i$ or `Answer`, transition $V_{i+1} = g(V_i, m_i)$, and episode termination at `Answer` or $i = N_{\max}$ [2511.23477]. CogCoM does not present an explicit policy $\pi_\theta(m_t \mid s_t)$; inference is treated as next-token prediction conditioned on the dialogue context, optionally aided by launching prompts that enumerate manipulation options [2402.04236]. The distinction is significant: CogCoM integrates CoM into a generative multi-turn VLM interface, whereas Video-CoM makes action selection itself the target of policy optimization.

In both systems, interpretability is not an auxiliary visualization layer but a structural property of inference. Each step produces intermediate artifacts—boxes, text strings, zoomed crops, frame selections, or segment retrievals—that can be inspected to trace both successful and failed reasoning [2402.04236][2511.23477].

## 4. Data construction and training methodology

CogCoM realizes CoM through a four-part roadmap: a flexible manipulation design, an automated data generation pipeline, a multi-turn multi-image architecture, and a training process for versatile capabilities [2402.04236]. Its CoM training data comprise 70K synthesized CoM chains from TextVQA, ST-VQA, and TDIUC-count, plus 11,677 implicit chains for TDIUC-absurd without manipulations to resist hallucinations [2402.04236]. The pipeline uses GPT-4 as a linguistic annotator, GroundingDINO for Grounding, PaddleOCR for OCR, and DFS over execution trees to identify positive paths that terminate at the correct answer [2402.04236]. Heuristic `CropZoomIn` insertion is triggered when a step requires a function of a bounding box and the area ratio is large, for example $r \ge 36$ [2402.04236].

CogCoM training proceeds in two stages. Stage 1 pretrains visual experts on 1.5B image-text pairs from LAION-2B and COYO-700M, then on 40M grounded image-QA triples from LAION-115M. Stage 2 aligns the model using 570K samples combining MultiInstruct, LLaVAR, ShareGPT4V, and CoM multi-turn chains; launching prompts $P^M$ are randomly prepended to encourage explicit elicitation of manipulations [2402.04236].

Video-CoM adopts a different training stack centered on manipulation supervision and reinforcement learning. Its `Video-CoM-Instruct` dataset contains 18K manipulation-driven instruction-tuning samples designed for multi-step CoM reasoning, derived from approximately 9K source videos selected from large existing corpora using an information-density score, plus 70 manually curated high-complexity videos from YouTube for RL [2511.23477]. Gemini-2.5-Pro generates 3–5 manipulation-targeted QAs per video, yielding approximately 25K candidates, and dual-format verification prunes this to 18K with correct step-by-step traces; spatial tubes for zoom are re-annotated with high-agreement filtering using Gemini + InternVL3-78B [2511.23477]. The split is 15K for SFT and 3K for GRPO with dense step-level spatio-temporal annotations $S^\*$, $F^\*$, and $B^\*$ [2511.23477].

Stage I of Video-CoM uses supervised fine-tuning on 25K videos + 180K images: 15K Video-CoM-Instruct SFT, 9K ActivityNet, and 180K Visual-CoT. Stage II applies reinforcement learning with reasoning-aware GRPO (RA-GRPO) [2511.23477]. Its central addition is a step-level reasoning reward:
$$
R_{\mathrm{reason}} = \frac{1}{N} \sum_{i=1}^{N} c_i,
$$
where $c_i = 1$ if a predicted manipulation matches a valid segment, valid frame, or a spatial box with $\mathrm{IoU}(b_i^\prime, b^\*) \ge \tau_b$, and $0$ otherwise; the paper sets $\tau_b = 0.3$ [2511.23477]. This is combined with an answer reward
$$
R_{\mathrm{accuracy}} =
\begin{cases}
1, & \hat y = y^\* \\
0, & \text{otherwise}
\end{cases}
$$
into a total reward
$$
R = R_{\mathrm{accuracy}} + R_{\mathrm{reason}}.
$$
For each input, a group of $K=8$ trajectories is sampled, and the group-relative advantage is
$$
A_k = R_k - \frac{1}{K}\sum_{j=1}^{K} R_j.
$$
The optimization objective includes a KL anchor to the SFT policy with $\lambda = 0.04$, and the formulation introduces no clipping or entropy bonuses [2511.23477].

This contrast is instructive. CogCoM learns CoM primarily through large-scale synthetic supervision and alignment data, whereas Video-CoM further operationalizes CoM as a policy problem over explicit manipulations. A plausible implication is that video CoM requires stronger trajectory-level supervision because temporal mislocalization can corrupt all later evidence-gathering steps.

## 5. Empirical performance and characteristic trajectories

CogCoM reports state-of-the-art performance across 9 benchmarks from 4 categories with a 17B model [2402.04236]. On detailed VQA, it achieves 71.7 on GQA, 84.0 on TallyVQA simple and 70.1 on TallyVQA complex, 71.1 on TextVQA, and 70.0 on ST-VQA. On visual grounding, it reports 92.34/94.57/89.15 on RefCOCO val/test-A/test-B, 88.19/92.80/82.08 on RefCOCO+ val/test-A/test-B, and 89.32/90.45 on RefCOCOg val/test. On POPE adversarial hallucination, it reports 87.8 F1 [2402.04236]. The paper also introduces AutoCoM-test, built from TextVQA validation, where training on CoM data alone raises zero-shot accuracy from 2.15 to 48.41 after 2k steps and to 55.59 after 8k steps; explanation scores rise proportionally with answer accuracy [2402.04236].

Video-CoM evaluates on nine video reasoning benchmarks and reports an average performance gain of +3.6% over recent SOTA across nine benchmarks, while training on only 25K SFT and 3K GRPO video samples [2511.23477]. On the new Video-CoM-Bench, it attains 68.7, compared with 60.2 for VideoRFT, 59.9 for VideoChat-R1.5, and 57.1 for Video-R1. On VideoMMMU/Perc it reports 70.0, on VideoMMMU/Avg 50.2, on ScienceVideoBench 27.6, on VideoMathQA 27.8, on MMVU-Val 65.4, and on Minerva 31.7 [2511.23477]. Its reasoning average over five reasoning-oriented benchmarks is 40.5, compared with 39.3 for VideoChat-R1.5, 38.8 for VideoChat-R1 and VideoRFT, and 37.9 for Video-R1 [2511.23477].

Ablations in Video-CoM isolate the role of reasoning-aware rewards and of the manipulation set. On Video-CoM-Bench, performance improves from 64.0 under SFT to 66.7 under answer-only GRPO and to 68.7 under RA-GRPO; average reasoning improves from 37.5 to 38.9 to 40.5 [2511.23477]. Enabling `find-segment` yields the largest single gain, and adding `find-frame` and then `spatial-zoom` progressively improves results; all three together perform best [2511.23477]. Reasoning accuracy and joint Acc@IoU also improve monotonically from SFT to RA-GRPO, for example Reasoning 49.2 → 51.0 → 53.8 and Acc@IoU 38.1 → 37.6 → 39.2 [2511.23477].

The qualitative trajectories reported in both papers clarify what CoM is meant to solve. In CogCoM, a typical sequence is `Grounding("the pillar") → bbx; CropZoomIn(bbx,2) → img1; OCR(img1) → txt ("STAFF ONLY"); Answer: "STAFF ONLY"` [2402.04236]. In Video-CoM, a representative trajectory for reading a brief on-screen timing graphic is `Find-Segment`, then `Find-Frame`, then `Spatial-Zoom`, so that the final answer quotes split times read from the zoomed overlay [2511.23477]. Another example revisits the moment with transition titles, isolates a frame reading “Narrated by Brian Pederson,” and grounds the answer in that frame rather than in world knowledge [2511.23477].

These cases underscore that CoM is especially advantageous when evidence is fleeting, small, or compositionally distributed across time and space: reading fine text, counting localized entities, tracing event order, or distinguishing short-lived visual states [2402.04236][2511.23477].

## 6. Interpretive scope, related paradigms, and limitations

CoM is often compared with Chain-of-Thought, but the distinction is formal rather than merely stylistic. Chain-of-Thought decomposes a task into linguistic reasoning steps; CoM decomposes it into evidence-bearing visual steps. In CogCoM, the chain consists of executable manipulations over images; in Video-CoM, the steps are explicit perception-action units over the video workspace [2402.04236][2511.23477]. Compared with tool-use agents that only re-sample frames, Video-CoM composes temporal and spatial operations into richer, multi-level reasoning chains [2511.23477].

A common source of confusion is terminological rather than methodological. In arXiv usage, “CoM” is overloaded. In the present topic it denotes Chain of Manipulations, as in CogCoM and Video-CoM [2402.04236][2511.23477]. Other contemporaneous papers use CoM for Chain-of-Memory, Chain of Modality, or Chain of Merges, and explicitly note that these are not Chain of Manipulations [2601.14287][2604.14520][2508.21421]. Any encyclopedic treatment therefore needs to separate the manipulation-based visual reasoning literature from unrelated CoM acronyms.

The limitations reported in the two principal CoM papers are also complementary. CogCoM identifies weaknesses in data generation, including limited linguistic step diversity from 5-shot prompting, coarse boxes from GroundingDINO, and OCR failures on slanted text and long strings; it also notes inference overhead from re-inputting manipulated images with hard prompts [2402.04236]. Video-CoM emphasizes that spatial localization is hard, especially for text regions in videos; bounding boxes for text are less reliable than for objects and required manual annotation in the GRPO subset. It also notes that long, homogeneous videos reduce the need for manipulations, error cascades can arise when an early segment choice is wrong, and interactive steps add compute and latency. On VCoM-Bench, average per-iteration latency is 25.90s for Video-CoM, compared with 20.27s for VideoChat-R1.5 and 19.04s for Video-R1 at the same 32-frame setting [2511.23477].

The future directions named in the video work indicate where the framework may expand next: richer action spaces such as speed changes, object tracking, motion magnification, and temporal differencing; better grounding metrics and automatic tubelet annotation; hierarchical CoM for very long videos; and joint audio-visual manipulations when audio is available [2511.23477]. CogCoM, by contrast, points toward in-vector manipulations that would avoid repeated image encoding, as well as more diverse manipulations and supervision for greater robustness [2402.04236].

Taken together, these papers define Chain of Manipulations as a general paradigm for grounded multimodal reasoning in which the model does not merely condition on visual input, but actively restructures that input through a sequence of evidence-producing operations. In the image case, this yields multi-turn chains over grounding, OCR, counting, calculation, and zoom. In the video case, it yields interactive trajectories over segment retrieval, frame selection, and spatial zoom. The unifying claim is that faithful reasoning improves when intermediate steps are themselves visual actions whose returns can be inspected, verified, and optimized [2402.04236][2511.23477].

Source: https://www.emergentmind.com/topics/chain-of-manipulations-com