---
title: 'OSR-CoT: State-Centric Reasoning'
url: https://www.emergentmind.com/topics/object-state-reasoning-chain-of-thought-osr-cot
type: topic
---

# OSR-CoT: State-Centric Reasoning

Searching arXiv for the specified OSR-CoT papers and closely related work.
Object State Reasoning Chain-of-Thought (OSR-CoT) denotes a class of chain-of-thought formulations in which intermediate reasoning is organized around explicit object states, state transitions, and inter-object relations rather than around an unstructured linear text trace. Across recent instantiations, OSR-CoT treats the “thought” process as grounded state evolution: detected agents in driving scenes are represented by structured tuples with uncertainty and intent variables; procedural planning steps are interpreted through before/after object-state changes; and multimodal reasoning can be externalized into mutable object graphs whose nodes are updated in place [2504.05908] [2509.21662] [2602.10494]. This perspective is not a single algorithm but a methodological family whose defining properties are explicit state representation, step-wise transition reasoning, and the use of those states to support interpretable decisions, cross-modal consistency, or grounded self-correction.

## 1. Concept and scope

In PRIMEDrive-CoT, OSR-CoT is defined as “a structured reasoning paradigm that decomposes scene understanding into explicit, step-wise inferences over object states and their interactions, while maintaining uncertainty estimates throughout” [2504.05908]. In MMPlanner, OSR-CoT is the prompting strategy that “explicitly model[s] object-state transitions” so that multimodal procedural plans remain consistent across text and images [2509.21662]. In Canvas-of-Thought, the same general idea is reformulated as reasoning over “an explicit, mutable visual substrate,” where the reasoning history is no longer an append-only text sequence but a stateful object graph subject to CRUD operations [2602.10494].

A common denominator across these formulations is that the unit of reasoning is not merely a sentence or token span, but an object together with attributes and relations. In driving, an object state may contain geometry, dynamics, category, uncertainty, and intent cues; in procedural planning, the relevant state is the physical configuration of task objects before and after each step; in canvas-based multimodal reasoning, the object state is embodied by DOM nodes, attributes, and structural relations [2504.05908] [2509.21662] [2602.10494]. This suggests a unifying interpretation of OSR-CoT as state-centric chain-of-thought: the chain is a sequence of state transformations, and interpretability arises from making those transformations explicit.

The notion also differs from generic chain-of-thought in a specific way stated in the MMPlanner work: OSR-CoT focuses explicitly on “Before/After” object states and integrates prior-step context tailored to multimodal planning, whereas generic CoT lacks explicit state-change reasoning [2509.21662]. Canvas-of-Thought sharpens this distinction further by arguing that ordinary CoT treats reasoning history as an immutable stream, forcing the model to implicitly maintain state, while OSR-CoT-style mutable state substrates preserve a single explicit “ground truth” state outside the token stream [2602.10494].

## 2. Formal object-state representations

The most explicit object-state formalization appears in PRIMEDrive-CoT. For each detected object \( i \), the state is

\[
s_i = (x_i, y_i, z_i, v_i, \theta_i, l_i, w_i, h_i, c_i, U_i, I_i),
\]

where \((x,y,z)\) is 3D position, \(v\) is speed or velocity vector, \(\theta\) is yaw, \((l,w,h)\) are box dimensions, \(c\) is class, \(U\) is uncertainty, and \(I\) summarizes intent cues such as deceleration or lane-change propensity inferred from motion and context [2504.05908]. This formulation makes OSR-CoT a measurable, machine-tractable reasoning scheme rather than merely a prompting heuristic.

Object interaction in the same framework is represented by graph edges parameterized by relative distance, closing speed, and contextual interaction intensity. The edge interaction energy is

\[
e_{ij} = \lambda_1 D_{ij} + \lambda_2 \Delta V_{ij} + \lambda_3 I_{ij},
\]

with graph topology and weights influenced by traffic rules, adjacency, and proximity [2504.05908]. Here OSR-CoT is relational: reasoning steps depend not only on per-object states but also on structured inter-object dependencies.

MMPlanner provides a different formalization suited to multimodal procedural planning. The multimodal goal is \( G = (G_t, G_v) \), the sequence of steps is \( S = \{s_1, s_2, \ldots, s_n\} \), with each step \( s_i = (t_i, v_i) \), and the generated image descriptions are \( D = \{d_1, \ldots, d_n\} \) [2509.21662]. The paper does not provide an explicit state-transition function of the form \( s_{t+1} = f(s_t, a_t) \); instead, state reasoning is operationalized through prompted before/after descriptions. That is, the state representation is partly linguistic, but still object-centered and temporally indexed.

Canvas-of-Thought gives a fully state-transition-centric formalization. The state at time \( t \) is

\[
\mathcal{S}_t = \{ \mathcal{V}_t, \mathcal{E}_t, \mathcal{P}_t \},
\]

where \( \mathcal{V}_t \) is the set of objects, \( \mathcal{E}_t \) the hierarchical or topological edges, and \( \mathcal{P}_t \) the per-node attributes [2602.10494]. The action space is

\[
a_t \in \{ \text{Insert}, \text{Replace}, \text{Modify}, \text{Delete} \},
\]

and transitions are realized by a deterministic executor,

\[
\mathcal{S}_{t+1} = \delta(\mathcal{S}_t, a_t).
\]

This is a particularly literal OSR-CoT formulation: the chain-of-thought becomes a sequence of addressable object-state mutations. A plausible implication is that Canvas-of-Thought makes OSR-CoT maximally explicit by externalizing state into a mutable substrate rather than encoding it in latent activations or text.

## 3. Architectural instantiations

PRIMEDrive-CoT instantiates OSR-CoT in autonomous driving through a five-stage pipeline. Stage 1 performs sensing and preprocessing on LiDAR point clouds and multi-view RGB images. LiDAR points \( P = \{(x_i, y_i, z_i, I_i)\} \) are voxelized, per-voxel centroids are computed, intensities are min–max normalized, and coordinates are range-normalized by \( R_{\max} \). Multi-view RGB images are resized to \(224 \times 224\) and normalized with \( \boldsymbol{\mu} = [0.485, 0.456, 0.406] \) and \( \boldsymbol{\sigma} = [0.229, 0.224, 0.225] \) [2504.05908]. Stage 2 uses an enhanced VoxelNet backbone for LiDAR and an ImageNet-pretrained ResNet34 for RGB, with MVX-Net fusion into a common latent space to output 3D boxes \( b_i = (x,y,z,l,w,h,\theta) \) and class scores \( p(c) \) [2504.05908].

Stage 3 computes uncertainty and proximity-aware risk. Classification uncertainty is Shannon entropy,

\[
H(p) = -\sum_k p_k \log p_k,
\]

orientation inconsistency is measured by

\[
\Delta\theta = \left|\theta_{\text{pred}} - \theta_{\text{ref}}\right|,
\]

and the unified uncertainty score is

\[
U = w_1 H(p) + w_2 \Delta\theta,\quad w_1,w_2>0.
\]

Proximity-aware risk is

\[
d_{\min} = \min_i \sqrt{x_i^2 + y_i^2 + z_i^2}, \qquad
R = \exp\!\left(-\frac{d_{\min}}{\lambda}\right).
\]

Stage 4 builds a graph \( G=(V,E) \) and performs Bayesian relational reasoning with a BGNN. Stage 5 converts BGNN outputs into step-by-step rationales and aligns Grad-CAM overlays with LiDAR detections for explanation [2504.05908]. In this pipeline, OSR-CoT is inseparable from uncertainty propagation and risk prioritization.

MMPlanner, referred to in the paper as mmPurple, uses OSR-CoT inside a three-component zero-shot multimodal procedural planning pipeline. A visual goal \( G_v \) is first generated from the textual goal \( G_t \) using Stable Diffusion, producing a multimodal goal \( G=(G_t,G_v) \). LLaVa-1.5-7B then generates the textual plan \( T \). For each textual step \( t_i \), GPT-3.5 receives an OSR-CoT prompt conditioned on the goal, the current step, and up to 10 previous steps, and outputs a structured description of object states before and after the step, followed by a concise image description \( d_i \). Stable Diffusion samples \( K \) candidate images, and BLIP-2 embeddings are used to select the best image via cosine similarity, \( \arg\max_k \operatorname{sim}(f_{ik}, f_{d_i}) \) [2509.21662]. Here OSR-CoT acts primarily as an image-description generator that resolves explicit and implicit state changes before image synthesis and selection.

Canvas-of-Thought uses a different substrate. The MLLM generates one-step reasoning coupled to an XML-like `<tool_call>` with JSON arguments for insert, modify, replace, or remove operations. A deterministic parsing layer validates fragments, an ID-addressable map \( \mathcal{H}: \mathcal{ID} \to \text{NodePointer} \) supports \( O(1) \) object lookup, and a headless browser renders the DOM state to \( V_{\text{state}} \) for comparison against the original visual evidence \( V_{\text{orig}} \) [2602.10494]. The resulting pipeline comprises atomic reasoning, state mutation, rendering, critique, and correction. Unlike the other instantiations, this one makes the reasoning substrate itself editable.

## 4. Uncertainty, validation, and grounded correction

A central feature of PRIMEDrive-CoT is the integration of aleatoric and epistemic uncertainty into OSR-CoT. Aleatoric uncertainty is captured by entropy \( H(p) \) and pose deviation \( \Delta\theta \), reflecting class ambiguity, noisy LiDAR returns, occlusions, and pose inconsistency. Epistemic uncertainty is represented through Bayesian parameter variability in the BGNN, using a variational approximation \( q_{\phi}(\theta) \approx p(\theta \mid D) \) and Monte Carlo sampling,

\[
\hat{p}(y \mid x, D) \approx \frac{1}{S}\sum_{s=1}^{S} p(y \mid x, \theta^{(s)}),\quad \theta^{(s)} \sim q_{\phi}(\theta).
\]

Downstream predictions marginalize over sampled parameters to estimate action posteriors such as Brake, SlowDown, or FollowAhead [2504.05908]. In this setting, OSR-CoT is explicitly uncertainty-aware: state variables carry uncertainty into graph reasoning and then into decision rationales.

The BGNN performs probabilistic message passing according to

\[
m_i^{(t)} = \sum_{j \in \mathcal{N}(i)} \alpha_{ij}^{(t)} \, \psi\!\big(h_i^{(t)}, h_j^{(t)}, r_{ij}^{(t)};\theta_\psi^{(t)}\big),
\]

\[
h_i^{(t+1)} = \phi\!\big(h_i^{(t)}, m_i^{(t)};\theta_\phi^{(t)}\big),
\]

with probabilistic attention weights

\[
\alpha_{ij}^{(t)} = \mathrm{softmax}_j\!\Big(f\big(e_{ij}^{(t)};\theta_\alpha^{(t)}\big)\Big).
\]

This permits uncertainty carry-through from node features \( h_i \), edge variability, and Bayesian parameter sampling [2504.05908]. A plausible implication is that PRIMEDrive-CoT operationalizes OSR-CoT not just as an explanatory overlay but as a full probabilistic state-estimation and relational-inference system.

Canvas-of-Thought introduces a different form of grounding: validation by rendering and critique. After each action \( a_t \), the current state is rendered,

\[
V_{\text{state}} = \mathcal{R}(\mathcal{S}_{t+1}),
\]

and a critic computes structured feedback,

\[
h_{t+1} = \mathcal{C}(V_{\text{orig}}, V_{\text{state}}, I).
\]

The validator may be binary, \( V(\mathcal{S}_{t+1}, C) \in \{0,1\} \), or penalty-based through \( \ell_C(\mathcal{S}_{t+1}) \), summarizing attribute errors, false existences, and spatial conflicts [2602.10494]. This is not uncertainty modeling in the Bayesian sense; rather, it is grounded self-correction through explicit state inspection. The paper’s phrase “visual gradient” emphasizes that critique functions as corrective signal at inference time rather than as a training loss [2602.10494].

MMPlanner addresses grounding through cross-modal selection and evaluation. After OSR-CoT generates a state-aware image description, BLIP-2 embeddings of candidate images and the description are compared, and the most aligned image is selected by cosine similarity [2509.21662]. The work also introduces imagetotextrelevance, a VLM-based score in which MiniGPT-4 first describes the image and then scores how well it matches the action and object states in the textual step. Human correlation is reported as \( \rho = 0.57 \), compared with \( \rho = 0.37 \) for CLIPScore [2509.21662]. In MMPlanner, OSR-CoT therefore supports cross-modal validation by producing more discriminative, state-specific prompts for image generation and selection.

## 5. Prompting, inference patterns, and training objectives

OSR-CoT appears in both prompted and model-architectural forms. In MMPlanner, the prompt itself is the main implementation vehicle. The OSR-CoT prompt proceeds in three phases: describe the current step using context from prior steps and the goal; reason about object state changes before and after the step; then write a concise image description containing those changes [2509.21662]. The prompt includes a one-shot in-context example and explicitly enforces “Before” and “After” structure. The paper contrasts this with a simpler prompt that only asks for an image after the current step and reports that the simple prompt tends to hallucinate irrelevant prior details and omit crucial state descriptors such as texture or mixture composition [2509.21662]. OSR-CoT thus functions as a prompt-level state-transition scaffold.

In PRIMEDrive-CoT, by contrast, OSR-CoT is downstream of perception and graph inference rather than being only a prompt template. The algorithmic summary is: preprocess LiDAR and RGB, detect boxes and classes via MVX-Net, compute entropy and orientation deviation, calculate per-object uncertainty \( U_i \), compute risk \( R_i \), construct the graph using node features and edge energies, run the BGNN for \( T \) layers with parameter sampling, aggregate the predictive \( p(a \mid G) \), and then generate a rationale ordered as objects \(\rightarrow\) uncertainty \(\rightarrow\) risk \(\rightarrow\) interactions \(\rightarrow\) decision [2504.05908]. The CoT module is lightweight, under \(1.2\) GFLOPs, and has no LLM dependence [2504.05908]. This is notable because OSR-CoT here is not synonymous with LLM prompting.

The training objectives in PRIMEDrive-CoT are explicit. Detection uses

\[
L_{\text{reg}} = \frac{1}{N}\sum_{i=1}^{N} \left\lVert \hat{b}_i - b_i \right\rVert_2^2,
\]

and, if a classification head is included,

\[
L_{\text{cls}} = -\sum_i \sum_k y_{ik} \log p_{ik}.
\]

The BGNN uses the ELBO-style objective

\[
\mathcal{L}_{\text{BGNN}} =
\mathbb{E}_{q_{\phi}(\theta)}\big[-\log p(D \mid \theta)\big]
+ \mathrm{KL}\big(q_{\phi}(\theta)\,\|\,p(\theta)\big),
\]

with optional action NLL,

\[
L_{\text{act}} = -\sum_n \log p(a_n \mid G_n, D),
\]

and the full training loss

\[
\mathcal{L} = \lambda_{\text{det}}(L_{\text{reg}} + L_{\text{cls}})
+ \lambda_{\text{bgnn}} \mathcal{L}_{\text{BGNN}}
+ \lambda_{\text{act}} L_{\text{act}}.
\]

DriveCoT uses a \(70\%/15\%/15\%\) train/validation/test split, training converges in about \(4.5\) hours on dual RTX 4090 GPUs, and inference runs at about \(38\) ms per frame, roughly \(18.7\) FPS, on an RTX 3090 with about \(41.9\) GFLOPs and less than \(1.2\) GB memory [2504.05908].

Canvas-of-Thought does not introduce supervised losses in the reported system. It is deployed via prompting, with “One Step Thinking,” thought pruning, and budgeted inference. The paper states that CoT uses budget \(1\), PoT, Iterative Reflection, and Canvas-CoT use budget \(6\), and ToT uses budget \(10\), with temperature \(0\) for stable trajectories [2602.10494]. It does, however, note conceptual training objectives that one could define, such as supervised CRUD loss, constraint penalty, and reinforcement signals; these are explicitly presented as conceptual rather than used in the work [2602.10494]. This distinction matters because it marks OSR-CoT as a reasoning format that can be realized either through learning objectives or through inference-time scaffolding.

## 6. Empirical performance and ablation evidence

The three papers report OSR-CoT gains in distinct evaluation regimes. PRIMEDrive-CoT evaluates on DriveCoT validation and reports speed-decision F1 scores of 0.85 for SpeedLimit, 0.82 for FollowAhead, 0.79 for SlowDown, 0.78 for SlowApproach, 0.86 for CautiousTurn, and 0.87 for Brake [2504.05908]. Path accuracy is \(87.6\%\) for Straight, \(77.6\%\) for Turn, and \(82.9\%\) for Lane Change. For LiDAR-based 3D detection, it reports accuracy \(89.39\%\), IoU \(0.78\), F1 \(0.85\), and deviation angle \(3.7^\circ\), compared with baseline VoxelNet at accuracy \(80.47\%\), IoU \(0.67\), F1 \(0.76\), and deviation \(6.1^\circ\) [2504.05908]. A 3-layer BGNN with 128-dimensional embeddings yields the best trade-off, with \(+3.2\%\) F1 and \(-14.5\%\) uncertainty in high-risk detections relative to a smaller BGNN. Removing multi-view RGB leaves detection largely unchanged because the system is LiDAR-primary, but interpretability drops because Grad-CAM overlays and cross-modal verification are lost [2504.05908].

MMPlanner reports state-of-the-art performance on RecipePlan and WikiPlan. On textual planning, mmPurple reaches PlanScore 82.05 on RecipePlan and 84.43 on WikiPlan, with the abstract summarizing a relative improvement of \(+6.8\%\) over baselines [2509.21662]. On cross-modal alignment, imagetotextrelevance improves to 77.07 on RecipePlan and 69.23 on WikiPlan; the abstract reports \(+11.9\%\) improvement [2509.21662]. On visual sequence ordering, RecipePlan accuracy rises to 27.50 and WikiPlan accuracy to 23.43, with the abstract reporting \(+26.7\%\) improvement [2509.21662]. Inference time is 52.02 seconds for mmPurple versus 96.77 seconds for TIP, a reduction of about \(46.25\%\) [2509.21662].

The ablation evidence in MMPlanner is especially relevant for isolating OSR-CoT’s contribution. Adding OSR-CoT to the LLaVa+SD baseline improves RecipePlan cross-modal alignment from 72.02 to 75.15 and visual ordering accuracy from 22.10 to 25.50; adding previous steps and then cross-modal selection raises performance further, culminating in RecipePlan imagetotextrelevance 77.07 and accuracy 27.50 [2509.21662]. A prompt ablation also shows steady gains from a prompt without special structure, to one-shot prompting, to the addition of a step description, and finally to the full state-based OSR-CoT prompt. On RecipePlan, full OSR-CoT reaches cross-modal alignment 77.07, \( \tau = 0.22 \), Dist 6.51, MS 2.39, and WMS 4.99, outperforming partial variants [2509.21662]. This supports the claim that explicit state reasoning, rather than mere prompt length, drives the improvements.

Canvas-of-Thought reports strong gains on VCode, RBench-V, and MathVista. On RBench-V with a GPT-5 backbone, the overall score is 32.4 for Canvas-CoT versus 17.4 for CoT, with Physics 47.1 versus 15.3, Math 48.9 versus 30.7, Counting 27.2 versus 12.3, and Game 16.7 versus 13.8, at a token budget of about \(1.0\)k versus \(0.7\)k for CoT and much lower than ToT or PoT [2602.10494]. On MathVista, Canvas-CoT improves several pass@1 and pass@2 scores for both GPT-5 and Gemini 2.5 Pro. On VCode, it raises MM-Vet average to 49.8 from 42.8 for Iterative Reflection and lifts overall average to 61.2 from 55.4 for CoT; on CV-Bench, SigLIP rises to 72.28, an \(+8.1\%\) improvement over CoT’s 66.88 [2602.10494].

The ablation in Canvas-of-Thought separates the effect of a DOM canvas from the effect of critique. On RBench-V and VCode subsets with GPT-5, Iterative Reflection scores 17.4 overall on RBench-V, adding a DOM canvas without critic increases this to 26.9, and adding critique yields the full 32.4 [2602.10494]. Similar additive gains hold for Gemini 2.5 Pro. This indicates that mutable object state and rendering-based validation contribute distinct benefits.

## 7. Interpretability, limitations, and broader significance

OSR-CoT is consistently presented as an interpretability mechanism, but the meaning of interpretability differs by domain. In PRIMEDrive-CoT, CoT narratives enumerate states, interactions, and uncertainties, while Grad-CAM highlights pedestrians, leading vehicles, occluding objects, brake lights, and lane boundaries to support human-in-the-loop verification [2504.05908]. An example pedestrian-crossing rationale explicitly traces perception, state estimation, risk scoring, interaction-graph construction, BGNN propagation, and a final action: “Brake now, then CautiousTurn after pedestrian clears” [2504.05908]. In this context, interpretability is tied to safety-critical decision auditability.

In MMPlanner, interpretability is bound to cross-modal consistency. The OSR-CoT prompt compels the model to articulate “Before” and “After” object states, including implicit changes not spelled out in the current textual step. Qualitative cases show that this helps depict “crumbly” texture, previously mixed bowl contents, “shredded cheese,” and a baked loaf even when these are not explicitly re-mentioned in the local step text [2509.21662]. This suggests that OSR-CoT functions as a temporal grounding device for multimodal generation, helping preserve latent procedural state across steps.

In Canvas-of-Thought, interpretability becomes externalized state transparency. Because the DOM is the single source of truth, with ID-addressable objects and explicit history of tool actions, the reasoning process is inspectable as a sequence of local object edits rather than a long text narrative [2602.10494]. The paper argues that this reduces serialization tax, avoids cascade errors, and supports non-monotonic corrections. A plausible implication is that OSR-CoT here approaches a white-box world-model paradigm for multimodal reasoning.

The limitations also differ. PRIMEDrive-CoT notes modest but non-zero computational overhead from BGNN sampling, false positives under extreme occlusion, myopic proximity risk that does not capture long-horizon tail risks such as CVaR, and domain shift beyond CARLA/DriveCoT that may require re-tuning \(\lambda\) values and priors [2504.05908]. MMPlanner notes that hallucinations remain possible, visual consistency for peripheral elements is not explicitly enforced, Stable Diffusion may fail on concepts absent from training data such as “beaten egg,” and PlanScore calibration can be imperfect because of LLM evaluator limitations [2509.21662]. Canvas-of-Thought identifies difficulties with complex 3D or occlusion-heavy scenes, dependence on external parsing, rendering, and critique modules, limited validator scope for subtle constraints, and possible reduced benefit on tasks where textual constraints dominate over visualizable state [2602.10494].

Across the three papers, OSR-CoT also has a broader methodological significance. PRIMEDrive-CoT explicitly argues that the framework generalizes to mobile robotics, healthcare, and logistics by redefining object states \( s_i \), edge energies \( e_{ij} \), and risk \( R \) [2504.05908]. MMPlanner suggests applicability to robotics and industrial workflows, especially where multimodal goals and step contexts are available [2509.21662]. Canvas-of-Thought proposes richer scene graphs, typed relations such as symmetry or tangency, formal geometry or SAT/SMT solvers, probabilistic state tracking, and broader substrates such as CAD systems, physics simulators, or spreadsheet-like state representations [2602.10494]. Taken together, these works indicate that OSR-CoT is best understood not as a domain-specific prompt trick, but as a general design principle for grounding reasoning in explicit stateful object models.

Source: https://www.emergentmind.com/topics/object-state-reasoning-chain-of-thought-osr-cot