Iterative Perception (ITP) Overview

• Iterative Perception (ITP) is a process where perception is refined through repeated action–perception cycles that leverage diagnostic feedback signals.
• ITP spans embodied interactions, where actions change the environment, and internal iterations that update latent representations for better inference.
• Contemporary ITP systems implement loops combining Bayesian updates, recurrent computations, and policy–critic models to enhance robustness and accuracy.

Iterative Perception (ITP) denotes perception organized as a repeated refinement process rather than a single-pass estimate. In the robotics literature surveyed in 2016, however, the standard term is Interactive Perception (IP), not ITP: IP is the principle that purposeful action creates diagnostic sensory signals and that repeatable regularities in the joint space of sensory data and action parameters over time, $S \times A \times t$, make prediction and interpretation simpler and more robust (Bohg et al., 2016). Subsequent papers use the language of ITP more explicitly in collaborative perception, large vision-LLMs, navigation, and recurrent vision backbones. This suggests that ITP is best understood as a cross-domain pattern—iterative refinement of perception through action, feedback, or recurrent computation—rather than as a single standardized field label (Huang et al., 2024, Wei et al., 9 Apr 2025, Yan et al., 25 Sep 2025).

1. Terminology, scope, and conceptual boundaries

In robotics, the established field name is Interactive Perception. If “Iterative Perception” is meant as repeated action–perception cycles that progressively refine a robot’s understanding, that idea is central to IP; if it is meant as repeated updates of a passive estimator on fixed data, it is not the same thing (Bohg et al., 2016). The survey makes the distinction explicit: passive vision may iteratively re-run an optimizer on fixed data, whereas IP changes the data by acting, and conditions interpretation on action parameters. The same source also distinguishes IP from Active Perception (AP): AP traditionally focuses on manipulating sensor configuration such as gaze control or viewpoint changes, whereas IP extends this to forceful interactions that change the environment itself (Bohg et al., 2016).

Two core principles define the classical robotics notion. The first is Purposeful action creates diagnostic sensory signals, sometimes summarized as CNS-only usage in the taxonomy: pushes, pulls, taps, shakes, and openings generate visual, haptic, or audio signals that would not exist under passive sensing. The second is Action–Perception Regularity (APR): there are repeatable regularities in how action parameters causally shape sensory responses over time, which simplifies prediction, belief updating, and informative action selection (Bohg et al., 2016). These principles make the loop inherently iterative: each action changes the observation, and the changed observation informs the next action.

Later literature broadens the label. In RoCo, ITP names an iterative object matching and pose adjustment loop for collaborative autonomous driving (Huang et al., 2024). In RePer, it denotes policy–critic reflection rounds in large vision-LLMs (Wei et al., 9 Apr 2025). In VideoChat-R1.5, it refers to visual test-time scaling through repeated clue-guided re-sampling of temporal segments and spatial regions (Yan et al., 25 Sep 2025). In Perceiver, PGT, and PRISM, the term is not always explicit, but the mechanism is iterative refinement of latent, group-token, or slot-memory representations (Jaegle et al., 2021, Deng et al., 2023, Wang et al., 29 May 2026). This suggests two broad regimes: embodied iterative perception, in which the agent changes the world or its sensing configuration, and internal iterative perception, in which the model revisits and revises intermediate representations.

2. Common formal structure

The canonical robotics formulation uses a partially observable control loop. With state $s \in S$, action $a \in A$, observation $o \in O$, transition model $T(s' \mid s,a)$, observation model $O(o \mid s')$, and belief $b(s)$, the Bayesian filter update is

$$b'(s') \propto O(o \mid s') \sum_s T(s' \mid s,a)\, b(s).$$

Informative action selection can then be written as expected entropy reduction,

$$a^{*} = \arg\max_a \ \mathbb{E}_o \left[ H(b) - H\big(b' \mid a,o\big) \right],$$

and sensorimotor regularities can be learned through predictive models of the form

$$\hat{o} = f_\theta(s,a), \qquad \min_\theta \ \mathbb{E}_{(s,a,o)}\left[\ell\big(f_\theta(s,a),o\big)\right].$$

The same survey frames IP as a POMDP $s \in S$0, where action selection trades off exploration and exploitation, and notes that exact solutions are intractable, so myopic, finite-horizon, and approximate global policies are common (Bohg et al., 2016).

Contemporary ITP systems instantiate this template with different state variables and update operators. RePer defines a state $s \in S$1, where a policy model $s \in S$2 produces a perception output and a critic $s \in S$3 returns feedback and a score; the loop stops when a score threshold is reached or a maximum number of rounds is exhausted (Wei et al., 9 Apr 2025). VideoChat-R1.5 expresses the next perceptual input as a function of the current one and the current textual hypothesis,

$s \in S$4

so the answer and the next visual sample are explicitly coupled (Yan et al., 25 Sep 2025). Perceiver repeatedly alternates asymmetric cross-attention from a latent array to the input with latent self-attention and MLP updates, summarized in the paper as

$s \in S$5

which implements re-entrant refinement at cost $s \in S$6 rather than full $s \in S$7 self-attention over inputs (Jaegle et al., 2021). PRISM similarly uses a recurrent organize–recall–refine loop, with object-centric slots, vector-quantized memory, and Adaptive Computation Time determining how many iterations a stage should run (Wang et al., 29 May 2026).

Across these formulations, the common invariant is not a particular architecture but a causal sequence: provisional estimate, targeted update, revised estimate, and an explicit rule for deciding what to process next.

3. Robotics origins and embodied realizations

The survey literature defines the core robotics agenda: manipulation-driven segmentation, pose estimation, articulation discovery, dynamics and material-property inference, tactile exploration, 3D reconstruction, and next-best-view/control all benefit from acting on the environment rather than merely observing it (Bohg et al., 2016). Representative mechanisms include pushing, poking, shaking, probing, tapping, sliding, grasping, opening articulated objects, and active viewpoint control with interaction. Across these tasks, interaction introduces motion or contact cues, reveals constrained motions and force responses, supports multimodal fusion, and improves identifiability and robustness relative to passive baselines (Bohg et al., 2016).

A particularly direct instantiation appears in ecological perception and affordance-map learning on PR2. In “Bootstrapping Robotic Ecological Perception from a Limited Set of Hypotheses Through Interactive Perception,” the robot builds a relevance map indicating the moveable parts of the scene, uses supervoxels represented by a 48-dimensional feature vector, trains Collaborative Mixture Models (CMMs) online, and selects the next region with which to interact according to a choice distribution that combines uncertainty and confidence (Goff et al., 2019). The loop is explicit: sense and segment, classify and compute relevance, evaluate uncertainty, act with a push primitive, detect change, update the classifier, and recompute the map. The minimal assumption is that “objects are parts of the environment that the robot can move,” so object categories and boundaries are not predefined (Goff et al., 2019).

“Building an Affordances Map with Interactive Perception” extends this logic from movability to affordances formalized as relations between action primitives, effects, and local visual features (Goff et al., 2019). The system oversegments RGB-D input into supervoxels with VCCS, represents each supervoxel by a 48-dimensional feature composed of a 15-dimensional CIELab histogram and a 33-dimensional FPFH descriptor, and learns a probabilistic map for pushable objects, activable push-buttons, and liftable objects. The operational loop is “segment, predict, select, act, detect, update,” with uncertainty-driven exploration and immediate online dataset augmentation. Liftability is treated as composite and filtered by pushability, illustrating how iterative perception can be chained across affordances rather than limited to a single predicate (Goff et al., 2019).

“Object Finding in Cluttered Scenes Using Interactive Perception” replaces hand-designed exploration heuristics with reinforcement learning (Novkovic et al., 2019). RGB-D observations and a target-color detector are fused into a TSDF and detection map, compressed into a 71-dimensional state centered on the end-effector frame, and mapped by PPO to continuous end-effector displacements plus a termination decision. The system closes both an active and interactive loop: camera motions strategically gather viewpoints, and end-effector motions physically reveal occluded objects. The abstract reports that the system, trained in simulation only, transfers smoothly to reality and can solve the task efficiently with more than 88% success rate (Novkovic et al., 2019).

Deformable-object manipulation intensifies the need for embodied iteration because of self-occlusions, underactuated dynamics, and partial observability. “Interactive Perception for Deformable Object Manipulation” formulates the problem as sequential decision making with a dual-arm setup: one UR10e arm manipulates a bag, and the other carries an active camera constrained to a hemispherical manifold (Weng et al., 2024). Its central construct, the Dynamic Active Vision Space (DAVS), is a time-varying manifold-with-boundary on the viewing hemisphere computed from the current structure of interest. The factorized policy $s \in S$8 exploits camera–manipulator coupling, and PPO uses visibility-related potential shaping to prefer actions that reveal the in-bag cube and the bag opening (Weng et al., 2024). Here, the iterative character lies not only in repeated belief revision but in repeated reconfiguration of both viewpoint and deformable state.

4. Contemporary extensions beyond classical interactive robotics

System	Domain	Iterative mechanism
RoCo (Huang et al., 2024)	Collaborative autonomous driving	object matching → pose graph optimization → pose update → re-matching
Perceiver (Jaegle et al., 2021)	Modality-agnostic perception	asymmetric cross-attention and latent self-attention interleaved over multiple steps
Perceptual Group Tokenizer (Deng et al., 2023)	Self-supervised vision	group formation → assignment → token update → context re-hypothesis
RePer (Wei et al., 9 Apr 2025)	Large vision-LLMs	policy model generates perceptions; critic model evaluates them with scores and fine-grained feedback
ST-Booster (Yue et al., 14 Apr 2025)	Vision-and-language navigation in continuous environments	dual-map update, multi-granularity aligned fusion, and Guided Attention Heatmaps
VideoChat-R1.5 (Yan et al., 25 Sep 2025)	Video MLLM inference	coarse answer and clues, then focused re-sampling of temporal segments and ROI crops
PRISM (Wang et al., 29 May 2026)	Recurrent vision backbone	organize–recall–refine with slot memory, prototype memory, and ACT

RoCo exemplifies a non-manipulative but still externally grounded ITP loop. Multiple agents share LiDAR-based features, object boxes, and poses; pose noise produces misalignment, so RoCo iterates between graph-guided object association and uncertainty-weighted pose-graph optimization until matching converges (Huang et al., 2024). The loop refines both correspondences and relative poses before final feature aggregation. Its notion of ITP is therefore neither passive optimization on fixed features nor classical forceful IP, but repeated correction of the perceptual geometry that underlies collaboration.

ST-Booster instantiates ITP in navigation as a temporal loop over global and local maps. At each step, a topological graph $s \in S$9 and egocentric grid $a \in A$0 are updated, aligned with instruction tokens through TCMT and GCMT, fused bidirectionally by Cell2Node and Node2Cell, and then used to generate Guided Attention Heatmaps that bias waypoint sampling (Yue et al., 14 Apr 2025). The paper does not explicitly use the term ITP, but it describes an “iterative spatiotemporal booster,” and the mechanism is unmistakably iterative: map construction, instruction alignment, heatmap guidance, action selection, observation update, and repetition.

RePer and VideoChat-R1.5 move ITP into multimodal LLMs. RePer alternates a policy and a critic so that a provisional caption or answer is evaluated with rationale and a scalar score, then revised in later rounds; Reflective Perceptual Learning trains the policy to imitate this improvement trajectory through reward-weighted sequence-level unlikelihood (Wei et al., 9 Apr 2025). VideoChat-R1.5 shifts the emphasis from textual reflection to perceptual test-time scaling: the model first emits tentative spatio-temporal clues, then re-samples densely inside predicted temporal windows and re-encodes cropped regions while preserving global context, repeating the cycle for a small budget of passes (Yan et al., 25 Sep 2025).

Perceiver, PGT, and PRISM represent a more internal notion of ITP. Perceiver repeatedly distills very large inputs into a latent bottleneck through cross-attention and re-entrant latent processing (Jaegle et al., 2021). PGT performs repeated grouping cycles in which sampled group tokens hypothesize context, bind input tokens, update the groups through GRU-based aggregation, and then refine the input features themselves (Deng et al., 2023). PRISM uses object-centric slots, vector-quantized stage-wise memory, and iterative redistribution back to spatial tokens to recover missing evidence under occlusion (Wang et al., 29 May 2026). These are not interactive in the robotics sense, because they do not change the world; instead, they iteratively revisit internal scene organization. This suggests that contemporary ITP spans a continuum from action-conditioned sensing to recurrent representational inference.

5. Empirical evidence and evaluation practices

The 2016 survey identifies a stable evaluation vocabulary for interactive and iterative perceptual systems: perception accuracy, uncertainty reduction, sample efficiency, policy effectiveness, downstream success rates, and robustness under clutter, occlusion, or noise relative to passive baselines (Bohg et al., 2016). It also notes evidence from biology and argues that interaction improves accuracy, robustness, identifiability, and sample complexity when latent properties such as weight or compliance matter for manipulation (Bohg et al., 2016). Later papers instantiate these evaluation axes in task-specific ways.

In collaborative perception, RoCo reports Average Precision at IoU $a \in A$1 and $a \in A$2 on DAIR-V2X and V2XSet under Gaussian pose noise (Huang et al., 2024). On DAIR-V2X [email protected], highlighted numbers include 76.3 vs 74.6 with no noise, 71.9 vs 70.0 at $a \in A$3, and 71.5 vs 69.2 at $a \in A$4; on [email protected], 62.0 vs 60.4 with no noise, 58.2 vs 57.0 at $a \in A$5, and 57.8 vs 56.9 at $a \in A$6 (Huang et al., 2024). The paper also reports qualitative reduction of pose errors and missed detections.

In embodied navigation, ST-Booster reports trajectory length, navigation error, oracle success rate, success rate, and SPL (Yue et al., 14 Apr 2025). On val unseen, it reports SR 61%, OSR 68%, NE 4.77m, and SPL 50%; on test unseen, SR 59%, SPL 50%, NE 4.83m, and OSR 66% (Yue et al., 14 Apr 2025). Under synthetic disturbances, the model maintains higher SR and SPL than the baseline in waypoint, global-perception, and local-perception disturbance settings, supporting the claim that iterative dual-map fusion and GAH guidance improve robustness (Yue et al., 14 Apr 2025).

In LVLM reflection, RePer evaluates detailed captioning, hallucination, and image-understanding benchmarks (Wei et al., 9 Apr 2025). On HallusionBench with the 13B backbone, accuracy improves from 14.95 to 20.00, aAcc from 43.85 to 51.00, and fAcc from 20.81 to 22.83. On MMHal-Bench, the 7B score rises from 2.02 to 2.51 while hallucination rate decreases from 0.61 to 0.53; the 13B score rises from 2.35 to 2.61 while hallucination rate decreases from 0.58 to 0.52. On DetailCaps-4870, Recall improves from 42.19 to 45.69 for 7B and from 43.77 to 49.10 for 13B (Wei et al., 9 Apr 2025). These metrics operationalize a form of iterative perceptual correction rather than physical interaction.

For recurrent visual backbones, PRISM emphasizes robustness under incomplete observations (Wang et al., 29 May 2026). On ImageNet-1K, it reports 80.3% Top-1 on clean inputs, 69.1% under PatchMask0.6, 44.6% under PatchMask0.8, 78.9% under BlockMask112, 69.2% under OnlyOne112, and 33.8% under OnlyOne56. On COCO with Mask R-CNN, it reports clean $a \in A$7 44.5 and $a \in A$8 40.8, degrading to 37.2 and 33.7 under M50 occlusion; the paper stresses that gains over a feed-forward baseline are larger under occlusion than on clean images (Wang et al., 29 May 2026). The reported feature-similarity analyses and halting dynamics are intended as evidence that iteration is most valuable when evidence is missing.

Embodied robotics papers report correspondingly embodied outcomes. The object-finding system in clutter reports I3DE 88.0% success in simulation on piles of cubes and 100% success in real-world experiments, with fewer steps than the hand-crafted exhaustive baseline (Novkovic et al., 2019). The affordance-map system reports that convergence typically occurred between 60 and 100 interactions, with accuracy up to about 0.8 in the better runs for pushable, push-button, and liftable affordances (Goff et al., 2019). These results fit the survey’s broader claim that interaction is most beneficial in clutter, occlusion, and tasks where latent physical properties matter (Bohg et al., 2016).

6. Misconceptions, limitations, and open problems

A persistent misconception is that any repeated computation over visual features counts as ITP in the same sense across fields. The robotics survey rejects this equivalence: passive iterative refinement re-runs an optimizer on fixed data, whereas IP changes the data by acting and exploits action–perception regularities (Bohg et al., 2016). A second misconception is to collapse ITP into Active Perception. AP can improve sensing by manipulating sensor configuration; IP extends this to forceful interactions that change the environment itself (Bohg et al., 2016). These distinctions matter because they determine which models, uncertainties, and control objectives are required.

The survey’s open problems remain structurally central: scalability to many objects and long horizons, safe and efficient exploration, policy learning under uncertainty with limited interaction data, expressive yet tractable uncertainty models, sim-to-real transfer, generalization across objects and environments, real-time constraints, and standardized benchmarking (Bohg et al., 2016). Robotics case studies expose these issues concretely. The ecological-perception and affordance-map papers note supervoxel instability, depth noise, action inaccuracy, mislabeled samples, and the requirement that overlap testing in CMMs needs more samples than feature dimension, which can delay structural adaptation (Goff et al., 2019, Goff et al., 2019). The deformable-object work frames the problem as a POMDP but does not maintain an explicit belief state in implementation; it also assumes accurate structure-of-interest extraction and a constrained hemispherical camera manifold (Weng et al., 2024).

Contemporary non-robotic systems inherit analogous problems in different guises. RoCo can stagnate under sparse overlaps or poor detections, is sensitive to $a \in A$9, $o \in O$0, and $o \in O$1, and assumes sufficiently static geometry during a time slice (Huang et al., 2024). RePer can suffer critic overfitting, feedback drift, and compounding errors when early feedback is incorrect; the paper also identifies stopping rules and uncertainty calibration as future directions (Wei et al., 9 Apr 2025). ST-Booster warns that coarse-grained Guided Attention Heatmaps can over-weight regions, that grid size trades off redundancy against noise, and that strong offline pretraining on expert trajectories may limit adaptability (Yue et al., 14 Apr 2025). VideoChat-R1.5 shows a roughly linear latency increase with the number of perception passes and highlights fast motion, heavy occlusion, and dispersed long-horizon clues as difficult cases (Yan et al., 25 Sep 2025). PRISM notes that compute increases on difficult inputs and that robustness results rely on synthetic occlusion protocols rather than a broader corruption suite (Wang et al., 29 May 2026).

Taken together, these limitations indicate that the decisive technical question is not whether perception is iterative, but how the loop is grounded: in embodied action, in cross-agent geometric correction, in policy–critic reflection, in clue-guided re-sampling, or in recurrent latent organization. The historical robotics literature reserves its strongest claims for the embodied form, where acting changes the evidence itself (Bohg et al., 2016). Contemporary usage broadens the term, but the conceptual center remains the same: perception improves when the system can use intermediate uncertainty or error signals to decide what to sense, update, or revisit next.