Perception Simulation Model Overview

• Perception Simulation Models are computational systems that transform natural images into condition-specific perceptual states using clinically grounded perturbations.
• They employ diverse architectures—from CNNs to Vision Transformers—to capture global context and simulate conditions like prosopagnosia and tunnel vision.
• These models integrate image transformation, autoregressive inference, and physical world simulation, offering insights for clinical, autonomous, and robotics applications.

In recent arXiv usage, a perception simulation model denotes a computational system that simulates perceptual outputs, perceptual states, or perception-conditioned scene representations from natural images, ground-truth scene descriptions, or partial sensory evidence. The most direct formulation appears in the Perceptual Reality Transformer (PRT), a conditional image-to-image framework that transforms ordinary natural images into approximations of how the world may appear under specific neurological or psychiatric perception conditions; adjacent literatures extend the same general idea to future-scene inference from incomplete observations, direct simulation of perception-system outputs, and internal simulation of perceptual states for cognition and action prediction (Lin, 13 Aug 2025).

1. Scope and conceptual boundaries

The Perceptual Reality Transformer is presented explicitly as a perception simulation model. Its purpose is not to generate stylized art, but to create scientifically grounded perceptual analogs that can help clinicians, students, caregivers, and technologists understand “what it might feel like” to perceive through the lens of a disorder. In that formulation, the model sits at the intersection of computer vision, computational neuroscience, and empathy-focused medical technology. PRT simulates eight neurological perception conditions mentioned in the paper: simultanagnosia, prosopagnosia, ADHD attention deficits, visual agnosia, depression-related perceptual changes, anxiety tunnel vision, Alzheimer’s memory effects, and related visual recognition/perceptual disruptions (Lin, 13 Aug 2025).

A broader reading of the literature suggests that the term covers more than condition-specific image transformation. P3Sim, for example, is proposed as a perception simulation model because it does not assume an idealized simulator with complete access to the scene’s full 3D state; instead, it predicts future scene states from partial inputs and incomplete 3D transformation signals. Its inputs include images, depth, optical flow, camera pose, and incomplete object-motion signals, and its output is a probabilistic simulation of what the scene could become next (Lee et al., 25 Jun 2026). In autonomous driving, Perception Error Models (PEMs) and related perception-informed Software-in-the-Loop systems simulate the effect of sensing and perception on the object list that the driving policy receives, without simulating raw sensor physics directly (Piazzoni et al., 2023).

This scope also includes low-level mechanistic models and internal cognitive models. A difference-of-Gaussians retinal stage followed by zero-crossing detection has been used to simulate stochastic resonance in early vision, thereby modeling noise-enhanced contrast sensitivity (Kundu et al., 2010). Semiotics Networks Representing Perceptual Inference and A-SOM-based intention-prediction systems simulate perceptual inference internally, either as convergence to a perceptual attractor or as continuation of an incomplete action-related perceptual sequence (Kupeev et al., 2023, Gharaee, 2022). A plausible implication is that the field does not define perception simulation at a single representational level: it ranges from low-level receptive-field models to task-level object-map simulators and high-level conditional neural generators.

2. Formal representations

PRT frames perception simulation as a conditional transformation problem. Given an image $I \in \mathbb{R}^{H \times W \times 3}$, a condition label $c \in \{0,1,\dots,7\}$, and severity $s \in [0,1]$, the model learns

$$f_{\theta}: \mathbb{R}^{H \times W \times 3} \times \{0, 1, ..., 7\} \times [0, 1] \rightarrow \mathbb{R}^{H \times W \times 3}$$

with $f_{\theta}(I,c,s)=I'$, where $I'$ is the simulated perceptual output. This formulation makes the simulation controllable: the same image can be mapped into different perceptual states, and the severity parameter allows graded progression from mild to severe symptoms (Lin, 13 Aug 2025).

P3Sim adopts a different formal layer. The scene is modeled as a set of random variables $\{x_p\}_{p\in\mathcal{P}}$, where each pointer $p$ identifies a persistent local scene element such as a spatial patch, a time index, or a modality like RGB, depth, or optical flow. The observed state is a partial function $$X:\mathrm{dom}(X)\subseteq\mathcal{P}\rightarrow\mathcal{V}$$, and the physical world model infers missing variables according to

$$\Psi:\,(X,\, p\notin\mathrm{dom}(X)) \mapsto \{\Pr[(p,v)\mid X]\mid v\in\mathcal{V}\}.$$

The same paper converts this graphical-model view into an autoregressive sequence model with pointer–value pairs, so that inference becomes

$c \in \{0,1,\dots,7\}$0

Here, perception simulation is probabilistic completion and future-state inference rather than deterministic image transformation (Lee et al., 25 Jun 2026).

In autonomous-driving virtual testing, the formal abstraction is often object-level. PEM defines the perceived world as

$c \in \{0,1,\dots,7\}$1

where $c \in \{0,1,\dots,7\}$2 is the world or ground truth set, $c \in \{0,1,\dots,7\}$3 is the object map from perception, and $c \in \{0,1,\dots,7\}$4 is the perception error. Downstream response is then

$c \in \{0,1,\dots,7\}$5

This representation makes explicit that the simulated quantity is not raw sensing but the deviation between world state and perceived world, with false negatives, false positives, misclassification, and parameter or localization error treated as distinct error types (Piazzoni et al., 2023).

Internal-simulation models use yet another formal vocabulary. In semiotic networks, perceptual inference is an observed-to-seen operator iterated until convergence:

$c \in \{0,1,\dots,7\}$6

with awareness defined by the fixed-point condition $c \in \{0,1,\dots,7\}$7. In the A-SOM intention-prediction architecture, internally simulated perceptual states are represented by action pattern vectors derived from the sequence of winning neurons over a skeletal action sequence, allowing continuation of the perceptual trajectory when native input is incomplete (Kupeev et al., 2023, Gharaee, 2022).

3. Architectural realizations

PRT benchmarks six neural architecture families, spanning classical convolutional, residual, transformer-based, recurrent, diffusion-based, and latent generative approaches. Each family corresponds to a different computational hypothesis about how perceptual distortions might be modeled (Lin, 13 Aug 2025).

Architecture family	Defining mechanism	Stated role
EncoderDecoderCNN	Convolutional encoder with mirrored decoder; condition and severity embeddings broadcast into feature maps	Simplest baseline
ResidualPerceptual	Predicts perturbations rather than a full image	Preserves image identity while introducing disorder-specific distortions
ViTPerceptual	Pretrained Vision Transformer (ViT-Base/16), $c \in \{0,1,\dots,7\}$8 patches, 12 transformer blocks, 768-dimensional embeddings, learned condition tokens in self-attention	Captures global context
RecurrentPerceptual	CNN features followed by LSTM layers	Mimics progressive or time-evolving symptoms
DiffusionPerceptual	Conditional DDPM-style U-Net with condition embeddings at multiple scales and a linear beta schedule from 0.0001 to 0.02 over 100 timesteps	Diffusion-based conditional simulation
GenerativePerceptual	VAE latent space with condition and severity embeddings combined with sampled latent codes before decoding	Latent generative approach

Among these, ViTPerceptual is the paper’s best-performing model. The interpretation given is that transformer architectures are especially effective for neurological simulation because they capture global context, which is essential for conditions like simultanagnosia and tunnel vision where relations among objects and scene-wide structure matter (Lin, 13 Aug 2025).

Perception simulation architectures also extend beyond image-to-image neural networks. P3Sim is composed of three interacting components: a learned physical world model $c \in \{0,1,\dots,7\}$9, a geometric conditioning module $s \in [0,1]$0, and a persistent scene memory $s \in [0,1]$1. $s \in [0,1]$2 converts known or partially known transformations into optical flow and sparse target depth; $s \in [0,1]$3 uses those prompts, together with current perceptual tokens, to infer missing scene variables; and $s \in [0,1]$4 accumulates the resulting estimates over time, preserving consistency across frames. The update rule

$s \in [0,1]$5

makes temporal coherence an explicit architectural object rather than an emergent property of a feed-forward model (Lee et al., 25 Jun 2026).

Other realizations are mechanistic or causal-probabilistic rather than purely neural. A low-level visual model based on a centre-surround receptive field, implemented as a difference-of-Gaussians filter modified by a Dirac delta term, followed by zero-crossing detection, simulates stochastic resonance without invoking higher cognition (Kundu et al., 2010). In automotive testing, causal Bayesian networks embedded in standardized scenario simulators map fog, rain, darkness, object proximity, and object overlap to contrast loss, sharpness degradation, and then to loss of detection, sizing errors, positioning offsets, and object merging (Fei et al., 5 Jun 2026). A plausible implication is that perception simulation architectures are best grouped by what they simulate—subjective appearance, future scene state, or functional insufficiency—rather than by whether they are neural, graphical, or mechanistic.

4. Supervision, conditioning, and fidelity

PRT constructs paired training data by turning natural images from ImageNet and CIFAR-10 into condition-specific targets using clinically grounded perturbation functions. These functions define the target perceptual state for each condition. Simultanagnosia is modeled as adaptive fragmentation that preserves objects but disrupts spatial or global scene integration; prosopagnosia as face-specific perturbation applied to detected face regions; ADHD as random visual elements and temporal variation to emulate attentional fluctuation; depression as reduced brightness and saturation with blue-shifting; anxiety tunnel vision as radial masking with exponential falloff; and Alzheimer’s effects as progressive blur, noise, and fading that intensify with severity. Training uses the composite loss

$s \in [0,1]$6

combining reconstruction loss, diversity loss, and severity loss. The system is trained for 50 epochs using AdamW with learning rates in the range of $s \in [0,1]$7 to $s \in [0,1]$8, with early stopping patience of 10 epochs (Lin, 13 Aug 2025).

The same paper evaluates fidelity along five axes: Reconstruction Quality (MSE), Condition Diversity, Severity Scaling, Literature Consistency, and Perceptual Distance (LPIPS). Across CIFAR-10 and ImageNet, ViTPerceptual performed best overall. On CIFAR-10 it achieved the lowest reconstruction MSE among the listed methods and the strongest severity scaling; on ImageNet it again had the best reconstruction MSE and very strong severity correlation. The CNN baseline was competitive and, on ImageNet, had the highest diversity and strong literature consistency. The VAE produced balanced but generally weaker results than ViT, the LSTM-based recurrent model was not as strong overall, and the diffusion model performed poorly for this specific task, with high reconstruction error and even negative severity correlation on CIFAR-10. ResidualNet, ViT, and the CNN baseline were also reported as the most stable across CIFAR-10 and ImageNet via coefficient of variation (Lin, 13 Aug 2025).

Adjacent literatures use different fidelity criteria, but the same methodological pattern recurs: simulation quality is assessed at the task level rather than by appearance alone. A physics-based automotive perception digital twin quantifies object detection with COCO-style average precision and defines $s \in [0,1]$9 as the object distance at which average precision falls to 0.50, showing that higher $$f_{\theta}: \mathbb{R}^{H \times W \times 3} \times \{0, 1, ..., 7\} \times [0, 1] \rightarrow \mathbb{R}^{H \times W \times 3}$$0 generally improves detection while nighttime illumination remains a stronger limiting factor than resolution (Liu et al., 2023). A perception-and-prediction simulator for self-driving testing evaluates not only AP, ADE, and FDE but also planner trajectory deviation, collision simulation metrics, jerk, acceleration, and behavior mismatch rate, because the simulated outputs are meant to test motion planning rather than visual realism (Wong et al., 2020). Rare-event testing based on perception error models replaces raw-sensor fidelity with accurate estimation of downstream safety-violation probabilities under adaptive importance sampling (Innes et al., 2022). This suggests that “fidelity” in perception simulation is domain-dependent: in some settings it means perceptual plausibility, and in others it means faithful reproduction of downstream functional consequences.

5. Application domains and exemplary systems

In medical and human-centered contexts, the most explicit application is neurological and psychiatric perception simulation. PRT positions its contributions as the first systematic benchmark for neurological perception simulation using deep learning and identifies immediate applications in medical education, empathy training, and assistive technology development. Its central claim is that it maps normal images into condition-specific perceptual states using clinically motivated perturbations and that Vision Transformers, with their global context modeling, are particularly well suited to simulating altered human perception (Lin, 13 Aug 2025).

In early-vision modeling, perception simulation can operate at a much lower level. A centre-surround receptive field followed by zero-crossing detection has been shown, through simulation, to exhibit stochastic resonance. The model uses additive Gaussian noise, a synthetic sinusoidal grating, and a threshold criterion based on the appearance of the second harmonic in the Fourier spectrum of the zero-crossing image. Threshold contrast is minimal at a non-zero noise level, so contrast sensitivity, defined as $$f_{\theta}: \mathbb{R}^{H \times W \times 3} \times \{0, 1, ..., 7\} \times [0, 1] \rightarrow \mathbb{R}^{H \times W \times 3}$$1, is maximized at an optimal amount of noise (Kundu et al., 2010). This is not a disorder simulator, but it is still a perception simulation model in the sense that it tries to explain how perceptual enhancement may arise from a nonlinear early-vision computation.

Autonomous-driving research uses perception simulation primarily for safety assessment and system validation. PEM-based virtual testing replaces explicit sensor simulation with a surrogate of the combined sensing-and-perception stack and studies how the resulting object-map errors affect the driving policy. Related work injects realistic loss of detection, sizing inaccuracies, positioning offsets, and object merging into Software-in-the-Loop environments by means of causal Bayesian networks conditioned on fog, rain, darkness, and object overlap. Another line learns to simulate the outputs of deployed perception and prediction systems directly from HD maps, bounding boxes, and trajectories, so that motion planners can be tested with realistic misses, false positives, mislocalization, and multimodal forecasting errors without raw sensor rendering (Piazzoni et al., 2023, Fei et al., 5 Jun 2026, Wong et al., 2020). The common feature is that the simulated object stream, not the sensor, becomes the planner’s perceptual input.

Perception simulation also appears as future-scene inference under incomplete information. P3Sim generalizes across novel view synthesis, object manipulation, and dynamic scene prediction by combining learned multimodal inference with explicit geometric conditioning and persistent scene memory. The paper also demonstrates the same simulator qualitatively on joint camera and object motion, deformable object manipulation, collision reasoning, multi-agent action prediction, and amodal completion or complete scene description (Lee et al., 25 Jun 2026).

Robotics and synthetic-data generation provide additional instantiations. Underwater active perception uses a modified Blender renderer with Fournier–Forand scattering, ten water types, and an MLP that predicts channel-wise contrast from distance to target, artificial illumination intensity, calibration inputs, current-operation inputs, and optimization variables, enabling adaptive distance-and-light control under turbidity and backscattering (Cardaillac et al., 23 Apr 2025). OceanSim uses NVIDIA Isaac Sim, Omniverse Replicator, and GPU ray tracing to render underwater cameras and imaging sonar with attenuation, backscatter, material-dependent sonar reflectance, and range-dependent noise, achieving real-time sonar rendering and fast synthetic data generation (Song et al., 3 Mar 2025). RoCo-Sim generates multi-view consistent roadside data by combining camera extrinsic optimization, a Multi-View Occlusion-Aware Sampler, DepthSAM, and a scalable post-processing toolkit, and it is presented as the first simulation framework for roadside collaborative perception (Du et al., 13 Mar 2025).

Internal cognitive models constitute another branch. Semiotic networks define perception as iterative transformation from observed to seen, converging to a percept image that satisfies a fixed-point awareness property, and use this operator to build perceptualized image classifiers (Kupeev et al., 2023). A hierarchical A-SOM architecture predicts intended human action by internally simulating perceptual states represented as action pattern vectors when sensory input is limited, and it reports that applying internally simulated perceptual states improves recognition performance in all experiments (Gharaee, 2022). These systems do not simulate cameras or object detectors; they simulate the internal continuation or stabilization of perceptual state itself.

6. Limitations, controversies, and research directions

The main limitation of PRT is already implicit in its construction. Its targets are generated by condition-specific perturbation functions grounded in clinical literature, so the simulations approximate the symptom structure rather than merely create visual effects. This suggests a persistent epistemic boundary: the generated outputs are scientifically motivated perceptual analogs, not direct measurements of subjective first-person phenomenology (Lin, 13 Aug 2025).

In safety-critical engineering, the central trade-off is between tractability and raw-sensor realism. PEMs, perception-and-prediction simulators, and causal perception-informed SIL frameworks are computationally cheaper and easier to parameterize than full sensor simulation, and they directly expose the planner to realistic perceptual insufficiencies. At the same time, object-level abstractions do not test failures caused directly at the sensor level, and some PEMs require each type of error to be defined and modeled explicitly; the reported PEM implementation, for example, focuses on missed detections and parameter errors for existing objects and does not generate false positives (Piazzoni et al., 2023, Wong et al., 2020, Fei et al., 5 Jun 2026).

Geometry-aware and physics-based systems depend strongly on auxiliary signals and simulator coverage. P3Sim states that its quality depends on the reliability of the depth, flow, segmentation, and pose signals used to build the conditioning, and that its strongest standardized quantitative evaluations are on novel view synthesis and 3D object motion rather than on collisions, deformation, and multi-agent interactions (Lee et al., 25 Jun 2026). Underwater active perception depends strongly on the diversity of its synthetic training set, assumes the inspected asset is relatively planar or can be summarized by a single target-distance value, and notes that far distances were not included (Cardaillac et al., 23 Apr 2025). OceanSim is explicit that it does not accurately model underwater vehicle dynamics and fluid dynamics and remains a perception-oriented simulator rather than a full underwater vehicle simulator (Song et al., 3 Mar 2025).

Mechanistic and internal-simulation models have their own constraints. The stochastic-resonance model is highly simplified, centered on sinusoidal gratings, and does not establish direct neurophysiological causation (Kundu et al., 2010). The A-SOM action-prediction architecture reports that recognition accuracy declines gradually at first and then much more strongly once the simulated portion becomes large; up to about 35% internal simulation, recognition accuracy drops only around 5%, but beyond 35% the reduction becomes much larger (Gharaee, 2022). A plausible implication is that internal completion is useful precisely when the missing region is structured enough to be constrained by learned perceptual dynamics.

Across these literatures, three recurring research directions are explicit. One is more faithful conditioning: clinically grounded symptom models, better depth or flow prompts, and causal environmental triggers. Another is stronger structural priors: geometric consistency, persistent memory, calibrated optics, and zone- or context-aware error models. The third is task-aware evaluation: literature consistency for disorder simulation, planner mismatch for autonomous driving, and closed-loop control effects for robotics. This suggests that the field is converging not on a single canonical perception simulation model, but on a family of hybrid systems that combine explicit structure, learned inference, and domain-specific notions of perceptual fidelity.