STEM Image-to-Code Translation

• STEM Image-to-Code Translation is a process that converts scientific visuals into structured, executable code while preserving key semantics like symbols, topology, and geometry.
• Techniques such as encoder–decoder architectures, attention mechanisms, and reinforcement learning enable the generation of outputs like LaTeX, Structured Text, and Python code.
• Applications span formula OCR, industrial P&ID control logic, and microscopy analysis, driving improved simulation, verification, and optimization in STEM research.

STEM image-to-code translation denotes the conversion of a STEM visual input into a structured machine-readable output whose semantics can be compiled, executed, simulated, or used for downstream control. In published work, the target representation ranges from LaTeX markup for printed formulas, IEC 61131-3 Structured Text for rasterized Piping-and-Instrumentation Diagrams, and Python/Matplotlib programs for visual reconstruction, to descriptor vectors, segmentation masks, component abundances, and rotationally invariant variational autoencoder latents that function as “code” for microscope steering and structural discovery (Singh, 2018, Koziolek et al., 2023, Guan et al., 11 Mar 2026, Creange et al., 2021). The common objective is not generic captioning, but faithful recovery of domain semantics—symbols, topology, geometry, periodicity, local atomic environments, spatial relations, and control intent—in a form that is directly actionable and, in several settings, executable or quantitatively verifiable (Periasami et al., 11 May 2026).

1. Definition and problem formulations

The task has been formalized in at least two distinct but related ways. In neural markup generation, the goal is to generate a token sequence $y=(y_1,\dots,y_\tau)$ from an image $x$ by maximizing the conditional probability

$P(y\mid x)=\prod_{t=1}^{\tau} P(y_t\mid y_{<t},a),$

where $a$ is the flattened visual feature sequence produced by the encoder (Singh, 2018). In executable reconstruction, the goal is to map an image $I$ to a program $P$ in a target language $L$ such that the rendered output $\hat I=R(P)$ reconstructs the input, with supervised training implemented as token-level negative log-likelihood over code tokens,

$$L_{\mathrm{SFT}}(\theta)=-\sum_t \log p_\theta(c_t\mid I,c_{<t}),$$

and reinforcement learning added through a GRPO objective defined over grouped rollouts and KL regularization (Guan et al., 11 Mar 2026).

A second formulation appears in automated microscopy, where “code” is not necessarily an executable source program but a compact representation of image content. Descriptor vectors $c(p)\in\mathbb{R}^d$ are computed per pixel neighborhood or sliding window, or aggregated into ROI-level codes $x$0, and then supplied to a Gaussian Process surrogate for Bayesian optimization of future sampling locations (Creange et al., 2021). Here the closed-loop structure is explicit: images are converted to code, the code is modeled by a surrogate, the surrogate selects new ROIs, and new images are acquired.

These formulations share a common premise: the output representation must preserve the structure that makes the STEM image meaningful. In formula OCR, this structure is syntactic and typographic; in P&IDs it is topological and control-theoretic; in microscopy it is crystallographic, mesoscale, or defect-related; and in reconstruction benchmarks it is the combination of geometry, labels, topology, and style required to regenerate the source image (Singh, 2018, Koziolek et al., 2023, Periasami et al., 11 May 2026).

2. Target representations and code semantics

The literature uses “code” in several technically distinct senses. Some outputs are human-readable symbolic programs or markup, whereas others are compact latent or geometric encodings intended for downstream optimization.

Target representation	Typical input	Operational role
Descriptor vectors, component abundances, segmentation-derived features, rVAE latents	STEM microscopy images	Surrogate modeling and Bayesian optimization for ROI selection
LaTeX markup	Single-line printed math formulas	Compilable symbolic reconstruction of formula structure
IEC 61131-3 Structured Text	Rasterized P&IDs	PLC/DCS control logic, including PID loops, interlocks, and startup logic
Python/Matplotlib code	Charts, geometry, graphs, chemistry, circuits, documents, spatial scenes	Executable visual reconstruction and benchmarkable perception

In microscopy, three representation families are explicitly implemented: engineered descriptors from sliding-window FFTs with linear unmixing by SVD, NMF, or N-FINDR; deep learning–based semantic segmentation or atom finding that yields probability maps, class masks, coordinates, and local geometric descriptors; and rotationally invariant VAE latents that encode windowed FFTs or atom-centered patches into low-dimensional variables, with three dedicated latent variables reserved for rotation angle and $x$1 offsets (Creange et al., 2021). The resulting code may be serialized in HDF5 or JSON, for example with fields such as "descriptor" and "component_abundances".

In formula translation, the target is canonicalized LaTeX markup. The model must learn syntax, nesting, and token order without explicit brace or environment constraints, and success is judged by BLEU, edit distance, compile success, and visual match rather than by latent separability (Singh, 2018).

In process automation, the output is syntactically compliant Structured Text validated by OpenPLC’s IEC2C compiler. The generated code includes PID function blocks, IF/THEN/ELSE control logic, timers, comments in (* ... *) notation, and sequential state/phase logic (Koziolek et al., 2023). The mapping is prompt-driven rather than grammar-driven; the paper explicitly states that no BNF or formal rule tables are presented.

In CodePercept and Vision2Code, the dominant executable target is Python with Matplotlib, typically with NumPy and sometimes SciPy or pandas. This target is used because it supports unified rendering across charts and plots, geometry, graphs, scientific imagery, documents, and 3D spatial scenes, while remaining executable in a controlled sandbox (Guan et al., 11 Mar 2026, Periasami et al., 11 May 2026).

3. Translation architectures and workflows

A canonical neural formulation is the encoder–attention–decoder transducer for Im2LaTeX. Images are centered, padded to a fixed size of $x$2, whitened to $x$3, and passed through five convolutional blocks with $x$4 kernels, stride 1, tanh activation, and $x$5 maxpool, producing a $x$6 feature grid (Singh, 2018). Two encoder variants are evaluated: i2l-nopool, which keeps a flattened sequence of length $x$7, and i2l-strips, which pools along height to obtain length $x$8. Decoding is performed by a two-layer LSTM stack with $x$9 and $P(y\mid x)=\prod_{t=1}^{\tau} P(y_t\mid y_{<t},a),$0, conditioned at each step on the previous token embedding and an attention context vector. The attention module computes $P(y\mid x)=\prod_{t=1}^{\tau} P(y_t\mid y_{<t},a),$1 over spatial locations and forms

$P(y\mid x)=\prod_{t=1}^{\tau} P(y_t\mid y_{<t},a),$2

Training uses teacher forcing, Adam with $P(y\mid x)=\prod_{t=1}^{\tau} P(y_t\mid y_{<t},a),$3 and $P(y\mid x)=\prod_{t=1}^{\tau} P(y_t\mid y_{<t},a),$4, minibatch size 56, L2 regularization with $P(y\mid x)=\prod_{t=1}^{\tau} P(y_t\mid y_{<t},a),$5, and beam search width 10 at inference (Singh, 2018).

Microscopy automation instantiates three end-to-end workflows: sliding-window FFT followed by SVD, NMF, or N-FINDR and then Bayesian optimization; sliding-window FFT followed by rVAE and latent selection; and DCNN atom finding followed by patch extraction, rVAE encoding, and latent selection (Creange et al., 2021). Preprocessing recommendations include drift correction via phase correlation, mild denoising, intensity normalization, background flattening, and patch extraction with window sizes typically between 16 and 64 pixels. The rVAE objective is the ELBO with Gaussian posteriors and reparameterization, and optimization is described with Adam, $P(y\mid x)=\prod_{t=1}^{\tau} P(y_t\mid y_{<t},a),$6, $P(y\mid x)=\prod_{t=1}^{\tau} P(y_t\mid y_{<t},a),$7, learning rate $P(y\mid x)=\prod_{t=1}^{\tau} P(y_t\mid y_{<t},a),$8, KL warmup, and early stopping. The selected component abundance or latent map becomes the Bayesian optimization target, modeled by a Gaussian Process with RBF, Matern, or Rational Quadratic kernels and acquisition by EI, UCB, or PI (Creange et al., 2021).

The P&ID-to-ST pipeline is structurally different. It relies on GPT-4V in interactive ChatGPT sessions, with manual segmentation of large diagrams into smaller cutouts, iterative prompts such as “Detect all controllers in this image” and “Write a self-contained IEC 61131-3 ST function block,” and post-generation import into OpenPLC via OpenPLC-Importer and IEC2C (Koziolek et al., 2023). No specialized CV detector, OCR engine, graph extraction algorithm, fine-tuning, or RAG is used. Recognition of symbols, tags, and topology is implicit in the multimodal LLM.

CodePercept generalizes executable image-to-code translation at scale. ICC-1M comprises over 1,000,000 Image-Caption-Code triplets and is built through three pipelines: Image Reproduction, Image Diversity, and Solid Geometry template instantiation (Guan et al., 11 Mar 2026). Code-grounded captions are produced by combining a caption draft with verified facts extracted from ground-truth executable code and an execution tracer $P(y\mid x)=\prod_{t=1}^{\tau} P(y_t\mid y_{<t},a),$9, which records coordinates, dimensions, counts, colors, and rendering semantics. Supervised fine-tuning jointly trains captioning and code generation, and reinforcement learning with GRPO adds rewards for fenced code format, executability, GPT-4o-based semantic equivalence, and GPT-4o-based image similarity (Guan et al., 11 Mar 2026).

4. Domains, case studies, and empirical results

Published applications span atomic-resolution imaging, mathematical notation, industrial diagrams, and multi-domain STEM reconstruction.

In automated microscopy, the method is evaluated on NiO pillars within a La:SrMnO$a$0 matrix, ferroelectric domains in Sm-doped BiFeO$a$1, and graphene with Si dopants and topological defects (Creange et al., 2021). For NiO-LSMO, Workflow 1, based on FFT with N-FINDR or NMF, yields robust component extraction and abundance maps that clearly localize pillars; the paper reports “up to 99% of ROI discovered in <200 points for favorable hyperparameters.” For BFO domain walls, Workflow 3, based on DCNN plus rVAE, is reported as best suited for atomic-scale features, although SSIM is “primarily below ~0.5 in reconstruction for small features due to GP smoothing and sparse sampling.” For graphene, Workflow 3 separates two trimer orientation classes through an interpretable angle latent, whereas FFT-window rVAE latents are less interpretable and more sensitive to hyperparameters (Creange et al., 2021).

In Im2LaTeX, the system achieves BLEU 89.0% on I2L-140K for both i2l-nopool and i2l-strips, with edit distances 0.0676 and 0.0671, visual match rates 70.37% and 69.24%, and compile rates 99.94% and 99.85%, respectively (Singh, 2018). On Im2latex-90k, i2l-strips reaches BLEU 88.19%, edit distance 0.0725, visual match 68.03%, and compile 99.81%. The paper notes improvement over Im2Tex at 87.73 BLEU on Im2latex-100k, while also observing that strict visual match can be lower because many non-identical LaTeX strings render equivalently (Singh, 2018).

For P&ID-to-ST translation, three case studies are reported: an Eastman Chemical plant P&ID, a DEXPI reference P&ID, and a butane regeneration air/water cooling system (Koziolek et al., 2023). In the Eastman case, GPT-4V correctly recognized all 14 control points, detected 1 correct feedforward cascade and missed 4 others, generated a 46-line cascade function block, 76 lines of interlock logic, and 55 lines of startup-step code, all compiled and simulated successfully. In the DEXPI case, the model correctly identified two controllers but misclassified HS 4750.01 and PI 4712.01 as controllers, generated a 38-line PID block with wrong I/O tag mapping, and hallucinated nozzle N4. In the butane case, the initial outputs included misclassification of fans as pumps and transmitters as controllers, but iterative prompting produced syntactically correct ST and an 83-line startup state machine after a bug fix (Koziolek et al., 2023).

CodePercept reports direct executable-reconstruction results on STEM2Code-Eval. CodePercept-32B-R1 reaches Image 68.97, Code 62.53, Avg 65.75, and Exec 95.90, compared with Qwen3-VL-32B-Instruct at Image 36.85, Code 39.98, Avg 38.42, and Exec 81.80 (Guan et al., 11 Mar 2026). The same paper also reports a two-stage captioner-solver evaluation across six STEM reasoning benchmarks: under the same solver, CodePercept-8B-S1 improves over Qwen3-VL-8B-Instruct by +3.0 on average, and CodePercept-32B-S1 improves over Qwen3-VL-32B-Instruct by +2.7; under a stronger solver, CodePercept-32B-S1 yields +3.6 (Guan et al., 11 Mar 2026).

Vision2Code broadens evaluation to 2,169 test examples from 15 source datasets spanning charts and plots, geometry, graphs, scientific imagery and engineering diagrams, documents, and 3D spatial scenes (Periasami et al., 11 May 2026). Overall macro-average $a$2 is 4.06 for GPT-5.4 Mini + reasoning, 4.05 for GPT-5.4, 3.14 for Gemini-3.1-Flash-Lite-Preview, and 1.60 for Qwen3.5-9B. The same benchmark shows domain dependence: models are strong on datasets such as DVQA and GraphVQA-Swift, but weak on SpatialVLM-QA, ChemVQA-2K, DocVQA, and EEE-Bench (Periasami et al., 11 May 2026). These results are not directly comparable to Im2LaTeX BLEU or microscopy SSIM because the evaluation target and score semantics differ.

5. Evaluation, verification, and interpretability

Evaluation protocols in this area are heterogeneous because the target artifacts differ. Im2LaTeX uses corpus BLEU-1/2/3/4, per-word Levenshtein edit distance, a visual exact-match rate with up to 5px shift, and compile success (Singh, 2018). The compile metric is particularly important because syntactic correctness is learned rather than hard-constrained. Attention heatmaps provide a second interpretability channel: the focal region is typically sharp and unimodal, localizing a 1–4-neighborhood around the current symbol or structure and revealing a left-to-right or left-right/up-down reading order depending on the encoder variant (Singh, 2018).

Microscopy workflows emphasize surrogate fidelity and structural localization rather than textual correctness. SSIM between reconstructed Gaussian Process maps and target maps is used to evaluate workflow quality, and abundance maps or selected latent overlays are inspected to validate whether the code tracks phases, domain walls, or defect orientations (Creange et al., 2021). The paper also reports that Workflow 1 is comparatively insensitive and consistently robust across kernels, acquisitions, and correlation lengths from 1 to 20, whereas workflows using rVAE show variability tied to convergence (Creange et al., 2021).

CodePercept defines a reconstruction-oriented benchmark, STEM2Code-Eval, that uses Image Score, Code Score, their arithmetic mean, and Exec Rate over 1,000 manually curated image-code pairs drawn from MathVision, MathVista, MathVerse, DynaMath, LogicVista, and WeMath (Guan et al., 11 Mar 2026). The benchmark is intentionally execution-based: candidates must produce executable Python code, and the output is judged for geometry completeness, positional and relational accuracy, annotations, stylistic consistency, legibility, and reproducibility. This design reflects the paper’s claim that executable code supports deterministic and verifiable assessment more directly than captioning or QA.

Vision2Code adopts a reference-code-free render-and-rate protocol. Models generate executable programs, the rendered output is scored by a VLM rater using dataset-specific four-category rubrics with weights summing to 1.0, and deterministic guardrails cap scores under severe semantic failures such as wrong chart orientation, ambiguous geometric labels, wrong graph topology, incorrect chemical structure, incorrect circuit connectivity, or collapsed 3D scenes (Periasami et al., 11 May 2026). Human validation on 539 source-output pairs rated by six STEM-literate annotators shows Pearson 0.723 and Spearman 0.703 for the dataset-specific rubric, compared with 0.626 and 0.633 for a generic rubric. The paper also argues that SSIM, PSNR, MSE, and embedding similarities are inadequate as sole metrics because high image similarity can coexist with semantic failure, such as wrong axis scale or inverted vector direction (Periasami et al., 11 May 2026).

A common misconception is that image-to-code quality can be read off from text-level exact match alone. The literature repeatedly rejects this simplification. In LaTeX generation, different correct strings can render identically; in microscopy, low SSIM can coexist with ROI-focused sampling; and in reference-code-free reconstruction, rendered semantic structure rather than source-program equivalence is the central object of evaluation (Singh, 2018, Creange et al., 2021, Periasami et al., 11 May 2026).

6. Limitations, misconceptions, and research trajectories

Several failure modes recur across domains. In microscopy, FFT descriptors can be sensitive to window size and prefiltering, Gaussian Process smoothness can wash out small features, and rVAE training can fail to converge because the latent space exhibits multiple minima (Creange et al., 2021). Pre-trained DCNN atom finders can also fail under domain shift in contrast or noise. In P&ID translation, the reported errors include symbol misclassification, topology errors caused by intersecting dashed lines, hallucinated tags, and wrong controller-to-sensor associations; the system has no confidence aggregation, no explicit graph model, no RAG, and no formal verification beyond compilation and simulation (Koziolek et al., 2023). In CodePercept, coverage remains limited for text-rich diagrams requiring OCR, non-synthetic or scanned imagery, and very intricate 3D or fine-grained engineering drawings (Guan et al., 11 Mar 2026). Vision2Code likewise finds that state-of-the-art systems remain weak on chemistry, circuits, dense documents, and especially 3D spatial scenes, even when render success is high (Periasami et al., 11 May 2026).

Another recurring issue is the relationship between perception and reasoning. CodePercept explicitly decouples the two and reports that scaling perception consistently outperforms scaling reasoning, presenting perception as “the true lever limiting current STEM visual reasoning” (Guan et al., 11 Mar 2026). This suggests that failures on downstream STEM tasks often originate in incomplete visual grounding rather than in purely symbolic inference. A plausible implication is that image-to-code translation is not just an output modality, but a diagnostic instrument for whether a model has actually internalized the structure of the input.

Research directions in the cited works converge on hybridization and stronger structural priors. For microscopy, future directions include physics-informed priors that incorporate lattice symmetry, reciprocal-space templates, and beam point-spread, multimodal data fusion with EELS or 4D-STEM, and reinforcement learning for probe paths and dwell schedules (Creange et al., 2021). For industrial diagrams, the proposed trajectory is hybrid CV/RAG pipelines, formal P&ID-to-graph schemas, and verified mappings to ST with static analysis and model checking for safety logic (Koziolek et al., 2023). For executable reconstruction, the open challenges include multi-language code generation beyond Python/Matplotlib, cross-engine consistency, stricter sandboxing, symbolic validators such as graph isomorphism, valency checks, netlist equivalence, and spatial relation checkers, and reference-code-free self-training using evaluator-filtered generations (Guan et al., 11 Mar 2026, Periasami et al., 11 May 2026).

Across these lines of work, STEM image-to-code translation emerges as a family of methods for turning visual STEM structure into operational representations. In some settings the representation is a latent coordinate, a component abundance map, or a segmentation-derived geometric descriptor; in others it is fully executable code or markup. What unifies them is the demand that the output preserve the semantics that matter to STEM practice: not merely what is visible, but what can be compiled, simulated, optimized, or experimentally acted upon (Creange et al., 2021, Singh, 2018, Guan et al., 11 Mar 2026).