Light-Bound Transformers: Hardware-Anchored Robustness for Silicon-Photonic Computer Vision Systems

Abstract: Deploying Vision Transformers (ViTs) on near-sensor analog accelerators demands training pipelines that are explicitly aligned with device-level noise and energy constraints. We introduce a compact framework for silicon-photonic execution of ViTs that integrates measured hardware noise, robust attention training, and an energy-aware processing flow. We first characterize bank-level noise in microring-resonator (MR) arrays, including fabrication variation, thermal drift, and amplitude noise, and convert these measurements into closed-form, activation-dependent variance proxies for attention logits and feed-forward activations. Using these proxies, we develop Chance-Constrained Training (CCT), which enforces variance-normalized logit margins to bound attention rank flips, and a noise-aware LayerNorm that stabilizes feature statistics without changing the optical schedule. These components yield a practical ``measure $\rightarrow$ model $\rightarrow$ train $\rightarrow$ run'' pipeline that optimizes accuracy under noise while respecting system energy limits. Hardware-in-the-loop experiments with MR photonic banks show that our approach restores near-clean accuracy under realistic noise budgets, with no in-situ learning or additional optical MACs.

• The paper introduces a hardware-anchored robustness framework that leverages measured device noise statistics and analytical variance proxies to improve ViT performance on silicon-photonic hardware.
• It employs chance-constrained attention training and noise-aware LayerNorm to mitigate errors from fabrication, thermal drift, and laser noise in microring-resonator arrays.
• Experimental evaluations show that the proposed method retains high accuracy and energy efficiency under high-noise conditions, outperforming conventional fine-tuning approaches.

Hardware-Anchored Robustness for Silicon-Photonic Vision Transformers

Introduction

This work introduces a comprehensive methodology for the robust deployment of Vision Transformers (ViTs) on silicon-photonic (SiPh) microring-resonator (MR) accelerator hardware. The novelty lies in a hardware-characterization-informed, analytic approach to noise modeling, algorithmic regularization via chance-constrained training (CCT) of attention, and noise-aware layer normalization. The pipeline explicitly aligns the model’s training objective with measured device-level noise characteristics, addressing the brittleness of digital-to-analog mappings in SiPh for computer vision workloads.

Characterizing Silicon-Photonic Hardware and its Noisy Substrate

Fabrication variability, thermal drift, and laser-induced amplitude noise are identified as principal degradations in MR-photonic architectures. The paper presents detailed spatial statistics of resonance wavelength shifts for MR banks, elucidating both intra-bank stochasticity and die-level heterogeneity:

Figure 2: (a) shows the distribution of resonance wavelength shifts for four MR banks across 100 random placements on the variation map; (b) gives a heat map of spatially correlated process-induced deviations for MR placement analysis.

These characterizations serve as ground truth for hardware-aware modeling: per-bank variance $\sigma_\text{fab}^2$ quantifies static mismatch, thermal crosstalk $\sigma_\text{thermal}^2$ captures run-time channel correlations, and $\sigma_\text{laser}^2$ models input (laser) amplitude noise. The analysis supports two regimes: pre-trim (dominated by stochastic but correctable fabrication variances, up to $\sigma_\text{fab}\approx 0.8$), and post-trim (low-jitter, trimmed, with $\sigma_\text{fab}$ reduced by over an order of magnitude).

Noise-Aware Vision Transformer Architecture and Training

The architecture maps the compute-intensive transformer primitives (primarily MatMuls for MHSA, FFN, and embedding layers) onto SiPh cores, leveraging spatial and wavelength division multiplexing. The key challenge is the mapping of high-dimensional, quantized weights and laser-encoded activations onto MR arrays, with substantial process variation and dynamic perturbation.

To counter these nonidealities, the authors propose a closed-form, variance-proxy-driven training and inference approach:

Figure 4: The end-to-end flow for noise-aware ViT: physical device statistics inform analytic variance proxies, which impose chance-constrained attention (CCT) training and noise-aware LayerNorm for robust deployment under fabrication, thermal, and laser noise.

Chance-Constrained Attention Training

Conventional ViT training using cross-entropy leaves attention logits vulnerable to noise-induced rank flips, resulting in cascading softmax misallocation. This work introduces CCT: for each attention head and query, CCT enforces a minimum margin-to-variance ratio $m_{t,ij}/\sigma_{t,ij} \geq z_\tau$ for all important competitors $j$, such that the probability of logit order flips is bounded by a target $\tau$ (quantile threshold).

Closed-form, activation-dependent proxies for the per-bank attention logit variances are computed analytically, without sampling. The loss aggregates a convex hinge over adversarial competitors, backpropagating the stability constraint directly onto learnable projections. Notably, this formulation provides direct, interpretable probabilistic robustness guarantees at the attention level, rather than generic perturbation resilience.

Noise-Aware Layer Normalization

Standard LayerNorm fails under hardware noise, as device-induced variance inflations bias both feature scaling and gradient stability. Here, a Noise-Aware LayerNorm (NALN) is introduced, applying a variance correction by subtracting device-inferred noise variance from empirical variance in both forward and backward passes, thereby preserving meaningful feature contrast and stabilizing optimization under quantization and device noise.

Experimental Evaluation and Numerical Findings

Evaluation spans ViT-Tiny/Small/Base models across CIFAR-10 and Tiny-ImageNet, as well as dense prediction tasks on COCO, with fabricated and simulated MR device parameters. Hardware-realistic noise is injected into all relevant ViT layers.

Figure 6: Top-1 mean accuracy for ViT-Tiny on CIFAR-10, plotted against increasing $\sigma_\mathrm{fab}$ for baseline inference, standard fine-tuning, CCT, and CCT+NALN.

Key results include:

• At $\sigma_\mathrm{fab}=0.20$, CCT+NALN achieves $\sigma_\text{thermal}^2$0 on CIFAR-10, matching clean accuracy; standard fine-tuning yields only partial recovery.
• Under extreme conditions ($\sigma_\text{thermal}^2$1), where vanilla inference fails catastrophically, CCT+NALN retains $\sigma_\text{thermal}^2$2 accuracy.
• The robustness gain is sustained for larger ViT variants and for Tiny-ImageNet, with CCT+NALN consistently outstripping ordinary fine-tuning by over $\sigma_\text{thermal}^2$3 in high-noise regimes.
• Object detection and segmentation mAP losses on COCO are similarly mitigated, with CCT+NALN narrowing the accuracy gap substantially under severe backbone noise.

Energy and performance evaluation reveals additional strong claims:

• The SiPh accelerator achieves $\sigma_\text{thermal}^2$4 KFps/W on INT8 ViT inference, $\sigma_\text{thermal}^2$5 higher than Xilinx VCK190 FPGA and $\sigma_\text{thermal}^2$6 that of NVIDIA A100 GPU under identical workloads.
• End-to-end energy and latency breakdowns show analog optics dominate computation time and energy; most inefficiency arises from ADC overhead, motivating further shift toward in-analog-domain operation for next-generation co-design.

Implications and Future Directions

The analytic, hardware-anchored robustness framework promoted in this paper is a substantive advancement for deploying complex DNNs, particularly ViTs, on noise-limited SiPh and other analog substrates. The method moves beyond generic noise injection, exploiting directly measured device statistics for actionable regularization, and thereby ensuring robust and predictable model behavior under true physical constraints—a necessity for near-sensor and edge-AI deployments.

Practically, this work bridges the gap between device/process engineering and algorithmic robustness, offering a fully differentiable pipeline compatible with large-scale, high-throughput applications. The approach is extensible to alternative analog fabrics (e.g., resistive IMC arrays), and suggests a paradigm for future hardware-software co-design: analytic characterization $\sigma_\text{thermal}^2$7 variance-aware modeling $\sigma_\text{thermal}^2$8 constrained learning.

Theoretically, integrating hardware-informed variance proxies and probabilistic guarantees into model training offers a path toward verifiable reliability under non-i.i.d. error processes, which will be increasingly critical as edge AI and sensor fusion systems push the boundaries of energy and area constraints.

Future developments may include integrating active device-characterization feedback for adaptive trimming, advanced hardware-aware quantization, and deployment of more granular, structured noise models as SiPh process controls improve.

Conclusion

This work establishes a rigorous, hardware-anchored methodology for deploying Vision Transformers on silicon-photonic accelerators, resilient to dominant process and device nonidealities. By translating measured hardware statistics into analytic variance proxies and imposing chance constraints at the attention level, alongside noise-aware normalization, the approach consistently delivers near-clean accuracy across diverse vision tasks under aggressive noise regimes and tight energy budgets. These results highlight the viability and necessity of hardware-aware learning principles for the next generation of edge and near-sensor AI systems.