OptoViT: Silicon Photonics ViT Accelerator

• OptoViT is a near-sensor, region-aware Vision Transformer accelerator that uses silicon photonics to perform parallel matrix multiplications via VCSELs and microring resonators.
• The design incorporates a hybrid electronic-photonic approach, pairing optical MAC operations with electronic nonlinear functions to optimize efficiency.
• It achieves up to 84% energy savings and 100.4 KFPS/W performance by integrating ROI pruning, quantization-aware training, and photonic-specific matrix decomposition.

Searching arXiv for OptoViT and closely related papers. arxiv_search(query="Opto-ViT silicon photonics Vision Transformer accelerator", max_results=5) Opto-ViT is a near-sensor, region-aware Vision Transformer accelerator that leverages silicon photonics for real-time and energy-efficient vision processing. It is formulated as a hybrid electronic-photonic system in which the optical core performs compute-intensive matrix multiplications using Vertical-Cavity Surface-Emitting Lasers (VCSELs) and Microring Resonators (MRs), while nonlinear functions and normalization remain electronic. The design integrates region-of-interest pruning, quantization-aware training, matrix decomposition tailored to photonic constraints, and device/circuit/architecture co-design to reduce redundant computation and enable scalable ViT inference at the edge (Morsali et al., 9 Jul 2025).

1. Research context and scope

Opto-ViT was introduced against the backdrop of two converging trends: the rise of Vision Transformers as high-capacity vision backbones, and the use of silicon photonics to accelerate neural-network linear algebra under energy and bandwidth constraints. The motivating problem is explicit: ViTs model long-range dependencies and global contextual relationships effectively, but their compute and memory demands hinder efficient deployment in scenarios with strict energy and bandwidth limitations (Morsali et al., 9 Jul 2025).

Within that setting, Opto-ViT is positioned as the first near-sensor, region-aware ViT accelerator leveraging silicon photonics. The emphasis on “near-sensor” is architecturally significant because it links ROI selection and photonic MAC execution to real-time vision pipelines rather than treating the accelerator as a generic off-chip tensor engine. The emphasis on “region-aware” is equally central because the proposal does not merely accelerate a full ViT; it reduces patch traffic before encoding.

A common source of confusion is the similarity between Opto-ViT and POViT. They address different problems. “POViT: Vision Transformer for Multi-objective Design and Characterization of Nanophotonic Devices” concerns the fast and accurate characterization of nanoscale photonic devices and proposes a Transformer model for semiconductor photonic-device design and simulation, rather than a silicon-photonic accelerator for ViT inference (Chen et al., 2022). This terminological distinction matters because one line of work applies transformers to photonics as a design domain, whereas Opto-ViT applies photonics to transformers as a hardware substrate.

2. Hybrid electronic-photonic organization

Opto-ViT adopts a two-tier compute fabric composed of an Optical Processing Block, an Electronic Processing Unit, and Buffer Memory (Morsali et al., 9 Jul 2025). The optical block contains five identical Optical Cores. Each core supports up to 32 parallel Wavelength-Division-Multiplexed channels and 64 waveguide arms, matching a typical ViT head dimension $d_k = 64$. Each core includes an array of 32 VCSELs generating light intensities proportional to input activations $x_j$ on wavelengths $\{\lambda_j\}$, MR banks organized as 64 arms $\times$ 32 rings per arm whose resonance tuning encodes weights $W_{ij}$, balanced photodetectors at each arm for summing transmitted optical powers into MAC results, and DAC-driven MR-tuning circuits together with VCSEL drivers.

For an input vector $\mathbf{x}\in\mathbb{R}^{n}$ and a weight matrix $\mathbf{W}\in\mathbb{R}^{m\times n}$, each core implements the matrix multiplication in parallel as

$y_i = \sum_{j=1}^n w_{ij}x_j.$

The idealized photonic compute-core model is

$$y_i \;=\;\mathrm{BPD}\Bigl(\sum_{j=1}^n |\,\mathrm{VCSEL}(x_j)\cdot\mathrm{MR}(w_{ij})|^2\Bigr) \quad\longrightarrow\quad \mathbf{y} \;=\;\mathbf{W}\,\mathbf{x}.$$

The electronic unit performs GELU, Softmax, Layer-Normalization, and residual additions. The reported dataflow is optical cores output raw MACs, then ADC, then electronic Softmax/GELU/LN, then DAC for the next optical stage. Buffer Memory stores weights and intermediate activations and interfaces to the cores via ADCs and DACs.

Subsystem	Reported role
Optical Processing Block	Matrix multiplications via VCSELs, MRs, and BPDs
Electronic Processing Unit	GELU, Softmax, LN, residual additions
Buffer Memory	Weight and activation storage; ADC/DAC interfacing

This partition makes clear that Opto-ViT is not an all-optical transformer. Its acceleration strategy is selective: linear transforms are photonic, while nonlinear and normalization-heavy stages remain electronic.

3. Region-of-interest pruning with MGNet

To reduce redundant patch processing, Opto-ViT inserts a lightweight Mask Generation Network, MGNet, before the main ViT backbone (Morsali et al., 9 Jul 2025). The input frame is patchified into $(H/p)\times(W/p)$ patches of size $x_j$0, linearly embedded, and passed through a single transformer encoder block consisting of norm, MHSA, and MLP. MGNet then uses a self-attention head on the [CLS] token to compute patch importance, followed by a linear projection to a scalar score per patch and a Sigmoid with hard threshold $x_j$1 to obtain a binary mask.

The scoring equations are

$x_j$2

and

$x_j$3

The loss is binary cross-entropy between $x_j$4 and ground-truth patch labels derived from bounding-box masks, and validation is reported using mean IoU. The practical consequence is a skip rate of approximately 60–70% of patches, yielding linear energy and compute savings in all downstream ViT layers.

MGNet is significant because it changes the accelerator’s operating point before the photonic MAC engine is engaged. Rather than only improving MAC efficiency, the design reduces the number of MACs presented to the photonic fabric. A plausible implication is that Opto-ViT’s energy gains depend jointly on photonic throughput and the quality of ROI estimation; the larger drop on Tiny-ImageNet, where fine-grained masks were unavailable, is consistent with that dependence.

4. ViT co-optimization for photonic constraints

Opto-ViT co-optimizes the ViT backbone through quantization-aware training and matrix decomposition tailored for photonic execution (Morsali et al., 9 Jul 2025). For QAT, both weights and activations use symmetric 8-bit uniform quantization. The forward quantization rule is

$x_j$5

where $x_j$6 is the dynamic range step and $x_j$7. The backward pass uses the Straight-Through Estimator.

The matrix-decomposition optimization targets the attention path. Standard self-attention requires $x_j$8, $x_j$9, followed by $\{\lambda_j\}$0. A naïve mapping would require reprogramming MRs with $\{\lambda_j\}$1 only after computing $\{\lambda_j\}$2. Opto-ViT instead applies the algebraic rewrite

$\{\lambda_j\}$3

At the start of the cycle, three cores are pre-tuned with $\{\lambda_j\}$4, $\{\lambda_j\}$5, and $\{\lambda_j\}$6, all treated as statically known for that stage. The reported five-core pipeline is: cores 1–3 process $\{\lambda_j\}$7, $\{\lambda_j\}$8, and related operands in parallel; cores 4–5 wait for Softmax and GELU, then apply to $\{\lambda_j\}$9.

The architectural consequence is explicit: one MR-tuning delay and the buffering of intermediate $\times$0 are eliminated. This suggests that Opto-ViT’s efficiency is not only a function of photonic MAC density, but also of how transformer algebra is rewritten to match MR programming and dataflow constraints.

5. Photonic device and circuit co-design

The device layer centers on the Microring Resonator and the VCSEL array (Morsali et al., 9 Jul 2025). For the MR, the target is 8-bit weight resolution, corresponding to a Q-factor of approximately 5 000. The reported geometry is waveguide width $\times$1 nm, ring width $\times$2 nm, and radius $\times$3. Crosstalk between MRs $\times$4 and $\times$5 is modeled as

$\times$6

The associated noise power is

$\times$7

with resolution approximately $\times$8.

The VCSEL array uses 32 wavelengths spanning the C-band, and its energy per bit is reported as lower than MOS-tuned MR input. On the integration side, more than 200 identical MR test-cells were fabricated on a $\times$9 mm$W_{ij}$0 SiPh die. Circuit simulation used Cadence Spectre with a 45 nm PDK for DAC/ADC, VCSEL driver, BPD MUX, and memory interface.

The reported latency and energy breakdown for Tiny-ViT on $W_{ij}$1 input further specifies the system bottlenecks. Energy is partitioned as ADC approximately 35%, tuning approximately 25%, VCSEL approximately 15%, BPD approximately 5%, and DAC/Memory/Electronic approximately 20%. Delay is partitioned as Optical MatMul approximately 60%, memory fetch approximately 25%, and Softmax/GELU approximately 15%. The energy model per MAC is summarized as

$W_{ij}$2

These results indicate that the design is full-stack in a literal sense: the reported claims extend from MR device physics and crosstalk modeling to PDK-level circuit simulation and transformer-level execution. A plausible implication is that ADC and MR tuning remain dominant non-ideal costs even when photonic matrix multiplication is highly parallel.

6. Reported performance, comparative position, and caveats

Opto-ViT reports 100.4 KFPS/W on ViT inference and up to 84% total system energy savings via ROI masking and photonic offload, while maintaining less than 1.6% accuracy loss across classification, detection, and video tasks (Morsali et al., 9 Jul 2025). The reference comparisons given are FPGA (Xilinx VCK190, INT8) at 1.42 KFPS/W and GPU (NVIDIA A100, TensorRT INT8) at 0.86 KFPS/W.

Setting	Reported result	Note
ViT inference efficiency	100.4 KFPS/W	Main Opto-ViT throughput metric
Total system energy savings	Up to 84%	Attributed to ROI masking and photonic offload
Accuracy impact	Less than 1.6% loss	Aggregate claim across tasks

For image classification, the reported figures are: ViT-Small at $W_{ij}$3 with no skipping, 97.86% FP32 versus 97.67% Opto-ViT Top-1; ViT-Base at $W_{ij}$4 with no skipping, 98.56% versus 98.16%; and ViT-Base Mask at $W_{ij}$5 with 67% skip, 85.51% versus 80.12%, with the explicit note that Tiny-ImageNet lacked fine-grained masks and therefore exhibited a larger drop. For object detection on COCO using Mask R-CNN with ViTDet-Base backbone, Full-Precision reports AP DET 30.35% and AP SEG 27.12%; Opto-ViT INT8 reports 30.53% and 26.95%; Opto-ViT Mask at 66% skip reports 30.44% and 26.85%. For video object detection on ImageNet-VID, the ViTDet baseline reports mAP 54.90%, mAP@50 80.51%, and mAP@75 62.52%; Opto-ViT INT8 reports 53.39%, 79.62%, and 60.99%; and Mask Pruning at 68% skip reports 53.01%, 79.12%, and 59.81%.

The comparison with other photonic DNN accelerators is more nuanced than a simple “best-in-class” reading. The reported KFPS/W values are LightBulb 57.75, HolyLight 3.30, Robin 46.50, CrossLight 52.59, Lightator up to 188.24, and Opto-ViT 100.4. Thus, Opto-ViT exceeds several listed photonic designs but not all of them. This is an important corrective to a common misconception that the use of silicon photonics alone guarantees the highest reported energy efficiency. The distinguishing feature of Opto-ViT is instead the combination of WDM-parallel photonic MACs, electronic nonlinear processing, and ROI-aware front-end pruning in a single near-sensor ViT deployment stack.