Scaling on-chip photonic neural processors using arbitrarily programmable wave propagation (2402.17750v1)
Abstract: On-chip photonic processors for neural networks have potential benefits in both speed and energy efficiency but have not yet reached the scale at which they can outperform electronic processors. The dominant paradigm for designing on-chip photonics is to make networks of relatively bulky discrete components connected by one-dimensional waveguides. A far more compact alternative is to avoid explicitly defining any components and instead sculpt the continuous substrate of the photonic processor to directly perform the computation using waves freely propagating in two dimensions. We propose and demonstrate a device whose refractive index as a function of space, $n(x,z)$, can be rapidly reprogrammed, allowing arbitrary control over the wave propagation in the device. Our device, a 2D-programmable waveguide, combines photoconductive gain with the electro-optic effect to achieve massively parallel modulation of the refractive index of a slab waveguide, with an index modulation depth of $10{-3}$ and approximately $104$ programmable degrees of freedom. We used a prototype device with a functional area of $12\,\text{mm}2$ to perform neural-network inference with up to 49-dimensional input vectors in a single pass, achieving 96% accuracy on vowel classification and 86% accuracy on $7 \times 7$-pixel MNIST handwritten-digit classification. This is a scale beyond that of previous photonic chips relying on discrete components, illustrating the benefit of the continuous-waves paradigm. In principle, with large enough chip area, the reprogrammability of the device's refractive index distribution enables the reconfigurable realization of any passive, linear photonic circuit or device. This promises the development of more compact and versatile photonic systems for a wide range of applications, including optical processing, smart sensing, spectroscopy, and optical communications.
- LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
- Patterson, D. et al. Carbon emissions and large neural network training. arXiv:2104.10350 (2021).
- Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M. & Villalobos, P. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN), 1–8 (2022).
- Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nature Photonics 11, 441–446 (2017).
- Wetzstein, G. et al. Inference in artificial intelligence with deep optics and photonics. Nature 588, 39–47 (2020).
- Shastri, B. J., Tait, A. N., Ferreira de Lima, T., Pernice, W. H., Bhaskaran, H., Wright, C. D. & Prucnal, P. R. Photonics for artificial intelligence and neuromorphic computing. Nature Photonics 15, 102–114 (2021).
- Tait, A. N., de Lima, T. F., Zhou, E., Wu, A. X., Nahmias, M. A., Shastri, B. J. & Prucnal, P. R. Neuromorphic photonic networks using silicon photonic weight banks. Scientific Reports 7 (2017).
- Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).
- Bandyopadhyay, S. et al. Single chip photonic deep neural network with accelerated training. arXiv:2208.01623 (2022).
- Feldmann, J. et al. Parallel convolutional processing using an integrated photonic tensor core. Nature 589, 52–58 (2021).
- Huang, C. et al. A silicon photonic–electronic neural network for fibre nonlinearity compensation. Nature Electronics 4, 837–844 (2021).
- Ashtiani, F., Geers, A. J. & Aflatouni, F. An on-chip photonic deep neural network for image classification. Nature 606, 501–506 (2022).
- Hamerly, R., Bernstein, L., Sludds, A., Soljačić, M. & Englund, D. Large-scale optical neural networks based on photoelectric multiplication. Physical Review X 9, 021032 (2019).
- Nahmias, M. A., De Lima, T. F., Tait, A. N., Peng, H.-T., Shastri, B. J. & Prucnal, P. R. Photonic multiply-accumulate operations for neural networks. IEEE Journal of Selected Topics in Quantum Electronics 26, 1–18 (2019).
- Anderson, M. G., Ma, S.-Y., Wang, T., Wright, L. G. & McMahon, P. L. Optical transformers. arXiv:2302.10360 (2023).
- McMahon, P. L. The physics of optical computing. Nature Reviews Physics 5, 717–734 (2023).
- Chrostowski, L., Wang, X., Flueckiger, J., Wu, Y., Wang, Y. & Fard, S. T. Impact of fabrication non-uniformity on chip-scale silicon photonic integrated circuits. In Optical Fiber Communication Conference, Th2A–37 (Optica Publishing Group, 2014).
- Burgwal, R., Clements, W. R., Smith, D. H., Gates, J. C., Kolthammer, W. S., Renema, J. J. & Walmsley, I. A. Using an imperfect photonic network to implement random unitaries. Optics Express 25, 28236–28245 (2017).
- Larocque, H. & Englund, D. Universal linear optics by programmable multimode interference. Optics Express 29, 38257–38267 (2021).
- Khoram, E., Chen, A., Liu, D., Ying, L., Wang, Q., Yuan, M. & Yu, Z. Nanophotonic media for artificial neural inference. Photonics Research 7, 823–827 (2019).
- Hughes, T. W., Williamson, I. A. D., Minkov, M. & Fan, S. Wave physics as an analog recurrent neural network. Science Advances 5, eaay6946 (2019).
- Nakajima, M., Tanaka, K. & Hashimoto, T. Neural Schrödinger equation: Physical law as deep neural network. IEEE Transactions on Neural Networks and Learning Systems 33, 2686–2700 (2022).
- Nikkhah, V., Pirmoradi, A., Ashtiani, F., Edwards, B., Aflatouni, F. & Engheta, N. Inverse-designed low-index-contrast structures on a silicon photonics platform for vector–matrix multiplication. Nature Photonics (2024).
- Wu, T., Menarini, M., Gao, Z. & Feng, L. Lithography-free reconfigurable integrated photonic processor. Nature Photonics 17, 710–716 (2023).
- Molesky, S., Lin, Z., Piggott, A. Y., Jin, W., Vucković, J. & Rodriguez, A. W. Inverse design in nanophotonics. Nature Photonics 12, 659–670 (2018).
- Psaltis, D., Brady, D., Gu, X.-G. & Lin, S. Holography in artificial neural networks. Nature 343, 325–330 (1990).
- Brady, D. J. & Psaltis, D. Holographic interconnections in photorefractive waveguides. Applied Optics 30, 2324 (1991).
- Delaney, M. et al. Nonvolatile programmable silicon photonics using an ultralow-loss Sb22{}_{2}start_FLOATSUBSCRIPT 2 end_FLOATSUBSCRIPTSe33{}_{3}start_FLOATSUBSCRIPT 3 end_FLOATSUBSCRIPT phase change material. Science Advances 7, eabg3500 (2021).
- Wu, C., Deng, H., Huang, Y.-S., Yu, H., Takeuchi, I., Ríos Ocampo, C. A. & Li, M. Freeform direct-write and rewritable photonic integrated circuits in phase-change thin films. Science Advances 10, eadk1361 (2024).
- Delaney, M., Zeimpekis, I., Lawson, D., Hewak, D. W. & Muskens, O. L. A new family of ultralow loss reversible phase‐change materials for photonic integrated circuits: Sb2S3subscriptSb2subscriptS3\mathrm{Sb}_{2}\mathrm{S}_{3}roman_Sb start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT roman_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT and Sb2Se3subscriptSb2subscriptSe3\mathrm{Sb}_{2}\mathrm{Se}_{3}roman_Sb start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT roman_Se start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT. Advanced Functional Materials 30, 2002447 (2020).
- Wright, L. G., Onodera, T., Stein, M. M., Wang, T., Schachter, D. T., Hu, Z. & McMahon, P. L. Deep physical neural networks trained with backpropagation. Nature 601, 549–555 (2022).
- Chiou, P. Y., Ohta, A. T. & Wu, M. C. Massively parallel manipulation of single cells and microparticles using optical images. Nature 436, 370–372 (2005).
- Wu, M. C. Optoelectronic tweezers. Nature Photonics 5, 322–324 (2011).
- Luennemann, M., Hartwig, U., Panotopoulos, G. & Buse, K. Electrooptic properties of lithium niobate crystals for extremely high external electric fields. Applied Physics B 76, 403–406 (2003).
- Hillenbrand, J., Getty, L. A., Clark, M. J. & Wheeler, K. Acoustic characteristics of american english vowels. The Journal of the Acoustical society of America 97, 3099–3111 (1995).
- LeCun, Y. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist/ (1998).
- Gu, J., Zhu, H., Feng, C., Jiang, Z., Chen, R. T. & Pan, D. Z. M3icro: Machine learning-enabled compact photonic tensor core based on programmable multi-operand multimode interference. arXiv:2305.19505 (2023).
- Huang, C. et al. Demonstration of scalable microring weight bank control for large-scale photonic integrated circuits. APL Photonics 5 (2020).
- Li, G. H., Sekine, R., Nehra, R., Gray, R. M., Ledezma, L., Guo, Q. & Marandi, A. All-optical ultrafast ReLU function for energy-efficient nanophotonic deep learning. Nanophotonics 12, 847–855 (2022).
- Cui, C., Zhang, L. & Fan, L. In situ control of effective Kerr nonlinearity with Pockels integrated photonics. Nature Physics 18, 497–501 (2022).
- Wang, C. et al. Integrated lithium niobate electro-optic modulators operating at CMOS-compatible voltages. Nature 562, 101–104 (2018).
- Ahn, G. H. et al. Platform-agnostic waveguide integration of high-speed photodetectors with evaporated tellurium thin films. Optica 10, 349 (2023).
- Bogaerts, W. et al. Programmable photonic circuits. Nature 586, 207–216 (2020).
- Tang, P., Meier, A. L., Towner, D. J. & Wessels, B. W. BaTiO33{}_{3}start_FLOATSUBSCRIPT 3 end_FLOATSUBSCRIPT thin-film waveguide modulator with a low voltage–length product at near-infrared wavelengths of 098 and 155 µm. Optics Letters 30, 254 (2005).
- Koeber, S. et al. Femtojoule electro-optic modulation using a silicon–organic hybrid device. Light: Science & Applications 4, e255 (2015).
- Davis, S. R., Farca, G., Rommel, S. D., Johnson, S. & Anderson, M. H. Liquid crystal waveguides: new devices enabled by >>>1000 waves of optical phase control. In Emerging Liquid Crystal Technologies V, vol. 7618, 76180E. International Society for Optics and Photonics (SPIE, 2010).
- Leuthold, J., Koos, C. & Freude, W. Nonlinear silicon photonics. Nature Photonics 4, 535–544 (2010).
- Blumenthal, D. J., Heideman, R., Geuzebroek, D., Leinse, A. & Roeloffzen, C. Silicon nitride in silicon photonics. Proceedings of the IEEE 106, 2209–2231 (2018).
- Jung, H., Yu, S.-P., Carlson, D. R., Drake, T. E., Briles, T. C. & Papp, S. B. Tantala Kerr nonlinear integrated photonics. Optica 8, 811 (2021).
- Timurdogan, E., Poulton, C. V., Byrd, M. J. & Watts, M. R. Electric field-induced second-order nonlinear optical effects in silicon waveguides. Nature Photonics 11, 200–206 (2017).
- Mohammadi Estakhri, N., Edwards, B. & Engheta, N. Inverse-designed metastructures that solve equations. Science 363, 1333–1338 (2019).
- Roques-Carmes, C. et al. Heuristic recurrent algorithms for photonic ising machines. Nature Communications 11, 249 (2020).
- Yamaguchi, T., Arai, K., Niiyama, T., Uchida, A. & Sunada, S. Time-domain photonic image processor based on speckle projection and reservoir computing. Communications Physics 6, 250 (2023).
- Yao, C., Xu, K., Zhang, W., Chen, M., Cheng, Q. & Penty, R. Integrated reconstructive spectrometer with programmable photonic circuits. Nature Communications 14, 6376 (2023).
- Yang, Z., Albrow-Owen, T., Cai, W. & Hasan, T. Miniaturization of optical spectrometers. Science 371, eabe0722 (2021).
- Cheng, Q., Rumley, S., Bahadori, M. & Bergman, K. Photonic switching in high performance datacenters. Optics express 26, 16022–16043 (2018).
- Yang, K. Y. et al. Multi-dimensional data transmission using inverse-designed silicon photonics and microcombs. Nature Communications 13, 7862 (2022).
- Habermehl, S., Apodaca, R. T. & Kaplar, R. J. On dielectric breakdown in silicon-rich silicon nitride thin films. Applied Physics Letters 94, 012905 (2009).
- Piccirillo, A. & Gobbi, A. L. Physical‐electrical properties of silicon nitride deposited by PECVD on III–V semiconductors. Journal of The Electrochemical Society 137, 3910–3917 (1990).
- Srivastava, J. K., Prasad, M. & Wagner, J. B. Electrical conductivity of silicon dioxide thermally grown on silicon. Journal of The Electrochemical Society 132, 955–963 (1985).
- Janotta, A. et al. Doping and its efficiency in a−SiOx:H:𝑎subscriptSiO𝑥Ha-\mathrm{SiO}_{x}:\mathrm{H}italic_a - roman_SiO start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT : roman_H. Phys. Rev. B 69, 115206 (2004).
- Piccoli, G., Sanna, M., Borghi, M., Pavesi, L. & Ghulinyan, M. Silicon oxynitride platform for linear and nonlinear photonics at nir wavelengths. Optical Materials Express 12, 3551 (2022).
- Ghatak, A., Thyagarajan, K. & Shenoy, M. Numerical analysis of planar optical waveguides using matrix approach. Journal of Lightwave Technology 5, 660–667 (1987).
- Zhang, M., Wang, C., Cheng, R., Shams-Ansari, A. & Lončar, M. Monolithic ultra-high-Q lithium niobate microring resonator. Optica 4, 1536 (2017).
- Bender, N., Yamilov, A., Goetschy, A., Yılmaz, H., Hsu, C. W. & Cao, H. Depth-targeted energy delivery deep inside scattering media. Nature Physics 18, 309–315 (2022).
- Frumker, E. & Silberberg, Y. Phase and amplitude pulse shaping with two-dimensional phase-only spatial light modulators. JOSA B 24, 2940–2947 (2007).
- Hughes, T. W., Minkov, M., Shi, Y. & Fan, S. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5, 864–871 (2018).
- Pai, S. et al. Experimentally realized in situ backpropagation for deep learning in photonic neural networks. Science 380, 398–404 (2023).
- Spall, J., Guo, X. & Lvovsky, A. I. Training neural networks with end-to-end optical backpropagation. arxiv:2308.05226 (2023).
- Iluz, M., Cohen, K., Kheireddine, J., Hazan, Y., Rosenthal, A., Tsesses, S. & Bartal, G. Unveiling the evolution of light within photonic integrated circuits. Optica 11, 42–47 (2024).
- Agrawal, G. Nonlinear fiber optics (fifth edition) (Academic Press, Boston, 2012).
- Schwesyg, J. R., Falk, M., Phillips, C. R., Jundt, D. H., Buse, K. & Fejer, M. M. Pyroelectrically induced photorefractive damage in magnesium-doped lithium niobate crystals. J. Opt. Soc. Am. B 28, 1973–1987 (2011).
- Paturzo, M. et al. On the origin of internal field in lithium niobate crystals directly observed by digital holography. Optics Express 13, 5416 (2005).
- Ramey, C. Silicon photonics for artificial intelligence acceleration. In 2020 IEEE Hot Chips 32 Symposium (HCS), 1–26 (2020).
- Zhang, X. et al. Heterogeneously integrated III–V-on-lithium niobate broadband light sources and photodetectors. Optics Letters 47, 4564 (2022).
- Zhu, S. et al. Waveguide‐integrated two‐dimensional material photodetectors in thin‐film lithium niobate. Advanced Photonics Research 4, 2300045 (2023).
- Liu, Z. et al. Micro-light-emitting diodes with quantum dots in display technology. Light: Science & Applications 9, 83 (2020).
- Choi, H. W., Liu, C., Gu, E., McConnell, G., Girkin, J. M., Watson, I. M. & Dawson, M. D. Gan micro-light-emitting diode arrays with monolithically integrated sapphire microlenses. Applied Physics Letters 84, 2253–2255 (2004).