Papers
Topics
Authors
Recent
Search
2000 character limit reached

Spatially Varying Exposure with 2-by-2 Multiplexing: Optimality and Universality

Published 30 Jun 2023 in eess.IV and cs.CV | (2306.17367v1)

Abstract: The advancement of new digital image sensors has enabled the design of exposure multiplexing schemes where a single image capture can have multiple exposures and conversion gains in an interlaced format, similar to that of a Bayer color filter array. In this paper, we ask the question of how to design such multiplexing schemes for adaptive high-dynamic range (HDR) imaging where the multiplexing scheme can be updated according to the scenes. We present two new findings. (i) We address the problem of design optimality. We show that given a multiplex pattern, the conventional optimality criteria based on the input/output-referred signal-to-noise ratio (SNR) of the independently measured pixels can lead to flawed decisions because it cannot encapsulate the location of the saturated pixels. We overcome the issue by proposing a new concept known as the spatially varying exposure risk (SVE-Risk) which is a pseudo-idealistic quantification of the amount of recoverable pixels. We present an efficient enumeration algorithm to select the optimal multiplex patterns. (ii) We report a design universality observation that the design of the multiplex pattern can be decoupled from the image reconstruction algorithm. This is a significant departure from the recent literature that the multiplex pattern should be jointly optimized with the reconstruction algorithm. Our finding suggests that in the context of exposure multiplexing, an end-to-end training may not be necessary.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. S. Nayar and T. Mitsunaga, “High dynamic range imaging: spatially varying pixel exposures,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 472–479, 2000.
  2. S. K. Nayar and V. Branzoi, “Adaptive dynamic range imaging: optical control of pixel exposures over space and time,” in Proc. IEEE International Conference on Computer Vision (ICCV), vol. 2, pp. 1168–1175, 2003.
  3. G. Wan, X. Li, G. Agranov, M. Levoy, and M. Horowitz, “CMOS image sensors with multi-bucket pixels for computational photography,” IEEE Journal of Solid-State Circuits, vol. 47, no. 4, pp. 1031–1042, 2012.
  4. C. M. Nguyen, J. N. P. Martel, and G. Wetzstein, “Learning spatially varying pixel exposures for motion deblurring,” in IEEE International Conference on Computational Photography (ICCP), pp. 1–11, 2022. Available online at https://arxiv.org/abs/2204.07267.
  5. T. Klinghoffer, S. Somasundaram, K. Tiwary, and R. Raskar, “Physics vs. learned priors: Rethinking camera and algorithm design for task-specific imaging,” 2022. Available online at https://arxiv.org/abs/2204.09871.
  6. C. Aguerrebere, A. Almansa, Y. Gousseau, J. Delon, and P. Musé, “Single shot high dynamic range imaging using piecewise linear estimators,” in 2014 IEEE International Conference on Computational Photography (ICCP), pp. 1–10, 2014.
  7. S. Hajisharif, J. Kronander, and J. Unger, “Adaptive Dual ISO HDR Reconstruction,” EURASIP Journal on Image and Video Processing, Dec 2015.
  8. M. Schoberl, A. Belz, A. Nowak, J. Seiler, A. Kaup, and S. Foessel, “Building a high dynamic range video sensor with spatially nonregular optical filtering,” in Applications of Digital Image Processing XXXV (A. G. Tescher, ed.), vol. 8499, p. 84990C, International Society for Optics and Photonics, SPIE, 2012.
  9. U. Cogalan, M. Bemana, K. Myszkowski, H. P. Seidel, and T. Ritschel, “Learning HDR video reconstruction for dual-exposure sensors with temporally-alternating exposures,” Computers & Graphics, vol. 105, pp. ”57–72”, 2022.
  10. F. Yasuma, T. Mitsunaga, D. Iso, and S. K. Nayar, “Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum,” IEEE Transactions on Image Processing, vol. 19, no. 9, pp. 2241–2253, 2010.
  11. M. Alghamdi, Q. Fu, A. Thabet, and W. Heidrich, “Reconfigurable snapshot HDR imaging using coded masks and inception network,” in Vision Modeling and Visualization, 01 2019.
  12. Y. Jiang, I. Choi, J. Jiang, and J. Gu, “HDR video reconstruction with tri-exposure quad-bayer sensors,” 2021.
  13. S. J. Carey, D. R. Barr, B. Wang, A. Lopich, and P. Dudek, “Mixed signal SIMD processor array vision chip for real-time image processing,” Analog Integr. Circuits Signal Process., vol. 77, p. 385–399, dec 2013.
  14. Y. Luo, D. Ho, and S. Mirabbasi, “Exposure-programmable CMOS pixel with selective charge storage and code memory for computational imaging,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 65, no. 5, pp. 1555–1566, 2018.
  15. H. Ke, N. Sarhangnejad, R. Gulve, Z. Xia, N. Gusev, N. Katic, K. N. Kutulakos, and R. Genov, “Extending image sensor dynamic range by scene-aware pixelwise-adaptive coded exposure,” in Proc. Int. Image Sensor Workshop, pp. 111–114, 2019.
  16. H. Reyserhove, A. S. Berkovich, and X. Liu, “Programmable pixel array.” U.S. Patent 20200195828A1, Jun. 2020.
  17. J. Zhang, R. Etienne-Cummings, T. Xiong, T. D. Tran, and S. H. Chin, “Flexible pixel-wise exposure control and readout.” U.S. Patent 20180115725A1, Nov. 2019.
  18. N. Sarhangnejad, N. Katic, Z. Xia, M. Wei, N. Gusev, G. Dutta, R. Gulve, H. Haim, M. M. Garcia, D. Stoppa, K. N. Kutulakos, and R. Genov, “5.5 dual-tap pipelined-code-memory coded-exposure-pixel cmos image sensor for multi-exposure single-frame computational imaging,” in 2019 IEEE International Solid- State Circuits Conference - (ISSCC), pp. 102–104, 2019.
  19. J. N. P. Martel, L. K. Müller, S. J. Carey, P. Dudek, and G. Wetzstein, “Neural sensors: Learning pixel exposures for HDR imaging and video compressive sensing with programmable sensors,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 7, pp. 1642–1653, 2020.
  20. J. Zhang, J. P. Newman, X. Wang, C. S. Thakur, J. Rattray, R. Etienne-Cummings, and M. A. Wilson, “A closed-loop, all-electronic pixel-wise adaptive imaging system for high dynamic range videography,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 67, no. 6, pp. 1803–1814, 2020.
  21. J. Kronander, S. Gustavson, and J. Bonnet, G.and Unger, “Unified HDR reconstruction from raw CFA data,” in IEEE International Conference on Computational Photography (ICCP), pp. 1–9, 2013.
  22. O. Yadid-Pecht and E. R. Fossum, “Wide intrascene dynamic range cmos aps using dual sampling,” IEEE Transactions on Electron Devices, vol. 44, no. 10, pp. 1721–1723, 1997.
  23. Y. Wang, S. Barna, S. Campbell, and E. R. Fossum, “A high dynamic range cmos aps image sensor,” in IEEE Workshop CCD and Advanced Image Sensors, Lake Tahoe, Nevada, USA, 2001.
  24. N. K. Kalantari and R. Ramamoorthi, “Deep high dynamic range imaging of dynamic scenes,” ACM Transactions on Graphics (TOG), vol. 36, no. 4, pp. 1–12, 2017.
  25. S. Wu, J. Xu, Y.-W. Tai, and C.-K. Tang, “Deep high dynamic range imaging with large foreground motions,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 117–132, 2018.
  26. Q. Yan, D. Gong, Q. Shi, A. v. d. Hengel, C. Shen, I. Reid, and Y. Zhang, “Attention-guided network for ghost-free high dynamic range imaging,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1751–1760, 2019.
  27. Q. Yan, D. Gong, P. Zhang, Q. Shi, J. Sun, I. Reid, and Y. Zhang, “Multi-scale dense networks for deep high dynamic range imaging,” in 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 41–50, IEEE, 2019.
  28. Q. Yan, L. Zhang, Y. Liu, Y. Zhu, J. Sun, Q. Shi, and Y. Zhang, “Deep HDR imaging via a non-local network,” IEEE Transactions on Image Processing, vol. 29, pp. 4308–4322, 2020.
  29. Y. Deng, Q. Liu, and T. Ikenaga, “Multi-scale contextual attention based hdr reconstruction of dynamic scenes,” in Twelfth International Conference on Digital Image Processing (ICDIP 2020), vol. 11519, pp. 413–419, SPIE, 2020.
  30. Q. Yan, S. Zhang, W. Chen, Y. Liu, Z. Zhang, Y. Zhang, J. Q. Shi, and D. Gong, “A lightweight network for high dynamic range imaging,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 824–832, 2022.
  31. L. Zhu, F. Zhou, B. Liu, and O. Göksel, “HDRfeat: A feature-rich network for high dynamic range image reconstruction,” arXiv preprint arXiv:2211.04238, 2022.
  32. Y. Chi, X. Zhang, and S. H. Chan, “HDR imaging with spatially varying signal-to-noise ratios,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5724–5734, 2023.
  33. R. Pourreza-Shahri and N. Kehtarnavaz, “Exposure bracketing via automatic exposure selection,” in 2015 IEEE International Conference on Image Processing (ICIP), pp. 320–323, 2015.
  34. N. Barakat, A. N. Hone, and T. E. Darcie, “Minimal-bracketing sets for high-dynamic-range image capture,” IEEE Transactions on Image Processing, vol. 17, no. 10, pp. 1864–1875, 2008.
  35. B. Guthier, S. Kopf, and W. Effelsberg, “Optimal shutter speed sequences for real-time HDR video,” in 2012 IEEE International Conference on Imaging Systems and Techniques Proceedings, pp. 303–308, 2012.
  36. K. F. Huang and J. C. Chiang, “Intelligent exposure determination for high quality HDR image generation,” in 2013 IEEE International Conference on Image Processing, pp. 3201–3205, 2013.
  37. K. Hirakawa and P. J. Wolfe, “Optimal exposure control for high dynamic range imaging,” in 2010 IEEE International Conference on Image Processing, pp. 3137–3140, 2010.
  38. S. W. Hasinoff, F. Durand, and W. T. Freeman, “Noise-optimal capture for high dynamic range photography,” in 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 553–560, 2010.
  39. Z. Wang, J. Zhang, M. Lin, J. Wang, P. Luo, and J. Ren, “Learning a reinforced agent for flexible exposure bracketing selection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1820–1828, 2020.
  40. Y. Chi, A. Gnanasambandam, V. Koltun, and S. H. Chan, “Dynamic low-light imaging with Quanta Image Sensors,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16, pp. 122–138, Springer, 2020.
  41. C. Li, X. Qu, A. Gnanasambandam, O. A. Elgendy, J. Ma, and S. H. Chan, “Photon-limited object detection using non-local feature matching and knowledge distillation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3976–3987, 2021.
  42. A. Gnanasambandam and S. H. Chan, “HDR imaging with Quanta Image Sensors: Theoretical limits and optimal reconstruction,” IEEE transactions on computational imaging, vol. 6, pp. 1571–1585, 2020.
  43. A. Ingle, T. Seets, M. Buttafava, S. Gupta, A. Tosi, M. Gupta, and A. Velten, “Passive inter-photon imaging,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8585–8595, 2021.
  44. S. H. Chan, “What does a one-bit Quanta Image Sensor offer?,” IEEE Transactions on Computational Imaging, vol. 8, pp. 770–783, 2022.
  45. S. H. Chan, “On the insensitivity of bit density to read noise in one-bit Quanta Image Sensors,” IEEE Sensors Journal, 2023.
  46. A. Gnanasambandam and S. H. Chan, “Exposure-referred signal-to-noise ratio for digital image sensors,” IEEE Transactions on Computational Imaging, vol. 8, pp. 561–575, 2022.
  47. S. H. Chan, X. Wang, and O. A. Elgendy, “Plug-and-play ADMM for image restoration: Fixed-point convergence and applications,” IEEE Transactions on Computational Imaging, vol. 3, no. 1, pp. 84–98, 2017.
  48. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M. H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in CVPR, 2022.
  49. E. P’erez-Pellitero, S. Catley-Chandar, A. Leonardis, and R. Timofte, “Ntire 2021 challenge on high dynamic range imaging: Dataset, methods and results,” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 691–700, 2021.
  50. P. Korshunov, H. Nemoto, A. Skodras, and T. Ebrahimi, “Crowdsourcing-based Evaluation of Privacy in HDR Images,” in Optics, Photonics, and Digital Technologies for Multimedia Applications III (P. Schelkens, T. Ebrahimi, G. Cristóbal, F. Truchetet, and P. Saarikko, eds.), vol. 9138, p. 913802, International Society for Optics and Photonics, SPIE, 2014.
  51. N. K. Kalantari and R. Ramamoorthi, “Deep high dynamic range imaging of dynamic scenes,” ACM Transactions on Graphics (Proceedings of SIGGRAPH 2017), vol. 36, no. 4, 2017.
  52. G. Yu, G. Sapiro, and S. Mallat, “Solving inverse problems with piecewise linear estimators: From gaussian mixture models to structured sparsity,” IEEE Transactions on Image Processing, vol. 21, no. 5, pp. 2481–2499, 2012.
  53. C. Bao, J. F. Cai, and H. Ji, “Fast sparsity-based orthogonal dictionary learning for image restoration,” in 2013 IEEE International Conference on Computer Vision, pp. 3384–3391, 2013.
  54. K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising,” IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3142–3155, 2017.
  55. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M. H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in CVPR, 2021.
  56. V. Monga, Y. Li, and Y. C. Eldar, “Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing,” IEEE Signal Processing Magazine, vol. 38, no. 2, pp. 18–44, 2021.
  57. D. Gilton, G. Ongie, and R. Willett, “Deep equilibrium architectures for inverse problems in imaging,” IEEE Transactions on Computational Imaging, vol. 7, pp. 1123–1133, 2021.
  58. Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.
  59. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018.
  60. D. Zwillinger and S. Kokoska, CRC Standard Probability and Statistics Tables and Formulae. Chapman & Hall, 2000.
Citations (6)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 16 likes about this paper.

Sign Up for Free