Forecasting Solar Energy Using a Single Image

Abstract: Solar panels are increasingly deployed in cities on rooftops, walls, and urban infrastructure. Although the panel costs have fallen in recent years, the soft costs of installing them have not. These soft costs include assessing the illumination (irradiance) of a panel, which is typically performed using a 3D model that fails to capture small nearby structures that impact the irradiance. Our approach uses a single image taken at the panel's location to forecast its irradiance at any time in the future. We use visual cues in the image to find the camera's orientation and the portion of the sky visible to the panel in order to forecast the irradiance due to the sun and the sky. In addition, we show that the irradiance due to reflections from nearby buildings varies smoothly over time and can be forecasted from the image. This approach enables assessing the solar energy potential of any surface and forecasting the temporal variation of a panel's irradiance. We validate our approach using real irradiance measurements in urban canyons. We show that our approach often yields more accurate irradiance forecasts compared to conventional irradiance-based transposition methods and 3D model-based simulations. We also show that a single spherical image can be used to find the best fixed orientation of a panel. Finally, we present Solaris, a device to capture the image seen by a panel in a variety of urban settings.

• The paper presents a novel vision pipeline that uses a single hemispherical image to accurately forecast future solar irradiance.
• It leverages neural networks, including a TinyViT backbone and vision transformers, to estimate camera orientation and regress scene irradiance.
• Field experiments in Manhattan demonstrate significant improvements, with optimal panel reorientation potentially increasing annual energy yield by up to 27%.

Forecasting Solar Energy Using a Single Image: Technical Summary

Introduction and Motivation

The assessment and forecasting of solar panel irradiance in urban environments is a critical and unsolved challenge due to the prevalence of fine-grained geometric obstructions near candidate installation sites. Traditional approaches rely heavily on explicit 3D environment models or long-term time-lapse image collection, both of which suffer from practical and cost limitations—particularly in urban canyons where small objects and recent changes substantially impact the incident irradiance. The paper "Forecasting Solar Energy Using a Single Image" (2604.21982) presents a novel computer vision pipeline that utilizes a single hemispherical (fisheye) image taken at the panel location to automatically forecast irradiance at any future time, for arbitrary sky conditions.

Methodology

Camera Orientation and Sky Aperture Estimation

A hemispherical image inherently encodes rich geometric cues, including shadows cast by fine-scale and large-scale structures, and visible scene lines. The method employs a neural network (TinyViT backbone and dual MLP heads) trained on a large rendered and real-world dataset to robustly estimate both the sun's and gravity's directions in the camera coordinate frame, even when the sun is not visible. This is necessary due to the unreliability of magnetometer-based north estimation in dense urban environments.

The sun's direction in the earth coordinate frame is derived from the image timestamp and GPS via standard solar position algorithms. The camera's rotation is then computed by matching directional correspondences (Kabsch algorithm or an alternate closed-form solution if IMU gravity is available).

Given the image and its orientation, a segmentation network is applied to extract the sky aperture mask, which precisely captures any sky visible to the panel, including occlusions due to small and large nearby objects.

Figure 1: An individual takes a single hemispherical photo; the image is sufficient to forecast irradiance for any future sky condition.

Figure 2: Visual cues (shadows, scene lines) are leveraged to estimate sun and gravity vectors, thus determining camera orientation.

Irradiance Modeling

Sun and Sky Contributions

Using the calibrated image, the sky aperture, and future sky parameters (DNI, DHI), the sun and sky irradiance are modeled analytically. The method integrates sky radiance over the aperture using the Perez sky model, and masks the direct sun contribution to only include intervals when the sun is visible.

Figure 3: The sky aperture is segmented and used with sky models to forecast time-dependent irradiance under different conditions.

Scene Irradiance (Reflections and Indirect Lighting)

Urban surfaces contribute a non-negligible portion of total irradiance via inter-reflections and indirect light, particularly in canyons. Large-scale physically based rendering experiments reveal that scene irradiance is a temporally smooth and low-dimensional function—invariant under scale and well approximated by a small number of principal components.

A vision transformer-based network is trained to regress, from a single image and sky aperture, the full scene irradiance function as a map over possible sun positions. Training on $>$1M rendered images demonstrates the viability of this approach, which predicts function-valued outputs (via discretization in sun zenith/azimuth).

Figure 4: Schematic of scene irradiance contributions, illustrating scale-invariance and the smooth temporal nature arising from Lambertian assumptions.

Figure 5: Principal component analysis shows that scene irradiance is highly compressible; a neural network predicts it accurately from a single image.

Figure 6: End-to-end diagram of the pipeline for computing forecasted irradiance from a single captured hemispherical image.

Validation and Evaluation

A comprehensive set of experiments were conducted in various Manhattan urban locations, including rooftops and deep street canyons. In each experiment, a single hemispherical image is used for forecasting, and ground-truth irradiance is provided by a calibrated pyranometer.

Quantitative results showcase that the proposed method outperforms both state-of-the-art transposition models (e.g., System Adviser Model/SAM, which relies on isotropic sky and sky view factor assumptions) and 3D model-based approaches (which use path tracing on LiDAR/satellite-derived geometry). The critical deficit of competing methods is their inability to resolve small-scale occlusions, leading to significant error margins (up to 181% overestimation in annual yield).

Figure 7: Comparison of forecasted and measured irradiance time series from four test sites, demonstrating accurate reproduction of rapid transitions and overall energy.

Figure 8: Forecasting irradiance using a 3D model lacking small-scale detail results in substantial overestimation compared to the image-based method.

A tabular summary of forecasting errors demonstrates the superiority of the image-based approach across diverse sky conditions, with notably low daily total error even in complex canyons.

Panel Orientation Optimization

A single omnidirectional (spherical) image is sufficient to not only assess a fixed panel orientation but also sweep all possible orientations to find the global optimum with respect to annual irradiance. By extracting the 3D plane of the existing panel surface directly from the image (via geometric constraints from visible panel edges or corners), the true and the optimal energy yield is computed. Real deployments in Manhattan indicate that observed installations are often suboptimal, and realignment could increase expected annual energy by up to 27%.

Figure 9: From a spherical image above the panel, corners and edges yield the orientation in the camera frame.

Figure 10: Optimization of panel orientation in the field yields substantial potential yield increases, highlighting suboptimal current installation practice.

Practical Deployment: Solaris Device

The method is instantiated as the Solaris device, a readily-deployable imaging system integrating a commercially available spherical camera and custom alignment chassis. It can be rapidly placed on any candidate surface—flat, vertical, or sloped—or used to image the view from an already installed panel.

Figure 11: Solaris device for capturing hemispherical/spherical images on various urban surfaces, enabling rapid in situ irradiance forecasting.

Numerical and Empirical Highlights

• The scene irradiance neural network achieves an average error of 0.5% in annual energy forecasting (relative to total annual irradiance), compared to a 12.3% systematic underestimation when scene irradiance is ignored.
• Field validation across 20 days and four environments shows the proposed method is less sensitive to sky conditions and captures step transitions produced by sun passage within/outside sky apertures.
• 3D model and transposition-based methods show significantly higher error and bias, confirming the irreplaceable role of small-scale geometric detail in complex urban domains.

Limitations

• The method assumes images are captured in sunny conditions with discernible shadows for orientation estimation; overcast acquisition is not supported.
• It cannot account for future modifications to the local geometry (e.g., construction, seasonal vegetation change) post-image capture, nor for transient effects from small, fast-moving clouds.
• For the estimation of irradiance under non-standard sky conditions, errors in satellite-derived DNI, DHI, or GHI inject forecasting inaccuracies shared across all compared methods.

Implications and Future Directions

This work redefines the minimum viable sensing required for high-fidelity urban solar forecasting by leveraging visual context and spatial priors rather than explicit 3D models or labor-intensive time series. The approach drastically reduces assessment costs and latency, enabling real-time in situ evaluation and retrofitting guidance.

Theoretically, the reduction of complex irradiance forecasting to a learned low-dimensional regression from an image highlights the high information content of visual encodings for global illumination phenomena, with broader implications for scene understanding and inverse rendering in uncontrolled environments.

Practical adoption is immediately feasible and could drive significant increases in urban solar integration by reducing assessment costs, improving placement, and facilitating reorientation audits. Prospective extensions include forecasting from images taken at distance (with transfer learning for novel-view hallucination) and adaptation to changing environments via periodic image recapture or integration with semantic change detection systems.

Conclusion

"Forecasting Solar Energy Using a Single Image" (2604.21982) demonstrates that a single zenithal or spherical image from the intended panel site is sufficient for accurate, physics-informed future irradiance forecasting. The proposed architecture achieves this via robust orientation inference using local and global shadows, precise aperture masking, and neural regression of temporally parameterized indirect scene contributions. This approach eliminates the dependence on explicit 3D geometric models and their associated costs, providing accurate, rapid, and practical tools for solar deployment in complex urban environments.