Systematic Review on Ground-Based Cloud Tracking Methods for Photovoltaics Nowcasting ()
1. Introduction
In recent years we have been witnessing unprecedented growth of renewable energy generation (Bojek, 2022), but renewables are highly dependent on local weather conditions (Ruther et al., 2003), making their resources intermittent. Photovoltaic (PV) was especially impacted by cost reduction in recent years (Kavlak, McNerney, & Trancik, 2018), but has its generation affected not only by the solar cycle but also by factors such as clouds, local temperature, and other climatic parameters (Bel & Bandi, 2019; Cai & Aliprantis, 2013; Denholm & Margolis, 2007; Golestaneh, Pinson, & Gooi, 2016; Iqbal, 1983; Duffie & Beckman, 2006).
Predicting local changes in irradiance due to passing clouds is a key challenge in solar power generation (Fouad, Shihata, & Morgan, 2017). The intermittency of solar resources, particularly during cloudy conditions (Jewell & Ramakumar, 1987; Lappalainen & Valkealahti, 2017; Shah, Mithulananthan, Bansal, & Ramachandaramurthy, 2015), necessitates the development of technologies and strategies to forecast and mitigate energy production fluctuations (Gonzalez-Moreno, Marcos, de la Parra, & Marroyo, 2022; Marcos, Storkël, Marroyo, Garcia, & Lorenzo, 2014). Clouds exhibit seasonal and daily variability, influenced by atmospheric stability and upper layer conditions, further complicating short-term forecasts (Arya, 2001). Cloud dynamics, including variations in thickness, shape, and volume, pose challenges for accurately estimating local cloud cover over specific PV plants (Arya, 2001). Factors such as cloud velocity and shadow coverage influence the rate of energy output fluctuations in PV plants (Jewell & Ramakumar, 1987; Marcos, Marroyo, Lorenzo, Alvira, & Izco, 2011a, 2011b).
Power variations of up to 90%/min. were observed for different locations and plant sizes (Marcos et al., 2011b). Figure 1 illustrates two atmospheric situations where large cloud-induced PV power fluctuations can repeatedly occur during a very short period. In the left illustration, (a, b) are moving shadows caused by individual passing clouds, while on the right one (c, d) are lighter regions areas generated by holes in a fast-moving dense cloud cover.
Figure 1. Atmospheric situations where large cloud-induced PV power fluctuations can repeatedly occur during a very short time span.
The impact of passing clouds on the energy ramp rate (RR) is demonstrated in Figure 2, depicting power production overlaid on sky imager photographs from the UFSC Photovoltaic Lab in Southern Brazil. In just 4 minutes, peak power from an experimental PV array fluctuates dramatically, from 1700 W (a) to 700 W (b), then back up to 1900 W (c). The surplus irradiation in (c) surpasses the normal clear sky rate due to the cloud-border effect, caused by multiple reflections of surrounding clouds (Pecenak, Mejia, Kurtz, Evan, & Kleissl, 2016; Thuillier, Perrin, Keckhut, & Huppert, 2013; Yordanov, Midtgård, Saetre, Nielsen, & Norum, 2013; Yordanov, 2015).
Figure 2. PV power fluctuation in a time window of 4 minutes showing the corresponding sky imager view.
According to Kariniotakis (2017) [sec 1.1.3, 7], the recommended temporal scale for observing ramps caused by clouds is almost like a “situation awareness” on the site. Shortest time span forecasting of solar irradiation and consequent PV fluctuations, inside of a time window of 5 minutes or less, also called nowcasting, can provide vital information for optimizing battery and grid usage (Gonzalez-Moreno et al., 2022).
Cloud data for nowcasting must be collected with the lowest latency possible and have to possess the highest spatial resolution possible, to identify individual small clouds minutes before they affect solar irradiation. These data can neither be provided by weather satellites in low Earth orbits, nor by geostationary satellites and have to be collected from ground-based observation.
Satellite images are discouraged for cloud observation due to limitations in spatial resolution, distortion, and latency in data transmission and processing. While exclusive satellite services could address these issues, their high costs typically exceed the budget of most renewable energy utility companies.
Another way to mitigate intermittent oscillations caused by clouds in the generation is photovoltaic-storage systems (PVS). Solutions like reused automotive (Sawin et al., 2018) or specially designed batteries (Fu, Remo, & Margolis, 2018) are being associated with PV systems to reduce intermittency. Nevertheless, forecasting is still necessary to manage the energy dispatch to keep grid reliability (Nottrott, Kleissl, & Washom, 2012), to avoid operational problems, additional maintenance, equipment stresses (Chen, Li, Brady, & Lehman, 2010; Inman, Chu, & Coimbra, 2016; Luoma, Kleissl, & Murray, 2011; Mccormick & Suehrcke, 2018), and cloud border over-irradiation (do Nascimento, de Souza Viana, Campos, & Rüther, 2019; Inman et al., 2016; Martins, Mantelli, & Rüther, 2022; Pecenak et al., 2016; Thuillier et al., 2013; Toreti Scarabelot, Arns Rampinelli, & Rambo, 2021; Yordanov et al., 2013; Yordanov, 2015).
Three previous reviews, predating our selected period, focused on Direct Normal Irradiance (DNI) for thermal plants. Pavlović, Radonjić, Milosavljević, & Pantić (2012) and Siva Reddy, Kaushik, Ranjan, & Tyagi (2013) examined existing concentrating solar plants (CSP) globally in 2012, emphasizing climate conditions and feasibility analysis, but did not address forecasting. Inman, Pedro, and Coimbra (2013) provided a comprehensive overview of forecasting methods, covering solar irradiance components, clear sky models, and evaluation techniques. The review encompassed regressive, AI, remote sensing, numerical weather prediction (NWP), and local variables for short-term and hybrid forecasting, without proposing new methods. The authors concluded with desired performance criteria for techniques.
The complexity of cloud-induced fluctuations in solar energy underscores the need for effective strategies to address them. For accurate nowcasting of local cloud coverage, especially for smaller PV plants, automated ground-based cloud observations are essential. Such a system should encompass cloud identification, measurement, classification, cloud movement tracking and prediction, and nowcasting of irradiance changes. Since Inman et al. (2013), significant advancements in Machine Learning (ML) methods for the PV sector have emerged. Task (a) was addressed in Juncklaus Martins et al. (2022), while this review focuses on task (b), systematically examining the state of the art in cloud movement estimation and tracking methods.
2. Observation Methods
The choice of the Ground-based Sky Image Acquisition System (GSIAS) is crucial for effectively capturing cloud images. Among the most used methods, the Whole Sky Imager and the Total Sky Imager (TSI) stand out. These devices capture hemispherical images of the sky through fisheye lenses, which allow a full 180-degree view of the sky. A detailed study of GSIAS can be found at Lin, Zhang, & Wang (2023).
These methods determine the quality and resolution of the images obtained, covering detailed information (Lin et al., 2023; Peng et al., 2015). Ground-based systems face challenges in accurately detecting and identifying cloud pixels due to lighting variations throughout the day and under different weather conditions (Peng et al., 2015). However, ground-based cameras face significant challenges in accurately detecting and identifying cloud pixels due to lighting variations throughout the day and under different weather conditions (Peng et al., 2015). These variations can introduce distortions and artifacts into images, complicating analysis and requiring advanced preprocessing techniques.
For the process of acquiring images captured on the ground when constructing the datasets, most of the selected articles (Table 1) used TSI to capture images, some used fisheye-equipped cameras and others used ground-based cameras.
Sky Image Preprocessing
Image pre-processing is an important step in the context of GSIAS. It involves applying various techniques to improve the quality of captured images and prepare the data for subsequent analysis. This process is crucial to deal with lighting variations throughout the day, different weather conditions and distortions introduced by atmospheric factors in the images captured by GSIA.
Table 1. List of GSIAS used in the articles.
Method |
Amount |
Articles |
TSI |
18 |
El Jaouhari, Zaz, & Masmoudi (2015); Marquez & Coimbra (2013); Sun et al. (2014); Quesada-Ruiz, Chu, Tovar-Pescador, Pedro, & Coimbra (2014); Chang, Yao, Li, Tong, & Tu
(2017); Peng et al. (2015); Xu et al. (2015); Cervantes,
Krishnaswami, Richardson, & Vega (2016); Peng, Yu,
Huang, Heiser, & Kalb (2016); Cheng (2017); Richardson,
Krishnaswami, Shephard, & Vega (2017a); Cañadillas, Gonz’alez-D’ıaz, Rodr’ıguez, Rodr’ıguez, & Guerrero-Lemus (2018); Bone, Pidgeon, Kearney, & Veeraragavan (2018); Zhang et al. (2019); Zhen et al. (2019); Tiwari, Sabzehgar,
& Rasouli (2019); Nouri et al. (2019); Caldas &
Alonso-Su’arez (2019) |
Fisheye Lens |
10 |
El Jaouhari et al. (2015); Bernecker, Riess, Angelopoulou, & Hornegger (2014); Chow, Belongie, & Kleissl (2015); Zhen, Wang, Mi, Sun, & Sun (2015); Richardson, Krishnaswami, Vega, & Cervantes (2017b); Dissawa, Ekanayake, Godaliyadda, Ekanayake, & Agalgaonkar (2017); Magnone, Sossan, Scolari, & Paolone (2017); Saleh, Meek, Masoum, & Abshar (2018);
Ao, Xuer, Salinas, & Chin (2019); Eşlik, Akarslan, &
Hocaoğlu (2021) |
Ground-based |
1 |
Zhang, Du, Chen, & Lim (2018) |
Preprocessing techniques include brightness and contrast correction, noise removal, correction of geometric distortions caused by fisheye lenses, and normalization of images to standardize lighting conditions. Some equipment, such as the TSI, has automatic lighting correction (Sawant, Shande, Feij’oo-Lorenzo, & Bokde, 2021; Zhang et al., 2018). Additionally, it is often necessary to convert from one color space to another. This conversion can highlight specific characteristics of clouds, facilitating segmentation and analysis. Different color spaces, such as RGB (Red, Green, Blue) and HSV (Hue, Saturation, Value), offer different advantages depending on the application (Sun et al., 2014). For example, the HSV color space can separate intensity information (Value) from color information (Hue and Saturation), making it easier to detect clouds under varying lighting conditions (El Jaouhari et al., 2015).
Techniques of Histogram Equalization (Peng et al., 2015; Zhen et al., 2019), Edge Detection (Eşlik et al., 2021), Background Subtraction (Zhang et al., 2018), Masking (Dissawa et al., 2017; Magnone et al., 2017; Zhang et al., 2019) are widely used to prepare the captured raw images by GSIAS for advanced analytics.
3. Cloud Recognition Methods
The following subsections explore the methodologies developed in selected articles for cloud identification. According to the study presented by Lin et al. (2023), identification techniques can be grouped into three fundamental categories: Thresholding, Advanced Image Segmentation and Mchine Learning (ML).
3.1. Thresholding
One of the methods and approaches applied to perform automatic image segmentation is thresholding. This method is widely adopted in studies (Bernecker et al., 2014; Bone et al., 2018; Caldas & Alonso-Su’arez, 2019; Cervantes et al., 2016; Chang et al., 2017; Dissawa et al., 2017; El Jaouhari et al., 2015; Marquez & Coimbra, 2013; Peng et al., 2015; Zhang et al., 2018; Zhang et al., 2019).
This technique consists of dividing the image into two distinct classes, normally selecting a threshold value. An evolution of this approach was employed in paper of Marquez and Coimbra (2013) based on work by Li, Lyu, and Yang (2011), which explored hybrid thresholds. In Quesada-Ruiz et al. (2014), the author presented a hybrid threshold classification algorithm, which combines several thresholding techniques to improve the accuracy of cloud identification.
In some studies (Caldas & Alonso-Su’arez 2019; Chow et al., 2011; Zhang et al., 2018), researchers adopted adaptive thresholding techniques for different pixels, with the aim of tackling classification errors resulting from lighting variations.
3.2. Advanced Image Segmentation
The advanced image segmentation category includes more complex methodologies such as edge-based segmentation, region growth, and feature detection algorithms. Edge-based segmentation seeks to identify cloud contours, while region growing groups similar pixels together to create larger regions.
In the study by Dissawa et al. (2017), an approach was employed to identify individual clouds between image frames. This approach involved using the normalized Cross-Correlation Method (CCM) to establish correspondences between clouds at different times.
To extract relevant features and vectors in Cheng (2017) and Dissawa et al. (2017), the feature detection algorithm Harris was adopted. The authors at Eşlik et al. (2021) have identified the most suitable spots for cloud tracking. In this research, the Shi-Tomasi algorithm was employed to detect cloud contours and corners.
When compared to threshold-based approaches, more advanced image segmentation techniques generally demonstrate superior performance in recognizing specific features. Image segmentation is more flexible and can consider a wider range of pixel attributes such as brightness, color, and texture, resulting in more accurate and detailed results (Lin et al., 2023).
3.3. Machine Learning
ML techniques play a crucial role in automating cloud identification in images, leveraging their ability to learn intricate patterns and extract relevant features. Methods like Support Vector Machines (SVM), Artificial Neural Networks (ANN), and Decision Trees enable automatic classification, surpassing manual approaches in efficiency and accuracy.
Studies such as Peng et al. (2015, 2016), Sun et al. (2014) and Xu et al. (2015) highlight SVM’s popularity for cloud identification. Additionally, classifiers like Random Forest and Bayesian Classifier show promise in this regard. The Boltzmann Constrained Machine (BCM), an ANN algorithm, aids in cloud segmentation by extracting significant features from images, and enhancing pattern representation (Bernecker et al., 2014).
In Peng et al. (2015), a two-tier cross-validation approach with SVM classifier yielded high accuracy in cloud identification, surpassing an 83.2% hit rate compared to manual classification masks. However, limitations were noted in detecting multilayer clouds near the sun’s position and thin clouds.
On the other hand, Zhen et al. (2019) employed K-means clustering for cloud identification and classification, along with texture features based on a gray-level cooccurrence matrix, enabling classification of various cloud classes.
In Cazorla, Olmo, and Alados-Arboledas (2008), MLP was utilized for real-time cloud classification based on internal sky images captured by a camera, distinguishing clear skies, dense clouds, and thin clouds.
ML techniques have proven highly effective in cloud detection, as highlighted in studies like Cheng & Lin (2017); Lin et al. (2023); Peng et al. (2015); Richardson et al. (2017a, 2017b), showcasing significant improvements in precision and accuracy.
4. Cloud Tracking Methods
Tracking objects is a challenging task, and tracking clouds is even more complex due to the multiple variables that influence their displacement and orientation. Clouds can suffer partial or total occlusion, rapid lighting variations, dynamic deformations and unpredictable movements influenced by meteorological factors. Furthermore, clouds vary significantly in scale and perspective, making predicting their direction and movement a difficult task (Arya, 2001; Chow et al., 2011; Hamblyn et al., 2021; Peng et al., 2015).
To perform cloud tracking, it is necessary to calculate the Cloud Motion Vector (CMV), which is a mathematical representation that describes the direction and speed of cloud movement in a sequence of images or videos. The main techniques used by the authors include three main CMV formats (Bernecker et al., 2014; Chang et al., 2017; Chow et al., 2011; Lin et al., 2023; Marquez & Coimbra, 2013; Zhen et al., 2019):
1) Global CMV (g-CMV): A single motion vector is used to represent the displacement of all clouds in the image.
2) Multiple CMVs (m-CMVs): Multiple motion vectors are assigned to different regions of the image.
3) Dense Vector Field (DVF): This is the most detailed representation, where each pixel in the image has a corresponding motion vector.
The choice between different CMV formats, such as global (g-CMV), local (m-CMV), and DVF, depends on the specific application requirements. While g-CMVs offer a broad view of cloud movement, m-CMV and DVF are preferred for capturing detailed local cloud behavior. Typically, g-CMV is derived from m-CMV or DVF using aggregation or analysis methods (Chow et al., 2011; Lin et al., 2023; Zhen et al., 2019).
Similarity Maximization (SM) is a central approach in many techniques used to calculate g-CMV, m-CMV and DVF. This technique involves finding the optimal geometric transformation to align multiple cloud images, maximizing their similarity. There are several SM techniques, and the main ones found in this study include the Block Matching (BM), Cross-Correlation Method (CCM), Optical Flow (OF), Feature Matching (FM) and Machine Learning approaches.
4.1. Block Matching
The BM approach involves dividing the images into blocks or patches and then looking for correspondences between these blocks in the images being compared. The fundamental task is to find the placements that optimize the similarity between the image blocks, thus adjusting the transformation that best aligns the cloud images. This technique was used by the authors in Magnone et al. (2017); Nouri et al. (2019); Peng et al. (2016); Zaher, Thil, Nou, Traor’e, & Grieu (2017); Zhen et al. (2019).
The BM approach to cloud tracking often combines this technique with other approaches to improve tracking accuracy and robustness. Combining BM with other techniques can increase the computational load, but it also improves the accuracy and robustness of cloud tracking. In the study of Peng et al. (2016), a hybrid tracking technique was proposed to incorporate the strength of BM and OF. The results show that the hybrid approach outperformed classical models, reducing at least 30% motion estimation errors compared to real motions in most simulated image sequences.
4.2. Cross-Correlation Method
The Normalized Cross Correlation (CCM) approach is used to establish correspondences between patterns in different frames, with the aim of determining the offset between these patterns. In this technique, the image is divided into small windows, where the correlation is calculated to find the correlation peak, which corresponds to the motion vector. The CCM was employed in several studies, including Ao et al. (2019); Caldas & Alonso-Su’arez (2019); Dissawa et al. (2017); Marquez & Coimbra (2013); Peng et al. (2015); Xu et al. (2015).
The CCM technique is robust and simple as it can effectively identify accurate matches even in the presence of noise and intensity variations. However, calculating the correlation for all possible windows can be computationally intensive, especially for high-resolution images. To mitigate this problem, optimized versions such as the normalized CCM can be used. In Dissawa et al. (2017), this normalization was used to find the correspondence of each cloud in each frame.
4.3. Optical Flow
One of the primary techniques for tracking clouds is Optical Flow (OF). This technique estimates pixel displacement between sequential frames based on intensity variations over time. It includes dense and sparse techniques: Sparse OF tracks key points, while Dense OF computes motion vectors for every pixel. Table 2 presents the selection of OF algorithms made by each article that used this method. The following sections will discuss the algorithms employed in the selected articles, providing a comprehensive analysis of their application and effectiveness.
Table 2. List of optical flow algorithms for cloud tracking.
4.3.1. Sparse Optical Flow
Sparse OF is a tracking technique that focuses on calculating the movement of selected points of interest. These points are typically chosen using feature detectors such as Harris, Shi-Tomasi, or SIFT. Sparse OF calculates movement only at these selected points, rather than across all pixels in the image, which makes it appealing due to its computational efficiency.
Among the sparse OF algorithms, Lucas-Kanade is the most chosen by the authors, as indicated in Table 2. This algorithm is widely preferred due to its computational efficiency, accuracy for small displacements and its ease of implementation. The LucasKanade is effective for tracking small movements and smooth intensity variations, and it may falter with large movements or abrupt intensity changes (Peng et al., 2016).
In studies like Dissawa et al. (2017) and El Jaouhari et al. (2015), cloud movements were tracked by detecting interest points using the Harris feature detector, then employing the Lucas-Kanade algorithm. Eşlik et al. (2021) calculated tracked points using the Shi-Tomasi algorithm, subsequently applying the Lucas-Kanade OF algorithm.
4.3.2. Dense Optical Flow
Dense optical flow is a technique used to estimate the motion of all pixels in a sequence of images. Unlike sparse optical flow, dense optical flow provides a comprehensive motion field for every pixel, offering a detailed representation of the motion across the entire image. Dense optical flow algorithms calculate the motion vectors for every pixel, resulting in a dense field of motion vectors that describe the displacement of each pixel from one frame to the next.
The main dense OF algorithms that have been used include Horn-Schunck, Gunnar-Farneback, Bruhn, SimpleFlow, Motion Template and the mixed Variational Optical Flow (VOF) algorithm (Table 2) The Horn-Schunck algorithm was the most chosen among the dense OF. For cloud tracking, this algorithm estimates the movement of pixels in image sequences based on equations that consider spatial smoothing and temporal continuity of pixel intensities. Although it is effective for calculating cloud motion vectors between frames, the Horn-Schunck algorithm may face limitations in scenes with abrupt motion changes or inconsistent textures.
The Gunnar Farneback algorithm is known for its resilience to significant changes and occlusions, is computationally efficient, but can have problems with heavily textured or repeated image sections. In Cañadillas et al. (2018), Gunnar-Farneback was chosen to determine cloud trajectories, with ellipse modeling to address fisheye lens effects. Cloud movement directions were obtained from weighted averages of angles and magnitudes derived from OF.
In Tiwari et al. (2019), Gunnar Farneback’s algorithm was employed in a two-step process to track image features. Coarse-to-fine OF with the Farneback method allowed efficient tracking of objects over long distances due to 10-minute image separation. In West et al. (2014), several OF algorithms, including Farneback, were evaluated. The Farneback algorithm was chosen for its flexible range of parameters, low computational intensity and high accuracy.
The Bruhn’s algorithm, a variational optimization technique, enhances OF accuracy in regions with significant intensity variations, complex textures, and occlusions. It employs an iterative energy function minimization process to optimize motion vectors for each image point, resulting in more robust and accurate OF estimates, particularly in challenging scenarios (Bruhn, Weickert, & Schnörr, 2005).
In Ao et al. (2019), a cloud tracking model based on Bruhn’s OF approach was developed and compared with traditional OF models. The model utilized a histogram and RGB channel for pixel identification, and Bruhn’s method was employed to determine OF between images. Comparisons with Lucas-Kanade, Horn-Schunck algorithms, and Crude Validation technique showed similar tracking accuracy, but Bruhn’s algorithm significantly improved computation time by 39.48%.
The research of Chow et al. (2015) employed the VOF algorithm, which differs from traditional OF methods. While conventional approaches focus on minimizing intensity errors between consecutive frames, VOF approaches the problem as a variational optimization process to minimize an energy function. This approach allows for the incorporation of additional considerations such as spatial smoothing and motion constraints, resulting in more accurate and robust OF estimates, especially in areas with varying intensity or occlusions.
4.4. Feature Matching
The Feature Matching technique involves finding correspondences between points of interest or features in different images. The objective is to identify points of interest that are invariant to transformations such as rotation, scale and lighting changes. For this, algorithms such as Harris-Stephens (Harris Corner Detector), SIFT (ScaleInvariant Feature Transform) and SURF (Speeded-Up Robust Features) are used. Table 3 shows the articles that used the feature matching technique.
Table 3. List of techniques for feature matching.
In Cañadillas et al. (2018), the SIFT algorithm was used to establish correspondences between images from the TSI cameras, followed by mapping into azimuthal and zenithal coordinates using the Gunnar-Farneback algorithm. On the other hand, Zhen et al. (2019) used the SURF algorithm in conjunction with the Lucas-Kanade algorithm. Meanwhile, Peng et al. (2016) proposed a hybrid cloud motion tracking model combining BM and OF. In Dissawa et al. (2017), Harris feature detection was used to identify characteristic points of clouds, allowing the detection of cloud deformation and movement speed.
The Feature Matching process involves several steps, each with its own computational complexity. Feature detection and description can be intensive, especially with algorithms like SIFT and SURF.
4.5. Machine Learning Approaches
In the field of terrestrial cloud image research, Machine Learning (ML) and Deep Learning (DL) techniques have been gaining prominence, predominantly in cloud recognition and classification, as illustrated in several studies (Arrais et al., 2022; Bernecker et al., 2014; Cazorla et al., 2008; Cheng & Lin, 2017; Juncklaus Martins et al., 2022; Martins et al., 2022, 2023; Peng et al., 2015, 2016; Richardson et al., 2017a; Sun et al., 2014; Xu et al., 2015; Zhen et al., 2019). However, the potential of neural networks extends beyond this, and can be applied in predicting cloud movements in terrestrial images, as evidenced by Arrais et al. (2022); Lu, Wang, Li, & Zhang (2021); Martins et al. (2022); Pierce, Stein, Braid, & Riley (2022); Su, Li, An, & Wang (2020).
Convolutional Neural Networks (CNNs) can be trained to directly estimate CMV from image sequences. Advanced models can learn to detect complex movement patterns that are difficult to capture with traditional methods. Pierce et al. (2022) used the convolutional autoencoder (CAE) technique, specifically U-net, to identify clouds and a particle tracker to predict cloud movement. Among the techniques used were the Otsu method, thresholding in RB and HSV, and the K-Means algorithm for color segmentation. In conclusion, the author highlighted that a minimum of labeled data is necessary to obtain state-of-the-art results, in addition to a fast and robust application for forecasting and tracking cloud movement.
In Lu et al. (2021), the Cascade Causal Long Short-Term Memory (CCLSTM) was utilized to estimate cloud morphology and displacement speed, supplemented by the Super-Resolution Network (SR-Net) to enhance results. However, the model struggled with clouds passing through shading belts, often misclassifying them as disappeared. Meanwhile, in Eşlik et al. (2021), researchers used the Shi-Tomasi algorithm to identify tracking points, followed by the Lucas-Kanade OF algorithm for monitoring. Cloud movement estimates were then obtained through a Feedforward Backpropagation ANN.
4.6. Advanced and Complementary Cloud Tracking Methods
While traditional and machine learning methods are widely used, there is a growing need for advanced and complementary techniques that can address the specific challenges of cloud tracking.
4.6.1. Displacement Vector Field Filtering
The Displacement Vector Field (DVF) filtering technique is mainly used in the context of OF to improve the accuracy and consistency of the calculated motion vectors. In dense OF, where the movement of all pixels is estimated, filtering is essential to smooth the vector field and eliminate noise, using methods such as mean, Gaussian or bilateral filters. In sparse OF, although less common, filtering can also be applied to refine the motion vectors of specific points of interest. Algorithms such as HornSchunck and Farnebäck incorporate smoothing and regularization techniques to obtain a more uniform and accurate vector field.
The DVF technique was implemented in Lin et al. (2023) and Peng et al. (2016) studies as a post-processing step to improve vector accuracy and remove noise and imperfections in the images. In Peng et al. (2016) techniques such as the vector median filter and the sky filter were applied to remove low-magnitude noise.
4.6.2. Particle Image Velocimetry
Particle Image Velocimetry (PIV) is a velocity measurement and flow analysis technique used primarily in fluid dynamics. PIV allows the visualization and quantification of velocity fields in a fluid by tracking the movement of tracer particles introduced into the fluid.
PIV iteratively applies the BM algorithm using progressively smaller block sizes. This hierarchical search generates DVF, providing comprehensive information about the movements of particles such as clouds. Studies such as Magnone et al. (2017); Marquez & Coimbra (2013); Peng et al. (2016); Quesada-Ruiz et al. (2014) chose the PIV technique to calculate cloud velocity fields. In Marquez & Coimbra (2013); Peng et al. (2016), the authors used the ’MPIV’ software developed by Mori (2002).
4.6.3. Global Cloud Motion Vectors
The g-CMV represents the overall movement of clouds in an image. Initially, feature points are selected to estimate cloud movement between frames using techniques like CCM, OF, and BM. Displacement vectors indicating direction and magnitude of cloud movement are then calculated for each point, and statistical measures are computed to obtain the g-CMV.
In Chow et al. (2011), a cloud movement prediction method is employed using a time map of clouds. By determining cloud direction and speed through CCM between current and past images, the cloud map is advected to predict future positions. This involves moving all images according to identified motion vectors, validated by comparing the forecasted cloud cover with observed reality 30 seconds later.
Both methodologies involve validation steps: comparing the advected map with observed reality for cloud cover forecasting, and comparing compensated images with subsequent images to assess g-CMV effectiveness. Additionally, in Bernecker et al. (2014), the technique from Chow et al. (2011) was used to select the highest frequency CMV, determining the strain field using the demons algorithm.
Another approach to simplify cloud motion vectors is by utilizing clustering algorithms on g-CMVs, categorizing and condensing vast amounts of vectors into a more manageable representation of global movement (Lin et al., 2023). Grouping similar CMVs together provides a more generalized view. For instance, algorithms like K-means have been used for this purpose (Marquez & Coimbra, 2013).
In Zhen et al. (2019), the authors proposed an algorithm combining three techniques, including CMM, where g-CMV represents a weighted combination of these results, with weights determined heuristically.
In contrast to traditional methods relying on m-CMVs and DVFs, newer algorithms aim to calculate g-CMVs directly. For example, Quesada-Ruiz et al. (2014) proposed an approach based on circular strips centered on the Sun, dividing images into distinct segments. Analyzing cloud changes in these segments helped determine the g-CMV. Subsequent work by Bone et al. (2018) modified the g-CMV calculation formula from Quesada-Ruiz et al. (2014) to enhance optimization and accuracy.
4.6.4. Cloud Distribution Extrapolation
Cloud Distribution Extrapolation (CDE) is a predictive method used to forecast cloud movement and evolution based on current observations and past patterns (Lin et al., 2023). Despite its effectiveness, CDE encounters challenges such as reduced accuracy over longer prediction horizons and abrupt changes in wind conditions. To overcome these limitations, researchers have explored alternative approaches.
One promising strategy involves utilizing CMV time series, allowing for a more comprehensive analysis by incorporating historical data (Bernecker et al., 2014; Dissawa et al., 2017; Lin et al., 2023). Various time series forecasting techniques, including linear regression, moving average filters, and the Kalman filter, have been investigated to enhance cloud distribution extrapolation (Bernecker et al., 2014). Additionally, particle filtering and vector autoregressive filters present viable options for improving the accuracy and reliability of cloud movement predictions (Bone et al., 2018; Dissawa et al., 2017).
4.7. Validation and Evaluation Metrics
Evaluating the accuracy and effectiveness of cloud tracking methods can be challenging due to the diversity of metrics used by researchers. Metrics such as Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and others play a crucial role in objectively evaluating the performance of cloud tracking models and algorithms. Studies like Peng et al. (2015) have utilized metrics like MAE and RMSE to assess cloud tracking accuracy, where MAE measures average accuracy and RMSE penalizes larger errors more significantly.
Similarly, Nouri et al. (2019) employed MAE and RMSE to validate cloud height and movement vectors, while Eşlik et al. (2021) utilized metrics like RMSE, MAE, and Mean Absolute Percentage Error (MAPE) to evaluate the effectiveness of ANN in predicting cloud movements. In Chow et al. (2015), the accuracy of the VOF method’s forecasts was evaluated by transforming binary cloud images into Cartesian coordinates to create a “cloud map”, which was then compared with predictions from the CCM method. The accuracy assessment involved superimposing the actual cloud map onto the “advected” cloud map and calculating the pixel-wise error, termed “Matching Error”.
In Magnone et al. (2017), it was utilized to assess the performance of cloud detection and movement identification. In Bernecker et al. (2014), Precision, Recall, and F2 Score were employed initially to evaluate occlusion predictions’ accuracy and recall. Later, RMSE was used to assess the second stage of the methodology.
Meanwhile, Peng et al. (2015) introduced the Success Tracking Index (STI), measuring the percentage of dataset instances where all 25 solar blocking pixels were successfully included. In the study by Peng et al. (2015), five metrics were utilized to evaluate the model’s performance. These metrics include the Colorful OF Map, Average Angular Error (AAE), Standard Deviation of Angular Error (STDANG), Average End Point Error (AEPE), and MAE. Each metric serves a specific purpose in assessing the accuracy and effectiveness of the model in predicting cloud movement.
In the study by Zaher et al. (2017), Peak Signal-to-Noise Ratio (PSNR) and MSE metrics were employed to assess algorithms, focusing on image intensity and reconstruction accuracy. In contrast, Ao et al. (2019) evaluated cloud tracking methods based on prediction accuracy, distinguishing correct and incorrect pixel predictions. They measured tracking accuracy by the proportion of correct predictions to the total number of pixels, excluding undefined pixels.
Each article presents its own methodology for detecting, calculating CMV and tracking clouds. Each technique used has specific metrics suitable for its evaluation, and it is common for articles to use more than one type of metric to ensure a comprehensive analysis. This diversity of metrics and methodologies, however, complicates direct comparison between studies, making it difficult to determine which is the best metric or tracking technique. Each set of metrics can highlight different aspects of a technique’s performance, such as accuracy, robustness, computational efficiency, or resilience to adverse conditions.
As a result, there is no clear way to identify the most effective metric or most accurate tracking technique without a unified data set and standardized benchmarking methodology. Cloud detection and tracking are fundamental for several areas, including the study of solar irradiation. Understanding the movement and distribution of clouds makes it possible to predict the amount of sunlight reaching the Earth’s surface, which is crucial for managing solar energy systems and predicting climate.
5. Solar Irradiation Prediction
Solar irradiance, encompassing Direct Normal Irradiance (DNI), Diffuse Horizontal Irradiance (DHI), and Global Horizontal Irradiance (GHI), plays a crucial role in PV systems, directly impacting energy output. DNI, with its direct and intense nature, is particularly significant for PV energy generation, while GHI serves as a pivotal parameter in assessing the capacity and energy potential of PV technology. Various studies, including Bernecker et al. (2014); Bone et al. (2018); Caldas & Alonso-Su’arez (2019); Cañadillas et al. (2018); Chow et al. (2011); Magnone et al. (2017); Marquez & Coimbra (2013); Peng et al. (2015); Quesada-Ruiz et al. (2014); Richardson et al. (2017a, 2017b); Xu et al. (2015); Zaher et al. (2017), have underscored the importance of these components.
In forecasting solar irradiance, quantitative methods predominate, providing numerical values related to irradiance. Three primary predictive models are identified in recent research, including Indirect Forecasting, ML Forecasting, and a hybrid approach that integrates both direct and indirect methods, as outlined in Lin et al. (2023).
5.1. Solar Indirect Forecast
Solar forecasting through the indirect method involves using cloud forecast results as input, focusing on interpreting cloud characteristics to deduce their impact on solar irradiation. Inputs like cloud rada results, estimated CVM, and cloud motion maps (CMM) serve as indicators of atmospheric state correlated with solar irradiation. Local information is leveraged in works such as Bone et al. (2018); Chang et al. (2017); Marquez & Coimbra (2013); Quesada-Ruiz et al. (2014), where segmentation techniques like block ladder and sector ladder methods are utilized to identify and categorize specific regions in images, aiding in delineating cloud areas for detailed analysis of their characteristics and movements.
In the study of Marquez and Coimbra (2013), the block ladder approach was adopted due to its ability to identify and track small regions of interest in the images. The sector ladder, used in Bone et al. (2018); Quesada-Ruiz et al. (2014), is particularly effective in distributing images in radial segments, facilitating the analysis of cloud movement patterns in specific directions.
An alternative approach to solar forecasting involves incorporating global scope information, focusing on broader-scale variables and trends. This strategy typically involves two phases: extrapolating cloud distribution using terrestrial images to anticipate future conditions, followed by correlating the projected image with effective solar irradiance.
Geometric models, as described in Cervantes et al. (2016); Chow et al. (2011), form the basis of the first approach. Here, pixels in the image are projected onto the ground following an optical path, with solar irradiance estimated using cloud radar (CR) results for the corresponding pixel.
In the data-driven approach, real datasets are utilized to train a function mapping images directly to solar irradiance. This function often incorporates selected features extracted directly from images or derived from CR results as primary inputs, as outlined in Marquez and Coimbra (2013).
Over time, various mapping functions have been explored and evaluated for their effectiveness in solar forecasting. These include traditional techniques like simple linear regression (Marquez & Coimbra, 2013) as well as more advanced ML methods incorporating non-linear models (Moncada, Richardson Jr., & Vega-Avila, 2018). However, both geometric and data-based approaches face challenges and limitations that can affect the accuracy of solar predictions.
Geometric methods, while theoretically accurate, have shown instability in solar forecasts, as noted in studies such as Richardson et al. (2017a, 2017b). On the other hand, data-based approaches require large datasets and are sensitive to noise and anomalies (Lin et al., 2023).
In both methodologies, abrupt variations in atmospheric conditions or unexpected occurrences can impact the accuracy of estimates, such as obscuration and reduction events. For instance, in Dissawa et al. (2017), the author initially identifies the solar position in the image and then tracks cloud movement towards the sun. However, this approach is prone to errors, leading to the proposal of more robust methods to address this deficiency (Lin et al., 2023).
5.2. Machine Learning-Based Forecasting
In the subsequent sections, we explore the potential and constraints of ML techniques in forecasting solar irradiation using ground cloud images.
5.2.1. Convolutional Neural Networks
In solar forecasting via the direct method, supervised ML techniques aim to establish a direct link between images and solar irradiance. The CNNs have emerged as powerful tools for this purpose due to their ability to recognize spatial patterns in images effectively, including cloud formations and light intensity (Lin et al., 2023).
CNNs excel in capturing complex spatial patterns and hierarchies in images, preserving and exploring both local and global relationships within the image data. However, in cases where subtle or less obvious image features are more relevant for prediction, algorithms focusing on one-dimensional input data may be more suitable. Manual feature extraction from images allows researchers to focus on specific image aspects that strongly correlate with solar irradiance.
The study by Paletta and Lasenby (2020) introduces an irradiance estimation model that combines a CNN, utilizing ResNet units to analyze sky images, with an ANN processing auxiliary data. The integration of both pieces of information by another ANN generates forecasts, showing that CNNs can effectively predict future irradiance based on past sky image sequences. However, the model has limitations in predicting sudden irradiance changes, suggesting the inclusion of historical data for improved analysis.
Similarly, Leelaruji and Teerakawanich (2020) and Tiwari et al. (2019) estimate cloud movement vector direction and velocity from images using the Farneback method. They propose a combined technique of image processing and a CNN based on ResNet to predict solar irradiance fluctuation from sky images and trigger alert systems 1 to 2 minutes in advance.
In CNN architecture for ground image-based strategies, challenges arise due to cloud images occupying large spaces, necessitating deeper convolution networks and expanded convolution cores for effective learning. To address this, dilated convolution can expand the convolution kernel size without increasing weight parameters. Additionally, replacing conventional convolution blocks with residual blocks helps mitigate setbacks from excessively deep CNNs.
5.2.2. Long-Short Term Memory
In studies by Eşlik et al. (2021); Eşlik, Akarslan, and Hocaoğlu (2022), the integration of conventional image processing techniques with LSTM models demonstrates a potent system for predicting irradiance from cloud image analysis. They employ the ShiTomasi method to identify characteristic points in celestial images and track them across sequences using the Lucas-Kanade algorithm. The LSTM model is then trained to predict real-time irradiance from these cloud image sequences, enabling short-term solar radiation estimation.
5.2.3. Regression Algorithms
Using linear regression for estimating solar irradiation from cloud images proves to be effective due to the continuous variability of solar irradiation. Regression models are adept at predicting values on a continuous scale, making them accurate instruments for evaluating irradiation levels. Moreover, they offer flexibility in integrating additional information, such as weather data, to enhance prediction accuracy.
In Al-lahham, Theeb, Elalem, Alshawi, and Alshebeili (2020), an algorithm is introduced for identifying distinct features in celestial images, employing ML strategies to calculate the IGH. Regression models, particularly Random Forest (RF) and K-Nearest Neighbors (KNN), are utilized to build these models.
In Chang et al. (2017), four methods are explored for solar irradiation estimation. These include Brightness Ratio Delta-Based Regression (delta RBR), conventional linear regression, Support Vector Regression (SVR) with linear and non-linear kernels. Both linear and non-linear models, based on features inferred from celestial images, exhibit significant accuracy improvements over the standard persistence model, offering more reliable insights into solar irradiation patterns.
5.3. Validation and Evaluation Metrics
Irradiance metrics in images serve as crucial parameters for evaluating the efficiency and accuracy of solar irradiation forecasting methods. In various studies such as Al-lahham et al. (2020); Bernecker et al. (2014); Bone et al. (2018); Cañadillas et al. (2018); Cervantes et al. (2016); Chang et al. (2017); Marquez & Coimbra (2013); Quesada-Ruiz et al. (2014); Richardson et al. (2017b); Tiwari et al. (2019), a range of metrics have been utilized for evaluating solar irradiance prediction models. These include MAE, RMSE, nMAPE, nRMSE, and MBE.
6. Discussion and Conclusion
In this article, we present a systematic review of the cloud tracking process using images obtained from the ground, aiming at nowcasting predictions. The organization of this article is structured into four fundamental steps: Observational Data, Cloud Recognition Methods, Cloud Tracking Methods and Solar Irradiation Prediction.
Cloud tracking articles that use terrestrial imagery rely on observational data collected by cameras with angled lenses such as TSI or fisheye lenses. Few studies have used cameras without angled lenses to collect data. Both approaches have advantages and disadvantages.
Cameras with angled lenses have the advantage of having a wide field of view, however, these lenses can introduce distortions that make data analysis more complex. Cameras without angled lenses have the advantage of less distortion and are more affordable. However, its field of view is more restricted, which results in more limited coverage.
One of the main problems and difficulties that writers encounter is cloud edge detection. The change in image brightness is highly unpredictable due to the changes that sky lighting can cause throughout the day and in different weather conditions. Image processing techniques are frequently used by authors, such as thresholding, color segmentation, filters and mathematical morphology. However, more recent research has employed machine learning techniques such as SVM and KNN for cloud detection and is showing promising results.
Detecting clouds can also be tricky when they move quickly, making real-time tracking equally challenging. The practice of cloud tracking based on terrestrial images has gained prominence, especially in real-time monitoring of energy systems powered by high-penetration solar generation.
Cloud tracking involves applying techniques such as BM, OF, FM and ML-based methods to calculate CMV. Techniques such as OF and DVF require significant computational resources, especially for high-resolution images. The technique most adopted by researchers to track clouds through terrestrial images is OF, which is based on the analysis of pixel displacement between consecutive images. Within this approach, the Lucas-Kanade algorithm is often used. Although this method offers several advantages, it has the limitation of being sensitive to changes in lighting and may be ineffective when clouds undergo rapid shape changes.
As highlighted previously, DL techniques have been used in more recent studies to identify and track clouds in images. These methods are capable of handling a wide range of cloud shapes and varieties and have the advantage of learning complex patterns directly from the data. However, they require large volumes of data and great computational power.
ML methods, such as CNN, LSTM and regression algorithms, are also increasingly used to predict solar irradiation and, consequently, energy generation in photovoltaic plants. Most articles employed hybrid models that combine different techniques and approaches to take advantage of the individual advantages of each method and overcome their limitations. In the context of cloud tracking and solar irradiance forecasting, hybrid models are especially useful for improving the accuracy and robustness of forecasts. The combination of OF and ML combines the accuracy of Optical Flow in estimating pixel-by-pixel motion with the learning ability of ML models to adjust and improve motion vectors.
The diversity of metrics and the lack of standardization make it difficult to directly compare results between different studies. Each method has advantages, limitations and disadvantages. However, given that existing studies greatly diversify the techniques used and rarely compare one model with another, it becomes a challenge to determine which approach is the most effective.