Generative Generative Adversarial Nets [7] (Generative Adversarial Nets, GAN) is the product of the combination of game theory and deep learning. It is an unsupervised probability distribution learning method that learns the distribution of real data, generating new datasets with high similarity. GAN consists of generator and discriminator. The generator learns the distribution of real sample data and generates the most realistic fake data. The discriminator is essentially a dual classifier that needs to identify whether the input data is a real sample or fake data generated by the generator. The main purpose of the generator is to try to generate fake data that is similar to the real data, and the main purpose of the discriminator is to try to distinguish the real data from the fake data. Therefore, the training process of Generative Adversarial Network is like a game game. In all possible function sums, find the equilibrium solution of both sides.

In order to learn the data distribution of the generator on the data x, first define a random noise variable z, which can be mapped to the corresponding data

space by the generator, which is defined as the function of the multi-layer perceptron representation in the generative adversarial model. Because it is the function represented by the second multilayer perceptron. Where D(x) represents the real data x. When training the model, we use Dl to take the maximum value of the probability value of the real sample and the generated sample, so it can be minimized to the training summation. In this case, the value function is played by the discriminator and the generator. The optimal solution has been reached, and an equilibrium solution for both sides is found.

min G max D V ( D , G ) = E x ~ p d a t a ( x ) [ log D ( x ) ] + E z ~ p z ( z ) [ log ( 1 − D ( G ( z ) ) ) ] (1)

The CGAN is the abbreviation of Conditional Generative Adversarial Nets [8], also known as Conditional Generative Adversarial Network. CGAN is an extended model of the original GAN, which adds a condition to the Generative Adversarial Network, making the Generative Adversarial Network a supervised learning. Both the generator and the discriminator add the corresponding information y as the training condition. In the generative model, the prior input noise p(z) and the conditional information y together form the joint hidden layer representation. In this way, in the Formula (2) mentioned above, a condition is added to allow the network to perform a minimax game under the condition y:

min G max D V ( D , G ) = E x ~ p d a t a ( x ) [ log D ( x / y ) ] + E z ~ p z ( z ) [ log ( 1 − D ( G ( x / y ) ) ) ] (2)

Attention mechanism is one of the hotspots of research in recent years. It helps deep neural networks suppress less significant pixels or channels. The Attention Mechanism (Normalization-based Attention Module, NAM) [9] proposed in 2021 is to use the contribution factor of the weight to improve the attention mechanism. Using a batch-normalized scale factor, utilizes a variance measure of the weights of the trained model to highlight salient features. This avoids adding the fully connected and convolutional layers used in (Bottlenet Attention Module, BAM) and (Convolutional Block Attention Module, CBAM).

NAM as an efficient and lightweight attention mechanism. The module integration from CBAM was adopted and the channel and spatial attention submodules were redesigned. A NAM module is embedded at the end of each network block. For residual networks, it is embedded at the end of the residual structure. For the channel attention submodule, a scale factor from batch normalization (BN) is used, as shown in publicity (3). The scale factor measures the variance of the channels and indicates their importance.

B o u t = B N ( B i n ) = γ B i n − μ Β σ Β 2 − ε + β (3)

Among them, μ Β and σ Β are the mean and standard deviation of the minimum batch B, respectively γ and β are trainable affine transformation parameters (scale and displacement). The channel attention sub-module is shown in Figure 2 and Equation (4), where M c represents the output feature. γ is the scale factor of each channel, and the weight is W γ = γ i / ∑ j = 0 γ j . A scale factor of BN is applied to the spatial dimension to measure the importance of pixels. Name it Pixel Normalization. The corresponding spatial attention sub-module is shown in Figure 3 and Equation (5), where the output is represented as Ms, λ is the scale factor, and the weight is W λ = λ i / ∑ j = 0 λ j .

To suppress less significant weights, we add a regularization term to the loss function, as shown in Equation (6). where x is the input, y is the output, and W is the network weight, l ( ⋅ ) represents the loss function, g ( ⋅ ) is the l 1 -norm penalty function, and p is the penalty function of the balance functions g ( γ ) and g ( λ ) .

M c = s i g m o i d ( W γ ( B N ( F 1 ) ) ) (4)

M s = s i g m o i d ( W λ ( B N s ( F 2 ) ) ) (5)

l o s s = ∑ ( x , y ) l ( f ( x , W ) , y ) + p ∑ g ( γ ) + p ∑ g ( λ ) (6)

Generative Adversarial Networks are not stable during training, we address this problem by introducing a perceptual loss into the network. A new refinement

loss function is proposed. Specifically, we combine pixel-to-pixel loss, perceptual loss, and adversarial loss with appropriate weights to form our new refinement loss function. The new loss function is then defined as follows:

L R P = λ e L E + λ a L A + λ p L P (7)

where L A represents the adversarial loss (loss of the discriminator D) function, L p is the perceptual loss function, L E is the normal per-pixel loss function. Here, λ e , λ a , and λ p are the predefined weights for perceptual loss and adversarial loss, respectively. If we set both λ a and λ p to 0, then the network reduces to a normal CNN configuration. If λ p is set to 0, the network will reduce to a normal GAN. If λ a is set to 0, the network reduces to the structure proposed in [10]. The three loss functions are defined as follows. Given an image { x , y b } with C channel. Given an image { x , y b } with C channel, width W and height H (i.e. C × W × H ), where is the input image x and y b is the corresponding ground truth, the per-pixel loss function is defined as:

L E = 1 C W H ∑ c = 1 C ∑ x = 1 W ∑ y = 1 H ‖ Φ E ( X ) c , w , h − ( y b ) c , w , h ‖ 2 2 (8)

where Φ E is the learning network G used to generate the derained image output. Suppose the output size of some advanced layers is C i × W i × H i . Likewise, the perceptual loss is also defined as:

L P = 1 C i W i H i ∑ c = 1 C i ∑ w = 1 W i ∑ h = 1 H i ‖ V ( Φ E ( X ) ) c , w , h − V ( y b ) c , w , h ‖ 2 2 (9)

Among them, V represents a nonlinear CNN transformation. Our goal is to minimize the distance between features. Given a set of N rain images.

Since the goal of single image rain removal is to generate pixel-level rain removal images, the generator should be able to remove rain streaks as much as possible without losing the detailed information of the background image. Therefore, the key to removing rain from a single image is to design a good generated by generator { Φ E ( X ) } i = 1 N , the entropy loss of the discriminator is defined as:

L A = − 1 N ∑ i = 1 N log ( D ( Φ E ( X ) ) ) (10)

In this section, we design a generative adversarial network model by incorporating the conditional generative network of the attention mechanism NAM for the problem of image rain removal. The entire network model architecture consists of two main parts, the first part generates images that are similar to the original images. The second part is to judge the most similar generated image by the contextual semantics of the original image, and iterate the first two steps through back-propagation. In this paper, the conditional generation network incorporating the attention mechanism NAM for image deraining model IDN-CGAN is shown in Figure 4, which clearly expresses the core idea of this chapter. The image features obtained by IDN-CGAN have good generality and generalization ability.

Since the goal of single image rain removal is to generate pixel-level rain removal images, the generator should be able to remove rain streaks as much as possible without losing the detailed information of the background image. Therefore, the key to removing rain from a single image is to design a good structure to generate a rain-removing image. Existing solutions, such as sparse coding- based methods [11] [12] [13], and neural network-based methods [14], all adopt a symmetric (encoding-decoding) structure. In these methods, a symmetric structure is used to form the generator network. The generator G directly learns the end-to-end mapping from the input rain image to the corresponding ground truth. In contrast to existing adversarial networks that use U-Net [15] [16] or ResNet [17] for occlusion-image-to-image translation in the generator, we use the recently introduced densely connected block [18]. These dense blocks allow gradients to flow, increasing parameter efficiency. Each layer in the dense block consists of three consecutive operations, batch normalization (BN), leaky rectified linear unit with leaky linear rectification function (Leaky ReLU), and 3 × 3 convolution. Each dense block is followed by a transition block (T) that functions as an upsampling (Tu), downsampling (Td), or no-sampling operation (Tn).

Furthermore, we embed a NAM module at the end of the generator network to effectively utilize different levels of features and guarantee better convergence.

This paper proposes a new multi-scale discriminator. This is inspired by the use of multi-scale features in objection detection [19] and semantic segmentation [20]. Similar to the structure proposed in [15], a convolutional layer with batch normalization and PReLU activation is used as the basis throughout the

discriminator network. Then, multi-scale pooling modules with different scale features are stacked at the end of the discriminator. The merged features are then upsampled and concatenated, followed by a 1 × 1 convolution and a sigmoid function to produce a normalized probability score between [0, 1]. By using features at different scales, we explicitly incorporate global hierarchical context into the discriminator. A scaling operation weights the normalized weights to the features of each channel, and prevents the adversarial network from collapsing and oscillating during training, improving stability.

The Due to the lack of large datasets for single-image rainfall training and evaluation, in order to verify the effectiveness of the proposed algorithm, we use not only real datasets but also synthetic datasets in our experiments. The training set consists of 800 images in total, divided into two types: severe rain images and light rain images, to ensure rain pixels with different intensities and directions, and generate different training and test sets. To demonstrate the effectiveness of the method on real-world data, we download a dataset of 50 rain images from the Internet. When creating this dataset, we took every care possible to ensure that the collected images differ in content, intensity and orientation of rain pixels. A more comprehensive effect has been achieved.

We employ Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) [21], and Visual Information Fidelity (VIF) [22] to evaluate the performance of different methods. All of these quantitative measurements are calculated using the luminance channel. These three criteria are often used to verify the performance of the network model and the robustness of the model. The peak signal to noise ratio (PSNR) of an image is given by the following formula:

PSNR = 10 log 10 ( MAX 2 MSE ) (11)

SSIM is used to judge whether the image is distorted, the objective standard of image noise level, and the maximum unit is expressed in decibels. The structural similarity index [21] (Structural similarity index, SSIM) is shown in the following formula:

SSIM = ( 2 μ x μ y + C 1 ) ( 2 σ x σ y + C 2 ) ( μ x 2 + μ y 2 + C 1 ) ( σ x 2 + σ y 2 + C 2 ) (12)

where x, y are the uncompressed undistorted image and the image contrasted by x, μ x μ y is a function of brightness, σ x σ y is a function of contrast, the constant C₁ is to ensure the stability of the system at μ x 2 + μ y 2 ⇒ 0 , and C₂ is also a constant C₃ = C₂/2. In the evaluation index, the larger the value of PSNR, VIF and SSIM, the better the image restoration and the smaller the image distortion.

The experimental parameter settings in this paper are done on the Tensorflow learning platform. The experiment was performed on a desktop computer, Intel-i5 3.3G CPU, 8GB RAM, Windows 10, TensorFlow 1.1, Opencv2.0, the learning rate was set to 0.5, the number of iterations was 50, and the batch size was 2. This paper selectively samples heavy rain images to show that our method performs well under difficult conditions. The performance of the proposed method and recent state-of-the-art methods is evaluated on real-world rainfall test images. The deraining results of all methods on the derained images input by two samples are shown in Figure 5. By comparing the proposed method with other methods, by observation, we can clearly observe that GMM [23] tends to add artifacts on derained images. Although CCR [24] is able to remove rain streaks, it produces blurry results, which are not visually appealing. JORDER [25] can reduce the intensity of rain or remove streaks in some areas, however, they cannot completely remove rain streaks. Raindrops are still visible in the magnified region of interest despite other methods, and good visual performance can be achieved. Compared with other methods, this method is able to successfully remove most of the rain streaks while maintaining the details of the derained images.

The three evaluation parameters PSNR, SSIM, UQI and IDN-CGAN are all learned by using training images of synthetic training datasets. Using the test images of the synthetic datasets discussed earlier, as shown in Table 1, it can be

seen that the introduction of adversarial loss improves the visual quality compared to the traditional CNN architecture. For comparison: the first indicator is the peak signal-to-noise ratio (PSNR). The PSNR of the model IDN-CGAN proposed in this paper is significantly higher than the other three methods in the heavily derained image and the lightly derained image. The second indicator is The estimated overall structural similarity index (SSIM), the SSIM values of GMM [23] and CCR [24] are both lower than JORDER [25], while the SSIM value of IDN-CGAN is slightly higher than that of JORDER [25]; Authenticity (VIF), although IDN-CGAN performs better than the other three models, the difference is not large, the reason may be that the proposed method misses some rain patterns in the output image. (The higher the PSNR and VIF evaluation indicators and the higher the SSIM indicator, the clearer the model repair results and the better the effect.)