In order to prove the superiority of the proposed fusion method, three sets of images are selected for multi-focus image fusion, as shown in Figures 1(a)-(c). The images on the top row are mainly focused on the foreground while the images on the bottom row are mainly focused on the background. To better evaluate the performance of the fusion method, the proposed method is compared with several current mainstream multi-focus image fusion methods based on DWT [13] , NSCT [14] , OPT [15] and LP [16] . All experiments are carried out in MATLAB2016a.

In the quality evaluation of no reference image [17] , the saliency of image is an important index to evaluate the quality of image. It can be better suited to human subjective feelings. If the image is not high in significance, the image is blurred. In this paper, the Laplacian gradient [10] is used.

The Laplacian operator is an important algorithm in the image processing, which is a marginal point detection operator that is independent of the edge direction. The Laplacian operator is a kind of second order differential operator. A continuous two-element function f (x, y), whose Laplacian operation is defined as

∇ 2 f = ∂ 2 f / ∂ x 2 + ∂ 2 f / ∂ y 2 (1)

For digital images, the Laplacian operation can be simplified as

g ( i , j ) = 4 f ( i , j ) − f ( i + 1 , j ) − f ( i − 1 , j ) − f ( i , j + 1 ) − f ( i , j − 1 ) (2)

At the same time the above formula can be expressed as a convolution form, that is

g ( i , j ) = ∑ r = − k k ∑ s = − l l f ( i − r , j − s ) H ( r , s ) (3)

In the above formula, i , j = 0 , 1 , 2 , ⋯ , N − 1 ; k = 1, l = 1, H(r, s) can take a lot of values, one of which is

H 1 = [ 0 1 0 1 − 4 1 0 1 0 ]

Experiments show that the higher the image saliency is, the greater the sum of the mean of the corresponding matrix is after being processed by the Laplacian operator. Therefore, the image saliency (D(f)) based on the Laplacian gradient function is defined as follows:

D ( f ) = ∑ y ∑ x | g ( x , y ) |   ( g ( x , y ) > T ) (4)

Among them, g (x, y) is the convolution of Laplacian operators at pixel points (x, y).

By using the value of D(f), it is easy to divide images with different clarity. Next, it is applied to the saliency decision of different regions of images. According to the above, the region saliency of an image can be defined as:

D I ( i , j ) = D ( I ( i − n : i + n , j − n : j + n ) ) (5)

Among them, D is the function of saliency method based on Laplacian gradient operator. D_I is the matrix of saliency of image I. And ( 2 n + 1 ) × ( 2 n + 1 ) is the scale of processing template.

In the multi-focus image processing, we can get significant matrices (D_I1, D_I2) of different focus images, obtain a decision matrix (M_decision) by comparing.

M decision = ( D I 1 ≥ D I 2 ) (6)

For various reasons, there are some noise and erroneous judgment in the decision map. It will affect the quality of image fusion. As for erroneous judgment, it will be mentioned later in the article.

In the first obtained decision map, there are often some noise and misjudged areas need to be corrected. In most methods, morphological processing is usually used to solve this problem. But this method often leads to the destruction of the boundary. H. Hariharan et al. [18] defined the focusing connectivity of the same focal plane. Most of the noise and misjudged areas can be corrected, according to it.

M DF-decision = Delete Larea ( M decision ) (7)

As for Delete_Larea, it needs to be mentioned that its function is to delete smaller connected areas which include most noise and misjudged areas.

At this stage, there is an important problem to be solved. The erroneous judgment adhered to the focus edge is not removed by the above method. When using the Laplacian method to deal with the edges of multi-focus images, there is often edge information interference, in the case of Figure 2. Because the black part is more than the white part in their corresponding templates, point A and the points around it will turn black. We can understand it from Formula 2. f uses 3 × 3 this module for processing. And it will also be false for other reasons. Therefore, we put forward a focus edge optimization method based on edge detection. Edge detection is used to find the edges of the original images. Using a module scans the edges to modify the area in the module. The g is an edge detection function. As shown in Formula 8, the h is the function that if it is found that one side of the edge is dominated by an element, all this side is modified to the element. Among them, A is a decision map, B is an edge map, and C is an optimized decision map.

[ 1 1 1 0 0 1 1 0 0 0 1 1 1 0 0 1 1 0 0 0 1 0 0 0 0 ] → f A = [ 1 1 1 0 0 1 1 1 0 0 1 1 0 0 0 1 1 0 0 0 1 0 0 0 0 ] → g B = [ 1 1 1 0 0 0 1 0 0 0 0 0   1   0 0 0 1 0 0 0 1 0 0 0 0 ] } → h C = [ 1 1 1 0 0 1 1 0 0 0 1 1 1 0 0 1 1 0 0 0 1 0 0 0 0 ] (8)

Image fusion is carried out according to the final decision map (D_final). Then, the fused image f (x, y) could be expressed as:

f ( x , y ) = D final × f 1 ( x , y ) + ( 1 − D final ) × f 2 ( x , y ) (9)

It means the fused image is composed by the focus regions in the image f₁ (x, y) and f₂ (x, y). Though these steps, a fused image fully focused could be obtained.

For more than two images to be fused, it is necessary to change the form of the decision map. It will storage the serial number of the most significant image in the corresponding region. Figure 3 is a schematic map of a decision map in the process of fusion of four multi-focus images. When image fusion is made, each point of the image is assigned according to the index value. The fused image f (x, y) could be expressed as:

f ( x , y ) = f D final ( x , y ) ( x , y ) (10)

The performance of the fusion algorithm can be evaluated subjectively and objectively. Since the evaluation is highly dependent on human visual characteristics, it is difficult to distinguish between the fused images when they are approximately similar. Therefore, one subjective evaluation method and four objective evaluation methods are adopted in this article.

1) Subjective evaluation method

a) Comparison of residual maps

The residual map can display the difference between two images in the image. We can observe the effect of image fusion by observing residual maps of different methods. The residual map I_r between the source image and the fused image is defined as follows:

I r = I origin − I fusion + max ( I origin ) / 2 (11)

2) Objective evaluation methods

a) Mutual information (MI)

The greater the sum of the mutual information between the fusion image and the source image, the richer the information obtained from the source image of the fused image, and the better the fusion effect. The MI between the source image and the fused image is defined as follows:

M I = ∑ k = 0 L ∑ i = 0 L p A F ( i , k ) log 2 p A F ( i , k ) p A ( i ) p F ( k ) + ∑ k = 0 L ∑ j = 0 L p B F ( j , k ) log 2 p B F ( j , k ) p A ( j ) p F ( k ) (12)

Among them, p_A, p_B and p_F are the normalized gray histogram of A, B and F. p_AF (i, k) and p_BF (j, k) are united gray histograms between the fused image and the source image. L is the number of intensity levels.

b) Peak signal to noise ratio (PSNR)

PSNR is the most common and widely used objective measure of image quality. The larger the PSNR, the less the distortion is represented. The PSNR is calculated as follows:

P S N R = 10 ⋅ log 10 ( M A X I 2 / ( 1 m n ∑ i = 1 m − 1 ∑ j = 1 n − 1 ‖ I ( i , j ) − K ( i , j ) ‖ 2 ) ) (13)

where A represents one of the pre-processed images, F represents the processed image, and MAX_I is the maximum value that represents the color of the image point.

c) Spatial frequencies (SF)

SF reflects the change of the pixel gray level of the image in space. To some extent, SF can reflect the clarity of images. SF is defined as follows:

S F = 1 M × N ∑ i = 1 M ∑ j = 2 N [ I ( i , j ) − I ( i , j − 1 ) ] 2 + 1 M × N ∑ i = 2 M ∑ j = 1 N [ I ( i , j ) − I ( i − 1 , j ) ] 2 (14)

where I (I, j) represents the image, and M and N represent the number of rows and columns of the image.

d) Edge intensity (EI)

EI is a measure of the local change intensity of the image in the normal direction along the edge, and also reflects the image sharpness to some extent. Its formula is expressed as:

E I = 1 M × N ∑ i = 1 M ∑ j = 1 N I x 2 ( i , j ) + I y 2 ( i , j ) (15)

where I_x (i, j) and I_y (i, j) represent horizontal gradient and longitudinal gradient of the image.

In order to test the accuracy of the above method, the experiment uses MATLAB language programming to achieve the above algorithm. Experimental pictures use Lena images. The image size is 512 × 512 pixels. Then, the four focus images are generated by blurring each with a Gaussian radius of 2.5, 5, 7.5, and 10, respectively. Five images of Lena, Lena 2.5, Lena 5, Lena 7.5, and Lena 10 are shown in Figures 4(a)-(e).

The five images were tested using the image saliency assessment method based on the Laplacian gradient. Get the corresponding D(f). The data is shown

that this method is very sensitive to fuzziness. Contrast experiments are performed using a group of multi-focus images in Figure 1(a). The results are shown in Figure 5(a).

From Figures 5(a)-(d), one can clearly see that the performance of these fusion methods showed difference when fused with the same multi-focus image. From a detailed observation, the fused image obtained by Tenengrad and SMD is not clear and there are a large number of residuals in Figure 5(c) and Figure 5(d). Besides, the edge of the object is fuzzy from the decision map of Tenengrad, SMD and energy gradient. At the same time, compared with the actual situation, it can be clearly seen that the decision map obtained by these fusion methods appeared to have more obvious false information. However, it can be easily observed that the fused image acquired by the image saliency assessment method based on the Laplacian gradient is more ideal in the subjective effect because the residual information is also less than other methods which means that the method transfer almost all focus information to the fused image. On the other hand, good preprocessing is very convenient for the later operation, especially in edge optimization.

In turn, the focusing connectivity of the same focal plane and a focus edge optimization method based on edge detection are used to deal with the initial decision map. We can see that obvious interference have been removed in decision map in Figure 6(e). It is not difficult to find that the edge is smoother in the decision map after edge optimization in Figure 6(f). And it shows more clearly in Figures 7(a)-(d).

The whole process can be summarized below. First, we choose a set of multi-focus images (Figure 6(a) and Figure 6(b)) for processing to obtain the corresponding saliency maps (Figure 6(c)). And then we can get an initial decision map (Figure 6(d)) through them. Next, the focusing connectivity of the same focal plane is used to remove most of the noise and misjudge areas. The edge correction method is used to optimize the decision map (Figure 6(e)). Finally,

the multi-focus images are fused according to the final decision map. In the final decision map (Figure 6(f)), it is clear from Figure 7(d) that the edge is more smooth and consistent with the actual situation. Finally, we got a globally clear image (Figure 6(g)).

The fused image and corresponding residual map of Backgammon Clock, and Lab using different method are shown in Figures 8(a)-(j), Figures 9(a)-(j) and Figures 10(a)-(j). From Figures 8(a)-(e), we see that the five algorithms can produce fused images separately, but it is very difficult to distinguish the differences between some fusion results only by visual observation. To better evaluate the visual quality of the fused image, it is a good method to compare their residual map shown in Figures 8(f)-(j). Comparing the residual maps of these methods, the results are obvious. The residual maps obtained by DWT, NSCT, OPT and LP has more residual information, but the proposed method has less.

What would be resulted from Figure 8(c) and Figure 9(c) is that the images are not enough clear using OPT. Many fusion errors exit at the right edge of a Gobang box in Figure 8(a) and Figure 8(b) and the surface of the back clock in Figure 9(a), Figure 9(b) and Figure 9(d). From the residual maps, there are more residual information on the surface of clock in Figure 10(f), Figure 10(h) and Figure 10(i). Compared with other methods, the method proposed in this paper has better subjective performance.

In the last part, the residual maps are used to compare the different image fusion methods. In order to further verify the performance of the proposed method, the objective quality evaluation is carried out. Objective evaluation indicators have been introduced above, including MI, PSNR, SF and EI. The evaluation results are shown in Table 1.

From the data in Table 1, the evaluation results are obvious. When the origin image is “Backgammon”, most values of four indexes of the proposed method are observably higher than those of the other methods. It has the same situation for “Clock” and “Lab”. By the meaning of several evaluation methods, it can be shown that the fusion image obtained by the proposed method contains more information, also shows that the fusion image has higher definition.