Multi-scale decomposition-based fusion methods use a multi-scale decomposition method to decompose multisource images into different scales and different resolutions, and consequently, to obtains low-frequency sub-bands containing image energy information and high-frequency sub-bands containing detailed information of different scales of images; in the next step, the fusion is performed according to different fusion rules on the low-frequency sub-band and

high-frequency sub-bands respectively; finally, the multi-scale reconstruction is performed through the fused sub-band image to obtain the final fused image. A schematic diagram of the images fusion scheme based on general multi-scale decomposition is illustrated in Figure 3. Another key factor that affects the fusion results is the fusion strategy. This is a process that determines the formation of the fused image, based on coefficients or pixels of the source images.

Classical multi-scale decomposition methods include pyramid transform and wavelet transform. Common pyramid transformations include the Laplacian pyramid (LP) [22], contrast pyramid (CP) [31] [32] [33], steerable pyramid [34], and gradient pyramid (GP) [35]. Moreover, wavelet transforms include: discrete wavelet transform (DWT) [36] [37] [38], discrete wavelet packet [39], binary wavelet, lifting wavelet [40], dual tree complex wavelet (DTCWT) [41], multi-wavelet [42], à trous wavelet [43] [44], discrete stationary wavelet transform (DSWT) [45] etc. The contrast pyramid transform is derived from the Laplacian pyramid transform, which involves calculating the ratio of two adjacent low-pass filtering images of the Gaussian pyramid; therefore, the model becomes able to consider local contrast [31]. The work of [32] proposed an infrared and visible

image fusion method that was based on the contrast pyramid and multi-objective evolutionary algorithm, in which the multi-objective evolutionary algorithm was utilized to optimize fusion coefficients.

The image fusion method based on pyramid transform is simple to implement; however, when the gray value difference between images is large, square traces will appear in the fusion result due to the correlation of features between adjacent scales after the image is transformed by the pyramid. The wavelet transform has a good ability to analysis local time domain and frequency domain for the signal, while the optimal representation of the signal point singularity can be obtained for the one-dimensional segmented smooth signal. For the two-dimensional image signal, however, the commonly used two-dimensional separable wavelet is a “non-sparse” image representation: this is a tensor product of one-dimensional wavelets, with only a finite direction, that is unable to optimally represent a two-dimensional image containing line singularities. In short, a common limitation of the methods in the wavelet family is that they find it difficult to accurately represent the curves and edges of the images.

In order to solve these problems and better represent the images, many multi-scale decomposition methods have been proposed and applied to the image fusion problem. Representative examples include Bandlet transform, which is based on image edge adaptive construction, Ridgelet transform that has a good approximation effect on linear singularity, curvelet transforms (CVT) [46] [47], which can produce a good approximation of curve singularity, contourlet transforms [48] [49], nonsubsampled contourlet transform (NSCT) [50], shearlets which allow for the efficient encoding of anisotropic features in multivariate problem classes and place no restrictions on the 110 directions for the shearing or the size of the supports [51], and so on. Through the use of the parabolic scaling law, the curvelet and contourlet can resolve the dichotomous points along a smooth curve. Furthermore, the multi-scale decomposition methods described above have also been successfully applied in multi-sensor image fusion tasks [46] [49] [50]. However, these multi-scale decomposition methods cannot sparsely represent any anisotropic structure; therefore, the ripplet transform is proposed [52]. In [53], moreover, a remote sensing image fusion method based on ripplet transform and compressed sensing theory was proposed in order to minimize the spectral distortion of the pansharpened MS bands relative to the original ones. The primary advantage of these multi-scale decomposition methods is their good ability to preserve the details of different source images. However, since spatial consistency is not adequately considered during the fusion process, the fusion of these methods may result in distortion of the brightness and color values [54].

Recently, edge-preserving filters [55] have been actively researched within the field of image processing, and have also been successfully applied in multi-sensor image fusion. In [55], Farbman et al. introduce a novel method for constructing edge-preserving multi-scale image decomposition, in which the edge-preserving multi-scale image decomposition is accomplished using an alternative edge-preserving smoothing operator based on a weighted least squares optimization framework, which is well-suited to progressive coarsening and the multi-scale detail extraction of images. Li et al. introduced a novel two-scale fusion method that does not need to rely heavily on a specific image decomposition method; moreover a new guided filtering-based weighted average technique is utilized to make full use of spatial consistency when fusing the base layers and detail layers [54]. Hu and Li introduced a non-downsampling direction filter bank into a multi-scale bilateral filter, further proposing a new multi-scale geometric analysis method-the multi-scale bidirectional filter (MDBF)—which can better represent the intrinsic geometrical structure of images and achieves better performance than the traditional fusion methods [56]. In addition, the combination of Gaussian and bilateral filters is also successfully applied in the fusion of infrared and visible images [57]. Furthermore, some computer vision tools—such as support value transform [58], log-Gabor transform [59], and anisotropic heat diffusion [60] —have also been applied for multi-scale decomposition-based fusion.

As the authors of [20] point out, the spatial quality of the fused images may be less satisfactory if fewer decomposition levels are applied, while both the performance and computational efficiency of the method may also be reduced if too many decomposition levels are applied. Thus, some researchers have attempted to determine the optimal number of decomposition levels that will yield optimal fusion quality. For example, the authors in [61] compared various multi-resolution decomposition algorithms, focusing on the influence of the decomposition level and filters on fusion performance. They subsequently concluded that a short filter usually provides better fusion results than a long filter, while the most appropriate number of decomposition levels is four. The work of [62] estimated the optimal number of decomposition levels for multi-spectral and panchromatic image fusion with a specific resolution ratio.

The research hotspots in the field of sparse representation include the approximate representation of models, uniqueness and stability of model solutions, performance analysis of sparse representations, model-solving algorithms, dictionary learning algorithms, sparse decomposition algorithms, super-complete atomic dictionaries, specific applications of sparse representations, and close contact compression sensing, among other aspects. Among them, specific applications include: image processing (such as compression, enhancement and super-resolution), audio processing (such as blind source separation) and pattern recognition (such as face and gesture recognition). From a practical perspective, targeted flexible models, computational speed, adaptive and high-performance representation results are key ways in which sparse representation methods can achieve advantage in the application domain.

The purpose of the signal sparse representation is to represent the signal with as few atoms as possible in a given over-complete dictionary: this allows a more concise representation of the signal to be obtained, making it easier and more convenient for us to obtain the information contained in the signal.

Utilizing the characteristics of sparse coefficients, Yang and Li were the first to apply sparse representation theory to image fusion. Firstly, the multi-source input images are divided into many overlapping patches in order to capture local salient features and maintain shift invariance. In the next step, in order to obtain the corresponding sparse coefficients, the overlapping patches from multi-source images are decomposed in the overcomplete dictionary. Subsequently, the sparse coefficients from multiple source images are applied to the fusion process. Finally, the image is reconstructed using the fusion coefficients and dictionary [63]. In [64], a novel pan-sharpening method (HR panchromatic and corresponding LR spectral channel fusion), named Sparse Fusion of Image (SparseFI), is proposed. Based on the theory of compressed sensing, it utilizes the sparse representation of the HR/LR multispectral image block in the panchromatic image, along with its down-sampled LR version. Yu et al. used the first model (JSM-1) from the joint sparse representation method proposed in [65] to achieve image fusion [66]. In [67], Liu et al. proposed an improved, compressed sensing-based image fusion scheme. In this method, the low and high sparse coefficients of the source images obtained by discrete wavelet transform are fused by means of an improved entropy-weighted fusion rule and max-abs-based fusion rule respectively. After using the local linear projection of random Gaussian matrix, the fusion image is reconstructed using a compression sampling matching tracking algorithm. Compared with the traditional transform-based image fusion methods, the proposed approach can retain more details, such as edges, lines and contours. Ma et al. proposed an image fusion algorithm based on sparse representation and guided filtering. Firstly, a sparse representation (SR)-based method was utilized to construct the initial fused image from the source images. Then, the spatial domain (SF)-based fused image was obtained in order to make full use of the spatial information of the source image. Finally, the final fused image was obtained by using guided filtering to reconstruct the SR-based fusion image and the SF-based fusion image [68]. In [69], a noisy remote sensing image fusion method based on joint sparse representation (JSR) was proposed to fuse SAR images with other source images. Firstly, the redundant complementary sub-images were obtained by the JSR method, and then the complementary sparse coefficients were fused together by using an improved fusion rule based on pulse coupled neural network (PCNN). At the same time, due to the different types of noises between the SAR image and other source images, these noises where treated as complementary information in the source images and suppressed in this step. Finally, the fused image is reconstructed by adding the fused complementary sub-images and the redundant information.

In addition to those based on multi-scale decomposition and sparse representation, there are also many fusion methods based on other theoretical knowledge.

For instance, [70] proposed a novel fusion framework based on the total spectrum change (TV) method and image enhancement. For the multi-scale decomposition of the spatial variation generated by the spectral variational framework, these authors verified that the decomposition components could be effectively modeled by the tail Rayleigh distribution (TRD) rather than the commonly used Gaussian distribution. Therefore, TRD-based saliency and matching measures were proposed to fuse each sub-band decomposition, while spatial intensity information is also employed to fuse the remaining image decomposition components. Zhang et al. reconstruct the infrared background using quad-tree decomposition and Bessel interpolation. Secondly, by subtracting the reconstructed background from the infrared image, the infrared brightness feature is extracted, and then refined by reducing the redundant background information. In order to resolve the overexposure problem, the fine infrared features are first adaptively suppressed and then added to the visual image, enabling the final fused image to be obtained [71]. Using fuzzy logic and population-based optimization, Kumar and Dass devised a new fusion method. In a departure from the weighted average of pixels method, the author proposed a method based on total variation to fuse the multiple sensor images. Under the proposed approach, the imaging process was modeled as a local affine model, while the fusion problem was framed as an inverse problem [72]. With the aim of further improving the fuse results, Shen et al. used maximum a posteriori (MAP) estimation in the hierarchical multivariate Gaussian conditional random field model to derive the optimal fusion weights [73]. In [74], the source images were first decomposed into a principal component matrix and a sparse matrix by means of robust principal component analysis; after that, the weights were estimated by taking the local sparse feature as the input of the pulse coupled neural network. [75] further proposed using a linear regression model to generate synthetic components, which were only partially replaced based on the correlation between the intensity component and the panchromatic image. In [76], firstly, the salient structure of the input image was fused in the gradient domain. In the next step, the fusion image was reconstructed by solving the Poisson equation, which ensures that the gradient of the fusion image approaches the fusion gradient. In [77], the intuitionistic fuzzy set theory was applied to image fusion, which involved transferring input image to the fuzzy domain, after which the maximum and minimum fusion operations were used to achieve fusion processing. In [78], Liu et al. proposed a novel focus evaluation operator based on max-min filter. In the proposed focus metric, the max-min filter was combined with the average filter and the median filter (MMAM) to evaluate the focus degree of the source images. After that, based on the structure-driven fusion region and the depth information of the blurred images, MMAM is utilized to achieve high-quality multi-focus fused image.

The most commonly used multi-scale decomposition methods are pyramid transform, wavelet transform, curved wave transform, non-downsampling contour wave transform, contourlet transform, Retinex transform, and morphological operators-based methods. Different methods have their own advantages and disadvantages in different application areas. Various fusion algorithms combining different transforms were subsequently proposed with the aim of fully utilizing these advantages, reducing defects, and ensuring that the methods employed complemented each other.

For example, in [79], a new image fusion methodology based on pulse couple neural networks and shearlets was proposed; this approach can not only extract more important visual information from the source images, but can also effectively prevent the introduction of manual information. Based on morphological component analysis and sparse representation, Jiang and Wang devised a novel fusion method in which the morphological component was utilized to decompose the source images, and then fused with the sparse representation-based method [80]. Moreover, with the goal of simultaneously preserving the appearance information of the visible image and the thermal radiation of the infrared image, Ma et al. put forward a novel fusion algorithm of visible and infrared images—referred to as Gradient Transfer Fusion (GTF)—based on gradient transfer and total variation (TV) minimization [81]. In [82], a Gaussian filter is utilized to decompose multi-source images, with a new combination of total variation rules employed to combine the base layers and a novel weight map construction method being proposed based on the saliency analysis. Furthermore, by using Nonsubsampled Contourlet Transform (NSCT) and sparse K-SVD dictionary learning to obtain the prominent features of the source images, Cai et al. proposed a novel fusion method which named as NSCT_SK_SVD [83]. In order to promote the performance of infrared and visual image fusion and provide better visual effects, Jin et al. proposed a novel image fusion method integrating DSWT, discrete cosine transform (DCT) and local spatial frequency (LSF). In this proposed methodology, DSWT was utilized to decompose the significant features of the source images into a series of sub-images, each with different levels and spatial frequencies; moreover, DCT was applied to separate the important details of the sub-images on the basis of their different energy frequencies while LSF was employed to enhance the regional features of DCT coefficients that could be useful for image feature extraction [45]. The work of [84] further proposed a new method for fusing visible and infrared images, referred to as DTCWT-ACCD, based on DTCWT and an adaptive combined clustered dictionary. Yang et al. put forward a novel remote sensing image fusion method based on adaptively weighted joint detail injection. In the proposed method, firstly, the spatial details were extracted from MS and PAN images through à trous wavelet transform and multi-scale guided filter; after that, the extracted details were sparsely represented to generate the main joint details by dictionary learning from the sub-images themselves; subsequently, in order to obtain the refined joint details information, an adaptive weight factor was designed considering the correlation and difference between the previous joint details and PAN image details; finally, the fused image was obtained by injecting refine joint details into the MS image using modulation coefficients [85]. In [86], the PAN and MS image pansharpening algorithm was proposed based on adaptive neural network and sparse representation in the nonsubsampled shearing domain. Aiming at obtaining better fusion performance in multi-focus image fusion based on transform domain, Liu et al. proposed a novel multi-focus image algorithm, which combined with the adaptive dual-channel spiking cortical model (SCM) in NSST domain and the differential image. Due to the global coupling of dual SCMs, the synchronization characteristics of pulses, and the multi-resolution and direction of NSST, the proposed algorithm can well retain the information of the source image and fuse a clear image that is more in line with human visual effects [87].

A new research direction and innovative idea for image fusion is Deep Learning-based methods which have achieved good results in image fusion.

To solve the problems of limited detail preservation and high sensitivity to false registration in most existing SR-based fusion methods, Liu et al. proposed a novel image fusion framework based on convolutional sparse representation (CSR), which was a new signal decomposition model. In this framework, each source image was decomposed into the base layer and the detail layer, so it could effectively overcome the above two problems, and then achieve high-quality multi-focus image fusion and multi peak image fusion [88].

In [89], Huang et al. proposed an improved sparse denoising auto-encoder (MSDA) algorithm, which was utilized to simulate the complex relationship between high resolution (HR) and low resolution (LR) PAN image blocks as a nonlinear mapping. By connecting a series of MSDAs, a stacked MSDA (S-MSDA) which could effectively pre-train DNNs was obtained. In addition, the entire DNN was trained again through the back propagation algorithm after pre-training to better train the DNN. Finally, assuming that the relationship between HR/LR MS image blocks was the same as the relationship between HR/LR PAN image blocks, the HR MS image would be reconstructed from the observed LR MS image using the trained DNN. In [90], a pan-sharpening method based on SRCNN model and Gran-Schmidt (GS) transform was proposed. In the proposed algorithm, the SRCNN model was used to enhance the spatial resolution of the MS image, and the average band of the enhanced MS image was calculated as a reference for histogram matching to modify the PAN image. Finally, the enhanced MS image and the modified PAN image were merged by GS transform to generate a high-resolution MS image. Another work that applies CNNs to pan-sharpening was introduced by Masi et al. [91], in which the SRCNN framework was also used to model the pan-sharpening process as an end-to-end mapping. In this algorithm, the input of the network was the superposition result of the up-sampled low-resolution MS image and PAN image, and the output of the network was the target high-resolution MS image.

In [92], Palsson et al. applied 3-D convolutional neural network to MS and hyperspectral (HS) image fusion, and obtained high-resolution HS image. The work of [93] put forward a superpixel-based multi-local convolutional neural network (SML-CNN) for panchromatic and MS image classification. To reduce the amount of the input data for CNN, a simple linear iterative clustering method was extended to segment MS images and generate superpixels, and replace pixels with superpixels as the basic analysis unit. In order to make full use of the spatial spectral information and environmental information of superpixels, a multi-local area joint representation method based on superpixels was proposed. Then, an SML-CNN model was established to extract effective joint feature representations, and a softmax layer was used to divide these features learned by multiple local CNNs into different categories. Finally, a multi-information modification strategy that combined the detailed information and semantic information was employed to eliminate the adverse effects on the classification results within and between superpixels, thereby improving the classification performance. Another work that applies CNNs to remote sensing image fusion was introduced by Liu et al. [94], in which an end-to-end learning framework based on deep multi-instance learning (DMIL) was utilized to classify MS and PAN images by using feature-based joint spectral and spatial information. The framework consists of two instances: one for capturing pan spatial information and one for describing MS spectral information, two examples of the characteristics obtained directly in line, could be regarded as simple fusion characteristics. In order to fully integrate spatial spectral information for further classification, the simple fusion feature is entered into a three-layer fully connected fusion network to learn the high-level fusion feature. And in [95], Ma et al. proposed a novel generation countermeasure network called FusionGAN to fuse infrared and visible images. In this network, the generator was utilized to generate the fused image with the main infrared intensity and visible gradient information. The discriminator aimed to prevent the fused image from containing more details in the visible image, so that the fusion image retains both the radiation information in the infrared image and the texture information in the visible image.

As the authors of [96] point out, a residual network (ResNet) was utilized to make full use of the high nonlinearity of deep learning model to realize image fusion task, and the proposed algorithm could achieve the highest spatial spectral unified accuracy. And in order to obtain clearer and richer texture feature panoramic images, Liu et al. proposed a novel multi focus image fusion algorithm, which combines NSST and ResNet. In the proposed algorithm, NSST was utilized to fully consider the high-frequency details in the image and low-frequency global features. For the high-frequency details, improved gradient sum of Laplace energy was employed to handle high frequency sub-band coefficients of different levels and directions by using different directional gradients. For the low-frequency details, ResNet with a deep network structure is used to obtain the spatial information characteristics of the low-frequency coefficient image [97]. For most fusion methods, the detection of the focus area is a key step. In view of this, Liu et al. proposed a multi-focus image fusion algorithm based on dual convolutional neural network (DualCNN). Firstly, the source images were input into the dual CNN to recover the details and structure from their super-resolution images and improve the contrast of the source images. Secondly, bilateral filtering was applied to reduce the noise of the fused image, and the guided filter was used to detect the focus area of the image and refine the decision map. Finally, the fusion image was obtained by weighting the source image according to the decision graph. Experimental results show that the algorithm can preserve image details well and maintain spatial consistency [98].

Another work that applies CNNs to image fusion was proposed by Zagoruyko and Komodakis, in which a CNN-based model was employed to directly learn a general similarity function for comparing image patches from the image data. Then, by exploring and studying a variety of neural network structures, it was proved that these network structures were particularly suitable for this task, which provides a new idea for studying the effectiveness of convolutional neural network design target fusion indicators [99].

Generally speaking, existing feature fusion techniques can be subdivided into two key categories: namely, feature selection-based and feature extraction-based.

In feature selection-based fusion methods, all sets of feature vectors are first grouped together, after which an appropriate feature selection method is utilized. Battiti [101] proposed a method using supervised neural networks, [102] presented a fusion method based on dynamic programming, and Shi and Zhang provided a method based on support vector machines (SVM) [103].

In the feature extraction-based methods, moreover the multiple feature vector sets are combined into a single set of feature vectors, which are input into a feature extractor for fusion [100] [104]. Moreover, the classical feature combination algorithm is feature extraction-based, meaning that it groups multiple sets of feature vectors into one union-vector (or super-vector) [104].

In [100], the union-vector-based feature fusion method is defined as serial feature fusion, while the feature fusion method based on the complex vector is called parallel feature fusion. Qin and Yung devised a method that used localized maximum-margin learning to fuse different types of features during BOCVW modeling for eventual scene classification [105]. A new feature extraction method, based on feature fusion, is proposed in [106] based on the idea of canonical correlation analysis (CCA). In order to classify very high resolution (VHR) satellite imagery, Huang et al. [107] presented a multi-scale feature fusion methodology based on wavelet transform. Furthermore, in order to detect nighttime vehicles effectively, a novel bio-inspired image enhancement method with a weighted feature fusion technique was devised by Kuang and Zhang [108]. Yang et al. [109] presented a novel feature combination strategy, the idea behind which involves combining two sets of feature vectors using a complex vector instead of a real union-vector. Fernandezbeltran et al. proposed a novel pLSA-based image fusion method designed to reveal multimodal modal patterns in SAR and MSI data, thus effectively merging and classifying Sentinel-1 and Sentinel-2 remote sensing data [110]. The work of [111] proposed a novel label fusion method, referred to as Feature Sensitive Label Prior (FSLP), which takes both the variety and consistency of the different features into account as was utilized to gather atlas priors.

Classical feature fusion methods represent feature data as real numbers. These and include inference [109] [112] and estimation-based methods [113] [114], as well as methods that employ certain types of feature data [115] [116] [117] [118]. Peng et al. [119] presented a quantum-inspired feature fusion method that uses the maximum mutual von Neumann entropy to define the relationship between quantized feature samples. Peng and Deng developed a quantum-inspired feature fusion method with collision and reaction mechanisms [120]. A new quantum-inspired feature fusion method, based around maximum fidelity in order to better improve the completeness and conciseness of existing feature data, was developed in [121]. In addition, a multi-modal feature fusion-based framework was proposed to improve geographic image annotation: this algorithm leverages a low-to-high learning flow for both the deep and shallow modality features, with the overall goal of achieving effective geographic images representation [122].