Video approach

As mentioned earlier, we have four approaches to compression Mastcam images. Two approaches were described in earlier papers. In this research, we propose a video approach to compression. Our overall technical approach can be summarized as follows. First, some preprocessing steps are used to make sure the image height and width are even numbered, normalized, and in double. Second, the nine-band image cube is converted to three groups of three-band images. Third, since different codecs require different input formats, the images are saved to appropriate formats such as Y4m or YUV444. Fourth, we will apply the various compression algorithms (JPEG, J2K, X264, X265, and Daala) to the three-frame video and generate various performance metrics. There are four performance metrics. Each metric is computed by comparing the reconstructed nine-band cube with the original 9-band cube. The compression ratio is generated by comparing the size the compressed file with the size of the original image cube.

Figure 1 illustrates the work flow of the video approach. We include some details for some of the blocks.

· Pre-processing

The preprocessing has a few components. First, it is important to ensure the input image dimensions to have even numbers because some codecs may crash if the image size has odd dimensions. Second, the input image is normalized to double precision with values between 0 and 1. Third, the different bands are saved into tiff format. Fourth, all the bands are written into YUV444 and Y4M formats.

· Video codecs

Different codecs have different requirements. For JPEG, we used FFMPEG to create a three-frame MPEG video with certain target quality parameter. The video is then decoded using FFMPEG. Finally, the decoded video is then converted to individual frames via the imread command in Matlab. For J2K, we used Matlab’s video writer to create a J2K video with certain quality parameters. We then used Matlab’s video reader to decode the video and the individual frames will be retrieved. For X264 and X265, the videos are encoded using the respective encoders with certain quality parameters. The video decoding was done within FFMPEG. For Daala, we directly used the Daala’s functions for encoding and decoding.

· Performance evaluation

In the evaluation part, each frame is reconstructed and compared to the original input band. Four performance metrics have been used.

Split Band (SB) Approach

Here, we briefly summarize an alternative approach, which was presented in [17]. Details can be found in [17]. In [17], we did not include Daala. Here, we include new results with Daala as one of the codecs. The overall architecture is very similar to that of the video approach, except that each 3-band image is now compressed independently.

Two-step approach

Some preliminary results of this two-step approach were presented in [16]. The detailed signal flow can be found in Figure 2. The first step is to apply PCA to compress the nine-band Mastcam image cube to three or six bands. The second step is to compress the three or six bands using conventional codecs. In [16], we found that six-band has better performance. It should be noted that we did not include Daala in [16] because at that time, we did not know Daala has a newer version that outperforms an old version developed about 2 years ago.

One-step PCA only approach

It is well-known that PCA can be used for data compression. Actually, PCA, unlike DCT and wavelet, is a data dependent approach that yields optimal data compression. Figure 3 illustrates the PCA approach.

Compressing RGB only

The 5, 4, 2 bands in the Mastcam image cube are the RGB bands. We have performed a separate study specifically for RGB bands. It is similar to the SB approach.

In this paper, we will compare image codecs in the market and objectively evaluate different codecs and eventually recommend the best codec to our customer.

With the above in mind, we performed a brief summary of the existing high performance codecs.

DCT based algorithms

· JPEG [10]:

JPEG is the very first image compression standard. The video counterparts are the MPEG-1 and MPEG-2 standards.

· JPEG-XR [24]:

It was developed by Microsoft. The performance is comparable to JPEG-2000.

It is mainly used for still image compression.

· VP8 and VP9 [25] [26]:

These video compression algorithms are owned by Google. The performance is somewhat close to X-264. We did include VP8 and VP9 in our study because they are not as popular as X264 and X265.

· X-264 [12]:

X264 is the current state-of-the-art in video compression. Youtube uses X264. It has good still image compression.

· X-265 [13]:

This is the next-generation video codec and has excellent still image compression and video compression. However, the computational complexity is much more than that of X264. In general, X265 has the same basic structure as previous standards and contains many incremental improvement over X264. It should be noted that X264 and X265 are optimized versions of H264 and H265, respectively.

· Daala [15]

Recently, there is a parallel activity at xiph.org foundation, which implements a compression codec called Daala [15]. It is based on DCT. There are pre- and post-filters to increase energy compaction and remove block artifacts. Daala borrows ideas from [14].

The block-coding framework in Daala can be illustrated in Figure 4.

Wavelet based algorithms

JPEP-2000 is a wavelet [11] [27] [28] [29] based compression standard. It has better performance than JPEG. However, JPEG-2000 requires the use of the whole image for coding and hence is not suitable for real-time applications. In addition, motion-JPEG-2000 for video compression is not popular in the market.

In almost all compression systems, researchers used peak signal-to-noise ratio (PSNR) or structural similarity (SSIM) to evaluate the compression algorithms.

Given a fixed compression ratio, algorithms that yield higher PSNR or SSIM will be regarded as better algorithms. However, PSNR or SSIM do not correlate well with human perception. Recently, a group of researchers investigated a number of different performance metrics [23]. Extensive experiments were performed to investigate the correlation between human perceptions with various performance metrics. According to the results found in [23], it was determined that two performance metrics correlate well with human perception. One image example shown in Figure 5 demonstrates that HVS and HVS-M have high correlation with human subjective evaluation results. In the past, we have used HVS and HVS-m in several applications [30] [31].

Mastcam imager information is shown in Figure 6 and Table 1. There are 6 overlapping bands and 3 non-overlapping bands (L3, L4 and L5 from the left camera and R3, R4, and R5 from the right camera). More details about Mastcam can be found in [2].

When using the Video approach for Daala, the nine-band image is saved as a three-frame video in the Y4M format. While Daala can also be used to encode still images, it was created with video compression as its primary use (Figure 7).

Figure 8 and Figure 9 show the comparison of different algorithms for 20 left and 20 right images, respectively. Table 2 and Table 3 summarize the performance metrics at 0.1 compression ratio for the left and right images, respectively. For left images, at 10 to 1 compression or 0.1 compression ratio, one can see that Daala is 5 and 10 dB better than that of JPEG in terms of HVS and HVSm, respectively. If we use the conventional metric (PSNR), then JPEG and J2K are

slightly better than others. In terms of SSIM, J2K and Daala are about 0.05 higher than that of the JPEG. For right images, the difference between Daala and JPEG is even bigger mainly because modern compression codecs are more efficient in compressing high resolution images.

Based on the results, we observe that Daala and J2K are very similar in performance. However when looking at the HVS and HVSm, Daala is the strongest method.

Figure 10, Figure 11, Table 4, and Table 5 show the different performance metrics using the SB approach. Let us focus on 0.1 compression ratio (10 times compression) for the left images first. For PSNR, J2K is about 3 dBs better than that of JPEG. For SSIM, J2K and Daala are 0.06 and 0.04 better than JPEG. For HVS and HVSm, we can see that Daala’s performance is 5 and 10 dBs higher than that of JPEG, respectively. This means Daala’s compression can preserve the perception quality in the reconstructed images.

In general, the performance metrics of right images are higher than those of the left images. The reason is that the modern codecs have better mechanisms to compress high resolution images.

PCA has been used as a compression tool in the past. It is also known as KLT (Kohonen Loeve Transform). Here, we briefly summarize the application of PCA only to compress left and right Mastcam images. Figure 12 and Figure 13 show that the compression performance of PCA3 and PCA6 is not enough to reach 10 to 1 compression. For instance, PCA3 achieved a PSNR of 41.25 dBs at a compression ratio of 0.18. We will see in Section 3.5 that it will be good to combine PCA with other codecs to further improve the compression performance to 10 to 1.

The first step performs the PCA compression. In the second step, we used the

video approach. In our earlier paper [16], we observed that PCA-6 is better than PCA-3. Here, we present new results for PCA-6 because: 1) we have included Daala results; 2) there are new updates of X265 and X264.

Figure 14, Table 6, Figure 15, and Table 7 show the performance metrics of the 2-step approach for left and right images. Let us focus again on the left images in the 0.1 compression ratio region. For PSNR, Daala is 6 dBs better than that of JPEG. For SSIM, Daala is 0.16 higher than that of JPEG. For HVS and HVSm, Daala has 8 and 10 dBs higher values than that of JPEG, respectively. For

right images, the differences between Daala and JPEG are even bigger.

Now, we would like to compare the Video and SB approaches. Figure 16 and Figure 17 show the detailed comparison between video and the SB approaches for the left and right images, respectively. Let us first focus on the left images near the compression rate of 0.1. We have the following observations:

· JPEG has the highest PSNR than others.

· Daala has a very high HVSm (>45 dB) in both video and SB approaches.

· Daala SB is slightly stronger than the Video approach.

· Daala SB produces a PSNR that is slightly weaker than J2K.

· Daala SB Produces a stronger SSIM than video but both approaches are below J2K.

For right images, we see a similar trend as above except that the metrics are all higher in right images.

Hence, video approach is slightly worse than SB because video has more overhead and there are only three frames. If there are more frames, the Video approach may have edges over the SB approach. Since our interest is to achieve perceptually lossless compression, Daala is a better choice than others. In addition, Daala is amenable to parallel processing whereas J2K requires the whole image for processing and is not suitable for parallel implementation.

Here, we would like to compare the video and the 2-step approaches. Figure 18 and Figure 19 show the detailed comparison between Video and the 2-step approaches for the left and right images, respectively. Let us first focus on the left images near the compression rate of 0.1. In terms of HVS and HVSm, the Video approach with Daala is better than others most of the time. The Daala’s performance is 7 dB and 10 dB better than JPEG in HVS and HVSm, respectively. X265 is the second best in HVS and HVSm. Finally, all the metrics in right images are higher than those corresponding left ones.

Figure 20, Figure 21, Table 8, and Table 9 show the metrics of different codecs for left and right RGB images, respectively. It can be seen that, at 0.1 compression

ratio, J2K performed the best in terms of PSNR and SSIM. However, in terms of HVS and HVSm, Daala is the best performing one. It is also somewhat surprising to notice that JPEG is consistently the third best one in all metrics.

To subjectively evaluate the different codecs, we include three case studies. Case 1 is for compression ratio near 0.1 compression ratio. Figure 22 shows the original and 5 reconstructed images from JPEG, J2K, X264, X265, and Daala. We observe no perceptual loss of quality as compared to the original image. Case 2 is for compression ratio near 0.05 compression ratio. Again, it is still difficult to spot any artifacts between the reconstructed images and the original images shown in Figure 23. Case 3 is for compression ratio near 0.0251 compression ratio, which corresponds to 40 to 1 compression. In this case, we start to visualize some artifacts (Figure 24) in JPEG, X264, and X265. However, Daala and J2K still do not have visible artifacts.

· Comparison of different approaches

In this paper, we propose a new video approach to compressing Mastcam multispectral images and compare with earlier approaches (PCA only, SB, and two-step). Through extensive experiments using actual Mastcam images, we observed that the SB approach yielded slightly better performance than others.

· Comparison of different codecs

In each approach, we have compared five codecs. Amongst them, we observed that Daala has the best performance in terms of HVS and HVSm. The improvement of Daala over JPEG is more than 5 dBs in those metrics at 10 to 1 compression.

Since HVS and HVSm correlate well with human perception, we believe Daala is a good candidate to replace JPEG. For the two conventional metrics (PSNR and SSIM), we occasionally observed that JPEG and J2K has slight edge over other codecs. It is somewhat surprising to notice that the two popular codecs, X264 and X265, did not yield good results at 10 to 1 compression. However, they did sometimes have reasonable performance at lower compression cases such as 5 to 1 or 3 to 1.

· Computational Complexity

Daala is DCT based and is hence amenable to parallel processing. J2K, on the other hand, is a wavelet based approach that requires the whole image. Hence, Daala is more efficient for fast processing.

· Subjective Comparisons

Through visual experiments using RGB images, we noticed that at 10:1 compression, all codecs have almost no loss. Even at 20:1 compression, it is still hard to notice any artifacts. However, at 40 to 1 compression, JPEG, X264 and X265 start to see some color distortions and block artifacts. Daala and J2K still performed reasonably well at 40:1.