In this paper, we employ a sensing scheme based on PCE or also known as Coded Aperture (CA) video frames as described in [12]. Figure 1 illustrates the differences between a conventional video sensing scheme and PCE, where random spatial pixel activation is combined with fixed temporal exposure duration. First, conventional cameras capture frames at certain frame rates such as 30 frames per second. In contrast, PCE camera captures a compressed frame called motion coded image over a fixed period of time (Tv). For example, a user can compress 30 conventional frames into a single motion coded frame. This will yield significant data compression ratio. Second, the PCE camera allows a user to use different exposure times for different pixel locations. For low lighting regions, more exposure times can be used and for strong light areas, short exposure can be exerted. This will allow high dynamic range. Moreover, power can also be saved via low sampling rate in the data acquisition process. As shown in Figure 1, one conventional approach to using the motion coded images is to apply sparse reconstruction to reconstruct the original frames and this process may be very time consuming.

Suppose the video scene is contained in a data cube X ∈ R M × N × T where M × N is the image size and T is the number of frames. A sensing data cube is defined by S ∈ R M × N × T which contains the exposure times for pixel located at (m, n, t). The value of S ( m , n , t ) is 1 for frames t ∈ [ t start , t end ] and 0 otherwise. [t_start, t_end] denotes the start and end frame numbers for a particular pixel.

The measured coded aperture image Y ∈ R M × N is obtained by

Y ( m , n ) = ∑ t = 1 T S ( m , n , t ) ⋅ X ( m , n , t ) (1)

The original video scene X ∈ R M × N × T can be reconstructed via sparsity methods (L₁ or L₀). Details can be found in [12].

Instead of doing sparse reconstruction on PCE images or frames, our scheme directly acts on the PCE or Coded Aperture Images, which contain raw sensing measurements without the need for any reconstruction effort. Utilizing raw measurements has several challenges. First, moving targets may be smeared if the exposure times are long. Second, there are also missing pixels in the raw measurements because not all pixels are activated during the data collection process. Third, there are much fewer frames in the raw video because many original frames are compressed into a single coded frame. Consequently, training data may be scarce.

In this study, we have focused our effort into simulating the measurements that should be produced by the PCE-based compressive sensing (CS) sensor. We then proceed to show that detecting, tracking, and even classifying moving objects of interest in the scene is feasible. We carried out multiple experiments with three diverse sensing models: PCE/CA Full, PCE/CA 50%, and PCE/CA 25%. PCE full refers to the compression of 30 frames to 1 with no missing pixels. PCE 50 is the case where we compress 30 frames to 1 and at the same time, only 50% of pixels are activated for a length of 4/30 seconds. PCE 25 is similar to PCE 50 except that only 25% of the pixels are activated for 4/30 seconds.

Table 1 below summarizes the comparison between the three sensing models. Details can be found in [19].

Strictly speaking, YOLO is a detector rather than a tracker. Here, tracking is done via detection. That is, we apply YOLO to detect multiple targets and the target locations are extracted in every frame. Collecting the location information from the various frames will then create target trajectories.

YOLO tracker [20] is fast and has similar performance as Faster R-CNN [24]. We picked YOLO because it is easy to install and is also compatible with our hardware, which seems to have a hard time to install and run Faster R-CNN. The training of YOLO is quite simple. Images with ground truth target locations are needed.

YOLO has 24 convolutional layers followed by 2 fully connected layers. Details can be found in [20]. The input images are resized to 448 × 448. It has some built-in capability to deal with different target sizes and illuminations. However, it is found that histogram matching is essential in order to make the tracker more robust to illumination changes.

YOLO also comes with a classification module. However, based on our evaluations, the classification accuracy using YOLO is not as good as ResNet in Section 3. This is perhaps due to a lack of training data.

The ResNet-18 model is an 18-layer convolutional neural network (CNN) that has the advantage of avoiding performance saturation and/or degradation when training deeper layers, which is a common problem among other CNN architectures. The ResNet-18 model avoids the performance saturation by implementing an identity shortcut connection, which skips one or more layers and learns the residual mapping of the layer rather than the original mapping.

Training of ResNet requires target patches. The targets are cropped from training videos. Mirror images are then created. We then perform data augmentation using scaling (larger and smaller), rotation (every 45 degrees), and illumination (brighter and dimmer) to create more training data. For each cropped target, we are able to create a data set with 64 more images.

We have mid-wave infrared (MWIR) and long-wave infrared (LWIR) videos from our sponsor. There are two videos from each imager: Video 4 and Video 5. Vehicles in Video 4 start from a parking lot and then travel to a remote location. Video 5 is just the opposite. Each frame contains up to three vehicles (Ram, Silverado, and Frontier), which are shown below in Figure 2.

It is challenging for target tracking and classification using the above videos for several reasons. First, the target orientation changes from the top view to side views. Second, the target size varies a lot in different frames. Third, the illumination is also different. Fourth, the vehicles look very similar to one another, as can be seen in Figure 2.

Here, we also briefly mention the image characteristics of SWIR, MWIR, and LWIR. From Figure 3 [25] , one can see the bands are different. SWIR lies in the range of 0.9 to 1.7 microns; MWIR is in the range of 3 to 5 microns; LWIR is within the range of 8 to 14 microns. Because of those different wavelength ranges, the image characteristics are very different, as can be seen in Figure 4 and Figure 5. The daytime and nighttime behaviors are also different.

We used the following metrics for evaluating the YOLO tracker performance:

· Center Location Error (CLE): It is the error between the center of the bounding box and the ground-truth bounding box.

· Distance Precision (DP): It is the percentage of frames where the centroids of detected bounding boxes are within 20 pixels of the centroid of ground-truth bounding boxes.

· EinGT: It is the percentage of the frames where the centroids of the detected bounding boxes are inside the ground-truth bounding boxes.

· Number of frames with detection: This is the total number of frames that have detection.

For classification, we used confusion matrix and classification accuracy as performance metrics.

Conventional tracker results

We first present some tracking results using a conventional tracker known as STAPLE [22]. STAPLE requires the target location to be known in the first frame. After that, STAPLE learns the target model online and tracks the target. However, even in PCE full cases as shown in Figure 6 for MWIR videos and in Figure 7 for LWIR videos. STAPLE was not able to track any targets in subsequent frames. This shows the difficulty of target tracking using PCE cameras.

MWIR: Train using Video 4 and Test using Video 5

We used YOLO tracker here. Video 4 was used for training and Video 5 for testing. Four performance metrics were used in the studies. Tables 2-4 show the tracking results for PCE full, PCE 50, and PCE 25, respectively. In Table 2 (PCE full case), one can see that the percentages of correct detection are very high. The CLE is around 5 pixels and the DP and EinGT values are all close to 100%. In Table 3 (PCE 50 case), we observe that the percentages of correct detection start to drop. The CLE values become higher as compared to PCE full. The DP and EinGT values are still good. For the PCE 25 case (Table 4), the percentages of frames with detection are even lower as compared to the other two cases. The CLE values are getting bigger. The general trend is that when the compression ratio increases, the performance drops accordingly. This can be corroborated in the snapshots shown in Figures 8-10 where more incorrect labels can be seen in the high compression cases. It should be noted that labels came from the YOLO tracker, which has inferior performance than ResNet. We will see more classification results in the later sections.

MWIR: Train using Video 5 and Test using Video 4

This is the reverse case where Video 5 was used for training and Video 4 for testing. Tables 5-7 show the tracking results for PCE full, PCE 50, and PCE 25, respectively. The trend is that when the compression ratio increases, the performance drops accordingly. This can be confirmed in the snapshots shown in Figures 11-13 where we can see that some targets do not have bounding boxes around them in the high compression cases. We also see that more incorrect labels in high compression cases.

Here, we applied two classifiers: YOLO and ResNet. It should be noted that classification is performed only when there are good detection results from the YOLO tracker. For some frames in the PCE 50 and PCE 25, there may not be positive detection results and for those frames, we do not generate any classification results.

MWIR: Training Using Video 4 and Testing Using Video 5

Here, Video 4 was used for training and Video 5 for testing. Tables 8-10 show the classification results using YOLO and ResNet for PCE full, PCE 50, and PCE 25, respectively. In each table, the left side shows the confusion matrix and the last column shows the classification accuracy. In all cases, the first observation is that the ResNet performance is better than that of YOLO. For instance, the averaged classification accuracy in ResNet is 0.5 and the averaged classification accuracy for YOLO is only 0.31 in the PCE full case. The second observation is that the classification performance deteriorates with high missing rates. Due to aggressive compression (>30:1), the ResNet classification rates are also low in the PCE 25 case. Third, we also notice that Frontier has higher classification accuracy than Ram and Silverado. This is probably because RAM and Silverado may have similar appearance, as can be seen from the confusion matrices in those tables.

MWIR: Training Using Video 5 and Testing Using Video 4

Here, Video 5 was used for training and Video 4 for testing. The observations in Tables 11-13 are similar to the earlier case. That is, ResNet is better than YOLO and classification performance drops with high compression rates.

LWIR: Train using Video 4 and Test using Video 5

From Table 14 (PCE full) case, the CLE, DP, and EinGT metrics all look normal. The numbers of frames with detection are lower than those of MWIR. Frontier has higher detections than Ram and Silverado. For PCE 50 (Table 15) and PCE 25 cases (Table 16), we observe that the YOLO tracker has more missed detections when the compression ratio increases. DP and EinGT scores are all high. CLE scores increase as compression increases. Moreover, from Figures 14-16, we can see that there are some missed bounding boxes as well as incorrect labels around the vehicles.

LWIR: Train using Video 5 and Test using Video 4

Figures 17-19 and Tables 17-19 summarize the LWIR study where Video 5 was used for training and Video 4 for testing. Similar to earlier section, we have decent performance when the compression is low. Some partial targets can be tracked. In general, the percentages of frames with detection are lower as compared to the case of using Video 4 for training and Video 5 for testing. Moreover, the overall performance of tracking of LWIR videos is inferior to that of MWIR videos.

LWIR: Training Using Video 4 and Testing Using Video 5

We performed a comparative study between ResNet and the built-in YOLO classifiers. Tables 20-22 summarize the classification results for PCE full, PCE 50, and PCE 25, respectively. In each table, the left side shows the confusion matrix and the last column shows the classification accuracies. We observe that ResNet has much higher classification accuracy than YOLO. The results here are similar to those in the MWIR cases. That is, ResNet classification results are much better than those of YOLO.

LWIR: Training Using Video 5 and Testing Using Video 4

We have similar observations as the previous LWIR case. In the PCE full case (Table 23), the ResNet results are very good for Ram and Frontier, but not for Silverado. When compression ratio is beyond 30 to 1, the classification rates drop significantly as can be seen in Table 24 and Table 25 in which Ram and Silverado have very poor classification rates. We think that, in the coded aperture camera case, we should not use high compression, as the targets will be smeared too much. Moreover, MWIR may be preferred over LWIR in target tracking and classification.