Figure 1(a) and Figure 1(b) show the conceptual block diagrams of the two networks used in this study, in which the data are input from the “Input” node on the left, flowed toward the bottom and then toward the top, processed at each node and output to the “Output” node on the right side. The basic U-net [5] that employs fully convolutional network architecture is shown in Figure 1(a). In Figure 1(a), the input data after the batch-normalization [10] first go through a series of down-sampling operations, i.e., the “Down_n” nodes on the left side. Each “Down_n” node has the same internal structures, including a convolution layer for feature extraction, followed by a dropout layer (drop rate = 0.5) to prevent the network from over-fitting, a Rectified Linear Unit (ReLU) [11] activation layer which is simply a piecewise function to keep only the positive value of input function to overcome the vanishing gradient problem, and a max pooling layer for down-sampling. The data then enter the “Bottom” node that has the same convolution operation but without the down-sampling layer.

The opposite operations of the convolutions in U-Net are a series of up-sampling (or deconvolutional) operations, i.e., the “Up_n” nodes on the right side of Figure 1(a). Similar to the “Down_n” nodes, each “Up_n” node has the same internal structures, particularly for up-sampling. Among these up-sampling nodes, the transposed convolution layer (TCL) [7] of the “Up_n” node is named as “Deconv” in Figure 1(a).

TCL is mainly the opposite of convolution by applying a de-convolution matrix on the input image, plus up-sampling by dilation that produces a higher resolution output from the lower-resolution input. The resulting layer are then added to the output of the corresponding resolution layer on the left side (i.e.,

the “Dow_n” operations). This addition, usually called skip connection, is shown as the line from the “Down_n” layer to the corresponding “Up_n” layer in the U-Net of Figure 1(a).

However, the outputs from these up-sampling nodes suffer from checkerboard artifacts that greatly limit the capabilities of the network model in generating smooth outputs on semantic segmentation. Detailed explanation of this artifact can be found in Odena, A., et al. [8]. In brief, as pointed out in Figure 2 and Figure 3 of Gao, H., et al. [9], standard deconvolutional operation can be decomposed into several convolutional operations, and these convoluted intermediate feature layers are shuffled and combined to produce the final layer. As a result, there is no direct relationship among these intermediate feature layers since they are generated by independent convolutional kernels. This independence usually causes uncorrelated values of adjacent pixels, leading to checkerboard artifacts [8], which greatly limit the capabilities of the network in generating smooth outputs on semantic segmentation.

To overcome the checkerboard problem, the Pixel De-convolutional layer (PDL) [9] was proposed to replace the TCL up-sampling layer in the up-sampling nodes so that the intermediate feature layers are generated sequentially from the previously generated ones. That is, the intermediate feature layers in a later stage are forced to depend on previous ones. This operation creates direct relationship between each intermediate feature layers. The U-Net with the original up-sampling layer replaced by the PDL is called the Pixel-Deconvolutional-Net (PDN) [9]. With PDN, the direct relationships between adjacent pixels on the output feature layers are established as explained in the following.

The up-sampling procedure of PDN is detailed using the “Up-2” up-sampling node in Figure 1(a) as an example by replacing the “Deconv” layer with the “PDL” layer in Figure 1(b). As shown in Figure 1(b) the output of the “Bottom” layer was fed into the up-sampling PDL. The PDL applies two sequential convolution operations on the input layer [9]. These two convolution layers were then dilated to higher dimensions and input to the final “Add” layer. The dilation operation is done by adding zeros between two adjacent voxels in the input matrix so the output matrix dimensions are doubled, for example, from 8 × 8 × 8 to 16 ×16 × 16. The output of the second convolution layer is further processed by the two “Mask Convolution Layers”. As shown in Figure 1(b), the “Mask Convolution Layer” (orange rectangular plates) is a combination of “LConv”, “Split” and “Dilate Shift”. The “LConv” is a convolution that results in 3 times (i.e., “×3” in the figure) the number of features of the input matrix (note that the number of features is indicated as the last number in the parenthesis after each layer name in Figure 1(a) and Figure 1(b)). For example, the input dimensions for “LConv” in Figure 1(b) is (?, 8, 8, 8, 32). After “LConv”, the output dimensions will become (?, 8, 8, 8, 96). The ×3 is just because there are three Cartesian coordinate (x/y/z) for the matrix. The following “Split” operation splits this output into three separate equal-sized matrix. The 3rd step, the “Dilate Shift” operation is a dilation followed by a one-voxel shift on the output matrix along one of x/y/z axis. As in the previous example, the (?, 8, 8, 8, 96) matrix will be split into three (?, 8, 8, 8, 32) matrices. Each matrix is dilated to size (?, 16, 16, 16, 32) and shifted one voxel along the x, y or z axis. So one “Mask Convolution Layer” generates three shifted dilated convoluted layers. Finally the two dilated layers and two shifted dilated convoluted layers are combined to form the intermediate output layer. The conceptual graph of the PDL is also shown in Figure 6 of Gao, H., et al. [9].

All codes were written in Python with TensorFlow 2.1 and running on a laptop with an Intel i7-7700HQ CPU, 16 GB memory and an NVIDIA GTX 1050 Ti GPU with 4 GB video memory. All the parameters in the network were chosen considering the achievable performance and endurability under these hardware limitations.

Seventy-eight liver cases were identified in the Eclipse treatment planning system of our facility and used to test the PDN. All patients received radiation treatments in the abdominal area and had a full contour of the liver. All contours in the data set were contoured by either a medical dosimetrist or a radiation oncologist and were reviewed and approved by the attending radiation oncologists of our facility during the daily smart rounds. The summary of the dataset was listed in Table 1.

For each case, the liver CT and structure sets were anonymized in the Eclipse treatment planning system [12] and exported to the local disk in DICOM RT format. With the help of SimpleITK [13] and Plastimatch [14], the CT DICOM

files were transformed to 3D images and the liver contour in the DICOM RT files were converted to mask images. Both image sets were represented as 3D array. For the CT images, each voxel used two bytes to represent its gray scale, while for the mask image only one bit (1 or 0) was necessary because its function was to indicate if a voxel was inside or outside the liver contour. In compliance with the 3D image format of Tensorflow [15], one color dimension was also added to the 3D array to form a 4D array. Because the CT and mask images were both grayscale, the added color dimension was one.

The next step was to unify the dimensions of these seventy-eight image sets. Although the axial dimensions were the same 512 (Height) × 512 (Width), the number of slices for each scan was different, ranging from 100 to 250 with a mean of ~167. This inconsistence in slice number caused problems in programming the Tensorflow. In addition, our hardware computation power was not sufficient to hand the 512 × 512 axial dimensions. As a result, the original CT and mask images were converted to lower dimensions. This was achieved by resizing each image to low dimension levels, that is, 64 (Depth/Slice) × 64 (Height) × 64 (Width) × 1 (Color). B-spline interpolation was applied in the resizing operation to minimize the loss of image information due to dimension reduction. The TFRecord format, a simple format for storing a sequence of binary records, was used as the pipeline storage during the training or testing of the networks.

Figure 2 shows the general process flow for network training and testing (or prediction) of this study. The exported data set, after going through the preprocessing as described above, was split into two groups: about 80% (62 cases) as the training set and the rest (16 cases) as the testing set. When the data of a training case were entered into the network, the 4D CT images were used as the input (i.e., the input in Figure 1(a)) from which the network tried to segment the liver, and the liver 4D mask image as the label, i.e., the ground truth of the contours, from which the network could compare and learn.

The network training was performed in a sequential manner by dividing the training process into multiple steps. That is, the training data set was first divided into subgroups with, e.g., four patients in each group. In each step, the network took one group (four patients) as input, predicted the mask image and compared the predicted mask with the label (ground truth) by calculating the Dice loss function between the two mask images. The parameters (i.e., the weights/bias for segmentation) used by the networks for segmentation, were adjusted automatically by stochastic gradient descent (SGD) [16], to minimize the loss function. Once the training of the current step was done, the network took the next group of patients. The same sequence was repeated until the network finished one epoch, or one complete pass through the training set. The training process was stopped after 200 epochs, considering the balance between the length of the training time and the convergence of the loss function.

The prediction path was similar to the training path except for a few steps. First, when constructing the low-resolution dataset, no liver contouring was needed so only CT images were fed into the network. Second, the network parameters were fixed and not adjustable. Lastly, only one patient dataset was used for each prediction.

The performance of the U-Net and PDN was evaluated for the low-dimension levels, i.e., 64 (Depth/Slice) × 64 (Height) × 64 (Width) × 1 (Color) using the 16 cases in the testing set, after the networks were trained with the 62 cases in the training set. The similarity Dice score of segmented livers was calculated as a guideline for evaluating the segmentation quality of the predicted mask results.

In addition, the predicted mask image was resized back to the original dimensions (i.e. 512 × 512 × 64 × 1), which is called the “Contour” image, so that it could be compared with the original contour. For this dimension level, the Average Hausdorff Distance (AHD) was also calculated to further evaluate the segmentation results. The AHD measures the distance between the ground true set (X) (i.e., label/mask images) and the predicted set (Y) (i.e., predicted label from network). It involves the calculation of two directed HDs: 1) the directed HD from X to Y, given by the sum of the shortest distances between all points of X and Y divided by the number of points in X, and 2) the directed HD from Y to X, which is calculated in a similar way with the roles of X and Y switched. AHD is the mean of these two directed HDs. Obviously, the smaller the AHD is, the more similarity between the two sets. Therefore, the final metrics were the Dice score for the predicted mask images, and the Dice score and AHD for the “Contour” images.

The performance of these two networks was directly compared using the same training (the first 62 cases) and testing (the last 16 cases) sets, each with 200 training epochs. The mean metrics scores of the sixteen testing cases for each network are shown in Table 2. The training time of each network is also presented in the last column of Table 2.

The similarity Dice score of predicted images and “Contour” images were almost the same for both networks. However the mean and standard deviation of “Contour AHD” for the PDN were respectively 6.0% and 51.5% better than that for the U-Net. Surprisingly, the training times for both networks were around 30 minutes for 200 epochs and only 3 more minutes for PDN although the PDL is much complicated than the TCL. The prediction time was negligible relative to the training time for both networks. Segmented livers on one slice of two testing cases for these two networks are shown in Figure 3.

The segmentation results for two challenging cases, one with a fiducial and the other with a hole inside the contour are shown in Figure 4. As demonstrated in Figure 4, neither U-Net nor PDN was able to achieve similar segmentation of the liver containing a fiducial or a hole to that performed by human experts. However, the PDN did outperform the U-Net for the “hole” case as the PDN did not accidently include the contrast from stomach into the liver contour as seen in the lower panel of Figure 4.