This section overviews several object detection techniques that use point clouds and how they differ from the model presented in this paper.

Autonomous driving datasets are the most common type of labeled LiDAR data, whereas the authors are not aware of the examples of marine datasets that are widely available. In the CARLA Simulator [4] , a vehicle is placed in a randomized environment to collect data. Unreal Engine [9] is used for the CARLA simulation environment. Various sensors are available, such as LiDAR, cameras, lane sensors, collision detectors, and inertial measurement units. Different environments and randomized environmental conditions are also available in CARLA, including traffic and other non-playable characters.

In [5] , CARLA was modified to produce a point cloud with automatically generated semantic segmentation labels. The researchers took their generated data and mixed it with real-life data from the Semantic-Kitti [10] dataset. They found that 1) training solely on simulation data did not perform as well as training on real data, 2) mixing the two types of data improved the model performance, and may save time on the rigorous labeling of real-world data. Most notably, infrequent objects such as emergency service vehicles were supplemented by increasing their appearances in the simulation dataset. This increased the model’s predictive power of specific objects in the real-life dataset. In the current paper, similar results are achieved when simulating maritime autonomy and LiDAR in Unity while additionally exploring how various combinations of real-life and simulation data may affect the results.

In Complex-Yolo [11] , object detection is performed with point clouds. This method prioritized speed which is necessary for autonomous vehicle operation. In [11] , the authors first flattened the point cloud from a birds-eye view, and then they encoded the height, intensity, and density of the points into the RGB channels of an image. Then 2D bounding boxes were generated for each of the objects, which included the angle, the class such as car, pedestrian, or cyclist, and the position, length, and width of the object. The flattened point cloud can lose some finer details of the input point cloud, as a trade-off for speed, and networks that directly operate on points are better suited for point-cloud predictions. In the current paper, fine-grained details of the dataset are retained with the proposed model architecture as described in Section 3 and model is tested in the maritime environment.

In this research, the main network employed is PointNet++ [7] . It uses several PointNet [12] models to represent local regions of space better. Since PointNet can produce a good global feature representation of the point cloud, it is used within [7] to produce several local feature vectors of increasing length and decreasing point density.

Figure 3 shows how PointNet++ down-samples a point cloud while increasing the feature size $k_0<k_1<k_2$, then up-samples it while decreasing the feature size $k_2>k_3>k_4$. This type of architecture resembles a U-Net [13] , where skip connections and a U-shaped down-sampling and up-sampling architecture are employed. Here, each skip connection represents the local features for a subset of points. The feature-length increases with each iteration of PointNet since there is a smaller number of points representing larger and larger regions. They are all combined in the end to give several feature sets of the point cloud with different levels of detail.

SphereFormer [8] is a transformer [14] module that utilizes radial window self-attention combined with a cubic self-attention window to process sparse point cloud data. Transformers comprise attention layers, which process data within attention windows. In the case of SphereFormer [8] , the attention windows capture groups of points and within each group, the features of points are used together to generate new features within that group. The radial window is a key factor in why this module improves upon models that process LiDAR point clouds. The radial window acknowledges the feature of LiDAR scans where the sensor rays get further apart radially as the distance from the origin of the sensor increases. Because of this feature, the surfaces of objects that are farther away appear more sparse within a cubic window. In a radial window, the volume of the window increases as the distance from the origin increases.

The SphereFormer module is not designed to be a standalone machine learning model for point clouds. Instead, it is meant to be inserted into existing models such as PointNet-based models or graph neural networks. In this work, it is inserted between layers of a PointNet++ machine learning model to boost its ability to learn to interpret point clouds from LiDAR scans. The SphereFormer module resulted in a high Mean Intersection over Union performance score on the Semantic-Kitti [10] dataset, and it specifically excelled for objects that were farther away, which is why it is utilized in this paper.

The following machine learning model architecture was developed (see Figure 4 ), in which the PointNet++ architecture has been fitted with SphereFormer modules.

The original PointNet architecture is able to perform semantic segmentation on point clouds while being inherently order-invariant. This gave it an advantage over most other networks at the time by turning a point cloud into a voxel representation first [12] [15] instead of working with point cloud directly. The PointNet++ architecture was chosen for this research as it is a substantial improvement over the original PointNet network [7] since it allows the model to learn finer details of a point cloud. The SphereFormer modules are used to further enhance this model’s learning ability, but for a LiDAR-specific domain. Other networks that use order-invariant techniques similar to PointNet++ [7] are usually derived from a Graph Neural Network (GNN), which operates on a graph. In a GNN, message-passing steps update the values of the graph nodes at every step [16] . A node can have any number of neighbors, so the messages passed from the neighbors need to be aggregated with an associative function, similar to the one used in [12] . Many ML techniques may be considered to be GNNs, such as the attention mechanism [14] , KPConv [17] , as well PointNet++ itself. In this paper, the PointNet++ technique has been used due to its superior ability to capture fine-grained details.

The VLP-16 LiDAR sensor fires its sixteen infra-red lasers (and corresponding detectors) at a rate of approximately 18,000 times per second in a regular spinning pattern [3] . These lasers reflect off objects nearby and the detected time-of-flight allows for the calculation of the distance to the point (in spherical coordinates) and the intensity is used to get the object reflectivity [3] . Translating the spherical coordinates produces a set of 3D Euclidean coordinates that represent the point cloud.

Robot Operating System (ROS) [18] is a robotics platform for control systems. In particular, messages can be sent with ROS between different sensors to different processes. These messages and their metadata, such as the timestamp and topic, can be recorded and saved to a ROSBAG-type file format. In this work, the LiDAR sensor messages along with other data about the WAM-V are saved to a ROSBAG file, and this file is used to load the point cloud data for labeling.

A remote-controlled catamaran unmanned vessel (see Figure 1 ) maneuvered around various buoys. The vessel moved such that the vessel-mounted Velodyne VLP-16 LiDAR sensor (see Figure 5 ) detected buoys at various distances, heights, and angles. Varying paths taken around the buoys by the WAM-V created a diverse set of LiDAR point-cloud data.

The ROSBAG file contains records from the Velodyne VLP-16 sensor having Velodyne coordinates for all the points that collided with objects in the maritime environment. Utilizing LiDAR Toolbox in MATLAB [19] enabled the identification followed by the labeling of points in the point cloud.

Unity Game Engine [2] is a development platform designed mainly to assist in creating video games, but can also be used for simulations. Unity was chosen due to a large pool of assets, such as 3D models and scripts, available within Unity’s large community. To simulate the local maritime environment, the Unity simulation needed to introduce environmental factors, such as waves, which was done via the Crest [20] software, shore trees, and docks. The main objective of the simulation was to produce high-fidelity, synthetic LiDAR data, so lighting conditions and materials were a low priority, whereas rendering high-resolution depth images for simulating the LiDAR sensor was the highest priority. Random perturbations to the environment were also mainly aimed at augmenting the LiDAR data. The success of the simulation in producing quality synthetic data was assessed by how well the ML model developed in this research performs when assisted by the inclusion of the simulation data.

A 3D model of the WAM-V vessel was placed in a Unity scene with randomly generated objects. The WAM-V vessel model follows tracks around simulated environments (see, e.g., Figure 6 ). The Crest wave simulation allows for variable wave parameters. At the start of every loop of the simulation, random wave parameters are used to calculate different Gerstner waves [21] to simulate realistic ocean wave behavior.

When generating points with the simulated LiDAR sensor, two depth maps are utilized. The first depth map contains all of the objects in the scene, such as the buoys, buildings, and ground. The second depth map solely pertains to the distance from the sensor to the surface of the water. The minimum depth of the two maps is taken to get the true depth value and generate a point. When using Crest [20] to simulate waves, the water depth map causes points to appear on the surface of individual waves. However, during data collection with the WAM-V vessel, points collected from the LiDAR scans that are under the Z-plane where the vessel touches the water are removed. During low swell activity, this is a good estimation of where the water is at a given point. However, passing boats or other wave sources could falsely appear to be buoys if the current model is used. To match the real-life data collection, this Z-plane filter was also used in the Unity simulation, thereby curtailing somewhat the Unity simulation capabilities of generating points on the surface of waves.

The LiDAR system in Unity was simulated using a compute shader. Compute shaders are a high-level graphical processing unit (GPU) programming language for writing parallel code. In this case, each point is handled individually, sampling from four different textures and four cameras pointing in the cardinal directions. From sampling the textures, we get the layer and object IDs as well as the object and water depth of each point. Taking many samples from spherical coordinates results in a point cloud similar to that of a real LiDAR sensor. In Figure 7 , a sample snapshot of the simulation shows a labeled point cloud.

To better mirror the real-world LiDAR sensor, the simulation data output required some post-processing. Real LiDAR data collected from the Velodyne VLP-16, shows that a large slice of the point cloud is missing from the back side of the WAM-V vessel, which is due to obstruction of the view caused by the WAM-V vessel itself. Conversely, the simulation data does not exhibit such an omission, and thus, needs to be adjusted. Fortunately, simulation data may be easily post-processed since the point cloud data includes horizontal and vertical IDs for each point. These IDs may be used to quickly determine which sensor produced which point and perform adjustments, such as reducing the field of view of either axis, creating dropout, and changing the resolution. In particular, the vertical field of view is randomly set for each point cloud between 20 and 35 degrees to enhance the generalizing ability of the model. The horizontal field of view extends entirely around the LiDAR sensor, and it is randomly sampled to create artificial obstructions that more closely mimic the obstructed view of the real-life craft.

Figure 8 compares the LiDAR sensor data from the real docks, (a), and data generated by the Unity simulation and post-processing, (b). The images exhibit a similar object sparsity, in general, which may point to the simulation data representing obstructions to a similar degree. Yet, the buoys (red and green dots) are more uniformly distributed in the simulation-generated image and there are more of them ( Figure 8(b) ). The latter underscores the goal of the simulation of generating more training data reliably containing buoy objects and is leveraged in the experimental procedure to obtain quality semantic segmentation as described in Section 5.

After training on the simulation data, validation on the real data showed that the model lacked some predictive capabilities. Figure 9 depicts the results in the form of the confusion matrix, which is a technique to report in a square table layout the number of true positives, true negatives, false positives, and false negatives. The diagonal of the matrix represents all instances that are correctly predicted. In this work, three classes are to be predicted: sphere, cylinder, and other (see Figure 8 ). The first two classes are to distinguish among the two types of buoys and the latter is for all the ‘non-buoy’ objects. The small number of classes justifies employing the confusion matrix to assess the results despite the noticeable prevalence of the other objects. It may be observed in Figure 9 that a point on sphere or cylinder is more often labeled as cylinder since there are more spherical buoys in the dataset. The matrix entry corresponding to (other, other) has the highest value correctly predicting the largest number of non-buoy objects in the real dataset. Hence, the confusion matrix indicates that the model is good at distinguishing a buoy from other objects, while it is very poor at finding out which type of buoy it is. Following the ideas in [5] , this work proposes to use the real data to supplement the training set in the LiDAR Unity maritime environment when the amount of real data is limited. The next Section 5.2 describes the obtained improvements.

A simple union of the real and simulation datasets exhibits a large imbalance between the simulation and real data samples, at around 74 simulation samples per 1 real sample. Compare also Figure 8(a) and Figure 8(b) for the differences in the number of objects. This imbalance may skew the training on the combined dataset. Therefore, the model is first trained on the entire simulation dataset (as in the first experiment in Section 5.1. After the training, the model is fine-tuned on a particular combination of the two datasets. Each such combination is defined by the proportions of simulation and real data in the mixed dataset used in the fine-tuning.

To fine-tune the model, its weights are initialized from the weights saved from training entirely on the simulation dataset, and the training proceeds on the combined dataset. Several cases of the combined dataset were considered, each containing a particular ratio of simulation to real data amounts. Specifically, the amount of the real data was fixed at ${\mathcal{D}}_R$ while the amount of simulation data ${\mathcal{D}}_S$ was taken proportionally in each case as $r\times {\mathcal{D}}_R$, where the factor $r=0,1,2,3,4,5$.

In all the different $r$-factor cases, Figure 10 displays the IoU metric calculated for the validation on the real and simulation datasets (blue and orange dots, respectively) as well as for the fine-tuning the model (green dots). Note that the IoU results with abscissa equal to −1 correspond to the model without fine-tuning, as described in Section 5.1. Without any exposure to the real data, the training model overfit the simulation data, and therefore, scored very low on the real data. At $r=0$, the model is trained only on the real data and predicted it—IoU = 0.8—while performing poorly on the simulation data, which may also be due to overfitting.

The model with the factor $r=4$ is showcased here since it has a high real data IoU while having a similar IoU on the simulation data, indicating that it may be less likely to overfit either of the datasets, which is also supported by the curve of loss values shown in Figure 11 . The IoU has improved to 0.84 for the real-life data validation from both $r=-1$ and $r=0$ cases. For the simulation data validation, the IoU of 0.80 is an improvement over the $r=0$ case and a very slight decrease of 0.02 from the simulation-data-only training ( $r=-1$). These IoU scores are similar to those of 0.74 and 0.81 obtained by using the SphereFormer on the Semantic-Kitti dataset benchmarks [10] .

The showcased factor $r=4$ dataset resulted in the confusion matrix shown in Figure 12 presenting an overall improvement from the first experiment, where the model failed to differentiate between the two buoy types, and more often mislabeled other objects as buoys. The other positive factor $r$ cases achieved similar confusion matrices.

It may be observed in Figure 10 that, for the other positive $r$ factors, the IoUs are higher on the real-life data, ranging between 0.86 and 0.89, than that of the showcased $r=4$ model (IoU of 0.84 ). For those $r$’s, however, the IoU scores were all either 0.78 or 0.79 on the simulation data. Hence, for those $r$’s, substantially higher differences may be observed between the IoU scores of the simulation and real-life data as compared with the difference in the $r=4$ case. The high differences may indicate that the model is overfitting the real data in those cases. Hence, the case of $r=4$ was showcased in Figure 11 and Figure 12 since it had the closest high IoU scores between real and simulation validation sets.