With the development of deep learning, vehicle tracking methods based on object detection have gradually become mainstream. Object detectors generate candidate bounding boxes by detecting each video frame, and vehicle trajectories are formed using subsequent association strategies.

1) YOLO Series Models

The YOLO (You Only Look Once) model, proposed by Redmon et al. [8] , pioneered single-stage object detection. The current popular version, YOLOv8, optimizes the network structure and feature fusion, demonstrating excellent accuracy and speed in vehicle detection tasks [9] [10] .

The advantage of the YOLOv8 model is its ability to achieve real-time vehicle detection with a high frame rate, performing well even in occlusion and low-light conditions. However, its detection results are highly dependent on the diversity of the training data, and there is limited coordination with subsequent tracking modules.

2) Faster R-CNN

Faster R-CNN, introduced by Ren et al. [11] , significantly improves detection accuracy by incorporating a Region Proposal Network (RPN). Zhang Ying et al. [12] designed a vehicle detection solution based on the Faster R-CNN model for unmanned aerial vehicle (UAV) platforms. Ouyang Bo et al. [13] proposed a lightweight tracking model, FA-SORT, which achieved a tracking speed of 29.93 FPS when tested on the UAVDT dataset.

The advantage of Faster R-CNN lies in its high accuracy, making it particularly suitable for complex backgrounds and dense target scenarios. However, its computational complexity is relatively high, making it difficult to meet real-time processing requirements.

Classic filtering methods, represented by the Kalman filter, estimate and update vehicle trajectories by combining detection results with predicted states. These methods are typically used in conjunction with object detectors for multi-target tracking tasks.

1) Kalman Filter (KF)

The Kalman Filter (KF) is an efficient linear state estimation method widely used for vehicle tracking in scenarios involving constant or linear motion. Gordon et al. [14] applied KF in real-time traffic flow monitoring, significantly improving the accuracy of vehicle trajectory predictions. However, KF performs poorly when dealing with nonlinear or complex motion.

2) Extended Kalman Filter (EKF)

The Extended Kalman Filter (EKF) extends the KF to nonlinear systems by performing a first-order linearization, making it suitable for a wider range of applications. Reid et al. [15] applied EKF to multi-target tracking, achieving high accuracy in vehicle scenarios involving rapid acceleration and sharp turns. However, its performance in highly nonlinear scenarios is still limited by linearization errors.

3) Unscented Kalman Filter (UKF)

The Unscented Kalman Filter (UKF) improves trajectory prediction accuracy in nonlinear systems by using sigma-point sampling, eliminating the need for linearization. The UKF algorithm, proposed by van der Merwe et al. [16] , demonstrates excellent performance in dynamic target tracking. Farag et al. [17] combined UKF with deep detectors to perform multi-target tracking in complex traffic environments.

In recent years, researchers have explored the deep integration of object detection and filtering techniques to optimize vehicle tracking performance. Bui T et al. [18] proposed a tracking framework that combines YOLOv5 with an optimized Kalman filter (KF) algorithm in DeepSORT, achieving a 15.5% improvement in mAP and a 14.2% increase in MOTA. Gao J et al. [19] reduced the impact of observation noise on detection accuracy during nonlinear motion by incorporating a lightweight channel block attention mechanism (LCBAM) and noise-adaptive Extended Kalman Filter (NSA-EKF). These methods demonstrate the complementarity of deep detectors and classical filtering techniques in vehicle tracking tasks, making them a current research hotspot.

Deep learning-based object detection methods offer efficient target recognition capabilities, while classical filtering methods provide irreplaceable advantages in state estimation and trajectory prediction. Integrating the strengths of both approaches into a unified tracking framework is an effective solution for vehicle tracking in complex dynamic environments.

In this study, YOLOv8 is responsible for the initial vehicle detection, providing high-quality target information for the subsequent tracking module. This information is then used to enhance the overall tracking performance through Kalman filtering and error feedback mechanisms. The following section outlines the configuration and structural optimizations of the YOLOv8 model used in this research.

1) Overall Framework Configuration

In this study, YOLOv8 still uses CSPDarknet as the backbone network, incorporating a bottleneck structure (Bottleneck CSP) to reduce redundant computations and lower model complexity, thereby improving detection speed while maintaining accuracy. Additionally, multi-scale feature fusion modules (FPN and PAN) are introduced in the structure to enhance the model’s performance in detecting vehicles of varying sizes. Furthermore, the application of depthwise separable convolutions significantly reduces the computational cost of YOLOv8, enabling real-time processing. Finally, to improve detection accuracy, a reinforced data augmentation strategy is employed during YOLOv8 training, including random scaling, translation, flipping, and color adjustment, which enhances the model’s robustness and generalization ability. The overall framework of YOLOv8 is shown in Figure 1 .

2) Input and Output Formats

The input format for the YOLOv8 detection module is a standard RGB image, which is normalized and resized to a specific dimension (e.g., 640 × 640) before being fed into the model. This ensures consistent detection accuracy across images with different resolutions. During the input process, YOLOv8 also incorporates anchor box adaptive adjustment techniques, allowing the model to more flexibly accommodate vehicles of various sizes and perspectives.

The output format of YOLOv8 includes the coordinates of the predicted bounding boxes, target categories, and confidence scores. For vehicle detection tasks, the model outputs the vehicle’s bounding box positions (x, y, width, height) along with the probability distribution for the corresponding category (e.g., car, truck, motorcycle), where the confidence score is used to filter out low-confidence targets. After post-processing with Non-Maximum Suppression (NMS), overlapping redundant boxes are eliminated to ensure the accuracy of the output results. In this study, the detection boxes output by YOLOv8 are directly fed into the Kalman filter module for further trajectory prediction and state estimation.

In vehicle tracking tasks, the Kalman filter is a commonly used state estimation method that combines observation and prediction to estimate the object’s position and motion state. While the standard Kalman filter (KF) performs excellently in linear systems with Gaussian noise, it has limitations in dynamic, nonlinear, and non-Gaussian environments. Therefore, this study introduces several variants of the Kalman filter, including the Extended Kalman Filter (EKF), Unscented Kalman Filter (UKF), and Interactive Multiple Model (IMM) filter. Additionally, an adaptive filter selection mechanism based on Support Vector Machine (SVM) is proposed to dynamically select the optimal filter in different scenarios, thus optimizing detection and tracking performance.

1) Standard Kalman Filter (KF)

The standard Kalman filter assumes that both the system state model and the measurement model are linear and that the noise follows a Gaussian distribution. The state estimation process consists of two stages: prediction and update.

The state prediction equations are given by the following Equation (1) and (2).

${\hat{x}}_{k|k-1}=F_k{\hat{x}}_{k-1|k-1}+B_k{u}_k$ (1)

$P_{k|k-1}=F_k{P}_{k-1|k-1}{F}_k^{\text{T}}+Q_k$ (2)

The state update equations are given by the following Equation (3), (4) and (5).

$K_k=P_{k|k-1}{H}_k^{\text{T}}{\left(H_k{P}_{k|k-1}{H}_k^{\text{T}}+R_k\right)}^{-1}$ (3)

${\hat{x}}_{k|k}={\hat{x}}_{k|k-1}+K_k\left(z_k-H_k{\hat{x}}_{k|k-1}\right)$ (4)

$P_{k|k}=\left(I-K_k{H}_k\right)P_{k|k-1}$ (5)

Wherein, $F_k$ represents the state transition matrix, $H_k$ is the measurement matrix, $Q_k$ and $R_k$ denote the process noise and measurement noise covariance matrices, respectively, and $P_k$ is the estimation error covariance matrix.

2) Extended Kalman Filter (EKF)

The Extended Kalman Filter (EKF) extends the linear assumption of the standard Kalman Filter (KF) to accommodate mildly nonlinear systems. The EKF achieves approximate linearization by performing a first-order Taylor expansion on the nonlinear state transition function and observation function.

The equations for the state prediction phase (nonlinear) are given by Equations (6) and (7) as follows.

${\hat{x}}_{k|k-1}=f\left({\hat{x}}_{k-1|k-1}\right)+B_k{u}_k$ (6)

$P_{k|k-1}=F_k{P}_{k-1|k-1}{F}_k^{\text{T}}+Q_k$ (7)

The state update equations are given by the following Equation (8), (9) and (10).

$K_k=P_{k|k-1}{H}_k^{\text{T}}{\left(H_k{P}_{k|k-1}{H}_k^{\text{T}}+R_k\right)}^{-1}$ (8)

${\hat{x}}_{k|k}={\hat{x}}_{k|k-1}+K_k\left(z_k-h\left({\hat{x}}_{k|k-1}\right)\right)$ (9)

$P_{k|k}=\left(I-K_k{H}_k\right)P_{k|k-1}$ (10)

In this context, $f(\cdot )$ and $h(\cdot )$ represent the nonlinear functions for state transition and observation, respectively.

3) Unscented Kalman Filter (UKF)

The Unscented Kalman Filter (UKF) addresses the state estimation problem in nonlinear systems through a set of weighted sample points known as “sigma points.” Without the need for linearization, the UKF achieves higher-precision state estimation under stronger nonlinear conditions.

Sigma Point Generation and Propagation: A set of weighted sigma points is generated, and the propagation results of each sigma point are used to update the mean and covariance, thereby calculating the estimated state distribution.

In complex vehicle tracking scenarios, a single filter struggles to cope with all situations due to the presence of different motion patterns (such as uniform speed, acceleration, nonlinear trajectories, etc.). Therefore, an adaptive filter selection mechanism based on Support Vector Machines (SVMs) is proposed. This mechanism utilizes SVMs to classify motion patterns in the input feature space, thereby selecting the most suitable Kalman filter variant according to the characteristics of the scene. This dynamic selection can significantly enhance the adaptability of the filter and ensure the accuracy and robustness of the tracking process.

The design of the SVM-based adaptive filter selection mechanism can be divided into three main steps: feature extraction, SVM classifier training, and dynamic filter selection.

1) Feature Extraction

To achieve accurate filter selection, the system needs to extract features that can effectively distinguish between motion patterns. These features include the target’s velocity ( $v$), acceleration ( $a$), and the rate of change in its motion trajectory (such as curvature changes). The distributions of these features under different motion patterns are typically distinct, serving as effective bases for classification. Therefore, we define the feature vector as shown in Equation (11):

$X=\left[v,a,\theta \right]$ (11)

Wherein, $v$ represents the instantaneous velocity of the target at the current moment; $a$ denotes the acceleration of the target; and $\theta$ indicates the change angle of the movement direction (i.e., steering angle or trajectory curvature).

2) SVM Classifier

Within the feature space, the feature vector X is input into the Support Vector Machine (SVM) classifier. The SVM utilizes the trained classification boundary to separate different motion patterns. Specifically, we train a multi-class SVM classifier that categorizes vehicle motion into three classes: uniform speed, acceleration, and complex nonlinear motion.

Assuming the training data is denoted as $D={\left\{\left(X_i,y_i\right)\right\}}_{i=1}^N$, where $X_i$ represents the input features and $y_i$ corresponds to the motion pattern labels (1 for uniform motion, 2 for accelerated motion, and 3 for nonlinear motion). The objective is to find an optimal classification boundary by solving the following optimization problem using SVM, as shown in Equations (12) and (13):

$\underset{w,b}{\min }\frac{1}{2}{\Vert w\Vert }^2+C{\displaystyle {\sum }_{i=1}^N{\epsilon }_i}$ (12)

subject to $y_i\left(w\cdot \varnothing \left(X_i\right)+b\right)\ge 1-{\epsilon }_i,{\epsilon }_i\ge 0,i=1,\cdots ,N$ (13)

Wherein, W represents the classification weight vector, B is the bias, CC stands for the slack variable, C is the penalty parameter, and QQ denotes the kernel function used to map features into a high-dimensional space. By solving the aforementioned optimization problem, the SVM obtains the optimal separating hyperplane, enabling classification of newly input data points.

3) Filter Selection Strategy

Once the current motion pattern is determined, the system dynamically selects the corresponding Kalman filter variant based on the SVM classification result. The specific selection strategy is as follows:

Therefore, given the input features X at the current time, the SVM outputs a label $y\in \left\{1,2,3\right\}$, and the corresponding filter is selected based on this label to enhance vehicle tracking performance.

Compared to a fixed filter, the SVM-based adaptive selection mechanism overcomes the limitations of a single filter in complex scenarios by dynamically switching filters, thereby improving tracking accuracy and stability.

Figure 2 illustrates the flowchart of the SVM-based adaptive filter selection mechanism.

Firstly, data preprocessing and feature extraction are conducted to compute the current speed, acceleration, and direction change angle from the target’s historical trajectory, forming the feature vector X.

Subsequently, the feature vector X is input into an SVM classifier, which outputs the current motion mode label y (in a multi-class SVM, based on the trained classification boundaries, the SVM can effectively distinguish between uniform, accelerated, and complex nonlinear motions).

Concurrently, based on the classification result y, the corresponding Kalman filter variant is selected. For instance, when y = 1, the system employs the standard Kalman Filter (KF); when y = 2, the Extended Kalman Filter (EKF) is used; and when y = 3, the Unscented Kalman Filter (UKF) is chosen.

Finally, with the selected filter, the YOLOv8 detection results are input as observations to the filter, and the updated target state is calculated to obtain more precise location information.

Apart from selecting the appropriate Kalman filter, dynamic changes or unexpected events in the environment (such as sudden acceleration, deceleration, occlusion, or unexpected direction changes of the target) can also lead to increased tracking errors. To enhance the system’s responsiveness to these unexpected events, this study proposes an error feedback mechanism that dynamically adjusts the parameters of the Kalman filter by feeding back detection errors. This mechanism not only compensates for errors generated during detection and tracking but also adapts when the filter’s response is poor, thereby improving the system’s stability and accuracy in complex environments.

The design of the error feedback mechanism is based on dynamic feedback adjustment of detection errors and mainly consists of the following steps:

1) Error Calculation

In each frame, the detection error $e_k$ is calculated based on the difference between the target position detected by YOLOv8 (detection result) and the predicted position by the Kalman filter (prediction result).

Assuming the detection result is $z_k$ (observation) and the filter’s prediction is ${\hat{x}}_{k|k-1}$, the error is defined as Equation (14):

$e_k=z_k-{\hat{x}}_{k|k-1}$ (14)

Wherein, $e_k$ denotes the detection error vector at time $k$, encompassing information on the differences in position and velocity.

2) Error Weight Calculation

Different feedback intensities should be applied to detection errors in various scenarios. By introducing an error weight $W_k$, the errors are weighted accordingly. The error weight can be dynamically adjusted based on the magnitude of the detection error and the frequency of occurrence of unexpected events. For instance, a larger feedback weight is assigned to significantly increased errors.

The feedback weight can be defined by Equation (15):

$W_k=\alpha \cdot \Vert e_k\Vert +\beta$ (15)

Wherein, $\alpha$ and $\beta$ are adjustable parameters, and $\Vert e_k\Vert$ represents the magnitude of the error. By adjusting the feedback weight, the filter’s responsiveness to unexpected events can be enhanced.

3) Filter Gain Adjustment

The error is fed back to the process noise covariance $Q_k$ and measurement noise covariance $R_k$ of the Kalman filter. By dynamically adjusting $Q_k$ and $R_k$, the filter can enhance its adaptability to different error scenarios, thereby improving tracking accuracy.

The adjustment model for the feedback mechanism can be expressed as Equations (16) and (17):

$Q_k=Q_k+W_k\cdot (e_k{e}_k^T)$ (16)

$R_k=R_k+W_k\cdot (e_k{e}_k^T)$ (17)

The error feedback adjusts the process noise $Q_k$ and the measurement noise covariance matrix $R_k$, enabling the filter to adaptively adjust its gain matrix based on the current error.

Figure 3 illustrates the complete overall process of adaptive tracking.

To comprehensively evaluate the performance of the vehicle detection and tracking system proposed in this study, two widely used standard datasets, KITTI [20] and UA-DETRAC [21] , are selected. These datasets cover a variety of scenarios (such as urban roads, highways, and adverse weather conditions) and different dynamic conditions (such as occlusion and low-light environments). Additionally, to quantify the model’s performance, mean Average Precision (mAP), Multiple Object Tracking Accuracy (MOTA), and Frames Per Second (FPS) for real-time performance are adopted as the core evaluation metrics.

1) KITTI Dataset

The KITTI dataset is one of the standard datasets for autonomous driving research. It primarily consists of images from common traffic scenarios such as urban roads and highways, including 7481 training images and 7518 test images.

2) UA-DETRAC Dataset

The UA-DETRAC dataset focuses on vehicle detection and tracking tasks. It covers high-density traffic areas such as urban roads and intersections. It provides 10 hours of video data, comprising 1210 video clips and 140,000 annotated frames.

Table 1 summarizes the main characteristics of the KITTI and UA-DETRAC datasets:

This paper quantitatively evaluates the system performance from both detection and tracking perspectives, mainly using the following metrics:

1) Mean Average Precision (mAP@0.5)

mAP is a core metric for object detection, used to measure the overall performance of a detection model across all categories. mAP@0.5 indicates the mean average precision calculated with an Intersection over Union (IoU) threshold of 0.5. Its calculation is shown in Equation (18):

$\text{mAP}@0.5=\frac{1}{N}{\displaystyle {\sum }_{i=1}^{N}A{P}_i}$ (18)

where, $N$ is the number of categories, and $A{P}_i$ is the average precision for the i-th category.

2) Multiple Object Tracking Accuracy (MOTA)

MOTA is used to quantify the overall performance of a tracking task, taking into account object losses, mis-tracking, and ID switches. Its calculation is shown in Equation (19):

$\text{MOTA}=1-\frac{{{\displaystyle \sum }}_t\left(F{N}_t+F{P}_t+ID{S}_t\right)}{{{\displaystyle \sum }}_{t}G{T}_t}$ (19)

where, $F{N}_t$, $F{P}_t$, and $ID{S}_t$ represent the number of false negatives, false positives, and ID switches at frame $t$, respectively, and $G{T}_t$ represents the total number of ground truth objects at frame $t$.

3) Real-time Performance (FPS)

Frames Per Second (FPS) is used to measure the speed performance of the model and is an important indicator of real-time performance. A high FPS value indicates that the system is suitable for practical applications.

The experiments in this study were conducted in a high-performance computing environment to ensure the real-time and reliable performance of vehicle detection and tracking tasks. The following details the software and hardware configurations of the experimental environment, as well as the specific hyperparameter settings for the YOLOv8 detection module and Kalman filter.

The software and hardware configurations used in the experiments are shown in Table 2 :

The GPU (RTX 3060) supports rapid training and inference for YOLOv8, and the sufficient memory ensures the loading and processing of the dataset. The detection part of the experiment is implemented based on the PyTorch framework, while the Kalman filter and data processing modules utilize NumPy for matrix operations.

The hyperparameters of the YOLOv8 model have been fine-tuned to ensure a balance between detection accuracy and speed. The specific configurations are shown in Table 3 :

These hyperparameter settings have been optimized on the KITTI and UA-DETRAC datasets, effectively enhancing the detection performance of YOLOv8 in various scenarios.

To adapt to dynamic scenarios in vehicle tracking, the initialization parameters for various Kalman filters are shown in Table 4 :

Where Q, R, and P represent the process noise covariance matrix, measurement noise covariance matrix, and initial error covariance matrix, respectively; I denotes the identity matrix.

The hardware and software environments of the experiment provide powerful computational support for large-scale detection and tracking tasks. The optimized hyperparameters of YOLOv8 enable it to be efficient and robust in multiple scenarios, while the reasonable initialization and dynamic adjustment strategies of the Kalman filter ensure the accuracy and real-time performance of the tracking module.

Vehicle detection serves as the foundational step in achieving multi-object tracking, with its detection accuracy directly impacting the predictive performance of subsequent filters. To evaluate the performance of the YOLOv8 detection model used in this paper, comprehensive comparative experiments were conducted with mainstream object detection models (such as YOLOv5, Faster R-CNN, and RetinaNet) across multiple typical scenarios (including daytime, nighttime, occlusion, low-light, etc.). Table 5 summarizes the performance of different detection models in these typical scenarios.

From the comparison of mean Average Precision (mAP) at IoU threshold of 0.5, it can be seen that YOLOv8 outperforms other models in all scenarios, particularly in nighttime and occlusion scenarios, achieving mAP values of 90.1% and 87.8%, respectively.

The final loss value of YOLOv8 is 0.43, which is approximately 25% lower than that of YOLOv5 and RetinaNet, and approximately 30% lower than that of Faster R-CNN. This verifies the rationality of selecting YOLOv8 as the detection module in this paper, providing robust support for subsequent tracking tasks.

Table 6 presents the performance of the SVM-based adaptive filter selection mechanism across different scenarios. The experimental results indicate that the adaptive selection mechanism significantly outperforms fixed filter schemes in terms of tracking accuracy and robustness in complex scenarios such as accelerated motion and nonlinear motion.

In scenarios involving occlusion, sudden motion, and multi-object environments, the error feedback + SVM dynamic selection mechanism significantly improves tracking accuracy compared to other methods. Below is a comparison of the Mean Squared Error (MSE) for each method across different scenarios.

As shown in Table 7 , the error feedback + SVM dynamic selection mechanism significantly reduces MSE in occlusion and sudden motion scenarios, with superior smoothness and real-time performance indicators compared to single filters and the standalone SVM dynamic selection mechanism.

As shown in Table 8 , a further evaluation is conducted to assess the difference in effectiveness between the standalone SVM dynamic selection mechanism and the SVM dynamic selection mechanism combined with error feedback. Two different error feedback mechanisms under distinct strategies are employed and compared across multiple scenarios.

1) Error Feedback Standalone Adjustment Strategy: Error feedback only adjusts the parameters of the currently selected filter without triggering a re-selection.

2) Error Feedback + Filter Re-selection Strategy: Error feedback not only adjusts the parameters but also triggers a re-selection of the filter.

The performance advantages of SVM in dynamic filter selection primarily stem from the following aspects:

a) Discriminative power of motion features: Experiments have demonstrated that velocity, acceleration, and direction change angles exhibit significant distribution differences across different motion patterns, providing a clear basis for filter classification.

b) Nonlinear classification capability: Compared to traditional linear classifiers, SVM effectively handles nonlinear patterns through kernel function mapping, enabling more precise selection among KF, EKF, and UKF.

c) Matching of errors with scenario characteristics: By incorporating an error feedback mechanism, SVM dynamically adjusts selection strategies based on real-time error variations, further enhancing the system’s adaptability.

To validate the robustness of the SVM-based adaptive Kalman filtering, this paper analyzes the visualization results of tracking and recognition in various complex environments, as shown in Figure 4 .

(a) The four frames in row (a) illustrate the tracking and recognition performance of the model for a target vehicle with ID 17 in a turning scenario. It can be observed that the model demonstrates good performance.

(b) The four frames in row (b) show the tracking and recognition performance of the model for a target vehicle with ID 1 at different angles during nighttime. It is evident that the model maintains good locking on the target vehicle, even during the process of being overtaken.

(c) The four frames in row (c) present the tracking and recognition performance of the model for a target vehicle with ID 1 across multiple scenarios on a road segment. It is clear that the model can effectively identify and track the target vehicle even in multi-target and complex scenarios.

From this, it can be concluded that the SVM-based adaptive Kalman filtering exhibits strong robustness in complex environments. It can provide decision support for intelligent transportation, autonomous driving, security, and other fields.

Through experimental comparisons with current mainstream vehicle tracking methods such as DeepSORT and FAIR MOT, the proposed research method in this study demonstrates advantages in detection accuracy, tracking accuracy, and system stability.

As shown in Table 9 , compared to DeepSORT’s 91.5% mAP on the KITTI dataset and FAIR MOT’s 89.2% mAP, the YOLOv8 detection module combined with a dynamic filter selection mechanism proposed in this paper achieves a 94.3% mAP, exhibiting stronger adaptability to complex scenarios. Additionally, compared to DeepSORT’s 35 FPS and FAIR MOT’s 30 FPS, the real-time performance of the proposed method (45 FPS) significantly enhances processing speed while meeting accuracy requirements, making it suitable for practical deployment.