Feature extraction is the basis of fusion identification, aiming to fully capture identity-related features from radar and video data to lay the foundation for subsequent fusion and matching, as shown in Figure 2.

Figure 2.Feature extraction processes of the two modalities.

2.1.1. Radar Feature Extraction

Millimeter-wave radar extracts features based on echo signals, mainly including range-Doppler features, spatial structure features, and motion trajectory features [6].

The radar data collection usually requires the target to move naturally within the sensing range [2] to obtain sufficient echo information. Typical radar feature extraction applications involve various feature types, data forms, and sample sizes, as shown in Table 1.

Table 1. Typical applications of radar data acquisition and feature extraction.

Common radar feature extraction techniques include two-dimensional Fast Fourier Transform (2D-FFT), Multiple Signal Classification (MUSIC) algorithm, and point cloud voxelization. 2D-FFT converts time-domain signals into Range-Doppler Maps (RDM) to extract distance and speed features [9]. For example, Caoet al. [3] used 2D-FFT to process radar echo signals to obtain range-Doppler map sequences containing gait motion information; the MUSIC algorithm is used for high-precision angle estimation to generate Angle-Time Maps (ATM) to supplement spatial orientation features; point cloud voxelization converts sparse point clouds into regular voxel grids to facilitate feature extraction by deep learning models. Luoet al. [6] adopted a sliding window-based method to convert single-frame point clouds into multi-frame time-series data, retaining the temporal motion features of skeletal postures.

2.1.2. Video Feature Extraction

Video feature extraction focuses on identity-related visual information, such as facial features, gait features, and skeletal features [7]. Facial features include texture, contour, and key point information, extracted through face detection and feature encoding methods [12]; gait features originate from motion patterns such as walking posture and step frequency [10], usually obtained by analyzing image sequences or contour sequences [3][4]; skeletal features extract 3D coordinates of human joints through pose estimation algorithms [6][13].

Typical video feature extraction applications involve various feature types and processing methods, as shown in Table 2.

Common data forms include Range-Doppler Spectrograms (RDM), 4D point clouds (x, y, z, v), time-frequency spectrograms, etc. Extraction technologies mainly include the following three categories:

Table 2.Typical applications of video data acquisition and feature extraction.

Signal transformation technology converts time-domain signals into range-Doppler spectrograms through 2D-FFT to capture target motion features [14] and generates time-frequency spectrograms using STFT to extract micro-Doppler features [15][16]. Spatial feature extraction processes sparse point clouds using networks such as PointNet and DGCNN to capture 3D human structure and contour topology [7]. It also achieves high-precision angle estimation through the MUSIC algorithm to generate ATM [17]. Temporal feature mining converts single-frame point clouds into time series using a sliding window mechanism to capture dynamic patterns of gaits and actions [18] and optimizes trajectory features through Kalman filtering to improve stability [19].

Feature fusion and alignment are key elements for solving the heterogeneity of millimeter-wave radar and video data, directly affecting the performance of the fusion identification system [20]. The specific implementation is shown in Figure 3.

Figure 3.Implementation flow of spatio-temporal alignment, feature distribution alignment, and fusion strategies.

2.2.1. Fusion Strategies

According to different fusion stages, fusion strategies can be divided into data-level fusion, feature-level fusion, and decision-level fusion [5].

Data-level fusion strategy directly integrates raw radar and video data before feature extraction. For example, Zong et al. [19] spliced radar range-angle information with video pixel data to form a multimodal data matrix, which is input into the fusion detection network. This strategy retains the most original information but has high requirements for data synchronization and noise reduction.

The feature-level fusion strategy integrates extracted radar and video features in a unified feature space, which is the most widely used fusion strategy in current research. For example, Liu et al. [7] designed a cross-modal similarity estimation method to fuse radar point cloud features and video 2D image features. Shi et al. [4] fused radar micro-motion features and video gait appearance features in a multi-scale feature space. Chen et al. [8] adopted a dual contrast learning framework to fuse radar Doppler features and video motion features. Feature-level fusion balances information retention and computational efficiency, and can adaptively adjust feature weights according to scenario changes.

Decision-level fusion strategy combines the recognition results of radar and video single-modal systems to obtain the final identity judgment [15]. It has strong robustness to single-modal failure but relies on the accuracy of single-modal decision results.

The three fusion strategies present significant trade-offs between computational cost and system latency. Data-level fusion entails processing raw heterogeneous data, yielding high computational complexity and maximum system latency. Feature-level fusion only integrates the extracted feature vectors, with moderate computational cost and latency, balancing information integrity and real-time performance. Decision-level fusion merely conducts posterior fusion on the recognition outcomes of single-modal systems and delivers minimal computational overhead and the lowest system latency, thus being better suited for application scenarios with stringent real-time requirements.

2.2.2. Alignment Methods

Alignment methods aim to eliminate spatio-temporal asynchrony and feature distribution differences between radar and video modalities [21], including spatio-temporal alignment and feature distribution alignment.

Spatio-temporal alignment ensures that the features of the two modalities correspond to the same target at the same time [6]. Time alignment is usually realized through sensor timestamp synchronization [7]. Spatial alignment uses calibration methods to map radar spatial coordinates (distance, angle) to video image coordinates to achieve spatial registration of targets. For example, Yang et al. proposed the BiCro method [22], which corrects noisy correspondences through bidirectional cross-modal similarity consistency to improve alignment accuracy.

Feature distribution alignment reduces the distribution gap between radar and video features. Metric learning is a common method. Wang et al. [20] learned a coupled feature space to map heterogeneous features to a unified space. Weiet al. [21] proposed a universal weighted metric learning method to adaptively adjust feature weights. Lionget al. [23] adopted deep coupled metric learning to realize nonlinear mapping of features. In addition, contrastive learning and knowledge distillation are also widely used in alignment tasks [12]. Chenet al. [8] transferred video facial identity knowledge to radar gait features through knowledge distillation to achieve semantic alignment.

Cross-modal matching algorithms [24] and recognition [25] complete identity verification based on fused and aligned features, which can be divided into machine learning algorithms and deep learning algorithms. Traditional machine learning relies on manually designed features, with lightweight models, fast training speed, good compatibility with small sample data, and strong interpretability. However, it has limited feature expression ability in complex scenarios and vulnerable generalization performance, and is suitable for classification and regression tasks with small data volume and clear features. Deep learning can automatically learn deep abstract features from massive data, with higher accuracy in complex data processing such as images and radar signals, but has large model parameters, a long training time, requires a large amount of labeled data, poor interpretability, and is prone to overfitting. Overall, machine learning is more advantageous in small-sample and low-dimensional scenarios; deep learning has significantly better feature extraction and recognition performance in high-dimensional and nonlinear data tasks such as millimeter-wave radar and video fusion.

2.3.1. Machine Learning Algorithms

Machine learning algorithms mainly use traditional classification and matching methods to process fused features [20]. Common algorithms include Dynamic Time Warping (DTW), K-Nearest Neighbor (KNN), Hidden Markov Model (HMM), and Random Forests.

DTW is used for matching time-series features such as gaits and motion trajectories, calculating the similarity between test sequences and reference templates through dynamic programming, suitable for radar motion feature matching. KNN performs identity classification based on feature similarity [9]. For example, Guo et al. [2] used KNN to classify fused vital sign features with an accuracy of over 85%. HMM models the temporal dependence of features, suitable for dynamic gesture and gait-based recognition [10]. Random Forest performs classification by integrating multiple decision trees and can handle high-dimensional fused features [11]. However, machine learning algorithms have a limited ability to process complex heterogeneous features and perform worse than deep learning algorithms on large-scale datasets.

2.3.2. Deep Learning Algorithms

Deep learning algorithms have powerful feature learning and pattern recognition capabilities, making them the mainstream algorithms in current fusion identification [7][12]. Common algorithms include Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), and Transformer.

CNN is used to extract spatial features from voxelized point clouds and multimodal feature maps [14]. For example, Shiet al. [4] used a deep CNN to fuse radar time-Doppler spectrograms and video gait energy images. Luoet al. [6] used CNN to extract spatial features of radar point clouds. LSTM is suitable for processing time-series features such as gait sequences and radar signal sequences. Shanet al. [12] used LSTM to model the temporal dependence of gait features to improve recognition robustness. Transformer captures global feature dependencies through a self-attention mechanism, suitable for cross-modal feature matching [18]. Chen et al. [8] used Transformer-based networks for temporal modeling and instance discrimination to enhance the discriminative ability of fused features.

Some innovative algorithms were proposed for these applications. Yaoet al. [25] proposed the semi-supervised recognition system mmSignature, achieving 96.3% accuracy with a small amount of labeled data. Shao et al. [18] realized fast action recognition based on point cloud sequence learning with inference time less than 0.2 ms. Deep learning algorithms can also be combined with multi-task learning, transfer learning, and other technologies to improve performance. For example, Shan et al. [12] used knowledge distillation for transfer learning, and Deng et al. [26] used synthetic data to enhance model generalization ability.

In summary, deep learning delivers superior performance in high-dimensional, heterogeneous cross-modal feature learning and complex-scenario recognition. By contrast, traditional machine learning models still present more prominent practical advantages for resource-constrained edge computing devices, few-shot annotation scenarios, and low-latency lightweight deployment, owing to their lightweight architecture, low computational cost, and fast inference speed.

This application aims to achieve real-time person identification and pedestrian re-identification in public places such as shopping malls, stations, and communities [27], ensure public safety, and provide cross-environment identity confirmation and trajectory tracking of personnel [6]. The core requirements are high precision and strong robustness to complex environments.

Caoet al. [3] proposed a cross vision-RF gait re-identification system based on low-cost RGB-D cameras and millimeter-wave radar, fusing gait motion features extracted by radar and visual gait features obtained by cameras, achieving about 92.5% top-1 accuracy and 97.5% top-5 accuracy among 56 volunteers, and stably identifying targets even in multi-person scenarios. Liu et al. [7] designed the Mission system, which detects targets through radar in camera-restricted areas and identifies person images from the camera network, achieving 85% top-1 accuracy and 90% top-5 accuracy among 58 volunteers, suitable for security monitoring scenarios.

The core challenge of intelligent security monitoring is to cope with environmental interference such as crowd density, occlusion, and illumination changes. Future research needs to focus on improving the real-time performance and multi-target discrimination ability of the system.

This application combines person identification with human-computer interaction, realizing identity authentication while completing gesture and action control, and is applied to smart homes, smart vehicles, and other scenarios. The core requirements are fast response and non-contact operation. Janakarajet al. [28] proposed the STAR system, realizing simultaneous tracking and recognition of persons using millimeter-wave radar and deep learning, fusing radar tracking information and video identity features, achieving real-time recognition in human-computer interaction scenarios with high stability. Huanget al. [11] proposed the RPCRS system that extracted human activity characteristics from millimeter-wave radar signals and combined them with video identity verification to achieve safe human-computer interaction, with recognition accuracy over 90%. The MatchAnything method, proposed by He et al. [29], is capable of matching diverse image data, making it applicable for cross-modal behavior recognition, where it can also be utilized to process radar signals to enhance recognition precision. Yaoet al.’s mmSignature system [30] realizes identity verification for interactive scenarios such as device unlocking and personalized services through semi-supervised learning, with an accuracy of 96.3%.

The core challenge of human-computer interaction identity authentication is to balance real-time performance and accuracy. Future research needs to design lightweight fusion models to reduce computational delays.

This application aims to continuously maintain identity and stably identify multiple moving personnel in complex and dense scenarios such as stations, squares, and parks. The core requirement is to distinguish different personnel identities and maintain recognition consistency even under multi-person interference, occlusion, and staggered movement. Luo et al. [6] adopted the CNN-BiGRU architecture to fuse radar point clouds and video joint information, realizing multi-target skeletal positioning and identity discrimination, with a multi-target identity identification accuracy of 89%. Caoet al. [3] fused radar gait features and video visual features, and can stably match target identities even in multi-person parallel movement scenarios, effectively avoiding target confusion and identity jumps. Shao et al. [18] realized real-time multi-target identity tracking and discrimination based on rapid modeling of radar point cloud sequences, maintaining high frame rate and identity consistency in complex crowd scenarios.

The core challenge of multi-target tracking identity identification is feature confusion and identity association errors caused by occlusion interference and target staggering. Future optimization of the multi-target association mechanism and cross-modal feature matching strategy is needed to improve the stability of identity identification in dense scenarios.

Table 3 briefly summarizes several mainstream applications, listing the goals and key elements of four typical application categories.

Table 3. Key features of four typical application categories.

Most current research experiments are conducted in laboratory environments with less interference [2][7]. However, in practical applications, the system is often affected by factors such as crowd density, occlusion, illumination changes, and electromagnetic interference [1]. For example, in dense crowd scenarios, radar signals are easily mixed with echoes from multiple targets, and video images are severely occluded, leading to a significant decrease in recognition accuracy [6]. In addition, severe weather such as rain and fog will attenuate radar signals and blur video images, further affecting system performance [5].

Although existing alignment methods have made progress, there are still gaps in spatio-temporal synchronization and feature distribution consistency [20][23]. Spatio-temporal asynchrony between radar and video sensors may lead to feature misalignment of the same target [6]. Feature distribution differences caused by modal heterogeneity [29] make it difficult to fully integrate complementary information, limiting the improvement of recognition accuracy [12]. Especially in dynamic scenarios, target movement will exacerbate alignment difficulties [8].

The collection and annotation of millimeter-wave radar and video fusion data are time-consuming and labor-intensive. Cross-scenario data collection and annotation are costly and privacy-sensitive, making it difficult to construct large-scale standardized datasets [26]. Currently, publicly available millimeter-wave radar and video fusion datasets for person identification are extremely scarce. Two representative datasets are as follows. The mmBody dataset provides synchronized millimeter-wave radar point clouds and RGBD video data, covering 20 subjects, 100 motions, and 7 scenes, serving as a commonly used multimodal benchmark for human perception [31]. Such public datasets generally suffer from limitations, including a small number of subjects, single-scene settings, and insufficient sample sizes, which still cannot meet the requirements of large-scale cross-scenario model training. Radar data require professional equipment and technicians for collection, and video data involve privacy issues, making it difficult to build large-scale annotated datasets [16]. Although some studies use synthetic data to supplement training samples [26], there is a distribution gap between synthetic data and real data, affecting model generalization ability [30].

Although millimeter-wave radar data is privacy-preserving and avoids direct collection of sensitive human visual information, fusion of radar data and video data forms a multimodal data linkage that can precisely associate individual identities, thereby introducing new privacy leakage and ethical risks. Cross-modal data association and matching may break through the privacy protection boundaries of a single modality. Without sufficient authorization, this can easily lead to the abuse of identity information and continuous tracking, violating the principles of data minimization and privacy compliance. This represents a critical ethical and compliance challenge that must be addressed in the practical deployment of fusion technologies.

To improve anti-interference ability, it is important to optimize feature extraction algorithms [3]. For example, we may use attention mechanisms to focus on extracting “illumination-independent, angle-independent, occlusion-robust” identity features, and use attention mechanisms, domain adaptation, and data enhancement to improve cross-environment generalization ability and suppress environmental noise [4][8]. We also design a multi-view sensor layout to reduce occlusion impact [32]. In addition, we may enhance model robustness through domain adaptation and data enhancement technologies [26].

We may develop more accurate spatio-temporal calibration methods to achieve millisecond-level time synchronization and pixel-level spatial registration [6]. At the same time, we can explore advanced feature alignment technologies such as cross-modal contrastive learning and generative adversarial networks to reduce feature distribution differences [12]. For dynamic scenarios, it is important to design adaptive alignment strategies that can be adjusted in real time according to target movement [8].

High-quality synthetic data may be generated to supplement real data [26] to reduce dependence on large-scale annotated data and achieve rapid adaptation to new locations and environments with a small number of samples. For example, [33] optimizes the video-to-radar conversion model to improve the authenticity of synthetic radar data [16]. We may use federated learning to achieve data sharing under privacy protection, expand the training data scale, and explore semi-supervised and unsupervised learning algorithms to reduce dependence on annotated data [29].

Unified feature space construction is a crucial research direction, as it can map data from multiple modalities into a common feature space. As a result, it reduces the discrepancies among different modalities, enables more diverse applications, and allows the use of more network models. Unified feature space construction can rely on modern architectures, including cross-modal Transformers, dual-branch contrastive learning frameworks, and deep cross-modal metric learning networks. It realizes global cross-modal feature interaction via self-attention mechanisms and narrows the feature distance of homologous targets through contrastive learning, ultimately achieving precise alignment and efficient representation of heterogeneous radar and video features in a unified latent space. This serves as the core technical pathway to address modal heterogeneity.