Pedestrian Re-identification is the task of associating observations of the same individual across a network of non-overlapping camera views [1] . Given a query image of a person of interest captured in one camera view, the objective is to retrieve all instances of the same person from a gallery set containing images from other camera views. It is fundamentally a matching problem, aiming to determine if two observations correspond to the same physical person despite variations in viewpoint, pose, illumination, occlusion, and background clutter. Early methods relied on hand-crafted features [8] , while modern approaches predominantly utilize deep learning to learn discriminative feature representations directly from data [3] [4] .

Multi-Modal Pedestrian ReID extends the traditional ReID paradigm by leveraging information from multiple sensing modalities to improve matching accuracy and robustness [5] . While visual data (RGB) is the most common modality, it can be limited under challenging conditions. Integrating complementary sensor data, such as depth maps (RGB-D), thermal imagery (RGB-T), or radio frequency signals (like mmWave radar), can provide additional cues to overcome the shortcomings of visual sensors alone. The core idea is that different modalities capture distinct aspects of a person or the environment, and their fusion can lead to a more comprehensive and resilient representation for matching [9] .

Understanding the individual strengths and weaknesses of visual cameras and mmWave radar is crucial for effective fusion.

The complementary nature of visual and mmWave data forms the core rationale for their fusion in pedestrian ReID. Cameras excel at capturing appearance details crucial for identification in good conditions, while mmWave radar, which persists even when visual data is degraded, provides robust geometric and velocity information.

Fusion aims to leverage these complementary strengths:

Compared with systems that rely on a single modality, this combination ensures that the application system is more robust and reliable in complex real-world environments [10] .

A typical mmWave-visual ReID system involves one or more cameras and mmWave radar sensors deployed to cover the area of interest.

This configuration facilitates calibration and ensures that both sensors observe the same scene region, simplifying data association. The diagram should also present the subsequent processing steps.

Temporal alignment of data streams from cameras and radars is critical for accurate fusion. Mismatched timestamps can lead to incorrect associations between visual features and radar points corresponding to the same pedestrian at a given moment. Common synchronization techniques include:

Spatial alignment, or calibration, is necessary to establish the geometric relationship between the coordinate systems of the camera(s) and the radar(s). This allows projecting radar points onto the image plane or transforming visual features into the radar’s coordinate system, enabling data association and fusion.

In multi-sensor systems, the collaboration between millimeter-wave radar and RGB cameras faces three core challenges: calibration, synchronization, and data association. Hardware-level synchronization can achieve microsecond-level precision but is costly, while software synchronization (NTP/PTP) requires motion compensation algorithms to correct the data misalignment caused by millisecond-level delays. In terms of external parameter calibration, traditional checkerboard calibration has high accuracy but is limited to static environments, whereas target-less calibration is flexible but relies on rich natural feature matching. In data association, the modal differences between the radar’s sparse point cloud and the camera’s dense image can easily lead to target mismatches, necessitating the fusion of probabilistic models and deep learning features to enhance consistency. In practical use, it is essential to balance cost and performance: first, use a calibration board for precise calibration and automatically adjust during operation; prioritize hardware synchronization, but if the budget is insufficient, resort to software synchronization with algorithm compensation. Ultimately, success depends on the collaboration between hardware and algorithms to adapt to environmental changes in real-time.

Raw data from cameras and radar require significant pre-processing before fusion.

Visual Data Pre-processing includes the following context:

mmWave Radar Data Pre-processing including the following context:

Effective pre-processing cleans the raw data, extracts relevant information (e.g. pedestrian bounding boxes, clustered radar points associated with pedestrians), and prepares it for subsequent fusion stages.

Data-level fusion involves merging raw or minimally processed data from mmWave radar and visual sensors before detection or identification. This approach leverages radar’s precise localization to generate regions of interest (ROI) in images, which are then processed for re-identification. Early methods relied on straightforward data integration, but recent advancements incorporate deep learning for enhanced performance.

One of the earliest works, Milch and Behrens [18] , proposed a method that uses a radar-generated target list to define the ROIs, and then verifies the pedestrian through visual profile analysis. This approach was computationally efficient but limited by the simplicity of visual features. Guo et al. [19] advanced this by introducing intra-frame clustering and inter-frame tracking, using radar data to filter noise and guide visual confirmation. Their method improved robustness in noisy environments.

More recently, Wang et al. [20] developed a three-layer fusion model integrating radar and monocular vision, using a visual attention mechanism to prioritize ROIs. This method enhanced detection in dynamic scenes. Similarly, Streubel and Yang [21] explored data-level fusion for indoor pedestrian tracking, projecting radar points onto stereo camera images. Their approach achieved high localization accuracy but required precise sensor calibration.

Data-level fusion excels in leveraging raw data complementarity but faces challenges in computational complexity and sensor synchronization. Recent studies, such as Plascencia et al. [22] , have begun integrating deep learning into process fused data, improving scalability. Table 1 compares key data-level fusion methods.

Feature-level fusion extracts features from radar and visual data separately, and then combines them for re-identification. This approach has gained prominence with deep learning, enabling models to learn complex feature representations and fusion strategies dynamically.

Nobis et al. [23] proposed a deep learning architecture, which fuses projected radar data with camera images within network layers. This method improved 2D detection accuracy by learning optimal fusion levels. Plascencia et al. [22] extended this concept, transforming radar and lidar data into 2D grayscale images and fusing them with RGB images using a SegNet-based network. Their approach enhanced pedestrian detection in cluttered environments.

In addition to that, attention mechanisms have further refined feature-level fusion. Li et al. [24] introduced an attention-based network for pedestrian liveness detection, combining radar cross-section (RCS) features with visual data to distinguish real pedestrians from static images. Similarly, Yu et al. [25] proposed a dual cross-attention module (DCAM) for feature fusion, initially for vehicle detection but adaptable to pedestrians. Liu et al. [26] developed a multi-modal network integrating radar gait features with visual appearance, achieving robust re-identification in occluded scenes.

Feature-level fusion benefits from deep learning’s ability to model complex relationships but requires substantial training data and computational resources. Table 2 summarizes key methods.

Decision-level fusion involves independent detection or identification by each sensor, followed by result integration. This modular approach is robust to sensor failures and suitable for real-time applications.

Yang et al. [27] proposed a decision-level fusion method using an unscented Kalman filter (UKF) for radar data and YOLOv5 with DeepSORT for visual tracking, matching targets in polar coordinates. This method achieved high tracking precision. An earlier study [28] introduced a multi-sensor tracking algorithm with back-projection and multi-hypothesis association, enhancing trajectory accuracy. Zhao et al. [29] focused on nighttime detection, combining infrared vision and radar with an improved YOLOv5 and extended Kalman filter.

Additional studies include Graves et al. [30] , who used decision-level fusion for pedestrian collision warning, integrating radar localization with visual classification, and Zhu et al. [31] , who developed a track-to-track fusion method for multi-pedestrian tracking. These methods prioritize simplicity and robustness but may miss early-stage data synergies. Table 3 compares key approaches.

The evolution of mmWave radar and visual fusion for pedestrian ReID reflects advancements in sensor technology and algorithms. Early methods (2000s) focused on data-level fusion, using radar to guide visual processing in simple scenarios. The 2010s saw decision-level fusion gain traction for its robustness, as seen in collision warning systems. Since 2015, feature-level fusion has dominated, driven by deep learning’s ability to model complex multi-modal relationships. Future directions may involve end-to-end deep learning models and efficient handling of sparse radar data, addressing real-time constraints in autonomous systems.

To better illustrate the fusion approach, we present a detailed explanation of several representative re-identification methods. This will facilitate a more comprehensive understanding of the advantages offered by millimeter-wave radar and vision fusion for re-identification tasks.

Zheng et al. [1] proposed the pedestrian alignment network (PAN) that utilizes two convolutional branches (the base branch and the alignment branch) and an affine estimation branch to simultaneously address pedestrian alignment and recognition issues. The base branch deploys a pre-trained ResNet-50 model on ImageNet and removes the final fully connected (FC) layer. The alignment branch consists of 3 ResBlocks and 1 average pooling layer, also adding an FC layer to predict multiclass probabilities. The affine estimation branch receives two activated input tensors from the base branch. The Res4 Feature Maps contain shallow feature maps of the original image, reflecting local pattern information; the Res2 Feature Maps are closer to the classification layer and encode attention and semantic cues for pedestrian recognition.

Zheng et al. [2] proposed a deep learning model that combines the advantages of verification models and recognition models. Establishing relationships through pairwise comparisons, such as partial matching and contrastive loss, is performed. Contrastive loss directly computes the Euclidean distance between two embeddings. In the recognition model, there exists an implicit relationship between the learned embeddings constructed using cross-entropy loss. The cross-entropy loss can be used. When the directions of the embedding vectors are similar, the network converges, thereby maintaining the similarity of the embeddings. The proposed model simultaneously utilizes both types of loss functions and benefits from pre-training on ImageNet, thereby overcoming the limitations of a single model.

In the performance evaluation of data-level fusion methods, computational complexity and real-time indicators have significant advantages. Regarding the issue of computational complexity, existing research mainly focuses on two dimensions: noise suppression and accuracy optimization. For example, Yu et al. [25] utilized modules to reduce technical complexity, providing a reference technical path for complexity control. In terms of real-time performance, most studies present a processing delay of less than 50 ms, which is largely attributed to the inherent characteristics of data-level fusion due to operating directly at the raw data layer, thus avoiding the time overhead associated with higher-level processing, such as feature extraction and decision reasoning.

Zheng et al. [1] adopted a multi-branch architecture (base branch, aligned branch, affine estimated branch), which has a higher complexity, resulting in a surge in memory and computation, and its multi-task joint optimization (recognition, alignment, and feature learning) further increases the training complexity. In addition, due to high-resolution feature map processing and multi-branch parallel computing, PAN has higher hardware requirements, resulting in large inference delays. The model proposed by Zheng et al. [2] fused the comparison loss of the verification model (based on Euclidean distance) and the cross-entropy loss of the recognition model, although it is manifested in the complexity of dual-objective optimization and high-dimensional embedding calculation. The transfer learning of the ImageNet pre-trained model (such as ResNet) reduces the training cost. In general, the model proposed by Zheng et al. [2] has made a breakthrough in complexity and real-time, which can be used as a good reference for similar processing of complexity and real-time in this project to achieve more accurate pedestrian re-identification.

Although the constructed models of the two are different, both have improved the accuracy of pedestrian re-identification, providing certain insights and guidance for the fusion re-identification system.

Nighttime or low-light conditions challenge visual sensors, but radar’s penetration ability ensures reliability. Zhao et al. [29] proposed a decision-level fusion framework using infrared vision and radar, achieving high accuracy in dark environments. Similarly, Zhang et al. [11] combined thermal imaging with radar for nighttime ReID, improving robustness.

Real-time pedestrian tracking supports path planning and collision avoidance. Yang et al. [27] developed a decision-level fusion method using UKF and DeepSORT, ensuring precise trajectory tracking. Zhu et al. [31] introduced track-to-track fusion for multi-pedestrian scenarios, handling occlusions effectively. These methods enable vehicles to anticipate pedestrian movements.

Fusion-based ReID methods significantly enhance safety and reliability in autonomous systems. Studies like [28] demonstrate reduced false positives compared to single-sensor approaches. Future advancements may leverage datasets and focus on real-time processing and adverse weather performance.

1) Lack of Dedicated Benchmarks: The absence of large-scale, diverse, and publicly available datasets specifically designed for mmWave-visual pedestrian ReID (with ground-truth IDs across non-overlapping views under various conditions) is arguably the biggest obstacle. This impedes standardized evaluation, fair comparison of methods, and the training of data-hungry deep learning models.

2) Data Heterogeneity and Representation: Effectively fusing the sparse, geometric, and point cloud data from radar with the dense, semantic, and appearance-rich pixel data from cameras remains fundamentally challenging. Finding optimal representations for radar data that facilitate effective fusion with visual features is crucial.

3) Radar Data Quality and Interpretation: mmWave radar data can be noisy, suffer from multipath reflections (ghost targets), and have low angular resolution compared to cameras. Sparsity makes it difficult to infer detailed shapes or associate points reliably to specific body parts for fine-grained gait analysis or ReID, especially in crowds. Extracting discriminative features solely from sparse radar points is non-trivial.

4) Complexity vs. Real-Time Performance: Sophisticated hybrid fusion models (e.g. using transformers or complex attention mechanisms) often achieve better performance but come with high computational costs, making real-time deployment on resource-constrained platforms (like robots or edge devices) challenging.

5) Generalization and Domain Adaptation: Models trained on data from one specific sensor setup, environment, or weather condition may not generalize well to others (domain shift). Variability in radar hardware, camera types, environmental clutter, and pedestrian densities poses significant generalization challenges.

1) Benchmark Dataset Development: Creating large-scale, diverse mmWave-visual pedestrian ReID datasets covering various environments (indoor/outdoor), weather conditions, pedestrian densities, and sensor viewpoints is paramount. Including annotations for persistent IDs across non-overlapping views is essential.

2) Advanced Fusion Architectures: Exploring novel deep learning architectures tailored for heterogeneous sensor fusion. This includes investigating more advanced transformer variants, graph neural networks specifically designed for radar point cloud structure and radar-visual interaction, and perhaps integrating neural rendering techniques to bridge the modality gap.

3) Exploiting Richer Radar Information: Moving beyond basic point clouds (x, y, z, v). Research into effectively incorporating radar micro-Doppler signatures for gait recognition utilizing the full Range-Azimuth-Elevation-Doppler radar tensor, or learning discriminative features directly from raw radar ADC data holds significant promise.

4) Real-Time Optimization: Studying model compression, quantization, knowledge distillation, and efficient network architectures to enable real-time execution of complex fusion models on edge devices.

5) Cross-Modal Adaptation: Developing geometry-aware fusion models with 3D pose estimation to overcome the core challenge of view-invariant matching in non-overlapping sensor configurations, while addressing domain shift through adversarial feature representation.

Overcoming these challenges and pursuing these research directions will be crucial for unlocking the full potential of mmWave-visual fusion and realizing truly robust and reliable pedestrian ReID systems for real-world applications.