The detailed flow of single-frame object recognition and inter-frame registration is shown in Figure 3 . Single-frame image object recognition is used to obtain the class, location and confidence information of potential objects. In the technical framework proposed herein, all object recognition algorithms that provide the above information can be used to recognize objects from single-frame images. Considering that the great success achieved by YOLOv3 algorithm in the field of object recognition in recent years, YOLOv3 is used herein to accomplish this step.

The step of inter-frame object registration is taken to establish the association among potential objects recognized at different moments. DeepSort, a technical framework commonly used for object tracking, is employed herein to accomplish this step. By first resorting to recursive Kalman filtering to predict the positions of all potential objects that have appeared before in the current frame (frame F), then matching the inter-frame objects by considering both motion information and appearance information. The formula for calculating the matching degree of motion information $d^{\left(1\right)}\left(u,v\right)$ is:

$d^{\left(1\right)}\left(u,v\right)={\left(d_u-y_v\right)}^T{S}_{uv}^{-1}\left(d_u-y_v\right)$ (1)

Wherein $d_u$ is the position of detection box of the u-th object identified by YOLOv3 in the image of the F-th frame, and $y_v$ is the very place that the object whose ID is $I_v$ and whose frames have appeared is in the image of the F-th frame as predicted by the recursive Kalman filter algorithm, and $S_{uv}$ is the covariance matrix between $d_u$ and $y_v$.

The formula for calculating the matching degree $d^{\left(2\right)}\left(u,v\right)$ of appearance information is:

$d^{\left(2\right)}\left(u,v\right)=\min \left\{1-r_u^T{r}_k^{\left(v\right)}|r_k^{\left(v\right)}\in R_v\right\}$ (2)

Wherein $r_u$ is the appearance feature vector extracted by $d_u$, and $r_v$ is the set of those q times appearance feature vectors of the object whose ID is $I_v$ that have appeared (q ≤ 100; 30 is assigned to it in this paper). DeepSort sets the linear weighted result $C_{u,v}$ from $d^{\left(1\right)}\left(u,v\right)$ and $d^{\left(2\right)}\left(u,v\right)$ to be the measure of how well the motion information matches the appearance information:

$C_{u,v}=\lambda d^{\left(1\right)}\left(u,v\right)+(1-\lambda )d^{\left(2\right)}\left(u,v\right)$ (3)

Wherein $\lambda$ is a hyperparameter that controls the influence of the two metrics on the results (0.01 is assigned to it in this paper). If $C_{u,v}$ falls within the threshold range (0.32 is assigned to it in this paper)where the two match each other, it is considered that the u-th potential object identified in the F-th frame is the very potential object with ID $I_v$ that has appeared before. Motion information and appearance information match successfully. The u-th potential object is identified after ID $I_v$ is assigned to the F-th frame. Otherwise, a further step should be taken to judge whether the object matches by the indicator Intersection Over Union (IOU) between $d_u$ and $y_v$.

The object detection and registration process of the method proposed herein is shown in Figure 3 . It should be pointed out that yet there is a combinatorial optimization problem in the one-to-one registration of the v potential objects that have appeared and the u objects identified in the F-th frame. The technical framework of DeepSort works by the Hungarian algorithm to recursively sort out the optimal match between the v potential objects that have appeared and the u object identified in the F-th frame.

Real objects usually appear simultaneously in multi-frame images captured consecutively. The method proposed herein fuses the single-frame detection confidences of the same potential object at multiple moments after object recognition in a single-frame image is registered with inter-frame object, and outputs the corrected new confidences. If after detection and registration, the confidences of object recognition for a potential object inconsecutive k frame images are $p_1$, $p_2$, …, $p_k$, respectively, then the confidence ${\hat{p}}_k$ obtained after information fusion and planning are:

${\hat{p}}_k=\{\begin{array}{c}{\beta }_k,{\beta }_k<1\\ 1\text{,}{\beta }_k\ge 1\end{array}$ (4)

Equation (4) ensures that the resulting confidence ${\hat{p}}_k$ is a constant between [0, 1]. And in the equation, ${\beta }_k$ is the calculated value for the multi-frame confidence fusion:

${\beta }_k=\frac{\left({\displaystyle \underset{i=1}{\overset{k}{\sum }}a{b}^{k-i}{p}_i}\right)f\left(x\right)}{{\displaystyle \underset{i=1}{\overset{k}{\sum }}a{b}^{k-i}}}$ (5)

Wherein a is the single-frame basic contribution; and b is the forgetting factor (b ∈ (0, 1)), whose function is to gradually reduce over time the weight of the DLOR result of a certain frame in the confidence fusion for the subsequent moments. $f\left(x\right)$ is a multiplication function, whose calculation formula is:

$f\left(x\right)=\{\begin{array}{l}1\text{,}x<z\hfill \\ \frac{c}{1+2e^{-d\left(x-z\right)}}\text{,}x\ge z\hfill \end{array}$ (6)

Wherein the variable x is the proportion of the images where YOLOv3 can detect an object.to a consecutive sequence of images captured since the first appearance of the object. If x is greater than or equal to z, the calculated value ${\beta }_k$ of the multi-frame confidence fusion will be greatly increased, otherwise it will stay as it is. c is the multiplication factor, and d, the multiplication adjustment factor. Parameters z, c, and d are valued as appropriate.

It is often that new objects appear in the process of object recognition in consecutive scenes. Let m be the minimum length of temporal sequence for the fusion of temporal sequence information under the method proposed herein (m is valued as appropriate). If the number of image frames collected for a new object from the time-series images in which it appears is less than m, it is obviously impossible to perform time-series information fusion by the method in Section 3.2. This problem happens to provide an interface for the method proposed herein to fuse scene priori information. The specific process to fuse scene priori information is shown in Figure 4 , where area A is the reasonable area delineated according to scene priori knowledge where a new object might appear for the first time, and area B is the unreasonable area where it might appear for the first time. The object recognition confidence of (m-k) moments missing from the fusion of temporal sequence information is set to be either the object recognition confidence of the current frame image if the new object appears in area A for the first time, or 0 and if it appears in area B for the first time. After the scene priori information is fused, the confidence for the new object can also be corrected by the method described in Section 3.2. And if the area where a new object appears for the first time does not match the scene priori information, the recognition confidence of the object will be significantly reduced.

For ease of understanding, scenes used in part 4 are taken as examples to illustrate how to the delineate areas based on the scene priori knowledge. As shown in Figure 5 , according to common sense, the central area in front of the observer and the areas on the left and right sides are reasonable areas where new object might appear for the first time in this scene and thus should be each delineated as area A, while the other area is the unreasonable area where new objects might appear for the first time and thus should be delineated as area B.

The configuration of the hardware platform for the object recognition algorithm in the experiment is: dual Nvidia GTX3080 graphics cards, and dual Intel(R) Xeon(R) Gold 6139M CPUs. The deep learning and information fusion algorithms are implemented using Pytorch (1.8.1) and CUDA (11.1) under the integrated development environment Pycharm (2020.2.3) in the Windows 10 operating system.

The information fusion object recognition method proposed herein this paper was first tested on public datasets. During the experiment, 80% of the samples were extracted from the “Urban object detection” dataset [29] for the training and testing of basic DLOR (the samples were divided into training set, validation set and test set in the proportion of 3:1:1). The method proposed herein is applicable to consecutive scenes only, but the “Urban object detection” dataset contains few consecutive scenes. Therefore, in the comparison of recognition accuracy between this method and basic DLOR, in addition to the two consecutive scenes taken from the “Urban object detection” data, four consecutive scenes were also extracted from the “BDD100K” dataset [30] . The scenes used are shown in Figure 6 , and the number of images, objects and types included in each scene, in Table 1 . The Areas A and B in each scene are divided according to the example in Section 3.3, and the parameter values for information fusion to be conducted by the method proposed herein are shown in Table 2 . Given 0.05 and 0.3 as the confidence thresholds to judge whether an object exists, the basic DLOR (YOLOv3) and the method proposed herein are compared and analyzed by the false positive rate and the miss rate, leading to the results shown in Table 3 and Table 4 .

It can be seen from Tables 1-4 that when the confidence threshold is set to 0.05, the performance of the method proposed herein is significantly better than that of basic DLOR in all the six scenes, to be specific, the false positive rate of various objects in each scene under the former algorithm is 1.98% lower than that under basic DLOR on average, and the miss rate, 9.18% lower than that under the latter. The object recognition of person in scene 4 by the method proposed herein shows the most significant performance improvement, as suggested by the miss rate and false positive rate reduced by 16% and 8%, respectively.

When the confidence threshold is set to 0.3, the missed detection rate performance of the method proposed herein is also significantly better than that of basic DLOR in all the six scenes, to be specific, the miss rate of various objects in each scene under the former algorithm is 8.6% lower than that under basic DLOR on average. The object recognition of traffic signal in scene 1 by the method proposed herein shows the most significant performance improvement, as suggested by the miss rate reduced by 13.2%. Regardless of whether the confidence threshold is 0.05 or 0.3, the two performance evaluation indicators in relation with the method proposed herein are not lower than those in relation with basic DLOR in all the scenes. The method proposed is a great improvement on the object recognition performance of the basic DLOR.

Figure 7(a) shows the object recognition result obtained by basic DLOR under the condition of threshold 0.05 in the seventh frame image in scene 2, and Figure 7(b) , the recognition result obtained by the method proposed herein under the same condition. It can be seen from the figures that due to the poor light conditions of the images, the Basic DLOR mistakenly recognizes a one-person object on the top of the car in the right of center of the image, which false positive recognition can be avoided by the method proposed herein.

Figure 8(a) shows the object recognition result obtained by Basic DLOR under the condition of threshold 0.3 in the fourth frame image in scene 4, and Figure 8(b) , the recognition result obtained by the method proposed herein under the same condition. It can be seen from the figures that due to the far distance, the Basic DLOR fails to recognize the car and the traffic signal in the middle of the image, which miss can be avoided by the method proposed herein.

Intelligent robot environment perception is one of the important applications of object recognition. In order to further verify the practicability of the method proposed, 1961 images from two production workshops were collected for this paper by reference to the working environment of industrial robots to build an industrial scene dataset for experiments. The dataset includes such two types of objects as machine tool and person. During the experiment, 90% of the samples were taken from the self-built industrial dataset for the training of the DLOR model and validation (the samples are split into the training set, the validation set and the test set in the proportion of 8:1:1), and two consecutive scenes with a large number of images were selected from the remaining images for the purpose of algorithm performance evaluation. The two scenes are shown in Figure 9 , among which, the first one includes 45 images and a total of 202 objects, and the second one, 45 images and a total of 189 objects.

Considering that in the images, some of the objects in this dataset were larger than those in the public dataset used in Section 4.2, the parameters W and R of the information fusion process in this experiment were increased to 540 and 155 respectively, while other parameters were the same as those in Section 4.2. After many attempts, it was found that both algorithms performed well when the confidence threshold was set to 0.3 on the dataset, hence this confidence threshold. The performance of Basic DLOR and our method is shown in Table 5 .

It can be seen from the table that there was no false positive in the two methods. This is because, limited by the research conditions, the appearance of the objects from the self-built dataset is less diversified than that from public dataset, and the difference between the objects in the training samples and the objects in the performance evaluation images is relatively small. The miss rate of our method is 7.1% lower on average than that of Basic DLOR. Figure 10(a) and Figure 10(b) show an example of reduced object miss rate obtained by our method in scene one, and Figure 10(c) and Figure 10(d) , scene two. Based on the experimental results in Section 4.2 and this Section, it is concluded that it is because of the expansion of information sources that the method proposed herein greatly improves the performance of object recognition algorithm, and that the quality of a single frame image becomes less influential on the recognition result.