Many datasets have been constructed in the field of face-mask detection problems. Those datasets can be categorized into real-world face mask datasets [7] [8] and simulated generated face mask datasets [9] [10] . By manually analyzing the classes in each dataset and mapping them to “with mask”, “poor mask”, “none mask”, and others to observe the distribution of the number of images within classes in each dataset. The other classes indicate that the class in this dataset cannot be mapped to one of the three classes defined in this paper.

The MAFA [7] dataset contains public web images with a minimum edge length of 80 pixels while removing images containing obscure faces, resulting in 30,811 images, each of which contained at least one face with a mask on.

The AIZOO [11] dataset selected 3894 images from the WIDER FACE [12] dataset and 4064 images from the MAFA dataset for a total of 7959 images. By re-labelling those images, the AIZOO dataset contains a total of 16,636 images with faces in it: including two classes “with mask” and “none mask”.

The MAFA-FMD [8] dataset selected 50% of the images from the AIZOO dataset and 50% of the MAFA images. They re-annotated the selected images into the same three classes of annotations as in this paper.

The NFM [9] dataset extracts 18,088 video frames from 17 street videos and obtains 581,108 faces, which contain many tiny and obscured faces. This dataset has a strong practical significance which can distinguish between the three types of classes: “masked”, “unmasked”, and “unknown”. In this article, “masked”, and “unmasked” map to “with mask” and “none mask”.

The MaskedFace-Net (MF-Net) [10] dataset selected images from the Flickr- Faces-HQ3 [13] dataset, which is constructed by combining unmasked faces and taking a generative adversarial network to generate images of faces that wear masks correctly and incorrectly.

The RMFRD [14] dataset contains only 5000 images of faces wearing masks and 90,000 images of faces not wearing masks, which are selected from Internet resources.

Other than using real-world data images, computer generated data are also contributing to this field. The SMFRD [14] dataset generates face images with simulated masks for the LFW [15] and WebFace [16] datasets by using simulated mask-wearing generation software.

Table 1 shows the detailed number of face annotations and the number of images within the dataset. The short-dashed line indicates no data of this category exists in the dataset. If the number of Totals is greater than the sum of the three classes, it means that the dataset has data from other classes. Furthermore, the number of Totals and the number of Images is the same means that all the images in this dataset contain only one face.

Up to now, there are fewer datasets with balanced samples containing three cases of wearing masks at the same time. For example, the number of samples with incorrect masks in MAFA-FMD is 20 times smaller from the other two samples, which will make it difficult for the algorithm to learn the feature information of the label sample.

In this paper, researchers construct two multi-category, multi-scene, multi- difficulty mask datasets. LMFD [17] is a raw image dataset containing a total of 8232 images with only mask labels inside each picture provided. MFD [17] is a new dataset composed of LMFD after data enhancement methods, which includes both image data before and after the image enhancement techniques. These datasets allow many researchers to focus their research efforts on algorithm innovation and model deployment, contributing to facilitating the application of

face mask detection algorithms. The LMFD and MFD datasets contains the seven datasets mentioned above, real-world face mask images publicly available on the Internet, and the simulated generated face mask images.

The resolution of images in the MFD and LMFD datasets is not the same. The resolution of squared images is 320 px × 320 px. Other rectangular images are mainly within 400 px in width and height. 4086 images are used to train the model under LMFD dataset, and 55,905 images are used to train the model under MFD dataset.

Images have properties including but not limited to width and height, resolution, color channels, contrast, brightness, light source properties, saturation, white balance, sharpening, exposure level, etc. Convolutional neural networks are more concerned with information about the absolute properties of a feature in a picture and are not sensitive to the relative relationship with the background or other features. Artificial image enhancement of images, such as rotating the angle, zooming in and out, panning and flipping, imitating perspective, changing saturation, contrast, brightness, sharpening, exposure, etc., can be effective in increasing the number of images and feature information [21] .

In this paper, when training the model, in order to make the model easily and comfortably detect pictures with different scenes, different clarity, different facial and mask features, five kinds of data enhancement methods are taken, and each picture is expanded 3 times by taking five data enhancement methods in turn. Finally, each image can be expanded 15 times, and the method significantly reduces the difference in data volume between samples. The experiments in Section 5 show that the method has a more obvious positive effect on the improvement of accuracy.

The ColorJitter [22] method randomly changes the brightness, contrast, saturation, and hue of the image to simulate different ambient light, the special color of the mask, etc. The GaussianBlur [23] method randomly selects a Gaussian blur to blur the image to simulate camera shake during shooting or high-speed movement of the subject. The Sharpness [24] method randomly adjusts the sharpness of the image to simulate the image taken under different sharpness conditions. The Posterize [25] method post-processes the image by reducing the number of bits in each color channel. The Solarize [26] method exposes the image by inverting all pixel values above a threshold value.

Examples of data augmentation results are shown in Figure 3 for images containing incorrect labels for wearing masks. After performing data enhancement, the dataset can handle the problem of sample imbalance.

Table 2 shows the number of labels for the LMFD and the MFD dataset which we construct for future researchers. When aiming for fast training and deployment, LMFD would be a better choice.

Experiments (Section 5) show that using LMFD can significantly improve the

detection accuracy of the algorithm and obtain an excellent model with a high yield at the cost of low consumption. And by using MFD, detection performance is good under the condition of blurred faces, dark environments, etc. Besides, both datasets have a balanced distribution of labels which can affect the stability of the model performance.

The convolutional kernel in the convolutional layer determines the perceptual field size of the feature map. The following problems exist in the traditional deep convolutional neural network:

1) The representation feature space obtained from the up-sampling and pooling layers does not change with different viewpoints, thus the model is not robust enough for handling situations with different viewing angles.

2) The internal data structure is lost; the spatial hierarchical information is lost.

3) It is hard to learn the inconspicuous features from smaller objects. Therefore, the detection accuracy of people far away from the cameras is low.

In order to solve the above issues, we use the Hybrid dilation convolution method in the mask detection network. The Hybrid dilation convolution is usually applied to the field of semantic segmentation after its initial proposal, and in this paper, we apply hybrid dilation convolution [27] to the muzzle detection subtask in target detection, which is found to be better than traditional convolution through the experiments discussed in section 5. By using hybrid dilation convolution, small object’s features can be more easily learned with the use of dilation factor. When the dilation factor value is 1, 2, and 3, the convolution kernel shape is shown in Figure 4, and the red points represent pixels that are involved in the convolution calculation. It is worth mentioning that the dilation factor r is equal to 1 in the case of traditional convolution.

When r is equal to 3, and the number of pixels involved in the calculation will be 9, the number of pixels ignored due to the expansion will increase to 40, and the information loss rate is 40/(40 + 9) = 82%. It is obvious that the larger the expansion factor, the larger the information loss rate. Therefore, it is more balanced and reasonable to use a mixture of convolution kernels with different dilation factors.

Besides, it is important to design a reasonable expansion rate by assuming that we have N convolution layers, the size of the convolution kernel is K × K , and the hybrid expansion rate is [ r 1 , ⋯ , r i , ⋯ , r n ] . The goal of the dilation convolution process is to make a series of convolution operations with continuous perceptual fields and no missing image information, and we define the “maximum distance between two non-zero values” as M i . The design requirements for the inflation rate need to satisfy condition (1).

M i = max [ M i + 1 − 2 r i , M i + 1 − 2 ( M i + 1 − r i ) , r i ] (1)

where:

M 2 ≤ K (2)

M n = r n (3)

r i is the expansion factor of layer i , and M i is the maximum expansion rate of layer i . The design principle is that the conventional number of the expansion rate of each layer cannot be greater than 1.

Figure 5 shows the perceptual field area size and the number of times each pixel point is involved in the convolution calculation when the convolution kernel size is 3 × 3 for the conventional convolution and the hybrid dilation convolution rate of [1, 2, 3]. When the convolution kernel size is 3 × 3, the perceptual field area size of conventional convolution is 7 × 7. When the hybrid inflated convolution method is used, the hybrid inflated convolution rate of [1, 2, 3] increases the perceptual field area to 13 × 13, which increases the perceptual field area by 344.9% and preserves more image feature information.

Another benefit of using hybrid dilation convolution is that it can use arbitrary expansion rates in the process, thus it can naturally expand the receptive domain of the network without adding additional modules, which is important for identifying relatively large objects. The key difference between the hybrid inflationary convolution approach and the Atrous Space Pyramid Pool (ASPP) module [28] or the contextual aggregation module [29] [30] is that the hybrid dilation convolution approach utilizes an expansion factor with a common factor no greater than 1. In addition, hybrid dilation convolution can naturally integrate with the original layers of the network without adding additional modules.

Faster R-CNN is a deep convolutional network used for object detection that appears to the user as a single, end-to-end, unified network [31] . The input image is extracted by a convolutional neural network for image features. The region proposal network generates proposal frames, and the proposal frames are

projected onto the feature map to obtain the corresponding feature matrices. Each feature matrix is scaled to a 7 × 7 sized feature map through a region of interest pooling layer, followed by spreading the feature map through a series of fully connected layers to obtain the prediction results [32] . We then train the Faster R-CNN target detection model and use the detection result accuracy as the baseline model, focusing on the data augmentation application effect, and the integration application effect of hybrid expanded convolution and residual network. Figure 6 shows the object detection algorithm Faster R-CNN.

The residual network [33] proposed by KaiMing He et al. in 2015 still has a wide influence today, in which the model with ResNet as the main backbone network continues to set new records for major datasets.

The residual network follows the complete convolutional layer design of VGG [34] . It starts with two convolutional layers with the same number of output channels. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. Then we skip these 2 convolutional operations through the cross-layer datapath and add the input directly before the final ReLU activation function. Such a design requires that the outputs of the two convolutional layers have the same shape as the inputs so that they can be added together. If you want to change the number of channels, you need to introduce an additional convolutional layer to transform the input into the desired shape before doing the summation operation.

ResNet50 uses four modules consisting of residual blocks, each using several residual blocks with the same number of output channels. The first module has the same number of channels as the input channels. Since a maximum convergence

layer with a step size of 2 has been used previously, there is no need to reduce the height and width. Each subsequent module doubles the number of channels of the previous module in the first residual block and halves the height and width. Figure 7 shows the network structure of the deep learning algorithm called RHF.

The momentum method [35] is an accelerated gradient descent method that simulates momentum in physics, accumulating velocity vectors, and for a given objective function f ( θ ) to be minimized, the momentum method is given by:

v t + 1 = μ v t − ε ∇ f ( θ t ) + λ θ t (4)

θ t + 1 = θ t + v t + 1 (5)

where ε > 0 is the learning rate, μ ∈ [ 0 , 1 ] is the momentum coefficient and ∇ f ( θ t ) is the gradient at θ t .

In the model training, various combinations of parameters are tested. We finally used the following parameters in our model based on the size of the device memory and the magnitude of the model parameters, taking into account the model training time and the model training effect. Higher learning rate will cause the model never converge, and the model cannot learn characteristics of the datasets at a lower learning rate.

The models with ResNet50 and RHF are developed on the Pytorch deep learning library [36] . The datasets used are LMFD and MFD. The model was trained for 20 epochs with a stochastic gradient descent (SGD) algorithm of learning rate ε = 0.005 and momentum μ = 0.9. The parameters used in the experiments include a batch size of 8, and weight decay λ is 0.0005. The learning rate decreases by 33% every three generations. An NVIDIA GeForce RTX 3090 GPU was employed.

COCO metrics were used to validate the model effects. The mAP and IoU are

used in this paper. The mAP is the average accuracy, and IoU is the intersection over union of prediction frames to object frames. The threshold of IoU is set to 0.6 because there are many small objects in datasets.

The performance and training time cost of ResNet50 and RHF models on the LMFD and the MFD dataset be shown in Table 3. Through observational studies the detection accuracy and the model on each label and the average precision of the detection on the dataset, it can be found that model training on RestNet50 costs more time than model training on RHF. The reduction in training time will allow the model to be trained faster on large datasets.

It took 1.5769 hours to train the ResNet50 model on LMFD, and only 1.3743 hours to train the RHF model. It took 23.5589 hours to train ResNet50 model on MFD, and only 13.9982 hours to train RHF model on LMFD.

By using the MFD dataset to train the RHF model, RHF shows good performance on the completely new images that never appeared in the original dataset before. The model inference 200 new images on NVIDIA GPU3090. The average testing time of each image was 0.0424 seconds, and the real-time frame rate reached 23.6FPS.

The loss rate of the training process is shown in Figure 8. The mAP of the training process is shown in Figure 9. Results indicate that the RHF model’s loss rate starts to converge after the 15^th epoch. The ResNet50 model only starts to converge in MFD dataset after the 18^th epoch. The loss rate curve shows that the hybrid dilation convolution accelerates the convergence of the model.

Figure 10 shows the detection results in multiple complex environments of the four models trained in this paper. The Original Image indicates the images without any image processing and data enhancement. Due to the length of the displayed images, only the detection results of the three data enhancement methods in multiple scenarios are used. The “GaussianBlur” means the images use a size of 3 × 7 Gaussian convolution kernel to blur. The “ColorJitter” method uses the brightness jitter factor in [0.5, 1.5], the hue jitter factor in [−0.3, 0.3]. The “Solarize” method inverts the pixel values exceeding 108 in each color channel.