Abu et al. 2020 [25] used SEVERSTAL dataset to predict surface defects in steel production using four deep learning techniques VGG16, MobileNet, Densenet121 and Resnet101. The dataset was pre-processed by using OpenCV to rescale images to 256 × 480 pixels while 50 epochs and 32 batch sizes were used for their models. The accuracy of their models is shown Table 3 .

The results showed that MobileNet yielded highest accuracy compared to others.

Akhyar et al. 2023 [22] investigated surface defects in steel manufacturing using SEVERSTAL steel dataset, NEU dataset and DACM dataset. They proposed a deep learning approach termed forceful steel defect detector FDD which is rooted in R-CNN with deformable convolution and deformable ROI pooling which integrate with the geometric shape of defects. Their models accuracy were measured in terms of average recall (AR) and mean average precision (mAP) and comparison were made ( Table 4 ).

In ML programming domain, dataset is necessary to build models that make predictions. The process involves inputting relevant dataset into algorithms that learn from the peculiar patters embedded in the data to make informed predictions [24] . Clearly, the development of effective models starts with data preparation [35] [36] . When relevant dataset for targeted investigation is acquired, the data is prepared through cleaning, transformation and making its ready for computational analysis. The manufacturing dataset acquired in this research was published by [37] . It was sourced from Severstal, a leading steel industry company located in Russia. Severstal created very large industrial data lake from production of flat sheet steel in order to monitor defects ( Figure 2 ).

The dataset is in three segments: train images (count = 12,568), test images (count = 5506) and train CSV (7095,3). The 12,568 images exist in four classes with various kinds of defects. Images with common defect are present in class 3 ( Figure 3 ).

The data preparation step involves resizing the images to match the fixed dimension of the input shape, creating arrays to hold the images and their corresponding class identifiers, and transforming the categorical labels into integers using LabelEncoder ( Figure 4 ). The images’ pixel intensities were also normalized to be within the range of 0 - 1. To handle overfitting and enable the model to learn from a more generalized set of features, data augmentation techniques were applied, including shifting, rotation, zooming, and flipping. Transforming the provided run-length encoded masks into binary masks for the segmentation task.

Machine learning systems are versatile tools capable of handling a variety of tasks which includes classification, regression, and segmentation [38] [39] . Classification tasks are common, and they involve categorization of data into predefined classes based on input features [40] [41] . When two label exist in the dataset, the classification is binary. When label is more than two, classification becomes multi-label. The Severstal dataset contains a target variable with more than two labels (defects). The dataset has four types of defects: defect 1, defect 2, defect 3 and defect 4. A multi-label classifier is employed in the ML system set-up of the current research. This enables prediction of different defect class in the dataset. The classifier is trained on the training set and evaluated using the validation set. The classifier’s input includes the generated training images (256 × 512) and corresponding labelled masks. As represented in Figure 1 , for each prediction, a classifier probability score is calculated. This reveals the model’s confidence in the presence of each defect that is detected.

Segmentation in machine learning is another important task and it is common with image processing, object identification and biomedical applications [42] - [44] . The main target of segmentation in all these applications is streamline the outlook of image being investigated into item that is more coherent for visual sighting [44] [45] . In this research, segmentation has been integrated with classification to further streamline the image of defect that is detected by classification. The multi-label classifier in Section (3.2) identifies types of defects that exist in a particular sheet material while segmentation is implemented to understand the part of the sheet where defect is detected. The multi-label classifier identifies different types of defects (Defect 1, Defect 2, Defect 3, Defect 4) that exist in an image. For each defect, predictions are produced that help in understanding which part of the image contains each defect.

Thresholding is important in situation where classification is integrated with segmentation. This enables proper refinement of predictions made by the classification algorithm. Basically, the purpose of thresholding is to establish a cutoff point for classifier outputs to decide which predictions should be considered valid detections of defects. With adoption of thresholds between the 2nd and 98th percentiles, the thresholding method ensures that predictions with low confidence (potential false positives or false negatives) are discarded. As affirms by [46] and [47] , thresholding increases the reliability of model’s performance. The remasking process that follows thresholding is specified to further improve the accuracy of the segmentation. Remasking involves reintroducing the predicted defects and comparing them against the original pixel mask (ground truth). This process facilitates refinement of the segmented regions through elimination of possible noise while validating the areas where defects are predicted [48] .

The re-masked images are resized back to the original resolution (256 × 1600) before final output. The final outputs consist of a CSV file containing: ImageId (the unique identifier for each image) ClassId (the defect class identified Defect 1, Defect 2, Defect 3 and Defect 4) and the Encoded Pixels (the encoded pixel values representing the locations of the defects)

Evaluation of machine learning models directly involves assessment of model performance [49] . The evaluation reflects the applicability and efficiency of the models across different domains. Evaluation is important to determine the suitability of models implemented for specific investigation while ensuring their effectiveness in real-world applications [50] . There are many metrics used in machine learning domain to measure model performances [51] . Based on the integration of classification and segmentation, the applicable metrics includes accuracy, precision, F1-score, recall, dice coefficient and dice loss. Accuracy measures the proportion of correct predictions made by a model out of all predictions [52] . Precision focuses on the ability of the model to correctly identify only the relevant positive instances [53] . It is computed as the ratio of true positives to the sum of true positives and false positives. Recall, also known as sensitivity, evaluates the model’s ability to detect all possible positive cases by dividing true positives by the sum of true positives and false negatives. The F1-score is the harmonic mean of precision and recall. It yields a balanced metric when trade-off exist between precision and recall. The Dice coefficient is one of the metrics for similarity [54] [55] . It measures the overlap between predicted and actual segmentations, with values ranging from 0 (no overlap) to 1 (perfect overlap). Conversely, dice loss is a loss function derived from the Dice coefficient. It is used to train models by penalizing the model for poor overlap between predicted and actual regions.

Prior testing before final model selection was adopted to yield improved model performance in the overall system. Initial experimentation was carried out using four models for classification: EfficientNetB1, ResNet50, DenseNet121 and VGG16. EfficientNetB1 is part of a family of models that uses compound scaling method, balancing network depth, width, and resolution to achieve high accuracy with fewer parameters [56] . The EfficientNet architecture is designed to optimize both accuracy and efficiency. This makes it suitable for large-scale image classification tasks [57] . Just like EffcientNetB1, ResNet50 is a member of family of ResNet. According to its name, ResNet50 maintain depth with 50 layers. ResNet50 is deep CNN architecture that function with consideration of residual connections [58] . The residual connections resolve the issue of vanishing gradients to boost training of deeper networks. ResNet50 has strength to capture detailed information in images with improved adaptability for item recognition across the images [59] . DesNet121 is a CNN architecture that work with the reuse of dense connections and features [60] . Fundamentally, in DenseNet, every layer is linked to every other layer in feed-forward settings. While in traditional deep learning setup, output layer usually acts as the only input to subsequent layers in DenseNet configuration, dense connections are introduced every layer to get feature, and they allow maps of layers before it [61] [62] . This special arrangement control growth of network and computational complexity of the algorithms which support improved performance [63] . VGG is also a CNN model developed by Visual Geometry Group (VGG) at the University of Oxford. The VGG16 architecture is the one with 16 layers with learnable weights [64] [65] . This architecture reduces the spatial dimensions of the input image while increasing the depth of the feature maps [66] . VGG16 produces better classification due to its ability to focus on targeted image area [67] .

As justified by many evidence of their strengths with handling of image classification, EfficientNetB1, ResNet50, DenseNet121 and VGG16 were implemented.

The experiment showed DenseNet121 performing better than the rest of the deep learning ( Table 5 ). Thus, denseNet121 was selected for classification in the final model as specified in the project workflow. DenseNet-121 has shown in various computer vision tasks, such as image classification, object detection, and image segmentation. The architecture of DenseNet-121 allows for better gradient flow throughout the network, reducing the vanishing gradient problem. Also, its dense connections promote feature reuse, allowing the network to learn more compact and efficient representations. This makes it suitable for the task of defect detection in steel plate images. Specifically, the DenseNet-121 algorithm was used for binary and multi-class classification in main analysis.

Similarly, U-Net and DeepLabV3 were tested for segmentation. The architecture of U-Net is characterized by its U-shaped structure. The “U” shape consists of a contracting path and an expansive path that allow the model to capture both context and precise localization of images during segmentation [68] . U-Net is adaptable, efficient and high performing [69] [70] . DeepLab is combined neural networks specifically designed for semantic image segmentation [71] [72] . Implementing these two models achieved results in Table 6 .

Based on the outcomes of initial experimentation, DenseNet121 and DeepLab is selected to build the classification and segmentation models in the system workflow. The input of DenseNet121 is configured with shape of 100 by 100 pixels with 3 colour channels (RGB) and set up with five layers ( Figure 5 ). The whole layers are divided into DenseBlocks where average pooling layer average each of the feature map and reduce spatial dimensions. Each of the fully-connected (dense) layers is followed by ReLu activation function. Batch normalization is applied to normalize the activations to stabilize the model and improve speed.

A drop-out of 0.3 was applied after the first two dense layers thus during model training, 30% of the units are randomly fix to zero in order to stop overfitting. Sigmoid activation function which is perfect for binary classification and is added at final dense layer. The DenseNet-121 model was trained on the pre-processed dataset. The training process involved backpropagation and gradient descent to update the model’s parameters. The model was trained to learn to identify and categorize the defects present in the images of steel plates.

The DeepLabV3+ model was setup after initializing the input size as (512, 512, 3) and the number of output classes as n_classes. The n_classes in the DeepLabV3+ model is applied to make the model flexible and adaptable to various defect classes in the dataset ( Figure 6 ). This design makes the model generalizable. The encoder block was configured with a Conv2D layer with 32 filters, a kernel size of (3, 3), and stride 2, followed by a custom convolutional layer (Conv2D_custom) with 64 filters, a kernel size of (1, 3), and stride 1, and a BatchNormalization layer. Xception blocks are applied for different sizes like (128, 128, 128), (256, 256, 256), and (728, 728, 728), with strides depending on the layer. The SeparableConv2D layers include filters of 256, with dilation rates of 6, 12, and 18.

The predicted class of defects from classifiers are fed into segmentation through decision flow of 2nd/98th percentile rule. This allow human decisions as areas outside the range between the 2nd percentile and the 98th percentile in the predicted masks are neglected. With application of thresholding, model performance is improved (reliability and robustness) by focusing on the area that is more concern with the study predictions and discard other regions.

After training, the classification model’s performance was evaluated using various metrics, including precision, recall, F1-score, and accuracy ( Figure 7 ). These metrics provided a quantitative measure of the model’s ability to accurately identify and categorize the defects in the steel plate images.

The segmentation models are trained using the Adam optimizer and the Dice loss function. The Dice coefficient is used as a metric for model evaluation. The models are trained for a specified number of epochs, with the best model saved based on the maximum Dice coefficient achieved on the validation set. Each trained model is then evaluated on the train, validation, and test sets. The evaluation scores (Dice loss and Dice coefficient) for each defect class are displayed ( Figure 8 ). Additionally, the model’s predictions on a subset of the train, validation, and test datasets are visualized by displaying the original image, the ground truth mask, and the predicted mask side by side ( Figures 9-12 ).

These models were trained and evaluated separately for each type of defect.

The coefficients range from 64.81% to 87.69% across different datasets and defect classes, indicating varying performance for different defects ( Table 9 ). Some insights can be drawn. Defect 4 consistently has the highest Dice Coefficients across all sets, suggesting that the model is best at detecting this defect. Defect 1 has the lowest Dice Coefficient in the testing set, hinting at potential challenges in detecting this specific defect.

Comparing this model with other commonly used models in the field of image recognition, such as VGG16, ResNet, DenseNet-121 showed superior performance. The superior performance of DenseNet-121 can be attributed to its unique architecture, where each layer is connected to every other layer in a feed-forward fashion, allowing for better gradient flow and feature reuse. This architecture enables DenseNet-121 to learn more compact and efficient representations, making it more suitable for complex tasks like defect detection in steel plate images. Different variations of the DeepLab models. The use of DeepLab V3 provides the benefits of a powerful, efficient convolutional network that scales well with increasing amounts of data and computational resources. Each model was trained separately to detect a specific type of defect in the steel plates. The model’s performance was evaluated using the Dice coefficient, a popular metric for image segmentation tasks that measures the overlap between the predicted and actual results. High Dice coefficient value ( Table 10 ) indicate a high degree of overlap and, thus, successful defect detection. The models demonstrated effective learning, with their performance improving over successive training epochs. The visualization of the model’s predictions further confirmed their ability to accurately detect and categorize defects.

VGG16 showed lowest accuracy due to its smallest number of layers embedded in its configuration. Though observation indicates that larger layer sizes do not ensure improve accuracy value. The study models achieved higher accuracies compared to [25] outcomes as configurations were specifically tailored towards improved performance with optimization of models’ parameters (pooling, activation functions, batch normalization).

The study segmentation model performed better than benchmarked paper from literature review, Amin and Akhter 2020. DeepLab V3 delivered higher dice coefficients across classes of the defects while Amin and Akhter 2020 model exhibited imbalanced performance across defect classes and failed to predict defects for classes 1, 2 and 4.

The proposed methodology offers several improvements over existing benchmarked works in defect detection. Deep learning models like VGG16 and ResNet101 used in research by [25] , yielded lower accuracies of 50% and 70.5%, respectively. In contrast, the proposed system utilizing DenseNet121 achieves a significantly higher accuracy of 92.34% ( Table 11 ). This demonstrates its superior capability with learning and identification of defect patterns. The DenseNet121 architecture outperforms other deep learning techniques by leveraging dense connections between layers, utilization of feature reuse, and improvement of gradient flow. All these allow DenseNet121 to mitigate vanishing gradient problem better. Also, the overall system in this research integrates multi-label classification and segmentation, providing both identification and precise localization of defects. This combination surpasses models that focus solely on classification like those by [22] and [25] which they achieved accuracies of 79.91% with MobileNet and 96.9% with the FDD model. Their models lacked segmentation capabilities, and this limits the practical application when localization of defect is a top target. Based on the initial experimentation and final modelling, the use of DeepLab V3 for segmentation in the current research further improves performance. The Dice coefficient for segmentation ranges from 64.81% to 87.69% across different defect classes. This provides more robust and reliable detection compared to previous approaches in research by [73] where their model only achieved a Dice coefficient of 54.30%. DenseNet121 maintains a higher feature map resolution in its configuration while the is densely connected. This allows DenseNet121 to capture more detailed spatial information and produced higher precision of 92.31% compared to YOLO models that downsample the input image multiple times and achieved highest precision of 65.20% in the work of Akhyar et al. (2023) [22] . Generally, the proposed methodology adopted in this research excels by offering a more integrated approach with the combination of classification and segmentation to improve accuracy and precision that surpasses models in existing works.