Datamining is a sub-category of machine learning, which uses labelled data to train algorithms to classify information or predict results [7] . Input data is incorporated into the model through weight adjustments so that the algorithm works correctly [7] . A supervised machine learning approach is used in cases where you already know what you will teach the machine ahead of time. In order for the algorithm to achieve the desired results, it is necessary to take it through a large training data set, examine it, and adjust the parameters until the desired results are achieved. By adjusting its own parameters in response to a loss function, the algorithm measures its accuracy in order to minimize the error as much as possible. A final test of the trained machine is to allow it to make predictions based on a validation set, which is a new set of unseen data that has not been seen before [7] . There are two types of algorithms which are used in supervised learning, namely regression algorithms and classification algorithms, as shown in Figure 1 .

An example of unsupervised learning is a method for learning without the need for a person to monitor the model as the model explores a data set. Therefore, the training process is characterized by a data set without any labels or classifications that have been previously defined [10] . Using only statistical properties as a basis to assess the hidden information, the algorithm tries to identify patterns that relate different variables in order to identify hidden information that may not have been spotted before they are scanned [6] . According to Figure 1 , there are three major types of unsupervised learning algorithms: clustering, association, and dimensionality reduction1.

The idea of semi-supervised learning is an important subcategory of machine learning. As part of the training stage, rather than labeling the entire analysed data set, only a small portion of the data is reviewed, hand-labeled, and then used to train a model by combining it with other data. Lastly, this trained model is able to correctly classify all the features that exist in a larger unlabelled data set that has not been labelled by humans [12] . There are two types of learning techniques: semi-supervised learning and supervised learning. It has been shown that semi-supervised learning is a more flexible method of learning than unsupervised learning and can be used for a wide range of classification, regression, clustering, and even association problems. Due to the fact that semi-supervised learning uses a small amount of labelled data as well as a large amount of unlabelled data, it is less computationally intensive and takes less time to prepare data than supervised learning [12] .

The idea of reinforcement learning is a subcategory of machine learning in which machines have the capability of interacting with environments. In a learning agent, behavior is learned by performing tasks and seeing the results of the actions they perform in a given environment. As a result of every good action that the agent takes, he or she receives positive feedback; as a result of every bad action that the agent takes, he or she receives negative feedback or a penalty. If the machine repeats the process thousands of times, it over time becomes more and more skilled as it learns and becomes more proficient [13] . There is one significant difference between reinforcement learning and traditional supervised learning and that is that in supervised learning; the answer is found in the training data. It is therefore possible to train the model with the correct solution as a result. In contrast to this, reinforcement learning is not a response, but rather an agent that decides what to do based on what happens, in order to respond or perform a task at hand. There are some significant differences between reinforcement learning and unsupervised learning in terms of the objectives that it pursues [14] .

Furthermore, there are a number of specific application cases and practical effects of machine learning and deep learning methods in optical communication systems that are: channel characterization and modelling, and network optimization and planning. The channel characterization and modelling. Modelling optical channel impairments and uncertainties using machine learning and deep learning. Optimising network architecture, transmission parameters, and resource allocation improves system performance and reliability. In order to decrease the complexity of networks, deep learning and machine learning methods use network optimization in order to reduce complexity. As a result of using these methods, it is possible to optimize the configuration of the network, minimize the energy consumption, and improve the efficiency and reliability of communications.

This article highlights the role of machine learning and deep learning in improving optical solutions. The design science approach and literature review conducted in this study provide valuable insights into the challenges and opportunities associated with utilizing these techniques. The successful implementation of machine learning and deep learning in optical solutions has the potential to revolutionize various industries, leading to improved efficiency, performance, and accuracy.

This article is organized as follows: the methodology is discussed in Section 2 whereas the conclusion and future works are offered in Section 3.

Optical communication systems have revolutionized the way we transmit information in recent years. Through optical communication systems, fast and high-capacity data transmission is possible. Since demand for faster and higher-capacity networks is increasing, traditional optical communication systems are reaching their limits. The use of machine learning and deep learning approaches has given rise to powerful tools that can significantly boost optical communication systems’ performance. This paper presents an overview of how machine learning (ML) and deep learning can be used to improve the performance of optical communication systems, including modulation techniques, channel modelling, equalization, and system optimization. According to the paper, these approaches offer advantages such as spectral efficiency improvements, reduced latency, and enhanced robustness to impairments, including spectrum degradation, in the channel. Furthermore, machine learning and deep learning are discussed in light of potential challenges and limitations. Moreover, their potential benefits are discussed. As a result of this paper, we gain an understanding and highlight the potential of these approaches to improve optical communication in the future.