In this algorithm, let the point labeled be x, and label the point closest (k = 1) or numerous points closest (k = n) to x be y. When there is a vast amount of data x, y will possibly be the same. For example, when a baised coin is tossed (the chance of getting one result is higher than the other) one million times, the result is nine hundred thousand times heads. Predictably, the next toss is very likely to be head. In this case, KNN uses a similar method.

C( x )=Y( 1 )

In Figure 1, K nearest neighbor classifier classifies this dataset into three groups of clusters represented by three different colors. The arrows are pointing to X’s nearest neighbors. In this case, X is classified as orange because it has the most nearest neighbors in the K value set (K = 5 in the figure).

The classifier C assigns x to its closest neighbor Y depending on the value of k (in this case k = 1). As the size of data set increases, the error rate would be guaranteed to be less than twice of the Bayes error rate (The minimum error rate based on the data’s distribution).

P * ≤ P ≤ P * ( 2 − ( C / ( C − 1 ) ) P * )

Above is the formula for obtaining the tight error boundary. P* is the Bayes error rate, C stands for the number of classes, and P is the nearest neighbor method’s error rate. For example, when there is a large amount of data, Y is the nearest neighbor, and X needs to be classified, it is very likely for X to be classified with Y. The chance of getting an error from this classification will be greater than the minimum rate based on the population and less than two times that error rate, which is shown in the formula above.

Random Forest can obtain a very accurate and precise result by planting huge amount of decision trees.

Training sets would be selected at random and be randomly distributed along X and Y vectors.

Then, tree bagging method (bootstrap aggregating) will choose a sample with replacement at random.

In Figure 2, random forest classifier plants several trees to form a forest. By calculating the mean of the results from the forest, the classifier generates a majority voting outcome, which is the final result of the classification.

For example, when selecting for B times, tree bagging would have B training samples from X and Y called X_b and Y_b. Next, it will make a decision tree fb based on X_b and Y_b. Therefore, after these trainings, the sample could be predicted from the mean number of the trees, with the formula:

f ^ = 1 B ∑ b = 1 B f b (x ′)

The majority vote would be the result of the classification. This method strengthens the performance of the model by limiting the variances without increasing the degree of bias. In the other hand, in the case of training with few and noisy datasets, tree bagging the model would perform better by decreasing the correlation between the trees.

Random Forest differs in the learning process for each individual tree. It uses a modified learning process called feature bagging. Different from tree bagging, feature bagging allows each decision tree to have a random subset of features. By doing so, it is able to intentionally prevent some significant predictor features and therefore make them correlated.

A basic neural network consists of three parts, as shown in Figure 3: the input layer (layer L₁), the hidden layers (layer L₂, L₃), and the output layer (layer L₄).

We can view the model as f ( x ) : x → y

f ( x ) = K ( Σ W i G i (x))

K in this equation is also known as the activation function, which is usually a defined function, such as the hyperbolic tangent function. The activation function helps address the values as the input layer change.

Each layer has its weight and the weight changes as the model goes through the hidden layers until it reaches the output layer. The components of each layer are completely independent of each other. As Figure 3 shows, more hidden layers would result in different outputs.

In another task, researchers from University of Toronto including Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov introduced the dropout method to neural networks. Normally, neural networks contain many hidden layers to express complicated relationships. However, when it is applied to test data with noisy training data, the models don’t perform as well as that for the training datasets. Dropout is one of the methods designed mainly for reducing the amount of computation and decreasing the chance of overfitting. It is proved that dropouts can effectively help a large model to achieve these goals by intentionally ignoring random units of random layers (Srivastava et al., 2014).

As shown in Figure 4, Dropout method can avoid overfitting and provide a way of combing exponential architectures for neural networks. This method temporarily removes randomly hidden or visible units with their income and outcome connections. When applying dropouts, the training dataset can be viewed as a smaller version of the testing dataset. For example, n units of neural networks can be viewed as a collection of 2n quantities of potential neural networks, yet, these networks still share the same weights so the number of total parameters is still the same, n2.

Table 1 shows data published by Sebastian Tomczak, a researcher at University of Science and Technology, Poland. This dataset predicts bankruptcy of Polish companies. To get the data, the already bankrupted companies were evaluated in the period of 2000-2012, and the other companies were studied from 2007-2013.

As shown in Table 2, truncate is imported for this task, which would only use the first quantity of the number previously set. Each algorithm would use truncate equal to 50 and 150, which means it would cut off the first 50 and 150 samples and later be used for accuracy calculation.

K value in Figure 5 is set to be 3. According to the figure, 3 appears to be a good k value because the classifier is able to classify many datasets outside of the majority neighbors.

K value in Figure 6 is set to be 5. According to the figure, 5 is a relatively alternative value for k. Although it is able to classify datasets, it lacks precision comparing to k value of 3.

K value in Figure 7 is set to be 7. According to the figure, 7 is the least alternate value for k among three experiments, because it is able to classify the least dataset that is outside of the majority neighbors.

Table 3 shows the result of accuracy rate based on the KNN calculation.

As shown in Table 4, a truncate of 50 results in an accuracy rate of 0.975, and that of 150 results in an ac-curacy rate of 0.99, which is relatively high.

Table 5 shows the results of accuracy and loss rate using Neural Networks when truncate is 50 and 150 respectively.

Table 6 shows the results of accuracy and loss rate using Neural Networks when truncate is 50 and 150 respectively, with two dropout rates applied. The first dropout rate tested is 0.5, and the second dropout rate is 0.3. The results of Nerual Networks model are visualized in Figure 8 and Figure 9.