The inductive method reconstructs original dataset P by parameter W × H . Once the reconstructed data and training data close enough, the reconstructed data also can automatically close to the missing value by gradient descent. First of all, the missing value matrix P ∈ R m × n . Given a sample of observed entries Ω from a true underlying matrix M ∈ R m × n , the goal is to estimate missing entries under additional assumptions on the structure of the matrix. The most common assumption is that the matrix is low-rank, i.e., M = W × H T , as shown in Figure 1.

We could solve the following optimization problem:

min W ∈ R m × k , H ∈ R m × k ∑ ( P − W T H ) 2 + 1 2 λ ( ‖ W ‖ F 2 + ‖ H ‖ F 2 ) ,

where λ is a regularization parameter, W and H denote the parameters to reconstruct. We want to learn factors W ∈ R m × k and H ∈ R n × k such that the estimated values are close to the observed entries, and the W H T is low rank. This problem can be represented as following figure:

Inspired by linear version IMC, we come up with nonlinear version called Nonlinear Inductive Matrix Completion. The main idea is adding nonlinear function into each parameter. Consider Inductive Matrix Completion method with nonlinear activation function with following optimization problem [4] :

min W ∈ R m × k , H ∈ R m × k ∑ ( P − ϕ ( W T ) ϕ ( H ) ) 2 + 1 2 λ ( ‖ W ‖ F 2 + ‖ H ‖ F 2 ) ,

Here function ϕ can be any nonlinear function such as sigmoid function, ReLU function, Tanh function. When function ϕ output is input itself, Nonlinear IMC will automatically become a linear version inductive matrix completion.

Inspired by one layer neural network [5] , we introduce multiple layers structure, we come up with deep version called Deep Inductive Matrix Completion. The main idea is adding multiple parameters layers. As Shown in Figure 2.

Consider Nonlinear Inductive Matrix Completion method involved in deep factor learning process. The optimization problem can be defined as following [5] :

min W ∈ R m × n , H ∈ R n × k ∑ ( P − ϕ ( W ϕ ( H ϕ ( H T ϕ ( W T ) ) ) ) ) 2 + 1 2 λ ( ‖ W ‖ F 2 + ‖ H ‖ F 2 ) ,

Here W ∈ R m × n and H ∈ R n × k , we have k < m , which can be clearly show as following picture:

min A ∈ R m × n , B ∈ R n × k , C ∈ R k × l ∑ ( P − ϕ ( A ϕ ( B ϕ ( C ⋯ ϕ ( C T ϕ ( B T ϕ ( A T ) ) ) ) ) ) ) 2 + 1 2 λ ( ‖ A ‖ F 2 + ‖ B ‖ F 2 + ‖ C ‖ F 2 + ⋯ ) ,

Here m > n > k > l . Noticed that deep IMC with only one layer will automatically become a linear version IMC.

The optimization process is to solve the above function with loss function. Our target is the original matrix P and the reconstructed matrix as close as possible. In all above objective function, we choose gradient descent method to solve it. Gradient descent is a first-order iterative optimization algorithm for finding the local minimum of a function. Here we will give mathematical detail about gradient descent to solve inductive matrix completion. The process can divide to 3 steps: firstly compute gradient for each parameters. Secondly update parameters in each iteration. Last but not least, repeat iteration until we find minimial of loss function.

For example we want to optimize linear version inductive matrix completion method. The first thing is to randomly initialize parameters W and H. Then fix parameters W to train another parameter H, and then fix parameter H to train parameter W until we have minimal loss. The loss function can be written as following [5] :

J ( θ ) = ∑ ( P − W T H ) 2 + 1 2 λ ( ‖ W ‖ F 2 + ‖ H ‖ F 2 ) ,

Here θ is parameter W and H.

Secondly, we can compute gradient to find the fastest direction from initialized point to the local minimal respect to W i and H j , we have:

∂ J ( θ ) ∂ W = 2 × ( P − W H H ) × ( − H ) + λ ‖ W ‖ F

Similarly,

∂ J ( θ ) ∂ H = 2 × ( P − W H T ) × ( − W ) + λ ‖ H ‖ F

Now, we can update parameters in each iteration as following:

W = W − α ∂ J ( θ ) ∂ W

H = H − α ∂ J ( θ ) ∂ H

Here W × H T is our reconstructed matrix with full observation.

1: Initialize all parameters

2: Split training data and test data

3: for i ∈ ( 1, ⋯ , N ) do

4: Set loss function

5: Calculate difference between original matrix and reconstructed matrix

6: Count the number of correct and total number of train/test value

7: Calculate training accuracy and test accuracy

8: end for

9: Gradient descent respect to loss function

10: Update parameters

In this paper, we focus on test accuracy since our target is to put an appropriate value in the missing position. However, we also need to define training accuracy since training accuracy shows the proportion of correct predicting value in total number of observation. We can track the change of training accuracy to know the progress of parameters training process. The training accuracy is defined as:

training   accuracy = number   of   correct   value   in   non-missing   part total   number   of   non-missing   part

Similarly, we can define test accuracy as the proportion of correct predicting value in total number of missing value. The following is formula:

test   accuracy = number   of   correct   value   in   missing   part total   number   of   missing   part

We regard ε = 10 − 2 in this test, which means when the difference between original matrix and reconstructed matrix element is less than 10^−2, we think the method predicts correct value.

Our target is to investigate this section will show how to generate fake linear data in detail. Firstly, let’s only consider a small size matrix. We generate the 100 × 100 original matrix P by W × H T for IMC and NIMC, here W and H are 100 × k vectors randomly generated within range 0 to 1. Notice that dimension k must be small number since the normal assumption is matrix P must be low rank so that the problem gradient descend method can solve. As shown in Figure 3.

Now we want to generate linear deep style structure data. Inspired by the neural network structure, we try to set linear data structure with multiple layers. Similarly, we generate the 100 × 100 original matrix P by W × H T but adding one more layer. The result is we set original matrix P by A × B × B T × A T , here A and B are 100 × m and m × k vectors, which randomly generate within range 0 to 1. Noticed that dimension k must be small number since matrix P must be low rank so that the problem gradient descend method can solve. The generation detail can be shown in Figure 4.

For above dataset, we randomly take out 30% value as missing value from P, hence, each test will use totally different original dataset.

Based on the linear data generation, we want to add nonlinear function to generate nonlinear data. In this way, we want to figure out how the nonlinear IMC method handle nonlinear data structure and how deep IMC handle deep style data structure. The generation of nonlinear data is much similar with linear data generation; however, the most important difference is to add different nonlinear function into each parameters column. The ϕ can be different nonlinear function such as sigmiod, RuLU, cos, sin or any nonlinear function. As shown in Figure 5.

After testing on fake data, we want to know how our method handle real data. Here we choose Russell 3000 as our research target. As shown in Figure 6.

The first time I met Russell 3000 is at Professor Randy Paffenroth group meeting. One of his PhD students Nitish research on finding financial market dimensions [6] . The main idea is to find out how many dimensions can represent the whole market data, which is a hard topic and enough to devote several years’ research.

As for my perspective, Russell 3000 is a dataset with largely empty. The reason why the Russell 3000 is largely empty is the stocks have not completed IPO process at the date or suddenly disappear in the financial market due to the company collapse, or company under suspend for a period of time. The target we choose this dataset is to find out the how our method handle real world dataset. Also, Nitish find out Russell 3000 has nonlinear data structure during his research. However, this is a guess or conclusion we have from experiment. Unfortunately, we do not have formal theory to prove it now. Besides, we can try IMC, NIMC and DIMC on Russell 3000, which is an alternative way to verify data structure for Russell 3000. The reason why we can think in this way is all three method only can solve the problem that has same data structure with method itself. Hence, we can verify Russell 3000 data structure by testing which method works.

Since we never know the what is the correct value in missing position since they does not exist in real world market, we randomly choose 2000 stocks from Russell 3000 with 2000 valid date. In this way, we have a 2000 × 2000 matrix from Russell 3000 with all observation. The next thing is pick missing value. We have two kinds of missing value position to model real world market since the real world market exists the two situation. The first one models the situation suspend happen in the market. We define Random as randomly taking out of 30% value from dataset as missing value. The second one models the situation company have not done IPO process yet or suddenly disappear in stock market. We define Cross as taking out of large number value as missing value while left only a few observations. The two situations are shown in Figure 7.

In experiment 1, we want to know the IMC method performance with linear low rank fake data on different dimensions. Also, this can be clearly shown in the relation between parameters dimension and test accuracy. Here we test IMC by setting parameters in dimensions 1, 5 and 20, as shown in Figure 8.

In experiment 2, we do the similar thing again but using NIMC in linear low rank data. We want to see how nonlinear method handle linear low rank data. Here we test NIMC by setting parameters in dimensions 1 and 5, as shown in Figure 9.

In experiment 3, we do the similar thing again but using DIMC in linear low rank data. We want to see how the Deep IMC method handle linear low rank data. Here we test DIMC by setting parameters in dimension 100 × 10 × 5. We show the performance of deep version IMC as Figure 10.

From experiments 1, 2, 3, we have IMC as the best method to solve linear low rank data instead of applying NIMC and DIMC. Now we want to test how the IMC, NIMC and DIMC handle nonlinear low rank fake data. We set the same parameters dimension by 5 in all three tests. The test result can be tracked as shown in Figure 11.

First we want to test how IMC handle Russell 3000 with random and cross missing value position. The test has two kinds of missing value position, which models two situation happens in real world. As shown in Figure 12.

This experiment we want to test NIMC performance on Russell 3000. Similar as the experiment 5, we test two missing value position on NIMC. Here is test with NIMC in parameters dimension 5. As shown in Figure 13.

From Figure 13, NIMC with Russell 3000 on parameters dimension 5 has huge information loss, which only has 14% correct missing value predicting. From the experiment from fake data, we know nonlinear method usually needs high dimension parameters to reconstruct matrix, hence, we try to increase parameters dimension by 80 as shown in Figure 14.

After we test Russell 3000 with NIMC, we know nonlinear method can handle it well in random missing value position instead of cross missing value position. Moreover, we want to know DIMC performance on the two kinds of missing value dataset. This test shows the performance that DIMC handle Russell 3000 with random and cross missing value position, as shown in Figure 15.

The last experiment we want to summarize information loss in the three methods on Russell 3000. We do experiment with three method but different parameter dimensions on 5, 10, 20, 50, 100, trying to find out how the dimension change affect test accuracy in real data Russell 3000. We do each test for 5 times and record error in Figure 16.

From Figure 16, the result shows the relation between accuracy loss and dimensionality change. We have NIMC and DIMC have large information loss in low dimension, especially DIMC suddenly decreases when dimension decreases by 5, however, IMC has less information loss on all dimensions, meaning NIMC and DIMC require high dimension to reconstruct dataset. However, we still need NIMC and DIMC to solve complex structured data since we know IMC only can solve linear structured data from fake data experiments.

In this case, we apply indicators as following: transport_injury, falls, drowning, fires_heat_hot_substances, poisonings, exposure_to_mechanical_forces, foreign_body, education, network_readiness, other_unintentional_injuries, expo-sure_to_forces_of_nature__diunintentional_injuries, ul_standards, govern-ment_effectiveness,, consume_protection_survey, ul_labor_rights_index, road_safety, population, population_under_15, population_over_65, dgp, un_region, who_region, iso_membership, iec_membership, un_sub_region, un_development_status. We have total 29 indicators to impute data. Moreover, we add 2018 UL Safety Index into model to control prediction since any information we provided will be benefit for result. The reason is machine learning algorithm would increase accuracy once we provide more information and this data is integral to the imputation process, as it provides the “pattern” that the method leverages. Hence, totally we have 29 indicators to impute data while 291 missing data points at this moment. Lastly, the method only can feed numerical value instead of category words, hence, we set factorize indicators to numercial value as following: un_region, un sub region, un development status, who region, iso membership, iec_membership.

The above process can be found in code “Get Origial Data” and “Get Categroy Data by Int” part.

For all 29 indicators, we want to normalize data by each indicators’ mean and standard deviation as following formula:

X ' = X − μ σ

Here, X' is normalized data, X is data point, µ is mean value of each indicators, σ is standard deviation of each indicators. Now, we have clean data to impute, which can be found as “df clean.csv” after run code provided.

For the data imputation part, we applied non-linear IMC with Tanh activation function since Tanh give us best training accuracy after try sigmoid function, ReLU function, sin function, cos function or even linear version IMC. Here we set parameters W and H randomly generated by 178 × 50 and 50 × 30 matrix, learning rate alpha is 10. Do 7000 iteration for each training. The result show 7000 iteration is enough to reach 98% training accuracy. The code can automatically output loss function and training accuracy on each iteration, then plot visualization graph. This section can be found in code “Data Imputation” part.

For the data we impute, most of them are “reasonable”, but a few of them are “unreasonable”. For example, for the ul_labor right, one value will excess 77, which is impossible considering the maximum value of the initial setting of ul labor right indicator is 77. Hence, we regard the data imputed which exceeds 2σ is abnormal point, and then we adjust these data by normalizing to the normal distribution by applying formula in 2.2. Totally we have about 1220 abnormal data point in each experiment. The value can be print from code “adjust normalize” part.

The last step is repeat above process to output a range for imputed data. After checking output by applying randomly input, we realized our model is stable since the imputed data follows normal distributed with N (0, 1).

By the result, we repeat step 3rd data imputation for 100 times, recording 100 times value then calculate mean value and standard deviation for each missing data points. Then pick half standard deviation above or below mean value as the appropriate boundary to calculate UL Safety Index. Therefore, the prediction of UL Safety Index is a range instead of a fixed value.

The data distribution shows most of data points follow the trend that high Consumer Protection score consistently with high Safety Index as shown in Figure 17.

Only one or two data point seems “unreasonable”. For example, the lowest red point with high Safety Index but a very low Consumer Protection score prediction. Indeed, we have to admit any prediction will surely have “mismatch” point. However, according to Stability analysis above, we can calculate prediction range instead of a fix value, which is close to the truth and acceptable for us.

The data distribution shows most of data points follow the trend that high Labor Right score consistently with high Safety Index as shown in Figure 18.

Only one or two data points seem “unreasonable”. For example, the upper left red point with low Safety Index but a very high Labor Right score prediction, or the lower right red point, has a high Safety Index but a very low Labor Right score. Also, we have to admit any prediction will surely have a “mismatch” point. However, according to the Stability analysis above, we can calculate the prediction range instead of a fixed value, which is close to the truth and acceptable for us.

See Figure 19.