SNPs and phenotypic effect for the four different types of cancers-acute myeloid leukemia (MIM # 601626), breast cancer (MIM#114480), colorectal cancer (MIM#114500) and esophageal cancer (MIM#114480) are collected from OMIM, one of the biggest databases which provides detailed information about phenotype-genotype relation [16].

To determine whether SNPs occur at protein-protein interaction sites, we used STRING database to identify the interaction sites for the affected proteins (i.e., the gene products) [17]. For Acute Myeloid Leukemia, 16 genes are in-

volved, which result in 171 proteins that have certain interactions with each other. Due to unsolved 3D protein structures the set is reduced to 111 proteins. There are several databases which provide SNP data, including SwissProt and dbSNP. For this study we used SNP from SwissProt database [18] because of its large collection as compared to other databases. The queries to SwissProt identified 1399 nsSNP for these 111 proteins. The same data collection protocol is used for the other three cancers as well. After filtering with required protein structural as well as sequence properties, the final data set consists of 4056 SNP’s in total, as listed in Table 1.

Using these nsSNP, feature vector was constructed using several properties of both sequence and their respective structure. FoldX was used to calculate parameters which are important for protein stability [19]. It provides several important features along with the calculation of total energy for the mutant and the wild type protein. Panther software calculates Substitution Position-Specific Evolutionary Conservation (subPSEC) Scores and it is based on hidden Markov model (HMM). It was used to collect subPSEC score. Fathmm was used to calculate HMM cancer-specific pathogenicity weights [21]. In total 21 features were collected and all these features are shown in supplementary data S1.

We also collect haplotype data for genes associated with Acute Myleoid Leukemia. A haplotype is considered as set of polymorphic, which are inherited together. It is referred to a combination of alleles or a set of SNP that are found on the same chromosome [15]. To collect haplotype information two databases were used in this study. One is HapMapProject and the other is UCSC genome browser [25] [26]. HapMap Project has a wide range of SNPs, which are collected from dbSNP. Since our dataset consists of SNPs collected from SwissProt, to collect as many as haplotype data, we incorporate UCSC genome browser, which provides gene based common allele variants taken from 1000 genome project [27].

To assess the prediction performance, we adopt the widely accepted cross-vali-

dation scheme. Specifically, we used 10-fold cross-validation. The data is randomly split to 10 equal-sized subsets, and one set is reserved for testing and the remaining 9 subsets are combined into a training set to train the classifier. This process is repeated 10 times, with each subset being used as test set once and the average performance from 10 runs is reported. We used some commonly used measurements to report the performance, which includes accuracy, precision, recall, F1 score, and MCC, defined as follows.

where TP stands for true positive when a SNP is correctly predicted as detrimental, TN for true negative when a SNP is correctly predicted as polymorphism, FP for false positive when a SNP is incorrectly predicted as detrimental; and FN for false negative when a SNP is incorrectly predicted as polymorphism.

We also evaluate the performance using receiver operating characteristic (ROC) curve and Receiver operating characteristic (ROC) score.ROC is a graphical representation that illustrates the performance of a binary classifier system. The plot is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. The true-positive rate is also known as sensitivity or recall while false-positive rate is also known (1 − specificity) [28].

For the 4056 SNP’s listed in Table 1, three different datasets were generated. All three datasets have the same number of instances but different dimensionality of the feature vector. First dataset had 3 (sequential) features in it, second dataset had 18 (structural) features and the last dataset had all 21features in it. Receiver operating characteristic (ROC) score was calculated for 10-Fold cross validation and the mean of those score is represented in Table 2 and Figure 2 respectively.

The results clearly show that using structural and sequence based features together for SNP Prediction provides better results as compared to individual protein properties. It also suggests that hybrid features provide better results for any combination of features used. It also shows that random forest performs better among other classifiers used in this task.

For this task, data was collected for four different cancers that are breast cancer,

colorectal cancer, esophageal cancer and acute myeloid leukemia, see Table 1. It was observed that very few genes, such as TP53, were common for all types of cancers collected for this study and generally in all types of cancers. It can be seen from Table 1 that the number of detrimental SNPs is low as compared to the polymorphic SNP’s. The difference is almost three times between two types of SNPs. Prediction performance for every classifier for each disease was studied. Table 3 lists the performance of each classifier on both detrimental as well as polymorphic SNP.

The above table represents that SVM RBF performs better for esophageal and colorectal cancer and SVM linear performed better for acute myeloid leukemia, while all classifiers performed about equally well on breast cancer. It also shows that for polymorphic SNP prediction precision and recall is much better as compared to the detrimental SNPs. This may be attributed to the skewed data distribution. It is also noticeable that in terms of accuracy there is only 1% difference while using different classifiers.

Further, all the cancer types were lumped together to analyze their performance (shown in Table 4). It showed that random forest once again performed better. In order to further evaluate predictive power without using a fixed threshold to determine positive versus negative, receiver operating characteristic (ROC) score was calculated for all classifiers using 10-fold cross validation. The mean ROC score is represented in Figure 3. Results from mean ROC score show that except for acute myeloid leukemia for each disease random forest provides better score. And in general, all the ROC Scores are above 0.70.

Initially, it was hypothesized that SNP classification for individual disease will be better than that of combine diseases but results reflect the opposite. In order to further investigate couple of tasks were performed. It was noticed that there were six genes which are common and associated with cancer types selected for this study. These common genes were completely removed from data set and classification was performed. Results showed that mean ROC score for all the cases was less than 0.6 (shown in Figure 3). It provides a clue that if there is no common gene among diseases than SNP prediction for individual cancer type

will be better but in general almost all the cancers have certain common genes.

Another task was performed to see how training be affected if the combination of all disease SNP without redundancy i.e. only single instance of SNP occur in the final dataset when this gene is shared by more than one cancer type. In this case ROC score was similar to every individual cancer type SNP classification.

It was noticed and mentioned earlier that detrimental SNP are much less in number than the polymorphic SNPs. It produces an unbalanced dataset. To see what impact data would make if the number of detrimental SNP is equal to polymorphic SNPs. Number of SNPs for polymorphic class was reduced and then classification task was performed. It does not show any change in ROC score for best classifier but the F1-score for detrimental SNPs was rapidly increased from 0.45 to 0.86. This change in detrimental SNP evaluation can be seen from Table 5 as well as from the Figure 4. It was noticed that when data is balanced it does not affect polymorphic SNPs but classification of detrimental SNP is significantly improved.

Lastly mean ROC score was calculated using 10-fold cross validation for each classifier and found that random forest provides better results as compared to any other classifier. Note that there is no change in the mean ROC score for best classifier but SVM with its different kernels is performing better.

To assess the statistical significance for the difference between that set of combine cancers and the set of Acute Myeloid Leukemia, a t-test was performed on the ROC score of both datasets using random forest, and p-value is 0.007458. This concludes that random forest performs better than other classifiers when SNP’s prediction is done for any type of cancer.

nsSNP prediction was done at interacting site as well as non-interacting site. 3DID database (release: June 2015) was used to observe presence of a particular residue at interacting site. It was found that among 40 proteins associated with acute myeloid leukemia having solved 3D structure and nsSNP there are only 18 proteins which had information for their interacting and non-interacting residues recorded in the database. Two subsets were created for this problem one having SNPs at interacting residues and the other with SNPs at non-interacting residues. Data distribution is shown in Table 6.

Classification prediction was performed using same classifiers. Their performance with reference to precision, recall, F1-Scoreand accuracy is given below in Table 7. Data distribution is balanced for both subsets and thus it provides im-

proved results for both datasets when compared to task one datasets in terms of polymorphic and detrimental prediction.

While the overall performance has been dropped, there is an improved performance for prediction of detrimental SNP’s. Further, ROC score was determined for all classifiers for both datasets as shown in Figure 6. The upper panel is for all the classifier trained and tested for SNPs at interacting sites and the lower panel is for non-interacting site SNP’s. Mean ROC score for SVM RBF and SVM polynomial were same i.e. 0.86 for both datasets but in case of non-interacting site residues SVM polynomial is performing better with 0.66 score. It concludes that when overall performance of two datasets is considered SVM polynomial has better performance than any other classifier. Lastly to verify the statistical significance of the performance difference, a t-test was performed on the 10-fold cross validation of SVM polynomial ROC score and it was found that p-value is 0.020197, confirming the statistical significance of the difference.

In this analysis, we examine predicting SNPs effect in the context of haplotype,

i.e., the prediction of individual SNPs versus SNPs within a known haplotype. The search against database from HapMap Project and the other is UCSC genome browser only identified haplotypes from 14 genes from the gene pool associated with acute myeloid leukemia. Haplotypes were considered in pair only that means each single SNP in haplotype was compared to every haplotype allelic change within same gene including self-replication. In this task, two subsets were generated: one set consists of haplotypes pairs and the other set consists of all individual SNPs associated with genes involved in acute myeloid leukemia. Data distribution for these two subsets is given in Table 8.

For training 10-fold cross validation was applied to both datasets using SVM with three kernels and random forest. The results for this classification problem are shown in Table 9.

In Table 9 we can see easily that the best accuracy in predicting haplotype pair is 0.91, a significant increase over 0.82, the best accuracy in predicting individual SNPs. Also, we notice a clear advantage of Random forest for predicting haplotype pairs across the board on all four metrics, whereas SVM Polynomial per-

forms slightly better for predicting of individual SNPs. In particularly, it is worth noting that the F1-score for haplotype pair of detrimental phenotype is 0.81 by Random Forest classifier, which is a very impressive performance given that the datasets (Table 8) are quite skewed toward polymorphic phenotype and therefore present a greater challenge for correctly predicting the detrimental phenotype. The four metrics used in Table 9 all depend on a fixed threshold for prediction. ROC curve and score can evaluate a classifier’s predictive power and performance without relying on a specific prediction threshold. In Figure 7, ROC curves and scores are shown for haplotype SNP pairs (top panel) and individual SNPs (bottom panel). The two key observations from Table 9 are essential maintained: a) pairing SNPs in haplotype help improve phenotype prediction (ROC score = 0.95, achieved by RF), as compared to predicting phenotype for individual SNPs (ROC score = 0.81, achieved by SVM-RBF); b) while RF generally performs better, SVM-RBF has a slight edge in predicting individual SNPs.

Again, a t-test was performed on ROC scores from the 10-fold cross validation using Random Forest for haplotype pair versus individual SNPs. The p-value is

7.8 × 10⁻¹⁵, confirming the statistical significance of the difference.

Overall, it suggests that Random forest is the better classifier for most of the tasks performed in this study. An exception was seen for task 3, where SVM polynomial is providing better results.