We present a methodology for constructing a short-term event risk score in heart failure patients from an ensemble predictor, using bootstrap samples, two different classification rules, logistic regression and linear discriminant analysis for mixed data, continuous or categorical, and random selection of explanatory variables to build individual predictors. We define a measure of the importance of each variable in the score and an event risk measure by an odds-ratio. Moreover, we establish a property of linear discriminant analysis for mixed data. This methodology is applied to EPHESUS trial patients on whom biological, clinical and medical history variables were measured.
In this study, we focus on the problem of constructing a short-term event risk score in heart failure patients based on observations of biological, clinical and medical history variables.
Numerous event risk scores in heart failure patients have been proposed in recent years, but one aspect is particularly important to consider in the construction of a score and in the relevance of the results obtained. This concerns the choice of classification models whose conditions of use may be restrictive. The most currently used classification models in these studies are logistic regression and Cox proportional hazard model. Quoting for example the Seattle Heart Failure Model (SHFM) risk score [
• SHFM risk score was derived in a cohort of 1153 patients with ejection fraction < 30% and New York Heart Association (NYHA) class III to IV and validated in 5 other cohorts of patients with similar characteristics. Area under ROC curve (AUC) at 1 year was 0.725 in resubstitution and ranged from 0.679 to 0.810 in the 5 validation cohorts.
• SPIM risk score was derived in a cohort of 6632 patients from the Eplerenone Post-Acute Myocardial Infarction Heart Failure Efficacy and Survival Study (EPHESUS) trial [
These two risk scores were developed using Cox proportional hazard model and characteristics available at baseline as explanatory variables. Overall, there are several limitations to using these risk scores. They were constructed using only data available at baseline. However, as many studies include inclusion criteria based on clinical or biological parameters measured at baseline, it is possible that some variables are not present in the score due to these inclusion criteria. For example, patients were included in the EPHESUS trial only if their potassium level at baseline was less than 5 mmol/L. This is a reason why potassium is not present in the SPIM score although this is an important parameter which moreover may evolve considerably over time. Concerning the model, the Cox proportional risk model assumes the proportionality of risks, an important condition not always obtained and verified.
In this study, we used a new approach:
• we develop a methodology for constructing a short-term event (death or hospitalization) risk score, taking into account the most recent values of the parameters and therefore the closest values of an event, in order to generate alerts and eventually immediately modify drug prescription; using EPHESUS trial data, we could only construct a score at 1 month in order not to have too few patients with event in the learning sample; but with the same methodology, a score could be constructed at a closer time;
• we use an ensemble predictor, that is more stable than a predictor built on a single learning sample, using bootstrap samples; this allows an internal validation of the score using AUC out-of-bag (OOB); moreover, we use two classification methods, logistic regression and linear discrimination analysis, and, in order to avoid overlearning, for each predictor we use a random selection of explanatory variables, after testing other methods of selection that did not give better results, the number of drawn variables being optimized after testing all possible choices;
• furthermore, our method of construction can be adapted to data streams: when patient data arrives continuously, the coefficients of variables in the score function can be updated online.
In the next section, we present how we defined the learning sample using the available data from EPHESUS trial and the list of explanatory variables used. In the third section, we state a property of linear discriminant analysis (LDA) for mixed data, continuous or categorical. In the fourth section, after presenting the methodology used to build a risk score and to reduce its variation scale from 0 to 100, we define a measure of the importance of variables or groups of correlated variables in the score and a measure of the event risk by an odds-ratio. In the fifth section, we describe the results obtained by applying our methodology to our data. The paper ends with a conclusion.
The database at our disposal was EPHESUS, a clinical trial that included 6632 patients with heart failure (HF) after acute myocardial infarction (MI) complicated by left ventricular systolic dysfunction (left ventricular ejection fraction < 40%) [
In this trial, each patient was regularly monitored, with visits at the inclusion in the study (baseline), 1 month after inclusion, 3 months later, then every 3 months until the end of follow-up. At each visit, biological, clinical parameters or medical history were observed. In addition, all adverse events (deaths, hospitalizations, diseases) that occurred during follow-up were collected.
To define the learning sample used to construct the short-term event risk score, we made the following working hypothesis: based on biological, clinical measurements or medical history on a patient at a fixed time, we sought to assess the risk that this patient has a short-term HF event. The individuals considered are couples (patient-month) without taking into account the link between several couples (patient-month) concerning the same patient. Therefore, it was assumed that the short-term future of a patient depends only on his current measures.
Firstly, we did a full review of the database in order to:
• identify the biological and clinical variables that were regularly measured at each visit,
• determine the medical history data that we could update from information collected during the follow-up.
We were thus able to define a set of 27 explanatory variables whose list is presented in
Then, we defined the response variable as the occurrence of a composite short-term HF event (death or hospitalization for progression of HF). In order to have enough events, we defined the short term as being equal to 30 days. Patient-months with a follow-up of less than 30 days and no short-term HF event during this incomplete follow-up period, were not taken into account.
There were finally 21,382 patient-months from 5937 different patients whose 317 with short-term HF event and 21,065 with no short-term event.
Denote A' the transposed of a matrix A.
In case of mixed data, categorical and continuous, a classical method to perform a discriminant analysis is:
1) perform a preliminary factorial analysis according to the nature of the data, such as multiple correspondence factorial analysis (MCFA) [
2) after defining a convenient distance, perform a discriminant analysis from the set of values of principal components, or factors.
See for example the DISQUAL (DIScrimination on QUALitative variables) method of Saporta [
Denote as usual T the total inertia matrix of a dataset partitioned in classes, W and B respectively its intraclass and interclass inertia matrix.
We show hereafter that when performing LDA with metrics T−1 or W−1, it is not necessary to perform a preliminary factorial analysis and LDA can be directly performed from the raw mixed data.
Metrics W−1 will be used in the following but can be replaced by T−1.
Let I = { 1 , 2 , ⋯ , n } a set of n individuals, partitioned in q disjoint classes I 1 , ⋯ , I q . Denote n k = c a r d ( I k ) , p k i the weight of ith individual of class I k ( i = 1 , ⋯ , n k ; k = 1 , ⋯ , q ) and P k = ∑ i = 1 n k p k i the weight of I k , with ∑ q k = 1 P k = 1 . p quantitative variables or indicators of modalities of categorical variables, denoted x 1 , ⋯ , x p , are observed on these individuals. Suppose that there exists no affine relation between these variables, especially for each categorical variable an indicator is removed.
For j = 1 , ⋯ , p , denote x k i j the value of x j for ith individual of class I k . Denote x k i the vector ( x k i 1 ⋯ x k i p ) ′ and g k the barycenter of the elements x k i for i ∈ I k :
g k = 1 P k ∑ i ∈ I k p k i x k i . (1)
Intraclass inertia ( p , p ) matrix W is supposed invertible:
W = ∑ k = 1 q ∑ i = 1 n k p k i ( x k i − g k ) ( x k i − g k ) ′ . (2)
A currently used distance in LDA d W − 1 ( a , b ) between two points a and b in ℝ p is such that:
d W − 1 2 ( a , b ) = ( a − b ) ′ W − 1 ( a − b ) . (3)
Suppose we want to classify an individual knowing the vector a of values of x 1 , ⋯ , x p . Principle of LDA is to classify it in I k such that d W − 1 2 ( a , g k ) is minimal.
Consider now new variables y 1 , ⋯ , y m affine combinations of x 1 , ⋯ , x p , with m ≥ p , such that:
y k i = A x k i + β , (4)
with y k i = ( y k i 1 ⋯ y k i m ) ′ , A a ( m , p ) matrix of rank p and β a vector in ℝ m .
Denote h k the barycenter of vectors y k i in ℝ m for i ∈ I k :
h k = 1 P k ∑ i ∈ I k p k i y k i = 1 P k ∑ i ∈ I k p k i ( A x k i + β ) = A g k + β , (5)
y k i − h k = A ( x k i − g k ) . (6)
Let Z the intraclass inertia ( m , m ) matrix of { y k i , i = 1 , ⋯ , n k ; k = 1 , ⋯ , q } :
Z = ∑ k = 1 q ∑ i ∈ I k p k i ( y k i − h k ) ( y k i − h k ) ′ = A W A ′ . (7)
The rank of Z is equal to the rank of A, p ≤ m . For m > p , the ( m , m ) matrix Z is not invertible. Then use in this case the pseudoinverse (or Moore-Penrose inverse) of Z, denoted Z+, which is equal to the inverse of Z when m = p , to define the pseudodistance denoted d Z + in ℝ m . The denomination pseudodistance is used because Z+ is not positive definite. Remind the definition of a pseudoinverse and two theorems [
Definition Let A a ( k , l ) matrix of rank r. The pseudo-inverse of A is the unique ( l , k ) matrix A+ such that:
1) A A + A = A ,
2) A + A A + = A + ,
3) ( A A + ) ′ = A A + ,
4) ( A + A ) ′ = A + A .
Theorem 1 Maximal rank decomposition
Let A a ( k , l ) matrix of rank r. Then there exist two full-rank (r) matrices, F of dimension ( k , r ) and G of dimension ( r , l ) ( r g ( F ) = r g ( G ) = r ) such that A = F G .
Theorem 2 Expression of A+
Let A = F G a full-rank decomposition of A. Then A + = G ′ ( F ′ A G ′ ) − 1 F ′ .
Prove now:
Proposition 1 d Z + 2 ( A a + β , A b + β ) = d W − 1 2 ( a , b ) .
Proof. Z = ( A W ) A ′ . AW and A' are of full-rank p. Applying theorem 2 yields:
Z + = A ( ( A W ) ′ A W A ′ A ) − 1 ( A W ) ′ (8)
= A ( A ′ A ) − 1 ( W A ′ A W ) − 1 ( A W ) ′ (9)
= A ( A ′ A ) − 1 W − 1 ( A ′ A ) − 1 A ′ . (10)
A ′ Z + A = W − 1 . (11)
Note that, when m = p , A is invertible and Z + = ( A W A ′ ) − 1 = Z − 1 .
d Z + 2 ( A a + β , A b + β ) = ( A ( a − b ) ) ′ Z + ( A ( a − b ) ) = ( a − b ) ′ W − 1 ( a − b ) . □
Thus:
Proposition 2 Let A a ( m , p ) matrix, m > p , of rank p and for k = 1 , ⋯ , q , i = 1 , ⋯ , n k , y k i = A x k i + β . The results of LDA of the dataset { x k i , k = 1 , ⋯ , q , i = 1 , ⋯ , n k } with the metrics W−1 on ℝ p are the same as those of LDA of the dataset { y k i , k = 1 , ⋯ , q , i = 1 , ⋯ , n k } with the pseudometrics Z + = ( A W A ′ ) + .
Applications
Denote x i j the value of the variable x j for individual i belonging to I, i = 1 , ⋯ , n , j = 1 , ⋯ , p and x i = ( x i 1 ⋯ x i p ) ′ the vector of values of ( x 1 , ⋯ , x p ) for individual i. Denote p i the weight of individual i, such that ∑ i = 1 n p i = 1 . To perform a factorial analysis of the dataset { x i , i = 1 , ⋯ , n } , the difference between two individuals i and i' is measured by a distance d ( i , i ′ ) defined on ℝ p associated to a metrics M, such that
d 2 ( i , i ′ ) = ( x i − x i ′ ) ′ M ( x i − x i ′ ) . (12)
Denote X the ( n , p ) matrix whose element ( i , j ) is x i j . Denote D the diagonal ( n , n ) matrix whose element ( i , i ) is p i .
Perform a factorial analysis of ( X , M , D ) , for instance principal component analysis (PCA) for continuous variables or MCFA for categorical variables or MDFA for mixed data. Suppose X of rank p. Denote u j = ( u j 1 ⋯ u j p ) ′ a unit vector of the jth principal axis. Denote c j = X M u j = ( c 1 j ⋯ c n j ) ′ the jth principal component. Denote U the ( p , p ) matrix ( u 1 ⋯ u p ) and C the ( n , p ) matrix ( c 1 ⋯ c p ) = X M U ; as u 1 , ⋯ , u p are M-orthonormal, U ′ M U = I and
C = X M U ⇔ X = C U ′ ⇔ for i = 1 , ⋯ , n , x i = U c i (13)
⇔ for i = 1 , ⋯ , n , c i = U ′ M x i . (14)
Using the metrics of intraclass inertia matrix inverse, LDA from C is equivalent to LDA from X.
Suppose now that the variable x p + 1 = 1 − x p is introduced; when x p is the indicator of a modality of a binary variable, x p + 1 is the indicator of the other modality. Then:
( x i 1 ⋮ x i p x i p + 1 ) = ( u 1 1 ⋯ u p 1 ⋮ ⋱ ⋮ u 1 p ⋯ u p p − u 1 p ⋯ − u p p ) ( c i 1 ⋮ c i p ) + ( 0 ⋮ 0 1 ) (15)
Denote X1 the ( n , p + 1 ) matrix whose element ( i , j ) is x i j . LDA from C with the metrics of intraclass inertia matrix inverse is equivalent to LDA from X1 with the metrics of intraclass inertia matrix pseudoinverse.
For instance:
1) If x 1 , ⋯ , x p are continuous variables, LDA from X is equivalent to LDA from C obtained by PCA, such as normed PCA, or generalized canonical correlation analysis (gCCA) [
2) If x 1 , ⋯ , x p are indicators of modalities of categorical variables, and if MCFA is performed to obtain C, LDA from C with the metrics of intraclass inertia matrix inverse is equivalent to LDA from X with the metrics of intraclass inertia matrix pseudoinverse.
3) Likewise, if x 1 , ⋯ , x p are continuous variables or indicators of modalities of categorical variables, and if MDFA [
Consider the problem of predicting an outcome variable y, continuous (in the case of regression) or categorical (in the case of classification) from observable explanatory variables x 1 , ⋯ , x p , continuous or categorical.
The principle of an ensemble method [
• in regression: the average of predictions y i ^ ;
• in classification: the rule of the majority vote or the average of the estimations of a posteriori class probabilities.
The ensemble predictor is expected to be better than each of the individual predictors. For this purpose [
• each single predictor must be relatively good,
• single predictors must be sufficiently different from each other.
To build a set of predictors, we can:
• use different classifiers,
• and/or use different samples (e.g. by bootstrapping, boosting, randomizing outputs) [
• and/or use different methods of variables selection (e.g. ascending, stepwise, shrinkage, random) [
• and/or in general, introduce randomness into the construction of predictors (e.g. in random forests [
In Random Generalized Linear Model (RGLM) [
• a bootstrap sample is drawn,
• a fixed number of variables are randomly selected,
• the selected variables are rank-ordered according to their individual association with the outcome variable y and only the top ranking variables are retained,
• an ascending selection of variables is made using Akaike information criterion (AIC) [
Tufféry [
• a bootstrap sample is drawn,
• variables are randomly selected,
• an ascending variables selection is performed using AIC [
Note that this method is in fact a particular case of RGLM method.
Present now the method used in this study to check the stability of the predictor obtained on the entire learning sample.
The steps of the method for constructing an ensemble predictor are presented in the form of a tree (
At first step, n1 classifiers are chosen.
At second step, n2 bootstrap samples are drawn and are the same for each classifier.
At third step, for each classifier and each bootstrap sample, n3 modalities of random selection of variables are chosen, a modality being defined either by a number of randomly drawn variables or by a number of predefined groups of correlated variables, which are randomly drawn, inside each of which a variable is randomly drawn.
At fourth step, for each classifier, each bootstrap sample and each modality of random selection of variables, one method of selection of variables is chosen, a stepwise or a shrinkage (LASSO, ridge or elastic net) method.
This yields a set of n 1 × n 2 × n 3 predictors, which are aggregated to obtain an ensemble predictor.
To assess accuracy of the ensemble predictor, the percentage of well-classified is currently used. But this criteria is not always convenient, especially in the
present case of unbalanced classes. We decided to use AUC. AUC in resubstitution being usually too optimistic, we used AUC OOB [
Two classifiers were used: logistic regression and LDA with metrics W−1. Other classifiers were tested but not retained because of their less good results, such as random forest-random input (RF-RI) [
1000 bootstrap samples were randomly drawn.
Three modalities of random selection were retained, firstly a random draw of a fixed number of variables, secondly and thirdly a random draw of a fixed number of predefined groups of correlated variables followed by a random draw of one variable inside each drawn group. The number of variables or of groups drawn was determined by optimization of AUC OOB.
Fourth step did not improve prediction accuracy and was not retained.
Denote n the total number of patient-months and p the number of variables. Denote x i j the value of variable x j for patient-month i, i = 1 , ⋯ , n , j = 1 , ⋯ , p . Each patient-month i is represented by a vector x i = ( x i 1 ⋯ x i p ) ′ in ℝ p .
In the case of two classes Ω 1 and Ω 0 , whose barycenters are respectively denoted g 1 and g 0 , Fisher linear discriminant function
S 1 ( x ) = ( x − g 1 + g 0 2 ) ′ W − 1 ( g 1 − g 0 ) = α ′ 1 x + β 1 (16)
can be used as score function. For logistic regression, the following score function can be used:
S 2 ( x ) = ln P ( Ω 1 | X = x ) P ( Ω 0 | X = x ) = α ′ 2 x + β 2 . (17)
Remind that, in the case of a multinormal model with homoscedasticity (covariance matrices within classes are equal), when P ( Ω 1 ) = P ( Ω 0 ) , logistic model is equivalent to LDA [
S 2 ( x ) = ln P ( Ω 1 | X = x ) P ( Ω 0 | X = x ) = ln P ( Ω 1 ) P ( Ω 0 ) + S 1 ( x ) = S 1 ( x ) . (18)
So we used the following method to aggregate the obtained predictors:
1) the score functions obtained by LDA are aggregated by averaging; denote now S1 the averaged score;
2) likewise the score functions obtained by logistic regression are aggregated by averaging; denote S2 the averaged score;
3) a combination of the two scores, λ S 1 + ( 1 − λ ) S 2 is defined, 0 ≤ λ ≤ 1 ; a value of λ that maximizes AUC OOB is retained; denote S0 the optimal score obtained by this method.
If s is an optimal cut-off, the ensemble classifier is defined by:
If S 0 ( x ) > s , x is classified in Ω 1 ; (19)
if not, x is classified in Ω 0 . (20)
The variation scale of the score function S 0 ( x ) was reduced from 0 to 100 using the following method. Denote:
S 0 ( x ) = α ′ 0 x + β 0 = ∑ j = 1 p α 0 j x j + β 0 . (21)
Denote for j = 1 , ⋯ , p :
P j = | α 0 j | ( max 1 ≤ i ≤ n x i j − min 1 ≤ i ≤ n x i j ) (22)
and
P = ∑ j = 1 p P j = ∑ j = 1 p | α 0 j | ( max 1 ≤ i ≤ n x i j − min 1 ≤ i ≤ n x i j ) . (23)
Let m j the minimal value of the variable x j if α 0 j > 0 , or its maximal value if α 0 j < 0 .
Denote S ( x ) the “normalized” score function, with values from 0 to 100, defined by:
S ( x ) = 100 P ∑ j = 1 p α 0 j ( x j − m j ) (24)
= 100 ∑ j = 1 p α 0 j ( x j − m j ) ∑ k = 1 p | α 0 k | ( max 1 ≤ i ≤ n x i k − min 1 ≤ i ≤ n x i k ) (25)
= α ′ x + β , with ( β α 1 ⋮ α p ) = ( − 100 P ∑ j = 1 p α 0 j m j 100 α 0 1 P ⋮ 100 α 0 p P ) . (26)
Explanatory variables are not expressed in the same unit. To assess their importance in the score, we used “standardized” coefficients, multiplying the coefficient of each variable in the score by its standard deviation. These coefficients are those associated with standardized variables and are directly comparable. For all variables, the absolute values of their standardized coefficient, from the greatest to the lowest, were plotted on a graph. The same type of plot was used for groups of correlated variables, whose importance is assessed by the sum of absolute values of their standardized coefficients.
Define a risk measure associated to a score s by an odds-ratio O R 1 ( s ) :
O R 1 ( s ) = P ( Y = 1 | S > s ) P ( Y = 0 | S > s ) P ( Y = 0 ) P ( Y = 1 ) = P ( S > s | Y = 1 ) P ( S > s | Y = 0 ) = S e ( s ) 1 − S p ( s ) . (27)
An estimation of O R 1 ( s ) , also denoted O R 1 ( s ) , is n 1 n 0 × N 0 N 1 with n k = # { S > s } ∩ { Y = k } and N k = # { Y = k } , k = 0 , 1 .
Note that:
• O R 1 ( s ) decreases when S e ( s ) decreases and S p ( s ) is constant. In practice, the decrease will be much smaller when there are many observations;
• O R 1 ( s ) is not defined when S p ( s ) is equal to 1.
For these reasons, the following definition can also be used:
O R 2 ( s ) = max t ≤ s : O R 1 ( t ) < ∞ O R 1 ( t ) . (28)
Note that O R 1 is the slope y/x of the line joining the origin to the point ( x , y ) of the ROC curve. In the case of an “ideal” ROC curve, supposed continuous above the diagonal line, assuming that there is no vertical segment in the curve, this slope increases from point ( 1,1 ) , corresponding to the minimal value of score, to point ( 0,0 ) , corresponding to its maximal value; the case of a vertical segment (Se decreases, Sp is constant), occurring when the score of a patient with event is between those of two patients without event, is particularly visible in the case of a small number of patients and also justifies the definition of O R 2 , whose curve fits that of O R 1 .
For very high score values, when n0 or n1 are too small, the estimation of O R 1 is no longer reliable. A reliability interval of the score could be defined, depending on the values of n0 and n1.
To avoid problems related to the presence of outliers or extreme data, all continuous variables were winsorized using the 1st percentile and the 99th percentile of each variable as limit values [
Among qualitative variables, two are ordinal: the NYHA class with 4 modalities and the number of myocardial infarction (no. MI) with 5 modalities. In order to preserve the ordinal nature of these variables, we chose to use an ordinal encoding. For NYHA, we therefore associated 3 binary variables: NYHA ≥ 2, NYHA ≥ 3 and NYHA ≥ 4. In the same way, for the no. MI, we considered 4 binary variables: no. MI ≥ 2, no. MI ≥ 3, no. MI ≥ 4 and no. MI ≥ 5.
On the other hand, continuous variables were transformed in the context of logistic regression. For each continuous variable, a linearity test was performed using the method of restricted cubic splines with 3 knots [
| Variable | p-value 1 | Transformation function f ( x ) | p-value 2 |
|---|---|---|---|
| Hemoglobin | 0.090 | ||
| Hematocrit | 0.007 | x − 2 | 1.00 |
| ePVS | 0.69 | ||
| Creatinine | 0.21 | ||
| eGFR Cockroft-Gault | <0.0001 | ln ( x ) | 0.40 |
| eGFR MDRD | <0.0001 | x − 0.5 | 0.79 |
| eGFR CKD-EPI | 0.005 | ln ( x ) | 0.90 |
| Sodium | 0.056 | ||
| Potassium | <0.0001 | ( x − 4.6 ) 2 | 0.47 |
| Heart rate | <0.0001 | ( x − 60 ) 2 | 0.91 |
| Systolic BP | <0.0001 | ( x − 140 ) 2 | 0.34 |
| Diastolic BP | <0.0001 | ( x − 84 ) 2 | 0.49 |
| Mean BP | <0.0001 | ( x − 102 ) 2 | 0.66 |
| Weight | 0.090 | ||
| BMI | 0.060 | ||
| Age | 0.64 |
At 5% level, linearity was rejected for 9 of 16 continuous variables. For each of these 9 variables, we represented graphically the relationship between the logit (natural logarithm of the ratio probability of event/probability of non-event) and the variable. An example of graphical representation is given for potassium: we observe a quadratic relationship between the logit and the potassium (
For hematocrit and the three variables of eGFR, the relationship is clearly monotonous. So we considered some simple monotonic transformation functions as f ( x ) = x a with a ∈ { − 2, − 1, − 0.5,0.5,1,2 } or f ( x ) = ln ( x ) , then we retained for each variable the transformation for which the likelihood under univariable logistic model was maximal (minimal p-value).
For other variables not checking linearity, namely potassium, the three blood pressure measures (systolic, diastolic and mean), and heart rate, the relationship between the logit and the variable was rather quadratic. We therefore applied a quadratic transformation function ( X − k * ) 2 with k ∗ an optimal value determined by maximizing likelihood under univariable logistic model. To compare, we also used the criterion of maximal AUC to determine an optimal value. These results are presented in
Also note that the transformation applied to potassium allows to take into account both hypokalemia and hyperkalemia, two different clinical situations pooled here that may increase the risk of death and/or hospitalization measured by the score.
| Variable | “Raw” variable X | Criterion 1 Maximizing likelihood for ( X − k * ) 2 | Criterion 2 Maximal AUC for ( X − k * ) 2 | ||
|---|---|---|---|---|---|
| AUC | k* | AUC | k* | AUC | |
| Systolic BP | 0.5818 | 140 | 0.5995 | 140 | 0.5995 |
| Diastolic BP | 0.5834 | 84 | 0.5970 | 84 | 0.5970 |
| Mean BP | 0.5915 | 102 | 0.6091 | 101 | 0.6094 |
| Potassium | 0.5312 | 4.6 | 0.5665 | 4.7 | 0.5676 |
| Heart rate | 0.6473 | 60 | 0.6521 | 60 | 0.6521 |
To verify that the transformation of the variables was good, a linearity test for each transformed variable was performed according to the previously detailed principle. All tests are not significant at the 5% level (see
As a first step, we applied our methodology with the following parameters:
• use of a single classification rule, logistic regression ( n 1 = 1 ),
• draw of 1000 bootstrap samples ( n 2 = 1000 ),
• random selection of variables according to a single modality ( n 3 = 1 ).
Three modalities for the random selection of variables were defined:
• 1st modality: random draw of m variables among 32,
• 2nd modality: random draw of m groups among 18, then one variable from each drawn group,
• 3rd modality: random draw of m groups among 24, then one variable from each drawn group.
The groups of variables considered for each modality are presented in
For each modality, an ensemble score was built for all possible values of m and the one that gave maximal AUC OOB was selected. In
The ensemble score by logistic regression, denoted S 2 ( x ) , obtained by averaging the three ensemble scores that we constructed, gave slightly better results, with AUC OOB of 0.8649.
The same methodology was used by simply replacing the classification rule (logistic regression) by LDA for mixed data and keeping the same other settings.
| Variables | Modality 1 | Modality 2 | Modality 3 |
|---|---|---|---|
| Systolic BP | - | Blood pressure | Blood pressure |
| Diastolic BP | - | ||
| Mean BP | - | ||
| Heart rate | - | - | - |
| Weight | - | Obesity | Obesity |
| BMI | - | ||
| NYHA ≥ 2 | - | NYHA | - |
| NYHA ≥ 3 | - | - | |
| NYHA ≥ 4 | - | - | |
| Age | - | - | - |
| Gender | - | - | - |
| Caucasian | - | - | - |
| Hemoglobin | - | Hematology | Hematology |
| Hematocrit | - | ||
| ePVS | - | ||
| Creatinine | - | Renal function | Renal function |
| eGFR Cockroft-Gault | - | ||
| eGFR MDRD | - | ||
| eGFR CKD-EPI | - | ||
| Potassium | - | - | - |
| Sodium | - | - | - |
| Hypertension | - | - | - |
| Diabetes | - | - | - |
| Hosp. for HF | - | Previous hosp. for HF | - |
| Hosp. for HF the previous month | - | - | |
| Hosp. for CV cause the previous month | - | - | - |
| Hosp. for other CV cause the previous month | - | - | - |
| Hosp. for non CV cause the previous month | - | - | - |
| No. MI ≥ 2 | - | No. MI | - |
| No. MI ≥ 3 | - | - | |
| No. MI ≥ 4 | - | - | |
| No. MI ≥ 5 | - | - |
| Parameters | AUC in resubstitution | AUC OOB |
|---|---|---|
| Modality 1 m = 19 | 0.8716 | 0.8616 |
| Modality 2 m = 14 | 0.8688 | 0.8611 |
| Modality 3 m = 8 | 0.8691 | 0.8634 |
| Ensemble score | 0.8728 | 0.8649 |
Again, for each modality, we searched the optimal m parameter. The obtained results are presented in
As for logistic regression, the best results were obtained for the third modality, with AUC OOB equal to 0.8638.
| Parameters | AUC in resubstitution | AUC OOB |
|---|---|---|
| Modality 1 m = 12 | 0.8679 | 0.8614 |
| Modality 2 m = 5 | 0.8673 | 0.8631 |
| Modality 3 m = 7 | 0.8690 | 0.8638 |
| Ensemble score | 0.8707 | 0.8654 |
The ensemble score by LDA, denoted S 1 ( x ) , yielded better results with AUC OOB equal to 0.8654.
The final ensemble score denoted S 0 ( x ) , obtained by synthesis of the two ensemble scores S 1 ( x ) and S 2 ( x ) presented previously, provided the best results with AUC equal to 0.8733 in resubstitution and 0.8667 in OOB.
This ensemble score corresponds to the one obtained by applying our methodology with the following parameters:
• two classification rules are used, logistic regression and LDA for mixed data ( n 1 = 2 ),
• 1000 bootstrap samples are drawn ( n 2 = 1000 ),
• m variables are randomly selected according to three modalities ( n 3 = 3 ).
The scale of variation of the score function S 0 ( x ) was reduced from 0 to 100 according to the procedure described previously. We denote this “normalized” score S ( x ) .
In
To have a global view of the importance of the variables in the “normalized” score, we represented on a graph the absolute value of standardized coefficient associated with each variable, from the largest value to the smallest (see
The same type of graph was made to represent the importance of the groups of variables in configuration 2 defined by the sum of the absolute values of the “standardized” coefficients associated with the variables of the group, from the largest sum to the smallest (see
| Variables | Ensemble score S 0 ( x ) | Ensemble score “normalized” S ( x ) | ||
|---|---|---|---|---|
| coefficient | Standardized coefficient | coefficient | Standardized coefficient | |
| Constant | −0.210 | −0.210 | 44.60 | 44.60 |
| Hemoglobin | −0.0580 | −0.0871 | −0.478 | −0.717 |
| Hematocrit−2 | 314.00 | 0.0442 | 2590.00 | 0.364 |
| ePVS | 0.131 | 0.107 | 1.07 | 0.877 |
| Creatinine | 0.00349 | 0.0964 | 0.0287 | 0.794 |
| Ln (eGFR Cockroft-Gault) | −0.0940 | −0.0396 | −0.774 | −0.326 |
| eGFR MDRD−0.5 | −0.892 | −0.0183 | −7.34 | −0.151 |
| Ln(eGFR CKD-EPI) | −0.175 | −0.0590 | −1.44 | −0.486 |
| Sodium | −0.0232 | −0.0861 | −0.191 | −0.709 |
| (Potassium-4.6)2 | 0.301 | 0.0889 | 2.48 | 0.732 |
| (Heart rate-60)2 | 0.000696 | 0.221 | 0.00572 | 1.82 |
| (Systolic BP-140)2 | 0.000125 | 0.0729 | 0.00103 | 0.600 |
| (Diastolic BP-84)2 | 0.0000985 | 0.0220 | 0.000810 | 0.181 |
| (Mean BP-102)2 | 0.000201 | 0.0545 | 0.00165 | 0.448 |
| Weight | 0.0000258 | 0.000374 | 0.000212 | 0.00308 |
| BMI | 0.00196 | 0.00844 | 0.0161 | 0.0695 |
| Age | 0.00449 | 0.0506 | 0.0370 | 0.416 |
| Caucasian | −0.162 | −0.0455 | −1.33 | −0.374 |
| Male | 0.0434 | 0.0195 | 0.357 | 0.161 |
| Hypertension | 0.136 | 0.0665 | 1.12 | 0.547 |
| Diabetes | 0.0904 | 0.0422 | 0.744 | 0.347 |
| Hosp. for HF | 0.549 | 0.175 | 4.52 | 1.44 |
| Hosp. for HF the previous month | 1.53 | 0.185 | 12.60 | 1.52 |
| Hosp. for CV cause the previous month | 0.403 | 0.168 | 3.31 | 1.38 |
| Hosp. for non-CV cause the previous month | 0.361 | 0.0486 | 2.97 | 0.400 |
| Hosp. for other CV cause the previous month | 0.104 | 0.0205 | 0.852 | 0.169 |
| No. MI ≥ 2 | 0.0840 | 0.0377 | 0.692 | 0.310 |
| No. MI ≥ 3 | 0.118 | 0.0323 | 0.973 | 0.266 |
| No. MI ≥ 4 | 0.242 | 0.0342 | 1.99 | 0.281 |
| No. MI ≥ 5 | 0.0443 | 0.00370 | 0.365 | 0.0304 |
| NYHA ≥ 2 | 0.309 | 0.150 | 2.54 | 1.23 |
| NYHA ≥ 3 | 0.612 | 0.194 | 5.04 | 1.60 |
| NYHA ≥ 4 | 1.65 | 0.142 | 13.60 | 1.16 |
We represented the variation of n 0 , n 1 , S e ( s ) , 1 − S p ( s ) , O R 1 ( s ) and O R 2 ( s ) according to the score s (
| s | n 0 | n 1 | S e ( s ) | 1 − S p ( s ) | O R 1 ( s ) | O R 2 ( s ) |
|---|---|---|---|---|---|---|
| s* = 23.7094 | 4527 | 250 | 0.7918 | 0.2149 | 3.6844 | 3.6844 |
| 11.8489 | 19683 | 317 | 1.0000 | 0.9344 | 1.0702 | 1.0702 |
| 13.7320 | 17684 | 316 | 0.9968 | 0.8395 | 1.1874 | 1.1874 |
| 15.1105 | 15684 | 316 | 0.9968 | 0.7446 | 1.3388 | 1.3388 |
| 16.3630 | 13686 | 314 | 0.9905 | 0.6498 | 1.5245 | 1.5245 |
| 17.6044 | 11689 | 311 | 0.9811 | 0.5549 | 1.7679 | 1.7679 |
| 18.9050 | 9697 | 303 | 0.9558 | 0.4604 | 2.0762 | 2.0762 |
| 20.4525 | 7709 | 291 | 0.9180 | 0.3660 | 2.5081 | 2.5081 |
| 22.3007 | 5729 | 271 | 0.8549 | 0.2720 | 3.1428 | 3.1428 |
| 24.7670 | 3766 | 234 | 0.7382 | 0.1788 | 4.1278 | 4.1278 |
| 28.8573 | 1822 | 178 | 0.5615 | 0.0865 | 6.4884 | 6.4884 |
| 33.2656 | 872 | 128 | 0.4038 | 0.0414 | 9.7431 | 9.8363 |
| 38.2403 | 414 | 86 | 0.2713 | 0.0197 | 13.7706 | 13.7706 |
| 49.1933 | 70 | 30 | 0.0978 | 0.0033 | 29.4283 | 31.5217 |
| 55.1424 | 28 | 22 | 0.0694 | 0.0014 | 50.4112 | 50.4112 |
| 58.0352 | 14 | 16 | 0.0505 | 0.0007 | 70.8812 | 74.7575 |
We represented the variation of odds-ratio O R 1 and O R 2 in this reliability interval (
example, P ( Y = 1 | S > 40 ) P ( Y = 0 | S > 40 ) is about 15 times higher than P ( Y = 1 ) P ( Y = 0 ) .
In this article, we presented a new methodology for constructing a short-term event risk score in heart failure patients, based on an ensemble predictor built using two classification rules (logistic regression and LDA for mixed data), 1000 bootstrap samples and three modalities of random selection of variables. This score was normalized on a scale from 0 to 100. AUC OOB is equal to 0.8667. Note
that an important variable such as potassium that does not appear in other scores (as SPIM risk score) is taken into account in this score.
Moreover, we defined a measure of the importance of each variable and each group of variables in the score and defined an event risk measure by an odds-ratio.
Due to the nature of the data available (data obtained from the EPHESUS study), we had to define the short term to 30 days in order to have enough patients with HF event. It would be better to have data of patients with shorter intervals, in order to have data the closest possible of an event and eventually improve the quality of the score. When such data will be available, it will be interesting to apply the same methodology to construct a new score.
Furthermore, we proved a property of linear discriminant analysis for mixed data.
Finally, this methodology can be adapted to the case of a data stream. Suppose that new data for heart failure patients arrives continuously. Data can be allocated to bootstrap samples using Poisson bootstrap [
Results incorporated in this article received funding from the Investments for the Future program under grant agreement No ANR-15-RHU-0004.
The authors declare no conflicts of interest regarding the publication of this paper.
Duarte, K., Monnez, J.-M. and Albuisson, E. (2018) Methodology for Constructing a Short-Term Event Risk Score in Heart Failure Patients. Applied Mathematics, 9, 954-974. https://doi.org/10.4236/am.2018.98065