Note that all stochastic approximation algorithms described in this section use an online standardization of the data. Indeed, in practical applications, an inadequate choice of step-size of these processes or the presence of heterogeneous data or outliers can lead to numerical explosion issues in the non-asymptotic phase of the stochastic approximation process. To avoid numerical explosions in the presence of heterogeneous data, an online standardization of the data is proposed [10] [11]; in the case of a data stream, the moments of the regression variables are a priori not known, but can be estimated online in order to perform the standardization. However, in this instance, the convergence of the stochastic approximation process is not ensured by classical theorems and was therefore proven in [10] in the case of linear regression, and in [11] in the case of logistic regression. Moreover, a too rapid decrease in step-size may reduce the speed of convergence in the non-asymptotic phase of the process. For this reason, following [12], the use of a decreasing piecewise constant step-size has been tested in [11].

Consider first the case of logistic regression. Let S be a random variable taking its values in { 0 , 1 } and R = ( R 1 ⋯ R p   1 ) ′ with R 1 , ⋯ , R p being random variables taking values in ℝ , m = ( E [ R 1 ] ⋯ E [ R p ] 0 ) ′ , R c = R − m , σ k the standard deviation of R k , Γ the diagonal square matrix with diagonal elements 1 σ 1 , ⋯ , 1 σ p , 1 , Z = Γ R c the standardized R vector, θ ( p + 1 , 1 ) the vector of parameters and h ( u ) = e u 1 + e u . The vector θ is the unique solution of the system of equations E [ ∇ x ln ( 1 + e Z ′ x e Z ′ x S ) ] = 0 , and thus of

E [ Z ( h ( Z ′ x ) − S ) ] = 0. (1)

Let ( ( R n , S n ) , n ≥ 1 ) denote an i.i.d. sample of ( R , S ) and for k ∈ { 1 , ⋯ , p } , R ¯ n k denote the average of the sample ( R 1 k , ⋯ , R n k ) of R k and ( V n k ) 2 = 1 n ∑ i = 1 n ( R i k − R ¯ n k ) 2 its variance (both computed recursively), R ¯ n the vector ( R ¯ n 1 ⋯ R ¯ n p   0 ) ′ and Γ n the diagonal matrix with diagonal elements 1 n n − 1 V n 1 , ⋯ , 1 n n − 1 V n p , 1 .

Assume that a mini-batch of m n new observatons ( R i , S i ) constituting an i.i.d sample of ( R , S ) is taken into account at step n. Denote M n = ∑ i = 1 n m i and I n = { M n − 1 + 1 , ⋯ , M n } . Define for j ∈ I n , Z ˜ j = Γ M n − 1 ( R j − R ¯ M n − 1 ) the vector R j standardized with respect to estimations of the means and variances of the components of R at step n − 1 . Recursively define the stochastic approximation process ( X n , n ≥ 1 ) and the averaged process ( X ¯ n , n ≥ 1 ) :

X n + 1 = X n − a n 1 m n ∑ j ∈ I n     Z ˜ j ( h ( Z ˜ ′ j X n ) − S j ) (2)

X ¯ n + 1 = 1 n + 1 ∑ i = 1 n + 1     X i = X ¯ n − 1 n + 1 ( X ¯ n − X n + 1 ) (3)

In the case of linear regression, the same type of process is used in [10] taking h ( u ) = u .

The following theorem is established for linear regression in [10] and for logistic regression in [11]. Assume:

(H1a) There is no affine relation between the components of R. (H1b) The moments of order 4 of R exist.

(H2a) a n > 0 , ∑ n = 1 ∞ a n = ∞ , ∑ n = 1 ∞ a n n < ∞ , ∑ n = 1 ∞ a n 2 < ∞ .

Theorem. Under H1a, H1b and H2a, ( X n , n ≥ 1 ) and ( X ¯ n , n ≥ 1 ) converge almost surely to θ .

In [10] and [11], these processes were compared to others (with or without online standardization, and with or without averaging) on real or simulated data. Empirical results showed the interest of using online standardization of the data to avoid numerical explosions as well as the better performance of averaged processes using a piecewise constant step-size (see Section 3).

Recursively define the stochastic approximation processes ( X n , n ≥ 1 ) and ( X ¯ n , n ≥ 1 ) :

X n + 1 = X n − a n 1 M n ∑ i = 1 n   ∑ j ∈ I i     Z ˜ j ( Z ˜ ′ j X n − S j ) ,   Z ˜ j = Γ M n ( R j − R ¯ M n ) (4)

X n + 1 = 1 n + 1 ∑ i = 1 n + 1     X i = X ¯ n − 1 n + 1 ( X ¯ n − X n + 1 ) (5)

Note that 1 M n ∑ i = 1 n ∑ j ∈ I i Z ˜ j Z ˜ ′ j = Γ M n ( 1 M n ∑ i = 1 n ∑ j ∈ I i R j R ′ j − R ¯ M n R ¯ ′ M n ) Γ M n and 1 M n ∑ i = 1 n ∑ j ∈ I i Z ˜ j S j = Γ M n ( 1 M n ∑ i = 1 n ∑ j ∈ I i R j S j − R ¯ M n S ¯ M n ) ,

S ¯ M n = 1 M n ∑ i = 1 M n S i . Thus, the updating does not necessitate storing previous data

since all empirical means and variances can be recursively computed. The same type of process would not be possible without storing the data for logistic regression, since in this case, Z ˜ j in Z ˜ j h ( Z ˜ ′ j X n ) should be updated for all j.

Denote by λ max the largest eigenvalue of the covariance matrix of R. Assume:

(H2b) ( a n = a < 1 λ max ) or ( a n → 0 , ∑ 1 ∞ a n = ∞ ) .

Theorem. Under H1a, H1b and H2b, ( X n , n ≥ 1 ) and ( X ¯ n , n ≥ 1 ) converge almost surely to θ .

This theorem was also proven in [10]. Empirical results again showed the interest of using online standardization of the data as well as all observations up to the current step to avoid numerical explosions and to increase the speed of convergence.

It is therefore possible to use the processes described in this section to update the predictors by linear discriminant analysis and logistic regression in the ensemble score, taking into account the sample of new data generated by the Poisson bootstrap at each step for each predictor.

Four datasets available on the Internet and one dataset derived from the EPHESUS study [17] were used, all of which have previously been utilized to test the performance of stochastic approximation processes with online standardized data in the case of online linear regression [10] and online logistic regression [11]. The Twonorm, Ringnorm, Quantum and Adult datasets are commonly used to test classification methods. Twonorm and Ringnorm, introduced by Breiman [18], contain simulated data with homogeneous variables. Quantum contains observed “clean” data, without outliers and with most of its variables on a similar scale. Adult and HOSPHF30D contain observed data with outliers, as well as heterogeneous variables of different types and scales. A summary of these datasets is provided in Table 1.

The following preprocessing was performed on the data:

· Twonorm and Ringnorm: no preprocessing.

· Quantum: a stepwise variable selection (using AIC) was performed on the 6197 observations without any missing value. The dataset with complete observations for the 12 selected variables was used.

· Adult2: from the Adult dataset, modalities of several categorical variables were merged (in order to obtain a larger number of observations for each modality) and all categorical variables were then replaced by sets of binary variables, leading to a dataset with 38 variables.

· HOSPHF30D: 13 variables were selected using a stepwise selection.

From each dataset, a data stream was simulated step by step by randomly drawing, with replacement, 100 new observations at each step. Online scores were then constructed and updated from these data streams.

For each dataset, a batch ensemble score was constructed using an adapted method from Duarte et al. [5] with the following parameters:

1) Two classification rules were used: linear discriminant analysis (LDA) and logistic regression (LR).

2) A total of 100 bootstrap samples were drawn for both rules (i.e. the same samples were used by each rule).

3) All available variables were included.

4) For each classification rule, the 100 associated predictors were aggregated by arithmetic mean and the coefficients subsequently normalized such that the score varied between 0 and 100 (as described in [5], Subsection 4.4.2).

5) The aggregation between the two intermediate scores S LDA and S LR was achieved by arithmetic mean: S = λ S LDA + ( 1 − λ ) S LR with λ = 0.5 .

The score obtained for each dataset was used as a “gold standard” to assess the convergence of the tested online processes (Figure 1).

N_a: number of available observations; N: number of selected observations; p_a: number of available parameters; p: number of selected parameters.

Types of processes: Three different types of stochastic processes ( X n ) were used as defined below.

1) “Classical” stochastic gradient (notation C_ _ _). At step n, card I n = m n observations ( R j , S j ) were taken into account and the process was updated recursively: X n + 1 = X n − a n 1 m n ∑ j ∈ I n Z ˜ j ( h ( Z ˜ ′ j X n ) − S j ) , with Z ˜ j the vector of standardized explanatory variables, S j ∈ { 0 , 1 } , h ( u ) = u for the LDA, and h ( u ) = e u 1 + e u for the LR.

2) “Averaged” stochastic gradient (notation A_ _ _): X ¯ n + 1 = 1 n + 1 ∑ i = 1 n + 1 X i .

3) Only in the case of the LDA: a process taking into account all of the previous observations ( R j , S j ) at each step until the current step, j ∈ I 1 ∪ ⋯ ∪ I n (final mention “all”) [10]: X n + 1 = X n − a n 1 M n ∑ i = 1 n ∑ j ∈ I i Z ˜ j ( Z ˜ ′ j X n − S j ) , Z ˜ j = Γ M n ( R j − R ¯ M n )

In all cases, the explanatory variables were standardized online (notation _S _ _): the principle and practicality of this method to avoid numerical explosions have already been shown [10] [11]. Indeed, for some datasets (Adult2, HOSPHF30D), processes with raw data led to a numerical explosion, contrary to those with online standardized data.

Step-size choice: Tested step-sizes a n were either:

1) Continuously decreasing: a n = c / ( b + n ) α (notation _ _ _V);

2) Constant: a n = 1 / p (with p the number of explanatory variables) (notation _ _ _C);

3) Piecewise constant [12]: a n = c / ( b + ⌊ n τ ⌋ ) α ( ⌊ . ⌋ being the integer part, τ the size of the level) (notation _ _ _P).

In all cases, α = 2 / 3 was taken as suggested by Xu [8] in the case of linear regression, b = 1 and c = 1 .

Tested processes: Six couples of processes were tested (Table 2). The latter were among those which performed best in the studies published for online LDA [10] and for online LR [11], or represented “usual” processes frequently used (apart from online data standardization). A total of 100 new observations were used per step. Each process was applied to each of the streams generated from the datasets.

In the notation describing a couple of processes, the first term is for the LDA and the second for the LR. For example, AS100Call-AS100P200 is the couple formed using for the LDA an averaged process (A) with online standardization of the data (S), 100 new observations per step (100), constant step-size (C), taking into account all the observations up to the current step (all), and for the LR an averaged process (A) with online standardization of the data (S), 100 new

All processes used online standardized data and 100 new observations per step.

observations per step (100) and a piecewise constant step-size with levels of size 200 (P200).

Note that the six couples of processes can be grouped in three pairs. In each pair, for the LDA part, one couple of processes uses 100 observations at each step and the other all observations up to the current step, the processes for the LR part being the same.

Convergence criterion: The convergence criterion used was the relative difference of the norms ‖ θ b − θ ^ N ‖ ‖ θ b ‖ between the θ b vector of coefficients obtained for the batch score and the θ ^ N vector of coefficients estimated by a

process after N iterations, the variables being standardized and the score being normalized to vary between 0 and 100 [5]. Convergence was considered to have occurred when the value of this criterion was less than the arbitrary threshold of 0.05. Three indicators were compared for each couple of processes: the criterion value for the synthetic score S_LDA obtained by aggregating the LDAs, the criterion value for the synthetic score S_LR obtained by aggregating the LRs, and the criterion value for the final score S.

In order to study the empirical convergence of the process, an analysis using a total of 10N observations was performed for each couple of processes. Since 100 observations are introduced at each step, the number of iterations of the process is N/10. Due to the stochastic nature of the processes studied, some variability is expected in the results. In order to evaluate this variability, the entire analysis using 10N observations was replicated 50 times for each couple of processes and for each dataset. The mean, standard deviation (SD) and relative standard deviation (RSD), i.e. the standard deviation divided by the mean, of the criterion values were studied for the intermediary and final scores. For each dataset, the average of the criterion values of all couples of processes was also studied.

For each replication and each dataset, the performance of the couples of processes were ranked from the best (lowest relative difference of the norms for the final score S) to the worst (highest relative difference of the norms for S). Thereafter, the mean rank of each couple and its associated standard deviation over the 50 replications were computed, first by dataset, and finally over all datasets.

To study the long-term convergence of the process, a single analysis using 100N observations was performed for each couple of processes. Again, for each dataset, the values of the criterion for the intermediary and final scores were studied, and the couples of processes were ranked from the best to the worst. The mean rank over all datasets was used to compare the global performance of the couples. All analyses were performed with R 3.6.2.

When replicating each couple of processes 50 times, the mean criterion values were lower than 0.05 for all couples of processes applied on Twonorm, Ringnorm and Quantum datasets (Table 3). However, only three out of six couples of processes converged for Adult2 (AS100C-AS100P200, AS100P50all-AS100P50 and AS100Call-AS100P200) as well as for HOSPHF30D (AS100P50-AS100P50, AS100P50all-AS100P50 and AS100Call-AS100P200). Note that for Twonorm, Ringnorm and Quantum, the maximum criterion values (not shown) for all couples of processes were always lower than 0.05 (i.e. even the worst performing processes still converged), whereas it was not the case for certain couples applied on Adult2 and HOSPHF30D.

Generally, intermediate LDA scores had smaller mean values, i.e. a faster convergence, than intermediate LR scores. However, the worst performing intermediary process was the LDA process AS100P50 applied on Adult2. In most cases, the mean criterion value for the final S score was between those of the two intermediate scores S_LDA and S_LR. For some couples of processes applied on some datasets (for instance AS100Call-AS100C on Adult2 or AS100C-AS100P200 on HOSPHF30D), this led to a convergence towards the reference of the final score while one of the intermediate scores had not yet converged according to the criterion.

When studying the rankings of the couples of processes over the 50 replications, the best couple of processes overall was AS100P50all-AS100P50. This couple was consistently among the three best couples, and had the best performance for three datasets. Note that the three best couples of processes across all datasets were those using all observations until the current step for the LDA intermediary scores.

*denotes criteria values < 0.05. SD: standard deviation; RSD: relative standard deviation.

The observed differences in the average criterion were greater between datasets rather than between couples of processes (Table 4). Indeed, the means of each couple of processes were the lowest for Twonorm and Ringnorm compared to the other datasets. Conversely, all couples had their worst results for HOSPHF30D. Generally, all couples of processes performed better when applied on simulated data (Twonorm and Ringnorm) rather than on observed data (Quantum, Adult2, HOSPHF30D). This was also true when comparing the standard deviations and RSDs.

When comparing the overall variability of the rankings between the couples, AS100P50all-AS100P50 and AS100Call-AS100P200, the two best performing couples of processes on average, also had the lowest standard deviations for the mean overall rank (1.17 and 1.12 respectively), while the couple with the largest standard deviation was CS100Vall-CS100V (1.72). It appears that the two best couples of processes consisted of averaged processes with a piecewise constant or a constant step-size for one process in the case of LDA. It also appears that these two processes in the case of LDA used all observations up to the current step instead of a mini-batch of observations.

When studying the couples of processes after using 100N observations (i.e. N iterations) in order to assess the “long-term” convergence, the final online S score was very similar (criterion value < 0.05) to the reference “batch” score for all of the couples on four of the five datasets tested (Table 5).

*denote criteria values < 0.05. First abbreviation: LDA process; Second abbreviation: LR process. Type of processes: C for classical SGD, A for ASGD. Data: R for raw data, S for online standardization of the data (1st number: number of new data per step). Step-size: V for continuously decreasing, C for constant, P for piecewise constant (2nd number: size of the steps of the piecewise constant step-size).

Only the AS100P50-AS100P50 couple applied to the Adult2 dataset did not converge after 100N iterations (criterion = 1.697). More precisely, the result for the LDA part of this couple differed substantially from its batch counterpart (criterion = 2.756), whereas the LR part appeared to converge to the batch LR part (criterion = 0.035).

For each couple of processes, the best performances were achieved for the Twonorm and Ringnorm datasets, which consist of simulated data. The worst performances were obtained for Adult2 and HOSPHF30D datasets, which contain observed data.

Although these results are not directly comparable with the average results using 10N observations presented in the previous subsection (since there was only one replication using 100N observations), it should be noted that the criterion values of all couples of processes on all datasets were lower after 100N observations than the mean values after 10N observations, except for the LDA and global scores of AS100P50-AS100P50 applied on Adult2.

When the couples of processes were ranked from best to worst for each dataset and the average ranks were calculated across all datasets (Table 5), the two worst performing couples were CS100Vall-CS100V and CS100V-CS100V, i.e. the only two couples using classical processes and a continuously decreasing step-size. The best couple was again AS100P50all-AS100P50. It appears that the two best couples of processes were the same as for 10N observations and that the two worst couples were classical processes with a continuously decreasing step-size.