Often in longitudinal studies, some subjects complete their follow-up visits, but others miss their visits due to various reasons. For those who miss follow-up visits, some of them might learn that the event of interest has already happened when they come back. In this case, not only are their event times interval-censored, but also their time-dependent measurements are incomplete. This problem was motivated by a national longitudinal survey of youth data. Maximum likelihood estimation (MLE) method based on expectation-maximization (EM) algorithm is used for parameter estimation. Then missing information principle is applied to estimate the variance-covariance matrix of the MLEs. Simulation studies demonstrate that the proposed method works well in terms of bias, standard error, and power for samples of moderate size. The national longitudinal survey of youth 1997 (NLSY97) data is analyzed for illustration.
In longitudinal studies, subjects who are likely to progress to a new state during the study are monitored over time. For example, in clinical trials, subjects who are at high risk of a certain disease are monitored and have follow-up visits. Some subjects complete all of their follow-up visits and their failure times are recorded. However, others miss their follow-up visits, and they may learn that the event of interest had already occurred when they came back. The event times for these patients are censored within the corresponding person-specific time intervals. Although there are multiple follow-up visiting intervals for each subject, researchers often use one particular interval that contains the true unknown failure time unless they had accurately determined the failure time. This is known as “partly interval-censored failure time data”. There are quite a few research works based on partly interval-censored data such as [
Another commonly available data type in longitudinal studies is called pooled repeated observations. Subjects have multiple follow-up visits as usual. From every visit, a subject obtains a binary outcome for the event of interest. All those repeated binary outcomes are pooled together to develop a model to analyze the effects of time-dependent covariates on the occurrence of the event. [
Now, we define pooled repeated partly interval-censored data. We have pooled repeated observations, but some binary outcomes and covariates are incomplete. They can only be determined with certain unknown probabilities within the corresponding specific follow-up visits. In this case, the analysis of such data requires a new method that combines a model that handles pooled repeated observations without censoring and a method that deals with partly or completely interval-censored data.
The main goal of this study is to estimate the effects of the time-dependent covariates on the occurrence of the event of interest (e.g., progression to a disease, becoming a frequent smoker, etc.). We extend the work of [
The organization of this paper is as follows. In Section 2, we present a logistic regression model for pooled repeated partly interval-censored data. In Section 3, we provide the details of computation of the MLEs of the regression parameter via EM algorithm and the variance estimation through the missing information principle. Section 4 displays the simulation study results. Section 5 illustrates an application to a real data set. Finally, Section 6 briefly summarizes what we have achieved and also discusses potential extensions of our work.
We consider a case of longitudinal studies, where subjects are at risk of an event of interest and have follow-up visits. Some subjects make complete follow-up visits, but others miss some of their follow-up appointments and come back after the event of interest has occurred. Whenever they miss a visit, both their binary outcome of the event of the interest and covariates are missing. Our proposed model estimates the effects of time-dependent covariates on the event of interest.
Let T i be the time subject i experiences the event of interest, i = 1, ⋯ , n . At the beginning of the study, every subject is assigned to the same follow-up visits, t j , j = 1, ⋯ , M . Let y i j be the indicator of whether or not subject i has had the event of interest in the jth interval given a subject was event-free through t j − 1 and x i j , the covariate at time t j − 1 . Since we are interested in modeling a binary outcome, we use a logit link to model the probability of event as in [
logit ( p i j ) = log ( p i j / ( 1 − p i j ) ) = α + β ′ x i j , (1)
where
p i j = P ( y i j = 1 | x i j , T i > t j − 1 ) . (2)
We construct the full (complete) log-likelihood, assuming as if there were no missing visits while subjects are in the study.
l = ∑ i = 1 n ∑ j = 1 M i [ − log ( 1 + exp ( α + β ′ x i j ) ) + y i j ( α + β ′ x i j ) ] , (3)
where M i is the index of the last time subject i was in the study.
Assume that the ith subject missed visits after time t L i and came back at t R i . L i is the index of the last time subject i made the visit and was event-free. R i is the index of the first time subject i was observed with the event of interest. Then y i L i = 0 , y i R i = 1 , and y i j is missing for L i + 1 ≤ j ≤ R i − 1 . For the subjects who do not miss visits, L i + 1 = R i . Whenever subjects miss visits, their covariate value, x i j , is also missing. We use the EM algorithm ( [
E-step: For individuals whose failure times are interval-censored, we need to estimate both y i j and x i j in the expression (3) for j ∈ { L i + 1, ⋯ , R i − 1 } .
x i j could be continuous or categorical ( [
x i j = θ 0 i + θ 1 i t j − 1 + ε i j , (4)
where ε i j ∼ N ( 0, σ ε 2 ) , c o v ( ε i j , ε i j ′ ) = 0, j ≠ j ′ . We estimate x i j by x ^ i j = θ ^ 0 i + θ ^ 1 i t j − 1 for L i + 1 ≤ j ≤ R i − 1 , where θ ^ 0 i and θ ^ 1 i are least squares estimators.
If x i j is ordinal, we assign numbers to corresponding categories. Then we again assume linear growth curve with fixed effects to estimate the missing x i j ’s. Let n c be the number of categories for this ordinal variable. For each individuali, the observed x i j ’s are used in model (4) to compute θ ^ 0 i and θ ^ 1 i . Then we compute x ^ i j = θ ^ 0 i + θ ^ 1 i t j − 1 as usual.
Next, we create n c − 1 thresholds in order to uniquely assign x ^ i j into one of the n c categories. Note that x ^ i j ∼ N ( θ 0 i + θ 1 i t j − 1 , σ x ^ i 2 ) and x ^ i = { x ^ i , L i + 1 , ⋯ , x ^ i , R i − 1 , x i , R i } . We use the quantiles of this normal distribution to define the thresholds. Since we need to compute σ ^ x ^ i 2 to define thresholds, we need at least three distinct observed covariate values, x i j ’s for each subject, otherwise, σ ^ x ^ i 2 would be undefined due to the zero degrees of freedom.
The observed ordinal covariates for some subjects do not include the entire ordinal categories. Therefore, the ordinal logistic regression model does not work for estimating ordinal covariates. In Appendix 2, we provide a detailed rationale for choosing fixed effects model, its extension in a general setting, and challenges with random effects model.
y ^ i j = E [ y i j | Y i , x ^ i , α , β , T i > t j − 1 ] = P [ T i = t j | t L i < T i ≤ t R i , Y i , x ^ i , α , β ] = ( p ^ i j p ^ i , L i + 1 + ∑ k = L i + 2 R i − 1 [ p ^ i k ∏ o = L i + 1 k − 1 ( 1 − p ^ i o ) ] + ∏ o = L i + 1 R i − 1 ( 1 − p ^ i o ) if j = L i + 1 , p ^ i j ∏ o = L i + 1 j − 1 ( 1 − p ^ i o ) p ^ i , L i + 1 + ∑ k = L i + 2 R i − 1 [ p ^ i k ∏ o = L i + 1 k − 1 ( 1 − p ^ i o ) ] + ∏ o = L i + 1 R i − 1 ( 1 − p ^ i o ) if j ∈ { L i + 2 , ⋯ , R i − 1 } , ∏ o = L i + 1 j − 1 ( 1 − p ^ i o ) p ^ i , L i + 1 + ∑ k = L i + 2 R i − 1 [ p ^ i k ∏ o = L i + 1 k − 1 ( 1 − p ^ i o ) ] + ∏ o = L i + 1 R i − 1 ( 1 − p ^ i o ) if j = R i , (5)
where
p ^ i j = exp ( α + β ′ x ^ i j ) 1 + exp ( α + β ′ x ^ i j ) (6)
This is an extension of a geometric-type experiment, where the probability of success (progression) changes at each follow-up visit, t j , L i + 1 ≤ j ≤ R i .
M-step: We find the values of α and β that maximize the expected value of log-likelihood in Equation (3), conditioned on the observed data. Therefore, we have
( α ^ , β ^ ) = arg max l α , β | y ^ i j , x ^ i j , (7)
where y ^ i j = y i j , x ^ i j = x i j , if uncensored.
Expressions (5)-(7) are repeated until convergence. As there are no closed forms for α ^ and β ^ , we used an optimization package optim in R to obtain ( α ^ , β ^ ) .
We apply Louis’ method for variance estimation using the notation in [
− ∂ 2 log P ( θ | W ) ∂ θ 2 = − ∫ v ∂ 2 log P ( θ | W , V ) ∂ θ 2 P ( V | θ , W ) d V − V a r ( − ∂ log P ( θ | W , V ) ∂ θ ) , (8)
where W is observed data, i.e., partly interval-censored pooled repeated observations. V is latent data, the true unknown counterpart of the interval-censored portion of W. θ | W is the observed posterior and θ | W , V is the augmented posterior.
The details of the expression (8) are provided in Appendix 3.
We considered n = 300 subjects who have M = 7 follow-up visits each. We generated covariates as follows:
x 1 i j ∼ N ( 5.8 + 0.3 t j − 1 ,0.1 ) .
x 2 i j ∼ N ( 0.4 + 0.15 t j − 1 ,0.1 ) .
x 1 i j represents a continuous covariate with larger values and faster growth rate over time, while x 2 i j represents one with smaller values and slower growth rate over time.
First, we generate n = 300 subjects who have complete follow-up visits. This makes the original complete data (OC), pooled repeated data. We randomly choose n 1 subjects out of these. This makes the exact data (E), a proper subset of the OC. For the remaining n 2 = n − n 1 subjects, we randomly designate some of their follow-up visits missing. This makes the pooled repeated interval-censored observations. The observed data (O), which is the pooled repeated partly interval-censored data, is the mix of pooled repeated data (E) and pooled repeated interval-censored data. We considered several values for n 1 and n 2 to cover different proportions of exact data.
We randomly sampled L i and R i for each patient. Note that for the exact data, we have R i = L i + 1 and for the pooled repeated interval-censored data, R i ≥ L i + 2 . Then for j = 1, ⋯ , L i , we have y i j = 0 and for j = R i , ⋯ , M , we have y i j = 1 . y i j is missing for j = L i + 1, ⋯ , R i − 1 in the pooled repeated interval-censored data. y i j is 1 when the ith subject at risk at the jth visit experiences the event of interest in thejth interval.
We computed the bias and variance for original complete data, exact data, and observed data based on B = 1500 replications. In addition, we investigated the power of our test.
We first considered the case where there was only one attribute in the model. The EM algorithm (Section 3.1) was used for the parameter estimation. The variance of the parameter estimator was calculated using Louis’ method (Section 3.2).
The results are shown in
Next, we computed the power of the test H 0 : β = β 0 vs. H 1 : β ≠ β 0 . We considered both one-dimensional covariate and two-dimensional covariates. We considered 3 sample sizes (100, 200, and 300), and for each of these sample sizes we ran B = 1000 replications of the test. The power was calculated as the proportion of times H 0 was rejected at 5% level of significance. Both
| ( n 1 , n 2 ) | B E | B O | B O C | σ E 2 | σ O 2 | σ O C 2 |
|---|---|---|---|---|---|---|
| (250, 50) | 0.559 | 0.241 | 0.021 | 0.043 | 0.028 | 0.017 |
| (200, 100) | 0.624 | 0.326 | 0.023 | 0.056 | 0.031 | 0.022 |
| (150, 150) | 0.769 | 0.457 | 0.025 | 0.059 | 0.034 | 0.022 |
| (100, 200) | 0.812 | 0.608 | 0.022 | 0.065 | 0.038 | 0.023 |
| (50, 250) | 0.838 | 0.809 | 0.023 | 0.078 | 0.044 | 0.026 |
| ( n 1 , n 2 ) | Parameter | σ O 2 | σ O C 2 | σ E 2 | σ B o o t 2 |
|---|---|---|---|---|---|
| (50, 250) | β 1 | 0.7312 | 0.1514 | 1.2726 | 0.1244 |
| β 2 | 0.0152 | 0.0049 | 0.0287 | 0.0035 | |
| (100 ,200) | β 1 | 0.4723 | 0.1617 | 0.5491 | 0.1157 |
| β 2 | 0.0108 | 0.0050 | 0.0160 | 0.0022 | |
| (150, 150) | β 1 | 0.2765 | 0.1463 | 0.3130 | 0.1175 |
| β 2 | 0.0076 | 0.0043 | 0.0087 | 0.0025 | |
| (200, 100) | β 1 | 0.1848 | 0.1442 | 0.2153 | 0.1391 |
| β 2 | 0.0064 | 0.0049 | 0.0077 | 0.0028 | |
| (250, 50) | β 1 | 0.1522 | 0.1342 | 0.1604 | 0.1274 |
| β 2 | 0.0053 | 0.0050 | 0.0059 | 0.0039 | |
| (270, 30) | β 1 | 0.1677 | 0.1621 | 0.1696 | 0.1497 |
| β 2 | 0.0052 | 0.0049 | 0.0055 | 0.0026 | |
| (290, 10) | β 1 | 0.1483 | 0.1461 | 0.1509 | 0.1321 |
| β 2 | 0.0047 | 0.0046 | 0.0048 | 0.0029 |
detecting the effect size of 0.75 for the proportional hazards model with a sample of size 300 using current status data. Considering that pooled repeated partly interval-censored data has more information than current status data, we fully
| ( n 1 , n 2 ) | β 1 | β 2 | Joint |
|---|---|---|---|
| (50, 250) | 0.878 | 0.853 | 0.835 |
| (100, 200) | 0.883 | 0.874 | 0.866 |
| (150, 150) | 0.899 | 0.886 | 0.881 |
| (200, 100) | 0.910 | 0.906 | 0.903 |
| (250, 50) | 0.947 | 0.931 | 0.936 |
agree with this better power result. The 95% coverage probabilities for different proportions of pooled repeated partly interval-censored data are shown in
In summary, even a small amount of pooled repeated interval-censored data within O does make our statistical inference more accurate and more powerful.
For more than 4 decades, the National Longitudinal Surveys (NLS) data have served as an important tool for economists, sociologists, and other researchers. The NLSY97 is a nationally representative sample of approximately 9000 youths who were 12 to 18 years old as of December 31, 1996. The NLSY97 is designed to document the transition from school to work and into adulthood. It collects extensive information about youths’ labor market behavior and educational experiences over time. In addition to educational and labor market experiences, the NLSY97 contains detailed information on many other topics. Some of the areas included in the data are criminal behavior, alcohol, and drug use. For the purpose of illustration of our methods, we use the NLSY97 data from 1997 to 2013 ( [
There are 8984 subjects in the data set. We analyze the 1822 subjects who did not smoke at the beginning of the study in 1997, but by the end of 2013 became frequent smokers (smoking for more than 10 days in a month). That is O. The response variable is defined as
y i j = ( 1, a frequent smoker 0, not a frequent smoker . (9)
Exact observations (E) are available in approximately 87.5% of those analyzed. The 1st covariate, x 1 i j , is the number of days an individual drank alcohol in the last 30 days. The 2nd covariate, x 2 i j , is an individual’s self-evaluation of “general state of health”. x 2 i j is defined as: 1 = excellent, 2 = very good, 3 = good, 4 = fair, and 5 = poor. The covariate effects are estimated by the EM algorithm in Section 0. The standard errors of these estimators are computed by Louis’ method in Section 0. The results are shown in
Additionally, we analyzed only E from O in order to see how much smaller the pooled repeated interval-censored data can help make the analysis more accurate. Another rationale for this is some practitioners often analyze only E due to the unavailability of software. The results are shown in
| α ^ | s e ( α ^ ) | β ^ 1 | s e ( β ^ 1 ) | β ^ 2 | s e ( β ^ 2 ) |
|---|---|---|---|---|---|
| −2.36 | 0.041 | 0.103 | 0.002 | 0.19 | 0.015 |
| α ^ | s e ( α ^ ) | β ^ 1 | s e ( β ^ 1 ) | β ^ 2 | s e ( β ^ 2 ) |
|---|---|---|---|---|---|
| −2.35 | 0.067 | 0.102 | 0.004 | 0.18 | 0.028 |
effect. Therefore, this data analysis reaffirms that even a small amount of pooled repeated interval-censored portion of O increases the sensitivity of the test.
We focused on developing a method to estimate the regression parameters and the variance-covariance matrix of those estimators for the pooled repeated partly interval-censored data logistic regression model. We employed the EM algorithm to estimate the parameters and missing information principle to estimate the variance-covariance matrix of those estimators.
Monte Carlo simulation demonstrates acceptable levels of bias, standard error, and power. To our knowledge, this is the first extensive power study for the pooled repeated partly interval-censored data logistic regression model. The simulation results suggest that in practice, one needs a sample of size around 300 to achieve an 80% power of the test to detect a very small effect size (0.45) for the regression parameter of interest. However, one needs a much smaller size, only around 200, for a bit larger effect size (0.55).
There are several potential extensions of our methods. Our methods can also be used when the predetermined follow-up visits were person-dependent. Our methods can be extended to handle correlated covariates by employing a ridge regression model ( [
Last but not least, we note that there are challenges in including either left-censoring or right-censoring. Refer to Appendix 1 for details.
The authors appreciate Dr. Alexis Dinno for introducing the data.
The authors declare no conflicts of interest regarding the publication of this paper.
Daneshi, N. and Kim, J.S. (2021) Maximum Likelihood Estimation for the Pooled Repeated Partly Interval-Censored Observations Logistic Regression Model. Open Journal of Statistics, 11, 230-242. https://doi.org/10.4236/ojs.2021.111012
In some special cases, the visiting time of some subjects in the data may have either right or left censoring. If a subject has not failed at the last visit ( y i L i = 0 ) and does not come back for the proceeding interview visits, then the subject’s time to the event of interest is right-censored. In this case L i = M i and R i = M . As NLSY predetermined M for all subjects, M plays the role of ∞ .
One may want to impute the covariate, x i j and reponse, y i j according to the procedures in Section 3.1. Unfortunately, extrapolating the covariates x i j for j > L i using the linear growth curve in Section 3.1 may well increase bias and variance.
If a subject’s first visit is at time k and the subject shows the symptoms of the event of interest, then both y i j and x i j are missing for j = 1, ⋯ , k − 1 , and y i k = 1 . Therefore, the covariate, x i j and response, y i j should be estimated for j ≤ k − 1 at E-step. We merely have L i = 0 , R i = k , and two observed covariate values x i 0 and x i k . Therefore, we cannot fit the subject-dependent growth curve to estimate the covariates at the missed visits.
In summary, there is no merit to include individuals whose event-times are either left-censored or right-censored when fitting a logistic regression model with pooled repeated observations.
In Section 3.1, we assumed that covariates have a linear growth curve with fixed effects. This was motivated by NLSY97 data. In NLSY97, follow-up interviews were relatively far apart (1 year). Additionally, some individuals had no change in their covariate values, e.g., some individuals had no drinking throughout the study. This motivated us to assume that for a given individual, the covariate values are uncorrelated at different follow-up visits, i.e., c o v ( ε i j , ε i j ′ ) = 0, j ≠ j ′ .
If the follow-up time intervals are relatively short and there are no constant covariate values for any individual over time, one may adopt a linear growth curve with fixed effects and autocorrelated errors. That is,
x i j = θ 0 i + θ 1 i t j − 1 + ε i j , (10)
where ε i j is an autoregressive process with lag 1, AR (1), ε i j ∼ N ( 0, σ ε 2 ) , c o v ( ε i , j , ε i , j + 1 ) = ρ , ρ ≠ 0 .
[
The variance estimation is based on Y i , the observed binary outcomes for subject i and the variability of missing response, y i j conditioned on x i , where x i = { x i 1 , x i 2 , ⋯ , x i , L i , x ^ i , L i + 1 , ⋯ , x ^ i , R i − 1 , x i , R i } . Let θ = ( α , β → ) , and z i = ( 1, x i ) . Then the complete information matrix in (8) can be computed by
∑ i z i z i T exp ( θ T z i ) ( 1 + exp ( θ T z i ) ) 2 . (11)
The missing information in (8) is computed by Monte Carlo simulations.
V a r ( − ∂ log P ( θ | W , V ) ∂ θ ) = E ( − ∂ log P ( θ | W , V ) ∂ θ ) 2 − [ E ( − ∂ log P ( θ | W , V ) ∂ θ ) ] 2 , (12)
where
E ( − ∂ log P ( θ | W , V ) ∂ θ ) ≈ 1 B ∑ b = 1 B [ − ∂ log P ( θ | W , V b ) ∂ θ ]
and
E ( − ∂ log P ( θ | W , V ) ∂ θ ) 2 ≈ 1 B ∑ b = 1 B [ − ∂ log P ( θ | W , V b ) ∂ θ ] 2 .