1. Introduction

OALibJ

Open Access Library Journal

2333-9705

Scientific Research Publishing

10.4236/oalib.1106030

OALibJ-98561

Articles

Biomedical&Life Sciences Business&Economics Chemistry&Materials Science Computer Science&Communications Earth&Environmental Sciences Engineering Medicine&Healthcare Physics&Mathematics Social Sciences&Humanities

A Comparison of Three Models in Multivariate Binary Longitudinal Data Analysis:

Hissah

Alzahrani

₁

Mathematical Science Department, College of applied sciences, Umm Al-Qura University, Mekkah, Saudi Arabia

03022020

070211326, December 201925, February 2020 28, February 2020

2014

This work is licensed under the Creative Commons Attribution International License (CC BY). http://creativecommons.org/licenses/by/4.0/

Multivariate longitudinal data analysis plays an important role in many biomedical and social problems. In this article, we present three methods for analyzing multiple and correlated binary outcomes; each one can be beneficial for determined aims. We review methods one and two, and we proposed method three. The three methods estimate the marginal means using the GEE approach for multivariate binary longitudinal data. The first method addresses the question of estimating one group of covariate parameters for many binary outcomes while accounting for their multivariate structure. The second method addresses the question of estimating the covariate parameters for each binary outcome separately. The third method is an estimation of the covariate parameters for each combination of outcomes. Our goal is to investigate the differences among the parameter estimations of the three methods. In the application element, we present a follow-up study (Florida Dental Care Study) that measured three binary outcomes and five covariates at four intervals. The FDCS study is useful explanation of the variation between outcomes since the outcomes were highly correlated.

Binary GEE Longitudinal Many Outcomes Correlation

1. Introduction

Many of the recent medical experiments and social research are characterized by multiple outcomes. In this research, we focus on the multivariate binary outcomes in longitudinal data. It is a general approach of the univariate longitudinal data when we have more than one outcome. Each individual i has a vector of responses for different outcomes, k = 1 , 2 , ⋯ , K , which are measured at different times or occasions, t = 1 , 2 , ⋯ , T i , and has cluster size T i = T i K . To simplify the notations, we will refer to T i as T which is the number of occasions or visit numbers over all observations as following:

Y i = [ Y i 11 , Y i 21 , ⋯ , Y i T 1 , Y i 12 , Y i 22 , Y i 32 , ⋯ , Y i T 2 , ⋯ , Y i 1 K , Y i 2 K , ⋯ , Y i T K ] T

There are multiple methods for analyzing multivariate binary longitudinal data, which are different depending on the research aims. Let’s start by referring “outcome” to the longitudinal readings within the same dependent variable and “response” for any longitudinal reading of any dependent variables without specification. If the outcomes are uncorrelated, then we can analyze them as univariate longitudinal data separately. This independence assumption rarely happens in real applications because the outcomes, which are taken from the same observation, are highly probably to be correlated. The best statistical analysis of multiple outcomes should account for the correlation among the outcomes and the occasions since it is longitudinal data [1] [2] [3] . The three methods of analyzing longitudinal data are:

1) Reducing many outcomes into one summary outcome which results in estimating a unique set of regression coefficients [3] .

2) Analyzing each outcome separately with account for the correlations, which results in estimating a set of regression coefficients for each outcome [4] .

3) Analyzing the outcomes jointly and accounting for the correlations, which results in estimating a set of regression coefficients for each combination case of them (the proposed method).

Our investigation method of the longitudinal data via the marginal models is using GEE approach. The GEE approach of longitudinal data analysis is constructed for discrete and continuous outcomes¹. Then, applied the alternating logistic regression in multivariate binary model, which is dependent on modeling the association between responses by the odds ratio. The specification of the odds ratio matrix R, is the most important point on the GEE analysis of the

marginal models. We need to estimate ( K T ) ( K T − 1 ) 2 = ( K T 2 ) parameters. It is

a large correlation matrix and it will rapidly increase when we have more parameters in R (correlation matrix). Since our interests are in binary multivariate longitudinal outcomes, then it is more natural to use the odds ratio to describe the association between the binary responses [3] . There are two applied methods to reduce the parameter estimations of the within subject association matrix R. The two methods are Kronecker product and regression model for pairwise odds ratio. The two methods are equivalent to each other and have approximately the same results. We will focus on the regression model for pairwise odds ratio to be method 1 in the comparison part.

Also, a SAS macro is built to analyze multivariate binary longitudinal outcomes using Kronecker product [4] . In this paper, we will consider the regression model for pairwise odds ratio to be the statistical tool of the first aim (method 1) [3] , and Kronecker product for the second aim (method 2) [4] . In the next section is an overview of the three methods. The application is in Section 3. The last section is about the discussion.

2. Methods2.1. Method 1: Regression Modeling for Odds Ratio

The GEE model is g ( μ i ) = X i β where E ( Y i ) = μ i , and X i is the covariate matrix, assuming the vector of responses for subject i is related to the covariates via logit link function. The primary advantage of using GEE in multivariate binary data is working in two-way marginal relationships between the pairs of the responses for any dimension of outcomes and occasions without high order associations. The penalty of this advantage is the cluster size which is T × K. It will cause a larger correlation matrix R since the dimension of R matrix is TK × TK. R

is a diagonal and symmetric matrix, thus we need to estimate ( T K 2 ) . For example,

if we have 5 outcomes obtained at 5 occasions, then we need to estimate 300 parameters, because that we need to obtain a parsimonious model of working

correlation R to be less than ( T K 2 ) .

As we mentioned previously, we can use the odds ratio instead of the correlation since the responses are binary. Then, the matrix R will consist of pairwise log odds ratios [3] . It is based on fitting a regression model for marginal pairwise

odds ratio to estimate less parameters than ( T K 2 ) [3] . Let γ be a vector of size ( T K 2 ) . of all non-redundant pairwise odds ratio in R. Here the goal is reducing

the length of the vector γ . Consider the matrix of the odds ratio R consist of three odds ratio types:

1) Let v t k . t k ′ be the inter-outcome odds ratio which compares the outcome k with the outcome k ′ at time t:

v t k . t k ′ = P ( Y t k = 1 , Y t k ′ = 1 ) P ( Y t k = 0 , Y t k ′ = 0 ) P ( Y t k = 1 , Y t k ′ = 0 ) P ( Y t k = 0 , Y t k ′ = 1 ) (1)

2) Let a α t k , t ′ k be the intra-outcome odds ratio which compares outcome k at time t with the same outcome at time t ′ :

α t k , t ′ k = P r ( Y t k = 1 , Y t ′ k = 1 ) P r ( Y t k = 0 , Y t ′ k = 0 ) P r ( Y t k = 1 , Y t ′ k = 0 ) P r ( Y t k = 0 , Y t ′ k = 1 ) (2)

3) Let T t k , t ′ k be the cross odds ratio which compares the outcome k at time t with outcome k at time t ′ :

T t k , t ′ k ′ = P r ( Y t k = 1 , Y t ′ k ′ = 1 ) P r ( Y t k = 0 , Y t k ′ = 0 ) P r ( Y t k = 1 , Y t ′ k ′ = 0 ) P r ( Y t k = 0 , Y t ′ k ′ = 1 ) (3)

The general regression model that relates the inter-outcome (υ), the intra-outcome (α) and the cross odds ratio (τ) to matrix z via log link function [2] , is:

log ( O R ) = Z γ (4)

where is γ a parsimonious vector of the odds ratio parameters that relates the ( K T 2 ) pairwise log odds ratio to Z. Here Z is a fixed indicator matrix that

specifies the outcomes, times and interactions between them which are under consideration. Using the regression model of (4) helps us to model the three types of association in matrix Z.

2.2. Method 2: Kronecker Product

To apply method 2 we used Kronecker Product approach [4] , they built a SAS macro to estimate separate sets of regression coefficients for each binary outcome. This method depends on the GEE estimation of the marginal models. The strength of this macro is creating a covariate design to allow for separate regression coefficients for each outcome. The GEE model is logit ( μ i ) = X i β where E ( Y i ) = μ i and X i is the covariate matrix and assuming the vector of responses for subject i is related to the covariate via logit link function.

Using Kronecker product of X matrix and K dimensional identity matrix be used to generate regression coefficients for each outcome. This method is beneficial to fit more than one longitudinal outcome using GEE approach, and accounts for the correlation among the outcomes. The results are separated estimation coefficients for each outcome, which is helpful for the second aim.

2.3. Method 3: The Proposed Approach

Let Y i t k denotes the value of k^th binary outcome measured at t occasion or visit number for individual i, where i = 1 , 2 , ⋯ , N , k = 1 , 2 , ⋯ , K , and t = 1 , 2 , ⋯ , T . Let X i t c denotes the value of the covariate C that is measured at occasion t for individual i where c = 1 , 2 , ⋯ , C (C is the number of covariate variables).

Let Y i ~ Bernolli ( π 1 ) , Y 2 ~ Bernolli ( π 2 ) , ⋯ , Y k ~ Bernolli ( π K ) . Assuming the outcomes are obtained in the same time for same C covariates. We will collapse K outcomes into one outcome Z. Since Multinomial trial process is a simple generation of Bernoulli trial processes, then the new variable Z has multinomial distribution with J = 2 K possible outcomes, Z 1 , Z 2 , ⋯ , Z J Suppose each possible outcome can occur with probability p 1 , p 2 , ⋯ , p J , then the probability of Z 1 occurs m 1 times, Z 2 occurs m 2 times, ⋯ , Z J occurs m J times is following the probability mass function:

f ( m 1 , m 2 , ⋯ , m J ) = ( N m 1 , m 2 , ⋯ , m J ) p 1 m 1 p 2 m 2 ⋯ p J m J (5)

where ∑ j − 1 J p j = 1 , ∑ j − 1 J m j = N . Then, Z = ( Z 1 , Z 2 , ⋯ , Z J ) T has a multinomial distribution with p = ( p 1 , p 2 , ⋯ , p J ) parameters. To collapse many binary vectors into one multinomial vector, we suggest to do transformation from a binary coding system to a decimal coding system. Then we analyze the new outcome Z instead of K binary vectors. Since the data is longitudinal, we consider a correlation among the new variable Z i t . Then we use an appropriate exciting method for analyzing correlated nominal multinomial responses.

For correlated multinomial responses, there are many methods using random effects model and marginal models. For random effect models, Hedeker adopts a method in Bock’s model [5] . For marginal models, the GEE approach avoids specification of the distribution of the multinomial outcome by adopting a “working” correlation matrix. However, a GEE approach using a local odds ratio parameterization is one appropriate method to analyze the correlated data [6] . In the multinomial responses, we set up one of the possible outcomes to be a reference level and estimate the parameter in term of J − 1 remaining levels compared to the reference level.

After estimating the parameters of nominal multinomial responses Z i t , we must move to the “decoding” step. The decoding step is to transform each category in Z i t j response to its original definition. For example, the parameter estimation β ^ 1 for category z = 3 is for the original code “011”. Then, expresses the estimated parameter of the logit probabilities of accruing outcome 1 and outcome 2 simultaneously. In conclusion, using the GEE approach for nominal multinomial correlated responses could be helpful to analyze the joint distribution of many binary longitudinal outcomes.

3. Application

The Florida Dental Care Study (FDCS) is a longitudinal study of oral health and dental service utilization. It is conducted in four centers in North Florida in the United States [7] . The sample size is 873 subjects who had baseline interview exams and four clinical examination interviews at 6, 12, 18 and 24 months past baseline visit. By the end of 24 months, 87.5% of 873 remained in the study. 3.3% refused to participate, 1.1% unable to participate due to medical issues, 3.3% were deceased and 4% were dropout. The issue of bias was discussed by comparing the characteristics of the patients who remained in the study at 24 months, with those who did not for any reason [8] . Also, there are three binary outcomes that were measured over four time intervals (0 - 6, 6 - 12, 12 - 18, 18 - 24 months). In this application, a subset of five covariates were chosen in Table 1. The three binary outcomes measured in this study are “problem oriented visit”, “dental cleaning” and “dental check up”. At the end of each interval, each subject was asked whether he had visited a dentist within the past 6 months “problem oriented visit” and whether this dental visit was for a “dental cleaning” or “check up”. The three binary outcomes are coded as (0 = “No”, 1 = “Yes”). First, we consider the following three models.

Model 1:

logit ( μ i ) = β 0 + β 1 X i r a + β 2 X g e n d e r + β 3 X c a v i t + β 4 X l o o s e + β 5 X a b l e

Model 2:

logit ( μ i ) = ∑ k = 1 3 ( β k 0 + β k 1 X i r a + β k 2 X g e n d e r + β k 3 X c a v i t + β k 4 X l o o s e + β k 5 X a b l e )

Model 3:

logit ( μ i ) = ∑ j = 1 7 ( β j 0 + β j 1 X i r a + β j 2 X g e n d e r + β j 3 X c a v i t + β j 4 X l o o s e + β j 5 X a b l e )

j = { 1 when ( Y i 1 = 0 ) & ( Y i 2 = 0 ) & ( Y i 3 = 1 ) 2 when ( Y i 1 = 0 ) & ( Y i 2 = 1 ) & ( Y i 3 = 0 ) 3 when ( Y i 1 = 0 ) & ( Y i 2 = 1 ) & ( Y i 3 = 1 ) 4 when ( Y i 1 = 1 ) & ( Y i 2 = 0 ) & ( Y i 3 = 0 ) 5 when ( Y i 1 = 1 ) & ( Y i 2 = 0 ) & ( Y i 3 = 1 ) 6 when ( Y i 1 = 1 ) & ( Y i 2 = 1 ) & ( Y i 3 = 0 ) 7 when ( Y i 1 = 1 ) & ( Y i 2 = 1 ) & ( Y i 3 = 1 )

The reference category is 0 when ( Y i 1 = 0 ) & ( Y i 2 = 0 ) & ( Y i 3 = 0 ). Before running the three models, we measured the association between the binary responses using the odds ratio instead of the correlation. The odds ratio and the 95% confidence intervals are in Table 2. The association parameters are larger than one and significant for all the odds ratios because the 95% confidence intervals exclude the value one. Then, we expect a large differences between the three models. The strongest association is between “cleaning” and “check up” outcomes.

We ran the three models with no constraints in the log odds ratio, then we got 66 unique estimated odds ratio (#α = 12, #υ = 18, #τ = 36) in models 1 and 2. The results of the three models are in Tables 3-5. The covariates Cavit and Able are significant in model 1 while Gender and IRA are close and Loose is not. In model 2, the covariates are estimated for each outcome separately. The Gender covariate still not significant for all. However, The Loose parameter estimation is changed completely from model 1 to model 2 to be significant for all. This change

Table 1 Covariates variables

Covariate	Definition
IRA	(1) The subject goes to a dentist regularly or occasionally whether or not has a problem, (0) The subject did go to a dental check-up once a year or more often in the previous 5 years.
Gender	(1) Female, (0) Male.
Cavit	(1) The subject reported having cavities (tooth decay) in the previous 6 months, (0) if not.
Loose	(l) The subject had a loose tooth, (0) had not
Able	(1) The subject able to pay unexpected US$ 500 dental bill, but with difficulty, (0) not able to pay

Table 2 Estimated odds ratio

Odds Ratio	Estimate	95% CI
OR (Y₁, Y₂)	1.977	(0.512, 0.851)
OR (Y₁, Y₃)	2.081	(0.551, 0.915)
OR (Y₂, Y₃)	42.147	(3.497, 3.985)

Table 3 Model 1 results

Parameter	Estimate	Stderr	z	P-value
Intercept	−1.1302	0.0651	−17.36	<0.0001
IRA	0.2379	0.1273	1.87	0.0616
Gender	−0.1649	0.0901	−1.83	0.0673
Cavit	0.6235	0.0795	7.84	<0.0001
Loose	0.0818	0.081	1.01	0.3126
Able	−0.2715	0.086	−3.16	0.0016

Table 4 Model 2 Results

Parameter	Estimate	Stderr	z	P-value
Intercept₁	−1.306	0.0827	−15.79	<0.0001
IRA_y₁	0.1447	0.1768	0.82	0.4129
Gender_y₁	−0.1904	0.116	−1.64	0.1006
Cavit_y₁	1.0137	0.1059	9.58	<0.0001
Loose_y₁	0.5367	0.1191	4.51	<0.0001
Able_y₁	−0.1008	0.1077	−0.94	0.3491
Intercept₂	−1.1788	0.0979	−12.04	<0.0001
IRA_y₂	0.4656	0.1836	2.54	0.0112
Gender_y₂	−0.1652	0.1323	−1.25	0.2118
Cavit_y₂	0.3919	0.1113	3.52	0.0004
Loose_y₂	−0.4316	0.1245	−3.47	0.0005
Able_y₂	−0.4329	0.1257	−3.44	0.0006
Intercept₃	−0.8009	O. 0945	− 8.48	<0.0001
IRA_y₃	0.328	0.1898	1.73	0.0839
Gender_y₃	−0.1609	0.1277	−1.26	0.2075
Cavit_y₃	0.2894	0.1065	2.72	0.0066
Loose_y₃	−0.4101	0.117	−3.51	0.0005
Able_y₃	−0.5959	0.1245	−4.79	<0.0001

Table 5 Model 3 results

Parameter	Estimate	Stderr	z	P-value
Intercept₁	2.31644	0.15412	15.0297	<2e⁻¹⁶
IRA_y₁	−0.44468	0.28851	−1.5413	0.12325
Gender_y₁	0.25399	0.20448	I. 2421	0.2142
Cavit_y₁	−1.59776	0.16413	−9.7347	<2e⁻¹⁶
Loose_y₁	0.30088	0.22217	1.3543	0.17565
Able_y₁	0.56503	0.19391	2.9139	0.00375
Intercept₂	0.2618	0.20957	1.2493	0.21157
IRA_y₂	−0.31291	0.41567	−0.7528	0.45157
Gender_y₂	0. 32744	0.26896	1.2174	0.22343
Cavit_y₂	−2.22835	0.32981	−6.7565	<2e⁻¹⁶
Loose_y₂	0.40086	0.27408	1.4626	0.14359
Able_y₂	0.08664	0.2673	0. 3241	0.74585
Intercept₃	−1.64295	0.3346	−4.9101	<2e⁻¹⁶
IRA_y₃	0. 34044	0.53157	0.6404	0.52188
Gender_y₃	0.09152	0.38695	0.2365	0.81304
Cavit_y₃	−1.67004	0. 46435	−3.5965	0.00032
Loose_y₃	0.4977	0.41352	1.2036	0.22876
Able_y₃	0.89498	0.36628	2.4434	0.01455
Intercept₄	1.08689	0.14938	7.2758	<2e⁻¹⁶
IRA_y₄	0.02381	0.3449	0.069	0. 94496
Gender_y₄	0.06488	0.21685	0.2992	0.76478
Cavit_y₄	−2.02666	0.21162	−9.577	<2e⁻¹⁶
Loose_y₄	−0.51225	0.25708	−1.9926	0.0463
Able_y₄	−0.07702	0.20244	−0.3805	0.7036
Intercept₅	0.7074	0.17148	4.1254	<4e⁻⁵
IRA_y₅	−0.22746	0.32465	−0.7006	0.48353
Gender_y₅	0.09493	0.2191	0.4333	0.66481
Cavit_y₅	−0.92957	0.19173	−4.8484	<2e⁻¹⁶
Loose_y₅	0.98945	0.22902	4.3204	<2e⁻⁵
Able_y₅	0.54333	0.20724	2.6218	0.00875
Intercept₆	−0.26746	0.214	−1.2498	0.21136
IRA_y₆	−0.19855	0.43232	−0.4593	0.64604
Gender_y₆	−0.20593	0.29128	−0.707	0.47957
Cavit_y₆	−0.77173	0.26226	−2.9426	0.00325
Loose_y₆	0.68872	0.29773	2.3133	0.02071
Able_y₆	−0.06312	0.26756	−0.2359	0.81351
Intercept₇	−1.41215	0.25849	−15.463	<2e⁻¹⁶
IRA_y₇	0.10559	0.46596	0.2266	0.82072
Gender_y₇	0.1786	0.32314	0.5527	0.58046
Cavit_y₇	−0.83001	0.33215	−12.4989	0.01246
Loose_y₇	0.94458	0.35612	2.6524	0.00799
Able_y₇	0.53696	0.30734	1.7471	0.08062

due to separate the effect of the covariate for each outcome, when they are highly correlated. In model 3, we find Gender parameter estimation still not significant for the all. Loose parameter estimation becomes significant only for outcome Y 1 , Y 1 & 2 , Y 1 & 3 and Y 1 , 2 & 3 which means the high association between the “problem visit” outcome and the others hides the effect of Loose covariate in the outcome in model 1. To get a closer look at the parameters’ differences in the three models, we aggregate the parameter estimations of all models for each covariate in Figure 1.

The dotplot in Figure 1 indicates many points. First, the variation of the parameter estimation between model 1, model 2 and model 3 are relatively. The variation in IRA and Gender parameter estimations over the three models is limited due to insignificant results. IRA parameter estimation is close to being significant in model 1, then model 2 explains the source of the effect which is the second outcome. The third model clarifies the effect after breaking up the responses for each joint case and finding that there is no significant estimations. In a contrary case, the parameter estimation of Able covariate starts significant in model 1 and 2 expect for the first outcome, then model 3 illustrates some significant parameters related to the significant association between outcomes 1 and 3. The Loose variable begins as not significant in model 1 but it is significant in model 2 for all outcomes while model 3 clarifies the sources of effect which is the associations between the three outcomes. The case of the intercept is clearly starting significant in model 1 and model 2, but model 3 explains the cause of the significance which are outcomes 1 and 3. The parameter of Cavit covariate shows how the third model changes the sign of the parameter to negative after using the coding method.

In summary, The FSCS study of three outcomes “problem visit”, “cleaning” and “check up” and four intervals is a good example of highly correlated responses that are used to investigate parameter estimation differences of the three models. Over this example, we see that the effect of each covariate on the responses varies over the three models. The most significant covariates over the three models are Cavity, Loose tooth, and the ability to pay an unexpected US$500 dental bill, but with difficulty.

4. Discussion

In this article, we review two existing methods for analyzing binary multivariate longitudinal data and proposed a third method. In most longitudinal experiments, there are more than one outcomes are obtained from the subjects over many occasions. Usually, the researcher is interested in investigating the effect of many independent variables on these outcomes. We applied three different methods to be beneficial in this case. The aim of this article is to give a more general overview of analyzing binary multivariate longitudinal data in case of expecting a correlation among the outcomes other than the correlation among the repeated measurements.

If the researcher is interested in estimating one group the estimated effects of covariates for all outcomes and accounting of the multivariate structure, then the first method is the most appropriate one. However, if the researcher is eager to separate the effects of covariates on the outcomes, then the second method is appropriate. The joint analysis of all of the outcomes in the third method is beneficial to investigate the relationship between the correlated outcomes and how the covariates are affecting on them. The third method consists of encoding and decoding stages. The encoding stage is a step of defining each case of associations between the outcomes into simple number. It is an idea of store of all the possible combinations of the outcomes into one outcome. We converted three binary outcomes into one multinomial outcome. The decoding stage is the last step of returning each code to its original definition.

Then, as a result, the joint analysis in method 3 is limited to small number of outcomes because the number of parameters of estimation is rapidly increasing when the number of outcomes is increased. It would be beneficial for future work to improve the third method to reduce the dimension of the parameters estimations by adding a filtering stage between the encoding and decoding steps. We presented a simulation part for many scenarios and application. The FCSD example is applied to the three methods and we saw how the parameter estimations of the three methods were changed over covariates. Each method explains more details starting from model 1, and moving to models 2 and 3; each method answers a different question. In conclusion, the ultimate chosen model depends on the research goal.

Conflicts of Interest

The author declares no conflicts of interest regarding the publication of this paper.

Cite this paper

Alzahrani, H. (2020) A Comparison of Three Models in Multivariate Binary Longitudinal Data Analysis: Application to FDCS Study. Open Access Library Journal, 7: e6030. https://doi.org/10.4236/oalib.1106030

References1

Liang, K.Y. and Zeger, S.L. (1986) Longitudinal Data Analysis Using Generalized Linear Models. Biometrika, 73, 13-22. https://doi.org/10.1093/biomet/73.1.13

Carey, V., Zeger, S. and Diggle, P. (1993) Modelling Multivariate Binary Data with Alternating Logistic Regressions. Biometrika, 80, 517-526.https://doi.org/10.1093/biomet/80.3.517

O’Brien, L. and Fitzmaurice, G. (2004) Analysis of Longitudinal Multiple-Source Binary Data Using Generalized Estimating Equations. Journal of the Royal Statistical Society: Series C (Applied Statistics), 53, 177-193.https://doi.org/10.1046/j.0035-9254.2003.05296.x

Shelton, B., Gilbert, G., Liu, B. and Fisher, M. (2004) A SAS Macro for the Analysis of Multivariate Longitudinal Binary Outcomes. Computer Methods and Programs in Biomedicine, 76, 163-175. https://doi.org/10.1016/j.cmpb.2004.05.005

Hedeker, D. (2003) A Mixed-Effects Multinomial Logistic Regression Model. Statistics in Medicine, 22, 1433-1446. https://doi.org/10.1002/sim.1522

Touloumis, A., Agresti, A. and Kateri, M. (2013) GEE for Multinomial Responses Using a Local Odds Ratios Parameterization. Biometrics, 69, 633-640.https://doi.org/10.1111/biom.12054

Gilbert, G., Duncan, R.P., Kulley, A., Coward, T. and Heft, M. (1997) Evaluation of Bias and Logistics in a Survey of Adults at Increased Risk for Oral Health Decrements. Journal of Public Health Dentistry, 57, 48-58.https://doi.org/10.1111/j.1752-7325.1997.tb02472.x

Gilbert, G., Duncan, R. and Vogel, W. (1998) Determinants of Dental Care Use in Dentate Adults: Six-Monthly Use during a 24-Month Period in the Florida Dental Care Study. Social Science U Medicine, 47, 727-737. https://doi.org/10.1016/S0277-9536(98)00148-8