As a way of handling missing data, multiple imputation (MI) has been steadily gaining popularity in the past several decades. When it comes to the question of how many imputations are sufficient in applying MI, different researchers have given different answers. Rubin suggested 2 to 5 [1] [2] . Schafer and Olsen suggested 3 to 5 [3] . Graham et al. suggested 20 or more [4] . Hershberger and Fisher suggested that several hundred imputations are often required [5] . Allison suggested that one may need more imputations than what were generally recommended in the literature [6] . The primary reason for the variation in recommended numbers of imputations is that different researchers may use different parameters or statistics and what is good for one parameter or statistic may not be good enough for a different parameter or statistic. For example, Rubin used the “relative efficiency” as defined by him to make the decision [1] , whereas Graham et al. used the testing power to make the decision [4] . When a data user decides to adopt MI, he or she does not have standard guidelines on how many imputations he or she should use for his or her particular data situation.

Variables that need imputation can be different, and the imputation models and/or methods can vary. The resources that can be allocated to MI application can also vary from survey to survey. It may not be practical to recommend a specific number of imputations that can universally fit all imputation models or data and management situations. Instead of attempting to prove or disprove recommendations found in the published literature, the current research examines an empirically-based method to determine the minimally sufficient number of imputations needed.

Let m be the number of imputations and η be the least number of imputations that is sufficient to meet a mini- mum requirement chosen by a data analyst. Among the various published answers to the “how-many-are-suffi- cient” question, the one given by Rubin (1987) [1] appears to be by far the most influential. In Chapter 4 of his classic book on MI, Rubin established that the relationship between, the large sample variance of a point estimator, Q, from a finite m, and, the variance from an infinite m, is

where is the population fraction of missing information. From this relationship, the relative efficiency (RE) measured in the units of standard errors from using MI is

Based on this RE, Rubin stated: “If, even two repeated imputations appear to result in accurate levels, and three repeated imputations result in accurate levels even when”.

The question is: Is the RE, as defined by Equation (2), the best and only parameter that can be used as the criterion to determine η? Different η recommendations from different researchers (e.g. [3] -[5] ) suggest this RE may not be the best and only parameter to determine η. None of these researchers including Rubin have explicitly claimed that his or her method is the only correct method to determine η.

The number of imputations affects the MI results in multiple ways as indicated by the following equations for MI data analyses [1] . The mean of the point estimator Q is

The average variance estimate of Q over m complete datasets from MI is

where is the variance estimate of the ith imputation. The estimated imputation variance is

The total variance estimate is

Equations (3) to (6) all contain m as a factor, indicating that m can affect the results of MI data analyses in multiple ways. The minimally sufficient number of imputations could be defined as the smallest m that would produce a sufficiently accurate, , B, or T as judged by the data user.

This paper presents a methodology to determine η by focusing on the effects of m on the accuracy of B as defined by Equation (5). The focus is on B because the primary advantage of MI over single imputation (SI) is that MI makes it possible to estimate B while SI cannot [1] [7] . For an m to become the η, this m should first be capable of allowing us to obtain a sufficiently accurate B. The accuracy of B can be measured by ω, the estimated standard error of B. For a particular variable, ω would be smaller as m increases. If ω becomes sufficiently small, then one may conclude that B is sufficiently accurate and the m corresponding to that ω value would represent η. Using the data from the 2012 wave of the National Ambulatory Medical Care Survey (NAMCS) Physician Workflow study (PWS), we were able to demonstrate that ω can be used to determine η. This method can be used by anyone to determine η for his or her own data.

Shifting from SI to MI allows an estimate of B, the imputation variance, and a more accurate estimate of the total variance, T, as defined by Equation (6) [1] [7] . If an MI protocol cannot give a reliable estimate of B, then the

major benefit of MI would be lost. Therefore, it makes sense to use an m-dependent measure of reliability for the estimate of B as a criterion in determining η, the minimally sufficient number of imputations. The results of this research indicate that B values obtained for m < 20 may not be reliable at least for these five variables tested using the imputation method described (Figure 2 and Figure 3). If one uses Rubin’s recommendation of m = 2 or 3, the B value obtained may be many times bigger or smaller than the true B value. Using the criterion of whether a reliable B can be obtained using the protocol described in this paper, the recommended m = 2 to 5 ([1] [2] ) are not sufficient for the five variables tested.

The variance of imputation variances is a good determiner of η for three reasons. First, it is critical for any MI procedure to produce a reliable estimate of B. Second, the effect of m on ω can be big and can be easily visualized, as shown in Figure 3. Furthermore, the effect of m on ω decreases with increased m, allowing the data user to set a cutoff point for an m to be recognized as the η. Third, the method of calculating ω is relatively simple and practical.

To determine the minimally sufficient number of imputations, one has to first define the criteria for being recognized as “sufficient”. Different data users may have different criteria. As a result, different η values may be obtained from the same m-ω relationship data, as exemplified by the results show in Table 5 for the two different η determination methods discussed above. This is another reason why a universal recommendation of η is not a good idea and may even be impossible.

The highest m value tested in this study was 100. There was an indication that even m = 100 still had some effect in reducing ω (Figure 3(b), Figure 3(c) and Figure 3(d)). Ten samples were pulled for each m. If one pulled 100 samples for each m and tested m values of 200, 300, or even greater, it may be possible to show that an increase in m would still be statistically effective in reducing the variance of B. Theoretically the effects of m in reducing ω would approach zero but may never become zero. Therefore, the argument by some researchers ([5] ) that η should be in hundreds is not totally unreasonable and has some support from this research. But because an increased m means a higher cost, it may not be justified to do many more imputations just for a tiny bit of gain. Balance of the gain against the cost based on one’s own situation plays an important role in η determination.

Based on discussions above, η can be defined as the value of m beyond which the cost of further increase in m would outweigh the benefit of reducing ω. Both the moving regression method and the confidence interval method were used to determine η based on the m-ω relationships obtained for five survey variables. The two methods are meant to be examples to illustrate that different η values may be obtained depending on the chosen method and the cutoff point. There may be other methods to determine η from a m-ω curve that fit one’s particular situation better. With the ever-improving information technology, the cost for increased number of imputations may not be a major source of cost in determining η. But there is always a need to consider one’s particular situation such as one’s tolerance level for the probability in determining η.

The five variables are different in their value ranges, variances and missing data percentages. Nevertheless the m-ω relationship curve can be used to determine η for all of the five variables tested. No obvious relationship can be established between the variable characteristics such as the variances and the η values obtained.

The NAMCS Physician Workflow mail survey is sponsored by the Office of the National Coordinator for Health Information Technology.