Problems in estimating “high” burnout prevalence in organizations arise because of a fundamental disparity between how burnout is conceptualized and how it is measured. Burnout is considered a univariate continuous phenomenon of “experienced feeling”, but a person’s level of “experienced feeling” is indicated in a non-additive non-linear way by a person’s joint levels on three aspects of the phenomena. These aspects, or what are probably better understood as symptoms, are one’s joint levels of emotional exhaustion, depersonalization, and sense of personal accomplishment (Maslach et al., 1996; Maslach, Schaufeli, & Leiter, 2001) . The highest state of “experienced feelings of burnout” is known as “Burnout Syndrome”, identifiable by the trivariate joint occurrence of “high” frequency of emotional exhaustion, “high” frequency of depersonalization, and “infrequent” sense of personal accomplishment in relation to one’s work (Maslach et al., 1996) . One can also identify low “experienced feeling of burnout” at work-when a person jointly experiences “infrequent” emotional exhaustion, “infrequent” depersonalization, and a “high” frequency of a sense of personal accomplishment. The measurement problem arises in deciding how to gauge the level of “experienced feeling of burnout” in people who report one of the other six possible trivariate configurations of “high” and “low” frequencies on the three “aspect” variables; as for example when one has both a frequent sense of personal accomplishment and infrequent emotional exhaustion, but also has a “high” frequency of depersonalization. What these signify about a person’s position on the univariate “experienced feelings of burnout” continuum remains uncertain (Boersma & Lindblom, 2009; Demerouti, Verbeke, & Bakker, 2005; Maslach et al., 1996) .

These uncertainties have led investigators to consider five different estimators of “high” burnout. Each is distinguished by which of the six “uncertain” joint configurations they classify as indicating “high” levels of burnout. In essence each “high” burnout estimator embodies a different hypothesis about what the six “uncertain” trivariate joint configurations imply about the person’s level of “experienced feelings of burnout”. Figure 1 shows the five “high” burnout estimators and how each maps (collapses) the trivariate configurations of “high” and “low” emotional exhaustion, depersonalization and sense of accomplishment into a dichotomous “univariate experience of burnout” scale. The joint configurations indicating “high” burnout in the respective five estimators are: 1) any configurations with “high” frequency of emotional exhaustion, 2) any with either “high” frequency of emotional exhaustion or “high” frequency of depersonalization, 3) any with “high” frequency of emotional exhaustion plus either “high” frequency of depersonalization or “low” frequency of personal accomplishment (a.k.a. emotional exhaustion plus one), 4) any with both “high” frequency of emotional exhaustion and depersonalization, and 5) only those with jointly “high” frequency of emotional exhaustion and depersonalization and “low” frequency of sense of personal accomplishment.

There is evidence within the burnout measurement literature to support the use of each of the estimators. This evidence includes: the intercorrelations among the three Maslach Burnout Inventory (MBI, Maslach et al., 1996 ) subscale scores, the MBI subscales’ scores’ correlations with job demands or resources (Lee & Ashforth, 1996; Schaufeli & Salanova, 2007; West et al., 2006) , and the MBI factor and bifactor loadings (Gil-Monte, 2005; Loera, Converso, & Viotti, 2014; Meszaros, Adam, Szabo, Szigeti, & Urban, 2014; Schaufeli, Bakker, Hoogduin, Schaap, & Kladler, 2001) . A few studies have examined some estimators’ discriminant validity or their utility in predictive models (Brenninkmeijer & Van Yperen, 2003; Schaufeli et al., 2001; West et al., 2012) . The use of the two most popular estimators, “‘high’ emotional exhaustion alone” or “either ‘high’ emotional exhaustion or depersonalization”, is supported by their associations with undesirable sequelae of “high” burnout, such as perceived medical errors, suicidality and alcoholism (Pedersen, Sorensen, Bruun, Christensen, & Vedsted, 2016; Oreskovich et al., 2012; Shanafelt et al., 2010; West et al., 2012) .

There are also weaknesses in the evidence and in its use. For example, the sequelae studies demonstrate only that some of the aspects of “Burnout Syndrome” are, individually, associated with potentially serious consequences (Ores- kovich et al., 2012; Shanafelt et al., 2010) . It is also unclear in many of the factor and bifactor studies that the authors have adequately considered the potential for spurious correlations due to skewedness and to extreme points and outliers in the tridimensional data. In addition, many studies do not adequately consider the potential effects of the non-additive nonlinear structure within the tridimensional configuration of the three variables with respect to burnout experience, as has been pointed out by Demerouti et al. (2005) . Also few studies have treated the sets of joint aspects that are queried by the five estimators as symptoms and examined their associations as markers of employee- perceived problems in job demands and resources in human-systems functioning.

A final problem is that evidence from some older studies, like those of Rafferty et al.’s (1986) and Schaufeli et al.’s (2001) , continue to be frequently cited as strong support for the use of some popular estimators even though the strength of the evidence in the studies is weak (Bianchi, Schonfeld, & Laurent, 2014, 2015a, 2015b; Kleijweg, Verbraak, & Van Dijk, 2013; Rafferty, Lemkau, Purdy, & Rudisill, 1986; Thomas, 2004; Van Dam, Keijsers, Verbraak, Eling, & Becker, 2015) . For example, Rafferty et al.’s (1986) used an older version of the MBI and found 67 residents’ emotional exhaustion frequency scores and intensity scores and their depersonalization intensity scores were statistically significantly correlated with their responses to a single burnout item (scaled as 1 (not at all burned out) to 9 (very burned out)). In spite of the small sample size, this result is often used to justify the claim that the sense of personal accomplishment aspect of burnout is less important than the other two aspects. The study does not justify the conclusion because the sense of personal accomplishment aspect of burnout was inadequately described in their single burnout item’s stem, thus biasing the results. In addition, the study also found that the residency-directors’ assessments of the residents’ burnout, were associated with the residents’ MBI sense of accomplishment frequencies but not with their depersonalization scores. Schaufeli et al.’s (2001) finding that the MBI emotional exhaustion and depersonalization subscales had discriminant validity for work-related neurasthenia is also often cited as strong support for the sole use of these two aspects as burnout estimators, even though both the neurasthenia disorder and the measurement of it used in the study have since lost their scientific support.

The concern over which of the five estimators to use is important because the prevalence of “high” burnout depends on which estimator is used. Some investigators argue strongly that some of the five estimators overestimate the prevalence of “high” burnout (Blanchard, Rodrigues, & Colombat, 2012; Kleijweg et al., 2013) . Others argue that restricting the estimator of “high” burnout to the occurrence of “Burnout Syndrome” alone (the joint prevalence of “high” emotional exhaustion and depersonalization and “low” sense of personal accomplishment) underestimates it (Campbell et al., 2011; Dyrbye & Shanafelt, 2015; Shanafelt, Dyrbye, & West, 2013) . Figure 1 shows mathematically that four of the five estimators will necessarily yield prevalences of “high” burnout that are greater than the prevalence of “Burnout Syndrome” alone. This is due to the fact that the occurrence of “Burnout Syndrome” can be no more prevalent than the least prevalent of any one of its aspects alone or in combinations. In addition, large differences in the “high” burnout prevalence will be observed when assessed in a single population at the same time using the five different estimators. This problem is easy to see in the literature, which we next describe.

In the studies we examined where the joint occurrence of all three aspects of Burnout Syndrome was used as the estimator of “high” burnout, prevalence ranged only from 2% to 13% (Al-Dubai & Rampal, 2010; Contag et al., 2010; Gabbe et al., 2008; Golub, Johns, Weiss, Ramesh, & Ossoff, 2008; Golub, Weiss, Ramesh, Ossoff, & Johns, 2007; Gorter, Albrecht, Hoogstraten, & Eijkman, 1999) . When data on this estimator were not directly available, we used the least prevalent aspect reported in the study as the upper bound on the possible joint prevalence of all three aspects and found the upper bounds ranged from 12.4% to 28% (Campbell, Prochazka, Yamashita, & Gopal, 2010; Dyrbye et al., 2008; Grassi & Magnani, 2000; Klimo Jr. et al., 2013; Shanafelt et al., 2012; Shanafelt et al., 2015) . The “high” burnout prevalences based on the bivariate joint occurrence of only two aspects―“both ‘high’ emotional exhaustion and depersonalization”- were similarly relatively low (for example, 17% in Gabbe et al., 2008 ), as were “high” burnout prevalences based on “‘high’ emotional exhaustion plus one other aspect” (for example, 18% to 29.9% in (Brenninkmeijer & Van Yperen, 2003; Dyrbye & Shanafelt, 2015; Schaufeli & Van Dierendonck, 2000; van der Wal, Bucx, Hendriks, Scheffer, & Prins, 2016) . In contrast, using the “‘high’ emotional exhaustion alone” estimator yielded alarmingly high prevalences of “high” burnout; from 27.5% to 63.2% (Al-Dubai & Rampal, 2010; Bressi et al., 2009; Campbell et al., 2010; Dyrbye et al., 2008; Grassi & Magnani, 2000; Grunfeld et al., 2000; Klersy et al., 2007; Shanafelt et al., 2012; Shanafelt et al., 2015) . This range was similar to the “18% to 76%” range for “high” burnout often cited in the organizational literature (Campbell et al., 2010; Prins et al., 2007) . The prevalence using the “either ‘high’ emotional exhaustion or ‘high’ depersonalization” estimator was similarly high (45.4% to 67%) (Campbell et al., 2011; Campbell et al., 2010; Dyrbye et al., 2015; Dyrbye & Shanafelt, 2015; Dyrbye et al., 2008; Shanafelt et al., 2012; Shanafelt et al., 2015) . Also, in studies where cost-efficient single-item “indicators” or “markers” (Indrayan, 2013) of the MBI subscale scores were substituted for the full MBI subscales, the relative magnitudes of the prevalences obtained from the five estimators were consistent with those obtained using the full MBI scales (Dolan et al., 2015; West et al., 2012; West, Dyrbye, Sloan, & Shanafelt, 2009) .

The differences in the prevalences of “high” burnout obtained from the five estimators poses a serious practical problem (Dyrbye, West, & Shanafelt, 2009) in health care organizations. One consequence is comparisons of the prevalences of “high” burnout across some institutions may be spurious. Another is that institutions are vulnerable to 1) spurious data on the prevalence of “high” burnout and burnout vulnerability in their employees, 2) spurious inferences about the effects of employee-reported human-systems problems on “high” burnout prevalence, and 3) spurious assessments of the need for and effects and effectiveness of attempted interventions.

Currently, managers have no research-based information on the association of the aspects of “high” burnout used in the estimators and the specific problems in job demands and resources in human-systems functioning their employees report in annual organizational health surveys. Knowledge of these associations could help to gauge the usefulness of the different “high” burnout estimators in practice and help managers to identify problems that may be affecting burnout. In the current study, we investigated the association (measured as the positive likelihood ratio (LR+)) between each of the subsets of aspects of “Burnout Syndrome” measured in the five burnout estimators and employee-perceptions of problems in 31 types of job demands and resources in human-systems functioning, to determine whether any of the subsets of aspects measured in the specific “high” burnout estimators might act as potential symptoms (or markers) of the employee-perceived problems. The specific research question was: How much more often were the respective three aspects of “Burnout Syndrome” (and their subsets used as estimators of “high” burnout) self-reported by persons who perceived and reported a specific problem in their workplace than in persons who did not (the positive likelihood ratio (LR+)) (McGee, 2002; Scott, Greenberg, & Poole, 2008) .

We used data from a cross-sectional organizational health census-survey conducted annually in the Veterans Health Administration (VHA) (N_{(all employees)} = 284,634). The estimated overall response rate was 56.3% and complete data were given by 97% of respondents. The survey was voluntary; advertised daily; and available in electronic, paper or telephone format during the last 3 weeks in September 2013. Both participation-confidentiality and response-anonymity were ensured by the data collection, storage and reporting procedures (Osatuke et al., 2012) . Use of the aggregated survey data for research and publication was approved by the Cincinnati (Ohio) Veterans Health Administration Facility IRB (signed documentation available on request).

Here we analyze the data from the 9543 respondents in seven health care provider groups (primary care physicians, surgeons, psychiatrists, anesthesiologists, medicine specialists, nurse practitioners, and physicians’ assistants) (N = 22,721) from among the 87,358 direct-care providers in 16 occupational groups (N_{(direct care providers)} = 160,124) who responded to the survey. Their estimated overall response rate in the direct care providers was 53%, and the complete data rate was 97%. Table 1 shows the specific sample sizes, estimated response rates, and demographic characteristics in each of the seven groups in the current study (n_min = 340 anesthesiologists, n_max = 2165 physician in all medical specialties, n_Mdn = 1265 psychiatrists; response rate_min = 34% in primary care physicians, response rate_max = 56% in psychiatrists, and response rate_Mdn = 41.5%).

All items used in this study were administered on the same survey as a standard

Note. The groups in this table are respectively: Primary Care Physicians, Surgeons, Psychiatrists, All Medicine-Specialists, Anesthesiologists, Nurse Practitioners (NP), Physicians Assistants (PA), and All Health Care Providers. ^aBlack or African American. ^bAmerican Indian or Alaskan Native. ^cNative Hawaiian or other Pacific Islander.

part of the VHA annual survey.

To measure associations, in each health care provider group, we calculated how much more the “high” burnout estimator’s aspects occurred among persons who perceived a workplace problem than in persons who did not (the positive likelihood ratio (LR+)). This statistic is frequently used in clinical medicine and epidemiology. We used the epiR program (Stevenson, 2015) in R (R Development Core Team, 2014) . To simplify interpretation, we translated the LR+ statistics into “how much a finding of the presence of the burnout estimator’s aspects increased the prevalence of the perception of the problem beyond the baseline prevalence in the group’ (the “added prevalence”). We used McGee’s (2002) formula: “added prevalence” = [100 × (0.19 × ln(LR+)). Like LR+, the “added prevalence” captures both linear and nonlinear associations in ordered categorical data, and is independent of the baseline prevalence of the problem in the population (Indrayan, 2013; Scott et al., 2008) . Our large sample sizes ensured statistical significance, so we judged “added prevalence” to be clinically significant if the lower bound on its 99% confidence interval was greater than 20% (McGee, 2002) .

We found that, within each group, the five “high” burnout estimators produced large differences in the estimated prevalence of “high” burnout, as shown in the

Note. For both Emotional Exhaustion (E) and Depersonalization (D), we defined frequencies as “High” (H) if the occurrence was “once a week” or more (4 or higher). For Sense of Personal Accomplishment (PA), we defined frequencies as “Low” (L) if the occurrence was “a few times a month” or less (3 or less). Groups in this table are respectively: Primary Care Physicians, Surgeons, Psychiatrists, All Medicine-Specialists, Anesthesiologists, Physicians Assistants, Nurse Practitioners, and All Health Care Providers. ^aThe 99% Confidence Intervals on the percent prevalence estimates were each smaller than ±0.02%. ^b“Burnout Syndrome”. ^cNo Aspects of “Burnout Syndrome”.

lower part of Table 3. For psychiatrists the estimated prevalence of “high” burnout using “‘high’ emotional exhaustion or depersonalization” (36.8%) was six times higher than the prevalence estimated using the trivariate joint occurrence of all three aspects of “Burnout Syndrome” (6.0%). The prevalence of “high” burnout was lowest in all groups when estimated as “Burnout Syndrome”. It ranged from 5.6% in all medical specialties and anesthesiologists to 13.4% in primary care physicians. The estimated prevalence of “high” burnout was highest in all groups when estimated by the combined occurrence of “either ‘high’

Note. The groups in this table are: Primary Care Physicians, Surgeons, Psychiatrists, All Medicine-Specialists, Anesthesiologists, Nurse Practitioners and Physicians Assistants. EE = “High” Emotional Exhaustion: Emotional Exhaustion frequencies of “once a week” or more (4 or higher). DP = “High” Depersonalization: Depersonalization frequencies of “once a week” or more (4 or higher). PA =”Low” Sense of Personal Accomplishment: Sense of Personal Accomplishment frequencies of “a few times a month” or less (3 or less). The five “high” burnout estimators and the single aspects of Burnout Syndrome for which clinically significant results were obtained are indicated as follows: EE ≡ “High” Emotional Exhaustion alone; DP ≡ “High” Depersonalization alone; EE OR DP ≡ High” Emotional Exhaustion or “High” Depersonalization; EE & (DP OR PA) ≡ High” Emotional Exhaustion and either (“High” Depersonalization or “Low” Sense of Personal Accomplishment); EE & DP ≡ High” Emotional Exhaustion and “High” Depersonalization; and EE & DP & PA ≡ the trivariate joint occurrence of High” Emotional Exhaustion and “High” Depersonalization and “Low” Sense of Personal Accomplishment. The “problem- base” rows provide the percent prevalence of the problem in the group. “Add, If” indicates that rows following it give the “Added Percent Prevalence” if the estimator occurs.. To calculate the “the percent prevalence of the human- systems problem, given the presence of the aspects of “Burnout Syndrome” queried by the estimator”, sum: the Base % Prevalence + Added % Prevalence. For example, Among Medicine-Specialists who self-report EE & DP & PA, the risk of perceived problems in “safety in disagreeing” is: 11% (base prevalence in our institution) + 34% = 45%. To calculate the positive likelihood ratio from the data use:. ^aThe full item queries related to the abbreviated problem description are listed in Table 1. ^bIf the base prevalence < 10%, the “Added Percent Prevalence” formula we used underestimates the change in probability. The exact value can be calculated from the information in the table by first calculating the posterior odds as. Then calculating the “exact value for added prevalence” = (posteriorodds/(1+ posteriorodds)) − base. ^cThe “Added Percent Prevalence” given in this section is for only for EE&DP&PA. The values for EE&DP are lower.

emotional exhaustion or ‘high’ depersonalization”. It ranged from 24.2% in anesthesiologists to 50.2% in primary care physicians.

We noted that reporting the prevalence of all eight trivariate configurations in a group, like those shown in the first part of Table 3, may reveal vulnerabilities within the group to large increases in burnout prevalence. For example, primary care physicians appeared vulnerable to an increase in the prevalence of “Burnout Syndrome” from 13.4% to as high as 33% if the 19.6% of them who were jointly “high” both on emotional exhaustion and depersonalization, but also currently “high” on sense of personal accomplishment (H, H, H) were to lose the factors in their work environment sustaining their “high” sense of personal accomplishment.

As Table 4 shows, when “high” burnout was estimated as the trivariate joint occurrence of “‘high’ emotional exhaustion, ‘high’ depersonalization and ‘low’ sense of personal accomplishment”, it was clinically-significantly associated with 97% (30) of the 31 problems studied in at least one of the health care provider groups.

In contrast, we found the two most commonly used “high” burnout estimators (“‘high’ emotional exhaustion alone”, and “either ‘high’ emotional exhaustion or ‘high’ depersonalization”) were associated only with six of the human- systems problems studied, but only within four provider groups. Table 4 shows both of the most commonly used estimators were associated with workload within surgeons and medicine specialists. “High” emotional exhaustion alone was associated with workload in psychiatrists and nurse practitioners. Within anesthesiologists, 1) both estimators were associated with cooperation among workgroups; 2) “‘high’ emotional exhaustion alone’ was associated with skills development and supervisor relationship; and 3) “either ‘high’ emotional exhaustion or ‘high’ depersonalization was associated with problems with respect at work. Turnover intentions were also associated with both commonly used estimators in several groups.

Although not shown in Table 4, “high” depersonalization alone was associated with more perceived human-systems problems (including job control, safety in problem solving, and work/family balance) than (a) the “‘high’ emotional exhaustion alone” estimator and (b) the “either ‘high’ emotional exhaustion or ‘high’ depersonalization” estimator.

Table 4 also shows the three more rarely used estimators were associated with many more of the human-systems problems we studied.