Reporting of Negative Randomized Trials in Three Major Medical Journals

doi:10.4236/ijcm.2013.42021

Paper Menu >>

Journal Menu >>

International Journal of Clinical Medicine, 2013, 4, 108-113

http://dx.doi.org/10.4236/ijcm.2013.42021 Published Online February 2013 (http://www.scirp.org/journal/ijcm)

Reporting of Negative Randomized Trials in Three Major

Medical Journals

Gabriel A. Brooks1, Daniel W. Bowles1,2, Daniel B. Jamieson1, Janice V. Huang1, Alexandra Smart1,

Kathleen A. Heist1, Traci E. Yamashita1, Allan V. Prochazka1,2, Ravi K. Gopal1,2

1Department of Medicine, University of Colorado School of Medicine, Aurora, USA; 2Denver VA Medical Center, Eastern Colorado

Healthcare System, Denver, USA.

Email: [email protected]

Received December 13th, 2012; revised January 15th, 2013; accepted January 22nd, 2013

ABSTRACT

Context: In recent years there has been increasing interest on publication bias and on initiatives to decrease bias, in-

cluding trial registration. Objective: To test whether there has been an increase in reports of randomized control trials

(RCT’s) with negative outcomes in major journals and to identify factors associated with these reports. Design: Retro-

spective review of reports of RCT’s published in the Journal of the American Medical Association, The Lancet and the

New England Journal of Medicine before (2002-’03, pre-registration era) and after (2007-’08, registration era) the in-

stitution of mandatory trial registration. Main Outcome Measure: The primary outcome was the proportion of RCT

reports with negative outcomes compared across the two eras. Secondary outcome includes other factors affecting pub-

lication. Results: We identified 917 reports of RCT’s published in the two study eras. No publications in the pre-regis-

tration era reported trial registration compared with 94.4% in the registration era (p < 0.001). There was a non-signifi-

cant increase in negative trials from the pre-registration to the registration era (29.1% vs. 34.1%, p = 0.10, OR 1.26,

95% CI 0.96 - 1.67). Study characteristics associated with negative outcomes include trials of drugs (OR 1.62, 95% CI

1.08 - 2.43), procedures or devices (OR 2.08, 95% CI 1.29 - 3.35), explicit identification of a single primary endpoint

(OR 1.70, 95% CI 1.40 - 2.47), and increasing sample size (OR 3.08, 95% CI 1.78 - 5.34). Non-inferiority study de-

sign was associated with a decreased likelihood of a negative outcome (OR 0.13, 95% CI 0.05 - 0.31). Conclusions:

The proportion of published RCT reports with negative outcomes in three major medical journals has not significantly

increased since the mandatory clinical trial registration policy. The observed prevalence of negative trials is associated

with increases in sample size and greater specificity in trial endpoints.

Keywords: Clinical Trial Registration; Publication Bias; Randomized Trials; Clinical Trials; Non-Inferiority Trials

1. Introduction

Health care professionals rely on up-to-date, compre-

hensive, and objective data to make clinical, research and

policy decisions. However, a growing body of data sug-

gests that the medical literature used to support these

decisions is subject to demonstrable publication bias,

with publication of study reports depending on character

of the results [1]. Trials with statistically significant out-

comes are nearly twice as likely to be published and are

published more quickly than so-called “negative trials”

[2,3]. Publication bias has been particularly evident

among studies of investigational drugs [4,5], where pro-

prietary interests may lead drug makers to withhold re-

sults of negative trials. These flaws in the medical litera-

ture skew systematic interpretation of clinical trial results,

misrepresent the experiences of patients who altruisti-

cally participate in these trials, and jeopardize the health

and safety of future patients.

In response to concerns about lack of transparency in

clinical trials, the United States Congress passed the

Food and Drug Administration Modernization Act in

1997 calling for the creation of a tool to permit public

access to information regarding ongoing clinical trials.

The National Institutes of Health (NIH) created the

clinicaltrials.gov registry in 2000 in response to this

mandate. Initially this registry was voluntary and limited

to trials involving medications targeted for “serious or

life threatening illnesses”. Even with this limited scope

there was poor compliance with registration of qualifying

studies.

In 2004 the International Committee of Medical Jour-

nal Editors (the ICMJE, made up of the editors of eleven

leading general medical journals) announced that effec-

tive September 2005, registration with clinicaltrials.gov

Reporting of Negative Randomized Trials in Three Major Medical Journals 109

or a similar registry would be required as a precondition

for publication [6]. Before the introduction of the ICM

JE’s policy there were fewer than 14,000 trials registered

at clinicaltrials.gov; within months of the policy’s intro-

duction the number of registered trials increased by 73%.

[7]. To date, over 136,000 trials from 182 countries are

registered at clinicaltrials.gov [8].

The establishment of clinicaltrials.gov by the NIH and

the ICMJE’s policy of mandatory clinical trial registra-

tion were both intended to increase the transparency of

clinical research. Based on anecdotal experience, we hy-

pothesized that the ICMJE’s mandatory clinical trial reg-

istration policy had resulted in a reduction in publication

bias by increasing the proportion of clinical trial reports

with negative outcomes published in major medical

journals. The primary aim of our study was to test whe-

ther there has been an increase in reports of randomized

control trials (RCT’s) with negative outcomes in major

journals and to identify factors associated with these re-

port.

2. Methods

2.1. Study Sample

Our study sample included all reports of RCT’s pub-

lished in the New England Journal of Medicine, The

Lancet and the Journal of the American Medical Asso-

ciation during 2002-’03 (preregistration era) and 2007-’08

(registration era). We selected these journals because

they represent the three general medicine journals with

the highest impact factors [9]. The two eras were cho-

sen to capture publication patterns before and after the

institution of mandatory clinical trial registration policies

(initiated in 2005). Two-year intervals were chosen to

represent each era based on a pilot sample of 50 articles

that suggested an 80% power to detect a 15% absolute

difference between eras in the proportion of published

trials with negative outcomes.

Reports of RCT’s were identified using the Cochrane

Central Register of Controlled Trials (Cochrane Register),

a publicly accessible database maintained by the Coch-

rane Group. We searched using the full journal name in

the source field of the Cochrane Register, limiting sear-

ches to the years of interest and excluding entries with

“comment” in the publication type field. When searching

for entries in the Lancet, we excluded entries with “on-

cology” or “neurology” in the source field to avoid que-

rying the database for RCT reports published in Lancet

Oncology and Lancet Neurology.

After our initial search, we excluded entries in the

Cochrane Register that were not reports of RCT’s (in-

cluding duplicate entries, entries for editorials and letters

to the editor and entries for trials other than RCT’s). We

defined an RCT as a prospective study randomizing hu-

man subjects into two or more groups for comparison of

a defined intervention. We excluded cost-effectiveness

trials because they have incremental outcomes that can-

not be categorized as positive or negative. We excluded

phase I and II trials because these studies frequently do

not have efficacy outcomes (phase I studies were initially

excluded from the ICMJE’s mandatory clinical trial reg-

istration policy).

2.2. Data Abstraction

For each included entry of an RCT report in the Coch-

rane Register we abstracted descriptive information and

assessed the statistical significance of the primary out-

come(s). One reviewer abstracted descriptive information

from the structured article abstracts within the Cochrane

tion on endpoints we accessed full reports at the on-line

sites of the respective journals. Descriptive information

abstracted included year and journal of publication, trial

registration number, intervention type (specified non-

exclusively as drug, procedure/device, surgery, behave-

ioral, diagnostic or treatment strategy), sample size,

number of study groups, use of placebo control, use of a

single primary endpoint and use of a non-inferiority

endpoint. We considered trials to be registered if any

registration number was reported in the abstract.

Two reviewers assessed each RCT report to categorize

the primary outcome(s) as positive or negative with re-

spect to statistical significance of the outcome. We de-

fined a statistically significant (positive) result as having

a p-value of less than 0.05 or reaching a pre-specified

non-inferiority parameter. Studies having more than one

primary outcome or without an explicitly identified pri-

mary outcome were categorized as positive if any main

outcome reported in the abstract was positive. In cases

where there was disagreement between the two reviewers

regarding the significance of a study outcome (positive

vs. negative), study interpretation was labeled as discor-

dant and a committee of three reviewers reached a con-

sensus determination (GB, AP and RG). This study was

deemed exempt from review by the Colorado Multiple

Institutional Review Board and the Department of Vet-

erans Affairs.

2.3. Statistical Analysis

We calculated descriptive statistics for study attributes

and compared these attributes across the two publication

eras using chi-squared or Wilcoxon rank-sum tests. At-

tributes with p ≤ 0.1 were included in a logistic regres-

sion model to identify study attributes that were associ-

ated with an increased likelihood of a negative outcome.

For model interpretability, sample size was categorized

as 200 - 900, 1000 - 4999, and ≥5000. A final regression

Reporting of Negative Randomized Trials in Three Major Medical Journals

110

model was created using a stepwise-selection regression

technique to identify attributes of RCT’s that had an in-

dependent effect on the likelihood of a reported negative

outcome. All analyses were performed using SAS Ver-

sion 9.2 (SAS Institute Inc., Cary, NC, USA).

3. Results

We identified 1091 entries in the Cochrane Central Reg-

ister published in the New England Journal of Medicine,

The Lancet and the Journal of the American Medical

Association in the pre-registration and registration eras.

Of the 1091 entries, 181 (16%) were excluded, leaving

917 reports of RCT’s for our analysis (468 and 449 re-

ports from the pre-registration and registration eras re-

spectively, see Figure 1).

Descriptive statistics for reports of RCT’s from each

era are illustrated in Table 1. We found a high level of

compliance with reporting of trial registration; 94.4% of

trials in the registration era reported registration com-

pared to no trials in the pre-registration era (p < 0.001).

There were more reports of drug trials in the registration

era compared to the pre-registration era (75.4% vs.

69.5%, p = 0.02), and there was a trend toward fewer

reports of trials with procedural or device therapies in the

registration era (13.4% vs. 17.7%, p = 0.07). Compared

to the pre-registration era, more RCT reports in the Reg-

istration era explicitly identified a primary endpoint

(56.0% vs. 69.5%, p ≤ 0.001) and more RCT’s had

non-inferiority primary endpoints (4.7% vs 9.1%, p =

0.01). Median sample sizes were larger for RCT’s re-

ported in the registration era (n = 750 vs. 385, p ≤

1091 entries identified in Cochrane Register

174 of 1091 entries excluded (16%)

- 17 duplicate entries

- 83 editorials/letters to the editor

- 58 studies other than RCT’s

- 13 RCT’s, phase I/II trials

- 3 RCT’s, economic endpoint

917 of 1091 RCT reports included (84%)

- 468 from pre-registration era 2002-’03 (51%)

- 166 from NEJM

- 142 from JAMA

-160 from Lancet

- 449 from registration era 2007-’08 (49%)

- 193 from NEJM

- 101 from JAMA

- 155 from Lanc et

Figure 1. Flow Diagram of RCT reports reviewed for inclu-

sion. Abbreviations: RCT, randomized controlled trial;

NEJM, New England Journal of Medicine; JAMA, Journal

of the American Medical Association.

0.001). With regard to the primary outcome, 29.1% of

RCT reports from the pre-registration era reported nega-

tive outcomes compared to 34.1% in the registration era.

This difference did not meet statistical significance (Ta-

ble 2, p = 0.10 for unadjusted analysis, OR 1.26%, 95%

CI 0.96 - 1.67). When analyzed by individual journal, a

statistically significant difference was noted only for the

New England Journal of Medicine, with a rate of nega-

tive publications of 24.1% in the pre-registration era and

35.2% in the registration era (p = 0.02).

While publication era did not predict a differential

likelihood of negative study outcomes, our logistic re-

gression model did identify a number of factors associ-

ated with an increased or decreased likelihood of nega-

tive outcomes (Table 3). The five factors associated with

an increased likelihood of a negative outcome were trials

of drug therapies, trials of procedure or device therapies,

explicit identification of a single primary endpoint, in-

creasing sample size and discordance (discordant inter-

pretation of the primary outcome by the two reviewers).

The only factor associated with a decreased likelihood of

Table 1. Characteristics of RCT reports.

Characteristic 2002-’03

% (n) 2007-’08

% (n) p-value

Registration reported 0 94.4% (424)<0.0001

Drug trial 69.5% (325) 75.4% (343)0.02

Procedure/device trial 17.7% (83) 13.4% (60)0.07

Surgery trial 4.3% (20) 2.7% (12)0.19

Behavioral trial 10.9% (51) 8.0% (36)0.14

Trial of diagnostic test 2.4% (11) 2.5% (11)0.92

Trial of treatment strategy 13.0% (61) 12.7% (57)0.88

Placebo controlled 40.8% (191) 42.3% (190)0.64

Non-inferiority design 4.7% (22) 9.1% (41)0.01

Single primary endpoint 56.0% (262) 69.5% (312)<0.0001

Discordant 10.3% (48) 6.5% (29)0.04

U.S. correspondence address42.3% (198) 39.0% (175)<0.0001

Median sample size 385 750

Table 2. Published RCT reports with negative outcomes, by

era.

Journal 2002-’03 % (n) 2007-’08 % (n) p-value

All journals29.1% (136) 34.1% (153) 0.10

NEJM 24.1% (40) 35.2% (68) 0.02

JAMA 38.7% (55) 43.6% (44) 0.45

Lancet 25.6% (41) 26.5% (41) 0.87

Reporting of Negative Randomized Trials in Three Major Medical Journals 111

Table 3. Logistic regression, modeling probability of a ne-

gative outcome (adjusted).

OR 95% CI p-value

Publication era

(relative to pre-registration era) 1.26 (0.93, 1.71) 0.14

Drug trial 1.62 (1.08, 2.43) 0.02

Procedure/device trial 2.08 (1.29, 3.35) 0.003

Non-inferiority trial .13 (0.05, 0.31) <0.0001

Single primary endpoint 1.79 (1.30, 2.47) 0.0004

Discordant 3.70 (2.13, 6.25) <0.0001

Sample size <0.0001

200 - 999 relative to <200 1.80 (1.18, 2.74)

1000 - 4999 relative to <200 2.54 (1.61, 4.00)

≥5000 relative to <200 3.08 (1.78, 5.34)

Journal <0.0001

Lancet relative to NEJM .82 (.58, 1.18)

JAMA relative to NEJM 1.87 (1.29, 2.71)

Abbreviations: OR, odds ratio; CI confidence interval; NEJM, New England

Journal of Medicine; JAMA, Journal of the American Medical Association

a negative outcome was a non-inferiority study design

(OR = 0.13, p < 0.001, 95% CI 0.05 - 0.31).

4. Discussion

We proposed that the anecdotal perception of an increase

in negative studies present in our academic journal club

is a real effect and that it would be associated with man-

datory clinical trial registration.

Our analysis did not find a significant difference in the

proportion of published trials with negative outcomes

between the two eras, even though there was a dramatic

increase in reporting of trial registration, suggesting that

trial registration may not have influenced the chances of

negative trials being published in major medical journals.

We also saw changes in the characteristics of reported

trials. Sample sizes of published studies increased, climb-

ing from a median of 385 to 750. The percentage of re-

ports explicitly identifying a primary outcome also in-

creased. Improvement in clinical trial reporting was re-

flected practically in our finding that it became easier for

abstracters to interpret trial outcomes, with discordant

interpretation of trial outcomes (positive vs. negative)

decreasing from the pre-registration era to the registra-

tion era. Though non-inferiority trials represented a small

minority of published trials, the percentage of trials using

a non-inferiority design nearly doubled across the two

study eras. Use of a non-inferiority design was the only

trial characteristic that was independently associated with

a decreased likelihood of a negative outcome. Character-

istics of trials that were associated with an increased

likelihood of a negative reported outcome were drug or

procedure interventions, larger sizes, and reporting of a

single primary endpoint.

To our knowledge this study is the first to examine the

effects of the ICMJE’s mandatory clinical trial registra-

tion policy on publication of negative studies. Previous

studies of publication bias have examined cohorts of tri-

als from within specific institutions or registries, and

have substantiated that negative trials are less likely to be

published [2,3]. We chose to examine the question of

publication bias at the level of the publishing journals,

since the ICMJE’s policy was enacted on this level and

this is the level of exposure for the journal reader. We

looked for evidence of changes in publication bias in the

general medicine journals with the highest impact factors

because research published in these journals is likely to

be influential and because all of these journals are mem-

bers of the ICMJE. In its 2004 policy statement, the

ICMJE cites two primary drivers of publication bias-

inconclusive or negative results that are perceived as

clinically uninformative and negative results that reflect

poorly on a proprietary intervention [6]. The ICMJE’s

policy highlights efforts from participating journals to

give appropriate weight to the importance of clinical trial

results, regardless of whether these results are positive or

negative. However, this policy will do little to affect

publication bias within high-profile journals that is

driven by clinically uninformative negative results; these

trials are likely to be published in journals with lower

impact factors. Our study was most likely to detect pub-

lication bias driven by negative results that reflected

poorly on proprietary interventions. Other recent studies

have demonstrated the connection between publication

bias and selective outcomes reporting, where primary

outcomes for trials of proprietary interventions appear to

have been changed post hoc but prior to publication.

[2,10,11]. Such pre-publication changes in reported out-

comes may partly account for our own negative findings.

We did observe two study characteristics that may

have driven this perceived increase in publication of

negative trials. Surprisingly, the larger the trial the more

likely it was to be negative.

This fact may represent studies in conditions that great

treatment improvements have already been achieved and

further therapies have marginal effect or therapies that

have pushed the envelope too far past benefit to detri-

ment. Also, the fact that more discordant studies were

noted to be negative may be related to selective reporting.

Negative trials may not have been as clear in their end-

points making study abstracters differ on their interpreta-

tion that the study was positive or negative. Only upon

reviewing the full article was one able to determine

Reporting of Negative Randomized Trials in Three Major Medical Journals

112

whether the study’s primary endpoint was truly negative.

Our study has a number of limitations. Our data are

based primarily on reviews of article abstracts, which

may not have contained the same information as full

publications. Our sample size was not adequate to detect

small differences between rates of publication of nega-

tive trials across the two study eras. Because we studied

RCT reports from two eras, it was not possible to control

fully for changes over time other than mandatory clinical

trial registration. The effect of mandatory clinical trial

registration on publication bias may be obscured by other

trends in the medical literature, such as selective out-

comes reporting or changes in the designs of randomized

controlled trials that favor positive outcomes. We note

that the editorial leadership at the three journals remained

constant over the two study eras. Also, because we stud-

ied publication bias in the three general medicine jour-

nals with the highest impact factors, our results do not

address changes in publication bias across the entire

medical literature or within medical journals of lower

impact factors.

Our study findings suggest that any effect of clinical

trial registration on publication bias within top-tier medi-

cine journals is likely to be small, if it exists. Other stud-

ies have shown substantial problems with selective out-

comes reporting even among registered trials. Do these

findings cast doubt on the utility of clinical trial registra-

tion as a tool to increase the transparency of clinical re-

search? It seems probable that standards of adequacy for

trial registration have not been sufficient [12,13], and

stricter enforcement of registration standards as well as

new laws requiring minimal outcomes reporting may

improve the utility of clinical trial registration to reduce

publication bias and selective outcomes reporting.

Our findings also suggest that studies with non-inferi-

ority designs are playing an increasingly important role

in the medical literature. Non-inferiority trials nearly

doubled as a proportion of all trials published in our

study sample, and these trials are much more likely to

have a positive outcome than trials with conventional

superiority designs. While the ascendance of non-inferi-

ority trials is a welcome trend in the arena of compara-

tive effectiveness research, this trend may be less benign

when applied in trials of proprietary therapies. In this

latter arena, a non-inferiority trial design may provide a

lower bar to evaluate the effectiveness of a therapy that is

seeking a wider market share [14].

Future studies examining the rates of published nega-

tive trials and the effect of trial registration on publica-

tion bias should look for this effect in a broader range of

journals, including subspecialty journals and smaller

general medicine journals. Because publication bias may

be driven by proprietary concerns, further inquiries

should shed more light on the interaction between trial

funding, publication bias and adoption of non-inferiority

study designs.

In conclusion, our results do not substantiate an asso-

ciation between clinical trial registration and a reduction

in publication bias in major medicine journals. Clinical

trial registration is meant as a means toward maximizing

the impact of clinical research activities and protecting

the contributions of altruistic patients. As we become

accustomed to the era of clinical trial registration (and

now, mandatory outcomes reporting) it is imperative to

learn what we can and cannot expect from this important

tool.

REFERENCES

[1] R. J. Simes, “Publication Bias: The Case for an Interna-

tional Registry of Clinical Trials,” Journal of Clinical

Oncology, Vol. 4, No. 10, 1986, pp. 1529-1541.

[2] K. Dwan., D. G. Altman, J. A. Arnaiz, et al., “Systematic

Review of the Empirical Evidence of Study Publication

Bias and Outcome Reporting Bias,” PLoS One, Vol. 3,

No. 8, 2008, e3081. doi:10.1371/journal.pone.0003081

[3] S. Hopewell, K. Loudon, M. J. Clarke, A. D. Oxman and

K. Dickersin, “Publication Bias in Clinical Trials Due to

Statistical Significance or Direction of Trial Results,”

Cochrane Database of Systematic Reviews, Vol. 1, 2009,

MR000006.

[4] K. Rising, P. Bachetti and L. Bero, “Reporting Bias in

Drug Trials Submitted to the Food and Drug Administra-

tion: Review of Publication and Presentation,” PLoS

Medicine, Vol. 5, No. 11, 2008, e217.

doi:10.1371/journal.pmed.0050217

[5] K. Lee, P. Bachetti and I. Sim, “Publication of Clinical

Trials Supporting Successful New Drug Applications: A

Literature Analysis,” PLoS Medicine, Vol. 5, No. 9, 2008,

e191. doi:10.1371/journal.pmed.0050191

[6] C. D. Deangelis, J. M. Drazen, F. A. Frizelle, C. Haug, J.

Hoey, R. Horton, S. Kotzin, C. Laine, A. Marusic, A. J.

Overbeke, T. V. Schroeder, H. C. Sox and M. B. Van Der

Weyden, “Clinical Trial Registration: A Statement from

the International Committee of Medical Journal Editors,”

Journal of the American Medical Association, Vol. 292,

No. 11, 2004, pp. 1363-1364.

doi:10.1001/jama.292.11.1363

[7] D. A. Zarin, T. Tse and N. C. Ide, “Trial Registration at

ClinicalTrials.gov between May and October 2005,” New

England Journal of Medicine, Vol. 353, No. 26, 2005, pp.

2779- 2787. doi:10.1056/NEJMsa053234

[8] Clinicaltrials.gov, “A Service of the US National Insti-

tutes of Health,” 2010. http://www.clinicaltrials.gov

[9] ISI Web of Knowledge, “Journal Citation Reports,” 2010.

http://0-admin-apps.isiknowledge.com.impulse.ucdenver.

edu/JCR

[10] S. S. Vedula, L. Bero, R. W. Scherer and K. Dickersin,

“Outcome Reporting in Industry-Sponsored Trials of Ga-

bapentin for Off-Label Use,” New England Journal of

Medicine, Vol. 361, No. 20, 2009, pp. 1963-1971.

Reporting of Negative Randomized Trials in Three Major Medical Journals

113

doi:10.1056/NEJMsa0906126

[11] S. Mathieu, I. Boutron, D. Moher, D. Altman and P.

Ravaud, “Comparison of Registered and Published Pri-

mary Outcomes in Randomized Controlled Trials,” Jour-

nal of the American Medical Association, Vol. 302, No. 9,

2009, pp. 977-984. doi:10.1001/jama.2009.1242

[12] C. D. Deangelis, J. M. Drazen, F. A. Frizelle, C. Haug, J.

Hoey, R. Horton, S. Kotzin, C. Laine, A. Marusic, A. J.

Overbeke, T. V. Schroeder, H. C. Sox and M. B. Van Der

Weyden, “Is This Clinical Trial Fully Registered? A

Statement from the International Committee of Medical

Journal Editors,” New England Journal of Medicine, Vol.

352, No. 23, 2005, pp. 2436-2438.

doi:10.1056/NEJMe058127

[13] J. S. Ross, G. K. Mulvey, E. M. Hines, S. E. Nissen and

H. M. Krumholz, “Trial Publication after Registration in

ClinicalTrials.gov: A Cross-Sectional Analysis,” PLoS

Medicine, Vol. 6, No. 9, 2009, e1000144.

doi:10.1371/journal.pmed.1000144

[14] S. Garatini and V. Bertele, “Non-Inferiority Trials Are

Unethical Because They Disregard Patients’ Interests,”

Lancet, Vol. 370, No. 9602, 2007, pp. 1875-1877.

doi:10.1016/S0140-6736(07)61604-3