Publication bias

From Wikipedia, the free encyclopedia
Jump to: navigation, search
Not to be confused with Reporting bias or Media bias.

Publication bias is a bias with regard to what is likely to be published, among what is available to be published. Not all bias is inherently problematic – for instance, a bias against publishing lies is often a desirable bias – but one problematic and much-discussed bias is the tendency of researchers, editors, and pharmaceutical companies to handle the reporting of experimental results that are positive (i.e. showing a significant finding) differently from results that are negative (i.e. supporting the null hypothesis) or inconclusive, leading to a misleading bias in the overall published literature.[1]

This is mostly a bias towards reporting significant results, despite the fact that studies with significant results do not appear to be superior to studies with a null result with respect to quality of design.[2] It has been found that statistically significant results are three times more likely to be published than papers affirming a null result.[3] However, when a positive result is well established, it may become newsworthy to publish papers affirming the null result.[4] It has been found that the most common reason for non-publication is an investigator's declining to submit results for publication (because of the investigator's loss of interest in the topic, the investigator's anticipation that others will not be interested in null results, etc.), a finding that underlines researchers’ role in publication bias phenomena.[2]

In an effort to decrease this problem, some prominent medical journals require registration of a trial before it commences so that unfavorable results are not withheld from publication. Several such registries exist, but researchers are often unaware of them. In addition, attempts to identify unpublished studies have proved very difficult and often unsatisfactory. Another strategy suggested by a meta-analysis is caution in the use of small and non-randomised clinical trials because of their demonstrated high susceptibility to error and bias.[2]


Publication bias occurs when the publication of research results depends not just on the quality of the research but on its nature and direction.[5]

Within psychology and related disciplines, publication bias is often called the file drawer effect, or file drawer problem. The origin of this term is that results not supporting the hypotheses of researchers are often lost in the researchers' file drawers, leading to a bias.[6] The term "file drawer problem" was coined by the psychologist Robert Rosenthal in 1979.[7]

Positive results bias, a type of publication bias, occurs when authors are more likely to submit, or editors accept, positive than negative or inconclusive results.[8]

Outcome reporting bias occurs when several outcomes within a study are measured but are reported selectively depending on the strength and direction of the results. A related term that has been coined is HARKing (Hypothesizing After the Results are Known).[9]

Evidence of publication bias[edit]

The presence of publication bias in the literature has been most extensively studied in biomedical research. Investigators following clinical trials from the submission of their protocols to ethics committees or regulatory authorities until the publication of their results observed that those with positive results are more likely to be published.[10][11][12] In addition, studies often fail to report negative results when published, as demonstrated by research comparing study protocols with published articles.[13][14]

The presence of publication bias has also been investigated in meta-analyses. The largest study on publication bias in meta-analyses to date investigated the presence of publication bias in systematic reviews of medical treatments from the Cochrane Library.[15] The study showed that positive statistically significant findings are more likely to be included in meta-analyses of efficacy than other findings and that results showing no evidence of adverse effects have a greater probability to enter meta-analyses of safety than statistically significant results showing that adverse effects exist. Evidence of publication bias has also been found in meta-analyses published in prominent medical journals.[16]

Effect on meta-analysis[edit]

As result of publication bias, published studies may not be truly representative of all valid studies undertaken, and this bias may distort meta-analyses and systematic reviews of large numbers of studies—on which evidence-based medicine, for example, increasingly relies. The problem may be particularly significant when the research is sponsored by entities that may have a financial or ideological interest in achieving favorable results.

Those undertaking meta-analyses and systematic reviews need to take account of publication bias by performing a thorough search for unpublished studies. Additionally, a number of publication bias methods have been developed, including selection models [15][17][18] and methods based on the funnel plot, such the Begg's test,[19] the Egger's test,[20] and the trim and fill method [21] However, since all publication bias methods are characterized by a relatively low power and are based on strong and unverifiable assumptions, their use does not guarantee the validity of conclusions from a meta-analysis.[22][23]


The antidepressant Reboxetine provides an example of experimental bias in clinical trials. It was originally passed as effective for treatment of depression in many countries in Europe in the UK in 2001 (though in practice it is rarely used). In 2010) a meta analysis concluded it is ineffective due to publication bias in the original trials published by the drug manufacturer Pfizer. A later (2011) meta analysis of the original data found flaws in the 2010 meta analysis and suggests that it can be effective after all, in severe cases of depression. See Reboxetine - Efficacy. Whatever the final outcome for Reboxetine, the original trials show a clear case of publication bias. More examples of publication bias are given by Ben Goldacre[24] and Peter Wilmhurst.[25]

In the social sciences, a study looks at published papers on the relationship between Corporate Social and Financial Performance, and found that "In economics, finance, and accounting journals, the average correlations were only about half the magnitude of the findings published in Social Issues Management, Business Ethics, or Business and Society journals".[26]

Publication bias is often cited in investigations of papers on the Paranormal. A recent example is a paper by Daryl Bem, which showed evidence of short term pre-cognition. Negative results by other researchers that attempted to duplicate his work were not published in the journals that published the original results.[27]

One study[28] compared Chinese and non-Chinese studies of gene-disease associations and found that "Chinese studies in general reported a stronger gene-disease association and more frequently a statistically significant result".[29] One possible interpretation of this result is selective publication (publication bias).


According to John Ioannidis, negative papers are most likely to be suppressed when:[30]

  1. the studies conducted in a field are smaller;
  2. effect sizes are smaller;
  3. there is a greater number and lesser preselection of tested relationships;
  4. there is greater flexibility in designs, definitions, outcomes, and analytical modes;
  5. there is greater financial and other interest and prejudice;
  6. more teams are involved in a scientific field in chase of statistical significance.

Ioannidis further asserts that "claimed research findings may often be simply accurate measures of the prevailing bias".


Ioannidis' remedies include:

  1. Better powered studies
    • Low-bias meta-analysis
    • Large studies where they can be expected to give very definitive results or test major, general concepts
  2. Enhanced research standards including
    • Pre-registration of protocols (as for randomized trials)
    • Registration or networking of data collections within fields (as in fields where researchers are expected to generate hypotheses after collecting data)
    • Adopting from randomized controlled trials the principles of developing and adhering to a protocol.
  3. Considering, before running an experiment, what they believe the chances are that they are testing a true or non-true relationship.
    • Properly assessing the false positive report probability based on the statistical power of the test[31]
    • Reconfirming (whenever ethically acceptable) established findings of "classic" studies, using large studies designed with minimal bias

Study registration[edit]

In September 2004, editors of several prominent medical journals (including the New England Journal of Medicine, The Lancet, Annals of Internal Medicine, and JAMA) announced that they would no longer publish results of drug research sponsored by pharmaceutical companies unless that research was registered in a public database from the start.[32] Furthermore, some journals, e.g. Trials, encourage publication of study protocols in their journals.[33] The World Health Organization agreed that basic information about all clinical trials should be registered, at inception, and that this information should be publicly accessible through the WHO International Clinical Trials Registry Platform. Additionally, public availability of full study protocols, alongside reports of trials is becoming more common for studies.[34] A recent study showed that publication bias is smaller in meta-analyses of more recent studies, supporting the effectiveness of the measures used to reduce publication bias in clinical trials.[15]

See also[edit]


  1. ^ Song, F.; Parekh, S.; Hooper, L.; Loke, Y. K.; Ryder, J.; Sutton, A. J.; Hing, C.; Kwok, C. S.; Pang, C.; Harvey, I. (2010). "Dissemination and publication of research findings: An updated review of related biases". Health technology assessment (Winchester, England) 14 (8): iii, iix–xi, iix–193. doi:10.3310/hta14080. PMID 20181324.  edit
  2. ^ a b c Easterbrook, P. J.; Berlin, J. A.; Gopalan, R.; Matthews, D. R. (1991). "Publication bias in clinical research". Lancet 337 (8746): 867–872. doi:10.1016/0140-6736(91)90201-Y. PMID 1672966. 
  3. ^ Dickersin, K.; Chan, S.; Chalmers, T. C. et al. (1987). "Publication bias and clinical trials". Controlled Clinical Trials 8 (4): 343–353. doi:10.1016/0197-2456(87)90155-3. 
  4. ^ Luijendijk, HJ; Koolman, X (May 2012). "The incentive to publish negative studies: how beta-blockers and depression got stuck in the publication cycle.". J Clin Epidemiol 65 (5): 488–92. doi:10.1016/j.jclinepi.2011.06.022. 
  5. ^ K. Dickersin (March 1990). "The existence of publication bias and risk factors for its occurrence". JAMA 263 (10): 1385–1359. doi:10.1001/jama.263.10.1385. PMID 2406472. 
  6. ^ Jeffrey D. Scargle (2000). "Publication bias: the "file-drawer problem" in scientific inference" (PDF). Journal of Scientific Exploration 14 (2): 94–106. 
  7. ^ Rosenthal R. File drawer problem and tolerance for null results. Psychol Bull 1979;86:638-41.
  8. ^ D.L. Sackett (1979). "Bias in analytic research". J Chronic Dis 32 (1–2): 51–63. doi:10.1016/0021-9681(79)90012-2. PMID 447779. 
  9. ^ N.L. Kerr (1998). "HARKing: Hypothesizing After the Results are Known". Personality and Social Psychology Review 2 (3): 196–217. doi:10.1207/s15327957pspr0203_4. PMID 15647155. 
  10. ^ Dickersin, K.; Min, Y.I. (1993). "NIH clinical trials and publication bias". Online J Curr Clin Trials. PMID 8306005. 
  11. ^ Decullier E, Lheritier V, Chapuis F. Fate of biomedical research protocols and publication bias in France: retrospective cohort study. BMJ 2005;331:19-22
  12. ^ Song F, Parekh-Bhurke S, Hooper L, Loke Y, Ryder J, Sutton A, et al. Extent of publication bias in different categories of research cohorts: a meta-analysis of empirical studies. BMC Med Res Methodol 2009;9:79
  13. ^ Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ 2005;330:753.
  14. ^ Riveros C, Dechartres A, Perrodeau E, Haneef R, Boutron I, Ravaud P. Timing and completeness of trial results posted at and published in journals. PLoS Med 2013;10:e1001566.
  15. ^ a b c Kicinski, M; Springate, D. A.; Kontopantelis, E (2015). "Publication bias in meta-analyses from the Cochrane Database of Systematic Reviews". Statistics in Medicine: n/a. doi:10.1002/sim.6525. PMID 25988604.  edit
  16. ^ Kicinski M. Publication bias in recent meta-analyses. PLoS ONE 2013;8:e81823
  17. ^ Silliman N. Hierarchical selection models with applications in meta-analysis. Journal of American Statistical Association 1997; 92(439):926-936. DOI: 10.1080/01621459.1997.10474047.
  18. ^ 36. Hedges L, Vevea J. Estimating effect size under publication bias: small sample properties and robustness of a random effects selection model. Journal of Educational and Behavioral Statistics 1996; 21(4):299-332. DOI: 10.3102/10769986021004299
  19. ^ Begg C, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics 1994; 50(4):1088-1101.
  20. ^ Egger M, Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. British Medical Journal 1997; 315:629-634. DOI: 10.1136/bmj.315.7109.629
  21. ^ Duval S, Tweedie R. Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000; 56(2):455-463. DOI: 10.1111/j.0006-341X.2000.00455.x.
  22. ^ Sutton AJ, Song F, Gilbody SM, Abrams KR (2000) Modelling publication bias in meta-analysis: a review. Stat Methods Med Res 9:421-445
  23. ^ Kicinski, M (2014). "How does under-reporting of negative and inconclusive results affect the false-positive rate in meta-analysis? A simulation study". BMJ Open 4 (8): e004831. doi:10.1136/bmjopen-2014-004831. PMC 4156818. PMID 25168036.  edit
  24. ^ Ben Goldacre What doctors don't know about the drugs they prescribe
  25. ^ Wilmshurst, Peter. "Dishonesty in Medical Research" (PDF). 
  26. ^ Marc Orlitzky Institutional Logics in the Study of Organizations: The Social Construction of the Relationship between Corporate Social and Financial Performance
  27. ^ Ben Goldacre Backwards step on looking into the future The Guardian, Saturday 23 April 2011
  28. ^ Zhenglun Pan, Thomas A. Trikalinos, Fotini K. Kavvoura, Joseph Lau, John P.A. Ioannidis, "Local literature bias in genetic epidemiology: An empirical evaluation of the Chinese literature". PLoS Medicine, 2(12):e334, 2005 December.
  29. ^ Jin Ling Tang, "Selection Bias in Meta-Analyses of Gene-Disease Associations", PLoS Medicine, 2(12):e409, 2005 December.
  30. ^ Ioannidis J (2005). "Why most published research findings are false". PLoS Med 2 (8): e124. doi:10.1371/journal.pmed.0020124. PMC 1182327. PMID 16060722. 
  31. ^ Wacholder, S.; Chanock, S; Garcia-Closas, M; El Ghormli, L; Rothman, N (March 2004). "Assessing the Probability That a Positive Report is False: An Approach for Molecular Epidemiology Studies". JNCI 96 (6): 434–42. doi:10.1093/jnci/djh075. PMID 15026468. 
  32. ^ (The Washington Post) (2004-09-10). "Medical journal editors take hard line on drug research". Retrieved 2008-02-03. 
  33. ^ "Instructions for Trials authors — Study protocol". 2009-02-15. 
  34. ^ Dickersin, K.; Chalmers, I. (2011). "Recognizing, investigating and dealing with incomplete and biased reporting of clinical research: from Francis Bacon to the WHO". J R Soc Med 104 (12): 532–538. doi:10.1258/jrsm.2011.11k042. PMID 22179297. 

External links[edit]