Jump to content

Reproducibility

From Wikipedia, the free encyclopedia

This is an old revision of this page, as edited by SHMGTS (talk | contribs) at 11:39, 30 October 2020 (→‎Reproducible research). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

Reproducibility is a major principle of the scientific method. It means that a result obtained by an experiment or observational study should be achieved again with a high degree of agreement when the study is replicated with the same methodology by different researchers. Only after one or several such successful replications should a result be recognized as scientific knowledge.

With a narrower scope, reproducibility has been introduced in computational sciences: Any results should be documented by making all data and code available in such a way that the computations can be executed again with idenitical results.

The terms replicability and repeatability are used in the context of reproducibility, see below.

In recent decades, there has been a rising concern that many published scientific results fail the test of reproducibility, evoking a reproducibility or replicability crisis.

History

Boyle's air pump was, in terms of the 17th century, a complicated and expensive scientific apparatus, making reproducibility of results difficult

The first to stress the importance of reproducibility in science was the Irish chemist Robert Boyle, in England in the 17th century. Boyle's air pump was designed to generate and study vacuum, which at the time was a very controversial concept. Indeed, distinguished philosophers such as René Descartes and Thomas Hobbes denied the very possibility of vacuum existence. Historians of science Steven Shapin and Simon Schaffer, in their 1985 book Leviathan and the Air-Pump, describe the debate between Boyle and Hobbes, ostensibly over the nature of vacuum, as fundamentally an argument about how useful knowledge should be gained. Boyle, a pioneer of the experimental method, maintained that the foundations of knowledge should be constituted by experimentally produced facts, which can be made believable to a scientific community by their reproducibility. By repeating the same experiment over and over again, Boyle argued, the certainty of fact will emerge.

The air pump, which in the 17th century was a complicated and expensive apparatus to build, also led to one of the first documented disputes over the reproducibility of a particular scientific phenomenon. In the 1660s, the Dutch scientist Christiaan Huygens built his own air pump in Amsterdam, the first one outside the direct management of Boyle and his assistant at the time Robert Hooke. Huygens reported an effect he termed "anomalous suspension", in which water appeared to levitate in a glass jar inside his air pump (in fact suspended over an air bubble), but Boyle and Hooke could not replicate this phenomenon in their own pumps. As Shapin and Schaffer describe, “it became clear that unless the phenomenon could be produced in England with one of the two pumps available, then no one in England would accept the claims Huygens had made, or his competence in working the pump”. Huygens was finally invited to England in 1663, and under his personal guidance Hooke was able to replicate anomalous suspension of water. Following this Huygens was elected a Foreign Member of the Royal Society. However, Shapin and Schaffer also note that “the accomplishment of replication was dependent on contingent acts of judgment. One cannot write down a formula saying when replication was or was not achieved”.[1]

The philosopher of science Karl Popper noted briefly in his famous 1934 book The Logic of Scientific Discovery that “non-reproducible single occurrences are of no significance to science”.[2] The statistician Ronald Fisher wrote in his 1935 book The Design of Experiments, which set the foundations for the modern scientific practice of hypothesis testing and statistical significance, that “we may say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us statistically significant results”.[3] Such assertions express a common dogma in modern science that reproducibility is a necessary condition (although not necessarily sufficient) for establishing a scientific fact, and in practice for establishing scientific authority in any field of knowledge. However, as noted above by Shapin and Schaffer, this dogma is not well-formulated quantitatively, such as statistical significance for instance, and therefore it is not explicitly established how many times must a fact be replicated to be considered reproducible.

Replicability, repeatability

Two major steps are naturally distinguished in connection with reproducibility of experimental or observational studies: When new data is obtained in the attempt to achieve it, the term replicability is often used, and the new study is a replication or replicate of the original one. Obtaining the same results when analyzing the data set of the original study again with the same procedures, many authors use the term reproducibility in a narrow, technical sense coming from its use in computational research. Repeatability is related to the repetition of the experiment within the same study by the same researchers. Reproducibility in the original, wide sense is only acknowledged if a replication performed by an independent researcher team is successful.

Unfortunately, the terms reproducibility and replicability appear in the literature also with reversed meaning. [4] [5]

Measures of reproducibility and repeatability

In chemistry, the terms reproducibility and repeatability are used with a specific quantitative meaning: In inter-laboratory experiments, a concentration or other quantity of a chemical substance is measured repeatedly in different laboratories to assess the variability of the measurements. Then, the standard deviation of the difference between two values obtained within the same laboratory is called repeatability. The standard deviation for the difference between two measurement from different laboratories is called reproducibility.[6] These measures are related to the more general concept of variance components in metrology.

Reproducible research

Reproducible research method

The term reproducible research refers to the idea that scientific results should be documented in such a way that their deduction is fully tranparent. This requires a detailed description of the methods used to obtain the data[7][8] and making the full dataset and the code to calculate the results easily accessible.[9][10][11][12][13] [14] This is the essential part of open science.

There are systems that facilitate such documentation, like the R Markdown language[15] or the Jupyter notebook.[16] The Open Science Framework provides a platform and useful tools to support reproducible research.

Reproducible research in practice

Psychology has seen a renewal of internal concerns about irreproducible results (see the entry on replicability crisis for empirical results on success rates of replications). Researchers showed in a 2006 study that, of 141 authors of a publication from the American Psychology Association (APA) empirical articles, 103 (73%) did not respond with their data over a six-month period.[17] In a follow up study published in 2015, it was found that 246 out of 394 contacted authors of papers in APA journals did not share their data upon request (62%).[18] In a 2012 paper, it was suggested that researchers should publish data along with their works, and a dataset was released alongside as a demonstration.[19] In 2017, an article published in Scientific Data suggested that this may not be sufficient and that the whole analysis context should be disclosed.[20]

A 2018 study published in the journal PLOS ONE found that 14.4% of a sample of public health researchers had shared their data or code or both.[21]

There have been initiatives to improve reporting and hence reproducibility in the medical literature for many years, beginning with the CONSORT initiative, which is now part of a wider initiative, the EQUATOR Network. This group has recently turned its attention to how better reporting might reduce waste in research,[22] especially biomedical research.

Reproducible research is key to new discoveries in pharmacology. A Phase I discovery will be followed by Phase II reproductions as a drug develops towards commercial production. In recent decades Phase II success has fallen from 28% to 18%. A 2011 study found that 65% of medical studies were inconsistent when re-tested, and only 6% were completely reproducible.[23]

Noteworthy irreproducible results

Hideyo Noguchi became famous for correctly identifying the bacterial agent of syphilis, but also claimed that he could culture this agent in his laboratory. Nobody else has been able to produce this latter result.[24]

In March 1989, University of Utah chemists Stanley Pons and Martin Fleischmann reported the production of excess heat that could only be explained by a nuclear process ("cold fusion"). The report was astounding given the simplicity of the equipment: it was essentially an electrolysis cell containing heavy water and a palladium cathode which rapidly absorbed the deuterium produced during electrolysis. The news media reported on the experiments widely, and it was a front-page item on many newspapers around the world (see science by press conference). Over the next several months others tried to replicate the experiment, but were unsuccessful.[25]

Nikola Tesla claimed as early as 1899 to have used a high frequency current to light gas-filled lamps from over 25 miles (40 km) away without using wires. In 1904 he built Wardenclyffe Tower on Long Island to demonstrate means to send and receive power without connecting wires. The facility was never fully operational and was not completed due to economic problems, so no attempt to reproduce his first result was ever carried out.[26]

Other examples which contrary evidence has refuted the original claim:

See also

References

  1. ^ Steven Shapin and Simon Schaffer, Leviathan and the Air-Pump, Princeton University Press, Princeton, New Jersey (1985).
  2. ^ This citation is from the 1959 translation to English, Karl Popper, The Logic of Scientific Discovery, Routledge, London, 1992, p. 66.
  3. ^ Ronald Fisher, The Design of Experiments, (1971) [1935](9th ed.), Macmillan, p. 14.
  4. ^ Barba, Lorena A. "Terminologies for Reproducible Research" (PDF). Retrieved 2020-10-15.
  5. ^ Liberman, Mark. "Replicability vs. reproducibility — or is it the other way round?". Retrieved 2020-10-15.
  6. ^ Subcommittee E11.20 on Test Method Evaluation and Quality Control (2014). "Standard Practice for Use of the Terms Precision and Bias in ASTM Test Methods". ASTM International. ASTM E177.{{cite web}}: CS1 maint: numeric names: authors list (link)(subscription required)
  7. ^ King, Gary (1995). "Replication, Replication". PS: Political Science and Politics. 28 (3): 444–452. doi:10.2307/420301. ISSN 1049-0965. JSTOR 420301.
  8. ^ Kühne, Martin; Liehr, Andreas W. (2009). "Improving the Traditional Information Management in Natural Sciences". Data Science Journal. 8 (1): 18–27. doi:10.2481/dsj.8.18.
  9. ^ Fomel, Sergey; Claerbout, Jon (2009). "Guest Editors' Introduction: Reproducible Research". Computing in Science and Engineering. 11 (1): 5–7. Bibcode:2009CSE....11a...5F. doi:10.1109/MCSE.2009.14.
  10. ^ Buckheit, Jonathan B.; Donoho, David L. (May 1995). WaveLab and Reproducible Research (PDF) (Report). California, United States: Stanford University, Department of Statistics. Technical Report No. 474. Retrieved 5 January 2015.
  11. ^ "The Yale Law School Round Table on Data and Core Sharing: "Reproducible Research"". Computing in Science and Engineering. 12 (5): 8–12. 2010. doi:10.1109/MCSE.2010.113.
  12. ^ Marwick, Ben (2016). "Computational reproducibility in archaeological research: Basic principles and a case study of their implementation". Journal of Archaeological Method and Theory. 24 (2): 424–450. doi:10.1007/s10816-015-9272-9. S2CID 43958561.
  13. ^ Goodman, Steven N.; Fanelli, Daniele; Ioannidis, John P. A. (1 June 2016). "What does research reproducibility mean?". Science Translational Medicine. 8 (341): 341ps12. doi:10.1126/scitranslmed.aaf5027. PMID 27252173.
  14. ^ Harris J.K; Johnson K.J; Combs T.B; Carothers B.J; Luke D.A; Wang X (2019). "Three Changes Public Health Scientists Can Make to Help Build a Culture of Reproducible Research". Public Health Rep. Public Health Reports. 134 (2): 109–111. doi:10.1177/0033354918821076. ISSN 0033-3549. OCLC 7991854250. PMC 6410469. PMID 30657732.
  15. ^ Marwick, Ben; Boettiger, Carl; Mullen, Lincoln (29 September 2017). "Packaging data analytical work reproducibly using R (and friends)". The American Statistician. 72: 80–88. doi:10.1080/00031305.2017.1375986. S2CID 125412832.
  16. ^ Kluyver, Thomas; Ragan-Kelley, Benjamin; Perez, Fernando; Granger, Brian; Bussonnier, Matthias; Frederic, Jonathan; Kelley, Kyle; Hamrick, Jessica; Grout, Jason; Corlay, Sylvain (2016). "Jupyter Notebooks-a publishing format for reproducible computational workflows". In Loizides, F; Schmidt, B (eds.). Positioning and Power in Academic Publishing: Players, Agents and Agendas. IOS Press. pp. 87–90. {{cite book}}: External link in |chapterurl= (help); Unknown parameter |booktitle= ignored (help); Unknown parameter |chapterurl= ignored (|chapter-url= suggested) (help)
  17. ^ Wicherts, J. M.; Borsboom, D.; Kats, J.; Molenaar, D. (2006). "The poor availability of psychological research data for reanalysis". American Psychologist. 61 (7): 726–728. doi:10.1037/0003-066X.61.7.726. PMID 17032082.
  18. ^ Vanpaemel, W.; Vermorgen, M.; Deriemaecker, L.; Storms, G. (2015). "Are we wasting a good crisis? The availability of psychological research data after the storm". Collabra. 1 (1): 1–5. doi:10.1525/collabra.13.
  19. ^ Wicherts, J. M.; Bakker, M. (2012). "Publish (your data) or (let the data) perish! Why not publish your data too?". Intelligence. 40 (2): 73–76. doi:10.1016/j.intell.2012.01.004.
  20. ^ Pasquier, Thomas; Lau, Matthew K.; Trisovic, Ana; Boose, Emery R.; Couturier, Ben; Crosas, Mercè; Ellison, Aaron M.; Gibson, Valerie; Jones, Chris R.; Seltzer, Margo (5 September 2017). "If these data could talk". Scientific Data. 4: 170114. Bibcode:2017NatSD...470114P. doi:10.1038/sdata.2017.114. PMC 5584398. PMID 28872630.
  21. ^ Harris, Jenine K.; Johnson, Kimberly J.; Carothers, Bobbi J.; Combs, Todd B.; Luke, Douglas A.; Wang, Xiaoyan (2018). "Use of reproducible research practices in public health: A survey of public health analysts". PLOS ONE. 13 (9): e0202447. Bibcode:2018PLoSO..1302447H. doi:10.1371/journal.pone.0202447. ISSN 1932-6203. OCLC 7891624396. PMC 6135378. PMID 30208041.{{cite journal}}: CS1 maint: unflagged free DOI (link)
  22. ^ "Research Waste/EQUATOR Conference | Research Waste". researchwaste.net. Archived from the original on 29 October 2016.
  23. ^ Prinz, F.; Schlange, T.; Asadullah, K. (2011). "Believe it or not: How much can we rely on published data on potential drug targets?". Nature Reviews Drug Discovery. 10 (9): 712. doi:10.1038/nrd3439-c1. PMID 21892149.
  24. ^ Tan, SY; Furubayashi, J (2014). "Hideyo Noguchi (1876-1928): Distinguished bacteriologist". Singapore Medical Journal. 55 (10): 550–551. doi:10.11622/smedj.2014140. ISSN 0037-5675. PMC 4293967. PMID 25631898.
  25. ^ Browne, Malcolm (3 May 1989). "Physicists Debunk Claim Of a New Kind of Fusion". New York Times. Retrieved 3 February 2017.
  26. ^ Cheney, Margaret (1999), Tesla, Master of Lightning, New York: Barnes & Noble Books, ISBN 0-7607-1005-8, pp. 107.; "Unable to overcome his financial burdens, he was forced to close the laboratory in 1905."
  27. ^ Dominus, Susan (October 18, 2017). "When the Revolution Came for Amy Cuddy". New York Times Magazine.

Further reading

External links

Look up reproducibility in Wiktionary, the free dictionary.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Reproducibility&oldid=986198042"