Skip to Main Content

The Reproducibility Crisis: Interesting Sources

Before you read, reflect: 

  • Have you ever talked about reproducibility in your time at Williams?
  • When and where are these conversation happening? Where are these conversations absent?

As you read through these sources, keep these questions in mind...

  • Based on what you read, what are the major drivers of the reproducibility crisis?
  • What are possible or observed impacts of publishing research that are ultimately not replicable?
  • How will you apply what you read to your own project? Do you notice any issues with replicability in your project?
  • Why do you think reproducibility ultimately matters?

The Reproducibility Crisis

1,500 Scientists Lift the Lid on Reproducibility

"Survey sheds light on the ‘crisis’ rocking research: More than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments. Those are some of the telling figures that emerged from Nature's survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research."

The Replication Crisis - Video by StatisFaction

Replication (re-running studies to confirm results) and reproducibility (the ability to repeat an analyses on data) have come under fire over the past few years. The foundation of science itself is built upon statistical analysis and yet there has been more and more evidence that suggests possibly even the majority of studies cannot be replicated. This "replication crisis" is likely being caused by a number of factors which we'll discuss as well as some of the proposed solutions to ensure that the results we're drawing from scientific studies are reliable. (Abstract from AlexanderStreet)

'Publish or Perish' culture blamed for reproducibility crisis

"Survey of more than 1,600 biomedical researchers also flagged small sample sizes and cherry-picking of data as leading causes of reproducibility problems." (subtitle of aritcle)

No Raw Data, No Science: another possible source of the reproducibility crisis

"In this editorial, I propose that a lack of raw data or data fabrication is another possible cause of irreproducibility. As an Editor-in-Chief of Molecular Brain , I have handled 180 manuscripts since early 2017 and have made 41 editorial decisions categorized as “Revise before review,” requesting that the authors provide raw data. Surprisingly, among those 41 manuscripts, 21 were withdrawn without providing raw data, indicating that requiring raw data drove away more than half of the manuscripts. I rejected 19 out of the remaining 20 manuscripts because of insufficient raw data. Thus, more than 97% of the 41 manuscripts did not present the raw data supporting their results when requested by an editor, suggesting a possibility that the raw data did not exist from the beginning, at least in some portions of these cases."

Replicability and Replication in the Humanities

"A large number of scientists and several news platforms have, over the last few years, been speaking of a replication crisis in various academic disciplines, especially the biomedical and social sciences. This paper answers the novel question of whether we should also pursue replication in the humanities. " (From abastract)

Gaming the Metrics: Misconduct and Manipulation in Academic Research

A book exploring how "the increasing reliance on metrics to evaluate scholarly publications has produced new forms of academic fraud and misconduct."

Transparent Methodology

How to Write Your Methods - PLOS

"What should you include in your methods section, and how much detail is appropriate?"

A Manifesto for Reproducible Science

Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. (Abstract from Nature)

Undergraduate Chefs Dishing Reproducible Research

"This is a recipe for a course that introduces students to the fundamentals of research and demonstrates the benefits of open, transparent, and reproducible research practices."

Resources to Browse

More on the Francesca Gino Scandal:

Data Falsificada by DataColada

Science Integrity Digest

A blog about science integrity, exploring such topics as disgruntled authors, post-publication peer review, problematic images, and more.

A project of The Center for Scientific Integrity

An organization promoting transparency and (you guessed it) integrity in science and scientific publishing, as well as to advocate for best practices and better efficiency in science.

Retraction Watch

A website that tracks retractions as "a window into the scientific process"

More in Retraction Watch:

Top 10 most highly cited retracted papers

Replication probe finds 'statistically improbable data' tied to institute in Bangladesh