Replication crisis

The replication crisis^[a] is an ongoing methodological crisis in which the results of many scientific studies are difficult or impossible to reproduce. Because the reproducibility of empirical results is an essential part of the scientific method,^[2] such failures undermine the credibility of theories building on them and potentially call into question substantial parts of scientific knowledge.

The replication crisis is frequently discussed in relation to psychology and medicine, where considerable efforts have been undertaken to reinvestigate classic results, to determine whether they are reliable, and if they turn out not to be, the reasons for the failure.^[3]^[4] Data strongly indicate that other natural and social sciences are affected as well.^[5]

The phrase replication crisis was coined in the early 2010s^[6] as part of a growing awareness of the problem. Considerations of causes and remedies have given rise to a new scientific discipline, metascience,^[7] which uses methods of empirical research to examine empirical research practice.^[8]

Considerations about reproducibility can be placed into two categories. Reproducibility in the narrow sense refers to re-examining and validating the analysis of a given set of data. Replication refers to repeating the experiment or study to obtain new, independent data with the goal of reaching the same or similar conclusions.

Background

Replication

Replication has been called "the cornerstone of science".^[9]^[10] Environmental health scientist Stefan Schmidt began a 2009 review with this description of replication:

Replication is one of the central issues in any empirical science. To confirm results or hypotheses by a repetition procedure is at the basis of any scientific conception. A replication experiment to demonstrate that the same findings can be obtained in any other place by any other researcher is conceived as an operationalization of objectivity. It is the proof that the experiment reflects knowledge that can be separated from the specific circumstances (such as time, place, or persons) under which it was gained.^[11]

But there is limited consensus on how to define replication and potentially related concepts.^[12]^[13]^[11] A number of types of replication have been identified:

Direct or exact replication, where an experimental procedure is repeated as closely as possible.^[11]^[14]
Systematic replication, where an experimental procedure is largely repeated, with some intentional changes.^[14]
Conceptual replication, where a finding or hypothesis is tested using a different procedure.^[11]^[14] Conceptual replication allows testing for generalizability and veracity of a result or hypothesis.^[14]

Reproducibility can also be distinguished from replication, as referring to reproducing the same results using the same data set. Reproducibility of this type is why many researchers make their data available to others for testing.^[15]

The replication crisis does not necessarily mean these fields are unscientific.^[16]^[17]^[18] Rather, this process is part of the scientific process in which old ideas or those that cannot withstand careful scrutiny are pruned,^[19]^[20] although this pruning process is not always effective.^[21]^[22]

A hypothesis is generally considered to be supported when the results match the predicted pattern and that pattern of results is found to be statistically significant. Results are considered significant whenever the relative frequency of the observed pattern falls below an arbitrarily chosen value (i.e. the significance level) when assuming the null hypothesis is true. This generally answers the question of how unlikely results would be if no difference existed at the level of the statistical population. If the probability associated with the test statistic exceeds the chosen critical value, the results are considered statistically significant.^[23] The corresponding probability of exceeding the critical value is depicted as p < 0.05, where p (typically referred to as the "p-value") is the probability level. This should result in 5% of hypotheses that are supported being false positives (an incorrect hypothesis being erroneously found correct), assuming the studies meet all of the statistical assumptions. Some fields use smaller p-values, such as p < 0.01 (1% chance of a false positive) or p < 0.001 (0.1% chance of a false positive). But a smaller chance of a false positive often requires greater sample sizes or a greater chance of a false negative (a correct hypothesis being erroneously found incorrect). Although p-value testing is the most commonly used method, it is not the only method.

Statistics

Certain terms commonly used in discussions of the replication crisis have technically precise meanings, which are presented here.^[1]

In the most common case, null hypothesis testing, there are two hypotheses, a null hypothesis $H_{0}$ and an alternative hypothesis $H_{1}$ . The null hypothesis is typically of the form "X and Y are statistically independent". For example, the null hypothesis might be "taking drug X does not change 1-year recovery rate from disease Y", and the alternative hypothesis is that it does change.

As testing for full statistical independence is difficult, the full null hypothesis is often reduced to a simplified null hypothesis "the effect size is 0", where "effect size" is a real number that is 0 if the full null hypothesis is true, and the larger the effect size is, the more the null hypothesis is false.^[24] For example, if X is binary, then the effect size might be defined as the change in the expectation of Y upon a change of X: $({\text{effect size}})=\mathbb {E} [Y|X=1]-\mathbb {E} [Y|X=0]$ Note that the effect size as defined above might be zero even if X and Y are not independent, such as when $Y\sim {\mathcal {N}}(0,1+X)$ . Since different definitions of "effect size" capture different ways for X and Y to be dependent, there are many different definitions of effect size.

In practice, effect sizes cannot be directly observed, but must be measured by statistical estimators. For example, the above definition of effect size is often measured by Cohen's d estimator. The same effect size might have multiple estimators, as they have tradeoffs between efficiency, bias, variance, etc. This further increases the number of possible statistical quantities that can be computed on a single dataset. When an estimator for an effect size is used for statistical testing, it is called a test statistic.

Illustration of the 4 possible outcomes of a null hypothesis test: false negative, true negative, false positive, true positive. In this illustration, the hypothesis test is a one-sided threshold test.

A null hypothesis test is a decision procedure which takes in some data, and outputs either $H_{0}$ or $H_{1}$ . If it outputs $H_{1}$ , it is usually stated as "there is a statistically significant effect" or "the null hypothesis is rejected".

Often, the statistical test is a (one-sided) threshold test, which is structured as follows:

Gather data $D$ .
Compute a test statistic $t[D]$ for the data.
Compare the test statistic against a critical value/threshold $t_{\text{threshold}}$ . If $t[D]>t_{\text{threshold}}$ , then output $H_{1}$ , else, output $H_{0}$ .

A two-sided threshold test is similar, but with two thresholds, such that it outputs $H_{1}$ if either $t[D]<t_{\text{threshold}}^{-}$ or $t[D]>t_{\text{threshold}}^{+}$

There are 4 possible outcomes of a null hypothesis test: false negative, true negative, false positive, true positive. A false negative means that $H_{0}$ is true, but the test outcome is $H_{1}$ ; a true negative means that $H_{0}$ is true, and the test outcome is $H_{0}$ , etc.

	Probability to reject $H_{0}$	Probability to not reject $H_{0}$
If $H_{0}$ is True	α	1-α
If $H_{1}$ is True	1-β (power)	β

Interaction between sample size, effect size, and statistical power. Distributions of sample means under the null (θ=0) and alternative hypotheses are shown. The shaded red area represents significance (α), held constant at 0.05, while the shaded green area represents statistical power (1-β). As the sample size increases, the distributions narrow, leading to clearer separation between the hypotheses and higher power. Similarly, a larger effect size increases the distance between the distributions, resulting in greater power.

Significance level, false positive rate, or the alpha level, is the probability of finding the alternative to be true when the null hypothesis is true: $({\text{significance}}):=\alpha :=Pr({\text{find }}H_{1}|H_{0})$ For example, when the test is a one-sided threshold test, then $\alpha =Pr_{D\sim H_{0}}(t[D]>t_{\text{threshold}})$ where $D\sim H_{0}$ means "the data is sampled from $H_{0}$ ".

Statistical power, true positive rate, is the probability of finding the alternative to be true when the alternative hypothesis is true: $({\text{power}}):=1-\beta :=Pr({\text{find }}H_{1}|H_{1})$ where $\beta$ is also called the false negative rate. For example, when the test is a one-sided threshold test, then $1-\beta =Pr_{D\sim H_{1}}(t[D]>t_{\text{threshold}})$ .

Given a statistical test and a data set $D$ , the corresponding p-value is the probability that the test statistic is at least as extreme, conditional on $H_{0}$ . For example, for a one-sided threshold test, $p[D]=Pr_{D'\sim H_{0}}(t[D']>t[D])$ If the null hypothesis is true, then the p-value is distributed uniformly on $[0,1]$ . Otherwise, it is typically peaked at $p=0.0$ and roughly exponential, though the precise shape of the p-value distribution depends on what the alternative hypothesis is.^[25]^[26]

Since the p-value is distributed uniformly on $[0,1]$ conditional on the null hypothesis, one may construct a statistical test with any significance level $\alpha$ by simply computing the p-value, then output $H_{1}$ if $p[D]<\alpha$ . This is usually stated as "the null hypothesis is rejected at significance level $\alpha$ ", or " $H_{1}\;(p<\alpha )$ ", such as "smoking is correlated with cancer (p < 0.001)".

History

The beginning of the replication crisis can be traced to a number of events in the early 2010s. Philosopher of science and social epistemologist Felipe Romero identified four events that can be considered precursors to the ongoing crisis:^[27]

Controversies around social priming research: In the early 2010s, the well-known "elderly-walking" study^[28] by social psychologist John Bargh and colleagues failed to replicate in two direct replications.^[29] This experiment was part of a series of three studies that had been widely cited throughout the years, was regularly taught in university courses, and had inspired a large number of conceptual replications. Failures to replicate the study led to much controversy and a heated debate involving the original authors.^[30] Notably, many of the conceptual replications of the original studies also failed to replicate in subsequent direct replications.^[31]^[32]^[33]^[34]
Controversies around experiments on extrasensory perception: Social psychologist Daryl Bem conducted a series of experiments supposedly providing evidence for the controversial phenomenon of extrasensory perception.^[35] Bem was highly criticized for his study's methodology and upon reanalysis of the data, no evidence was found for the existence of extrasensory perception.^[36] The experiment also failed to replicate in subsequent direct replications.^[37] According to Romero, what the community found particularly upsetting was that many of the flawed procedures and statistical tools used in Bem's studies were part of common research practice in psychology.
Amgen and Bayer reports on lack of replicability in biomedical research: Scientists from biotech companies Amgen and Bayer Healthcare reported alarmingly low replication rates (11–20%) of landmark findings in preclinical oncological research.^[38]
Publication of studies on p-hacking and questionable research practices: Since the late 2000s, a number of studies in metascience showed how commonly adopted practices in many scientific fields, such as exploiting the flexibility of the process of data collection and reporting, could greatly increase the probability of false positive results.^[39]^[40]^[41] These studies suggested how a significant proportion of published literature in several scientific fields could be nonreplicable research.

This series of events generated a great deal of skepticism about the validity of existing research in light of widespread methodological flaws and failures to replicate findings. This led prominent scholars to declare a "crisis of confidence" in psychology and other fields,^[42] and the ensuing situation came to be known as the "replication crisis".

Although the beginning of the replication crisis can be traced to the early 2010s, some authors point out that concerns about replicability and research practices in the social sciences had been expressed much earlier. Romero notes that authors voiced concerns about the lack of direct replications in psychological research in the late 1960s and early 1970s.^[43]^[44] He also writes that certain studies in the 1990s were already reporting that journal editors and reviewers are generally biased against publishing replication studies.^[45]^[46]

In the social sciences, the blog Data Colada (whose three authors coined the term "p-hacking" in a 2014 paper) has been credited with contributing to the start of the replication crisis.^[47]^[48]^[49]

University of Virginia professor and cognitive psychologist Barbara A. Spellman has written that many criticisms of research practices and concerns about replicability of research are not new.^[50] She reports that between the late 1950s and the 1990s, scholars were already expressing concerns about a possible crisis of replication,^[51] a suspiciously high rate of positive findings,^[52] questionable research practices (QRPs),^[53] the effects of publication bias,^[54] issues with statistical power,^[55]^[56] and bad standards of reporting.^[51]

Spellman also identifies reasons that the reiteration of these criticisms and concerns in recent years led to a full-blown crisis and challenges to the status quo. First, technological improvements facilitated conducting and disseminating replication studies, and analyzing large swaths of literature for systemic problems. Second, the research community's increasing size and diversity made the work of established members more easily scrutinized by other community members unfamiliar with them. According to Spellman, these factors, coupled with increasingly limited resources and misaligned incentives for doing scientific work, led to a crisis in psychology and other fields.^[50]

According to Andrew Gelman,^[57] the works of Paul Meehl, Jacob Cohen, and Tversky and Kahneman in the 1960s-70s were early warnings of replication crisis. In discussing the origins of the problem, Kahneman himself noted historical precedents in subliminal perception and dissonance reduction replication failures.^[58]

It had been repeatedly pointed out since 1962^[55] that most psychological studies have low power (true positive rate), but low power persisted for 50 years, indicating a structural and persistent problem in psychological research.^[59]^[60]

Prevalence

In psychology

Several factors have combined to put psychology at the center of the conversation.^[61]^[62] Some areas of psychology once considered solid, such as social priming and ego depletion,^[63] have come under increased scrutiny due to failed replications.^[64] Much of the focus has been on social psychology,^[65] although other areas of psychology such as clinical psychology,^[66]^[67]^[68] developmental psychology,^[69]^[70]^[71] and educational research have also been implicated.^[72]^[73]^[74]^[75]^[76]

In August 2015, the first open empirical study of reproducibility in psychology was published, called The Reproducibility Project: Psychology. Coordinated by psychologist Brian Nosek, researchers redid 100 studies in psychological science from three high-ranking psychology journals (Journal of Personality and Social Psychology, Journal of Experimental Psychology: Learning, Memory, and Cognition, and Psychological Science). 97 of the original studies had significant effects, but of those 97, only 36% of the replications yielded significant findings (p value below 0.05).^[12] The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies. The same paper examined the reproducibility rates and effect sizes by journal and discipline. Study replication rates were 23% for the Journal of Personality and Social Psychology, 48% for Journal of Experimental Psychology: Learning, Memory, and Cognition, and 38% for Psychological Science. Studies in the field of cognitive psychology had a higher replication rate (50%) than studies in the field of social psychology (25%).^[77]

Of the 64% of non-replications, only 25% disproved the original result (at statistical significance). The other 49% were inconclusive, neither supporting nor contradicting the original result. This is because many replications were underpowered, with a sample 2.5 times smaller than the original.^[78]

A study published in 2018 in Nature Human Behaviour replicated 21 social and behavioral science papers from Nature and Science, finding that only about 62% could successfully reproduce original results.^[79]^[80]

Similarly, in a study conducted under the auspices of the Center for Open Science, a team of 186 researchers from 60 different laboratories (representing 36 different nationalities from six different continents) conducted replications of 28 classic and contemporary findings in psychology.^[81]^[82] The study's focus was not only whether the original papers' findings replicated but also the extent to which findings varied as a function of variations in samples and contexts. Overall, 50% of the 28 findings failed to replicate despite massive sample sizes. But if a finding replicated, then it replicated in most samples. If a finding was not replicated, then it failed to replicate with little variation across samples and contexts. This evidence is inconsistent with a proposed explanation that failures to replicate in psychology are likely due to changes in the sample between the original and replication study.^[82]

Results of a 2022 study suggest that many earlier brain–phenotype studies ("brain-wide association studies" (BWAS)) produced invalid conclusions as the replication of such studies requires samples from thousands of individuals due to small effect sizes.^[83]^[84]

In medicine

Results from The Reproducibility Project: Cancer Biology suggest most studies of the cancer research sector may not be replicable.

Of 49 medical studies from 1990 to 2003 with more than 1000 citations, 92% found that the studied therapies were effective. Of these studies, 16% were contradicted by subsequent studies, 16% had found stronger effects than did subsequent studies, 44% were replicated, and 24% remained largely unchallenged.^[85] A 2011 analysis by researchers with pharmaceutical company Bayer found that, at most, a quarter of Bayer's in-house findings replicated the original results.^[86] But the analysis of Bayer's results found that the results that did replicate could often be successfully used for clinical applications.^[87]

In a 2012 paper, C. Glenn Begley, a biotech consultant working at Amgen, and Lee Ellis, a medical researcher at the University of Texas, found that only 11% of 53 pre-clinical cancer studies had replications that could confirm conclusions from the original studies.^[38] In late 2021, The Reproducibility Project: Cancer Biology examined 53 top papers about cancer published between 2010 and 2012 and showed that among studies that provided sufficient information to be redone, the effect sizes were 85% smaller on average than the original findings.^[88]^[89] A survey of cancer researchers found that half of them had been unable to reproduce a published result.^[90] Another report estimated that almost half of randomized controlled trials contained flawed data (based on the analysis of anonymized individual participant data (IPD) from more than 150 trials).^[91]

In other disciplines

In economics

Economics has lagged behind other social sciences and psychology in its attempts to assess replication rates and increase the number of studies that attempt replication.^[13] A 2016 study in the journal Science replicated 18 experimental studies published in two leading economics journals, The American Economic Review and the Quarterly Journal of Economics, between 2011 and 2014. It found that about 39% failed to reproduce the original results.^[92]^[93]^[94] About 20% of studies published in The American Economic Review are contradicted by other studies despite relying on the same or similar data sets.^[95] A study of empirical findings in the Strategic Management Journal found that about 30% of 27 retested articles showed statistically insignificant results for previously significant findings, whereas about 4% showed statistically significant results for previously insignificant findings.^[96]

In water resource management

A 2019 study in Scientific Data estimated with 95% confidence that of 1,989 articles on water resources and management published in 2017, study results might be reproduced for only 0.6% to 6.8%, largely because the articles did not provide sufficient information to allow for replication.^[97]

Across fields

A 2016 survey by Nature on 1,576 researchers who took a brief online questionnaire on reproducibility found that more than 70% of researchers have tried and failed to reproduce another scientist's experiment results (including 87% of chemists, 77% of biologists, 69% of physicists and engineers, 67% of medical researchers, 64% of earth and environmental scientists, and 62% of all others), and more than half have failed to reproduce their own experiments. But fewer than 20% had been contacted by another researcher unable to reproduce their work. The survey found that fewer than 31% of researchers believe that failure to reproduce results means that the original result is probably wrong, although 52% agree that a significant replication crisis exists. Most researchers said they still trust the published literature.^[5]^[98] In 2010, Fanelli (2010)^[99] found that 91.5% of psychiatry/psychology studies confirmed the effects they were looking for, and concluded that the odds of this happening (a positive result) was around five times higher than in fields such as astronomy or geosciences. Fanelli argued that this is because researchers in "softer" sciences have fewer constraints to their conscious and unconscious biases.

Early analysis of result-blind peer review, which is less affected by publication bias, has estimated that 61% of result-blind studies in biomedicine and psychology have led to null results, in contrast to an estimated 5% to 20% in earlier research.^[100]

In 2021, a study conducted by University of California, San Diego found that papers that cannot be replicated are more likely to be cited.^[101] Nonreplicable publications are often cited more even after a replication study is published.^[102]

Causes

There are many proposed causes for the replication crisis.

Historical and sociological causes

The replication crisis may be triggered by the "generation of new data and scientific publications at an unprecedented rate" that leads to "desperation to publish or perish" and failure to adhere to good scientific practice.^[103]

Predictions of an impending crisis in the quality-control mechanism of science can be traced back several decades. Derek de Solla Price—considered the father of scientometrics, the quantitative study of science—predicted in 1963 that science could reach "senility" as a result of its own exponential growth.^[104] Some present-day literature seems to vindicate this "overflow" prophecy, lamenting the decay in both attention and quality.^[105]^[106]

Historian Philip Mirowski argues that the decline of scientific quality can be connected to its commodification, especially spurred by major corporations' profit-driven decision to outsource their research to universities and contract research organizations.^[107]

Social systems theory, as expounded in the work of German sociologist Niklas Luhmann, inspires a similar diagnosis. This theory holds that each system, such as economy, science, religion, and media, communicates using its own code: true and false for science, profit and loss for the economy, news and no-news for the media, and so on.^[108]^[109] According to some sociologists, science's mediatization,^[110] commodification,^[107] and politicization,^[110]^[111] as a result of the structural coupling among systems, have led to a confusion of the original system codes.

Problems with the publication system in science

Publication bias

A major cause of low reproducibility is the publication bias stemming from the fact that statistically non-significant results and seemingly unoriginal replications are rarely published. Only a very small proportion of academic journals in psychology and neurosciences explicitly welcomed submissions of replication studies in their aim and scope or instructions to authors.^[112]^[113] This does not encourage reporting on, or even attempts to perform, replication studies. Among 1,576 researchers Nature surveyed in 2016, only a minority had ever attempted to publish a replication, and several respondents who had published failed replications noted that editors and reviewers demanded that they play down comparisons with the original studies.^[5]^[98] An analysis of 4,270 empirical studies in 18 business journals from 1970 to 1991 reported that less than 10% of accounting, economics, and finance articles and 5% of management and marketing articles were replication studies.^[92]^[114] Publication bias is augmented by the pressure to publish and the author's own confirmation bias,^[b] and is an inherent hazard in the field, requiring a certain degree of skepticism on the part of readers.^[41]

Publication bias leads to what psychologist Robert Rosenthal calls the "file drawer effect". The file drawer effect is the idea that as a consequence of the publication bias, a significant number of negative results^[c] are not published. According to philosopher of science Felipe Romero, this tends to produce "misleading literature and biased meta-analytic studies",^[27] and when publication bias is considered along with the fact that a majority of tested hypotheses might be false a priori, it is plausible that a considerable proportion of research findings might be false positives, as shown by metascientist John Ioannidis.^[1] In turn, a high proportion of false positives in the published literature can explain why many findings are nonreproducible.^[27]

Another publication bias is that studies that do not reject the null hypothesis are scrutinized asymmetrically. For example, they are likely to be rejected as being difficult to interpret or having a Type II error. Studies that do reject the null hypothesis are not likely to be rejected for those reasons.^[116]

In popular media, there is another element of publication bias: the desire to make research accessible to the public led to oversimplification and exaggeration of findings, creating unrealistic expectations and amplifying the impact of non-replications. In contrast, null results and failures to replicate tend to go unreported. This explanation may apply to power posing's replication crisis.^[117]

Mathematical errors

Even high-impact journals have a significant fraction of mathematical errors in their use of statistics. For example, 11% of statistical results published in Nature and BMJ in 2001 are "incongruent", meaning that the reported p-value is mathematically different from what it should be if it were correctly calculated from the reported test statistic. These errors were likely from typesetting, rounding, and transcription errors.^[118]

Among 157 neuroscience papers published in five top-ranking journals that attempt to show that two experimental effects are different, 78 erroneously tested instead for whether one effect is significant while the other is not, and 79 correctly tested for whether their difference is significantly different from 0.^[119]

"Publish or perish" culture

The consequences for replicability of the publication bias are exacerbated by academia's "publish or perish" culture. As explained by metascientist Daniele Fanelli, "publish or perish" culture is a sociological aspect of academia whereby scientists work in an environment with very high pressure to have their work published in recognized journals. This is the consequence of the academic work environment being hypercompetitive and of bibliometric parameters (e.g., number of publications) being increasingly used to evaluate scientific careers.^[120] According to Fanelli, this pushes scientists to employ a number of strategies aimed at making results "publishable". In the context of publication bias, this can mean adopting behaviors aimed at making results positive or statistically significant, often at the expense of their validity (see QRPs, section 4.3).^[120]

According to Center for Open Science founder Brian Nosek and his colleagues, "publish or perish" culture created a situation whereby the goals and values of single scientists (e.g., publishability) are not aligned with the general goals of science (e.g., pursuing scientific truth). This is detrimental to the validity of published findings.^[121]

Philosopher Brian D. Earp and psychologist Jim A. C. Everett argue that, although replication is in the best interests of academics and researchers as a group, features of academic psychological culture discourage replication by individual researchers. They argue that performing replications can be time-consuming, and take away resources from projects that reflect the researcher's original thinking. They are harder to publish, largely because they are unoriginal, and even when they can be published they are unlikely to be viewed as major contributions to the field. Replications "bring less recognition and reward, including grant money, to their authors".^[122]

In his 1971 book Scientific Knowledge and Its Social Problems, philosopher and historian of science Jerome R. Ravetz predicted that science—in its progression from "little" science composed of isolated communities of researchers to "big" science or "techno-science"—would suffer major problems in its internal system of quality control. He recognized that the incentive structure for modern scientists could become dysfunctional, creating perverse incentives to publish any findings, however dubious. According to Ravetz, quality in science is maintained only when there is a community of scholars, linked by a set of shared norms and standards, who are willing and able to hold each other accountable.

Standards of reporting

Certain publishing practices also make it difficult to conduct replications and to monitor the severity of the reproducibility crisis, for articles often come with insufficient descriptions for other scholars to reproduce the study. The Reproducibility Project: Cancer Biology showed that of 193 experiments from 53 top papers about cancer published between 2010 and 2012, only 50 experiments from 23 papers have authors who provided enough information for researchers to redo the studies, sometimes with modifications. None of the 193 papers examined had its experimental protocols fully described and replicating 70% of experiments required asking for key reagents.^[88]^[89] The aforementioned study of empirical findings in the Strategic Management Journal found that 70% of 88 articles could not be replicated due to a lack of sufficient information for data or procedures.^[92]^[96] In water resources and management, most of 1,987 articles published in 2017 were not replicable because of a lack of available information shared online.^[97] In studies of event-related potentials, only two-thirds the information needed to replicate a study were reported in a sample of 150 studies, highlighting that there are substantial gaps in reporting.^[123]

Procedural bias

By the Duhem-Quine thesis, scientific results are interpreted by both a substantive theory and a theory of instruments. For example, astronomical observations depend both on the theory of astronomical objects and the theory of telescopes. A large amount of non-replicable research might accumulate if there is a bias of the following kind: faced with a null result, a scientist prefers to treat the data as saying the instrument is insufficient; faced with a non-null result, a scientist prefers to accept the instrument as good, and treat the data as saying something about the substantive theory.^[124]

Cultural evolution

Smaldino and McElreath^[60] proposed a simple model for the cultural evolution of scientific practice. Each lab randomly decides to produce novel research or replication research, at different fixed levels of false positive rate, true positive rate, replication rate, and productivity (its "traits"). A lab might use more "effort", making the ROC curve more convex but decreasing productivity. A lab accumulates a score over its lifetime that increases with publications and decreases when another lab fails to replicate its results. At regular intervals, a random lab "dies" and another "reproduces" a child lab with a similar trait as its parent. Labs with higher scores are more likely to reproduce. Under certain parameter settings, the population of labs converge to maximum productivity even at the price of very high false positive rates.

Questionable research practices and fraud

Questionable research practices (QRPs) are intentional behaviors that capitalize on the gray area of acceptable scientific behavior or exploit the researcher degrees of freedom (researcher DF), which can contribute to the irreproducibility of results by increasing the probability of false positive results.^[125]^[126]^[41] Researcher DF are seen in hypothesis formulation, design of experiments, data collection and analysis, and reporting of research.^[126] Some examples of QRPs are data dredging,^[126]^[127]^[40]^[d] selective reporting,^[125]^[126]^[127]^[40]^[e] and HARKing (hypothesising after results are known).^[126]^[127]^[40]^[f] In medicine, irreproducible studies have six features in common. These include investigators not being blinded to the experimental versus the control arms, a failure to repeat experiments, a lack of positive and negative controls, failing to report all the data, inappropriate use of statistical tests, and use of reagents that were not appropriately validated.^[129]

QRPs do not include more explicit violations of scientific integrity, such as data falsification.^[125]^[126] Fraudulent research does occur, as in the case of scientific fraud by social psychologist Diederik Stapel,^[130]^[14] cognitive psychologist Marc Hauser and social psychologist Lawrence Sanna,^[14] but it appears to be uncommon.^[14]

Prevalence

According to IU professor Ernest O’Boyle and psychologist Martin Götz, around 50% of researchers surveyed across various studies admitted engaging in HARKing.^[131] In a survey of 2,000 psychologists by behavioral scientist Leslie K. John and colleagues, around 94% of psychologists admitted having employed at least one QRP. More specifically, 63% admitted failing to report all of a study's dependent measures, 28% to report all of a study's conditions, and 46% to selectively reporting studies that produced the desired pattern of results. In addition, 56% admitted having collected more data after having inspected already collected data, and 16% to having stopped data collection because the desired result was already visible.^[40] According to biotechnology researcher J. Leslie Glick's estimate in 1992, 10% to 20% of research and development studies involved either QRPs or outright fraud.^[132] The methodology used to estimate QRPs has been contested, and more recent studies suggested lower prevalence rates on average.^[133]

A 2009 meta-analysis found that 2% of scientists across fields admitted falsifying studies at least once and 14% admitted knowing someone who did. Such misconduct was, according to one study, reported more frequently by medical researchers than by others.^[134]

Statistical issues

Low statistical power

According to Deakin University professor Tom Stanley and colleagues, one plausible reason studies fail to replicate is low statistical power. This happens for three reasons. First, a replication study with low power is unlikely to succeed since, by definition, it has a low probability to detect a true effect. Second, if the original study has low power, it will yield biased effect size estimates. When conducting a priori power analysis for the replication study, this will result in underestimation of the required sample size. Third, if the original study has low power, the post-study odds of a statistically significant finding reflecting a true effect are quite low. It is therefore likely that a replication attempt of the original study would fail.^[15]

Mathematically, the probability of replicating a previous publication that rejected a null hypothesis $H_{0}$ in favor of an alternative $H_{1}$ is $({\text{significance}})Pr(H_{0}|{\text{publication}})+({\text{power}})Pr(H_{1}|{\text{publication}})\leq ({\text{power}})$ assuming significance is less than power. Thus, low power implies low probability of replication, regardless of how the previous publication was designed, and regardless of which hypothesis is really true.^[78]

Stanley and colleagues estimated the average statistical power of psychological literature by analyzing data from 200 meta-analyses. They found that on average, psychology studies have between 33.1% and 36.4% statistical power. These values are quite low compared to the 80% considered adequate statistical power for an experiment. Across the 200 meta-analyses, the median of studies with adequate statistical power was between 7.7% and 9.1%, implying that a positive result would replicate with probability less than 10%, regardless of whether the positive result was a true positive or a false positive.^[15]

The statistical power of neuroscience studies is quite low. The estimated statistical power of fMRI research is between .08 and .31,^[135] and that of studies of event-related potentials was estimated as .72‒.98 for large effect sizes, .35‒.73 for medium effects, and .10‒.18 for small effects.^[123]

In a study published in Nature, psychologist Katherine Button and colleagues conducted a similar study with 49 meta-analyses in neuroscience, estimating a median statistical power of 21%.^[136] Meta-scientist John Ioannidis and colleagues computed an estimate of average power for empirical economic research, finding a median power of 18% based on literature drawing upon 6.700 studies.^[137] In light of these results, it is plausible that a major reason for widespread failures to replicate in several scientific fields might be very low statistical power on average.

The same statistical test with the same significance level will have lower statistical power if the effect size is small under the alternative hypothesis. Complex inheritable traits are typically correlated with a large number of genes, each of small effect size, so high power requires a large sample size. In particular, many results from the candidate gene literature suffered from small effect sizes and small sample sizes and would not replicate. More data from genome-wide association studies (GWAS) come close to solving this problem.^[138]^[139] As a numeric example, most genes associated with schizophrenia risk have low effect size (genotypic relative risk, GRR). A statistical study with 1000 cases and 1000 controls has 0.03% power for a gene with GRR = 1.15, which is already large for schizophrenia. In contrast, the largest GWAS to date has ~100% power for it.^[140]

Positive effect size bias

Even when the study replicates, the replication typically have smaller effect size. Underpowered studies have a large effect size bias.^[141]

Distribution of statistically significant estimates of the regression factor in a linear model in the presence of added error. When the sample size is small, adding noise overestimates the regression factor about 50% of the times. When the sample size is small, it consistently underestimates. Figure appeared in^[142].

In studies that statistically estimate a regression factor, such as the $k$ in $Y=kX+b$ , when the dataset is large, noise tends to cause the regression factor to be underestimated, but when the dataset is small, noise tends to cause the regression factor to be overestimated.^[142]

Problems of meta-analysis

Meta-analyses have their own methodological problems and disputes, which leads to rejection of the meta-analytic method by researchers whose theory is challenged by meta-analysis.^[116]

Rosenthal proposed the "fail-safe number" (FSN)^[54] to avoid the publication bias against null results. It is defined as follows: Suppose the null hypothesis is true; how many publications would be required to make the current result indistinguishable from the null hypothesis?

Rosenthal's point is that certain effect sizes are large enough, such that even if there is a total publication bias against null results (the "file drawer problem"), the number of unpublished null results would be impossibly large to swamp out the effect size. Thus, the effect size must be statistically significant even after accounting for unpublished null results.

One objection to the FSN is that it is calculated as if unpublished results are unbiased samples from the null hypothesis. But if the file drawer problem is true, then unpublished results would have effect sizes concentrated around 0. Thus fewer unpublished null results would be necessary to swap out the effect size, and so the FSN is an overestimate.^[116]

Another problem with meta-analysis is that bad studies are "infectious" in the sense that one bad study might cause the entire meta-analysis to overestimate statistical significance.^[78]

P-hacking

Various statistical methods can be applied to make the p-value appear smaller than it really is. This need not be malicious, as moderately flexible data analysis, routine in research, can increase the false-positive rate to above 60%.^[41]

For example, if one collects some data, applies several different significance tests to it, and publishes only the one that happens to have a p-value less than 0.05, then the total p-value for "at least one significance test reaches p < 0.05" can be much larger than 0.05, because even if the null hypothesis were true, the probability that one out of many significance tests is extreme is not itself extreme.

Typically, a statistical study has multiple steps, with several choices at each step, such as during data collection, outlier rejection, choice of test statistic, choice of one-tailed or two-tailed test, etc. These choices in the "garden of forking paths" multiply, creating many "researcher degrees of freedom". The effect is similar to the file-drawer problem, as the paths not taken are not published.^[143]

Consider a simple illustration. Suppose the null hypothesis is true, and we have 20 possible significance tests to apply to the dataset. Also suppose the outcomes to the significance tests are independent. By definition of "significance", each test has probability 0.05 to pass with significance level 0.05. The probability that at least 1 out of 20 is significant is, by assumption of independence, $1-(1-0.05)^{20}=0.64$ .^[144]

Another possibility is the multiple comparisons problem. In 2009, it was twice noted that fMRI studies had a suspicious number of positive results with large effect sizes, more than would be expected since the studies have low power (one example^[145] had only 13 subjects). It pointed out that over half of the studies would test for correlation between a phenomenon and individual fMRI voxels, and only report on voxels exceeding chosen thresholds.^[146]

The figure shows the change in p-values computed from a t-test as the sample size increases, and how early stopping can allow for p-hacking even when the null hypothesis is exactly true. Data is drawn from two identical normal distributions,

N(0,10)

. For each sample size

n

, ranging from 5 to

10^{4}

, a t-test is performed on the first

n

samples from each distribution, and the resulting p-value is plotted. The red dashed line indicates the commonly used significance level of 0.05. If the data collection or analysis were to stop at a point where the p-value happened to fall below the significance level, a spurious statistically significant difference could be reported.

Optional stopping is a practice where one collects data until some stopping criterion is reached. Though a valid procedure, it is easily misused. The problem is that p-value of an optionally stopped statistical test is larger than it seems. Intuitively, this is because the p-value is supposed to be the sum of all events at least as rare as what is observed. With optional stopping, there are even rarer events that are difficult to account for, i.e. not triggering the optional stopping rule, and collecting even more data before stopping. Neglecting these events leads to a p-value that is too low. In fact, if the null hypothesis is true, any significance level can be reached if one is allowed to keep collecting data and stop when the desired p-value (calculated as if one has always been planning to collect exactly this much data) is obtained.^[147] For a concrete example of testing for a fair coin, see p-value#optional stopping.

More succinctly, the proper calculation of p-value requires accounting for counterfactuals, that is, what the experimenter could have done in reaction to data that might have been. Accounting for what might have been is hard even for honest researchers.^[147] One benefit of preregistration is to account for all counterfactuals, allowing the p-value to be calculated correctly.^[148]

The problem of early stopping is not just limited to researcher misconduct. There is often pressure to stop early if the cost of collecting data is high. Some animal ethics boards even mandate early stopping if the study obtains a significant result midway.^[144]

Such practices are widespread in psychology. In a 2012 survey, 56% of psychologists admitted to early stopping, 46% to only reporting analyses that "worked", and 38% to post hoc exclusion, that is, removing some data after analysis was already performed on the data before reanalyzing the remaining data (often on the premise of "outlier removal").^[40]

Statistical heterogeneity

As also reported by Stanley and colleagues, a further reason studies might fail to replicate is high heterogeneity of the to-be-replicated effects. In meta-analysis, "heterogeneity" refers to the variance in research findings that results from there being no single true effect size. Instead, findings in such cases are better seen as a distribution of true effects.^[15] Statistical heterogeneity is calculated using the I-squared statistic,^[149] defined as "the proportion (or percentage) of observed variation among reported effect sizes that cannot be explained by the calculated standard errors associated with these reported effect sizes".^[15] This variation can be due to differences in experimental methods, populations, cohorts, and statistical methods between replication studies. Heterogeneity poses a challenge to studies attempting to replicate previously found effect sizes. When heterogeneity is high, subsequent replications have a high probability of finding an effect size radically different than that of the original study.^[g]

Importantly, significant levels of heterogeneity are also found in direct/exact replications of a study. Stanley and colleagues discuss this while reporting a study by quantitative behavioral scientist Richard Klein and colleagues, where the authors attempted to replicate 15 psychological effects across 36 different sites in Europe and the U.S. In the study, Klein and colleagues found significant amounts of heterogeneity in 8 out of 16 effects (I-squared = 23% to 91%). Importantly, while the replication sites intentionally differed on a variety of characteristics, such differences could account for very little heterogeneity . According to Stanley and colleagues, this suggested that heterogeneity could have been a genuine characteristic of the phenomena being investigated. For instance, phenomena might be influenced by so-called "hidden moderators" – relevant factors that were previously not understood to be important in the production of a certain effect.

In their analysis of 200 meta-analyses of psychological effects, Stanley and colleagues found a median percent of heterogeneity of I-squared = 74%. According to the authors, this level of heterogeneity can be considered "huge". It is three times larger than the random sampling variance of effect sizes measured in their study. If considered along sampling error, heterogeneity yields a standard deviation from one study to the next even larger than the median effect size of the 200 meta-analyses they investigated.^[h] The authors conclude that if replication is defined by a subsequent study finding a sufficiently similar effect size to the original, replication success is not likely even if replications have very large sample sizes. Importantly, this occurs even if replications are direct or exact since heterogeneity nonetheless remains relatively high in these cases.

Others

Within economics, the replication crisis may be also exacerbated because econometric results are fragile:^[150] using different but plausible estimation procedures or data preprocessing techniques can lead to conflicting results.^[151]^[152]^[153]

Context sensitivity

New York University professor Jay Van Bavel and colleagues argue that a further reason findings are difficult to replicate is the sensitivity to context of certain psychological effects. On this view, failures to replicate might be explained by contextual differences between the original experiment and the replication, often called "hidden moderators".^[154] Van Bavel and colleagues tested the influence of context sensitivity by reanalyzing the data of the widely cited Reproducibility Project carried out by the Open Science Collaboration.^[12] They re-coded effects according to their sensitivity to contextual factors and then tested the relationship between context sensitivity and replication success in various regression models.

Context sensitivity was found to negatively correlate with replication success, such that higher ratings of context sensitivity were associated with lower probabilities of replicating an effect.^[i] Importantly, context sensitivity significantly correlated with replication success even when adjusting for other factors considered important for reproducing results (e.g., effect size and sample size of original, statistical power of the replication, methodological similarity between original and replication).^[j] In light of the results, the authors concluded that attempting a replication in a different time, place or with a different sample can significantly alter an experiment's results. Context sensitivity thus may be a reason certain effects fail to replicate in psychology.^[154]

Bayesian explanation

In the framework of Bayesian probability, by Bayes' theorem, rejecting the null hypothesis at significance level 5% does not mean that the posterior probability for the alternative hypothesis is 95%, and the posterior probability is also different from the probability of replication.^[155]^[156] Consider a simplified case where there are only two hypotheses. Let the prior probability of the null hypothesis be $Pr(H_{0})$ , and the alternative $Pr(H_{1})=1-Pr(H_{0})$ . For a given statistical study, let its false positive rate (significance level) be $Pr({\text{find }}H_{1}|H_{0})$ , and true positive rate (power) be $Pr({\text{find }}H_{1}|H_{1})$ . For illustrative purposes, let significance level be 0.05 and power be 0.45 (underpowered).

Now, by Bayes' theorem, conditional on the statistical studying finding $H_{1}$ to be true, the posterior probability of $H_{1}$ actually being true is not $1-Pr({\text{find }}H_{1}|H_{0})=0.95$ , but

$Pr(H_{1}|{\text{ find }}H_{1})={\frac {Pr({\text{ find }}H_{1}|H_{1})Pr(H_{1})}{Pr({\text{ find }}H_{1}|H_{0})Pr(H_{0})+Pr({\text{ find }}H_{1}|H_{1})Pr(H_{1})}}$

and the probability of replicating the statistical study is $Pr({\text{replication}}|{\text{ find }}H_{1})=Pr({\text{find }}H_{1}|H_{1})Pr(H_{1}|{\text{ find }}H_{1})+Pr({\text{find }}H_{1}|H_{0})Pr(H_{0}|{\text{ find }}H_{1})$ which is also different from $Pr(H_{1}|{\text{ find }}H_{1})$ . In particular, for a fixed level of significance, the probability of replication increases with power, and prior probability for $H_{1}$ . If the prior probability for $H_{1}$ is small, then one would require a high power for replication.

For example, if the prior probability of the null hypothesis is $Pr(H_{0})=0.9$ , and the study found a positive result, then the posterior probability for $H_{1}$ is $Pr(H_{1}|{\text{ find }}H_{1})=0.50$ , and the replication probability is $Pr({\text{replication}}|{\text{ find }}H_{1})=0.25$ .

Problem with null hypothesis testing

Some argue that null hypothesis testing is itself inappropriate, especially in "soft sciences" like social psychology.^[157]^[158]

As repeatedly observed by statisticians,^[159] in complex systems, such as social psychology, "the null hypothesis is always false", or "everything is correlated". If so, then if the null hypothesis is not rejected, that does not show that the null hypothesis is true, but merely that it was a false negative, typically due to low power.^[160] Low power is especially prevalent in subject areas where effect sizes are small and data is expensive to acquire, such as social psychology.^[157]^[161]

Furthermore, when the null hypothesis is rejected, it might not be evidence for the substantial alternative hypothesis. In soft sciences, many hypotheses can predict a correlation between two variables. Thus, evidence against the null hypothesis "there is no correlation" is no evidence for one of the many alternative hypotheses that equally well predict "there is a correlation". Fisher developed the NHST for agronomy, where rejecting the null hypothesis is usually good proof of the alternative hypothesis, since there are not many of them. Rejecting the hypothesis "fertilizer does not help" is evidence for "fertilizer helps". But in psychology, there are many alternative hypotheses for every null hypothesis.^[161]^[162]

In particular, when statistical studies on extrasensory perception reject the null hypothesis at extremely low p-value (as in the case of Daryl Bem), it does not imply the alternative hypothesis "ESP exists". Far more likely is that there was a small (non-ESP) signal in the experiment setup that has been measured precisely.^[163]

Paul Meehl noted that statistical hypothesis testing is used differently in "soft" psychology (personality, social, etc.) from physics. In physics, a theory makes a quantitative prediction and is tested by checking whether the prediction falls within the statistically measured interval. In soft psychology, a theory makes a directional prediction and is tested by checking whether the null hypothesis is rejected in the right direction. Consequently, improved experimental technique makes theories more likely to be falsified in physics but less likely to be falsified in soft psychology, as the null hypothesis is always false since any two variables are correlated by a "crud factor" of about 0.30. The net effect is an accumulation of theories that remain unfalsified, but with no empirical evidence for preferring one over the others.^[23]^[162]

Base rate fallacy

According to philosopher Alexander Bird, a possible reason for the low rates of replicability in certain scientific fields is that a majority of tested hypotheses are false a priori.^[164] On this view, low rates of replicability could be consistent with quality science. Relatedly, the expectation that most findings should replicate would be misguided and, according to Bird, a form of base rate fallacy. Bird's argument works as follows. Assuming an ideal situation of a test of significance, whereby the probability of incorrectly rejecting the null hypothesis is 5% (i.e. Type I error) and the probability of correctly rejecting the null hypothesis is 80% (i.e. Power), in a context where a high proportion of tested hypotheses are false, it is conceivable that the number of false positives would be high compared to those of true positives.^[164] For example, in a situation where only 10% of tested hypotheses are actually true, one can calculate that as many as 36% of results will be false positives.^[k]

The claim that the falsity of most tested hypotheses can explain low rates of replicability is even more relevant when considering that the average power for statistical tests in certain fields might be much lower than 80%. For example, the proportion of false positives increases to a value between 55.2% and 57.6% when calculated with the estimates of an average power between 34.1% and 36.4% for psychology studies, as provided by Stanley and colleagues in their analysis of 200 meta-analyses in the field.^[15] A high proportion of false positives would then result in many research findings being non-replicable.

Bird notes that the claim that a majority of tested hypotheses are false a priori in certain scientific fields might be plausible given factors such as the complexity of the phenomena under investigation, the fact that theories are seldom undisputed, the "inferential distance" between theories and hypotheses, and the ease with which hypotheses can be generated. In this respect, the fields Bird takes as examples are clinical medicine, genetic and molecular epidemiology, and social psychology. This situation is radically different in fields where theories have outstanding empirical basis and hypotheses can be easily derived from theories (e.g., experimental physics).^[164]

Consequences

When effects are wrongly stated as relevant in the literature, failure to detect this by replication will lead to the canonization of such false facts.^[165]

A 2021 study found that papers in leading general interest, psychology and economics journals with findings that could not be replicated tend to be cited more over time than reproducible research papers, likely because these results are surprising or interesting. The trend is not affected by publication of failed reproductions, after which only 12% of papers that cite the original research will mention the failed replication.^[166]^[167] Further, experts are able to predict which studies will be replicable, leading the authors of the 2021 study, Marta Serra-Garcia and Uri Gneezy, to conclude that experts apply lower standards to interesting results when deciding whether to publish them.^[167]

Public awareness and perceptions

Concerns have been expressed within the scientific community that the general public may consider science less credible due to failed replications.^[168] Research supporting this concern is sparse, but a nationally representative survey in Germany showed that more than 75% of Germans have not heard of replication failures in science.^[169] The study also found that most Germans have positive perceptions of replication efforts: only 18% think that non-replicability shows that science cannot be trusted, while 65% think that replication research shows that science applies quality control, and 80% agree that errors and corrections are part of science.^[169]

Response in academia

With the replication crisis of psychology earning attention, Princeton University psychologist Susan Fiske drew controversy for speaking against critics of psychology for what she called bullying and undermining the science.^[170]^[171]^[172]^[173] She called these unidentified "adversaries" names such as "methodological terrorist" and "self-appointed data police", saying that criticism of psychology should be expressed only in private or by contacting the journals.^[170] Columbia University statistician and political scientist Andrew Gelman responded to Fiske, saying that she had found herself willing to tolerate the "dead paradigm" of faulty statistics and had refused to retract publications even when errors were pointed out.^[170] He added that her tenure as editor had been abysmal and that a number of published papers she edited were found to be based on extremely weak statistics; one of Fiske's own published papers had a major statistical error and "impossible" conclusions.^[170]

Credibility revolution

Some researchers in psychology indicate that the replication crisis is a foundation for a "credibility revolution", where changes in standards by which psychological science are evaluated may include emphasizing transparency and openness, preregistering research projects, and replicating research with higher standards for evidence to improve the strength of scientific claims.^[174] Such changes may diminish the productivity of individual researchers, but this effect could be avoided by data sharing and greater collaboration.^[174] A credibility revolution could be good for the research environment.^[175]

Remedies

Focus on the replication crisis has led to renewed efforts in psychology to retest important findings.^[41]^[176] A 2013 special edition of the journal Social Psychology focused on replication studies.^[13]

Standardization as well as (requiring) transparency of the used statistical and experimental methods have been proposed.^[177] Careful documentation of the experimental set-up is considered crucial for replicability of experiments and various variables may not be documented and standardized such as animals' diets in animal studies.^[178]

A 2016 article by John Ioannidis elaborated on "Why Most Clinical Research Is Not Useful".^[179] Ioannidis describes what he views as some of the problems and calls for reform, characterizing certain points for medical research to be useful again; one example he makes is the need for medicine to be patient-centered (e.g. in the form of the Patient-Centered Outcomes Research Institute) instead of the current practice to mainly take care of "the needs of physicians, investigators, or sponsors".

Reform in scientific publishing

Metascience

Metascience is the use of scientific methodology to study science itself. It seeks to increase the quality of scientific research while reducing waste. It is also known as "research on research" and "the science of science", as it uses research methods to study how research is done and where improvements can be made. Metascience is concerned with all fields of research and has been called "a bird's eye view of science."^[180] In Ioannidis's words, "Science is the best thing that has happened to human beings ... but we can do it better."^[181]

Meta-research continues to be conducted to identify the roots of the crisis and to address them. Methods of addressing the crisis include pre-registration of scientific studies and clinical trials as well as the founding of organizations such as CONSORT and the EQUATOR Network that issue guidelines for methodology and reporting. Efforts continue to reform the system of academic incentives, improve the peer review process, reduce the misuse of statistics, combat bias in scientific literature, and increase the overall quality and efficiency of the scientific process.

Presentation of methodology

Some authors have argued that the insufficient communication of experimental methods is a major contributor to the reproducibility crisis and that better reporting of experimental design and statistical analyses would improve the situation. These authors tend to plead for both a broad cultural change in the scientific community of how statistics are considered and a more coercive push from scientific journals and funding bodies.^[182] But concerns have been raised about the potential for standards for transparency and replication to be misapplied to qualitative as well as quantitative studies.^[183]

Business and management journals that have introduced editorial policies on data accessibility, replication, and transparency include the Strategic Management Journal, the Journal of International Business Studies, and the Management and Organization Review.^[92]

Result-blind peer review

In response to concerns in psychology about publication bias and data dredging, more than 140 psychology journals have adopted result-blind peer review. In this approach, studies are accepted not on the basis of their findings and after the studies are completed, but before they are conducted and on the basis of the methodological rigor of their experimental designs, and the theoretical justifications for their statistical analysis techniques before data collection or analysis is done.^[184] Early analysis of this procedure has estimated that 61% of result-blind studies have led to null results, in contrast to an estimated 5% to 20% in earlier research.^[100] In addition, large-scale collaborations between researchers working in multiple labs in different countries that regularly make their data openly available for different researchers to assess have become much more common in psychology.^[185]

Pre-registration of studies

Scientific publishing has begun using pre-registration reports to address the replication crisis.^[186]^[187] The registered report format requires authors to submit a description of the study methods and analyses prior to data collection. Once the method and analysis plan is vetted through peer-review, publication of the findings is provisionally guaranteed, based on whether the authors follow the proposed protocol. One goal of registered reports is to circumvent the publication bias toward significant findings that can lead to implementation of questionable research practices. Another is to encourage publication of studies with rigorous methods.

The journal Psychological Science has encouraged the preregistration of studies and the reporting of effect sizes and confidence intervals.^[188] The editor in chief also noted that the editorial staff will be asking for replication of studies with surprising findings from examinations using small sample sizes before allowing the manuscripts to be published.

Metadata and digital tools for tracking replications

It has been suggested that "a simple way to check how often studies have been repeated, and whether or not the original findings are confirmed" is needed.^[166] Categorizations and ratings of reproducibility at the study or results level, as well as addition of links to and rating of third-party confirmations, could be conducted by the peer-reviewers, the scientific journal, or by readers in combination with novel digital platforms or tools.

Statistical reform

Requiring smaller p-values

Many publications require a p-value of p < 0.05 to claim statistical significance. The paper "Redefine statistical significance",^[189] signed by a large number of scientists and mathematicians, proposes that in "fields where the threshold for defining statistical significance for new discoveries is p < 0.05, we propose a change to p < 0.005. This simple step would immediately improve the reproducibility of scientific research in many fields." Their rationale is that "a leading cause of non-reproducibility (is that the) statistical standards of evidence for claiming new discoveries in many fields of science are simply too low. Associating 'statistically significant' findings with p < 0.05 results in a high rate of false positives even in the absence of other experimental, procedural and reporting problems."^[189]

This call was subsequently criticised by another large group, who argued that "redefining" the threshold would not fix current problems, would lead to some new ones, and that in the end, all thresholds needed to be justified case-by-case instead of following general conventions.^[190]

Addressing misinterpretation of p-values

Although statisticians are unanimous that the use of "p < 0.05" as a standard for significance provides weaker evidence than is generally appreciated, there is a lack of unanimity about what should be done about it. Some have advocated that Bayesian methods should replace p-values. This has not happened on a wide scale, partly because it is complicated and partly because many users distrust the specification of prior distributions in the absence of hard data. A simplified version of the Bayesian argument, based on testing a point null hypothesis was suggested by pharmacologist David Colquhoun.^[191]^[192] The logical problems of inductive inference were discussed in "The Problem with p-values" (2016).^[193]

The hazards of reliance on p-values arises partly because even an observation of p = 0.001 is not necessarily strong evidence against the null hypothesis.^[192] Despite the fact that the likelihood ratio in favor of the alternative hypothesis over the null is close to 100, if the hypothesis was implausible, with a prior probability of a real effect being 0.1, even the observation of p = 0.001 would have a false positive risk of 8 percent. It would still fail to reach the 5 percent level.

It was recommended that the terms "significant" and "non-significant" should not be used.^[192] p-values and confidence intervals should still be specified, but they should be accompanied by an indication of the false-positive risk. It was suggested that the best way to do this is to calculate the prior probability that would be necessary to believe in order to achieve a false positive risk of a certain level, such as 5%. The calculations can be done with various computer software.^[192]^[194] This reverse Bayesian approach, which physicist Robert Matthews suggested in 2001,^[195] is one way to avoid the problem that the prior probability is rarely known.

Encouraging larger sample sizes

To improve the quality of replications, larger sample sizes than those used in the original study are often needed.^[196] Larger sample sizes are needed because estimates of effect sizes in published work are often exaggerated due to publication bias and large sampling variability associated with small sample sizes in an original study.^[197]^[198]^[199] Further, using significance thresholds usually leads to inflated effects, because particularly with small sample sizes, only the largest effects will become significant.^[158]

Cross-validation

One common statistical problem is overfitting, that is, when researchers fit a regression model over a large number of variables but a small number of data points. For example, a typical fMRI study of emotion, personality, and social cognition has fewer than 100 subjects, but each subject has 10,000 voxels. The study would fit a sparse linear regression model that uses the voxels to predict a variable of interest, such as self-reported stress. But the study would then report on the p-value of the model on the same data it was fitted to. The standard approach in statistics, where data is split into a training and a validation set, is resisted because test subjects are expensive to acquire.^[146]^[200]

One possible solution is cross-validation, which allows model validation while also allowing the whole dataset to be used for model-fitting.^[201]

Replication efforts

Funding

In July 2016, the Netherlands Organisation for Scientific Research made €3 million available for replication studies. The funding is for replication based on reanalysis of existing data and replication by collecting and analysing new data. Funding is available in the areas of social sciences, health research and healthcare innovation.^[202]

In 2013, the Laura and John Arnold Foundation funded the launch of The Center for Open Science with a $5.25 million grant. By 2017, it provided an additional $10 million in funding.^[203] It also funded the launch of the Meta-Research Innovation Center at Stanford at Stanford University run by Ioannidis and medical scientist Steven Goodman to study ways to improve scientific research.^[203] It also provided funding for the AllTrials initiative led in part by medical scientist Ben Goldacre.^[203]

Emphasis in post-secondary education

Based on coursework in experimental methods at MIT, Stanford, and the University of Washington, it has been suggested that methods courses in psychology and other fields should emphasize replication attempts rather than original studies.^[204]^[205]^[206] Such an approach would help students learn scientific methodology and provide numerous independent replications of meaningful scientific findings that would test the replicability of scientific findings. Some have recommended that graduate students should be required to publish a high-quality replication attempt on a topic related to their doctoral research prior to graduation.^[207]

Replication database

There has been a concern that replication attempts have been growing.^[208]^[209]^[210] As a result, this may lead to lead to research waste.^[211] In turn, this has led to a need to systematically track replication attempts. As a result, several databases have been created (e.g. ^[212] ^[213]). The databases have created a Replication Database that includes psychology and speech-language therapy among other disciplines to promote theory-driven research and optimize the use of academic and institutional resource, while promoting trust in science.^[214]

Final year thesis

Some institutions require undergraduate students to submit a final year thesis that consists of an original piece of research. Daniel Quintana, a psychologist at the University of Oslo in Norway, has recommended that students should be encouraged to perform replication studies in thesis projects, as well as being taught about open science.^[215]

Semi-automated

"The overall process of testing the reproducibility and robustness of the cancer biology literature by robot. First, text mining is used to extract statements about the effect of drugs on gene expression in breast cancer. Then two different teams semi-automatically tested these statements using two different protocols, and two different cell lines (MCF7 and MDA-MB-231) using the laboratory automation system Eve."

Researchers demonstrated a way of semi-automated testing for reproducibility: statements about experimental results were extracted from, as of 2022 non-semantic, gene expression cancer research papers and subsequently reproduced via robot scientist "Eve".^[216]^[217] Problems of this approach include that it may not be feasible for many areas of research and that sufficient experimental data may not get extracted from some or many papers even if available.

Involving original authors

Psychologist Daniel Kahneman argued that, in psychology, the original authors should be involved in the replication effort because the published methods are often too vague.^[218]^[219] Others, such as psychologist Andrew Wilson, disagree, arguing that the original authors should write down the methods in detail.^[218] An investigation of replication rates in psychology in 2012 indicated higher success rates of replication in replication studies when there was author overlap with the original authors of a study^[220] (91.7% successful replication rates in studies with author overlap compared to 64.6% successful replication rates without author overlap).

Big team science

The replication crisis has led to the formation and development of various large-scale and collaborative communities to pool their resources to address a single question across cultures, countries and disciplines.^[221] The focus is on replication, to ensure that the effect generalizes beyond a specific culture and investigate whether the effect is replicable and genuine.^[222] This allows interdisciplinary internal reviews, multiple perspectives, uniform protocols across labs, and recruiting larger and more diverse samples.^[222] Researchers can collaborate by coordinating data collection or fund data collection by researchers who may not have access to the funds, allowing larger sample sizes and increasing the robustness of the conclusions.

Broader changes to scientific approach

Emphasize triangulation, not just replication

Psychologist Marcus R. Munafò and Epidemiologist George Davey Smith argue, in a piece published by Nature, that research should emphasize triangulation, not just replication, to protect against flawed ideas. They claim that,

replication alone will get us only so far (and) might actually make matters worse ... [Triangulation] is the strategic use of multiple approaches to address one question. Each approach has its own unrelated assumptions, strengths and weaknesses. Results that agree across different methodologies are less likely to be artefacts. ... Maybe one reason replication has captured so much interest is the often-repeated idea that falsification is at the heart of the scientific enterprise. This idea was popularized by Karl Popper's 1950s maxim that theories can never be proved, only falsified. Yet an overemphasis on repeating experiments could provide an unfounded sense of certainty about findings that rely on a single approach. ... philosophers of science have moved on since Popper. Better descriptions of how scientists actually work include what epistemologist Peter Lipton called in 1991 "inference to the best explanation".^[223]

Complex systems paradigm

The dominant scientific and statistical model of causation is the linear model.^[224] The linear model assumes that mental variables are stable properties which are independent of each other. In other words, these variables are not expected to influence each other. Instead, the model assumes that the variables will have an independent, linear effect on observable outcomes.^[224]

Social scientists Sebastian Wallot and Damian Kelty-Stephen argue that the linear model is not always appropriate.^[224] An alternative is the complex system model which assumes that mental variables are interdependent. These variables are not assumed to be stable, rather they will interact and adapt to each specific context.^[224] They argue that the complex system model is often more appropriate in psychology, and that the use of the linear model when the complex system model is more appropriate will result in failed replications.^[224]

...psychology may be hoping for replications in the very measurements and under the very conditions where a growing body of psychological evidence explicitly discourages predicting replication. Failures to replicate may be plainly baked into the potentially incomplete, but broadly sweeping failure of human behavior to conform to the standard of independen[ce] ...^[224]

Replication should seek to revise theories

Replication is fundamental for scientific progress to confirm original findings. However, replication alone is not sufficient to resolve the replication crisis. Replication efforts should seek not just to support or question the original findings, but also to replace them with revised, stronger theories with greater explanatory power. This approach therefore involves pruning existing theories, comparing all the alternative theories, and making replication efforts more generative and engaged in theory-building.^[225]^[226] However, replication alone is not enough, it is important to assess the extent that results generalise across geographical, historical and social contexts is important for several scientific fields, especially practitioners and policy makers to make analyses in order to guide important strategic decisions. Reproducible and replicable findings was the best predictor of generalisability beyond historical and geographical contexts, indicating that for social sciences, results from a certain time period and place can meaningfully drive as to what is universally present in individuals.^[227]

Open science

Tenets of open science

Open data, open source software and open source hardware all are critical to enabling reproducibility in the sense of validation of the original data analysis. The use of proprietary software, the lack of the publication of analysis software and the lack of open data prevents the replication of studies. Unless software used in research is open source, reproducing results with different software and hardware configurations is impossible.^[228] CERN has both Open Data and CERN Analysis Preservation projects for storing data, all relevant information, and all software and tools needed to preserve an analysis at the large experiments of the LHC. Aside from all software and data, preserved analysis assets include metadata that enable understanding of the analysis workflow, related software, systematic uncertainties, statistics procedures and meaningful ways to search for the analysis, as well as references to publications and to backup material.^[229] CERN software is open source and available for use outside of particle physics and there is some guidance provided to other fields on the broad approaches and strategies used for open science in contemporary particle physics.^[230]

Online repositories where data, protocols, and findings can be stored and evaluated by the public seek to improve the integrity and reproducibility of research. Examples of such repositories include the Open Science Framework, Registry of Research Data Repositories, and Psychfiledrawer.org. Sites like Open Science Framework offer badges for using open science practices in an effort to incentivize scientists. However, there have been concerns that those who are most likely to provide their data and code for analyses are the researchers that are likely the most sophisticated.^[231] Ioannidis suggested that "the paradox may arise that the most meticulous and sophisticated and method-savvy and careful researchers may become more susceptible to criticism and reputation attacks by reanalyzers who hunt for errors, no matter how negligible these errors are".^[231]

Notes

^ Also called the replicability crisis, reproducibility crisis, reproduction crisis, or irreproducibility crisis.
^ According to the APA Dictionary of Psychology, confirmation bias is "the tendency to gather evidence that confirms preexisting expectations, typically by emphasizing or pursuing supporting evidence while dismissing or failing to seek contradictory evidence".^[115]
^ In the context of null-hypothesis significance testing, results that are not statistically significant
^ Data dredging, also known as p-hacking or p-fishing, is misuse of data, through myriad techniques, to find support for hypotheses that the data is inadequate for.^[128]
^ Selective reporting is also known as partial publication. Reporting is an opportunity to disclose all of the researcher degrees of freedom used or exploited. Selective reporting is a failure to report relevant details or choices, such as some independent and dependent variables, missing data, data exclusions, and outlier exclusions.^[126]
^ HARKing, also known as post-hoc storytelling, is when an exploratory analysis is framed as a confirmatory analysis. It involves changing a hypothesis after research has been done, so that the new hypothesis is able to be confirmed by the results of the experiment.^[126]
^ The authors make an example whereby assuming that the true mean correlation reflecting an effect is 0.2 and the standard deviation of the distribution of effects is also 0.2, a replication study will have a 62% probability of finding either a medium-to-large true effect (r > 0.3) or a negligible true effect (r < 0.1).
^ 0.412 against 0.389 in units of standardized mean differences (SMD).
^ The main DV used was the subjective binary rating (i.e replicated/ not replicated) used in the original study by OSC. The authors also measured correlations with other measures of reproducibility (e.g. Confidence intervals) and found nearly-equal correlations between context-sensitivity and replication success
^ The independent effect of context-sensitivity could be observed both in a multiple logistic regression and in a hierarchical regression model. In the latter case, context-sensitivity was included in step 2 of the hierarchy and change in the coefficient of multiple determination turned out to be significant
^
Following Bird's argument this percentage is obtained by calculating the False Positive Report Probability (FPRP) as follows.
- FPRP = Number of false positives / Number of total positives
- Number of false positives = Probability of obtaining a false positive x Number of negative tests
- Number of true positives = Probability of obtaining a true positive x Number of positive tests
Assuming:
- Number of tests = 1000
- Proportion of true hypotheses p = 0.10
- Probability of obtaining a false positive a = 0.05
- Probability of obtaining a true positive 1 – B = 0.8
Then FPRP = (0.05 x 900)/(0.05 x 900 + 0.8 x 100) = 0.36

References

^ ^a ^b ^c Ioannidis JP (August 2005). "Why most published research findings are false". PLOS Medicine. 2 (8): e124. doi:10.1371/journal.pmed.0020124. PMC 1182327. PMID 16060722.
^ John S (8 December 2017). Scientific Method. New York, NY: Routledge. doi:10.4324/9781315100708. ISBN 978-1-315-10070-8. S2CID 201781341.
^ Lehrer J (13 December 2010). "The Truth Wears Off". The New Yorker. Retrieved 2020-01-30.
^ Marcus G (1 May 2013). "The Crisis in Social Psychology That Isn't". The New Yorker. Retrieved 2020-01-30.
^ ^a ^b ^c Baker M (May 2016). "1,500 scientists lift the lid on reproducibility". Nature (News Feature). 533 (7604). Springer Nature: 452–454. Bibcode:2016Natur.533..452B. doi:10.1038/533452a. PMID 27225100. S2CID 4460617. (Erratum: [1])
^ Pashler H, Harris CR (November 2012). "Is the Replicability Crisis Overblown? Three Arguments Examined". Perspectives on Psychological Science. 7 (6): 531–536. doi:10.1177/1745691612463401. PMID 26168109. S2CID 1342421.
^ Fidler F, Wilcox J (2018). "Reproducibility of Scientific Results". The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University. Retrieved 2019-05-19.
^ Korbmacher M, Azevedo F, Pennington CR, Hartmann H, Pownall M, Schmidt K, et al. (25 July 2023). "The replication crisis has led to positive structural, procedural, and community changes". Communications Psychology. 1 (1): 3. doi:10.1038/s44271-023-00003-2. ISSN 2731-9121. PMC 11290608. PMID 39242883.
^ Moonesinghe R, Khoury MJ, Janssens AC (February 2007). "Most published research findings are false-but a little replication goes a long way". PLOS Medicine. 4 (2): e28. doi:10.1371/journal.pmed.0040028. PMC 1808082. PMID 17326704.
^ Simons DJ (January 2014). "The Value of Direct Replication". Perspectives on Psychological Science. 9 (1): 76–80. doi:10.1177/1745691613514755. PMID 26173243. S2CID 1149441.
^ ^a ^b ^c ^d Schmidt S (2009). "Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences". Review of General Psychology. 13 (2). SAGE Publications: 90–100. doi:10.1037/a0015108. ISSN 1089-2680. S2CID 143855611.
^ ^a ^b ^c Open Science Collaboration (August 2015). "PSYCHOLOGY. Estimating the reproducibility of psychological science" (PDF). Science. 349 (6251): aac4716. doi:10.1126/science.aac4716. hdl:10722/230596. PMID 26315443. S2CID 218065162.
^ ^a ^b ^c Duvendack M, Palmer-Jones R, Reed RW (May 2017). "What Is Meant by "Replication" and Why Does It Encounter Resistance in Economics?". American Economic Review. 107 (5): 46–51. doi:10.1257/aer.p20171031. ISSN 0002-8282.
^ ^a ^b ^c ^d ^e ^f ^g Shrout PE, Rodgers JL (January 2018). "Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis". Annual Review of Psychology. 69 (1). Annual Reviews: 487–510. doi:10.1146/annurev-psych-122216-011845. PMID 29300688. S2CID 19593610.
^ ^a ^b ^c ^d ^e ^f Stanley TD, Carter EC, Doucouliagos H (December 2018). "What meta-analyses reveal about the replicability of psychological research". Psychological Bulletin. 144 (12): 1325–1346. doi:10.1037/bul0000169. PMID 30321017. S2CID 51951232.
^ Meyer C, Chabris C (31 July 2014). "Why Psychologists' Food Fight Matters". Slate.
^ Aschwanden C (19 August 2015). "Science Isn't Broken". FiveThirtyEight. Retrieved 2020-01-30.
^ Aschwanden C (27 August 2015). "Psychology Is Starting To Deal With Its Replication Problem". FiveThirtyEight. Retrieved 2020-01-30.
^ Etchells P (28 May 2014). "Psychology's replication drive: it's not about you". The Guardian.
^ Wagenmakers EJ, Wetzels R, Borsboom D, van der Maas HL, Kievit RA (November 2012). "An Agenda for Purely Confirmatory Research". Perspectives on Psychological Science. 7 (6): 632–638. doi:10.1177/1745691612463078. PMID 26168122. S2CID 5096417.
^ Ioannidis JP (November 2012). "Why Science Is Not Necessarily Self-Correcting". Perspectives on Psychological Science. 7 (6): 645–654. doi:10.1177/1745691612464056. PMID 26168125. S2CID 11798785.
^ Pashler H, Harris CR (November 2012). "Is the Replicability Crisis Overblown? Three Arguments Examined". Perspectives on Psychological Science. 7 (6): 531–536. doi:10.1177/1745691612463401. PMID 26168109. S2CID 1342421.
^ ^a ^b Meehl PE (1967). "Theory-Testing in Psychology and Physics: A Methodological Paradox". Philosophy of Science. 34 (2): 103–115. doi:10.1086/288135. ISSN 0031-8248. JSTOR 186099. S2CID 96422880.
^ Kelley K, Preacher KJ (June 2012). "On effect size". Psychological Methods. 17 (2): 137–152. doi:10.1037/a0028086. PMID 22545595.
^ Simonsohn U, Nelson LD, Simmons JP (November 2014). "p-Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results". Perspectives on Psychological Science. 9 (6): 666–681. doi:10.1177/1745691614553988. PMID 26186117.
^ Simonsohn U, Nelson LD, Simmons JP (April 2014). "P-curve: a key to the file-drawer". Journal of Experimental Psychology. General. 143 (2): 534–547. doi:10.1037/a0033242. PMID 23855496.
^ ^a ^b ^c Romero F (November 2019). "Philosophy of science and the replicability crisis". Philosophy Compass. 14 (11). doi:10.1111/phc3.12633. ISSN 1747-9991. S2CID 202261836.
^ Bargh JA, Chen M, Burrows L (August 1996). "Automaticity of social behavior: direct effects of trait construct and stereotype-activation on action". Journal of Personality and Social Psychology. 71 (2): 230–244. doi:10.1037/0022-3514.71.2.230. PMID 8765481. S2CID 6654763.
^ Doyen S, Klein O, Pichon CL, Cleeremans A (18 January 2012). Lauwereyns J (ed.). "Behavioral priming: it's all in the mind, but whose mind?". PLOS ONE. 7 (1): e29081. Bibcode:2012PLoSO...729081D. doi:10.1371/journal.pone.0029081. PMC 3261136. PMID 22279526.
^ Yong E (10 March 2012). "A failed replication draws a scathing personal attack from a psychology professor". National Geographic. Archived from the original on 2021-02-25. Retrieved 2023-07-04.
^ Pashler H, Coburn N, Harris CR (29 August 2012). "Priming of social distance? Failure to replicate effects on social and food judgments". PLOS ONE. 7 (8): e42510. Bibcode:2012PLoSO...742510P. doi:10.1371/journal.pone.0042510. PMC 3430642. PMID 22952597.
^ Harris CR, Coburn N, Rohrer D, Pashler H (16 August 2013). "Two failures to replicate high-performance-goal priming effects". PLOS ONE. 8 (8): e72467. Bibcode:2013PLoSO...872467H. doi:10.1371/journal.pone.0072467. PMC 3745413. PMID 23977304.
^ Shanks DR, Newell BR, Lee EH, Balakrishnan D, Ekelund L, Cenac Z, et al. (24 April 2013). "Priming intelligent behavior: an elusive phenomenon". PLOS ONE. 8 (4): e56515. Bibcode:2013PLoSO...856515S. doi:10.1371/journal.pone.0056515. PMC 3634790. PMID 23637732.
^ Klein RA, Ratliff KA, Vianello M, Adams RB, Bahník Š, Bernstein MJ, et al. (May 2014). "Investigating Variation in Replicability". Social Psychology. 45 (3): 142–152. doi:10.1027/1864-9335/a000178. hdl:2066/131506. ISSN 1864-9335.
^ Bem DJ (March 2011). "Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect". Journal of Personality and Social Psychology. 100 (3): 407–425. doi:10.1037/a0021524. PMID 21280961. S2CID 1961013.
^ Wagenmakers EJ, Wetzels R, Borsboom D, van der Maas HL (March 2011). "Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011)". Journal of Personality and Social Psychology. 100 (3): 426–432. doi:10.1037/a0022790. PMID 21280965.
^ Galak J, LeBoeuf RA, Nelson LD, Simmons JP (December 2012). "Correcting the past: failures to replicate ψ". Journal of Personality and Social Psychology. 103 (6): 933–948. doi:10.1037/a0029709. PMID 22924750.
^ ^a ^b Begley CG, Ellis LM (March 2012). "Drug development: Raise standards for preclinical cancer research". Nature (Comment article). 483 (7391): 531–533. Bibcode:2012Natur.483..531B. doi:10.1038/483531a. PMID 22460880. S2CID 4326966. (Erratum: doi:10.1038/485041e)
^ Ioannidis JP (September 2008). "Why most discovered true associations are inflated". Epidemiology. 19 (5): 640–648. doi:10.1097/EDE.0b013e31818131e7. PMID 18633328. S2CID 15440816.
^ ^a ^b ^c ^d ^e ^f John LK, Loewenstein G, Prelec D (May 2012). "Measuring the prevalence of questionable research practices with incentives for truth telling". Psychological Science. 23 (5): 524–532. doi:10.1177/0956797611430953. PMID 22508865. S2CID 8400625.
^ ^a ^b ^c ^d ^e Simmons JP, Nelson LD, Simonsohn U (November 2011). "False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant". Psychological Science. 22 (11): 1359–1366. doi:10.1177/0956797611417632. PMID 22006061. S2CID 13802986.
^ Pashler H, Wagenmakers EJ (November 2012). "Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?". Perspectives on Psychological Science. 7 (6): 528–530. doi:10.1177/1745691612465253. PMID 26168108. S2CID 26361121.
^ Ahlgren A (April 1969). "A modest proposal for encouraging replication". American Psychologist. 24 (4): 471. doi:10.1037/h0037798. ISSN 1935-990X.
^ Smith NC (October 1970). "Replication studies: A neglected aspect of psychological research". American Psychologist. 25 (10): 970–975. doi:10.1037/h0029774. ISSN 1935-990X.
^ Neuliep JW, Crandall R (1993). "Reviewer bias against replication research". Journal of Social Behavior and Personality. 8 (6): 21–29. ProQuest 1292304227 – via ProQuest.
^ Neuliep JW, Crandall R (1990). "Editorial bias against replication research". Journal of Social Behavior and Personality. 5 (4): 85–90 – via ProQuest.
^ Lewis-Kraus G (30 September 2023). "They Studied Dishonesty. Was Their Work a Lie?". The New Yorker. ISSN 0028-792X. Retrieved 2023-10-01.
^ Subbaraman N (24 September 2023). "The Band of Debunkers Busting Bad Scientists". Wall Street Journal. Archived from the original on 2023-09-24. Retrieved 2023-10-08.
^ "APA PsycNet". psycnet.apa.org. Retrieved 2023-10-08.
^ ^a ^b Spellman BA (November 2015). "A Short (Personal) Future History of Revolution 2.0". Perspectives on Psychological Science. 10 (6): 886–899. doi:10.1177/1745691615609918. PMID 26581743. S2CID 206778431.
^ ^a ^b Greenwald AG, ed. (January 1976). "An editorial". Journal of Personality and Social Psychology. 33 (1): 1–7. doi:10.1037/h0078635. ISSN 1939-1315.
^ Sterling TD (1959). "Publication Decisions and Their Possible Effects on Inferences Drawn from Tests of Significance--Or Vice Versa". Journal of the American Statistical Association. 54 (285): 30–34. doi:10.2307/2282137. ISSN 0162-1459. JSTOR 2282137.
^ Mills JL (October 1993). "Data torturing". The New England Journal of Medicine. 329 (16): 1196–1199. doi:10.1056/NEJM199310143291613. PMID 8166792.
^ ^a ^b Rosenthal R (May 1979). "The file drawer problem and tolerance for null results". Psychological Bulletin. 86 (3): 638–641. doi:10.1037/0033-2909.86.3.638. ISSN 1939-1455. S2CID 36070395.
^ ^a ^b Cohen J (September 1962). "The statistical power of abnormal-social psychological research: a review". Journal of Abnormal and Social Psychology. 65: 145–153. doi:10.1037/h0045186. PMID 13880271.
^ Sedlmeier P, Gigerenzer G (March 1989). "Do studies of statistical power have an effect on the power of studies?". Psychological Bulletin. 105 (2): 309–316. doi:10.1037/0033-2909.105.2.309. hdl:21.11116/0000-0000-B883-C. ISSN 1939-1455.
^ Gelman A (21 September 2016). "What has happened down here is the winds have changed". Statistical Modeling, Causal Inference, and Social Science.
^ Yong E (3 October 2012). "Nobel laureate challenges psychologists to clean up their act". Nature. doi:10.1038/nature.2012.11535. ISSN 1476-4687.
^ Vankov I, Bowers J, Munafò MR (May 2014). "On the persistence of low power in psychological science". Quarterly Journal of Experimental Psychology. 67 (5): 1037–1040. doi:10.1080/17470218.2014.885986. PMC 4961230. PMID 24528377.
^ ^a ^b Smaldino PE, McElreath R (September 2016). "The natural selection of bad science". Royal Society Open Science. 3 (9): 160384. arXiv:1605.09511. Bibcode:2016RSOS....360384S. doi:10.1098/rsos.160384. PMC 5043322. PMID 27703703.
^ Achenbach J. "No, science's reproducibility problem is not limited to psychology". The Washington Post. Retrieved 2015-09-10.
^ Wiggins BJ, Christopherson C (2019). "The replication crisis in psychology: An overview for theoretical and philosophical psychology". Journal of Theoretical and Philosophical Psychology. 39 (4): 202–217. doi:10.1037/teo0000137. ISSN 2151-3341. S2CID 210567289.
^ Hagger MS, Chatzisarantis NL, Alberts H, Anggono CO, Batailler C, Birt AR, et al. (July 2016). "A Multilab Preregistered Replication of the Ego-Depletion Effect". Perspectives on Psychological Science. 11 (4): 546–573. doi:10.1177/1745691616652873. hdl:20.500.11937/16871. PMID 27474142.
^ Bartlett T (30 January 2013). "Power of Suggestion". The Chronicle of Higher Education.
^ Dominus S (18 October 2017). "When the Revolution Came for Amy Cuddy". The New York Times. ISSN 0362-4331. Retrieved 2017-10-19.
^ Duncan LE, Keller MC (October 2011). "A critical review of the first 10 years of candidate gene-by-environment interaction research in psychiatry". The American Journal of Psychiatry. 168 (10): 1041–1049. doi:10.1176/appi.ajp.2011.11020191. PMC 3222234. PMID 21890791.
^ Leichsenring F, Abbass A, Hilsenroth MJ, Leweke F, Luyten P, Keefe JR, et al. (April 2017). "Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research". Psychological Medicine. 47 (6): 1000–1011. doi:10.1017/S003329171600324X. PMID 27955715. S2CID 1872762.
^ Hengartner MP (28 February 2018). "Raising Awareness for the Replication Crisis in Clinical Psychology by Focusing on Inconsistencies in Psychotherapy Research: How Much Can We Rely on Published Findings from Efficacy Trials?". Frontiers in Psychology. 9. Frontiers Media: 256. doi:10.3389/fpsyg.2018.00256. PMC 5835722. PMID 29541051.
^ Frank MC, Bergelson E, Bergmann C, Cristia A, Floccia C, Gervain J, et al. (9 March 2017). "A Collaborative Approach to Infant Research: Promoting Reproducibility, Best Practices, and Theory-Building". Infancy. 22 (4): 421–435. doi:10.1111/infa.12182. hdl:10026.1/9942. PMC 6879177. PMID 31772509.
^ Harris JR (2009) [1998]. The Nurture Assumption: Why Children Turn Out the Way They Do (2nd ed.). New York: Free Press. ISBN 978-1439101650.
^ Harris HR (2006). No Two Alike: Human Nature and Human Individuality. New York: W. W. Norton & Company. ISBN 978-0393329711.
^ Tyson C (14 August 2014). "Failure to Replicate". Inside Higher Ed. Retrieved 2018-12-19.
^ Makel MC, Plucker JA (1 August 2014). "Facts Are More Important Than Novelty: Replication in the Education Sciences". Educational Researcher. 43 (6): 304–316. doi:10.3102/0013189X14545513. S2CID 145571836. Retrieved 2018-12-19.
^ Kirschner PA, Sweller J, Clark RE (2006). "Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching". Educational Psychologist. 41 (2). Routledge: 75–86. doi:10.1207/s15326985ep4102_1. S2CID 17067829.
^ Foundations for Success: The Final Report of the National Mathematics Advisory Panel (PDF) (Report). United States Department of Education. 2008. pp. 45–46. Archived (PDF) from the original on 2018-01-18. Retrieved 2020-11-03.
^ Pashler H, McDaniel M, Rohrer D, Bjork R (December 2008). "Learning Styles: Concepts and Evidence". Psychological Science in the Public Interest. 9 (3). SAGE Publications: 105–119. doi:10.1111/j.1539-6053.2009.01038.x. PMID 26162104. S2CID 2112166.
^ Nosek BA, Cohoon J, Kidwell MC, Spies JR (2018) [2015]. "Summary of reproducibility rates and effect sizes for original and replication studies overall and by journal/discipline". Estimating the Reproducibility of Psychological Science (table). Reproducibility Project: Psychology. Retrieved 2019-10-16.
^ ^a ^b ^c Nelson LD, Simmons J, Simonsohn U (January 2018). "Psychology's Renaissance". Annual Review of Psychology. 69 (1): 511–534. doi:10.1146/annurev-psych-122216-011836. PMID 29068778.
^ Roger A (27 August 2018). "The Science Behind Social Science Gets Shaken Up—Again". Wired. Retrieved 2018-08-28.
^ Camerer CF, Dreber A, Holzmeister F, Ho TH, Huber J, Johannesson M, et al. (September 2018). "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015". Nature Human Behaviour. 2 (9): 637–644. doi:10.1038/s41562-018-0399-z. PMID 31346273. S2CID 52098703.
^ Klein RA (2018). "Many Labs 2: Investigating Variation in Replicability Across Samples and Settings". Advances in Methods and Practices in Psychological Science. 1 (4): 443–490. doi:10.1177/2515245918810225. hdl:1854/LU-8637133.
^ ^a ^b Witkowski T (2019). "Is the glass half empty or half full? Latest results in the replication crisis in Psychology" (PDF). Skeptical Inquirer. Vol. 43, no. 2. pp. 5–6. Archived from the original (PDF) on 2020-01-30.
^ Richtel M (16 March 2022). "Brain-Imaging Studies Hampered by Small Data Sets, Study Finds". The New York Times.
^ Marek S, Tervo-Clemmens B, Calabro FJ, Montez DF, Kay BP, Hatoum AS, et al. (March 2022). "Reproducible brain-wide association studies require thousands of individuals". Nature. 603 (7902): 654–660. Bibcode:2022Natur.603..654M. doi:10.1038/s41586-022-04492-9. PMC 8991999. PMID 35296861.
^ Ioannidis JP (July 2005). "Contradicted and initially stronger effects in highly cited clinical research". JAMA. 294 (2): 218–228. doi:10.1001/jama.294.2.218. PMID 16014596. S2CID 16749356.
^ Prinz F, Schlange T, Asadullah K (August 2011). "Believe it or not: how much can we rely on published data on potential drug targets?". Nature Reviews. Drug Discovery. 10 (9): 712. doi:10.1038/nrd3439-c1. PMID 21892149.
^ Wheeling K (12 May 2016). "Big Pharma Reveals a Biomedical Replication Crisis". Pacific Standard. Retrieved 2020-01-30. Updated on 14 June 2017
^ ^a ^b Haelle T (7 December 2021). "Dozens of major cancer studies can't be replicated". Science News. Retrieved 2022-01-19.
^ ^a ^b "Reproducibility Project: Cancer Biology". www.cos.io. Center for Open Science. Retrieved 2022-01-19.
^ Mobley A, Linder SK, Braeuer R, Ellis LM, Zwelling L (2013). Arakawa H (ed.). "A survey on data reproducibility in cancer research provides insights into our limited ability to translate findings from the laboratory to the clinic". PLOS ONE. 8 (5): e63221. Bibcode:2013PLoSO...863221M. doi:10.1371/journal.pone.0063221. PMC 3655010. PMID 23691000.
^ Van Noorden R (July 2023). "Medicine is plagued by untrustworthy clinical trials. How many studies are faked or flawed?". Nature. 619 (7970): 454–458. Bibcode:2023Natur.619..454V. doi:10.1038/d41586-023-02299-w. PMID 37464079.
^ ^a ^b ^c ^d Tsui AS (21 January 2022). "From Traditional Research to Responsible Research: The Necessity of Scientific Freedom and Scientific Responsibility for Better Societies". Annual Review of Organizational Psychology and Organizational Behavior. 9 (1): 1–32. doi:10.1146/annurev-orgpsych-062021-021303. ISSN 2327-0608. S2CID 244238570.
^ Camerer CF, Dreber A, Forsell E, Ho TH, Huber J, Johannesson M, et al. (March 2016). "Evaluating replicability of laboratory experiments in economics". Science. 351 (6280): 1433–1436. Bibcode:2016Sci...351.1433C. doi:10.1126/science.aaf0918. PMID 26940865.
^ Bohannon J (3 March 2016). "About 40% of economics experiments fail replication survey". Science. doi:10.1126/science.aaf4141. Retrieved 2017-10-25.
^ Goldfarb RS (1 December 1997). "Now you see it, now you don't: emerging contrary results in economics". Journal of Economic Methodology. 4 (2): 221–244. doi:10.1080/13501789700000016. ISSN 1350-178X.
^ ^a ^b Bergh DD, Sharp BM, Aguinis H, Li M (6 April 2017). "Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings". Strategic Organization. 15 (3): 423–436. doi:10.1177/1476127017701076. ISSN 1476-1270. S2CID 44024633.
^ ^a ^b Stagge JH, Rosenberg DE, Abdallah AM, Akbar H, Attallah NA, James R (February 2019). "Assessing data availability and research reproducibility in hydrology and water resources". Scientific Data. 6: 190030. Bibcode:2019NatSD...690030S. doi:10.1038/sdata.2019.30. PMC 6390703. PMID 30806638.
^ ^a ^b Nature Video (28 May 2016). "Is There a Reproducibility Crisis in Science?". Scientific American. Retrieved 2019-08-15.
^ Fanelli D (April 2010). Scalas E (ed.). ""Positive" results increase down the Hierarchy of the Sciences". PLOS ONE. 5 (4): e10068. Bibcode:2010PLoSO...510068F. doi:10.1371/journal.pone.0010068. PMC 2850928. PMID 20383332.
^ ^a ^b Allen C, Mehler DM (May 2019). "Open science challenges, benefits and tips in early career and beyond". PLOS Biology. 17 (5). Public Library of Science: e3000246. doi:10.1371/journal.pbio.3000246. PMC 6513108. PMID 31042704.
^ "A New Replication Crisis: Research that is Less Likely to be True is Cited More". University of California, San Diego. 21 May 2021. Archived from the original on 2024-04-13.
^ Serra-Garcia M, Gneezy U (May 2021). "Nonreplicable publications are cited more than replicable ones". Science Advances. 7 (21). Bibcode:2021SciA....7.1705S. doi:10.1126/sciadv.abd1705. PMC 8139580. PMID 34020944.
^ Begley CG, Ioannidis JP (January 2015). "Reproducibility in science: improving the standard for basic and preclinical research". Circulation Research. 116 (1): 116–126. doi:10.1161/CIRCRESAHA.114.303819. PMID 25552691. S2CID 3587510.
^ Price DJ (1963). Little science big science. Columbia University Press. p. 32. ISBN 9780231085625.
^ Siebert S, Machesky LM, Insall RH (September 2015). "Overflow in science and its implications for trust". eLife. 4: e10825. doi:10.7554/eLife.10825. PMC 4563216. PMID 26365552.
^ Della Briotta Parolo P, Pan RK, Ghosh R, Huberman BA, Kaski K, Fortunato S (2015). "Attention decay in science". Journal of Informetrics. 9 (4): 734–745. arXiv:1503.01881. Bibcode:2015arXiv150301881D. doi:10.1016/j.joi.2015.07.006. S2CID 10949754.
^ ^a ^b Mirowski P (2011). Science-Mart. Harvard University Press. pp. 2, 24. ISBN 978-0-674-06113-2.
^ Moeller HG (2006). Luhmann explained: from souls to systems. Chicago: Open Court. p. 25. ISBN 0-8126-9598-4. OCLC 68694011.
^ Luhmann N (1995). Social systems. Stanford, CA: Stanford University Press. p. 288. ISBN 978-0-8047-2625-2. OCLC 31710315.
^ ^a ^b Scheufele DA (September 2014). "Science communication as political communication". Proceedings of the National Academy of Sciences of the United States of America. 111 (Suppl 4): 13585–13592. Bibcode:2014PNAS..111S3585S. doi:10.1073/pnas.1317516111. PMC 4183176. PMID 25225389.
^ Pielke R (2007). The honest broker : making sense of science in policy and politics. Cambridge: Cambridge University Press. doi:10.1017/CBO9780511818110. ISBN 978-0-511-81811-0. OCLC 162145073.
^ Martin GN, Clarke RM (2017). "Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices". Frontiers in Psychology. 8: 523. doi:10.3389/fpsyg.2017.00523. PMC 5387793. PMID 28443044.
^ Yeung AW (2017). "Do Neuroscience Journals Accept Replications? A Survey of Literature". Frontiers in Human Neuroscience. 11: 468. doi:10.3389/fnhum.2017.00468. PMC 5611708. PMID 28979201.
^ Hubbard R, Vetter DE (1 February 1996). "An empirical comparison of published replication research in accounting, economics, finance, management, and marketing". Journal of Business Research. 35 (2): 153–164. doi:10.1016/0148-2963(95)00084-4. ISSN 0148-2963.
^ "Confirmation bias". APA Dictionary of Psychology. Washington, DC: American Psychological Association. n.d. Retrieved 2022-02-02.{{cite encyclopedia}}: CS1 maint: year (link)
^ ^a ^b ^c Ferguson CJ, Heene M (November 2012). "A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science's Aversion to the Null". Perspectives on Psychological Science. 7 (6): 555–561. doi:10.1177/1745691612459059. PMID 26168112.
^ Dominus S (18 October 2017). "When the Revolution Came for Amy Cuddy". New York Times Magazine.
^ García-Berthou E, Alcaraz C (May 2004). "Incongruence between test statistics and P values in medical papers". BMC Medical Research Methodology. 4 (1): 13. doi:10.1186/1471-2288-4-13. PMC 443510. PMID 15169550.
^ Nieuwenhuis S, Forstmann BU, Wagenmakers EJ (August 2011). "Erroneous analyses of interactions in neuroscience: a problem of significance". Nature Neuroscience. 14 (9): 1105–1107. doi:10.1038/nn.2886. PMID 21878926.
^ ^a ^b Fanelli D (April 2010). "Do pressures to publish increase scientists' bias? An empirical support from US States Data". PLOS ONE. 5 (4): e10271. Bibcode:2010PLoSO...510271F. doi:10.1371/journal.pone.0010271. PMC 2858206. PMID 20422014.
^ Nosek BA, Spies JR, Motyl M (November 2012). "Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability". Perspectives on Psychological Science. 7 (6): 615–631. arXiv:1205.4251. doi:10.1177/1745691612459058. PMC 10540222. PMID 26168121. S2CID 23602412.
^ Everett JA, Earp BD (1 January 2015). "A tragedy of the (academic) commons: interpreting the replication crisis in psychology as a social dilemma for early-career researchers". Frontiers in Psychology. 6: 1152. doi:10.3389/fpsyg.2015.01152. PMC 4527093. PMID 26300832.
^ ^a ^b Clayson PE, Carbine KA, Baldwin SA, Larson MJ (November 2019). "Methodological reporting behavior, sample sizes, and statistical power in studies of event-related potentials: Barriers to reproducibility and replicability". Psychophysiology. 56 (11): e13437. doi:10.1111/psyp.13437. PMID 31322285.
^ LeBel EP, Peters KR (December 2011). "Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice". Review of General Psychology. 15 (4): 371–379. doi:10.1037/a0025172. ISSN 1089-2680.
^ ^a ^b ^c "Research misconduct – The grey area of Questionable Research Practices". www.vib.be. Vlaams Instituut voor Biotechnologie. 30 September 2013. Archived from the original on 2014-10-31. Retrieved 2015-11-13.
^ ^a ^b ^c ^d ^e ^f ^g ^h Wicherts JM, Veldkamp CL, Augusteijn HE, Bakker M, van Aert RC, van Assen MA (2016). "Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking". Frontiers in Psychology. 7: 1832. doi:10.3389/fpsyg.2016.01832. PMC 5122713. PMID 27933012.
^ ^a ^b ^c "The Nine Circles of Scientific Hell". Perspectives on Psychological Science (Opinion). 7 (6): 643–644. November 2012. doi:10.1177/1745691612459519. PMID 26168124. S2CID 45328962.
^ "Data dredging". APA Dictionary of Psychology. Washington, DC: American Psychological Association. n.d. Retrieved 2022-01-09. The inappropriate practice of searching through large files of information to try to confirm a preconceived hypothesis or belief without an adequate design that controls for possible confounds or alternate hypotheses. Data dredging may involve selecting which parts of a large data set to retain to get specific, desired results.{{cite encyclopedia}}: CS1 maint: year (link)
^ Begley CG (May 2013). "Six red flags for suspect work". Nature (Comment article). 497 (7450): 433–434. Bibcode:2013Natur.497..433B. doi:10.1038/497433a. PMID 23698428. S2CID 4312732.
^ Shea C (13 November 2011). "Fraud Scandal Fuels Debate Over Practices of Social Psychology". The Chronicle of Higher Education.
^ O'Boyle EH, Götz M (2022). "Questionable Research Practices". Research Integrity: Best Practices for the Social and Behavioral Sciences. Oxford University Press. pp. 261–294. ISBN 978-0190938550.
^ Glick JL (1992). "Scientific data audit—A key management tool". Accountability in Research. 2 (3): 153–168. doi:10.1080/08989629208573811.
^ Fiedler K, Schwarz N (19 October 2015). "Questionable Research Practices Revisited". Social Psychological and Personality Science. 7: 45–52. doi:10.1177/1948550615612150. ISSN 1948-5506. S2CID 146717227.
^ Fanelli D (May 2009). "How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data". PLOS ONE. 4 (5): e5738. Bibcode:2009PLoSO...4.5738F. doi:10.1371/journal.pone.0005738. PMC 2685008. PMID 19478950.
^ Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, et al. (May 2013). "Power failure: why small sample size undermines the reliability of neuroscience". Nature Reviews. Neuroscience. 14 (5): 365–376. doi:10.1038/nrn3475. PMID 23571845.
^ Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, et al. (May 2013). "Power failure: why small sample size undermines the reliability of neuroscience". Nature Reviews. Neuroscience. 14 (5): 365–376. doi:10.1038/nrn3475. PMID 23571845. S2CID 455476.
^ Ioannidis JP, Stanley TD, Doucouliagos H (1 October 2017). "The Power of Bias in Economics Research". The Economic Journal. 127 (605): F236–F265. doi:10.1111/ecoj.12461. ISSN 0013-0133. S2CID 158829482.
^ Flint J, Munafò MR (February 2013). "Candidate and non-candidate genes in behavior genetics". Current Opinion in Neurobiology. 23 (1): 57–61. doi:10.1016/j.conb.2012.07.005. PMC 3752971. PMID 22878161.
^ Dumas-Mallet E, Button KS, Boraud T, Gonon F, Munafò MR (February 2017). "Low statistical power in biomedical science: a review of three human research domains". Royal Society Open Science. 4 (2): 160254. Bibcode:2017RSOS....460254D. doi:10.1098/rsos.160254. PMC 5367316. PMID 28386409.
^ Farrell MS, Werge T, Sklar P, Owen MJ, Ophoff RA, O'Donovan MC, et al. (May 2015). "Evaluating historical candidate genes for schizophrenia". Molecular Psychiatry. 20 (5): 555–562. doi:10.1038/mp.2015.16. PMC 4414705. PMID 25754081.
^ Protzko J, Schooler JW (21 February 2017), Lilienfeld SO, Waldman ID (eds.), "Decline Effects: Types, Mechanisms, and Personal Reflections", Psychological Science Under Scrutiny (1st ed.), Wiley, pp. 85–107, doi:10.1002/9781119095910.ch6, ISBN 978-1-118-66107-9, retrieved 2024-07-26
^ ^a ^b Loken E, Gelman A (February 2017). "Measurement error and the replication crisis". Science. 355 (6325): 584–585. Bibcode:2017Sci...355..584L. doi:10.1126/science.aal3618. PMID 28183939.
^ Gelman, Andrew, and Eric Loken. "The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time." Department of Statistics, Columbia University 348.1-17 (2013): 3.
^ ^a ^b Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD (March 2015). "The extent and consequences of p-hacking in science". PLOS Biology. 13 (3): e1002106. doi:10.1371/journal.pbio.1002106. PMC 4359000. PMID 25768323.
^ Eisenberger NI, Lieberman MD, Williams KD (October 2003). "Does rejection hurt? An FMRI study of social exclusion". Science. 302 (5643): 290–292. Bibcode:2003Sci...302..290E. doi:10.1126/science.1089134. PMID 14551436.
^ ^a ^b Vul E, Harris C, Winkielman P, Pashler H (May 2009). "Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition". Perspectives on Psychological Science. 4 (3): 274–290. doi:10.1111/j.1745-6924.2009.01125.x. PMID 26158964.
^ ^a ^b Wagenmakers EJ (October 2007). "A practical solution to the pervasive problems of p values". Psychonomic Bulletin & Review. 14 (5): 779–804. doi:10.3758/BF03194105. PMID 18087943.
^ Wicherts JM, Veldkamp CL, Augusteijn HE, Bakker M, van Aert RC, van Assen MA (25 November 2016). "Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking". Frontiers in Psychology. 7: 1832. doi:10.3389/fpsyg.2016.01832. PMC 5122713. PMID 27933012.
^ Higgins JP, Thompson SG (June 2002). "Quantifying heterogeneity in a meta-analysis". Statistics in Medicine. 21 (11): 1539–1558. doi:10.1002/sim.1186. PMID 12111919. S2CID 6319826.
^ Moosa IA (2 October 2019). "The fragility of results and bias in empirical research: an exploratory exposition". Journal of Economic Methodology. 26 (4): 347–360. doi:10.1080/1350178X.2018.1556798. ISSN 1350-178X. S2CID 158504639.
^ Granger CW (1999). Empirical Modeling in Economics: Specification and Evaluation. Cambridge University Press. p. 5. doi:10.1017/CBO9780511492327. ISBN 978-0-521-77825-1.
^ Maziarz M (1 December 2021). "Resolving empirical controversies with mechanistic evidence". Synthese. 199 (3): 9957–9978. doi:10.1007/s11229-021-03232-2. ISSN 1573-0964. S2CID 236249427.
^ Morgan MS, Magnus JR (September 1997). "The experiment in applied econometrics". Journal of Applied Econometrics. 12 (5): 459–661. ISSN 1099-1255.
^ ^a ^b Van Bavel JJ, Mende-Siedlecki P, Brady WJ, Reinero DA (June 2016). "Contextual sensitivity in scientific reproducibility". Proceedings of the National Academy of Sciences of the United States of America. 113 (23): 6454–6459. Bibcode:2016PNAS..113.6454V. doi:10.1073/pnas.1521897113. JSTOR 26470212. PMC 4988618. PMID 27217556.
^ Trafimow D (July 2003). "Hypothesis testing and theory evaluation at the boundaries: surprising insights from Bayes's theorem". Psychological Review. 110 (3): 526–535. doi:10.1037/0033-295X.110.3.526. PMID 12885113.
^ Wagenmakers EJ (October 2007). "A practical solution to the pervasive problems of p values". Psychonomic Bulletin & Review. 14 (5): 779–804. doi:10.3758/BF03194105. PMID 18087943.
^ ^a ^b Cohen J (December 1994). "The earth is round (p < .05)". American Psychologist. 49 (12): 997–1003. doi:10.1037/0003-066X.49.12.997. ISSN 1935-990X.
^ ^a ^b Amrhein V, Korner-Nievergelt F, Roth T (2017). "The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research". PeerJ. 5: e3544. doi:10.7717/peerj.3544. PMC 5502092. PMID 28698825.
^ Branwen G (30 April 2023). "Everything Is Correlated". gwern.net.
^ Cohen J (1992). "Things I have learned (so far).". In Kazdin AE (ed.). Methodological issues & strategies in clinical research. Washington: American Psychological Association. pp. 315–333. doi:10.1037/10109-028. ISBN 978-1-55798-154-7. Retrieved 2024-07-26.
^ ^a ^b Meehl PE (1992). "Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology.". In Miller RB (ed.). The restoration of dialogue: Readings in the philosophy of clinical psychology. Washington: American Psychological Association. pp. 523–555. doi:10.1037/10112-043. ISBN 978-1-55798-157-8.
^ ^a ^b Paul Meehl (1986). What social scientists don't understand. In D. W. Fiske & R. A. Shweder (Eds.), Metatheory in social science: Pluralisms and subjectivities (pp. 315-338). Chicago: University of Chicago Press.
^ Jaynes ET, Bretthorst GL (2003). "5. Queer uses for probability theory". Probability theory: the logic of science. Cambridge, UK ; New York, NY: Cambridge University Press. ISBN 978-0-521-59271-0.
^ ^a ^b ^c Bird A (1 December 2021). "Understanding the Replication Crisis as a Base Rate Fallacy". The British Journal for the Philosophy of Science. 72 (4): 965–993. doi:10.1093/bjps/axy051. ISSN 0007-0882.
^ Nissen SB, Magidson T, Gross K, Bergstrom CT (December 2016). "Publication bias and the canonization of false facts". eLife. 5: e21451. arXiv:1609.00494. doi:10.7554/eLife.21451. PMC 5173326. PMID 27995896.
^ ^a ^b University of California San Diego (May 2021). "A new replication crisis: Research that is less likely to be true is cited more". phys.org. Retrieved 2021-06-14.
^ ^a ^b Serra-Garcia M, Gneezy U (May 2021). "Nonreplicable publications are cited more than replicable ones". Science Advances. 7 (21): eabd1705. Bibcode:2021SciA....7.1705S. doi:10.1126/sciadv.abd1705. PMC 8139580. PMID 34020944.
^ Białek M (January 2018). "Replications can cause distorted belief in scientific progress". The Behavioral and Brain Sciences. 41: e122. doi:10.1017/S0140525X18000584. PMID 31064528. S2CID 147705650.
^ ^a ^b Mede NG, Schäfer MS, Ziegler R, Weißkopf M (January 2021). "The "replication crisis" in the public eye: Germans' awareness and perceptions of the (ir)reproducibility of scientific research". Public Understanding of Science. 30 (1): 91–102. doi:10.1177/0963662520954370. PMID 32924865. S2CID 221723269.
^ ^a ^b ^c ^d Letzter R (22 September 2016). "Scientists are furious after a famous psychologist accused her peers of 'methodological terrorism'". Business Insider. Retrieved 2020-01-30.
^ "Draft of Observer Column Sparks Strong Social Media Response". APS Observer. Association for Psychological Science. September 2016. Retrieved 2017-10-04.
^ Fiske ST (31 October 2016). "A Call to Change Science's Culture of Shaming". APS Observer. 29 (9).
^ Singal J (12 October 2016). "Inside Psychology's 'Methodological Terrorism' Debate". NY Mag. Retrieved 2017-10-04.
^ ^a ^b Vazire S (July 2018). "Implications of the Credibility Revolution for Productivity, Creativity, and Progress". Perspectives on Psychological Science. 13 (4): 411–417. doi:10.1177/1745691617751884. PMID 29961410. S2CID 49647586.
^ Korbmacher M, Azevedo F, Pennington CR, Hartmann H, Pownall M, Schmidt K, et al. (25 July 2023). "The replication crisis has led to positive structural, procedural, and community changes". Communications Psychology. 1 (1): 3. doi:10.1038/s44271-023-00003-2. hdl:10852/106350. ISSN 2731-9121. PMC 11290608. PMID 39242883.
^ Stroebe W, Strack F (January 2014). "The Alleged Crisis and the Illusion of Exact Replication". Perspectives on Psychological Science. 9 (1): 59–71. doi:10.1177/1745691613514450. PMID 26173241. S2CID 31938129.
^ Jensen A (7 May 2019). "Replication as Success and Unsuccessful Replication". College of Liberal Arts, Department of Philosophy. University of Minnesota. Retrieved 2022-05-25.
^ Madhusoodanan J (May 2022). "The overlooked variable in animal studies: why diet makes a difference". Nature. 605 (7911): 778–779. Bibcode:2022Natur.605..778M. doi:10.1038/d41586-022-01393-9. PMID 35606524. S2CID 249015202.
^ Ioannidis JP (June 2016). "Why Most Clinical Research Is Not Useful". PLOS Medicine. 13 (6): e1002049. doi:10.1371/journal.pmed.1002049. PMC 4915619. PMID 27328301.
^ Ioannidis JP, Fanelli D, Dunne DD, Goodman SN (October 2015). "Meta-research: Evaluation and Improvement of Research Methods and Practices". PLOS Biology. 13 (10): e1002264. doi:10.1371/journal.pbio.1002264. PMC 4592065. PMID 26431313.
^ Bach B (8 December 2015). "On communicating science and uncertainty: A podcast with John Ioannidis". Scope. Retrieved 2019-05-20.
^ Gosselin RD (January 2020). "Statistical Analysis Must Improve to Address the Reproducibility Crisis: The ACcess to Transparent Statistics (ACTS) Call to Action". BioEssays. 42 (1): e1900189. doi:10.1002/bies.201900189. PMID 31755115. S2CID 208228664.
^ Pratt MG, Kaplan S, Whittington R (6 November 2019). "Editorial Essay: The Tumult over Transparency: Decoupling Transparency from Replication in Establishing Trustworthy Qualitative Research". Administrative Science Quarterly. 65 (1): 1–19. doi:10.1177/0001839219887663. ISSN 0001-8392. S2CID 210537501.
^ Aschwanden C (6 December 2018). "Psychology's Replication Crisis Has Made The Field Better". FiveThirtyEight. Retrieved 2018-12-19.
^ Chartier C, Kline M, McCarthy R, Nuijten M, Dunleavy DJ, Ledgerwood A (December 2018), "The Cooperative Revolution Is Making Psychological Science Better", Observer, 31 (10), retrieved 2018-12-19
^ "Registered Replication Reports". Association for Psychological Science. Retrieved 2015-11-13.
^ Chambers C (20 May 2014). "Psychology's 'registration revolution'". The Guardian. Retrieved 2015-11-13.
^ Lindsay DS (December 2015). "Replication in Psychological Science". Psychological Science. 26 (12): 1827–1832. doi:10.1177/0956797615616374. PMID 26553013.
^ ^a ^b Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers EJ, Berk R, et al. (January 2018). "Redefine statistical significance". Nature Human Behaviour. 2 (1): 6–10. doi:10.1038/s41562-017-0189-z. hdl:10281/184094. PMID 30980045.
^ Lakens D, Adolfi FG, Albers CJ, Anvari F, Apps MA, Argamon SE, et al. (March 2018). "Justify your alpha". Nature Human Behaviour. 2 (3): 168–171. doi:10.1038/s41562-018-0311-x. hdl:21.11116/0000-0004-9413-F. ISSN 2397-3374. S2CID 3692182.
^ Colquhoun D (November 2014). "An investigation of the false discovery rate and the misinterpretation of p-values". Royal Society Open Science. 1 (3): 140216. arXiv:1407.5296. Bibcode:2014RSOS....140216C. doi:10.1098/rsos.140216. PMC 4448847. PMID 26064558.
^ ^a ^b ^c ^d Colquhoun D (December 2017). "The reproducibility of research and the misinterpretation of p-values". Royal Society Open Science. 4 (12): 171085. doi:10.1098/rsos.171085. PMC 5750014. PMID 29308247.
^ Colquhoun D (11 October 2016). "The problem with p-values". Aeon Magazine. Retrieved 2016-12-11.
^ Longstaff C, Colquhoun D. "Calculator for false positive risk (FPR)". University College London. version 1.7.
^ Matthews RA (2001). "Why should clinicians care about Bayesian methods?". Journal of Statistical Planning and Inference. 94: 43–58. doi:10.1016/S0378-3758(00)00232-9.
^ Maxwell SE, Lau MY, Howard GS (September 2015). "Is psychology suffering from a replication crisis? What does "failure to replicate" really mean?". The American Psychologist. 70 (6): 487–498. doi:10.1037/a0039400. PMID 26348332.
^ IntHout J, Ioannidis JP, Borm GF, Goeman JJ (August 2015). "Small studies are more heterogeneous than large ones: a meta-meta-analysis". Journal of Clinical Epidemiology. 68 (8): 860–869. doi:10.1016/j.jclinepi.2015.03.017. hdl:2066/153978. PMID 25959635.
^ Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, et al. (May 2013). "Power failure: why small sample size undermines the reliability of neuroscience". Nature Reviews. Neuroscience. 14 (5): 365–376. doi:10.1038/nrn3475. PMID 23571845.
^ Greenwald AG (1975). "Consequences of prejudice against the null hypothesis" (PDF). Psychological Bulletin. 82 (1): 1–20. doi:10.1037/h0076157.
^ Kriegeskorte N, Simmons WK, Bellgowan PS, Baker CI (May 2009). "Circular analysis in systems neuroscience: the dangers of double dipping". Nature Neuroscience. 12 (5): 535–540. doi:10.1038/nn.2303. PMC 2841687. PMID 19396166.
^ Yarkoni T, Westfall J (November 2017). "Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning". Perspectives on Psychological Science. 12 (6): 1100–1122. doi:10.1177/1745691617693393. PMC 6603289. PMID 28841086.
^ "NWO makes 3 million available for Replication Studies pilot". Netherlands Organisation for Scientific Research (Press release). July 2016. Archived from the original on 2016-07-22.
^ ^a ^b ^c Apple S (22 January 2017). "The Young Billionaire Behind the War on Bad Science". Wired.
^ Frank MC, Saxe R (November 2012). "Teaching Replication". Perspectives on Psychological Science. 7 (6): 600–604. doi:10.1177/1745691612460686. PMID 26168118. S2CID 33661604.
^ Grahe JE, Reifman A, Hermann AD, Walker M, Oleson KC, Nario-Redmond M, et al. (November 2012). "Harnessing the Undiscovered Resource of Student Research Projects". Perspectives on Psychological Science. 7 (6): 605–607. doi:10.1177/1745691612459057. PMID 26168119.
^ Marwick B, Wang L, Robinson R, Loiselle H (22 October 2019). "How to Use Replication Assignments for Teaching Integrity in Empirical Archaeology". Advances in Archaeological Practice. 8: 78–86. doi:10.1017/aap.2019.38.
^ Everett JA, Earp BD (1 January 2015). "A tragedy of the (academic) commons: interpreting the replication crisis in psychology as a social dilemma for early-career researchers". Frontiers in Psychology. 6: 1152. doi:10.3389/fpsyg.2015.01152. PMC 4527093. PMID 26300832.
^ Ziano I, Mok PY, Feldman G (August 2021). "Replication and Extension of Alicke (1985) Better-Than-Average Effect for Desirable and Controllable Traits". Social Psychological and Personality Science. 12 (6): 1005–1017. doi:10.1177/1948550620948973. ISSN 1948-5506.
^ Korbmacher M, Azevedo F, Pennington CR, Hartmann H, Pownall M, Schmidt K, et al. (25 July 2023). "The replication crisis has led to positive structural, procedural, and community changes". Communications Psychology. 1 (1): 3. doi:10.1038/s44271-023-00003-2. ISSN 2731-9121. PMC 11290608. PMID 39242883.
^ Pennington CR (2023). A student's guide to open science: using the replication crisis to reform psychology. Maidenhead: Open University Press. ISBN 978-0-335-25117-9.
^ Kulke L, Rakoczy H (1 February 2018). "Implicit Theory of Mind – An overview of current replications and non-replications". Data in Brief. 16: 101–104. Bibcode:2018DIB....16..101K. doi:10.1016/j.dib.2017.11.016. ISSN 2352-3409. PMC 5694957. PMID 29188228.
^ "Curate Science". curatescience.org. Retrieved 2024-09-19.
^ LeBel EP, McCarthy RJ, Earp BD, Elson M, Vanpaemel W (September 2018). "A Unified Framework to Quantify the Credibility of Scientific Findings". Advances in Methods and Practices in Psychological Science. 1 (3): 389–402. doi:10.1177/2515245918787489. ISSN 2515-2459.
^ Röseler L, Kaiser L, Doetsch C, Klett N, Seida C, Schütz A, et al. (11 September 2024). "The Replication Database: Documenting the Replicability of Psychological Science". Journal of Open Psychology Data. 12 (1): 8. doi:10.5334/jopd.101. ISSN 2050-9863.
^ Quintana DS (September 2021). "Replication studies for undergraduate theses to improve science and education". Nature Human Behaviour (World View article). 5 (9): 1117–1118. doi:10.1038/s41562-021-01192-8. PMID 34493847. S2CID 237439956.
^ University of Cambridge (April 2022). "'Robot scientist' Eve finds that less than one-third of scientific results are reproducible". Techxplore. Retrieved 2022-05-15.
^ Roper K, Abdel-Rehim A, Hubbard S, Carpenter M, Rzhetsky A, Soldatova L, et al. (April 2022). "Testing the reproducibility and robustness of the cancer biology literature by robot". Journal of the Royal Society, Interface. 19 (189): 20210821. doi:10.1098/rsif.2021.0821. PMC 8984295. PMID 35382578.
^ ^a ^b Chambers C (10 June 2014). "Physics envy: Do 'hard' sciences hold the solution to the replication crisis in psychology?". The Guardian.
^ Kahneman D (2014). "A New Etiquette for Replication". Social Psychology (Commentary). Commentaries and Rejoinder on. 45 (4): 310–311. doi:10.1027/1864-9335/a000202.
^ Makel MC, Plucker JA, Hegarty B (November 2012). "Replications in Psychology Research: How Often Do They Really Occur?". Perspectives on Psychological Science. 7 (6): 537–542. doi:10.1177/1745691612460688. PMID 26168110.
^ Uhlmann EL, Ebersole CR, Chartier CR, Errington TM, Kidwell MC, Lai CK, et al. (September 2019). "Scientific Utopia III: Crowdsourcing Science". Perspectives on Psychological Science. 14 (5): 711–733. doi:10.1177/1745691619850561. PMID 31260639.
^ ^a ^b Forscher PS, Wagenmakers EJ, Coles NA, Silan MA, Dutra N, Basnight-Brown D, et al. (May 2023). "The Benefits, Barriers, and Risks of Big-Team Science". Perspectives on Psychological Science. 18 (3): 607–623. doi:10.1177/17456916221082970. PMID 36190899. S2CID 236816530.
^ Munafò MR, Davey Smith G (January 2018). "Robust research needs many lines of evidence". Nature. 553 (7689): 399–401. Bibcode:2018Natur.553..399M. doi:10.1038/d41586-018-01023-3. PMID 29368721.
^ ^a ^b ^c ^d ^e ^f Wallot S, Kelty-Stephen DG (1 June 2018). "Interaction-Dominant Causation in Mind and Brain, and Its Implication for Questions of Generalization and Replication". Minds and Machines. 28 (2): 353–374. doi:10.1007/s11023-017-9455-0. hdl:21.11116/0000-0001-AC9C-E. ISSN 1572-8641.
^ Tierney W, Hardy JH, Ebersole CR, Leavitt K, Viganola D, Clemente EG, et al. (1 November 2020). "Creative destruction in science". Organizational Behavior and Human Decision Processes. 161: 291–309. doi:10.1016/j.obhdp.2020.07.002. hdl:2066/228242. ISSN 0749-5978. S2CID 224979451.
^ Tierney W, Hardy J, Ebersole CR, Viganola D, Clemente EG, Gordon M, et al. (1 March 2021). "A creative destruction approach to replication: Implicit work and sex morality across cultures". Journal of Experimental Social Psychology. 93: 104060. doi:10.1016/j.jesp.2020.104060. hdl:10037/24275. ISSN 0022-1031. S2CID 229028797.
^ Delios A, Clemente EG, Wu T, Tan H, Wang Y, Gordon M, et al. (July 2022). "Examining the generalizability of research findings from archival data". Proceedings of the National Academy of Sciences of the United States of America. 119 (30): e2120377119. Bibcode:2022PNAS..11920377D. doi:10.1073/pnas.2120377119. PMC 9335312. PMID 35858443.
^ Ince DC, Hatton L, Graham-Cumming J (February 2012). "The case for open computer programs". Nature. 482 (7386): 485–488. Bibcode:2012Natur.482..485I. doi:10.1038/nature10836. PMID 22358837.
^ Vuong QH (January 2018). "The (ir)rational consideration of the cost of science in transition economies". Nature Human Behaviour. 2 (1): 5. doi:10.1038/s41562-017-0281-4. PMID 30980055. S2CID 46878093.
^ Junk TR, Lyons L (21 December 2020). "Reproducibility and Replication of Experimental Particle Physics Results". Harvard Data Science Review. 2 (4). arXiv:2009.06864. doi:10.1162/99608f92.250f995b. S2CID 221703733.
^ ^a ^b Ioannidis JP (February 2016). "Anticipating consequences of sharing raw data and code and of awarding badges for sharing". Journal of Clinical Epidemiology (Commentary). 70: 258–260. doi:10.1016/j.jclinepi.2015.04.015. PMID 26163123.

Replication crisis

Background

Replication

Statistics

History

Prevalence

In psychology

In medicine

In other disciplines

In economics

In water resource management

Across fields

Causes

Historical and sociological causes

Problems with the publication system in science

Publication bias

Mathematical errors

"Publish or perish" culture

Standards of reporting

Procedural bias

Cultural evolution

Questionable research practices and fraud

Prevalence

Statistical issues

Low statistical power

Positive effect size bias

Problems of meta-analysis

P-hacking

Statistical heterogeneity

Others

Context sensitivity

Bayesian explanation

Problem with null hypothesis testing

Base rate fallacy

Consequences

Public awareness and perceptions

Response in academia

Credibility revolution

Remedies

Reform in scientific publishing

Metascience

Presentation of methodology

Result-blind peer review

Pre-registration of studies

Metadata and digital tools for tracking replications

Statistical reform

Requiring smaller p-values

Addressing misinterpretation of p-values

Encouraging larger sample sizes

Cross-validation

Replication efforts

Funding

Emphasis in post-secondary education

Replication database

Final year thesis

Semi-automated

Involving original authors

Big team science

Broader changes to scientific approach

Emphasize triangulation, not just replication

Complex systems paradigm

Replication should seek to revise theories

Open science

See also

Notes

References

Further reading