Unveiling replication crisis: Are most published research findings false?

In a groundbreaking journal article entitled "Why most published research findings are false" published in 2005, John Ioannidis from the Stanford University ignited a crucial discussion on the credibility of scientific research. It was revealed that the pervasive issue of false positives in academic literature is due to various factors in the research ecosystem, including the pressure to publish by researchers, and the pressure to publish ‘impactful’ results. This state of affairs has led to a phenomenon, termed "The Replication Crisis," highlighting the urgency for the scientific community to adopt more robust practices and recognise the role of replication studies. Three talented researchers at the UC Business School, Professors Bob Reed and Tom Coupe, and Dr Sanghyun Hong, delve into the heart of this issue.

Motivation behind the research

Bob's interest in this subject began 15 years ago when he attempted to reproduce the findings of previous studies but yielded inconsistent results. This sparked a research journey for him that still continues to this day. Bob’s multifaceted research approach spans replication studies (replications of other people’s work to determine if it is reliable), statistical meta-analyses (study of statistical methods for synthesizing the results from many studies), and advocacy for replication research dissemination. Through initiatives like editing the Journal of Comments and Replications in Economics (where provides a place for researchers to publish replications) and founding a website called “The Replication Network” (where provides a searchable database of replications in economics), Bob is committed to foster a culture of transparency and reliability in academic research. Apart from those initiatives, he leads a renowned research cluster at UC called “UCMeta” together with Tom and Sanghyun, where they collectively explore various topics within “meta-science” with a network of enthusiastic researchers.

Anticipated impacts

The team is championing a transformative shift in academic research, paving the way for a more transparent and robust scientific landscape. In the short term, the team seeks to encourage economists and scientific journals to publish replication studies despite challenges such as diminished attention and potential backlash. By advocating for improved meta-analytic methodologies, the team seeks to enhance the reliability of research synthesis, enabling clearer insights amidst the maze of published empirical studies.

Looking ahead, the long-term goal for the team is to address "The Replication Crisis" through innovative solutions. Collaboration has been developed with scholars at the University of Texas at Austin (US), the St. Louis Federal Reserve (US), the Centre for Open Science (US), ZBW – Leibniz Information Centre for Economics (Germany), the University of Essex (UK), and MetaMelb Research Initiative (Australia). The team continues to explore effective strategies to address this complex issue.

With unwavering commitment and inventive approaches, the team seamlessly integrates their research discoveries into Economics undergraduate courses at the UC Business School, including Introduction to Econometrics (ECON213) and Econometrics (ECON324). By imparting knowledge on the significance of replication studies and fostering a culture of critical evaluation of research findings, the team strives to empower students to contribute to enhancing research integrity and reliability in their academic and professional endeavours. 

Next steps

An exciting new initiative that the team is currently working on is the development of an automatic alert system to notify authors of “working papers” (unpublished research that is seeking to be published in a journal) about relevant replication studies of the original papers that they have cited. By increasing awareness and citation rates of replication studies, this endeavour seeks to incentivise further replication efforts and encourage journals to publish them, ultimately fostering a culture of scientific rigor.

For more information about this research, please contact Bob.

Related media

Guest blog - The Replication Network

Outputs

Journal articles

Hong, S., & Reed, W.R. (2024). Meta‐analysis and partial correlation coefficients: A matter of weights. Research Synthesis Methods, 15(2), 303-312.

Coupé, T., Reed, W.R., & Zimmermann, C. (2023). Getting seen: Results from an online experiment to draw more attention to replications. Research Policy, 52(8), 104841.

Hong, S., & Reed, W.R. (2021). Using Monte Carlo experiments to select meta‐analytic estimators. Research Synthesis Methods, 12(2), 192-215.

Havránek, T., Stanley, T.D., Doucouliagos, H., Bom, P., Geyer‐Klingeberg, J., Iwasaki, I., Reed, W.R., Rost, K.,& van Aert, R.C.M. (2020). Reporting guidelines for meta‐analysis in economics. Journal of Economic Surveys, 34(3), 469-475.

Xue, X., Reed, W.R., & Menclova, A. (2020). Social capital and health: A meta-analysis. Journal of Health Economics, 72, 102317.

Duvendack, M., Palmer-Jones, R., & Reed, W.R. (2017). What is meant by “replication” and why does it encounter resistance in economics?. American Economic Review, 107(5), 46-51.