Open Science Series: A replication crisis in methodological computational research?

Open Science Webinar Series

A replication crisis in methodological computational research?

Dr. Anne-Laure Boulesteix, Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig Maximilians University, Munich, Germany

Statisticians are often keen to analyze the statistical aspects of the so-called “replication crisis”. They condemn fishing expeditions and publication bias across empirical scientific fields applying statistical methods. But what about good practice issues in their own - methodological - research, i.e. research considering statistical methods as research objects? When developing and evaluating new statistical methods and data analysis tools, do statisticians adhere to the good practice principles they promote in fields which apply statistics? I argue that statisticians should make substantial efforts to address what may be called the replication crisis in the context of methodological research in statistics and data science, in particular by trying to avoid bias in their comparison studies based on simulated or real data. I will discuss topics such as publication bias, the design and necessity of neutral comparison studies and the importance of appropriate reporting and research synthesis in the context of methodological (bio)statistical research by drawing an analogy with clinical research.