1 pm : 3 pm

University of Amsterdam REC A 1.03

Addressing the Theory Crisis

A worrying number of psychological findings have turned out to not be replicable. Diagnoses of the causes of this “replication crisis”, and recommendations to address it, have thus far nearly exclusively focused on methods of data collection, analysis, and reporting. We argue that a further cause of poor replicability is the often weak logical link between theories and their empirical tests, and that the replication crisis will not be resolved without a deeper understanding of how psychologists formulate theories and link them to data collection. We propose a distinction between discovery-oriented and theory-testing research. In discovery-oriented research, researchers do not strongly imply hypotheses by which their theories can be tested, but rather define a search space for the discovery of effects that would support them, and failures to find these effects do not question the theory. This endeavor necessarily engenders a high risk of Type-I errors, that is, publication of findings that will not replicate. Theory-testing research, by contrast, relies on theories that strongly imply hypotheses, such that disconfirmation of the hypothesis provides evidence against the theory. Theory-testing research engenders a smaller risk of Type-I errors. We argue that a strong link between theories and hypotheses is best achieved by encouraging researchers to formalize theoretical assumptions as computational models. We critically revisit contemporary recommendations for addressing the “replication crisis”, such as the proposal to distinguish exploratory from confirmatory research, and the preregistration of hypotheses and analysis plans.

Short Bio

Professor Stephan Lewandowsky is a cognitive scientist at the University of Bristol. He was an Australian Professorial Fellow from 2007 to 2012, and was awarded a Discovery Outstanding Researcher Award from the Australian Research Council in 2011. He held a Revesz Visiting Professorship at the University of Amsterdam in 2012, and received a Wolfson Research Merit Fellowship from the Royal Society upon moving to the UK in 2013. He was appointed a Fellow of the Academy of Social Science and a Fellow of the Association of Psychological Science in 2017. In 2016, he was appointed a fellow of the Committee for Skeptical Inquiry for his commitment to science, rational inquiry and public education. In 2019, he received a Humboldt Research Award from the Humboldt Foundation in Germany. His research examines people’s memory, decision making, and knowledge structures, with a particular emphasis on how people update information in memory. His most recent research interests examine the potential conflict between human cognition and the physics of the global climate, which has led him into research in climate science and climate modeling.

Organized by Open Science Community Amsterdam (OSCA)