Projects
Replication Target Selection
Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arise: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic, and transparently communicated.
Using modified Delphi-procedure, we constructed a consensus-based checklist for transparent reporting of replication target selection. Item selection was informed by applied practices and expert opinion. The resulting list is designed to assist social scientists in making informed decisions about which studies to replicate. It further serves to transparently and systematically report these justifications.
The checklist for transparent replication target selection, will enable evaluation of future decisions to replicate and aid discussion about how resources are allocated and which studies ought to be prioritised. This checklist is useful to researchers that intend to replicate a study in making concrete their decision process as they prepare their study protocol. The checklist could also be used to score and evaluate funding applications for replication studies.
This project is conducted as a Registered Report and received IPA from the Royal Society Open Science in September 2021. All information can be found here.
In several projects, we explore and describe the evidential strength for efficacy underpinning endorsement decisions by the Food and Drug Administration (FDA). To this end, we adopt a Bayesian framework and quantify evidential strength using Bayes Factors. In contrast to p-values, Bayes Factors quantify evidence in favour of both the null hypothesis (i.e. no treatment effect) and the alternative hypothesis (i.e. a treatment effect) by comparing the relative likelihood of the observed data under either hypothesis.
Psychotropic drugs
We examined (1) whether psychotropic drugs are supported by substantial evidence (at the time of approval), and (2) whether there are systematic differences across drug groups. Data from short-term, placebo-controlled phase II/III clinical trials for antipsychotics, antidepressants for depression and anxiety disorders, and drugs for attention deficit hyperactivity disorder (ADHD) were extracted from FDA reviews. Bayesian model-averaged meta-analysis was performed and strength of evidence was quantified (i.e. BFBMA). Strength of evidence and trialling varied between drugs. Median evidential strength was extreme for ADHD medication, moderate for antipsychotics, and considerably lower and more frequently classified as weak or moderate for antidepressants for depression and anxiety disorders. Varying median effect sizes, sample sizes, and numbers of trials might account for differences. Although most drugs were supported by strong evidence at the time of approval, some only had moderate or ambiguous evidence. These results showed the need for more systematic quantification and classification of statistical evidence for psychotropic drugs. Evidential strength should be communicated transparently and clearly towards clinical decision makers.
Oncological drugs
In an upcoming project, we aim to demonstrate the utility of adopting a Bayesian framework to quantify evidential strength and uncertainty for statistical evaluations of benefits of novel cancer drugs at the time of approval. Reanalysing data from the CEIT-Cancer project, we aim to (1) describe evidential strength for clinical trials of drug and biologic products approved for the treatment of cancer in the last two decades; (2) contrast evidential standards between endpoints, accelerated vs. non-accelerated approval, lines of treatment, and type of cancer to describe potential differences in evidential strength.
Evidential Strength – Drug Endorsement
Merle-Marie Pittelkow 