This master class introduces and applies recently developed methods of meta-analysis to integrate, summarize and understand rapidly expanding research in the health, business and the social sciences. We demonstrate how these methods address and mitigate the sources of the current credibility crisis across the disciplines.
Dr Emily Kothe is the Co-Director of DeLMAR and the Stream Leader for Systematic Review and Meta-Analysis in the School of Psychology Data Sciences Unit. She has conducted systematic reviews and meta-analyses across a number of topics within psychology including: attachment, cognitive dissonance, health behaviour and behaviour change, and body image.
Selective reporting, questionable research practices, low statistical power, and wide variation of findings across an area of research are the causes of the current credibility/replication crisis. Meta-analysis and meta-regression analysis can identify the reach of these problems in your area of research, filter out much of their effects, and thereby reduce their influence. When carefully conducted, meta-analysis can greatly increase research credibility.
The advanced methods presented in this course are designed to address these issues directly. Meta-regression analysis routinely explains much of the wide and often conflicting variation typically found among reported findings. Recently developed meta-analysis methods use a study’s statistical power to reduce these potential biases. We aim to provide participants with the tools needed to produce state-of-the-art research synthesis and meta-analyses.
The workshop is relevant to all researchers in health, medicine, psychology, economics, business and other social sciences could benefit from this class if they are already somewhat knowledgeable about meta-analysis.
Day 1
1) The Current Replication/Credibility ‘Crisis’ (John et al., 2012; Ioannidis et al., 2017)
a) Publication bias, questionable research practices and heterogeneity
b) What meta-analyses tell us about contemporary research.
i) Replicability (OSC, 2015; Stanley et al., 2017)
ii) Statistical power (Ioannidis et al., 2017; Stanley et al., 2017)
iii) Publication bias
iv) Heterogeneity
2) Review of Meta-Analysis (MA) (Hunter & Schmidt, 2000, Borenstein et al., 2009)
a) Effect sizes
b) Fixed and Random Effects
c) Application to psychology and health research
d) Issues and Limitations
3) Methods to Accommodate and Reduce Publication Bias (Stanley, 2008)
a) Causes: Questionable Research Practices (Ferguson & Heene,2012; Franco et al.,2014)
b) Funnel graphs and motivating examples (Stanley and Doucouliagos, 2010)
c) Unrestricted Weighted Least Squares (WLS) (Stanley and Doucouliagos, 2015)
d) FAT-PET-PEESE and other methods (Stanley and Doucouliagos, 2014)
Day 2
1) High Heterogeneity among Research Results (Higgins & Thompson, 2002)
a) How high heterogeneity limits research synthesis
b) How prevalent is it? (Ioannidis et al, 2017; Stanley et al, 2017)
c) Meta-analysis methods to measure and accommodate heterogeneity
2) Introduction to Meta-Regression Analysis (MRA) (Borenstein et al., 2009)
a) Basic MRA models and their interpretation
b) Applications from psychology, economics and health
3) MRA: Advanced Topics with extensive applications (Stanley and Doucouliagos, 2012)
a) Robustness of MRA findings
b) Seeing through the fog of Multicollinearity
i) General-to-specific modelling
ii) Bayesian Model Averaging
c) Dependence across reported estimates (Hedges et al., 2010)
i) Cluster-robust standard errors
ii) Fixed and random-effects panel MRA
iii) Multilevel models (Cheung, 2014).
4) ‘Hands on’ lab: Using R and STATA for MA and MRA
This two day master-class will be held in the ACSPRI Office in Melbourne.
Attendees should bring a laptop with either (a) STATA or (b) R and RStudio installed. Instructions with how to install R and RStudio will be circulated before the workshop. If you don’t have a copy of Stata, could you please contact us in advance & we will organise a trial version for you for the course.
Participants should be conducting a meta-analysis or have attended a previous meta-analysis workshop or class. Participants also need to be knowledgeable about basic research statistics and regression analysis.
There are no required readings for this class, although before attending, we recommend reading:
- Doucouliagos, C. (2016) Meta-regression analysis: Producing credible estimates from diverse evidence. IZA World of Labor: 320. doi:10.15185/izawol.320.
Further References:
- Borenstein, M., Hedges, L. V., Higgins, J. P. T. and Rothstein, H. R. (2009). Introduction to meta-analysis, Chichester, U.K. John Wiley & Sons.
- Cheung, M. W.-L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19(2), 211-229.
- Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502-1505.
- Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7(6),555-561.
- Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta‐regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39-65.
- Higgins, J., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta‐analysis. Statistics in Medicine, 21(11), 1539-1558.
- Hunter, J. E., & Schmidt, F. L. (2000). Fixed effects vs. random effects meta‐analysis models: Implications for cumulative research knowledge. International Journal of Selection and Assessment, 8(4), 275-292.
- Ioannidis, J.P.A., Stanley, T.D. and Doucouliagos, C. (2017). “The power of bias in economics research,” The Economic Journal, 127: F236-265.
- John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524-532.
- Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716–aac4716. doi:10.1126/science.aac4716
- Stanley, T.D. (2008). Meta-regression methods for detecting and estimating empirical effect in the presence of publication selection. Oxford Bulletin of Economics and Statistics, 70:103-27.
- Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2017). What meta-analyses reveal about the replicability of psychological research. Unpublished manuscript.
- Stanley, T.D and Doucouliagos, H. (2010). Picture this: A simple graph that reveals much ado about research.” Journal of Economic Surveys, 24: 170-91.
- Stanley, T.D. and Doucouliagos, H. (2012). Meta-regression analysis in economics and business. Oxford: Routledge.
- Stanley, T. D., & Doucouliagos, H. (2014). Meta‐regression approximations to reduce publication selection bias. Research Synthesis Methods, 5(1), 60-78.
- Stanley, T.D. and Doucouliagos, C. “Neither fixed nor random: Weighted least squares meta-analysis,” Statistics in Medicine 34 (2015), 2116-27.
- Stanley, T.D., Doucouliagos, C. and Ioannidis, J.P.A. (2017). “Finding the Power to Reduce Publication Bias,” Statistics in Medicine, 36: 1580-1598.
Emily was excellent - R studio exercises are fantastic! (October 2018)
Thanks for an excellent and informative course, the two different convenors worked very well. (October 2018)