Abstract
Meta-analysis synthesizes findings across studies to estimate overall effects, but effect size variability often remains unexplained despite traditional moderator analyses. Study quality may account for some of this heterogeneity. This study explores whether mixture modeling can serve as an alternative moderator analysis technique by identifying latent subpopulations of studies based on quality indicators. The current research addressed two questions: (1) Can mixture modeling be used to identify distinct populations of studies using quality indicators as moderating variables?, and (2) What methods for quantifying study quality are optimal for use when employing a mixture modeling approach to moderator analysis in meta-analysis?
A simulation study was conducted varying six parameters: true effect size (d), mean difference between populations (md), between-study variance (t2), variance ratio across groups, number of studies (k), and the correlation between study quality and effect size. Five models were tested using different approaches to quality scoring: sum, mean, factor score, latent variable, and multivariate mixture modeling. Percent agreement, bias, and mean squared error were used to evaluate performance.
Results showed that mixture modeling using continuous quality indicators struggled to classify studies accurately. The multivariate approach outperformed others in reducing bias and MSE, particularly when the correlation between quality and effect size was strong.
These findings suggest that while mixture modeling has potential as a moderator analysis method, its effectiveness is limited when study quality has weak associations with effect sizes. The added complexity of multivariate models may not be warranted over simpler quality scoring approaches in typical research scenarios. Future work should explore categorical moderators and alternative approaches for capturing study quality effects in meta-analysis.