The following is a version of the argument that I advanced in yesterday’s panel discussion about meta-analysis and meta-meta-analysis at researchED 2018. It’s probably a little more coherent than the live version due to the ability to edit. It is worth reading Robert Coe’s posts on the same issue.
There are two interlinked problems with the kind of meta-meta-analysis represented by the Education Endowment Foundation’s Toolkit and John Hattie’s Visible Learning. The first is the way that the effect size metric is used (quoted as ‘months of additional progress’ in the case of the Education Endowment Foundation) and the second is the way that very different kinds of interventions are packaged together.
The effect size metric
There is, in my opinion, nothing intrinsically wrong with effect sizes. This is not an argument about whether effect sizes are good or bad, it is about the validity of the inferences that…
View original post 746 more words