Simpson, A. (2017). The misdirection of public policy: comparing and combining standardised effect sizes. Journal of Education Policy, 32(4), pp.450-466.

Fascinating paper by Adrian Simpson pulls no punches in arguing that the way meta-analyses are constructed in education leads us to entirely wrong conclusions ($) tandfonline.com/doi/abs/10.108…

Media preview
I would question the claim that meta-analyses are generally used uncritically: you hear tales of this, but I’d say there’s still a huge amount of resistance.
Simpson 3.png The fundamental issue with meta-analyses is that researchers’ task is to find a difference, but the (legitimate) design choices they make to do so invalidate meta-analyses.
Simpson 4.pngSimpson 11.png
For example, meta-analyses combine different comparison groups in ways which are contradictory.
 Simpson 5.pngSimpson 6.png

If you just analyse the effects on a sub-population, you get a higher effect size: again, these are combined uncritically.

Simpson 8.pngSimpson 9.png And, as we know, a researcher-designed instrument suited to the test gets a higher result, but again, these are combined uncritically.

Simpson 10.png

 So they don’t help us know what makes a bigger difference.
Simpson 12.png
Simpson 13.png

Conclusion: this is a wrong direction.

Simpson 14.png
Dylan 2.png

One thought on “Simpson, A. (2017). The misdirection of public policy: comparing and combining standardised effect sizes. Journal of Education Policy, 32(4), pp.450-466.

  1. Thanks this is an interesting article and this backs up Bergeron who published similar arguments in a paper here- http://mje.mcgill.ca/article/view/9475/7229

    Also, Adrian Simpson gave an interview with Ollie Lovell that i was present at, that extends his arguments – it should be posted here soon – http://www.ollielovell.com/podcast/

    I am putting together a critique of Hattie and am looking for anyone interested to help here –

    https://visablelearning.blogspot.com.au/

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s