Meta-analytic thinking
Thompson (2002, p.28) defines meta analytic thinking as: “a) the prospective formulation of study expectations and design by explicitly invoking prior effect size measures and b) the retrospective interpretation of new results, once they are in hand, via explicit, direct comparison with the prior effect sizes in the related literature”
Prospective formulation involves reviewing the literature for other relevant studies that provide insight into the estimated effect size. Meta analyses can be particularly useful for this purpose.
Retrospective interpretation emphasises not being overly focused on the results of any one study. The approach is somewhat bayesian in its orientation. Results of the focal study are interpreted within a context of prior research. The stronger and more relevant the prior research, and the weaker the current research, the less the results of the current research is given credence. In many respects, each study is interpreted as addition to a much larger meta analysis.
There is an overall emphasis on aligning research questions with estimates of population parameters. These parameters might be correlations (e.g., correlation between intelligence and job performance), standardised group differences (e.g., the effectiveness of a drug in reducing depression relative to a control group). Any one study will get a point estimate of the effect size and this is what is most commonly reported (e.g., the correlation between x and y is .5). Meta analytic thinking emphasises the importance of also obtaining confidence intervals around effect sizes (e.g., we can be 95% confident that the correlation between x and y is between .3 and .65). This approach highlights the uncertainty associated with our knowledge of the practical importance of an effect. As studies accumulate and sample sizes increase, confidence intervals get progressively smaller.
References
- Thompson, B. (2002). What future quantitative social science research could look like: Confidence intervals for effect size. Educational Researcher, 31(3), 25.