Rethinking Research Sample Size - Beyond the Rules of Thumb

Deep Dive Podcast: Rethinking Research Sample Size - Beyond the Rules of Thumb
Are you relying on common sample size rules of thumb in your research? A recent critical analysis by Aguinis and Harden challenges these practices, shedding light on their limitations and alternative approaches. Here are 5 key takeaways:
1️⃣ Power Analysis Isn’t One-Size-Fits-All: Using Cohen’s definitions of small, medium, and large effect sizes for power analysis is common but often misapplied. These values are subjective and should only be a last resort when other context-specific methods aren’t feasible.
2️⃣ Adjusting Type I Error Rates Needs Thoughtfulness: Increasing the Type I error rate (e.g., to 0.10) to boost power with small samples is widespread. However, this adjustment is often arbitrary and overlooks critical trade-offs between Type I and Type II errors.
3️⃣ The 5:1 Rule of Thumb - A Lower Bound, Not a Goal: The guideline suggesting five observations per parameter in covariance structure analysis is oversimplified and only valid under rare multivariate normality. Most cases demand at least 10 observations per parameter.
4️⃣ Context Matters for Effect Size and Error Rates: Small effects can have significant implications in some research contexts, contradicting the assumption that only large effects matter. Similarly, Type I error adjustments should align with the specific study’s stakes and consequences.
5️⃣ Critical Thinking Over Convenience: Reliance on these rules of thumb often stems from convenience or reviewer expectations rather than methodological rigor. Researchers must prioritize contextually informed decisions over blanket rules.
Let’s innovate how we design and evaluate research for meaningful insights. What are your experiences with these rules of thumb?
Get article: Aguinis, H., & Harden, E. E. 2009. Sample size rules of thumb: Evaluating three common practices. In C. E. Lance and R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Received doctrine, verity, and fable in the organizational and social sciences: 269-288. New York: Routledge. Available at www.hermanagui...