Restriction of range

Studies of cognitive ability often use samples that are not representative of the population. This is especially salient if the subjects are university students, a highly selected group. This practice means that correlations with cognitive ability found in studies using students are too low if they are meant as an estimate of the effect size in the general population. Fortunately, methods exist that can correct for this bias, see Methods of Meta-analysis by Frank Schmidt and John Hunter.

Two methods for restricting the range are provided. In direct restriction, we select only cases within two cutoffs and leave everything else out. This scenario can happen when using IQ tests as a hiring tool in top-down selection. However, it is more commonly the case that cognitive abiltiy is not the sole criterion or that its influence is only probabilistic. In indirect restriction, each case has a probability of being excluded based on a function. Direct restriction is a special case of indirect restriction where the sd of the restriction function is 0.

The figure below visualizes the effect by color-coding the excluded datapoints and showing the correlation in the population, the sample, the reduction in the correlation in %, and the reduction in variance in %.


Made by Emil O. W. Kirkegaard using Shiny for R. Source code available on Github.