In research on people, scientists are typically interested in the group data – the mean, median, and variance of a sample of people. But according to a provocative new paper out in PNAS, the statistics of a group can obscure the variability within individuals, over time.

The paper, from Aaron J. Fisher, John D. Medaglia, and Bertus F. Jeronimus, isn’t really making a new point. The pitfalls of generalizing from the group to the individual level have long been known – but these issues are typically discussed in the form of hypothetical scenarios or contrived examples. Fisher et al. show how these issues apply to real-world data.

The authors took datasets from six psychology studies, all of which involved repeated measures from each participant: for instance, in Study #1, 43 people suffering from depression or anxiety had to rate their mood, four times each day for one month.

Because each participant gave multiple measures, Fisher et al. were able to calculate a series of between-subjects statistics (e.g. group mean at time 1, group variance at time 1, and so on) as well as within-subjects stats (mean across participant 1, variance across participant 1).

This revealed that the within-subjects standard deviation was on average nearly eight times higher than the corresponding between-subjects standard deviation. What’s more, for some pairs of variables, correlations calculated between subjects were much higher than correlations within subjects, as in this case, for the correlation between Depressed Mood and Worry in Study #1:

Between-subjects, the correlation was consistently around 0.7, but within-subjects, the same correlation averaged about 0.4, and was much more varied, with some participants showing no correlation or even a slightly negative one.

This could have important implications. Fisher et al. explain that the depressed mood vs. worry correlation is known from many studies (between-subjects ones), and it’s so strong that it has led to the view that depression and anxiety disorders are variants of the same condition. But,

The putative covariance between these syndromes in the extant literature may underestimate the considerable variability in the relationship across individuals. Thus, taxonomic decisions based on the putative group correlation between these dimensions would likely lack ecological and clinical validity, and may undermine treatment planning and outcomes. These results also suggest that our taxonomies are unlikely to capture the diversity of natural kinds that may exist across humans.

The authors conclude that the problem is ultimately due to the fact that most data from humans is non-ergodic – meaning, in a nutshell, that people are not interchangable – and that taking account of this non-ergodicity is necessary to perform statistical inference. Again, it’s not a new point, but many of the examples in this paper are pretty convincing illustrations.