Earlier this year, I posted on how Sergio Della Salla, the editor of Cortex, criticized a headline-grabbing JAMA paper that had reported neuropsychological abnormalities in US embassy staff exposed to the mysterious Havana ‘sonic attack’. According to Della Salla, the evidence presented didn’t suggest enduring cognitive deficits in the victims.

Now, Della Salla is back (along with co-authors) for round two with a new paper, called Cognitive symptoms in US government personnel in Cuba: The mending is worse than the hole. He argues that a new clarification of the JAMA paper’s methodology makes even less sense than the original.

As I discussed in my first post, the bone of contention here is how we define a test score as ‘abnormal’ or ‘impaired’. In the original JAMA paper, the authors stated that they defined abnormality on a test as a score below the 40th percentile.

The 40th percentile means a score which, according to previous studies, 4 in 10 people would score lower than. As Della Salla said in his first paper, this is a very liberal definition of abnormality, especially since each patient received 37 different scores from a battery of tests, making the chance of seeing one or more scores <40th percentile very high even if the patient had normal function.

After Della Salla’s first piece was published, the authors of the criticized JAMA paper posted a rebuttal (to a different set of critics) in which they seemingly switched to a different definition of dysfunction.

In the rebuttal, the JAMA authors said that:

Interpretation of these results is more nuanced than a simple counting of scores lower than conventional percentile cutoffs. Within-individual deviations from an average performance are considered signs of brain dysfunction. Percentile scores in our report showed that all impaired patients had several scores that deviated by more than 1 SD from their respective means, some exceeding 2 SDs, which translates to more than 40 percentile points below their means (below 10th percentile relative to their average performance). This meets standard criteria for neuropsychological impairment

As Della Salla et al. say in the new piece, this is a “confusing, ambiguous statement”. Less politely, one could say that it doesn’t make any sense.

Having puzzled over the statement for a while, I think what the JAMA authors are saying is that all of the patients had some test scores which were at least 1 standard deviation (SD) lower than that patient’s own personal mean score across all 37 tests. Some of the patients had scores that fell over 2 SD below their own mean score.

Assuming this is what is meant, it’s still a fatally flawed definition of abnormality, as Della Salla et al. go on to explain:

Within-subject profiling can be a valid approach to understanding variations in a person’s abilities across domains, but such profiles can only be evaluated by reference to the distribution of between-test differences in a normative sample. The within-subject approach that Hampton and colleagues’ statement implies has no normative point of reference, and would be meaningless.

Incidentally, I wonder if the confusing statement was meant to justify the heavily criticized original “40th percentile” rule? This would explain the strange and awkward statement that “2 SDs translates to more than 40 percentile points below their means”. Except as a deliberate reference to the 40th percentile, I can see no reason you would write this. 2 SD below the mean is about 43 centiles below it, not 40, and centiles below the mean is an odd thing to talk about in the first place.

Overall, I agree with Della Salla that there are big problems with the JAMA paper. This doesn’t mean that there was no sonic attack in Havana (or microwave attack). The JAMA paper wasn’t about the nature of the alleged attack; it was a study of the symptoms in those affected. That said, if there is no evidence of lasting neuropsychological dysfunction, this might be taken as evidence against the idea that a brain-damaging weapon was involved.