Guest post: Read the methods first

Originally a comment by Claire on Researchers found.

I can’t access the paper because it’s behind a paywall and I’m not at work. The appendix has the methods and detailed description of how they collected the data and that’s all I really care about. I always read the methods first. If I think the methods are garbage then the paper is garbage and I can devote my valuable time to something else.

PNAS is a good journal and I’m a little shocked that the paper was published. Statistically, this paper is flawed in many ways. Firstly, none of the methods adjusted for confounders. Confounders are elements that you have not accounted for in a study that may be coincidentally correlated with the trait of interest.

Here, they report that an overwhelming majority of parents identified as “liberal”. This is a problem in a study like this. I’m sure you can all see it already, parental attitudes to the trans movement are highly correlated with their overall political stances. Conservative parents are much more likely to disapprove of any expression of difference in gender presentation and sexual orientation, even at a young age. So you have already introduced selection bias right from the beginning. It’s right there in the data, you can see it.

Next confounder: locations of recruitment. This is a problem in all studies, including the sort of work I do. But I adjust for it! Cities are more liberal than rural areas as well as being more populous (easier to recruit in big cities vs small communities) and San Francisco is very different to Oklahoma City, politically speaking.

There are others, but I think I’ve made my point.

The methods themselves are terrible for these kinds of problems. Tests like t-tests, chi-sqs and even the more complex tests like ANOVA are not capable of adjusting for confounders (ANCOVA would work, but they didn’t use it).

There are other problems with the statistics too; some of the tests are inappropriate because of the “small cell” problem, they can’t report odds ratios or betas because they didn’t do the right tests. But I don’t want to get into the weeds here.

Finally, they don’t seem to have the faintest idea of how hypothesis testing works. They state their null hypothesis (H0) and their alternate hypothesis (HA) as two separate hypotheses (1a and 1b and so on). This indicates a lack of understanding of what they are doing. They list several hypothese in this paper (not counting the whole weird H0/HA presentation) and this hurts them.

Statistical power is the probability you will correctly reject the null hypothesis, i.e. the alternate hypothesis really is true. But every time you add another test, you have to adjust for it, which reduces the power. Here they claim good power but do not present their power calculations.

I’ve only outlined a few of the most pressing issues with it; there are more but I don’t want to bore you all. This is almost certainly a terrible paper, based on the methods description. I tried to be as open and unbiased as I could, reading it as if I were a reviewer of a paper in my own field. If I had reviewed this paper, I would have been very concerned about the standard of statistical expertise in this study and probably written to the editor to ask it be improved before I was even willing to do the review.

9 Responses to “Guest post: Read the methods first”