Further reading
The reaction to the NASA-arsenic based life story makes a nice study guide to epistemology and how scientists think and how various distorting influences (like media priorities and funding needs) can bollix things up. PZ set us straight last week almost before the ink was dry, Rosie Redfield wrote a scathing analysis on Saturday, Carl Zimmer talked to a dozen experts on Monday.
Almost unanimously, they think the NASA scientists have failed to make their case. “It would be really cool if such a bug existed,” said San Diego State University’s Forest Rohwer, a microbiologist who looks for new species of bacteria and viruses in coral reefs. But, he added, “none of the arguments are very convincing on their own.” That was about as positive as the critics could get. “This paper should not have been published,” said Shelley Copley of the University of Colorado.
Read the whole article for details; read Redfield’s post; read Jerry Coyne’s post including comments, most of which are from people who know something relevant.
I started to worry as soon as Nasa started making the “stay-tuned in a couple of days” announcements.
One thing that bothers me is that, if I understand it correctly, the real breaking news would have been the determination that the arsenic was in fact being taken up to construct the DNA backbone with sugars in place of phosphates. Yet this appears to be only “implied” or “inferred” and not definitively proven. So why all the hype?
I would be very interested to see the reviews by the anonymous reviewers prior to publication. The distorting influences that you speak of could have either ignored or marginalized them if they were, in fact, critical of this work (I admit this is pure speculation).
I saw the live press conference by NASA over this paper. And it was pretty shambolic and embarrassing. I wasn’t sure if the press conference was aimed at children or adults, and when the press finally started asking adult questions, things went a bit awry, and one of the co-writers of the paper started bickering with another scientist.
The good news is, that scientists are sceptics and also human with egos, while science is rather objective in the end. And so the scepticism, no matter how harsh and cruel, is part of the process. It is a shame that politics and government funding interfere with science, but that is to be expected with NASA.
NASA lost sight of their mission years ago and for the last few decades has struggled for non-space related relevance. They keep staging dog and pony shows but the public is bored.
Martin Robbins’ Lay Scientist blog. probably gives the best sequence of events of this story here:
http://www.guardian.co.uk/science/the-lay-scientist/2010/dec/08/2
And it does show how communication breakdown is why things got so unreasonable. And this is a lesson we must always remember: communication is key to rational debate, and stonewalling is definitely antithetical to rational debate.
I was pleased to see Zimmer’s article picked up by Gawker yesterday. Kudos to popular media sites that bring the scientific conversation to a general audience.
That is some harsh post-publication peer review from one of the most distinguished authorities on extremophiles.
We don’t yet know, though, whether bad reviewers are at fault here. It is not ultimately the reviewers who decide if a paper sees publication. It’s the editors, supposedly informed by the reviewers. It may be that Science has become too enamored of blockbuster stories.
“It is not ultimately the reviewers who decide if a paper sees publication. It’s the editors, supposedly informed by the reviewers.”
The other way to put that statement is to say that for a bad paper to get published it requires both the reviewers AND the editor getting it wrong.
Unfortunately that is not an infrequent occurrence. “Peer review” sounds like a simple process but it is fraught with potential for both error and unethical behavior. Some journals are better than others at addressing this matter but ‘Science’ is not one of them.
How can a paper like the one discussed here get published?
Simple. Send it to a journal with a letter over hyping the findings as a revolutionary paper in astrobiology and suggesting some reviewers that are ‘friendly’ towards you (a lot of journals now ask the authors to suggest reviewers!). Either those suggested by the authors or other reviewers who are not specialists in the particular topic at hand are chosen by the sub-editor as reviewers – i.e. no biochemists or microbiologists, just astronomers are the reviewers. The reviewers see no great problem with the paper (since they don’t understand the biochemistry involved – or don’t want to reject a paper from a friend) and recommend acceptance. The editor sees the reviewers recommendation and accepts the paper.
And this is simply the tip of the iceberg as regards the faults in the current peer review process – probably every single working scientist will have horror stories to tell about their experiences in this regard.
Oh, and don’t miss the graphic that Gizmodo used in their feature on it. lulz (Story was also cross-posted to Jezebel)
http://gizmodo.com/5708875/why-are-some-scientists-attacking-nasas-new-life-discovery
More detailed analysis by Carl Zimmer and thirteen other scientists can be found here:
http://blogs.discovermagazine.com/loom/2010/12/08/of-arsenic-and-aliens-what-the-critics-said/
This is a fascinating insight into scientific criticism.
Illustrates nicely what we were talking about just the other day. Scientists need humility – if you make some unsupported claims you get shot down; in this case very quickly. Is this what Feynman meant by “pseudoscientists”?
@#8, re the graphic for the Gizmodo story… too funny!
Sigmund – really?! Journals get non-experts to review articles? I’m staggered. I’m also naïve – I simply assumed the opposite. I knew mainstream media do that, sometimes as it were on principle (or to put it another way, in order to kick off a shitstorm, as when the NY Times had Leon Wieseltier review Dan Dennett’s book) – but I thought that was one way journals differed from mainstream mass media.
It’s easy to think of good reasons to include non-expert reviewers: for instance to see if the article is comprehensible to non-experts. But it’s certainly not easy to think of good reasons to exclude expert reviewers.
As for some kind of peer-review reform, I found this comment interesting from Rosie Redfield’s blog:
https://www.blogger.com/comment.g?blogID=32079676&postID=188341271861072078
All reviewers for scientific papers will be “experts”. The trouble is that they may not be experts in the appropriate topic. It’s common in medical research for reviewers to be either clinicians or scientists – usually molecular biologists. For an individual paper, however, the actual ratio of molecular biology to clinical data may be something like 9:1
In that scenario loading the reviewing panel with clinicians will result in a set of reviewers that may not be able to understand some basic flaws in the molecular biology of the study.
In the case of the arsenic paper it looks like something similar happened. The time from submission to acceptance was 1 month – astonishingly short for a high ranked journal. While it is possible to get an initial review done within that timespan, the normal result is the request that one undertakes a series of controls and additional experiments – something that usually takes months to complete. The 1 month timespan for the arsenic paper suggests that those reviewing it did not realize that many controls were missing- leading me to the conclusion that the reviewing panel was skewed in some way.