In Search of Peer Reviewers

I found this article the other day in Science and thought it was relevant to our recent discussions about journals and the process of peer-review. It's kind of inside look at some of the problems that occur as some journals attempt to find people to review new articles and the attitude some members of the scientific community have towards doing this kind of work.
---
'As a past editor of Marine Mammal Science and a present associate editor of the Journal of Mammalogy, I have had great difficulty in lining up reviewers. Sometimes it takes 8 or 10 tries to find someone who will agree to review a paper. The typical excuse is "I'm too busy."

First I try the people who have published the most relevant and recent papers on the topic in question. Then I move down the range of choices. The temptation, and sometimes the need, is to turn to potential reviewers in less-related fields or those who are not so "busy" (i.e., are not producing much themselves). This inevitably leads to less-knowledgeable reviewers and often reviews of lesser quality, which of course complicates the editor's job and sometimes enrages the authors.

If an average acceptance rate of 50% is assumed, and if each paper needs at least two reviews, then each paper published represents at least four reviews. Following this logic, if you publish three or four papers a year, you should be doing at least 12 to 16 reviews. Anything less means that you are sloughing off the work to others who are perhaps less knowledgeable and capable than you in your specialty, and you should not be upset when someone reviewing a paper of yours "doesn't have a clue."

Doing a fair share of peer reviews should be a recognized and expected part of the job for scientific professionals; it should be written into the job descriptions of salaried scientists and be considered in evaluating junior faculty for tenure. The caution should be "Publish and review, or perish."'

How publishing of science articles has changed in the last 100 years

The Physical Review Letters is one of the most important journals in science. In a short essay that appeared last week, one of the former editors talks about some of the changes in science communication -- such as variations in peer review, the race for priority, the need to keep some stuff secret -- that have affected science publishing.

Essay: Physical Review Letters; Sam Goudsmit’s Vision

More on suppressed findings

Another story today about the need to publish "failed" studies, this time about cholesterol:

From: What That Cholesterol Trial Didn’t Show

The Vytorin makers dragged their feet on releasing the results, issued the findings in a press release rather than a medical journal and made a lot of people mad.

“Failed trials are important,” said Dr. Steven Nissen, chairman of cardiovascular medicine at the Cleveland Clinic. “The reason the medical community is upset about the delay in reporting this is that we learn as much from a study that fails as we do from a study that succeeds. People are not happy that we didn’t learn about this sooner.”

Notice that this quote also addresses the issue of "science by press release."

How complete is the peer-reviewed medical literature?

Many studies have suggested that there's a "positive effect" bias in publication -- if your study proves what you expected, it gets published. Otherwise, you set it aside. (This is sometimes called "the bottom drawer" effect, as well -- the unpublished studies get put into the bottom drawer of your desk.) One key question: is the bias unintentional (scientists don't publish negative results because they're not interesting), or is it actively harmful (because other scientists think that the research area is more successful than it really is)?

Another common concern is that the bias is actually deliberate, especially when it involves pharmaceutical-company-sponsored studies of drug effectiveness. That's what a new study claims:

Antidepressant Studies Unpublished - New York Times

What counts as "science by press conference"?

In May 2007, the Mars rover Spirit discovered silica-rich soils that suggest the possibility of microbial life on Mars. That finding has been widely publicized (see examples below). But, it has NOT been peer-reviewed, in the formal, publication sense. (Or so I'm told by colleagues who should know.) So, does it count as that evil "science by press conference"?

I personally heard about these findings in November 2007, while listening to a graduate student defend her research interests. It happens, of course, that I'm at the institution that leads the rover missions. But since I'm not a specialist, I didn't know that I was hearing "unpublished" results, and had the occasion arisen, I could have circulated the results more widely.

Then, a few weeks later, the mission PI (who is very experienced at dealing with the media) spoke about the results at a well-attended scientific meeting, the American Geophysical Union, where the media are often present. His comments were widely reported:

The BBC story
The Mars Society story

It's also true that NASA issued a press release about his talk, as did Cornell (and these press releases were widely posted on news sites around the web). But, given the frequent press coverage of AGU meetings, that's probably immaterial.

Many of the stories that appeared were written by well-known science writers. For example, a New York Times article by Kenneth Chang was reprinted widely (here, in the Portland Oregonian).

So, given all the warnings against science by press conference, how does this happen? Or does it mean (as Hilgartner 1990 suggests) that the charge of "science by press conference" is only used when someone wants to criticize another scientist, while "acceptable" popularizations are labeled as such?

How important is peer review?

SCIENCE AND COMMERCE: Who Owns Glycobiology? -- Kaiser 318 (5851): 734 -- Science

Read this article about a controversy between a set of scientists (glycobiologists) and a commercial company that claims the science supports its "nutraceuticals" (food supplements that might -- or might not -- make you healthier). Look especially for the role of publication and peer review in the dispute.

Superconductivity and the Y/Yb episode

Peers under pressure : Nature, Nature 413, 102-104 (13 September 2001) | doi:10.1038/35093252

As I mentioned in class today, the high-temperature-superconductivity case in the late 1980s involved a (probably deliberate) typographic error. The article above describes the incident in the context of a general discussion of peer review. The link below provides additional evidence.
[Letter to the editor elaborating on the Y/Yb episode, Nature 414, 393-394 (22 November 2001) | doi:10.1038/35106722]