This is a tangential follow-up to yesterday's Global Warming presentations - specifically to the comment James Wilson made about his experience at the Climate Project training workshop where someone in his group completely misunderstood/misrepresented the peer-review process.
To paraphrase James, this person (who apparently thought him/herself an expert on the matter) described peer-review as the process whereby when a scientist submits their research for publication, other scientists (the peers) actually re-do all of the work, to replicate every experiment and observation, to be sure it works, before the paper passes peer-review! What a caricature of real peer-review that is. As James put it, it is hard enough to get the funding and the resources to do the original research in the first place! Imagine if we had to keep reproducing work that has already been done for "peer-review"! Science would grind to a halt, indeed.
On the other hand, later last night, I came across another take on peer-review by John Dennehy, the Evilutionary Biologist, writing about a recent proposal in PLoS Biology to incentivise peer-review in order increase the efficiency of the process - primarily turnaround time.What is most surprising to me is that many scientists shirk peer review when asked to perform it. I recall one colleague who suggested the following strategy: after acquiescing to a request for peer review, ignore it until after the deadline, until the editor writes asking where it is. Only then should you perform the task and submit your comments. The colleague reasoned, perhaps then the editors would leave you alone and not request future reviews from you. The strategy, in addition to being appalling and uncollegial, is particularly shortsighted. If the strategy spread, it eventually would be applied to your own submissions as well.
In a letter to PLoS Biology, Marc Hauser and Ernst Fehr propose to incentivise the peer review process by punishing transgressors. Here editors would keep databases on when papers were sent to reviewers and when the reviews were returned. The late reviews would be punished accordingly: "For every day since receipt of the manuscript for review plus the number of days past the deadline, the reviewer's next personal submission to the journal will be held in editorial limbo for twice as long before it is sent for review."
Sound like it might work? Perhaps, but there are pitfalls:
Naturally the system is not without bugs. Who is punished when multi-authored papers are submitted? What happens if one of the authors is a timely reviewer and another is a slacker? Hauser and Fehr suggest penalizing only the primary corresponding author, but I think that might easily be gamed by making the least penalized author the correspondent. Also slackers could avoid penalties by refusing requests for reviews. Hauser and Fehr suggest penalizing them by adding a one-week delay their own next submission. However, this also penalizes those who turn down reviews because of time constraints and or because they feel unqualified to do the service.
In addition, I would particularly like to emphasize and elaborate upon that last point, about time constraints, because that really hits close to home for me, especially this semester! At a place like CSU-Fresno (non-Ph.D.-granting; lower-tier in terms of research universities), you always hear us complain about the workload, given how many classes we have to teach, and the increasing desire (which I wholeheartedly support) to increase our research activities and output. As a still-green faculty at such an institution, I am struggling to come to grips not only with balancing the teaching/research workload, but also a number of subtler constraints that don't often get recognized. While it is hard enough for us to compete with the big-boys of research universities in terms of attracting grants, one of the challenges is to even remain in circulation/contention among the big-boys at all, especially when transitioning from a strong research postdoc background. To me, participation in peer-reviews is one way to remain on the radar, so to speak, even as I try to keep up my research productivity while teaching 3-4 classes a semester. Which is why I have found it difficult to say no when an editor solicits a peer-review, even though I probably should decline more often. As a result I am guilty (even right now) of slow turnaround on some peer-reviews. And this is also tempering my own previous impatient reactions to prolonged peer-reviews.
So what would be my options under Hauser & Fehr's proposal, even with Dennehy's caveats? Of course, the first thing I have to try and do is to manage my limited time better, and meet deadlines (and likely sleep even less than I already do!). But if, given the multiple demands on us CSU-type faculty, the many hats we have to wear, I slip the deadline - I will get punished by encumbering a hold on my own future papers. If, on the other hand, I decline to review at all, I will also get whacked with a similar hold. Either way, my already slowing productivity will plummet further down the spiral, pushing me farther out of the orbit of higher-level research! Great - yet another constraint; just what we need. I don't want to shirk peer-review, nor deliberately push past deadlines as strategy to avoid being called upon. But given everything else we have to deal with, please don't make it any harder for us to participate in the scientific research community!
As Daniel Ebbole of Texas A&M wrote in a comment on the PLoS Biology site:
The time demands on potential reviewers have become enormous. The love of science is really the only motivation for agreeing to review. The funding situation is testing that love. No serious scientist has much time for reviewing, and certainly no time to respond to such a silly suggestion as punishing people who are too busy to turn in reviews on time. Of course, unless you are from Harvard, you have no time to write such a silly suggestion in the first place...
But then Ebbole goes on to suggest a monetary incentive - paying reviewers (or rather, their institutions) for timely reviews - in addition to punishment. I don't like that idea either. As Dennehy points out in another posting today, we scientists and our students are already paying three-times over for publishing/reading our own work, so adding to these publishing costs is not a smart idea. Instead, I'm inclined towards more open peer review, an experiment Nature tried out last year, but which apparently did not quite catch on.
As for faculty workloads at institutions other than Harvard and its ilk, how about changing the tenure-evaluation criteria so that the time spent on peer-review is not completely discounted when evaluating our job performance?!
Enough venting, I guess - I better get back to finishing up that review now!
Alternatively, I could always add endlessly to my CV through this golden opportunity for publishing in a different kind of "peer-reviewed journal"!
Read more...
... you know, the one that opened in Petersburg, Kentucky this Memorial Day, and has been all over the news? Well, you may visit that if you are in the region and want some amusement - or need to raise your blood pressure. Like that pathetic "fair-and-balanced" review in the New York Times did to me. Fortunately, there is an excellent antidote, from (who else) Dr. PZ Myers, who has waded through a flood of media coverage and blog commentary to compile a carnival: The Creation Museum. That's the one you should be visiting this sunday.
