11 dreams for the publishing debate — #3 get real credit for refereeing

Meanwhile back on the ranch...
Each new post will start from the top, so scroll down a little if you've read the previous one -- but also check out the first post for some motivation.

These are dreams. Some are realistic---perhaps just around the corner; others are way out there---basically crazy. Some will apply to everyone, others only to some. But all have diversity in mind, diversity in our expectations of who researchers are and what they do.

1. write fewer original-research papers

I know what you're going to say. But hear me out. This is at the core: to enable researchers to publish fewer "new result"-papers.

I believe all major problems brought up in the debate are, at the heart, caused by the immense increase in publications -- but not the global increase, the personal one. You have to publish far too much/big these days to get a half-decent position/grant. Increasing publication numbers did not increase the quality of research or, for that matter, the "quality of life" in our communities.

Instead, the massive inflation is killing us, devaluating everything we do as researchers. More papers mean that our papers are worth less. Having to publish more papers means we produce more research of questionable quality (unintentionally and otherwise). Especially young researchers have to publish for metrics instead of quality. Worst of all, evaluating researchers only by this steam-punkesque output means that the jobs go to people with this one singular skill -- writing the right kind of papers to please the editorial boards of the right kind of journals -- leading to an intellectual monoculture instead of diverse, stable, rich communities. In particular, the pressure works against women and minorities as they often start with and continue to face disadvantages in their careers that make it harder to produce the desired "output" in the desired time frame.

(If you're wondering why I'm not bashing "evil publishers". I don't think they are the problem -- we are. If you are happy with the inflation of papers-in-journals, then big publishers are what you need.)

2. get real credit for surveys, reviews and exposition

Surveys, reviews and expositions are research -- nothing more, nothing less. We live in a time where it is actually more important to write expository work. Why? Because we've optimized our production pipeline so well that there is no shortage of new (looking) results. Yes, you can argue about "relevance" (I won't, but I can't stop you). But if you're serious about all that "research for research's-sake"-talk, then you might notice that we've figured out how to educate people in the tens of thousands every year that do nothing else but churn out result after result.

As Felix aptly wrote: we need to move beyond "new" results. And this means to step back from them and, well, review. Surveys are the glue that holds our fields together, holds our communities together. We do it all the time -- if you read a paper thoroughly, you'll most likely write a review anyway. As we aggregate, work on larger projects or grant proposals these aggregate and easily become surveys and expositions. We need to make all of these public so that the community and the original author(!) can benefit from this enormous creative output that otherwise might only show up in a reading seminar or a journal club.

Mathematicians usually do even more. If we read a paper, we create our own version of the results. Mathematics need these just like music needs different interpretations. We need to give people who are "just" re-writing proofs the credit they deserve, the credit for keeping results alive, accessible and understandable to a wider audience than the original author and referee.

And with credit, I don't mean vouchers for some bookstore. Surveys must be first-class citizens! Surveying and reviewing other peoples work belongs on your CV, it is no less original than your "original research" and every department should have people that are exceptional at it, that are better at surveys, reviews or exposition than "new" research and add to the diversity of a department.

3. get real credit for refereeing

Refereeing is research, just like surveys and "new results". It might seem redundant after dream #2 (and in many ways I would say that's the goal), but given the importance of refereeing right now and the way we do it, it's separate.

Currently, good refereeing seems nearly impossible. With the increase of publications we can't even catch up with what's happening in our own narrowly focused interests -- how, then, can we expect to referee properly? Sam tells his own funny but sad story of refereeing honestly but the problem runs deep. Good referees are hard to find and when you find them, they could be writing a paper instead of refereeing -- and why should they not?

There are many ways we could improve refereeing. We can (and should) split up the different stages of refereeing (reference checking, correctness checking on different levels, opinion gathering etc), we should open pre- and post-publication peer review (and in-between peer review), we should use alternative methods to collect these reviews (instead of review journals that have at most one opinion per paper).

One key of refereeing is forming an opinion -- and voicing it. That's very hard, in particular for young and disadvantaged researchers who fear significant repercussions. But we don't need less people sharing their opinions on other researchers' output, we need more (whether they agree or not); it's a responsibility both to our own research communities and to a larger society.

But all of these are very unlikely to work, if we don't find a way to give referees the credit (and criticism) they deserve. Before we can praise referees we need tools to evaluate refereeing, both in current and future form. Above all, we need to be able to put refereeing on the CV, it's part of the research qualities every strong department should strive for.