Peer Review

Peer review. The human algorithm behind the world of scholarly and scientific publication. Some say it is a bit like democracy, a system full of problems, but the least worst we have. Some say it is like poetry, or love, or justice. Really quite hard to define or describe operationally. At least one thing can be said of peer review: it has become a cliché of academic life...

Victorian stairwells, dust-mottled sunlight, rough carpets and terra-cotta tiles, glass, steel, rubber coated wires, fans, electricity, fluorescent lights, silky heavy paper-back books, dehydrated plants, screens, backlights, the sinus burning smells of warming plastic, fan-heaters, cast-iron heaters, long-tables, too many chairs, white-boards, flip-charts, projectors, markers, chalk, so many black-boxes, spread-sheets, emails, recordings of one kind or another. The everyday material world of academia, walked over, passed by, touched, trained and tweaked, but hardly ever really noticed. And those are the fairly tangible materials. Imagine the intangibles – the multi-lingual oral cultures of talk and body-language and silence. In science and technology, conversation seems to tend towards argument, as if everything becomes a hypotheses to be disproved. A sprawling zero-sum game, a kind of maze…

***

In this context, “Humour is the way we cope,” Harun tells me. We’re chatting over a cup of tea in the kitchen of CONNECT at Dunlop Oriel House, on the fringes of Trinity College Dublin.  “When we make a joke, it's not just to let things slide better into another persons ear, it's to console ourselves as well.” During this conversation, we choose Peer-Review as our first seed for Stranger Fictions. We are in the mood for a bit of mischief, to perform a little institutional critique. What's important here?

“Science Communication,” Harun says, “is really important”.

“What do you understand by that?,” I ask.

“Both worlds - the scientific world, communicating scientific results to other scientists,” he says, “but more importantly communicating science to a wider public, because I see a huge gap there…”

“So in Science Communication,” I say, “...there’s one challenge of communicating science to scientists… and that’s part of the everyday of what you have to do [as a researcher]… all the peer-review stuff. And you are fairly well trained, probably, to do that? Would you say? Are you? Where are the gaps there..?”

“Well, the thing is you are trained by combat,” Harun says, setting up the foundational metaphor for our first stranger fictions session, “You are thrown into this ring, into this [academic] field to fight, and whatever you learn its your experience, or [what you learn from the] experience of others, and the lack of… you know you never really get a peer-review seminar… no journals or publications whatsoever give you an instruction of how to review a paper or give you bullet points or even some sort of guidelines. You are usually on your own because of a common knowledge assumption that you are ready to do it. So when you collect those bits and pieces of the peer-review skill on both sides [as writer and reviewer], it’s actually ‘monkey does what monkey sees’. And the trouble is our peer reviewers usually train us as writers, as authors. Once we get the peer-review from them, to state this and that in such a fashion, it’s more likely that tomorrow we are going to act [and write] just like them.”

Peer-review,
It's monkey-see-monkey-do.

Harun sends me some gently satirical papers on peer-review. One is called Referee's Report by Colin Adams [PDF]. The other is an essay he wrote himself in response to our conversation above. It's called Where do the Rejected Papers go to Die? [PDF]. If peer-review is monkey-see-monkey-do, then what if we showed ourselves something a little bit different? And what if we doubled-down on the combat metaphor? You want a fight? I'll give you fight! So, we decided to host our session on peer-review in a boxing club, to bring the clichéd logic of the academic game into plain sight, and to keep us on our toes.

Monkey-see-monkey-do,
Harun flips the Peer-review.

Instead of keeping the peer-review as a private correspondence between editors, authors and reviewers of an academic journal, Harun decided it was time to make the reviewers comments public. So he established JASPERS, the Journal of Astounding Peer Review. This journal aims to publish the very best peer-reviews of academic papers, but not the academic papers themselves. “Science doesn't know everything,” Harun says, “We trust science as a process, not as a result.” JASPERS shows the process of science at work. The first session of Stranger Fictions became the first meeting of JASPERS editorial board, with each participant in the session taking on the role of Peer-Reviewer.

***

A hesitant snow was beginning to fall around the boxing club, the air so cold each flake seemed suspended along with our breaths, marking time and movement like a Muybridge movie. Every oil and fan heater we could source we brought to bear on the warehouse space of the boxing club, and soon we had blown the fuse. Luckily, we had a warm-up planned.

Phase One:

Perform the combative metaphor of traditional academic culture by using Augusto Boal's 'Boxing Match' exercise (2002, pg. 141). Now how do we feel?

Phase Two:

Seeding Peer-Review: Harun welcomes us to the first editorial meeting of JASPERS, outlines the purpose of the Journal and our tasks as Peer-Reviewers.

Phase Three:

The Writing Prompt: JASPERS Issue 1

If you review the academic piece
You must show not tell;
If you review the fiction/poetry piece
You must deconstruct like hell.

Everyone is a reviewer.

Choose one of the five papers to review (excerpts from academic papers are laid out on the edge of the Boxing Ring, drawn from Computer Science, Science and Technology Studies, Ecocriticism, Poetry and Scientific Satire.)

Read and decide Yay or Nay for publishing.

You have approx. 20 mins to write your review.

Reviewers can appeal for more time.

Phase Four:

Reviewer's regroup with their reviews in hand and under the direction of Reviewer-in-Chief (Harun Siljak) each will read their review and offer their professional opinion of whether the article should be published, Yay or Nay.

Phase Five:

Conclude the session – step outside the fiction of JASPERS and offer any insights or responses.

***

***

When I first introduced Stranger Fictions to the EDGE researchers, I drew upon the story of six blind men and an elephant. It captures the challenge of communicating from different positions, points-of-view, experiences and knowledges. It also works as a metaphor for peer-review – a warning not to get bogged down in zero-sum argument, but to remain open to the plurality of experience and knowledge.

A group of blind men, who have never encountered an elephant before, meet one. They learn to conceptualize what an elephant is by touching it. They describe and define the elephant according to their own partial experience. As they share their experiences, they begin to fall into disagreement with each other – each claiming their own partial experience as the whole truth... [Read John Godfrey Saxe's poem from 1872 here).

Writer and Lawyer, Vanessa Place, reminds us that there are various versions of this parable, but there is a common conclusion: “that all are correct and yet each incomplete, that there is a truth and that this truth is beholden to the eye of the beholder, perforce limited, potentially dogmatic” (I’ll Drown My Book, 2012, pg. 445)

Peer-review. Monkey-see-monkey-do. The world makes sense with others. Take on other points of view, other models, see what aspects of the world become open for you...