When you receive an invitation to peer review, you should be sent a copy of the paper's abstract to help you decide whether you wish to do the review. Try to respond to invitations promptly - it will prevent delays. It is also important at this stage to declare any potential Conflict of Interest.
The structure of the review report varies between journals. Some follow an informal structure, while others have a more formal approach. "Number your comments. " (Jonathon Halbesleben, former Editor of Journal of Occupational and Organizational Psychology) Informal Structure Many journals don't provide criteria for reviews beyond asking for your 'analysis of merits'. In this case, you may wish to familiarize yourself with examples of other reviews done for the journal, which the editor should be able to provide or, as you gain experience, rely on your own evolving style. Formal Structure Other journals require a more formal approach. Sometimes they will ask you to address specific questions in your review via a questionnaire. Or they might want you to rate the manuscript on various attributes using a scorecard. Often you can't see these until you log in to submit your review. So when you agree to the work, it's worth checking for any journal-specific guidelines and requirements. If there are formal guidelines, let them direct the structure of your review. In Both Cases Whether specifically required by the reporting format or not, you should expect to compile comments to authors and possibly confidential ones to editors only.
Following the invitation to review, when you'll have received the article abstract, you should already understand the aims, key data and conclusions of the manuscript. If you don't, make a note now that you need to feedback on how to improve those sections. The first read-through is a skim-read. It will help you form an initial impression of the paper and get a sense of whether your eventual recommendation will be to accept or reject the paper.
While you should read the whole paper, making the right choice of what to read first can save time by flagging major problems early on.
Editors say, "Specific recommendations for remedying flaws are VERY welcome."
Examples of possibly major flaws include:
If experimental design features prominently in the paper, first check that the methodology is sound - if not, this is likely to be a major flaw.
You might examine:
Major Flaws in Information
If methodology is less of an issue, it's often a good idea to look at the data tables, figures or images first. Especially in science research, it's all about the information gathered. If there are critical flaws in this, it's very likely the manuscript will need to be rejected. Such issues include:
If you find a major problem, note your reasoning and clear supporting evidence (including citations).
After the initial read and using your notes, including those of any major flaws you found, draft the first two paragraphs of your review - the first summarizing the research question addressed and the second the contribution of the work. If the journal has a prescribed reporting format, this draft will still help you compose your thoughts.
The First Paragraph
This should state the main question addressed by the research and summarize the goals, approaches, and conclusions of the paper. It should:
The Second Paragraph
This should provide a conceptual overview of the contribution of the research. So consider:
After drafting these two paragraphs, you should be in a position to decide whether this manuscript is seriously flawed and should be rejected (see the next section). Or whether it is publishable in principle and merits a detailed, careful read through.
Even if you are coming to the opinion that an article has serious flaws, make sure you read the whole paper. This is very important because you may find some really positive aspects that can be communicated to the author. This could help them with future submissions.
A full read-through will also make sure that any initial concerns are indeed correct and fair. After all, you need the context of the whole paper before deciding to reject. If you still intend to recommend rejection, see the section "When recommending rejection."
Once the paper has passed your first read and you've decided the article is publishable in principle, one purpose of the second, detailed read-through is to help prepare the manuscript for publication. You may still decide to recommend rejection following a second reading.
"Offer clear suggestions for how the authors can address the concerns raised. In other words, if you're going to raise a problem, provide a solution." (Jonathon Halbesleben, Editor of Journal of Occupational and Organizational Psychology)
Preparation
To save time and simplify the review:
You may also wish to consider:
Check the Language
Not every submission is well written. Part of your role is to make sure that the text’s meaning is clear.
Editors say, "If a manuscript has many English language and editing issues, please do not try and fix it. If it is too bad, note that in your review and it should be up to the authors to have the manuscript edited."
If the article is difficult to understand, you should have rejected it already. However, if the language is poor but you understand the core message, see if you can suggest improvements to fix the problem:
On Grammar and Punctuation
Your primary role is judging the research content. Don't spend time polishing grammar or spelling. Editors will make sure that the text is at a high standard before publication. However, if you spot grammatical errors that affect clarity of meaning, then it's important to highlight these. Expect to suggest such amendments - it's rare for a manuscript to pass review with no corrections.
A 2010 study of nursing journals found that 79% of recommendations by reviewers were influenced by grammar and writing style (Shattel, et al., 2010).
1. The Introduction
A well-written introduction:
Originality and Topicality
Originality and topicality can only be established in the light of recent authoritative research. For example, it's impossible to argue that there is a conflict in current understanding by referencing articles that are 10 years old.
Authors may make the case that a topic hasn't been investigated in several years and that new research is required. This point is only valid if researchers can point to recent developments in data gathering techniques or to research in indirectly related fields that suggest the topic needs revisiting. Clearly, authors can only do this by referencing recent literature. Obviously, where older research is seminal or where aspects of the methodology rely upon it, then it is perfectly appropriate for authors to cite some older papers.
Editors say, "Is the report providing new information; is it novel or just confirmatory of well-known outcomes?"
It's common for the introduction to end by stating the research aims. By this point you should already have a good impression of them - if the explicit aims come as a surprise, then the introduction needs improvement.
2. Materials and Methods
Academic research should be replicable, repeatable and robust - and follow best practice.
Replicable Research
This makes sufficient use of:
These are used to make sure observed trends are not due to chance and that the same experiment could be repeated by other researchers - and result in the same outcome. Statistical analyses will not be sound if methods are not replicable. Where research is not replicable, the paper should be recommended for rejection.
Repeatable Methods
These give enough detail so that other researchers are able to carry out the same research. For example, equipment used or sampling methods should all be described in detail so that others could follow the same steps. Where methods are not detailed enough, it's usual to ask for the methods section to be revised.
Robust Research
This has enough data points to make sure the data are reliable. If there are insufficient data, it might be appropriate to recommend revision. You should also consider whether there is any in-built bias not nullified by the control experiments.
Best Practice
During these checks you should keep in mind best practice:
If the research fails to reach relevant best practice standards, it's usual to recommend rejection. What's more, you don't then need to read any further.
3. Results and Discussion
This section should tell a coherent story - What happened? What was discovered or confirmed?
Certain patterns of good reporting need to be followed by the author:
Discussion should always, at some point, gather all the information together into a single whole. Authors should describe and discuss the overall story formed. If there are gaps or inconsistencies in the story, they should address these and suggest ways future research might confirm the findings or take the research forward.
4. Conclusions
This section is usually no more than a few paragraphs and may be presented as part of the results and discussion, or in a separate section. The conclusions should reflect upon the aims - whether they were achieved or not - and, just like the aims, should not be surprising. If the conclusions are not evidence-based, it's appropriate to ask for them to be re-written.
5. Information Gathered: Images, Graphs and Data Tables
If you find yourself looking at a piece of information from which you cannot discern a story, then you should ask for improvements in presentation. This could be an issue with titles, labels, statistical notation or image quality.
Where information is clear, you should check that:
You should also check whether images have been edited or manipulated to emphasize the story they tell. This may be appropriate but only if authors report on how the image has been edited (e.g. by highlighting certain parts of an image). Where you feel that an image has been edited or manipulated without explanation, you should highlight this in a confidential comment to the editor in your report.
6. List of References
You will need to check referencing for accuracy, adequacy and balance.
Where a cited article is central to the author's argument, you should check the accuracy and format of the reference - and bear in mind different subject areas may use citations differently. Otherwise, it's the editor’s role to exhaustively check the reference section for accuracy and format.
You should consider if the referencing is adequate:
Check for a well-balanced list of references that is:
You should be able to evaluate whether the article meets the criteria for balanced referencing without looking up every reference.
By now you will have a deep understanding of the paper's content - and you may have some concerns about plagiarism.
Identified Concern
If you find - or already knew of - a very similar paper, this may be because the author overlooked it in their own literature search. Or it may be because it is very recent or published in a journal slightly outside their usual field.
You may feel you can advise the author how to emphasize the novel aspects of their own study, so as to better differentiate it from similar research. If so, you may ask the author to discuss their aims and results, or modify their conclusions, in light of the similar article. Of course, the research similarities may be so great that they render the work unoriginal and you have no choice but to recommend rejection.
"It's very helpful when a reviewer can point out recent similar publications on the same topic by other groups, or that the authors have already published some data elsewhere." (Editor feedback)
Suspected Concern
If you suspect plagiarism, including self-plagiarism, but cannot recall or locate exactly what is being plagiarized, notify the editor of your suspicion and ask for guidance.
Most editors have access to software that can check for plagiarism.
Editors are not out to police every paper, but when plagiarism is discovered during peer review it can be properly addressed ahead of publication. If plagiarism is discovered only after publication, the consequences are worse for both authors and readers, because a retraction may be necessary.
8. Search Engine Optimization (SEO)
After the detailed read-through, you will be in a position to advise whether the title, abstract and key words are optimized for search purposes. In order to be effective, good SEO terms will reflect the aims of the research.
A clear title and abstract will improve the paper's search engine rankings and will influence whether the user finds and then decides to navigate to the main article. The title should contain the relevant SEO terms early on. This has a major effect on the impact of a paper, since it helps it appear in search results. A poor abstract can then lose the reader's interest and undo the benefit of an effective title - whilst the paper's abstract may appear in search results, the potential reader may go no further.
So ask yourself, while the abstract may have seemed adequate during earlier checks, does it:
Editors say, "Does the Abstract highlight the important findings of the study?"
If there is a formal report format, remember to follow it. This will often comprise a range of questions followed by comment sections. Try to answer all the questions. They are there because the editor felt that they are important. If you're following an informal report format you could structure your report in three sections: summary, major issues, minor issues.
Major Issues
Minor Issues
Your review should ultimately help the author improve their article. So be polite, honest and clear. You should also try to be objective and constructive, not subjective and destructive.
You should also:
Most journals give reviewers the option to provide some confidential comments to editors. Often this is where editors will want reviewers to state their recommendation - see the next section - but otherwise this area is best reserved for communicating malpractice such as suspected plagiarism, fraud, unattributed work, unethical procedures, duplicate publication, bias or other conflicts of interest.
However, this doesn't give reviewers permission to 'backstab' the author. Authors can't see this feedback and are unable to give their side of the story unless the editor asks them to. So in the spirit of fairness, write comments to editors as though authors might read them too.
Reviewers should check the preferences of individual journals as to where they want review decisions to be stated. In particular, bear in mind that some journals will not want the recommendation included in any comments to authors, as this can cause editors difficulty later - see Section 11 for more advice about working with editors.
You will normally be asked to indicate your recommendation (e.g. accept, reject, revise and resubmit, etc.) from a fixed-choice list and then to enter your comments into a separate text box.
Recommending Acceptance
If you're recommending acceptance, give details outlining why, and if there are any areas that could be improved. Don't just give a short, cursory remark such as 'great, accept'. See Improving the Manuscript
Recommending Revision
Where improvements are needed, a recommendation for major or minor revision is typical. You may also choose to state whether you opt in or out of the post-revision review too. If recommending revision, state specific changes you feel need to be made. The author can then reply to each point in turn.
Some journals offer the option to recommend rejection with the possibility of resubmission – this is most relevant where substantial, major revision is necessary.
What can reviewers do to help? "Be clear in their comments to the author (or editor) which points are absolutely critical if the paper is given an opportunity for revision." (Jonathon Halbesleben, Editor of Journal of Occupational and Organizational Psychology)
Recommending Rejection
If recommending rejection or major revision, state this clearly in your review (and see the next section, 'When recommending rejection').
Where manuscripts have serious flaws you should not spend any time polishing the review you've drafted or give detailed advice on presentation.
Editors say, "If a reviewer suggests a rejection, but her/his comments are not detailed or helpful, it does not help the editor in making a decision."
In your recommendations for the author, you should:
Remember to give constructive criticism even if recommending rejection. This helps developing researchers improve their work and explains to the editor why you felt the manuscript should not be published.
"When the comments seem really positive, but the recommendation is rejection…it puts the editor in a tough position of having to reject a paper when the comments make it sound like a great paper." (Jonathon Halbesleben, Editor of Journal of Occupational and Organizational Psychology)
Visit our Wiley Author Learning and Training Channel for expert advice on peer review.
Watch the video, Ethical considerations of Peer Review
