Scientific research is a collaborative endeavour, and that collaboration doesn’t stop once the lab work is complete. Each paper is the result of multiple people and perspectives, reaching across the world to combine authors, reviewers, editors, and the manifold researchers who will go on to be inspired and influenced by the papers being submitted for publication today.

Detailed, expert advice from researchers active in the same field as the submitted paper is invaluable to assess whether or not that work is scientifically sound, what value it adds to the field, and what additional support or corrections may be needed to validate the authors’ interpretations and conclusions.

Thorough and constructively critical reviewer advice, when carefully considered and acted on by the paper’s authors, can improve the quality and accuracy of the presented content, immeasurably increasing the value of that paper to the scientific community. It can help a paper go from ‘unpublishable’ to ‘outstanding’.

Even the infamous Reviewer #2 plays an important role.

And that’s really the key to this whole endeavour: improvement. The right reports can not only improve the paper being assessed, but also influence the way its authors work from that day on. Peer review is far more than an editorial filter: it is a mechanism for learning and exchange.

Can’t We Just Use AI?

AI tools are making striking headway assessing the novelty and accuracy of a paper’s claims by comparing it to published works on the same topic. Development continues apace, but these tools are not quite ready to replace humans on the task.

While we wait for that milestone, there’s another important consideration: copyright. Publication offers an official record not only of what was done, but of who did it: authors have the right to be recognised and credited for their work.

Uploading someone’s confidential, unpublished results into an AI tool, which will often store that information and potentially spit out recognisable parts of it when asked similar questions by other users, has obvious consequences for the original authors and what we consider the official record.

As a reviewer, you are responsible for maintaining the strict confidentiality of any unpublished materials with which you are entrusted. Using AI tools to help generate your review could therefore be a breach of the expected ethics and of the trust placed in you when you’re invited to review a paper.

It could also harm your reputation: editors can often spot an AI-generated report and, though we may not say anything, we’ll probably be wary of any content coming from you thereafter.

For many reasons, it’s better to engage your analytical skills to assess the paper and prepare a report yourself, and you can follow these tips to help you do it.

Tip 1: Take Your Time

Assessing the work of your peers should not be undertaken lightly.

The degree to which a reviewer report is useful to the author and, by extension, the value added to that paper during the review process, is proportional to the depth of your analysis; the report must go beyond a superficial summary to highlight problems or inconsistencies, specify what’s missing, and guide the author toward any necessary additions and corrections.

A reviewer must also be able to compare the work in question to the published body of literature on the topic, to determine what new value the paper adds to that literature or to identify what’s required to convincingly present results which may contradict current trends or established theories.

A good report – one useful to both the author and the editor – must therefore clearly indicate a close evaluation of the text and data, and a familiarity and understanding of the work right from the big picture context down to the smallest details.

To do that job properly, you can’t afford to rush.

Tip 2: Be Specific

Even a report which carefully outlines all the paper’s strengths and weaknesses is not necessarily useful.

Just as the authors must provide sufficient evidence to support their claims, so too must you support your reviewer comments by referencing the exact parts of the manuscript or the data where you’ve observed something, or by citing relevant literature.

For example, it is not enough to say ‘the work is not novel’; be specific about what aspect is not novel and back that up with relevant references: ‘the same concept/material/method/etc has already been studied, see [citations]’.

Remembering that the ultimate goal is to guide the authors to improve their work, it is also necessary to propose what specific actions should be taken and what changes must be made. For example, rather than saying ‘this claim is unsupported’, indicate what is missing or incorrect: ‘to support the claim that …, I would need to see …’. Instead of ‘there is a problem with this analysis’, try stating exactly what you think may be wrong: ‘I believe … assumption is incorrect, leading to exaggerated values for …’

This is the key to ‘constructive’ criticism: consider what specific actions you might take if it were your paper, and detail them in your report.

Tip 3: Use a Clear Layout

There are many ways to present your assessment and recommendations, with three such possibilities described below. No matter which way you choose, it is always a good idea to use numbering. Not only can this assist you to better organise your thoughts, it also helps to prevent any point from being overlooked and makes it clear at a glance which point is being addressed in the author’s response.

Section by Section or Line by Line

Some reviewers like to present their thoughts in exactly the order they see things as they read the paper.

These comments may be grouped under the same subheadings as the paper itself (title, abstract, introduction, etc.), or they may be more granular, prefaced by the respective line number where the reviewer found something interesting or problematic.

This layout makes it very easy to see exactly which part of the paper needs to be addressed and whether or not each point was resolved in the revision, which is helpful for everyone involved.

It could also mean that the same concern is raised repeatedly throughout the report. This redundancy requires more mental filtering by the reader; where feedback for the same key point is split over different parts of the report, it is easier for the author or editor to miss something, and can generate an apparent emphasis on the concerns raised more frequently, rather than those which are the most important.

To get around this, a short summary of the major observations and expected impact of the paper is often placed at the end of the report.

Major and Minor Concerns

By far the most common layout that I see, this report divides the reviewer’s observations into two groups: things which will determine whether or not to proceed with publication (major concerns), and things which will only delay it (minor concerns).

Minor concerns might include presentation issues, problems with the language or clarity of expression, abbreviations not defined, or content that should be moved between the main manuscript and the supporting information. It may be as simple as a missing scale bar or incomplete reference information.

Major concerns relate to the completeness, accuracy, and quality of the work. They could include unscientific content, fundamental flaws in the concepts, methods, or data analysis, unsupported or incorrect statements and conclusions, or even suspected ethical problems. Presentation issues may be mentioned here if they are so numerous that they raise concerns about the paper preparation and seriously affect its overall quality and readability.

Reports using this layout often begin with a summary of the general impressions and expected impact of the paper, including the reviewer’s opinion on its suitability for the journal based on the overall novelty and quality. No final summary of key points is required.

This layout is easy to read and understand, but can sometimes tend more toward general observations rather than constructive details. Refer to Tip 2 to ensure that the wording of each point drives toward a solution rather than focusing only on the problem.

Key Concerns and Potential Solutions

Another way to lay out your report is to use key points as subheadings.

This could adopt the paper’s major claims and conclusions as subheadings, and assess their plausibility by considering what evidence would be needed, what evidence is presented, and how reliable and convincing you find those results and interpretations.

Alternatively, you might use the checklist items below or the individual concerns you noticed when reading the paper as subheadings, with your comments underneath those headings indicating which parts of the paper are affected and what would be needed to resolve them.

This is a very solution-focused style of report, but you need to be careful that you don’t get bogged down in the details and miss something important, and you may end up with a ‘miscellaneous’ section for minor issues at the end.

Tip 4: Complete the Checklist

What things should your assessment cover?

• It is helpful to provide a big-picture opinion on the more editorial aspects: novelty, general quality, and the prospective impact or value to the community. What novelties or insights will this paper add to the current body of literature? How great an advance is it, beyond what has already been done? How big is the audience which will be interested in it?
• Is the background/context sufficiently clear and complete to explain the motivation and goals of the work? Does the work actually address the stated challenges?
• Is the work scientifically and technically sound? Consider whether or not the statements, concepts, methodology, and interpretations/analyses seem correct. Do the data appear to have any problems, or do different pieces of evidence contradict one another?
• Is the dataset complete? Does it tell a complete story, or is anything missing which you consider required proof of the authors’ claims or which is standard for reporting in your field?
• Similarly, are all conclusions fully supported? If not, which ones need additional support, and what exact evidence is missing?
• Is the reference list comprehensive, unbiased, and relevant? Does it provide a true picture of the current state of the field? (Keep in mind that any references you suggest should also be directly relevant to the point you are addressing and representative of the literature for the field, without bias toward any one group. Suggesting many of your own papers, especially without clarifying how they will help to address a particular concern about the paper, may be considered an attempt at citation manipulation.)

This list isn’t exhaustive, but it covers the bigger questions that should be in your mind when preparing your report.

Tip 5: Write the Report You’d Want to Receive

Reviewers are also authors: you also hope that others will take the time to review your works carefully and constructively, to help you improve what you do.

Therefore, once you have finished drafting your report, take another look and consider:

• Is the language neutral and objective?
• Is the layout clear?
• Has it addressed the whole checklist?
• Does it provide detailed, actionable items for each concern that you’ve raised?
• Overall, is this report just an opinion to the editor, or will it help the authors improve their work – whether or not you think it is ready or suitable for publication at this particular journal?

Remember that you are writing your report to other human beings, with similar needs and motivations to yourself. Keep it professional and, any time you’re not sure, let this thought guide you: would you find it useful, if you were the author?

These are just one editor’s tips. Want even more? Wiley Author Services provides additional materials to assist you with peer review activities: https://authorservices.wiley.com/Reviewers/journal-reviewers/becoming-a-reviewer.html/peer-review-training.html

Featured image from Kaitlyn Baker via Unsplash.