How to write a high-quality peer review – JMIR Publications - Support - KB

Background

A high-quality peer review can provide authors with thoughtful and respectful criticisms of the work, with the aims of improving the quality of the manuscript and its potential contributions to existing scientific knowledge. Additionally, such a peer review encourages authors to produce high-quality research and can help maintain scientific integrity and authenticity (including validity, significance, and originality of a study).

This article is intended to provide general guidance and suggestions to peer reviewers on what JMIR Publications considers to be a high-quality peer review. A high-quality peer review report on a scientific paper typically contains several key components that help evaluate the quality and validity of the research.

Content:

Anatomy of a high-quality peer review
Common pitfalls to avoid
Review Quality
Generative AI support in peer review
Related Important Links
Reviewer Recognition

Anatomy of a high-quality peer review

While this is an extensive list, peer reviewers may not necessarily be required or expected to address every point for any given article type that is under review. However, many of these principles are best practices, and the editor may ask peer reviewers to clarify elements that they may deem important or vital to the peer review of a manuscript.

Here are some important elements commonly found in a comprehensive peer review report:

Summary or Overview: (optional) Begin the report with a concise summary of the paper's main findings, objectives, significance, strengths and limitations. This helps provide an overview for the editor and authors.
For Original Papers and Literature Reviews: (What are the article types for JMIR journals?)
- Abstract: For structured abstracts, is the content of each section sufficiently detailed enough to understand the complete work within the 450-word limit?
- Introduction: Evaluate the clarity and coherence of the introduction, including the presentation of relevant background information, the research question (or aims, hypotheses) and the significance of the study.
  - Literature overview and references: Evaluate the authors' knowledge of relevant prior research and their appropriate citation and referencing of previous studies. Identify any major gaps or omissions in the literature review.
- Methods: Assess the methodology used in the research, including the study design, data collection methods, sample size, statistical analyses, and any potential limitations or biases. Does the manuscript reporting adhere to a relevant reporting guideline for the methods used? (What reporting guidelines should I follow for my article?) Are Ethical Considerations or Approvals included? ((for authors) Institutional Research Board / Research Ethics Board and Informed Consent )
- Results: Evaluate the presentation and accuracy of the results. Check if they align with the research question, whether the statistical analyses are appropriate, and if the interpretation of the results is logical and supported by the data.
- Discussion: Do the authors cover Principal Findings (and Strengths), Comparison to Prior Work, Limitations, and Conclusion? Assess the coherence of the discussion, the authors' interpretation of the results, and their ability to draw meaningful conclusions. Identify any unsupported claims, overgeneralizations, or opinions (e.g., recommendations, lessons learned, or otherwise) that are not based upon empirical findings or cited existing literature. Check whether the limitations of the study are properly acknowledged in the discussion section. All statements in the Discussion/Conclusions appropriately match the conclusions that can be drawn, i.e. ensure that conclusions can be drawn based on the methodology and findings of the study. (This applies to both the Abstract and the manuscript text.)
Clarity, organization, readability: Assess the clarity, organization, and overall flow of the paper. Identify any sections that need improvement or restructuring. Comment on the use of headings, subheadings, and other structural elements.
- For Viewpoint manuscripts, these points can be especially important to clearly state the opinion or perspective taken by the authors, including key messages and supporting evidence for the argumentation included. If the Viewpoint instead describes an experience or lessons learned, the key aim and messages should be clearly stated and supported by specific examples from the experiences or observations of of the authors. Sufficient citation of existing scientific evidence would still be important to help place such experiences in context.
Language and style: Although peer reviewers may evaluate the quality of writing, including grammar, spelling, and sentence structure, this should not be the sole focus of the peer review report. Provide suggestions for improvement if necessary, and specifically in instances where the language quality interferes with the clear communication of the scientific work. Regarding style, ensure that the language is concise and accessible to the intended audience. JMIR Publications offers an Extended Copyediting Service when needed after a manuscript is accepted to help refine the English language.
Ethical Considerations: Identify any potential ethical concerns, such as plagiarism, conflicts of interest, or issues related to human or animal subjects. Comment on the ethical soundness and compliance of the study.
Recommendations: Offer constructive feedback and specific recommendations for improvement. Highlight any major revisions that should be made, and suggest additional experiments or analyses if needed.
Confidential Comments: Peer reviewers may provide optional confidential comments to the editor.

Important: Although some peer reviewers may opt to use available generative artificial intelligence tools (E.g., GPT-4, etc) to assist in their process of peer reviewing manuscripts, we expect peer reviewers to disclose usage of such tools in their peer review reports. This is similar to the JMIR Publications policies for authors (How should the "Acknowledgments" section be formatted?) because ultimately, the peer reviewer is accountable for the submitted peer review report and comments.

Common pitfalls to avoid

Lack of specificity: A poor-quality review report often lacks specific and detailed comments. It may contain vague statements or generalizations without providing specific examples or evidence to support the reviewer's critique or suggestions. On the other hand, excessive focus on copyediting related comments is not recommended.
- Failure to Consider the Paper's Context: A subpar review may not take into account the specific aims, scope, or target audience of the journal or conference for which the paper is submitted. It may overlook the relevance or significance of the research within the broader scientific community.
- Inadequate assessment: The reviewer may fail to critically evaluate the research methodology, results, or conclusions. They may overlook important flaws, errors, or inconsistencies in the study, or fail to identify gaps in the literature or limitations of the research. If the peer reviewer does not have the expertise to assess certain components of the study, this is acceptable and should be disclosed in Confidential Comments to the Editor.
- Insufficient evaluation of ethical considerations: A poor-quality review report may overlook potential ethical issues, such as plagiarism, conflicts of interest, or inadequate protection of human subjects. Ethical concerns should be carefully evaluated and addressed in a review report. (for authors) Institutional Research Board / Research Ethics Board and Informed Consent)
- Inconsistent or incoherent feedback: The feedback provided in a subpar review report may be inconsistent, contradictory, or incoherent. The reviewer may jump between different topics without maintaining a logical flow or fail to provide a clear overall assessment.
Lack of Constructive Feedback: Focusing only on identifying problems without offering specific recommendations or solutions to address them is a missed opportunity to provide constructive feedback or suggestions for improvement.
- Bias, prejudice, or judgmental language: Use of neutral language is strongly advised. A reviewer may demonstrate bias or prejudice by favoring or opposing certain ideas, authors, or research groups without sound scientific reasoning.
- Unprofessional, rude, or offensive language: The use of unprofessional language, personal attacks, or offensive comments undermines the professionalism and collegiality expected in the peer review process. A review report should focus on the scientific content and should refrain from using derogatory or inflammatory language.
Unethical peer review behaviors:
- Adherence to ethical best practices: Peer reviewers are expected to adhere to ethical best practices in peer review. Potential misconduct will be investigated and handled according to COPE guidance.
- Requests for self-citation: Suggesting citation of one's own published manuscripts when those manuscripts are irrelevant to the manuscript under review is an example of citation manipulation and can be considered scientific misconduct. JMIR Publications adheres to COPE guidance on such matters (COPE: Citation Manipulation) and peer reviews may be removed from consideration in such situations.
- Appropriation of author material: Suspected cases of reviewer appropriate of author material will be investigated and handled according to COPE guidance on appropriation of authors' ideas or data.
Lack of confidentiality: A poor-quality review report may breach the confidentiality of the peer review process by disclosing the reviewer's identity (What does the peer-review process at JMIR journals look like?), or sharing the content of the review report with unauthorized individuals. Peer reviewers are accountable for maintaining privacy and confidentiality of the peer review process.

Review Quality

Peer review reports are rated on a scale of 1-5 in their quality. Rating definitions are provided here.

Score: 5	Summary: Exemplary & Definitive
Outstanding review that sets a high standard for rigor, quality, clarity, and professionalism. Quality of Analysis: Exceptionally detailed, scientifically rigorous, and comprehensive; offers deep insight grounded in relevant literature. Constructiveness: Provides highly specific, actionable recommendations that meaningfully improve the manuscript. Ethics & Professionalism: Thorough ethical evaluation (IRB/consent, privacy, inclusivity); exemplary objectivity and responsibility. Aligns fully with JMIR Publications policies on all applicable ethical policies for the authors’ work and adheres to highest standards of ethical peer review conduct. Utility: Fully informs and strengthens the editorial decision.
Score: 4	Summary: Highly Useful & Thorough
Strong and well-reasoned review that significantly aids author revisions and editorial decision-making. Quality of Analysis: Detailed and thorough; critiques are clearly justified, often with supporting references or justification. Constructiveness: Offers practical, actionable suggestions for improvement. Ethics & Professionalism: Identifies clear ethical issues in the reviewed work; maintains confidentiality and neutral, professional language and adheres to standards of ethical peer review conduct. Utility: Dependable input for editors and authors.
Score: 3	Summary: Somewhat Useful & Sufficient
Acceptable review that provides the essentials but lacks depth or balance. Quality of Analysis: Moderately detailed but uneven; some sections are thoroughly addressed while others are overlooked. Constructiveness: Identifies major issues but provides limited, generic, or broad sweeping recommendations. Ethics & Professionalism: Addresses ethical issues in the work in a general way and objectively; may overlook elements such as potential biases or inclusive language. Sufficiently adheres to standards of ethical peer review conduct and language use. Utility: Sufficient for editorial purposes, especially when the manuscript’s strengths/weaknesses are obvious.
Score: 2	Summary: Weak or Insufficient
Underdeveloped review that offers minimal value; the review does not support editorial decision-making at all. Quality of Analysis: Superficial or narrowly focused; lacks evidence-based critique or engagement with key scientific elements. Constructiveness: Feedback is vague or generic, with little practical guidance. Ethics & Professionalism: May neglect ethical review or methodological assessment, and/or may overlook aspects of ethical peer review conduct, e.g., using biased or non-neutral language. Utility: Provides limited support for the editorial decision.
Score: 1	Summary: Unsuitable or Unprofessional
Unacceptable review that fails to meet basic standards of rigor or conduct. Quality of Analysis: Inaccurate, careless, or largely absent critique. It seems like the reviewer did not read the paper. Constructiveness: Offers no meaningful recommendations. Ethics & Professionalism: Contains inappropriate language, ethical breaches (e.g., undisclosed AI use, confidentiality violations), or citation manipulation. Utility: Provides no usable input; the reviewer should generally not be invited again.

Generative AI support in peer review

Read this policy before considering generative AI use to support peer review.

There are key principles and what they mean for you as a peer reviewer:

Accountability

You are accountable for final submitted peer review comments, whether or not generative AI was used.

The quality and content of the peer review is held to usual high standards, as described above for a high-quality peer review.

Do not engage in AI plagiarism, i.e., do not claim credit for the production of statements that were generated by AI.

Confidentiality

You must maintain confidentiality of the manuscript and materials entrusted to you for peer review.

Many commercially available generative AI tools, including large language models, reuse prompted information for training. Peer reviewers who enter confidential information into commercial tools are potentially breaching confidentiality, which is an unethical peer review practice.

Transparency

If you used generative AI to assist in peer review, you are required to disclose use completely and precisely. The author(s) and editor have the right to transparency of peer reviewers' use of generative AI as a part of the revision and decision-making process.

For example, a bare minimum statement includes the following: “I conducted this review with the assistance of [ProductName, Version, from CompanyName, Year].”

Sufficient detail must be provided so that an editor has a clear and complete understanding of the role of AI in peer review report generation. The handling editor may request the peer reviewer to provide more detail, for example, the prompts used and the responses generated by AI.

Peer reviewers are required to adhere to the policies with regards to peer reviewer accountability, transparency, and maintenance of confidentiality of any content that is generated by AI and incorporated into a peer review report submitted for any JMIR Publications journal.

Peer reviewers may also wish to carefully read the editorial policy to inform themselves of what JMIR Publications expects of authors when they use and disclose use of generative AI in producing parts of a manuscript submitted to a JMIR Publications journal.

**Peer reviewers are advised to carefully read the full published editorial policy on generative AI usage to ensure their adherence to the required appropriate statements and/or supplementary materials.**

Because of the rapidly evolving nature of AI technologies, related policies, regulation, and best practices, JMIR Publications may periodically modify this editorial policy. It is the responsibility of the peer reviewers to adhere to these principles and practices provided.

Reviewer Recognition

Footnote: Article sections on Anatomy of a high-quality peer review and Common pitfalls to avoid were originally, in part, generated using ChatGPT-3.5 (OpenAI), however, it was edited and enhanced by friendly humans from JMIR Publications staff.