Peer Review Week 2023 Q4: AI-Generated Peer Review Reports

[Xiaohui Li]

For the manuscripts I contributed to recently, we have not received easy-to-spot-on AI-generated review reports. However, when I reviewed manuscripts and read the review report from other reviewers, I have encountered several times of AI-generated reports. One time it was so obvious that it even had a remark of “generated by ChatGPT” embeded in the texts, likely due to the use of ChatGPT through a third-party application. In all these cases, the comments were very general and vague, with no clear actions suggested. For example, if I ask chatGPT to write a review report for a “plant biology manuscript”, it will produce something like “Conclusions: The conclusions drawn are largely supported by the data, but the authors should avoid overgeneralizing their findings, especially considering the limitations mentioned in the discussion.” Yet, this is exactly what I saw from real review reports recently.

If reviewers are simply using AI to finish this “task”, without producing meaningful input tailored to a specific manuscript, then it will be a big problem for several reasons:

1. It hinders the generation of diverse opinions. It will all be AI’s “opinion”.

2. The process of peer review becomes meaningless. The authors, or anyone, could use AI to get the same comments.

3. There is usually no clear action items in the AI-generated reports, which does not help the authors to improve their manuscript.

4. It generates a feeling that the authors are now working for AI, and usually the same AI, instead of working with the editor and reviewers to improve the quality of science.

5. Many times people only feed the abstract to AI, due to character limit in the input. This again leads to vague review report.

I feel there is a need for a community-wide discussion about the use of AI in peer review, about what is accepted, and what should the editor/journal do when the reviewer’s report relies heavily on AI.

• This reply was modified 1 year ago by  Xiaohui Li.