JMIR Publications

ABSTRACT

Background:

ChatGPT may act as a research assistant to help organize the direction of thinking and summarize research findings.

Objective:

However, few studies have examined the quality, similarity, plagiarism, and accuracy of the abstracts generated by ChatGPT when researchers provide full-text basic research papers.

Methods:

We selected 30 basic research papers from Nature, Genome Biology, and Biological Psychiatry. Excluding abstracts, we fed the full text into the ChatGPT model (using ChatPDF) and prompted it to generate abstracts with the same style as those in the original papers. Eight experts were invited to evaluate the quality of these abstracts (Likert scale:0–10) and identified which abstracts were generated by ChatGPT using a blind approach. These abstracts were also evaluated for similarity, plagiarism, and accuracy of the AI content.

Results:

The quality of the ChatGPT-generated abstracts was lower than that of the actual abstracts (Likert scale of quality, mean (standard deviation), 4.72 (2.09) vs 8.09 (1.03); p<0.001), and the quality difference was large in the unstructured format (mat (-4.33; -4.79 to -3.86) and small in the four-subheading structured format (-2.33; -2.79 to -1.86). Among the 30 ChatGPT-generated abstracts, three showed wrong conclusions and ten were identified as AI content. Similarity (2.10%-4.40%) and plagiarism (5.01%-6.70%) were not high. The blinded reviewers achieved a 93% accuracy rate in guessing which abstract was written using ChatGPT.

Conclusions:

Using ChatGPT to generate a scientific abstract may not produce the problems of similarity and plagiarism when feeding real full-texts written by humans. However, the quality of the ChatGPT-generated abstract was sub-optimal, and the accuracy was not 100%.