JMIR Publications

ABSTRACT

Background:

Plain language summaries (PLSs) of Cochrane systematic reviews are a simple format for presenting medical information to the lay public, which is particularly important in oncology, where patients have a more active role in decision making. However, current PLS formats are significantly above readability requirements for the general population. Cost-effective solutions to this problem are still lacking.

Objective:

Plain language summaries (PLSs) of Cochrane systematic reviews are a simple format for presenting medical information to the lay public, which is particularly important in oncology, where patients have a more active role in decision-making. The aim of this study was to assess whether a large language model (e.g. Chat Generative Pre-trained Transformer (ChatGPT)) can improve the readability and linguistic characteristics of Cochrane PLSs about oncology interventions, without changing evidence synthesis conclusions, to save resources of medical researchers.

Methods:

The sample included 275 scientific abstracts (SAs) and corresponding PLSs of Cochrane systematic reviews about oncology interventions. ChatGPT-4 was tasked to make each scientific abstract (SA) into a PLS with these prompts: 1) Can you rewrite this Cochrane SA into a Cochrane PLS so that your text has a Simple Measure of Gobbledygook (SMOG) index of 6?; followed by 2) Can you rewrite PLS from prompt 1 so it is more emotional?; and 3) Can you rewrite this SA so it is simpler, easier to read and more appropriate for the lay audience? For each PLS produced by ChatGPT, we analyzed word count, level of readability (SMOG index), and linguistic characteristics using Language Inquiry and Word Count (LIWC) software, and compared it to the original PLS. We also analyzed the category of conclusiveness and compared it to the corresponding SA.

Results:

The PLSs generated by the first prompt (write at SMOG index 6) were the shortest and had the lowest SMOG level (median score (Md)=8.2, 95% confidence interval (CI)=8-8.4), compared to original PLSs (Md=13.1, 95% CI=12.9-13.4). The second prompt (revise above to more emotional) generated PLSs with lower readability (Md=11.4, 95% CI=11.1-12). PLSs produced by the third prompt (write simpler and easier) had the median SMOG index score of 8.7 (95% CI=8.4-8.8). Regarding linguistic characteristics, ChatGPT-generated PLSs from all three prompts used less analytical tone and more authenticity, clout, and emotional tone when compared to the original PLSs written by the authors. ChatGPT-generated PLSs did not change the conclusiveness category of the original SA.

Conclusions:

ChatGPT can be a valuable tool in simplifying medically related formats for lay audiences, such as PLSs. It may be better than researchers in simplifying complex medical texts. More research is needed, including about oversight mechanisms to ensure that the information is accurate and reliable.