Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.
Who will be affected?
Readers: No access to all 28 journals. We recommend accessing our articles via PubMed Central
Authors: No access to the submission form or your user account.
Reviewers: No access to your user account. Please download manuscripts you are reviewing for offline reading before Wednesday, July 01, 2020 at 7:00 PM.
Editors: No access to your user account to assign reviewers or make decisions.
Copyeditors: No access to user account. Please download manuscripts you are copyediting before Wednesday, July 01, 2020 at 7:00 PM.
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Leveraging Artificial Intelligence Large Language Models for Writing Clinical Trial Proposals in Dermatology
Megan Hauptman;
Daniel Copley;
Kelly Young;
Tran Do;
Joseph Durgin;
Albert Yang;
Jungsoo Chang;
Allison Billi;
Mio Nakamura;
Trilokraj Tejasvi
ABSTRACT
Background:
Large language models (LLMs) are becoming increasingly popular in clinical trial design but have been underutilized in research proposal development.
Objective:
This study compares the performance of commonly used open access LLMs versus human proposal composition and review.
Methods:
Ten LLMs were prompted to write a research proposal. Six physicians and each of the LLMs assessed 11 blinded proposals for capabilities and limitations in accuracy and comprehensiveness.
Results:
Chat GPT o1 was rated the most accurate and LLaMA 3.1 the least accurate by human scorers. LLM scorers rated Chat GPT o1 and Deepseek R1 the most accurate. Chat GPT o1 was the most comprehensive and LLaMA 3.1 the least comprehensive by human and LLM scorers. LLMs performed poorly on scoring proposals, and on average rated proposals 1.9 points higher than humans for both accuracy and comprehensiveness.
Conclusions:
Paid versions of ChatGPT remain the highest quality and most versatile option of available LLMs. These tools cannot replace expert input but serve as powerful assistants, streamlining the development process and enhancing productivity. Clinical Trial: n/a
Citation
Please cite as:
Hauptman M, Copley D, Young K, Do T, Durgin J, Yang A, Chang J, Billi A, Nakamura M, Tejasvi T
Leveraging AI Large Language Models for Writing Clinical Trial Proposals in Dermatology: Instrument Validation Study