JMIR Publications

ABSTRACT

Background:

Patients find technology tools to be more approachable for seeking sensitive health-related information, such as reproductive health information. The inventive conversational ability of artificial intelligence (AI) chatbots, such as ChatGPT, offers a potential means for patients to effectively locate answers to their health-related questions online.

Objective:

ChatGPT and Google Search were evaluated to compare their ability to offer accurate, effective, and current information regarding proceeding action after missing a dose of oral contraceptive pill (OCP).

Methods:

A sequence of eleven questions, mimicking a patient inquiring about action to take after missing a dose of OCP, were input into ChatGPT as a cascade, given the conversational ability of ChatGPT. The questions were input into four different ChatGPT accounts with the account holders being of various demographics to evaluate potential differences and biases in the responses given to different account holders. The leading question, “what should I do if I missed a day of my oral contraception birth control?”, alone was then input into Google Search, given its non-conversational nature. The results from the ChatGPT questions and the Google Search results to the leading question were evaluated on their readability, accuracy, and effective delivery of information.

Results:

The ChatGPT results were determined to be at an overall higher grade reading level, longer reading duration (Table 2), less accurate, less current, and a less effective delivery of information. In contrast, the Google Search resulting answer box and snippets were at a lower grade reading level, shorter reading duration, more current, able to reference the origin of the information (transparent), and provided the information in various formats in addition to text.

Conclusions:

ChatGPT has room for improvement in accuracy, transparency, recency, and reliability before it can equitably be implemented into healthcare information delivery and provide the potential benefits it poses. However, AI may be used as a tool for providers to educate their patients in preferred, creative, and efficient ways, such as using AI to generate accessible short educational videos from healthcare provider-vetted information.