JMIR Publications

ABSTRACT

Background:

Medical students may increasingly use Large Language Models(LLM) in their learning. ChatGPT is an LLM at the forefront of this new development in medical education, with the capacity to respond to multi-disciplinary questions.

Objective:

The present study evaluated the ability of ChatGPT 3.5 to answer the Indian Undergraduate Medical Examination in the subject of community medicine and compared ChatGPT scores with the scores obtained by the students.

Methods:

The study was conducted at a public-funded medical college in Hyderabad, India. The study was based on the internal assessment examination conducted in January 2023 for students in the MBBS Final year Part-I, which included 40 questions from the community medicine syllabus. The same questions were administered as prompts to ChatGPT 3.5, and the responses were recorded. Apart from scoring ChatGPT responses, the two independent evaluators explored the responses to each question under three sub-domains to further analyse their quality: relevancy, coherence, and completeness.

Results:

ChatGPT 3.5 scored 72.3% in Paper I and 61% in Paper II. The mean score of the 94 students was 43% in Paper I and 45% in Paper II. The responses of ChatGPT 3.5 were also rated to be satisfactorily relevant, coherent, and complete for most of the questions (>80%).

Conclusions:

ChatGPT 3.5 might have substantial knowledge to understand and answer the Indian medical undergraduate subject of community medicine. ChatGPT may be introduced to students to enable the self-directed learning of community medicine in the pilot mode under faculty oversight, as it is still in the initial stages where its potential and reliability of medical contents from Indian context needs to be explored, satisfactorily.