JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) have revolutionized Natural Language Processing (NLP) with their ability to generate human-like text through extensive training on large datasets. These models, including ChatGPT-3.5, GPT-4, and Bard, find applications beyond NLP, attracting interest from academia and industry. Students are actively leveraging LLMs to enhance learning experiences and prepare for high-stakes exams, such as the National Eligibility cum Entrance Test (NEET) in India.

Objective:

This comparative analysis aims to evaluate the performance of ChatGPT-3.5, GPT-4, and Bard in answering NEET-2023 questions.

Methods:

In this paper, we test the performance of ChatGPT 3.5, GPT-4, and Bard on pre-medical exam in India, NEET-2023. The questions of NEET were provided to these AI models, and the responses were recorded. Precision, recall, accuracy and F1 score were used to evaluate the performance of all three models.

Results:

GPT-4 demonstrated consistent superiority over Bard and ChatGPT-3.5 in all three subjects. Specifically, GPT-4 achieved accuracy rates of 72.5% in Physics, 44.44% in Chemistry, and 50.5% in Biology.

Conclusions:

The study's findings provide valuable insights into the performance of GPT-3.5, GPT-4, and Bard in answering NEET-2023 questions. GPT-4 emerged as the most accurate model, highlighting its potential for educational applications. The results underscore the suitability of LLMs for high-stakes exams and their positive impact on education. Additionally, the study establishes a benchmark for evaluating and enhancing LLMs' performance in educational tasks, promoting responsible and informed use of these models in diverse learning environments.