JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) are usually built (pretrained) on general domain data. To adapt LLMs better to health care needs, it might be beneficial for health care institutions to create LLMs pretrained on their own EHR data. Nevertheless, there are concerns related to the training budget, the type of pretraining (pretraining from scratch vs. pretraining on top of pretrained LLMs), and the EHR database size.

Objective:

The objective of this manuscript is to address these concerns and to help decision-makers at health care institutions better understand opportunities and challenges for LLM development.

Methods:

We use published work on empirical experience in pretraining LLMs, and establish a relationship between the training budget, LLM size, pretraining database size and training time.

Results:

We found that pretraining a modest-sized LLM (13 billion parameters) from scratch requires around 800 GB of EHR data and costs 127,000 USD. For a 65 billion parameter LLM, the EHR requirement is 4000 GB, costing 3 million USD. In comparison, continued pretraining using 500 GB EHR on a 65 billion parameter pretrained LLM would cost 400,000 USD.

Conclusions:

Building private LLMs on EHR data can be challenging due to budget and database size limitations. Most health care institutions will not have the required resources to train their own LLMs from scratch, and even continued pretraining will be challenging for many. This study provides a framework for health care institutions when deciding about the deployment of LLMs in their systems.