JMIR Publications

ABSTRACT

Background:

Free-text clinical data that is unstructured and narrative in nature can provide a rich source of patient information. However, the information contained within routinely collected health data is typically captured as free-text, and extracting research quality clinical phenotypes from these data remains a challenge. Manually reviewing and extracting clinical phenotypes from free-text patient notes is a time-consuming process and not suitable for large scale datasets. On the other hand, automatically extracting clinical phenotypes can be a challenging task due to medical researchers lacking gold-standard annotated references and other purpose-built resources including software. Recent large language models (LLMs) have the ability to understand natural language instructions (prompts) which helps them adapt to different domains and tasks without the need for specific training data. This makes them suitable for clinical applications, though their use in this field is still limited.

Objective:

In this paper, we analyse how large language models (LLMs) and in-context-learning techniques can be utilised for text classification in a real-world medical research scenario, where only a few training examples are available. For this, we sought to develop a pipeline, based on the “few-shot” learning framework, for extracting information from free-text clinical summaries derived from discharge patient notes and clinical interviews in a cohort of 1121 individuals with severe mental illnesses. Our main aim was to identify those with confirmed or strongly suspected comorbid intellectual disability (ID). A secondary aim was to take advantage of complementary and independent information from this cohort, in the form of a high-quality genomic dataset. For this we designed a proof-of-concept study to assess whether the individuals identified by our LLM-based models as having ID were also carriers of genetic variants known to confer risk of ID in the general population.

Methods:

We explore novel approaches for performing classification in free-text patient summaries, based on using an intermediate Information Extraction step and human-in-the-loop approach in order to maximise the performance of in-context-learning techniques and LLMs. We perform our experiments with two models, Flan-T5 and LLaMA, and a real-world research dataset with free text clinical information from individuals diagnosed with severe mental illness. For our genetics-based proof-of-concept experiment, we use a published dataset where de novo mutations have been called. On this, we perform rare variant association analyses through Firth’s penalised likelihood approach, a logistic regression framework appropriate for sparse data.

Results:

Results show that a two-stage approach consisting of information extraction and manual validation can be effective for identifying individuals with suspected intellectual disabilities within free-text patient information, only needing a single training example per classification label. This approach was further validated by demonstrating that individuals classified by LLMs to have suspected intellectual disabilities were significantly enriched for rare genetic variants in genes associated with intellectual disability.

Conclusions:

Recently developed LLMs and in-context learning techniques combined with human-in-the-loop approaches, can be highly beneficial for extraction and categorisation of information from free-text clinical data when there is lack of training data. In this proof-of-concept study, we show that LLMs can be used to identify individuals with a severe mental illness who also have suspected intellectual disability, which is a biologically and clinically meaningful subgroup of patients. This showcases one of the applications of our method and suggests broad uses for supporting research on clinical records.