JMIR Publications

ABSTRACT

Background:

Assessment of activities of daily living (ADLs) and instrumental activities of daily living (iADLs) is key to determining severity of dementia and care needs among older adults. However, such information is often only documented in free-text clinical notes within the electronic health record (EHR) and can be challenging to find.

Objective:

To develop and validate machine learning models to determine status of ADL and iADL impairments based on clinical notes.

Methods:

This cross-sectional study leveraged EHR clinical notes from Mass General Brigham’s Research Patient Data Repository linked with Medicare fee-for-service claims data from 2007-2017 to identify individuals aged 65 and older with at least one diagnosis of dementia. Notes for encounters both 180 days before and after the first date of dementia diagnosis were randomly sampled. Models were trained and validated using note sentences filtered by expert-curated keywords (filtered cohort) and further evaluated using unfiltered sentences (unfiltered cohort). Models’ performance was compared using area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC).

Results:

The study included 10,000 key-term filtered sentences representing 441 subjects (283 [64.17%] women, mean [SD] age 82.7 [7.9] years), and 1,000 unfiltered sentences representing 80 subjects (56 [70%] women, mean [SD] age 82.8 [7.5] years). AUROC was high for the best-performing ADL and iADL models on both cohorts (>0.97). For ADL impairment identification, the random forest model achieved the best AUPRC (0.89; 95% CI, 0.86-0.91) on the filtered cohort; the SVM model achieved the highest AUPRC (0.82; 95% CI, 0.75-0.89) for the unfiltered cohort. For iADL impairment, the Bio+Clinical BERT model had the highest AUPRC (0.76 filtered; 95% CI, 0.68-0.82; 0.58 unfiltered; 95% CI, 0.001-1.0). Compared with a keyword-search approach on the unfiltered cohort, machine learning reduced false positive rates from 4.5% to 0.2% for ADL and 1.8% to 0.1% for iADL.

Conclusions:

In this study, we demonstrated the ability of machine learning models to accurately identify ADL and iADL impairment based on free-text clinical notes, which could be useful in determining the severity of dementia.