JMIR Publications

ABSTRACT

Background:

Discourse analysis can reliably predict real-world communicative success, yet is rarely implemented clinically, due to challenges in manually generating stimulus-specific main concept inventories and scoring patient narratives.

Objective:

We evaluated (1) the quality of main concept inventories generated by a vision-language model relative to main concepts produced by healthy controls and (2) the value of automatically derived macrolinguistic and microlinguistic features extracted from the narrations for classification of persons with aphasia versus healthy controls and prediction of aphasia severity.

Methods:

Main concept inventories were generated for each movie clip with Gemini 2.5 Pro, a vision-language model, from the clip’s video frames, audio, and subtitles. Five sets of concepts were generated independently per clip and assembled into a single inventory. Features were extracted from narrative transcripts of all participant-clip combinations (51 healthy controls, 54 persons with aphasia, 8 clips), resulting in 688 narrations. Group comparisons used analysis of covariance with age as a covariate. Least absolute shrinkage and selection operator logistic regression with leave-two-out cross-validation classified the groups, and models restricted to the 15 microlinguistic or to the 4 macrolinguistic features were compared with the full model. Within the subsample of persons with aphasia, associations between each feature and the Western Aphasia Battery-Revised Aphasia Quotient (AQ) were estimated using partial correlations adjusting for age.

Results:

The main concept inventories generated directly from each clip’s video, audio, and subtitles using the vision-language model approximated the main concepts healthy controls produced. Persons with aphasia produced fewer unique main concepts than healthy controls (19% vs 46% of the inventory; d=−1.84, 95% CI −2.30 to −1.38). Group classification was excellent, with an area under the receiver operating characteristic curve (AUC) of 0.948 (95% CI 0.897-0.986) and an accuracy of 0.895 (95% CI 0.838-0.952). The AUC of neither restricted model differed significantly from the full model (microlinguistic only: AUC 0.914, P=.07; macrolinguistic only: AUC 0.908). More severe aphasia (lower AQ) was associated with greater distance of produced concepts from the main concept inventory (r=−0.79, 95% CI −0.87 to −0.66), lower main concept completeness (r=0.66), and more negative sentiment (r=0.65). In a hierarchical regression, distance to the centroid explained AQ variance beyond age, completeness, and the microlinguistic features associated with AQ (ΔR²=0.069, 95% CI 0.001-0.115; F1,41=16.62; P<.001).

Conclusions:

Main concept inventories generated directly from multimodal movie input using a vision-language model approximated the main concepts healthy speakers produced, making them adequate to serve as reference inventories for computerized main concept analysis. The automatically derived macrolinguistic features improved prediction of aphasia severity beyond the traditional microlinguistic measures alone. The new methodology opens a path to libraries of discourse assessment stimuli that are more realistic, diverse, and adaptable to varying cultural and clinical needs.