Currently accepted at: JMIR Formative Research
Date Submitted: Nov 3, 2025
Date Accepted: Feb 27, 2026
This paper has been accepted and is currently in production.
It will appear shortly on 10.2196/87054
The final accepted version (not copyedited yet) is in this tab.
Conversational Platform for Multimodal Digital Biomarkers of Fatigue, Cognition, and Mental Health (Okaya): A Feasibility Study
ABSTRACT
Background:
Collection of multimodal data (video, audio, text) can yield digital biomarkers relevant to mental health, fatigue, and cognition. However, the feasibility and signal characteristics in operational populations remain underexplored.
Objective:
The objectives of this study are to (1) extracting an evidence-based library of vision, speech, and language biomarkers; (2) assess the feasibility of a fully remote conversational platform (Okaya) for collecting analyzable multimodal data; and (3) conduct preliminary signal checks for depression, fatigue, and cognition.
Methods:
Participants were recruited from the US Air Force and US Space Force. All participants completed the Okaya check-in which included a voice conversation with a large language model (LLM). A total of 66 visual, acoustic, and text features were extracted from each interaction between the participant and the LLM. For validation purposes, the study also collected a measure of depression (Patient Health Questionnaire; PHQ-9), fatigue (Cancer Fatigue Scale; CFS), and cognition (Trail Making Test; TMT). We evaluate the feasibility of the platform and evaluate correlation between the extracted features and the validated assessments.
Results:
8 unique participants contributed 62 sessions over a period from March 6, 2025 to August 6, 2025. The platform was deemed feasible for participants since 6 of the 8 participants opted to complete more than one session, and the three participants who provided feedback reported high overall experience and usability. From the data perspective, preliminary correlations produced significant results for multiple digital biomarkers: (1) pitch, volume standard deviation, volume slope, ARI complexity, Flesch-Kincaid complexity, and Gunning complexity for depression; (2) pitch, volume standard deviation, volume slope, average F2 formant frequency, Gunning complexity, and eyelid droop for fatigue; and (3) shimmer for cognition. We also observed how features vary over time among participants with multiple sessions.
Conclusions:
The conversational and AI-enabled platform was feasible among an operational sample as a method to detect depression, fatigue and cognition. These results further validate previously discovered digital biomarkers of mental health, fatigue, and cognition and inform the development of personalized models for each user while detecting anomalies in a remote monitoring setting.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.