JMIR Publications

ABSTRACT

Background:

The use of Artificial Intelligence (AI) to analyze healthcare data has become common in behavioral health sciences. However, the lack of training opportunities for mental health professionals limits clinicians' ability to adopt AI in clinical settings. AI education is essential for trainees, equipping them with the literacy needed to implement AI tools in practice, collaborate effectively with data scientists, and develop as interdisciplinary researchers with computing skills.

Objective:

As part of the Penn Innovation in Suicide Prevention Implementation Research (INSPIRE) Center, we developed, implemented, and evaluated a virtual workshop to educate psychiatry and psychology trainees on using AI for suicide prevention research.

Methods:

The workshop introduced trainees to natural language processing (NLP) concepts and Python coding skills using jupyter notebooks within a secure Microsoft Azure Databricks cloud computing and analytics environment. We designed a three-hour workshop covered four key NLP topics: data characterization, data standardization, concept extraction, and statistical analysis. To demonstrate real-world applications, we processed chief complaints from electronic health record to compare the prevalence of suicide-related encounters across populations by race/ethnicity and age. Training materials were developed based on standard NLP techniques and domain-specific tasks, such as preprocessing psychiatry-related acronyms. Two researchers drafted and demonstrated the code, incorporating feedback from the INSPIRE Methods Core to refine the materials. To evaluate the workshop’s effectiveness, we used the Kirkpatrick program evaluation model, focusing on participants' reactions (Level 1) and learning outcomes (Level 2). Confidence changes in knowledge and skills before and after the workshop were assessed using paired t-tests, and open-ended questions were included to gather feedback for future improvements.

Results:

Ten attendees participated in the workshop virtually, including residents, postdocs, and graduate students from the psychiatry and psychology departments. Only two participants had experience with python or NLP prior to this workshop. They found the workshop helpful (mean = 3.17 on a scale of 1-4, SD = 0.41). Their overall confidence in NLP knowledge significantly increased (p < 0.002), from 1.35 (SD = 0.47) to 2.79 (SD = 0.46). Confidence in coding abilities also improved significantly (p = 0.013), increasing from 1.33 (SD = 0.60) to 2.25 (SD = 0.42). Open-ended feedback suggested incorporating theme analysis and exploring additional datasets for future workshops.

Conclusions:

This study illustrates the effectiveness of a tailored data science workshop for trainees in psychiatry and psychology, focusing on applying NLP techniques and suicide prevention research. The workshop significantly enhanced participants' confidence in conducting data science research. Future workshops will cover additional topics of interest, such as working with large language models, thematic analysis, diverse datasets, and multifaceted outcomes. This includes examining how participants' learning impacts their practice and research, as well as assessing knowledge and skills beyond self-reported confidence through methods like case studies for deeper insights. Clinical Trial: Not applicable