JMIR Publications

ABSTRACT

Background:

Introduction The current decade has seen much technological progress, but one of the most impactful and controversial areas has been Generative AI. Since AI was first discussed in the 1950’s [1] and especially since it exploded onto the technological scene in the 2020s, it has been a source of both wonder and fear about how it might help, hinder, and otherwise permanently alter human life. Today, in 2025, AI is a household term and a daily topic of news and discussion. It is easy to imagine a future in which AI is even more ubiquitous than it already is, but two of the most talked about areas of AI incorporation today are education and healthcare. AI use has both supporters and detractors in education and healthcare. Proponents of AI in education tout its potential to individualize learning experiences and thereby increase student engagement; they also advocate for AI’s many supportive functions such as analyzing and managing various student data and streamlining administrative tasks [2]. Meanwhile, AI has been used in healthcare since the development of a glaucoma consultation program at Rutgers University in 1976 [1]. Since then, exponential advancements in AI system design have led to a proliferation of clinical uses. ChatGPT, for example, is being used in applications such as answering patient questions; assisting in clinical data analysis and decision making; responding to emergencies; assisting in practice management; and assisting in data management and other aspects of medical research [3]. Along with the advantages and benefits of AI’s educational and healthcare applications, each presents specific challenges and raises questions. Detractors of AI in education cite the potential for students to cheat with AI. Some also warn against students leaning too heavily on AI to the diminishment of their own critical thinking abilities [2]. In both education and healthcare, questions arise about output accuracy plus such ethical issues as data privacy and output bias, among others. Health informatics lies at the intersection of technology and healthcare, and as health informatics educators, we realize that whether we welcome it or fear it, AI will play a large role in our students’ futures. More precisely, our students will graduate into a healthcare profession that is already regularly using AI, and they will work for employers who expect that they are well prepared to utilize it in their jobs. In fact, employers already prioritize AI competencies over prior non-AI experience [4] in the job candidates’ profiles and believe AI skill sets are more important than many others [5].

Objective:

Knowing that technology, including AI, advances vastly more quickly than official university curricula, we felt it imperative to transition some of our assignments so our students could begin using AI in a health informatics context now. We represent two very distinct Master’s in Health Informatics Programs. University of Illinois at Chicago (UIC) is an all-online, asynchronous program founded in 1999 and has been completely online since 2010. Many of our students are returning to school after time spent working, thus they are older than traditional graduate students, and many already possess significant experience in healthcare, business, or technical fields. The UIC program has an overall focus on population-level social informatics and offers students courses and specialization options in data science, mobile health, and leadership. University of San Francisco (USF) is a hybrid program. Students spend over half their time in-person on campus and the remainder of their time online. This relatively new program was founded ten years ago. Many students are recent bachelor’s degree recipients with backgrounds in public health, business, health professions, and technology. The USF program focuses on data analytics, digital health, and clinical informatics, as well as offering course options in public health informatics, clinical leadership, and nursing informatics. In our endeavor to rapidly infuse AI into our curricula, we focused on four knowledge domains of AI Competencies for health informatics education: Essentials of AI; Applications of AI to Health Informatics; AI Transformations to Information and Knowledge; and Organizational Change and Adoption of AI Within the Healthcare Organization. We based these categories on prior work done in the development of a health data science concentration within a health informatics curriculum [6]. Within each of these AI competency categories, we listed knowledge, skills, and attitudes we believed our programs would need to provide for our students, leaving room for future ideas. Our new work builds on the previous work incorporating Gen AI skills and competencies into the same Knowledge Domains. The skills and competencies enabled us to develop Gen AI assignments using a backward design approach, beginning by laying out what we wanted our students to learn and then using that information to create assignments to help them learn it [7]. During the fall 2024 semester each program incorporated a Gen AI assignment into a course, assessing each student’s preliminary knowledge of AI, their knowledge of AI at the conclusion of the assignment, and then each student’s reflections on AI. UIC students participated in this study during BHIS 593, their Capstone experience, the culminating course in the Master’s in Health Informatics. This course requires students to research a topic of interest over the 16-week semester, delivering a paper or project at the end. The goal is for students to synthesize their learnings from their degree program and demonstrate competency as health informaticians. USF students participated in this study during HS 633, Exploring Gen AI Ethics: Intersection of Education and Health Ecosystems, which they take approximately half-way through their master’s program. This course, a hands-on workshop, incorporates the latest AI literature and tools and focuses on Gen AI, AI ethics concepts, AI applications, use cases, frameworks, and AI policies with special emphasis on healthcare. The goal is to equip students with foundational knowledge and practical skills in Gen AI, preparing them to navigate the Silicon Valley tech ecosystem and the traditional healthcare landscape. USF's GenAI Ethics course (HS 633) was conceived not as a technical deep dive into Gen AI models and methods but as a framework‑building course that foregrounds ethics and governance. This course was offered for the first time in Fall 2024 to coincide with this research study. Our research questions for this multi-site study were: Did students learn (develop knowledge) about Gen AI by doing the assignments? Did students say they developed Gen AI skills and professional attitudes by completing the assignments?

Methods:

Methods - Design, Setting, Participant Recruitment This was a multisite study of assessment of assignments completed by Master of Health Informatics students at UIC and USF campuses in the fall of 2024. Across both sites, a total of 18 students participated in the research. UIC At UIC the research was implemented in the online Health Informatics curricula for Biomedical Health Information Sciences in the Capstone course required to complete the MSHI. There were eleven participants from UIC (N=11). Students chose from 4 areas of Gen AI practice and developed their specific topics working with faculty to define the scope and deliverables for their project. Topics explored real-world questions of interest for which students developed a blueprint / prototype solution or a use case. Table 1 contains information about the four topic areas. The institutional review board of the University of Illinois Chicago approved this study under the exempt research determination. Table 1. Gen AI topics Topics Focusing Questions Description Clinical uses How is Gen AI being used to augment provider and clinician workflows? What is the potential for Gen AI to assist in improving health outcomes? The information needed to make medical decisions (e.g., medical history, laboratory and imaging results, unstructured clinical notes) can be scattered across multiple records that exist in myriad formats and locations. Gen AI could be used to compile and organize this information—and put it into a format that is accessible and clinician-friendly—to accelerate and augment critical thinking. In addition, Gen AI-enabled ambient documentation could pull information from clinician conversations and generate natural-sounding notes. Technology could also be trained to identify patterns that are too subtle for a human to recognize. Patients/ Consumers What is the consumer or patient perspective on using Gen AI for healthcare? In what ways can Gen AI improve the patient experience / patient engagement to manage chronic conditions? Accurate real-time audio and text messages could be generated instantly, and in different languages, as frontline workers interact with people for health care, and social services. Gen AI also could translate documents, websites, laws, regulations, and policies. Health advisories could make essential information accessible to a diverse population. Gen AI could also play a central role in optimizing and mitigating health and safety risks by generating worksite-specific safety training that replicates real-world settings and critical scenarios. Frontline/ first responders and public health, community social services use of AI? How might Gen AI be used to improve patient engagement (from the public health perspective)? In what ways can Gen AI be used to streamline emergency response in the field, urgent care or the hospital emergency department? Operational inefficiencies or limited capacity in the call center can translate to decreased customer satisfaction. Gen AI could help to create hyper-personalized experiences with customers and patients. It could also help efficiently support customers and reduce call volume handled by associates. The technology might also assist human staff in generating responses to customer questions, insurance coverage and other plan details. The customer service experience can have a direct impact on patient perception, even without any change in charged costs or appointment wait times. Ethical Use of Gen AI How can organizations create an ethical framework for Gen AI in healthcare (consider bias and hallucinations)? How can they use an ethical framework and still support team science and innovation? What are the data governance policies and steps needed for quality control and validation of a Gen AI model? Set up an experiment that identifies and examines these issues. Recommendations for how organizations should proceed to build this into governance policies for Gen AI and how it should be considered in building capabilities for using Gen AI to increase AI literacy in the workforce. All students enrolled in the course were given the opportunity to participate in the research at the beginning of the semester. Participation was voluntary and students that chose not to do the research were still able to complete the course. All data was collected via the Blackboard Learning Management System. In Blackboard, students completed a pre-test to assess their baseline knowledge of Gen AI. The same questions were given as a post-test at the end of the course to assess student learning. Additionally, students completed self-reflections about skills developed and their attitudes toward using Gen AI and how they thought it might impact their health informatics work and careers. The reflections were also completed at the beginning and at the end of the semester (see Table 2). Table 2. Student reflections Reflection Questions Reflection 1 As you look to the future, to what degree do you think Gen AI will be useful to your work in health informatics? To what degree do you think it might become part of your work process? What Gen AI skills do you think will be most valuable to you in your health informatics career? To what extent do you think Gen AI has the potential to enhance your ability to do health informatics work or enhance your productivity? What concerns do you have about using Gen AI in your professional work? Are there aspects of Gen AI that you anticipate might be problematic? What do you see as some of the drawbacks or challenges of Gen AI use? Reflection 2 Describe how did using Gen AI for your capstone impacted your satisfaction with the work you produced. Do you think you were more or less satisfied with your results than you would have been without using Gen AI? Did you experience disappointment or worry/anxiety about using Gen AI to augment your capstone project? Describe any way in which using Gen AI was disappointing or led to worry for you. Did this change over the course of the semester? What impact did using Gen AI have on your critical thinking as you worked on your capstone? Did it enhance your critical thinking? Did you experience reduced critical thinking due to overreliance on AI technology? Overall, what was the best thing (the thing you enjoyed the most) about using Gen AI for your capstone project? What was the worst thing (thing you enjoyed the least) about using Gen AI for your capstone? Please add any additional comments you would like to make about your learning experience and the use of Gen AI. For their projects, students completed foundational readings. They conducted additional research exploring their chosen topic and developed prompts to use Gen AI to assist in their research, brainstorm, and synthesize information. Student projects were submitted at defined checkpoints during the semester and faculty provided feedback to guide the projects’ development. Knowledge was assessed using multiple choice and short answer tests. A pre-test was given at the start of the semester, and an identical post-test was completed at the end of the semester. The tests were open book, administered on a learning management system with unlimited attempts. However, students were advised that these tests had no bearing on their grade for the course but were only for research purposes to get a baseline of what students know about Gen AI. An important part of this study was to identify skills that students will need in the workforce. AI skills and attitudes were identified using student reflections on student engagement topics exploring how they would use Gen AI in their future career, concerns with using Gen AI, and their satisfaction with the end product for their courses [8]( At UIC reflections were given at the beginning and end of the semester. USF The USF participants for this study are seven (N=7) out of eight students who took the Gen AI Ethics course in the fall of 2024. A student absent from the initial survey distribution at the beginning of the fall 2024 semester was excluded from this study. The participants are in their second year of the master's degree program in digital health informatics. USF students were introduced in class to the UIC+USF study and explained the purpose, inclusion criteria, privacy, and harm risk per the IRB-approved documentation. The two components of the survey (pretest / posttest and reflections) were explained to students. Students were told that participation was voluntary and would not affect their grades. The initial USF test (pre) was distributed during in-person class time. Students were given ample time, up to one hour to complete their surveys. Surveys identical to those used at UIC were delivered using the Google Survey application. The USF results were de-identified and remained in Google Cloud until the end of the semester. At the end of the semester, a second survey was administered via Google. This survey had two sections. The first section was identical to the pre-test at the beginning of the semester. The additional second section was for the reflections of students who completed Gen AI semester-long projects. USFs IRB approved the second test before distribution to the study participants. The second USF survey, like the first, was distributed during in-person class time, their last before students presented their final semester-long Gen AI Ethics projects. Participants were given ample time, up to one hour in class to answer and reflect on their semester. The USF participants in the second survey were seven (N=7) out of eight students who took the Gen AI Ethics course in the fall of 2024. The student absent from the initial study was excluded from the second survey for consistency. Once USF students completed the second survey, de-identified survey results were uploaded to UIC’s secure Box system for analysis. The UIC Capstone experience is a culmination of learning throughout the MSHI program. In their capstone, students personalize their project by pursuing an area of interest to them. This has the advantage of incorporating real-world scenarios and teaches critical divergent thinking and active learning. The USF Gen AI ethics course equips students with foundational AI knowledge that they can then bring into their work at the intersection of technology and healthcare. In both programs, assignment instructions included use of Gen AI to support the creation of a final product. Applying backward design principles Our approach applied backwards design principles to develop learning assignments that explore the use of Gen AI in real-world practical examples. In this method of instructional design, an instructor determines desired results first, then identifies the evidence required to show that learners have attained those results. The instructor then develops learning activities that will provide that evidence [7]. To develop our Gen AI assignments, we first examined knowledge domains for health informatics education used in previous work by these authors to create a data science concentration within the MSHI [6] and adapted them for AI competencies: Essentials of AI Application of AI to Health Informatics AI Transformations to Information and Knowledge Organizational Change and Adoption of AI within the Healthcare Organization A preliminary list of 33 competencies was developed based on our professional experience, recommendations in published articles, and job requirements actually posted. Thinking of these competencies as the desired outcomes of students’ Gen AI experiences in our courses, we then used these categories and competencies as guidelines for working backwards to write our pre-test, post-test, and reflection questions, endeavoring to guide and assess students’ development of competency in these areas. Main measures We collected data about developing student competencies in Gen AI: knowledge, skills, and professional attitudes. Research Question 1: Did students learn (develop knowledge) about Gen AI by doing the assignments? Knowledge – Students completed a pre and posttest survey with 23 items each worth 1 point. Final scores for pretest and posttest were compared to see if students demonstrated higher scores on the posttest. Research Question 2: Did students say they developed Gen AI skills and professional attitudes by completing the assignments? Skills and professional attitudes – student reflection data Categorization of student responses was completed by 2 faculty from UIC and 1 from USF. Each student response was evaluated to select the one primary element it contained and then was assigned to one competency category. Responses that were not related to the subject received no assignment, (ex: “thank you”, “happy I took the class”). Differences in faculty categorization were examined together in a work session to develop a consensus. To help us come to a consensus we used some basic rules, i.e. reflections related to ethics or accuracy were assigned to the AI Transformations to Information and Knowledge domain; and for longer student responses with multiple perspectives, raters assigned the knowledge domain that they agreed was the predominant domain.

Results:

Results Pre/post-test - quantitative Students completed a pre-test survey of 23 multiple choice and short answer questions. Students demonstrated an improvement in knowledge from pre-test to post-test in both courses with UIC students improving from 81% to 93% at the end of the capstone course. USF students also improved from 77% to 80% by the end of the Gen AI Ethics course. Table 3. Knowledge assessment Site Pre-test Post-test UIC 81% (n=10) 93% (n=8) USF 77% (n=7) 80% (n=7)

Conclusions:

Proliferation of Gen AI and digital health education in the health professions curricula is in the early stages and is poised to grow into a large-scale academic debate about implementation, with consensus on the need to embed elements of artificial intelligence expected from professional societies and graduate education commissions. Therefore, the need to define AI competencies and include them among educational outcomes stretches beyond informatics programs, expanding into the existing health professions [19] as well as emerging specialties such as Master of Digital Health [20]. There are ongoing attempts to define a clear set of skills to be taught in such digital health programs, with many of the grassroots efforts originating from experiences of the faculty [19]. By beginning with the competencies and using backward design to develop learning experiences based on them [7], we innovatively addressed a gap in Gen AI education in a health informatics program, tested two distinct cohorts of students in both online and hybrid content delivery programs, and share our experiences and recommendations for rapidly infusing Gen AI into curriculum. These recommendations are generalizable to a wide variety of informatics and health professions education programs, to address the growing need for academia to update programs with the latest technology developments in clinical care and life sciences. Clinical Trial: N/A