JMIR Publications

ABSTRACT

Background:

E-learning has taken a firm place in postgraduate medical education. Whereas 10 years ago it was promising, it now has a definite niche and is more than a hype: indeed, it is clearly here to stay. However, evaluating the effect of postgraduate medical e-learning (PGMeL) and, improving upon it, is can be complicated. While the learning aims of e-learning are evaluated, there are no instruments to evaluate the instructional design of PGMeL. Such an evaluation instrument may be developed by following the Association for Medical Education in Europe (AMEE) seven-step process. The first five steps of this process were previously performed by literature reviews, focus group discussion and an international Delphi.

Objective:

This study will continue with step six and seven and answer the research question: is a content-validated PGMeL Evaluation Survey useful, understandable and of added value for creators of such an e-learning?

Methods:

There are five steps in this study: 1) creating a survey from 37 items; 2) testing readability and question interpretation; 3) adjusting, rewriting and translating; 4) gathering filled out surveys from three PGMeLs; and 5) holding focus group discussions with the e-learning authors. Step five is carried out by presenting the results of the evaluations from step four, followed by a group discussion. There are three groups of participants in this study. Group A are experienced end-users of PGMeL, and participated in step two. Group B are users who undertook e-learning and were asked to fill out the survey in step four. Group C are the authors of the e-learnings described above.

Results:

From a list of 36 items we developed a postgraduate Medical E-learning Evaluation Survey (MEES). Seven residents participated in the step-two group discussion; four items were interpreted differently, three were not readable and two items were double. The items from step two were rewritten and, after adjustment, were understood correctly. The MEES was translated into Dutch and again pilot tested. All items were clear, and were understood correctly. The MEES version used for the evaluation contained three positive domains (motivation, learning enhancers and real-world translation) and two negative domains (barriers and learning discouragers), with thirty-six items in those domains, five Likert-scale questions of 1 to 10, and five open questions asking the participants to give their own comments in each domain. Three e-learnings were evaluated during the period July to November 2018. There was a total of 158 responses from a Dutch e-learning, a European OB/GYN e-learning and a surgical e-learning offered worldwide. Finally, three focus group discussions took place with a total of ten participants. The usefulness was much appreciated, understandability was good and added value was high. Four items needed additional explanation by the authors and a Creators’ Manual was created at their request.

Conclusions:

The MEES is the first survey to evaluate instructional design of PGMeL and is constructed following all seven steps of the AMEE. This study completes the design of the survey and shows its usefulness and added value to the authors. It finishes with a final, publicly available survey which includes a Creators’ Manual. We briefly discuss the number of responses needed and conclude that more is better; in the end, however, one has to work with what is available. The next steps would be to see whether improvement can be measured by using the MEES, and to continue to work on the end understandability in different languages and cultural groups.