Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Jun 5, 2022
Open Peer Review Period: Jun 5, 2022 - Jul 31, 2022
Date Accepted: Oct 29, 2022
(closed for review but you can still tweet)
The agreement between virtual patients and unannounced standardized patients in evaluating primary healthcare quality: a multicentered, cross-sectional pilot study in seven provinces of China
ABSTRACT
Background:
The unannounced standardized patient (USP) is the gold standard for primary healthcare (PHC) quality assessment but has many restrictions associated with high human and resource costs. Virtual patient (VP) is a valid, low-cost software for simulating clinical scenarios, and is widely used in medical education. Whether it can be used to assess the quality of PHC remains unknown.
Objective:
This study aimed to examine the agreement of VP and USP in the PHC quality assessment, and to identify factors influencing the VP-USP agreement.
Methods:
Eleven matched USP-VP cases were developed based on clinical guidelines and were implemented on a convenient sample of urban PHCs in the capital city of the seven-study provinces. 720 USP visits were conducted, whereby on-duty primary healthcare providers who met inclusion criteria were randomly selected by the USPs and further received a VP assessment of the same case condition at least a week later. The VP-USP agreement was measured by concordance correlation coefficient (CCC) for continuity scores and weighted kappa for diagnosis. Multiple linear regression was applied to identify factors influencing VP-USP agreement.
Results:
Only 146 VP scores were matched with original USP scores. The CCC of medical history was 0.37 (95%CI 0.24, 0.49), physical examination was 0.27 (95%CI 0.12, 0.42), laboratory and imaging test was -0.03 (95%CI -0.20, 0.14), and treatment was 0.22 (95%CI 0.07, 0.37). The weighted kappa of diagnosis was 0.32 (95%CI 0.13, 0.52). The multiple linear regression model indicated that VP tests were significantly influenced by the different case conditions and the cities studied.
Conclusions:
The agreement was low between VP and USP in PHC quality assessment. It may reflect the know-do gap, while VP test results were also influenced by different case conditions, VP’s interactive design and usability. Further modifications for VP’s usability should be user-centred, paying attention to balance usability enhancement and hints avoidance.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.