Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Dec 4, 2023
Date Accepted: Apr 29, 2024
The Evaluation and Comparison of Academic Impact and Disruptive Innovation Level of Medical Journals: Bibliometrics Analysis and Disruptive Evaluation
ABSTRACT
Background:
The quality and results of medical research have a direct impact on human health and well-being. As an important platform for researchers to present their academic findings, medical journals have a close relationship between their evaluation orientation and the value orientation of their published research results. However, no study has yet examined the differences between the academic impact and disruptive innovation level of medical journals.
Objective:
This study will provide reference for correctly recognizing the innovation level of journal published results, conducting scientific and reasonable medical journal evaluation and constructing a scientific, objective and fair academic evaluation system.
Methods:
We chose the general & internal medicine SCIE journals in 2018 as the study object to explore the differences in the academic impact and disruptive innovation level of medical journals based on the POCI and H1Connect databases respectively and compared them with the results of peer review.
Results:
(1) The correlation coefficients of JDI with CIF5, JIF and JCI are 0.677, 0.585 and 0.621 respectively and the correlation coefficient of Dz with CC5 is 0.635. However, the average difference in the disruptive innovation and academic influence rankings of journals' reaches 20 places (about 17.5%) and the average difference in the disruptive innovation and influence rankings of research papers' reaches about 2700 places (about 17.7%), reflecting the essential difference between the two different evaluation systems. (2) The top 7 journals selected based on JDI, CIF5, JIF and JCI are the same and all of them are H-journals. While 8 (53.33%), 96 (64%) and 880 (58.67%) of the Top0.1%, Top1% and Top10% papers selected based on Dz and CC5 are the same respectively, there are 8 (53.33%), 65 (43.33%) and 187 (12.47%) H-papers among Top0.1%, Top1% and Top10% papers selected based on Dz. There were 5 (33.33%), 74 (49.33%) and 220 (14.67%) H-papers among Top0.1%, Top1% and Top10% papers selected based on CC5, respectively. (3) Research papers with the Changes Clinical Practice tag showed only moderate innovation (4.96) and impact level (241.67), but had high levels of peer-reviewed recognition (6.00) and attention (2.83).
Conclusions:
The study shows that the scientific research evaluation based on the disruptive innovation index is detached from the traditional impact evaluation system, the three evaluation systems only have high consistency for authoritative journals and top papers and neither the single impact index nor the innovation index can directly reflect the impact of medical research on clinical practice. How to establish a set of integrated, comprehensive, scientific and reasonable journal evaluation system to improve the existing evaluation system of medical journals still needs further research.
Citation
Per the author's request the PDF is not available.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.