JMIR Publications

ABSTRACT

Background:

The “Archive of German language general practice” (ADAM) stores about 800 paper based doctoral theses from 1965 till today. While they have been grouped in different categories no deeper systematic process of information extraction (IE) has been performed yet. Recently developed Large Language Models (LLMs) like ChatGPT have been attributed the potential to help in IE of medical documents. However, there are concerns about hallucination of LLM. Furthermore, there have not been reports regarding their usage in non-recent doctoral theses yet.

Objective:

To analyze if LLMs can help to extract information from doctor thesis by using GPT-4o and gemini-1.5-flash for paper based doctoral theses in ADAM.

Methods:

We randomly selected ten doctoral theses from 1965 and 2022. After preprocessing we used two different LLM pipelines, using models of OpenAI and Google, respectively. Pipelines were used to extract dissertation characteristics and generate uniform abstracts. Furthermore, one pooled human-generated abstract was written. Blinded raters were then asked to evaluate LLM-generated abstracts in comparison to human-generated ones.

Results:

Relevant dissertation characteristics and keywords could be extracted for all theses (n=10): name and place of the institute, title of theses, name(s) of author and year of publication. For all except one doctoral thesis an abstract could be generated using GPT-4o, while gemini-1.5-flash provided abstracts in all cases (n=10). Translation from German into English did not re-sult in a loss of information in all cases (n=10). The modality of abstract generation showed no influence in raters evaluation using ANOVA (P=.23). Creation of LLM-generated abstracts was estimated 24 to 36 times faster than creation by humans.

Conclusions:

An accumulating body of not published doctoral theses makes it difficult to extract relevant information. Recently, great hopes have been placed in LLMs like ChatGPT, which to this day have not yet been used in IE of “historic” medical documents. This feasibility study shows that both models, GPT-4o and gemini-1.5-flash helped to accurately simplify and condense doctor-al theses into relevant information while LLM-generated abstracts being perceived similar and taking about thirty times less time in comparison to human-generated ones. Thus, LLMs can be used to extract relevant information and produce accurate abstracts from doctoral theses in ADAM when used cautiously. Taken together this information could help to better search the scientific background from family medicine from the last 60 years, helping other researchers and thus strengthen research in family medicine.