JMIR Publications

ABSTRACT

Background:

Screening for clinical trials is challenging for clinicians due to its time consuming and repetitive nature. The rise of artificial intelligence (AI) offers an opportunity to improve screening productivity and reproducibility. Pancreatic cancer has an increasing incidence, poor outcomes, and a critical need for improved management.

Objective:

To assess the performance of AI in evaluating clinical trial inclusion and exclusion criteria, compared to a double-blind human gold standard on a retrospective cohort .

Methods:

In this pilot study, we retrospectively reviewed cases from our institutional database of patients with advanced pancreatic cancer presented at tumor board meetings between January 2018 and December 2023. Each patient was screened for clinical trials open for inclusion at the time of the multidisciplinary meeting. Manual screening of eligibility criteria for each patient-trial was performed by two blinded oncologists to determine potential eligibility (gold standard), with a third resolving discrepancies. Potential eligibility was also assessed using three AI language models (ChatGPT-4.5, Claude-3.7-Sonnet, and Mistral-7b-Instruct v0.3).

Results:

Across 341 patient-trial pairings, the AI models demonstrated high sensitivity, ranging from 83.3% to 92.2%. Analysis of the criteria showed a correlation between the risk of failure and the number of words and the number of characters of the criteria (respectively p = 0.02 and p= <0.01). Overall screening time for manual assessment was significatively longer for human gold standard assessment than AI.

Conclusions:

While our study highlights the promising performance of AI in clinical trial screening, several ways for improvement can be identified. Refining prompt engineering and fine-tuning models with domain-specific datasets could further enhance accuracy and specificity. Future work should explore integration with structured clinical data, such as laboratory values or radiological findings, to improve multimodal comprehension. Expanding the evaluation to a broader range of tumor types and multicenter datasets would improve generalizability. Finally, real-time prospective validation and workflow integration with electronic health records will be critical to assess the feasibility and clinical impact of LLM-assisted screening in daily oncology practice. Addressing these challenges will be essential to move from proof-of-concept to scalable clinical implementation.