Currently submitted to: JMIR Medical Informatics
Date Submitted: Nov 5, 2026
Open Peer Review Period: Nov 12, 2026 - Jan 7, 2027
(currently open for review)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Optimizing Cancer Trial Eligibility: A Performance Assessment of PENELOPE’s Criteria Detection and Rephrasing
ABSTRACT
Background:
Configuring Electronic Health Record (EHR) based Clinical Trial (CT) recruitment support systems requires transforming free-text eligibility criteria (EC) into executable queries. This step remains largely manual, particularly during patient prescreening prior to official enrolment.
Objective:
To develop and evaluate PENELOPE C2PAC, a pipeline designed to extract EC from CT protocols, detect Prescreening-oriented Atomic Criteria (PAC), and rephrase them into standardized sentences.
Methods:
The pipeline consists of three modules: an EC extractor request ClinicalTrials.gov API; a lemma-based PAC detector relying on the PENELOPE knowledge base (which contains standard concepts like SNOMED CT, LOINC, ICD-10, ICDO3, ATC, RxNorm) enriched with a domain ontology; and a PAC editor powered by a large language model (LLM) to generate PAC standard free text. The pipeline was evaluated on interventional CTs. Three experts (a medical oncologist, a pathologist and a medical informatics engineer) independently rated each PAC detected within the inclusion criteria, establishing a gold standard by consensus. Inter-rater agreement was measured using Cohen’s kappa, and the PAC detector’s performance was assessed via precision, recall, and F1 score.
Results:
Out of 36 clinical trials, the extractor retrieved 481 inclusion and 621 exclusion criteria, with 4 of the 36 CT failing to extract EC, therefore requiring manual correction. The detector identified 270 inclusion criteria with at least one PAC, totaling to 676 PACs. Compared to the gold standard (Cohen’s kappa ranging from 0.64 to 0.68), the detector achieved a precision, recall and F1 score of 0.63, 0.60 and 0.62, respectively. Main error sources included non-standard acronyms in ECs, missing concepts within the PENELOPE knowledge base, lack of negation detection, and insufficient management of ordinal variables. The LLM produced acceptable phrasings for 435 PACs (64.3%), yielding 146 unique candidates for integration to the PENELOPE PAC library, though agreement on phrasing quality was only slight to fair (Cohen’s kappa ranging from 0.05 to 0.36).
Conclusions:
PENELOPE C2PAC semi-automatically transforms free text EC into standardized PAC through a transparent, human-in-the-loop process. While the system demonstrates promising extraction and detection capabilities, refining acronym processing, negation handling, and knowledge base coverage will be essential before integrating the pipeline into live research workflows.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.