JMIR Publications

ABSTRACT

Background:

For the public health community, monitoring recently published articles is crucial for staying informed about the latest research developments. However, identifying publications about studies with specific research designs from the extensive body of public health publications is a challenge with currently available methods.

Objective:

Our objective is to develop a fine-tuned pre-trained language model that can accurately identify publications from clinical trials that use a Group- or Cluster-Randomized Trial (GRT), Individually Randomized Group-Treatment Trial (IRGT), or Stepped Wedge Group- or Cluster-Randomized Trial (SWGRT) design within the biomedical literature.

Methods:

We fine-tuned the BioMedBERT language model using a dataset of biomedical literature from the Office of Disease Prevention at the National Institutes of Health. The model was trained to classify publications into three categories of clinical trials that use nested designs. The model performance was evaluated on unseen data and demonstrated high sensitivity and specificity for each class.

Results:

When our proposed model was tested for generalizability with unseen data, it delivered high sensitivity and specificity for each class as follows: Non-randomized trials (0.95 and 0.93), GRTs (0.94 and 0.90), IRGTs (0.81 and 0.97), and SWGRTs (0.96 and 0.99), respectively.

Conclusions:

This model offers a valuable tool for the public health community to directly identify publications from clinical trials that use one of three classes of nested designs.