JMIR Publications

ABSTRACT

Background:

Privacy concerns represent a primary barrier to the adoption of large language model (LLM) applications, particularly in sensitive domains such as healthcare. Although a growing body of survey-based research has examined user privacy concerns with LLMs, no standardized instrument exists for this purpose. Existing studies have each developed their own instruments grounded in different theoretical frameworks and designed for different purposes. This heterogeneity limits cross-study comparability and leaves researchers or practitioners without a validated, ready-to-use tool for assessing privacy concerns with a specific LLM system.

Objective:

This study aimed to develop and validate a concise, standardized questionnaire for assessing user privacy concerns with LLM-based systems, suitable for use by both researchers and practitioners such as AI application developers and healthcare professionals.

Methods:

A literature review was conducted to identify privacy-related domains relevant to LLM use, encompassing technology vendor trust, informed consent, data handling practices, user control, and transparency. A draft questionnaire comprising one item per domain was developed and subjected to expert content validity review by independent researchers. All items achieved acceptable content validity (item-level content validity index >.80; scale-level content validity index [S-CVI/Ave] = .964). The finalized instrument was piloted internally and then administered via Qualtrics to a sample of adults recruited through the University of Pittsburgh, LinkedIn, and the Pitt+Me research participant registry. Data were collected between April 2025 and July 2026. Eligible participants were adults aged 18 years or older with awareness of LLM-based applications. Following data screening, psychometric analyses were conducted to assess internal consistency reliability (Cronbach's alpha and McDonald’s omega) and factor structure. Factor structure was examined using a split-sample design, with exploratory factor analysis (EFA) conducted on a random one third and confirmatory factor analysis (CFA) on the holdout two thirds. EFA used maximum likelihood extraction on a polychoric correlation matrix, with parallel analysis used to determine the number of factors. CFA used diagonally weighted least squares estimation on the same matrix type. Both global and local fit were performed for the CFA models.

Results:

A total of 603 responses were collected. After removing pilot, ineligible, and incomplete records, the final analytic sample comprised 361 participants. The sample was predominantly younger than 35 years old (54.8%), female (63.7%), White (78.9%), and highly educated, with 88.1% holding at least a bachelor's degree. The composition constrains the generalizability of the factor structure and score distributions. The instrument was refined from seven to six items after two general-concern items were found to be near-collinear (polychoric r = .91). The final six-item questionnaire showed high internal consistency (Cronbach's α = .90; CFA-based McDonald’s ω = .90). EFA (n = 121) supported a unidimensional structure with all standardized loadings ≥ .70 (range: .71-.88). CFA on an independent holdout subsample (n = 240) showed excellent fit (CFI = .996; TLI = .993; RMSEA = .046; SRMR = .057). A one- versus two-factor comparison indicated only a non-discriminant secondary facet (factor correlation r = .83), and information criteria favored the unidimensional model. Together, these results support the unidimensional structure for measuring privacy concern with LLMs.

Conclusions:

The LLM Privacy Concern Assessment (LLM-PCA) is a concise, psychometrically sound six-item instrument that captures a unidimensional privacy concern construct specific to LLM-based systems. Unlike prior instruments developed within specific theoretical frameworks or for narrow research purposes, this scale is designed for broad applicability, enabling standardized measurement across studies and practical privacy assessment in real-world LLM applications.