JMIR Publications

ABSTRACT

Background:

Test case authoring in healthcare domain QA presents unique challenges due to complex form structures, nested validation logic, and domain-specific constraints such as age-based eligibility rules, insurance tiers, and dependent relationships. Traditional test development is not only labor-intensive but often overlooks edge cases and combinatorial conditions inherent to healthcare systems. In this study, we explore the application of Commercial LLM (OpenAI GPT-4), a state-of-the-art large language model (LLM), for the automatic generation of functional test cases from unstructured natural language requirements. We present a hybrid architecture that combines GPT-4-based prompt engineering with a Playwright automation framework, enabling dynamic test generation, code integration, and reuse of test components across healthcare applications. Our results show a 70% reduction in test authoring time, improved accuracy in form validation coverage, and faster onboarding of QA resources. The framework further supports document upload testing (PDF parsing) and edge logic validation, which are critical in regulatory-driven domains. This work demonstrates the feasibility of integrating LLMs into test engineering pipelines and lays the groundwork for future research in AI-augmented software testing.

Objective:

As a QA Engineer in healthcare domain, each test I execute carries both regulatory and operational significance as well as real world implications. This is not simply a matter of finding bugs – it’s about making sure patient records are treated properly, that sensitive records are checked with great accuracy, and workflows are wound up to critical compliance standards such as HIPAA. A minor misstep – like improperly handling the way a form processes the demographic data of a child or a spouse or taking in an invalid PDF document – can easily become a major compliance violation or crash of the actual users. Healthcare QA is unusual because of the quantity of diseases and variations that are present. The forms usually contain conditional fields on condition of age, relationship, or health status. The validations need to ensure the dynamical enforcement of rules, and that the data is complete, secure, and traceable. Every change – no matter how trivial – needs to be verified in a variety of platforms (web, mobile, third-party integrations), all whilst producing audit- friendly logs. This high-pressure situation makes manual testing a bottleneck and a risk very quickly. Key Challenges: Requirements are in the form of natural language documents that must be converted into test scripts manually. • Features consist of several combinations (e.g. child1 to child9, spouse DOB, insurance tier). • We validate across diverse platforms: mobile, PDFs, Workday integrations. • The compliance teams must be audit-ready for every test. It took ~45 minutes to write these tests manually for each case, a lot of duplicate logic with minor changes. We could see that we required a smarter, scalable approach.

Methods:

I built a workflow around Commercial LLM (OpenAI GPT-4) that turns plain-English functional requirements into executable test cases — using Playwright and Gherkin syntax as outputs. How It Works: Input: Paste a requirement into a prompt. Processing: GPT-4 interprets the user actions, validations, and edge cases. Output: A complete Playwright test, ready to run. Sample Prompt: { "prompt": "Generate end-to-end Playwright test cases based on the following requirement:\n- User logs into the health insurance portal\n- User selects ‘Add Spouse’ option\n- Fills in spouse details (DOB, ethnicity, race)\n- Uploads supporting PDF document\n- Submits the form\n\n Include test validations for field presence, PDF format, and success message.", "model": "gpt-4" }

Results:

Metric Before GPT-4 After GPT-4 Avg. time per test case ~45 minutes ~8–12 minutes Edge case coverage 50–60% >90% QA onboarding duration ~2 weeks 4–5 days Manual QA effort eliminated ~40% ~70%

Conclusions:

Healthcare QA is very different from the traditional software fields like e-commerce or SaaS, because of the regulatory, structural and compliance limits. In this space, precision, completeness, and traceability are needed during testing. Key factors include: • Age-Based Logic: Examples of conditional validations which are the auto populating of smoking status ‘N’ for applicants under 21 years. • Nested Form Complexity: Dynamic inputs from child1 to child9 which are age defendant and conditionally displayed. • PDF Document Validation: Uploaded documents should be in the conforming format and data integrity criteria with those of the regulator. • Audit and Compliance Requirements: A monitoring of QA processes must ensure that complete logs are kept for auditability purposes, especially in the HIPAA provisions. Before the GPT-4 integration, edge case modeling was a slow and inaccurate process. The changes brought about by AI brought out great efficiency and coverage. 3.1 Quantitative Gains When the GPT-4 was implemented into the healthcare QA workflow, there were significant gains regarding productivity in several areas. One of the most dramatic improvements in the change processes was the decrease in test case creation time from a 45-minute time to about 8 to 12 minutes only. This increase in efficiency does not only speed up the overall QA cycle but also adds free time from highly valuable engineers for more strategic teamwork. Edge case coverage also benefited from huge improvements, from an original 50%-60% to over 90% after the help by GPT-4. It translated to fewer blind spots in testing and increased confidence in the resume of system reliability in complex scenarios. The time for onboarding for new QA engineers was also improved. Earlier on, it would take two weeks for a new team member to be productive in writing tests. With GPT-4 making the process easier and diminishing the learning curve, onboarding time has been reduced to mere 4-5 days. Finally, a lot of manual QA effort was saved. Works which previously needed human input on a highly considerable scale were now partially or fully automated and the manual workload decreased from 40% to 70%. Such quantitative returns represent the transformational potential in integrating a high-compliance, high-precision QA arena such as healthcare with GPT-4. Metric Before GPT-4 After GPT-4 Avg. time per test case ~45 minutes ~8–12 minutes Edge case coverage 50–60% >90% QA onboarding duration ~2 weeks 4–5 days Manual QA effort eliminated ~40% ~70% Table I. 3.2 Real-World Outcomes Critical Defects Detected: Identified two major production issues • Incorrect age validation for child DOB • Improper handling of PDF upload format. CI/CD Efficiency: Test suites now execute nightly in Browser Stack, with automatic logging and stable pass rates. Scalability: New test cases are generated in minutes, allowing rapid regression cycles across multiple platforms. 4. Challenges and Key Learnings Bringing GPT-4 into the healthcare QA workflow provided significant productivity benefits. Success had to be built through careful prompt engineering and iterative workflow optimization. 4.1 Prompt Design Challenges Clarity Matters: The early prompts with indistinct instructions produced results inconsistent or irrelevant. Specifically defined prompts in reference to UI elements, rules of validation and expected responses set clear ground much for better outcomes. • Contextual Precision: Addition of application-specific constraints (for example: mandatory fields, value range, platform behaviors) improved the accuracy and appropriateness of test cases. • Examples Improve Consistency: By offering output examples (as is Gherkin syntax or Playwright- compatible formats), GPT-4 was led toward conformity with QA standards. 4.2 Model Behavior and Limitations Tendency to Overgeneralize: GPT-4 sometimes added redundant validations, particularly for the common fields like email or name inputs. • Non-Deterministic Outputs: Same prompts sometimes resulted in different responses, which made it difficult to keep track of versions and reproduction. • Post-Processing Required: Minor manual adjustments are usually needed to trim up selectors, to take care of edge cases, or to follow platform specifics. 4.3 Practical Lessons for QA Automation Version Control: The auto-generated test cases were managed and used in conjunction with other people by tracking them in Git for change management and cooperation. • Reusable Prompt Templates: Keeping a library of good prompts ensuring uniform and standard output throughout the team. • Human Oversight: All AI-developed test cases were reviewed by QA engineers and delivered to CI pipelines while ensuring the quality assurance in vulnerable healthcare settings. 5. Conclusion and Future Directions This project is a major shift forward in quality assurance in high stake industries such as healthcare. By using GPT-4 in the test case generation process, we automated what has traditionally been a manual and error-prone job, delivering dramatic timesaving with respect to test case authoring, increased coverage of edge cases, and a speedier onboarding of new QA team members while maintaining rigid compliance requirements like HIPAA. Besides the operational advantages, this combination is part of the general QA philosophy change. from laborious, process-oriented validation to autonomous and intelligent process. With GPT-4, we were able to switch off deterministic test design, meaning QA engineers could work on strategic quality initiatives, complex root-cause analysis, system-level resilience rather than test design. Future Directions Capitalizing on this basis, several major innovations are to be introduced by us: Open-Source CLI Tool: Not yet completed, this tool is designed so that users can upload functional requirements (.txt or .docx), activate GPT-4 to interpret them, and get structured .spec.ts files ready for Playwright automation as a result. AI-Powered Locator Mapping: By using DOM parsing together with the GPT-4 Inference, We have implemented a strategy to dynamically generate and maintain UI selectors, without making tests fragile with changes in the UI. Automated Error Classification: We’re investigating the application of GPT-4 to examine failed test cases and make recommendations for potential root causes (such as logic bugs, network issues, or UI regressions), to make triage and resolution of failed test cases a cinch. These improvements are aimed at scaling our automation capabilities and encourage more use of AI-guided QA in regulated industries. We will also share our findings and tools internally through an AI engineering meetup to support cross-functional innovation.