JMIR Publications

ABSTRACT

Background:

Augmented reality (AR) has emerged as a promising tool to enhance surgical precision during robot-assisted partial nephrectomy (RAPN), particularly by enabling the intraoperative overlay of three-dimensional (3D) anatomical models. However, real-time AR implementation requires robust segmentation of anatomical structures, such as the kidney, which remains technically challenging in dynamic laparoscopic environments.

Objective:

This study aimed to develop and validate a large, annotated image dataset to train deep learning models for automated segmentation of the renal parenchyma during RAPN, as a prerequisite for real-time AR guidance.

Methods:

We conducted a single-center, observational image annotation study using prospectively collected surgical videos from 131 RAPN procedures performed between 2022 and 2024. Patients had localized renal tumors, with 11 presenting multifocal disease (160 tumors in total). A total of 48,000 frames were extracted based on sharpness, diversity, and exclusion of artifacts. A subset of 454 images was annotated by nine contributors (surgeons, engineers, and non-experts) after structured training. Inter-annotator agreement was assessed using Dice Similarity Coefficient (DSC) and sensitivity against expert reference. A convolutional neural network (AlbuNet-34 architecture) was trained using 15,683 annotated images and evaluated on a validation set of 3,137 images. Model performance was analyzed globally and across surgical phases.

Results:

Annotators achieved high agreement, with median DSC values ranging from 0.91 to 0.95 and sensitivity consistently above 0.89. The deep learning model reached a mean DSC of 0.75 (SD 0.23) and a sensitivity of 0.71 (SD 0.24) on the validation set. Segmentation accuracy varied significantly by surgical phase, with lower performance observed during tumor resection and renal bed reconstruction due to increased visual complexity.

Conclusions:

This study demonstrates the feasibility of automated renal parenchyma segmentation using deep learning in real-world intraoperative settings. While current performance remains below expert-level annotations, the creation of a large annotated dataset and the implementation of a structured multi-annotator workflow represent key milestones toward reliable AR-assisted surgery. Ongoing refinements in annotation quality, dataset diversity, and neural network optimization are expected to enhance future real-time AR applications in urology.