JMIR Publications

ABSTRACT

Background:

Generative Artificial Intelligence (GAI) is expected to enhance the productivity of public social and healthcare sector while maintaining, at minimum, current standards of quality and user experience. However, empirical evidence on GAI impacts in practical, real-life settings remains limited.

Objective:

This study investigates productivity, machine translation quality, and user experience impacts of GPT-4 language model in an in-house language translation services team of a large wellbeing services county in Finland.

Methods:

The study employes a mixed-methods approach. Quantitative data of 908 translation segments was collected in real-life conditions using the computer-assisted language translation software Trados to assess productivity differences between machine and human translation. Similarly, a separate data set of 1387 segment pairs were collected and analyzed to estimate machine translation quality. Additionally, user experience was investigated through qualitative data from translator interviews. The data was collected between March and June 2024 with the team of 4 translators.

Results:

The findings indicate that, on average, post-editing machine translations is 14% faster than translating texts from scratch (P=.028), and 37% faster when number of segments is equalized across translators. Nonetheless, insights from translator interviews underscore that further productivity gains could be achieved by optimizing language model performance and redesigning operational processes and workflows. Additionally, the translators perceive that the quality of machine translation affects both productivity and user experience.

Conclusions:

The results highlight that the full productivity potential of GAI technology is constrained by the organization’s capabilities in managing and utilizing artificial intelligence effectively.