Accepted for/Published in: JMIR mHealth and uHealth
Date Submitted: Sep 21, 2020
Date Accepted: Nov 30, 2020
The human factor in automated image-based nutrition apps; An analysis of common mistakes using goFOOD[TM] Lite application
ABSTRACT
Background:
Technological advancements have enabled nutrient estimation by smartphone apps, such as goFOODTM. This is an artificial intelligence-based smartphone system, which uses food images or video captured by the user as input, and then translates these into estimates of nutrient content. The quality of the data is highly dependent on the images the user makes. This can lead to major loss of data and impaired quality. Instead of removing these data from the study, in-depth analysis is needed to explore common mistakes and to use them for further improvement of automated apps for nutrition assessment.
Objective:
The aim of this study was to analyse common mistakes made by participants using the goFOODTM Lite app, a version of goFOODTM, that was designed for food-logging, but without providing results to the users, in order to improve both the instructions provided and the automated functionalities of the app.
Methods:
The 48 study participants were given face-to-face instructions for goFOODTM Lite and were asked to record two pictures (one recording) before and two after the daily consumption of each food or beverage, using a reference card as fiducial marker. All pictures that were discarded for processing due to mistakes were analysed, in order to record the main mistakes made by users.
Results:
Of the 468 recordings of non-packaged food items captured by the app, 60 (12.8%) had to be discarded due to errors in the capturing procedure. The principle problems were: 1) wrong fiducial marker or improper marker use (19 recordings), 2) plate issues such as a non-compatible or visible plate (8 recordings), 3) a combination of various issues (17 recordings), 4) other reasons such as obstacles (hand) in front of the camera or matching recording pairs (16 recordings).
Conclusions:
No other study has focused on the principle problems in the use of automatic applications for assessing nutritional intake. This study showed that it is extremely important to provide study participants with excellent instructions, if high quality data are to be obtained. Future developments could focus on making it easier to recognise food on various plates from its colour or shape and on exploring alternatives to using fiducial markers. It is also essential for future studies to understand the training needed by the participants, as well as to enhance the app’s user-friendliness and to develop automatic image checks based on participant feedback.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.