JMIR Publications

ABSTRACT

Background:

Recent research has developed many information technology (IT) supported systems for health care control, including systems estimating nutrition from images of meals. Systems that capture data about eating and exercise are useful for diabetics as well as people who are simply on a diet. Continuous monitoring is key to effective dietary control, requiring systems that are simple to use and motivate users to pay attention to their meals. Unfortunately, most current systems are complex or fail to motivate. Systems require some manual inputs such as selection of an icon or image or inputting the category of the user's food. The nutrition information fed back to users is not especially helpful, as they are told only the estimated detailed nutritional values contained in the meal.

Objective:

In this paper, we introduce healthiness of meals as a more useful and meaningful general standard, and present a novel algorithm that can estimate it from meal images without manual inputs.

Methods:

We propose a system that estimates meal healthiness using a deep neural network that extracts features and a ranking network that learns the relationship between the degrees of healthiness of the meals, using a dataset prepared by a human dietary expert.

Results:

The ranking estimated by the proposed network and the ranking of healthiness based on the dietitian's judgement have relation with a high correlation coefficient; 0.715. In addition, we show that extracting network features through pre-training with a publicly available large meal dataset lets us overcome the limited availability of specific healthiness data.

Conclusions:

We have presented an image-based system that can rank meals in terms of the overall healthiness of the dishes constituting the meal. We showed that the ranking has good correlation to nutritional-value-based ranking. We then proposed a network that allows conditions that are important for judging the meal image, extracting features that eliminate background information and extracting features that are independent of location. Under these conditions, the experimental results show that our network achieves a higher accuracy of healthiness ranking estimation than the conventional image ranking method. The results of this experiment detecting unhealthy meals suggest that our system can be used to assist health care workers in making meal plans for diabetic patients who need advice in choosing healthy meals.