Trade-off between Automation and Accuracy in Mobile Photo Recognition Food Logging

Trade-off between Automation and Accuracy in Mobile Photo Recognition Food Logging


Brian Y. Lim, Xinni Chng, Shengdong Zhao



Food logging can help users understand their food choices and encourage healthier eating habits. However, current apps still pose many usability challenges, including tedious manual text entry of food names. Recently, advances in computer vision and deep learning are enabling automatic food recognition for instant and convenient logging. However, as a nascent technology, this suffers from inaccuracy, which may lead to poor adoption or misuse. We investigated the trade-off between accuracy and convenience of automatic photo recognition in comparison to manual search logging. Specifically, we have developed a mobile app prototype that integrates both photo recognition and search logging capabilities, and conducted formative investigations on the usability and usage of automatic photo recognition in food logging in a series of studies: online requirements survey, usability lab study, and 1-week field trial in an Asian country. Participants were interested in convenient, automatic photo logging, but dominantly used manual search logging due to a lack of data coverage and accuracy. We identified reasons for poor accuracy and highlight complications in using inaccurate automatic photo logging. We further discuss opportunities for design and technology to address these challenges.

Author Keywords

Food Journals; Food Logging; Image Recognition; Mobile Applications; User Experience; Field Study

ACM Classification Keywords

H.5.2 User Interfaces: Mobile Dietary System


Shen is an HCI professor at the National University of Singapore working on realizing his vision of HeadsUp Computing, a new Interaction paradigm that can transform the way we live and interact with computers. In his free time, Shen loves to read, run, spend time with family and friends, and explore nature.

Leave a Reply