FiNC is a health-tech startup specialized in preventive healthcare leveraging Artificial Intelligence technology. With their mission “Personalized AI for everyone’s wellness”, they support people’s health not only in Japan but worldwide.
Technology Development Department
In May 2018, FiNC undertook a project from the Japanese Government. Japan Science and Technology Agency (JST) requested us to develop an AI model that analyzes nutritional values from an image of a meal.
They also gave us some requirements. “We want an AI model that can identify the nutritional values from a photo of a meal. It should be able to distinguish the ingredients and estimate the amount of each ingredient used at the same time, and calculate calories precisely.”
There already existed calorie value predictors that showed the standard calories from photos of meals using image recognition technology. However, in this project, JST wanted us to research and develop a refined AI model that can predict precise nutritional values. For example, if we look at photos of curry dishes, they may look very similar but the calories can differ depending on the ingredients like beef or pork. As far as we knew, nobody had yet developed such a precise AI model up to then.
At the time, I remember telling my manager “I think this will be an extremely challenging project”.
After many discussions, gradually I was able to see how we could make this possible. My plan was to detect each ingredients used in the meal from the photo, then calculate the calories accurately in detail. I thought it would be possible to achieve it by object detection once we had accumulated enough training data.
In order to calculate the actual nutritional values and calories, we need data that indicates the ingredients and the amount of each ingredient used in the meal. Our company already had a huge amount of data of meal images, but that did not include information about the amount of the ingredients.
So, we decided to start the project from actually cooking meals from scratch. We collected the data of ingredients and the amount of each ingredient used. Then, we went through a process called Annotation where we classified all the ingredients used in the photos of the cooked meals.
The goal was to cook 2,000 to 3,000 recipes, take photos of the cooked meals from various angles, and create training dataset that consists of a set of meal photos, names of ingredients on the photos, and the amount of each ingredient used in the meal. We had to do this in only 3 months, by the end of November 2018. That meant we had to cook 50 recipes per day.
Prior to the project, an acquaintance of mine introduced me a member of ABEJA. I told him about the project and he said “it’s an extremely tough project, one of the hardest projects I’ve ever handled” and it was no surprise. However, he added, “but we can try our best to achieve the objectives as much as possible.”
We chose ABEJA because of its wide support. ABEJA offered to support us with not only annotation but also the overall operation such as cooking and taking photos.
We are engineers, so we had no experience in carrying out cooking operations. What’s worse, we had to cook 2,000+ recipes in just 3 months. To be honest, I thought it would be impossible. Then, a project manager from ABEJA joined us who had a lot of experience in complex operations. We really appreciated his advice.
“Let’s contact ex-kitchen managers for the cooking part.”
“Let’s call this number for a chef outsourcing agency.”
“Let’s start off with cooking 10 recipes a day and see how it goes. Depending on how it works, we can do more.”
With the support by ABEJA’s PM, we planned the project together. Thanks to ABEJA, we started the project extremely smoothly. I was relieved and felt very positive.
The project started in full swing from September. We started gathering image data using our corporate kitchen and a studio. We had an experienced kitchen manager and a few part time assistants cooking 50 recipes each day. To secure more data per meal, we changed presentations and took 10 shots per each meal. Including cooking and shooting assistants, around 10 people were involved in the project each day.
Cooking and shooting 50 recipes per day was tough enough, but annotation to distinguish ingredients on the plate was also challenging. For example, in a Stir Fry dish, various vegetables are chopped and cooked together and they are on top of each other when it’s served. In case of curry, cooked vegetables don’t hold their shapes. Despite this, we had to specify if it’s cabbage, beansprout, carrot or meat. To annotate, we had to identify and mark the outlines of the ingredients precisely and colour each item. This process required unimaginable patience.
As we went along, occasionally we found some image data defective. Or, some days we couldn’t finish cooking 50 recipes. We discussed with the kitchen manager and improved the workflow of the cooking and the annotation process. We divided the process into small tasks - selecting the ingredients, chopping, getting the seasonings ready, cooking and dishing it up. We assigned one staff for each task which helped them focus on just one task and made the process more efficient.
ABEJA’s project manager also came up with an idea to improve the annotation process. Up to then, one annotator worked on one picture, coloring all the ingredients on it. It was time consuming and sometimes lacked accuracy. Instead, he suggested that we assign one annotator for each ingredient. One annotator colores just cabbage, another annotator colores just bean sprouts. As a result, the annotation cost went down and we had better accuracy.
After a series of trial and error, we were able to cook 50 recipes regularly each day by October 2018. By the end of November, we achieved the original goal of 2,000 recipes. In February 2019, we were able to submit the final research result report.
Although I started this project by myself, the project manager from ABEJA supported me from day 1 and worked along with me right until the end. He really empowered me all the way through.
“Serve people for their precious life” is FiNC’s corporate vision. We develop products that help people manage their health, e.g., a healthcare food diary app. Our vision accorded with the purpose of this R&D project to develop the AI model.
FiNC has an application product, and there is a function where AI distinguishes meal images and calculates the calories. However, it doesn’t recognize some images occasionally. When this happens, the users have to search and input the information themselves, and it’s the only way at the moment. I’ve always wanted to improve this recognition accuracy.
Through the R&D in this project, I’ve learnt the difficulties in each process and the cost it would take to make the refined model. As the next step, with this experience, my aim is to refine the function and bring it up to the level to be added in our product line-up as FiNC’s new solution.
Please feel free to ask anything about our products.
We will get back to you promptly.