Computer scientists at the University of Bonn have developed a self-learning computer program that can predict, with startling accuracy, what will happen in the future based on a typical sequence of actions.
The program first learns a set of consistent actions, such as making a salad, from video sequences. Based on the patterns it learned from the videos, it can predict what a chef would do next when watching a new video.
The scientists inputted 40 videos of human chefs preparing different salads. The videos had an average of 20 different actions over a span of about six minutes and also contained exact details of what time each action started and its duration.
After four hours of watching salads being tossed, the algorithm had learned which actions typically follow others during the task and how long each action took. It also had to account for the variations of individual chef styles and differences in recipes.
Then the team, led by Dr. Jürgen Gall, tested how well the program learned to make salad by showing it new videos and asking it to anticipate what would happen next. The computer was told what is shown in the first 20 or 30 percent of one of the new videos. It then had to predict what would happen during the rest of the film.
The program did surprisingly well on the test. “Accuracy was over 40 percent for short forecast periods,” said Dr. Gall. However, the accuracy dropped significantly the farther the algorithm had to look into the future; for steps that were more than three minutes in the future, the computer was right only 15 percent of the time. In addition, the algorithm performed noticeably worse if it had to recognize on its own what happened in the first part of the video, rather than being told it was about preparing a new salad.
Gall and his colleagues want the study to be understood only as a first step into the new field of activity prediction. One of the factors the researchers admit they need to work on is better analyzing “noisy” data—filtering out valuable data from additional irrelevant information. “Our process does work with it,” he said. “But unfortunately nowhere near as well.”
Once the program’s predictive capabilities become more precise, it could lead to the development of software and robots that can anticipate a user’s needs— including that long-sought-after robotic butler that can pass the salt and pepper before you even have to ask.
Read more about developments in machine learning at Autonomous Vehicles Armed with Machine Learning Algorithms.