People depicted in static images the world over may soon be brought to life through augmented reality (AR). Researchers from the University of Washington have developed a new algorithm that is able to take a still, 2D photograph and give the people depicted in it the ability to walk, run, jump or dance. The tool, Photo Wake-Up, solves a critical challenge in the field of computer vision: how to overcome the fact that the visual input on this type of image comes from just one camera position.
How the “Magic” Happens
Unlocking locomotion for 2D characters, and making them appear 3D, requires some complex graphical manipulation. Photo Wake-Up first takes an image and analyzes it to highlight the outlines of the people to be animated. It then generates a 3D template that corresponds to the 2D silhouette and adjusts that template to mirror the subject’s body positioning in the original image. Next, it must flip the 3D computer generation back into 2D in order to actually make the movement happen. The researchers say it’s very difficult to animate a still image in 3D with precision.
Once this template is complete, the algorithm has to find a way to let the subject move without leaving the rest of the image looking unnatural. To do this, it takes information from the remainder of the image, the background in most cases, and splices together a “patch” it can use to occupy the space vacated by the subject. When Steph Curry runs in the video above, for instance, Photo Wake-Up is actually generating an artificial but matching background to fill the void behind him.
A Diverse Set of Potential Applications
As with most nascent technologies, the computer scientists’ first effort is not without limitations. For example, it’s not effective in animating objects that are not obviously human. Those humans also need to be facing forward with most of their body fully exposed. If someone’s legs are crossed or they are partially obscured by something in the foreground, it doesn’t work. Those developments will come in time, the team says, but even now, the list of potential uses for Photo Wake-Up is robust.
They imagine a world where visiting an art museum can reach a new level of interactivity, citing the example of, “sitting down for tea with Mona Lisa.” Or one in which budding art students could dream up their drawings with future locomotion in mind. That doesn’t even touch on the possibilities in areas like gaming, where players could build avatars that both look and move like they do. While further development is necessary, the possibilities for ensuing generations of Photo Wake-Up are enticing.