A new algorithm from MIT researchers could soon apply signature photographic styles to your cell phone photos.
The success of renown portrait photographers is often attributed to a unique visual styles, sometimes accomplished through careful tinkering of studio lighting. Now, MIT researchers along with colleagues from Adobe Systems and the University of Virginia have developed an algorithm to mimic signature styles and apply them to photos and videos. Think of it as a much more impressive version of the filters we use daily on Instagram and other photo editing applications.
The process is called style transfer and is currently a hot topic for graphics research. While methods for style transfer have long existed, they weren’t great with close-ups of human faces.
YiChang Shih, an MIT graduate student in electrical engineering and computer science explains why:
Most previous methods are global: From this example, you figure out some global parameters, like exposure, color shift, global contrast. We started with those filters but just found that they didn’t work well with human faces. Our eyes are so sensitive to human faces. We’re just intolerant to any minor errors.
Shih and his collaborators instead developed what he describes as a “local transfer.” What this amounts to is pairing an initial photo with a photo of the desired style to create the final outcome. When the results still weren’t natural, they added multi-scale matching to their algorithm to ensure that both large and small elements are respected.
“Human faces consist of textures of different scales,” Shih says. “You want the small scale — which corresponds to face pores and hairs — to be similar, but you also want the large scale to be similar — like nose, mouth, lighting.”
To avoid the problem of undoing modifications made at one scale while changing another, the algorithm makes a Laplacian pyramid, a graphical representation that allows it to identify characteristics distinctive of different scales and concentrate modifications on those areas. One of the more exciting revelations is that it translates quite well to video.
While the techniques work best when you start out with two well-paired photos, it has been producing solid results overall. The next steps for the technology include figuring out the best consumer applications, given the upgrade over conventional filtration techniques. Robert Bailey, senior innovator at Adobe’s Disruptive Innovation Group and former director of design at Picasa agrees: “You can’t get stylizations that are this strong with those kinds of filters. You can increase the contrast, you can make it look grungy, but you’re not going to fundamentally be able to change the lighting effect on the face.”
It’ll be interesting to see how creative people can get. Besides better selfies that will inevitably shared on social media, it’s looking like this could be a new tool for photographers and filmmakers.