Two Minute Papers: Perfect Virtual Hands – But At A Cost! Two Minute Papers: Virtual Characters Learn To Work Out … and Undergo Surgery Two Minute Papers: This is What Abraham Lincoln May Have Looked Like! Two Minute Papers: This AI Learned Boxing … with Serious Knockout Power! Two Minute Papers: Everybody Can Make Deepfakes Now! Two Minute Papers: AI Learns To Compute Game Physics In Microseconds! Two Minute Papers: DeepFake Detector AIs Are Good Too! Two Minute Papers: This AI Clones Your Voice After Listening for 5 Seconds Two Minute Papers: This AI Does Nothing In Games … And Still Wins! Two Minute Papers: OpenAI Five Beats World Champion DOTA2 Team 2-0! Two Minute Papers: 6 Life Lessons I Learned From AI Research Two Minute Papers: DeepMind’s AlphaStar Beats Humans 10-0 (or 1) Two Minute Papers: OpenAI Plays Hide and Seek … and Breaks The Game! Two Minute Papers: 4 Experiments Where the AI Outsmarted Its Creators Two Minute Papers: AI Learns to Animate Humanoids Two Minute Papers: Ken Burns Effect, Now In 3D! • Two Minute Papers: This AI Creates Human Faces From Your Sketches! • Two Minute Papers: Google’s New AI Puts Video Calls On Steroids! • Two Minute Papers: New AI Research Work Fixes Your Choppy Videos! • Two Minute Papers: How Does Deep Learning Work? #24 • Two Minute Papers: Can an AI Learn Lip Reading? • Two Minute Papers: Two Shots of Green Screen Please! • Two Minute Papers: How To Get Started With Machine Learning? #51 Two Minute Papers: This AI Creates Dessert Photos … and more! • Two Minute Papers: NVIDIA’s AI Dreams Up Imaginary Celebrities #207 • Two Minute Papers: Beautiful Gooey Simulations, Now 10 Times Faster • Two Minute Papers: DeepMind’s New AI Dreams Up Videos on Many Topics • Two Minute Papers: How Do Genetic Algorithms Work? #32 • Two Minute Papers: AI Makes 3D Models From Photos #122 • Two Minute Papers: What is De-Aging? • Two Minute Papers: This AI Made Me Look Like Obi-Wan Kenobi! • Two Minute Papers: DeepMind’s AI Learns Locomotion From Scratch | Two Minute Papers #190 Two Minute Papers: DeepMind’s WaveNet, 1000 Times Faster | Two Minute Papers #232 Two Minute Papers: This is How You Hack A Neural Network Two Minute Papers: We Can All Be Video Game Characters With This AI ★★★★★ Two Minute Papers: DeepMind’s New AI Helps Detecting Breast Cancer • Two Minute Papers: Artistic Style Transfer For Videos #68 •

Two Minute Papers: Artistic Style Transfer For Videos #68 •

In this AI video ...

Artificial neural networks were inspired by the human brain and simulate how neurons behave when they are shown a sensory input (e.g. images, sounds, etc). They are known to be excellent tools for image recognition, any many other problems beyond that – they also excel at weather predictions, breast cancer cell mitosis detection, brain image segmentation and toxicity prediction among many others. Deep learning means that we use an artificial neural network with multiple layers, making it even more powerful for more difficult tasks.

This time they have been shown to be apt at reproducing the artistic style of many famous painters, such as Vincent Van Gogh and Pablo Picasso among many others. All the user needs to do is provide an input photograph and a target image from which the artistic style will be learned.

And now, onto the next frontier: transferring artistic style to videos!

The paper “Artistic style transfer for videos” is available here. The implementation of this technique is also available here.

Recommended for you:
Deep Neural Network Learns Van Gogh’s Art
Deep Learning Program Learns to Paint: https://www.youtube.com/watch?v=UGAzi1QBVEg
From Doodles To Paintings With Deep Learning: https://www.youtube.com/watch?v=jMZqxfTls-0

Sintel Movie copyright: Blender Foundation https://durian.blender.org/sharing/

The thumbnail background image was taken from the corresponding paper.
Splash screen/thumbnail design: Felícia Fehér – http://felicia.hu

Video Transcript

This transcript was generated by an AI at Otter.ai

Dear Fellow Scholars,

this is Two Minute Papers with Károly Zsolnai-Fehér. Here we have previously talked about a technique that used a deep neural network to transfer the artistic style of a painting to any arbitrary image, for instance to a photograph. As always, if you’re not familiar with some of these terms, we have discussed them in previous episodes and links are available in the YouTube description, make sure to check them out.

Style transfer is possible on still images but as there is currently no technique to apply this to videos, it is hopefully abundantly clear that a lot of potential still lies dormant inside. But can we apply this artistic style transfer to videos? Would it work if we would simply try? For an experienced researcher it is blatantly obvious that it’s an understatement to say that it wouldn’t work. It would fail in a spectacular manner, as you can see here, but with this new technique, it apparently works quite well.

To be frank, the results look gorgeous. So how does it work? Now don’t be afraid you’ll be presented with a concise but deliberately obscure statement. This technique preserves temporal coherence when applying the artistic style by incorporating the optical flow of the input video. Now the only question is what temporal coherence and optical flow means. Temporal coherence is a term that was used by physicists to describe for instance, how the behavior of a wave of light changes or stays the same if we observe it at different times. In computer graphics, it is also an important term because oftentimes, we have techniques that we can apply to one image, but not necessarily to a video, because the behavior of the technique changes drastically from frame to frame introducing a disturbing flickering effect that you can see in this video.

Here, we have the same if we do the artistic style transfer. Because there is no communication between the individual images of the video, the technique has no idea that most of the time we’re looking at the same things. And if so their artistic style would have to be applied the same way over and over to these regions, we are clearly lacking temporal coherence. Now, onto optical flows. Imagine a flying drone that takes a series of photographs while hovering and looking around the boss. To write sophisticated navigation algorithms, the drone would have to know which object is which across many of these photographs. If we have slightly turned most of what we see is the same and only a small part of this new image is new information. But the computer doesn’t know that as all it sees is a bunch of pixels. Optical flow algorithms help us achieving this by describing the possible motions that give us photograph B from photograph a in this application.

What this means is that there is some interframe communication the algorithm will know that if I color this person this way a moment ago, I cannot drastically change the style of that region on a whim. It is now easy to see why naively applying such techniques to many individual frames would be a flippant attempt to create beautiful smooth looking videos. So now it hopefully makes a bit more sense. This technique preserves temporal coherence when applying the artistic style by incorporating the optical flow of the input video.

Such great progress in so little time, loving it. Thanks for watching and for your generous support and I’ll see you next time.

YouTube Source for this AI/Deep Learning/Research Video

AI video(s) you might be interested in …