Deferred Neural Rendering: Image Synthesis using Neural Textures (SIGGRAPH 2019)

SIGGRAPH 2019 Technical Paper Video
Project Page: niessnerlab.org/projects/thie...
Paper Abstract
The modern computer graphics pipeline can synthesize images at remarkable visual quality; however, it requires well-defined, high-quality 3D content as input. In this work, we explore the use of imperfect 3D content, for instance,obtained from photo-metric reconstructions with noisy and incomplete surface geometry, while still aiming to produce photo-realistic (re-)renderings. To address this challenging problem, we introduce Deferred Neural Rendering, a new paradigm for image synthesis that combines the traditional graphics pipeline with learnable components. Specifically, we propose Neural Textures, which are learned feature maps that are trained as part of the scene capture process. Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline. Both neural textures and deferred neural renderer are trained end-to-end, enabling us to synthesize photo-realistic images even when the original 3D content was imperfect. In contrast to traditional, black-box 2D generative neural networks, our 3D representation gives us explicit control over the generated output, and allows for a wide range of application domains. For instance, we can synthesize temporally-consistent video re-renderings of recorded 3D scenes as our representation is inherently embedded in 3D space. This way, neural textures can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates. We show the effectiveness of our approach in several experiments on novel view synthesis, scene editing, and facial reenactment, and compare to state-of-the-art approaches that leverage the standard graphics pipeline as well as conventional generative neural networks.

[@jakeniemi6533]

This is super impressive. The last example was simply mind-boggling, how accurate your approach is considering how little information was available in the input (bounding box). Simply incredible!

[@RuofeiDu]

Great work!

[@u3k1m6]

Spooky. I want to see the comments when this video blows up

[@aniareyouok]

Wonderful work, please consider releasing the models and training data as open source!

[@MysteryPancake]

is it possible to duplicate humans and other moving objects as well?

[@Ben-rz9cf]

can you make a plugin that can output the textured geometry? i need it for workflow purposes

[@gustavosantos106]

WWW3 incoming! :D

[@wiccanconservative]

This tech is scary... oh the possibilities.

[@attiliohollige]

Realities

[@011THEWALL]

Wow amazing. How can I get my hands on this tech to demo it?

[@thomaat460]

It's software and this particular software isn't publicly released in a state that anyone but a computer scientist can make it work.
There's DeepFake too though. Albeit of slightly different quality, it essentially does the same. There's some applications (like FakeApp) that allow any person with a bit of computer knowledge to change faces.

[@rallyworld3417]

Wait sid it also generated mocing hands

[@user-ux8iv1zk5e]

这个是模拟生成嘴部的吗？

[@llermy3371]

不，模拟生成嘴部是他们方法的其中一个应用而已

[@Mooncake69420]

Okay I'm just going to come out and say it. It's impressive but there is no chance in HELL this could ever fool anyone. At least not at the current level it's at

[@RizzlePrivate]

You are either underestimating peoples stupidity or overestimating peoples intelligence. This would definitely fool most people when used in ways that aren't too absurd. You might be able to spot the difference, but you would have to be critical of every image you see and scrutinise them. Most people just don't work that way.

[@attiliohollige]

@@RizzlePrivate It's fooling billions of people right now imo

[@tupacandeazyefan]

please tell me how you can tell apart the two balls for example

[@TheBanjoShowOfficial]

when you are not consciously thinking that a video may be deepfaked, you are far far far more likely to be fooled by it than if you had previous knowledge of it being so. you overestimate your conscious abilities.