Microsoft's New AI: Virtual Humans Became Real! 🤯
ฝัง
- เผยแพร่เมื่อ 23 ก.ค. 2024
- ❤️ Check out Runway and try it for free here: runwayml.com/papers/
📝 The paper "3D Face Reconstruction with Dense Landmarks" is available here:
microsoft.github.io/DenseLand...
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Balfanz, Alex Haro, Andrew Melnychuk, Benji Rabhan, Bryan Learn, B Shang, Christian Ahlin, Eric Martel, Geronimo Moralez, Gordon Child, Ivo Galic, Jace O'Brien, Jack Lukic, John Le, Jonas, Jonathan, Kenneth Davis, Klaus Busse, Kyle Davis, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Michael Albrecht, Michael Tedder, Nevin Spoljaric, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Rajarshi Nigam, Ramsey Elbasheer, Steef, Taras Bobrovytsky, Ted Johnson, Thomas Krcmar, Timothy Sum Hon Mun, Torsten Reil, Tybie Fitzhugh, Ueli Gallizzi.
If you wish to appear here or pick up other perks, click here: / twominutepapers
Chapters:
0:00 - Teaser
0:19 - Use virtual worlds!
0:39 Is that a good idea?
1:28 Does this really work?
1:51 Now 10 times more!
2:13 Previous method
2:35 New method
3:15 It gets better!
3:52 From simulation to reality
4:35 "Gloves"
5:07 How fast is it?
5:35 VS Apple's ARKit
6:25 Application to DeepFakes
Thumbnail background design: Felícia Zsolnai-Fehér - felicia.hu
Károly Zsolnai-Fehér's links:
Instagram: / twominutepapers
Twitter: / twominutepapers
Web: cg.tuwien.ac.at/~zsolnai/ - วิทยาศาสตร์และเทคโนโลยี
![THE TOYS x NONT TANONT - ดอกไม้ที่รอฝน (spring) [Official MV]](http://i.ytimg.com/vi/pPa1d5cC8M4/mqdefault.jpg)
![รวมเพลงลูกทุ่ง 6 หนุ่มไทบ้าน ฮิตติดกระแส ชุดที่ 1 l มนต์แคน ไมค์ ไผ่ เสถียร ไหมไทย พี [Longplay]](http://i.ytimg.com/vi/GD2v1OjMw5I/mqdefault.jpg)
"...even the movement of our nostrils will be translated. What a time to be alive!" - Dr. Károly Zsolnai-Fehér, nose enthusiast
or a time to not be alive...
@@user-wq9mw2xz3j To phrase it slightly less darkly, 'What a time to be virtually alive!'.
Please let it not track what’s coming out of the nose during winter… I’d prefer to be blissfully ignorant of that at least in VR! ;)
I just hope this technique won't be weaponize by some evil gov to make fake thing to smear others, instead, should use it in a good & fun way.
So that's how his name is spelled.
Is there a better channel than demonstrates bleeding edge technology?
No, this is it. If you’re not subscribed, you’re missing out.
You are too kind Phil, thank you so much! 🙏
@@TwoMinutePapers no, thank you. I mean it. Nobody else does it like Two Minute Papers.
Absolutely correct
But he didn't talk about Stable Diffusion yet.
I have not seen one. If you want to get ahead of the competition, especially when it comes to multimedia, just follow this channel to see what's coming down the pike.
Wow, can you imagine movies or games based on historical figures being reproduced by photo real 3d actors? This will be wild. Also terrifying that AI programs will probably be used for people to recreate and interact with digital versions of their dead spouses or children, this might mess some peoples heads up.
Imagine if we could some how train AI based popular literature and have it recreate the authors personality
@@Rctdcttecededtef I mean it can already imitate the writing style they used in their books, if you have a bunch of personal writings from a person I don’t see a reason you couldn’t generate text that matches them.
If you ask me...
In order to lose your sanity due to virtual world, it must already be lost.
Most commonly, being superstitious and or religious.
@@Numbabu that's a much better explanation of what I was imagining
@@GreatMCGamer that's cool but what if you're brought up under this tech since 0 years of age
We'll need a minimum of 10 more years of training for it to capture Jim Carrey's expressions accurately. It'll most likely be way to confused
He'll be the stress test for the technology. Kind of like the meme 'can it run Cryengine?', but it'll be 'but can it do Jim Carrey?'.
5:40 you could use this to insert yourself into a game and actually talk as yourself (not just for VR)... Imagine having virtual conversations with other players in a 3rd personal online game.. where other players talk realistically
Star citizen already does this. Called face over IP (FOIP)
I can guarantee you that people will just meme the shit out of each other
Player customization will be more advanced soon hopefully
Star Citizen has this. Though its its a little outdated but they could totally update it with an improved version of something like this. current face tracking is goofy but does work
Wow a new tool to commit fraud with!
Love this channel. Wish more were as upbeat as yours ... Cheers.
make one!
It sounds slightly hyperbolic and too whispery/aspirated at times though. It's a bit distracting when you constantly feel like it's a bit ad like.
3:38 I like how confused the AI is when the lips are behind the tea cup.
Input Video: "Siiip"
AI: "schlop schlop schlop?"
If only this was on the market last year when I spent 6 intense months tracking dots on faces on mo-cap actors for a video game!
There was so much manual cleanup and fixing because the captured facecam data wasn't always great, it was a lot like 2:19
I want to do what you do when I grow up, but I don't know what you are doing and where I can start 😂
Well here's your chance to do something with this paper. Create a company and offer your skills in facial tracking. If you're cheaper than you are right now, you'll make big bucks!
In time you could become very proficient in this type of motion capture. Let's say it's weird with capturing blinking. Well "just" write an algo that detects when the top and bottom of the eye-lids are close to each other, in order to detect blinking. Then delete the blinking data from ~0.3 seconds before that detection until ~0.3 seconds after, and replace it with a standard blinking motion. Stuff like that will be worth gold, since you'll be able to do so much more work so much faster than industry standards.
“But can it track Jim Carry” feels like the “but can it run Crysis” of face tracking solutions lol
Dear Fellow Scholars, we have a new sponsor in our ranks! Make sure to try their amazing AI-assisted editor here: runwayml.com/papers/
Sponsorship WOOOOO!
This could take animation to another level. Nvidia's Audio2Face can already take text and generate emotions. Avatars that are indistinguishable from humans is probably not more than a year or two down the line.
indistinguishable through a casual look, sure, but algorithmically generated faces also generate data artifacts which can be traced by software
@@theamici Yes, totally perfect deep fakes are not yet close but the "video feed" you send to that Teams meeting can be easily virtual presentation of yourself in perfect lighting in a studio and nice clothing where ever you're actually attending from.
@@MikkoRantalainen or a president is seen on video doing something or saying something he didn’t.
Scary Barry
@@Juanpopspacks Yes, the general public should be educated that a video can lie worth 10000 words.
This tech would be extremely useful for streaming: from Vtubers to facial tracking for green screens!
I could also see this being used as a kind of encoder-decoder pair for telepresence applications, where one’s head and shoulders are sent as position data rather than a high resolution image, and then reconstructed on the receiver side for high speed, high fidelity, lighting insensitive video calls!
It would also be useful to be used in the reverse for more accurate facial expressions of characters in video games. Collecting enough data, and turning it all into vector space, and thus being able to perform more human expressions.
There have been a lot of games throughout the year that have promised "realistic" facial expressions, but none of them have been up to par, with this they could quite easily use the data to improve that.
Now the only thing that needs work is lighting of non-static objects.
nice idea
That teleconference idea exists already - I think, if I remember correctly, by Nvidia. This channel did a video on it a couple of years ago.
Here it is th-cam.com/video/dVa1xRaHTA0/w-d-xo.html
@@livedandletdie oh ur 100% right about that
But we probably won't see it any time soon
The face tracking technology (without tracking points) is getting wild! The digital animation workflow is going to be sped up two fold in the next year! (Or at least, I think)
possible uses:
motion capture
facial animation
facial sentiment detection (embedding, augmentation)
facial recognition
automated lip reading (embedding, augmentation)
speech audio-to-animation (data augmentation)
eye tracking
I can't wait until you can walk around in an open-world game map in VR and engage any NPC in conversation and the AI will generate in it an actual personality with unique character traits - though the player will be able to set parameters barring types they don't want to run into!
Definitely what I hope will happen too!
so basically an echo chamber?
@@proloycodes If you want to set your parameters for provocative conversations or constant arguments go right ahead!
No, if the AI is smart enough to do that, you’ll have absolutely zero say in what it’s personality is. Lol
Ladies and gentlemen, an ERP junkie.
What a time to be *AI*
This is literally another one of those papers that I've been waiting for I do work in animation i got my degree in 2d animation and this tool along with various other things that are coming out are going to revolutionize the animation more than it already is
How does a 3D tool help with 2D? On a side note I doubt anything made by A.I will ever look as good as Akira or Ghibli stuff.
@@AhmedAli-kt1ez 3D models made to look 2D
@@AhmedAli-kt1ez the data can easily be flattened to 2D. Actually they already are in the end
is there a way to export blender files from it or do you know how to use it ? :)
@@lovejsouvzacne this is a real-time image to facial feature. I don’t think it handle the mesh generation. Also if it would it would be to industry standard format most likely and not Blender format
VR team meetings where you don't need a fake avatar.
Mind blowing.
extremely impressive. is this facial performance capture now good enough for real movie-making? also it would be very useful to see the motion being targeted onto animations, especially animals.
Probably not if you can't tweak the results but it won't be long
It's actually even more impressive than what was stated in the video, because it's not only running on the CPU, it's running on one CPU core
If they gave it depth too then pretty sure it would
the result from motion capture often are jagged, and the result will be baked on every frame,
it's hard to clean it up , unless if they somehow invent the way to clean up the jagged in mocap.
@@jensenraylight8011 yes, cleaning up mocap has always been a total nightmare - very expensive and very tedious. if AI can clean up mocap and get rid of the unnatural jinks and jumps then that would be a huge leap forwards. recently, i have seen improvements, especially in fixing footsliding. maybe karoly find a paper that finally really fixes messy mocap.
I wonder if expensive stuff you need for mocap will have their prices go down once these AIs reach a point where they’re on par and available to the general public
It's going to happen the same thing that happened with the GPS market. It used to be very expensive to own one and there was a yearly fee for updated maps. Now we've got Google Maps for free on our phones!
This is a huge W for the gaming industry.
You know what would be really cool on a testing program like this? Use some glass materials to block people's faces in various ways, and film them in visible light and in IR, and then let the AI work with the IR footage to figure things out, meaning that the person's face is occluded, but then you would have visual footage from the same angle that you could overlay, so that you could see what it got right and got wrong perfectly.
I suppose that footage could also be used for training the AI to correct it's mistakes.
And then measure the gaps for every data-point it got wrong, and then optimize those gaps via feeding them as training data.
Really love your video summaries. Excellent work. Thanks for posting!
Always love the reveal with "hold onto your papers!" or even the SqUeEzE your papers on this one. :) Great content as always, thank you!
Even slight misplacement of virtual face from real face is noticed by our brains
This is going to be great for XR applications!
Combine this tech with 3D animation, Voice synthesis, AI script writing, AI story creation, AI generated artwork, targeted advertising, cameras in theatres to gage response and interest.
We can have "Hollywood in a box"
and potentially add the tech of Brain Machine Interfacing,(the first of which has become commercially available very recently) we could even eliminate theatres and get better response from the audience.
Of course, we all know what this will ACTUALLY be used for...
Add to that CRISPR babies and the enormous benefits it has and in a couple of generations human beings as we know them will be a thing of the past. It's weird, almost every tech development , from self driving cars, boston dynamic's robots, midjourney AI, 3D printing etc is replacing physical labour. It's like man's answer to the Garden of Eden. We don't need God to go back there. We can counter "in the sweat of your brow" with and machine server power.
The writing is on the wall. I mean, when you have silicon valley intrepreneurs waving normal human beings as a burden and glorifying a "new designed haminity" then we are pretty much damned.
This is great because image based tracking is always cheaper then any special sensors. Now nobody will need expensive headsets or iphone sensors for good face tracking.
Your enthusiasm is infectious.
Does anyone know if this is available to the public? Like, can I download and run this model with the pretrained weights?
Wow, that’s amazing. ARKit still does some things better, because of the depth data. For example, in those side-by-side videos, you can see that ARKit better tracks the lips, around the eyes, and nose/nostrils. It’s the “nooks & crannies” of our faces that depth sensing can better interpret. The color algorithms are getting better & better though! They are still just a little more jiggly compared to color + depth. Thanks for sharing this cool video!
because this works so well with obstructions, this might be useful for vrchat in the future...
You could use this for real time augmented reality layovers of someone's virtual avatar over their face, that could be pretty cool.
You make me squeeze the papers so hard they turned into Diamane
Started with the uncanny valley in the simulations and then somehow got every more unsettling with the wireframe robo-masks
I'm excited to see this easier&higher quality capture in Vtubing
This is outstanding, can't wait to try this new thing.
1:15 Baby wake up, a new NFT collection just dropped!
Nice Nice, seems like 2020-2030 will be a decade where AI will become common, in a big way, in software runing on desktop, laptops, smartphones.
It's time for a lot of new cool software that will increase productivity and enable us to make new things that were previusly imposibile to make.
Great recession with global unemployment.
@@rettenthetetlen8759 exactly, and ultimately a big war, but there is a hope, a communist revolution making obsolete money, market, wage work, exploitment, capital
In the 1970's German television had a movie "Welt am Draht" (World on a wire), where computer experts created a virtual world and were able to enter this. One of the virtual beings found a way in the real world and said, that this isn't the real world yet, finally it came, that the computer experts of the real world gave the technology to the virtual people.
At least, the concept of virtual world is something, where we go to far.
Whenever I watch one of these with virtual humans in it, I can't help feel disapointed that we'll still have to wait so much longer for the same to be possible with virtual not-humans. One day though we will be able to get an AI to generate animated or realistic characters of any kind from humans to fantasy creatures to aliens and beyond all with the same algorithm. And maybe ten papers down the line that same AI would be able to study a video of godzilla and create him. And one day all of this will lead to the destination I dream of.
This is already possible today. It’s just what mesh you want to use as output. Instagram etc snapshat filters do this.
Yeah… Those with the craziest fetishes will definitely be satisfied.
Wonderful as always
Said it before, we're right in the age that's making all the building blocks for what will one day be common a place Holodeck-like technology.
any info on when this will become a product i can purchase? any clear date on the horizon?
The fact, that news outlets are using this technology, is troubling.
We need good vr hand tracking!!
There is a sample for testing ir the source code si available?
I see what pushes you forward, the rewrite of the Game of Thrones final series! I cannot wait
I've thought for years that AI will eventually be able to break down movies and TV shows into it's elements, including actor appearances, voice and acting styles. This could be used as part of that by capturing the facial expressions of actors. One thing I see that could be improved is by identifying expression anchor points of different faces and people. Topographical maps isn't enough when it comes to faces. People typically have different points on their faces where expressions seem to radiate from. It's probably tied to either muscle anchor points or just how different people learn to control their facial muscles as they grow up.
Omg seeing Mona Lisa move like that was awesome! I love technology ❤️
Let's use it for VR remote conferences.
If it can track Jim Carrey, it can track everyone
"what a time to be alive!" is my favorite catchphrase
love your videos! amazing tech!
I love your delivery!
Signed up on Runway, excited to see what it can do.
It's still super weird to me that you can train and AI on computer-generated data... it's like the computers can teach themselves based off of their own imaginations... Love it! and thank you!
It's important to mention that the entire dataset is procedurally generated, but not via an AI. Thus it's 100% precise.
@@spideyninja Fair enough. :)
AlphaZero (Go software with superhuman performance) was also completely trained on synthetic data.
so is this available to download yet to test?
Look forward to converting scanned faces to a miniature model, so that players can literally play as themselves in DnD
I must say that ideas like using data generation from virtual environment like using footage from virtual environment for training driving AI is just something that I haven't heard of but it so much on the surface, considering that you get both structured and unstructured data distributions at the same time, and very little work is needed by human. If someone knows papers on successful application of virtual environment used for training (outside of RL of course, I am talking more CV), please link it here, would really appreciate!
Amazing work. I love it :*
Let be honest folks: I know that y'all wanna a massive open world RPG like Skyrim where every single NPC is lifelike and totally unique and will pass the Turing test blindfolded.
your completely right
I mean, yes, I do, but we are so far from that. Look at Evermore: it's basically a real life version of this, with real actors playing the parts of NPCs, and you *still* can't get anything approaching an authentic experience. Even *actual people* are no good at passing the Turing test in a fictional setting.
Dr! I just saw you are presenting at GTC this year! Signed up immediately for your presentation when I saw your name
I wasn't ready for the Two Minute Papers face reveal, but I'm glad i now have a face to the soothing voice!
Get these virtual fellers hooked up with a gpt3 brain, stick em in gta v, and boom simulation confirmed
Regarding deep-fakes-.. Imagine this, having a photo of a relative or someone who's passed, or a dog, and preset movements- or ai to make them more life-like. Sortof like the pictures and paintings in Harry Potter. Digital frames already exist- but "magical" moving people frames don't. Fun idea for anyone savvy out there!
I'm gonna use this in the Metaverse.
0:33 This makes me think that we are living in a simulation. Dang.
almost makes me wonder if they couldn't just use existing data to determine what's behind the obstructions. If it can match a face to an existing model, and can position that over the tracking data, then use points on the model to realign those it can't see. So then it should match pretty well. Obviously it's less than trivial to program, but I think, the answer to tracking things that are obstructed is simply to use similar models and track them in the parts they can't see.
6:23 in sync with the narrator! :D Amaaaazing! lol
6:40 and 7:00 - can we use this nowadays? or is this some concept? care to share a link or name of the git repository / paper? I love the possibility to move some face inside a photo using my own face
Amazing! but i didn't get where I can find this tech to try it???
So will we be seeing cameras auto-orienting a person's avatar so that they look like their staring into a camera when it's actually off-center? I could see the system calculating the center of a screen-partner's face as the center of where a camera should be, and adjusting the avatar of the caller (so it seems to be oriented as though the camera were placed there). It could take the communication awkwardness out of video calls and teleconferences. Moreover, I think these intereractions are like talking to a person who avoids eye contact, essentially undermining confidence and hindering the adoption of video conferencing at a psychological level.
If that's the only think you want to fix, half mirror in front of your camera and teleprompter-like screen setup is all you need.
However, if you want to show yourself in perfect lighting and some specific clothing and background, then you need the tech from this video.
Elképesztő lett a kontent, csak így tovább!❤
Nem simán kontent! Valódi mondanivaló. ;)
when is this available? Will it just be an app on a phone?
Great content, thank you
How can we use this stuff?
I think we all know what people are going to use this for
?
Big facts 😂🤌
My goodness! Love it!
I can't hold onto my papers! My papers flew out of my hands before the video even started!
can't believe humans are real now. terrifying.
very interesting concept!
When I see this sort of thing I immediately think of the benefits it would have for visual effects in movies, specifically in regards to augmenting an actor's face with some CG _without even needing to place dots on their faces anymore!_ Think of the old TV shows like Star Trek The Next Generation where characters like Worf, the Klingon, had to endure 3 or more hours of makeup every single day. Using the above technology, Michael Dorn could just come into work wearing his Starfleet costume and the rest would be done in post! And the best part is that it would actually look BETTER. Even up close shots would likely be perfect, especially once this technology is further developed.
What a time to be alive!
AI research always impresses me. As a dreamer myself, I see every new paper as a new tool that could be used for virtual worlds. Now I would love to see what AI research could do with discrete BCIs like Neuralink or Synchron. I feel like BCI research has been quite isolated from other fields outside of the startup world, and it could gain from exploring neural nets a bit more (especially with the results we saw from Neuralink).
5:24 "Squeeze that paper" ...... When holding on to your papers just doesn't cut it anymore.
Id love to use it for facial tracking simulations, where the face is a real persons, but Ai virtual.
Im sure that could be a method that would create better facial capture tracks.
any extended research done on lip reading ?
the question is , where can i get it and how do i use it on blender and so on,
Naon jó lett.Csak így tovább. Imadom a videoidat.
wow, i was looking in all internet some appropiate topic regarding virtual costumes for filmmakers... wow, it's insane... this will be a serios plus for filmmakers's buget... Waiting for unreal engine to implement all this stuff soon
08:09 "instantly remove unwanted objects" removes humans 😂
Lets hope this is open source or it somehow finds its way over to the boys at Epic. This algorithm needs to be implemented in live link face like, yesterday. Imagine metahumans with this level of mocap fidelity.
It’s Microsoft, so who knows. They probably patented it, and likely won’t open source the most critical parts
But the technique seems to have some temporal inconsistency when occlusion happens. Would be awesome to fix that too.
This is really a behind the scenes for Grand Theft Auto 6
Facial recognition
The dream tool of all governments....
As I make characters in Character Creator.. lol this is amazing.
Awesome where do I download it?
Hi Karoly,
I have a degree in maths and went into teaching. What would you recommend to someone looking to get into AI? I'm hoping to transition but have little coding experience.
I find every video you produce fascinating and want to get on the other side of the curtain.
Thank you for your efforts. Ben
Unfortunately, I think that coding is required to work in AI related fields. Personally, I'd recommend you to read a few technical books about AI, just to make sure that you really want to switch from teaching to AI, and to get some basics in programming. I think that "Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow" by Aurelien Geron is a great start. Just be sure to have the second edition and not the first, because the first edition uses an outdated python library. In any case, you'll also have to learn to code in python, and there are probably a lot of books or tutorials online for that as well.
@@slangster42 Lovely reply. Thank you.
I've started teaching myself python by doing the first 20 ish Euler problems.
Thank you for the recommendation. I'll check it out :)
Imagine having to undergo surgery. Your body get scanned, as well as your brain under different circumstances like joy, rest, pain etc. Your full body scan including all the interior tissue gets fed into an AI operated robot surgeon. It analyses all the data, figures out the best procedure for this surgery to be a succes, with minimal incisions, movements and risks. The AI then explains the whole procedure to the doctors involved, so they understand what it is going to do. It monitors your brain patterns to be sure you're actually under anesthesia during the procedure. Difficult procedures that once took many hours and involved huge risks, could now be done in 45 minutes, with next to no risk involved. Doctors would still be in the room, to intervein when necessary, as one of many failsafe systems of course.