A pokemon trainer game where you're telling your pokemon where to move to in the battlefield and which moves to use. "Pikachu, dodge right by using quick attack!" just like in the cartoons.
This suggests a future where a game is powered by an LLM in the background, guiding NPCs and generating complex worlds. I can envision a scenario where an audience gathers in a theater, shouting commands to the characters, and the story unfolds in real time based on their input.
What if the play time for GTA VII grinds into the infinite because of all the lore, world building, side quests that are generated by talking to a random NPC.😅
Entirely AI generated, from environment, movement, voices, characters, and stories with dynamic changes. It's going to be amazing. Sci-fi, historical, fantasy, training, simulator, real-world, you name it.
@@hao21291 Military robotics don't tend to rely on complex armatures that would benefit from the kind of context aware kinematics you're picturing; It doesn't need to look like a person to mount a weapon on it, or to defuse/detonate an explosive, and there's not really advantage in doing so.
It is called inverse Kinematics for robotics! Dude I've been ranting about why we don't have this tech after seeing a tiny 3 seconds sound bite of something similar. I didn't have a name for it. Now I do! This is awesome. This is the future of video games. Imagine having a sports game where tackling is realistic. Or having the scale and weight of playing a super human like an Astartes would feel real. I called it Ragpuppet instead of Ragdoll, lol. This is what the actual next leap in gaming is. What consoles were made for. Having a dedicated hardware to do this like an NPU or physics card sounds like a no brainer for Microsoft and Sony, along with "AI".
I can't wait to have games where the player and the NPCs are smoothly animated based on the context they're in. Games where the NPCs have a personality but they can basically do and say anything. A game that uses ray tracing to have realistic light and AI text generation for the dialogs, AI text to speech for the NPC voices, AI decision making for the NPCs, AI animated movements, ...The possibilities are nearly endless.
Can’t wait for this tech three or four papers down the line! This would be awesome for making cutscenes in a videogame instead of doing expensive motion capture.
Martial artist here. Most martial arts put heavy emphasis on footwork. Often you will have new people only practice footwork, because little else matters until the footwork is first figured out. If you think about it, our feet and legs are our primary ways of interacting with gravity and gravity is omnipresent and eternal. So nearly every action you do actually starts in you feet. So maybe a future solution for an animation ai system would be to break the training down into chunks. First do feet and legs only, then add a torso and head, then finally add arms. Just a thought.
You cannot train the feet on their own and expect to perform just as well if you add a torso or whole body to it. The feets are the primary interface with the ground, therefore the most important element in our interaction with gravity, but our body has its own weight, inertia, force, etc... and you need to have that there while training the feet so they can take it into account.
I feel like stability on one's feet should be fairly simple for AI, you can see Boston Dynamics robots holding themselves upright reliably because falling is a pretty obvious problem to detect and then solve for avoiding
I think that has a lot to do with the way humans learn and our attention span and focus. I would imagine AI to learn better with the whole system and more iterations.
@ Biped Incorrect. A lot of martial techniques can just be done with the feet. The issue is just that gravity is the main factor in a fight and in life. For the most part we use our legs and feet to resist gravity and to reposition. If you want an example a good one would be a bullfighter. Their main technique is to sidestep their opponent. Emphasis on step. Footwork simply matters more, that's why you practice it the most.
@ I'm not talking about stability, I'm taking about strategic positioning and footwork. Sure, you need balance but what's more important is simply where you place your feet in anticipation of an attack and to set yourself up to launch a counter attack.
Imagine seeing npcs just walking around and doing things like real people? Not awkwardly transitioning from animations, but making up their own decisions on the fly. Especially if the characters can be given tasks, i imagine something like chatgpt could give it a full day's schedule of tasks that update as it meets the objectives, adding infinite variation on what they are doing and how they do it. Plus, if npc dialogue can begin using realistic voices and responses that stay in character and feel natural, we are really getting close to something cool and scary
@@opitopit2105 considering that most AI technology wasn't possible in 2018, I'd say things are moving pretty fast. I'm not sure where we're headed, but it will likely happen quickly. Even AI videos are brand new. Two years ago it was almost impossible to get something that was kinda real looking.
This could probably even be used to show a model how you can move AND probably by filming you giving you physio training to expand your movement range, etc.
With an EEG cap or some other method of reading brain activity, you could combine that with this to pretty much allow players to guide their character's internal thoughts and everything else is generated. The player wants to walk through a doorway, and their wanted action becomes their AI-counterparts thoughts, and eventual actions. Not much button pressing and would allow for quicker interfacing and control over a character where new animations could come up for any situation.
Having this and other Ai functions could be so amazing in games especially mmo’s. You could probably even retrofit old mmos with an interactive ai model, allowing npc interactions to have a natural conversational feel but still limit them to stay within the confines of the story/quest info.
Well, it is a tough problem indeed. I've just open-sourced my pipeline for training quadruped agents to fight in an arena, and it's very difficult to make them attack without going berserk (falling along with the enemy agent). Making them more aggressive causes them to go berserk, while making them more defensive turns them into cowards. I'm still trying to balance their behavior through the reward function!
@@sanchitagrawal1214 I can not even mention the video of my agents fighting, the comment gets deleted. I do not know, what triggers this insane automatic premoderation, I did not even post the link itself.
Didn't we see a paper, say 4-5 months ago (I think from Ubisoft or NVidia) that had solved that animation problem supposedly?? And it was also chaining them as well?
I think it'd be interesting for a single 2MP video on both Titans and Transformer-Squared; I'd love to hear Dr. Karoly's thoughts on the results of combining them.
Text to physics based animation seems absolutely insane because LLMs take text input and provide a text response... we're actually on the cusp of being able to run scenarios entirely generated by AI instead of being strictly stuck to text.
They should train an ai model to look at video and learn from it. For example you show the robot/agent a video of how to do something, and it will be able to do it. Just have a tv showing stuff inside millions of simulations with tasks that become easier when learning from the tv.
video has no fundamentals, you can mimic to make fake videos like veo 2 and sora but its not useful on a robot, you need real physics and data like we are doing with the new simulation models
I am a solo noob developer creating a virtual table-top RPG in Unreal Engine I am excited for this, as it would allow me to setup standard sets of animations for various skeletons, and then hopefully create new animations for given commands. For example: Player wants there character to Dash forward towards an enemy and attack with weapon while leaping into the air. The character can select their character and choose options, select target, choose options. When the round is started an everyone has entered their commands (told the DM what they want to do), the DM can then hit play and everyone can watch everything play out in real time. So, if that character strated dashing towards the enemy, but the enemy suddenly changed directions, the attacker animation would change direction to follow, and then leap in that direction and animations would all be adjusted accordingly. Or say, animate a four legged winged dragon in a realistic manner with only a small subset of given animations (like, walk, run, fly etc.)
Good luck, for what you described, animation is probly the easier part, coding gona be the hardest part, and since that code barrely exist even in private entity, AI will not help you, and blueprint for such complex intneraction will make your game slow, which probly result in you understanding C++, and know how to optimise such a cluster of conflicting animation/cancel animation ( the terms are Animation priority or animation Blenndinng )
@@alixamdali1782 Um, what? I don't understand why everyone disses blueprints. And, the code already exists, I am taking elements from the Animation Sample Project to accomplish this...I have no clue where you are coming from with this lol
We have seen these videos for 10 years now. Innovations, improvements, mind boggling stuff. But yet none of these animation innovations ever made its way into games.
we did? almost all of them became reality and are in the games we play, you just got used to it so you think its normal but look at games 10 years ago and now, its in the details
Yeah but honestly the title and thumbnail don't really give you much info. Too many videos on this channel are named "Nvidia's new ai" which just doesn't seem too interesting
i would use this for a game in which you "control" a group of fighters, maybe an army, and you play vs other players, and you can upgrade your individual fighters, some are mages, some have weapons, each has individual skill trees you can choose upgrades from, and you can tell them how to win for example "arrow guy and mage guy try to use your most damage abilities when the enemy is stunned by warrior guy" or something like this, and have your own little minion army fighting other people and hehhehehehehe this is my necromancer fantasy since i was a child :D maybe if i turn 60 in 35 years this will be a reality
success rate of getting up from the couch has got to be my favorite metric in any paper so far hahah
amazing work!
It's at least 50% more successful at getting off the couch than I am!
This is so impressive, actually. It might even inspire me to start making some papers of my own!
Thats so nice! Actually it is VERY SIMPLE to train these Agents. On my channel, I teach how to get started easily with Unity :)
Robot : "I know Kung-Fu"
Robot(0.0): "I invented Kung Fu"
Show Me
Other robot "whoa"
Aye nice matrix reference
Instead of a controller, I want a game where I can scream at my chacrater how to fight.
To catch them is my real test, to train them is my cause.
Suddenly memories of Binary Dominion, shouting "BO!"
A pokemon trainer game where you're telling your pokemon where to move to in the battlefield and which moves to use. "Pikachu, dodge right by using quick attack!" just like in the cartoons.
🤣
Gang leader
This suggests a future where a game is powered by an LLM in the background, guiding NPCs and generating complex worlds. I can envision a scenario where an audience gathers in a theater, shouting commands to the characters, and the story unfolds in real time based on their input.
yes very likely to have 19th century cabaret shows with AI HAHHAHAHAHA
What if the play time for GTA VII grinds into the infinite because of all the lore, world building, side quests that are generated by talking to a random NPC.😅
Entirely AI generated, from environment, movement, voices, characters, and stories with dynamic changes. It's going to be amazing. Sci-fi, historical, fantasy, training, simulator, real-world, you name it.
Congratulations you described a reverse twitch plays pokemon scenario.
That's just the CharacterAI app.
This seems relevant to much more than just games, I imagine this might be adaptable to real world robotics with enough work.
i'm kinda worried on the negative side. it will 100% be used in military
They're already 2 steps ahead, they don't need this.
@@hao21291 Military robotics don't tend to rely on complex armatures that would benefit from the kind of context aware kinematics you're picturing; It doesn't need to look like a person to mount a weapon on it, or to defuse/detonate an explosive, and there's not really advantage in doing so.
@@ezgarrth4555 I'm suspicious of how easily you seemed to be able to dismantle their argument 🤔 what you hiding?
It is called inverse Kinematics for robotics!
Dude I've been ranting about why we don't have this tech after seeing a tiny 3 seconds sound bite of something similar.
I didn't have a name for it. Now I do! This is awesome.
This is the future of video games. Imagine having a sports game where tackling is realistic. Or having the scale and weight of playing a super human like an Astartes would feel real.
I called it Ragpuppet instead of Ragdoll, lol.
This is what the actual next leap in gaming is. What consoles were made for. Having a dedicated hardware to do this like an NPU or physics card sounds like a no brainer for Microsoft and Sony, along with "AI".
Actually, the technique on the punch is not bad - straight extension from the shoulder, with a little hip twist without dropping guard.
Thanks for sharing! Can't wait to explore these, especially the one on simulations that look almost real. Keep up the great work!
I can't wait to have games where the player and the NPCs are smoothly animated based on the context they're in. Games where the NPCs have a personality but they can basically do and say anything. A game that uses ray tracing to have realistic light and AI text generation for the dialogs, AI text to speech for the NPC voices, AI decision making for the NPCs, AI animated movements, ...The possibilities are nearly endless.
That game is called life.
@@CSPlayerDamon unfortunately life can do this thing where it eats you alive when you try to go adventuring
@@Liam-ke2hv Indeed. I was simply briefly stating that overuse of AI in a game might be a bad design.
Like that video game in the movie "Her"
@@CSPlayerDamon I mean if you'd like to refer to other people as NPCs... should prolly stick to games
Can’t wait for this tech three or four papers down the line! This would be awesome for making cutscenes in a videogame instead of doing expensive motion capture.
Thank you for showing us the features, would've appreciated a deeper dive into the technical aspects of the paper
Martial artist here. Most martial arts put heavy emphasis on footwork. Often you will have new people only practice footwork, because little else matters until the footwork is first figured out.
If you think about it, our feet and legs are our primary ways of interacting with gravity and gravity is omnipresent and eternal. So nearly every action you do actually starts in you feet.
So maybe a future solution for an animation ai system would be to break the training down into chunks. First do feet and legs only, then add a torso and head, then finally add arms. Just a thought.
You cannot train the feet on their own and expect to perform just as well if you add a torso or whole body to it.
The feets are the primary interface with the ground, therefore the most important element in our interaction with gravity, but our body has its own weight, inertia, force, etc... and you need to have that there while training the feet so they can take it into account.
I feel like stability on one's feet should be fairly simple for AI, you can see Boston Dynamics robots holding themselves upright reliably because falling is a pretty obvious problem to detect and then solve for avoiding
I think that has a lot to do with the way humans learn and our attention span and focus. I would imagine AI to learn better with the whole system and more iterations.
@ Biped
Incorrect.
A lot of martial techniques can just be done with the feet.
The issue is just that gravity is the main factor in a fight and in life. For the most part we use our legs and feet to resist gravity and to reposition.
If you want an example a good one would be a bullfighter. Their main technique is to sidestep their opponent. Emphasis on step.
Footwork simply matters more, that's why you practice it the most.
@ I'm not talking about stability, I'm taking about strategic positioning and footwork. Sure, you need balance but what's more important is simply where you place your feet in anticipation of an attack and to set yourself up to launch a counter attack.
As a video game animator i am so excited about this.
Do you plan to enjoy unemployment?
@@eleklink8406 we will all enjoy it when all the jobs are gone, we will be free to use our time the way we want
@@eleklink8406 My thoughts exactly. I cant do good movement animations much less for something oganic but with this I don't need to
@@eleklink8406he can be even better now. Experts in any field only become more powerful with ai.
as a video game animator, you must not be very good at it if you are excited for this.
Imagine seeing npcs just walking around and doing things like real people? Not awkwardly transitioning from animations, but making up their own decisions on the fly. Especially if the characters can be given tasks, i imagine something like chatgpt could give it a full day's schedule of tasks that update as it meets the objectives, adding infinite variation on what they are doing and how they do it. Plus, if npc dialogue can begin using realistic voices and responses that stay in character and feel natural, we are really getting close to something cool and scary
Cool and scary is correct.
i doubt that, this model needs 4GB of vram (for 1 npc) and still doesn't look natural
@@opitopit2105 considering that most AI technology wasn't possible in 2018, I'd say things are moving pretty fast. I'm not sure where we're headed, but it will likely happen quickly. Even AI videos are brand new. Two years ago it was almost impossible to get something that was kinda real looking.
This could probably even be used to show a model how you can move AND probably by filming you giving you physio training to expand your movement range, etc.
With an EEG cap or some other method of reading brain activity, you could combine that with this to pretty much allow players to guide their character's internal thoughts and everything else is generated. The player wants to walk through a doorway, and their wanted action becomes their AI-counterparts thoughts, and eventual actions. Not much button pressing and would allow for quicker interfacing and control over a character where new animations could come up for any situation.
Having this and other Ai functions could be so amazing in games especially mmo’s. You could probably even retrofit old mmos with an interactive ai model, allowing npc interactions to have a natural conversational feel but still limit them to stay within the confines of the story/quest info.
never thought I'd prefer hearing AI's voice narrating the video...
Well, it is a tough problem indeed. I've just open-sourced my pipeline for training quadruped agents to fight in an arena, and it's very difficult to make them attack without going berserk (falling along with the enemy agent). Making them more aggressive causes them to go berserk, while making them more defensive turns them into cowards. I'm still trying to balance their behavior through the reward function!
Open source link..??.. I would love to contribute
@@sanchitagrawal1214 I can not even mention the video of my agents fighting, the comment gets deleted. I do not know, what triggers this insane automatic premoderation, I did not even post the link itself.
@@debuggers_processTH-cam is on a comment banning rage these days.
Been watching you from WELL before chatgpt came out and people started paying attention. Still my favorite A.I. channel!
I remember how real-time physic was introduced to games in Half-Life 2. Introducing AI will be such life changing moment.
Didn't we see a paper, say 4-5 months ago (I think from Ubisoft or NVidia) that had solved that animation problem supposedly?? And it was also chaining them as well?
1:45
Such a funny name for a paper
"Shi et al 2024"
Thats not the name of the paper
...for a paper, ehm yes, that's exactly it!!
In the next paper, the AI will be a 3rd degree black belt in Brazilian Jujitsu.
seems similar to the Multi-AMP technique, you should review that paper too!!!
Next can you tell us about the new titans architecture?
I think it'd be interesting for a single 2MP video on both Titans and Transformer-Squared; I'd love to hear Dr. Karoly's thoughts on the results of combining them.
Text to physics based animation seems absolutely insane because LLMs take text input and provide a text response... we're actually on the cusp of being able to run scenarios entirely generated by AI instead of being strictly stuck to text.
Cooking AI in every direction. Until there’s no directions left.
Then they cook us
True!! What a time to be alive😍
imagine NPCs in RPG, reacting dynamically like this
imagine horror games
This is one small couch for roboto, one giant HAIYA for robot-kind.
Yeah, I saw this a while back ago. There was this one other place that was training AI to do NPC walking and combat movements
It looks like it just moved the pop. It's still there at 2:30
wake up babe- a new Six Minute Nvidia Advertisements video just dropped
It’s not an advertisement thought 😭
show me on the doll where Nvidia touched you
Im gonna teach my teacher to watch my cat
Amazing! Love your videos. :D
this is really a nice addition
1:39
This looks so much like a TH-cam Poop
a sims-like game with this tech would be amazing
at this point every ai model released is a game changing model
this is the future of animation, no doubt about this. and probably not even that far away.
Lol, the old AI was drunk!!
Nice. I just hope this tech could be utilized locally. Having to reach out to OpenAI’s servers to do it is not interesting IMO.
This is an NVIDIA model; it has nothing to do with OpenAI. It runs locally on 4GB of VRAM (though you'll need about 50GB of VRAM to train it).
@ Thanks! Even training should be doable locally then with the new generation of GPUs and AI computers like Nvidia’s DIGITS. Fun times ahead!
They should train an ai model to look at video and learn from it. For example you show the robot/agent a video of how to do something, and it will be able to do it. Just have a tv showing stuff inside millions of simulations with tasks that become easier when learning from the tv.
That's a thing. It's just not as effective as other techniques.
video has no fundamentals, you can mimic to make fake videos like veo 2 and sora but its not useful on a robot, you need real physics and data like we are doing with the new simulation models
I am a solo noob developer creating a virtual table-top RPG in Unreal Engine
I am excited for this, as it would allow me to setup standard sets of animations for various skeletons, and then hopefully create new animations for given commands. For example: Player wants there character to Dash forward towards an enemy and attack with weapon while leaping into the air. The character can select their character and choose options, select target, choose options. When the round is started an everyone has entered their commands (told the DM what they want to do), the DM can then hit play and everyone can watch everything play out in real time. So, if that character strated dashing towards the enemy, but the enemy suddenly changed directions, the attacker animation would change direction to follow, and then leap in that direction and animations would all be adjusted accordingly. Or say, animate a four legged winged dragon in a realistic manner with only a small subset of given animations (like, walk, run, fly etc.)
Good luck, for what you described, animation is probly the easier part, coding gona be the hardest part, and since that code barrely exist even in private entity, AI will not help you, and blueprint for such complex intneraction will make your game slow, which probly result in you understanding C++, and know how to optimise such a cluster of conflicting animation/cancel animation ( the terms are Animation priority or animation Blenndinng )
@@alixamdali1782 Um, what? I don't understand why everyone disses blueprints. And, the code already exists, I am taking elements from the Animation Sample Project to accomplish this...I have no clue where you are coming from with this lol
It's paper time!!!
Remembers me from that one game called ToriBash, but now instead of players, prompts control the model.
1:36 why do I laugh so hard at this.🤣
If you want a picture of the future, imagine a boot stamping on a human face-for ever.
Man I can see this in gaming in a few years
Doesnt the new architecture look a lot like predictive coding thing in own minds? I got a bit excited when I saw the diagram
Oh cool, finally advancements in AI that aren't trying to steal *my* job, but someone else's.
i wish we had more papers on simulating stuff like bread tearing instead of just ai every day.
This would be a game changer for making emotes in games
This is the gap in AI we need to fully solve to get personal robots like in the movies.
Soon as this is perfected, its star wars bro.
When is someone going to put all of these tools into a package that we can use stand-alone without a subscription. lol
Sarah Conor ?
This is great, but canthis be done in real time to basically make appropriate animations in real time in game?
Yep. If I'm reading the paper right (though I skimmed it, tbh), inference on an RTX 3090 can output up to 3500 fps of animation data.
im surprised they are using a transformer model instead of a diffusion model since its a sequence of poses
It's a diffusion transformer. The best of both worlds.
Looks like like a drunk person's movements.
we should make reforestation robots with that. plant 10 amazon forests
I tried to prompt myself with "Hold onto the [papers]", but it's a failure case of my algorithm.
this isn't 2 minutes! it's 6 minutes!
I guess we’re in the Interloper timeline, just a little bit different
We have seen these videos for 10 years now. Innovations, improvements, mind boggling stuff.
But yet none of these animation innovations ever made its way into games.
we did? almost all of them became reality and are in the games we play, you just got used to it so you think its normal but look at games 10 years ago and now, its in the details
@shirowolff9147 Now I'm curious which games you speak of.
Agent smith in the making!
let's hope this isn't used in the real world to create war bots
WHAT A TIME TO BE ALIVE!
Is it balancing or just trying to look plausible for animation purposes? I take it it's the latter
I know -Kung fu- how to get off the couch
👁️👄👁️
When we have fully controllable slaves in our phone, I can't imagine which industry will be the one to utilise them
youre already a slave to your phone obviously, its you whos being utilized fool.
Thats so nice! Actually it is VERY SIMPLE to train these Agents. On my channel, I teach how to get started easily with Unity :)
GTA 7 in 3020 will be wild.
what could POSSIBLY go wrong
The ai would be good only to be used as some sandbox thing, replacing game developers would break
my heart.
ok we're done. Kidding but...who knows. Next, AI use weapon (eazy) lulz... thx 4 the news Karoly 🙂
Now we just need to use nvidia realty physics clone to make robots
Anyone else thinking of Ren & Stimpy
This is massive for the "Livestreams of copyrighted characters reading AI-generated scripts with AI-generated voices" fandom
Man I wish I was smart like this.
Better than a killbot... an assaultbot.
In now AI can Animation look like totally accurate battle simulator in units NPC
We are coocked when chat gpt release that he can use this on the killing robots.
It doesn't need to be perfect but acceptable
Uncharted 5 is gonna blow my mind
My chance getting up from sofa is 0%
5:02 I laughed out loud
the AI voice needs a lot of work still
I am about to lose my dream job
Can we tell it to kick flip onto the sofa and the front flip to get up
Let's just hope Millennial devs don't get their hands on this - all NPC's will be jumping everywhere. (WoW)
把它用作现实中的机器人上面，哇！
not sure if it's a great news... was wondering what it could looks in a real humanoid robot.
"That's better than what most humans can do" Seriously? Most humans can not get out of the couch for 2 of 100 tries? :D (@5:01)
NPCs are going t be soo cool in a few years when the software catches up
What the heck? Two minute papers fell of this hard? 3.3K views in an hour? Wasn't this better?
Yeah but honestly the title and thumbnail don't really give you much info. Too many videos on this channel are named "Nvidia's new ai" which just doesn't seem too interesting
14k in 5h
get a job and stop caring about other peoples views
@ Right on it!
Can it be used to animate "adult" activities? Asking for a friend 😏
teaching ai to fight for ai video generation would be great, because they suck at fighting atm.
i would use this for a game in which you "control" a group of fighters, maybe an army, and you play vs other players, and you can upgrade your individual fighters, some are mages, some have weapons, each has individual skill trees you can choose upgrades from, and you can tell them how to win for example "arrow guy and mage guy try to use your most damage abilities when the enemy is stunned by warrior guy" or something like this, and have your own little minion army fighting other people and hehhehehehehe this is my necromancer fantasy since i was a child :D maybe if i turn 60 in 35 years this will be a reality
Holodeck VR is one step closer. Fire up your Picard Private Investigator mysteries brothers and sisters, it's not much longer now.
We are all about to be homeless. And we are training these AIs very hard to achieve this.