(Super) Quick instructions for Puppeteer setup
ฝัง
- เผยแพร่เมื่อ 28 พ.ย. 2021
- I made a tool to try and improve the performance I could get out of UE5 Live Link Face with Metahuman and here's the instructions on how to get it set up in your own project.
Here's the video demoing it as a comparison with the default setup, with terrible audio: • A tool for improving L...
And a video explaining how the blueprints work: • How does the Puppeteer...
And this is my video explaining it, way too fast. But at least the video isn't 30 minutes long right?
Here's the GitHub link: github.com/ViktroNZ/Puppeteer
#UE5 #Metahuman #LiveLinkFace #UnrealEngine - ภาพยนตร์และแอนิเมชัน
![[TH] 2024 PMWC x EWC Main Tournament Day 3 | PUBG MOBILE WORLD CUP x ESPORTS WORLD CUP](http://i.ytimg.com/vi/3EM6gpHgLk4/mqdefault.jpg)
![มองหน้ากันไม่ติด (Awkward) - โอ๊ต ปราโมทย์ FEAT. MAIYARAP [Official MV]](http://i.ytimg.com/vi/4b02SY_r6Gw/mqdefault.jpg)
Hello, for those who don't see the puppeteer content, check "show the plugin content" in the content browser settings , hope it helps !
Hah really - is it that simple! That's awesome, thanks heaps Pasquale. That would explain why it works for some and not others
You're a life saver
Just discovered it. Thank you so much for all your hard work and for sharing it with us. Super excited to try it out tomorrow. (:
This is a seriously great plugin, it really adds some depth to animating metahumans. I'm very new to Unreal, this is really simple and easy to understand. great job.
Thank you so much, that's really incredible, hope to test that soon :)
You are brilliant! Your channel name is very apt, thank you for this! :)
how does it feel to know that you changed the life of an Unreal engine newbie? Great work. Couldn't be more grateful
This is such a lifesaver. Setting it up today
Glad to hear it! I should've been asking this earlier but what sort of project are you using it with?
I've just put a video together explaining the blueprints: th-cam.com/video/fC3UHKr0Wpg/w-d-xo.html
dude thanks so much, your tool helps a ton! :)
Great to hear it! Looking forward to seeing what you do with it
What a legend, this is awesome!
Thank you so much for sharing this! :)
this is awesom going to try hopefully it will work with the lastest metahuman
I haven't run it yet, but thank you already!
Awesome!!!!! You're genius!!!!
your a legend brother
Nice, I just got my iphone X yesterday to play with Live Link Face and noticed the jitter was a bit much. I was going to write my own version of this and then decided to look for other people doing the same thing :) thanks! saved me a bit of work.
Did you notice your phone overheating by the way? I got a used iphone X for this and if I have the camera on it gets a bit toasty and the framerate drops. I have heard that things improve if you don't use wireless too. Anyway, thanks again!
Glad I could help! Yep, I found my phone overheated as well. I wonder if attaching a flat plate of metal to the back of the phone while it's in a stand would help dissipate the heat. Or just use an external fan :)
@@_theartofvictory Yeah, I did consider that. I saw another YT where they suggested using an ipad because of its better thermals. That wouldn't really fly though if you want to mount it to your head :)
This works amazingly well! Thank you so much for creating this tool.
I've run into a problem trying to bake the animation sequence to the face skeleton as it appears to reference the control rig, and that doesn't seem able to 'see' the recording. Any thoughts on how to fix this?
The Puppeteer_Anim_BP never asked me about the skeleton, so maybe that has something to do with it. How can I manually input the Facial_Archetype into the blueprint?
Ah this is something that I haven't tried to do so I don't have a good answer and I'm also not super experienced with Sequencer.
But one thing to consider is that the way the rig is being driven, is that Puppeteer modifies the curve values on the skeletal mesh, not the Control Rig. Therefore the Control Rig isn't actually being driven by LiveLink in this instance.
In Sequencer, is there a way to add the animated curves as tracks so that they're recorded, instead of the Control Rig?
Another thing to look at is whether the AnimBP could be modified to drive the Control Rig?
Great plug-in. Can you add a tutorial how to record the animations with sequencer? This would be very nice :-)
Thanks Andreas! I’m sorry I’m not planning to do anymore videos for Puppeteer at this stage.
I believe the generic tutorials for take recorder should give you what you need. You would need to record the curves of the head skeletal mesh
Bit late but if you or anyone else is still wondering: Have Take Recorder open in the MainLevel, press play and then drag the BP_(your metahuman) into Take Recorder (you have to do this during Play as the Metahuman doesn't exist in the level otherwise). While you're doing the live link stuff press the Record button and then when you're finished hit Stop and the animation will be saved in Cinematics>Takes>[Date]>[Subscenes]>Animation
@@Monoville This reply saved this plugin. Should be up to top!🤟
crazy!
Hi Victor, thanks for your work and great contribution. One question: I was able to open your Main level an set my custom MH and it worked great, just that pressing 1 did not open your UI and second, when i returned to my own level (even i saw your Puppeteer BP instead of the Face_Anim BP) it did not work? Any ideas? Thx, Mark PS: I am on UE 5.0.3
Hey Mark, it seems like a common issue that 1 doesn't open the UI now and I'm not sure why that is. If you sort the comments in here by Newest First, you'll see a few posts down Pixel Bokeh Productions asks the same question. Some of my suggestions there might help.
With your second question, I'm not sure what you mean by "it did not work". Do you mean the UI wasn't showing up or the calibration wasn't working?
If I recall correctly, the way I've set it up, you need to be using the Puppeteer_Actor_BP in the level and make sure it's set to the correct Metahuman BP inside the details panel. That'll spawn the correct Metahuman and then you can bring up the UI using 1 or whatever alternative option you might come up with.
Capturing in Sequencer is a little trickier because of the way I set it up. It's not ideal, but you can begin play on the level and then add the Metahuman BP into Sequencer after you're playing already. This way you can reference the Control Rig in Sequencer. I'm not actually sure if that works - but I did see the Control Rig when I tried it quickly.
Hopefully that gives you some ideas to start with!
you should get a epic grant for this :D
its hidden gem, thank you so much for sharing , but can i change the value manually? or it just can edit to minim or max value?
Thanks :). The value of each control is driven by FaceAR via LiveLink so it's not setup to take in a manual value say for debugging. There's nothing stopping you from adding that functionality though. You could setup a debugging mode where you pause live link and then drag sliders to create a pose. It's designed for calibration, so it's only really designed for setting min/max currently.
Great work ! Did you manage to improve speech and lips animation ?
Thanks Vincent, no I didn't. It does a very poor job of capturing speech. The ARKit tech isn't really designed for photo realistic movements yet - more for their memojis that have very simple geometry.
If I were to try and tackle it, the first things I'd look at are making sure I've got a good frame rate, perhaps minimising some of the curves so that it isolates some of the primary shapes rather than doing a whole bunch at the same time. Perhaps inferring intermediate controls by taking two ARKit blendshape outputs controls and doing some on the fly calculations.
Other issues I've found are around jaw movements driving aspects of the upper face undesirably. On the iPhone, you can see how moving the jaw pulls at the whole tracked mesh
@@_theartofvictory did you xanadu youtube channel ? Speech and lips are pretty impressive. I wonder how he did it.
@@vincenttoujas yeah indeed! My guess would be that there's a manual cleanup process involved? His work is impressive
Hey man, amazing stuff that you're doing!!! I am actually struggling at the step where you have to select the sceleton. Where exactly in the puppeter anim bp do I pick the face archetype?
When I open up the bp, hovering on the skeleton Icon shows me face_archetype_skeleton already. But later when I go into the metahuman bp, I cant find the puppeteeranim_bp_c, only the one without the c
@@habibmensch7368 have you tried the one without the _c?
You shouldn’t need to go into the AnimBP and select the skeleton - this happens when the AnimBP first gets created.
@@_theartofvictory I tried it with the one without the c and live link wasnt picking up my facial motion :/
But how can we adjust it to achieve the best effect? Are there any related tutorials? We don't have any experience in facial expression debugging
Hey sorry I don't know of any related tutorials for this. As a starting point you want to get it to match your own facial expression as best as possible. Reference is key
Victor, hello. one question, can this be used in Unreal 4? how would be the process? I'm about to use your tool in Unreal 5, no problem, I do have a problem with the metahuman, in which the binding assets are not asigned and when I try, unreal is crashing. but I don't think your tool has anything to do with that. some unreal Bug. so I would like to try it in Unreal 4
Hey CJ, it’s possible to set up in Unreal Engine 4 as well but I think it would be a manual process of copying the setup in. The assets are versioned for UE5 so can’t be used in UE4.
You could try migrate the assets from a UE5 project into the UE4 project using the built in migration tool in the editor but I’m not sure if the assets can be backwards converted like that. Simply copying the assets in Windows Explorer doesn’t work.
It could be automatic you could run the face recognition simultaneously on the metahuman, compare both recognitions and auto adjust the valúes
Thanks! Does it support the final version of UE5?
Hey I haven’t tested it but I expect it will do
Questions.
1.- How to reach you? Email, discord ? anything?
2.- It seems to me that the version in github is updated compared to this video. Im trying to use your AnimBp but doesnt really work. Im trying to figure out as I write this message.
3.- The hotkey 1 to bring the UI doesnt work for me. I am tho not working in Play mode but Live instead.
Overall, Im looking more into the plugin, I like what you are doing! Best
After confirming Puppeteer plugin installed, I do not see "Puppeteer Content" under main content so I do not have an option to load main level.
Hmmm, can you check the following:
1. The path of the plugin is //Plugins/Puppeteer/Puppeteer.uplugin (presumably it is because you said it's installed)
2. Are you running Unreal Engine 5?
@@_theartofvictory im in this exact position myself, correct folder and plug in installed but no map :S
Hey it looks like the issue might be that the "show plugin content" checkbox hasn't been checked in the Content browser. Give that a go and see if that fixes it
same problem as above. Impossible to see Puppeteer Content even with enabled plugin checked and restart ... any clue ?
Hey Vincent, I'm not sure why that issue is coming up. I'm also not sure if it's affecting everyone but it looks like a reasonable workaround is to import the "Content" folder within the Puppeter plugin straight into your own Content folder for your project.
To do that, download and export the Puppeteer plugin from GitHub to any location on your computer.
Create a sub folder in the Content folder of your own project called "Puppeteer".
Copy the contents of the Puppeteer plugin's Content folder into the folder you created within your project. You should now be able to access the Puppeteer assets within Unreal Editor.
Give that a go and see if that works :)
@@_theartofvictory thanks a lot for your help !!!
will this also work on UE 4.27.2?
Hey I'm pretty sure it won't. Any time I've tried to use assets created in a newer version in an older version it doesn't work.
You could refer to my other video that explains how it works to try and recreate it?
Oh another you thing you could try is get the plugin into the UE5 project and then use the "migrate assets" function to try get it into your 4.27 project
How can I export the facial animation key ??
I believe what you're looking for is the take recorder. When you're in Play mode, you should be able to use the take recorder to capture the facial animation to sequencer.
Have a search on TH-cam for take recorder and Metahuman
Will this work with a non-metahuman rig/character?
It won't out of the box but it is possible to modify it to work with other rigs.
The main change would be how the apple blendshape values are converted to rig blendshape or joint transformation values
thanks for sharing , but is it works in UE5? I can't see the folder after install, thanks
Hey Yu Joe, yes it's built for UE5. You might need to enable "Show Plugin Content" in Show View Options in the Content Browser
@@_theartofvictory Thanks you so much for reply~~~ it works~~~
After numerous attempts I’m not able to spawn metahuman with BP In my custom map, So I figured I’d just record it in the example map, but there I just can’t record the animation in sequencer as It has no inputs whatsoever like “control rig” or anything :/ I’m rly loving the plugin it’s super useful… if u could tell us how to fix these issues
Thanks❤
Thanks and I'm afraid I didn't get as far as to make it user friendly to work with sequencer. I just had a quick look at my blueprints and I can see that you can't access the Control Rig in sequencer because the Actor for the Metahuman isn't spawned until it starts playing.
One thing you could try is begin play in editor, then open up your Level Sequence and add in the BP Actor of your Metahuman that gets spawned into the level. From there you'll see you can now access the Control Rig in the Level Sequence. I tried to capture on mine after doing this but it crashed - but I didn't have Live Link hooked up or any animation happening and I think this might have caused an issue.
Give that a try and see if that works
@@_theartofvictory Thanks for the reply
@@lukagogoshvili6787 A simple way to address this might be to duplicate my level and then copy your assets into it. Otherwise you could copy the assets from the MainLevel into your level - mainly the Puppeteer_Actor if I recall correctly.
With the Puppeteer_Actor, in the details, make sure you set the Metahuman Class to spawn as well - this should be the BP_ from the Metahuman folder that gets created when you import the Metahuman.
For anybody else trying to use Puppeteer with sequencer, I just figured out how to make it work.
Record a Live Link take using the take recorder like normal, and instead of dragging the spawned metahuman into the sequence you need to open that recorded sequence in the take recorder.
Then you disable recording the Live Link source because you already have that saved, and while the editor is still playing you add the spawned metahuman as a source to the take recorder. Then you just hit record and finish that take. Afterwards, you can just go to the metahuman's face track in that recorded sequence, right click it, and then bake it to the control rig.
The editor crashed for me the first time I tried it, but the 2nd time it worked perfectly.
How we can record the animation?
You should be able to use take recorder to capture the animation. I haven't used it much myself so I can't explain how but I would suggest looking for tutorials on that first
hi, when i use the puppeteer and input the mocap body make head not follow body mocap, how to fix it?
Hey have a look in the animation blueprint. There'll be something in there that's taking the head motion input from LiveLink and applying it. Disconnect this and it should stop LiveLink from fighting your body mocap setup.
so i must apply the head motion or disconnecting the head motion? it just for head rotation
@@zdhy261430 You'll need to disconnect the head motion that's coming from the LiveLink node in the Animation Blueprint for the Metahuman. I can't tell for sure - but I expect that's what's happening
Great tutorial and overview of the plug-in. Not able to get the to get the interface screen to come up in UE 5.0.2 by pressing 1.Viewing with Default ViewPort. Have everything else working and set up. I have tried remapping the input key from the BP but still no luck. The only difference is I'm streaming data in via Rokoko LiveLink and I can see it is coming in on the Metahuman when in Play. Anything I can try or review to see if I can get this to work? Thanks!
Hmmm a couple of other people have had issues as well and I've been trying to reproduce locally but haven't been able to. I'm on UE5.0.2 as well.
I'd be keen to eliminate key events as an issue first. In the blueprint where the key event is triggered, are you able to put in a Print String node and print some message to screen? That way when you press the 1 key, you'll know whether it's a key event or the UI is not actually being created.
Make sure you've clicked in the viewport before press 1 as well.
Another thought might be that the player controller is not 0 in your setup. Perhaps using Rokoko LiveLink does something with the Player Controller IDs? If the Key Event is working, you could get it to print out the result of "Get Num Player Controllers" function. That should be 1. If it's more than 1, then you could change the function "Get Player Controller" to use the Player Index of 1 instead of 0.
Another workaround would be to hook up Event Begin Play and replace the Key Event. It'll load up the calibration on Begin Play, but that will tell you whether the UI is working or not. If that was the only way to go forward, you could add a button to the UI that closes the widget after you're done with it. A bit hacky, but could get you by in a pinch.
Have a try of those things and report back :)
@@_theartofvictory Thanks for the great suggestions. I will give them a try and see what happens. If successful, I will update.
None of the solutions above solved this problem for me , I’ve even tried to change the key but absolutely nothing happens… anyone was able to solve it ?
I see everything working fine but the UI doesn’t come up and I can’t calibrate it.
@@rastamami I'm not exactly sure why there's this issue of using the 1 key event. Perhaps something has happened in the latest UE5 update that breaks my implementation of the key event.
I haven't had time to investigate further (and I'm not doing any more development on this at this stage, though I feel like I might need to with the number of people using it!).
I suggest using you connect Event Begin Play to create the UI and replace the Key Event 1. Someone else has tried this and it worked for them - but then you need to close the UI, which you might just need to create a button in the UI to close it. Sorry I know this isn't the best solution, but without further investigation I don't have any better ideas at this stage
@@_theartofvictory here too, the same problem, just updated to 5.0.3 and still no success... tried to change the keys too, I was thinking it could be a problem of azerty / qwerty keyboards...
hi , is this working with 5.1? cause for me I cant see the content
In the content browser, have you checked "Show plugin content" in the view options?
@@_theartofvictory yes, and not working , I nee to try again, in fact my live link is meta human face is vibrating , not usable at all , I wish I could show you
Hi! Any idea why i cannot see the interface when i press ”1”?
Hey are you running the level that came with the plugin? In the level blueprint you'll see the logic that creates the interface
@@_theartofvictory I have the same issue. the "1" seems to be mapped to another function by default.
@@_theartofvictory same issure,, I have changed key to Ctrl+1, still not working, need help , thanks master
@@junzheshen3343 Hey sorry for the late reply. If you open User_BP and have a look in the main event graph, you'll see a key event for "1". Change that key event to something else that's not used.
@@yujoe5338 Hey Yu Joe, I just replied above as well, but have a look at User_BP and change the key event binding to something else. You mentioned you changed to Ctrl+1 and it's still not working. It's worth trying another key event that's not using 1. 1 is still working for me, but I haven't changed any of the default UE5 settings so that might be why?
great tutorial! does puppeteer work on 4.26 please ?
It doesn't unfortunately, only UE5 on
@@_theartofvictory thanks! I have my link link face working on 4.26 but for some reason it isn't working on 5.0.2, do you think there's any difference in getting them to work on either version? cos I basically replicated the process i used in 4.26. I'd appreciate any help
never mind! got it to work thanks
I appear to have the same issue as The Cosmos above ^ Using UE5 and plugin is in the correct folder, I see the .uplugin where it should be. Have also toggled it on and off a few times in Edit --> Plugins menu hmm. Anything else I could try? Unreal 5.0.0-16682836
I was able to get it working by manually importing the Puppeteer Content into the Content Browser (it wasn't autopopulating, even on restart). After that, everything seems to be working well! I wonder though how I can use it in 1) My own Level or 2) My own sequence inside your level. If I add a camera track, and lock viewport to my new camera, the livelink motion is not hooked up.. If I *don't* lock viewport to my own CineCamera, and I just hit Play, your app loads, but I cant move the camera around etc.. (Also wondering if its possible to use in another level altogether after saving the calibration). Thank you for your great work with this Plugin! I'm happy to shoot over a tip etc if you have BuyMeACoffee setup etc
Oh strange! I've checked my version and I'm running the same as you: Version: 5.0.0-16682836+++UE5+Release-5.0-EarlyAccess
Here's a video of what happens on mine: th-cam.com/video/ymlzmCvS_wA/w-d-xo.html
But then it's also on my computer so there's no issues with versions etc? Ideally it would be all packaged nicely as a marketplace plugin but this is just a little side project so wasn't planning to go and do that.
It's the first time I've put a plugin together so no doubt I've done something wrong too :D
Great work and yeah that's a good workaround given all the content is just there to pull out and chuck in your project!
You'll be able to use it on your own level and with your own camera setup too.
The Puppeteer_Actor is the main thing you need to drop into your level and then configure the correct class settings and Live Link Subject in the details pane of the Puppeteer_Actor instance.
I haven't done much with Sequencer so I'm not exactly sure of the flow without looking into it.
In the way I've set it up, you've got to have the level playing to have the blueprints working and processing Live Link. If I recall correctly, there's a way to have Live Link processing as a preview thing (i.e. without Play in Editor) - but I'm not sure how to do that off the top of my head.
Perhaps you could create a CineCamera that you then reference in Sequencer. In the level blueprint, you set the CineCamera as the view target with "Set View Target" for Player Controller 0. That when when you Begin Play, you're looking from the CineCamera you are referencing in Sequencer.
With regards to the calibration, what's happening is that a save game is getting created with all the min and max values. You should be able to load the values in using Load Calibration regardless of the level that you're running the tool on.
Thanks for the offer of a tip, but I'll just get a kick out of someone being able to do something interesting with the plugin if it works :).
@@whiletrueify I just recorded a video explaining the blueprint. Hopefully that helps you pick it apart: th-cam.com/video/fC3UHKr0Wpg/w-d-xo.html