If you're curious about the license and terms of use, you can see it in detail here: github.com/ReForge-Mode/Radish_Tools_Phoneme_Extractor This is the longest video I've made so far. The process of research literally started three years ago. So if you want to support me on making these free useful tools, you can do it via Patreon and Ko-FI - www.patreon.com/ReForgeMode - ko-fi.com/reforgemode
Hey, recommend the best way to get vaginal and rectal blendshapes functioning using VROID models? Been working on a new electronics prototype and it's not as fun with only her mouth.
Yes you can! There are two ways to so it. You can either: - Copy paste the keyframes from one clip to the end of another clip. Not recommended because it could easily mess up the timing. - Use Timelines. Timeline is a feature in Unity that allows you to make cutscenes. It is a series of animation played one after another. Think of it like a video editor with multiple tracks for different objects. You would use Timeline for lipsync anyway in conversation, so might as well handle everything in timelines. I've included Timeline examples in the project files when you download this tool from my Ko-Fi page.
I see... This plugin has not been updated in a long time. I am considering to update it sometime in the future. I recommend checking uLipSync instead: th-cam.com/video/k5CtTsIKwE4/w-d-xo.html
Glad you like it! Maybe I'll do it somewhere in the future! I'm sure there are some methods I can use. It could certainly be more accurate than this method!
Hi, maybe I'm too much of a newbie to Unity, but I can't get it to work in another model after importing into the scene, could you help me with that please?
Sure! Today is the right time to start learning Unity! Let me know what is the problem in details. You can also send screenshots so I can help you better by joining our Discord server! Open up a forum there and we'll help you directly.
Thank you so much for this! I've been able to test your project, and I was wanting to import it into my existing project which is already using UniVRM 1. How hard would that be, or do I need to remove UniVRM and import your project as a package?
Glad I could help! You don't need to remove anything. Though, I would recommend doing the lipsync animation on a separate project and just take the animation files to your real project. Because when you try to build the game, the build will throw out errors. All codes I used in this project only runs in the Editor, not in the built version. You can run this in your project, but don't forget to delete my script or move the script to another folder before you build the exe file.
Yes, I'm pretty sure this is compatible with any platform, since the end result is just an animation file. The only caveat is that you should not include this tool in your game build. I will update it soon so it will automatically be discarded once the project is built 👍👍
Hmmm... No, I don't think there is. But the source code is available, so if you want, you can build the executable yourself. Or maybe you can use Virtual Machine that runs Windows to do it. The Phoneme Extractor is not a heavy program by any means. Of course, I have very little experience on doing these, so you might have to do some googling.
Thank you for this! I had a question regarding loading custom audio files - I am unable to create the animations for them as it throws an ArgumentOutOfRangeException.
@@ReForgeMode It is a red error warning. The audio file is 16-bit .wav file. The error comes up when I am loading my audio.wav folder to Generate the LipSync Animation. To be precise, the error thrown says ArgumentOutOfRangeException : Index should be non-negative and less than the size of the collection. Thank you for your help!!
It seems to be a list issue. Can you post the screenshot on my Discord server? (Link is in the description) If there is any major error, I will need to fix it. Thanks for reporting this!
Hi, I got an error at compilation: "Assets\Scripts\CreateAnimationCurse\CreateAnimationCurve.cs(6,26): error CS0234: The type or namespace name 'AnimationUtility' does not exist in the namespace 'UnityEditor' (are you missing an assembly reference?) " Did I miss something?
Huh, it looks like Unity messing up again. Did you open the project in different version of Unity than what I noted on the github page? That could be the problem.
Oh, seems that this project is not intended for build. I found an explanation related to this problem: "UnityEditor is a namespace specific for Unity Editor internal stuff and custom editor scripts ... it is not available in a build so you don't want to use anything from it in a MonoBehaviour "
Oh no. The lipsync tool is not meant to be in built. This is because the function that generate the animation file can only be run in Unity Editor. it can't be run in the finished game. You only want to use the finished lipsync animation file in your final game build. But if you want to include this in your build and not cause errors, you can add: #if UNITY_EDITOR #endif This will make it so that this script won't be added into your final build, so no errors there.
Heyy there !! I m a beginner to unity and i don't know anything about it can u guide me that how can i use this lipsync tool for converting tts tokens real time in a video call and integrating bert or gpt for understanding so that avatar replies accordingly and can engage in conversation
Hello! Your use case falls outside the scope of this video, and would require a dedicated project for it. If you'd like, you can commission me to work on your project. Feel free to chat on my Discord server, link in the description. I have experience in doing this for several clients.
@ReForgeMode okay...also i wasn't able to mention before but this tool works wonder and this project really increased my understanding but i was very curious to know about head moving..hair flowing animations of this avatar..needles to say these tools are life saver but i m still trying to learn to use them for my custom needs
@@AaronAsherRandall Yes, the example in the intro is recorded from the engine. You are free to try it yourself. It's free unless you choose to donate to me.
Very cool! Do let me know how it works and when the video is uploaded! Of course, this video is quite outdated. My latest lipsync video is using an open source uLipSync plugin that captures mic sound in real time. But it's not accurate as I would like it to be. I was wondering on how accurate your script will be, and if it allows for live mic lipsync. If you open-source it, I think it will help many people in their gamedev journey. That's always been my goal after all.
@@ReForgeMode Hey, its a rather simple solution I came up with for my own game. It has no "live mic" option, so I'm sorry if it sounded misleading. It works with text based conversations. (Which can be coupled with a voiceover of the same text). I simpley provide a phonetic string, and the script loops over it, and moves the A, I, O, E, U blendshapes. So if the voiceover is "Good Night", i provide a text "gud nait" and it will give me a good enough result, by moving the U, A, I blendshapes, during the voiceover duration. Obviously i can improve the percision of this script, but for now its good enough. Today or tomorrow I will add a Short video to my channel showing how it looks, for a tutorial it will probably take me some time though.
@@Mikes-Code I see! So you're handing the automatic phoneme translation to the user instead. Yes, that would work for most cases. I wonder if it works for different people with different accent... I guess you can just change the phoneme translation. Then again, you won't notice the viseme that closely either. I still want to redo this tutorial sometime in the future. But automating the phoneme translation process is such a pain, because I need to rewrite the whole Radish tools into Unity. That's gonna be a struggle and a half. Looking forward to your video though! Are you sharing the repo publicly?
Major Fail. I was hoping this system could be used on a custom made character, but your scripts are hardcoded looking for significant words. There is also no ability to link Blendshapes within your scripts
I will update this later. For now, you will need to modify the blendshape names in the script. This is no excuse for me not to make it easier to use. Thanks for notifying me. Could you tell me what you need so it could work in your character? Like the option to specify where the blendshape is located?
@@ReForgeMode Hey sorry for being an asshole. I tried editing the Clipcurve lines of your code but it wont link to the proper mesh -- not sure why, but its most likely a unity bug back from 2018. I bypassed this problem by manually copy pasting the actual keyframes into a new Clip that was created in the editor. Many thanks for the free tool.
It's pretty difficult if you never do any blendshapes modeling in Blender. I'm mostly a programmer, so I never even tried those. I suggest finding tools or plugins to make it.
Hello, this is great work. I appreciate the time you dedicate to it. I would really like you to help me switch to another model and use several at the same time in one scene. Please tell me how much you would charge me in dollars in a private message. Thank you very much.
Why yall acting like this looks good? Barely any of the phonemes is being reflected in the animation. If you're just post processing audio, you're better off faking it via audio amplitude and detecting specific phonemes like "O" and "U" than going this route.
If you're curious about the license and terms of use, you can see it in detail here:
github.com/ReForge-Mode/Radish_Tools_Phoneme_Extractor
This is the longest video I've made so far. The process of research literally started three years ago. So if you want to support me on making these free useful tools, you can do it via Patreon and Ko-FI
- www.patreon.com/ReForgeMode
- ko-fi.com/reforgemode
Hey, recommend the best way to get vaginal and rectal blendshapes functioning using VROID models? Been working on a new electronics prototype and it's not as fun with only her mouth.
holy shit you are literall one of the best people I've ever had the pleasure of interacting with on the internet; thanks!
Thank you! I'm just here to help everyone 😄😄
You never disappoint thanks for everything you've done for the community much love
Haha, thank you! I have an even bigger things in the horizon. Stay tuned! Our Unity VRoid game will be made easier!
Those results seem soo good, wow!
Thank you! Glad you like it!
Thank you. I was wondering whether I can combine those lip sync animation clips with other clips so the same characters plays them at the same time?
Yes you can! There are two ways to so it. You can either:
- Copy paste the keyframes from one clip to the end of another clip. Not recommended because it could easily mess up the timing.
- Use Timelines. Timeline is a feature in Unity that allows you to make cutscenes. It is a series of animation played one after another. Think of it like a video editor with multiple tracks for different objects.
You would use Timeline for lipsync anyway in conversation, so might as well handle everything in timelines.
I've included Timeline examples in the project files when you download this tool from my Ko-Fi page.
theres a error that states audio phoneme extractor returned empty result could you help? thanks
I see... This plugin has not been updated in a long time. I am considering to update it sometime in the future. I recommend checking uLipSync instead: th-cam.com/video/k5CtTsIKwE4/w-d-xo.html
@@ReForgeMode Thanks!
This is awesome! I wonder how it would be to get that lip sync by webcam
Glad you like it!
Maybe I'll do it somewhere in the future! I'm sure there are some methods I can use. It could certainly be more accurate than this method!
@@ReForgeMode that would be so cool! i hope you do! than you so much! xo
Hi, maybe I'm too much of a newbie to Unity, but I can't get it to work in another model after importing into the scene, could you help me with that please?
Sure! Today is the right time to start learning Unity! Let me know what is the problem in details.
You can also send screenshots so I can help you better by joining our Discord server! Open up a forum there and we'll help you directly.
Has anyone been able to do it?
Thank you so much for this! I've been able to test your project, and I was wanting to import it into my existing project which is already using UniVRM 1. How hard would that be, or do I need to remove UniVRM and import your project as a package?
Glad I could help!
You don't need to remove anything. Though, I would recommend doing the lipsync animation on a separate project and just take the animation files to your real project.
Because when you try to build the game, the build will throw out errors. All codes I used in this project only runs in the Editor, not in the built version.
You can run this in your project, but don't forget to delete my script or move the script to another folder before you build the exe file.
Amazing work!!! I wanted to know if it is compatible with Unity WebGL
Yes, I'm pretty sure this is compatible with any platform, since the end result is just an animation file.
The only caveat is that you should not include this tool in your game build. I will update it soon so it will automatically be discarded once the project is built 👍👍
Amazing!
Thank you!
Thank you for the great videos. They’re crystal clear! Do you know of any version of the phoneme extractor that would work for mac?
Hmmm... No, I don't think there is. But the source code is available, so if you want, you can build the executable yourself.
Or maybe you can use Virtual Machine that runs Windows to do it. The Phoneme Extractor is not a heavy program by any means.
Of course, I have very little experience on doing these, so you might have to do some googling.
Thank you for this!
I had a question regarding loading custom audio files - I am unable to create the animations for them as it throws an ArgumentOutOfRangeException.
Is it a red error or yellow warning? Unity is rather picky with the audio files. Make sure it's .wav just to be safe.
@@ReForgeMode It is a red error warning. The audio file is 16-bit .wav file. The error comes up when I am loading my audio.wav folder to Generate the LipSync Animation. To be precise, the error thrown says ArgumentOutOfRangeException : Index should be non-negative and less than the size of the collection.
Thank you for your help!!
It seems to be a list issue. Can you post the screenshot on my Discord server? (Link is in the description)
If there is any major error, I will need to fix it. Thanks for reporting this!
Amazing Work! Thanks a lot for everything.
Glad you like it! 😄😄
Hi, I got an error at compilation: "Assets\Scripts\CreateAnimationCurse\CreateAnimationCurve.cs(6,26): error CS0234: The type or namespace name 'AnimationUtility' does not exist in the namespace 'UnityEditor' (are you missing an assembly reference?) "
Did I miss something?
Huh, it looks like Unity messing up again. Did you open the project in different version of Unity than what I noted on the github page? That could be the problem.
Hello, I used 2022.2.6f1 . I wish I could give you a screenshot with errors, but TH-cam blocks link to it.
Then, I recommend sending the screenshot to our Discord server instead. We can chat more in there to help resolve your issues.
Oh, seems that this project is not intended for build. I found an explanation related to this problem:
"UnityEditor is a namespace specific for Unity Editor internal stuff and custom editor scripts ... it is not available in a build so you don't want to use anything from it in a MonoBehaviour "
Oh no. The lipsync tool is not meant to be in built. This is because the function that generate the animation file can only be run in Unity Editor. it can't be run in the finished game.
You only want to use the finished lipsync animation file in your final game build.
But if you want to include this in your build and not cause errors, you can add:
#if UNITY_EDITOR
#endif
This will make it so that this script won't be added into your final build, so no errors there.
Heyy there !!
I m a beginner to unity and i don't know anything about it can u guide me that how can i use this lipsync tool for converting tts tokens real time in a video call and integrating bert or gpt for understanding so that avatar replies accordingly and can engage in conversation
Hello! Your use case falls outside the scope of this video, and would require a dedicated project for it. If you'd like, you can commission me to work on your project. Feel free to chat on my Discord server, link in the description. I have experience in doing this for several clients.
@ReForgeMode okay...also i wasn't able to mention before but this tool works wonder and this project really increased my understanding but i was very curious to know about head moving..hair flowing animations of this avatar..needles to say these tools are life saver but i m still trying to learn to use them for my custom needs
Thank you!
Glad I could help 😄😄!
Wow, so the audio files are automatically synced in time to the lip-sync?
Not real time, no. It is prerecorded, so you prepare the animation beforehand.
@@ReForgeMode ah sorry, my question was not well worded. Is the lip-sync in time with the pre-recorded audio?
@@AaronAsherRandall Yes, the example in the intro is recorded from the engine. You are free to try it yourself. It's free unless you choose to donate to me.
Thank you so much +1 sub!
Glad I could help!
can this work for ready player me character please
I recommend using uLipsync instead: th-cam.com/video/k5CtTsIKwE4/w-d-xo.html
👍👍👍👍
can this work with Mixamo characters?
Not exactly. I recommend using uLipSync instead: th-cam.com/video/k5CtTsIKwE4/w-d-xo.html
@@ReForgeMode Thanks
I created a script that does all of this without any plugins or tools.
Just one script.
Maybe I will create a tutorial about it some time.
Very cool! Do let me know how it works and when the video is uploaded!
Of course, this video is quite outdated. My latest lipsync video is using an open source uLipSync plugin that captures mic sound in real time. But it's not accurate as I would like it to be.
I was wondering on how accurate your script will be, and if it allows for live mic lipsync. If you open-source it, I think it will help many people in their gamedev journey. That's always been my goal after all.
@@ReForgeMode
Hey, its a rather simple solution I came up with for my own game.
It has no "live mic" option, so I'm sorry if it sounded misleading.
It works with text based conversations.
(Which can be coupled with a voiceover of the same text).
I simpley provide a phonetic string, and the script loops over it, and moves the A, I, O, E, U blendshapes.
So if the voiceover is "Good Night", i provide a text "gud nait" and it will give me a good enough result, by moving the U, A, I blendshapes, during the voiceover duration.
Obviously i can improve the percision of this script, but for now its good enough.
Today or tomorrow I will add a Short video to my channel showing how it looks, for a tutorial it will probably take me some time though.
@@Mikes-Code I see! So you're handing the automatic phoneme translation to the user instead. Yes, that would work for most cases. I wonder if it works for different people with different accent... I guess you can just change the phoneme translation. Then again, you won't notice the viseme that closely either.
I still want to redo this tutorial sometime in the future. But automating the phoneme translation process is such a pain, because I need to rewrite the whole Radish tools into Unity. That's gonna be a struggle and a half.
Looking forward to your video though! Are you sharing the repo publicly?
Major Fail. I was hoping this system could be used on a custom made character, but your scripts are hardcoded looking for significant words. There is also no ability to link Blendshapes within your scripts
I will update this later. For now, you will need to modify the blendshape names in the script. This is no excuse for me not to make it easier to use. Thanks for notifying me.
Could you tell me what you need so it could work in your character? Like the option to specify where the blendshape is located?
@@ReForgeMode Hey sorry for being an asshole. I tried editing the Clipcurve lines of your code but it wont link to the proper mesh -- not sure why, but its most likely a unity bug back from 2018.
I bypassed this problem by manually copy pasting the actual keyframes into a new Clip that was created in the editor.
Many thanks for the free tool.
@@lorencebuzon5023 Glad that works out in the end!
Don't worry, I need to get used to these kind of comments from time to time. It's humbling.
The only thing i Never learned to teach myself.. creating Mouth Blends... xD... I bet it's super easy since iO nly do Jaw Animations. xD
It's pretty difficult if you never do any blendshapes modeling in Blender. I'm mostly a programmer, so I never even tried those. I suggest finding tools or plugins to make it.
Hello, this is great work. I appreciate the time you dedicate to it. I would really like you to help me switch to another model and use several at the same time in one scene. Please tell me how much you would charge me in dollars in a private message. Thank you very much.
Thank you! Let's talk about it in our discord server!
Why yall acting like this looks good? Barely any of the phonemes is being reflected in the animation. If you're just post processing audio, you're better off faking it via audio amplitude and detecting specific phonemes like "O" and "U" than going this route.
I certainly welcome anyone who can create a better lipsync tools for free in Unity. Because when I created this video, there isn't any.