Super Fast Voice To Voice AI! | Voice Cloning with so-vits-svc
ฝัง
- เผยแพร่เมื่อ 24 ก.ค. 2024
- Clone any voice! Have you ever wanted to sing a song in a language foreign to you? Can't be bothered to actually learn how to speak that language? No problem! Thanks to the amazing power of FREE open source Voice To Voice technology, anyone can remix any audio with the voice of their choosing - in just SECONDS from any trained model! So fast, you can even run it in real-time :)
German, French, Dutch, Japanese, English, Welsh, Cornish, Spanish, Portuguese, Bengali, Greek, Latin, Sumerian, Akkadian, Kawishana, Paakantyi - whatever language you speak or want to speak (or sing!) it does them all! Any language really does actually mean exactly that 😉
Super quick and easy to install locally for maximum fun! Not suitable for children.
Some ideas for research:
* Want to hear a loved one again? If you’ve got their voice recorded, then you can do exactly that
* Lost your voice but have some old recordings? Hear yourself speak or sing again!
* Develop a way to have a chatbot speak to you using your voice
* Take audio generated by a TTS and convert it into your own voice
All free, ready for you to run on your own computer at home (or using Google Colab).
Enjoy!
Update: for Issues relating to “fairseq” on MS Windows, see the GitHub page for fixes!
== Links! ==
* so-vits-svc-fork - github.com/voicepaw/so-vits-s...
* song - pixabay.com/music/pop-french-...
* anaconda - www.anaconda.com/
* pytorch - pytorch.org/get-started/locally/
* AudioSlicer - github.com/henrymaas/AudioSlicer
* Spleeter - github.com/deezer/spleeter
* Demucs - github.com/facebookresearch/d...
* Installing Anaconda for MS Windows Beginners - • Anaconda - Python Inst...
* Text To Speech with Tortoise TTS - • AI Voice Cloning - Tor...
== Stable Diffusion ==
* Talking Faces! - • Create your own animat...
* Stable Diffusion Playlist! - th-cam.com/play/PLj.html...
* Interested in adding things to your AI Art? Try these!
Dreambooth Playlist - • Stable Diffusion Dream...
* Textual Inversion Playlist - • Stable Diffusion Textu...
Note that GitHub repositories change often. Be sure to check for any changes!
0:00 Introduction to voice to voice cloning
1:34 so-vits-svc installation
3:56 Downloading so-vits-svc pre-trained models
4:06 voice dataset for so-vits-svc
5:56 training so-vits-svc
10:26 so-vits-svc inference - วิทยาศาสตร์และเทคโนโลยี
![คุณอาเรียโต๊ะข้างๆ พูดรัสเซียหวานใส่ซะหัวใจจะวาย ตอนที่ 4 [ซับไทย] 【Ani-One Thailand】](http://i.ytimg.com/vi/_vK_cden2f8/mqdefault.jpg)
Nerdy Rodent I am always amazed at your brilliance. You are at the top of your game as always! Thank you for your talent.
Thank you too 😀
Brilliant tutorial. I love your videos especially because you use open source programs. I wish you would make an ai music generator tutorial next. Thanks again for your great work.
just stumbled upon this channel while looking for a tutorial for so vits svc. I'm thoroughly impressed by the quality of this tutorial. you couldn't be more thorough and clearer. not only you provided a tutorial but also explained what to do about splicing and removing the vocals, you've earned a subscriber.
Great to hear - thanks! :)
Really like your tuts on newest neuro repos! Thank you, Nerdy Rodent!
Glad you like them!
Congrats. You got credited as a contributor to the so-vits-svc-fork repo. 👏👏👏
Oh cool! 😀
Amazing video, so well done and loved your way of explaining it!! ♥
Glad it was helpful!
This must be one of the clearest tutorials about a collaboration of github / CLI tools that I have ever stumbled upon. Stunning work 👍 thank you
Glad it was helpful!
You sing really well pal 😆 While there's many idiots doing crap videos about it, yours is an excelent tutorial, covers up all the necesary stuff. Congrats from Buenos Aires.
Glad I could help!
WOW!!! Thats awesome!! I was wating for something like this! Thank you very much
Glad you liked it!
About a year ago I thought nothing could top Dall-E... the last two weeks we seem to be taking 10 steps forward every day. I fancy hearing Donald Duck singing some ACDC... and now it's possible.
Because why not, eh?
Lmfao dude ai has limitless potential. Seriously, as a kid I knew this would happen one day and it so happy it's happening in my teen years.
One day, AI will be able to generate any scenario prompt you throw at it to amazing detail in a virtual reality space
On the contrary this is only starting
The AI french wasn't that bad. Nice, would love some more in depth guides/tutos to this
There’s not much else to it tbh! 😃
Nerdy Rodent always dropping the best AI tuts around .Now just replace Snoop Dogg's voice with yours and make a sick intro to your vids 🎤🎵
Wow!! I'm training my voice right now! Thank you for the instructions.
You got this! 👍
how did you get this to work?? ive been at this legit 2 hours and i cant get the gui to open
Thank you so much, this was super helpful! I have mine training now 🎉
😀
My Nerdy Buddy again gave me exactly what i was looking for! Nice, mate! 👊😎 Although, in the " AnimateDiff New Model! A1111, ComfyUI, ControlNet " i did not understand what you meant with Conda, than when I installed an other AI program it messed up the AnimateDiff ( : ... So its very cool that you speaking about this virtual environment feature at 02:35 !
For anyone wondering why nothing showed up in the folder you set as the install path like me(im very new to all this). just create a dataset_raw folder then a folder with the speaker id(this can be anything) then put the wav files there and then run commands as shown in the video. The folders you see in the video will be created as you run these commands, they aren't instantly there after you run the command to install the so vits fork. also great video 💜
Awesome tutorial as always!
Would love to see some free, open source text to voice in a good quality
This is indeed free and open source!
you may already know this but TorToiSe TTS is currently the best open source TTS AI. Its not as good as eleven labs but its pretty powerful, you only need a handful of example audio and a few minutes of training (each run, that's why its called Tortoise). There's also an AI called Bark that focuses more on inflections and emotions but its currently in its infancy and doesn't reliably output usable results
Here's a tip I've prepared for those encountering the 'ValueError: Invalid integer data type 'f'' error in AudioSlicer. I'd like to share my experience with you.
Wav files can be represented in a manner using bit depth as integers or as floats. If you're dealing with a Wav file in float format, you'll encounter an error. Therefore, you need to export your Wav file in int16 format. Keep this in mind.
This is awesome!! Some day imma donate like $100 to this channel! Don’t have the money yet, but you’ve helped sooo much!
Held out to the last minute and BOOM! was worth it. This is amazing technology. Thanks for sharing. I'm super interested to see a heavy metal drum being transformed into a softer brush percussion for like a jazz piano vibe. We already trained for speech .... and the separation part with spleeter is straight forward.... how can we do it..... some one must be working on this.
I sort of want to try with out of scope things like animal noises and other weird sounds 😆
Oh there's people working on it alright, these tools are getting crazier by the day
you have a great voice btw!
Thanks 😉
I dont know anything about python nor anything related to Ai... man I cant even speak in English properly and see me here, trying to learn how to use this, thx for teaching me and keep up the great work!😁😁
Excellent work.
Awesome content!
Thank you for the step by step guide and immense insight into the process and steps required to train and infer. We all really appreciate it! ☺️
You're very welcome, and thank you! 😀
These techs really heats my spirit
Amazing voice !
Damn another rabbit hole. Ahh. Awesome
Weeee! 😆
Awesome video! I've already made a dataset and it turned out great. However, if I want to create a separate new dataset of another voice, do I simply move my old datasets & log folders out and replace them with new ones and start training from there? Thank you
A very good tool. well explained thank you sir
it is so different when using G collab I kind of wish you also did a tutorial along side that as well. There is a lot of guessing and pretty much no info on this online except for this one video on the non g collab version.
So, this was the surprise. Cool 😎🥶
Nice tutorial, but I wish you'd demonstrated it on normal speech too. Singing, I think, is actually easier to make sound realistic/good than normal speech, since it's got regular pitch and rhythm, and we the listeners are used to compression and loads of effects, etc.
I had a listen to the Element Song short, and it sounded good, though at places it sounded like you had a sore throat. Not sure if that was the model training or poorly split voice/music tracks.
Great content, thanks for the tutorial!
😀
So we could get Elvis interviews for his voice and then have him sing Where The Streets Have No Name by U2?
Merci pour cette super vidéo mon rat préféré !
🥰
merci, et vous êtes les bienvenus!
i've never found the video on youtube, but back in the 90s, ray kurzweil demonstrated this technology (cepstral deconvolution, match phoneme).
Is there any way to make the realtime voice changer be instant or near instant without degrading the quality or is that just a "in the future" thing. Because it sounds good but the issue is when I say something it doesn't go through for 3 to 4 seconds. I already got a means to use it in games through Voicemeeter but that delay is a bit of a killer.
pretty cool!
Thank you very much!! great tutorial. but i stil cant get it to work with collab. But sadly, I dont have enough VRAM to train local. Would you please make a tutorial on how to train in collab??
hello, could you tell me what the command would be like to execute the inference in colab without the auto prediction and to be able to choose the type of inference to crepe?
Fab Fab Fabulous!
So cool!
This is an impressive tutorial! I want to have voice-to-voice in Italian. Should I record my voice in English or Italian? I guess the model is pretrained only on English voices
Either. You can just use the any language. Most of the examples were in Japanese.
Very nice and it works on OSX to, a bit tricky with pyenv & ancanda but works great even without GPU
Awesome, good to know!
great tutorial! i'm missing a few things, What if you start a new training? you simply delete everything from the logs/44k folder and start training again? or is there some way u can devide trainings in seperated folders? Also what's really the difference between the D_ and G_ part of the files output?
Yup, you can save your old files before starting a new project. You only need the generator if you’re not going to continue training, so it’s fine to remove the discriminator if you’re only doing inference
Whatever you do, do no make your mom sing this song, lol.
Very impressive tech and the install doesn't look too frustrating, I'll try it out! Do I have to read lines to clone a voice, or can I use samples? I'd love to clone some character's voices.
Great video as always! I would love to hear an English song so I can pick up on the tone and inflection errors. Curious if any French speakers can comment on how well it did.
Then check out the yt short where I sing the elements song - particularly near the end with the non-standard pronunciation of the word “discovered” 😉
As a French speaker I can tell you that it sounds a little bit strange. It is like he has an English accent. Not native French speaker. But it’s still quite good
is there a way to add additional information to a training model? or do you have to just retrain the entire data set?
great tutorial!
but can you show how to continue developing the model with newly added sample data?
An other interesting video, thanks again.
Glad you enjoyed it 😀
super cool!
Thank you for your video. Informative. Important
You are most welcome. You may like my latest video about RVC which has much of this built into a single web interface and it trains faster!
7:30 I'm a little confused here. I'm not sure if I have any VRAM, because I don't have an external GPU, only a CPU. I'm having trouble figuring out what settings to use. 🤔
Great video!
I have a question…
It’s not important if some audios contain background music?
The following is from Google Translate. Sorry, my English is not good. I deleted some of the old D and G models, and only kept some of the new ones, and they corresponded one-to-one. However, now choosing to continue training prompts me: load old checkpoint failed and will start training from the beginning, what can I do to get it to continue training from the last time?
Hay Nerdy Robert. That looks awesome. Is there a way to make this as portable AI that can be taken with all of it's data and use it on different computers running it from a disk on key or external hard drive?
Sure, you could indeed add it to your Ubuntu portable usb stick
When I use the train command, I get this error:
Expected 2D (unbatched) or 3D (batched) input to conv1d, but got input of size: [16, 2, 80, 929]
What can I do?
We need a model repository for this project
Where do you run the pre -resample command? In the command prompt that pops up after the gui does? Or your own command prompt? Because you can't type in the gui prompt and it says svc isn't a recognized command in my own cmd
brilliant
Any ideas why pip doesn't work for me? I've tried so many solutions but I just can't get it to install anything! It always says pip isn't recognised. i've tried reinstalling python/pip. Tried changing it via the control panel and Ive tried various commands, this is so annoying!
Hi nerdy rodent, im trying to use this tut and so far its been very comprehensive (tysm!) however i have an amd card and over at pytorch its saying that rocm isnt available for windows. im not exactly very intelligent when it comes to this coding stuff so i was hoping you might ahve asolution. thank you!
Yes, Linux is best for AI performance, stability, ease of use and compatibility.
Can’t wait til this gets streamlined lmao
Great video.
Thank you so much
Man your timing on this video is perfect, just what I was looking for! You are using Linux in this tutorial right? If so, is it Ubuntu or something else?
Yup, Ubuntu 22.04 with an Nvidia GPU!
nice video!! could you give me a link or a website where I can find other voices like Kendrick's one for example
i need help lol
when i do svc pre-resample
it tells me Error: Invalid value for '-i' / '--input-dir': Path 'dataset_raw' does not exist.
please i need help what do i do
Please, a video for google colab! Thank you so much!
I was thinking, is possible to use this method as a sort of voice changer? For example, i want a character giving me a line with a particular tone, but tts don't give much acting. So i record myself percorming the line, and then use that lime a sort of acapella base.
Yup!
Thank you for sharing! ❤
I got couple of questions, I hope you don't mind:
1 - Can I create the dataset from another Language, or it must be English?
2 - Running this locally on my Windows PC - Will it work with 4GB Nvidia with the right settings? or by default it needs something crazy as 14GB VRAM?
Thanks ahead!
For training you’ll need at least a mid-range consumer GPU, so colab for that! Inference you may get away with it…
@@NerdyRodent Thank you!
I hope it can work with other languages for data set I mean.
Noice will check it out
What would happen if I combine my voice samples with a friend of mine's voice samples? Does it give some sort of hyrbid between the two?
Awesome, the fact it can do different languages gives it and edge from other available options like xVASynth. What is considered a "reasonable" GPU? How much VRAM is required to do training and generating?
You should be able to get away with 8GB VRAM
Another fine video! I’m trying to learn basic code to keep up..😂..does this mean you can use this sorcery and create the voice track to any video? I want to use with a talking head as per your other vids? Thanks
Yup. Any voice to any voice!
Excellent! So can you then use this audio track for your thin plate spine driving video? Guess you need to somehow replace the original audio file? I’m leaving out lip syncing for now lol
@@musumo1908 Sure. Wav2lip can sort out the lip sync
im using an a6000 gpu with 48gb of ram. if i increase the batch to 30 does this increase the quality or just the amount of time it will take? I have noticed that stuff at 10k epochs sounds better then ones at 20k. How do you know if it is over training? been using data set of 200 samples. Are more samples better or is it overkill at a certain point? Alot of questions sorry lol
How would I go about adding more samples to an already trained model?
Great video - these trainings are hitting my CPU, and meanwhile my GPU is chillin' not doing much at all. When Tortoise is processing, it makes a satisfying 'working' sound. It's an Asus 3090 Strix. Not sure if I need to specify GPU anywhere, but I'm using Cuda 11.7 in my env, and followed the default instructions (avoiding the cpu/amd ones). 62c on CPU, 43c on GPU without anything running except, well, this video. - just wondering if I'm missing out by letting my ryzen do the lifting
The only way I can think of to not use the GPU would be to install the CPU only version of pytorch
The GUI version works amazingly well, but I can't seem to get the CLI version going... Anyone been down that road?
Do you know where the HuBERT model of so-vits-svc-fork is?
Thanks Man !!! I just wonder if i can do this on mac mini or macbook pro ? I was no idea at all. or should i buy a laptop with window software. thanks again.
Not locally that I’m aware of - but you could still use the colab. Linux + Nvidia is best if you’re getting new stuff!
can you do other things than switching the language
I heard that it can slightly alter the lycris while still keeping the same energy
I have a question. If i want to train a certain speaker, what command do i have to type to train only that certain speaker ? thank you for answer and this brilliant tutorial.
That would be just like I show in the video, where only 1 speaker is trained 😃
One of my biggest questions trying to learn how to do all this is that if im wanting to make an ai voice of a character I like and then use him for cover songs, but the problem is there's not many singing clips of his voice so i've been just wondering do they having to be singing clips? or can they be talking normally and that's fine? (ik this is probably a stupid question but it's been a huge struggle just trying to learn all this)
They need to be audio clips of the voice. Try to avoid any backing music.
You always make videos about things I'm looking for or didn't know I needed. Thanks very much 🫶
😀 Glad to be of service! And much thanks to you too
I just spent an hour scouring the internet for a copy of the "practice for speech quality measurement" text - I can't find access to that file anywhere. Help me out bro -thanks..
Not sure where I went wrong. Environment installed but i get svcg is not a recognized command.
haha as a french the intro song was hilarious to hear
How is the latency in the real-time voice changing? Let's say, when the latest RTX cards are used? Can it benefit from multiple cards?
Not too bad, just a slight delay
Why do my models sounds like freaky ass robots?
My friend and I are in a band, I took a vocal recording from one of our songs, and ran it to train on 66 (10second) segments. I let it run for 10000 epochs on the RTX 4090, the end result was pretty shiet tbh! I don't know what I'm doing wrong, is it just training duration? Should I let it run for 10+ hours? I can do that, the GPU is water cooled and the CPU is water cooled I have no issue running this for 24/7
what to write in speaker id? when I press infer I'm getting this error ValueError: Speaker_id 0 is not found.
Nice! I wonder if you can take the output of a text to speech program like Tortoise TTS or coqui-ai as the input for so-vits-svc to match someone's voice and/or improve quality.
Yes, you obviously can ;)
Yup. Or tts in another language too!
@@ChristianIce I suppose what I'm really asking is how much less robotic the voice would sound if you do. TTS also often sounds over-compressed like an 8KHz telephone conversation rather than a 44.1KHz studio recording.
(Now I'm also asking myself if I care enough to start experimenting.)
@@nathanbanks2354 i have the same question, pls update if you try it out...
@@nathanbanks2354
It will sadly sound robotic exactly the same.
This technique mimicks the input voice, that's what it makes more natural compared to TTS, to the point it can mimick any other language, no matter what.
This sadly means that if you input a flat inexpressive voice, you would just get a flat inexpressive voice as output.
May I ask what the final music synthesis tool is?
Using Google Collab, I can't seem to get the Inference GUI active. Any ideas?
Great tutorial, thanks! Just a few questions: if I have dataset of 5-7 mins of audio in total for voice, how much epochs is better to train? Because in automatically generated config, there are 10000 epochs and it seems quite a lot
Yeah, I start at 20k steps
@@NerdyRodent My 128/16 * 3125 = 25,000 steps, so my epoch is 3125, is this alright?
Hi,
after a couple of runs, Im getting a CUDA out of memory exception...how do I fix this? thanks
WIld, I already created a soundpack for RC radios with 850 sound files I've recorded and cleaned up of my voice with Audacity... I wasn't using the best mic at th time... some junk from Walmart, that I've since upgraded to a K668B, but still... wild that I already have a massive dataset of audio files of my voice before even getting started.
Sounds great!
I'm a producer who writes and compose the melody and lyrics for all my songs. I don't have a marketable voice so I've always had a hard time finding the right singers for my tracks. My understanding of this technology is limited for the moment but, does anybody here knows if currently one can make a custom voice with AI to overlay my own voice and be able to produce vocals that sound great without the need of a real singer???
love ya