- 37
- 29 870
Linguflex
เข้าร่วมเมื่อ 24 มิ.ย. 2023
A humble attempt to bring sci-fi dreams of capable Jarvis-style AI companions closer to reality:
This channel is about my journey to write an open-source, ultra-responsive, capable personal assistant with a high quality voice and tons of features.
This channel is about my journey to write an open-source, ultra-responsive, capable personal assistant with a high quality voice and tons of features.
Speech endpoint detection algorithm (new version)
This script listens and tries to figure out when a thought or sentence is finished.
github.com/KoljaB/RealtimeSTT/blob/dev/tests/realtimestt_speechendpoint.py
The algorithm works like this:
- it changes how long it waits before deciding speaking is done
- it waits long if you stop mid-sentence
- medium if it's unsure
- short when you finish with a clear ending punctuation
After each piece of text, it asks an AI model if the sentence is complete. If the AI says yes, it only waits for a very short time.
This helps capturing full ideas more accurately. It uses the AI check to be very responsive when it's confident you've finished a thought, while still giving you more time if you're in the middle of speaking. So it listens more smartly and can quickly recognize when you've finished a complete idea, but also knows when to wait if you're still thinking.
github.com/KoljaB/RealtimeSTT/blob/dev/tests/realtimestt_speechendpoint.py
The algorithm works like this:
- it changes how long it waits before deciding speaking is done
- it waits long if you stop mid-sentence
- medium if it's unsure
- short when you finish with a clear ending punctuation
After each piece of text, it asks an AI model if the sentence is complete. If the AI says yes, it only waits for a very short time.
This helps capturing full ideas more accurately. It uses the AI check to be very responsive when it's confident you've finished a thought, while still giving you more time if you're in the middle of speaking. So it listens more smartly and can quickly recognize when you've finished a complete idea, but also knows when to wait if you're still thinking.
มุมมอง: 1 063
วีดีโอ
Working on a new realtime speech endpoint detection algorithm
มุมมอง 83วันที่ผ่านมา
Working on a new realtime speech endpoint detection algorithm
Second demo: fast, high quality realtime speech to text with new RealtimeSTT library v0.3.0
มุมมอง 46414 วันที่ผ่านมา
Showcasing video transcription recorded over stereomix. This demo uses realtimestt_test_stereomix.py from the tests folder of the free opensource RealtimeSTT github repository: github.com/KoljaB/RealtimeSTT/
Fast and high quality realtime transcriptions with new github RealtimeSTT library version 0.3.0
มุมมอง 8714 วันที่ผ่านมา
Showcasing realtime video transcription, recorded over microphone. This demo uses realtimestt_test.py from the tests folder of the free opensource RealtimeSTT github repository: github.com/KoljaB/RealtimeSTT/
AI at Your Fingertips: Powerful CLI Tools for Speech, Text, and Language Processing
มุมมอง 204หลายเดือนก่อน
github.com/KoljaB/ai_cli_tools This video demonstrates the capabilities of ai_cli_tools, a suite of command-line interfaces for core AI services. Speech-to-text transcription, interaction with large language models, and text-to-speech synthesis - all from the command line. Basic usage, advanced pipelines, and real-time processing, these tools can be combined for powerful AI workflows.
Working on a general memory for Linguflex
มุมมอง 1262 หลายเดือนก่อน
AI memory solution with minimal latency that automatically extracts, stores and retrieves user information during conversations. In systems like MemGPT users must explicitly say "save this info". With this approach the AI chooses when to save information autonomously. So when a user says "my name is Steve" this system automatically stores this detail without additional prompts. With MemGPT, the...
Linguflex with custom wakewords
มุมมอง 1703 หลายเดือนก่อน
Custom wakewords (like "samantha" and "linguflex") now possible in Linguflex. github.com/KoljaB/Linguflex/
Ollama & Home Assistant: Ultimate Privacy with Linguflex for Smart Homes
มุมมอง 6463 หลายเดือนก่อน
Say hello to total privacy with #Ollama support for #HomeAssistant in Linguflex. Smart home automation running fully locally. #OpenSource #SmartHome github.com/KoljaB/Linguflex github.com/KoljaB/Linguflex/blob/main/docs/home.md github.com/KoljaB/Linguflex/tree/main/lingu/modules/home
Linguflex remote controlled with phone browser
มุมมอง 3484 หลายเดือนก่อน
Linguflex remote controlled with phone browser
Milkshake Scene with realtime speaker diarization algorithm
มุมมอง 3816 หลายเดือนก่อน
3 Speakers, short sentences and music at the end. Code available here: github.com/KoljaB/WhoSpeaks File used: realtime_diarize.py
Realtime speaker diarization algorithm
มุมมอง 2.1K6 หลายเดือนก่อน
Realtime diarization of the coin toss scene from "No Country for Old Men", which is a challenge for every speaker diarization engine. Both speakers have a similar voice characteristics, they speak quiet and are hard to understand. This tests a new voice characteristics grouping algorithm with automatical speaker number detection that learns and gets better with time. Code is online here: github...
Current progress: developing a realtime speaker diarization algorithm
มุมมอง 2186 หลายเดือนก่อน
Completely unpolished tests. Basic idea seems to work, yet very early work state still, needs more tests for verification and lots of refinement.
First realtime speaker diarization algorithm test
มุมมอง 1986 หลายเดือนก่อน
Completely unpolished first test. Basic idea seems to work, yet very early work state still, needs more tests for verification and lots of refinement.
Linguflex 2.0 with Samuel L. Jackson voice
มุมมอง 1626 หลายเดือนก่อน
Not Elevenlabs, just XTTS and RVC postprocessing with finetuned models. All local generated in realtime.
Linguflex 2.0 with David Attenborough voice
มุมมอง 1.4K6 หลายเดือนก่อน
Linguflex 2.0 with David Attenborough voice
MoneyPrinterTurbo AI English Installation Manual
มุมมอง 6616 หลายเดือนก่อน
MoneyPrinterTurbo AI English Installation Manual
Setup Guide: Linguflex 2.0 on Windows
มุมมอง 9246 หลายเดือนก่อน
Setup Guide: Linguflex 2.0 on Windows
Speech interruption in linguflex 2.0, a free open-source personal ai assistant
มุมมอง 3387 หลายเดือนก่อน
Speech interruption in linguflex 2.0, a free open-source personal ai assistant
Replacing six speakers at once with azure voices with a single cli command
มุมมอง 15610 หลายเดือนก่อน
Replacing six speakers at once with azure voices with a single cli command
#coqui ➕ https://github.com/KoljaB/TurnVoice = 🔥
มุมมอง 22210 หลายเดือนก่อน
#coqui ➕ github.com/KoljaB/TurnVoice = 🔥
RealtimeTTS v0.3.3 now supports OpenAI TTS
มุมมอง 49710 หลายเดือนก่อน
RealtimeTTS v0.3.3 now supports OpenAI TTS
RealtimeTTS v0.3.0 with simplified chinese support
มุมมอง 16110 หลายเดือนก่อน
RealtimeTTS v0.3.0 with simplified chinese support
Fast local AI talk with a custom voice based on Zephyr model, RealtimeSTT and RealtimeTTS libraries.
มุมมอง 1.7K11 หลายเดือนก่อน
Fast local AI talk with a custom voice based on Zephyr model, RealtimeSTT and RealtimeTTS libraries.
Realtime translation in 6 languages with RealtimeSTT library in under 80 lines of code.
มุมมอง 2.5Kปีที่แล้ว
Realtime translation in 6 languages with RealtimeSTT library in under 80 lines of code.
RealtimeSTT: A low-latency speech-to-text library with advanced voice activity detection
มุมมอง 2.7Kปีที่แล้ว
RealtimeSTT: A low-latency speech-to-text library with advanced voice activity detection
Really impressive results and as you say, this is how people often speak, so handling this well is really important!
hi i am new to coding, want to know how to launch the interface where I can paste text
Code is here: github.com/KoljaB/RealtimeTTS/blob/master/tests/pyqt6_speed_test_chinese.py
wow.. really cool
He's so creepy.
Interesting! What kind of models are you using
faster_whisper for speech to text, Silero VAD + webrtcvad for voice activity detection, llama 3.1 8b for sentence end verification
@@Linguflex llama api?
@@parthpatwari3174 LMStudio
Thanks for working on these projects
Awesome 👏💪
how we can do it. is your code availble and open source . please guide
work in progress: github.com/KoljaB/RealtimeSTT/blob/dev/tests/realtimestt_speechendpoint.py
Thanks for the amazing work . Will it support automatic multi language detection as well? Pls advise
@@SaddamBinSyedWouldn't be hard to implement. faster_whisper is currently buggy when no language is set
Wow, looks like a great tool and you've chosen a great text-to-speech voice! Where is it from?
Voice is mashed together in Elevenlabs, a bit complicated process involving training a voice from different voice sources
why she sounds like sneaky 😁😁🤣🤣
Nice. I like the self-correction.
nice
I'll try to use this and customize it for our Undergrad Thesis, is it okay?
Yes sure. It's MIT license so you can use it for whatever you like.
very nice!
I'd like to use it but there isn't code snippets with just specific functions. I only want the live speech transcription and real time TTS in a simple script, I don't need all of the code for wake words etc etc etc etc, And there's like 2000 encompassing lines of code so I cant even figure out what parts are that specific feature.
Please look into github.com/KoljaB/LocalAIVoiceChat/ project and how both Realtime libraries are used in the ai_voicetalk_local.py
Also please look into the tests folder of both Realtime libs. There are simple code examples without usage of wakewords in the RealtimeSTT tests folder.
@@Linguflex Okay, thanks for replying!
This looks really promising, i will try to test it on my programs! Thanks for the work my dude!
Absolutely amazing, great work!!
u got github ?
Yes, github.com/KoljaB. No code for this one up yet tho, bcs it's too early work state
Can I run it in a raspberry pi 3 B?
Does it need gpu cause I have a i3 only
Can you help me? :( TomlDecodeError: Reserved escape sequence used (line 100 column 1 char 3696) Traceback: File "C:\Users\tiago\miniconda3\envs\MoneyPrinterTurbo\lib\site-packages\streamlit untime\scriptrunner\script_runner.py", line 584, in _run_script exec(code, module.__dict__) File "C:\Users\tiago\Desktop\Try\MoneyPrinterTurbo\webui\Main.py", line 34, in <module> from app.services import task as tm, llm, voice File "C:\Users\tiago\Desktop\Try\MoneyPrinterTurbo\app\services\task.py", line 8, in <module> from app.config import config File "C:\Users\tiago\Desktop\Try\MoneyPrinterTurbo\app\config\__init__.py", line 6, in <module> from app.config import config File "C:\Users\tiago\Desktop\Try\MoneyPrinterTurbo\app\config\config.py", line 42, in <module> _cfg = load_config() File "C:\Users\tiago\Desktop\Try\MoneyPrinterTurbo\app\config\config.py", line 30, in load_config _config_ = toml.loads(_cfg_content) File "C:\Users\tiago\miniconda3\envs\MoneyPrinterTurbo\lib\site-packages\toml\decoder.py", line 514, in loads raise TomlDecodeError(str(err), original, pos)
Sorry, can't help. I have nothing to do with MoneyPrinterTurbo. This is a translation example as showcase for my TurnVoice GitHub project.
You gotta put the pexels API key in quotes like this - """ pexels_api_keys = [ "nE113EOVlRVbpWvRE0yZFuy6KmM9WAqvelyadayadayada",] """.
Action latency is yet to be improved , amazing project❤
Great work..do you have any ideas to reduce latency in text to speech..im working on it..
Looking promising!
Does Linguflex allow you to interrupt the model?
Yes, it does. Voicebased or via escape key. See: th-cam.com/video/uQ8jJtalc9M/w-d-xo.html
Hey mate love your work. Unfortunately i cant get this software to run. Finally got your chat package working though (forget the name), adapted it for ollama (using langchain) as i just cant get the cuda version of llama cpp working (so takes about 20 secs for a response)...langchain just seems a little more straight forward and ollama is blazing fast. I was also experiencing a lot of those pipe issues, changing python version fixed it for me. Seems a difficult bit of software to maintain (like all AI packages) given the number of dependencies which are very sensitive to exact versions of other packages. But keep it up, amazing work.
Codebase for this: github.com/KoljaB/EmoTTS It uses OpenAI. Ollama is best choice I think for local models. You're right about maintaining AI projects ends up in dependency nightmares. Of course you can set fixed versions, but if not all dependent libs do this too it will mess up the environment nevertheless sooner or later.
@@Linguflex will be amazing when local llms are capable of this. Ive never bothered with openAI for these things, though I am a paid subscriber, aren't the costs in addition to that to use their API? whilst I only have 12GB of vram on my laptop, stuck with the comparatively dumb llms. Though even some of the coqui voice models at time express emotion, the quality of the text they are reading obviously plays a huge role. The thing. I hate about all ai libraries being stuck to specific python versions and package versions...is I must have installed a venv for just about every bloody python version and every bloody py torch version. My love/hate relationship with python.
@@bigdaddy5303 Very true words. It was an insane hazzle to combine all needed libs for linguflex into one single environment. We already have ollama support for linguflex in some implementations on our discord (so I have and another dev but we didnt release it yet). Should be coming soon...
would this project be implementable into a business
depends (tts license etc), but sure
I'm attempting to run a few example clients provided in the git. Maybe there's something obvious I'm missing, but an error I commonly run into is: ModuleNotFoundError: No module named 'RealtimeSTT'
If you did perform "pip install realtimestt" before then probably your pip/python env is somehow corrupted. You could try it in a virtual env.
@@Linguflex I knew it was something obvious 🤦 Don't know how I missed that step lol Thank you for your help
im gonna try to make a Vrchat STT app that puts the words above my head using their osc system :D
Awesome. How is interruptions implemented here? Thank
The interruptions are triggered by the volume level detected on the microphone. It's a straightforward implementation that unfortunately also gets interrupted if the text-to-speech (TTS) output is too loud. To eliminate this issue, implementing echo cancellation is necessary, but that process is quite complex and not trivial
Very nice. Greatjob❤ Out of curiosity, how would you handle back to back conversation with interruptionhandling without using space?
Thank you. We talked about how to do solid interruption in my discord channel recently: discord.gg/f556hqRjpv Highly encourage you to join, it's a great place to ask questions, share progress and get support from tech enthusiasts. Would love to see you there!
I really love your content and aspire to be like you. Although I am new to programming, I am learning Python and JavaScript simultaneously, which has been quite challenging. I live in the Kakuma refugee camp, where few people are interested in contributing to technology, focusing more on their own lives. Could you please update your full roadmap so that I can follow it? Additionally, it would be great if you could create a Discord community for discussions and support.
Thank you so much for your kind words! ❤ Great to hear that you're learning both Python and JavaScript at the same time in Kenya, I think your determination will definitely pay off. I don't have a specific roadmap since I usually just develop projects based on inspiration and current interests. I highly encourage you to join our Discord community: discord.gg/f556hqRjpv It's a great place to ask questions, share progress and get support from tech enthusiasts. Would love to see you there!
🤑🥰🥰🥰🥰🥰
I was looking to a way to add auto generated subtitle to offline videos. Can it be used for this porpuse? Didn't it need any speech recognition model?
I wouldn't use my realtime libraries for this: they are optimized for speed, not quality. I'd generate a srt file transcript with word-level timestamps with a good STT library like stable_whisper, then add the subtitles to the video using that srt file (ffmpeg can do this for example).
That's incredibly accurate. Nice work! Can you active-transcribe AND wake-word for commands? It'd be great if you could have it always listening and then do something on wake word.
No, currently not. The idea is good, I can see some use-cases for this. I'll think about that.
I love you work
Can I have it
Sure, here's the repo: github.com/KoljaB/WhoSpeaks
Can this be used hands free or does the mic need toggle per reply?
Can be used hands free
This amazing! Is there anyway I could run this on my computer then talk to it using a headset connected to my phone so it’s mobile?
Hello man I have difficulties running your library, can you help me ?
Sure. Could you write your problems as an issue, so others can see how it's solved afterwards? Depending on if it's input or output: github.com/KoljaB/RealtimeSTT/issues github.com/KoljaB/RealtimeTTS/issues
I don't understand how to use it...
What do you want to do? The "tests" folder contains some examples how you can use it: github.com/KoljaB/RealtimeSTT/tree/master/tests Maybe also the "tests" of RealtimeTTS can help, they also use RealtimeSTT a lot: github.com/KoljaB/RealtimeTTS/tree/master/tests
Impressive work, thanks
at the prerequisites 0:39 i downloaded cuDNN as a zip file, but i have no idea where to place it. is there an executable inside or do i extract them somewhere specific? (edit, for people that will be wondering the same: th-cam.com/video/OEFKlRSd8Ic/w-d-xo.html ) as well as ffmpeg. as a non developer, it's not pretty straightforward :( (edit: tutorial for this one th-cam.com/video/jZLqNocSQDM/w-d-xo.html )
Thanks a lot for this information, as a dev I'm often just not aware of the problems the users have. I'll rework the Prerequisites section, thx again.
Yeah, I followed that tutorial above to install cuDNN to a tee, but no matter what I do, when I try to install Linguflex, it says cuDNN installation not found. I know it must be that my PATH's are incorrect, but I can't figure out what is wrong with them. I've been trying to get this to work now for 5 hours... Bottom line is that I simply can't install cuDNN successfully. At this point, I have so many PATH's in my Environmental Variables, I don't know if maybe they are conflicting with eachother or what... I added them for me as a User and in the System variables as well but it always tells me cuDNN installation not found. It's a bummer, I would really, really like to use Linguflex... I installed the correct Python version, ffmpeg, the CUDA Toolkit, all of that is fine... just the cuDNN is the brick wall I can't get past.
And this is the worst it will be. Can’t wait for next update.
Diarrheazation?
This is epic, thank you for sharing!
Insanely good distinction after just a moment. Meanwhile i had to try and close my eyes and try to guess myself. Might aswell have been a monologue
Interesting but clearly struggling a bit lol
What language you used?
Python
I know basic python, where do I go next?
@@niscchay he literally posted the github page lol
Wheat
Hello, thank you for this great project. I know it's not the point but would it be possible to add the option to use text as input for the user ?
Yes absolutely, we need that. Any ideas where to place it in the UI?
@@Linguflex I know there are quiet few icons already but the simplest would be maybe to add another icon that would open a chat window ?
Super cool! is this in a public repo?
Not yet but soon