this is so great!!! what fun! I picture the underqualified candidate who's code crashes mid interview and all of a sudden can't answer 1+1. (: But seriously, I have some cognitive disabilities, even something to keep me on track and succinct in answering my questions is absolutely invaluable. I am seeing this as a reminder tool for me, not an answer tool per se. so powerful and enabling for somebody like me. I imagine it helping my confidence which will allow me to be more of myself and let my natural attitude come out instead of being clouded with mumbling, with racing thoughts.. (as many of my other high-frequency colleagues have) thank you very very much for taking the time to share with us!
🎯 Key Takeaways for quick navigation: 00:00 *🎬 Introduction to Real-Time Conversation Co-pilots* - Introduction to the concept of real-time conversation co-pilots. - Overview of the challenges and potential benefits of using AI in conversations. - Discussion of past attempts and the need for low-latency solutions. 01:39 *🚀 Real-Time Co-pilots in Professional Settings* - Examples of real-time conversation co-pilots in professional contexts, such as aerospace engineering and job interviews. - Consideration of the value and ethical implications of using AI in interviews and professional settings. - Potential for improving interview processes and enhancing communication skills. 03:29 *🤝 Real-Time Co-pilots for Social Interactions* - Exploration of the benefits of real-time conversation co-pilots in social interactions. - Personal anecdotes about the challenges of social interactions and the potential support AI could provide. - Discussion of the broader applications beyond professional contexts. 04:24 *📱 Building Real-Time Conversation Co-pilots: Web and Mobile Apps* - Overview of building real-time conversation co-pilots, including web and mobile applications. - Introduction to the technical components required for such applications. - Step-by-step guide on using platforms like Replicate for deploying AI models. 04:53 *⚙️ Technical Challenges: Real-Time Transcript and Fast Inference* - Discussion of the technical challenges involved in achieving real-time transcription and fast inference. - Solutions for real-time transcription, including recurrent loops and optimizations for accuracy. - Strategies for achieving fast inference with large language models, such as model selection and optimization techniques. 10:19 *🛠️ Implementing Real-Time Conversation Co-pilots: Demo and Iterations* - Overview of the iterative process of building real-time conversation co-pilots. - Demonstration of a web application prototype and its functionality. - Integration of AWS services and Replicate for deploying and running AI models. 17:30 *🛠️ Backend service setup and frontend basic structure* - Setting up the backend service involves defining routes and handling requests. - The frontend structure includes defining HTML elements and basic CSS styling. - Functionality such as recording audio and fetching suggestions is outlined. 23:18 *📱 Exploring Whisper Kit for mobile deployment* - Whisper Kit, an open-source fine-tuned model, enables deploying speech-to-text models on mobile devices. - Real-time transcription with minimal latency is demonstrated on devices like the iPhone. - Whisper Kit's optimization allows for efficient use of resources on mobile hardware. 24:01 *🎁 Unboxing and setup of Apple Vision Pro* - The unboxing and setup process of Apple's Vision Pro headset is showcased. - Details about accessories, boot-up procedure, and initial impressions are provided. - The demonstration highlights optimizations for improved accuracy in AI-generated transcripts. 25:14 *💻 Development using Whisper Kit for iOS apps* - Utilizing Whisper Kit Swift package to develop iOS apps for real-time transcription. - Setting up the project in Xcode and configuring model selection and streaming options. - Overview of the app's structure and explanation of key functionalities for transcription and model loading. 31:03 *🚀 Integration and testing of conversation co-pilot features* - Defining variables and functions for user input, API interaction, and response handling. - Adding UI components for user prompts, transcription display, and interaction buttons. - Demonstrating the setup process and testing the app on an iPhone for real-time transcription and suggestion generation. Made with HARPA AI
@@bassamel-ashkar4005 phone is cool but just generally a conversational agent to utilize when having any conversation anywhere. It seems to be an extremely useful concept in so many ways.
Holy, whisper kit is insane, running the model on mobile device directly gonna be the future; Also thanks for sharing Replicate, didn't know it is free to use!
Hey Jason, interested in the iPhone project (tho android would be better). How about adding an extra layer to this flow that translates the text to a different language and speaks it out loud? We could then have real-time conversations with someone who isn't the same language as us. Here's the use case. You hand over the left earbud to your Polish friend. You use the right earbud. Ideally both earbud would have a mic. The application on the phone would: listen to the speech in English => do a transcribe on the chunk => do a translate to Polish on the chunk => speak out the chunk in Polish. Then when the Polish person is talking, it'd work the opposite way.
I need this for screening consultants. They are very smooth talkers, the minute you scratch their gloss they fall through, unless they actually know what they are talking about. Can't wait to see your app. Thanks for sharing!
All your videos are amazing! One gotcha, if AWS give you an "access denied" error you may need to update S3 policy to add a "/*" onto the back of the resource ARN - the wildcard permits access to all bucket objects
This is amazing Jason. Just subscribed to your channel and am very interested to see the iPhone app you build and other AI projects that you're working on. I see a lot of amazing use cases for this like overcoming objections on sales calls, asking better questions on podcasts / interviews, etc.
I tried to build a similar interview co-pilot before too, but the latency made it not usable; Can't believe how far we went with those model performance past few month!
You do know that you can use OpenAI API to get sub 2-3 sec outputs right? The only thing not possible on GPT is a system prompt (I think, I have never needed to use OpenAI's API). But on my Mac with a capable 7b parameter like Mistral, or Mixtral model, the output is also within 5 secs (especially when loaded in GPU memory). I prefer local generation vs online since in local you can modify the system prompt and you can customize the output a lot more.
Kindly make a tutorial to fine tune an open source LLM model on many pdfs data. The fine tuned LLM must be able to answer the questions from the pdfs accurately.
This is amazing, thank you for great content. I wonder, I tried this (i'm a beginner with a newly found passion for learning python). I don't have the same amount response on the web app like you have, I get a 8-10 second delay both with the transcript and suggestion. How can I fix this?
Ok now i want to build an agent that can listen to videos, copy and build the apps. There is way too much cool AI stuff to try and i have other hobbies and a life too xD
In the future, Neuralink will be wired directly to the brain activating the vocal chords so that the interviewer is mind blown on how you're answering the questions even though it appears you fell asleep during the interview.
Me: "I think I can do this. I'm going to give it a shot!" Tries executing this following Jason's steps. Gets error message at first step when installing replicate into VSC. "command not found" . Watches 3 videos to see if I can figure out why VSC is giving me this error. Still not working. Feels defeated and quits :(
This is a great tutorial and illustration of how to use services, but your bucket policy on Amazon needs to be locked to just your user so nobody can mess with your bucket and hijack it. It's one of the most common ways people get their data leaked.
What's the point of passing the interview when there is a probation period in which real tasks need to be accomplished. And if those tasks are still doable by LLMs, it is just a matter of time until that position will be completely automated.
I don’t do the ‘TH-cam’ other than to watch. I liked. Subscribed. And now I am kindly asking you how can I give you money? What you have here can help so many people. I eagerly await for this blessing to come to fruition.
If you get a teleprompter like the Elgato Prompter it acts as a 2nd monitor so the other person will never see it, plus you can be looking directly into the camera (while reading the info) at the same time so it looks super natural!
30 sec. and i find so pathetic the use that some people give to the tools , faking an interview , if you suck at work pls , dont do an interview imgonna fired you after a month , why you are gonna lie for a month of pay , if you suck at one job maybe you can spend the improving your skills , but young people of this days really Suck so badly
Would be funny when both parties of an interview who uses the co-pilot finds out about each other.
Oh it'll happen, it already does to an extent, we use ai to help us find jobs, they use them to scim threw applicants
You just gave me an idea 👍
Lmao😂😂😂
@@figs3284 I love random ideas :0, can I have a spoiler? 🙂
this is so great!!! what fun! I picture the underqualified candidate who's code crashes mid interview and all of a sudden can't answer 1+1. (:
But seriously, I have some cognitive disabilities, even something to keep me on track and succinct in answering my questions is absolutely invaluable. I am seeing this as a reminder tool for me, not an answer tool per se. so powerful and enabling for somebody like me. I imagine it helping my confidence which will allow me to be more of myself and let my natural attitude come out instead of being clouded with mumbling, with racing thoughts.. (as many of my other high-frequency colleagues have)
thank you very very much for taking the time to share with us!
Thanks!
🎯 Key Takeaways for quick navigation:
00:00 *🎬 Introduction to Real-Time Conversation Co-pilots*
- Introduction to the concept of real-time conversation co-pilots.
- Overview of the challenges and potential benefits of using AI in conversations.
- Discussion of past attempts and the need for low-latency solutions.
01:39 *🚀 Real-Time Co-pilots in Professional Settings*
- Examples of real-time conversation co-pilots in professional contexts, such as aerospace engineering and job interviews.
- Consideration of the value and ethical implications of using AI in interviews and professional settings.
- Potential for improving interview processes and enhancing communication skills.
03:29 *🤝 Real-Time Co-pilots for Social Interactions*
- Exploration of the benefits of real-time conversation co-pilots in social interactions.
- Personal anecdotes about the challenges of social interactions and the potential support AI could provide.
- Discussion of the broader applications beyond professional contexts.
04:24 *📱 Building Real-Time Conversation Co-pilots: Web and Mobile Apps*
- Overview of building real-time conversation co-pilots, including web and mobile applications.
- Introduction to the technical components required for such applications.
- Step-by-step guide on using platforms like Replicate for deploying AI models.
04:53 *⚙️ Technical Challenges: Real-Time Transcript and Fast Inference*
- Discussion of the technical challenges involved in achieving real-time transcription and fast inference.
- Solutions for real-time transcription, including recurrent loops and optimizations for accuracy.
- Strategies for achieving fast inference with large language models, such as model selection and optimization techniques.
10:19 *🛠️ Implementing Real-Time Conversation Co-pilots: Demo and Iterations*
- Overview of the iterative process of building real-time conversation co-pilots.
- Demonstration of a web application prototype and its functionality.
- Integration of AWS services and Replicate for deploying and running AI models.
17:30 *🛠️ Backend service setup and frontend basic structure*
- Setting up the backend service involves defining routes and handling requests.
- The frontend structure includes defining HTML elements and basic CSS styling.
- Functionality such as recording audio and fetching suggestions is outlined.
23:18 *📱 Exploring Whisper Kit for mobile deployment*
- Whisper Kit, an open-source fine-tuned model, enables deploying speech-to-text models on mobile devices.
- Real-time transcription with minimal latency is demonstrated on devices like the iPhone.
- Whisper Kit's optimization allows for efficient use of resources on mobile hardware.
24:01 *🎁 Unboxing and setup of Apple Vision Pro*
- The unboxing and setup process of Apple's Vision Pro headset is showcased.
- Details about accessories, boot-up procedure, and initial impressions are provided.
- The demonstration highlights optimizations for improved accuracy in AI-generated transcripts.
25:14 *💻 Development using Whisper Kit for iOS apps*
- Utilizing Whisper Kit Swift package to develop iOS apps for real-time transcription.
- Setting up the project in Xcode and configuring model selection and streaming options.
- Overview of the app's structure and explanation of key functionalities for transcription and model loading.
31:03 *🚀 Integration and testing of conversation co-pilot features*
- Defining variables and functions for user input, API interaction, and response handling.
- Adding UI components for user prompts, transcription display, and interaction buttons.
- Demonstrating the setup process and testing the app on an iPhone for real-time transcription and suggestion generation.
Made with HARPA AI
this is crazy. Love your content. I will definitely use the end product if you publish one. Keep the great work up man!
Best AI channel hands down
I've been thinking about this idea for the last 8 months - so pumped to give it a shot, thanks a ton for sharing!
Are you trying to build a phone agent?
@@bassamel-ashkar4005 phone is cool but just generally a conversational agent to utilize when having any conversation anywhere. It seems to be an extremely useful concept in so many ways.
Holy, whisper kit is insane, running the model on mobile device directly gonna be the future;
Also thanks for sharing Replicate, didn't know it is free to use!
"If you’re new to Replicate, you can try us out for free, but eventually you’ll need to enter a credit card."
Hey Jason, interested in the iPhone project (tho android would be better).
How about adding an extra layer to this flow that translates the text to a different language and speaks it out loud?
We could then have real-time conversations with someone who isn't the same language as us.
Here's the use case. You hand over the left earbud to your Polish friend. You use the right earbud. Ideally both earbud would have a mic.
The application on the phone would:
listen to the speech in English => do a transcribe on the chunk => do a translate to Polish on the chunk => speak out the chunk in Polish.
Then when the Polish person is talking, it'd work the opposite way.
I need this for screening consultants. They are very smooth talkers, the minute you scratch their gloss they fall through, unless they actually know what they are talking about. Can't wait to see your app. Thanks for sharing!
There can be lots of good applications for this. Happy to explore and share. Keep the good work 👍
Every video you drop is a gem.
Deepgram STT and Groq's LPUs make this possible.
Awesome Job. You really need to go commercial with your ideas .
All your videos are amazing!
One gotcha, if AWS give you an "access denied" error you may need to update S3 policy to add a "/*" onto the back of the resource ARN - the wildcard permits access to all bucket objects
thanks bro it worked!
This is amazing Jason. Just subscribed to your channel and am very interested to see the iPhone app you build and other AI projects that you're working on.
I see a lot of amazing use cases for this like overcoming objections on sales calls, asking better questions on podcasts / interviews, etc.
I look forward to you rolling out you app Jason! It would be amazing to run that locally on my phone!
Awesome video again Jason good stuff
Yes definitely interested in the app, please notify when done!
tremendous work dude
This running on the new Groq arquitecture would be awesome
I tried to build a similar interview co-pilot before too, but the latency made it not usable;
Can't believe how far we went with those model performance past few month!
You do know that you can use OpenAI API to get sub 2-3 sec outputs right? The only thing not possible on GPT is a system prompt (I think, I have never needed to use OpenAI's API). But on my Mac with a capable 7b parameter like Mistral, or Mixtral model, the output is also within 5 secs (especially when loaded in GPU memory). I prefer local generation vs online since in local you can modify the system prompt and you can customize the output a lot more.
Yes would be highly interessted to see this app in the store!
Love your work. Great content. Yes, please let me know about the end product once you polish it please.
Love your videos, you do an amazing job of packing high quality information into a 30 minutes ish video. 🎉 thanks a lot!
thank you so much for your feedback!
Excellent video and very timely for my interests. Thank you very much.
Great production, thanks for sharing
Don’t have enough thumbs for that, great content.
Excellent content
so in real life f2f talk we have to hold mobile to have a convo with other?
Could you make a detailed video on how to make this?
I would be super interested. Thanks for your magic!
Definitely interested
quick question, If I were to run this offline what would the token speed look like?
Why use small model when you can use large model and Qroq?
why no search or/and RAG func call?
with thsi even the small fats model can become more knowledge
Impressive, I was thinking about to implement this in Android
Why can't I see this video from your channel on my laptop? Lol. Im on my phone now but is anyone able to see this video from the computer?
Kindly make a tutorial to fine tune an open source LLM model on many pdfs data. The fine tuned LLM must be able to answer the questions from the pdfs accurately.
Is it possible to have an AI copilot real time in game ,steamvr rec room ?
Very cool! 👍
Phenomenal
This is amazing, thank you for great content. I wonder, I tried this (i'm a beginner with a newly found passion for learning python). I don't have the same amount response on the web app like you have, I get a 8-10 second delay both with the transcript and suggestion. How can I fix this?
hell yeah im interested in the app
When your app will be published in istore? Or maybe it is already?
Ok now i want to build an agent that can listen to videos, copy and build the apps. There is way too much cool AI stuff to try and i have other hobbies and a life too xD
Thanks for sharing. Don’t sit on the translation potential for this as well ;)
Is there something like this for the android?
In the future, Neuralink will be wired directly to the brain activating the vocal chords so that the interviewer is mind blown on how you're answering the questions even though it appears you fell asleep during the interview.
Me: "I think I can do this. I'm going to give it a shot!"
Tries executing this following Jason's steps. Gets error message at first step when installing replicate into VSC. "command not found" .
Watches 3 videos to see if I can figure out why VSC is giving me this error. Still not working.
Feels defeated and quits :(
Can we do this but in Google meet or similar?
Yes! I need this for zoom, teams and meet
It's not going to work if the interviewer is able to track your eye movements through AI which can tell you are reading from a script.
Connect it then to a suitable glasses
Have you heard about eye tracking by nvidia
Imagine using this with Groq api inference speeds
exactly my thoughts.
For sure- two orders of magnitude faster inference is bringing us a whole new world and fast - by the end of march it will be evident
Can we use this model on ESP 32?
oh, another step to local speaking AI Assistant like Cortana or Jarvis
This is a great tutorial and illustration of how to use services, but your bucket policy on Amazon needs to be locked to just your user so nobody can mess with your bucket and hijack it. It's one of the most common ways people get their data leaked.
Hi i am very interested in this development
What's the point of passing the interview when there is a probation period in which real tasks need to be accomplished. And if those tasks are still doable by LLMs, it is just a matter of time until that position will be completely automated.
lol cheesing interview
seeing this and thinking google audio recorder transcript and gemma
interested!
👏👏
Please publish as an Android app too!
I don’t do the ‘TH-cam’ other than to watch. I liked. Subscribed. And now I am kindly asking you how can I give you money?
What you have here can help so many people. I eagerly await for this blessing to come to fruition.
hah thanks bro!
Horrorfying😮 Terminator has arrived i guess. Better to late than never.
this is what the deaf waiting for long time👍🏻
true!
what if interviewer ask to share the screen😅
If you get a teleprompter like the Elgato Prompter it acts as a 2nd monitor so the other person will never see it, plus you can be looking directly into the camera (while reading the info) at the same time so it looks super natural!
I will only do AI on my mobile if I can use it in Airplane mode.
this model load locally so yes it works in airplane mode!
Ok so what if i want to use this instead to argue with ppl and win every debate always forever. I gotta grind them TH-cam comment wins or something
I’ll find anyone who is serious about building a ai call software
just make sure you throw in a "right" every 6 to 9 words and your interviewer will never know your full of shit and not reading.
hahaha so funny
I'm high
30 sec. and i find so pathetic the use that some people give to the tools , faking an interview , if you suck at work pls , dont do an interview imgonna fired you after a month , why you are gonna lie for a month of pay , if you suck at one job maybe you can spend the improving your skills , but young people of this days really Suck so badly
Thanks!
thank you sir!