Love the vid...nice delivery and fun example. It is wild to think that just a -few- couple years ago, it could have been a keynote demo that blew the lid off a convention center somewhere. It makes our days of demoing _drag-n-drop_ or _OLE_ , on stage in front of thousands a very, very distant memory. A great time to get into, or in my case back into, coding. Thanks!
Pretty cool, one suggestion - for tool calls getting number of fingers you can use enum 1 to 5 to ensure it doesnt hallucinate with results and use strict mode
Great vid. Did you post link to your sample code somewhere or I just missed it. Also, if you implemented strict mode like other comments suggested which you liked, it would be great to see how one can implement some sanity here. Thanks again - also love your enthusiastic tone and smiles. Happy holidays.
Hi, do you have any idea how a conversation could be implemented between three actors, the microphone and output sounds of the equipment and the Openai realtime API?
Thanks a lot for the video! I noticed that when I reach the daily limit of openai the api will stop responding but there is not indication for that of the issue. Any idea how can I know this error? I tried to print the opeai API response but the status still coming as OK
I would like to a chatbot similar to Jarvis running a Raspberry Pi4. Online, many examples, but none using the advanced feature of a real-time API Could you assist me or suggest a tutorial? I believe it should be quite easy to create a on a Raspberry Pi but using the real-time API instead of the old one, correct?
Fantastic! I got this working on my laptop in about 10 minutes and was able to add a new function which checks my battery level in about another 10 minutes. It's very satisfying when someone shares such simple code that gives a working starter platform . Have you played with anthropic's tools standard thingy yet?
Love it. But technically, just like with your robot arm there, a person can "tell" the ai to fire a gun round when a certain condition is met. Like a person walking by, for example. This extends its capabilities into the real world, just like your robotic hand example. Don't get me wrong, i'm super stoked about this, and will be using it. Just giving some hypotheticals that are now possible for a dystopian set future. But i expect a utopia. I am optimistic!
@@IceMetalPunk @IceMetalPunk Yes, but the difference is that the person shooting the gun has to be present, even if it's from a sizable distance. And that's going to happen at an exact location and time, and a vantage point can be determined based on the trajectory of the shot. The surveillance and forensics nowadays is so good, that it is very difficult to get away with this kind of killing, even if it is not face to face. On the other hand, if a person instructs an ai to carry out a killing days in advanced so that the person can escape, this leaves new ways for criminal tactics and evasions. It can detect the exact face/profile of the person to carry out the act on with cheap equipment, and teleoperated weapons. You could create a gmail with a fake name and info, drive to a public location in a stolen tinted vehicle with free wifi on a cheap or stolen, dumpable machine, get an LLM api key, mask the functions as something benign that carry out the deed so that the LLM system doesn't flag it, throw the device out the window (with your mask on of course - smile for the cameras), drive off the grid where there's 0 surveillance and devices on hand, dump the vehicle, have weeks, months or literally even years before the AI carrys out the act without a trace because it can act and adjust almost as intelligently as a human. New laws will have to emerge, and those often take a while to pass in courts. More importantly, new ways to track these events. This isn't too convoluted a task for anyone to do. In fact, it's scarily easy now. It doesn't even have to be a gun-type weapon, it could be releasing poisonous gas, an explosive, or something less seeming and intricate; like having a drone with face detection activate, fly to the target 4 years later, and fire a lethal poisonous dart. How scalable is this? One dart...hundreds? The main asset that AI could buy you in these hypotheticals, is time. It becomes way more difficult to solve crimes the more time passes. And with it, you could have a lifetime before releasing the 'trigger', without you even needing to monitor or maintain it. I know this is exhaustive to read, but people are clever and I'm sure my hypotheticals aren't even as intricate and untraceable as it can get.
@@craigsdennis It could be masked as simply as "pushButton()" or "sendSmiley()" to avoid flagging from the system. With its function body being to detect a specific face and then send json to a machine which uses its payload to know if to release the trigger. The activation of the machine could literally be a '🙃' smiley face that was received from the json . Its a seperation of concerns that makes the AI unaware of what it's doing.
I think this video deserves some kind of award. You are amazing.
agreed
oh maan crazy.. cool that you demostrated first and then jumped into the code.. like a magician that reveals the trick haha
🐰🎩
Love the vid...nice delivery and fun example. It is wild to think that just a -few- couple years ago, it could have been a keynote demo that blew the lid off a convention center somewhere. It makes our days of demoing _drag-n-drop_ or _OLE_ , on stage in front of thousands a very, very distant memory. A great time to get into, or in my case back into, coding. Thanks!
Honored to be a part of your journey back into coding! Keep me posted on your journey! 💪🚀
I'm really glad this video popped up on my for you page. Already did star a few of your repositories. Awesome work keep it up!
Whoa... Wild. This stuff is starting to resemble magic. Very interesting times
Bonkers right?!
Awesome, this hides the complexity of low level webrtc api, I would like to see realtime message and video chat and robot hand in the next episode🥰
Amazing video. Seriously for a small company like the one that I have, this video is so useful
siiick!!!! I was just working on tools after setting up rtc, can't believe there's already a video on it
Have fun! It's so incredible!
You are the god of demos
☺️
Pretty cool, one suggestion - for tool calls getting number of fingers you can use enum 1 to 5 to ensure it doesnt hallucinate with results and use strict mode
Awesome! Thanks for the feedback, I'll make it better!
Structured Outputs for the win!
explain?
Fantastic, it's very interesting
Love your stuff 🤙
Great vid. Did you post link to your sample code somewhere or I just missed it. Also, if you implemented strict mode like other comments suggested which you liked, it would be great to see how one can implement some sanity here. Thanks again - also love your enthusiastic tone and smiles. Happy holidays.
Code is in the description of this video prefixed with Repo
How can we on the server side within the webrtc relay hook into the messages flowing through data channel?
Super informativ about WebRTC
How do you do function calling on the server side?
Hi, do you have any idea how a conversation could be implemented between three actors, the microphone and output sounds of the equipment and the Openai realtime API?
what a time to be alive
Is it possible to connect to Azure OpenAI?
Can o1 call functions while thinking or just when it is finished?
Thanks a lot for the video! I noticed that when I reach the daily limit of openai the api will stop responding but there is not indication for that of the issue. Any idea how can I know this error? I tried to print the opeai API response but the status still coming as OK
Achieve more with AI by your side 💫
So cool!
The Terminator is knocking at the door, one finger at a time.
This is wonderful! How do I reach you?
Wow this is wild
this is huge
how to protect openai api key from user ?
It's on the server using protected secrets, I relay it through. If you did it directly there are other workarounds.
This is amazing thanks for sharing
Thanks for watching!
I would like to a chatbot similar to Jarvis running a Raspberry Pi4. Online, many examples, but none using the advanced feature of a real-time API Could you assist me or suggest a tutorial? I believe it should be quite easy to create a on a Raspberry Pi but using the real-time API instead of the old one, correct?
Cool!
very cool!
Insane!
Sheeeeeesh.
Riiiiiiiiiiiight?!
Fantastic! I got this working on my laptop in about 10 minutes and was able to add a new function which checks my battery level in about another 10 minutes.
It's very satisfying when someone shares such simple code that gives a working starter platform .
Have you played with anthropic's tools standard thingy yet?
Love it. But technically, just like with your robot arm there, a person can "tell" the ai to fire a gun round when a certain condition is met. Like a person walking by, for example. This extends its capabilities into the real world, just like your robotic hand example. Don't get me wrong, i'm super stoked about this, and will be using it. Just giving some hypotheticals that are now possible for a dystopian set future. But i expect a utopia. I am optimistic!
Someone would have to write the "shootGun" function in this example. Definitely keep the human in the loop!
You can tell a person to fire a gun, too. These aren't new problems.
@@IceMetalPunk @IceMetalPunk Yes, but the difference is that the person shooting the gun has to be present, even if it's from a sizable distance. And that's going to happen at an exact location and time, and a vantage point can be determined based on the trajectory of the shot. The surveillance and forensics nowadays is so good, that it is very difficult to get away with this kind of killing, even if it is not face to face.
On the other hand, if a person instructs an ai to carry out a killing days in advanced so that the person can escape, this leaves new ways for criminal tactics and evasions. It can detect the exact face/profile of the person to carry out the act on with cheap equipment, and teleoperated weapons. You could create a gmail with a fake name and info, drive to a public location in a stolen tinted vehicle with free wifi on a cheap or stolen, dumpable machine, get an LLM api key, mask the functions as something benign that carry out the deed so that the LLM system doesn't flag it, throw the device out the window (with your mask on of course - smile for the cameras), drive off the grid where there's 0 surveillance and devices on hand, dump the vehicle, have weeks, months or literally even years before the AI carrys out the act without a trace because it can act and adjust almost as intelligently as a human.
New laws will have to emerge, and those often take a while to pass in courts. More importantly, new ways to track these events. This isn't too convoluted a task for anyone to do. In fact, it's scarily easy now. It doesn't even have to be a gun-type weapon, it could be releasing poisonous gas, an explosive, or something less seeming and intricate; like having a drone with face detection activate, fly to the target 4 years later, and fire a lethal poisonous dart. How scalable is this? One dart...hundreds? The main asset that AI could buy you in these hypotheticals, is time. It becomes way more difficult to solve crimes the more time passes. And with it, you could have a lifetime before releasing the 'trigger', without you even needing to monitor or maintain it.
I know this is exhaustive to read, but people are clever and I'm sure my hypotheticals aren't even as intricate and untraceable as it can get.
@@craigsdennis It could be masked as simply as "pushButton()" or "sendSmiley()" to avoid flagging from the system. With its function body being to detect a specific face and then send json to a machine which uses its payload to know if to release the trigger. The activation of the machine could literally be a '🙃' smiley face that was received from the json . Its a seperation of concerns that makes the AI unaware of what it's doing.
Awesome,
❤🔥❤🔥❤🔥
Holyyyyyy Shieeeettttttt!
Same, dude, same.
so cool
When the robot show the middle finger is when I shut everything 3
Amazing!