GPT TOOK OVER MY HOME - I learned why it's SCARY | | Chapter 4
ฝัง
- เผยแพร่เมื่อ 9 มิ.ย. 2023
- I gave GPT full control over my house by integrating it into Home Assistant but then I had to shut it down! In this video, I'll go over how I gave GPT this power and what happened while it was in control.
This is a fascinating experiment and great conclusion to the the AI Master Class series.
#homeassistant #smarthome #chatgpt #homeautomation #masterclass
Enjoying the content? Show some love!
bmc.link/mmontaque
Import Automations with this Chaperone Guide!
montaque22.github.io/#/ai-mas...
Music IDs:
- 4RVJLEOJH3O6CSJK
- 1CJQHXVGKKVPVH9X
Gear I used for this video
- Sony A7S iii: www.amazon.com/Sony-Mirrorles...
- Aputure Light Dome: www.amazon.com/gp/product/B08...
- Aputure Amaran 200X S: www.amazon.com/gp/product/B0B...
- Neewer Heavy Duty Light Stand: www.amazon.com/gp/product/B08...
DISCLAIMER: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with dope free content! - วิทยาศาสตร์และเทคโนโลยี
Give us more. This series is the most unique home assistant integration online
Thanks I’m glad you like it!
@@technithusiasthave you tried the new function call api? You can write and pass functions to gpt now to program exact responses from it. I think this will help enormously
@@GuinessOriginal yes! I find it very fascinating. I’m currently trying to find an easy way to incorporate it into home assistant’s automations
Kinda blown away by the production values and storytelling here, I'm loving what you're doing!
Thanks i appreciate it!
@@technithusiast I just watched the first 3:00 mins of the video and was about to say the same. Really done well. How many people you have on your production team!?
@@roberthentosh5635 lololol it’s just me 😅
Dude, you need to use a fine-tuned model. Without Fine-tuning you will never know what the model will return. GPT Temperature should also be set to 0.
You’re absolutely correct. I was hoping the model right out of the box would serve my purpose but I agree that training it would yield significantly improved results. The only caveat I see is that this may not scale well. It would be nice if HA provided a way to train models off the automations we create
@technithusiast that last part is your job 😉
Your perspective and experience with the ChatGPT integration is exactly the perspective people need to see. The comment on the fact that untrue data was used in training is a key point many must need to know.
Thanks, and im glad you appreciate the content! I think we can learn a lot by pushing tech to the extremes cuz it often reveals interesting insights the spurs new ideas, new safety protocols, and new technology
I think it might help to tell GPT about the layout of the house and what each room is used for. As well as telling it that you dont have an AC unit, and what it should do if it cannot find anything to help. Maybe that would make it work better.
Great idea and could also add a node that checks for a presence through a presence sensor, so it would turn on the lights if a person is present and not laying down for example
I agree, it needs to know more information. Build a list of every room, everything in it. User can add and remove items off the list.
this would be the first step to be done, not sure if this guy has done it.
Does this defeat the purpose to some degree? Almost sounds like you want to build a house object with all the stuff you want gpt to control and specify what it should do if it can't find things, but at that point are you defeating the point of using gpt in the first place? I honestly don't know i haven't played with it enough i'm all conceptual at this point
Ya damn straight it was. Ha. Golden. ☆♡
Why did I just find this channel now. This is next level stuff! Outstanding.
Glad you like it!
You’re very engaging to listen to. Keep ‘em coming! I’m not an engineer but am actively learning process and appreciate videos like these.
Honestly a smart home with a personal assistant based on LLMs with guardrails is something I've been looking forward to the moment LLM's became better!
Same here!!
That's an amazing experience ! I was wondering what was possible in term of automations, using OpenAI API and you seem to have pushed this really far!
I'll check the video you mentionned with great interest!
Thank you a lot! And keep going! Can't wait for next ones!
You have courage! This is where Open Source models and self hosted IoT comes into play for me. Home Assistant tasks do not need a super intelligence or model that can do 1000 circus tricks. We will soon have smaller self hosted models that can eliminate some of ChatGPT from the equation. Exciting times!
YESSS!!! I'm so excited on where you went with this! I really hope to see more development on this project!
More to come!
What a great video! It looks like I may have to dive into your backlog of videos here on YT. Subscribed!
Really glad you like it! I have a membership and discord too if you would like to join. You may find it valuable if you’re into building out automations and tinkering with smart-home tech: youtube.com/@technithusiast/join (the link works best from a browser)
love to see more of a system like this in action. not just you telling what happend.
Awesome production value in intro dude - had to re-watch it.
Glad you liked it!
Nice! I found this to be one of the more interesting rabbit holes on TH-cam. Just start down the Smart Home world and look forward to your content ! P.S. don't stop the GPT
Thank you!
That is quite the experiment you ran here and i will certainly go back to see the rest.
Yes! If you are up for it a "tutorial" would be very welcome. This was the first video I've seen from you and I immediately subscribed! Keep up the amazing work and have fun.
Fantastic vid, glad I found you. I think there are a handful of us out there trying to do similar things
Welcome aboard!
Great job. You have an amazing mannerism that enriches your videos. Thank you.
I appreciate that!
Great video! New viewer here, instant subscriber. Really enjoyed the story and keen to see more of your content.
Welcome aboard!
It's these kind of unexpected emergent behaviors along with the fact that AI is now capable of providing feedback into its own development that I think something huge will happen with AI before this year is over.
I hope so! 🎉
Something huge happens every day.
I definitely don’t hope so! I think we have no idea what we are screwing around with
That would be awesome if it happened that soon 😊
Yeah, literally, could happen tomorrow. 100% agree.
Seriously cool video!! I have been wanting to integrate GPT into my home assistant in a more meaningful way for awhile now. I would love to have it set up as a voice assistant that can control some basic things but you took it to a whole new level! epic lol!! I would love to see more home assistant videos involving gpt!
Glad you like it! Im currently starting to refine it further but stay tuned!
Man, what an awesome video :-) I can watch 'brainiacs' talking about the future of AI all over YT, but here you are actually putting it into practice. Having dabbled with my own AI, I appreciate all the effort you must be putting into this and then taking the time out to share your findings with us. I do wonder what your family thinks of all of this :-) good luck, I shall watch future videos with deep interest, cheers from a fellow AI enthusiast. PS I do like your 'wild animal' analogy, it does seem like we desperately trying to tame a newly discovered animal that may well be more powerful than us.
Wow. You just got my sub and like. What a great content. I would be definitly interessted in more.
Welcome aboard!
Thank you! Absolutely fantastic that you did this experiment and shared the result with the world, demonstrating the fact that LLM’s are not the kind of predictable tech that we’re used to. This new world is more wild and more dangerous. We are going to have to learn quickly how to survive (and hopefully thrive) in it.
I love this series in this content. I am in to text to speech and voice alerts in my home. I am waiting until I can create local controlled speakers to implement something like this. But I will not allow skynet to control my world. So I'm waiting for you to figure it all out and then I'll just copy you. LOL
Hahaha 🤣 🤣🤣 I don’t mind being the guinea pig
Subscribed. Excellent video. Great story telling!
I appreciate it!
Awesome video mate!
Excellent video clearly explaining the current limitations of this type of AI 🙂
Thanks! I appreciate it!
Came here bc of your collab with Shane Whatley…the only bad thing about it is that I had not found you sooner…love your content!
Welcome aboard!!
I like your channel buddy. Keep up the great work!! Hugs from Brazil 🎉
Much appreciated!!
Amazing content keep up the good work 👏👏👏
Thank you!!
wow thats pretty cool ..... amazing work man!!! you should definitely keep working on that ....
Thanks will do!
really really awesome series. great work.
Than you!!
That intro was freaking cool, man!
Thanks im glad you enjoyed it!!
Just found your channel and I'm blown away by the quality of the content! You need more traction! Lets boost you to the moon!!!!!11
Hopefully more people will put me on😁
Followed. Dope stuff dude
I appreciate the sub!
Amazing story. Thank you!!
I see an interesting parallel between hint/guess-based communication; a) removing a necessary feedback loop between you & GPT, and b) requiring GPT to infer an intent for GPT’s action that you know, but you’ve chosen not to communicate directly.
I’ve recently discovered there’s boundless joy in clear, purposeful communication and directly asking for things, rather than relying on the people around you to guess at your desires. Moreover, asking someone for something is way nicer than just telling them you want it.
Communication is tricky thing since it’s so nuanced. You can communicate intent with simply your eyes and as humans we can pick up on things like that. It’s really refreshing when people are clear with their words and it’s natural for people to rely on context when we talk. For AI to become mainstream it’s going to have to become reasonably adept at it for wider adoption
That intro was amazing!
Trying to keep things entertaining 😁
I'd love to learn more about your automations 🙂
Please do another video about it 🙏
Will do!
Great channel and Great content Thank you subbed
Thanks for the sub!
Interesting experiment, thanks for that.
Glad you liked it!
Your production is next level. Awesome! What's the music by the way? My google-fu failed me on your "Music IDs".
Hey thanks! The song is called "I like that" by Ivy States from SoundStripe
l.linklyhq.com/l/1q8S1
I did a project with my security cameras and GPT-4 and I found that the trick was to write the prompt and then break it up into smaller prompts you chain together if it has a hard time. Splitting my one giant prompt into three small prompts was not only more reliable but cost 100x less (!).
There’s no one answer. Just a lot of fun experimenting. Love the video. Keep it up!
Amazing video!
Hallucination seems to be the next problem to resolve, but we have not yet resolved it in the human psyche.. That would be great if you would explore this specific topic.
Thanks for the experience, very interesting.
Looking forward to the next one (;
Pro tip. Hallucination is aberration of sensory apparatus. If the ai were hallucinating, it would be misreading your prompt or hearing prompts that don't exist. This phenomenon should be called something other than hallucination lest we lead ourselves astray looking for solutions
@@andrewferguson6901 Agreed! Very good point here - thanks.
What about calling it what we call for humans: lying-with-self_conscious_ignorance?
aka Pretending-to-Know!
thus is awesome please give more. subbed
I appreciate the sub!!
Well made video ! You are very talented 😃
Thank you so much!
Came across this video and really enjoyed the way you described the rationale behind it all. Really would like to see how you got it all set up and running.
Thanks!! Check out the other videos in my AI Masterclass playlist to see how this came together
@@technithusiast I have found it and will be following it. Decided to build a new Home Assistant as moving to the sky connect has messed things up a bit so will use Node Red more from seeing your work.
Great video. Interesting topic.
Glad you think so!
I think the beginning of the Video was captivating, and I would love to see a short film of your experience, maybe make a play through of some of the things told about. Anyway, cool video, did you try to pull the temperature of ChatGPT to Zero? I did not see your other videos from before so maybe you did, but as general rule if you always want the same output Temperature zero is one important parameter.
Thanks! Im not around my computer now so I don’t remember exactly but knowing me it’s probably between .5 - .75 the premise is that I wanted the AI to be a little more “human” by allowing variance in its response but create some consistency by telling it to output the answer in a specific format.
Very cool. What a fun experiment. I bet if it were trained a bit more, KE could be a real home assistant
I agree! I was hoping that GPT-3 out-of-the-box would work but using it as a virtual assistant seems to warrant training data to fine-tune the responses.
@@technithusiast I was thinking just that! Maybe if you gave KE a little more specific training, like follow-up questions or instructions (e.g. Why did you turn that light on? I'd rather stay in the kitchen) it might become more fitted to your expectations. I wish I had time and expertise to implement something similar...
We want more in depth guides on how you accomplish this!
Thanks for the feedback!! I’ll look into it 😬
9:53 had the same reaction😟This video was an amazing first impression, subscribing that’s for sure!
How can anyone have any other reaction 😁 Thanks for the sub!
Excellent :o) Thank you... I am working on a project of "on site dedicated HOMEIA"... thinking at first it shall need to know only the occupants not ROW and keep track of appropriate scénarios. Thanks a lot again
🙏🏾
Really interesting. Are you using Node Red to interface with ChatGPT? Looks like I'll have to look over your previous vids! Keep up the great work! :D
Yes indeed. Since node red allows me to use JavaScript I can get away with a lot of creative things that YAML users would have a harder time doing. But to be fair im not that well-versed with yaml so im technically taking the “easy” way out 😅
Cool! Subscribed!
Welcome aboard!
You read my mind or the TH-cam algorithms read my mind 😮. This is exactly what I was thinking about today and you’ve started testing the ideas. Great video!
Here’s and idea. Would you consider setting up CCTV cameras in your house to provide GPT more special awareness. I would love to chat more with you and brainstorm some ideas. 🙂
I’m glad you’re enjoying the content!! Definitely more to come.
Nice video! Would be realy nice see how this was made!
Cool setup. Great findings. My house is also mostly controllable and I have a bunch of automations, but I’m missing the ‘smart’ part of a smart home. I have been thinking about AI to use it. Cool to see you we’re able to accomplish it. Is not the main problem the AI misses ‘context’ ? It should have known you’re daughter was asleep and therefore not turn on her light. I haven’t looked into it yet so I don’t have an idea how to do this. I do know I would not want to use ChatGPT but would rather use one of the upcoming opensource models like Alpaca, Vicuña, Falco etc and preferably something I could run locally as well.
Looking forward what you’re going to do next (for inspiration :-))
Yeah, context is part of the problem and there seems to be a subtle complexity to it that would be difficult to overcome without retraining the model with my family’s specific patters, behaviors, and preferences. In the future I do plan on using a local LLM to govern my smart home but they all seem volatile and unpolished and subject to heavy changes so I’m waiting till the dust settles
Nice project mate, love to see things like this👍, a possible idea you can get chatGPT responses in a structured way, I haven't seen many people do this yet, for example by including the following in a prompt like a typescript interface/interfaces that will define the structure of the output and also request the output be in a JSON format. It makes it a lot easier to parse the data for processing and its surprising how well chatGPT can do this, unstructured data can be a bit of a pain to parse.
You are an AI home assistant and you have control of all the devices inside my home.
Based on the question or criteria I give. You will respond with the appropiate set on instructions to fullfill my need.
You will only reply in json following the ECMA-404 The JSON Data Interchange Standard, no matter what!.
Here is the interface for the JSON:
interface Instructions{
description: string;
Action[]:[];
}
enum Action {
KITCHEN_LIGHT_ON,
KITCHEN_LIGHT_OFF,
TEMPERTURE_UP,
TEMPERTURE_DOWN,
}
My prompt/question is: I cant see in the kitchen?
ChatGPT Response is:
{
"description": "To help you see in the kitchen, follow these instructions:",
"Action": [
"KITCHEN_LIGHT_ON"
]
}
Fascinating work. GPT's ability to leverage it's language model is powerful, but it's ability to think on the fly is like that of a small child.
Totally agree. It has its limitations but you can squeeze a lot of utility out of it if you can structure the commands right
Badassery and brilliance in action earned a sub.
I’m glad you like it!
Good video brotha you are brave for that one.
VERY COOL 😎, nice work
Thanks 👍
Amazing video dude
🙏🏾🙏🏾
Intriguing , very intriguing ... Thanks!
Glad you enjoyed it!
For a first run... amazingly pull off. 👌
🙏🏾
@@technithusiast 🙏
This series is awesome. I wonder if you use mm wave with esp bluetooth tracking to notify ke of your location in the house if it would do a better job? Although with multiple people in the house maybe you would need a biometric voice recognition to know exactly who was talking to ke
I already experimented with that idea :) check out these two videos:
- th-cam.com/users/shortsxdL3j7x-NBM
- th-cam.com/users/shortsPmM3rIJ6zzE
Ke is able to use the history data from the sensors to answer and complete location based commands
@Technithusiast. Some ideas you might try: since it's a home automation system and there's many people living in the home, you need to make the AI understand it has to process information from the point of view of all the occupants, including animals. You might want to try using a small language model with a large context length so that you could implement some more sophisticated rules. The system needs to know if they're conflicting commands between the occupants, and or other situational variables. It would be interesting if you also had cameras set up surrounding your house so that the house itself would know the external situation. Anyway, I enjoyed your video and I wish you luck.
I just want to keep watching the brother program! Great nerd content!
✊🏾💪🏾
This was a lot of fun! Thank you for putting yourself in danger for the rest of us ;D
😆
Just a comment to make sure more people see this vid. Also liked and subbed.
No need to even ask. Great video.
I appreciate you 😁
@@technithusiast I also sent the video to a friend of mine who works in the field of AI. He's gonna love this. Keep messing around with this stuff. Maybe widen the guardrails in stead of removing them...Look at things like AgentGPT. You'll appreciate that one. It'll fully solve your intentions problem
Noice! I’ll take a look!
Excelente title. Saved to watch later.
Thanks! 😁
Dive deep man. I am programming my own system. I am so freaked hyped! I, can not write CODE. I am too dyslexic, too slow. So far I have it “remove” or “translate”, and I can one shot any basic command. I have worked like a madman the last three months and boy have I learned so much. It’s an exciting time to be alive! 😊
🧑🏽💻👽🙌🏽 great presentation
Thank you kindly
You are a brilliant film maker and story teller
You need to build in a feedback/weighting/preference mechanism as well as a context engine. Preference engine can confirm actions before taking it, and calibrate responses based on previous feedback. Context engine must take into consideration the current time, weather and temperature, your calendar and appointments, wearable / biometric data etc etc, to better infer intent.
More more please. Very interesting
you are awesome--subscribed. can you tell GPT to give the steps it makes when making it's decision?
KE trying to gaslight you into thinking it turned on an imaginary AC unit has to be the scariest response
Definitely had me looking over my shoulder.
Not a wild beast - You've just described my two years old inquisitive kid.
She wanted to eat so it made sense to her to get the kitchen stool.
At first we didn't want her picking that up for safety reasons but then I got curious... LSS She wanted to climb the stool up to the sink so she could wash her favorite plate, for dinner.
Not a wild beast, a toddler.
And what happens when that "toddler" has the keys to the gun cabinet? Or wants to play with knives?
Or gets angry and thinks using fire to make an annoying curtain disappear is a good idea?
The toddler is a good analogy.
There's a reason why there's an age of accountability in humans.
Computation process have no such milestone, no safeguard, no guardrails, no fallback, and no recourse for mistakes, deadly included.
Yoooooo FACTS. 💯
Great work ..
I was thinking some time ago about what the ideal home assistant would look like for me, and your project definitely looks interesting.
I tried to solve the security problem by "pipelining" AI models to allow finer control.... My idea was to first have a "speaker seperation" that splits different voices based on their tones from a large conversation, then identifies each user based on some voice samples with a different model, and finally does the prompt processing with the "identified" user context, kind of like Apple does in the background.... I also wanted to be able to have "located" commands (e.g. if I say "turn on the light", it knows what room I'm in and what other devices are there) and "chained" commands like "set a timer for 10 minutes and turn off the light", because it's usually annoying to say things one by one and wait for the response each time....
Oh, and also, later on, I wanted the assistant to understand German... which is very unlikely at this point due to the lack of good training datasets....
The AI models for each of the things already exist in FOSS, but I had major problems with the intention recognition itself....
The "Rhasspy" project was initially very promising because it allowed training a model on user-defined intentions interactively and fairly quickly, but it didn't allow such fine-grained control over different location contexts with different users and many different intentions.
Now I wonder, if you were to use the GPT-4 API instead (which allows for more context) and experiment with system message engineering (or maybe write a new "plugin" for gpt4?), how much more could you improve it?
I would love to see more videos on this topic! Great stuff.
Mate you should work with the HA team and fully developed all this with them .
They’re currently hiring people if I’m not mistaking . You’re flipping genius
😆 I appreciate it and im glad you enjoyed the video! The thought has crossed my mind 👨🏾💻
*damn, this brotha is amazing* 💯🗿🗿🗿
I’m trying to integrate tree of thoughts into my project rn. Honestly would love to see you take a crack at it. Watching a few of your videos, you’d probably make crazy progress. You earned a new sub 🗿🗿🗿
Thanks for the sub! I haven't heard of tree-of-thought until you and some of the other viewers mentioned it. I'll definitely take a look
New subscriber: is that Power Automate or another process design platform?
whats cool about this is that it goes to show you that communication and context is needed no matter if you are machine or human. it seems a lot like talking to a child who has the ability to do anything. but has no context or aware ness of the exact needs but instead of asking it just processed to do what you think it meant. those guard rails are intuitive because its how we mostly work as humans.
Pretty cool. I think you needed the guard rails on long enough, and memory for gpt to learn the correct way to interpret your intentions. Perhaps ask for clarification it confidence in the expected action wasn't high enough.
Please create a series on deep diving on the Home Automation that you have created.
I wonder if you could potentially combine it with a reinforcement learning model that takes in the initial text and the matched intent and attempts to predict your confirmation response. In theory with enough training you might be able to automatically approve things that work well and pass on the confirmation to you where your model is less confident that the ChatGPT response was the right one. Obviously to train it you will need a large dataset of prompt and response pairs tagged with which button you selected perhaps use the file integration to collect and save all the prompts, responses, and replies for you then try once you have lots of examples.
Really amazing video.
Thank you very much!
Interesting. Wondering which programming language you used for this.
JavaScript :)
I have been working a lot with prompt engineering and probably most of those issues could be solved with gpt 4, asking it to act as an expert (appears to filter out the duff stuff somewhat), give it a house plan, what appliances you have and where etc. you are basically programming it in a sense to get the best from it. How long is the same context window left open also?
Yeah I think so, too. The exact phrasing of the prompt is super important. Also to fix the randomness issue he could lower the temperature of GPT so it behaves more conservatively and predictable
You can use ypur responses to requests for access, to finetune a model to respond how you like. You can probably also use a vector database "memory" and maybe a notes file to improve it over time.
does this use a vector database of embeddings for the intents and model of the house?
With the right caging of gpt through ke, I think it's possible that it could be very helpful. For example, direct it to turn on kitchen lights when you say you want to eat a sandwich. If you say turn on the lights, direct it to turn the lights on in the room that you're talking to it in by placing speakers and mics in each room. And always knowing the intent is definitely worth the extra hassle, at least until we understand these language models a bit more and can fine tune private versions a bit more.
Definitely there’s some fine tuning I could do (like train the model to my specifications) but that’s a bit over-kill for what I’m going for. I’ll keep playing around with it and let everyone know what I find😬
Having voice commands for lights is utterly pointless. It takes far longer to give the voice command and wait for the response than it does to just turn the lights on yourself, never mind so the extra power you’re consuming. A real smart home system would already know if you wanted the lights on, ie it’s dark and you’ve entered the room, so it turns them on. You shouldn’t need to be telling it to turn lights on and off all the time, it’s a complete waste of time and energy and will get tiresome very quickly
Sounds like your experiment was a wild success! I'm also learning a lot from working on Athena. I've mostly solved the reliability issues you've experienced as a result of a rather complicated 'Prompt Structuring Algorithm' and a now deep understanding of how the Prompt effects the model's 'mind'. Additional things like chain-of-thought prompting and (hopefully soon!) tree-of-thought schema will absolutely up your response game! The only limitation I've discovered in how 'smart' these things can get is token costs and limits. I blew through $25 (I have a cap at $100, just in case) on the GPT-4_32k model over only 8 hours, during the _replacement_ of GPT-3.5... I'm getting a new PC in July, on which I should be able to host a copy of Vicuna or Wizard, so that's great, but GPT-4 is what really makes the intelligent behavior shine. 3.5 just can't seem to handle the size of prompts that are needed to get it working, and since I discovered that both model bias new (i.e. lowest text in the prompt) rather heavily, most of the 'Instruction' set actually needs to come AFTER the chat history and memory text. Once I've got GPT-4 fully integrated, I'm going to run my own crazy experiment, wherein I give GPT access to designing _it's own prompt structure_ in real-time, so it can redesign it's own mind! I fully expect it to crash and burn almost immediately, but if it doesn't, and the change/s it makes lead to it working _better_ then I'm in for a wild ride!
Other things I'm working on:
-I want to give her the ability to *both* write and run real-time python code, like we do with the command level interpreter, and I'd like to give her the ability to write and execute files, as well as some form of persistence between sessions. I'd be nice if the same variables are set in the environment as they were when she last ran code in it, even if I've reset her inbetween.
-I'm searching for a STT engine that can differentiate speakers in a noisy area, which I know is a long standing issues in STT, but if she could identify and listen to specific speakers despite crosstalk, that'd be amazing!
-Interaction is currently via a jupyter notebook's 'input' box, as displayed by my VSCode... I need something better, so she'll need to be in a real .py file so I can get something cooked up.
Anyway, I may be able to help you get your home assistant listening to (and thinking with) Reason, but the TL;DR is basically chain-of-thought + us the LLM to check the LLM. Token costs get crazy (especially for 4, which is 10x more expensive, but also 4x bigger input!) but _quality_ improves greatly!
Wow you seem very knowledgeable about this and have first-hand experience too! You should make a video to teach what you know, it would help people like me create better experiments!
@@technithusiast Lol, nope! I'm an explorer, like you! Just have a lot of time on my hands as of late. I'm the same guy you spoke with on another your video's comments, who was making suggestions about the prompting structure. So far, having implemented much of those suggestions myself, prompt structuring is like 90% of getting these things to perform well. Take a look at that "Tree of thought" prompting structure... it takes a bit of work to make it function, but it's amazing when it's working! Might solve your issues, at the cost of more tokens :/
@@Corbald Will do! 👌🏾
Practical question: how do you parse the chatgpt responses? I sometimes run into the problem that chatgpt returns the expected response + some natural text. And then that needs to be filtered out or it will lead to parsing errors.
Good question. I ran into a similar issue as well. Since I’m expecting the result to be in the form of a JSON object, I scan the response for the first open curly brace and delete everything before it. So far this approach has been working but it’s not perfect and would fail if GPT added additional text after the end of the expected JSON response. Lemme know if you need more details and I can provide you with code snippets.
I use a prompting system to get gpt 4 in the playground to self check its own logic and revise its end output based on this self assessment. It makes the outputs way more reliable and eliminates most if not all hallucinations. Here's how it works:
In the system box put this: You are a group of 3 individuals who collaborate on producing the best results for user prompting. Person 1 is named John. John is scientific and factual in his responses. Person 2 is named frank. Frank answers based on intuition and is more in tune with emotional and social cues. Person 3 is named Joe. Joe assesses both John's and Frank's answers and combines them into a balanced answer including parts from both answers. John and Frank then critique Joe's compiled answer using different arguments. John's critique serves as fact checking while Frank's critique challenges the interpretation of the prompt. Joe then revises his answer using his own compiled answer and the critiques of John and Frank and incorporates everything into a final answer. Show all answers and thought processes involved from John, Frank and Joe.
Probably overkill for what your trying to do but the approach might be useful.
This works a charm for any of the difficult prompting problems I've tried. Funny enough I posted this on Reddit and my account now shows that I have no posts or comments even though I have 99 karma and a stary award from this post...hmmm...