A great companion to your earlier ai masterclass videos. Also a great example of why I like Node-RED: the ability to exchange automations or fragments. HA Blueprints eat your heart out!
This is exciting, thank you! I'd love to see a more in depth video on using this to automate the 'Good Morning' message you touched on in a previous video. This would push the WAF over the top!
I’m really glad I stumbled across your channel. You’ve got some great content man. 👍🏻 I also love that you’re a Node-Red guy. I can’t stand setting up automations and such in HA.
Thanks, I really appreciate it! After seeing your video on LocalAI, I'm looking into adding new node that allows users to use any custom LLM as long as it is compatible with GPT's API.
OMG I really love your Project! Is there a way to specify past messages like assistant or user messages to get a continous conversation? :) Regards from Switzerland
Hey I’m glad you like the project!! Currently the nodes are pretty open ended so you could do it manually by saving the response and including it in subsequent calls. I have some updates I’m working on to easily facilitate conversations and using a local LLM instead of needing OpenAI
Hey, thanks again for this great video and all the effort you put into this. Will definitely take a closer look. I found extended openai conversation a couple of days ago and was just about to comment until you also mentioned it. It's like magic. I asked it yesterday if it could tell me what time I made a fire in the fireplace due to a rapid temperature increase in the living room and it nailed it. GPT4 though, I figured results are so much better with the latest model. Right now I'm installing localai for a test because I hit my limit in gpt4 rather fast with extended openai integration. It's just so much fun to use. Now I'm thinking about getting new hardware to run the assist pipeline and LLM locally. 20 buck a month for gpt 4 is quite a lot if you don't really need it. Keep up the great work! Greetings from Germany
I always forget how global TH-cam is and that people from all over the world can find my content. I like your enthusiasm and im glad you're enjoying my content.
Fantastic nodes! I'm genuinely impressed by them. Your content is outstanding. Could you delve deeper into how to tap into home assistant with a template node to grab information from home assistant? I feel as though I'm not completely understand the full range of possibilities you're seeing here.
I glad you’re enjoying the content! I have a few videos hat teach how to use some of the nodes ( like this video: th-cam.com/video/oWP8es4g4D0/w-d-xo.html ) and the past 2-3 recent videos give examples of different automations ideas that use these nodes.
Terrific Video and Nodes! Got it going but have a question. I'm planning on using this outside HA for a museum educational application and I would like to limit the chat to a specific area as: const query = datastore .createQuery('Animal') .filter('type', '=', 'marine mammal') .filter('familyFriendliness', '=', true); Thanks this video helped so much.
Im glad this helped and very interesting use case you have!! Yeah, I wanted something that was pretty flexible where i don't have to send it a ton of data. You can definitely scope down the information however you like before sending the data.
This is so cool! I am so pumped for this! I tried to set it up but I run into problems with intent. It says "The config is missing Intent Name or missing payload.functionName" but shouldn't that be added automagically when you register the intent?
Love the enthusiasm! Another viewer ran into a similar issue and was able to solve it by attaching a debug node and reading the data. If you're still unable to get it working let me know and i can sync up with you offline for additional help.
@@technithusiast I have been experimenting with setting up voice assistant at home and I have gotten it to work with the home assistant “assist” but it sucks so bad. So that’s why I’m happy to see movement on this! I would love to help you on this. I’m a programmer with 9 years of experience working as a consultant in various fields. And I’m a huge home assistant nerd. I use all node red for my automations. At the moment I have loads of time since I’m between assignments.
@@technithusiast I looked at the debug and I’m not sure what I should be looking for. But I did see one strange thing. It says there is a registered intent that is called “lightOn” but there isn’t. There was one before but I renamed it. Edit: looked at this some more. And I’m almost positive there is a bug when deleting “register intent” nodes that causes them to not be removed from the tools visible to OpenAI
It is definitely possible as I’ve had issues around that use case. To help debug, can you send me two things: First a screen recording of how you setup the automation and the error results of what happened. Second, export your entire flow as a json. Send both the json and screen recording to technithusiast+help@gmail.com
Great video tutorial as always. I know it’s slow but have you thought of making a video that uses local AI to keep information safe and if your like me and internet is spotty.
First of all thank you for your inspiring videos. You deserve much more subscribers. I like your nodes for node-red very much. But what i ask me all the time is: How can i extract the contenct from the homeassistant assist or the stt from my sattelite to use this content as a comand for your nodes? Please help me with that. Thanks
Thank you and I’m glad you’re enjoying the content! I’ve seen some code from other clever folks that’s been able to get get and return text to home assistant assist. I’ve been only able to get text from it (I’m still trying to figure out how to send text back asynchronously) You can use the “sentence” node in node red and use {wildcard} in the sentences box. This will trigger this node with the text from assist. However this is a one way street and I haven’t figured out how to respond after I processed the text
Hey this is probably something I would be better suited to help as a channel member as you would have access to the discord server so we can converse faster and share code easier. However based on what you’re mentioning Assist doesn’t talk directly to node-red. If you use the conversation api in node-red it will definitely pick it up but the Assist dialog in HA requires a custom plug-in that would act as a conversation agent and that agent would send the messages to your node-red instance.
@@technithusiast thanks :) I feel in time I will begin to understand because of how you present. Kind of a “due to exposure, I learned” without a deep dive focus if that makes sense.
@@florianosabatini4016 what are u using to search? Try using the Samba addon. I’ll try and update the readme with a screen grab of what I see if that doesn’t work try using the config in chat node
@@technithusiast love your content. I'm having the same issue. I have nodered under my custom components but I don't see that folder. got samba installed and while I see my home assistant device on my network, I don't see how to access the file structure. any ideas on where to find the settings.js file?
Hey bro, thanks for your outstanding videos. I get the following debug message when I try to ask what the color of the sky is; Error: 400 [] is too short - 'tools'. I can't figure out why this is happening. :/
This is amazing! Already started implementing and it looks super promising :) I did notice that since I want to expose it to a lot of entities, and I have added some extra rules and functions, I end up sending A LOT of text with every request, and end up exceeding the token limit. I was wondering if there's a way to create the bot in my openai account, define the rules and functions there, and communicate with it instead of a fresh chat each time. If it is possible, it will allow me to only send the entities list with each request. What do you think?
I like your thought process! Currently i don't think there is a way to do this well. The caveat at the moment is that if an AI is suppose to control our smart home it needs to know the state of all our devices in every single conversation. For example if I sent the Chatbot the state of all my devices once and then i turn on the light. If i asked the bot which light is on, it will get the answer wrong because the state changed and we did not update the bot. (This is why Assistant API from OpenAI doesn't work for this use case either). The best strategy so far is splitting your automations in such a way where it doesn't need to know a lot about everything. Or even more complex, force the bot to ask you about the state of specific devices before it tries to execute commands. We're currently talking about this in our channel membership is you would like to participate 😁 th-cam.com/channels/N4_NYvCU-tvIG3F1QSVdSw.htmljoin
I’m a bad litmus test since I have to test this extensively. And it would also depend on the automations. Back when I first started $5 was enough. But if I were to guess, $10 should be enough for most users but this is just a guess.
Hi, it's me again. I have finished the video, tried to follow each step but i get the error that you show with "OpenAI Tool Enabled" "call service node is missing api "domain" property, not found in config or payload" i have the OpenAI Tool in my flow enabled correctly. i tried to copy the lines from your video (here is the result) { "type": "function", "function": { "name": "light", "description": "turn on or off the lights", "parameters": { "type": "object", "properties": { "action": { "type": "string", "description": "action to run which will turn on or off a light", "enum": [ "turn_on", "turn_off" ] }, "entities": { "type": "array", "items": { "type": "string" } }, "response": { "type": "string", "description": "A friendly response to the given command" } }, "required": [ "action", "entities", "response" ] } } } but still i get the same error in the call service. "missing api domain" any ideas?
Sorry for any confusion the video may have caused. There are so many moving pieces that a few details may have slipped through the cracks on my side. Seeing that many may be inclined to try node-red due to this plugin. I will see if i can create a Node-Red crash course.
@@technithusiast bugs and issues are part of the experience. thank you for your work. i am trying to figure out how i can make it more efficient. how can i give OpenAI more info like "can you turn off the light that is turned on the longest in my office?"
If you know you want GPT to answer specific kinds of questions you should update the data you pass to the System node. For example you can do some preprocessing to the entities and include a “duration” property. Or since entities already have a “last_updated” property, you can give GPT the instructions on how to calculate duration if asked.
Interesting video! I am actually developing a similar tool where users can build statemachines to run automation with ai actions and api calls. It also supports local language models. Pm me if interested, I will provide a free and a premium tier of the software.
can anyone help me out? i am trying to follow the video. i have created the api key, i have paid 10 euros as a "bank" for the chatgpt. i have installed the plugin in the node red. i have followed the first steps where you place the timestamp, openaiuser... openai chat and debug.... these are my node red debug issues.... (even though i even created the openAI credentials) 1/6/2024, 8:15:00 PMnode: Home Assistant Sers msg : error "Error: No credentials provided" 1/6/2024, 8:15:00 PMnode: Home Assistant msg : error "Error: No credentials provided" 1/6/2024, 8:15:00 PMnode: Check how many humans msg : error "Error: Invalid server config" 1/6/2024, 8:15:00 PMnode: Check how many humans msg : error "Error: Invalid server config" 1/6/2024, 8:15:00 PMnode: Take Snapshot msg : error "Error: Server config node not found" 1/6/2024, 8:15:08 PMnode: 9df8b46cec9967d8 msg : error "Error: 404 The model `gpt-4-1106-preview` does not exist or you do not have access to it. Learn more: help.openai.com/en/articles/7102672-how-can-i-access-gpt-4."
A great companion to your earlier ai masterclass videos. Also a great example of why I like Node-RED: the ability to exchange automations or fragments. HA Blueprints eat your heart out!
Glad you like it. I’m dropping more videos that go into detail for each node. Stay tuned! #NodeRedIsTheBest
This is exciting, thank you! I'd love to see a more in depth video on using this to automate the 'Good Morning' message you touched on in a previous video. This would push the WAF over the top!
😂😂 sure thing! I’ll go over a few of the GPT related automations I have.
Best tutorial I have seen so far. Greetings from Germany.
Glad you liked it!
I’m really glad I stumbled across your channel. You’ve got some great content man. 👍🏻
I also love that you’re a Node-Red guy. I can’t stand setting up automations and such in HA.
lol you’re my peoples. It feel rare to find folks who prefers node-red over HAs automation
Wow!! Simply amazing, thank you. A little over my head at the moment, but looking forward to learn more.
Glad you enjoyed. I have more tutorials on my channel to help you get started 😁
Huge! Man this is so flexible/powerful. Thanks so much for building this! Excellent work! 🎉
Thanks, I really appreciate it! After seeing your video on LocalAI, I'm looking into adding new node that allows users to use any custom LLM as long as it is compatible with GPT's API.
OMG I really love your Project!
Is there a way to specify past messages like assistant or user messages to get a continous conversation? :)
Regards from Switzerland
Hey I’m glad you like the project!! Currently the nodes are pretty open ended so you could do it manually by saving the response and including it in subsequent calls. I have some updates I’m working on to easily facilitate conversations and using a local LLM instead of needing OpenAI
Hey, thanks again for this great video and all the effort you put into this. Will definitely take a closer look. I found extended openai conversation a couple of days ago and was just about to comment until you also mentioned it. It's like magic. I asked it yesterday if it could tell me what time I made a fire in the fireplace due to a rapid temperature increase in the living room and it nailed it. GPT4 though, I figured results are so much better with the latest model. Right now I'm installing localai for a test because I hit my limit in gpt4 rather fast with extended openai integration. It's just so much fun to use. Now I'm thinking about getting new hardware to run the assist pipeline and LLM locally. 20 buck a month for gpt 4 is quite a lot if you don't really need it.
Keep up the great work! Greetings from Germany
I always forget how global TH-cam is and that people from all over the world can find my content. I like your enthusiasm and im glad you're enjoying my content.
Great stuff. I am looking for your Membership but I am not seeing it from the TH-cam iOS app. Am I missing something or does it need to be enabled?
It's active. Im not sure why iOS devices are having problems. Try this link: www.youtube.com/@technithusiast/join
Fantastic nodes! I'm genuinely impressed by them. Your content is outstanding. Could you delve deeper into how to tap into home assistant with a template node to grab information from home assistant? I feel as though I'm not completely understand the full range of possibilities you're seeing here.
I glad you’re enjoying the content! I have a few videos hat teach how to use some of the nodes ( like this video:
th-cam.com/video/oWP8es4g4D0/w-d-xo.html ) and the past 2-3 recent videos give examples of different automations ideas that use these nodes.
Ahh, thank you. The Battery Notification example video (th-cam.com/video/T-cIfzZxTmc/w-d-xo.html) explained that to me! Very good, thank you!
Awesome, I’m glad it worked out!
Terrific Video and Nodes! Got it going but have a question. I'm planning on using this outside HA for a museum educational application and I would like to limit the chat to a specific area as:
const query = datastore
.createQuery('Animal')
.filter('type', '=', 'marine mammal')
.filter('familyFriendliness', '=', true);
Thanks this video helped so much.
Im glad this helped and very interesting use case you have!! Yeah, I wanted something that was pretty flexible where i don't have to send it a ton of data. You can definitely scope down the information however you like before sending the data.
@@technithusiast any suggestions on how this would be done with your nodes? Not sure at all how to add it. Thanks for the response.
@@technithusiast Actually I got it. Just added the criterial to the user node and it works, much obliged.
This is so cool! I am so pumped for this! I tried to set it up but I run into problems with intent. It says "The config is missing Intent Name or missing payload.functionName" but shouldn't that be added automagically when you register the intent?
Love the enthusiasm! Another viewer ran into a similar issue and was able to solve it by attaching a debug node and reading the data. If you're still unable to get it working let me know and i can sync up with you offline for additional help.
@@technithusiast I have been experimenting with setting up voice assistant at home and I have gotten it to work with the home assistant “assist” but it sucks so bad. So that’s why I’m happy to see movement on this! I would love to help you on this. I’m a programmer with 9 years of experience working as a consultant in various fields. And I’m a huge home assistant nerd. I use all node red for my automations.
At the moment I have loads of time since I’m between assignments.
@@technithusiast I looked at the debug and I’m not sure what I should be looking for. But I did see one strange thing. It says there is a registered intent that is called “lightOn” but there isn’t. There was one before but I renamed it.
Edit: looked at this some more. And I’m almost positive there is a bug when deleting “register intent” nodes that causes them to not be removed from the tools visible to OpenAI
It is definitely possible as I’ve had issues around that use case. To help debug, can you send me two things:
First a screen recording of how you setup the automation and the error results of what happened.
Second, export your entire flow as a json.
Send both the json and screen recording to technithusiast+help@gmail.com
@@technithusiast I sent you an email
HI sir, can we use gpt for predictive maintenance?
This is really awesome. Thanks for this great tutorial. This opens for me so many ways to improve my smarthome.
🙌🏾🙌🏾
This is epic. Thanks for open-sourcing it!
Glad you like it!
Great video tutorial as always. I know it’s slow but have you thought of making a video that uses local AI to keep information safe and if your like me and internet is spotty.
Yes! I’m currently exploring that route but I’m having trouble getting localai to work 😓
Are there updated instructions for 2024.10.2 13.2?
First of all thank you for your inspiring videos. You deserve much more subscribers. I like your nodes for node-red very much. But what i ask me all the time is: How can i extract the contenct from the homeassistant assist or the stt from my sattelite to use this content as a comand for your nodes? Please help me with that. Thanks
Thank you and I’m glad you’re enjoying the content! I’ve seen some code from other clever folks that’s been able to get get and return text to home assistant assist. I’ve been only able to get text from it (I’m still trying to figure out how to send text back asynchronously)
You can use the “sentence” node in node red and use {wildcard} in the sentences box. This will trigger this node with the text from assist. However this is a one way street and I haven’t figured out how to respond after I processed the text
Ooh great. At least a start. Thanks for replying.
@@technithusiast Ive tested your suggestion. Sadly i dont get any output when I create a sentence node and Insert {wildcard} into the sentence box. Ive connected the sentence node to a debug node wich debugging the whole msg. My goal would be to aks assit for example "how late it is?" and this sentence should be send to node red to use it for controll your created openai nodes.
[
{
"id": "4b30d2bbfa1115cb",
"type": "ha-sentence",
"z": "92271de62069540b",
"name": "",
"server": "c1ba182f9955500a",
"version": 1,
"outputs": 1,
"exposeAsEntityConfig": "",
"sentences": [
"{wildcard}"
],
"response": "",
"outputProperties": [
{
"property": "topic",
"propertyType": "msg",
"value": "",
"valueType": "triggerId"
},
{
"property": "payload",
"propertyType": "msg",
"value": "",
"valueType": "triggerId"
}
],
"x": 260,
"y": 6010,
"wires": [
[
"8902f606ae8c66cd"
]
]
},
{
"id": "8902f606ae8c66cd",
"type": "debug",
"z": "92271de62069540b",
"name": "debug 137",
"active": true,
"tosidebar": true,
"console": false,
"tostatus": false,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 475,
"y": 6010,
"wires": []
},
{
"id": "c1ba182f9955500a",
"type": "server",
"name": "Home Assistant",
"version": 5,
"addon": true,
"rejectUnauthorizedCerts": true,
"ha_boolean": "y|yes|true|on|home|open",
"connectionDelay": true,
"cacheJson": true,
"heartbeat": false,
"heartbeatInterval": "30",
"areaSelector": "friendlyName",
"deviceSelector": "friendlyName",
"entitySelector": "friendlyName",
"statusSeparator": ": ",
"statusYear": "hidden",
"statusMonth": "short",
"statusDay": "numeric",
"statusHourCycle": "h23",
"statusTimeFormat": "h:m",
"enableGlobalContextStore": false
}
]
Hey this is probably something I would be better suited to help as a channel member as you would have access to the discord server so we can converse faster and share code easier. However based on what you’re mentioning Assist doesn’t talk directly to node-red. If you use the conversation api in node-red it will definitely pick it up but the Assist dialog in HA requires a custom plug-in that would act as a conversation agent and that agent would send the messages to your node-red instance.
I don’t understand any of this, but I find it exciting and in time, maybe something I will attempt.
No worries! I'll be releasing videos to show some of the automations you can pull of using Node-Red + AI-Intent plugin 😁
@@technithusiast thanks :) I feel in time I will begin to understand because of how you present. Kind of a “due to exposure, I learned” without a deep dive focus if that makes sense.
Hi, I have Node-red as HA integration and I'm not able to find the settings.js file to enter the AI API
If you are using node-red in Home Assistant, you may need to look for a folder called addon_configs and look for a generated folder ending in _nodered
@@technithusiast unfortunately there are no folders with such names in my HA
@@florianosabatini4016 what are u using to search? Try using the Samba addon. I’ll try and update the readme with a screen grab of what I see if that doesn’t work try using the config in chat node
@@technithusiast love your content. I'm having the same issue. I have nodered under my custom components but I don't see that folder. got samba installed and while I see my home assistant device on my network, I don't see how to access the file structure. any ideas on where to find the settings.js file?
figured out samba
Hey bro, thanks for your outstanding videos. I get the following debug message when I try to ask what the color of the sky is; Error: 400 [] is too short - 'tools'. I can't figure out why this is happening. :/
Could be a regression. Try setting the tool choice option to “None” if you’re not using the OpenAI tool node
This is amazing!
Already started implementing and it looks super promising :)
I did notice that since I want to expose it to a lot of entities, and I have added some extra rules and functions, I end up sending A LOT of text with every request, and end up exceeding the token limit.
I was wondering if there's a way to create the bot in my openai account, define the rules and functions there, and communicate with it instead of a fresh chat each time.
If it is possible, it will allow me to only send the entities list with each request.
What do you think?
I like your thought process! Currently i don't think there is a way to do this well. The caveat at the moment is that if an AI is suppose to control our smart home it needs to know the state of all our devices in every single conversation. For example if I sent the Chatbot the state of all my devices once and then i turn on the light. If i asked the bot which light is on, it will get the answer wrong because the state changed and we did not update the bot. (This is why Assistant API from OpenAI doesn't work for this use case either). The best strategy so far is splitting your automations in such a way where it doesn't need to know a lot about everything. Or even more complex, force the bot to ask you about the state of specific devices before it tries to execute commands. We're currently talking about this in our channel membership is you would like to participate 😁 th-cam.com/channels/N4_NYvCU-tvIG3F1QSVdSw.htmljoin
are we going to be able to use Ollama with this method?
Not at the moment. For version 2, I’m looking into creating a “3rd-party” node that would enable the use of compatible open source LLMs
Hmmm...my NodeRed installation seems to not have a settings.js file. It is nowhere to be found
If you are using node-red in Home Assistant, you may need to look for a folder called addon_configs and look for a generated folder ending in _nodered
@@technithusiast Thanks, I found it by connecting to my HA via Samba. Now it works like charm an opens a whole lot of possibilities.
I think I've said this before, but I want to be you when I grow up. First, I have to go back in time quite a bit. But, that's just a detail.
🤣🤣
Boom! Wow. Thank you
You bet!
Fantastic add my thank you in the form of a message to help the algorithm if it does
Lol I’m glad you like it ✌🏾
Hey! Just curious is there any chance you can tell/show us how much your usage is with OpenAi per month?
I’m a bad litmus test since I have to test this extensively. And it would also depend on the automations. Back when I first started $5 was enough. But if I were to guess, $10 should be enough for most users but this is just a guess.
@@technithusiast Hello, perfect thank you :) I have enabled the CAP just in case!
@@aydanabrahams9665 glad I could help!
Hi, it's me again. I have finished the video, tried to follow each step but i get the error that you show with "OpenAI Tool Enabled"
"call service node is missing api "domain" property, not found in config or payload"
i have the OpenAI Tool in my flow enabled correctly. i tried to copy the lines from your video (here is the result)
{
"type": "function",
"function": {
"name": "light",
"description": "turn on or off the lights",
"parameters": {
"type": "object",
"properties": {
"action": {
"type": "string",
"description": "action to run which will turn on or off a light",
"enum": [
"turn_on",
"turn_off"
]
},
"entities": {
"type": "array",
"items": {
"type": "string"
}
},
"response": {
"type": "string",
"description": "A friendly response to the given command"
}
},
"required": [
"action",
"entities",
"response"
]
}
}
}
but still i get the same error in the call service. "missing api domain"
any ideas?
and i figured it out with debug... the issue is with "functionName" , when it should be "domain: msg.payload[0].nodeName" instead
Sorry for any confusion the video may have caused. There are so many moving pieces that a few details may have slipped through the cracks on my side. Seeing that many may be inclined to try node-red due to this plugin. I will see if i can create a Node-Red crash course.
@@technithusiast bugs and issues are part of the experience. thank you for your work.
i am trying to figure out how i can make it more efficient. how can i give OpenAI more info like "can you turn off the light that is turned on the longest in my office?"
If you know you want GPT to answer specific kinds of questions you should update the data you pass to the System node. For example you can do some preprocessing to the entities and include a “duration” property. Or since entities already have a “last_updated” property, you can give GPT the instructions on how to calculate duration if asked.
Interesting video! I am actually developing a similar tool where users can build statemachines to run automation with ai actions and api calls. It also supports local language models. Pm me if interested, I will provide a free and a premium tier of the software.
Sounds great! What is it called?
can anyone help me out? i am trying to follow the video. i have created the api key, i have paid 10 euros as a "bank" for the chatgpt. i have installed the plugin in the node red. i have followed the first steps where you place the timestamp, openaiuser... openai chat and debug....
these are my node red debug issues.... (even though i even created the openAI credentials)
1/6/2024, 8:15:00 PMnode: Home Assistant Sers
msg : error
"Error: No credentials provided"
1/6/2024, 8:15:00 PMnode: Home Assistant
msg : error
"Error: No credentials provided"
1/6/2024, 8:15:00 PMnode: Check how many humans
msg : error
"Error: Invalid server config"
1/6/2024, 8:15:00 PMnode: Check how many humans
msg : error
"Error: Invalid server config"
1/6/2024, 8:15:00 PMnode: Take Snapshot
msg : error
"Error: Server config node not found"
1/6/2024, 8:15:08 PMnode: 9df8b46cec9967d8
msg : error
"Error: 404 The model `gpt-4-1106-preview` does not exist or you do not have access to it. Learn more: help.openai.com/en/articles/7102672-how-can-i-access-gpt-4."
well i guess it needed time because now i can get a response... let me move on with the video :)
Glad it's working!