SuperHouse #41: Datalogging with MQTT, Node-RED, InfluxDB, and Grafana
ฝัง
- เผยแพร่เมื่อ 11 ธ.ค. 2020
- * www.superhouse.tv/41-datalogg...
* www.superhouse.tv/episodes
Recording sensor data and then reporting it is a very important element of many automation systems. By setting up a datalogging system using Mosquitto, Node-RED, InfluxDB, and Grafana you can record data from many different sources and then show critical information and trends over time.
See the link above for a step by step guide, including all the commands so that you can easily copy and paste them.
SuperHouse:
- www.superhouse.tv
- / superhousetv
- / superhousetv
Jonathan Oxer:
- / jonoxer
- / jonoxer
Support me on GitHub Sponsors or Patreon to receive exclusive discount codes, access to exclusive sections of the SuperHouse Discord, and early access to new episodes: www.superhouse.tv/support - วิทยาศาสตร์และเทคโนโลยี
That is by far the best video I have seen on Node Red! Most “How to” videos are so quick that it is really hard to follow, yours is perfectly paced and beautifully explained. I now feel confident in setting up InfluxdDB and Grafana. Thank you so much
I was waiting for someone to make a video like this a long time ago! With exactly all this elements, thank you so much!!
This is an excellent, detailed yet understandable walk through of this whole setup. Absolutely awesome work!
I wish this video existed months ago when I was trying to figure this out. Great job! Thank you!!
Great video Jon, thanks for filling in a lot of gaps for me in the whole sensor to presentation stream.
Node Red and Grafana look like really useful tools for this application. Much appreciated.
Great! You did an awesome job explaining everything in a way anyone can follow!
Dear Jonathan. Thank you for making this video and making it available for us. I had been looking around for a informative video on the topic, since past few days. You have made it easy for us to understand the flow and enabled us to implement it ourselves.
This makes me want to have another go at Node-Red. Thanks for the great information.
Great tutorial! Thank you! I was struggling with Nr and Grafana.
Excellent! Thoughtful balance of flexibility vs complexity. Can’t wait to build my own stack on my RPi. Well done, great demo!
Oh man, you saved my day! Formating the data with json in Nodered was exactly was i was looking for - THE WHOLE DAY! Now my Jelly switch transfers numbers and not strings anymore. Thank you SOOO MUCH!
Thank you this !!! This is really great. Explanations are clear, easy to follow, and the seed is great. You explained so much in a single video. Please keep posting!
Nice tutorial. I’m a big fan of mqtt, node red, influxdb and Grafana and wild agree that this is the way to go to support future projects. 100k tantalisingly close - well done Jon, well deserved.
Thanks Adam!
Great tutorial! I've always wondered about Node Red and now this makes sense on how to set it up. I'm probably going to do this as a container in ProxMox as am concerned about the excessive writing to the SD card on the Raspberry Pi.
Clean and simple demonstration of all the essential tools importantly with good documentation to allow me to follow it - Fantastic kick start for automation and data recording - Thanks
*Really good and comprehensive tutorial. Thank You very much!*
This is by far THE BEST video on "how to use" .... all these appts !!!
Nice walk through!
I am just starting with node red. You can set the format on the mqtt input node to JSON, so you don't need the JSON node.
Having played with Node-RED a few years back and all that, this was a beautiful refresher and extension adding InfluxDB and Grafana. Thanks!
Fantastic walk-through - exactly what I have been looking for. Thanks!
Thank you for that great tutorial! I am sure I can use the information directly as I am redesigning my smart home.
Glad to have found your channel Johnathon, I really enjoyed this and hope to put it into practice.
this is like a one stop shop tutorial. Brilliant!
Super video... yes, long but worthwhile, educational and tells the story perfectly
Thanks for the video. I heard about Node-Red, InfluxDB and Grafana combination and never explore it. This video make the process easy to understand.
Great work Jon!! I have seen performance issue using Nodered. Rest of the components works flawless. I had switched to telegraf and the performance was far better.
Stumbled on this video just now… It’s one of the most helpfull vids i’ve seen on this topic! My compliments and thanks! .. subscribed immediately!!
Fantastic video. I wish I had know more about grafana two years ago.
I spent years creating a new concept for my company adding embedded “senors” to monitor the operation of a real-time core 5G network component and the only missing element was a presentation layer (GUI).
All the data (sensors) were in a database yet we couldn’t get funding for the final missing GUI/presentation layer.
I’m actually gutted I couldn’t complete my concept that would have revolutionised the monitoring of this critical product ☹️
Open Source is frankly amazing! 🥰
Absolutely fantastic video as everyone else is saying. Fun, inspiring and useful!
Great video, just what I was looking for, Thanks very much Jonathan!
I just wanted to say thank you for this very in-depth and great explanation! definitely gained a new subscriber.
Thanks for this informative video! Double thumbs up!
This is my reference video, very well made and even suitable for beginners!
Thank you so much for putting this together!
Very well explained. Thank you
Very good explanation of how to setup an extensive logging and visualization platform. Thanks!
I indeed came here for informations on telegraf, but I'm also relieved that one does not have to use telegraf.
I did this. Excellent video mate. After all, I needed to know my beer fridg ewas always at the right temp.HeHe. Not joking about that either. Grafana says my beer is -0.69C right this second.
nice overview and video... now I want to build an air-quality monitor as well!
Great video, I love the considerations for re-use, the versatility of the system and the excellent choice of software.
Super cool tutorial. Pulling data from a Sensirion SCD30 into a 3 gauge cluster for CO2, RH, and temperature. I need to learn how to combine outputs on Arduino to create one json instead of three. Thanks again.
Excellent video/ tutorial as always Jonathan. Thank you for all the time you spend creating them. I'm just about to teach my students IOT, what a better way of backing up my teaching by pointing them at your videos... Hope life is getting easier in Sunny Australia..
That's fantastic, thanks John! I'm glad the videos are useful. Since you teach this sort of thing professionally (I'm just an amateur who makes it up as I go along) if there are specific topics or ideas that would be useful for your students, please let me know.
Bravo, Got me thinking again
Great detailed video! uUst what I needed to know.
SPLENDID, JON! THANK-YOU SO MUCH!
I very much needed a detailed walk-through like this to get from zero to first-working. Trying to do this kind of first-time integration on my own is beyond painful, especially given how little spare time I have after my day-job. It takes weeks instead of hours for me to get things done starting from scratch, so not very many things get done!
My wish-list item would be for one more level of time savings: Automation to create database entries on the fly for each mqtt data item the first time it is seen, then add line charts to a dashboard to display them. I'd very much rather prune unwanted items than add the wanted ones one at a time! I've never used NodeRED, but I suspect it should be able to handle this task.
NodeRED looks to be a powerful OpenSource alternative to some of the National Instruments LabVIEW tools I've used in the past (and learned to love to hate). And I see PyNodeRED (pypi.org/project/pynodered/) will let me create and import modules written in my language of choice, Python, which I use at work for data transformation and analysis (and everything Bash won't do).
Again, this video is awesome. It feels as if it were created just for me, to greatly accelerate my learning curve and make me instantly productive. I didn't even know the questions to ask, and you answered them all. THANKS!
Next step: On to the web page for this episode!
Edit: Really, this combined with your Tasmota walk-throughs gives me a soup-to-nuts recipe for data acquisition, transmission, archive and display. Thanks!
Hi BobC, thanks for your positive comments! Being able to automate the end-to-end setup of a new data point would be fantastic. Since recording this video I've been told about a way to simplify the JSON-to-InfluxDB step, so I'm going to add that info to the tutorial page as I'm finishing it off. It doesn't automate the process of creating dashboard widgets, but at least it could save some time setting up Node-RED for each individual measurement.
Excellent tutorial. Brilliantly explained. You should be a teacher.
Wow. It finaly makes sense :) Succinct and clear explanations. I'm in! Thank you.
Hi, I really like your videos and have followed them for years. I always learn a lot from you. On this one i will suggest that you use a SSD drive in stead of a SD card for the system and the database. in this way you increase the reliability of the system over time.
You really made a very good job, in all of your video. It's very interesting and very useful. Many thanks !
Excellent video. Yesterday I added InfluxDB, Grafana and Telegram to my OpenHab + NodeRed + Zigbee2mqtt setup and I was interested to see if I missed anything. I didn't, but it this tutorial would have definitely saved me time in working out how to set it all up. I'm using node-red as the data bridge and rules engine with telegram only for alerts (and, later, maybe specific controls) - mainly because node-red is very powerful and local, whereas telegram relies on the internet. So for my water leak sensors and mains water valve cutoff system, I want that to operate simply and locally.
Bravo
You are a very good 'teacher' also for non uk native language like me.
Thanks
As always well presented!
Super Video from Superhouse....big fan 👍👍👍
Excellent explanation 👍
Amazing video, you describe it very well! ✌️
Excellent video ! Great Job !
Really helpful and easy to follow tutorial!!! 😃😃
Pretty god damn awesome. If you didn't do code reviews as in the previous video in this series I wouldn't have been as interested. I think the format of doing code walk throughs of microcontroller projects is one of the best ways to teach your viewers what's happening! One tip in general for the code you do as a good habit to develop. Instead of calling a function again and again and again in your main loop that always returns again and again the same return call data value such as when you are doing all the sprintf function calls to get the ESP ID move the function call out of the main execution loop() and up into the one time setup section. Then you only need one single call to that function for the entire time the microcontroller project is running -- assign the return value of the single ESP ID function call in setup and assign the data value in the return call to a variable -- or the hexadecimal equivalent that you want for this particular example. Then in your main loop() just reference the variable directly to save repetitive calls which when you design all your code this way can lead to faster execution times, faster time to reach that critical next sleep call and therefore more power savings -- less critical when powered by a wall wart power supply but very critical and good habit for when you start to build battery powered microcontroller solutions where every ounce of runtime life you can squeeze out per battery charge counts for something.
Thank you...
It’s very useful information
Congrats my friend! Very nice 👍🏼
You can also take the json object and send to influx. If you do that you dont have to split each value Up. And if your sensors follow a Common system in terms of json data vs topic then you can have 1 single Line that takes all data from your sensors to the influx. Then you dont have to care when you add more sensors :) i do that for both tasmota, My own sensors and others. The data ends Up tagged and with proper series names IF done properly :)
That's a fantastic tip, thanks!
@@SuperHouseTV also use tags/Keys in influx. Makes it easier to Group and show data without having to do queries for each sensor but instead use filters :) i have way more than 1000 data sources above is a must :) you could argue to send the data to influx directly or use telegraf but i like having all My data distributed via mqtt :) since i also use alot of the data for automations where they dont exist in ha. (I have used node red longer than ha)
Great video really good explanation 👍
Very Good presentation. Thank you for this, so useful to my application.
Excellent video.
Very good explanation for the combined software projects. It helped me a lot.
Thanx for your work!!!
Much appreciated. Thanks.
Excellent content and well presented
Superb video and information!!!
Wonderful, sir!
Great work. Thanks.
Well done, Very good video!
Thank you for your video.
Got it working very nice instructions. The only issue is that when you share Grafana, any random user can edit the Grafana dashboard. I could share just the plots (Kiosk mode), but when you hit Esc you go back to dashboard.
awesome video !
Exactly what I was after and a great pace and level thank you. My next task is how to store pairs of JSON data in influx (lat and long) to track and chart stuff - wish me luck
Great video!!!!! Thanks!!!!
Great tutorial!
Well done 👍
This is outstanding
Hi, I came across your channel today. I've to ask a favor: in which video do you explain the json setup for your sensor? I like to do that for my BME280. THX for that super video. Now I now where and how my weather station project goes.
brilliant thank you.
Great video. What Air Quality Sensor are you using? I'm looking for something similar with a MQTT output
Nice video! For step 7, is there a way to automatically split all the JSON items and store each one in the DB without having to manually copy/paste/change each entry?
Hi thanks a lot for the video. Also what model is your air quality sensor please
I love you. Thats exactly what I was looking for.
Or in German: Danke bester, genau nach dem Zeug mit deinem Vorgehen habe ich gesucht! :D
thank you so much !!
Great ! ... Thank you.
Very nice video...
Wery useful video, thank you Jon!
I would like to noticed: if someone wanna use the authentication from the version of mosquitto 1.5 , then need to write an additional row into the mosquitto.conf:
per_listener_settings true
If this row is missing, than the authentication will not work (actually without password will work the mosquitto_sub example (at 4:13).
Excellents video. Quick question, what would be the procedure if I wanted to access Grafana from a website instead of locally?
Thanks a lot
Thanks !!! Super
Thank You
Hi Jonathan thanks for this great tutorial .I've achieved to send the data from my Higrow sensor to Influx and Grafana.Now I need your help .How can we send the data to a sensor in Homeassistant and see it in Lovelace.
Amazing
awesome!
Danke!
Very interesting, loved it. Can you make a graph using take data from a raspberry pi (ex. temperature, CPU load, ram uses, and so on) in Grafana
Love it
You have a link for the mods for the ikea tables for that enclosure? :)
Very nice video. Thanks. It's exactly like my setup. I hope there will be a sequel.
I have been trying for a long time to figure out how to handle long-term data. For example, when I want to draw an annual graph, Raspberry is weak, it takes a very long time and it heats up a lot. So it is necessary to downsample the data. But this downsampled data goes into new metrics, and that means a new graph for long-term data. That seems silly to me.
Have you looked into InfluxDB retention policies? As part of the retention policy you average older data points so to minimise space at the expenses of accuracy
Yes, retention policies are something I should cover in a follow-up. It's very important on any datalogging system, but particularly on low-end devices like a Pi.