Andreas - another perfectly formed video - in every aspect. We appear to run parallel lives as I was working on this for my open air pollution monitoring project. There must be more than 24 hours in the Swiss day as I can't for the life of me see how you manage to produce a) so much high quality material, b) over such a diverse range of subjects c) in such depth in such a short space of time and d) in a Swiss accent!!!).
@@AndreasSpiess Yes, it will. The issue is the simplest way of amassing JSON over https from many disparate sources. Not sure how to organise the secure apis.
This set of tools is very flexible, I am using it myself and I was amassed at how easy it was to add a new device and process it's data once the whole system is configured. In my case I didn't have to go through the setup process because I just used a ready made raspberry pi image from the BigClown IoT bundle.
Super useful! Displaying data in a meaningful & useful way is always a great challenge between providing what is of interest to a range of user interests & storing what is needed to create this output in such a way that it is easily accessed & does not need a lot of storage. There were several great ideas within this both as shown with this software or for writing ones own software on different hardware. Thank you for sharing!
I took the newest RasPi-4 & OS Buster, used your selections @2:53 and the latest version of peter's script. It took 17 minutes for a full installation. Thanks for this video
Just to remind that "Retention Policies" in InfluxDB comes connected with the concept of "Continuous Queries". Maybe in a future video you can explain that. Great video as always. Thanks!
This video just gave me the push to convert my smart energy meter and heating data into Grafana and Influxdb. Until now I was using Highcharts and the Dashboard of Node-RED which is so difficult to setup. Now I can add historical data. Great video, thanks Andreas!
Well done you. This, #255, is my 'eureka' tutorial. Thank you for explaining why AS WELL AS how to. I've read the rest of the Internet and really should have started here. Sensors have been my professional 'thing' for decades and I can now see a better way to process and examine the data. Thank you.
I swear, you are a Swiss mind reader or something! I was sitting here, building my Node Red + Grafana + InfluxDB setup and your video got uploaded. O_O
You are seriously amazing. Thank you so much for this video - I've spend days trying to get this to work, and with just a couple hours and following your video, I've got everything up and running.
Cool, I use InfluxDB for my home, but never understood the retention policy, thank you. I have three point about your setup: - InfluxDB is part of the TICK stack, that has Chronograf as data visualization tool in place of Grafana, a different flavour of the same concept someone might prefer, and has Telegraf agent you can use to collect raspberry's own metrics to track its health, feeding them through the same InfluxDB and Grafana combination - This raspberry use case tends to put some pressure on the SD because of the big quantity of data written and red, it is important to consider an external HDD and a backup strategy (obviously you already know this) - All those "installations" and "SD images" are things of the past, you need to switch to the container approach, I am talking about Docker; that will give you ease of installation, configuration, resource balancing, sharing, testing different setup, replication of setup and tenths of other things you will discover
1. I made a video about SD cards and showed how you can enable an HDD. So far I had no problems with my SQLite database. The number of points written is not a lot. 2. As soon as Docker gives full HW access it will be a good solution also for the RPI. I had problems to access a simple USB port on my PC. Otherwise a good concept with a lot of potentials.
I've been agonizing over my heating system lately (I have been using a homebrew controller for my parents house heating unit for over 10 years now). Now I can just copy your image and dump my logging data in there while using a esp32 to control all the relais and sensors. Thank you so much!
Amazing video for home use 😇 I have a similar setup running in my new flat, including Gas-Heater control and a combination of all sorts of different mcus with sensors and actuators based on various projects with a centralized mqtt/influx/nodered/grafana server (mostly own-software, tasmota...) Keep up the amazing work - your channel is still one of the best (mostly because you don't have annoying intros and music or semi-funny jokes ;) ) Greetings from Austria
Thank you! Jokes are not easy in an international environment. Glad to read that you like them. I like music, but in videos like that, I hate it myself. This is why I do not use it...
Lovely!. I've used InfluxDB + Grafana for almost a year, but for me the confusing part is Node-Red. I've instead used a InfluxDB libary for my ESP8266 codes, there is also for Python, etc.. and a small program "Telegraf" can send data from windows, macs, and linux machines.
I did not know a library exists. Thanks for the tip. In my case, it is more flexible to use Node-Red as an "intermediate". And the rest (telegraph, python) is maybe for other channels ;-)
You can easily send data our from Arduino and the ESP8266 as HTTP, ingest it in a Node-Red inout, then send it out from Node-Red in MQTT on port 1883 or 8883 depending if you need certs for TLS1.2. It can do a lot as an intermediary, as Andreas said.
Very nice Andreas - I've just been playing with InfluxDB and Grafans with InfluxDB fed with SNMP data by telegraf, all in docker containers. This video has given me an idea to move that on and try to connect my weather station rather than just looking at my Internet traffic :) I'll need to try to understand my stations data stream format which should be fun. Thanks for another great video and the idea...
Big thanks. I was using the Node Red Dashboard and it was awful, nevermind browser issues if you left it on the dashboard page too long. Pushing my data to InfluxDB and then viewing beautiful charts in Grafana from that DB information is awesome. Thanks,
@@AndreasSpiess I would also add that I'm using Python on a (Pi Zero W) to push MQTT data to a remote server listening for MQTT and is hosting InfluxDB, Node Red, and Grafana. You just need to install Paho MQTT on the Pi (pypi.org/project/paho-mqtt/). Then in the script just import it (import paho.mqtt.client as mqtt), define the remote server (mqttBroker = 'foo.server.com'), set the client with client = mqtt.Client(), connect to the remote server, client.connect(mqttBroker), and then for every variable assign to your data feed, you then just publish it with something like this: client.publish('Temperature', temperature). Where temperature = bmp.temperature, a feed coming from a BMP390 sensor. I any case, following your instructions on Influx DB, Node Red (MQTT->InfluxDB), and Grafana has been a life saver. It sure beats trying to chart something from file data. :-)
Excellent video Andreas. Clear instructions and to the point. We have used mqtt (activeMQ) +nodered+mysql and we were planning to use influx dB since its performance is more suited to IoT and time series data. Node red is really amazing and can help you to get to the results without the need of programming. Grafana (even if its free) its excellent visualisation tool. We have everything working under Ubuntu VPS, docker images of MQTT, node red, mysql, grafana etc and the performance is more than enough (more than 25k measurements in one minute transfered from the remote sensor via mqtt to the database without any issues)
it would be great if any of you wants to collaborate for any EU funded research based projects. (as mentioned most of us live parallel lives.. and do similar things)
25k points per minute are quite a lot! I did not know this is possible. Docker is a good concept. But with Peter's script, it is also quite easy to install. And we have full access to the RPI hardware...
This is a new concept for me (and it's very interesting), I have used a more traditional sql approach based on the work by Csongor Varga "Working with time series data in Node Red with SQLite" on TH-cam. Your viewers might find the comparison interesting.
@@AndreasSpiess problems were caused by a double install of sqlite node, 1 in Pete's node selection, 1 after that with unsafe-perms, and the sqldbs node which is unmainained from 2 years, we removed it... and now nodes are installed 1 by 1, instead on a single batch, so if 1 fails, it's just that, it does not make the others fail all together... now seems all fine
@@AndreasSpiess minor issues solved this morning: switched to official grafana repository instead of the previous one, as now it exists and it's working properly (it was not so when i added grafana and influxdb to script, a year ago or so...)
@Andreas did you consider writing the name field not as a measurement but as a tag? This is achieved by writing not one object as msg.payload to the InfluxDB node, but writing a list [] of two objects, first one with the measurements, second one with the tags. If you set the name field as a tag, it should also show up automatically as a completion choice in Grafana.
Hi Andreas, you might want to consider hass.io which has all as point and click install packages for all you talked about today! Great also to create automation for your tv lighting etc.. all over a web gui so easy it will withstand the wife test.
Thank you so mucho for this. I succesfully followed this tutorial and now have an energy monitor and a water pressure sensor working and displaying data in grafana.
Hi! Could you please upload the NR-flow? I am really interested in the details in the function-nodes. I am struggling with my own input from NR to InfluxDB. :)
I started out using node-red and MongoDB, because you used to be able to get free cloud-based MongoDB services, but that time-based data consolidation is really nice, so I'll probably switch to InfluxDB for my projects from now on.
@@AndreasSpiess yeah, MongoDB seems to store a lot of metadata for every record, which could be useful for some things, but not so much for simple IoT projects.
Andreas did you stop numbering your videos ( #Number). The last one was #245. In my opinion, it facilitates finding specific videos and contents. I hope you continue using it. Thanks
Thank you - I installed Grafana and InfluxDB on my existing Raspberry Pi 3+ with an already installed Node-Red and Mosquitto. For the installation of Grafana and InfluxDB I found some instructions on the internet. I use Node-Red and Mosquitto a lot for my home automation and used until now SQLite or flat files. Your Video helped me to setup it - and it runs fine. Now I'm going to adapt my home automation to the new data sources.
can you make a video on the topic of retention strategies, it seems to me to be a little more complicated? Would you use this configuration, shown in the video, for everyday use?
Dear Andreas, great Video! Quite complex things explain so amazing short. Stuff like this can only see very rarely. Love your comments why you choose this tools. Thank You! Greets Thomas
Thanks for a great video! I use Influx and Grafana professionally at work to analyze metric data from a mobile application. Thankfully I have an entire team of people to manage maintaining those systems. This kind of inspires me to see if I can hack my solar system on my roof (it uses zigbee apparently) to get data into influx and match it with real time weather so I can analyze better how my solar is performing.
Great, finally this seems to solve my IT problems for IoT. HomeAssistant is also good but seems to have an easy start with challenging future. (developing all the plugins etc. for simple additional custom features!)
@@AndreasSpiess For whom, who might be interested... Home Assistant (HA) has MQTT capability. An MQTT broker can direct messages to HA and HA will store them and display in a nice manner. Only by writing some making configuration files, one can trigger alerts, actions etc... e.g. I could set up a threshold for humidity sensor, after crossing this level, HA triggered a relay, which controls the fan. It is neat but extending it seems to be difficult.
Thanks for warning Giuseppe Suanno. I was about to do this on a zero because i happen to have one available. Even if i suspected it might need a gruntier board like PI3. Why only on Pi3? Can you please elaborate or provide source?
I've got it running on a RasPi 2, and have also ran it on the original 512MB RasPi Model B. Granted it's slower on the 1st generation Pi, but it runs with no unreasonable difference in speed on a Pi 2 compared to a Pi 3. (which I've also run it on)
In the measurement "stations". Wouldn't it be the better option (faster for queries) to store the "name" as tag and not as field? Tags are indexed and fields aren't. Still starting with InfluxDB but that's how i understand it.
Did you set your retention policy for the weather station to do hourly averages after one year? If so, how? I am seeing it is ideally done during the database creation but can be done after. Eitherway the syntax isn't so clear to me, and I like the retention policy you mention.
Playing with influx and grafana for computer (cpu ram ssd-cache and disk usage) and backup status monitoring. Got it working with docker containers after a few hours tinkering. Did not see the need for node red, though. Using curl for posting, and trying to get raspberry+Screenly working for display.
I like node-red because I know it ;-) there are many other possibilities to bridge from MQTT into influxDB. Did you try docker to use GPIOs or Serial on the RPi?
@@AndreasSpiess I used docker for influx and grafana, not for the data collector /sensor program. You may be able to use gpio and serial inside a container by usingal a privileged container and mounting /sys and /dev/tty*** as volumes on it.
I would like to see also a video about continuous queries and retention policies in detail with real examples related to weather (and possibly energy data). I have about three years of data every few seconds for my many weather and energy sensors but find very cumbersome to implement retention policies and continuous queries and to downsample the already existing data.
I do not see that this is a topic with lots of interested people. I think you will find some information on the net. Your situation is perfect as you can play with your data and find out which policy fits your needs
Nice video Andreas! In my case, instead of manual instalation, since I started using Home Assistant a few days ago, I just added InfluxDb and Grafana pluggins in Home Assistant which was like 5 minutes effort (Node Red I already added before). Anyway your instructions helped a lot for a fast start. And a sugestion - if you didn't already - check out Home Assistant. Nice platform + combined with Mosquitto, Node Red and now InfluxDb and Grafana ... make it very interesting. I use some manually programmed microcontrolers (bathroom humidity and fan control), Nextion displays from your video over MQTT, and dwo days ago I reprogram own application for LED strip and Sonoff Basic with esphomeyaml. Addionally using Alexa over Node Red, that fits very niceli into Home Assistant. New graphs from bathroom humidity, temperature and fan control is today nice addon. Point is that any additional knowledge nicely fits in picture. Constant evolution...
Hello Andreas, does this work with an DHT22 Sensor or an Adafruit Sensirion SHT31-D? Where would I need to set the sensors up, in Node-Red? Thank you - great video!
Thank you for the video, otherwise I have a question is it possible to integrate Grafana in a php page or is it an open source that can be easily integrated into a web page in your site?
What do you think about using the SD card long term with the amount of writing influx will be doing? Would you use an SSD if you were planning on keeping your setup long term?
Hi Andreas! Love this video, and thanks for putting together the image. I recently ordered a few Rasperry Pi Zero 2 Ws, and I'm trying to load the image you provided to one of them. When I load the flashed SD card and plug in the power, the device doesn't boot. No power light, no indication that it's connected to wifi (after adding wpa_supplicant.conf and SSH to the root directory). Is there any reason the image as provided wouldn't work on a raspi zero 2 W? Thanks again!
I do not know if it works on the 2W. Two things to consider: 1. The memory of the 2W is very low for such applications. I would go for at least 2G 2. Maybe you watch the video about IOTstack. There you find another way of installing all the goodies...
Hi Andreas, just to share. I created a "home heating system monitor" based on some arduino-nano, reading NTCs/DS1820, and raspberry interconnected by field-bus based on modbus (RS485). Raspberry is running a python script to collect data and store it on influxbd, Grafana is used to visualize data and correlate info. GOAL: integrate solar PV, heat pump system and Gas heat generator, control floor heating and try to be more green. ... will see...
It sounds like an interesting project. Maybe you have also planned to add something like hass.io? And maybe you also watch the video about IOTstack (if not already done)
... Yes I am looking deep in Hass.io ... but ... before go in Strong IOT every one should read 'Abusing the Internet of Things'. For now l prefer to stay with a well lock down reporting tools.
Hi Andreas, relational databases do not necessarily have to be setup with fields like "Temp, Pressure, ...", you can as well do an different approach and use a dedicated field to indicate the type of the value. Something like "Timestamp, Type, Value", that way the only restriction is the type of value you want to store ( integer, float, string, ... ). If you want to get completely rid of that you should have a look at elasticsearch, it's a database system that is highly scalable and very dynamic. You can easily scale it up from one DB node to hundreds in seconds, simply by adding additional nodes and the same way back (in reality there is a little downside in downsizing - let me know if you are interested). Elasticsearch does NOT stick on types, or even database table schemas. It simply stores what you send - if there is no - for example Temperature - it just creates it dynamically for you. Additionally the values are the same flexible - no need to preselect a type like float or a length, it's just dynamically created on the fly. It's even able to store >100k Messages/s depending on the machine it's running on. In reality I had soe time ago a DB running where I was feeding event log data from 9 Active Directory domain controllers and keeping ~1 year of history of that data. Peaks where around 500k messages/s and around 5-7 million entries a day on a 6 year old reused hardware. Basically a perfect match for MQTT where we do not have strict type/name rules as well.
You are right. You can store these primitive sensor values in every database without any problems. AFAIK Elasticsearch is considered to be a "NoSQL" database. They are very good at certain things like you describe: Unstructured data and also full-text search. As I said, the strength of time-series databases is not the storage side but on the retention as well as the extraction of time-based data. I do not want to focus on database concepts on this channel because there are people which are much more capable than me in this matter.
It's a search engine, why should you use it as a database? How do you efficiently get the average of a field (or even better over a time period)? Types and structure shouldn't be abandoned if you don't have to deal with dynamic data.
@@RalfVogler : Elasticsearch is NOT a dumb search engine, it's a NoSQL database ! Beside that you can use several mathematical functions on fields, similar to a relational database. As MQTT as well is a typeless protocol elasticsearch would be a perfect match to store anything that would possibly arrive over MQTT. Along with Kibana it's quite easy to visualize the data, even other solutions ( including Grafana ) are possible. The answer to your questions is here : www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-avg-aggregation.html, it can be expanded to any other critera the system is aware of ( time, name, field type, ... ). Given the fact that we talk about sensor data to be stored, mainly through MQTT, it might be worth looking at something like Elasticsearch that could easily handle . However, it's anyway recommended to use the DBMS you prefer an feel safe with ! Additionally it of course depends on the needs and given infrastructure. If you try to store (any) Windows eventlog data, especially from a whole domain, into a SQL database you'll understand the advantages of Elasticsearch and it's scaleability ... I'm just bringing it to the table, it's everyone's decission to have a closer look or not. My recommendation is : If you run just a couple of sensors use what you already have. If you want to scale up to hundreds of sensors you might have a closer look to Elasticsearch.
"use a dedicated field to indicate the type of the value" My point was that if you do that, you can't do things like average (which you are probably interested in for sensor values). Correct me if I'm wrong, I never used Elasticsearch. Of course you can do anything. Just writing to a file is also fast. I would assume that even for storing data of hundreds of sensors the choice of database doesn't matter. Where you will see a difference in performance is when you search in non-indexed fields, aggregate data over wrong types or use joins when the database doesn't support it. The built-in capabilities is the other important point: does Elasticsearch support querying time series data? Concerning types: either you declare the types for each MQTT topic or you let something like InfluxDB infer them from the first message.
Hi @Andreas, thanks for this video. Definitely got me excited to revise my home setup. Unfortunately, I'm running into a lot of issues. I had two different RPis to run PiHole and Home Assistant. I wanted to move to an SSD to store and analyze data locally (influx, grafana). I had a MQTT broker set up in HA already and was interested to migrate this to a container. Unfortunately I can't get the MQTT addon or integration working in HA. I tried to contact the HA discord support and they were very clear this is not a supported installation method. It seems to mix the supervised/Haasio and core/docker methods. So...I'm concerned about maintainability of this method. Hoping you have some thoughts on viability here. Thanks!
@@AndreasSpiess apologies, looks like I commented on the wrong video. Yes, I am using IOTstack. The Home Assistant community was clear this is not a supported way of installing HA. Seems a big downside to not be able to leverage such a well documented and supported community. Hoping it can evolve.
Thanks for suggesting the script from Peter Scargill. However the script did not install properly and the pi4 seems to be slow now. Is there any way to roll back the script? I rather install the tools one by one.
@@AndreasSpiess The InfluxDB was really fast and easy! I installed influxDB in a chrooted Linux on an Android TV box where I am using it as a mqtt broker and node-red administrator. Now I am testing the first data recording from my air conditioning energy meter. Next step to install Grafana. And all these thanks to you again :)
Hi Andreas, i've tried to use your SD card image on two different SD cards and i cant get any of them to boot, i know this post is two years old, is there an updated version of this i can try? Thank you for your work on the videos they are very interesting and i try to learn from them.
Andreas, can you share in some way settings of nodes that you use in node-red. I try to use image you shared, but my PI is 4b and probably this is why it doesn't boot with image you shared load in my SD card.
I have some problem with selecting appropriate choices when running Peter's script through putty ssh. What key do I have to press when I want to select/deselect the available choices?
I suppose all the credentials should be changed to use the SD card image securely? What kind of changes should be done? Paswords for the Raspberry Pi, Node_red, influxDB and Grafana should be changed? It would be nice to have a guide how to do this and how changing the password of one app affects to another app.. in other words, how to keep the system working after passwords are changed.
Thank you for the hundreds of really useful, interesting, up to date tutorials and info, you're pure gold! I have a question regarding the pi node-red server side, i need to have it in a distant location and feed it with 12V batteries (car battery) and let it run node-red influx db and grafana. What would be a good way to convert 12V down to 5V for the pi? Would one of these cheap buck converters do? What do you suggest? thank you in advance!
Grüezi Andreas, as always a great video with many explanation that everybody can follow and learn a lot. todays topic is very sexy, since data/process mining is the new lubricant of the world. as already told on TTN con I love your 'Mutterwitz'. cheers Michael (guy form Sulzer)
Very nice! The problem with the Pi for DBs is the SD. Even good SDs tend to worn out fast and you lost your valuable DB. I would recomend to store db in other disk even at the expense of use USB disk. There are other SBC with SATA/PCIe that will be more suited this purpose.
Its all the craze doing opensauce. I wonder what made u not choose non elk and or prometheus. I suppose grafana is top for these requirements. Miss lora and esp32 a bit... a lot to be said about long range iot.
Is it possible to use "measurements" from different Influxdb databases in the same "panel" in Grafana? I can't figure out the syntax for the FROM field in the query editor.
Hello Andreas. Do you know about any more or less procedure to have "secured":-) access to a local instance of node-red with MQTT, outside the local network with dynamic IP ? The methods i've found so far are riddled with cautionary measures such as different port allocation and the like... i'm not planning to control a high pressure boiler, but it would be nice to have a minimum sense of "security":-):-)...
I think a VPN would be the best solution. You get some free services. Maybe I make once a video about it. Usually, you need a piece of SW running on the RPI which connects to a server in the cloud. And you connect to it, too.
Thanks. I'll try with PiVPN and Google DNS and see how it performs. I've tried IBM Watson, which is a bit complicated, with a steep learning curve, and performed beautifully, but it has too many restrictions in free version.
Seems retention policy only tells the data base what old data to delete. Achieving the reduced resolution for older data requires a lot more work and down samples it in separate place. Then it is not possible to have a single graph in graphana. Or am I missing something?
@@AndreasSpiess Indeed, this seems to be complicated, but there is a solution here pommi.nethuis.nl/collectd-influxdb-grafana-with-downsampling/ I will see where this leads..
@@AndreasSpiess Unfortunately it only works, as the author says, when the end of plotted time is always now. On top of this, it does not take too much query to kill the rpi. it runs out of RAM fast. This kills mosquitto and node red for a while.
Great video and very timely for me as I am looking to complement my old RRD based graphics site with Grafana and Infuxdb. I have used Pete’s ( and Mr Shark’s) script along with his node-red big timer for years! While I do enjoy node red, I am looking to lower my dependence on it. Thus hints of ways to direct mqtt data directly to influxdb are high on my curiosity list!!
Are the same results possible if I have multiple Pis spread further out? I assume there'll be more setup and configuration bit I'm just dabbling into this for the first time. This is really close to what I want to do to monitor my property with multiple raspberry pis scattered about with sensors monitoring propane level, power generation ect.
I do not know why you would use more Raspberries. It complicates the setup. If one Raspberry is to weak you can change to a powerful PC with the same tools. If you want to have multiple installations you can watch my video #295 (which uses Docker) and Google for Kubernetes.
@@AndreasSpiess mainly cause I don't want to run sensor wires for hundreds of feet that's allot of wire and probably signal degradation. I want to put a Pi down by the propane tanks, one about a mile away at the well and one near my battery bank. All these things are far apart from each other. I want to do exactly this but through multiple Pis. One Pi just wouldn't be practical considering the physical distance.
I understand. For sensors I propose not to use raspberries. Small MCUs are better fitted for that purpose (cheaper, smaller, less power consumption, less maintenance). Maybe you watch also my LoRa videos if you need to bridge longer distances without wires. For small distances, Wi-Fi and ESP32 boards are a good choice. Maybe you google "farm relay system". (TH-cam deletes links)
@@AndreasSpiess thankfully I have really reliable wifi across my whole property off my neighbors wind turbine (don't worry he's aware incase you were wondering lol) time to drive deep back into this rabbit hole again, thanks for the replies!! This is the first video I've seen Grafana in and I'm sold on that already. Your videos have given me tons of starting points too!
Andreas. I downloaded your SD Card image and ran on my pi. Using your passwords as given at the end of the video, none of them work for me, in particular, the log-in credentials for the Raspian System are not accepted??. Alan
Hello Andreas! Thanks for the great video. I have already tried Pete's Script some time ago and am using node red for over 2 years knnow! Worked great, so i'm doing a fresh install now on the latest Raspberry and I am adding Influx DB now for the first time. Iw wanna try to store and read power consumption data that I am transmittting from an ESP8266 in my fuse box. Have you tried anything like that yet?
Hi, thank you for the video, I have a question regarding influxDB and Grafana, I want to use InfluxDb and Grafana not for IoT purpose but to a bi purpose, I have an app that logs events of user action and I want to create some dashboard of analytics of those events, will Grafana and influxDB are the right tools, and if yes, My logs will be kept on a device that doesn't have internet connection all the time, so the logs will upload to the influxDB in a later time, how can I use the event timestamp and not the timestamp of when I'm adding the log to influxDB? thank you in adavance
"We like programming, but we love result" Quote of the day. Thank you sir
You are welcome!
Even I can confirm, and I'm a developer.
this :]
Andreas - another perfectly formed video - in every aspect. We appear to run parallel lives as I was working on this for my open air pollution monitoring project. There must be more than 24 hours in the Swiss day as I can't for the life of me see how you manage to produce a) so much high quality material, b) over such a diverse range of subjects c) in such depth in such a short space of time and d) in a Swiss accent!!!).
I agree 100%. Andreas is amazing!! Thank you so much Sir for sharing knowledge with everyone and great commitment on making amazing videos !!
We are slow here. That is probably why our time runs a little slower... I hope the content still helps you a little for your pollution project!
@@AndreasSpiess Yes, it will. The issue is the simplest way of amassing JSON over https from many disparate sources. Not sure how to organise the secure apis.
Security is not my specialty, fortunately ;-)
Amazing! I had to stop the video on two occasions I was so overwhelmed by the sheer power of this suite of FREE software! Thanks Andreas!
That is the power of TH-cam. You can use it at your pace ;-)
The timestamps of the example data are from 2019-02-06. Please choose an appropriate date range in Grafana if you want to display them
Extremely usefull video - I am very happy to be one of your Patrion-supporters! Keep it coming, Andreas!
Thank you for your support! It is always good to read that my videos are useful...
This set of tools is very flexible, I am using it myself and I was amassed at how easy it was to add a new device and process it's data once the whole system is configured. In my case I didn't have to go through the setup process because I just used a ready made raspberry pi image from the BigClown IoT bundle.
I think it is not so easy if you have to do everything from the ground. So we both were happy we were able to stand on the shoulders of others ;-)
Mr Spiess, my deepest gratitude for your work and for sharing great content in a video like this. I became your Patreon today to support your work!
Thank you for supporting the channel. I appreciate it!
Super useful! Displaying data in a meaningful & useful way is always a great challenge between providing what is of interest to a range of user interests & storing what is needed to create this output in such a way that it is easily accessed & does not need a lot of storage. There were several great ideas within this both as shown with this software or for writing ones own software on different hardware. Thank you for sharing!
You are welcome. And I agree that retention strategies are not easy to define these days with very cheap disk space...
I took the newest RasPi-4 & OS Buster, used your selections @2:53 and the latest version of peter's script. It took 17 minutes for a full installation. Thanks for this video
Very fast! Did you try the docker Installation?
no I did not. I'll try to create a surveillance for 24/7 running PCs (if message does not arrive periodically - alert)@@AndreasSpiess
Interesting project!
We created a whole project with lots of useful containers (IOTstack)...
Just to remind that "Retention Policies" in InfluxDB comes connected with the concept of "Continuous Queries". Maybe in a future video you can explain that. Great video as always. Thanks!
I mentioned the compression, but not in detail. I do not plan a video on that because it is very special. Maybe once part of another one. Who knows?
This video just gave me the push to convert my smart energy meter and heating data into Grafana and Influxdb. Until now I was using Highcharts and the Dashboard of Node-RED which is so difficult to setup. Now I can add historical data. Great video, thanks Andreas!
I transferred historical data exporting to CSV and creating the needed input strings for influxDB with Excel.
Well done you. This, #255, is my 'eureka' tutorial. Thank you for explaining why AS WELL AS how to. I've read the rest of the Internet and really should have started here. Sensors have been my professional 'thing' for decades and I can now see a better way to process and examine the data. Thank you.
Maybe you also watch my videos about IOTstack...
@@AndreasSpiess I did Andreas and for the first time doing useful things with this stack. More thanks. My stuff is at rogerfrost.com. Best wishes.
Nice to see you're making good use of "the script" Andreass.
I always like to stand on the shoulders of big men ;-) Makes me more productive! Like that, everybody can profit.
@@AndreasSpiess Hah. I do that all the time. My script enables software like Node-Red, written by people cleverer than me!
Thank you both for your wonderful work. I have a beautiful image on my Pi and it’s all working 😁🏴
I have no words Andreas. Astonishing. Thanks.
You are welcome!
I swear, you are a Swiss mind reader or something! I was sitting here, building my Node Red + Grafana + InfluxDB setup and your video got uploaded. O_O
Maybe you can save some time now ;-)
You are seriously amazing. Thank you so much for this video - I've spend days trying to get this to work, and with just a couple hours and following your video, I've got everything up and running.
Glad to read that!
Cool, I use InfluxDB for my home, but never understood the retention policy, thank you.
I have three point about your setup:
- InfluxDB is part of the TICK stack, that has Chronograf as data visualization tool in place of Grafana, a different flavour of the same concept someone might prefer, and has Telegraf agent you can use to collect raspberry's own metrics to track its health, feeding them through the same InfluxDB and Grafana combination
- This raspberry use case tends to put some pressure on the SD because of the big quantity of data written and red, it is important to consider an external HDD and a backup strategy (obviously you already know this)
- All those "installations" and "SD images" are things of the past, you need to switch to the container approach, I am talking about Docker; that will give you ease of installation, configuration, resource balancing, sharing, testing different setup, replication of setup and tenths of other things you will discover
1. I made a video about SD cards and showed how you can enable an HDD. So far I had no problems with my SQLite database. The number of points written is not a lot.
2. As soon as Docker gives full HW access it will be a good solution also for the RPI. I had problems to access a simple USB port on my PC. Otherwise a good concept with a lot of potentials.
I've been agonizing over my heating system lately (I have been using a homebrew controller for my parents house heating unit for over 10 years now). Now I can just copy your image and dump my logging data in there while using a esp32 to control all the relais and sensors. Thank you so much!
I hope you will be successful in transferring the data from your source into InfluxDB.
Amazing video for home use 😇 I have a similar setup running in my new flat, including Gas-Heater control and a combination of all sorts of different mcus with sensors and actuators based on various projects with a centralized mqtt/influx/nodered/grafana server (mostly own-software, tasmota...)
Keep up the amazing work - your channel is still one of the best (mostly because you don't have annoying intros and music or semi-funny jokes ;) )
Greetings from Austria
Thank you! Jokes are not easy in an international environment. Glad to read that you like them.
I like music, but in videos like that, I hate it myself. This is why I do not use it...
Lovely!. I've used InfluxDB + Grafana for almost a year, but for me the confusing part is Node-Red. I've instead used a InfluxDB libary for my ESP8266 codes, there is also for Python, etc.. and a small program "Telegraf" can send data from windows, macs, and linux machines.
I did not know a library exists. Thanks for the tip. In my case, it is more flexible to use Node-Red as an "intermediate". And the rest (telegraph, python) is maybe for other channels ;-)
You can easily send data our from Arduino and the ESP8266 as HTTP, ingest it in a Node-Red inout, then send it out from Node-Red in MQTT on port 1883 or 8883 depending if you need certs for TLS1.2. It can do a lot as an intermediary, as Andreas said.
Very nice Andreas - I've just been playing with InfluxDB and Grafans with InfluxDB fed with SNMP data by telegraf, all in docker containers. This video has given me an idea to move that on and try to connect my weather station rather than just looking at my Internet traffic :) I'll need to try to understand my stations data stream format which should be fun.
Thanks for another great video and the idea...
There are a few other videos about weather station hacking and Pi and Docker (IOTstack)
Thank You!
I love how well structured your videos are. Suits my taste as german engineer ;)
Thank you! Maybe you also watch the one about Docker. It simplifies the installation of those programs
@@AndreasSpiess Yes i did and everything is working nicely together. :)
Big thanks. I was using the Node Red Dashboard and it was awful, nevermind browser issues if you left it on the dashboard page too long. Pushing my data to InfluxDB and then viewing beautiful charts in Grafana from that DB information is awesome. Thanks,
I also like Grafana...
@@AndreasSpiess I would also add that I'm using Python on a (Pi Zero W) to push MQTT data to a remote server listening for MQTT and is hosting InfluxDB, Node Red, and Grafana. You just need to install Paho MQTT on the Pi (pypi.org/project/paho-mqtt/). Then in the script just import it (import paho.mqtt.client as mqtt), define the remote server (mqttBroker = 'foo.server.com'), set the client with client = mqtt.Client(), connect to the remote server, client.connect(mqttBroker), and then for every variable assign to your data feed, you then just publish it with something like this: client.publish('Temperature', temperature). Where temperature = bmp.temperature, a feed coming from a BMP390 sensor. I any case, following your instructions on Influx DB, Node Red (MQTT->InfluxDB), and Grafana has been a life saver. It sure beats trying to chart something from file data. :-)
Excellent video Andreas. Clear instructions and to the point. We have used mqtt (activeMQ) +nodered+mysql and we were planning to use influx dB since its performance is more suited to IoT and time series data. Node red is really amazing and can help you to get to the results without the need of programming. Grafana (even if its free) its excellent visualisation tool. We have everything working under Ubuntu VPS, docker images of MQTT, node red, mysql, grafana etc and the performance is more than enough (more than 25k measurements in one minute transfered from the remote sensor via mqtt to the database without any issues)
it would be great if any of you wants to collaborate for any EU funded research based projects. (as mentioned most of us live parallel lives.. and do similar things)
25k points per minute are quite a lot! I did not know this is possible. Docker is a good concept. But with Peter's script, it is also quite easy to install. And we have full access to the RPI hardware...
This is a new concept for me (and it's very interesting), I have used a more traditional sql approach based on the work by Csongor Varga "Working with time series data in Node Red with SQLite" on TH-cam. Your viewers might find the comparison interesting.
SQL is always a possibility. It is stable and does its job.
Your channel is pure gold!!
Greetings from Chile!
Thank you very much!
@Andreas issues are now solved, as you know by latest comments on Pete's blog... i'm MrShark :)
Thanks! The video was already out then. I pin your comment that everyone can read it.
@@AndreasSpiess problems were caused by a double install of sqlite node, 1 in Pete's node selection, 1 after that with unsafe-perms, and the sqldbs node which is unmainained from 2 years, we removed it... and now nodes are installed 1 by 1, instead on a single batch, so if 1 fails, it's just that, it does not make the others fail all together... now seems all fine
@@AndreasSpiess minor issues solved this morning: switched to official grafana repository instead of the previous one, as now it exists and it's working properly (it was not so when i added grafana and influxdb to script, a year ago or so...)
@@squalazzo Influxdb node still requires manual installation - tried just a couple of hours ago
@Andreas did you consider writing the name field not as a measurement but as a tag? This is achieved by writing not one object as msg.payload to the InfluxDB node, but writing a list [] of two objects, first one with the measurements, second one with the tags. If you set the name field as a tag, it should also show up automatically as a completion choice in Grafana.
I did this after the video. It is easier for the selection in the dashboard. But during the take I did not know how to do it :-(
Hi Andreas, you might want to consider hass.io which has all as point and click install packages for all you talked about today! Great also to create automation for your tv lighting etc.. all over a web gui so easy it will withstand the wife test.
Maybe I have to look at hass.io once...
this was an eye opener for me. always a good time learning from you
Glad to hear it!
Thank you so mucho for this. I succesfully followed this tutorial and now have an energy monitor and a water pressure sensor working and displaying data in grafana.
Excellent!
Everything new. I had watched twice to understand. I want this setup too for my home sensors. Thank you dear.
You are welcome!
Im now successfully using grafana influxdb with mosquitto (EspMqtt) using node red on my OrangePi Lite.
Hi!
Could you please upload the NR-flow? I am really interested in the details in the function-nodes. I am struggling with my own input from NR to InfluxDB. :)
Amzaing effort from you and Pete's , keep the videos coming Andreas 👍
Thank you!
I started out using node-red and MongoDB, because you used to be able to get free cloud-based MongoDB services, but that time-based data consolidation is really nice, so I'll probably switch to InfluxDB for my projects from now on.
MongoDB seems to use quite a lot of disk space because it is made for other purposes. There is a comparison on the influxDB page...
@@AndreasSpiess yeah, MongoDB seems to store a lot of metadata for every record, which could be useful for some things, but not so much for simple IoT projects.
Really helpful explanation of the benefits of time series databases for IoT, thank you 👍👍
Glad it was helpful!
I'm a programming student and I was this kind of data retention policy saves a lot of work. Glad I don't have to reinvent the wheel.
We live in a good time with all this open source software...
Andreas did you stop numbering your videos ( #Number). The last one was #245. In my opinion, it facilitates finding specific videos and contents. I hope you continue using it. Thanks
No, but I always have about the newest 10 videos without numbers.
Thank you - I installed Grafana and InfluxDB on my existing Raspberry Pi 3+ with an already installed Node-Red and Mosquitto.
For the installation of Grafana and InfluxDB I found some instructions on the internet.
I use Node-Red and Mosquitto a lot for my home automation and used until now SQLite or flat files.
Your Video helped me to setup it - and it runs fine. Now I'm going to adapt my home automation to the new data sources.
Now you can watch the newer version (dockerized) of this setup.
Thanks Andreas, did you consider to wrap this config in Docker/Docker Compose instead SD ? It would be easier to upgrade
No, not for the moment. Maybe later...
Nice work Andreas! Thank you for putting this together, very informative.
My pleasure!
Interesting and useful video!
It seems we are heading back to remote sensing? I like that.
Sensing and acting based on these data is IOT for me ;-)
can you make a video on the topic of retention strategies, it seems to me to be a little more complicated?
Would you use this configuration, shown in the video, for everyday use?
I am not sure if this is stuff which interests a lot of people. But you find infos on the net.
Dear Andreas, great Video! Quite complex things explain so amazing short. Stuff like this can only see very rarely. Love your comments why you choose this tools. Thank You! Greets Thomas
Thank you for your kind words!
Thanks for a great video! I use Influx and Grafana professionally at work to analyze metric data from a mobile application. Thankfully I have an entire team of people to manage maintaining those systems. This kind of inspires me to see if I can hack my solar system on my roof (it uses zigbee apparently) to get data into influx and match it with real time weather so I can analyze better how my solar is performing.
At least you would have great support for your project. Not as we "simple people" ;-)
Excellent demonstration of how little you need to collect and visualise data. I have a spare PINE64 which I'd like to try using these building blocks
You have to check which operating systems the script supports if you want to use it.
Great, finally this seems to solve my IT problems for IoT. HomeAssistant is also good but seems to have an easy start with challenging future. (developing all the plugins etc. for simple additional custom features!)
So far I always used Node-Red and do not know Home Assistant.
@@AndreasSpiess For whom, who might be interested... Home Assistant (HA) has MQTT capability. An MQTT broker can direct messages to HA and HA will store them and display in a nice manner. Only by writing some making configuration files, one can trigger alerts, actions etc... e.g. I could set up a threshold for humidity sensor, after crossing this level, HA triggered a relay, which controls the fan. It is neat but extending it seems to be difficult.
you'd better warn that grafana works on RPI3 only and not on previous raspberry boards.
I did not know that.
Thanks for warning Giuseppe Suanno. I was about to do this on a zero because i happen to have one available. Even if i suspected it might need a gruntier board like PI3. Why only on Pi3? Can you please elaborate or provide source?
I am using it on orangepi lite.
I've got it running on a RasPi 2, and have also ran it on the original 512MB RasPi Model B. Granted it's slower on the 1st generation Pi, but it runs with no unreasonable difference in speed on a Pi 2 compared to a Pi 3. (which I've also run it on)
In the measurement "stations". Wouldn't it be the better option (faster for queries) to store the "name" as tag and not as field? Tags are indexed and fields aren't. Still starting with InfluxDB but that's how i understand it.
I am not a specialist in InfluxDB and my data size is small. So I never investigated.
Thank you. I've been struggling for years with mysql and an inflexible php graphing library.
This is maybe a little more flexible...
Did you set your retention policy for the weather station to do hourly averages after one year? If so, how? I am seeing it is ideally done during the database creation but can be done after. Eitherway the syntax isn't so clear to me, and I like the retention policy you mention.
So far I do not use retention policies. The RPI4 is still fast enough.
Playing with influx and grafana for computer (cpu ram ssd-cache and disk usage) and backup status monitoring. Got it working with docker containers after a few hours tinkering. Did not see the need for node red, though. Using curl for posting, and trying to get raspberry+Screenly working for display.
I like node-red because I know it ;-) there are many other possibilities to bridge from MQTT into influxDB. Did you try docker to use GPIOs or Serial on the RPi?
@@AndreasSpiess I used docker for influx and grafana, not for the data collector /sensor program. You may be able to use gpio and serial inside a container by usingal a privileged container and mounting /sys and /dev/tty*** as volumes on it.
I would like to see also a video about continuous queries and retention policies in detail with real examples related to weather (and possibly energy data). I have about three years of data every few seconds for my many weather and energy sensors but find very cumbersome to implement retention policies and continuous queries and to downsample the already existing data.
I do not see that this is a topic with lots of interested people. I think you will find some information on the net.
Your situation is perfect as you can play with your data and find out which policy fits your needs
Nice video Andreas!
In my case, instead of manual instalation, since I started using Home Assistant a few days ago, I just added InfluxDb and Grafana pluggins in Home Assistant which was like 5 minutes effort (Node Red I already added before).
Anyway your instructions helped a lot for a fast start. And a sugestion - if you didn't already - check out Home Assistant. Nice platform + combined with Mosquitto, Node Red and now InfluxDb and Grafana ... make it very interesting.
I use some manually programmed microcontrolers (bathroom humidity and fan control), Nextion displays from your video over MQTT, and dwo days ago I reprogram own application for LED strip and Sonoff Basic with esphomeyaml. Addionally using Alexa over Node Red, that fits very niceli into Home Assistant. New graphs from bathroom humidity, temperature and fan control is today nice addon.
Point is that any additional knowledge nicely fits in picture. Constant evolution...
Other commenters also mentioned that it is easy with Home assistant. Maybe I have to try it one...
Thank you for your great video. These are the projects that I can learn from a lot....
You are welcome!
Hello Andreas, does this work with an DHT22 Sensor or an Adafruit Sensirion SHT31-D? Where would I need to set the sensors up, in Node-Red?
Thank you - great video!
I usually do not connect sensors to the Raspberry because they have to be at different places. I use ESP32s and connect them with node-red via MQTT.
Thank you for the video, otherwise I have a question is it possible to integrate Grafana in a php page or is it an open source that can be easily integrated into a web page in your site?
What do you think about using the SD card long term with the amount of writing influx will be doing? Would you use an SSD if you were planning on keeping your setup long term?
No
Hi Andreas! Love this video, and thanks for putting together the image. I recently ordered a few Rasperry Pi Zero 2 Ws, and I'm trying to load the image you provided to one of them. When I load the flashed SD card and plug in the power, the device doesn't boot. No power light, no indication that it's connected to wifi (after adding wpa_supplicant.conf and SSH to the root directory). Is there any reason the image as provided wouldn't work on a raspi zero 2 W? Thanks again!
I do not know if it works on the 2W. Two things to consider:
1. The memory of the 2W is very low for such applications. I would go for at least 2G
2. Maybe you watch the video about IOTstack. There you find another way of installing all the goodies...
Awesome guide to getting Grafana up and running, thank you :)
You are welcome!
Great video, Andreas. Really enjoyed this one.
Thank you!
Hi sir. Nice tutorial, I am quite newbie here. Is there a way to export the data from the dashboard trends to excel or CSV?
Yes. Grafana has an export function. Maybe you watch my newer videos about this topic?
@@AndreasSpiess Thanks mate!. Your tutorials are one of the best in this topic
Thank you , always something to learn every week in an easy way
You are welcome!
Hi Andreas, just to share. I created a "home heating system monitor" based on some arduino-nano, reading NTCs/DS1820, and raspberry interconnected by field-bus based on modbus (RS485). Raspberry is running a python script to collect data and store it on influxbd, Grafana is used to visualize data and correlate info.
GOAL: integrate solar PV, heat pump system and Gas heat generator, control floor heating and try to be more green. ... will see...
It sounds like an interesting project. Maybe you have also planned to add something like hass.io? And maybe you also watch the video about IOTstack (if not already done)
... Yes I am looking deep in Hass.io ... but ...
before go in Strong IOT every one should read 'Abusing the Internet of Things'.
For now l prefer to stay with a well lock down reporting tools.
Hi Andreas, relational databases do not necessarily have to be setup with fields like "Temp, Pressure, ...", you can as well do an different approach and use a dedicated field to indicate the type of the value. Something like "Timestamp, Type, Value", that way the only restriction is the type of value you want to store ( integer, float, string, ... ). If you want to get completely rid of that you should have a look at elasticsearch, it's a database system that is highly scalable and very dynamic. You can easily scale it up from one DB node to hundreds in seconds, simply by adding additional nodes and the same way back (in reality there is a little downside in downsizing - let me know if you are interested). Elasticsearch does NOT stick on types, or even database table schemas. It simply stores what you send - if there is no - for example Temperature - it just creates it dynamically for you. Additionally the values are the same flexible - no need to preselect a type like float or a length, it's just dynamically created on the fly. It's even able to store >100k Messages/s depending on the machine it's running on. In reality I had soe time ago a DB running where I was feeding event log data from 9 Active Directory domain controllers and keeping ~1 year of history of that data. Peaks where around 500k messages/s and around 5-7 million entries a day on a 6 year old reused hardware. Basically a perfect match for MQTT where we do not have strict type/name rules as well.
You are right. You can store these primitive sensor values in every database without any problems. AFAIK Elasticsearch is considered to be a "NoSQL" database. They are very good at certain things like you describe: Unstructured data and also full-text search.
As I said, the strength of time-series databases is not the storage side but on the retention as well as the extraction of time-based data.
I do not want to focus on database concepts on this channel because there are people which are much more capable than me in this matter.
It's a search engine, why should you use it as a database? How do you efficiently get the average of a field (or even better over a time period)?
Types and structure shouldn't be abandoned if you don't have to deal with dynamic data.
@@RalfVogler : Elasticsearch is NOT a dumb search engine, it's a NoSQL database ! Beside that you can use several mathematical functions on fields, similar to a relational database. As MQTT as well is a typeless protocol elasticsearch would be a perfect match to store anything that would possibly arrive over MQTT. Along with Kibana it's quite easy to visualize the data, even other solutions ( including Grafana ) are possible.
The answer to your questions is here : www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-avg-aggregation.html, it can be expanded to any other critera the system is aware of ( time, name, field type, ... ).
Given the fact that we talk about sensor data to be stored, mainly through MQTT, it might be worth looking at something like Elasticsearch that could easily handle . However, it's anyway recommended to use the DBMS you prefer an feel safe with ! Additionally it of course depends on the needs and given infrastructure.
If you try to store (any) Windows eventlog data, especially from a whole domain, into a SQL database you'll understand the advantages of Elasticsearch and it's scaleability ...
I'm just bringing it to the table, it's everyone's decission to have a closer look or not. My recommendation is : If you run just a couple of sensors use what you already have. If you want to scale up to hundreds of sensors you might have a closer look to Elasticsearch.
"use a dedicated field to indicate the type of the value"
My point was that if you do that, you can't do things like average (which you are probably interested in for sensor values). Correct me if I'm wrong, I never used Elasticsearch.
Of course you can do anything. Just writing to a file is also fast. I would assume that even for storing data of hundreds of sensors the choice of database doesn't matter.
Where you will see a difference in performance is when you search in non-indexed fields, aggregate data over wrong types or use joins when the database doesn't support it.
The built-in capabilities is the other important point: does Elasticsearch support querying time series data?
Concerning types: either you declare the types for each MQTT topic or you let something like InfluxDB infer them from the first message.
thanks a lot for this research and experimentation. we will definitely use this for our farm.
I hope your application will add some value :-)
Hi @Andreas, thanks for this video. Definitely got me excited to revise my home setup. Unfortunately, I'm running into a lot of issues. I had two different RPis to run PiHole and Home Assistant. I wanted to move to an SSD to store and analyze data locally (influx, grafana). I had a MQTT broker set up in HA already and was interested to migrate this to a container. Unfortunately I can't get the MQTT addon or integration working in HA. I tried to contact the HA discord support and they were very clear this is not a supported installation method. It seems to mix the supervised/Haasio and core/docker methods. So...I'm concerned about maintainability of this method. Hoping you have some thoughts on viability here. Thanks!
Maybe you have a look at IOTstack. But I am not sure if it makes sense to mix HA with other installations, becasue you can add aeverything inside HA.
@@AndreasSpiess apologies, looks like I commented on the wrong video. Yes, I am using IOTstack. The Home Assistant community was clear this is not a supported way of installing HA. Seems a big downside to not be able to leverage such a well documented and supported community. Hoping it can evolve.
Thanks for suggesting the script from Peter Scargill. However the script did not install properly and the pi4 seems to be slow now. Is there any way to roll back the script? I rather install the tools one by one.
Maybe you watch my newer videos about IOTstack. Much easier and faster.
What another great video, thank you Andreas! Your video inspired me for changing my implementation on how to use historical data.
I hope you will be successful!
@@AndreasSpiess The InfluxDB was really fast and easy! I installed influxDB in a chrooted Linux on an Android TV box where I am using it as a mqtt broker and node-red administrator. Now I am testing the first data recording from my air conditioning energy meter. Next step to install Grafana.
And all these thanks to you again :)
Hi, great project. I want to store DHT sensor data to monogodb locally, how can i do that?
I do not know, I never used Mongodb
Hi Andreas, i've tried to use your SD card image on two different SD cards and i cant get any of them to boot, i know this post is two years old, is there an updated version of this i can try? Thank you for your work on the videos they are very interesting and i try to learn from them.
I will have a look at it.
Andreas, can you share in some way settings of nodes that you use in node-red. I try to use image you shared, but my PI is 4b and probably this is why it doesn't boot with image you shared load in my SD card.
Maybe you use the newer version using Docker. There is a video about it
Hi ,thanks for your work. Is possible to download node-red flow from this video?
It is here: drive.google.com/open?id=1u2y31cGnGtL0Acd3l0l5BwxRgtIrOqyq
I have some problem with selecting appropriate choices when running Peter's script through putty ssh. What key do I have to press when I want to select/deselect the available choices?
Use the space bar.
I suppose all the credentials should be changed to use the SD card image securely? What kind of changes should be done? Paswords for the Raspberry Pi, Node_red, influxDB and Grafana should be changed? It would be nice to have a guide how to do this and how changing the password of one app affects to another app.. in other words, how to keep the system working after passwords are changed.
You can decide what you want to change. Because the server is behind my firewall I leave them as they are.
Thank you for the hundreds of really useful, interesting, up to date tutorials and info, you're pure gold! I have a question regarding the pi node-red server side, i need to have it in a distant location and feed it with 12V batteries (car battery) and let it run node-red influx db and grafana. What would be a good way to convert 12V down to 5V for the pi? Would one of these cheap buck converters do? What do you suggest?
thank you in advance!
A buck converter with at least 3A should do it. There are many projects around if you search with Google
You're awesome, thanks for sharing this knowledge. I've been learning a lot from you.
You are welcome!
Grüezi Andreas, as always a great video with many explanation that everybody can follow and learn a lot. todays topic is very sexy, since data/process mining is the new lubricant of the world. as already told on TTN con I love your 'Mutterwitz'. cheers Michael (guy form Sulzer)
It was long overdue for me as I wanted to use it for my installation.
I hope you had a good trip home!
Very nice! The problem with the Pi for DBs is the SD. Even good SDs tend to worn out fast and you lost your valuable DB.
I would recomend to store db in other disk even at the expense of use USB disk. There are other SBC with SATA/PCIe that will be more suited this purpose.
I once did a video about that. For me so far I had no bad experience after more than a year (not Grafana,SQLite)
Cool. Great stuff Andreas.
Its all the craze doing opensauce. I wonder what made u not choose non elk and or prometheus. I suppose grafana is top for these requirements. Miss lora and esp32 a bit... a lot to be said about long range iot.
Thank you!
Is it possible to use "measurements" from different Influxdb databases in the same "panel" in Grafana? I can't figure out the syntax for the FROM field in the query editor.
I only use one database for one panel. So I do not know if you can use measurements of several databases in one panel.
@@AndreasSpiess I figured it out. grafana.com/docs/grafana/latest/features/datasources/mixed/
Its a pity the wind direction is not 360 degrees
It shows 360 degrees, but not nicely. There is a plugin which can be installed to change that. But I do not yet know how to install it :-(
Wao!!!! Great idea!!!
:-)
Hello Andreas. Do you know about any more or less procedure to have "secured":-) access to a local instance of node-red with MQTT, outside the local network with dynamic IP ? The methods i've found so far are riddled with cautionary measures such as different port allocation and the like... i'm not planning to control a high pressure boiler, but it would be nice to have a minimum sense of "security":-):-)...
I think a VPN would be the best solution. You get some free services. Maybe I make once a video about it. Usually, you need a piece of SW running on the RPI which connects to a server in the cloud. And you connect to it, too.
Thanks. I'll try with PiVPN and Google DNS and see how it performs. I've tried IBM Watson, which is a bit complicated, with a steep learning curve, and performed beautifully, but it has too many restrictions in free version.
Hey Andreas - You have a good source to learn how to use Node-Red? Oh, and thank you very much for this video... just what the doctor ordered.
I do not know of a good source. I usually search if I have a particular problem.
You can find docss with examples in nodered.org/docs/
Useful and interesting as usual 😉
:-)
Excellent video. Thanks.
Greetings from the galilee
Glad you liked it!
Happy birthday Andreas, hope you have a good one.
Yes, I had a good one. My new bike for the Danube tour arrived right that day :-)
Seems retention policy only tells the data base what old data to delete. Achieving the reduced resolution for older data requires a lot more work and down samples it in separate place. Then it is not possible to have a single graph in graphana. Or am I missing something?
I never tried it so far. Stuff for some future work ;-)
@@AndreasSpiess Indeed, this seems to be complicated, but there is a solution here pommi.nethuis.nl/collectd-influxdb-grafana-with-downsampling/ I will see where this leads..
Thank you for the link!
@@AndreasSpiess Unfortunately it only works, as the author says, when the end of plotted time is always now. On top of this, it does not take too much query to kill the rpi. it runs out of RAM fast. This kills mosquitto and node red for a while.
issue is still open github.com/grafana/grafana/issues/4262
Great video and very timely for me as I am looking to complement my old RRD based graphics site with Grafana and Infuxdb.
I have used Pete’s ( and Mr Shark’s) script along with his node-red big timer for years!
While I do enjoy node red, I am looking to lower my dependence on it. Thus hints of ways to direct mqtt data directly to influxdb are high on my curiosity list!!
A other viewer commented he is using python scripts instead of Node-red
github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer
Are the same results possible if I have multiple Pis spread further out? I assume there'll be more setup and configuration bit I'm just dabbling into this for the first time. This is really close to what I want to do to monitor my property with multiple raspberry pis scattered about with sensors monitoring propane level, power generation ect.
I do not know why you would use more Raspberries. It complicates the setup. If one Raspberry is to weak you can change to a powerful PC with the same tools.
If you want to have multiple installations you can watch my video #295 (which uses Docker) and Google for Kubernetes.
@@AndreasSpiess mainly cause I don't want to run sensor wires for hundreds of feet that's allot of wire and probably signal degradation. I want to put a Pi down by the propane tanks, one about a mile away at the well and one near my battery bank. All these things are far apart from each other. I want to do exactly this but through multiple Pis. One Pi just wouldn't be practical considering the physical distance.
I understand. For sensors I propose not to use raspberries. Small MCUs are better fitted for that purpose (cheaper, smaller, less power consumption, less maintenance).
Maybe you watch also my LoRa videos if you need to bridge longer distances without wires. For small distances, Wi-Fi and ESP32 boards are a good choice. Maybe you google "farm relay system". (TH-cam deletes links)
@@AndreasSpiess thankfully I have really reliable wifi across my whole property off my neighbors wind turbine (don't worry he's aware incase you were wondering lol) time to drive deep back into this rabbit hole again, thanks for the replies!! This is the first video I've seen Grafana in and I'm sold on that already. Your videos have given me tons of starting points too!
Many Thanks Andreas - Well Done and hope to see more
You are welcome!
Andreas. I downloaded your SD Card image and ran on my pi. Using your passwords as given at the end of the video, none of them work for me, in particular, the log-in credentials for the Raspian System are not accepted??. Alan
Raspbian uses user pi and password raspberry.
Very good! Is there a connector for TTN? I have LoRa Explorer device.
You can connect to TTN via MQTT connector in Node-Red
@@AndreasSpiess one more question pls: in the SD ready image is there Raspbian and Peter's script besides Node-Red, Grafana..?
Yes
@@AndreasSpiess ok so i just need to burn with win32 Disk Imager the Grafan.img file, right?
Yes
Really inspiring ideas, thank you! I'm thinking to move my solar panel yield visualization to an influxdb and grafana stack.
Good idea!
Hello Andreas! Thanks for the great video. I have already tried Pete's Script some time ago and am using node red for over 2 years knnow! Worked great, so i'm doing a fresh install now on the latest Raspberry and I am adding Influx DB now for the first time. Iw wanna try to store and read power consumption data that I am transmittting from an ESP8266 in my fuse box. Have you tried anything like that yet?
I made a video and an image is there for download. Today I worked on the alarming from Grafana via Telegram to the mobile.
Hi, thank you for the video, I have a question regarding influxDB and Grafana, I want to use InfluxDb and Grafana not for IoT purpose but to a bi purpose, I have an app that logs events of user action and I want to create some dashboard of analytics of those events, will Grafana and influxDB are the right tools, and if yes, My logs will be kept on a device that doesn't have internet connection all the time, so the logs will upload to the influxDB in a later time, how can I use the event timestamp and not the timestamp of when I'm adding the log to influxDB? thank you in adavance
I do not know. But maybe you google to find the solution for your particular problem?
Which weather station do you use?
There is a video abou 433 MHz. There you should find the link and how to read it.
@@AndreasSpiess thanx, don't know why I didn't get notification that you replied to my message though...
Just wondering, why do you use nodered when you can program the weather station to send the data to influxdb directly?
Derkades weather station is in a remote location only equipped with sensors, as it should be. RPi is the central server station.
I like node-red. And it is anyway running in my server. But you can do it otherwise, of course.
Fantastic job!
Just what I was needing for my project.
Thank you!
You are welcome!