Thanks for your outstanding work. I was searching for Monitoring Proxmon with Prometheus and found your "Monitor Proxmox VE With Prometheus And Grafana". I really appreciate your practice of referencing the prerequisite videos at the start. I worked my way back through the chain to installing Docker in a VM under Proxmox. These were all very clear and informative. NOW I'M AN EXPERT! (ahem....) Thanks again for your fine work!
Extremely useful, detailed and well explained tutorial, thank you so much. Finally somebody that explained each and every point understandably for a beginner
Thanks for the feedback I'm finding these to be a really good combination Definitely useful to be able to run them as containers And there's so much flexibility thanks to the community
Useful breakdown of the steps involved. I did have a go at this about a year ago and made a bit of a pigs ear of the process. Think I'll be more successful this time following your guide :-)
I'm really liking this combination There are a lot of exporters/agents available so I've got it monitoring my Proxmox cluster, TLS certificate expiry, etc But running it on Docker makes it so much easier to backup and a restore would take next to no time
The folder location is up to you But, I have a dedicated user account for Docker, so I run everything from that user's home folder to make sure that user has access to files I also use Docker Compose and so the YAML file for that goes into the home folder Within that home folder, each container has it's own sub folders to store configuration files in
@@TechTutorialsDavidMcKone Thanks for the reply, I've only messed about with Docker a little and having a dedicated account seems like a good idea as i had issues with permissions previously and that would help.
Q: So when i use node-exporter to scrape matrics , anyone knowing my node IP could see metrics, right. Is there any safer way to scrape matrics so it won't be open to all. Correct me if I'am wrong :)
@@AshutoshAtnoor You should be blocking access to any service on a computer by using a personal firewall Although access to servers should be blocked by placing them behind a dedicated firewall anyway Because I'm running Docker for instance in a VM in Proxmox VE, the hypervisor firewall is blocking access If node exporter was running on a Linux computer somewhere else I would use something like Uncomplicated Firewall (UFW) to do that instead
Prometheus scrapes data from exporters and node exporter is an exporter that's being used to gather metrics from the host computer It makes sense to get an exporter up and running first and make sure it's working before configuring Prometheus There's nothing to stop you configuring Prometheus first, but it won't be able to do anything until the exporter is ready
Are you running this in Linux or Windows? I ask because I've seen problems reported in forums when using Windows This is because the mount path is for a Linux host
@@TechTutorialsDavidMcKone Actally i rectify my error but now i am dealing with in prometheus my node-exporter state is down i am getting this error Get "192.168.8.53:9100/metrics": context deadline exceeded. can you please help me ?
My OCD really hates Docker Compose. Having 500 folders with just a single yaml file in them drives me up the wall, I wish we could normalize just using Docker run.
Tempting but I've come to find that one big file gets harder and harder to maintain over time Maybe one day I can just leave this all to an AI to manage...
@@TechTutorialsDavidMcKone That's part of the reason I prefer using Docker run. Similar to Compose, I use individual bash scripts to run each container. The scripts take down any existing container with that name, pull the latest image, and spin it back up again. Plus, this allows me to organize my scripts however I like without having a ton of subfolders for each one. I can then schedule them in Cron and easily update my images regularly automatically. Because of my use case, Run just feels more flexible. I could accomplish the same with Compose, but the process just feels more....clunky. Great tutorial by the way!
Hi David, very clear tutorial, thanks! I have one question, do you know how to setup prometheus so that it monitors the containers of the hosts? (how many are running, how many are stopped, etc.) I could not find information that works on the net... :(
I only monitor the VM I run this on as a whole as I don't apply quotas for instance But if you want more details on the containers, then the suggestion is to use cAdvisor prometheus.io/docs/guides/cadvisor/ Have a look for a Grafana dashboard that suits your needs, but I did a quick search and found this one grafana.com/grafana/dashboards/13946-docker-cadvisor/
¿Estás usando Ubuntu? No tengo conocimiento de ningún problema con node-exporter, pero solo lo he estado ejecutando en Debian Puede comprobar los registros en busca de posibles errores mediante un comando como este Docker container logs node-exporter Porque me pregunto si la computadora está bloqueando el acceso para el exportador de nodos Probablemente esté usando la IP correcta, pero le pediría que lo verifique nuevamente, ya que la que uso solo funcionaría para mi computadora No sé si ya ha revisado la publicación del blog, pero todas las configuraciones se muestran allí si necesita verificarlas dos veces www.techtutorials.tv/sections/management-and-monitoring/install-prometheus-and-grafana/
@@TechTutorialsDavidMcKone Ya e visto donde me falla , cuando tu hacer el docker compose up -d , se te levanta el docker prometheus con un puerto y a mi se me levanta pero sin el puerto 9090 .
@@santicapellalorenzo2936 Intente verificar si algo más está usando el puerto 9090 sudo ss-tunlp En el medio debería haber una columna, Dirección local:Puerto. Eso muestra en qué puertos se están escuchando. Es posible que tengas que ampliar el terminal, pero la columna de la derecha llamada Proceso muestra qué proceso está usando ese puerto.
Good to know you've resolved this I keep forgetting about UFW myself and I was going to suggest that amongst other things But as frustrating as it can be, it's still necessary
After hours of struggling through these steps, I cannot get past the fact that node exporter is not accessible no matter what port I set. If I try a curl, it fails. Prometheus runs fine but node exporter never gets a connection. Surely Im not the only person with this issue?
I think you'll want to check the log files to try and find out what's going wrong For example docker container logs node_exporter Maybe there's another application or container using the same port You should be able to check using this command sudo ss -tunlp In my case I can see "node_exporter" listening on TCP port 9100
@@TechTutorialsDavidMcKone Thank you for the reply David. I’ll try these steps in the morning although I did try changing the port earlier abs that didn’t work so I suspect it’s something else. Let me just clarify that my frustration is at Docker, not your tutorial. The steps you provided were clear, and the tutorial was great.
@@TechTutorialsDavidMcKone It is listening on port 9100 but its not connecting. I am on Windows btw. I can get node exporter to run if I run it outside of the container (not in with prometheus and grafana) but as soon as its in that container, it is never accessible
So I am using portainer. I am trying to create a stack for this. I paste the first one node exporter. That one worked fine. When I tried to paste prometheus, I get this error: "failed to deploy a stack: Network monitoring_default Creating Network monitoring_default Created Volume "monitoring_prometheus-data" Creating Volume "monitoring_prometheus-data" Created Container prometheus Creating Container node_exporter Running Container prometheus Created Container prometheus Starting Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/data/compose/29/prometheus/prometheus.yml" to rootfs at "/etc/prometheus/prometheus.yml": mount /data/compose/29/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type"
I've barely touched Portainer as it's GUI based and I prefer to use docker compose I don't know what the compatibility is for importing but it sounds like either the config file for prometheus is missing or the path needs correcting because it's looking for prometheus.yml and it can't find it My docker compose file was based on the file prometheus/prometheus.yml with an expectation this would be accessible via the current working folder, hence the preceding ./ in the line In your case it seems to be beginning its search from /data/compose/29/ So I'd suggest looking there
@@TechTutorialsDavidMcKone I got it to work in portainer stack but had to use this instead: volumes: - /etc/prometheus:/etc/prometheus - prometheus-data:/prometheus
Port 9100 respond and have data, 9090 garter information and display it, Grafana make connection with Prometheus but no data is displayed. When I enter in Grafana container and curl prometheus:9090/metrics data appears
What response did you get when you setup the data source in Grafana? When you first install Grafana you have to pick a data source, so in this case it would be Prometheus You then have to enter the URL for that e.g. prometheus:9090 The you click Save and Test and it should give you a status check If it's not working you'll want to double the URL is correct You can also check the logs of the container for more details E.g. docker container logs grafana If it's not reporting any problems then maybe there is issue with the Dashboard Again check the logs, otherwise I would check the forum for possible causes
Thanks for your outstanding work. I was searching for Monitoring Proxmon with Prometheus and found your "Monitor Proxmox VE With Prometheus And Grafana". I really appreciate your practice of referencing the prerequisite videos at the start. I worked my way back through the chain to installing Docker in a VM under Proxmox. These were all very clear and informative. NOW I'M AN EXPERT! (ahem....)
Thanks again for your fine work!
Thanks for the feedback
Good to know the video was helpful
Extremely useful, detailed and well explained tutorial, thank you so much. Finally somebody that explained each and every point understandably for a beginner
Thanks for taking the time to comment
Good to know the video was helpful
Extremely clear, great stuff! Finally my homelab will have monitoring beyond the Proxmox summary.
Thanks for the feedback, and good to know the video was helpful
Excellent tutorial David. Thank you for making it. I loved the step by step checks on all the components
Thanks for the feedback and good to know the video was helpful
I absolutely agree. David does a fantastic job. His work is a benchmark for others to work towards.
The best tutorial on the subject!
Thanks for the feedback
I'm finding these to be a really good combination
Definitely useful to be able to run them as containers
And there's so much flexibility thanks to the community
Useful breakdown of the steps involved. I did have a go at this about a year ago and made a bit of a pigs ear of the process. Think I'll be more successful this time following your guide :-)
Good to know the video was useful
Best tutorial I was able to find for this topic. Very easy to follow.
Thanks for the feedback, always appreciated
And good to know the video was useful
Appreciate the clear instructions, sir! Well done, keep rocking!
Glad you liked the video and thanks for the feedback
Thanks
Thanks for supporting the channel
Thanks for explaining what 'unless-stopped' means 4 times😅
Great video👍
Wow, that many times
Thank you so much sr.!! And hugs from Brazil!
It's a very useful combination
Good to hear the video was helpful
Excellent tutorial David. Thank you for making it.
Thanks for the feedback and good to know the video was helpful
That's David, much appreciated, no doubt I'll get to in at some point in my homelab.
I'm really liking this combination
There are a lot of exporters/agents available so I've got it monitoring my Proxmox cluster, TLS certificate expiry, etc
But running it on Docker makes it so much easier to backup and a restore would take next to no time
Thank you for the excellent video. Calm, Clear and Detail Oriented. Love it 🙏
Thanks for the feedback, I really appreciate it
Followed this and it worked a treat, I was wondering about the folder location "mkdir prometheus" and if it should be in my home or not. Thanks
The folder location is up to you
But, I have a dedicated user account for Docker, so I run everything from that user's home folder to make sure that user has access to files
I also use Docker Compose and so the YAML file for that goes into the home folder
Within that home folder, each container has it's own sub folders to store configuration files in
@@TechTutorialsDavidMcKone Thanks for the reply, I've only messed about with Docker a little and having a dedicated account seems like a good idea as i had issues with permissions previously and that would help.
Thanks David!
Thanks for taking the time to leave a comment
Q: So when i use node-exporter to scrape matrics , anyone knowing my node IP could see metrics, right. Is there any safer way to scrape matrics so it won't be open to all.
Correct me if I'am wrong :)
@@AshutoshAtnoor You should be blocking access to any service on a computer by using a personal firewall
Although access to servers should be blocked by placing them behind a dedicated firewall anyway
Because I'm running Docker for instance in a VM in Proxmox VE, the hypervisor firewall is blocking access
If node exporter was running on a Linux computer somewhere else I would use something like Uncomplicated Firewall (UFW) to do that instead
could you please make a video about , monitor nginx logs in grafana with prometheus
I'll add it to my list
@@TechTutorialsDavidMcKone thank you man 👍🏻
Hey nice video, I am new to Dockers. Just want to ask you why are we installing node exportor first and not Prometheus?
Prometheus scrapes data from exporters and node exporter is an exporter that's being used to gather metrics from the host computer
It makes sense to get an exporter up and running first and make sure it's working before configuring Prometheus
There's nothing to stop you configuring Prometheus first, but it won't be able to do anything until the exporter is ready
@@TechTutorialsDavidMcKone thanks, it all makes sense now. Really appreciate your quick response and thank you for the awesome video tutorials
Error response from daemon: path / is mounted on/ but it is not a shared or slave mount ? Anyone please clarify this error
Are you running this in Linux or Windows?
I ask because I've seen problems reported in forums when using Windows
This is because the mount path is for a Linux host
@@TechTutorialsDavidMcKone i am using linux ubuntu 22.04
can you please help me to resolve this error
@@GowriAyyanar-el3fn I suggest checking on the forum as there are various responses and it seems to depend on various things
@@TechTutorialsDavidMcKone Actally i rectify my error but now i am dealing with in prometheus my node-exporter state is down i am getting this error Get "192.168.8.53:9100/metrics": context deadline exceeded. can you please help me ?
My OCD really hates Docker Compose. Having 500 folders with just a single yaml file in them drives me up the wall, I wish we could normalize just using Docker run.
Tempting but I've come to find that one big file gets harder and harder to maintain over time
Maybe one day I can just leave this all to an AI to manage...
@@TechTutorialsDavidMcKone That's part of the reason I prefer using Docker run. Similar to Compose, I use individual bash scripts to run each container. The scripts take down any existing container with that name, pull the latest image, and spin it back up again. Plus, this allows me to organize my scripts however I like without having a ton of subfolders for each one. I can then schedule them in Cron and easily update my images regularly automatically. Because of my use case, Run just feels more flexible. I could accomplish the same with Compose, but the process just feels more....clunky. Great tutorial by the way!
Have a makefile and create a "run" command to cd into each directory and run docker compose on appropriate files.
Hi David, very clear tutorial, thanks! I have one question, do you know how to setup prometheus so that it monitors the containers of the hosts? (how many are running, how many are stopped, etc.) I could not find information that works on the net... :(
I only monitor the VM I run this on as a whole as I don't apply quotas for instance
But if you want more details on the containers, then the suggestion is to use cAdvisor
prometheus.io/docs/guides/cadvisor/
Have a look for a Grafana dashboard that suits your needs, but I did a quick search and found this one
grafana.com/grafana/dashboards/13946-docker-cadvisor/
@@TechTutorialsDavidMcKone Thanks David! I saw cAdvisor, I thought that maybe there were a way to monitor the containers without this.
Great video yay!
Good to know the video was helpful
Thanks for the feedback
He seguido los pasos pero cuando llego al apartado de prometehus, me da error de ip y api , y no me deja ver la pantalla del puerto 9100
¿Estás usando Ubuntu? No tengo conocimiento de ningún problema con node-exporter, pero solo lo he estado ejecutando en Debian Puede comprobar los registros en busca de posibles errores mediante un comando como este Docker container logs node-exporter Porque me pregunto si la computadora está bloqueando el acceso para el exportador de nodos Probablemente esté usando la IP correcta, pero le pediría que lo verifique nuevamente, ya que la que uso solo funcionaría para mi computadora No sé si ya ha revisado la publicación del blog, pero todas las configuraciones se muestran allí si necesita verificarlas dos veces
www.techtutorials.tv/sections/management-and-monitoring/install-prometheus-and-grafana/
@@TechTutorialsDavidMcKone Ya e visto donde me falla , cuando tu hacer el docker compose up -d , se te levanta el docker prometheus con un puerto y a mi se me levanta pero sin el puerto 9090
.
@@santicapellalorenzo2936 Intente verificar si algo más está usando el puerto 9090
sudo ss-tunlp
En el medio debería haber una columna, Dirección local:Puerto.
Eso muestra en qué puertos se están escuchando.
Es posible que tengas que ampliar el terminal, pero la columna de la derecha llamada Proceso muestra qué proceso está usando ese puerto.
I can not get node Exporter working. Prometheus target is up. Node exporter is down...
I figured it out. I had to open ports in ubuntu firewall even though those ports were already open in pfsense
Good to know you've resolved this
I keep forgetting about UFW myself and I was going to suggest that amongst other things
But as frustrating as it can be, it's still necessary
After hours of struggling through these steps, I cannot get past the fact that node exporter is not accessible no matter what port I set. If I try a curl, it fails. Prometheus runs fine but node exporter never gets a connection. Surely Im not the only person with this issue?
I think you'll want to check the log files to try and find out what's going wrong
For example
docker container logs node_exporter
Maybe there's another application or container using the same port
You should be able to check using this command
sudo ss -tunlp
In my case I can see "node_exporter" listening on TCP port 9100
@@TechTutorialsDavidMcKone Thank you for the reply David. I’ll try these steps in the morning although I did try changing the port earlier abs that didn’t work so I suspect it’s something else.
Let me just clarify that my frustration is at Docker, not your tutorial. The steps you provided were clear, and the tutorial was great.
@@TechTutorialsDavidMcKone It is listening on port 9100 but its not connecting. I am on Windows btw. I can get node exporter to run if I run it outside of the container (not in with prometheus and grafana) but as soon as its in that container, it is never accessible
@@babababa201Well the node exporter is for *NIX systems so it won't probably do anything for Windows
For that you'll want the Windows exporter
Uhhh....Royalty!
+1
I must admit I've been having fun with this
The combination of these two is really useful
So I am using portainer. I am trying to create a stack for this. I paste the first one node exporter. That one worked fine. When I tried to paste prometheus, I get this error:
"failed to deploy a stack: Network monitoring_default Creating Network monitoring_default Created Volume "monitoring_prometheus-data" Creating Volume "monitoring_prometheus-data" Created Container prometheus Creating Container node_exporter Running Container prometheus Created Container prometheus Starting Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/data/compose/29/prometheus/prometheus.yml" to rootfs at "/etc/prometheus/prometheus.yml": mount /data/compose/29/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type"
I've barely touched Portainer as it's GUI based and I prefer to use docker compose
I don't know what the compatibility is for importing but it sounds like either the config file for prometheus is missing or the path needs correcting because it's looking for prometheus.yml and it can't find it
My docker compose file was based on the file prometheus/prometheus.yml with an expectation this would be accessible via the current working folder, hence the preceding ./ in the line
In your case it seems to be beginning its search from /data/compose/29/
So I'd suggest looking there
@@TechTutorialsDavidMcKone I got it to work in portainer stack but had to use this instead: volumes:
- /etc/prometheus:/etc/prometheus
- prometheus-data:/prometheus
@@jcmichel5768 Good to know, its working thanks for sharing
Port 9100 respond and have data, 9090 garter information and display it, Grafana make connection with Prometheus but no data is displayed. When I enter in Grafana container and curl prometheus:9090/metrics data appears
What response did you get when you setup the data source in Grafana?
When you first install Grafana you have to pick a data source, so in this case it would be Prometheus
You then have to enter the URL for that e.g. prometheus:9090
The you click Save and Test and it should give you a status check
If it's not working you'll want to double the URL is correct
You can also check the logs of the container for more details
E.g. docker container logs grafana
If it's not reporting any problems then maybe there is issue with the Dashboard
Again check the logs, otherwise I would check the forum for possible causes
@@TechTutorialsDavidMcKone It makes connection and lights 2 green ticks :) upper right corner and bottom left