Relabeling in Prometheus | Relabeling Architecture and Flow, Configuration, Examples, Debugging

In this video, I explain the concept of relabeling in Prometheus, a powerful tool for filtering and transforming your flows of labeled objects in the Prometheus server. I go into the motivation for relabeling, explain how relabeling fits into the main Prometheus configuration file, and then show how the relabeling steps and flow works. I also discuss the structure of a single relabeling rule, give several examples of relabeling rules, as well as show you how to debug your relabeling rules.
Learn about all relabeling actions in my free relabeling training:
training.promlabs.com/trainin...
Use the PromLabs Relabeler tool to visualize and debug your relabeling rules:
relabeler.promlabs.com/
Also check out my other Prometheus training courses if you want to learn Prometheus in a structured way from the ground up:
training.promlabs.com/
Chapters:
00:00 Introduction
00:37 Motivation for relabeling
02:04 Relabeling in the Prometheus configuration file
03:09 Relabeling steps and flow
03:50 Relabeling rule structure and fields
04:44 Target metadata labels
06:07 The relabeling visualizer tool
06:32 Example 1: Setting a fixed label value
07:01 Example 2: Overriding the scrape port
08:04 Example 3: Mapping over label patterns
09:01 Example 4: Dropping scraped samples
09:35 Debugging relabeling rules
10:31 Outro & PromLabs Trainings
---------------------------------------------------------------------------
CREDITS: "Subscribe Button" by MrNumber112 • Free Download: Subscri...

[@ivivasal]

I just finished watching the video series and would like to congratulate you. They are awesome. Especially the animations to explain how prometheus works. Really brilliant. Thank you very much for your effort.

[@avr4you]

Thanks for this video. Very clear and helpful for the start in relabeling.

[@ricardohincapie1537]

Thanks. So useful!

[@csabadobo]

Hi, may I ask for some help?
I am using a kube-prometheus-stack. Federating 2 remote prometheus alerts into a central prometheus. I use externalLabels and they appear just fine. Problem is, the metrics/ alerts do not have on central prometheus the set externaLabels. How to fix that?

[@yerramsettivijay5194]

Thanks for the overview I want to add condition so that alerts should be dropped before reaching Alertmanager. The alert should visible in Prometheus but not in alert manager is it possible?
Thank you

[@PromLabs]

Yes, if you do it via alert relabeling, you could drop those alerts and they will only be visible on the Prometheus side and not in the Alertmanager. However, it's more common to still send all alerts to the Alertmanager, but set some label (like severity="info") and then just configure Alertmanager to not route them to anywhere urgent.

[@yerramsettivijay5194]

@@PromLabsthank you I have added like this to prom configuration (Prometheus.yml) but I can still see that alert in alert manager
alerting:
alert_relabel_configs:
- action: drop
source_labels: [severity]
Regex: “info”
alertmanagers:
- static_configs:
- targets:
# - alertmanager: 9093

[@EricOnYouTube]

Do you have a video that can help me with this: I have prometheus scraping data from 3 different Raspberry Pis. I want to show to visualize the data. How do I go about doing that for data that is very similar but sourced from different Pis.

[@PromLabs]

If you are looking for host metrics, take a look at my Node Exporter video, especially the part at the end where I import the Node Exporter dashboard into Grafana, which then lets you select a host to show metrics for: th-cam.com/video/dcb59H_iLj4/w-d-xo.html. If it's other metrics, the approach would be similar in terms of using template variables in Grafana to allow you to select the specific host / IP to show metrics for, so you only need to create one dashboard for all hosts / Pis. You can also find more info on that in my Grafana training at training.promlabs.com/.

[@mariarti7459]

Is there any performance drawbacks when using relabeling?

[@PromLabs]

For target relabeling, the impact is usually negligible since targets are not rediscovered / recreated at super high frequency. For metric relabeling there can be a bigger hit since the relabeling rules are applied to every sample coming from a target (which might be a lot) on every scrape. Still, you'd have to measure how impactful that really is. Similar for remote write and alert relabeling.

[@madhand6045]

make a video about json exporter please and make more demos using various exporters

[@Draxx1206]

Hi sir. I have a question. I have successfully installed Prometheus, and it is running on port 9090. After that, I also successfully installed Grafana, but when I add Prometheus as a datasource in Grafana, it cannot be found. What could be the issue? Thank you very much.

[@PromLabs]

Hi! For concrete support questions like that, I recommend using the open community channels (see prometheus.io/community/). There, many people can help and benefit from the answers, and it's also easier to give enough relevant context to a question. E.g. I'm not sure what "it cannot be found" means - do you mean that Grafana reports a connection error when you try entering localhost:9090/ as the Prometheus datasource URL? In case you are running Grafana in a Docker container, you may have to specify the host's IP (not localhost), since localhost within the Grafana Docker container will be namespaced and separate from the host where Prometheus is running.

[@Draxx1206]

@@PromLabs Thank you sir, for fixing that issue. Could you please create more videos about Prometheus + Grafana + Alertmanager? I think it would be very helpful. Thank you so much.