IMPORTANT: I made a mistake in the video by saying that AlertManager is querying Prometheus. That's incorrect. It's the other way around. Prometheus is evaluating the rules and sending alerts to AlertManager which, in turn, is forwarding them to final destinations like Slack, email, etc. What do you use for monitoring, logging, and alerting? What's your favorite stack?
I highly recommend kube-prometheus-stack, all-in-one helm chart to deploy prometheus, grafana and alertmanager, each with its own operators. So instead of pre-defining things in values.yaml, you can use CRs to define targets, rules, alerts, dashboards, datasources, etc - in a Kubernetes way. For the logging part, I found banzaiclud's logging-operator to be very interesting, again a way to simplify the deployment of software for logging collection, aggregation and shipment (Loki being just one possible destination). It is also built around an operator and deploys instances of fluentd and fluentbit.
It would be interesting to see an example using opentelemetry to gather the observability data (avoding agents vendor lock-in) and use the otel pipelines to expose the data to different vendor solutions.
Great video!!! Observability is so important and allow a lot off evolution not yet explored today I have the mnemonic word AMLET for alerting, monitoring, logging, eventing (context and others) and tracing I think also that grafana is the de facto place to have all data to observe even as a saas) thanks to tempo and loki we can add more meaning to metrics dashboards (and I have a small preference for sensu go over robusta to serve as a glue around all that) and leverage all that with a runbook system for auto remediation (stackstorm, awx, ansible platform, jenkins, rundeck....). The dream!
Would be interesting to have a deeper dive, things like Thanos, Tempo, Mimir, etc. Also, what do you think of using their jsonnet libraries to manage those? I found the community helm charts to be not that well maintained and jsonnet is actually pretty flexible for an enterprise setup
If you are serious about monitoring, you need to setup your own monitoring system even on managed kubernetes like EKS, GKE and AKS. I hope you will take the topic of monitoring further with introduction of Prometheus Operator, Grafana Cloud Agent (and GCA Operator), Grafana Operator and perhaps also Grafana Tempo. I would also love to see separate video about VictoriaMetrics that is much better than Prometheus itself.
This could not have come at a better time! Looking forward to part2 with tracing, open telemetry etc. and maybe also cover the maintenance aspects. Prometheus does automatic data purging which makes it maintenance free; how does loki compare with it. With logs the data volumes are going larger and much more workload dependent so one could easily overwhelm the system. Plus some organizations may need log archives to be kept for several years, how loki supports that use case would be interesting to see. My organization uses elastic search. Can loki be a replacement for elastic search today, or in future? The reason I would prefer loki over elastic is because I can co-relate logs with metrics, events and maybe even traces. In case of java/spring boot based apps, tracing can be very simple to achieve with auto-instrumentation. This would provide great visibility into the working of the application. I am myself exploring it this week.
Adding those to my TODO list... :) In the meantime, you might want to join th-cam.com/video/Jqe7kkXZzCo/w-d-xo.html and ask those questions there as well.
Great stuff, but the in my opinion the really tricky part is managing these things at scale. First of all there is the storage aspect, but also Prometheus seems to breakdown when the cluster gets too big. At that point you either need to use a federated setup or something else and it would be useful to hear your thoughts on that.
You're right. When running at scale, a single Prometheus does not work (it cannot scale). I already have it on my TODO list to tackle that subject in one of the upcoming videos.
After few mo later Grafana stack extended make more flexible Grafana tempo + open telemetry for auto instrument + Grafana agent Grafana loki Prometheus Grafana Alertmanger Basically included Metrics , log, apm/tracing and alert Also Grafana able to adding silence by UI so we don’t need expose Prometheus alertmanger to make alert mute
I concur with others that Tracing is conspicuously absent, as is OpenTelemetry (OTEL), which is the emerging standard that ties all these CNCF pieces together with others such as Fluent Bit
You're right. I got complaints in the past that the videos were too long so I started making shorter ones and that often results in things missing. I'll work on a second part of that video with tracing and open. telemetry
I finally got around making a video about OpenTelemetry (I wanted to explore it separately first). It's available at th-cam.com/video/oe5YYh9mhzw/w-d-xo.html. Tracing is coming next.
@@DevOpsToolkit Thanks. I mean more on the storage side. Like if I want to move away from AWS open search to decrease cost. Are there any alternatives than Loki?
@@DevOpsToolkit My pleasure! Can’t help but contributing a bit as I’m binge watching your videos. Plus, it was a pleasure collaborating with you on Geekle’s conferences. Keep up the great work!
As far as I know, there is no SSO in Grafana open-source version (not sure about Enterprise). I would probably try something like Teleport (you'll find a video in this channel).
Grafana is a great tools but somehow like Jenkins If the plug-in no longer maintain then you may get trouble it have chance may your Grafana unable to perform upgrade otherwise graph will dead But I do love the trend they go for Loki+Tempo+Display Prometheus Love to see if there have tools to generate dashboard and integrated 3 of them Do you have plan or already have video for Tempo?
IMPORTANT: I made a mistake in the video by saying that AlertManager is querying Prometheus. That's incorrect. It's the other way around. Prometheus is evaluating the rules and sending alerts to AlertManager which, in turn, is forwarding them to final destinations like Slack, email, etc.
What do you use for monitoring, logging, and alerting? What's your favorite stack?
Prometheus, Loki, alert manager, grafana, slack
pin this comment!! great video, loving it
I highly recommend kube-prometheus-stack, all-in-one helm chart to deploy prometheus, grafana and alertmanager, each with its own operators. So instead of pre-defining things in values.yaml, you can use CRs to define targets, rules, alerts, dashboards, datasources, etc - in a Kubernetes way. For the logging part, I found banzaiclud's logging-operator to be very interesting, again a way to simplify the deployment of software for logging collection, aggregation and shipment (Loki being just one possible destination). It is also built around an operator and deploys instances of fluentd and fluentbit.
I think we should include Tracing here. It can be Jaeger, Temp or something else. And all those thing should be standardized by OpenTelemetry.
You're right. I should have added tracing to that video. I'll work on making a follow up with tracing
Here's Jaeger: th-cam.com/video/FK0uh-7nDSg/w-d-xo.html
It would be interesting to see an example using opentelemetry to gather the observability data (avoding agents vendor lock-in) and use the otel pipelines to expose the data to different vendor solutions.
Adding it to my TODO list... :)
Great video!!! Observability is so important and allow a lot off evolution not yet explored today
I have the mnemonic word AMLET for alerting, monitoring, logging, eventing (context and others) and tracing
I think also that grafana is the de facto place to have all data to observe even as a saas)
thanks to tempo and loki we can add more meaning to metrics dashboards (and I have a small preference for sensu go over robusta to serve as a glue around all that) and leverage all that with a runbook system for auto remediation (stackstorm, awx, ansible platform, jenkins, rundeck....). The dream!
Would be interesting to have a deeper dive, things like Thanos, Tempo, Mimir, etc. Also, what do you think of using their jsonnet libraries to manage those? I found the community helm charts to be not that well maintained and jsonnet is actually pretty flexible for an enterprise setup
I have Tanos on my TODO list. Adding tempo and Mimir as well... :)
I second that request for tempo. Great video as always viktor.
I suggest you also add the alert example just like you do with querying. Otherwise great video enjoyed it 👍🏽
If you are serious about monitoring, you need to setup your own monitoring system even on managed kubernetes like EKS, GKE and AKS. I hope you will take the topic of monitoring further with introduction of Prometheus Operator, Grafana Cloud Agent (and GCA Operator), Grafana Operator and perhaps also Grafana Tempo. I would also love to see separate video about VictoriaMetrics that is much better than Prometheus itself.
as ever you are rigth, i try loki and wooow! woks perfect with grafana thanks a lot genius!
This could not have come at a better time! Looking forward to part2 with tracing, open telemetry etc. and maybe also cover the maintenance aspects. Prometheus does automatic data purging which makes it maintenance free; how does loki compare with it. With logs the data volumes are going larger and much more workload dependent so one could easily overwhelm the system. Plus some organizations may need log archives to be kept for several years, how loki supports that use case would be interesting to see. My organization uses elastic search. Can loki be a replacement for elastic search today, or in future? The reason I would prefer loki over elastic is because I can co-relate logs with metrics, events and maybe even traces.
In case of java/spring boot based apps, tracing can be very simple to achieve with auto-instrumentation. This would provide great visibility into the working of the application. I am myself exploring it this week.
Adding those to my TODO list... :)
In the meantime, you might want to join th-cam.com/video/Jqe7kkXZzCo/w-d-xo.html and ask those questions there as well.
@@DevOpsToolkit Sure, added a reminder
OpenTelemetry is finally finished and available at th-cam.com/video/oe5YYh9mhzw/w-d-xo.html. Tracing is coming next.
Great stuff, but the in my opinion the really tricky part is managing these things at scale. First of all there is the storage aspect, but also Prometheus seems to breakdown when the cluster gets too big. At that point you either need to use a federated setup or something else and it would be useful to hear your thoughts on that.
You're right. When running at scale, a single Prometheus does not work (it cannot scale). I already have it on my TODO list to tackle that subject in one of the upcoming videos.
@@DevOpsToolkit amazing! Very much looking forward to that
After few mo later
Grafana stack extended make more flexible
Grafana tempo + open telemetry for auto instrument + Grafana agent
Grafana loki
Prometheus
Grafana Alertmanger
Basically included
Metrics , log, apm/tracing and alert
Also Grafana able to adding silence by UI so we don’t need expose Prometheus alertmanger to make alert mute
In the latest versions grafana also shows the alert manager alerts and can be silenced from there too (bell icon)
I was not aware of that. That's great news.
MELT stack = Monitoring, Event (alerting +OnCall), Logging and Tracing
I concur with others that Tracing is conspicuously absent, as is OpenTelemetry (OTEL), which is the emerging standard that ties all these CNCF pieces together with others such as Fluent Bit
You're right.
I got complaints in the past that the videos were too long so I started making shorter ones and that often results in things missing. I'll work on a second part of that video with tracing and open. telemetry
I finally got around making a video about OpenTelemetry (I wanted to explore it separately first). It's available at th-cam.com/video/oe5YYh9mhzw/w-d-xo.html. Tracing is coming next.
Thanks Viktor for your nice video & informative , really helpful
Great video.
Are there any other self-managed logging solutions other than ELK/EFK and Loki-Grafana?
There's fluentd and fluentbit for shipping logs.
@@DevOpsToolkit Thanks. I mean more on the storage side. Like if I want to move away from AWS open search to decrease cost. Are there any alternatives than Loki?
@AhmedAyman-gs7oz Loki is the only one I used besides elasticsearch (excluding managed solutions).
VictoriaMetrics (Operator)
How did I miss this video, 2 days wasted. Thanks
Nice explanation! Thank you very much.
Yes!
Fantastic!
You're awesome
Thanks a ton Zach
Thanks!
Thanks a ton!
@@DevOpsToolkit My pleasure! Can’t help but contributing a bit as I’m binge watching your videos. Plus, it was a pleasure collaborating with you on Geekle’s conferences. Keep up the great work!
Thanks for this great video , what about black-box exporter ?
I should probably create a separate video about a selection of Prometheus exporters.
@@DevOpsToolkit this would be nice 😊. Thank you 🙏
very interesting
How do we configure SSO for Grafana login?
As far as I know, there is no SSO in Grafana open-source version (not sure about Enterprise). I would probably try something like Teleport (you'll find a video in this channel).
Grafana is a great tools but somehow like Jenkins
If the plug-in no longer maintain then you may get trouble it have chance
may your Grafana unable to perform upgrade otherwise graph will dead
But I do love the trend they go for Loki+Tempo+Display Prometheus
Love to see if there have tools to generate dashboard and integrated 3 of them
Do you have plan or already have video for Tempo?
Tempo is on my todo list :)