How do you conclude when to restart a pod though via the liveness probe? For example if you have 100 pods all relying on a MySQL database. That MySQL database goes down, you don't want to cause 100 pods to keep restarting. What do you use to know if it is a configuration error and requires restart vs temporary communication problem that we can auto reconnect to once it is back up? I think it would actually be better to keep the app up, correlate other observability tools that are providing database health and response codes to determine that there is a DB issue. What are your thoughts?
You're absolutely right, restarting all 100 pods just because the MySQL database goes down isn't ideal. Instead of relying purely on the liveness probe to restart the pods, a better approach would be using the readiness probe. This way, the pods stay up but won’t serve any traffic until the database is back online, avoiding unnecessary restarts while keeping the app alive and ready to reconnect when things are back to normal. When it comes to figuring out if it's a database issue or something else, I completely agree with you-using observability tools like Prometheus or Grafana is a much smarter way to go. These tools give you better insight into the health of your database and can alert you to actual issues without causing chaos in your pods. Instead of restarting everything, you can use these tools to get a clearer picture of what's really going on. Also, having some built-in retry logic in the app is essential. If it’s just a temporary issue, your app can keep attempting to reconnect without needing a full restart. So, yeah, keeping the app up, using observability tools, and waiting for the database to recover is definitely the better strategy in this case. Thanks for bringing that up-it’s a really important distinction!
I did make a video on Distributed Tracing previously. Here th-cam.com/video/XYvQHjWJJTE/w-d-xo.html I will deep dive into one of the tools in future, thanks for the support!
Check out the playlist in description. If you are a fresher I would suggest to start with System Design Basics, and then look into Microservices. Best!
How do you conclude when to restart a pod though via the liveness probe? For example if you have 100 pods all relying on a MySQL database. That MySQL database goes down, you don't want to cause 100 pods to keep restarting. What do you use to know if it is a configuration error and requires restart vs temporary communication problem that we can auto reconnect to once it is back up?
I think it would actually be better to keep the app up, correlate other observability tools that are providing database health and response codes to determine that there is a DB issue.
What are your thoughts?
You're absolutely right, restarting all 100 pods just because the MySQL database goes down isn't ideal. Instead of relying purely on the liveness probe to restart the pods, a better approach would be using the readiness probe. This way, the pods stay up but won’t serve any traffic until the database is back online, avoiding unnecessary restarts while keeping the app alive and ready to reconnect when things are back to normal.
When it comes to figuring out if it's a database issue or something else, I completely agree with you-using observability tools like Prometheus or Grafana is a much smarter way to go. These tools give you better insight into the health of your database and can alert you to actual issues without causing chaos in your pods. Instead of restarting everything, you can use these tools to get a clearer picture of what's really going on.
Also, having some built-in retry logic in the app is essential. If it’s just a temporary issue, your app can keep attempting to reconnect without needing a full restart. So, yeah, keeping the app up, using observability tools, and waiting for the database to recover is definitely the better strategy in this case. Thanks for bringing that up-it’s a really important distinction!
Can you share which tool you are using for this animated videos😎
bro I love watching your videos , please make a video on distributed tracing platforms like jaegar tracing
I did make a video on Distributed Tracing previously. Here th-cam.com/video/XYvQHjWJJTE/w-d-xo.html
I will deep dive into one of the tools in future, thanks for the support!
This topic comes under which domain?
I want to learn more about these things, i am just a fresher please guide me with resources
Check out the playlist in description. If you are a fresher I would suggest to start with System Design Basics, and then look into Microservices. Best!