The CORRECT way to implement Retries in .NET
ฝัง
- เผยแพร่เมื่อ 18 ก.ย. 2022
- Get started with Octopus Deploy: oc.to/nickchapsas
Join Octopus Deploy's free Slack community: oc.to/SlackCommunity
Check out my courses: dometrain.com
Become a Patreon and get source code access: / nickchapsas
Hello everybody I'm Nick and in this video I will show you how you can make your code more resilient by implementing retry policies. Retry policies power pretty much every microservice out there in some capacity, but due to the nature of having multiple systems coming together and needing to deal with things that might fail.
Don't forget to comment, like and subscribe :)
Social Media:
Follow me on GitHub: bit.ly/ChapsasGitHub
Follow me on Twitter: bit.ly/ChapsasTwitter
Connect on LinkedIn: bit.ly/ChapsasLinkedIn
Keep coding merch: keepcoding.shop
#csharp #dotnet
I've been using Polly for a long time. One of the things you have to be careful of though is when making HttpRequests and getting a timeout. If your request should mutate data, and the request is not idempotent, then recalling the method on a timeout (that actually succeeded) during a retry could cause duplicate calls. Most of my API work unfortunately falls under that umbrella, so I don't set a global retry policy, and handle each call individually.
If it's your own APIs that have this weakness, it's a good idea to do it within a Transactional block, so the work can get rolled back if it times out too early; CancellationTokens are your friend here.
And if the call has a point-of-no-return (e.g. because it interacts with software outside the API), I've managed that by requiring unique message GUIDs for those requests.
These get recorded in a database so I can't reprocess them by accident, and once processed, I also record the response. So if they time out and resend the request, I just return that stored response instead.
Correct, once you distribute your application over the network, you have to start caring about all those wonderful distributed computing problems and timeouts are the one that are often forgotten about (did it complete? did it not complete?). Idempotency and eventual consistency are then the solution but that often involves shifting to using queues/streams with retries rather than something as straightforward as just wrapping http calls with Polly.
Yes it is always a blessing when using a new Api to see that it accepts some kind of idempotencyKey in their Post endpoints. Otherwise I wouldn't recommend at all retrying any kind of Post request.
I have my services split by "Command" and "Query" -- a light-style "CQRS" pattern. I can apply a retry policy on the "Query" services, while I leave the "Command" services to fail fast.
I'm really looking forward to see what other videos you come up with on the subject of resiliency. It's so often just retries, but so extra topics would be amazing!
Love this channel. just started a new role where I am using Polly, had a rough idea of what was going on but not a full understanding, totally get it now, including Jitter which I seen being used but didn't really know what was going on! Thanks :)
Your content is always top notch! Can´t wait to watch your take on the Circuit Breaker. It was the first ¨resilency pattern¨ I learnt and I think it´s one of the hidden gems when talking about resilency.
Polly is one of my favorite packages. I can't image life with out it.
As usual, great content, easily explained, just wanted to say thanks for that 👏🏾
This is really helpful. I love your enthusiasm and knowledge. Thx.
Thank you! This is exactly what I need in my project right now!
Great content as usual, I remember we talked about the jitter a couple of years ago when you did a video on retries. Regarding status codes to retry on, I think 429 that for instance APIM deliver if you add the Throttling Policy can be good to retry on. Also, when I get the 429 from Cosmos I first retry a couple of time with the RetryAfterMs that the DocumentException property deliver and if I dont succeed I respond to caller with a Responseheader that have the RetryAfterMs value so the Client can retry after a "recommended" wait time. Same with 403 Forbidden, some APIs incuding google use 403 instead of 429 if a client call them to many times in a short period so that can also be good to retry on against some APIs, while against other APIs it can be a waste of resources... Happy Coding! :)
I really loved this video... Thank you so much Nick!
As always love your explanations!
Very good to know this info. I do approach things differently: I have a background service that polls external APIs at regular interval and caches the data locally. Then, my API will send cached data to my clients. My polling service would do the retries in the background and if, for some reason, fails, the client's experience is not affected as they get the last cached data. This is great for a couple of reasons:
Client experience is not interrupted nor delayed.
External calls are only made at predefined intervals - so light on the provider's end (no "punishments" for hitting their service too hard).
External API usage costs based on hits/traffic can be reduced greatly, thus saving money: Say you had a 3rd party service that allowed 5000 hits per month for $100, and the next level was 50K hits for $500, and, without pre-polling/caching, you were calling ~30K hits per month. You would have your background poll every 10 minutes, or 4320hits/mo and save $400 every month - right off the bat. It would also save on bandwidth (while making it far more predictable), and with cloud services nickel-and-diming you to death on just about everything, you might save a few hundred more dollars every month on top of that! I love cloud tech, but it's still absurdly expensive and there's always those invoice shockers. Caching data is a good way to mitigate some cloud-sticker-shock.
+1 That's good info, but assumes you know in advance what the API call and its parameters will be. Obviously you could use a LRU cache or similar in your service if you don't know in advance.
I started proxying external services (that i've no control over) a decade ago for security purposes primarily and still do to this day as a matter of policy.
Great insight, i was on the fence about using polly
I came here to see polly implementation and that is excactly whats there, kudos
Really proper way for retries is to implement circuit breaker pattern. Jitter and exponential retries just smooth out load on downstream API, but it almost doesn't reduce number of requests which you will fire into it when it fails.
Awesome solution! Keep up the good work! 👍
I'll say that this scenario probably isn't appropriate for a retry anyway. If your API simply proxies another API, and the other API is down, then in reality your service is down too. Delegate the retry logic to the client, because the client can determine what kind of UX it wants. Is it okay with a user waiting 16 seconds as it spins? Is it going to throw up an error message on immediate failure and passively retry in the background? Is it going to stop trying and prompt manual reattempts at some point? They're all fine, but the thing is the backend doesn't need to worry about it at all, it does its one-time proxy and if it fails, it fails. A circuit breaker of some sort may still be appropriate to avoid contributing to an accidental DDoS, because a client doesn't have to respect your retry policy anyway and can just spam requests until you DDoS anyway.
There are situations where this kind of resiliency *is* more important, though, and the concepts you demonstrated here + the use of polly is great. Maybe in aggregation requests where your API is calling 4 or 5 APIs and you don't want a transient failure in one of them to kill the entire request is probably the best example. Another popular one is if you expect call chains across dozens of services waiting for your final return, but if that's what the company has built then then murder is probably a more appropriate solution than retry policies. Passive background processing services with no user on the other end are probably acceptable as well, but such a process is probably already on an infinite loop of some sort anyway so it really doesn't matter there.
Nice summary - in short it depends.
on what is mission critical and what is not.
If someone is trying to implement a transaction scope across 4 or 5 API calls, then at some point murphy's law is going to kick in. Most likely a failure is when you least want it. In that case, I would move that processing into a single API call which internally has its own rollback mechanisms, and simplifies consumption.
Identify the mission critical failures and then separating those out. For example, if an order is being completed, and a payment token needs to be finalized as part of that, then
1) Saving of data in a pre-finalization state
2) Obtaining the token (and rollback above it user cancels or fails - if the rollback fails no damage is done except for orphaned data which is already flagged as such)
3) Charging the token
4) Finalizing the order, including notifications, inventory, etc.
- The pre-finalization state you can roll back easily if needed, and have the customer try again.. The purpose of this is to lesson the workload in the finalization step, and those less things that can go wrong, as well as providing a final order id when getting the token.
- If unable to obtain the token you can direct the customer to try again (and roll back above).
- Charging the token is the critical point of failure
- Finalizing the order you can recover from if using a queue that records the failure point, so you no the state, and this can have retries for extended period, and administrative solutions if needed.
So in this case the mission critical step is charging the token. If a call is made, and it simply evaporates, then additional steps need to ascertain if the token was indeed charged or not. It does not matter if the user needs to wait 20 seconds in this rare case, because if it is not ascertained then it is a critical system error,. You do not want to roll back, because that could make things worse.
_
For a weather app, the weather API provider would most likely have a cache, and can do a couple retries if cache does not have an entry yet. If that fails, then they return the appropriate message. Otherwise they could use stale cache until recovered. None of it is mission critical.
Then if a client wants to add there own retry logic, it is unlikely to result in a DOS failure of the weather app.
--
Just this second, a lighting bolt came, then followed by thunder, and power flashed off and back on (for real). Lets see if this reply makes it through.
Excellent video and attention to detail
Beautifully explained 👌
Very cool! Thanks for sharing!
Love this style of production-ready solution to common problems that are easy to mess up on your own. Same goes for the working with secrets video. Considering how many current and future software disasters you've enabled your viewers to avert. That's more than just good content - it's good karma.
I like that the 0.69 and 420 were not left out. Nice.
Me Too!😃
Please don't use goto in public. A lot of guys, especially newbies don't even know it exists and this is so good
Great upload nick was about to search this haha
I've used Polly in the past; simple and elegant.
Awesome video as always! I'm building my first Blazor WASM application and wondering if thats a good use case for polly on the backend calls.
Any plans to add a new course anytime soon ? Maybe something about microservices ? They are so popular nowadays and hard to find some good explanation about properly implementing them covering failure cases and scaling.
try this : th-cam.com/video/DgVjEo3OGBI/w-d-xo.html
Approved!
Lesson 1 of that course should be "get really flipping good at CICD" and all brands of automation"
Nice one! I also like the circuit breaker in Polly package
It’s coming next week
Thanks for the video
Super useful video, thank you very much
This is a very nice topic that does not get much love -- tks
Quality content, cheers
Hello Nick. As always bringing the good content. I have been following you for a quite time and you´re a machine. I have been implementing multiple approaches from other videos in my company, I really wanted to say thank you in the first place. I have one question, would this approach be useful to implement a token expiration to an api call? For example if the token expires, implement the retry policy to get a new refresh token? Or the better one is to implement a delegatingHandler in HttpClientFactory?
Thanks!
Great content as always. I'm wondering though, why didn't you use the presented approach together with the circuit breaker pattern? For the example you've shown it seems like a good idea.
Hii, thanl you for greate videos.
I want to ask about dependency injection of my own services.
it's pretty easy to use in contorllers.
But what if i want to use it in other classes or for static variables ? Is it ok to get the ServiceProvider (app.services) and assign its value to a static variable ?? (eg. TestClass.Provider = app.Services)
Would you implement the same policy for database stuff?
Is it possible to register for example SqlConnection with policy?
Could you do videos on authentication and security ? you are the best love your videos!
Thank you for your videos!
I have one question: Can I use this library when successfully request status is defined one response content level, not status code?
Absolutely. It supports reading the content of the http response body
I think a good thing to mention here is idempotent design and when not to retry or how to make post endpoints idempotent.
Do you recommend using an interface to replace the direct dependency on Polly? How would you do it
This feature will make if faster to write some C# code, but, increases what you need to learn to read that code later. As I get older I tend to prefer code that's easier to read even if it's more characters, at least for long-lived projects. For quick short-lived scripts these features are nice and taken to the extreme you end up with a "write-only language" like perl.
Nice video... as always. I was also wondering whether you are planning to make a video about unsafe performance optimizations that can be done. Even though they might not be used regularly, they would still be very interesting.
Ohh that's a great idea!
@@nickchapsas YES!
we never randomized with the power before and Jitter, we just randomized. I never knew this is called the Jitter even when you already know the feature you can learn something new!
Awesome vídeo Nick!! Love it.
PS: in your opinion, what is the best way to test Store Procedures integration?
In my opinion you should not be using stored procedures
@@nickchapsas Why?
with the timeouts, should not we also capture TaskCanceledExceptions? my experience is that usually that's thrown during http call timeouts with httpclients, and HandleTransientHttpError is not covered by that (we also add SocketExceptions as well)...
Or are we using the httpclient wrong? (although some of it comes from company-internal libs...)
Good vid 🥰
Polly is awesome. Also been using this since forever. Love the fact you can reduce everything to just 2 lines. Can you now do it with just 1 character? 😛 Maybe with .NET 15?
@Nick, could you please make a video on reactive programming and how it is implemented (if it is) in C#?
Great! Add circuit break and your golden 😉
awesome
What about maintaining a status for the backing api that sets it to offline when there's a connection problem.
You can then start polling until you get a response again and continue serving calls for all clients.
Also, you specified internal server error, but that's can be specific to the call.
Shouldn't there be an error specific to the api not being available?
You managed to smoothly sneak both 69 and 420 into the video. Nice job m8.
Hey Nick,
Which CI tool do you use with Octopus Deploy? I remember that you use Rider (like me) so I figured it might be TeamCity, but I think TeamCity is maybe a bit expensive if you don't need a lot of fancy stuff. There are a lot of different tools so I figured you may have some good input.
Cool video as usual!
I’ve been using TeamCity indeed
Thank you! Is there something standard for 429 Too Many Requests handling?
The Retry method has the ability to provide a function to consume the context provided by Polly to determine the sleep duration. This is more flexible because some APIs wont use the standard retry header, and even more so some libraries like Azure Cosmos DB will wrap the recommended wait time into an exception.
And what would you suggest if I have an azure function, and I need to await a larger delay between retries, like 2 minutes.
I don't think keeping a serveless app awake up to 10 minutes (per request) is a good idea. 😅
Thanks for the great video. We have a service (s1) with a similar retry policy, calling another service (s2) that also has a retry policy. The second service is also calling a service (s3), which is the last service being called. For some reason s2 is retrying to call s3 and then s1 is starting to retry calling s2. We are seeing http cancellation exceptions, but can not figure out what the underlying issue is. Can retry policies in multiple services cause issues like this? Thanks
Maybe because of timeout between s1 et s2 while s2 retry to call X time s3?
Can polly retry errors that occur while reading the response body? For instance, a connection error when downloading a large blob to a Stream. If not, how would you handle that?
it can retry enything, so any exception in a code block will result in calling it again
Could you please make some videos on Owin, OAuth and Saml Authentication
"or maybe another time..." proceeds to silently write 0.69
that's a lot of patrons!
Why do you have httpClientFactory.CreateClient() in the method and not in the constructor? I missed if the nuance as to why it moved was in your last video. For a high throughput application wouldn’t it be a lot of scoped allocation on that var client variable?
It's the same amount of allocations. The HttpClient is transient even in the constructor. It's the handlers behind the scenes that are pooled. Check my HttpClient video
You could also implement the retry inside the HTTP pipeline by the custom handler
Yes you can. The advantage of Polly is that, aside from beignt easy to implement, you can add a retry to basically anything, i use it for event sending, api calls, ftp file writing, almost everything that can fail for a transient error
Hello Nick, loved your advice! But doesn't that breaks the 'open closed' principle from S.O.L.I.D.?
It doesn't. Open-closed can also be achieved with composition, you don't need inheritance
Interesting - I have always skipped retries on 500 on the assumption that it could be my input that caused the error.
Thinking back to it though, lots of the services I've worked with would indeed return 500 for transient or operational issues.
Nice
Can you be notified when a retry is taking place so that you can inform a user than an error is being elegantly handled? For instance, "now retrying 1/x ... 2/x ... 3/x" perhaps with a Cancel button?
Technically you can stream a response during retrying but the handling of that streamed response is very client specific.
Definitely, depends on implementation. Maybe with hub? I was just logging timeout with logger to see if it is not persistent error in my app and worked like a charm. Cancel would be trickier. Just realized, why not using cancellation token ;)
Hmm, an event or callback would be nice, as well as a cancelation token. 🤔
If you have clients accessing the API at random times, rather than synchronized, shouldn't that imbue the effective jitter necessary?
Yes
Great video, thank you. What pen and tablet do you use for the whiteboard app?
Look at the application title at the top.
@@CharlesBurnsPrime or better yet, try reading my post properly. I didn't ask about the name of the application. I asked about the hardware (pen and tablet).
@@mil3761 You are right, I did not read your post properly. No need to be rude about it
@@CharlesBurnsPrime that's funny cos "rude" is kinda how your initial reply came across.
@@mil3761 I can see that.
Why not wait for 4.20?
Does any tools for implementation of retry policy for not http calls, like database or grpc calls?
Polly can handle any type of method call
What if my company puts the restriction to the nuget packages that I'm allowed to use. Is there any way to make it right without Polly?
You should educate the persons in your organization that placed a restriction on Polly.
cool..!!!
Wow awesome, Polly evolved.
Hi, can You make video about installing minimal API project on Linux?
Can Polly be used for APIs rate limit?
It can! It has a dedicated Rate Limiting functionality
also Polly is not restircited to be used with HttpClient, it can be used for any method call.
This is mentioned in the video
It's interesting that the exact same code which is production grade by the end of the video is not allowed anywhere near production at 2:50. I'm curious where you think the responsibility belongs for implementing the retry policy: the author of the client library...or the consumer of the client? Also, I was expecting you to mention the circuit breaker policy which I think would help in the scenario you describe. Nonetheless enjoyed the video.
Circuit breaker video is coming next week
Hi Nick, your httpclient video link does not appear when you mention it should.
Which IDE you are using?
Rider
In my case I need to get the error results from all the iterations. For example if I am retrying 2 times without success I need to return {errors:[ "timeout", "timeout","timeout"]} from my API. Can I achieve this with polly ?
Of course, you can get everything about the response body
@@nickchapsas I am not sure you understand my question. Sample scenario: I want to write to file and retry 2 times if it fails due to Exception. If all of my retries fails I want to get all 3 exceptions that occurs (during retries by polly). The problem is ExecuteAsync throws only the last Exception.
@@Martin-kj1od if you just want to e.g. log it, there's a separate handler for that: WaitAndRetryAsync has an optional function which is called on every failure with the details. If you want to collect it for later use - like at the end throw an AggregateException - well, i am not sure about that, my guess is that you have to solve yourself
I always enjoy your videos. But just an idea. Your face cam always hides input in the IDE whereas under the solution explorer is mostly empty space. Then your cam would not hide any content :)
Fascinating...
You don’t need that jittering randomization, as it will happen naturally during execution, as there are naturally happening network delays and thread scheduling jitters.
Also, it is much better if you use prime numbers as delay intervals: 1, 3, 5, 7, 11, 13
Jittering is important when you're calling an API with a naturally small response time. Go read the documentation on Polly's pages, because they have done way more research than reasonable on why a smidge of random is important.
Why are prime numbers better for delay intervals ?
@@DarraghJones less chance for concurrency even if two clients hit the same thing on precisely synchronized clocks.
Curious no mention of 429!
Nick, why aren’t you suffixing your async methods names with “Async”?
Convention. Good practice.
As soon as you see suffix you know to await the call
@@ibrahimhussain3248 I know. I asked WHY he’s not doing that.
@@fraysa oh! hehe. my bad
you should 429 codes to the policy as well.
11:05 Why not 4.20 ?
God damnit
420, nice.
Am I the only person who finds it hard to get Visual Studio to run unit tests in the latest versions of .NET Core?
Actually, studies show that a user will wait for 20 seconds before assuming something has gone wrong.
@11.12 missed a 4.20
Why do this in code? Isn't this something better suited for service Mesh
Why not let the server be responsible for deciding on whether to retry
handling a request in case it failed? I expect that in most cases the client should not be responsible for that. Clients should not care about why the server failed. There's a response coming from the server and the client should handle that and continue. Timeouts are there to make sure that clients don't wait forever.
complinted tNice tutorials guys voice (that I like as well).
0.69 420 😅
Your videos are good but can you use less click-bait titles on banners please.
Sadly, this is how TH-cam works. Sorry for that but it can't change
GTFO w the controllers bro is 2022 we on that minimal apiiiiiiiiiiiiiiiiiiii
When will you stop reminding your API key is invalid?
Never
@@nickchapsas So irritating. Be pragmatic.
@@anatolia23 Nah
If you have clients accessing the API at random times, rather than synchronized, shouldn't that imbue the effective jitter necessary?
Yes