Explained nicely. Here is what I can think about your question. The API, which is used for shortening the URL, should first look into the DB (or wherever its stored) for the given input (assuming it would be URL) . in case URL exists in the DB, API should return the short URL for the same. if DB does't contain then only API should generate the new short URL and return. Assuming older data in DB is cleaned up or archived periodically.
Love you Hussein thank you very much for contributing a lot to community we see how hard it is how much time it takes and we appreciate that, keep going man!
Greate video Hussein! I really love how you explained it. I've always had a small understanding of this topic and with your video, I get it completely now.
But if you keep refreshing the page it would generate a new key so even if your POST indempotent, since the new key is generated, it would still create a new resource. What am I missing here?
To deal with idempotent request you can maintain checkoutId in checkout page and in the order page you can pass that checkoutId and you can ensure if an order is placed or not with that checkoutId
I would hash the passed in url and store this hash and data in a database. The hash would be the primary key, so for every future call I can do a quick search and return what is already stored, otherwise run it through the url shortener and store it.
We are facing this problem at work big time. We are trying to build a massively high available system and we need our apis / services to be idempotent but many sub systems we manage via APIs are inherently not idempotent so it’s a high challenge.
for url shortner idempotency I would hash the payload and will take it as an idempotency key. this should be done because for a particular user requesting for shortening a URL with some properties i.e. short url being private / public etc will uniquely identify a short url request
I have a partial resource update so I made that API as PATCH. And the operation and implementation is idempotent. But in REST PATCH is not idempotent. What's your thought on this? Is going with PUT a better design or PATCH with idempotent?
Think about it semantically. If you issue that resource update multiple times what well happen? If it is safe, use PUT else continue using patch or POST
Hi , Does anyone have any implementation suggestion to handle de-duplication at Kafka consumer side ? i.e to make idempotent consumer which will discard dupllicate messages from Kafka in the event of any broker or offset issues. One solution aware of is to store last read consumer offset number in a database. Store the offset number and msg values in single update call to ensure atomicity. Whenever consumer reads again, it check in the DB to ensure is not reading offset number less than already in the DB. Downside is if DB is huge , it will be perfomance impact. Another possible way is store topic name and last read offset number in a seperate table. Any other recommended practice ?
In case of url shortener we will save real link in db and when user send a request again we check if the link already existed in db we will send the shorten link of that link else we generate a new link if I t doesn't exist
Great video Hussein. However I have one doubt. In the video you explained that for making the request as idempotent, we can create a client id for a request. So , suppose if two POST requests are coming from two different devices ( not concurrently) , then there can be a case where client id becomes different. So it can end up in creating two records in the DB. But , my requirement is to update the record created by the 1st POST request. In such scenario, how do we handle idempotency? Or can we use client id approach?
You use PUT requests to update and use POST for inserts. Therefore if you send a post request from two different clients it should enter two new records. But lets say you can’t do that. In that case you will have to create some sort of a hash key from fields which are not gonna get updated and unique to that record.
when you give example related to real world problems like the transaction one....video become much more interesting...........try make more relations with practial implementation of those please!!
Quarantine Coder thank you 🙏I did give an example in this video right? The ordering system ? Were you looking for something else specific let me know so i do better next time 😊
@@hnasr yeah.. Gave one in this.. That.. Why I liked this one.. I mean to say your experience with solving real world problem .. That amplifys the video
Your question is very interesting. Typically we want to allow multiple shortened URLs for the same input URL (for different reasons) but at the same time we want to avoid loss of idempotency. I would take a hash of the URL and store it in the database. I would also store the Datetime of said request in YYYYmmDD HH format. I would create a unique index in this table as the composite of the hash and datetime. The result is we will not allow the same URL to create multiple shortened URLs in the same hour. If user requests the same URL in the same hour, our endpoint can return the pre-existing value from the database.
@@rampandey191 Bit late to the party, but I'd assume for analytics (as an example) you'd want to allow multiple. Let's say I create a shortened URL that leads to a popular website and someone else does the same. We only want to see however how much traffic that shortened URL is getting from wherever we have shared the URL to. If both people get the same shortened URL, then the analytics would get mixed up together. By creating 2 shortened URLs for the same URL this wouldn't happen and each of the 2 users would have their own URL that they can share and keep track of how much traffic they are getting from it.
Football Hero اهلا عزيزي، القراءه المستمره، استماع للبودكاست ، مشاهده فيديوهات. رما زلت اعتبر نفسي لا اعرف اي شي. وعاء العلم لا يمتلى بما فيه. بالتوفيق شاهد هذا الفيديو اتكلم بالتفصيل My Preferred Method of Learning Backend Engineering Technologies th-cam.com/video/4NsWnT_-FoE/w-d-xo.html
Love you being a quick and simple solver.
Explained nicely.
Here is what I can think about your question. The API, which is used for shortening the URL, should first look into the DB (or wherever its stored) for the given input (assuming it would be URL) . in case URL exists in the DB, API should return the short URL for the same. if DB does't contain then only API should generate the new short URL and return.
Assuming older data in DB is cleaned up or archived periodically.
Awesome video again, lot of these concepts I came to know cuz of your videos.
Love you Hussein thank you very much for contributing a lot to community we see how hard it is how much time it takes and we appreciate that, keep going man!
Good stuff. Your question: Validate url to be created against the db, if found=send that, else create new one (store in db) and send short url
Greate video Hussein! I really love how you explained it. I've always had a small understanding of this topic and with your video, I get it completely now.
You have a really wonderful voice and way of speaking!
Hussein Nasser, you are a gem! May god bless you.
Nikhil Tripathy thanks Nikhil God bless ❤️ enjoy the content
Another great video man!
Thanks for all the content!! Super helpful👍👍
But if you keep refreshing the page it would generate a new key so even if your POST indempotent, since the new key is generated, it would still create a new resource. What am I missing here?
To deal with idempotent request you can maintain checkoutId in checkout page and in the order page you can pass that checkoutId and you can ensure if an order is placed or not with that checkoutId
I would hash the passed in url and store this hash and data in a database. The hash would be the primary key, so for every future call I can do a quick search and return what is already stored, otherwise run it through the url shortener and store it.
We are facing this problem at work big time. We are trying to build a massively high available system and we need our apis / services to be idempotent but many sub systems we manage via APIs are inherently not idempotent so it’s a high challenge.
Very well explained !!
All your videos are awesome.
for url shortner idempotency I would hash the payload and will take it as an idempotency key.
this should be done because for a particular user requesting for shortening a URL with some properties i.e. short url being private / public etc will uniquely identify a short url request
Hi Hussein, shouldn't we use put request instead of post in these cases to get idempotency ? That's why put requests are made, right ?
I have a partial resource update so I made that API as PATCH. And the operation and implementation is idempotent. But in REST PATCH is not idempotent.
What's your thought on this?
Is going with PUT a better design or PATCH with idempotent?
Think about it semantically. If you issue that resource update multiple times what well happen? If it is safe, use PUT else continue using patch or POST
Great video, thank you
HTTP POSTs are defined as NOT idempotent. Please use PUT when you define an idempotent endpoint to write to db.
Hi , Does anyone have any implementation suggestion to handle de-duplication at Kafka consumer side ? i.e to make idempotent consumer which will discard dupllicate messages from Kafka in the event of any broker or offset issues.
One solution aware of is to store last read consumer offset number in a database. Store the offset number and msg values in single update call to ensure atomicity. Whenever consumer reads again, it check in the DB to ensure is not reading offset number less than already in the DB. Downside is if DB is huge , it will be perfomance impact.
Another possible way is store topic name and last read offset number in a seperate table.
Any other recommended practice ?
In case of url shortener we will save real link in db and when user send a request again we check if the link already existed in db we will send the shorten link of that link else we generate a new link if I t doesn't exist
URL Idempotent: Can we put URL Mapping in the Protobuf?
Great video Hussein. However I have one doubt. In the video you explained that for making the request as idempotent, we can create a client id for a request. So , suppose if two POST requests are coming from two different devices ( not concurrently) , then there can be a case where client id becomes different. So it can end up in creating two records in the DB. But , my requirement is to update the record created by the 1st POST request. In such scenario, how do we handle idempotency? Or can we use client id approach?
You use PUT requests to update and use POST for inserts. Therefore if you send a post request from two different clients it should enter two new records. But lets say you can’t do that. In that case you will have to create some sort of a hash key from fields which are not gonna get updated and unique to that record.
@@amila325 Thanks Ami, I was also thinking about a similar kind of solution.
when you give example related to real world problems like the transaction one....video become much more interesting...........try make more relations with practial implementation of those please!!
Quarantine Coder thank you 🙏I did give an example in this video right? The ordering system ? Were you looking for something else specific let me know so i do better next time 😊
@@hnasr yeah.. Gave one in this.. That.. Why I liked this one.. I mean to say your experience with solving real world problem .. That amplifys the video
يعطيك العافية
ممكن فيديوهات عن الفرونت اند والاخطاء الي يجب تجنبها مثل المذكورة في الفيديو وبلتحديد في اطر العمل مثل React js
Your question is very interesting. Typically we want to allow multiple shortened URLs for the same input URL (for different reasons) but at the same time we want to avoid loss of idempotency. I would take a hash of the URL and store it in the database. I would also store the Datetime of said request in YYYYmmDD HH format. I would create a unique index in this table as the composite of the hash and datetime. The result is we will not allow the same URL to create multiple shortened URLs in the same hour. If user requests the same URL in the same hour, our endpoint can return the pre-existing value from the database.
Hey can you explain the reason to create different shortened URLs for same URL? Because I think it should not be allowed
@@rampandey191 Bit late to the party, but I'd assume for analytics (as an example) you'd want to allow multiple. Let's say I create a shortened URL that leads to a popular website and someone else does the same. We only want to see however how much traffic that shortened URL is getting from wherever we have shared the URL to. If both people get the same shortened URL, then the analytics would get mixed up together. By creating 2 shortened URLs for the same URL this wouldn't happen and each of the 2 users would have their own URL that they can share and keep track of how much traffic they are getting from it.
@@lonerider543 makes sense thanks
Nice ! Followed you .
Crisp! 😀
Great video!
i love that moment when Hussein laughs at some hole in the system.
How about.... I have an idea.... how about..... a video that shows how to do retries
استاذ حسين كنت عاوز اعرف منين انتا عارف كل هذي الاشياء انا اول مرة اعرفها لما دخلت على قناتك لو تدلني على مكان اقدر اعرفها زيك اكون شاكرلك بقية حياتي
Football Hero اهلا عزيزي، القراءه المستمره، استماع للبودكاست ، مشاهده فيديوهات. رما زلت اعتبر نفسي لا اعرف اي شي. وعاء العلم لا يمتلى بما فيه.
بالتوفيق
شاهد هذا الفيديو اتكلم بالتفصيل
My Preferred Method of Learning Backend Engineering Technologies
th-cam.com/video/4NsWnT_-FoE/w-d-xo.html
@@hnasr اشكرك استاذ ناصر على كرمك و مساعدتك لنا و انا اوعدك في يوم حردلك هذا الجميل 💙💙💙💙
you can explain stuff in few minutes and that's an art. people make 30mins videos on this lol. just puts me off
I just know about this from this video
It is an important concept, learned about it while researching Kafka around a year ago. Thanks for watching 🙏
Just a suggestion- Its a knowledge video, it would be better if you clearly narrate the things instead of speaking in an unclear accent.