You should continue this podcast series. Out of all the podcasts or technical tutorials, there is no one in this niche. Students or experienced professionals will learn a lot from this. Kudos to you.
*timestamps:* 0:00 Introduction 2:48 Initial architecture/techstack of RazorPay 4:30 ensuring strong consistency on monolithic architecture 9:06 check constraints 10:47 deciding to move from monolith to microservices 15:23 splitting of databases/application logic when moving to microservices 18:25 dual writing of data into monolith as well as microservice dbs(during transition phase), transaction outbox pattern 21:19 databases in current microservice architecture 22:45 deciding on SQL vs NOSQL 25:10 ensuring consistency in microservice architecture 28:10 payment flow with eventual consistency 34:40 prioritization for settlements@T+0 37:20 chaos testing 39:00 integration tests on interface between 2 services 41:00 incidents when things went wrong 47:40 outro
The way you progress the communication through passively narrating and curiously listening made this podcast different from other existing system design podcasts.
This is truly asli engineering . Seedhi Baat no bakwaas, Real domain challenges and solution . I know that no architectures is The BEST. OutBox CDC ... amazing stuff ... please continue making stuff like this.
These 49.48 mins went pass like some couple of minutes. This is so much learning and that also in such such basic terminology. That's one heck of a enginnering. Thanks for the podcast Arpit and Arjun. I wish this series never comes to an end ❤️
Few things amazed me and I am not sure about whether its worth it. 1. Just for connection pooling they migrated monolith to microservice. 2. They remove foreign key constraints in their table and made custom orm.
Great podcast Arpit. Best thing I loved was that you were paraphrasing most of the things that Arjun said. I think you already know most of the things and still you sounded like a newbie with excitement who's listening about systems for the first time :)
Real engineering hits more than any dopamine. Got facinated by the number of DBs razorpay is using depends on seevice. Really cool stuff with cool Engineer
Just one word.. Brilliant Podcast This is really Asli Engineering Stuff. Worth watching every second of this podcast, can't skip anything. Very Insightful.
loved this episode, so much to learn from razorpay. The data consistency section and ledger settlement is such a creative insight for anybody who hasn't worked on it yet. Thanks for recording this session and happy 1 month @google Arpit!!!
woow, my mind blows away, so much knowledge on a plate, lots of things to get to know about how the actual digital payment system works, which may appear simple from a normal person's perspective. Thank you, Arpit.
Hi Arpit & Arjun, I've been thoroughly enjoying your content and have been binge-watching your videos. At 18:25, Arjun discussed implementing the outbox pattern for dual writes. While I understand the concept well (and have implemented something similar), I wanted to delve deeper into the actual implementation. The traditional use case for this pattern involves achieving ACD (a form of transactionality) for database writes and event publication in the message queue or a different DB. Once the data is transactionally written to both the monolith table and the outbox table, an asynchronous job, using transactional log tailing (binlog consumption), allows us to consistently write to the microservice table. However, let's consider a scenario where the broker is down (acting as a SPoF). How does the OUTBOX PATTERN ensure consistency in this situation? In my implementation, instead of relying on binlog consumption, I've set up a cronjob (using a serverless service) to read from the outbox table and publish the event to the message queue. If it succeeds, it updates the status in the outbox table to prevent the next run of the cronjob from picking it up again. I'd love to hear your thoughts.
Wow! What a knowledge packed podcast and this is just ep1! So glad this exists! Really enjoyed conversations between both of you! Excited for future eps! Thanks Arpit!
Really inspiring series of videos, arpit. Keep it going I am sure it is inspiring a lot of Indian techies to dive deep and try these things on, it will definitely make better quality made in india for the world systems at an unbelievable price point.
10:10 Arpit one question Isn't it risky that even Check constraints are removed from DB, bcoz maybe possible that DBA can accidentally run a insert or update query with Bad data.. This can lead to data corruption in DB if certain bad record make there way to the DB.. any thoughts ??
This is Asli Hardcore Engineering! Thanks a lot Arpit for bringing this amazing episode. Really amazed by the engineering practices at Razorpay. The way Arjun explained each and every detail was enlightening; definitely a skill to be learned by every software engineer. It is one of the best engineering podcasts and I got to learn so much!
A brilliant show. Thank you guys. One thing that I am still interested to know. I have seen so many people talks about a lot of requests we were getting. But there is no number. So if you could tell this in your next video. Then it would be really great.
Great watch. Couldn't understand few of the technology used as my experience is mostly related to MS stack. But it has generated lot of curiosity about the stack used at RazorPay. Looking for more such podcasts. Thanks Arjan/Arpit for this wonderful discussion.
Great video, one interesting question would’ve been what have been some issues with microservice architecture which still they’re in process of solving or need to evolve more
The interviewee is providing calm and composed answers. I liked the clarity and the interviewer has to improve and avoid rephrasing every answer for such.a long time
It’s really difficult to think that a foreign key validation at PHP (application) level would be faster than what MySQL can do. Database are optimized for these things.
Would love a 20 mins video where you can make notes about this conversation and explain. There were some terms and flows which i couldnt follow. CDC was new, sounds like ddb streams. Also the part where they ensure the ledger update was pretty cool but I couldnt understand it fully.
Hey Arpit, Do read LedgerStore paper by Uber. With that in mind, you will love the beauty of both razorpay and Ubers ledger system, especially the way uber migrated their db from aws to in house and writing to both db parallels to test thing out.
@@AsliEngineering Please do make video on same if possible. I think it is good enough for everyone to learn something from a large db migration. Love your content and open for any help you need (not for technical perspective as I am still learning and growing but any other task or any initiative) in growing your community.
When we said MySQL is high on consistency. which consistency we r referring here..ACID one or CAP one.. it sound like acid one but then you removed all the constraints from db.. which is bit conflicting @arujn
I have always wondered about this scenario with the foreign key/constraint validation being done in application code because I have seen a couple of threads where big companies don't use foreign key constraints. Consider a situation where you are persisting entity X (which must have a unique name) and it also holds a bunch of foreign refs(say Y,Z,..). Won't performing the validation in the application code involve querying the entire X table with the name projection (check unique), querying the Y,Z,.. other tables by ID? Why is this a better option than letting the relational database handle this ? As far as I understand it, this approach will make you perform more queries than required. What if there are millions of rows (say users table) in each of the tables ?
Payments data can be huge is rdbms enough to support the data , are you removing payments data periodically to some cold store from the transactional rdbms?
Deadlock not raising exception (Causing dirty ready) should this not be happening by default in all rdbms databases? Curious is this also the case in other db's which guarantee acid?. The last section on incidents and their resolution was good.
Good one Arpit and Arjun learned a lot, I am so stuck with AWS that can only relate to amazon specific service be it queue or notification topics, this is triggering an alarm for me to look into other similar products used at scale worldwide. PS:~ Not sure their UTs are with 95% code coverage; hahaha Arjun can reply and confirm.
Hmm, if im not wrong then every DB has to do foreign key check before insert, update or delete of each linked tables. This slow downs the entire thing.
Nice idea but validation on application will still need atleast the primary key on the table and transaction between app and db and will need to avoid any such case where there is m:n cardinality between entities if we have such cases then the only better way is to use relation tables where foreign key constraint is inevitable.
Additional way is to even manage that relation table manually through app instead of using foreign key on the relation table , but how efficient will it be is a question again. May be it more efficient as it has to do a lot less that db has to do.
You should continue this podcast series. Out of all the podcasts or technical tutorials, there is no one in this niche. Students or experienced professionals will learn a lot from this. Kudos to you.
Check out other videos on my channel. You will find them interesting.
Pp0
*timestamps:*
0:00 Introduction
2:48 Initial architecture/techstack of RazorPay
4:30 ensuring strong consistency on monolithic architecture
9:06 check constraints
10:47 deciding to move from monolith to microservices
15:23 splitting of databases/application logic when moving to microservices
18:25 dual writing of data into monolith as well as microservice dbs(during transition phase), transaction outbox pattern
21:19 databases in current microservice architecture
22:45 deciding on SQL vs NOSQL
25:10 ensuring consistency in microservice architecture
28:10 payment flow with eventual consistency
34:40 prioritization for settlements@T+0
37:20 chaos testing
39:00 integration tests on interface between 2 services
41:00 incidents when things went wrong
47:40 outro
Thank you so much for this.
The way you progress the communication through passively narrating and curiously listening made this podcast different from other existing system design podcasts.
This is truly asli engineering . Seedhi Baat no bakwaas, Real domain challenges and solution . I know that no architectures is The BEST. OutBox CDC ... amazing stuff ... please continue making stuff like this.
Such a technical dense podcast, rare to find such podcasts, nothing less than a technical conference talk. Great job Arpit !!
These 49.48 mins went pass like some couple of minutes.
This is so much learning and that also in such such basic terminology. That's one heck of a enginnering.
Thanks for the podcast Arpit and Arjun. I wish this series never comes to an end ❤️
Few things amazed me and I am not sure about whether its worth it.
1. Just for connection pooling they migrated monolith to microservice.
2. They remove foreign key constraints in their table and made custom orm.
I can surely say just connection pooling will be not the single reason.
Can’t state how insightful it was. It blew my mind. Discussion really took my understanding to next level.
Great podcast Arpit. Best thing I loved was that you were paraphrasing most of the things that Arjun said. I think you already know most of the things and still you sounded like a newbie with excitement who's listening about systems for the first time :)
This is one of the best Podcasts I have ever seen. Such deep and niche insights. Beauty of Engineering!
Real engineering hits more than any dopamine.
Got facinated by the number of DBs razorpay is using depends on seevice.
Really cool stuff with cool Engineer
Just one word.. Brilliant Podcast
This is really Asli Engineering Stuff. Worth watching every second of this podcast, can't skip anything. Very Insightful.
This style of system design discussion is lit and interesting af
There are bunch of videos on my channel on real-world System Design. You will definitely find them interesting.
As always superb content and also I had recently cracked offered from Razorpay and Its Interview Process is also Amazing.
loved this episode, so much to learn from razorpay. The data consistency section and ledger settlement is such a creative insight for anybody who hasn't worked on it yet. Thanks for recording this session and happy 1 month @google Arpit!!!
Keep this series going!!! So much knowledge in one video
I have a ton of videos on various topics that you'll find more interesting than this. Scroll through my channel once.
woow, my mind blows away, so much knowledge on a plate, lots of things to get to know about how the actual digital payment system works, which may appear simple from a normal person's perspective. Thank you, Arpit.
Loved every bit of it . Please continue this series .
Hi Arpit & Arjun,
I've been thoroughly enjoying your content and have been binge-watching your videos.
At 18:25, Arjun discussed implementing the outbox pattern for dual writes. While I understand the concept well (and have implemented something similar), I wanted to delve deeper into the actual implementation. The traditional use case for this pattern involves achieving ACD (a form of transactionality) for database writes and event publication in the message queue or a different DB.
Once the data is transactionally written to both the monolith table and the outbox table, an asynchronous job, using transactional log tailing (binlog consumption), allows us to consistently write to the microservice table.
However, let's consider a scenario where the broker is down (acting as a SPoF). How does the OUTBOX PATTERN ensure consistency in this situation?
In my implementation, instead of relying on binlog consumption, I've set up a cronjob (using a serverless service) to read from the outbox table and publish the event to the message queue. If it succeeds, it updates the status in the outbox table to prevent the next run of the cronjob from picking it up again.
I'd love to hear your thoughts.
Thank you so muchh arpit.... It is true knowledge that will help engineers in real time, who are newly taking you challenges.
Wow great video, such complex concepts put so simple without using a single diagram.
Good going arjun…
video aane se pehle hi dekh liya tumne
@@saumitrashukla591 ji shukla ji, it was open for sometime, you got late
Thanks Varun 😊
Pure gold content 👏
Such an amazing and detailed conversation. Absolutely loved it 💯
Wow Arpit....very knowledgeable session, Thanks for bringing him and the insights
You must continue this series.... Phenomenal insights... Waiting for more
Thanks for the video to both of you, the content is really phaad!!
Great conversation !! Covered almost end to end high level architecture, challenges etc.
need more podcasts like this
Wow! What a knowledge packed podcast and this is just ep1! So glad this exists! Really enjoyed conversations between both of you!
Excited for future eps! Thanks Arpit!
Really inspiring series of videos, arpit. Keep it going I am sure it is inspiring a lot of Indian techies to dive deep and try these things on, it will definitely make better quality made in india for the world systems at an unbelievable price point.
Brilliant Engineering by razorpay.
10:10 Arpit one question Isn't it risky that even Check constraints are removed from DB, bcoz maybe possible that DBA can accidentally run a insert or update query with Bad data.. This can lead to data corruption in DB if certain bad record make there way to the DB.. any thoughts ??
but if check constraints are becoming a bottleneck for you to scale then this is exactly what you have to do
Brilliant value addition to engineering contents, real stuff! ❤️
This is Asli Hardcore Engineering! Thanks a lot Arpit for bringing this amazing episode. Really amazed by the engineering practices at Razorpay. The way Arjun explained each and every detail was enlightening; definitely a skill to be learned by every software engineer. It is one of the best engineering podcasts and I got to learn so much!
Wow! what an amazing podcast. Looking forward for more such Videos.
Learned a lot from these 49 minutes! Hats off to the engineers!!
loved It, amazing things are doing by Razorpay Engineers. 😍
Informative session. Learnt so much!
i am very excited for this series.
#thanks#gratefulToArpit
Best engineering interview out there. loved it.
Awesome talk. Thanks for doing this.
A brilliant show. Thank you guys. One thing that I am still interested to know. I have seen so many people talks about a lot of requests we were getting. But there is no number. So if you could tell this in your next video. Then it would be really great.
Thanks Arpit Bhaiya for this awesome podcast.
Amazing video, both legends and true engineers
Brilliant podcast! Thank you arpit for doing this!!
gold content. I got to learn from this video a lot .Thanks Arpit .
So true!
Great start of the day.
Great watch. Couldn't understand few of the technology used as my experience is mostly related to MS stack. But it has generated lot of curiosity about the stack used at RazorPay. Looking for more such podcasts. Thanks Arjan/Arpit for this wonderful discussion.
Amazing stuff! Looking forward to more such podcasts.
Thanks arpit such a wonderful podcast
We really need these kind of podcasts. Thank you Arpit
Would recommend going through other videos of my channel.
Great video, one interesting question would’ve been what have been some issues with microservice architecture which still they’re in process of solving or need to evolve more
Was waiting for this one.
Really Enjoyed it
thanks
Need More Sir
Really loved this one, thanks Arpit for bringing such insightful engineering podcast
Very interesting and informative. Loved it.
The interviewee is providing calm and composed answers. I liked the clarity and the interviewer has to improve and avoid rephrasing every answer for such.a long time
Looking forward!!
It’s really difficult to think that a foreign key validation at PHP (application) level would be faster than what MySQL can do. Database are optimized for these things.
not about the speed. But having the ability to split.
It always amazes to see real world problems.
Thanks for this amazing video.
really insightful video , keep up the good work
Intial tech stack:Php, Larvel, Route 53, alb, mysql rds
Lots of engineering insights and technical points. Got to learn many new things.
Would love a 20 mins video where you can make notes about this conversation and explain. There were some terms and flows which i couldnt follow. CDC was new, sounds like ddb streams. Also the part where they ensure the ledger update was pretty cool but I couldnt understand it fully.
Great discussion. Enjoyed and got to learn new things. 🔥
As Always Arjun is great at explaining!
Hey Arpit,
Do read LedgerStore paper by Uber. With that in mind, you will love the beauty of both razorpay and Ubers ledger system, especially the way uber migrated their db from aws to in house and writing to both db parallels to test thing out.
Thanks a ton for sharing. Definitely going through it.
@@AsliEngineering
Please do make video on same if possible. I think it is good enough for everyone to learn something from a large db migration.
Love your content and open for any help you need (not for technical perspective as I am still learning and growing but any other task or any initiative) in growing your community.
Really insightful and learnt a lot.
Tazorpay walo ne abhi naya merchant ko onboard nhi kar raha hai 😭😭😭😭Muze phonepe or paym pg jaana pad raha hai.
Loved it.. insightful
Thankyou for providing this content ❤, really insightful
When we said MySQL is high on consistency. which consistency we r referring here..ACID one or CAP one.. it sound like acid one but then you removed all the constraints from db.. which is bit conflicting @arujn
Mind blown podcast 🤯Thanks Arpit. You really inspires many like me. 😁
Great video, we want more of these
I think one more step before dual write is to backup the existing data so that traffic can be served
Pure Gold
maja agaya sir bas 🙏
ase podcast kar te rahiye
superb… really learned lot of things from this session.🤘🏻
I have always wondered about this scenario with the foreign key/constraint validation being done in application code because I have seen a couple of threads where big companies don't use foreign key constraints. Consider a situation where you are persisting entity X (which must have a unique name) and it also holds a bunch of foreign refs(say Y,Z,..). Won't performing the validation in the application code involve querying the entire X table with the name projection (check unique), querying the Y,Z,.. other tables by ID? Why is this a better option than letting the relational database handle this ? As far as I understand it, this approach will make you perform more queries than required. What if there are millions of rows (say users table) in each of the tables ?
the orm does this only for data which they wouldn't have to query for.
Great podcast very informative, thanks
Very nice podcast , Enjoying the technology
just one word: wow!
Payments data can be huge is rdbms enough to support the data , are you removing payments data periodically to some cold store from the transactional rdbms?
Deadlock not raising exception (Causing dirty ready) should this not be happening by default in all rdbms databases? Curious is this also the case in other db's which guarantee acid?.
The last section on incidents and their resolution was good.
Curious about how uniqueness & race conditions regd database writes were achieved by moving db validations & constraints to application layers.
Good one Arpit and Arjun learned a lot, I am so stuck with AWS that can only relate to amazon specific service be it queue or notification topics, this is triggering an alarm for me to look into other similar products used at scale worldwide.
PS:~ Not sure their UTs are with 95% code coverage; hahaha Arjun can reply and confirm.
Hmm, if im not wrong then every DB has to do foreign key check before insert, update or delete of each linked tables. This slow downs the entire thing.
What a fantastic session 👌
Very informative content
How can you decide to which bank you should route request its chooseen by customer
Hi Arpit! Eager to listen to Dr. Kailash of Zerodha in such podcasts as well. Thanks!
Amazing 48:05
Please get this up on Spotify 🥺
AsliEngineering is already on Spotify. I think it's would be available on Spotify by default.
Will upload on it soon.
@@AsliEngineering also upload to other podcast platforms like Apple Podcasts, Amazon Music etc.
You can add visual explanations of some technical concepts as every one might not understand/know all the concepts.
Amazing discussion! :D
Being a part of debugging through the last issue mentioned would be lit af :D
Yeah. Even I got engrossed and tried to get max info on it 😀
All the possible links in the description but NO mention of the speaker's linkedin or socials.
It is there when posted on LinkedIn and Twitter. Missed adding it in description. Just search Arjun Tomar on LinkedIn and you'll find him.
@@AsliEngineering Thanks.
Instead of outboxer pattern cant we use read replica
Please get more such videos
Nice idea but validation on application will still need atleast the primary key on the table and transaction between app and db and will need to avoid any such case where there is m:n cardinality between entities if we have such cases then the only better way is to use relation tables where foreign key constraint is inevitable.
Additional way is to even manage that relation table manually through app instead of using foreign key on the relation table , but how efficient will it be is a question again. May be it more efficient as it has to do a lot less that db has to do.