Stop messing up your stream processing joins! | Systems Design Interview 0 to 1 with Ex-Google SWE
ฝัง
- เผยแพร่เมื่อ 15 ส.ค. 2023
- Remember ladies and gents - when someone presents you with their stream, make sure you keep it in your memory.
- วิทยาศาสตร์และเทคโนโลยี
Congratulations for 10K Jordan!
Congratz with your data intensive applications youtube channel
Thank you Martin kleppman has no idea
Can you review some cases where we’d do table to table joins with streams? Is it mainly to offload work from the db(separation of consumers and db)? Or improve performance?
If you want realtime joins that actually update for you, it would be useful.
Think of a books table and an author's table. They're both big, so you don't want to query them both many times over.
Maybe a new book gets added, and it turns out many authors had written to it, some of which were already in the table. Let's do a join.
Maybe a new author gets added, and it turns out they contributed to many books in the table. Now we can only fetch the books we need without having to redo a full join.
Hi thanks for mentioning the Issues in depth at the end, was about to post a comment asking for how will InMemory in all Consumers work and more.
Just as a side note if this problem was to be done without Flink or any tool as such, is there are any other approach?
I think you'd probably end up reinventing the wheel - taking distributed snapshots is really expensive so flink found a great way to do it without hugely impacting performance
@@jordanhasnolife5163makes sense
Hi @jordan Awesome video is there a video on CDC deep dive? Is it a queue push after db save ? and if yes then it is similar to 2-phase commit , isnt it?
Similar, however keep in mind that the push to the queue is *from* the db! And we don't need that push to happen for the write to be committed to the database. If the db goes down, or can't communicate to the queue, it's ok if we place those writes there later. That's the main difference.
Would it make sense to have to query the db as an option for data that is not in memory and there is no CDC event in the queue for that ID? If you have several TBs of data in DB, maintaining such a large number of consumers may be rather costly, right(As RAM is costly)? Is there a more cost-effective solution?
You could theoretically store the flink data on disk I believe, but yeah if latency isn't a main concern you could always just query a db
So if the in-memory tables are huge you mentioned we can partition the info coming into multiple queues - does this mean we need to maintain multiple consumers with each consuming from a specific queue?
That's correct!
I noticed that a majority of your designs lean heavily towards kafka + Flink combination. Is this a personal choice, or do you run your designs by Senior Engineers? Don't get me wrong, these designs are not textbook-y, they are real and heavily tend towards what is "in" now (as against Alex xu's designs which seem, for lack of a better word, a bit dated.
I have 17 years of experience in the industry (I am a fossil), but I get to learn many things from your discussions. This is incredibly rare for someone with 2 years of experience. Thanks for everything that you do!
Mainly a personal choice, but I experience the pitfalls every day at work of using non replayable message brokers, so I'm definitely pro log based mq's and stateful consumers whenever I notice the opportunity to use them.
That being said, I understand in practice this may have a high cost to implement due to storage and or latency and many people might compromise and opt for fewer partitions/simpler solutions. My designs are generally pretty idealized and not at all optimized for cost, which is why you don't see stuff like this too much IRL.
@@jordanhasnolife5163 On the contrary, I honestly believe these *are* real world and not textbook-y and cookie-cutter like "Grokking". If someone were to make these systems in 2023, they would choose this tech, as against say 2017-18 when the "system design" questions became mainstream.
do you make videos after waking up and smoking one joint?
No I do them right between when I smoke crack and go to bed
congrats on 10k hottie
I literally spontaneously combusted
where do i buy the foot picks?
You don't actually buy them, I just give them out in exchange for job referrals
video from jordan finaaaaaaaaaaaaaaaaaaalllllllllyyyyyyyyyy😍😍😍😍😍