- 164
- 521 894
The Geek Narrator
เข้าร่วมเมื่อ 15 พ.ค. 2022
The GeekNarrator is here to make you curious, excited and inspired about Technology and Software Engineering. You will see in depth technical discussions with actionable insights from the industry experts. These insights will help you get better at Software Engineering, as it comes from the real world engineering experience.
Practical Systems Learning & Verification with Jack Vanlightly
Practical Systems Learning & Verification with Jack Vanlightly
Welcome to The GeekNarrator podcast! In this episode, host Kaivalya Apte goes deeper into the practical applications of formal methods with Jack Vanlightly, a principal technologist at Confluent. With years of experience in distributed systems, Jack discusses his journey and how formal methods have been instrumental in system design verification and bug detection. The conversation covers Jack's background, his process of using formal methods, the significance of modelling, verification, documentation, and systems learning, as well as the future evolution of tooling and its applications. Tune in to understand the intricacies of how formal methods can transform your approach to distributed systems!
Chapters:
00:00 Introduction to the episode
00:37 Meet Jack VanLightly: Principal Technologist at Confluent
02:17 Jack's Journey into Distributed Systems
04:29 Discovering the Power of Formal Methods
08:11 Modeling and Simulation in Formal Methods
13:43 Verification and Safety Properties
19:02 Documentation and Communication Challenges
20:43 Formal Methods as a Systems Learning Tool
24:26 Practical Applications and Case Studies
56:38 Future of Formal Verification and Closing Thoughts
Jack's Blog: jack-vanlightly.com/
Become a member of The GeekNarrator to get access to member only videos, notes and monthly 1:1 with me.
Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription.
app.codecrafters.io/join?via=geeknarrator
If you like this episode, please hit the like button and share it with your network.
Also please subscribe if you haven't yet.
Database internals series: th-cam.com/video/yV_Zp0Mi3xs/w-d-xo.html
Popular playlists:
Realtime streaming systems: th-cam.com/play/PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-.html
Software Engineering: th-cam.com/play/PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17.html
Distributed systems and databases: th-cam.com/play/PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d.html
Modern databases: th-cam.com/play/PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN.html
Stay Curios! Keep Learning!
Welcome to The GeekNarrator podcast! In this episode, host Kaivalya Apte goes deeper into the practical applications of formal methods with Jack Vanlightly, a principal technologist at Confluent. With years of experience in distributed systems, Jack discusses his journey and how formal methods have been instrumental in system design verification and bug detection. The conversation covers Jack's background, his process of using formal methods, the significance of modelling, verification, documentation, and systems learning, as well as the future evolution of tooling and its applications. Tune in to understand the intricacies of how formal methods can transform your approach to distributed systems!
Chapters:
00:00 Introduction to the episode
00:37 Meet Jack VanLightly: Principal Technologist at Confluent
02:17 Jack's Journey into Distributed Systems
04:29 Discovering the Power of Formal Methods
08:11 Modeling and Simulation in Formal Methods
13:43 Verification and Safety Properties
19:02 Documentation and Communication Challenges
20:43 Formal Methods as a Systems Learning Tool
24:26 Practical Applications and Case Studies
56:38 Future of Formal Verification and Closing Thoughts
Jack's Blog: jack-vanlightly.com/
Become a member of The GeekNarrator to get access to member only videos, notes and monthly 1:1 with me.
Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription.
app.codecrafters.io/join?via=geeknarrator
If you like this episode, please hit the like button and share it with your network.
Also please subscribe if you haven't yet.
Database internals series: th-cam.com/video/yV_Zp0Mi3xs/w-d-xo.html
Popular playlists:
Realtime streaming systems: th-cam.com/play/PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-.html
Software Engineering: th-cam.com/play/PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17.html
Distributed systems and databases: th-cam.com/play/PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d.html
Modern databases: th-cam.com/play/PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN.html
Stay Curios! Keep Learning!
มุมมอง: 268
วีดีโอ
Database Internals - NileDB Postgres re-engineered for multitenant apps
มุมมอง 19K14 วันที่ผ่านมา
Database Internals - NileDB: Postgres Re-engineered for Multitenant Apps with Gwen Shapira Join us in this episode as we dive deep into the intricacies of NileDB, a groundbreaking database re-engineered for multi-tenant applications. Our special guest, Gwen Shapira, co-founder of NileDB and a notable figure in the database field, shares her insights and technical know-how on solving the common ...
AutoMQ: A New Kafka Alternative on S3 That Addresses Your Costs and Pain Points
มุมมอง 1.1K21 วันที่ผ่านมา
In this video, we dive into AutoMQ, an interesting streaming project that reinvent Apache Kafka’s storage layer to gain many attractive features, mainly including: - 10x cost-effective than Apache Kafka: AutoScaling、Support Spot Instance、Separate Storage to S3. All this make AutoMQ 10x cost-effective than Apache Kafka. - Easy to operate: No need to manage the cluster's capacity yourself. Statel...
Building a continuous profiler with Frederic from Polar Signals
มุมมอง 15K21 วันที่ผ่านมา
Building a Continuous Profiler with Frederic from Polar Signals | Geek Narrator Podcast In this episode we chat with Frederic from Polar Signals. We dive deep into the intricacies of building a continuous profiler, the challenges faced, and the unique solutions developed by Polar Signals. Frederic shares insights from his background in observability and discusses the innovations in FrostDB, a c...
Database Internals - SlateDB with Chris Riccomini
มุมมอง 14Kหลายเดือนก่อน
Welcome back to another episode! Today, I have a special guest, Chris Riccomini, joining me to delve into the exciting world of databases. In this episode, we focus on SlateDB, a new and innovative database that's making waves in the tech community. We'll cover a wide range of topics, including the architecture of SlateDB, its internals, design decisions, and some fascinating use cases. Chris, ...
System Design the formal way with FizzBee
มุมมอง 793หลายเดือนก่อน
In this video I talk to Jayaprabhakar Kadarkarai aka JP who is the founder of FizzBee. FizzBee is a design specification language and model checker to help developers verify their design before writing even a single line of implementation code. We have discussed where it is applicable, what are the benefits, how does it work and many other interesting challenges with examples. Chapters: 00:00 I...
Learnings from building Open Source Distributed Systems with Kishore Gopalakrishna
มุมมอง 29K3 หลายเดือนก่อน
In this episode of The Geek Narrator podcast, hosted by Kaivalya Apte, we welcome a special guest, Kishore Gopalakrishna from StarTree, co-author of Apache Pinot and other notable projects. Kishore shares his extensive experience in building real-time analytics and streaming systems, including Apache Pino, Espresso, Apache Helix, and Third Eye. The episode delves into the motivations and challe...
WarpStream: A drop-in replacement for Kafka
มุมมอง 13K4 หลายเดือนก่อน
In this episode of The GeekNarrator podcast, host Kaivalya Apte interviews Ryan and Richie, the founders of WarpStream. They discuss the architecture, benefits, and core functionalities of WarpStream, a drop-in replacement for Apache Kafka. The conversation covers their experience with Kafka, the design decisions behind WarpStream, and the operational challenges it addresses. They also delve in...
XTDB - An Immutable SQL Database
มุมมอง 11K4 หลายเดือนก่อน
Exploring XTDB with Jeremy Taylor & Malcolm Sparks: An In-Depth Dive into Immutability and Database Internals In this episode of the Geek Narrator Podcast, host Kaivalya is joined by Jeremy Taylor and Malcolm Sparks from Juxt to explore XTDB, an immutable database designed to handle complex historical and financial data with precision. They delve into the architecture, internal mechanics, and u...
Turso - SQLite for production
มุมมอง 17K4 หลายเดือนก่อน
Exploring Turso with Glauber Costa: Insights on SQLite for Production In this episode of The GeekNarrator podcast, host Kaivalya Apte interviews Glauber Costa, founder and CEO of TursoDB. They discuss the inception of TursoDB, Glauber's background in Linux kernel development, and the journey from unikernel projects to founding a database company. Glauber explains TursoDB's enhancements to SQLit...
Testing Distributed Systems the right way ft. Will Wilson
มุมมอง 8K5 หลายเดือนก่อน
In this episode of The GeekNarrator podcast, host Kaivalya Apte dives into the complexities of testing distributed systems with Will Wilson from Antithesis. If you’re grappling with the challenges of testing databases, micro-services, and distributed systems, this episode is a must-watch. Will Wilson demystifies the concept of deterministic simulation testing, shares insights about its advantag...
Taking Postgres to the next level with Neon
มุมมอง 18K5 หลายเดือนก่อน
Deep Dive into Serverless Databases with Neon: Featuring Heikki Linnakangas In this episode of the Geek Narrator podcast, host Kaivalya Apte is joined by Heikki Linnakangas, co-founder of Neon, to explore the innovative world of serverless databases. They discuss Neon's unique approach to separating compute and storage, the benefits of serverless architecture for modern applications, and dive i...
Scaling Derived Data for Planet-Scale Applications at Linkedin
มุมมอง 4825 หลายเดือนก่อน
In this video I speak with Felix GV, who is a Principal Staff Engineer at Linkedin, and has done major contributions to the data infrastructure and Linkedin, including VeniceDB. This episode will give you a good understanding of why we need a new database for storing "Derived Data" in a low latency, high performance manner, which is very important for Machine Learning workloads. Chapters: 00:00...
SuperCharging PostgreSQL for Search and Analytics - ParadeDB (Philippe Noël)
มุมมอง 18K5 หลายเดือนก่อน
SuperCharging PostgreSQL for Search and Analytics - ParadeDB (Philippe Noël)
Modern OLAP Database System Design with FDAP (Andrew Lamb)
มุมมอง 14K5 หลายเดือนก่อน
Modern OLAP Database System Design with FDAP (Andrew Lamb)
The ultimate multi-model Database, SurrealDB with Pratim Bhosale
มุมมอง 7816 หลายเดือนก่อน
The ultimate multi-model Database, SurrealDB with Pratim Bhosale
Demystifying Real-time Analytics, Search and Hybrid Search with Dhruba, CTO @Rockset
มุมมอง 3.6K6 หลายเดือนก่อน
Demystifying Real-time Analytics, Search and Hybrid Search with Dhruba, CTO @Rockset
Rapidly Simulate Production Traffic ft. Michael Drogalis
มุมมอง 6436 หลายเดือนก่อน
Rapidly Simulate Production Traffic ft. Michael Drogalis
Demystifying Stream Processing Fundamentals
มุมมอง 7466 หลายเดือนก่อน
Demystifying Stream Processing Fundamentals
High Performance with GraalVM - Alina Yurenko
มุมมอง 5K6 หลายเดือนก่อน
High Performance with GraalVM - Alina Yurenko
Taming TimeSeries Data with QuestDB - Javier Ramirez
มุมมอง 4157 หลายเดือนก่อน
Taming TimeSeries Data with QuestDB - Javier Ramirez
Beat the CAP Theorem : Make Distributed consistency simple
มุมมอง 1.6K7 หลายเดือนก่อน
Beat the CAP Theorem : Make Distributed consistency simple
KuzuDB: The Future of Graph Databases (w/ The Founder of KuzuDB)
มุมมอง 1K7 หลายเดือนก่อน
KuzuDB: The Future of Graph Databases (w/ The Founder of KuzuDB)
Restate - making distributed systems simple with Stephan Ewen
มุมมอง 7757 หลายเดือนก่อน
Restate - making distributed systems simple with Stephan Ewen
Volt Active Data: Low Latency Stream processing
มุมมอง 4948 หลายเดือนก่อน
Volt Active Data: Low Latency Stream processing
Stream Processing 101 - An Introduction
มุมมอง 1.3K8 หลายเดือนก่อน
Stream Processing 101 - An Introduction
TigerBeetle: World’s Fastest Financial Transactions Database
มุมมอง 2.9K8 หลายเดือนก่อน
TigerBeetle: World’s Fastest Financial Transactions Database
Durable Async-Await with Dominik Tornow
มุมมอง 6649 หลายเดือนก่อน
Durable Async-Await with Dominik Tornow
Observability Engineering with Liz Fong-Jones
มุมมอง 1.3K9 หลายเดือนก่อน
Observability Engineering with Liz Fong-Jones
This was awesome!
Thanks a lot Vipul 🙏🏻
Heikki rocks!
Your podcasts and the way you dig down to detailed level is revolutionary...please keep making such insightful videos ...this has enhanced my knowledge many folds .. let's catch up if you want
Please remove members only videos. It doesn't make sense yet for your channel.
Thanks for the feedback. I am happy you added “yet”, which tells you believe 😀🙏🏻
😅o
The proper way is to rewrite using native languages like C++ or Rust. Use of java again is a plain joke.
Language has never been an issue with Kafka. Embracing the Kafka ecosystem and being fully compatible with Kafka is an important trade-off.
@@kaimingwan526 Efficiency is the issue. It translates to number of resources you have to pay for monthly. That's why Redpanda exists.
Thank you Kaivaly for the video, it was really well explained and the animations are amazing. I would like to point just one thing the title "Reinventing Kafka the right way" I think is not precise once the part that AutoMQ are improving is the Storage Engine system and not the Message Middleware Broker (if I understood correctly based on this video). It was more an opportunity to enhance an isolated component from Kafka for cloud storage instead a full rewrite. An example of "Reinventing the right way" in my point of view would be the ScyllaDB which has backwards compatibility with Cassandra although it was rewritten using C++. I wish the best brow keep going!!!! EDIT 1: And they created a platform over na existent Kafka concept
Thanks for watching and the feedback. I appreciate it. I think you made a fair point, I guess re-engineered would have been a more precise word. I will change that.
Great video, learnt a lot!
Thanks for this interview. Great as always! One question: so if the WAL writes are batched and periodically flushed to storage, there in a window in which if the process crashes, the writes will be lost? I.e the client response that write is complete is not durable.
Thanks for watching. Yes, there are ways to configure the behaviour. This might help you slatedb.io/docs/faq/#what-happens-if-the-process-goes-down-before-slatedb-flushes-data-to-object-storage
Yeah that’s the “linger” stuff they spoke about. The client won’t get an ack about the write being successful until the flush happens. Therefore it would be as if the write request never happened (in practice the client would retry)
The storage architecture reminds me of AWS RDS (Aurora) Architecture with the control plane managing a muli tenant storage fleet and having write quoram for strong consistency. Great discussion, loved it.
Wonderful video
Information dense conversation! And isn't it expected that size is sufficiently large then it would increase number of put calls to object storage. Why don't you do buffering based on time plus size based criteria?
Good question. I think there are plans but haven’t really finalised yet. But maybe in the future a hybrid setting will be available. IIRC I asked this question to Chris.
Thank you for this, looking forward to assisting with the golang port! This was a great overview of the project, and a great way to get acquainted with some of the design decisions.
Thanks for watching 🙏🏻
I love how you simplify complex topics in the crypto space.
Like all duckdb users, I really enjoy hearing Mark Raasveldt talk. You did a great job of guiding the conversation into intesting and informative areas!
Thanks a lot watching and sharing your thoughts 🙏🏻😀
This is super cool . Probably the only youtube video [> 1hr ] , which I watched in whole. I took a whole week to watch it ... made sure I understood every idea presented and discussed. Thank you so much. I have a question => The partition placement , is DDB still using consistent hashing , for deciding which partition goes to which node[physical server] ... or there is some other algorithm used now. The original Dynamo paper mentioned consistent hashing [I read half the paper] ... Precisely => 1. How is it decided which key goes to which partition ? 2. How is it decided which partition goes to which node[physical server] ? Consistent hashing ... works in a bit different way ... all of the people watching this video knows how . For example in normal consistent hashing I guess it is hard to keep a limit on partition size ... I have read some part of the original consistent hashing paper ... in my quest to understand that paper ... somehow I reached your video .. and I thank myself for that.
Well presented and explained well. The best part when Matthias use pointer to exactly mention what part in presentation is under discussion. Thanks 👏
❤
Thanks 🤔
Min-blowing.
@TheGeedNarrator Great content, thanks! However, requesting you to not highlight the current word in the closed captions. It is distracting, especially in the selected color scheme. Had to zoom-in the browser window till the captions disappear.
Thanks for watching and sharing your feedback. I have addressed that in all my recent videos.
@@TheGeekNarrator Great, thanks! Please keep up the good work. Already subscribed!
🙏🏻 thanks. Really appreciate it
Brilliant one. I used to work on ES in FAANG, and can totally relate to this one
Thanks 🙏🏻
Always lot of things to learn from Arpit 👆👆
Truly awesome insights!! Thanks a lot, these are really informative (love to understand more like these)
Thanks for watching 🙏🏻
Absolutely loved the conversation. Thank you
Thank you for watching 🙏🏻
Dhruba seems such a chill guy. Very passionate about his work too. Amazing podcast.
Thanks a lot 🙏🏻😀
Very technical and insightful. Thanks for bringing great folks to the podcast. 🎉
Thanks Abhishek 🙏🏻😀
Is there a version without the blinking text at the bottom ? It is extremely distracting
Sorry for that. I have removed the captions from all my recent videos. Here is an alternative you can choose to listen, but watching is the same experience. podcasters.spotify.com/pod/show/kaivalya-apte/episodes/TigerBeetle-Worlds-Fastest-Financial-Transactions-Database-e2g5qur
@@TheGeekNarrator Thank you ! The content is very good, i moved the video frame down so its out of sight. Btw the tech behind the subtitles is awesome and quality is great, it was just very distracting when trying to just watch.
Great podcast, hats off to you for doing this consistently. One suggestion is to have some kind of architecture diagram maybe to help understand better while explaining things. I know it could be too much to ask but if it's feasible please try to do that.
Thanks Sudhanshu. Your feedback is noted.
Amazing 🔥🔥
I saw a small video of tiered storage in StarTree channel but this video is something that is much needed for in-depth understanding of what’s going on. Kudos to Neha for explaining it so clearly.
Great interview!
the initial story about how turso was started was awesome!
why couldn't LiteFS just observe writes to .wal and propagate only those like you are doing at Turso? Also ... if I understood correctly the primary is location based so you'll have fast primary in Europe, as example, but writings from US will inevitably need to end up there first, right? I think this is also common for other DBs, but I just would like to be confirmed reads are always fast (once in each replica) but writes are also "primary location latency" based. Is this correct? Last, but not least, please don't talk trash on JS, it's pretty awesome after all :P
this is great product .. i will give it a try.. also get quine folks @TheGeekNarrator . think of graph with triggers
Great video, Thank you!
❤❤❤❤ great talk!
Great episode, highly informative regarding search, RDBMs, abd everything in between
Great episode, I’m a fan of arrow and datafusion
Thank you 🙏🏻😀
This was completely amazing!! Thanks a ton
Thanks 🙏🏻
What is the name of the app you using for drawings?
Goodnotes
Super cool
Dropped all the things no marriage no function only work
Everything wonderful especial the quest. But please do not include this annoying subtitles. When I will then I enable the once from TH-cam. You're can I not disable. And when I am trying to concentrate on the story, they really annoing
Thanks for watching and your feedback. This has been a feedback by many folks so I have removed the captions from the latest episodes. I hope this won’t be a problem going forward.
@@TheGeekNarratori disagree to him, the subtitle helps a lot for non-english speaker in understanding the podcast. If you will, at least, please insert it in the caption feature (not the auto-translate one), so we can still activate or deactivate the subtitle
Absolute fruitful discussion, Loved it. looking forward for master Template of Envelope of Estimation :)
What a Masterclass with the Master! Its a great summary of cassandra documentation and parts of it are covered in DDIA. Also, observe how the master crafts concise and clear explanations to the questions. great questions too Kaivalya - Loved it
thanks a lot for this one, keep going, you are doing really well
Thanks a lot 🙏🏻😀
cozodb is also interesting
Thanks - I didn’t know about it.
Thank you, looking forward for more videos about Cassandra under layers