P99 CONF 2023 | Cost-Effective Burst Scaling For Distributed Query Execution by Dan Harris

Go to www.p99conf.io/ for P99 CONF talks on demand and learn more.
. . . . .
Building a query engine that scales efficiently is a difficult task. Queries over big datasets stored in Object Storage require a large amount of IO and compute power. Keeping the latency of expensive queries acceptable when using a fixed size compute cluster is only possible when over-provisioning a cluster, while dynamically up- or downscaling is too slow for interactive queries.
To overcome these challenges, we built a distributed execution model which allows us to dynamically execute across both AWS Lambda and EC2 resources. With this model we can shed excess load to lambda functions to preserve low latency while we scale EC2 capacity to manage costs.
. . . . .
P99 CONF is the technical conference for anyone who obsesses over high-performance, low-latency applications. Engineers from Netflix, Google, Meta, Twitter, TikTok, Uber + more will be sharing 50+ talks on topics like Rust, Go, Zig, distributed data systems, Kubernetes, edge, and AI/ML. Learn from experts like Gwen Shapira, Bryan Cantrill, Jon Haddad, Armin Ronacher, Liz Rice & more - for free, from anywhere. Learn more and see P99 CONF talks on www.p99conf.io/