Mohit Kumar - Billion Rows Walk Into a Go Bar

In this talk from GopherCon India 2024, we delve into the quest for extreme performance optimization by tackling the 1BRC (1 Billion Row Challenge). Discover how a program initially taking several minutes to complete was transformed to run in just a few seconds. Along the way, we’ll explore how to move beyond abstractions and deal directly with data at massive scale.
We’ll examine practical techniques, such as using pprof for pinpointing memory allocations, garbage collection overhead, and performance bottlenecks. Learn about the subtle code changes-like one that cut memory allocations from 15+GB down to a mere 2MB-and gain insights that you can immediately apply to your own Go projects.
By watching this talk, you’ll:
Develop “mechanical sympathy” when handling large-scale data.
Understand how pprof can identify the hidden bottlenecks lurking in your code.
Learn to write more efficient and performant Go programs in your everyday development work.