Migrating Hybrid Big Data Platforms at Scale to Kubernetes - Sunil Govindan, Cloudera

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon Europe in Paris from March 19-22, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at kubecon.io
Migrating Hybrid Big Data Platforms at Scale to Kubernetes - Sunil Govindan, Cloudera
A significant challenge in managing bigdata workloads is the ability to extract accurate business insights from data consistently. Although K8s is a widely accepted platform for running stateful workloads in containerized form, legacy big data workloads are struggling to transition to containerized K8s due to gaps in enterprise-ready big data management. In this talk, we will share how Cloudera is pioneering the path toward K8s from legacy Hadoop with open-source technologies around resource management and multi-tenancy architecture. Most big data applications are closely integrated stateful microservices, and migrating them to enterprise-hardened multi-tenant platform is challenging. We will showcase a case study that highlights challenges of managing noisy workloads in a shared multi-tenant K8s cluster at scale. Topics covered will include concurrency-based autoscaling, latencies in disaggregated storage from compute, and an effective hierarchical quota management solution for K8s.