Tracing SSL/TLS Encrypted Microservices with eBPF - Dom Del Nano , Twitter
ฝัง
- เผยแพร่เมื่อ 21 ส.ค. 2024
- Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Tracing SSL/TLS Encrypted Microservices with eBPF - Dom Del Nano , Twitter
SSL/TLS adoption in the Cloud Native environments is growing rapidly. While great for security, the encryption in such environments pose a unique challenge for observability tools. Many traffic sniffing tools can only collect the encrypted data, which is of limited value to the application developer. Important attributes like the operation, the endpoint and the payload are undecipherable. To truly help in the troubleshooting process, application developers need to be able to see these messages and their contents. In this talk, we present how eBPF can be used to tracing SSL/TLS connections. The method we present is used by tools like BCC’s sslsniff and Pixie’s protocol tracer. Specifically, we cover how eBPF uprobes can be attached to popular SSL/TLS libraries, including OpenSSL, BoringSSL and goTLS. We show how eBPF enables us to collect clear text data directly from the TLS library, while discussing the challenges of tracing dynamically vs statically linked TLS libraries. Finally, we also present how this feature could help with improving application observability at some of the largest engineering organizations without disrupting their production environment.
In case of service mesh approach like bastion(tls) ( meshProx(mTls) service(not encrypted ) ) I can do an standard obeservation. Am I wrong? What is your general experience with service meshes ?
If the traffic is in plaintext at any point in the path, which would be the case for the situation you described, then this tracing technique wouldn't be necessary. However, at Twitter we don't use side cars because finagle has an intelligent client side load balancer within it. Therefore the traffic is encrypted end to end.
If there is traffic that bypasses the service mesh, such as requests to Internet facing services, then you would still benefit from tracing the fully encrypted traffic.