- 53
- 20 874
Container Bytes
United States
เข้าร่วมเมื่อ 22 พ.ค. 2023
Containers can be complex. We are here to make it simple one byte at a time.
Improve Infrastructure Autoscaling with Custom Compute Classes in GKE
In this video we'll explore the new custom compute class feature in GKE, which improves infrastructure autoscaling with fall-back compute priorities and advanced node configuration in a new declarative, CRD-based API.
For more information see: cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes
Blog Post: cloud.google.com/blog/products/containers-kubernetes/introducing-new-gke-custom-compute-class-api/
For more information see: cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes
Blog Post: cloud.google.com/blog/products/containers-kubernetes/introducing-new-gke-custom-compute-class-api/
มุมมอง: 194
วีดีโอ
GPU Sharing on GKE with Multi Instance GPU
มุมมอง 8721 วันที่ผ่านมา
The A100 GPU and H100 GPU consist of seven compute units and eight memory units, which can be partitioned into GPU instances of varying sizes. Overview of all GPU Sharing Techniques on GKE: th-cam.com/video/Y7hdy-mFLlU/w-d-xo.html GKE Multi Instance GPU: cloud.google.com/kubernetes-engine/docs/how-to/gpus-multi NVIDIA Partitioning Table: docs.nvidia.com/datacenter/tesla/mig-user-guide/index.htm...
Different ways of Running RayJob on Kubernetes
มุมมอง 9221 วันที่ผ่านมา
RayJob is a Kubernetes Custom Resource Definition that combines both the Ray Job (A packaged ray application) and the submitter (something that sends a Ray Job to a RayCluster). In this video Kent and Ali covers the different ways of running a RayJob on Kubernetes using Persistent or Ephemeral clusters.
Simplify Kuberay with Ray Operator on GKE
มุมมอง 89หลายเดือนก่อน
KubeRay is a powerful, open-source Kubernetes operator that simplifies the deployment and management of Ray applications on Kubernetes. Ray Operator addon on GKE makes it super easy to install and setup Ray on GKE. In this video Mofi Rahman and Ali Zaidi walks through the steps to install Ray Addon on an existing cluster and setting up an Ray Cluster. Simple Command To Run Ray Job: ray job subm...
GKE Multi Tenancy with Teams
มุมมอง 59หลายเดือนก่อน
In this video Nick walks us through another GKE Enterprise feature, Teams.
Fleet Level Feature Management with Feature Manager
มุมมอง 3332 หลายเดือนก่อน
The Feature Management view lets you view the state of GKE Enterprise features for your fleet clusters. In this video Nick goes over what Feature Manager is and the key features you can enable in your fleet. Intro to GKE Enterprise: th-cam.com/video/sXcA-Zsxvvw/w-d-xo.html
Build Internal Developer Platforms on GKE using GKE Enterprise
มุมมอง 2423 หลายเดือนก่อน
In this video Nick Eberts walks us through how to build a Internal Developer Platform on top of GKE using GKE Enterprise. The topics covered in this video are, GKE Enterprise, Fleets and Rollout Sequencing. Learn About GKE Enterprise: cloud.google.com/kubernetes-engine/enterprise/docs/concepts/gke-editions Fleets: cloud.google.com/kubernetes-engine/docs/fleets-overview Rollout Sequencing: cloud...
Tips for Securing your Ray Cluster on GKE
มุมมอง 2253 หลายเดือนก่อน
As a distributed compute framework for AI applications, Ray has grown in popularity in recent years, and deploying it on GKE is a popular choice that provides flexibility and configurable orchestration. Learn how to secure Ray on GKE in this video. How to secure Ray on Google Kubernetes Engine: cloud.google.com/blog/products/containers-kubernetes/securing-ray-to-run-on-google-kubernetes-engine ...
Effective GPU Sharing Strategies in GKE
มุมมอง 2423 หลายเดือนก่อน
GPU sharing strategies allow multiple containers to efficiently use your attached GPUs and save running costs. GKE provides the following GPU sharing strategies: - Multi-instance GPU - GPU time-sharing - NVIDIA MPS Learn More: cloud.google.com/kubernetes-engine/docs/concepts/timesharing-gpus
Serving Gemma on GKE on TPU using Jetstream
มุมมอง 1433 หลายเดือนก่อน
Gemma is a family of lightweight, state-of-the-art open models built from the same research and technology used to create the Gemini models. JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs In this video Mofi Rahman and Ali Zaidi walks through the process of deploying Gemma on GKE on TPU using JetStream. Find Gemma on Huggingface - huggi...
Improve Resource Obtainability (GPUs, TPUs) with Dynamic Workload Scheduler on GCP
มุมมอง 2104 หลายเดือนก่อน
Dynamic Workload Scheduler is a resource management and job scheduling platform designed for AI Hypercomputer. Dynamic Workload Scheduler improves your access to AI/ML resources, helps you optimize your spend, and can improve the experience of workloads such as training and fine-tuning jobs, by scheduling all the accelerators needed simultaneously. Dynamic Workload Scheduler supports TPUs and N...
Reducing data pre-processing time by 95% using Ray
มุมมอง 1.5K5 หลายเดือนก่อน
In ML your model is as good as your data. Often this crucial step is overlooked. In this video Google Solutions Architect Shobhit and Kavitha walks through a reference solution where they managed to bring down data pre-processing of downloading over 20,000 data points each with multiple image URLs from 8 hour to just about 20 minutes using Ray workers. Ray is an open-source unified compute fram...
Serving Gemma on GKE using Nvidia TRT LLM and Triton Server
มุมมอง 7096 หลายเดือนก่อน
Gemma is a family of lightweight, state-of-the-art open models built from the same research and technology used to create the Gemini models. Nvidia TensorRT is an SDK for high-performance deep learning inference, includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for inference applications. Triton Inference Server is an open source inference s...
Serving Gemma on GKE using Text Generation Inference (TGI)
มุมมอง 4456 หลายเดือนก่อน
Gemma is a family of lightweight, state-of-the-art open models built from the same research and technology used to create the Gemini models. Text Generation Inference (TGI) is a toolkit for deploying and serving Large Language Models (LLMs). In this video Mofi Rahman and Ali Zaidi walks through the process of deploying Gemma on GKE using TGI serving engine. Find Gemma on Huggingface - huggingfa...
Serving Gemma on GKE using vLLM
มุมมอง 5296 หลายเดือนก่อน
Gemma is a family of lightweight, state-of-the-art open models built from the same research and technology used to create the Gemini models. vLLM is a fast and easy-to-use library for LLM inference and serving. In this video Mofi Rahman and Ali Zaidi walks through the process of deploying Gemma on GKE using vLLM serving engine. Find Gemma on Huggingface - huggingface.co/google Follow along the ...
Improve LLM accuracy and performance with Retrieval Augmented Generation
มุมมอง 1.3K7 หลายเดือนก่อน
Improve LLM accuracy and performance with Retrieval Augmented Generation
Monitoring ML Training Platform using Kueue Metrics and Cloud Monitoring
มุมมอง 2117 หลายเดือนก่อน
Monitoring ML Training Platform using Kueue Metrics and Cloud Monitoring
Architecture of a ML Platform with Resource Sharing on Kubernetes
มุมมอง 4908 หลายเดือนก่อน
Architecture of a ML Platform with Resource Sharing on Kubernetes
Serve LLM on Google Kubernetes Engine on L4 GPUs
มุมมอง 4448 หลายเดือนก่อน
Serve LLM on Google Kubernetes Engine on L4 GPUs
Building a Batch Platform on Kubernetes
มุมมอง 22511 หลายเดือนก่อน
Building a Batch Platform on Kubernetes
What is HPC and Overview of a HPC architecture
มุมมอง 87211 หลายเดือนก่อน
What is HPC and Overview of a HPC architecture
Understanding Horizontal Pod Autoscaling
มุมมอง 191ปีที่แล้ว
Understanding Horizontal Pod Autoscaling
Kubernetes Job YAML Fields You Should Know
มุมมอง 164ปีที่แล้ว
Kubernetes Job YAML Fields You Should Know
Containerize Java Spring Boot Application
มุมมอง 403ปีที่แล้ว
Containerize Java Spring Boot Application
Thanks for sharing this.
Could you also share the code for this video ?
I believe in the method you are using in each pod gemma 2b model is downloaded and it will increase the size of all 24 pods . So could we store it in S3 bucket or some storage service in Gcloud and reduce the size of pods ? Correct me if i am wrong
For performance reasons we definitely recommend caching the model in some storage to save on bandwidth downloading the model over and over again. You should also look into GKE secondary boot disk as well to preload the container image, massively improving pod startup time for inferencing.
@@ContainerBytes when u r free could you make a video on that like caching the model or storing in S3. Also MIG can be used only in high end GPUs like A100 etc... could u do some video on GPU sharing methods like time slicing in low end GPUs
can i do this on the free gcloud tier with 300$ of free cloud credits ? i am ok with enabling billing but will the free tier suffice this ?
If your project has the quota for it, you could run this for a few hours.
Very cool, cant wait to try these out!
1:54 “I sort of trust Mofi” is an important security policy to live by. 😂
“Trust but verify”, Motto to live by.
thank you
Honestly, this is channel has the best explanations I've come across.
Glad you like them!
It's grate been looking into GCP docs forever...... Please make another one on using it with GCE
Thanks for the great tutorial! Is there a github repo where we can find the code used in the tutorial? Specifically, k8s files
Thanks for the video. I was under impression resource request is mandatory to decide auto scaling. Ex: if i set req cpu 100m and limit cpu 500m , average utilization is 50%, so, the 50% is mapped with request or limit ?
prefer to refer to what I am doing with it (ie the subcommand) and not speak kube-C-T-L otherwise :)
cube C-T-L 👍
Can you give the link to notebook to practice
Please add links you mentioned
Coming soon!
Great video. Will target Ray at next
Hey there, i have finetuned Mistral Modela and I have also created TensorRT engine. I wanted to ask do I need preprocessing and postprocessing script or do I just need pbtxt file to serve it on Triton Inference server? Shall I need to follow what you did for gemma?
thanks for the video but a little disappointed that the future video never came :(
Hello there, we actually have a few videos on the channel showing the containerization and pushing of images to Artifacts Registry. Will update the description to link these too. Thanks for watching.
I'll still pronounce it as, "cube cuddle" because that's where the Borg cuddle you: in a cube.
I wrote my Cloud Engineer exam few months ago, and I am writing my Solutions Architect next week. I hope I pass.
You got this!!!
Hi thanks for this video! Is there a video where you push to artifactory?
Are you asking about JFrog Artifactory? We don't have a video planned for artifactory at the moment. But there developer docs should have some guidance.
Good work guys. Love your very simple, easy and clean approach. Excited to wait for upcoming videos.
Thank You! Glad you enjoyed it!
Cube C T L team!
Yes, that's a real problem in big projects
I'm Cube C-T-L as well!
I pronounce it kubectl
How is the correct or most economical way to run the rest api?
Correct is hard to say. Most economical way probably would be to do something in Cloud Run. You get 2 Million requests per month for free. For most hobby to small projects 2 million request a month is more that what you can ever get to. In the case you get beyond you are already fairly successful as a business and the cost would be negligible. Cloud run is also easy (comparatively) to get started with.
but is it necessary to create a virtual machine? or just with the mkdir command?
Not sure I follow. We did not create a virtual machine. You might be talking about the cloud shell editor? Your local machine would work just fine if you have the necessary tools installed. Mainly the node runtime. I will add some links to setup local dev environments in the description.
Can you make these videos move beginer friendly? I don't have any idea about Google cloud
Of course. Anything specific that was confusing? Like the commands I used or tools used?