Persistent Volumes on Kubernetes for beginners
ฝัง
- เผยแพร่เมื่อ 1 ก.ค. 2024
- Subscribe to show your support! goo.gl/1Ty1Q2 .
Patreon 👉🏽 / marceldempers
In today's video, we're going to be talking about persistent volumes and how they help us to run workloads that require persistence. These can include databases, caches, message queues and anything that requires data to be stored.
But remember applications that require persistence can be complex. To help run these complex applications, Kubernetes uses StatefulSets instead of deployments.
Remember to checkout my video on StatefulSets here - • Understanding Stateful...
Also if you want to support the channel further, become a member 😎
marceldempers.dev/join
Checkout "That DevOps Community" too
marceldempers.dev/community
If you are new to Kubernetes, check out my getting started playlist on Kubernetes below :)
Checkout the source code below and follow along :)
Kubernetes Guide for Beginners:
---------------------------------------------------
• Kubernetes development...
Kubernetes Monitoring Guide:
-----------------------------------------------
• Kubernetes Monitoring ...
Kubernetes Secret Management Guide:
--------------------------------------------------------------
• Kubernetes Secret Mana...
Like and Subscribe for more :)
Source Code
--------------------------------------------------------------
github.com/marcel-dempers/doc...
Follow me on socials!
marceldempers.dev
Twitter | / marceldempers
GitHub | github.com/marcel-dempers
Facebook | thatdevopsguy
LinkedIn | / marceldempers
Instagram | / thatdevopsguy
Music:
Track: Fox Beat 2 - Jeff Kalee - Pillow Talk - Royalty Free Vlog Music [BUY=FREE] | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / jeff-kalee-pillow-talk...
Track: Amine Maxwell - Le Soir | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / le-soir
Track: Sappheiros - Spirit | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / spirit
Track: n0tavailable - mourner - sad trap beat | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / mourner
Track: souKo - souKo - Parallel | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / parallel - วิทยาศาสตร์และเทคโนโลยี
Understanding StatefulSets for beginners👉🏽 th-cam.com/video/zj6r_EEhv6s/w-d-xo.html
You're an excellent teacher Marcel, thank you for making these videos. Your vlog format is groundbreaking in tech. While there are plenty of others on youtube offering fantastic content, you've introduced (at least to me) an element that has made it so much easier to take the information in. Well done man!
Marcel, your stateful and persistance videos demos are brilliant, thank you for putting in the work to share your knowledge, your delivery is brilliant
Hi Marcel
Thank you so much for this simple explanation with a REAL WORLD example of persistent volumes. I have been wracking my brains trying to tie together pv, pvc, and statefulsets. You explained and demonstrated in 20 minutes what other people couldn't get across in hours of demos. Love your channel, keep it up mate 👌👍🙂
Possibly the best tutorial on the topic!!
You're on another level mate, so concise, so clever, so well spoken, love your videos
You are too good at this, your videos have helped me understand and breakdown those heavy jargons on K8s doc pages. Thank you.
Great explaination!! After watching at least 10 other tutorials, yours was the first one that really worked!
Thanks Marcel! Would really love to see one on Argo Workflows/Events for setting up and triggering pipelines in kubernetes!
Thanks for this amazing video, I was really struggling with this part in Kubernetes. It helped me a lot :)
Thanks a lot, Marcel this is the best video about the concept that I've found on TH-cam, your explanation coming from docker volumes creates a solid ground to easily understand the kubernetes part :smile:
Marcel you're a good man. You always explain things very clearly and dough you type as fast as the light you repo in GitHub shows everything. So many many thanks. I was stuck trying to bind a local PV to a mongo db StatefulSet , looking at your code(dough for Postgres ) made me find a way to debug my pv.yaml.. it results that was a typo in the nodeAffinity key.. thank you so much again. Vinny
Man this is just awesome, I'm beating myself for not understanding, after watching this I can assure myself I got the idea
I remember this quote,
You can't connect the dots forward - only backward
- Steve Jobs
Thank you for the lecture marcel, You made my day.
You sir are amazing! Love your style and your editing and your teaching ability. You're like the Kubernetes god! Keep up the great work!
this devops guy LIFTS!!
💪🏽
Your channel is a godsend 🙏🙏🙏
excellent and well detailed! thank you for the explanation dude!
Thanks for sharing. Gives me better understading than k8s doc.
Bro out here making every DevOps engineers life so much easier, thanks the school system around the globe should take you as an example on how to transfer knowledge....
This video is almost as good as you triceps. Respect
Just Amazing ! Thank you very much man ! Keep Going !
Thank you so much for this. I’ve been in class all week and studying my ass off and just have NOT understood the pv claim until now. Now I can get some sleep. Lol
Excellent video It helped me to clear my concept Thanks.
Thanks alot Marcel, I finally understood the concepts of PV, PVC and how to use them with Postgres thanks to you. Massive thanks!
Also I would like to request if you could teach how to do authentication and authorizations from nginx-ingress using Keycloak, as I am sure many people struggle with it... And there arent any TH-camrs like you, who teach from the basics and build up concepts upon that, rather they usually just assume that we just know stuff. Also, since KeyCloak is really popular Auth as a Service, so I am sure it would be very helpful for others too, so please consider doing a video on it, please. Thanks a lot in advance.
Thanks for the great content Marcel
Great explaination!
Thank you for this video, please make a video on the production-ready Postgres database.
Good video generally, but there's a misconception about containers. A container is not a process, it's a sandboxed runtime environment in which you can run one or more processes. You can stop a container (`docker stop `) which terminate all processes inside of a container and then run it again (`docker start `) and the files will still be there. Kubernetes never bothers to restart a container - when it restarts a pod it recreates containers with the same IP address assigned, etc. so you might think that containers are completely stateless (and they're typically treated as such) but a container keeps its filesystem state after all of its processes terminate which is pretty useful when it comes to retrieving logs and inspecting what caused the termination.
Best in class 🙌✌️
thanks a lot , very informative and useful
Very informative
very good video, tks Marcel.
You're great. Good job!
Great video!
Every sec has information in your videos.
Thank you, the guide was very clear and concise.
Do you have a guide on how to handle a "real world" project in the different phases? For example I imagine one would develop in local on minikube, then deploy on the cloud (AWS, GCP, etc), maybe having TEST/PRODUCTION as different clusters. How do you manage all this? Does it make sense to have in the same git repo different kubernetes folders, one for each environment, containing the appropriate yml?
Or would you rather have different branches? A guide on this would be great.
great video thank you
You are really good at explaining things. Can you do a tutorial on tricep extensions? Thanks!
super underrated comment lol
Hi Marcel and thanks you for the great job you do!
Can you give a hint or direction please. I've deployed wp with pvc using nfs-provisioner on bare-metal. nfs share is located on the neighbouring vm host in the same subnet and basically both k8s nodes and nfs node use the same physical disk. What I see is that i can easily store anything on this share manually from any pod but WP + mysql is very slow when using this nfs (e.g. it take up to 15 sec to read a few KB files). perms on the share are 777. Access mode readwritemany if it makes any difference. I've googled for this issues and it turns out quite common but there is no specific solution or reason that I could find. Any advices please?
Thanks! I have a single tennet app that will have multiple copies of the same pod running in different namespaces with different persistant volume claims. Many of these resources have the same name, but are just deployed in different namespaces. It would be great to see a video about how a single pod .yaml could be used in that way to create different volumes.
Thank you so much
Thanks brother
Thank you
Greate, thanks a lot for this video it cant be better
Great Video, breaking down the content in ingestible bytes. One improvement I'd suggest is timestamping your video so that people can skip ahead to the exact part they are looking for. Keep doing you man your content is great
What is the best way to setup storage server and its configuration for kubernetes cluster persistence volume for production system? NFS or ISCASI any hardware brand specific. Thanks !
The local file system path you specified, you would see files in there after creating the table right?
awesome !!
How does it work, then, if I have multiple apps that need to share the same physical storage. From what it sounds, I still need to create a pv for all of them, but can mount them to the same physical path in the storage class, nfs for example.
But then how does the storage amount come into play? How can a pv reliably tell a claim “hey, I have this much storage available, use me.” If there is overlap in the physical storage?
Thank you kindly for this video. One question though. Why would a pvc specifies a storage class? isn't it already defined in a pv?
Great video Marcel!
However, I do have a question.
I am able to persist data through the lifetime of the container / pod, but not able to do so if I restart my cluster. How do I achieve this?
I am planning to use local storage class
HERO!
Thx Great Video 👍👍👍👍
how would I add nodeselectos and affinity rules for statefulsets?
@13:40, How is the PersistentVolumeClaim referencing the PeristentVolume (example-volume) ? How does the pvc know which PV to use?
Excellent
That Brilliant Guy who is Awesomeeeeee.....
Hey Marcel, love the videos! I think there is a small mistake with your explanation on PVCs claiming storage from PVs. You cannot have several PVCs bound to one PV, once a PVC is bound to a PV the PV will not be able to bind to another PVC but many pods can use the same PVC. You create PVs to configure a certain way to claim storage and to use it you bind a PVC to it. If you aren't using PVs the storage class dynamically creates and provisions a PV for the requested PVC. Keep up the great content!
Hi Gaurav, is this end of K8s series, please add if few more topics pending.
and i must saym your tutorial made my concep clear and handson experianced.
Suffering for the past week figuring out how to persist data on a database container has finalldy led me here to find the solution...
Perfect thanks! Super .......
Please a video on storage class for aws
Hi Marcel, I've created a PersistentVolume using the hostpath StorageClassName and I mounted it in a mysql pod. The Persistent Volume hostpath is "/var/folders/data", and it's mounted in "/var/lib/mysql" in the Container. What's strange is that when I list the PV hostpath "/var/folders/data" it's empty. But when I create a test database and then delete and recreate the deployment the test database is still there. You know where the container "/var/lib/mysql" files could be found? I was expecting to find them in the PV hostpath "/var/folders/data". I'm using docker for mac.
where is /var/lib/postgresql/data located? Is this location inside pod?
Hey Marcel, once again thanks for another excellent video. One question though. Would the Postgres data survive a Docker Desktop restart?
Technically, it should since the example used host path persistence so unless the host is destroyed, it should persist yes
@@MarcelDempers Hey Macel, thanks for the video. unfortunately the data being lost when docker restarts.
@@MarcelDempers data loss after restarting the docker desktop
how to backup and restore database in this case ?
Hey Marcel, great videos. After I altered the ConfigMap to use a different POSTGRES_USER and deleted/recreated everything including the PV, the old database user & table existed.
Can you explain why that happened even after I deleted the PV? Is there more to "scrub" here?
I assumed recreating everything would result in a fresh database with the new user credentials. Thanks!
This highly depends on the storage class you use. In this demo I use host path which means data is persisted on the node at a path (which can be changed).
The get a fresh storage, you either have to change the path to get a new folder on the host. Remember your old data would remain. Or SSH to the node, clean it up, Or run another container with a host mount and carefully run "rm" command to delete the content
@@MarcelDempers Thank you for such a quick response! I used hostPath/your tutorial code too. If you have any advice on ssh-ing to the docker-desktop (Mac) node to clean up, I'm all ears. Thanks again for these videos, very helpful.
Hi Thanks you for great video.! I did the same and tried to deploy jenkins but when i create pv and pvc . mnt/data is empty. and i used volumemount to var/jenkins_home.
If you need help with Jenkins checkout my CI/CD playlist, there are two Jenkins guides with source code for you 💪🏽👊🏽
Hello. I have a question. What if you would want to save and mount the logs of a server application? Should you still use deployment or statefulset?
For logging, I have a separate video in the pipeline, You would rather export logs from the app and not store it in a volume. Take a look at the EFK\ELK concepts, where basically every pod in K8s writes logs to stdout. It is best practise not to write logs to file and rather use stdout\stderr. These logs will automatically end up in the host docker logs which you can push out of the system using Fluentd\FluentBit. This allows you to centralise logs for all deployments\statefulsets running in your cluster under something like Elastic Search or 3rd party application.
@@MarcelDempers oh. interesting. I'll look it up! Thank youuu so much. been watching your videos. so much helpful for me. great videos btw :)))
but If we have a three node cluster one is master other two are worker first time our Postgres pod is schedule on worker1 now its PV on worker1 and second time after deleting the pod k8s schedule it on Worker2 now pod create its PV on Worker2 but now data is not persistence. because data on worker1 can not allocate to worker2 pod? What's a solution?
This is because you are using a deployment with PVs which is not meant for stateful workloads. You'll need to learn about StatefulSets
th-cam.com/video/zj6r_EEhv6s/w-d-xo.html
Hi Marcel, my setup is a bit unique, I have created a K8S cluster on a few VMs in our own private cloud and all the nodes are mounted with a single NFS drive (say /data), can I define a storageclass of hostpath type using /data? Does PV has to be mounted to all the nodes?
That depends on the app you're planning to run and how it uses the storage. If you plan on using a statefulset its recommended that each pod has its own storage. StatefulSet will use a storage class to provision a PV per pod and not share the same storage. Hostpath will work, but if you mount a shared storage to all the nodes, they may override each other's data. So unless you mount unique storage to each node, it may not be ideal. I'm new to NFS, but i would see if its possible to provision NFS dynamically using a storage class and see if the statefulset can issue PV + PVCs per pods using NFS dynamic provisioning. That way each pod will get its own NFS storage provisioned by k8s when you scale up and down
Checkout NFS provisioning using Rook rook.io/docs/rook/v1.4/nfs.html
Seems interesting.
If you have not worked with Statefulsets I'd highly recommend to th-cam.com/video/zj6r_EEhv6s/w-d-xo.html
@@MarcelDempers thanks Marcel, I have deployed Rook Ceph RBD in my cluster and now the applications now works like charm
Hi, i got a qusetion, if my volume is stored application's log file, this volume should be oversized, how could i logrotate in this volume to manage the log file size? do u have any suggestion? thank u so much
I would recommend looking into Fluentd for shipping logs and not store logs in a volume.
K8s has pod logs go to a container runtime folder and Fluentd mounts that directory to forward logs out.
The container runtime is designed to rotate logs automatically.
th-cam.com/video/6kmHvXdAzIM/w-d-xo.html
@@MarcelDempers thank u so much! its useful to a kubernetes beginner user.
The tutorial doesn't work on my end. Keeps getting /var/lib/postgresql/data invalid permission.
Im running docker with k8s on windows 11
You requested for 50mb you got 1gb :P :D
I believe capacity stated on the PVC is that of the PV and not the requested of the claim :)
Why does the table already exist @4:45 if you just created the volume. I'm guessing you edited out the part when you first created it.
Do people really prefer the background music ? Maybe you should poll your subscribers, just distracting.
Great video!