HDFS is a file system while we talked about object storage. File system is hierarchical while object storage is flat, which allows for infinite storage and scalability. File system on the other hand has limitations in that regards. File system might be better in terms of speed and performance but again depends on the total size etc. so yes the approach we discussed is purely on top of cloud object storage.
These podcasts are goldmine!
Also I watched your only livestream today while I was in metro. Got many insights. You should do more livestreams.
Thanks 🙏🏻 I will make a note for livestreams.
Thanks for updating the video. Great content. Keep it up.
Your feedback helped me. thanks :)
Pushing the algorithm ❤
What is the rust library Simon talked about building flexible storage formats? I heard something like "archive". But failing to find the right one.
Good question github.com/rkyv/rkyv
rkyv
How is Hdfs different from the above mentioned approach?
HDFS is a file system while we talked about object storage. File system is hierarchical while object storage is flat, which allows for infinite storage and scalability. File system on the other hand has limitations in that regards. File system might be better in terms of speed and performance but again depends on the total size etc. so yes the approach we discussed is purely on top of cloud object storage.
❤ this content is just awesome