OCI as a Standard for ML Artifact Storage and Retrieval - Peyman Norouzi & Eric Koepfle, Bloomberg

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon North America in Salt Lake City from November 12 - 15, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at kubecon.io
OCI as a Standard for ML Artifact Storage and Retrieval - Peyman Norouzi & Eric Koepfle, Bloomberg
Bloomberg's internal Data Science Platform (DSP) provides infrastructure for the development and management of its ML initiatives. The DSP recently added a storage layer tailored to machine learning’s unique demands, specifically aspects like consistency, efficiency, provenance, and governance for ML assets. As container image registries become essential in cloud-native infrastructure, the Open Container Initiative's (OCI) artifact distribution specification has emerged as a versatile standard for managing ML assets. Utilizing container layering, versioning, and metadata addresses these ML-specific challenges. This talk presents our approach to storing and sharing ML models and datasets as OCI Artifacts, which are integrated into our platform - from model building to serving. The lessons learned from our experience will help those who want to adopt similar methods or explore OCI's role in the field of ML.

[@priyankpatil]

Great talk. I have also been thinking about the model storage and retrieval problem since some time. I think there needs to be a standard for model repositories on the lines of maven or other package/artifact management solutions. I had raised this in Sonatype Nexus forum (to add support for AI/ML models) but it seemed not many people were aware of AI/ML models storage issues on the forum. Artifactory and Hugging Face have come up with a solution - Hugging Face Repository type in Artifactory. At this point Hugging Face occupies the same space as Docker for containers or Maven Central for java artifacts. Hence if the Hugging Face repo structure has an open enough licence, it can be adopted as a standard with minor modifications if needed. Similar to Docker & OCI. Another issue that was not brought up in the talk is AI models are huge and a local (office network) proxy repo is a must for teams as downloading models from internet every time will be bandwidth intensive and slow. On-prem Maven proxy repos are very popular Java world. Same can be done for AI model repos.

[@priyankpatil]

I also feel there is a need for a Kubernetes primitive similar to volumes but for immutable artifacts. I would like to mount a container image as volume. This way static artifacts can be mounted without using init-containers. Alternative is NFS volumes but NFS is slow.

[@jonaszasut17]

@@priyankpatil Good news is that from K8S 1.31.0 you can mount OCI artifacts as read-only volumes. Now it's matter of packaging LLM images as OCI artifact and here you go