Hugging Face and SkyPilot: Revolutionizing Cloud Storage

Running AI Workloads Without Egress Fees

In the field of artificial intelligence, teams often face a major issue: models and datasets are typically hosted in a specific region of a cloud provider, while the GPUs needed for development, training, or deployment may be located on another cloud. This separation leads to data transfer fees between clouds, a significant expense for accessing one's own data. With the rise of cloud computing technologies, this problem is becoming increasingly common and costly.

Hugging Face, in collaboration with SkyPilot, offers a solution to this problem. By storing models and datasets on the Hugging Face Hub and using SkyPilot to execute computations on any GPU-equipped cluster, users can avoid these fees. It is simply a matter of mounting a Hugging Face Bucket or any Hub repository in a SkyPilot job using an hf:// URL and the HF_TOKEN already in the user's possession. Storage on Hugging Face does not charge egress fees, meaning that reading data on these GPUs is free, regardless of the cloud used.

New and Improved Features

The new features offered by this collaboration are varied and provide significant improvements:

• Hub Data Integration in Any Job: With store: hf, it is possible to mount a Hugging Face Bucket in read-write mode or any model, dataset, or Space repository in read-only mode in a SkyPilot task. This is done via an hf:// URL and the existing HF_TOKEN, using the MOUNT or COPY options. This integration greatly simplifies the data management process for developers.

• Execution on Any GPU, on Any Cloud: SkyPilot can find the necessary computing resources for a job across more than 20 clouds, Kubernetes, Slurm, and even on-premises. This allows the use of any available GPU, whether reserved or on-demand, regardless of the provider. This flexibility is crucial for optimizing resource utilization and reducing costs.

• No Egress Fees for Data Reading: Hugging Face storage does not charge egress fees or CDN fees. Thus, no matter where SkyPilot executes the job, models and datasets can be read directly from the same bucket, without requiring copies between clouds and without additional costs. This represents substantial savings for companies using multiple cloud resources.

• Xet-Based Deduplication: Buckets are built on Xet, meaning that only the modified pieces of incremental checkpoints and model variants are stored and transferred. This deduplication technology significantly reduces the volume of data to be transferred, optimizing storage efficiency.

• Joint Development: Hugging Face and SkyPilot have worked together to develop these features, with the integration of hf-mount FUSE patches by the Hugging Face team to enable operation in unprivileged containers. This collaboration has led to the creation of a robust and flexible solution tailored to the needs of modern developers.

Hugging Face Storage: A Major Asset for SkyPilot

SkyPilot tasks can already interact with various cloud object storages such as S3, GCS, Azure, R2, and others by mounting them to a local path. Now, Hugging Face Storage joins this list under store: hf, accessible via the hf:// schema. This covers the entire data lifecycle, from reading models and datasets to writing checkpoints and publishing finished models. This comprehensive integration simplifies developers' workflows and enhances operational efficiency.

The hf-mount FUSE backend from Hugging Face allows a bucket or repository to appear as a local path, thus facilitating integration with other SkyPilot FUSE mounts. Downloads occur at the filesystem level, meaning that only the necessary data is transferred, and a disk cache is maintained to optimize repeated reads. This approach reduces latency and improves overall system performance.

The Benefits of Lazy Reading

Lazy reading presents a significant advantage: a process can start working on a large file before it is fully downloaded. This allows the GPU to remain active almost immediately, training on the data as it arrives, rather than remaining idle while the entire dataset or checkpoint is transferred. This method is particularly beneficial during the first epoch when nothing is yet cached. COPY offers an alternative by downloading data via huggingface_hub without special requirements.

Simplifying Authentication

Authentication is simplified through the use of the token already in the user's possession. It is sufficient to set HF_TOKEN in the environment and pass it to an execution with --secret HF_TOKEN. SkyPilot uses this token for mounting on any cloud where the job is executed, whether it is AWS, GCP, Azure, Nebius, Lambda, or a personal Kubernetes cluster. This simplification reduces the complexity of managing access keys and enhances security.

Eliminating Egress Fees: A Paradigm Shift

The availability of GPUs is no longer limited to a single provider. To obtain enough H100 and H200 GPUs, teams maintain capacity across multiple providers simultaneously. SkyPilot is designed to manage this diversity, scheduling jobs across more than 20 clouds, Kubernetes, and on-premises, and executing them on any available reserved cluster. This ability to adapt to available resources is essential for maximizing the use of existing infrastructures.

Object storage has often been a barrier due to its regional and cloud-specific nature. Feeding a GPU or inference server located in another provider's data center required either duplicating data in each provider's bucket or paying for transfers. Egress fees can amount to around $0.09/GB outside of AWS, and even between regions within the same cloud. Hugging Face Storage eliminates these costs on the reading side, offering storage at $12-18/TB/month, accessible for free from any cluster. This reduction in operational costs is a major asset for companies looking to optimize their cloud spending.

A Revealing Benchmark

To evaluate the effectiveness of this solution, a small adjustment was made: the Qwen/Qwen3.5-4B model was trained on the HuggingFaceH4/Multilingual-Thinking dataset using TRL's SFTTrainer, mounting the model in read-only mode from its Hub repository and writing each checkpoint into a Hugging Face Bucket. The same SkyPilot YAML was executed on AWS, GCP, and Lambda, changing only --infra.

Measured Results:

• The model was loaded for free on each cloud, thanks to lazy reads that only pull the necessary data. The model was ready to be trained in about 30 seconds, with no egress fees. This rapid implementation is crucial for projects with tight deadlines.

• Checkpoints were streamed to the bucket at speeds of up to 170 MB/s, persisting beyond the GPU instance. This transfer efficiency ensures that data is available where needed, without delay.

• Per cloud, checkpoints were written to the bucket at optimal speeds, demonstrating the solution's effectiveness. This consistent performance across different cloud providers is a significant advantage for companies operating in multi-cloud environments.

Xet-Based Storage: Effective Deduplication

Hugging Face Buckets, based on Xet, use content-defined chunking to divide files into pieces of about 64 KB, storing each unique piece only once. This allows for effective deduplication in several scenarios:

• Incremental checkpoints and adapters: only modified pieces are uploaded, thus reducing the amount of data transferred.

• Model variants sharing a base: shared pieces are stored only once, optimizing storage space utilization.

• Augmented datasets: only new rows are transferred, significantly reducing the volume of data moved. This optimization is particularly useful for constantly evolving datasets.

• Optimized re-downloads: only piece hashes are transferred, speeding up the process. This method reduces the necessary bandwidth and accelerates download operations.

Conclusion

Automatic deduplication allows for significant savings on storage and data transfer. To get started, simply install SkyPilot with Hugging Face and configure an hf:// mount for any SkyPilot task. MOUNT requires a base image with glibc 2.34+ and /dev/fuse. This simplicity of installation and use is a boon for developers looking to quickly integrate this technology into their workflows.

A Fruitful Collaboration

The initial support for store: hf was initiated by Nikhil Jha, with the Hugging Face team integrating the hf-mount FUSE patches for mounting in unprivileged containers. The SkyPilot team incorporated this functionality into the storage backend, making the entire process open source and accessible to all. This collaborative and open approach fosters innovation and adoption by the developer community.