Using HuggingFace Datasets Offline

This is pretty simple, but quite helpful if you're running jobs on a compute node that doesn't have internet access.

On the login node or another machine with internet access, run the following Python code:

import datasets
 
x = datasets.load_dataset("my_dataset")
 
x.save_to_disk("./my_dataset_local")

Then, if needed, copy the files to the machine running your job. Now, from that offline machine, loading the dataset is simple!

y = datasets.load_from_disk("./hellaswag_local")
y # DatasetDict({...})