OSError: Not enough disk space.

[qqaatw]

Describe the bug

I'm trying to download natural_questions dataset from the Internet, and I've specified the cache_dir which locates in a mounted disk and has enough disk space. However, even though the space is enough, the disk space checking function still reports the space of root / disk having no enough space.

The file system structure is like below. The root / has 115G disk space available, and the sda1 is mounted to /mnt, which has 1.2T disk space available:

/
/mnt/sda1/path/to/args.dataset_cache_dir 

Steps to reproduce the bug

dataset_config = DownloadConfig(
    cache_dir=os.path.abspath(args.dataset_cache_dir),
    resume_download=True,
)
dataset = load_dataset("natural_questions", download_config=dataset_config)

Expected results

Can download the dataset without an error.

Actual results

The following error raised:

OSError: Not enough disk space. Needed: 134.92 GiB (download: 41.97 GiB, generated: 92.95 GiB, post-processed: Unknown size)

Environment info

• datasets version: 1.9.0
• Platform: Ubuntu 18.04
• Python version: 3.8.10
• PyArrow version:

[qqaatw]

Maybe we can change the disk space calculating API from shutil.disk_usage to os.statvfs in UNIX-like system, which can provide correct results.

statvfs = os.statvfs('path')
avail_space_bytes = statvfs.f_frsize * statvfs.f_bavail

[albertvillanova]

Hi @qqaatw, thanks for reporting.

Could you please try:

dataset = load_dataset("natural_questions", cache_dir=os.path.abspath(args.dataset_cache_dir))

[qqaatw]

@albertvillanova it works! Thanks for your suggestion. Is that a bug of DownloadConfig?

[albertvillanova]

DownloadConfig only sets the location to download the files. On the other hand, cache_dir sets the location for both downloading and caching the data. You can find more information here: https://huggingface.co/docs/datasets/loading_datasets.html#cache-directory

[davidtan-tw]

I had encountered the same error when running a command ds = load_dataset('food101') in a docker container. The error I got: OSError: Not enough disk space. Needed: 9.43 GiB (download: 4.65 GiB, generated: 4.77 GiB, post-processed: Unknown size)

In case anyone encountered the same issue, this was my fix:

# starting the container (mount project directory onto /app, so that the code and data in my project directory are available in the container)
docker run -it --rm -v $(pwd):/app my-demo:latest bash

# other code ...
ds = load_dataset('food101', cache_dir="/app/data") # set cache_dir to the absolute path of a directory (e.g. /app/data) that's mounted from the host (MacOS in my case) into the docker container

# this assumes ./data directory exists in your project folder. If not, create it or point it to any other existing directory where you want to store the cache

Thanks @albertvillanova for posting the fix above :-)

[sankexin]

import datasets
datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True