How to Manage Storage for Notebooks

Notebooks are a web-based Jupyter IDE with shared persistent storage for long-term development and inter-notebook collaboration, backed by accelerated compute.

Storage management allows you to modify and delete files and directories in Linux-based systems. To manage your storage on a Linux system, we recommend using the terminal.

You can also run Bash commands in the Jupyter Notebooks to interact with the underlying OS environment using a Code Cell in single line. To do so, start your line using the ! operator. If you need to add more lines, you can connect them using the && operator.

Determine Folder Size

To determine your folder size when managing your files, first list folders located in the current folder using the following commands.

du -sch .[!.]* * | sort -h

!du -sch .[!.]* * | sort -h

If you want to see the list of folders in a different folder, you can use the following commands, replacing <folder-name> with the path to the folder you want to analyze:

cd <folder-name>
du -sch .[!.]* * | sort -h

!cd <folder-name> && du -sch .[!.]* * | sort -h

Determine File Size

Once you know where the data is, you can determine the file sizes. This information helps in determining which individual files have the greatest impact on your storage. To list files located in the current folder, use the following commands.

find . -print0 | du -ch --files0-from=- | sort -h

!find . -print0 | du -ch --files0-from=- | sort -h

If you want to see the information stored in a different folder, you can use the following commands, replacing <folder-name> with the path to the folder you want to analyze:

cd <folder-name>
find . -print0 | du -ch --files0-from=- | sort -h

!cd <folder-name> && find . -print0 | du -ch --files0-from=- | sort -h

Once you have determined which files you might want to delete, we recommend moving them to a single folder so that you can delete everything together. To do so, you can use the following commands, replacing <folder-name>, <file-name>, and <destination-folder> values with your own:

cd <folder-name>
mv <file-name> <destination-folder>

!cd <folder-name> && mv <file-name> <destination-folder>

Delete Folders and Files

Once you have determined which folders and files you want to delete, use the following commands, replacing <folder-name> and <data-name> values with your own:

cd <folder-name>
rm -r <data-name> ## <data-name> is the name of the folder/file you want to delete

!cd <folder-name> && rm -r <data-name>

Other Tools and Alternatives

If you have access to the Linux Terminal and you prefer a visual interface to determine folder and file sizes, you can use Ncdu. This application scans your machine and shows all folders listed by size in the terminal. It also allows you to move around the file system to open each folder, see it’s content and delete any file or folder.

To use Ncdu, follow these instructions:

1. In the terminal, update your machine’s APT:

sudo apt-get update

2. Install ncdu.

sudo apt install ncdu

3. Run ncdu.

ncdu

Delete Datasets

Datasets are a storage source that allow you to save and/or share large amounts of data. They are part of the main storage even though you manage datasets separately from the main storage.

Datasets work as read-only sources. Once you add files to a dataset, you can’t edit, move, or delete them. Therefore, you can only remove those files from your storage by deleting the entire dataset itself.

To delete a dataset, follow these instructions:

1. In the Paperspace console, select any project.
2. Click on the Data tab and search for the dataset you want to delete.
3. Click the … menu that corresponds to the dataset you want to delete and select Delete.

Delete Models

Depending on their size, some models can take up a significant amount of storage in your account and require you to delete some of your models to free up some of that space.

You can delete a model from the model repository using the Paperspace console or the CLI.

• Web UI
• CLI

To delete a model, follow these instructions:

1. In the Paperspace console, select any project.
2. Click on the Models tab to see a list of your trained models.
3. Search for the model you want to delete.
4. Click the … menu that corresponds to the model you want to delete and select Delete.

You can delete a model using the CLI with the following command:

gradient models delete --id <your-model-id>