Data

Jobs should only be launched from a project directory. Home directories are not intended to host large amount of data.

Storage

Different volumes available

Several volumes of data storage are available on the iPOP-UP@RPBS cluster.

Directory Usage
/shared/home Home directory (personal environment and data). Not intended to store large amount of data.
/shared/projects Scientific and project data. Used to write hot data (Slurm jobs).
/shared/banks Common banks. Read-only.
/cold-storage Scientific and project data. Used to store processed data.

We operate quotas to limit the amount of disk space a group or a project can use on the hot storage filesystem.

Check available space for your project

You can check the space available for your projects with the following two commands :

  • Check space available for all your project :
status_bars
  • Check space available for a specific project :
lfsquotas myproject

Notes

  • Quotas are applied on file groups, not on directories.

  • If you move files from your home directory to a project directory, files ownership may not be correctly set. To set correct ownership, simply type repairacl myproject.

  • Processed data should be moved as soon as possible on the cold-storage directory or downloaded on an external drive.

  • Old data present in projects directories can be deleted without further notice.

Data Transfer

You can use different clients to download/upload your data from/to the cluster like scp or rsync

Transfer file from the server to my computer with scp

Gif scp

While being on your computer :

MyGreatComputer:~/$ scp <username>@ipop-up.rpbs.univ-paris-diderot.fr:Path_To_file/my_file path_to_local_folder

Don't forget the -r flag for recursive copy (for folders).

You can also use graphic clients like FileZilla or simply use your file manager with SSHFS.