Data

Jobs should only be launched from a project directory. Home directories are not intended to host data.

Storage

Different volumes available

Several volumes of data storage are available on the iPOP-UP@RPBS cluster.

Directory Usage
/shared/home Home directory (user environments). Not intended to store data.
/shared/projects Scientific and project data. Used to write hot data (Slurm jobs).
/shared/banks Common banks. Read-only.
/cold-storage Scientific and project data. Used to store processed data. Replicated.

We operate quotas to limit the amount of disk space a group or a project can use on the hot storage filesystem.

Check available space for your project

You can check the space available for your projects with the following two commands:

  • Check space available for all your project:
status_bars
  • Check space available for a specific project:
lfsquotas myproject

Important notes

  • Quotas are applied on files groups, not on directory trees.

  • If you move files from your home directory to a project directory, files ownership may not be correctly set. To set correct ownership, simply type repairacl myproject.

  • Processed data should be moved as soon as possible on the cold-storage directory or downloaded on an external drive.

  • Old data present in projects directories can be deleted without further notice.

Data Transfer

You can use different clients to download/upload your data from/to the cluster like scp or rsync.

Gif scp

While being on your computer:

MyGreatComputer:~/$ scp <username>@ipop-up.rpbs.univ-paris-diderot.fr:Path_To_file/my_file path_to_local_folder

Don't forget the -r flag for recursive copy (for directories).

You can also use graphic clients like FileZilla or simply use your file manager with SSHFS.