Data
Jobs should only be launched from a project directory. Home directories are not intended to host data.
Storage
Different volumes available
Several volumes of data storage are available on the iPOP-UP@RPBS cluster.
Directory | Usage |
---|---|
/shared/home | Home directory (user environments). Not intended to store data. |
/shared/projects | Scientific and project data. Used to write hot data (Slurm jobs). |
/shared/banks | Common banks. Read-only. |
/cold-storage | Scientific and project data. Used to store processed data. Replicated. |
We operate quotas to limit the amount of disk space a group or a project can use on the hot storage filesystem.
Check available space for your project
You can check the space available for your projects with the following two commands:
- Check space available for all your project:
status_bars
- Check space available for a specific project:
lfsquotas myproject
Important notes
-
Quotas are applied on files groups, not on directory trees.
-
If you move files from your home directory to a project directory, files ownership may not be correctly set. To set correct ownership, simply type
repairacl myproject
. -
Processed data should be moved as soon as possible on the cold-storage directory or downloaded on an external drive.
-
Old data present in projects directories can be deleted without further notice.
Data Transfer
You can use different clients to download/upload your data from/to the cluster like scp or rsync.
While being on your computer:
MyGreatComputer:~/$ scp <username>@ipop-up.rpbs.univ-paris-diderot.fr:Path_To_file/my_file path_to_local_folder
Don't forget the -r
flag for recursive copy (for directories).
You can also use graphic clients like FileZilla or simply use your file manager with SSHFS.