Quick start guide

Access IPOP-UP

Access: SSH
Server: ipop-up.rpbs.univ-paris-diderot.fr

Linux/Mac

On Linux/Mac, you can use ssh client like OpenSSH :

  • Connect via ssh with the following command :
ssh <username>@ipop-up.rpbs.univ-paris-diderot.fr

Windows

On Windows, you can use clients like PuTTY.

More details on login here.

Working directory/Data usage

Jobs should only be launched from a project directory.

Home directories are not intended to host large amount of data.

  • Go to your project directory with the following command :
cd /shared/projects/myproject

More details on data here.

Use Software

You must load software (such as blast, python, gcc, etc) in their environment by using the module commands (Environment Modules):

  • Load a software environment with the following command :
module load my_super_software

More details on softwares environment here

Submit a job

Computing work is done by submitting jobs to the workload manager Slurm.

You must use Slurm to execute your jobs.

Demo : Using slurn with sbatch

1 - Write a bash script

First, go into your project directory.

cd /shared/projects/myproject

This script must contain your commands to execute. Here, inside myscript.sh, we launch a bowtie command and just print some truth. 

#!/bin/bash

bowtie2 -x hg19 -1 sample_R1.fq.gz -2 sample_R2.fq.gz -S sample_hg19.sam

echo "Enjoy slurm ! It's highly addictive."

2 - Add options and requirements

You can specify several options for your jobs (name, number of CPU, amount of memory, time limit, etc.). Extra parameters are set at the beginning of the script with the #SBATCH directive (just after the shebang #!/bin/bash) You can see here a more complete list of options.

Here we specify the partition (--partition=MY_SUPER_PARTITION), the account (--account=MY_SUPER_ACCOUNT), the job name (--job-name=MY_SUPER_JOB_NAME) and the amount of memory required (--mem=40GB).

#!/bin/bash

#SBATCH --partition=MY_SUPER_PARTITION
#SBATCH --account=MY_SUPER_ACCOUNT
#SBATCH --job-name=MY_SUPER_JOB_NAME
#SBATCH --mem=40GB

bowtie2 -x hg19 -1 sample_R1.fq.gz -2 sample_R2.fq.gz -S sample_hg19.sam

echo "Enjoy slurm ! It's highly addictive."

Advice : - Set as many parameters as you can in the script in order to keep a track of your execution parameters for a future submission. - Keep a personal sbatch template with all the redundant option already written.

3 - Launch your job with the sbatch command

sbatch myscript.sh

The command returns a jobid to identify your job.

4 - Follow your job

To see all the jobs curently running :

squeue

To cancel/stop a job:

scancel <jobid>

The different status are :

  • PENDING : PD

  • RUNNING : R

  • COMPLETED : C

If your job is not displayed, your job is finished (with success or with error).

5 - See output

The output of the script (standard output and standard error) is live written. Default output file is slurm-[jobid].out in your working directory. And of course, if you have some result files like sample_hg19.sam, these files will be available.

See more information on slurm use here

Notes

  • All nodes have access to the data (/shared/home, /shared/projects or /shared/banks).

  • All softwares are available on all nodes, but we have to load them inside the script with command module load [module].

  • All jobs are contained and can not use more resources than defined (CPU, memory).

  • By default, only one CPU and 2GB of RAM are allocated.

  • Jobs which exceed limits (memory or time, values by default or set) are killed.