Quick start guide

Access iPOP-UP

Access: SSH
Server: ipop-up.rpbs.univ-paris-diderot.fr

Linux/Mac

On Linux/Mac, you can use ssh client like OpenSSH:

ssh <username>@ipop-up.rpbs.univ-paris-diderot.fr

Windows

On Windows, you can use clients like PuTTY.

More details on login here.

Working directory

Jobs should only be launched from a project directory. 

cd /shared/projects/myproject

Home directories are not intended to host data.

More details on data here.

Software

You must load software (such as blast, python, gcc, etc) in their environment by using the module commands (Environment Modules):

module load my_super_software

More details on softwares environment here.

Submit a job

Computing work is done by submitting jobs to the workload manager Slurm.

You must use Slurm to execute your jobs.

1 - Write a bash script

First, go into your project directory:

cd /shared/projects/myproject

This script must contain your commands to execute. Here, inside myscript.sh, we launch a bowtie command and just print some truth:

#!/bin/bash

bowtie2 -x hg19 -1 sample_R1.fq.gz -2 sample_R2.fq.gz -S sample_hg19.sam

echo "Enjoy slurm ! It's highly addictive."

2 - Add options and requirements

You can specify several options for your jobs (name, number of CPU, amount of memory, time limit, etc.). Extra parameters are set at the beginning of the script with the #SBATCH directive (just after the shebang #!/bin/bash) You can see here a more complete list of options.

Here we specify the partition (--partition=MY_SUPER_PARTITION), the project account (--account=MY_SUPER_ACCOUNT), the job name (--job-name=MY_SUPER_JOB_NAME) and the required amount of memory (--mem=40GB):

#!/bin/bash

#SBATCH --partition=MY_SUPER_PARTITION
#SBATCH --account=MY_SUPER_ACCOUNT
#SBATCH --job-name=MY_SUPER_JOB_NAME
#SBATCH --mem=40GB

bowtie2 -x hg19 -1 sample_R1.fq.gz -2 sample_R2.fq.gz -S sample_hg19.sam

echo "Enjoy slurm ! It's highly addictive."

Advices:

  • Set as many parameters as you can in the script in order to keep track of your execution parameters for a future submission.
  • Keep a personal sbatch template with all the redundant option already written.

3 - Launch your job with the sbatch command

sbatch myscript.sh

The command returns a job id to identify your job.

4 - Follow your job

To see all the jobs curently running:

squeue

To cancel/stop a job:

scancel <jobid>

The different status are:

  • PENDING: PD

  • RUNNING: R

  • COMPLETING: CG

If your job is not displayed, your job has finished (with success or with error).

5 - See output

The output of the script (standard output and standard error) is live written. Default output file is slurm-[jobid].out in your working directory. And of course, if you have some result files like sample_hg19.sam, these files will be available.

See more information on slurm use here.

Notes

  • All nodes have access to the data (/shared/home, /shared/projects or /shared/banks).

  • All softwares are available on all nodes, but we have to load them inside the script with command module load [module].

  • All jobs are contained and can not use more resources than defined (CPU, memory).

  • By default, only one CPU and 2GB of RAM are allocated.

  • Jobs which exceed limits (memory or time, values by default or set) are killed.