Singularity advanced guide

About Singularity

Singularity is a free, cross-platform and open-source computer program that performs operating-system-level virtualization also known as containerization.

One of the main uses of Singularity is to bring containers and reproducibility to scientific computing and the high-performance computing (HPC) world.

The need for reproducibility requires the ability to use containers to move applications from system to system.

Using Singularity containers, developers can work in reproducible environments of their choosing and design, and these complete environments can easily be copied and executed on other platforms.

(Singularity wikipedia page)

How using Singularity

$ srun singularity run shub://GodloveD/lolcow
Progress |===================================| 100.0%
________________________________________
/ RTFM:                                 \
\ Read The Fucking Manual               /
----------------------------------------
       \   ^__^
        \  (oo)\_______
           (__)\       )\/\
               ||----w |
               ||     ||

The documentation is here: User Guide

A selection for survival:

Where finding Singularity images: BioContainer

BioContainers is a community-driven project that provides the infrastructure and basic guidelines to create, manage and distribute bioinformatics packages (e.g conda) and containers (e.g docker, singularity). BioContainers is based on the popular frameworks Conda, Docker and Singularity. (biocontainers docs)

For each BioConda packages, a Docker container is build and push on Quay.io and DockerHub (example: Bioconda/Fastqc)

(source)

BioContainer repositories:

  • Quay.io: https://quay.io/repository/biocontainers/fastqc
  • DockerHub: https://hub.docker.com/r/biocontainers/fastqc

Docker 2 Singularity

Singularity accept to run Docker containers

$ singularity pull docker://godlovedc/lolcow
$ srun ./lolcow_latest.sif
____________________________________
/ You're currently going through a   \
| difficult transition period called |
\ "Life."                            /
------------------------------------
       \   ^__^
        \  (oo)\_______
           (__)\       )\/\
               ||----w |
               ||     ||

A Full Example

The script slurm/sbatch

singularity_bowtie2.slurm

#!/bin/bash
#SBATCH -o slurm.%N.%j.out
#SBATCH -e slurm.%N.%j.err
#SBATCH --partition fast
#SBATCH --cpus-per-task 2
#SBATCH --mem 10GB

SIMAGES=$PWD/singularity_images

#### Do it once
#mkdir -p $SIMAGES; cd $SIMAGES
#singularity pull docker://quay.io/biocontainers/sickle-trim:1.33--hed695b0_5
#singularity pull docker://quay.io/biocontainers/bowtie2:2.4.1--py37h8270d21_3
#singularity pull docker://quay.io/biocontainers/samtools:1.6--h9dace67_6
#cd -

mkdir fastq
wget http://denis.puthier.perso.luminy.univ-amu.fr/siNT_ER_E2_r3_chr21.fastq.gz
gunzip siNT_ER_E2_r3_chr21.fastq.gz
mv siNT_ER_E2_r3_chr21.fastq fastq

mkdir trimmed
$SIMAGES/sickle-trim_1.33--hed695b0_5.sif sickle se -f fastq/siNT_ER_E2_r3_chr21.fastq -t sanger -o trimmed/siNT_ER_E2_r3_chr21_trim.fastq

mkdir index
wget http://hgdownload.soe.ucsc.edu/goldenPath/hg19/chromosomes/chr21.fa.gz
gunzip chr21.fa.gz
mv chr21.fa index
$SIMAGES/bowtie2_2.4.1--py37h8270d21_3.sif bowtie2-build index/chr21.fa index/chr21_hg19

mkdir bam
$SIMAGES/bowtie2_2.4.1--py37h8270d21_3.sif bowtie2 --threads $SLURM_CPUS_PER_TASK -x index/chr21_hg19 -U trimmed/siNT_ER_E2_r3_chr21_trim.fastq -S bam/siNT_ER_E2_r3_chr21_trim.sam

$SIMAGES/samtools_1.6--h9dace67_6.sif samtools view -hbS -q 30 bam/siNT_ER_E2_r3_chr21_trim.sam | $SIMAGES/samtools_1.6--h9dace67_6.sif samtools sort > bam/siNT_ER_E2_r3_chr21_trim.bam

$SIMAGES/samtools_1.6--h9dace67_6.sif samtools index bam/siNT_ER_E2_r3_chr21_trim.bam

$SIMAGES/samtools_1.6--h9dace67_6.sif samtools view -h bam/siNT_ER_E2_r3_chr21_trim.bam | head -20

sbatch

sbatch singularity_bowtie2.slurm

Result

$ cat slurm.*.out

@HD  VN:1.0  SO:coordinate
@SQ     SN:chr21        LN:48129895
@PG     ID:bowtie2      PN:bowtie2      VN:2.3.3.1      CL:"/usr/local/bin/bowtie2-align-s --wrapper basic-0 -x /data/index/chr21_hg19 -S /data/bam/siNT_ER_E2_r3_chr21_trim.sam -U /data/trimmed/siNT_ER_E2_r3_chr21_trim.fastq"
SRR540192.3916166       16      chr21   9411377 42      37M     *       0       0       CTCTGTTCCTTGCTGACCTCCAATTTTGTCTATCCTT   GHFFFEGBHHHHHBHHHHHHHHFFDEHHHHHHHHHHH   AS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:37 YT:Z:UU
SRR540192.7673103       0       chr21   9411551 42      37M     *       0       0       CCGGCCCTTTATCAAGCCCTGGCCACCATGATAGTCA   HHHHFHHHHHHHHHHDHHHHGGGGFHGDHHGGGGEGG   AS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:37 YT:Z:UU
SRR540192.25047223      16      chr21   9411593 42      37M     *       0       0       TCCAATTGTTGTCTATGCAGGCCTACCAGATTTCTAA   IIEIIIIDIIIIIIIIIIIIIIIIIIIIIIIIIIIII   AS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:37 YT:Z:UU
SRR540192.11503770      0       chr21   9411670 42      37M     *       0       0       CAAATGTATCCAAATGAAAGGCTGTGGAGAATGTTGA   IIIIIIIIIIIIIIIIIIIIIIIIGIIIIHIFHIHII   AS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:37 YT:Z:UU
SRR540192.21957275      0       chr21   9411756 42      37M     *       0       0       TCTGGATGCTTTGATTGCTATCAGAAGCCGTTAAATA   IIIIIIIIIIIIIIIIIIIHIIIHHIIIIIIIIIIHG   AS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:37 YT:Z:UU
SRR540192.2371243       0       chr21   9412286 42      37M     *       0       0       CATAAAATGAGTTCTAGAGTTTATTTCTTTACTGCAT   GDG<GEAAEA=5;==DDBB?EBED@=@E@@ADBGG><   AS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:37 YT:Z:UU
SRR540192.16647265      0       chr21   9412303 42      37M     *       0       0       AGTTTATTTCTTTACTGCATCATTCTATTTTCAAGTC   IIIIIIIIIIIIIIIIIIIIIIIIIIHIIIIIIIIII   AS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:37 YT:Z:UU
SRR540192.8808648       16      chr21   9412397 42      37M     *       0       0       TTCATATTTTATTTTTTATTTACTGTATAATTCAGTA   IIIHHIGIIIGIIIIIGHIHHIIIIIIIIIIIIIIII   AS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:37 YT:Z:UU
SRR540192.24964502      0       chr21   9412572 42      37M     *       0       0       CAAATTTGGAAAACTGGAAAAAATATACATGGCAACA   IIIIIIIIIIIIIIIIIGIIIIIIIIIIIIIIIIIII   AS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:37 YT:Z:UU
SRR540192.14173679      0       chr21   9412633 42      37M     *       0       0       CAAACAACTATAAATATTGTTCCACCCAAACAACTAT   IIIIIIIIIIIIIIHIIIHIII3IIIIIIIIIIIIHH   AS:i:-3 XN:i:0  XM:i:1  XO:i:0  XG:i:0  NM:i:1  MD:Z:22T14      YT:Z:UU
SRR540192.28855342      0       chr21   9412728 42      37M     *       0       0       ATTTACAATGGTGTAAACTGTTATACACCATTTATTT   IIIHIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII   AS:i:0  XN:i:0  XM:i:0  XO:i:0  XG:i:0  NM:i:0  MD:Z:37 YT:Z:UU