ETH Zurich Euler cluster configuration
Configuration file to run nf-core pipelines on the Euler cluster of ETH Zurich.
To use the ETH Zurich Euler cluster configuration, run the pipeline with -profile ethz_euler. This will download and launch the ethz_euler.config, which has been pre-configured with a setup suitable for the Euler cluster. Using this profile, a docker image containing all of the required software will be downloaded and converted to a Singularity container before the execution of the pipeline.
Before running the pipeline
Currently, Nextflow is not installed by default on the Euler cluster. You have to install Nextflow version >= 23.07.0-edge on your custom modules.
Important: Previous Nextflow versions will fail to run the job, since the executor option ‘perCpuMemAllocation’ is only available since 23.07.0-edge. This option specifies memory allocations for SLURM jobs as —mem-per-cpu <task.memory / task.cpus> instead of —mem <task.memory>.
To run Nextflow on Euler, you will need to load the following modules:
openjdk - a free and opensource java implementation;
eth_proxy - for the compute nodes communicate with the Internet through the ETH Zurich proxy server;
nextflow - enables scalable and reproducible scientific workflows using software containers (this module can have a user-defined name since was a custom installation).
module load openjdk eth_proxy nextflow
Finally, you will also need to specify the Singularity cache directory with the environmental variable NXF_SINGULARITY_CACHEDIR. (https://www.nextflow.io/docs/latest/singularity.html)
Genomes
In /cluster/project/igenomes, the Euler cluster provides a set of reference genomes and annotations (illumina igenomes) for a selection of model organism. The genomes and annotations were downloaded from: https://support.illumina.com/sequencing/sequencing_software/igenome.html
The path for the igenomes saved in the Euler cluster is already assigned to the parameter variable igenomes_base and the parameter variable igenomes_ignore is set to false so that it loads the igenomes.config when running the pipeline. These default values can be overwritten when running the Nextflow command.
igenomes_base = '/cluster/project/igenomes'
igenomes_ignore = false
(For more information : https://scicomp.ethz.ch/wiki/Reference_genomes)
Config file
/*
* ------------------------------------------------------------------
* nf-core pipelines config file for the ETH Zurich Euler cluster
* ------------------------------------------------------------------
*/
params {
config_profile_name = 'ethz_euler'
config_profile_description = 'ETH Zurich Euler cluster profile -- provided by nf-core/configs'
config_profile_contact = 'Joao Agostinho de Sousa (github: @jpadesousa; email: joao.agostinhodesousa@hest.ethz.ch)'
config_profile_url = 'https://scicomp.ethz.ch/wiki/Euler'
max_memory = 4.TB
max_cpus = 128
max_time = 120.h
igenomes_base = '/cluster/project/igenomes'
igenomes_ignore = false
}
executor {
name = 'slurm'
perCpuMemAllocation = true
}
process {
resourceLimits = [
memory: 4.TB,
cpus: 128,
time: 120.h
]
scratch = 'true'
errorStrategy = 'retry'
maxRetries = 3
beforeScript = 'module load eth_proxy'
}
singularity {
enabled = true
autoMounts = true
}
trace {
trace.overwrite = true
enabled = true
}
// On a successful completion of a Nextflow run, automatically delete all intermediate files stored in the work/ directory
cleanup = true
// Allows to override the default cleanup = true behaviour for debugging
debug {
cleanup = false
}
homepage
contact
