├── chapter_03_grouping ├── main.nf └── data │ ├── reads │ ├── treatmentA │ │ ├── sampleA_rep1_normal_R1.fastq.gz │ │ ├── sampleA_rep1_normal_R2.fastq.gz │ │ ├── sampleA_rep1_tumor_R1.fastq.gz │ │ ├── sampleA_rep1_tumor_R2.fastq.gz │ │ ├── sampleA_rep2_normal_R1.fastq.gz │ │ ├── sampleA_rep2_normal_R2.fastq.gz │ │ ├── sampleA_rep2_tumor_R1.fastq.gz │ │ ├── sampleA_rep2_tumor_R2.fastq.gz │ │ ├── sampleB_rep1_normal_R1.fastq.gz │ │ ├── sampleB_rep1_normal_R2.fastq.gz │ │ ├── sampleB_rep1_tumor_R1.fastq.gz │ │ ├── sampleB_rep1_tumor_R2.fastq.gz │ │ ├── sampleC_rep1_normal_R1.fastq.gz │ │ ├── sampleC_rep1_normal_R2.fastq.gz │ │ ├── sampleC_rep1_tumor_R1.fastq.gz │ │ └── sampleC_rep1_tumor_R2.fastq.gz │ └── treatmentB │ │ ├── sampleA_rep1_normal_R1.fastq.gz │ │ ├── sampleA_rep1_normal_R2.fastq.gz │ │ ├── sampleA_rep1_tumor_R1.fastq.gz │ │ └── sampleA_rep1_tumor_R2.fastq.gz │ ├── genome.fasta.fai │ ├── genome.fasta │ ├── intervals.bed │ ├── samplesheet.ugly.csv │ └── samplesheet.csv ├── docs ├── .gitignore ├── summary.qmd ├── references.qmd ├── static │ └── img │ │ ├── metadata.png │ │ ├── seqera-logo-black.png │ │ └── codespaces_screenshot.png ├── _book │ ├── static │ │ └── img │ │ │ ├── metadata.png │ │ │ └── seqera-logo-black.png │ ├── site_libs │ │ ├── bootstrap │ │ │ └── bootstrap-icons.woff │ │ ├── quarto-html │ │ │ ├── tippy.css │ │ │ ├── zenscroll-min.js │ │ │ ├── quarto-syntax-highlighting.css │ │ │ └── anchor.min.js │ │ ├── quarto-nav │ │ │ ├── headroom.min.js │ │ │ └── quarto-nav.js │ │ └── clipboard │ │ │ └── clipboard.min.js │ ├── references.html │ ├── summary.html │ ├── support.html │ ├── index.html │ ├── testing.html │ └── configuration.html ├── support.qmd ├── workshop.scss ├── development.qmd ├── testing.qmd ├── index.qmd ├── _quarto.yml ├── configuration.qmd ├── metadata.qmd ├── structure.qmd ├── grouping.qmd ├── groovy.qmd └── operators.qmd ├── .gitignore ├── chapter_01_operators ├── data │ ├── reads │ │ ├── sampleA_rep1_normal_R1.fastq.gz │ │ ├── sampleA_rep1_normal_R2.fastq.gz │ │ ├── sampleA_rep1_tumor_R1.fastq.gz │ │ ├── sampleA_rep1_tumor_R2.fastq.gz │ │ ├── sampleA_rep2_normal_R1.fastq.gz │ │ ├── sampleA_rep2_normal_R2.fastq.gz │ │ ├── sampleA_rep2_tumor_R1.fastq.gz │ │ ├── sampleA_rep2_tumor_R2.fastq.gz │ │ ├── sampleB_rep1_normal_R1.fastq.gz │ │ ├── sampleB_rep1_normal_R2.fastq.gz │ │ ├── sampleB_rep1_tumor_R1.fastq.gz │ │ ├── sampleB_rep1_tumor_R2.fastq.gz │ │ ├── sampleC_rep1_normal_R1.fastq.gz │ │ ├── sampleC_rep1_normal_R2.fastq.gz │ │ ├── sampleC_rep1_tumor_R1.fastq.gz │ │ └── sampleC_rep1_tumor_R2.fastq.gz │ ├── samplesheet.ugly.csv │ └── samplesheet.csv └── main.nf ├── run.sh ├── chapter_02_metadata ├── data │ ├── reads │ │ ├── treatmentA │ │ │ ├── sampleA_rep1_normal_R1.fastq.gz │ │ │ ├── sampleA_rep1_normal_R2.fastq.gz │ │ │ ├── sampleA_rep1_tumor_R1.fastq.gz │ │ │ ├── sampleA_rep1_tumor_R2.fastq.gz │ │ │ ├── sampleA_rep2_normal_R1.fastq.gz │ │ │ ├── sampleA_rep2_normal_R2.fastq.gz │ │ │ ├── sampleA_rep2_tumor_R1.fastq.gz │ │ │ ├── sampleA_rep2_tumor_R2.fastq.gz │ │ │ ├── sampleB_rep1_normal_R1.fastq.gz │ │ │ ├── sampleB_rep1_normal_R2.fastq.gz │ │ │ ├── sampleB_rep1_tumor_R1.fastq.gz │ │ │ ├── sampleB_rep1_tumor_R2.fastq.gz │ │ │ ├── sampleC_rep1_normal_R1.fastq.gz │ │ │ ├── sampleC_rep1_normal_R2.fastq.gz │ │ │ ├── sampleC_rep1_tumor_R1.fastq.gz │ │ │ └── sampleC_rep1_tumor_R2.fastq.gz │ │ └── treatmentB │ │ │ ├── sampleA_rep1_normal_R1.fastq.gz │ │ │ ├── sampleA_rep1_normal_R2.fastq.gz │ │ │ ├── sampleA_rep1_tumor_R1.fastq.gz │ │ │ └── sampleA_rep1_tumor_R2.fastq.gz │ ├── samplesheet.ugly.csv │ └── samplesheet.csv └── main.nf ├── chapter_05_structure ├── lib │ └── Food.groovy ├── main.nf ├── templates │ ├── adder.py │ └── demo_script.sh └── bin │ └── cars.R ├── .devcontainer ├── nf-test-feature │ ├── devcontainer-feature.json │ └── install.sh ├── nextflow-feature │ ├── install.sh │ └── devcontainer-feature.json └── devcontainer.json ├── chapter_04_groovy ├── main.nf └── modules │ └── local │ └── fastp │ └── main.nf ├── README.md ├── Dockerfile └── .github └── workflows └── publish-image.yml /chapter_03_grouping/main.nf: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs/.gitignore: -------------------------------------------------------------------------------- 1 | /.quarto/ 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | chapter_*/.nextflow* 2 | chapter_*/work -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleA_rep1_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleA_rep1_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleA_rep1_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleA_rep1_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleA_rep2_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleA_rep2_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleA_rep2_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleA_rep2_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleB_rep1_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleB_rep1_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleB_rep1_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleB_rep1_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleC_rep1_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleC_rep1_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleC_rep1_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_01_operators/data/reads/sampleC_rep1_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs/summary.qmd: -------------------------------------------------------------------------------- 1 | # Summary 2 | 3 | Summary available on day #2 -------------------------------------------------------------------------------- /run.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | cd docs/_book 4 | python -m http.server -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleA_rep1_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleA_rep1_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleA_rep1_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleA_rep1_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleA_rep2_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleA_rep2_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleA_rep2_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleA_rep2_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleB_rep1_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleB_rep1_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleB_rep1_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleB_rep1_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleC_rep1_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleC_rep1_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleC_rep1_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentA/sampleC_rep1_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentB/sampleA_rep1_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentB/sampleA_rep1_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentB/sampleA_rep1_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/reads/treatmentB/sampleA_rep1_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleA_rep1_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleA_rep1_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleA_rep1_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleA_rep1_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleA_rep2_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleA_rep2_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleA_rep2_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleA_rep2_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleB_rep1_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleB_rep1_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleB_rep1_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleB_rep1_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleC_rep1_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleC_rep1_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleC_rep1_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentA/sampleC_rep1_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentB/sampleA_rep1_normal_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentB/sampleA_rep1_normal_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentB/sampleA_rep1_tumor_R1.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/reads/treatmentB/sampleA_rep1_tumor_R2.fastq.gz: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /chapter_05_structure/lib/Food.groovy: -------------------------------------------------------------------------------- 1 | enum Food { 2 | BONE, MEAT, BISCUIT 3 | } -------------------------------------------------------------------------------- /docs/references.qmd: -------------------------------------------------------------------------------- 1 | # References {.unnumbered} 2 | 3 | ::: {#refs} 4 | ::: 5 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/genome.fasta.fai: -------------------------------------------------------------------------------- 1 | chr1 11 6 11 12 2 | chr2 12 24 12 13 3 | chr3 13 43 13 14 4 | -------------------------------------------------------------------------------- /chapter_05_structure/main.nf: -------------------------------------------------------------------------------- 1 | workflow { 2 | names = Channel.of("Argente", "Absolon", "Chowne") 3 | } 4 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/genome.fasta: -------------------------------------------------------------------------------- 1 | >chr1 2 | AGCTATCTGAC 3 | >chr2 4 | CAGTACGTTGAC 5 | >chr3 6 | ACGTACGCGTACG -------------------------------------------------------------------------------- /chapter_03_grouping/data/intervals.bed: -------------------------------------------------------------------------------- 1 | chr1 0 11 interval1 2 | chr2 0 12 interval2 3 | chr3 0 13 interval3 4 | -------------------------------------------------------------------------------- /chapter_02_metadata/main.nf: -------------------------------------------------------------------------------- 1 | workflow { 2 | Channel.fromFilePairs("data/reads/*/*_R{1,2}.fastq.gz") 3 | | view 4 | } -------------------------------------------------------------------------------- /chapter_05_structure/templates/adder.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | print("Hello $name!") 4 | print("Process completed") -------------------------------------------------------------------------------- /docs/static/img/metadata.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seqeralabs/nf-training-advanced/HEAD/docs/static/img/metadata.png -------------------------------------------------------------------------------- /chapter_01_operators/main.nf: -------------------------------------------------------------------------------- 1 | workflow { 2 | Channel.of( 1, 2, 3, 4, 5 ) 3 | | map { it * it } 4 | | view 5 | } 6 | -------------------------------------------------------------------------------- /docs/_book/static/img/metadata.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seqeralabs/nf-training-advanced/HEAD/docs/_book/static/img/metadata.png -------------------------------------------------------------------------------- /docs/static/img/seqera-logo-black.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seqeralabs/nf-training-advanced/HEAD/docs/static/img/seqera-logo-black.png -------------------------------------------------------------------------------- /docs/_book/static/img/seqera-logo-black.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seqeralabs/nf-training-advanced/HEAD/docs/_book/static/img/seqera-logo-black.png -------------------------------------------------------------------------------- /docs/static/img/codespaces_screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seqeralabs/nf-training-advanced/HEAD/docs/static/img/codespaces_screenshot.png -------------------------------------------------------------------------------- /chapter_05_structure/templates/demo_script.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "process started at `date`" 4 | echo Hi there, $name 5 | echo "process completed" 6 | -------------------------------------------------------------------------------- /docs/_book/site_libs/bootstrap/bootstrap-icons.woff: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seqeralabs/nf-training-advanced/HEAD/docs/_book/site_libs/bootstrap/bootstrap-icons.woff -------------------------------------------------------------------------------- /docs/support.qmd: -------------------------------------------------------------------------------- 1 | # Links 2 | 3 | - Nextflow documentation: [www.nextflow.io/docs/latest](https://www.nextflow.io/docs/latest/) 4 | - Seqera support: [support@seqera.io](mailto:support@seqera.io) -------------------------------------------------------------------------------- /.devcontainer/nf-test-feature/devcontainer-feature.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "nf-test", 3 | "id": "nf-test", 4 | "version": "1.0.0", 5 | "description": "nf-test feature", 6 | "installsAfter": ["./nextflow-feature"] 7 | } 8 | -------------------------------------------------------------------------------- /.devcontainer/nextflow-feature/install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -e 3 | 4 | echo "Activating feature 'nextflow'" 5 | 6 | su ${_REMOTE_USER} -c "mkdir -p ~/.local/bin && cd ~/.local/bin && curl -s https://get.nextflow.io | bash" 7 | 8 | 9 | 10 | -------------------------------------------------------------------------------- /.devcontainer/nextflow-feature/devcontainer-feature.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "Nextflow", 3 | "id": "nextflow", 4 | "version": "1.0.0", 5 | "description": "Nextflow feature", 6 | "installsAfter": ["ghcr.io/devcontainers/features/java:1"] 7 | } 8 | -------------------------------------------------------------------------------- /chapter_05_structure/bin/cars.R: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env Rscript 2 | 3 | library(tidyverse) 4 | 5 | plot <- ggplot(mpg, aes(displ, hwy, colour = class)) + geom_point() 6 | 7 | mtcars |> write_tsv("cars.tsv") 8 | 9 | ggsave("cars.png", plot = plot) 10 | 11 | -------------------------------------------------------------------------------- /.devcontainer/nf-test-feature/install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -e 3 | 4 | echo "Activating feature 'nf-test'" 5 | 6 | su ${_REMOTE_USER} -c "mkdir -p ~/.local/bin && cd ~/.local/bin && curl -fsSL https://code.askimed.com/install/nf-test | bash" 7 | 8 | 9 | 10 | -------------------------------------------------------------------------------- /chapter_04_groovy/main.nf: -------------------------------------------------------------------------------- 1 | include { FASTP } from './modules/local/fastp/main.nf' 2 | 3 | workflow { 4 | params.input = "https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/samplesheet/v3.4/samplesheet_test.csv" 5 | 6 | Channel.fromPath(params.input) 7 | | splitCsv(header: true) 8 | | view 9 | } -------------------------------------------------------------------------------- /docs/workshop.scss: -------------------------------------------------------------------------------- 1 | /*-- scss:defaults --*/ 2 | 3 | $primary: #637238 !default; 4 | 5 | .sidebar-title { 6 | color: #27ae60; 7 | } 8 | 9 | .result { 10 | background-color: rgba(233,236,239,.65); 11 | color: #575f66; 12 | border-radius: .25rem; 13 | border: 1px solid #575f66; 14 | } 15 | 16 | -------------------------------------------------------------------------------- /docs/development.qmd: -------------------------------------------------------------------------------- 1 | # Iterative Development 2 | 3 | How to most effectively use caching features for rapid workflow development. Advocating for judicious use of the “view” operator to inspect channel outputs. TODO: Check dump channels 4 | Subscribe operator - Recent ONT example 5 | 6 | This chapter is a gentle instroduction to the workshop style and method by constructing a very small pipeline and going though the 7 | 8 | Nextflow scripts are -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Seqera Customer Training Workshop 2 | 3 | This repository contains materials for the Seqera customer training workshop. 4 | 5 | To participate in the workshop, select the green "Code" button at top-right and then "Create codespace on master": 6 | 7 | ![Opening codespace](docs/static/img/codespaces_screenshot.png) 8 | 9 | When the codespace container is ready, from the command line run the `run.sh` script to open up the materials in a new browser. 10 | 11 | ```bash 12 | ./run.sh 13 | ``` 14 | 15 | 16 | 17 | -------------------------------------------------------------------------------- /docs/testing.qmd: -------------------------------------------------------------------------------- 1 | # Troubleshooting 2 | 3 | To be introduced on day #2 4 | 5 | 13 | -------------------------------------------------------------------------------- /docs/index.qmd: -------------------------------------------------------------------------------- 1 | # Welcome {.unnumbered} 2 | 3 | Welcome to our Nextflow workshop for intermediate and advanced users! In this workshop, we will explore the advanced features of the Nextflow language and runtime, and learn how to use them to write efficient and scalable data-intensive workflows. We will cover topics such as parallel execution, error handling, and workflow customization. Please note that this is not an introductory workshop, and we will assume some basic familiarity with Nextflow. By the end of this workshop, you will have the skills and knowledge to create complex and powerful Nextflow pipelines for your own data analysis projects. Let's get started! 4 | 5 | -------------------------------------------------------------------------------- /chapter_02_metadata/data/samplesheet.ugly.csv: -------------------------------------------------------------------------------- 1 | id,repeat,normal_fastq_1,normal_fastq_2,tumor_fastq_1,tumor_fastq_2 2 | sampleA,1,data/reads/sampleA_normal_R1.fastq.gz,data/reads/sampleA_normal_R2.fastq.gz,data/reads/sampleA_tumor_R1.fastq.gz,data/reads/sampleA_tumor_R2.fastq.gz 3 | sampleA,2,data/reads/sampleA_normal_R1.fastq.gz,data/reads/sampleA_normal_R2.fastq.gz,data/reads/sampleA_tumor_R1.fastq.gz,data/reads/sampleA_tumor_R2.fastq.gz 4 | sampleB,1,data/reads/sampleB_normal_R1.fastq.gz,data/reads/sampleB_normal_R2.fastq.gz,data/reads/sampleB_tumor_R1.fastq.gz,data/reads/sampleB_tumor_R2.fastq.gz 5 | sampleC,1,data/reads/sampleC_normal_R1.fastq.gz,data/reads/sampleC_normal_R2.fastq.gz,data/reads/sampleC_tumor_R1.fastq.gz,data/reads/sampleC_tumor_R2.fastq.gz 6 | -------------------------------------------------------------------------------- /chapter_03_grouping/data/samplesheet.ugly.csv: -------------------------------------------------------------------------------- 1 | id,repeat,normal_fastq_1,normal_fastq_2,tumor_fastq_1,tumor_fastq_2 2 | sampleA,1,data/reads/sampleA_normal_R1.fastq.gz,data/reads/sampleA_normal_R2.fastq.gz,data/reads/sampleA_tumor_R1.fastq.gz,data/reads/sampleA_tumor_R2.fastq.gz 3 | sampleA,2,data/reads/sampleA_normal_R1.fastq.gz,data/reads/sampleA_normal_R2.fastq.gz,data/reads/sampleA_tumor_R1.fastq.gz,data/reads/sampleA_tumor_R2.fastq.gz 4 | sampleB,1,data/reads/sampleB_normal_R1.fastq.gz,data/reads/sampleB_normal_R2.fastq.gz,data/reads/sampleB_tumor_R1.fastq.gz,data/reads/sampleB_tumor_R2.fastq.gz 5 | sampleC,1,data/reads/sampleC_normal_R1.fastq.gz,data/reads/sampleC_normal_R2.fastq.gz,data/reads/sampleC_tumor_R1.fastq.gz,data/reads/sampleC_tumor_R2.fastq.gz 6 | -------------------------------------------------------------------------------- /chapter_01_operators/data/samplesheet.ugly.csv: -------------------------------------------------------------------------------- 1 | id,repeat,normal_fastq_1,normal_fastq_2,tumor_fastq_1,tumor_fastq_2 2 | sampleA,1,data/reads/sampleA_normal_R1.fastq.gz,data/reads/sampleA_normal_R2.fastq.gz,data/reads/sampleA_tumor_R1.fastq.gz,data/reads/sampleA_tumor_R2.fastq.gz 3 | sampleA,2,data/reads/sampleA_normal_R1.fastq.gz,data/reads/sampleA_normal_R2.fastq.gz,data/reads/sampleA_tumor_R1.fastq.gz,data/reads/sampleA_tumor_R2.fastq.gz 4 | sampleB,1,data/reads/sampleB_normal_R1.fastq.gz,data/reads/sampleB_normal_R2.fastq.gz,data/reads/sampleB_tumor_R1.fastq.gz,data/reads/sampleB_tumor_R2.fastq.gz 5 | sampleC,1,data/reads/sampleC_normal_R1.fastq.gz,data/reads/sampleC_normal_R2.fastq.gz,data/reads/sampleC_tumor_R1.fastq.gz,data/reads/sampleC_tumor_R2.fastq.gz 6 | -------------------------------------------------------------------------------- /docs/_quarto.yml: -------------------------------------------------------------------------------- 1 | project: 2 | type: book 3 | 4 | book: 5 | title: "Intermediate/Advanced Nextflow" 6 | navbar: 7 | logo: static/img/seqera-logo-black.png 8 | background: "#ecf0f1" 9 | output-file: "my-book" 10 | chapters: 11 | - index.qmd 12 | - part: Curriculum 13 | chapters: 14 | - operators.qmd 15 | - metadata.qmd 16 | - grouping.qmd 17 | - groovy.qmd 18 | - structure.qmd 19 | - configuration.qmd 20 | - testing.qmd 21 | - summary.qmd 22 | - part: Support 23 | chapters: 24 | - support.qmd 25 | - references.qmd 26 | 27 | format: 28 | html: 29 | smooth-scroll: true 30 | theme: 31 | - cosmo 32 | - workshop.scss 33 | 34 | highlight-style: ayu 35 | number-depth: 2 36 | execute: 37 | eval: false 38 | 39 | -------------------------------------------------------------------------------- /chapter_01_operators/data/samplesheet.csv: -------------------------------------------------------------------------------- 1 | id,repeat,type,fastq1,fastq2 2 | sampleA,1,normal,data/reads/sampleA_rep1_normal_R1.fastq.gz,data/reads/sampleA_rep1_normal_R2.fastq.gz 3 | sampleA,1,tumor,data/reads/sampleA_rep1_tumor_R1.fastq.gz,data/reads/sampleA_rep1_tumor_R2.fastq.gz 4 | sampleA,2,normal,data/reads/sampleA_rep2_normal_R1.fastq.gz,data/reads/sampleA_rep2_normal_R2.fastq.gz 5 | sampleA,2,tumor,data/reads/sampleA_rep2_tumor_R1.fastq.gz,data/reads/sampleA_rep2_tumor_R2.fastq.gz 6 | sampleB,1,normal,data/reads/sampleB_rep1_normal_R1.fastq.gz,data/reads/sampleB_rep1_normal_R2.fastq.gz 7 | sampleB,1,tumor,data/reads/sampleB_rep1_tumor_R1.fastq.gz,data/reads/sampleB_rep1_tumor_R2.fastq.gz 8 | sampleC,1,normal,data/reads/sampleC_rep1_normal_R1.fastq.gz,data/reads/sampleC_rep1_normal_R2.fastq.gz 9 | sampleC,1,tumor,data/reads/sampleC_rep1_tumor_R1.fastq.gz,data/reads/sampleC_rep1_tumor_R2.fastq.gz -------------------------------------------------------------------------------- /chapter_02_metadata/data/samplesheet.csv: -------------------------------------------------------------------------------- 1 | id,repeat,type,fastq1,fastq2 2 | sampleA,1,normal,data/reads/treatmentA/sampleA_rep1_normal_R1.fastq.gz,data/reads/treatmentA/sampleA_rep1_normal_R2.fastq.gz 3 | sampleA,1,tumor,data/reads/treatmentA/sampleA_rep1_tumor_R1.fastq.gz,data/reads/treatmentA/sampleA_rep1_tumor_R2.fastq.gz 4 | sampleA,2,normal,data/reads/treatmentA/sampleA_rep2_normal_R1.fastq.gz,data/reads/treatmentA/sampleA_rep2_normal_R2.fastq.gz 5 | sampleA,2,tumor,data/reads/treatmentA/sampleA_rep2_tumor_R1.fastq.gz,data/reads/treatmentA/sampleA_rep2_tumor_R2.fastq.gz 6 | sampleB,1,normal,data/reads/treatmentA/sampleB_rep1_normal_R1.fastq.gz,data/reads/treatmentA/sampleB_rep1_normal_R2.fastq.gz 7 | sampleB,1,tumor,data/reads/treatmentA/sampleB_rep1_tumor_R1.fastq.gz,data/reads/treatmentA/sampleB_rep1_tumor_R2.fastq.gz 8 | sampleC,1,normal,data/reads/treatmentA/sampleC_rep1_normal_R1.fastq.gz,data/reads/treatmentA/sampleC_rep1_normal_R2.fastq.gz 9 | sampleC,1,tumor,data/reads/treatmentA/sampleC_rep1_tumor_R1.fastq.gz,data/reads/treatmentA/sampleC_rep1_tumor_R2.fastq.gz -------------------------------------------------------------------------------- /chapter_03_grouping/data/samplesheet.csv: -------------------------------------------------------------------------------- 1 | id,repeat,type,fastq1,fastq2 2 | sampleA,1,normal,data/reads/treatmentA/sampleA_rep1_normal_R1.fastq.gz,data/reads/treatmentA/sampleA_rep1_normal_R2.fastq.gz 3 | sampleA,1,tumor,data/reads/treatmentA/sampleA_rep1_tumor_R1.fastq.gz,data/reads/treatmentA/sampleA_rep1_tumor_R2.fastq.gz 4 | sampleA,2,normal,data/reads/treatmentA/sampleA_rep2_normal_R1.fastq.gz,data/reads/treatmentA/sampleA_rep2_normal_R2.fastq.gz 5 | sampleA,2,tumor,data/reads/treatmentA/sampleA_rep2_tumor_R1.fastq.gz,data/reads/treatmentA/sampleA_rep2_tumor_R2.fastq.gz 6 | sampleB,1,normal,data/reads/treatmentA/sampleB_rep1_normal_R1.fastq.gz,data/reads/treatmentA/sampleB_rep1_normal_R2.fastq.gz 7 | sampleB,1,tumor,data/reads/treatmentA/sampleB_rep1_tumor_R1.fastq.gz,data/reads/treatmentA/sampleB_rep1_tumor_R2.fastq.gz 8 | sampleC,1,normal,data/reads/treatmentA/sampleC_rep1_normal_R1.fastq.gz,data/reads/treatmentA/sampleC_rep1_normal_R2.fastq.gz 9 | sampleC,1,tumor,data/reads/treatmentA/sampleC_rep1_tumor_R1.fastq.gz,data/reads/treatmentA/sampleC_rep1_tumor_R2.fastq.gz -------------------------------------------------------------------------------- /.devcontainer/devcontainer.json: -------------------------------------------------------------------------------- 1 | // For format details, see https://aka.ms/devcontainer.json. For config options, see the 2 | // README at: https://github.com/devcontainers/templates/tree/main/src/ubuntu 3 | { 4 | "name": "Ubuntu", 5 | "image": "ghcr.io/robsyme/workshop:release", 6 | "features": { 7 | "ghcr.io/devcontainers/features/conda:1": {}, 8 | "ghcr.io/devcontainers/features/docker-in-docker:2": {}, 9 | "./nextflow-feature": {}, 10 | "./nf-test-feature": {} 11 | }, 12 | 13 | // Features to add to the dev container. More info: https://containers.dev/features. 14 | // "features": {}, 15 | 16 | // Use 'forwardPorts' to make a list of ports inside the container available locally. 17 | // "forwardPorts": [], 18 | 19 | // Use 'postCreateCommand' to run commands after the container is created. 20 | // "postCreateCommand": "cd docs/_book && python -m http.server", 21 | 22 | // Configure tool-specific properties. 23 | // "customizations": {}, 24 | 25 | // Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root. 26 | "remoteUser": "workshopper", 27 | "extensions": ["quarto.quarto", "REditorSupport.r", "nf-core.nf-core-extensionpack"] 28 | } 29 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ubuntu:jammy 2 | 3 | ENV DEBIAN_FRONTEND noninteractive 4 | RUN apt-get update && apt-get install --yes --no-install-recommends wget curl tree unzip zip ca-certificates sudo htop 5 | RUN apt-get install --yes --no-install-recommends default-jre r-base r-base-dev libcurl4-openssl-dev libssl-dev libxml2-dev texlive-base texlive-xetex 6 | RUN wget -q -O- https://eddelbuettel.github.io/r2u/assets/dirk_eddelbuettel_key.asc | tee -a /etc/apt/trusted.gpg.d/cranapt_key.asc \ 7 | && echo "deb [arch=amd64] https://dirk.eddelbuettel.com/cranapt jammy main" > /etc/apt/sources.list.d/cranapt.list 8 | RUN wget -q -O- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc \ 9 | && echo "deb [arch=amd64] https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/" > /etc/apt/sources.list.d/cran-ubuntu.list && apt update 10 | RUN apt-get install --yes r-cran-tidyverse r-cran-rmarkdown r-cran-magrittr r-cran-quarto r-cran-languageserver 11 | RUN cd /tmp && wget https://github.com/quarto-dev/quarto-cli/releases/download/v1.2.269/quarto-1.2.269-linux-amd64.deb && dpkg -i quarto-1.2.269-linux-amd64.deb && rm *.deb 12 | RUN useradd -ms /bin/bash workshopper && adduser workshopper sudo 13 | RUN groupadd docker && usermod -aG docker workshopper && echo "workshopper:workshopper" | chpasswd 14 | # USER workshopper 15 | # WORKDIR /home/workshopper -------------------------------------------------------------------------------- /docs/_book/site_libs/quarto-html/tippy.css: -------------------------------------------------------------------------------- 1 | .tippy-box[data-animation=fade][data-state=hidden]{opacity:0}[data-tippy-root]{max-width:calc(100vw - 10px)}.tippy-box{position:relative;background-color:#333;color:#fff;border-radius:4px;font-size:14px;line-height:1.4;white-space:normal;outline:0;transition-property:transform,visibility,opacity}.tippy-box[data-placement^=top]>.tippy-arrow{bottom:0}.tippy-box[data-placement^=top]>.tippy-arrow:before{bottom:-7px;left:0;border-width:8px 8px 0;border-top-color:initial;transform-origin:center top}.tippy-box[data-placement^=bottom]>.tippy-arrow{top:0}.tippy-box[data-placement^=bottom]>.tippy-arrow:before{top:-7px;left:0;border-width:0 8px 8px;border-bottom-color:initial;transform-origin:center bottom}.tippy-box[data-placement^=left]>.tippy-arrow{right:0}.tippy-box[data-placement^=left]>.tippy-arrow:before{border-width:8px 0 8px 8px;border-left-color:initial;right:-7px;transform-origin:center left}.tippy-box[data-placement^=right]>.tippy-arrow{left:0}.tippy-box[data-placement^=right]>.tippy-arrow:before{left:-7px;border-width:8px 8px 8px 0;border-right-color:initial;transform-origin:center right}.tippy-box[data-inertia][data-state=visible]{transition-timing-function:cubic-bezier(.54,1.5,.38,1.11)}.tippy-arrow{width:16px;height:16px;color:#333}.tippy-arrow:before{content:"";position:absolute;border-color:transparent;border-style:solid}.tippy-content{position:relative;padding:5px 9px;z-index:1} -------------------------------------------------------------------------------- /docs/configuration.qmd: -------------------------------------------------------------------------------- 1 | # Configuration 2 | To be introduced on day #2 3 | 4 | This is an aspect of Nextflow that can be confusing. There are multiple ways of loading configuration and parameters into a Nextflow. This gives us two complications: 5 | 6 | - At which location should I be loading a configuration value? 7 | - Given a particular parameter, how do I know where it was set? 8 | 9 | ## Precedence 10 | 11 | 1. Parameters specified on the command line (--something value) 12 | 2. Parameters provided using the -params-file option 13 | 3. Config file specified using the -c my_config option 14 | 4. The config file named nextflow.config in the current directory 15 | 5. The config file named nextflow.config in the workflow project directory 16 | 6. The config file $HOME/.nextflow/config 17 | 7. Values defined within the pipeline script itself (e.g. main.nf) 18 | 19 | 28 | -------------------------------------------------------------------------------- /chapter_04_groovy/modules/local/fastp/main.nf: -------------------------------------------------------------------------------- 1 | process FASTP { 2 | container 'quay.io/biocontainers/fastp:0.23.2--h79da9fb_0' 3 | 4 | input: 5 | tuple val(meta), path(reads) 6 | 7 | output: 8 | tuple val(meta), path('*.fastp.fastq.gz') , optional:true, emit: reads 9 | tuple val(meta), path('*.json') , emit: json 10 | 11 | script: 12 | def prefix = task.ext.prefix ?: meta.id 13 | if (meta.single_end) { 14 | """ 15 | [ ! -f ${prefix}.fastq.gz ] && ln -sf $reads ${prefix}.fastq.gz 16 | fastp \\ 17 | --in1 ${prefix}.fastq.gz \\ 18 | --out1 ${prefix}.fastp.fastq.gz \\ 19 | --json ${prefix}.fastp.json \\ 20 | --html ${prefix}.fastp.html \\ 21 | --thread $task.cpus \\ 22 | --detect_adapter_for_pe \\ 23 | 2> ${prefix}.fastp.log 24 | """ 25 | } else { 26 | """ 27 | [ ! -f ${prefix}_1.fastq.gz ] && ln -sf ${reads[0]} ${prefix}_1.fastq.gz 28 | [ ! -f ${prefix}_2.fastq.gz ] && ln -sf ${reads[1]} ${prefix}_2.fastq.gz 29 | fastp \\ 30 | --in1 ${prefix}_1.fastq.gz \\ 31 | --in2 ${prefix}_2.fastq.gz \\ 32 | --out1 ${prefix}_1.fastp.fastq.gz \\ 33 | --out2 ${prefix}_2.fastp.fastq.gz \\ 34 | --json ${prefix}.fastp.json \\ 35 | --html ${prefix}.fastp.html \\ 36 | --thread $task.cpus \\ 37 | --detect_adapter_for_pe \\ 38 | 2> ${prefix}.fastp.log 39 | """ 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /.github/workflows/publish-image.yml: -------------------------------------------------------------------------------- 1 | # This workflow uses actions that are not certified by GitHub. 2 | # They are provided by a third-party and are governed by 3 | # separate terms of service, privacy policy, and support 4 | # documentation. 5 | 6 | # GitHub recommends pinning actions to a commit SHA. 7 | # To get a newer version, you will need to update the SHA. 8 | # You can also reference a tag or branch, but the action may change without warning. 9 | 10 | name: Create and publish a Docker image 11 | 12 | on: 13 | push: 14 | branches: ['release'] 15 | 16 | env: 17 | REGISTRY: ghcr.io 18 | IMAGE_NAME: ${{ github.repository }} 19 | 20 | jobs: 21 | build-and-push-image: 22 | runs-on: ubuntu-latest 23 | permissions: 24 | contents: read 25 | packages: write 26 | 27 | steps: 28 | - name: Checkout repository 29 | uses: actions/checkout@v3 30 | 31 | - name: Log in to the Container registry 32 | uses: docker/login-action@f054a8b539a109f9f41c372932f1ae047eff08c9 33 | with: 34 | registry: ${{ env.REGISTRY }} 35 | username: ${{ github.actor }} 36 | password: ${{ secrets.GITHUB_TOKEN }} 37 | 38 | - name: Extract metadata (tags, labels) for Docker 39 | id: meta 40 | uses: docker/metadata-action@98669ae865ea3cffbcbaa878cf57c20bbf1c6c38 41 | with: 42 | images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} 43 | 44 | - name: Build and push Docker image 45 | uses: docker/build-push-action@ad44023a93711e3deb337508980b4b5e9bcdc5dc 46 | with: 47 | context: . 48 | push: true 49 | tags: ${{ steps.meta.outputs.tags }} 50 | labels: ${{ steps.meta.outputs.labels }} -------------------------------------------------------------------------------- /docs/_book/site_libs/quarto-html/zenscroll-min.js: -------------------------------------------------------------------------------- 1 | !function(t,e){"function"==typeof define&&define.amd?define([],e()):"object"==typeof module&&module.exports?module.exports=e():function n(){document&&document.body?t.zenscroll=e():setTimeout(n,9)}()}(this,function(){"use strict";var t=function(t){return t&&"getComputedStyle"in window&&"smooth"===window.getComputedStyle(t)["scroll-behavior"]};if("undefined"==typeof window||!("document"in window))return{};var e=function(e,n,o){n=n||999,o||0===o||(o=9);var i,r=function(t){i=t},u=function(){clearTimeout(i),r(0)},c=function(t){return Math.max(0,e.getTopOf(t)-o)},a=function(o,i,c){if(u(),0===i||i&&i<0||t(e.body))e.toY(o),c&&c();else{var a=e.getY(),f=Math.max(0,o)-a,s=(new Date).getTime();i=i||Math.min(Math.abs(f),n),function t(){r(setTimeout(function(){var n=Math.min(1,((new Date).getTime()-s)/i),o=Math.max(0,Math.floor(a+f*(n<.5?2*n*n:n*(4-2*n)-1)));e.toY(o),n<1&&e.getHeight()+os?f(t,n,i):u+o>d?a(u-s+o,n,i):i&&i()},l=function(t,n,o,i){a(Math.max(0,e.getTopOf(t)-e.getHeight()/2+(o||t.getBoundingClientRect().height/2)),n,i)};return{setup:function(t,e){return(0===t||t)&&(n=t),(0===e||e)&&(o=e),{defaultDuration:n,edgeOffset:o}},to:f,toY:a,intoView:s,center:l,stop:u,moving:function(){return!!i},getY:e.getY,getTopOf:e.getTopOf}},n=document.documentElement,o=function(){return window.scrollY||n.scrollTop},i=e({body:document.scrollingElement||document.body,toY:function(t){window.scrollTo(0,t)},getY:o,getHeight:function(){return window.innerHeight||n.clientHeight},getTopOf:function(t){return t.getBoundingClientRect().top+o()-n.offsetTop}});if(i.createScroller=function(t,o,i){return e({body:t,toY:function(e){t.scrollTop=e},getY:function(){return t.scrollTop},getHeight:function(){return Math.min(t.clientHeight,window.innerHeight||n.clientHeight)},getTopOf:function(t){return t.offsetTop}},o,i)},"addEventListener"in window&&!window.noZensmooth&&!t(document.body)){var r="history"in window&&"pushState"in history,u=r&&"scrollRestoration"in history;u&&(history.scrollRestoration="auto"),window.addEventListener("load",function(){u&&(setTimeout(function(){history.scrollRestoration="manual"},9),window.addEventListener("popstate",function(t){t.state&&"zenscrollY"in t.state&&i.toY(t.state.zenscrollY)},!1)),window.location.hash&&setTimeout(function(){var t=i.setup().edgeOffset;if(t){var e=document.getElementById(window.location.href.split("#")[1]);if(e){var n=Math.max(0,i.getTopOf(e)-t),o=i.getY()-n;0<=o&&o<9&&window.scrollTo(0,n)}}},9)},!1);var c=new RegExp("(^|\\s)noZensmooth(\\s|$)");window.addEventListener("click",function(t){for(var e=t.target;e&&"A"!==e.tagName;)e=e.parentNode;if(!(!e||1!==t.which||t.shiftKey||t.metaKey||t.ctrlKey||t.altKey)){if(u){var n=history.state&&"object"==typeof history.state?history.state:{};n.zenscrollY=i.getY();try{history.replaceState(n,"")}catch(t){}}var o=e.getAttribute("href")||"";if(0===o.indexOf("#")&&!c.test(e.className)){var a=0,f=document.getElementById(o.substring(1));if("#"!==o){if(!f)return;a=i.getTopOf(f)}t.preventDefault();var s=function(){window.location=o},l=i.setup().edgeOffset;l&&(a=Math.max(0,a-l),r&&(s=function(){history.pushState({},"",o)})),i.toY(a,null,s)}}},!1)}return i}); -------------------------------------------------------------------------------- /docs/_book/site_libs/quarto-html/quarto-syntax-highlighting.css: -------------------------------------------------------------------------------- 1 | /* quarto syntax highlight colors */ 2 | :root { 3 | --quarto-hl-al-color: #f51818; 4 | --quarto-hl-an-color: #e6ba7e; 5 | --quarto-hl-at-color: #399ee6; 6 | --quarto-hl-bn-color: #ff9940; 7 | --quarto-hl-bu-color: #4cbf99; 8 | --quarto-hl-ch-color: #4cbf99; 9 | --quarto-hl-co-color: #607880; 10 | --quarto-hl-cv-color: #a37acc; 11 | --quarto-hl-cn-color: #a37acc; 12 | --quarto-hl-cf-color: #fa8d3e; 13 | --quarto-hl-dt-color: #fa8d3e; 14 | --quarto-hl-dv-color: #ff9940; 15 | --quarto-hl-do-color: #607880; 16 | --quarto-hl-er-color: #f51818; 17 | --quarto-hl-ex-color: #399ee6; 18 | --quarto-hl-fl-color: #ff9940; 19 | --quarto-hl-fu-color: #f2ae49; 20 | --quarto-hl-im-color: #86b300; 21 | --quarto-hl-in-color: #ff9940; 22 | --quarto-hl-kw-color: #fa8d3e; 23 | --quarto-hl-op-color: #ed9366; 24 | --quarto-hl-pp-color: #f07171; 25 | --quarto-hl-re-color: #399ee6; 26 | --quarto-hl-sc-color: #4cbf99; 27 | --quarto-hl-ss-color: #4cbf99; 28 | --quarto-hl-st-color: #86b300; 29 | --quarto-hl-va-color: #55b4d4; 30 | --quarto-hl-vs-color: #86b300; 31 | --quarto-hl-wa-color: #f07171; 32 | } 33 | 34 | /* other quarto variables */ 35 | :root { 36 | --quarto-font-monospace: SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace; 37 | } 38 | 39 | code span.al { 40 | font-weight: bold; 41 | color: #f51818; 42 | } 43 | 44 | code span.an { 45 | color: #e6ba7e; 46 | } 47 | 48 | code span.at { 49 | color: #399ee6; 50 | } 51 | 52 | code span.bn { 53 | color: #ff9940; 54 | } 55 | 56 | code span.bu { 57 | color: #4cbf99; 58 | } 59 | 60 | code span.ch { 61 | color: #4cbf99; 62 | } 63 | 64 | code span.co { 65 | font-style: italic; 66 | color: #607880; 67 | } 68 | 69 | code span.cv { 70 | color: #a37acc; 71 | } 72 | 73 | code span.cn { 74 | color: #a37acc; 75 | } 76 | 77 | code span.cf { 78 | font-weight: bold; 79 | color: #fa8d3e; 80 | } 81 | 82 | code span.dt { 83 | color: #fa8d3e; 84 | } 85 | 86 | code span.dv { 87 | color: #ff9940; 88 | } 89 | 90 | code span.do { 91 | color: #607880; 92 | } 93 | 94 | code span.er { 95 | color: #f51818; 96 | text-decoration: underline; 97 | } 98 | 99 | code span.ex { 100 | font-weight: bold; 101 | color: #399ee6; 102 | } 103 | 104 | code span.fl { 105 | color: #ff9940; 106 | } 107 | 108 | code span.fu { 109 | color: #f2ae49; 110 | } 111 | 112 | code span.im { 113 | color: #86b300; 114 | } 115 | 116 | code span.in { 117 | color: #ff9940; 118 | } 119 | 120 | code span.kw { 121 | font-weight: bold; 122 | color: #fa8d3e; 123 | } 124 | 125 | pre > code.sourceCode > span { 126 | color: #575f66; 127 | } 128 | 129 | code span { 130 | color: #575f66; 131 | } 132 | 133 | code.sourceCode > span { 134 | color: #575f66; 135 | } 136 | 137 | div.sourceCode, 138 | div.sourceCode pre.sourceCode { 139 | color: #575f66; 140 | } 141 | 142 | code span.op { 143 | color: #ed9366; 144 | } 145 | 146 | code span.pp { 147 | color: #f07171; 148 | } 149 | 150 | code span.re { 151 | color: #399ee6; 152 | } 153 | 154 | code span.sc { 155 | color: #4cbf99; 156 | } 157 | 158 | code span.ss { 159 | color: #4cbf99; 160 | } 161 | 162 | code span.st { 163 | color: #86b300; 164 | } 165 | 166 | code span.va { 167 | color: #55b4d4; 168 | } 169 | 170 | code span.vs { 171 | color: #86b300; 172 | } 173 | 174 | code span.wa { 175 | color: #f07171; 176 | } 177 | 178 | .prevent-inlining { 179 | content: "s.tolerance[a.direction],e(a),l=t,i=!1}function h(){i||(i=!0,n=requestAnimationFrame(c))}var u=!!o&&{passive:!0,capture:!1};return t.addEventListener("scroll",h,u),c(),{destroy:function(){cancelAnimationFrame(n),t.removeEventListener("scroll",h,u)}}}function o(t){return t===Object(t)?t:{down:t,up:t}}function s(t,n){n=n||{},Object.assign(this,s.options,n),this.classes=Object.assign({},s.options.classes,n.classes),this.elem=t,this.tolerance=o(this.tolerance),this.offset=o(this.offset),this.initialised=!1,this.frozen=!1}return s.prototype={constructor:s,init:function(){return s.cutsTheMustard&&!this.initialised&&(this.addClass("initial"),this.initialised=!0,setTimeout(function(t){t.scrollTracker=n(t.scroller,{offset:t.offset,tolerance:t.tolerance},t.update.bind(t))},100,this)),this},destroy:function(){this.initialised=!1,Object.keys(this.classes).forEach(this.removeClass,this),this.scrollTracker.destroy()},unpin:function(){!this.hasClass("pinned")&&this.hasClass("unpinned")||(this.addClass("unpinned"),this.removeClass("pinned"),this.onUnpin&&this.onUnpin.call(this))},pin:function(){this.hasClass("unpinned")&&(this.addClass("pinned"),this.removeClass("unpinned"),this.onPin&&this.onPin.call(this))},freeze:function(){this.frozen=!0,this.addClass("frozen")},unfreeze:function(){this.frozen=!1,this.removeClass("frozen")},top:function(){this.hasClass("top")||(this.addClass("top"),this.removeClass("notTop"),this.onTop&&this.onTop.call(this))},notTop:function(){this.hasClass("notTop")||(this.addClass("notTop"),this.removeClass("top"),this.onNotTop&&this.onNotTop.call(this))},bottom:function(){this.hasClass("bottom")||(this.addClass("bottom"),this.removeClass("notBottom"),this.onBottom&&this.onBottom.call(this))},notBottom:function(){this.hasClass("notBottom")||(this.addClass("notBottom"),this.removeClass("bottom"),this.onNotBottom&&this.onNotBottom.call(this))},shouldUnpin:function(t){return"down"===t.direction&&!t.top&&t.toleranceExceeded},shouldPin:function(t){return"up"===t.direction&&t.toleranceExceeded||t.top},addClass:function(t){this.elem.classList.add.apply(this.elem.classList,this.classes[t].split(" "))},removeClass:function(t){this.elem.classList.remove.apply(this.elem.classList,this.classes[t].split(" "))},hasClass:function(t){return this.classes[t].split(" ").every(function(t){return this.classList.contains(t)},this.elem)},update:function(t){t.isOutOfBounds||!0!==this.frozen&&(t.top?this.top():this.notTop(),t.bottom?this.bottom():this.notBottom(),this.shouldUnpin(t)?this.unpin():this.shouldPin(t)&&this.pin())}},s.options={tolerance:{up:0,down:0},offset:0,scroller:t()?window:null,classes:{frozen:"headroom--frozen",pinned:"headroom--pinned",unpinned:"headroom--unpinned",top:"headroom--top",notTop:"headroom--not-top",bottom:"headroom--bottom",notBottom:"headroom--not-bottom",initial:"headroom"}},s.cutsTheMustard=!!(t()&&function(){}.bind&&"classList"in document.documentElement&&Object.assign&&Object.keys&&requestAnimationFrame),s}); 8 | -------------------------------------------------------------------------------- /docs/_book/site_libs/quarto-html/anchor.min.js: -------------------------------------------------------------------------------- 1 | // @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&dn=expat.txt Expat 2 | // 3 | // AnchorJS - v4.3.1 - 2021-04-17 4 | // https://www.bryanbraun.com/anchorjs/ 5 | // Copyright (c) 2021 Bryan Braun; Licensed MIT 6 | // 7 | // @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&dn=expat.txt Expat 8 | !function(A,e){"use strict";"function"==typeof define&&define.amd?define([],e):"object"==typeof module&&module.exports?module.exports=e():(A.AnchorJS=e(),A.anchors=new A.AnchorJS)}(this,function(){"use strict";return function(A){function d(A){A.icon=Object.prototype.hasOwnProperty.call(A,"icon")?A.icon:"",A.visible=Object.prototype.hasOwnProperty.call(A,"visible")?A.visible:"hover",A.placement=Object.prototype.hasOwnProperty.call(A,"placement")?A.placement:"right",A.ariaLabel=Object.prototype.hasOwnProperty.call(A,"ariaLabel")?A.ariaLabel:"Anchor",A.class=Object.prototype.hasOwnProperty.call(A,"class")?A.class:"",A.base=Object.prototype.hasOwnProperty.call(A,"base")?A.base:"",A.truncate=Object.prototype.hasOwnProperty.call(A,"truncate")?Math.floor(A.truncate):64,A.titleText=Object.prototype.hasOwnProperty.call(A,"titleText")?A.titleText:""}function w(A){var e;if("string"==typeof A||A instanceof String)e=[].slice.call(document.querySelectorAll(A));else{if(!(Array.isArray(A)||A instanceof NodeList))throw new TypeError("The selector provided to AnchorJS was invalid.");e=[].slice.call(A)}return e}this.options=A||{},this.elements=[],d(this.options),this.isTouchDevice=function(){return Boolean("ontouchstart"in window||window.TouchEvent||window.DocumentTouch&&document instanceof DocumentTouch)},this.add=function(A){var e,t,o,i,n,s,a,c,r,l,h,u,p=[];if(d(this.options),"touch"===(l=this.options.visible)&&(l=this.isTouchDevice()?"always":"hover"),0===(e=w(A=A||"h2, h3, h4, h5, h6")).length)return this;for(null===document.head.querySelector("style.anchorjs")&&((u=document.createElement("style")).className="anchorjs",u.appendChild(document.createTextNode("")),void 0===(A=document.head.querySelector('[rel="stylesheet"],style'))?document.head.appendChild(u):document.head.insertBefore(u,A),u.sheet.insertRule(".anchorjs-link{opacity:0;text-decoration:none;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}",u.sheet.cssRules.length),u.sheet.insertRule(":hover>.anchorjs-link,.anchorjs-link:focus{opacity:1}",u.sheet.cssRules.length),u.sheet.insertRule("[data-anchorjs-icon]::after{content:attr(data-anchorjs-icon)}",u.sheet.cssRules.length),u.sheet.insertRule('@font-face{font-family:anchorjs-icons;src:url(data:n/a;base64,AAEAAAALAIAAAwAwT1MvMg8yG2cAAAE4AAAAYGNtYXDp3gC3AAABpAAAAExnYXNwAAAAEAAAA9wAAAAIZ2x5ZlQCcfwAAAH4AAABCGhlYWQHFvHyAAAAvAAAADZoaGVhBnACFwAAAPQAAAAkaG10eASAADEAAAGYAAAADGxvY2EACACEAAAB8AAAAAhtYXhwAAYAVwAAARgAAAAgbmFtZQGOH9cAAAMAAAAAunBvc3QAAwAAAAADvAAAACAAAQAAAAEAAHzE2p9fDzz1AAkEAAAAAADRecUWAAAAANQA6R8AAAAAAoACwAAAAAgAAgAAAAAAAAABAAADwP/AAAACgAAA/9MCrQABAAAAAAAAAAAAAAAAAAAAAwABAAAAAwBVAAIAAAAAAAIAAAAAAAAAAAAAAAAAAAAAAAMCQAGQAAUAAAKZAswAAACPApkCzAAAAesAMwEJAAAAAAAAAAAAAAAAAAAAARAAAAAAAAAAAAAAAAAAAAAAQAAg//0DwP/AAEADwABAAAAAAQAAAAAAAAAAAAAAIAAAAAAAAAIAAAACgAAxAAAAAwAAAAMAAAAcAAEAAwAAABwAAwABAAAAHAAEADAAAAAIAAgAAgAAACDpy//9//8AAAAg6cv//f///+EWNwADAAEAAAAAAAAAAAAAAAAACACEAAEAAAAAAAAAAAAAAAAxAAACAAQARAKAAsAAKwBUAAABIiYnJjQ3NzY2MzIWFxYUBwcGIicmNDc3NjQnJiYjIgYHBwYUFxYUBwYGIwciJicmNDc3NjIXFhQHBwYUFxYWMzI2Nzc2NCcmNDc2MhcWFAcHBgYjARQGDAUtLXoWOR8fORYtLTgKGwoKCjgaGg0gEhIgDXoaGgkJBQwHdR85Fi0tOAobCgoKOBoaDSASEiANehoaCQkKGwotLXoWOR8BMwUFLYEuehYXFxYugC44CQkKGwo4GkoaDQ0NDXoaShoKGwoFBe8XFi6ALjgJCQobCjgaShoNDQ0NehpKGgobCgoKLYEuehYXAAAADACWAAEAAAAAAAEACAAAAAEAAAAAAAIAAwAIAAEAAAAAAAMACAAAAAEAAAAAAAQACAAAAAEAAAAAAAUAAQALAAEAAAAAAAYACAAAAAMAAQQJAAEAEAAMAAMAAQQJAAIABgAcAAMAAQQJAAMAEAAMAAMAAQQJAAQAEAAMAAMAAQQJAAUAAgAiAAMAAQQJAAYAEAAMYW5jaG9yanM0MDBAAGEAbgBjAGgAbwByAGoAcwA0ADAAMABAAAAAAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAH//wAP) format("truetype")}',u.sheet.cssRules.length)),u=document.querySelectorAll("[id]"),t=[].map.call(u,function(A){return A.id}),i=0;i\]./()*\\\n\t\b\v\u00A0]/g,"-").replace(/-{2,}/g,"-").substring(0,this.options.truncate).replace(/^-+|-+$/gm,"").toLowerCase()},this.hasAnchorJSLink=function(A){var e=A.firstChild&&-1<(" "+A.firstChild.className+" ").indexOf(" anchorjs-link "),A=A.lastChild&&-1<(" "+A.lastChild.className+" ").indexOf(" anchorjs-link ");return e||A||!1}}}); 9 | // @license-end -------------------------------------------------------------------------------- /docs/_book/site_libs/quarto-nav/quarto-nav.js: -------------------------------------------------------------------------------- 1 | const headroomChanged = new CustomEvent("quarto-hrChanged", { 2 | detail: {}, 3 | bubbles: true, 4 | cancelable: false, 5 | composed: false, 6 | }); 7 | 8 | window.document.addEventListener("DOMContentLoaded", function () { 9 | let init = false; 10 | 11 | function throttle(func, wait) { 12 | var timeout; 13 | return function () { 14 | const context = this; 15 | const args = arguments; 16 | const later = function () { 17 | clearTimeout(timeout); 18 | timeout = null; 19 | func.apply(context, args); 20 | }; 21 | 22 | if (!timeout) { 23 | timeout = setTimeout(later, wait); 24 | } 25 | }; 26 | } 27 | 28 | function headerOffset() { 29 | // Set an offset if there is are fixed top navbar 30 | const headerEl = window.document.querySelector("header.fixed-top"); 31 | if (headerEl) { 32 | return headerEl.clientHeight; 33 | } else { 34 | return 0; 35 | } 36 | } 37 | 38 | function footerOffset() { 39 | const footerEl = window.document.querySelector("footer.footer"); 40 | if (footerEl) { 41 | return footerEl.clientHeight; 42 | } else { 43 | return 0; 44 | } 45 | } 46 | 47 | function updateDocumentOffsetWithoutAnimation() { 48 | updateDocumentOffset(false); 49 | } 50 | 51 | function updateDocumentOffset(animated) { 52 | // set body offset 53 | const topOffset = headerOffset(); 54 | const bodyOffset = topOffset + footerOffset(); 55 | const bodyEl = window.document.body; 56 | bodyEl.setAttribute("data-bs-offset", topOffset); 57 | bodyEl.style.paddingTop = topOffset + "px"; 58 | 59 | // deal with sidebar offsets 60 | const sidebars = window.document.querySelectorAll( 61 | ".sidebar, .headroom-target" 62 | ); 63 | sidebars.forEach((sidebar) => { 64 | if (!animated) { 65 | sidebar.classList.add("notransition"); 66 | // Remove the no transition class after the animation has time to complete 67 | setTimeout(function () { 68 | sidebar.classList.remove("notransition"); 69 | }, 201); 70 | } 71 | 72 | if (window.Headroom && sidebar.classList.contains("sidebar-unpinned")) { 73 | sidebar.style.top = "0"; 74 | sidebar.style.maxHeight = "100vh"; 75 | } else { 76 | sidebar.style.top = topOffset + "px"; 77 | sidebar.style.maxHeight = "calc(100vh - " + topOffset + "px)"; 78 | } 79 | }); 80 | 81 | // allow space for footer 82 | const mainContainer = window.document.querySelector(".quarto-container"); 83 | if (mainContainer) { 84 | mainContainer.style.minHeight = "calc(100vh - " + bodyOffset + "px)"; 85 | } 86 | 87 | // link offset 88 | let linkStyle = window.document.querySelector("#quarto-target-style"); 89 | if (!linkStyle) { 90 | linkStyle = window.document.createElement("style"); 91 | linkStyle.setAttribute("id", "quarto-target-style"); 92 | window.document.head.appendChild(linkStyle); 93 | } 94 | while (linkStyle.firstChild) { 95 | linkStyle.removeChild(linkStyle.firstChild); 96 | } 97 | if (topOffset > 0) { 98 | linkStyle.appendChild( 99 | window.document.createTextNode(` 100 | section:target::before { 101 | content: ""; 102 | display: block; 103 | height: ${topOffset}px; 104 | margin: -${topOffset}px 0 0; 105 | }`) 106 | ); 107 | } 108 | if (init) { 109 | window.dispatchEvent(headroomChanged); 110 | } 111 | init = true; 112 | } 113 | 114 | // initialize headroom 115 | var header = window.document.querySelector("#quarto-header"); 116 | if (header && window.Headroom) { 117 | const headroom = new window.Headroom(header, { 118 | tolerance: 5, 119 | onPin: function () { 120 | const sidebars = window.document.querySelectorAll( 121 | ".sidebar, .headroom-target" 122 | ); 123 | sidebars.forEach((sidebar) => { 124 | sidebar.classList.remove("sidebar-unpinned"); 125 | }); 126 | updateDocumentOffset(); 127 | }, 128 | onUnpin: function () { 129 | const sidebars = window.document.querySelectorAll( 130 | ".sidebar, .headroom-target" 131 | ); 132 | sidebars.forEach((sidebar) => { 133 | sidebar.classList.add("sidebar-unpinned"); 134 | }); 135 | updateDocumentOffset(); 136 | }, 137 | }); 138 | headroom.init(); 139 | 140 | let frozen = false; 141 | window.quartoToggleHeadroom = function () { 142 | if (frozen) { 143 | headroom.unfreeze(); 144 | frozen = false; 145 | } else { 146 | headroom.freeze(); 147 | frozen = true; 148 | } 149 | }; 150 | } 151 | 152 | // Observe size changed for the header 153 | const headerEl = window.document.querySelector("header.fixed-top"); 154 | if (headerEl && window.ResizeObserver) { 155 | const observer = new window.ResizeObserver( 156 | updateDocumentOffsetWithoutAnimation 157 | ); 158 | observer.observe(headerEl, { 159 | attributes: true, 160 | childList: true, 161 | characterData: true, 162 | }); 163 | } else { 164 | window.addEventListener( 165 | "resize", 166 | throttle(updateDocumentOffsetWithoutAnimation, 50) 167 | ); 168 | } 169 | setTimeout(updateDocumentOffsetWithoutAnimation, 250); 170 | 171 | // fixup index.html links if we aren't on the filesystem 172 | if (window.location.protocol !== "file:") { 173 | const links = window.document.querySelectorAll("a"); 174 | for (let i = 0; i < links.length; i++) { 175 | links[i].href = links[i].href.replace(/\/index\.html/, "/"); 176 | } 177 | 178 | // Fixup any sharing links that require urls 179 | // Append url to any sharing urls 180 | const sharingLinks = window.document.querySelectorAll( 181 | "a.sidebar-tools-main-item" 182 | ); 183 | for (let i = 0; i < sharingLinks.length; i++) { 184 | const sharingLink = sharingLinks[i]; 185 | const href = sharingLink.getAttribute("href"); 186 | if (href) { 187 | sharingLink.setAttribute( 188 | "href", 189 | href.replace("|url|", window.location.href) 190 | ); 191 | } 192 | } 193 | 194 | // Scroll the active navigation item into view, if necessary 195 | const navSidebar = window.document.querySelector("nav#quarto-sidebar"); 196 | if (navSidebar) { 197 | // Find the active item 198 | const activeItem = navSidebar.querySelector("li.sidebar-item a.active"); 199 | if (activeItem) { 200 | // Wait for the scroll height and height to resolve by observing size changes on the 201 | // nav element that is scrollable 202 | const resizeObserver = new ResizeObserver((_entries) => { 203 | // The bottom of the element 204 | const elBottom = activeItem.offsetTop; 205 | const viewBottom = navSidebar.scrollTop + navSidebar.clientHeight; 206 | 207 | // The element height and scroll height are the same, then we are still loading 208 | if (viewBottom !== navSidebar.scrollHeight) { 209 | // Determine if the item isn't visible and scroll to it 210 | if (elBottom >= viewBottom) { 211 | navSidebar.scrollTop = elBottom; 212 | } 213 | 214 | // stop observing now since we've completed the scroll 215 | resizeObserver.unobserve(navSidebar); 216 | } 217 | }); 218 | resizeObserver.observe(navSidebar); 219 | } 220 | } 221 | } 222 | }); 223 | -------------------------------------------------------------------------------- /docs/_book/site_libs/clipboard/clipboard.min.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * clipboard.js v2.0.10 3 | * https://clipboardjs.com/ 4 | * 5 | * Licensed MIT © Zeno Rocha 6 | */ 7 | !function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return n={686:function(t,e,n){"use strict";n.d(e,{default:function(){return o}});var e=n(279),i=n.n(e),e=n(370),u=n.n(e),e=n(817),c=n.n(e);function a(t){try{return document.execCommand(t)}catch(t){return}}var f=function(t){t=c()(t);return a("cut"),t};var l=function(t){var e,n,o,r=1 98 | tokens = id.tokenize("_") 99 | } 100 | | view 101 | } 102 | ``` 103 | 104 | If we are confident about the stability of the naming scheme, we can destructure the list returned by `tokenize` and assign them to variables directly: 105 | 106 | ```{groovy} 107 | map { id, reads -> 108 | (sample, replicate, type) = id.tokenize("_") 109 | meta = [sample:sample, replicate:replicate, type:type] 110 | [meta, reads] 111 | } 112 | ``` 113 | 114 | ::: {.callout-note} 115 | ## Destructuring requires parentheses 116 | Make sure that you're using a tuple with parentheses e.g. `(one, two)` rather than a List e.g. `[one, two]` 117 | ::: 118 | 119 | Another option is to use the [`transpose`](https://docs.groovy-lang.org/latest/html/api/groovy/util/GroovyCollections.html#transpose(java.util.List)) method with the [`collectEntries()`](https://docs.groovy-lang.org/latest/html/api/org/codehaus/groovy/runtime/DefaultGroovyMethods.html#collectEntries(E[])) to produce the same map. I'd warn that this method is bordering on a little 'too clever' and is more difficult to read. It also assumes that the order of the filename-encoded metadata is consistent. 120 | 121 | ```{groovy} 122 | map { id, reads -> 123 | meta = [id.tokenize("_"), ['sample', 'replicate', 'type']] 124 | .transpose() 125 | .collectEntries() 126 | [meta, reads] 127 | } 128 | ``` 129 | 130 | If we move back to the previous method, but decided that the 'rep' prefix on the replicate should be removed, we can use regular expressions to simply "subtract" pieces of a string. Here we remove a 'rep' prefix from the `replicate` variable if the prefix is present: 131 | 132 | ```{groovy} 133 | map { id, reads -> 134 | (sample, replicate, type) = id.tokenize("_") 135 | replicate -= ~/^rep/ 136 | meta = [sample:sample, replicate:replicate, type:type] 137 | [meta, reads] 138 | } 139 | ``` 140 | 141 | ::: {.callout-tip} 142 | ## Trim strings 143 | Groovy has a lot of very helpful syntactic sugar for string manipulation. You can trim parts of a string by simply subtracting another string: 144 | 145 | ```{groovy} 146 | demo = "one two three" 147 | assertEquals(demo - "two ", "one three") 148 | ``` 149 | 150 | ... or by subtracting a regular expression: 151 | 152 | ```{groovy} 153 | demo = "one two three" 154 | assertEquals(demo - ~/t.o ?/, "one three") 155 | ``` 156 | 157 | To quickly sanity-check a groovy expression, try the [Groovy web console](https://groovyconsole.appspot.com/) 158 | ::: 159 | 160 | We are almost there, but the we still don't have the "treatment" metadata captured in our meta map. The treament is encoded in this example in the name of the parent directory relative to the reads. Inside the map object, the reads are a list of two UnixPath objects. These objects implement the [`java.nio.Path`](https://docs.oracle.com/javase/7/docs/api/java/nio/file/Path.html) interface, which provides us many useful methods, including `getParent()`. 161 | 162 | We can call the `getParent()` method on each of the paths like so: 163 | 164 | ```{groovy} 165 | map { id, reads -> 166 | reads.collect { it.getParent() } 167 | } 168 | ``` 169 | 170 | If we want to call a set method on every item in a Collection, Groovy provides this convenient "spread dot" notation: 171 | 172 | ```{groovy} 173 | map { id, reads -> 174 | reads*.getParent() 175 | } 176 | ``` 177 | 178 | this returns another Path object, but we only want the name of the last directory, so we need to call `.getName()` method on each of these Paths. We can use the spread-dot notation again: 179 | 180 | ```{groovy} 181 | map { id, reads -> 182 | reads*.getParent()*.getName() 183 | } 184 | ``` 185 | 186 | The last piece of Groovy sugar is to note that methods with `get` and `set` prefixes can be called with a property-style notation, converting `getParent()` to `parent` and `getName()` to `name`: 187 | 188 | ```{groovy} 189 | map { id, reads -> 190 | reads*.parent*.name 191 | } 192 | ``` 193 | 194 | If we wanted to remove the "treatment" prefix, we can combine this new notation with the "minus" method which we used earlier in the aliased `-` form. 195 | 196 | ```{groovy} 197 | map { id, reads -> 198 | reads*.parent*.name*.minus(~/treatment/) 199 | } 200 | ``` 201 | 202 | In this particular example, we know ahead of time that the treatments must be the same because of the way the `fromFilePairs` method gathers pairs, but we'll continue for the sake of the demonstration. Our final `map` closure might look like: 203 | 204 | ```{groovy} 205 | workflow { 206 | Channel.fromFilePairs("data/reads/*/*_R{1,2}.fastq.gz") 207 | | map { id, reads -> 208 | (sample, replicate, type) = id.tokenize("_") 209 | (treatmentFwd, treatmentRev) = reads*.parent*.name*.minus(~/treatment/) 210 | meta = [ 211 | sample:sample, 212 | replicate:replicate, 213 | type:type, 214 | treatmentFwd:treatmentFwd, 215 | treatmentRev:treatmentRev, 216 | ] 217 | [meta, reads] 218 | } 219 | | view 220 | } 221 | ``` 222 | 223 | 224 | This metadata map can be passed through the workflow with the reads and used to split, join and recombine the data. The resulting channel would be suitable for any Nextflow process with inputs of the form 225 | 226 | ```{groovy} 227 | process ExampleProcess { 228 | input: 229 | tuple val(meta), path(reads) 230 | 231 | // ... 232 | ``` 233 | 234 | This channel "shape" or cardinarlity is extremely common in nf-core modules and subworkflows and is critical to enabling reusability of these modules. 235 | 236 | -------------------------------------------------------------------------------- /docs/structure.qmd: -------------------------------------------------------------------------------- 1 | --- 2 | engine: knitr 3 | --- 4 | 5 | # Workflow Structure 6 | 7 | There are three directories in a Nextflow workflow repository that have a special purpose: 8 | 9 | ## `./bin` 10 | The `bin` directory (if it exists) is always added to the `$PATH` for all tasks. If the tasks are performed on a remote machine, the directory is copied across to the new machine before the task begins. It is important to know that Nextflow will take care of updating `$PATH` and ensuring the files are available wherever the task is running, but will not change the permissions of any files in that directory. If a file is called by a task as an executable, the workflow developer must ensure that the file has the correct permissions to be executed. 11 | 12 | 13 | 14 | ## `./templates` 15 | If a process script block is becoming too long, it can be moved to a template file. The template file can then be imported into the process script block using the `template` method. This is useful for keeping the process block tidy and readable. Nextflow's use of `$` to indicate variables also allows for directly testing the template file by running it as a script. 16 | 17 | The chapter_05_structure directory already contains an example template - a very simple python script. We can add a new process that uses this template: 18 | 19 | ```groovy 20 | process SayHiTemplate { 21 | debug true 22 | input: val(name) 23 | script: template 'adder.py' 24 | } 25 | ``` 26 | 27 | ## `./lib` 28 | In the previous chapter, we saw the addition of small helper Groovy functions to the `main.nf` file. It may at times be helpful to bundle functionality into a new Groovy class. Any classes defined in the `lib` directory are available for use in the workflow - both `main.nf` and any imported modules. 29 | 30 | Classes defined in `lib` can be used for a variety of purposes. For example, the [nf-core/rnaseq](https://github.com/nf-core/rnaseq/tree/master/lib) workflow uses five custom classes: 31 | 1) `NfcoreSchema.groovy` for parsing the schema.json file and validating the workflow parameters. 32 | 2) `NfcoreTemplate.groovy` for email templating and nf-core utility functions. 33 | 3) `Utils.groovy` for provision of a single `checkCondaChannels` method. 34 | 4) `WorkflowMain.groovy` for workflow setup and to call the `NfcoreTemplate` class. 35 | 5) `WorkflowRnaseq.groovy` for the workflow-specific functions. 36 | 37 | The classes listed above all provide utility executed at the beginning of a workflow, and are generally used to "set up" the workflow. However, classes defined in `lib` can also be used to provide functionality to the workflow itself. 38 | 39 | ### Making a Metadata Class 40 | 41 | Let's consider an example where we create our own custom class to handle metadata. We can create a new class in `./lib/Metadata.groovy`. We'll extend the built-in `HashMap` class, and add a simple method to return a value: 42 | 43 | ```groovy 44 | class Metadata extends HashMap { 45 | def hi() { 46 | return "Hello, workshop participants!" 47 | } 48 | } 49 | ``` 50 | 51 | We can then use this class in our workflow: 52 | 53 | ```groovy 54 | workflow { 55 | Channel.of("Montreal") 56 | | map { new Metadata() } 57 | | view 58 | } 59 | ``` 60 | 61 | We can use the new `hi` method in the workflow: 62 | 63 | ```groovy 64 | workflow { 65 | Channel.of("Montreal") 66 | | map { new Metadata() } 67 | | view { it.hi() } 68 | } 69 | ``` 70 | 71 | At the moment, the `Metadata` class is not making use of the "Montreal" being passed into the closure. Let's change that by adding a constructor to the class: 72 | 73 | ```groovy 74 | class Metadata extends HashMap { 75 | Metadata(String location) { 76 | this.location = location 77 | } 78 | 79 | def hi() { 80 | return this.location ? "Hello, from ${this.location}!" : "Hello, workshop participants!" 81 | } 82 | } 83 | ``` 84 | 85 | Which we can use like so: 86 | 87 | ```groovy 88 | workflow { 89 | Channel.of("Montreal") 90 | | map { place -> new Metadata(place) } 91 | | view { it.hi() } 92 | } 93 | ``` 94 | 95 | We can also use this method when passing the object to a process: 96 | 97 | ```groovy 98 | process UseMeta { 99 | input: val(meta) 100 | output: path("out.txt") 101 | script: "echo '${meta.hi()}' | tee out.txt" 102 | } 103 | 104 | workflow { 105 | Channel.of("Montreal") 106 | | map { place -> new Metadata(place) } 107 | | UseMeta 108 | | view 109 | } 110 | ``` 111 | 112 | Why might this be helpful? You can add extra classes to the metadata which can be computed from the existing metadata. For exmaple, we might want want to grab the adapter prefix: 113 | 114 | ```groovy 115 | def getAdapterStart() { 116 | this.adapter?.substring(0, 3) 117 | } 118 | ``` 119 | 120 | Which we might use like so: 121 | ```groovy 122 | process UseMeta { 123 | input: val(meta) 124 | output: path("out.txt") 125 | script: "echo '${meta.adapter} prefix is ${meta.getAdapterStart()}' | tee out.txt" 126 | } 127 | 128 | workflow { 129 | Channel.of("Montreal") 130 | | map { place -> new Metadata(place) } 131 | | map { it + [adapter:"AACGTAGCTTGAC"] } 132 | | UseMeta 133 | | view 134 | } 135 | ``` 136 | 137 | You might even want to reach out to external services such as a LIMS or e-utilis API. Here we add a dummy "getSampleName()" method that reaches out to a public API: 138 | 139 | ```groovy 140 | def getSampleName() { 141 | def get = new URL('https://postman-echo.com/get?sampleName=Fido').openConnection() 142 | def getRC = get.getResponseCode(); 143 | if (getRC.equals(200)) { 144 | JsonSlurper jsonSlurper = new JsonSlurper() 145 | def json = jsonSlurper.parseText(get.getInputStream().getText()) 146 | return json.args.sampleName 147 | } 148 | } 149 | ``` 150 | 151 | Which we can use like so: 152 | 153 | ```groovy 154 | process UseMeta { 155 | input: val(meta) 156 | output: path("out.txt") 157 | script: 158 | "echo '${meta.adapter} prefix is ${meta.getAdapterStart()} with sampleName ${meta.getSampleName()}' | tee out.txt" 159 | } 160 | ``` 161 | 162 | ::: {.callout-note} 163 | ## Nextflow Caching 164 | When we start passing custom classes through the workflow, it's important to understand a little about the Nextflow caching mechanism. When a task is run, a unique hash is calculated based on the task name, the input files/values, and the input parameters. Our class extends from `HashMap`, which means that the hash will be calculated based on the contents of the `HashMap`. If we add a new method to the class, or ammend a class method, this does not change the value of the objects in the hash, which means that the hash will not change. 165 | ::: 166 | 167 | ### Exercise 168 | 169 | Can you show changing a method in our `Metadata` class does not change the hash? 170 | 171 |
172 | 173 | Show answer 174 | 175 | 176 | We might increase the length of the adapter prefix to 5 characters: 177 | 178 | ```{groovy} 179 | def getAdapterStart() { 180 | this.adapter?.substring(0, 5) 181 | } 182 | ``` 183 | 184 | Changing this method and resuming the workflow will not change the hash, and the existing method will be used. 185 |
186 | 187 | 188 | We are not limited to using or extending the built-in Groovy classes. Let's start by creating a `Dog` class in `./lib/Dog.groovy`: 189 | 190 | ```groovy 191 | class Dog { 192 | String name 193 | Boolean isHungry = true 194 | } 195 | ``` 196 | 197 | We can create a new dog at the beginning of the workflow: 198 | 199 | ```groovy 200 | workflow { 201 | dog = new Dog(name: "fido") 202 | log.info "Found a new dog: $dog" 203 | } 204 | ``` 205 | 206 | We can pass objects of our class through channels. Here we take a channel of dog names and create a channel of dogs: 207 | 208 | ```groovy 209 | workflow { 210 | Channel.of("Argente", "Absolon", "Chowne") 211 | | map { new Dog(name: it) } 212 | | view 213 | } 214 | ``` 215 | 216 | If we try to use this new class in a resumed process, no caches will be used. 217 | 218 | ### Exercise 219 | 220 | Show that the `Dog` class is not cached when resuming a workflow. 221 | 222 | ### Making a ValueObject 223 | 224 | 225 | Nextflow has provided a decorator to help serialize your custom classes. By adding `@ValueObject` to the class definition, Nextflow will automatically serialize the class and cache it. This is useful if you want to pass a custom class through a channel, or if you want to use the class in a resumed workflow. 226 | 227 | Let's add the decorator to our `Dog` class: 228 | 229 | ```groovy 230 | import nextflow.io.ValueObject 231 | 232 | @ValueObject 233 | class Dog { 234 | String name 235 | Boolean isHungry = true 236 | } 237 | ``` 238 | 239 | Lastly, we will need to register the class with Kyro, the Java serialization framework. Again, Nextflow provides a helper method to do this. We can add the following to the `main.nf` file: 240 | 241 | ```groovy 242 | import nextflow.util.KryoHelper 243 | 244 | KryoHelper.register(Dog) 245 | ``` 246 | 247 | ### Exercise 248 | 249 | Show that the `Dog` class can now be used in processes and cached correctly. 250 | 251 | 252 | 253 | 254 | 288 | -------------------------------------------------------------------------------- /docs/grouping.qmd: -------------------------------------------------------------------------------- 1 | --- 2 | engine: knitr 3 | --- 4 | 5 | # Grouping and Splitting 6 | 7 | ## Grouping using subMap 8 | 9 | Now that we have a channel that conforms to the `tuple val(meta) path()` pattern, we can investigate splitting and grouping patterns. 10 | 11 | We'll start with a simple main.nf in the `chapter_03_grouping` directory 12 | 13 | ```{bash} 14 | cd chapter_03_grouping 15 | ``` 16 | 17 | ```{groovy filename="main.nf"} 18 | workflow { 19 | Channel.fromPath("data/samplesheet.csv") 20 | | splitCsv( header:true ) 21 | | map { row -> 22 | meta = [id:row.id, repeat:row.repeat, type:row.type] 23 | [meta, [ 24 | file(row.fastq1, checkIfExists: true), 25 | file(row.fastq2, checkIfExists: true)]] 26 | } 27 | | view 28 | } 29 | ``` 30 | 31 | The first change we're going to make is to correct some repetitive code that we've seen quite a lot already in this workshop. The construction of the meta map from this row stutters quite a lot. We can make use of the [`subMap`](https://docs.groovy-lang.org/latest/html/groovy-jdk/java/util/Map.html#subMap(java.util.Collection)) method available Maps to quickly return a new map constructed from the subset of an existing map: 32 | 33 | ```{groovy} 34 | workflow { 35 | Channel.fromPath("data/samplesheet.csv") 36 | | splitCsv( header:true ) 37 | | map { row -> 38 | meta = row.subMap('id', 'repeat', 'type') 39 | [meta, [ 40 | file(row.fastq1, checkIfExists: true), 41 | file(row.fastq2, checkIfExists: true)]] 42 | } 43 | | view 44 | } 45 | ``` 46 | 47 | ::: {.callout-note} 48 | ## Complete meta map safety 49 | The `subMap` method will take a collection of keys and construct a _new_ map with just the keys listed in the collection. This method, in combination with the `plus` or `+` method for combining maps and resetting values should allow all contraction, expansion and modification of maps safely. 50 | ::: 51 | 52 | ### Exercise 53 | 54 | Can you extend our workflow in an _unsafe_ manner? Use the `set` operator to name the channel in our workflow above, and then `map` (the operator) over that without modification. In a separate map operation, try modifying the meta map in a way that is reflected in the first map. 55 | 56 | Note that we're trying to do the _wrong_ thing in this example to clarify what the correct approach might be. 57 | 58 |
59 | 60 | Show answer 61 | 62 | 63 | To ensure that the modification of the map happens first, we introduce a `sleep` into the first map operation. This `sleep` emulates a long-running Nextflow process. 64 | 65 | ```{groovy} 66 | workflow { 67 | Channel.fromPath("data/samplesheet.csv") 68 | | splitCsv( header:true ) 69 | | map { row -> 70 | meta = row.subMap('id', 'repeat', 'type') 71 | [meta, [ 72 | file(row.fastq1, checkIfExists: true), 73 | file(row.fastq2, checkIfExists: true)]] 74 | } 75 | | set { samples } 76 | 77 | 78 | samples 79 | | map { sleep 10; it } 80 | | view { meta, reads -> "Should be unmodified: $meta" } 81 | 82 | samples 83 | | map { meta, reads -> 84 | meta.type = meta.type == "tumor" ? "abnormal" : "normal" 85 | [meta, reads] 86 | } 87 | | view { meta, reads -> "Should be modified: $meta" } 88 | } 89 | ``` 90 |
91 | 92 | ### Exercise 93 | 94 | How would you fix the example above to use the safe operators `plus` and `subMap` to ensure that the original map remains unmodified? 95 | 96 |
97 | Show answer 98 | 99 | ```{groovy} 100 | workflow { 101 | Channel.fromPath("data/samplesheet.csv") 102 | | splitCsv( header:true ) 103 | | map { row -> 104 | meta = row.subMap('id', 'repeat', 'type') 105 | [meta, [ 106 | file(row.fastq1, checkIfExists: true), 107 | file(row.fastq2, checkIfExists: true)]] 108 | } 109 | | set { samples } 110 | 111 | 112 | samples 113 | | map { sleep 10; it } 114 | | view { meta, reads -> "Should be unmodified: $meta" } 115 | 116 | samples 117 | | map { meta, reads -> 118 | newmap = [type: meta.type == "tumor" ? "abnormal" : "normal"] 119 | [meta + newmap, reads] 120 | } 121 | | view { meta, reads -> "Should be modified: $meta" } 122 | } 123 | ``` 124 |
125 | 126 | ## Passing maps through processes 127 | 128 | Let's construct a dummy read mapping process. This is not a bioinformatics workshop, so we can 'cheat' in the interests of time. 129 | 130 | ```{groovy} 131 | process MapReads { 132 | input: 133 | tuple val(meta), path(reads) 134 | path(genome) 135 | 136 | output: 137 | tuple val(meta), path("*.bam") 138 | 139 | "touch out.bam" 140 | } 141 | 142 | workflow { 143 | reference = Channel.fromPath("data/genome.fasta").first() 144 | 145 | Channel.fromPath("data/samplesheet.csv") 146 | | splitCsv( header:true ) 147 | | map { row -> 148 | meta = row.subMap('id', 'repeat', 'type') 149 | [meta, [ 150 | file(row.fastq1, checkIfExists: true), 151 | file(row.fastq2, checkIfExists: true)]] 152 | } 153 | | set { samples } 154 | 155 | MapReads( samples, reference ) 156 | | view 157 | } 158 | ``` 159 | 160 | Let's consider that we might now want to merge the repeats. We'll need to group bams that share the `id` and `type` attributes. 161 | 162 | ```{groovy} 163 | MapReads( samples, reference ) 164 | | map { meta, bam -> [meta.subMap('id', 'type'), bam]} 165 | | groupTuple 166 | | view 167 | ``` 168 | 169 | This is easy enough, but the `groupTuple` operator has to wait until all items are emitted from the incoming queue before it is able to reassemble the output queue. If even one read mapping job takes a long time, the processing of all other samples is held up. We need a way of signalling to nextflow how many items are in a given group so that items can be emitted as early as possible. 170 | 171 | By default, the `groupTuple` operator groups on the first item in the element, which at the moment is a `Map`. We can turn this map into a special class using the `groupKey` method, which takes our grouping object as a first parameter and the number of expected elements in the second parameter. 172 | 173 | ```{groovy} 174 | MapReads( samples, reference ) 175 | | map { meta, bam -> 176 | key = groupKey(meta.subMap('id', 'type'), NUMBER_OF_ITEMS_IN_GROUP) 177 | [key, bam] 178 | } 179 | | groupTuple 180 | | view 181 | ``` 182 | 183 | ### Exercise 184 | 185 | How might we modify the upstream channels to the number of repeats into the metamap? 186 | 187 |
188 | Show answer 189 | 190 | ```{groovy} 191 | workflow { 192 | reference = Channel.fromPath("data/genome.fasta").first() 193 | 194 | Channel.fromPath("data/samplesheet.csv") 195 | | splitCsv( header:true ) 196 | | map { row -> 197 | meta = row.subMap('id', 'repeat', 'type') 198 | [meta, [file(row.fastq1, checkIfExists: true), file(row.fastq2, checkIfExists: true)]] 199 | } 200 | | map { meta, reads -> [meta.subMap('id', 'type'), meta.repeat, reads] } 201 | | groupTuple 202 | | map { meta, repeats, reads -> [meta + [repeatcount:repeats.size()], repeats, reads] } 203 | | transpose 204 | | map { meta, repeat, reads -> [meta + [repeat:repeat], reads]} 205 | | set { samples } 206 | 207 | MapReads( samples, reference ) 208 | | map { meta, bam -> 209 | key = groupKey(meta.subMap('id', 'type'), meta.repeatcount) 210 | [key, bam] 211 | } 212 | | groupTuple 213 | | view 214 | } 215 | ``` 216 |
217 | 218 | Now that we have our repeats together in an output channel, we can combine them using "advanced bioinformatics": 219 | 220 | ```{groovy} 221 | process CombineBams { 222 | input: 223 | tuple val(meta), path("input/in_*_.bam") 224 | 225 | output: 226 | tuple val(meta), path("combined.bam") 227 | 228 | "cat input/*.bam > combined.bam" 229 | } 230 | ``` 231 | 232 | In our workflow: 233 | 234 | ```{groovy} 235 | MapReads( samples, reference ) 236 | | map { meta, bam -> 237 | key = groupKey(meta.subMap('id', 'type'), meta.repeatcount) 238 | [key, bam] 239 | } 240 | | groupTuple 241 | | CombineBams 242 | ``` 243 | 244 | ## Fanning out over intervals 245 | 246 | The previous exercise demonstrated the fan-in approach using `groupTuple` and `groupKey`, but we might want to fan out our processes. An example might be computing over some intervals - genotyping over intervals, for example. 247 | 248 | We can take an existing bed file, for example and turn it into a channel of Maps. 249 | 250 | ```{groovy} 251 | Channel.fromPath("data/intervals.bed") 252 | | splitCsv(header: ['chr', 'start', 'stop', 'name'], sep: '\t') 253 | | collectFile { entry -> ["${entry.name}.bed", entry*.value.join("\t")] } 254 | | view 255 | | set { intervals } 256 | 257 | return 258 | ``` 259 | 260 | ::: {.callout-note} 261 | ## Quick return 262 | In the example above, I add a `return` statement for quick debugging. This ensures that all workflow operations after the `return` are not included in the process DAG. 263 | ::: 264 | 265 | Given a dummy genotyping process 266 | 267 | ```{groovy} 268 | process GenotypeOnInterval { 269 | input: 270 | tuple val(meta), path(bam), path(bed) 271 | 272 | output: 273 | tuple val(meta), path("genotyped.bam") 274 | 275 | "cat $bam $bed > genotyped.bam" 276 | } 277 | ``` 278 | 279 | We can use the `combine` operator to emit a new channel where each combined bam is attached to each bed file. These can then be piped into the genotyping process: 280 | 281 | ```{groovy} 282 | MapReads( samples, reference ) 283 | | map { meta, bam -> 284 | key = groupKey(meta.subMap('id', 'type'), meta.repeatcount) 285 | [key, bam] 286 | } 287 | | groupTuple 288 | | CombineBams 289 | | combine( intervals ) 290 | | GenotypeOnInterval 291 | | view 292 | ``` 293 | 294 | 295 | Finally, we can combine these genotyped bams back using `groupTuple` and another bam merge process: 296 | 297 | ```{groovy} 298 | process MergeGenotyped { 299 | input: 300 | tuple val(meta), path("input/in_*_.bam") 301 | 302 | output: 303 | tuple val(meta), path("merged.genotyped.bam") 304 | 305 | "cat input/*.bam > merged.genotyped.bam" 306 | } 307 | ``` 308 | 309 | ```{groovy} 310 | MapReads( samples, reference ) 311 | | map { meta, bam -> 312 | key = groupKey(meta.subMap('id', 'type'), meta.repeatcount) 313 | [key, bam] 314 | } 315 | | groupTuple 316 | | CombineBams 317 | | map { meta, bam -> [meta.subMap('id', 'type'), bam] } 318 | | combine( intervals ) 319 | | GenotypeOnInterval 320 | | groupTuple 321 | | MergeGenotyped 322 | | view 323 | ``` 324 | 325 | 326 | -------------------------------------------------------------------------------- /docs/groovy.qmd: -------------------------------------------------------------------------------- 1 | --- 2 | engine: knitr 3 | --- 4 | 5 | # Groovy Imports 6 | 7 | ```{r setup, eval=TRUE, include=FALSE} 8 | # knitr::knit_engines$set(groovy = function(options) { 9 | # code <- paste(options$code, collapse = '\n') 10 | # if(options$eval) { 11 | # workDir <- tempdir() 12 | # file.copy("../chapter_01_operators/data", workDir, recursive=TRUE) 13 | # scriptPath <- file.path(workDir, 'main.nf') 14 | # scriptConn<-file(scriptPath) 15 | # writeLines(options$code, scriptConn) 16 | # close(scriptConn) 17 | # cur <- getwd() 18 | # on.exit(setwd(cur)) 19 | # setwd(workDir) 20 | # out <- system2('nextflow', args = c('run', '.'), stdout = TRUE) 21 | # message(length(out)) 22 | # } else { 23 | # out <- '' 24 | # } 25 | # knitr::engine_output(options, code, out) 26 | # }) 27 | ``` 28 | 29 | There exists in Groovy a wealth of helper classes that can be imported into Nextflow scripts. In this chapter, we create a very small Workflow using the FastP tool to investigate importing the Groovy JSONSlurper class. 30 | 31 | First, let's move into the chapter 4 directory: 32 | 33 | ```{bash} 34 | cd chapter_04_groovy 35 | ``` 36 | 37 | Let's assume that we would like to pull in a samplesheet, parse the entries and run them through the FastP tool. So far, we have been concerned with local files, but Nextflow will handle remote files transparently: 38 | 39 | ```{groovy} 40 | workflow { 41 | params.input = "https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/samplesheet/v3.4/samplesheet_test.csv" 42 | 43 | Channel.fromPath(params.input) 44 | | splitCsv(header: true) 45 | | view 46 | } 47 | ``` 48 | 49 | Let's write a small closure to parse each row into the now-familiar map + files shape. We might start by constructing the meta-map: 50 | 51 | ```{groovy} 52 | workflow { 53 | params.input = "https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/samplesheet/v3.4/samplesheet_test.csv" 54 | 55 | Channel.fromPath(params.input) 56 | | splitCsv(header: true) 57 | | map { row -> 58 | meta = row.subMap('sample', 'strandedness') 59 | meta 60 | } 61 | | view 62 | } 63 | ``` 64 | 65 | ... but this precludes the possibility of adding additional columns to the samplesheet. We might to ensure the parsing will capture any extra metadata columns should they be added. Instead, let's partition the column names into those that begin with "fastq" and those that don't: 66 | 67 | ```{groovy} 68 | (readKeys, metaKeys) = row.keySet().split { it =~ /^fastq/ } 69 | ``` 70 | 71 | ::: {.callout-note} 72 | ## New methods 73 | We've introduced a new keySet method here. This is a method on Java's LinkedHashMap class ([docs here](https://docs.oracle.com/javase/8/docs/api/java/util/LinkedHashMap.html#keySet--)) 74 | 75 | We're also using the `.split()` method, which divides collection based on the return value of the closure. The mrhaki blog [provides a succinct summary](https://blog.mrhaki.com/2009/12/groovy-goodness-splitting-with-closures.html). 76 | ::: 77 | 78 | From here, let's 79 | 80 | ```{groovy} 81 | reads = row.subMap(readKeys).values().collect { file(it) } 82 | ``` 83 | 84 | ... but we run into an error: 85 | 86 | ```{groovy} 87 | Argument of `file` function cannot be empty 88 | ``` 89 | 90 | If we have a closer look at the samplesheet, we notice that not all rows have two read pairs. Let's add a condition 91 | 92 | ```{groovy} 93 | reads = row 94 | .subMap(readKeys) 95 | .values() 96 | .findAll { it != "" } // Single-end reads will have an empty string 97 | .collect { file(it) } // Turn those strings into paths 98 | ``` 99 | 100 | Now we need to construct the meta map. Let's have a quick look at the FASTP module that I've already pre-defined: 101 | 102 | ```{groovy} 103 | process FASTP { 104 | container 'quay.io/biocontainers/fastp:0.23.2--h79da9fb_0' 105 | 106 | input: 107 | tuple val(meta), path(reads) 108 | 109 | output: 110 | tuple val(meta), path('*.fastp.fastq.gz') , optional:true, emit: reads 111 | tuple val(meta), path('*.json') , emit: json 112 | 113 | script: 114 | def prefix = task.ext.prefix ?: meta.id 115 | if (meta.single_end) { 116 | // SNIP 117 | } else { 118 | // SNIP 119 | } 120 | ``` 121 | 122 | I can see that we require two extra keys, `id` and `single_end`: 123 | 124 | ```{groovy} 125 | meta = row.subMap(metaKeys) 126 | meta.id ?= meta.sample 127 | meta.single_end = reads.size == 1 128 | ``` 129 | 130 | This is now able to be passed through to our FASTP process: 131 | 132 | ```{groovy} 133 | Channel.fromPath(params.input) 134 | | splitCsv(header: true) 135 | | map { row -> 136 | (readKeys, metaKeys) = row.keySet().split { it =~ /^fastq/ } 137 | reads = row.subMap(readKeys).values() 138 | .findAll { it != "" } // Single-end reads will have an empty string 139 | .collect { file(it) } // Turn those strings into paths 140 | meta = row.subMap(metaKeys) 141 | meta.id ?= meta.sample 142 | meta.single_end = reads.size == 1 143 | [meta, reads] 144 | } 145 | | FASTP 146 | 147 | FASTP.out.json | view 148 | ``` 149 | 150 | Let's assume that we want to pull some information out of these JSON files. To make our lives a little more convenient, let's "publish" these json files so that they are more convenient. We're going to discuss configuration more completely in a later chapter, but that's no reason not to dabble a bit here. 151 | 152 | We'd like to add a `publishDir` directive to our FASTP process. 153 | 154 | ```{groovy} 155 | process { 156 | withName: 'FASTP' { 157 | publishDir = [ 158 | path: { "results/fastp/json" }, 159 | saveAs: { filename -> filename.endsWith('.json') ? filename : null }, 160 | ] 161 | } 162 | } 163 | ``` 164 | 165 | ::: {.callout-note} 166 | ## Groovy Tip: Elvis Operator 167 | This pattern of returning something if it is true and `somethingElse` if not: 168 | 169 | ```{groovy} 170 | somethingThatMightBeFalsey ? somethingThatMightBeFalsey : somethingElse 171 | ``` 172 | 173 | has a shortcut in Groovy - the "Elvis" operator: 174 | 175 | ```{groovy} 176 | somethingThatMightBeFalsey ?: somethingElse 177 | ``` 178 | ::: 179 | 180 | This enables us to iterate quickly to test out our JSON parsing without waiting on the FASTP caching to calculate on these slow virtual machines. 181 | 182 | ```{bash} 183 | nextflow run . -resume 184 | ``` 185 | 186 | Let's consider the possibility that we'd like to capture some of these metrics so that they can be used downstream. First, we'll have a quick peek at the [Groovy docs](https://groovy-lang.org/documentation.html) and I see that I need to import a `JsonSlurper`: 187 | 188 | ```{groovy} 189 | import groovy.json.JsonSlurper 190 | 191 | // We can also import a Yaml parser just as easily: 192 | // import org.yaml.snakeyaml.Yaml 193 | // new Yaml().load(new FileReader('your/data.yml')) 194 | ``` 195 | 196 | Now let's create a second entrypoint to quickly pass these JSON files through some tests: 197 | 198 | ::: {.callout-note} 199 | ## Entrypoint developing 200 | Using a second Entrypoint allows us to do quick debugging or development using a small section of the workflow without disturbing the main flow. 201 | ::: 202 | 203 | ```{groovy} 204 | workflow Jsontest { 205 | Channel.fromPath("results/fastp/json/*.json") 206 | | view 207 | } 208 | ``` 209 | 210 | which we run with 211 | 212 | ```{bash} 213 | nextflow run . -resume -entry Jsontest 214 | ``` 215 | 216 | Let's create a small function at the top of the workflow to take the JSON path and pull out some basic metrics: 217 | 218 | ```{bash} 219 | def getFilteringResult(json_file) { 220 | fastpResult = new JsonSlurper().parseText(json_file.text) 221 | } 222 | ``` 223 | 224 | ### Exercise 225 | 226 | The `fastpResult` returned from the `parseText` method is a large Map - a class which we're already familiar with. Modify the `getFilteringResult` function to return just the `after_filtering` section of the report. 227 | 228 |
229 | Reveal answer 230 | 231 | Here is one potential solution. 232 | 233 | ```{groovy} 234 | def getFilteringResult(json_file) { 235 | new JsonSlurper().parseText(json_file.text) 236 | ?.summary 237 | ?.after_filtering 238 | } 239 | ``` 240 | 241 | ::: {.callout-note} 242 | ## New notation: ?. 243 | This new notation is a null-safe access operator. The `?.summary` will access the summary property if the property exists. 244 | ::: 245 |
246 | 247 | We can then join this new map back to the original reads using the `join` operator: 248 | 249 | ```{groovy} 250 | FASTP.out.json 251 | | map { meta, json -> [meta, getFilteringResult(json)] } 252 | | join( FASTP.out.reads ) 253 | | view 254 | ``` 255 | 256 | ### Exercise 257 | 258 | Can you amend this pipeline to create two channels that filter the reads to exclude any samples where the Q30 rate is less than 93.5% 259 | 260 |
261 | Reveal answer 262 | 263 | ```{groovy} 264 | FASTP.out.json 265 | | map { meta, json -> [meta, getFilteringResult(json)] } 266 | | join( FASTP.out.reads ) 267 | | map { meta, fastpMap, reads -> [meta + fastpMap, reads] } 268 | | branch { meta, reads -> 269 | pass: meta.q30_rate >= 0.935 270 | fail: true 271 | } 272 | | set { reads } 273 | 274 | reads.fail | view { meta, reads -> "Failed: ${meta.id}" } 275 | reads.pass | view { meta, reads -> "Passed: ${meta.id}" } 276 | ``` 277 | 278 | 279 |
280 | 281 | 322 | -------------------------------------------------------------------------------- /docs/operators.qmd: -------------------------------------------------------------------------------- 1 | --- 2 | engine: knitr 3 | --- 4 | 5 | ```{r setup, eval=TRUE, include=FALSE} 6 | knitr::knit_engines$set(groovy = function(options) { 7 | code <- paste(options$code, collapse = '\n') 8 | if(options$eval) { 9 | workDir <- tempdir() 10 | file.copy("../chapter_01_operators/data", workDir, recursive=TRUE) 11 | scriptPath <- file.path(workDir, 'main.nf') 12 | scriptConn<-file(scriptPath) 13 | writeLines(options$code, scriptConn) 14 | close(scriptConn) 15 | cur <- getwd() 16 | on.exit(setwd(cur)) 17 | setwd(workDir) 18 | out <- system2('nextflow', args = c('run', '.'), stdout = TRUE) 19 | message(length(out)) 20 | } else { 21 | out <- '' 22 | } 23 | knitr::engine_output(options, code, out) 24 | }) 25 | ``` 26 | 27 | # Operator Tour 28 | 29 | In this chapter, we take a curated tour of the Nextflow operators. Commonly used and well understood operators are not covered here - only those that we've seen could use more attention or those where the usage could be more elaborate. 30 | 31 | These modest set of operators have been chosen to simultaneously demo tangential concepts and Nextflow features. 32 | 33 | ## `map` 34 | 35 | ### Basics 36 | 37 | Map is certainly the most commonly used of the operators covered here. It's a way to supply a closure through which each element in the channel is passed. The return value of the closure is emitted as an element in a new output channel. A canonical example is a closure that multiplies two numbers: 38 | 39 | ```{groovy eval=FALSE} 40 | workflow { 41 | Channel.of( 1, 2, 3, 4, 5 ) 42 | | map { it * it } 43 | | view 44 | } 45 | ``` 46 | 47 | By default, the element being passed to the closure is given the default name `it`. The variable can be named by using the `->` notation: 48 | 49 | ```{groovy eval=FALSE} 50 | workflow { 51 | Channel.of( 1, 2, 3, 4, 5 ) 52 | | map { num -> num * num } 53 | | view 54 | } 55 | ``` 56 | 57 | Groovy is an optionally typed language, and it is possible to specify the type of the argument passed to the closure. 58 | 59 | ```{groovy eval=FALSE} 60 | workflow { 61 | Channel.of( 1, 2, 3, 4, 5 ) 62 | | map { Integer num -> num * num } 63 | | view 64 | } 65 | ``` 66 | 67 | ### Named Closures 68 | 69 | If you find yourself re-using the same closure multiple times in your pipeline, the closure can be named and referenced: 70 | 71 | ```{groovy eval=FALSE} 72 | def squareIt = { Integer num -> num * num } 73 | 74 | workflow { 75 | Channel.of( 1, 2, 3, 4, 5 ) 76 | | map( squareIt ) 77 | | view 78 | } 79 | ``` 80 | 81 | If you have these re-usable closures defined, you can compose them together. 82 | 83 | ```{groovy eval=TRUE} 84 | def squareIt = { it * it } 85 | def addTwo = { it + 2 } 86 | 87 | workflow { 88 | Channel.of( 1, 2, 3, 4, 5 ) 89 | | map( squareIt >> addTwo ) 90 | | view 91 | } 92 | ``` 93 | 94 | The above is the same as writing: 95 | 96 | ```{groovy eval=FALSE} 97 | def squareIt = { it * it } 98 | def addTwo = { it + 2 } 99 | 100 | workflow { 101 | Channel.of( 1, 2, 3, 4, 5 ) 102 | | map( squareIt ) 103 | | map( addTwo ) 104 | | view 105 | } 106 | ``` 107 | 108 | For those inclined towards functional programming, you'll be happy to know that closures can be curried: 109 | 110 | ```{groovy eval=FALSE} 111 | def timesN = { multiplier, it -> it * multiplier } 112 | def timesTen = timesN.curry(10) 113 | 114 | workflow { 115 | Channel.of( 1, 2, 3, 4, 5 ) 116 | | map( timesTen ) 117 | | view 118 | } 119 | ``` 120 | 121 | ## `view` 122 | 123 | In addition to the argument-less usage of `view` as shown above, this operator can also take a closure to customize the stdout message. We can create a closure to print the value of the elements in a channel as well as their type, for example: 124 | 125 | ```{groovy eval=TRUE} 126 | def timesN = { multiplier, it -> it * multiplier } 127 | def timesTen = timesN.curry(10) 128 | def prettyPrint = { "Found '$it' (${it.getClass()})"} 129 | 130 | workflow { 131 | Channel.of( 1, 2, 3, 4, 5 ) 132 | | map( timesTen ) 133 | | view( prettyPrint ) 134 | } 135 | ``` 136 | 137 | ::: {.callout-note} 138 | ## Most closures will remain anonymous 139 | In many cases, it is simply cleaner to keep the closure anonymous, defined inline. Giving closures a name is only recommended when you find yourself defining the same or similar closures repeatedly in a given workflow. 140 | ::: 141 | 142 | ## `splitCsv` 143 | 144 | It is common that a samplesheet is passed as input into a Nextflow workflow. We'll see some more complicated ways to manage these inputs later on in the workshop, but the `splitCsv` is an excellent tool to have in a pinch. 145 | 146 | ```{groovy eval=FALSE} 147 | workflow { 148 | Channel.fromPath("data/samplesheet.csv") 149 | | splitCsv( header: true ) 150 | | view 151 | } 152 | ``` 153 | 154 | ### Exercise 155 | 156 | From the directory `chapter_01_operators`, use the `splitCsv` and `map` operators to create a channel that would be suitable input to the 157 | 158 | ```{groovy} 159 | process FastQC { 160 | input: 161 | tuple val(id), path(fastqs) 162 | // 163 | ``` 164 | 165 |
166 | Show answer 167 | 168 | Specifying the `header` argument in the `splitCsv` operator, we have convenient named access to csv elements. The closure returns a list of two elements where the second element a list of paths. 169 | 170 | ```{groovy} 171 | workflow { 172 | Channel.fromPath("data/samplesheet.csv") 173 | | splitCsv( header: true ) 174 | | map { row -> 175 | [row.id, [file(row.fastq1), file(row.fastq2)]] 176 | } 177 | | view 178 | } 179 | ``` 180 | 181 | ::: {.callout-warning} 182 | ## Convert Strings to Paths 183 | The fastq paths are simple strings in the context of a csv row. In order to pass them as paths to a Nextflow process, they need to be converted into objects that adjere to the `Path` interface. This is accomplished by wrapping them in `file`. 184 | ::: 185 | 186 | 187 | In the sample above, we've lost an important piece of metadata - the tumor/normal classification, choosing only the sample id as the first element in the output list. 188 | In the next chapter, we'll discuss the "meta map" pattern in more detail, but we can preview that here. 189 | 190 | ```{groovy} 191 | workflow { 192 | Channel.fromPath("data/samplesheet.csv") 193 | | splitCsv( header: true ) 194 | | map { row -> 195 | metaMap = [id: row.id, type: row.type, repeat: row.repeat] 196 | [metaMap, [file(row.fastq1), file(row.fastq2)]] 197 | } 198 | | view 199 | } 200 | ``` 201 | 202 | The construction of this map is very repetitive, and in the next chapter, we'll discuss some Groovy methods available on the `Map` class that can make this pattern more concise and less error-prone. 203 | 204 |
205 | 206 | ## `multiMap` 207 | 208 | The `multiMap` operator is a way of creating multiple channels from a single source. 209 | 210 | Let's assume we've been given a samplesheet that has tumor/normal pairs bundled together on the same row. 211 | 212 | ```{bash} 213 | cd chapter_01_operators 214 | cat data/samplesheet.ugly.csv 215 | ``` 216 | 217 | Using the `splitCsv` operator would give us one entry that would contain all four fastq files. Let's consider that we wanted to split these fastqs into separate channels for tumor and normal, we could use `multiMap`: 218 | 219 | ```{groovy} 220 | workflow { 221 | Channel.fromPath("data/samplesheet.ugly.csv") 222 | | splitCsv( header: true ) 223 | | multiMap { row -> 224 | tumor: 225 | metamap = [id: row.id, type:'tumor', repeat:row.repeat] 226 | [metamap, file(row.tumor_fastq_1), file(row.tumor_fastq_2)] 227 | normal: 228 | metamap = [id: row.id, type:'normal', repeat:row.repeat] 229 | [metamap, file(row.normal_fastq_1), file(row.normal_fastq_2)] 230 | } 231 | | set { samples } 232 | 233 | samples.tumor | view { "Tumor: $it"} 234 | samples.normal | view { "Normal: $it"} 235 | } 236 | ``` 237 | 238 | ::: {.callout-tip} 239 | ## Tip: `multiMapCriteria` 240 | The closure supplied to `multiMap` needs to return multiple channels, so using named closures as described in the `map` section above will not work. Fortunately, Nextflow provides the convenience `multiMapCriteria` method to allow you to define named `multiMap` closures should you need them. See the [`multiMap` documentation](https://www.nextflow.io/docs/latest/operator.html#multimap) for more info. 241 | ::: 242 | 243 | ## `branch` 244 | 245 | In the example above, the `multiMap` operator was necessary because we were supplied with a samplesheet that combined two pairs of fastq per row. If we were to use the neater samplesheet, we could use the `branch` operator to achieve the same result. 246 | 247 | ```{groovy} 248 | workflow { 249 | Channel.fromPath("data/samplesheet.csv") 250 | | splitCsv( header: true ) 251 | | map { row -> [[id: row.id, repeat: row.repeat, type: row.type], [file(row.fastq1), file(row.fastq2)]] } 252 | | branch { meta, reads -> 253 | tumor: meta.type == "tumor" 254 | normal: meta.type == "normal" 255 | } 256 | | set { samples } 257 | 258 | samples.tumor | view { "Tumor: $it"} 259 | samples.normal | view { "Normal: $it"} 260 | } 261 | ``` 262 | 263 | An element is only emitted to the first channel were the test condition is met. If an element does not meet any of the tests, it is not emitted to any of the output channels. You can 'catch' any such samples by specifying `true` as a condition. If we knew that all samples would be either tumor or normal and no third 'type', we could write 264 | 265 | ```{groovy} 266 | branch { meta, reads -> 267 | tumor: meta.type == "tumor" 268 | normal: true 269 | } 270 | ``` 271 | 272 | We can optionally return a new element to one or more of the output channels. For example, to add an extra key in the meta map of the tumor samples, we add a new line under the condition and return our new element. In this example, we modify the first element of the `List` to be a new list that is the result of merging the existing meta map with a new map containing a single key: 273 | 274 | ```{groovy} 275 | branch { meta, reads -> 276 | tumor: meta.type == "tumor" 277 | return [meta + [newKey: 'myValue'], reads] 278 | normal: true 279 | } 280 | ``` 281 | 282 | 283 | ### Exercise 284 | 285 | How would you modify the element returned in the `tumor` channel to have the key:value pair `type:'abnormal'` instead of `type:'tumor'`? 286 | 287 |
288 | 289 | Show answer 290 | 291 | 292 | There are many ways to accomplish this, but the map merging pattern introduced above can also be used to safely and concisely rename values in a map. 293 | 294 | ```{groovy} 295 | branch { meta, reads -> 296 | tumor: meta.type == "tumor" 297 | return [meta + [type: 'abnormal'], reads] 298 | normal: true 299 | } 300 | ``` 301 | 302 | ::: {.callout-note} 303 | ## Merging maps is safe 304 | Using the `+` operator to merge two or more Maps returns a _new_ Map. There are rare edge cases where modification of map rather than returning a new map can affect other channels. We discuss this further in the next chapter, but just be aware that this `+` operator is safer and often more convenient than modifying the `meta` object directly. 305 | 306 | See the Groovy [Map documentation](https://docs.groovy-lang.org/latest/html/groovy-jdk/java/util/Map.html#plus(java.util.Map)) for details. 307 | ::: 308 | 309 |
310 | 311 | ### Multi-channel Objects 312 | 313 | Some Nextflow operators return objects that contain _multiple_ channels. The `multiMap` and `branch` operators are excellent examples. In most instances, the output is assigned to a variable and then addressed by name: 314 | 315 | ```{groovy} 316 | numbers = Channel.from(1,2,3,4,5) 317 | | multiMap { 318 | small: it 319 | large: it * 10 320 | } 321 | numbers.small | view { num -> "Small: $num"} 322 | numbers.large | view { num -> "Large: $num"} 323 | ``` 324 | 325 | or by using `set`: 326 | 327 | ```{groovy} 328 | Channel.from(1,2,3,4,5) 329 | | multiMap { 330 | small: it 331 | large: it * 10 332 | } 333 | | set { numbers } 334 | 335 | numbers.small | view { num -> "Small: $num"} 336 | numbers.large | view { num -> "Large: $num"} 337 | ``` 338 | 339 | Given a process that takes multiple channels 340 | 341 | ```{groovy} 342 | process MultiInput { 343 | debug true 344 | input: 345 | val(smallNum) 346 | val(bigNum) 347 | 348 | "echo -n small is $smallNum and big is $bigNum" 349 | } 350 | ``` 351 | 352 | You can either provide the channels individually: 353 | 354 | ```{groovy} 355 | Channel.from(1,2,3,4,5) 356 | | multiMap { 357 | small: it 358 | large: it * 10 359 | } 360 | | set { numbers } 361 | 362 | MultiInput(numbers.small, numbers.large) 363 | ``` 364 | 365 | or you can provide the multichannel as a single input: 366 | 367 | ```{groovy} 368 | Channel.from(1,2,3,4,5) 369 | | multiMap { 370 | small: it 371 | large: it * 10 372 | } 373 | | set { numbers } 374 | 375 | MultiInput(numbers) 376 | ``` 377 | 378 | This also means you can skip the `set` operator for the cleanest solution: 379 | 380 | ```{groovy} 381 | Channel.from(1,2,3,4,5) 382 | | multiMap { 383 | small: it 384 | large: it * 10 385 | } 386 | | MultiInput 387 | ``` 388 | 389 | If you have processes that output multiple channels and input multiple channels and the cardinality matches, they can be chained together in the same manner. 390 | 391 | ## `transpose` 392 | 393 | The transpose operator is often misunderstood. It can be thought of as the inverse of the `groupTuple` operator. Give the following workflow, the `groupTuple` and `transpose` operators cancel each other out. Removing lines 8 and 9 returns the same result. 394 | 395 | Given a workflow that returns one element per sample, where we have grouped the samplesheet lines on a meta containing only id and type: 396 | 397 | ```{groovy eval=TRUE} 398 | workflow { 399 | Channel.fromPath("data/samplesheet.csv") 400 | | splitCsv(header: true) 401 | | map { row -> 402 | meta = [id: row.id, type: row.type] 403 | [meta, row.repeat, [row.fastq1, row.fastq2]] 404 | } 405 | | groupTuple 406 | | view 407 | } 408 | ``` 409 | 410 | If we add in a `transpose`, each repeat number is matched back to the appropriate list of reads: 411 | 412 | ```{groovy eval=TRUE} 413 | workflow { 414 | Channel.fromPath("data/samplesheet.csv") 415 | | splitCsv(header: true) 416 | | map { row -> 417 | meta = [id: row.id, type: row.type] 418 | [meta, row.repeat, [row.fastq1, row.fastq2]] 419 | } 420 | | groupTuple 421 | | transpose 422 | | view 423 | } 424 | ``` 425 | 426 | 431 | 432 | -------------------------------------------------------------------------------- /docs/_book/references.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Intermediate/Advanced Nextflow - References 11 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 |
89 |
90 | 103 | 111 |
112 | 113 |
114 | 115 | 201 | 202 | 205 | 206 |
207 | 208 |
209 |
210 |

References

211 |
212 | 213 | 214 | 215 |
216 | 217 | 218 | 219 | 220 |
221 | 222 | 223 |
224 | 225 |
226 | 227 |
228 | 229 | 230 | 231 |
232 | 365 | 374 |
375 | 376 | 377 | 378 | 379 | -------------------------------------------------------------------------------- /docs/_book/summary.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Intermediate/Advanced Nextflow - 8  Summary 11 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 |
71 |
72 | 85 | 93 |
94 | 95 |
96 | 97 | 183 | 184 | 187 | 188 |
189 | 190 |
191 |
192 |

8  Summary

193 |
194 | 195 | 196 | 197 |
198 | 199 | 200 | 201 | 202 |
203 | 204 | 205 |
206 | 207 |

Summary available on day #2

208 | 209 | 210 | 211 |
212 | 345 | 357 |
358 | 359 | 360 | 361 | 362 | -------------------------------------------------------------------------------- /docs/_book/support.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Intermediate/Advanced Nextflow - 9  Links 11 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 |
71 |
72 | 85 | 93 |
94 | 95 |
96 | 97 | 183 | 184 | 187 | 188 |
189 | 190 |
191 |
192 |

9  Links

193 |
194 | 195 | 196 | 197 |
198 | 199 | 200 | 201 | 202 |
203 | 204 | 205 |
206 | 207 | 211 | 212 | 213 | 214 |
215 | 348 | 360 |
361 | 362 | 363 | 364 | 365 | -------------------------------------------------------------------------------- /docs/_book/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Intermediate/Advanced Nextflow 11 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 |
70 |
71 | 84 | 92 |
93 | 94 |
95 | 96 | 182 | 183 | 192 | 193 |
194 | 195 |
196 |
197 |

Intermediate/Advanced Nextflow

198 |
199 | 200 | 201 | 202 |
203 | 204 | 205 | 206 | 207 |
208 | 209 | 210 |
211 | 212 |
213 |

Welcome

214 |

Welcome to our Nextflow workshop for intermediate and advanced users! In this workshop, we will explore the advanced features of the Nextflow language and runtime, and learn how to use them to write efficient and scalable data-intensive workflows. We will cover topics such as parallel execution, error handling, and workflow customization. Please note that this is not an introductory workshop, and we will assume some basic familiarity with Nextflow. By the end of this workshop, you will have the skills and knowledge to create complex and powerful Nextflow pipelines for your own data analysis projects. Let’s get started!

215 | 216 | 217 |
218 | 219 |
220 | 353 | 362 |
363 | 364 | 365 | 366 | 367 | -------------------------------------------------------------------------------- /docs/_book/testing.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Intermediate/Advanced Nextflow - 7  Troubleshooting 11 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 |
71 |
72 | 85 | 93 |
94 | 95 |
96 | 97 | 183 | 184 | 187 | 188 |
189 | 190 |
191 |
192 |

7  Troubleshooting

193 |
194 | 195 | 196 | 197 |
198 | 199 | 200 | 201 | 202 |
203 | 204 | 205 |
206 | 207 |

To be introduced on day #2

208 | 216 | 217 | 218 | 219 |
220 | 353 | 365 |
366 | 367 | 368 | 369 | 370 | -------------------------------------------------------------------------------- /docs/_book/configuration.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Intermediate/Advanced Nextflow - 6  Configuration 11 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 |
71 |
72 | 85 | 93 |
94 | 95 |
96 | 97 | 183 | 184 | 193 | 194 |
195 | 196 |
197 |
198 |

6  Configuration

199 |
200 | 201 | 202 | 203 |
204 | 205 | 206 | 207 | 208 |
209 | 210 | 211 |
212 | 213 |

To be introduced on day #2

214 |

This is an aspect of Nextflow that can be confusing. There are multiple ways of loading configuration and parameters into a Nextflow. This gives us two complications:

215 |
    216 |
  • At which location should I be loading a configuration value?
    • 217 |
  • Given a particular parameter, how do I know where it was set?
    • 218 |
219 |
220 |

6.1 Precedence

221 |
    222 |
  1. Parameters specified on the command line (–something value)
    2. 223 |
  2. Parameters provided using the -params-file option
    3. 224 |
  3. Config file specified using the -c my_config option
    4. 225 |
  4. The config file named nextflow.config in the current directory
    5. 226 |
  5. The config file named nextflow.config in the workflow project directory
    6. 227 |
  6. The config file $HOME/.nextflow/config
    7. 228 |
  7. Values defined within the pipeline script itself (e.g. main.nf)
    8. 229 |
230 | 239 | 240 | 241 |
242 | 243 |
244 | 377 | 389 |
390 | 391 | 392 | 393 | 394 | --------------------------------------------------------------------------------