├── .github ├── ISSUE_TEMPLATE │ ├── claim-software.md │ ├── modifying-a-workflow.md │ └── workflow-contribution.md ├── PULL_REQUEST_TEMPLATE │ ├── modifying-a-workflow.md │ └── new-workflow.md └── workflows │ ├── CSIS.yml │ ├── busco.yml │ ├── centrifuge.yml │ ├── checkm.yml │ ├── genotyphi.yml │ ├── ivar.yml │ ├── kraken.yml │ ├── kraken2.yml │ ├── krakenuniq.yml │ ├── pangolin.yml │ ├── prokka.yml │ ├── quast.yml │ ├── shovill.yml │ ├── skesa.yml │ ├── trycycler.yml │ └── unicycler.yml ├── CITATION.cff ├── CONTRIBUTING.md ├── EXAMPLES.md ├── HACKATHON.md ├── LICENSE ├── README.md ├── TEAMS.md └── templates ├── bug_template.md ├── feedback_template.md ├── support_template.md └── test_template.md /.github/ISSUE_TEMPLATE/claim-software.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Claim software (do this first!) 3 | about: Tell us that you are working on unit testing for some software 4 | title: "[CLAIM]" 5 | labels: Claim 6 | assignees: lskatz, boasvdp 7 | 8 | --- 9 | 10 | **What is the software?** 11 | Provide a name and URL 12 | 13 | **Which team is working on it?** 14 | State your team name here 15 | 16 | **Is any other team working on it?** 17 | Tell us that you made sure that no one else has claimed this software. 18 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/modifying-a-workflow.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Modifying a workflow 3 | about: Modify a workflow 4 | title: "[WORKFLOW MODIFICATION]" 5 | labels: '' 6 | assignees: lskatz, boasvdp 7 | 8 | --- 9 | 10 | **Which workflow are you modifying** 11 | Provide a link or filename to the yaml file 12 | 13 | **Ownership** 14 | Provide a link to the issue where your team claimed this workflow 15 | If this is not your team's workflow, tag the username(s) of another team 16 | so that they can discuss your changes and write their approval. 17 | 18 | **Changes** 19 | What did you modify? 20 | 21 | **Further** 22 | Provide any additional details. 23 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/workflow-contribution.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Workflow contribution 3 | about: Contribute a new workflow for unit testing 4 | title: "[NEW WORKFLOW]" 5 | labels: New workflow 6 | assignees: lskatz, boasvdp 7 | 8 | --- 9 | 10 | **What is the software** 11 | Provide a name and a URL to the software, preferably with the specific version you've tested 12 | 13 | **The claim** 14 | Provide a link to where you claimed this software in Issues. 15 | 16 | **Further information** 17 | Please provide any details here 18 | 19 | **Checklist** 20 | * Naming :name\_badge: 21 | * [ ] The workflow `.yml` file matches the name of the software. 22 | * Trigger :boom: 23 | * [ ] The workflow is triggered on pushes 24 | * [ ] The workflow is triggered on pull requests 25 | * [ ] The workflow is triggered on the main branch 26 | * [ ] The workflow is triggered on the dev branch 27 | * Operating systems :penguin: 28 | * [ ] Linux 29 | * [ ] Mac (optional) 30 | * [ ] Windows (optional) 31 | * Minimal testing requirements :ok\_hand: 32 | * [ ] Installation works 33 | * [ ] Flags are used correctly 34 | * [ ] Small test file processed correctly 35 | 36 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE/modifying-a-workflow.md: -------------------------------------------------------------------------------- 1 | # Workflow modification 2 | 3 | ## Link to issue 4 | 5 | Provide a link to the GitHub issue here: ... 6 | 7 | ## What the workflow did 8 | 9 | ## What we changed 10 | 11 | - used to do X 12 | - Now it does Y 13 | - used to do A 14 | - Now it does B 15 | 16 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE/new-workflow.md: -------------------------------------------------------------------------------- 1 | # New workflow 2 | 3 | Please fill in the table below: 4 | 5 | | Question | Your answer | 6 | | --- | --- | 7 | | Which software is tested? | SOFTWARE NAME | 8 | | Do you have a link to your Github Actions YAML file? | LINK TO YML | 9 | 10 | ## Examples 11 | 12 | For an example of a basic testing workflow, see [this Prokka workflow](../workflows/prokka.yml). 13 | 14 | For an example of an advanced testing workflow, see [this CheckM workflow](../workflows/checkm.yml), courtesy of [Chris Gulvik](https://github.com/chrisgulvik)). 15 | 16 | ## Checklist 17 | 18 | Please fill in the checklists below: 19 | 20 | - [ ] I've read the guidelines for contributions: [CONTRIBUTING.md](../../CONTRIBUTING.md) 21 | 22 | Checking the `name` and `on` sections (all boxes should be checked): 23 | - [ ] The name of the test matches the name of the .yml file 24 | - [ ] The tests are triggered on both push or pull requests, on the `main` and `dev` branches 25 | 26 | These tests run on (at least one of these boxes should be checked): 27 | - [ ] ubuntu-16.04, ubuntu-18.04, ubuntu-20.04 or ubuntu-latest (currently 18.04) 28 | - [ ] macos-10.15, macos-11.0, or macos-latest (currently 10.15) 29 | - [ ] windows-2019 or windows-latest (currently 2019) 30 | 31 | The tests include (all boxes should be checked, see [this workflow as an example for a basic example](../workflows/prokka.yaml)): 32 | - [ ] Installation of the software you're testing 33 | - [ ] Checking what happens when the software is invoked without commands (e.g. `prokka`) 34 | - [ ] Checking whether the `-h` and/or `--help` switches do not raise an error (e.g. `prokka -h` and/or `prokka --help`) 35 | - [ ] Checking whether the `-v` and/or `--version` switches do not raise an error (e.g. `prokka -v` and/or `prokka --version`) 36 | - [ ] Checking whether an error is raised when a non-sensical command is specified (e.g. `prokka --doesnotexist`) 37 | - [ ] Performing a small, functional/black box test (does the tool complete on a toy dataset without raising errors?) 38 | 39 | Optional tests (see [this workflow as an example for an advanced workflow](../workflows/checkm.yaml), courtesy of @chrisgulvik: 40 | - [ ] Checking whether an error is correctly raised when invalid input file is specified (e.g. a truncated FastQ file) 41 | - [ ] Checking whether dependencies are installed correctly 42 | - [ ] Installing dependencies from source 43 | - [ ] Checking whether dependencies pass their own tests 44 | -------------------------------------------------------------------------------- /.github/workflows/CSIS.yml: -------------------------------------------------------------------------------- 1 | # This is a basic workflow to help you get started with Actions 2 | 3 | name: CSIS 4 | 5 | # Controls when the action will run. Triggers the workflow on push or pull request 6 | # events but only for the main branch 7 | on: 8 | push: 9 | branches: [ main, dev ] 10 | pull_request: 11 | branches: [ main, dev ] 12 | 13 | # A workflow run is made up of one or more jobs that can run sequentially or in parallel 14 | jobs: 15 | # This workflow contains a single job called "build" 16 | build: 17 | # The type of runner that the job will run on 18 | runs-on: ubuntu-latest 19 | 20 | # Steps represent a sequence of tasks that will be executed as part of the job 21 | steps: 22 | # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it 23 | - uses: actions/checkout@v2 24 | with: 25 | path: CSIS 26 | - name: apt 27 | run: sudo apt-get install tree 28 | # Runs a single command using the runners shell 29 | - name: Run a one-line script 30 | run: echo Hello, world! 31 | - name: list shallow contents 32 | run: tree -L 2 -d 33 | - name: list contents 34 | run: tree -d 35 | 36 | # Runs a set of commands using the runners shell 37 | - name: Run a multi-line script 38 | run: | 39 | echo Add other actions to build, 40 | echo test, and deploy your project. 41 | -------------------------------------------------------------------------------- /.github/workflows/busco.yml: -------------------------------------------------------------------------------- 1 | name: busco 2 | 3 | on: 4 | push: 5 | branches: [main, dev] 6 | pull_request: 7 | branches: [main, dev] 8 | schedule: 9 | - cron: '55 5 1 * *' 10 | 11 | jobs: 12 | build: 13 | runs-on: ${{ matrix.os }} 14 | strategy: 15 | matrix: 16 | os: [ubuntu-20.04] #arch and centos both unavailable https://github.com/actions/virtual-environments#available-environments 17 | python-version: [3.6, 3.7, 3.8] #3.9 not working 18 | 19 | steps: 20 | - name: initialize github actions test environment 21 | uses: actions/checkout@v2 22 | 23 | - name: report environment variables for debugging 24 | run: | 25 | printf '%s\n' \ 26 | "Workflow ${GITHUB_WORKFLOW}" \ 27 | "Action Number ${GITHUB_ACTION}" \ 28 | "Run ID ${GITHUB_RUN_ID}" \ 29 | "Run Number ${GITHUB_RUN_NUMBER}" \ 30 | "Workspace ${GITHUB_WORKSPACE}" 31 | 32 | - name: report conda version 33 | run: | 34 | conda --version 35 | 36 | - name: setup workspace 37 | run: | 38 | mkdir -v "${GITHUB_WORKSPACE}/Downloads" 39 | 40 | - name: install busco with conda 41 | run: | 42 | conda create -n my_env python=${{ matrix.python-version }} \ 43 | busco=4.1.4 -c conda-forge -c bioconda -y 44 | 45 | - name: report conda version and config info 46 | shell: bash -l {0} 47 | run: | 48 | conda activate my_env 49 | conda info 50 | conda list 51 | conda config --show-sources 52 | conda config --show 53 | 54 | - name: test busco 55 | run: | 56 | # NOTE: conda environment still usable without login shell as above 57 | source "${CONDA}/bin/activate" my_env 58 | # busco #empty args doesn't print help menu; exit code 1 59 | busco -h 60 | busco --help 61 | busco -v 62 | busco --version 63 | mkdir Downloads/gitlab_src_busco 64 | wget -O Downloads/gitlab_src_busco/busco-4.1.4.tar.gz \ 65 | https://gitlab.com/ezlab/busco/-/archive/4.1.4/busco-4.1.4.tar.gz 66 | tar zxvf Downloads/gitlab_src_busco/busco-4.1.4.tar.gz \ 67 | --directory Downloads/gitlab_src_busco --strip-components=1 68 | busco --in Downloads/gitlab_src_busco/test_data/bacteria/genome.fna \ 69 | --mode geno --out outdir 70 | # diff outdir/logs/busco.log \ 71 | # Downloads/gitlab_src_busco/test_data/bacteria/expected_log.txt 72 | # # NOTE: diff fails because paths and timestamps are different 73 | # # https://gitlab.com/ezlab/busco/-/blob/b40b3ba2d3c5b4793bc9d42416d9a3aebccb6752/test_data/bacteria/expected_log.txt 74 | # # recommends comparing logfile but tough due to paths and timestamps 75 | grep $'\t|C:' outdir/logs/busco.log > test_output 76 | grep $'\t|C:' \ 77 | Downloads/gitlab_src_busco/test_data/bacteria/expected_log.txt \ 78 | > expected_output 79 | diff test_output expected_output 80 | echo "diff comparison returned: $?" 81 | echo 'INFO: tests complete' 82 | -------------------------------------------------------------------------------- /.github/workflows/centrifuge.yml: -------------------------------------------------------------------------------- 1 | # This is a workflow for testing "centrifuge" (https://github.com/DaehwanKimLab/centrifuge) 2 | 3 | name: centrifuge 4 | 5 | on: 6 | push: 7 | branches: [ main, dev ] 8 | pull_request: 9 | branches: [ main, dev ] 10 | 11 | jobs: 12 | # This workflow contains a single job called "build-test" 13 | build-test: 14 | 15 | # The type of runner that the job will run on 16 | runs-on: ubuntu-latest 17 | 18 | steps: 19 | - name: Install centrifuge 20 | run: conda create -y -n centrifuge -c bioconda centrifuge 21 | 22 | - name: Clone centrifuge repo # for test data 23 | uses: actions/checkout@v2 24 | with: 25 | repository: DaehwanKimLab/centrifuge 26 | path: centrifuge 27 | ref: v1.0.4-beta 28 | 29 | - name: Staging 30 | run: | 31 | echo "${CONDA}/envs/centrifuge/bin" >> $GITHUB_PATH 32 | 33 | - name: Test centrifuge-build 34 | run: | 35 | which centrifuge-build 36 | centrifuge-build --version 37 | centrifuge-build --help 38 | centrifuge-build --conversion-table centrifuge/example/reference/gi_to_tid.dmp --taxonomy-tree centrifuge/example/reference/nodes.dmp --name-table centrifuge/example/reference/nodes.dmp centrifuge/example/reference/test.fa index 39 | 40 | - name: Test centrifuge-inspect 41 | run: | 42 | which centrifuge-inspect 43 | centrifuge-inspect --version 44 | centrifuge-inspect --help 45 | centrifuge-inspect index 46 | 47 | - name: Test centrifuge 48 | run: | 49 | which centrifuge 50 | centrifuge --version 51 | centrifuge --help 52 | centrifuge -f -x index centrifuge/example/reads/input.fa > centrifuge-report.tsv 53 | 54 | - name: Test centrifuge-kreport # convert centrifuge output to kraken format 55 | run: | 56 | which centrifuge-kreport 57 | centrifuge-kreport -x index centrifuge-report.tsv 58 | 59 | -------------------------------------------------------------------------------- /.github/workflows/checkm.yml: -------------------------------------------------------------------------------- 1 | name: checkm 2 | 3 | on: 4 | push: 5 | branches: [main, dev] 6 | pull_request: 7 | branches: [main, dev] 8 | schedule: 9 | - cron: '55 5 1 * *' 10 | 11 | jobs: 12 | build: 13 | runs-on: ${{ matrix.os }} 14 | strategy: 15 | matrix: 16 | os: [ubuntu-20.04] #arch and centos both unavailable https://github.com/actions/virtual-environments#available-environments 17 | python-version: [3.6, 3.7, 3.8, 3.9] 18 | 19 | steps: 20 | - name: initialize github actions test environment 21 | uses: actions/checkout@v2 22 | 23 | - name: setup python ${{ matrix.python-version }} 24 | uses: actions/setup-python@v2 25 | with: 26 | python-version: ${{ matrix.python-version }} 27 | architecture: x64 28 | 29 | - name: update pip package manager and the setuptools and wheel libraries 30 | run: | 31 | python -m pip install -U pip setuptools wheel 32 | # GitHub does not always have up-to-date pip, and 33 | # installation from pre-built binary archives (with pip) 34 | # will complain if pip is out of date. 35 | # Up-to-date setuptools and wheel libraries 36 | # enable installation from source archives. 37 | 38 | - name: report pip, setuptools, and wheel versions 39 | run: | 40 | pip --version 41 | _ver=$(python -c 'import setuptools; print(setuptools.__version__)') 42 | echo "setuptools version $_ver" 43 | _ver=$(python -c 'import wheel; print(wheel.__version__)') 44 | echo "wheel version $_ver" 45 | 46 | - name: report environment variables for debugging 47 | run: | 48 | printf '%s\n' \ 49 | "Workflow ${GITHUB_WORKFLOW}" \ 50 | "Action Number ${GITHUB_ACTION}" \ 51 | "Run ID ${GITHUB_RUN_ID}" \ 52 | "Run Number ${GITHUB_RUN_NUMBER}" \ 53 | "Workspace ${GITHUB_WORKSPACE}" 54 | 55 | - name: report python version 56 | run: | 57 | python -c 'import sys; print(sys.version)' 58 | 59 | - name: setup workspace 60 | run: | 61 | mkdir -v "${GITHUB_WORKSPACE}"/{bin,Downloads} 62 | echo "${GITHUB_WORKSPACE}"/bin >> "${GITHUB_PATH}" 63 | 64 | - name: install pplacer 65 | run: | 66 | wget https://github.com/matsen/pplacer/releases/download/v1.1.alpha19/pplacer-linux-v1.1.alpha19.zip \ 67 | -O ${GITHUB_WORKSPACE}/Downloads/pplacer-linux-v1.1.alpha19.zip 68 | unzip ${GITHUB_WORKSPACE}/Downloads/pplacer-linux-v1.1.alpha19.zip 69 | mv -v ${GITHUB_WORKSPACE}/pplacer-Linux-v1.1.alpha19/guppy \ 70 | ${GITHUB_WORKSPACE}/bin 71 | mv -v ${GITHUB_WORKSPACE}/pplacer-Linux-v1.1.alpha19/pplacer \ 72 | ${GITHUB_WORKSPACE}/bin 73 | # NOTE: compiling from src requires old OCaml version so avoid for now 74 | # https://github.com/matsen/pplacer/issues/354 75 | # sudo apt-get install libgsl-dev libgslcblas0 libsqlite3-dev -y 76 | 77 | - name: test pplacer 78 | run: | 79 | pplacer 80 | # `pplacer -h` prints help menu but gives exit code 2 81 | pplacer -help 82 | pplacer --help 83 | pplacer --version 84 | 85 | - name: install hmmer3 86 | run: | 87 | wget http://eddylab.org/software/hmmer/hmmer-3.3.2.tar.gz \ 88 | -O ${GITHUB_WORKSPACE}/Downloads/hmmer-3.3.2.tar.gz 89 | tar zxvf ${GITHUB_WORKSPACE}/Downloads/hmmer-3.3.2.tar.gz 90 | cd ${GITHUB_WORKSPACE}/hmmer-3.3.2 91 | ./configure --prefix ${GITHUB_WORKSPACE} 92 | make 93 | make install 94 | make check 95 | 96 | - name: test hmmer3 97 | run: | 98 | # not all binaries are used for checkm; only check those called in 99 | # https://github.com/Ecogenomics/CheckM/blob/master/checkm/hmmer.py 100 | # skip no argument tests because return value is 1 101 | # ` --help` nonexistent 102 | for _bin in hmmalign hmmfetch hmmpress hmmscan hmmsearch; do $_bin -h; man -P cat $_bin; done 103 | 104 | - name: checkout prodigal 105 | uses: actions/checkout@v2 106 | with: 107 | repository: hyattpd/Prodigal 108 | path: github_src_prodigal 109 | ref: v2.6.3 110 | 111 | - name: install prodigal 112 | run: | 113 | cd ${GITHUB_WORKSPACE}/github_src_prodigal 114 | make 115 | chmod +x prodigal 116 | mv -v prodigal \ 117 | ${GITHUB_WORKSPACE}/bin 118 | 119 | - name: test prodigal 120 | run: | 121 | # prodigal #Error: Process completed with exit code 9. 122 | prodigal -h 123 | # `prodigal --help` nonexistent option 124 | prodigal -v 125 | cd ${GITHUB_WORKSPACE}/github_src_prodigal 126 | # NOTE: v2.6.3 (from 2016) lacks test files but repo now has them 127 | git pull origin 6ccc8d1ca126969be402e5bf495ed31a74f7edd5 128 | prodigal -i anthus_aco.fas -a proteins.faa -d genes.ffn 129 | 130 | - name: install python test libraries 131 | run: | 132 | pip install click hypothesis nose pytest 133 | 134 | - name: install python library dependencies and checkm 135 | run: | 136 | pip install checkm-genome==1.1.3 dendropy matplotlib numpy pysam scipy 137 | 138 | - name: report python test libraries' versions 139 | run: | 140 | hypothesis --version 141 | nosetests --version 142 | pytest --version 143 | 144 | - name: test dendropy 145 | run: | 146 | python -c 'import dendropy' 147 | #TO-DO pytest run the DendroPy/tests; dir missing from pip install 148 | 149 | - name: test matplotlib 150 | run: | 151 | python -c 'import matplotlib' 152 | # NOTE: additional pytest fails but most of lib not used in checkm 153 | # _ver=$(python -c 'import matplotlib; print(matplotlib.__version__)') 154 | # wget -O ${GITHUB_WORKSPACE}/Downloads/matplotlib-v${_ver}.tar.gz \ 155 | # https://github.com/matplotlib/matplotlib/archive/v${_ver}.tar.gz 156 | # tar zxvf ${GITHUB_WORKSPACE}/Downloads/matplotlib-v${_ver}.tar.gz \ 157 | # --directory ${GITHUB_WORKSPACE}/Downloads 158 | # cd ${GITHUB_WORKSPACE}/Downloads/matplotlib-${_ver} 159 | # sudo apt-get install freetype inkscape 160 | # python -m pytest --maxfail=50 161 | 162 | - name: test numpy 163 | run: | 164 | python -c 'import numpy; numpy.test()' 165 | 166 | - name: test pysam 167 | run: | 168 | python -c 'import pysam' #TO-DO pytest run pysam/tests; how? 169 | 170 | - name: test scipy 171 | run: | 172 | python -c 'import scipy; scipy.test()' 173 | 174 | - name: checkout checkm 175 | uses: actions/checkout@v2 176 | with: 177 | repository: Ecogenomics/CheckM 178 | path: github_src_checkm 179 | ref: v1.1.3 180 | 181 | - name: download checkm data 182 | run: | 183 | mkdir github_src_checkm/database 184 | wget -O github_src_checkm/database/checkm_data_2015_01_16.tar.gz \ 185 | https://data.ace.uq.edu.au/public/CheckM_databases/checkm_data_2015_01_16.tar.gz 186 | tar zxvf github_src_checkm/database/checkm_data_2015_01_16.tar.gz \ 187 | --directory github_src_checkm/database 188 | 189 | - name: test checkm 190 | run: | 191 | checkm data setRoot github_src_checkm/database 192 | checkm 193 | checkm -h 194 | # `checkm --help` nonexistent option 195 | cd ${GITHUB_WORKSPACE}/github_src_checkm 196 | rm -v checkm/test/test_genomicSignatures.py 197 | python -m pytest 198 | # checkm test checkm_test_results 199 | # NOTE: `checkm test` only fails because pplacer takes a long time to 200 | # run and gets killed, so it's disabled until longer runtime allowed 201 | # for github actions testing 202 | echo 'INFO: tests complete' 203 | -------------------------------------------------------------------------------- /.github/workflows/genotyphi.yml: -------------------------------------------------------------------------------- 1 | name: genotyphi 2 | 3 | on: [push] 4 | 5 | jobs: 6 | unit-testing: 7 | 8 | runs-on: ubuntu-latest 9 | strategy: 10 | matrix: 11 | # only works in version 2.7.5+ 12 | python-version: [2.7] 13 | 14 | steps: 15 | - uses: actions/checkout@v2 16 | with: 17 | repository: katholt/genotyphi 18 | - name: Set up Python ${{ matrix.python-version }} 19 | uses: actions/setup-python@v2 20 | with: 21 | python-version: ${{ matrix.python-version }} 22 | - name: Install dependencies 23 | run: | 24 | sudo apt-get install -y --no-install-recommends samtools bcftools bowtie2 25 | - name: Get data 26 | run: | 27 | wget 'https://www.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=nucleotide&id=AL513382.1&rettype=fasta' -O tmp.fasta 28 | cat tmp.fasta | perl -lane 'if(/>/){s/.+/>AL513382.1/;} print;' > CT18.fasta 29 | grep -A 1 ">" CT18.fasta 30 | wget ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR343/ERR343343/ERR343343_1.fastq.gz 31 | wget ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR343/ERR343343/ERR343343_2.fastq.gz 32 | - name: map 33 | run: | 34 | bowtie2-build CT18.fasta CT18.fasta 35 | bowtie2 -p 2 -x CT18.fasta -1 ERR343343_1.fastq.gz -2 ERR343343_2.fastq.gz -S output.sam 36 | - name: sam to bam 37 | run: samtools view -hbS output.sam > unsorted_output.bam 38 | - name: sort bam 39 | run: samtools sort -o output.bam unsorted_output.bam 40 | - name: genotyphi 41 | run: python genotyphi.py --mode bam --bam output.bam --ref CT18.fasta --ref_id AL513382.1 --output genotypes_test.txt 42 | # File Final_call Final_call_support Subclade Clade PrimaryClade Support_Subclade Support_Clade Support_PrimaryClade Number of SNPs called QRDR mutations 43 | # output.vcf 4.3.1.2 1.0 4.3.1.2 4 1.0 1.0 77 gyrA-D87G 44 | - name: test output 45 | run: | 46 | ls -lht 47 | resultsfile=$(\ls -t *genotypes_test.txt | head -n 1) 48 | echo "Results file is $resultsfile" 49 | Final_call="4.3.1.2" 50 | PrimaryClade="4" 51 | Final_call_observed=$(tail -n 1 $resultsfile | cut -f 2) 52 | PrimaryClade_observed=$(tail -n 1 $resultsfile | cut -f 6) 53 | [ "$PrimaryClade" == "$PrimaryClade_observed" ] && echo "Primary clade value was as expected, $PrimaryClade_observed" 54 | [ "$Final_call" == "$Final_call_observed" ] && echo "Final call was as expected, $Final_call_observed_observed" 55 | -------------------------------------------------------------------------------- /.github/workflows/ivar.yml: -------------------------------------------------------------------------------- 1 | name: ivar 2 | 3 | on: 4 | push: 5 | branches: [ main, dev ] 6 | pull_request: 7 | branches: [ main, dev ] 8 | 9 | jobs: 10 | functional-test: 11 | runs-on: ubuntu-latest 12 | steps: 13 | - name: install ivar and dependencies 14 | run: conda create -n ivar -c defaults -c bioconda -c conda-forge ivar samtools 15 | - uses: actions/checkout@v2 16 | with: 17 | repository: andersen-lab/ivar 18 | path: ivar 19 | - name: test ivar 20 | run: | 21 | source ${CONDA}/bin/activate ivar 22 | which ivar 23 | ivar version 24 | - name: test ivar trim 25 | run: | 26 | source ${CONDA}/bin/activate ivar 27 | cd $GITHUB_WORKSPACE/ivar/data 28 | ivar trim -b test.bed -i test.sorted.bam -p ivar_test.trimmed -q 15 -m 50 -s 4 29 | [ $(samtools view ivar_test.trimmed.bam | grep "^test" -c) == 2 ] 30 | - name: test ivar variants 31 | run: | 32 | source ${CONDA}/bin/activate ivar 33 | cd $GITHUB_WORKSPACE/ivar/data 34 | samtools mpileup -aa -A -d 600000 -B -Q 0 ivar_test.trimmed.bam | ivar variants -p ivar_test.trimmed.variants -q 20 -t 0.03 -r db/test_ref.fa -g test.gff 35 | [ $(cat ivar_test.trimmed.variants.tsv | grep "^test" -c) == 8 ] 36 | - name: test ivar filtervariants 37 | run: | 38 | source ${CONDA}/bin/activate ivar 39 | cd $GITHUB_WORKSPACE/ivar/data 40 | ivar filtervariants -t 0.5 -p ivar_test.filtered test.filtered.tsv 41 | [ $(cat ivar_test.filtered.tsv | grep "^Consensus_test" -c) == 4 ] 42 | echo "test.filtered.tsv" > filter_files.txt 43 | ivar filtervariants -t 0.5 -p ivar_test.filtered -f filter_files.txt 44 | [ $(cat ivar_test.filtered.tsv | grep "^Consensus_test" -c) == 4 ] 45 | - name: test ivar consensus 46 | run: | 47 | source ${CONDA}/bin/activate ivar 48 | cd $GITHUB_WORKSPACE/ivar/data 49 | samtools mpileup -d 1000 -A -Q 0 test.sorted.bam | ivar consensus -p ivar_test -q 20 -t 0 -m 1 50 | [ $(grep -o "A" ivar_test.fa | grep -c "A") == 93 ] 51 | - name: test ivar getmasked 52 | run: | 53 | source ${CONDA}/bin/activate ivar 54 | cd $GITHUB_WORKSPACE/ivar/data 55 | ivar getmasked -i test.filtered.tsv -b test.bed -f pair_information.tsv -p ivar_test.masked 56 | [ $(cat ivar_test.masked.txt | cut -f3) == "WNV_400_2_LEFT_alt" ] -------------------------------------------------------------------------------- /.github/workflows/kraken.yml: -------------------------------------------------------------------------------- 1 | # This is a basic workflow to help you get started with Actions 2 | 3 | name: kraken 4 | 5 | # Controls when the action will run. Triggers the workflow on push or pull request 6 | # events but only for the main branch 7 | on: 8 | push: 9 | branches: [ main, dev ] 10 | pull_request: 11 | branches: [ main, dev ] 12 | 13 | # A workflow run is made up of one or more jobs that can run sequentially or in parallel 14 | jobs: 15 | # This workflow contains a single job called "build" 16 | build: 17 | # The type of runner that the job will run on 18 | runs-on: ubuntu-latest 19 | 20 | # Steps represent a sequence of tasks that will be executed as part of the job 21 | steps: 22 | - name: Install kraken 23 | run: conda create -y -n kraken -c conda-forge -c bioconda kraken 24 | - uses: actions/checkout@v2 25 | with: 26 | repository: broadinstitute/viral-classify 27 | path: viral-classify 28 | ref: v2.1.10.0 29 | - name: Staging 30 | run: | 31 | echo "${CONDA}/envs/kraken/bin" >> $GITHUB_PATH 32 | mv $GITHUB_WORKSPACE/viral-classify/test/input/TestMetagenomicsViralMix/db ./ 33 | mv $GITHUB_WORKSPACE/viral-classify/test/input/TestSplitReads/in.fastq ./ 34 | - name: Test kraken-build 35 | run: | 36 | which kraken-build 37 | kraken-build --help 38 | kraken-build --version 39 | kraken-build --build --kmer-len 11 --minimizer-len 5 --db db 40 | - name: Test kraken 41 | run: | 42 | which kraken 43 | kraken --version 44 | kraken --help 45 | kraken --db db in.fastq > kraken.out 46 | - name: Test kraken-report 47 | run: | 48 | which kraken-report 49 | kraken-report --version 50 | kraken-report --help 51 | kraken-report --db db --show-zeros kraken.out > kraken.report 52 | grep "root" kraken.report 53 | - name: Test kraken-mpa-report 54 | run: | 55 | which kraken-mpa-report 56 | kraken-mpa-report --version 57 | kraken-mpa-report --help 58 | kraken-mpa-report --db db --show-zeros kraken.out > kraken.mpa-report 59 | grep "d__Viruses" kraken.mpa-report 60 | -------------------------------------------------------------------------------- /.github/workflows/kraken2.yml: -------------------------------------------------------------------------------- 1 | # This is a basic workflow to help you get started with Actions 2 | 3 | name: kraken2 4 | 5 | # Controls when the action will run. Triggers the workflow on push or pull request 6 | # events but only for the main branch 7 | on: 8 | push: 9 | branches: [ main, dev ] 10 | pull_request: 11 | branches: [ main, dev ] 12 | 13 | # A workflow run is made up of one or more jobs that can run sequentially or in parallel 14 | jobs: 15 | # This workflow contains a single job called "build" 16 | build: 17 | # The type of runner that the job will run on 18 | runs-on: ubuntu-latest 19 | 20 | # Steps represent a sequence of tasks that will be executed as part of the job 21 | steps: 22 | - name: Install kraken2 23 | run: conda create -y -n kraken2 -c conda-forge -c bioconda kraken2 24 | - uses: actions/checkout@v2 25 | with: 26 | repository: broadinstitute/viral-classify 27 | path: viral-classify 28 | ref: v2.1.10.0 29 | - name: Staging 30 | run: | 31 | echo "${CONDA}/envs/kraken2/bin" >> $GITHUB_PATH 32 | mv $GITHUB_WORKSPACE/viral-classify/test/input/TestMetagenomicsViralMix/db ./ 33 | mv $GITHUB_WORKSPACE/viral-classify/test/input/TestSplitReads/in.fastq ./ 34 | - name: Test kraken2-build 35 | run: | 36 | which kraken2-build 37 | kraken2-build --help 38 | kraken2-build --version 39 | kraken2-build --build --db db 40 | - name: Test kraken2-inspect 41 | run: | 42 | kraken2-inspect --help 43 | kraken2-inspect --version 44 | kraken2-inspect --db db 45 | - name: Test kraken2 46 | run: | 47 | kraken2 --version 48 | kraken2 --help 49 | kraken2 --db db --report report.txt --report-zero-counts in.fastq 50 | grep "root" report.txt 51 | -------------------------------------------------------------------------------- /.github/workflows/krakenuniq.yml: -------------------------------------------------------------------------------- 1 | # This is a basic workflow to help you get started with Actions 2 | 3 | name: krakenuniq 4 | 5 | # Controls when the action will run. Triggers the workflow on push or pull request 6 | # events but only for the main branch 7 | on: 8 | push: 9 | branches: [ main, dev ] 10 | pull_request: 11 | branches: [ main, dev ] 12 | 13 | # A workflow run is made up of one or more jobs that can run sequentially or in parallel 14 | jobs: 15 | # This workflow contains a single job called "build" 16 | build: 17 | # The type of runner that the job will run on 18 | runs-on: ubuntu-latest 19 | 20 | # Steps represent a sequence of tasks that will be executed as part of the job 21 | steps: 22 | - name: Install krakenuniq 23 | run: conda create -y -n krakenuniq -c conda-forge -c bioconda krakenuniq 24 | - uses: actions/checkout@v2 25 | with: 26 | repository: broadinstitute/viral-classify 27 | path: viral-classify 28 | ref: v2.1.10.0 29 | - name: Staging 30 | run: | 31 | echo "${CONDA}/envs/krakenuniq/bin" >> $GITHUB_PATH 32 | mv $GITHUB_WORKSPACE/viral-classify/test/input/TestMetagenomicsViralMix/db ./ 33 | mv $GITHUB_WORKSPACE/viral-classify/test/input/TestSplitReads/in.fastq ./ 34 | - name: Test krakenuniq-build 35 | run: | 36 | which krakenuniq-build 37 | krakenuniq-build --help 38 | krakenuniq-build --version 39 | krakenuniq-build --build --kmer-len 11 --minimizer-len 5 --db db 40 | - name: Test krakenuniq 41 | run: | 42 | which krakenuniq 43 | krakenuniq --version 44 | krakenuniq --help 45 | krakenuniq --db db in.fastq > krakenuniq.out 46 | - name: Test krakenuniq-report 47 | run: | 48 | which krakenuniq-report 49 | krakenuniq-report --version 50 | krakenuniq-report --help 51 | krakenuniq-report --db db --show-zeros krakenuniq.out > krakenuniq.report 52 | grep "root" krakenuniq.report 53 | - name: Test krakenuniq-mpa-report 54 | run: | 55 | which krakenuniq-mpa-report 56 | krakenuniq-mpa-report --version 57 | krakenuniq-mpa-report --help 58 | krakenuniq-mpa-report --db db --show-zeros krakenuniq.out > krakenuniq.mpa-report 59 | grep "d__Viruses" krakenuniq.mpa-report 60 | -------------------------------------------------------------------------------- /.github/workflows/pangolin.yml: -------------------------------------------------------------------------------- 1 | name: pangolin 2 | 3 | on: [push, pull_request] 4 | 5 | jobs: 6 | run_pangolin_test: 7 | name: Pangolin test on ${{ matrix.os }} 8 | runs-on: ${{ matrix.os }} 9 | strategy: 10 | matrix: 11 | os: ["ubuntu-latest", "macos-latest"] 12 | steps: 13 | - uses: actions/checkout@v2 14 | with: 15 | repository: cov-lineages/pangolin 16 | ref: v2.0.8 17 | path: pangolin 18 | - uses: conda-incubator/setup-miniconda@v2 19 | with: 20 | environment-file: pangolin/environment.yml 21 | activate-environment: pangolin 22 | channels: conda-forge,bioconda,defaults 23 | mamba-version: "*" 24 | - name: Install pangolin 25 | shell: bash -l {0} 26 | run: | 27 | cd $GITHUB_WORKSPACE/pangolin 28 | pip install -e . 29 | - name: Check pangolin version 30 | shell: bash -l {0} 31 | run: pangolin --version 32 | - name: Run pangolin with test data 33 | shell: bash -l {0} 34 | run: pangolin $GITHUB_WORKSPACE/pangolin/pangolin/test/test_seqs.fasta 2>&1 | tee pangolin.log 35 | # add more tests here -------------------------------------------------------------------------------- /.github/workflows/prokka.yml: -------------------------------------------------------------------------------- 1 | # This is a basic workflow to help you get started with Actions 2 | 3 | name: prokka 4 | 5 | # Controls when the action will run. Triggers the workflow on push or pull request 6 | # events but only for the main branch 7 | on: 8 | push: 9 | branches: [ main, dev ] 10 | pull_request: 11 | branches: [ main, dev ] 12 | 13 | # A workflow run is made up of one or more jobs that can run sequentially or in parallel 14 | jobs: 15 | # This workflow contains a single job called "build" 16 | build: 17 | # The type of runner that the job will run on 18 | runs-on: ubuntu-latest 19 | 20 | # Steps represent a sequence of tasks that will be executed as part of the job 21 | steps: 22 | - name: apt 23 | run: sudo apt-get install --no-install-recommends tree libdb-dev libbio-perl-perl libxml-simple-perl 24 | # Checks-out prokka under $GITHUB_WORKSPACE, so your job can access it 25 | - uses: actions/checkout@v2 26 | with: 27 | repository: tseemann/prokka 28 | path: prokka 29 | - name: directory structure 30 | run: tree -d $GITHUB_WORKSPACE 31 | - name: ls bin dir 32 | run: ls -lh $GITHUB_WORKSPACE/prokka/bin $GITHUB_WORKSPACE/prokka/binaries/linux 33 | - name: test prokka 34 | run: | 35 | export PATH=$GITHUB_WORKSPACE/prokka/bin:$PATH 36 | export PATH=$GITHUB_WORKSPACE/prokka/binaries/linux:$PATH 37 | cd $GITHUB_WORKSPACE/prokka 38 | which prokka 39 | which makeblastdb 40 | prokka --version 41 | prokka --help 42 | ! prokka --doesnotexist 43 | prokka --depends 44 | prokka --setupdb 45 | prokka --listdb 46 | prokka --cpus 2 --outdir asm --prefix asm test/plasmid.fna 47 | grep '>' asm/asm.fna 48 | prokka --cleandb 49 | 50 | -------------------------------------------------------------------------------- /.github/workflows/quast.yml: -------------------------------------------------------------------------------- 1 | name: quast 2 | 3 | on: 4 | push: 5 | branches: [ main, dev ] 6 | pull_request: 7 | branches: [ main, dev ] 8 | 9 | jobs: 10 | build: 11 | runs-on: ubuntu-latest 12 | strategy: 13 | matrix: 14 | python-version: [3.5, 3.6, 3.7, 3.8] 15 | 16 | steps: 17 | - uses: actions/checkout@v2 18 | - name: Set up Python ${{ matrix.python-version }} 19 | uses: actions/setup-python@v2 20 | with: 21 | python-version: ${{ matrix.python-version }} 22 | - name: apt 23 | run: | 24 | sudo apt-get install tree 25 | - uses: actions/checkout@v2 26 | with: 27 | repository: ablab/quast 28 | path: quast 29 | - name: Install quast 30 | run: | 31 | cd $GITHUB_WORKSPACE/quast 32 | ./setup.py install 33 | - name: test 1 34 | run: | 35 | wget quast.sf.net/test_data.tar.gz && tar xzf test_data.tar.gz 36 | tree test_data 37 | which quast.py 38 | quast.py --threads 1 --test 39 | -------------------------------------------------------------------------------- /.github/workflows/shovill.yml: -------------------------------------------------------------------------------- 1 | name: shovill 2 | 3 | on: 4 | push: 5 | branches: [main, dev] 6 | pull_request: 7 | branches: [main, dev] 8 | 9 | jobs: 10 | build: 11 | runs-on: ${{ matrix.os }} 12 | strategy: 13 | matrix: 14 | os: [ubuntu-18.04, ubuntu-20.04] 15 | 16 | steps: 17 | - name: conda 18 | run: | 19 | conda create -y -n env_shovill -c bioconda -c conda-forge shovill 20 | - name: Staging 21 | run: | 22 | echo "${CONDA}/envs/env_shovill/bin" >> $GITHUB_PATH 23 | - name: download test data 24 | run: | 25 | # Download test data from ENA 26 | wget ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR112/094/SRR11283994/SRR11283994_1.fastq.gz 27 | wget ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR112/094/SRR11283994/SRR11283994_2.fastq.gz 28 | 29 | # Download test data from SPAdes repo 30 | wget https://github.com/ablab/spades/raw/spades_3.14.1/assembler/test_dataset/ecoli_1K_1.fq.gz 31 | wget https://github.com/ablab/spades/raw/spades_3.14.1/assembler/test_dataset/ecoli_1K_2.fq.gz 32 | 33 | - name: create faulty data 34 | run: | 35 | # Remove last 5 characters from R1 file, creating a mismatch between lengths of sequence and quality score in that file 36 | zcat SRR11283994_1.fastq.gz | sed '$ s/.....$//' | gzip > SRR11283994_truncated_1.fastq.gz 37 | 38 | # Remove read 26 from the R1 file only 39 | zcat SRR11283994_1.fastq.gz | awk 'NR < 100 && NR > 103 {print $0}' | gzip > SRR11283994_missing_1.fastq.gz 40 | 41 | - name: test shovill-skesa, shovill-spades and shovill-megahit 42 | run: | 43 | echo "which shovill" 44 | which shovill 45 | 46 | echo "shovill" 47 | ! shovill 48 | 49 | echo "shovill -h" 50 | shovill -h 51 | 52 | echo "shovill --help" 53 | shovill --help 54 | 55 | echo "shovill --version" 56 | shovill --version 57 | 58 | echo "shovill --doesnotexist" 59 | ! shovill --doesnotexist 60 | 61 | echo "Start assembly using SKESA" 62 | shovill --R1 SRR11283994_1.fastq.gz --R2 SRR11283994_2.fastq.gz --assembler skesa --outdir skesa_standard 63 | grep -c '>' skesa_standard/contigs.fa 64 | 65 | echo "Start assembly using SPAdes" 66 | shovill --R1 ecoli_1K_1.fq.gz --R2 ecoli_1K_2.fq.gz --assembler spades --outdir spades_standard 67 | grep -c '>' spades_standard/contigs.fa 68 | 69 | echo "Start assembly using Megahit" 70 | shovill --R1 ecoli_1K_1.fq.gz --R2 ecoli_1K_2.fq.gz --assembler megahit --outdir megahit_standard 71 | grep -c '>' megahit_standard/contigs.fa 72 | 73 | echo "test whether Shovill catches a truncated quality score" 74 | ! shovill --R1 SRR11283994_truncated_1.fastq.gz --R2 SRR11283994_2.fastq.gz --assembler skesa --outdir skesa_truncated 75 | ! grep -c '>' skesa_truncated/contigs.fa 76 | 77 | echo "test whether Shovill catches when one half of a read pair is missing" 78 | ! shovill --R1 SRR11283994_missing_1.fastq.gz --R2 SRR11283994_2.fastq.gz --assembler skesa --outdir skesa_missing 79 | ! grep -c '>' skesa_missing/contigs.fa 80 | -------------------------------------------------------------------------------- /.github/workflows/skesa.yml: -------------------------------------------------------------------------------- 1 | name: skesa 2 | 3 | on: 4 | push: 5 | branches: [main, dev] 6 | pull_request: 7 | branches: [main, dev] 8 | 9 | jobs: 10 | build: 11 | runs-on: ${{ matrix.os }} 12 | strategy: 13 | matrix: 14 | os: [ubuntu-20.04] 15 | boost-version: [1.67, 1.71] 16 | 17 | steps: 18 | - name: apt 19 | run: | 20 | apt-cache madison libboost-all-dev 21 | sudo apt-get install libboost${{ matrix.boost-version }}-all-dev 22 | - name: check out SKESA 23 | uses: actions/checkout@v2 24 | with: 25 | repository: ncbi/SKESA 26 | path: SKESA 27 | - name: compile 28 | run: | 29 | cd ${GITHUB_WORKSPACE}/SKESA 30 | make 31 | - name: download test data 32 | run: | 33 | cd ${GITHUB_WORKSPACE}/SKESA 34 | ls -ltrh 35 | 36 | # Download test data from SPAdes repo 37 | wget ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR112/094/SRR11283994/SRR11283994_1.fastq.gz 38 | wget ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR112/094/SRR11283994/SRR11283994_2.fastq.gz 39 | - name: create faulty data 40 | run: | 41 | cd ${GITHUB_WORKSPACE}/SKESA 42 | 43 | # Remove last 5 characters from R1 file, creating a mismatch between lengths of sequence and quality score in that file 44 | zcat SRR11283994_1.fastq.gz | sed '$ s/.....$//' | gzip > SRR11283994_truncated_1.fastq.gz 45 | 46 | # Remove read 26 from the R1 file only 47 | zcat SRR11283994_1.fastq.gz | awk 'NR < 100 || NR > 103 {print $0}' | gzip > SRR11283994_missing_1.fastq.gz 48 | - name: test 49 | run: | 50 | cd ${GITHUB_WORKSPACE}/SKESA 51 | 52 | echo "which skesa" 53 | which ./skesa 54 | 55 | echo "./skesa gives exit code 1" 56 | ! ./skesa # raises an error 57 | 58 | echo "./skesa --help" 59 | ./skesa --help 60 | 61 | echo "./skesa --version" 62 | ./skesa --version 63 | 64 | echo "./skesa --doesnotexist" 65 | ! ./skesa --doesnotexist 66 | 67 | echo "Start assembly" 68 | ./skesa --fastq SRR11283994_1.fastq.gz,SRR11283994_2.fastq.gz > asm_local.fa 69 | grep -c '>' asm_local.fa 70 | 71 | echo "small test using --sra_run on SRR11283994" 72 | ./skesa --sra_run SRR11283994 > asm_sra.fa 73 | grep -c '>' asm_sra.fa 74 | 75 | echo "check whether assemblies are identical" 76 | diff asm_local.fa asm_sra.fa 77 | 78 | echo "test whether SKESA catches a truncated quality score" 79 | ./skesa --fastq SRR11283994_truncated_1.fastq.gz,SRR11283994_2.fastq.gz > asm_truncated.fa 80 | grep -c '>' asm_truncated.fa 81 | 82 | echo "test whether truncated assembly is different from normal assembly" 83 | diff asm_local.fa asm_truncated.fa 84 | 85 | echo "test whether SKESA catches when the forward read of a read pair is missing" 86 | ./skesa --fastq SRR11283994_missing_1.fastq.gz,SRR11283994_2.fastq.gz > asm_missing.fa 87 | grep -c '>' asm_missing.fa 88 | 89 | echo "test whether a missing read gives a different assembly" 90 | ! diff asm_local.fa asm_missing.fa 91 | -------------------------------------------------------------------------------- /.github/workflows/trycycler.yml: -------------------------------------------------------------------------------- 1 | name: trycycler 2 | 3 | on: 4 | push: 5 | branches: [main, dev] 6 | pull_request: 7 | branches: [main, dev] 8 | schedule: 9 | - cron: '55 5 1 * *' 10 | 11 | jobs: 12 | build: 13 | runs-on: ${{ matrix.os }} 14 | strategy: 15 | matrix: 16 | os: [ubuntu-20.04] #arch and centos both unavailable https://github.com/actions/virtual-environments#available-environments 17 | python-version: [3.6, 3.7, 3.8, 3.9] 18 | 19 | steps: 20 | - name: initialize github actions test environment 21 | uses: actions/checkout@v2 22 | 23 | - name: setup python ${{ matrix.python-version }} 24 | uses: actions/setup-python@v2 25 | with: 26 | python-version: ${{ matrix.python-version }} 27 | architecture: x64 28 | 29 | - name: update pip package manager and the setuptools and wheel libraries 30 | run: | 31 | python -m pip install -U pip setuptools wheel 32 | # GitHub does not always have up-to-date pip, and 33 | # installation from pre-built binary archives (with pip) 34 | # will complain if pip is out of date. 35 | # Up-to-date setuptools and wheel libraries 36 | # enable installation from source archives. 37 | 38 | - name: report pip, setuptools, and wheel versions 39 | run: | 40 | pip --version 41 | _ver=$(python -c 'import setuptools; print(setuptools.__version__)') 42 | echo "setuptools version $_ver" 43 | _ver=$(python -c 'import wheel; print(wheel.__version__)') 44 | echo "wheel version $_ver" 45 | 46 | - name: report environment variables for debugging 47 | run: | 48 | printf '%s\n' \ 49 | "Workflow ${GITHUB_WORKFLOW}" \ 50 | "Action Number ${GITHUB_ACTION}" \ 51 | "Run ID ${GITHUB_RUN_ID}" \ 52 | "Run Number ${GITHUB_RUN_NUMBER}" \ 53 | "Workspace ${GITHUB_WORKSPACE}" 54 | 55 | - name: report python version 56 | run: | 57 | python -c 'import sys; print(sys.version)' 58 | 59 | - name: checkout trycycler 60 | uses: actions/checkout@v2 61 | with: 62 | repository: rrwick/Trycycler 63 | path: github_src_trycycler 64 | ref: v0.3.3 65 | 66 | - name: install dependencies 67 | run: | 68 | sudo apt-get install mash minimap2 muscle r-base 69 | sudo R -e "install.packages(c('ape', 'phangorn'), \ 70 | repos = 'http://cran.us.r-project.org')" 71 | 72 | - name: report dependency versions 73 | run: | 74 | _ver=$(mash --version) 75 | echo "mash ${_ver}" 76 | _ver=$(minimap2 --version) 77 | echo "minimap2 ${_ver}" 78 | muscle -version 79 | echo 'sessionInfo()' | R --vanilla 80 | 81 | - name: install trycycler 82 | run: | 83 | cd ${GITHUB_WORKSPACE}/github_src_trycycler 84 | pip install -e . 85 | 86 | - name: test trycycler 87 | run: | 88 | # trycycler #prints usage but exit code 1 89 | trycycler -h 90 | trycycler --help 91 | trycycler --version 92 | cd ${GITHUB_WORKSPACE}/github_src_trycycler/test 93 | python test_cluster.py 94 | python test_software.py 95 | echo 'INFO: tests complete' 96 | -------------------------------------------------------------------------------- /.github/workflows/unicycler.yml: -------------------------------------------------------------------------------- 1 | name: unicycler 2 | 3 | on: 4 | push: 5 | branches: [main, dev] 6 | pull_request: 7 | branches: [main, dev] 8 | schedule: 9 | - cron: '55 5 1 * *' 10 | 11 | jobs: 12 | build: 13 | runs-on: ${{ matrix.os }} 14 | strategy: 15 | matrix: 16 | os: [ubuntu-20.04] #arch and centos both unavailable https://github.com/actions/virtual-environments#available-environments 17 | python-version: [3.6, 3.7, 3.8] #3.9 not working in conda 18 | 19 | steps: 20 | - name: initialize github actions test environment 21 | uses: actions/checkout@v2 22 | 23 | - name: report environment variables for debugging 24 | run: | 25 | printf '%s\n' \ 26 | "Workflow ${GITHUB_WORKFLOW}" \ 27 | "Action Number ${GITHUB_ACTION}" \ 28 | "Run ID ${GITHUB_RUN_ID}" \ 29 | "Run Number ${GITHUB_RUN_NUMBER}" \ 30 | "Workspace ${GITHUB_WORKSPACE}" 31 | 32 | - name: setup workspace 33 | run: | 34 | mkdir -v "${GITHUB_WORKSPACE}/Downloads" 35 | 36 | - name: setup conda environment 37 | uses: conda-incubator/setup-miniconda@v2 38 | with: 39 | activate-environment: my_env 40 | channels: conda-forge,bioconda,defaults 41 | channel-priority: true 42 | mamba-version: "*" 43 | python-version: ${{ matrix.python-version }} 44 | 45 | - name: report conda version, mamba version, and config info 46 | shell: bash -l {0} 47 | run: | 48 | conda activate my_env 49 | conda info 50 | conda list 51 | conda config --show-sources 52 | conda config --show 53 | mamba --version 54 | 55 | - name: install unicycler with mamba 56 | shell: bash -l {0} 57 | run: | 58 | conda activate my_env 59 | mamba install unicycler=0.4.8 -y 60 | 61 | - name: test unicycler 62 | shell: bash -l {0} 63 | run: | 64 | conda activate my_env 65 | export TERM=xterm 66 | # unicycler #Error: Process completed with exit code 1. 67 | unicycler -h 68 | unicycler --help 69 | unicycler --help_all 70 | unicycler --version 71 | _ver=$(unicycler --version | awk '{print $2}') 72 | mkdir Downloads/github_src_unicycler 73 | git config --global advice.detachedHead false 74 | git clone --branch "${_ver}" https://github.com/rrwick/Unicycler.git \ 75 | Downloads/github_src_unicycler 76 | cd Downloads/github_src_unicycler 77 | python test/test_assembly_graph.py 78 | python test/test_assembly_graph_copy_depth.py 79 | python test/test_blast_func.py 80 | python test/test_dependencies.py 81 | python test/test_semi_global_alignment.py 82 | python test/test_spades_func.py 83 | echo 'INFO: tests complete' 84 | -------------------------------------------------------------------------------- /CITATION.cff: -------------------------------------------------------------------------------- 1 | type: article 2 | title: 'Software testing in microbial bioinformatics: a call to action' 3 | authors: 4 | - family-names: van der Putten 5 | given-names: Boas C.L. 6 | - family-names: Mendes 7 | given-names: C I 8 | - family-names: Talbot 9 | given-names: Brooke M 10 | - family-names: de Korne-Elenbaas 11 | given-names: Jolinda 12 | - family-names: Mamede 13 | given-names: Rafael 14 | - family-names: Vila-Cerqueira 15 | given-names: Pedro 16 | - family-names: Coelho 17 | given-names: Luis Pedro 18 | - family-names: Gulvik 19 | given-names: Christopher A 20 | - family-names: Katz 21 | given-names: Lee S 22 | - family-names: participants 23 | given-names: The ASM NGS 2020 Hackathon 24 | journal: Microbial Genomics 25 | year: '2022' 26 | publisher: 27 | name: Microbiology Society 28 | volume: '8' 29 | issue: '3' 30 | keywords: 31 | - software testing 32 | - continuous integration 33 | - computational biology 34 | abstract: Computational algorithms have become an essential component of research, 35 | with great efforts by the scientific community to raise standards on development 36 | and distribution of code. Despite these efforts, sustainability and reproducibility 37 | are major issues since continued validation through software testing is still not 38 | a widely adopted practice. Here, we report seven recommendations that help researchers 39 | implement software testing in microbial bioinformatics. We have developed these 40 | recommendations based on our experience from a collaborative hackathon organised 41 | prior to the American Society for Microbiology Next Generation Sequencing (ASM NGS) 42 | 2020 conference. We also present a repository hosting examples and guidelines for 43 | testing, available from https://github.com/microbinfie-hackathon2020/CSIS. 44 | issn: 2057-5858 45 | doi: 10.1099/mgen.0.000790 46 | url: https://www.microbiologyresearch.org/content/journal/mgen/10.1099/mgen.0.000790 47 | identifiers: 48 | - type: url 49 | value: https://doi.org/https://doi.org/10.1099/mgen.0.000790 -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # CONTRIBUTING 2 | 3 | The hackathon has been completed and our collaborative manuscript has been published in Microbial Genomics (https://doi.org/10.1099/mgen.0.000790). However, we are still welcoming contributions to this repository. 4 | 5 | We especially welcome contributions of two kinds: 6 | 1. Adding or improving guidelines for software testing. 7 | 2. Testing workflows which serve as good examples for the microbial bioinformatics community. 8 | 9 | For all other contributions, we will first assess whether these fit the intended scope of this repository. 10 | 11 | 12 | 13 | ## THE INFORMATION BELOW DESCRIBES CONTRIBUTING GUIDELINES DURING THE HACKATHON (DECEMBER 2020). CURRENT CONTRIBUTING GUIDELINES ARE LISTED ABOVE 14 | *During the hackathon, contributions focused on implementing software testing workflows using GitHub Actions* 15 | 16 | 17 | ## Workflows 18 | 19 | Your team can create a workflow in the `.github/workflows` folder. 20 | It needs to be a file with extension `.yml`. 21 | It should be named after the software being tested. 22 | The `name` parameter in the file needs to be named after the software too. 23 | Structure of the workflow is described [here](https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/introduction-to-github-actions). Some examples can be found in [EXAMPLES](EXAMPLES.md). 24 | 25 | ## Pull request etiquette 26 | 27 | Please fork the repo, create a separate branch in your repo. 28 | Make changes. 29 | When changes are done, create a pull request from your new branch with the dev branch on this repo. 30 | 31 | From time to time, the owners of this repo will merge the dev branch with the main branch. 32 | 33 | Please see [Read Git Flow](https://leanpub.com/git-flow/read#leanpub-auto-git-flow-example) for more information, 34 | or also please see this depiction for our workflow: https://leanpub.com/site_images/git-flow/git-workflow-release-cycle-3release.png 35 | 36 | ### Basic test ideas 37 | 38 | _These are ideas and are not mandatory_ 39 | 40 | 1. See if running the command with `--version` gives a version number and exits with 0 41 | 2. See if running the command with `--help` gives a usage with stdout and exits with 0 42 | 3. Run the most basic command and compare it with a simple expected value. 43 | 4. More ideas can be found in this publication: https://academic.oup.com/gigascience/article/2/1/2047-217X-2-15/2656133 44 | 45 | ## Code of conduct 46 | 47 | [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-v2.0%20adopted-ff69b4.svg)](code_of_conduct.md) 48 | 49 | To stimulate a positive environment for all contributors, we have adopted v2.0 of the Contributor Covenant: https://www.contributor-covenant.org/version/2/0/code_of_conduct/. 50 | 51 | ## Practicalities 52 | 53 | Please see [HACKATHON.md](HACKATHON.md). 54 | 55 | ## Git flow depiction 56 | 57 | ![Git flow](https://leanpub.com/site_images/git-flow/git-workflow-release-cycle-3release.png) 58 | -------------------------------------------------------------------------------- /EXAMPLES.md: -------------------------------------------------------------------------------- 1 | # Examples 2 | 3 | This file contains two examples of GitHub Actions workflows: 4 | 5 | 1. An install of Prokka on Ubuntu, with some basic command tests and a small annotation command 6 | 2. Four installs of Quast using different versions of Python, using the `--test` command of Quast 7 | 8 | ## Prokka 9 | 10 | In [.github/workflows/](.github/workflows), examples are provided. If we look at the example of Prokka (`prokka.yml`), the following sections are important to study: 11 | 12 | ### Name of the workflow 13 | 14 | From [this line](.github/workflows/prokka.yml#L3): 15 | 16 | ``` 17 | name: prokka 18 | ``` 19 | 20 | This section defines the name of the software you want to test. This needs to match the name of the YAML file (without the `.yml` extension). 21 | 22 | ### Events triggering the tests 23 | 24 | From [this line](.github/workflows/prokka.yml#L5): 25 | 26 | ``` 27 | # Controls when the action will run. Triggers the workflow on push or pull request 28 | # events but only for the main branch 29 | on: 30 | push: 31 | branches: [ main, dev ] 32 | pull_request: 33 | branches: [ main, dev ] 34 | ``` 35 | 36 | This section describes the events that trigger automatic testing, and on which branches this should be done. In this case, every time an update gets pushed or a pull request is opened on the main or dev branches, the automatic tests will be performed. 37 | 38 | ### Specifying jobs within a workflow 39 | 40 | From [this line](.github/workflows/prokka.yml#L13): 41 | 42 | ``` 43 | # A workflow run is made up of one or more jobs that can run sequentially or in parallel 44 | jobs: 45 | # This workflow contains a single job called "build" 46 | build: 47 | # The type of runner that the job will run on 48 | runs-on: ubuntu-latest 49 | ``` 50 | 51 | This section specifies which jobs should be run within this workflow. In this case, a single job (`build`) is specified. This job runs on the latest version of Ubuntu, hosted by GitHub itself. 52 | 53 | ### Specifying steps within a job 54 | 55 | From [this line](.github/workflows/prokka.yml#L20): 56 | 57 | ``` 58 | # Steps represent a sequence of tasks that will be executed as part of the job 59 | steps: 60 | - name: apt 61 | run: sudo apt-get install tree libdb-dev libbio-perl-perl libxml-simple-perl 62 | # Checks-out prokka under $GITHUB_WORKSPACE, so your job can access it 63 | - uses: actions/checkout@v2 64 | with: 65 | repository: tseemann/prokka 66 | path: prokka 67 | - name: directory structure 68 | run: tree -d $GITHUB_WORKSPACE 69 | - name: ls bin dir 70 | run: ls -lh $GITHUB_WORKSPACE/prokka/bin $GITHUB_WORKSPACE/prokka/binaries/linux 71 | - name: test prokka 72 | run: | 73 | export PATH=$GITHUB_WORKSPACE/prokka/bin:$PATH 74 | export PATH=$GITHUB_WORKSPACE/prokka/binaries/linux:$PATH 75 | cd $GITHUB_WORKSPACE/prokka 76 | which prokka 77 | which makeblastdb 78 | prokka --version 79 | prokka --help 80 | ! prokka --doesnotexist 81 | prokka --depends 82 | prokka --setupdb 83 | prokka --listdb 84 | prokka --cpus 2 --outdir asm --prefix asm test/plasmid.fna 85 | grep '>' asm/asm.fna 86 | prokka --cleandb 87 | ``` 88 | 89 | This is where the actual testing happens. Within the `build` job, these five steps are defined. The first couple of steps are used to install dependencies through apt (`name: apt`), download the Prokka repository to the virtual machine (`uses: actions/checkout@v2`), print the directory structure (`name: directory structure`), check whether the correct executables are present (`name: ls bin dir`) and finally performs the test (`name: test prokka`). 90 | 91 | In this example, the tests that are included are: 92 | 93 | - Whether prokka and makeblastdb can be found in PATH (using `which`) 94 | - Whether `prokka --version` and `prokka --help` work and exit without raising an error 95 | - Whether a nonsense option does raise an error (`! prokka --doesnotexist`) 96 | - Whether different Prokka commands work (`--depends`, `--setupdb`, `--listdb`, `--cleandb`) 97 | - Whether a small test file can be annotated `prokka --cpus 2 --outdir asm --prefix asm test/plasmid.fna; grep '>' asm/asm.fna` 98 | 99 | As can be seen in the Actions tab of the CSIS GitHub repo, the Prokka test works as expected and raises no errors. Example tests are taken from https://github.com/tseemann/prokka/blob/master/.travis.yml 100 | 101 | ## Quast 102 | 103 | ### Name and events 104 | 105 | From [this line](.github/workflows/quast.yml#L1): 106 | 107 | ``` 108 | name: quast 109 | 110 | on: 111 | push: 112 | branches: [ main, dev ] 113 | pull_request: 114 | branches: [ main, dev ] 115 | ``` 116 | 117 | This is the same as for prokka: `name: quast` specifies the name of the workflow and `on: ` specifies which events should trigger Actions. 118 | 119 | ### Specifying a job using a matrix 120 | 121 | From [this line](.github/workflows/quast.yml#L9): 122 | 123 | ``` 124 | jobs: 125 | build: 126 | runs-on: ubuntu-latest 127 | strategy: 128 | matrix: 129 | python-version: [3.5, 3.6, 3.7, 3.8] 130 | ``` 131 | 132 | This is mostly identical to the prokka example above, except for the `strategy: `. In this case, we define a matrix of Python versions to test the tool on. Also see https://docs.github.com/en/free-pro-team@latest/actions/guides/building-and-testing-python. 133 | 134 | ### Specifying steps within the build job 135 | 136 | From [this line](.github/workflows/quast.yml#16): 137 | 138 | ``` 139 | steps: 140 | - uses: actions/checkout@v2 141 | - name: Set up Python ${{ matrix.python-version }} 142 | uses: actions/setup-python@v2 143 | with: 144 | python-version: ${{ matrix.python-version }} 145 | - name: apt 146 | run: | 147 | sudo apt-get install tree 148 | - uses: actions/checkout@v2 149 | with: 150 | repository: ablab/quast 151 | path: quast 152 | - name: Install quast 153 | run: | 154 | cd $GITHUB_WORKSPACE/quast 155 | ./setup.py install 156 | - name: test 1 157 | run: | 158 | wget quast.sf.net/test_data.tar.gz && tar xzf test_data.tar.gz 159 | tree test_data 160 | which quast.py 161 | quast.py --threads 1 --test 162 | ``` 163 | 164 | Again, largely similar to the prokka example above. The largest difference is in the first step, where in this case we checkout the Python versions defined in the matrix before. To check out Python versions, use the predefined `setup-python` action from https://github.com/marketplace/actions/setup-python. 165 | 166 | The steps thereafter are pretty similar: we install tree through apt (`name: apt`, to get an idea of the repository structure, mostly needed for debugging). Then we check out the Quast GitHub repo using the predefined checkout action (from https://github.com/marketplace/actions/checkout). We then install quast using their `setup.py` script (`name: Install quast`). Finally, we download the Quast test data and run their own tests (`name: test 1`). 167 | 168 | As we have defined a matrix of Python versions, this job is repeated for Python 3.5, 3.6, 3.7 and 3.8. It's also possible to define multiple matrices (e.g. Python versions and operating systems), which then get combined. 169 | -------------------------------------------------------------------------------- /HACKATHON.md: -------------------------------------------------------------------------------- 1 | # Hackathon 2 | 3 | ## Goals 4 | 5 | 1. Introduce automated tests for bioinformatic tools. 6 | 2. Educate ourselves in unit testing, continuous integration (CI), and YAML markup language. 7 | 3. Network with other bioinformaticians. 8 | 4. Create a community resource for identifying software that can pass unit tests. 9 | 5. Publish. 10 | 11 | ## Program 12 | 13 | ### At a glance 14 | 15 | * Register at https://forms.gle/rSSkQZv3pZXEAh3a7 16 | 17 | **NB: This is a tentative planning and might be subject to change.** 18 | 19 | Tentative dates: December 2nd to 4th (Wed-Fri), 2020. ASM NGS will take place from 7 to 11 December (Mon-Fri). 20 | We adhere to the same timezone as the ASM NGS conference itself, meaning the default timezone will be Eastern Standard Time (GMT -5:00). 21 | However, we will try to organize the schedule in such a way that takes into account timezones of all participants. 22 | 23 | The main program will be in Eastern Standard Time, but individual groups can work in local timezones. 24 | 25 | ### December 2nd 26 | 27 | **Orientation, team building, and project formation.** 28 | 29 | _Times are in Eastern Time to coincide with ASM NGS_ 30 | 31 | | time | what | presentor | 32 | |--------|------|-----------| 33 | | 9am | Orientation - the hackathon concept, Zoom, Discord | Lee Katz | 34 | | 9:30am | Go through an example | Boas van der Putten | 35 | | 10am | Team formation, network with team members, choose team captain, choose software for unit testing | via Discord | 36 | | 10:30am| Record team names and team members on `TEAMS.md` | Individual groups | 37 | | 11am | Individual team discussions and logistics | Discord | 38 | | 12pm | close | - | 39 | 40 | After hours: contact Lee Katz and Boas van der Putten 41 | * New team formation 42 | * Questions 43 | * etc 44 | 45 | ### December 3rd 46 | 47 | **Check in** 48 | 49 | _Times are in Eastern Time to coincide with ASM NGS_ 50 | 51 | | time | what | presentor | 52 | |--------|------|-----------| 53 | | 9am | Day 2 introduction | Boas van der Putten | 54 | | 9:15am | Keynote: _The why and the how of testing scientific software_ | Luis Pedro Coelho | 55 | | 10am | Team updates | team captains | 56 | | 11:30am| Closeout | Lee Katz | 57 | 58 | ### December 4th 59 | 60 | **Wrap up** 61 | 62 | _Times are in Eastern Time to coincide with ASM NGS_ 63 | 64 | | time | what | presentor | 65 | |--------|------|-----------| 66 | | 9am | Overview of contributions | Boas van der Putten | 67 | | 9:30am | Presentations (5 min for presentation + 5 min for challenges faced) | Team captains | 68 | | 11am | Discussion of future plans | Lee Katz | 69 | | 12pm | Happy hour | In Discord | 70 | 71 | ## Teams 72 | 73 | To stimulate interaction, we will subdivide participants into teams of 3-4 people. 74 | Ideally, these teams will cover different levels of experience with Git, GitHub Actions, and software testing. 75 | This way, every team can contribute to testing software. 76 | 77 | To register your team, please edit and submit a pull request to [TEAMS.md](/TEAMS.md) according to the [contribution etiquette](/CONTRIBUTING.md#pull-request-etiquette). 78 | 79 | ## Communication 80 | 81 | All links to Discord and Zoom will be emailed separately to help avoid "Zoom bombing." 82 | 83 | To register, please go to https://forms.gle/rSSkQZv3pZXEAh3a7 84 | 85 | * We will use a Discord server, to be determined. 86 | Each team will have a channel, named after their team. 87 | Although Discord is encouraged, each team can decide if there is a different platform that they would like to set up for themselves. 88 | * To communicate new pull requests, please start with a new issue in GitHub. 89 | * The hackathon itself will be held over Zoom. 90 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Lee Katz 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Software testing in microbial bioinformatics: a call to action 2 | 3 | :detective: Code Safety Inspection Service 4 | 5 | ## The CSIS 6 | 7 | Prior to the American Society for Microbiology Conference on Rapid Applied Microbial Next-Generation Sequencing and Bioinformatic Pipelines held in December 2020, we organized a collaborative three-day “Hackathon” and brought together more than twenty researchers in the field of microbial bioinformatics from five continents. 8 | The goal of our Hackathon was to explore how to employ software testing in microbial bioinformatics. 9 | 10 | The ASM NGS 2020 Hackathon aimed to promote the uptake of testing practices and engage the community in its adoption for public health. 11 | This repository is an open-source project that gathers guidance, guidelines and examples for software testing for microbial bioinformatics researchers. 12 | 13 | ## Why Software Testing 14 | 15 | Computational algorithms have become an essential component of research, with great efforts of the scientific community to raise standards on development and distribution of code. 16 | Despite these efforts, sustainability and reproducibility are major issues since continued validation through software testing is still not a widely-adopted practice. 17 | 18 | Based on the experiences from our Hackathon, we developed a set of seven recommendations for researchers seeking to improve the quality and reproducibility of their analyses through software testing. 19 | We propose collaborative software testing as an opportunity to continuously engage software users, developers, and students to unify scientific work across domains. 20 | 21 | ## Our Aim 22 | 23 | In the field of microbial bioinformatics, good software engineering practices are not widely adopted (yet). Many microbial bioinformaticians start out as (micro)biologists and subsequently learn how to code. 24 | Without abundant formal training, a lot of education about good software engineering practices comes down to an exchange of information within the microbial bioinformatics community. 25 | That is also where we aim to position our repository: as a resource that could help microbial bioinformaticians get started with software testing if they have not had formal training. 26 | 27 | ## Our Recommendations 28 | 29 | As automated software testing remains underused in scientific software, our set of recommendations not only ensures appropriate effort can be invested into producing a high quality and robust software, but also increases engagement in its sustainability. 30 | 31 | Here we propose seven recommendations that should be followed during software development. 32 | 33 | ### 1. Establish software needs and testing goals 34 | 35 | Manually testing the functionality of a tool is feasible in early development, but can become laborious as software matures. 36 | We recommend: 37 | 38 | - Developers establish software needs and testing goals during planning and designing stages to ensure an efficient testing structure; 39 | - A minimal test set should address the validation of core components or the program as a whole (Blackbox testing) and gradually progress toward verification of key functions which can accommodate code changes over time (Whitebox testing). 40 | 41 | The following [table](#Table-1:-Overview-of-testing-approaches) provides an overview of testing methodologies and can serve as a guide to developers that aim to implement testing practices. 42 | 43 | #### Table 1: Overview of testing approaches 44 | 45 | 46 | 47 | 48 | 49 | 50 | 52 | 53 | 54 | 55 | 56 | 58 | 59 | 60 | 63 | 65 | 67 | 68 | 69 | 71 | 73 | 75 | 76 | 77 | 79 | 81 | 83 | 84 | 85 | 87 | 89 | 91 | 92 | 93 | 95 | 97 | 99 | 100 | 101 | 104 | 105 | 106 | 108 | 110 | 112 | 113 | 114 | 116 | 118 | 120 | 121 | 122 | 124 | 126 | 128 | 129 | 130 | 133 | 134 | 135 | 137 | 139 | 141 | 142 | 143 | 145 | 147 | 149 | 150 | 151 | 153 | 155 | 157 | 158 |
51 | NameDescriptionExample
57 | Installation testing: can the software be invoked on different setups?
Installation testing 64 | Can the software be installed on different platforms? 66 | Test whether Software X can be installed using apt-get, pip, conda and from source.
70 | Configuration testing 72 | With which dependencies can the software be used? 74 | Test whether Software X can be used with different versions of BLAST+.
78 | Implementation testing 80 | Do different implementations work similarly enough? 82 | Test whether Software X works the same between the standalone and webserver versions.
86 | Compatibility testing 88 | Are newer versions compatible with previous input/output? 90 | Test whether Software X can be used with older versions of the UniProtKB database.
94 | Static testing 96 | Is the source code syntactically correct? 98 | Check whether all opening braces have corresponding closing braces or whether code is indented correctly in Software X.
Standard functionality testing: does the software do what it should in daily use?
107 | Use case testing 109 | Can the software do what it is supposed to do regularly? 111 | Test whether Software X can annotate a small plasmid.
115 | Workflow testing 117 | Can the software succesfully traverse each path in the analysis? 119 | Test whether Software X works in different modes (using fast mode, using rnammer over barrnap or using rfam mode).
123 | Sanity testing 125 | Can the software be invoked without errors? 127 | Test whether Software X works correctly without flags, or when checking dependencies or displaying help info.
132 | Destructive testing: what makes the software fail?
136 | Mutation testing 138 | How do the current tests handle harmful alterations to the software? 140 | Test whether changing a single addition to a subtraction within Software X causes the test suite to fail.
144 | Load testing 146 | At what input size does the software fail? 148 | Test whether Software X can annotate a small plasmid (10 Kbp), a medium-size genome (2 Mbp) or an unrealistically large genome for a prokaryote (1 Gbp).
152 | Fault injection 154 | Does the software fail if faults are introduced and how is this handled? 156 | Test whether Software X fails if nonsense functions are introduced in the gene calling code.
159 |
160 | 161 | ### 2. Input test files: the good, the bad, and the ugly 162 | 163 | When testing, it is important to include test files with known expected outcomes for a successful run. 164 | However, it is equally important to include files on which the tool is expected to fail. 165 | For example, some tools should recognize and report an empty input file or a wrong input format. 166 | Examples of valid and invalid file formats are available through the [BioJulia project](https://github.com/BioJulia/BioFmtSpecimens). 167 | 168 | ### 3. Use an easy-to-follow language format to implement testing 169 | 170 | Understanding the test workflow is necessary not only to ensure continued software development but also the integrity of the project for developers and users. 171 | This can be done through the adoption of a standardized and easy-to-follow format, such as [YAML](https://yaml.org/). 172 | 173 | Additionally, testing packages or frameworks offer an efficient approach to test creation and design. 174 | Frameworks such as [unittest](https://docs.python.org/3/library/unittest.html) or [pytest](https://docs.pytest.org/en/stable/) for Python improve test efficiency, help bug detection and reduce manual intervention. 175 | When possible frameworks should be integrated into test workflows. 176 | 177 | ### 4. Testing is good, automated testing is better 178 | 179 | When designing tests for your software, plan to automate. 180 | Whether your tests are small or comprehensive, automatic triggering of tests will help reduce your workload. 181 | 182 | Many platforms trigger tests automatically based on a set of user-defined conditions. Platforms such as [GitHub Actions](https://github.com/features/actions), [GitLab CI](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration), [CircleCI](https://circleci.com/), [Travis CI](https://www.travis-ci.com/) or [Jenkins](https://www.jenkins.io/) offer straightforward automated testing of code seamlessly upon deployment. 183 | 184 | ### 5. Try the test once, then try it again 185 | 186 | The result of an automated test in the context of one computational workspace does not ensure the same result will be obtained in a different setup. 187 | Although package managers and containers have reduced variability between workspaces, it is still important to ensure your software can be installed and used across supported platforms. 188 | One way to ensure this is to test on different environments, with varying dependency versions (e.g., multiple Python versions, instead of only the most recent one). 189 | 190 | ### 6. Showcase the tests 191 | 192 | For prospective users, it is good to know whether you have tested your software and, if so, which tests you have included. 193 | This can be done by displaying [a badge](https://shields.io/) in your [Github README](https://github.com/microbinfie-hackathon2020/CSIS#example-software-testing), or linking to your defined testing strategy e.g. a Github Actions YAML, (see [recommendation #2](https://github.com/microbinfie-hackathon2020/CSIS#2-input-test-files-the-good-the-bad-and-the-ugly)). 194 | 195 | Documenting the testing goal and process enables end-users to easily check tool functionality and the level of testing. 196 | 197 | We recommend contacting the authors, directly or through issues in the code repository, whose software you have tested to share successful outcomes or if you encountered abnormal behavior or component failures. 198 | An external perspective can be useful to find bugs that the authors are unaware of. 199 | 200 | ### 7. Encourage others to test your software 201 | 202 | Software testing can be crowd-sourced, as showcased by the ASM NGS 2020 Hackathon. 203 | Software suites such as [Pangolin](https://github.com/cov-lineages/pangolin) and [chewBBACA](https://github.com/B-UMMI/chewBBACA) have implemented automated testing developed during the Hackathon. 204 | 205 | For developers, crowd-sourcing offers the benefits of fresh eyes on your software. 206 | Feedback and contributions from users can expedite the implementation of software testing practices. 207 | It also contributes to software sustainability by creating community buy-in, which ultimately helps the software maintainers keep pace with dependency changes, and identify current user needs. 208 | 209 | ## Example software testing 210 | 211 | ### Tools with integrated testing 212 | 213 | | Software | Badge with link to CI | Version badge | Yaml | 214 | |----------|-----------------------|---------------|------| 215 | | [This repo](https://github.com/microbinfie-hackathon2020/CSIS/workflows/CSIS) | [![CSIS](https://github.com/microbinfie-hackathon2020/CSIS/workflows/CSIS/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3ACSIS) | ![GitHub release (latest by date)](https://img.shields.io/github/v/release/microbinfie-hackathon2020/CSIS?color=brightgreen&label=version) | [CSIS.yml](/.github/workflows/CSIS.yml) | 216 | |[Bactopia](https://bactopia.github.io/) | [![All Bactopia Tests](https://github.com/bactopia/bactopia/actions/workflows/all-bactopia-tests.yml/badge.svg)](https://github.com/bactopia/bactopia/actions/workflows/all-bactopia-tests.yml) | [![GitHub release (latest by date)](https://img.shields.io/github/v/release/bactopia/bactopia?color=brightgreen)](https://github.com/bactopia/bactopia/releases) | [all-bactopia-tests.yml](https://github.com/bactopia/bactopia/blob/master/.github/workflows/all-bactopia-tests.yml) | 217 | | [chewBBACA](https://github.com/B-UMMI/chewBBACA) | [![chewBBACA](https://github.com/B-UMMI/chewBBACA/workflows/chewbbaca/badge.svg)](https://github.com/B-UMMI/chewBBACA/actions?query=workflow%3Achewbbaca) | ![GitHub release (latest by date)](https://img.shields.io/github/v/release/B-UMMI/chewBBACA?color=brightgreen&label=version) | [chewbbaca.yml](https://github.com/B-UMMI/chewBBACA/blob/master/.github/workflows/chewbbaca.yml) 218 | | [Pangolin](https://github.com/cov-lineages/pangolin) | [![pangolin](https://github.com/microbinfie-hackathon2020/CSIS/workflows/pangolin/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Apangolin) | ![GitHub release (latest by date)](https://img.shields.io/github/v/release/cov-lineages/pangolin?color=brightgreen&label=version) | [pangolin.yml](/.github/workflows/pangolin.yml) | 219 | 220 | ### Tools with external testing 221 | 222 | | Software | Badge with link to CI | Version badge | Yaml | 223 | |----------|-----------------------|---------------|------| 224 | | [Genotyphi](https://github.com/katholt/genotyphi) | [![genotyphi](https://github.com/microbinfie-hackathon2020/CSIS/workflows/genotyphi/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Agenotyphi) | [![Tested Version](https://img.shields.io/badge/version-4296976-brightgreen)](https://github.com/katholt/genotyphi/tree/42969769753a53bb74f15bfb60846b5828ff91) | [genotyphi.yml](/.github/workflows/genotyphi.yml) | 225 | | [Kraken](https://github.com/DerrickWood/kraken) | [![kraken](https://github.com/microbinfie-hackathon2020/CSIS/workflows/kraken/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Akraken) | [![Tested Version](https://img.shields.io/badge/version-v1.1.1-brightgreen)](https://github.com/DerrickWood/kraken/releases/tag/v1.1.1) | [kraken.yml](/.github/workflows/kraken.yml) | 226 | | [KrakenUniq](https://github.com/fbreitwieser/krakenuniq) | [![krakenuniq](https://github.com/microbinfie-hackathon2020/CSIS/workflows/krakenuniq/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Akrakenuniq) | [![Tested Version](https://img.shields.io/badge/version-v0.5.8-brightgreen)](https://github.com/fbreitwieser/krakenuniq/releases/tag/v0.5.8) | [krakenuniq.yml](/.github/workflows/krakenuniq.yml) | 227 | | [Kraken2](https://github.com/DerrickWood/kraken2) | [![kraken2](https://github.com/microbinfie-hackathon2020/CSIS/workflows/kraken2/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Akraken2) | [![Tested Version](https://img.shields.io/badge/version-v2.1.1-brightgreen)](https://github.com/DerrickWood/kraken2/releases/tag/v2.1.1) | [kraken2.yml](/.github/workflows/kraken2.yml) | 228 | | [Centrifuge](https://github.com/DaehwanKimLab/centrifuge) | [![centrifuge](https://github.com/microbinfie-hackathon2020/CSIS/workflows/centrifuge/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Acentrifuge) | [![Tested Version](https://img.shields.io/badge/version-v1.0.4_beta-brightgreen)](https://github.com/DaehwanKimLab/centrifuge/releases/tag/v1.0.4-beta) | [centrifuge.yml](/.github/workflows/centrifuge.yml) | 229 | | [Prokka](https://github.com/tseemann/prokka) | [![prokka](https://github.com/microbinfie-hackathon2020/CSIS/workflows/prokka/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Aprokka) | [![Tested Version](https://img.shields.io/badge/version-v1.14.5-brightgreen)](https://github.com/tseemann/prokka/releases/tag/v1.14.5) | [prokka.yml](/.github/workflows/prokka.yml) | 230 | | [Quast](https://github.com/ablab/quast) | [![quast](https://github.com/microbinfie-hackathon2020/CSIS/workflows/quast/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Aquast) | [![Tested Version](https://img.shields.io/badge/version-v5.0.2-brightgreen)](https://github.com/ablab/quast/releases/tag/quast_5.0.2) | [quast.yml](/.github/workflows/quast.yml) | 231 | | [SKESA](https://github.com/ncbi/SKESA) | [![SKESA](https://github.com/microbinfie-hackathon2020/CSIS/workflows/skesa/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Askesa) | [![Tested Version](https://img.shields.io/badge/version-2.4.0-brightgreen)](https://github.com/ncbi/SKESA/releases/tag/2.4.0) | [skesa.yml](/.github/workflows/skesa.yml) | 232 | | [Shovill](https://github.com/tseemann/shovill) | [![shovill](https://github.com/microbinfie-hackathon2020/CSIS/workflows/shovill/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Ashovill) | [![Tested Version](https://img.shields.io/badge/version-v1.1.0-brightgreen)](https://github.com/tseemann/shovill/releases/tag/v1.1.0) | [shovill.yml](/.github/workflows/shovill.yml) | 233 | | [BUSCO](https://gitlab.com/ezlab/busco) | [![BUSCO](https://github.com/microbinfie-hackathon2020/CSIS/workflows/busco/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Abusco) | [![Tested Version](https://img.shields.io/badge/version-v4.1.4-brightgreen)](https://gitlab.com/ezlab/busco/-/tags/4.1.4) | [busco.yml](/.github/workflows/busco.yml) | 234 | | [Unicycler](https://github.com/rrwick/Unicycler) | [![unicycler](https://github.com/microbinfie-hackathon2020/CSIS/workflows/unicycler/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Aunicycler) | [![Tested Version](https://img.shields.io/badge/version-v0.4.8-brightgreen)](https://github.com/rrwick/Unicycler/releases/tag/v0.4.8) | [unicycler.yml](/.github/workflows/unicycler.yml) | 235 | | [Trycycler](https://github.com/rrwick/Trycycler) | [![trycycler](https://github.com/microbinfie-hackathon2020/CSIS/workflows/trycycler/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Atrycycler) | [![Tested Version](https://img.shields.io/badge/version-v0.4.1-brightgreen)](https://github.com/rrwick/Trycycler/releases/tag/v0.4.1) | [trycycler.yml](/.github/workflows/trycycler.yml) | 236 | | [CheckM](https://github.com/Ecogenomics/CheckM) | [![checkm](https://github.com/microbinfie-hackathon2020/CSIS/workflows/checkm/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Acheckm) | [![Tested Version](https://img.shields.io/badge/version-v1.1.3-brightgreen)](https://github.com/Ecogenomics/CheckM/releases/tag/v1.1.3) | [checkm.yml](/.github/workflows/checkm.yml) | 237 | | [iVar](https://github.com/andersen-lab/ivar) | [![iVar](https://github.com/microbinfie-hackathon2020/CSIS/workflows/ivar/badge.svg)](https://github.com/microbinfie-hackathon2020/CSIS/actions?query=workflow%3Aivar) | [![Tested Version](https://img.shields.io/badge/version-v1.3-brightgreen)](https://github.com/andersen-lab/ivar) | [ivar.yml](/.github/workflows/ivar.yml) | 238 | 239 | ## Etymology 240 | 241 | CSIS is a play on the acronym for the [United States Food Safety Inspection Service](https://www.fsis.usda.gov). 242 | Additionally, it has CSI in the acronym (Crime Scene Investigation) which has a sort of detective feel to it. 243 | 244 | ## Contributors 245 | The following participants were responsible for compiling the set of recommendations presented in this repository: [Boas van der Putten](https://github.com/boasvdp) [Inês Mendes](https://github.com/cimendes), [Brook Talbot](https://github.com/bmtalbot), [Jolinda de Korne-Elenbaas](https://github.com/jolindadekorne), [Rafael Mamede](https://github.com/rfm-targa), [Pedro Vila-Cerqueira](https://github.com/pedrorvc), [Luis Pedro Coelho](https://github.com/luispedro), [Christopher A. Gulvik](https://github.com/chrisgulvik), [Lee S. Katz](https://github.com/lskatz). 246 | 247 | The following participants were contributed in automating tests for bioinformatics and contributing a community resource for identifying software that can pass unit tests, available in this repository: [Áine O'Toole](https://github.com/aineniamh), [Justin Payne](https://github.com/crashfrog), [Mário Ramirez](https://github.com/orgs/B-UMMI/people/ramirma), [Peter van Heusden](https://github.com/pvanheus), [Robert A. Petit III](https://github.com/rpetit3), [Verity Hill](), [Yvette Unoarumhi](). 248 | -------------------------------------------------------------------------------- /TEAMS.md: -------------------------------------------------------------------------------- 1 | # Teams 2 | 3 | ## Each team 4 | 5 | Please fill out the table to join a team. 6 | Include a link to your github username. 7 | Under your GitHub username, please also have a basic introduction. 8 | If you need a primer, here are [instructions](https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-your-github-profile/about-your-profile). 9 | 10 | ### Example team 11 | 12 | * Main time zone: Eastern (UTC -5) 13 | * Goal: unit testing QUAST 14 | 15 | Name | GitHub profile | What knowledge I bring to the group 16 | --------------------|--------------------------------------|------------------------------------ 17 | Lee Katz | [lskatz](https://github.com/lskatz) | GitHub actions, yaml, unit testing 18 | Boas van der Putten | [boasvdp](https://github.com/boasvdp)| GitHub actions, yaml, species-specific knowledge (_Escherichia coli_) 19 | 20 | ### Team 1 - Chewie! 21 | Name | GitHub profile | What knowledge I bring to the group 22 | --------------------|-------------------------------------------|------------------------------------ 23 | Inês Mendes | [cimendes](https://github.com/cimendes) | Python, basic knowledge of pytest 24 | Rafael Mamede | [rfm-targa](https://github.com/rfm-targa) | Python, yaml, chewBBACA main dev 25 | Pedro Cerqueira | [pedrorvc](https://github.com/pedrorvc) | Python, yaml, chewBBACA main dev 26 | 27 | 28 | ### Team *All the Krakens* 29 | 30 | Name | GitHub profile | What knowledge I bring to the group 31 | --------------------|--------------------------------------|------------------------------------ 32 | Robert Petit | [rpetit3](https://github.com/rpetit3) | Python, gitlab-ci, debugging 33 | Justin Payne | [crashfrog](https://github.com/crashfrog) | Python, cloud, semver, Hypothesis framework for unit testing in Python 34 | Amit Yadav | [ay-amityadav](https://github.com/ay-amityadav) | Python, git (basic) 35 | 36 | ### Team bio-cwl-tools 37 | * Make bio-cwl-tools (the CWL bioinformatics tools repository) working 38 | * Main time zone: Central African Time (GMT+2) 39 | * Goals: 40 | * Expand cwltest to make it suitable for testing tools 41 | * Work on integrating cwltest and pytest: https://github.com/common-workflow-language/cwltest/pull/74 42 | * Learn Github Actions 43 | * Apply Github Actions with cwltest to create tests for [bio-cwl-tools](https://github.com/common-workflow-library/bio-cwl-tools) 44 | 45 | Name | GitHub profile | What knowledge I bring to the group 46 | --------------------|-------------------------------------------|------------------------------------ 47 | Peter van Heusden | [pvanheus](https://github.com/pvanheus) | Python, CWL, bioinformatics 48 | 49 | ### Team SARS-CoV-2 50 | Name | GitHub profile | What knowledge I bring to the group 51 | --------------------|-------------------------------------------|------------------------------------ 52 | Brooke Talbot | bmtalbot | 53 | Aine O'Toole | aineniamh | 54 | Verity Hill | viralverity | 55 | Po-E Paul Li | poeli | 56 | Peter van Heusden | pvanheus | 57 | Jolinda de Korne | jolindadekorne | 58 | Yvette Unoarumhi | | 59 | 60 | -------------------------------------------------------------------------------- /templates/bug_template.md: -------------------------------------------------------------------------------- 1 | # Bug report 2 | 3 | ## Bug description 4 | A clear and concise description of what the bug is. 5 | 6 | ## To Reproduce 7 | Steps to reproduce the behavior, please provide code snippets or a repository: 8 | 9 | 1. Go to '…' 10 | 2. Click on '…' 11 | 3. Scroll down to '…' 12 | 4. See error 13 | 14 | ## Expected behavior 15 | A clear and concise description of what you expected to happen. 16 | 17 | ## Screenshots (optional) 18 | If applicable, add screenshots to help explain your problem. 19 | 20 | ## System information 21 | - OS: [e.g. Ubunto, macOS, Windows] 22 | - Software version [e.g. 0.0.2, commit id ff2cb9e] 23 | - Dependecy version (if applicable) [e.g. Python 3.8.0] 24 | - Browser (if applicable) [e.g. Chrome, Safari] 25 | 26 | ## Additional context 27 | Add any other context about the problem here, such as if it was installed through Conda, PyPi or compiled from source. 28 | 29 | Adapted from @gummyg in supabase/supabase#1208 -------------------------------------------------------------------------------- /templates/feedback_template.md: -------------------------------------------------------------------------------- 1 | # Feedback 2 | 3 | ## Summary 4 | A clear and concise description of what the enhancement is and its scope, including what motivated it. 5 | 6 | ## Expected behavior 7 | A clear and concise description of what you expected to happen. 8 | 9 | ## Proposal 10 | An actual plan of action of how the enhancement will be implemented. 11 | 12 | ## Testing 13 | Notes on how to implement testing of the new feature, with defined test cases for failing and passing. 14 | 15 | ## Additional context 16 | Add any other context about the problem here, such as if it was installed through Conda, PyPi or compiled from source. 17 | 18 | Adapted from @damian-romero and @astrochun in UAL-RE/LD-Cool-P#132 -------------------------------------------------------------------------------- /templates/support_template.md: -------------------------------------------------------------------------------- 1 | # Support question 2 | 3 | ## What is the current behaviour? 4 | 5 | Add an explanation of the steps you've run, or that you're planning on running, and the logic behind it for your data. 6 | 7 | ## Main question 8 | 9 | What behaviour would your like to have a better understanding of? Ask consise and direct questions, directly related to what was addressed in the [What is the current behaviour?](#What-is-the-current-behaviour?) section. 10 | 11 | ## To Reproduce (optional) 12 | Steps to reproduce the behavior that lead to your question. Please provide code snippets or a repository: 13 | 14 | 1. Go to '…' 15 | 2. Click on '…' 16 | 3. Scroll down to '…' 17 | 4. See result 'X' 18 | 19 | ## Screenshots (optional) 20 | If applicable, add screenshots to help explain your problem. 21 | 22 | ## System information 23 | - OS: [e.g. Ubunto, macOS, Windows] 24 | - Software version [e.g. 0.0.2, commit id ff2cb9e] 25 | - Dependecy version (if applicable) [e.g. Python 3.8.0] 26 | - Browser (if applicable) [e.g. Chrome, Safari] 27 | 28 | ## Additional context 29 | Add any other context about the problem here, such as if it was installed through Conda, PyPi or compiled from source. 30 | -------------------------------------------------------------------------------- /templates/test_template.md: -------------------------------------------------------------------------------- 1 | 2 | # Test implementation template 3 | 4 | ## Scope of test 5 | Add a description of the scope of the test implemented [e.g. installation testing, use case testig, sanity testing] 6 | 7 | ## Test data 8 | If applicable, add a description of the test data, including provenance. 9 | 10 | ## Checklist 11 | Please fill in the checklists below: 12 | 13 | - [ ] I've read the guidelines for contributions: [CONTRIBUTING.md](../../CONTRIBUTING.md) 14 | 15 | Checking the `name` and `on` sections (all boxes should be checked): 16 | - [ ] The name of the test matches the name of the .yml file 17 | - [ ] The tests are triggered on both push or pull requests, on the `main` and `dev` branches 18 | 19 | These tests run on (at least one of these boxes should be checked): 20 | - [ ] ubuntu-16.04, ubuntu-18.04, ubuntu-20.04 or ubuntu-latest (currently 18.04) 21 | - [ ] macos-10.15, macos-11.0, or macos-latest (currently 10.15) 22 | - [ ] windows-2019 or windows-latest (currently 2019) 23 | 24 | The tests include (all boxes should be checked, see [this workflow as an example for a basic example](../workflows/prokka.yaml)): 25 | - [ ] Installation of the software you're testing 26 | - [ ] Checking what happens when the software is invoked without commands (e.g. `prokka`) 27 | - [ ] Checking whether the `-h` and/or `--help` switches do not raise an error (e.g. `prokka -h` and/or `prokka --help`) 28 | - [ ] Checking whether the `-v` and/or `--version` switches do not raise an error (e.g. `prokka -v` and/or `prokka --version`) 29 | - [ ] Checking whether an error is raised when a non-sensical command is specified (e.g. `prokka --doesnotexist`) 30 | - [ ] Performing a small, functional/black box test (does the tool complete on a toy dataset without raising errors?) 31 | 32 | Optional tests (see [this workflow as an example for an advanced workflow](../workflows/checkm.yaml), courtesy of @chrisgulvik: 33 | - [ ] Checking whether an error is correctly raised when invalid input file is specified (e.g. a truncated FastQ file) 34 | - [ ] Checking whether dependencies are installed correctly 35 | - [ ] Installing dependencies from source 36 | - [ ] Checking whether dependencies pass their own tests --------------------------------------------------------------------------------