├── LICENSE ├── README.md ├── embed-script-and-stage-files.cwl ├── embed.cwl ├── embed2.cwl ├── include-file ├── external-python-no-expansion.cwl ├── external-python.cwl ├── fixed-script.py └── myscript.py ├── javascript_snippets ├── array-to-dir.cwl ├── batch-array.cwl ├── flatten-nestedarray.cwl ├── get-vcfs.cwl └── nestedarray-to-dir.cwl ├── list-of-files-1.cwl ├── list-of-files-bash.cwl ├── list-of-files-python.cwl ├── list-of-files.job.yml ├── manifest.cwl ├── port-matching ├── tool.cwl ├── workflow-value-from.cwl └── workflow.cwl ├── stage-files.cwl └── workflow_patterns_initiative ├── README.md ├── control └── README.md ├── data └── README.md ├── event └── README.md ├── exception └── README.md ├── presentation └── README.md └── resource └── README.md /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | CWL patterns 2 | ------------ 3 | 4 | _Some patterns commonly encountered when writing CWL workflows_ 5 | 6 | 7 | 8 | - [Manifest file via Javascript](#manifest-file-via-javascript) 9 | - [Embedding scripts](#embedding-scripts) 10 | - [I have my script in a separate file](#i-have-my-script-in-a-separate-file) 11 | - [I have `$` signs in my script :(](#i-have--signs-in-my-script-) 12 | - [I don't want to escape the `$`](#i-dont-want-to-escape-the-) 13 | - [Embedding a bash script (style 2)](#embedding-a-bash-script-style-2) 14 | - [Manipulating a list of files using expressions](#manipulating-a-list-of-files-using-expressions) 15 | - [Link input files to working directory](#link-input-files-to-working-directory) 16 | - [How to handle port type mismatches](#how-to-handle-port-type-mismatches) 17 | 18 | 19 | 20 | [Which of the Workflow Patterns Initiative patterns does CWL support?](workflow_patterns_initiative/README.md) 21 | 22 | ## Manifest file via Javascript 23 | > My tool takes in a list of filenames as input. I used to pass each file name 24 | > on the command line, but when I scale up I run into unix "Argument list too 25 | > long" errors. What can I do? 26 | 27 | If the program accepts, or can be altered to accept, a manifest file containing 28 | a list of input file paths, the CWL can be written to generate this manifest 29 | file on the fly before invoking the tool ([example](manifest.cwl)). 30 | 31 | 32 | ## Embedding scripts 33 | > I have a Python script that I want to embed in my tool wrapper (not make part 34 | > of the docker image). How would I do this? 35 | 36 | You can embed the script as an `InitialWorkDirRequirement` 37 | ([example](embed.cwl)). 38 | 39 | ### I have my script in a separate file 40 | > I use good software practices. My Python/Ruby/Haskell/... script is in a 41 | > separate file from my CWL wrapper. 42 | 43 | You can use the `$include` directive to pull in the file into your CWL. 44 | You can use it in `InitialWorkDirRequirement` 45 | ([example](include-file/external-python.cwl)) or in File Contents ([example](include-file/external-python-no-expansion.cwl)) 46 | 47 | 48 | ### I have `$` signs in my script :( 49 | > My embedded (Bash/Python/R) script has "$" signs in it and this is conflicting 50 | > with CWL parameter references. How do get them to play nicely together? 51 | 52 | You can escape the `$` in your script with `\$` 53 | 54 | ### I don't want to escape the `$` 55 | > I'd rather just paste the script into the CWL and not have to go through it, 56 | > escaping the "$" signs. Do I have another option? 57 | 58 | You could embed your script in the "contents" field of the default value of a 59 | file. To note in this solution is that a user can supply a different file to the 60 | script input and override the default script. This can be considered a bug or a 61 | feature, depending on your use case ([example](embed2.cwl)). 62 | 63 | ## Embedding a bash script (style 2) 64 | 65 | A bash script can be passed as a string via the command line and invoked from 66 | the command line. ([example](list-of-files-bash.cwl)). 67 | 68 | ## Manipulating a list of files using expressions 69 | > I have an input that is a list of files. I wish to do processing based on the 70 | > file paths. 71 | 72 | This depends a bit on what the expression is intended to do. The easiest is if 73 | the whole processing can be done in javascript. In this case the pattern looks 74 | like 75 | 76 | ``` 77 | ${ 78 | var cmd = ""; 79 | for( var i = 0; i < inputs.files.length; i++) { 80 | cmd += "\n echo " + inputs.files[i].path; 81 | } 82 | return cmd; 83 | } 84 | ``` 85 | ([example](list-of-files-1.cwl)) 86 | 87 | This works well for when we can do with a JSON or simple string return value 88 | from the JS code. If what we really want is an embedded script (say a bash or 89 | Python script) to be generated, say via the `InitialWorkDirRequirement` it 90 | becomes cumbersome to write in this fashion. One way of doing this is to write a 91 | succinct JS expression that converts the passed JSON object into a list of paths 92 | in the syntax accepted by the script. 93 | 94 | Here is an example for [Python](list-of-files-python.cwl) and for 95 | [bash](list-of-files-bash.cwl) 96 | 97 | 98 | ## Link input files to working directory 99 | > I have a tool that does not do well with arbitrary file paths. I'd like to 100 | > link the files into the working directory so I don't have to deal with 101 | > arbitrary mount paths and so on. 102 | 103 | You can use `InitialWorkDirRequirement` to link the files 104 | ([example](stage-files.cwl)). 105 | 106 | You can mix this with embedding scripts 107 | ([example](embed-script-and-stage-files.cwl)). 108 | 109 | ## How to handle port type mismatches 110 | 111 | - Tool A produces a list of Files (or strings, ints ...) 112 | - Tool B accepts only a single File (or string, int ...) 113 | - How do I connect A to B? 114 | 115 | If you are _sure_ this is not going to be a problem, e.g. in this context A will 116 | only ever produce one file, or you are only interested in one file, you can use 117 | a step `valueFrom` expression to convert the types. 118 | 119 | [Here](port-matching/workflow.cwl) is a workflow that will raise validation 120 | warnings and will fail on execution because of port type mismatches. 121 | 122 | [Here](port-matching/workflow-value-from.cwl) is the same workflow with 123 | `valueFrom` added to make the port types match. 124 | 125 | 126 | ## Some commonly used Javascript expression tools 127 | You can tailor the input/output types to your situation. 128 | 129 | ### Flatten a nested array into an array 130 | [flatten-nestedarray.cwl](javascript_snippets/flatten-nestedarray.cwl) shows how to flatten an 131 | array such as \[\["a0", "a1"\], \["b0", "b1"\]\] into \["a0", "a1", "b0", "b1"\]. 132 | 133 | ### Divide an array into batches with a given batch size 134 | [batch-array.cwl](javascript_snippets/batch-array.cwl) shows how to divide an array such as 135 | \["a0", "a1", "a2", "a3", "a4"\] into a nested array 136 | \[\["a0", "a1"\], \["a2", "a3"\], \["a4"\]\] with batch size 2. With batch size 137 | 3, the batched nested array would be [\["a0", "a1", "a2"\], \["a3", "a4"\]\]. 138 | 139 | ### Stage an array of 'File' and 'Directory' type in a 'Directory' 140 | [array-to-dir.cwl](javascript_snippets/array-to-dir.cwl) shows how to stage an array such as 141 | \[file0, dir0, file1, dir1\] in a new directory, with file0, dir0, file1, dir1 142 | as its content. 143 | 144 | ### Stage a nested array of 'File' and 'Directory' type in a 'Directory' 145 | [nestedarray-to-dir.cwl](javascript_snippets/nestedarray-to-dir.cwl) shows how to stage a nested array 146 | such as \[\[file0, dir0\], \[file1, dir1\]\] in a new directory, with 147 | file0, dir0, file1, dir1 as its content. 148 | 149 | ### Process a directory of bgzipped and indexed vcf files into an array 150 | [get-vcfs.cwl](javascript_snippets/get-vcfs.cwl) is a bioinformatics specific tool that 151 | processes a 'Directory' with files such as a.vcf.gz, a.vcf.gz.tbi, b.vcf.gz, 152 | b.vcf.gz.tbi into an array \[a.vcf.gz, b.vcf.gz\], where a.vcf.gz.tbi is 153 | the secondary file of a.vcf.gz, and b.vcf.gz.tbi is the secondary file of 154 | b.vcf.gz, respectively. 155 | -------------------------------------------------------------------------------- /embed-script-and-stage-files.cwl: -------------------------------------------------------------------------------- 1 | cwlVersion: v1.0 2 | class: CommandLineTool 3 | inputs: 4 | files: File[] 5 | outputs: 6 | out: 7 | type: stdout 8 | stdout: output.txt 9 | baseCommand: [sh, script.sh] 10 | requirements: 11 | InitialWorkDirRequirement: 12 | listing: 13 | - entryname: script.sh 14 | entry: |- 15 | set -e 16 | ls -al 17 | - $(inputs.files) 18 | -------------------------------------------------------------------------------- /embed.cwl: -------------------------------------------------------------------------------- 1 | class: CommandLineTool 2 | cwlVersion: v1.0 3 | 4 | inputs: 5 | message: string 6 | 7 | baseCommand: ["python", "myscript.py"] 8 | 9 | requirements: 10 | InitialWorkDirRequirement: 11 | listing: 12 | - entryname: myscript.py 13 | entry: |- 14 | print("$(inputs.message)") 15 | 16 | outputs: 17 | example_out: 18 | type: stdout 19 | stdout: output.txt -------------------------------------------------------------------------------- /embed2.cwl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env cwl-runner 2 | cwlVersion: v1.0 3 | class: CommandLineTool 4 | label: Run an embedded Python script 5 | hints: 6 | DockerRequirement: 7 | dockerPull: python:3 8 | baseCommand: python 9 | 10 | inputs: 11 | script: 12 | type: File 13 | inputBinding: 14 | position: 1 15 | default: 16 | class: File 17 | basename: "script.py" 18 | contents: |- 19 | cash = 256.75 20 | print("This costs ${}".format(cash)) 21 | 22 | outputs: 23 | results: 24 | type: stdout -------------------------------------------------------------------------------- /include-file/external-python-no-expansion.cwl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env cwl-runner 2 | class: CommandLineTool 3 | cwlVersion: v1.0 4 | label: Run an embedded Python script 5 | 6 | inputs: 7 | script: 8 | type: File 9 | default: 10 | class: File 11 | basename: script.py 12 | contents: 13 | $include: fixed-script.py 14 | inputBinding: 15 | position: 1 16 | 17 | outputs: 18 | results: 19 | type: stdout 20 | stdout: out.txt 21 | 22 | baseCommand: python 23 | -------------------------------------------------------------------------------- /include-file/external-python.cwl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env cwl-runner 2 | 3 | class: CommandLineTool 4 | cwlVersion: v1.0 5 | 6 | requirements: 7 | InitialWorkDirRequirement: 8 | listing: 9 | - entryname: myscript.py 10 | entry: 11 | $include: myscript.py 12 | 13 | inputs: 14 | message: string 15 | 16 | outputs: 17 | example_out: 18 | type: stdout 19 | stdout: output.txt 20 | 21 | baseCommand: 22 | - python 23 | - myscript.py 24 | -------------------------------------------------------------------------------- /include-file/fixed-script.py: -------------------------------------------------------------------------------- 1 | # This script has to be included in a File contents 2 | # field to avoid ${} being interpreted as a JS expression 3 | cash = 256.75 4 | print("This costs ${}".format(cash)) 5 | -------------------------------------------------------------------------------- /include-file/myscript.py: -------------------------------------------------------------------------------- 1 | print("$(inputs.message)") 2 | -------------------------------------------------------------------------------- /javascript_snippets/array-to-dir.cwl: -------------------------------------------------------------------------------- 1 | cwlVersion: v1.2 2 | class: ExpressionTool 3 | doc: "Stage an array of 'File' and 'Directory' type (potentially mixed) in a 'Directory'." 4 | requirements: 5 | InlineJavascriptRequirement: {} 6 | inputs: 7 | arr: 8 | type: 9 | type: array 10 | items: [File, Directory] 11 | dirname: 12 | type: string 13 | outputs: 14 | dir: Directory 15 | expression: | 16 | ${ 17 | var dir = {"class": "Directory", 18 | "basename": inputs.dirname, 19 | "listing": inputs.arr}; 20 | return {"dir": dir}; 21 | } 22 | -------------------------------------------------------------------------------- /javascript_snippets/batch-array.cwl: -------------------------------------------------------------------------------- 1 | cwlVersion: v1.2 2 | class: ExpressionTool 3 | doc: "Divide an array of 'Any' type into batches with a given batch size." 4 | requirements: 5 | InlineJavascriptRequirement: {} 6 | inputs: 7 | arr: 8 | type: 9 | type: array 10 | items: Any 11 | batchsize: 12 | type: int 13 | outputs: 14 | batches: 15 | type: 16 | type: array 17 | items: 18 | type: array 19 | items: Any 20 | expression: | 21 | ${ 22 | var batches = []; 23 | for (var i = 0; i < inputs.arr.length; i+=inputs.batchsize) { 24 | var batch = inputs.arr.slice(i, i+inputs.batchsize); 25 | batches.push(batch); 26 | } 27 | return {"batches": batches}; 28 | } 29 | -------------------------------------------------------------------------------- /javascript_snippets/flatten-nestedarray.cwl: -------------------------------------------------------------------------------- 1 | cwlVersion: v1.2 2 | class: ExpressionTool 3 | doc: "Flatten a nested array of 'Any' type into an array." 4 | requirements: 5 | InlineJavascriptRequirement: {} 6 | inputs: 7 | nestedarray: 8 | type: 9 | type: array 10 | items: 11 | type: array 12 | items: ["null", Any] 13 | outputs: 14 | flattenedarray: 15 | type: 16 | type: array 17 | items: Any 18 | expression: | 19 | ${ 20 | var flattenedarray = []; 21 | for (var i = 0; i < inputs.nestedarray.length; i++) { 22 | for (var j = 0; j < inputs.nestedarray[i].length; j++) { 23 | if (inputs.nestedarray[i][j] != null) { 24 | flattenedarray.push(inputs.nestedarray[i][j]); 25 | } 26 | } 27 | } 28 | return {"flattenedarray": flattenedarray}; 29 | } 30 | -------------------------------------------------------------------------------- /javascript_snippets/get-vcfs.cwl: -------------------------------------------------------------------------------- 1 | cwlVersion: v1.2 2 | class: ExpressionTool 3 | doc: "Process a directory of bgzipped and indexed vcf files into an array with indices as secondary files." 4 | requirements: 5 | InlineJavascriptRequirement: {} 6 | hints: 7 | LoadListingRequirement: 8 | loadListing: shallow_listing 9 | inputs: 10 | vcfsdir: Directory 11 | outputs: 12 | vcfs: 13 | type: File[] 14 | secondaryFiles: [.tbi] 15 | expression: | 16 | ${ 17 | var vcfs = []; 18 | for (var i = 0; i < inputs.vcfsdir.listing.length; i++) { 19 | var file = inputs.vcfsdir.listing[i]; 20 | if (file.nameext == '.gz') { 21 | var main = file; 22 | for (var j = 0; j < inputs.vcfsdir.listing.length; j++) { 23 | var file = inputs.vcfsdir.listing[j]; 24 | if (file.basename == main.basename+".tbi") { 25 | main.secondaryFiles = [file]; 26 | break; 27 | } 28 | } 29 | vcfs.push(main); 30 | } 31 | } 32 | return {"vcfs": vcfs}; 33 | } 34 | -------------------------------------------------------------------------------- /javascript_snippets/nestedarray-to-dir.cwl: -------------------------------------------------------------------------------- 1 | cwlVersion: v1.2 2 | class: ExpressionTool 3 | doc: "Stage a nested array of 'File' and 'Directory' type (potentially mixed) in a 'Directory'." 4 | requirements: 5 | InlineJavascriptRequirement: {} 6 | inputs: 7 | nestedarr: 8 | type: 9 | type: array 10 | items: 11 | type: array 12 | items: [File, Directory] 13 | dirname: 14 | type: string 15 | outputs: 16 | dir: Directory 17 | expression: | 18 | ${ 19 | var dir = {"class": "Directory", 20 | "basename": inputs.dirname, 21 | "listing": []}; 22 | for (var i = 0; i < inputs.nestedarr.length; i++) { 23 | for (var j = 0; j < inputs.nestedarr[i].length; j++) { 24 | dir.listing.push(inputs.nestedarr[i][j]); 25 | } 26 | } 27 | return {"dir": dir}; 28 | } 29 | -------------------------------------------------------------------------------- /list-of-files-1.cwl: -------------------------------------------------------------------------------- 1 | cwlVersion: v1.0 2 | class: CommandLineTool 3 | requirements: 4 | ShellCommandRequirement: {} 5 | InlineJavascriptRequirement: {} 6 | inputs: 7 | files: 8 | type: 9 | type: array 10 | items: File 11 | outputs: 12 | example_out: 13 | type: stdout 14 | stdout: output.txt 15 | arguments: 16 | - shellQuote: false 17 | valueFrom: > 18 | ${ 19 | var cmd = ""; 20 | for( var i = 0; i < inputs.files.length; i++) { 21 | cmd += "\n echo " + inputs.files[i].path; 22 | } 23 | return cmd; 24 | } -------------------------------------------------------------------------------- /list-of-files-bash.cwl: -------------------------------------------------------------------------------- 1 | # The script in the valueFrom field creates a bash list by concatenating 2 | # the file paths in quotes with spaces inbetween. 3 | 4 | cwlVersion: v1.0 5 | class: CommandLineTool 6 | requirements: 7 | ShellCommandRequirement: {} 8 | InlineJavascriptRequirement: {} 9 | inputs: 10 | files: 11 | type: 12 | type: array 13 | items: File 14 | outputs: 15 | example_out: 16 | type: stdout 17 | stdout: output.txt 18 | arguments: 19 | - shellQuote: false 20 | valueFrom: > 21 | file_array=(${ 22 | var cmd = ""; 23 | for( var i = 0; i < inputs.files.length; i++) { 24 | cmd += "\"" + inputs.files[i].path + "\" "; 25 | } 26 | return cmd; 27 | }); 28 | 29 | for ((i=0; i < \${#file_array[@]}; i++)) 30 | do 31 | echo \${file_array[\$i]}; 32 | done -------------------------------------------------------------------------------- /list-of-files-python.cwl: -------------------------------------------------------------------------------- 1 | # The javascript expression returns a string that is code for a python list 2 | 3 | cwlVersion: v1.0 4 | class: CommandLineTool 5 | requirements: 6 | InlineJavascriptRequirement: {} 7 | InitialWorkDirRequirement: 8 | listing: 9 | - entryname: script.py 10 | entry: |- 11 | # The javascript expression returns a string that is code for 12 | # a python list 13 | file_array=[ ${ 14 | var l = ""; 15 | for( var i = 0; i < inputs.files.length; i++) { 16 | if (i > 0) { l += ", "; } 17 | l += "'" + inputs.files[i].path + "'"; 18 | } 19 | return l; 20 | }] 21 | 22 | print(file_array) 23 | 24 | inputs: 25 | files: 26 | type: 27 | type: array 28 | items: File 29 | outputs: 30 | example_out: 31 | type: stdout 32 | stdout: output.txt 33 | baseCommand: ["python", "script.py"] 34 | -------------------------------------------------------------------------------- /list-of-files.job.yml: -------------------------------------------------------------------------------- 1 | files: 2 | - class: File 3 | path: README.md 4 | - class: File 5 | path: LICENSE 6 | -------------------------------------------------------------------------------- /manifest.cwl: -------------------------------------------------------------------------------- 1 | class: CommandLineTool 2 | cwlVersion: v1.0 3 | 4 | inputs: 5 | input: 'File[]?' 6 | 7 | baseCommand: cat 8 | arguments: [manifest.txt] 9 | 10 | requirements: 11 | - class: DockerRequirement 12 | dockerPull: alpine 13 | - class: InitialWorkDirRequirement 14 | listing: 15 | - entryname: manifest.txt 16 | entry: |- 17 | ${ 18 | var x = "" 19 | for (var i = 0; i < inputs.input.length; i++) 20 | { 21 | x += inputs.input[i].path + "\n"; 22 | } 23 | return x; 24 | } 25 | writable: false 26 | - class: InlineJavascriptRequirement 27 | 28 | outputs: 29 | output: stdout 30 | 31 | stdout: out.txt -------------------------------------------------------------------------------- /port-matching/tool.cwl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env cwl-runner 2 | 3 | class: CommandLineTool 4 | cwlVersion: v1.0 5 | 6 | inputs: 7 | in1: string? 8 | in2: string[]? 9 | 10 | outputs: 11 | out1: 12 | type: string? 13 | outputBinding: 14 | outputEval: $(inputs.in1) 15 | out2: 16 | type: string[]? 17 | outputBinding: 18 | outputEval: $(inputs.in2) 19 | 20 | baseCommand: [echo] 21 | arguments: [] 22 | -------------------------------------------------------------------------------- /port-matching/workflow-value-from.cwl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env cwl-runner 2 | 3 | class: Workflow 4 | cwlVersion: v1.0 5 | 6 | requirements: 7 | StepInputExpressionRequirement: {} 8 | InlineJavascriptRequirement: {} 9 | 10 | inputs: 11 | in1: string? 12 | in2: string[]? 13 | 14 | outputs: 15 | out1: 16 | type: string? 17 | outputSource: step1/out1 18 | out2: 19 | type: string[]? 20 | outputSource: step1/out2 21 | 22 | steps: 23 | step1: 24 | in: 25 | in1: 26 | source: in2 27 | valueFrom: $(self[0]) 28 | in2: 29 | source: in1 30 | valueFrom: $([self]) 31 | run: tool.cwl 32 | out: 33 | - out1 34 | - out2 35 | -------------------------------------------------------------------------------- /port-matching/workflow.cwl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env cwl-runner 2 | 3 | class: Workflow 4 | cwlVersion: v1.0 5 | 6 | inputs: 7 | in1: string? 8 | in2: string[]? 9 | 10 | outputs: 11 | out1: 12 | type: string? 13 | outputSource: step1/out1 14 | out2: 15 | type: string[]? 16 | outputSource: step1/out2 17 | 18 | steps: 19 | step1: 20 | in: 21 | in1: in2 22 | in2: in1 23 | run: tool.cwl 24 | out: 25 | - out1 26 | - out2 27 | -------------------------------------------------------------------------------- /stage-files.cwl: -------------------------------------------------------------------------------- 1 | cwlVersion: v1.0 2 | class: CommandLineTool 3 | inputs: 4 | files: File[] 5 | outputs: 6 | out: 7 | type: stdout 8 | stdout: output.txt 9 | baseCommand: ["ls", "-al"] 10 | requirements: 11 | InitialWorkDirRequirement: 12 | listing: $(inputs.files) 13 | -------------------------------------------------------------------------------- /workflow_patterns_initiative/README.md: -------------------------------------------------------------------------------- 1 | http://www.workflowpatterns.com/ 2 | 3 | Each sub-folder analyzes the features of CWL against a category of patterns found in the Workflow Patterns Initiative's website. 4 | 5 | - [Control](control/README.md) patterns 6 | - [Resource](resource/README.md) patterns 7 | - [Data](data/README.md) patterns 8 | - [Exception](exception/README.md) Handling patterns 9 | - [Presentation](presentation/README.md) patterns 10 | - [Event](event/README.md) Log Imperfection patterns 11 | 12 | 13 | Note: the WPI's definition of a workflow is much broader than the type of workflows CWL aims to describe. 14 | 15 | CWL targets workflows made of command line tools with no real-time communication between steps nor outside services. 16 | WPI's definition of workflows includes (business) process modeling which have many control-flow features. 17 | 18 | Therefore these are not to be treated as "to do" lists for CWL, but a way to describ and demonstrate CWL's features and abilities in the larger workflow space. 19 | -------------------------------------------------------------------------------- /workflow_patterns_initiative/control/README.md: -------------------------------------------------------------------------------- 1 | 2 | # WPI's [Workflow Control Patterns](http://www.workflowpatterns.com/patterns/control/) and CWL 3 | 4 | Note: CWL is a data-driven (dataflow) workflow standard. These patterns use control-flow language and thus we speak of the equivalent data-flow constructs. 5 | There is no concept of "thread of control" in CWL. 6 | 7 | Of the 43 WPI Workflow Control patterns, 8 patterns are supported by CWL v1.2 or earlier and 35 patterns are unsupported. 8 | 9 | [Back to the list of WPI pattern categories](../README.md) 10 | 11 | ## "Basic Control Flow Patterns" 12 | 13 | * Pattern 1 (Sequence) 14 | 15 | Yes. CWL is a DAG based workflow language with explicit dependencies between steps. 16 | 17 | CWL versions: all 18 | 19 | * Pattern 2 (Parallel Split) 20 | 21 | Yes. Anytime a step in a CWL workflow has all of its inputs available it is allowed to be executed. 22 | 23 | CWL versions: all 24 | 25 | * Pattern 3 (Synchronization) 26 | 27 | Yes. One can have a step in a CWL workflow that requires inputs from the result of multiple other steps. 28 | Downstream steps that require the output of that step (and their descendents) will not be available for execution until the combination step has finished. 29 | 30 | CWL versions: all 31 | 32 | * Pattern 4 (Exclusive choice) 33 | 34 | Yes. No explicit construct, but can be achieved by marking the two downstream steps with the `when` CWL conditional step 35 | execution feature where the logic for one step's `when` field is the inverse of the other step's `when` field. 36 | 37 | If there is a downstream workflow step that would inherit an value from one of the exclusive choice steps into the same incoming input port, then `pickValue: the_only_non_null` may be helpful to ensure that both "exclusive" choices didn't execute due to a misconfiguration in their `when` logics. 38 | 39 | CWL versions: v1.2+ 40 | 41 | * Pattern 5 (Simple Merge) 42 | 43 | Yes. Any workflow can include another workflow as a step, thus allowing re-use. 44 | 45 | CWL versions: all 46 | 47 | ## "Advanced Branching and Synchronization Patterns" 48 | 49 | * Pattern 6 (Multi-Choice) 50 | 51 | No, as there is not a construct in CWL to implement this directly, as required by the WPI definition of this pattern. 52 | Can be emulated using the CWL v1.2 `when` similar to the method to implement Pattern 4 (Exclusive choice), but with 3 or more steps using matched `when` markers 53 | 54 | * Pattern 7 (Structured Synchronizing Merge) 55 | 56 | No, as there is not a construct in CWL to implement this directly, as required by the WPI definition of this pattern. 57 | One way to achieve this is to hide the choice inside a sub-workflow which will provide a stable set of outputs to connect to other steps. 58 | As this pattern implies the conditional execution of prior steps, the use of the `pickValue` construct in the merge step will likely be useful. 59 | 60 | * Pattern 8 (Multi-Merge) 61 | 62 | No, as CWL does not have a construct for the WPI Multi-Choice pattern, which is a pre-requisite for the WPI definition of the Multi-Merge pattern. 63 | 64 | However, any CWL step can rely on inputs from multiple other steps. This functionality is available in all CWL versions. 65 | 66 | * Pattern 9 (Structured Discriminator) 67 | 68 | No. CWL does not support canceling the execution of tasks according to some criteria. 69 | 70 | * Pattern 28 (Blocking Discriminator) 71 | 72 | No. State in CWL is read-only, therefore this pattern is not supported as it would be impossible to reset the discriminator. 73 | This pattern is likely not possible in CWL for other reasons as well. 74 | 75 | * Pattern 29 (Cancelling Discriminator) 76 | 77 | No. CWL does not support canceling the execution of tasks according to some criteria. 78 | Additionally, all variables in CWL are read-only so there is no ability to "reset [a] construct". 79 | 80 | * Pattern 30 (Structured Partial Join) 81 | 82 | No. CWL does not support canceling the execution of tasks according to some dynamic criteria. 83 | 84 | * Pattern 31 (Blocking Partial Join) 85 | 86 | No. CWL does not support re-ordering of tasks. 87 | 88 | * Pattern 32 (Cancelling Partial Join) 89 | 90 | No. CWL does not support canceling the execution of tasks according to some criteria. 91 | 92 | * Pattern 33 (Generalized AND-Join) 93 | 94 | No. CWL is a dataflow workflow language, not a control-flow workflow language. All inputs must be known at the time of workflow enactment. 95 | 96 | * Pattern 37 (Local Synchronizing Merge) 97 | 98 | No. CWL is a dataflow workflow language, not a control-flow workflow language. 99 | 100 | A CWL step can not accumulate inputs (or continue to emit outputs) over time. 101 | CWL steps are only executed once (unless `scatter` is used) and the results of that step are only made available once execution has finished. 102 | 103 | See also the evaluation of CWL against Pattern 7 (Structured Synchronizing Merge). 104 | 105 | * Pattern 38 (General Synchronizing Merge) 106 | 107 | Yes, as this was easy to implement due to two factors related to CWL being a dataflow workflow language and not a control-flow workflow language: 108 | 109 | 1. CWL steps are only executed once (unless `scatter` is used) and the results of that step are only made available once execution has finished. 110 | 111 | 2. A CWL step is available for execution when all of the upstream steps have been executed (or permanently skipped due to the use of the `when` construct). 112 | 113 | CWL versions: 114 | without conditional steps: all 115 | with contional steps: v1.2+ 116 | 117 | * Pattern 41 (Thread Merge) 118 | 119 | No, CWL has no concept of threads at the workflow language level. The underlying applications may use POSIX threads, but that is not managed by CWL. 120 | 121 | * Pattern 42 (Thread Split) 122 | 123 | No, CWL has no concept of threads at the workflow language level. The underlying applications may use POSIX threads, but that is not managed by CWL. 124 | 125 | ## "Multiple Instance Patterns" 126 | 127 | * Pattern 12 (Multiple Instances without Synchronization) 128 | 129 | No. While CWL has the `scatter` construct, using this implies an implicit "gather"ing of the results before downstreams steps can use the results. 130 | This means that execution of subsequent steps is delayed for the slowest execution of the the `scatter`ed tasks, even if that particular result isn't needed right away. 131 | 132 | Additionally, CWL does not have a `loop` construct, which would be another way one could implement this pattern. 133 | 134 | * Pattern 13 (Multiple Instances with a priori Design-Time Knowledge) 135 | 136 | No. There is no CWL construct to execute a step or task a specific N number of times, where N is a concrete number (like 23) specified by the workflow author. 137 | 138 | Additionally, CWL does not have a `loop` construct, which would be another way one could implement this pattern. 139 | 140 | However: this pattern could be emulated in a CWL editor or language that converts to CWL syntax by creating explicit steps (23 in our example) in the CWL workflow description. 141 | 142 | * Pattern 14 (Multiple Instances with a priori Run-Time Knowledge) 143 | 144 | Yes. CWL has the `scatter` construct which allows a compact directive that a given step is to be executed multiple times where most inputs are fixed except for those `scatter`ed specified inputs. 145 | 146 | * Pattern 15 (Multiple Instances without a priori Run-Time Knowledge) 147 | 148 | No. CWL lacks any construct that can spawn additional tasks dynamically according to criteria. 149 | 150 | * Pattern 34 (Static Partial Join for Multiple Instances) 151 | 152 | No. CWL does not have the ability to allow execution of a downstream step when N of M tasks have completed. 153 | 154 | * Pattern 35 (Cancelling Partial Join for Multiple Instances) 155 | 156 | No. CWL does not support canceling the execution of tasks according to some criteria. 157 | 158 | * Pattern 36 (Dynamic Partial Join for Multiple Instances) 159 | 160 | No. CWL lacks any construct that can spawn additional tasks dynamically according to criteria. 161 | 162 | ## "State-based Patterns" 163 | 164 | * Pattern 16 (Deferred Choice) 165 | 166 | No. The CWL v1.2+ conditional workflow step ability is only based upon explict inputs to the workflow step. 167 | 168 | Additionally, in CWL there is no concept of an "operating environment". 169 | 170 | * Pattern 17 (Interleaved Parallel Routing) 171 | 172 | No. Ordering of tasks in CWL is not fixed, but the dependecy graph is explicit. 173 | 174 | There is no CWl construct to limit the number of parallel tasks being executed, that is up to the workflow engine. 175 | 176 | * Pattern 18 (Milestone) 177 | 178 | No. In CWL, parameter values are read-only; they can not change over time. 179 | 180 | It is not possible to query the state of another CWL task. 181 | 182 | A step in a CWL workflow can only recieve specific outputs from other steps that have already completed, and this wiring is fixed prior to workflow execution. 183 | 184 | * Pattern 39 (Critical Section) 185 | 186 | No. Sub sections of CWL workflows can not be bidirectionally connected. One could depend on another (by putting the other in a sub-workflow, if it isn't one already). 187 | 188 | It is not possible to query the state of another CWL task. 189 | 190 | A step in a CWL workflow can only recieve specific outputs from other steps that have already completed, and this wiring is fixed prior to workflow execution. 191 | 192 | * Pattern 40 (Interleaved Routing) 193 | 194 | No. Ordering of tasks in CWL is not fixed, but the dependecy graph is explicit. 195 | 196 | There is no CWl construct to limit the number of parallel tasks being executed, that is up to the workflow engine. 197 | 198 | ## "Cancellation and Force Completion Patterns" 199 | 200 | * Pattern 19 (Cancel Task) 201 | 202 | No. CWL does not support canceling the execution of tasks according to some criteria. 203 | 204 | * Pattern 20 (Cancel Case) 205 | 206 | No. CWL does not support canceling the execution of tasks according to some criteria. 207 | 208 | Additionally, CWL does not have the concept of a "case". 209 | 210 | * Pattern 25 (Cancel Region) 211 | 212 | No. CWL does not support canceling the execution of tasks according to some criteria. 213 | 214 | * Pattern 26 (Cancel Multiple Instance Task) 215 | 216 | No. CWL does not support canceling the execution of tasks according to some criteria. 217 | 218 | * Pattern 27 (Complete Multiple Instance Task) 219 | 220 | No. CWL does not support canceling the execution of tasks according to some criteria. 221 | 222 | ## "Iteration Patterns" 223 | 224 | * Pattern 10 (Arbitrary Cycles) 225 | 226 | No. CWL has no loop construct. 227 | 228 | * Pattern 21 (Structured Loop) 229 | 230 | No. CWL has no loop construct. 231 | 232 | * Pattern 22 (Recursion) 233 | 234 | No. CWL does not support recursion. 235 | 236 | ## "Termination Patterns" 237 | 238 | * Pattern 11 (Implicit Termination) 239 | 240 | Yes. CWL workflows are finished when all of the required outputs are available. 241 | 242 | * Pattern 43 (Explicit Termination) 243 | 244 | No. CWL does not have an explicit "end node" construct. 245 | 246 | ## "Trigger Patterns" 247 | 248 | * Pattern 23 (Transient Trigger) 249 | 250 | No. CWL does not support the concept of triggers or signals. 251 | 252 | * Pattern 24 (Persistent Trigger) 253 | 254 | No. CWL does not support the concept of triggers or signals. 255 | -------------------------------------------------------------------------------- /workflow_patterns_initiative/data/README.md: -------------------------------------------------------------------------------- 1 | 2 | # WPI's [Workflow Data Patterns](http://www.workflowpatterns.com/patterns/data/) and CWL 3 | 4 | Of the 40 WPI Workflow Data patterns, 17 patterns are supported by CWL v1.2 or earlier and 23 patterns are unsupported. 5 | 6 | [Back to the list of WPI pattern categories](../README.md) 7 | 8 | * Pattern 1 (Task Data) 9 | 10 | "Data can be explicitly declared at task level with task level scoping" -> Yes, you can have a variable or value defined in a CommandLineTool, ExpressionTool, or Operation that is not visible to other CWL Processes. They are initialized with a value provided by the Process author. 11 | 12 | * Pattern 2 (Block Data) 13 | 14 | "Data can be explicitly declared at block task level with block task level scoping" -> Yes, one can declare an extra variable at the step level or sub-workflow level. 15 | 16 | "Facilities exist for formal parameter passing to and from a block" -> sub-Processes can inherit a step level variable when it is connected to one of the Process’s inputs. They cannot access any given step-level input by name or any other mechanism, only by its own defined input parameters. Arbitrary access to block level parameters from another block is not possible in CWL. The only outputs from a step are the outputs from the underlying Process that have been marked for export. For sub-workflows, their steps may connect workflow-level data (inputs parameters) to a specific sub-Processes by identifiers. The only data from a sub-workflow that is available to a sibling Process or the parent Workflow are the explicit output parameters that connect to specific step outputs in the sub-Workflow. 17 | 18 | * Pattern 3 (Scope Data) 19 | 20 | No. CWL does not support Scope Data as scopes are defined as not creating a new address space; the closest approximation (a CWL sub-Workflow) does create a new address space. 21 | 22 | * Pattern 4 (Multiple Instance Data) 23 | 24 | "The data element is capable of being replicated or partitioned across multiple tasks": Yes. 25 | 26 | "Each of these data instances exist in their own address space": Yes. 27 | 28 | A Process can be run with unique data inputs multiple times in a CWL workflow and multiple times in the same step when using the ‘scatter’ feature. Each run of that Process has access to only the specific data that was connected to it, and not to the data from other runs. 29 | 30 | "The instances are able to be accessed from a higher level in the process hierarchy" Only after execution is finished, and only for the explicit output values, yes. 31 | 32 | * Pattern 5 (Case Data) 33 | 34 | "Direct tool support for data elements at case level with case level scoping. Case data visible to all components of the case.": No, in CWL each Process only has access to its own defined inputs. Those inputs might be connected to Workflow-scope inputs (or the outputs from other steps) but there is no “global” or case namespace accessible from all levels of a workflow. 35 | 36 | * Pattern 6 (Folder Data) 37 | No. As CWL does not have a Case Data concept, it can not have a Folder Data concept either. 38 | 39 | * Pattern 7 (Workflow Data) 40 | 41 | "Workflow data visible to all components of a workflow". Yes, all steps can reference workflow level inputs (including inputs with default values that are rarely overridden by users) in setting up the inputs to their Process. However, inside those Processes there is no visibility into the step level or workflow level namespaces. Only those data values that have been propagated to the Process via one of its own named inputs are available. 42 | 43 | * Pattern 8 (Environment Data) 44 | 45 | No. There is no CWL construct for Environment Data. If network access is allowed by the executor, then the applications executed by CommandLineTools can access external data, though this is not advised for repeatability and resiliency reasons. Other sources of data gathering besides IP network access by CommandLineTools are possible, but not guaranteed by the CWL standards. They would certainly not be portable and such are not recommended. 46 | 47 | * Pattern 9 (Task to Task) 48 | 49 | Yes, CWL supports Task to Task data communication via Distinct data channels. That is, CWL Processes can inherit data from another Process and their outputs can in turn become inputs to other Processes, all defined by the `in` mapping in each CWL Workflow step. Concurrency is only an issue when `InplaceUpdateRequirement` is implemented. 50 | 51 | * Pattern 10 (Block Task to Sub-Workflow Decomposition) 52 | 53 | "Data elements available to a block task are able to be passed to or are accessible in the associated sub-workflow". Yes, step level inputs are connected to sub-Workflows. 54 | 55 | "There is some degree of control over which elements at block task level are made accessible in the sub-workflow" Yes, only explicit step level values are connected to the pre-defined sub-workflow inputs. 56 | 57 | * Pattern 11 (Sub-Workflow Decomposition to Block Task) 58 | 59 | "Data elements at sub-workflow level can be passed to or made accessible in the corresponding block task" Yes, via the ‘outputs’ section in the CWL (sub-)Workflow step definition. 60 | 61 | * Pattern 12 (To Multiple Instance Task) 62 | 63 | "Multiple instance tasks directly supported". Yes. 64 | 65 | "Data elements can be passed from an atomic task to all instances of a multiple instance task". Yes, via ‘scatter’. 66 | 67 | "Workflow handles synchronization of data passing and any necessary data replication" Yes, this is a defined responsibility of a CWL compliant workflow engine 68 | 69 | "Facilities are available to allocate sections of an aggregate data element to specific task instances". Yes. 70 | 71 | "Data elements in each task instance are independent of those in other task instances". Yes 72 | 73 | * Pattern 13 (From Multiple Instance Task) 74 | 75 | "Multiple instance tasks directly supported". Yes 76 | 77 | "Data elements can be aggregated from multiple task instances and forwarded to subsequent task instance(s)" Yes, via an implicit gather after the execution of a CWL Workflow step marked with “scatter”. 78 | 79 | "Workflow handles synchronization of data passing and any necessary data replication". Yes, this is a defined responsibility of a CWL compliant workflow engine 80 | 81 | * Pattern 14 (Case to Case) 82 | 83 | No, CWL explicitly and purposefully does not support interactions between concurrently executed Processes. CWL is not a service orchestration language. There is no CWL construct to say that two or more Processes should execute simultaneously or be overlapping. The software run by CommandLineTool processes might communicate data with an external service that might allow for exchanging data between CommandLineTool Processes that might be coincidentally executing concurrently. However, this is highly not recommended nor it is portable. There is a proposal (NOTE: https://github.com/common-workflow-language/cwltool#running-mpi-based-tools-that-need-to-be-launched) for an addition to a future version of the CWL standards of a "MPIRequirement’ that is available via a special flag in the CWL reference runner that achieves a similar functionality for a single CommandLineTool that should be executed on many nodes concurrently with intra-node communication set up by a system compliant with the MPI standard. 84 | 85 | * Pattern 15 (Task to Environment - Push), Pattern 16 (Environment to Task - Pull), Pattern 19 (Case to Environment - Push), Pattern 20 (Environment to Case - Pull), Pattern 23 (Workflow to Environment - Push), Pattern 24 (Environment to Workflow - Pull) 86 | 87 | No. This is not a feature of CWL. A CommandLineTool Process could communicate with an external service or resource via IP networking, but this is not explicitly supported by CWL and is not recommended. Interactions with stateful services are features of other workflow languages, like the now-defunct (NOTE: https://lists.apache.org/thread.html/r19322d54fd6aae5778aff46717dea2fbd37c3b64571300ad9cee0191%40%3Cdev.taverna.apache.org%3E 88 | ) Taverna, but they entail significant complexity and implementation costs to handle error states and other common challenges. 89 | 90 | * Pattern 17 (Environment to Task - Push), Pattern 18 (Task to Environment - Pull), Pattern 21 (Environment to Case - Push), Pattern 22 (Case to Environment - Pull), Pattern 25 (Environment to Workflow - Push), Pattern 26 (Workflow to Environment - Pull) 91 | 92 | No. While the "NetworkAccess" requirement enables a CommandLineTool to be marked as requiring network (IP) access the CWL standards state that “Enabling network access does not imply a publically routable IP address or the ability to accept inbound connections.” 93 | 94 | * Pattern 27 (Data Transfer by Value - Incoming) 95 | 96 | "Workflow components are able to accept data elements passed to them as value": Yes. The Processes connected to each step in a CWL workflow receive data by value to each of their required input parameters and zero or more of their optional input parameters. 97 | 98 | * Pattern 28 (Data Transfer by Value - Outgoing) 99 | 100 | "Workflow components are able to pass data elements to subsequent components by value": Yes, the results from a CWL Process are available as named outputs for connecting to the inputs of other Processes (and as final Workflow outputs). 101 | 102 | * Pattern 29 (Data Transfer - Copy In/Copy Out) 103 | 104 | No. In CWL we do not overwrite prior data except when the optional ‘InplaceUpdateRequirement’ is used. Even then there isn’t a copy, the specified Files and Directories are directly modified by the CommandLineTool. 105 | 106 | * Pattern 30 (Data Transfer by Reference - Unlocked), Pattern 31 (Data Transfer by Reference - With Lock) 107 | 108 | No. While the CWL object model does use IRIs/URIs ‘locations’ to identify specific File and Directories, those underlying bitstreams are invariant unless ‘InplaceUpdateRequirement’ is used. Those ‘locations’ are transformed to local file paths just prior to CommandLineTool execution. 109 | 110 | * Pattern 32 (Data Transformation - Input) 111 | 112 | Yes. There are many opportunities to transform data in CWL. In CWL Workflow step definitions once can extract a subset or perform another transformation using the ‘valueFrom’ field and a CWL expression or CWL parameter reference. 113 | 114 | * Pattern 33 (Data Transformation - Output) 115 | 116 | No. As a workaround, another Process (ExpressionTool, CommandLineTool, or sub-Workflow) can be used to modify an output before further use. Within a CommandLineTool Process the ‘outputEval’ field can do modifications. 117 | 118 | * Pattern 34 (Task Precondition - Data Existence) 119 | 120 | "Direct precondition support for evaluation data element existence at task instance level“: Yes. This is the basis for task dependency in CWL. 121 | 122 | * Pattern 35 (Task Precondition - Data Value) 123 | 124 | No. Once a Process has completed then all of its named outputs are available to sibling Workflow steps. While these can be evaluated using the new `when` field, this only decides if execution can take place. It does not delay execution as there is no capacity to update the inputs later; only to make new inputs under different steps. 125 | 126 | * Pattern 36 (Task Postcondition - Data Existence), Pattern 37 (Task Postcondition - Data Value) 127 | 128 | No. CommandLineTools and ExpressionTools terminate when their underlying tools finish execution. (sub-)Workflows terminate when all steps required for the outputs have finished. While the outputs can be examined in a CWL Expression and an exception thrown if they do not meet the given requirements, this does not cause continued execution or re-execution, but a permanent failure of the Process. 129 | 130 | * Pattern 38 (Event-Based Task Trigger) , Pattern 39 (Data-Based Task Trigger) 131 | 132 | No, this is not part of the CWL standards. But another system could initiate the execution of a CWL Process based upon an event outside the CWL Process description itself (or via an unofficial extension to the CWL standards). 133 | 134 | * Pattern 40 (Data-Based Routing) 135 | 136 | * "Any data element accessible at case level can be utilised in a routing construct". Any output from a sibling CWL step can be used to decide if a CWL step should be executed, yes. 137 | “Direct workflow support”: Yes, using the “when” field in a CWL workflow step definition. 138 | 139 | "Support for both exclusive choice and multi-choice constructs" These can be emulated, but not directly enforced. 140 | 141 | -------------------------------------------------------------------------------- /workflow_patterns_initiative/event/README.md: -------------------------------------------------------------------------------- 1 | 2 | # WPI's [Event Log Imperfection Patterns](http://www.workflowpatterns.com/patterns/logimperfection/) and CWL 3 | 4 | Of the 11 WPI Event Log Imperfection patterns, 0 patterns are supported by CWL v1.2 or earlier and 11 patterns are unsupported. Most of the patterns below are more like "anti-patterns" which should be avoided as they make process mining analysis more complicated. 5 | 6 | CWL specification doesn't define any special mechanisms for capturing and saving event logs. Redirecting `stdout` and `stderr` streams to files is not part of the event logs system. Some of the Event Log Imperfection patterns may still occur on the workflow management system level. 7 | 8 | [Back to the list of WPI pattern categories](../README.md) 9 | 10 | 11 | * :negative_squared_cross_mark: [Pattern 1 (Form-based Event Capture)](http://www.workflowpatterns.com/patterns/logimperfection/elp1.php) 12 | 13 | *Data in event log is captured from an electronic form. All data items captured on the form have the same timestamp.* 14 | 15 | **No.** 16 | 17 | * :negative_squared_cross_mark: [Pattern 2 (Inadvertent Time Travel)](http://www.workflowpatterns.com/patterns/logimperfection/elp2.php) 18 | 19 | *Entries in a log are recorded with an incorrect timestamp due to the ‘proximity’ of the correct value to the erroneously recorded value.* 20 | 21 | **No.** However, this pattern may occur on the workflow management system level. As an example, for a long time Airflow (and, as a result, CWL-Airflow too) internally used only UTC time zone. Users working in a different time zone could have potential difficulties in interpreting event logs. 22 | 23 | * :negative_squared_cross_mark: [Pattern 3 (Unanchored Event)](http://www.workflowpatterns.com/patterns/logimperfection/elp3.php) 24 | 25 | *The timestamp values of an event log are recorded in a format different from that which is expected by the tools used to process the event log.* 26 | 27 | **No.** However, this pattern may occur on the workflow management system level. As an example, Airflow allows to define custom format for event log messages. This may bring potential problems when parsing event log files. 28 | 29 | * :negative_squared_cross_mark: [Pattern 4 (Scattered Event)](http://www.workflowpatterns.com/patterns/logimperfection/elp4.php) 30 | 31 | *This pattern refers to events in an event log which have attributes that contain further information that can be used to derive new events.* 32 | 33 | **No.** However, this pattern may occur on the workflow management system level. As an example, by default CWL-Airflow allows several retries for the failed task. Every task execution results in creation of the separate event log file, however they are all still accessible by the retry attempt number. 34 | 35 | * :negative_squared_cross_mark: [Pattern 5 (Elusive Case)](http://www.workflowpatterns.com/patterns/logimperfection/elp5.php) 36 | 37 | *Events in a log are not explicitly linked to their case identifiers.* 38 | 39 | **No.** However, this pattern may occur on the workflow management system level. As an example, for bioinformatics workflows the case identifier will be the job description object that defines input data for workflow run. If event logs omit that information, users may have problems with linking the log events to their case identifiers. 40 | 41 | * :negative_squared_cross_mark: [Pattern 6 (Scattered Case)](http://www.workflowpatterns.com/patterns/logimperfection/elp6.php) 42 | 43 | *Key process steps are missing in the event log, but they are recorded elsewhere, thus bringing a potential 'record linkage' problem.* 44 | 45 | **No.** However, this pattern may occur on the workflow management system level. As an example, if CWL-Airflow used SubDags to run subworkflows, their event logs would be disconnected from the event log of the main workflow. 46 | 47 | * :negative_squared_cross_mark: [Pattern 7 (Collateral Events)](http://www.workflowpatterns.com/patterns/logimperfection/elp7.php) 48 | 49 | *Multiple events in a log that refer to a single process step within a case.* 50 | 51 | **No.** However, this pattern may occur on the workflow management system level. As an example, jobs from the scattered workflow step report events in parallel. 52 | 53 | * :negative_squared_cross_mark: [Pattern 8 (Polluted Label)](http://www.workflowpatterns.com/patterns/logimperfection/elp8.php) 54 | 55 | *This pattern refers to the presence of a group of event attribute values that are structurally the same, yet are distinct from each other due to differences in the exact attribute values that further qualifies the meaning of the value.* 56 | 57 | **No.** However, this pattern may occur on the workflow management system level. As an example, workflow run identifier may include workflow name and some unique identifier. All the events related to multiple runs of the same workflow will include both immutable (workflow name) and mutable (unique identifier) parts. This may potentially result in over-fitting the event log as there will be many specific activities that should have been firstly abstracted out. 58 | 59 | * :negative_squared_cross_mark: [Pattern 9 (Distorted Label)](http://www.workflowpatterns.com/patterns/logimperfection/elp9.php) 60 | 61 | *This pattern refers to the existence of two or more values of an event attribute that are not an exact match with each other but have strong similarities syntactically and semantically.* 62 | 63 | **No.** 64 | 65 | * :negative_squared_cross_mark: [Pattern 10 (Synonymous Labels)](http://www.workflowpatterns.com/patterns/logimperfection/elp10.php) 66 | 67 | *The synonymous label pattern refers to a situation whereby there is a group of values that are syntactically different but semantically similar.* 68 | 69 | **No.** 70 | 71 | * :negative_squared_cross_mark: [Pattern 11 (Homonymous Label)](http://www.workflowpatterns.com/patterns/logimperfection/elp11.php) 72 | 73 | *This pattern describes a situation where an activity is repeated multiple times within a case, but the interpretation of the activity, from a process perspective, differs between first and subsequent occurrences.* 74 | 75 | **No.** However, this pattern may occur on the workflow management system level. As an example, jobs from the scattered workflow step report events with the same identifier and as a result are mistakenly grouped together. -------------------------------------------------------------------------------- /workflow_patterns_initiative/exception/README.md: -------------------------------------------------------------------------------- 1 | # WPI's [Exception Handling Patterns](http://www.workflowpatterns.com/patterns/exception/) and CWL 2 | 3 | Broadly speaking, the CWL standards have little support for exception handling. 4 | 5 | [Back to the list of WPI pattern categories](../README.md) 6 | 7 | ## [WPI Exception Types](http://www.workflowpatterns.com/patterns/exception/exceptiontypes.php) 8 | 9 | ### "Work Item Failure" 10 | 11 | `CommandLineTool`s can define what exit codes represent successful execution, 12 | or temporary or permanenet task execution failures. While there isn't a specific 13 | CWL construct to verify the correct execution of a previous step, one can 14 | evaluate any outputs (or even the workflow level inputs provided by the user) 15 | and halt execution if they do not satisfy some criteria. 16 | 17 | Starting with CWL v1.2 there can be alternative and/or supplementary execution 18 | paths to cope with a step that ran without error but did not produce some 19 | desired output or meet a particular threshold. However, as of CWL v1.2, there 20 | is no ability in CWL to `loop` over a remediation step until a particular type 21 | of result is obtained. 22 | 23 | ### "Deadline Expiry" 24 | 25 | CWL has no concept of a task deadline. 26 | 27 | Starting with CWL v1.1 it is possible to annotate CWL `CommandLineTool`s with a 28 | [`ToolTimeLimit`](https://www.commonwl.org/v1.1/CommandLineTool.html#ToolTimeLimit) 29 | that specified the maximum computation time that may pass once execution of a 30 | `CommandLineTool` has begun.. 31 | However the result of missing the time limit is a permanent failure and CWL 32 | does have any automatic recovery mechanism for permanent failures. 33 | 34 | ### "Resource Unavailability" 35 | 36 | While the lack of needed resources can happen to a CWL workflow execution, 37 | there is no CWL construct to automatically scale or otherwise attempt to 38 | recover from such a situation. The providers of CWL compatible workflow 39 | execution are allowed to implement their own solutions, otherwise manual 40 | recovery is required. 41 | 42 | ### "External Trigger" 43 | 44 | There is no concept of triggers in CWL. 45 | 46 | ### "Constraint Violation" 47 | 48 | Beyond validation of the workflow description and the type compatibility 49 | between processes, the only mechanism to check for constraint violation is 50 | `pickValue`. `pickValue` appeared in version 1.2 of the CWL standards to 51 | primarily cope with the results of conditional execution using the `when` 52 | modifier on Workflow steps. 53 | 54 | ## ["Exception Handling at Work Item Level"](http://www.workflowpatterns.com/patterns/exception/exception_workitemlevel.php) patterns 55 | 56 | There are no CWL constructs corresponding to any of the WPI exception handling 57 | at the work item level patterns. However, many of these techniques are 58 | used automatically by many CWL executors. 59 | 60 | 61 | -------------------------------------------------------------------------------- /workflow_patterns_initiative/presentation/README.md: -------------------------------------------------------------------------------- 1 | 2 | # WPI's [Workflow Presentation Patterns](http://www.workflowpatterns.com/patterns/presentation/) and CWL 3 | 4 | Of the 20 WPI Workflow Presentation patterns, 19 patterns are supported by CWL v1.2 or earlier and 1 patterns is unsupported. 5 | 6 | Note, as CWL specificaltion does not have any graphical notation, the Concrete Syntax Patterns will highly depend on the platform responsible for rendering the visual representation of the workflow. 7 | 8 | [Back to the list of WPI pattern categories](../README.md) 9 | ## Abstract Syntax Patterns 10 | 11 | The abstract syntax of a process model relates to the formal structure of process elements and the relationships among them. 12 | 13 | ![](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/images/fig2.jpg) 14 | 15 | * :white_check_mark: [Pattern 1 (Block Structuring)](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/asm1.php) 16 | 17 | *In a block-structured process model, each split element has a corresponding join element of the same type, and split-join pairs are properly nested.* 18 | 19 | **Yes.** A combination of `MultipleInputFeatureRequirement`, `StepInputExpressionRequirement`, and `InlineJavascriptRequirement` allows to implement proper split and join elements, thus creating a block-structured process model. 20 | 21 | * :white_check_mark: [Pattern 2 (Duplication)](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/asm2.php) 22 | 23 | *Introduces controlled redundancy in a process model by repeating model elements.* 24 | 25 | **Yes.** If necessary, a contolled redundancy can be achieved by having several steps that run exactly the same subworkflow or command line tool, but are connected to the different upstream tasks. Additionally, `$import` and `$include` CWL directives can be used. 26 | 27 | * :white_check_mark: [Pattern 3 (Compacting)](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/asm3.php) 28 | 29 | *The pattern refers to methods to remove redundant or superfluous elements in a process model. Thus, it may revert the effects of Duplication.* 30 | 31 | **Yes.** The redundant step can be easily removed from the workflow. If the remaining step needs to combine outputs from the several upstream tasks, `MultipleInputFeatureRequirement` and `InlineJavascriptRequirement` can be used. 32 | 33 | * :white_check_mark: [Pattern 4 (Vertical Modularization)](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/asm4.php) 34 | 35 | *The pattern captures features to decompose a model into vertical modules, i.e. subprocesses, according to a hierarchical structure.* 36 | 37 | **Yes.** CWL allows to reuse command line tools and subworkflows multiple times within the same main workflow. 38 | 39 | * :white_check_mark: [Pattern 5 (Horizontal Modularization)](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/asm5.php) 40 | 41 | *The pattern captures features to partition a process model into smaller and more easily manageable peer modules. Hence, to facilitate collaboration, and to foster reuse of modules within the same process model or across different process models.* 42 | 43 | **Yes.** CWL workflows and command line tools follow the principles of portability, interoperability, and reusability, thus can be reused within the same or multiple pipelines. Workflows and command line tools developed and maintained by different teams can be parts of the same pipeline. 44 | 45 | * :white_check_mark: [Pattern 6 (Orthogonal Modularization)](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/asm6.php) 46 | 47 | *The pattern captures features to decompose a process model along the crosscutting concerns of the modeling domain, which are scattered across several model elements or modules.* 48 | 49 | **Yes.** If necessary, a group of steps that belong to a certain modeling domain can be put into a subworkflow and reused multiple times within the same main workflow. As an example, all the steps related to QC analysis of genomic data can be placed into a separate subworkflow. 50 | 51 | * :white_check_mark: [Pattern 7 (Composition)](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/asm7.php) 52 | 53 | *The pattern describes features for constructing a consolidated process model from different disjoint modules, thus possibly reversing the effects of modularization.* 54 | 55 | **Yes.** Any subworkflow can be replaced by a set of command line tools and other subworkflows it consists of. The logic of the main pipeline will remain the same. 56 | 57 | * :white_check_mark: [Pattern 8 (Merging)](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/asm8.php) 58 | 59 | *The pattern describes features for merging similar process models based on their commonalities, i.e. their identical model elements.* 60 | 61 | **Yes.** Workflows of the same or similar functionality can be merged into a single more complex pipeline. For additional functionality extra inputs can be added. 62 | 63 | * :white_check_mark: [Pattern 9 (Omission)](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/asm9.php) 64 | 65 | *The pattern captures features to remove one or more elements from a process model and reconnect the remaining ones. It implies information lost.* 66 | 67 | **Yes.** Unused tasks can be removed from the workflow as long as all broken connections between the remaining tasks are restored. 68 | 69 | * :white_check_mark: [Pattern 10 (Collapse)](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/asm10.php) 70 | 71 | *The pattern describes features to synthesize multiple model elements into a single one of more abstract nature, where the distinction among the constituent elements is no longer relevant. It implies information synthesis.* 72 | 73 | **Yes.** Workflow steps can be grouped together and put into a separate subworkflow, thus making the process model more abstract and more understandable. 74 | 75 | * :white_check_mark: [Pattern 11 (Restriction)](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/asm11.php) 76 | 77 | *The pattern captures features to restrict the syntax and semantics of a process modeling language, by removing modeling concepts from the language's meta-model or restricting the existing ones.* 78 | 79 | **Yes.** Certain syntax, semantic and functional restrictions can be applied when omitting 80 | `SubworkflowFeatureRequirement`, `ScatterFeatureRequirement`, `MultipleInputFeatureRequirement`, `StepInputExpressionRequirement`, `InplaceUpdateRequirement`, `InlineJavascriptRequirement`, `SchemaDefRequirement`, etc. 81 | 82 | * :white_check_mark: [Pattern 12 (Extension)](http://www.workflowpatterns.com/patterns/presentation/abstractsyntax/asm12.php) 83 | 84 | *The pattern captures features to extend the syntax and the semantics of a process modeling language, by adding new modeling concepts to the language's meta-model, or refining the existing ones* 85 | 86 | **Yes.** Implementation extensions not required for correct execution and metadata may be provided as additional fields on any object. Such extensions fields must use a namespace prefix listed in the workflow's `$namespaces` section. 87 | ## Concrete Syntax Patterns 88 | The concrete syntax of a process models deals with its visual appearance including symbols, colors and positions. 89 | 90 | ![](http://www.workflowpatterns.com/patterns/presentation/concretesyntax/images/Fig01.jpg) 91 | 92 | * :white_check_mark: [Pattern 1 (Layout Guidance)](http://www.workflowpatterns.com/patterns/presentation/concretesyntax/csm1.php) 93 | 94 | *The pattern refers to the availability of layout conventions or advice to organize the various model elements on a canvas.* 95 | 96 | **Yes.** CWL allows to provide additional metadata for any object. As an example, Seven Bridges platform extends the description of each model element with `sbg:x` and `sbg:y` fields to define the locations of the items on canvas. 97 | 98 | * :white_check_mark: [Pattern 2 (Enclosure Highlight)](http://www.workflowpatterns.com/patterns/presentation/concretesyntax/csm2.php) 99 | 100 | *The pattern refers to the availability of modeling constructs to visually enclose a set of logically-related model elements.* 101 | 102 | **Yes.** CWL allows to provide additional metadata for any object. As an example, SciDAP platform extends the descriptions of workflow inputs and outputs with `sd:layout` and `sd:visualPlugins` fields to group logically-related model elements. 103 | 104 | * :white_check_mark: [Pattern 3 (Graphical Highlight)](http://www.workflowpatterns.com/patterns/presentation/concretesyntax/csm3.php) 105 | 106 | *The pattern refers to the availability of features to change the visual appearance of model elements, such as shape, line thickness and type, background color, font type and color.* 107 | 108 | **Yes.** CWL allows to provide additional metadata for any object. That, if necessary, can be used to define visual appearence of the model elements. 109 | 110 | * :white_check_mark: [Pattern 4 (Pictorial Annotation)](http://www.workflowpatterns.com/patterns/presentation/concretesyntax/csm4.php) 111 | 112 | *The pattern denotes the availability of features to assign pictorial elements, such as icons or images, to modeling elements.* 113 | 114 | **Yes.** CWL allows to provide additional metadata for any object. That, if necessary, can be used to assign pictorial elements, such as icons or images, to modeling elements. 115 | 116 | * :white_check_mark: [Pattern 5 (Textual Annotation)](http://www.workflowpatterns.com/patterns/presentation/concretesyntax/csm5.php) 117 | 118 | *The pattern denotes the availability of features to visually represent free-form text in the canvas, which can be attached to modeling elements without changing semantics.* 119 | 120 | **Yes.** CWL allows to use human-readable labels and documentation strings set in the `label` and `doc` fields. In addition, any CWL object can be extended with the extra metadata fields. 121 | 122 | * :white_check_mark: [Pattern 6 (Explicit Representation)](http://www.workflowpatterns.com/patterns/presentation/concretesyntax/csm6.php) 123 | 124 | *The pattern denotes the ability to capture process modeling concepts via a dedicated graphical notation.* 125 | 126 | **Yes.** CWL allows to provide additional metadata for any object. That, if necessary, can be used to capture and highlight process modeling concepts. 127 | 128 | * :negative_squared_cross_mark: [Pattern 7 (Alternative Representation)](http://www.workflowpatterns.com/patterns/presentation/concretesyntax/csm7.php) 129 | 130 | *The pattern denotes the ability to capture process modeling concepts without the use of their primary graphical notation.* 131 | 132 | **No.** Even if CWL allows to provide additional metadata for any object, it will be more like an *Explicit Representation* rather then *Alternative Representation*. 133 | 134 | * :white_check_mark: [Pattern 8 (Naming Guidance)](http://www.workflowpatterns.com/patterns/presentation/concretesyntax/csm8.php) 135 | 136 | *The pattern refers to the availability of naming conventions or advice for model elements’ labels.* 137 | 138 | **Yes.** CWL supports any human-readable labels and documentation strings set in the `label` and `doc` fields of the model elements. -------------------------------------------------------------------------------- /workflow_patterns_initiative/resource/README.md: -------------------------------------------------------------------------------- 1 | # WPI's [Workflow Resource Patterns](http://www.workflowpatterns.com/patterns/resource/) and CWL 2 | 3 | Note: Most of the WPI resource handling patterns concern the allocation of business 4 | tasks to human resources. 5 | 6 | The CWL standards define a language to describe batch computation tasks and 7 | workflows made from these tasks. CWL purposely does not include non-computational 8 | tasks. Distribution of tasks in CWL is up to the implementing workflow engine to 9 | comply with any specified computational resource requirements (which are mandatory) 10 | or hints (which are optional). 11 | 12 | While only 2 of the 43 of the WPI Resource Patterns have an analogue in the 13 | CWL standards, many of these other patterns are used by CWL compatible 14 | workflow management systems; either controlled by algorithmic choices or 15 | user/operator input. 16 | 17 | [Back to the list of WPI pattern categories](../README.md) 18 | 19 | ## "Creation Patterns" 20 | 21 | * [Pattern 1 (Direct Distribution)](http://www.workflowpatterns.com/patterns/resource/creation/wrp1.php) 22 | 23 | No. There is no concept of a (computational) resource identity in the CWL standards. 24 | There is no method to specify that a given task should be execution on a specific computer. 25 | 26 | * [Pattern 2 (Role-Based Distribution)](http://www.workflowpatterns.com/patterns/resource/creation/wrp2.php) 27 | 28 | No. 29 | 30 | * [Pattern 3 (Deferred Distribution)](http://www.workflowpatterns.com/patterns/resource/creation/wrp3.php) 31 | 32 | No. The choice of up front scheduling, deferred scheduling, or hybrid 33 | scheduling is not a concept in CWL. The CWL standards do not specify a 34 | scheduling modality nor do they forbid any scheduling modality. 35 | 36 | * [Pattern 4 (Authorization)](http://www.workflowpatterns.com/patterns/resource/creation/wrp4.php) 37 | 38 | No. 39 | 40 | * [Pattern 5 (Separation of Duties)](http://www.workflowpatterns.com/patterns/resource/creation/wrp5.php) 41 | 42 | No. 43 | 44 | * [Pattern 6 (Case Handling)](http://www.workflowpatterns.com/patterns/resource/creation/wrp6.php) 45 | 46 | No. The CWL standards have no concept of a case. 47 | 48 | * [Pattern 7 (Retain Familiar)](http://www.workflowpatterns.com/patterns/resource/creation/wrp7.php) 49 | 50 | No. However a workflow engine is allowed to schedule/place jobs to take 51 | advantage of data locality as an optimization. 52 | 53 | * [Pattern 8 (Capability-Based Distribution)](http://www.workflowpatterns.com/patterns/resource/creation/wrp8.php) 54 | 55 | Yes, except there is no concept of "organisational model" in CWL. 56 | CWL `CommandLineTool`s can define minimum needed data storage and memory needs, along with the 57 | minimum number of CPU cores for efficient computation. These `ResourceRequirements` 58 | influence task placement. 59 | 60 | CWL versions: all 61 | 62 | * [Pattern 9 (History-Based Distribution)](http://www.workflowpatterns.com/patterns/resource/creation/wrp9.php) 63 | 64 | No. Like Pattern 7 (Retain Faimiliar), this could be implemented by a workflow 65 | engine to optimize based upon the availability of intermediate results left 66 | over (or nearby) by a previous task. 67 | 68 | * [Pattern 10 (Organisational Distribution)](http://www.workflowpatterns.com/patterns/resource/creation/wrp10.php) 69 | 70 | No. CWL has no concept of an "organization" or any entity similar to that. 71 | 72 | * [Pattern 11 (Automatic Execution)](http://www.workflowpatterns.com/patterns/resource/creation/wrp11.php) 73 | 74 | By definition, all tasks in CWL are automatic as CWL has no concept of sending a task to a human. 75 | 76 | CWL versions: all 77 | 78 | ## "Push Patterns" 79 | 80 | * [Pattern 12 (Distribution by Offer - Single Resource)](http://www.workflowpatterns.com/patterns/resource/push/wrp12.php) 81 | 82 | No. 83 | 84 | * [Pattern 13 (Distribution by Offer - Multiple Resources)](http://www.workflowpatterns.com/patterns/resource/push/wrp13.php) 85 | 86 | No. 87 | 88 | * [Pattern 14 (Distribution by Allocation - Single Resource)](http://www.workflowpatterns.com/patterns/resource/push/wrp14.php) 89 | 90 | No. 91 | 92 | * [Pattern 15 (Random Allocation)](http://www.workflowpatterns.com/patterns/resource/push/wrp15.php) 93 | 94 | No. 95 | 96 | * [Pattern 16 (Round Robin Allocation)](http://www.workflowpatterns.com/patterns/resource/push/wrp16.php) 97 | 98 | No. 99 | 100 | * [Pattern 17 (Shortest Queue)](http://www.workflowpatterns.com/patterns/resource/push/wrp17.php) 101 | 102 | No. 103 | 104 | * [Pattern 18 (Early Distribution)](http://www.workflowpatterns.com/patterns/resource/push/wrp18.php) 105 | 106 | No. 107 | 108 | * [Pattern 19 (Distribution on Enablement)](http://www.workflowpatterns.com/patterns/resource/push/wrp19.php) 109 | 110 | No. 111 | 112 | * [Pattern 20 (Late Distribution)](http://www.workflowpatterns.com/patterns/resource/push/wrp20.php) 113 | 114 | No. 115 | 116 | ## "Pull Patterns" 117 | 118 | * [Pattern 21 (Resource-Initiated Allocation)](http://www.workflowpatterns.com/patterns/resource/pull/wrp21.php) 119 | 120 | No. 121 | 122 | * [Pattern 22 (Resource-Initiated Execution - Allocated Work Item](http://www.workflowpatterns.com/patterns/resource/pull/wrp22.php) 123 | 124 | No. 125 | 126 | * [Pattern 23 (Resource-Initiated Execution - Offered Work Item)](http://www.workflowpatterns.com/patterns/resource/pull/wrp23.php) 127 | 128 | No. 129 | 130 | * [Pattern 24 (System-Determined Work Queue Content)](http://www.workflowpatterns.com/patterns/resource/pull/wrp24.php) 131 | 132 | No. 133 | 134 | * [Pattern 25 (Resource-Determined Work Queue Content)](http://www.workflowpatterns.com/patterns/resource/pull/wrp25.php) 135 | 136 | No. 137 | 138 | * [Pattern 26 (Selection Autonomy)](http://www.workflowpatterns.com/patterns/resource/pull/wrp26.php) 139 | 140 | No. 141 | 142 | * [Pattern 21 (Resource-Initiated Allocation)](http://www.workflowpatterns.com/patterns/resource/pull/wrp21.php) 143 | 144 | No. 145 | 146 | ## "Detour Patterns" 147 | 148 | * [Pattern 27 (Delegation)](http://www.workflowpatterns.com/patterns/resource/detour/wrp27.php) 149 | 150 | No. 151 | 152 | * [Pattern 28 (Escalation)](http://www.workflowpatterns.com/patterns/resource/detour/wrp28.php) 153 | 154 | No. 155 | 156 | * [Pattern 29 (Deallocation)](http://www.workflowpatterns.com/patterns/resource/detour/wrp29.php) 157 | 158 | No. 159 | 160 | * [Pattern 30 (Stateful Reallocation)](http://www.workflowpatterns.com/patterns/resource/detour/wrp30.php) 161 | 162 | No. 163 | 164 | * [Pattern 31 (Stateless Reallocation)](http://www.workflowpatterns.com/patterns/resource/detour/wrp31.php) 165 | 166 | No. 167 | 168 | * [Pattern 32 (Suspension/Resumption)](http://www.workflowpatterns.com/patterns/resource/detour/wrp32.php) 169 | 170 | No. 171 | 172 | * [Pattern 33 (Skip)](http://www.workflowpatterns.com/patterns/resource/detour/wrp33.php) 173 | 174 | No. 175 | 176 | * [Pattern 34 (Redo)](http://www.workflowpatterns.com/patterns/resource/detour/wrp34.php) 177 | 178 | No. There is no method in CWL for a task to specify that some *other* 179 | already completed task must be redone. 180 | 181 | However, if a `CommandLineTool` exits with an return code indicated in the specified 182 | `temporaryFailCodes` list, then the task is available for retrying again. 183 | The CWL standards do not specify how many retries (if any) should be attempted. 184 | 185 | * [Pattern 35 (Pre-Do)](http://www.workflowpatterns.com/patterns/resource/detour/wrp35.php) 186 | 187 | No. In CWL it is assumed that once a task has all of its inputs it is made 188 | available for execution. 189 | 190 | ## "Auto-Start Patterns" 191 | 192 | * [Pattern 36 (Commencement on Creation)](http://www.workflowpatterns.com/patterns/resource/autostart/wrp36.php) 193 | 194 | No. 195 | 196 | * [Pattern 37 (Commencement on Allocation)](http://www.workflowpatterns.com/patterns/resource/autostart/wrp37.php) 197 | 198 | No. 199 | 200 | * [Pattern 38 (Piled Execution)](http://www.workflowpatterns.com/patterns/resource/autostart/wrp38.php) 201 | 202 | No. There is no specific construct in CWL to force pipelined execution, 203 | however the presence of `streamable: true` on matching inputs/outputs of 204 | adjacent tasks is a strong hint for those system who wish to do so. 205 | 206 | * [Pattern 39 (Chained Execution)](http://www.workflowpatterns.com/patterns/resource/autostart/wrp39.php) 207 | 208 | No. 209 | 210 | ## "Visibility Patterns" 211 | 212 | * [Pattern 40 (Configurable Unallocated Work Item Visibility)](http://www.workflowpatterns.com/patterns/resource/visibility/wrp40.php) 213 | 214 | No. 215 | 216 | * [Pattern 41 (Configurable Allocated Work Item Visibility)](http://www.workflowpatterns.com/patterns/resource/visibility/wrp41.php) 217 | 218 | No. 219 | 220 | ## "Multiple Resource Patterns" 221 | 222 | * [Pattern 42 (Simultaneous Execution)](http://www.workflowpatterns.com/patterns/resource/multiple_resources/wrp42.php) 223 | 224 | No, there is no explicit CWL construct for this. It is assumed that parallel 225 | execution, when possible, is wanted by the user. All known CWL implementations 226 | support parallel execution, and almost all of them do so by default. 227 | 228 | * [Pattern 43 (Additional Resources)](http://www.workflowpatterns.com/patterns/resource/multiple_resources/wrp43.php) 229 | 230 | No. 231 | --------------------------------------------------------------------------------