├── .gitignore
├── LICENSE
├── Makefile
├── README.md
├── pipelines.nimble
├── pipelines
    ├── pipelines
    └── pipelines.nim
└── tests
    ├── ages.pipeline
    ├── ages.py
    ├── ages_data.csv
    ├── ages_utils.py
    ├── fizzbuzz.pipeline
    ├── fizzbuzz.py
    ├── fizzbuzz_client.py
    └── fizzbuzz_utils.py


/.gitignore:
--------------------------------------------------------------------------------
1 | piipelines/pipelines
2 | pipelines/nimcache/
3 | *.pyc
4 | *i~
5 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2019 Caleb Winston
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/Makefile:
--------------------------------------------------------------------------------
1 | all:
2 | 	nimble install


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | <p align="center">
  2 | <!-- <img width="250px" src="https://i.imgur.com/kTap42K.png"/> -->
  3 |     <img height="90px" src="https://i.imgur.com/rbx2Hlh.png"/><!--https://i.imgur.com/YfK7YdY.png-->
  4 | </p>
  5 | <!--- https://i.imgur.com/rbx2Hlh.png or https://i.imgur.com/YfK7YdY.png) --->
  6 | <!--- https://carbon.now.sh/?bg=rgba(239%2C228%2C176%2C1)&t=zenburn&wt=none&l=python&ds=true&dsyoff=20px&dsblur=68px&wc=false&wa=true&pv=56px&ph=56px&ln=false&fm=Ubuntu%20Mono&fs=17px&lh=136%25&si=false&code=from%2520utils%2520import%2520customers%2520%2520%2520%2520%2520%2520%2520%2520%2520%2520%2520%2520%2520as%2520customers%2520%2523%2520a%2520generator%2520function%2520in%2520the%2520utils%2520module%250Afrom%2520utils%2520import%2520parse_row%2520%2520%2520%2520%2520%2520%2520%2520%2520%2520%2520%2520%2520as%2520parser%250Afrom%2520utils%2520import%2520get_recommendations%2520%2520%2520as%2520recommender%250Afrom%2520utils%2520import%2520print_recommendations%2520as%2520printer%250A%250Acustomers%2520%257C%253E%2520parser%2520%257C%253E%2520recommender%2520%257C%253E%2520printer&es=2x&wm=false --->
  7 | 
  8 | Pipelines is a language and runtime for crafting massively parallel pipelines. Unlike other languages for defining data flow, the Pipeline language requires implementation of components to be defined separately in the Python scripting language. This allows the details of implementations to be separated from the structure of the pipeline, while providing access to thousands of active libraries for machine learning, data analysis and processing. Skip to [Getting Started](https://github.com/calebwin/pipelines#some-next-steps) to install the Pipeline compiler.
  9 | 
 10 | ### An example
 11 | 
 12 | As an introductory example, a simple pipeline for Fizz Buzz on even numbers could be written as follows -
 13 | 
 14 | ```python
 15 | from fizzbuzz import numbers
 16 | from fizzbuzz import even
 17 | from fizzbuzz import fizzbuzz
 18 | from fizzbuzz import printer
 19 | 
 20 | numbers
 21 | /> even 
 22 | |> fizzbuzz where (number=*, fizz="Fizz", buzz="Buzz")
 23 | |> printer
 24 | ```
 25 | 
 26 | Meanwhile, the implementation of the components would be written in Python -
 27 | 
 28 | ```python
 29 | def numbers():
 30 |     for number in range(1, 100):
 31 |         yield number
 32 | 
 33 | def even(number):
 34 |     return number % 2 == 0
 35 | 
 36 | def fizzbuzz(number, fizz, buzz):
 37 |     if number % 15 == 0: return fizz + buzz
 38 |     elif number % 3 == 0: return fizz
 39 |     elif number % 5 == 0: return buzz
 40 |     else: return number
 41 | 
 42 | def printer(number):
 43 |     print(number)
 44 | ```
 45 | 
 46 | Running the Pipeline document would safely execute each component of the pipeline in parallel and output the expected result.
 47 | 
 48 | ### The imports
 49 | 
 50 | Components are scripted in Python and linked into a pipeline using imports. The syntax for an import has 3 parts - (1) the path to the module, (2) the name of the function, and (3) the alias for the component. Here's an example -
 51 | ```python
 52 | from parser import parse_fasta as parse
 53 | ```
 54 | That's really all there is to imports. Once a component is imported it can be referenced anywhere in the document with the alias.
 55 | 
 56 | ### The stream
 57 | 
 58 | Every pipeline is operated on a stream of data. The stream of data is created by a Python [generator](https://docs.python.org/3/tutorial/classes.html#generators). The following is an example of a generator that generates a stream of numbers from 0 to 1000.
 59 | ```python
 60 | def numbers():
 61 |     for number in range(0, 1000):
 62 |         yield number
 63 | ```
 64 | Here's a generator that reads entries from a file
 65 | ```python
 66 | def customers():
 67 |     for line in open("customers.csv", 'r'):
 68 |         yield line
 69 | ```
 70 | The first component in a pipeline is always the generator. The generator is run in parallel with all other components and each element of data is passed through the other components.
 71 | ```python
 72 | from utils import customers             as customers # a generator function in the utils module
 73 | from utils import parse_row             as parser
 74 | from utils import get_recommendations   as recommender
 75 | from utils import print_recommendations as printer
 76 | 
 77 | customers |> parser |> recommender |> printer
 78 | ```
 79 | 
 80 | ### The pipes
 81 | 
 82 | Pipes are what connect components together to form a pipeline. As of now, there are 2 types of pipes in the Pipeline language - (1) transformer pipes, and (2) filter pipes. Transformer pipes are used when input is to be passed through a component. For example, a function can be defined to determine the potential of a particle and a function can be defined to print the potential.
 83 | ```python
 84 | particles |> get_potential |> printer
 85 | ```
 86 | The above pipeline code would pass data from the stream generated by `particles` through `get_potential` and then the output of `get_potential` through `printer`. Filter pipes work similarly except they use the following component to filter data. For example, a function can be defined to determine if a person is over 50 and then print their names to a file.
 87 | ```python
 88 | population /> over_50 |> printer
 89 | ```
 90 | This would use the function referenced by `over_50` to filter out data from the stream generated by `population` and then pass output to `printer`.
 91 | 
 92 | ### The `where` keyword
 93 | 
 94 | The `where` keyword lets you pass in multiple parameters to a component as opposed to just what the output from the previous component was. For example, a function can be defined to print to a file the names of all applicants under a certain age.
 95 | ```python
 96 | applicants
 97 | |> printer where (person=*, age_limit=21)
 98 | ```
 99 | This could be done using a filter as well.
100 | ```python
101 | applicants
102 | /> age_limit where (person=*, age=21)
103 | |> printer
104 | ```
105 | In this case, the function for `age_limit` could look something like this -
106 | ```python
107 | def age_limit(person, age):
108 |     return person.age <= age
109 | ```
110 | Note that this function still has just one return value - the boolean expression that is used to determine wether input to the component is passed on as output.
111 | 
112 | ### The `to` keyword
113 | The `to` keyword is for when you want the previous component has multiple return values and you want to specify which ones to pass on to the next component. As an example, if you had a function for calculating the electronegativity and electron affinity of an atom, you could use it in a pipeline as follows -
114 | ```python
115 | atoms
116 | |> calculator to (electronegativity, electron_affinity)
117 | |> printer where (line=electronegativity)
118 | ```
119 | Here's an example using a filter.
120 | ```python
121 | atoms
122 | /> below where (atom=*, limit=2) to (is_below, electronegativity, electron_affinity) with is_below
123 | |> printer where (line=electronegativity)
124 | ```
125 | Note the use of the `with` keyword here. This is necessary for filters to specify which return value of the function is used to filter out elements in the stream.
126 | 
127 | ### Getting started
128 | All you need to get started is the Pipelines compiler. You can install it by downloading the executable from [Releases](https://github.com/calebwin/pipelines/releases).
129 | > If you have the [Nimble](https://github.com/nim-lang/nimble/) package manager installed and `~/.nimble/bin` permanantly added to your PATH environment variable (look this up > if you don't know how to do this), you can also install by running the following command.
130 | > ```
131 | > nimble install pipelines
132 | > ```
133 | Pipelines' only dependency is [the Python interpreter](https://www.python.org/downloads/release/python-2715/) being installed on your system. At the moment, most versions 2.7 and earlier are supported and support for Python 3 is in the works. Once Pipelines is installed and added to your PATH, you can create a `.pipeline` file, run or compile anywhere on your system -
134 | ```console
135 | $ pipelines
136 | the .pipeline compiler (v:0.1.0)
137 | 
138 | usage:
139 |   pipelines                Show this
140 |   pipelines <file>         Compile .pipeline file
141 |   pipelines <folder>       Compile all .pipeline files in folder
142 |   pipelines run <file>     Run .pipeline file
143 |   pipelines clean <folder> Remove all compiled .py files from folder
144 | 
145 | for more info, go to github.com/calebwin/pipelines
146 | ```
147 | 
148 | ### Some next steps
149 | 
150 | There are several things I'm hoping to implement in the future for this project. I'm hoping to implement some sort of `and` operator for piping data from the stream into multiple components in parallel with the output ending up in the stream in a nondeterministic order. Further down the line, I plan on porting the whole thing to C and putting in a complete error handling system
151 | <!---
152 | - String imports
153 | - Control allocation of processes with Pool
154 | - Use Pipe instead of multiple Queue
155 | - Only have num_cpus running at one time
156 | --->
157 | 


--------------------------------------------------------------------------------
/pipelines.nimble:
--------------------------------------------------------------------------------
 1 | # Package
 2 | 
 3 | version       = "0.1.2"
 4 | author        = "calebwin"
 5 | description   = "A tiny framework & language for crafting massively parallel data pipelines"
 6 | license       = "MIT"
 7 | bin           = @["pipelines/pipelines"]
 8 | 
 9 | # Dependencies
10 | 
11 | requires "nim >= 0.17.2", "python >= 0.0.0"
12 | 


--------------------------------------------------------------------------------
/pipelines/pipelines:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/calebwin/pipelines/e623d2d03e42467eb50dd1882e3864108fbff0ed/pipelines/pipelines


--------------------------------------------------------------------------------
/pipelines/pipelines.nim:
--------------------------------------------------------------------------------
  1 | import python # TODO support python3
  2 | import os, strutils, sequtils, tables
  3 | 
  4 | const debug = false
  5 | 
  6 | type
  7 |   Path = tuple[module: string, function: string] # path to a function - defined by name of module and name of function in it
  8 |   Pipe = tuple[origin: string, modifiers: tuple[mWhere: string, mTo: string, mWith: string], destination: string, pipeType: Pipes] # pipe between 2 components - defined by origin component and destination component
  9 |   Pipes = enum
 10 |     pTransformer, pFilter
 11 | 
 12 | # compiles pipeline document at given path to python code
 13 | # TODO parallelize compilation
 14 | proc compile*(path: string): string =
 15 |   # get contents of file
 16 |   let contents: string = readFile(path)
 17 | 
 18 |   # (1) parse contents of file
 19 | 
 20 |   var
 21 |     paths: Table[string, Path] = initTable[string, Path]() # table mapping alias to path
 22 |     pipes: seq[Pipe] = @[] # pipes connecting components
 23 | 
 24 |   # get tokens from contents
 25 |   var 
 26 |     tokens: seq[string] = @[]
 27 |     token: string = ""
 28 |     index: int = 0
 29 | 
 30 |   # iterate through characters in contents and tokenize
 31 |   while index <= contents.len - 1:
 32 |     let character: string = $contents[index] # get character at current index
 33 | 
 34 |     # handle character being an end-of-token character
 35 |     if character.isNilOrWhitespace or character == "(" or character == "#":
 36 |       if token.strip().len > 0: 
 37 |         tokens.add(token.strip()) # add token to tokens if current character is an end-of-token-character
 38 |       token = ""
 39 | 
 40 |     # handle character being start of parameter declaration
 41 |     if character == "(":
 42 |       let parameterDeclaration: string = contents[index .. contents.find(')', index)] # get whole parameter declaration
 43 |       tokens.add(parameterDeclaration)
 44 |       index = contents.find(')', index) + 1 # move to end of parameter declaration
 45 |     # handle character being start of comment
 46 |     elif character == "#":
 47 |       index = contents.find(NewLines, index) + 1 # move to end of comment
 48 |     else:
 49 |       token &= character # otherwise, append character to token
 50 |       # the character is only appended if it's not the end of a token or the start of a parameter declaration
 51 |       index += 1 # move to next character
 52 | 
 53 |   # parse tokens to paths and pipes
 54 |   var tokenIndex: int = 0
 55 |   for token in tokens:
 56 |     case token:
 57 |     of "import":
 58 |       # parse import statement
 59 |       let
 60 |         newPathModule: string = tokens[tokenIndex - 1] # module for new path
 61 |         newPathFunction: string = tokens[tokenIndex + 1] # function for new path
 62 |         newPathAlias: string = if tokens[tokenIndex + 1 + 1] == "as": tokens[tokenIndex + 1 + 2] else: newPathFunction # alias of new path
 63 |         newPath: Path = (module : newPathModule, function : newPathFunction) # new path
 64 | 
 65 |       # add new path to paths
 66 |       paths[newPathAlias] = newPath
 67 |     of "|>":
 68 |       # parse tranformer pipe statement
 69 |       let
 70 |         prevToken: string = tokens[tokenIndex - 1] # previous token from contents
 71 |         nextToken: string = tokens[tokenIndex + 1] # next token from contnets
 72 |         newPipeOrigin: string = prevToken # origin of new pipe
 73 |         newPipeDestination: string = nextToken # destination of new pipe
 74 | 
 75 |       # extract where, to, with modifiers
 76 |       var
 77 |         numModifiers: int = 0
 78 |         modifiers: tuple[mWhere: string, mTo: string, mWith: string] = (mWhere : "", mTo : "", mWith : "")
 79 | 
 80 |       if numModifiers == 0 and tokenIndex + 3 <= tokens.len - 1:
 81 |         numModifiers += 1 # increment number of modifiers 
 82 |         case tokens[tokenIndex + 2]:
 83 |         of "where": modifiers.mWhere = tokens[tokenIndex + 3]
 84 |         of "to": modifiers.mTo = tokens[tokenIndex + 3]
 85 |         of "with": modifiers.mWith = tokens[tokenIndex + 3]
 86 | 
 87 |       if numModifiers == 1 and tokenIndex + 5 <= tokens.len - 1:
 88 |         numModifiers += 1 # increment number of modifiers
 89 |         case tokens[tokenIndex + 4]:
 90 |         of "where": modifiers.mWhere = tokens[tokenIndex + 5]
 91 |         of "to": modifiers.mTo = tokens[tokenIndex + 5]
 92 |         of "with": modifiers.mWith = tokens[tokenIndex + 5]
 93 | 
 94 |       if numModifiers == 2 and tokenIndex + 7 <= tokens.len - 1:
 95 |         numModifiers += 1 # increment number of modifiers
 96 |         case tokens[tokenIndex + 6]:
 97 |         of "where": modifiers.mWhere = tokens[tokenIndex + 7]
 98 |         of "to": modifiers.mTo = tokens[tokenIndex + 7]
 99 |         of "with": modifiers.mWith = tokens[tokenIndex + 7]
100 | 
101 |       # add new pipe
102 |       pipes.add((origin : newPipeOrigin, modifiers : modifiers, destination : newPipeDestination, pipeType : pTransformer))
103 |     of "/>":
104 |       # parse tranformer pipe statement
105 |       let
106 |         prevToken: string = tokens[tokenIndex - 1] # previous token from contents
107 |         nextToken: string = tokens[tokenIndex + 1] # next token from contnets
108 |         newPipeOrigin: string = prevToken # origin of new pipe
109 |         newPipeDestination: string = nextToken # destination of new pipe
110 | 
111 |       # extract where, to, with modifiers
112 |       var
113 |         numModifiers: int = 0
114 |         modifiers: tuple[mWhere: string, mTo: string, mWith: string] = (mWhere : "", mTo : "", mWith : "")
115 | 
116 |       if numModifiers == 0 and tokenIndex + 3 <= tokens.len - 1:
117 |         numModifiers += 1 # increment number of modifiers 
118 |         case tokens[tokenIndex + 2]:
119 |         of "where": modifiers.mWhere = tokens[tokenIndex + 3]
120 |         of "to": modifiers.mTo = tokens[tokenIndex + 3]
121 |         of "with": modifiers.mWith = tokens[tokenIndex + 3]
122 |         else: numModifiers -= 1 # undo increment
123 | 
124 |       if numModifiers == 1 and tokenIndex + 5 <= tokens.len - 1:
125 |         numModifiers += 1 # increment number of modifiers
126 |         case tokens[tokenIndex + 4]:
127 |         of "where": modifiers.mWhere = tokens[tokenIndex + 5]
128 |         of "to": modifiers.mTo = tokens[tokenIndex + 5]
129 |         of "with": modifiers.mWith = tokens[tokenIndex + 5]
130 |         else: numModifiers -= 1 # undo increment
131 | 
132 |       if numModifiers == 2 and tokenIndex + 7 <= tokens.len - 1:
133 |         numModifiers += 1 # increment number of modifiers
134 |         case tokens[tokenIndex + 6]:
135 |         of "where": modifiers.mWhere = tokens[tokenIndex + 7]
136 |         of "to": modifiers.mTo = tokens[tokenIndex + 7]
137 |         of "with": modifiers.mWith = tokens[tokenIndex + 7]
138 |         else: numModifiers -= 1 # undo increment
139 | 
140 |       # add new pipe
141 |       pipes.add((origin : newPipeOrigin, modifiers : modifiers, destination : newPipeDestination, pipeType : pFilter))
142 |     else:
143 |       discard
144 | 
145 |     # update index
146 |     tokenIndex += 1
147 | 
148 |   # (2) generate target code
149 | 
150 |   # initialize code with import statements
151 |   var
152 |     code: string = "from multiprocessing import Process, Queue\n"
153 |     mainCode: string = ""
154 | 
155 |   # IMPORTS
156 | 
157 |   # import functions
158 |   for alias, path in pairs(paths):
159 |     code &= "from " & path.module & " import " & path.function & " as " & alias & "\n"
160 | 
161 |   # FUNCTIONS
162 | 
163 |   # define pipe sentinel
164 |   code &= "class PLPipeSentinel: pass\n"
165 | 
166 |   # define function to run generator
167 |   let generatorName: string = pipes[0].origin
168 | 
169 |   code &= "def pl_run_" & generatorName & "(pl_stream, pl_out_queue):\n"
170 |   code &= "\tfor pl_data in pl_stream:\n"
171 |   code &= "\t\tpl_out_queue.put(pl_data)\n"
172 |   code &= "\tpl_out_queue.put(PLPipeSentinel())\n"
173 | 
174 |   # define functions for process for each component
175 |   var componentIndex: int = 0
176 |   for pipe in pipes:
177 |     let 
178 |       component: string = pipe.destination # destination of pipe ~ the component to define a function for
179 |       modifiers: tuple[mWhere: string, mTo: string, mWith: string] = pipe.modifiers # modifiers of pipe
180 |       parameters: string = if modifiers.mWhere.len > 0: modifiers.mWhere.replace("*", "pl_inp") else: "(pl_inp)" # parameters of pipe
181 |       inputs: string = if componentIndex > 0: pipes[componentIndex - 1].modifiers.mTo.replace("(", "").replace(")", "") else: "" # names of inputs into current compoent
182 |       # TODO handle with modifier
183 | 
184 |     var componentCode: string = "" # code for running component
185 | 
186 |     # initialize to variables to None
187 |     for to in modifiers.mTo.replace("(", "").replace(")", "").split(","):
188 |       componentCode &= to.strip() & " = None\n"
189 | 
190 |     # loop indefinitely
191 |     componentCode &= "while 1:\n"
192 | 
193 |     # block and get next element from in queue`
194 |     componentCode &= "\tpl_inp = pl_in_queue.get()\n"
195 | 
196 |     # unpackaget tuple input and reate local variables to access inputs if necessary
197 |     # this is only done when input is from a transformer pipe (filter pipes always have only one input)
198 |     if inputs != "" and pipes[componentIndex - 1].pipeType == pTransformer:
199 |       componentCode &= "\t" & inputs & " = pl_inp\n"
200 | 
201 |     # TODO use unique sentinel value
202 |     # check if element from in queue is sentinel value
203 |     componentCode &= "\tif isinstance(pl_inp, PLPipeSentinel):\n"
204 |     # put sentinel value in queue to next component
205 |     componentCode &= "\t\tpl_outp = PLPipeSentinel()\n"
206 | 
207 |     case pipe.pipeType:
208 |     of pTransformer:
209 |       # get output from passing element into component if element is not sentinel
210 |       componentCode &= "\tif not isinstance(pl_inp, PLPipeSentinel):\n"
211 |       componentCode &= "\t\tpl_outp = " & component & parameters & "\n"
212 |       # unpackage output and update to variables
213 |       if modifiers.mTo.len > 0:
214 |         componentCode &= "\t\t" & modifiers.mTo.replace("(", "").replace(")", "") & " = pl_outp\n"
215 |     of pFilter:
216 |       # get output from passing element into component if element is not sentinel
217 |       componentCode &= "\tif not isinstance(pl_inp, PLPipeSentinel):\n"
218 |       componentCode &= "\t\tpl_result = " & component & parameters & "\n"
219 |       if modifiers.mTo.len > 0:
220 |         componentCode &= "\t\t" & modifiers.mTo.replace("(", "").replace(")", "") & " = pl_result\n" # unpackage output and update to variables
221 |       if modifiers.mWith.len > 0:
222 |         componentCode &= "\t\tif " & modifiers.mWith.replace("(", "").replace(")", "") & ":\n" # send input through filter  
223 |       else:
224 |         componentCode &= "\t\tif pl_result:\n" # send input through filter
225 |       componentCode &= "\t\t\tpl_outp = pl_inp\n" # pass on input as output
226 |       componentCode &= "\t\telse:\n"
227 |       componentCode &= "\t\t\tcontinue\n" # otherwise continue to next input
228 | 
229 |     # check if queue to next component exists
230 |     componentCode &= "\tif pl_out_queue is not None:\n"
231 | 
232 |     # put output into queue to next component if queue to next component exists
233 |     componentCode &= "\t\tpl_out_queue.put(pl_outp)\n"
234 | 
235 |     # break if element from in queue is sentinel value
236 |     componentCode &= "\tif isinstance(pl_inp, PLPipeSentinel):\n"
237 |     componentCode &= "\t\tbreak\n"
238 | 
239 |     # update index of component
240 |     componentIndex += 1
241 | 
242 |     # append component code to code
243 |     code &= "def pl_run_" & component & "(pl_in_queue, pl_out_queue):\n" # function header
244 |     componentCode.removeSuffix("\n") # remove last newline
245 |     code &= componentCode.indent(1, "\t") & "\n" # indent and add newline
246 | 
247 |   # MAIN
248 | 
249 |   # get iterator over stream of data
250 |   let iteratorModule: string = pipes[0].origin
251 |   mainCode &= "pl_data = " & iteratorModule & "()\n"
252 | 
253 |   # create queues into components
254 |   for pipe in pipes:
255 |     let component: string = pipe.destination # component to create a queue into
256 | 
257 |     # create queue into component
258 |     mainCode &= "pl_in_" & component & " = Queue()\n"
259 | 
260 |   # create process for generator
261 |   mainCode &= "pl_" & generatorName & "_process = Process(target=pl_run_" & generatorName & ", args=(pl_data, pl_in_" & pipes[0].destination & "))\n"
262 | 
263 |   # create processes for each component
264 |   var pipeIndex: int = 0
265 |   for pipe in pipes:
266 |     let 
267 |       component: string = pipes[pipeIndex].destination # component to create process for
268 |       componentQueue: string = "pl_in_" & component # queue to component to create process for
269 |       nextComponentQueue: string = if pipeIndex < pipes.len - 1: "pl_in_" & pipes[pipeIndex + 1].destination else: "None" # queue to next component in pipeline
270 | 
271 |     # creat process for component passing in queue to the component and queue to next component
272 |     mainCode &= "pl_" & component & "_process = Process(target=pl_run_" & component & ", args=(" & componentQueue  & ","  & nextComponentQueue & ",))\n"
273 | 
274 |     # update index in pipes
275 |     pipeIndex += 1
276 | 
277 |   # start generator process
278 |   mainCode &= "pl_" & generatorName & "_process.start()\n"
279 | 
280 |   # start processes
281 |   for pipe in pipes:
282 |     let component: string = pipe.destination # component to start process of
283 | 
284 |     # start process of component
285 |     mainCode &= "pl_" & component & "_process.start()\n"
286 | 
287 |   # block main process till all process have finished
288 |   for pipe in pipes:
289 |     let component: string = pipe.destination # component to join process of
290 | 
291 |     # join process of component
292 |     mainCode &= "pl_" & component & "_process.join()\n"
293 | 
294 |   # append main code to code
295 |   code &= "if __name__ == \"__main__\":\n"
296 |   mainCode.removeSuffix("\n") # remove last newline
297 |   code &= mainCode.indent(1, "\t") & "\n" # indent main code and add newline
298 | 
299 |   # TODO append main code to code as "execute" method
300 |   code &= "def execute():\n"
301 |   mainCode.removeSuffix("\n") # remove last newline
302 |   code &= mainCode.indent(1, "\t") & "\n" # indent main code and add newline
303 | 
304 |   # return code
305 |   result = code
306 | 
307 |   if debug:
308 |     echo(code)
309 |  
310 | # runs pipeline document at given path
311 | proc runFile*(path: string) =
312 |   # get compiled Python code
313 |   let code: cstring = compile(path)
314 | 
315 |   # initialize the Python interpreter
316 |   initialize()
317 | 
318 |   # add directory containing .pipeline file to PYTHONPATH
319 |   when defined windows:
320 |     syssetpath($getPath() & ";" & path[0 .. path.rfind("/")])
321 |   else:
322 |     syssetpath($getPath() & ":" & path[0 .. path.rfind("/")])
323 | 
324 |   # run code
325 |   discard runSimpleString(code)
326 | 
327 |   # finalize the Python interpreter
328 |   finalize()
329 | 
330 | # compiles pipeline document at given path to python file at same path with ,py file extension
331 | proc compileFile*(path: string) =
332 |   # compile pipeline document at given path
333 |   let code: string = compile(path)
334 | 
335 |   # get path to new python file
336 |   let targetPath: string = path.replace(".pipeline", ".py")
337 | 
338 |   # print code to file at target path
339 |   writeFile(targetPath, code)
340 | 
341 | proc main() =
342 |   case paramCount():
343 |   of 0:
344 |     # print welcome message
345 |     echo("the .pipeline compiler (v:0.1.0)\n")
346 | 
347 |     # print usage
348 |     echo("usage:")
349 |     echo("  pipelines                Show this")
350 |     echo("  pipelines <file>         Compile .pipeline file")
351 |     echo("  pipelines <folder>       Compile all .pipeline files in folder")
352 |     echo("  pipelines run <file>     Run .pipeline file")
353 |     echo("  pipelines clean <folder> Remove all compiled .py files from folder\n")
354 | 
355 |     # print info
356 |     echo("for more info, go to github.com/calebwin/pipelines")
357 |   of 1:
358 |     # get path to file to run
359 |     let path: string = paramStr(1)
360 | 
361 |     # files compiled
362 |     var filesCompiled: seq[string] = @[]
363 | 
364 |     # run file
365 |     if fileExists(path):
366 |       compileFile(path)
367 |       filesCompiled.add(path)
368 |     elif dirExists(path):
369 |       for kind, file in walkDir(path):
370 |         if file.endsWith(".pipeline"):
371 |           compileFile(file)
372 |           filesCompiled.add(file)
373 |     else:
374 |       echo(path & " not found")
375 | 
376 |     # print info
377 |     echo($filesCompiled.len & " " & (if filesCompiled.len == 1: "file" else: "files") & " compiled" & (if dirExists(path): " at " & path else: "") & ":")
378 |     for file in filesCompiled:
379 |       echo("  " & file)
380 |   of 2:
381 |     case paramStr(1):
382 |     of "r", "run":
383 |       # get path to file to run
384 |       let path: string = paramStr(2)
385 | 
386 |       # run file
387 |       if existsFile(path):
388 |         runFile(path)
389 |       elif existsDir(path):
390 |         for kind, file in walkDir(path):
391 |           if file.endsWith(".pipeline"):
392 |             runFile(file)
393 |       else:
394 |         echo(path & " not found")
395 |     of "c", "clean":
396 |       # get path to folder to clean
397 |       let path: string = paramStr(2)
398 | 
399 |       # number of files removed
400 |       var filesRemoved: seq[string] = @[]
401 | 
402 |       # clean folder
403 |       for kind, file in walkDir(path):
404 |         # compiled .py file
405 |         let compiledFile: string = if file.endsWith(".pipeline"): file.replace(".pipeline", ".py") else: ""
406 | 
407 |         # remove compiled file if it exists
408 |         if compiledFile != "":
409 |           removeFile(compiledFile)
410 |           filesRemoved.add(compiledFile)
411 | 
412 |       # print info
413 |       echo($filesRemoved.len & " " & (if filesRemoved.len == 1: "file" else: "files") & " removed from " & path & ":")
414 |       for file in filesRemoved:
415 |         echo("  " & file)
416 |   else:
417 |     discard
418 | 
419 | main()


--------------------------------------------------------------------------------
/tests/ages.pipeline:
--------------------------------------------------------------------------------
1 | from ages_utils import ages
2 | from ages_utils import age_segment
3 | from ages_utils import is_under_age
4 | from ages_utils import print_age
5 | 
6 | # TODO fix error where you need to have a line at end
7 | 
8 | ages |> age_segment /> is_under_age |> print_age
9 | 


--------------------------------------------------------------------------------
/tests/ages.py:
--------------------------------------------------------------------------------
 1 | from multiprocessing import Process, Queue
 2 | from ages_utils import age_segment as age_segment
 3 | from ages_utils import is_under_age as is_under_age
 4 | from ages_utils import ages as ages
 5 | from ages_utils import print_age as print_age
 6 | class PLPipeSentinel: pass
 7 | def run_ages(stream, out_queue):
 8 | 	for data in stream:
 9 | 		out_queue.put(data)
10 | 	out_queue.put(PLPipeSentinel())
11 | def run_age_segment(in_queue, out_queue):
12 | 	while 1:
13 | 		inp = in_queue.get()
14 | 		if isinstance(inp, PLPipeSentinel):
15 | 			outp = PLPipeSentinel()
16 | 		if not isinstance(inp, PLPipeSentinel):
17 | 			outp = age_segment(inp)
18 | 		if out_queue is not None:
19 | 			out_queue.put(outp)
20 | 		if isinstance(inp, PLPipeSentinel):
21 | 			break
22 | def run_is_under_age(in_queue, out_queue):
23 | 	while 1:
24 | 		inp = in_queue.get()
25 | 		if isinstance(inp, PLPipeSentinel):
26 | 			outp = PLPipeSentinel()
27 | 		if not isinstance(inp, PLPipeSentinel):
28 | 			result = is_under_age(inp)
29 | 			if result:
30 | 				outp = inp
31 | 			else:
32 | 				continue
33 | 		if out_queue is not None:
34 | 			out_queue.put(outp)
35 | 		if isinstance(inp, PLPipeSentinel):
36 | 			break
37 | def run_print_age(in_queue, out_queue):
38 | 	while 1:
39 | 		inp = in_queue.get()
40 | 		if isinstance(inp, PLPipeSentinel):
41 | 			outp = PLPipeSentinel()
42 | 		if not isinstance(inp, PLPipeSentinel):
43 | 			outp = print_age(inp)
44 | 		if out_queue is not None:
45 | 			out_queue.put(outp)
46 | 		if isinstance(inp, PLPipeSentinel):
47 | 			break
48 | if __name__ == "__main__":
49 | 	data = ages()
50 | 	in_age_segment = Queue()
51 | 	in_is_under_age = Queue()
52 | 	in_print_age = Queue()
53 | 	ages_process = Process(target=run_ages, args=(data, in_age_segment))
54 | 	age_segment_process = Process(target=run_age_segment, args=(in_age_segment,in_is_under_age,))
55 | 	is_under_age_process = Process(target=run_is_under_age, args=(in_is_under_age,in_print_age,))
56 | 	print_age_process = Process(target=run_print_age, args=(in_print_age,None,))
57 | 	ages_process.start()
58 | 	age_segment_process.start()
59 | 	is_under_age_process.start()
60 | 	print_age_process.start()
61 | 	age_segment_process.join()
62 | 	is_under_age_process.join()
63 | 	print_age_process.join()
64 | 


--------------------------------------------------------------------------------
/tests/ages_data.csv:
--------------------------------------------------------------------------------
 1 | 5
 2 | 19
 3 | 89
 4 | 26
 5 | 28
 6 | 48
 7 | 79
 8 | 48
 9 | 20
10 | 34
11 | 41
12 | 39
13 | 69
14 | 73
15 | 29
16 | 28
17 | 13
18 | 5
19 | 7
20 | 9


--------------------------------------------------------------------------------
/tests/ages_utils.py:
--------------------------------------------------------------------------------
 1 | def ages():
 2 | 	# read lines of data
 3 | 	lines = []
 4 | 	with open('ages_data.csv') as f:
 5 | 		lines = f.read().splitlines()
 6 | 
 7 | 	# get numbers from strings
 8 | 	ages = [int(age) for age in lines]
 9 | 
10 | 	for age in ages:
11 | 		yield age
12 | 
13 | def age_segment(age):
14 |     if age < 12: return 0
15 |     elif age < 18: return 1
16 |     elif age < 28: return 2
17 |     elif age < 48: return 3
18 |     elif age < 68: return 4
19 |     elif age < 88: return 5
20 |     else: return 6
21 | 
22 | def is_under_age(age_segment):
23 | 	if age_segment <= 1:
24 | 	    return True
25 | 	return False
26 | 
27 | def print_age(age_segment):
28 |     if age_segment == 0: print("0-12")
29 |     elif age_segment == 1: print("12-18")
30 |     elif age_segment == 2: print("18-18")
31 |     elif age_segment == 3: print("28-48")
32 |     elif age_segment == 4: print("48-68")
33 |     elif age_segment == 5: print("68-88")
34 |     elif age_segment == 6: print("88+")


--------------------------------------------------------------------------------
/tests/fizzbuzz.pipeline:
--------------------------------------------------------------------------------
 1 | from fizzbuzz_utils import numbers  # a sequence of numbers
 2 | from fizzbuzz_utils import even     # a filter for even numbers
 3 | from fizzbuzz_utils import fizzbuzz # fizzbuzz
 4 | from fizzbuzz_utils import printer  # a printer
 5 | 
 6 | numbers
 7 | /> even     where (number=*, counter=count) to (is_even, count) with is_even
 8 | |> fizzbuzz where (number=*, fizz="fizz", buzz="buzz") to (number)
 9 | |> printer  where (number=number)
10 | 


--------------------------------------------------------------------------------
/tests/fizzbuzz.py:
--------------------------------------------------------------------------------
 1 | from multiprocessing import Process, Queue
 2 | from fizzbuzz_utils import fizzbuzz as fizzbuzz
 3 | from fizzbuzz_utils import printer as printer
 4 | from fizzbuzz_utils import even as even
 5 | from fizzbuzz_utils import numbers as numbers
 6 | class PLPipeSentinel: pass
 7 | def pl_run_numbers(pl_stream, pl_out_queue):
 8 | 	for pl_data in pl_stream:
 9 | 		pl_out_queue.put(pl_data)
10 | 	pl_out_queue.put(PLPipeSentinel())
11 | def pl_run_even(pl_in_queue, pl_out_queue):
12 | 	is_even = None
13 | 	count = None
14 | 	while 1:
15 | 		pl_inp = pl_in_queue.get()
16 | 		if isinstance(pl_inp, PLPipeSentinel):
17 | 			pl_outp = PLPipeSentinel()
18 | 		if not isinstance(pl_inp, PLPipeSentinel):
19 | 			pl_result = even(number=pl_inp, counter=count)
20 | 			is_even, count = pl_result
21 | 			if is_even:
22 | 				pl_outp = pl_inp
23 | 			else:
24 | 				continue
25 | 		if pl_out_queue is not None:
26 | 			pl_out_queue.put(pl_outp)
27 | 		if isinstance(pl_inp, PLPipeSentinel):
28 | 			break
29 | def pl_run_fizzbuzz(pl_in_queue, pl_out_queue):
30 | 	number = None
31 | 	while 1:
32 | 		pl_inp = pl_in_queue.get()
33 | 		if isinstance(pl_inp, PLPipeSentinel):
34 | 			pl_outp = PLPipeSentinel()
35 | 		if not isinstance(pl_inp, PLPipeSentinel):
36 | 			pl_outp = fizzbuzz(number=pl_inp, fizz="fizz", buzz="buzz")
37 | 			number = pl_outp
38 | 		if pl_out_queue is not None:
39 | 			pl_out_queue.put(pl_outp)
40 | 		if isinstance(pl_inp, PLPipeSentinel):
41 | 			break
42 | def pl_run_printer(pl_in_queue, pl_out_queue):
43 | 	while 1:
44 | 		pl_inp = pl_in_queue.get()
45 | 		number = pl_inp
46 | 		if isinstance(pl_inp, PLPipeSentinel):
47 | 			pl_outp = PLPipeSentinel()
48 | 		if not isinstance(pl_inp, PLPipeSentinel):
49 | 			pl_outp = printer(number=number)
50 | 		if pl_out_queue is not None:
51 | 			pl_out_queue.put(pl_outp)
52 | 		if isinstance(pl_inp, PLPipeSentinel):
53 | 			break
54 | if __name__ == "__main__":
55 | 	pl_data = numbers()
56 | 	pl_in_even = Queue()
57 | 	pl_in_fizzbuzz = Queue()
58 | 	pl_in_printer = Queue()
59 | 	pl_numbers_process = Process(target=pl_run_numbers, args=(pl_data, pl_in_even))
60 | 	pl_even_process = Process(target=pl_run_even, args=(pl_in_even,pl_in_fizzbuzz,))
61 | 	pl_fizzbuzz_process = Process(target=pl_run_fizzbuzz, args=(pl_in_fizzbuzz,pl_in_printer,))
62 | 	pl_printer_process = Process(target=pl_run_printer, args=(pl_in_printer,None,))
63 | 	pl_numbers_process.start()
64 | 	pl_even_process.start()
65 | 	pl_fizzbuzz_process.start()
66 | 	pl_printer_process.start()
67 | 	pl_even_process.join()
68 | 	pl_fizzbuzz_process.join()
69 | 	pl_printer_process.join()
70 | 


--------------------------------------------------------------------------------
/tests/fizzbuzz_client.py:
--------------------------------------------------------------------------------
1 | import fizzbuzz
2 | 
3 | fizzbuzz.execute()


--------------------------------------------------------------------------------
/tests/fizzbuzz_utils.py:
--------------------------------------------------------------------------------
 1 | def numbers():
 2 | 	for number in range(1, 101):
 3 | 		yield number
 4 | 
 5 | def even(number, counter):
 6 | 	return number % 2 == 0, 0 if counter is None else counter + 1
 7 | 
 8 | def fizzbuzz(number, fizz, buzz):
 9 | 	if number % 15 == 0: return fizz + buzz
10 | 	elif number % 3 == 0: return fizz
11 | 	elif number % 5 == 0: return buzz
12 | 	else: return number
13 | 
14 | def printer(number):
15 | 	print(number)


--------------------------------------------------------------------------------