├── .gitignore ├── README.md ├── ffmpeg.yml ├── ffmpeg └── handler.sh └── template ├── bash-streaming ├── .dockerignore ├── Dockerfile ├── function │ └── handler.sh ├── index.sh └── template.yml └── node-streaming ├── .dockerignore ├── Dockerfile ├── function ├── handler.js └── package.json ├── index.js ├── package.json └── template.yml /.gitignore: -------------------------------------------------------------------------------- 1 | template 2 | build 3 | .secrets 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## openfaas-streaming-templates 2 | 3 | Examples of `of-watchdog` from OpenFaaS used for streaming data with: 4 | 5 | * Node.js 18 - to stream responses, either text or big blobs of binary data. 6 | * Bash - to execute arbitrary commands and bash as a "HTTP" API 7 | * ffmpeg - to produce a gif from a `mov` QuickTime file 8 | 9 | This is a real-world example requested by Luca Morandini, Data Architect at AURIN, University of Melbourne. 10 | 11 | See also: [A collection of demos for the streaming functionality of openfaas by Netflix](https://github.com/cconger/openfaas-streaming-demos) 12 | 13 | ## The examples 14 | 15 | Whilst the data does stream, there are buffers in Golang's i/o packages set at around 32-64KB, which means that you may print out `1-10000` via STDOUT, but the first output your client may receive is 1-6770, followed by the remainder up to 10000. 16 | 17 | ### Example with Node.js 18 | 19 | * Create new 20 | 21 | ``` 22 | $ faas-cli new --lang node-streaming stream-this 23 | ``` 24 | 25 | * Add code with timer 26 | 27 | ```javascript 28 | "use strict" 29 | 30 | module.exports = (context, callback) => { 31 | 32 | var count = 0; 33 | var max = 8000; 34 | 35 | var timer = setInterval(function() { 36 | process.stdout.write("Message " + count.toString()+"\n"); 37 | count++; 38 | 39 | if(count > max) { 40 | clearInterval(timer); 41 | callback(undefined, undefined); 42 | } 43 | }, 1); 44 | } 45 | ``` 46 | 47 | * Set the timeout to be more generous 48 | 49 | ```yaml 50 | version: 1.0 51 | provider: 52 | name: openfaas 53 | gateway: http://127.0.0.1:8080 54 | functions: 55 | stream-this: 56 | lang: node-streaming 57 | handler: ./stream-this 58 | image: stream-this:latest 59 | environment: 60 | write_timeout: 1m 61 | read_timeout: 1m 62 | exec_timeout: 1m 63 | ``` 64 | 65 | * Deploy 66 | 67 | ``` 68 | faas-cli up -f stream-this 69 | ``` 70 | 71 | * Invoke: 72 | 73 | ```sh 74 | curl http://127.0.0.1:8080/function/stream-this 75 | Message 0 76 | Message 1 77 | Message 2 78 | Message 3 79 | Message 4 80 | Message 5 81 | Message 6 82 | Message 7 83 | Message 8 84 | Message 9 85 | Message 10 86 | ``` 87 | 88 | Generally, I saw this take 13s, with the first buffer printing at 2500, 2nd at around 5000 and finally the 8000 message. 89 | 90 | Execute via Docker without the OpenFaaS gateway: 91 | 92 | ```sh 93 | docker run --name stream-this -d -p 8000:8000 stream-this 94 | 95 | curl localhost:8000 96 | Message 0 97 | Message 1 98 | Message 2 99 | Message 3 100 | Message 4 101 | Message 5 102 | Message 6 103 | Message 7 104 | Message 8 105 | Message 9 106 | Message 10 107 | ``` 108 | 109 | 110 | ### Example with bash: 111 | 112 | ``` 113 | faas-cli new --lang bash-streaming printr 114 | ``` 115 | 116 | * Edit `printr/handler.sh` 117 | 118 | ``` 119 | #!/bin/sh 120 | 121 | for i in $(seq 1 10000) ; do sleep 0.001 && echo $i; done 122 | ``` 123 | 124 | * Now set a bigger timeout 125 | 126 | ```yaml 127 | version: 1.0 128 | provider: 129 | name: openfaas 130 | gateway: http://127.0.0.1:8080 131 | functions: 132 | printr: 133 | lang: bash-streaming 134 | handler: ./printr 135 | image: printr:latest 136 | environment: 137 | write_timeout: 1m 138 | read_timeout: 1m 139 | exec_timeout: 1m 140 | 141 | ``` 142 | 143 | Output: 144 | 145 | ``` 146 | time curl -i http://127.0.0.1:8080/function/printr 147 | HTTP/1.1 200 OK 148 | Content-Length: 292 149 | Content-Type: application/octet-stream 150 | Date: Sat, 31 Aug 2019 09:12:37 GMT 151 | X-Call-Id: cc54c283-fc56-4c4b-9fbe-70638eb1a6dc 152 | X-Start-Time: 1567242756314135880 153 | 154 | 1 155 | 2 156 | 3 157 | ... 158 | 98 159 | 99 160 | 100 161 | ``` 162 | ### Example with ffmpeg 163 | 164 | See the ffmpeg.yml file and `./ffmpeg/handler.sh` for how this works. 165 | 166 | * Take a short video with the webcam on your MacBook Pro using QuickTime and "Record Movie" 167 | * Or download a short `mp4` with the `youtubedl` function of the Function Store i.e. `https://www.youtube.com/watch?v=eznZ0PlWGGE` 168 | 169 | ```sh 170 | faas-cli store deploy youtubedl 171 | echo https://www.youtube.com/watch?v=eznZ0PlWGGE | faas-cli invoke youtubedl > blinkt.mp4 172 | ``` 173 | 174 | * Deploy the ffmpeg function to your OpenFaaS installation 175 | 176 | Specify ffmpeg via the `build_options` mechanism. See also template.yml for pre-defined build options. 177 | 178 | ```yaml 179 | functions: 180 | ffmpeg: 181 | build_options: 182 | - ffmpeg 183 | ``` 184 | 185 | Or specify it via a build-arg: 186 | 187 | ``` 188 | faas-cli up -f ffmpeg --build-arg ADDITIONAL_PKG=ffmpeg 189 | ``` 190 | 191 | Any other Alpine Linux packages can be added to ADDITIONAL_PKG in a similar way such as `curl` 192 | 193 | Example of the script: 194 | 195 | ```sh 196 | #!/bin/sh 197 | 198 | # Create a temporary filename 199 | export nano=`date +%s%N` 200 | 201 | export OUT=/tmp/$nano.mov 202 | 203 | # Save stdin to a temp file 204 | cat - > ${OUT} 205 | 206 | # Scale down to 50% 207 | # Use format rgb24 208 | # Reduce to 5 FPS to reduce the size 209 | # Use "gif" as output format 210 | # Use "pipe:1" (STDOUT) to write the binary data 211 | ffmpeg -i ${OUT} -vf scale=iw*.5:ih*.5 -pix_fmt rgb24 -r 5 -f gif -hide_banner pipe:1 212 | 213 | # After printing to stdout, the client has received the data via streaming 214 | # Now we delete the temporary file 215 | rm ${OUT} 216 | ``` 217 | 218 | * Invoke the function 219 | * Review your gif and share with your friends on Twitter or with [@alexellisuk](https://twitter.com/alexellisuk) 220 | 221 | ```sh 222 | export OPENFAAS_URL=http://127.0.0.1:8080 223 | 224 | curl -SLsf http://$OPENFAAS_URL/function/ffmpeg --data-binary @$HOME/Desktop/my-video.mov > my-gif.gif 225 | ``` 226 | 227 | You can also limit concurrency by adding `max_inflight=1` to only allow one video to be processed at once, or up the value to whatever you feel is a sane limit like `max_inflight=10`. 228 | -------------------------------------------------------------------------------- /ffmpeg.yml: -------------------------------------------------------------------------------- 1 | version: 1.0 2 | provider: 3 | name: openfaas 4 | gateway: http://127.0.0.1:8080 5 | functions: 6 | ffmpeg: 7 | lang: bash-streaming 8 | handler: ./ffmpeg 9 | image: alexellis2/openfaas-mov-to-gif:0.0.1 10 | build_options: 11 | - ffmpeg 12 | environment: 13 | write_timeout: 2m5s 14 | read_timeout: 2m5s 15 | exec_timeout: 2m 16 | -------------------------------------------------------------------------------- /ffmpeg/handler.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | # Create a temporary filename 4 | export nano=`date +%s%N` 5 | 6 | export OUT=/tmp/$nano.mov 7 | 8 | # Save stdin to a temp file 9 | cat - > ${OUT} 10 | 11 | # Scale down to 50% 12 | # Use format rgb24 13 | # Reduce to 5 FPS to reduce the size 14 | # Use "gif" as output format 15 | # Use "pipe:1" (STDOUT) to write the binary data 16 | ffmpeg -i ${OUT} -vf scale=iw*.5:ih*.5 -pix_fmt rgb24 -r 5 -f gif -hide_banner pipe:1 17 | 18 | # After printing to stdout, the client has received the data via streaming 19 | # Now we delete the temporary file 20 | rm ${OUT} 21 | -------------------------------------------------------------------------------- /template/bash-streaming/.dockerignore: -------------------------------------------------------------------------------- 1 | */node_modules 2 | -------------------------------------------------------------------------------- /template/bash-streaming/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ghcr.io/openfaas/of-watchdog:0.9.11 as watchdog 2 | 3 | FROM alpine:3.17.3 as build 4 | 5 | ARG ADDITIONAL_PACKAGE="" 6 | 7 | COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog 8 | RUN chmod +x /usr/bin/fwatchdog 9 | 10 | RUN addgroup -S app && adduser app -S -G app 11 | 12 | RUN apk add --no-cache ${ADDITIONAL_PACKAGE} 13 | 14 | WORKDIR /root/ 15 | 16 | RUN mkdir -p /home/app 17 | 18 | # Wrapper/boot-strapper 19 | WORKDIR /home/app 20 | COPY index.sh ./ 21 | 22 | WORKDIR /home/app/function 23 | COPY function/*.sh ./ 24 | 25 | # Copy in additional function files and folders 26 | COPY --chown=app:app function/ . 27 | 28 | WORKDIR /home/app/ 29 | 30 | # chmod for tmp is for a buildkit issue (@alexellis) 31 | RUN chmod +rx -R ./function \ 32 | && chown app:app -R /home/app \ 33 | && chmod 777 /tmp 34 | 35 | USER app 36 | 37 | ENV mode="streaming" 38 | 39 | ENV fprocess="sh ./index.sh" 40 | EXPOSE 8080 41 | 42 | HEALTHCHECK --interval=3s CMD [ -e /tmp/.lock ] || exit 1 43 | 44 | CMD ["fwatchdog"] 45 | -------------------------------------------------------------------------------- /template/bash-streaming/function/handler.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | echo "Hi" 4 | -------------------------------------------------------------------------------- /template/bash-streaming/index.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | sh ./function/handler.sh 4 | 5 | -------------------------------------------------------------------------------- /template/bash-streaming/template.yml: -------------------------------------------------------------------------------- 1 | language: bash-streaming 2 | fprocess: sh ./index.sh 3 | build_options: 4 | - name: ffmpeg 5 | packages: 6 | - ffmpeg 7 | - name: curl 8 | packages: 9 | - curl 10 | -------------------------------------------------------------------------------- /template/node-streaming/.dockerignore: -------------------------------------------------------------------------------- 1 | */node_modules 2 | -------------------------------------------------------------------------------- /template/node-streaming/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ghcr.io/openfaas/of-watchdog:0.9.11 as watchdog 2 | 3 | FROM node:18-alpine as build 4 | 5 | ARG ADDITIONAL_PACKAGE="" 6 | 7 | COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog 8 | RUN chmod +x /usr/bin/fwatchdog 9 | 10 | RUN addgroup -S app && adduser app -S -G app 11 | 12 | RUN apk add --no-cache ${ADDITIONAL_PACKAGE} 13 | 14 | WORKDIR /root/ 15 | 16 | # Turn down the verbosity to default level. 17 | ENV NPM_CONFIG_LOGLEVEL warn 18 | 19 | RUN mkdir -p /home/app 20 | 21 | # Wrapper/boot-strapper 22 | WORKDIR /home/app 23 | COPY package.json ./ 24 | 25 | # This ordering means the npm installation is cached for the outer function handler. 26 | RUN npm i --production 27 | 28 | # Copy outer function handler 29 | COPY index.js ./ 30 | 31 | # COPY function node packages and install, adding this as a separate 32 | # entry allows caching of npm install runtime dependencies 33 | WORKDIR /home/app/function 34 | COPY function/*.json ./ 35 | RUN npm i --production || : 36 | 37 | # Copy in additional function files and folders 38 | COPY --chown=app:app function/ . 39 | 40 | WORKDIR /home/app/ 41 | 42 | # chmod for tmp is for a buildkit issue (@alexellis) 43 | RUN chmod +rx -R ./function \ 44 | && chown app:app -R /home/app \ 45 | && chmod 777 /tmp 46 | 47 | USER app 48 | 49 | ENV mode="streaming" 50 | 51 | ENV fprocess="node index.js" 52 | EXPOSE 8080 53 | 54 | HEALTHCHECK --interval=3s CMD [ -e /tmp/.lock ] || exit 1 55 | 56 | CMD ["fwatchdog"] 57 | -------------------------------------------------------------------------------- /template/node-streaming/function/handler.js: -------------------------------------------------------------------------------- 1 | "use strict" 2 | 3 | module.exports = (context, callback) => { 4 | callback(undefined, {"status": "done"}); 5 | } 6 | -------------------------------------------------------------------------------- /template/node-streaming/function/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "function", 3 | "version": "1.0.0", 4 | "description": "", 5 | "main": "handler.js", 6 | "scripts": { 7 | "test": "echo \"Error: no test specified\" && exit 1" 8 | }, 9 | "keywords": [], 10 | "author": "", 11 | "license": "ISC" 12 | } 13 | -------------------------------------------------------------------------------- /template/node-streaming/index.js: -------------------------------------------------------------------------------- 1 | // Copyright (c) Alex Ellis 2017. All rights reserved. 2 | // Copyright (c) OpenFaaS Author(s) 2018. All rights reserved. 3 | // Licensed under the MIT license. See LICENSE file in the project root for full license information. 4 | 5 | "use strict" 6 | 7 | const getStdin = require('get-stdin'); 8 | 9 | const handler = require('./function/handler'); 10 | 11 | getStdin().then(val => { 12 | handler(val, (err, res) => { 13 | if (err) { 14 | return console.error(err); 15 | } 16 | 17 | if (!res) { 18 | return; 19 | } 20 | 21 | if (isArray(res) || isObject(res)) { 22 | console.log(JSON.stringify(res)); 23 | } else { 24 | process.stdout.write(res); 25 | } 26 | }); 27 | }).catch(e => { 28 | console.error(e.stack); 29 | }); 30 | 31 | const isArray = (a) => { 32 | return (!!a) && (a.constructor === Array); 33 | }; 34 | 35 | const isObject = (a) => { 36 | return (!!a) && (a.constructor === Object); 37 | }; 38 | -------------------------------------------------------------------------------- /template/node-streaming/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "NodejsBase", 3 | "version": "1.0.0", 4 | "description": "", 5 | "main": "faas_index.js", 6 | "scripts": { 7 | "test": "echo \"Error: no test specified\" && exit 1" 8 | }, 9 | "keywords": [], 10 | "author": "", 11 | "license": "ISC", 12 | "dependencies": { 13 | "get-stdin": "^5.0.1" 14 | } 15 | } 16 | -------------------------------------------------------------------------------- /template/node-streaming/template.yml: -------------------------------------------------------------------------------- 1 | language: node-streaming 2 | fprocess: node index.js 3 | welcome_message: | 4 | You have created a new function which uses Node.js 18 and Alpine Linux. 5 | npm i --save can be used to add third-party packages like request or cheerio 6 | npm documentation: https://docs.npmjs.com/ 7 | build_options: 8 | - name: ffmpeg 9 | packages: 10 | - ffmpeg 11 | - name: curl 12 | packages: 13 | - curl 14 | --------------------------------------------------------------------------------