├── .gitignore ├── README.md ├── step0 ├── README.md └── simulator.sh ├── step1 ├── README.md ├── nodejs │ ├── README.md │ ├── app.js │ ├── index.js │ ├── package.json │ └── test │ │ └── functional.js └── simulator.sh ├── step10 ├── README.md ├── docker-compose.yml ├── images │ ├── 00-circleci-signup.png │ ├── 01-circleci-link-github.png │ ├── 02-circleci-add-project.png │ ├── 03-circleci-project-env-variable.png │ ├── 04-circleci-build-project.png │ ├── 05-code-push-in-github.png │ ├── 06-circle-success-build.png │ ├── 07-docker-hub-image-created.png │ ├── ci-web-services.png │ └── ci-workflow.png └── nodejs │ ├── Dockerfile │ ├── README.md │ ├── app.js │ ├── circle.yml │ ├── compose-test.yml │ ├── index.js │ ├── package.json │ ├── test.sh │ └── test │ ├── functional.js │ └── mocha.opts ├── step11 ├── README.md ├── docker-cloud.yml ├── docker-compose.yml ├── images │ ├── App-01-DB-Create.png │ ├── App-02-Dashboard.png │ ├── DockerCloud-01-CloudRegistry.png │ ├── DockerCloud-02-CI.png │ ├── DockerCloud-03-ApplicationDeployment.png │ ├── DockerCloud-04-CD.png │ ├── DockerCloud-05-Teams.png │ ├── Flow-01-GitHub.png │ ├── Flow-02-CircleCI.png │ ├── Flow-03-DockerHub.png │ ├── Flow-04-DockerCloud.png │ ├── Nodes-00_CloudProviders.png │ ├── Nodes-01-Menu.png │ ├── Nodes-02-Create.png │ ├── Nodes-03-Deploying.png │ ├── Stack-01-Create.png │ └── Stack-02-Deploy.png ├── nodejs │ ├── Dockerfile │ ├── README.md │ ├── app.js │ ├── circle.yml │ ├── compose-test-ci.yml │ ├── compose-test.yml │ ├── compose-test.yml-save │ ├── index.js │ ├── package.json │ ├── test.sh │ └── test │ │ ├── functional.js │ │ └── mocha.opts └── simulator.sh ├── step2 ├── README.md └── nodejs │ ├── Dockerfile │ ├── README.md │ ├── app.js │ ├── index.js │ ├── package.json │ └── test │ ├── functional.js │ └── mocha.opts ├── step3 ├── .gitignore ├── README.md └── images │ ├── 01-repository-creation.png │ ├── 02-repository-created.png │ └── 03-image-pushed.png ├── step4 ├── README.md ├── images │ ├── influxdb-admin.png │ ├── influxdb-query-example.png │ └── iot-database.png └── influxdb.conf ├── step5 ├── README.md ├── images │ └── docker-hub-iot-api-images.png └── nodejs │ ├── Dockerfile │ ├── README.md │ ├── app.js │ ├── index.js │ ├── package.json │ └── test │ ├── functional.js │ └── mocha.opts ├── step6 ├── .gitignore ├── README.md ├── docker-compose.yml ├── images │ └── 01-create-iot-db.png ├── influxdb.conf └── nodejs │ ├── Dockerfile │ ├── README.md │ ├── app.js │ ├── index.js │ ├── package.json │ └── test │ ├── functional.js │ └── mocha.opts ├── step7 ├── .gitignore ├── README.md ├── docker-compose.yml ├── images │ ├── 00-grafana-dashboard-example.png │ ├── 01-create-iot-db.png │ ├── 02-grafana-admin.png │ ├── 03-grafana-welcome.png │ ├── 04-grafana-datasource.png │ ├── 05-grafana-dashboard.png │ └── 06-grafana-data-samples.png ├── influxdb.conf ├── nodejs │ ├── Dockerfile │ ├── README.md │ ├── app.js │ ├── index.js │ ├── package.json │ └── test │ │ ├── functional.js │ │ └── mocha.opts └── simulator.sh ├── step8 ├── .gitignore ├── README.md ├── docker-compose.yml ├── images │ ├── 01-influxdb-db-creation.png │ └── 02-grafana-data-samples.png └── simulator.sh └── step9 ├── .gitignore ├── README.md ├── docker-compose.yml ├── images ├── 01-influxdb-db-create.png └── 02-grafana-data-samples.png └── simulator.sh /.gitignore: -------------------------------------------------------------------------------- 1 | **/node_modules 2 | **/.DS_Store 3 | **/*.log 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Purpose 2 | 3 | This project is used to illustrate the [Docker from the ground up](https://www.udemy.com/docker-from-the-ground-up/) online course. 4 | 5 | Several Docker related concepts and tools will be covers: 6 | * Dockerfile 7 | * images 8 | * volumes 9 | * Docker Compose 10 | * Docker Machine 11 | * Swarm 12 | * and some other cool stuff... 13 | 14 | To do this, we will follow several steps to create a simple application and improve it. 15 | 16 | * [Create the data simulator](./step0/) 17 | * [Create the API](./step1/) 18 | * [Containerize the API](./step2/) 19 | * [Use Docker Hub to store the API image](./step3/) 20 | * [Run InfluxDB in a container](./step4/) 21 | * [Modify the API to persist data into InfluxDB](./step5/) 22 | * [Move into a multi-containers Docker Compose application](./step6/) 23 | * [Add Grafana visualisation service](./step7/) 24 | * [Deploy the application on a remote host created with Docker Machine](./step8/) 25 | * [Deploy the application on a Docker Swarm](./step9/) 26 | * [Setup a simple Continuous Integration pipeline](./step10/) 27 | * [Setup a simple Continuous Deployment pipeline](./step11/) 28 | * and more to come... 29 | 30 | # The application 31 | 32 | The project that we will create is a backend for IoT that is to say an application used to collect data coming from IoT devices. 33 | 34 | To better understand it, let's consider the 2 sides of the project 35 | - the device that measures data and send them to a backend 36 | - the backend that collects data and exposes them through an API 37 | 38 | ## Device 39 | 40 | There are a lot of mass market IoT devices out there 41 | - [Raspberry PI](https://www.raspberrypi.org/), that comes in different flavors 42 | - [C.H.I.P](https://getchip.com/) 43 | - [Orange Pi](http://www.orangepi.org/) 44 | - ... 45 | 46 | Those devices are really cheap and embeds a Linux distribution (or if its not there, it's easy to install one). 47 | Also, a lot of sensors can be attached to those devices, making them interacting with the external world a breaze. 48 | Having those devices send information over HTTP or other lower level network protocols is very easy. 49 | 50 | In this project, we will start by using a bash script to simulate the sending of temperature information. 51 | At the end, we will setup a real device, a Raspberry PI, to get real temperature and send it to our backend. 52 | 53 | ## Backend 54 | 55 | The backend is a HTTP API that 56 | - receives the data 57 | - exposes those data to the external world 58 | 59 | The data received are stored in [InfluxDB](https://github.com/influxdata/influxdb) that is a great database dedicated to time serie data. 60 | 61 | The visualisation is done with [Grafana](http://grafana.org), an excellent tool to build metrics and analytic dashboards. 62 | 63 | The example implementation of the backend is done with Node.js but Python, Java and any other programming languages will also be used in a near future. 64 | Pull Request are welcome shall you want to provide an implementation in another language. 65 | 66 | # Status 67 | 68 | In the current version, this application is not suitable for production use but it's a good starting point to create a backend for personal or startup projects. 69 | 70 | # Prerequisites 71 | 72 | In order to start this project, you only need to have an account on [GitHub](https://github.com) and a new repository named *iot-api* (or any other name you choose). 73 | In the different steps of this project, you will be asked to modify the code within this repository. 74 | 75 | # Go ! 76 | 77 | Let's start with [the first step](./step0) and create a simple simulator in Bash. 78 | 79 | # Licence 80 | 81 | The MIT License (MIT) 82 | 83 | Copyright (c) [2017] [Luc Juggery] 84 | 85 | Permission is hereby granted, free of charge, to any person obtaining a copy 86 | of this software and associated documentation files (the "Software"), to deal 87 | in the Software without restriction, including without limitation the rights 88 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 89 | copies of the Software, and to permit persons to whom the Software is 90 | furnished to do so, subject to the following conditions: 91 | 92 | The above copyright notice and this permission notice shall be included in all 93 | copies or substantial portions of the Software. 94 | 95 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 96 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 97 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 98 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 99 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 100 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 101 | SOFTWARE. 102 | -------------------------------------------------------------------------------- /step0/README.md: -------------------------------------------------------------------------------- 1 | # Objectives 2 | 3 | Create a simulator that send temperature data in the following JSON format: 4 | 5 | ```` 6 | { 7 | ts: TIMESTAMP 8 | type: "temp" 9 | value: VALUE, 10 | sensor_id: ID_OF_THE_DEVICE_SENDING_THE_DATA 11 | } 12 | ```` 13 | 14 | For instance, a valid data could be the following one 15 | 16 | ```` 17 | { 18 | ts: "2017-03-01T23:12:52Z", 19 | type: "temp", 20 | value: 34, 21 | sensor_id: 123 22 | } 23 | ```` 24 | 25 | In the real world, those data would be sent by a real device and represent the temperature with value *34* sent by the device which id is *123* and collected on the *2017-03-01T23:12:52Z* (date in ISO8601 format). 26 | 27 | # The simulator 28 | 29 | The simulator can be done in any language, on the current example it's done in bash shell. 30 | It's a very minimal, and only 31 | * get the current date 32 | * generate a random number for the temperature (quite hot location btw) 33 | * send the data in JSON using the *curl* command (note: the targeted API will run on port 1337 as we will see in the next step) 34 | * wait 1 seconds and repeat 35 | 36 | An example of the simulator is provided in the step0 folder, you can copy it to the root of the *iot-api* folder. 37 | 38 | ----- 39 | [Next >](../step1) 40 | -------------------------------------------------------------------------------- /step0/simulator.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | while(true); do 4 | # Current date 5 | d=$(date -u +"%Y-%m-%dT%H:%M:%SZ") 6 | 7 | # Random temperature between 20 and 34°C 8 | temp=$(( ( RANDOM % 15 ) + 20 )) 9 | 10 | # Send data to API 11 | curl -XPOST -H "Content-Type: application/json" -d '{"ts":"'$d'", "type": "temp", "value": '$temp', "sensor_id": 123 }' http://localhost:1337/data 12 | 13 | sleep 1 14 | done 15 | -------------------------------------------------------------------------------- /step1/README.md: -------------------------------------------------------------------------------- 1 | # Objectives 2 | 3 | Setting up a basic HTTP API that collects time series data sent in JSON. 4 | The format of the input data is the following one: 5 | 6 | ```` 7 | { 8 | ts: TIMESTAMP 9 | type: TYPE_OF_DATA 10 | value: VALUE, 11 | sensor_id: ID_OF_THE_DEVICE_SENDING_THE_DATA 12 | } 13 | ```` 14 | 15 | Example of a valid input data: 16 | 17 | ```` 18 | { 19 | ts: "2017-03-01T23:12:52Z", 20 | type: "temp", 21 | value: 34, 22 | sensor_id: 123 23 | } 24 | ```` 25 | 26 | # The API 27 | 28 | In order to create this application you can choose the language you want based on your preferences. 29 | In the current version Node.js will be used as an example. 30 | 31 | This first (and very simple) version of the API needs to 32 | 33 | * implement a HTTP Post endpoint on /data 34 | * listen on port 1337 unless the PORT environment variable is provided 35 | * reply with a 201 HTTP Status Code (creation) 36 | * displays the received data on the standard output 37 | * add a simple test to check the implementation (the test should send the data example defined above) 38 | 39 | # Implementation's example 40 | 41 | You are invited to develop your own implementation. 42 | Otherwise, you can also copy, at the root of your *iot-api* folder, the content of the *nodejs* folder (Node.js example implementation) if you want to. 43 | 44 | [Node.js](./nodejs) 45 | 46 | Note: implementation examples in other languages will be provided soon. 47 | 48 | ----- 49 | [< Previous](../step0) - [Next >](../step2) 50 | -------------------------------------------------------------------------------- /step1/nodejs/README.md: -------------------------------------------------------------------------------- 1 | In this implementation several files are defined 2 | * package.json that handles all the dependencies 3 | * app.js that defines the expressjs web server 4 | * index.js that run the server 5 | 6 | On top of those, the file test/functional.js defines a simple test that starts the API, send data and verify the returned HTTP Status Code is 201. 7 | 8 | ## Starting the API 9 | 10 | ```` 11 | npm start 12 | ```` 13 | 14 | ## Running the test 15 | 16 | ```` 17 | npm test 18 | 19 | > iot@1.0.0 test /Users/luc/IoT-demo-project/step1/nodejs 20 | > mocha test/functional.js 21 | 22 | Creation 23 | info: server listening on port 3000 24 | info: ts=2017-03-11T15:00:53Z, type=temp, value=34, sensor_id=123 25 | ✓ should create dummy data (45ms) 26 | 27 | 1 passing (58ms) 28 | ```` 29 | -------------------------------------------------------------------------------- /step1/nodejs/app.js: -------------------------------------------------------------------------------- 1 | // Load dependencies 2 | const express = require('express'), 3 | bodyParser = require('body-parser'), 4 | winston = require('winston'); 5 | 6 | // Create express application 7 | let app = module.exports = express(); 8 | 9 | // Body parser configuration 10 | app.use(bodyParser.json()); 11 | app.use(bodyParser.urlencoded({ extended: true })); 12 | 13 | // Handle inconming data 14 | app.post('/data', 15 | (req, res, next) => { 16 | winston.info(req.body); 17 | return res.sendStatus(201); 18 | }); 19 | -------------------------------------------------------------------------------- /step1/nodejs/index.js: -------------------------------------------------------------------------------- 1 | // Load dependencies 2 | const util = require('util'), 3 | winston = require('winston'), 4 | app = require('./app'); 5 | 6 | // Define API port 7 | let port = process.env.PORT || 1337; 8 | 9 | // Run API 10 | app.listen(port, function(){ 11 | winston.info(util.format("server listening on port %s", port)); 12 | }); 13 | -------------------------------------------------------------------------------- /step1/nodejs/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "iot", 3 | "version": "1.0.0", 4 | "description": "IoT example project", 5 | "main": "index.js", 6 | "scripts": { 7 | "start": "node index.js", 8 | "test": "./node_modules/.bin/mocha test/functional.js" 9 | }, 10 | "author": "Luc Juggery", 11 | "license": "MIT", 12 | "dependencies": { 13 | "body-parser": "^1.17.1", 14 | "express": "^4.15.2", 15 | "winston": "^2.3.1" 16 | }, 17 | "devDependencies": { 18 | "mocha": "^3.2.0", 19 | "supertest": "^3.0.0" 20 | } 21 | } 22 | -------------------------------------------------------------------------------- /step1/nodejs/test/functional.js: -------------------------------------------------------------------------------- 1 | const request = require('supertest'), 2 | app = require('../app'); 3 | winston = require('winston'), 4 | util = require('util'), 5 | port = 3000, 6 | baseURL = util.format('http://localhost:%s', port); 7 | 8 | before(function(){ 9 | app.listen(port, function(){ 10 | winston.info(util.format("server listening on port %s", port)); 11 | }); 12 | }); 13 | 14 | describe('Creation', function(){ 15 | it('should create dummy data', function(done){ 16 | request(baseURL) 17 | .post('/data') 18 | .set('Content-Type', 'application/json') 19 | .send({"ts":"2017-03-11T15:00:53Z", "type": "temp", "value": 34, "sensor_id": 123 }) 20 | .expect(201) 21 | .end(function(err, res){ 22 | if (err) throw err; 23 | done(); 24 | }) 25 | }); 26 | }); 27 | -------------------------------------------------------------------------------- /step1/simulator.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | while(true); do 4 | # Current date 5 | d=$(date -u +"%Y-%m-%dT%H:%M:%SZ") 6 | 7 | # Random temperature between 20 and 34°C 8 | temp=$(( ( RANDOM % 15 ) + 20 )) 9 | 10 | # Send data to API 11 | curl -XPOST -H "Content-Type: application/json" -d '{"ts":"'$d'", "type": "temp", "value": '$temp', "sensor_id": 123 }' http://localhost:1337/data 12 | 13 | sleep 1 14 | done 15 | -------------------------------------------------------------------------------- /step10/README.md: -------------------------------------------------------------------------------- 1 | # Objectives 2 | 3 | In this step, we will setup a basic CI (Continuous Integration) pipeline. 4 | 5 | # Some tools of the CI lanscape 6 | 7 | A lot of online services exists for each of the versionning, test and images storage part. The image below list some of them. 8 | 9 | ![Landscape examples](./images/ci-web-services.png) 10 | 11 | Note: we only consider SaaS platforms here but a lot of tools can be installed on prems as well. 12 | 13 | # The workflow 14 | 15 | The workflow we will consider is the following one 16 | 17 | * code pushed in a repository of our versionning tool 18 | * tests are automatically ran 19 | * image is created and pushed to a registry if tests succeed 20 | 21 | Note: 2 tags of the same image will be created, the first one is the name of the branch where the code changes have been done, the second one is the value of the GitHub commit of thoses changes. 22 | 23 | Among the tools available we will use the following ones 24 | 25 | * GitHub to handle the source code (we already using it though) 26 | * [CircleCI](http://circleci.com) to run the tests, they offer a great free tier btw 27 | * [Docker Hub](https://hub.docker.com) to store the Docker images of the application 28 | 29 | ![The workflow](./images/ci-workflow.png) 30 | 31 | # Adding CircleCI in the picture 32 | 33 | The main testing platforms are configured in pretty much the same way, and this applies to CircleCI as well 34 | 35 | * a link needs to be done with a GitHub account 36 | * a configuration file needs to be created at the root of the GitHub repository that will be tested 37 | 38 | Let's review the process in details. 39 | 40 | 1. We first need to link our GitHub account 41 | 42 | CircleCI eases this process as it offers the possibility to signup directly using GitHub. 43 | 44 | ![CircleCI signup](./images/00-circleci-signup.png) 45 | 46 | The GitHub account is then linked to the platform 47 | 48 | ![GitHub account linked](./images/01-circleci-link-github.png) 49 | 50 | 2. Select the project to be handled in CircleCI 51 | 52 | Going into the project section, we can add a new project selecting one of our GitHub repositories. 53 | 54 | ![Add a project](./images/02-circleci-add-project.png) 55 | 56 | Within the project settings, we then set our Docker Hub credentials as environment variables (this second step is need to that CircleCI can push images to the Hub). 57 | 58 | ![Set Docker Hub credentials](./images/03-circleci-project-env-variable.png) 59 | 60 | 3. Add a circle.yml configuration file 61 | 62 | When running the tests for a project, CircleCI expects to have a *circle.yml* file at the root of the project (GitHub repository). This file is where all the steps are defined. 63 | 64 | Let's have a look at the way tests are done in the [Node.js example implementation](./nodejs/) 65 | 66 | 4. Build the project 67 | 68 | The project is now ready to be built. Clicking on the *build* button will retrieve the source from Github and perform the tests and the associated actions. 69 | 70 | ![Build](./images/04-circleci-build-project.png) 71 | 72 | 5. See all of this into account 73 | 74 | Let's now see this simple workflow into action 75 | 76 | Commit changes and push it into GitHub 77 | 78 | ![Push changes](./images/05-code-push-in-github.png) 79 | 80 | Check that the tests are automatically triggered in CircleCI 81 | 82 | ![CircleCI tests](./images/06-circle-success-build.png) 83 | 84 | Going into Docker Hub we can see the image has been created with both tags 85 | * master as this is the name of the branch on which the changes have been done 86 | * GitHub's COMMIT_ID so the version of the code can be easily matched against the image 87 | 88 | ![Image creation on Docker Hub](./images/07-docker-hub-image-created.png) 89 | -------------------------------------------------------------------------------- /step10/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.1" 2 | services: 3 | db: 4 | image: influxdb:1.2.4 5 | environment: 6 | - INFLUXDB_ADMIN_ENABLED=true 7 | ports: 8 | - 8086:8086 9 | - 8083:8083 10 | volumes: 11 | - influxdata:/var/lib/influxdb 12 | api: 13 | image: lucj/iot-api:v3 14 | command: npm start 15 | ports: 16 | - 1337:1337 17 | dashboard: 18 | image: grafana/grafana 19 | ports: 20 | - 3000:3000 21 | volumes: 22 | influxdata: 23 | -------------------------------------------------------------------------------- /step10/images/00-circleci-signup.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step10/images/00-circleci-signup.png -------------------------------------------------------------------------------- /step10/images/01-circleci-link-github.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step10/images/01-circleci-link-github.png -------------------------------------------------------------------------------- /step10/images/02-circleci-add-project.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step10/images/02-circleci-add-project.png -------------------------------------------------------------------------------- /step10/images/03-circleci-project-env-variable.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step10/images/03-circleci-project-env-variable.png -------------------------------------------------------------------------------- /step10/images/04-circleci-build-project.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step10/images/04-circleci-build-project.png -------------------------------------------------------------------------------- /step10/images/05-code-push-in-github.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step10/images/05-code-push-in-github.png -------------------------------------------------------------------------------- /step10/images/06-circle-success-build.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step10/images/06-circle-success-build.png -------------------------------------------------------------------------------- /step10/images/07-docker-hub-image-created.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step10/images/07-docker-hub-image-created.png -------------------------------------------------------------------------------- /step10/images/ci-web-services.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step10/images/ci-web-services.png -------------------------------------------------------------------------------- /step10/images/ci-workflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step10/images/ci-workflow.png -------------------------------------------------------------------------------- /step10/nodejs/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM mhart/alpine-node:7.7.1 2 | ENV LAST_UPDATED 20170301T231500 3 | 4 | # Install curl 5 | RUN apk add -U curl 6 | 7 | # Copy list of server side dependencies 8 | COPY package.json /tmp/package.json 9 | 10 | # Install dependencies 11 | RUN cd /tmp && npm install 12 | 13 | # Copy dependencies libraries 14 | RUN mkdir /app && cp -a /tmp/node_modules /app/ 15 | 16 | # Copy src files 17 | COPY . /app/ 18 | 19 | # Use /app working directory 20 | WORKDIR /app 21 | 22 | # Expose http port 23 | EXPOSE 1337 24 | 25 | # Run application 26 | CMD ["npm", "start"] 27 | -------------------------------------------------------------------------------- /step10/nodejs/README.md: -------------------------------------------------------------------------------- 1 | # Foreword 2 | 3 | When we have initiated this API we have created a simple test just to setup the testing process (even if it does not a lot of things). It's now time to enhance the way tests are ran in order to better automate them. 4 | 5 | # How testing is done 6 | 7 | In order to run the tests in the current impplementation, we need to: run an instance of InfluxDB, create the *iot* database and execute the test defined in the API. 8 | 9 | Run the database and create the *iot* database 10 | 11 | ```` 12 | docker container run -d -p 8083:8083 -p 8086:8086 --name influx influxdb 13 | curl -i -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE iot" 14 | ```` 15 | 16 | Run the test defined in *test/functional.js* 17 | 18 | ```` 19 | $ INFLUXDB_HOST=localhost npm test 20 | 21 | > iot@1.0.0 test /Users/luc/perso/Dropbox/Work/Side/Docker/IoT-demo-project/step10/nodejs 22 | > mocha test/functional.js 23 | 24 | Creation 25 | info: server listening on port 3000 26 | info: ts=2017-03-11T15:00:53Z, type=temp, value=34, sensor_id=123 27 | ✓ should create dummy data (90ms) 28 | 29 | 1 passing (103ms) 30 | ```` 31 | 32 | # Simplify the testing process 33 | 34 | ## Using a bash script 35 | 36 | We can create a shell script with the above commands and run it during the test phase. An example of the *test.sh* script has been added to the [Node.js example implementation](./nodejs/). 37 | 38 | This is not the option we will use when running the test on CircleCI though. 39 | 40 | ## Using a Docker Compose file 41 | 42 | We can also use another approach using Docker Compose. The *compose-test.yml* file illustrate this approach. It basically define 2 services: 43 | * db: the InfluxDB we need to run so the API can persist the data it receives 44 | * sut (system under test): used to build the image and run the test on it using the underlying database 45 | 46 | Note:*sut* is used as this is the default service name ran by Docker Cloud when used as a CI tool (instead of CircleCI). We have decided to keepo the same name in this example. 47 | 48 | Using this file, the test are ran simply using the *docke-compose build* and a *docker-compose run* command. 49 | 50 | At the same time, we have added the installation of *curl* in the base image, this is very light and will help to create the database when running the tests. 51 | 52 | Let's see it in action. 53 | 54 | ```` 55 | $ docker-compose -f compose-test.yml build sut 56 | Building sut 57 | Step 1/10 : FROM mhart/alpine-node:7.7.1 58 | ---> e1a533c514f2 59 | Step 2/10 : ENV LAST_UPDATED 20170301T231500 60 | ---> Using cache 61 | ---> 4a6e696daca9 62 | Step 3/10 : RUN apk add -U curl 63 | ---> Running in 26662acf2633 64 | fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz 65 | fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz 66 | (1/4) Installing ca-certificates (20161130-r0) 67 | (2/4) Installing libssh2 (1.7.0-r2) 68 | (3/4) Installing libcurl (7.52.1-r2) 69 | (4/4) Installing curl (7.52.1-r2) 70 | Executing busybox-1.25.1-r0.trigger 71 | Executing ca-certificates-20161130-r0.trigger 72 | OK: 7 MiB in 17 packages 73 | ---> c3d6acf8023f 74 | Removing intermediate container 26662acf2633 75 | Step 4/10 : COPY package.json /tmp/package.json 76 | ---> c4039033878c 77 | Removing intermediate container 630cc2aff5bf 78 | Step 5/10 : RUN cd /tmp && npm install 79 | ---> Running in 90ca79f79211 80 | iot@1.0.0 /tmp 81 | +-- body-parser@1.17.1 82 | ... 83 | npm WARN iot@1.0.0 No repository field. 84 | ---> 1a328e9a8fe6 85 | Removing intermediate container 90ca79f79211 86 | Step 6/10 : RUN mkdir /app && cp -a /tmp/node_modules /app/ 87 | ---> Running in 07d503221dcf 88 | ---> 0d4d2b07da98 89 | Removing intermediate container 07d503221dcf 90 | Step 7/10 : COPY . /app/ 91 | ---> 26322e176963 92 | Removing intermediate container d688a9402aaf 93 | Step 8/10 : WORKDIR /app 94 | ---> 448232deadbe 95 | Removing intermediate container 8de202977bb7 96 | Step 9/10 : EXPOSE 1337 97 | ---> Running in 7ec87ac42cee 98 | ---> 34b8c67bacf5 99 | Removing intermediate container 7ec87ac42cee 100 | Step 10/10 : CMD npm start 101 | ---> Running in ef18e2dcd7be 102 | ---> 319d46689567 103 | Removing intermediate container ef18e2dcd7be 104 | Successfully built 319d46689567 105 | ```` 106 | 107 | Then, we run the tests on the image previously built. 108 | 109 | ```` 110 | $ docker-compose -f compose-test.yml run sut 111 | HTTP/1.1 200 OK 112 | Connection: close 113 | Content-Type: application/json 114 | Request-Id: a661cdb3-0d59-11e7-8001-000000000000 115 | X-Influxdb-Version: 1.2.1 116 | Date: Mon, 20 Mar 2017 10:40:35 GMT 117 | Transfer-Encoding: chunked 118 | 119 | {"results":[{"statement_id":0}]} 120 | 121 | > iot@1.0.0 test /app 122 | > mocha test/functional.js 123 | 124 | Creation 125 | info: server listening on port 3000 126 | info: ts=2017-03-11T15:00:53Z, type=temp, value=34, sensor_id=123 127 | ✓ should create dummy data (83ms) 128 | 129 | 1 passing (99ms) 130 | ```` 131 | 132 | # Adding circle.yml file 133 | 134 | In order for the tests to be done in CircleCI, we need to specify the different phases in a *circle.yml* file that will be created at the root of the repository. 135 | 136 | Basically, this file defines 137 | * the environment CircleCI platform needs to set up for the tests: only docker/docker-compose are needed here 138 | * the test commands that needs to be ran: run the test service defined in the compose file 139 | * the deployment that needs to be done when the tests succeed: create an image and push it to the Docker Hub 140 | 141 | Notes: 142 | - there are obviously much more available options but those ones will match our needs for now 143 | - the test phase specified in circle.yml does not use run the build phase as by default it will be ran each time the tests are executed on the platform (new CircleCI environment spin up each time) 144 | -------------------------------------------------------------------------------- /step10/nodejs/app.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | // Load dependencies 4 | const express = require('express'), 5 | Influx = require('influx'), 6 | bodyParser = require('body-parser'), 7 | winston = require('winston'); 8 | 9 | // Create express application 10 | let app = module.exports = express(); 11 | 12 | // Create a client towards InfluxDB 13 | let influx = new Influx.InfluxDB({ 14 | host: process.env.INFLUXDB_HOST || "db", 15 | database: 'iot' 16 | }); 17 | 18 | // Body parser configuration 19 | app.use(bodyParser.json()); 20 | app.use(bodyParser.urlencoded({ extended: true })); 21 | 22 | // Handle inconming data 23 | app.post('/data', 24 | function(req, res, next){ 25 | influx.writePoints([ 26 | { 27 | measurement: 'data', 28 | tags: { type: req.body.type }, 29 | fields: { sensor_id: req.body.sensor_id, value: req.body.value }, 30 | timestamp: new Date(req.body.ts).getTime() * 1000000 31 | } 32 | ]).then(() => { 33 | winston.info(req.body); 34 | return res.sendStatus(201); 35 | }) 36 | .catch( err => { 37 | winston.error(err.message); 38 | return res.sendStatus(500); 39 | }); 40 | }); 41 | -------------------------------------------------------------------------------- /step10/nodejs/circle.yml: -------------------------------------------------------------------------------- 1 | machine: 2 | services: 3 | - docker 4 | 5 | test: 6 | override: 7 | - docker-compose -f compose-test-ci.yml run test 8 | 9 | deployment: 10 | hub: 11 | branch: [ master ] 12 | commands: 13 | - docker build --rm=false -t lucj/iot-api:$CIRCLE_BRANCH . 14 | - docker tag lucj/iot-api:$CIRCLE_BRANCH lucj/iot-api:$CIRCLE_SHA1 15 | - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS 16 | - docker push lucj/iot-api:$CIRCLE_BRANCH 17 | - docker push lucj/iot-api:$CIRCLE_SHA1 18 | -------------------------------------------------------------------------------- /step10/nodejs/compose-test.yml: -------------------------------------------------------------------------------- 1 | db: 2 | image: influxdb:1.2.4 3 | 4 | sut: 5 | build: . 6 | command: /bin/sh -c 'curl -i -XPOST http://db:8086/query --data-urlencode "q=CREATE DATABASE iot" && npm test' 7 | links: 8 | - db 9 | -------------------------------------------------------------------------------- /step10/nodejs/index.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | // Load dependencies 4 | const util = require('util'), 5 | winston = require('winston'), 6 | app = require('./app'); 7 | 8 | // Define API port 9 | let port = process.env.PORT || 1337; 10 | 11 | // Run API 12 | app.listen(port, function(){ 13 | winston.info(util.format("server listening on port %s", port)); 14 | }); 15 | -------------------------------------------------------------------------------- /step10/nodejs/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "iot", 3 | "version": "1.0.0", 4 | "description": "IoT example project", 5 | "main": "index.js", 6 | "scripts": { 7 | "start": "node index.js", 8 | "test": "./node_modules/.bin/mocha test/functional.js" 9 | }, 10 | "author": "Luc Juggery", 11 | "license": "MIT", 12 | "dependencies": { 13 | "body-parser": "^1.17.1", 14 | "express": "^4.15.2", 15 | "influx": "^5.0.6", 16 | "winston": "^2.3.1" 17 | }, 18 | "devDependencies": { 19 | "mocha": "^3.2.0", 20 | "supertest": "^3.0.0" 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /step10/nodejs/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Run InfluxDB in a container 4 | docker container run -d -p 8083:8083 -p 8086:8086 --name influx influxdb 5 | 6 | # Wait for the database to be ready 7 | curl -sL -I localhost:8086/ping 8 | while [ $? -ne 0 ];do 9 | sleep 2 10 | curl -sL -I localhost:8086/ping 11 | done 12 | 13 | # Create the iot database 14 | curl -i -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE iot" 15 | 16 | # Run the tests specifying the InfluxDB host 17 | INFLUXDB_HOST=localhost npm test 18 | 19 | # Remove the InfluxDB container 20 | docker container stop influx && docker container rm influx 21 | -------------------------------------------------------------------------------- /step10/nodejs/test/functional.js: -------------------------------------------------------------------------------- 1 | const request = require('supertest'), 2 | app = require('../app'); 3 | winston = require('winston'), 4 | util = require('util'), 5 | port = 3000, 6 | baseURL = util.format('http://localhost:%s', port); 7 | 8 | before(function(){ 9 | app.listen(port, function(){ 10 | winston.info(util.format("server listening on port %s", port)); 11 | }); 12 | }); 13 | 14 | describe('Creation', function(){ 15 | it('should create dummy data', function(done){ 16 | request(baseURL) 17 | .post('/data') 18 | .set('Content-Type', 'application/json') 19 | .send({"ts":"2017-03-11T15:00:53Z", "type": "temp", "value": 34, "sensor_id": 123 }) 20 | .expect(201) 21 | .end(function(err, res){ 22 | if (err) throw err; 23 | done(); 24 | }) 25 | }); 26 | }); 27 | -------------------------------------------------------------------------------- /step10/nodejs/test/mocha.opts: -------------------------------------------------------------------------------- 1 | --timeout 5s 2 | -------------------------------------------------------------------------------- /step11/README.md: -------------------------------------------------------------------------------- 1 | # WARNING 2 | 3 | Docker recently announced the discontinuation of DockerCloud on May 21th. 4 | 5 | This step will be removed soon and updated with a continuous deployment on a Swarm 6 | 7 | # Purpose 8 | 9 | Now that we have a simple continuous integration pipeline that create the image of our application, we will go one step further and automatically deploy the new image. 10 | 11 | # What we will do 12 | 13 | * Introduce Docker Cloud fonctionalities and concepts 14 | * Setup a node 15 | * Create the application stack 16 | * Test the complete CI/CD flow 17 | 18 | # Docker Cloud 19 | 20 | Basically, [Docker Cloud](https://cloud.docker.com) is the 100% web based solution that enables to manage multi-containers application in an easy way, from the setup of the underlying infrastrucute to the redeployment at the end of the CI/CD pipeline. 21 | 22 | The definition in Docker's documentation is the following one: 23 | 24 | ``` 25 | Docker Cloud provides a hosted registry service with build and testing facilities for Dockerized application images; tools to help you set up and manage host infrastructure; and application lifecycle features to automate deploying (and redeploying) services created from images. 26 | ``` 27 | 28 | That is a lot of fonctionnalities and we will now illustrate some of them. 29 | 30 | We can log on [Docker Cloud](https://cloud.docker.com) using the Docker Hub credential. 31 | Once log, the welcome page is the following one. 32 | 33 | ![Docker Cloud Home](./images/DockerCloud-01-CloudRegistry.png) 34 | 35 | As we can see there is a lot of features here: 36 | * Image registry 37 | * Application deployment 38 | * CI/CD workflow 39 | * Teams / Users management 40 | 41 | In the following we will deploy our 3 containers applications (api, db, dashboard) and before that we will create a node on [DigitalOcean](https://digitalocean.com), from Docker Cloud interface, where the application will run. 42 | 43 | # Setup a node 44 | 45 | Setting up a node is really easy, the first step is to go into the *Cloud Settings* menu and select a cloud provider (among AWS, DigitalOcean, Microsoft Azure, SoftLayer, Packet). In this example, we use DigitalOcean, if you do not have an account on DigitalOcean (DO), yet, I really recommend you create one and have a try (a couple of bucks will be enough to get started). 46 | 47 | ![Linking DO](./images/Nodes-00_CloudProviders.png) 48 | 49 | When the cloud provider is linked, we can select it when creating a new node, from the Node menu. 50 | 51 | ![Create a node](./images/Nodes-02-Create.png) 52 | 53 | Note: depending upon the cloud provider used, the available options will be different though. 54 | 55 | After a couple of seconds, the node is created and ready to host our application. 56 | 57 | # The application stack 58 | 59 | In the previous step we have built the application so it is now a multi-containers app handled by Docker Compose. In the current version of Docker Cloud (that might change in the future), a docker-compose.yml file cannot be used as is (which is a pity I have to admit). But... the positive thing is that it only requires some little changes in the docker-compose.yml file. We created the *docker-cloud.yml* file out of *docker-compose.yml* and this is the file we will use in Docker Cloud. 60 | 61 | As we can see, we need to remove the top level keys (version, services, volumes) and that is pretty much all. 62 | 63 | In Docker cloud, there is the notion of Stack, which is a set of services, each services being instanciated in one or several containers. Our application will be deployed as a stack by draging the *docker-cloud.yml* file in the interface. 64 | 65 | ![Create a stack](./images/Stack-01-Create.png) 66 | 67 | From there, we just need to click on the "Create and Deploy" button and our stack will come to life. 68 | 69 | ![Deploy a stack](./images/Stack-02-Deploy.png) 70 | 71 | At the bottom of the screenshot, the exposed endPoints are listed, we will use them to access the services. Let's first use the InfluxDB endPoint exposing the administration interface to create the *iot* database. 72 | 73 | ![Create the database](./images/App-01-DB-Create.png) 74 | 75 | # Run the simulator 76 | 77 | Let's run the simulator using the endPoint exposed by the *api* 78 | 79 | ```` 80 | $ ./simulator.sh -h api.iot.edcca32d.svc.dockerapp.io -p 1337 81 | => About to send data to api.iot.edcca32d.svc.dockerapp.io on port 1337 82 | ```` 83 | 84 | The simulator seems to be working fine, let's now check if the application is receiving the data. 85 | 86 | # Visualize the data 87 | 88 | The data are correctly received as we can see from the dashboard. 89 | 90 | ![Data visualisation](./images/App-02-Dashboard.png) 91 | 92 | # Test the CI/CD flow 93 | 94 | Let's now make a small change to the code of *iot-api* and push those changes to GitHub. 95 | We will simply enable to pass the database name through the INFLUXDB_DATABASE environment variable and leave the default to "iot". 96 | 97 | Let's do this change in our *iot-api* repository and push it to GitHub. 98 | 99 | 1. Changes pushed to GitHub 100 | 101 | ![Code push](./images/Flow-01-GitHub.png) 102 | 103 | 2. Test triggered on CircleCI 104 | 105 | ![Test triggered](./images/Flow-02-CircleCI.png) 106 | 107 | 3. Image pushed on Docker Hub 108 | 109 | ![Image pushed](./images/Flow-03-DockerHub.png) 110 | 111 | 4. Image redeployed via Docker Cloud 112 | 113 | ![API redeployed](./images/Flow-04-DockerCloud.png) 114 | 115 | # Summary 116 | 117 | In this step we have added the Continuous Deployment in just a couple of steps. 118 | 119 | Note: as we can see from the Docker Cloud welcome page, we could have used Docker Cloud for the Continuous Integration as well but I think it's good to see some other services of the ecosystem and CircleCI definitely deserves a look IMHO. 120 | 121 | -------------------------------------------------------------------------------- /step11/docker-cloud.yml: -------------------------------------------------------------------------------- 1 | db: 2 | image: influxdb:1.2.4 3 | environment: 4 | - INFLUXDB_ADMIN_ENABLED=true 5 | ports: 6 | - 8086:8086 7 | - 8083:8083 8 | restart: on-failure 9 | api: 10 | autoredeploy: true 11 | image: lucj/iot-api:master 12 | command: npm start 13 | ports: 14 | - 1337:1337 15 | restart: on-failure 16 | dashboard: 17 | image: grafana/grafana 18 | ports: 19 | - 3000:3000 20 | restart: on-failure 21 | -------------------------------------------------------------------------------- /step11/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.1" 2 | services: 3 | db: 4 | image: influxdb:1.2.4 5 | environment: 6 | - INFLUXDB_ADMIN_ENABLED=true 7 | ports: 8 | - 8086:8086 9 | - 8083:8083 10 | volumes: 11 | - influxdata:/var/lib/influxdb 12 | api: 13 | image: lucj/iot-api:v3 14 | command: npm start 15 | ports: 16 | - 1337:1337 17 | dashboard: 18 | image: grafana/grafana 19 | ports: 20 | - 3000:3000 21 | volumes: 22 | influxdata: 23 | -------------------------------------------------------------------------------- /step11/images/App-01-DB-Create.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/App-01-DB-Create.png -------------------------------------------------------------------------------- /step11/images/App-02-Dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/App-02-Dashboard.png -------------------------------------------------------------------------------- /step11/images/DockerCloud-01-CloudRegistry.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/DockerCloud-01-CloudRegistry.png -------------------------------------------------------------------------------- /step11/images/DockerCloud-02-CI.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/DockerCloud-02-CI.png -------------------------------------------------------------------------------- /step11/images/DockerCloud-03-ApplicationDeployment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/DockerCloud-03-ApplicationDeployment.png -------------------------------------------------------------------------------- /step11/images/DockerCloud-04-CD.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/DockerCloud-04-CD.png -------------------------------------------------------------------------------- /step11/images/DockerCloud-05-Teams.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/DockerCloud-05-Teams.png -------------------------------------------------------------------------------- /step11/images/Flow-01-GitHub.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/Flow-01-GitHub.png -------------------------------------------------------------------------------- /step11/images/Flow-02-CircleCI.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/Flow-02-CircleCI.png -------------------------------------------------------------------------------- /step11/images/Flow-03-DockerHub.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/Flow-03-DockerHub.png -------------------------------------------------------------------------------- /step11/images/Flow-04-DockerCloud.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/Flow-04-DockerCloud.png -------------------------------------------------------------------------------- /step11/images/Nodes-00_CloudProviders.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/Nodes-00_CloudProviders.png -------------------------------------------------------------------------------- /step11/images/Nodes-01-Menu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/Nodes-01-Menu.png -------------------------------------------------------------------------------- /step11/images/Nodes-02-Create.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/Nodes-02-Create.png -------------------------------------------------------------------------------- /step11/images/Nodes-03-Deploying.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/Nodes-03-Deploying.png -------------------------------------------------------------------------------- /step11/images/Stack-01-Create.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/Stack-01-Create.png -------------------------------------------------------------------------------- /step11/images/Stack-02-Deploy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/images/Stack-02-Deploy.png -------------------------------------------------------------------------------- /step11/nodejs/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM mhart/alpine-node:7.7.1 2 | ENV LAST_UPDATED 20170301T231500 3 | 4 | # Copy list of server side dependencies 5 | COPY package.json /tmp/package.json 6 | 7 | # Install dependencies 8 | RUN cd /tmp && npm install 9 | 10 | # Copy dependencies libraries 11 | RUN mkdir /app && cp -a /tmp/node_modules /app/ 12 | 13 | # Copy src files 14 | COPY . /app/ 15 | 16 | # Use /app working directory 17 | WORKDIR /app 18 | 19 | # Expose http port 20 | EXPOSE 1337 21 | 22 | # Run application 23 | CMD ["npm", "start"] 24 | -------------------------------------------------------------------------------- /step11/nodejs/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step11/nodejs/README.md -------------------------------------------------------------------------------- /step11/nodejs/app.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | // Load dependencies 4 | const express = require('express'), 5 | Influx = require('influx'), 6 | bodyParser = require('body-parser'), 7 | winston = require('winston'); 8 | 9 | // Create express application 10 | let app = module.exports = express(); 11 | 12 | // Create a client towards InfluxDB 13 | let influx = new Influx.InfluxDB({ 14 | host: process.env.INFLUXDB_HOST || "db", 15 | database: process.env.INFLUXDB_DATABASE || 'iot' 16 | }); 17 | 18 | // Body parser configuration 19 | app.use(bodyParser.json()); 20 | app.use(bodyParser.urlencoded({ extended: true })); 21 | 22 | // Handle inconming data 23 | app.post('/data', 24 | function(req, res, next){ 25 | influx.writePoints([ 26 | { 27 | measurement: 'data', 28 | tags: { type: req.body.type }, 29 | fields: { sensor_id: req.body.sensor_id, value: req.body.value }, 30 | timestamp: new Date(req.body.ts).getTime() * 1000000 31 | } 32 | ]).then(() => { 33 | winston.info(req.body); 34 | return res.sendStatus(201); 35 | }) 36 | .catch( err => { 37 | winston.error(err.message); 38 | return res.sendStatus(500); 39 | }); 40 | }); 41 | -------------------------------------------------------------------------------- /step11/nodejs/circle.yml: -------------------------------------------------------------------------------- 1 | machine: 2 | services: 3 | - docker 4 | 5 | test: 6 | override: 7 | - docker-compose -f compose-test-ci.yml run test 8 | 9 | deployment: 10 | hub: 11 | branch: [ master ] 12 | commands: 13 | - docker build --rm=false -t lucj/iot-api:$CIRCLE_BRANCH . 14 | - docker tag lucj/iot-api:$CIRCLE_BRANCH lucj/iot-api:$CIRCLE_SHA1 15 | - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS 16 | - docker push lucj/iot-api:$CIRCLE_BRANCH 17 | - docker push lucj/iot-api:$CIRCLE_SHA1 18 | -------------------------------------------------------------------------------- /step11/nodejs/compose-test-ci.yml: -------------------------------------------------------------------------------- 1 | db: 2 | image: influxdb:1.2.4 3 | 4 | # Create iot database 5 | init: 6 | image: lucj/alpine-curl:1.0 7 | command: curl -i -XPOST http://db:8086/query --data-urlencode "q=CREATE DATABASE iot" && sleep 100 8 | links: 9 | - db:db 10 | 11 | # Run application test 12 | test: 13 | build: . 14 | command: npm test 15 | links: 16 | - init 17 | - db 18 | -------------------------------------------------------------------------------- /step11/nodejs/compose-test.yml: -------------------------------------------------------------------------------- 1 | version: '3.1' 2 | services: 3 | db: 4 | image: influxdb:1.2.4 5 | 6 | # Create iot database 7 | init: 8 | image: appropriate/curl 9 | command: -i -XPOST http://db:8086/query --data-urlencode "q=CREATE DATABASE iot" 10 | depends_on: 11 | - db 12 | 13 | # Run application test 14 | test: 15 | build: . 16 | command: npm test 17 | depends_on: 18 | - init 19 | - db 20 | -------------------------------------------------------------------------------- /step11/nodejs/compose-test.yml-save: -------------------------------------------------------------------------------- 1 | version: '3.1' 2 | services: 3 | db: 4 | image: influxdb 5 | volumes: 6 | - influxdata:/var/lib/influxdb 7 | 8 | # Create iot database 9 | init: 10 | image: appropriate/curl 11 | command: -i -XPOST http://db:8086/query --data-urlencode "q=CREATE DATABASE iot" 12 | depends_on: 13 | - db 14 | 15 | # Run application test 16 | test: 17 | build: . 18 | command: npm test 19 | depends_on: 20 | - init 21 | - db 22 | volumes: 23 | influxdata: 24 | -------------------------------------------------------------------------------- /step11/nodejs/index.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | // Load dependencies 4 | const util = require('util'), 5 | winston = require('winston'), 6 | app = require('./app'); 7 | 8 | // Define API port 9 | let port = process.env.PORT || 1337; 10 | 11 | // Run API 12 | app.listen(port, function(){ 13 | winston.info(util.format("server listening on port %s", port)); 14 | }); 15 | -------------------------------------------------------------------------------- /step11/nodejs/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "iot", 3 | "version": "1.0.0", 4 | "description": "IoT example project", 5 | "main": "index.js", 6 | "scripts": { 7 | "start": "node index.js", 8 | "test": "./node_modules/.bin/mocha test/functional.js" 9 | }, 10 | "author": "Luc Juggery", 11 | "license": "MIT", 12 | "dependencies": { 13 | "body-parser": "^1.17.1", 14 | "express": "^4.15.2", 15 | "influx": "^5.0.6", 16 | "winston": "^2.3.1" 17 | }, 18 | "devDependencies": { 19 | "mocha": "^3.2.0", 20 | "supertest": "^3.0.0" 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /step11/nodejs/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Run InfluxDB in a container 4 | docker container run -d -p 8083:8083 -p 8086:8086 --name influx influxdb 5 | 6 | # Wait for the database to be ready 7 | curl -sL -I localhost:8086/ping 8 | while [ $? -ne 0 ];do 9 | sleep 2 10 | curl -sL -I localhost:8086/ping 11 | done 12 | 13 | # Create the iot database 14 | curl -i -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE iot" 15 | 16 | # Run the tests specifying the InfluxDB host 17 | INFLUXDB_HOST=localhost npm test 18 | 19 | # Remove the InfluxDB container 20 | docker container stop influx && docker container rm influx 21 | -------------------------------------------------------------------------------- /step11/nodejs/test/functional.js: -------------------------------------------------------------------------------- 1 | const request = require('supertest'), 2 | app = require('../app'); 3 | winston = require('winston'), 4 | util = require('util'), 5 | port = 3000, 6 | baseURL = util.format('http://localhost:%s', port); 7 | 8 | before(function(){ 9 | app.listen(port, function(){ 10 | winston.info(util.format("server listening on port %s", port)); 11 | }); 12 | }); 13 | 14 | describe('Creation', function(){ 15 | it('should create dummy data', function(done){ 16 | request(baseURL) 17 | .post('/data') 18 | .set('Content-Type', 'application/json') 19 | .send({"ts":"2017-03-11T15:00:53Z", "type": "temp", "value": 34, "sensor_id": 123 }) 20 | .expect(201) 21 | .end(function(err, res){ 22 | if (err) throw err; 23 | done(); 24 | }) 25 | }); 26 | }); 27 | -------------------------------------------------------------------------------- /step11/nodejs/test/mocha.opts: -------------------------------------------------------------------------------- 1 | --timeout 5s 2 | -------------------------------------------------------------------------------- /step11/simulator.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Default HOST:PORT tragetted 4 | HOST="localhost" 5 | PORT=1337 6 | 7 | function usage { 8 | echo "Usage: simulator.sh [-h HOST] [-p PORT]" 9 | exit 1 10 | } 11 | 12 | # Parse arguments 13 | while getopts h:p: FLAG; do 14 | case $FLAG in 15 | h) 16 | HOST=$OPTARG 17 | ;; 18 | p) 19 | PORT=$OPTARG 20 | ;; 21 | \?) 22 | usage 23 | ;; 24 | esac 25 | done 26 | 27 | echo "=> About to send data to $HOST on port $PORT" 28 | echo 29 | 30 | # Generate and send random data 31 | while(true); do 32 | # Current date 33 | d=$(date -u +"%Y-%m-%dT%H:%M:%SZ") 34 | 35 | # Random temperature between 20 and 34°C 36 | temp=$(( ( RANDOM % 15 ) + 20 )) 37 | 38 | # Send data to API 39 | curl -XPOST -H "Content-Type: application/json" -d '{"ts":"'$d'", "type": "temp", "value": '$temp', "sensor_id": 123 }' http://$HOST:$PORT/data 40 | 41 | sleep 1 42 | done 43 | -------------------------------------------------------------------------------- /step2/README.md: -------------------------------------------------------------------------------- 1 | # Objectives 2 | 3 | *Containerize* the application adding a Dockerfile to the picture. 4 | 5 | # Instructions 6 | 7 | * Create a Dockerfile at the root of the project source code 8 | * Choose one of the base image that embeds the runtime of your selected language 9 | * Expose port 1337 in the Dockerfile 10 | * Add a .dockerignore to prevent local node_modules folder to be taken into account in the context 11 | 12 | # Build the image 13 | 14 | When the Dockerfile is ready, the image can be built with the following command: 15 | 16 | ```` 17 | docker image build -t iot:v1 . 18 | ```` 19 | 20 | Note: the image has been tagged with *v1*, this version will be incremented with the changes done in the next steps. 21 | 22 | # Run the application within a container 23 | 24 | Now the application and all its dependencies and its runtime is packaged into an image, we can run it within a container. 25 | 26 | ```` 27 | docker container run -p 1337:1337 iot:v1 28 | ```` 29 | 30 | Note: the *-p 1337:1337* option is specified for the port 1337 within the container to be published on the Docker host 31 | 32 | # Example 33 | 34 | The Node.js example implementation as been updated in the [nodejs](./nodejs) folder. 35 | 36 | # Status 37 | 38 | The image named *iot* and tagged with *v1* has been created locally. It then appears in the list of images available on the localhost. 39 | 40 | ```` 41 | $ docker image ls iot 42 | REPOSITORY TAG IMAGE ID CREATED SIZE 43 | iot v1 c243ef955948 8 seconds ago 71 MB 44 | ```` 45 | 46 | ----- 47 | [< Previous](../step1) - [Next >](../step3) 48 | -------------------------------------------------------------------------------- /step2/nodejs/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM mhart/alpine-node:6.11.0 2 | ENV LAST_UPDATED 20170624T180000 3 | 4 | # Copy list of server side dependencies 5 | COPY package.json /tmp/package.json 6 | 7 | # Install dependencies 8 | RUN cd /tmp && npm install 9 | 10 | # Copy dependencies libraries 11 | RUN mkdir /app && cp -a /tmp/node_modules /app/ 12 | 13 | # Copy src files 14 | COPY . /app/ 15 | 16 | # Use /app working directory 17 | WORKDIR /app 18 | 19 | # Expose http port 20 | EXPOSE 1337 21 | 22 | # Run application 23 | CMD ["npm", "start"] 24 | -------------------------------------------------------------------------------- /step2/nodejs/README.md: -------------------------------------------------------------------------------- 1 | In this implementation several files are defined 2 | * package.json that handles all the dependencies 3 | * app.js that defines the expressjs web server 4 | * index.js that run the server 5 | 6 | On top of those, the file test/functional.js defines a simple test that starts the API, send data and verify the returned HTTP Status Code is 201. 7 | 8 | ## Starting the API 9 | 10 | ```` 11 | npm start 12 | ```` 13 | 14 | ## Running the test 15 | 16 | ```` 17 | npm test 18 | 19 | > iot@1.0.0 test /Users/luc/IoT-demo-project/step1/nodejs 20 | > mocha test/functional.js 21 | 22 | Creation 23 | info: server listening on port 3000 24 | info: ts=2017-03-11T15:00:53Z, type=temp, value=34, sensor_id=123 25 | ✓ should create dummy data (45ms) 26 | 27 | 1 passing (58ms) 28 | ```` 29 | -------------------------------------------------------------------------------- /step2/nodejs/app.js: -------------------------------------------------------------------------------- 1 | // Load dependencies 2 | const express = require('express'), 3 | bodyParser = require('body-parser'), 4 | winston = require('winston'); 5 | 6 | // Create express application 7 | let app = module.exports = express(); 8 | 9 | // Body parser configuration 10 | app.use(bodyParser.json()); 11 | app.use(bodyParser.urlencoded({ extended: true })); 12 | 13 | // Handle inconming data 14 | app.post('/data', 15 | (req, res, next) => { 16 | winston.info(req.body); 17 | return res.sendStatus(201); 18 | }); 19 | -------------------------------------------------------------------------------- /step2/nodejs/index.js: -------------------------------------------------------------------------------- 1 | // Load dependencies 2 | const util = require('util'), 3 | winston = require('winston'), 4 | app = require('./app'); 5 | 6 | // Define API port 7 | let port = process.env.PORT || 1337; 8 | 9 | // Run API 10 | app.listen(port, function(){ 11 | winston.info(util.format("server listening on port %s", port)); 12 | }); 13 | -------------------------------------------------------------------------------- /step2/nodejs/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "iot", 3 | "version": "1.0.0", 4 | "description": "IoT example project", 5 | "main": "index.js", 6 | "scripts": { 7 | "start": "node index.js", 8 | "test": "./node_modules/.bin/mocha test/functional.js" 9 | }, 10 | "author": "Luc Juggery", 11 | "license": "MIT", 12 | "dependencies": { 13 | "body-parser": "^1.17.1", 14 | "express": "^4.15.2", 15 | "winston": "^2.3.1" 16 | }, 17 | "devDependencies": { 18 | "mocha": "^3.2.0", 19 | "supertest": "^3.0.0" 20 | } 21 | } 22 | -------------------------------------------------------------------------------- /step2/nodejs/test/functional.js: -------------------------------------------------------------------------------- 1 | const request = require('supertest'), 2 | app = require('../app'); 3 | winston = require('winston'), 4 | util = require('util'), 5 | port = 3000, 6 | baseURL = util.format('http://localhost:%s', port); 7 | 8 | before(function(){ 9 | app.listen(port, function(){ 10 | winston.info(util.format("server listening on port %s", port)); 11 | }); 12 | }); 13 | 14 | describe('Creation', function(){ 15 | it('should create dummy data', function(done){ 16 | request(baseURL) 17 | .post('/data') 18 | .set('Content-Type', 'application/json') 19 | .send({"ts":"2017-03-11T15:00:53Z", "type": "temp", "value": 34, "sensor_id": 123 }) 20 | .expect(201) 21 | .end(function(err, res){ 22 | if (err) throw err; 23 | done(); 24 | }) 25 | }); 26 | }); 27 | -------------------------------------------------------------------------------- /step2/nodejs/test/mocha.opts: -------------------------------------------------------------------------------- 1 | --timeout 5s 2 | -------------------------------------------------------------------------------- /step3/.gitignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | -------------------------------------------------------------------------------- /step3/README.md: -------------------------------------------------------------------------------- 1 | # Objectives 2 | 3 | In this step, we will create a repository in the Docker Hub that will be used to distribute the *api* image to other environments 4 | 5 | # Instructions 6 | 7 | * Create an account on [Docker Hub](https://hub.docker/com) 8 | * Create a repository named *iot-api* (we might create some other services in the next step, so using the *iot* prefix could be convenient) 9 | * Tag the *iot:v1* image so it matches the repository format 10 | * Push the new tagged to the Docker Hub 11 | 12 | # Note on the iot images 13 | 14 | Until now, we have created 1 version of the *iot* image as the following output confirms 15 | 16 | ```` 17 | $ docker image ls iot 18 | REPOSITORY TAG IMAGE ID CREATED SIZE 19 | iot v1 c243ef955948 6 minutes ago 71 MB 20 | ```` 21 | 22 | As this image only exist locally, we need a way to distribute it so it can be used in other environments. 23 | This is where a registry comes into the picture and in this project we will rely on the official [Docker Hub](https://hub.docker.com). 24 | 25 | # Create a Docker Hub account 26 | 27 | Account creation on the Docker Hub is a simple process that can be done right from the [Docker Hub](https://hub.docker.com) index page. 28 | Just select a username and a password and you'r done. 29 | 30 | # Create a repository 31 | 32 | When logged in the Docker Hub, select the "Create" menu and click on the "Create Repository". In the form displayed, we need to provide the following things 33 | * The name of the repository: *iot-api* in this case 34 | * A short description 35 | * The visibility (set to Public by default) that enables to specify if the image can be downloaded by everybody or only by a limited list of users. 36 | 37 | ![Repository Creation](./images/01-repository-creation.png) 38 | 39 | When the repository is created, the details will be displayed like in the following screenshot. 40 | 41 | ![Repository Created](./images/02-repository-created.png) 42 | 43 | # Tag the existing image 44 | 45 | The last image created locally is the *iot:v1*, as we are pretty happy with this version (at least for now on), we will publish it into our newly created repository. 46 | In order to do so, we first need to tag the image so it follows the USERNAME/REPOSITORY:VERSION format. 47 | 48 | This can easily be done using the following command 49 | 50 | ```` 51 | docker image tag iot:v1 lucj/iot-api:v1 52 | ```` 53 | 54 | We basically tell Docker to add a new tag on the existing *iot:v1* image 55 | 56 | Note: make sure to use your Docker Hub username when tagging the image. 57 | 58 | # Push the new tag to the Docker Hub 59 | 60 | The first step is to login to the Docker Hub througt the command line using the *docker login* command. 61 | 62 | Example: 63 | 64 | ```` 65 | $ docker login 66 | Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. 67 | Username (lucj): 68 | Password: 69 | Login Succeeded 70 | ```` 71 | 72 | Once logged in, the image can be pushed with the *docker image push* command. 73 | 74 | Example: 75 | 76 | ```` 77 | $ docker push lucj/iot-api:v1 78 | The push refers to a repository [docker.io/lucj/iot-api] 79 | 0894754c3439: Pushed 80 | b07357a57dbe: Pushed 81 | 9ba73e8cc696: Pushed 82 | 2eecbdd0e662: Pushed 83 | 8e254b51dfd6: Mounted from mhart/alpine-node 84 | v1: digest: sha256:6d6bec9e59de2e3c59feeb918c2aeba3b3bf09165d0194e5b6968147d52e9355 size: 1580 85 | ```` 86 | 87 | Going back to the Docker Hub dashboard, we can see that a tag now exist for this image. 88 | 89 | ![Image pushed](./images/03-image-pushed.png) 90 | 91 | # Summary 92 | 93 | The image now exist on the Docker Hub and can be easily distributed. 94 | 95 | 96 | ----- 97 | [< Previous](../step2) - [Next >](../step4) 98 | -------------------------------------------------------------------------------- /step3/images/01-repository-creation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step3/images/01-repository-creation.png -------------------------------------------------------------------------------- /step3/images/02-repository-created.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step3/images/02-repository-created.png -------------------------------------------------------------------------------- /step3/images/03-image-pushed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step3/images/03-image-pushed.png -------------------------------------------------------------------------------- /step4/README.md: -------------------------------------------------------------------------------- 1 | # Objectives 2 | 3 | We will not touch our application code in this step and focus instead on the underlying database that we will use to persist the data. 4 | In the next step, we will see how to integrate it on our application. 5 | 6 | To persist the data, we will use [InfluxDB](https://github.com/influxdata/influxdb), a great open source Time Serie Database. 7 | 8 | If you want to know more about InfluxDB, I hightly recommend you have a look at the [documentation](https://docs.influxdata.com/influxdb/v1.2/concepts/key_concepts/). 9 | 10 | # Instructions 11 | 12 | * Go to the Docker Hub page of the official [Influxdb image](https://hub.docker.com/_/influxdb/) 13 | 14 | A lot of information are provided concerning the usage of this image, its configuration, the port that needs to be exported, ... 15 | 16 | * Use the following command to generate the default configuration file 17 | 18 | ```` 19 | docker container run --rm influxdb influxd config > influxdb.conf 20 | ```` 21 | 22 | * Modify the admin part of the configuration file so that the administration interface is enabled 23 | 24 | ```` 25 | [admin] 26 | enabled = true 27 | bind-address = ":8083" 28 | https-enabled = false 29 | https-certificate = "/etc/ssl/influxdb.pem" 30 | ```` 31 | 32 | * Create a volume named *influxdata* 33 | 34 | For this, we use the Docker's volume CLI with the default driver 35 | 36 | ```` 37 | docker volume create influxdata 38 | ```` 39 | 40 | * Run a container using the named volume and the modified configuration file 41 | 42 | To run a InfluxDB container, several option need to be provided to the *docker container run* command to 43 | 44 | * specify the port to publish on the host (8086 is the API port, 8083 is the port used to save the administration web interface) 45 | * bind mount the configuration file from the local folder into the container's /etc/influxdb folder 46 | * specify the volume to use to persist the data 47 | * specify a name that will make it handy to get container's information (more on that soon) 48 | 49 | ```` 50 | docker container run -p 8083:8083 -p 8086:8086 \ 51 | -v $PWD/influxdb.conf:/etc/influxdb/influxdb.conf:ro \ 52 | -v influxdata:/var/lib/influxdb \ 53 | --name influx \ 54 | influxdb -config /etc/influxdb/influxdb.conf 55 | ```` 56 | 57 | For the next step, we will need to get the IP address of the Influx container, let's get it with the following command 58 | 59 | ```` 60 | $ docker container inspect -f '{{ .NetworkSettings.IPAddress }}' influx 61 | 172.17.0.2 62 | ```` 63 | 64 | Note: you might not get the same one in your machine 65 | 66 | # Setting up the database 67 | 68 | Influxdb administration interface should be available on (http://localhost:8083) 69 | 70 | ![Influxdb admin](./images/influxdb-admin.png) 71 | 72 | We will start be creating a database named *iot* using the following HTTP request targeting Influxdb's API 73 | 74 | ```` 75 | curl -i -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE iot" 76 | ```` 77 | 78 | The database creation can easily be done using the administration web interface but it's interesting to see that under the hoods everything happends through HTTP requests. 79 | 80 | ![Iot database](./images/iot-database.png) 81 | 82 | Now that the database is ready, let's create a test entry using InfluxDB's HTTP API. 83 | 84 | ```` 85 | curl -i -XPOST 'http://localhost:8086/write?db=iot' --data-binary 'data,type=temp,sensor_id=123 value=34 1483481572000000000' 86 | ```` 87 | 88 | This HTTP Post request creates a *point*, in the data *measurement*. This point has the fields *sensor_id* and *value* with the respective values of 123 and 34. *type* is a *tag* with the value *temp*. 89 | *measurement*, *point*, *field*, *tag*, ... are all InfluxDB terminology that is cover in the [InfluxDB key concepts](https://docs.influxdata.com/influxdb/v1.2/concepts/key_concepts/). 90 | 91 | Basically, in a first approximation, we can see a *measurment* as a regular SQL table, a *point* as a record of this table. A *point* is a list of *fields* and their value and a list of *tags* and their value, *tags* and *fields* are different in the sense that *tags* are indexed when *fields* are not. 92 | 93 | * Verify using the administration interface 94 | 95 | Issuing the following SQL command within the administration interface enables to retrive the *point* created above. 96 | 97 | ![Iot database](./images/influxdb-query-example.png) 98 | 99 | 100 | ----- 101 | [< Previous](../step3) - [Next >](../step5) 102 | -------------------------------------------------------------------------------- /step4/images/influxdb-admin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step4/images/influxdb-admin.png -------------------------------------------------------------------------------- /step4/images/influxdb-query-example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step4/images/influxdb-query-example.png -------------------------------------------------------------------------------- /step4/images/iot-database.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step4/images/iot-database.png -------------------------------------------------------------------------------- /step4/influxdb.conf: -------------------------------------------------------------------------------- 1 | reporting-disabled = false 2 | bind-address = ":8088" 3 | 4 | [meta] 5 | dir = "/var/lib/influxdb/meta" 6 | retention-autocreate = true 7 | logging-enabled = true 8 | 9 | [data] 10 | dir = "/var/lib/influxdb/data" 11 | wal-dir = "/var/lib/influxdb/wal" 12 | query-log-enabled = true 13 | cache-max-memory-size = 1073741824 14 | cache-snapshot-memory-size = 26214400 15 | cache-snapshot-write-cold-duration = "10m0s" 16 | compact-full-write-cold-duration = "4h0m0s" 17 | max-series-per-database = 1000000 18 | max-values-per-tag = 100000 19 | trace-logging-enabled = false 20 | 21 | [coordinator] 22 | write-timeout = "10s" 23 | max-concurrent-queries = 0 24 | query-timeout = "0s" 25 | log-queries-after = "0s" 26 | max-select-point = 0 27 | max-select-series = 0 28 | max-select-buckets = 0 29 | 30 | [retention] 31 | enabled = true 32 | check-interval = "30m0s" 33 | 34 | [shard-precreation] 35 | enabled = true 36 | check-interval = "10m0s" 37 | advance-period = "30m0s" 38 | 39 | [admin] 40 | enabled = true 41 | bind-address = ":8083" 42 | https-enabled = false 43 | https-certificate = "/etc/ssl/influxdb.pem" 44 | 45 | [monitor] 46 | store-enabled = true 47 | store-database = "_internal" 48 | store-interval = "10s" 49 | 50 | [subscriber] 51 | enabled = true 52 | http-timeout = "30s" 53 | insecure-skip-verify = false 54 | ca-certs = "" 55 | write-concurrency = 40 56 | write-buffer-size = 1000 57 | 58 | [http] 59 | enabled = true 60 | bind-address = ":8086" 61 | auth-enabled = false 62 | log-enabled = true 63 | write-tracing = false 64 | pprof-enabled = true 65 | https-enabled = false 66 | https-certificate = "/etc/ssl/influxdb.pem" 67 | https-private-key = "" 68 | max-row-limit = 10000 69 | max-connection-limit = 0 70 | shared-secret = "" 71 | realm = "InfluxDB" 72 | unix-socket-enabled = false 73 | bind-socket = "/var/run/influxdb.sock" 74 | 75 | [[graphite]] 76 | enabled = false 77 | bind-address = ":2003" 78 | database = "graphite" 79 | retention-policy = "" 80 | protocol = "tcp" 81 | batch-size = 5000 82 | batch-pending = 10 83 | batch-timeout = "1s" 84 | consistency-level = "one" 85 | separator = "." 86 | udp-read-buffer = 0 87 | 88 | [[collectd]] 89 | enabled = false 90 | bind-address = ":25826" 91 | database = "collectd" 92 | retention-policy = "" 93 | batch-size = 5000 94 | batch-pending = 10 95 | batch-timeout = "10s" 96 | read-buffer = 0 97 | typesdb = "/usr/share/collectd/types.db" 98 | security-level = "none" 99 | auth-file = "/etc/collectd/auth_file" 100 | 101 | [[opentsdb]] 102 | enabled = false 103 | bind-address = ":4242" 104 | database = "opentsdb" 105 | retention-policy = "" 106 | consistency-level = "one" 107 | tls-enabled = false 108 | certificate = "/etc/ssl/influxdb.pem" 109 | batch-size = 1000 110 | batch-pending = 5 111 | batch-timeout = "1s" 112 | log-point-errors = true 113 | 114 | [[udp]] 115 | enabled = false 116 | bind-address = ":8089" 117 | database = "udp" 118 | retention-policy = "" 119 | batch-size = 5000 120 | batch-pending = 10 121 | read-buffer = 0 122 | batch-timeout = "1s" 123 | precision = "" 124 | 125 | [continuous_queries] 126 | log-enabled = true 127 | enabled = true 128 | run-interval = "1s" 129 | 130 | -------------------------------------------------------------------------------- /step5/README.md: -------------------------------------------------------------------------------- 1 | # Objectives 2 | 3 | Modify the application so it uses InfluxDB to persist the data received 4 | 5 | # Instructions 6 | 7 | * Use an InfluxDB driver for the langage you selected 8 | * Use this driver to persist the json data received 9 | * Get the *type* tag, the *sensor_id* and *value* fields from the request's body 10 | * Specify the InfluxDB host through the INFLUXDB_HOST environment variable 11 | 12 | # Implementation 13 | 14 | [Node.js](./nodejs) 15 | 16 | # Building the new image 17 | 18 | When the changes above are done in the code, we create a new version of the image. Instead of tagging the image to the format expected by the registry in a second step, we can name use the target image name right away. You will need to use your Docker Hub username though. 19 | 20 | ```` 21 | docker image build -t lucj/iot-api:v2 . 22 | ```` 23 | 24 | # Run a container based on the new image 25 | 26 | As the application now relies on the underlying InfluxDB database, we will provide the IP of the container running InfluxDB to the API container (172.17.0.2 in our example, but yours might be different). 27 | 28 | ```` 29 | docker container run -e "INFLUXDB_HOST=172.17.0.2" -p 1337:1337 lucj/iot-api:v2 30 | ```` 31 | 32 | Notes 33 | * this is a temporary setup to show an example on how a container can use the service of another one. We will enhance this in the next steps when dealing with Docker Compose. 34 | * containers can see each other using their IP addresses as they are on the same *Docker0* default bridge network. 35 | 36 | # Push the new version of image to the registry 37 | 38 | As the image is already tagged into the correct format, we just need to push it to the Docker Hub. 39 | 40 | ```` 41 | $ docker image push lucj/iot-api:v2 42 | The push refers to a repository [docker.io/lucj/iot-api] 43 | f3e959ead19c: Pushed 44 | e8800c4ab975: Pushed 45 | 03383574868d: Pushed 46 | b2f6f834570f: Pushed 47 | 8e254b51dfd6: Layer already exists 48 | 60ab55d3379d: Layer already exists 49 | v2: digest: sha256:7a9616b6ae3b1b57f48a96c5b3c9dfcf27bd3069bbedbdde449d09e1cd759499 size: 1580 50 | ```` 51 | 52 | Both tags, *v1* and *v2* are available on the Docker Hub. 53 | 54 | ![Image list](./images/docker-hub-iot-api-images.png) 55 | 56 | ----- 57 | [< Previous](../step4) - [Next >](../step6) 58 | -------------------------------------------------------------------------------- /step5/images/docker-hub-iot-api-images.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step5/images/docker-hub-iot-api-images.png -------------------------------------------------------------------------------- /step5/nodejs/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM mhart/alpine-node:7.7.1 2 | ENV LAST_UPDATED 20170301T231500 3 | 4 | # Copy list of server side dependencies 5 | COPY package.json /tmp/package.json 6 | 7 | # Install dependencies 8 | RUN cd /tmp && npm install 9 | 10 | # Copy dependencies libraries 11 | RUN mkdir /app && cp -a /tmp/node_modules /app/ 12 | 13 | # Copy src files 14 | COPY . /app/ 15 | 16 | # Use /app working directory 17 | WORKDIR /app 18 | 19 | # Expose http port 20 | EXPOSE 1337 21 | 22 | # Run application 23 | CMD ["npm", "start"] 24 | -------------------------------------------------------------------------------- /step5/nodejs/README.md: -------------------------------------------------------------------------------- 1 | In this implementation, we are using the *influx* npm library to connect to the underlying InfluxDB database. 2 | 3 | ## Starting the API 4 | 5 | To start the API, the IP address of the InfluxDB host needs to be provided. 6 | 7 | ```` 8 | INFLUXDB_HOST=localhost npm start 9 | ```` 10 | 11 | ## Running the test 12 | 13 | To test the API, the IP address of the underlying InfluxDB database also need to be provided 14 | 15 | ```` 16 | INFLUXDB_HOST=localhost npm test 17 | 18 | > iot@1.0.0 test /Users/luc/perso/Dropbox/Work/Side/Docker/IoT-demo-project/step4/nodejs 19 | > mocha test/functional.js 20 | 21 | 22 | Creation 23 | info: server listening on port 3000 24 | info: ts=2017-03-11T15:00:53Z, type=temp, value=34, sensor_id=123 25 | ✓ should create dummy data (83ms) 26 | 27 | 28 | 1 passing (97ms) 29 | ```` 30 | -------------------------------------------------------------------------------- /step5/nodejs/app.js: -------------------------------------------------------------------------------- 1 | // Load dependencies 2 | const express = require('express'), 3 | Influx = require('influx'), 4 | bodyParser = require('body-parser'), 5 | winston = require('winston'); 6 | 7 | // Create express application 8 | let app = module.exports = express(); 9 | 10 | // Create a client towards InfluxDB 11 | let influx = new Influx.InfluxDB({ 12 | host: process.env.INFLUXDB_HOST, 13 | database: 'iot' 14 | }); 15 | 16 | // Body parser configuration 17 | app.use(bodyParser.json()); 18 | app.use(bodyParser.urlencoded({ extended: true })); 19 | 20 | // Handle inconming data 21 | app.post('/data', 22 | function(req, res, next){ 23 | influx.writePoints([ 24 | { 25 | measurement: 'data', 26 | tags: { type: req.body.type }, 27 | fields: { sensor_id: req.body.sensor_id, value: req.body.value }, 28 | timestamp: new Date(req.body.ts).getTime() * 1000000 29 | } 30 | ]).then(() => { 31 | winston.info(req.body); 32 | return res.sendStatus(201); 33 | }) 34 | .catch( err => { 35 | winston.error(err.message); 36 | return res.sendStatus(500); 37 | }); 38 | }); 39 | -------------------------------------------------------------------------------- /step5/nodejs/index.js: -------------------------------------------------------------------------------- 1 | // Load dependencies 2 | const util = require('util'), 3 | winston = require('winston'), 4 | app = require('./app'); 5 | 6 | // Define API port 7 | let port = process.env.PORT || 1337; 8 | 9 | // Run API 10 | app.listen(port, function(){ 11 | winston.info(util.format("server listening on port %s", port)); 12 | }); 13 | -------------------------------------------------------------------------------- /step5/nodejs/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "iot", 3 | "version": "1.0.0", 4 | "description": "IoT example project", 5 | "main": "index.js", 6 | "scripts": { 7 | "start": "node index.js", 8 | "test": "./node_modules/.bin/mocha test/functional.js" 9 | }, 10 | "author": "Luc Juggery", 11 | "license": "MIT", 12 | "dependencies": { 13 | "body-parser": "^1.17.1", 14 | "express": "^4.15.2", 15 | "influx": "^5.0.6", 16 | "winston": "^2.3.1" 17 | }, 18 | "devDependencies": { 19 | "mocha": "^3.2.0", 20 | "supertest": "^3.0.0" 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /step5/nodejs/test/functional.js: -------------------------------------------------------------------------------- 1 | const request = require('supertest'), 2 | app = require('../app'); 3 | winston = require('winston'), 4 | util = require('util'), 5 | port = 3000, 6 | baseURL = util.format('http://localhost:%s', port); 7 | 8 | before(function(){ 9 | app.listen(port, function(){ 10 | winston.info(util.format("server listening on port %s", port)); 11 | }); 12 | }); 13 | 14 | describe('Creation', function(){ 15 | it('should create dummy data', function(done){ 16 | request(baseURL) 17 | .post('/data') 18 | .set('Content-Type', 'application/json') 19 | .send({"ts":"2017-03-11T15:00:53Z", "type": "temp", "value": 34, "sensor_id": 123 }) 20 | .expect(201) 21 | .end(function(err, res){ 22 | if (err) throw err; 23 | done(); 24 | }) 25 | }); 26 | }); 27 | -------------------------------------------------------------------------------- /step5/nodejs/test/mocha.opts: -------------------------------------------------------------------------------- 1 | --timeout 5s 2 | -------------------------------------------------------------------------------- /step6/.gitignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | -------------------------------------------------------------------------------- /step6/README.md: -------------------------------------------------------------------------------- 1 | # Objectives 2 | 3 | In this step, we will add Docker Compose to the picture and define the multi containers application 4 | 5 | # Instructions 6 | 7 | Change the code so that the default INFLUXDB_HOST is "db" (the name of the InfluxDB service) if no other value is provided as an environment variable. 8 | 9 | # Example implementation 10 | 11 | [Node.js](./nodejs) 12 | 13 | # Compose file 14 | 15 | The compose file is defined as follow 16 | 17 | * 2 services: *db* and *api* 18 | * definition of *influxdata* volume and mount it on /var/lib/influxdb of the *db* service 19 | * usage of the influxdb.conf file present locally 20 | * publication of port 8083 and 8086 of the *db* service 21 | * publication of port 1337 of the *api* service 22 | 23 | # Running the application 24 | 25 | To take into account the changes done in the code (modification of the default INFLUXDB_HOST), let's run the application using the *--build* option. 26 | 27 | Note: we are using here the *build* and the *image* instructions so the version *v3* of the image can be created directly with Docker Compose using the following command: 28 | 29 | ```` 30 | $ docker-compose up --build 31 | Creating network "step5_default" with the default driver 32 | Creating volume "step5_influxdata" with default driver 33 | Building api 34 | Step 1/9 : FROM mhart/alpine-node:7.7.1 35 | ---> e1a533c514f2 36 | Step 2/9 : ENV LAST_UPDATED 20170301T231500 37 | ---> Using cache 38 | ---> 4a6e696daca9 39 | Step 3/9 : COPY package.json /tmp/package.json 40 | ---> 47f600d37818 41 | Removing intermediate container 886efe383c7d 42 | Step 4/9 : RUN cd /tmp && npm install 43 | ---> Running in f5398f02b98c 44 | ... 45 | Removing intermediate container f5398f02b98c 46 | Step 5/9 : RUN mkdir /app && cp -a /tmp/node_modules /app/ 47 | ---> Running in 1f78c73d24ec 48 | ---> 7f7e7667f3db 49 | Removing intermediate container 1f78c73d24ec 50 | Step 6/9 : COPY . /app/ 51 | ---> aa3f46efeb3f 52 | Removing intermediate container 0b3225eb929b 53 | Step 7/9 : WORKDIR /app 54 | ---> 053f8371a52f 55 | Removing intermediate container 62d4ab2dcdcc 56 | Step 8/9 : EXPOSE 1337 57 | ---> Running in 013e278b6ad5 58 | ---> 85ac56c8afd2 59 | Removing intermediate container 013e278b6ad5 60 | Step 9/9 : CMD npm start 61 | ---> Running in c9fdf62f3dad 62 | ---> dea68591ed39 63 | Removing intermediate container c9fdf62f3dad 64 | Successfully built dea68591ed39 65 | Creating step5_api_1 66 | Creating step5_db_1 67 | Attaching to step5_api_1, step5_db_1 68 | db_1 | [I] 2017-03-12T21:10:21Z InfluxDB starting, version 1.2.1, branch master, commit 3ec60fe2649b51a85cd1db6c8937320a80a64c35 69 | db_1 | [I] 2017-03-12T21:10:21Z Go version go1.7.4, GOMAXPROCS set to 4 70 | db_1 | [I] 2017-03-12T21:10:21Z Using configuration at: /etc/influxdb/influxdb.conf 71 | db_1 | 72 | db_1 | 8888888 .d888 888 8888888b. 888888b. 73 | db_1 | 888 d88P" 888 888 "Y88b 888 "88b 74 | db_1 | 888 888 888 888 888 888 .88P 75 | db_1 | 888 88888b. 888888 888 888 888 888 888 888 888 8888888K. 76 | db_1 | 888 888 "88b 888 888 888 888 Y8bd8P' 888 888 888 "Y88b 77 | db_1 | 888 888 888 888 888 888 888 X88K 888 888 888 888 78 | db_1 | 888 888 888 888 888 Y88b 888 .d8""8b. 888 .d88P 888 d88P 79 | db_1 | 8888888 888 888 888 888 "Y88888 888 888 8888888P" 8888888P" 80 | db_1 | 81 | api_1 | 82 | api_1 | > iot@1.0.0 start /app 83 | api_1 | > node index.js 84 | api_1 | 85 | db_1 | [I] 2017-03-12T21:10:21Z Using data dir: /var/lib/influxdb/data service=store 86 | db_1 | [I] 2017-03-12T21:10:21Z opened service service=subscriber 87 | db_1 | [I] 2017-03-12T21:10:21Z Starting monitor system service=monitor 88 | db_1 | [I] 2017-03-12T21:10:21Z 'build' registered for diagnostics monitoring service=monitor 89 | db_1 | [I] 2017-03-12T21:10:21Z 'runtime' registered for diagnostics monitoring service=monitor 90 | db_1 | [I] 2017-03-12T21:10:21Z 'network' registered for diagnostics monitoring service=monitor 91 | db_1 | [I] 2017-03-12T21:10:21Z 'system' registered for diagnostics monitoring service=monitor 92 | db_1 | [I] 2017-03-12T21:10:21Z Starting precreation service with check interval of 10m0s, advance period of 30m0s service=shard-precreation 93 | db_1 | [I] 2017-03-12T21:10:21Z Starting snapshot service service=snapshot 94 | db_1 | [I] 2017-03-12T21:10:21Z Starting admin service service=admin 95 | db_1 | [I] 2017-03-12T21:10:21Z DEPRECATED: This plugin is deprecated as of 1.1.0 and will be removed in a future release service=admin 96 | db_1 | [I] 2017-03-12T21:10:21Z Listening on HTTP: [::]:8083 service=admin 97 | db_1 | [I] 2017-03-12T21:10:21Z Starting continuous query service service=continuous_querier 98 | db_1 | [I] 2017-03-12T21:10:21Z Starting HTTP service service=httpd 99 | db_1 | [I] 2017-03-12T21:10:21Z Authentication enabled:false service=httpd 100 | db_1 | [I] 2017-03-12T21:10:21Z Listening on HTTP:[::]:8086 service=httpd 101 | db_1 | [I] 2017-03-12T21:10:21Z Starting retention policy enforcement service with check interval of 30m0s service=retention 102 | db_1 | [I] 2017-03-12T21:10:21Z Listening for signals 103 | db_1 | [I] 2017-03-12T21:10:21Z Storing statistics in database '_internal' retention policy 'monitor', at interval 10s service=monitor 104 | db_1 | [I] 2017-03-12T21:10:21Z Sending usage statistics to usage.influxdata.com 105 | api_1 | info: server listening on port 1337 106 | ... 107 | ```` 108 | 109 | From the output we can see the creation of several components 110 | * the user defined bridge network the containers will be connected to 111 | * the volume defined to handle InfluxDB data 112 | * the build of the *api* image 113 | * the services' containers, *db* and *api* 114 | 115 | # Create the *iot* database 116 | 117 | Let's use the InfluxDB administration interface to create our database, named *iot*. 118 | 119 | ![Create database from administration interface](./images/01-create-iot-db.png) 120 | 121 | # Push the newly created image 122 | 123 | Using the *--build* option when running *docker-compose up*, the *v3* version of the image has been created. 124 | 125 | Let's push it to Docker Hub. 126 | 127 | ```` 128 | $ docker image push lucj/iot-api:v3 129 | The push refers to a repository [docker.io/lucj/iot-api] 130 | c77035b16cb3: Pushed 131 | 906e9be4d576: Pushed 132 | d712cb66e5d4: Pushed 133 | 264232f2575c: Pushed 134 | 8e254b51dfd6: Mounted from mhart/alpine-node 135 | 60ab55d3379d: Mounted from mhart/alpine-node 136 | v3: digest: sha256:4d6163739eb6cc2ce936864c30af5b7a71263d7e4121ad2f9482481d28147cf5 size: 1580 137 | ```` 138 | 139 | ----- 140 | [< Previous](../step5) - [Next >](../step7) 141 | -------------------------------------------------------------------------------- /step6/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.1" 2 | services: 3 | db: 4 | image: influxdb:1.2.4 5 | command: -config /etc/influxdb/influxdb.conf 6 | ports: 7 | - 8083:8083 8 | - 8086:8086 9 | volumes: 10 | - ./influxdb.conf:/etc/influxdb/influxdb.conf:ro 11 | - influxdata:/var/lib/influxdb 12 | api: 13 | image: lucj/iot-api:v3 14 | build: ./nodejs 15 | command: npm start 16 | ports: 17 | - 1337:1337 18 | volumes: 19 | influxdata: 20 | -------------------------------------------------------------------------------- /step6/images/01-create-iot-db.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step6/images/01-create-iot-db.png -------------------------------------------------------------------------------- /step6/influxdb.conf: -------------------------------------------------------------------------------- 1 | reporting-disabled = false 2 | bind-address = ":8088" 3 | 4 | [meta] 5 | dir = "/var/lib/influxdb/meta" 6 | retention-autocreate = true 7 | logging-enabled = true 8 | 9 | [data] 10 | dir = "/var/lib/influxdb/data" 11 | wal-dir = "/var/lib/influxdb/wal" 12 | query-log-enabled = true 13 | cache-max-memory-size = 1073741824 14 | cache-snapshot-memory-size = 26214400 15 | cache-snapshot-write-cold-duration = "10m0s" 16 | compact-full-write-cold-duration = "4h0m0s" 17 | max-series-per-database = 1000000 18 | max-values-per-tag = 100000 19 | trace-logging-enabled = false 20 | 21 | [coordinator] 22 | write-timeout = "10s" 23 | max-concurrent-queries = 0 24 | query-timeout = "0s" 25 | log-queries-after = "0s" 26 | max-select-point = 0 27 | max-select-series = 0 28 | max-select-buckets = 0 29 | 30 | [retention] 31 | enabled = true 32 | check-interval = "30m0s" 33 | 34 | [shard-precreation] 35 | enabled = true 36 | check-interval = "10m0s" 37 | advance-period = "30m0s" 38 | 39 | [admin] 40 | enabled = true 41 | bind-address = ":8083" 42 | https-enabled = false 43 | https-certificate = "/etc/ssl/influxdb.pem" 44 | 45 | [monitor] 46 | store-enabled = true 47 | store-database = "_internal" 48 | store-interval = "10s" 49 | 50 | [subscriber] 51 | enabled = true 52 | http-timeout = "30s" 53 | insecure-skip-verify = false 54 | ca-certs = "" 55 | write-concurrency = 40 56 | write-buffer-size = 1000 57 | 58 | [http] 59 | enabled = true 60 | bind-address = ":8086" 61 | auth-enabled = false 62 | log-enabled = true 63 | write-tracing = false 64 | pprof-enabled = true 65 | https-enabled = false 66 | https-certificate = "/etc/ssl/influxdb.pem" 67 | https-private-key = "" 68 | max-row-limit = 10000 69 | max-connection-limit = 0 70 | shared-secret = "" 71 | realm = "InfluxDB" 72 | unix-socket-enabled = false 73 | bind-socket = "/var/run/influxdb.sock" 74 | 75 | [[graphite]] 76 | enabled = false 77 | bind-address = ":2003" 78 | database = "graphite" 79 | retention-policy = "" 80 | protocol = "tcp" 81 | batch-size = 5000 82 | batch-pending = 10 83 | batch-timeout = "1s" 84 | consistency-level = "one" 85 | separator = "." 86 | udp-read-buffer = 0 87 | 88 | [[collectd]] 89 | enabled = false 90 | bind-address = ":25826" 91 | database = "collectd" 92 | retention-policy = "" 93 | batch-size = 5000 94 | batch-pending = 10 95 | batch-timeout = "10s" 96 | read-buffer = 0 97 | typesdb = "/usr/share/collectd/types.db" 98 | security-level = "none" 99 | auth-file = "/etc/collectd/auth_file" 100 | 101 | [[opentsdb]] 102 | enabled = false 103 | bind-address = ":4242" 104 | database = "opentsdb" 105 | retention-policy = "" 106 | consistency-level = "one" 107 | tls-enabled = false 108 | certificate = "/etc/ssl/influxdb.pem" 109 | batch-size = 1000 110 | batch-pending = 5 111 | batch-timeout = "1s" 112 | log-point-errors = true 113 | 114 | [[udp]] 115 | enabled = false 116 | bind-address = ":8089" 117 | database = "udp" 118 | retention-policy = "" 119 | batch-size = 5000 120 | batch-pending = 10 121 | read-buffer = 0 122 | batch-timeout = "1s" 123 | precision = "" 124 | 125 | [continuous_queries] 126 | log-enabled = true 127 | enabled = true 128 | run-interval = "1s" 129 | 130 | -------------------------------------------------------------------------------- /step6/nodejs/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM mhart/alpine-node:7.7.1 2 | ENV LAST_UPDATED 20170301T231500 3 | 4 | # Copy list of server side dependencies 5 | COPY package.json /tmp/package.json 6 | 7 | # Install dependencies 8 | RUN cd /tmp && npm install 9 | 10 | # Copy dependencies libraries 11 | RUN mkdir /app && cp -a /tmp/node_modules /app/ 12 | 13 | # Copy src files 14 | COPY . /app/ 15 | 16 | # Use /app working directory 17 | WORKDIR /app 18 | 19 | # Expose http port 20 | EXPOSE 1337 21 | 22 | # Run application 23 | CMD ["npm", "start"] 24 | -------------------------------------------------------------------------------- /step6/nodejs/README.md: -------------------------------------------------------------------------------- 1 | In this step, we define a default value, "db", for the InfluxDB host the *api* needs to connect to. 2 | 3 | ```` 4 | // Create a client towards InfluxDB 5 | let influx = new Influx.InfluxDB({ 6 | host: process.env.INFLUXDB_HOST || "db", 7 | database: 'iot' 8 | }); 9 | ```` 10 | 11 | As each service of a Docker Compose application can communicate with other services using their name, this code changes replaces the environment variable that would be defined in the *docker-compose.yml* file otherwise. 12 | -------------------------------------------------------------------------------- /step6/nodejs/app.js: -------------------------------------------------------------------------------- 1 | // Load dependencies 2 | const express = require('express'), 3 | Influx = require('influx'), 4 | bodyParser = require('body-parser'), 5 | winston = require('winston'); 6 | 7 | // Create express application 8 | let app = module.exports = express(); 9 | 10 | // Create a client towards InfluxDB 11 | let influx = new Influx.InfluxDB({ 12 | host: process.env.INFLUXDB_HOST || "db", 13 | database: 'iot' 14 | }); 15 | 16 | // Body parser configuration 17 | app.use(bodyParser.json()); 18 | app.use(bodyParser.urlencoded({ extended: true })); 19 | 20 | // Handle inconming data 21 | app.post('/data', 22 | function(req, res, next){ 23 | influx.writePoints([ 24 | { 25 | measurement: 'data', 26 | tags: { type: req.body.type }, 27 | fields: { sensor_id: req.body.sensor_id, value: req.body.value }, 28 | timestamp: new Date(req.body.ts).getTime() * 1000000 29 | } 30 | ]).then(() => { 31 | winston.info(req.body); 32 | return res.sendStatus(201); 33 | }) 34 | .catch( err => { 35 | winston.error(err.message); 36 | return res.sendStatus(500); 37 | }); 38 | }); 39 | -------------------------------------------------------------------------------- /step6/nodejs/index.js: -------------------------------------------------------------------------------- 1 | // Load dependencies 2 | const util = require('util'), 3 | winston = require('winston'), 4 | app = require('./app'); 5 | 6 | // Define API port 7 | let port = process.env.PORT || 1337; 8 | 9 | // Run API 10 | app.listen(port, function(){ 11 | winston.info(util.format("server listening on port %s", port)); 12 | }); 13 | -------------------------------------------------------------------------------- /step6/nodejs/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "iot", 3 | "version": "1.0.0", 4 | "description": "IoT example project", 5 | "main": "index.js", 6 | "scripts": { 7 | "start": "node index.js", 8 | "test": "./node_modules/.bin/mocha test/functional.js" 9 | }, 10 | "author": "Luc Juggery", 11 | "license": "MIT", 12 | "dependencies": { 13 | "body-parser": "^1.17.1", 14 | "express": "^4.15.2", 15 | "influx": "^5.0.6", 16 | "winston": "^2.3.1" 17 | }, 18 | "devDependencies": { 19 | "mocha": "^3.2.0", 20 | "supertest": "^3.0.0" 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /step6/nodejs/test/functional.js: -------------------------------------------------------------------------------- 1 | const request = require('supertest'), 2 | app = require('../app'); 3 | winston = require('winston'), 4 | util = require('util'), 5 | port = 3000, 6 | baseURL = util.format('http://localhost:%s', port); 7 | 8 | before(function(){ 9 | app.listen(port, function(){ 10 | winston.info(util.format("server listening on port %s", port)); 11 | }); 12 | }); 13 | 14 | describe('Creation', function(){ 15 | it('should create dummy data', function(done){ 16 | request(baseURL) 17 | .post('/data') 18 | .set('Content-Type', 'application/json') 19 | .send({"ts":"2017-03-11T15:00:53Z", "type": "temp", "value": 34, "sensor_id": 123 }) 20 | .expect(201) 21 | .end(function(err, res){ 22 | if (err) throw err; 23 | done(); 24 | }) 25 | }); 26 | }); 27 | -------------------------------------------------------------------------------- /step6/nodejs/test/mocha.opts: -------------------------------------------------------------------------------- 1 | --timeout 5s 2 | -------------------------------------------------------------------------------- /step7/.gitignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | -------------------------------------------------------------------------------- /step7/README.md: -------------------------------------------------------------------------------- 1 | # Objectives 2 | 3 | In this step, we will add Grafana as the visualisation service of our application. 4 | [Grafana](http://grafana.org/) enables to create really great and neat dashboards. 5 | We will see how easily this can be added to our Docker Compose application. 6 | 7 | # Instructions 8 | 9 | * Add a service, named dashboard, in the docker-compose file using grafana image 10 | * Publish the port 3000 of the Grafana container onto the port 3000 of the host 11 | * Run the compose application 12 | * Run the simulator 13 | 14 | # Details 15 | 16 | ## About Grafana 17 | 18 | Grafana enables to create great looking dashboard, just have a look at the following one... neat isn't it ? 19 | 20 | ![Grafana dashboard example](./images/00-grafana-dashboard-example.png) 21 | 22 | Checking the official Grafana image from the [Docker Hub](https://hub.docker.com/r/grafana/grafana/) we get information on how to run the application and which port needs to be exposed. 23 | This is as easy as this: 24 | 25 | ```` 26 | docker container run -p 3000:3000 grafana/grafana 27 | ```` 28 | 29 | ## Adding the dashboard service 30 | 31 | In the docker-commose.yml file, we will add a new service, named *dashbaord*, based on the Grafana image. 32 | The service will be simply defined as follow: 33 | 34 | ```` 35 | dashboard: 36 | image: grafana/grafana 37 | ports: 38 | - 3000:3000 39 | ```` 40 | 41 | ## *iot* database 42 | 43 | Let's use the InfluxDB administration interface to create our database, named *iot*. 44 | 45 | ![Create database from administration interface](./images/01-create-iot-db.png) 46 | 47 | ## First step with Grafana 48 | 49 | The Grafana interface is available on port 3000 (port exposed in the docker-compose.yml file). 50 | The default administration credentials are *admin*/*admin*. 51 | 52 | ![Grafana login page](./images/02-grafana-admin.png) 53 | 54 | Once logged-in the next step is to configure the source of data we need to use to get the data. 55 | 56 | ![Grafana welcome](./images/03-grafana-welcome.png) 57 | 58 | ## Configure the data source 59 | 60 | By default, Grafana can get data from several data sources and InfluxDB is one of them. The screenshot below shows the configuration that needs to be used. 61 | The important things to note here is the URL used to target InfluxDB's api: *http://db:8086*. This is possible because Docker Compose enables the services to communicates with each other using their names. 62 | 63 | ![Grafana data source creation](./images/04-grafana-datasource.png) 64 | 65 | ## Create a dashboard 66 | 67 | The screenshot below shows how to modify the InfluxDB request to get all the data from the *data* measurement of the *iot* database. 68 | 69 | ![Grafana dashboard creation](./images/05-grafana-dashboard.png) 70 | 71 | ## Run the simulator 72 | 73 | Let's run the simulator, wait a couple of seconds and check the data appearing on the dashboard created above. 74 | 75 | ```` 76 | ./simulator.sh 77 | ```` 78 | 79 | ![Grafana data sample](./images/06-grafana-data-samples.png) 80 | 81 | 82 | ----- 83 | [< Previous](../step6) - [Next >](../step8) 84 | -------------------------------------------------------------------------------- /step7/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.1" 2 | services: 3 | db: 4 | image: influxdb:1.2.4 5 | command: -config /etc/influxdb/influxdb.conf 6 | ports: 7 | - 8083:8083 8 | - 8086:8086 9 | volumes: 10 | - ./influxdb.conf:/etc/influxdb/influxdb.conf:ro 11 | - influxdata:/var/lib/influxdb 12 | api: 13 | image: iot:v3 14 | build: ./nodejs 15 | command: npm start 16 | ports: 17 | - 1337:1337 18 | dashboard: 19 | image: grafana/grafana 20 | ports: 21 | - 3000:3000 22 | volumes: 23 | influxdata: 24 | -------------------------------------------------------------------------------- /step7/images/00-grafana-dashboard-example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step7/images/00-grafana-dashboard-example.png -------------------------------------------------------------------------------- /step7/images/01-create-iot-db.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step7/images/01-create-iot-db.png -------------------------------------------------------------------------------- /step7/images/02-grafana-admin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step7/images/02-grafana-admin.png -------------------------------------------------------------------------------- /step7/images/03-grafana-welcome.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step7/images/03-grafana-welcome.png -------------------------------------------------------------------------------- /step7/images/04-grafana-datasource.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step7/images/04-grafana-datasource.png -------------------------------------------------------------------------------- /step7/images/05-grafana-dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step7/images/05-grafana-dashboard.png -------------------------------------------------------------------------------- /step7/images/06-grafana-data-samples.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step7/images/06-grafana-data-samples.png -------------------------------------------------------------------------------- /step7/influxdb.conf: -------------------------------------------------------------------------------- 1 | reporting-disabled = false 2 | bind-address = ":8088" 3 | 4 | [meta] 5 | dir = "/var/lib/influxdb/meta" 6 | retention-autocreate = true 7 | logging-enabled = true 8 | 9 | [data] 10 | dir = "/var/lib/influxdb/data" 11 | wal-dir = "/var/lib/influxdb/wal" 12 | query-log-enabled = true 13 | cache-max-memory-size = 1073741824 14 | cache-snapshot-memory-size = 26214400 15 | cache-snapshot-write-cold-duration = "10m0s" 16 | compact-full-write-cold-duration = "4h0m0s" 17 | max-series-per-database = 1000000 18 | max-values-per-tag = 100000 19 | trace-logging-enabled = false 20 | 21 | [coordinator] 22 | write-timeout = "10s" 23 | max-concurrent-queries = 0 24 | query-timeout = "0s" 25 | log-queries-after = "0s" 26 | max-select-point = 0 27 | max-select-series = 0 28 | max-select-buckets = 0 29 | 30 | [retention] 31 | enabled = true 32 | check-interval = "30m0s" 33 | 34 | [shard-precreation] 35 | enabled = true 36 | check-interval = "10m0s" 37 | advance-period = "30m0s" 38 | 39 | [admin] 40 | enabled = true 41 | bind-address = ":8083" 42 | https-enabled = false 43 | https-certificate = "/etc/ssl/influxdb.pem" 44 | 45 | [monitor] 46 | store-enabled = true 47 | store-database = "_internal" 48 | store-interval = "10s" 49 | 50 | [subscriber] 51 | enabled = true 52 | http-timeout = "30s" 53 | insecure-skip-verify = false 54 | ca-certs = "" 55 | write-concurrency = 40 56 | write-buffer-size = 1000 57 | 58 | [http] 59 | enabled = true 60 | bind-address = ":8086" 61 | auth-enabled = false 62 | log-enabled = true 63 | write-tracing = false 64 | pprof-enabled = true 65 | https-enabled = false 66 | https-certificate = "/etc/ssl/influxdb.pem" 67 | https-private-key = "" 68 | max-row-limit = 10000 69 | max-connection-limit = 0 70 | shared-secret = "" 71 | realm = "InfluxDB" 72 | unix-socket-enabled = false 73 | bind-socket = "/var/run/influxdb.sock" 74 | 75 | [[graphite]] 76 | enabled = false 77 | bind-address = ":2003" 78 | database = "graphite" 79 | retention-policy = "" 80 | protocol = "tcp" 81 | batch-size = 5000 82 | batch-pending = 10 83 | batch-timeout = "1s" 84 | consistency-level = "one" 85 | separator = "." 86 | udp-read-buffer = 0 87 | 88 | [[collectd]] 89 | enabled = false 90 | bind-address = ":25826" 91 | database = "collectd" 92 | retention-policy = "" 93 | batch-size = 5000 94 | batch-pending = 10 95 | batch-timeout = "10s" 96 | read-buffer = 0 97 | typesdb = "/usr/share/collectd/types.db" 98 | security-level = "none" 99 | auth-file = "/etc/collectd/auth_file" 100 | 101 | [[opentsdb]] 102 | enabled = false 103 | bind-address = ":4242" 104 | database = "opentsdb" 105 | retention-policy = "" 106 | consistency-level = "one" 107 | tls-enabled = false 108 | certificate = "/etc/ssl/influxdb.pem" 109 | batch-size = 1000 110 | batch-pending = 5 111 | batch-timeout = "1s" 112 | log-point-errors = true 113 | 114 | [[udp]] 115 | enabled = false 116 | bind-address = ":8089" 117 | database = "udp" 118 | retention-policy = "" 119 | batch-size = 5000 120 | batch-pending = 10 121 | read-buffer = 0 122 | batch-timeout = "1s" 123 | precision = "" 124 | 125 | [continuous_queries] 126 | log-enabled = true 127 | enabled = true 128 | run-interval = "1s" 129 | 130 | -------------------------------------------------------------------------------- /step7/nodejs/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM mhart/alpine-node:7.7.1 2 | ENV LAST_UPDATED 20170301T231500 3 | 4 | # Copy list of server side dependencies 5 | COPY package.json /tmp/package.json 6 | 7 | # Install dependencies 8 | RUN cd /tmp && npm install 9 | 10 | # Copy dependencies libraries 11 | RUN mkdir /app && cp -a /tmp/node_modules /app/ 12 | 13 | # Copy src files 14 | COPY . /app/ 15 | 16 | # Use /app working directory 17 | WORKDIR /app 18 | 19 | # Expose http port 20 | EXPOSE 1337 21 | 22 | # Run application 23 | CMD ["npm", "start"] 24 | -------------------------------------------------------------------------------- /step7/nodejs/README.md: -------------------------------------------------------------------------------- 1 | In this implementation, we are using the *influx* npm library to connect to the underlying InfluxDB database. 2 | 3 | ## Starting the API 4 | 5 | To start the API, the IP address of the InfluxDB host needs to be provided. 6 | 7 | ```` 8 | INFLUXDB_HOST=localhost npm start 9 | ```` 10 | 11 | ## Running the test 12 | 13 | To test the API, the IP address of the underlying InfluxDB database also need to be provided 14 | 15 | ```` 16 | INFLUXDB_HOST=localhost npm test 17 | 18 | > iot@1.0.0 test /Users/luc/perso/Dropbox/Work/Side/Docker/IoT-demo-project/step4/nodejs 19 | > mocha test/functional.js 20 | 21 | 22 | Creation 23 | info: server listening on port 3000 24 | info: ts=2017-03-11T15:00:53Z, type=temp, value=34, sensor_id=123 25 | ✓ should create dummy data (83ms) 26 | 27 | 28 | 1 passing (97ms) 29 | ```` 30 | -------------------------------------------------------------------------------- /step7/nodejs/app.js: -------------------------------------------------------------------------------- 1 | // Load dependencies 2 | const express = require('express'), 3 | Influx = require('influx'), 4 | bodyParser = require('body-parser'), 5 | winston = require('winston'); 6 | 7 | // Create express application 8 | let app = module.exports = express(); 9 | 10 | // Create a client towards InfluxDB 11 | let influx = new Influx.InfluxDB({ 12 | host: process.env.INFLUXDB_HOST || "db", 13 | database: 'iot' 14 | }); 15 | 16 | // Body parser configuration 17 | app.use(bodyParser.json()); 18 | app.use(bodyParser.urlencoded({ extended: true })); 19 | 20 | // Handle inconming data 21 | app.post('/data', 22 | function(req, res, next){ 23 | influx.writePoints([ 24 | { 25 | measurement: 'data', 26 | tags: { type: req.body.type }, 27 | fields: { sensor_id: req.body.sensor_id, value: req.body.value }, 28 | timestamp: new Date(req.body.ts).getTime() * 1000000 29 | } 30 | ]).then(() => { 31 | winston.info(req.body); 32 | return res.sendStatus(201); 33 | }) 34 | .catch( err => { 35 | winston.error(err.message); 36 | return res.sendStatus(500); 37 | }); 38 | }); 39 | -------------------------------------------------------------------------------- /step7/nodejs/index.js: -------------------------------------------------------------------------------- 1 | // Load dependencies 2 | const util = require('util'), 3 | winston = require('winston'), 4 | app = require('./app'); 5 | 6 | // Define API port 7 | let port = process.env.PORT || 1337; 8 | 9 | // Run API 10 | app.listen(port, function(){ 11 | winston.info(util.format("server listening on port %s", port)); 12 | }); 13 | -------------------------------------------------------------------------------- /step7/nodejs/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "iot", 3 | "version": "1.0.0", 4 | "description": "IoT example project", 5 | "main": "index.js", 6 | "scripts": { 7 | "start": "node index.js", 8 | "test": "./node_modules/.bin/mocha test/functional.js" 9 | }, 10 | "author": "Luc Juggery", 11 | "license": "MIT", 12 | "dependencies": { 13 | "body-parser": "^1.17.1", 14 | "express": "^4.15.2", 15 | "influx": "^5.0.6", 16 | "winston": "^2.3.1" 17 | }, 18 | "devDependencies": { 19 | "mocha": "^3.2.0", 20 | "supertest": "^3.0.0" 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /step7/nodejs/test/functional.js: -------------------------------------------------------------------------------- 1 | const request = require('supertest'), 2 | app = require('../app'); 3 | winston = require('winston'), 4 | util = require('util'), 5 | port = 3000, 6 | baseURL = util.format('http://localhost:%s', port); 7 | 8 | before(function(){ 9 | app.listen(port, function(){ 10 | winston.info(util.format("server listening on port %s", port)); 11 | }); 12 | }); 13 | 14 | describe('Creation', function(){ 15 | it('should create dummy data', function(done){ 16 | request(baseURL) 17 | .post('/data') 18 | .set('Content-Type', 'application/json') 19 | .send({"ts":"2017-03-11T15:00:53Z", "type": "temp", "value": 34, "sensor_id": 123 }) 20 | .expect(201) 21 | .end(function(err, res){ 22 | if (err) throw err; 23 | done(); 24 | }) 25 | }); 26 | }); 27 | -------------------------------------------------------------------------------- /step7/nodejs/test/mocha.opts: -------------------------------------------------------------------------------- 1 | --timeout 5s 2 | -------------------------------------------------------------------------------- /step7/simulator.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | while(true); do 4 | # Current date 5 | d=$(date -u +"%Y-%m-%dT%H:%M:%SZ") 6 | 7 | # Random temperature between 20 and 34°C 8 | temp=$(( ( RANDOM % 15 ) + 20 )) 9 | 10 | # Send data to API 11 | curl -XPOST -H "Content-Type: application/json" -d '{"ts":"'$d'", "type": "temp", "value": '$temp', "sensor_id": 123 }' http://localhost:1337/data 12 | 13 | sleep 1 14 | done 15 | -------------------------------------------------------------------------------- /step8/.gitignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | -------------------------------------------------------------------------------- /step8/README.md: -------------------------------------------------------------------------------- 1 | # Objectives 2 | 3 | In this step, we will deploy the application on a remote machine. 4 | 5 | # What we will do 6 | 7 | * Create a Docker host using the Docker Machine tool 8 | * Slightly modify the docker-compose.yml file to use the image from the Docker Hub 9 | * Run the application 10 | 11 | # Creation of a Docker Host 12 | 13 | Let's use Docker Machine, with the driver of your choice, to create a new Docker Host 14 | 15 | For simplicity, we use the *virtualbox* driver, but *aws*, *digitalocean*, ... can of course be used. 16 | 17 | ```` 18 | docker-machine create --driver virtualbox iot 19 | ```` 20 | 21 | Switch to the context of the newly created *iot* machine so the local client will target the daemon running on this host. 22 | 23 | ```` 24 | eval $(docker-machine env iot) 25 | ```` 26 | 27 | # Modification of the docker-compose.yml file 28 | 29 | * db service 30 | 31 | The usage of the local InfluxDB configuration file has been removed as relying on an external file is not portable. We will use the environment variable instead (configuration details can be found [InfluxDB Docker Hub page](https://hub.docker.com/_/influxdb/). 32 | 33 | * api service 34 | 35 | The *image* has been changed so it matches the name of the one that is available on the Docker Hub. Also, the *build* instruction has been removed as we will not build new image in this step but only rely on the image stored on Docker Hub. 36 | 37 | # Run the application 38 | 39 | The application can then be ran with the following command 40 | 41 | ```` 42 | docker-compose up 43 | ```` 44 | 45 | # Create the database 46 | 47 | As we did before, we create the *iot* database through InfluxDB's administration interface. This one is available on port *8083* of the *iot* machine. 48 | 49 | ![Database creation](./images/01-influxdb-db-creation.png) 50 | 51 | # Run the simulator 52 | 53 | We have slightly modified the simulator so that *-h* and *-p* options can be specified to target a specific host:port as by default it only targets the localhost on port 1337. 54 | 55 | To target the *iot* machine we can then run the following command 56 | 57 | ```` 58 | ./simulator.sh -h $(docker-machine ip iot) -p 1337 59 | ```` 60 | 61 | Note: the IP of the machine created is retrieved from the *docker-machine ip* command. 62 | 63 | # Visualisation with Grafana 64 | 65 | The Grafana interface is avialable on the *MACHINE_IP:3000*. After having defined the dashboard as we did in a previous step, we can visualize the data sent by the simulator. 66 | 67 | [Grafana data samples](./images/02-grafana-data-samples) 68 | 69 | # Summary 70 | 71 | We are now able to run the application on any host using the images stored on the Docker Registry. 72 | 73 | 74 | ----- 75 | [< Previous](../step7) - [Next >](../step9) 76 | -------------------------------------------------------------------------------- /step8/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.1" 2 | services: 3 | db: 4 | image: influxdb:1.2.4 5 | environment: 6 | - INFLUXDB_ADMIN_ENABLED=true 7 | ports: 8 | - 8086:8086 9 | - 8083:8083 10 | volumes: 11 | - influxdata:/var/lib/influxdb 12 | api: 13 | image: lucj/iot-api:v3 14 | command: npm start 15 | ports: 16 | - 1337:1337 17 | dashboard: 18 | image: grafana/grafana 19 | ports: 20 | - 3000:3000 21 | volumes: 22 | influxdata: 23 | -------------------------------------------------------------------------------- /step8/images/01-influxdb-db-creation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step8/images/01-influxdb-db-creation.png -------------------------------------------------------------------------------- /step8/images/02-grafana-data-samples.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step8/images/02-grafana-data-samples.png -------------------------------------------------------------------------------- /step8/simulator.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Default HOST:PORT tragetted 4 | HOST="localhost" 5 | PORT=1337 6 | 7 | function usage { 8 | echo "Usage: simulator.sh [-h HOST] [-p PORT]" 9 | exit 1 10 | } 11 | 12 | # Parse arguments 13 | while getopts h:p: FLAG; do 14 | case $FLAG in 15 | h) 16 | HOST=$OPTARG 17 | ;; 18 | p) 19 | PORT=$OPTARG 20 | ;; 21 | \?) 22 | usage 23 | ;; 24 | esac 25 | done 26 | 27 | echo "=> About to send data to $HOST on port $PORT" 28 | echo 29 | 30 | # Generate and send random data 31 | while(true); do 32 | # Current date 33 | d=$(date -u +"%Y-%m-%dT%H:%M:%SZ") 34 | 35 | # Random temperature between 20 and 34°C 36 | temp=$(( ( RANDOM % 15 ) + 20 )) 37 | 38 | # Send data to API 39 | curl -XPOST -H "Content-Type: application/json" -d '{"ts":"'$d'", "type": "temp", "value": '$temp', "sensor_id": 123 }' http://$HOST:$PORT/data 40 | 41 | sleep 1 42 | done 43 | -------------------------------------------------------------------------------- /step9/.gitignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | -------------------------------------------------------------------------------- /step9/README.md: -------------------------------------------------------------------------------- 1 | # Objectives 2 | 3 | In this step, we will deploy the application on Docker Swarm. 4 | 5 | # What we will do 6 | 7 | * Create a 2 nodes Swarm 8 | * Run the Docker Compose application as a Stack 9 | 10 | # Creation of the Docker Swarm 11 | 12 | We will use Docker Machine, with the *virtualbox* driver to create 2 hosts 13 | 14 | Note: feel free to use the driver of your choice (*aws*, *digitalocean*,...). 15 | 16 | ```` 17 | docker-machine create --driver virtualbox node1 18 | docker-machine create --driver virtualbox node2 19 | ```` 20 | 21 | Let's create the Swarm from *node1*, which make this guy the **Leader** of the future managers. 22 | 23 | ```` 24 | $ eval $(docker-machine env node1) 25 | 26 | $ docker swarm init --advertise-addr eth1 27 | Swarm initialized: current node (r6dr61umtoybvjtzk4etw8ffk) is now a manager. 28 | 29 | To add a worker to this swarm, run the following command: 30 | 31 | docker swarm join \ 32 | --token SWMTKN-1-6bkjrg17vocrzi767d1wd6px586dy719vutigejlnt4jqq2hvc-4slm5bpf0cydkwq4up1jggh9n \ 33 | 192.168.99.100:2377 34 | 35 | To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. 36 | ```` 37 | 38 | Following the instruction provided above, we can add *node2* as a worker 39 | 40 | ```` 41 | $ eval $(docker-machine env node2) 42 | 43 | $ docker swarm join \ 44 | --token SWMTKN-1-6bkjrg17vocrzi767d1wd6px586dy719vutigejlnt4jqq2hvc-4slm5bpf0cydkwq4up1jggh9n \ 45 | 192.168.99.100:2377 46 | This node joined a swarm as a worker. 47 | ```` 48 | 49 | # Run the application 50 | 51 | From Docker version 1.13, a Docker Compose application can be ran on a Swarm using the *docker stack deploy* command and specifying the docker-compose file. 52 | Let's deploy our application. 53 | 54 | ```` 55 | $ docker stack deploy -c docker-compose.yml iot 56 | Creating network iot_default 57 | Creating service iot_dashboard 58 | Creating service iot_db 59 | Creating service iot_api 60 | ```` 61 | 62 | The stack named *iot* has been created and contains 3 services 63 | 64 | ```` 65 | $ docker stack ls 66 | NAME SERVICES 67 | iot 3 68 | ```` 69 | 70 | Let's have some additional details on the running services 71 | 72 | ```` 73 | $ docker service ls 74 | ID NAME MODE REPLICAS IMAGE 75 | gjrt9ox21epb iot_db replicated 1/1 influxdb:latest 76 | wgyu3jvec8yc iot_api replicated 1/1 lucj/iot-api:v3 77 | yyzjng3vk8d8 iot_dashboard replicated 1/1 grafana/grafana:latest 78 | ```` 79 | 80 | The services are present, and seem to be running fine, we now check on which node each one is running. 81 | 82 | ```` 83 | $ docker service ps iot_db 84 | ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 85 | d72piw9k0fia iot_db.1 influxdb:latest node2 Running Running 5 minutes ago 86 | 87 | $ docker service ps iot_api 88 | ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 89 | ud2m9j1d3y3n iot_api.1 lucj/iot-api:v3 node1 Running Running 5 minutes ago 90 | 91 | $ docker service ps iot_dashboard 92 | ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 93 | l9j2va01ysuh iot_dashboard.1 grafana/grafana:latest node1 Running Running 5 minutes ago 94 | ```` 95 | 96 | We can see from those outputs that instances (read containers) of both *dashboard* and *api* services are running on *node1*. The instance of *db* is running on node2. 97 | 98 | # Create the database 99 | 100 | The routing mesh feature of Docker Swarm makes each port, published by a service, available on any node. 101 | As the InfluxDB is running on *node2*, let's access the administration interface from *node1* and create the *iot* database. 102 | 103 | ![Database creation](./images/01-influxdb-db-create.png) 104 | 105 | # Run the simulator 106 | 107 | To verify once again the routing mesh feature, let's run the simulator towards *node2* (as the API is running on *node1*). 108 | 109 | ```` 110 | ./simulator.sh -h $(docker-machine ip node2) -p 1337 111 | ```` 112 | 113 | # Visualize the data 114 | 115 | As we did several times, we need to create a datasource and a dashboard form Grafana web interface. 116 | Once this is done, we should see the data sent by the simulator. 117 | 118 | ![Data visualization](./images/02-grafana-data-samples.png) 119 | 120 | 121 | ----- 122 | [< Previous](../step8) 123 | -------------------------------------------------------------------------------- /step9/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.1" 2 | services: 3 | db: 4 | image: influxdb:1.2.4 5 | environment: 6 | - INFLUXDB_ADMIN_ENABLED=true 7 | ports: 8 | - 8086:8086 9 | - 8083:8083 10 | volumes: 11 | - influxdata:/var/lib/influxdb 12 | api: 13 | image: lucj/iot-api:v3 14 | command: npm start 15 | ports: 16 | - 1337:1337 17 | dashboard: 18 | image: grafana/grafana 19 | ports: 20 | - 3000:3000 21 | volumes: 22 | influxdata: 23 | -------------------------------------------------------------------------------- /step9/images/01-influxdb-db-create.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step9/images/01-influxdb-db-create.png -------------------------------------------------------------------------------- /step9/images/02-grafana-data-samples.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lucj/IoT-demo-project/aeeb1e443a1ab0fe0eb6c94ce4525e6ce4cdba30/step9/images/02-grafana-data-samples.png -------------------------------------------------------------------------------- /step9/simulator.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Default HOST:PORT tragetted 4 | HOST="localhost" 5 | PORT=1337 6 | 7 | function usage { 8 | echo "Usage: simulator.sh [-h HOST] [-p PORT]" 9 | exit 1 10 | } 11 | 12 | # Parse arguments 13 | while getopts h:p: FLAG; do 14 | case $FLAG in 15 | h) 16 | HOST=$OPTARG 17 | ;; 18 | p) 19 | PORT=$OPTARG 20 | ;; 21 | \?) 22 | usage 23 | ;; 24 | esac 25 | done 26 | 27 | echo "=> About to send data to $HOST on port $PORT" 28 | echo 29 | 30 | # Generate and send random data 31 | while(true); do 32 | # Current date 33 | d=$(date -u +"%Y-%m-%dT%H:%M:%SZ") 34 | 35 | # Random temperature between 20 and 34°C 36 | temp=$(( ( RANDOM % 15 ) + 20 )) 37 | 38 | # Send data to API 39 | curl -XPOST -H "Content-Type: application/json" -d '{"ts":"'$d'", "type": "temp", "value": '$temp', "sensor_id": 123 }' http://$HOST:$PORT/data 40 | 41 | sleep 1 42 | done 43 | --------------------------------------------------------------------------------