├── .gitmodules ├── LICENSE ├── README.md └── docker-compose.yml /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "arango"] 2 | path = arango 3 | url = https://github.com/abdes/submodule-docker-dev-workflow-arango.git 4 | [submodule "server"] 5 | path = server 6 | url = https://github.com/abdes/submodule-docker-dev-workflow-server.git 7 | [submodule "web"] 8 | path = web 9 | url = https://github.com/abdes/submodule-docker-dev-workflow-web.git 10 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Abdessattar Sassi 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Development workflow using _git submodules_ and _docker_ 2 | 3 | This setup aims at simplifying the development workflow and increasing its 4 | efficiency no matter whether the developer is (remote, together with the dev 5 | team), and the experience she/he has with the system(familiar with the full 6 | system or just contributing to a specific part). 7 | 8 | ## Table of Content 9 | 10 | 11 | - [Table of Content](#table-of-content) 12 | - [Credits](#credits) 13 | - [Prerequisites](#prerequisites) 14 | - [Git configuration](#git-configuration) 15 | - [Background and Assumptions](#background-and-assumptions) 16 | - [Creating the git repo for the development environment](#creating-the-git-repo-for-the-development-environment) 17 | - [Adding submodules](#adding-submodules) 18 | - [Removing a submodule](#removing-a-submodule) 19 | - [Temporarily removing a submodule](#temporarily-removing-a-submodule) 20 | - [Permanently removing a submodule](#permanently-removing-a-submodule) 21 | - [Working with container and submodules](#working-with-container-and-submodules) 22 | - [Setting up the development environment as a developer](#setting-up-the-development-environment-as-a-developer) 23 | - [Working independently on a module](#working-independently-on-a-module) 24 | - [Getting an update from the submodule’s remote](#getting-an-update-from-the-submodules-remote) 25 | - [Making updates to the container](#making-updates-to-the-container) 26 | - [Grabbing container updates](#grabbing-container-updates) 27 | - [Updating a submodule in-place in the container](#updating-a-submodule-in-place-in-the-container) 28 | - [Dockerizing everything](#dockerizing-everything) 29 | - [Quick start](#quick-start) 30 | - [How it's done](#how-its-done) 31 | - [arangodb](#arangodb) 32 | - [server](#server) 33 | - [web](#web) 34 | - [nginx - reverse proxy and router](#nginx-reverse-proxy-and-router) 35 | - [Conclusion](#conclusion) 36 | 37 | 38 | 39 | ## Credits 40 | 41 | * The original idea by Amine Mouafik [Efficient development workflow using Git submodules and Docker Compose](https://www.airpair.com/docker/posts/efficiant-development-workfow-using-git-submodules-and-docker-compose). 42 | * The excellent [git documentation](https://git-scm.com/doc) 43 | * [Mastering Git submodules](https://medium.com/@porteneuve/mastering-git-submodules-34c65e940407#.s51gcqy2p) 44 | by Christophe Porteneuve 45 | 46 | ## Prerequisites 47 | 48 | ```shell 49 | $ git -v 50 | $ docker -v 51 | $ docker-compose -v 52 | ``` 53 | Detailed instructions for installing and configuring the above tools are 54 | available on the internet: 55 | * git: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git 56 | * docker: https://docs.docker.com/engine/installation/ 57 | * docker-compose: https://docs.docker.com/compose/install/ 58 | 59 | Docker containers are an important part of this development stack and its 60 | associated workflow. Docker tools must be properly installed and configured 61 | on the devloper workstation. 62 | 63 | > On Mac OS X, it is not recommended to use the Docker Toolbox due to several 64 | > issues with volumes through the virtualbox VM. The newer Docker for Mac, 65 | > which comes with the lightweight [xhyve](https://github.com/mist64/xhyve/) 66 | > virtualization layer, should be used instead. 67 | 68 | Git is the chosen SCM for this development environment/workflow and git 69 | submodules are an important part of it. The same principles and methodology 70 | can be used with other SCM tools as long as they can provide a similar 71 | mechanism than git submodules to integrate source for multiple repositories 72 | into the file tree of a master project. 73 | 74 | > Git submodules are tricky to use and may seem a bit daunting at first but 75 | > they have been chosen for this workflow over other alternatives for the 76 | > following reasons: 77 | > * built into git with no need for any extra tools 78 | > * flexibility to work with each module as a separate repository or to 79 | > work from within the enclosing project. 80 | > 81 | > Alternative to git submodules include 82 | [git subtree](https://github.com/git/git/blob/master/contrib/subtree/git-subtree.txt) 83 | and [git-subrepo _(3rd party tool)_](https://github.com/ingydotnet/git-subrepo). 84 | 85 | ### Git configuration 86 | To make the work with git submodules more reliable, it is strongly recommended 87 | to have the following git configuration setup: 88 | ```shell 89 | $ git config --global diff.submodule log 90 | ``` 91 | Equivalent to `git diff --submodules=log`. 92 | 93 | ```shell 94 | git config --global status.submoduleSummary true 95 | ``` 96 | Extend the status information to add the commits in the included submodules. 97 | 98 | ```shell 99 | git config --global alias.spush 'push --recurse-submodules=on-demand' 100 | ``` 101 | An alias that can be used when pushing commits to the remote repos to make sure 102 | that all submodules commits are pushed when pushing the main project. 103 | > Failure to do so will create issues when developers need to update their 104 | > environment with missing commits from the submodules and errors when trying 105 | > to update them. 106 | 107 | ## Background and Assumptions 108 | 109 | The example application we will be working on is composed of the following modules: 110 | * arango: the backend database, using [ArangoDB](https://www.arangodb.com) 111 | * server: server example application, using [node.js](https://nodejs.org/en/), for the 112 | implementation of the REST API 113 | * web: the public web site for the example application, served through 114 | [nginx](https://www.nginx.com) 115 | 116 | The domain name 'example.local' will be used locally for the development environment 117 | and two host entries need to be configured in **/etc/hosts** for the _server_ and 118 | _web_ modules as following: 119 | 120 | ``` 121 | 127.0.0.1 api.example.lo 122 | 127.0.0.1 www.example.lo 123 | ``` 124 | To test that the host names are properly setup, simply use ping as following: 125 | ```shell 126 | $ ping api.example.lo 127 | PING api.example.lo (127.0.0.1): 56 data bytes 128 | 64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.044 ms 129 | 130 | --- api.example.lo ping statistics --- 131 | 1 packets transmitted, 1 packets received, 0.0% packet loss 132 | round-trip min/avg/max/stddev = 0.044/0.044/0.044/0.000 ms 133 | ``` 134 | ```shell 135 | $ ping www.example.lo 136 | PING www.example.lo (127.0.0.1): 56 data bytes 137 | 64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.045 ms 138 | 139 | --- www.example.lo ping statistics --- 140 | 1 packets transmitted, 1 packets received, 0.0% packet loss 141 | round-trip min/avg/max/stddev = 0.045/0.045/0.045/0.000 ms 142 | $ 143 | ``` 144 | 145 | We assume that git is running on the localhost as a server, and that the 146 | following repositories already exist: 147 | * /var/git/example/arango.git, for the arango module 148 | * /var/git/example/server.git, for the server module 149 | * /var/git/example/web.git, for the web module 150 | 151 | ## Creating the git repo for the development environment 152 | 153 | > These steps are only required when setting up the development workflow fo the 154 | > first time. Once it's done, developers being onboarded on this workflow only 155 | > need to start with 'Setting up the development environment' 156 | 157 | On the git server (localhost), login as the git user and create a new bare 158 | repository with the name 'development-local.git' 159 | ```shell 160 | $ git init --bare /var/git/example/development-local.git 161 | Initialized empty Git repository in /var/git/example/development-local.git/ 162 | ``` 163 | We will work on that repo to finish its setup from a regular user on her local 164 | environment. 165 | 166 | We'll clone the development-local git repo, add the various application modules 167 | to it, populate it with the docker files we need to configure and manage the 168 | lifecycle of the different docker containers used for the testing. 169 | 170 | ```shell 171 | $ git clone git@localhost:/var/git/example/development-local.git development-local 172 | Cloning into 'development-local'... 173 | warning: You appear to have cloned an empty repository. 174 | Checking connectivity... done. 175 | $ cd development-local 176 | $ touch README.md 177 | $ git add README.md 178 | $ git commit -m "Empty README.md" 179 | $ git push 180 | ``` 181 | We have just added an empty README.md file to the development-local repo and 182 | pushed the commit to the remote. The development repo should be populated with 183 | at least the basic workflow documentation. You can use this README.md file and 184 | adapt as necessary for your own needs. 185 | 186 | ## Adding submodules 187 | 188 | Let's add the submodules... 189 | ```shell 190 | $ git submodule add git@localhost:/var/git/example/arango.git 191 | $ git submodule add git@localhost:/var/git/example/server.git 192 | $ git submodule add git@localhost:/var/git/example/web.git 193 | ``` 194 | This will have created a .gitmodules in the development-local repository so 195 | that future developers will be able to fetch the submodules while fetching the 196 | development repo. 197 | 198 | By default, submodules will add the subproject into a directory named the same 199 | as the repository, in this case “DbConnector”. You can add a different path at 200 | the end of the command if you want it to go elsewhere. 201 | 202 | Finally we can commit the changes and push to the remote (either using the 203 | previously defined alias `git spush` or simply `git push` as we did not make 204 | any changes to the submodules after adding them). 205 | 206 | ```shell 207 | $ git commit -m "Added submodules" 208 | $ git spush 209 | ``` 210 | 211 | ## Removing a submodule 212 | There are two situations where you’d want to “remove” a submodule: 213 | * You just want to clear the working directory (perhaps before archiving the 214 | container’s WD) but want to retain the possibility of restoring it later (so 215 | it has to remain in .gitmodules and .git/modules); 216 | * You wish to definitively remove it from the current branch. 217 | 218 | Let’s see each case in turn. 219 | 220 | ### Temporarily removing a submodule 221 | The first situation is easily handled by git submodule deinit. 222 | ```shell 223 | $ git submodule deinit web 224 | Cleared directory 'web' 225 | Submodule 'web' (git@localhost:/var/git/example/web.git) unregistered for path 'web' 226 | ``` 227 | This has no impact on the container status whatsoever. The submodule is not 228 | locally known anymore (it’s gone from .git/config), so its absence from the 229 | working directory goes unnoticed. We still have the vendor/plugins/demo directory 230 | but it’s empty; we could strip it with no consequence. 231 | 232 | The submodule must not have any local modifications when you do this, otherwise 233 | you’d need to --force the call. 234 | 235 | Any later subcommand of git submodule will blissfully ignore this submodule until 236 | you init it again, as the submodule won’t even be in local config. Such commands 237 | include update, foreach and sync. 238 | 239 | On the other hand, the submodule remains defined in .gitmodules: an init followed 240 | by an update (or a single update --init) will restore it as new: 241 | 242 | ```shell 243 | $ git submodule update --init web 244 | Submodule 'web' (git@localhost:/var/git/example/web.git) registered for path 'web' 245 | Submodule path 'web': checked out 'ab2e7566ad972378dd51e98dff74250a4ff58b28' 246 | ``` 247 | 248 | ### Permanently removing a submodule 249 | This means you want to get rid of the submodule for good: a regular git rm will 250 | do, just like for any other part of the working directory. This will only work 251 | if your submodule uses a gitfile (a .git which is a file, not a directory), 252 | which is the case starting with Git 1.7.8. 253 | 254 | In addition to stripping the submodule from the working directory, the command 255 | will update the .gitmodules file so it does not reference the submodule anymore. 256 | 257 | For a comprehensive cleanup, it is recommended to do both in sequence, first the 258 | `deinit` then the `rm` so as not to end up with the module still in the local 259 | config. 260 | 261 | ```shell 262 | $ git submodule deinit web 263 | $ git rm web 264 | $ git commit -m "Removed web module" 265 | $ git push 266 | ``` 267 | Regardless of the approach, the submodule’s repo remains present in 268 | .git/modules/web. If you need to add the module back again, you need to kill 269 | that before. 270 | 271 | ## Working with container and submodules 272 | 273 | ### Setting up the development environment as a developer 274 | 275 | To get quickly setup as a developer, you simply need to do the following 276 | instructions (after ensuring the prerequisites and additional configuration for 277 | git is done): 278 | 279 | ```shell 280 | $ git clone --recursive git@localhost:/var/git/example/development-local.git development-local 281 | ``` 282 | This will have the effect of cloning the development-local repo, initializing 283 | (`git submodule init`) and updating (`git submodule update`) the enclosed 284 | submodules, all in one go. 285 | 286 | ``` 287 | development-local 288 | ├── README.md 289 | ├── arango 290 | │   └── README.md 291 | ├── server 292 | │   └── README.md 293 | └── web 294 | └── README.md 295 | ``` 296 | 297 | Since Git 1.7.8, submodules use a simple .git file with a single gitdir: line 298 | mentioning a relative path to the actual repo folder, now located inside the 299 | container’s .git/modules. This is mostly useful when the container has branches 300 | that don’t have the submodule at all: this avoid having to scrap the submodule’s 301 | repo when switching to such a container branch. 302 | ```shell 303 | $ cd web 304 | $ ls -la 305 | total 8 306 | drwxr-xr-x 4 abdessattar staff 136 Jul 16 14:47 . 307 | drwxr-xr-x 8 abdessattar staff 272 Jul 16 14:47 .. 308 | -rw-r--r-- 1 abdessattar staff 84 Jul 16 14:47 .git 309 | -rw-r--r-- 1 abdessattar staff 0 Jul 16 14:47 README.md 310 | $ cat .git 311 | gitdir: /Users/abdessattar/Documents/example-dev/development-local/.git/modules/web 312 | ``` 313 | 314 | Be that as it may, the container and the submodule truly act as independent 315 | repos: they each have their own history (log), status, diff, etc. Therefore be 316 | mindful of your current directory when reading your prompt or typing commands: 317 | depending on whether you’re inside the submodule or outside of it, the context 318 | and impact of your commands differ drastically! 319 | 320 | > The container and the submodule truly act as independent repos. 321 | 322 | Finally, the submodule commit referenced by the container is stored using its 323 | SHA1, not a volatile reference (such as a branch name). Because of this, a 324 | submodule does not automatically upgrade which is a blessing in disguise when 325 | it comes to reliability, maintenance and QA. 326 | 327 | Because of this, most of the time a submodule is in detached head state inside 328 | its containers, as it’s updated by checking out a SHA1 (regardless of whether 329 | that commit is the branch tip at that time). 330 | ```shell 331 | $ cd web 332 | $ git status 333 | HEAD detached at ab2e756 334 | nothing to commit, working directory clean 335 | ``` 336 | 337 | ### Working independently on a module 338 | 339 | It is perfectly fine to work on a module independently of the development setup. 340 | We will see later how we can grab the updated made to a submodule from its 341 | remote and integrate them into the development container. 342 | 343 | Let's assume a developer needs to add a node.js basic application scaffolding 344 | into the server submodule. The updated will be done independently on the server 345 | repo. 346 | 347 | ```shell 348 | $ git clone git@localhost:/var/git/example/server.git server 349 | ``` 350 | 351 | Add an `index.js` file: 352 | ```js 353 | // index.js 354 | var express = require('express'); 355 | var app = express(); 356 | 357 | app.get('/hello', function (req, res) { 358 | res.send('Hello World from node.js!'); 359 | }); 360 | 361 | app.listen(3000, function () { 362 | console.log('Example app listening on port 3000!'); 363 | }); 364 | ``` 365 | 366 | Add a `package.json` file: 367 | ```json 368 | # package.json 369 | { 370 | "name": "example", 371 | "version": "1.0.0", 372 | "description": "Example application server", 373 | "main": "index.js", 374 | "scripts": { 375 | "start": "node index.js" 376 | }, 377 | "repository": { 378 | "type": "git", 379 | "url": "git@localhost:/var/git/example/server.git" 380 | }, 381 | "dependencies": { 382 | "express": "^4.14.0" 383 | } 384 | } 385 | ``` 386 | 387 | Add `.gitignore` file: 388 | ``` 389 | node_modules 390 | ``` 391 | 392 | Commit updates and push to remote: 393 | ```shell 394 | $ git add -all 395 | $ git commit -m "Added simple hello world node/express app" 396 | $ git push 397 | 398 | -> The rsulting history tree would like like this: 399 | 400 | (10 seconds ago) * 03d50da (HEAD -> master, origin/master, origin/HEAD) 401 | | 402 | (3 hours ago) * ab2e756 403 | ``` 404 | 405 | ### Getting an update from the submodule’s remote 406 | 407 | Let's assume now that a second developer is setting up a full development 408 | environment will all submodules. 409 | 410 | ```shell 411 | $ git clone --recursive git@localhost:/var/git/example/development-local.git development-local 412 | 413 | $ cd development-local 414 | $ git status 415 | On branch master 416 | Your branch is up-to-date with 'origin/master'. 417 | nothing to commit, working directory clean 418 | 419 | $ git submodule status 420 | 3a16594c9272c2c521801bac93d3ac958deca336 arango (heads/master) 421 | ab2e7566ad972378dd51e98dff74250a4ff58b28 server (ab2e756) 422 | ab2e7566ad972378dd51e98dff74250a4ff58b28 web (heads/master) 423 | 424 | $ cd server 425 | $ git status 426 | HEAD detached at ab2e756 427 | nothing to commit, working directory clean 428 | ``` 429 | 430 | Notice that the server module is in a detached HEAD mode and no longer tracking 431 | the remote master because it has been modified on the remote and the container 432 | only includes the submodule through a specific commit SHA1 (ab2e756). 433 | 434 | Suppose we want to grab the latest commits made on the remote/master in our 435 | local submodule. 436 | 437 | > On a side note, I would not recommend using pull for this kind of update. 438 | > To properly get the updates in the working directory, this command requires 439 | > that you’re on the proper active branch, which you usually aren’t (you’re on 440 | > a detached head most of the time). You’d have to start with a checkout of 441 | > that branch. But more importantly, the remote branch could very well have 442 | > moved further ahead since the commit you want to set on, and a pull would 443 | > inject commits you may not want in your local codebase. 444 | 445 | Therefore, I recommend splitting the process manually: first git fetch to get 446 | all new data from the remote in local cache, then log to verify what you have 447 | and checkout on the desired SHA1. In addition to finer-grained control, this 448 | approach has the added benefit of working regardless of your current state 449 | (active branch or detached head). 450 | 451 | ```shell 452 | $ git fetch 453 | 454 | $ git log --oneline origin/master 455 | 03d50da Added simple hello world node/express app 456 | ab2e756 Empty README.md 457 | 458 | $ git checkout 03d50da 459 | Previous HEAD position was ab2e756... Empty README.md 460 | HEAD is now at 03d50da... Added simple hello world node/express app 461 | 462 | $ ls -1 463 | README.md 464 | index.js 465 | package.json 466 | ``` 467 | As a result of the above, the parent container is modified and we can see the 468 | detailed status as below: 469 | ```shell 470 | $ cd - 471 | $ git status 472 | On branch master 473 | Your branch is up-to-date with 'origin/master'. 474 | Changes not staged for commit: 475 | (use "git add ..." to update what will be committed) 476 | (use "git checkout -- ..." to discard changes in working directory) 477 | 478 | modified: server (new commits) 479 | 480 | Submodules changed but not updated: 481 | 482 | * server ab2e756...03d50da (1): 483 | > Added simple hello world node/express app 484 | 485 | no changes added to commit (use "git add" and/or "git commit -a") 486 | ``` 487 | 488 | We now only need to perform the container commit that finalizes our submodule’s 489 | update. 490 | 491 | > If you had to touch the container’s code to make it work with this update, 492 | > commit it along, naturally. On the other hand, avoid mixing submodule-related 493 | > changes and other stuff that would just pertain to the container code: by 494 | > neatly separating the two, later migrations to other code-reuse approaches 495 | > are made easier (also, as usual, atomic commits FTW). 496 | 497 | ```shell 498 | $ git commit -am "Moved server submodule to 03d50da" 499 | $ git push 500 | ``` 501 | 502 | ### Making updates to the container 503 | 504 | Making changes and committing/pushing them within the container and not the 505 | submodules is starightforward. It is however recommended to keep such updates 506 | separate from submodule version changes and updates for the sake of better 507 | code management. 508 | 509 | Let's update the container with an enhanced REDME.md file that contains detailed 510 | instructions and push the update to the remote. 511 | 512 | ```shell 513 | $ vim README.md 514 | $ git status 515 | On branch master 516 | Your branch is up-to-date with 'origin/master'. 517 | Changes not staged for commit: 518 | (use "git add ..." to update what will be committed) 519 | (use "git checkout -- ..." to discard changes in working directory) 520 | 521 | modified: README.md 522 | 523 | no changes added to commit (use "git add" and/or "git commit -a") 524 | 525 | $ git commit -am "Updated README.md" 526 | $ git push 527 | ``` 528 | 529 | ### Grabbing container updates 530 | 531 | When pulling updates from the conatiner repo's remote, **Git auto-fetches, but 532 | does not auto-update**. The local cache is up-to-date with the submodule’s 533 | remote, but the submodule’s working directory stays with its former contents. 534 | Although this auto-fetching is limited to already-known submodules: any new 535 | ones, not yet copied into local configuration, are not auto-fetched. 536 | 537 | > If you don’t explicitly update the submodule’s working directory, your next 538 | > container commit will regress the submodule. Is is therefore mandatory that 539 | > you finalize the update. 540 | 541 | ```shell 542 | $ git pull 543 | $ git submodule sync --recursive 544 | $ git submodule update --init --recursive 545 | ``` 546 | 547 | ### Updating a submodule in-place in the container 548 | 549 | This scenario should be limited to the cases where a submodule cannot be 550 | compiled or tested outside the container. Otherwise it is preferred to modify 551 | submodules within their independent repos and pull the updates in the container 552 | afterwards. 553 | 554 | Let's update the web submodule to add a simple index.html page. 555 | 556 | First, we need to make sure we are starting from a clean start, which means the 557 | tip of a branch. Let's assume we can add on top of the current submodule master 558 | branch without breaking the rest of the container and its submodules (if the 559 | risk of breaking the container happens often, it probably means that embedding 560 | this code as a submodule was not a good design architectural choice, and it maybe 561 | more appropriate to just embed its code inline in the container or somewhere else). 562 | 563 | ```shell 564 | $ cd web 565 | $ git checkout master 566 | $ git pull --rebase 567 | ``` 568 | 569 | We can now edit the code, test it etc. then perform the **two commits and the 570 | two necessary pushes** to get the updates to the remote so other developers 571 | can see them them and grab them. 572 | ```shell 573 | $ git add index.html 574 | $ git status -s 575 | A index.html 576 | ``` 577 | 578 | Commit the submodule updates: 579 | ```shell 580 | $ git commit -am "Added simple index.html" 581 | ``` 582 | 583 | Commit the container to move the submodule to the new version: 584 | ```shell 585 | $ cd .. 586 | $ git commit -am "Moving submodule web to 7ff5ca7" 587 | ``` 588 | 589 | Finally, push the container and the submodule either in one go with 590 | `git push --recurse-submodules=on-demand` or by doing two `push` operations, 591 | one in the submodule and one in the container. 592 | ```shell 593 | $ git push --recurse-submodules=on-demand 594 | ``` 595 | 596 | ## Dockerizing everything 597 | 598 | ### Quick start 599 | 600 | All modules are dockerized. To get started with the testing, clone the 601 | development repo as described in '[Setting up the development environment as a developer](#setting-up-the-development-environment-as-a-developer)' 602 | and do this from the development repo root: 603 | 604 | ```shell 605 | $ cd server && npm install && npm run build && cd .. 606 | $ cd arango && npm install && cd .. 607 | $ docker-compose -p example up 608 | ``` 609 | 610 | To tear down the docker containers, simply do this from the development 611 | repo root: 612 | ```shell 613 | $ docker-compose -p example down 614 | ``` 615 | 616 | > It is important to use `docker-compose -p ` to make sure 617 | > docker-compose does not prepend the current directory name to all containers 618 | > it creates. We want those container names to be predictable as they are used 619 | > in the tools within the submodules. 620 | 621 | ### How it's done 622 | 623 | Docker containers enable us to automate the deployment of applications or 624 | application modules inside lightweight containers. No more spending hours or 625 | days making sure all dependencies with correct versions are downloaded and 626 | installed. No more problems due to environment pollution on the test systems. 627 | Simply pull the right container, deploy it and run it. 628 | 629 | With this docker-compose comes to play as a simple but straightforward 630 | tool to define and run multi-container applications, providing basic 631 | orchestration capabilities. 632 | 633 | There are certainly many more sophisticated tools and platforms, such as Apache 634 | Mesos, Kubernetes, etc... to manage complex environments and orchestrate large 635 | clusters of multi-container applications. But we are here for the main purpose 636 | of efficiently setting up a development environment and supporting our 637 | multi-module development workflow with the minimal upfront requirements. 638 | 639 | As we will see, docker and docker-compose will allow us to build everything we 640 | need and get supported our workflow with simply a docker-compose descriptor 641 | file. That in itself is amazing! 642 | 643 | Let's recall the overall application architecture: 644 | * We mainly provide a set of REST APIs served out of our node.js/express 645 | application server, located in the `server` module. 646 | * Our `server` modules relies on a backend database, implemented using ArangoDB. 647 | The choice of ArangoDB is motivated by its versatility to support document 648 | storage and graph databases. Any other alternate database that satisfies the 649 | need may be used, and a docker container would most likely be available of 650 | docker hub. The scripts and optional Foxx microservices files for ArangoDB are 651 | hosted in our `arango` module. 652 | * We also provide a nice web site for our app, served by nginx our of the `web` 653 | module. The web site being fully static or using dynamic pages would not 654 | impact at all our deployment architecture. We simply would need to augment 655 | the container with the required additional capabilities. 656 | * Finally, to protect our application modules and enable us to leverage all the 657 | goodies that can come with `nginx`, we front all our external interfaces with 658 | a reverse proxy. There is no special module in our application repo dor that. 659 | The whole thing can be implemented within the nginx container definition. 660 | Should we need to add some artificats in the future to support enhancements 661 | to our nginx reverse proxy, such as certificates and keys for TLS, we simply 662 | need to create a new application module and add it to our development repo. 663 | 664 | The required containers and their resources are summarized in the table below: 665 | 666 | | Container | Main function | Baseline Docker Container | Volumes | Ports | 667 | |-----------|------------------------------------------------|--------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|-----------| 668 | | arangodb | backend document and graph database | Official container image from docker hub arangodb/arangodb | data: docker volume apps: optional Foxx microservices source code, embedded in arango submodule | 8529:8529 | 669 | | server | REST API and business logic application server | Official stable release of node.js from docker hub node:argon | app: mapped from the server submodule | 3000:3000 | 670 | | web | public web site for the application | Alpine nginx container image from docker hub evild/alpine-nginx | site: mapped from the web submodule config: mapped from the web submodule | 8000:80 | 671 | | nginx | reverse proxy | The amazing auto-configured reverse proxy container based on nginx from docker hub jwilder/nginx-proxy | internal mapping needed for the docker.sock, not related to our application modules | 80:80 | 672 | 673 | > NOTE: docker-compose.yml uses version 2 configuration format to get access to 674 | > the volume declarations and to get a cleaner overall descriptor. 675 | 676 | #### arangodb 677 | 678 | A local volume created and managed by docker is used for the database storage, 679 | while we use our arango submodule to provide ArangoDB with the application 680 | source code for Foxx microservices. 681 | 682 | ```yml 683 | arangodb: 684 | image: arangodb/arangodb 685 | environment: 686 | - ARANGO_NO_AUTH=1 687 | ports: 688 | - "8529:8529" 689 | volumes: 690 | - arangodb-data:/var/lib/arangodb3/ 691 | - arangodb-apps:/var/lib/arangodb3-apps/ 692 | - ./arango/foxx:/var/lib/example-apps/:ro 693 | - ./arango/scripts/:/var/lib/arangodb3-scripts/:ro 694 | ``` 695 | 696 | #### server 697 | 698 | We need to link this container to the `arangodb` container for our application 699 | server to be able to communicate with the database via the exposed ports. 700 | 701 | ```yml 702 | server: 703 | image: node:argon 704 | volumes: 705 | - ./server/:/server/ 706 | working_dir: /server/ 707 | command: npm start 708 | environment: 709 | - VIRTUAL_HOST=api.example.lo 710 | - VIRTUAL_PORT=3000 711 | ports: 712 | - "3000:3000" 713 | links: 714 | - arangodb 715 | ``` 716 | 717 | #### web 718 | 719 | We don't want to override the entire nginx configuration filesystem in the 720 | container, so we only mount volumes for the `nginx.conf` file and for the 721 | `conf.d/` sub-folder. 722 | 723 | ```yml 724 | web: 725 | image: evild/alpine-nginx 726 | volumes: 727 | - ./web/public/:/usr/share/nginx/html/:ro 728 | - ./web/nginx/nginx.conf:/etc/nginx/nginx.conf:ro 729 | - ./web/nginx/conf.d/:/etc/nginx/conf.d/:ro 730 | environment: 731 | - VIRTUAL_HOST=www.example.lo 732 | - VIRTUAL_PORT=80 733 | ports: 734 | - "8000:80" 735 | ``` 736 | 737 | #### nginx - reverse proxy and router 738 | 739 | J. Wilder's nginx reverse proxy container provides an easy and straightforward 740 | method to automatically and dynamically configure all our containers for 741 | reverse proxying by nginx. To mark a container for configuration inside nginx, 742 | you simply add the environment variables `VIRTUAL_HOST` and `VIRTUAL_PORT` to 743 | its container definition. 744 | 745 | For our application, we have the following needs: 746 | * Everything coming to `app.example.lo` should be served by the node.js/express 747 | inside the `server` container. 748 | * Everything coming to `www.example.lo` should be served by the nginx web server 749 | inside the `web` container. 750 | 751 | > The virtual hosts are locally defined in the machine's /etc/hosts file for 752 | > the purpose of our development workflow. In production they can be replaced 753 | > with the real hosts/domain. 754 | 755 | J. Wilder's automated proxy container can do a lot more and can be extensiely 756 | customized to support complex configurations of nginx, TLS, multiple virtual 757 | hosts, etc... For reference, the full documentation is available at 758 | https://github.com/jwilder/nginx-proxy. 759 | 760 | ```yml 761 | nginx: 762 | image: jwilder/nginx-proxy 763 | volumes: 764 | - /var/run/docker.sock:/tmp/docker.sock 765 | ports: 766 | - "80:80" 767 | ``` 768 | 769 | ## Conclusion 770 | 771 | With a fully working developer workflow, supported by git and docker, we can 772 | get a full developer team or a single developer started in a matter of minutes 773 | rather than days. Moreover, with the flexibility that is offered by docker 774 | and the power of git submodules, we can easily adapt the application sturctures 775 | to add more modules. 776 | 777 | Microservices architectures, coupled with containers and container composition 778 | are powerful software architecture tools to build powerful systems while 779 | significantly reducing the complexity overhead on the development workflows. 780 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '2' 2 | 3 | services: 4 | arangodb: 5 | image: arangodb/arangodb 6 | environment: 7 | - ARANGO_NO_AUTH=1 8 | ports: 9 | - "8529:8529" 10 | volumes: 11 | - arangodb-data:/var/lib/arangodb3/ 12 | - ./arango/apps/:/var/lib/arangodb3-apps/ 13 | 14 | server: 15 | image: node:argon 16 | volumes: 17 | - ./server/:/server/ 18 | working_dir: /server/ 19 | command: npm start 20 | environment: 21 | - VIRTUAL_HOST=api.example.lo 22 | - VIRTUAL_PORT=3000 23 | ports: 24 | - "3000:3000" 25 | links: 26 | - arangodb 27 | 28 | web: 29 | image: evild/alpine-nginx 30 | volumes: 31 | - ./web/public/:/usr/share/nginx/html/:ro 32 | - ./web/nginx/nginx.conf:/etc/nginx/nginx.conf:ro 33 | - ./web/nginx/conf.d/:/etc/nginx/conf.d/:ro 34 | environment: 35 | - VIRTUAL_HOST=www.example.lo 36 | - VIRTUAL_PORT=80 37 | ports: 38 | - "8000:80" 39 | 40 | nginx: 41 | image: jwilder/nginx-proxy 42 | volumes: 43 | - /var/run/docker.sock:/tmp/docker.sock 44 | ports: 45 | - "80:80" 46 | 47 | volumes: 48 | arangodb-data: 49 | driver: local 50 | --------------------------------------------------------------------------------