├── CHANGELOG.md ├── LICENSE ├── README.md ├── bin ├── minipot-init └── minipot-start ├── etc ├── consul.d │ └── minipot-agent.json ├── cron.d │ └── pot-prune ├── minipot-traefik.toml.sample ├── newsyslog.conf.d │ ├── consul.conf │ ├── nomad.conf │ └── traefik.conf ├── nomad │ └── minipot-server.hcl.sample └── syslog.d │ ├── consul.conf │ └── nomad.conf └── share └── examples └── minipot ├── nginx-full.job └── nginx.job /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | All notable changes to this project will be documented in this file. 3 | 4 | The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) 5 | and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). 6 | 7 | ## [0.2.4] 2024-12-03 8 | ### Changed 9 | nomad: Change settings to adapt to modified nomad rc script 10 | 11 | ## [0.2.3] 2021-01-28 12 | ### Changed 13 | traefix: update the sample to work with traefik 2.x, now the standard version in the portstree 14 | 15 | ## [0.2.2] 2020-01-31 16 | cron: fix a nasty bug in the PATH, causing pot to be destroyed, even if running 17 | 18 | ## [0.2.1] 2019-12-17 19 | ### Changed 20 | examples: nginx.job is now really minial, while nginx-full.job is a bit more complete 21 | 22 | ## [0.2.0] 2019-12-17 23 | ### Added 24 | nomad: add the automatic configuration of the client's network interface 25 | 26 | ## [0.1.0] 2019-12-10 27 | ### Added 28 | README: quickstart guide 29 | pot: add initialization steps 30 | traefik: proxy configuration file 31 | newsyslogd: log rotation for traefik 32 | cron: pot prune cron job (every 15 minutes) 33 | minipot-start.sh: easy script that start all the needed services 34 | 35 | ### Changed 36 | minipot scripts: remove .sh suffix 37 | consul: agent bind to 0.0.0.0 38 | nomad: small naming improvements 39 | minipot-init.sh: renamed from bootstrap.sh 40 | Provide the IP is not mandatory anymore 41 | 42 | ## [0.0.1] 2019-11-13 43 | ### Added 44 | consul: agent configuration file 45 | nomad: server configuration file 46 | syslogd: configuration files 47 | newsyslogd: log rotation configuration files 48 | bootstrap.sh: script with additional conifguration and preparation 49 | example: add a job example 50 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2017, Luca Pizzamiglio 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | * Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # THIS REPOSITORY IS NO LONGER ACTIVELY MAINTAINED, IT HAS MOVED TO https://codeberg.org/bsdpot/minipot 2 | # minipot 3 | 4 | This project is a collection of configuration files that allows to emulate a nomad cluster in one server/VM 5 | It's like minikube, but it' for FreeBSD, and based on pot and nomad. 6 | 7 | ## A service mesh based on FreeBSD 8 | 9 | minipot will install everything you need to run a single-node service mesh. 10 | This **NOT** meant to be use for production, but to easily have all the service already configured on one node and to play with it. 11 | 12 | ### The components 13 | 14 | minipot is based on pot and nomad: 15 | * pot is a jail framework, that allows you to create and import jail images. 16 | * nomad is a container orchestrator 17 | Additionally, there are other applications: 18 | * consul: a service discovery application, it's needed to work with nomad. In consul, you can see all the services running in your mesh, where they are running and their health status 19 | * traefik: the http proxy/loadbalancer. traefik read the service catalag provided by consul and make all the services avaialable. 20 | 21 | ## How to install minipot 22 | 23 | The easiest way is to install the package: 24 | ```console 25 | pkg install minipot 26 | ``` 27 | The package will install all the needed software and the configuration files. 28 | `pot` has to be configured, before to run the minipot initialization. To be more precise, its configuration file (`/usr/local/etc/pot/pot.conf`) needs your attention. If you have trouble to configure it, please refer to the [`pot` installation guide](https://github.com/pizzamig/pot/blob/master/share/doc/pot/Installation.md) 29 | 30 | If you are already using nomad, traefik or consul, their configuration files will be copied using the suffix `.bkp`. 31 | 32 | Once you are ready, the init script will conclude the installation process: 33 | ```console 34 | minipot-init 35 | ``` 36 | This script will modify your `/etc/rc.conf` to add the last pieces of configuration 37 | 38 | **NOTE** If your machine has 2 routable IP addresses, then you have to specify which address should be used to make your services available, for instance: 39 | ```console 40 | minipot-init -i 192.168.0.1 41 | ``` 42 | 43 | In order to show resource usage to nomad, `pot` needs to activate the resource limits framework, via a loader tunable. You can check if it's active via the command: 44 | ```console 45 | sysctl kern.racct.enable 46 | ``` 47 | If the output is `0`, then you need to modify the `/boot/loader.conf` file and reboot. 48 | If the output is `1`, you are ready to go. 49 | 50 | ## How to start minipot 51 | If you had to reboot your system, you can skip to the next section. 52 | If you didn't reboot your system, you can use an additional script, to start everything you need: 53 | ```console 54 | minipot-start 55 | ``` 56 | 57 | This script will: 58 | * restart syslogd (to manage the new log files) 59 | * restart crond (to manage new cron entries) 60 | * start consul 61 | * start nomad 62 | * start traefik 63 | 64 | **NOTE** If you have issue with consul, try to invoke it as 65 | ``` 66 | consul agent 67 | ``` 68 | and read the output. If the deamon fails to start, no output will be written to the log. 69 | ## Run the example 70 | In the minipot examples folder (`/usr/local/share/examples/minipot`), you can find a simple example you can use to deploy a nginx instance on your minipot. 71 | 72 | ```console 73 | cd /usr/local/share/examples/minipot 74 | nomad run nginx.job 75 | ``` 76 | 77 | ## A bit of diagnostic 78 | 79 | At port 8500, you can reach the consul web user interface. 80 | At port 4646, you can reach the nomad web user interface. 81 | At port 9002, you can reach the traefik web use interface. 82 | 83 | In consul or in nomad you can see at which address/port your nginx instance is running. 84 | The service is named `hello-web`. 85 | The nomad job is called `nginx-minipot`. 86 | 87 | `traefik` is listening on port 8080 to route http traffic to service registered in consul, depending on the host header. 88 | You can use the following command line to reach your nginx instance via traefik: 89 | ``` console 90 | curl -H 'host: hello-web.minipot' 127.0.0.1:8080 91 | ``` 92 | 93 | ### Log files 94 | 95 | Every component has its own log file: 96 | * `/var/log/consul/consul.log` is the consul log file (uses syslogd) 97 | * `/var/log/nomad/nomad.log` is the nomad log file (uses syslogd) 98 | * `/var/log/traefik.log` is the traefik error log file 99 | * `/var/log/traefik-access.log` is the traefik access log file 100 | 101 | minipot is already configured to do a log file rotation. 102 | 103 | ## Troubleshooting 104 | 105 | If you shutdown the machine without stopping the nomad jobs, it can happen that nomad will have some difficulties restarting. 106 | If nomad won't restart, you can delete its temporary database removing the folder `/var/tmp/nomad` 107 | 108 | -------------------------------------------------------------------------------- /bin/minipot-init: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | print_syntax () 4 | { 5 | echo "Syntax: $0 [-i IPADDRESS]" 6 | echo "Please use the IP address of the machine where minipot is supposed to run" 7 | echo "The -i option is mandatory if the your machine has more than one routable IP address" 8 | } 9 | 10 | # $1 the IP address 11 | _inject_ip() 12 | { 13 | local ip="$1" 14 | # configure consul 15 | sysrc consul_args="-advertise $ip" 16 | } 17 | 18 | # $1 the IP address 19 | _find_netif() 20 | { 21 | local ifs ip 22 | ip="$1" 23 | ifs="$(ifconfig | grep '^[a-z]' | cut -f 1 -d ":" | tr '\n' ' ')" 24 | for netif in $ifs ; do 25 | if ifconfig "$netif" | grep -q -F " $ip " ; then 26 | echo $netif 27 | return 28 | fi 29 | done 30 | } 31 | 32 | while getopts :i:h arg; do 33 | case $arg in 34 | h) 35 | print_syntax 36 | exit 0 37 | ;; 38 | i) 39 | ip="$OPTARG" 40 | netif="$( _find_netif "$ip" )" 41 | if [ -z "$netif" ]; then 42 | echo "$ip is not bound to any interface - aborting" 43 | exit 1 44 | fi 45 | ;; 46 | ?) 47 | print_syntax 48 | exit 1 49 | ;; 50 | esac 51 | done 52 | 53 | if ! grep -c "^kern.racct.enable=1" /boot/loader.conf ; then 54 | echo kern.racct.enable=1 >> /boot/loader.conf 55 | fi 56 | 57 | if [ ! -e /etc/pf.conf.potbkp ] && [ -e /etc/pf.conf ]; then 58 | cp /etc/pf.conf /etc/pf.conf.potbkp 59 | fi 60 | pot init 61 | 62 | sysrc nomad_user="root" 63 | sysrc nomad_dir="/var/tmp/nomad" 64 | sysrc nomad_env="PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/sbin:/bin" 65 | sysrc nomad_args="-config=/usr/local/etc/nomad/minipot-server.hcl -data-dir=/var/tmp/nomad" 66 | [ -n "$netif" ] && sysrc nomad_args+=" -network-interface=$netif" 67 | sysrc consul_enable="YES" 68 | sysrc nomad_enable="YES" 69 | sysrc traefik_enable="YES" 70 | sysrc traefik_conf="/usr/local/etc/minipot-traefik.toml" 71 | 72 | mkdir -p /var/log/consul 73 | mkdir -p /var/log/nomad 74 | touch /var/log/consul/consul.log 75 | touch /var/log/nomad/nomad.log 76 | touch /var/log/traefik.log 77 | touch /var/log/traefik-access.log 78 | chown traefik:traefik /var/log/traefik.log 79 | chown traefik:traefik /var/log/traefik-access.log 80 | 81 | if [ -n "$ip" ]; then 82 | _inject_ip "$ip" 83 | fi 84 | 85 | -------------------------------------------------------------------------------- /bin/minipot-start: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | service syslogd restart 4 | service cron restart 5 | 6 | service consul start 7 | sleep 5 8 | service nomad start 9 | service traefik start 10 | -------------------------------------------------------------------------------- /etc/consul.d/minipot-agent.json: -------------------------------------------------------------------------------- 1 | { 2 | "bind_addr": "0.0.0.0", 3 | "client_addr": "0.0.0.0", 4 | "datacenter": "mini-dc", 5 | "dns_config": { 6 | "a_record_limit": 3, 7 | "enable_truncate": true 8 | }, 9 | "enable_syslog": true, 10 | "leave_on_terminate": true, 11 | "log_level": "WARN", 12 | "node_name": "mini-pot", 13 | "translate_wan_addrs": true, 14 | "ui": true, 15 | "server": true, 16 | "bootstrap_expect": 1 17 | } 18 | -------------------------------------------------------------------------------- /etc/cron.d/pot-prune: -------------------------------------------------------------------------------- 1 | SHELL=/bin/sh 2 | PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin 3 | 4 | */15 * * * * root /usr/local/bin/pot prune 5 | -------------------------------------------------------------------------------- /etc/minipot-traefik.toml.sample: -------------------------------------------------------------------------------- 1 | [entryPoints] 2 | [entryPoints.http] 3 | address = "0.0.0.0:8080" 4 | [entryPoints.traefik] 5 | address = "0.0.0.0:9002" 6 | 7 | [log] 8 | filePath = "/var/log/traefik.log" 9 | format = "common" 10 | level="INFO" 11 | 12 | [accessLog] 13 | filePath = "/var/log/traefik-access.log" 14 | format = "common" 15 | 16 | [api] 17 | insecure = true 18 | dashboard = true 19 | 20 | [providers] 21 | [providers.consulCatalog] 22 | exposedByDefault = true 23 | defaultRule = "Host(`{{ .Name }}.minipot`)" 24 | stale = false 25 | [providers.consulCatalog.endpoint] 26 | address = "http://127.0.0.1:8500" 27 | -------------------------------------------------------------------------------- /etc/newsyslog.conf.d/consul.conf: -------------------------------------------------------------------------------- 1 | /var/log/consul/consul.log 640 10 10000 @T CpX 2 | -------------------------------------------------------------------------------- /etc/newsyslog.conf.d/nomad.conf: -------------------------------------------------------------------------------- 1 | /var/log/nomad/nomad.log 640 10 10000 @T CpX 2 | -------------------------------------------------------------------------------- /etc/newsyslog.conf.d/traefik.conf: -------------------------------------------------------------------------------- 1 | /var/log/traefik.log traefik:traefik 640 10 10000 @T CpX /var/run/traefik.pid SIGUSR1 2 | /var/log/traefik-access.log traefik:traefik 640 10 10000 @T CpX /var/run/traefik.pid SIGUSR1 3 | -------------------------------------------------------------------------------- /etc/nomad/minipot-server.hcl.sample: -------------------------------------------------------------------------------- 1 | bind_addr = "0.0.0.0" 2 | plugin_dir = "/usr/local/libexec/nomad/plugins" 3 | plugin "nomad-pot-driver" {} 4 | datacenter = "minipot" 5 | 6 | advertise { 7 | # This should be the IP of THIS MACHINE and must be routable by every node 8 | # in your cluster 9 | http = "127.0.0.1:4646" 10 | } 11 | 12 | server { 13 | enabled = true 14 | bootstrap_expect = 1 15 | } 16 | 17 | client { 18 | enabled = true 19 | options { 20 | "driver.raw_exec.enable" = "1" 21 | } 22 | } 23 | 24 | consul { 25 | # The address to the Consul agent. 26 | address = "127.0.0.1:8500" 27 | 28 | # The service name to register the server and client with Consul. 29 | server_service_name = "minipot-server" 30 | client_service_name = "minipot-client" 31 | 32 | # Enables automatically registering the services. 33 | auto_advertise = true 34 | 35 | # Enabling the server and client to bootstrap using Consul. 36 | server_auto_join = false 37 | client_auto_join = true 38 | } 39 | 40 | enable_syslog=true 41 | log_level="INFO" 42 | syslog_facility="LOCAL1" 43 | 44 | -------------------------------------------------------------------------------- /etc/syslog.d/consul.conf: -------------------------------------------------------------------------------- 1 | local0.* /var/log/consul/consul.log 2 | 3 | -------------------------------------------------------------------------------- /etc/syslog.d/nomad.conf: -------------------------------------------------------------------------------- 1 | local1.* /var/log/nomad/nomad.log 2 | 3 | -------------------------------------------------------------------------------- /share/examples/minipot/nginx-full.job: -------------------------------------------------------------------------------- 1 | job "nginx-minipot" { 2 | datacenters = ["minipot"] 3 | type = "service" 4 | 5 | group "group1" { 6 | count = 1 7 | 8 | update { 9 | max_parallel = 1 10 | min_healthy_time = "10s" 11 | healthy_deadline = "8m" 12 | auto_revert = true 13 | } 14 | 15 | task "www1" { 16 | driver = "pot" 17 | 18 | service { 19 | tags = ["nginx", "www"] 20 | name = "hello-web" 21 | port = "http" 22 | 23 | check { 24 | type = "tcp" 25 | name = "tcp" 26 | interval = "5s" 27 | timeout = "2s" 28 | } 29 | } 30 | 31 | config { 32 | image = "https://pot-registry.zapto.org/registry/" 33 | pot = "FBSD120-nginx" 34 | tag = "1.2" 35 | command = "nginx" 36 | args = [ 37 | " -g 'daemon off;'" 38 | ] 39 | 40 | port_map = { 41 | http = "80" 42 | } 43 | } 44 | 45 | resources { 46 | cpu = 200 47 | memory = 64 48 | 49 | network { 50 | mbits = 10 51 | port "http" {} 52 | } 53 | } 54 | } 55 | } 56 | } 57 | -------------------------------------------------------------------------------- /share/examples/minipot/nginx.job: -------------------------------------------------------------------------------- 1 | job "nginx-minipot" { 2 | datacenters = ["minipot"] 3 | type = "service" 4 | 5 | group "group1" { 6 | count = 1 7 | 8 | task "www1" { 9 | driver = "pot" 10 | 11 | service { 12 | tags = ["nginx", "www"] 13 | name = "hello-web" 14 | port = "http" 15 | 16 | check { 17 | type = "tcp" 18 | name = "tcp" 19 | interval = "5s" 20 | timeout = "2s" 21 | } 22 | } 23 | 24 | config { 25 | image = "https://pot-registry.zapto.org/registry/" 26 | pot = "FBSD120-nginx" 27 | tag = "1.2" 28 | 29 | port_map = { 30 | http = "80" 31 | } 32 | } 33 | 34 | resources { 35 | cpu = 200 36 | memory = 64 37 | 38 | network { 39 | mbits = 10 40 | port "http" {} 41 | } 42 | } 43 | } 44 | } 45 | } 46 | --------------------------------------------------------------------------------