├── SingleNode └── UPI-cgruver.md ├── GCP └── IPI-default.md ├── AWS └── IPI-default.md ├── Azure └── IPI-default.md ├── vSphere ├── IPI-default.md ├── automated-testing-system.md └── UPI-Prerequisites.md ├── README.md ├── Homelab ├── UPI-vrutkovs.md └── UPI-SriRamanujam.md ├── guide-template.md └── LICENSE /SingleNode/UPI-cgruver.md: -------------------------------------------------------------------------------- 1 | # Single Node OKD Installation 2 | 3 | This document outlines how to deploy a single node OKD cluster using virt. 4 | 5 | ## Requirements 6 | 7 | - Host with a minimal CentOS Stream, Fedora, or CentOS-8 installed (*do not create a /home filesystem*) 8 | - Monitor, mouse, and keyboard attached to the host 9 | - Static IP for the host 10 | - The following packages installed: virt, wget, git, net-tools, bind, bind-utils, bash-completion, rsync, libguestfs-tools, virt-install, epel-release, libvirt-devel, httpd-tools, snf, nginx 11 | 12 | ## Procedure 13 | 14 | For the complete procedure, please see [Building an OKD4 single node cluster with minimal resources](https://cgruver.github.io/okd4-single-node-cluster) 15 | -------------------------------------------------------------------------------- /GCP/IPI-default.md: -------------------------------------------------------------------------------- 1 | --- 2 | authors: 3 | - "@elmiko" 4 | last-updated: "2021-03-24" 5 | okd-version: "4.7" 6 | --- 7 | # GCP IPI Default Deployment 8 | 9 | This describes the resources used by OpenShift after perfoming an installation 10 | using the default options for the installer. 11 | 12 | ## Infrastructure 13 | 14 | ### Compute 15 | 16 | * 3 control plane nodes 17 | * instance type `n1-standard-4` 18 | * 3 compute nodes 19 | * instance type `n1-standard-2` 20 | * 1 image 21 | 22 | ### Networking 23 | 24 | * 2 networks 25 | * 2 subnetworks 26 | * 3 static IP addresses 27 | * 1 router 28 | * 2 routes 29 | * 3 target pools 30 | * 10 firewall rules 31 | * 2 forwarding rules 32 | * 3 in-use global IP addresses 33 | * 3 health checks 34 | 35 | ### Platform 36 | 37 | * 5 IAM service accounts 38 | 39 | ## Deployment 40 | 41 | See the [OKD documentation](https://docs.okd.io/latest/installing/installing_gcp/installing-gcp-account.html) 42 | to proceed with deployment. 43 | -------------------------------------------------------------------------------- /AWS/IPI-default.md: -------------------------------------------------------------------------------- 1 | --- 2 | authors: 3 | - "@elmiko" 4 | last-updated: "2021-03-24" 5 | okd-version: "4.7" 6 | --- 7 | # AWS IPI Default Deployment 8 | 9 | This describes the resources used by OpenShift after perfoming an installation 10 | using the default options for the installer. 11 | 12 | ## Infrastructure 13 | 14 | ### Compute 15 | 16 | * 3 control plane nodes 17 | * instance type `m4.xlarge`, or `m5.xlarge` if previous not available in the region 18 | * 3 compute nodes 19 | * instance type `m4.large`, or `m5.large` if previous not available in the region 20 | 21 | ### Networking 22 | 23 | * 1 virtual private cloud 24 | * 1 public subnet per availability zone in the region 25 | * 1 private subnet per availability zone in the region 26 | * 1 NAT gateway per availability zone 27 | * 1 elastic IP address per NAT gateway 28 | * 3 elastic load balancers 29 | * 1 external network load balancer for the master API server 30 | * 1 internal network load balancer for the master API server 31 | * 1 classic load balancer for the router 32 | * 21 elastic network interfaces, plus 1 interface per availability zone 33 | * 1 virtual private cloud gateway 34 | * 10 distinct security groups 35 | 36 | ## Deployment 37 | 38 | See the [OKD documentation](https://docs.okd.io/latest/installing/installing_aws/preparing-to-install-on-aws.html) 39 | to proceed with deployment. 40 | -------------------------------------------------------------------------------- /Azure/IPI-default.md: -------------------------------------------------------------------------------- 1 | --- 2 | authors: 3 | - "@elmiko" 4 | last-updated: "2021-03-24" 5 | okd-version: "4.7" 6 | --- 7 | # Azure IPI Default Deployment 8 | 9 | This describes the resources used by OpenShift after perfoming an installation 10 | using the default options for the installer. 11 | 12 | ## Infrastructure 13 | 14 | ### Compute 15 | 16 | * 3 control plane nodes 17 | * instance type `Standard_D8s_v3` 18 | * 3 compute nodes 19 | * instance type `Standard_D4s_v3` 20 | 21 | ### Networking 22 | 23 | * 1 virtual network (VNet) containing 2 subnets 24 | * 6 network interfaces 25 | * 3 network load balancers 26 | * 1 public for compute node access 27 | * 1 private for control plane access 28 | * 1 public for control plane access 29 | * 2 public IP addresses 30 | * 1 for the public compute load balancer 31 | * 1 for the public control plane load balancer 32 | * 7 private IP addresses 33 | * 1 per control plane node 34 | * 1 per compute node 35 | * 1 for the private control plane load balancer 36 | * 2 network security groups 37 | * 1 for control plane allowing traffic on port 6443 from anywhere 38 | * 1 for compute allowing traffic on ports 80 and 443 from the internet 39 | 40 | ## Deployment 41 | 42 | See the [OKD documentation](https://docs.okd.io/latest/installing/installing_azure/installing-azure-account.html) 43 | to proceed with deployment. 44 | -------------------------------------------------------------------------------- /vSphere/IPI-default.md: -------------------------------------------------------------------------------- 1 | --- 2 | authors: 3 | - "@lobziik" 4 | last-updated: "2021-03-24" 5 | okd-version: "4.7" 6 | --- 7 | # vSphere IPI Deployment 8 | 9 | This describes the resources used by OpenShift after perfoming an installation 10 | using the required options for the installer. 11 | 12 | ## Infrastructure 13 | 14 | ### Compute 15 | All vms stored within folder described above and tagged with tag created by installer. 16 | 17 | * 3 control plane vms (name format: `{cluster name}-{generated cluster id}-master-{0,1,2}`) 18 | * 4 vCPU 19 | * 16 GB RAM 20 | * 120 GB storage 21 | 22 | * 3 worker vms (name format: `{cluster name}-{generated cluster id}-master-{generated worker id}`) 23 | * 2 vCPU 24 | * 8 GB RAM 25 | * 120 GB storage 26 | 27 | ### Networking 28 | 29 | Should be set up by user. Installer doesn't create anything there. Network name should be provided as installer argument. 30 | 31 | ### Miscellaneous 32 | 33 | * tag category with format `openshift-{cluster name}-{generated cluster id}` 34 | * tag with format `{cluster name}-{generated cluster id}` 35 | * folder with title format `{cluster name}-{generated cluster id}` 36 | * disabled virtual machine with name `{cluster name}-rhcos-{generated cluster id}` which using as template for further scaling 37 | 38 | ## Deployment 39 | 40 | See the [OKD documentation](https://docs.okd.io/latest/installing/installing_vsphere/installing-vsphere-installer-provisioned.html) 41 | to proceed with deployment. 42 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # OKD Deployment Configuration Guides 2 | 3 | This repository contains a collection of community curated documents describing known deployment configurations 4 | for [OKD](https://okd.io). The information is categorized by deployment platform/style and installation options. At a minimum, 5 | these guides describe the necessary infrastructure and configuration for an OKD deployment. Where possible, 6 | the guides go into deeper detail about installation and deployment options and proceedures used by the authors. 7 | 8 | These guides serve as a common place for sharing platform configurations, and exposing more of the 9 | [tribal knowledge](https://en.wikipedia.org/wiki/Tribal_knowledge) that has been aquired by 10 | developers and operators working closely with OKD. 11 | 12 | ## Repository structure 13 | 14 | The files in this repository are divided into directories for each specific deployment platform or style. 15 | Within each directory are [Markdown](https://www.markdownguide.org/) files for different deployment 16 | scenarios. For example, most platforms will have a "Default IPI" document which describes the configuration 17 | for a cluster that has been created using the [OpenShift installer](https://github.com/openshift/installer) 18 | with a minimal configuration file. 19 | 20 | ## Contributing to the notes 21 | 22 | Contributions to the notes are welcomed through pull requests from forked versions of this repository. 23 | 24 | When proposing a new guide, please use the [Guide Template](guide-template.md) to create your content. 25 | You should place your content in a platform directory associated with your specific deployment. You 26 | should name your document using the installation method and brief descriptive text of the options 27 | you are detailing. For example, if writing about deploying in a disconnected network scenario, you might 28 | create a file named `UPI-disconnected-network.md`. If you are documenting your personal "home lab" setup, 29 | please use your github username in the filename, for example `UPI-elmiko.md`. 30 | 31 | And if all else fails, please do not hesitate to open your pull request and start a discussions with your 32 | questions and concerns about implementation. 33 | -------------------------------------------------------------------------------- /Homelab/UPI-vrutkovs.md: -------------------------------------------------------------------------------- 1 | # Vadim's homelab 2 | 3 | This describes the resources used by OpenShift after perfoming an installation 4 | to make it similar to my homelab setup. 5 | 6 | ## Compute 7 | 8 | * 1 Ubiquity EdgeRouter ER-X 9 | * runs DHCP (embedded), custom DNS server via AdGuard 10 | * [pic](https://dl.vrutkovs.eu/public/homelab/erx.jpg) 11 | * 1 NAS/Bastion host 12 | * haproxy for loadbalancer 13 | * ceph cluster for PVs 14 | * NFS server for shared data 15 | * [pic](https://dl.vrutkovs.eu/public/homelab/neptr-gumball.jpg) 16 | * 1 control plane 17 | * Intel i5 CPU, 16+4 GB RAM 18 | * 120 GB NVME disk 19 | * [pic](https://dl.vrutkovs.eu/public/homelab/bmo.jpg) 20 | * 1 compute nodes 21 | * Lenovo X220 laptop 22 | * [pic](https://dl.vrutkovs.eu/public/homelab/neptr-gumball.jpg) 23 | 24 | ## Router setup 25 | Once nodes have booted assign static IPs using MAC pinning. 26 | 27 | EdgeRouter has dnsmasq to support custom DNS entries, but I wanted to have a network-wide ad filtering 28 | and DNS-over-TLS for free, so I followed [this guide](https://medium.com/@casonadams/edgerouter-x-adguardhome-b9d453f5725b) 29 | to install [AdGuard Home](https://adguard.com/en/adguard-home/overview.html) on the router. 30 | This gives a fancy UI for DNS rewrites and gives a useful stats about the nodes on the network. 31 | 32 | ## NAS/Bastion setup 33 | HAProxy setup is fairly standard - see [ocp4-helpernode](https://github.com/RedHatOfficial/ocp4-helpernode) for idea. 34 | 35 | Along with (fairly standard) NFS server I also run a single node Ceph cluster, so that I could benefit 36 | from CSI / autoprovision / snapshots etc. 37 | 38 | ## Installation 39 | 40 | Currently "single node install" requires a dedicated throwaway bootstrap node, so I used future compute node (x220 laptop) 41 | as a bootstrap node. Once master was installed, the laptop was re-provisioned to become a compute node. 42 | 43 | ## Upgrading 44 | 45 | Since I use a single master install, upgrades are bit complicated. Both nodes are labelled as workers, so upgrading those is not an issue. 46 | Upgrading single master is tricky, so I use [this script](https://github.com/vrutkovs/okd-installer/blob/master/manifests/singlenode/upgrade-master.sh) to pivot the node into expected master ignition content, which runs `rpm-ostree rebase `. This script needs to be cancelled before it starts installing OS extensions (NetworkManager-ovs etc.) as its necessary. 47 | 48 | This issue as a class would be addressed in 4.8. 49 | 50 | ## Useful software 51 | 52 | [Grafana operator](https://operatorhub.io/operator/grafana-operator) is incredibly useful to setup monitoring. 53 | This operator helps me to define a configuration for various datasources (i.e. [Promtail+Loki](https://grafana.com/oss/loki/)) and control dashboard source code using CRs. 54 | 55 | [SnapScheduler](https://operatorhub.io/operator/snapscheduler) makes periodic snapshots of some PVs so that risky changes could be reverted. 56 | 57 | [Tekton](https://operatorhub.io/operator/tektoncd-operator) operator is helping me to run a few clean up jobs in cluster periodically. 58 | Most useful pipeline I've been using is running `oc adm must-gather` on this cluster, unpacking it and storing it in Git. This helps me keep track of changes in the cluster in a git repo - and, unlike gitops solution like ArgoCD - I can still tinker with things in the console. 59 | 60 | Other useful software running in my cluster: 61 | 62 | * [Gitea](https://gitea.io) - git server 63 | * [HomeAssistant](https://home-assistant.io/) - controls smart home devices 64 | * [BitWarden_rs](https://github.com/dani-garcia/bitwarden_rs) - password storage 65 | * [Minio](https://min.io) - S3-like storage 66 | * [Nextcloud](https://nextcloud.com) - file sync software 67 | * [Navidrome](https://www.navidrome.org) - music server 68 | * [MiniFlux](https://miniflux.app) - RSS reader 69 | * [Matrix Synapse](https://matrix.org) - federated chat app 70 | * [Pleroma](https://pleroma.social) - federated microblogging app 71 | * [Wallabag](https://www.wallabag.it) - Read-It-Later app 72 | -------------------------------------------------------------------------------- /vSphere/automated-testing-system.md: -------------------------------------------------------------------------------- 1 | # Implementing an Automated Installation Solution for OKD on vSphere with User Provisioned Infrastructure (UPI) 2 | 3 | ## Introduction 4 | 5 | It's possible to completely automate the process of installing OpenShift/OKD on vSphere with User Provisioned Infrastructure by chaining together the various functions of [OCT](https://github.com/JaimeMagiera/oct) via a wrapper script. 6 | 7 | ## Steps 8 | 9 | 1. Deploy the DNS, DHCP, and load balancer infrastructure outlined in the [Prerequisites](#Prerequisites) section. 10 | 2. Create an install-config.yaml.template file based on the format outlined in the section [Sample install-config.yaml file for VMware vSphere](https://docs.okd.io/latest/installing/installing_vsphere/installing-vsphere.html#installation-vsphere-config-yaml_installing-vsphere) of the OKD docs. Do not add a pull secret. The script will query you for one or it will insert a default one if you use the --auto-secret flag. 11 | 3. Create a [wrapper script](#Wrapper-Script) that: 12 | * Installs the desired FCOS image 13 | * Downloads the oc and openshift-installer binaries for your desired release version 14 | * Generates and modifies the ignition files appropriately 15 | * Builds the cluster nodes 16 | * Triggers the installation process. 17 | 18 | ## Prerequisites 19 | 20 | ### DNS 21 | 22 | * 1 entry for the bootstrap node of the format bootstrap.[cluster].domain.tld 23 | * 3 entries for the master nodes of the form master-[n].[cluster].domain.tld 24 | * An entry for each of the desired worker nodes in the form worker-[n].[cluster].domain.tld 25 | * 1 entry for the API endpoint in the form api.[cluster].domain.tld 26 | * 1 entry for the API internal endpoint in the form api-int.[cluster].domain.tld 27 | * 1 wilcard entry for the Ingress endpoint in the form \*.apps.[cluster].domain.tld 28 | 29 | ### DHCP 30 | ### Load Balancer 31 | 32 | vSphere UPI requires the use of a load balancer. There needs to be two pools. 33 | 34 | * API: This pool should contain your master nodes. 35 | * Ingress: This pool should contain your worker nodes. 36 | 37 | ### Proxy (Optional) 38 | 39 | If the cluster will sit on a private network, you'll need a proxy for outgoing traffic, both for the install process and for regular operation. In the case of the former, the installer needs to pull containers from the external registires. In the case of the latter, the proxy is needed when applicaton containers need access to the outside world (e.g. yum installs, external code repositories like gitlab, etc.) 40 | 41 | The proxy should be configured to accept conections from the IP subnet for your cluster. A simple proxy to use for this purpose is [squid](http://www.squid-cache.org) 42 | 43 | ## Wrapper Script 44 | 45 | ``` bash 46 | #!/bin/bash 47 | 48 | masters_count=3 49 | workers_count=2 50 | template_url="https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/33.20210314.2.0/x86_64/fedora-coreos-33.20210314.2.0-vmware.x86_64.ova" 51 | template_name="fedora-coreos-33.20210201.2.1-vmware.x86_64" 52 | library="Linux ISOs" 53 | cluster_name="mycluster" 54 | cluster_folder="/MyVSPHERE/vm/Linux/OKD/mycluster" 55 | network_name="VM Network" 56 | install_folder=`pwd` 57 | 58 | # Import the template 59 | ./oct.sh --import-template --library "${library}" --template-url "${template_url}" 60 | 61 | # Install the desired OKD tools 62 | oct.sh --install-tools --release 4.6 63 | 64 | # Launch the prerun to generate and modify the ignition files 65 | oct.sh --prerun --auto-secret 66 | 67 | # Deploy the nodes for the cluster with the appropriate ignition data 68 | oct.sh --build --template-name "${template_name}" --library "${library}" --cluster-name "${cluster_name}" --cluster-folder "${cluster_folder}" --network-name "${network_name}" --installation-folder "${install_folder}" --master-node-count ${masters_count} --worker-node-count ${workers_count} 69 | 70 | # Turn on the cluster nodes 71 | oct.sh --cluster-power on --cluster-name "${cluster_name}" --master-node-count ${masters_count} --worker-node-count ${workers_count} 72 | 73 | # Run the OpenShift installer 74 | bin/openshift-install --dir=$(pwd) wait-for bootstrap-complete --log-level=info 75 | 76 | ``` 77 | 78 | ## Future Updates 79 | * Generating the install-config template 80 | * Pull directly from FCOS release feed 81 | 82 | -------------------------------------------------------------------------------- /guide-template.md: -------------------------------------------------------------------------------- 1 | --- 2 | authors: 3 | - "@elmiko" 4 | last-updated: "2021-03-24" 5 | okd-version: "X.Y" 6 | --- 7 | # OKD Deployment Configuration Guide Template 8 | 9 | ``` 10 | 11 | 12 | Hello! This is the template for creating new OKD deployment configuration guides. 13 | The text in this template provides instructions and suggestions to help you in 14 | creating your guide, it is meant to be replaced as you write. 15 | 16 | First steps: 17 | * Update the header material. The section at the top of this document contains information about 18 | the document authors, the last date of update, and the target okd version for the guide. Please use 19 | your GitHub handle in the author section, and the date formatted as YYYY-MM-DD. 20 | * Change the title. It is useful to specify what distinguishes your guide from others. For example 21 | "IPI AWS Deployment Using Cluster-Wide Proxy", or "Charro's Single Node LibVirt UPI Deployment". 22 | * Follow the sections of this document and replace the content with your guide. 23 | ``` 24 | 25 | ## Audience 26 | 27 | ``` 28 | 29 | 30 | Who is the primary intended audience for this guide? Is it the home hacker, or the engineer interested 31 | in improving their cloud infrastructure, or someone else completely? This section should be used to highlight 32 | what groups might get the most benefit from this guide. 33 | ``` 34 | 35 | ## Prerequisites 36 | 37 | Before beginning this guide, please review these prerequisites: 38 | 39 | ``` 40 | 41 | 42 | Use this section to talk about things that will be needed, or assumed available, for your deployment 43 | configuration. 44 | 45 | * Basic familiarity with Linux command line tools and installation process 46 | * A network with a configurable router for your cluster 47 | * USB thumb drive for boot media 48 | * Access to a DNS server 49 | ``` 50 | 51 | ## Infrastructure 52 | 53 | The following hardware and services were used to construct this OKD cluster. 54 | 55 | ``` 56 | 57 | 58 | This section should describe the hardware configuration and topology you have used for your deployment. 59 | Feel free to create sub-headings for special equipment or configurations you might have used, along 60 | with descriptions of their usage. 61 | ``` 62 | 63 | ### Compute 64 | 65 | ``` 66 | 67 | 68 | * Control plane nodes, 3 x AMD Ryzen 3 3300U, 64 GB RAM, 500 GB SSD 69 | * Worker nodes, 2 x Intel Core i7-6800K, 128 GB RAM, 500 GB SSD 70 | * Big worker node, 1 x AMD Ryzen 3990X, 1 TB RAM, 4 TB SSD 71 | * 1 x NVIDIA V100 72 | * Storage is 3 x 2 TB SSD in RAID configuration 73 | ``` 74 | 75 | ### Networking 76 | 77 | ``` 78 | 79 | 80 | * Gigabit LAN between workstation, bootstrap, master node 81 | * Gigabit LAN between master and worker nodes 82 | * Gigabit LAN between master and router 83 | * the LAN's and workstation's nameservice delegating requests of the wildcard domain to the master nodes 84 | * 2 x GCP Load Balancers 85 | ``` 86 | 87 | ## Deployment 88 | 89 | ``` 90 | 91 | 92 | This section is where you will describe your deployment and how you created and configured it. The 93 | subsections here are given as inspiration for your content, feel free to remove or add where 94 | appropriate. 95 | ``` 96 | 97 | ### Setup 98 | 99 | ``` 100 | 101 | 102 | This section is for in-depth explanations of any setup work that was done before deployment. 103 | 104 | * How did you prepare for your deployment? 105 | * Did you create any extra services that a reader should know about? 106 | * Describe the steps you took to prepare your infrastructure for deployment. 107 | * Use `code blocks` to share relevant scripts and commands that your ran. 108 | ``` 109 | 110 | ### Installation 111 | 112 | ``` 113 | 114 | 115 | This section is where you can talk about how you ran the installation process, and what options you chose along the way. 116 | 117 | * How did you install your cluster? 118 | * Are there any special considerations before attempting a similar process? 119 | * Describe the steps you took to install OKD on your infrastructure. 120 | * Use `code blocks` to share relevant scripts and commands that your ran. 121 | ``` 122 | 123 | ### Upgrade 124 | 125 | ``` 126 | 127 | 128 | If you have performed upgrades on your deployment, this is the section to talk about how you did them. 129 | 130 | * Are upgrade a part of your deployment? 131 | * How do you plan and release them? 132 | * Describe how you have performed upgrades and any extra steps you needed. 133 | ``` 134 | 135 | ## Day 2 Operations 136 | 137 | ``` 138 | 139 | 140 | This section is for detailing things you have done with your cluster after the basic installation. 141 | 142 | * After deployment what have you done to configure your cluster further? 143 | * Are there other services or operators that you install? 144 | * Is there additional automation that you use on cluster? 145 | ``` 146 | 147 | ## Additional Resources 148 | 149 | ``` 150 | 151 | 152 | Use this section to add links to any materials that you referenced or would like to call out specifically. 153 | 154 | You can also use this section to embed videos demonstrating your deployment and its configuration. 155 | ``` 156 | 157 | -------------------------------------------------------------------------------- /Homelab/UPI-SriRamanujam.md: -------------------------------------------------------------------------------- 1 | # Sri's Overkill Homelab Setup 2 | 3 | This document lays out the resources used to create my completely-overkill homelab. This cluster provides all the compute and storage I think I'll need for the foreseeable future, and the CPU, RAM, and storage can all be scaled vertically independently of each other. Not that I think I'll need to do that for a while. 4 | 5 | More detail into the deployment and my homelab's Terraform configuration can be found [here](https://github.com/SriRamanujam/okd-deployment). 6 | 7 | ## Hardware 8 | 9 | * 3 hyper-converged hypervisors 10 | * Ryzen 5 3600 11 | * 64 GiB RAM 12 | * 3x 4TiB HDD 13 | * 2x 500GiB SSD in RAID1 14 | * 1x 256GiB NVME for boot disk 15 | 16 | * 1 NUC I had laying around gathering dust 17 | * Intel Core i3-5010U 18 | * 16 GiB RAM 19 | * 500GiB SSD 20 | 21 | ## Main cluster 22 | 23 | My hypervisors each host an identical workload. The total size of this cluster is 3 control plane nodes, and 9 worker nodes. 24 | So it splits very nicely three ways. Each hypervisor hosts 1 control plane VM and 3 worker VMs. 25 | 26 | * 3 control plane nodes 27 | * 4x CPU 28 | * 10 GiB RAM 29 | * 50 GiB disk 30 | 31 | * 9 worker nodes 32 | * 8x CPU 33 | * 16 GiB RAM 34 | * 50 GiB root disk 35 | * 4 TiB HDD for workload use 36 | 37 | * 1 bootstrap node (temporary, taken down after inital setup is complete) 38 | * 4 vCPU 39 | * 8 GiB RAM 40 | * 120 GiB root disk 41 | 42 | ## Supporting infrastructure 43 | 44 | ### Networking 45 | 46 | OKD, and especially baremetal UPI OKD, requires a very specific network setup. You will most likely need something more 47 | flexible than your ISP's router to get everything fully configured. [The documentation](https://docs.okd.io/latest/installing/installing_bare_metal/installing-bare-metal.html#installation-network-user-infra_installing-bare-metal) is very clear on the various DNS records and DHCP static allocations you will need to make, so I won't go into them here. 48 | 49 | However, there are a couple extra things that you may want to set for best results. In particular, I make sure that I have [PTR records](https://www.cloudflare.com/learning/dns/dns-records/dns-ptr-record/) set up for all my cluster nodes. This is extremely important as the nodes need a correct PTR record set up for them to auto-discover their hostname. Clusters typically do not set themselves up properly if there are hostname collisions! 50 | 51 | 52 | ### API load balancer 53 | 54 | I run a separate smaller VM on the NUC as a single-purpose load balancer appliance, running HAProxy. 55 | 56 | * 1 load balancer VM 57 | * 2x vCPU 58 | * 256MiB RAM 59 | * 10GiB disk 60 | 61 | The HAProxy config is straightforward. I adapted mine from the example config file created by the [ocp4-helpernode](https://github.com/RedHatOfficial/ocp4-helpernode/blob/master/templates/haproxy.cfg.j2) playbook. 62 | 63 | ## Deployment 64 | 65 | I create the VMs on the hypervisors using Terraform. The [Terraform Libvirt provider](https://github.com/dmacvicar/terraform-provider-libvirt) is very, very cool. It's also used by `openshift-install` for its Libvirt-based deployments, so it supports everything needed to deploy OKD nodes. Most importantly, I can use Terraform to supply the VMs with their Ignition configs, which means I don't have to worry about passing kernel args manually or setting up a PXE server to get things going like the official OKD docs would have you do. Terraform also makes it easy to tear down the cluster and reset in case something goes wrong. 66 | 67 | ## Post-Bootstrap One-Time Setup 68 | 69 | ### Storage with Rook and Ceph 70 | 71 | I deploy a Ceph cluster into OKD using Rook. The Rook configuration deploys OSDs on top of the 4TiB HDDs assigned to each worker. I deploy an erasure-coded CephFS pool (6+2) for RWX workloads and a 3x replica block pool for RWO workloads. 72 | 73 | ### Monitoring and Alerting 74 | 75 | OKD comes with a very comprehensive monitoring and alerting suite, and it would be a shame not to take advantage of it. I set up an Alertmanager webhook to send any alerts to a [small program I wrote that posts the alerts to Discord](https://github.com/SriRamanujam/alertmanager-discord-bridge). 76 | 77 | I also deploy a Prometheus + Grafana set up into the cluster that collects metrics from the various hypervisors and supporting infrastructure VMs. I use Grafana's built-in Discord alerting mechanism to post those alerts. 78 | 79 | ### LoadBalancer with MetalLB 80 | 81 | [MetalLB](https://metallb.universe.tf) is a piece of fantastic software that allows on-prem or otherwise non-public-cloud Kubernetes clusters to enjoy the luxury of `LoadBalancer` type services. It's dead simple to set up and makes you feel you're in a real datacenter. I deploy several workloads that don't use standard HTTP and so can't be deployed behind a `Route`. Without MetalLB, I wouldn't be able to deploy these workloads on OKD at all but with it, I can! 82 | 83 | ## Software I Run 84 | 85 | I maintain an ansible playbook that handles deploying my workloads into the cluster. I prefer Ansible over other tools like Helm because it has more robust capabilities to store secrets, I find its templating capabilities more flexible and powerful than Helm's (especially when it comes to inlining config files into config maps or creating templated Dockerfiles for BuildConfigs), and because I am already familiar with Ansible and know how it works. 86 | 87 | * [paperless-ng](https://github.com/jonaswinkler/paperless-ng) - A document organizer that uses machine learning to automatically classify and organize 88 | * [bitwarden_rs](https://github.com/dani-garcia/bitwarden_rs) - Password manager 89 | * [Jellyfin](https://jellyfin.org/) - Media management 90 | * [Samba](https://www.samba.org/) - I joined a StatefulSet to my AD domain and it serves an authenticated SMB share 91 | * [Netbox](https://github.com/netbox-community/netbox) - Infrastructure management tool 92 | * [Quassel](https://quassel-irc.org) - IRC bouncer 93 | * [Ukulele](https://github.com/Frederikam/ukulele) - Bot that plays music into Discord channels 94 | * RPM and deb package repos for internal packages 95 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /vSphere/UPI-Prerequisites.md: -------------------------------------------------------------------------------- 1 | # Prerequites for vSphere UPI 2 | 3 | In this example I describe the setup of a DNS/DHCP server and a Load Balancer on a Raspberry PI microcomputer. The instructions most certainly will also work for other environments. 4 | 5 | I use Raspberry Pi OS (debian based). 6 | 7 | ## IP Addresses of components in this example 8 | * Homelab subnet: **192.168.178.0/24** 9 | * DSL router/gateway: **192.168.178.1** 10 | * IP address of Raspberry Pi (DHCP/DNS/Load Balancer): **192.168.178.5** 11 | * local domain: **homelab.net** 12 | * local cluster (name: c1) domain: **c1.homelab.net** 13 | * DHCP range: 192.168.178.40 ... 192.168.178.199 14 | * Static IPs for OKD's bootstrap, masters and workers 15 | 16 | ## Upgrade Raspberry Pi 17 | 18 | ``` 19 | sudo apt-get update 20 | sudo apt-get upgrade 21 | sudo reboot 22 | ``` 23 | 24 | ## Set static IP address on Raspberry Pi 25 | 26 | Add this: 27 | ``` 28 | interface eth0 29 | static ip_address=192.168.178.5/24 30 | static routers=192.168.178.1 31 | static domain_name_servers=192.168.178.5 8.8.8.8 32 | ``` 33 | to 34 | 35 | /etc/dhcpcd.conf 36 | 37 | ## DHCP 38 | Ensure that no other DHCP servers are activated in the network of your homelab e.g. in your internet router. 39 | 40 | The DHCP server in this example is setup with DDNS (Dynamic DNS) enabled. 41 | 42 | ### Install 43 | 44 | ``` 45 | sudo apt-get install isc-dhcp-server 46 | ``` 47 | 48 | ### Configure 49 | 50 | Enable DHCP server for IPv4 on eth0: 51 | 52 | /etc/default/isc-dhcp-server 53 | ``` 54 | INTERFACESv4="eth0" 55 | INTERFACESv6="" 56 | ``` 57 | 58 | /etc/dhcp/dhcpd.conf 59 | ``` 60 | # dhcpd.conf 61 | # 62 | 63 | #################################################################################### 64 | # Configuration for Dynamic DNS (DDNS) updates # 65 | # Clients requesting an IP and sending their hostname for domain *.homelab.net # 66 | # will be auto registered in the DNS server. # 67 | #################################################################################### 68 | ddns-updates on; 69 | ddns-update-style standard; 70 | 71 | # This option points to the copy rndc.key we created for bind9. 72 | include "/etc/bind/rndc.key"; 73 | 74 | allow unknown-clients; 75 | use-host-decl-names on; 76 | default-lease-time 300; # 5 minutes 77 | max-lease-time 300; # 5 minutes 78 | 79 | # homelab.net DNS zones 80 | zone homelab.net. { 81 | primary 192.168.178.5; # This server is the primary DNS server for the zone 82 | key rndc-key; # Use the key we defined earlier for dynamic updates 83 | } 84 | zone 178.168.192.in-addr.arpa. { 85 | primary 192.168.178.5; # This server is the primary reverse DNS for the zone 86 | key rndc-key; # Use the key we defined earlier for dynamic updates 87 | } 88 | 89 | ddns-domainname "homelab.net."; 90 | ddns-rev-domainname "in-addr.arpa."; 91 | #################################################################################### 92 | 93 | 94 | #################################################################################### 95 | # Basic configuration # 96 | #################################################################################### 97 | # option definitions common to all supported networks... 98 | default-lease-time 300; 99 | max-lease-time 300; 100 | 101 | # If this DHCP server is the official DHCP server for the local 102 | # network, the authoritative directive should be uncommented. 103 | authoritative; 104 | 105 | # Parts of this section will be put in the /etc/resolv.conf of your hosts later 106 | option domain-name "homelab.net"; 107 | option routers 192.168.178.1; 108 | option subnet-mask 255.255.255.0; 109 | option domain-name-servers 192.168.178.5; 110 | 111 | subnet 192.168.178.0 netmask 255.255.255.0 { 112 | range 192.168.178.40 192.168.178.199; 113 | } 114 | #################################################################################### 115 | 116 | 117 | #################################################################################### 118 | # Static IP addresses # 119 | # (Replace the MAC addresses here with the ones you set in vsphere for your vms) # 120 | #################################################################################### 121 | group { 122 | host bootstrap { 123 | hardware ethernet 00:1c:00:00:00:00; 124 | fixed-address 192.168.178.200; 125 | } 126 | 127 | host master0 { 128 | hardware ethernet 00:1c:00:00:00:10; 129 | fixed-address 192.168.178.210; 130 | } 131 | 132 | host master1 { 133 | hardware ethernet 00:1c:00:00:00:11; 134 | fixed-address 192.168.178.211; 135 | } 136 | 137 | host master2 { 138 | hardware ethernet 00:1c:00:00:00:12; 139 | fixed-address 192.168.178.212; 140 | } 141 | 142 | host worker0 { 143 | hardware ethernet 00:1c:00:00:00:20; 144 | fixed-address 192.168.178.220; 145 | } 146 | 147 | host worker1 { 148 | hardware ethernet 00:1c:00:00:00:21; 149 | fixed-address 192.168.178.221; 150 | } 151 | 152 | host worker2 { 153 | hardware ethernet 00:1c:00:00:00:22; 154 | fixed-address 192.168.178.222; 155 | } 156 | } 157 | ``` 158 | 159 | ## DNS 160 | 161 | ### Install 162 | 163 | ``` 164 | sudo apt install bind9 dnsutils 165 | ``` 166 | 167 | ### Basic configuration 168 | 169 | /etc/bind/named.conf.options 170 | ``` 171 | include "/etc/bind/rndc.key"; 172 | 173 | acl internals { 174 | // lo adapter 175 | 127.0.0.1; 176 | 177 | // CIDR for your homelab network 178 | 192.168.178.0/24; 179 | }; 180 | 181 | options { 182 | directory "/var/cache/bind"; 183 | 184 | // If there is a firewall between you and nameservers you want 185 | // to talk to, you may need to fix the firewall to allow multiple 186 | // ports to talk. See http://www.kb.cert.org/vuls/id/800113 187 | 188 | // If your ISP provided one or more IP addresses for stable 189 | // nameservers, you probably want to use them as forwarders. 190 | // Uncomment the following block, and insert the addresses replacing 191 | // the all-0's placeholder. 192 | 193 | forwarders { 194 | 8.8.8.8; 195 | 8.8.4.4; 196 | }; 197 | forward only; 198 | 199 | //======================================================================== 200 | // If BIND logs error messages about the root key being expired, 201 | // you will need to update your keys. See https://www.isc.org/bind-keys 202 | //======================================================================== 203 | dnssec-validation no; 204 | 205 | listen-on-v6 { none; }; 206 | auth-nxdomain no; 207 | listen-on port 53 { any; }; 208 | 209 | // Allow queries from my Homelab and also from Wireguard Clients. 210 | allow-query { internals; }; 211 | allow-query-cache { internals; }; 212 | allow-update { internals; }; 213 | recursion yes; 214 | allow-recursion { internals; }; 215 | allow-transfer { internals; }; 216 | 217 | dnssec-enable no; 218 | 219 | check-names master ignore; 220 | check-names slave ignore; 221 | check-names response ignore; 222 | }; 223 | ``` 224 | 225 | /etc/bind/named.conf.local 226 | ``` 227 | #include "/etc/bind/rndc.key"; 228 | 229 | // 230 | // Do any local configuration here 231 | // 232 | 233 | // Consider adding the 1918 zones here, if they are not used in your 234 | // organization 235 | //include "/etc/bind/zones.rfc1918"; 236 | 237 | # All devices that don't belong to the OKD cluster will be maintained here. 238 | zone "homelab.net" { 239 | type master; 240 | file "/etc/bind/forward.homelab.net"; 241 | allow-update { key rndc-key; }; 242 | }; 243 | 244 | zone "c1.homelab.net" { 245 | type master; 246 | file "/etc/bind/forward.c1.homelab.net"; 247 | allow-update { key rndc-key; }; 248 | }; 249 | 250 | zone "178.168.192.in-addr.arpa" { 251 | type master; 252 | notify no; 253 | file "/etc/bind/178.168.192.in-addr.arpa"; 254 | allow-update { key rndc-key; }; 255 | }; 256 | ``` 257 | 258 | Zone file for **homlab.net**: 259 | /etc/bind/forward.homelab.net 260 | ``` 261 | ; 262 | ; BIND data file for local loopback interface 263 | ; 264 | $TTL 604800 265 | @ IN SOA homelab.net. root.homelab.net. ( 266 | 2 ; Serial 267 | 604800 ; Refresh 268 | 86400 ; Retry 269 | 2419200 ; Expire 270 | 604800 ) ; Negative Cache TTL 271 | ; 272 | @ IN NS homelab.net. 273 | @ IN A 192.168.178.5 274 | @ IN AAAA ::1 275 | ``` 276 | 277 | 278 | The name of the next file depends on the subnet that is used: 279 | 280 | /etc/bind/178.168.192.in-addr.arpa 281 | ``` 282 | $TTL 1W 283 | @ IN SOA ns1.homelab.net. root.homelab.net. ( 284 | 2019070742 ; serial 285 | 10800 ; refresh (3 hours) 286 | 1800 ; retry (30 minutes) 287 | 1209600 ; expire (2 weeks) 288 | 604800 ; minimum (1 week) 289 | ) 290 | NS ns1.homelab.net. 291 | 292 | 200 PTR bootstrap.c1.homelab.net. 293 | 294 | 210 PTR master0.c1.homelab.net. 295 | 211 PTR master1.c1.homelab.net. 296 | 212 PTR master2.c1.homelab.net. 297 | 298 | 220 PTR worker0.c1.homelab.net. 299 | 221 PTR worker1.c1.homelab.net. 300 | 222 PTR worker2.c1.homelab.net. 301 | 302 | 5 PTR api.c1.homelab.net. 303 | 5 PTR api-int.c1.homelab.net. 304 | ``` 305 | 306 | ### DNS records for OKD 4 307 | 308 | Zone file for **c1.homelab.net** (our OKD 4 cluster will be in this domain): 309 | 310 | /etc/bind/forward.c1.homelab.net 311 | ``` 312 | ; 313 | ; BIND data file for local loopback interface 314 | ; 315 | $TTL 604800 316 | @ IN SOA c1.homelab.net. root.c1.homelab.net. ( 317 | 2 ; Serial 318 | 604800 ; Refresh 319 | 86400 ; Retry 320 | 2419200 ; Expire 321 | 604800 ) ; Negative Cache TTL 322 | ; 323 | @ IN NS c1.homelab.net. 324 | @ IN A 192.168.178.5 325 | @ IN AAAA ::1 326 | 327 | load-balancer IN A 192.168.178.5 328 | 329 | bootstrap IN A 192.168.178.200 330 | 331 | master0 IN A 192.168.178.210 332 | master1 IN A 192.168.178.211 333 | master2 IN A 192.168.178.212 334 | 335 | worker0 IN A 192.168.178.220 336 | worker1 IN A 192.168.178.221 337 | worker2 IN A 192.168.178.222 338 | worker3 IN A 192.168.178.223 339 | 340 | *.apps.c1.homelab.net. IN CNAME load-balancer.c1.homelab.net. 341 | api-int.c1.homelab.net. IN CNAME load-balancer.c1.homelab.net. 342 | api.c1.homelab.net. IN CNAME load-balancer.c1.homelab.net. 343 | ``` 344 | 345 | ## Set file permissions 346 | 347 | For dynamic DNS (ddns) to work you should do this: 348 | ``` 349 | sudo chown -R bind:bind /etc/bind 350 | ``` 351 | 352 | ## Load Balancer 353 | 354 | ### Install 355 | 356 | ``` 357 | sudo apt-get install haproxy 358 | ``` 359 | 360 | ### Configure 361 | 362 | /etc/haproxy/haproxy.cfg 363 | ``` 364 | global 365 | log /dev/log local0 366 | log /dev/log local1 notice 367 | chroot /var/lib/haproxy 368 | stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners 369 | stats timeout 30s 370 | user haproxy 371 | group haproxy 372 | daemon 373 | 374 | # Default SSL material locations 375 | ca-base /etc/ssl/certs 376 | crt-base /etc/ssl/private 377 | 378 | # Default ciphers to use on SSL-enabled listening sockets. 379 | # For more information, see ciphers(1SSL). This list is from: 380 | # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ 381 | # An alternative list with additional directives can be obtained from 382 | # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy 383 | ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS 384 | ssl-default-bind-options no-sslv3 385 | 386 | defaults 387 | log global 388 | mode http 389 | option httplog 390 | option dontlognull 391 | timeout connect 20000 392 | timeout client 10000 393 | timeout server 10000 394 | errorfile 400 /etc/haproxy/errors/400.http 395 | errorfile 403 /etc/haproxy/errors/403.http 396 | errorfile 408 /etc/haproxy/errors/408.http 397 | errorfile 500 /etc/haproxy/errors/500.http 398 | errorfile 502 /etc/haproxy/errors/502.http 399 | errorfile 503 /etc/haproxy/errors/503.http 400 | errorfile 504 /etc/haproxy/errors/504.http 401 | 402 | 403 | # You can see the stats and observe OKD's bootstrap process by opening 404 | # http://:4321/haproxy?stats 405 | listen stats 406 | bind :4321 407 | mode http 408 | log global 409 | maxconn 10 410 | 411 | timeout client 100s 412 | timeout server 100s 413 | timeout connect 100s 414 | timeout queue 100s 415 | 416 | stats enable 417 | stats hide-version 418 | stats refresh 30s 419 | stats show-node 420 | stats auth admin:password 421 | stats uri /haproxy?stats 422 | 423 | 424 | frontend openshift-api-server 425 | bind *:6443 426 | default_backend openshift-api-server 427 | mode tcp 428 | option tcplog 429 | 430 | backend openshift-api-server 431 | balance source 432 | mode tcp 433 | server bootstrap bootstrap.c1.homelab.net:6443 check 434 | server master0 master0.c1.homelab.net:6443 check 435 | server master1 master1.c1.homelab.net:6443 check 436 | server master2 master2.c1.homelab.net:6443 check 437 | 438 | 439 | frontend machine-config-server 440 | bind *:22623 441 | default_backend machine-config-server 442 | mode tcp 443 | option tcplog 444 | 445 | backend machine-config-server 446 | balance source 447 | mode tcp 448 | server bootstrap bootstrap.c1.homelab.net:22623 check 449 | server master0 master0.c1.homelab.net:22623 check 450 | server master1 master1.c1.homelab.net:22623 check 451 | server master2 master2.c1.homelab.net:22623 check 452 | 453 | 454 | frontend ingress-http 455 | bind *:80 456 | default_backend ingress-http 457 | mode tcp 458 | option tcplog 459 | 460 | backend ingress-http 461 | balance source 462 | mode tcp 463 | server master0 master0.c1.homelab.net:80 check 464 | server master1 master1.c1.homelab.net:80 check 465 | server master2 master2.c1.homelab.net:80 check 466 | 467 | server worker0 worker0.c1.homelab.net:80 check 468 | server worker1 worker1.c1.homelab.net:80 check 469 | server worker2 worker2.c1.homelab.net:80 check 470 | server worker3 worker3.c1.homelab.net:80 check 471 | 472 | 473 | frontend ingress-https 474 | bind *:443 475 | default_backend ingress-https 476 | mode tcp 477 | option tcplog 478 | 479 | backend ingress-https 480 | balance source 481 | mode tcp 482 | 483 | server master0 master0.c1.homelab.net:443 check 484 | server master1 master1.c1.homelab.net:443 check 485 | server master2 master2.c1.homelab.net:443 check 486 | 487 | server worker0 worker0.c1.homelab.net:443 check 488 | server worker1 worker1.c1.homelab.net:443 check 489 | server worker2 worker2.c1.homelab.net:443 check 490 | server worker3 worker3.c1.homelab.net:443 check 491 | ``` 492 | 493 | ## Reboot and check status 494 | 495 | Reboot Raspberry Pi: 496 | ``` 497 | sudo reboot 498 | ``` 499 | 500 | Check status of DNS/DHCP server and Load Balancer: 501 | ``` 502 | sudo systemctl status haproxy.service 503 | sudo systemctl status isc-dhcp-server.service 504 | sudo systemctl status bind9 505 | ``` 506 | 507 | ## Proxy (if on a private network) 508 | 509 | If the cluster will sit on a private network, you'll need a proxy for outgoing traffic, both for the install process and for regular operation. In the case of the former, the installer needs to pull containers from the external registires. In the case of the latter, the proxy is needed when applicaton containers need access to the outside world (e.g. yum installs, external code repositories like gitlab, etc.) 510 | 511 | The proxy should be configured to accept conections from the IP subnet for your cluster. A simple proxy to use for this purpose is [squid](http://www.squid-cache.org) 512 | --------------------------------------------------------------------------------