├── set-env ├── static ├── OCP_4x_bootstrap.png ├── pxe-boot-master.png ├── pxe-boot-worker.png ├── pxe-write-image.png ├── bootstrap-progress.png ├── pxe-boot-bootstrap.png └── pxe-boot-local-disk.png ├── utils ├── htpasswd.sample ├── DNS │ ├── 10-static-dns.conf │ ├── 10-populate-resolv.conf │ ├── named-reverse-zone.conf │ ├── named-zone.conf │ └── dnsmasq.conf ├── PXE │ ├── pxelinux.cfg-localboot │ ├── pxelinux-cfg-bootstrap │ ├── dnsmasq-pxe.conf │ ├── dnsmasq-ipxe-matchbox.conf │ ├── openstack-pxe-boot.md │ ├── sample.ipxe │ ├── pxelinux-cfg-default-uefi │ ├── pxelinux.cfg-default-bios │ └── pxelinux.cfg-default-4.6 ├── BareMetal │ ├── ipmi-shutdown.sh │ └── ocp4.3-metal3-config.yaml ├── NFS │ ├── nfs-export.txt │ └── nfs-pv-image-registry.yaml ├── nm-patch.json ├── mcp │ ├── mcp-worker-bm.yaml │ ├── mcp-worker-bm-mc.yaml │ └── rhel-rt-mc-and-mcp.md ├── Ignition │ ├── worker-ignition.json │ ├── ignition-template_ign.json │ ├── static-name-dns-ignition.json │ └── static-ip-ignition.json ├── scripts │ ├── encode_kubeadmin_password.py │ └── patch-systemd-units.py ├── OSP │ ├── troubleshooting-cloud-credentials.md │ ├── ocp-osp-static-ip.md │ └── security-group.md ├── Registry │ ├── registry_config.yml │ ├── local_registries.conf │ ├── local-registry-docker-client.md │ ├── local-registry-dedicated-disk.md │ ├── registries.conf │ ├── local-registry.md │ ├── local-registry-pull-secret.md │ └── local-registry-rpm.md ├── ocp-http.servcies ├── ocp-lb.services ├── poc-registry.service └── haproxy.cfg ├── customizations ├── templates │ ├── 99_kubeadmin-password-secret.yml │ ├── 50-router-replicas.yaml │ ├── 97-master-proxy-UPDATETHIS.yaml │ ├── 97-worker-proxy-UPDATETHIS.yaml │ ├── 98-master-registries-UPDATETHIS.yaml │ ├── 98-worker-registries-UPDATETHIS.yaml │ ├── 11-master-bond1-UPDATETHIS.yaml │ └── 11-worker-bond1-UPDATETHIS.yaml ├── 10-master-nm-workaround.yaml └── 10-worker-nm-workaround.yaml ├── MASTER-WORKER-COMBINED.md ├── README_AIRGAPPED.md ├── install-config-UPDATETHIS.yaml ├── CUSTOMIZATIONS.md ├── poc.sh ├── LICENSE └── README.md /set-env: -------------------------------------------------------------------------------- 1 | # Source this file to get definition 2 | export KUBECONFIG=`pwd`/ocp4/auth/kubeconfig 3 | -------------------------------------------------------------------------------- /static/OCP_4x_bootstrap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/williamcaban/ocp4-lab/HEAD/static/OCP_4x_bootstrap.png -------------------------------------------------------------------------------- /static/pxe-boot-master.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/williamcaban/ocp4-lab/HEAD/static/pxe-boot-master.png -------------------------------------------------------------------------------- /static/pxe-boot-worker.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/williamcaban/ocp4-lab/HEAD/static/pxe-boot-worker.png -------------------------------------------------------------------------------- /static/pxe-write-image.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/williamcaban/ocp4-lab/HEAD/static/pxe-write-image.png -------------------------------------------------------------------------------- /static/bootstrap-progress.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/williamcaban/ocp4-lab/HEAD/static/bootstrap-progress.png -------------------------------------------------------------------------------- /static/pxe-boot-bootstrap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/williamcaban/ocp4-lab/HEAD/static/pxe-boot-bootstrap.png -------------------------------------------------------------------------------- /static/pxe-boot-local-disk.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/williamcaban/ocp4-lab/HEAD/static/pxe-boot-local-disk.png -------------------------------------------------------------------------------- /utils/htpasswd.sample: -------------------------------------------------------------------------------- 1 | ocpadmin:$apr1$ZuJlQr.Y$6abuePAhKG0iY8QDNWoq80 2 | developer:$apr1$QE2hKzLx$4ZeptR1hHNP538zRh/Pew. 3 | 4 | -------------------------------------------------------------------------------- /utils/DNS/10-static-dns.conf: -------------------------------------------------------------------------------- 1 | # /etc/NetworkManager/conf.d/10-static-dns.conf 2 | # NOTE: Used to disable DHCP client from updating /etc/resolv.conf 3 | [main] 4 | dns=none 5 | -------------------------------------------------------------------------------- /utils/PXE/pxelinux.cfg-localboot: -------------------------------------------------------------------------------- 1 | DEFAULT LOCAL 2 | PROMPT 0 3 | TIMEOUT 0 4 | ONTIMEOUT LOCAL 5 | 6 | LABEL LOCAL 7 | MENU LABEL Boot from Local Disk 8 | MENU DEFAULT 9 | LOCALBOOT 0 -------------------------------------------------------------------------------- /customizations/templates/99_kubeadmin-password-secret.yml: -------------------------------------------------------------------------------- 1 | kind: Secret 2 | apiVersion: v1 3 | metadata: 4 | namespace: kube-system 5 | name: kubeadmin 6 | data: 7 | # a1b2c-3d4e5-f6g7h-8i9ab 8 | kubeadmin: JDJiJDEyJERZVnBadS9WLzVlQjB1cG5CVmpFYmVWM0xwY2h1VlFYZ1U0d0xCRkhUaVFTbGs5NXRlTmUu -------------------------------------------------------------------------------- /utils/DNS/10-populate-resolv.conf: -------------------------------------------------------------------------------- 1 | # /etc/NetworkManager/conf.d/10-populate-resolv.conf 2 | # NOTE: Used to predefine NM entries for /etc/resolv.conf 3 | [main] 4 | dns=default 5 | 6 | [global-dns] 7 | searches=ocp4poc.example.com 8 | 9 | [global-dns-domain-*] 10 | servers=8.8.8.8 -------------------------------------------------------------------------------- /utils/BareMetal/ipmi-shutdown.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | export IPMI_USER=user 4 | export IPMI_PASS=password 5 | 6 | export HOST_LIST=(host1 host2 host3 host4) 7 | 8 | for host in "${HOST_LIST[@]}"; do 9 | ipmitool -I lanplus -H ${host} -U ${IPMI_USER} -P ${IPMI_PASS} chassis power off 10 | done -------------------------------------------------------------------------------- /utils/NFS/nfs-export.txt: -------------------------------------------------------------------------------- 1 | # Sample NFS 2 | /opt/imageregistry *(rw,sync,no_wdaly,no_root_squash,insecure) 3 | /opt/clusterlogging-0 *(rw,sync,no_wdaly,no_root_squash,insecure) 4 | /opt/clusterlogging-1 *(rw,sync,no_wdaly,no_root_squash,insecure) 5 | /opt/clusterlogging-2 *(rw,sync,no_wdaly,no_root_squash,insecure) -------------------------------------------------------------------------------- /utils/nm-patch.json: -------------------------------------------------------------------------------- 1 | { 2 | "systemd": { 3 | "units": [ 4 | { 5 | "name": "NetworkManager-wait-online.service", 6 | "dropins": [{ 7 | "name": "timeout.conf", 8 | "contents": "[Service]\nExecStart=\nExecStart=/usr/bin/nm-online -s -q --timeout=300" 9 | }] 10 | } 11 | ] 12 | } 13 | } -------------------------------------------------------------------------------- /customizations/templates/50-router-replicas.yaml: -------------------------------------------------------------------------------- 1 | # oc get ingresscontrollers -n openshift-ingress-operator default -o yaml 2 | # oc edit ingresscontrollers -n openshift-ingress-operator default 3 | apiVersion: operator.openshift.io/v1 4 | kind: IngressController 5 | metadata: 6 | name: default 7 | namespace: openshift-ingress-operator 8 | spec: 9 | replicas: 3 -------------------------------------------------------------------------------- /utils/mcp/mcp-worker-bm.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: machineconfiguration.openshift.io/v1 2 | kind: MachineConfigPool 3 | metadata: 4 | name: worker-bm 5 | spec: 6 | machineConfigSelector: 7 | matchExpressions: 8 | - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-bm]} 9 | nodeSelector: 10 | matchLabels: 11 | node-role.kubernetes.io/worker-bm: "" -------------------------------------------------------------------------------- /utils/PXE/pxelinux-cfg-bootstrap: -------------------------------------------------------------------------------- 1 | DEFAULT pxeboot 2 | PROMPT 0 3 | TIMEOUT 0 4 | ONTIMEOUT pxeboot 5 | 6 | LABEL pxeboot 7 | MENU DEFAULT 8 | KERNEL rhcos/rhcos-kernel 9 | APPEND ip=eno1:dhcp rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.ignition_url=http://192.168.1.1:8000/bootstrap.ign coreos.inst.image_url=http://192.168.1.1:8000/metal/rhcos-4.1.0-x86_64-metal-bios.raw.gz -------------------------------------------------------------------------------- /utils/Ignition/worker-ignition.json: -------------------------------------------------------------------------------- 1 | { 2 | "ignition": { 3 | "version": "2.2.0", 4 | "config": { 5 | "append": [ 6 | { 7 | "source": "http://bastion.example.com:8000/worker.ign", 8 | "verification": {} 9 | } 10 | ] 11 | } 12 | }, 13 | "networkd": {}, 14 | "passwd": {}, 15 | "storage": {}, 16 | "systemd": {} 17 | } 18 | -------------------------------------------------------------------------------- /utils/Ignition/ignition-template_ign.json: -------------------------------------------------------------------------------- 1 | { 2 | "ignition": { 3 | "version": "2.2.0", 4 | "config": { 5 | "append": [ 6 | { 7 | "source": "http://bastion.example.com:8000/worker.ign", 8 | "verification": {} 9 | } 10 | ] 11 | } 12 | }, 13 | "networkd": {}, 14 | "passwd": {}, 15 | "storage": {}, 16 | "systemd": {} 17 | } 18 | } 19 | -------------------------------------------------------------------------------- /utils/NFS/nfs-pv-image-registry.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: PersistentVolume 3 | metadata: 4 | name: pv-image-registry 5 | spec: 6 | capacity: 7 | storage: 100Gi 8 | accessModes: 9 | - ReadWriteMany 10 | nfs: 11 | path: /opt/nfs/image-registry 12 | server: 192.168.1.1 13 | persistentVolumeReclaimPolicy: Retain 14 | 15 | # NOTE: If using RHEL NFS Server with SELinux 16 | # chown nobody.nobody /opt/nfs/imageregistry 17 | # chcon -Rt nfs_t /opt/nfs/imageregistry 18 | -------------------------------------------------------------------------------- /utils/scripts/encode_kubeadmin_password.py: -------------------------------------------------------------------------------- 1 | #!/bin/env python3 2 | 3 | try: 4 | import bcrypt, base64 5 | except ImportError: 6 | raise ImportError("Missing 'bcrypt' or 'base64' libraries ") 7 | 8 | # Password Format: 5char-5char-5char-5char 9 | MYPASSWORD = b"a1b2c-3d4e5-f6g7h-8i9ab" 10 | 11 | # Generate ascii encoded encrypted password with random salt 12 | encrypted_password = bcrypt.hashpw(MYPASSWORD,bcrypt.gensalt()) 13 | 14 | b64_password = base64.b64encode(encrypted_password) 15 | 16 | print ("OCP base64 encoded password:",b64_password) -------------------------------------------------------------------------------- /utils/OSP/troubleshooting-cloud-credentials.md: -------------------------------------------------------------------------------- 1 | # Troubleshooting 2 | 3 | ``` 4 | oc get cm -n openshift-kube-apiserver -o yaml 5 | oc get cm -n openshift-config -o yaml 6 | ``` 7 | 8 | By default the OpenStack credentials are set via a Secret: 9 | - secret-name: `openstack-credentials` 10 | - secret-namespace: `kube-system` 11 | 12 | ``` 13 | oc get secret -n kube-system openstack-credentials --template='{{index .data "clouds.yaml"}}' | base64 -d 14 | oc get secret -n kube-system openstack-credentials --template='{{index .data "clouds.conf"}}' | base64 -d 15 | ``` -------------------------------------------------------------------------------- /utils/Registry/registry_config.yml: -------------------------------------------------------------------------------- 1 | version: 0.1 2 | log: 3 | fields: 4 | service: registry 5 | environment: development 6 | storage: 7 | cache: 8 | layerinfo: inmemory 9 | filesystem: 10 | rootdirectory: /opt/docker-registry 11 | delete: 12 | enabled: true 13 | http: 14 | addr: :5000 15 | tls: 16 | certificate: /etc/docker-distribution/certs/domain.crt 17 | key: /etc/docker-distribution/certs/domain.key 18 | host: https://test.example.com:5000 19 | secret: testsecret1 20 | relativeurls: false 21 | auth: 22 | htpasswd: 23 | realm: basic-realm 24 | path: /etc/docker-distribution/registry_passwd 25 | -------------------------------------------------------------------------------- /utils/BareMetal/ocp4.3-metal3-config.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: metal3-config 5 | namespace: openshift-machine-api 6 | data: 7 | cache_url: '' 8 | deploy_kernel_url: http://172.22.0.3:6180/images/ironic-python-agent.kernel 9 | deploy_ramdisk_url: http://172.22.0.3:6180/images/ironic-python-agent.initramfs 10 | dhcp_range: 172.22.0.10,172.22.0.100 11 | http_port: "6180" 12 | ironic_endpoint: http://172.22.0.3:6385/v1/ 13 | ironic_inspector_endpoint: http://172.22.0.3:5050/v1/ 14 | provisioning_interface: eno1 15 | provisioning_ip: 172.22.0.3/24 16 | rhcos_image_url: http://192.168.1.1/images/rhcos-43.81.-openstack.x86_64.qcow2.ga?sha256= 17 | -------------------------------------------------------------------------------- /utils/ocp-http.servcies: -------------------------------------------------------------------------------- 1 | # 2 | # Copy (chmod 664) to /etc/systemd/system/ocp-http.service 3 | # 4 | # systemctl daemon-reload 5 | # systemctl start ocp-http 6 | # systemctl status ocp-http 7 | # systemctl enable ocp-http 8 | # 9 | 10 | [Unit] 11 | Description=OpenShift HTTP for Ignition files and PXE Configs 12 | After=network.target syslog.target 13 | 14 | [Service] 15 | Type=simple 16 | TimeoutStartSec=5m 17 | ExecStartPre=-/usr/bin/podman rm "ocp-http" 18 | 19 | ExecStart=/usr/bin/podman run --name ocp-http -p 8000:80 -v /opt/nginx/html:/usr/share/nginx/html:ro nginx 20 | 21 | ExecReload=-/usr/bin/podman stop "ocp-http" 22 | ExecReload=-/usr/bin/podman rm "ocp-http" 23 | ExecStop=-/usr/bin/podman stop "ocp-http" 24 | Restart=always 25 | RestartSec=30 26 | 27 | [Install] 28 | WantedBy=multi-user.target 29 | -------------------------------------------------------------------------------- /utils/ocp-lb.services: -------------------------------------------------------------------------------- 1 | # /etc/systemd/system/ocp-lb.service 2 | # 3 | # Copy (chmod 664) to /etc/systemd/system/ocp-lb.service 4 | # 5 | # systemctl daemon-reload 6 | # systemctl start ocp-lb 7 | # systemctl status ocp-lb 8 | # systemctl enable ocp-lb 9 | # 10 | 11 | [Unit] 12 | Description=OpenShift External Load Balancer 13 | After=network.target syslog.target 14 | 15 | [Service] 16 | Type=simple 17 | TimeoutStartSec=5m 18 | ExecStartPre=-/usr/bin/podman rm "ocp-lb" 19 | 20 | ExecStart=/usr/bin/podman run --name ocp-lb --net host -v /opt/haproxy:/usr/local/etc/haproxy:ro haproxy 21 | 22 | ExecReload=-/usr/bin/podman stop "ocp-lb" 23 | ExecReload=-/usr/bin/podman rm "ocp-lb" 24 | ExecStop=-/usr/bin/podman stop "ocp-lb" 25 | Restart=always 26 | RestartSec=30 27 | 28 | [Install] 29 | WantedBy=multi-user.target 30 | -------------------------------------------------------------------------------- /customizations/10-master-nm-workaround.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: machineconfiguration.openshift.io/v1 3 | kind: MachineConfig 4 | metadata: 5 | creationTimestamp: null 6 | labels: 7 | machineconfiguration.openshift.io/role: master 8 | name: 10-master-nm-workaround 9 | spec: 10 | config: 11 | ignition: 12 | config: {} 13 | security: 14 | tls: {} 15 | timeouts: {} 16 | version: 2.2.0 17 | networkd: {} 18 | passwd: {} 19 | storage: {} 20 | systemd: { 21 | "units": [ 22 | { 23 | "name": "NetworkManager-wait-online.service", 24 | "dropins": [{ 25 | "name": "timeout.conf", 26 | "contents": "[Service]\nExecStart=\nExecStart=/usr/bin/nm-online -s -q --timeout=300" 27 | }] 28 | } 29 | ] 30 | } 31 | osImageURL: "" -------------------------------------------------------------------------------- /customizations/10-worker-nm-workaround.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: machineconfiguration.openshift.io/v1 3 | kind: MachineConfig 4 | metadata: 5 | creationTimestamp: null 6 | labels: 7 | machineconfiguration.openshift.io/role: worker 8 | name: 10-worker-nm-workaround 9 | spec: 10 | config: 11 | ignition: 12 | config: {} 13 | security: 14 | tls: {} 15 | timeouts: {} 16 | version: 2.2.0 17 | networkd: {} 18 | passwd: {} 19 | storage: {} 20 | systemd: { 21 | "units": [ 22 | { 23 | "name": "NetworkManager-wait-online.service", 24 | "dropins": [{ 25 | "name": "timeout.conf", 26 | "contents": "[Service]\nExecStart=\nExecStart=/usr/bin/nm-online -s -q --timeout=300" 27 | }] 28 | } 29 | ] 30 | } 31 | osImageURL: "" -------------------------------------------------------------------------------- /customizations/templates/97-master-proxy-UPDATETHIS.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: machineconfiguration.openshift.io/v1 2 | kind: MachineConfig 3 | metadata: 4 | creationTimestamp: null 5 | labels: 6 | machineconfiguration.openshift.io/role: master 7 | name: 97-master-proxy 8 | spec: 9 | config: 10 | ignition: 11 | config: {} 12 | security: 13 | tls: {} 14 | timeouts: {} 15 | version: 2.2.0 16 | networkd: {} 17 | passwd: {} 18 | storage: { 19 | "files": [ 20 | { 21 | "path": "/etc/environment", 22 | "filesystem": "root", 23 | "mode": 420, 24 | "contents": { 25 | "source": "data:;base64," 26 | } 27 | } 28 | ] 29 | } 30 | systemd: {} 31 | osImageURL: "" -------------------------------------------------------------------------------- /customizations/templates/97-worker-proxy-UPDATETHIS.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: machineconfiguration.openshift.io/v1 2 | kind: MachineConfig 3 | metadata: 4 | creationTimestamp: null 5 | labels: 6 | machineconfiguration.openshift.io/role: worker 7 | name: 97-worker-proxy 8 | spec: 9 | config: 10 | ignition: 11 | config: {} 12 | security: 13 | tls: {} 14 | timeouts: {} 15 | version: 2.2.0 16 | networkd: {} 17 | passwd: {} 18 | storage: { 19 | "files": [ 20 | { 21 | "path": "/etc/environment", 22 | "filesystem": "root", 23 | "mode": 420, 24 | "contents": { 25 | "source": "data:;base64," 26 | } 27 | } 28 | ] 29 | } 30 | systemd: {} 31 | osImageURL: "" -------------------------------------------------------------------------------- /utils/scripts/patch-systemd-units.py: -------------------------------------------------------------------------------- 1 | #!/bin/env python3 2 | 3 | import argparse 4 | import json 5 | 6 | __version__ ='0.1.0' 7 | 8 | def patch_units(ignition, patch): 9 | d1 = json.load(open(ignition,'r')) 10 | d2 = json.load(open(patch,'r')) 11 | 12 | d1["systemd"]["units"] += d2["systemd"]["units"] 13 | 14 | print (json.dumps(d1)) 15 | 16 | 17 | def main(): 18 | """ 19 | Main entry point 20 | """ 21 | parser = argparse.ArgumentParser() 22 | parser.add_argument( 23 | '-i', '--ignition', help='Path to ignition file to use as the base', 24 | required=True) 25 | parser.add_argument( 26 | '-p', '--patch', help='Patch for systemd.units in JSON', 27 | required=True) 28 | parser.add_argument( 29 | '--version', action='version', 30 | version='%(prog)s {}'.format(__version__)) 31 | 32 | args = parser.parse_args() 33 | 34 | patch_units(args.ignition, args.patch) 35 | 36 | 37 | if __name__ == '__main__': 38 | main() 39 | -------------------------------------------------------------------------------- /utils/PXE/dnsmasq-pxe.conf: -------------------------------------------------------------------------------- 1 | # OCP4 PXE BOOT Lab 2 | ### dnsmasq configurations 3 | # disable DNS /etc/dnsmasq.conf set port=0 4 | # 5 | no-dhcp-interface=eth0 6 | interface=eth1 7 | 8 | #domain=ocp4poc.example.com 9 | 10 | #### DHCP (dnsmasq --help dhcp) 11 | dhcp-range=eth1,192.168.1.10,192.168.1.200,24h 12 | dhcp-option=option:netmask,255.255.255.0 13 | dhcp-option=option:router,192.168.1.1 14 | dhcp-option=option:dns-server,192.168.1.1 15 | dhcp-option=option:ntp-server,204.11.201.10 16 | 17 | # Bootstrap 18 | dhcp-host=02:01:01:01:01:01,192.168.1.10 19 | 20 | # master-0, master-1, master-2 21 | dhcp-host=02:00:00:00:01:01,192.168.1.11 22 | dhcp-host=02:00:00:00:01:02,192.168.1.12 23 | dhcp-host=02:00:00:00:01:03,192.168.1.13 24 | 25 | # worker-0, worker-1, worker-2 26 | dhcp-host=02:00:00:00:02:01,192.168.1.15 27 | dhcp-host=02:00:00:00:02:01,192.168.1.16 28 | dhcp-host=02:00:00:00:02:01,192.168.1.17 29 | 30 | #### PXE 31 | enable-tftp 32 | tftp-root=/var/lib/tftpboot,eth1 33 | dhcp-boot=pxelinux.0 34 | 35 | # 36 | # END OF FILE 37 | # 38 | -------------------------------------------------------------------------------- /utils/Registry/local_registries.conf: -------------------------------------------------------------------------------- 1 | # This is a system-wide configuration file used to 2 | # keep track of registries for various container backends. 3 | # It adheres to TOML format and does not support recursive 4 | # lists of registries. 5 | 6 | # The default location for this configuration file is /etc/containers/registries.conf. 7 | 8 | # The only valid categories are: 'registries.search', 'registries.insecure', 9 | # and 'registries.block'. 10 | 11 | [registries.search] 12 | registries = ['test.example.com:5000','registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org'] 13 | 14 | # If you need to access insecure registries, add the registry's fully-qualified name. 15 | # An insecure registry is one that does not have a valid SSL certificate or only does HTTP. 16 | [registries.insecure] 17 | registries = [] 18 | 19 | 20 | # If you need to block pull access from a registry, uncomment the section below 21 | # and add the registries fully-qualified name. 22 | # 23 | # Docker only 24 | [registries.block] 25 | registries = [] 26 | -------------------------------------------------------------------------------- /utils/PXE/dnsmasq-ipxe-matchbox.conf: -------------------------------------------------------------------------------- 1 | # Example iPXE configuration with Matchbox 2 | # clusterName = ocp4poc 3 | # baseDomain = example.com 4 | 5 | ## iPXE configuration 6 | 7 | # ipxe server 8 | address=/matchbox.ocp4poc.example.com/192.168.1.1 9 | 10 | # TFTP parameters 11 | enable-tftp 12 | tftp-root=/tftpboot 13 | tftp-no-fail 14 | tftp-secure 15 | #tftp-mtu=1500 16 | 17 | # Legacy PXE 18 | dhcp-match=set:bios,option:client-arch,0 19 | dhcp-boot=tag:bios,undionly.kpxe,matchbox.ocp4poc.example.com,192.168.1.1 20 | 21 | # UEFI 22 | dhcp-match=set:efi32,option:client-arch,6 23 | dhcp-boot=tag:efi32,ipxe.efi,matchbox.ocp4poc.example.com,192.168.1.1 24 | dhcp-match=set:efibc,option:client-arch,7 25 | dhcp-boot=tag:efibc,ipxe.efi,matchbox.ocp4poc.example.com,192.168.1.1 26 | dhcp-match=set:efi64,option:client-arch,9 27 | dhcp-boot=tag:efi64,ipxe.efi,matchbox.ocp4poc.example.com,192.168.1.1 28 | 29 | # iPXE - chainload to matchbox.ocp4poc.example.com ipxe boot script 30 | dhcp-userclass=set:ipxe,iPXE 31 | dhcp-boot=tag:ipxe,http://matchbox.ocp4poc.example.com:8080/boot.ipxe 32 | 33 | # 34 | # END OF FILE 35 | # -------------------------------------------------------------------------------- /utils/Ignition/static-name-dns-ignition.json: -------------------------------------------------------------------------------- 1 | { 2 | "ignition": { 3 | "version": "2.2.0", 4 | "config": { 5 | "append": [ 6 | { 7 | "source": "http://bastion.example.com:8000/worker.ign", 8 | "verification": {} 9 | } 10 | ] 11 | } 12 | }, 13 | "networkd": {}, 14 | "passwd": {}, 15 | "storage": { 16 | "files": [ 17 | { 18 | "filesystem": "root", 19 | "path": "/etc/resolv.conf", 20 | "contents": { 21 | "source": "data:;base64,", 22 | "verification": {} 23 | }, 24 | "mode": 420 25 | }, 26 | { 27 | "filesystem": "root", 28 | "path": "/etc/hostname", 29 | "contents": { 30 | "source": "data:;base64,", 31 | "verification": {} 32 | }, 33 | "mode": 420 34 | } 35 | ] 36 | }, 37 | "systemd": {} 38 | } 39 | -------------------------------------------------------------------------------- /utils/mcp/mcp-worker-bm-mc.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: machineconfiguration.openshift.io/v1 2 | kind: MachineConfigPool 3 | metadata: 4 | name: worker-bm 5 | spec: 6 | machineConfigSelector: 7 | matchExpressions: 8 | - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-bm]} 9 | nodeSelector: 10 | matchLabels: 11 | node-role.kubernetes.io/worker-bm: "" 12 | --- 13 | apiVersion: machineconfiguration.openshift.io/v1 14 | kind: MachineConfig 15 | metadata: 16 | labels: 17 | machineconfiguration.openshift.io/role: worker-bm 18 | name: 50-worker-bm 19 | spec: 20 | config: 21 | ignition: 22 | config: {} 23 | security: 24 | tls: {} 25 | timeouts: {} 26 | version: 2.2.0 27 | networkd: {} 28 | passwd: {} 29 | storage: 30 | files: 31 | - path: "/etc/foo/foo.conf" 32 | filesystem: root 33 | mode: 420 34 | contents: 35 | source: data:;base64, 36 | - path: "/etc/foo/foo-other.conf" 37 | filesystem: root 38 | mode: 420 39 | contents: 40 | source: data:;base64, -------------------------------------------------------------------------------- /utils/DNS/named-reverse-zone.conf: -------------------------------------------------------------------------------- 1 | ; /var/named/1.168.192.in-addr.arpa 2 | $TTL 1h 3 | $ORIGIN 1.168.192.IN-ADDR.ARPA. 4 | 5 | @ 1h IN SOA bastion.ocp4poc.example.com. root.ocp4poc.example.com. ( 6 | 2019052901 ; serial 7 | 2H ; refresh 8 | 15 ; retry 9 | 1W ; expiry 10 | 2H ) ; minimum 11 | 12 | IN NS bastion.ocp4poc.example.com. 13 | 14 | 1 IN PTR bastion.ocp4poc.example.com. 15 | 16 | 10 IN PTR bootstrap.ocp4poc.example.com. 17 | 18 | 11 IN PTR master-0.ocp4poc.example.com. 19 | 12 IN PTR master-1.ocp4poc.example.com. 20 | 13 IN PTR master-2.ocp4poc.example.com. 21 | 22 | 15 IN PTR worker-0.ocp4poc.example.com. 23 | 16 IN PTR worker-1.ocp4poc.example.com. 24 | 17 IN PTR worker-2.ocp4poc.example.com. 25 | 26 | 100 IN PTR lb.ocp4poc.example.com. 27 | 28 | ;------------------------------------------------------------------------------------------------------ 29 | ; append to /etc/named.comf 30 | zone "1.168.192.in-addr.arpa" IN { 31 | type master; 32 | file "1.168.192.in-addr.arpa"; 33 | allow-update { none; }; 34 | }; 35 | -------------------------------------------------------------------------------- /MASTER-WORKER-COMBINED.md: -------------------------------------------------------------------------------- 1 | # Combined `Master` and `Worker` Roles on Masters Nodes 2 | 3 | The OCP4.2 UPI deployment may configure the Master nodes with `master` and `worker` roles. If this is not the desired configuration: 4 | 5 | ## Disable combined `master` + `worker` role from the Master Nodes 6 | - Edit the cluster scheduler operator configuration to remove the `worker` role from the Master nodes and set `mastersSchedulable: false` 7 | ``` 8 | oc edit schedulers cluster 9 | ``` 10 | 11 | - The end result should be similar to the following: 12 | ``` 13 | apiVersion: config.openshift.io/v1 14 | kind: Scheduler 15 | metadata: 16 | name: cluster 17 | spec: 18 | mastersSchedulable: false 19 | ``` 20 | 21 | ## Enable combined `master` + `worker` role from the Master Nodes 22 | 23 | - To re-enable `worker` role into the Master Nodes set `mastersSchedulable: true`. 24 | ``` 25 | oc edit schedulers cluster 26 | ``` 27 | - The end result should be similar to the following: 28 | ``` 29 | apiVersion: config.openshift.io/v1 30 | kind: Scheduler 31 | metadata: 32 | name: cluster 33 | spec: 34 | mastersSchedulable: true 35 | ``` 36 | -------------------------------------------------------------------------------- /utils/PXE/openstack-pxe-boot.md: -------------------------------------------------------------------------------- 1 | # Creating PXE-Boot Image for OpenStack 2 | 3 | 1. Create an empty disk file with DOS filesysm 4 | ``` 5 | dd if=/dev/zero of=pxeboot.img bs=1M count=4 6 | mkdosfs pxeboot.img 7 | ``` 8 | 2. Make it bootable 9 | ``` 10 | losetup /dev/loop0 pxeboot.img 11 | mount /dev/loop0 /mnt 12 | syslinux --install /dev/loop0 13 | ``` 14 | 3. Install iPXE Kernel. Stup syslinux.cfg to load it at boot time 15 | ``` 16 | curl -O http://boot.ipxe.org/ipxe.iso 17 | 18 | mount -o loop,ro ipxe.iso /media 19 | 20 | cp /media/ipxe.krn /mnt 21 | 22 | cat > /mnt/syslinux.cfg < 36 | openstack subnet set --port-security-enabled=False 37 | ``` 38 | 39 | # Credits 40 | 41 | Thanks to David Cain for the guidance. 42 | 43 | This information is based on work by kimizhang at https://kimizhang.wordpress.com/2013/08/26/create-pxe-boot-image-for-openstack/ -------------------------------------------------------------------------------- /README_AIRGAPPED.md: -------------------------------------------------------------------------------- 1 | # Disconnected/Airgapped Install with OCP4.2+ 2 | 3 | Preparing the installation configuration file for Disconnected/AirGapped install: 4 | 5 | - Clone ***this*** repo to the Bastion Node 6 | 7 | 8 | - Edit `install-config.yaml` with the pull secrets of your local registry: 9 | 10 | ``` 11 | 12 | 13 | pullSecret: '{"auths": {"registry.ocp4poc.example.com:5000": { 14 | "auth": "ZHVtbXk6ZHVtbXk=", 15 | "email": "noemail@example.com" 16 | }}}' 17 | sshKey: 'ssh-rsa ...' 18 | ``` 19 | - Include the certificate for your local registries at `additionalTrustBundle` in the `install-config.yaml` 20 | ``` 21 | additionalTrustBundle: | 22 | -----BEGIN CERTIFICATE----- 23 | YOUR CERTIFICATE BUNDLE HERE 24 | -----END CERTIFICATE----- 25 | ``` 26 | - Update `install-config.yaml` with the information of the local registry and specifying the original source for the content: (required for correctly interpreting the release metadata) 27 | ``` 28 | imageContentSources: 29 | - mirrors: 30 | - registry.ocp4poc.example.com:5000//release 31 | source: quay.io/openshift-release-dev/ocp-release 32 | ``` 33 | 34 | ## Next Steps 35 | 36 | Follow the regular UPI installation procedure. 37 | 38 | 39 | -------------------------------------------------------------------------------- /install-config-UPDATETHIS.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | baseDomain: example.com 3 | compute: 4 | - hyperthreading: Enabled 5 | name: worker 6 | replicas: 0 7 | controlPlane: 8 | hyperthreading: Enabled 9 | name: master 10 | replicas: 3 11 | metadata: 12 | name: ocp4poc 13 | networking: 14 | # networkType: OpenShiftSDN 15 | networkType: OVNKubernetes 16 | clusterNetworks: 17 | - cidr: 10.128.0.0/14 18 | hostPrefix: 23 19 | serviceNetwork: 20 | - 172.30.0.0/16 21 | # machineCIDR: 192.168.1.0/24. 22 | # externalIP: 23 | # policy: {} 24 | platform: 25 | none: {} 26 | sshKey: 'ssh-ed25519 AAAA...' 27 | pullSecret: '{"auths": ...}' 28 | # pullSecret: '{"auths":{":5000": {"auth": "","email": "you@example.com"}}}' 29 | # proxy: 30 | # httpProxy: http://:@: 31 | # httpsProxy: http://:@: 32 | # noProxy: example.com 33 | # additionalTrustBundle: | 34 | # -----BEGIN CERTIFICATE----- 35 | # ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ 36 | # -----END CERTIFICATE----- 37 | # Record the imageContentSources section from the output of the ./poc mirror command. 38 | # imageContentSources: 39 | # - mirrors: 40 | # - :5000//release 41 | # source: quay.io/openshift-release-dev/ocp-release 42 | # - mirrors: 43 | # - :5000//release 44 | # source: registry.svc.ci.openshift.org/ocp/release 45 | -------------------------------------------------------------------------------- /utils/Ignition/static-ip-ignition.json: -------------------------------------------------------------------------------- 1 | { 2 | "ignition": { 3 | "version": "2.2.0", 4 | "config": { 5 | "append": [ 6 | { 7 | "source": "http://bastion.example.com:8000/worker.ign", 8 | "verification": {} 9 | } 10 | ] 11 | } 12 | }, 13 | "networkd": {}, 14 | "passwd": {}, 15 | "storage": { 16 | "files": [ 17 | { 18 | "filesystem": "root", 19 | "path": "/etc/sysconfig/network-scripts/ifcfg-enp1s0", 20 | "contents": { 21 | "source": "data:;base64,", 22 | "verification": {} 23 | }, 24 | "mode": 420 25 | }, 26 | { 27 | "filesystem": "root", 28 | "path": "/etc/resolv.conf", 29 | "contents": { 30 | "source": "data:;base64,", 31 | "verification": {} 32 | }, 33 | "mode": 420 34 | }, 35 | { 36 | "filesystem": "root", 37 | "path": "/etc/hostname", 38 | "contents": { 39 | "source": "data:;base64,", 40 | "verification": {} 41 | }, 42 | "mode": 420 43 | } 44 | ] 45 | }, 46 | "systemd": {} 47 | } 48 | -------------------------------------------------------------------------------- /utils/Registry/local-registry-docker-client.md: -------------------------------------------------------------------------------- 1 | # Configuring the Docker client 2 | 1. Configure docker client with self-signed certificate 3 | 4 | - Create the directory under certs.d depending on the FQDN that you have configured while creating the certificate 5 | 6 | ``` 7 | mkdir -p /etc/docker/certs.d/:5000 8 | 9 | #example 10 | mkdir -p /etc/docker/certs.d/bastion.example.com:5000 11 | 12 | ``` 13 | 14 | 2. Copy the `domain.crt` to the new directory 15 | 16 | ``` 17 | cp /etc/docker-distribution/certs/domain.crt /etc/docker/certs.d/bastion.example.com\:5000/domain.crt 18 | 19 | # NOTE: replace the bastion.example.com with the FQDN of local registry 20 | ``` 21 | 22 | 3. Trust this certificate 23 | 24 | ``` 25 | cp /etc/docker-distribution/certs/domain.crt /etc/pki/ca-trust/source/anchors/bastion.example.com.crt 26 | 27 | # NOTE: must be named with the FQDN of the registry 28 | 29 | update-ca-trust 30 | 31 | ``` 32 | 33 | 4. Add the newly created registry to the `/etc/containers/registries.conf` 34 | 35 | ``` 36 | registries: 37 | - registry.access.redhat.com 38 | - bastion.example.com:5000 39 | ``` 40 | 41 | 5. Restart the docker-distribution service 42 | 43 | ``` 44 | systemctl restart docker-distribution 45 | 46 | ``` 47 | 48 | 49 | ## Troubleshooting 50 | - If needed, to test or debug the registry configuration run 51 | ``` 52 | /usr/bin/registry serve /etc/docker-distribution/registry/config.yml 53 | ``` -------------------------------------------------------------------------------- /utils/PXE/sample.ipxe: -------------------------------------------------------------------------------- 1 | :local 2 | exit 3 | 4 | :shell 5 | echo Type 'exist' to get back to the menu 6 | set menu-timeout 0 7 | set submenu-timeout 0 8 | goto start 9 | 10 | :bootstrap 11 | set base-url http://192.168.1.1:8000 12 | kernel ${base-url}/install/rhcos-installer-kernel initrd=rhcos-installer-initramfs.img ip=dhcp rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.first_boot=1 coreos.inst.image_url=http://192.168.1.1:8000/install/rhcos-metal.raw.gz coreos.inst.ignition_url=http://192.168.1.1:8000/ignition/bootstrap.ign 13 | initrd ${base-url}/install/rhcos-installer-initramfs.img 14 | boot || goto failed 15 | 16 | :master 17 | set base-url http://192.168.1.1:8000 18 | kernel ${base-url}/install/rhcos-installer-kernel initrd=rhcos-installer-initramfs.img ip=dhcp rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.first_boot=1 coreos.inst.image_url=http://192.168.1.1:8000/install/rhcos-metal.raw.gz coreos.inst.ignition_url=http://192.168.1.1:8000/ignition/master.ign 19 | initrd ${base-url}/install/rhcos-installer-initramfs.img 20 | boot || goto failed 21 | 22 | :worker 23 | set base-url http://192.168.1.1:8000 24 | kernel ${base-url}/install/rhcos-installer-kernel initrd=rhcos-installer-initramfs.img ip=dhcp rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.first_boot=1 coreos.inst.image_url=http://192.168.1.1:8000/install/rhcos-metal.raw.gz coreos.inst.ignition_url=http://192.168.1.1:8000/ignition/worker.ign 25 | initrd ${base-url}/install/rhcos-installer-initramfs.img 26 | boot || goto failed -------------------------------------------------------------------------------- /utils/Registry/local-registry-dedicated-disk.md: -------------------------------------------------------------------------------- 1 | # (optional) Using Dedicate Disk for registry 2 | 3 | - NOTE: If you see following error message while uploading images into your local registry then either you create dedciated disk with fstype=1 or re-install your server with disk setup as fstype=1 4 | 5 | Error message: 6 | ``` 7 | "Could not get runtime: kernel does not support overlay fs: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported.: driver not supported" 8 | ``` 9 | 10 | 1. Create partition in disk 11 | 12 | ``` 13 | fdisk /dev/sdb 14 | type (t): 8e 15 | ``` 16 | 17 | 2. Create physical volume 18 | 19 | ``` 20 | pvcreate /dev/sdb1 21 | ``` 22 | 23 | 3. Create volume group 24 | 25 | ``` 26 | vgcreate vg_registry /dev/sdb1 27 | 28 | vgs 29 | ``` 30 | 31 | 4. Create logical volume 32 | 33 | ``` 34 | lvcreate -l 100%FREE -n lv_registry vg_registry 35 | 36 | lvdisplay vg_registry/lv_registry 37 | ``` 38 | 39 | 5. Format logical volume 40 | ``` 41 | mkfs.ext4 /dev/vg_registry/lv_registry 42 | ``` 43 | 44 | 6. Identify devices UUID 45 | ``` 46 | blkid /dev/vg_registry/lv_registry 47 | 48 | [root@bastion ~]# blkid /dev/vg_registry/lv_registry 49 | /dev/vg_registry/lv_registry: UUID="60c3366b-1c3d-41da-bf62-7e00d88d3f0b" TYPE="ext4" 50 | 51 | ``` 52 | 53 | 7. /etc/fstab 54 | 55 | ``` 56 | UUID= /opt/registry ext4 defaults 0 0 57 | ``` 58 | -------------------------------------------------------------------------------- /utils/poc-registry.service: -------------------------------------------------------------------------------- 1 | # 2 | # Copy (chmod 664) to /etc/systemd/system/poc-registry.service 3 | # 4 | # systemctl daemon-reload 5 | # systemctl start poc-registry 6 | # systemctl status poc-registry 7 | # systemctl enable poc-registry 8 | # 9 | 10 | [Unit] 11 | Description=OpenShift POC Container Registry 12 | After=network.target syslog.target 13 | 14 | [Service] 15 | Type=simple 16 | TimeoutStartSec=5m 17 | ExecStartPre=-/usr/bin/podman rm "poc-registry" 18 | 19 | ExecStart=/usr/bin/podman run --name poc-registry -p 5000:5000 \ 20 | -v /opt/registry/data:/var/lib/registry:z \ 21 | -v /opt/registry/auth:/auth:z \ 22 | -e "REGISTRY_AUTH=htpasswd" \ 23 | -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry" \ 24 | -e "REGISTRY_HTTP_SECRET=ALongRandomSecretForRegistry" \ 25 | -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \ 26 | -v /opt/registry/certs:/certs:z \ 27 | -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \ 28 | -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \ 29 | docker.io/library/registry:2 30 | 31 | ExecReload=-/usr/bin/podman stop "poc-registry" 32 | ExecReload=-/usr/bin/podman rm "poc-registry" 33 | ExecStop=-/usr/bin/podman stop "poc-registry" 34 | Restart=always 35 | RestartSec=30 36 | 37 | [Install] 38 | WantedBy=multi-user.target 39 | -------------------------------------------------------------------------------- /utils/Registry/registries.conf: -------------------------------------------------------------------------------- 1 | ############################################################# 2 | # SAMPLE /etc/containers/registries.conf 3 | ############################################################# 4 | # The only valid categories are: 'registries.search', 'registries.insecure', 5 | # and 'registries.block'. 6 | 7 | ############################################################# 8 | # List the valid registries to seach. 9 | # - Remove any non used or authorized registries here 10 | ############################################################# 11 | [registries.search] 12 | registries = ['registry.ocp4poc.example.com:5000','registry.redhat.io', 'quay.io', 'docker.io'] 13 | 14 | ############################################################# 15 | # Add other registries from the organization in this section 16 | ############################################################# 17 | # An insecure registry is one that does not have a valid SSL certificate or only does HTTP. 18 | [registries.insecure] 19 | registries = ['registry.ocp4poc.example.com:5000','other-registry.example.com:5000'] 20 | 21 | ############################################################# 22 | # If you need to block pull access from a registry, uncomment 23 | # the section below and add the registries fully-qualified name. 24 | ############################################################# 25 | # Docker only 26 | [registries.block] 27 | registries = [] 28 | 29 | ############################################################# 30 | # END OF FILE 31 | ############################################################# -------------------------------------------------------------------------------- /customizations/templates/98-master-registries-UPDATETHIS.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: machineconfiguration.openshift.io/v1 3 | kind: MachineConfig 4 | metadata: 5 | creationTimestamp: null 6 | labels: 7 | machineconfiguration.openshift.io/role: master 8 | name: 98-master-registries 9 | spec: 10 | config: 11 | ignition: 12 | config: {} 13 | security: 14 | tls: {} 15 | timeouts: {} 16 | version: 2.2.0 17 | networkd: {} 18 | passwd: {} 19 | storage: { 20 | "files": [ 21 | { 22 | "path": "/etc/containers/registries.conf", 23 | "filesystem": "root", 24 | "mode": 420, 25 | "contents": { 26 | "source": "data:;base64," 27 | } 28 | }, 29 | { 30 | "path": "/etc/pki/ca-trust/source/anchors/registry.ocp4poc.example.com.crt", 31 | "filesystem": "root", 32 | "mode": 420, 33 | "contents": { 34 | "source": "data:;base64," 35 | } 36 | } 37 | ] 38 | } 39 | systemd: { 40 | "units": [ 41 | { 42 | "contents": "[Unit]\nDescription=Update system CA\nAfter=syslog.target network.target\n\n[Service]\nType=oneshot\nExecStart=/usr/bin/update-ca-trust\nRemainAfterExit=true\n\n[Install]\nWantedBy=multi-user.target\n", 43 | "enabled": true, 44 | "name": "update-ca.service" 45 | } 46 | ] 47 | } 48 | osImageURL: "" -------------------------------------------------------------------------------- /customizations/templates/98-worker-registries-UPDATETHIS.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: machineconfiguration.openshift.io/v1 3 | kind: MachineConfig 4 | metadata: 5 | creationTimestamp: null 6 | labels: 7 | machineconfiguration.openshift.io/role: worker 8 | name: 98-worker-registries 9 | spec: 10 | config: 11 | ignition: 12 | config: {} 13 | security: 14 | tls: {} 15 | timeouts: {} 16 | version: 2.2.0 17 | networkd: {} 18 | passwd: {} 19 | storage: { 20 | "files": [ 21 | { 22 | "path": "/etc/containers/registries.conf", 23 | "filesystem": "root", 24 | "mode": 420, 25 | "contents": { 26 | "source": "data:;base64," 27 | } 28 | }, 29 | { 30 | "path": "/etc/pki/ca-trust/source/anchors/registry.ocp4poc.example.com.crt", 31 | "filesystem": "root", 32 | "mode": 420, 33 | "contents": { 34 | "source": "data:;base64," 35 | } 36 | } 37 | ] 38 | } 39 | systemd: { 40 | "units": [ 41 | { 42 | "contents": "[Unit]\nDescription=Update system CA\nAfter=syslog.target network.target\n\n[Service]\nType=oneshot\nExecStart=/usr/bin/update-ca-trust\nRemainAfterExit=true\n\n[Install]\nWantedBy=multi-user.target\n", 43 | "enabled": true, 44 | "name": "update-ca.service" 45 | } 46 | ] 47 | } 48 | osImageURL: "" -------------------------------------------------------------------------------- /utils/PXE/pxelinux-cfg-default-uefi: -------------------------------------------------------------------------------- 1 | UI vesamenu.c32 2 | #DEFAULT menu.c32 3 | DEFAULT LOCAL 4 | PROMPT 0 5 | TIMEOUT 200 6 | ONTIMEOUT LOCAL 7 | 8 | #MENU BACKGROUND rhcos/ocp4.png 9 | MENU TITLE PXE BOOT MENU 10 | 11 | LABEL WORKER-UEFI 12 | MENU LABEL ^1 WORKER (UEFI) 13 | KERNEL rhcos/rhcos-kernel 14 | APPEND rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.ignition_url=http://192.168.1.1:8000/worker.ign coreos.inst.image_url=http://192.168.1.1:8000/metal/rhcos-4.1.0-x86_64-metal-uefi.raw.gzip=eno1:dhcp 15 | 16 | LABEL MASTER-UEFI 17 | MENU LABEL ^2 MASTER (UEFI) 18 | KERNEL rhcos/rhcos-kernel 19 | APPEND rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.ignition_url=http://192.168.1.1:8000/master.ign coreos.inst.image_url=http://192.168.1.1:8000/metal/rhcos-4.1.0-x86_64-metal-uefi.raw.gzip=eno1:dhcp 20 | 21 | LABEL BOOTSTRAP-UEFI 22 | MENU LABEL ^3 BOOTSTRAP (UEFI) 23 | KERNEL rhcos/rhcos-kernel 24 | APPEND rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.ignition_url=http://192.168.1.1:8000/bootstrap.ign coreos.inst.image_url=http://192.168.1.1:8000/metal/rhcos-4.1.0-x86_64-metal-uefi.raw.gzip=eno1:dhcp 25 | 26 | LABEL LOCAL 27 | MENU LABEL ^7 Boot from Local Disk 28 | MENU DEFAULT 29 | LOCALBOOT 0 30 | 31 | LABEL RECOVERY1 32 | MENU LABEL ^8 Recovery (initqueue) 33 | KERNEL rhcos/rhcos-kernel 34 | APPEND rd.break=initqueue rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 ip=eno1:dhcp 35 | 36 | LABEL RECOVERY2 37 | MENU LABEL ^9 Recovery (pre-mount) 38 | KERNEL rhcos/rhcos-kernel 39 | APPEND rd.break=pre-mount rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 ip=eno1:dhcp 40 | -------------------------------------------------------------------------------- /CUSTOMIZATIONS.md: -------------------------------------------------------------------------------- 1 | # Working with Customizations 2 | 3 | There are two ways to apply customizations to OpenShift 4.x: 4 | - Customziations to RHCOS before the OCP platform is operational. These customizations require editing the initial Ignition file. 5 | - Customizations after the platform is up and running. At that time we use MachineConfig resources for the customization. 6 | 7 | ## Customization of RHCOS for bootstrap and installation processes 8 | 9 | This type of customization is used to setup advanced RHCOS parameters like: 10 | - Customized Disk Partitions 11 | - Advanced Networking Configurations 12 | 13 | These cutomization use a `filetranspiler` tool that reads a `root disk` layout structure (`/`) from a specific folder and encodes it into the corresponding Ignition file. 14 | 15 | The installation script in this repo can do this for `bootstrap`, `master` and `worker` Ignition files. It will apply these if it detect the corresponding `fake root layout` under the `./customizations/` path. 16 | 17 | The following is an example of injecting network configurations into the `worker` Nodes. 18 | 19 | ``` 20 | $ tree customizations 21 | customizations 22 | └── worker 23 | └── etc 24 | ├── resolve.conf 25 | └── sysconfig 26 | └── network-scripts 27 | ├── ifcfg-blah 28 | └── ifcfg-fake 29 | ``` 30 | 31 | ## Customization after deployment of the platform 32 | 33 | This type of customization uses MachineConfig resources. The installation script in this repo assumes any YAML file under `./customizations/*.yaml` is a MachineConfig and will inject those into the customization of the deployment. 34 | 35 | The following is an example of injecting MachineConfig for customizations. 36 | 37 | ``` 38 | $ tree ./customizations/ 39 | ./customizations/ 40 | ├── 10-master-nm-workaround.yaml 41 | └── 10-worker-nm-workaround.yaml 42 | ``` 43 | -------------------------------------------------------------------------------- /utils/DNS/named-zone.conf: -------------------------------------------------------------------------------- 1 | ;------------------------------------------------------------------------------------------------------ 2 | ; /var/named/ocp4poc.example.com 3 | $TTL 1D 4 | @ IN SOA bastion.ocp4poc.example.com. root.ocp4poc.example.com. ( 5 | 2019052001 ; serial 6 | 1D ; refresh 7 | 2H ; retry 8 | 1W ; expiry 9 | 2D ) ; minimum 10 | 11 | @ IN NS bastion.ocp4poc.example.com. 12 | @ IN A 192.168.1.1 13 | 14 | ; Ancillary services 15 | lb IN A 192.168.1.200 16 | lb-ext IN A 10.10.10.10 17 | 18 | ; Bastion or Jumphost 19 | bastion IN A 192.168.1.1 20 | 21 | ; OCP Cluster 22 | bootstrap IN A 192.168.1.10 23 | 24 | master-0 IN A 192.168.1.11 25 | master-1 IN A 192.168.1.12 26 | master-2 IN A 192.168.1.13 27 | 28 | worker-0 IN A 192.168.1.15 29 | worker-1 IN A 192.168.1.16 30 | worker-2 IN A 192.168.1.17 31 | 32 | etcd-0 IN A 192.168.1.11 33 | etcd-1 IN A 192.168.1.12 34 | etcd-2 IN A 192.168.1.13 35 | 36 | _etcd-server-ssl._tcp.ocp4poc.example.com. IN SRV 0 0 2380 etcd-0.ocp4poc.example.com. 37 | IN SRV 0 0 2380 etcd-1.ocp4poc.example.com. 38 | IN SRV 0 0 2380 etcd-2.ocp4poc.example.com. 39 | 40 | api IN CNAME lb-ext ; external LB interface 41 | api-int IN CNAME lb ; internal LB interface 42 | 43 | apps IN CNAME lb-ext 44 | *.apps IN CNAME lb-ext 45 | 46 | ;------------------------------------------------------------------------------------------------------ 47 | ; append to /etc/named.comf 48 | zone "ocp4poc.example.com" IN { 49 | type master; 50 | file "ocp4poc.example.com"; 51 | allow-update { none; }; 52 | }; 53 | 54 | -------------------------------------------------------------------------------- /utils/OSP/ocp-osp-static-ip.md: -------------------------------------------------------------------------------- 1 | # Create ports with static IPs 2 | 3 | ``` 4 | openstack port create ocp.sdn.port.bootstrap --network ocp.sdn.net --security-group ocp-cluster-sg --fixed-ip subnet=ocp.sdn.subnet,ip-address=192.168.1.10 5 | 6 | openstack port create ocp.sdn.port.master-0 --network ocp.sdn.net --security-group ocp-cluster-sg --fixed-ip subnet=ocp.sdn.subnet,ip-address=192.168.1.11 7 | openstack port create ocp.sdn.port.master-1 --network ocp.sdn.net --security-group ocp-cluster-sg --fixed-ip subnet=ocp.sdn.subnet,ip-address=192.168.1.12 8 | openstack port create ocp.sdn.port.master-2 --network ocp.sdn.net --security-group ocp-cluster-sg --fixed-ip subnet=ocp.sdn.subnet,ip-address=192.168.1.13 9 | 10 | openstack port create ocp.sdn.port.worker-0 --network ocp.sdn.net --security-group ocp-cluster-sg --fixed-ip subnet=ocp.sdn.subnet,ip-address=192.168.1.15 11 | openstack port create ocp.sdn.port.worker-1 --network ocp.sdn.net --security-group ocp-cluster-sg --fixed-ip subnet=ocp.sdn.subnet,ip-address=192.168.1.16 12 | openstack port create ocp.sdn.port.worker-2 --network ocp.sdn.net --security-group ocp-cluster-sg --fixed-ip subnet=ocp.sdn.subnet,ip-address=192.168.1.17 13 | ``` 14 | 15 | ## Create Bootstrap 16 | ``` 17 | openstack server create --image rhcos-4.3.0 --flavor --key-name ocp-key --user-data bootstrap.json --port ocp.sdn.port.bootstrap bootstrap.ocp4.example.com 18 | ``` 19 | 20 | ## Create Master nodes 21 | ``` 22 | for i in {0..2}; do 23 | openstack server create --image rhcos4.3.0 --flavor --key-name ocp-key --user-data master-${i}.json --port ocp.sdn.port.master-${i} master-${i}.ocp4.example.com 24 | done 25 | ``` 26 | 27 | ## Create Worker nodes 28 | ``` 29 | for i in {0..2}; do 30 | openstack server create --image rhcos4.3.0 --flavor --key-name ocp-key --user-data worker-${i}.json --port ocp.sdn.port.worker-${i} worker-${i}.ocp4.example.com 31 | done 32 | ``` 33 | 34 | ## Troubleshooting 35 | - Retrieve console URL for instance 36 | ``` 37 | openstack console url show bootstap.ocp4.example.com 38 | ``` -------------------------------------------------------------------------------- /utils/Registry/local-registry.md: -------------------------------------------------------------------------------- 1 | # Containerized local Container Registry Server 2 | 3 | The following instructions use `/opt/registry` for the locations of the volumes of the container registry. 4 | 5 | 1. Create folders for registry 6 | ``` 7 | mkdir -p /opt/registry/{auth,certs,data} 8 | ``` 9 | 10 | 2. Generate self-signed certificate 11 | ``` 12 | cd /opt/registry/certs 13 | openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt 14 | ``` 15 | 16 | 3. Generate username and password (must use bcrpt formated passwords) 17 | ``` 18 | htpasswd -bBc /opt/registry/auth/htpasswd dummy dummy 19 | ``` 20 | NOTE: If `htpasswd` not available install HTTP tools `yum -y install httpd-tools` 21 | 22 | 4. Install and run the `poc-registry.service`: 23 | ``` 24 | cp ./utils/poc-registry.service /etc/systemd/system/poc-registry.service 25 | 26 | podman pull registry:2 27 | 28 | systemctl daemon-reload 29 | 30 | systemctl start poc-registry 31 | systemctl status poc-registry 32 | systemctl enable poc-registry 33 | ``` 34 | 35 | 5. (if needed) Add port 5000 to the internal and public zone 36 | 37 | ``` 38 | firewall-cmd --add-port=5000/tcp --zone=internal --permanent 39 | firewall-cmd --add-port=5000/tcp --zone=public --permanent 40 | firewall-cmd --reload 41 | ``` 42 | 43 | 6. Verify whether the docker registry is up using the curl command 44 | 45 | ``` 46 | curl -u dummy:dummy -k https://bastion.example.com:5000/v2/_catalog 47 | 48 | # NOTE: It should list an empty repository 49 | ``` 50 | 51 | ## Next steps: 52 | 53 | 1. Create [pull secret](local-registry-pull-secret.md) for Container Registry and run test 54 | 55 | 2. (optional) Configure [Container Registry](local-registry-dedicated-disk.md) with dedicated storage 56 | 57 | 3. (optional) Configure [docker client](local-registry-docker-client.md) to use local registry 58 | 59 | NOTE 1: This is a containerized deployment. If prefer an RPM installation refer to [local-registry-rpm.md](local-registry-rpm.md) -------------------------------------------------------------------------------- /customizations/templates/11-master-bond1-UPDATETHIS.yaml: -------------------------------------------------------------------------------- 1 | # REMOVE THESE COMMENTS 2 | # To obtain the BASE64 encoding: 3 | # cat /etc/sysconfig/network-scripts/ifcfg-bond1 | base64 -w0 4 | # cat /etc/sysconfig/network-scripts/ifcfg-ens3f0 | base64 -w0 5 | # cat /etc/sysconfig/network-scripts/ifcfg-ens3f1 | base64 -w0 6 | # cat /etc/sysconfig/network-scripts/ifcfg-vlan403 | base64 -w0 7 | # 8 | apiVersion: machineconfiguration.openshift.io/v1 9 | kind: MachineConfig 10 | metadata: 11 | creationTimestamp: null 12 | labels: 13 | machineconfiguration.openshift.io/role: master 14 | name: 11-master-bond1 15 | spec: 16 | config: 17 | ignition: 18 | config: {} 19 | security: 20 | tls: {} 21 | timeouts: {} 22 | version: 2.2.0 23 | networkd: {} 24 | passwd: {} 25 | storage: { 26 | "files": [ 27 | { 28 | "path": "/etc/sysconfig/network-scripts/ifcfg-bond1", 29 | "filesystem": "root", 30 | "mode": 420, 31 | "contents": { 32 | "source": "data:;base64," 33 | } 34 | }, 35 | { 36 | "path": "/etc/sysconfig/network-scripts/ifcfg-ens3f0", 37 | "filesystem": "root", 38 | "mode": 420, 39 | "contents": { 40 | "source": "data:;base64," 41 | } 42 | }, 43 | { 44 | "path": "/etc/sysconfig/network-scripts/ifcfg-ens3f1", 45 | "filesystem": "root", 46 | "mode": 420, 47 | "contents": { 48 | "source": "data:;base64," 49 | } 50 | }, 51 | { 52 | "path": "/etc/sysconfig/network-scripts/ifcfg-vlan403", 53 | "filesystem": "root", 54 | "mode": 420, 55 | "contents": { 56 | "source": "data:;base64," 57 | } 58 | } 59 | ] 60 | } 61 | -------------------------------------------------------------------------------- /customizations/templates/11-worker-bond1-UPDATETHIS.yaml: -------------------------------------------------------------------------------- 1 | # REMOVE THESE COMMENTS 2 | # To obtain the BASE64 encoding: 3 | # cat /etc/sysconfig/network-scripts/ifcfg-bond1 | base64 -w0 4 | # cat /etc/sysconfig/network-scripts/ifcfg-ens3f0 | base64 -w0 5 | # cat /etc/sysconfig/network-scripts/ifcfg-ens3f1 | base64 -w0 6 | # cat /etc/sysconfig/network-scripts/ifcfg-vlan403 | base64 -w0 7 | # 8 | apiVersion: machineconfiguration.openshift.io/v1 9 | kind: MachineConfig 10 | metadata: 11 | creationTimestamp: null 12 | labels: 13 | machineconfiguration.openshift.io/role: worker 14 | name: 11-worker-bond1 15 | spec: 16 | config: 17 | ignition: 18 | config: {} 19 | security: 20 | tls: {} 21 | timeouts: {} 22 | version: 2.2.0 23 | networkd: {} 24 | passwd: {} 25 | storage: { 26 | "files": [ 27 | { 28 | "path": "/etc/sysconfig/network-scripts/ifcfg-bond1", 29 | "filesystem": "root", 30 | "mode": 420, 31 | "contents": { 32 | "source": "data:;base64," 33 | } 34 | }, 35 | { 36 | "path": "/etc/sysconfig/network-scripts/ifcfg-ens3f0", 37 | "filesystem": "root", 38 | "mode": 420, 39 | "contents": { 40 | "source": "data:;base64," 41 | } 42 | }, 43 | { 44 | "path": "/etc/sysconfig/network-scripts/ifcfg-ens3f1", 45 | "filesystem": "root", 46 | "mode": 420, 47 | "contents": { 48 | "source": "data:;base64," 49 | } 50 | }, 51 | { 52 | "path": "/etc/sysconfig/network-scripts/ifcfg-vlan403", 53 | "filesystem": "root", 54 | "mode": 420, 55 | "contents": { 56 | "source": "data:;base64," 57 | } 58 | } 59 | ] 60 | } 61 | -------------------------------------------------------------------------------- /utils/OSP/security-group.md: -------------------------------------------------------------------------------- 1 | # Reference SecurityGroups for OCP over OSP 2 | 3 | ``` 4 | export OSP_EXT_NET=10.134.0.0/16 5 | export OSP_TENANT_NET=192.168.50.0/24 6 | ``` 7 | 8 | Reference SecurityGroup for Bastion Node: 9 | ``` 10 | openstack security group create bastion-sg 11 | 12 | openstack security group rule create --ingress --dst-port 22 --protocol tcp --remote-ip ${OSP_EXT_NET} bastion-sg 13 | 14 | openstack security group rule create --ingress --dst-port 6443 --protocol tcp --remote-ip ${OSP_TENANT_NET} bastion-sg 15 | openstack security group rule create --ingress --dst-port 22623 --protocol tcp --remote-ip ${OSP_TENANT_NET} bastion-sg 16 | openstack security group rule create --ingress --dst-port 80 --protocol tcp --remote-ip ${OSP_TENANT_NET} bastion-sg 17 | openstack security group rule create --ingress --dst-port 443 --protocol tcp --remote-ip ${OSP_TENANT_NET} bastion-sg 18 | openstack security group rule create --ingress --dst-port 53 --protocol tcp --remote-ip ${OSP_TENANT_NET} bastion-sg 19 | openstack security group rule create --ingress --dst-port 53 --protocol udp --remote-ip ${OSP_TENANT_NET} bastion-sg 20 | openstack security group rule create --ingress --dst-port 8000 --protocol tcp --remote-ip ${OSP_TENANT_NET} bastion-sg 21 | openstack security group rule create --ingress --dst-port 22 --protocol tcp --remote-ip ${OSP_TENANT_NET} bastion-sg 22 | ``` 23 | 24 | SecurityGroup for OCP Nodes: 25 | ``` 26 | openstack security group create ocp-cluster-sg 27 | 28 | openstack security group rule create --ingress --protocol tcp --remote-ip ${OSP_TENANT_NET} ocp-cluster-sg 29 | openstack security group rule create --ingress --protocol udp --remote-ip ${OSP_TENANT_NET} ocp-cluster-sg 30 | openstack security group rule create --ingress --protocol icmp --remote-ip ${OSP_TENANT_NET} ocp-cluster-sg 31 | 32 | # Allow external access for K8s API and OCP Ingress 33 | openstack security group rule create --ingress --dst-port 6443 --protocol tcp --remote-ip ${OSP_EXT_NET} ocp-cluster-sg 34 | openstack security group rule create --ingress --dst-port 80 --protocol tcp --remote-ip ${OSP_EXT_NET} ocp-cluster-sg 35 | openstack security group rule create --ingress --dst-port 443 --protocol tcp --remote-ip ${OSP_EXT_NET} ocp-cluster-sg 36 | ``` 37 | 38 | ## If using secondary networks with MACVLAN 39 | 40 | ```bash 41 | openstack port set --no-security-group --disable-port-security 42 | ``` 43 | -------------------------------------------------------------------------------- /utils/haproxy.cfg: -------------------------------------------------------------------------------- 1 | # haproxy.cfg 2 | defaults 3 | mode http 4 | log global 5 | option httplog 6 | option dontlognull 7 | option forwardfor except 127.0.0.0/8 8 | option redispatch 9 | retries 3 10 | timeout http-request 10s 11 | timeout queue 1m 12 | timeout connect 10s 13 | timeout client 300s 14 | timeout server 300s 15 | timeout http-keep-alive 10s 16 | timeout check 10s 17 | maxconn 20000 18 | 19 | frontend openshift-api-server 20 | bind *:6443 21 | default_backend openshift-api-server 22 | mode tcp 23 | option tcplog 24 | 25 | backend openshift-api-server 26 | balance source 27 | mode tcp 28 | server bootstrap 192.168.1.10:6443 check 29 | server master-0 192.168.1.11:6443 check 30 | server master-1 192.168.1.12:6443 check 31 | server master-2 192.168.1.13:6443 check 32 | 33 | frontend machine-config-server 34 | bind *:22623 35 | default_backend machine-config-server 36 | mode tcp 37 | option tcplog 38 | 39 | backend machine-config-server 40 | balance source 41 | mode tcp 42 | server bootstrap 192.168.1.10:22623 check 43 | server master-0 192.168.1.11:22623 check 44 | server master-1 192.168.1.12:22623 check 45 | server master-2 192.168.1.13:22623 check 46 | 47 | frontend ingress-http 48 | bind *:80 49 | default_backend ingress-http 50 | mode tcp 51 | option tcplog 52 | 53 | backend ingress-http 54 | balance source 55 | mode tcp 56 | # If combining master/worker role 57 | # server master-0 192.168.1.11:80 check 58 | # server master-1 192.168.1.12:80 check 59 | # server master-2 192.168.1.13:80 check 60 | 61 | server worker-0 192.168.1.15:80 check 62 | server worker-1 192.168.1.16:80 check 63 | server worker-2 192.168.1.17:80 check 64 | 65 | frontend ingress-https 66 | bind *:443 67 | default_backend ingress-https 68 | mode tcp 69 | option tcplog 70 | 71 | backend ingress-https 72 | balance source 73 | mode tcp 74 | # If combining master/worker role 75 | # server master-0 192.168.1.11:443 check 76 | # server master-1 192.168.1.12:443 check 77 | # server master-2 192.168.1.13:443 check 78 | 79 | server worker-0 192.168.1.15:443 check 80 | server worker-1 192.168.1.16:443 check 81 | server worker-2 192.168.1.17:443 check 82 | 83 | # NOTE: Not recommended for production 84 | #listen stats 85 | # bind :9000 86 | # mode http 87 | # stats enable 88 | # stats uri / 89 | 90 | # 91 | # END OF FILE 92 | # 93 | -------------------------------------------------------------------------------- /utils/PXE/pxelinux.cfg-default-bios: -------------------------------------------------------------------------------- 1 | ########################################################### 2 | # Reference: /var/lib/tftpboot/pxelinux.cfg/default 3 | ########################################################### 4 | UI vesamenu.c32 5 | #DEFAULT menu.c32 6 | DEFAULT LOCAL 7 | PROMPT 0 8 | TIMEOUT 200 9 | ONTIMEOUT LOCAL 10 | 11 | #MENU BACKGROUND rhcos/ocp4.png 12 | MENU TITLE PXE BOOT MENU 13 | 14 | LABEL WORKER-BIOS 15 | MENU LABEL ^1 WORKER (BIOS) 16 | KERNEL rhcos/rhcos-kernel 17 | APPEND rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.ignition_url=http://192.168.1.1:8000/worker.ign coreos.inst.image_url=http://192.168.1.1:8000/metal/rhcos-4.1.0-x86_64-metal-bios.raw.gz ip=eno1:dhcp 18 | 19 | LABEL MASTER-BIOS 20 | MENU LABEL ^2 MASTER (BIOS) 21 | KERNEL rhcos/rhcos-kernel 22 | APPEND rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.ignition_url=http://192.168.1.1:8000/master.ign coreos.inst.image_url=http://192.168.1.1:8000/metal/rhcos-4.1.0-x86_64-metal-bios.raw.gz ip=eno1:dhcp 23 | 24 | LABEL BOOTSTRAP-BIOS 25 | MENU LABEL ^3 BOOTSTRAP (BIOS) 26 | KERNEL rhcos/rhcos-kernel 27 | APPEND rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.ignition_url=http://192.168.1.1:8000/bootstrap.ign coreos.inst.image_url=http://192.168.1.1:8000/metal/rhcos-4.1.0-x86_64-metal-bios.raw.gz ip=eno1:dhcp 28 | 29 | LABEL LOCAL 30 | MENU LABEL ^4 BOOT FROM LOCAL DISK 31 | MENU DEFAULT 32 | LOCALBOOT 0 33 | 34 | ########################################################### 35 | # SECTION FOR ADVANCED TROUBLESHOOTING 36 | # man dracut.cmdline for options 37 | ########################################################### 38 | LABEL RECOVERY1 39 | MENU LABEL ^8 Recovery (initqueue) 40 | KERNEL rhcos/rhcos-kernel 41 | APPEND rd.break=initqueue rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 ip=eno1:dhcp 42 | 43 | # NOTES: Using rd.break=initqueue 44 | # Identify disk: lsblk 45 | # Mount root partition: mkdir /mnt ; mount /dev/sda2 /mnt ; cd /mnt 46 | 47 | LABEL RECOVERY2 48 | MENU LABEL ^9 Recovery (pre-mount) 49 | KERNEL rhcos/rhcos-kernel 50 | APPEND rd.break=pre-mount rd.neednet=1 initrd=rhcos/rhcos-initramfs.img console=tty0 ip=eno1:dhcp 51 | 52 | # NOTES: Using rd.break=pre-mount 53 | # cat /proc/cmdline 54 | ########################################################### 55 | # END OF FILE 56 | ########################################################### -------------------------------------------------------------------------------- /utils/PXE/pxelinux.cfg-default-4.6: -------------------------------------------------------------------------------- 1 | UI vesamenu.c32 2 | #DEFAULT menu.c32 3 | DEFAULT LOCAL 4 | PROMPT 0 5 | #TIMEOUT 200 6 | TIMEOUT 100 7 | ONTIMEOUT LOCAL 8 | 9 | #MENU BACKGROUND rhcos/ocp4.png 10 | MENU TITLE PXE BOOT MENU 11 | 12 | LABEL WORKER 13 | MENU LABEL ^1 WORKER 14 | KERNEL http://198.18.100.24:8080/rhcos/rhcos-kernel 15 | APPEND rd.neednet=1 initrd=http://198.18.100.24:8080/rhcos/rhcos-initramfs.img console=tty0,115200n8 coreos.inst=yes coreos.inst.install_dev=/dev/vda coreos.inst.ignition_url=http://198.18.100.24:8080/ignition/worker.ign coreos.live.rootfs_url=http://198.18.100.24:8080/rhcos/rhcos-live-rootfs ip=dhcp 16 | 17 | LABEL MASTER 18 | MENU LABEL ^2 MASTER 19 | KERNEL http://198.18.100.24:8080/rhcos/rhcos-kernel 20 | APPEND rd.neednet=1 initrd=http://198.18.100.24:8080/rhcos/rhcos-initramfs.img console=tty0,115200n8 coreos.inst=yes coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://198.18.100.24:8080/ignition/master.ign coreos.live.rootfs_url=http://198.18.100.24:8080/rhcos/rhcos-live-rootfs ip=dhcp 21 | 22 | LABEL BOOTSTRAP 23 | MENU LABEL ^3 BOOTSTRAP 24 | KERNEL http://198.18.100.24:8080/rhcos/rhcos-kernel 25 | APPEND rd.neednet=1 initrd=http://198.18.100.24:8080/rhcos/rhcos-initramfs.img console=tty0,115200n8 coreos.inst=yes coreos.inst.install_dev=/dev/vda coreos.inst.ignition_url=http://198.18.100.24:8080/ignition/bootstrap.ign coreos.live.rootfs_url=http://198.18.100.24:8080/rhcos/rhcos-live-rootfs ip=dhcp 26 | 27 | LABEL LOCAL 28 | MENU LABEL ^4 BOOT FROM LOCAL DISK 29 | MENU DEFAULT 30 | LOCALBOOT 0 31 | 32 | ########################################################### 33 | # SECTION FOR ADVANCED TROUBLESHOOTING 34 | # man dracut.cmdline for options 35 | ########################################################### 36 | LABEL DRACUT1 37 | MENU LABEL ^8 DRACUT (initqueue) 38 | KERNEL http://198.18.100.24:8080/rhcos/rhcos-kernel 39 | APPEND rd.break=initqueue rd.neednet=1 initrd=http://198.18.100.24:8080/rhcos/rhcos-initramfs.img console=tty0,115200n8 ip=dhcp 40 | 41 | # NOTES: Using rd.break=initqueue 42 | # Identify disk: lsblk 43 | # Mount root partition: mkdir /mnt ; mount /dev/vda2 /mnt ; cd /mnt 44 | 45 | LABEL DRACUT2 46 | MENU LABEL ^9 DRACUT (pre-mount) 47 | KERNEL http://198.18.100.24:8080/rhcos/rhcos-kernel 48 | APPEND rd.break=pre-mount rd.neednet=1 initrd=http://198.18.100.24:8080/rhcos/rhcos-initramfs.img console=tty0,115200n8 ip=dhcp 49 | 50 | # NOTES: Using rd.break=pre-mount 51 | # cat /proc/cmdline 52 | ########################################################### 53 | # END OF FILE 54 | ########################################################### 55 | -------------------------------------------------------------------------------- /utils/Registry/local-registry-pull-secret.md: -------------------------------------------------------------------------------- 1 | # Create pull secret for new registry 2 | 3 | 1. Add credential information of local registry into the `pull-secret.json` file 4 | 5 | ``` 6 | # Example user and password 7 | # NOTE: use the '-n' to generate a valid encrypted password 8 | echo -n "dummy:dummy" | base64 -w0 9 | ZHVtbXk6ZHVtbXk= 10 | 11 | # Edit pull-secret.json 12 | vi pull-secret.json 13 | 14 | ... 15 | "auths": { 16 | "registry.example.com:5000": { 17 | "auth": "ZHVtbXk6ZHVtbXk=", 18 | "email": "noemail@example.com" 19 | }, 20 | ... 21 | ``` 22 | 23 | 2. Test the registry is operational 24 | 25 | ``` 26 | ##################################### 27 | # Test login to the registry 28 | ##################################### 29 | 30 | $ podman login bastion.example.com:5000 31 | Username: dummy 32 | Password: 33 | Login Succeeded! 34 | 35 | ##################################### 36 | # Test pushing a container image 37 | ##################################### 38 | 39 | # Get an image 40 | $ podman pull registry.access.redhat.com/ubi8/ubi-minimal 41 | Trying to pull registry.access.redhat.com/ubi8/ubi-minimal...Getting image source signatures 42 | Copying blob sha256:ed6b7e8623ef8ca893d44d01fc88999684cc0209bc48cd71c6b5a696ed1d60f5 43 | 32.50 MB / ? [-------------------------------------------------=----------] 3s 44 | Copying blob sha256:5b86d995ed7f224d4e810d76a4a7a87702338f37abbd7df916f99e1549e1f68d 45 | 1.41 KB / ? [-----------------------------------------=-------------------] 0s 46 | Copying config sha256:3bfa511b67f82778ace94aaedb7da39d353f33eabc9ae24abad47805b6cef9c3 47 | 4.28 KB / 4.28 KB [========================================================] 0s 48 | Writing manifest to image destination 49 | Storing signatures 50 | 3bfa511b67f82778ace94aaedb7da39d353f33eabc9ae24abad47805b6cef9c3 51 | 52 | # Tag image with local registry 53 | $ podman tag registry.access.redhat.com/ubi8/ubi-minimal:latest bastion.example.com:5000/ubi8/ubi-minimal:latest 54 | 55 | # Push image to local registry 56 | $ podman push bastion.example.com:5000/ubi8/ubi-minimal:latest 57 | Getting image source signatures 58 | Copying blob sha256:62373019ab2eec9b927fd44c87720cd05f675888d11903581e60edeec3d985c2 59 | 87.44 MB / 87.44 MB [=====================================================] 10s 60 | Copying blob sha256:44d5dd834e48e686666301fbc4478baecb1e68ec5eb289b80c096f78da30977d 61 | 20.00 KB / 20.00 KB [======================================================] 0s 62 | Copying config sha256:3bfa511b67f82778ace94aaedb7da39d353f33eabc9ae24abad47805b6cef9c3 63 | 4.28 KB / 4.28 KB [========================================================] 0s 64 | Writing manifest to image destination 65 | Storing signatures 66 | ``` 67 | 68 | -------------------------------------------------------------------------------- /utils/Registry/local-registry-rpm.md: -------------------------------------------------------------------------------- 1 | # Setting local container registry with self-signed CA 2 | 3 | NOTE: RHEL `rhel-7-server-extras-rpms` repo is required for these RPMs. 4 | 5 | 1. Install registry dependencies: 6 | 7 | ``` 8 | yum install -y docker-distribution skopeo podman 9 | 10 | Please note if you are using RHEL7 subscriptions following packages needs to be manually pre-installed in order to have podman and httpd-tools 11 | 12 | libnet-1.1.6-7.el7.x86_64.rpm 13 | python-IPy-0.75-6.el7.noarch.rpm 14 | apr-util-1.5.2-6.el7.x86_64.rpm 15 | 16 | ``` 17 | 18 | 2. Setup registry configuration 19 | 20 | - Generate the required certificate file for the docker-distribution service. 21 | Note: Ensure you use the registry FQDN as the CN when generating the certificates. 22 | 23 | ``` 24 | mkdir /etc/docker-distribution/certs 25 | cd /etc/docker-distribution/certs 26 | openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt 27 | ``` 28 | 29 | - Generate htpasswd based authentication 30 | ``` 31 | htpasswd -cB /etc/docker-distribution/registry/htpasswd dummy dummy 32 | ``` 33 | 34 | - Backup of the existing configuration file `/etc/docker-distribution/registry/config.yml` 35 | 36 | ``` 37 | mv /etc/docker-distribution/registry/config.yml /root/original-docker-distribution-config.xml 38 | ``` 39 | 40 | - Create new `/etc/docker-distribution/registry/config.yml` with content similar to: 41 | ``` 42 | version: 0.1 43 | log: 44 | fields: 45 | service: registry 46 | storage: 47 | cache: 48 | blobdescriptor: inmemory 49 | filesystem: 50 | rootdirectory: /opt/registry 51 | auth: 52 | htpasswd: 53 | realm: basic-realm 54 | # bcrpt formatted passwords 55 | path: /etc/docker-distribution/registry/htpasswd 56 | http: 57 | addr: :5000 58 | host: https://bastion.example.com:5000 59 | secret: myverysecretregistry 60 | tls: 61 | certificate: /etc/docker-distribution/registry/certs/domain.crt 62 | key: /etc/docker-distribution/registry/certs/domain.key 63 | 64 | log: 65 | accesslog: 66 | disabled: false 67 | level: info 68 | formatter: text 69 | fields: 70 | service: registry 71 | environment: staging 72 | ``` 73 | 74 | 75 | - Notes: 76 | - Replace the "host" line appropriately with the FQDN 77 | - Replace the "secret" with a random value 78 | - Replace the rootdirectory as required 79 | - Indentation should be properly maintained 80 | - The password format of /etc/docker-distribution/registry_passwd must be bcrypt 81 | 82 | 83 | 3. Start the `docker-distribution.service` and add port 5000 to the internal and public zone 84 | 85 | ``` 86 | systemctl start docker-distribution 87 | 88 | firewall-cmd --add-port=5000/tcp --zone=internal --permanent 89 | firewall-cmd --add-port=5000/tcp --zone=public --permanent 90 | firewall-cmd --reload 91 | ``` 92 | 93 | 4. Verify whether the docker registry is up using the curl command 94 | 95 | ``` 96 | curl -u dummy:dummy -k https://bastion.example.com:5000/v2/_catalog 97 | 98 | # NOTE: It should list an empty repository 99 | ``` 100 | 101 | ## Next steps: 102 | 103 | 1. Create [pull secret](local-registry-pull-secret.md) for Container Registry and run test 104 | 105 | 2. (optional) Configure [Container Registry](local-registry-dedicated-disk.md) with dedicated storage 106 | 107 | 3. (optional) Configure [docker client](local-registry-docker-client.md) to use local registry 108 | 109 | NOTE 1: This is an RPM deployment. If prefer a Containerized deployment refer to [local-registry.md](local-registry.md) 110 | 111 | 112 | -------------------------------------------------------------------------------- /utils/DNS/dnsmasq.conf: -------------------------------------------------------------------------------- 1 | # Example DHCP and Forward & Reverse DNS configuration 2 | # clusterName = ocp4poc 3 | # baseDomain = example.com 4 | 5 | # Recursive DNS 6 | server=8.8.8.8 7 | #server=8.8.4.4 8 | 9 | domain=ocp4poc.example.com 10 | 11 | # using specific NTP servers 12 | #dhcp-option=option:ntp-server,198.255.68.106,18.222.40.121 13 | 14 | # External API endpoint (External VIP) 15 | address=/api.ocp4poc.example.com/192.168.53.110 16 | ptr-record=110.53.168.192.in-addr.arpa,api.ocp4poc.example.com 17 | 18 | # Internal API endpoint (Internal VIP) 19 | address=/api-int.ocp4poc.example.com/192.168.53.110 20 | ptr-record=110.53.168.192.in-addr.arpa,api-int.ocp4poc.example.com 21 | 22 | # wildcard domain *.apps.. (External VIP) 23 | address=/.apps.ocp4poc.example.com/192.168.53.110 24 | 25 | # etcd SRV records 26 | srv-host=_etcd-server-ssl._tcp.ocp4poc.example.com,etcd-0.ocp4poc.example.com,2380 27 | srv-host=_etcd-server-ssl._tcp.ocp4poc.example.com,etcd-1.ocp4poc.example.com,2380 28 | srv-host=_etcd-server-ssl._tcp.ocp4poc.example.com,etcd-2.ocp4poc.example.com,2380 29 | 30 | # Bootstrap 31 | dhcp-host=00:00:00:00:00:00,192.168.53.120,bootstrap.ocp4poc.example.com 32 | address=/bootstrap.ocp4poc.example.com/192.168.53.120 33 | ptr-record=120.53.168.192.in-addr.arpa,bootstrap.ocp4poc.example.com 34 | 35 | # Master-0 / etcd-0 36 | # NOTE: update 00:00:00:00:00:00 with the actual mac addreses of the Node 37 | dhcp-host=00:00:00:00:00:00,192.168.53.130,master-0.ocp4poc.example.com 38 | address=/master-0.ocp4poc.example.com/192.168.53.130 39 | ptr-record=130.53.168.192.in-addr.arpa,master-0.ocp4poc.example.com 40 | address=/etcd-0.ocp4poc.example.com/192.168.53.130 41 | 42 | # Master-1 / etcd-1 43 | # NOTE: update 00:00:00:00:00:00 with the actual mac addreses of the Node 44 | dhcp-host=00:00:00:00:00:00,192.168.53.131,master-1.ocp4poc.example.com 45 | address=/master-1.ocp4poc.example.com/192.168.53.131 46 | ptr-record=131.53.168.192.in-addr.arpa,master-1.ocp4poc.example.com 47 | address=/etcd-1.ocp4poc.example.com/192.168.53.131 48 | 49 | # Master-2 / etcd-2 50 | # NOTE: update 00:00:00:00:00:00 with the actual mac addreses of the Node 51 | dhcp-host=00:00:00:00:00:00,192.168.53.132,master-2.ocp4poc.example.com 52 | address=/master-2.ocp4poc.example.com/192.168.53.132 53 | ptr-record=132.53.168.192.in-addr.arpa,master-2.ocp4poc.example.com 54 | address=/etcd-2.ocp4poc.example.com/192.168.53.132 55 | 56 | # Worker-0 57 | # NOTE: update 00:00:00:00:00:00 with the actual mac addreses of the Node 58 | dhcp-host=00:00:00:00:00:00,192.168.53.140,worker-0.ocp4poc.example.com 59 | address=/worker-0.ocp4poc.example.com/192.168.53.140 60 | ptr-record=140.53.168.192.in-addr.arpa,worker-0.ocp4poc.example.com 61 | 62 | # Worker-1 63 | # NOTE: update 00:00:00:00:00:00 with the actual mac addreses of the Node 64 | dhcp-host=00:00:00:00:00:00,192.168.53.141,worker-1.ocp4poc.example.com 65 | address=/worker-1.ocp4poc.example.com/192.168.53.141 66 | ptr-record=141.53.168.192.in-addr.arpa,worker-1.ocp4poc.example.com 67 | 68 | # Worker-2 69 | # NOTE: update 00:00:00:00:00:00 with the actual mac addreses of the Node 70 | dhcp-host=00:00:00:00:00:00,192.168.53.142,worker-2.ocp4poc.example.com 71 | address=/worker-2.ocp4poc.example.com/192.168.53.142 72 | ptr-record=142.53.168.192.in-addr.arpa,worker-2.ocp4poc.example.com 73 | 74 | # Worker-3 75 | # NOTE: update 00:00:00:00:00:00 with the actual mac addreses of the Node 76 | dhcp-host=00:00:00:00:00:00,192.168.53.143,worker-3.ocp4poc.example.com 77 | address=/worker-3.ocp4poc.example.com/192.168.53.143 78 | ptr-record=143.53.168.192.in-addr.arpa,worker-3.ocp4poc.example.com 79 | 80 | # Worker-4 81 | # NOTE: update 00:00:00:00:00:00 with the actual mac addreses of the Node 82 | dhcp-host=00:00:00:00:00:00,192.168.53.144,worker-4.ocp4poc.example.com 83 | address=/worker-4.ocp4poc.example.com/192.168.53.144 84 | ptr-record=144.53.168.192.in-addr.arpa,worker-4.ocp4poc.example.com 85 | 86 | # 87 | # END OF FILE 88 | # 89 | -------------------------------------------------------------------------------- /utils/mcp/rhel-rt-mc-and-mcp.md: -------------------------------------------------------------------------------- 1 | # RHEL-RT MachineConfig (MC) and MachineConfigPool (MCP) 2 | 3 | ``` 4 | ################################################ 5 | NO DOT FOLLOW THIS PROCEDURE FOR GENERAL MCP's. 6 | FOR GENERAL MCP's DEFINITION USE DERIVED MC's. 7 | ################################################ 8 | ``` 9 | 10 | Create a `worker-rt` MachineConfigPool with the corresponding `MachineConfig` 11 | 12 | ## Step 1 13 | ``` 14 | oc get machineconfig --selector='machineconfiguration.openshift.io/role=worker' 15 | ``` 16 | 17 | Output should be similar to this. 18 | ``` 19 | $ oc get machineconfig --selector='machineconfiguration.openshift.io/role=worker' 20 | NAME GENERATEDBYCONTROLLER IGNITIONVERSION CREATED 21 | 00-worker 02c07496ba0417b3e12b78fb32baf6293d314f79 2.2.0 12d 22 | 01-worker-container-runtime 02c07496ba0417b3e12b78fb32baf6293d314f79 2.2.0 12d 23 | 01-worker-kubelet 02c07496ba0417b3e12b78fb32baf6293d314f79 2.2.0 12d 24 | 98-worker-registries 2.2.0 10d 25 | 99-worker-d75f2daf-9778-11e9-83e8-001a4a16010a-registries 02c07496ba0417b3e12b78fb32baf6293d314f79 2.2.0 12d 26 | 99-worker-ssh 2.2.0 12d 27 | ``` 28 | 29 | In this case `99-worker-d75f2daf-9778-11e9-83e8-001a4a16010a-registries` will be ignored. 30 | 31 | ## Step 2 32 | For every non-generated MachineConfig listed in `Step 1` export a copy and remove the `ownerReferences` section and any `selfLink` and `annotations` sections. 33 | 34 | ``` 35 | oc get --export machineconfig 00-worker -o yaml > 00-worker-rt.yaml 36 | oc get --export machineconfig 01-worker-container-runtime -o yaml > 01-worker-rt-container-runtime.yaml 37 | oc get --export machineconfig 01-worker-kubelet -o yaml > 01-worker-rt-kubelet.yaml 38 | oc get --export machineconfig 98-worker-registries -o yaml > 98-worker-rt-registries.yaml 39 | oc get --export machineconfig 99-worker-ssh -o yaml > 99-worker-rt-ssh.yaml 40 | 41 | sed -i 's/worker/worker-rt/g' 00-worker-rt.yaml 42 | sed -i '/ownerReferences/,+7d' 00-worker-rt.yaml 43 | sed -i '/annotations/,+1d' 00-worker-rt.yaml 44 | 45 | sed -i 's/worker/worker-rt/g' 01-worker-rt-container-runtime.yaml 46 | sed -i '/ownerReferences/,+7d' 01-worker-rt-container-runtime.yaml 47 | sed -i '/annotations/,+1d' 01-worker-rt-container-runtime.yaml 48 | 49 | sed -i 's/worker/worker-rt/g' 01-worker-rt-kubelet.yaml 50 | sed -i '/ownerReferences/,+7d' 01-worker-rt-kubelet.yaml 51 | sed -i '/annotations/,+1d' 01-worker-rt-kubelet.yaml 52 | 53 | sed -i 's/worker/worker-rt/g' 98-worker-rt-registries.yaml 54 | sed -i '/ownerReferences/,+7d' 98-worker-rt-registries.yaml 55 | sed -i '/annotations/,+1d' 98-worker-rt-registries.yaml 56 | sed -i '/selfLink/d' 98-worker-rt-registries.yaml 57 | 58 | 59 | sed -i 's/worker/worker-rt/g' 99-worker-rt-ssh.yaml 60 | sed -i '/ownerReferences/,+7d' 99-worker-rt-ssh.yaml 61 | sed -i '/annotations/,+1d' 99-worker-rt-ssh.yaml 62 | sed -i '/selfLink/d' 99-worker-rt-ssh.yaml 63 | ``` 64 | 65 | ## Step 3 66 | 67 | Crete the new MachineConfigs 68 | 69 | ``` 70 | oc create -f 00-worker-rt.yaml 71 | oc create -f 01-worker-rt-container-runtime.yaml 72 | oc create -f 01-worker-rt-kubelet.yaml 73 | oc create -f 98-worker-rt-registries.yaml 74 | oc create -f 99-worker-rt-ssh.yaml 75 | ``` 76 | 77 | Create a new MachineConfigPool for the new configurations. 78 | 79 | ``` 80 | cat < ./mcp-worker-rt.yaml 81 | apiVersion: machineconfiguration.openshift.io/v1 82 | kind: MachineConfigPool 83 | metadata: 84 | generation: 1 85 | name: worker-rt 86 | spec: 87 | machineConfigSelector: 88 | matchLabels: 89 | machineconfiguration.openshift.io/role: worker-rt 90 | nodeSelector: 91 | matchLabels: 92 | node-role.kubernetes.io/worker-rt: "" 93 | EOF 94 | ``` 95 | 96 | ``` 97 | oc create -f mcp-worker-rt.yaml 98 | ``` 99 | 100 | ## Step 4 101 | 102 | ***NOTE:*** These steps need to be documented. 103 | 104 | Apply the real-time `tuned` profile to `worker-rt` MCP. 105 | 106 | 107 | ## Tagging Nodes for the MachineConfigPool 108 | Apply the corresponding label to the RHEL-RT node: `node-role.kubernetes.io/worker-rt` and remove the standard `node-role.kubernetes.io/worker` label. -------------------------------------------------------------------------------- /poc.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ############################################################## 3 | # UPDATE TO MATCH YOUR ENVIRONMENT 4 | ############################################################## 5 | 6 | # Must match directory at: https://mirror.openshift.com/pub/openshift-v4/clients/${OCP_RELEASE_PATH} 7 | OCP_RELEASE_PATH=ocp # valid options are "ocp" or "ocp-dev-preview" 8 | OCP_SUBRELEASE=4.6.0-rc.3 9 | 10 | # Must match directory at https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/ 11 | RHCOS_RELEASE=pre-release # "4.5" for latest stable, "pre-release" for nightly 12 | 13 | # ancillary services 14 | WEBROOT=/opt/nginx/html 15 | TFTPROOT=/opt/dnsmasq # Using dnsmasq container as tftpserver, otherwise /tftpboot or /var/lib/tftpboot 16 | POCDIR=ocp4 17 | 18 | ############################################################# 19 | # EXPERIMENTAL 20 | ############################################################## 21 | 22 | #LAST_3_OCP_RELEASES=$(curl -s https://quay.io/api/v1/repository/${UPSTREAM_REPO}/ocp-release/tag/\?limit=3\&page=1\&onlyActiveTags=true | jq -r '.tags[].name') 23 | 24 | AIRGAP_REG='registry.ocp4poc.example.com:5000' 25 | AIRGAP_REPO='ocp4/ocp-release' 26 | 27 | UPSTREAM_REPO='openshift-release-dev' 28 | RELEASE_NAME='ocp-release' 29 | AIRGAP_SECRET_JSON='pull-secret-2.json' 30 | 31 | # NOT NEEDED FOR IF INSTALLER IS RETRIEVED FROM THE SAME REPO 32 | #export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=${AIRGAP_REG}/${AIRGAP_REPO}:${OCP_SUBRELEASE} 33 | 34 | ############################################################## 35 | # DO NOT MODIFY AFTER THIS LINE 36 | ############################################################## 37 | 38 | usage() { 39 | echo -e "Usage: $0 [ clean | ignition | custom | prep_ign | approve ] " 40 | echo -e "\t\t(extras) [ mirror | get_images | prep_installer | prep_images | debug_bootstrap | debug_install ]" 41 | } 42 | 43 | get_images() { 44 | mkdir images ; cd images 45 | 46 | # https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/ 47 | curl -J -L -O https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/${RHCOS_RELEASE}/latest/rhcos-live-initramfs.x86_64.img 48 | curl -J -L -O https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/${RHCOS_RELEASE}/latest/rhcos-live-kernel-x86_64 49 | curl -J -L -O https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/${RHCOS_RELEASE}/latest/rhcos-live-rootfs.x86_64.img 50 | 51 | curl -J -L -O https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/${RHCOS_RELEASE}/latest/rhcos-live.x86_64.iso 52 | #curl -J -L -O https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/${RHCOS_RELEASE}/latest/rhcos-metal.x86_64.raw.gz 53 | #curl -J -L -O https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/${RHCOS_RELEASE}/latest/rhcos-qemu.x86_64.qcow2.gz 54 | 55 | # https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.2.0/ 56 | curl -J -L -O https://mirror.openshift.com/pub/openshift-v4/clients/${OCP_RELEASE_PATH}/${OCP_SUBRELEASE}/openshift-client-linux-${OCP_SUBRELEASE}.tar.gz 57 | curl -J -L -O https://mirror.openshift.com/pub/openshift-v4/clients/${OCP_RELEASE_PATH}/${OCP_SUBRELEASE}/openshift-install-linux-${OCP_SUBRELEASE}.tar.gz 58 | 59 | cd .. 60 | tree images 61 | } 62 | 63 | install_tools() { 64 | echo -e "NOTE: Tools used by $0 are not installed by this script. Manually install one of the following options:" 65 | echo -e "\n\t yum -y install tftp-server dnsmasq syslinux-tftpboot tree python36 jq oniguruma" 66 | 67 | echo "Downloading `filestranspiler`" 68 | curl -o ./utils/filetranspile https://raw.githubusercontent.com/ashcrow/filetranspiler/master/filetranspile 69 | chmod +x ./utils/filetranspile 70 | } 71 | 72 | mirror () { 73 | echo "Mirroring from Quay into Local Registry" 74 | # 4.2 75 | # Note: This option keep old metadata references to quay.io 76 | ./oc adm release mirror -a ${AIRGAP_SECRET_JSON} --insecure=true --from=quay.io/${UPSTREAM_REPO}/${RELEASE_NAME}:${OCP_SUBRELEASE} \ 77 | --to-release-image=${AIRGAP_REG}/${AIRGAP_REPO}:${OCP_SUBRELEASE} --to=${AIRGAP_REG}/${AIRGAP_REPO} 78 | 79 | # Unsupported procedure for OCP 4.1 80 | # echo "WARNING: This is an unsupported procedure" 81 | # ./oc adm release new -a ${AIRGAP_SECRET_JSON} --insecure --from-release=quay.io/${UPSTREAM_REPO}/ocp-release:${OCP_SUBRELEASE} \ 82 | # --mirror=${AIRGAP_REG}/${AIRGAP_REPO} --to-image=${AIRGAP_REG}/${AIRGAP_REPO}:${OCP_SUBRELEASE} 83 | 84 | # NOTE: When using the local `openshift-install` binary (it should not require the image override env variable) 85 | echo "Retrieving 'openshift-install' from local container repository" 86 | ./oc adm --insecure=true -a ${AIRGAP_SECRET_JSON} release extract --command='openshift-install' ${AIRGAP_REG}/${AIRGAP_REPO}:${OCP_SUBRELEASE} 87 | } 88 | 89 | clean() { 90 | echo "Removing installation folder" 91 | rm -fr ${POCDIR} 92 | } 93 | 94 | ignition() { 95 | echo "Creating and populating installation folder" 96 | mkdir ${POCDIR} 97 | cp ./install-config.yaml ${POCDIR} 98 | echo "Generating ignition files" 99 | ./openshift-install create ignition-configs --dir=${POCDIR} 100 | } 101 | 102 | customizations () { 103 | if [[ ! -f ./utils/filetranspile ]]; then 104 | echo "Missing customization tool. Downloading `filestranspiler`" 105 | curl -o ./utils/filetranspile https://raw.githubusercontent.com/ashcrow/filetranspiler/master/filetranspile 106 | chmod +x ./utils/filetranspile 107 | fi 108 | 109 | echo "Generate manifests to apply customizations" 110 | ./openshift-install create manifests --dir=${POCDIR} 111 | 112 | # workaround be applied to the initial ignition files 113 | cp -r ./customizations/*.yaml ./${POCDIR}/openshift/ 114 | 115 | echo "Generating new Ignition Configs" 116 | ./openshift-install create ignition-configs --dir=${POCDIR} 117 | 118 | # echo "Create backup of Ignition files to apply additional customizations" 119 | # mv ${POCDIR}/master.ign ${POCDIR}/master.ign-bkup 120 | # mv ${POCDIR}/worker.ign ${POCDIR}/worker.ign-bkup 121 | 122 | # echo "Updating Master and Workers Ignition files with NetworkManager patch" 123 | # jq -s '.[0] * .[1]' ${POCDIR}/master.ign-bkup ./utils/nm-patch.json > ${POCDIR}/master.ign 124 | # jq -s '.[0] * .[1]' ${POCDIR}/worker.ign-bkup ./utils/nm-patch.json > ${POCDIR}/worker.ign 125 | 126 | # echo "Updating Bootstrap Ignition file to apply NetworkManager patch" 127 | # mv ${POCDIR}/bootstrap.ign ${POCDIR}/bootstrap.ign-bkup 128 | #./utils/scripts/patch-systemd-units.py -i ./${POCDIR}/bootstrap.ign-bkup -p ./utils/nm-patch.json > ./${POCDIR}/bootstrap.ign 129 | 130 | # Check if there are additional configuration customizations for bootstrap.ign 131 | if [[ -d ./customizations/bootstrap ]]; then 132 | echo "Found bootstrap customization directory. Encoding additional configuration files into bootstrap.ign" 133 | mv ${POCDIR}/bootstrap.ign ${POCDIR}/bootstrap.ign-bkup 134 | ./utils/filetranspile -i ./${POCDIR}/bootstrap.ign-bkup -f ./customizations/bootstrap > ./${POCDIR}/bootstrap.ign 135 | fi 136 | 137 | # Check if there are additional configuration customizations for master.ign 138 | if [[ -d ./customizations/master ]]; then 139 | echo "Found master customization directory. Encoding additional configuration files into master.ign" 140 | mv ${POCDIR}/master.ign ${POCDIR}/master.ign-bkup 141 | ./utils/filetranspile -i ./${POCDIR}/master.ign-bkup -f ./customizations/master > ./${POCDIR}/master.ign 142 | fi 143 | 144 | # Check if there are additional configuration customizations for worker.ign 145 | if [[ -d ./customizations/worker ]]; then 146 | echo "Found worker customization directory. Encoding additional configuration files into worker.ign" 147 | mv ${POCDIR}/worker.ign ${POCDIR}/worker.ign-bkup 148 | ./utils/filetranspile -i ./${POCDIR}/worker.ign-bkup -f ./customizations/worker > ./${POCDIR}/worker.ign 149 | fi 150 | 151 | echo "Customizations done." 152 | echo "export KUBECONFIG=`pwd`/${POCDIR}/auth/kubeconfig" > ./set-env 153 | } 154 | 155 | prep_installer () { 156 | echo "Uncompressing installer and client binaries" 157 | tar -xzf ./images/openshift-client-linux-${OCP_SUBRELEASE}.tar.gz 158 | tar -xaf ./images/openshift-install-linux-${OCP_SUBRELEASE}.tar.gz 159 | } 160 | 161 | prep_images () { 162 | mkdir ${WEBROOT}/rhcos/ 163 | 164 | if [[ -f ./images/rhcos-metal.x86_64.raw.gz ]]; then 165 | echo "Copying RHCOS OS Image to ${WEBROOT}" 166 | cp -f ./images/rhcos-metal.x86_64.raw.gz ${WEBROOT}/rhcos/ 167 | fi 168 | 169 | echo "Copying RHCOS PXE Boot Images to ${WEBROOT}" 170 | cp -f ./images/rhcos-live-rootfs.x86_64.img ${WEBROOT}/rhcos/rhcos-live-rootfs 171 | cp -f ./images/rhcos-live-initramfs.x86_64.img ${WEBROOT}/rhcos/rhcos-initramfs.img 172 | cp -f ./images/rhcos-live-kernel-x86_64 ${WEBROOT}/rhcos/rhcos-kernel 173 | 174 | tree ${WEBROOT}/rhcos/ 175 | } 176 | 177 | prep_ign () { 178 | echo "Installing Ignition files into web path" 179 | mkdir ${WEBROOT}/ignition/ 180 | cp -f ${POCDIR}/*.ign ${WEBROOT}/ignition/ 181 | chmod +r ${WEBROOT}/ignition/*.ign 182 | tree ${WEBROOT}/ignition 183 | } 184 | 185 | bootstrap () { 186 | echo "Assuming PXE boot process in progress" 187 | ./openshift-install wait-for bootstrap-complete --dir=${POCDIR} --log-level debug 188 | echo "Enable cluster credentials: 'export KUBECONFIG=${POCDIR}/auth/kubeconfig'" 189 | export KUBECONFIG=${POCDIR}/auth/kubeconfig 190 | } 191 | 192 | install () { 193 | echo "Assuming PXE boot process in progress" 194 | ./openshift-install wait-for install-complete --dir=${POCDIR} --log-level debug 195 | } 196 | 197 | approve () { 198 | export KUBECONFIG=${POCDIR}/auth/kubeconfig 199 | ./oc get csr 200 | ./oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs ./oc adm certificate approve 201 | #sleep 3 202 | #./oc get csr 203 | } 204 | 205 | # Capture First param 206 | key="$1" 207 | 208 | case $key in 209 | tools) 210 | install_tools 211 | ;; 212 | get_images|images) 213 | get_images 214 | ;; 215 | mirror) 216 | mirror 217 | ;; 218 | clean) 219 | clean 220 | ;; 221 | ignition) 222 | ignition 223 | ;; 224 | custom|customizations) 225 | customizations 226 | ;; 227 | prep_ign) 228 | prep_ign 229 | ;; 230 | prep_installer) 231 | prep_installer 232 | ;; 233 | prep_images) 234 | prep_images 235 | ;; 236 | debug_bootstrap|bootstrap) 237 | bootstrap 238 | ;; 239 | debug_install|install) 240 | install 241 | ;; 242 | approve) 243 | approve 244 | ;; 245 | *) 246 | usage 247 | ;; 248 | esac 249 | 250 | ############################################################## 251 | # END OF FILE 252 | ############################################################## 253 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | Copyright 2014 Red Hat, Inc. 180 | 181 | Licensed under the Apache License, Version 2.0 (the "License"); 182 | you may not use this file except in compliance with the License. 183 | You may obtain a copy of the License at 184 | 185 | http://www.apache.org/licenses/LICENSE-2.0 186 | 187 | Unless required by applicable law or agreed to in writing, software 188 | distributed under the License is distributed on an "AS IS" BASIS, 189 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 190 | See the License for the specific language governing permissions and 191 | limitations under the License. 192 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # OpenShift 4.6 UPI bare-metal (using PXE boot) 2 | 3 | ```bash 4 | NOTE: THIS REPO IS NOT LONGER MAINTAINED OR UPDATED. IT IS HERE FOR HISTORICAL REFERNCES 5 | ``` 6 | 7 | > :heavy_exclamation_mark: *Red Hat does not provide commercial support for the content of this repo* 8 | 9 | ```bash 10 | ############################################################################## 11 | DISCLAIMER: THE CONTENT OF THIS REPO IS PROVIDED "AS-IS" 12 | 13 | THE CONTENT IS PROVIDED AS REFERENCE WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 16 | ############################################################################## 17 | ``` 18 | 19 | This is a reference documentation for POCs of OpenShift 4.6 UPI bare-metal deployment using PXE Boot. 20 | 21 | The initial deployment is as per the following diagram 22 | 23 | ![OCP Bootstrap Deployment](static/OCP_4x_bootstrap.png) 24 | 25 | ***NOTE:*** Even when the diagram shows two different load balancers, the access for the operations and administration tasks as well as the applications traffic are documented to go over the same load balancer. For a production grade deployment, the control and operation ports (K8s API, Machine Server, etc.) should not be exposed outside the organization. 26 | 27 | # TL;DR 28 | 29 | 1. Clone this repo to the Bastion Node 30 | 2. Edit the `install-config.yaml` to match your environment 31 | 3. Execute `poc.sh` script: 32 | ``` 33 | ./poc.sh clean 34 | 35 | ./poc.sh ignition 36 | 37 | ./poc.sh custom 38 | 39 | ./poc.sh prep_ign 40 | ``` 41 | 4. Power up the Bootstrap Node and PXE install the RHCOS 42 | 5. Power up the Master Nodes and PXE install the RHCOS 43 | ``` 44 | # Monitor bootstrap progress: 45 | ./poc.sh debug_bootstrap 46 | ``` 47 | 6. Once bootstrap complete, shutdown Bootstrap Node 48 | 7. Power up the Worker Nodes and PXE install 49 | ``` 50 | # Monitor OCP install 51 | ./poc.sh debug_install 52 | ``` 53 | 8. Monitor CSR requests from the Worker Nodes and accept the certificates 54 | 55 | NOTE: There are two CSR's per Node that need to be accepted. 56 | ``` 57 | $ export KUBECONFIG=./ocp4poc/auth/kubeconfig 58 | 59 | $ ./oc get csr 60 | 61 | $ ./oc adm certificate approve 62 | 63 | ``` 64 | 65 | 66 | # Reference Environment 67 | 68 | - Base Domain: example.com 69 | - Cluster Name: ocp4poc 70 | 71 | | NODE | IP ADDRESS | 72 | |:----------|:-------------| 73 | | bootstrap | 192.168.1.10 | 74 | | master-0 | 192.168.1.11 | 75 | | master-1 | 192.168.1.12 | 76 | | master-2 | 192.168.1.13 | 77 | | worker-0 | 192.168.1.15 | 78 | | worker-1 | 192.168.1.16 | 79 | | worker-2 | 192.168.1.17 | 80 | 81 | ***NOTE:*** For easy customization, clone this repo to your Bastion Node. 82 | 83 | Modify the following environment variables of `./poc.sh` script to match your environment. 84 | ``` 85 | OCP_RELEASE_PATH=ocp # valid options are "ocp" or "ocp-dev-preview" 86 | OCP_SUBRELEASE=4.6.0-rc.3 87 | 88 | RHCOS_RELEASE=pre-release # "4.5" for latest stable, "pre-release" for nightly 89 | 90 | WEBROOT=/opt/nginx/html 91 | TFTPROOT=/opt/dnsmasq # Using dnsmasq container as tftpserver, otherwise /tftpboot or /var/lib/tftpboot 92 | POCDIR=ocp4 93 | ``` 94 | ***NOTE:*** Next instructions assume this has been customized. 95 | 96 | ## Pre-requisites 97 | ### DNS Configuration 98 | - Example of [forward DNS records](utils/named-zone.conf) included in the `utils` folder 99 | - Example of [reverse records](utils/named-reverse-zone.conf) included in the `utils` folder 100 | - DNS must have entries for: 101 | - DNS forward records for all Nodes 102 | - DNS reverse records for all the Nodes 103 | - DNS entries for special records used by OCP: `etcd`, `etcd srv`, `api`, `api-int`, and `*.apps` wildcard subdomain 104 | 105 | Reference official [documentation](https://docs.openshift.com/container-platform/4.2/installing/installing_bare_metal/installing-bare-metal.html#installation-dns-user-infra_installing-bare-metal) for details on special entries required in DNS. 106 | 107 | ***NOTE:*** If there is NO way to reach out to the external NTP and DNS servers from the masters and workers node then you can set proxy DNS . DNS/NTP query in this will flow like this . 108 | - master node > bastion node > external DNS/NTP server 109 | - worker node > bastion node > external DNS/NTP server 110 | 111 | In this case no need to use bastion as recursive DNS, set port=0 in /etc/dnsmasq.conf file. 112 | 113 | ***NOTE:*** While setting up the external DNS server make sure the A records priorities for master and etcd are set properly . We want to have higher priority for master A records as compared to etcd A record . Otherwise what will happen is when we try to install master nodes ,during the reverse lookup step it might get the etcd FQDN instead of master FQDN!! 114 | 115 | ***NOTE:*** If using Microsoft external DNS server you can use CNAME for etcd FQDN's 116 | 117 | ### Load Balancer Configuration 118 | - Setup load balancerconfiguration in ***pass-through*** for Kubernetes API (`tcp/6443`), Machine Server Config (`tcp/22623`), OpenShift Routers HTTP and HTTPS (`tcp/80`, `tcp/443`) 119 | 120 | Reference Load Balancer configurations available in the `utils` folder (use one of the two): 121 | - Load balancer using [HAProxy](utils/haproxy.cfg) at system level (installed from RPM) 122 | - Load balancer using HAProxy as [System Container](utils/ocp-lb.service) managed by systemd 123 | 124 | NOTE: If seeing port bind errors starting the load balancer check SELinux settings: 125 | ``` 126 | # List the permited ports 127 | semanage port -l | grep http_port_t 128 | 129 | # If need to add ports 130 | semanage port -a -t http_port_t -p tcp 6443 131 | semanage port -a -t http_port_t -p tcp 22623 132 | semanage port -m -t http_port_t -p tcp 8000 133 | ``` 134 | 135 | # Preparing the Bastion Node 136 | 137 | - Install PXE Boot pre-requisites 138 | ``` 139 | yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm 140 | yum -y install tftp-server dnsmasq syslinux-tftpboot tree python36 jq oniguruma 141 | 142 | subscription-manager repos --enable rhel-7-server-extras-rpms 143 | yum -y install podman skopeo 144 | ``` 145 | 146 | - (optional) Setup HAProxy as load balancer 147 | - Update `./utils/haproxy.cfg` to match your environment 148 | ``` 149 | mkdir -pv /opt/haproxy 150 | 151 | cp ./utils/haproxy.cfg /opt/haproxy 152 | cp ./utils/ocp-lb.service /etc/systemd/system/ocp-lb.service 153 | 154 | podman pull haproxy 155 | 156 | systemctl daemon-reload 157 | 158 | systemctl start ocp-lb 159 | systemctl status ocp-lb 160 | systemctl enable ocp-lb 161 | ``` 162 | 163 | Note: To accept asymmetrically routed packets set rp_filter = 2 (Credits: Thanks to Jay Cromer) 164 | ``` 165 | echo "net.ipv4.conf.default.rp_filter = 2" >> /etc/sysctl.conf 166 | echo "net.ipv4.conf.all.rp_filter = 2" >> /etc/sysctl.conf 167 | 168 | echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter 169 | echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter 170 | ``` 171 | 172 | - Setup web server for Ignition and PXE files 173 | ``` 174 | mkdir -pv /opt/nginx/html/metal 175 | 176 | cp ./utils/ocp-http.service /etc/systemd/system/ocp-http.service 177 | 178 | podman pull nginx 179 | 180 | systemctl daemon-reload 181 | 182 | systemctl start ocp-http 183 | systemctl status ocp-http 184 | systemctl enable ocp-http 185 | ``` 186 | 187 | ## Setting up DNSmasq for PXE Boot 188 | 189 | 1. (if already running Bind server in Bastion) Disable DNS in DNSmasq by setting `port=0` 190 | ``` 191 | vi /etc/dnsmasq.conf 192 | ... 193 | port=0 194 | ... 195 | ``` 196 | 2. Configure DHCP and DHCP PXE Options following the reference [dnsmasq-pxe.conf](utils/dnsmasq-pxe.conf) 197 | ``` 198 | cp ./utils/dnsmasq-pxe.conf /etc/dnsmasq.d/dnsmasq-pxe.conf 199 | ``` 200 | NOTE: Update `/etc/dnsmasq.d/dnsmasq-pxe.conf` to match environment 201 | 202 | ## Setup PXE Boot Configurations 203 | 204 | 1. Create PXE Boot menu to be used by the environment [/var/lib/tftpboot/pxelinux.cfg/default](utils/PXE/pxelinux.cfg-default-bios) 205 | 206 | ``` 207 | mkdir -pv /var/lib/tftpboot/pxelinux.cfg/ 208 | 209 | copy ./utils/pxelinux.cfg-default-bios /var/lib/tftpboot/pxelinux.cfg/default 210 | ``` 211 | NOTE: Update `/var/lib/tftpboot/pxelinux.cfg/default` to match environment. 212 | 213 | 214 | 2. Download RHCOS images. 215 | 216 | - Running `./poc.sh get_images` download all the images to `./images` on your current directory. It should be similar to this list (versions may be different): 217 | 218 | ``` 219 | images/ 220 | ├── openshift-client-linux-4.6.0-rc.3.tar.gz 221 | ├── openshift-install-linux-4.6.0-rc.3.tar.gz 222 | ├── rhcos-live-initramfs.x86_64.img 223 | ├── rhcos-live-kernel-x86_64 224 | ├── rhcos-live-rootfs.x86_64.img 225 | └── rhcos-live.x86_64.iso 226 | ``` 227 | 228 | 3. Open the `openshift-client-linux-.tar.gz` and the `openshift-install-linux-.tar.gz` into your current directory. This will provide the `openshift-installer`, `oc` and `kubectl` binaries. 229 | 230 | 4. Copy RHCOS PXE images and RHCOS images into the corresponding folders 231 | 232 | ``` 233 | ./poc.sh prep_images 234 | ``` 235 | 236 | 5. Uncompress installer and client binaries into current directory 237 | 238 | ``` 239 | ./poc.sh prep_installer 240 | ``` 241 | 242 | ## Ports to open if Bastion node running all the ancillary services 243 | 244 | ``` 245 | firewall-cmd --zone=public --change-interface=eth0 246 | firewall-cmd --zone=trusted --change-interface=eth1 247 | 248 | firewall-cmd --get-active-zones 249 | 250 | firewall-cmd --zone=public --permanent --add-port=6443/tcp 251 | firewall-cmd --zone=public --permanent --add-port=22623/tcp 252 | firewall-cmd --zone=public --permanent --add-service=http 253 | firewall-cmd --zone=public --permanent --add-service=https 254 | firewall-cmd --zone=public --permanent --add-service=dns 255 | 256 | firewall-cmd --reload 257 | 258 | firewall-cmd --zone=public --list-services 259 | firewall-cmd --zone=public --list-ports 260 | 261 | ``` 262 | 263 | # INSTALLATION 264 | 265 | - Create or edit `install-config.yaml` to include the pull secret obtained from [https://try.openshift.com](https://try.openshift.com) 266 | 267 | - Add the SSH Key to be used to access the Bootstrap Node to the `install-config.yaml` 268 | 269 | - Generate the Ignition files: 270 | 271 | `./poc.sh ignition` 272 | 273 | - Apply custom patch for NetworkManager: 274 | 275 | `./poc.sh custom` 276 | 277 | - Copy Ignition files into web server path 278 | 279 | `./poc.sh prep_ign` 280 | 281 | - Boot the Bootstrap Node and at the PXE MENU select the `BOOTSTRAP` option and press `Enter` 282 | 283 | - This will start the process of installing RHCOS in the Bootstrap Node 284 | 285 | ![Bootstrap Menu](static/pxe-boot-bootstrap.png) 286 | 287 | - Wait until the bootstrap Node is up and showing the login prompt 288 | 289 | (missing image here) 290 | 291 | - Monitor bootstrap progress using the script or the equivalent `openshift-install wait-for ...` command 292 | 293 | `./poc.sh bootstrap` 294 | 295 | - Start the Master Nodes and select the `MASTER` option from the PXE menu 296 | 297 | ![Bootstrap Menu](static/pxe-boot-master.png) 298 | 299 | - The boot process of the Masters will start the RHCOS installation and use the Ignition configuration during the first install. 300 | - After RHCOS is installed in the Node, it will be up for a few minutes showing the login prompt and eventually will reboot. This is a normal part of the automatic upgrade process of RHCOS. 301 | - The Master node receives the actual configuration as a rendered Machine Config from the Bootstrap Nodes. 302 | - During the process, the Cluster Version Operator instruct the Master Nodes to start deploying the components corresponding to the Master Node. 303 | - One of the actions is the upgrade of RHCOS so the Node will download latest RHCOS version and will apply it. 304 | - After upgrading the OS, the Nodes reboot into the latest version of RHCOS 305 | - During the process, the Cluster Version Operator instruct the Master Nodes to start deploying the components corresponding to the Master Node. 306 | 307 | - Once the Bootstrap Node reports the bootstrap process as completed and indicates it is safe to shutdown the Bootstrap Node, proceed to shutdown the Node. 308 | - At this point the etcd have achieved quorum and the Master Nodes are fully operational 309 | 310 | (missing image here) 311 | 312 | - Proceed to monitor the Installation process using the script or the equivalent `openshift-install wait-for ...` command 313 | 314 | `./poc install` 315 | 316 | - Proceed to boot the Worker Nodes and select the `WORKER` option fromt he PXE menu 317 | 318 | ![Bootstrap Menu](static/pxe-boot-worker.png) 319 | 320 | - After RHCOS is installed in the Worker node, it will go over a similar process as with the Master Nodes but this time, there will be a Certificate Signing Request (CSR) that need to be accepted before it proceeds. 321 | - The script provides `./poc approve` to automattically approve any pending certificate. NOTE: In a production environment, the certificates should be approved ONLY after confirming the CSR is from a valid and authorized Worker Node. 322 | 323 | (missing image here) 324 | 325 | - A successful installation will show all the cluster operators available. The `image-registry` will not become active until a cluster administrator configure storage for the registry. 326 | 327 | ``` 328 | # export KUBECONFIG=`pwd`/ocp4poc/auth/kubeconfig 329 | 330 | # oc get co 331 | NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE 332 | authentication 4.6.0-rc.3 True False False 8m42s 333 | cloud-credential 4.6.0-rc.3 True False False 69m 334 | cluster-autoscaler 4.6.0-rc.3 True False False 63m 335 | config-operator 4.6.0-rc.3 True False False 65m 336 | console 4.6.0-rc.3 True False False 38m 337 | csi-snapshot-controller 4.6.0-rc.3 True False False 64m 338 | dns 4.6.0-rc.3 True False False 63m 339 | etcd 4.6.0-rc.3 True False False 63m 340 | image-registry 4.6.0-rc.3 True False False 45m 341 | ingress 4.6.0-rc.3 True False False 42m 342 | insights 4.6.0-rc.3 True False False 65m 343 | kube-apiserver 4.6.0-rc.3 True False False 62m 344 | kube-controller-manager 4.6.0-rc.3 True False False 62m 345 | kube-scheduler 4.6.0-rc.3 True False False 62m 346 | kube-storage-version-migrator 4.6.0-rc.3 True False False 16m 347 | machine-api 4.6.0-rc.3 True False False 63m 348 | machine-approver 4.6.0-rc.3 True False False 63m 349 | machine-config 4.6.0-rc.3 True False False 63m 350 | marketplace 4.6.0-rc.3 True False False 63m 351 | monitoring 4.6.0-rc.3 True False False 41m 352 | network 4.6.0-rc.3 True False False 65m 353 | node-tuning 4.6.0-rc.3 True False False 65m 354 | openshift-apiserver 4.6.0-rc.3 True False False 44m 355 | openshift-controller-manager 4.6.0-rc.3 True False False 63m 356 | openshift-samples 4.6.0-rc.3 True False False 44m 357 | operator-lifecycle-manager 4.6.0-rc.3 True False False 64m 358 | operator-lifecycle-manager-catalog 4.6.0-rc.3 True False False 64m 359 | operator-lifecycle-manager-packageserver 4.6.0-rc.3 True False False 42m 360 | service-ca 4.6.0-rc.3 True False False 64m 361 | storage 4.6.0-rc.3 True False False 65m 362 | ``` 363 | 364 | 365 | - Adding ephemeral storaste to the image registry can be done with the following command: 366 | - ***NOTE:*** ONLY USE THIS TYPE OF STORAGE FOR TESTING 367 | 368 | ``` 369 | oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}' 370 | ``` 371 | 372 | # Assigning persistent storage for image-registry 373 | 374 | - Set the PVC name for the image registry 375 | ``` 376 | oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"pvc":{"claim":"{changeme}"}}}}' 377 | ``` 378 | 379 | - For dynamic PVC creation by the operator edit the configuration and delete the name of the claim 380 | ``` 381 | oc edit configs.imageregistry.operator.openshift.io 382 | 383 | # Remove the 'changeme' (including the '') and leave 'claim:' only. 384 | ... 385 | storage: 386 | pvc: 387 | claim: 388 | ... 389 | ``` 390 | 391 | - Create PV for the image registry. 392 | 393 | Example: `pv-image-registry.yaml` 394 | ``` 395 | apiVersion: v1 396 | kind: PersistentVolume 397 | metadata: 398 | name: pv-image-registry 399 | spec: 400 | capacity: 401 | storage: 100Gi 402 | accessModes: 403 | - ReadWriteMany 404 | nfs: 405 | path: /path/to/nfs/image-registry 406 | server: 192.168.88.7 407 | persistentVolumeReclaimPolicy: Retain 408 | ``` 409 | --------------------------------------------------------------------------------