├── .gitignore ├── README.md ├── k3s ├── .terraform.lock.hcl ├── identity.tf ├── load_balancer.tf ├── network_security.tf ├── networking.tf ├── nodes.tf ├── templates │ └── cloudinit.tftpl ├── terraform.tfvars.example ├── variables.tf └── versions.tf └── oke ├── oke.tf ├── outputs.tf ├── terraform.tfvars.example ├── variables.tf └── versions.tf /.gitignore: -------------------------------------------------------------------------------- 1 | .terraform/ 2 | generated/ 3 | env.sh 4 | backend.tf 5 | terraform.tfvars 6 | terraform.tfstate* 7 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Kubernetes using Oracle Cloud Always Free resources 2 | 3 | This repository contains two methods of getting a working Kubernetes cluster on Oracle Cloud using their Always Free resources. 4 | 5 | 1. Using their managed Kubernetes service: Oracle Container Engine for Kubernetes (OKE) 6 | 2. Using k3s with a highly available (embedded etcd) control plane 7 | 8 | In both cases the nodes will be of the Ampere ARM64 variety, because you get 4 cores and 24GB RAM worth of nodes this way. You only get a couple of 1 core 1GB RAM nodes using x86 processors. 9 | 10 | I make no promises to keep this repo up to date, but it should serve as a good example of how to get started with Kubernetes on Oracle Cloud Infrastructure. 11 | 12 | ## Creating the cluster 13 | 14 | ### Initial configuration 15 | 16 | You will need an account on [Oracle Cloud](https://cloud.oracle.com). You'll be given some credits (£250 in the UK) initially but you shouldn't see them being used by anything in this repository; everything should be covered by the [Always Free resources](https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier_topic-Always_Free_Resources.htm) and OKE is free anyway. 17 | 18 | When your account is activated you'll need to install and configure the [OCI CLI](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/cliconcepts.htm) get some details to populate the `terraform.tfvars` file, which are used to authentiate to your account. The information of how to get these details are in [Oracle's Documentation](https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm#configuring_the_terraform_provider). 19 | 20 | At the end of the trial period you will need to convert your account to a paid account rather than leaving it as a free tier account. This is because whilst OKE is free, it is not supported (yet) on a free account. [The documentation](https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier.htm) also says that the Ampere instances will be terminated and need reprovisioning, although I didn't encounter this. 21 | 22 | I recommend you setup [Budgets](https://docs.oracle.com/en-us/iaas/Content/Billing/Concepts/budgetsoverview.htm) on your account to ensure you get alerted if you are running up a bill. Unfortunately I am currently being billed £0.05 a day for the boot volume of one of my nodes, but I can live with that. This might be down to the many things I've tried on my account and I'd be interested to hear if others also get the same issue. 23 | 24 | ### Applying the Terraform 25 | 26 | With the account created and `terraform.tfvars` file populated, creating your cluster should then be as simple as opening a terminal in the relevant directory and running `terraform init` followed by `terraform apply`. 27 | 28 | If you want to use [Remote State](https://www.terraform.io/docs/language/state/remote.html) for your Terraform state file you will need to perform additional configuration. I am storing mine in OCI Object Storage using a Pre Authenticated Request URL, using the method in [this medium post](https://medium.com/oracledevs/storing-terraform-remote-state-to-oracle-cloud-infrastructure-object-storage-b32fe7402781). You could also use Oracle Cloud's [Resource Manager](https://docs.oracle.com/en-us/iaas/Content/ResourceManager/Concepts/resourcemanager.htm) managed Terraform service. 29 | 30 | ## Connecting to your Kubernetes Cluster 31 | ### OKE 32 | 33 | The Terraform creates a `generated` directory containing your kubeconfig file. You can either use this where it is by running `export KUBECONFIG=/path/to/generated/kubeconfig` or copy it to the default path used by `kubectl` at `~/.kube/config`. 34 | 35 | At this point you will need to make sure you have the [OCI CLI](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/cliconcepts.htm) installed, as that is used for authentication to your new cluster. 36 | 37 | You should now be able to run commands like `kubectl get no` and see you have a cluster up and running with two nodes. 38 | 39 | ### k3s 40 | 41 | You will need to use Oracle managed bastion service (already configured by the terraform) to SSH into one of the control plane nodes. The username will be `opc`. You will find a kubeconfig file in `/etc/rancher/k3s/k3s.yaml` but for it to work locally you will need to change the IP address to that of your Load Balancer using https on port 6443. 42 | 43 | You will see there are 4 nodes; 3 control plane and 1 worker. All with 1 CPU core and 6 GB RAM. 44 | -------------------------------------------------------------------------------- /k3s/.terraform.lock.hcl: -------------------------------------------------------------------------------- 1 | # This file is maintained automatically by "terraform init". 2 | # Manual edits may be lost in future updates. 3 | 4 | provider "registry.terraform.io/hashicorp/cloudinit" { 5 | version = "2.2.0" 6 | hashes = [ 7 | "h1:CUOSIT4XzkJXEpPp4dBL+rxxLgqamwiMvuhBVNHC7AI=", 8 | "h1:Id6dDkpuSSLbGPTdbw49bVS/7XXHu/+d7CJoGDqtk5g=", 9 | "h1:jjiYQ9lHpy5Ca9GoWbsXtuDr2HLgDQY8my1gIrp1lSo=", 10 | "h1:siiI0wK6/jUDdA5P8ifTO0yc9YmXHml4hz5K9I9N+MA=", 11 | "h1:tQLNREqesrdCQ/bIJnl0+yUK+XfdWzAG0wo4lp10LvM=", 12 | "zh:76825122171f9ea2287fd27e23e80a7eb482f6491a4f41a096d77b666896ee96", 13 | "zh:795a36dee548e30ca9c9d474af9ad6d29290e0a9816154ad38d55381cd0ab12d", 14 | "zh:9200f02cb917fb99e44b40a68936fd60d338e4d30a718b7e2e48024a795a61b9", 15 | "zh:a33cf255dc670c20678063aa84218e2c1b7a67d557f480d8ec0f68bc428ed472", 16 | "zh:ba3c1b2cd0879286c1f531862c027ec04783ece81de67c9a3b97076f1ce7f58f", 17 | "zh:bd575456394428a1a02191d2e46af0c00e41fd4f28cfe117d57b6aeb5154a0fb", 18 | "zh:c68dd1db83d8437c36c92dc3fc11d71ced9def3483dd28c45f8640cfcd59de9a", 19 | "zh:cbfe34a90852ed03cc074601527bb580a648127255c08589bc3ef4bf4f2e7e0c", 20 | "zh:d6ffd7398c6d1f359b96f5b757e77b99b339fbb91df1b96ac974fe71bc87695c", 21 | "zh:d9c15285f847d7a52df59e044184fb3ba1b7679fd0386291ed183782683d9517", 22 | "zh:f7dd02f6d36844da23c9a27bb084503812c29c1aec4aba97237fec16860fdc8c", 23 | ] 24 | } 25 | 26 | provider "registry.terraform.io/hashicorp/random" { 27 | version = "3.1.3" 28 | hashes = [ 29 | "h1:7+wnAXQM7IpNEAQ6WZXdO0ZfQW/ncQFXYJ5T2KaR+Z8=", 30 | "h1:KkBerDx4KhflLuwYJ/dl3zOTBY0qi4sdTj4nvwPIS7g=", 31 | "h1:LPSVX+oXKGaZmxgtaPf2USxoEsWK/pnhmm/5FKw+PtU=", 32 | "h1:d4M8bOY9r99suD5EYfdZUvbhtq6hzCHa2SeY+T+IRlA=", 33 | "h1:nLWniS8xhb32qRQy+n4bDPjQ7YWZPVMR3v1vSrx7QyY=", 34 | "zh:26e07aa32e403303fc212a4367b4d67188ac965c37a9812e07acee1470687a73", 35 | "zh:27386f48e9c9d849fbb5a8828d461fde35e71f6b6c9fc235bc4ae8403eb9c92d", 36 | "zh:5f4edda4c94240297bbd9b83618fd362348cadf6bf24ea65ea0e1844d7ccedc0", 37 | "zh:646313a907126cd5e69f6a9fafe816e9154fccdc04541e06fed02bb3a8fa2d2e", 38 | "zh:7349692932a5d462f8dee1500ab60401594dddb94e9aa6bf6c4c0bd53e91bbb8", 39 | "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", 40 | "zh:9034daba8d9b32b35930d168f363af04cecb153d5849a7e4a5966c97c5dc956e", 41 | "zh:bb81dfca59ef5f949ef39f19ea4f4de25479907abc28cdaa36d12ecd7c0a9699", 42 | "zh:bcf7806b99b4c248439ae02c8e21f77aff9fadbc019ce619b929eef09d1221bb", 43 | "zh:d708e14d169e61f326535dd08eecd3811cd4942555a6f8efabc37dbff9c6fc61", 44 | "zh:dc294e19a46e1cefb9e557a7b789c8dd8f319beca99b8c265181bc633dc434cc", 45 | "zh:f9d758ee53c55dc016dd736427b6b0c3c8eb4d0dbbc785b6a3579b0ffedd9e42", 46 | ] 47 | } 48 | 49 | provider "registry.terraform.io/oracle/oci" { 50 | version = "4.75.0" 51 | constraints = ">= 4.67.3" 52 | hashes = [ 53 | "h1:7mA4N3Snal+MjgATEm2NCjiVlLYYAaNMFRn7+IYF3KE=", 54 | "h1:RlYetIy/lQQ4uL3dfOY1Kq5MsRL86xxWyhPqp520fqs=", 55 | "h1:RsrBRtygE17vqHmrclCu6vwggQta5nydw/S7Tt0SKZ0=", 56 | "h1:fhuVWZ0a26hMLIOh2pKc78yR7ML/dr0ZPTTDnq56uhM=", 57 | "h1:lV4Z/CuFOVjyivvAyZEJBAmFVLIm153a3Zi49lRHjKU=", 58 | "zh:0390dd200ab50344f1014306dc186c21a8fb8fd881585b3a8299b1261ad41ca4", 59 | "zh:0f46b164faaad0325006f3fdab4b699f1b26203e46cf9df33891ab483455c2ce", 60 | "zh:0fb4c9ae86b24c11369bb8097a4461cde5f4f69736871c0c9d3401a2791fd402", 61 | "zh:17bc369131353acdbce7dd132bac1caf44ec9a7ad5eb66dc33a1cf7c48eaac0e", 62 | "zh:2cb11c8658480240e6f60ab6cc84b748bf76e08a821fc73a493a052bf8e17183", 63 | "zh:3ae19ec595635b7fd5046669e1b0112db3bc7f378d6f8132955963a623520d20", 64 | "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425", 65 | "zh:b342b03ef66c754d86e1296dbbed533b660bc477371b5ca0918eb06704503314", 66 | "zh:b49b11491c9e4be5417f73766d05afbac0901659196aa03147db0a86eedae1f3", 67 | "zh:c093376cb83d66d2c46f7f83534461728d2c8a9aff25df80bbd3826da09dd4bd", 68 | "zh:cc7fa18d38f32bd7a49dc457d1b5131ded0ca402764594d88bbb288d52e25e7e", 69 | "zh:d0da468726410393fe9546ee23eae7056ca9a517d5ccf90baf704f4ec49f621a", 70 | "zh:d79f133c6add2bc9a901d6941bd6b87c6d0df07eb38eed4174625697e6283dd4", 71 | "zh:d90c8340b74defee4c8d86df230e4135cfa08ac6339cc58d92c6a833a9a32070", 72 | "zh:da871ab255a31ebfb7a97489cf84411df8c48968f97b8048930f5a52fb3914a0", 73 | ] 74 | } 75 | -------------------------------------------------------------------------------- /k3s/identity.tf: -------------------------------------------------------------------------------- 1 | resource "oci_identity_compartment" "kubernetes" { 2 | compartment_id = var.tenancy_ocid 3 | description = "Compartment for Terraform resources." 4 | name = var.environment_name 5 | enable_delete = true 6 | freeform_tags = local.freeform_tags 7 | } 8 | 9 | resource "oci_identity_tag_namespace" "security" { 10 | compartment_id = oci_identity_compartment.kubernetes.id 11 | description = "Tags used to assign identity policies via dynamic groups" 12 | name = "Security" 13 | } 14 | 15 | resource "oci_identity_tag" "instance_group" { 16 | description = "Used to define the group of instances to determine dynamic group membership" 17 | name = "Instance-Group" 18 | tag_namespace_id = oci_identity_tag_namespace.security.id 19 | } 20 | 21 | resource "oci_identity_dynamic_group" "kubernetes_control_plane" { 22 | compartment_id = var.tenancy_ocid 23 | description = "Dynamic group which contains instances tagged as Kubernetes Control Plane nodes" 24 | matching_rule = < /etc/resolv-k3s.conf 7 | echo "nameserver 1.0.0.1" >> /etc/resolv-k3s.conf 8 | 9 | # Disable and stop firewalld as per k3s docs and other memory hogs we don't need 10 | systemctl disable firewalld --now 11 | systemctl disable sssd --now 12 | systemctl disable tuned --now 13 | 14 | # Install htop (because I like it...) and upgrade everything 15 | dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm 16 | dnf upgrade -y 17 | dnf install -y htop python36-oci-cli 18 | 19 | # Install K3S 20 | export INSTALL_K3S_VERSION="${k3s_version}" 21 | export K3S_TOKEN="${k3s_token}" 22 | 23 | %{ if node_role == "control-plane" } 24 | # Credit to Lorenzo Garuti for script logic: https://github.com/garutilorenzo/k3s-aws-terraform-cluster 25 | # Find out OCID of the first node to be created by the instance-pool and whether it is us. Winner gets to be first control plane node. 26 | export OCI_CLI_AUTH=instance_principal 27 | first_instance=$(oci compute-management instance-pool list-instances --compartment-id $(oci-metadata --value-only -g compartmentId) --instance-pool-id $(oci-metadata --value-only -g instancePoolId) --sort-by TIMECREATED --sort-order ASC | jq -r .data[0].id) 28 | instance_id=$(oci-metadata -g id --value-only) 29 | 30 | if [[ "$first_instance" == "$instance_id" ]]; then 31 | echo "I'm the first! Woohoo! Cluster init!" 32 | export INSTALL_K3S_EXEC="server --disable traefik --disable local-storage --disable-cloud-controller --resolv-conf /etc/resolv-k3s.conf --cluster-init --kubelet-arg kube-reserved=cpu=50m,memory=1536Mi,ephemeral-storage=1Gi --kubelet-arg system-reserved=cpu=10m,memory=640Mi,ephemeral-storage=1Gi" 33 | else 34 | echo "Someone else got there first. Cluster join..." 35 | export INSTALL_K3S_EXEC="server --disable traefik --disable local-storage --disable-cloud-controller --resolv-conf /etc/resolv-k3s.conf --server https://${lb_ip_address}:6443 --kubelet-arg kube-reserved=cpu=50m,memory=1536Mi,ephemeral-storage=1Gi --kubelet-arg system-reserved=cpu=10m,memory=640Mi,ephemeral-storage=1Gi" 36 | fi 37 | %{ else } 38 | export INSTALL_K3S_EXEC="agent --resolv-conf /etc/resolv-k3s.conf --server https://${lb_ip_address}:6443 --kubelet-arg kube-reserved=cpu=10m,memory=160Mi,ephemeral-storage=1Gi --kubelet-arg system-reserved=cpu=10m,memory=640Mi,ephemeral-storage=1Gi" 39 | %{ endif } 40 | 41 | until (curl -sfL https://get.k3s.io | sh -); do 42 | echo 'k3s did not install correctly, retrying' 43 | sleep 2 44 | done 45 | -------------------------------------------------------------------------------- /k3s/terraform.tfvars.example: -------------------------------------------------------------------------------- 1 | tenancy_ocid = "ocid1.tenancy.oc1..stuff" 2 | region = "uk-london-1" 3 | ssh_public_key_path = "~/.ssh/id_rsa.pub" 4 | -------------------------------------------------------------------------------- /k3s/variables.tf: -------------------------------------------------------------------------------- 1 | variable "ssh_public_key_path" { 2 | description = "The filesystem path of an SSH public key to authorised for host authentication" 3 | default = "~/.ssh/id_rsa.pub" 4 | } 5 | 6 | variable "region" { 7 | description = "The OCI region we're using" 8 | default = "uk-london-1" 9 | } 10 | 11 | variable "tenancy_ocid" { 12 | description = "The OCID of the parent tenancy in which we're creating a compartment" 13 | } 14 | 15 | variable "environment_name" { 16 | description = "The name to give the OCI compartment" 17 | default = "kubernetes" 18 | } 19 | 20 | variable "cidr_block" { 21 | description = "The CIDR /16 block to use for the VCN" 22 | default = "10.200.0.0" 23 | } 24 | 25 | variable "k3s_version" { 26 | description = "The GitHub tag to use for k3s installation" 27 | default = "v1.23.6+k3s1" 28 | } 29 | 30 | locals { 31 | freeform_tags = { 32 | "provisioner" = "terraform" 33 | "environment" = var.environment_name 34 | } 35 | resource_name = "${var.environment_name}-control-plane" 36 | vcn_cidr = "${var.cidr_block}/16" 37 | private_cidr = cidrsubnet(local.vcn_cidr, 2, 0) 38 | public_cidr = cidrsubnet(local.vcn_cidr, 2, 1) 39 | } 40 | -------------------------------------------------------------------------------- /k3s/versions.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | oci = { 4 | source = "oracle/oci" 5 | } 6 | } 7 | required_version = ">= 1.1" 8 | } 9 | -------------------------------------------------------------------------------- /oke/oke.tf: -------------------------------------------------------------------------------- 1 | resource "oci_identity_compartment" "oke" { 2 | compartment_id = var.tenancy_ocid 3 | description = "Compartment for Terraform resources." 4 | name = var.compartment_name 5 | enable_delete = true 6 | } 7 | 8 | data "oci_core_images" "bastion" { 9 | compartment_id = oci_identity_compartment.oke.id 10 | operating_system = "Oracle Linux" 11 | operating_system_version = "8" 12 | shape = "VM.Standard.E2.1.Micro" 13 | sort_by = "TIMECREATED" 14 | sort_order = "DESC" 15 | } 16 | 17 | module "oke" { 18 | source = "oracle-terraform-modules/oke/oci" 19 | 20 | compartment_id = oci_identity_compartment.oke.id 21 | 22 | region = var.region 23 | home_region = var.region 24 | tenancy_id = var.tenancy_ocid 25 | ssh_public_key_path = var.ssh_public_key_path 26 | create_operator = false 27 | bastion_shape = { 28 | shape = "VM.Standard.E2.1.Micro", 29 | } 30 | bastion_image_id = data.oci_core_images.bastion.images.0.id 31 | allow_worker_ssh_access = true 32 | control_plane_allowed_cidrs = ["0.0.0.0/0"] 33 | kubernetes_version = "v1.22.5" 34 | subnets = { 35 | bastion = { netnum = 0, newbits = 13 } 36 | operator = { netnum = 1, newbits = 13 } 37 | cp = { netnum = 2, newbits = 13 } 38 | int_lb = { netnum = 16, newbits = 11 } 39 | pub_lb = { netnum = 17, newbits = 11 } 40 | workers = { netnum = 1, newbits = 2 } 41 | fss = { netnum = 18, newbits = 11 } 42 | } 43 | node_pools = { 44 | np1 = { 45 | boot_volume_size = 50 46 | node_pool_size = 2 47 | ocpus = 2 48 | shape = "VM.Standard.A1.Flex" 49 | memory = 12 50 | } 51 | } 52 | providers = { 53 | oci.home = oci 54 | } 55 | } 56 | -------------------------------------------------------------------------------- /oke/outputs.tf: -------------------------------------------------------------------------------- 1 | output "kubeconfig" { value = module.oke.kubeconfig } 2 | 3 | -------------------------------------------------------------------------------- /oke/terraform.tfvars.example: -------------------------------------------------------------------------------- 1 | tenancy_ocid = "ocid1.tenancy.oc1..stuff" 2 | region = "uk-london-1" 3 | ssh_public_key_path = "~/.ssh/id_rsa.pub" 4 | -------------------------------------------------------------------------------- /oke/variables.tf: -------------------------------------------------------------------------------- 1 | variable "ssh_public_key_path" { 2 | description = "The filesystem path of an SSH public key to authorised for host authentication" 3 | default = "~/.ssh/id_rsa.pub" 4 | } 5 | 6 | variable "region" { 7 | description = "The OCI region we're using" 8 | default = "uk-london-1" 9 | } 10 | 11 | variable "tenancy_ocid" { 12 | description = "The OCID of the parent tenancy in which we're creating a compartment" 13 | } 14 | 15 | variable "compartment_name" { 16 | default = "kubernetes" 17 | } 18 | 19 | -------------------------------------------------------------------------------- /oke/versions.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | oci = { 4 | source = "oracle/oci" 5 | } 6 | } 7 | required_version = ">= 1.1" 8 | } 9 | --------------------------------------------------------------------------------