├── .gitignore ├── CONTRIBUTING.md ├── README.md ├── hooks.tf ├── main.tf ├── outputs.tf ├── scripts ├── boot-master │ ├── copy_cluster_skel.sh │ ├── functions.sh │ ├── generate_hostsfiles.sh │ ├── get-args.sh │ ├── install-docker.sh │ ├── load-config.py │ ├── load-image.sh │ ├── parse-hostgroups.py │ ├── scaleworkers.sh │ └── start_install.sh └── common │ ├── docker-user.sh │ ├── prereqs.sh │ └── version-specific.sh ├── tests ├── README.md ├── container-runtest.sh ├── helpers.bash ├── install-docker.bats ├── load-image.bats ├── run_bats.sh └── terraform-sanity.bats └── variables.tf /.gitignore: -------------------------------------------------------------------------------- 1 | # Local .terraform directories 2 | **/.terraform/* 3 | 4 | # .tfstate files 5 | *.tfstate 6 | *.tfstate.* 7 | 8 | # .tfvars files 9 | *.tfvars 10 | 11 | # generated key files 12 | *.pem 13 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | ## Contributing to IBM Cloud Architecture reference applications 2 | Anyone can contribute to IBM Cloud Architecture reference applications and their associated projects, whether you are an IBMer or not. 3 | We welcome your collaboration & contributions happily, as our reference applications are meant to reflect your real world scenarios. 4 | There are multiple ways to contribute: report bugs and improvement suggestions, improve documentation, and contribute code. 5 | 6 | 7 | ## Bug reports, documentation changes, and feature requests 8 | 9 | If you would like to contribute your experience with an IBM Cloud Architecture project back to the project in the form of encountered bug reports, necessary documentation changes, or new feature requests, this can be done through the use of the repository's [**Issues**](#) list. 10 | 11 | Before opening a new issue, please reference the existing list to make sure a similar or duplicate item does not already exist. Otherwise, please be as explicit as possible when creating the new item and be sure to include the following: 12 | 13 | - **Bug reports** 14 | - Specific Project Version 15 | - Deployment environment 16 | - A minimal, but complete, setup of steps to recreate the problem 17 | - **Documentation changes** 18 | - URL to existing incorrect or incomplete documentation (either in the project's GitHub repo or external product documentation) 19 | - Updates required to correct current inconsistency 20 | - If possible, a link to a project fork, sample, or workflow to expose the gap in documentation. 21 | - **Feature requests** 22 | - Complete description of project feature request, including but not limited to, components of the existing project that are impacted, as well as additional components that may need to be created. 23 | - A minimal, but complete, setup of steps to recreate environment necessary to identify the new feature's current gap. 24 | 25 | The more explicit and thorough you are in opening GitHub Issues, the more efficient your interaction with the maintainers will be. When creating the GitHub Issue for your bug report, documentation change, or feature request, be sure to add as many relevant labels as necessary (that are defined for that specific project). These will vary by project, but will be helpful to the maintainers in quickly triaging your new GitHub issues. 26 | 27 | ## Code contributions 28 | 29 | We really value contributions, and to maximize the impact of code contributions, we request that any contributions follow the guidelines below. If you are new to open source contribution and would like some more pointers or guidance, you may want to check out [**Your First PR**](http://yourfirstpr.github.io/) and [**First Timers Only**](https://www.firsttimersonly.com/). These are a few projects that help on-board new contributors to the overall process. 30 | 31 | ### Coding and Pull Requests best practices 32 | - Please ensure you follow the coding standard and code formatting used throughout the existing code base. 33 | - This may vary project by project, but any specific diversion from normal language standards will be explicitly noted. 34 | - One feature / bug fix / documentation update per pull request 35 | - Always pull the latest changes from upstream and rebase before creating any pull request. 36 | - New pull requests should be created against the `integration` branch of the repository, if available. 37 | - This ensures new code is included in full-stack integration tests before being merged into the `master` branch 38 | - All new features must be accompanied by associated tests. 39 | - Make sure all tests pass locally before submitting a pull request. 40 | - Include tests with every feature enhancement, improve tests with every bug fix 41 | 42 | ### Github and git flow 43 | 44 | The internet is littered with guides and information on how to use and understand git. 45 | However, here's a compact guide that follows the suggested workflow 46 | 47 | ![Github flow](https://ibm-cloud-architecture.github.io/assets/img/github_flow.png) 48 | 49 | 1. Fork the desired repo in github. 50 | 51 | 2. Clone your repo to your local computer. 52 | 53 | 3. Add the upstream repository 54 | 55 | Note: Guide for step 1-3 here: [forking a repo](https://help.github.com/articles/fork-a-repo/) 56 | 57 | 4. Create new development branch off the targeted upstream branch. This will often be `master`. 58 | 59 | ``` 60 | git checkout -b master 61 | ``` 62 | 63 | 5. Do your work: 64 | - Write your code 65 | - Write your tests 66 | - Pass your tests locally 67 | - Commit your intermediate changes as you go and as appropriate 68 | - Repeat until satisfied 69 | 70 | 6. Fetch latest upstream changes (in case other changes had been delivered upstream while you were developing your new feature). 71 | 72 | ``` 73 | git fetch upstream 74 | ``` 75 | 7. Rebase to the latest upstream changes, resolving any conflicts. This will 'replay' your local commits, one by one, after the changes delivered upstream while you were locally developing, letting you manually resolve any conflict. 76 | 77 | ``` 78 | git branch --set-upstream-to=upstream/master 79 | git rebase 80 | ``` 81 | Instructions on how to manually resolve a conflict and commit the new change or skip your local replayed commit will be presented on screen by the git CLI. 82 | 83 | 8. Push the changes to your repository 84 | 85 | ``` 86 | git push origin 87 | ``` 88 | 89 | 9. Create a pull request against the same targeted upstream branch. 90 | 91 | [Creating a pull request](https://help.github.com/articles/creating-a-pull-request/) 92 | 93 | Once the pull request has been reviewed, accepted and merged into the main github repository, you should synchronise your remote and local forked github repository `master` branch with the upstream master branch. To do so: 94 | 95 | 10. Pull to your local forked repository the latest changes upstream (that is, the pull request). 96 | 97 | ``` 98 | git pull upstream master 99 | ``` 100 | 101 | 11. Push those latest upstream changes pulled locally to your remote forked repository. 102 | 103 | ``` 104 | git push origin master 105 | ``` 106 | 107 | ### What happens next? 108 | - All pull requests will be automatically built and unit tested by travis-ci, when implemented by that specific project. 109 | - You can determine if a given project is enabled for travis-ci unit tests by the existence of a `.travis.yml` file in the root of the repository or branch. 110 | - When in use, all travis-ci unit tests must pass completely before any further review or discussion takes place. 111 | - The repository maintainer will then inspect the commit and, if accepted, will pull the code into the upstream branch. 112 | - Should a maintainer or reviewer ask for changes to be made to the pull request, these can be made locally and pushed to your forked repository and branch. 113 | - Commits passing this stage will make it into the next release cycle for the given project. 114 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Terraform ICP Provision Module 2 | This terraform module can be used to deploy IBM Cloud Private on any supported infrastructure vendor. 3 | Tested on Ubuntu 16.04 and RHEL 7 on SoftLayer, VMware, AWS and Azure. 4 | 5 | ### Pre-requisites 6 | 7 | If the default SSH user is not the root user, the default user must have password-less sudo access. 8 | 9 | 10 | ## Inputs 11 | 12 | | Variable | Default |Required| Description | 13 | |--------------------|---------------|--------|----------------------------------------| 14 | | **Cluster settings** | 15 | |icp-inception | |No* |Version of ICP to provision. Not required when See below for details on using private registry| 16 | |icp-master | |No* |IP address of ICP Masters. Required if you don't use icp-host-groups | 17 | |icp-worker | |No* |IP addresses of ICP Worker nodes. Required if you don't use icp-host-groups | 18 | |icp-proxy | |No* |IP addresses of ICP Proxy nodes. Required if you don't use icp-host-groups | 19 | |icp-management | |No |IP addresses of ICP Management Nodes, if management is to be separated from master nodes. Optional| 20 | | icp-host-groups | |No* | Map of host types and IPs. See below for details. | 21 | | boot-node | |No* | IP Address of boot node. Needed when using icp-host-groups or when using a boot node separate from first master node. If separate it must be included in `cluster_size` | 22 | |cluster_size | |Yes |Define total clustersize. Workaround for terraform issue #10857. Normally computed| 23 | | **ICP Configuration** | 24 | |icp_config_file | |No |Yaml configuration file for ICP installation.| 25 | |icp_configuration | |No |Configuration items for ICP installation. See [KnowledgeCenter](https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0/installing/config_yaml.html) for reference. Note: Boolean values (true/false) must be supplied as strings| 26 | |config_strategy |merge |No |Strategy for original config.yaml shipped with ICP. Default is merge, everything else means override. | 27 | | **ICP Boot node to cluster communication** | 28 | |generate_key |True |No |Whether to generate a new ssh key for use by ICP Boot Master to communicate with other nodes| 29 | |icp_pub_key | |No |Public ssh key for ICP Boot master to connect to ICP Cluster. Only use when generate_key = false| 30 | |icp_priv_key | |No |Private ssh key for ICP Boot master to connect to ICP Cluster. Only use when generate_key = false| 31 | | **Terraform installation process** | 32 | |hooks | |No |Hooks into different stages in the cluster setup process. See below for details| 33 | |local-hooks | |No |Locally run hooks at different stages in the cluster setup process. See below for details| 34 | |on_hook_failure |fail | |Behavior when hooks fail. Anything other than `fail` will `continue`| 35 | |install-verbosity | |No | Verbosity of the icp ansible installer. -v to -vvvv. See ansible documentation for verbosity information | 36 | |install-command |install |No | Installer command to run | 37 | |cluster-directory |/opt/ibm/cluster |No | Location to use for the cluster directory | 38 | |cluster_dir_owner | |No |Username to own cluster directory after an install. Defaults to `ssh_user`| 39 | | **Terraform to cluster ssh configuration**| 40 | |ssh_user |root |No |Username for Terraform to ssh into the ICP cluster. This is typically the default user with for the relevant cloud vendor| 41 | |ssh_key_base64 | |No |base64 encoded content of private ssh key| 42 | |ssh_agent |True |No |Enable or disable SSH Agent. Can correct some connectivity issues. Default: true (enabled)| 43 | |bastion_host | |No |Specify hostname or IP to connect to nodes through a SSH bastion host. Assumes same SSH key and username as cluster nodes| 44 | | **Docker and ICP Enterprise Edition Image configuration** | 45 | |docker_package_location| |No |http or nfs location of docker installer which ships with ICP. Typically used for RHEL which does not support docker-ce| 46 | |docker_image_name |docker-ce |No |Name of docker image to install when installing from Ubuntu or Centos repository | 47 | |docker_version |latest |No |Version of docker image to install from Ubuntu or Centos repository. i.e. `18.06.1` `latest` install latest version. | 48 | |image_location | |No |Location of image file. Start with `nfs:` or `http` to indicate protocol to download with. Starting with `/` indicates local file. See example below | 49 | |image_locations | | No | List of images in same format as `image_location`. Typically used for multi-arch deployments | 50 | |image_location_user | | No | Username if authentication required for image_location | 51 | |image_location_pass | | No | Password if authentication required for image_location | 52 | 53 | ## Outputs 54 | 55 | - icp_public_key 56 | * The public key used for boot master to connect via ssh for cluster setup 57 | - icp_private_key 58 | * The public key used for boot master to connect via ssh for cluster setup 59 | - install_complete 60 | * Boolean value that is set to true when ICP installation process is completed 61 | - icp_version 62 | * The ICP version that has been installed 63 | - cluster_ips 64 | * List of IPs of the cluster 65 | 66 | ### ICP Version specifications 67 | The `icp-inception` field supports the format `org`/`repo`:`version`. `ibmcom` is the default organisation and `icp-inception` is the default repo, so if you're installing for example version `2.1.0.2` from Docker Hub it's sufficient to specify `2.1.0.2` as the version number. 68 | 69 | It is also supported to install from private docker registries. 70 | In this case the format is: 71 | `username`:`password`@`private_registry_server`/`org`/`repo`:`version`. 72 | 73 | So for exmaple 74 | 75 | `myuser:SomeLongPassword@myprivateregistry.local/ibmcom/icp-inception:2.1.0.2` 76 | 77 | 78 | ### Remote Execution Hooks 79 | It is possible to execute arbitrary commands between various phases of the cluster setup and installation process. 80 | Currently, the following hooks are defined. Each hook must be a list of commands to run. 81 | 82 | | Hook name | Where executed | When executed | 83 | |--------------------|----------------|------------------------------------------------------------| 84 | | cluster-preconfig | all nodes | Before any of the module scripts | 85 | | cluster-postconfig | all nodes | After prerequisites are installed | 86 | | boot-preconfig | boot master | Before any module scripts on boot master | 87 | | preinstall | boot master | After configuration image load and configuration generation| 88 | | postinstall | boot master | After successful ICP installation | 89 | 90 | ### Local Execution Hooks 91 | It is possible to execute arbitrary commands between various phases of the cluster setup and installation process. 92 | Currently, the following hooks are defined. Each hook must be a single command to run. 93 | 94 | | Hook name | When executed | 95 | |--------------------|------------------------------------------------------------| 96 | | local-preinstall | After configuration and preinstall remote hook | 97 | | local-postinstall | After successful ICP installation | 98 | 99 | These hooks support the execution of a single command or a local script. While this is a local-exec [command](https://www.terraform.io/docs/provisioners/local-exec.html#command), passing additional interpreter/environment parameters are not supported and therefore everything will be treated as a BASH script. 100 | 101 | 102 | ### Host groups 103 | In ICP version 2.1.0.2 the concept of host groups were introduced. This allows users to define groups of hosts by an arbritrary name that will be labelled such that they can be dedicated to particular workloads. You can read more about host groups on the [KnowledgeCenter](https://www.ibm.com/support/knowledgecenter/en/SSBS6K_2.1.0.2/installing/hosts.html) 104 | 105 | To support this an input map called `icp-host-groups` were introduced, and this can be used to generate the relevant hosts file for the ICP installer. When using this field it should be used **instead of** the `icp-master`, `icp-worker`, etc fields. 106 | 107 | ## Usage example 108 | 109 | ### Using hooks 110 | 111 | ```hcl 112 | module "icpprovision" { 113 | source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0" 114 | 115 | icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"] 116 | icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"] 117 | icp-proxy = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"] 118 | 119 | icp-inception = "3.1.2" 120 | 121 | cluster_size = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}" 122 | 123 | icp_configuration = { 124 | "network_cidr" = "192.168.0.0/16" 125 | "service_cluster_ip_range" = "172.16.0.1/24" 126 | "default_admin_password" = "My0wnPassw0rd" 127 | } 128 | 129 | generate_key = true 130 | 131 | ssh_user = "ubuntu" 132 | ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}" 133 | 134 | hooks = { 135 | "cluster-preconfig" = [ 136 | "echo This will run on all nodes", 137 | "echo And I can run as many commands", 138 | "echo as I want", 139 | "echo ....they will run in order" 140 | ] 141 | "postinstall" = [ 142 | "echo Performing some post install backup", 143 | "${ var.postinstallbackup != "true" ? "" : "sudo chmod a+x /tmp/icp_backup.sh ; sudo /tmp/icp_backup.sh" }" 144 | ] 145 | } 146 | } 147 | ``` 148 | 149 | 150 | #### Using HostGroups 151 | 152 | ```hcl 153 | module "icpprovision" { 154 | source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0" 155 | 156 | # We will define master, management, worker, proxy and va (Vulnerability Assistant) as well as a custom db2 group 157 | icp-host-groups = { 158 | master = "${openstack_compute_instance_v2.icpmaster.*.access_ip_v4}" 159 | management = "${openstack_compute_instance_v2.icpmanagement.*.access_ip_v4}" 160 | worker = "${openstack_compute_instance_v2.icpworker.*.access_ip_v4}" 161 | proxy = "${openstack_compute_instance_v2.icpproxy.*.access_ip_v4}" 162 | va = "${openstack_compute_instance_v2.icpva.*.access_ip_v4}" 163 | 164 | hostgroup-db2 = "${openstack_compute_instance_v2.icpdb2.*.access_ip_v4}" 165 | } 166 | 167 | # We always have to specify a node to bootstrap the cluster. It can be any of the cluster nodes, or a separate node that has network access to the cluster. 168 | # We will use the first master node as boot node to run the ansible installer from 169 | boot-node = "${openstack_compute_instance_v2.icpmaster.0.access_ip_v4}" 170 | 171 | icp-inception = "3.1.2" 172 | 173 | cluster_size = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}" 174 | 175 | icp_configuration = { 176 | "network_cidr" = "192.168.0.0/16" 177 | "service_cluster_ip_range" = "172.16.0.1/24" 178 | "default_admin_password" = "My0wnPassw0rd" 179 | } 180 | } 181 | ``` 182 | 183 | #### Community Edition 184 | 185 | ```hcl 186 | module "icpprovision" { 187 | source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0" 188 | 189 | icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"] 190 | icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"] 191 | icp-proxy = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"] 192 | 193 | icp-inception = "3.1.2" 194 | 195 | cluster_size = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}" 196 | 197 | icp_configuration = { 198 | "network_cidr" = "192.168.0.0/16" 199 | "service_cluster_ip_range" = "172.16.0.1/24" 200 | "default_admin_password" = "My0wnPassw0rd" 201 | } 202 | 203 | generate_key = true 204 | 205 | ssh_user = "ubuntu" 206 | ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}" 207 | 208 | } 209 | ``` 210 | 211 | #### Enterprise Edition 212 | 213 | From NFS location 214 | 215 | ```hcl 216 | module "icpprovision" { 217 | source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0" 218 | 219 | icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"] 220 | icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"] 221 | icp-proxy = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"] 222 | 223 | image_location = "nfs:fsf-lon0601b-fz.adn.networklayer.com:/IBM02S6275/data01/ibm-cloud-private-x86_64-2.1.0.1.tar.gz" 224 | 225 | cluster_size = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}" 226 | 227 | icp_configuration = { 228 | "network_cidr" = "192.168.0.0/16" 229 | "service_cluster_ip_range" = "172.16.0.1/24" 230 | "default_admin_password" = "My0wnPassw0rd" 231 | } 232 | 233 | generate_key = true 234 | 235 | ssh_user = "ubuntu" 236 | ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}" 237 | 238 | } 239 | ``` 240 | 241 | From HTTP location 242 | 243 | ```hcl 244 | module "icpprovision" { 245 | source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.0.0" 246 | 247 | icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"] 248 | icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"] 249 | icp-proxy = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"] 250 | 251 | image_location = "https://myserver.host/myfiles/ibm-cloud-private-x86_64-2.1.0.1.tar.gz" 252 | 253 | 254 | cluster_size = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}" 255 | 256 | icp_configuration = { 257 | "network_cidr" = "192.168.0.0/16" 258 | "service_cluster_ip_range" = "172.16.0.1/24" 259 | "default_admin_password" = "My0wnPassw0rd" 260 | } 261 | 262 | generate_key = true 263 | 264 | ssh_user = "ubuntu" 265 | ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}" 266 | 267 | } 268 | ``` 269 | 270 | Load tarball already existing on the boot node. 271 | 272 | ```hcl 273 | module "icpprovision" { 274 | source = "github.com/ibm-cloud-architecture/terraform-module-icp-deploy?ref=3.1.0" 275 | 276 | icp-master = ["${softlayer_virtual_guest.icpmaster.ipv4_address}"] 277 | icp-worker = ["${softlayer_virtual_guest.icpworker.*.ipv4_address}"] 278 | icp-proxy = ["${softlayer_virtual_guest.icpproxy.*.ipv4_address}"] 279 | 280 | image_location = "/opt/ibm/cluster/images/ibm-cloud-private-x86_64-3.1.2.tar.gz" 281 | 282 | 283 | cluster_size = "${var.master["nodes"] + var.worker["nodes"] + var.proxy["nodes"]}" 284 | 285 | icp_configuration = { 286 | "network_cidr" = "192.168.0.0/16" 287 | "service_cluster_ip_range" = "172.16.0.1/24" 288 | "default_admin_password" = "My0wnPassw0rd" 289 | } 290 | 291 | generate_key = true 292 | 293 | ssh_user = "ubuntu" 294 | ssh_key_base64 = "${base64encode(file("~/.ssh/id_rsa"))}" 295 | 296 | } 297 | ``` 298 | 299 | 300 | 301 | There are several examples for different providers available from [IBM Cloud Architecture Solutions Group github page](https://github.com/ibm-cloud-architecture) 302 | - [ICP on SoftLayer](https://github.com/ibm-cloud-architecture/terraform-icp-softlayer) 303 | - [ICP on VMware](https://github.com/ibm-cloud-architecture/terraform-icp-vmware) 304 | 305 | 306 | ### ICP Configuration 307 | Configuration file is generated from items in the following order 308 | 309 | 1. config.yaml shipped with ICP (if config_strategy = merge, else blank) 310 | 2. config.yaml specified in `icp_config_file` 311 | 3. key: value items specified in `icp_configuration` 312 | 313 | Details on configuration items on ICP KnowledgeCenter 314 | * [ICP 1.2.0](https://www.ibm.com/support/knowledgecenter/SSBS6K_1.2.0/installing/config_yaml.html) 315 | * [ICP 2.1.0](https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0/installing/config_yaml.html) 316 | 317 | 318 | ### Scaling 319 | The module supports automatic scaling of worker nodes. 320 | To scale simply add more nodes in the root resource supplying the `icp-worker` variable. 321 | You can see working examples for softlayer [in the icp-softlayer](https://github.com/ibm-cloud-architecture/terraform-icp-softlayer) repository 322 | 323 | Please note, because of how terraform handles module dependencies and triggers, it is currently necessary to retrigger the scaling resource **after scaling down** nodes. 324 | If you don't do this ICP will continue to report inactive nodes until the next scaling event. 325 | To manually trigger the removal of deleted node, run these commands: 326 | 327 | 1. `terraform taint --module icpprovision null_resource.icp-worker-scaler` 328 | 2. `terraform apply` 329 | 330 | 331 | ## Module Versions 332 | 333 | As new use cases and best practices emerge code will be added to and changed the in module. Any changes in the code leads to a new release version. The module versions follow a [semantic versioning](https://semver.org/) scheme. 334 | 335 | To avoid breaking existing templates which depends on the module it is recommended to add a version tag to the module source when pulling directly from git. 336 | 337 | 338 | ### Versions and changes 339 | #### 3.1.1 340 | - Fix issues with offline install 341 | - Fix issue with config generation 342 | 343 | #### 3.1.0 344 | - Allow cluster directory to be specified 345 | - Allow other targets to be called from `icp-inception` 346 | - Fix issues when owner of cluster files are something other than `ssh_user` 347 | - Allow the cluster directory to be owned by arbitrary user after install 348 | - Accept local files as valid location for `image_location` 349 | 350 | #### 3.0.8 351 | - Fix docker install from yum repo for non-root user on RHEL 352 | 353 | #### 3.0.7 354 | - Fix password length to comply with ICP 3.1.2 password rules 355 | 356 | #### 3.0.6 357 | - Fix verbose installation option 358 | 359 | #### 3.0.5 360 | - Fix docker install on ppc64le and s390x 361 | 362 | #### 3.0.4 363 | - Fix default_admin_password output when provided as empty string in icp_config 364 | 365 | #### 3.0.3 366 | - Fix blank icp_configuration["default_admin_password"] not generating random password 367 | 368 | #### 3.0.2 369 | - Fix remote hook issue 370 | 371 | #### 3.0.1 372 | - Fix local-hook issue 373 | 374 | #### 3.0.0 375 | - Retire parallel image pull 376 | - Retire unused variables (enterprise-edition, image_file, ssh_key, ssh_key_file) 377 | - Default to generate strong default admin password if no password is specified 378 | - Detect inception image when installing from offline tarball 379 | - Rename `icp-version` variable to more descriptive `icp-inception` 380 | - Overhaul of scaler function 381 | - Add support for automatic installation of docker on RHEL and Centos 382 | - Add support for downloading multi-arch images 383 | - Fix wget output spamming during wget image download 384 | - Include BATS test for testing scripts locally 385 | - Rewrite image-load to take username and password separately when downloading from HTTP source 386 | 387 | #### 2.4.0 388 | - Add support for local hooks 389 | - Support specifying docker version when installing docker with apt (Ubuntu only) 390 | - Ensure /opt/ibm is present before copying cluster skeleton 391 | 392 | #### 2.3.7 393 | - Add retry logic to apt-get when installing prerequisites. Sometimes cloud-init or some other startup process can hold a lock on apt. 394 | 395 | #### 2.3.6 396 | - Retry ssh from boot to cluster nodes when generating /etc/hosts entries. Fixes issues when some cluster nodes are provisioned substantially slower. 397 | - Report exit code from docker when running ansible installer, rather than the last command in the pipelist (tee) 398 | 399 | #### 2.3.5 400 | - Skip blanks when generating config.yaml as yaml.safe_dump exports them as '' which ansible installer doesn't like 401 | 402 | #### 2.3.4 403 | - Create backup copy of original config.yaml to keep options and comments 404 | - Support nested dictionaries when parsing `icp_configuration` to convert true/false strings to booleans 405 | 406 | #### 2.3.3 407 | - Fix empty icp-master list issue when using icp-host-groups 408 | - Fix issue with docker package install from nfs source 409 | - Make docker check silent when docker is not installed 410 | 411 | #### 2.3.2 412 | - Fix issues with terraform formatting of boolean values in config.yaml 413 | 414 | #### 2.3.1 415 | - Fix issue with non-hostgroups installations not generating hosts files 416 | - Fix boot-node not being optional in non-hostgroups installations 417 | - Fix issue with boot node trying to ssh itself 418 | - Install docker from repository if no other method selected (ubuntu only) 419 | - Fix apt install issue for prerequisites 420 | 421 | #### 2.3.0 422 | - Add full support for separate boot node 423 | - Save icp install log output to /tmp/icp-install-log.txt 424 | - Add option for verbosity on icp install log output 425 | 426 | #### 2.2.2 427 | - Fix issues with email usernames when using private registry 428 | - Fix passwords containing ':' when using private registry 429 | 430 | #### 2.2.1 431 | - Fix scaler error when using hostgroups 432 | 433 | #### 2.2.0 434 | - Added support for hostgroups 435 | - Updated preprequisites scripts to avoid emediate failure in airgapped installations 436 | - Include module outputs 437 | 438 | 439 | #### 2.1.0 440 | - Added support for install hooks 441 | - Added support for converged proxy nodes (combined master/proxy) 442 | - Added support for private docker registry 443 | 444 | #### 2.0.1 445 | - Fixed problem with worker scaler 446 | 447 | #### 2.0.0 448 | - Added support for ssh bastion host 449 | - Added support for dedicated management hosts 450 | - Split up null_resource provisioners to increase granularity 451 | - Added support for parallel load of EE images 452 | - Various fixes 453 | 454 | #### 1.0.0 455 | - Initial release 456 | -------------------------------------------------------------------------------- /hooks.tf: -------------------------------------------------------------------------------- 1 | ####### 2 | ## Hooks to run before any other module actions 3 | ####### 4 | 5 | ## Cluster Pre-config hook 6 | resource "null_resource" "icp-cluster-preconfig-hook-stop-on-fail" { 7 | count = "${var.on_hook_failure == "fail" ? var.cluster_size : 0}" 8 | 9 | connection { 10 | host = "${element(local.icp-ips, count.index)}" 11 | user = "${var.ssh_user}" 12 | private_key = "${local.ssh_key}" 13 | agent = "${var.ssh_agent}" 14 | bastion_host = "${var.bastion_host}" 15 | } 16 | 17 | # Run cluster-preconfig commands 18 | provisioner "remote-exec" { 19 | inline = [ 20 | "${var.hooks["cluster-preconfig"]}" 21 | ] 22 | on_failure = "fail" 23 | } 24 | } 25 | resource "null_resource" "icp-cluster-preconfig-hook-continue-on-fail" { 26 | count = "${var.on_hook_failure != "fail" ? var.cluster_size : 0}" 27 | 28 | connection { 29 | host = "${element(local.icp-ips, count.index)}" 30 | user = "${var.ssh_user}" 31 | private_key = "${local.ssh_key}" 32 | agent = "${var.ssh_agent}" 33 | bastion_host = "${var.bastion_host}" 34 | } 35 | 36 | # Run cluster-preconfig commands 37 | provisioner "remote-exec" { 38 | inline = [ 39 | "${var.hooks["cluster-preconfig"]}" 40 | ] 41 | on_failure = "continue" 42 | } 43 | } 44 | 45 | ####### 46 | ## Hooks to run after cluster prereqs install 47 | ####### 48 | 49 | ## Cluster postconfig hook 50 | resource "null_resource" "icp-cluster-postconfig-hook-stop-on-fail" { 51 | depends_on = ["null_resource.icp-cluster"] 52 | count = "${var.on_hook_failure == "fail" ? var.cluster_size : 0}" 53 | 54 | connection { 55 | host = "${element(local.icp-ips, count.index)}" 56 | user = "${var.ssh_user}" 57 | private_key = "${local.ssh_key}" 58 | agent = "${var.ssh_agent}" 59 | bastion_host = "${var.bastion_host}" 60 | } 61 | 62 | # Run cluster-postconfig commands 63 | provisioner "remote-exec" { 64 | inline = [ 65 | "${var.hooks["cluster-postconfig"]}" 66 | ] 67 | on_failure = "fail" 68 | } 69 | } 70 | resource "null_resource" "icp-cluster-postconfig-hook-continue-on-fail" { 71 | depends_on = ["null_resource.icp-cluster"] 72 | count = "${var.on_hook_failure != "fail" ? var.cluster_size : 0}" 73 | 74 | connection { 75 | host = "${element(local.icp-ips, count.index)}" 76 | user = "${var.ssh_user}" 77 | private_key = "${local.ssh_key}" 78 | agent = "${var.ssh_agent}" 79 | bastion_host = "${var.bastion_host}" 80 | } 81 | 82 | # Run cluster-postconfig commands 83 | provisioner "remote-exec" { 84 | inline = [ 85 | "${var.hooks["cluster-postconfig"]}" 86 | ] 87 | on_failure = "continue" 88 | } 89 | } 90 | 91 | 92 | # First hook for Boot node 93 | resource "null_resource" "icp-boot-preconfig-stop-on-fail" { 94 | depends_on = ["null_resource.icp-cluster-postconfig-hook-continue-on-fail", "null_resource.icp-cluster-postconfig-hook-stop-on-fail", "null_resource.icp-cluster"] 95 | count = "${var.on_hook_failure == "fail" ? 1 : 0}" 96 | 97 | # The first master is always the boot master where we run provisioning jobs from 98 | connection { 99 | host = "${local.boot-node}" 100 | user = "${var.ssh_user}" 101 | private_key = "${local.ssh_key}" 102 | agent = "${var.ssh_agent}" 103 | bastion_host = "${var.bastion_host}" 104 | } 105 | 106 | # Run stage hook commands 107 | provisioner "remote-exec" { 108 | inline = [ 109 | "${var.hooks["boot-preconfig"]}" 110 | ] 111 | on_failure = "fail" 112 | } 113 | } 114 | resource "null_resource" "icp-boot-preconfig-continue-on-fail" { 115 | depends_on = ["null_resource.icp-cluster-postconfig-hook-stop-on-fail", "null_resource.icp-cluster-postconfig-hook-continue-on-fail", "null_resource.icp-cluster"] 116 | count = "${var.on_hook_failure != "fail" ? 1 : 0}" 117 | 118 | # The first master is always the boot master where we run provisioning jobs from 119 | connection { 120 | host = "${local.boot-node}" 121 | user = "${var.ssh_user}" 122 | private_key = "${local.ssh_key}" 123 | agent = "${var.ssh_agent}" 124 | bastion_host = "${var.bastion_host}" 125 | } 126 | 127 | # Run stage hook commands 128 | provisioner "remote-exec" { 129 | inline = [ 130 | "${var.hooks["boot-preconfig"]}" 131 | ] 132 | on_failure = "continue" 133 | } 134 | } 135 | 136 | ####### 137 | ## Hooks to run before installation 138 | ####### 139 | # Boot node hook 140 | resource "null_resource" "icp-preinstall-hook-stop-on-fail" { 141 | depends_on = ["null_resource.icp-generate-hosts-files"] 142 | count = "${var.on_hook_failure == "fail" ? 1 : 0}" 143 | 144 | # The first master is always the boot master where we run provisioning jobs from 145 | connection { 146 | host = "${local.boot-node}" 147 | user = "${var.ssh_user}" 148 | private_key = "${local.ssh_key}" 149 | agent = "${var.ssh_agent}" 150 | bastion_host = "${var.bastion_host}" 151 | } 152 | 153 | # Run stage hook commands 154 | provisioner "remote-exec" { 155 | inline = [ 156 | "${var.hooks["preinstall"]}" 157 | ] 158 | on_failure = "fail" 159 | } 160 | } 161 | resource "null_resource" "icp-preinstall-hook-continue-on-fail" { 162 | depends_on = ["null_resource.icp-generate-hosts-files"] 163 | count = "${var.on_hook_failure != "fail" ? 1 : 0}" 164 | 165 | # The first master is always the boot master where we run provisioning jobs from 166 | connection { 167 | host = "${local.boot-node}" 168 | user = "${var.ssh_user}" 169 | private_key = "${local.ssh_key}" 170 | agent = "${var.ssh_agent}" 171 | bastion_host = "${var.bastion_host}" 172 | } 173 | 174 | # Run stage hook commands 175 | provisioner "remote-exec" { 176 | inline = [ 177 | "${var.hooks["preinstall"]}" 178 | ] 179 | on_failure = "continue" 180 | } 181 | } 182 | 183 | # Local preinstall hook 184 | resource "null_resource" "local-preinstall-hook-stop-on-fail" { 185 | depends_on = ["null_resource.icp-preinstall-hook-continue-on-fail", "null_resource.icp-preinstall-hook-stop-on-fail"] 186 | count = "${var.on_hook_failure == "fail" ? 1 : 0}" 187 | 188 | provisioner "local-exec" { 189 | command = "${var.local-hooks["local-preinstall"]}" 190 | on_failure = "fail" 191 | } 192 | } 193 | resource "null_resource" "local-preinstall-hook-continue-on-fail" { 194 | depends_on = ["null_resource.icp-preinstall-hook-continue-on-fail", "null_resource.icp-preinstall-hook-stop-on-fail"] 195 | count = "${var.on_hook_failure != "fail" ? 1 : 0}" 196 | 197 | provisioner "local-exec" { 198 | command = "${var.local-hooks["local-preinstall"]}" 199 | on_failure = "continue" 200 | } 201 | } 202 | 203 | 204 | # Local postinstall hook 205 | resource "null_resource" "local-postinstall-hook-stop-on-fail" { 206 | depends_on = ["null_resource.icp-install"] 207 | count = "${var.on_hook_failure == "fail" ? 1 : 0}" 208 | 209 | provisioner "local-exec" { 210 | command = "${var.local-hooks["local-postinstall"]}" 211 | on_failure = "fail" 212 | } 213 | } 214 | resource "null_resource" "local-postinstall-hook-continue-on-fail" { 215 | depends_on = ["null_resource.icp-install"] 216 | count = "${var.on_hook_failure != "fail" ? 1 : 0}" 217 | 218 | provisioner "local-exec" { 219 | command = "${var.local-hooks["local-postinstall"]}" 220 | on_failure = "continue" 221 | } 222 | } 223 | 224 | ####### 225 | ## Hooks to run after installation complete 226 | ####### 227 | 228 | # Hook for Boot node 229 | resource "null_resource" "icp-postinstall-hook-stop-on-fail" { 230 | depends_on = ["null_resource.icp-install"] 231 | count = "${var.on_hook_failure == "fail" ? 1 : 0}" 232 | 233 | # The first master is always the boot master where we run provisioning jobs from 234 | connection { 235 | host = "${local.boot-node}" 236 | user = "${var.ssh_user}" 237 | private_key = "${local.ssh_key}" 238 | agent = "${var.ssh_agent}" 239 | bastion_host = "${var.bastion_host}" 240 | } 241 | 242 | # Run stage hook commands 243 | provisioner "remote-exec" { 244 | inline = [ 245 | "${var.hooks["postinstall"]}" 246 | ] 247 | on_failure = "fail" 248 | } 249 | } 250 | resource "null_resource" "icp-postinstall-hook-continue-on-fail" { 251 | depends_on = ["null_resource.icp-install"] 252 | count = "${var.on_hook_failure != "fail" ? 1 : 0}" 253 | 254 | # The first master is always the boot master where we run provisioning jobs from 255 | connection { 256 | host = "${local.boot-node}" 257 | user = "${var.ssh_user}" 258 | private_key = "${local.ssh_key}" 259 | agent = "${var.ssh_agent}" 260 | bastion_host = "${var.bastion_host}" 261 | } 262 | 263 | # Run stage hook commands 264 | provisioner "remote-exec" { 265 | inline = [ 266 | "${var.hooks["postinstall"]}" 267 | ] 268 | on_failure = "continue" 269 | } 270 | } 271 | -------------------------------------------------------------------------------- /main.tf: -------------------------------------------------------------------------------- 1 | 2 | # Generate a new key if this is required 3 | resource "tls_private_key" "icpkey" { 4 | algorithm = "RSA" 5 | } 6 | 7 | # Generate a random string for password if required 8 | resource "random_string" "generated_password" { 9 | length = "32" 10 | special = "false" 11 | } 12 | 13 | ## cluster-preconfig hooks are run before icp-cluster if defined 14 | 15 | 16 | ## Actions that has to be taken on all nodes in the cluster 17 | resource "null_resource" "icp-cluster" { 18 | depends_on = ["null_resource.icp-cluster-preconfig-hook-continue-on-fail", "null_resource.icp-cluster-preconfig-hook-stop-on-fail"] 19 | count = "${var.cluster_size}" 20 | 21 | connection { 22 | host = "${element(local.icp-ips, count.index)}" 23 | user = "${var.ssh_user}" 24 | private_key = "${local.ssh_key}" 25 | agent = "${var.ssh_agent}" 26 | bastion_host = "${var.bastion_host}" 27 | } 28 | 29 | # Validate we can do passwordless sudo in case we are not root 30 | provisioner "remote-exec" { 31 | inline = [ 32 | "sudo -n echo This will fail unless we have passwordless sudo access" 33 | ] 34 | } 35 | 36 | provisioner "remote-exec" { 37 | inline = [ 38 | "mkdir -p /tmp/icp-common-scripts" 39 | ] 40 | } 41 | provisioner "file" { 42 | source = "${path.module}/scripts/common/" 43 | destination = "/tmp/icp-common-scripts" 44 | } 45 | 46 | provisioner "remote-exec" { 47 | inline = [ 48 | "mkdir -p ~/.ssh", 49 | "echo '${var.generate_key ? tls_private_key.icpkey.public_key_openssh : var.icp_pub_key}' | tee -a ~/.ssh/authorized_keys", 50 | "chmod a+x /tmp/icp-common-scripts/*", 51 | "/tmp/icp-common-scripts/prereqs.sh", 52 | "/tmp/icp-common-scripts/version-specific.sh ${var.icp-inception}", 53 | "/tmp/icp-common-scripts/docker-user.sh" 54 | ] 55 | } 56 | } 57 | 58 | ## icp-boot-preconfig hooks are run before icp-docker, if defined 59 | 60 | # To make script parameters more consistent we'll define a common set here 61 | locals { 62 | script_options = "${join(" -", list(""), compact(list( 63 | var.icp-inception == "" ? "" : "i ${var.icp-inception}", 64 | var.cluster-directory == "" ? "" : "d ${var.cluster-directory}", 65 | var.install-verbosity == "" ? "" : "l ${var.install-verbosity}", 66 | var.install-command == "" ? "" : "c ${var.install-command}", 67 | var.image_location_user == "" ? "" : "u ${var.image_location_user}", 68 | var.image_location_pass == "" ? "" : "p ${var.image_location_pass}", 69 | var.image_location == "" ? "" : "l ${var.image_location}", 70 | length(var.image_locations) == 0 ? "" : "l ${join(" -l ", var.image_locations )}", 71 | var.docker_package_location == "" ? "" : "o ${var.docker_package_location}", 72 | var.docker_image_name == "" ? "" : "k ${var.docker_image_name}", 73 | var.docker_version == "" ? "" : "s ${var.docker_version}" 74 | )))}" 75 | } 76 | 77 | resource "null_resource" "icp-docker" { 78 | depends_on = ["null_resource.icp-boot-preconfig-continue-on-fail", "null_resource.icp-boot-preconfig-stop-on-fail", "null_resource.icp-cluster"] 79 | 80 | # Boot node is always the first entry in the IP list, so if we're not pulling in parallel this will only happen on boot node 81 | connection { 82 | host = "${element(local.icp-ips, 0)}" 83 | user = "${var.ssh_user}" 84 | private_key = "${local.ssh_key}" 85 | agent = "${var.ssh_agent}" 86 | bastion_host = "${var.bastion_host}" 87 | } 88 | 89 | provisioner "remote-exec" { 90 | inline = [ 91 | "mkdir -p /tmp/icp-bootmaster-scripts" 92 | ] 93 | } 94 | 95 | provisioner "file" { 96 | source = "${path.module}/scripts/boot-master/" 97 | destination = "/tmp/icp-bootmaster-scripts" 98 | } 99 | 100 | # Make sure scripts are executable and docker installed 101 | provisioner "remote-exec" { 102 | inline = [ 103 | "chmod a+x /tmp/icp-bootmaster-scripts/*.sh", 104 | "/tmp/icp-bootmaster-scripts/install-docker.sh ${local.script_options}" 105 | ] 106 | } 107 | } 108 | 109 | resource "null_resource" "icp-image" { 110 | depends_on = ["null_resource.icp-docker"] 111 | 112 | # Boot node is always the first entry in the IP list 113 | connection { 114 | host = "${element(local.icp-ips, 0)}" 115 | user = "${var.ssh_user}" 116 | private_key = "${local.ssh_key}" 117 | agent = "${var.ssh_agent}" 118 | bastion_host = "${var.bastion_host}" 119 | } 120 | 121 | provisioner "remote-exec" { 122 | inline = [ 123 | "echo \"Loading image ${var.icp-inception} ${var.image_location}\"", 124 | "/tmp/icp-bootmaster-scripts/load-image.sh ${local.script_options}" 125 | ] 126 | } 127 | } 128 | 129 | 130 | # First make sure scripts and configuration files are copied 131 | resource "null_resource" "icp-boot" { 132 | 133 | depends_on = ["null_resource.icp-image"] 134 | 135 | # The first master is always the boot master where we run provisioning jobs from 136 | connection { 137 | host = "${local.boot-node}" 138 | user = "${var.ssh_user}" 139 | private_key = "${local.ssh_key}" 140 | agent = "${var.ssh_agent}" 141 | bastion_host = "${var.bastion_host}" 142 | } 143 | 144 | 145 | # store config yaml if it was specified 146 | provisioner "file" { 147 | source = "${var.icp_config_file}" 148 | destination = "/tmp/config.yaml" 149 | } 150 | 151 | # JSON dump the contents of icp_configuration items 152 | provisioner "file" { 153 | content = "${jsonencode(var.icp_configuration)}" 154 | destination = "/tmp/items-config.yaml" 155 | } 156 | } 157 | 158 | # Generate all necessary configuration files, load image files, etc 159 | resource "null_resource" "icp-config" { 160 | depends_on = ["null_resource.icp-boot"] 161 | 162 | # The first master is always the boot master where we run provisioning jobs from 163 | connection { 164 | host = "${local.boot-node}" 165 | user = "${var.ssh_user}" 166 | private_key = "${local.ssh_key}" 167 | agent = "${var.ssh_agent}" 168 | bastion_host = "${var.bastion_host}" 169 | } 170 | 171 | provisioner "remote-exec" { 172 | inline = [ 173 | "/tmp/icp-bootmaster-scripts/copy_cluster_skel.sh ${local.script_options}", 174 | "python /tmp/icp-bootmaster-scripts/load-config.py ${var.cluster-directory} ${var.config_strategy} ${random_string.generated_password.result}" 175 | ] 176 | } 177 | 178 | # Copy the provided or generated private key 179 | provisioner "file" { 180 | content = "${var.generate_key ? tls_private_key.icpkey.private_key_pem : var.icp_priv_key}" 181 | destination = "/tmp/icp/cluster/ssh_key" 182 | } 183 | 184 | # Since the file provisioner deals badly with empty lists, we'll create the optional management nodes differently 185 | # Later we may refactor to use this method for all node types for consistency 186 | provisioner "remote-exec" { 187 | inline = [ 188 | "echo -n ${join(",", var.icp-master)} > /tmp/icp/cluster/masterlist.txt", 189 | "echo -n ${join(",", var.icp-proxy)} > /tmp/icp/cluster/proxylist.txt", 190 | "echo -n ${join(",", var.icp-worker)} > /tmp/icp/cluster/workerlist.txt", 191 | "echo -n ${join(",", var.icp-management)} > /tmp/icp/cluster/managementlist.txt", 192 | "mv -f /tmp/icp/cluster/* ${var.cluster-directory}/", 193 | "chmod 600 ${var.cluster-directory}/ssh_key" 194 | ] 195 | } 196 | 197 | # JSON dump the contents of icp-host-groups items 198 | provisioner "file" { 199 | content = "${jsonencode(var.icp-host-groups)}" 200 | destination = "/tmp/icp-host-groups.json" 201 | } 202 | } 203 | 204 | 205 | 206 | # Generate the hosts files on the cluster 207 | resource "null_resource" "icp-generate-hosts-files" { 208 | depends_on = ["null_resource.icp-config"] 209 | 210 | # The first master is always the boot master where we run provisioning jobs from 211 | connection { 212 | host = "${local.boot-node}" 213 | user = "${var.ssh_user}" 214 | private_key = "${local.ssh_key}" 215 | agent = "${var.ssh_agent}" 216 | bastion_host = "${var.bastion_host}" 217 | } 218 | 219 | provisioner "remote-exec" { 220 | inline = [ 221 | "/tmp/icp-bootmaster-scripts/generate_hostsfiles.sh ${local.script_options}" 222 | ] 223 | } 224 | } 225 | 226 | # Boot node and local hooks are run before install if defined 227 | 228 | # Start the installer 229 | resource "null_resource" "icp-install" { 230 | depends_on = ["null_resource.local-preinstall-hook-continue-on-fail", "null_resource.local-preinstall-hook-stop-on-fail", "null_resource.icp-generate-hosts-files"] 231 | 232 | # The first master is always the boot master where we run provisioning jobs from 233 | connection { 234 | host = "${local.boot-node}" 235 | user = "${var.ssh_user}" 236 | private_key = "${local.ssh_key}" 237 | agent = "${var.ssh_agent}" 238 | bastion_host = "${var.bastion_host}" 239 | } 240 | 241 | 242 | provisioner "remote-exec" { 243 | inline = [ 244 | "/tmp/icp-bootmaster-scripts/start_install.sh ${local.script_options}" 245 | ] 246 | } 247 | } 248 | 249 | ## Post install hooks are run after installation if defined 250 | 251 | resource "null_resource" "icp-worker-scaler" { 252 | depends_on = ["null_resource.icp-cluster", "null_resource.icp-install"] 253 | 254 | triggers { 255 | workers = "${join(",", var.icp-worker)}" 256 | } 257 | 258 | connection { 259 | host = "${local.boot-node}" 260 | user = "${var.ssh_user}" 261 | private_key = "${local.ssh_key}" 262 | agent = "${var.ssh_agent}" 263 | bastion_host = "${var.bastion_host}" 264 | } 265 | 266 | provisioner "remote-exec" { 267 | inline = [ 268 | "echo -n ${join(",", var.icp-master)} > /tmp/masterlist.txt", 269 | "echo -n ${join(",", var.icp-proxy)} > /tmp/proxylist.txt", 270 | "echo -n ${join(",", var.icp-worker)} > /tmp/workerlist.txt", 271 | "echo -n ${join(",", var.icp-management)} > /tmp/managementlist.txt" 272 | ] 273 | } 274 | 275 | # JSON dump the contents of icp-host-groups items 276 | provisioner "file" { 277 | content = "${jsonencode(var.icp-host-groups)}" 278 | destination = "/tmp/scaled-host-groups.json" 279 | } 280 | 281 | 282 | provisioner "file" { 283 | source = "${path.module}/scripts/boot-master/scaleworkers.sh" 284 | destination = "/tmp/icp-bootmaster-scripts/scaleworkers.sh" 285 | } 286 | 287 | provisioner "remote-exec" { 288 | inline = [ 289 | "chmod a+x /tmp/icp-bootmaster-scripts/scaleworkers.sh", 290 | "sudo chown ${var.ssh_user}:${var.ssh_user} -R ${var.cluster-directory}", 291 | "/tmp/icp-bootmaster-scripts/scaleworkers.sh ${var.icp-inception}", 292 | "sudo chown ${local.cluster_dir_owner}:${local.cluster_dir_owner} -R ${var.cluster-directory}" 293 | ] 294 | } 295 | } 296 | 297 | resource "null_resource" "icp-cluster-owner" { 298 | depends_on = ["null_resource.icp-worker-scaler", "null_resource.icp-postinstall-hook-continue-on-fail", "null_resource.icp-postinstall-hook-stop-on-fail"] 299 | 300 | # Change the owner of the cluster directory to the desired user 301 | connection { 302 | host = "${local.boot-node}" 303 | user = "${var.ssh_user}" 304 | private_key = "${local.ssh_key}" 305 | agent = "${var.ssh_agent}" 306 | bastion_host = "${var.bastion_host}" 307 | } 308 | 309 | provisioner "remote-exec" { 310 | inline = [ 311 | "sudo chown ${local.cluster_dir_owner}:${local.cluster_dir_owner} -R ${var.cluster-directory}", 312 | ] 313 | } 314 | } 315 | -------------------------------------------------------------------------------- /outputs.tf: -------------------------------------------------------------------------------- 1 | 2 | output "icp_public_key" { 3 | description = "The public key used for boot master to connect via ssh for cluster setup" 4 | value = "${var.generate_key ? tls_private_key.icpkey.public_key_openssh : var.icp_pub_key}" 5 | } 6 | 7 | output "icp_private_key" { 8 | description = "The public key used for boot master to connect via ssh for cluster setup" 9 | value = "${var.generate_key ? tls_private_key.icpkey.private_key_pem : var.icp_priv_key}" 10 | } 11 | 12 | output "install_complete" { 13 | depends_on = ["null_resource.icp-install"] 14 | description = "Boolean value that is set to true when ICP installation process is completed" 15 | value = "true" 16 | } 17 | 18 | output "icp_version" { 19 | value = "${var.icp-inception}" 20 | } 21 | 22 | output "cluster_ips" { 23 | value = "${local.icp-ips}" 24 | } 25 | 26 | locals { 27 | default_admin_password = "${lookup(var.icp_configuration, "default_admin_password", random_string.generated_password.result)}" 28 | } 29 | 30 | output "default_admin_password" { 31 | value = "${local.default_admin_password != "" ? local.default_admin_password : random_string.generated_password.result}" 32 | } 33 | -------------------------------------------------------------------------------- /scripts/boot-master/copy_cluster_skel.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | LOGFILE=/tmp/copyclusterskel.log 3 | exec 3>&1 4 | exec > >(tee -a ${LOGFILE} >/dev/null) 2> >(tee -a ${LOGFILE} >&3) 5 | 6 | echo "Script started with inputs $@" 7 | 8 | source /tmp/icp-bootmaster-scripts/functions.sh 9 | source /tmp/icp-bootmaster-scripts/get-args.sh 10 | 11 | # If loaded from tarball, icp version may not be specified in terraform 12 | if [[ -z "${icp_version}" ]]; then 13 | icp_version=$(get_inception_image) 14 | fi 15 | 16 | # Figure out the version 17 | # This will populate $org $repo and $tag 18 | parse_icpversion ${icp_version} 19 | echo "registry=${registry:-not specified} org=$org repo=$repo tag=$tag" 20 | 21 | # Copy the default data to the cluster directory 22 | docker run -e LICENSE=accept -v /tmp/icp:/data ${registry}${registry:+/}${org}/${repo}:${tag} cp -r cluster /data 23 | sudo chown $(whoami):$(whoami) -R /tmp/icp 24 | ensure_directory_reachable ${cluster_dir} 25 | sudo mv /tmp/icp/cluster/* ${cluster_dir} 26 | 27 | # Take a backup of original config file, to keep a record of original settings and comments 28 | cp ${cluster_dir}/config.yaml ${cluster_dir}/config.yaml-original 29 | -------------------------------------------------------------------------------- /scripts/boot-master/functions.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | DefaultOrg="ibmcom" 3 | DefaultRepo="icp-inception" 4 | 5 | # Populates globals $org $repo $tag 6 | function parse_icpversion() { 7 | 8 | # Determine if registry is also specified 9 | if [[ $1 =~ .*/.*/.* ]] 10 | then 11 | # Save the registry section of the string 12 | local r=$(echo ${1} | cut -d/ -f1) 13 | # Save username password if specified for registry 14 | if [[ $r =~ .*@.* ]] 15 | then 16 | local up=${r%@*} 17 | username=${up%%:*} 18 | password=${up#*:} 19 | registry=${r##*@} 20 | else 21 | registry=${r} 22 | fi 23 | org=$(echo ${1} | cut -d/ -f2) 24 | elif [[ $1 =~ .*/.* ]] 25 | # Determine organisation 26 | then 27 | org=$(echo ${1} | cut -d/ -f1) 28 | else 29 | org=$DefaultOrg 30 | fi 31 | 32 | # Determine repository and tag 33 | if [[ $1 =~ .*:.* ]] 34 | then 35 | repo=$(echo ${1##*/} | cut -d: -f1) 36 | tag=$(echo ${1##*/} | cut -d/ -f2 | cut -d: -f2) 37 | elif [[ "$1" == "" ]] 38 | then 39 | # We should autodetect the version if loaded from tarball 40 | # For now we grab the first matching image in docker image list. 41 | read -r repo tag <<<$(sudo docker image list | awk -v pat=$DefaultRepo ' $1 ~ pat { print $1 " " $2 ; exit }') 42 | 43 | # As a last resort we'll use the latest tag from docker hub 44 | if [[ -z $tag ]]; then 45 | repo=$DefaultRepo 46 | tag="latest" 47 | fi 48 | else 49 | # The last supported approach is to just supply version number 50 | repo=$DefaultRepo 51 | tag=$1 52 | fi 53 | } 54 | 55 | function get_inception_image() { 56 | # In cases where inception image has not been specified 57 | # we may look for inception image locally 58 | image=$(sudo docker image list | grep -m 1 inception | awk '{ print $1 ":" $2 }') 59 | echo $image 60 | } 61 | 62 | function ensure_directory_reachable() { 63 | # Ensure that the directory exists and is reachable: 64 | # 1) It is owned by the current user 65 | # 2) All parent directories are executable 66 | test_dir=$1 67 | sudo mkdir -p ${test_dir} 68 | sudo chown $(whoami):$(whoami) ${test_dir} 69 | f=${test_dir} 70 | while [[ $f != / ]]; do sudo chmod a+x "$f"; f=$(dirname "$f"); done; 71 | } 72 | -------------------------------------------------------------------------------- /scripts/boot-master/generate_hostsfiles.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | LOGFILE=/tmp/generate_hostsfiles.log 3 | exec 3>&1 4 | exec > >(tee -a ${LOGFILE} >/dev/null) 2> >(tee -a ${LOGFILE} >&3) 5 | 6 | source /tmp/icp-bootmaster-scripts/get-args.sh 7 | 8 | # Make sure ssh key has correct permissions set before using 9 | chmod 600 ${cluster_dir}/ssh_key 10 | 11 | # Global array variable for holding all cluster ip/hostnames 12 | declare -A cluster 13 | 14 | # Make sure /tmp/hosts is empty, in case we running this a second time 15 | [[ -e /tmp/hosts ]] && rm /tmp/hosts 16 | 17 | # Functions for "traditional" groups (master, proxy, worker, management) 18 | read_from_groupfiles() { 19 | ## First compile a list of all nodes in the cluster with ip addresses and associated hostnames 20 | declare -a master_ips 21 | IFS=', ' read -r -a master_ips <<< $(cat ${cluster_dir}/masterlist.txt) 22 | 23 | declare -a worker_ips 24 | IFS=', ' read -r -a worker_ips <<< $(cat ${cluster_dir}/workerlist.txt) 25 | 26 | declare -a proxy_ips 27 | IFS=', ' read -r -a proxy_ips <<< $(cat ${cluster_dir}/proxylist.txt) 28 | 29 | ## First gather all the hostnames and link them with ip addresses 30 | declare -A workers 31 | for worker in "${worker_ips[@]}"; do 32 | workers[$worker]=$(ssh -o StrictHostKeyChecking=no -i ${cluster_dir}/ssh_key ${worker} hostname) 33 | cluster[$worker]=${workers[$worker]} 34 | printf "%s %s\n" "$worker" "${cluster[$worker]}" >> /tmp/hosts 35 | done 36 | 37 | declare -A proxies 38 | for proxy in "${proxy_ips[@]}"; do 39 | proxies[$proxy]=$(ssh -o StrictHostKeyChecking=no -i ${cluster_dir}/ssh_key ${proxy} hostname) 40 | cluster[$proxy]=${proxies[$proxy]} 41 | printf "%s %s\n" "$proxy" "${cluster[$proxy]}" >> /tmp/hosts 42 | done 43 | 44 | declare -A masters 45 | for m in "${master_ips[@]}"; do 46 | # No need to ssh to self 47 | if hostname -I | grep -w $m &>/dev/null 48 | then 49 | masters[$m]=$(hostname) 50 | else 51 | masters[$m]=$(ssh -o StrictHostKeyChecking=no -i ${cluster_dir}/ssh_key ${m} hostname) 52 | fi 53 | cluster[$m]=${masters[$m]} 54 | printf "%s %s\n" "$m" "${cluster[$m]}" >> /tmp/hosts 55 | done 56 | 57 | # Add management nodes if separate from master nodes 58 | if [[ -s ${cluster_dir}/managementlist.txt ]] 59 | then 60 | declare -a management_ips 61 | IFS=', ' read -r -a management_ips <<< $(cat ${cluster_dir}/managementlist.txt) 62 | 63 | declare -A mngrs 64 | for m in "${management_ips[@]}"; do 65 | mngrs[$m]=$(ssh -o StrictHostKeyChecking=no -i ${cluster_dir}/ssh_key ${m} hostname) 66 | cluster[$m]=${mngrs[$m]} 67 | printf "%s %s\n" "$m" "${cluster[$m]}" >> /tmp/hosts 68 | done 69 | fi 70 | 71 | ## Generate the hosts file for the ICP installation 72 | echo '[master]' > ${cluster_dir}/hosts 73 | for master in "${master_ips[@]}"; do 74 | echo $master >> ${cluster_dir}/hosts 75 | done 76 | 77 | echo >> ${cluster_dir}/hosts 78 | echo '[worker]' >> ${cluster_dir}/hosts 79 | for worker in "${worker_ips[@]}"; do 80 | echo $worker >> ${cluster_dir}/hosts 81 | done 82 | 83 | echo >> ${cluster_dir}/hosts 84 | echo '[proxy]' >> ${cluster_dir}/hosts 85 | for proxy in "${proxy_ips[@]}"; do 86 | echo $proxy >> ${cluster_dir}/hosts 87 | done 88 | 89 | # Add management host entries if separate from master nodes 90 | if [[ ! -z ${management_ips} ]] 91 | then 92 | echo >> ${cluster_dir}/hosts 93 | echo '[management]' >> ${cluster_dir}/hosts 94 | for m in "${management_ips[@]}"; do 95 | echo $m >> ${cluster_dir}/hosts 96 | done 97 | fi 98 | } 99 | 100 | 101 | read_from_hostgroups() { 102 | # First parse the hostgroup json 103 | python /tmp/icp-bootmaster-scripts/parse-hostgroups.py ${cluster_dir} 104 | 105 | # Get the cluster ips 106 | declare -a cluster_ips 107 | IFS=',' read -r -a cluster_ips <<< $(cat /tmp/cluster-ips.txt) 108 | 109 | # Generate the hostname/ip combination 110 | for node in "${cluster_ips[@]}"; do 111 | cluster[$node]=$(ssh -o StrictHostKeyChecking=no -o ConnectionAttempts=100 -i ${cluster_dir}/ssh_key ${node} hostname) 112 | printf "%s %s\n" "$node" "${cluster[$node]}" >> /tmp/hosts 113 | done 114 | 115 | } 116 | 117 | #TODO: Figure out the situation when using separate boot node 118 | #TODO: Make sure /tmp/hosts is empty, so we don't double up all the time 119 | update_etchosts() { 120 | ## Update all hostfiles in all nodes in the cluster 121 | ## also remove the line for 127.0.1.1 122 | for node in "${!cluster[@]}"; do 123 | # No need to ssh to self 124 | if hostname -I | grep -w $node &>/dev/null 125 | then 126 | cat /tmp/hosts | cat - /etc/hosts | sed -e "/127.0.1.1/d" | sudo tee /etc/hosts 127 | else 128 | cat /tmp/hosts | ssh -i ${cluster_dir}/ssh_key ${node} 'cat - /etc/hosts | sed -e "/127.0.1.1/d" | sudo tee /etc/hosts' 129 | fi 130 | done 131 | } 132 | 133 | 134 | if [[ $( stat -c%s /tmp/icp-host-groups.json ) -gt 2 ]]; then 135 | read_from_hostgroups 136 | elif [[ -s ${cluster_dir}/masterlist.txt ]]; then 137 | read_from_groupfiles 138 | else 139 | echo "Couldn't find any hosts" >&2 140 | exit 1 141 | fi 142 | 143 | update_etchosts 144 | -------------------------------------------------------------------------------- /scripts/boot-master/get-args.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | while getopts ":i:d:v:c:l:u:p:o:k:s:" arg; do 4 | case "${arg}" in 5 | i) 6 | icp_inception=${OPTARG} 7 | ;; 8 | d) 9 | cluster_dir=${OPTARG} 10 | ;; 11 | v) 12 | log_verbosity=${OPTARG} 13 | ;; 14 | c) 15 | install_command=${OPTARG} 16 | ;; 17 | l) 18 | locations+=( ${OPTARG} ) 19 | ;; 20 | u) 21 | username=${OPTARG} 22 | ;; 23 | p) 24 | password=${OPTARG} 25 | ;; 26 | o) 27 | docker_package_location=${OPTARG} 28 | ;; 29 | k) 30 | docker_image=${OPTARG} 31 | ;; 32 | s) 33 | docker_version=${OPTARG} 34 | ;; 35 | \?) 36 | echo "Invalid option : -$OPTARG in commmand $0 $*" >&2 37 | exit 1 38 | ;; 39 | :) 40 | echo "Missing option argument for -$OPTARG in command $0 $*" >&2 41 | exit 1 42 | ;; 43 | esac 44 | done -------------------------------------------------------------------------------- /scripts/boot-master/install-docker.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ############################### 3 | ## 4 | ## This script will handle the installation of docker. 5 | ## 6 | ## 7 | ############################### 8 | LOGFILE=/tmp/install-docker.log 9 | exec 3>&1 10 | exec > >(tee -a ${LOGFILE} >/dev/null) 2> >(tee -a ${LOGFILE} >&3) 11 | 12 | echo "Script started with inputs $@" 13 | source /tmp/icp-bootmaster-scripts/get-args.sh 14 | 15 | sourcedir=/tmp/icp-docker 16 | 17 | function rhel_docker_install { 18 | # Process for RedHat VMs 19 | echo "Update RedHat or CentOS with latest patches" 20 | 21 | # Add the Extra Repo from RedHat to be able to support extra tools that needed 22 | sudo subscription-manager repos --enable=rhel-7-server-extras-rpms 23 | sudo yum update -y 24 | 25 | # Installing nesscarry tools for ICP to work including Netstat for tracing 26 | sudo yum install -y net-tools yum-utils device-mapper-persistent-data lvm2 27 | 28 | # Register Docker Community Edition repo for CentOS and RedHat 29 | sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo 30 | sudo yum install -y ${docker_image}${docker_version} 31 | if [[ $? -gt 0 ]]; then 32 | if which tail &>> /dev/null; then 33 | tail -n 5 $LOGFILE >&2 34 | fi 35 | 36 | echo "Error installing ${docker_image}${docker_version}" >&2 37 | exit 1 38 | fi 39 | 40 | # Start Docker locally on the host 41 | sudo systemctl enable docker 42 | sudo systemctl start docker 43 | } 44 | 45 | function ubuntu_docker_install { 46 | # Process for Ubuntu VMs 47 | echo "Installing ${docker_version:-latest} docker from docker repository" >&2 48 | sudo apt-get -q update 49 | # Make sure preprequisites are installed 50 | sudo apt-get -y -q install \ 51 | apt-transport-https \ 52 | ca-certificates \ 53 | curl \ 54 | software-properties-common 55 | 56 | # Add docker gpg key 57 | curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - 58 | 59 | # Right now hard code adding x86 repo 60 | local arch=$(uname -m | sed -e 's/x86_64/amd64/g' -e 's/ppc64le/ppc64el/g') 61 | sudo add-apt-repository \ 62 | "deb [arch=${arch}] https://download.docker.com/linux/ubuntu \ 63 | $(lsb_release -cs) \ 64 | stable" 65 | 66 | sudo apt-get -q update 67 | sudo apt-get -y -q install ${docker_image}${docker_version} 68 | if [[ $? -gt 0 ]]; then 69 | if which tail &>> /dev/null; then 70 | tail -n 5 $LOGFILE >&2 71 | fi 72 | 73 | echo "Error installing ${docker_image}${docker_version}" >&2 74 | exit 1 75 | fi 76 | } 77 | 78 | # Nothing to do here if we have docker already 79 | if docker --version &>> /dev/null 80 | then 81 | # Make sure the current user has permission to use docker 82 | /tmp/icp-common-scripts/docker-user.sh 83 | echo "Docker already installed. Exiting" >&2 84 | exit 0 85 | fi 86 | 87 | # Check if we're asked to install off-line or ICP provided docker bundle 88 | if [[ -z ${docker_package_location} ]] 89 | then 90 | echo "Not required to install ICP provided docker." 91 | else 92 | echo "Starting installation of ${docker_package_location}" 93 | mkdir -p ${sourcedir} 94 | 95 | # Decide which protocol to use 96 | if [[ "${docker_package_location:0:3}" == "nfs" ]] 97 | then 98 | # Separate out the filename and path 99 | nfs_mount=$(dirname ${docker_package_location:4}) 100 | package_file="${sourcedir}/$(basename ${docker_package_location})" 101 | # Mount 102 | sudo mount.nfs $nfs_mount $sourcedir 103 | elif [[ "${docker_package_location:0:4}" == "http" ]] 104 | then 105 | # Figure out what we should name the file 106 | filename="icp-docker.bin" 107 | mkdir -p ${sourcedir} 108 | curl -o ${sourcedir}/${filename} "${docker_package_location#http:}" 109 | package_file="${sourcedir}/${filename}" 110 | fi 111 | 112 | chmod a+x ${package_file} 113 | sudo ${package_file} --install 114 | 115 | # Make sure our user is added to the docker group if needed 116 | /tmp/icp-common-scripts/docker-user.sh 117 | 118 | echo "Install complete..." 119 | exit 0 120 | fi 121 | 122 | ## At this stage we better attempt to install from repository 123 | if grep -q -i ubuntu /etc/*release 124 | then 125 | OSLEVEL=ubuntu 126 | 127 | elif grep -q -i 'red hat' /etc/*release 128 | then 129 | OSLEVEL=redhat 130 | 131 | elif grep -q -i 'CentOS' /etc/*release 132 | then 133 | OSLEVEL=redhat 134 | else 135 | OSLEVEL=other 136 | fi 137 | echo "Operating System is $OSLEVEL" 138 | 139 | if [[ "${OSLEVEL}" == "ubuntu" ]] 140 | then 141 | if [ "${docker_version}" == "latest" ]; 142 | then 143 | docker_version="" 144 | else 145 | docker_version="=${docker_version}*" 146 | fi 147 | 148 | ubuntu_docker_install 149 | 150 | # Make sure our user is added to the docker group if needed 151 | /tmp/icp-common-scripts/docker-user.sh 152 | exit 0 153 | 154 | elif [[ "${OSLEVEL}" == "redhat" ]] 155 | then 156 | if [ "${docker_version}" == "latest" ] 157 | then 158 | docker_version="" 159 | else 160 | docker_version="-${docker_version}*" 161 | fi 162 | rhel_docker_install 163 | 164 | # Make sure our user is added to the docker group if needed 165 | /tmp/icp-common-scripts/docker-user.sh 166 | exit 0 167 | 168 | else 169 | echo "Only Ubuntu supported for repository install for now..." >&2 170 | echo "Please install docker manually, or with ICP provided docker bundle" >&2 171 | exit 1 172 | fi 173 | -------------------------------------------------------------------------------- /scripts/boot-master/load-config.py: -------------------------------------------------------------------------------- 1 | #!/bin/env python 2 | import os, sys, yaml, json, getpass 3 | config_file = '/tmp/config.yaml' 4 | config_items = '/tmp/items-config.yaml' 5 | config_orig = None 6 | 7 | # Load config from config file if provided 8 | if os.stat(config_file).st_size == 0: 9 | config_f = {} 10 | else: 11 | with open(config_file, 'r') as stream: 12 | try: 13 | config_f = yaml.load(stream) 14 | except yaml.YAMLError as exc: 15 | print(exc) 16 | 17 | 18 | # Load config items if provided 19 | with open(config_items, 'r') as stream: 20 | config_i = json.load(stream) 21 | 22 | supplied_cluster_dir = "" 23 | if len(sys.argv) > 1: 24 | supplied_cluster_dir = sys.argv[1] 25 | config_orig = os.path.join(supplied_cluster_dir,"config.yaml") 26 | if not os.path.exists(config_orig): 27 | config_orig = None 28 | 29 | # If merging changes, start by loading default values. Else start with empty dict 30 | if len(sys.argv) > 2: 31 | if sys.argv[2] == "merge": 32 | if config_orig is None: 33 | raise Exception("Invalid cluster directory provided: {}\n\tconfig.yaml could not be found.".format(supplied_cluster_dir)) 34 | with open(config_orig, 'r') as stream: 35 | try: 36 | config_o = yaml.load(stream) 37 | except yaml.YAMLError as exc: 38 | print(exc) 39 | else: 40 | config_o = {} 41 | 42 | 43 | # First accept any changes from supplied config file 44 | config_o.update(config_f) 45 | 46 | # Second accept any changes from supplied config items 47 | config_o.update(config_i) 48 | 49 | # Automatically add the ansible_become if it does not exist, and if we are not root 50 | if not 'ansible_user' in config_o and getpass.getuser() != 'root': 51 | config_o['ansible_user'] = getpass.getuser() 52 | config_o['ansible_become'] = True 53 | 54 | # Detect if a default admin password has been provided by user, set the terraform generated password if not 55 | if ((not 'default_admin_password' in config_f and 56 | not 'default_admin_password' in config_i) or 57 | config_o['default_admin_password'] == ''): 58 | if len(sys.argv) > 3: 59 | config_o['default_admin_password'] = sys.argv[3] 60 | else: 61 | raise Exception("default_admin_password not set and none provided from terraform") 62 | 63 | # to handle terraform bug regarding booleans, we must parse dictionaries to find strings "true" or "false" 64 | # and convert them to booleans. 65 | # Also skip blanks as yaml.safe_dump dumps them as '' which ansible installer does not like 66 | def parsedict(d): 67 | 68 | t_dict = {} 69 | if type(d) is dict: 70 | for key, value in d.iteritems(): 71 | 72 | # Handle nested dictionaries 73 | if type(value) is dict: 74 | t_dict[key] = parsedict(value) 75 | 76 | elif type(value) is str or type(value) is unicode: 77 | if value.lower() == 'true': 78 | t_dict[key] = True 79 | elif value.lower() == 'false': 80 | t_dict[key] = False 81 | elif value == '': 82 | # Skip blanks 83 | continue 84 | else: 85 | t_dict[key] = value 86 | 87 | else: 88 | # We will not look for booleans in lists and such things 89 | t_dict[key] = value 90 | 91 | return t_dict 92 | 93 | 94 | # Write the new configuration 95 | with open(config_orig, 'w') as of: 96 | yaml.safe_dump(parsedict(config_o), of, explicit_start=True, default_flow_style = False) 97 | -------------------------------------------------------------------------------- /scripts/boot-master/load-image.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | LOGFILE=/tmp/loadimage.log 3 | exec 3>&1 4 | exec > >(tee -a ${LOGFILE} >/dev/null) 2> >(tee -a ${LOGFILE} >&3) 5 | 6 | echo "Got the parameters $@" 7 | # Defaults 8 | source /tmp/icp-bootmaster-scripts/functions.sh 9 | source /tmp/icp-bootmaster-scripts/get-args.sh 10 | declare -a locations 11 | 12 | 13 | 14 | 15 | # Figure out the version 16 | # This will populate $org $repo and $tag 17 | parse_icpversion ${icp_inception} 18 | echo "registry=${registry:-not specified} org=$org repo=$repo tag=$tag" 19 | 20 | if [[ ! -z ${icp_inception} ]]; then 21 | # Figure out the version 22 | # This will populate $org $repo and $tag 23 | parse_icpversion ${icp_inception} 24 | echo "registry=${registry:-not specified} org=$org repo=$repo tag=$tag" 25 | fi 26 | 27 | # Allow downloading multiple tarballs, 28 | # which is required in multi-arch deployments 29 | for image_location in ${locations[@]} ; do 30 | imagedir=${cluster_dir}/images 31 | ensure_directory_reachable ${imagedir} 32 | 33 | # Detect if image is local, or else which protocol to use 34 | if [[ "${image_location}" == "/" ]]; then 35 | # Tarball already exists locally, use that directly 36 | echo "${image_location} is deemed to be local, setting image_file" 37 | image_file=${image_location} 38 | 39 | elif [[ "${image_location:0:4}" == "http" ]]; then 40 | # Extract filename from URL if possible 41 | if [[ "${image_location: -2}" == "gz" ]]; then 42 | # Assume a sensible filename can be extracted from URL 43 | filename=$(basename ${image_location}) 44 | else 45 | # TODO We might be able to use some magic to extract actual filename. 46 | # For now, hard-code for x86 47 | echo "Not able to determine filename from URL ${image_location}" >&2 48 | filename="ibm-cloud-private-x86_64${tag}.tar.gz" 49 | echo "Set it to ${filename}" >&2 50 | 51 | fi 52 | 53 | # Download the file using auth if provided 54 | echo "Downloading ${image_location}" >&2 55 | echo "This can take a very long time" >&2 56 | wget -nv --continue ${username:+--user} ${username} ${password:+--password} ${password} \ 57 | -O ${imagedir}/${filename} "${image_location}" 58 | 59 | if [[ $? -gt 0 ]]; then 60 | echo "Error downloading ${image_location}" >&2 61 | exit 1 62 | fi 63 | 64 | # Set the image file name if we're on the same platform 65 | if [[ ${filename} =~ .*$(uname -m).* ]]; then 66 | echo "Setting image_file to ${imagedir}/${filename}" 67 | image_file="${imagedir}/${filename}" 68 | fi 69 | else 70 | # Assume NFS since this is the only other supported protocol 71 | # Separate out the filename and path 72 | nfs_mount=$(dirname ${image_location}) 73 | image_file="${imagedir}/$(basename ${image_location})" 74 | # Mount 75 | sudo mount.nfs $nfs_mount $imagedir 76 | 77 | fi 78 | done 79 | 80 | # Load the offline tarball to get the inception image 81 | if [[ -s "$image_file" ]] 82 | then 83 | echo "Loading image file ${image_file}. This can take a very long time" >&2 84 | # If we have pv installed, we can use that for improved reporting 85 | if which pv >>/dev/null; then 86 | pv --interval 30 ${image_file} | tar zxf - -O | sudo docker load >&2 87 | else 88 | tar xf ${image_file} -O | sudo docker load >&2 89 | fi 90 | else 91 | # If we don't have an image locally we'll pull from docker registry 92 | if [[ -z $(docker images -q ${registry}${registry:+/}${org}/${repo}:${tag}) ]]; then 93 | # If this is a private registry we may need to log in 94 | [[ ! -z "$username" ]] && [[ ! -z "$password" ]] && docker login -u ${username} -p ${password} ${registry} 95 | # ${registry}${registry:+/} adds / only if registry is specified 96 | docker pull ${registry}${registry:+/}${org}/${repo}:${tag} 97 | fi 98 | fi 99 | -------------------------------------------------------------------------------- /scripts/boot-master/parse-hostgroups.py: -------------------------------------------------------------------------------- 1 | import os, sys, json, ConfigParser 2 | 3 | hgfile = '/tmp/icp-host-groups.json' 4 | hostfile = None 5 | ipfile = '/tmp/cluster-ips.txt' 6 | 7 | supplied_cluster_dir = "" 8 | if len(sys.argv) > 1: 9 | supplied_cluster_dir = sys.argv[1] 10 | hostfile = os.path.join(supplied_cluster_dir,"hosts") 11 | if not os.path.isdir(supplied_cluster_dir): 12 | hostfile = None 13 | 14 | if hostfile is None: 15 | raise Exception("Invalid cluster directory provided: {}".format(supplied_cluster_dir)) 16 | 17 | # Exit if we don't need to do anything 18 | if os.stat(hgfile).st_size == 0: 19 | exit(1) 20 | 21 | # Load the hostgroup info from file if provided 22 | with open(hgfile, 'r') as stream: 23 | hostgroups = json.load(stream) 24 | 25 | # Create the hostfile 26 | hf = open(hostfile, 'w') 27 | h = ConfigParser.ConfigParser(allow_no_value=True) 28 | 29 | ips = [] 30 | for group in hostgroups.keys(): 31 | h.add_section(group) 32 | for host in hostgroups[group]: 33 | ips.append(host) 34 | h.set(group, host) 35 | 36 | h.write(hf) 37 | hf.close() 38 | 39 | # Write a list of ip addresses, removing duplicates 40 | i = open(ipfile, 'w') 41 | i.write(",".join(list(set(ips)))) 42 | i.close() 43 | -------------------------------------------------------------------------------- /scripts/boot-master/scaleworkers.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source /tmp/icp-bootmaster-scripts/get-args.sh 3 | source /tmp/icp-bootmaster-scripts/functions.sh 4 | 5 | NEWLIST=/tmp/workerlist.txt 6 | OLDLIST=${cluster_dir}/workerlist.txt 7 | 8 | # Figure out the version 9 | # This will populate $org $repo and $tag 10 | parse_icpversion ${1} 11 | 12 | # Compare new and old list of workers 13 | declare -a newlist 14 | IFS=', ' read -r -a newlist <<< $(cat ${NEWLIST}) 15 | 16 | declare -a oldlist 17 | IFS=', ' read -r -a oldlist <<< $(cat ${OLDLIST}) 18 | 19 | declare -a added 20 | declare -a removed 21 | 22 | # As a precausion, if either list is empty, something might have gone wrong and we should exit in case we delete all nodes in error 23 | # When using hostgroups this is expected behaviour, so need to exit 0 rather than cause error. 24 | if [ ${#newlist[@]} -eq 0 ]; then 25 | echo "Couldn't find any entries in new list of workers. Exiting'" 26 | exit 0 27 | fi 28 | if [ ${#oldlist[@]} -eq 0 ]; then 29 | echo "Couldn't find any entries in old list of workers. Exiting'" 30 | exit 0 31 | fi 32 | 33 | 34 | # Cycle through old ips to find removed workers 35 | for oip in "${oldlist[@]}"; do 36 | if [[ "${newlist[@]}" =~ "${oip}" ]]; then 37 | echo "${oip} is still here" 38 | fi 39 | 40 | if [[ ! " ${newlist[@]} " =~ " ${oip} " ]]; then 41 | # whatever you want to do when arr doesn't contain value 42 | removed+=(${oip}) 43 | fi 44 | done 45 | 46 | # Cycle through new ips to find added workers 47 | for nip in "${newlist[@]}"; do 48 | if [[ "${oldlist[@]}" =~ "${nip}" ]]; then 49 | echo "${nip} is still here" 50 | fi 51 | 52 | if [[ ! " ${oldlist[@]} " =~ " ${nip} " ]]; then 53 | # whatever you want to do when arr doesn't contain value 54 | added+=(${nip}) 55 | fi 56 | done 57 | 58 | 59 | 60 | if [[ -n ${removed} ]] 61 | then 62 | echo "removing workers: ${removed[@]}" 63 | 64 | ### Setup kubectl 65 | 66 | # use kubectl from container 67 | kubectl="sudo docker run -e LICENSE=accept --net=host -v ${cluster_dir}:/installer/cluster -v /root:/root ${registry}${registry:+/}${org}/${repo}:${tag} kubectl" 68 | 69 | $kubectl config set-cluster cfc-cluster --server=https://localhost:8001 --insecure-skip-tls-verify=true 70 | $kubectl config set-context kubectl --cluster=cfc-cluster 71 | $kubectl config set-credentials user --client-certificate=/installer/cluster/cfc-certs/kubecfg.crt --client-key=/installer/cluster/cfc-certs/kubecfg.key 72 | $kubectl config set-context kubectl --user=user 73 | $kubectl config use-context kubectl 74 | 75 | list=$(IFS=, ; echo "${removed[*]}") 76 | 77 | for ip in "${removed[@]}"; do 78 | $kubectl delete node $ip 79 | sudo sed -i "/^${ip} /d" /etc/hosts 80 | sudo sed -i "/^${ip} /d" ${cluster_dir}/hosts 81 | done 82 | 83 | fi 84 | 85 | if [[ -n ${added} ]] 86 | then 87 | echo "Adding: ${added[@]}" 88 | # Collect node names 89 | 90 | # Update /etc/hosts 91 | for node in "${added[@]}" ; do 92 | nodename=$(ssh -o StrictHostKeyChecking=no -i ${cluster_dir}/ssh_key ${node} hostname) 93 | printf "%s %s\n" "$node" "$nodename" | cat - /etc/hosts | sudo sponge /etc/hosts 94 | printf "%s %s\n" "$node" "$nodename" | ssh -o StrictHostKeyChecking=no -i ${cluster_dir}/ssh_key ${node} 'cat - /etc/hosts | sudo sponge /etc/hosts' 95 | done 96 | 97 | list=$(IFS=, ; echo "${added[*]}") 98 | docker run -e LICENSE=accept --net=host -v "${cluster_dir}":/installer/cluster \ 99 | ${registry}${registry:+/}${org}/${repo}:${tag} install -l ${list} 100 | fi 101 | 102 | 103 | # Backup the origin list and replace 104 | mv ${OLDLIST} ${OLDLIST}-$(date +%Y%m%dT%H%M%S) 105 | mv ${NEWLIST} ${OLDLIST} 106 | -------------------------------------------------------------------------------- /scripts/boot-master/start_install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source /tmp/icp-bootmaster-scripts/get-args.sh 3 | source /tmp/icp-bootmaster-scripts/functions.sh 4 | 5 | # If loaded from tarball, icp version may not be specified in terraform 6 | if [[ -z "${icp_version}" ]]; then 7 | icp_version=$(get_inception_image) 8 | fi 9 | 10 | # Figure out the version 11 | # This will populate $org $repo and $tag 12 | parse_icpversion ${icp_version} 13 | echo "registry=${registry:-not specified} org=$org repo=$repo tag=$tag" 14 | 15 | docker run -e LICENSE=accept -e ANSIBLE_CALLBACK_WHITELIST=profile_tasks,timer --net=host -t -v ${cluster_dir}:/installer/cluster ${registry}${registry:+/}${org}/${repo}:${tag} ${install_command} ${log_verbosity} |& tee /tmp/icp-${install_command}-log.txt 16 | 17 | exit ${PIPESTATUS[0]} 18 | -------------------------------------------------------------------------------- /scripts/common/docker-user.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Some RHEL based installations may not have docker installed yet. 4 | # Only aattempt to add user to group if docker is installed and the user is not root 5 | if grep -q docker /etc/group 6 | then 7 | iam=$(whoami) 8 | 9 | if [[ $iam != "root" ]] 10 | then 11 | sudo usermod -a -G docker $iam 12 | fi 13 | fi 14 | -------------------------------------------------------------------------------- /scripts/common/prereqs.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #This script updates hosts with required prereqs 3 | #if [ $# -lt 1 ]; then 4 | # echo "Usage $0 " 5 | # exit 1 6 | #fi 7 | 8 | #HOSTNAME=$1 9 | 10 | LOGFILE=/tmp/prereqs.log 11 | exec 3>&1 12 | exec > >(tee -a ${LOGFILE} >/dev/null) 2> >(tee -a ${LOGFILE} >&3) 13 | 14 | #Find Linux Distro 15 | if grep -q -i ubuntu /etc/*release 16 | then 17 | OSLEVEL=ubuntu 18 | else 19 | OSLEVEL=other 20 | fi 21 | echo "Operating System is $OSLEVEL" 22 | 23 | ubuntu_install(){ 24 | packages_to_check="\ 25 | python-yaml \ 26 | thin-provisioning-tools \ 27 | lvm2" 28 | sudo sysctl -w vm.max_map_count=262144 29 | packages_to_install="" 30 | 31 | for package in ${packages_to_check}; do 32 | if ! dpkg -l ${package} &> /dev/null; then 33 | packages_to_install="${packages_to_install} ${package}" 34 | fi 35 | done 36 | 37 | if [ ! -z "${packages_to_install}" ]; then 38 | # attempt to install, probably won't work airgapped but we'll get an error immediately 39 | echo "Attempting to install: ${packages_to_install} ..." 40 | retries=20 41 | sudo apt-get update 42 | while [ $? -ne 0 -a "$retries" -gt 0 ]; do 43 | retries=$((retries-1)) 44 | echo "Another process has acquired the apt-get update lock; waiting 10s" >&2 45 | sleep 10; 46 | sudo apt-get update 47 | done 48 | if [ $? -ne 0 -a "$retries" -eq 0 ] ; then 49 | echo "Maximum number of retries (${retries}) for apt-get update attempted; quitting" >&2 50 | exit 1 51 | fi 52 | 53 | retries=20 54 | sudo apt-get install -y ${packages_to_install} 55 | while [ $? -ne 0 -a "$retries" -gt 0 ]; do 56 | retries=$((retries-1)) 57 | echo "Another process has acquired the apt-get install/upgrade lock; waiting 10s" >&2 58 | sleep 10; 59 | sudo apt-get install -y ${packages_to_install} 60 | done 61 | if [ $? -ne 0 -a "$retries" -eq 0 ] ; then 62 | echo "Maximum number of retries (20) for apt-get install attempted; quitting" >&2 63 | exit 1 64 | fi 65 | fi 66 | } 67 | 68 | crlinux_install(){ 69 | packages_to_check="\ 70 | PyYAML \ 71 | device-mapper \ 72 | libseccomp \ 73 | libtool-ltdl \ 74 | libcgroup \ 75 | iptables \ 76 | device-mapper-persistent-data \ 77 | lvm2" 78 | 79 | for package in ${packages_to_check}; do 80 | if ! rpm -q ${package} &> /dev/null; then 81 | packages_to_install="${packages_to_install} ${package}" 82 | fi 83 | done 84 | 85 | if [ ! -z "${packages_to_install}" ]; then 86 | # attempt to install, probably won't work airgapped but we'll get an error immediately 87 | echo "Attempting to install: ${packages_to_install} ..." 88 | sudo yum install -y ${packages_to_install} 89 | fi 90 | } 91 | 92 | if [ "$OSLEVEL" == "ubuntu" ]; then 93 | ubuntu_install 94 | else 95 | crlinux_install 96 | fi 97 | 98 | echo "Complete.." 99 | exit 0 100 | -------------------------------------------------------------------------------- /scripts/common/version-specific.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ver=$1 4 | SCRIPT=$(realpath -s $0) 5 | SCRIPTPATH=$(dirname $SCRIPT) 6 | 7 | for SCRIPT in ${SCRIPTPATH}/${ver}-* 8 | do 9 | if [ -f $SCRIPT -a -x $SCRIPT ] 10 | then 11 | source $SCRIPT 12 | fi 13 | done 14 | 15 | exit 0 16 | -------------------------------------------------------------------------------- /tests/README.md: -------------------------------------------------------------------------------- 1 | # Tests for the deploy module 2 | 3 | Various unit and validation tests for the module. 4 | 5 | Most tests run individual scripts in containers to validate expected behavior. 6 | This allows the scripts to easily be validated in a number of linux distributions and versions. 7 | 8 | There is a helper script `container-runtest.sh` which will run inside the test container. 9 | The script will setup the directory structure, install pre-requisites and run validation tests. 10 | `container-runtest.sh` will return the sum of tested script exit code + validation code exit code. 11 | 12 | ## Pre-requisits 13 | 14 | To run the tests you'll need the following installed 15 | 16 | - [BATS](https://github.com/bats-core/bats-core) 17 | - [Terraform](https://www.terraform.io/) 18 | - Docker (it must be running) 19 | 20 | ## Running tests 21 | 22 | To run all existing tests 23 | ``` 24 | ./run_bats.sh all 25 | ``` 26 | 27 | To run single test, for example load-image 28 | ``` 29 | ./run_bats.sh load-image.bats 30 | ``` 31 | 32 | 33 | Most tests are run in docker containers, so you can monitor the tests by running 34 | `docker logs -f ` on the active container 35 | -------------------------------------------------------------------------------- /tests/container-runtest.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ##################################### 3 | ## Script to run a script inside a docker container 4 | ## 5 | ## Will install any package in YUM_PREREQS or APT_PREREQS environment variables 6 | ## prior to running any other script 7 | ## 8 | ## Inputs 9 | ## -s