├── .github └── ISSUE_TEMPLATE │ ├── bug_report.md │ └── feature_request.md ├── .gitignore ├── CHANGELOG.md ├── LICENSE ├── README.md ├── SECURITY.md ├── ansible ├── README.md ├── playbooks │ └── create.yml └── vars.example.yml ├── azuredeploy.json ├── azuredeploy.parameters.json ├── azuredeploy.parameters.sample.json ├── azuredeployinfraonly.json ├── azuredeployinfraonlyc1.json ├── images ├── openshiftdiagram.jpg └── openshiftiambcd.jpg ├── nested ├── bastionprep.json ├── gallerybasic.json ├── galleryclustercnsnode.json ├── galleryclusternode.json ├── masterprep.json ├── nodeprep.json └── openshiftdeploy.json └── scripts ├── bastionPrep.sh ├── deployOpenShift.sh ├── infraPrep.sh ├── masterPrep.sh └── nodePrep.sh /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **stdout** 27 | Include the last 100 lines of stdout from Bastion host - see troubleshooting https://docs.microsoft.com/en-us/azure/virtual-machines/linux/openshift-troubleshooting 28 | 29 | **Template Information (please complete the following information):** 30 | - OS: [e.g. iOS] 31 | - Branch: [e.g. master] 32 | 33 | **Additional context** 34 | Add any other context about the problem here. 35 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | 4 | # User-specific files 5 | *.suo 6 | *.user 7 | *.userosscache 8 | *.sln.docstates 9 | 10 | # User-specific files (MonoDevelop/Xamarin Studio) 11 | *.userprefs 12 | 13 | # Build results 14 | [Dd]ebug/ 15 | [Dd]ebugPublic/ 16 | [Rr]elease/ 17 | [Rr]eleases/ 18 | x64/ 19 | x86/ 20 | bld/ 21 | [Bb]in/ 22 | [Oo]bj/ 23 | [Ll]og/ 24 | 25 | # Visual Studio 2015 cache/options directory 26 | .vs/ 27 | # Uncomment if you have tasks that create the project's static files in wwwroot 28 | #wwwroot/ 29 | 30 | # MSTest test Results 31 | [Tt]est[Rr]esult*/ 32 | [Bb]uild[Ll]og.* 33 | 34 | # NUNIT 35 | *.VisualState.xml 36 | TestResult.xml 37 | 38 | # Build Results of an ATL Project 39 | [Dd]ebugPS/ 40 | [Rr]eleasePS/ 41 | dlldata.c 42 | 43 | # DNX 44 | project.lock.json 45 | artifacts/ 46 | 47 | *_i.c 48 | *_p.c 49 | *_i.h 50 | *.ilk 51 | *.meta 52 | *.obj 53 | *.pch 54 | *.pdb 55 | *.pgc 56 | *.pgd 57 | *.rsp 58 | *.sbr 59 | *.tlb 60 | *.tli 61 | *.tlh 62 | *.tmp 63 | *.tmp_proj 64 | *.log 65 | *.vspscc 66 | *.vssscc 67 | .builds 68 | *.pidb 69 | *.svclog 70 | *.scc 71 | 72 | # Chutzpah Test files 73 | _Chutzpah* 74 | 75 | # Visual C++ cache files 76 | ipch/ 77 | *.aps 78 | *.ncb 79 | *.opendb 80 | *.opensdf 81 | *.sdf 82 | *.cachefile 83 | *.VC.db 84 | *.VC.VC.opendb 85 | 86 | # Visual Studio profiler 87 | *.psess 88 | *.vsp 89 | *.vspx 90 | *.sap 91 | 92 | # TFS 2012 Local Workspace 93 | $tf/ 94 | 95 | # Guidance Automation Toolkit 96 | *.gpState 97 | 98 | # ReSharper is a .NET coding add-in 99 | _ReSharper*/ 100 | *.[Rr]e[Ss]harper 101 | *.DotSettings.user 102 | 103 | # JustCode is a .NET coding add-in 104 | .JustCode 105 | 106 | # TeamCity is a build add-in 107 | _TeamCity* 108 | 109 | # DotCover is a Code Coverage Tool 110 | *.dotCover 111 | 112 | # NCrunch 113 | _NCrunch_* 114 | .*crunch*.local.xml 115 | nCrunchTemp_* 116 | 117 | # MightyMoose 118 | *.mm.* 119 | AutoTest.Net/ 120 | 121 | # Web workbench (sass) 122 | .sass-cache/ 123 | 124 | # Installshield output folder 125 | [Ee]xpress/ 126 | 127 | # DocProject is a documentation generator add-in 128 | DocProject/buildhelp/ 129 | DocProject/Help/*.HxT 130 | DocProject/Help/*.HxC 131 | DocProject/Help/*.hhc 132 | DocProject/Help/*.hhk 133 | DocProject/Help/*.hhp 134 | DocProject/Help/Html2 135 | DocProject/Help/html 136 | 137 | # Click-Once directory 138 | publish/ 139 | 140 | # Publish Web Output 141 | *.[Pp]ublish.xml 142 | *.azurePubxml 143 | # TODO: Comment the next line if you want to checkin your web deploy settings 144 | # but database connection strings (with potential passwords) will be unencrypted 145 | *.pubxml 146 | *.publishproj 147 | 148 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 149 | # checkin your Azure Web App publish settings, but sensitive information contained 150 | # in these scripts will be unencrypted 151 | PublishScripts/ 152 | 153 | # NuGet Packages 154 | *.nupkg 155 | # The packages folder can be ignored because of Package Restore 156 | **/packages/* 157 | # except build/, which is used as an MSBuild target. 158 | !**/packages/build/ 159 | # Uncomment if necessary however generally it will be regenerated when needed 160 | #!**/packages/repositories.config 161 | # NuGet v3's project.json files produces more ignoreable files 162 | *.nuget.props 163 | *.nuget.targets 164 | 165 | # Microsoft Azure Build Output 166 | csx/ 167 | *.build.csdef 168 | 169 | # Microsoft Azure Emulator 170 | ecf/ 171 | rcf/ 172 | 173 | # Windows Store app package directories and files 174 | AppPackages/ 175 | BundleArtifacts/ 176 | Package.StoreAssociation.xml 177 | _pkginfo.txt 178 | 179 | # Visual Studio cache files 180 | # files ending in .cache can be ignored 181 | *.[Cc]ache 182 | # but keep track of directories ending in .cache 183 | !*.[Cc]ache/ 184 | 185 | # Others 186 | ClientBin/ 187 | ~$* 188 | *~ 189 | *.dbmdl 190 | *.dbproj.schemaview 191 | *.pfx 192 | *.publishsettings 193 | node_modules/ 194 | orleans.codegen.cs 195 | 196 | # Since there are multiple workflows, uncomment next line to ignore bower_components 197 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 198 | #bower_components/ 199 | 200 | # RIA/Silverlight projects 201 | Generated_Code/ 202 | 203 | # Backup & report files from converting an old project file 204 | # to a newer Visual Studio version. Backup files are not needed, 205 | # because we have git ;-) 206 | _UpgradeReport_Files/ 207 | Backup*/ 208 | UpgradeLog*.XML 209 | UpgradeLog*.htm 210 | 211 | # SQL Server files 212 | *.mdf 213 | *.ldf 214 | 215 | # Business Intelligence projects 216 | *.rdl.data 217 | *.bim.layout 218 | *.bim_*.settings 219 | 220 | # Microsoft Fakes 221 | FakesAssemblies/ 222 | 223 | # GhostDoc plugin setting file 224 | *.GhostDoc.xml 225 | 226 | # Node.js Tools for Visual Studio 227 | .ntvs_analysis.dat 228 | 229 | # Visual Studio 6 build log 230 | *.plg 231 | 232 | # Visual Studio 6 workspace options file 233 | *.opt 234 | 235 | # Visual Studio LightSwitch build output 236 | **/*.HTMLClient/GeneratedArtifacts 237 | **/*.DesktopClient/GeneratedArtifacts 238 | **/*.DesktopClient/ModelManifest.xml 239 | **/*.Server/GeneratedArtifacts 240 | **/*.Server/ModelManifest.xml 241 | _Pvt_Extensions 242 | 243 | # Paket dependency manager 244 | .paket/paket.exe 245 | paket-files/ 246 | 247 | # FAKE - F# Make 248 | .fake/ 249 | 250 | # JetBrains Rider 251 | .idea/ 252 | *.sln.iml 253 | 254 | # Ansible 255 | vars.yml 256 | *.retry 257 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | This CHANGELOG.md file will contain the update log for the latest set of updates to the templates 2 | 3 | 4 | # UPDATES for Master (Release 3.11) - November 19, 2018 5 | 6 | 1. Update to deploy 3.11 7 | 8 | 9 | # UPDATES for Master (Release 3.10) - November 19, 2018 10 | 11 | 1. Add support for custom SSL Certificates 12 | 2. Add support for proxy settings 13 | 3. Change configuration for networking. For new vNet, supply vNet name, address CIDR and all subnet information 14 | 4. For existing vNet, provide full resource ID of each subnet. 15 | 16 | # UPDATES for Master (Release 3.10) - September 13, 2018 17 | 18 | 1. Update to deploy 3.10 19 | 2. Add support for 3rd party marketplace image 20 | 3. Add support for broker pool ID in addition to compute pool ID 21 | 22 | 23 | # UPDATES for Master (Release 3.9) - August 28, 2018 24 | 25 | 1. Lock version to 3.9.33 - Azure Cloud Provider setup issues in 3.9.40 26 | 27 | 28 | # UPDATES for Master (Release 3.9) - August 6, 2018 29 | 30 | 1. Added support for private master nodes 31 | 2. Addes support for private infra nodes 32 | 3. Removed inbound NAT rules for master LB to better secure master nodes 33 | 34 | 35 | # UPDATES for Master (Release 3.9) - July 14, 2018 36 | 37 | 1. Added support for Accelerated Networking 38 | 2. Added support for existing or new VNet 39 | 40 | 41 | # UPDATES for Master (Release 3.9) - May 22, 2018 42 | 43 | 1. Added parameter for CNS VM Size 44 | 2. Added support for non-HA masters by allowing a single master 45 | 3. Cleaned up Azure Cloud Provider configuration 46 | 47 | 48 | # UPDATES for Master (Release 3.9) - May 19, 2018 49 | 50 | 1. Updated scripts to support 3.9.27 51 | 2. Added Support for RHEL 7.5 52 | 3. Added Container Native Storage (CNS) support 53 | 4. Added support for custom IP range for the Virtual Network 54 | 55 | # UPDATES for Master (Release 3.9) - March 28, 2018 56 | 57 | 1. Create Release 3.9 Branch 58 | 2. Updating scripts for 3.9 repository 59 | 3. Switch to port 443 for web console 60 | 4. Remove old unused resources 61 | 62 | 63 | # UPDATES for Master (Release 3.7) - February 14, 2018 64 | 65 | 1. Created Release 3.7 Branch 66 | 2. Update deployOpenShift.sh file to separate out Ansible Playbooks 67 | 3. Created separate repo for OpenShift installation Playbooks 68 | 69 | 70 | # UPDATES for Release 3.7 - January 12, 2018 71 | 72 | 1. Inject the Private Key into Bastion host during prep. 73 | 2. Add support for managed and unmanaged disks. 74 | 3. Update prep script to simplify Cloud Access registration for username/password or activation key/organization id. 75 | 4. Update Azure Cloud Provider playbooks - no need to delete node and include cluster reboot. 76 | 5. Include additional data disk sizes. 77 | 6. Create storage class based on managed or unmanaged disk. 78 | 7. General cleanup. 79 | 80 | 81 | # UPDATES for Release 3.6 - September 29, 2017 82 | 83 | 1. Removed installation of Azure CLI as this is no longer needed. 84 | 2. Removed dnslabel parameters and made them variables to simplify deployment. 85 | 3. Added new D2-64v3, D2s-64sv3, E2-64v3, and E2s-64sv3 VM types. 86 | 4. Updated prep scripts to include additional documented pre-requisites. 87 | 5. Set OS disk size to 64 GB and updated prep scripts to expand root partition. 88 | 6. Removed option to install single master cluster. Now supports 3 or 5 Masters and 2 or 3 Infra nodes. 89 | 7. Configure RHEL to use NetworkManager on eth0. 90 | 8. Added additional troubleshooting for Azure Cloud Configuration playbooks (Exit Codes 7 - 10). 91 | 9. Updated to latest versions of APIs - includes reworking of Storage Account creation. 92 | 10. Bastion Host - separate Storage Account and VM size definition. 93 | 11. Enabled Diagnostics Storage for all VMs. 94 | 12. Added Tags to all resources. 95 | 13. Switched to nip.io (versus xip.io). 96 | 14. Added option to enable Azure Cloud Provider (true or false). 97 | 15. Moved Metric and Logging setup to post cluster install. 98 | 16. General cleanup (removed unnecessary resources, variables, etc.). 99 | 100 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Microsoft 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # OpenShift Container Platform 3 Deployment Template 2 | 3 | ## NOTE: Structure of Repo 4 | 5 | **The Master branch has been updated to deploy version 3.11** 6 | 7 | **MAJOR UPDATES HAVE BEEN MADE - READ BEFORE DEPLOYING** 8 | 9 | The master branch contains the most current release of OpenShift Container Platform 3, which is currently version 3.11. We will maintain the templates for the current version of OCP only, as version 3.10 is [no longer commercially suported by Red Hat](https://access.redhat.com/support/policy/updates/openshift_noncurrent). The older branches will not be deleted but will no longer be maintained or updated. 10 | 11 | New as of August 27, 2019: I have added the azurestack-release-3.11 branch with templates and scripts for deploying OCP 3.11 to Azure Stack. 12 | 13 | The following branches exist: 14 | 15 | **Commercial Azure** 16 | - Release-3.6 (As is; no longer updated) 17 | - Release-3.7 (As is; no longer updated) 18 | - Release-3.9 (As is; no longer updated) 19 | - Release-3.10 (As is; no longer updated) 20 | 21 | **Azure Stack** 22 | - azurestack-release-3.7 (As is; no longer updated) 23 | - azurestack-release-3.9 (As is; no longer updated) 24 | - azurestack-release-3.11 25 | 26 | Bookmark [aka.ms/OpenShift](http://aka.ms/OpenShift) for future reference. 27 | 28 | **For OpenShift Origin refer to https://github.com/Microsoft/openshift-origin** 29 | 30 | ## OpenShift Container Platform 3.11 with Username / Password authentication for OpenShift 31 | 32 | 1. Single master option available 33 | 2. VM types that support Accelerated Networking will automatically have this feature enabled 34 | 3. Custom and existing Vnet 35 | 4. Support cluster with private masters (no public IP on load balancer in front of master nodes) 36 | 5. Support cluster with private router (no public IP on load balancer in front of infra nodes) 37 | 6. Support broker pool ID (for master and infra nodes) along with compute pool ID (for compute nodes) 38 | 7. Support for default gallery RHEL On Demand image and 3rd party Marketplace offer such as BYOS image in Private Marketplace 39 | 8. Support self-signed certificates or custom SSL certificates for master load balancer (Web Console) 40 | 9. Support self-signed certificates or custom SSL certificates for infra load balancer (Router) 41 | 42 | 43 | This template deploys OpenShift Container Platform with basic username / password for authentication to OpenShift. It includes the following resources: 44 | 45 | |Resource |Properties | 46 | |-----------------------|------------------------------------------------------------------------------------------------------------------------------------| 47 | |Virtual Network
Default |**Address prefix:** 10.0.0.0/14
**Master subnet:** 10.1.0.0/16
**Infra subnet:** 10.2.0.0/16
**Node subnet:** 10.3.0.0/16 | 48 | |Virtual Network
Custom |**Address prefix:** Your Choice
**Master subnet:** Your Choice
**Infra subnet:** Your Choice
**CNS subnet:** Your Choice
**Node subnet:** Your Choice | 49 | |Master Load Balancer |1 probe and 1 rule for TCP 443 | 50 | |Infra Load Balancer |2 probes and 2 rules for TCP 80 and TCP 443 | 51 | |Public IP Addresses |Bastion Public IP for Bastion Node
OpenShift Master public IP attached to Master Load Balancer (if masters are public)
OpenShift Router public IP attached to Infra Load Balancer (if router is public) | 52 | |Storage Accounts
Unmanaged Disks |1 Storage Account for Bastion VM
1 Storage Account for Master VMs
1 Storage Account for Infra VMs
2 Storage Accounts for Node VMs
2 Storage Accounts for Diagnostics Logs
1 Storage Account for Private Docker Registry | 53 | |Storage Accounts
Managed Disks |2 Storage Accounts for Diagnostics Logs
1 Storage Account for Private Docker Registry | 54 | |Network Security Groups|1 Network Security Group for Bastion VM
1 Network Security Group Master VMs
1 Network Security Group for Infra VMs
1 Network Security Group for CNS VMs (if CNS enabled)
1 Network Security Group for Node VMs | 55 | |Availability Sets |1 Availability Set for Master VMs
1 Availability Set for Infra VMs
1 Availability Set for CNS VMs (if CNS enabled)
1 Availability Set for Node VMs | 56 | |Virtual Machines |1 Bastion Node - Used to run ansible playbook for OpenShift deployment
1, 3 or 5 Master Nodes
1, 2 or 3 Infra Nodes
3 or 4 CNS Nodes (if CNS enabled)
User-defined number of Nodes (1 to 30)
All VMs include a single attached data disk for Docker thin pool logical volume
CNS VMs include 3 additional data disks for glusterfs storage (if CNS enabled)| 57 | 58 | ![Cluster Diagram](images/openshiftdiagram.jpg) 59 | 60 | ## READ the instructions in its entirety before deploying! 61 | 62 | Additional documentation for deploying OpenShift in Azure can be found here: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/openshift-get-started 63 | 64 | This template deploys multiple VMs and requires some pre-work before you can successfully deploy the OpenShift Cluster. If you don't complete the pre-work correctly, you will most likely fail to deploy the cluster using this template. Please read the instructions completely before you proceed. 65 | 66 | By default, this template uses the On-Demand Red Hat Enterprise Linux image from the Azure Gallery. 67 | >When using the On-Demand image, there is an additional hourly RHEL subscription charge for using this image on top of the normal compute, network and storage costs. At the same time, the instance will be registered to your Red Hat subscription, so you will also be using one of your entitlements. This will lead to "double billing". To avoid this, you would need to build your own RHEL image, which is defined in [this Red Hat KB article](https://access.redhat.com/articles/uploading-rhel-image-to-azure). 68 | 69 | If you have a valid Red Hat subscription, register for Cloud Access and [request access](http://aka.ms/rhel-byos) to the BYOS RHEL image in the Private Azure Marketplace to avoid the double billing. To use a 3rd party marketplace offer (such as the BYOS private image), you need to provide the following information for the offer - publisher, offer, sku, and version. You also need to enable the offer for programmatic deployment. 70 | 71 | If you are only using one pool ID for all nodes, then enter the same pool ID for both 'rhsmPoolId' and 'rhsmBrokerPoolId'. 72 | 73 | **Private Clusters** 74 | 75 | Deploying private OpenShift clusters requires more than just not having a public IP associated to the master load balancer (web console) or to the infra load balancer (router). A private cluster generally uses a custom DNS server (not the default Azure DNS), a custom domain name (such as contoso.com), and pre-defined virtual network(s). For private clusters, you will need to configure your virtual network with all the appropriate subnets and DNS server settings in advance. Then use **existingMasterSubnetReference**, **existingInfraSubnetReference**, **existingCnsSubnetReference**, and **existingNodeSubnetReference** to specify the existing subnet for use by the cluster. 76 | 77 | If private masters is selected (**masterClusterType**=private), a static private IP needs to be specified for **masterPrivateClusterIp** which will be assigned to the front end of the master load balancer. This must be within the CIDR for the master subnet and not already in use. **masterClusterDnsType** must be set to "custom" and the master DNS name must be provided for **masterClusterDns** and this needs to map to the static Private IP and will be used to access the console on the master nodes. 78 | 79 | If private router is selected (**routerClusterType**=private), a static private IP needs to be specified for **routerPrivateClusterIp** which will be assigned to the front end of the infra load balancer. This must be within the CIDR for the infra subnet and not already in use. **routingSubDomainType** must be set to "custom" and the wildcard DNS name for routing must be provided for **routingSubDomain**. 80 | 81 | If private masters and private router is selected, the custom domain name must also be entered for **domainName** 82 | 83 | After successful deployment, the Bastion Node is the only node with a public IP that you can ssh into. Even if the master nodes are configured for public access, they are not exposed for ssh access. 84 | 85 | ## Prerequisites 86 | 87 | ### Create Key Vault to store secret based information 88 | 89 | You will need to create a Key Vault to store various secret information that will then be used as part of the deployment so that the information is not exposed via the parameters file. Secrets will need to be created for the SSH private key (**sshPrivateKey**), Azure AD client secret (**aadClientSecret**), OpenShift admin password (**openshiftPassword**), and Red Hat Subscription Manager password or activation key (**rhsmPasswordOrActivationKey**). Additionally, if custom SSL certificates are used, then 6 additional secrets will need to be created - **routingcafile**, **routingcertfile**, **routingkeyfile**, **mastercafile**, **mastercertfile**, and **masterkeyfile**. These will be explained in more detail. 90 | 91 | The template references specific secret names so you **must** use the bolded names listed above (case sensitive). 92 | 93 | It is recommend to create a separate Resource Group specifically to store the KeyVault. This way, you can reuse the KeyVault for other deployments and you won't have to create this every time you chose to deploy another OpenShift cluster. 94 | 95 | **Create Key Vault using Azure CLI** 96 | 1. Create new Resource Group: az group create -n \ -l \
97 | Ex: `az group create -n KeyVaultResourceGroupName -l 'East US'`
98 | 1. Create Key Vault: az keyvault create -n \ -g \ -l \ --enabled-for-template-deployment true
99 | Ex: `az keyvault create -n KeyVaultName -g KeyVaultResourceGroupName -l 'East US' --enabled-for-template-deployment true`
100 | 101 | ### Generate SSH Keys 102 | 103 | You'll need to generate an SSH key pair (Public / Private) in order to provision this template. Ensure that you do **NOT** include a passphrase with the private key. 104 | 105 | If you are using a Windows computer, you can download puttygen.exe. You will need to export to OpenSSH (from Conversions menu) to get a valid Private Key for use in the Template. 106 | 107 | From a Linux or Mac, you can just use the ssh-keygen command. Once you are finished deploying the cluster, you can always generate new keys that uses a passphrase and replace the original ones used during initial deployment. 108 | 109 | **Store SSH Private key in Secret** 110 | 111 | 1. Create Secret: az keyvault secret set --vault-name \ -n \ --file \
112 | Ex: `az keyvault secret set --vault-name KeyVaultName -n sshPrivateKey --file ~/.ssh/id_rsa`
113 | 114 | ### Generate Azure Active Directory (AAD) Service Principal 115 | 116 | To configure Azure as the Cloud Provider for OpenShift Container Platform, you will need to create an Azure Active Directory Service Principal. The easiest way to perform this task is via the Azure CLI. Below are the steps for doing this. 117 | 118 | Assigning permissions to the entire Subscription is the easiest method but does give the Service Principal permissions to all resources in the Subscription. Assigning permissions to only the Resource Group is the most secure as the Service Principal is restricted to only that one Resource Group. 119 | 120 | **Azure CLI 2.0** 121 | 122 | 1. **Create Service Principal and assign permissions to Subscription**
123 | a. az ad sp create-for-rbac -n \ --password \ --role contributor --scopes /subscriptions/\
124 | Ex: `az ad sp create-for-rbac -n openshiftcloudprovider --password Pass@word1 --role contributor --scopes /subscriptions/555a123b-1234-5ccc-defgh-6789abcdef01`
125 | 126 | 2. **Create Service Principal and assign permissions to Resource Group**
127 | a. If you use this option, you must have created the Resource Group first. Be sure you don't create any resources in this Resource Group before deploying the cluster.
128 | b. az ad sp create-for-rbac -n \ --password \ --role contributor --scopes /subscriptions/\/resourceGroups/\
129 | Ex: `az ad sp create-for-rbac -n openshiftcloudprovider --password Pass@word1 --role contributor --scopes /subscriptions/555a123b-1234-5ccc-defgh-6789abcdef01/resourceGroups/00000test`
130 | 131 | 3. **Create Service Principal without assigning permissions to Resource Group**
132 | a. If you use this option, you will need to assign permissions to either the Subscription or the newly created Resource Group shortly after you initiate the deployment of the cluster or the post installation scripts will fail when configuring Azure as the Cloud Provider.
133 | b. az ad sp create-for-rbac -n \ --password \ --role contributor --skip-assignment
134 | Ex: `az ad sp create-for-rbac -n openshiftcloudprovider --password Pass@word1 --role contributor --skip-assignment`
135 | 136 | You will get an output similar to: 137 | 138 | ```javascript 139 | { 140 | "appId": "2c8c6a58-44ac-452e-95d8-a790f6ade583", 141 | "displayName": "openshiftcloudprovider", 142 | "name": "http://openshiftcloudprovider", 143 | "password": "Pass@word1", 144 | "tenant": "12a345bc-1234-dddd-12ab-34cdef56ab78" 145 | } 146 | ``` 147 | 148 | The appId is used for the aadClientId parameter. Store the password in the Key Vault. 149 | 150 | ```bash 151 | az keyvault secret set --vault-name KeyVaultName -n aadClientSecret --value Pass@word1 152 | ``` 153 | 154 | ### OpenShift Admin Password 155 | 156 | An initial OpenShift Cluster Admin user will be created after the cluster is deployed. This admin user will need a password. Store the password that you want to use in the Key Vault. 157 | 158 | ```bash 159 | az keyvault secret set --vault-name KeyVaultName -n openshiftPassword --value Pass@word1 160 | ``` 161 | 162 | ### Red Hat Subscription Access 163 | 164 | For security reasons, the method for registering the RHEL system allows the use of an Organization ID and Activation Key as well as a Username and Password. Please know that it is more secure to use the Organization ID and Activation Key. 165 | 166 | You can determine your Organization ID by running ```subscription-manager identity``` on a registered machine. To create or find your Activation Key, please go here: https://access.redhat.com/management/activation_keys. 167 | 168 | You will also need to get the Pool ID that contains your entitlements for OpenShift. You can retrieve this from the Red Hat portal by examining the details of the subscription that has the OpenShift entitlements. Or you can contact your Red Hat administrator to help you. 169 | 170 | Store the password or activation key that you want to use in the Key Vault. 171 | 172 | ```bash 173 | az keyvault secret set --vault-name KeyVaultName -n rhsmPasswordOrActivationKey --value Pass@word1 174 | ``` 175 | 176 | ### Custom Certificates 177 | 178 | By default, the template will deploy an OpenShift cluster using self-signed certificates for the OpenShift web console and the routing domain. If you want to use custom SSL certificates, set 'routingCertType' to 'custom' and 'masterCertType' to 'custom'. You will need the CA, Cert, and Key files in .pem format for the certificates. 179 | 180 | You will need to store these files in Key Vault secrets. Use the same Key Vault as the one used for the private key. Rather than require 6 additional inputs for the secret names, the template is hard-coded to use specific secret names for each of the SSL certificate files. Store the certficiate data using the information from the following table. 181 | 182 | | Secret Name | Certificate file | 183 | |------------------|--------------------| 184 | | mastercafile | master CA file | 185 | | mastercertfile | master CERT file | 186 | | masterkeyfile | master Key file | 187 | | routingcafile | routing CA file | 188 | | routingcertfile | routing CERT file | 189 | | routingkeyfile | routing Key file | 190 | 191 | Create the secrets using the Azure CLI. Below is an example. 192 | 193 | ```bash 194 | az keyvault secret set --vault-name KeyVaultName -n mastercafile --file ~/certificates/masterca.pem 195 | ``` 196 | 197 | ## azuredeploy.Parameters.json File Explained 198 | | Property | Description | Valid options | Default value | 199 | |-----------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|---------------| 200 | | `_artifactsLocation` | URL for artifacts (json, scripts, etc.) | | https://raw.githubusercontent.com/Microsoft/openshift-container-platform/master | 201 | | `location` | Azure region to deploy resources to | | | 202 | | `masterVmSize` | Size of the Master VM. Select from one of the allowed VM sizes listed in the azuredeploy.json file | | Standard_E2s_v3 | 203 | | `infraVmSize` | Size of the Infra VM. Select from one of the allowed VM sizes listed in the azuredeploy.json file | | Standard_D4s_v3 | 204 | | `nodeVmSize` | Size of the App Node VM. Select from one of the allowed VM sizes listed in the azuredeploy.json file | | Standard_D4s_v3 | 205 | | `cnsVmSize` | Size of the CNS Node VM. Select from one of the allowed VM sizes listed in the azuredeploy.json file | | Standard_E4s_v3 | 206 | | `osImageType` | The RHEL image to use. defaultgallery: On-Demand; marketplace: 3rd Party image | - "defaultgallery"

- "marketplace" | defaultgallery | 207 | | `marketplaceOsImage` | If `osImageType` is marketplace, then enter the appropriate values for 'publisher', 'offer', 'sku', 'version' of the marketplace offer. This is an object type | | | 208 | | `storageKind` | The type of storage to be used. | - "managed"
- "unmanaged" | managed | 209 | | `openshiftClusterPrefix` | Cluster Prefix used to configure hostnames for all nodes. Between 1 and 20 characters | | mycluster | 210 | | `minoVersion` | The minor version of OpenShift Container Platform 3.11 to deploy | | 188 | 211 | | `masterInstanceCount` | Number of Masters nodes to deploy | - 1, 3, 5 | 3 | 212 | | `infraInstanceCount` | Number of infra nodes to deploy | - 1, 2, 3 | 3 | 213 | | `nodeInstanceCount` | Number of Nodes to deploy | - 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 | 2 | 214 | | `cnsInstanceCount` | Number of CNS nodes to deploy | - 3, 4 | 3 | 215 | | `osDiskSize` | Size of OS disk for the VM (in GB) | - 64
- 128
- 256
- 512
- 1024
- 2048 | 64 | 216 | | `dataDiskSize` | Size of data disk to attach to nodes for Docker volume (in GB) | - 32
- 64
- 128
- 256
- 512
- 1024
- 2048 | 128 | 217 | | `cnsGlusterDiskSize` | Size of data disk to attach to CNS nodes for use by gluster (in GB) | - 32
- 64
- 128
- 256
- 512
- 1024
- 2048 | 128 | 218 | | `adminUsername` | Admin username for both OS (VM) login and initial OpenShift user | | ocpadmin | 219 | | `enableMetrics` | Enable Metrics. Metrics require more resources so select proper size for Infra VM | - "true"
- "false" | false | 220 | | `enableLogging` | Enable Logging. elasticsearch pod requires 8 GB RAM so select proper size for Infra VM | - "true"
- "false" | false | 221 | | `enableCNS` | Enable Container Native Storage (CNS) | - "true"
- "false" | false | 222 | | `rhsmUsernameOrOrgId` | Red Hat Subscription Manager Username or Organization ID | | | 223 | | `rhsmPoolId` | The Red Hat Subscription Manager Pool ID that contains your OpenShift entitlements for compute nodes | | | 224 | | `rhsmBrokerPoolId` | The Red Hat Subscription Manager Pool ID that contains your OpenShift entitlements for masters and infra nodes. If you don't have different pool IDs, enter same pool ID as 'rhsmPoolId' | | 225 | | `sshPublicKey` | Copy your SSH Public Key here | | | 226 | | `keyVaultSubscriptionId` | The Subscription ID of the subscription that contains the Key Vault | | | 227 | | `keyVaultResourceGroup` | The name of the Resource Group that contains the Key Vault | | | 228 | | `keyVaultName` | The name of the Key Vault you created | | | 229 | | `enableAzure` | Enable Azure Cloud Provider | - "true"
- "false" | true | 230 | | `aadClientId` | Azure Active Directory Client ID also known as Application ID for Service Principal | | | 231 | | `domainName` | Name of the custom domain name to use (if applicable). Set to "none" if not deploying fully private cluster | | none | 232 | | `masterClusterDnsType` | Domain type for OpenShift web console. 'default' will use DNS label of master infra public IP. 'custom' allows you to define your own name. | - "default"
- "custom" | default | 233 | | `masterClusterDns` | The custom DNS name to use to access the OpenShift web console if you selected 'custom' for `masterClusterDnsType` | | console.contoso.com | 234 | | `routingSubDomainType` | This will either be nipio (if you don't have your own domain) or 'custom' if you have your own domain that you would like to use for routing | - "nipio"
- "custom" | nipio | 235 | | `routingSubDomain` | The wildcard DNS name you would like to use for routing if you selected 'custom' for `routingSubDomainType` | | apps.contoso.com | 236 | | `virtualNetworkNewOrExisting` | Select whether to use an existing Virtual Network or create a new Virtual Network | - "existing"
- "new" | new | 237 | | `virtualNetworkResourceGroupName` | Name of the Resource Group for the new Virtual Network if you selected 'new' for `virtualNetworkNewOrExisting` | | resourceGroup().name | 238 | | `virtualNetworkName` | The name of the new Virtual Network to create if you selected 'new' for `virtualNetworkNewOrExisting` | | openshiftvnet | 239 | | `addressPrefixes` | Address prefix of the new virtual network | | 10.0.0.0/14 | 240 | | `masterSubnetName` | The name of the master subnet | | mastersubnet | 241 | | `masterSubnetPrefix` | CIDR used for the master subnet - needs to be a subset of the addressPrefix | | 10.1.0.0/16 | 242 | | `infraSubnetName` | The name of the infra subnet | | infrasubnet | 243 | | `infraSubnetPrefix` | CIDR used for the infra subnet - needs to be a subset of the addressPrefix | | 10.2.0.0/16 | 244 | | `nodeSubnetName` | The name of the node subnet | | nodesubnet | 245 | | `nodeSubnetPrefix` | CIDR used for the node subnet - needs to be a subset of the addressPrefix | | 10.3.0.0/16 | 246 | | `existingMasterSubnetReference` | Full reference to existing subnet for master nodes. Not needed if creating new vNet / Subnet | | | 247 | | `existingInfraSubnetReference` | Full reference to existing subnet for infra nodes. Not needed if creating new vNet / Subnet | | | 248 | | `existingCnsSubnetReference` | Full reference to existing subnet for cns nodes. Not needed if creating new vNet / Subnet | | | 249 | | `existingNodeSubnetReference` | Full reference to existing subnet for compute nodes. Not needed if creating new vNet / Subnet | | | 250 | | `masterClusterType` | Specify whether the cluster uses private or public master nodes. If private is chosen, the master nodes will not be exposed to the Internet via a public IP. Instead, it will use the private IP specified in the `masterPrivateClusterIp` | - "public"
- "private" | public | 251 | | `masterPrivateClusterIp` | If private master nodes is selected, then a private IP address must be specified for use by the internal load balancer for master nodes. This will be a static IP so it must reside within the CIDR block for the master subnet and not already in use. If public master nodes is selected, this value will not be used but must still be specified. | | 10.1.0.200 | 252 | | `routerClusterType` | Specify whether the cluster uses private or public infra nodes. If private is chosen, the infra nodes will not be exposed to the Internet via a public IP. Instead, it will use the private IP specified in the `routerPrivateClusterIp` | - "public"
- "private" | public | 253 | | `routerPrivateClusterIp` | If private infra nodes is selected, then a private IP address must be specified for use by the internal load balancer for infra nodes. This will be a static IP so it must reside within the CIDR block for the master subnet and not already in use. If public infra nodes is selected, this value will not be used but must still be specified. | | 10.2.0.200 | 254 | | `routingCertType` | Use custom certificate for routing domain or the default self-signed certificate - follow instructions in **Custom Certificates** section | - "selfsigned"
- "custom" | selfsigned | 255 | | `masterCertType` | Use custom certificate for master domain or the default self-signed certificate - follow instructions in **Custom Certificates** section | - "selfsigned"
- "custom" | selfsigned | 256 | 257 | 258 | 259 | ## Deploy Template 260 | 261 | Once you have collected all of the prerequisites for the template, you can deploy the template by populating the **azuredeploy.parameters.json** file and executing Resource Manager deployment commands with PowerShell or the Azure CLI. 262 | 263 | **Azure CLI 2.0** 264 | 265 | 1. Create Resource Group: az group create -n \ -l \
266 | Ex: `az group create -n openshift-cluster -l westus` 267 | 2. Create Resource Group Deployment: az group deployment create --name \ --template-file \ --parameters @\ --resource-group \ --nowait
268 | Ex: `az group deployment create --name ocpdeployment --template-file azuredeploy.json --parameters @azuredeploy.parameters.json --resource-group openshift-cluster --no-wait` 269 | 270 | 271 | ### NOTE 272 | 273 | The OpenShift Ansible playbook does take a while to run when using VMs backed by Standard Storage. VMs backed by Premium Storage are faster. If you want Premium Storage, select a DS, Es, or GS series VM. It is highly recommended that Premium storage be used. 274 |
275 | 276 | If the Azure Cloud Provider is not enabled, then the Service Catalog and Ansible Template Service Broker will not be installed as Service Catalog requires persistent storage. 277 | 278 | Be sure to follow the OpenShift instructions to create the necessary DNS entry for the OpenShift Router for access to applications.
279 | 280 | A Standard Storage Account is provisioned to provide persistent storage for the integrated OpenShift Registry as Premium Storage does not support storage of anything but VHD files. 281 | 282 | 283 | ### TROUBLESHOOTING 284 | 285 | If you encounter an error during deployment of the cluster, please view the deployment status. The following Error Codes will help to narrow things down. 286 | 287 | 1. Exit Code 3: Your Red Hat Subscription User Name / Password or Organization ID / Activation Key is incorrect 288 | 2. Exit Code 4: Your Red Hat Pool ID is incorrect or there are no entitlements available 289 | 3. Exit Code 5: Unable to provision Docker Thin Pool Volume 290 | 4. Exit Code 99: Configuration playbooks were not downloaded 291 | 292 | Before opening an issue, ssh to the Bastion node and review the stdout and stderr files as explained below. The stdout file will most likely contain the most useful information so please do include the last 50 lines of the stdout file in the issue description. Do NOT copy the error output from the Azure portal. 293 | 294 | You can SSH to the Bastion node and from there SSH to each of the nodes in the cluster and fix the issues. 295 | 296 | A common cause for the failures related to the node service not starting is the Service Principal did not have proper permissions to the Subscription or the Resource Group. If this is indeed the issue, then assign the correct permissions and manually re-run the script that failed an all subsequent scripts. Be sure to restart the service that failed (e.g. systemctl restart atomic-openshift-node.service) before executing the scripts again. 297 | 298 | For further troubleshooting, please SSH into your Bastion node on port 22. You will need to be root **(sudo su -)** and then navigate to the following directory: **/var/lib/waagent/custom-script/download**

299 | You should see a folder named '0' and '1'. In each of these folders, you will see two files, stderr and stdout. You can look through these files to determine where the failure occurred. 300 | 301 | ## Post-Deployment Operations 302 | 303 | ### Service Catalog 304 | 305 | **Service Catalog** 306 | 307 | If you enable Azure or CNS for storage these scripts will deploy the service catalog as a post deployment option. 308 | 309 | ### Metrics and logging 310 | 311 | **Metrics** 312 | 313 | If you deployed Metrics, it will take a few extra minutes for deployment to complete. Please be patient. 314 | 315 | Once the deployment is complete, log into the OpenShift Web Console and complete an addition configuration step. Go to the openshift-infra project, click on Hawkster metrics route, and accept the SSL exception in your browser. 316 | 317 | **Logging** 318 | 319 | If you deployed Logging, it will take a few extra minutes for deployment to complete. Please be patient. 320 | 321 | Once the deployment is complete, log into the OpenShift Web Console and complete an addition configuration step. Go to the logging project, click on the Kubana route, and accept the SSL exception in your browser. 322 | 323 | ### Creation of additional users 324 | 325 | To create additional (non-admin) users in your environment, login to your master server(s) via SSH and run: 326 |
htpasswd /etc/origin/master/htpasswd mynewuser 327 | 328 | ### Additional OpenShift Configuration Options 329 | 330 | You can configure additional settings per the official (OpenShift Container Platform Documentation). 331 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). 6 | 7 | If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below. 8 | 9 | ## Reporting Security Issues 10 | 11 | **Please do not report security vulnerabilities through public GitHub issues.** 12 | 13 | Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report). 14 | 15 | If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey). 16 | 17 | You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). 18 | 19 | Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: 20 | 21 | * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) 22 | * Full paths of source file(s) related to the manifestation of the issue 23 | * The location of the affected source code (tag/branch/commit or direct URL) 24 | * Any special configuration required to reproduce the issue 25 | * Step-by-step instructions to reproduce the issue 26 | * Proof-of-concept or exploit code (if possible) 27 | * Impact of the issue, including how an attacker might exploit the issue 28 | 29 | This information will help us triage your report more quickly. 30 | 31 | If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs. 32 | 33 | ## Preferred Languages 34 | 35 | We prefer all communications to be in English. 36 | 37 | ## Policy 38 | 39 | Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd). 40 | 41 | 42 | -------------------------------------------------------------------------------- /ansible/README.md: -------------------------------------------------------------------------------- 1 | # OpenShift Container Platform Deployment using Ansible 2 | 3 | ## Prerequisites 4 | 5 | - Ansible 2.7.x 6 | - Azure service principal 7 | - SSH public and private keys generated 8 | 9 | ### Red Hat Subscription Access 10 | 11 | For security reasons, the method for registering the RHEL system allows the use of an Organization ID and Activation Key as well as a Username and Password. Please know that it is more secure to use the Organization ID and Activation Key. 12 | 13 | You can determine your Organization ID by running ```subscription-manager identity``` on a registered machine. To create or find your Activation Key, please go here: https://access.redhat.com/management/activation_keys. 14 | 15 | You will also need to get the Pool ID that contains your entitlements for OpenShift. You can retrieve this from the Red Hat portal by examining the details of the subscription that has the OpenShift entitlements. Or you can contact your Red Hat administrator to help you. 16 | 17 | ## Setting up OpenShift 18 | 19 | Copy **vars.example.yml** to **vars.yml** and edit the file to update all the variables with your information. 20 | 21 | In general the only thing you will have to do is to make sure you have proper SSH keys available. By default your private key will be used from **~/.ssh/id_rsa**. Copy your public key content to **admin_pubkey:** 22 | 23 | In addition you need to provide your RHEL username/password or organisation/key in following fields: 24 | - **rhsm_username_org** 25 | - **rhsm_password_key** 26 | Please check last paragraph of this document to learn more. 27 | 28 | Run the playbook: 29 | 30 | ```bash 31 | ansible-playbook playbooks/create.yml -e @vars.yml 32 | ``` 33 | This playbook will deploy OpenShift Container Platform with basic username / password for authentication to OpenShift. It includes the following resources: 34 | 35 | |Resource |Properties | 36 | |-----------------------|------------------------------------------------------------------------------------------------------------------------------------| 37 | |Virtual Network
Default |**Address prefix:** 10.0.0.0/14
**Master subnet:** 10.1.0.0/16
**Node subnet:** 10.2.0.0/16 | 38 | |Master Load Balancer |1 probe and 1 rule for TCP 443 | 39 | |Infra Load Balancer |2 probes and 2 rules for TCP 80 and TCP 443 | 40 | |Public IP Addresses |Bastion Public IP for Bastion Node
OpenShift Master public IP attached to Master Load Balancer (if masters are public)
OpenShift Router public IP attached to Infra Load Balancer (if router is public) | 41 | |Storage Accounts|1 Storage Account for Registry| 42 | |Network Security Groups|1 Network Security Group for Bastion VM
1 Network Security Group Master VMs
1 Network Security Group for Infra VMs
1 Network Security Group for Node VMs | 43 | |Availability Sets |1 Availability Set for Master VMs
1 Availability Set for Infra VMs
1 Availability Set for Node VMs | 44 | |Virtual Machines |1 Bastion Node - Used to Run Ansible Playbook for OpenShift deployment
1, 3 or 5 Master Nodes
1, 2 or 3 Infra Nodes
User-defined number of Nodes (1 to 30)
All VMs include a single attached data disk for Docker thin pool logical volume| 45 | 46 | ![Cluster Diagram](../images/openshiftdiagram.jpg) 47 | 48 | 49 | ## Playbook Explanation 50 | 51 | Playbook execution can be divided into a few phases. During these phases tasks run in parallel to save time 52 | 53 | During fist phase following resources are created in parallel: 54 | - Public IP addresses 55 | - subnets 56 | - network security groups 57 | - availability sets 58 | 59 | In the second phase following resources are created: 60 | - load balancers for master and infrastructure nodes 61 | - network interfaces for bastion, master, infra and node VMs 62 | - storage - synchronous 63 | 64 | In the third phase: 65 | - bastion VM 66 | - master VMs 67 | - node VMs 68 | - infra VMs 69 | 70 | In the fourth phase, after all the virtual machines are successfully deployed: 71 | - execute custom scripts on master infra and node VMs to set them up using **azure_rm_virtualmachine_extension** module - these tasks are performed asynchronously, but we won't wait for the result 72 | - execute tasks on bastion VM to install OpenShift - these tasks will be performed synchronously 73 | 74 | 75 | ## Parameters Explanation 76 | 77 | ## azuredeploy.Parameters.json File Explained 78 | | Property | Description | Valid options | Default value | 79 | |-----------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|---------------| 80 | |`location`|Azure region for deployment||| 81 | |`resource_group`|Resource group name||| 82 | |`cluster_prefix`|Cluster name prefix, used as prefix for VM names||| 83 | |`master_count`|Number of Masters nodes to deploy||3| 84 | |`infra_count`|Number of infra nodes to deploy||3| 85 | |`node_count`| Number of Nodes to deploy||2| 86 | |`vm_size_master`|Size of the Master VM.||Standard_D2s_v3| 87 | |`vm_size_infra`|Size of the Infra VM.||Standard_D4s_v3| 88 | |`vm_size_node`|Size of the App Node VM.|| Standard_D2s_v3| 89 | |`vm_size_bastion`|Size of the Bastion Node VM.||Standard_D2s_v3| 90 | |`os_disk_size`|Size of OS disk|min 64 GB|64| 91 | |`data_disk_size`|Size of data disk to attach to nodes for Docker volume|- 32 GB
- 64 GB
- 128 GB
- 256 GB
- 512 GB
- 1024 GB
- 2048 GB|64| 92 | |`managed_disk_type`|Type of managed disk|- Premium_LRS|Premium_LRS| 93 | |`admin_username`| Admin username for both OS (VM) login and initial OpenShift user||azureuser| 94 | |`admin_pubkey`|Admin public key||azureuser| 95 | |`admin_privkey`|Admin private key location||~/.ssh/id_rsa| 96 | |`aad_client_id`|||| 97 | |`aad_client_secret`|||| 98 | |`subscription_id`|||| 99 | |`tenant_id`|||| 100 | |`rhsm_username_org`|||| 101 | |`rhsm_username_key`|||| 102 | |`rhsm_pool`|||| 103 | |`virtual_network_name`|||| 104 | |`virtual_network_cidr`|||| 105 | |`master_subnet_cidr`|||| 106 | |`infra_subnet_cidr`|||| 107 | |`node_subnet_cidr`|||| 108 | |`cns_subnet_cidr`|||| 109 | |`bastion_publicip`|||| 110 | |`master_lb_public_ip`|||| 111 | |`routing`|||| 112 | |`router_lb_public_ip`|||| 113 | |`registry_storage_account`|||| 114 | |`unmanaged_storage_class_account`|||| 115 | |`ocp_admin_passwd`|||| 116 | |`deploy_cns`|||| 117 | |`deploy_logging`|||| 118 | |`deploy_azure_cloud_provider`|||| 119 | 120 | ### Red Hat Subscription Access 121 | 122 | For security reasons, the method for registering the RHEL system allows the use of an Organization ID and Activation Key as well as a Username and Password. Please know that it is more secure to use the Organization ID and Activation Key. 123 | 124 | You can determine your Organization ID by running ```subscription-manager identity``` on a registered machine. To create or find your Activation Key, please go here: https://access.redhat.com/management/activation_keys. 125 | 126 | You will also need to get the Pool ID that contains your entitlements for OpenShift. You can retrieve this from the Red Hat portal by examining the details of the subscription that has the OpenShift entitlements. Or you can contact your Red Hat administrator to help you. 127 | -------------------------------------------------------------------------------- /ansible/playbooks/create.yml: -------------------------------------------------------------------------------- 1 | # Description 2 | # =========== 3 | # OpenShift sample for Azure 4 | 5 | --- 6 | - hosts: localhost 7 | tasks: 8 | # --- PRE TASKS --- 9 | # All the other tasks require these tasks 10 | - name: Create a resource group 11 | azure_rm_resourcegroup: 12 | name: "{{ resource_group }}" 13 | location: "{{ location }}" 14 | 15 | - name: Create virtual network 16 | azure_rm_virtualnetwork: 17 | resource_group: "{{ resource_group }}" 18 | name: "{{ virtual_network_name }}" 19 | address_prefixes: "{{ virtual_network_cidr }}" 20 | 21 | # --- PARALLELIZABLE TASKS --- 22 | # All these tasks have no requirements other than RG and vNet 23 | # They can be run in any order 24 | - name: Create public IP address for Master LB 25 | azure_rm_publicipaddress: 26 | resource_group: "{{ resource_group }}" 27 | allocation_method: Static 28 | domain_name: "{{ master_lb_public_ip }}" 29 | name: "{{ master_lb_public_ip }}" 30 | async: 150 31 | poll: 0 32 | register: master_ip_output 33 | 34 | - name: Create public IP address for Infra LB 35 | azure_rm_publicipaddress: 36 | resource_group: "{{ resource_group }}" 37 | allocation_method: Static 38 | domain_name: "{{ router_lb_public_ip }}" 39 | name: "{{ router_lb_public_ip }}" 40 | async: 150 41 | poll: 0 42 | register: infra_ip_output 43 | 44 | - name: Create public IP address for Bastion node 45 | azure_rm_publicipaddress: 46 | resource_group: "{{ resource_group }}" 47 | allocation_method: Static 48 | domain_name: "{{ bastion_publicip }}" 49 | name: "{{ bastion_publicip }}" 50 | async: 150 51 | poll: 0 52 | register: bastion_ip_output 53 | 54 | - name: Add master subnet 55 | azure_rm_subnet: 56 | resource_group: "{{ resource_group }}" 57 | name: mastersubnet 58 | address_prefix: "{{ master_subnet_cidr }}" 59 | virtual_network: "{{ virtual_network_name }}" 60 | 61 | - name: Add node subnet 62 | azure_rm_subnet: 63 | resource_group: "{{ resource_group }}" 64 | name: nodesubnet 65 | address_prefix: "{{ node_subnet_cidr }}" 66 | virtual_network: "{{ virtual_network_name }}" 67 | 68 | - name: Add infra subnet 69 | azure_rm_subnet: 70 | resource_group: "{{ resource_group }}" 71 | name: infrasubnet 72 | address_prefix: "{{ infra_subnet_cidr }}" 73 | virtual_network: "{{ virtual_network_name }}" 74 | 75 | - name: Create infrastructure NSG 76 | azure_rm_securitygroup: 77 | resource_group: "{{ resource_group }}" 78 | name: "{{ cluster_prefix }}-infra-nsg" 79 | rules: 80 | - name: allowSSHin_all 81 | protocol: Tcp 82 | destination_port_range: 22 83 | access: Allow 84 | priority: 100 85 | direction: Inbound 86 | - name: allowHTTPSIn_all 87 | protocol: Tcp 88 | destination_port_range: 443 89 | access: Allow 90 | priority: 200 91 | direction: Inbound 92 | - name: allowHTTPIn_all 93 | protocol: Tcp 94 | destination_port_range: 80 95 | access: Allow 96 | priority: 300 97 | direction: Inbound 98 | async: 150 99 | poll: 0 100 | 101 | - name: Create master NSG 102 | azure_rm_securitygroup: 103 | resource_group: "{{ resource_group }}" 104 | name: "{{ cluster_prefix }}-master-nsg" 105 | rules: 106 | - name: allowSSHin_all 107 | protocol: Tcp 108 | destination_port_range: 22 109 | access: Allow 110 | priority: 100 111 | direction: Inbound 112 | - name: allowHTTPSIn_all 113 | protocol: Tcp 114 | destination_port_range: 443 115 | access: Allow 116 | priority: 200 117 | direction: Inbound 118 | async: 150 119 | poll: 0 120 | 121 | - name: Create node NSG 122 | azure_rm_securitygroup: 123 | resource_group: "{{ resource_group }}" 124 | name: "{{ cluster_prefix }}-node-nsg" 125 | rules: 126 | - name: allowSSHin_all 127 | protocol: Tcp 128 | destination_port_range: 22 129 | access: Allow 130 | priority: 100 131 | direction: Inbound 132 | - name: allowHTTPSIn_all 133 | protocol: Tcp 134 | destination_port_range: 443 135 | access: Allow 136 | priority: 200 137 | direction: Inbound 138 | - name: allowHTTPIn_all 139 | protocol: Tcp 140 | destination_port_range: 80 141 | access: Allow 142 | priority: 300 143 | direction: Inbound 144 | async: 150 145 | poll: 0 146 | 147 | - name: Create master availability set 148 | azure_rm_availabilityset: 149 | name: masteravailabilityset 150 | location: "{{ location }}" 151 | resource_group: "{{ resource_group }}" 152 | sku: Aligned 153 | async: 150 154 | poll: 0 155 | register: master_as_output 156 | 157 | - name: Create node availability set 158 | azure_rm_availabilityset: 159 | name: nodeavailabilityset 160 | location: "{{ location }}" 161 | resource_group: "{{ resource_group }}" 162 | sku: Aligned 163 | async: 150 164 | poll: 0 165 | register: node_as_output 166 | 167 | - name: Create infra availability set 168 | azure_rm_availabilityset: 169 | name: infraavailabilityset 170 | location: "{{ location }}" 171 | resource_group: "{{ resource_group }}" 172 | sku: Aligned 173 | async: 150 174 | poll: 0 175 | register: infra_as_output 176 | 177 | - name: Wait for IP addresses, and Availability Sets to be ready 178 | async_status: 179 | jid: "{{ item.ansible_job_id }}" 180 | mode: status 181 | register: item_result 182 | with_items: 183 | - "{{ bastion_ip_output }}" 184 | - "{{ master_ip_output }}" 185 | - "{{ infra_ip_output }}" 186 | - "{{ infra_as_output }}" 187 | - "{{ master_as_output }}" 188 | - "{{ node_as_output }}" 189 | until: item_result.finished 190 | retries: 5 191 | delay: 30 192 | 193 | # --- Load Balancers --- 194 | # These require the public IPs: master and infra respectively 195 | - name: Create master load balancer 196 | azure_rm_loadbalancer: 197 | name: "{{ cluster_prefix }}-masterlb" 198 | location: "{{ location }}" 199 | resource_group: "{{ resource_group }}" 200 | frontend_ip_configurations: 201 | - name: frontendip 202 | public_ip_address: "{{ master_lb_public_ip }}" 203 | load_balancing_rules: 204 | - name: OpenShiftAdminConsole 205 | frontend_ip_configuration: frontendip 206 | load_distribution: SourceIP 207 | idle_timeout: 15 208 | probe: httpsprobe 209 | backend_address_pool: backendaddrpool0 210 | frontend_port: 443 211 | backend_port: 443 212 | backend_address_pools: 213 | - name: backendaddrpool0 214 | probes: 215 | - name: httpsprobe 216 | port: 443 217 | 218 | - name: Create infra load balancer 219 | azure_rm_loadbalancer: 220 | name: "{{ cluster_prefix }}-infralb" 221 | location: "{{ location }}" 222 | resource_group: "{{ resource_group }}" 223 | frontend_ip_configurations: 224 | - name: frontendip 225 | public_ip_address: "{{ router_lb_public_ip }}" 226 | load_balancing_rules: 227 | - name: OpenShiftRouterHTTP 228 | frontend_ip_configuration: frontendip 229 | probe: httpprobe 230 | backend_address_pool: backendaddrpool0 231 | frontend_port: 80 232 | backend_port: 80 233 | - name: OpenShiftRouterHTTPS 234 | frontend_ip_configuration: frontendip 235 | probe: httpsprobe 236 | backend_address_pool: backendaddrpool0 237 | frontend_port: 443 238 | backend_port: 443 239 | backend_address_pools: 240 | - name: backendaddrpool0 241 | probes: 242 | - name: httpsprobe 243 | port: 443 244 | - name: httpprobe 245 | port: 80 246 | 247 | # --- NICs --- 248 | # These require: Subnets and NSGs 249 | - name: Create virtual network interface card for bastion 250 | azure_rm_networkinterface: 251 | resource_group: "{{ resource_group }}" 252 | name: "{{ cluster_prefix }}-bastion-nic" 253 | virtual_network: "{{ virtual_network_name }}" 254 | subnet: infrasubnet 255 | security_group: "{{ cluster_prefix }}-infra-nsg" 256 | ip_configurations: 257 | - name: ipconfig0 258 | public_ip_address_name: "{{ bastion_publicip }}" 259 | async: 400 260 | poll: 0 261 | register: output_nic_bastion 262 | 263 | - name: Create storage account for Registry 264 | azure_rm_storageaccount: 265 | resource_group: "{{ resource_group }}" 266 | name: "{{ registry_storage_account }}" 267 | type: Standard_LRS 268 | 269 | - name: Get storage keys 270 | azure_rm_resource: 271 | resource_group: "{{ resource_group }}" 272 | provider: storage 273 | resource_type: storageAccounts 274 | resource_name: "{{ registry_storage_account }}" 275 | subresource: 276 | - type: listkeys 277 | api_version: '2018-03-01-preview' 278 | method: POST 279 | register: keys 280 | 281 | - name: store key as fact 282 | set_fact: registry_storage_account_key="{{ keys['response']['keys'][0]['value'] }}" 283 | 284 | - name: Create virtual network interface card for master 285 | azure_rm_networkinterface: 286 | resource_group: "{{ resource_group }}" 287 | name: "{{ cluster_prefix }}-master-{{ item }}-nic" 288 | virtual_network: "{{ virtual_network_name }}" 289 | subnet: mastersubnet 290 | security_group: "{{ cluster_prefix }}-master-nsg" 291 | ip_configurations: 292 | - name: ipconfig1 293 | load_balancer_backend_address_pools: 294 | - name: backendaddrpool0 295 | load_balancer: "{{ cluster_prefix }}-masterlb" 296 | with_sequence: start=0 end={{ master_count - 1 }} 297 | async: 400 298 | poll: 0 299 | register: output_nics_master 300 | 301 | - name: Create virtual network interface card for node 302 | azure_rm_networkinterface: 303 | resource_group: "{{ resource_group }}" 304 | name: "{{ cluster_prefix }}-node-{{ item }}-nic" 305 | virtual_network: "{{ virtual_network_name }}" 306 | subnet: nodesubnet 307 | security_group: "{{ cluster_prefix }}-node-nsg" 308 | ip_configurations: 309 | - name: ipconfig1 310 | with_sequence: start=0 end={{ node_count - 1 }} 311 | async: 400 312 | poll: 0 313 | register: output_nics_node 314 | 315 | - name: Create virtual network interface card for infra 316 | azure_rm_networkinterface: 317 | resource_group: "{{ resource_group }}" 318 | name: "{{ cluster_prefix }}-infra-{{ item }}-nic" 319 | virtual_network: "{{ virtual_network_name }}" 320 | subnet: infrasubnet 321 | security_group: "{{ cluster_prefix }}-infra-nsg" 322 | ip_configurations: 323 | - name: ipconfig1 324 | load_balancer_backend_address_pools: 325 | - name: backendaddrpool0 326 | load_balancer: "{{ cluster_prefix }}-infralb" 327 | with_sequence: start=0 end={{ infra_count - 1 }} 328 | async: 400 329 | poll: 0 330 | register: output_nics_infra 331 | 332 | - name: Wait for NICs to be ready 333 | async_status: 334 | jid: "{{ item.ansible_job_id }}" 335 | mode: status 336 | register: async_poll_results 337 | loop: "{{ [ output_nic_bastion ] + output_nics_master.results + output_nics_node.results + output_nics_infra.results }}" 338 | until: async_poll_results.finished 339 | retries: 30 340 | delay: 60 341 | 342 | # Create bastion 343 | - name: Create bastion VM 344 | azure_rm_virtualmachine: 345 | resource_group: "{{ resource_group }}" 346 | name: "{{ cluster_prefix }}-bastion" 347 | managed_disk_type: Standard_LRS 348 | vm_size: "{{ vm_size_bastion }}" 349 | admin_username: "{{ admin_username }}" 350 | ssh_password_enabled: false 351 | ssh_public_keys: 352 | - path: /home/{{ admin_username }}/.ssh/authorized_keys 353 | key_data: "{{ admin_pubkey }}" 354 | network_interfaces: "{{ cluster_prefix }}-bastion-nic" 355 | image: 356 | offer: RHEL 357 | publisher: RedHat 358 | sku: 7-RAW 359 | version: latest 360 | data_disks: 361 | - lun: 0 362 | disk_size_gb: "{{ data_disk_size }}" 363 | managed_disk_type: Standard_LRS 364 | async: 1000 365 | poll: 0 366 | register: output_vm_bastion 367 | 368 | # Creating master 369 | - name: Create master VMs 370 | azure_rm_virtualmachine: 371 | resource_group: "{{ resource_group }}" 372 | name: "{{ cluster_prefix }}-master-{{ item }}" 373 | managed_disk_type: "{{ managed_disk_type }}" 374 | vm_size: "{{ vm_size_master }}" 375 | admin_username: "{{ admin_username }}" 376 | ssh_password_enabled: false 377 | ssh_public_keys: 378 | - path: /home/{{ admin_username }}/.ssh/authorized_keys 379 | key_data: "{{ admin_pubkey }}" 380 | network_interfaces: "{{ cluster_prefix }}-master-{{ item }}-nic" 381 | image: 382 | offer: RHEL 383 | publisher: RedHat 384 | sku: 7-RAW 385 | version: latest 386 | data_disks: 387 | - lun: 0 388 | disk_size_gb: "{{ data_disk_size }}" 389 | managed_disk_type: "{{ managed_disk_type }}" 390 | availability_set: masteravailabilityset 391 | os_disk_size_gb: 64 392 | with_sequence: start=0 end={{ master_count - 1 }} 393 | async: 1000 394 | poll: 0 395 | register: output_vms_master 396 | 397 | - name: Create node VMs 398 | azure_rm_virtualmachine: 399 | resource_group: "{{ resource_group }}" 400 | name: "{{ cluster_prefix }}-node-{{ item }}" 401 | managed_disk_type: "{{ managed_disk_type }}" 402 | vm_size: "{{ vm_size_node }}" 403 | admin_username: "{{ admin_username }}" 404 | ssh_password_enabled: false 405 | ssh_public_keys: 406 | - path: /home/{{ admin_username }}/.ssh/authorized_keys 407 | key_data: "{{ admin_pubkey }}" 408 | network_interfaces: "{{ cluster_prefix }}-node-{{ item }}-nic" 409 | image: 410 | offer: RHEL 411 | publisher: RedHat 412 | sku: 7-RAW 413 | version: latest 414 | data_disks: 415 | - lun: 0 416 | disk_size_gb: "{{ data_disk_size }}" 417 | managed_disk_type: "{{ managed_disk_type }}" 418 | availability_set: nodeavailabilityset 419 | os_disk_size_gb: 64 420 | with_sequence: start=0 end={{ node_count - 1 }} 421 | async: 1000 422 | poll: 0 423 | register: output_vms_node 424 | 425 | - name: Create infra VMs 426 | azure_rm_virtualmachine: 427 | resource_group: "{{ resource_group }}" 428 | name: "{{ cluster_prefix }}-infra-{{ item }}" 429 | managed_disk_type: "{{ managed_disk_type }}" 430 | vm_size: "{{ vm_size_infra }}" 431 | admin_username: "{{ admin_username }}" 432 | ssh_password_enabled: false 433 | ssh_public_keys: 434 | - path: /home/{{ admin_username }}/.ssh/authorized_keys 435 | key_data: "{{ admin_pubkey }}" 436 | network_interfaces: "{{ cluster_prefix }}-infra-{{ item }}-nic" 437 | image: 438 | offer: RHEL 439 | publisher: RedHat 440 | sku: 7-RAW 441 | version: latest 442 | data_disks: 443 | - lun: 0 444 | disk_size_gb: "{{ data_disk_size }}" 445 | managed_disk_type: "{{ managed_disk_type }}" 446 | availability_set: infraavailabilityset 447 | os_disk_size_gb: 64 448 | with_sequence: start=0 end={{ infra_count - 1 }} 449 | async: 1000 450 | poll: 0 451 | register: output_vms_infra 452 | 453 | - name: Get bastion Public IP 454 | azure_rm_publicipaddress_facts: 455 | resource_group: "{{ resource_group }}" 456 | name: "{{ bastion_publicip }}" 457 | register: bastion_ip 458 | 459 | - set_fact: 460 | bastion_public_ip: "{{ bastion_ip.ansible_facts.azure_publicipaddresses[0].properties.ipAddress }}" 461 | 462 | - name: Get master Public IP 463 | azure_rm_publicipaddress_facts: 464 | resource_group: "{{ resource_group }}" 465 | name: "{{ master_lb_public_ip }}" 466 | register: master_lb_ip 467 | 468 | - set_fact: 469 | master_publicip: "{{ master_lb_ip.ansible_facts.azure_publicipaddresses[0].properties.ipAddress }}" 470 | master_publicip_fqdn: "{{ master_lb_ip.ansible_facts.azure_publicipaddresses[0].properties.dnsSettings.fqdn }}" 471 | 472 | - name: Get infra Public IP 473 | azure_rm_publicipaddress_facts: 474 | resource_group: "{{ resource_group }}" 475 | name: "{{ router_lb_public_ip }}" 476 | register: router_lb_ip 477 | 478 | - set_fact: 479 | router_publicip: "{{ router_lb_ip.ansible_facts.azure_publicipaddresses[0].properties.ipAddress }}" 480 | 481 | - set_fact: 482 | nip_io_domain: "{{ router_publicip }}.nip.io" 483 | 484 | - set_fact: 485 | routing: "{{ nip_io_domain }}" 486 | when: routing is undefined or routing is none or routing == "" 487 | 488 | - name: Add Bastion to host group 489 | add_host: 490 | hostname: "{{ bastion_public_ip }}" 491 | groupname: bastion_vm 492 | 493 | - name: Wait for VMs to be ready 494 | async_status: 495 | jid: "{{ item.ansible_job_id }}" 496 | mode: status 497 | register: async_poll_results 498 | loop: "{{ [ output_vm_bastion ] + output_vms_master.results + output_vms_node.results + output_vms_infra.results }}" 499 | until: async_poll_results.finished 500 | retries: 30 501 | delay: 60 502 | 503 | # --- VM Extensions --- 504 | # These require: VMs 505 | - name: Create master VM extension 506 | azure_rm_virtualmachine_extension: 507 | resource_group: "{{ resource_group }}" 508 | name: deployOpenShift 509 | virtual_machine_name: "{{ cluster_prefix }}-master-{{ item }}" 510 | publisher: Microsoft.Azure.Extensions 511 | virtual_machine_extension_type: CustomScript 512 | type_handler_version: 2.0 513 | settings: 514 | fileUris: 515 | - 'https://raw.githubusercontent.com/Microsoft/openshift-container-platform/master/scripts/masterPrep.sh' 516 | commandToExecute: "bash masterPrep.sh {{ rhsm_username_org }} '{{ rhsm_password_key }}' {{ rhsm_pool }} {{ admin_username }} {{ location }} {{ unmanaged_storage_class_account }}" 517 | with_sequence: start=0 end={{ master_count - 1 }} 518 | when: runvmext == "true" 519 | async: 1000 520 | poll: 0 521 | register: output_vmext_master 522 | 523 | 524 | - name: Create infra VM extension 525 | azure_rm_virtualmachine_extension: 526 | resource_group: "{{ resource_group }}" 527 | name: deployOpenShift 528 | virtual_machine_name: "{{ cluster_prefix }}-infra-{{ item }}" 529 | publisher: Microsoft.Azure.Extensions 530 | virtual_machine_extension_type: CustomScript 531 | type_handler_version: 2.0 532 | settings: 533 | fileUris: 534 | - 'https://raw.githubusercontent.com/Microsoft/openshift-container-platform/master/scripts/nodePrep.sh' 535 | commandToExecute: "bash nodePrep.sh {{ rhsm_username_org }} '{{ rhsm_password_key }}' {{ rhsm_pool }}" 536 | with_sequence: start=0 end={{ infra_count - 1 }} 537 | when: runvmext == "true" 538 | async: 1000 539 | poll: 0 540 | register: output_vmext_infra 541 | 542 | - name: Create node VM extension 543 | azure_rm_virtualmachine_extension: 544 | resource_group: "{{ resource_group }}" 545 | name: deployOpenShift 546 | virtual_machine_name: "{{ cluster_prefix }}-node-{{ item }}" 547 | publisher: Microsoft.Azure.Extensions 548 | virtual_machine_extension_type: CustomScript 549 | type_handler_version: 2.0 550 | settings: 551 | fileUris: 552 | - 'https://raw.githubusercontent.com/Microsoft/openshift-container-platform/master/scripts/nodePrep.sh' 553 | commandToExecute: "bash nodePrep.sh {{ rhsm_username_org }} '{{ rhsm_password_key }}' {{ rhsm_pool }}" 554 | with_sequence: start=0 end={{ node_count - 1 }} 555 | when: runvmext == "true" 556 | async: 1000 557 | poll: 0 558 | register: output_vmext_node 559 | 560 | - name: Wait for VM extensions setup to be completed 561 | async_status: 562 | jid: "{{ item.ansible_job_id }}" 563 | mode: status 564 | register: async_poll_results 565 | loop: "{{ output_vmext_master.results + output_vmext_node.results + output_vmext_infra.results }}" 566 | until: async_poll_results.finished 567 | retries: 30 568 | delay: 60 569 | 570 | 571 | # Bastion playbook tasks to set it up 572 | - hosts: bastion_vm 573 | user: "{{ admin_username }}" 574 | become: yes 575 | vars: 576 | dest_key: "/home/{{ admin_username }}/.ssh/id_rsa" 577 | empty_var: "" 578 | tasks: 579 | - name: Ensure .ssh directory exists in root home folder. 580 | file: 581 | dest: "{{ dest_key | dirname }}" 582 | mode: 0700 583 | owner: "{{ admin_username }}" 584 | state: directory 585 | 586 | - name: Install SSH private key on Bastion 587 | copy: 588 | src: "{{ admin_privkey }}" 589 | dest: "{{ dest_key }}" 590 | mode: 0600 591 | owner: "{{ admin_username }}" 592 | 593 | - name: Remove RHUI 594 | shell: | 595 | rm -f /etc/yum.repos.d/rh-cloud.repo 596 | sleep 10 597 | 598 | - name: Register as user with password and auto-subscribe to available content. 599 | redhat_subscription: 600 | state: present 601 | username: "{{ rhsm_username_org }}" 602 | password: "{{ rhsm_password_key }}" 603 | pool_ids: "{{ rhsm_pool }}" 604 | auto_attach: yes 605 | 606 | - name: Disable all RHSM repositories 607 | rhsm_repository: 608 | name: '*' 609 | state: disabled 610 | 611 | - name: Enable RHEL RHSCL and ansible repo 612 | rhsm_repository: 613 | name: 614 | - rhel-7-server-rpms 615 | - rhel-7-server-extras-rpms 616 | - rhel-7-server-ose-3.9-rpms 617 | - rhel-7-server-ansible-2.4-rpms 618 | - rhel-7-fast-datapath-rpms 619 | - rh-gluster-3-client-for-rhel-7-server-rpms 620 | state: enabled 621 | 622 | - name: Update system to latest packages 623 | yum: 624 | name: '*' 625 | state: latest 626 | exclude: WALinuxAgent 627 | 628 | - name: Az CLI import rpm key 629 | rpm_key: 630 | state: present 631 | key: https://packages.microsoft.com/keys/microsoft.asc 632 | 633 | - name: Add Azure repository 634 | yum_repository: 635 | name: azure-cli 636 | description: Azure CLI 637 | baseurl: https://packages.microsoft.com/yumrepos/azure-cli 638 | gpgcheck: yes 639 | gpgkey: https://packages.microsoft.com/keys/microsoft.asc 640 | 641 | #TODO: Fix versions 642 | - name: Install ALL packages 643 | yum: 644 | name: 645 | - wget 646 | - net-tools 647 | - bind-utils 648 | - iptables-services 649 | - bridge-utils 650 | - bash-completion 651 | - httpd-tools 652 | - kexec-tools 653 | - sos 654 | - psacct 655 | - ansible 656 | - glusterfs-fuse 657 | - gcc 658 | - python-devel 659 | - git 660 | - azure-cli 661 | #- python27-python-pip 662 | - atomic-openshift-excluder 663 | - atomic-openshift-docker-excluder 664 | state: present 665 | 666 | - name: Excluders for OpenShift 667 | shell: atomic-openshift-excluder unexclude 668 | 669 | - name: Installing OpenShift utilities 670 | yum: 671 | name: 672 | - atomic-openshift-utils 673 | state: present 674 | 675 | - name: Adding DOMAIN to search for resolv.conf 676 | shell: | 677 | echo "DOMAIN=`domainname -d`" >> /etc/sysconfig/network-scripts/ifcfg-eth0 678 | 679 | - name: Get updateansiblecfg.yaml playbook 680 | get_url: 681 | url: https://raw.githubusercontent.com/microsoft/openshift-container-platform-playbooks/master/updateansiblecfg.yaml 682 | dest: . 683 | 684 | - name: Run updateansiblecfg.yaml playbook 685 | shell: ansible-playbook -f 25 updateansiblecfg.yaml 686 | 687 | - name: Get "deployOpenShift.sh" 688 | get_url: 689 | url: https://gist.githubusercontent.com/brusMX/b8ba89a4ce7063363315e6ce69b4a1d3/raw/652d8c73bc5f45e0383d0781c9e5293e56c4dc1f/export-all.sh 690 | dest: . 691 | - file: 692 | path: export-all.sh 693 | mode: "a+x" 694 | 695 | # Deploying openshift CP with deployOpenShift.sh 696 | # 'bash ', parameters('openshiftDeploymentScriptFileName') 697 | - name: Run deployOpenShift.sh Container Platform 698 | shell: | 699 | args=( 700 | # $1 SUDOUSER=$1 - parameters('adminUsername') 701 | "{{ admin_username }}" 702 | # $2 PASSWORD="$2" - parameters('openshiftPassword') 703 | "{{ ocp_admin_passwd }}" \ 704 | # $3 MASTER=$3 - parameters('openshiftMasterHostname') 705 | "{{ cluster_prefix }}-master" \ 706 | # $4 MASTERPUBLICIPHOSTNAME=$4 - parameters('openshiftMasterPublicIpFqdn') 707 | "{{ hostvars['localhost']['master_publicip_fqdn'] }}" \ 708 | # $5 MASTERPUBLICIPADDRESS=$5 - parameters('openshiftMasterPublicIpAddress') 709 | "{{ hostvars['localhost']['master_publicip'] }}" \ 710 | # $6 INFRA=$6 - parameters('openshiftInfraHostname') 711 | "{{ cluster_prefix }}-infra" \ 712 | # $7 NODE=$7 - parameters('openshiftNodeHostname') 713 | "{{ cluster_prefix }}-node" \ 714 | # $8 NODECOUNT=$8 - parameters('nodeInstanceCount') 715 | "{{ node_count }}" \ 716 | # $9 INFRACOUNT=${9} - parameters('infraInstanceCount') 717 | "{{ infra_count }}" \ 718 | # $10 MASTERCOUNT=${10} - parameters('MasterInstanceCount') 719 | "{{ master_count }}" \ 720 | # $11 ROUTING=${11} - parameters(parameters('subDomainChosen')) 721 | "{{ routing }}" \ 722 | # $12 REGISTRYSA=${12} - parameters('newStorageAccountRegistry') 723 | "{{ registry_storage_account }}" \ 724 | # $13 ACCOUNTKEY="${13}" - parameters('newStorageAccountKey') 725 | "{{ hostvars['localhost']['registry_storage_account_key'] }}" \ 726 | # $14 METRICS=${14} - parameters('enableMetrics') 727 | "{{ deploy_metrics }}" 728 | # $15 LOGGING=${15} 729 | "{{ deploy_logging }}" 730 | # $16 TENANTID=${16} - variables('tenantId') 731 | "{{ tenant_id }}" \ 732 | # $17 SUBSCRIPTIONID=${17} - variables('subscriptionId') 733 | "{{ subscription_id }}" \ 734 | # $18 AADCLIENTID=${18} - parameters('aadClientId') 735 | "{{ aad_client_id }}" \ 736 | # $19 AADCLIENTSECRET="${19}" - parameters('aadClientSecret') 737 | "{{ aad_client_secret }}" \ 738 | # $20 RESOURCEGROUP=${20} - variables('resourceGroupName') 739 | "{{ resource_group }}" \ 740 | # $21 LOCATION=${21} - variables('location') 741 | "{{ location }}" \ 742 | # $22 AZURE=${22} - parameters('enableAzure') 743 | "{{ deploy_azure_cloud_provider }}" \ 744 | # $23 STORAGEKIND=${23} 745 | "{{ disk_type }}" 746 | # $24 ENABLECNS 747 | "{{ deploy_cns }}" 748 | # $25 CNS 749 | "{{ cluster_prefix }}-cns" 750 | # $26 CNSCOUNT 751 | "{{ cns_count }}" 752 | # $27 VNETNAME #TODO: remove var from script is not utilized 753 | "{{ empty_var }}" 754 | # $28 NODENSG #TODO: remove var from script is not utilized 755 | "{{ empty_var }}" 756 | # $29 NODEAVAILIBILITYSET #TODO: remove var from script is not utilized 757 | "{{ empty_var }}" 758 | ) 759 | ./export-all.sh "${args[@]}" 760 | -------------------------------------------------------------------------------- /ansible/vars.example.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # defaults file for create.yaml 3 | state: present 4 | location: eastus 5 | resource_group: "myclusterrg" 6 | cluster_prefix: mycluster 7 | 8 | runvmext: "true" 9 | unique_string: "{{ resource_group | md5 | truncate(8, True, '') }}" 10 | 11 | ####################### 12 | # Number of nodes 13 | ####################### 14 | master_count: 3 15 | infra_count: 3 16 | node_count: 3 17 | # Min of 3 required and add odd nums only for CNS 18 | cns_count: 0 # [3,5,7] 19 | 20 | ####################### 21 | # vm sizes and other vm configurations (adapt) 22 | # az vm list-sizes -l -o table 23 | ####################### 24 | 25 | vm_size_master: Standard_D2s_v3 26 | vm_size_infra: Standard_D2s_v3 27 | vm_size_node: Standard_D2s_v3 28 | vm_size_bastion: Standard_D2s_v3 29 | # vm_size_cns: Standard_D8s_v3 30 | 31 | # Disk sizes for OS Disk and Data Disk in GB 32 | os_disk_size: 64 33 | data_disk_size: 64 34 | 35 | # Type of disk used 36 | # disk_type: managed 37 | managed_disk_type: Premium_LRS 38 | 39 | ####################### 40 | # SSH user for all VMs created 41 | ####################### 42 | admin_username: azureuser 43 | # This will be added to each instance's authorized_keys file 44 | admin_pubkey: 'ssh-rsa LDKJFDLKJDFLDKJDF' 45 | # Path for the private key which will be added to localhost ssh_config for proxying through the bastion 46 | admin_privkey: '~/.ssh/id_rsa' 47 | 48 | # If set, creates local users with password-less sudo on the bastion. 49 | # bastion_users: 50 | # user1: 'ssh-rsa LDKJFDLKJDFLDKJDF' 51 | # user2: 'ssh-rsa FDLKJDFLDKJDFLDKJ' 52 | 53 | ####################### 54 | # Service Principal 55 | ####################### 56 | # This is used to template the cloud config file on each node for azure 57 | aad_client_id: "" 58 | 59 | # Place this in vars file that can be encrypted in root of play and not checked into SCM 60 | aad_client_secret: "" 61 | subscription_id: "" 62 | tenant_id: "" 63 | 64 | ####################### 65 | # RHSM 66 | # Username / Password or 67 | # Activation key / OrgId 68 | ####################### 69 | rhsm_username_org: '' 70 | rhsm_password_key: '' 71 | # rhsm_key: "" 72 | # rhsm_org: "" 73 | # Can specify separate pools so only Application nodes use paid subs 74 | # or keep the same if only using one pool 75 | rhsm_pool: "" 76 | # rhsm_broker_pool: "" 77 | # rhsm_node_pool: "" 78 | # rhsm_repos: 79 | # - rhel-7-server-rpms 80 | # - rhel-7-server-extras-rpms 81 | # - rhel-7-server-ose-3.9-rpms 82 | # - rhel-7-fast-datapath-rpms 83 | # - rhel-7-server-ansible-2.4-rpms 84 | # bastion_pkgs: 85 | # - ansible 86 | # - atomic-openshift-utils 87 | # - atomic-openshift-clients 88 | # - git 89 | # - tmux 90 | # - screen 91 | 92 | ####################### 93 | # Network 94 | ####################### 95 | virtual_network_name: "openshiftvnet" 96 | virtual_network_cidr: "10.0.0.0/14" 97 | master_subnet_cidr: "10.1.0.0/16" 98 | infra_subnet_cidr: "10.2.0.0/16" 99 | node_subnet_cidr: "10.3.0.0/16" 100 | cns_subnet_cidr: "10.4.0.0/16" 101 | 102 | bastion_publicip: "bastion{{ unique_string }}" 103 | # api_port: 443 104 | 105 | # If this is set, the bastion host is configured with the given static 106 | # IP address. A public IP address is still assigned to the bastion 107 | # because it is required during installation, but can be removed in 108 | # Azure after the installation is done. This is useful in environments 109 | # where the VNet is accessible from a private network. 110 | # bastion_private_ip: 192.168.0.20 111 | 112 | # Override the defaults for the SDN cluster network and the services 113 | # network. 114 | # osm_cluster_network_cidr: 10.29.0.0/16 115 | # openshift_portal_net: 10.28.0.0/16 116 | 117 | ####################### 118 | # LB 119 | ####################### 120 | # Name of Master LB public IP address 121 | master_lb_public_ip: "masterdns{{ unique_string }}" 122 | routing: "apps.contoso.com" 123 | 124 | # If this is set, the master load balancer will be configured as an 125 | # internal Azure load balancer with the given IP address as its 126 | # frontend IP address. There will be no public IP address assigned to 127 | # the load balancer. This is useful in environments where the VNet is 128 | # accessible from a private network. 129 | #master_lb_private_ip: 192.168.0.21 130 | #master_lb_private_dns: ocp-console.example.com 131 | 132 | # Name of Router LB 133 | router_lb_public_ip: "routerdns{{ unique_string }}" 134 | 135 | # registry Storage Account Name 136 | registry_storage_account: "registry{{ unique_string }}" 137 | 138 | # Storage account for default Storage Class if using unmanaged disks 139 | unmanaged_storage_class_account: "scsa{{ unique_string }}" 140 | 141 | ####################### 142 | # Custom Named Certs 143 | ####################### 144 | # Router 145 | #router_cert: '{"cafile": "/path/to/ca_cert", "certfile": "/path/to/fullchain.cer", "keyfile": "/vagrant/keys/domain.key"}' 146 | ## Master 147 | #master_cert: '[{"cafile": "/path/to/ca_cert", "certfile": "/path/to/fullchain.cer", "keyfile": "/path/to/key/domain.key", "names": ["openshift.console.domain.name"]}]' 148 | 149 | 150 | ####################### 151 | # OCP Identity 152 | # Use 'htpasswd -n ' to generate password hash. (htpasswd from httpd-tools RPM) 153 | ####################### 154 | # Example with admin:changeme 155 | # openshift_master_htpasswd_users: {'admin': '$apr1$zAhyA9Ko$rBxBOwAwwtRuuaw8OtCwH0'} 156 | # openshift_master_identity_providers: [{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}] 157 | 158 | # 159 | ocp_admin_passwd: 'redhat@123' 160 | 161 | ####################### 162 | # deploy CNS 163 | ####################### 164 | deploy_cns: false 165 | # deploy_cns_on_infra: false 166 | 167 | ####################### 168 | # deploy additional components 169 | ####################### 170 | 171 | deploy_metrics: true 172 | deploy_logging: true 173 | # deploy_prometheus: true 174 | deploy_azure_cloud_provider: true 175 | 176 | # volume sizing defaults 177 | # metrics_volume_size: '20Gi' 178 | # logging_volume_size: '100Gi' 179 | # prometheus_volume_size: '20Gi' 180 | 181 | ####################### 182 | # UI customization 183 | ####################### 184 | 185 | #project_request_message: To create a new project, contact your team administrator. 186 | 187 | ####################### 188 | ####################### 189 | # Don't change 190 | ####################### 191 | ####################### 192 | # bastion: "{{ resource_group }}b.{{ location }}.cloudapp.azure.com" 193 | -------------------------------------------------------------------------------- /azuredeploy.parameters.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "_artifactsLocation": { 6 | "value": "https://raw.githubusercontent.com/Microsoft/openshift-container-platform/master" 7 | }, 8 | "masterVmSize": { 9 | "value": "Standard_E2s_v3" 10 | }, 11 | "infraVmSize": { 12 | "value": "Standard_D4s_v3" 13 | }, 14 | "nodeVmSize": { 15 | "value": "Standard_D4s_v3" 16 | }, 17 | "cnsVmSize": { 18 | "value": "Standard_E4s_v3" 19 | }, 20 | "osImageType": { 21 | "value": "defaultgallery" 22 | }, 23 | "marketplaceOsImage": { 24 | "value": { 25 | "publisher": "RedHat", 26 | "offer": "RHEL", 27 | "sku": "7-RAW", 28 | "version": "latest" 29 | } 30 | }, 31 | "storageKind": { 32 | "value": "changeme" 33 | }, 34 | "openshiftClusterPrefix": { 35 | "value": "changeme" 36 | }, 37 | "minorVersion": { 38 | "value": "69" 39 | }, 40 | "masterInstanceCount": { 41 | "value": 3 42 | }, 43 | "infraInstanceCount": { 44 | "value": 3 45 | }, 46 | "nodeInstanceCount": { 47 | "value": 3 48 | }, 49 | "cnsInstanceCount": { 50 | "value": 3 51 | }, 52 | "dataDiskSize": { 53 | "value": 128 54 | }, 55 | "cnsGlusterDiskSize": { 56 | "value": 128 57 | }, 58 | "adminUsername": { 59 | "value": "changeme" 60 | }, 61 | "enableMetrics": { 62 | "value": "false" 63 | }, 64 | "enableLogging": { 65 | "value": "false" 66 | }, 67 | "enableCNS": { 68 | "value": "false" 69 | }, 70 | "rhsmUsernameOrOrgId": { 71 | "value": "changeme" 72 | }, 73 | "rhsmPoolId": { 74 | "value": "changeme" 75 | }, 76 | "rhsmBrokerPoolId": { 77 | "value": "changeme" 78 | }, 79 | "sshPublicKey": { 80 | "value": "GEN-SSH-PUB-KEY" 81 | }, 82 | "keyVaultSubscriptionId": { 83 | "value": "255a325e-8276-4ada-af8f-33af5658eb34" 84 | }, 85 | "keyVaultResourceGroup": { 86 | "value": "changeme" 87 | }, 88 | "keyVaultName": { 89 | "value": "changeme" 90 | }, 91 | "enableAzure": { 92 | "value": "true" 93 | }, 94 | "aadClientId": { 95 | "value": "changeme" 96 | }, 97 | "domainName": { 98 | "value": "none" 99 | }, 100 | "masterClusterDnsType": { 101 | "value": "default" 102 | }, 103 | "masterClusterDns": { 104 | "value": "console.contoso.com" 105 | }, 106 | "routingSubDomainType": { 107 | "value": "nipio" 108 | }, 109 | "routingSubDomain": { 110 | "value": "apps.contoso.com" 111 | }, 112 | "virtualNetworkNewOrExisting": { 113 | "value": "new" 114 | }, 115 | "virtualNetworkName": { 116 | "value": "changeme" 117 | }, 118 | "addressPrefixes": { 119 | "value": "10.0.0.0/14" 120 | }, 121 | "masterSubnetName": { 122 | "value": "changeme" 123 | }, 124 | "masterSubnetPrefix": { 125 | "value": "10.1.0.0/16" 126 | }, 127 | "infraSubnetName": { 128 | "value": "changeme" 129 | }, 130 | "infraSubnetPrefix": { 131 | "value": "10.2.0.0/16" 132 | }, 133 | "nodeSubnetName": { 134 | "value": "changeme" 135 | }, 136 | "nodeSubnetPrefix": { 137 | "value": "10.3.0.0/16" 138 | }, 139 | "existingMasterSubnetReference": { 140 | "value": "/subscriptions/abc686f6-963b-4e64-bff4-99dc369ab1cd/resourceGroups/vnetresourcegroup/providers/Microsoft.Network/virtualNetworks/openshiftvnet/subnets/mastersubnet" 141 | }, 142 | "existingInfraSubnetReference": { 143 | "value": "/subscriptions/abc686f6-963b-4e64-bff4-99dc369ab1cd/resourceGroups/vnetresourcegroup/providers/Microsoft.Network/virtualNetworks/openshiftvnet/subnets/infrasubnet" 144 | }, 145 | "existingCnsSubnetReference": { 146 | "value": "/subscriptions/abc686f6-963b-4e64-bff4-99dc369ab1cd/resourceGroups/vnetresourcegroup/providers/Microsoft.Network/virtualNetworks/openshiftvnet/subnets/cnssubnet" 147 | }, 148 | "existingNodeSubnetReference": { 149 | "value": "/subscriptions/abc686f6-963b-4e64-bff4-99dc369ab1cd/resourceGroups/vnetresourcegroup/providers/Microsoft.Network/virtualNetworks/openshiftvnet/subnets/nodesubnet" 150 | }, 151 | "masterClusterType": { 152 | "value": "public" 153 | }, 154 | "masterPrivateClusterIp": { 155 | "value": "10.1.0.200" 156 | }, 157 | "routerClusterType": { 158 | "value": "public" 159 | }, 160 | "routerPrivateClusterIp": { 161 | "value": "10.2.0.200" 162 | }, 163 | "routingCertType": { 164 | "value": "selfsigned" 165 | }, 166 | "masterCertType": { 167 | "value": "selfsigned" 168 | } 169 | } 170 | } 171 | -------------------------------------------------------------------------------- /azuredeploy.parameters.sample.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "_artifactsLocation": { 6 | "value": "https://raw.githubusercontent.com/Microsoft/openshift-container-platform/master" 7 | }, 8 | "masterVmSize": { 9 | "value": "Standard_E2s_v3" 10 | }, 11 | "infraVmSize": { 12 | "value": "Standard_D4s_v3" 13 | }, 14 | "nodeVmSize": { 15 | "value": "Standard_D4s_v3" 16 | }, 17 | "cnsVmSize": { 18 | "value": "Standard_E4s_v3" 19 | }, 20 | "osImageType": { 21 | "value": "defaultgallery" 22 | }, 23 | "marketplaceOsImage": { 24 | "value": { 25 | "publisher": "RedHat", 26 | "offer": "RHEL", 27 | "sku": "7-RAW", 28 | "version": "latest" 29 | } 30 | }, 31 | "storageKind": { 32 | "value": "managed" 33 | }, 34 | "openshiftClusterPrefix": { 35 | "value": "ocpcluster" 36 | }, 37 | "minorVersion": { 38 | "value": "69" 39 | }, 40 | "masterInstanceCount": { 41 | "value": 3 42 | }, 43 | "infraInstanceCount": { 44 | "value": 3 45 | }, 46 | "nodeInstanceCount": { 47 | "value": 3 48 | }, 49 | "cnsInstanceCount": { 50 | "value": 3 51 | }, 52 | "dataDiskSize": { 53 | "value": 128 54 | }, 55 | "cnsGlusterDiskSize": { 56 | "value": 128 57 | }, 58 | "adminUsername": { 59 | "value": "ocpadmin" 60 | }, 61 | "enableMetrics": { 62 | "value": "false" 63 | }, 64 | "enableLogging": { 65 | "value": "false" 66 | }, 67 | "enableCNS": { 68 | "value": "false" 69 | }, 70 | "rhsmUsernameOrOrgId": { 71 | "value": "myrhsmusername" 72 | }, 73 | "rhsmPoolId": { 74 | "value": "1a23f12345c8c8380166abcde01a251b" 75 | }, 76 | "rhsmBrokerPoolId": { 77 | "value": "1a23f12345c8c8380166abcde01a111a" 78 | }, 79 | "sshPublicKey": { 80 | "value": "GEN-SSH-PUB-KEY" 81 | }, 82 | "keyVaultSubscriptionId": { 83 | "value": "255a325e-8276-4ada-af8f-33af5658eb34" 84 | }, 85 | "keyVaultResourceGroup": { 86 | "value": "keyvaultrg" 87 | }, 88 | "keyVaultName": { 89 | "value": "keyvault" 90 | }, 91 | "enableAzure": { 92 | "value": "true" 93 | }, 94 | "aadClientId": { 95 | "value": "aa1f18d9-1234-4f11-11cb-d6da4f6def11" 96 | }, 97 | "domainName": { 98 | "value": "none" 99 | }, 100 | "masterClusterDnsType": { 101 | "value": "default" 102 | }, 103 | "masterClusterDns": { 104 | "value": "console.contoso.com" 105 | }, 106 | "routingSubDomainType": { 107 | "value": "nipio" 108 | }, 109 | "routingSubDomain": { 110 | "value": "apps.contoso.com" 111 | }, 112 | "virtualNetworkNewOrExisting": { 113 | "value": "new" 114 | }, 115 | "virtualNetworkName": { 116 | "value": "ocpvnet" 117 | }, 118 | "addressPrefixes": { 119 | "value": "10.0.0.0/14" 120 | }, 121 | "masterSubnetName": { 122 | "value": "mastersubnet" 123 | }, 124 | "masterSubnetPrefix": { 125 | "value": "10.1.0.0/16" 126 | }, 127 | "infraSubnetName": { 128 | "value": "infrasubnet" 129 | }, 130 | "infraSubnetPrefix": { 131 | "value": "10.2.0.0/16" 132 | }, 133 | "nodeSubnetName": { 134 | "value": "nodesubnet" 135 | }, 136 | "nodeSubnetPrefix": { 137 | "value": "10.3.0.0/16" 138 | }, 139 | "existingMasterSubnetReference": { 140 | "value": "/subscriptions/abc686f6-963b-4e64-bff4-99dc369ab1cd/resourceGroups/vnetresourcegroup/providers/Microsoft.Network/virtualNetworks/openshiftvnet/subnets/mastersubnet" 141 | }, 142 | "existingInfraSubnetReference": { 143 | "value": "/subscriptions/abc686f6-963b-4e64-bff4-99dc369ab1cd/resourceGroups/vnetresourcegroup/providers/Microsoft.Network/virtualNetworks/openshiftvnet/subnets/infrasubnet" 144 | }, 145 | "existingCnsSubnetReference": { 146 | "value": "/subscriptions/abc686f6-963b-4e64-bff4-99dc369ab1cd/resourceGroups/vnetresourcegroup/providers/Microsoft.Network/virtualNetworks/openshiftvnet/subnets/cnssubnet" 147 | }, 148 | "existingNodeSubnetReference": { 149 | "value": "/subscriptions/abc686f6-963b-4e64-bff4-99dc369ab1cd/resourceGroups/vnetresourcegroup/providers/Microsoft.Network/virtualNetworks/openshiftvnet/subnets/nodesubnet" 150 | }, 151 | "masterClusterType": { 152 | "value": "public" 153 | }, 154 | "masterPrivateClusterIp": { 155 | "value": "10.1.0.200" 156 | }, 157 | "routerClusterType": { 158 | "value": "public" 159 | }, 160 | "routerPrivateClusterIp": { 161 | "value": "10.2.0.200" 162 | }, 163 | "routingCertType": { 164 | "value": "selfsigned" 165 | }, 166 | "masterCertType": { 167 | "value": "selfsigned" 168 | } 169 | } 170 | } 171 | -------------------------------------------------------------------------------- /images/openshiftdiagram.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/openshift-container-platform/5560bb14c41899104d49e34cd3736b0206caa942/images/openshiftdiagram.jpg -------------------------------------------------------------------------------- /images/openshiftiambcd.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/openshift-container-platform/5560bb14c41899104d49e34cd3736b0206caa942/images/openshiftiambcd.jpg -------------------------------------------------------------------------------- /nested/bastionprep.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "location": { 6 | "type": "string", 7 | "metadata": { 8 | "description": "Datacenter Region Location" 9 | } 10 | }, 11 | "apiVersionCompute": { 12 | "type": "string", 13 | "metadata": { 14 | "description": "API version for compute resources" 15 | } 16 | }, 17 | "bastionPrepScriptUrl": { 18 | "type": "string", 19 | "metadata": { 20 | "description": "Bastion prep script Url" 21 | } 22 | }, 23 | "bastionPrepScriptFileName": { 24 | "type": "string", 25 | "metadata": { 26 | "description": "Bastion prep script file name" 27 | } 28 | }, 29 | "openshiftBastionHostname": { 30 | "type": "string", 31 | "metadata": { 32 | "description": "OpenShift Bastion hostname" 33 | } 34 | }, 35 | "adminUsername": { 36 | "type": "string", 37 | "metadata": { 38 | "description": "Administrator username on all VMs" 39 | } 40 | }, 41 | "domainName": { 42 | "type": "string", 43 | "metadata": { 44 | "description": "Domain name search" 45 | } 46 | }, 47 | "rhsmUsernameOrOrgId": { 48 | "type": "string", 49 | "metadata": { 50 | "description": "Red Hat Subscription Manager username or Organization Id" 51 | } 52 | }, 53 | "rhsmPasswordOrActivationKey": { 54 | "type": "securestring", 55 | "metadata": { 56 | "description": "Red Hat Subscription Manager password or activation key" 57 | } 58 | }, 59 | "rhsmPoolId": { 60 | "type": "string", 61 | "metadata": { 62 | "description": "Pool Id with OpenShift entitlements" 63 | } 64 | }, 65 | "sshPrivateKey": { 66 | "type": "securestring", 67 | "metadata": { 68 | "description": "SSH private key for deploying OpenShift" 69 | } 70 | }, 71 | "routingCertType": { 72 | "type": "string", 73 | "metadata": { 74 | "description": "Use custom certificate for routing domain or the default self-signed certificate" 75 | } 76 | }, 77 | "masterCertType": { 78 | "type": "string", 79 | "metadata": { 80 | "description": "Use custom certificate for master domain or the default self-signed certificate" 81 | } 82 | }, 83 | "customRoutingCaFile": { 84 | "type": "securestring", 85 | "metadata": { 86 | "description": "Routing domain CA file" 87 | } 88 | }, 89 | "customRoutingCertFile": { 90 | "type": "securestring", 91 | "metadata": { 92 | "description": "Routing domain Cert file" 93 | } 94 | }, 95 | "customRoutingKeyFile": { 96 | "type": "securestring", 97 | "metadata": { 98 | "description": "Routing domain Key file" 99 | } 100 | }, 101 | "customMasterCaFile": { 102 | "type": "securestring", 103 | "metadata": { 104 | "description": "Master domain CA file" 105 | } 106 | }, 107 | "customMasterCertFile": { 108 | "type": "securestring", 109 | "metadata": { 110 | "description": "Master domain Cert file" 111 | } 112 | }, 113 | "customMasterKeyFile": { 114 | "type": "securestring", 115 | "metadata": { 116 | "description": "Master domain Key file" 117 | } 118 | }, 119 | "minorVersion": { 120 | "type": "string", 121 | "metadata": { 122 | "description": "Minor version of OpenShift 3.11 to deploy" 123 | } 124 | }, 125 | "masterClusterDnsType": { 126 | "type": "string", 127 | "metadata": { 128 | "description": "Domain type for web console - default (use DNS label of master infra public IP) or custom (defined in next parameter)" 129 | } 130 | }, 131 | "routingSubDomainType": { 132 | "type": "string", 133 | "metadata": { 134 | "description": "Default Subdomain type - nip.io or custom (defined in next parameter)" 135 | } 136 | }, 137 | "redHatTags": { 138 | "type": "object", 139 | "metadata": { 140 | "description": "Red Hat Tags" 141 | } 142 | } 143 | }, 144 | "variables": { 145 | "singlequote": "'" 146 | }, 147 | "resources": [{ 148 | "type": "Microsoft.Compute/virtualMachines/extensions", 149 | "name": "[concat(parameters('openshiftBastionHostname'), '/deployOpenShift')]", 150 | "location": "[parameters('location')]", 151 | "apiVersion": "[parameters('apiVersionCompute')]", 152 | "tags": { 153 | "displayName": "PrepBastion", 154 | "provider": "[parameters('redHatTags').provider]", 155 | "app": "[parameters('redHatTags').app]", 156 | "version": "[parameters('redHatTags').version]", 157 | "platform": "[parameters('redHatTags').platform]" 158 | }, 159 | "properties": { 160 | "publisher": "Microsoft.Azure.Extensions", 161 | "type": "CustomScript", 162 | "typeHandlerVersion": "2.0", 163 | "autoUpgradeMinorVersion": true, 164 | "settings": { 165 | "fileUris": [ 166 | "[parameters('bastionPrepScriptUrl')]" 167 | ] 168 | }, 169 | "protectedSettings": { 170 | "commandToExecute": "[concat('bash ', parameters('bastionPrepScriptFileName'), ' ', parameters('rhsmUsernameOrOrgId'), ' ',variables('singlequote'), parameters('rhsmPasswordOrActivationKey'), variables('singlequote'), ' ', parameters('rhsmPoolId'), ' ', variables('singlequote'), parameters('sshPrivateKey'), variables('singlequote'), ' ', variables('singlequote'), parameters('adminUsername'), variables('singlequote'), ' ', variables('singlequote'), parameters('routingCertType'), variables('singlequote'), ' ', variables('singlequote'), parameters('masterCertType'), variables('singlequote'), ' ', variables('singlequote'), parameters('customRoutingCaFile'), variables('singlequote'), ' ', variables('singlequote'), parameters('customRoutingCertFile'), variables('singlequote'), ' ', variables('singlequote'), parameters('customRoutingKeyFile'), variables('singlequote'), ' ', variables('singlequote'), parameters('customMasterCaFile'), variables('singlequote'), ' ', variables('singlequote'), parameters('customMasterCertFile'), variables('singlequote'), ' ', variables('singlequote'), parameters('customMasterKeyFile'), variables('singlequote'), ' ', variables('singlequote'), parameters('domainName'), variables('singlequote'), ' ', variables('singlequote'), parameters('minorVersion'), variables('singlequote'), ' ', variables('singlequote'), parameters('masterClusterDnsType'), variables('singlequote'), ' ', variables('singlequote'), parameters('routingSubDomainType'), variables('singlequote'))]" 171 | } 172 | } 173 | }], 174 | "outputs": {} 175 | } 176 | -------------------------------------------------------------------------------- /nested/gallerybasic.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "location": { 6 | "type": "string", 7 | "metadata": { 8 | "description": "Datacenter Region Location" 9 | } 10 | }, 11 | "sshKeyPath": { 12 | "type": "string", 13 | "metadata": { 14 | "description": "SSH Public Key Path" 15 | } 16 | }, 17 | "sshPublicKey": { 18 | "type": "string", 19 | "metadata": { 20 | "description": "SSH Public Key" 21 | } 22 | }, 23 | "adminUsername": { 24 | "type": "string", 25 | "metadata": { 26 | "description": "Admin Username" 27 | } 28 | }, 29 | "vmSize": { 30 | "type": "string", 31 | "metadata": { 32 | "description": "VM Size" 33 | } 34 | }, 35 | "osImageType": { 36 | "type": "string", 37 | "metadata": { 38 | "description": "OpenShift OS image type. 'defaultgallery' will use the on demand RHEL image. 'marketplace' allows you to define a third party marketplace image to use." 39 | } 40 | }, 41 | "marketplaceOsImage": { 42 | "type": "object", 43 | "metadata": { 44 | "description": "Enter the appropriate publisher, offer, sku and version values for the appropriate marketplace image you want to use." 45 | } 46 | }, 47 | "marketplacePlan": { 48 | "type": "object", 49 | "metadata": { 50 | "description": "Plan object for marketplace offer." 51 | } 52 | }, 53 | "hostName": { 54 | "type": "string", 55 | "metadata": { 56 | "description": "VM Hostname" 57 | } 58 | }, 59 | "unmanagedOsDiskUri": { 60 | "type": "string", 61 | "metadata": { 62 | "description": "Unmanaged OS disk uri" 63 | } 64 | }, 65 | "role": { 66 | "type": "string", 67 | "metadata": { 68 | "description": "VM Role for tag" 69 | } 70 | }, 71 | "vmStorageType": { 72 | "type": "string", 73 | "metadata": { 74 | "description": "VM Storage Type" 75 | } 76 | }, 77 | "storageKind": { 78 | "type": "string", 79 | "metadata": { 80 | "description": "Managed or Unmanaged disk" 81 | } 82 | }, 83 | "newStorageAccount": { 84 | "type": "string", 85 | "metadata": { 86 | "description": "Storage Account" 87 | } 88 | }, 89 | "diagStorageAccount": { 90 | "type": "string", 91 | "metadata": { 92 | "description": "Diagnostics Storage Account" 93 | } 94 | }, 95 | "apiVersionStorage": { 96 | "type": "string", 97 | "metadata": { 98 | "description": "Storage API Version" 99 | } 100 | }, 101 | "apiVersionCompute": { 102 | "type": "string", 103 | "metadata": { 104 | "description": "Compute API Version" 105 | } 106 | }, 107 | "imageReference": { 108 | "type": "object", 109 | "metadata": { 110 | "description": "Image Reference" 111 | } 112 | }, 113 | "redHatTags": { 114 | "type": "object", 115 | "metadata": { 116 | "description": "Red Hat Tags" 117 | } 118 | } 119 | }, 120 | "variables": { 121 | "managedStorageProfile": { 122 | "imageReference": "[if(equals(parameters('osImageType'), 'marketplace'), parameters('marketplaceOsImage'), parameters('imageReference'))]", 123 | "osDisk": { 124 | "name": "[concat(parameters('hostName'), '-osdisk')]", 125 | "managedDisk": { 126 | "storageAccountType": "[parameters('vmStorageType')]" 127 | }, 128 | "caching": "ReadWrite", 129 | "createOption": "FromImage", 130 | "osType": "Linux" 131 | } 132 | }, 133 | "unmanagedStorageProfile": { 134 | "imageReference": "[if(equals(parameters('osImageType'), 'marketplace'), parameters('marketplaceOsImage'), parameters('imageReference'))]", 135 | "osDisk": { 136 | "name": "[concat(parameters('hostName'), '-osdisk')]", 137 | "vhd": { 138 | "uri": "[concat(parameters('unmanagedOsDiskUri'), 'vhds/', parameters('hostname'), '-osdisk.vhd')]" 139 | }, 140 | "caching": "ReadWrite", 141 | "createOption": "FromImage" 142 | } 143 | }, 144 | "storageProfile": "[concat(parameters('storageKind'), 'StorageProfile')]" 145 | }, 146 | "resources": [{ 147 | "type": "Microsoft.Compute/virtualMachines", 148 | "name": "[parameters('hostName')]", 149 | "location": "[parameters('location')]", 150 | "apiVersion": "[parameters('apiVersionCompute')]", 151 | "tags": { 152 | "Role": "[parameters('role')]", 153 | "provider": "[parameters('redHatTags').provider]", 154 | "app": "[parameters('redHatTags').app]", 155 | "version": "[parameters('redHatTags').version]", 156 | "platform": "[parameters('redHatTags').platform]" 157 | }, 158 | "plan": "[if(equals(parameters('osImageType'), 'marketplace'), parameters('marketplacePlan'), json('null'))]", 159 | "properties": { 160 | "hardwareProfile": { 161 | "vmSize": "[parameters('vmSize')]" 162 | }, 163 | "osProfile": { 164 | "computerName": "[parameters('hostName')]", 165 | "adminUsername": "[parameters('adminUsername')]", 166 | "linuxConfiguration": { 167 | "disablePasswordAuthentication": true, 168 | "ssh": { 169 | "publicKeys": [{ 170 | "path": "[parameters('sshKeyPath')]", 171 | "keyData": "[parameters('sshPublicKey')]" 172 | }] 173 | } 174 | } 175 | }, 176 | "storageProfile": "[variables(variables('storageProfile'))]", 177 | "networkProfile": { 178 | "networkInterfaces": [{ 179 | "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(parameters('hostName'), '-nic'))]" 180 | }] 181 | }, 182 | "diagnosticsProfile": { 183 | "bootDiagnostics": { 184 | "enabled": true, 185 | "storageUri": "[concat(concat(reference(resourceId(resourceGroup().name, 'Microsoft.Storage/storageAccounts', parameters('diagStorageAccount')), parameters('apiVersionStorage')).primaryEndpoints['blob']))]" 186 | } 187 | } 188 | } 189 | }], 190 | "outputs": {} 191 | } 192 | -------------------------------------------------------------------------------- /nested/galleryclustercnsnode.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "location": { 6 | "type": "string", 7 | "metadata": { 8 | "description": "Datacenter Region Location" 9 | } 10 | }, 11 | "sshKeyPath": { 12 | "type": "string", 13 | "metadata": { 14 | "description": "SSH Public Key Path" 15 | } 16 | }, 17 | "sshPublicKey": { 18 | "type": "string", 19 | "metadata": { 20 | "description": "SSH Public Key" 21 | } 22 | }, 23 | "osDiskSize": { 24 | "type": "int", 25 | "metadata": { 26 | "description": "Size of Data Disk" 27 | } 28 | }, 29 | "dataDiskSize": { 30 | "type": "int", 31 | "metadata": { 32 | "description": "Size of Data Disk" 33 | } 34 | }, 35 | "glusterDiskSize": { 36 | "type": "int", 37 | "metadata": { 38 | "description": "Size of Data Disks for Gluster" 39 | } 40 | }, 41 | "adminUsername": { 42 | "type": "string", 43 | "metadata": { 44 | "description": "Admin Username" 45 | } 46 | }, 47 | "vmSize": { 48 | "type": "string", 49 | "metadata": { 50 | "description": "VM Size" 51 | } 52 | }, 53 | "osImageType": { 54 | "type": "string", 55 | "metadata": { 56 | "description": "OpenShift OS image type. 'defaultgallery' will use the on demand RHEL image. 'marketplace' allows you to define a third party marketplace image to use." 57 | } 58 | }, 59 | "marketplaceOsImage": { 60 | "type": "object", 61 | "metadata": { 62 | "description": "Enter the appropriate publisher, offer, sku and version values for the appropriate marketplace image you want to use." 63 | } 64 | }, 65 | "marketplacePlan": { 66 | "type": "object", 67 | "metadata": { 68 | "description": "Plan object for marketplace offer." 69 | } 70 | }, 71 | "availabilitySet": { 72 | "type": "string", 73 | "metadata": { 74 | "description": "Name of Availibility Set" 75 | } 76 | }, 77 | "hostName": { 78 | "type": "string", 79 | "metadata": { 80 | "description": "VM Hostname" 81 | } 82 | }, 83 | "unmanagedOsDiskUri": { 84 | "type": "string", 85 | "metadata": { 86 | "description": "Unmanaged OS disk uri" 87 | } 88 | }, 89 | "unmanagedDataDiskUri": { 90 | "type": "string", 91 | "metadata": { 92 | "description": "Unmanaged data disk uri" 93 | } 94 | }, 95 | "role": { 96 | "type": "string", 97 | "metadata": { 98 | "description": "VM Role for tag" 99 | } 100 | }, 101 | "vmStorageType": { 102 | "type": "string", 103 | "metadata": { 104 | "description": "VM Storage Type" 105 | } 106 | }, 107 | "storageKind": { 108 | "type": "string", 109 | "metadata": { 110 | "description": "Managed or Unmanaged disk" 111 | } 112 | }, 113 | "newStorageAccountOs": { 114 | "type": "string", 115 | "metadata": { 116 | "description": "Storage Account for OS disk" 117 | } 118 | }, 119 | "newStorageAccountData": { 120 | "type": "string", 121 | "metadata": { 122 | "description": "Storage Account for data disk" 123 | } 124 | }, 125 | "diagStorageAccount": { 126 | "type": "string", 127 | "metadata": { 128 | "description": "Diagnostics Storage Account" 129 | } 130 | }, 131 | "apiVersionStorage": { 132 | "type": "string", 133 | "metadata": { 134 | "description": "Storage API Version" 135 | } 136 | }, 137 | "apiVersionCompute": { 138 | "type": "string", 139 | "metadata": { 140 | "description": "Compute API Version" 141 | } 142 | }, 143 | "imageReference": { 144 | "type": "object", 145 | "metadata": { 146 | "description": "Image Reference" 147 | } 148 | }, 149 | "redHatTags": { 150 | "type": "object", 151 | "metadata": { 152 | "description": "Red Hat Tags" 153 | } 154 | } 155 | }, 156 | "variables": { 157 | "managedStorageProfile": { 158 | "imageReference": "[if(equals(parameters('osImageType'), 'marketplace'), parameters('marketplaceOsImage'), parameters('imageReference'))]", 159 | "osDisk": { 160 | "name": "[concat(parameters('hostName'), '-osdisk')]", 161 | "managedDisk": { 162 | "storageAccountType": "[parameters('vmStorageType')]" 163 | }, 164 | "caching": "ReadWrite", 165 | "createOption": "FromImage", 166 | "diskSizeGB": "[parameters('osDiskSize')]", 167 | "osType": "Linux" 168 | }, 169 | "dataDisks": [{ 170 | "name": "[concat(parameters('hostName'), '-docker-pool')]", 171 | "diskSizeGB": "[parameters('dataDiskSize')]", 172 | "lun": 0, 173 | "managedDisk": { 174 | "storageAccountType": "[parameters('vmStorageType')]" 175 | }, 176 | "createOption": "Empty" 177 | }, { 178 | "name": "[concat(parameters('hostName'), '-gluster-disk0')]", 179 | "diskSizeGB": "[parameters('glusterDiskSize')]", 180 | "lun": 1, 181 | "managedDisk": { 182 | "storageAccountType": "[parameters('vmStorageType')]" 183 | }, 184 | "createOption": "Empty" 185 | }, { 186 | "name": "[concat(parameters('hostName'), '-gluster-disk1')]", 187 | "diskSizeGB": "[parameters('glusterDiskSize')]", 188 | "lun": 2, 189 | "managedDisk": { 190 | "storageAccountType": "[parameters('vmStorageType')]" 191 | }, 192 | "createOption": "Empty" 193 | }, { 194 | "name": "[concat(parameters('hostName'), '-gluster-disk2')]", 195 | "diskSizeGB": "[parameters('glusterDiskSize')]", 196 | "lun": 3, 197 | "managedDisk": { 198 | "storageAccountType": "[parameters('vmStorageType')]" 199 | }, 200 | "createOption": "Empty" 201 | }] 202 | }, 203 | "unmanagedStorageProfile": { 204 | "imageReference": "[if(equals(parameters('osImageType'), 'marketplace'), parameters('marketplaceOsImage'), parameters('imageReference'))]", 205 | "osDisk": { 206 | "name": "[concat(parameters('hostName'), '-osdisk')]", 207 | "vhd": { 208 | "uri": "[concat(parameters('unmanagedOsDiskUri'), 'vhds/', parameters('hostname'), '-osdisk.vhd')]" 209 | }, 210 | "caching": "ReadWrite", 211 | "createOption": "FromImage", 212 | "diskSizeGB": 64 213 | }, 214 | "dataDisks": [{ 215 | "name": "[concat(parameters('hostname'), '-docker-pool')]", 216 | "diskSizeGB": "[parameters('dataDiskSize')]", 217 | "lun": 0, 218 | "vhd": { 219 | "uri": "[concat(parameters('unmanagedDataDiskUri'), 'vhds/', parameters('hostname'), '-docker-pool.vhd')]" 220 | }, 221 | "createOption": "Empty" 222 | }, { 223 | "name": "[concat(parameters('hostname'), '-gluster-disk0')]", 224 | "diskSizeGB": "[parameters('glusterDiskSize')]", 225 | "lun": 1, 226 | "vhd": { 227 | "uri": "[concat(parameters('unmanagedDataDiskUri'), 'vhds/', parameters('hostname'), '-gluster-disk0.vhd')]" 228 | }, 229 | "createOption": "Empty" 230 | }, { 231 | "name": "[concat(parameters('hostname'), '-gluster-disk1')]", 232 | "diskSizeGB": "[parameters('glusterDiskSize')]", 233 | "lun": 2, 234 | "vhd": { 235 | "uri": "[concat(parameters('unmanagedDataDiskUri'), 'vhds/', parameters('hostname'), '-gluster-disk1.vhd')]" 236 | }, 237 | "createOption": "Empty" 238 | }, { 239 | "name": "[concat(parameters('hostname'), '-gluster-disk2')]", 240 | "diskSizeGB": "[parameters('glusterDiskSize')]", 241 | "lun": 3, 242 | "vhd": { 243 | "uri": "[concat(parameters('unmanagedDataDiskUri'), 'vhds/', parameters('hostname'), '-gluster-disk2.vhd')]" 244 | }, 245 | "createOption": "Empty" 246 | }] 247 | }, 248 | "storageProfile": "[concat(parameters('storageKind'), 'StorageProfile')]" 249 | }, 250 | "resources": [{ 251 | "type": "Microsoft.Compute/virtualMachines", 252 | "name": "[parameters('hostName')]", 253 | "location": "[parameters('location')]", 254 | "apiVersion": "[parameters('apiVersionCompute')]", 255 | "tags": { 256 | "Role": "[parameters('role')]", 257 | "provider": "[parameters('redHatTags').provider]", 258 | "app": "[parameters('redHatTags').app]", 259 | "version": "[parameters('redHatTags').version]", 260 | "platform": "[parameters('redHatTags').platform]" 261 | }, 262 | "plan": "[if(equals(parameters('osImageType'), 'marketplace'), parameters('marketplacePlan'), json('null'))]", 263 | "properties": { 264 | "availabilitySet": { 265 | "id": "[resourceId('Microsoft.Compute/availabilitySets', parameters('availabilitySet'))]" 266 | }, 267 | "hardwareProfile": { 268 | "vmSize": "[parameters('vmSize')]" 269 | }, 270 | "osProfile": { 271 | "computerName": "[parameters('hostName')]", 272 | "adminUsername": "[parameters('adminUsername')]", 273 | "linuxConfiguration": { 274 | "disablePasswordAuthentication": true, 275 | "ssh": { 276 | "publicKeys": [{ 277 | "path": "[parameters('sshKeyPath')]", 278 | "keyData": "[parameters('sshPublicKey')]" 279 | }] 280 | } 281 | } 282 | }, 283 | "storageProfile": "[variables(variables('storageProfile'))]", 284 | "networkProfile": { 285 | "networkInterfaces": [{ 286 | "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(parameters('hostName'), '-nic'))]" 287 | }] 288 | }, 289 | "diagnosticsProfile": { 290 | "bootDiagnostics": { 291 | "enabled": true, 292 | "storageUri": "[concat(concat(reference(resourceId(resourceGroup().name, 'Microsoft.Storage/storageAccounts', parameters('diagStorageAccount')), parameters('apiVersionStorage')).primaryEndpoints['blob']))]" 293 | } 294 | } 295 | } 296 | }], 297 | "outputs": {} 298 | } 299 | -------------------------------------------------------------------------------- /nested/galleryclusternode.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "location": { 6 | "type": "string", 7 | "metadata": { 8 | "description": "Datacenter Region Location" 9 | } 10 | }, 11 | "sshKeyPath": { 12 | "type": "string", 13 | "metadata": { 14 | "description": "SSH Public Key Path" 15 | } 16 | }, 17 | "sshPublicKey": { 18 | "type": "string", 19 | "metadata": { 20 | "description": "SSH Public Key" 21 | } 22 | }, 23 | "osDiskSize": { 24 | "type": "int", 25 | "metadata": { 26 | "description": "Size of Data Disk" 27 | } 28 | }, 29 | "dataDiskSize": { 30 | "type": "int", 31 | "metadata": { 32 | "description": "Size of Data Disk" 33 | } 34 | }, 35 | "adminUsername": { 36 | "type": "string", 37 | "metadata": { 38 | "description": "Admin Username" 39 | } 40 | }, 41 | "vmSize": { 42 | "type": "string", 43 | "metadata": { 44 | "description": "VM Size" 45 | } 46 | }, 47 | "osImageType": { 48 | "type": "string", 49 | "metadata": { 50 | "description": "OpenShift OS image type. 'defaultgallery' will use the on demand RHEL image. 'marketplace' allows you to define a third party marketplace image to use." 51 | } 52 | }, 53 | "marketplaceOsImage": { 54 | "type": "object", 55 | "metadata": { 56 | "description": "Enter the appropriate publisher, offer, sku and version values for the appropriate marketplace image you want to use." 57 | } 58 | }, 59 | "marketplacePlan": { 60 | "type": "object", 61 | "metadata": { 62 | "description": "Plan object for marketplace offer." 63 | } 64 | }, 65 | "availabilitySet": { 66 | "type": "string", 67 | "metadata": { 68 | "description": "Name of Availibility Set" 69 | } 70 | }, 71 | "hostName": { 72 | "type": "string", 73 | "metadata": { 74 | "description": "VM Hostname" 75 | } 76 | }, 77 | "unmanagedOsDiskUri": { 78 | "type": "string", 79 | "metadata": { 80 | "description": "Unmanaged OS disk uri" 81 | } 82 | }, 83 | "unmanagedDataDiskUri": { 84 | "type": "string", 85 | "metadata": { 86 | "description": "Unmanaged data disk uri" 87 | } 88 | }, 89 | "role": { 90 | "type": "string", 91 | "metadata": { 92 | "description": "VM Role for tag" 93 | } 94 | }, 95 | "vmStorageType": { 96 | "type": "string", 97 | "metadata": { 98 | "description": "VM Storage Type" 99 | } 100 | }, 101 | "storageKind": { 102 | "type": "string", 103 | "metadata": { 104 | "description": "Managed or Unmanaged disk" 105 | } 106 | }, 107 | "newStorageAccountOs": { 108 | "type": "string", 109 | "metadata": { 110 | "description": "Storage Account for OS disk" 111 | } 112 | }, 113 | "newStorageAccountData": { 114 | "type": "string", 115 | "metadata": { 116 | "description": "Storage Account for data disk" 117 | } 118 | }, 119 | "diagStorageAccount": { 120 | "type": "string", 121 | "metadata": { 122 | "description": "Diagnostics Storage Account" 123 | } 124 | }, 125 | "apiVersionStorage": { 126 | "type": "string", 127 | "metadata": { 128 | "description": "Storage API Version" 129 | } 130 | }, 131 | "apiVersionCompute": { 132 | "type": "string", 133 | "metadata": { 134 | "description": "Compute API Version" 135 | } 136 | }, 137 | "imageReference": { 138 | "type": "object", 139 | "metadata": { 140 | "description": "Image Reference" 141 | } 142 | }, 143 | "redHatTags": { 144 | "type": "object", 145 | "metadata": { 146 | "description": "Red Hat Tags" 147 | } 148 | } 149 | }, 150 | "variables": { 151 | "managedStorageProfile": { 152 | "imageReference": "[if(equals(parameters('osImageType'), 'marketplace'), parameters('marketplaceOsImage'), parameters('imageReference'))]", 153 | "osDisk": { 154 | "name": "[concat(parameters('hostName'), '-osdisk')]", 155 | "managedDisk": { 156 | "storageAccountType": "[parameters('vmStorageType')]" 157 | }, 158 | "caching": "ReadWrite", 159 | "createOption": "FromImage", 160 | "diskSizeGB": "[parameters('osDiskSize')]", 161 | "osType": "Linux" 162 | }, 163 | "dataDisks": [{ 164 | "name": "[concat(parameters('hostName'), '-docker-pool')]", 165 | "diskSizeGB": "[parameters('dataDiskSize')]", 166 | "lun": 0, 167 | "managedDisk": { 168 | "storageAccountType": "[parameters('vmStorageType')]" 169 | }, 170 | "createOption": "Empty" 171 | }] 172 | }, 173 | "unmanagedStorageProfile": { 174 | "imageReference": "[if(equals(parameters('osImageType'), 'marketplace'), parameters('marketplaceOsImage'), parameters('imageReference'))]", 175 | "osDisk": { 176 | "name": "[concat(parameters('hostName'), '-osdisk')]", 177 | "vhd": { 178 | "uri": "[concat(parameters('unmanagedOsDiskUri'), 'vhds/', parameters('hostname'), '-osdisk.vhd')]" 179 | }, 180 | "caching": "ReadWrite", 181 | "createOption": "FromImage", 182 | "diskSizeGB": 64 183 | }, 184 | "dataDisks": [{ 185 | "name": "[concat(parameters('hostname'), '-docker-pool')]", 186 | "diskSizeGB": "[parameters('dataDiskSize')]", 187 | "lun": 0, 188 | "vhd": { 189 | "uri": "[concat(parameters('unmanagedDataDiskUri'), 'vhds/', parameters('hostname'), '-docker-pool.vhd')]" 190 | }, 191 | "createOption": "Empty" 192 | }] 193 | }, 194 | "storageProfile": "[concat(parameters('storageKind'), 'StorageProfile')]" 195 | }, 196 | "resources": [{ 197 | "type": "Microsoft.Compute/virtualMachines", 198 | "name": "[parameters('hostName')]", 199 | "location": "[parameters('location')]", 200 | "apiVersion": "[parameters('apiVersionCompute')]", 201 | "tags": { 202 | "Role": "[parameters('role')]", 203 | "provider": "[parameters('redHatTags').provider]", 204 | "app": "[parameters('redHatTags').app]", 205 | "version": "[parameters('redHatTags').version]", 206 | "platform": "[parameters('redHatTags').platform]" 207 | }, 208 | "plan": "[if(equals(parameters('osImageType'), 'marketplace'), parameters('marketplacePlan'), json('null'))]", 209 | "properties": { 210 | "availabilitySet": { 211 | "id": "[resourceId('Microsoft.Compute/availabilitySets', parameters('availabilitySet'))]" 212 | }, 213 | "hardwareProfile": { 214 | "vmSize": "[parameters('vmSize')]" 215 | }, 216 | "osProfile": { 217 | "computerName": "[parameters('hostName')]", 218 | "adminUsername": "[parameters('adminUsername')]", 219 | "linuxConfiguration": { 220 | "disablePasswordAuthentication": true, 221 | "ssh": { 222 | "publicKeys": [{ 223 | "path": "[parameters('sshKeyPath')]", 224 | "keyData": "[parameters('sshPublicKey')]" 225 | }] 226 | } 227 | } 228 | }, 229 | "storageProfile": "[variables(variables('storageProfile'))]", 230 | "networkProfile": { 231 | "networkInterfaces": [{ 232 | "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(parameters('hostName'), '-nic'))]" 233 | }] 234 | }, 235 | "diagnosticsProfile": { 236 | "bootDiagnostics": { 237 | "enabled": true, 238 | "storageUri": "[concat(concat(reference(resourceId(resourceGroup().name, 'Microsoft.Storage/storageAccounts', parameters('diagStorageAccount')), parameters('apiVersionStorage')).primaryEndpoints['blob']))]" 239 | } 240 | } 241 | } 242 | }], 243 | "outputs": {} 244 | } 245 | -------------------------------------------------------------------------------- /nested/masterprep.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "location": { 6 | "type": "string", 7 | "metadata": { 8 | "description": "Datacenter Region Location" 9 | } 10 | }, 11 | "apiVersionCompute": { 12 | "type": "string", 13 | "metadata": { 14 | "description": "API version for compute resources" 15 | } 16 | }, 17 | "masterPrepScriptUrl": { 18 | "type": "string", 19 | "metadata": { 20 | "description": "Master prep script Url" 21 | } 22 | }, 23 | "masterPrepScriptFileName": { 24 | "type": "string", 25 | "metadata": { 26 | "description": "Master prep script file name" 27 | } 28 | }, 29 | "openshiftMasterHostname": { 30 | "type": "string", 31 | "metadata": { 32 | "description": "OpenShift Master hostname" 33 | } 34 | }, 35 | "adminUsername": { 36 | "type": "string", 37 | "metadata": { 38 | "description": "Administrator username on all VMs" 39 | } 40 | }, 41 | "rhsmUsernameOrOrgId": { 42 | "type": "string", 43 | "metadata": { 44 | "description": "Red Hat Subscription Manager username or Organization Id" 45 | } 46 | }, 47 | "rhsmPasswordOrActivationKey": { 48 | "type": "securestring", 49 | "metadata": { 50 | "description": "Red Hat Subscription Manager password or activation key" 51 | } 52 | }, 53 | "rhsmPoolId": { 54 | "type": "string", 55 | "metadata": { 56 | "description": "Pool Id with OpenShift entitlements" 57 | } 58 | }, 59 | "redHatTags": { 60 | "type": "object", 61 | "metadata": { 62 | "description": "Red Hat Tags" 63 | } 64 | } 65 | }, 66 | "variables": { 67 | "singlequote": "'" 68 | }, 69 | "resources": [{ 70 | "type": "Microsoft.Compute/virtualMachines/extensions", 71 | "name": "[concat(parameters('openshiftMasterHostname'), '/prepMasters')]", 72 | "location": "[parameters('location')]", 73 | "apiVersion": "[parameters('apiVersionCompute')]", 74 | "tags": { 75 | "displayName": "PrepMaster", 76 | "provider": "[parameters('redHatTags').provider]", 77 | "app": "[parameters('redHatTags').app]", 78 | "version": "[parameters('redHatTags').version]", 79 | "platform": "[parameters('redHatTags').platform]" 80 | }, 81 | "properties": { 82 | "publisher": "Microsoft.Azure.Extensions", 83 | "type": "CustomScript", 84 | "typeHandlerVersion": "2.0", 85 | "autoUpgradeMinorVersion": true, 86 | "settings": { 87 | "fileUris": [ 88 | "[parameters('masterPrepScriptUrl')]" 89 | ] 90 | }, 91 | "protectedSettings": { 92 | "commandToExecute": "[concat('bash ', parameters('masterPrepScriptFileName'), ' ', variables('singlequote'), parameters('rhsmUsernameOrOrgId'), variables('singlequote'), ' ', variables('singlequote'), parameters('rhsmPasswordOrActivationKey'), variables('singlequote'), ' ', variables('singlequote'), parameters('rhsmPoolId'), variables('singlequote'))]" 93 | } 94 | } 95 | }], 96 | "outputs": {} 97 | } 98 | -------------------------------------------------------------------------------- /nested/nodeprep.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "location": { 6 | "type": "string", 7 | "metadata": { 8 | "description": "Datacenter Region Location" 9 | } 10 | }, 11 | "apiVersionCompute": { 12 | "type": "string", 13 | "metadata": { 14 | "description": "API version for compute resources" 15 | } 16 | }, 17 | "nodePrepScriptUrl": { 18 | "type": "string", 19 | "metadata": { 20 | "description": "Compute node prep script Url" 21 | } 22 | }, 23 | "nodePrepScriptFileName": { 24 | "type": "string", 25 | "metadata": { 26 | "description": "Compute node prep script file name" 27 | } 28 | }, 29 | "openshiftNodeHostname": { 30 | "type": "string", 31 | "metadata": { 32 | "description": "OpenShift Node hostname" 33 | } 34 | }, 35 | "rhsmUsernameOrOrgId": { 36 | "type": "string", 37 | "metadata": { 38 | "description": "Red Hat Subscription Manager username or Organization Id" 39 | } 40 | }, 41 | "rhsmPasswordOrActivationKey": { 42 | "type": "securestring", 43 | "metadata": { 44 | "description": "Red Hat Subscription Manager password or activation key" 45 | } 46 | }, 47 | "rhsmPoolId": { 48 | "type": "string", 49 | "metadata": { 50 | "description": "Pool Id with OpenShift entitlements" 51 | } 52 | }, 53 | "redHatTags": { 54 | "type": "object", 55 | "metadata": { 56 | "description": "Red Hat Tags" 57 | } 58 | } 59 | }, 60 | "variables": { 61 | "singlequote": "'" 62 | }, 63 | "resources": [{ 64 | "type": "Microsoft.Compute/virtualMachines/extensions", 65 | "name": "[concat(parameters('openshiftNodeHostname'), '/prepNodes')]", 66 | "location": "[parameters('location')]", 67 | "apiVersion": "[parameters('apiVersionCompute')]", 68 | "tags": { 69 | "displayName": "PrepMaster", 70 | "provider": "[parameters('redHatTags').provider]", 71 | "app": "[parameters('redHatTags').app]", 72 | "version": "[parameters('redHatTags').version]", 73 | "platform": "[parameters('redHatTags').platform]" 74 | }, 75 | "properties": { 76 | "publisher": "Microsoft.Azure.Extensions", 77 | "type": "CustomScript", 78 | "typeHandlerVersion": "2.0", 79 | "autoUpgradeMinorVersion": true, 80 | "settings": { 81 | "fileUris": [ 82 | "[parameters('nodePrepScriptUrl')]" 83 | ] 84 | }, 85 | "protectedSettings": { 86 | "commandToExecute": "[concat('bash ', parameters('nodePrepScriptFileName'), ' ', variables('singlequote'), parameters('rhsmUsernameOrOrgId'), variables('singlequote'), ' ',variables('singlequote'), parameters('rhsmPasswordOrActivationKey'), variables('singlequote'), ' ', variables('singlequote'), parameters('rhsmPoolId'), variables('singlequote'))]" 87 | } 88 | } 89 | }], 90 | "outputs": {} 91 | } 92 | -------------------------------------------------------------------------------- /nested/openshiftdeploy.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "location": { 6 | "type": "string", 7 | "metadata": { 8 | "description": "Datacenter Region Location" 9 | } 10 | }, 11 | "tenantId": { 12 | "type": "string", 13 | "metadata": { 14 | "description": "Tenant ID" 15 | } 16 | }, 17 | "subscriptionId": { 18 | "type": "string", 19 | "metadata": { 20 | "description": "Subscription ID" 21 | } 22 | }, 23 | "resourceGroupName": { 24 | "type": "string", 25 | "metadata": { 26 | "description": "Resource Group Name" 27 | } 28 | }, 29 | "apiVersionCompute": { 30 | "type": "string", 31 | "metadata": { 32 | "description": "API version for compute resources" 33 | } 34 | }, 35 | "openshiftDeploymentScriptUrl": { 36 | "type": "string", 37 | "metadata": { 38 | "description": "Bastion prep script Url" 39 | } 40 | }, 41 | "openshiftDeploymentScriptFileName": { 42 | "type": "string", 43 | "metadata": { 44 | "description": "Bastion prep script file name" 45 | } 46 | }, 47 | "newStorageAccountRegistry": { 48 | "type": "string", 49 | "metadata": { 50 | "description": "Storage Account for Docker Registry" 51 | } 52 | }, 53 | "newStorageAccountKey": { 54 | "type": "string", 55 | "metadata": { 56 | "description": "Storage Account Key for Docker Registry" 57 | } 58 | }, 59 | "openshiftBastionHostname": { 60 | "type": "string", 61 | "metadata": { 62 | "description": "OpenShift Bastion hostname" 63 | } 64 | }, 65 | "openshiftMasterHostname": { 66 | "type": "string", 67 | "metadata": { 68 | "description": "OpenShift Master hostname" 69 | } 70 | }, 71 | "openshiftMasterPublicIpFqdn": { 72 | "type": "string", 73 | "metadata": { 74 | "description": "OpenShift Master VM public IP fully qualified domain name" 75 | } 76 | }, 77 | "openshiftMasterPublicIpAddress": { 78 | "type": "string", 79 | "metadata": { 80 | "description": "OpenShift Master VM public IP address" 81 | } 82 | }, 83 | "openshiftInfraHostname": { 84 | "type": "string", 85 | "metadata": { 86 | "description": "OpenShift Infra Node hostname" 87 | } 88 | }, 89 | "openshiftNodeHostname": { 90 | "type": "string", 91 | "metadata": { 92 | "description": "OpenShift Node hostname" 93 | } 94 | }, 95 | "openshiftCnsHostname": { 96 | "type": "string", 97 | "metadata": { 98 | "description": "OpenShift CNS Node hostname" 99 | } 100 | }, 101 | "masterInstanceCount": { 102 | "type": "int", 103 | "metadata": { 104 | "description": "Number of OpenShift Master nodes" 105 | } 106 | }, 107 | "infraInstanceCount": { 108 | "type": "int", 109 | "metadata": { 110 | "description": "Number of OpenShift Infra nodes" 111 | } 112 | }, 113 | "nodeInstanceCount": { 114 | "type": "int", 115 | "metadata": { 116 | "description": "Number of OpenShift nodes" 117 | } 118 | }, 119 | "cnsInstanceCount": { 120 | "type": "int", 121 | "metadata": { 122 | "description": "Number of CNS nodes" 123 | } 124 | }, 125 | "storageKind": { 126 | "type": "string", 127 | "metadata": { 128 | "description": "Managed or unmanaged disk" 129 | } 130 | }, 131 | "adminUsername": { 132 | "type": "string", 133 | "metadata": { 134 | "description": "Administrator username on all VMs" 135 | } 136 | }, 137 | "openshiftPassword": { 138 | "type": "securestring", 139 | "metadata": { 140 | "description": "Administrator password for OpenShift Console" 141 | } 142 | }, 143 | "enableCNS": { 144 | "type": "string", 145 | "metadata": { 146 | "description": "Enable Container Native Storage: true or false" 147 | } 148 | }, 149 | "enableMetrics": { 150 | "type": "string", 151 | "metadata": { 152 | "description": "Enable OpenShift Metrics: true or false" 153 | } 154 | }, 155 | "enableLogging": { 156 | "type": "string", 157 | "metadata": { 158 | "description": "Enable OpenShift Logging: true or false" 159 | } 160 | }, 161 | "enableAzure": { 162 | "type": "string", 163 | "metadata": { 164 | "description": "Enable Azure: true or false" 165 | } 166 | }, 167 | "aadClientId": { 168 | "type": "string", 169 | "metadata": { 170 | "description": "Azure AD Client ID" 171 | } 172 | }, 173 | "aadClientSecret": { 174 | "type": "securestring", 175 | "metadata": { 176 | "description": "Azure AD Client Secret" 177 | } 178 | }, 179 | "nipioDomain": { 180 | "type": "string", 181 | "metadata": { 182 | "description": "nip.io Subdomain for application routing" 183 | } 184 | }, 185 | "customDomain": { 186 | "type": "string", 187 | "metadata": { 188 | "description": "custom Subdomain for application routing" 189 | } 190 | }, 191 | "subDomainChosen": { 192 | "type": "string", 193 | "metadata": { 194 | "description": "Subdomain chosen for application routing" 195 | } 196 | }, 197 | "redHatTags": { 198 | "type": "object", 199 | "metadata": { 200 | "description": "Red Hat Tags" 201 | } 202 | }, 203 | "virtualNetworkName": { 204 | "type": "string", 205 | "metadata": { 206 | "description": "Name of the Virtual Network" 207 | } 208 | }, 209 | "nodeNetworkSecurityGroup": { 210 | "type": "string", 211 | "metadata": { 212 | "description": "Name of Node Network Security Group" 213 | } 214 | }, 215 | "nodeAvailabilitySet": { 216 | "type": "string", 217 | "metadata": { 218 | "description": "Name of Node Availability Set" 219 | } 220 | }, 221 | "masterClusterType": { 222 | "type": "string", 223 | "metadata": { 224 | "description": "Cluster Type" 225 | } 226 | }, 227 | "masterPrivateClusterIp": { 228 | "type": "string", 229 | "metadata": { 230 | "description": "Private IP Address" 231 | } 232 | }, 233 | "masterClusterDns": { 234 | "type": "string", 235 | "metadata": { 236 | "description": "Master Cluster DNS name" 237 | } 238 | }, 239 | "masterPublicIpName": { 240 | "type": "string", 241 | "metadata": { 242 | "description": "Master Public IP Name" 243 | } 244 | }, 245 | "routerClusterType": { 246 | "type": "string", 247 | "metadata": { 248 | "description": "Cluster Type" 249 | } 250 | }, 251 | "routerPublicIpName": { 252 | "type": "string", 253 | "metadata": { 254 | "description": "Infra Public IP Name" 255 | } 256 | }, 257 | "routingCertType": { 258 | "type": "string", 259 | "metadata": { 260 | "description": "Use custom certificate for routing domain or self-signed certificate" 261 | } 262 | }, 263 | "masterCertType": { 264 | "type": "string", 265 | "metadata": { 266 | "description": "Use custom certificate for master domain or self-signed certificate" 267 | } 268 | }, 269 | "minorVersion": { 270 | "type": "string", 271 | "metadata": { 272 | "description": "Minor version of OpenShift 3.11 to deploy" 273 | } 274 | } 275 | }, 276 | "variables": { 277 | "singlequote": "'" 278 | }, 279 | "resources": [{ 280 | "type": "Microsoft.Compute/virtualMachines/extensions", 281 | "name": "[concat(parameters('openshiftBastionHostname'), '/deployOpenShift')]", 282 | "location": "[parameters('location')]", 283 | "apiVersion": "[parameters('apiVersionCompute')]", 284 | "tags": { 285 | "displayName": "DeployOpenShift", 286 | "provider": "[parameters('redHatTags').provider]", 287 | "app": "[parameters('redHatTags').app]", 288 | "version": "[parameters('redHatTags').version]", 289 | "platform": "[parameters('redHatTags').platform]" 290 | }, 291 | "properties": { 292 | "publisher": "Microsoft.Azure.Extensions", 293 | "type": "CustomScript", 294 | "typeHandlerVersion": "2.0", 295 | "autoUpgradeMinorVersion": true, 296 | "settings": { 297 | "fileUris": [ 298 | "[parameters('openshiftDeploymentScriptUrl')]" 299 | ] 300 | }, 301 | "protectedSettings": { 302 | "commandToExecute": "[concat('bash ', parameters('openshiftDeploymentScriptFileName'), ' \"', parameters('adminUsername'), '\" ', variables('singlequote'), parameters('openshiftPassword'), variables('singlequote'), ' \"', parameters('openshiftMasterHostname'), '\" \"', parameters('openshiftMasterPublicIpFqdn'), '\" \"', parameters('openshiftMasterPublicIpAddress'), '\" \"', parameters('openshiftInfraHostname'), '\" \"', parameters('openshiftNodeHostname'), '\" \"', parameters('nodeInstanceCount'), '\" \"', parameters('infraInstanceCount'), '\" \"', parameters('MasterInstanceCount'), '\" \"', parameters(parameters('subDomainChosen')), '\" \"', parameters('newStorageAccountRegistry'), '\" ', variables('singlequote'), parameters('newStorageAccountKey'), variables('singlequote'), ' \"', parameters('enableMetrics'), '\" \"', parameters('enableLogging'), '\" ', variables('singlequote'), parameters('tenantId'), variables('singlequote'), ' ', variables('singlequote'), parameters('subscriptionId'), variables('singlequote'), ' ', variables('singlequote'), parameters('aadClientId'), variables('singlequote'), ' ', variables('singlequote'), parameters('aadClientSecret'), variables('singlequote'), ' ', variables('singlequote'), parameters('resourceGroupName'), variables('singlequote'), ' ', variables('singlequote'), parameters('location'), variables('singlequote'), ' ', variables('singlequote'), parameters('enableAzure'), variables('singlequote'), ' ', variables('singlequote'), parameters('storageKind'), variables('singlequote'), ' ', variables('singlequote'), parameters('enableCNS'), variables('singlequote'), ' ', variables('singlequote'), parameters('openshiftCnsHostname'), variables('singlequote'),' ', variables('singlequote'), parameters('cnsInstanceCount'), variables('singlequote'), ' ', variables('singlequote'), parameters('virtualNetworkName'), variables('singlequote'), ' ', variables('singlequote'), parameters('nodeNetworkSecurityGroup'), variables('singlequote'), ' ', variables('singlequote'), parameters('nodeAvailabilitySet'), variables('singlequote'), ' ', variables('singlequote'), parameters('masterClusterType'), variables('singlequote'), ' ', variables('singlequote'), parameters('masterPrivateClusterIp'), variables('singlequote'), ' ', variables('singlequote'), parameters('masterClusterDns'), variables('singlequote'), ' ', variables('singlequote'), parameters('masterPublicIpName'), variables('singlequote'), ' ', variables('singlequote'), parameters('routerClusterType'), variables('singlequote'), ' ', variables('singlequote'), parameters('routerPublicIpName'), variables('singlequote'), ' ', variables('singlequote'), parameters('routingCertType'), variables('singlequote'), ' ', variables('singlequote'), parameters('masterCertType'), variables('singlequote'), ' ', variables('singlequote'), parameters('minorVersion'), variables('singlequote'))]" 303 | } 304 | } 305 | }], 306 | "outputs": {} 307 | } 308 | -------------------------------------------------------------------------------- /scripts/bastionPrep.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | echo $(date) " - Starting Bastion Prep Script" 3 | 4 | export USERNAME_ORG=$1 5 | export PASSWORD_ACT_KEY="$2" 6 | export POOL_ID=$3 7 | export PRIVATEKEY=$4 8 | export SUDOUSER=$5 9 | export CUSTOMROUTINGCERTTYPE=$6 10 | export CUSTOMMASTERCERTTYPE=$7 11 | export CUSTOMROUTINGCAFILE="$8" 12 | export CUSTOMROUTINGCERTFILE="$9" 13 | export CUSTOMROUTINGKEYFILE="${10}" 14 | export CUSTOMMASTERCAFILE="${11}" 15 | export CUSTOMMASTERCERTFILE="${12}" 16 | export CUSTOMMASTERKEYFILE="${13}" 17 | export CUSTOMDOMAIN="${14}" 18 | export MINORVERSION=${15} 19 | export CUSTOMMASTERTYPE=${16} 20 | export CUSTOMROUTINGTYPE=${17} 21 | 22 | # Generate private keys for use by Ansible 23 | echo $(date) " - Generating Private keys for use by Ansible for OpenShift Installation" 24 | 25 | runuser -l $SUDOUSER -c "echo \"$PRIVATEKEY\" > ~/.ssh/id_rsa" 26 | runuser -l $SUDOUSER -c "chmod 600 ~/.ssh/id_rsa*" 27 | 28 | # Remove RHUI 29 | 30 | rm -f /etc/yum.repos.d/rh-cloud.repo 31 | sleep 10 32 | 33 | # Register Host with Cloud Access Subscription 34 | echo $(date) " - Register host with Cloud Access Subscription" 35 | 36 | subscription-manager register --force --username="$USERNAME_ORG" --password="$PASSWORD_ACT_KEY" || subscription-manager register --force --activationkey="$PASSWORD_ACT_KEY" --org="$USERNAME_ORG" 37 | RETCODE=$? 38 | 39 | if [ $RETCODE -eq 0 ] 40 | then 41 | echo "Subscribed successfully" 42 | elif [ $RETCODE -eq 64 ] 43 | then 44 | echo "This system is already registered." 45 | else 46 | echo "Incorrect Username / Password or Organization ID / Activation Key specified" 47 | exit 3 48 | fi 49 | 50 | subscription-manager attach --pool=$POOL_ID > attach.log 51 | if [ $? -eq 0 ] 52 | then 53 | echo "Pool attached successfully" 54 | else 55 | grep attached attach.log 56 | if [ $? -eq 0 ] 57 | then 58 | echo "Pool $POOL_ID was already attached and was not attached again." 59 | else 60 | echo "Incorrect Pool ID or no entitlements available" 61 | exit 4 62 | fi 63 | fi 64 | 65 | # Disable all repositories and enable only the required ones 66 | echo $(date) " - Disabling all repositories and enabling only the required repos" 67 | 68 | subscription-manager repos --disable="*" 69 | 70 | subscription-manager repos \ 71 | --enable="rhel-7-server-rpms" \ 72 | --enable="rhel-7-server-extras-rpms" \ 73 | --enable="rhel-7-server-ose-3.11-rpms" \ 74 | --enable="rhel-7-server-ansible-2.6-rpms" \ 75 | --enable="rhel-7-fast-datapath-rpms" \ 76 | --enable="rh-gluster-3-client-for-rhel-7-server-rpms" \ 77 | --enable="rhel-7-server-optional-rpms" 78 | 79 | # Update system to latest packages 80 | echo $(date) " - Update system to latest packages" 81 | yum -y update --exclude=WALinuxAgent 82 | echo $(date) " - System update complete" 83 | 84 | # Install base packages and update system to latest packages 85 | echo $(date) " - Install base packages" 86 | yum -y install wget git net-tools bind-utils iptables-services bridge-utils bash-completion httpd-tools kexec-tools sos psacct ansible 87 | yum -y update glusterfs-fuse 88 | echo $(date) " - Base package installation complete" 89 | 90 | # Install OpenShift utilities 91 | echo $(date) " - Installing OpenShift utilities" 92 | yum -y install openshift-ansible-3.11.${MINORVERSION} 93 | echo $(date) " - OpenShift utilities installation complete" 94 | 95 | # Installing Azure CLI 96 | # From https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-yum 97 | echo $(date) " - Installing Azure CLI" 98 | sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc 99 | sudo sh -c 'echo -e "[azure-cli]\nname=Azure CLI\nbaseurl=https://packages.microsoft.com/yumrepos/azure-cli\nenabled=1\ngpgcheck=1\ngpgkey=https://packages.microsoft.com/keys/microsoft.asc" > /etc/yum.repos.d/azure-cli.repo' 100 | sudo yum install -y azure-cli 101 | echo $(date) " - Azure CLI installation complete" 102 | 103 | # Install ImageMagick to resize image for Custom Header 104 | sudo yum install -y ImageMagick 105 | 106 | # Configure DNS so it always has the domain name 107 | echo $(date) " - Adding DOMAIN to search for resolv.conf" 108 | if [[ $CUSTOMDOMAIN == "none" ]] 109 | then 110 | DOMAINNAME=`domainname -d` 111 | else 112 | DOMAINNAME=$CUSTOMDOMAIN 113 | fi 114 | 115 | echo "DOMAIN=$DOMAINNAME" >> /etc/sysconfig/network-scripts/ifcfg-eth0 116 | 117 | echo $(date) " - Restarting NetworkManager" 118 | runuser -l $SUDOUSER -c "ansible localhost -o -b -m service -a \"name=NetworkManager state=restarted\"" 119 | echo $(date) " - NetworkManager configuration complete" 120 | 121 | # Run Ansible Playbook to update ansible.cfg file 122 | echo $(date) " - Updating ansible.cfg file" 123 | wget --retry-connrefused --waitretry=1 --read-timeout=20 --timeout=15 -t 5 https://raw.githubusercontent.com/microsoft/openshift-container-platform-playbooks/master/updateansiblecfg.yaml 124 | ansible-playbook -f 10 ./updateansiblecfg.yaml 125 | 126 | # Create certificate files 127 | 128 | if [[ $CUSTOMMASTERCERTTYPE == "custom" ]] 129 | then 130 | echo $(date) " - Creating custom master certificate files" 131 | runuser -l $SUDOUSER -c "echo \"$CUSTOMMASTERCAFILE\" > /tmp/masterca.pem" 132 | runuser -l $SUDOUSER -c "echo \"$CUSTOMMASTERCERTFILE\" > /tmp/mastercert.pem" 133 | runuser -l $SUDOUSER -c "echo \"$CUSTOMMASTERKEYFILE\" > /tmp/masterkey.pem" 134 | echo $(date) " - Custom master certificate files masterca.pem, mastercert.pem, masterkey.pem created in /tmp" 135 | fi 136 | 137 | if [ $CUSTOMROUTINGCERTTYPE == "custom" ] 138 | then 139 | echo $(date) " - Creating custom routing certificate files" 140 | runuser -l $SUDOUSER -c "echo \"$CUSTOMROUTINGCAFILE\" > /tmp/routingca.pem" 141 | runuser -l $SUDOUSER -c "echo \"$CUSTOMROUTINGCERTFILE\" > /tmp/routingcert.pem" 142 | runuser -l $SUDOUSER -c "echo \"$CUSTOMROUTINGKEYFILE\" > /tmp/routingkey.pem" 143 | echo $(date) " - Custom routing certificate files routingca.pem, routingcert.pem, routingkey.pem created in /tmp" 144 | fi 145 | 146 | echo $(date) " - Script Complete" 147 | 148 | -------------------------------------------------------------------------------- /scripts/deployOpenShift.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo $(date) " - Starting Script" 4 | 5 | set -e 6 | 7 | export SUDOUSER=$1 8 | export PASSWORD="$2" 9 | export MASTER=$3 10 | export MASTERPUBLICIPHOSTNAME=$4 11 | export MASTERPUBLICIPADDRESS=$5 12 | export INFRA=$6 13 | export NODE=$7 14 | export NODECOUNT=$8 15 | export INFRACOUNT=$9 16 | export MASTERCOUNT=${10} 17 | export ROUTING=${11} 18 | export REGISTRYSA=${12} 19 | export ACCOUNTKEY="${13}" 20 | export METRICS=${14} 21 | export LOGGING=${15} 22 | export TENANTID=${16} 23 | export SUBSCRIPTIONID=${17} 24 | export AADCLIENTID=${18} 25 | export AADCLIENTSECRET="${19}" 26 | export RESOURCEGROUP=${20} 27 | export LOCATION=${21} 28 | export AZURE=${22} 29 | export STORAGEKIND=${23} 30 | export ENABLECNS=${24} 31 | export CNS=${25} 32 | export CNSCOUNT=${26} 33 | export VNETNAME=${27} 34 | export NODENSG=${28} 35 | export NODEAVAILIBILITYSET=${29} 36 | export MASTERCLUSTERTYPE=${30} 37 | export PRIVATEIP=${31} 38 | export PRIVATEDNS=${32} 39 | export MASTERPIPNAME=${33} 40 | export ROUTERCLUSTERTYPE=${34} 41 | export INFRAPIPNAME=${35} 42 | export CUSTOMROUTINGCERTTYPE=${36} 43 | export CUSTOMMASTERCERTTYPE=${37} 44 | export MINORVERSION=${38} 45 | export BASTION=$(hostname) 46 | 47 | # Set CNS to default storage type. Will be overridden later if Azure is true 48 | export CNS_DEFAULT_STORAGE=true 49 | 50 | # Setting DOMAIN variable 51 | export DOMAIN=`domainname -d` 52 | 53 | # Determine if Commercial Azure or Azure Government 54 | CLOUD=$( curl -H Metadata:true "http://169.254.169.254/metadata/instance/compute/location?api-version=2017-04-02&format=text" | cut -c 1-2 ) 55 | export CLOUD=${CLOUD^^} 56 | 57 | export MASTERLOOP=$((MASTERCOUNT - 1)) 58 | export INFRALOOP=$((INFRACOUNT - 1)) 59 | export NODELOOP=$((NODECOUNT - 1)) 60 | 61 | echo $(date) " - Configuring SSH ControlPath to use shorter path name" 62 | 63 | sed -i -e "s/^# control_path = %(directory)s\/%%h-%%r/control_path = %(directory)s\/%%h-%%r/" /etc/ansible/ansible.cfg 64 | sed -i -e "s/^#host_key_checking = False/host_key_checking = False/" /etc/ansible/ansible.cfg 65 | sed -i -e "s/^#pty=False/pty=False/" /etc/ansible/ansible.cfg 66 | sed -i -e "s/^#stdout_callback = skippy/stdout_callback = skippy/" /etc/ansible/ansible.cfg 67 | sed -i -e "s/^#pipelining = False/pipelining = True/" /etc/ansible/ansible.cfg 68 | 69 | # echo $(date) " - Modifying sudoers" 70 | sed -i -e "s/Defaults requiretty/# Defaults requiretty/" /etc/sudoers 71 | sed -i -e '/Defaults env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"/aDefaults env_keep += "PATH"' /etc/sudoers 72 | 73 | # Create docker registry config based on Commercial Azure or Azure Government 74 | if [[ $CLOUD == "US" ]] 75 | then 76 | export DOCKERREGISTRYREALM=core.usgovcloudapi.net 77 | export CLOUDNAME="AzureUSGovernmentCloud" 78 | else 79 | export DOCKERREGISTRYREALM=core.windows.net 80 | export CLOUDNAME="AzurePublicCloud" 81 | fi 82 | 83 | # Setting the default openshift_cloudprovider_kind if Azure enabled 84 | if [[ $AZURE == "true" ]] 85 | then 86 | CLOUDKIND="openshift_cloudprovider_kind=azure 87 | openshift_cloudprovider_azure_client_id=$AADCLIENTID 88 | openshift_cloudprovider_azure_client_secret=$AADCLIENTSECRET 89 | openshift_cloudprovider_azure_tenant_id=$TENANTID 90 | openshift_cloudprovider_azure_subscription_id=$SUBSCRIPTIONID 91 | openshift_cloudprovider_azure_cloud=$CLOUDNAME 92 | openshift_cloudprovider_azure_vnet_name=$VNETNAME 93 | openshift_cloudprovider_azure_security_group_name=$NODENSG 94 | openshift_cloudprovider_azure_availability_set_name=$NODEAVAILIBILITYSET 95 | openshift_cloudprovider_azure_resource_group=$RESOURCEGROUP 96 | openshift_cloudprovider_azure_location=$LOCATION" 97 | CNS_DEFAULT_STORAGE=false 98 | if [[ $STORAGEKIND == "managed" ]] 99 | then 100 | SCKIND="openshift_storageclass_parameters={'kind': 'managed', 'storageaccounttype': 'Premium_LRS'}" 101 | else 102 | SCKIND="openshift_storageclass_parameters={'kind': 'shared', 'storageaccounttype': 'Premium_LRS'}" 103 | fi 104 | fi 105 | 106 | # Cloning Ansible playbook repository 107 | 108 | echo $(date) " - Cloning Ansible playbook repository" 109 | 110 | ((cd /home/$SUDOUSER && git clone https://github.com/Microsoft/openshift-container-platform-playbooks.git) || (cd /home/$SUDOUSER/openshift-container-platform-playbooks && git pull)) 111 | 112 | if [ -d /home/${SUDOUSER}/openshift-container-platform-playbooks ] 113 | then 114 | echo " - Retrieved playbooks successfully" 115 | else 116 | echo " - Retrieval of playbooks failed" 117 | exit 7 118 | fi 119 | 120 | # Configure custom routing certificate 121 | echo $(date) " - Create variable for routing certificate based on certificate type" 122 | if [[ $CUSTOMROUTINGCERTTYPE == "custom" ]] 123 | then 124 | ROUTINGCERTIFICATE="openshift_hosted_router_certificate={\"cafile\": \"/tmp/routingca.pem\", \"certfile\": \"/tmp/routingcert.pem\", \"keyfile\": \"/tmp/routingkey.pem\"}" 125 | else 126 | ROUTINGCERTIFICATE="" 127 | fi 128 | 129 | # Configure custom master API certificate 130 | echo $(date) " - Create variable for master api certificate based on certificate type" 131 | if [[ $CUSTOMMASTERCERTTYPE == "custom" ]] 132 | then 133 | MASTERCERTIFICATE="openshift_master_overwrite_named_certificates=true 134 | openshift_master_named_certificates=[{\"names\": [\"$MASTERPUBLICIPHOSTNAME\"], \"cafile\": \"/tmp/masterca.pem\", \"certfile\": \"/tmp/mastercert.pem\", \"keyfile\": \"/tmp/masterkey.pem\"}]" 135 | else 136 | MASTERCERTIFICATE="" 137 | fi 138 | 139 | # Configure master cluster address information based on Cluster type (private or public) 140 | echo $(date) " - Create variable for master cluster address based on cluster type" 141 | if [[ $MASTERCLUSTERTYPE == "private" ]] 142 | then 143 | MASTERCLUSTERADDRESS="openshift_master_cluster_hostname=$MASTER01 144 | openshift_master_cluster_public_hostname=$PRIVATEDNS 145 | openshift_master_cluster_public_vip=$PRIVATEIP" 146 | else 147 | MASTERCLUSTERADDRESS="openshift_master_cluster_hostname=$MASTERPUBLICIPHOSTNAME 148 | openshift_master_cluster_public_hostname=$MASTERPUBLICIPHOSTNAME 149 | openshift_master_cluster_public_vip=$MASTERPUBLICIPADDRESS" 150 | fi 151 | 152 | # Create Master nodes grouping 153 | echo $(date) " - Creating Master nodes grouping" 154 | MASTERLIST="0$MASTERCOUNT" 155 | for (( c=1; c<=$MASTERCOUNT; c++ )) 156 | do 157 | mastergroup="$mastergroup 158 | ${MASTER}0$c openshift_node_group_name='node-config-master'" 159 | done 160 | 161 | # Create Infra nodes grouping 162 | echo $(date) " - Creating Infra nodes grouping" 163 | for (( c=1; c<=$INFRACOUNT; c++ )) 164 | do 165 | infragroup="$infragroup 166 | ${INFRA}0$c openshift_node_group_name='node-config-infra'" 167 | done 168 | 169 | # Create Nodes grouping 170 | echo $(date) " - Creating Nodes grouping" 171 | if [ $NODECOUNT -gt 9 ] 172 | then 173 | # If more than 10 compute nodes need to create groups 01 - 09 separately than 10 and higher 174 | for (( c=1; c<=9; c++ )) 175 | do 176 | nodegroup="$nodegroup 177 | ${NODE}0$c openshift_node_group_name='node-config-compute'" 178 | done 179 | 180 | for (( c=10; c<=$NODECOUNT; c++ )) 181 | do 182 | nodegroup="$nodegroup 183 | ${NODE}$c openshift_node_group_name='node-config-compute'" 184 | done 185 | else 186 | # If less than 10 compout nodes 187 | for (( c=1; c<=$NODECOUNT; c++ )) 188 | do 189 | nodegroup="$nodegroup 190 | ${NODE}0$c openshift_node_group_name='node-config-compute'" 191 | done 192 | fi 193 | 194 | # Create CNS nodes grouping if CNS is enabled 195 | if [[ $ENABLECNS == "true" ]] 196 | then 197 | echo $(date) " - Creating CNS nodes grouping" 198 | 199 | for (( c=1; c<=$CNSCOUNT; c++ )) 200 | do 201 | cnsgroup="$cnsgroup 202 | ${CNS}0$c openshift_node_group_name='node-config-compute'" 203 | done 204 | fi 205 | 206 | # Setting the HA Mode if more than one master 207 | if [ $MASTERCOUNT != 1 ] 208 | then 209 | echo $(date) " - Enabling HA mode for masters" 210 | export HAMODE="openshift_master_cluster_method=native" 211 | fi 212 | 213 | # Create Temp Ansible Hosts File 214 | echo $(date) " - Create Ansible Hosts file" 215 | 216 | cat > /etc/ansible/hosts <> ~/.ssh/known_hosts" 241 | drive=$(runuser $SUDOUSER -c "ssh ${CNS}0$c 'sudo /usr/sbin/fdisk -l'" | awk '$1 == "Disk" && $2 ~ /^\// && ! /mapper/ {if (drive) print drive; drive = $2; sub(":", "", drive);} drive && /^\// {drive = ""} END {if (drive) print drive;}') 242 | drive1=$(echo $drive | cut -d ' ' -f 1) 243 | drive2=$(echo $drive | cut -d ' ' -f 2) 244 | drive3=$(echo $drive | cut -d ' ' -f 3) 245 | cnsglusterinfo="$cnsglusterinfo 246 | ${CNS}0$c glusterfs_devices='[ \"${drive1}\", \"${drive2}\", \"${drive3}\" ]'" 247 | done 248 | fi 249 | 250 | # Create Ansible Hosts File 251 | echo $(date) " - Create Ansible Hosts file" 252 | 253 | cat > /etc/ansible/hosts < /home/$SUDOUSER/openshift-container-platform-playbooks/vars.yaml < attach.log 43 | if [ $? -eq 0 ] 44 | then 45 | echo "Pool attached successfully" 46 | else 47 | grep attached attach.log 48 | if [ $? -eq 0 ] 49 | then 50 | echo "Pool $POOL_ID was already attached and was not attached again." 51 | else 52 | echo "Incorrect Pool ID or no entitlements available" 53 | exit 4 54 | fi 55 | fi 56 | 57 | # Disable all repositories and enable only the required ones 58 | echo $(date) " - Disabling all repositories and enabling only the required repos" 59 | 60 | subscription-manager repos --disable="*" 61 | 62 | subscription-manager repos \ 63 | --enable="rhel-7-server-rpms" \ 64 | --enable="rhel-7-server-extras-rpms" \ 65 | --enable="rhel-7-server-ose-3.11-rpms" \ 66 | --enable="rhel-7-server-ansible-2.6-rpms" \ 67 | --enable="rhel-7-fast-datapath-rpms" \ 68 | --enable="rh-gluster-3-client-for-rhel-7-server-rpms" 69 | 70 | # Install base packages and update system to latest packages 71 | echo $(date) " - Install base packages and update system to latest packages" 72 | 73 | yum -y install wget git net-tools bind-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct 74 | yum -y install cloud-utils-growpart.noarch 75 | yum -y install ansible 76 | yum -y update glusterfs-fuse 77 | yum -y update --exclude=WALinuxAgent 78 | echo $(date) " - Base package insallation and updates complete" 79 | 80 | # Grow Root File System 81 | echo $(date) " - Grow Root FS" 82 | 83 | rootdev=`findmnt --target / -o SOURCE -n` 84 | rootdrivename=`lsblk -no pkname $rootdev` 85 | rootdrive="/dev/"$rootdrivename 86 | name=`lsblk $rootdev -o NAME | tail -1` 87 | part_number=${name#*${rootdrivename}} 88 | 89 | growpart $rootdrive $part_number -u on 90 | xfs_growfs $rootdev 91 | 92 | # Install Docker 93 | echo $(date) " - Installing Docker" 94 | yum -y install docker 95 | 96 | # Update docker config for insecure registry 97 | echo " 98 | # Adding insecure-registry option required by OpenShift 99 | OPTIONS=\"\$OPTIONS --insecure-registry 172.30.0.0/16\" 100 | " >> /etc/sysconfig/docker 101 | 102 | # Create thin pool logical volume for Docker 103 | echo $(date) " - Creating thin pool logical volume for Docker and starting service" 104 | 105 | DOCKERVG=$( parted -m /dev/sda print all 2>/dev/null | grep unknown | grep /dev/sd | cut -d':' -f1 | head -n1 ) 106 | 107 | echo " 108 | # Adding OpenShift data disk for docker 109 | DEVS=${DOCKERVG} 110 | VG=docker-vg 111 | " >> /etc/sysconfig/docker-storage-setup 112 | 113 | # Running setup for docker storage 114 | docker-storage-setup 115 | if [ $? -eq 0 ] 116 | then 117 | echo "Docker thin pool logical volume created successfully" 118 | else 119 | echo "Error creating logical volume for Docker" 120 | exit 5 121 | fi 122 | 123 | # Enable and start Docker services 124 | 125 | systemctl enable docker 126 | systemctl start docker 127 | 128 | echo $(date) " - Script Complete" 129 | 130 | -------------------------------------------------------------------------------- /scripts/masterPrep.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | echo $(date) " - Starting Master Prep Script" 3 | 4 | export USERNAME_ORG=$1 5 | export PASSWORD_ACT_KEY="$2" 6 | export POOL_ID=$3 7 | 8 | # Remove RHUI 9 | 10 | rm -f /etc/yum.repos.d/rh-cloud.repo 11 | sleep 10 12 | 13 | # Register Host with Cloud Access Subscription 14 | echo $(date) " - Register host with Cloud Access Subscription" 15 | 16 | subscription-manager register --force --username="$USERNAME_ORG" --password="$PASSWORD_ACT_KEY" || subscription-manager register --force --activationkey="$PASSWORD_ACT_KEY" --org="$USERNAME_ORG" 17 | RETCODE=$? 18 | 19 | if [ $RETCODE -eq 0 ] 20 | then 21 | echo "Subscribed successfully" 22 | elif [ $RETCODE -eq 64 ] 23 | then 24 | echo "This system is already registered." 25 | else 26 | sleep 5 27 | subscription-manager register --force --username="$USERNAME_ORG" --password="$PASSWORD_ACT_KEY" || subscription-manager register --force --activationkey="$PASSWORD_ACT_KEY" --org="$USERNAME_ORG" 28 | RETCODE2=$? 29 | if [ $RETCODE2 -eq 0 ] 30 | then 31 | echo "Subscribed successfully" 32 | elif [ $RETCODE2 -eq 64 ] 33 | then 34 | echo "This system is already registered." 35 | else 36 | echo "Incorrect Username / Password or Organization ID / Activation Key specified. Unregistering system from RHSM" 37 | subscription-manager unregister 38 | exit 3 39 | fi 40 | fi 41 | 42 | subscription-manager attach --pool=$POOL_ID > attach.log 43 | if [ $? -eq 0 ] 44 | then 45 | echo "Pool attached successfully" 46 | else 47 | grep attached attach.log 48 | if [ $? -eq 0 ] 49 | then 50 | echo "Pool $POOL_ID was already attached and was not attached again." 51 | else 52 | echo "Incorrect Pool ID or no entitlements available" 53 | exit 4 54 | fi 55 | fi 56 | 57 | # Disable all repositories and enable only the required ones 58 | echo $(date) " - Disabling all repositories and enabling only the required repos" 59 | 60 | subscription-manager repos --disable="*" 61 | 62 | subscription-manager repos \ 63 | --enable="rhel-7-server-rpms" \ 64 | --enable="rhel-7-server-extras-rpms" \ 65 | --enable="rhel-7-server-ose-3.11-rpms" \ 66 | --enable="rhel-7-server-ansible-2.6-rpms" \ 67 | --enable="rhel-7-fast-datapath-rpms" \ 68 | --enable="rh-gluster-3-client-for-rhel-7-server-rpms" \ 69 | --enable="rhel-7-server-optional-rpms" 70 | 71 | # Install base packages and update system to latest packages 72 | echo $(date) " - Install base packages and update system to latest packages" 73 | 74 | yum -y install wget git net-tools bind-utils iptables-services bridge-utils bash-completion httpd-tools kexec-tools sos psacct ansible 75 | yum -y install cloud-utils-growpart.noarch 76 | yum -y update glusterfs-fuse 77 | yum -y update --exclude=WALinuxAgent 78 | echo $(date) " - Base package insallation and updates complete" 79 | 80 | # Grow Root File System 81 | echo $(date) " - Grow Root FS" 82 | 83 | rootdev=`findmnt --target / -o SOURCE -n` 84 | rootdrivename=`lsblk -no pkname $rootdev` 85 | rootdrive="/dev/"$rootdrivename 86 | name=`lsblk $rootdev -o NAME | tail -1` 87 | part_number=${name#*${rootdrivename}} 88 | 89 | growpart $rootdrive $part_number -u on 90 | xfs_growfs $rootdev 91 | 92 | if [ $? -eq 0 ] 93 | then 94 | echo "Root partition expanded" 95 | else 96 | echo "Root partition failed to expand" 97 | exit 6 98 | fi 99 | 100 | # Install Docker 101 | echo $(date) " - Installing Docker" 102 | yum -y install docker 103 | 104 | # Update docker config for insecure registry 105 | echo " 106 | # Adding insecure-registry option required by OpenShift 107 | OPTIONS=\"\$OPTIONS --insecure-registry 172.30.0.0/16\" 108 | " >> /etc/sysconfig/docker 109 | 110 | # Create thin pool logical volume for Docker 111 | echo $(date) " - Creating thin pool logical volume for Docker and staring service" 112 | 113 | DOCKERVG=$( parted -m /dev/sda print all 2>/dev/null | grep unknown | grep /dev/sd | cut -d':' -f1 ) 114 | 115 | echo " 116 | # Adding OpenShift data disk for docker 117 | DEVS=${DOCKERVG} 118 | VG=docker-vg 119 | " >> /etc/sysconfig/docker-storage-setup 120 | 121 | # Running setup for docker storage 122 | docker-storage-setup 123 | if [ $? -eq 0 ] 124 | then 125 | echo "Docker thin pool logical volume created successfully" 126 | else 127 | echo "Error creating logical volume for Docker" 128 | exit 5 129 | fi 130 | 131 | # Enable and start Docker services 132 | 133 | systemctl enable docker 134 | systemctl start docker 135 | 136 | echo $(date) " - Script Complete" -------------------------------------------------------------------------------- /scripts/nodePrep.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | echo $(date) " - Starting Infra / Node Prep Script" 3 | 4 | export USERNAME_ORG=$1 5 | export PASSWORD_ACT_KEY="$2" 6 | export POOL_ID=$3 7 | 8 | # Remove RHUI 9 | 10 | rm -f /etc/yum.repos.d/rh-cloud.repo 11 | sleep 10 12 | 13 | # Register Host with Cloud Access Subscription 14 | echo $(date) " - Register host with Cloud Access Subscription" 15 | 16 | subscription-manager register --force --username="$USERNAME_ORG" --password="$PASSWORD_ACT_KEY" || subscription-manager register --force --activationkey="$PASSWORD_ACT_KEY" --org="$USERNAME_ORG" 17 | RETCODE=$? 18 | 19 | if [ $RETCODE -eq 0 ] 20 | then 21 | echo "Subscribed successfully" 22 | elif [ $RETCODE -eq 64 ] 23 | then 24 | echo "This system is already registered." 25 | else 26 | sleep 5 27 | subscription-manager register --force --username="$USERNAME_ORG" --password="$PASSWORD_ACT_KEY" || subscription-manager register --force --activationkey="$PASSWORD_ACT_KEY" --org="$USERNAME_ORG" 28 | RETCODE2=$? 29 | if [ $RETCODE2 -eq 0 ] 30 | then 31 | echo "Subscribed successfully" 32 | elif [ $RETCODE2 -eq 64 ] 33 | then 34 | echo "This system is already registered." 35 | else 36 | echo "Incorrect Username / Password or Organization ID / Activation Key specified. Unregistering system from RHSM" 37 | subscription-manager unregister 38 | exit 3 39 | fi 40 | fi 41 | 42 | subscription-manager attach --pool=$POOL_ID > attach.log 43 | if [ $? -eq 0 ] 44 | then 45 | echo "Pool attached successfully" 46 | else 47 | grep attached attach.log 48 | if [ $? -eq 0 ] 49 | then 50 | echo "Pool $POOL_ID was already attached and was not attached again." 51 | else 52 | echo "Incorrect Pool ID or no entitlements available" 53 | exit 4 54 | fi 55 | fi 56 | 57 | # Disable all repositories and enable only the required ones 58 | echo $(date) " - Disabling all repositories and enabling only the required repos" 59 | 60 | subscription-manager repos --disable="*" 61 | 62 | subscription-manager repos \ 63 | --enable="rhel-7-server-rpms" \ 64 | --enable="rhel-7-server-extras-rpms" \ 65 | --enable="rhel-7-server-ose-3.11-rpms" \ 66 | --enable="rhel-7-server-ansible-2.6-rpms" \ 67 | --enable="rhel-7-fast-datapath-rpms" \ 68 | --enable="rh-gluster-3-client-for-rhel-7-server-rpms" \ 69 | --enable="rhel-7-server-optional-rpms" 70 | 71 | # Install base packages and update system to latest packages 72 | echo $(date) " - Install base packages and update system to latest packages" 73 | 74 | yum -y install wget git net-tools bind-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct ansible 75 | yum -y install cloud-utils-growpart.noarch 76 | yum -y update glusterfs-fuse 77 | yum -y update --exclude=WALinuxAgent 78 | echo $(date) " - Base package insallation and updates complete" 79 | 80 | # Grow Root File System 81 | echo $(date) " - Grow Root FS" 82 | 83 | rootdev=`findmnt --target / -o SOURCE -n` 84 | rootdrivename=`lsblk -no pkname $rootdev` 85 | rootdrive="/dev/"$rootdrivename 86 | name=`lsblk $rootdev -o NAME | tail -1` 87 | part_number=${name#*${rootdrivename}} 88 | 89 | growpart $rootdrive $part_number -u on 90 | xfs_growfs $rootdev 91 | 92 | if [ $? -eq 0 ] 93 | then 94 | echo "Root partition expanded" 95 | else 96 | echo "Root partition failed to expand" 97 | exit 6 98 | fi 99 | 100 | # Install Docker 101 | echo $(date) " - Installing Docker" 102 | yum -y install docker 103 | 104 | # Update docker config for insecure registry 105 | echo " 106 | # Adding insecure-registry option required by OpenShift 107 | OPTIONS=\"\$OPTIONS --insecure-registry 172.30.0.0/16\" 108 | " >> /etc/sysconfig/docker 109 | 110 | # Create thin pool logical volume for Docker 111 | echo $(date) " - Creating thin pool logical volume for Docker and staring service" 112 | 113 | DOCKERVG=$( parted -m /dev/sda print all 2>/dev/null | grep unknown | grep /dev/sd | cut -d':' -f1 | head -n1 ) 114 | 115 | echo " 116 | # Adding OpenShift data disk for docker 117 | DEVS=${DOCKERVG} 118 | VG=docker-vg 119 | " >> /etc/sysconfig/docker-storage-setup 120 | 121 | # Running setup for docker storage 122 | docker-storage-setup 123 | if [ $? -eq 0 ] 124 | then 125 | echo "Docker thin pool logical volume created successfully" 126 | else 127 | echo "Error creating logical volume for Docker" 128 | exit 5 129 | fi 130 | 131 | # Enable and start Docker services 132 | 133 | systemctl enable docker 134 | systemctl start docker 135 | 136 | echo $(date) " - Script Complete" 137 | 138 | --------------------------------------------------------------------------------