├── .github └── ISSUE_TEMPLATE │ ├── bug_report.md │ └── feature_request.md ├── .gitignore ├── CODE_OF_CONDUCT.md ├── LICENSE ├── README.md ├── _config.yml ├── main.tf ├── modules ├── base_node │ ├── main.tf │ ├── outputs.tf │ └── variables.tf ├── base_sec_group │ ├── main.tf │ ├── outputs.tf │ └── variables.tf ├── control_center │ ├── README.md │ ├── main.tf │ ├── outputs.tf │ └── variables.tf ├── kafka_broker │ ├── README.md │ ├── main.tf │ ├── outputs.tf │ └── variables.tf ├── kafka_connect │ ├── README.md │ ├── main.tf │ ├── output.tf │ └── variables.tf ├── ksql │ ├── README.md │ ├── main.tf │ ├── outputs.tf │ └── variables.tf ├── rest_proxy │ ├── README.md │ ├── main.tf │ ├── outputs.tf │ └── variables.tf ├── schema_registry │ ├── README.md │ ├── main.tf │ ├── outputs.tf │ └── variables.tf └── zookeeper │ ├── README.md │ ├── main.tf │ ├── outputs.tf │ └── variables.tf ├── outputs.tf └── variables.tf /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **Desktop (please complete the following information):** 27 | - OS: [e.g. iOS] 28 | - Browser [e.g. chrome, safari] 29 | - Version [e.g. 22] 30 | 31 | **Smartphone (please complete the following information):** 32 | - Device: [e.g. iPhone6] 33 | - OS: [e.g. iOS8.1] 34 | - Browser [e.g. stock browser, safari] 35 | - Version [e.g. 22] 36 | 37 | **Additional context** 38 | Add any other context about the problem here. 39 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .project 2 | .terraform.lock.hcl 3 | .terraform/ -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, sex characteristics, gender identity and expression, 9 | level of experience, education, socio-economic status, nationality, personal 10 | appearance, race, religion, or sexual identity and orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies both within project spaces and in public spaces 49 | when an individual is representing the project or its community. Examples of 50 | representing a project or community include using an official project e-mail 51 | address, posting via an official social media account, or acting as an appointed 52 | representative at an online or offline event. Representation of a project may be 53 | further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at nerdynick@gmail.com. All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ## Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 71 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 72 | 73 | [homepage]: https://www.contributor-covenant.org 74 | 75 | For answers to common questions about this code of conduct, see 76 | https://www.contributor-covenant.org/faq 77 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # terraform-aws-confluent-platform 2 | 3 | Terraform Module(s) for Deploying the Confluent Platform within AWS. 4 | 5 | # Features 6 | 7 | | | Zookeeper | Kafka Broker | Kafka Connect | ksqlDB | Rest Proxy | Schema Registry | Control Center | 8 | |:------------------- |:---------:|:------------:|:-------------:|:------:|:----------:|:---------------:|:--------------:| 9 | | EC2 Instance | X | X | X | X | X | X | X | 10 | | EBS Volumes | X | X | X | X | X | X | X | 11 | | Route53 DNS | X | X | X | X | X | X | X | 12 | | Security Groups | X | X | X | X | X | X | X | 13 | | Load Balancers | N/A | | | | | | N/A | 14 | | Multi AZ | X | X | X | X | X | X | X | 15 | | Auto Scaling Groups | N/A | N/A | | | | N/A | N/A | 16 | | Multi Cluster | N/A | N/A | | | N/A | N/A | N/A | 17 | | Monitoring | X | X | X | X | X | X | X | 18 | | EC2 User Data | X | X | X | X | X | X | X | 19 | 20 | ## Feature Limitations 21 | 22 | 1. Out of the box all nodes and security groups are required to live in the same VPC 23 | 24 | **NOTE:** If you leverage the individual component modules, some of these limitations can be worked around. 25 | These limitation just haven't be able to be baked into a single unified parent module yet, or may not be possible to at all. 26 | 27 | ## Monitoring 28 | 29 | Currently Prometheus and Jolokia are the only supported monitoring platforms. 30 | The contained features around them will Setup Security Groups and EC2 instance tags appropriate to each component. 31 | The EC2 tags are designed to allow you to use Prometheus' EC2 Service Discovery feature, 32 | [as discussed here](https://medium.com/investing-in-tech/automatic-monitoring-for-all-new-aws-instances-using-prometheus-service-discovery-97d37a5b2ea2), 33 | to find each component and automaticly start reading from that component. 34 | 35 | # Pre-Defined Template Varaibles 36 | 37 | | Variable | Availability | Desc | 38 | |:---------------------|:---------------|:-----| 39 | | node_count | ALL | Node index within the component's collection of nodes | 40 | | component_name | ALL | Full name of component. Ex: `Control Center`, `Zookeeper`, `Kafka Broker` | 41 | | component_short_name | ALL | Short name of component. Ex: `c3`, `zk`, `kfk`, `sr` | 42 | | node_name | DNS, User_Data | The rendered node name template for the given instance | 43 | | node_dns | User_Data | The rendered node DNS template for the given instance | 44 | | node_devices | User_Data | Comma separated list of devices/volumes that will be attached to this instance | 45 | 46 | # How To 47 | 48 | ## Provide User Data to EC2 Instances 49 | 50 | Each component has support to render a template and use that as the `user_data` that gets provided to created instances. 51 | These templates will be given a default set of varaibles, as outlined above, as well as any variables you add to the `extra_template_vars`, `*_extra_template_vars`, `user_data_template_vars`, `*_user_data_template_vars` variables. 52 | Of note, the `extra_template_vars` and `user_data_template_vars` are treated as global collections, and will be provided to all components/sub_modules. 53 | 54 | ## Defining Separate Providers 55 | 56 | As of 2.4.3 you can now define separate providers for your EC2 instance creation and your DNS creation. 57 | This is to support those users that may need to use separate IAM accounts to for DNS then they do to create EC2, EBS, and SecGroups. 58 | 59 | These provider alias are as follows: 60 | 61 | ``` 62 | provider "aws" { 63 | alias = "default" 64 | } 65 | provider "aws" { 66 | alias = "dns" 67 | } 68 | ``` 69 | 70 | Example of how to pass these references: 71 | 72 | ``` 73 | module "shared-cp-aws" { 74 | source = "nerdynick/confluent-platform/aws" 75 | version = "2.4.4" 76 | 77 | providers = { 78 | aws.default = aws.default 79 | aws.dns = aws.dns 80 | } 81 | 82 | .... 83 | } 84 | ``` 85 | 86 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-cayman -------------------------------------------------------------------------------- /main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | aws = { 4 | source = "hashicorp/aws" 5 | version = "~> 4.5.0" 6 | configuration_aliases = [aws.dns] 7 | } 8 | template = { 9 | source = "hashicorp/template" 10 | version = "~> 2.2.0" 11 | } 12 | } 13 | } 14 | 15 | module "cp-aws-zookeeper" { 16 | source = "./modules/zookeeper" 17 | 18 | providers = { 19 | aws = aws 20 | aws.dns = aws.dns 21 | } 22 | 23 | servers = var.zookeeper_servers 24 | image_id = var.zookeeper_image_id == "" ? var.image_id : var.zookeeper_image_id 25 | instance_type = var.zookeeper_instance_type 26 | root_volume_size = var.zookeeper_root_volume_size 27 | key_pair = var.zookeeper_key_pair == "" ? var.key_pair : var.zookeeper_key_pair 28 | tags = merge(var.default_tags, var.zookeeper_tags) 29 | subnet_ids = length(var.zookeeper_subnet_ids) > 0 ? var.zookeeper_subnet_ids : var.subnet_ids 30 | multi_az = var.multi_az && var.zookeeper_multi_az 31 | security_group_ids = length(var.zookeeper_security_group_ids) > 0 ? var.zookeeper_security_group_ids : var.security_group_ids 32 | dns_zone_id = var.zookeeper_dns_zone_id == "" ? var.dns_zone_id : var.zookeeper_dns_zone_id 33 | dns_ttl = var.zookeeper_dns_ttl 34 | name_template = var.zookeeper_name_template 35 | dns_template = var.zookeeper_dns_template 36 | sg_name = var.zookeeper_sg_name 37 | kafka_broker_sg_id = module.cp-aws-kafka_broker.security_group 38 | 39 | extra_template_vars = merge(var.zookeeper_extra_template_vars, var.extra_template_vars) 40 | user_data_template_vars = merge(var.zookeeper_user_data_template_vars, var.user_data_template_vars) 41 | user_data_template = var.zookeeper_user_data_template 42 | vpc_id = var.vpc_id 43 | enable_sg_creation = var.enable_sg_creation ? var.zookeeper_enable_sg_creation : false 44 | enable_dns_creation = var.enable_dns_creation && var.zookeeper_enable_dns_creation 45 | 46 | port_external_range = var.zookeeper_port_external_range 47 | port_internal_range = var.zookeeper_port_internal_range 48 | external_access_security_group_ids = var.zookeeper_external_access_security_group_ids 49 | external_access_cidrs = var.zookeeper_external_access_cidrs 50 | 51 | extra_ebs_volumes = var.zookeeper_extra_ebs_volumes 52 | vol_trans_log_size = var.zookeeper_vol_trans_log_size 53 | vol_trans_log_device_name = var.zookeeper_vol_trans_log_device_name 54 | vol_storage_size = var.zookeeper_vol_storage_size 55 | vol_storage_device_name = var.zookeeper_vol_storage_device_name 56 | 57 | monitoring_security_group_ids = var.monitoring_security_group_ids 58 | monitoring_cidrs = var.monitoring_cidrs 59 | prometheus_enabled = var.prometheus_enabled 60 | jolokia_enabled = var.jolokia_enabled 61 | prometheus_port = var.zookeeper_prometheus_port 62 | jolokia_port = var.zookeeper_jolokia_port 63 | } 64 | 65 | module "cp-aws-kafka_broker" { 66 | source = "./modules/kafka_broker" 67 | 68 | providers = { 69 | aws = aws 70 | aws.dns = aws.dns 71 | } 72 | 73 | servers = var.kafka_broker_servers 74 | image_id = var.kafka_broker_image_id == "" ? var.image_id : var.kafka_broker_image_id 75 | instance_type = var.kafka_broker_instance_type 76 | root_volume_size = var.kafka_broker_root_volume_size 77 | key_pair = var.kafka_broker_key_pair == "" ? var.key_pair : var.kafka_broker_key_pair 78 | tags = merge(var.default_tags, var.kafka_broker_tags) 79 | subnet_ids = length(var.kafka_broker_subnet_ids) > 0 ? var.kafka_broker_subnet_ids : var.subnet_ids 80 | multi_az = var.multi_az && var.kafka_broker_multi_az 81 | security_group_ids = length(var.kafka_broker_security_group_ids) > 0 ? var.kafka_broker_security_group_ids : var.security_group_ids 82 | dns_zone_id = var.kafka_broker_dns_zone_id == "" ? var.dns_zone_id : var.kafka_broker_dns_zone_id 83 | dns_ttl = var.kafka_broker_dns_ttl 84 | name_template = var.kafka_broker_name_template 85 | dns_template = var.kafka_broker_dns_template 86 | sg_name = var.kafka_broker_sg_name 87 | kafka_connect_sg_ids = [module.cp-aws-kafka_connect.security_group] 88 | ksql_sg_ids = [module.cp-aws-ksql.security_group] 89 | rest_proxy_sg_id = module.cp-aws-rest_proxy.security_group 90 | schema_registry_sg_id = module.cp-aws-schema_registry.security_group 91 | control_center_sg_id = module.cp-aws-control_center.security_group 92 | 93 | extra_template_vars = merge(var.kafka_broker_extra_template_vars, var.extra_template_vars) 94 | user_data_template_vars = merge(var.kafka_broker_user_data_template_vars, var.user_data_template_vars) 95 | user_data_template = var.kafka_broker_user_data_template 96 | vpc_id = var.vpc_id 97 | enable_sg_creation = var.enable_sg_creation ? var.kafka_broker_enable_sg_creation : false 98 | enable_dns_creation = var.enable_dns_creation && var.kafka_broker_enable_dns_creation 99 | 100 | port_external_range = var.kafka_broker_port_external_range 101 | port_internal_range = var.kafka_broker_port_internal_range 102 | external_access_security_group_ids = var.kafka_broker_external_access_security_group_ids 103 | external_access_cidrs = var.kafka_broker_external_access_cidrs 104 | 105 | extra_ebs_volumes = var.kafka_broker_extra_ebs_volumes 106 | vol_data_size = var.kafka_broker_vol_data_size 107 | vol_data_device_name = var.kafka_broker_vol_data_device_name 108 | vol_data_type = var.kafka_broker_vol_data_type 109 | 110 | monitoring_security_group_ids = var.monitoring_security_group_ids 111 | monitoring_cidrs = var.monitoring_cidrs 112 | prometheus_enabled = var.prometheus_enabled 113 | jolokia_enabled = var.jolokia_enabled 114 | prometheus_port = var.kafka_broker_prometheus_port 115 | jolokia_port = var.kafka_broker_jolokia_port 116 | } 117 | 118 | module "cp-aws-kafka_connect" { 119 | source = "./modules/kafka_connect" 120 | 121 | providers = { 122 | aws = aws 123 | aws.dns = aws.dns 124 | } 125 | 126 | servers = var.kafka_connect_servers 127 | image_id = var.kafka_connect_image_id == "" ? var.image_id : var.kafka_connect_image_id 128 | instance_type = var.kafka_connect_instance_type 129 | root_volume_size = var.kafka_connect_root_volume_size 130 | key_pair = var.kafka_connect_key_pair == "" ? var.key_pair : var.kafka_connect_key_pair 131 | tags = merge(var.default_tags, var.kafka_connect_tags) 132 | subnet_ids = length(var.kafka_connect_subnet_ids) > 0 ? var.kafka_connect_subnet_ids : var.subnet_ids 133 | multi_az = var.multi_az && var.kafka_connect_multi_az 134 | security_group_ids = length(var.kafka_connect_security_group_ids) > 0 ? var.kafka_connect_security_group_ids : var.security_group_ids 135 | dns_zone_id = var.kafka_connect_dns_zone_id == "" ? var.dns_zone_id : var.kafka_connect_dns_zone_id 136 | dns_ttl = var.kafka_connect_dns_ttl 137 | name_template = var.kafka_connect_name_template 138 | dns_template = var.kafka_connect_dns_template 139 | sg_name = var.kafka_connect_sg_name 140 | control_center_sg_id = module.cp-aws-control_center.security_group 141 | 142 | extra_template_vars = merge(var.kafka_connect_extra_template_vars, var.extra_template_vars) 143 | user_data_template_vars = merge(var.kafka_connect_user_data_template_vars, var.user_data_template_vars) 144 | user_data_template = var.kafka_connect_user_data_template 145 | vpc_id = var.vpc_id 146 | enable_sg_creation = var.enable_sg_creation ? var.kafka_connect_enable_sg_creation : false 147 | enable_dns_creation = var.enable_dns_creation && var.kafka_connect_enable_dns_creation 148 | 149 | port_external_range = var.kafka_connect_port_external_range 150 | port_internal_range = var.kafka_connect_port_internal_range 151 | external_access_security_group_ids = var.kafka_connect_external_access_security_group_ids 152 | external_access_cidrs = var.kafka_connect_external_access_cidrs 153 | 154 | extra_ebs_volumes = var.kafka_connect_extra_ebs_volumes 155 | 156 | monitoring_security_group_ids = var.monitoring_security_group_ids 157 | monitoring_cidrs = var.monitoring_cidrs 158 | prometheus_enabled = var.prometheus_enabled 159 | jolokia_enabled = var.jolokia_enabled 160 | prometheus_port = var.kafka_connect_prometheus_port 161 | jolokia_port = var.kafka_connect_jolokia_port 162 | } 163 | 164 | module "cp-aws-control_center" { 165 | source = "./modules/control_center" 166 | 167 | providers = { 168 | aws = aws 169 | aws.dns = aws.dns 170 | } 171 | 172 | servers = var.control_center_servers 173 | image_id = var.control_center_image_id == "" ? var.image_id : var.control_center_image_id 174 | instance_type = var.control_center_instance_type 175 | root_volume_size = var.control_center_root_volume_size 176 | key_pair = var.control_center_key_pair == "" ? var.key_pair : var.control_center_key_pair 177 | tags = merge(var.default_tags, var.control_center_tags) 178 | subnet_ids = length(var.control_center_subnet_ids) > 0 ? var.control_center_subnet_ids : var.subnet_ids 179 | multi_az = var.multi_az && var.control_center_multi_az 180 | security_group_ids = length(var.control_center_security_group_ids) > 0 ? var.control_center_security_group_ids : var.security_group_ids 181 | dns_zone_id = var.control_center_dns_zone_id == "" ? var.dns_zone_id : var.control_center_dns_zone_id 182 | dns_ttl = var.control_center_dns_ttl 183 | name_template = var.control_center_name_template 184 | dns_template = var.control_center_dns_template 185 | sg_name = var.control_center_sg_name 186 | 187 | extra_template_vars = merge(var.control_center_extra_template_vars, var.extra_template_vars) 188 | user_data_template_vars = merge(var.control_center_user_data_template_vars, var.user_data_template_vars) 189 | user_data_template = var.control_center_user_data_template 190 | vpc_id = var.vpc_id 191 | enable_sg_creation = var.enable_sg_creation ? var.control_center_enable_sg_creation : false 192 | enable_dns_creation = var.enable_dns_creation && var.control_center_enable_dns_creation 193 | 194 | port_external_range = var.control_center_port_external_range 195 | port_internal_range = var.control_center_port_internal_range 196 | external_access_security_group_ids = var.control_center_external_access_security_group_ids 197 | external_access_cidrs = var.control_center_external_access_cidrs 198 | 199 | extra_ebs_volumes = var.control_center_extra_ebs_volumes 200 | vol_data_size = var.control_center_vol_data_size 201 | vol_data_device_name = var.control_center_vol_data_device_name 202 | } 203 | 204 | module "cp-aws-ksql" { 205 | source = "./modules/ksql" 206 | 207 | providers = { 208 | aws = aws 209 | aws.dns = aws.dns 210 | } 211 | 212 | servers = var.ksql_servers 213 | image_id = var.ksql_image_id == "" ? var.image_id : var.ksql_image_id 214 | instance_type = var.ksql_instance_type 215 | root_volume_size = var.ksql_root_volume_size 216 | key_pair = var.ksql_key_pair == "" ? var.key_pair : var.ksql_key_pair 217 | tags = merge(var.default_tags, var.ksql_tags) 218 | subnet_ids = length(var.ksql_subnet_ids) > 0 ? var.ksql_subnet_ids : var.subnet_ids 219 | multi_az = var.multi_az && var.ksql_multi_az 220 | security_group_ids = length(var.ksql_security_group_ids) > 0 ? var.ksql_security_group_ids : var.security_group_ids 221 | dns_zone_id = var.ksql_dns_zone_id == "" ? var.dns_zone_id : var.ksql_dns_zone_id 222 | dns_ttl = var.ksql_dns_ttl 223 | name_template = var.ksql_name_template 224 | dns_template = var.ksql_dns_template 225 | sg_name = var.ksql_sg_name 226 | control_center_sg_id = module.cp-aws-control_center.security_group 227 | 228 | extra_template_vars = merge(var.ksql_extra_template_vars, var.extra_template_vars) 229 | user_data_template_vars = merge(var.ksql_user_data_template_vars, var.user_data_template_vars) 230 | user_data_template = var.ksql_user_data_template 231 | vpc_id = var.vpc_id 232 | enable_sg_creation = var.enable_sg_creation ? var.ksql_enable_sg_creation : false 233 | enable_dns_creation = var.enable_dns_creation && var.ksql_enable_dns_creation 234 | 235 | port_external_range = var.ksql_port_external_range 236 | port_internal_range = var.ksql_port_internal_range 237 | external_access_security_group_ids = var.ksql_external_access_security_group_ids 238 | external_access_cidrs = var.ksql_external_access_cidrs 239 | 240 | extra_ebs_volumes = var.ksql_extra_ebs_volumes 241 | vol_data_size = var.ksql_vol_data_size 242 | vol_data_device_name = var.ksql_vol_data_device_name 243 | 244 | monitoring_security_group_ids = var.monitoring_security_group_ids 245 | monitoring_cidrs = var.monitoring_cidrs 246 | prometheus_enabled = var.prometheus_enabled 247 | jolokia_enabled = var.jolokia_enabled 248 | prometheus_port = var.ksql_prometheus_port 249 | jolokia_port = var.ksql_jolokia_port 250 | } 251 | 252 | module "cp-aws-rest_proxy" { 253 | source = "./modules/rest_proxy" 254 | 255 | providers = { 256 | aws = aws 257 | aws.dns = aws.dns 258 | } 259 | 260 | servers = var.rest_proxy_servers 261 | image_id = var.rest_proxy_image_id == "" ? var.image_id : var.rest_proxy_image_id 262 | instance_type = var.rest_proxy_instance_type 263 | root_volume_size = var.rest_proxy_root_volume_size 264 | key_pair = var.rest_proxy_key_pair == "" ? var.key_pair : var.rest_proxy_key_pair 265 | tags = merge(var.default_tags, var.rest_proxy_tags) 266 | subnet_ids = length(var.rest_proxy_subnet_ids) > 0 ? var.rest_proxy_subnet_ids : var.subnet_ids 267 | multi_az = var.multi_az && var.rest_proxy_multi_az 268 | security_group_ids = length(var.rest_proxy_security_group_ids) > 0 ? var.rest_proxy_security_group_ids : var.security_group_ids 269 | dns_zone_id = var.rest_proxy_dns_zone_id == "" ? var.dns_zone_id : var.rest_proxy_dns_zone_id 270 | dns_ttl = var.rest_proxy_dns_ttl 271 | name_template = var.rest_proxy_name_template 272 | dns_template = var.rest_proxy_dns_template 273 | sg_name = var.rest_proxy_sg_name 274 | 275 | extra_template_vars = merge(var.rest_proxy_extra_template_vars, var.extra_template_vars) 276 | user_data_template_vars = merge(var.rest_proxy_user_data_template_vars, var.user_data_template_vars) 277 | user_data_template = var.rest_proxy_user_data_template 278 | vpc_id = var.vpc_id 279 | enable_sg_creation = var.enable_sg_creation ? var.rest_proxy_enable_sg_creation : false 280 | enable_dns_creation = var.enable_dns_creation && var.rest_proxy_enable_dns_creation 281 | 282 | port_external_range = var.rest_proxy_port_external_range 283 | port_internal_range = var.rest_proxy_port_internal_range 284 | external_access_security_group_ids = var.rest_proxy_external_access_security_group_ids 285 | external_access_cidrs = var.rest_proxy_external_access_cidrs 286 | 287 | monitoring_security_group_ids = var.monitoring_security_group_ids 288 | monitoring_cidrs = var.monitoring_cidrs 289 | prometheus_enabled = var.prometheus_enabled 290 | jolokia_enabled = var.jolokia_enabled 291 | prometheus_port = var.rest_proxy_prometheus_port 292 | jolokia_port = var.rest_proxy_jolokia_port 293 | } 294 | 295 | module "cp-aws-schema_registry" { 296 | source = "./modules/schema_registry" 297 | 298 | providers = { 299 | aws = aws 300 | aws.dns = aws.dns 301 | } 302 | 303 | servers = var.schema_registry_servers 304 | image_id = var.schema_registry_image_id == "" ? var.image_id : var.schema_registry_image_id 305 | instance_type = var.schema_registry_instance_type 306 | root_volume_size = var.schema_registry_root_volume_size 307 | key_pair = var.schema_registry_key_pair == "" ? var.key_pair : var.schema_registry_key_pair 308 | tags = merge(var.default_tags, var.schema_registry_tags) 309 | subnet_ids = length(var.schema_registry_subnet_ids) > 0 ? var.schema_registry_subnet_ids : var.subnet_ids 310 | multi_az = var.multi_az && var.schema_registry_multi_az 311 | security_group_ids = length(var.schema_registry_security_group_ids) > 0 ? var.schema_registry_security_group_ids : var.security_group_ids 312 | dns_zone_id = var.schema_registry_dns_zone_id == "" ? var.dns_zone_id : var.schema_registry_dns_zone_id 313 | dns_ttl = var.schema_registry_dns_ttl 314 | name_template = var.schema_registry_name_template 315 | dns_template = var.schema_registry_dns_template 316 | sg_name = var.schema_registry_sg_name 317 | kafka_connect_sg_ids = [module.cp-aws-kafka_connect.security_group] 318 | ksql_sg_ids = [module.cp-aws-ksql.security_group] 319 | rest_proxy_sg_id = module.cp-aws-rest_proxy.security_group 320 | control_center_sg_id = module.cp-aws-control_center.security_group 321 | 322 | extra_template_vars = merge(var.schema_registry_extra_template_vars, var.extra_template_vars) 323 | user_data_template_vars = merge(var.schema_registry_user_data_template_vars, var.user_data_template_vars) 324 | user_data_template = var.schema_registry_user_data_template 325 | vpc_id = var.vpc_id 326 | enable_sg_creation = var.enable_sg_creation ? var.schema_registry_enable_sg_creation : false 327 | enable_dns_creation = var.enable_dns_creation && var.schema_registry_enable_dns_creation 328 | 329 | port_external_range = var.schema_registry_port_external_range 330 | port_internal_range = var.schema_registry_port_internal_range 331 | external_access_security_group_ids = var.schema_registry_external_access_security_group_ids 332 | external_access_cidrs = var.schema_registry_external_access_cidrs 333 | 334 | monitoring_security_group_ids = var.monitoring_security_group_ids 335 | monitoring_cidrs = var.monitoring_cidrs 336 | prometheus_enabled = var.prometheus_enabled 337 | jolokia_enabled = var.jolokia_enabled 338 | prometheus_port = var.schema_registry_prometheus_port 339 | jolokia_port = var.schema_registry_jolokia_port 340 | } -------------------------------------------------------------------------------- /modules/base_node/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | aws = { 4 | source = "hashicorp/aws" 5 | version = "~> 4.5.0" 6 | configuration_aliases = [aws.dns] 7 | } 8 | template = { 9 | source = "hashicorp/template" 10 | version = "~> 2.2.0" 11 | } 12 | } 13 | } 14 | 15 | locals { 16 | volumes_to_mount = setproduct(range(var.servers), var.ebs_volumes) 17 | 18 | public_dns = aws_instance.instance.*.public_dns 19 | private_dns = aws_instance.instance.*.private_dns 20 | dns = compact(concat(local.public_dns, local.private_dns)) 21 | public_ip = aws_instance.instance.*.public_ip 22 | private_ip = aws_instance.instance.*.private_ip 23 | ips = compact(concat(local.public_ip, local.private_ip)) 24 | 25 | prometheus_tags = var.prometheus_enabled ? {"PROMETHEUS_PORT": var.prometheus_port} : {} 26 | jolokia_tags = var.jolokia_enabled ? {"JOLOKIA_PORT": var.jolokia_port} : {} 27 | monitoring_tags = merge(local.jolokia_tags, local.prometheus_tags) 28 | 29 | user_data_template_vars = merge({}, var.user_data_template_vars, var.extra_template_vars) 30 | devices = join(",", [ 31 | for device in var.ebs_volumes: device.device_name 32 | ]) 33 | } 34 | data "template_file" "node_name" { 35 | count = var.servers 36 | template = var.name_template 37 | 38 | vars = merge(var.extra_template_vars, { 39 | "node_count" = "${count.index+1}" 40 | "itemcount" = "${count.index+1}" 41 | }) 42 | } 43 | data "template_file" "node_dns" { 44 | count = var.servers 45 | template = var.dns_template 46 | vars = merge(var.extra_template_vars, { 47 | "node_count" = "${count.index+1}" 48 | "node_name" = data.template_file.node_name[count.index].rendered 49 | "itemcount" = "${count.index+1}" 50 | "name" = data.template_file.node_name[count.index].rendered 51 | }) 52 | } 53 | data "template_file" "user_data" { 54 | count = var.servers 55 | template = var.user_data_template 56 | vars = merge(local.user_data_template_vars, { 57 | "node_count" = "${count.index+1}" 58 | "node_name" = data.template_file.node_name[count.index].rendered 59 | "node_dns" = data.template_file.node_dns[count.index].rendered 60 | "node_devices" = local.devices 61 | }) 62 | } 63 | resource "aws_instance" "instance" { 64 | provider = aws 65 | count = var.servers 66 | ami = var.image_id 67 | instance_type = var.instance_type 68 | key_name = var.key_pair 69 | subnet_id = var.multi_az ? var.subnet_ids[(count.index%length(var.subnet_ids))] : element(var.subnet_ids, 0) 70 | vpc_security_group_ids = compact(var.security_group_ids) 71 | 72 | user_data = data.template_file.user_data[count.index].rendered 73 | 74 | tags = merge(local.monitoring_tags, var.tags, { 75 | "Name"=data.template_file.node_name[count.index].rendered 76 | }) 77 | 78 | volume_tags = merge(var.tags, { 79 | "Name"=data.template_file.node_name[count.index].rendered 80 | }) 81 | 82 | root_block_device { 83 | volume_size = var.root_volume_size 84 | delete_on_termination = true 85 | volume_type = "gp2" 86 | } 87 | } 88 | 89 | resource "aws_ebs_volume" "instance_volume" { 90 | provider = aws 91 | count = length(local.volumes_to_mount) 92 | 93 | availability_zone = aws_instance.instance[local.volumes_to_mount[count.index][0]].availability_zone 94 | encrypted = local.volumes_to_mount[count.index][1].encrypted 95 | size = local.volumes_to_mount[count.index][1].size 96 | type = local.volumes_to_mount[count.index][1].type 97 | kms_key_id = local.volumes_to_mount[count.index][1].kms_key_id 98 | tags = merge(var.tags, local.volumes_to_mount[count.index][1].tags, { 99 | Name: "${aws_instance.instance[local.volumes_to_mount[count.index][0]].tags["Name"]}-${local.volumes_to_mount[count.index][1].name}" 100 | }) 101 | } 102 | 103 | resource "aws_volume_attachment" "instance_volume_attach" { 104 | provider = aws 105 | count = length(local.volumes_to_mount) 106 | 107 | device_name = local.volumes_to_mount[count.index][1].device_name 108 | volume_id = aws_ebs_volume.instance_volume[count.index].id 109 | instance_id = aws_instance.instance[local.volumes_to_mount[count.index][0]].id 110 | } 111 | 112 | resource "aws_route53_record" "dns_record" { 113 | provider = aws.dns 114 | count = var.enable_dns_creation ? var.servers : 0 115 | zone_id = var.dns_zone_id 116 | name = data.template_file.node_dns[count.index].rendered 117 | type = length(local.dns) > 0 ? "CNAME" : "A" 118 | ttl = var.dns_ttl 119 | records = [element(compact(concat([element(local.public_dns, count.index)], [element(local.private_dns, count.index)], [element(local.private_ip, count.index)], [element(local.private_ip, count.index)])), 0)] 120 | } -------------------------------------------------------------------------------- /modules/base_node/outputs.tf: -------------------------------------------------------------------------------- 1 | output "instances" { 2 | value = aws_instance.instance 3 | } 4 | 5 | output "dns_records" { 6 | value = aws_route53_record.dns_record 7 | } -------------------------------------------------------------------------------- /modules/base_node/variables.tf: -------------------------------------------------------------------------------- 1 | variable "servers" { 2 | type = number 3 | default = 0 4 | description = "Number of Nodes/Instances to create" 5 | } 6 | 7 | #Instance Related Vars 8 | variable "image_id" { 9 | type = string 10 | description = "AMI/VM image ID to use in creating each instance" 11 | } 12 | variable "instance_type" { 13 | type = string 14 | default = "t3.medium" 15 | description = "Instance Type to use as defined by the provide" 16 | } 17 | variable "root_volume_size" { 18 | type = number 19 | default = 15 20 | description = "Root ephemeral disk size in GBs" 21 | } 22 | variable "key_pair" { 23 | type = string 24 | description = "Key Pairs - Key Name to install/use for the ROOT user" 25 | } 26 | variable "tags" { 27 | type = map 28 | description = "Default set of tags to include on all entries where tagging is available. By default `name` and `Name` will be included where logical." 29 | } 30 | 31 | #Network Related Vars 32 | variable "multi_az" { 33 | type = bool 34 | default = true 35 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 36 | } 37 | variable "subnet_ids" { 38 | type = list(string) 39 | description = "Subnets where all instances should live within" 40 | } 41 | 42 | variable "security_group_ids" { 43 | type = list(string) 44 | default = [] 45 | description = "Extra security groups that you wish to have the instances added to" 46 | } 47 | 48 | #DNS Related Vars 49 | variable "dns_zone_id" { 50 | type = string 51 | description = "Route53 Zone ID to add each instances DNS record to" 52 | } 53 | variable "dns_ttl" { 54 | type = string 55 | default = "300" 56 | description = "Default DNS record TTL" 57 | } 58 | variable "enable_dns_creation" { 59 | type = bool 60 | default = true 61 | description = "Generate Route53 entries for all created resources" 62 | } 63 | 64 | variable "name_template" { 65 | type = string 66 | default = "$${format(\"%02.0f\", itemcount)}" 67 | description = "Terraform template string to generate instance names with. `itemcount` will be provided, along with the value provided in `extra_template_vars`." 68 | } 69 | variable "dns_template" { 70 | type = string 71 | default = "$${name}" 72 | description = "Terraform template string to generate the DNS entry value. `itemcount` and `name` will be provided, along with the value provided in `extra_template_vars`." 73 | } 74 | variable "extra_template_vars" { 75 | type = map 76 | default = {} 77 | description = "Default set of additional template vars to include when rendering the `dns_template` and `name_template`. Note `name` and `itemcount` are reserved names." 78 | } 79 | 80 | #EBS Volumes 81 | variable "ebs_volumes" { 82 | type = list(object({ 83 | name: string, 84 | device_name: string, 85 | encrypted: bool, 86 | kms_key_id: string, 87 | size: number, 88 | type: string, 89 | tags: map(string) 90 | })) 91 | default = [] 92 | } 93 | 94 | #Monitoring 95 | variable "prometheus_port" { 96 | type = number 97 | } 98 | variable "prometheus_enabled" { 99 | type = bool 100 | } 101 | 102 | variable "jolokia_port" { 103 | type = number 104 | } 105 | variable "jolokia_enabled" { 106 | type = bool 107 | } 108 | 109 | #User Data 110 | variable "user_data_template" { 111 | type = string 112 | default = "" 113 | description = "A Shell script to run upon instance startup" 114 | } 115 | 116 | variable "user_data_template_vars" { 117 | type = map 118 | default = {} 119 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the extra_template_vars param." 120 | } -------------------------------------------------------------------------------- /modules/base_sec_group/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | aws = { 4 | source = "hashicorp/aws" 5 | version = "~> 4.5.0" 6 | } 7 | } 8 | } 9 | 10 | locals { 11 | prometheus_port_range = var.prometheus_enabled ? [{from: var.prometheus_port,to: var.prometheus_port}] : [] 12 | jolokia_port_range = var.jolokia_enabled ? [{from: var.jolokia_port,to: var.jolokia_port}] : [] 13 | monitoring_port_range = concat(local.prometheus_port_range, local.jolokia_port_range) 14 | } 15 | resource "aws_security_group" "my_security_group" { 16 | provider = aws 17 | count = var.enable_sg_creation ? 1 : 0 18 | name = var.sg_name 19 | description = "Confluent Platform - ${var.component_name}" 20 | vpc_id = var.vpc_id 21 | 22 | tags = merge({ 23 | Name: var.sg_name 24 | }, var.tags) 25 | 26 | dynamic "ingress" { 27 | for_each = var.port_external_range 28 | content { 29 | description = "${var.component_name} - External - SecGroups" 30 | from_port = ingress.value.from 31 | to_port = ingress.value.to 32 | protocol = "tcp" 33 | security_groups = compact(var.external_access_security_group_ids) 34 | } 35 | } 36 | dynamic "ingress" { 37 | for_each = var.port_external_range 38 | content { 39 | description = "${var.component_name} - External - CIDRs" 40 | from_port = ingress.value.from 41 | to_port = ingress.value.to 42 | protocol = "tcp" 43 | cidr_blocks = compact(var.external_access_cidrs) 44 | } 45 | } 46 | dynamic "ingress" { 47 | for_each = var.port_internal_range 48 | content { 49 | description = "${var.component_name} - Internal" 50 | from_port = ingress.value.from 51 | to_port = ingress.value.to 52 | protocol = "tcp" 53 | self = true 54 | } 55 | } 56 | 57 | dynamic "ingress" { 58 | for_each = local.monitoring_port_range 59 | content { 60 | description = "${var.component_name} - Monitoring - SecGroups" 61 | from_port = ingress.value.from 62 | to_port = ingress.value.to 63 | protocol = "tcp" 64 | security_groups = compact(var.monitoring_security_group_ids) 65 | } 66 | } 67 | dynamic "ingress" { 68 | for_each = local.monitoring_port_range 69 | content { 70 | description = "${var.component_name} - Monitoring - CIDRs" 71 | from_port = ingress.value.from 72 | to_port = ingress.value.to 73 | protocol = "tcp" 74 | cidr_blocks = compact(var.monitoring_cidrs) 75 | } 76 | } 77 | } -------------------------------------------------------------------------------- /modules/base_sec_group/outputs.tf: -------------------------------------------------------------------------------- 1 | output "security_group" { 2 | value = length(aws_security_group.my_security_group) > 0 ? aws_security_group.my_security_group[0].id : "" 3 | } -------------------------------------------------------------------------------- /modules/base_sec_group/variables.tf: -------------------------------------------------------------------------------- 1 | variable "vpc_id" { 2 | type = string 3 | description = "VPC ID that Resources should exist in" 4 | } 5 | variable "enable_sg_creation" { 6 | type = bool 7 | default = true 8 | description = "Flag to enable the creation of needed Security Group(s)" 9 | } 10 | variable "sg_name" { 11 | type = string 12 | description = "Name/Name Prefix to use when creating Security Groups(s)" 13 | } 14 | variable "port_external_range" { 15 | type = list(object({from=number,to=number})) 16 | description = "External Port Ranges to Expose" 17 | } 18 | variable "port_internal_range" { 19 | type = list(object({from=number,to=number})) 20 | description = "Internal Port Ranges to Expose" 21 | } 22 | variable "external_access_security_group_ids" { 23 | type = list 24 | default = [] 25 | description = "Other Security Groups you will tro grant access to the externalized ports" 26 | } 27 | variable "external_access_cidrs" { 28 | type = list 29 | default = [] 30 | description = "CIDRs you will tro grant access to the externalized ports" 31 | } 32 | variable "component_name" { 33 | type = string 34 | description = "Component name to use in templated Names/Decriptions of policies and groups" 35 | } 36 | variable "tags" { 37 | type = map 38 | default = {} 39 | description = "What tags to assign to the Security Group(s)" 40 | } 41 | 42 | #Monitoring 43 | variable "monitoring_security_group_ids"{ 44 | type = list 45 | default = [] 46 | description = "Collection of Security Groups that need access to monitoring this component" 47 | } 48 | variable "monitoring_cidrs"{ 49 | type = list 50 | default = [] 51 | description = "Collection of CIDRS that need access to monitoring this component" 52 | } 53 | variable "prometheus_port" { 54 | type = number 55 | } 56 | variable "prometheus_enabled" { 57 | type = bool 58 | } 59 | 60 | variable "jolokia_port" { 61 | type = number 62 | } 63 | variable "jolokia_enabled" { 64 | type = bool 65 | } -------------------------------------------------------------------------------- /modules/control_center/README.md: -------------------------------------------------------------------------------- 1 | # AWS - Confluent Platform - Control Center Terraform Module 2 | 3 | Independent module for creating Control Center EC2 instances, Route53 DNS addresses, and Security Group(s). -------------------------------------------------------------------------------- /modules/control_center/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | aws = { 4 | source = "hashicorp/aws" 5 | version = "~> 4.5.0" 6 | configuration_aliases = [aws.dns] 7 | } 8 | template = { 9 | source = "hashicorp/template" 10 | version = "~> 2.2.0" 11 | } 12 | } 13 | } 14 | 15 | locals { 16 | component_name = "Control Center" 17 | component_short_name = "c3" 18 | } 19 | 20 | module "my_sec_group" { 21 | source = "../base_sec_group" 22 | 23 | providers = { 24 | aws = aws 25 | } 26 | 27 | component_name = local.component_name 28 | 29 | vpc_id = var.vpc_id 30 | enable_sg_creation = var.enable_sg_creation 31 | sg_name = var.sg_name 32 | port_external_range = var.port_external_range 33 | port_internal_range = var.port_internal_range 34 | external_access_security_group_ids = var.external_access_security_group_ids 35 | external_access_cidrs = var.external_access_cidrs 36 | tags = var.tags 37 | 38 | prometheus_enabled = false 39 | prometheus_port = 0 40 | jolokia_enabled = false 41 | jolokia_port = 0 42 | } 43 | 44 | module "my_instance" { 45 | source = "../base_node" 46 | 47 | providers = { 48 | aws = aws 49 | aws.dns = aws.dns 50 | } 51 | 52 | extra_template_vars = merge({ 53 | component_name = local.component_name 54 | component_short_name = local.component_short_name 55 | }, var.extra_template_vars) 56 | 57 | servers = var.servers 58 | image_id = var.image_id 59 | instance_type = var.instance_type 60 | root_volume_size = var.root_volume_size 61 | key_pair = var.key_pair 62 | tags = var.tags 63 | multi_az = var.multi_az 64 | subnet_ids = var.subnet_ids 65 | security_group_ids = concat(var.security_group_ids, [module.my_sec_group.security_group]) 66 | dns_zone_id = var.dns_zone_id 67 | dns_ttl = var.dns_ttl 68 | name_template = var.name_template 69 | dns_template = var.dns_template 70 | enable_dns_creation = var.enable_dns_creation 71 | 72 | prometheus_enabled = false 73 | prometheus_port = 0 74 | jolokia_enabled = false 75 | jolokia_port = 0 76 | 77 | user_data_template = var.user_data_template 78 | user_data_template_vars = var.user_data_template_vars 79 | 80 | ebs_volumes = concat([ 81 | {name:"data", device_name:var.vol_data_device_name, encrypted:false, kms_key_id="", size:var.vol_data_size, type:"gp2", tags:{}} 82 | ], var.extra_ebs_volumes) 83 | } -------------------------------------------------------------------------------- /modules/control_center/outputs.tf: -------------------------------------------------------------------------------- 1 | output "instances" { 2 | value = module.my_instance.instances 3 | } 4 | 5 | output "dns_records" { 6 | value = module.my_instance.dns_records 7 | } 8 | 9 | output "security_group" { 10 | value = module.my_sec_group.security_group 11 | } -------------------------------------------------------------------------------- /modules/control_center/variables.tf: -------------------------------------------------------------------------------- 1 | ########################### 2 | # C3 Vars 3 | ########################### 4 | variable "servers" { 5 | type = number 6 | default = 0 7 | } 8 | 9 | #Instance Related Vars 10 | variable "image_id" { 11 | type = string 12 | } 13 | variable "instance_type" { 14 | type = string 15 | default = "t3.medium" 16 | } 17 | variable "root_volume_size" { 18 | type = number 19 | default = 15 20 | } 21 | variable "key_pair" { 22 | type = string 23 | } 24 | variable "tags" { 25 | type = map 26 | default = {} 27 | description = "What tags to assign to the Security Group(s) and EC2 instances" 28 | } 29 | variable "multi_az" { 30 | type = bool 31 | default = true 32 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 33 | } 34 | 35 | #Network Related Vars 36 | variable "subnet_ids" { 37 | type = list(string) 38 | } 39 | 40 | variable "security_group_ids" { 41 | type = list 42 | default = [] 43 | } 44 | 45 | #DNS Related Vars 46 | variable "dns_zone_id" { 47 | type = string 48 | } 49 | variable "dns_ttl" { 50 | type = string 51 | default = "300" 52 | } 53 | 54 | variable "name_template" { 55 | type = string 56 | default = "ccc$${format(\"%02.0f\", itemcount)}" 57 | } 58 | variable "dns_template" { 59 | type = string 60 | default = "$${name}" 61 | } 62 | variable "enable_dns_creation" { 63 | type = bool 64 | default = true 65 | description = "Generate Route53 entries for all created resources" 66 | } 67 | variable "extra_template_vars" { 68 | type = map 69 | default = {} 70 | } 71 | 72 | #SG Related Vars 73 | variable "vpc_id" { 74 | type = string 75 | description = "VPC ID that Resources should exist in" 76 | } 77 | variable "enable_sg_creation" { 78 | type = bool 79 | default = true 80 | description = "Flag to enable the creation of needed Security Group(s)" 81 | } 82 | variable "sg_name" { 83 | type = string 84 | default = "CP_Control_Center" 85 | description = "Name/Name Prefix to use when creating Security Groups(s)" 86 | } 87 | variable "port_external_range" { 88 | type = list(object({from=number,to=number})) 89 | default = [{from=9021,to=9021}] 90 | description = "External Port Ranges to Expose" 91 | } 92 | variable "port_internal_range" { 93 | type = list(object({from=number,to=number})) 94 | default = [{from=9021,to=9021}] 95 | description = "Internal Port Ranges to Expose" 96 | } 97 | variable "external_access_security_group_ids" { 98 | type = list 99 | default = [] 100 | description = "Other Security Groups you will tro grant access to the externalized ports" 101 | } 102 | variable "external_access_cidrs" { 103 | type = list 104 | default = [] 105 | description = "CIDRs you will tro grant access to the externalized ports" 106 | } 107 | 108 | 109 | #EBS Volumes 110 | variable "extra_ebs_volumes" { 111 | type = list(object({ 112 | name: string, 113 | device_name: string, 114 | encrypted: bool, 115 | kms_key_id: string, 116 | size: number, 117 | type: string, 118 | tags: map(string) 119 | })) 120 | default = [] 121 | } 122 | 123 | variable "vol_data_size" { 124 | type = number 125 | default = 300 126 | } 127 | variable "vol_data_device_name" { 128 | type = string 129 | default = "/dev/sdf" 130 | } 131 | 132 | #User Data 133 | variable "user_data_template" { 134 | type = string 135 | default = "" 136 | description = "A Shell script to run upon instance startup" 137 | } 138 | 139 | variable "user_data_template_vars" { 140 | type = map 141 | default = {} 142 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the extra_template_vars param." 143 | } -------------------------------------------------------------------------------- /modules/kafka_broker/README.md: -------------------------------------------------------------------------------- 1 | # AWS - Confluent Platform - Kafka Broker Terraform Module 2 | 3 | Independent module for creating Kafka Broker EC2 instances, Route53 DNS addresses, and Security Group(s). -------------------------------------------------------------------------------- /modules/kafka_broker/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | aws = { 4 | source = "hashicorp/aws" 5 | version = "~> 4.5.0" 6 | configuration_aliases = [aws.dns] 7 | } 8 | template = { 9 | source = "hashicorp/template" 10 | version = "~> 2.2.0" 11 | } 12 | } 13 | } 14 | 15 | locals { 16 | component_name = "Kafka Broker" 17 | component_short_name = "kfk" 18 | } 19 | 20 | module "my_sec_group" { 21 | source = "../base_sec_group" 22 | 23 | providers = { 24 | aws = aws 25 | } 26 | 27 | component_name = local.component_name 28 | 29 | vpc_id = var.vpc_id 30 | enable_sg_creation = var.enable_sg_creation 31 | sg_name = var.sg_name 32 | port_external_range = var.port_external_range 33 | port_internal_range = var.port_internal_range 34 | external_access_security_group_ids = concat([ 35 | var.rest_proxy_sg_id, 36 | var.schema_registry_sg_id, 37 | var.control_center_sg_id 38 | ], var.kafka_connect_sg_ids, var.ksql_sg_ids, var.external_access_security_group_ids) 39 | external_access_cidrs = var.external_access_cidrs 40 | tags = var.tags 41 | 42 | monitoring_security_group_ids = var.monitoring_security_group_ids 43 | monitoring_cidrs = var.monitoring_cidrs 44 | prometheus_enabled = var.prometheus_enabled 45 | jolokia_enabled = var.jolokia_enabled 46 | prometheus_port = var.prometheus_port 47 | jolokia_port = var.jolokia_port 48 | } 49 | 50 | module "my_instance" { 51 | source = "../base_node" 52 | 53 | providers = { 54 | aws = aws 55 | aws.dns = aws.dns 56 | } 57 | 58 | extra_template_vars = merge({ 59 | component_name = local.component_name 60 | component_short_name = local.component_short_name 61 | }, var.extra_template_vars) 62 | 63 | servers = var.servers 64 | image_id = var.image_id 65 | instance_type = var.instance_type 66 | root_volume_size = var.root_volume_size 67 | key_pair = var.key_pair 68 | tags = var.tags 69 | multi_az = var.multi_az 70 | subnet_ids = var.subnet_ids 71 | security_group_ids = concat(var.security_group_ids, [module.my_sec_group.security_group]) 72 | dns_zone_id = var.dns_zone_id 73 | dns_ttl = var.dns_ttl 74 | name_template = var.name_template 75 | dns_template = var.dns_template 76 | enable_dns_creation = var.enable_dns_creation 77 | 78 | prometheus_enabled = var.prometheus_enabled 79 | jolokia_enabled = var.jolokia_enabled 80 | prometheus_port = var.prometheus_port 81 | jolokia_port = var.jolokia_port 82 | 83 | user_data_template = var.user_data_template 84 | user_data_template_vars = var.user_data_template_vars 85 | 86 | ebs_volumes = concat([ 87 | { 88 | name:"data", 89 | device_name:var.vol_data_device_name, 90 | encrypted:false, 91 | kms_key_id="", 92 | size:var.vol_data_size, 93 | type: var.vol_data_type == null ? (var.vol_data_size > 500 ? "st1" : "gp2") : var.vol_data_type, 94 | tags:{} 95 | } 96 | ], var.extra_ebs_volumes) 97 | } -------------------------------------------------------------------------------- /modules/kafka_broker/outputs.tf: -------------------------------------------------------------------------------- 1 | output "instances" { 2 | value = module.my_instance.instances 3 | } 4 | 5 | output "dns_records" { 6 | value = module.my_instance.dns_records 7 | } 8 | 9 | output "security_group" { 10 | value = module.my_sec_group.security_group 11 | } -------------------------------------------------------------------------------- /modules/kafka_broker/variables.tf: -------------------------------------------------------------------------------- 1 | ########################### 2 | # Broker Vars 3 | ########################### 4 | variable "servers" { 5 | type = number 6 | default = 1 7 | } 8 | 9 | #Instance Related Vars 10 | variable "image_id" { 11 | type = string 12 | } 13 | variable "instance_type" { 14 | type = string 15 | default = "t3.medium" 16 | } 17 | variable "root_volume_size" { 18 | type = number 19 | default = 30 20 | } 21 | variable "key_pair" { 22 | type = string 23 | } 24 | variable "tags" { 25 | type = map 26 | } 27 | variable "multi_az" { 28 | type = bool 29 | default = true 30 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 31 | } 32 | 33 | #Network Related Vars 34 | variable "subnet_ids" { 35 | type = list(string) 36 | } 37 | 38 | variable "security_group_ids" { 39 | type = list 40 | default = [] 41 | } 42 | 43 | #DNS Related Vars 44 | variable "dns_zone_id" { 45 | type = string 46 | } 47 | variable "dns_ttl" { 48 | type = string 49 | default = "300" 50 | } 51 | 52 | variable "name_template" { 53 | type = string 54 | default = "kfk$${format(\"%02.0f\", itemcount)}" 55 | } 56 | variable "dns_template" { 57 | type = string 58 | default = "$${name}" 59 | } 60 | variable "enable_dns_creation" { 61 | type = bool 62 | default = true 63 | description = "Generate Route53 entries for all created resources" 64 | } 65 | variable "extra_template_vars" { 66 | type = map 67 | default = {} 68 | } 69 | 70 | #SG Related Vars 71 | variable "vpc_id" { 72 | type = string 73 | description = "VPC ID that Resources should exist in" 74 | } 75 | variable "enable_sg_creation" { 76 | type = bool 77 | default = true 78 | description = "Flag to enable the creation of needed Security Group(s)" 79 | } 80 | variable "sg_name" { 81 | type = string 82 | default = "CP_Kafka_Broker" 83 | description = "Name/Name Prefix to use when creating Security Groups(s)" 84 | } 85 | variable "port_external_range" { 86 | type = list(object({from=number,to=number})) 87 | default = [{from=9091,to=9093},{from:8090,to:8091}] 88 | description = "External Port Ranges to Expose" 89 | } 90 | variable "port_internal_range" { 91 | type = list(object({from=number,to=number})) 92 | default = [{from=9091,to=9092},{from:8090,to:8091}] 93 | description = "Internal Port Ranges to Expose" 94 | } 95 | variable "external_access_security_group_ids" { 96 | type = list 97 | default = [] 98 | description = "Other Security Groups you will tro grant access to the externalized ports" 99 | } 100 | variable "external_access_cidrs" { 101 | type = list 102 | default = [] 103 | description = "CIDRs you will tro grant access to the externalized ports" 104 | } 105 | variable "kafka_connect_sg_ids" { 106 | type = list 107 | default = [] 108 | } 109 | variable "ksql_sg_ids" { 110 | type = list 111 | default = [] 112 | } 113 | variable "rest_proxy_sg_id" { 114 | type = string 115 | default = "" 116 | } 117 | variable "schema_registry_sg_id" { 118 | type = string 119 | default = "" 120 | } 121 | variable "control_center_sg_id" { 122 | type = string 123 | default = "" 124 | } 125 | 126 | 127 | #EBS Volumes 128 | variable "extra_ebs_volumes" { 129 | type = list(object({ 130 | name: string, 131 | device_name: string, 132 | encrypted: bool, 133 | kms_key_id: string, 134 | size: number, 135 | type: string, 136 | tags: map(string) 137 | })) 138 | default = [] 139 | } 140 | 141 | variable "vol_data_size" { 142 | type = number 143 | default = 500 144 | } 145 | variable "vol_data_device_name" { 146 | type = string 147 | default = "/dev/sdf" 148 | } 149 | variable "vol_data_type" { 150 | type = string 151 | default = null 152 | } 153 | 154 | #Monitoring 155 | variable "monitoring_security_group_ids"{ 156 | type = list 157 | default = [] 158 | description = "Collection of Security Groups that need access to monitoring this component" 159 | } 160 | variable "monitoring_cidrs"{ 161 | type = list 162 | default = [] 163 | description = "Collection of CIDRS that need access to monitoring this component" 164 | } 165 | 166 | variable "prometheus_port" { 167 | type = number 168 | default = 8080 169 | description = "Port on which the Prometheus Agent is running" 170 | } 171 | variable "prometheus_enabled" { 172 | type = bool 173 | default = true 174 | } 175 | 176 | variable "jolokia_port" { 177 | type = number 178 | default = 7771 179 | description = "Port on which the Jolokia Agent is running" 180 | } 181 | variable "jolokia_enabled" { 182 | type = bool 183 | default = true 184 | } 185 | 186 | #User Data 187 | variable "user_data_template" { 188 | type = string 189 | default = "" 190 | description = "A Shell script to run upon instance startup" 191 | } 192 | 193 | variable "user_data_template_vars" { 194 | type = map 195 | default = {} 196 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the extra_template_vars param." 197 | } -------------------------------------------------------------------------------- /modules/kafka_connect/README.md: -------------------------------------------------------------------------------- 1 | # AWS - Confluent Platform - Kafka Connect Terraform Module 2 | 3 | Independent module for creating kafka Connect EC2 instances, Route53 DNS addresses, and Security Group(s). -------------------------------------------------------------------------------- /modules/kafka_connect/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | aws = { 4 | source = "hashicorp/aws" 5 | version = "~> 4.5.0" 6 | configuration_aliases = [aws.dns] 7 | } 8 | template = { 9 | source = "hashicorp/template" 10 | version = "~> 2.2.0" 11 | } 12 | } 13 | } 14 | 15 | locals { 16 | component_name = "Kafka Connect" 17 | component_short_name = "connect" 18 | } 19 | 20 | module "my_sec_group" { 21 | source = "../base_sec_group" 22 | 23 | providers = { 24 | aws = aws 25 | } 26 | 27 | component_name = local.component_name 28 | 29 | vpc_id = var.vpc_id 30 | enable_sg_creation = var.enable_sg_creation 31 | sg_name = var.sg_name 32 | port_external_range = var.port_external_range 33 | port_internal_range = var.port_internal_range 34 | external_access_security_group_ids = concat([ 35 | var.control_center_sg_id 36 | ], var.external_access_security_group_ids) 37 | external_access_cidrs = var.external_access_cidrs 38 | tags = var.tags 39 | 40 | monitoring_security_group_ids = var.monitoring_security_group_ids 41 | monitoring_cidrs = var.monitoring_cidrs 42 | prometheus_enabled = var.prometheus_enabled 43 | jolokia_enabled = var.jolokia_enabled 44 | prometheus_port = var.prometheus_port 45 | jolokia_port = var.jolokia_port 46 | } 47 | 48 | module "my_instance" { 49 | source = "../base_node" 50 | 51 | providers = { 52 | aws = aws 53 | aws.dns = aws.dns 54 | } 55 | 56 | extra_template_vars = merge({ 57 | component_name = local.component_name 58 | component_short_name = local.component_short_name 59 | }, var.extra_template_vars) 60 | 61 | servers = var.servers 62 | image_id = var.image_id 63 | instance_type = var.instance_type 64 | root_volume_size = var.root_volume_size 65 | key_pair = var.key_pair 66 | tags = var.tags 67 | multi_az = var.multi_az 68 | subnet_ids = var.subnet_ids 69 | security_group_ids = concat(var.security_group_ids, [module.my_sec_group.security_group]) 70 | dns_zone_id = var.dns_zone_id 71 | dns_ttl = var.dns_ttl 72 | name_template = var.name_template 73 | dns_template = var.dns_template 74 | enable_dns_creation = var.enable_dns_creation 75 | 76 | prometheus_enabled = var.prometheus_enabled 77 | jolokia_enabled = var.jolokia_enabled 78 | prometheus_port = var.prometheus_port 79 | jolokia_port = var.jolokia_port 80 | 81 | user_data_template = var.user_data_template 82 | user_data_template_vars = var.user_data_template_vars 83 | 84 | 85 | ebs_volumes = var.extra_ebs_volumes 86 | } -------------------------------------------------------------------------------- /modules/kafka_connect/output.tf: -------------------------------------------------------------------------------- 1 | output "instances" { 2 | value = module.my_instance.instances 3 | } 4 | 5 | output "dns_records" { 6 | value = module.my_instance.dns_records 7 | } 8 | 9 | output "security_group" { 10 | value = module.my_sec_group.security_group 11 | } -------------------------------------------------------------------------------- /modules/kafka_connect/variables.tf: -------------------------------------------------------------------------------- 1 | ########################### 2 | # Connect Vars 3 | ########################### 4 | variable "servers" { 5 | type = number 6 | default = 0 7 | } 8 | 9 | #Instance Related Vars 10 | variable "image_id" { 11 | type = string 12 | } 13 | variable "instance_type" { 14 | type = string 15 | default = "t3.medium" 16 | } 17 | variable "root_volume_size" { 18 | type = number 19 | default = 15 20 | } 21 | variable "key_pair" { 22 | type = string 23 | } 24 | variable "tags" { 25 | type = map 26 | } 27 | variable "multi_az" { 28 | type = bool 29 | default = true 30 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 31 | } 32 | 33 | #Network Related Vars 34 | variable "subnet_ids" { 35 | type = list(string) 36 | } 37 | 38 | variable "security_group_ids" { 39 | type = list 40 | default = [] 41 | } 42 | 43 | #DNS Related Vars 44 | variable "dns_zone_id" { 45 | type = string 46 | } 47 | variable "dns_ttl" { 48 | type = string 49 | default = "300" 50 | } 51 | 52 | variable "name_template" { 53 | type = string 54 | default = "connect$${format(\"%02.0f\", itemcount)}" 55 | } 56 | variable "dns_template" { 57 | type = string 58 | default = "$${name}" 59 | } 60 | variable "enable_dns_creation" { 61 | type = bool 62 | default = true 63 | description = "Generate Route53 entries for all created resources" 64 | } 65 | variable "extra_template_vars" { 66 | type = map 67 | default = {} 68 | } 69 | 70 | #SG Related Vars 71 | variable "vpc_id" { 72 | type = string 73 | description = "VPC ID that Resources should exist in" 74 | } 75 | variable "enable_sg_creation" { 76 | type = bool 77 | default = true 78 | description = "Flag to enable the creation of needed Security Group(s)" 79 | } 80 | variable "sg_name" { 81 | type = string 82 | default = "CP_Kafka_Connect" 83 | description = "Name/Name Prefix to use when creating Security Groups(s)" 84 | } 85 | variable "port_external_range" { 86 | type = list(object({from=number,to=number})) 87 | default = [{from=8083,to=8083}] 88 | description = "External Port Ranges to Expose" 89 | } 90 | variable "port_internal_range" { 91 | type = list(object({from=number,to=number})) 92 | default = [{from=8083,to=8083}] 93 | description = "Internal Port Ranges to Expose" 94 | } 95 | variable "external_access_security_group_ids" { 96 | type = list 97 | default = [] 98 | description = "Other Security Groups you will tro grant access to the externalized ports" 99 | } 100 | variable "external_access_cidrs" { 101 | type = list 102 | default = [] 103 | description = "CIDRs you will tro grant access to the externalized ports" 104 | } 105 | variable "control_center_sg_id" { 106 | type = string 107 | default = "" 108 | } 109 | 110 | #EBS Volumes 111 | variable "extra_ebs_volumes" { 112 | type = list(object({ 113 | name: string, 114 | device_name: string, 115 | encrypted: bool, 116 | kms_key_id: string, 117 | size: number, 118 | type: string, 119 | tags: map(string) 120 | })) 121 | default = [] 122 | } 123 | 124 | #Monitoring 125 | variable "monitoring_security_group_ids"{ 126 | type = list 127 | default = [] 128 | description = "Collection of Security Groups that need access to monitoring this component" 129 | } 130 | variable "monitoring_cidrs"{ 131 | type = list 132 | default = [] 133 | description = "Collection of CIDRS that need access to monitoring this component" 134 | } 135 | 136 | variable "prometheus_port" { 137 | type = number 138 | default = 8077 139 | description = "Port on which the Prometheus Agent is running" 140 | } 141 | variable "prometheus_enabled" { 142 | type = bool 143 | default = true 144 | } 145 | 146 | variable "jolokia_port" { 147 | type = number 148 | default = 7773 149 | description = "Port on which the Jolokia Agent is running" 150 | } 151 | variable "jolokia_enabled" { 152 | type = bool 153 | default = true 154 | } 155 | 156 | #User Data 157 | variable "user_data_template" { 158 | type = string 159 | default = "" 160 | description = "A Shell script to run upon instance startup" 161 | } 162 | 163 | variable "user_data_template_vars" { 164 | type = map 165 | default = {} 166 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the extra_template_vars param." 167 | } -------------------------------------------------------------------------------- /modules/ksql/README.md: -------------------------------------------------------------------------------- 1 | # AWS - Confluent Platform - KSQL Terraform Module 2 | 3 | Independent module for creating KSQL EC2 instances, Route53 DNS addresses, and Security Group(s). -------------------------------------------------------------------------------- /modules/ksql/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | aws = { 4 | source = "hashicorp/aws" 5 | version = "~> 4.5.0" 6 | configuration_aliases = [aws.dns] 7 | } 8 | template = { 9 | source = "hashicorp/template" 10 | version = "~> 2.2.0" 11 | } 12 | } 13 | } 14 | 15 | locals { 16 | component_name = "KSQL" 17 | component_short_name = "ksql" 18 | } 19 | 20 | module "my_sec_group" { 21 | source = "../base_sec_group" 22 | 23 | providers = { 24 | aws = aws 25 | } 26 | 27 | component_name = local.component_name 28 | 29 | vpc_id = var.vpc_id 30 | enable_sg_creation = var.enable_sg_creation 31 | sg_name = var.sg_name 32 | port_external_range = var.port_external_range 33 | port_internal_range = var.port_internal_range 34 | external_access_security_group_ids = concat([ 35 | var.control_center_sg_id 36 | ], var.external_access_security_group_ids) 37 | external_access_cidrs = var.external_access_cidrs 38 | tags = var.tags 39 | 40 | monitoring_security_group_ids = var.monitoring_security_group_ids 41 | monitoring_cidrs = var.monitoring_cidrs 42 | prometheus_enabled = var.prometheus_enabled 43 | jolokia_enabled = var.jolokia_enabled 44 | prometheus_port = var.prometheus_port 45 | jolokia_port = var.jolokia_port 46 | } 47 | 48 | module "my_instance" { 49 | source = "../base_node" 50 | 51 | providers = { 52 | aws = aws 53 | aws.dns = aws.dns 54 | } 55 | 56 | extra_template_vars = merge({ 57 | component_name = local.component_name 58 | component_short_name = local.component_short_name 59 | }, var.extra_template_vars) 60 | 61 | servers = var.servers 62 | image_id = var.image_id 63 | instance_type = var.instance_type 64 | root_volume_size = var.root_volume_size 65 | key_pair = var.key_pair 66 | tags = var.tags 67 | multi_az = var.multi_az 68 | subnet_ids = var.subnet_ids 69 | security_group_ids = concat(var.security_group_ids, [module.my_sec_group.security_group]) 70 | dns_zone_id = var.dns_zone_id 71 | dns_ttl = var.dns_ttl 72 | name_template = var.name_template 73 | dns_template = var.dns_template 74 | enable_dns_creation = var.enable_dns_creation 75 | 76 | prometheus_enabled = var.prometheus_enabled 77 | jolokia_enabled = var.jolokia_enabled 78 | prometheus_port = var.prometheus_port 79 | jolokia_port = var.jolokia_port 80 | 81 | user_data_template = var.user_data_template 82 | user_data_template_vars = var.user_data_template_vars 83 | 84 | 85 | ebs_volumes = concat([ 86 | {name:"data", device_name:var.vol_data_device_name, encrypted:false, kms_key_id="", size:var.vol_data_size, type:"gp2", tags:{}} 87 | ], var.extra_ebs_volumes) 88 | } -------------------------------------------------------------------------------- /modules/ksql/outputs.tf: -------------------------------------------------------------------------------- 1 | output "instances" { 2 | value = module.my_instance.instances 3 | } 4 | 5 | output "dns_records" { 6 | value = module.my_instance.dns_records 7 | } 8 | 9 | output "security_group" { 10 | value = module.my_sec_group.security_group 11 | } -------------------------------------------------------------------------------- /modules/ksql/variables.tf: -------------------------------------------------------------------------------- 1 | ########################### 2 | # KSQL Vars 3 | ########################### 4 | variable "servers" { 5 | type = number 6 | default = 0 7 | } 8 | 9 | #Instance Related Vars 10 | variable "image_id" { 11 | type = string 12 | } 13 | variable "instance_type" { 14 | type = string 15 | default = "t3.small" 16 | } 17 | variable "root_volume_size" { 18 | type = number 19 | default = 15 20 | } 21 | variable "key_pair" { 22 | type = string 23 | } 24 | variable "tags" { 25 | type = map 26 | } 27 | variable "multi_az" { 28 | type = bool 29 | default = true 30 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 31 | } 32 | 33 | #Network Related Vars 34 | variable "subnet_ids" { 35 | type = list(string) 36 | } 37 | 38 | variable "security_group_ids" { 39 | type = list 40 | default = [] 41 | } 42 | 43 | #DNS Related Vars 44 | variable "dns_zone_id" { 45 | type = string 46 | } 47 | variable "dns_ttl" { 48 | type = string 49 | default = "300" 50 | } 51 | 52 | variable "name_template" { 53 | type = string 54 | default = "ksql$${format(\"%02.0f\", itemcount)}" 55 | } 56 | variable "dns_template" { 57 | type = string 58 | default = "$${name}" 59 | } 60 | variable "enable_dns_creation" { 61 | type = bool 62 | default = true 63 | description = "Generate Route53 entries for all created resources" 64 | } 65 | variable "extra_template_vars" { 66 | type = map 67 | default = {} 68 | } 69 | 70 | #SG Related Vars 71 | variable "vpc_id" { 72 | type = string 73 | description = "VPC ID that Resources should exist in" 74 | } 75 | variable "enable_sg_creation" { 76 | type = bool 77 | default = true 78 | description = "Flag to enable the creation of needed Security Group(s)" 79 | } 80 | variable "sg_name" { 81 | type = string 82 | default = "CP_KSQL" 83 | description = "Name/Name Prefix to use when creating Security Groups(s)" 84 | } 85 | variable "port_external_range" { 86 | type = list(object({from=number,to=number})) 87 | default = [{from=8088,to=8088}] 88 | description = "External Port Ranges to Expose" 89 | } 90 | variable "port_internal_range" { 91 | type = list(object({from=number,to=number})) 92 | default = [{from=8088,to=8088}] 93 | description = "Internal Port Ranges to Expose" 94 | } 95 | variable "external_access_security_group_ids" { 96 | type = list 97 | default = [] 98 | description = "Other Security Groups you will tro grant access to the externalized ports" 99 | } 100 | variable "external_access_cidrs" { 101 | type = list 102 | default = [] 103 | description = "CIDRs you will tro grant access to the externalized ports" 104 | } 105 | variable "control_center_sg_id" { 106 | type = string 107 | default = "" 108 | } 109 | 110 | 111 | #EBS Volumes 112 | variable "extra_ebs_volumes" { 113 | type = list(object({ 114 | name: string, 115 | device_name: string, 116 | encrypted: bool, 117 | kms_key_id: string, 118 | size: number, 119 | type: string, 120 | tags: map(string) 121 | })) 122 | default = [] 123 | } 124 | 125 | variable "vol_data_size" { 126 | type = number 127 | default = 100 128 | } 129 | variable "vol_data_device_name" { 130 | type = string 131 | default = "/dev/sdf" 132 | } 133 | 134 | #Monitoring 135 | variable "monitoring_security_group_ids"{ 136 | type = list 137 | default = [] 138 | description = "Collection of Security Groups that need access to monitoring this component" 139 | } 140 | variable "monitoring_cidrs"{ 141 | type = list 142 | default = [] 143 | description = "Collection of CIDRS that need access to monitoring this component" 144 | } 145 | 146 | variable "prometheus_port" { 147 | type = number 148 | default = 8076 149 | description = "Port on which the Prometheus Agent is running" 150 | } 151 | variable "prometheus_enabled" { 152 | type = bool 153 | default = true 154 | } 155 | 156 | variable "jolokia_port" { 157 | type = number 158 | default = 7774 159 | description = "Port on which the Jolokia Agent is running" 160 | } 161 | variable "jolokia_enabled" { 162 | type = bool 163 | default = true 164 | } 165 | 166 | #User Data 167 | variable "user_data_template" { 168 | type = string 169 | default = "" 170 | description = "A Shell script to run upon instance startup" 171 | } 172 | 173 | variable "user_data_template_vars" { 174 | type = map 175 | default = {} 176 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the extra_template_vars param." 177 | } -------------------------------------------------------------------------------- /modules/rest_proxy/README.md: -------------------------------------------------------------------------------- 1 | # AWS - Confluent Platform - REST Proxy Terraform Module 2 | 3 | Independent module for creating REST Proxy EC2 instances, Route53 DNS addresses, and Security Group(s). -------------------------------------------------------------------------------- /modules/rest_proxy/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | aws = { 4 | source = "hashicorp/aws" 5 | version = "~> 4.5.0" 6 | configuration_aliases = [aws.dns] 7 | } 8 | template = { 9 | source = "hashicorp/template" 10 | version = "~> 2.2.0" 11 | } 12 | } 13 | } 14 | 15 | locals { 16 | component_name = "REST Proxy" 17 | component_short_name = "rest" 18 | } 19 | 20 | module "my_sec_group" { 21 | source = "../base_sec_group" 22 | 23 | providers = { 24 | aws = aws 25 | } 26 | 27 | component_name = local.component_name 28 | 29 | vpc_id = var.vpc_id 30 | enable_sg_creation = var.enable_sg_creation 31 | sg_name = var.sg_name 32 | port_external_range = var.port_external_range 33 | port_internal_range = var.port_internal_range 34 | external_access_security_group_ids = var.external_access_security_group_ids 35 | external_access_cidrs = var.external_access_cidrs 36 | tags = var.tags 37 | 38 | monitoring_security_group_ids = var.monitoring_security_group_ids 39 | monitoring_cidrs = var.monitoring_cidrs 40 | prometheus_enabled = var.prometheus_enabled 41 | jolokia_enabled = var.jolokia_enabled 42 | prometheus_port = var.prometheus_port 43 | jolokia_port = var.jolokia_port 44 | } 45 | 46 | module "my_instance" { 47 | source = "../base_node" 48 | 49 | providers = { 50 | aws = aws 51 | aws.dns = aws.dns 52 | } 53 | 54 | extra_template_vars = merge({ 55 | component_name = local.component_name 56 | component_short_name = local.component_short_name 57 | }, var.extra_template_vars) 58 | 59 | servers = var.servers 60 | image_id = var.image_id 61 | instance_type = var.instance_type 62 | root_volume_size = var.root_volume_size 63 | key_pair = var.key_pair 64 | tags = var.tags 65 | multi_az = var.multi_az 66 | subnet_ids = var.subnet_ids 67 | security_group_ids = concat(var.security_group_ids, [module.my_sec_group.security_group]) 68 | dns_zone_id = var.dns_zone_id 69 | dns_ttl = var.dns_ttl 70 | name_template = var.name_template 71 | dns_template = var.dns_template 72 | enable_dns_creation = var.enable_dns_creation 73 | 74 | prometheus_enabled = var.prometheus_enabled 75 | jolokia_enabled = var.jolokia_enabled 76 | prometheus_port = var.prometheus_port 77 | jolokia_port = var.jolokia_port 78 | 79 | user_data_template = var.user_data_template 80 | user_data_template_vars = var.user_data_template_vars 81 | } -------------------------------------------------------------------------------- /modules/rest_proxy/outputs.tf: -------------------------------------------------------------------------------- 1 | output "instances" { 2 | value = module.my_instance.instances 3 | } 4 | 5 | output "dns_records" { 6 | value = module.my_instance.dns_records 7 | } 8 | 9 | output "security_group" { 10 | value = module.my_sec_group.security_group 11 | } -------------------------------------------------------------------------------- /modules/rest_proxy/variables.tf: -------------------------------------------------------------------------------- 1 | ########################### 2 | # RESTProxy Vars 3 | ########################### 4 | variable "servers" { 5 | type = number 6 | default = 0 7 | } 8 | 9 | #Instance Related Vars 10 | variable "image_id" { 11 | type = string 12 | } 13 | variable "instance_type" { 14 | type = string 15 | default = "t3.small" 16 | } 17 | variable "root_volume_size" { 18 | type = number 19 | default = 15 20 | } 21 | variable "key_pair" { 22 | type = string 23 | } 24 | variable "tags" { 25 | type = map 26 | } 27 | variable "multi_az" { 28 | type = bool 29 | default = true 30 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 31 | } 32 | 33 | #Network Related Vars 34 | variable "subnet_ids" { 35 | type = list(string) 36 | } 37 | 38 | variable "security_group_ids" { 39 | type = list 40 | default = [] 41 | } 42 | 43 | #DNS Related Vars 44 | variable "dns_zone_id" { 45 | type = string 46 | } 47 | variable "dns_ttl" { 48 | type = string 49 | default = "300" 50 | } 51 | variable "name_template" { 52 | type = string 53 | default = "rest$${format(\"%02.0f\", itemcount)}" 54 | } 55 | variable "dns_template" { 56 | type = string 57 | default = "$${name}" 58 | } 59 | variable "enable_dns_creation" { 60 | type = bool 61 | default = true 62 | description = "Generate Route53 entries for all created resources" 63 | } 64 | variable "extra_template_vars" { 65 | type = map 66 | default = {} 67 | } 68 | 69 | #SG Related Vars 70 | variable "vpc_id" { 71 | type = string 72 | description = "VPC ID that Resources should exist in" 73 | } 74 | variable "enable_sg_creation" { 75 | type = bool 76 | default = true 77 | description = "Flag to enable the creation of needed Security Group(s)" 78 | } 79 | variable "sg_name" { 80 | type = string 81 | default = "CP_REST_Proxy" 82 | description = "Name/Name Prefix to use when creating Security Groups(s)" 83 | } 84 | variable "port_external_range" { 85 | type = list(object({from=number,to=number})) 86 | default = [{from=8082,to=8082}] 87 | description = "External Port Ranges to Expose" 88 | } 89 | variable "port_internal_range" { 90 | type = list(object({from=number,to=number})) 91 | default = [{from=8082,to=8082}] 92 | description = "Internal Port Ranges to Expose" 93 | } 94 | variable "external_access_security_group_ids" { 95 | type = list 96 | default = [] 97 | description = "Other Security Groups you will tro grant access to the externalized ports" 98 | } 99 | variable "external_access_cidrs" { 100 | type = list 101 | default = [] 102 | description = "CIDRs you will tro grant access to the externalized ports" 103 | } 104 | 105 | #Monitoring 106 | variable "monitoring_security_group_ids"{ 107 | type = list 108 | default = [] 109 | description = "Collection of Security Groups that need access to monitoring this component" 110 | } 111 | variable "monitoring_cidrs"{ 112 | type = list 113 | default = [] 114 | description = "Collection of CIDRS that need access to monitoring this component" 115 | } 116 | 117 | variable "prometheus_port" { 118 | type = number 119 | default = 8075 120 | description = "Port on which the Prometheus Agent is running" 121 | } 122 | variable "prometheus_enabled" { 123 | type = bool 124 | default = true 125 | } 126 | 127 | variable "jolokia_port" { 128 | type = number 129 | default = 7775 130 | description = "Port on which the Jolokia Agent is running" 131 | } 132 | variable "jolokia_enabled" { 133 | type = bool 134 | default = true 135 | } 136 | 137 | #User Data 138 | variable "user_data_template" { 139 | type = string 140 | default = "" 141 | description = "A Shell script to run upon instance startup" 142 | } 143 | 144 | variable "user_data_template_vars" { 145 | type = map 146 | default = {} 147 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the extra_template_vars param." 148 | } -------------------------------------------------------------------------------- /modules/schema_registry/README.md: -------------------------------------------------------------------------------- 1 | # AWS - Confluent Platform - Schema Registry Terraform Module 2 | 3 | Independent module for creating Schema Registry EC2 instances, Route53 DNS addresses, and Security Group(s). -------------------------------------------------------------------------------- /modules/schema_registry/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | aws = { 4 | source = "hashicorp/aws" 5 | version = "~> 4.5.0" 6 | configuration_aliases = [aws.dns] 7 | } 8 | template = { 9 | source = "hashicorp/template" 10 | version = "~> 2.2.0" 11 | } 12 | } 13 | } 14 | 15 | locals { 16 | component_name = "Schema Registry" 17 | component_short_name = "sr" 18 | } 19 | 20 | module "my_sec_group" { 21 | source = "../base_sec_group" 22 | 23 | providers = { 24 | aws = aws 25 | } 26 | 27 | component_name = local.component_name 28 | 29 | vpc_id = var.vpc_id 30 | enable_sg_creation = var.enable_sg_creation 31 | sg_name = var.sg_name 32 | port_external_range = var.port_external_range 33 | port_internal_range = var.port_internal_range 34 | external_access_security_group_ids = concat([ 35 | var.rest_proxy_sg_id, 36 | var.control_center_sg_id 37 | ], var.kafka_connect_sg_ids, var.ksql_sg_ids, var.external_access_security_group_ids) 38 | external_access_cidrs = var.external_access_cidrs 39 | tags = var.tags 40 | 41 | monitoring_security_group_ids = var.monitoring_security_group_ids 42 | monitoring_cidrs = var.monitoring_cidrs 43 | prometheus_enabled = var.prometheus_enabled 44 | jolokia_enabled = var.jolokia_enabled 45 | prometheus_port = var.prometheus_port 46 | jolokia_port = var.jolokia_port 47 | } 48 | 49 | module "my_instance" { 50 | source = "../base_node" 51 | 52 | providers = { 53 | aws = aws 54 | aws.dns = aws.dns 55 | } 56 | 57 | extra_template_vars = merge({ 58 | component_name = local.component_name 59 | component_short_name = local.component_short_name 60 | }, var.extra_template_vars) 61 | 62 | servers = var.servers 63 | image_id = var.image_id 64 | instance_type = var.instance_type 65 | root_volume_size = var.root_volume_size 66 | key_pair = var.key_pair 67 | tags = var.tags 68 | multi_az = var.multi_az 69 | subnet_ids = var.subnet_ids 70 | security_group_ids = concat(var.security_group_ids, [module.my_sec_group.security_group]) 71 | dns_zone_id = var.dns_zone_id 72 | dns_ttl = var.dns_ttl 73 | name_template = var.name_template 74 | dns_template = var.dns_template 75 | enable_dns_creation = var.enable_dns_creation 76 | 77 | prometheus_enabled = var.prometheus_enabled 78 | jolokia_enabled = var.jolokia_enabled 79 | prometheus_port = var.prometheus_port 80 | jolokia_port = var.jolokia_port 81 | 82 | user_data_template = var.user_data_template 83 | user_data_template_vars = var.user_data_template_vars 84 | } -------------------------------------------------------------------------------- /modules/schema_registry/outputs.tf: -------------------------------------------------------------------------------- 1 | output "instances" { 2 | value = module.my_instance.instances 3 | } 4 | 5 | output "dns_records" { 6 | value = module.my_instance.dns_records 7 | } 8 | 9 | output "security_group" { 10 | value = module.my_sec_group.security_group 11 | } -------------------------------------------------------------------------------- /modules/schema_registry/variables.tf: -------------------------------------------------------------------------------- 1 | ########################### 2 | # SchemaReg Vars 3 | ########################### 4 | variable "servers" { 5 | type = number 6 | default = 0 7 | } 8 | 9 | #Instance Related Vars 10 | variable "image_id" { 11 | type = string 12 | } 13 | variable "instance_type" { 14 | type = string 15 | default = "t3.small" 16 | } 17 | variable "root_volume_size" { 18 | type = number 19 | default = 15 20 | } 21 | variable "key_pair" { 22 | type = string 23 | } 24 | variable "tags" { 25 | type = map 26 | } 27 | variable "multi_az" { 28 | type = bool 29 | default = true 30 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 31 | } 32 | 33 | #Network Related Vars 34 | variable "subnet_ids" { 35 | type = list(string) 36 | } 37 | 38 | variable "security_group_ids" { 39 | type = list 40 | default = [] 41 | } 42 | 43 | #DNS Related Vars 44 | variable "dns_zone_id" { 45 | type = string 46 | } 47 | variable "dns_ttl" { 48 | type = string 49 | default = "300" 50 | } 51 | 52 | variable "name_template" { 53 | type = string 54 | default = "sr$${format(\"%02.0f\", itemcount)}" 55 | } 56 | variable "dns_template" { 57 | type = string 58 | default = "$${name}" 59 | } 60 | variable "enable_dns_creation" { 61 | type = bool 62 | default = true 63 | description = "Generate Route53 entries for all created resources" 64 | } 65 | variable "extra_template_vars" { 66 | type = map 67 | default = {} 68 | } 69 | 70 | #SG Related Vars 71 | variable "vpc_id" { 72 | type = string 73 | description = "VPC ID that Resources should exist in" 74 | } 75 | variable "enable_sg_creation" { 76 | type = bool 77 | default = true 78 | description = "Flag to enable the creation of needed Security Group(s)" 79 | } 80 | variable "sg_name" { 81 | type = string 82 | default = "CP_Schema_Registry" 83 | description = "Name/Name Prefix to use when creating Security Groups(s)" 84 | } 85 | variable "port_external_range" { 86 | type = list(object({from=number,to=number})) 87 | default = [{from=8081,to=8081}] 88 | description = "External Port Ranges to Expose" 89 | } 90 | variable "port_internal_range" { 91 | type = list(object({from=number,to=number})) 92 | default = [{from=8081,to=8081}] 93 | description = "Internal Port Ranges to Expose" 94 | } 95 | variable "external_access_security_group_ids" { 96 | type = list 97 | default = [] 98 | description = "Other Security Groups you will tro grant access to the externalized ports" 99 | } 100 | variable "external_access_cidrs" { 101 | type = list 102 | default = [] 103 | description = "CIDRs you will tro grant access to the externalized ports" 104 | } 105 | variable "kafka_connect_sg_ids" { 106 | type = list 107 | default = [] 108 | } 109 | variable "ksql_sg_ids" { 110 | type = list 111 | default = [] 112 | } 113 | variable "rest_proxy_sg_id" { 114 | type = string 115 | default = "" 116 | } 117 | variable "control_center_sg_id" { 118 | type = string 119 | default = "" 120 | } 121 | 122 | #Monitoring 123 | variable "monitoring_security_group_ids"{ 124 | type = list 125 | default = [] 126 | description = "Collection of Security Groups that need access to monitoring this component" 127 | } 128 | variable "monitoring_cidrs"{ 129 | type = list 130 | default = [] 131 | description = "Collection of CIDRS that need access to monitoring this component" 132 | } 133 | 134 | variable "prometheus_port" { 135 | type = number 136 | default = 8078 137 | description = "Port on which the Prometheus Agent is running" 138 | } 139 | variable "prometheus_enabled" { 140 | type = bool 141 | default = true 142 | } 143 | 144 | variable "jolokia_port" { 145 | type = number 146 | default = 7772 147 | description = "Port on which the Jolokia Agent is running" 148 | } 149 | variable "jolokia_enabled" { 150 | type = bool 151 | default = true 152 | } 153 | 154 | #User Data 155 | variable "user_data_template" { 156 | type = string 157 | default = "" 158 | description = "A Shell script to run upon instance startup" 159 | } 160 | 161 | variable "user_data_template_vars" { 162 | type = map 163 | default = {} 164 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the extra_template_vars param." 165 | } -------------------------------------------------------------------------------- /modules/zookeeper/README.md: -------------------------------------------------------------------------------- 1 | # AWS - Confluent Platform - Zookeeper Terraform Module 2 | 3 | Independent module for creating Zookeeper EC2 instances, Route53 DNS addresses, and Security Group(s). -------------------------------------------------------------------------------- /modules/zookeeper/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | aws = { 4 | source = "hashicorp/aws" 5 | version = "~> 4.5.0" 6 | configuration_aliases = [aws.dns] 7 | } 8 | template = { 9 | source = "hashicorp/template" 10 | version = "~> 2.2.0" 11 | } 12 | } 13 | } 14 | 15 | locals { 16 | component_name = "Zookeeper" 17 | component_short_name = "zk" 18 | } 19 | 20 | module "my_sec_group" { 21 | source = "../base_sec_group" 22 | 23 | providers = { 24 | aws = aws 25 | } 26 | 27 | component_name = local.component_name 28 | 29 | vpc_id = var.vpc_id 30 | enable_sg_creation = var.enable_sg_creation 31 | sg_name = var.sg_name 32 | port_external_range = var.port_external_range 33 | port_internal_range = var.port_internal_range 34 | external_access_security_group_ids = concat([ 35 | var.kafka_broker_sg_id 36 | ], var.external_access_security_group_ids) 37 | external_access_cidrs = var.external_access_cidrs 38 | tags = var.tags 39 | 40 | monitoring_security_group_ids = var.monitoring_security_group_ids 41 | monitoring_cidrs = var.monitoring_cidrs 42 | prometheus_enabled = var.prometheus_enabled 43 | jolokia_enabled = var.jolokia_enabled 44 | prometheus_port = var.prometheus_port 45 | jolokia_port = var.jolokia_port 46 | } 47 | 48 | module "my_instance" { 49 | source = "../base_node" 50 | 51 | providers = { 52 | aws = aws 53 | aws.dns = aws.dns 54 | } 55 | 56 | extra_template_vars = merge({ 57 | component_name = local.component_name 58 | component_short_name = local.component_short_name 59 | }, var.extra_template_vars) 60 | 61 | servers = var.servers 62 | image_id = var.image_id 63 | instance_type = var.instance_type 64 | root_volume_size = var.root_volume_size 65 | key_pair = var.key_pair 66 | tags = var.tags 67 | multi_az = var.multi_az 68 | subnet_ids = var.subnet_ids 69 | security_group_ids = concat(var.security_group_ids, [module.my_sec_group.security_group]) 70 | dns_zone_id = var.dns_zone_id 71 | dns_ttl = var.dns_ttl 72 | name_template = var.name_template 73 | dns_template = var.dns_template 74 | enable_dns_creation = var.enable_dns_creation 75 | 76 | prometheus_enabled = var.prometheus_enabled 77 | jolokia_enabled = var.jolokia_enabled 78 | prometheus_port = var.prometheus_port 79 | jolokia_port = var.jolokia_port 80 | 81 | user_data_template = var.user_data_template 82 | user_data_template_vars = var.user_data_template_vars 83 | 84 | ebs_volumes = concat([ 85 | {name:"transaction-log", device_name:var.vol_trans_log_device_name, encrypted:false, kms_key_id="", size:var.vol_trans_log_size, type:"gp2", tags:{}}, 86 | {name:"storage", device_name:var.vol_storage_device_name, encrypted:false, kms_key_id="", size:var.vol_storage_size, type:"gp2", tags:{}} 87 | ], var.extra_ebs_volumes) 88 | } -------------------------------------------------------------------------------- /modules/zookeeper/outputs.tf: -------------------------------------------------------------------------------- 1 | output "instances" { 2 | value = module.my_instance.instances 3 | } 4 | 5 | output "dns_records" { 6 | value = module.my_instance.dns_records 7 | } 8 | 9 | output "security_group" { 10 | value = module.my_sec_group.security_group 11 | } -------------------------------------------------------------------------------- /modules/zookeeper/variables.tf: -------------------------------------------------------------------------------- 1 | ########################### 2 | # Zookeeper Vars 3 | ########################### 4 | variable "servers" { 5 | type = number 6 | default = 1 7 | } 8 | 9 | #Instance Related Vars 10 | variable "image_id" { 11 | type = string 12 | } 13 | variable "instance_type" { 14 | type = string 15 | default = "t3.small" 16 | } 17 | variable "root_volume_size" { 18 | type = number 19 | default = 15 20 | } 21 | variable "key_pair" { 22 | type = string 23 | } 24 | variable "tags" { 25 | type = map 26 | } 27 | variable "multi_az" { 28 | type = bool 29 | default = true 30 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 31 | } 32 | 33 | #Network Related Vars 34 | variable "subnet_ids" { 35 | type = list(string) 36 | } 37 | 38 | variable "security_group_ids" { 39 | type = list 40 | default = [] 41 | } 42 | 43 | #DNS Related Vars 44 | variable "dns_zone_id" { 45 | type = string 46 | } 47 | variable "dns_ttl" { 48 | type = string 49 | default = "300" 50 | } 51 | 52 | variable "name_template" { 53 | type = string 54 | default = "zk$${format(\"%02.0f\", itemcount)}" 55 | } 56 | variable "dns_template" { 57 | type = string 58 | default = "$${name}" 59 | } 60 | variable "enable_dns_creation" { 61 | type = bool 62 | default = true 63 | description = "Generate Route53 entries for all created resources" 64 | } 65 | 66 | variable "extra_template_vars" { 67 | type = map 68 | default = {} 69 | } 70 | 71 | #SG Related Vars 72 | variable "vpc_id" { 73 | type = string 74 | description = "VPC ID that Resources should exist in" 75 | } 76 | variable "enable_sg_creation" { 77 | type = bool 78 | default = true 79 | description = "Flag to enable the creation of needed Security Group(s)" 80 | } 81 | variable "sg_name" { 82 | type = string 83 | default = "CP_Zookeeper" 84 | description = "Name/Name Prefix to use when creating Security Groups(s)" 85 | } 86 | variable "port_external_range" { 87 | type = list(object({from=number,to=number})) 88 | default = [{from=2181,to=2181}] 89 | description = "External Port Ranges to Expose" 90 | } 91 | variable "port_internal_range" { 92 | type = list(object({from=number,to=number})) 93 | default = [{from=2181,to=2181},{from=2888,to=2888},{from=3888,to=3888}] 94 | description = "Internal Port Ranges to Expose" 95 | } 96 | variable "external_access_security_group_ids" { 97 | type = list 98 | default = [] 99 | description = "Other Security Groups you will tro grant access to the externalized ports" 100 | } 101 | variable "external_access_cidrs" { 102 | type = list 103 | default = [] 104 | description = "CIDRs you will tro grant access to the externalized ports" 105 | } 106 | variable "kafka_broker_sg_id" { 107 | type = string 108 | default = "" 109 | } 110 | 111 | 112 | #EBS Volumes 113 | variable "extra_ebs_volumes" { 114 | type = list(object({ 115 | name: string, 116 | device_name: string, 117 | encrypted: bool, 118 | kms_key_id: string, 119 | size: number, 120 | type: string, 121 | tags: map(string) 122 | })) 123 | default = [] 124 | } 125 | 126 | variable "vol_trans_log_size" { 127 | type = number 128 | default = 500 129 | } 130 | variable "vol_trans_log_device_name" { 131 | type = string 132 | default = "/dev/sdf" 133 | } 134 | variable "vol_storage_size" { 135 | type = number 136 | default = 500 137 | } 138 | variable "vol_storage_device_name" { 139 | type = string 140 | default = "/dev/sdg" 141 | } 142 | 143 | #Monitoring 144 | variable "monitoring_security_group_ids"{ 145 | type = list 146 | default = [] 147 | description = "Collection of Security Groups that need access to monitoring this component" 148 | } 149 | variable "monitoring_cidrs"{ 150 | type = list 151 | default = [] 152 | description = "Collection of CIDRS that need access to monitoring this component" 153 | } 154 | 155 | variable "prometheus_port" { 156 | type = number 157 | default = 8079 158 | description = "Port on which the Prometheus Agent is running" 159 | } 160 | variable "prometheus_enabled" { 161 | type = bool 162 | default = true 163 | } 164 | 165 | variable "jolokia_port" { 166 | type = number 167 | default = 7770 168 | description = "Port on which the Jolokia Agent is running" 169 | } 170 | variable "jolokia_enabled" { 171 | type = bool 172 | default = true 173 | } 174 | 175 | #User Data 176 | variable "user_data_template" { 177 | type = string 178 | default = "" 179 | description = "A Shell script to run upon instance startup" 180 | } 181 | 182 | variable "user_data_template_vars" { 183 | type = map 184 | default = {} 185 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the extra_template_vars param." 186 | } -------------------------------------------------------------------------------- /outputs.tf: -------------------------------------------------------------------------------- 1 | output "zookeeper_instances" { 2 | value = module.cp-aws-zookeeper.instances 3 | } 4 | output "zookeeper_dns_records" { 5 | value = module.cp-aws-zookeeper.dns_records 6 | } 7 | output "zookeeper_security_group" { 8 | value = module.cp-aws-zookeeper.security_group 9 | } 10 | 11 | output "kafka_broker_instances" { 12 | value = module.cp-aws-kafka_broker.instances 13 | } 14 | output "kafka_broker_dns_records" { 15 | value = module.cp-aws-kafka_broker.dns_records 16 | } 17 | output "kafka_broker_security_group" { 18 | value = module.cp-aws-kafka_broker.security_group 19 | } 20 | 21 | output "kafka_connect_instances" { 22 | value = module.cp-aws-kafka_connect.instances 23 | } 24 | output "kafka_connect_dns_records" { 25 | value = module.cp-aws-kafka_connect.dns_records 26 | } 27 | output "kafka_connect_security_group" { 28 | value = module.cp-aws-kafka_connect.security_group 29 | } 30 | 31 | output "control_center_instances" { 32 | value = module.cp-aws-control_center.instances 33 | } 34 | output "control_center_dns_records" { 35 | value = module.cp-aws-control_center.dns_records 36 | } 37 | output "control_center_security_group" { 38 | value = module.cp-aws-control_center.security_group 39 | } 40 | 41 | output "ksql_instances" { 42 | value = module.cp-aws-ksql.instances 43 | } 44 | output "ksql_dns_records" { 45 | value = module.cp-aws-ksql.dns_records 46 | } 47 | output "ksql_security_group" { 48 | value = module.cp-aws-ksql.security_group 49 | } 50 | 51 | output "rest_proxy_instances" { 52 | value = module.cp-aws-rest_proxy.instances 53 | } 54 | output "rest_proxy_dns_records" { 55 | value = module.cp-aws-rest_proxy.dns_records 56 | } 57 | output "rest_proxy_security_group" { 58 | value = module.cp-aws-rest_proxy.security_group 59 | } 60 | 61 | output "schema_registry_instances" { 62 | value = module.cp-aws-schema_registry.instances 63 | } 64 | output "schema_registry_dns_records" { 65 | value = module.cp-aws-schema_registry.dns_records 66 | } 67 | output "schema_registry_security_group" { 68 | value = module.cp-aws-schema_registry.security_group 69 | } 70 | 71 | -------------------------------------------------------------------------------- /variables.tf: -------------------------------------------------------------------------------- 1 | variable "extra_template_vars" { 2 | type = map 3 | default = {} 4 | description = "A collection of variables to use in all template rendering." 5 | } 6 | variable "vpc_id" { 7 | type = string 8 | } 9 | variable "subnet_ids" { 10 | type = list(string) 11 | } 12 | variable "multi_az" { 13 | type = bool 14 | default = true 15 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 16 | } 17 | variable "image_id" { 18 | type = string 19 | } 20 | variable "enable_sg_creation" { 21 | type = bool 22 | default = true 23 | } 24 | variable "security_group_ids" { 25 | type = list(string) 26 | default = [] 27 | } 28 | variable "key_pair" { 29 | type = string 30 | } 31 | variable "default_tags" { 32 | type = map 33 | default = {} 34 | } 35 | variable "dns_zone_id" { 36 | type = string 37 | } 38 | variable "enable_dns_creation" { 39 | type = bool 40 | default = true 41 | description = "Generate Route53 entries for all created resources" 42 | } 43 | 44 | #Monitoring 45 | variable "monitoring_security_group_ids"{ 46 | type = list 47 | default = [] 48 | description = "Collection of Security Groups that need access to monitoring this component" 49 | } 50 | variable "monitoring_cidrs"{ 51 | type = list 52 | default = [] 53 | description = "Collection of CIDRS that need access to monitoring this component" 54 | } 55 | variable "prometheus_enabled" { 56 | type = bool 57 | default = true 58 | } 59 | variable "jolokia_enabled" { 60 | type = bool 61 | default = true 62 | } 63 | 64 | #User Data 65 | variable "user_data_template_vars" { 66 | type = map 67 | default = {} 68 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the extra_template_vars param as well as node_count, node_name, node_dns, node_devices." 69 | } 70 | 71 | 72 | ########################### 73 | # Zookeeper Vars 74 | ########################### 75 | variable "zookeeper_servers" { 76 | type = number 77 | default = 1 78 | } 79 | 80 | #Instance Related Vars 81 | variable "zookeeper_image_id" { 82 | type = string 83 | default = "" 84 | } 85 | variable "zookeeper_instance_type" { 86 | type = string 87 | default = "t3.small" 88 | } 89 | variable "zookeeper_root_volume_size" { 90 | type = number 91 | default = 15 92 | } 93 | variable "zookeeper_key_pair" { 94 | type = string 95 | default = "" 96 | } 97 | variable "zookeeper_tags" { 98 | type = map 99 | default = {} 100 | } 101 | 102 | #Network Related Vars 103 | variable "zookeeper_subnet_ids" { 104 | type = list(string) 105 | default = [] 106 | } 107 | variable "zookeeper_multi_az" { 108 | type = bool 109 | default = true 110 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 111 | } 112 | 113 | variable "zookeeper_security_group_ids" { 114 | type = list 115 | default = [] 116 | } 117 | 118 | #DNS Related Vars 119 | variable "zookeeper_dns_zone_id" { 120 | type = string 121 | default = "" 122 | } 123 | variable "zookeeper_dns_ttl" { 124 | type = string 125 | default = "300" 126 | } 127 | 128 | variable "zookeeper_name_template" { 129 | type = string 130 | default = "zk$${format('%02f', itemcount)}" 131 | } 132 | variable "zookeeper_dns_template" { 133 | type = string 134 | default = "$${name}" 135 | } 136 | variable "zookeeper_sg_name" { 137 | type = string 138 | default = "CP_Zookeeper" 139 | } 140 | variable "zookeeper_enable_sg_creation" { 141 | type = bool 142 | default = true 143 | } 144 | variable "zookeeper_enable_dns_creation" { 145 | type = bool 146 | default = true 147 | description = "Generate Route53 entries for all created resources" 148 | } 149 | variable "zookeeper_port_external_range" { 150 | type = list(object({from=number,to=number})) 151 | default = [{from=2181,to=2181}] 152 | description = "External Port Ranges to Expose" 153 | } 154 | variable "zookeeper_port_internal_range" { 155 | type = list(object({from=number,to=number})) 156 | default = [{from=2181,to=2181},{from=2888,to=2888},{from=3888,to=3888}] 157 | description = "Internal Port Ranges to Expose" 158 | } 159 | variable "zookeeper_external_access_security_group_ids" { 160 | type = list 161 | default = [] 162 | description = "Other Security Groups you will tro grant access to the externalized ports" 163 | } 164 | variable "zookeeper_external_access_cidrs" { 165 | type = list 166 | default = [] 167 | description = "CIDRs you will tro grant access to the externalized ports" 168 | } 169 | variable "kafka_broker_sg_id" { 170 | type = string 171 | default = "" 172 | } 173 | 174 | 175 | #EBS Volumes 176 | variable "zookeeper_extra_ebs_volumes" { 177 | type = list(object({ 178 | name: string, 179 | device_name: string, 180 | encrypted: bool, 181 | kms_key_id: string, 182 | size: number, 183 | type: string, 184 | tags: map(string) 185 | })) 186 | default = [] 187 | } 188 | 189 | variable "zookeeper_vol_trans_log_size" { 190 | type = number 191 | default = 500 192 | } 193 | variable "zookeeper_vol_trans_log_device_name" { 194 | type = string 195 | default = "/dev/sdf" 196 | } 197 | variable "zookeeper_vol_storage_size" { 198 | type = number 199 | default = 500 200 | } 201 | variable "zookeeper_vol_storage_device_name" { 202 | type = string 203 | default = "/dev/sdg" 204 | } 205 | 206 | #Monitoring 207 | variable "zookeeper_prometheus_port" { 208 | type = number 209 | default = 8079 210 | description = "Port on which the Prometheus Agent is running" 211 | } 212 | variable "zookeeper_jolokia_port" { 213 | type = number 214 | default = 7770 215 | description = "Port on which the Jolokia Agent is running" 216 | } 217 | 218 | #User Data 219 | variable "zookeeper_extra_template_vars" { 220 | type = map 221 | default = {} 222 | } 223 | variable "zookeeper_user_data_template" { 224 | type = string 225 | default = "" 226 | description = "A Shell script to run upon instance startup" 227 | } 228 | 229 | variable "zookeeper_user_data_template_vars" { 230 | type = map 231 | default = {} 232 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the zookeeper_extra_template_vars param as well as node_count, node_name, node_dns, node_devices." 233 | } 234 | 235 | ########################### 236 | # Broker Vars 237 | ########################### 238 | variable "kafka_broker_servers" { 239 | type = number 240 | default = 1 241 | } 242 | 243 | #Instance Related Vars 244 | variable "kafka_broker_image_id" { 245 | type = string 246 | default = "" 247 | } 248 | variable "kafka_broker_instance_type" { 249 | type = string 250 | default = "t3.medium" 251 | } 252 | variable "kafka_broker_root_volume_size" { 253 | type = number 254 | default = 30 255 | } 256 | variable "kafka_broker_key_pair" { 257 | type = string 258 | default = "" 259 | } 260 | variable "kafka_broker_tags" { 261 | type = map 262 | default = {} 263 | } 264 | 265 | #Network Related Vars 266 | variable "kafka_broker_subnet_ids" { 267 | type = list(string) 268 | default = [] 269 | } 270 | variable "kafka_broker_multi_az" { 271 | type = bool 272 | default = true 273 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 274 | } 275 | 276 | variable "kafka_broker_security_group_ids" { 277 | type = list 278 | default = [] 279 | } 280 | 281 | #DNS Related Vars 282 | variable "kafka_broker_dns_zone_id" { 283 | type = string 284 | default = "" 285 | } 286 | variable "kafka_broker_dns_ttl" { 287 | type = string 288 | default = "300" 289 | } 290 | 291 | variable "kafka_broker_name_template" { 292 | type = string 293 | default = "kfk$${format('%02f', itemcount)}" 294 | } 295 | variable "kafka_broker_dns_template" { 296 | type = string 297 | default = "$${name}" 298 | } 299 | variable "kafka_broker_sg_name" { 300 | type = string 301 | default = "CP_Kafka_Broker" 302 | } 303 | variable "kafka_broker_port_external_range" { 304 | type = list(object({from=number,to=number})) 305 | default = [{from=9091,to=9093},{from:8090,to:8091}] 306 | description = "External Port Ranges to Expose" 307 | } 308 | variable "kafka_broker_port_internal_range" { 309 | type = list(object({from=number,to=number})) 310 | default = [{from=9091,to=9092},{from:8090,to:8091}] 311 | description = "Internal Port Ranges to Expose" 312 | } 313 | variable "kafka_broker_external_access_security_group_ids" { 314 | type = list 315 | default = [] 316 | description = "Other Security Groups you will tro grant access to the externalized ports" 317 | } 318 | variable "kafka_broker_external_access_cidrs" { 319 | type = list 320 | default = [] 321 | description = "CIDRs you will tro grant access to the externalized ports" 322 | } 323 | variable "kafka_broker_enable_sg_creation" { 324 | type = bool 325 | default = true 326 | } 327 | variable "kafka_broker_enable_dns_creation" { 328 | type = bool 329 | default = true 330 | description = "Generate Route53 entries for all created resources" 331 | } 332 | 333 | 334 | #EBS Volumes 335 | variable "kafka_broker_extra_ebs_volumes" { 336 | type = list(object({ 337 | name: string, 338 | device_name: string, 339 | encrypted: bool, 340 | kms_key_id: string, 341 | size: number, 342 | type: string, 343 | tags: map(string) 344 | })) 345 | default = [] 346 | } 347 | 348 | variable "kafka_broker_vol_data_size" { 349 | type = number 350 | default = 500 351 | } 352 | variable "kafka_broker_vol_data_device_name" { 353 | type = string 354 | default = "/dev/sdf" 355 | } 356 | variable "kafka_broker_vol_data_type" { 357 | type = string 358 | default = null 359 | } 360 | 361 | #Monitoring 362 | variable "kafka_broker_prometheus_port" { 363 | type = number 364 | default = 8080 365 | description = "Port on which the Prometheus Agent is running" 366 | } 367 | variable "kafka_broker_jolokia_port" { 368 | type = number 369 | default = 7771 370 | description = "Port on which the Jolokia Agent is running" 371 | } 372 | 373 | #User Data 374 | variable "kafka_broker_extra_template_vars" { 375 | type = map 376 | default = {} 377 | } 378 | variable "kafka_broker_user_data_template" { 379 | type = string 380 | default = "" 381 | description = "A Shell script to run upon instance startup" 382 | } 383 | 384 | variable "kafka_broker_user_data_template_vars" { 385 | type = map 386 | default = {} 387 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the kafka_broker_extra_template_vars param as well as node_count, node_name, node_dns, node_devices." 388 | } 389 | 390 | ########################### 391 | # Connect Vars 392 | ########################### 393 | variable "kafka_connect_servers" { 394 | type = number 395 | default = 0 396 | } 397 | 398 | #Instance Related Vars 399 | variable "kafka_connect_image_id" { 400 | type = string 401 | default = "" 402 | } 403 | variable "kafka_connect_instance_type" { 404 | type = string 405 | default = "t3.medium" 406 | } 407 | variable "kafka_connect_root_volume_size" { 408 | type = number 409 | default = 15 410 | } 411 | variable "kafka_connect_key_pair" { 412 | type = string 413 | default = "" 414 | } 415 | variable "kafka_connect_tags" { 416 | type = map 417 | default = {} 418 | } 419 | 420 | #Network Related Vars 421 | variable "kafka_connect_subnet_ids" { 422 | type = list(string) 423 | default = [] 424 | } 425 | variable "kafka_connect_multi_az" { 426 | type = bool 427 | default = true 428 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 429 | } 430 | 431 | variable "kafka_connect_security_group_ids" { 432 | type = list 433 | default = [] 434 | } 435 | 436 | #DNS Related Vars 437 | variable "kafka_connect_dns_zone_id" { 438 | type = string 439 | default = "" 440 | } 441 | variable "kafka_connect_dns_ttl" { 442 | type = string 443 | default = "300" 444 | } 445 | 446 | variable "kafka_connect_name_template" { 447 | type = string 448 | default = "connect$${format('%02f', itemcount)}" 449 | } 450 | variable "kafka_connect_dns_template" { 451 | type = string 452 | default = "$${name}" 453 | } 454 | variable "kafka_connect_sg_name" { 455 | type = string 456 | default = "CP_Kafka_Connect" 457 | } 458 | variable "kafka_connect_enable_sg_creation" { 459 | type = bool 460 | default = true 461 | } 462 | variable "kafka_connect_enable_dns_creation" { 463 | type = bool 464 | default = true 465 | description = "Generate Route53 entries for all created resources" 466 | } 467 | variable "kafka_connect_port_external_range" { 468 | type = list(object({from=number,to=number})) 469 | default = [{from=8083,to=8083}] 470 | description = "External Port Ranges to Expose" 471 | } 472 | variable "kafka_connect_port_internal_range" { 473 | type = list(object({from=number,to=number})) 474 | default = [{from=8083,to=8083}] 475 | description = "Internal Port Ranges to Expose" 476 | } 477 | variable "kafka_connect_external_access_security_group_ids" { 478 | type = list 479 | default = [] 480 | description = "Other Security Groups you will tro grant access to the externalized ports" 481 | } 482 | variable "kafka_connect_external_access_cidrs" { 483 | type = list 484 | default = [] 485 | description = "CIDRs you will tro grant access to the externalized ports" 486 | } 487 | 488 | #EBS Volumes 489 | variable "kafka_connect_extra_ebs_volumes" { 490 | type = list(object({ 491 | name: string, 492 | device_name: string, 493 | encrypted: bool, 494 | kms_key_id: string, 495 | size: number, 496 | type: string, 497 | tags: map(string) 498 | })) 499 | default = [] 500 | } 501 | 502 | #Monitoring 503 | variable "kafka_connect_prometheus_port" { 504 | type = number 505 | default = 8077 506 | description = "Port on which the Prometheus Agent is running" 507 | } 508 | variable "kafka_connect_jolokia_port" { 509 | type = number 510 | default = 7773 511 | description = "Port on which the Jolokia Agent is running" 512 | } 513 | 514 | #User Data 515 | variable "kafka_connect_extra_template_vars" { 516 | type = map 517 | default = {} 518 | } 519 | variable "kafka_connect_user_data_template" { 520 | type = string 521 | default = "" 522 | description = "A Shell script to run upon instance startup" 523 | } 524 | 525 | variable "kafka_connect_user_data_template_vars" { 526 | type = map 527 | default = {} 528 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the kafka_connect_extra_template_vars param as well as node_count, node_name, node_dns, node_devices." 529 | } 530 | 531 | ########################### 532 | # Control Center Vars 533 | ########################### 534 | variable "control_center_servers" { 535 | type = number 536 | default = 0 537 | } 538 | 539 | #Instance Related Vars 540 | variable "control_center_image_id" { 541 | type = string 542 | default = "" 543 | } 544 | variable "control_center_instance_type" { 545 | type = string 546 | default = "t3.medium" 547 | } 548 | variable "control_center_root_volume_size" { 549 | type = number 550 | default = 15 551 | } 552 | variable "control_center_key_pair" { 553 | type = string 554 | default = "" 555 | } 556 | variable "control_center_tags" { 557 | type = map 558 | default = {} 559 | } 560 | 561 | #Network Related Vars 562 | variable "control_center_subnet_ids" { 563 | type = list(string) 564 | default = [] 565 | } 566 | variable "control_center_multi_az" { 567 | type = bool 568 | default = true 569 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 570 | } 571 | 572 | variable "control_center_security_group_ids" { 573 | type = list 574 | default = [] 575 | } 576 | 577 | #DNS Related Vars 578 | variable "control_center_dns_zone_id" { 579 | type = string 580 | default = "" 581 | } 582 | variable "control_center_dns_ttl" { 583 | type = string 584 | default = "300" 585 | } 586 | 587 | variable "control_center_name_template" { 588 | type = string 589 | default = "ccc$${format('%02f', itemcount)}" 590 | } 591 | variable "control_center_dns_template" { 592 | type = string 593 | default = "$${name}" 594 | } 595 | variable "control_center_sg_name" { 596 | type = string 597 | default = "CP_Control_Center" 598 | } 599 | variable "control_center_enable_sg_creation" { 600 | type = bool 601 | default = true 602 | } 603 | variable "control_center_enable_dns_creation" { 604 | type = bool 605 | default = true 606 | description = "Generate Route53 entries for all created resources" 607 | } 608 | variable "control_center_port_external_range" { 609 | type = list(object({from=number,to=number})) 610 | default = [{from=9021,to=9021}] 611 | description = "External Port Ranges to Expose" 612 | } 613 | variable "control_center_port_internal_range" { 614 | type = list(object({from=number,to=number})) 615 | default = [{from=9021,to=9021}] 616 | description = "Internal Port Ranges to Expose" 617 | } 618 | variable "control_center_external_access_security_group_ids" { 619 | type = list 620 | default = [] 621 | description = "Other Security Groups you will tro grant access to the externalized ports" 622 | } 623 | variable "control_center_external_access_cidrs" { 624 | type = list 625 | default = [] 626 | description = "CIDRs you will tro grant access to the externalized ports" 627 | } 628 | 629 | 630 | #EBS Volumes 631 | variable "control_center_extra_ebs_volumes" { 632 | type = list(object({ 633 | name: string, 634 | device_name: string, 635 | encrypted: bool, 636 | kms_key_id: string, 637 | size: number, 638 | type: string, 639 | tags: map(string) 640 | })) 641 | default = [] 642 | } 643 | 644 | variable "control_center_vol_data_size" { 645 | type = number 646 | default = 300 647 | } 648 | variable "control_center_vol_data_device_name" { 649 | type = string 650 | default = "/dev/sdf" 651 | } 652 | 653 | #User Data 654 | variable "control_center_extra_template_vars" { 655 | type = map 656 | default = {} 657 | } 658 | variable "control_center_user_data_template" { 659 | type = string 660 | default = "" 661 | description = "A Shell script to run upon instance startup" 662 | } 663 | 664 | variable "control_center_user_data_template_vars" { 665 | type = map 666 | default = {} 667 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the control_center_extra_template_vars param as well as node_count, node_name, node_dns, node_devices." 668 | } 669 | 670 | ########################### 671 | # KSQL Vars 672 | ########################### 673 | variable "ksql_servers" { 674 | type = number 675 | default = 0 676 | } 677 | 678 | #Instance Related Vars 679 | variable "ksql_image_id" { 680 | type = string 681 | default = "" 682 | } 683 | variable "ksql_instance_type" { 684 | type = string 685 | default = "t3.medium" 686 | } 687 | variable "ksql_root_volume_size" { 688 | type = number 689 | default = 15 690 | } 691 | variable "ksql_key_pair" { 692 | type = string 693 | default = "" 694 | } 695 | variable "ksql_tags" { 696 | type = map 697 | default = {} 698 | } 699 | 700 | #Network Related Vars 701 | variable "ksql_subnet_ids" { 702 | type = list(string) 703 | default = [] 704 | } 705 | variable "ksql_multi_az" { 706 | type = bool 707 | default = true 708 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 709 | } 710 | 711 | variable "ksql_security_group_ids" { 712 | type = list 713 | default = [] 714 | } 715 | 716 | #DNS Related Vars 717 | variable "ksql_dns_zone_id" { 718 | type = string 719 | default = "" 720 | } 721 | variable "ksql_dns_ttl" { 722 | type = string 723 | default = "300" 724 | } 725 | 726 | variable "ksql_name_template" { 727 | type = string 728 | default = "ksql$${format('%02f', itemcount)}" 729 | } 730 | variable "ksql_dns_template" { 731 | type = string 732 | default = "$${name}" 733 | } 734 | variable "ksql_sg_name" { 735 | type = string 736 | default = "CP_KSQL" 737 | } 738 | variable "ksql_enable_sg_creation" { 739 | type = bool 740 | default = true 741 | } 742 | variable "ksql_enable_dns_creation" { 743 | type = bool 744 | default = true 745 | description = "Generate Route53 entries for all created resources" 746 | } 747 | variable "ksql_port_external_range" { 748 | type = list(object({from=number,to=number})) 749 | default = [{from=8088,to=8088}] 750 | description = "External Port Ranges to Expose" 751 | } 752 | variable "ksql_port_internal_range" { 753 | type = list(object({from=number,to=number})) 754 | default = [{from=8088,to=8088}] 755 | description = "Internal Port Ranges to Expose" 756 | } 757 | variable "ksql_external_access_security_group_ids" { 758 | type = list 759 | default = [] 760 | description = "Other Security Groups you will tro grant access to the externalized ports" 761 | } 762 | variable "ksql_external_access_cidrs" { 763 | type = list 764 | default = [] 765 | description = "CIDRs you will tro grant access to the externalized ports" 766 | } 767 | 768 | 769 | #EBS Volumes 770 | variable "ksql_extra_ebs_volumes" { 771 | type = list(object({ 772 | name: string, 773 | device_name: string, 774 | encrypted: bool, 775 | kms_key_id: string, 776 | size: number, 777 | type: string, 778 | tags: map(string) 779 | })) 780 | default = [] 781 | } 782 | 783 | variable "ksql_vol_data_size" { 784 | type = number 785 | default = 100 786 | } 787 | variable "ksql_vol_data_device_name" { 788 | type = string 789 | default = "/dev/sdf" 790 | } 791 | 792 | #Monitoring 793 | variable "ksql_prometheus_port" { 794 | type = number 795 | default = 8076 796 | description = "Port on which the Prometheus Agent is running" 797 | } 798 | variable "ksql_jolokia_port" { 799 | type = number 800 | default = 7774 801 | description = "Port on which the Jolokia Agent is running" 802 | } 803 | 804 | #User Data 805 | variable "ksql_extra_template_vars" { 806 | type = map 807 | default = {} 808 | } 809 | variable "ksql_user_data_template" { 810 | type = string 811 | default = "" 812 | description = "A Shell script to run upon instance startup" 813 | } 814 | 815 | variable "ksql_user_data_template_vars" { 816 | type = map 817 | default = {} 818 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the ksql_extra_template_vars param as well as node_count, node_name, node_dns, node_devices." 819 | } 820 | 821 | ########################### 822 | # RESTProxy Vars 823 | ########################### 824 | variable "rest_proxy_servers" { 825 | type = number 826 | default = 0 827 | } 828 | 829 | #Instance Related Vars 830 | variable "rest_proxy_image_id" { 831 | type = string 832 | default = "" 833 | } 834 | variable "rest_proxy_instance_type" { 835 | type = string 836 | default = "t3.medium" 837 | } 838 | variable "rest_proxy_root_volume_size" { 839 | type = number 840 | default = 15 841 | } 842 | variable "rest_proxy_key_pair" { 843 | type = string 844 | default = "" 845 | } 846 | variable "rest_proxy_tags" { 847 | type = map 848 | default = {} 849 | } 850 | 851 | #Network Related Vars 852 | variable "rest_proxy_subnet_ids" { 853 | type = list(string) 854 | default = [] 855 | } 856 | variable "rest_proxy_multi_az" { 857 | type = bool 858 | default = true 859 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 860 | } 861 | 862 | variable "rest_proxy_security_group_ids" { 863 | type = list 864 | default = [] 865 | } 866 | 867 | #DNS Related Vars 868 | variable "rest_proxy_dns_zone_id" { 869 | type = string 870 | default = "" 871 | } 872 | variable "rest_proxy_dns_ttl" { 873 | type = string 874 | default = "300" 875 | } 876 | 877 | variable "rest_proxy_name_template" { 878 | type = string 879 | default = "rest$${format('%02f', itemcount)}" 880 | } 881 | variable "rest_proxy_dns_template" { 882 | type = string 883 | default = "$${name}" 884 | } 885 | variable "rest_proxy_sg_name" { 886 | type = string 887 | default = "CP_REST_Proxy" 888 | } 889 | variable "rest_proxy_enable_sg_creation" { 890 | type = bool 891 | default = true 892 | } 893 | variable "rest_proxy_enable_dns_creation" { 894 | type = bool 895 | default = true 896 | description = "Generate Route53 entries for all created resources" 897 | } 898 | variable "rest_proxy_port_external_range" { 899 | type = list(object({from=number,to=number})) 900 | default = [{from=8082,to=8082}] 901 | description = "External Port Ranges to Expose" 902 | } 903 | variable "rest_proxy_port_internal_range" { 904 | type = list(object({from=number,to=number})) 905 | default = [{from=8082,to=8082}] 906 | description = "Internal Port Ranges to Expose" 907 | } 908 | variable "rest_proxy_external_access_security_group_ids" { 909 | type = list 910 | default = [] 911 | description = "Other Security Groups you will tro grant access to the externalized ports" 912 | } 913 | variable "rest_proxy_external_access_cidrs" { 914 | type = list 915 | default = [] 916 | description = "CIDRs you will tro grant access to the externalized ports" 917 | } 918 | 919 | #Monitoring 920 | variable "rest_proxy_prometheus_port" { 921 | type = number 922 | default = 8075 923 | description = "Port on which the Prometheus Agent is running" 924 | } 925 | variable "rest_proxy_jolokia_port" { 926 | type = number 927 | default = 7775 928 | description = "Port on which the Jolokia Agent is running" 929 | } 930 | 931 | #User Data 932 | variable "rest_proxy_extra_template_vars" { 933 | type = map 934 | default = {} 935 | } 936 | variable "rest_proxy_user_data_template" { 937 | type = string 938 | default = "" 939 | description = "A Shell script to run upon instance startup" 940 | } 941 | 942 | variable "rest_proxy_user_data_template_vars" { 943 | type = map 944 | default = {} 945 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the rest_proxy_extra_template_vars param as well as node_count, node_name, node_dns, node_devices." 946 | } 947 | 948 | ########################### 949 | # Schema Registry Vars 950 | ########################### 951 | variable "schema_registry_servers" { 952 | type = number 953 | default = 0 954 | } 955 | 956 | #Instance Related Vars 957 | variable "schema_registry_image_id" { 958 | type = string 959 | default = "" 960 | } 961 | variable "schema_registry_instance_type" { 962 | type = string 963 | default = "t3.medium" 964 | } 965 | variable "schema_registry_root_volume_size" { 966 | type = number 967 | default = 15 968 | } 969 | variable "schema_registry_key_pair" { 970 | type = string 971 | default = "" 972 | } 973 | variable "schema_registry_tags" { 974 | type = map 975 | default = {} 976 | } 977 | 978 | #Network Related Vars 979 | variable "schema_registry_subnet_ids" { 980 | type = list(string) 981 | default = [] 982 | } 983 | variable "schema_registry_multi_az" { 984 | type = bool 985 | default = true 986 | description = "Should all the instances be proportianently spread among all the Subnets or just stay in the first subnet" 987 | } 988 | 989 | variable "schema_registry_security_group_ids" { 990 | type = list 991 | default = [] 992 | } 993 | 994 | #DNS Related Vars 995 | variable "schema_registry_dns_zone_id" { 996 | type = string 997 | default = "" 998 | } 999 | variable "schema_registry_dns_ttl" { 1000 | type = string 1001 | default = "300" 1002 | } 1003 | 1004 | variable "schema_registry_name_template" { 1005 | type = string 1006 | default = "sr$${format('%02f', itemcount)}" 1007 | } 1008 | variable "schema_registry_dns_template" { 1009 | type = string 1010 | default = "$${name}" 1011 | } 1012 | variable "schema_registry_sg_name" { 1013 | type = string 1014 | default = "CP_Schema_Registry" 1015 | } 1016 | variable "schema_registry_enable_sg_creation" { 1017 | type = bool 1018 | default = true 1019 | } 1020 | variable "schema_registry_enable_dns_creation" { 1021 | type = bool 1022 | default = true 1023 | description = "Generate Route53 entries for all created resources" 1024 | } 1025 | variable "schema_registry_port_external_range" { 1026 | type = list(object({from=number,to=number})) 1027 | default = [{from=8081,to=8081}] 1028 | description = "External Port Ranges to Expose" 1029 | } 1030 | variable "schema_registry_port_internal_range" { 1031 | type = list(object({from=number,to=number})) 1032 | default = [{from=8081,to=8081}] 1033 | description = "Internal Port Ranges to Expose" 1034 | } 1035 | variable "schema_registry_external_access_security_group_ids" { 1036 | type = list 1037 | default = [] 1038 | description = "Other Security Groups you will tro grant access to the externalized ports" 1039 | } 1040 | variable "schema_registry_external_access_cidrs" { 1041 | type = list 1042 | default = [] 1043 | description = "CIDRs you will tro grant access to the externalized ports" 1044 | } 1045 | 1046 | #Monitoring 1047 | variable "schema_registry_prometheus_port" { 1048 | type = number 1049 | default = 8078 1050 | description = "Port on which the Prometheus Agent is running" 1051 | } 1052 | variable "schema_registry_jolokia_port" { 1053 | type = number 1054 | default = 7772 1055 | description = "Port on which the Jolokia Agent is running" 1056 | } 1057 | 1058 | #User Data 1059 | variable "schema_registry_extra_template_vars" { 1060 | type = map 1061 | default = {} 1062 | } 1063 | variable "schema_registry_user_data_template" { 1064 | type = string 1065 | default = "" 1066 | description = "A Shell script to run upon instance startup" 1067 | } 1068 | 1069 | variable "schema_registry_user_data_template_vars" { 1070 | type = map 1071 | default = {} 1072 | description = "A collection of variables to give to the user data template during render. These will be in addition to the vars already passed in the schema_registry_extra_template_vars param as well as node_count, node_name, node_dns, node_devices." 1073 | } 1074 | --------------------------------------------------------------------------------