├── .gitignore ├── ARCHITECTURE.md ├── CLEANUP.md ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── DEPLOYMENT.md ├── Dockerfile ├── LICENSE ├── MAINTENANCE.md ├── NOTICE ├── POST_DEPLOYMENT.md ├── README.md ├── REQUIREMENTS.md ├── SECURITY.md ├── TROUBLESHOOTING.md ├── UPDATE.md ├── cid └── containers_cost_allocation.yaml ├── helm └── kubecost_s3_exporter │ ├── Chart.yaml │ ├── clusters_values │ └── .gitkeep │ ├── templates │ ├── cron.yaml │ └── serviceaccount.yaml │ ├── values.schema.json │ └── values.yaml ├── main.py ├── requirements.txt ├── screenshots └── architecture_diagram.png ├── terraform └── terraform-aws-cca │ ├── README.md │ ├── examples │ └── root_module │ │ ├── main.tf │ │ ├── outputs.tf │ │ ├── providers.tf │ │ ├── terraform.tfvars │ │ └── variables.tf │ ├── main.tf │ ├── modules │ ├── common_locals │ │ ├── locals.tf │ │ └── outputs.tf │ ├── kubecost_s3_exporter │ │ ├── README.md │ │ ├── locals.tf │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf │ └── pipeline │ │ ├── README.md │ │ ├── locals.tf │ │ ├── main.tf │ │ ├── outputs.tf │ │ ├── secret_policy.tpl │ │ └── variables.tf │ ├── outputs.tf │ ├── providers.tf │ ├── terraform.tfvars │ └── variables.tf └── timezones.txt /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | .DS_Store 3 | cid/cca.yaml 4 | cid/cid.log 5 | helm/kubecost_s3_exporter/clusters_values/*.yaml 6 | -------------------------------------------------------------------------------- /ARCHITECTURE.md: -------------------------------------------------------------------------------- 1 | # Architecture 2 | 3 | The following is the solution's architecture: 4 | 5 | ![Screenshot of the solution's architecture](screenshots/architecture_diagram.png) 6 | 7 | ## Solution's Components 8 | 9 | The solution is composed of the following resources (note - deployment instructions can be found later in the documentation): 10 | 11 | 1. A data collection pod (referred to as "Kubecost S3 Exporter" throughout some parts of the documentation). 12 | This project deploys a CronJob controller and a Service Account to each EKS cluster you choose. 13 | The CronJob controller creates the data collection pod on a schedule. 14 | These components are used to collect data from Kubecost. 15 | 2. A pipeline that is composed of the following components: 16 | 1. An Amazon S3 bucket that stores the Kubecost data 17 | 2. AWS Glue database, table and crawler 18 | 3. Amazon Athena workgroup 19 | 4. Relevant IAM roles and policies 20 | 5. AWS Secrets Manager secret. 21 | Optional, used for storing root CA certificate when TLS is enabled on Kubecost frontend container 22 | 3. Amazon QuickSight dashboard, data source and dataset 23 | 24 | Most of the above components are deployed using a Terraform module. 25 | The K8s resources are deployed using a Helm chart (that is invoked by the Terraform module or by the user). 26 | The QuickSight dashboard is deployed using the `cid-cmd` CLI tool. 27 | Deployment instructions can be found in the [`DEPLOYMENT.md`](DEPLOYMENT.md) file 28 | 29 | ## High-Level Logic 30 | 31 | 1. The CronJob K8s controller runs daily and creates a pod that collects cost allocation data from Kubecost. 32 | It uses the [Kubecost Allocation API](https://docs.kubecost.com/apis/apis-overview/api-allocation) to retrieve the cost allocation data. 33 | In most cases, it collects the data between 72 hours ago 00:00:00 and 48 hours ago 00:00:00. 34 | In some cases it may collect more, if it identifies gaps between the data available in Kubecost and in the S3 bucket. 35 | Once data is collected, it's then converted to a Parquet, compressed and uploaded to an S3 bucket of your choice. 36 | 2. The data is made available in Athena using AWS Glue database, table and crawler. 37 | The crawler runs daily (using a defined schedule), to create or update partitions. 38 | 3. QuickSight uses the Athena table as a data source to visualize the data. 39 | The data in the QuickSight dataset is refreshed daily according to a defined schedule. 40 | 41 | ## Authentication Logic 42 | 43 | ### IAM Roles for Service Account 44 | 45 | This solution uses IRSA for authentication with AWS resources. 46 | The authentication flow changes according to the following conditions: 47 | 48 | * The EKS cluster and the target resource (S3 bucket or Secret Manager secret) are in the same account: 49 | In this case, the Terraform module creates one IAM Role for Service Account in the EKS cluster's account. 50 | The Kubecost S3 Exporter script will assume the role and perform the relevant actions. 51 | * The EKS cluster and the target resource (S3 bucket or Secret Manager secret) are in different accounts: 52 | In this case, the Terraform module creates two IAM roles: 53 | A child IAM Role Service Account in the EKS cluster's account, and parent IAM role in the pipline account. 54 | The Kubecost S3 Exporter script will first assume the child role, then the parent role, and perform the relevant actions. 55 | This is referred to as IAM role chaining, and is used to support cross-account authentication. 56 | 57 | **_Important Note:_** 58 | The inline policy created for the IRSA includes some wildcards. 59 | The reason for using these wildcards is to specify: 60 | * All months (part of the S3 bucket prefix) 61 | * All years (part of the S3 bucket prefix) 62 | * All dates in the Parquet file name that is being uploaded to the bucket 63 | 64 | Even with these wildcards, the policy restricts access only to a very specific prefix of the bucket. 65 | This is done specifying the account ID, region and EKS cluster name as part of the resource in the inline policy. 66 | This is possible because the prefix we use in the S3 bucket includes the account and region for each cluster, and the Parquet file name includes the EKS cluster name. 67 | 68 | ### S3 Bucket Policy 69 | 70 | In addition, a sample S3 bucket policy is provided as part of this documentation. 71 | This is for the bucket that is used to store the Kubecost data. 72 | See ["Using an S3 Bucket Policy on the Kubecost Data Bucket" in the SECURITY.md file](SECURITY.md/.#using-an-s3-bucket-policy-on-the-kubecost-data-bucket). 73 | The Terraform module that's provided with this solution does not create it, because it doesn't create the S3 bucket. 74 | It's up to you to use it on your S3 bucket. 75 | 76 | ## Kubecost APIs Used by this Project 77 | 78 | This project uses the [Kubecost Allocation API](https://docs.kubecost.com/apis/apis-overview/api-allocation) to retrieve the cost allocation data 79 | 80 | ## Back-filling Past Data 81 | 82 | This solution supports back-filling past data up to the Kubecost retention limits (15 days for the free tier and EKS-optimized bundle). 83 | The back-filling is done automatically by the data collection pod if it identifies gaps in the S3 data compared to the Kubecost data. 84 | The way it works is as follows: 85 | 86 | 1. An environment variable is passed to the data collection pod (`BACKFILL_PERIOD_DAYS` in Helm, `backfill_period_days` in Terraform). 87 | The default value is 15 days (according to the Kubecost free tier and EKS-optimized bundle retention limit), but it can be changed. 88 | 2. Every time the data collection pod runs, it performs the following: 89 | 1. Identifies the available data in Kubecost for the back-fill period. 90 | This is done by querying the Allocation API for the given period, in daily granularity and `cluster` aggregation. 91 | This API call intentionally uses high granularity and high aggregation levels, because the cost data isn't the purpose of this call. 92 | The purpose of this call is to identify the dates where Kubecost data is available. 93 | 2. Identifies the available data in the S3 bucket for the back-fill period. 94 | This is done by querying Amazon S3 API for the given bucket, using the `s3:ListObjectV2` API call. 95 | The dates are then extracted from the Parquet files names. 96 | 3. The dates extracted from Kubecost Allocation API and Amazon S3 `s3:ListObjectV2` API are compared. 97 | If there are dates in the Kubecost API response that aren't available in the S3 bucket, data collection is performed from Kubecost for these dates. 98 | 99 | On a regular basis, this logic is simply used to perform the daily data collection. 100 | It'll always identify one day gap between Kubecost and S3, and will collect the missing day. 101 | However, in cases of missing data for other dates, the above logic is used to back-fill the missing data. 102 | This is instead of simply running the data collection always on a given timeframe (e.g., 3 days ago), which will only work for daily collection. 103 | This automatic back-filling solution can fit the following use-cases: 104 | 105 | 1. Back-filling data for a newly deployed data collection pod, if Kubecost was already deployed on the same cluster for multiple days 106 | 2. Back-filling data for clusters that are regularly powered off for certain days: 107 | On the days they're powered off, the data collection pod isn't running, and therefore isn't collecting the data relative to those dates (3 days back). 108 | The missing data will be automatically back-filled the next time the job runs after the cluster was powered back up. 109 | 3. Back-filling for failed jobs: 110 | It could be that the data collection pod failed for some reason, more than the maximum number of job failures. 111 | Assuming the issue is fixed within the Kubecost retention limit, the missing data will be back-filled automatically the next time the job runs successfully. 112 | 4. Back-filling for accidental deletion of Parquet files: 113 | If Parquet files within the Kubecost retention limit timeframe were accidentally deleted, the missing data will be automatically back-filled. 114 | 115 | Notes: 116 | 117 | 1. The back-filling solution supports back-filling data only up to the Kubecost retention limit (15 days for the free tier and EKS-optimized bundle) 118 | 2. The back-filling solution is automatic, and does not support force-back-filling of data that already exists in the S3 bucket. 119 | If you'd like to force-back-fill existing data, you must delete the Parquet file for the desired date, and then run the data collection (please back up first). 120 | An example reason for such a scenario is that an issue was fixed or a feature was added to the solution, and you'd like it to be applied for past data. 121 | Notice that this is possible only up to the Kubecost retention limit (15 days for the free tier and EKS-optimized bundle). 122 | -------------------------------------------------------------------------------- /CLEANUP.md: -------------------------------------------------------------------------------- 1 | # Cleanup 2 | 3 | ## QuickSight Cleanup - Resources created using `cid-cmd` CLI Tool 4 | 5 | 1. Log in to QuickSight 6 | 2. Manually delete any analysis you created from the dashboard 7 | 3. Manually delete the dashboard 8 | 9 | ## AWS and K8s Resources Cleanup 10 | 11 | 1. Follow the ["Complete Cleanup" section in the Terraform module README file](terraform/terraform-aws-cca/README.md/.#complete-cleanup) 12 | 2. Manually remove the CloudWatch Log Stream that was created by the AWS Glue Crawler 13 | 3. Manually empty and delete the S3 bucket you created, if not used for other use-cases 14 | 15 | ## Helm K8s Resources Cleanup 16 | 17 | For clusters on which the K8s resources were deployed using "Deployment Option 2", run the following Helm command per cluster: 18 | 19 | helm uninstall kubecost-s3-exporter -n --kube-context 20 | 21 | ## Remove Namespaces 22 | 23 | For each cluster, remove the namespace by running `kubectl delete ns --context ` per cluster. 24 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | -------------------------------------------------------------------------------- /DEPLOYMENT.md: -------------------------------------------------------------------------------- 1 | # Deployment 2 | 3 | Clone the repo: 4 | 5 | git clone https://github.com/awslabs/containers-cost-allocation-dashboard.git 6 | 7 | There are 3 high-level steps to deploy the solution: 8 | 9 | 1. [Build and Push the Container Image](#step-1-build-and-push-the-container-image) 10 | 2. [Deploy the AWS and K8s Resources](#step-2-deploy-the-aws-and-k8s-resources) 11 | 3. [Dashboard Deployment](#step-3-dashboard-deployment) 12 | 13 | ## Step 1: Build and Push the Container Image 14 | 15 | This project doesn't provide a public image, so you'll need to build an image and push it to the registry and repository of your choice. 16 | We recommend using Private Repository in Amazon Elastic Container Registry (ECR). 17 | You can find instructions on creating a Private Repository in ECR in [this document](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html), and pricing information can be found [here](https://aws.amazon.com/ecr/pricing/). 18 | The name for the repository can be any valid name you'd like - for example, you can use `kubecost-s3-exporter`. 19 | If you decided to use Private Repository in ECR, you'll have to configure your Docker client to log in to it first, before pushing the image to it. 20 | You can find instructions on logging in to a Private Repository in ECR using Docker client, in [this document](https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html). 21 | 22 | Note for the image build process: 23 | You might want to build for a target platform which is different from the source machine. 24 | In this case, make sure you use [QEMU emulation](https://docs.docker.com/build/building/multi-platform/#qemu). 25 | Please note that currently, the `Dockerfile` can be used to build images for `amd64` and `arm64` architectures. 26 | 27 | In this section, choose either [Build and Push for a Single Platform](#build-and-push-for-a-single-platform) or [Build and Push for Multiple Platforms](#build-and-push-for-multiple-platforms). 28 | 29 | ### Build and Push for a Single Platform 30 | 31 | Build for a platform as the source machine: 32 | 33 | docker build -t /: . 34 | 35 | Build for a specific target platform: 36 | 37 | docker build --platform linux/amd64 -t /: . 38 | 39 | Push: 40 | 41 | docker push /: 42 | 43 | ### Build and Push for Multiple Platforms 44 | 45 | docker buildx build --push --platform linux/amd64,linux/arm64/v8 --tag /: . 46 | 47 | ## Step 2: Deploy the AWS and K8s Resources 48 | 49 | This solution provides a Terraform module for deployment of both the AWS the K8s resources. 50 | There are 2 options to use it: 51 | * Deployment Option 1: Deploy both the AWS resources and the K8s resources using Terraform (K8s resources are deployed by invoking Helm) 52 | * Deployment Option 2: Deploy only the AWS resources using Terraform, and deploy the K8s resources using the `helm` command. 53 | With this option, Terraform will create a cluster-specific `values.yaml` file (with a unique name) for each cluster, which you can use. 54 | 55 | You can use a mix of these options. 56 | On some clusters, you can choose to deploy the K8s resources by having Terraform invoke Helm (the first option). 57 | On other clusters, you can choose to deploy the K8s resources yourself using the `helm` command (the second option). 58 | 59 | ### Deployment Option 1 60 | 61 | With this deployment option, Terraform deploys both the AWS resources and the K8s resources (by invoking Helm). 62 | 63 | 1. Open the [`providers.tf`](terraform/terraform-aws-cca/providers.tf) file and define the providers. 64 | Follow the sections and the comments in the file, which provide instructions. 65 | 2. Open the [`terraform.tfvars`](terraform/terraform-aws-cca/terraform.tfvars) file and provide common root module variable values. 66 | Follow the comments in the file, which provide instructions. 67 | 3. Open the [`main.tf`](terraform/terraform-aws-cca/main.tf) file and define the calling modules. 68 | Follow the sections and the comments in the file, which provide instructions. 69 | 4. Run `terraform init` 70 | 5. Run `terraform apply` 71 | 72 | If you want more detailed information, please follow the instructions in the [Terraform module README file](terraform/terraform-aws-cca/README.md). 73 | For the initial deployment, you need to go through the [Requirements](terraform/terraform-aws-cca/README.md/.#requirements), [Structure](terraform/terraform-aws-cca/README.md/.#structure) and [Initial Deployment](terraform/terraform-aws-cca/README.md/.#initial-deployment) sections. 74 | 75 | Once you're done with Terraform, continue to [step 3](#step-3-dashboard-deployment) below. 76 | 77 | ### Deployment Option 2 78 | 79 | With this deployment option, Terraform deploys only the AWS resources, and the K8s resources are deployed using the `helm` command. 80 | 81 | 1. Open the [`providers.tf`](terraform/terraform-aws-cca/providers.tf) file and define the providers. 82 | Follow the sections and the comments in the file, which provide instructions. 83 | 2. Open the [`terraform.tfvars`](terraform/terraform-aws-cca/terraform.tfvars) file and provide common root module variable values. 84 | Follow the comments in the file, which provide instructions. 85 | 3. Open the [`main.tf`](terraform/terraform-aws-cca/main.tf) file and define the calling modules. 86 | Follow the sections and the comments in the file, which provide instructions. 87 | Make sure you use `invoke_helm` input set to `false` in each cluster's calling module. 88 | 4. Run `terraform init` 89 | 5. Run `terraform apply` 90 | 91 | If you want more detailed information, please follow the instructions in the [Terraform module README file](terraform/terraform-aws-cca/README.md). 92 | For the initial deployment, you need to go through the [Requirements](terraform/terraform-aws-cca/README.md/.#requirements), [Structure](terraform/terraform-aws-cca/README.md/.#structure) and [Initial Deployment](terraform/terraform-aws-cca/README.md/.#initial-deployment) sections. 93 | 94 | After applying the Terraform configuration, a YAML file will be created per cluster, containing the Helm values for this cluster. 95 | The YAML file for each cluster will be named `___values.yaml`. 96 | The YAML files will be created in the `helm/kubecost_s3_exporter/clusters_values` directory. 97 | Then, for each cluster, deploy the K8s resources by executing Helm. 98 | Executing Helm when you're still in the Terraform module root directory: 99 | 100 | helm upgrade -i kubecost-s3-exporter ../../helm/kubecost_s3_exporter/ -n --values ../../../helm/kubecost_s3_exporter/clusters_values/.yaml --create-namespace --kube-context 101 | 102 | Executing Helm when in the `helm` directory: 103 | 104 | helm upgrade -i kubecost-s3-exporter kubecost_s3_exporter/ -n --values kubecost_s3_exporter/clusters_values/.yaml --create-namespace --kube-context 105 | 106 | Once you're done, continue to [step 3](#step-3-dashboard-deployment) below. 107 | 108 | ## Step 3: Dashboard Deployment 109 | 110 | Follow all subsections below to deploy the dashboard and be able to use it. 111 | 112 | ### Deploy the QuickSight Assets 113 | 114 | From the `cid` folder, run the following command: 115 | 116 | cid-cmd deploy --resources containers_cost_allocation.yaml --dashboard-id containers-cost-allocation --athena-database kubecost_db --quicksight-datasource-id cca --athena-workgroup primary --timezone 117 | 118 | Replace `` with a timezone from the lists in the [timezones.txt file](timezones.txt) in the project's root directory. 119 | You can also remove the `--timezone` argument, and the CLI tool will present timezones list for you. 120 | Make sure you provide credentials as environment variables or by passing `--profile_name` argument to the above command. 121 | Make sure you provide region as environment variable or by passing `--region_name` argument to the above command. 122 | The output after executing the above command, should be similar to the below: 123 | 124 | CLOUD INTELLIGENCE DASHBOARDS (CID) CLI 0.2.39 Beta 125 | 126 | Loading plugins... 127 | Core loaded 128 | 129 | 130 | Checking AWS environment... 131 | profile name: 132 | accountId: 133 | AWS userId: 134 | Region: 135 | 136 | 137 | Discovering deployed dashboards... [####################################] 100% "KPI Dashboard" (kpi_dashboard) 138 | 139 | Latest template: arn:aws:quicksight:us-east-1:223485597511:template/containers-cost-allocation/version/1 140 | Dashboard "containers-cost-allocation" is not deployed 141 | 142 | Required datasets: 143 | - cca_kubecost_view 144 | 145 | 146 | Looking by DataSetId defined in template...complete 147 | 148 | There are still 1 datasets missing: cca_kubecost_view 149 | Creating dataset: cca_kubecost_view 150 | Detected views: kubecost_view 151 | Dataset "cca_kubecost_view" created 152 | Using dataset cca_kubecost_view: 53076fa4-4238-a2e1-8672-3909f0621986 153 | Deploying dashboard containers-cost-allocation 154 | 155 | ####### 156 | ####### Congratulations! 157 | ####### Containers Cost Allocation (CCA) is available at: https://.quicksight.aws.amazon.com/sn/dashboards/containers-cost-allocation 158 | ####### 159 | 160 | ? [share-with-account] Share this dashboard with everyone in the account?: (Use arrow keys) 161 | » yes 162 | no 163 | 164 | Choose whether to share the dashboard with everyone in this account. 165 | Selecting "yes" will result in an output similar to the below: 166 | 167 | ? [share-with-account] Share this dashboard with everyone in the account?: yes 168 | Sharing complete 169 | 170 | Selecting "no" will result in an output similar to the below: 171 | 172 | ? [share-with-account] Share this dashboard with everyone in the account?: no 173 | 174 | Any of the above selections will complete the deployment. 175 | 176 | ### What Needs to Happen for Data to Appear on the Dashboard? 177 | 178 | Before you start using the dashboard, make sure the following is true: 179 | 180 | * Data must be present in the S3 bucket. 181 | For this, the Kubecost S3 Exporter container must have collected data for at least one day. 182 | Note: since it collects data from 72 hours ago 00:00:00 to 48 hours ago 00:00:00, it might find no data on new Kubecost deployments. 183 | Wait until enough data was collected by Kubecost, so that the Kubecost S3 Exporter can collect data. 184 | * The Glue crawler must have successfully run after data was already uploaded by Kubecost S3 Exporter to the S3 bucket. 185 | Note that there must not be any files in the S3 bucket, other than the ones uploaded by the Kubecost S3 Exporter. 186 | * The QuickSight dataset must have refreshed successfully after the Glue crawler ran successfully 187 | 188 | ### Save and Publish the Dashboard 189 | 190 | If you added labels to the dataset (using the `k8s_labels` and `k8s_annotations` Terraform variables): 191 | 1. Log in to QuickSight and go to the "Datasets" menu on the left 192 | 2. Click the `cca_kubecost_view` dataset, then click "EDIT DATASET" 193 | 3. On the top right, click "SAVE & PUBLISH" 194 | 4. To monitor the process, click the "QuickSight" icon on the top right. 195 | Then, go to the "Datasets" menu and click the `cca_kubecost_view` dataset again. 196 | 5. Click the "Refresh" tab. 197 | On the "History" table, you should see the most recent refresh with "Refresh type" column of "Manual, Edit". 198 | Wait until it's successfully finished, then you can start using the dashboard. 199 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | # Docker image for Kubecost S3 Exporter 2 | # This image is without shell and package managers, and contains only the app's binary and required libraries 3 | 4 | ############### 5 | # Build Stage # 6 | ############### 7 | 8 | # Using Debian image as a source to build the app binary with the required libraries and a specific Python version 9 | FROM python:3.12.1-slim-bookworm AS build 10 | 11 | # Fetching binutils which is required for building the Python binary using PyInstaller 12 | RUN set -ex \ 13 | && apt-get update \ 14 | && apt-get -y install binutils \ 15 | && apt-get -y autoremove \ 16 | && apt-get -y clean \ 17 | && rm -rf /var/lib/apt/lists/* 18 | 19 | # Uninstalling PIP so that it can be later installed as a non-root user 20 | RUN pip3 uninstall -y pip 21 | 22 | # Adding a non-root user (named "nonroot") 23 | RUN useradd -u 65532 nonroot -m 24 | USER nonroot 25 | ENV PATH="/home/nonroot/.local/bin:${PATH}" 26 | 27 | # Installing and upgrading PIP using the non-root user 28 | RUN python3 -m ensurepip --user 29 | RUN pip3 install --user --upgrade pip==23.3.1 30 | 31 | # Installing Kubecost S3 Exporter requirements and PyInstaller 32 | COPY --chown=nonroot:nonroot requirements.txt . 33 | RUN pip3 install -r requirements.txt 34 | RUN pip3 install pyinstaller==6.3.0 35 | 36 | # PIP cleanup 37 | RUN pip3 uninstall -y pip 38 | 39 | # Creating the binary 40 | ## Creating "app" directory and switching to it. This directory is used to host the application files 41 | ## Copying the main Python script 42 | ## Creating the binary using PyInstaller 43 | WORKDIR /home/nonroot/app 44 | COPY --chown=nonroot:nonroot main.py . 45 | RUN pyinstaller -F main.py --specpath . --hidden-import pyarrow.vendored.version --collect-all dateutil 46 | 47 | ################################ 48 | # Non-Root User Creation Stage # 49 | ################################ 50 | 51 | # In this stage, we create the non-root user and switch to it 52 | # We use a separate stage for this, to not duplicate code, as the user is later referenced in multiple other stages 53 | 54 | # Building the image from "scratch" (an empty image), to keep it secure, clean and minimal 55 | FROM scratch AS create_non_root_user 56 | 57 | # Copying the users/passwords file from the build stage and switching to the non-root user 58 | COPY --from=build /etc/passwd /etc/passwd 59 | USER nonroot 60 | 61 | ############################## 62 | # Copy Shared Objects Stages # 63 | ############################## 64 | 65 | # In the following stages, we copy the Shared Objects per target architecture 66 | # These stages are used as triggers, invoked later by the runtime stage according to the target archtecture 67 | # A separate stage per architecture is required, because the files names are different in each architecture 68 | 69 | # # 70 | # Copy Shared Objects Stage for amd64 Architecture # 71 | # # 72 | 73 | # Starting the Copy Shared Objects stage from the create_non_root_user stage, for amd64 architecture 74 | FROM create_non_root_user AS copy_so_amd64 75 | 76 | # Copying Shared Objects which are common to both Python and the app's binary 77 | ONBUILD COPY --from=build --chown=nonroot:nonroot /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc.so.6 78 | ONBUILD COPY --from=build --chown=nonroot:nonroot /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2 79 | 80 | # Copying Python's Shared Objects from the build stage, for amd64 architecture 81 | ONBUILD COPY --from=build --chown=nonroot:nonroot /usr/local/bin/../lib/libpython3.12.so.1.0 /usr/local/bin/../lib/libpython3.12.so.1.0 82 | ONBUILD COPY --from=build --chown=nonroot:nonroot /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libm.so.6 83 | 84 | # Copying the app binary's Shared Objects from the build stage, for amd64 architecture 85 | ONBUILD COPY --from=build --chown=nonroot:nonroot /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libdl.so.2 86 | ONBUILD COPY --from=build --chown=nonroot:nonroot /lib/x86_64-linux-gnu/libz.so.1 /lib/x86_64-linux-gnu/libz.so.1 87 | ONBUILD COPY --from=build --chown=nonroot:nonroot /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libpthread.so.0 88 | 89 | # Copying PyArrow's Shared Objects from the build stage, for amd64 architecture 90 | ONBUILD COPY --from=build --chown=nonroot:nonroot /usr/lib/x86_64-linux-gnu/librt.so.1 /usr/lib/x86_64-linux-gnu/librt.so.1 91 | 92 | # # 93 | # Copy Shared Objects Stage for arm64 Architecture # 94 | # # 95 | 96 | # Starting the Copy Shared Objects stage from the create_non_root_user stage, for arm64 architecture 97 | FROM create_non_root_user AS copy_so_arm64 98 | 99 | # Copying Shared Objects which are common to both Python and the app's binary 100 | ONBUILD COPY --from=build --chown=nonroot:nonroot /lib/aarch64-linux-gnu/libc.so.6 /lib/aarch64-linux-gnu/libc.so.6 101 | ONBUILD COPY --from=build --chown=nonroot:nonroot /lib/ld-linux-aarch64.so.1 /lib/ld-linux-aarch64.so.1 102 | 103 | # Copying Python's Shared Objects from the build stage, for arm64 architecture 104 | ONBUILD COPY --from=build --chown=nonroot:nonroot /usr/local/bin/../lib/libpython3.12.so.1.0 /usr/local/bin/../lib/libpython3.12.so.1.0 105 | ONBUILD COPY --from=build --chown=nonroot:nonroot /lib/aarch64-linux-gnu/libm.so.6 /lib/aarch64-linux-gnu/libm.so.6 106 | 107 | # Copying the app binary's Shared Objects from the build stage, for arm64 architecture 108 | ONBUILD COPY --from=build --chown=nonroot:nonroot /lib/aarch64-linux-gnu/libdl.so.2 /lib/aarch64-linux-gnu/libdl.so.2 109 | ONBUILD COPY --from=build --chown=nonroot:nonroot /lib/aarch64-linux-gnu/libz.so.1 /lib/aarch64-linux-gnu/libz.so.1 110 | ONBUILD COPY --from=build --chown=nonroot:nonroot /lib/aarch64-linux-gnu/libpthread.so.0 /lib/aarch64-linux-gnu/libpthread.so.0 111 | 112 | # Copying PyArrow's Shared Objects from the build stage, for arm64 architecture 113 | ONBUILD COPY --from=build --chown=nonroot:nonroot /usr/lib/aarch64-linux-gnu/librt.so.1 /usr/lib/aarch64-linux-gnu/librt.so.1 114 | 115 | ################# 116 | # Runtime Stage # 117 | ################# 118 | 119 | # Here we perform the final actions to prepare the runtime image 120 | # This stage is referencing the target architecture in its "FROM" instructions 121 | # This is so that the relevant target architecture's "Copy Shared Objects" stage will be invoked 122 | # When this is done, the relevant Shared Objects files for the target architecture will be copied 123 | 124 | # Setting the target architecture argument 125 | ARG TARGETARCH 126 | 127 | # Building the runtime image from the relevant "Copy Shared Objects" stage based on the target architecture 128 | FROM copy_so_$TARGETARCH 129 | 130 | # Creating the "/tmp" directory 131 | # Using "WORKDIR" is a trick, because "mkdir" command isn't available in this image 132 | # The "/tmp" directory is required for the app to store temporary files in an ephemeral volume 133 | WORKDIR /tmp 134 | 135 | # Copying the main binary 136 | ## Creating "app" directory and switching to it. This directory is used to host the application files 137 | ## Copying the main binary from the build image 138 | WORKDIR /home/nonroot/app 139 | COPY --from=build --chown=nonroot:nonroot /home/nonroot/app/dist/main . 140 | 141 | # Executing the main binary 142 | CMD ["./main"] 143 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | -------------------------------------------------------------------------------- /MAINTENANCE.md: -------------------------------------------------------------------------------- 1 | # Maintenance 2 | 3 | After the solution is initially deployed, you might want to make changes. 4 | Below are instruction for some common changes that you might do after the initial deployment. 5 | 6 | ## Deploying on Additional Clusters 7 | 8 | To add additional clusters to the dashboard, you need to add them to the Terraform module and apply it. 9 | Please follow the ["Maintenance -> Deploying on Additional Clusters" part in the Terraform module README file](terraform/terraform-aws-cca/README.md/.#deploying-on-additional-clusters). 10 | Wait for the next schedule of the Kubecost S3 Exporter and QuickSight refresh, so that it'll collect the new data. 11 | 12 | Alternatively, you can run the Kubecost S3 Exporter on-demand according to [Running the Kubecost S3 Exporter Pod On-Demand](#running-the-kubecost-s3-exporter-pod-on-demand) section. 13 | Then, manually run the Glue Crawler and manually refresh the QuickSight dataset. 14 | 15 | ## Removing Kubecost S3 Exporter from Specific Clusters 16 | 17 | Please follow the ["Cleanup -> Removing Kubecost S3 Exporter from Specific Clusters" part in the Terraform module README file](terraform/terraform-aws-cca/README.md/.#removing-kubecost-s3-exporter-from-specific-clusters). 18 | 19 | ## Adding/Removing Labels/Annotations to/from the Dataset 20 | 21 | After the initial deployment, you might want to add or remove labels or annotations for some or all clusters, to/from the dataset. 22 | To do this, perform the following: 23 | 24 | 1. Add/remove the labels/annotations to/from the Terraform module and apply it. 25 | Please follow the ["Maintenance -> Adding/Removing Labels/annotations to/from the Dataset" part in the Terraform module README file](terraform/terraform-aws-cca/README.md/.#addingremoving-labelsannotations-tofrom-the-dataset). 26 | 2. Wait for the next Kubecost S3 Exporter schedule so that it'll collect the labels/annotations. 27 | Alternatively, you can run the Kubecost S3 Exporter on-demand according to [Running the Kubecost S3 Exporter Pod On-Demand](#running-the-kubecost-s3-exporter-pod-on-demand) section. 28 | 29 | **_Note about annotations:_** 30 | 31 | While K8s labels are included by default in Kubecost Allocation API response, K8s annotations aren't. 32 | To include K8s annotations in the Kubecost Allocation API response, follow [this document](https://docs.kubecost.com/install-and-configure/advanced-configuration/annotations). 33 | 34 | ## Running the Kubecost S3 Exporter Pod On-Demand 35 | 36 | In some cases, you'd like to run the Kubecost S3 Exporter pod on-demand. 37 | For example, you may want to test it, or you may have added some data and would like to see it immediately. 38 | To run the Kubecost S3 Exporter pod on-demand, run the following command (replace `` with your namespace and `` with your cluster context: 39 | 40 | kubectl create job --from=cronjob/kubecost-s3-exporter kubecost-s3-exporter1 -n --context 41 | 42 | You can see the status by running `kubectl get all -n --context `. 43 | 44 | Please note that due to the automatic back-filling solution, you can't run the data collection on-demand for data that already exists in S3. 45 | If data already exists for the date you'd like to run the collection for, you must delete the Parquet file for this date first. 46 | This will trigger the automatic back-filling solution to identify the missing date and back-fill it, when you run the data collection. 47 | Notice that this is possible only up to the Kubecost retention limit (15 days for the free tier, 30 days for the business tier). 48 | 49 | ## Getting Logs from the Kubecost S3 Exporter Pod 50 | 51 | To see the logs of the Kubecost S3 Exporter pod, you need to first get the list of pods by running the following command: 52 | 53 | kubectl get all -n --context 54 | 55 | Then, run the following command to get the logs: 56 | 57 | kubectl logs -c kubecost-s3-exporter -n --context 58 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Containers Cost Allocation (CCA) Dashboard 2 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | -------------------------------------------------------------------------------- /POST_DEPLOYMENT.md: -------------------------------------------------------------------------------- 1 | # Post-Deployment 2 | 3 | This document includes post-deployment steps. 4 | All sections are optional, but it's advised to review them. 5 | 6 | ## Share the Dashboard with Users 7 | 8 | To share the dashboard with users, for them to be able to view it and create Analysis from it, see [this link](https://catalog.workshops.aws/awscid/en-US/dashboards/share). 9 | 10 | ## Create an Analysis from the Dashboard 11 | 12 | Create an Analysis from the Dashboard, to edit it and create custom visuals: 13 | 14 | 1. Log in to QuickSight, navigate to "Dashboards" and click the `Containers Cost Allocation (CCA)` dashboard 15 | 2. Go to step 9 in [this link](https://catalog.workshops.aws/awscid/en-US/dashboards/share) to allow "Save as" functionality for your user. 16 | Once done, go back to the dashboard and refresh the page (it's required to see the "Save as" icon). 17 | 3. On the top right part of the dashboard, click the "Save as" icon, name the Analysis, then click "SAVE". 18 | You'll now be navigated to the Analysis. 19 | 4. You can edit the Analysis as you wish, and save it again as a dashboard, by clicking the "Share" icon on the top right, then click "Publish dashboard". 20 | Please note that if you customize the dashboard, and would later like to update to a new dashboard version, you must maintain a separate dashboard. 21 | You can update a non-customized dashboard, then apply your customizations again on it. 22 | 23 | ## Tuning Schedules 24 | 25 | All components in this solution that are running on schedule, have default schedule set, to simplify the deployment. 26 | Following are the schedules: 27 | 28 | * Kubecost S3 Exporter CronJob default schedule: 00:00:00 UTC, daily 29 | * Glue crawler schedule: 01:00:00 UTC, daily 30 | * QuickSight dataset refresh schedule varies (changes in different deployments). 31 | The timezone for the QuickSight dataset refresh schedule is set based your selection when deploying the dashboard. 32 | 33 | The Kubecost S3 Exporter CronJob schedule and the Glue crawler schedule, are based on cron expressions. 34 | Since cron expressions are always in UTC, the result may be that some components may be scheduled to run in a non-ideal order. 35 | This may result in data being available in the QuickSight dashboard, only 24 hours after it was uploaded to the S3 bucket. 36 | The most ideal schedule order is as follows: 37 | 38 | 1. All Kubecost S3 Exporter CronJobs on all clusters should run in a chosen schedule, possibly close to each other 39 | 2. The Glue crawler should run after all Kubecost S3 Exporter CronJobs finished running (possibly 1 hour gap would be a good idea) 40 | 3. The QuickSight dataset refresh should run after the Glue crawler finished running (possibly 1 hour gap would be a good idea) 41 | 42 | It's advised to adjust these schedules as instructed above, using the relevant Terraform variables: 43 | 44 | * For the Kubecost S3 Exporter CronJob schedule: 45 | Adjust the `kubecost_s3_exporter_cronjob_schedule` variable in the `kubecost_s3_exporter` Terraform reusable module. 46 | See [the `kubecost_s3_exporter` Terraform reusable module's variables.tf file](terraform/terraform-aws-cca/modules/kubecost_s3_exporter/variables.tf) for more information on this variable. 47 | * For the Glue crawler schedule: 48 | Adjust the `glue_crawler_schedule` variable in the `pipeline` module. 49 | See [the `pipeline` Terraform reusable module's variables.tf file](terraform/terraform-aws-cca/modules/pipeline/variables.tf) for more information on this variable. 50 | * For the QuickSight dataset refresh schedule: 51 | Follow [this document](https://docs.aws.amazon.com/quicksight/latest/user/refreshing-imported-data.html#schedule-data-refresh). 52 | Make sure you select "Full refresh". 53 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Containers Cost Allocation (CCA) Dashboard 2 | 3 | Welcome! 4 | This repository contains a project for visualizing data from [Kubecost](https://www.kubecost.com/) in Amazon QuickSight, as part of [CID (Cloud Intelligence Dashboards)](https://catalog.workshops.aws/awscid/en-US). 5 | The dashboard provides visibility into EKS in-cluster cost and usage in a multi-cluster environment, using data from a [self-hosted Kubecost pod](https://www.kubecost.com/products/self-hosted). 6 | 7 | This project can work with any Kubecost tier: 8 | 9 | * Kubecost EKS-optimized bundle 10 | * Kubecost free tier 11 | * Kubecost enterprise tier, with the following limitations: 12 | * Data for all clusters are included in a single file per day instead of file per cluster per day 13 | * AWS account ID will not be shown 14 | * The `properties.eksclustername` dataset field will show the primary cluster name for any cluster. 15 | Instead, you can customize the dashboard and use the `properties.cluster` field 16 | 17 | Please note that [OpenCost](https://www.opencost.io/) is not supported. 18 | 19 | More information on the Kubecost EKS-optimized bundle can be found in the below resources: 20 | 21 | * [Launch blog post](https://aws.amazon.com/blogs/containers/aws-and-kubecost-collaborate-to-deliver-cost-monitoring-for-eks-customers/) 22 | * [AMP integration blog post](https://aws.amazon.com/blogs/mt/integrating-kubecost-with-amazon-managed-service-for-prometheus/) 23 | * [Multi-cluster visibility blog post](https://aws.amazon.com/blogs/containers/multi-cluster-cost-monitoring-using-kubecost-with-amazon-eks-and-amazon-managed-service-for-prometheus/) 24 | * [Amazon Cognito integration blog post](https://aws.amazon.com/blogs/containers/securing-kubecost-access-with-amazon-cognito/) 25 | * [EKS user guide](https://docs.aws.amazon.com/eks/latest/userguide/cost-monitoring.html) 26 | * [Kubecost on the AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-asiz4x22pm2n2?sr=0-2&ref_=beagle&applicationId=AWSMPContessa) 27 | * [EKS Blueprints Add-on](https://aws-quickstart.github.io/cdk-eks-blueprints/addons/kubecost/) 28 | * [EKS Add-on](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html#workloads-add-ons-available-vendors) 29 | 30 | More information on Kubecost pricing can be found [here](https://www.kubecost.com/pricing). 31 | Feature comparison between Kubecost tiers can be found [here](https://docs.kubecost.com/architecture/opencost-product-comparison). 32 | 33 | ## Architecture 34 | 35 | The following is the solution's architecture: 36 | 37 | ![Screenshot of the solution's architecture](screenshots/architecture_diagram.png) 38 | 39 | This solution is composed of the following components (in high-level): 40 | 1. A data collection pod (referred to as "Kubecost S3 Exporter" throughout some parts of the documentation). 41 | It's used to collect the data from Kubecost and upload it to an S3 bucket that you own. 42 | 2. A pipeline that makes the data available to be queried in Athena 43 | 3. A QuickSight dashboard, along with its QuickSight assets (data source, dataset) 44 | 45 | The AWS resources in this solution are deployed using a Terraform module. 46 | The K8s resources in this solution are deployed using a Helm chart. 47 | It's invoked by Terraform by default, but you can choose to deploy it yourself. 48 | The QuickSight dashboard is deployed using the `cid-cmd` CLI tool. 49 | 50 | More detailed information on the architecture and logic are found in the [`ARCHITECTURE.md`](ARCHITECTURE.md) file. 51 | Before proceeding to the requirements and deployment of this solution, it's highly recommended you review. 52 | For information related to security in this project, please refer to the [`SECURITY.md`](SECURITY.md) file. 53 | 54 | ## Requirements 55 | 56 | Before proceeding to the deployment of this solution, please complete the requirements, as outlined in the [`REQUIREMENTS.md`](REQUIREMENTS.md) file. 57 | 58 | ## Deployment 59 | 60 | For instructions on deploying this solution, please refer to the [`DEPLOYMENT.md`](DEPLOYMENT.md) file. 61 | When done, proceed to the [Post-Deployment Steps](#post-deployment-steps) section below. 62 | 63 | ## Post-Deployment Steps 64 | 65 | Before proceeding to use the dashboard, please complete the post-deployment steps outlined in the [`POST_DEPLOYMENT.md`](POST_DEPLOYMENT.md) file. 66 | In addition, if you'd like to deploy the Kubecost-S3-Exporter on additional clusters: 67 | See the ["Deploying on Additional Clusters" section in the MAINTENANCE.md file](MAINTENANCE.md/.#deploying-on-additional-clusters). 68 | If you'd like to add/remove K8s labels/annotations to/from the dataset: 69 | See the ["Adding/Removing Labels/Annotations to/from the Dataset" section in the MAINTENANCE.md file](MAINTENANCE.md/.#addingremoving-labelsannotations-tofrom-the-dataset). 70 | 71 | ## Update the Solution 72 | 73 | For update instructions, please refer to the [`UPDATE.md`](UPDATE.md) file. 74 | 75 | ## Maintenance 76 | 77 | For instructions on common maintenance tasks related to this solution, please refer to the [`MAINTENANCE.md`](MAINTENANCE.md) file. 78 | 79 | ## Troubleshooting 80 | 81 | For instructions on troubleshooting common issues related to this solution, please refer to the [`TROUBLESHOOTING.md`](TROUBLESHOOTING.md) file. 82 | 83 | ## Cleanup 84 | 85 | For instructions on cleanup, please refer to the [`CLEANUP.md`](CLEANUP.md) file. 86 | 87 | ## Security 88 | 89 | See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information on how to report security issues. 90 | See [SECURITY.md](SECURITY.md) for more information related to security in this solution. 91 | 92 | ## License 93 | 94 | This project is licensed under the Apache-2.0 License. 95 | -------------------------------------------------------------------------------- /REQUIREMENTS.md: -------------------------------------------------------------------------------- 1 | # Requirements 2 | 3 | Following are the requirements before deploying this solution: 4 | 5 | * An S3 bucket, which will be used to store the Kubecost data. 6 | It is not created by the Terraform module, you need to create it in advance. 7 | * Athena workgroup (you can also use the default `primary` workgroup) 8 | * An S3 bucket to be used for the Athena workgroup query results location (see [detailed instructions](#athena-requirements)) 9 | * QuickSight Enterprise Edition, with the following (see [detailed instructions](#quicksight-requirements)): 10 | * Permissions to access the Kubecost S3 bucket and the Athena query results location S3 bucket 11 | * Enough SPICE capacity 12 | * For each EKS cluster, have the following: 13 | * An [IAM OIDC Provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html). 14 | The IAM OIDC Provider must be created in the EKS cluster's account. 15 | * Kubecost deployed in the EKS cluster. In addition, the following is optional but recommended: 16 | * The get the most accurate cost data from Kubecost (such as RIs, SPs and Spot), [integrate it with CUR](https://docs.kubecost.com/install-and-configure/install/cloud-integration/aws-cloud-integrations) and [Spot Data Feed](https://docs.kubecost.com/install-and-configure/install/cloud-integration/aws-cloud-integrations/aws-spot-instances). 17 | * To get accurate network costs from Kubecost, please follow the [Kubecost network cost allocation guide](https://docs.kubecost.com/using-kubecost/getting-started/cost-allocation/network-allocation) and deploy [the network costs DaemonSet](https://docs.kubecost.com/install-and-configure/advanced-configuration/network-costs-configuration). 18 | * To see K8s annotations in each allocation, you must [enable Annotation Emission in Kubecost](https://docs.kubecost.com/install-and-configure/advanced-configuration/annotations) 19 | * To see node-related data for each allocation, [add node labels in Kubecost's values.yaml](https://github.com/kubecost/cost-analyzer-helm-chart/blob/develop/cost-analyzer/values.yaml). 20 | See more information in the [Adding Node Labels](#adding-node-labels) section 21 | * Terraform 1.3.x or higher 22 | * Helm 3.x or higher 23 | * The `cid-cmd` tool ([install with PIP](https://pypi.org/project/cid-cmd/)) 24 | 25 | Please continue reading the below sections. 26 | They include more detailed instructions for some of the above requirements. 27 | 28 | ## Kubecost S3 Exporter Container 29 | 30 | ### Setting Requests and Limits 31 | 32 | As explained in [`ARCHITECTURE.md`](ARCHITECTURE.md), this solution deploys a container that collects data from Kubecost. 33 | This container currently doesn't have requests and limits set. 34 | It's highly advised that you first test it in a dev/QA environment that is similar to your production environment. 35 | During testing, monitor the CPU and RAM usage of the container, and set the requests and limits accordingly. 36 | 37 | ## Kubecost Requirements 38 | 39 | ### Adding Node Labels 40 | 41 | The QuickSight dashboard includes the capability to group and filter allocations by node-related data. 42 | This data is based on K8s node labels, which must be available in the Kubecost Allocation API. 43 | The following node labels are support by the QuickSight dashboard: 44 | 45 | node.kubernetes.io/instance-type 46 | topology.kubernetes.io/region 47 | topology.kubernetes.io/zone 48 | kubernetes.io/arch 49 | kubernetes.io/os 50 | eks.amazonaws.com/nodegroup 51 | eks.amazonaws.com/nodegroup_image 52 | eks.amazonaws.com/capacityType 53 | karpenter.sh/capacity-type 54 | karpenter.sh/provisioner-name 55 | karpenter.k8s.aws/instance-ami-id 56 | 57 | However, by default, the Kubecost Allocation API response only includes a few specific node labels. 58 | For the QuickSight dashboard to support all of the above node labels, you must add them to [Kubecost values.yaml](https://github.com/kubecost/cost-analyzer-helm-chart/blob/develop/cost-analyzer/values.yaml). 59 | 60 | Here's an example of adding these node labels using `--set` option when running `helm upgrade -i`: 61 | 62 | helm upgrade -i kubecost \ 63 | oci://public.ecr.aws/kubecost/cost-analyzer --version 1.103.4 \ 64 | --namespace kubecost \ 65 | -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/develop/cost-analyzer/values-eks-cost-monitoring.yaml \ 66 | --set networkCosts.enabled=true \ 67 | --set networkCosts.config.services.amazon-web-services=true \ 68 | --set kubecostModel.allocation.nodeLabels.includeList="node.kubernetes.io\/instance-type\,topology.kubernetes.io\/region\,topology.kubernetes.io\/zone\,kubernetes.io\/arch\,kubernetes.io\/os\,eks.amazonaws.com\/nodegroup\,eks.amazonaws.com\/nodegroup_image\,eks.amazonaws.com\/capacityType\,karpenter.sh\/capacity-type\,karpenter.sh\/provisioner-name\,karpenter.k8s.aws\/instance-ami-id" 69 | 70 | ## Athena Requirements 71 | 72 | 1. Create an S3 bucket that will be used for the Athena workgroup query results location. 73 | It must be different from the S3 bucket that you're using to store the Kubecost data. 74 | It must be in the same region as the QuickSight region where you plan to deploy the dashboard. 75 | 2. Create an Athena workgroup if you don't want to use the default `primary` workgroup. 76 | It must be in the same region as the QuickSight region where you plan to deploy the dashboard. 77 | Follow [this document](https://docs.aws.amazon.com/athena/latest/ug/workgroups-create-update-delete.html#creating-workgroups) for instructions. 78 | 3. Whether you decided to use the default `primary` workgroup or created a new one: 79 | You must set an Athena query results location for the workgroup. 80 | Follow [this document](https://docs.aws.amazon.com/athena/latest/ug/querying.html#query-results-specify-location-workgroup) for instructions. 81 | It's advised that as part of the settings, you choose to encrypt the query results. 82 | 83 | ## QuickSight Requirements 84 | 85 | ### Configure QuickSight Permissions for the S3 Buckets 86 | 87 | 1. Navigate to “Manage QuickSight → Security & permissions” 88 | 2. Under “Access granted to X services”, click “Manage” 89 | 3. Under “S3 Bucket”: 90 | 1. Check the checkbox for the S3 bucket you created for the Kubecost data. 91 | You only need to check the checkbox next to the S3 bucket name for this bucket. 92 | No need to check the checkbox under "Write permission for Athena Workgroup" for this bucket. 93 | 2. Check the checkbox for the S3 bucket you created for the Athena workgroup query results location. 94 | Make sure you check both the checkbox next to the S3 bucket name and the checkbox under "Write permission for Athena Workgroup". 95 | 4. Click "Finish" and "Save". 96 | 97 | ### Verifying Enough QuickSight SPICE Capacity 98 | 99 | Make sure you have enough QuickSight SPICE capacity to create the QuickSight dataset and store the data. 100 | The required capacity depends on the size of your EKS clusters from which Kubecost data is collected. 101 | You may start with small SPICE capacity and adjust as needed. 102 | Make sure it's at least larger than 0, so Terraform can create the QuickSight dataset. 103 | Make sure that you purchase SPICE capacity in the region where you intend to deploy the dashboard. 104 | 105 | To add SPICE capacity, follow [this document](https://docs.aws.amazon.com/quicksight/latest/user/managing-spice-capacity.html). 106 | -------------------------------------------------------------------------------- /TROUBLESHOOTING.md: -------------------------------------------------------------------------------- 1 | # Troubleshooting 2 | 3 | This document includes some common issues and possible solutions. 4 | 5 | ## Kubecost S3 Exporter Data Collection Issues 6 | 7 | This section includes common issues related to the Kubecost S3 Exporter. 8 | 9 | ### The Data Collection Pod is in Status of `Completed`, But There's No Data in the S3 Bucket 10 | 11 | The data collection container collects data between 72 hours ago 00:00:00.000 and 48 hours ago 00:00:00.000. 12 | Your Kubecost server still have missing data in this timeframe. 13 | Please check the data collection container logs, and if you see the below message, it means you still don't have enough data: 14 | 15 | ERROR kubecost-s3-exporter: API response appears to be empty. 16 | This script collects data between 72 hours ago and 48 hours ago. 17 | Make sure that you have data at least within this timeframe. 18 | 19 | In this case, please wait for Kubecost to collect data for 72 hours ago, and then check again. 20 | 21 | ### The Data Pod Container is in Status of `Error` 22 | 23 | This could be for various reasons. 24 | Below are a couple of scenarios caught by the data collection container, and their logs you should expect to see. 25 | 26 | #### A Connection Establishment Timeout 27 | 28 | In case of a connection establishment timeout, the container logs will show the following log: 29 | 30 | ERROR kubecost-s3-exporter: Timed out waiting for TCP connection establishment in the given time ({connection_timeout}s). Consider increasing the connection timeout value. 31 | 32 | In this case, please check the following: 33 | 34 | 1. That you specified the correct Kubecost API endpoint in the `kubecost_api_endpoint` input. 35 | This should be the Kubecost cost-analyzer service. 36 | Usually, you should be able to specify `http://.:[port]`, and this DNS name will be resolved. 37 | The default service name for Kubecost cost-analyzer service is `kubecost-cost-analyzer`, and the default namespace it's created in is `kubecost`. 38 | The default port the Kubecost cost-analyzer service listens on is TCP 9090. 39 | Unless you changed the namespace, service name or port, you should be good with the default value of the `kubecost_api_endpoint` input. 40 | If you changed any of the above, make sure you change the `kubecost_api_endpoint` input value accordingly. 41 | 2. If the `kubecost_api_endpoint` input has the correct value, try increasing the `connection_timeout` input value 42 | 3. If you still get the same error, check network connectivity between the data collection pod and the Kubecost cost-analyzer service 43 | 44 | #### An HTTP Server Response Timeout 45 | 46 | In case of HTTP server response timeout, the container logs will show one of the following logs (depends on the API being queried): 47 | 48 | ERROR kubecost-s3-exporter: Timed out waiting for Kubecost Allocation On-Demand API to send an HTTP response in the given time ({read_timeout}s). Consider increasing the read timeout value. 49 | 50 | ERROR kubecost-s3-exporter: Timed out waiting for Kubecost Assets API to send an HTTP response in the given time ({read_timeout}s). Consider increasing the read timeout value. 51 | 52 | If this is for the Allocation On-Demand API call, please follow the recommendations in the "Clarifications on the Allocation On-Demand API" part on the Appendix. 53 | If this is for the Assets API call, please try increasing the `kubecost_assets_api_read_timeout` input value. 54 | -------------------------------------------------------------------------------- /UPDATE.md: -------------------------------------------------------------------------------- 1 | # Update the Solution 2 | 3 | ## Updates to the Kubecost S3 Exporter 4 | 5 | The following are considered updates to the Kubecost S3 Exporter: 6 | 7 | * Updates to the Kubecost S3 Exporter Python script 8 | * Updates to the Python dependencies of the Kubecost S3 Exporter Python script 9 | * Updates to the content of the `Dockerfile` 10 | 11 | When any of the above is changed, follow the below steps to perform an update: 12 | 13 | 1. Build and push the Docker image. 14 | You can follow the same steps as in [Step 1: Build and Push the Container Image in the `DEPLOYMENT.md` file](DEPLOYMENT.md/.#step-1-build-and-push-the-container-image). 15 | 2. Depending on how you deployed the K8s resources, choose one of the below: 16 | 1. If you used [Deployment Option 1](DEPLOYMENT.md/.#deployment-option-1): 17 | Run `terraform apply` on the Terraform module, from the root directory of the Terraform module. 18 | 2. If you used [Deployment Option 2](DEPLOYMENT.md/.#deployment-option-2): 19 | Run `helm upgrade` on the new Helm chart. 20 | 21 | ## Updates to the Helm Chart 22 | 23 | In case of updates to the Helm chart only, run `helm upgrade` 24 | 25 | ## Updates to Resources Created by the Terraform Module 26 | 27 | In case of updates to resources created by the Terraform module: 28 | Running `terraform apply` will detect the changes, show you what's changed, and then you can apply the changes. 29 | 30 | ## Updates to the Dashboard 31 | 32 | Note for users who customized the dashboard: 33 | Make sure to keep a copy of the customized dashboard, update the original one, and merge with your customizations. 34 | 35 | In case of a new dashboard version, run the following command from the `cid` folder: 36 | 37 | cid-cmd update --resources containers_cost_allocation.yaml --dashboard-id containers-cost-allocation 38 | 39 | Make sure you provide credentials as environment variables or by passing `--profile_name` argument to the above command. 40 | Make sure you provide region as environment variable or by passing `--region_name` argument to the above command. 41 | The output after executing the above command, should be similar to the below: 42 | 43 | CLOUD INTELLIGENCE DASHBOARDS (CID) CLI 0.2.39 Beta 44 | 45 | Loading plugins... 46 | Core loaded 47 | 48 | 49 | Checking AWS environment... 50 | profile name: 51 | accountId: 52 | AWS userId: 53 | Region: 54 | 55 | 56 | Discovering deployed dashboards... [####################################] 100% "KPI Dashboard" (kpi_dashboard) 57 | 58 | Latest template: arn:aws:quicksight:us-east-1:223485597511:template/containers-cost-allocation/version/2 59 | An update is available: 60 | Deployed -> Latest 61 | Version v0.3.2 v0.3.3 62 | VersionId 1 2 63 | Using dataset cca_kubecost_view: 53076fa4-4238-a2e1-8672-3909f0621986 64 | 65 | Checking for updates... 66 | Deployed template: arn:aws:quicksight:us-east-1:223485597511:template/containers-cost-allocation/version/1 67 | Latest template: arn:aws:quicksight:us-east-1:223485597511:template/containers-cost-allocation/version/2 68 | 69 | Updating containers-cost-allocation 70 | Update completed 71 | 72 | ####### 73 | ####### Containers Cost Allocation (CCA) is available at: https://.quicksight.aws.amazon.com/sn/dashboards/containers-cost-allocation 74 | ####### 75 | 76 | If there's no updated version of the dashboard, that output should be similar to the below: 77 | 78 | CLOUD INTELLIGENCE DASHBOARDS (CID) CLI 0.2.39 Beta 79 | 80 | Loading plugins... 81 | Core loaded 82 | 83 | 84 | Checking AWS environment... 85 | profile name: 86 | accountId: 87 | AWS userId: 88 | Region: 89 | 90 | 91 | Discovering deployed dashboards... [####################################] 100% "KPI Dashboard" (kpi_dashboard) 92 | 93 | Latest template: arn:aws:quicksight:us-east-1:223485597511:template/containers-cost-allocation/version/1 94 | You are up to date! 95 | Version v0.3.2 96 | VersionId 1 97 | Using dataset cca_kubecost_view: 53076fa4-4238-a2e1-8672-3909f0621986 98 | 99 | Checking for updates... 100 | Deployed template: arn:aws:quicksight:us-east-1:223485597511:template/containers-cost-allocation/version/1 101 | Latest template: arn:aws:quicksight:us-east-1:223485597511:template/containers-cost-allocation/version/1 102 | 103 | ? [confirm-update] No updates available, should I update it anyway?: (Use arrow keys) 104 | » yes 105 | no 106 | 107 | Select "no", and upon selection, you should see output similar to the below: 108 | 109 | ? [confirm-update] No updates available, should I update it anyway?: no 110 | 111 | If you need to force update, select "yes", and you should see an output similar to the below: 112 | 113 | ? [confirm-update] No updates available, should I update it anyway?: yes 114 | 115 | Updating containers-cost-allocation 116 | Update completed 117 | 118 | ####### 119 | ####### Containers Cost Allocation (CCA) is available at: https://eu-north-1.quicksight.aws.amazon.com/sn/dashboards/containers-cost-allocation 120 | ####### 121 | -------------------------------------------------------------------------------- /cid/containers_cost_allocation.yaml: -------------------------------------------------------------------------------- 1 | dashboards: 2 | CONTAINERS COST ALLOCATION (CCA): 3 | dependsOn: 4 | datasets: 5 | - cca_kubecost_view 6 | name: Containers Cost Allocation (CCA) 7 | dashboardId: containers-cost-allocation 8 | category: Custom 9 | templateId: containers-cost-allocation 10 | datasets: 11 | cca_kubecost_view: 12 | data: 13 | DataSetId: 53076fa4-4238-a2e1-8672-3909f0621986 14 | Name: cca_kubecost_view 15 | PhysicalTableMap: 16 | 476e7a0e-2e42-df7a-ff77-1555e92677fe: 17 | RelationalTable: 18 | DataSourceArn: ${athena_datasource_arn} 19 | Catalog: AwsDataCatalog 20 | Schema: ${athena_database_name} 21 | Name: kubecost_view 22 | InputColumns: 23 | - Name: name 24 | Type: STRING 25 | - Name: window.start 26 | Type: DATETIME 27 | - Name: window.end 28 | Type: DATETIME 29 | - Name: minutes 30 | Type: DECIMAL 31 | SubType: FLOAT 32 | - Name: cpucores 33 | Type: DECIMAL 34 | SubType: FLOAT 35 | - Name: cpucorerequestaverage 36 | Type: DECIMAL 37 | SubType: FLOAT 38 | - Name: cpucoreusageaverage 39 | Type: DECIMAL 40 | SubType: FLOAT 41 | - Name: cpucorehours 42 | Type: DECIMAL 43 | SubType: FLOAT 44 | - Name: cpucost 45 | Type: DECIMAL 46 | SubType: FLOAT 47 | - Name: cpucostadjustment 48 | Type: DECIMAL 49 | SubType: FLOAT 50 | - Name: cpuefficiency 51 | Type: DECIMAL 52 | SubType: FLOAT 53 | - Name: gpucount 54 | Type: DECIMAL 55 | SubType: FLOAT 56 | - Name: gpuhours 57 | Type: DECIMAL 58 | SubType: FLOAT 59 | - Name: gpucost 60 | Type: DECIMAL 61 | SubType: FLOAT 62 | - Name: gpucostadjustment 63 | Type: DECIMAL 64 | SubType: FLOAT 65 | - Name: networktransferbytes 66 | Type: DECIMAL 67 | SubType: FLOAT 68 | - Name: networkreceivebytes 69 | Type: DECIMAL 70 | SubType: FLOAT 71 | - Name: networkcost 72 | Type: DECIMAL 73 | SubType: FLOAT 74 | - Name: networkcrosszonecost 75 | Type: DECIMAL 76 | SubType: FLOAT 77 | - Name: networkcrossregioncost 78 | Type: DECIMAL 79 | SubType: FLOAT 80 | - Name: networkinternetcost 81 | Type: DECIMAL 82 | SubType: FLOAT 83 | - Name: networkcostadjustment 84 | Type: DECIMAL 85 | SubType: FLOAT 86 | - Name: loadbalancercost 87 | Type: DECIMAL 88 | SubType: FLOAT 89 | - Name: loadbalancercostadjustment 90 | Type: DECIMAL 91 | SubType: FLOAT 92 | - Name: pvbytes 93 | Type: DECIMAL 94 | SubType: FLOAT 95 | - Name: pvbytehours 96 | Type: DECIMAL 97 | SubType: FLOAT 98 | - Name: pvcost 99 | Type: DECIMAL 100 | SubType: FLOAT 101 | - Name: pvcostadjustment 102 | Type: DECIMAL 103 | SubType: FLOAT 104 | - Name: rambytes 105 | Type: DECIMAL 106 | SubType: FLOAT 107 | - Name: rambyterequestaverage 108 | Type: DECIMAL 109 | SubType: FLOAT 110 | - Name: rambyteusageaverage 111 | Type: DECIMAL 112 | SubType: FLOAT 113 | - Name: rambytehours 114 | Type: DECIMAL 115 | SubType: FLOAT 116 | - Name: ramcost 117 | Type: DECIMAL 118 | SubType: FLOAT 119 | - Name: ramcostadjustment 120 | Type: DECIMAL 121 | SubType: FLOAT 122 | - Name: ramefficiency 123 | Type: DECIMAL 124 | SubType: FLOAT 125 | - Name: sharedcost 126 | Type: DECIMAL 127 | SubType: FLOAT 128 | - Name: externalcost 129 | Type: DECIMAL 130 | SubType: FLOAT 131 | - Name: totalcost 132 | Type: DECIMAL 133 | SubType: FLOAT 134 | - Name: totalefficiency 135 | Type: DECIMAL 136 | SubType: FLOAT 137 | - Name: properties.provider 138 | Type: STRING 139 | - Name: properties.region 140 | Type: STRING 141 | - Name: properties.cluster 142 | Type: STRING 143 | - Name: properties.clusterid 144 | Type: STRING 145 | - Name: properties.eksclustername 146 | Type: STRING 147 | - Name: properties.container 148 | Type: STRING 149 | - Name: properties.namespace 150 | Type: STRING 151 | - Name: properties.pod 152 | Type: STRING 153 | - Name: properties.node 154 | Type: STRING 155 | - Name: properties.node_instance_type 156 | Type: STRING 157 | - Name: properties.node_availability_zone 158 | Type: STRING 159 | - Name: properties.node_capacity_type 160 | Type: STRING 161 | - Name: properties.node_architecture 162 | Type: STRING 163 | - Name: properties.node_os 164 | Type: STRING 165 | - Name: properties.node_nodegroup 166 | Type: STRING 167 | - Name: properties.node_nodegroup_image 168 | Type: STRING 169 | - Name: properties.controller 170 | Type: STRING 171 | - Name: properties.controllerkind 172 | Type: STRING 173 | - Name: properties.providerid 174 | Type: STRING 175 | - Name: account_id 176 | Type: STRING 177 | - Name: region 178 | Type: STRING 179 | - Name: year 180 | Type: STRING 181 | - Name: month 182 | Type: STRING 183 | LogicalTableMap: 184 | 476e7a0e-2e42-df7a-ff77-1555e92677fe: 185 | Alias: kubecost_view 186 | DataTransforms: 187 | - CreateColumnsOperation: 188 | Columns: 189 | - ColumnName: region_code 190 | ColumnId: 2e12d931-4fae-9f31-c3ea-0c9b47fcc401 191 | Expression: |- 192 | ifelse( 193 | isNotNull({properties.region}), {properties.region}, 194 | region 195 | ) 196 | - ProjectOperation: 197 | ProjectedColumns: 198 | - name 199 | - window.start 200 | - window.end 201 | - minutes 202 | - cpucores 203 | - cpucorerequestaverage 204 | - cpucoreusageaverage 205 | - cpucorehours 206 | - cpucost 207 | - cpucostadjustment 208 | - cpuefficiency 209 | - gpucount 210 | - gpuhours 211 | - gpucost 212 | - gpucostadjustment 213 | - networktransferbytes 214 | - networkreceivebytes 215 | - networkcost 216 | - networkcrosszonecost 217 | - networkcrossregioncost 218 | - networkinternetcost 219 | - networkcostadjustment 220 | - loadbalancercost 221 | - loadbalancercostadjustment 222 | - pvbytes 223 | - pvbytehours 224 | - pvcost 225 | - pvcostadjustment 226 | - rambytes 227 | - rambyterequestaverage 228 | - rambyteusageaverage 229 | - rambytehours 230 | - ramcost 231 | - ramcostadjustment 232 | - ramefficiency 233 | - sharedcost 234 | - externalcost 235 | - totalcost 236 | - totalefficiency 237 | - properties.provider 238 | - properties.region 239 | - properties.cluster 240 | - properties.clusterid 241 | - properties.eksclustername 242 | - properties.container 243 | - properties.namespace 244 | - properties.pod 245 | - properties.node 246 | - properties.node_instance_type 247 | - properties.node_availability_zone 248 | - properties.node_capacity_type 249 | - properties.node_architecture 250 | - properties.node_os 251 | - properties.node_nodegroup 252 | - properties.node_nodegroup_image 253 | - properties.controller 254 | - properties.controllerkind 255 | - properties.providerid 256 | - account_id 257 | - region 258 | - year 259 | - month 260 | - region_code 261 | - TagColumnOperation: 262 | ColumnName: region 263 | Tags: 264 | - ColumnGeographicRole: STATE 265 | - TagColumnOperation: 266 | ColumnName: region_code 267 | Tags: 268 | - ColumnGeographicRole: STATE 269 | Source: 270 | PhysicalTableId: 476e7a0e-2e42-df7a-ff77-1555e92677fe 271 | ImportMode: SPICE 272 | schedules: 273 | - default 274 | views: {} 275 | -------------------------------------------------------------------------------- /helm/kubecost_s3_exporter/Chart.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v2 2 | name: kubecost-s3-exporter 3 | description: A Python exporter from Kubecost to S3 4 | type: application 5 | version: 0.1.1 6 | -------------------------------------------------------------------------------- /helm/kubecost_s3_exporter/clusters_values/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/containers-cost-allocation-dashboard/8a581332a70ae55d53464e52a0bb8b3dd64cb425/helm/kubecost_s3_exporter/clusters_values/.gitkeep -------------------------------------------------------------------------------- /helm/kubecost_s3_exporter/templates/cron.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: batch/v1 2 | kind: CronJob 3 | metadata: 4 | name: {{ .Values.cronJob.name }} 5 | namespace: {{ .Values.cronJob.namespace }} 6 | spec: 7 | schedule: {{ .Values.cronJob.schedule }} 8 | failedJobsHistoryLimit: 3 9 | jobTemplate: 10 | spec: 11 | template: 12 | spec: 13 | automountServiceAccountToken: false 14 | securityContext: 15 | runAsNonRoot: true 16 | runAsUser: 65532 17 | seccompProfile: 18 | type: RuntimeDefault 19 | serviceAccountName: {{ .Values.serviceAccount.name }} 20 | restartPolicy: OnFailure 21 | containers: 22 | - name: kubecost-s3-exporter 23 | image: {{ .Values.image }} 24 | imagePullPolicy: {{ .Values.imagePullPolicy }} 25 | securityContext: 26 | allowPrivilegeEscalation: false 27 | readOnlyRootFilesystem: true 28 | capabilities: 29 | drop: 30 | - ALL 31 | env: 32 | {{- range .Values.env }} 33 | - name: "{{ .name }}" 34 | value: "{{ .value }}" 35 | {{- end }} 36 | volumeMounts: 37 | - mountPath: /tmp 38 | name: kubecost-s3-exporter 39 | volumes: 40 | - name: kubecost-s3-exporter 41 | emptyDir: 42 | sizeLimit: {{ .Values.ephemeralVolumeSize }} 43 | -------------------------------------------------------------------------------- /helm/kubecost_s3_exporter/templates/serviceaccount.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ServiceAccount 3 | metadata: 4 | name: {{ .Values.serviceAccount.name }} 5 | namespace: {{ .Values.serviceAccount.namespace }} 6 | annotations: 7 | eks.amazonaws.com/role-arn: {{ .Values.serviceAccount.role }} 8 | automountServiceAccountToken: false 9 | -------------------------------------------------------------------------------- /helm/kubecost_s3_exporter/values.yaml: -------------------------------------------------------------------------------- 1 | # Default values for kubecost-s3-exporter. 2 | # This is a YAML-formatted file. 3 | # Declare variables to be passed into your templates. 4 | 5 | namespace: "kubecost-s3-exporter" 6 | 7 | image: "" # Add the Docker container image in the format of /: 8 | imagePullPolicy: Always 9 | ephemeralVolumeSize: "50Mi" 10 | 11 | cronJob: 12 | name: "kubecost-s3-exporter" 13 | schedule: "0 0 * * *" 14 | 15 | serviceAccount: 16 | create: true 17 | name: "kubecost-s3-exporter" 18 | role: "" # Example: arn:aws:iam:::role/ 19 | 20 | env: 21 | - name: "S3_BUCKET_NAME" 22 | value: "" # Add S3 bucket name 23 | - name: "KUBECOST_API_ENDPOINT" 24 | value: "http://kubecost-cost-analyzer.kubecost:9090" # Change to your Kubecost endpoint if necessary 25 | - name: "BACKFILL_PERIOD_DAYS" 26 | value: 15 27 | - name: "CLUSTER_ID" 28 | value: "" # Change to your EKS cluster ARN 29 | - name: "IRSA_PARENT_IAM_ROLE_ARN" 30 | value: "" 31 | - name: "AGGREGATION" 32 | value: "container" 33 | - name: "KUBECOST_ALLOCATION_API_PAGINATE" 34 | value: "False" 35 | - name: "CONNECTION_TIMEOUT" 36 | value: 10 37 | - name: "KUBECOST_ALLOCATION_API_READ_TIMEOUT" 38 | value: 60 39 | - name: "TLS_VERIFY" 40 | value: "True" 41 | - name: "KUBECOST_CA_CERTIFICATE_SECRET_NAME" 42 | value: "" 43 | - name: "KUBECOST_CA_CERTIFICATE_SECRET_REGION" 44 | value: "" 45 | - name: "LABELS" 46 | value: "" # Comma-separated list of labels. Example: "app, chart, app.kubernetes.io/version" 47 | - name: "ANNOTATIONS" 48 | value: "" # Comma-separated list of annotations. Example: "kubernetes.io/psp, eks.amazonaws.com/compute_type, team" 49 | - name: "PYTHONUNBUFFERED" 50 | value: "1" 51 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | boto3==1.34.11 2 | botocore==1.34.11 3 | pandas==2.1.4 4 | pyarrow==14.0.2 5 | requests==2.31.0 6 | -------------------------------------------------------------------------------- /screenshots/architecture_diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/awslabs/containers-cost-allocation-dashboard/8a581332a70ae55d53464e52a0bb8b3dd64cb425/screenshots/architecture_diagram.png -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/README.md: -------------------------------------------------------------------------------- 1 | # Containers Cost Allocation (CCA) Dashboard Terraform Module 2 | 3 | This Terraform Module is used to deploy the resources required for the Containers Cost Allocation (CCA) dashboard. 4 | It's suitable to deploy the resources in multi-account, multi-region, multi-cluster environments. 5 | It's used to deploy the following: 6 | 7 | 1. The AWS resources that support the solution 8 | 2. The K8s resources (Kubecost S3 Exporter CronJob and Service Account) on each cluster. 9 | This is done either by invoking Helm, or by generating a Helm `values.yaml` for you to deploy. 10 | These options are configurable per cluster by using the `invoke_helm` variable. 11 | If set, Terraform will invoke Helm and will deploy the K8s resources This is the default option. 12 | If not set, Terraform will generate a Helm `values.yaml` for this cluster. 13 | You'll then need to deploy the K8s resources yourself using the `helm` command. 14 | 15 | This guide is composed of the following sections: 16 | 17 | 1. Requirements: 18 | A list of the requirements to use this module. 19 | 2. Structure: 20 | Shows the module's structure. 21 | 3. Initial Deployment: 22 | Provides initial deployment instructions. 23 | 4. Maintenance: 24 | Provides information on common changes that might be done after the initial deployment. 25 | 5. Cleanup: 26 | Provides information on the process to clean up resources. 27 | 28 | ## Requirements 29 | 30 | This Terraform module requires the following: 31 | 32 | * That you completed all requirements in the project's [REQUIREMENTS.md](../../REQUIREMENTS.md) file 33 | * That you [built and pushed the Kubecost S3 Exporter container image to ECR](../../DEPLOYMENT.md/.#step-1-build-and-push-the-container-image) 34 | * That you manage your AWS credentials using [shared configuration and credentials files](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). 35 | This is required because this Terraform module is meant to create or access resources in different AWS accounts that may require different sets of credentials. 36 | * In your kubeconfig file, each EKS cluster should reference the AWS profile from the shared configuration file. 37 | This is so that Helm (invoked by Terraform or manually) can tell which AWS credentials to use when communicating with the cluster. 38 | 39 | ## Structure 40 | 41 | Below is the complete module structure, followed by details on each directory/module: 42 | 43 | terraform-aws-cca/ 44 | ├── README.md 45 | ├── main.tf 46 | ├── outputs.tf 47 | ├── providers.tf 48 | ├── terraform.tfvars 49 | ├── variables.tf 50 | ├── examples 51 | │   └── root_module 52 | │   ├── main.tf 53 | │   ├── outputs.tf 54 | │   ├── providers.tf 55 | │   ├── terraform.tfvars 56 | │   └── variables.tf 57 | └── modules 58 | ├── common_locals 59 | │   ├── locals.tf 60 | │   └── outputs.tf 61 | ├── kubecost_s3_exporter 62 | │   ├── README.md 63 | │   ├── locals.tf 64 | │   ├── main.tf 65 | │   ├── outputs.tf 66 | │   └── variables.tf 67 | └── pipeline 68 | ├── README.md 69 | ├── locals.tf 70 | ├── main.tf 71 | ├── outputs.tf 72 | ├── secret_policy.tpl 73 | └── variables.tf 74 | 75 | ### The Root Directory 76 | 77 | The root directory is the root module. 78 | It contains the following files: 79 | * The [`main.tf`](main.tf) file, which is the entry point to this module. 80 | Thi is where you call the reusable modules to deploy the resources: 81 | * The `pipline` reusable module that deploys the pipeline resources 82 | * The `kubecost_s3_exporter` reusable module for each cluster, to deploy the Kubecost S3 Exporter on your clusters 83 | * The [`providers.tf`](providers.tf) file, where you add a provider configuration for each module 84 | * The [`outputs.tf`](outputs.tf) file, to be used to add your required outputs 85 | * The [`variables.tf`](variables.tf) file, where the root module common variables are defined. 86 | These variables are used by the reusable modules. 87 | * The [`terraform.tfvars`](terraform.tfvars), where you provide values for the root module common variables 88 | 89 | ### The `modules` Directory 90 | 91 | The `modules` directory contains the reusable Terraform child modules used to deploy the solution. 92 | It contains several modules, as follows: 93 | 94 | #### The `common_locals` Module 95 | 96 | The `common_locals` reusable module in the `common_locals` directory only has locals and outputs. 97 | It contains the common locals that are used by other modules. It does not contain resources. 98 | 99 | #### The `pipeline` Module 100 | 101 | The `pipeline` reusable module in the `pipeline` directory contains the Terraform IaC required to deploy the AWS pipeline resources. 102 | It contains module-specific variables, outputs, and resources. 103 | It also contains a template file (secret_policy.tpl) that contains IAM policy for AWS Secrets Manager secret policy. 104 | 105 | #### The `kubecost_s3_exporter` Module 106 | 107 | The `kubecost_s3_exporter` reusable module in the `kubecost_s3_exporter` directory contains the Terraform IaC required to deploy: 108 | 109 | * The K8s resources (the CronJob used to create the Kubecost S3 Exporter pod, and a service account) on each EKS cluster 110 | * For each EKS cluster: 111 | * The IRSA (IAM Role for Service Account) in the EKS cluster's account 112 | * If the cluster is in different account than the S3 bucket, also a parent IAM role is created, in the S3 bucket's account 113 | 114 | It contains module-specific variables, outputs, and resources. 115 | 116 | ### The `examples` Directory 117 | 118 | The `examples` directory currently includes only the `root_module` directory. 119 | It includes examples of the following files from the root directory (root module): 120 | * The [`main.tf`](examples/root_module/main.tf) file 121 | * The [`outputs.tf`](examples/root_module/outputs.tf) file 122 | * The [`providers.tf`](examples/root_module/providers.tf) file 123 | * The [`terraform.tfvars`](terraform.tfvars) file 124 | 125 | These files give some useful examples for you to get started when modifying the actual files. 126 | 127 | ## Initial Deployment 128 | 129 | Deployment of the Containers Cost Allocation (CCA) Dashboard using this Terraform module requires the following steps: 130 | 131 | 1. Add provider configuration for each reusable module, in the [`providers.tf`](providers.tf) file in the root module 132 | 1. Add provider for the `pipeline` reusable module. For more information and examples: 133 | See the ["Define Provider" section in the module's README.md file](modules/pipeline/README.md/.#define-provider). 134 | 2. Add provider for the `kubecost_s3_exporter` reusable module. For more information and examples: 135 | See the ["Define Provider for each EKS cluster" section in the module's README.md file](modules/kubecost_s3_exporter/README.md/.#define-provider-for-each-eks-cluster). 136 | 2. Provide common root module variable values in the [`terraform.tfvars`](terraform.tfvars) file. 137 | The only required variable is `bucket_arn`. 138 | 3. Provide variables values in the [`main.tf`](main.tf) file in the root module, for: 139 | 1. The `pipeline` reusable module. For more information and examples: 140 | See the ["Create a Calling Module and Provide Variables Values" section in the module's README.md file](modules/pipeline/README.md/.#create-a-calling-module-and-provide-variables-values). 141 | 2. The `kubecost_s3_exporter` reusable module for each cluster. For more information and examples: 142 | See the ["Create a Calling Module and Provide Variables Values" section in the module's README.md file](modules/kubecost_s3_exporter/README.md/.#create-a-calling-module-and-provide-variables-values). 143 | 4. Optionally, add outputs to the [`outputs.tf`](outputs.tf) file in the root module. 144 | See more information on the `kubecost_s3_exporter` module's outputs in the [module's README.md file](modules/kubecost_s3_exporter/README.md/.#adding-outputs-for-each-cluster). 145 | 5. Deploy: 146 | From the root directory of the Terraform module: 147 | 1. Run `terraform init` 148 | 2. Run `terraform apply` 149 | 150 | ## Maintenance 151 | 152 | After the solution is initially deployed, you might want to make changes. 153 | Below are instruction for some common changes that you might do after the initial deployment. 154 | 155 | ### Deploying on Additional Clusters 156 | 157 | When adding additional clusters after the initial deployment, not all the initial deployment steps are required. 158 | To continue adding additional clusters after the initial deployment, the only required steps are as follows, for each cluster: 159 | 160 | 1. Create an [IAM OIDC Provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) in the EKS cluster's account 161 | 2. Define additional providers for the clusters 162 | 3. Create additional calling modules of the `kubecost_s3_exporter` reusable module, for each cluster, and provide variables values. 163 | This is done in the [`main.tf`](main.tf) file in the root directory. 164 | You can follow the instructions in the [`kubecost_s3_exporter` calling module creation steps](modules/kubecost_s3_exporter/README.md/.#create-a-calling-module-and-provide-variables-values) 165 | 4. If you need to add labels or annotations for this cluster, follow the [Maintenance -> Adding/Removing Labels/Annotations to/from the Dataset section](#addingremoving-labelsannotations-tofrom-the-dataset) 166 | 5. Optionally, add cluster output for the IRSA (IAM Role for Service Account) and parent IAM role, for each cluster 167 | 168 | Then, from the root directory, run `terraform init` and `terraform apply` 169 | 170 | ### Updating Clusters Inputs 171 | 172 | After the initial deployment, you might want to change parameters. 173 | Below are instructions for doing so: 174 | 175 | #### Updating Inputs for Existing Clusters 176 | 177 | To update inputs for existing clusters (all or some), perform the following: 178 | 179 | 1. In the root directory, open the [`main.tf`](main.tf) file 180 | 2. Change the relevant inputs in the calling modules of the clusters you wish to update 181 | 3. From the root directory, run `terraform apply` 182 | 183 | ### Adding/Removing Labels/Annotations to/from the Dataset 184 | 185 | After the initial deployment, you might want to add or remove labels or annotations for some or all clusters, to/from the dataset. 186 | To do this, perform the following in the [`terraform.tfvars`](terraform.tfvars) file in the root directory: 187 | 1. To add labels, add the K8s labels keys in the `k8s_labels` variable. 188 | This list should include all K8s labels from all clusters, that you wish to include in the dataset. 189 | 2. To add annotations, add the K8s annotations keys in the `k8s_annotations` variable. 190 | This list should include all K8s annotations from all clusters, that you wish to include in the dataset. 191 | 192 | As an example, see the below table, showing a possible list of labels and annotations for different clusters: 193 | 194 | | Cluster | Labels | Labels Wanted in the Dataset | Annotations | Annotations Wanted in the Dataset | 195 | |-------------------------------------|------------|------------------------------|--------------|-----------------------------------| 196 | | cluster\_a | a, b, c, d | a, b, c | 1, 2, 3, 4 | 1, 2, 3 | 197 | | cluster\_b | a, f, g, h | f, g | 1, 6, 7, 8 | 5, 6 | 198 | | cluster\_c | x, y, z, a | | 9, 10, 11, 1 | | 199 | 200 | In this case, this is how the `k8s_labels` and `k8s_annotations` variables will look like: 201 | 202 | k8s_labels = ["a", "b", "c", "f", "g"] # Optionally, add K8s labels you'd like to be present in the dataset 203 | k8s_annotations = ["1", "2", "3", "5", "6"] # Optionally, add K8s annotations you'd like to be present in the dataset 204 | 205 | 3. From the root directory, run `terraform apply`. 206 | Terraform will output the new list of labels and annotations when the deployment is completed. 207 | 4. Save and publish the dataset by following the ["Save and Publish the Dashboard" section in the DEPLOYMENT.md file](../../DEPLOYMENT.md/.#save-and-publish-the-dashboard) 208 | 209 | **_Note about annotations:_** 210 | 211 | While K8s labels are included by default in Kubecost Allocation API response, K8s annotations aren't. 212 | To include K8s annotations in the Kubecost Allocation API response, following [this document](https://docs.kubecost.com/install-and-configure/advanced-configuration/annotations). 213 | 214 | ## Cleanup 215 | 216 | ### Removing Kubecost S3 Exporter from Specific Clusters 217 | 218 | To remove the Kubecost S3 Exporter from a specific cluster, perform the following: 219 | 220 | 1. From the [`main.tf`](main.tf) file in the root directory, remove the calling module instance of the cluster 221 | 2. From the [`outputs.tf`](outputs.tf) file in the root directory, remove the outputs of the cluster, if any 222 | 3. Run `terraform apply` 223 | 4. From the [`providers.tf`](providers.tf) file in the root directory, remove the providers of the cluster 224 | You must remove the providers only after you did step 1-3 above, otherwise the above steps will fail 225 | 5. If Kubecost S3 Exporter was deployed on this cluster using `invoke_helm=false`, you also need to uninstall the chart: 226 | `helm uninstall kubecost-s3-exporter -n --kube-context ` 227 | 228 | ### Complete Cleanup 229 | 230 | To completely clean up the entire setup, run `terraform destroy` from the root directory. 231 | Then, follow the "Cleanup" section of the main README.md to clean up other resources that weren't created by Terraform. 232 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/examples/root_module/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_version = ">= 1.3.0" 3 | required_providers { 4 | aws = { 5 | source = "hashicorp/aws" 6 | version = "~> 5.26.0" 7 | } 8 | helm = { 9 | source = "hashicorp/helm" 10 | version = "~> 2.9.0" 11 | } 12 | local = { 13 | source = "hashicorp/local" 14 | version = "~> 2.4.0" 15 | } 16 | } 17 | } 18 | 19 | ###################################### 20 | # Section 1 - AWS Pipeline Resources # 21 | ###################################### 22 | 23 | module "pipeline" { 24 | source = "./modules/pipeline" 25 | 26 | # # 27 | # Root Module Common Variables # 28 | # # 29 | 30 | # References to root module common variables, do not remove or change 31 | 32 | bucket_arn = var.bucket_arn 33 | k8s_labels = var.k8s_labels 34 | k8s_annotations = var.k8s_annotations 35 | aws_common_tags = var.aws_common_tags 36 | 37 | # # 38 | # Pipeline Module Variables # 39 | # # 40 | 41 | # Provide pipeline module variables values here 42 | 43 | 44 | } 45 | 46 | ######################################################### 47 | # Section 2 - Data Collection Pod Deployment using Helm # 48 | ######################################################### 49 | 50 | # # 51 | # Clusters in Account 111111111111 # 52 | # # 53 | 54 | # Clusters in Region us-east-1 # 55 | 56 | module "us-east-1-111111111111-cluster1" { 57 | source = "./modules/kubecost_s3_exporetr" 58 | 59 | providers = { 60 | aws.pipeline = aws 61 | aws.eks = aws.us-east-1-111111111111-cluster1 62 | helm = helm.us-east-1-111111111111-cluster1 63 | } 64 | 65 | # # 66 | # Root Module Common Variables # 67 | # # 68 | 69 | # References to root module common variables 70 | # Always include when creating new calling module, and do not remove or change 71 | 72 | bucket_arn = var.bucket_arn 73 | k8s_labels = var.k8s_labels 74 | k8s_annotations = var.k8s_annotations 75 | aws_common_tags = var.aws_common_tags 76 | 77 | # # 78 | # Kubecost S3 Exporter Module Variables # 79 | # # 80 | 81 | # Provide kubecost_s3_exporter module variables values here 82 | 83 | cluster_arn = "arn:aws:eks:us-east-1:111111111111:cluster/cluster1" 84 | kubecost_s3_exporter_container_image = "111111111111.dkr.ecr.us-east-1.amazonaws.com/kubecost_cid:0.1.0" 85 | kubecost_api_endpoint = "https://kubecost-eks-cost-analyzer.kubecost-eks" 86 | connection_timeout = 5 87 | kubecost_allocation_api_read_timeout = 30 88 | tls_verify = "no" 89 | kubecost_ca_certificate_secret_name = "kubecost" 90 | kubecost_ca_certificate_secrets = module.pipeline.kubecost_ca_cert_secret 91 | kubecost_ephemeral_volume_size = "100Mi" 92 | backfill_period_days = 5 93 | } 94 | 95 | module "us-east-1-111111111111-cluster2" { 96 | source = "./modules/kubecost_s3_exporetr" 97 | 98 | providers = { 99 | aws.pipeline = aws 100 | aws.eks = aws.us-east-1-111111111111-cluster2 101 | helm = helm.us-east-1-111111111111-cluster2 102 | } 103 | 104 | # # 105 | # Root Module Common Variables # 106 | # # 107 | 108 | # References to root module common variables 109 | # Always include when creating new calling module, and do not remove or change 110 | 111 | bucket_arn = var.bucket_arn 112 | k8s_labels = var.k8s_labels 113 | k8s_annotations = var.k8s_annotations 114 | aws_common_tags = var.aws_common_tags 115 | 116 | # # 117 | # Kubecost S3 Exporter Module Variables # 118 | # # 119 | 120 | # Provide kubecost_s3_exporter module variables values here 121 | 122 | cluster_arn = "arn:aws:eks:us-east-1:111111111111:cluster/cluster2" 123 | kubecost_s3_exporter_container_image = "111111111111.dkr.ecr.us-east-1.amazonaws.com/kubecost_cid:0.1.0" 124 | namespace = "kubecost-s3-exporter-2" 125 | service_account = "kubecost-s3-exporter-2" 126 | kubecost_allocation_api_paginate = "Yes" 127 | } 128 | 129 | # Clusters in Region us-east-2 # 130 | 131 | module "us-east-2-111111111111-cluster1" { 132 | source = "./modules/kubecost_s3_exporetr" 133 | 134 | providers = { 135 | aws.pipeline = aws 136 | aws.eks = aws.us-east-2-111111111111-cluster1 137 | helm = helm.us-east-2-111111111111-cluster1 138 | } 139 | 140 | # # 141 | # Root Module Common Variables # 142 | # # 143 | 144 | # References to root module common variables 145 | # Always include when creating new calling module, and do not remove or change 146 | 147 | bucket_arn = var.bucket_arn 148 | k8s_labels = var.k8s_labels 149 | k8s_annotations = var.k8s_annotations 150 | aws_common_tags = var.aws_common_tags 151 | 152 | # # 153 | # Kubecost S3 Exporter Module Variables # 154 | # # 155 | 156 | # Provide kubecost_s3_exporter module variables values here 157 | 158 | cluster_arn = "arn:aws:eks:us-east-2:111111111111:cluster/cluster1" 159 | kubecost_s3_exporter_container_image = "111111111111.dkr.ecr.us-east-1.amazonaws.com/kubecost_cid:0.1.0" 160 | kubecost_s3_exporter_container_image_pull_policy = "IfNotPresent" 161 | kubecost_s3_exporter_pod_schedule = "0 0 * * 5" 162 | kubecost_api_endpoint = "http://kubecost-eks-cost-analyzer.kubecost-eks:9090" 163 | } 164 | 165 | module "us-east-2-111111111111-cluster2" { 166 | source = "./modules/kubecost_s3_exporetr" 167 | 168 | providers = { 169 | aws.pipeline = aws 170 | aws.eks = aws.us-east-2-111111111111-cluster2 171 | } 172 | 173 | # # 174 | # Root Module Common Variables # 175 | # # 176 | 177 | # References to root module common variables 178 | # Always include when creating new calling module, and do not remove or change 179 | 180 | bucket_arn = var.bucket_arn 181 | k8s_labels = var.k8s_labels 182 | k8s_annotations = var.k8s_annotations 183 | aws_common_tags = var.aws_common_tags 184 | 185 | # # 186 | # Kubecost S3 Exporter Module Variables # 187 | # # 188 | 189 | # Provide kubecost_s3_exporter module variables values here 190 | 191 | cluster_arn = "arn:aws:eks:us-east-2:111111111111:cluster/cluster2" 192 | kubecost_s3_exporter_container_image = "111111111111.dkr.ecr.us-east-1.amazonaws.com/kubecost_cid:0.1.0" 193 | invoke_helm = false 194 | } 195 | 196 | # # 197 | # Clusters in Account 222222222222 # 198 | # # 199 | 200 | # Clusters in Region us-east-1 # 201 | 202 | module "us-east-1-222222222222-cluster1" { 203 | source = "./modules/kubecost_s3_exporetr" 204 | 205 | providers = { 206 | aws.pipeline = aws 207 | aws.eks = aws.us-east-1-222222222222-cluster1 208 | helm = helm.us-east-1-222222222222-cluster1 209 | } 210 | 211 | # # 212 | # Root Module Common Variables # 213 | # # 214 | 215 | # References to root module common variables 216 | # Always include when creating new calling module, and do not remove or change 217 | 218 | bucket_arn = var.bucket_arn 219 | k8s_labels = var.k8s_labels 220 | k8s_annotations = var.k8s_annotations 221 | aws_common_tags = var.aws_common_tags 222 | 223 | # # 224 | # Kubecost S3 Exporter Module Variables # 225 | # # 226 | 227 | # Provide kubecost_s3_exporter module variables values here 228 | 229 | cluster_arn = "arn:aws:eks:us-east-1:222222222222:cluster/cluster1" 230 | kubecost_s3_exporter_container_image = "222222222222.dkr.ecr.us-east-1.amazonaws.com/kubecost_cid:0.1.0" 231 | kubecost_api_endpoint = "http://kubecost-eks-cost-analyzer.kubecost-eks:9090" 232 | k8s_config_path = "~/configs/k8s/config" 233 | } 234 | 235 | module "us-east-1-222222222222-cluster2" { 236 | source = "./modules/kubecost_s3_exporetr" 237 | 238 | providers = { 239 | aws.pipeline = aws 240 | aws.eks = aws.us-east-1-222222222222-cluster2 241 | helm = helm.us-east-1-222222222222-cluster2 242 | } 243 | 244 | # # 245 | # Root Module Common Variables # 246 | # # 247 | 248 | # References to root module common variables 249 | # Always include when creating new calling module, and do not remove or change 250 | 251 | bucket_arn = var.bucket_arn 252 | k8s_labels = var.k8s_labels 253 | k8s_annotations = var.k8s_annotations 254 | aws_common_tags = var.aws_common_tags 255 | 256 | # # 257 | # Kubecost S3 Exporter Module Variables # 258 | # # 259 | 260 | # Provide kubecost_s3_exporter module variables values here 261 | 262 | cluster_arn = "arn:aws:eks:us-east-1:222222222222:cluster/cluster2" 263 | kubecost_s3_exporter_container_image = "222222222222.dkr.ecr.us-east-1.amazonaws.com/kubecost_cid:0.1.0" 264 | } 265 | 266 | # Clusters in Region us-east-2 # 267 | 268 | module "us-east-2-222222222222-cluster1" { 269 | source = "./modules/kubecost_s3_exporetr" 270 | 271 | providers = { 272 | aws.pipeline = aws 273 | aws.eks = aws.us-east-2-222222222222-cluster1 274 | helm = helm.us-east-2-222222222222-cluster1 275 | } 276 | 277 | # # 278 | # Root Module Common Variables # 279 | # # 280 | 281 | # References to root module common variables 282 | # Always include when creating new calling module, and do not remove or change 283 | 284 | bucket_arn = var.bucket_arn 285 | k8s_labels = var.k8s_labels 286 | k8s_annotations = var.k8s_annotations 287 | aws_common_tags = var.aws_common_tags 288 | 289 | # # 290 | # Kubecost S3 Exporter Module Variables # 291 | # # 292 | 293 | # Provide kubecost_s3_exporter module variables values here 294 | 295 | cluster_arn = "arn:aws:eks:us-east-2:222222222222:cluster/cluster1" 296 | kubecost_s3_exporter_container_image = "222222222222.dkr.ecr.us-east-1.amazonaws.com/kubecost_cid:0.1.0" 297 | kubecost_api_endpoint = "http://kubecost-eks-cost-analyzer.kubecost-eks:9090" 298 | invoke_helm = false 299 | } 300 | 301 | module "us-east-2-222222222222-cluster2" { 302 | source = "./modules/kubecost_s3_exporetr" 303 | 304 | providers = { 305 | aws.pipeline = aws 306 | aws.eks = aws.us-east-2-222222222222-cluster2 307 | helm = helm.us-east-2-222222222222-cluster2 308 | } 309 | 310 | # # 311 | # Root Module Common Variables # 312 | # # 313 | 314 | # References to root module common variables 315 | # Always include when creating new calling module, and do not remove or change 316 | 317 | bucket_arn = var.bucket_arn 318 | k8s_labels = var.k8s_labels 319 | k8s_annotations = var.k8s_annotations 320 | aws_common_tags = var.aws_common_tags 321 | 322 | # # 323 | # Kubecost S3 Exporter Module Variables # 324 | # # 325 | 326 | # Provide kubecost_s3_exporter module variables values here 327 | 328 | cluster_arn = "arn:aws:eks:us-east-2:222222222222:cluster/cluster2" 329 | kubecost_s3_exporter_container_image = "222222222222.dkr.ecr.us-east-1.amazonaws.com/kubecost_cid:0.1.0" 330 | namespace = "kubecost-s3-exporter-2" 331 | create_namespace = false 332 | service_account = "kubecost-s3-exporter-2" 333 | create_service_account = false 334 | } 335 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/examples/root_module/outputs.tf: -------------------------------------------------------------------------------- 1 | # Output for showing the distinct labels from all clusters, collected from the "k8s_labels" common input 2 | output "labels" { 3 | value = length(var.k8s_labels) > 0 ? join(", ", distinct(var.k8s_labels)) : null 4 | description = "A list of the distinct labels of all clusters, that'll be added to the dataset" 5 | } 6 | 7 | # Output for showing the distinct annotations from all clusters, collected from the "k8s_annotations" common input 8 | output "annotations" { 9 | value = length(var.k8s_annotations) > 0 ? join(", ", distinct(var.k8s_annotations)) : null 10 | description = "A list of the distinct annotations of all clusters, that'll be added to the dataset" 11 | } 12 | 13 | # Clusters outputs 14 | output "us-east-1-111111111111-cluster1" { 15 | value = module.us-east-1-111111111111-cluster1 16 | description = "The outputs for 'us-east-1-111111111111-cluster1'" 17 | } 18 | 19 | output "us-east-1-111111111111-cluster2" { 20 | value = module.us-east-1-111111111111-cluster2 21 | description = "The outputs for 'us-east-1-111111111111-cluster2'" 22 | } 23 | 24 | output "us-east-2-111111111111-cluster1" { 25 | value = module.us-east-2-111111111111-cluster1 26 | description = "The outputs for 'us-east-2-111111111111-cluster1'" 27 | } 28 | 29 | output "us-east-2-111111111111-cluster2" { 30 | value = module.us-east-2-111111111111-cluster2 31 | description = "The outputs for 'us-east-2-111111111111-cluster2'" 32 | } 33 | 34 | output "us-east-1-222222222222-cluster1" { 35 | value = module.us-east-1-222222222222-cluster1 36 | description = "The outputs for 'us-east-1-222222222222-cluster1'" 37 | } 38 | 39 | output "us-east-1-222222222222-cluster2" { 40 | value = module.us-east-1-222222222222-cluster2 41 | description = "The outputs for 'us-east-1-222222222222-cluster2'" 42 | } 43 | 44 | output "us-east-2-222222222222-cluster1" { 45 | value = module.us-east-2-222222222222-cluster1 46 | description = "The outputs for 'us-east-2-222222222222-cluster1'" 47 | } 48 | 49 | output "us-east-2-222222222222-cluster2" { 50 | value = module.us-east-2-222222222222-cluster2 51 | description = "The outputs for 'us-east-2-222222222222-cluster2'" 52 | } 53 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/examples/root_module/providers.tf: -------------------------------------------------------------------------------- 1 | ##################################### 2 | # Section 1 - Pipeline AWS Provider # 3 | ##################################### 4 | 5 | provider "aws" { 6 | region = "us-east-1" 7 | shared_config_files = ["~/.aws/config"] 8 | shared_credentials_files = ["~/.aws/credentials"] 9 | profile = "pipeline_profile" 10 | default_tags { 11 | tags = var.aws_common_tags 12 | } 13 | } 14 | 15 | ########################################################### 16 | # Section 2 - Kubecost S3 Exporter AWS and Helm Providers # 17 | ########################################################### 18 | 19 | # # 20 | # Clusters in Account 111111111111 # 21 | # # 22 | 23 | # Clusters in Region us-east-1 # 24 | 25 | provider "aws" { 26 | alias = "us-east-1-111111111111-cluster1" 27 | 28 | region = "us-east-1" 29 | shared_config_files = ["~/.aws/config"] 30 | shared_credentials_files = ["~/.aws/credentials"] 31 | profile = "profile1" 32 | default_tags { 33 | tags = var.aws_common_tags 34 | } 35 | } 36 | 37 | provider "helm" { 38 | alias = "us-east-1-111111111111-cluster1" 39 | 40 | kubernetes { 41 | config_context = "arn:aws:eks:us-east-1:111111111111:cluster/cluster1" 42 | config_path = "~/.kube/config" 43 | } 44 | } 45 | 46 | provider "aws" { 47 | alias = "us-east-1-111111111111-cluster2" 48 | 49 | region = "us-east-1" 50 | shared_config_files = ["~/.aws/config"] 51 | shared_credentials_files = ["~/.aws/credentials"] 52 | profile = "profile1" 53 | default_tags { 54 | tags = var.aws_common_tags 55 | } 56 | } 57 | 58 | provider "helm" { 59 | alias = "us-east-1-111111111111-cluster2" 60 | 61 | kubernetes { 62 | config_context = "arn:aws:eks:us-east-1:111111111111:cluster/cluster2" 63 | config_path = "~/.kube/config" 64 | } 65 | } 66 | 67 | # Clusters in Region us-east-2 # 68 | 69 | provider "aws" { 70 | alias = "us-east-2-111111111111-cluster1" 71 | 72 | region = "us-east-2" 73 | shared_config_files = ["~/.aws/config"] 74 | shared_credentials_files = ["~/.aws/credentials"] 75 | profile = "profile1" 76 | default_tags { 77 | tags = var.aws_common_tags 78 | } 79 | } 80 | 81 | provider "helm" { 82 | alias = "us-east-2-111111111111-cluster1" 83 | 84 | kubernetes { 85 | config_context = "arn:aws:eks:us-east-1:111111111111:cluster/cluster1" 86 | config_path = "~/.kube/config" 87 | } 88 | } 89 | 90 | provider "aws" { 91 | alias = "us-east-2-111111111111-cluster2" 92 | 93 | region = "us-east-2" 94 | shared_config_files = ["~/.aws/config"] 95 | shared_credentials_files = ["~/.aws/credentials"] 96 | profile = "profile1" 97 | default_tags { 98 | tags = var.aws_common_tags 99 | } 100 | } 101 | 102 | # # 103 | # Clusters in Account 222222222222 # 104 | # # 105 | 106 | # Clusters in Region us-east-1 # 107 | 108 | provider "aws" { 109 | alias = "us-east-1-222222222222-cluster1" 110 | 111 | region = "us-east-1" 112 | shared_config_files = ["~/.aws/config"] 113 | shared_credentials_files = ["~/.aws/credentials"] 114 | profile = "profile2" 115 | default_tags { 116 | tags = var.aws_common_tags 117 | } 118 | } 119 | 120 | provider "helm" { 121 | alias = "us-east-1-222222222222-cluster1" 122 | 123 | kubernetes { 124 | config_context = "arn:aws:eks:us-east-1:222222222222:cluster/cluster1" 125 | config_path = "~/.kube/config" 126 | } 127 | } 128 | 129 | provider "aws" { 130 | alias = "us-east-1-222222222222-cluster2" 131 | 132 | region = "us-east-1" 133 | shared_config_files = ["~/.aws/config"] 134 | shared_credentials_files = ["~/.aws/credentials"] 135 | profile = "profile2" 136 | default_tags { 137 | tags = var.aws_common_tags 138 | } 139 | } 140 | 141 | provider "helm" { 142 | alias = "us-east-1-222222222222-cluster2" 143 | 144 | kubernetes { 145 | config_context = "arn:aws:eks:us-east-1:222222222222:cluster/cluster2" 146 | config_path = "~/.kube/config" 147 | } 148 | } 149 | 150 | # Clusters in Region us-east-2 # 151 | 152 | provider "aws" { 153 | alias = "us-east-2-222222222222-cluster1" 154 | 155 | region = "us-east-2" 156 | shared_config_files = ["~/.aws/config"] 157 | shared_credentials_files = ["~/.aws/credentials"] 158 | profile = "profile2" 159 | default_tags { 160 | tags = var.aws_common_tags 161 | } 162 | } 163 | 164 | provider "aws" { 165 | alias = "us-east-2-222222222222-cluster2" 166 | 167 | region = "us-east-2" 168 | shared_config_files = ["~/.aws/config"] 169 | shared_credentials_files = ["~/.aws/credentials"] 170 | profile = "profile2" 171 | default_tags { 172 | tags = var.aws_common_tags 173 | } 174 | } 175 | 176 | provider "helm" { 177 | alias = "us-east-2-222222222222-cluster2" 178 | 179 | kubernetes { 180 | config_context = "arn:aws:eks:us-east-1:222222222222:cluster/cluster2" 181 | config_path = "~/.kube/config" 182 | } 183 | } 184 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/examples/root_module/terraform.tfvars: -------------------------------------------------------------------------------- 1 | bucket_arn = "arn:aws:s3:::bucket-name" 2 | k8s_labels = ["app", "chart", "component", "app.kubernetes.io/version", "app.kubernetes.io/managed_by", "app.kubernetes.io/part_of"] 3 | k8s_annotations = ["kubernetes.io/psp", "eks.amazonaws.com/compute_type"] 4 | aws_common_tags = { 5 | tag_key1 = "tag_value1" 6 | tak_key2 = "tag_value2" 7 | } 8 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/examples/root_module/variables.tf: -------------------------------------------------------------------------------- 1 | variable "bucket_arn" { 2 | description = <<-EOF 3 | (Required) The ARN of the S3 Bucket to which the Kubecost data will be uploaded. 4 | Possible values: A valid S3 bucket ARN. 5 | EOF 6 | 7 | type = string 8 | 9 | # Note - the full regex should have been "^arn:(?:aws|aws-cn|aws-us-gov):s3:::(?!(xn--|.+-s3alias$))[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$" 10 | # The "(?!(xn--|.+-s3alias$))" part has been omitted because Terraform regex engine doesn't support negative lookahead (the "?!" part) 11 | # Therefore, it has been removed, and instead, "!startswith" and "!endswith" conditions have been added, to complement this missing functionality 12 | validation { 13 | condition = ( 14 | can(regex("^arn:(?:aws|aws-cn|aws-us-gov):s3:::[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$", var.bucket_arn)) && 15 | !startswith(element(split(":::", var.bucket_arn), 1), "xn--") && 16 | !endswith(element(split(":::", var.bucket_arn), 1), "-s3alias") 17 | ) 18 | error_message = "The 'bucket_arn' variable contains an invalid ARN" 19 | } 20 | } 21 | 22 | variable "k8s_labels" { 23 | description = <<-EOF 24 | (Optional) K8s labels common across all clusters, that you wish to include in the dataset. 25 | Possible values: A list of strings, each one should be a valid K8s label key. 26 | Default value: empty list ([]). 27 | EOF 28 | 29 | type = list(string) 30 | default = [] 31 | 32 | validation { 33 | condition = ( 34 | length([ 35 | for k8s_label in var.k8s_labels : k8s_label 36 | if can(regex("^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9-]*[a-zA-Z0-9])\\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9-]*[A-Za-z0-9])\\/[a-zA-Z0-9][-\\w.]{0,61}[a-zA-Z0-9]$|^[a-zA-Z0-9][-\\w.]{0,61}[a-zA-Z0-9]$", k8s_label)) 37 | ]) == length(var.k8s_labels) 38 | ) 39 | error_message = "At least one of the items the 'k8s_labels' list, contains an invalid K8s label key" 40 | } 41 | } 42 | 43 | variable "k8s_annotations" { 44 | description = <<-EOF 45 | (Optional) K8s annotations common across all clusters, that you wish to include in the dataset. 46 | Possible values: A list of strings, each one should be a valid K8s annotation key. 47 | Default value: empty list ([]). 48 | EOF 49 | 50 | type = list(string) 51 | default = [] 52 | 53 | validation { 54 | condition = ( 55 | length([ 56 | for k8s_annotation in var.k8s_annotations : k8s_annotation 57 | if can(regex("^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9-]*[a-zA-Z0-9])\\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9-]*[A-Za-z0-9])\\/[a-zA-Z0-9][-\\w.]{0,61}[a-zA-Z0-9]$|^[a-zA-Z0-9][-\\w.]{0,61}[a-zA-Z0-9]$", k8s_annotation)) 58 | ]) == length(var.k8s_annotations) 59 | ) 60 | error_message = "At least one of the items the 'k8s_annotations' list, contains an invalid K8s annotation key" 61 | } 62 | } 63 | 64 | variable "aws_common_tags" { 65 | description = <<-EOF 66 | (Optional) Common AWS tags to be used on all AWS resources created by Terraform. 67 | Possible values: Each field in the map must have a valid AWS tag key and an optional value. 68 | Default value: empty map ({}). 69 | EOF 70 | 71 | type = map(any) 72 | default = {} 73 | } 74 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/main.tf: -------------------------------------------------------------------------------- 1 | # This is the main entry point file where the calling modules are defined 2 | # Follow the sections and the comments inside them, which provide instructions 3 | 4 | terraform { 5 | required_version = ">= 1.3.0" 6 | required_providers { 7 | aws = { 8 | source = "hashicorp/aws" 9 | version = "~> 5.26.0" 10 | } 11 | helm = { 12 | source = "hashicorp/helm" 13 | version = "~> 2.9.0" 14 | } 15 | local = { 16 | source = "hashicorp/local" 17 | version = "~> 2.4.0" 18 | } 19 | } 20 | } 21 | 22 | ###################################### 23 | # Section 1 - AWS Pipeline Resources # 24 | ###################################### 25 | 26 | # Calling module for the pipeline module, to create the AWS pipeline resources 27 | module "pipeline" { 28 | source = "./modules/pipeline" 29 | 30 | # # 31 | # Root Module Common Variables # 32 | # # 33 | 34 | # References to root module common variables, do not remove or change 35 | 36 | bucket_arn = var.bucket_arn 37 | k8s_labels = var.k8s_labels 38 | k8s_annotations = var.k8s_annotations 39 | aws_common_tags = var.aws_common_tags 40 | 41 | # # 42 | # Pipeline Module Variables # 43 | # # 44 | 45 | # Provide optional pipeline module variables values here, if needed 46 | 47 | } 48 | 49 | ######################################################### 50 | # Section 2 - Data Collection Pod Deployment using Helm # 51 | ######################################################### 52 | 53 | # Calling modules for the kubecost_s3_exporter module. 54 | # Deploys the K8s resources on clusters, and creates IRSA in cluster's accounts 55 | # There are 2 deployment options: 56 | # 57 | # 1. Deploy the K8s resources by having Terraform invoke Helm 58 | # This option is shown in the "cluster1" calling module example 59 | # 2. Deploy the K8s resources by having Terraform generate a Helm values.yaml, then you deploy it using Helm 60 | # This option is shown in the "cluster2" calling module example 61 | 62 | # Example calling module for cluster with Helm invocation 63 | # Use it if you'd like Terraform to invoke Helm to deploy the K8s resources 64 | # Replace "cluster1" with a unique name to identify the cluster 65 | # Duplicate the calling module for each cluster on which you wish to deploy the Kubecost S3 Exporter 66 | module "cluster1" { 67 | 68 | # This is an example, to help you get started 69 | 70 | source = "./modules/kubecost_s3_exporter" 71 | 72 | providers = { 73 | aws.pipeline = aws 74 | aws.eks = aws.us-east-1-111111111111-cluster1 # Replace with the AWS provider alias for the cluster 75 | helm = helm.us-east-1-111111111111-cluster1 # Replace with the Helm provider alias for the cluster 76 | } 77 | 78 | # # 79 | # Root Module Common Variables # 80 | # # 81 | 82 | # References to root module common variables 83 | # Always include when creating new calling module, and do not remove or change 84 | 85 | bucket_arn = var.bucket_arn 86 | k8s_labels = var.k8s_labels 87 | k8s_annotations = var.k8s_annotations 88 | aws_common_tags = var.aws_common_tags 89 | 90 | # # 91 | # Kubecost S3 Exporter Module Variables # 92 | # # 93 | 94 | # Provide kubecost_s3_exporter module variables values here 95 | 96 | cluster_arn = "" # Add the EKS cluster ARN here 97 | kubecost_s3_exporter_container_image = "" # Add the Kubecost S3 Exporter container image here (example: 111111111111.dkr.ecr.us-east-1.amazonaws.com/kubecost_s3_exporter:0.1.0) 98 | } 99 | 100 | # Example calling module for cluster without Helm invocation 101 | # Use it if you'd like Terraform to generate a Helm values.yaml, then you deploy it using Helm 102 | # Replace "cluster2" with a unique name to identify the cluster 103 | # Duplicate the calling module for each cluster on which you wish to deploy the Kubecost S3 Exporter 104 | module "cluster2" { 105 | 106 | # This is an example, to help you get started 107 | 108 | source = "./modules/kubecost_s3_exporter" 109 | 110 | providers = { 111 | aws.pipeline = aws 112 | aws.eks = aws.us-east-1-111111111111-cluster2 # Replace with the AWS provider alias for the cluster 113 | } 114 | 115 | # # 116 | # Root Module Common Variables # 117 | # # 118 | 119 | # References to root module common variables 120 | # Always include when creating new calling module, and do not remove or change 121 | 122 | bucket_arn = var.bucket_arn 123 | k8s_labels = var.k8s_labels 124 | k8s_annotations = var.k8s_annotations 125 | aws_common_tags = var.aws_common_tags 126 | 127 | # # 128 | # Kubecost S3 Exporter Module Variables # 129 | # # 130 | 131 | # Provide kubecost_s3_exporter module variables values here 132 | 133 | cluster_arn = "" # Add the EKS cluster ARN here 134 | kubecost_s3_exporter_container_image = "" # Add the Kubecost S3 Exporter container image here (example: 111111111111.dkr.ecr.us-east-1.amazonaws.com/kubecost_s3_exporter:0.1.0) 135 | invoke_helm = false 136 | } 137 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/modules/common_locals/locals.tf: -------------------------------------------------------------------------------- 1 | locals { 2 | 3 | # The below local variable is used to define the static columns of the schema 4 | # It maps each column to hive, presto and QuickSight dataset data types 5 | # The hive type is used in AWS Glue table, the presto type is used in the Athena view 6 | static_columns = [ 7 | { 8 | name = "name" 9 | qs_data_set_type = "STRING" 10 | hive_type = "string" 11 | presto_type = "varchar" 12 | }, 13 | { 14 | name = "window.start" 15 | qs_data_set_type = "DATETIME" 16 | hive_type = "timestamp" 17 | presto_type = "timestamp" 18 | }, 19 | { 20 | name = "window.end" 21 | qs_data_set_type = "DATETIME" 22 | hive_type = "timestamp" 23 | presto_type = "timestamp" 24 | }, 25 | { 26 | name = "minutes" 27 | qs_data_set_type = "DECIMAL" 28 | hive_type = "double" 29 | presto_type = "double" 30 | }, 31 | { 32 | name = "cpucores" 33 | qs_data_set_type = "DECIMAL" 34 | hive_type = "double" 35 | presto_type = "double" 36 | }, 37 | { 38 | name = "cpucorerequestaverage" 39 | qs_data_set_type = "DECIMAL" 40 | hive_type = "double" 41 | presto_type = "double" 42 | }, 43 | { 44 | name = "cpucoreusageaverage" 45 | qs_data_set_type = "DECIMAL" 46 | hive_type = "double" 47 | presto_type = "double" 48 | }, 49 | { 50 | name = "cpucorehours" 51 | qs_data_set_type = "DECIMAL" 52 | hive_type = "double" 53 | presto_type = "double" 54 | }, 55 | { 56 | name = "cpucost" 57 | qs_data_set_type = "DECIMAL" 58 | hive_type = "double" 59 | presto_type = "double" 60 | }, 61 | { 62 | name = "cpucostadjustment" 63 | qs_data_set_type = "DECIMAL" 64 | hive_type = "double" 65 | presto_type = "double" 66 | }, 67 | { 68 | name = "cpuefficiency" 69 | qs_data_set_type = "DECIMAL" 70 | hive_type = "double" 71 | presto_type = "double" 72 | }, 73 | { 74 | name = "gpucount" 75 | qs_data_set_type = "DECIMAL" 76 | hive_type = "double" 77 | presto_type = "double" 78 | }, 79 | { 80 | name = "gpuhours" 81 | qs_data_set_type = "DECIMAL" 82 | hive_type = "double" 83 | presto_type = "double" 84 | }, 85 | { 86 | name = "gpucost" 87 | qs_data_set_type = "DECIMAL" 88 | hive_type = "double" 89 | presto_type = "double" 90 | }, 91 | { 92 | name = "gpucostadjustment" 93 | qs_data_set_type = "DECIMAL" 94 | hive_type = "double" 95 | presto_type = "double" 96 | }, 97 | { 98 | name = "networktransferbytes" 99 | qs_data_set_type = "DECIMAL" 100 | hive_type = "double" 101 | presto_type = "double" 102 | }, 103 | { 104 | name = "networkreceivebytes" 105 | qs_data_set_type = "DECIMAL" 106 | hive_type = "double" 107 | presto_type = "double" 108 | }, 109 | { 110 | name = "networkcost" 111 | qs_data_set_type = "DECIMAL" 112 | hive_type = "double" 113 | presto_type = "double" 114 | }, 115 | { 116 | name = "networkcrosszonecost" 117 | qs_data_set_type = "DECIMAL" 118 | hive_type = "double" 119 | presto_type = "double" 120 | }, 121 | { 122 | name = "networkcrossregioncost" 123 | qs_data_set_type = "DECIMAL" 124 | hive_type = "double" 125 | presto_type = "double" 126 | }, 127 | { 128 | name = "networkinternetcost" 129 | qs_data_set_type = "DECIMAL" 130 | hive_type = "double" 131 | presto_type = "double" 132 | }, 133 | { 134 | name = "networkcostadjustment" 135 | qs_data_set_type = "DECIMAL" 136 | hive_type = "double" 137 | presto_type = "double" 138 | }, 139 | { 140 | name = "loadbalancercost" 141 | qs_data_set_type = "DECIMAL" 142 | hive_type = "double" 143 | presto_type = "double" 144 | }, 145 | { 146 | name = "loadbalancercostadjustment" 147 | qs_data_set_type = "DECIMAL" 148 | hive_type = "double" 149 | presto_type = "double" 150 | }, 151 | { 152 | name = "pvbytes" 153 | qs_data_set_type = "DECIMAL" 154 | hive_type = "double" 155 | presto_type = "double" 156 | }, 157 | { 158 | name = "pvbytehours" 159 | qs_data_set_type = "DECIMAL" 160 | hive_type = "double" 161 | presto_type = "double" 162 | }, 163 | { 164 | name = "pvcost" 165 | qs_data_set_type = "DECIMAL" 166 | hive_type = "double" 167 | presto_type = "double" 168 | }, 169 | { 170 | name = "pvcostadjustment" 171 | qs_data_set_type = "DECIMAL" 172 | hive_type = "double" 173 | presto_type = "double" 174 | }, 175 | { 176 | name = "rambytes" 177 | qs_data_set_type = "DECIMAL" 178 | hive_type = "double" 179 | presto_type = "double" 180 | }, 181 | { 182 | name = "rambyterequestaverage" 183 | qs_data_set_type = "DECIMAL" 184 | hive_type = "double" 185 | presto_type = "double" 186 | }, 187 | { 188 | name = "rambyteusageaverage" 189 | qs_data_set_type = "DECIMAL" 190 | hive_type = "double" 191 | presto_type = "double" 192 | }, 193 | { 194 | name = "rambytehours" 195 | qs_data_set_type = "DECIMAL" 196 | hive_type = "double" 197 | presto_type = "double" 198 | }, 199 | { 200 | name = "ramcost" 201 | qs_data_set_type = "DECIMAL" 202 | hive_type = "double" 203 | presto_type = "double" 204 | }, 205 | { 206 | name = "ramcostadjustment" 207 | qs_data_set_type = "DECIMAL" 208 | hive_type = "double" 209 | presto_type = "double" 210 | }, 211 | { 212 | name = "ramefficiency" 213 | qs_data_set_type = "DECIMAL" 214 | hive_type = "double" 215 | presto_type = "double" 216 | }, 217 | { 218 | name = "sharedcost" 219 | qs_data_set_type = "DECIMAL" 220 | hive_type = "double" 221 | presto_type = "double" 222 | }, 223 | { 224 | name = "externalcost" 225 | qs_data_set_type = "DECIMAL" 226 | hive_type = "double" 227 | presto_type = "double" 228 | }, 229 | { 230 | name = "totalcost" 231 | qs_data_set_type = "DECIMAL" 232 | hive_type = "double" 233 | presto_type = "double" 234 | }, 235 | { 236 | name = "totalefficiency" 237 | qs_data_set_type = "DECIMAL" 238 | hive_type = "double" 239 | presto_type = "double" 240 | }, 241 | { 242 | name = "properties.provider" 243 | qs_data_set_type = "STRING" 244 | hive_type = "string" 245 | presto_type = "varchar" 246 | }, 247 | { 248 | name = "properties.region" 249 | qs_data_set_type = "STRING" 250 | hive_type = "string" 251 | presto_type = "varchar" 252 | }, 253 | { 254 | name = "properties.cluster" 255 | qs_data_set_type = "STRING" 256 | hive_type = "string" 257 | presto_type = "varchar" 258 | }, 259 | { 260 | name = "properties.clusterid" 261 | qs_data_set_type = "STRING" 262 | hive_type = "string" 263 | presto_type = "varchar" 264 | }, 265 | { 266 | name = "properties.eksclustername" 267 | qs_data_set_type = "STRING" 268 | hive_type = "string" 269 | presto_type = "varchar" 270 | }, 271 | { 272 | name = "properties.container" 273 | qs_data_set_type = "STRING" 274 | hive_type = "string" 275 | presto_type = "varchar" 276 | }, 277 | { 278 | name = "properties.namespace" 279 | qs_data_set_type = "STRING" 280 | hive_type = "string" 281 | presto_type = "varchar" 282 | }, 283 | { 284 | name = "properties.pod" 285 | qs_data_set_type = "STRING" 286 | hive_type = "string" 287 | presto_type = "varchar" 288 | }, 289 | { 290 | name = "properties.node" 291 | qs_data_set_type = "STRING" 292 | hive_type = "string" 293 | presto_type = "varchar" 294 | }, 295 | { 296 | name = "properties.node_instance_type" 297 | qs_data_set_type = "STRING" 298 | hive_type = "string" 299 | presto_type = "varchar" 300 | }, 301 | { 302 | name = "properties.node_availability_zone" 303 | qs_data_set_type = "STRING" 304 | hive_type = "string" 305 | presto_type = "varchar" 306 | }, 307 | { 308 | name = "properties.node_capacity_type" 309 | qs_data_set_type = "STRING" 310 | hive_type = "string" 311 | presto_type = "varchar" 312 | }, 313 | { 314 | name = "properties.node_architecture" 315 | qs_data_set_type = "STRING" 316 | hive_type = "string" 317 | presto_type = "varchar" 318 | }, 319 | { 320 | name = "properties.node_os" 321 | qs_data_set_type = "STRING" 322 | hive_type = "string" 323 | presto_type = "varchar" 324 | }, 325 | { 326 | name = "properties.node_nodegroup" 327 | qs_data_set_type = "STRING" 328 | hive_type = "string" 329 | presto_type = "varchar" 330 | }, 331 | { 332 | name = "properties.node_nodegroup_image" 333 | qs_data_set_type = "STRING" 334 | hive_type = "string" 335 | presto_type = "varchar" 336 | }, 337 | { 338 | name = "properties.controller" 339 | qs_data_set_type = "STRING" 340 | hive_type = "string" 341 | presto_type = "varchar" 342 | }, 343 | { 344 | name = "properties.controllerkind" 345 | qs_data_set_type = "STRING" 346 | hive_type = "string" 347 | presto_type = "varchar" 348 | }, 349 | { 350 | name = "properties.providerid" 351 | qs_data_set_type = "STRING" 352 | hive_type = "string" 353 | presto_type = "varchar" 354 | } 355 | ] 356 | 357 | # The below local is used to define the partition keys of the schema that is used in both the AWS Glue Table and the QuickSight Dataset 358 | partition_keys = [ 359 | { 360 | name = "account_id" 361 | qs_data_set_type = "STRING" 362 | hive_type = "string" 363 | presto_type = "varchar" 364 | }, 365 | { 366 | name = "region" 367 | qs_data_set_type = "STRING" 368 | hive_type = "string" 369 | presto_type = "varchar" 370 | }, 371 | { 372 | name = "year" 373 | qs_data_set_type = "STRING" 374 | hive_type = "string" 375 | presto_type = "varchar" 376 | }, 377 | { 378 | name = "month" 379 | qs_data_set_type = "STRING" 380 | hive_type = "string" 381 | presto_type = "varchar" 382 | } 383 | ] 384 | } 385 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/modules/common_locals/outputs.tf: -------------------------------------------------------------------------------- 1 | output "static_columns" { 2 | description = "A list of the schema's static columns, mapped to their QuickSight Dataset types, hive types and presto types" 3 | value = local.static_columns 4 | } 5 | 6 | output "partition_keys" { 7 | description = "A list of the schema's partition keys, mapped to their QuickSight Dataset types, hive types and presto types" 8 | value = local.partition_keys 9 | } 10 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/modules/kubecost_s3_exporter/locals.tf: -------------------------------------------------------------------------------- 1 | locals { 2 | bucket_name = element(split(":::", var.bucket_arn), 1) 3 | cluster_name = element(split("/", data.aws_arn.eks_cluster.resource), 1) 4 | cluster_oidc_provider_id = element(split("/", data.aws_iam_openid_connect_provider.this.arn), 3) 5 | pipeline_partition = element(split(":", data.aws_caller_identity.pipeline.arn), 1) 6 | kubecost_ca_certificate_secret_arn = length(var.kubecost_ca_certificate_secrets) > 0 ? lookup(element(var.kubecost_ca_certificate_secrets, index(var.kubecost_ca_certificate_secrets.*.name, var.kubecost_ca_certificate_secret_name)), "arn", "") : "" 7 | helm_chart_location = "${path.module}/../../../../helm/kubecost_s3_exporter" 8 | helm_values_yaml = yamlencode( 9 | { 10 | "namespace" : var.namespace 11 | "image" : var.kubecost_s3_exporter_container_image 12 | "imagePullPolicy" : var.kubecost_s3_exporter_container_image_pull_policy 13 | "ephemeralVolumeSize" : var.kubecost_s3_exporter_ephemeral_volume_size 14 | "cronJob" : { 15 | "name" : "kubecost-s3-exporter", 16 | "schedule" : var.kubecost_s3_exporter_cronjob_schedule 17 | } 18 | "serviceAccount" : { 19 | "name" : var.service_account 20 | "create" : var.create_service_account 21 | "role" : length(aws_iam_role.kubecost_s3_exporter_irsa_child) > 0 ? aws_iam_role.kubecost_s3_exporter_irsa_child[0].arn : aws_iam_role.kubecost_s3_exporter_irsa[0].arn 22 | } 23 | "env" : [ 24 | { 25 | "name" : "S3_BUCKET_NAME", 26 | "value" : local.bucket_name 27 | }, 28 | { 29 | "name" : "KUBECOST_API_ENDPOINT", 30 | "value" : var.kubecost_api_endpoint 31 | }, 32 | { 33 | "name" : "BACKFILL_PERIOD_DAYS", 34 | "value" : var.backfill_period_days 35 | }, 36 | { 37 | "name" : "CLUSTER_ID", 38 | "value" : var.cluster_arn 39 | }, 40 | { 41 | "name" : "IRSA_PARENT_IAM_ROLE_ARN", 42 | "value" : length(aws_iam_role.kubecost_s3_exporter_irsa_parent) > 0 ? aws_iam_role.kubecost_s3_exporter_irsa_parent[0].arn : "" 43 | }, 44 | { 45 | "name" : "AGGREGATION", 46 | "value" : var.aggregation 47 | }, 48 | { 49 | "name" : "KUBECOST_ALLOCATION_API_PAGINATE", 50 | "value" : var.kubecost_allocation_api_paginate 51 | }, 52 | { 53 | "name" : "CONNECTION_TIMEOUT", 54 | "value" : var.connection_timeout 55 | }, 56 | { 57 | "name" : "KUBECOST_ALLOCATION_API_READ_TIMEOUT", 58 | "value" : var.kubecost_allocation_api_read_timeout 59 | }, 60 | { 61 | "name" : "TLS_VERIFY", 62 | "value" : var.tls_verify 63 | }, 64 | { 65 | "name" : "KUBECOST_CA_CERTIFICATE_SECRET_NAME", 66 | "value" : length(local.kubecost_ca_certificate_secret_arn) > 0 ? var.kubecost_ca_certificate_secret_name : "" 67 | }, 68 | { 69 | "name" : "KUBECOST_CA_CERTIFICATE_SECRET_REGION", 70 | "value" : length(local.kubecost_ca_certificate_secret_arn) > 0 ? element(split(":", local.kubecost_ca_certificate_secret_arn), 3) : "" 71 | }, 72 | { 73 | "name" : "LABELS", 74 | "value" : join(", ", distinct(var.k8s_labels)) 75 | }, 76 | { 77 | "name" : "ANNOTATIONS", 78 | "value" : join(", ", distinct(var.k8s_annotations)) 79 | }, 80 | { 81 | "name" : "PYTHONUNBUFFERED", 82 | "value" : "1" 83 | } 84 | ] 85 | } 86 | ) 87 | } 88 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/modules/kubecost_s3_exporter/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_version = ">= 1.3.0" 3 | required_providers { 4 | aws = { 5 | source = "hashicorp/aws" 6 | version = ">= 4.63.0" 7 | configuration_aliases = [aws.pipeline, aws.eks] 8 | } 9 | helm = { 10 | source = "hashicorp/helm" 11 | version = ">= 2.9.0" 12 | } 13 | local = { 14 | source = "hashicorp/local" 15 | version = ">= 2.4.0" 16 | } 17 | } 18 | } 19 | 20 | data "aws_caller_identity" "pipeline" { 21 | provider = aws.pipeline 22 | } 23 | 24 | data "aws_caller_identity" "eks_cluster" { 25 | provider = aws.eks 26 | } 27 | 28 | data "aws_arn" "eks_cluster" { 29 | provider = aws.eks 30 | 31 | arn = var.cluster_arn 32 | } 33 | 34 | data "aws_eks_cluster" "this" { 35 | provider = aws.eks 36 | 37 | name = local.cluster_name 38 | } 39 | 40 | data "aws_iam_openid_connect_provider" "this" { 41 | provider = aws.eks 42 | 43 | url = data.aws_eks_cluster.this.identity.0.oidc.0.issuer 44 | } 45 | 46 | # The below resource is created conditionally, if the EKS cluster account ID and the pipeline account ID are the same 47 | # This means that a single IAM Role (IRSA) is created in the account, as cross-account authentication isn't required 48 | resource "aws_iam_role" "kubecost_s3_exporter_irsa" { 49 | provider = aws.eks 50 | 51 | count = data.aws_caller_identity.eks_cluster.account_id == data.aws_caller_identity.pipeline.account_id ? 1 : 0 52 | 53 | name = "kubecost_s3_exporter_irsa_${local.cluster_oidc_provider_id}" 54 | assume_role_policy = jsonencode( 55 | { 56 | Version = "2012-10-17" 57 | Statement = [ 58 | { 59 | Effect = "Allow" 60 | Principal = { 61 | Federated = data.aws_iam_openid_connect_provider.this.arn 62 | } 63 | Action = "sts:AssumeRoleWithWebIdentity" 64 | Condition = { 65 | StringEquals = { 66 | "${element(split(":oidc-provider/", data.aws_iam_openid_connect_provider.this.arn), 1)}:aud" = "sts.amazonaws.com" 67 | "${element(split(":oidc-provider/", data.aws_iam_openid_connect_provider.this.arn), 1)}:sub" = "system:serviceaccount:${var.namespace}:${var.service_account}" 68 | } 69 | } 70 | } 71 | ] 72 | } 73 | ) 74 | 75 | inline_policy { 76 | name = "kubecost_s3_exporter_parent_put_object" 77 | policy = jsonencode( 78 | { 79 | Statement = [ 80 | { 81 | Action = "s3:PutObject" 82 | Effect = "Allow" 83 | Resource = "${var.bucket_arn}/account_id=${data.aws_arn.eks_cluster.account}/region=${data.aws_arn.eks_cluster.region}/year=*/month=*/*_${local.cluster_name}.snappy.parquet" 84 | } 85 | ] 86 | Version = "2012-10-17" 87 | } 88 | ) 89 | } 90 | 91 | inline_policy { 92 | name = "kubecost_s3_exporter_parent_list_bucket" 93 | policy = jsonencode( 94 | { 95 | Statement = [ 96 | { 97 | Action = "s3:ListBucket" 98 | Effect = "Allow" 99 | Resource = var.bucket_arn 100 | } 101 | ] 102 | Version = "2012-10-17" 103 | } 104 | ) 105 | } 106 | 107 | # The below inline policy is conditionally created 108 | # If the "kubecost_ca_certificate_secret_arn" local contains a value, the below inline policy is added 109 | # Else, it won't be added 110 | dynamic "inline_policy" { 111 | for_each = length(local.kubecost_ca_certificate_secret_arn) > 0 ? [1] : [] 112 | content { 113 | name = "kubecost_s3_exporter_parent_get_secret_value" 114 | policy = jsonencode( 115 | { 116 | Statement = [ 117 | { 118 | Action = "secretsmanager:GetSecretValue" 119 | Effect = "Allow" 120 | Resource = local.kubecost_ca_certificate_secret_arn 121 | } 122 | ] 123 | Version = "2012-10-17" 124 | } 125 | ) 126 | } 127 | } 128 | 129 | tags = { 130 | irsa-kubecost-s3-exporter = "true" 131 | irsa-kubecost-s3-exporter-sm = length(local.kubecost_ca_certificate_secret_arn) > 0 ? "true" : "false" 132 | } 133 | } 134 | 135 | # The below 2 resources are created conditionally, if the EKS cluster account ID and the pipeline account ID are different 136 | # This means that a child IAM Role (IRSA) is created in the EKS account, and a parent IAM role is created in the pipeline accoubt 137 | # This is for cross-account authentication using IAM Role Chaining 138 | resource "aws_iam_role" "kubecost_s3_exporter_irsa_child" { 139 | provider = aws.eks 140 | 141 | count = data.aws_caller_identity.eks_cluster.account_id != data.aws_caller_identity.pipeline.account_id ? 1 : 0 142 | 143 | name = "kubecost_s3_exporter_irsa_${local.cluster_oidc_provider_id}" 144 | assume_role_policy = jsonencode( 145 | { 146 | Version = "2012-10-17" 147 | Statement = [ 148 | { 149 | Effect = "Allow" 150 | Principal = { 151 | Federated = data.aws_iam_openid_connect_provider.this.arn 152 | } 153 | Action = "sts:AssumeRoleWithWebIdentity" 154 | Condition = { 155 | StringEquals = { 156 | "${element(split(":oidc-provider/", data.aws_iam_openid_connect_provider.this.arn), 1)}:aud" = "sts.amazonaws.com" 157 | "${element(split(":oidc-provider/", data.aws_iam_openid_connect_provider.this.arn), 1)}:sub" = "system:serviceaccount:${var.namespace}:${var.service_account}" 158 | } 159 | } 160 | } 161 | ] 162 | } 163 | ) 164 | 165 | inline_policy { 166 | name = "kubecost_s3_exporter_irsa_${local.cluster_oidc_provider_id}" 167 | policy = jsonencode( 168 | { 169 | Statement = [ 170 | { 171 | Action = "sts:AssumeRole" 172 | Effect = "Allow" 173 | Resource = "arn:${local.pipeline_partition}:iam::${data.aws_caller_identity.pipeline.account_id}:role/kubecost_s3_exporter_parent_${local.cluster_oidc_provider_id}" 174 | } 175 | ] 176 | Version = "2012-10-17" 177 | } 178 | ) 179 | } 180 | } 181 | 182 | resource "aws_iam_role" "kubecost_s3_exporter_irsa_parent" { 183 | provider = aws.pipeline 184 | 185 | count = data.aws_caller_identity.eks_cluster.account_id != data.aws_caller_identity.pipeline.account_id ? 1 : 0 186 | 187 | name = "kubecost_s3_exporter_parent_${local.cluster_oidc_provider_id}" 188 | assume_role_policy = jsonencode( 189 | { 190 | Version = "2012-10-17" 191 | Statement = [ 192 | { 193 | Effect = "Allow" 194 | Principal = { 195 | AWS = aws_iam_role.kubecost_s3_exporter_irsa_child[0].arn 196 | } 197 | Action = "sts:AssumeRole" 198 | } 199 | ] 200 | } 201 | ) 202 | 203 | inline_policy { 204 | name = "kubecost_s3_exporter_parent_put_object" 205 | policy = jsonencode( 206 | { 207 | Statement = [ 208 | { 209 | Action = "s3:PutObject" 210 | Effect = "Allow" 211 | Resource = "${var.bucket_arn}/account_id=${data.aws_arn.eks_cluster.account}/region=${data.aws_arn.eks_cluster.region}/year=*/month=*/*_${local.cluster_name}.snappy.parquet" 212 | } 213 | ] 214 | Version = "2012-10-17" 215 | } 216 | ) 217 | } 218 | 219 | inline_policy { 220 | name = "kubecost_s3_exporter_parent_list_bucket" 221 | policy = jsonencode( 222 | { 223 | Statement = [ 224 | { 225 | Action = "s3:ListBucket" 226 | Effect = "Allow" 227 | Resource = var.bucket_arn 228 | } 229 | ] 230 | Version = "2012-10-17" 231 | } 232 | ) 233 | } 234 | 235 | # The below inline policy is conditionally created 236 | # If the "kubecost_ca_certificate_secret_arn" local contains a value, the below inline policy is added 237 | # Else, it won't be added 238 | dynamic "inline_policy" { 239 | for_each = length(local.kubecost_ca_certificate_secret_arn) > 0 ? [1] : [] 240 | content { 241 | name = "kubecost_s3_exporter_parent_get_secret_value" 242 | policy = jsonencode( 243 | { 244 | Statement = [ 245 | { 246 | Action = "secretsmanager:GetSecretValue" 247 | Effect = "Allow" 248 | Resource = local.kubecost_ca_certificate_secret_arn 249 | } 250 | ] 251 | Version = "2012-10-17" 252 | } 253 | ) 254 | } 255 | } 256 | 257 | tags = { 258 | irsa-kubecost-s3-exporter = "true" 259 | irsa-kubecost-s3-exporter-sm = length(local.kubecost_ca_certificate_secret_arn) > 0 ? "true" : "false" 260 | } 261 | } 262 | 263 | # The below 2 resources are conditionally created 264 | # If the "invoke_helm" variable is set, the first resource ("helm_release") is created 265 | # This deploys the K8s resources (data collection pod and service account) in the cluster by invoking Helm 266 | # Else, the second resource ("local_file") is created 267 | # This will NOT invoke Helm to deploy the K8s resources (data collection pod and service account) in the cluster 268 | # Instead, it'll create a local values.yaml file in the Helm chart's directory, to be used by the user to deploy the K8s using the "helm" command 269 | # The local file name will be "___values.yaml", so it'll be unique 270 | resource "helm_release" "kubecost_s3_exporter" { 271 | count = var.invoke_helm ? 1 : 0 272 | 273 | name = "kubecost-s3-exporter" 274 | chart = local.helm_chart_location 275 | namespace = var.namespace 276 | create_namespace = var.create_namespace 277 | values = [local.helm_values_yaml] 278 | } 279 | 280 | resource "local_file" "kubecost_s3_exporter_helm_values_yaml" { 281 | count = var.invoke_helm ? 0 : 1 282 | 283 | filename = "${local.helm_chart_location}/clusters_values/${data.aws_arn.eks_cluster.account}_${data.aws_arn.eks_cluster.region}_${local.cluster_name}_values.yaml" 284 | directory_permission = "0400" 285 | file_permission = "0400" 286 | content = <<-EOT 287 | ${local.helm_values_yaml} 288 | EOT 289 | } 290 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/modules/kubecost_s3_exporter/outputs.tf: -------------------------------------------------------------------------------- 1 | output "irsa" { 2 | description = <= 1. For example, 10Mi, 50Mi, 100Mi, 150Mi. 167 | Default value: 50Mi 168 | EOF 169 | 170 | type = string 171 | default = "50Mi" 172 | 173 | validation { 174 | condition = can(regex("^[1-9]\\d?Mi.*$", var.kubecost_s3_exporter_ephemeral_volume_size)) 175 | error_message = "The 'kubecost_s3_exporter_ephemeral_volume_size' variable must be in format of 'NMi', where N >= 1.\nFor example, 10Mi, 50Mi, 100Mi, 150Mi." 176 | } 177 | } 178 | 179 | variable "kubecost_api_endpoint" { 180 | description = <<-EOF 181 | (Optional) The Kubecost API endpoint in format of 'http://:'. 182 | Possible values: URI in the format of 'http://:[port]' or 'https://:[port]'. 183 | Default value: http://kubecost-cost-analyzer.kubecost:9090 184 | EOF 185 | 186 | type = string 187 | default = "http://kubecost-cost-analyzer.kubecost:9090" 188 | 189 | validation { 190 | condition = can(regex("^https?://.+$", var.kubecost_api_endpoint)) 191 | error_message = "The Kubecost API endpoint is invalid. It must be in the format of 'http://:[port]' or 'https://:[port]'" 192 | } 193 | } 194 | 195 | variable "backfill_period_days" { 196 | description = <<-EOF 197 | (Optional) The number of days to check for backfilling. 198 | Possible values: A positive integer equal or larger than 3. 199 | Default value: 15 200 | EOF 201 | 202 | type = number 203 | default = 15 204 | 205 | validation { 206 | condition = var.backfill_period_days >= 3 207 | error_message = "The 'backfill_period_days' variable must be a positive integer equal to or larger than 3" 208 | } 209 | } 210 | 211 | variable "aggregation" { 212 | description = <<-EOF 213 | (Optional) The aggregation to use for returning the Kubecost Allocation API results. 214 | Possible values: "container", "pod", "namespace", "controller", "controllerKind", "node" or "cluster". 215 | Default value: container 216 | EOF 217 | 218 | type = string 219 | default = "container" 220 | 221 | validation { 222 | condition = contains(["container", "pod", "namespace", "controller", "controllerKind", "node", "cluster"], var.aggregation) 223 | error_message = "The 'aggregation' variable includes an invalid value. It should be one of 'container', 'pod', 'namespace', 'controller', 'controllerKind', 'node', or 'cluster'" 224 | } 225 | } 226 | 227 | variable "kubecost_allocation_api_paginate" { 228 | description = <<-EOF 229 | (Optional) Dictates whether to paginate using 1-hour time ranges (relevant for 1h step). 230 | Possible values: "Yes", "No", "Y", "N", "True" or "False" 231 | Default value: False 232 | EOF 233 | 234 | type = string 235 | default = "False" 236 | 237 | validation { 238 | condition = can(regex("^(?i)(Yes|No|Y|N|True|False)$", var.kubecost_allocation_api_paginate)) 239 | error_message = "The 'kubecost_allocation_api_paginate' variable must be one of 'Yes', 'No', 'Y', 'N', 'True' or 'False' (case-insensitive)" 240 | } 241 | } 242 | 243 | variable "connection_timeout" { 244 | description = <<-EOF 245 | (Optional) The time (in seconds) to wait for TCP connection establishment. 246 | Possible values: A non-zero positive integer. 247 | Default value: 10 248 | EOF 249 | 250 | type = number 251 | default = 10 252 | 253 | validation { 254 | condition = var.connection_timeout > 0 255 | error_message = "The connection timeout must be a non-zero positive integer" 256 | } 257 | } 258 | 259 | variable "kubecost_allocation_api_read_timeout" { 260 | description = <<-EOF 261 | (Optional) The time (in seconds) to wait for the Kubecost Allocation API to send an HTTP response. 262 | Possible values: A non-zero positive integer. 263 | Default value: 60 264 | EOF 265 | 266 | type = number 267 | default = 60 268 | 269 | validation { 270 | condition = var.kubecost_allocation_api_read_timeout > 0 271 | error_message = "The read timeout must be a non-zero positive float" 272 | } 273 | } 274 | 275 | variable "tls_verify" { 276 | description = <<-EOF 277 | (Optional) Dictates whether TLS certificate verification is done for HTTPS connections. 278 | Possible values: "Yes", "No", "Y", "N", "True" or "False" 279 | Default value: True 280 | EOF 281 | 282 | type = string 283 | default = "True" 284 | 285 | validation { 286 | condition = can(regex("^(?i)(Yes|No|Y|N|True|False)$", var.tls_verify)) 287 | error_message = "The 'tls_verify' variable must be one of 'Yes', 'No', 'Y', 'N', 'True' or 'False' (case-insensitive)" 288 | } 289 | } 290 | 291 | variable "kubecost_ca_certificate_secret_name" { 292 | description = <<-EOF 293 | (Optional) The AWS Secrets Manager secret name, for the CA certificate used for verifying Kubecost's server certificate when using HTTPS. 294 | Possible values: A valid AWS Secrets Manager secret name. 295 | Default value: empty string ("") 296 | EOF 297 | 298 | type = string 299 | default = "" 300 | 301 | validation { 302 | condition = can(regex("^$|^[\\w/+=.@-]{1,512}$", var.kubecost_ca_certificate_secret_name)) 303 | error_message = "The 'kubecost_ca_certificate_secret_name' variable contains an invalid secret name" 304 | } 305 | } 306 | 307 | variable "namespace" { 308 | description = <<-EOF 309 | (Optional) The namespace in which the Kubecost S3 Exporter pod and service account will be created. 310 | Possible values: A valid K8s namespace name. 311 | Default value: kubecost-s3-exporter 312 | EOF 313 | 314 | type = string 315 | default = "kubecost-s3-exporter" 316 | 317 | validation { 318 | condition = can(regex("^[a-z0-9]([-a-z0-9]{0,62}[a-z0-9])?$", var.namespace)) 319 | error_message = "The 'namespace' variable contains an invalid Namespace name" 320 | } 321 | } 322 | 323 | variable "create_namespace" { 324 | description = <<-EOF 325 | (Optional) Dictates whether to create the namespace as part of the Helm Chart deployment. 326 | Possible values: true or false. 327 | Default value: true 328 | EOF 329 | 330 | type = bool 331 | default = true 332 | } 333 | 334 | variable "service_account" { 335 | description = <<-EOF 336 | (Optional) The service account for the Kubecost S3 Exporter pod. 337 | Possible values: A valid K8s service account name. 338 | Default value: kubecost-s3-exporter 339 | EOF 340 | 341 | type = string 342 | default = "kubecost-s3-exporter" 343 | 344 | validation { 345 | condition = can(regex("^[a-z0-9]([-a-z0-9]{0,252}[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?){0,252}$", var.service_account)) 346 | error_message = "The 'service_account' variable contains an invalid Service Account name" 347 | } 348 | } 349 | 350 | variable "create_service_account" { 351 | description = <<-EOF 352 | (Optional) Dictates whether to create the service account as part of the Helm Chart deployment. 353 | Possible values: true or false. 354 | Default value: true 355 | EOF 356 | 357 | type = bool 358 | default = true 359 | } 360 | 361 | variable "invoke_helm" { 362 | description = <<-EOF 363 | (Optional) Dictates whether to invoke Helm to deploy the K8s resources (the kubecost-s3-exporter CronJob and the Service Account). 364 | Possible values: true or false. 365 | Default value: true 366 | EOF 367 | 368 | type = bool 369 | default = true 370 | } 371 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/modules/pipeline/README.md: -------------------------------------------------------------------------------- 1 | # Pipeline Module 2 | 3 | The Pipeline reusable module is used deploy the pipeline AWS resources for this solution. 4 | 5 | ## Define Provider 6 | 7 | In the [`providers.tf`](../../providers.tf) file in the root directory, you'll find a pre-created `aws` provider for the `pipeline` module: 8 | 9 | ##################################### 10 | # Section 1 - Pipeline AWS Provider # 11 | ##################################### 12 | 13 | # Provider for the pipeline module 14 | provider "aws" { 15 | 16 | # This is an example, to help you get started 17 | 18 | region = "us-east-1" # Change the region if necessary 19 | shared_config_files = ["~/.aws/config"] # Change the path to the shared config file, if necessary 20 | shared_credentials_files = ["~/.aws/credentials"] # Change the path to the shared credential file, if necessary 21 | profile = "pipeline_profile" # Change to the profile that will be used for the account and region where the pipeline resources will be deployed 22 | default_tags { 23 | tags = module.common_variables.aws_common_tags 24 | } 25 | } 26 | 27 | * Change the `region` field if needed 28 | * Change the `shared_config_files` and `shared_credentials_files` if needed 29 | * Change the `profile` field to the AWS Profile that Terraform should use to create the pipeline resources 30 | 31 | Examples can be found in the [`examples/root_module/providers.tf`](../../examples/root_module/providers.tf) file. 32 | 33 | ## Create a Calling Module and Provide Variables Values 34 | 35 | In the [`main.tf`](../../main.tf) file in the root directory, you'll find a pre-created `pipeline` calling module: 36 | 37 | ###################################### 38 | # Section 2 - AWS Pipeline Resources # 39 | ###################################### 40 | 41 | # Calling module for the pipeline module, to create the AWS pipeline resources 42 | module "pipeline" { 43 | source = "./modules/pipeline" 44 | 45 | # # 46 | # Common Module Variables # 47 | # # 48 | 49 | # References to variables outputs from the common module, do not remove or change 50 | 51 | bucket_arn = module.common_variables.bucket_arn 52 | k8s_labels = module.common_variables.k8s_labels 53 | k8s_annotations = module.common_variables.k8s_annotations 54 | aws_common_tags = module.common_variables.aws_common_tags 55 | 56 | # # 57 | # Pipeline Module Variables # 58 | # # 59 | 60 | # Provide optional pipeline module variables values here, if needed 61 | 62 | } 63 | 64 | Variables referenced from the `common_variables` module are already present, please do not change or remove them. 65 | The `pipeline` module's own variables are all optional. 66 | If you don't need to change one of the optional variables, you can leave the pre-created calling module as is. 67 | 68 | For more information on the variables, see this module's [`variables.tf`](variables.tf) file. 69 | For examples, see the [`examples/root_module/main.tf`](../../examples/root_module/main.tf) file. 70 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/modules/pipeline/locals.tf: -------------------------------------------------------------------------------- 1 | locals { 2 | 3 | bucket_name = element(split(":::", var.bucket_arn), 1) 4 | 5 | athena_view_sql = <<-EOF 6 | SELECT 7 | * 8 | FROM 9 | "${aws_glue_catalog_database.kubecost.name}"."${aws_glue_catalog_table.kubecost.name}" 10 | WHERE 11 | ( 12 | (current_date - INTERVAL '${var.athena_view_data_retention_months}' MONTH) <= "window.start" 13 | ) 14 | EOF 15 | 16 | static_columns = [for column in module.common_locals.static_columns : { name = column.name, type = column.presto_type }] 17 | labels_columns = [for column in var.k8s_labels : { name = "properties.labels.${column}", type = "varchar" }] 18 | annotations_columns = [for column in var.k8s_annotations : { name = "properties.annotations.${column}", type = "varchar" }] 19 | partition_keys_columns = [for column in module.common_locals.partition_keys : { name = column.name, type = column.presto_type }] 20 | 21 | presto_view = jsonencode({ 22 | originalSql = local.athena_view_sql, 23 | catalog = "awsdatacatalog", 24 | schema = var.glue_database_name, 25 | columns = concat(local.static_columns, local.labels_columns, local.annotations_columns, local.partition_keys_columns) 26 | }) 27 | 28 | } 29 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/modules/pipeline/main.tf: -------------------------------------------------------------------------------- 1 | module "common_locals" { 2 | source = "../common_locals" 3 | } 4 | 5 | terraform { 6 | required_version = ">= 1.3.0" 7 | required_providers { 8 | aws = { 9 | source = "hashicorp/aws" 10 | version = ">= 4.63.0" 11 | } 12 | } 13 | } 14 | 15 | data "aws_partition" "pipeline" {} 16 | data "aws_region" "pipeline" {} 17 | data "aws_caller_identity" "pipeline" {} 18 | 19 | resource "aws_iam_policy" "kubecost_glue_crawler" { 20 | name = "kubecost_glue_crawler_policy" 21 | policy = jsonencode( 22 | { 23 | Statement = [ 24 | { 25 | Action = [ 26 | "glue:GetTable", 27 | "glue:GetDatabase", 28 | "glue:GetPartition", 29 | "glue:CreatePartition", 30 | "glue:UpdatePartition", 31 | "glue:BatchGetPartition", 32 | "glue:BatchCreatePartition", 33 | "glue:BatchUpdatePartition" 34 | ] 35 | Effect = "Allow" 36 | Resource = [ 37 | "arn:${data.aws_partition.pipeline.partition}:glue:${data.aws_region.pipeline.name}:${data.aws_caller_identity.pipeline.account_id}:catalog", 38 | aws_glue_catalog_database.kubecost.arn, 39 | aws_glue_catalog_table.kubecost.arn, 40 | "${replace(aws_glue_catalog_table.kubecost.arn, aws_glue_catalog_table.kubecost.name, replace(local.bucket_name, "-", "_"))}*" 41 | ] 42 | Sid = "AllowGlueKubecostTable" 43 | }, 44 | { 45 | Action = [ 46 | "s3:GetObject", 47 | "s3:ListBucket", 48 | ] 49 | Effect = "Allow" 50 | Resource = [ 51 | "${var.bucket_arn}/*", 52 | var.bucket_arn, 53 | ] 54 | Sid = "AllowS3KubecostBucket" 55 | }, 56 | { 57 | Action = "logs:CreateLogGroup" 58 | Effect = "Allow" 59 | Resource = "arn:${data.aws_partition.pipeline.partition}:logs:${data.aws_region.pipeline.name}:${data.aws_caller_identity.pipeline.account_id}:*" 60 | Sid = "AllowCloudWatchLogsCreateLogGroupForGlueCrawlers" 61 | }, 62 | { 63 | Action = "logs:CreateLogStream" 64 | Effect = "Allow" 65 | Resource = "arn:${data.aws_partition.pipeline.partition}:logs:${data.aws_region.pipeline.name}:${data.aws_caller_identity.pipeline.account_id}:log-group:/aws-glue/crawlers:*" 66 | Sid = "AllowCloudWatchLogsCreateLogStreamForKubecostCrawler" 67 | }, 68 | { 69 | Action = "logs:PutLogEvents" 70 | Effect = "Allow" 71 | Resource = "arn:${data.aws_partition.pipeline.partition}:logs:${data.aws_region.pipeline.name}:${data.aws_caller_identity.pipeline.account_id}:log-group:/aws-glue/crawlers:log-stream:kubecost_crawler" 72 | Sid = "AllowCloudWatchLogsPutLogs" 73 | } 74 | ] 75 | Version = "2012-10-17" 76 | } 77 | ) 78 | } 79 | 80 | resource "aws_iam_role" "kubecost_glue_crawler" { 81 | name = "kubecost_glue_crawler_role" 82 | assume_role_policy = jsonencode( 83 | { 84 | Statement = [ 85 | { 86 | Action = "sts:AssumeRole" 87 | Effect = "Allow" 88 | Principal = { 89 | Service = "glue.amazonaws.com" 90 | } 91 | }, 92 | ] 93 | Version = "2012-10-17" 94 | } 95 | ) 96 | managed_policy_arns = [aws_iam_policy.kubecost_glue_crawler.arn] 97 | } 98 | 99 | resource "aws_glue_catalog_database" "kubecost" { 100 | name = var.glue_database_name 101 | } 102 | 103 | resource "aws_glue_catalog_table" "kubecost" { 104 | name = var.glue_table_name 105 | database_name = aws_glue_catalog_database.kubecost.name 106 | parameters = { 107 | "classification" = "parquet" 108 | } 109 | 110 | table_type = "EXTERNAL_TABLE" 111 | 112 | dynamic "partition_keys" { 113 | for_each = [for partition_key in module.common_locals.partition_keys : partition_key] 114 | content { 115 | name = partition_keys.value.name 116 | type = partition_keys.value.hive_type 117 | } 118 | } 119 | 120 | storage_descriptor { 121 | location = "s3://${local.bucket_name}/" 122 | input_format = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat" 123 | output_format = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat" 124 | parameters = { 125 | "classification" = "parquet" 126 | } 127 | 128 | ser_de_info { 129 | name = "kubecost_table_parquet_serde" 130 | serialization_library = "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe" 131 | } 132 | 133 | dynamic "columns" { 134 | for_each = [for static_column in module.common_locals.static_columns : static_column] 135 | content { 136 | name = columns.value.name 137 | type = columns.value.hive_type 138 | } 139 | } 140 | dynamic "columns" { 141 | for_each = [for k8s_label in distinct(var.k8s_labels) : k8s_label] 142 | content { 143 | name = "properties.labels.${columns.value}" 144 | type = "string" 145 | } 146 | } 147 | dynamic "columns" { 148 | for_each = [for k8s_annotation in distinct(var.k8s_annotations) : k8s_annotation] 149 | content { 150 | name = "properties.annotations.${columns.value}" 151 | type = "string" 152 | } 153 | } 154 | } 155 | } 156 | 157 | resource "aws_glue_catalog_table" "kubecost_view" { 158 | name = var.glue_view_name 159 | database_name = aws_glue_catalog_database.kubecost.name 160 | parameters = { 161 | presto_view = "true" 162 | comment = "Presto View" 163 | } 164 | 165 | table_type = "VIRTUAL_VIEW" 166 | view_expanded_text = "/* Presto View */" 167 | view_original_text = "/* Presto View: ${base64encode(local.presto_view)} */" 168 | 169 | storage_descriptor { 170 | dynamic "columns" { 171 | for_each = [for static_column in module.common_locals.static_columns : static_column] 172 | content { 173 | name = columns.value.name 174 | type = columns.value.hive_type 175 | } 176 | } 177 | dynamic "columns" { 178 | for_each = [for k8s_label in distinct(var.k8s_labels) : k8s_label] 179 | content { 180 | name = "properties.labels.${columns.value}" 181 | type = "string" 182 | } 183 | } 184 | dynamic "columns" { 185 | for_each = [for k8s_annotation in distinct(var.k8s_annotations) : k8s_annotation] 186 | content { 187 | name = "properties.annotations.${columns.value}" 188 | type = "string" 189 | } 190 | } 191 | dynamic "columns" { 192 | for_each = [for partition_key in module.common_locals.partition_keys : partition_key] 193 | content { 194 | name = columns.value.name 195 | type = columns.value.hive_type 196 | } 197 | } 198 | } 199 | } 200 | 201 | resource "aws_glue_crawler" "kubecost" { 202 | name = var.glue_crawler_name 203 | database_name = aws_glue_catalog_database.kubecost.name 204 | schedule = "cron(${var.glue_crawler_schedule})" 205 | role = aws_iam_role.kubecost_glue_crawler.name 206 | 207 | catalog_target { 208 | database_name = aws_glue_catalog_database.kubecost.name 209 | tables = [ 210 | aws_glue_catalog_table.kubecost.name 211 | ] 212 | } 213 | 214 | schema_change_policy { 215 | delete_behavior = "LOG" 216 | update_behavior = "LOG" 217 | } 218 | 219 | configuration = jsonencode( 220 | { 221 | CrawlerOutput = { 222 | Partitions = { 223 | AddOrUpdateBehavior = "InheritFromTable" 224 | } 225 | } 226 | Grouping = { 227 | TableGroupingPolicy = "CombineCompatibleSchemas" 228 | } 229 | Version = 1 230 | } 231 | ) 232 | } 233 | 234 | # The next 3 resources are conditionally created 235 | # If the "kubecost_ca_certificates_list" variable isn't empty, a secret containing the CA certificate will be created 236 | # Else, it won't be created 237 | resource "aws_secretsmanager_secret" "kubecost_ca_cert" { 238 | count = length(var.kubecost_ca_certificates_list) > 0 ? length(var.kubecost_ca_certificates_list) : 0 239 | 240 | name = var.kubecost_ca_certificates_list[count.index].cert_secret_name 241 | recovery_window_in_days = 0 242 | } 243 | 244 | resource "aws_secretsmanager_secret_version" "kubecost_ca_cert" { 245 | count = length(var.kubecost_ca_certificates_list) > 0 ? length(var.kubecost_ca_certificates_list) : 0 246 | 247 | secret_id = aws_secretsmanager_secret.kubecost_ca_cert[count.index].id 248 | secret_string = file(var.kubecost_ca_certificates_list[count.index].cert_path) 249 | } 250 | 251 | resource "aws_secretsmanager_secret_policy" "kubecost_ca_cert" { 252 | count = length(var.kubecost_ca_certificates_list) > 0 ? length(var.kubecost_ca_certificates_list) : 0 253 | 254 | secret_arn = aws_secretsmanager_secret.kubecost_ca_cert[count.index].arn 255 | policy = templatefile("${path.module}/secret_policy.tpl", { 256 | arn = aws_secretsmanager_secret.kubecost_ca_cert[count.index].id 257 | principals = var.kubecost_ca_certificates_list[count.index].cert_secret_allowed_principals 258 | }) 259 | } 260 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/modules/pipeline/outputs.tf: -------------------------------------------------------------------------------- 1 | output "glue_database_name" { 2 | description = "The AWS Glue Database name" 3 | value = aws_glue_catalog_database.kubecost.name 4 | } 5 | 6 | output "glue_view_name" { 7 | description = "The AWS Glue Table name for the Athena view" 8 | value = aws_glue_catalog_table.kubecost_view.name 9 | } 10 | 11 | output "kubecost_ca_cert_secret" { 12 | description = "All AWS Secrets Manager Secrets" 13 | value = aws_secretsmanager_secret.kubecost_ca_cert 14 | } 15 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/modules/pipeline/secret_policy.tpl: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": 4 | [ 5 | { 6 | "Effect": "Deny", 7 | "Principal": "*", 8 | "Action": "secretsmanager:*", 9 | "Resource": "${arn}", 10 | "Condition": 11 | { 12 | "Bool": 13 | { 14 | "aws:SecureTransport": "false" 15 | }, 16 | "StringNotEquals": 17 | { 18 | %{ if principals != null } 19 | %{ if length(principals) > 0 } 20 | "aws:PrincipalTag/irsa-kubecost-s3-exporter-sm": "true", 21 | "aws:PrincipalArns": ${jsonencode(principals)} 22 | %{ else } 23 | "aws:PrincipalTag/irsa-kubecost-s3-exporter-sm": "true" 24 | %{ endif } 25 | %{ else } 26 | "aws:PrincipalTag/irsa-kubecost-s3-exporter-sm": "true" 27 | %{ endif } 28 | } 29 | } 30 | } 31 | ] 32 | } 33 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/modules/pipeline/variables.tf: -------------------------------------------------------------------------------- 1 | # # 2 | # Root Module's Common Variables # 3 | # # 4 | 5 | variable "bucket_arn" { 6 | description = <<-EOF 7 | (Required) The ARN of the S3 Bucket to which the Kubecost data will be uploaded. 8 | Meant to only take a reference to the "bucket_arn" variable from the root module. 9 | Possible values: Only "var.bucket_arn" (without the double quotes). 10 | EOF 11 | 12 | type = string 13 | 14 | # Note - the full regex should have been "^arn:(?:aws|aws-cn|aws-us-gov):s3:::(?!(xn--|.+-s3alias$))[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$" 15 | # The "(?!(xn--|.+-s3alias$))" part has been omitted because Terraform regex engine doesn't support negative lookahead (the "?!" part) 16 | # Therefore, it has been removed, and instead, "!startswith" and "!endswith" conditions have been added, to complement this missing functionality 17 | validation { 18 | condition = ( 19 | can(regex("^arn:(?:aws|aws-cn|aws-us-gov):s3:::[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$", var.bucket_arn)) && 20 | !startswith(element(split(":::", var.bucket_arn), 1), "xn--") && 21 | !endswith(element(split(":::", var.bucket_arn), 1), "-s3alias") 22 | ) 23 | error_message = "The 'bucket_arn' variable contains an invalid ARN" 24 | } 25 | } 26 | 27 | variable "k8s_labels" { 28 | description = <<-EOF 29 | (Optional) K8s labels common across all clusters, that you wish to include in the dataset. 30 | Meant to only take a reference to the "k8s_labels" variables from the root module. 31 | Possible values: Only "var.k8s_labels" (without the double quotes). 32 | Default value: empty list ([]). 33 | EOF 34 | 35 | type = list(string) 36 | default = [] 37 | 38 | validation { 39 | condition = ( 40 | length([ 41 | for k8s_label in var.k8s_labels : k8s_label 42 | if can(regex("^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9-]*[a-zA-Z0-9])\\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9-]*[A-Za-z0-9])\\/[a-zA-Z0-9][-\\w.]{0,61}[a-zA-Z0-9]$|^[a-zA-Z0-9][-\\w.]{0,61}[a-zA-Z0-9]$", k8s_label)) 43 | ]) == length(var.k8s_labels) 44 | ) 45 | error_message = "At least one of the items the 'k8s_labels' list, contains an invalid K8s label key" 46 | } 47 | } 48 | 49 | variable "k8s_annotations" { 50 | description = <<-EOF 51 | (Optional) K8s annotations common across all clusters, that you wish to include in the dataset. 52 | Meant to only take a reference to the "k8s_annotations" variable from the root module. 53 | Possible values: Only "var.k8s_annotations" (without the double quotes). 54 | Default value: empty list ([]). 55 | EOF 56 | 57 | type = list(string) 58 | default = [] 59 | 60 | validation { 61 | condition = ( 62 | length([ 63 | for k8s_annotation in var.k8s_annotations : k8s_annotation 64 | if can(regex("^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9-]*[a-zA-Z0-9])\\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9-]*[A-Za-z0-9])\\/[a-zA-Z0-9][-\\w.]{0,61}[a-zA-Z0-9]$|^[a-zA-Z0-9][-\\w.]{0,61}[a-zA-Z0-9]$", k8s_annotation)) 65 | ]) == length(var.k8s_annotations) 66 | ) 67 | error_message = "At least one of the items the 'k8s_annotations' list, contains an invalid K8s annotation key" 68 | } 69 | } 70 | 71 | variable "aws_common_tags" { 72 | description = <<-EOF 73 | (Optional) Common AWS tags to be used on all AWS resources created by Terraform. 74 | Meant to only take a reference to the "aws_common_tags" variable from the root module. 75 | Possible values: Only "var.aws_common_tags" (without the double quotes). 76 | Default value: empty map ({}). 77 | EOF 78 | 79 | type = map(any) 80 | default = {} 81 | } 82 | 83 | # # 84 | # This Module's Variables # 85 | # # 86 | 87 | variable "glue_database_name" { 88 | description = <<-EOF 89 | (Optional) The AWS Glue database name. 90 | Possible values: A valid AWS Glue database name. 91 | Default value: kubecost_db 92 | EOF 93 | 94 | type = string 95 | default = "kubecost_db" 96 | 97 | validation { 98 | condition = can(regex("^[a-z0-9_]{1,255}$", var.glue_database_name)) 99 | error_message = "The 'glue_database_name' variable contains an invalid AWS Glue Database name" 100 | } 101 | } 102 | 103 | variable "glue_table_name" { 104 | description = <<-EOF 105 | (Optional) The AWS Glue table name. 106 | Possible values: A valid AWS Glue table name. 107 | Default value: kubecost_table 108 | EOF 109 | 110 | type = string 111 | default = "kubecost_table" 112 | 113 | validation { 114 | condition = can(regex("^[a-z0-9_]{1,255}$", var.glue_table_name)) 115 | error_message = "The 'glue_table_name' variable contains an invalid AWS Glue Table name" 116 | } 117 | } 118 | 119 | variable "glue_view_name" { 120 | description = <<-EOF 121 | (Optional) The AWS Glue Table name for the Athena view 122 | Possible values: A valid AWS Glue table name. 123 | Default value: kubecost_view 124 | EOF 125 | 126 | type = string 127 | default = "kubecost_view" 128 | 129 | validation { 130 | condition = can(regex("^[a-z0-9_]{1,255}$", var.glue_view_name)) 131 | error_message = "The 'glue_view_name' variable contains an invalid AWS Glue Table name" 132 | } 133 | } 134 | 135 | variable "glue_crawler_name" { 136 | description = <<-EOF 137 | (Optional) The AWS Glue Crawler name 138 | Possible values: A valid AWS Glue crawler name. 139 | Default value: kubecost_crawler 140 | EOF 141 | 142 | type = string 143 | default = "kubecost_crawler" 144 | 145 | validation { 146 | condition = can(regex("^[a-z0-9_]{1,255}$", var.glue_crawler_name)) 147 | error_message = "The 'glue_crawler_name' variable contains an invalid AWS Crawler Table name" 148 | } 149 | } 150 | 151 | variable "glue_crawler_schedule" { 152 | description = <<-EOF 153 | (Optional) The schedule for the Glue Crawler, in Cron format. Make sure to set it after the last Kubecost S3 Exporter Cron schedule. 154 | Possible values: A valid cron expression. 155 | Default value: 0 1 * * ? * 156 | 157 | EOF 158 | 159 | type = string 160 | default = "0 1 * * ? *" 161 | 162 | validation { 163 | condition = can(regex("(@(annually|yearly|monthly|weekly|daily|hourly|reboot))|(@every (\\d+(ns|us|µs|ms|s|m|h))+)|((((\\d+,)+\\d+|(\\d+([/\\-])\\d+)|\\d+|\\*|\\?) ?){5,7})", var.glue_crawler_schedule)) 164 | error_message = "The 'glue_crawler_schedule' variable contains an invalid Cron expression" 165 | } 166 | } 167 | 168 | variable "athena_view_data_retention_months" { 169 | description = <<-EOF 170 | (Optional) The amount of months back to keep data in the Athena view. 171 | Possible values: A non-zero positive integer. 172 | Default value: 6 173 | EOF 174 | 175 | type = string 176 | default = 6 177 | 178 | validation { 179 | condition = can(regex("^[1-9][0-9]*$", var.athena_view_data_retention_months)) 180 | error_message = "The 'athena_view_data_retention_months' variable can take only a non-zero positive integer" 181 | } 182 | } 183 | 184 | variable "kubecost_ca_certificates_list" { 185 | description = <<-EOF 186 | (Optional) A list root CA certificates paths and their configuration for AWS Secrets Manager. Used for TLS communication with Kubecost. 187 | This is a consolidated list of all root CA certificates that are needed for all Kubecost endpoints. 188 | 189 | (Required) cert_path: The full local path to the root CA certificate. 190 | Possible values: A valid Linux path. 191 | (Required) cert_secret_name: The name to use for the AWS Secrets Manager Secret that will be created for this root CA certificate. 192 | Possible values: A valid AWS Secret Manager secret name. 193 | (Optional) cert_secret_allowed_principals: A list of principals to include in the AWS Secrets Manager Secret policy (in addition to the principal that identify the cluster, which will be automatically added by Terraform). 194 | Possible values: A list of IAM principals (users, roles) ARNs. 195 | Default value: empty list ([]). 196 | EOF 197 | 198 | type = list(object({ 199 | cert_path = string 200 | cert_secret_name = string 201 | cert_secret_allowed_principals = optional(list(string)) 202 | })) 203 | 204 | default = [] 205 | 206 | validation { 207 | condition = ( 208 | length([ 209 | for cert_path in var.kubecost_ca_certificates_list.*.cert_path : cert_path 210 | if can(regex("^(~|\\/[ \\w.-]+)+$", cert_path)) 211 | ]) == length(var.kubecost_ca_certificates_list) && 212 | length([ 213 | for cert_secret_name in var.kubecost_ca_certificates_list.*.cert_secret_name : cert_secret_name 214 | if can(regex("^[\\w/+=.@-]{1,512}$", cert_secret_name)) 215 | ]) == length(var.kubecost_ca_certificates_list) 216 | ) 217 | error_message = <<-EOF 218 | At least one of the below is invalid in one of the items in "kubecost_ca_certificates_list" list: 219 | 1. One of the "cert_path" values 220 | 2. One of the "cert_secret_name" values 221 | EOF 222 | } 223 | } 224 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/outputs.tf: -------------------------------------------------------------------------------- 1 | # Output for showing the distinct labels from all clusters, collected from the "k8s_labels" common input 2 | output "labels" { 3 | value = length(var.k8s_labels) > 0 ? join(", ", distinct(var.k8s_labels)) : null 4 | description = "A list of the distinct labels of all clusters, that'll be added to the dataset" 5 | } 6 | 7 | # Output for showing the distinct annotations from all clusters, collected from the "k8s_annotations" common input 8 | output "annotations" { 9 | value = length(var.k8s_annotations) > 0 ? join(", ", distinct(var.k8s_annotations)) : null 10 | description = "A list of the distinct annotations of all clusters, that'll be added to the dataset" 11 | } 12 | 13 | # Clusters outputs 14 | #output "cluster1" { 15 | # 16 | # # This is an example, to help you get started 17 | # 18 | # value = module.cluster1 19 | # description = "The outputs for 'cluster1'" 20 | #} 21 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/providers.tf: -------------------------------------------------------------------------------- 1 | # This is the providers blocks file 2 | # Follow the sections and the comments inside them, which provide instructions 3 | 4 | ##################################### 5 | # Section 1 - Pipeline AWS Provider # 6 | ##################################### 7 | 8 | # Provider for the pipeline module 9 | provider "aws" { 10 | 11 | # This is an example, to help you get started 12 | 13 | region = "us-east-1" # Change the region if necessary 14 | shared_config_files = ["~/.aws/config"] # Change the path to the shared config file, if necessary 15 | shared_credentials_files = ["~/.aws/credentials"] # Change the path to the shared credential file, if necessary 16 | profile = "pipeline_profile" # Change to the profile that will be used for the account and region where the pipeline resources will be deployed 17 | default_tags { 18 | tags = var.aws_common_tags 19 | } 20 | } 21 | 22 | ########################################################### 23 | # Section 2 - Kubecost S3 Exporter AWS and Helm Providers # 24 | ########################################################### 25 | 26 | # Providers for the kubecost_s3_exporter module. 27 | # Used to deploy the K8s resources on clusters, and creates IRSA in cluster's accounts 28 | # There are 2 deployment options: 29 | # 30 | # 1. Deploy the K8s resources by having Terraform invoke Helm 31 | # In this case, you have to define 2 providers per cluster - an AWS provider and a Helm provider 32 | # 2. Deploy the K8s resources by having Terraform generate a Helm values.yaml, then you deploy it using Helm 33 | # In this case, you have to define 1 provider per cluster - an AWS provider 34 | 35 | # # 36 | # Example providers for cluster with Helm invocation # 37 | # # 38 | 39 | # Use these providers if you'd like Terraform to invoke Helm to deploy the K8s resources 40 | # Duplicate the providers for each cluster on which you wish to deploy the Kubecost S3 Exporter 41 | 42 | provider "aws" { 43 | 44 | # This is an example, to help you get started 45 | 46 | alias = "us-east-1-111111111111-cluster1" # Change to an alias that uniquely identifies the cluster within all the AWS provider blocks 47 | 48 | region = "us-east-1" # Change the region if necessary 49 | shared_config_files = ["~/.aws/config"] # Change the path to the shared config file, if necessary 50 | shared_credentials_files = ["~/.aws/credentials"] # Change the path to the shared credential file, if necessary 51 | profile = "profile1" # Change to the profile that identifies the account and region where the cluster is 52 | default_tags { 53 | tags = var.aws_common_tags 54 | } 55 | } 56 | 57 | provider "helm" { 58 | 59 | # This is an example, to help you get started 60 | 61 | alias = "us-east-1-111111111111-cluster1" # Change to an alias that uniquely identifies the cluster within all the Helm provider blocks 62 | 63 | kubernetes { 64 | config_context = "arn:aws:eks:us-east-1:111111111111:cluster/cluster1" # Change to the context that identifies the cluster in the K8s config file (in many cases it's the cluster ARN) 65 | config_path = "~/.kube/config" # Change to the full path of the K8s config file 66 | } 67 | } 68 | 69 | # # 70 | # Example provider for cluster without Helm invocation # 71 | # # 72 | 73 | # Use this provider if you'd like Terraform to generate a Helm values.yaml, then you deploy it using Helm 74 | # Duplicate the provider for each cluster on which you wish to deploy the Kubecost S3 Exporter 75 | provider "aws" { 76 | 77 | # This is an example, to help you get started 78 | 79 | alias = "us-east-1-111111111111-cluster2" # Change to an alias that uniquely identifies the cluster within all AWS Helm provider blocks 80 | 81 | region = "us-east-1" # Change the region if necessary 82 | shared_config_files = ["~/.aws/config"] # Change the path to the shared config file, if necessary 83 | shared_credentials_files = ["~/.aws/credentials"] # Change the path to the shared credential file, if necessary 84 | profile = "profile1" # Change to the profile that identifies the account and region where the cluster is 85 | default_tags { 86 | tags = var.aws_common_tags 87 | } 88 | } 89 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/terraform.tfvars: -------------------------------------------------------------------------------- 1 | bucket_arn = "" # Add S3 bucket ARN here, of the bucket that will be used to store the data collected from Kubecost 2 | k8s_labels = [] # Optionally, add K8s labels you'd like to be present in the dataset 3 | k8s_annotations = [] # Optionally, add K8s annotations you'd like to be present in the dataset 4 | aws_common_tags = {} # Optionally, add AWS common tags you'd like to be created on all resources 5 | -------------------------------------------------------------------------------- /terraform/terraform-aws-cca/variables.tf: -------------------------------------------------------------------------------- 1 | variable "bucket_arn" { 2 | description = <<-EOF 3 | (Required) The ARN of the S3 Bucket to which the Kubecost data will be uploaded. 4 | Possible values: A valid S3 bucket ARN. 5 | EOF 6 | 7 | type = string 8 | 9 | # Note - the full regex should have been "^arn:(?:aws|aws-cn|aws-us-gov):s3:::(?!(xn--|.+-s3alias$))[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$" 10 | # The "(?!(xn--|.+-s3alias$))" part has been omitted because Terraform regex engine doesn't support negative lookahead (the "?!" part) 11 | # Therefore, it has been removed, and instead, "!startswith" and "!endswith" conditions have been added, to complement this missing functionality 12 | validation { 13 | condition = ( 14 | can(regex("^arn:(?:aws|aws-cn|aws-us-gov):s3:::[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$", var.bucket_arn)) && 15 | !startswith(element(split(":::", var.bucket_arn), 1), "xn--") && 16 | !endswith(element(split(":::", var.bucket_arn), 1), "-s3alias") 17 | ) 18 | error_message = "The 'bucket_arn' variable contains an invalid ARN" 19 | } 20 | } 21 | 22 | variable "k8s_labels" { 23 | description = <<-EOF 24 | (Optional) K8s labels common across all clusters, that you wish to include in the dataset. 25 | Possible values: A list of strings, each one should be a valid K8s label key. 26 | Default value: empty list ([]). 27 | EOF 28 | 29 | type = list(string) 30 | default = [] 31 | 32 | validation { 33 | condition = ( 34 | length([ 35 | for k8s_label in var.k8s_labels : k8s_label 36 | if can(regex("^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9-]*[a-zA-Z0-9])\\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9-]*[A-Za-z0-9])\\/[a-zA-Z0-9][-\\w.]{0,61}[a-zA-Z0-9]$|^[a-zA-Z0-9][-\\w.]{0,61}[a-zA-Z0-9]$", k8s_label)) 37 | ]) == length(var.k8s_labels) 38 | ) 39 | error_message = "At least one of the items the 'k8s_labels' list, contains an invalid K8s label key" 40 | } 41 | } 42 | 43 | variable "k8s_annotations" { 44 | description = <<-EOF 45 | (Optional) K8s annotations common across all clusters, that you wish to include in the dataset. 46 | Possible values: A list of strings, each one should be a valid K8s annotation key. 47 | Default value: empty list ([]). 48 | EOF 49 | 50 | type = list(string) 51 | default = [] 52 | 53 | validation { 54 | condition = ( 55 | length([ 56 | for k8s_annotation in var.k8s_annotations : k8s_annotation 57 | if can(regex("^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9-]*[a-zA-Z0-9])\\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9-]*[A-Za-z0-9])\\/[a-zA-Z0-9][-\\w.]{0,61}[a-zA-Z0-9]$|^[a-zA-Z0-9][-\\w.]{0,61}[a-zA-Z0-9]$", k8s_annotation)) 58 | ]) == length(var.k8s_annotations) 59 | ) 60 | error_message = "At least one of the items the 'k8s_annotations' list, contains an invalid K8s annotation key" 61 | } 62 | } 63 | 64 | variable "aws_common_tags" { 65 | description = <<-EOF 66 | (Optional) Common AWS tags to be used on all AWS resources created by Terraform. 67 | Possible values: Each field in the map must have a valid AWS tag key and an optional value. 68 | Default value: empty map ({}). 69 | EOF 70 | 71 | type = map(any) 72 | default = {} 73 | } 74 | -------------------------------------------------------------------------------- /timezones.txt: -------------------------------------------------------------------------------- 1 | Africa/Abidjan 2 | Africa/Accra 3 | Africa/Addis_Ababa 4 | Africa/Algiers 5 | Africa/Asmara 6 | Africa/Asmera 7 | Africa/Bamako 8 | Africa/Bangui 9 | Africa/Banjul 10 | Africa/Bissau 11 | Africa/Blantyre 12 | Africa/Brazzaville 13 | Africa/Bujumbura 14 | Africa/Cairo 15 | Africa/Casablanca 16 | Africa/Ceuta 17 | Africa/Conakry 18 | Africa/Dakar 19 | Africa/Dar_es_Salaam 20 | Africa/Djibouti 21 | Africa/Douala 22 | Africa/El_Aaiun 23 | Africa/Freetown 24 | Africa/Gaborone 25 | Africa/Harare 26 | Africa/Johannesburg 27 | Africa/Juba 28 | Africa/Kampala 29 | Africa/Khartoum 30 | Africa/Kigali 31 | Africa/Kinshasa 32 | Africa/Lagos 33 | Africa/Libreville 34 | Africa/Lome 35 | Africa/Luanda 36 | Africa/Lubumbashi 37 | Africa/Lusaka 38 | Africa/Malabo 39 | Africa/Maputo 40 | Africa/Maseru 41 | Africa/Mbabane 42 | Africa/Mogadishu 43 | Africa/Monrovia 44 | Africa/Nairobi 45 | Africa/Ndjamena 46 | Africa/Niamey 47 | Africa/Nouakchott 48 | Africa/Ouagadougou 49 | Africa/Porto-Novo 50 | Africa/Sao_Tome 51 | Africa/Timbuktu 52 | Africa/Tripoli 53 | Africa/Tunis 54 | Africa/Windhoek 55 | America/Adak 56 | America/Anchorage 57 | America/Anguilla 58 | America/Antigua 59 | America/Araguaina 60 | America/Argentina/Buenos_Aires 61 | America/Argentina/Catamarca 62 | America/Argentina/ComodRivadavia 63 | America/Argentina/Cordoba 64 | America/Argentina/Jujuy 65 | America/Argentina/La_Rioja 66 | America/Argentina/Mendoza 67 | America/Argentina/Rio_Gallegos 68 | America/Argentina/Salta 69 | America/Argentina/San_Juan 70 | America/Argentina/San_Luis 71 | America/Argentina/Tucuman 72 | America/Argentina/Ushuaia 73 | America/Aruba 74 | America/Asuncion 75 | America/Atikokan 76 | America/Atka 77 | America/Bahia 78 | America/Bahia_Banderas 79 | America/Barbados 80 | America/Belem 81 | America/Belize 82 | America/Blanc-Sablon 83 | America/Boa_Vista 84 | America/Bogota 85 | America/Boise 86 | America/Buenos_Aires 87 | America/Cambridge_Bay 88 | America/Campo_Grande 89 | America/Cancun 90 | America/Caracas 91 | America/Catamarca 92 | America/Cayenne 93 | America/Cayman 94 | America/Chicago 95 | America/Chihuahua 96 | America/Coral_Harbour 97 | America/Cordoba 98 | America/Costa_Rica 99 | America/Creston 100 | America/Cuiaba 101 | America/Curacao 102 | America/Danmarkshavn 103 | America/Dawson 104 | America/Dawson_Creek 105 | America/Denver 106 | America/Detroit 107 | America/Dominica 108 | America/Edmonton 109 | America/Eirunepe 110 | America/El_Salvador 111 | America/Ensenada 112 | America/Fort_Nelson 113 | America/Fort_Wayne 114 | America/Fortaleza 115 | America/Glace_Bay 116 | America/Godthab 117 | America/Goose_Bay 118 | America/Grand_Turk 119 | America/Grenada 120 | America/Guadeloupe 121 | America/Guatemala 122 | America/Guayaquil 123 | America/Guyana 124 | America/Halifax 125 | America/Havana 126 | America/Hermosillo 127 | America/Indiana/Indianapolis 128 | America/Indiana/Knox 129 | America/Indiana/Marengo 130 | America/Indiana/Petersburg 131 | America/Indiana/Tell_City 132 | America/Indiana/Vevay 133 | America/Indiana/Vincennes 134 | America/Indiana/Winamac 135 | America/Indianapolis 136 | America/Inuvik 137 | America/Iqaluit 138 | America/Jamaica 139 | America/Jujuy 140 | America/Juneau 141 | America/Kentucky/Louisville 142 | America/Kentucky/Monticello 143 | America/Knox_IN 144 | America/Kralendijk 145 | America/La_Paz 146 | America/Lima 147 | America/Los_Angeles 148 | America/Louisville 149 | America/Lower_Princes 150 | America/Maceio 151 | America/Managua 152 | America/Manaus 153 | America/Marigot 154 | America/Martinique 155 | America/Matamoros 156 | America/Mazatlan 157 | America/Mendoza 158 | America/Menominee 159 | America/Merida 160 | America/Metlakatla 161 | America/Mexico_City 162 | America/Miquelon 163 | America/Moncton 164 | America/Monterrey 165 | America/Montevideo 166 | America/Montreal 167 | America/Montserrat 168 | America/Nassau 169 | America/New_York 170 | America/Nipigon 171 | America/Nome 172 | America/Noronha 173 | America/North_Dakota/Beulah 174 | America/North_Dakota/Center 175 | America/North_Dakota/New_Salem 176 | America/Nuuk 177 | America/Ojinaga 178 | America/Panama 179 | America/Pangnirtung 180 | America/Paramaribo 181 | America/Phoenix 182 | America/Port-au-Prince 183 | America/Port_of_Spain 184 | America/Porto_Acre 185 | America/Porto_Velho 186 | America/Puerto_Rico 187 | America/Punta_Arenas 188 | America/Rainy_River 189 | America/Rankin_Inlet 190 | America/Recife 191 | America/Regina 192 | America/Resolute 193 | America/Rio_Branco 194 | America/Rosario 195 | America/Santa_Isabel 196 | America/Santarem 197 | America/Santiago 198 | America/Santo_Domingo 199 | America/Sao_Paulo 200 | America/Scoresbysund 201 | America/Shiprock 202 | America/Sitka 203 | America/St_Barthelemy 204 | America/St_Johns 205 | America/St_Kitts 206 | America/St_Lucia 207 | America/St_Thomas 208 | America/St_Vincent 209 | America/Swift_Current 210 | America/Tegucigalpa 211 | America/Thule 212 | America/Thunder_Bay 213 | America/Tijuana 214 | America/Toronto 215 | America/Tortola 216 | America/Vancouver 217 | America/Virgin 218 | America/Whitehorse 219 | America/Winnipeg 220 | America/Yakutat 221 | America/Yellowknife 222 | Antarctica/Casey 223 | Antarctica/Davis 224 | Antarctica/DumontDUrville 225 | Antarctica/Macquarie 226 | Antarctica/Mawson 227 | Antarctica/McMurdo 228 | Antarctica/Palmer 229 | Antarctica/Rothera 230 | Antarctica/South_Pole 231 | Antarctica/Syowa 232 | Antarctica/Troll 233 | Antarctica/Vostok 234 | Arctic/Longyearbyen 235 | Asia/Aden 236 | Asia/Almaty 237 | Asia/Amman 238 | Asia/Anadyr 239 | Asia/Aqtau 240 | Asia/Aqtobe 241 | Asia/Ashgabat 242 | Asia/Ashkhabad 243 | Asia/Atyrau 244 | Asia/Baghdad 245 | Asia/Bahrain 246 | Asia/Baku 247 | Asia/Bangkok 248 | Asia/Barnaul 249 | Asia/Beirut 250 | Asia/Bishkek 251 | Asia/Brunei 252 | Asia/Calcutta 253 | Asia/Chita 254 | Asia/Choibalsan 255 | Asia/Chongqing 256 | Asia/Chungking 257 | Asia/Colombo 258 | Asia/Dacca 259 | Asia/Damascus 260 | Asia/Dhaka 261 | Asia/Dili 262 | Asia/Dubai 263 | Asia/Dushanbe 264 | Asia/Famagusta 265 | Asia/Gaza 266 | Asia/Harbin 267 | Asia/Hebron 268 | Asia/Ho_Chi_Minh 269 | Asia/Hong_Kong 270 | Asia/Hovd 271 | Asia/Irkutsk 272 | Asia/Istanbul 273 | Asia/Jakarta 274 | Asia/Jayapura 275 | Asia/Jerusalem 276 | Asia/Kabul 277 | Asia/Kamchatka 278 | Asia/Karachi 279 | Asia/Kashgar 280 | Asia/Kathmandu 281 | Asia/Katmandu 282 | Asia/Khandyga 283 | Asia/Kolkata 284 | Asia/Krasnoyarsk 285 | Asia/Kuala_Lumpur 286 | Asia/Kuching 287 | Asia/Kuwait 288 | Asia/Macao 289 | Asia/Macau 290 | Asia/Magadan 291 | Asia/Makassar 292 | Asia/Manila 293 | Asia/Muscat 294 | Asia/Nicosia 295 | Asia/Novokuznetsk 296 | Asia/Novosibirsk 297 | Asia/Omsk 298 | Asia/Oral 299 | Asia/Phnom_Penh 300 | Asia/Pontianak 301 | Asia/Pyongyang 302 | Asia/Qatar 303 | Asia/Qostanay 304 | Asia/Qyzylorda 305 | Asia/Rangoon 306 | Asia/Riyadh 307 | Asia/Saigon 308 | Asia/Sakhalin 309 | Asia/Samarkand 310 | Asia/Seoul 311 | Asia/Shanghai 312 | Asia/Singapore 313 | Asia/Srednekolymsk 314 | Asia/Taipei 315 | Asia/Tashkent 316 | Asia/Tbilisi 317 | Asia/Tehran 318 | Asia/Tel_Aviv 319 | Asia/Thimbu 320 | Asia/Thimphu 321 | Asia/Tokyo 322 | Asia/Tomsk 323 | Asia/Ujung_Pandang 324 | Asia/Ulaanbaatar 325 | Asia/Ulan_Bator 326 | Asia/Urumqi 327 | Asia/Ust-Nera 328 | Asia/Vientiane 329 | Asia/Vladivostok 330 | Asia/Yakutsk 331 | Asia/Yangon 332 | Asia/Yekaterinburg 333 | Asia/Yerevan 334 | Atlantic/Azores 335 | Atlantic/Bermuda 336 | Atlantic/Canary 337 | Atlantic/Cape_Verde 338 | Atlantic/Faeroe 339 | Atlantic/Faroe 340 | Atlantic/Jan_Mayen 341 | Atlantic/Madeira 342 | Atlantic/Reykjavik 343 | Atlantic/South_Georgia 344 | Atlantic/St_Helena 345 | Atlantic/Stanley 346 | Australia/ACT 347 | Australia/Adelaide 348 | Australia/Brisbane 349 | Australia/Broken_Hill 350 | Australia/Canberra 351 | Australia/Currie 352 | Australia/Darwin 353 | Australia/Eucla 354 | Australia/Hobart 355 | Australia/LHI 356 | Australia/Lindeman 357 | Australia/Lord_Howe 358 | Australia/Melbourne 359 | Australia/NSW 360 | Australia/North 361 | Australia/Perth 362 | Australia/Queensland 363 | Australia/South 364 | Australia/Sydney 365 | Australia/Tasmania 366 | Australia/Victoria 367 | Australia/West 368 | Australia/Yancowinna 369 | Brazil/Acre 370 | Brazil/DeNoronha 371 | Brazil/East 372 | Brazil/West 373 | CET 374 | CST6CDT 375 | Canada/Atlantic 376 | Canada/Central 377 | Canada/Eastern 378 | Canada/Mountain 379 | Canada/Newfoundland 380 | Canada/Pacific 381 | Canada/Saskatchewan 382 | Canada/Yukon 383 | Chile/Continental 384 | Chile/EasterIsland 385 | Cuba 386 | EET 387 | EST5EDT 388 | Egypt 389 | Eire 390 | Etc/GMT 391 | Etc/GMT+0 392 | Etc/GMT+1 393 | Etc/GMT+10 394 | Etc/GMT+11 395 | Etc/GMT+12 396 | Etc/GMT+2 397 | Etc/GMT+3 398 | Etc/GMT+4 399 | Etc/GMT+5 400 | Etc/GMT+6 401 | Etc/GMT+7 402 | Etc/GMT+8 403 | Etc/GMT+9 404 | Etc/GMT-0 405 | Etc/GMT-1 406 | Etc/GMT-10 407 | Etc/GMT-11 408 | Etc/GMT-12 409 | Etc/GMT-13 410 | Etc/GMT-14 411 | Etc/GMT-2 412 | Etc/GMT-3 413 | Etc/GMT-4 414 | Etc/GMT-5 415 | Etc/GMT-6 416 | Etc/GMT-7 417 | Etc/GMT-8 418 | Etc/GMT-9 419 | Etc/GMT0 420 | Etc/Greenwich 421 | Etc/UCT 422 | Etc/UTC 423 | Etc/Universal 424 | Etc/Zulu 425 | Europe/Amsterdam 426 | Europe/Andorra 427 | Europe/Astrakhan 428 | Europe/Athens 429 | Europe/Belfast 430 | Europe/Belgrade 431 | Europe/Berlin 432 | Europe/Bratislava 433 | Europe/Brussels 434 | Europe/Bucharest 435 | Europe/Budapest 436 | Europe/Busingen 437 | Europe/Chisinau 438 | Europe/Copenhagen 439 | Europe/Dublin 440 | Europe/Gibraltar 441 | Europe/Guernsey 442 | Europe/Helsinki 443 | Europe/Isle_of_Man 444 | Europe/Istanbul 445 | Europe/Jersey 446 | Europe/Kaliningrad 447 | Europe/Kiev 448 | Europe/Kirov 449 | Europe/Kyiv 450 | Europe/Lisbon 451 | Europe/Ljubljana 452 | Europe/London 453 | Europe/Luxembourg 454 | Europe/Madrid 455 | Europe/Malta 456 | Europe/Mariehamn 457 | Europe/Minsk 458 | Europe/Monaco 459 | Europe/Moscow 460 | Europe/Nicosia 461 | Europe/Oslo 462 | Europe/Paris 463 | Europe/Podgorica 464 | Europe/Prague 465 | Europe/Riga 466 | Europe/Rome 467 | Europe/Samara 468 | Europe/San_Marino 469 | Europe/Sarajevo 470 | Europe/Saratov 471 | Europe/Simferopol 472 | Europe/Skopje 473 | Europe/Sofia 474 | Europe/Stockholm 475 | Europe/Tallinn 476 | Europe/Tirane 477 | Europe/Tiraspol 478 | Europe/Ulyanovsk 479 | Europe/Uzhgorod 480 | Europe/Vaduz 481 | Europe/Vatican 482 | Europe/Vienna 483 | Europe/Vilnius 484 | Europe/Volgograd 485 | Europe/Warsaw 486 | Europe/Zagreb 487 | Europe/Zaporozhye 488 | Europe/Zurich 489 | GB 490 | GB-Eire 491 | GMT 492 | GMT0 493 | Greenwich 494 | Hongkong 495 | Iceland 496 | Indian/Antananarivo 497 | Indian/Chagos 498 | Indian/Christmas 499 | Indian/Cocos 500 | Indian/Comoro 501 | Indian/Kerguelen 502 | Indian/Mahe 503 | Indian/Maldives 504 | Indian/Mauritius 505 | Indian/Mayotte 506 | Indian/Reunion 507 | Iran 508 | Israel 509 | Jamaica 510 | Japan 511 | Kwajalein 512 | Libya 513 | MET 514 | MST7MDT 515 | Mexico/BajaNorte 516 | Mexico/BajaSur 517 | Mexico/General 518 | NZ 519 | NZ-CHAT 520 | Navajo 521 | PRC 522 | PST8PDT 523 | Pacific/Apia 524 | Pacific/Auckland 525 | Pacific/Bougainville 526 | Pacific/Chatham 527 | Pacific/Chuuk 528 | Pacific/Easter 529 | Pacific/Efate 530 | Pacific/Enderbury 531 | Pacific/Fakaofo 532 | Pacific/Fiji 533 | Pacific/Funafuti 534 | Pacific/Galapagos 535 | Pacific/Gambier 536 | Pacific/Guadalcanal 537 | Pacific/Guam 538 | Pacific/Honolulu 539 | Pacific/Johnston 540 | Pacific/Kanton 541 | Pacific/Kiritimati 542 | Pacific/Kosrae 543 | Pacific/Kwajalein 544 | Pacific/Majuro 545 | Pacific/Marquesas 546 | Pacific/Midway 547 | Pacific/Nauru 548 | Pacific/Niue 549 | Pacific/Norfolk 550 | Pacific/Noumea 551 | Pacific/Pago_Pago 552 | Pacific/Palau 553 | Pacific/Pitcairn 554 | Pacific/Pohnpei 555 | Pacific/Ponape 556 | Pacific/Port_Moresby 557 | Pacific/Rarotonga 558 | Pacific/Saipan 559 | Pacific/Samoa 560 | Pacific/Tahiti 561 | Pacific/Tarawa 562 | Pacific/Tongatapu 563 | Pacific/Truk 564 | Pacific/Wake 565 | Pacific/Wallis 566 | Pacific/Yap 567 | Poland 568 | Portugal 569 | ROK 570 | Singapore 571 | SystemV/AST4 572 | SystemV/AST4ADT 573 | SystemV/CST6 574 | SystemV/CST6CDT 575 | SystemV/EST5 576 | SystemV/EST5EDT 577 | SystemV/HST10 578 | SystemV/MST7 579 | SystemV/MST7MDT 580 | SystemV/PST8 581 | SystemV/PST8PDT 582 | SystemV/YST9 583 | SystemV/YST9YDT 584 | Turkey 585 | UCT 586 | US/Alaska 587 | US/Aleutian 588 | US/Arizona 589 | US/Central 590 | US/East-Indiana 591 | US/Eastern 592 | US/Hawaii 593 | US/Indiana-Starke 594 | US/Michigan 595 | US/Mountain 596 | US/Pacific 597 | US/Samoa 598 | UTC 599 | Universal 600 | W-SU 601 | WET 602 | Zulu 603 | EST 604 | HST 605 | MST 606 | ACT 607 | AET 608 | AGT 609 | ART 610 | AST 611 | BET 612 | BST 613 | CAT 614 | CNT 615 | CST 616 | CTT 617 | EAT 618 | ECT 619 | IET 620 | IST 621 | JST 622 | MIT 623 | NET 624 | NST 625 | PLT 626 | PNT 627 | PRT 628 | PST 629 | SST 630 | VST 631 | --------------------------------------------------------------------------------