├── .eslintignore ├── .eslintrc.json ├── .gitignore ├── .prettierrc ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── bin └── app.ts ├── cdk.json ├── config └── config.interface.ts ├── dashboards ├── geth_system_resources.json ├── geth_transaction_processing.json └── lighthouse_dashboard.json ├── images └── ethereum_staking_eks.jpg ├── lib ├── cloudwatch-widgets.json ├── custom-resources │ └── eksloggingOnEvent.py ├── eks.ts ├── k8s-baseline.ts ├── nodegroup.ts ├── observe.ts ├── policies │ └── policies.ts ├── storageClass.yaml └── vpc.ts ├── manifests ├── consoleViewOnlyGroup.yaml └── fluentBitSetup.yaml ├── package-lock.json ├── package.json └── tsconfig.json /.eslintignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | 3 | -------------------------------------------------------------------------------- /.eslintrc.json: -------------------------------------------------------------------------------- 1 | { 2 | "env": { 3 | "browser": true, 4 | "es2021": true, 5 | "node": true 6 | }, 7 | "extends": [ 8 | "eslint:recommended", 9 | "plugin:@typescript-eslint/recommended", 10 | "prettier" 11 | ], 12 | "overrides": [], 13 | "parser": "@typescript-eslint/parser", 14 | "parserOptions": { 15 | "ecmaVersion": "latest", 16 | "sourceType": "module" 17 | }, 18 | "plugins": ["@typescript-eslint"], 19 | "rules": { 20 | "@typescript-eslint/no-unused-vars": "error", 21 | "@typescript-eslint/consistent-type-definitions": ["error", "type"], 22 | "@typescript-eslint/no-var-requires": ["off"] 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | node_modules/ 2 | cdk.out/ 3 | cdk.context.json 4 | dist/ 5 | **/*id_rsa* 6 | **/.DS_Store -------------------------------------------------------------------------------- /.prettierrc: -------------------------------------------------------------------------------- 1 | { 2 | "trailingComma": "es5", 3 | "tabWidth": 2, 4 | "semi": true, 5 | "singleQuote": true 6 | } 7 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Deploy an Ethereum staking solution with Amazon EKS and Amazon Managed Prometheus and Grafana 2 | This repository contains a CDK template for deploying an Ethereum execution client, such as Geth or Erigon, and an Ethereum consensus client, such as Lighthouse or Prysm, on an EKS cluster across a configurable set of Availability Zones. This repository also demonstrates how to deploy a Prometheus metrics server and provision a Grafana dashboard with Amazon Managed Service for Grafana. 3 | 4 | Two Grafana dashboards, and an Amazon CloudWatch dashboard are deployed to provide insights into the health of the EKS cluster, and the Ethereum clients. 5 | 6 | The cluster nodes are deployed on `r7g` Graviton instances which provide the best performance to cost ratio for running Ethereum clients on AWS. Client storage is on Amazon EBS `gp3` volumes. These settings can be easily configured within the deployment template. A Bastion host is also deployed from which we manage the EKS cluster. The EKS cluster is deployed in private subnets, while the Bastion host is in a public subnet. 7 | 8 | A couple of things to note. First, in order to provision the Grafana dashboard, your AWS account needs to be part of an Organization, with IAM Identity Center enabled. Instructions to do these can be found below. 9 | 10 | Second, the CDK creates an SSH key that you will use to connect to the Bastion host. The key is saved in Parameter Store. For security considerations, it's advised that you delete the key from Parameter Store after you have downloaded it. 11 | 12 | Third, the walkthrough deploys this CDK in `us-east-1` in Availability Zones `us-east-1a`, `us-east-1b`, and `us-east-1c`. 13 | 14 | Fourth, due to the size of the Ethereum blockchain, client sync time can take a day, to a week, or even longer. The example below installs the Geth client which can be synced within a day using its [snap sync](https://geth.ethereum.org/docs/fundamentals/sync-modes) feature. 15 | 16 | With all that behind us, let's move on to deploying the solution. 17 | 18 | ## Solution Architecture 19 | 20 | The diagram below shows the primary components involved in the solution. 21 | ![ethereum_staking_eks](./images/ethereum_staking_eks.jpg "Architecture") 22 | 23 | 24 | ## Deploy this solution 25 | 26 | Before deploying, ensure your local environment is configured with your AWS credentials and the region to deploy to. See the [CDK docs](https://docs.aws.amazon.com/cdk/v2/guide/cli.html#getting_started_auth) for more information. 27 | 28 | ``` 29 | git clone 30 | cd 31 | npm install 32 | ``` 33 | 34 | #### Install CDK 35 | You need to have the CDK cli installed locally. [Follow the steps here](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) to install it. Note: version 2.81 of the CDK has a known issue that prevents all the stacks from being deployed at once. Please use version 2.71. If you are installing via npm, you can run ```npm install -g aws-cdk@v2.71.0```. 36 | 37 | #### Configure CDK AWS authentication 38 | [Follow the steps here](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_auth) to configure your AWS credentials. 39 | 40 | #### Bootstrap CDK 41 | If this is the first time CDK is being deployed in this account, you need to bootstrap the account: 42 | ```cdk bootstrap``` 43 | 44 | #### Create an Organization 45 | An Organization is required to use Amazon Managed Service for Grafana. If the AWS account is not already part of an Organzation, you can create one on the [Organization page](https://us-east-1.console.aws.amazon.com/organizations/v2/home/accounts). 46 | 47 | #### Enable IAM Identity Center and Create a User 48 | In order to log into the Grafana dashboard, you need to create a user from the IAM Identity Center. Visit here to enable [Identity Center](https://us-east-1.console.aws.amazon.com/singlesignon/home), and to create a user. 49 | 50 | #### Deploy the CDK 51 | ```cdk deploy -c cluster_name=ethereum-staking-cluster -c availability_zones=us-east-1a,us-east-1b,us-east-1c --app "npx ts-node --prefer-ts-exts bin/app.ts" --all --require-approval never``` 52 | 53 | This will deploy multiple Cloudformation stacks. This will take around one hour to complete. Once the stacks have deployed, we can deploy the Ethereum clients. 54 | 55 | ### Connect to kubectl 56 | We need to communicate with our EKS cluster from our Bastion host, as the cluster is not accessible via the public internet. We can either connect to the Bastion Host via Session Manager, or ssh from our local environment. In this walk through we will connect from our local environment. 57 | 58 | 1. Download the ssh key we created in the solution. The key is in [Parameter Store](https://us-east-1.console.aws.amazon.com/systems-manager/parameters/?region=us-east-1&tab=Table) and has a naming convention like `/ec2/keypair/key-123abc...` 59 | 2. Click the key name, and then click **Show**. 60 | 3. On your local device, `cd ~/.ssh` and create a new file called `bastionHostKeyPair.pem`. For example, with `vi`: ```vi bastionHostKeyPair.pem``` 61 | 4. Add the contents of the key you copied from Parameter Store into `bastionHostKeyPair.pem` and save the file. 62 | 5. Set the correct file permissions: ```chmod 400 bastionHostKeyPair.pem``` 63 | 5. Connect to the Bastion Host. You can find the public IP address of your Bastion Host on the [EC2 console](https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#). You can find an example of how to connect on the `Connect` page of the `EKS/BastionEKSHost` instance. It will be something like this: ```ssh -i ~/.ssh/bastionHostKeyPair.pem ec2-user@``` 64 | 65 | ### Update the kubeconfig file 66 | In order to access the EKS cluster, we must first authorize our local environment. The **Outputs** tab of our `EKS` Cloudformation stack includes the command we can run to authorize our local environment to work with the cluster. 67 | 1. From the [Cloudformation Console](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1), open the `EKS` stack you just deployed, and browse the Outputs tab. Look for a Key called `EKSClusterConfigCommand`. Copy the value. 68 | 2. From the Bastion host, paste this value, and click enter. 69 | 70 | Your local environment is now authorized to communicate with your EKS cluster. 71 | 72 | Next we need to configure your local environment's `kubeconfig` file for `kubectl`. We need to supply the name of our EKS cluster to the command below. The EKS cluster name can be found within the command you ran above for updating the kubeconfig file. You can also find it on the [EKS console](https://us-east-1.console.aws.amazon.com/eks/home?region=us-east-1#/clusters): ```aws eks update-kubeconfig --name --region us-east-1``` 73 | 74 | ### Install Helm 75 | Helm should already be installed on the Bastion host. If you need to install it, you can follow the steps on the [Helm website](https://v3-1-0.helm.sh/docs/intro/install/#from-script). 76 | 77 | At this point we have a running EKS cluster and a Bastion host that is authorized to manage the cluster. Pat yourself on the back, you've already come a long way. From here on out, we are going to be installing the Ethereum clients. 78 | 79 | ## Install the Ethereum clients and Prometheus monitoring 80 | We use [Helm charts from Stakewise](https://docs.stakewise.io/node-operator/stakewise-infra-package/usage#eth2-consensus-beacon-nodes) to deploy the clients. 81 | 82 | From the Bastion host: 83 | 84 | ### Install the StorageClass 85 | 1. We need to create our `storageClass` that our clients will use. We create a storageclass for `gp3` volumes. Copy the contents of the `lib/storageClass.yaml` file into an identically named file, on the Bastion host, and then run: ```kubectl apply -f storageClass.yaml``` 86 | 87 | ### Generate a shared JWT 88 | 2. The execution and consensus client share a JWT to authenticate. We generate and set one in our environment. 89 | ```export JWT=`openssl rand -hex 32` ``` 90 | 91 | 92 | ### Install the Geth execution client 93 | We install three replicas of the Geth client. 94 | 95 | ``` 96 | helm repo add stakewise https://charts.stakewise.io 97 | helm repo update 98 | helm upgrade --debug --install geth stakewise/geth \ 99 | --set='global.replicaCount=3' \ 100 | --set='global.network=mainnet' \ 101 | --set='global.metrics.enabled=true' \ 102 | --set='global.metrics.serviceMonitor.enabled=true' \ 103 | --set='global.metrics.prometheusRule.enabled=true' \ 104 | --set='global.livenessProbe.enabled=false' \ 105 | --set='global.readinessProbe.enabled=false' \ 106 | --set='persistence.storageClassName=ebs-gp3-storageclass' \ 107 | --set='metrics.serviceMonitor.namespace=monitoring' \ 108 | --set='metrics.serviceMonitor.additionalLabels.release=kube-prometheus' \ 109 | --set='global.JWTSecret='${JWT} \ 110 | --create-namespace \ 111 | --namespace chain 112 | ``` 113 | 114 | ### Install the Lighthouse consensus client 115 | Install Lighthouse, once the geth pods are up and running. 116 | ``` 117 | helm upgrade --debug --install lighthouse stakewise/lighthouse \ 118 | --set='global.network=mainnet' \ 119 | --set='global.JWTSecret='${JWT} \ 120 | --set='global.executionEndpoints[0]=http://geth-0.geth.chain:8551' \ 121 | --set='global.executionEndpoints[1]=http://geth-1.geth.chain:8551' \ 122 | --set='global.executionEndpoints[2]=http://geth-2.geth.chain:8551' \ 123 | --set='global.metrics.enabled=true' \ 124 | --set='global.metrics.serviceMonitor.enabled=true' \ 125 | --set='global.metrics.prometheusRule.enabled=true' \ 126 | --set='global.livenessProbe.enabled=false' \ 127 | --set='global.readinessProbe.enabled=false' \ 128 | --set='checkpointSyncUrl=https://beaconstate.ethstaker.cc/' \ 129 | --set='persistence.storageClassName=ebs-gp3-storageclass' \ 130 | --set='metrics.serviceMonitor.namespace=monitoring' \ 131 | --set='metrics.serviceMonitor.additionalLabels.release=kube-prometheus' \ 132 | --create-namespace \ 133 | --namespace chain 134 | ``` 135 | 136 | ## Review the setup 137 | To monitor the pods and logs, these commands may be helpful: 138 | ```kubectl get pods -A``` 139 | ```kubectl logs geth-0 -n chain``` 140 | ```kubectl describe pod geth-0 -n chain``` 141 | ```kubectl exec -i -t geth-0 --container geth -n chain -- sh``` -- connect to the Container 142 | 143 | 144 | ## Logging and Monitoring 145 | 146 | ### Logging 147 | 148 | Check `/aws/containerinsights//application` CloudWatch log group for the client logs. For checking EKS/System logs, check the following logs groups: 149 | ``` 150 | /aws/containerinsights//host 151 | /aws/containerinsights//dataplane 152 | ``` 153 | 154 | ### Monitoring 155 | For accessing Grafana, [follow the steps here](https://docs.aws.amazon.com/grafana/latest/userguide/AMG-manage-users-and-groups-AMG.html) to configure AWS IAM Identity Center users/user groups. 156 | 157 | Once you setup the users & obtain the access to Grafana dashboard, add the Amazon Prometheus as a data source by [following the steps here](https://docs.aws.amazon.com/grafana/latest/userguide/AMP-adding-AWS-config.html). 158 | 159 | [Follow the steps here](https://docs.aws.amazon.com/grafana/latest/userguide/v9-dash-managing-dashboards.html#v9-dash-export-import-dashboards) to import the following dashboards to visualize EKS & Geth metrics: 160 | 161 | | Dashboard | ID/Json | 162 | |:-------------------------------------------|:--------| 163 | | Geth System Resources | [dashboard.json](./dashboards/geth_system_resources.json) | 164 | | Geth Transaction Processing | [dashboard.json](./dashboards/geth_transaction_processing.json) | 165 | | Lighthouse Consensus | [dashboard.json](./dashboards/lighthouse_dashboard.json) | 166 | | k8s-views-global | 15757 | 167 | | k8s-views-namespaces | 15758 | 168 | | k8s-views-nodes | 15759 | 169 | | k8s-views-pods | 15760 | 170 | | k8s-system-api-server | 15761 | 171 | | k8s-system-coredns | 15762 | 172 | | k8s-addons-prometheus | 19105 | 173 | 174 | ## Security 175 | 176 | See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information. 177 | 178 | ## License 179 | 180 | This library is licensed under the MIT-0 License. See the LICENSE file. 181 | -------------------------------------------------------------------------------- /bin/app.ts: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | import { App, Aspects } from 'aws-cdk-lib'; 3 | import { Construct } from 'constructs'; 4 | import { EKS } from '../lib/eks'; 5 | import { EKSk8sBaseline } from '../lib/k8s-baseline'; 6 | import { NodeGroup } from '../lib/nodegroup'; 7 | import { Observe } from '../lib/observe'; 8 | import { VPC } from '../lib/vpc'; 9 | import { AwsSolutionsChecks, NagSuppressions } from 'cdk-nag'; 10 | 11 | const DEFAULT_CONFIG = { 12 | env: { 13 | account: process.env.CDK_DEFAULT_ACCOUNT, 14 | region: process.env.CDK_DEFAULT_REGION, 15 | }, 16 | }; 17 | 18 | const app = new App(); 19 | Aspects.of(app).add(new AwsSolutionsChecks({verbose: true})); 20 | const prefix = stackPrefix(app); 21 | 22 | const vpc = new VPC(app, 'VPC', { 23 | env: DEFAULT_CONFIG.env, 24 | }); 25 | 26 | const eks = new EKS(app, 'EKS', { 27 | env: DEFAULT_CONFIG.env, 28 | stackName: `${prefix}EKS`, 29 | eksVpc: vpc.vpc, 30 | }); 31 | 32 | const nodegroups = new NodeGroup(app, 'NodeGroup', { 33 | env: DEFAULT_CONFIG.env, 34 | stackName: `${prefix}NodeGroup`, 35 | eksVpc: vpc.vpc, 36 | eksCluster: eks.cluster, 37 | nodeGroupRole: eks.createNodegroupRole('erigon-ng'), 38 | bastionSecurityGroup: eks.bastionSecurityGroup, 39 | eksKms: eks.kms, 40 | }); 41 | 42 | const k8sbase = new EKSk8sBaseline(app, 'EKSK8sBaseline', { 43 | env: DEFAULT_CONFIG.env, 44 | stackName: `${prefix}EKSK8sBaseline`, 45 | eksCluster: eks.cluster, 46 | eksKms: eks.kms, 47 | }); 48 | 49 | const amp = new Observe(app, 'Observe', { 50 | env: DEFAULT_CONFIG.env, 51 | stackName: `${prefix}Observe`, 52 | eksKms: eks.kms, 53 | eksCluster: eks.cluster, 54 | vpc: vpc.vpc, 55 | }); 56 | 57 | NagSuppressions.addStackSuppressions(nodegroups,[ 58 | { id: 'AwsSolutions-EC23', reason: 'Nodes are run within private VPC' } 59 | ]); 60 | 61 | NagSuppressions.addStackSuppressions(vpc,[ 62 | { id: 'AwsSolutions-VPC7', reason: 'No VPC Flow Log required for PoC-grade deployment' }, 63 | ]); 64 | 65 | NagSuppressions.addStackSuppressions(k8sbase,[ 66 | { id: 'AwsSolutions-IAM5', reason: 'Permission to read CF stack is restrictive enough' }, 67 | ], true); 68 | 69 | NagSuppressions.addStackSuppressions(eks,[ 70 | { id: 'AwsSolutions-IAM4', reason: 'AWSLambdaBasicExecutionRole, AWSLambdaVPCAccessExecutionRole, AmazonEKS* are restrictive roles' }, 71 | { id: 'AwsSolutions-IAM5', reason: 'Permission to read CF stack is restrictive enough' }, 72 | { id: 'AwsSolutions-L1', reason: 'Non-container Lambda function managed by predefined EKS templates for CDK' }, 73 | { id: 'AwsSolutions-EC23', reason: 'Bastion host must be publicly accessible' }, 74 | { id: 'AwsSolutions-EC28', reason: 'Detailed monitoring not required for PoC-grade deployment' }, 75 | { id: 'AwsSolutions-EC29', reason: 'Termination is disabled via property override, which cdk-nag does not account for' }, 76 | ], true); 77 | 78 | NagSuppressions.addStackSuppressions(amp,[ 79 | { id: 'AwsSolutions-IAM5', reason: 'Permission to read CF stack is restrictive enough' }, 80 | ], true); 81 | 82 | nodegroups.addDependency(eks); 83 | k8sbase.addDependency(nodegroups); 84 | amp.addDependency(nodegroups); 85 | 86 | function stackPrefix(stack: Construct): string { 87 | const prefixValue = stack.node.tryGetContext('stack_prefix'); 88 | 89 | if (prefixValue !== undefined) { 90 | return prefixValue.trim(); 91 | } 92 | // if no stack_prefix return empty string 93 | return ''; 94 | } 95 | app.synth(); 96 | -------------------------------------------------------------------------------- /cdk.json: -------------------------------------------------------------------------------- 1 | { 2 | "app": "npx ts-node --prefer-ts-exts bin/app.ts", 3 | "watch": { 4 | "include": ["**"], 5 | "exclude": [ 6 | "README.md", 7 | "cdk*.json", 8 | "**/*.d.ts", 9 | "**/*.js", 10 | "tsconfig.json", 11 | "package*.json", 12 | "yarn.lock", 13 | "node_modules", 14 | "test" 15 | ] 16 | }, 17 | "context": { 18 | "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true, 19 | "@aws-cdk/core:stackRelativeExports": true, 20 | "@aws-cdk/aws-rds:lowercaseDbIdentifier": true, 21 | "@aws-cdk/aws-lambda:recognizeVersionProps": true, 22 | "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true, 23 | "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true, 24 | "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true, 25 | "@aws-cdk/core:target-partitions": ["aws", "aws-cn"], 26 | "vpcCidr": "10.5.0.0/16", 27 | "nodeGroupMaxSize": 10, 28 | "nodeGroupDesiredSize": 2, 29 | "nodeGroupMinSize": 2 30 | } 31 | } 32 | -------------------------------------------------------------------------------- /config/config.interface.ts: -------------------------------------------------------------------------------- 1 | import { 2 | InstanceType, 3 | AmazonLinuxImage, 4 | BlockDevice, 5 | EbsDeviceOptions, 6 | } from 'aws-cdk-lib/aws-ec2'; 7 | 8 | export interface Config { 9 | accountId: string; 10 | 11 | region: string; 12 | 13 | // Cloudformation Stack Name 14 | stackName: string; 15 | 16 | vpcId?: string; 17 | 18 | instanceName: string; 19 | 20 | availabilityZone: string; 21 | 22 | userDataPath: string; 23 | 24 | instanceType: InstanceType; 25 | 26 | machineImage: AmazonLinuxImage; 27 | 28 | blockDevices?: BlockDevice[]; 29 | } 30 | -------------------------------------------------------------------------------- /dashboards/geth_system_resources.json: -------------------------------------------------------------------------------- 1 | { 2 | "annotations": { 3 | "list": [ 4 | { 5 | "builtIn": 1, 6 | "datasource": "-- Grafana --", 7 | "enable": true, 8 | "hide": true, 9 | "iconColor": "rgba(0, 211, 255, 1)", 10 | "name": "Annotations & Alerts", 11 | "target": { 12 | "limit": 100, 13 | "matchAny": false, 14 | "tags": [], 15 | "type": "dashboard" 16 | }, 17 | "type": "dashboard" 18 | } 19 | ] 20 | }, 21 | "editable": true, 22 | "fiscalYearStartMonth": 0, 23 | "graphTooltip": 0, 24 | "id": 157, 25 | "iteration": 1650530164929, 26 | "links": [], 27 | "liveNow": false, 28 | "panels": [ 29 | { 30 | "datasource": { 31 | "type": "prometheus", 32 | "uid": "${datasource}" 33 | }, 34 | "description": "", 35 | "fieldConfig": { 36 | "defaults": { 37 | "color": { 38 | "mode": "thresholds" 39 | }, 40 | "mappings": [ 41 | { 42 | "options": { 43 | "0": { 44 | "color": "text", 45 | "index": 0, 46 | "text": "syncing" 47 | } 48 | }, 49 | "type": "value" 50 | } 51 | ], 52 | "thresholds": { 53 | "mode": "absolute", 54 | "steps": [ 55 | { 56 | "color": "blue", 57 | "value": null 58 | } 59 | ] 60 | }, 61 | "unit": "locale" 62 | }, 63 | "overrides": [] 64 | }, 65 | "gridPos": { 66 | "h": 3, 67 | "w": 4, 68 | "x": 0, 69 | "y": 0 70 | }, 71 | "id": 4, 72 | "links": [], 73 | "maxDataPoints": 100, 74 | "options": { 75 | "colorMode": "background", 76 | "graphMode": "none", 77 | "justifyMode": "auto", 78 | "orientation": "auto", 79 | "reduceOptions": { 80 | "calcs": [ 81 | "lastNotNull" 82 | ], 83 | "fields": "", 84 | "values": false 85 | }, 86 | "text": {}, 87 | "textMode": "auto" 88 | }, 89 | "pluginVersion": "8.4.2", 90 | "targets": [ 91 | { 92 | "datasource": { 93 | "type": "prometheus", 94 | "uid": "${datasource}" 95 | }, 96 | "exemplar": false, 97 | "expr": "chain_head_block{namespace=\"$namespace\",pod=\"$pod\"}", 98 | "format": "time_series", 99 | "instant": true, 100 | "interval": "", 101 | "intervalFactor": 1, 102 | "legendFormat": "{{pod}}", 103 | "refId": "A" 104 | } 105 | ], 106 | "title": "Current Block", 107 | "transparent": true, 108 | "type": "stat" 109 | }, 110 | { 111 | "datasource": { 112 | "type": "prometheus", 113 | "uid": "${datasource}" 114 | }, 115 | "fieldConfig": { 116 | "defaults": { 117 | "color": { 118 | "mode": "palette-classic" 119 | }, 120 | "custom": { 121 | "axisLabel": "", 122 | "axisPlacement": "auto", 123 | "barAlignment": 0, 124 | "drawStyle": "bars", 125 | "fillOpacity": 100, 126 | "gradientMode": "none", 127 | "hideFrom": { 128 | "legend": false, 129 | "tooltip": false, 130 | "viz": false 131 | }, 132 | "lineInterpolation": "linear", 133 | "lineWidth": 0, 134 | "pointSize": 5, 135 | "scaleDistribution": { 136 | "type": "linear" 137 | }, 138 | "showPoints": "never", 139 | "spanNulls": true, 140 | "stacking": { 141 | "group": "A", 142 | "mode": "none" 143 | }, 144 | "thresholdsStyle": { 145 | "mode": "off" 146 | } 147 | }, 148 | "mappings": [], 149 | "thresholds": { 150 | "mode": "absolute", 151 | "steps": [ 152 | { 153 | "color": "green", 154 | "value": null 155 | }, 156 | { 157 | "color": "red", 158 | "value": 80 159 | } 160 | ] 161 | }, 162 | "unit": "locale" 163 | }, 164 | "overrides": [ 165 | { 166 | "matcher": { 167 | "id": "byName", 168 | "options": "Transactions" 169 | }, 170 | "properties": [ 171 | { 172 | "id": "color", 173 | "value": { 174 | "fixedColor": "#e0752d", 175 | "mode": "fixed" 176 | } 177 | } 178 | ] 179 | } 180 | ] 181 | }, 182 | "gridPos": { 183 | "h": 6, 184 | "w": 7, 185 | "x": 4, 186 | "y": 0 187 | }, 188 | "id": 8, 189 | "links": [], 190 | "options": { 191 | "legend": { 192 | "calcs": [ 193 | "mean", 194 | "max", 195 | "min" 196 | ], 197 | "displayMode": "list", 198 | "placement": "bottom" 199 | }, 200 | "tooltip": { 201 | "mode": "single", 202 | "sort": "none" 203 | } 204 | }, 205 | "pluginVersion": "8.3.6", 206 | "targets": [ 207 | { 208 | "datasource": { 209 | "type": "prometheus", 210 | "uid": "${datasource}" 211 | }, 212 | "exemplar": false, 213 | "expr": "rate(txpool_known{namespace=\"$namespace\",pod=\"$pod\"}[1m])", 214 | "format": "time_series", 215 | "instant": false, 216 | "interval": "", 217 | "intervalFactor": 1, 218 | "legendFormat": "{{pod}}", 219 | "refId": "A" 220 | } 221 | ], 222 | "title": "Transaction Rate (tx/s)", 223 | "type": "timeseries" 224 | }, 225 | { 226 | "datasource": { 227 | "type": "prometheus", 228 | "uid": "${datasource}" 229 | }, 230 | "fieldConfig": { 231 | "defaults": { 232 | "color": { 233 | "mode": "palette-classic" 234 | }, 235 | "custom": { 236 | "axisLabel": "", 237 | "axisPlacement": "auto", 238 | "barAlignment": 0, 239 | "drawStyle": "bars", 240 | "fillOpacity": 100, 241 | "gradientMode": "none", 242 | "hideFrom": { 243 | "legend": false, 244 | "tooltip": false, 245 | "viz": false 246 | }, 247 | "lineInterpolation": "linear", 248 | "lineWidth": 1, 249 | "pointSize": 5, 250 | "scaleDistribution": { 251 | "type": "linear" 252 | }, 253 | "showPoints": "never", 254 | "spanNulls": true, 255 | "stacking": { 256 | "group": "A", 257 | "mode": "none" 258 | }, 259 | "thresholdsStyle": { 260 | "mode": "off" 261 | } 262 | }, 263 | "mappings": [], 264 | "thresholds": { 265 | "mode": "absolute", 266 | "steps": [ 267 | { 268 | "color": "green", 269 | "value": null 270 | }, 271 | { 272 | "color": "red", 273 | "value": 80 274 | } 275 | ] 276 | }, 277 | "unit": "locale" 278 | }, 279 | "overrides": [ 280 | { 281 | "matcher": { 282 | "id": "byName", 283 | "options": "gwei" 284 | }, 285 | "properties": [ 286 | { 287 | "id": "color", 288 | "value": { 289 | "fixedColor": "#0a50a1", 290 | "mode": "fixed" 291 | } 292 | } 293 | ] 294 | }, 295 | { 296 | "matcher": { 297 | "id": "byName", 298 | "options": "transactions" 299 | }, 300 | "properties": [ 301 | { 302 | "id": "color", 303 | "value": { 304 | "fixedColor": "#cca300", 305 | "mode": "fixed" 306 | } 307 | } 308 | ] 309 | } 310 | ] 311 | }, 312 | "gridPos": { 313 | "h": 6, 314 | "w": 7, 315 | "x": 11, 316 | "y": 0 317 | }, 318 | "id": 10, 319 | "links": [], 320 | "options": { 321 | "legend": { 322 | "calcs": [ 323 | "sum" 324 | ], 325 | "displayMode": "list", 326 | "placement": "bottom" 327 | }, 328 | "tooltip": { 329 | "mode": "single", 330 | "sort": "none" 331 | } 332 | }, 333 | "pluginVersion": "8.3.6", 334 | "targets": [ 335 | { 336 | "datasource": { 337 | "type": "prometheus", 338 | "uid": "${datasource}" 339 | }, 340 | "exemplar": true, 341 | "expr": "txpool_pending{namespace=\"$namespace\",pod=\"$pod\"}", 342 | "format": "time_series", 343 | "instant": false, 344 | "interval": "", 345 | "intervalFactor": 1, 346 | "legendFormat": "{{pod}}", 347 | "refId": "A" 348 | } 349 | ], 350 | "title": "Pending Transactions", 351 | "type": "timeseries" 352 | }, 353 | { 354 | "datasource": { 355 | "type": "prometheus", 356 | "uid": "${datasource}" 357 | }, 358 | "fieldConfig": { 359 | "defaults": { 360 | "color": { 361 | "mode": "thresholds" 362 | }, 363 | "mappings": [], 364 | "thresholds": { 365 | "mode": "absolute", 366 | "steps": [ 367 | { 368 | "color": "super-light-yellow", 369 | "value": null 370 | } 371 | ] 372 | }, 373 | "unit": "bytes" 374 | }, 375 | "overrides": [] 376 | }, 377 | "gridPos": { 378 | "h": 3, 379 | "w": 3, 380 | "x": 18, 381 | "y": 0 382 | }, 383 | "id": 12, 384 | "options": { 385 | "colorMode": "none", 386 | "graphMode": "none", 387 | "justifyMode": "auto", 388 | "orientation": "auto", 389 | "reduceOptions": { 390 | "calcs": [ 391 | "lastNotNull" 392 | ], 393 | "fields": "", 394 | "values": false 395 | }, 396 | "textMode": "auto" 397 | }, 398 | "pluginVersion": "8.4.2", 399 | "targets": [ 400 | { 401 | "datasource": { 402 | "type": "prometheus", 403 | "uid": "${DS_STAKEWISE-PROD-VALIDATORS}" 404 | }, 405 | "exemplar": true, 406 | "expr": "system_memory_used", 407 | "interval": "", 408 | "legendFormat": "", 409 | "refId": "A" 410 | } 411 | ], 412 | "title": "Geth Memory Usage", 413 | "transparent": true, 414 | "type": "stat" 415 | }, 416 | { 417 | "datasource": { 418 | "type": "prometheus", 419 | "uid": "${datasource}" 420 | }, 421 | "fieldConfig": { 422 | "defaults": { 423 | "color": { 424 | "mode": "thresholds" 425 | }, 426 | "mappings": [], 427 | "thresholds": { 428 | "mode": "absolute", 429 | "steps": [ 430 | { 431 | "color": "super-light-yellow", 432 | "value": null 433 | } 434 | ] 435 | }, 436 | "unit": "none" 437 | }, 438 | "overrides": [] 439 | }, 440 | "gridPos": { 441 | "h": 3, 442 | "w": 3, 443 | "x": 21, 444 | "y": 0 445 | }, 446 | "id": 22, 447 | "options": { 448 | "colorMode": "none", 449 | "graphMode": "none", 450 | "justifyMode": "auto", 451 | "orientation": "auto", 452 | "reduceOptions": { 453 | "calcs": [ 454 | "lastNotNull" 455 | ], 456 | "fields": "", 457 | "values": false 458 | }, 459 | "textMode": "auto" 460 | }, 461 | "pluginVersion": "8.4.2", 462 | "targets": [ 463 | { 464 | "datasource": { 465 | "type": "prometheus", 466 | "uid": "${DS_STAKEWISE-PROD-VALIDATORS}" 467 | }, 468 | "exemplar": true, 469 | "expr": "system_cpu_threads", 470 | "interval": "", 471 | "legendFormat": "", 472 | "refId": "A" 473 | } 474 | ], 475 | "title": "Geth CPU Threads", 476 | "transparent": true, 477 | "type": "stat" 478 | }, 479 | { 480 | "datasource": { 481 | "type": "prometheus", 482 | "uid": "${datasource}" 483 | }, 484 | "fieldConfig": { 485 | "defaults": { 486 | "color": { 487 | "mode": "thresholds" 488 | }, 489 | "decimals": 2, 490 | "mappings": [ 491 | { 492 | "options": { 493 | "match": "null", 494 | "result": { 495 | "text": "N/A" 496 | } 497 | }, 498 | "type": "special" 499 | } 500 | ], 501 | "thresholds": { 502 | "mode": "absolute", 503 | "steps": [ 504 | { 505 | "color": "text", 506 | "value": null 507 | } 508 | ] 509 | }, 510 | "unit": "locale" 511 | }, 512 | "overrides": [] 513 | }, 514 | "gridPos": { 515 | "h": 3, 516 | "w": 4, 517 | "x": 0, 518 | "y": 3 519 | }, 520 | "id": 6, 521 | "links": [], 522 | "maxDataPoints": 100, 523 | "options": { 524 | "colorMode": "value", 525 | "graphMode": "area", 526 | "justifyMode": "auto", 527 | "orientation": "horizontal", 528 | "reduceOptions": { 529 | "calcs": [ 530 | "lastNotNull" 531 | ], 532 | "fields": "", 533 | "values": false 534 | }, 535 | "textMode": "auto" 536 | }, 537 | "pluginVersion": "8.4.2", 538 | "targets": [ 539 | { 540 | "datasource": { 541 | "type": "prometheus", 542 | "uid": "${datasource}" 543 | }, 544 | "exemplar": false, 545 | "expr": "chain_head_receipt{namespace=\"$namespace\",pod=\"$pod\"}", 546 | "format": "time_series", 547 | "instant": true, 548 | "interval": "", 549 | "intervalFactor": 1, 550 | "legendFormat": "{{pod}}", 551 | "refId": "A" 552 | } 553 | ], 554 | "title": "Last Received Block", 555 | "type": "stat" 556 | }, 557 | { 558 | "datasource": { 559 | "type": "prometheus", 560 | "uid": "${datasource}" 561 | }, 562 | "fieldConfig": { 563 | "defaults": { 564 | "color": { 565 | "mode": "thresholds" 566 | }, 567 | "mappings": [], 568 | "thresholds": { 569 | "mode": "absolute", 570 | "steps": [ 571 | { 572 | "color": "super-light-yellow", 573 | "value": null 574 | } 575 | ] 576 | }, 577 | "unit": "bytes" 578 | }, 579 | "overrides": [] 580 | }, 581 | "gridPos": { 582 | "h": 3, 583 | "w": 3, 584 | "x": 18, 585 | "y": 3 586 | }, 587 | "id": 16, 588 | "options": { 589 | "colorMode": "none", 590 | "graphMode": "none", 591 | "justifyMode": "auto", 592 | "orientation": "auto", 593 | "reduceOptions": { 594 | "calcs": [ 595 | "lastNotNull" 596 | ], 597 | "fields": "", 598 | "values": false 599 | }, 600 | "textMode": "auto" 601 | }, 602 | "pluginVersion": "8.4.2", 603 | "targets": [ 604 | { 605 | "datasource": { 606 | "type": "prometheus", 607 | "uid": "${datasource}" 608 | }, 609 | "exemplar": false, 610 | "expr": "node_memory_MemTotal_bytes", 611 | "instant": true, 612 | "interval": "", 613 | "legendFormat": "", 614 | "refId": "A" 615 | } 616 | ], 617 | "title": "Total System Memory", 618 | "transparent": true, 619 | "type": "stat" 620 | }, 621 | { 622 | "datasource": { 623 | "type": "prometheus", 624 | "uid": "${datasource}" 625 | }, 626 | "fieldConfig": { 627 | "defaults": { 628 | "color": { 629 | "mode": "palette-classic" 630 | }, 631 | "custom": { 632 | "axisLabel": "", 633 | "axisPlacement": "auto", 634 | "barAlignment": 0, 635 | "drawStyle": "line", 636 | "fillOpacity": 0, 637 | "gradientMode": "none", 638 | "hideFrom": { 639 | "legend": false, 640 | "tooltip": false, 641 | "viz": false 642 | }, 643 | "lineInterpolation": "linear", 644 | "lineWidth": 1, 645 | "pointSize": 5, 646 | "scaleDistribution": { 647 | "type": "linear" 648 | }, 649 | "showPoints": "auto", 650 | "spanNulls": false, 651 | "stacking": { 652 | "group": "A", 653 | "mode": "none" 654 | }, 655 | "thresholdsStyle": { 656 | "mode": "off" 657 | } 658 | }, 659 | "mappings": [], 660 | "thresholds": { 661 | "mode": "absolute", 662 | "steps": [ 663 | { 664 | "color": "super-light-yellow", 665 | "value": null 666 | } 667 | ] 668 | }, 669 | "unit": "none" 670 | }, 671 | "overrides": [] 672 | }, 673 | "gridPos": { 674 | "h": 3, 675 | "w": 3, 676 | "x": 21, 677 | "y": 3 678 | }, 679 | "id": 20, 680 | "options": { 681 | "legend": { 682 | "calcs": [], 683 | "displayMode": "hidden", 684 | "placement": "bottom" 685 | }, 686 | "tooltip": { 687 | "mode": "single", 688 | "sort": "none" 689 | } 690 | }, 691 | "pluginVersion": "8.3.6", 692 | "targets": [ 693 | { 694 | "datasource": { 695 | "type": "prometheus", 696 | "uid": "${DS_STAKEWISE-PROD-VALIDATORS}" 697 | }, 698 | "exemplar": true, 699 | "expr": "system_cpu_procload", 700 | "interval": "", 701 | "legendFormat": "", 702 | "refId": "A" 703 | } 704 | ], 705 | "title": "Geth Processor Loading", 706 | "transparent": true, 707 | "type": "timeseries" 708 | }, 709 | { 710 | "datasource": { 711 | "type": "prometheus", 712 | "uid": "${datasource}" 713 | }, 714 | "fieldConfig": { 715 | "defaults": { 716 | "color": { 717 | "mode": "thresholds" 718 | }, 719 | "mappings": [], 720 | "thresholds": { 721 | "mode": "absolute", 722 | "steps": [ 723 | { 724 | "color": "green", 725 | "value": null 726 | } 727 | ] 728 | } 729 | }, 730 | "overrides": [] 731 | }, 732 | "gridPos": { 733 | "h": 6, 734 | "w": 4, 735 | "x": 0, 736 | "y": 6 737 | }, 738 | "id": 24, 739 | "options": { 740 | "orientation": "auto", 741 | "reduceOptions": { 742 | "calcs": [ 743 | "lastNotNull" 744 | ], 745 | "fields": "", 746 | "values": false 747 | }, 748 | "showThresholdLabels": false, 749 | "showThresholdMarkers": true 750 | }, 751 | "pluginVersion": "8.4.2", 752 | "targets": [ 753 | { 754 | "datasource": { 755 | "type": "prometheus", 756 | "uid": "${datasource}" 757 | }, 758 | "exemplar": true, 759 | "expr": "p2p_peers{namespace=\"$namespace\",pod=\"$pod\"}", 760 | "instant": false, 761 | "interval": "", 762 | "legendFormat": "{{pod}}", 763 | "refId": "A" 764 | } 765 | ], 766 | "title": "Peer Count", 767 | "type": "gauge" 768 | }, 769 | { 770 | "datasource": { 771 | "type": "prometheus", 772 | "uid": "${datasource}" 773 | }, 774 | "fieldConfig": { 775 | "defaults": { 776 | "color": { 777 | "mode": "thresholds" 778 | }, 779 | "mappings": [], 780 | "thresholds": { 781 | "mode": "absolute", 782 | "steps": [ 783 | { 784 | "color": "super-light-yellow", 785 | "value": null 786 | } 787 | ] 788 | }, 789 | "unit": "bytes" 790 | }, 791 | "overrides": [] 792 | }, 793 | "gridPos": { 794 | "h": 6, 795 | "w": 4, 796 | "x": 4, 797 | "y": 6 798 | }, 799 | "id": 28, 800 | "options": { 801 | "colorMode": "background", 802 | "graphMode": "none", 803 | "justifyMode": "center", 804 | "orientation": "auto", 805 | "reduceOptions": { 806 | "calcs": [ 807 | "lastNotNull" 808 | ], 809 | "fields": "", 810 | "values": false 811 | }, 812 | "textMode": "auto" 813 | }, 814 | "pluginVersion": "8.4.2", 815 | "targets": [ 816 | { 817 | "datasource": { 818 | "type": "prometheus", 819 | "uid": "${datasource}" 820 | }, 821 | "exemplar": true, 822 | "expr": "rate(p2p_egress{namespace=\"$namespace\",pod=\"$pod\"}[5m])", 823 | "interval": "", 824 | "legendFormat": "Out: {{pod}}", 825 | "refId": "A" 826 | }, 827 | { 828 | "datasource": { 829 | "type": "prometheus", 830 | "uid": "${datasource}" 831 | }, 832 | "exemplar": true, 833 | "expr": "rate(p2p_ingress{namespace=\"$namespace\",pod=\"$pod\"}[5m])", 834 | "hide": false, 835 | "interval": "", 836 | "legendFormat": "In: {{pod}}", 837 | "refId": "B" 838 | } 839 | ], 840 | "title": "Network Traffic (bytes/s)", 841 | "type": "stat" 842 | }, 843 | { 844 | "datasource": { 845 | "type": "prometheus", 846 | "uid": "${datasource}" 847 | }, 848 | "fieldConfig": { 849 | "defaults": { 850 | "color": { 851 | "mode": "thresholds" 852 | }, 853 | "mappings": [], 854 | "thresholds": { 855 | "mode": "absolute", 856 | "steps": [ 857 | { 858 | "color": "super-light-purple", 859 | "value": null 860 | } 861 | ] 862 | }, 863 | "unit": "bytes" 864 | }, 865 | "overrides": [] 866 | }, 867 | "gridPos": { 868 | "h": 6, 869 | "w": 4, 870 | "x": 8, 871 | "y": 6 872 | }, 873 | "id": 30, 874 | "options": { 875 | "colorMode": "background", 876 | "graphMode": "none", 877 | "justifyMode": "center", 878 | "orientation": "auto", 879 | "reduceOptions": { 880 | "calcs": [ 881 | "lastNotNull" 882 | ], 883 | "fields": "", 884 | "values": false 885 | }, 886 | "textMode": "auto" 887 | }, 888 | "pluginVersion": "8.4.2", 889 | "targets": [ 890 | { 891 | "datasource": { 892 | "type": "prometheus", 893 | "uid": "${datasource}" 894 | }, 895 | "exemplar": true, 896 | "expr": "rate(system_disk_readbytes{namespace=\"$namespace\",pod=\"$pod\"}[5m])", 897 | "interval": "", 898 | "legendFormat": "Read: {{pod}}", 899 | "refId": "A" 900 | }, 901 | { 902 | "datasource": { 903 | "type": "prometheus", 904 | "uid": "${datasource}" 905 | }, 906 | "exemplar": true, 907 | "expr": "rate(system_disk_writebytes{namespace=\"$namespace\",pod=\"$pod\"}[5m])", 908 | "hide": false, 909 | "interval": "", 910 | "legendFormat": "Write: {{pod}}", 911 | "refId": "B" 912 | } 913 | ], 914 | "title": "Disk Activity (bytes/s)", 915 | "type": "stat" 916 | }, 917 | { 918 | "datasource": { 919 | "type": "prometheus", 920 | "uid": "${datasource}" 921 | }, 922 | "fieldConfig": { 923 | "defaults": { 924 | "color": { 925 | "fixedColor": "super-light-purple", 926 | "mode": "fixed" 927 | }, 928 | "custom": { 929 | "axisLabel": "", 930 | "axisPlacement": "auto", 931 | "barAlignment": 0, 932 | "drawStyle": "line", 933 | "fillOpacity": 100, 934 | "gradientMode": "none", 935 | "hideFrom": { 936 | "legend": false, 937 | "tooltip": false, 938 | "viz": false 939 | }, 940 | "lineInterpolation": "linear", 941 | "lineStyle": { 942 | "fill": "solid" 943 | }, 944 | "lineWidth": 1, 945 | "pointSize": 5, 946 | "scaleDistribution": { 947 | "type": "linear" 948 | }, 949 | "showPoints": "never", 950 | "spanNulls": true, 951 | "stacking": { 952 | "group": "A", 953 | "mode": "none" 954 | }, 955 | "thresholdsStyle": { 956 | "mode": "off" 957 | } 958 | }, 959 | "mappings": [], 960 | "thresholds": { 961 | "mode": "absolute", 962 | "steps": [ 963 | { 964 | "color": "green", 965 | "value": null 966 | }, 967 | { 968 | "color": "red", 969 | "value": 80 970 | } 971 | ] 972 | }, 973 | "unit": "bytes" 974 | }, 975 | "overrides": [ 976 | { 977 | "matcher": { 978 | "id": "byName", 979 | "options": "Difficulty" 980 | }, 981 | "properties": [ 982 | { 983 | "id": "color", 984 | "value": { 985 | "fixedColor": "rgb(199, 70, 70)", 986 | "mode": "fixed" 987 | } 988 | } 989 | ] 990 | } 991 | ] 992 | }, 993 | "gridPos": { 994 | "h": 6, 995 | "w": 6, 996 | "x": 12, 997 | "y": 6 998 | }, 999 | "id": 42, 1000 | "links": [], 1001 | "options": { 1002 | "legend": { 1003 | "calcs": [ 1004 | "mean", 1005 | "lastNotNull", 1006 | "max", 1007 | "min" 1008 | ], 1009 | "displayMode": "table", 1010 | "placement": "bottom" 1011 | }, 1012 | "tooltip": { 1013 | "mode": "single", 1014 | "sort": "none" 1015 | } 1016 | }, 1017 | "pluginVersion": "8.3.6", 1018 | "targets": [ 1019 | { 1020 | "datasource": { 1021 | "type": "prometheus", 1022 | "uid": "${datasource}" 1023 | }, 1024 | "exemplar": false, 1025 | "expr": "eth_db_chaindata_disk_size{namespace=\"$namespace\",pod=\"$pod\"}", 1026 | "format": "time_series", 1027 | "interval": "", 1028 | "intervalFactor": 1, 1029 | "legendFormat": "{{pod}}", 1030 | "refId": "A" 1031 | } 1032 | ], 1033 | "title": "Chain Data Disk Size", 1034 | "type": "timeseries" 1035 | }, 1036 | { 1037 | "datasource": { 1038 | "type": "prometheus", 1039 | "uid": "${datasource}" 1040 | }, 1041 | "fieldConfig": { 1042 | "defaults": { 1043 | "color": { 1044 | "fixedColor": "text", 1045 | "mode": "fixed" 1046 | }, 1047 | "mappings": [], 1048 | "thresholds": { 1049 | "mode": "absolute", 1050 | "steps": [ 1051 | { 1052 | "color": "green", 1053 | "value": null 1054 | }, 1055 | { 1056 | "color": "red", 1057 | "value": 80 1058 | } 1059 | ] 1060 | } 1061 | }, 1062 | "overrides": [] 1063 | }, 1064 | "gridPos": { 1065 | "h": 6, 1066 | "w": 6, 1067 | "x": 18, 1068 | "y": 6 1069 | }, 1070 | "id": 32, 1071 | "options": { 1072 | "colorMode": "value", 1073 | "graphMode": "area", 1074 | "justifyMode": "auto", 1075 | "orientation": "auto", 1076 | "reduceOptions": { 1077 | "calcs": [ 1078 | "lastNotNull" 1079 | ], 1080 | "fields": "", 1081 | "values": false 1082 | }, 1083 | "textMode": "auto" 1084 | }, 1085 | "pluginVersion": "8.4.2", 1086 | "targets": [ 1087 | { 1088 | "datasource": { 1089 | "type": "prometheus", 1090 | "uid": "${datasource}" 1091 | }, 1092 | "exemplar": true, 1093 | "expr": "rpc_requests{namespace=\"$namespace\",pod=\"$pod\"}", 1094 | "interval": "", 1095 | "legendFormat": "{{pod}}", 1096 | "refId": "A" 1097 | } 1098 | ], 1099 | "title": "RPC Requests", 1100 | "type": "stat" 1101 | }, 1102 | { 1103 | "datasource": { 1104 | "type": "prometheus", 1105 | "uid": "${datasource}" 1106 | }, 1107 | "fieldConfig": { 1108 | "defaults": { 1109 | "color": { 1110 | "fixedColor": "light-purple", 1111 | "mode": "fixed" 1112 | }, 1113 | "custom": { 1114 | "axisLabel": "", 1115 | "axisPlacement": "auto", 1116 | "barAlignment": 0, 1117 | "drawStyle": "line", 1118 | "fillOpacity": 0, 1119 | "gradientMode": "none", 1120 | "hideFrom": { 1121 | "legend": false, 1122 | "tooltip": false, 1123 | "viz": false 1124 | }, 1125 | "lineInterpolation": "linear", 1126 | "lineStyle": { 1127 | "fill": "solid" 1128 | }, 1129 | "lineWidth": 1, 1130 | "pointSize": 3, 1131 | "scaleDistribution": { 1132 | "type": "linear" 1133 | }, 1134 | "showPoints": "always", 1135 | "spanNulls": true, 1136 | "stacking": { 1137 | "group": "A", 1138 | "mode": "none" 1139 | }, 1140 | "thresholdsStyle": { 1141 | "mode": "off" 1142 | } 1143 | }, 1144 | "mappings": [], 1145 | "thresholds": { 1146 | "mode": "absolute", 1147 | "steps": [ 1148 | { 1149 | "color": "green", 1150 | "value": null 1151 | }, 1152 | { 1153 | "color": "red", 1154 | "value": 80 1155 | } 1156 | ] 1157 | }, 1158 | "unit": "bytes" 1159 | }, 1160 | "overrides": [] 1161 | }, 1162 | "gridPos": { 1163 | "h": 6, 1164 | "w": 8, 1165 | "x": 0, 1166 | "y": 12 1167 | }, 1168 | "id": 40, 1169 | "links": [], 1170 | "options": { 1171 | "legend": { 1172 | "calcs": [ 1173 | "mean", 1174 | "lastNotNull", 1175 | "max", 1176 | "min" 1177 | ], 1178 | "displayMode": "table", 1179 | "placement": "bottom" 1180 | }, 1181 | "tooltip": { 1182 | "mode": "single", 1183 | "sort": "none" 1184 | } 1185 | }, 1186 | "pluginVersion": "8.3.6", 1187 | "targets": [ 1188 | { 1189 | "datasource": { 1190 | "type": "prometheus", 1191 | "uid": "${datasource}" 1192 | }, 1193 | "exemplar": true, 1194 | "expr": "rate(system_disk_writebytes{namespace=\"$namespace\",pod=\"$pod\"}[1m])", 1195 | "format": "time_series", 1196 | "interval": "", 1197 | "intervalFactor": 1, 1198 | "legendFormat": "{{pod}}", 1199 | "refId": "A" 1200 | } 1201 | ], 1202 | "title": "Disk Write Activity (bytes/s)", 1203 | "type": "timeseries" 1204 | }, 1205 | { 1206 | "datasource": { 1207 | "type": "prometheus", 1208 | "uid": "${datasource}" 1209 | }, 1210 | "fieldConfig": { 1211 | "defaults": { 1212 | "color": { 1213 | "fixedColor": "light-purple", 1214 | "mode": "fixed" 1215 | }, 1216 | "custom": { 1217 | "axisLabel": "", 1218 | "axisPlacement": "auto", 1219 | "barAlignment": 0, 1220 | "drawStyle": "line", 1221 | "fillOpacity": 0, 1222 | "gradientMode": "none", 1223 | "hideFrom": { 1224 | "legend": false, 1225 | "tooltip": false, 1226 | "viz": false 1227 | }, 1228 | "lineInterpolation": "linear", 1229 | "lineStyle": { 1230 | "fill": "solid" 1231 | }, 1232 | "lineWidth": 1, 1233 | "pointSize": 3, 1234 | "scaleDistribution": { 1235 | "type": "linear" 1236 | }, 1237 | "showPoints": "always", 1238 | "spanNulls": true, 1239 | "stacking": { 1240 | "group": "A", 1241 | "mode": "none" 1242 | }, 1243 | "thresholdsStyle": { 1244 | "mode": "off" 1245 | } 1246 | }, 1247 | "mappings": [], 1248 | "thresholds": { 1249 | "mode": "absolute", 1250 | "steps": [ 1251 | { 1252 | "color": "green", 1253 | "value": null 1254 | }, 1255 | { 1256 | "color": "red", 1257 | "value": 80 1258 | } 1259 | ] 1260 | }, 1261 | "unit": "bytes" 1262 | }, 1263 | "overrides": [] 1264 | }, 1265 | "gridPos": { 1266 | "h": 6, 1267 | "w": 8, 1268 | "x": 8, 1269 | "y": 12 1270 | }, 1271 | "id": 38, 1272 | "links": [], 1273 | "options": { 1274 | "legend": { 1275 | "calcs": [ 1276 | "mean", 1277 | "lastNotNull", 1278 | "max", 1279 | "min" 1280 | ], 1281 | "displayMode": "table", 1282 | "placement": "bottom" 1283 | }, 1284 | "tooltip": { 1285 | "mode": "single", 1286 | "sort": "none" 1287 | } 1288 | }, 1289 | "pluginVersion": "8.3.6", 1290 | "targets": [ 1291 | { 1292 | "datasource": { 1293 | "type": "prometheus", 1294 | "uid": "${datasource}" 1295 | }, 1296 | "exemplar": true, 1297 | "expr": "rate(system_disk_readbytes{namespace=\"$namespace\",pod=\"$pod\"}[1m])", 1298 | "format": "time_series", 1299 | "interval": "", 1300 | "intervalFactor": 1, 1301 | "legendFormat": "{{pod}}", 1302 | "refId": "A" 1303 | } 1304 | ], 1305 | "title": "Disk Read Activity (bytes/s)", 1306 | "type": "timeseries" 1307 | }, 1308 | { 1309 | "datasource": { 1310 | "type": "prometheus", 1311 | "uid": "${datasource}" 1312 | }, 1313 | "fieldConfig": { 1314 | "defaults": { 1315 | "color": { 1316 | "fixedColor": "yellow", 1317 | "mode": "fixed" 1318 | }, 1319 | "custom": { 1320 | "axisLabel": "", 1321 | "axisPlacement": "auto", 1322 | "barAlignment": 0, 1323 | "drawStyle": "line", 1324 | "fillOpacity": 15, 1325 | "gradientMode": "none", 1326 | "hideFrom": { 1327 | "legend": false, 1328 | "tooltip": false, 1329 | "viz": false 1330 | }, 1331 | "lineInterpolation": "stepAfter", 1332 | "lineWidth": 1, 1333 | "pointSize": 5, 1334 | "scaleDistribution": { 1335 | "type": "linear" 1336 | }, 1337 | "showPoints": "never", 1338 | "spanNulls": true, 1339 | "stacking": { 1340 | "group": "A", 1341 | "mode": "none" 1342 | }, 1343 | "thresholdsStyle": { 1344 | "mode": "off" 1345 | } 1346 | }, 1347 | "mappings": [], 1348 | "thresholds": { 1349 | "mode": "absolute", 1350 | "steps": [ 1351 | { 1352 | "color": "green", 1353 | "value": null 1354 | } 1355 | ] 1356 | }, 1357 | "unit": "bytes" 1358 | }, 1359 | "overrides": [] 1360 | }, 1361 | "gridPos": { 1362 | "h": 6, 1363 | "w": 8, 1364 | "x": 16, 1365 | "y": 12 1366 | }, 1367 | "id": 36, 1368 | "links": [], 1369 | "options": { 1370 | "legend": { 1371 | "calcs": [ 1372 | "last", 1373 | "min", 1374 | "max" 1375 | ], 1376 | "displayMode": "table", 1377 | "placement": "bottom" 1378 | }, 1379 | "tooltip": { 1380 | "mode": "single", 1381 | "sort": "none" 1382 | } 1383 | }, 1384 | "pluginVersion": "8.3.6", 1385 | "targets": [ 1386 | { 1387 | "datasource": { 1388 | "type": "prometheus", 1389 | "uid": "${datasource}" 1390 | }, 1391 | "exemplar": true, 1392 | "expr": "rate(p2p_egress{namespace=\"$namespace\",pod=\"$pod\"}[5m])", 1393 | "format": "time_series", 1394 | "interval": "", 1395 | "intervalFactor": 1, 1396 | "legendFormat": "{{pod}}", 1397 | "refId": "A" 1398 | } 1399 | ], 1400 | "title": "Outgoing Network (bytes/s)", 1401 | "type": "timeseries" 1402 | } 1403 | ], 1404 | "refresh": "1m", 1405 | "schemaVersion": 35, 1406 | "style": "dark", 1407 | "tags": [ 1408 | "eth1.0", 1409 | "ethereum", 1410 | "geth" 1411 | ], 1412 | "templating": { 1413 | "list": [ 1414 | { 1415 | "current": { 1416 | "selected": false, 1417 | "text": "default", 1418 | "value": "default" 1419 | }, 1420 | "hide": 0, 1421 | "includeAll": false, 1422 | "multi": false, 1423 | "name": "datasource", 1424 | "options": [], 1425 | "query": "prometheus", 1426 | "queryValue": "", 1427 | "refresh": 1, 1428 | "regex": "", 1429 | "skipUrlSync": false, 1430 | "type": "datasource" 1431 | }, 1432 | { 1433 | "current": { 1434 | "selected": false, 1435 | "text": "goerli", 1436 | "value": "goerli" 1437 | }, 1438 | "datasource": { 1439 | "type": "prometheus", 1440 | "uid": "${datasource}" 1441 | }, 1442 | "definition": "label_values(vflux_server_active_count{}, namespace)", 1443 | "hide": 0, 1444 | "includeAll": false, 1445 | "multi": false, 1446 | "name": "namespace", 1447 | "options": [], 1448 | "query": { 1449 | "query": "label_values(vflux_server_active_count{}, namespace)", 1450 | "refId": "StandardVariableQuery" 1451 | }, 1452 | "refresh": 1, 1453 | "regex": "", 1454 | "skipUrlSync": false, 1455 | "sort": 0, 1456 | "type": "query" 1457 | }, 1458 | { 1459 | "current": { 1460 | "selected": true, 1461 | "text": "geth-0", 1462 | "value": "geth-0" 1463 | }, 1464 | "datasource": { 1465 | "type": "prometheus", 1466 | "uid": "${datasource}" 1467 | }, 1468 | "definition": "label_values(vflux_server_active_count{}, pod)", 1469 | "hide": 0, 1470 | "includeAll": false, 1471 | "multi": false, 1472 | "name": "pod", 1473 | "options": [], 1474 | "query": { 1475 | "query": "label_values(vflux_server_active_count{}, pod)", 1476 | "refId": "StandardVariableQuery" 1477 | }, 1478 | "refresh": 1, 1479 | "regex": "", 1480 | "skipUrlSync": false, 1481 | "sort": 0, 1482 | "type": "query" 1483 | } 1484 | ] 1485 | }, 1486 | "time": { 1487 | "from": "now-1h", 1488 | "to": "now" 1489 | }, 1490 | "timepicker": {}, 1491 | "timezone": "", 1492 | "title": "Geth System Resources", 1493 | "uid": "C9EVmtFMb3pKJUqdasdvMwVDN", 1494 | "version": 2, 1495 | "weekStart": "" 1496 | } -------------------------------------------------------------------------------- /dashboards/geth_transaction_processing.json: -------------------------------------------------------------------------------- 1 | { 2 | "__inputs": [ 3 | { 4 | "name": "DS_PROMETHEUS", 5 | "label": "Prometheus", 6 | "description": "", 7 | "type": "datasource", 8 | "pluginId": "prometheus", 9 | "pluginName": "Prometheus" 10 | }, 11 | { 12 | "name": "VAR_JOB", 13 | "type": "constant", 14 | "label": "job", 15 | "value": "geth", 16 | "description": "" 17 | } 18 | ], 19 | "__requires": [ 20 | { 21 | "type": "grafana", 22 | "id": "grafana", 23 | "name": "Grafana", 24 | "version": "7.4.0" 25 | }, 26 | { 27 | "type": "panel", 28 | "id": "graph", 29 | "name": "Graph", 30 | "version": "" 31 | }, 32 | { 33 | "type": "datasource", 34 | "id": "prometheus", 35 | "name": "Prometheus", 36 | "version": "1.0.0" 37 | }, 38 | { 39 | "type": "panel", 40 | "id": "stat", 41 | "name": "Stat", 42 | "version": "" 43 | } 44 | ], 45 | "annotations": { 46 | "list": [ 47 | { 48 | "builtIn": 1, 49 | "datasource": "-- Grafana --", 50 | "enable": true, 51 | "hide": true, 52 | "iconColor": "rgba(0, 211, 255, 1)", 53 | "name": "Annotations & Alerts", 54 | "type": "dashboard" 55 | } 56 | ] 57 | }, 58 | "editable": true, 59 | "gnetId": 14053, 60 | "graphTooltip": 0, 61 | "id": null, 62 | "iteration": 1615554713540, 63 | "links": [], 64 | "panels": [ 65 | { 66 | "collapsed": false, 67 | "datasource": "${DS_PROMETHEUS}", 68 | "gridPos": { 69 | "h": 1, 70 | "w": 24, 71 | "x": 0, 72 | "y": 0 73 | }, 74 | "id": 82, 75 | "panels": [], 76 | "title": "System", 77 | "type": "row" 78 | }, 79 | { 80 | "aliasColors": {}, 81 | "bars": false, 82 | "dashLength": 10, 83 | "dashes": false, 84 | "datasource": "${DS_PROMETHEUS}", 85 | "fieldConfig": { 86 | "defaults": { 87 | "custom": {} 88 | }, 89 | "overrides": [] 90 | }, 91 | "fill": 1, 92 | "fillGradient": 0, 93 | "gridPos": { 94 | "h": 6, 95 | "w": 8, 96 | "x": 0, 97 | "y": 1 98 | }, 99 | "hiddenSeries": false, 100 | "id": 106, 101 | "legend": { 102 | "alignAsTable": false, 103 | "avg": false, 104 | "current": false, 105 | "max": false, 106 | "min": false, 107 | "rightSide": false, 108 | "show": true, 109 | "total": false, 110 | "values": false 111 | }, 112 | "lines": true, 113 | "linewidth": 1, 114 | "links": [], 115 | "nullPointMode": "null", 116 | "options": { 117 | "alertThreshold": true 118 | }, 119 | "percentage": false, 120 | "pluginVersion": "7.4.0", 121 | "pointradius": 5, 122 | "points": false, 123 | "renderer": "flot", 124 | "seriesOverrides": [], 125 | "spaceLength": 10, 126 | "stack": false, 127 | "steppedLine": false, 128 | "targets": [ 129 | { 130 | "expr": "system_cpu_sysload{instance=~\"$instance\", job=~\"$job\"}", 131 | "format": "time_series", 132 | "interval": "", 133 | "intervalFactor": 1, 134 | "legendFormat": "system", 135 | "refId": "A" 136 | }, 137 | { 138 | "expr": "system_cpu_syswait{instance=~\"$instance\", job=~\"$job\"}", 139 | "format": "time_series", 140 | "interval": "", 141 | "intervalFactor": 1, 142 | "legendFormat": "iowait", 143 | "refId": "B" 144 | }, 145 | { 146 | "expr": "system_cpu_procload{instance=~\"$instance\", job=~\"$job\"}", 147 | "format": "time_series", 148 | "interval": "", 149 | "intervalFactor": 1, 150 | "legendFormat": "geth", 151 | "refId": "C" 152 | } 153 | ], 154 | "thresholds": [], 155 | "timeFrom": null, 156 | "timeRegions": [], 157 | "timeShift": null, 158 | "title": "CPU", 159 | "tooltip": { 160 | "shared": true, 161 | "sort": 0, 162 | "value_type": "individual" 163 | }, 164 | "type": "graph", 165 | "xaxis": { 166 | "buckets": null, 167 | "mode": "time", 168 | "name": null, 169 | "show": true, 170 | "values": [] 171 | }, 172 | "yaxes": [ 173 | { 174 | "format": "percent", 175 | "label": null, 176 | "logBase": 1, 177 | "max": null, 178 | "min": null, 179 | "show": true 180 | }, 181 | { 182 | "format": "short", 183 | "label": null, 184 | "logBase": 1, 185 | "max": null, 186 | "min": null, 187 | "show": true 188 | } 189 | ], 190 | "yaxis": { 191 | "align": false, 192 | "alignLevel": null 193 | } 194 | }, 195 | { 196 | "aliasColors": {}, 197 | "bars": false, 198 | "dashLength": 10, 199 | "dashes": false, 200 | "datasource": "${DS_PROMETHEUS}", 201 | "fieldConfig": { 202 | "defaults": { 203 | "custom": {} 204 | }, 205 | "overrides": [] 206 | }, 207 | "fill": 1, 208 | "fillGradient": 0, 209 | "gridPos": { 210 | "h": 6, 211 | "w": 8, 212 | "x": 8, 213 | "y": 1 214 | }, 215 | "hiddenSeries": false, 216 | "id": 86, 217 | "legend": { 218 | "alignAsTable": false, 219 | "avg": false, 220 | "current": false, 221 | "max": false, 222 | "min": false, 223 | "rightSide": false, 224 | "show": true, 225 | "total": false, 226 | "values": false 227 | }, 228 | "lines": true, 229 | "linewidth": 1, 230 | "links": [], 231 | "nullPointMode": "null", 232 | "options": { 233 | "alertThreshold": true 234 | }, 235 | "percentage": false, 236 | "pluginVersion": "7.4.0", 237 | "pointradius": 5, 238 | "points": false, 239 | "renderer": "flot", 240 | "seriesOverrides": [], 241 | "spaceLength": 10, 242 | "stack": false, 243 | "steppedLine": false, 244 | "targets": [ 245 | { 246 | "expr": "rate(system_memory_allocs{instance=~\"$instance\", job=~\"$job\"}[1m])", 247 | "format": "time_series", 248 | "interval": "", 249 | "intervalFactor": 1, 250 | "legendFormat": "alloc", 251 | "refId": "A" 252 | }, 253 | { 254 | "expr": "system_memory_used{instance=~\"$instance\", job=~\"$job\"}", 255 | "format": "time_series", 256 | "interval": "", 257 | "intervalFactor": 1, 258 | "legendFormat": "used", 259 | "refId": "B" 260 | }, 261 | { 262 | "expr": "system_memory_held{instance=~\"$instance\", job=~\"$job\"}", 263 | "format": "time_series", 264 | "interval": "", 265 | "intervalFactor": 1, 266 | "legendFormat": "held", 267 | "refId": "C" 268 | } 269 | ], 270 | "thresholds": [], 271 | "timeFrom": null, 272 | "timeRegions": [], 273 | "timeShift": null, 274 | "title": "Memory", 275 | "tooltip": { 276 | "shared": true, 277 | "sort": 0, 278 | "value_type": "individual" 279 | }, 280 | "type": "graph", 281 | "xaxis": { 282 | "buckets": null, 283 | "mode": "time", 284 | "name": null, 285 | "show": true, 286 | "values": [] 287 | }, 288 | "yaxes": [ 289 | { 290 | "format": "bytes", 291 | "label": null, 292 | "logBase": 1, 293 | "max": null, 294 | "min": null, 295 | "show": true 296 | }, 297 | { 298 | "format": "short", 299 | "label": null, 300 | "logBase": 1, 301 | "max": null, 302 | "min": null, 303 | "show": true 304 | } 305 | ], 306 | "yaxis": { 307 | "align": false, 308 | "alignLevel": null 309 | } 310 | }, 311 | { 312 | "aliasColors": {}, 313 | "bars": false, 314 | "dashLength": 10, 315 | "dashes": false, 316 | "datasource": "${DS_PROMETHEUS}", 317 | "fieldConfig": { 318 | "defaults": { 319 | "custom": {} 320 | }, 321 | "overrides": [] 322 | }, 323 | "fill": 1, 324 | "fillGradient": 0, 325 | "gridPos": { 326 | "h": 6, 327 | "w": 8, 328 | "x": 16, 329 | "y": 1 330 | }, 331 | "hiddenSeries": false, 332 | "id": 85, 333 | "legend": { 334 | "alignAsTable": false, 335 | "avg": false, 336 | "current": false, 337 | "max": false, 338 | "min": false, 339 | "rightSide": false, 340 | "show": true, 341 | "total": false, 342 | "values": false 343 | }, 344 | "lines": true, 345 | "linewidth": 1, 346 | "links": [], 347 | "nullPointMode": "null", 348 | "options": { 349 | "alertThreshold": true 350 | }, 351 | "percentage": false, 352 | "pluginVersion": "7.4.0", 353 | "pointradius": 5, 354 | "points": false, 355 | "renderer": "flot", 356 | "seriesOverrides": [], 357 | "spaceLength": 10, 358 | "stack": false, 359 | "steppedLine": false, 360 | "targets": [ 361 | { 362 | "expr": "rate(system_disk_readbytes{instance=~\"$instance\", job=~\"$job\"}[1m])", 363 | "format": "time_series", 364 | "interval": "", 365 | "intervalFactor": 1, 366 | "legendFormat": "read", 367 | "refId": "A" 368 | }, 369 | { 370 | "expr": "rate(system_disk_writebytes{instance=~\"$instance\", job=~\"$job\"}[1m])", 371 | "format": "time_series", 372 | "interval": "", 373 | "intervalFactor": 1, 374 | "legendFormat": "write", 375 | "refId": "B" 376 | } 377 | ], 378 | "thresholds": [], 379 | "timeFrom": null, 380 | "timeRegions": [], 381 | "timeShift": null, 382 | "title": "Disk", 383 | "tooltip": { 384 | "shared": true, 385 | "sort": 0, 386 | "value_type": "individual" 387 | }, 388 | "type": "graph", 389 | "xaxis": { 390 | "buckets": null, 391 | "mode": "time", 392 | "name": null, 393 | "show": true, 394 | "values": [] 395 | }, 396 | "yaxes": [ 397 | { 398 | "format": "Bps", 399 | "label": null, 400 | "logBase": 1, 401 | "max": null, 402 | "min": null, 403 | "show": true 404 | }, 405 | { 406 | "format": "short", 407 | "label": null, 408 | "logBase": 1, 409 | "max": null, 410 | "min": null, 411 | "show": true 412 | } 413 | ], 414 | "yaxis": { 415 | "align": false, 416 | "alignLevel": null 417 | } 418 | }, 419 | { 420 | "collapsed": false, 421 | "datasource": "${DS_PROMETHEUS}", 422 | "gridPos": { 423 | "h": 1, 424 | "w": 24, 425 | "x": 0, 426 | "y": 7 427 | }, 428 | "id": 75, 429 | "panels": [], 430 | "title": "Network", 431 | "type": "row" 432 | }, 433 | { 434 | "aliasColors": {}, 435 | "bars": false, 436 | "dashLength": 10, 437 | "dashes": false, 438 | "datasource": "${DS_PROMETHEUS}", 439 | "fieldConfig": { 440 | "defaults": { 441 | "custom": {} 442 | }, 443 | "overrides": [] 444 | }, 445 | "fill": 1, 446 | "fillGradient": 0, 447 | "gridPos": { 448 | "h": 6, 449 | "w": 12, 450 | "x": 0, 451 | "y": 8 452 | }, 453 | "hiddenSeries": false, 454 | "id": 96, 455 | "legend": { 456 | "alignAsTable": true, 457 | "avg": false, 458 | "current": true, 459 | "max": false, 460 | "min": false, 461 | "rightSide": true, 462 | "show": true, 463 | "total": false, 464 | "values": true 465 | }, 466 | "lines": true, 467 | "linewidth": 1, 468 | "links": [], 469 | "nullPointMode": "null", 470 | "options": { 471 | "alertThreshold": true 472 | }, 473 | "percentage": false, 474 | "pluginVersion": "7.4.0", 475 | "pointradius": 5, 476 | "points": false, 477 | "renderer": "flot", 478 | "seriesOverrides": [], 479 | "spaceLength": 10, 480 | "stack": false, 481 | "steppedLine": false, 482 | "targets": [ 483 | { 484 | "expr": "rate(p2p_ingress{instance=~\"$instance\", job=~\"$job\"}[1m])", 485 | "format": "time_series", 486 | "interval": "", 487 | "intervalFactor": 1, 488 | "legendFormat": "ingress", 489 | "refId": "B" 490 | }, 491 | { 492 | "expr": "rate(p2p_egress{instance=~\"$instance\", job=~\"$job\"}[1m])", 493 | "format": "time_series", 494 | "interval": "", 495 | "intervalFactor": 1, 496 | "legendFormat": "egress", 497 | "refId": "C" 498 | } 499 | ], 500 | "thresholds": [], 501 | "timeFrom": null, 502 | "timeRegions": [], 503 | "timeShift": null, 504 | "title": "Traffic", 505 | "tooltip": { 506 | "shared": true, 507 | "sort": 0, 508 | "value_type": "individual" 509 | }, 510 | "type": "graph", 511 | "xaxis": { 512 | "buckets": null, 513 | "mode": "time", 514 | "name": null, 515 | "show": true, 516 | "values": [] 517 | }, 518 | "yaxes": [ 519 | { 520 | "format": "Bps", 521 | "label": null, 522 | "logBase": 1, 523 | "max": null, 524 | "min": null, 525 | "show": true 526 | }, 527 | { 528 | "format": "short", 529 | "label": null, 530 | "logBase": 1, 531 | "max": null, 532 | "min": null, 533 | "show": true 534 | } 535 | ], 536 | "yaxis": { 537 | "align": false, 538 | "alignLevel": null 539 | } 540 | }, 541 | { 542 | "aliasColors": {}, 543 | "bars": false, 544 | "dashLength": 10, 545 | "dashes": false, 546 | "datasource": "${DS_PROMETHEUS}", 547 | "fieldConfig": { 548 | "defaults": { 549 | "custom": {} 550 | }, 551 | "overrides": [] 552 | }, 553 | "fill": 1, 554 | "fillGradient": 0, 555 | "gridPos": { 556 | "h": 6, 557 | "w": 12, 558 | "x": 12, 559 | "y": 8 560 | }, 561 | "hiddenSeries": false, 562 | "id": 77, 563 | "legend": { 564 | "alignAsTable": true, 565 | "avg": false, 566 | "current": true, 567 | "max": false, 568 | "min": false, 569 | "rightSide": true, 570 | "show": true, 571 | "total": false, 572 | "values": true 573 | }, 574 | "lines": true, 575 | "linewidth": 1, 576 | "links": [], 577 | "nullPointMode": "null", 578 | "options": { 579 | "alertThreshold": true 580 | }, 581 | "percentage": false, 582 | "pluginVersion": "7.4.0", 583 | "pointradius": 5, 584 | "points": false, 585 | "renderer": "flot", 586 | "seriesOverrides": [], 587 | "spaceLength": 10, 588 | "stack": false, 589 | "steppedLine": false, 590 | "targets": [ 591 | { 592 | "expr": "p2p_peers{instance=~\"$instance\", job=~\"$job\"}", 593 | "format": "time_series", 594 | "interval": "", 595 | "intervalFactor": 1, 596 | "legendFormat": "peers", 597 | "refId": "A" 598 | }, 599 | { 600 | "expr": "rate(p2p_dials{instance=~\"$instance\", job=~\"$job\"}[1m])", 601 | "format": "time_series", 602 | "interval": "", 603 | "intervalFactor": 1, 604 | "legendFormat": "dials", 605 | "refId": "B" 606 | }, 607 | { 608 | "expr": "rate(p2p_serves{instance=~\"$instance\", job=~\"$job\"}[1m])", 609 | "format": "time_series", 610 | "interval": "", 611 | "intervalFactor": 1, 612 | "legendFormat": "serves", 613 | "refId": "C" 614 | } 615 | ], 616 | "thresholds": [], 617 | "timeFrom": null, 618 | "timeRegions": [], 619 | "timeShift": null, 620 | "title": "Peers", 621 | "tooltip": { 622 | "shared": true, 623 | "sort": 0, 624 | "value_type": "individual" 625 | }, 626 | "type": "graph", 627 | "xaxis": { 628 | "buckets": null, 629 | "mode": "time", 630 | "name": null, 631 | "show": true, 632 | "values": [] 633 | }, 634 | "yaxes": [ 635 | { 636 | "format": "none", 637 | "label": null, 638 | "logBase": 1, 639 | "max": null, 640 | "min": null, 641 | "show": true 642 | }, 643 | { 644 | "format": "short", 645 | "label": null, 646 | "logBase": 1, 647 | "max": null, 648 | "min": null, 649 | "show": true 650 | } 651 | ], 652 | "yaxis": { 653 | "align": false, 654 | "alignLevel": null 655 | } 656 | }, 657 | { 658 | "collapsed": false, 659 | "datasource": "${DS_PROMETHEUS}", 660 | "gridPos": { 661 | "h": 1, 662 | "w": 24, 663 | "x": 0, 664 | "y": 14 665 | }, 666 | "id": 4, 667 | "panels": [], 668 | "title": "Blockchain", 669 | "type": "row" 670 | }, 671 | { 672 | "cacheTimeout": null, 673 | "datasource": "${DS_PROMETHEUS}", 674 | "fieldConfig": { 675 | "defaults": { 676 | "color": { 677 | "mode": "thresholds" 678 | }, 679 | "custom": {}, 680 | "mappings": [ 681 | { 682 | "id": 0, 683 | "op": "=", 684 | "text": "N/A", 685 | "type": 1, 686 | "value": "null" 687 | } 688 | ], 689 | "thresholds": { 690 | "mode": "absolute", 691 | "steps": [ 692 | { 693 | "color": "rgb(252, 252, 252)", 694 | "value": null 695 | } 696 | ] 697 | }, 698 | "unit": "none" 699 | }, 700 | "overrides": [] 701 | }, 702 | "gridPos": { 703 | "h": 2, 704 | "w": 3, 705 | "x": 0, 706 | "y": 15 707 | }, 708 | "id": 108, 709 | "interval": null, 710 | "links": [], 711 | "maxDataPoints": 100, 712 | "options": { 713 | "colorMode": "value", 714 | "graphMode": "none", 715 | "justifyMode": "auto", 716 | "orientation": "horizontal", 717 | "reduceOptions": { 718 | "calcs": [ 719 | "mean" 720 | ], 721 | "fields": "", 722 | "values": false 723 | }, 724 | "text": {}, 725 | "textMode": "auto" 726 | }, 727 | "pluginVersion": "7.4.0", 728 | "targets": [ 729 | { 730 | "expr": "chain_head_header{instance=~\"$instance\", job=~\"$job\"}", 731 | "format": "time_series", 732 | "interval": "", 733 | "intervalFactor": 1, 734 | "legendFormat": "", 735 | "refId": "A" 736 | } 737 | ], 738 | "timeFrom": null, 739 | "timeShift": null, 740 | "title": "Latest header", 741 | "type": "stat" 742 | }, 743 | { 744 | "aliasColors": {}, 745 | "bars": false, 746 | "cacheTimeout": null, 747 | "dashLength": 10, 748 | "dashes": false, 749 | "datasource": "${DS_PROMETHEUS}", 750 | "fieldConfig": { 751 | "defaults": { 752 | "custom": {} 753 | }, 754 | "overrides": [] 755 | }, 756 | "fill": 1, 757 | "fillGradient": 0, 758 | "gridPos": { 759 | "h": 6, 760 | "w": 9, 761 | "x": 3, 762 | "y": 15 763 | }, 764 | "hiddenSeries": false, 765 | "id": 110, 766 | "legend": { 767 | "avg": false, 768 | "current": false, 769 | "max": false, 770 | "min": false, 771 | "show": true, 772 | "total": false, 773 | "values": false 774 | }, 775 | "lines": true, 776 | "linewidth": 1, 777 | "links": [], 778 | "nullPointMode": "null", 779 | "options": { 780 | "alertThreshold": true 781 | }, 782 | "percentage": false, 783 | "pluginVersion": "7.4.0", 784 | "pointradius": 2, 785 | "points": false, 786 | "renderer": "flot", 787 | "seriesOverrides": [], 788 | "spaceLength": 10, 789 | "stack": false, 790 | "steppedLine": false, 791 | "targets": [ 792 | { 793 | "expr": "chain_head_header{instance=~\"$instance\", job=~\"$job\"}", 794 | "format": "time_series", 795 | "interval": "", 796 | "intervalFactor": 1, 797 | "legendFormat": "header", 798 | "refId": "A" 799 | }, 800 | { 801 | "expr": "chain_head_receipt{instance=~\"$instance\", job=~\"$job\"}", 802 | "format": "time_series", 803 | "interval": "", 804 | "intervalFactor": 1, 805 | "legendFormat": "receipt", 806 | "refId": "B" 807 | }, 808 | { 809 | "expr": "chain_head_block{instance=~\"$instance\", job=~\"$job\"}", 810 | "format": "time_series", 811 | "interval": "", 812 | "intervalFactor": 1, 813 | "legendFormat": "block", 814 | "refId": "C" 815 | } 816 | ], 817 | "thresholds": [], 818 | "timeFrom": null, 819 | "timeRegions": [], 820 | "timeShift": null, 821 | "title": "Chain head", 822 | "tooltip": { 823 | "shared": true, 824 | "sort": 0, 825 | "value_type": "individual" 826 | }, 827 | "type": "graph", 828 | "xaxis": { 829 | "buckets": null, 830 | "mode": "time", 831 | "name": null, 832 | "show": true, 833 | "values": [] 834 | }, 835 | "yaxes": [ 836 | { 837 | "format": "short", 838 | "label": null, 839 | "logBase": 1, 840 | "max": null, 841 | "min": null, 842 | "show": true 843 | }, 844 | { 845 | "format": "short", 846 | "label": null, 847 | "logBase": 1, 848 | "max": null, 849 | "min": null, 850 | "show": true 851 | } 852 | ], 853 | "yaxis": { 854 | "align": false, 855 | "alignLevel": null 856 | } 857 | }, 858 | { 859 | "cacheTimeout": null, 860 | "datasource": "${DS_PROMETHEUS}", 861 | "fieldConfig": { 862 | "defaults": { 863 | "color": { 864 | "mode": "thresholds" 865 | }, 866 | "custom": {}, 867 | "mappings": [ 868 | { 869 | "id": 0, 870 | "op": "=", 871 | "text": "N/A", 872 | "type": 1, 873 | "value": "null" 874 | } 875 | ], 876 | "thresholds": { 877 | "mode": "absolute", 878 | "steps": [ 879 | { 880 | "color": "rgb(255, 255, 255)", 881 | "value": null 882 | } 883 | ] 884 | }, 885 | "unit": "none" 886 | }, 887 | "overrides": [] 888 | }, 889 | "gridPos": { 890 | "h": 2, 891 | "w": 3, 892 | "x": 12, 893 | "y": 15 894 | }, 895 | "id": 113, 896 | "interval": null, 897 | "links": [], 898 | "maxDataPoints": 100, 899 | "options": { 900 | "colorMode": "value", 901 | "graphMode": "none", 902 | "justifyMode": "auto", 903 | "orientation": "horizontal", 904 | "reduceOptions": { 905 | "calcs": [ 906 | "lastNotNull" 907 | ], 908 | "fields": "", 909 | "values": false 910 | }, 911 | "text": {}, 912 | "textMode": "auto" 913 | }, 914 | "pluginVersion": "7.4.0", 915 | "targets": [ 916 | { 917 | "expr": "txpool_pending{instance=~\"$instance\", job=~\"$job\"}", 918 | "format": "time_series", 919 | "interval": "", 920 | "intervalFactor": 1, 921 | "legendFormat": "{{ instance }}", 922 | "refId": "A" 923 | } 924 | ], 925 | "timeFrom": null, 926 | "timeShift": null, 927 | "title": "Executable transactions", 928 | "type": "stat" 929 | }, 930 | { 931 | "aliasColors": {}, 932 | "bars": false, 933 | "cacheTimeout": null, 934 | "dashLength": 10, 935 | "dashes": false, 936 | "datasource": "${DS_PROMETHEUS}", 937 | "fieldConfig": { 938 | "defaults": { 939 | "custom": {} 940 | }, 941 | "overrides": [] 942 | }, 943 | "fill": 1, 944 | "fillGradient": 0, 945 | "gridPos": { 946 | "h": 6, 947 | "w": 9, 948 | "x": 15, 949 | "y": 15 950 | }, 951 | "hiddenSeries": false, 952 | "id": 116, 953 | "legend": { 954 | "avg": false, 955 | "current": false, 956 | "max": false, 957 | "min": false, 958 | "show": true, 959 | "total": false, 960 | "values": false 961 | }, 962 | "lines": true, 963 | "linewidth": 1, 964 | "links": [], 965 | "nullPointMode": "null", 966 | "options": { 967 | "alertThreshold": true 968 | }, 969 | "percentage": false, 970 | "pluginVersion": "7.4.0", 971 | "pointradius": 2, 972 | "points": false, 973 | "renderer": "flot", 974 | "seriesOverrides": [], 975 | "spaceLength": 10, 976 | "stack": false, 977 | "steppedLine": false, 978 | "targets": [ 979 | { 980 | "expr": "txpool_pending{instance=~\"$instance\", job=~\"$job\"}", 981 | "format": "time_series", 982 | "interval": "", 983 | "intervalFactor": 1, 984 | "legendFormat": "executable", 985 | "refId": "A" 986 | }, 987 | { 988 | "expr": "txpool_queued{instance=~\"$instance\", job=~\"$job\"}", 989 | "format": "time_series", 990 | "interval": "", 991 | "intervalFactor": 1, 992 | "legendFormat": "gapped", 993 | "refId": "B" 994 | }, 995 | { 996 | "expr": "txpool_local{instance=~\"$instance\", job=~\"$job\"}", 997 | "format": "time_series", 998 | "interval": "", 999 | "intervalFactor": 1, 1000 | "legendFormat": "local", 1001 | "refId": "C" 1002 | } 1003 | ], 1004 | "thresholds": [], 1005 | "timeFrom": null, 1006 | "timeRegions": [], 1007 | "timeShift": null, 1008 | "title": "Transaction pool", 1009 | "tooltip": { 1010 | "shared": true, 1011 | "sort": 0, 1012 | "value_type": "individual" 1013 | }, 1014 | "type": "graph", 1015 | "xaxis": { 1016 | "buckets": null, 1017 | "mode": "time", 1018 | "name": null, 1019 | "show": true, 1020 | "values": [] 1021 | }, 1022 | "yaxes": [ 1023 | { 1024 | "format": "short", 1025 | "label": null, 1026 | "logBase": 1, 1027 | "max": null, 1028 | "min": null, 1029 | "show": true 1030 | }, 1031 | { 1032 | "format": "short", 1033 | "label": null, 1034 | "logBase": 1, 1035 | "max": null, 1036 | "min": null, 1037 | "show": true 1038 | } 1039 | ], 1040 | "yaxis": { 1041 | "align": false, 1042 | "alignLevel": null 1043 | } 1044 | }, 1045 | { 1046 | "cacheTimeout": null, 1047 | "datasource": "${DS_PROMETHEUS}", 1048 | "fieldConfig": { 1049 | "defaults": { 1050 | "color": { 1051 | "mode": "thresholds" 1052 | }, 1053 | "custom": {}, 1054 | "mappings": [ 1055 | { 1056 | "id": 0, 1057 | "op": "=", 1058 | "text": "N/A", 1059 | "type": 1, 1060 | "value": "null" 1061 | } 1062 | ], 1063 | "thresholds": { 1064 | "mode": "absolute", 1065 | "steps": [ 1066 | { 1067 | "color": "rgb(255, 255, 255)", 1068 | "value": null 1069 | } 1070 | ] 1071 | }, 1072 | "unit": "none" 1073 | }, 1074 | "overrides": [] 1075 | }, 1076 | "gridPos": { 1077 | "h": 2, 1078 | "w": 3, 1079 | "x": 0, 1080 | "y": 17 1081 | }, 1082 | "id": 111, 1083 | "interval": null, 1084 | "links": [], 1085 | "maxDataPoints": 100, 1086 | "options": { 1087 | "colorMode": "value", 1088 | "graphMode": "none", 1089 | "justifyMode": "auto", 1090 | "orientation": "horizontal", 1091 | "reduceOptions": { 1092 | "calcs": [ 1093 | "lastNotNull" 1094 | ], 1095 | "fields": "", 1096 | "values": false 1097 | }, 1098 | "text": {}, 1099 | "textMode": "auto" 1100 | }, 1101 | "pluginVersion": "7.4.0", 1102 | "targets": [ 1103 | { 1104 | "expr": "chain_head_receipt{instance=~\"$instance\", job=~\"$job\"}", 1105 | "format": "time_series", 1106 | "interval": "", 1107 | "intervalFactor": 1, 1108 | "legendFormat": "", 1109 | "refId": "A" 1110 | } 1111 | ], 1112 | "timeFrom": null, 1113 | "timeShift": null, 1114 | "title": "Latest receipt", 1115 | "type": "stat" 1116 | }, 1117 | { 1118 | "cacheTimeout": null, 1119 | "datasource": "${DS_PROMETHEUS}", 1120 | "fieldConfig": { 1121 | "defaults": { 1122 | "color": { 1123 | "mode": "thresholds" 1124 | }, 1125 | "custom": {}, 1126 | "mappings": [ 1127 | { 1128 | "id": 0, 1129 | "op": "=", 1130 | "text": "N/A", 1131 | "type": 1, 1132 | "value": "null" 1133 | } 1134 | ], 1135 | "thresholds": { 1136 | "mode": "absolute", 1137 | "steps": [ 1138 | { 1139 | "color": "rgb(255, 255, 255)", 1140 | "value": null 1141 | } 1142 | ] 1143 | }, 1144 | "unit": "none" 1145 | }, 1146 | "overrides": [] 1147 | }, 1148 | "gridPos": { 1149 | "h": 2, 1150 | "w": 3, 1151 | "x": 12, 1152 | "y": 17 1153 | }, 1154 | "id": 114, 1155 | "interval": null, 1156 | "links": [], 1157 | "maxDataPoints": 100, 1158 | "options": { 1159 | "colorMode": "value", 1160 | "graphMode": "none", 1161 | "justifyMode": "auto", 1162 | "orientation": "horizontal", 1163 | "reduceOptions": { 1164 | "calcs": [ 1165 | "lastNotNull" 1166 | ], 1167 | "fields": "", 1168 | "values": false 1169 | }, 1170 | "text": {}, 1171 | "textMode": "auto" 1172 | }, 1173 | "pluginVersion": "7.4.0", 1174 | "targets": [ 1175 | { 1176 | "expr": "txpool_queued{instance=~\"$instance\", job=~\"$job\"}", 1177 | "format": "time_series", 1178 | "interval": "", 1179 | "intervalFactor": 1, 1180 | "legendFormat": "", 1181 | "refId": "A" 1182 | } 1183 | ], 1184 | "timeFrom": null, 1185 | "timeShift": null, 1186 | "title": "Gapped transactions", 1187 | "type": "stat" 1188 | }, 1189 | { 1190 | "cacheTimeout": null, 1191 | "datasource": "${DS_PROMETHEUS}", 1192 | "fieldConfig": { 1193 | "defaults": { 1194 | "color": { 1195 | "mode": "thresholds" 1196 | }, 1197 | "custom": {}, 1198 | "mappings": [ 1199 | { 1200 | "id": 0, 1201 | "op": "=", 1202 | "text": "N/A", 1203 | "type": 1, 1204 | "value": "null" 1205 | } 1206 | ], 1207 | "thresholds": { 1208 | "mode": "absolute", 1209 | "steps": [ 1210 | { 1211 | "color": "rgb(255, 255, 255)", 1212 | "value": null 1213 | } 1214 | ] 1215 | }, 1216 | "unit": "none" 1217 | }, 1218 | "overrides": [] 1219 | }, 1220 | "gridPos": { 1221 | "h": 2, 1222 | "w": 3, 1223 | "x": 0, 1224 | "y": 19 1225 | }, 1226 | "id": 109, 1227 | "interval": null, 1228 | "links": [], 1229 | "maxDataPoints": 100, 1230 | "options": { 1231 | "colorMode": "value", 1232 | "graphMode": "none", 1233 | "justifyMode": "auto", 1234 | "orientation": "horizontal", 1235 | "reduceOptions": { 1236 | "calcs": [ 1237 | "lastNotNull" 1238 | ], 1239 | "fields": "", 1240 | "values": false 1241 | }, 1242 | "text": {}, 1243 | "textMode": "auto" 1244 | }, 1245 | "pluginVersion": "7.4.0", 1246 | "targets": [ 1247 | { 1248 | "expr": "chain_head_block{instance=~\"$instance\", job=~\"$job\"}", 1249 | "format": "time_series", 1250 | "interval": "", 1251 | "intervalFactor": 1, 1252 | "legendFormat": "", 1253 | "refId": "A" 1254 | } 1255 | ], 1256 | "timeFrom": null, 1257 | "timeShift": null, 1258 | "title": "Latest block", 1259 | "type": "stat" 1260 | }, 1261 | { 1262 | "cacheTimeout": null, 1263 | "datasource": "${DS_PROMETHEUS}", 1264 | "fieldConfig": { 1265 | "defaults": { 1266 | "color": { 1267 | "mode": "thresholds" 1268 | }, 1269 | "custom": {}, 1270 | "mappings": [ 1271 | { 1272 | "id": 0, 1273 | "op": "=", 1274 | "text": "N/A", 1275 | "type": 1, 1276 | "value": "null" 1277 | } 1278 | ], 1279 | "thresholds": { 1280 | "mode": "absolute", 1281 | "steps": [ 1282 | { 1283 | "color": "rgb(255, 255, 255)", 1284 | "value": null 1285 | } 1286 | ] 1287 | }, 1288 | "unit": "none" 1289 | }, 1290 | "overrides": [] 1291 | }, 1292 | "gridPos": { 1293 | "h": 2, 1294 | "w": 3, 1295 | "x": 12, 1296 | "y": 19 1297 | }, 1298 | "id": 115, 1299 | "interval": null, 1300 | "links": [], 1301 | "maxDataPoints": 100, 1302 | "options": { 1303 | "colorMode": "value", 1304 | "graphMode": "none", 1305 | "justifyMode": "auto", 1306 | "orientation": "horizontal", 1307 | "reduceOptions": { 1308 | "calcs": [ 1309 | "lastNotNull" 1310 | ], 1311 | "fields": "", 1312 | "values": false 1313 | }, 1314 | "text": {}, 1315 | "textMode": "auto" 1316 | }, 1317 | "pluginVersion": "7.4.0", 1318 | "targets": [ 1319 | { 1320 | "expr": "txpool_local{instance=~\"$instance\", job=~\"$job\"}", 1321 | "format": "time_series", 1322 | "interval": "", 1323 | "intervalFactor": 1, 1324 | "legendFormat": "", 1325 | "refId": "A" 1326 | } 1327 | ], 1328 | "timeFrom": null, 1329 | "timeShift": null, 1330 | "title": "Local transactions", 1331 | "type": "stat" 1332 | }, 1333 | { 1334 | "aliasColors": {}, 1335 | "bars": false, 1336 | "cacheTimeout": null, 1337 | "dashLength": 10, 1338 | "dashes": false, 1339 | "datasource": "${DS_PROMETHEUS}", 1340 | "fieldConfig": { 1341 | "defaults": { 1342 | "custom": {} 1343 | }, 1344 | "overrides": [] 1345 | }, 1346 | "fill": 1, 1347 | "fillGradient": 0, 1348 | "gridPos": { 1349 | "h": 9, 1350 | "w": 24, 1351 | "x": 0, 1352 | "y": 21 1353 | }, 1354 | "hiddenSeries": false, 1355 | "id": 112, 1356 | "legend": { 1357 | "alignAsTable": true, 1358 | "avg": false, 1359 | "current": true, 1360 | "max": true, 1361 | "min": true, 1362 | "rightSide": true, 1363 | "show": true, 1364 | "total": false, 1365 | "values": true 1366 | }, 1367 | "lines": true, 1368 | "linewidth": 1, 1369 | "links": [], 1370 | "nullPointMode": "null", 1371 | "options": { 1372 | "alertThreshold": true 1373 | }, 1374 | "percentage": false, 1375 | "pluginVersion": "7.4.0", 1376 | "pointradius": 2, 1377 | "points": false, 1378 | "renderer": "flot", 1379 | "seriesOverrides": [], 1380 | "spaceLength": 10, 1381 | "stack": true, 1382 | "steppedLine": false, 1383 | "targets": [ 1384 | { 1385 | "expr": "chain_execution{quantile=\"$quantile\", instance=~\"$instance\", job=~\"$job\"}", 1386 | "format": "time_series", 1387 | "interval": "", 1388 | "intervalFactor": 1, 1389 | "legendFormat": "execution (q=$quantile)", 1390 | "refId": "A" 1391 | }, 1392 | { 1393 | "expr": "chain_validation{quantile=\"$quantile\", instance=~\"$instance\", job=~\"$job\"}", 1394 | "format": "time_series", 1395 | "hide": false, 1396 | "interval": "", 1397 | "intervalFactor": 1, 1398 | "legendFormat": "validation (q=$quantile)", 1399 | "refId": "B" 1400 | }, 1401 | { 1402 | "expr": "chain_write{quantile=\"$quantile\", instance=~\"$instance\", job=~\"$job\"}", 1403 | "format": "time_series", 1404 | "hide": false, 1405 | "interval": "", 1406 | "intervalFactor": 1, 1407 | "legendFormat": "commit (q=$quantile)", 1408 | "refId": "C" 1409 | }, 1410 | { 1411 | "expr": "chain_account_reads{quantile=\"$quantile\", instance=~\"$instance\", job=~\"$job\"}", 1412 | "format": "time_series", 1413 | "interval": "", 1414 | "intervalFactor": 1, 1415 | "legendFormat": "account read (q=$quantile)", 1416 | "refId": "D" 1417 | }, 1418 | { 1419 | "expr": "chain_account_updates{quantile=\"$quantile\", instance=~\"$instance\", job=~\"$job\"}", 1420 | "format": "time_series", 1421 | "interval": "", 1422 | "intervalFactor": 1, 1423 | "legendFormat": "account update (q=$quantile)", 1424 | "refId": "E" 1425 | }, 1426 | { 1427 | "expr": "chain_account_hashes{quantile=\"$quantile\", instance=~\"$instance\", job=~\"$job\"}", 1428 | "format": "time_series", 1429 | "interval": "", 1430 | "intervalFactor": 1, 1431 | "legendFormat": "account hashe (q=$quantile)", 1432 | "refId": "F" 1433 | }, 1434 | { 1435 | "expr": "chain_account_commits{quantile=\"$quantile\", instance=~\"$instance\", job=~\"$job\"}", 1436 | "format": "time_series", 1437 | "interval": "", 1438 | "intervalFactor": 1, 1439 | "legendFormat": "account commit (q=$quantile)", 1440 | "refId": "G" 1441 | }, 1442 | { 1443 | "expr": "chain_storage_reads{quantile=\"$quantile\", instance=~\"$instance\", job=~\"$job\"}", 1444 | "format": "time_series", 1445 | "interval": "", 1446 | "intervalFactor": 1, 1447 | "legendFormat": "storage read (q=$quantile)", 1448 | "refId": "H" 1449 | }, 1450 | { 1451 | "expr": "chain_storage_updates{quantile=\"$quantile\", instance=~\"$instance\", job=~\"$job\"}", 1452 | "format": "time_series", 1453 | "interval": "", 1454 | "intervalFactor": 1, 1455 | "legendFormat": "storage update (q=$quantile)", 1456 | "refId": "I" 1457 | }, 1458 | { 1459 | "expr": "chain_storage_hashes{quantile=\"$quantile\", instance=~\"$instance\", job=~\"$job\"}", 1460 | "format": "time_series", 1461 | "interval": "", 1462 | "intervalFactor": 1, 1463 | "legendFormat": "storage hashe (q=$quantile)", 1464 | "refId": "J" 1465 | }, 1466 | { 1467 | "expr": "chain_storage_commits{quantile=\"$quantile\", instance=~\"$instance\", job=~\"$job\"}", 1468 | "format": "time_series", 1469 | "interval": "", 1470 | "intervalFactor": 1, 1471 | "legendFormat": "storage commit (q=$quantile)", 1472 | "refId": "K" 1473 | } 1474 | ], 1475 | "thresholds": [], 1476 | "timeFrom": null, 1477 | "timeRegions": [], 1478 | "timeShift": null, 1479 | "title": "Block processing", 1480 | "tooltip": { 1481 | "shared": true, 1482 | "sort": 0, 1483 | "value_type": "individual" 1484 | }, 1485 | "type": "graph", 1486 | "xaxis": { 1487 | "buckets": null, 1488 | "mode": "time", 1489 | "name": null, 1490 | "show": true, 1491 | "values": [] 1492 | }, 1493 | "yaxes": [ 1494 | { 1495 | "format": "ns", 1496 | "label": null, 1497 | "logBase": 1, 1498 | "max": null, 1499 | "min": null, 1500 | "show": true 1501 | }, 1502 | { 1503 | "format": "short", 1504 | "label": null, 1505 | "logBase": 1, 1506 | "max": null, 1507 | "min": null, 1508 | "show": true 1509 | } 1510 | ], 1511 | "yaxis": { 1512 | "align": false, 1513 | "alignLevel": null 1514 | } 1515 | }, 1516 | { 1517 | "aliasColors": {}, 1518 | "bars": false, 1519 | "cacheTimeout": null, 1520 | "dashLength": 10, 1521 | "dashes": false, 1522 | "datasource": "${DS_PROMETHEUS}", 1523 | "fieldConfig": { 1524 | "defaults": { 1525 | "custom": {} 1526 | }, 1527 | "overrides": [] 1528 | }, 1529 | "fill": 1, 1530 | "fillGradient": 0, 1531 | "gridPos": { 1532 | "h": 9, 1533 | "w": 24, 1534 | "x": 0, 1535 | "y": 30 1536 | }, 1537 | "hiddenSeries": false, 1538 | "id": 117, 1539 | "legend": { 1540 | "alignAsTable": true, 1541 | "avg": false, 1542 | "current": true, 1543 | "max": true, 1544 | "min": true, 1545 | "rightSide": true, 1546 | "show": true, 1547 | "total": false, 1548 | "values": true 1549 | }, 1550 | "lines": true, 1551 | "linewidth": 1, 1552 | "links": [], 1553 | "nullPointMode": "null", 1554 | "options": { 1555 | "alertThreshold": true 1556 | }, 1557 | "percentage": false, 1558 | "pluginVersion": "7.4.0", 1559 | "pointradius": 2, 1560 | "points": false, 1561 | "renderer": "flot", 1562 | "seriesOverrides": [], 1563 | "spaceLength": 10, 1564 | "stack": false, 1565 | "steppedLine": false, 1566 | "targets": [ 1567 | { 1568 | "expr": "rate(txpool_valid{instance=~\"$instance\", job=~\"$job\"}[1m])", 1569 | "format": "time_series", 1570 | "interval": "", 1571 | "intervalFactor": 1, 1572 | "legendFormat": "valid", 1573 | "refId": "K" 1574 | }, 1575 | { 1576 | "expr": "rate(txpool_invalid{instance=~\"$instance\", job=~\"$job\"}[1m])", 1577 | "format": "time_series", 1578 | "interval": "", 1579 | "intervalFactor": 1, 1580 | "legendFormat": "invalid", 1581 | "refId": "A" 1582 | }, 1583 | { 1584 | "expr": "rate(txpool_underpriced{instance=~\"$instance\", job=~\"$job\"}[1m])", 1585 | "format": "time_series", 1586 | "hide": false, 1587 | "interval": "", 1588 | "intervalFactor": 1, 1589 | "legendFormat": "underpriced", 1590 | "refId": "B" 1591 | }, 1592 | { 1593 | "expr": "rate(txpool_pending_discard{instance=~\"$instance\", job=~\"$job\"}[1m])", 1594 | "format": "time_series", 1595 | "hide": false, 1596 | "interval": "", 1597 | "intervalFactor": 1, 1598 | "legendFormat": "executable discard", 1599 | "refId": "C" 1600 | }, 1601 | { 1602 | "expr": "rate(txpool_pending_replace{instance=~\"$instance\", job=~\"$job\"}[1m])", 1603 | "format": "time_series", 1604 | "interval": "", 1605 | "intervalFactor": 1, 1606 | "legendFormat": "executable replace", 1607 | "refId": "D" 1608 | }, 1609 | { 1610 | "expr": "rate(txpool_pending_ratelimit{instance=~\"$instance\", job=~\"$job\"}[1m])", 1611 | "format": "time_series", 1612 | "interval": "", 1613 | "intervalFactor": 1, 1614 | "legendFormat": "executable ratelimit", 1615 | "refId": "E" 1616 | }, 1617 | { 1618 | "expr": "rate(txpool_pending_nofunds{instance=~\"$instance\", job=~\"$job\"}[1m])", 1619 | "format": "time_series", 1620 | "interval": "", 1621 | "intervalFactor": 1, 1622 | "legendFormat": "executable nofunds", 1623 | "refId": "F" 1624 | }, 1625 | { 1626 | "expr": "rate(txpool_queued_discard{instance=~\"$instance\", job=~\"$job\"}[1m])", 1627 | "format": "time_series", 1628 | "hide": false, 1629 | "interval": "", 1630 | "intervalFactor": 1, 1631 | "legendFormat": "gapped discard", 1632 | "refId": "G" 1633 | }, 1634 | { 1635 | "expr": "rate(txpool_queued_replace{instance=~\"$instance\", job=~\"$job\"}[1m])", 1636 | "format": "time_series", 1637 | "hide": false, 1638 | "interval": "", 1639 | "intervalFactor": 1, 1640 | "legendFormat": "gapped replace", 1641 | "refId": "H" 1642 | }, 1643 | { 1644 | "expr": "rate(txpool_queued_ratelimit{instance=~\"$instance\", job=~\"$job\"}[1m])", 1645 | "format": "time_series", 1646 | "hide": false, 1647 | "interval": "", 1648 | "intervalFactor": 1, 1649 | "legendFormat": "gapped ratelimit", 1650 | "refId": "I" 1651 | }, 1652 | { 1653 | "expr": "rate(txpool_queued_nofunds{instance=~\"$instance\", job=~\"$job\"}[1m])", 1654 | "format": "time_series", 1655 | "hide": false, 1656 | "interval": "", 1657 | "intervalFactor": 1, 1658 | "legendFormat": "gapped nofunds", 1659 | "refId": "J" 1660 | } 1661 | ], 1662 | "thresholds": [], 1663 | "timeFrom": null, 1664 | "timeRegions": [], 1665 | "timeShift": null, 1666 | "title": "Transaction propagation", 1667 | "tooltip": { 1668 | "shared": true, 1669 | "sort": 0, 1670 | "value_type": "individual" 1671 | }, 1672 | "type": "graph", 1673 | "xaxis": { 1674 | "buckets": null, 1675 | "mode": "time", 1676 | "name": null, 1677 | "show": true, 1678 | "values": [] 1679 | }, 1680 | "yaxes": [ 1681 | { 1682 | "format": "none", 1683 | "label": null, 1684 | "logBase": 1, 1685 | "max": null, 1686 | "min": null, 1687 | "show": true 1688 | }, 1689 | { 1690 | "format": "short", 1691 | "label": null, 1692 | "logBase": 1, 1693 | "max": null, 1694 | "min": null, 1695 | "show": true 1696 | } 1697 | ], 1698 | "yaxis": { 1699 | "align": false, 1700 | "alignLevel": null 1701 | } 1702 | }, 1703 | { 1704 | "collapsed": false, 1705 | "datasource": "${DS_PROMETHEUS}", 1706 | "gridPos": { 1707 | "h": 1, 1708 | "w": 24, 1709 | "x": 0, 1710 | "y": 39 1711 | }, 1712 | "id": 17, 1713 | "panels": [], 1714 | "title": "Database", 1715 | "type": "row" 1716 | }, 1717 | { 1718 | "aliasColors": {}, 1719 | "bars": false, 1720 | "dashLength": 10, 1721 | "dashes": false, 1722 | "datasource": "${DS_PROMETHEUS}", 1723 | "fieldConfig": { 1724 | "defaults": { 1725 | "custom": {} 1726 | }, 1727 | "overrides": [] 1728 | }, 1729 | "fill": 1, 1730 | "fillGradient": 0, 1731 | "gridPos": { 1732 | "h": 6, 1733 | "w": 8, 1734 | "x": 0, 1735 | "y": 40 1736 | }, 1737 | "hiddenSeries": false, 1738 | "id": 35, 1739 | "legend": { 1740 | "avg": false, 1741 | "current": false, 1742 | "max": false, 1743 | "min": false, 1744 | "show": true, 1745 | "total": false, 1746 | "values": false 1747 | }, 1748 | "lines": true, 1749 | "linewidth": 1, 1750 | "links": [], 1751 | "nullPointMode": "null", 1752 | "options": { 1753 | "alertThreshold": true 1754 | }, 1755 | "percentage": false, 1756 | "pluginVersion": "7.4.0", 1757 | "pointradius": 5, 1758 | "points": false, 1759 | "renderer": "flot", 1760 | "seriesOverrides": [], 1761 | "spaceLength": 10, 1762 | "stack": false, 1763 | "steppedLine": false, 1764 | "targets": [ 1765 | { 1766 | "expr": "rate(eth_db_chaindata_disk_read{instance=~\"$instance\", job=~\"$job\"}[1m])", 1767 | "format": "time_series", 1768 | "interval": "", 1769 | "intervalFactor": 1, 1770 | "legendFormat": "leveldb read", 1771 | "refId": "B" 1772 | }, 1773 | { 1774 | "expr": "rate(eth_db_chaindata_disk_write{instance=~\"$instance\", job=~\"$job\"}[1m])", 1775 | "format": "time_series", 1776 | "interval": "", 1777 | "intervalFactor": 1, 1778 | "legendFormat": "leveldb write", 1779 | "refId": "A" 1780 | }, 1781 | { 1782 | "expr": "rate(eth_db_chaindata_ancient_read{instance=~\"$instance\", job=~\"$job\"}[1m])", 1783 | "format": "time_series", 1784 | "interval": "", 1785 | "intervalFactor": 1, 1786 | "legendFormat": "ancient read", 1787 | "refId": "C" 1788 | }, 1789 | { 1790 | "expr": "rate(eth_db_chaindata_ancient_write{instance=~\"$instance\", job=~\"$job\"}[1m])", 1791 | "format": "time_series", 1792 | "interval": "", 1793 | "intervalFactor": 1, 1794 | "legendFormat": "ancient write", 1795 | "refId": "D" 1796 | } 1797 | ], 1798 | "thresholds": [], 1799 | "timeFrom": null, 1800 | "timeRegions": [], 1801 | "timeShift": null, 1802 | "title": "Data rate", 1803 | "tooltip": { 1804 | "shared": true, 1805 | "sort": 0, 1806 | "value_type": "individual" 1807 | }, 1808 | "type": "graph", 1809 | "xaxis": { 1810 | "buckets": null, 1811 | "mode": "time", 1812 | "name": null, 1813 | "show": true, 1814 | "values": [] 1815 | }, 1816 | "yaxes": [ 1817 | { 1818 | "format": "Bps", 1819 | "label": null, 1820 | "logBase": 1, 1821 | "max": null, 1822 | "min": null, 1823 | "show": true 1824 | }, 1825 | { 1826 | "format": "short", 1827 | "label": null, 1828 | "logBase": 1, 1829 | "max": null, 1830 | "min": null, 1831 | "show": true 1832 | } 1833 | ], 1834 | "yaxis": { 1835 | "align": false, 1836 | "alignLevel": null 1837 | } 1838 | }, 1839 | { 1840 | "aliasColors": {}, 1841 | "bars": false, 1842 | "dashLength": 10, 1843 | "dashes": false, 1844 | "datasource": "${DS_PROMETHEUS}", 1845 | "fieldConfig": { 1846 | "defaults": { 1847 | "custom": {} 1848 | }, 1849 | "overrides": [] 1850 | }, 1851 | "fill": 1, 1852 | "fillGradient": 0, 1853 | "gridPos": { 1854 | "h": 6, 1855 | "w": 8, 1856 | "x": 8, 1857 | "y": 40 1858 | }, 1859 | "hiddenSeries": false, 1860 | "id": 118, 1861 | "legend": { 1862 | "avg": false, 1863 | "current": false, 1864 | "max": false, 1865 | "min": false, 1866 | "show": true, 1867 | "total": false, 1868 | "values": false 1869 | }, 1870 | "lines": true, 1871 | "linewidth": 1, 1872 | "links": [], 1873 | "nullPointMode": "null", 1874 | "options": { 1875 | "alertThreshold": true 1876 | }, 1877 | "percentage": false, 1878 | "pluginVersion": "7.4.0", 1879 | "pointradius": 5, 1880 | "points": false, 1881 | "renderer": "flot", 1882 | "seriesOverrides": [], 1883 | "spaceLength": 10, 1884 | "stack": false, 1885 | "steppedLine": false, 1886 | "targets": [ 1887 | { 1888 | "expr": "eth_db_chaindata_disk_read{instance=~\"$instance\", job=~\"$job\"}", 1889 | "format": "time_series", 1890 | "interval": "", 1891 | "intervalFactor": 1, 1892 | "legendFormat": "leveldb read", 1893 | "refId": "B" 1894 | }, 1895 | { 1896 | "expr": "eth_db_chaindata_disk_write{instance=~\"$instance\", job=~\"$job\"}", 1897 | "format": "time_series", 1898 | "interval": "", 1899 | "intervalFactor": 1, 1900 | "legendFormat": "leveldb write", 1901 | "refId": "A" 1902 | }, 1903 | { 1904 | "expr": "eth_db_chaindata_ancient_read{instance=~\"$instance\", job=~\"$job\"}", 1905 | "format": "time_series", 1906 | "interval": "", 1907 | "intervalFactor": 1, 1908 | "legendFormat": "ancient read", 1909 | "refId": "C" 1910 | }, 1911 | { 1912 | "expr": "eth_db_chaindata_ancient_write{instance=~\"$instance\", job=~\"$job\"}", 1913 | "format": "time_series", 1914 | "interval": "", 1915 | "intervalFactor": 1, 1916 | "legendFormat": "ancient write", 1917 | "refId": "D" 1918 | } 1919 | ], 1920 | "thresholds": [], 1921 | "timeFrom": null, 1922 | "timeRegions": [], 1923 | "timeShift": null, 1924 | "title": "Session totals", 1925 | "tooltip": { 1926 | "shared": true, 1927 | "sort": 0, 1928 | "value_type": "individual" 1929 | }, 1930 | "type": "graph", 1931 | "xaxis": { 1932 | "buckets": null, 1933 | "mode": "time", 1934 | "name": null, 1935 | "show": true, 1936 | "values": [] 1937 | }, 1938 | "yaxes": [ 1939 | { 1940 | "format": "decbytes", 1941 | "label": null, 1942 | "logBase": 1, 1943 | "max": null, 1944 | "min": null, 1945 | "show": true 1946 | }, 1947 | { 1948 | "format": "short", 1949 | "label": null, 1950 | "logBase": 1, 1951 | "max": null, 1952 | "min": null, 1953 | "show": true 1954 | } 1955 | ], 1956 | "yaxis": { 1957 | "align": false, 1958 | "alignLevel": null 1959 | } 1960 | }, 1961 | { 1962 | "aliasColors": {}, 1963 | "bars": false, 1964 | "dashLength": 10, 1965 | "dashes": false, 1966 | "datasource": "${DS_PROMETHEUS}", 1967 | "fieldConfig": { 1968 | "defaults": { 1969 | "custom": {} 1970 | }, 1971 | "overrides": [] 1972 | }, 1973 | "fill": 1, 1974 | "fillGradient": 0, 1975 | "gridPos": { 1976 | "h": 6, 1977 | "w": 8, 1978 | "x": 16, 1979 | "y": 40 1980 | }, 1981 | "hiddenSeries": false, 1982 | "id": 119, 1983 | "legend": { 1984 | "alignAsTable": false, 1985 | "avg": false, 1986 | "current": false, 1987 | "max": false, 1988 | "min": false, 1989 | "rightSide": false, 1990 | "show": true, 1991 | "total": false, 1992 | "values": false 1993 | }, 1994 | "lines": true, 1995 | "linewidth": 1, 1996 | "links": [], 1997 | "nullPointMode": "null", 1998 | "options": { 1999 | "alertThreshold": true 2000 | }, 2001 | "percentage": false, 2002 | "pluginVersion": "7.4.0", 2003 | "pointradius": 5, 2004 | "points": false, 2005 | "renderer": "flot", 2006 | "seriesOverrides": [], 2007 | "spaceLength": 10, 2008 | "stack": false, 2009 | "steppedLine": false, 2010 | "targets": [ 2011 | { 2012 | "expr": "eth_db_chaindata_disk_size{instance=~\"$instance\", job=~\"$job\"}", 2013 | "format": "time_series", 2014 | "interval": "", 2015 | "intervalFactor": 1, 2016 | "legendFormat": "leveldb", 2017 | "refId": "B" 2018 | }, 2019 | { 2020 | "expr": "eth_db_chaindata_ancient_size{instance=~\"$instance\", job=~\"$job\"}", 2021 | "format": "time_series", 2022 | "interval": "", 2023 | "intervalFactor": 1, 2024 | "legendFormat": "ancient", 2025 | "refId": "A" 2026 | } 2027 | ], 2028 | "thresholds": [], 2029 | "timeFrom": null, 2030 | "timeRegions": [], 2031 | "timeShift": null, 2032 | "title": "Storage size", 2033 | "tooltip": { 2034 | "shared": true, 2035 | "sort": 0, 2036 | "value_type": "individual" 2037 | }, 2038 | "type": "graph", 2039 | "xaxis": { 2040 | "buckets": null, 2041 | "mode": "time", 2042 | "name": null, 2043 | "show": true, 2044 | "values": [] 2045 | }, 2046 | "yaxes": [ 2047 | { 2048 | "format": "decbytes", 2049 | "label": null, 2050 | "logBase": 1, 2051 | "max": null, 2052 | "min": null, 2053 | "show": true 2054 | }, 2055 | { 2056 | "format": "short", 2057 | "label": null, 2058 | "logBase": 1, 2059 | "max": null, 2060 | "min": null, 2061 | "show": true 2062 | } 2063 | ], 2064 | "yaxis": { 2065 | "align": false, 2066 | "alignLevel": null 2067 | } 2068 | }, 2069 | { 2070 | "collapsed": false, 2071 | "datasource": "${DS_PROMETHEUS}", 2072 | "gridPos": { 2073 | "h": 1, 2074 | "w": 24, 2075 | "x": 0, 2076 | "y": 46 2077 | }, 2078 | "id": 37, 2079 | "panels": [], 2080 | "title": "Trie Stats", 2081 | "type": "row" 2082 | }, 2083 | { 2084 | "aliasColors": {}, 2085 | "bars": false, 2086 | "dashLength": 10, 2087 | "dashes": false, 2088 | "datasource": "${DS_PROMETHEUS}", 2089 | "fieldConfig": { 2090 | "defaults": { 2091 | "custom": {} 2092 | }, 2093 | "overrides": [] 2094 | }, 2095 | "fill": 1, 2096 | "fillGradient": 0, 2097 | "gridPos": { 2098 | "h": 6, 2099 | "w": 12, 2100 | "x": 0, 2101 | "y": 47 2102 | }, 2103 | "hiddenSeries": false, 2104 | "id": 120, 2105 | "legend": { 2106 | "avg": false, 2107 | "current": false, 2108 | "max": false, 2109 | "min": false, 2110 | "show": true, 2111 | "total": false, 2112 | "values": false 2113 | }, 2114 | "lines": true, 2115 | "linewidth": 1, 2116 | "links": [], 2117 | "nullPointMode": "null", 2118 | "options": { 2119 | "alertThreshold": true 2120 | }, 2121 | "percentage": false, 2122 | "pluginVersion": "7.4.0", 2123 | "pointradius": 5, 2124 | "points": false, 2125 | "renderer": "flot", 2126 | "seriesOverrides": [], 2127 | "spaceLength": 10, 2128 | "stack": false, 2129 | "steppedLine": false, 2130 | "targets": [ 2131 | { 2132 | "expr": "rate(trie_memcache_clean_read{instance=~\"$instance\", job=~\"$job\"}[1m])", 2133 | "format": "time_series", 2134 | "hide": false, 2135 | "interval": "", 2136 | "intervalFactor": 1, 2137 | "legendFormat": "hit", 2138 | "refId": "C" 2139 | }, 2140 | { 2141 | "expr": "rate(trie_memcache_clean_write{instance=~\"$instance\", job=~\"$job\"}[1m])", 2142 | "format": "time_series", 2143 | "hide": false, 2144 | "interval": "", 2145 | "intervalFactor": 1, 2146 | "legendFormat": "miss", 2147 | "refId": "B" 2148 | } 2149 | ], 2150 | "thresholds": [], 2151 | "timeFrom": null, 2152 | "timeRegions": [], 2153 | "timeShift": null, 2154 | "title": "Clean cache", 2155 | "tooltip": { 2156 | "shared": true, 2157 | "sort": 0, 2158 | "value_type": "individual" 2159 | }, 2160 | "type": "graph", 2161 | "xaxis": { 2162 | "buckets": null, 2163 | "mode": "time", 2164 | "name": null, 2165 | "show": true, 2166 | "values": [] 2167 | }, 2168 | "yaxes": [ 2169 | { 2170 | "format": "Bps", 2171 | "label": null, 2172 | "logBase": 1, 2173 | "max": null, 2174 | "min": null, 2175 | "show": true 2176 | }, 2177 | { 2178 | "format": "short", 2179 | "label": null, 2180 | "logBase": 1, 2181 | "max": null, 2182 | "min": null, 2183 | "show": true 2184 | } 2185 | ], 2186 | "yaxis": { 2187 | "align": false, 2188 | "alignLevel": null 2189 | } 2190 | }, 2191 | { 2192 | "aliasColors": {}, 2193 | "bars": false, 2194 | "dashLength": 10, 2195 | "dashes": false, 2196 | "datasource": "${DS_PROMETHEUS}", 2197 | "fieldConfig": { 2198 | "defaults": { 2199 | "custom": {} 2200 | }, 2201 | "overrides": [] 2202 | }, 2203 | "fill": 1, 2204 | "fillGradient": 0, 2205 | "gridPos": { 2206 | "h": 6, 2207 | "w": 12, 2208 | "x": 12, 2209 | "y": 47 2210 | }, 2211 | "hiddenSeries": false, 2212 | "id": 56, 2213 | "legend": { 2214 | "avg": false, 2215 | "current": false, 2216 | "max": false, 2217 | "min": false, 2218 | "show": true, 2219 | "total": false, 2220 | "values": false 2221 | }, 2222 | "lines": true, 2223 | "linewidth": 1, 2224 | "links": [], 2225 | "nullPointMode": "null", 2226 | "options": { 2227 | "alertThreshold": true 2228 | }, 2229 | "percentage": false, 2230 | "pluginVersion": "7.4.0", 2231 | "pointradius": 5, 2232 | "points": false, 2233 | "renderer": "flot", 2234 | "seriesOverrides": [], 2235 | "spaceLength": 10, 2236 | "stack": false, 2237 | "steppedLine": false, 2238 | "targets": [ 2239 | { 2240 | "expr": "rate(trie_memcache_gc_size{instance=~\"$instance\", job=~\"$job\"}[1m])", 2241 | "format": "time_series", 2242 | "hide": false, 2243 | "interval": "", 2244 | "intervalFactor": 1, 2245 | "legendFormat": "gc", 2246 | "refId": "C" 2247 | }, 2248 | { 2249 | "expr": "rate(trie_memcache_flush_size{instance=~\"$instance\", job=~\"$job\"}[1m])", 2250 | "format": "time_series", 2251 | "hide": false, 2252 | "interval": "", 2253 | "intervalFactor": 1, 2254 | "legendFormat": "overflow", 2255 | "refId": "B" 2256 | }, 2257 | { 2258 | "expr": "rate(trie_memcache_commit_size{instance=~\"$instance\", job=~\"$job\"}[1m])", 2259 | "format": "time_series", 2260 | "interval": "", 2261 | "intervalFactor": 1, 2262 | "legendFormat": "commit", 2263 | "refId": "A" 2264 | } 2265 | ], 2266 | "thresholds": [], 2267 | "timeFrom": null, 2268 | "timeRegions": [], 2269 | "timeShift": null, 2270 | "title": "Dirty cache", 2271 | "tooltip": { 2272 | "shared": true, 2273 | "sort": 0, 2274 | "value_type": "individual" 2275 | }, 2276 | "type": "graph", 2277 | "xaxis": { 2278 | "buckets": null, 2279 | "mode": "time", 2280 | "name": null, 2281 | "show": true, 2282 | "values": [] 2283 | }, 2284 | "yaxes": [ 2285 | { 2286 | "format": "Bps", 2287 | "label": null, 2288 | "logBase": 1, 2289 | "max": null, 2290 | "min": null, 2291 | "show": true 2292 | }, 2293 | { 2294 | "format": "short", 2295 | "label": null, 2296 | "logBase": 1, 2297 | "max": null, 2298 | "min": null, 2299 | "show": true 2300 | } 2301 | ], 2302 | "yaxis": { 2303 | "align": false, 2304 | "alignLevel": null 2305 | } 2306 | } 2307 | ], 2308 | "refresh": "10s", 2309 | "schemaVersion": 27, 2310 | "style": "dark", 2311 | "tags": [ 2312 | "eth1.0", 2313 | "ethereum", 2314 | "geth" 2315 | ], 2316 | "templating": { 2317 | "list": [ 2318 | { 2319 | "current": { 2320 | "selected": false, 2321 | "text": "Prometheus", 2322 | "value": "Prometheus" 2323 | }, 2324 | "description": null, 2325 | "error": null, 2326 | "hide": 0, 2327 | "includeAll": false, 2328 | "label": "datasource", 2329 | "multi": false, 2330 | "name": "datasource", 2331 | "options": [], 2332 | "query": "prometheus", 2333 | "queryValue": "", 2334 | "refresh": 1, 2335 | "regex": "", 2336 | "skipUrlSync": false, 2337 | "type": "datasource" 2338 | }, 2339 | { 2340 | "description": null, 2341 | "error": null, 2342 | "hide": 2, 2343 | "label": "job", 2344 | "name": "job", 2345 | "query": "${VAR_JOB}", 2346 | "skipUrlSync": false, 2347 | "type": "constant", 2348 | "current": { 2349 | "value": "${VAR_JOB}", 2350 | "text": "${VAR_JOB}", 2351 | "selected": false 2352 | }, 2353 | "options": [ 2354 | { 2355 | "value": "${VAR_JOB}", 2356 | "text": "${VAR_JOB}", 2357 | "selected": false 2358 | } 2359 | ] 2360 | }, 2361 | { 2362 | "allValue": null, 2363 | "current": {}, 2364 | "datasource": "${DS_PROMETHEUS}", 2365 | "definition": "label_values(chain_head_block{}, instance)", 2366 | "description": null, 2367 | "error": null, 2368 | "hide": 0, 2369 | "includeAll": true, 2370 | "label": "instance", 2371 | "multi": false, 2372 | "name": "instance", 2373 | "options": [], 2374 | "query": { 2375 | "query": "label_values(chain_head_block{}, instance)", 2376 | "refId": "StandardVariableQuery" 2377 | }, 2378 | "refresh": 1, 2379 | "regex": "", 2380 | "skipUrlSync": false, 2381 | "sort": 0, 2382 | "tagValuesQuery": "", 2383 | "tags": [], 2384 | "tagsQuery": "", 2385 | "type": "query", 2386 | "useTags": false 2387 | }, 2388 | { 2389 | "allValue": null, 2390 | "current": { 2391 | "selected": false, 2392 | "text": "0.5", 2393 | "value": "0.5" 2394 | }, 2395 | "description": null, 2396 | "error": null, 2397 | "hide": 0, 2398 | "includeAll": false, 2399 | "label": null, 2400 | "multi": false, 2401 | "name": "quantile", 2402 | "options": [ 2403 | { 2404 | "selected": true, 2405 | "text": "0.5", 2406 | "value": "0.5" 2407 | }, 2408 | { 2409 | "selected": false, 2410 | "text": "0.75", 2411 | "value": "0.75" 2412 | }, 2413 | { 2414 | "selected": false, 2415 | "text": "0.95", 2416 | "value": "0.95" 2417 | }, 2418 | { 2419 | "selected": false, 2420 | "text": "0.99", 2421 | "value": "0.99" 2422 | }, 2423 | { 2424 | "selected": false, 2425 | "text": "0.999", 2426 | "value": "0.999" 2427 | }, 2428 | { 2429 | "selected": false, 2430 | "text": "0.9999", 2431 | "value": "0.9999" 2432 | } 2433 | ], 2434 | "query": "0.5, 0.75, 0.95, 0.99, 0.999, 0.9999", 2435 | "queryValue": "", 2436 | "skipUrlSync": false, 2437 | "type": "custom" 2438 | } 2439 | ] 2440 | }, 2441 | "time": { 2442 | "from": "now-3h", 2443 | "to": "now" 2444 | }, 2445 | "timepicker": { 2446 | "refresh_intervals": [ 2447 | "5s", 2448 | "10s", 2449 | "30s", 2450 | "1m", 2451 | "5m", 2452 | "15m", 2453 | "30m", 2454 | "1h", 2455 | "2h", 2456 | "1d" 2457 | ], 2458 | "time_options": [ 2459 | "5m", 2460 | "15m", 2461 | "1h", 2462 | "6h", 2463 | "12h", 2464 | "24h", 2465 | "2d", 2466 | "7d", 2467 | "30d" 2468 | ] 2469 | }, 2470 | "timezone": "", 2471 | "title": "Geth Transaction Processing", 2472 | "uid": "FPpjH6Hik", 2473 | "version": 17, 2474 | "description": "Geth metrics overview dashboard." 2475 | } -------------------------------------------------------------------------------- /images/ethereum_staking_eks.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-eks-ethereum-staking/d51b332e5ffa15a42c3e3949aca9b35621fe20ef/images/ethereum_staking_eks.jpg -------------------------------------------------------------------------------- /lib/cloudwatch-widgets.json: -------------------------------------------------------------------------------- 1 | { 2 | "widgets": [ 3 | { 4 | "height": 4, 5 | "width": 6, 6 | "y": 0, 7 | "x": 0, 8 | "type": "metric", 9 | "properties": { 10 | "view": "timeSeries", 11 | "stat": "Average", 12 | "period": 300, 13 | "stacked": false, 14 | "yAxis": { 15 | "left": { 16 | "min": 0 17 | } 18 | }, 19 | "region": "us-east-2", 20 | "metrics": [ 21 | [ 22 | "AWS/EC2", 23 | "CPUUtilization", 24 | "InstanceId", 25 | "i-02cfe6aa7d59d7d45", 26 | { "label": "i-02cfe6aa7d59d7d45" } 27 | ] 28 | ], 29 | "title": "CPU utilization (%)" 30 | } 31 | }, 32 | { 33 | "height": 4, 34 | "width": 6, 35 | "y": 4, 36 | "x": 6, 37 | "type": "metric", 38 | "properties": { 39 | "view": "timeSeries", 40 | "stat": "Average", 41 | "period": 300, 42 | "stacked": false, 43 | "yAxis": { 44 | "left": { 45 | "min": 0 46 | } 47 | }, 48 | "region": "us-east-2", 49 | "metrics": [ 50 | [ 51 | "AWS/EC2", 52 | "NetworkIn", 53 | "InstanceId", 54 | "i-02cfe6aa7d59d7d45", 55 | { "label": "i-02cfe6aa7d59d7d45" } 56 | ] 57 | ], 58 | "title": "Network in (bytes)" 59 | } 60 | }, 61 | { 62 | "height": 4, 63 | "width": 6, 64 | "y": 4, 65 | "x": 0, 66 | "type": "metric", 67 | "properties": { 68 | "view": "timeSeries", 69 | "stat": "Average", 70 | "period": 300, 71 | "stacked": false, 72 | "yAxis": { 73 | "left": { 74 | "min": 0 75 | } 76 | }, 77 | "region": "us-east-2", 78 | "metrics": [ 79 | [ 80 | "AWS/EC2", 81 | "NetworkOut", 82 | "InstanceId", 83 | "i-02cfe6aa7d59d7d45", 84 | { "label": "i-02cfe6aa7d59d7d45" } 85 | ] 86 | ], 87 | "title": "Network out (bytes)" 88 | } 89 | }, 90 | { 91 | "height": 4, 92 | "width": 6, 93 | "y": 0, 94 | "x": 6, 95 | "type": "metric", 96 | "properties": { 97 | "metrics": [ 98 | ["CWAgent", "mem_used_percent", "InstanceId", "i-02cfe6aa7d59d7d45"] 99 | ], 100 | "view": "timeSeries", 101 | "stacked": false, 102 | "region": "us-east-2", 103 | "stat": "Average", 104 | "period": 300, 105 | "title": "mem_used_percent" 106 | } 107 | }, 108 | { 109 | "height": 4, 110 | "width": 6, 111 | "y": 0, 112 | "x": 17, 113 | "type": "metric", 114 | "properties": { 115 | "metrics": [ 116 | [ 117 | { 118 | "expression": "m1/PERIOD(m1)", 119 | "label": "diskio_writes/sec", 120 | "id": "e1", 121 | "period": 60, 122 | "region": "us-east-2" 123 | } 124 | ], 125 | [ 126 | "CWAgent", 127 | "diskio_writes", 128 | "InstanceId", 129 | "i-02cfe6aa7d59d7d45", 130 | "name", 131 | "nvme0n1", 132 | "ImageId", 133 | "ami-0f91cedb707b09db0", 134 | "InstanceType", 135 | "m6g.xlarge", 136 | { "id": "m1", "stat": "Sum", "period": 60, "visible": false } 137 | ] 138 | ], 139 | "view": "timeSeries", 140 | "stacked": false, 141 | "region": "us-east-2", 142 | "stat": "Average", 143 | "period": 300, 144 | "title": "diskio_writes/s (nvme0n1)" 145 | } 146 | }, 147 | { 148 | "height": 4, 149 | "width": 5, 150 | "y": 0, 151 | "x": 12, 152 | "type": "metric", 153 | "properties": { 154 | "metrics": [ 155 | [ 156 | { 157 | "expression": "m1/PERIOD(m1)", 158 | "label": "diskio_reads/sec", 159 | "id": "e1", 160 | "period": 60, 161 | "region": "us-east-2" 162 | } 163 | ], 164 | [ 165 | "CWAgent", 166 | "diskio_reads", 167 | "InstanceId", 168 | "i-02cfe6aa7d59d7d45", 169 | "name", 170 | "nvme0n1", 171 | "ImageId", 172 | "ami-0f91cedb707b09db0", 173 | "InstanceType", 174 | "m6g.xlarge", 175 | { "id": "m1", "stat": "Sum", "visible": false, "period": 60 } 176 | ] 177 | ], 178 | "view": "timeSeries", 179 | "stacked": false, 180 | "region": "us-east-2", 181 | "stat": "Average", 182 | "period": 300, 183 | "title": "diskio_reads/s (nvme0n1)" 184 | } 185 | } 186 | ] 187 | } 188 | -------------------------------------------------------------------------------- /lib/custom-resources/eksloggingOnEvent.py: -------------------------------------------------------------------------------- 1 | import time 2 | import boto3,botocore 3 | import json 4 | import logging 5 | 6 | import botocore 7 | boto3.set_stream_logger('botocore.waiter', logging.DEBUG) 8 | 9 | client = boto3.client('eks') 10 | 11 | ALL_LOGGING_OPTS = [ 'api','audit','authenticator','controllerManager','scheduler'] 12 | 13 | def on_event(event, context): 14 | print(json.dumps(event)) 15 | request_type = event['RequestType'] 16 | if request_type == 'Create': return on_create(event) 17 | if request_type == 'Update': return on_update(event) 18 | if request_type == 'Delete': return on_delete(event) 19 | raise Exception("Invalid request type: %s" % request_type) 20 | 21 | def on_create(event): 22 | props = event["ResourceProperties"] 23 | print("Update %s with logging configuration %s" % (props['eksCluster'], props['loggingOpts'])) 24 | try: 25 | currentLoggingOpts = { 26 | 'enabled': [], 27 | 'disabled': [] 28 | } 29 | eks_info = client.describe_cluster( 30 | name=props['eksCluster'] 31 | ) 32 | for item in eks_info['cluster']['logging']['clusterLogging']: 33 | if (item['enabled']== True): 34 | currentLoggingOpts['enabled'] = item 35 | elif (item['enabled'] == False): 36 | currentLoggingOpts['disabled'] = item 37 | else: 38 | print("Unable to parse current eks logging configuration %s" % eks_info['cluster']['logging']['clusterLogging']) 39 | raise ValueError() 40 | updateLogOpts=log_config_diff(currentLoggingOpts,props['loggingOpts']) 41 | print(updateLogOpts) 42 | eks=client.update_cluster_config( 43 | name= props['eksCluster'], 44 | logging={ 45 | 'clusterLogging': updateLogOpts 46 | }) 47 | print("%s" % eks) 48 | except client.exceptions.InvalidParameterException as i: 49 | if ('No changes needed for the logging config provided' == i.response['Error']['Message']): 50 | print('No changes needed for the logging config provided, current config is valid, skipping action returning success') 51 | pass 52 | except Exception as e: 53 | print(e) 54 | raise 55 | return {'Status': 'SUCCESS', 'PhysicalResourceId': props['eksCluster'], 'StackId': event['StackId'], 'LogicalResourceId': event['LogicalResourceId']} 56 | 57 | def on_update(event): 58 | physical_id = event["PhysicalResourceId"] 59 | props = event["ResourceProperties"] 60 | print("update resource %s with props %s" % (physical_id, props)) 61 | on_create(event) 62 | 63 | def on_delete(event): 64 | physical_id = event["PhysicalResourceId"] 65 | print("Skip delete since eks cluster will be deleted %s" % physical_id) 66 | return {'Status': 'SUCCESS', 'PhysicalResourceId': physical_id, 'StackId': event['StackId'], 'LogicalResourceId': event['LogicalResourceId']} 67 | 68 | 69 | 70 | def is_complete(event,context): 71 | print(event) 72 | props = event["ResourceProperties"] 73 | print("Starting Waiter") 74 | # Sleep for 30 seconds before checking for cluster status to change 75 | time.sleep(30) 76 | print("Waited 30 seconds") 77 | try: 78 | waiter = client.get_waiter('cluster_active') 79 | waiter.wait( 80 | name=props['eksCluster'], 81 | WaiterConfig={ 82 | 'Delay': 10, 83 | 'MaxAttempts' : 40 84 | } 85 | ) 86 | eks_info = client.describe_cluster( 87 | name=props['eksCluster'] 88 | ) 89 | print(eks_info['cluster']['logging']) 90 | print("Completed wait") 91 | except botocore.exceptions.WaiterError as e: 92 | print(e) 93 | if "Max attempts exceeded" in e.message: 94 | return {'IsComplete': False} 95 | except Exception as e: 96 | print(e) 97 | raise 98 | 99 | return { 'IsComplete': True } 100 | 101 | 102 | def log_config_diff(currentOpts,newOpts): 103 | print ("Evaluate for updated logging to be enabled") 104 | if "" in newOpts: 105 | newOpts.remove("") 106 | if 'types' not in currentOpts['enabled']: 107 | currentOpts['enabled'] = {'types': []} 108 | 109 | enabledOpts = set(newOpts) - set(currentOpts['enabled']['types']) 110 | print("Need \"%s\" logging to be enabled" % list(enabledOpts)) 111 | disabledOpts = set(ALL_LOGGING_OPTS) - set(newOpts) 112 | print ("Need \"%s\" logging to be disabled" % list(disabledOpts)) 113 | return [{ 114 | 'types': list(enabledOpts), 115 | 'enabled': True 116 | }, 117 | { 118 | 'types': list(disabledOpts), 119 | 'enabled': False 120 | }] 121 | 122 | if __name__ == "__main__": 123 | jsonObj = """{ 124 | "RequestType": "Create", 125 | "ServiceToken": "EXAMPLE", 126 | "ResponseURL": "EXAMPLE", 127 | "StackId": "arn:aws:cloudformation:us-west-2:ACCOUNT_ID:stack/EKS_STACK/EXAMPLE", 128 | "RequestId": "EXAMPLE", 129 | "LogicalResourceId": "eksLoggingCustomResource", 130 | "ResourceType": "AWS::CloudFormation::CustomResource", 131 | "ResourceProperties": { 132 | "ServiceToken": "test", 133 | "eksCluster": "myekscluster", 134 | "enable": "true", 135 | "loggingOpts": [ "" 136 | ] 137 | } 138 | }""" 139 | print(on_event(json.loads(jsonObj),'')) 140 | print(is_complete(json.loads(jsonObj),'')) -------------------------------------------------------------------------------- /lib/eks.ts: -------------------------------------------------------------------------------- 1 | import * as fs from 'fs'; 2 | import * as cdk from 'aws-cdk-lib'; 3 | import { KubectlV25Layer } from "@aws-cdk/lambda-layer-kubectl-v25"; 4 | import { CfnJson, CfnResource } from 'aws-cdk-lib'; 5 | import * as ec2 from 'aws-cdk-lib/aws-ec2'; 6 | import * as eks from 'aws-cdk-lib/aws-eks'; 7 | import * as iam from 'aws-cdk-lib/aws-iam'; 8 | import { Key } from 'aws-cdk-lib/aws-kms'; 9 | import { Construct } from 'constructs'; 10 | import * as yaml from 'js-yaml'; 11 | 12 | type eksStackProps = cdk.StackProps & { 13 | eksVpc: ec2.Vpc; 14 | }; 15 | 16 | export class EKS extends cdk.Stack { 17 | public readonly cluster: eks.Cluster; 18 | public readonly awsauth: eks.AwsAuth; 19 | public readonly bastionSecurityGroup: ec2.SecurityGroup; 20 | public readonly kms: Key; 21 | 22 | constructor(scope: Construct, id: string, props: eksStackProps) { 23 | super(scope, id, props); 24 | this.bastionSecurityGroup = new ec2.SecurityGroup( 25 | this, 26 | 'bastionHostSecurityGroup', 27 | { 28 | allowAllOutbound: false, 29 | vpc: props.eksVpc, 30 | } 31 | ); 32 | // Recommended to use connections to manage ingress/egress for security groups 33 | this.bastionSecurityGroup.connections.allowTo( 34 | ec2.Peer.anyIpv4(), 35 | ec2.Port.tcp(443), 36 | 'Outbound to 443 only' 37 | ); 38 | this.bastionSecurityGroup.connections.allowFrom( 39 | ec2.Peer.anyIpv4(), 40 | ec2.Port.tcp(22), 41 | 'Allow SSH' 42 | ); 43 | 44 | // Create Custom IAM Role and Policies for Bastion Host 45 | // https://docs.aws.amazon.com/eks/latest/userguide/security_iam_id-based-policy-examples.html#policy_example3 46 | const bastionHostPolicy = new iam.ManagedPolicy( 47 | this, 48 | 'bastionHostManagedPolicy' 49 | ); 50 | bastionHostPolicy.addStatements( 51 | new iam.PolicyStatement({ 52 | resources: ['*'], 53 | actions: [ 54 | 'eks:DescribeNodegroup', 55 | 'eks:ListNodegroups', 56 | 'eks:DescribeCluster', 57 | 'eks:ListClusters', 58 | 'eks:AccessKubernetesApi', 59 | 'eks:ListUpdates', 60 | 'eks:ListFargateProfiles', 61 | ], 62 | effect: iam.Effect.ALLOW, 63 | sid: 'EKSReadonly', 64 | }) 65 | ); 66 | const bastionHostRole = new iam.Role(this, 'bastionHostRole', { 67 | assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'), 68 | managedPolicies: [ 69 | iam.ManagedPolicy.fromAwsManagedPolicyName( 70 | 'AmazonSSMManagedInstanceCore' 71 | ), 72 | bastionHostPolicy, 73 | ], 74 | }); 75 | 76 | const KEY_PAIR_NAME = 'bastionHostKeyPair'; 77 | const cfnKeyPair = new ec2.CfnKeyPair(this, 'BastionHost', { 78 | keyName: KEY_PAIR_NAME, 79 | }); 80 | 81 | // Create Bastion Host, connect using Session Manager, or SSH 82 | const bastionHostLinux = new ec2.Instance(this, 'BastionEKSHost', { 83 | vpc: props.eksVpc, 84 | vpcSubnets: 85 | { 86 | subnetGroupName: 'eks-dmz', 87 | }, 88 | keyName: KEY_PAIR_NAME, 89 | instanceType: new ec2.InstanceType('t4g.nano'), 90 | machineImage: ec2.MachineImage.fromSsmParameter( 91 | '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-arm64-gp2' 92 | ), 93 | securityGroup: this.bastionSecurityGroup, 94 | role: bastionHostRole, 95 | blockDevices: [ 96 | { 97 | deviceName: '/dev/xvda', 98 | volume: ec2.BlockDeviceVolume.ebs(10, { 99 | volumeType: ec2.EbsDeviceVolumeType.GP3, 100 | encrypted: true, 101 | }), 102 | }, 103 | ], 104 | }); 105 | 106 | (bastionHostLinux.node.defaultChild as ec2.CfnInstance).addPropertyOverride("DisableApiTermination", true); 107 | 108 | // Need KMS Key for EKS Envelope Encryption, if deleted, KMS will wait default (30 days) time before removal. 109 | this.kms = new Key(this, 'ekskmskey', { 110 | enableKeyRotation: true, 111 | }); 112 | 113 | this.cluster = new eks.Cluster(this, 'EKSCluster', { 114 | version: eks.KubernetesVersion.V1_25, 115 | defaultCapacity: 0, 116 | endpointAccess: eks.EndpointAccess.PRIVATE, 117 | vpc: props.eksVpc, 118 | kubectlLayer: new KubectlV25Layer(this, 'KubectlLayer'), 119 | secretsEncryptionKey: this.kms, 120 | mastersRole: bastionHostLinux.role, 121 | vpcSubnets: [ 122 | { 123 | subnetGroupName: 'eks-cluster', 124 | }, 125 | ], 126 | // Ensure EKS helper lambdas are in private subnets 127 | placeClusterHandlerInVpc: true, 128 | clusterLogging: [ 129 | eks.ClusterLoggingTypes.API, 130 | eks.ClusterLoggingTypes.AUTHENTICATOR, 131 | eks.ClusterLoggingTypes.SCHEDULER, 132 | eks.ClusterLoggingTypes.AUDIT, 133 | eks.ClusterLoggingTypes.CONTROLLER_MANAGER, 134 | ], 135 | }); 136 | 137 | this.cluster.clusterSecurityGroup.addIngressRule( 138 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 139 | ec2.Port.allTraffic(), 140 | 'Allow VPC' 141 | ); 142 | 143 | this.kms.addToResourcePolicy( 144 | new iam.PolicyStatement({ 145 | principals: [ 146 | new iam.ServicePrincipal(`logs.${this.region}.amazonaws.com`), 147 | ], 148 | actions: [ 149 | 'kms:GenerateDataKey*', 150 | 'kms:Decrypt*', 151 | 'kms:Encrypt*', 152 | 'kms:Describe*', 153 | 'kms:ReEncrypt*', 154 | ], 155 | effect: iam.Effect.ALLOW, 156 | resources: ['*'], 157 | conditions: { 158 | StringEquals: { 159 | 'aws:SourceAccount': this.account, 160 | }, 161 | }, 162 | }) 163 | ); 164 | 165 | this.kms.addToResourcePolicy( 166 | new iam.PolicyStatement({ 167 | principals: [new iam.ArnPrincipal('*')], 168 | actions: [ 169 | 'kms:Encrypt', 170 | 'kms:Decrypt', 171 | 'kms:ReEncrypt*', 172 | 'kms:GenerateDataKey*', 173 | 'kms:DescribeKey', 174 | ], 175 | effect: iam.Effect.ALLOW, 176 | resources: ['*'], 177 | conditions: { 178 | StringEquals: { 179 | 'kms:CallerAccount': this.account, 180 | 'kms:ViaService': `ec2.${this.region}.amazonaws.com`, 181 | }, 182 | 'ForAnyValue:StringEquals': { 183 | 'kms:EncryptionContextKeys': 'aws:ebs:id', 184 | }, 185 | }, 186 | }) 187 | ); 188 | 189 | // Allow BastionHost security group access to EKS Control Plane 190 | bastionHostLinux.connections.allowTo( 191 | this.cluster, 192 | ec2.Port.tcp(443), 193 | 'Allow between BastionHost and EKS ' 194 | ); 195 | // Install kubectl version similar to EKS k8s version 196 | bastionHostLinux.userData.addCommands( 197 | 'yum update -y', 198 | 'yum install -y git', 199 | 'yum remove -y awscli', 200 | 'rm -rf /usr/local/aws-cli', 201 | 'curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o "awscliv2.zip"', 202 | 'unzip awscliv2.zip -d awscliv2', 203 | './awscliv2/aws/install ', 204 | 'ln -s /usr/local/bin/aws /usr/bin/aws', 205 | 'rm -rf awscliv2.zip', 206 | 'curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/1.24.10/2023-01-30/bin/linux/arm64/kubectl', 207 | 'chmod +x ./kubectl', 208 | 'mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin', 209 | "echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc", 210 | 'curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash', 211 | 'curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3', 212 | 'chmod 700 get_helm.sh', 213 | './get_helm.sh', 214 | 'rm -rf get_helm.sh', 215 | `aws eks update-kubeconfig --name ${this.cluster.clusterName} --region ${this.region}` 216 | ); 217 | this.awsauth = new eks.AwsAuth(this, 'EKS_AWSAUTH', { 218 | cluster: this.cluster, 219 | }); 220 | 221 | // deploy Custom k8s RBAC group to provide EKS Web Console read only permissions https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html 222 | // https://aws.github.io/aws-eks-best-practices/security/docs/iam.html#employ-least-privileged-access-when-creating-rolebindings-and-clusterrolebindings 223 | const manifestConsoleViewGroup: Array = yaml.loadAll( 224 | fs.readFileSync('manifests/consoleViewOnlyGroup.yaml', 'utf-8') 225 | ); 226 | const manifestConsoleViewGroupDeploy = new eks.KubernetesManifest( 227 | this, 228 | 'eks-group-view-only', 229 | { 230 | cluster: this.cluster, 231 | manifest: manifestConsoleViewGroup, 232 | } 233 | ); 234 | this.awsauth.node.addDependency(manifestConsoleViewGroupDeploy); 235 | this.awsauth.addMastersRole( 236 | bastionHostLinux.role, 237 | `${bastionHostLinux.role.roleArn}/{{SessionName}}` 238 | ); 239 | // Patch aws-node daemonset to use IRSA via EKS Addons, do before nodes are created 240 | // https://aws.github.io/aws-eks-best-practices/security/docs/iam/#update-the-aws-node-daemonset-to-use-irsa 241 | const awsNodeconditionsPolicy = new CfnJson( 242 | this, 243 | 'awsVpcCniconditionPolicy', 244 | { 245 | value: { 246 | [`${this.cluster.openIdConnectProvider.openIdConnectProviderIssuer}:aud`]: 247 | 'sts.amazonaws.com', 248 | [`${this.cluster.openIdConnectProvider.openIdConnectProviderIssuer}:sub`]: 249 | 'system:serviceaccount:kube-system:aws-node', 250 | }, 251 | } 252 | ); 253 | const awsNodePrincipal = new iam.OpenIdConnectPrincipal( 254 | this.cluster.openIdConnectProvider 255 | ).withConditions({ 256 | StringEquals: awsNodeconditionsPolicy, 257 | }); 258 | const awsVpcCniRole = new iam.Role(this, 'awsVpcCniRole', { 259 | assumedBy: awsNodePrincipal, 260 | }); 261 | 262 | awsVpcCniRole.addManagedPolicy( 263 | iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKS_CNI_Policy') 264 | ); 265 | (() => 266 | new eks.CfnAddon(this, 'vpc-cni', { 267 | addonName: 'vpc-cni', 268 | resolveConflicts: 'OVERWRITE', 269 | serviceAccountRoleArn: awsVpcCniRole.roleArn, 270 | clusterName: this.cluster.clusterName, 271 | addonVersion: this.node.tryGetContext('eks-addon-vpc-cni-version'), 272 | }))(); 273 | (() => 274 | new eks.CfnAddon(this, 'kube-proxy', { 275 | addonName: 'kube-proxy', 276 | resolveConflicts: 'OVERWRITE', 277 | clusterName: this.cluster.clusterName, 278 | addonVersion: this.node.tryGetContext('eks-addon-kube-proxy-version'), 279 | }))(); 280 | (() => 281 | new eks.CfnAddon(this, 'core-dns', { 282 | addonName: 'coredns', 283 | resolveConflicts: 'OVERWRITE', 284 | clusterName: this.cluster.clusterName, 285 | addonVersion: this.node.tryGetContext('eks-addon-coredns-version'), 286 | }))(); 287 | } 288 | 289 | // Create nodegroup IAM role in same stack as eks cluster to ensure there is not a circular dependency 290 | public createNodegroupRole(id: string): iam.Role { 291 | const role = new iam.Role(this, id, { 292 | assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'), 293 | }); 294 | role.addManagedPolicy( 295 | iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSWorkerNodePolicy') 296 | ); 297 | role.addManagedPolicy( 298 | iam.ManagedPolicy.fromAwsManagedPolicyName( 299 | 'AmazonEC2ContainerRegistryReadOnly' 300 | ) 301 | ); 302 | role.addManagedPolicy( 303 | iam.ManagedPolicy.fromAwsManagedPolicyName( 304 | 'CloudWatchAgentServerPolicy' 305 | ) 306 | ); 307 | this.awsauth.addRoleMapping(role, { 308 | username: 'system:node:{{EC2PrivateDNSName}}', 309 | groups: ['system:bootstrappers', 'system:nodes'], 310 | }); 311 | return role; 312 | } 313 | 314 | } 315 | -------------------------------------------------------------------------------- /lib/k8s-baseline.ts: -------------------------------------------------------------------------------- 1 | import * as fs from 'fs'; 2 | import * as eks from 'aws-cdk-lib/aws-eks'; 3 | import * as iam from 'aws-cdk-lib/aws-iam'; 4 | import * as kms from 'aws-cdk-lib/aws-kms'; 5 | import * as cdk from 'aws-cdk-lib'; 6 | import { Construct } from 'constructs'; 7 | import * as yaml from 'js-yaml'; 8 | import * as genPolicy from './policies/policies'; 9 | 10 | type k8sBaselineProps = cdk.StackProps & { 11 | eksCluster: eks.Cluster; 12 | eksKms: kms.Key; 13 | }; 14 | 15 | export class EKSk8sBaseline extends cdk.Stack { 16 | constructor(scope: Construct, id: string, props: k8sBaselineProps) { 17 | super(scope, id, props); 18 | // ============================================================================================================================================ 19 | // Resource Creation 20 | // ============================================================================================================================================ 21 | /* 22 | Service Account Resources will be created in CDK to ensure proper IAM to K8s RBAC Mapping 23 | Helm Chart Version are taken from cdk.json file or from command line parameter -c 24 | Helm Chart full version list can be found via helm repo list or viewing yaml file on github directly, see README. 25 | */ 26 | 27 | /* 28 | Resources needed to create Fluent Bit DaemonSet 29 | Namespace 30 | Service Account Role 31 | IAM Policy 32 | K8s Manifest 33 | 34 | Current Config pushes to Cloudwatch , other outputs found here https://docs.fluentbit.io/manual/pipeline/outputs 35 | Fluentbit does not support IMDSv2 36 | https://github.com/fluent/fluent-bit/issues/2840#issuecomment-774393238 37 | */ 38 | 39 | // YAML contains fluentbit parser configurations, remove namespace and serviceaccount from yaml to properly annotate with IAM Role 40 | const manifestFluentBitSetup = this.cleanManifest( 41 | 'manifests/fluentBitSetup.yaml' 42 | ); 43 | const fluentBitNamespace = new eks.KubernetesManifest( 44 | this, 45 | 'amazon-cloudwatch-namespace', 46 | { 47 | cluster: props.eksCluster, 48 | manifest: [ 49 | { 50 | apiVersion: 'v1', 51 | kind: 'Namespace', 52 | metadata: { 53 | name: 'amazon-cloudwatch', 54 | labels: { 55 | name: 'amazon-cloudwatch', 56 | }, 57 | }, 58 | }, 59 | ], 60 | } 61 | ); 62 | const fluentBitSA = new eks.ServiceAccount(this, 'fluentbit-sa', { 63 | name: 'fluent-bit', 64 | namespace: 'amazon-cloudwatch', 65 | cluster: props.eksCluster, 66 | }); 67 | fluentBitSA.node.addDependency(fluentBitNamespace); 68 | genPolicy.createFluentbitPolicy( 69 | this, 70 | props.eksCluster.clusterName, 71 | fluentBitSA.role 72 | ); 73 | // Configurable variables for manifests/fluentBitSetup.yaml 74 | const fluentBitClusterInfo = new eks.KubernetesManifest( 75 | this, 76 | 'fluentbit-cluster-info', 77 | { 78 | cluster: props.eksCluster, 79 | manifest: [ 80 | { 81 | apiVersion: 'v1', 82 | kind: 'ConfigMap', 83 | metadata: { 84 | name: 'fluent-bit-cluster-info', 85 | namespace: 'amazon-cloudwatch', 86 | labels: { 87 | name: 'fluent-bit-cluster-info', 88 | }, 89 | }, 90 | data: { 91 | 'cluster.name': props.eksCluster.clusterName, 92 | 'http.port': '2020', 93 | 'http.server': 'On', 94 | 'logs.region': this.region, 95 | 'read.head': 'Off', 96 | 'read.tail': 'On', 97 | }, 98 | }, 99 | ], 100 | } 101 | ); 102 | fluentBitClusterInfo.node.addDependency(fluentBitNamespace); 103 | const fluentBitResource = new eks.KubernetesManifest( 104 | this, 105 | 'fluentbit-resource', 106 | { 107 | cluster: props.eksCluster, 108 | manifest: manifestFluentBitSetup, 109 | } 110 | ); 111 | fluentBitResource.node.addDependency(fluentBitSA); 112 | fluentBitResource.node.addDependency(fluentBitClusterInfo); 113 | 114 | /* 115 | Resources needed to create ALB Ingress Controller 116 | Namespace 117 | Service Account Role 118 | IAM Policy 119 | Helm Chart 120 | AddOn: https://github.com/aws/containers-roadmap/issues/1162 121 | */ 122 | 123 | // Create Namespace and Service Account for ALB Ingress 124 | const albNamespace = new eks.KubernetesManifest( 125 | this, 126 | 'alb-ingress-controller-namespace', 127 | { 128 | cluster: props.eksCluster, 129 | manifest: [ 130 | { 131 | apiVersion: 'v1', 132 | kind: 'Namespace', 133 | metadata: { 134 | name: 'alb-ingress-controller', 135 | labels: { 136 | name: 'alb-ingress-controller', 137 | }, 138 | }, 139 | }, 140 | ], 141 | } 142 | ); 143 | const albSA = new eks.ServiceAccount(this, 'alb-ingress-controller-sa', { 144 | name: 'alb-ingress-controller-sa', 145 | namespace: 'alb-ingress-controller', 146 | cluster: props.eksCluster, 147 | }); 148 | albSA.node.addDependency(albNamespace); 149 | 150 | // ALB Controller IAMPolicy 151 | genPolicy.createAlbIngressPolicy( 152 | this, 153 | props.eksCluster.clusterName, 154 | albSA.role 155 | ); 156 | // https://github.com/aws/eks-charts/blob/master/stable/aws-load-balancer-controller/values.yaml 157 | const albIngressHelmChart = new eks.HelmChart( 158 | this, 159 | 'alb-ingress-controller-chart', 160 | { 161 | chart: 'aws-load-balancer-controller', 162 | cluster: props.eksCluster, 163 | repository: 'https://aws.github.io/eks-charts', 164 | wait: true, 165 | release: 'aws-load-balancer-controller', 166 | createNamespace: true, 167 | namespace: 'alb-ingress-controller', 168 | // https://github.com/aws/eks-charts/blob/gh-pages/index.yaml 169 | version: this.node.tryGetContext( 170 | 'aws-load-balancer-controller-helm-version' 171 | ), 172 | values: { 173 | clusterName: props.eksCluster.clusterName, 174 | defaultTags: { 175 | 'eks:cluster-name': props.eksCluster.clusterName, 176 | }, 177 | // Start - values needed if ec2metadata endpoint is unavailable - https://github.com/aws/eks-charts/tree/master/stable/aws-load-balancer-controller#configuration 178 | region: this.region, 179 | vpcId: props.eksCluster.vpc.vpcId, 180 | // End - values needed if ec2metadata endpoint is unavailable 181 | serviceAccount: { 182 | create: false, 183 | name: albSA.serviceAccountName, 184 | }, 185 | }, 186 | } 187 | ); 188 | albIngressHelmChart.node.addDependency(albSA); 189 | /* 190 | Resources needed to create EBS CSI Driver 191 | Service Account Role 192 | IAM Policy 193 | Helm Chart 194 | Add On: https://github.com/aws/containers-roadmap/issues/247 195 | */ 196 | 197 | // Create Service Account (Pod IAM Role Mapping) for EBS Controller 198 | const ebsSA = new eks.ServiceAccount(this, 'ebs-csi-controller-sa', { 199 | name: 'ebs-csi-controller-sa', 200 | namespace: 'kube-system', 201 | cluster: props.eksCluster, 202 | }); 203 | 204 | // EBS Controller IAMPolicyDoc 205 | genPolicy.createEBSPolicy(this, props.eksCluster.clusterName, ebsSA.role); 206 | 207 | ebsSA.role.attachInlinePolicy( 208 | new iam.Policy(this, 'EncryptEBS', { 209 | statements: [ 210 | new iam.PolicyStatement({ 211 | actions: ['kms:CreateGrant', 'kms:ListGrants', 'kms:RevokeGrant'], 212 | effect: iam.Effect.ALLOW, 213 | resources: [props.eksKms.keyArn], 214 | conditions: { 215 | Bool: { 216 | 'kms:GrantIsForAWSResource': 'true', 217 | }, 218 | }, 219 | }), 220 | new iam.PolicyStatement({ 221 | actions: [ 222 | 'kms:Encrypt', 223 | 'kms:Decrypt', 224 | 'kms:ReEncrypt*', 225 | 'kms:GenerateDataKey*', 226 | 'kms:DescribeKey', 227 | ], 228 | effect: iam.Effect.ALLOW, 229 | resources: [props.eksKms.keyArn], 230 | }), 231 | ], 232 | }) 233 | ); 234 | 235 | // Helm Chart Values: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/charts/aws-ebs-csi-driver/values.yaml 236 | const ebsCsiHelmChart = new eks.HelmChart(this, 'ebs-csi-helm-chart', { 237 | chart: 'aws-ebs-csi-driver', 238 | cluster: props.eksCluster, 239 | createNamespace: true, 240 | repository: 'https://kubernetes-sigs.github.io/aws-ebs-csi-driver', 241 | release: 'aws-ebs-csi-driver', 242 | namespace: 'kube-system', 243 | wait: true, 244 | // Helm Chart Versions: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/gh-pages/index.yaml 245 | version: this.node.tryGetContext('aws-ebs-csi-driver-helm-version'), 246 | values: { 247 | controller: { 248 | serviceAccount: { 249 | create: false, 250 | name: ebsSA.serviceAccountName, 251 | }, 252 | extraVolumeTags: { 253 | 'eks:cluster-name': props.eksCluster.clusterName, 254 | }, 255 | }, 256 | }, 257 | }); 258 | 259 | ebsCsiHelmChart.node.addDependency(ebsSA); 260 | } 261 | 262 | // Removes namespace and ServiceAccount objects from manifests, performing this in code to keep original manifest files. 263 | cleanManifest(file: string) { 264 | const manifest: Array = yaml.loadAll( 265 | fs.readFileSync(file, 'utf-8'), 266 | null, 267 | { schema: yaml.JSON_SCHEMA } 268 | ); 269 | return manifest.filter( 270 | (element) => 271 | element.kind !== 'Namespace' && element.kind !== 'ServiceAccount' 272 | ); 273 | } 274 | } 275 | -------------------------------------------------------------------------------- /lib/nodegroup.ts: -------------------------------------------------------------------------------- 1 | import * as cdk from 'aws-cdk-lib'; 2 | import { Fn } from 'aws-cdk-lib'; 3 | import * as ec2 from 'aws-cdk-lib/aws-ec2'; 4 | import * as eks from 'aws-cdk-lib/aws-eks'; 5 | import * as iam from 'aws-cdk-lib/aws-iam'; 6 | import * as kms from 'aws-cdk-lib/aws-kms'; 7 | import { Construct } from 'constructs'; 8 | 9 | type k8snodegroupsProps = cdk.StackProps & { 10 | eksVpc: ec2.Vpc; 11 | eksCluster: eks.Cluster; 12 | nodeGroupRole: iam.Role; 13 | bastionSecurityGroup: ec2.SecurityGroup; 14 | eksKms: kms.Key; 15 | }; 16 | 17 | export class NodeGroup extends cdk.Stack { 18 | constructor(scope: Construct, id: string, props: k8snodegroupsProps) { 19 | super(scope, id, props); 20 | 21 | const userData = ec2.UserData.forLinux(); 22 | userData.addCommands('#!/bin/bash', 'yum update -y'); 23 | 24 | const multipartUserData = new ec2.MultipartUserData(); 25 | multipartUserData.addPart(ec2.MultipartBody.fromUserData(userData)); 26 | 27 | const ngsg = new ec2.SecurityGroup(this, 'NodeGroupSG', { 28 | vpc: props.eksVpc, 29 | allowAllOutbound: true, 30 | description: 'Node Group SG', 31 | }); 32 | 33 | // For Erigon: https://github.com/ledgerwatch/erigon#default-ports-and-protocols--firewalls 34 | // public 35 | ngsg.addIngressRule( 36 | ec2.Peer.anyIpv4(), 37 | ec2.Port.tcp(30303), 38 | 'eth/66 peering' 39 | ); 40 | ngsg.addIngressRule( 41 | ec2.Peer.anyIpv4(), 42 | ec2.Port.udp(30303), 43 | 'eth/66 peering' 44 | ); 45 | ngsg.addIngressRule( 46 | ec2.Peer.anyIpv4(), 47 | ec2.Port.tcp(30304), 48 | 'eth/67 peering' 49 | ); 50 | ngsg.addIngressRule( 51 | ec2.Peer.anyIpv4(), 52 | ec2.Port.udp(30304), 53 | 'eth/67 peering' 54 | ); 55 | ngsg.addIngressRule( 56 | ec2.Peer.anyIpv4(), 57 | ec2.Port.tcp(42069), 58 | 'Snap sync (Bittorrent)' 59 | ); 60 | ngsg.addIngressRule( 61 | ec2.Peer.anyIpv4(), 62 | ec2.Port.udp(42069), 63 | 'Snap sync (Bittorrent)' 64 | ); 65 | ngsg.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.udp(4000), 'Peering'); 66 | ngsg.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.udp(4001), 'Peering'); 67 | 68 | // private 69 | ngsg.addIngressRule( 70 | ec2.Peer.securityGroupId(props.bastionSecurityGroup.securityGroupId), 71 | ec2.Port.tcp(443), 72 | 'Bastion Host' 73 | ); 74 | ngsg.addIngressRule( 75 | ec2.Peer.securityGroupId(props.eksCluster.clusterSecurityGroupId), 76 | ec2.Port.allTraffic(), 77 | 'EKS' 78 | ); 79 | ngsg.addIngressRule( 80 | ngsg, 81 | ec2.Port.udpRange(20,60), 82 | 'Kubernetes DNS' 83 | ); 84 | ngsg.addIngressRule( 85 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 86 | ec2.Port.tcp(9090), 87 | 'gRPC Connections' 88 | ); 89 | ngsg.addIngressRule( 90 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 91 | ec2.Port.tcp(6060), 92 | 'Metrics or Pprof' 93 | ); 94 | ngsg.addIngressRule( 95 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 96 | ec2.Port.tcp(8551), 97 | 'Engine API (JWT auth)' 98 | ); 99 | ngsg.addIngressRule( 100 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 101 | ec2.Port.tcp(8545), 102 | 'RPC' 103 | ); 104 | ngsg.addIngressRule( 105 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 106 | ec2.Port.tcp(9091), 107 | 'gRPC Connections' 108 | ); 109 | ngsg.addIngressRule( 110 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 111 | ec2.Port.tcp(7777), 112 | 'gRPC Connections' 113 | ); 114 | ngsg.addIngressRule( 115 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 116 | ec2.Port.tcp(6060), 117 | 'pprof/metrics' 118 | ); 119 | ngsg.addIngressRule( 120 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 121 | ec2.Port.tcp(9092), 122 | 'gRPC (reserved)' 123 | ); 124 | ngsg.addIngressRule( 125 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 126 | ec2.Port.tcp(9093), 127 | 'gRPC (reserved)' 128 | ); 129 | ngsg.addIngressRule( 130 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 131 | ec2.Port.tcp(9094), 132 | 'gRPC (reserved)' 133 | ); 134 | ngsg.addIngressRule( 135 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 136 | ec2.Port.tcp(9100), 137 | 'prometheus node-exporter metrics' 138 | ); 139 | ngsg.addIngressRule( 140 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 141 | ec2.Port.tcp(10249), 142 | 'prometheus kube-proxy metrics' 143 | ); 144 | ngsg.addIngressRule( 145 | ec2.Peer.ipv4(props.eksVpc.vpcCidrBlock), 146 | ec2.Port.tcp(10250), 147 | 'prometheus kubelete metrics' 148 | ); 149 | 150 | const launchtemplate = new ec2.CfnLaunchTemplate(this, 'LaunchTemplate', { 151 | launchTemplateData: { 152 | instanceType: 'r7g.2xlarge', 153 | securityGroupIds: [ngsg.securityGroupId], 154 | userData: Fn.base64(multipartUserData.render()), 155 | }, 156 | }); 157 | 158 | function getNodeGroupAzs(azs: string | undefined): string[] | undefined { 159 | return azs?.split(',') ?? undefined; 160 | } 161 | 162 | new eks.Nodegroup(this, 'NodeGroup', { 163 | amiType: eks.NodegroupAmiType.AL2_ARM_64, 164 | cluster: props.eksCluster, 165 | nodeRole: props.nodeGroupRole, 166 | maxSize: this.node.tryGetContext('nodeGroupMaxSize'), 167 | desiredSize: this.node.tryGetContext('nodeGroupDesiredSize'), 168 | minSize: this.node.tryGetContext('nodeGroupMinSize'), 169 | subnets: { 170 | availabilityZones: getNodeGroupAzs( 171 | this.node.tryGetContext('availability_zones') 172 | ), 173 | subnetGroupName: 'eks-nodes', 174 | }, 175 | launchTemplateSpec: { 176 | id: launchtemplate.ref, 177 | version: launchtemplate.attrLatestVersionNumber, 178 | }, 179 | tags: { 180 | Name: Fn.join('-', [props.eksCluster.clusterName, 'WorkerNodes']), 181 | }, 182 | }); 183 | 184 | // Permissions for SSM Manager for core functionality 185 | props.nodeGroupRole.addManagedPolicy( 186 | iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonSSMManagedInstanceCore') 187 | ); 188 | } 189 | } 190 | -------------------------------------------------------------------------------- /lib/observe.ts: -------------------------------------------------------------------------------- 1 | import * as cdk from 'aws-cdk-lib'; 2 | import * as aps from 'aws-cdk-lib/aws-aps'; 3 | import * as logs from 'aws-cdk-lib/aws-logs'; 4 | import * as eks from 'aws-cdk-lib/aws-eks'; 5 | import * as iam from 'aws-cdk-lib/aws-iam'; 6 | import * as ec2 from 'aws-cdk-lib/aws-ec2'; 7 | import * as kms from 'aws-cdk-lib/aws-kms'; 8 | import { aws_grafana as grafana } from 'aws-cdk-lib'; 9 | import { Construct } from 'constructs'; 10 | import { RemovalPolicy } from 'aws-cdk-lib'; 11 | import { CfnJson } from 'aws-cdk-lib'; 12 | 13 | type observeProps = cdk.StackProps & { 14 | eksCluster: eks.Cluster; 15 | eksKms: kms.Key; 16 | vpc: ec2.Vpc; 17 | }; 18 | 19 | export class Observe extends cdk.Stack { 20 | constructor(scope: Construct, id: string, props: observeProps) { 21 | super(scope, id, props); 22 | 23 | const ampLogGroup = new logs.LogGroup(this, 'AmpLogGroup', { 24 | retention: logs.RetentionDays.ONE_WEEK, 25 | encryptionKey: props.eksKms, 26 | removalPolicy: RemovalPolicy.DESTROY, 27 | }); 28 | 29 | const cfnWorkspace = new aps.CfnWorkspace(this, 'PrometheusWorkspace', { 30 | loggingConfiguration: { 31 | logGroupArn: ampLogGroup.logGroupArn, 32 | }, 33 | }); 34 | 35 | const ingestCondition = new CfnJson(this, 'IngestCondition', { 36 | value: { 37 | [`${props.eksCluster.clusterOpenIdConnectIssuer}:sub`]: 38 | 'system:serviceaccount:monitoring:amp-iamproxy-ingest-service-account', 39 | }, 40 | }); 41 | 42 | const ingestRole = new iam.Role(this, 'IngestRole', { 43 | assumedBy: new iam.PrincipalWithConditions( 44 | new iam.WebIdentityPrincipal( 45 | `arn:aws:iam::${this.account}:oidc-provider/${props.eksCluster.openIdConnectProvider.openIdConnectProviderIssuer}` 46 | ), 47 | { 48 | StringEquals: ingestCondition, 49 | } 50 | ), 51 | description: 'Role for ingesting Prometheus metrics', 52 | inlinePolicies: { 53 | amp: new iam.PolicyDocument({ 54 | statements: [ 55 | new iam.PolicyStatement({ 56 | actions: [ 57 | 'aps:RemoteWrite', 58 | 'aps:GetSeries', 59 | 'aps:GetLabels', 60 | 'aps:GetMetricMetadata', 61 | ], 62 | effect: iam.Effect.ALLOW, 63 | resources: ['*'], 64 | }), 65 | ], 66 | }), 67 | }, 68 | }); 69 | 70 | const queryCondition = new CfnJson(this, 'QueryCondition', { 71 | value: { 72 | [`${props.eksCluster.clusterOpenIdConnectIssuer}:sub`]: [ 73 | 'system:serviceaccount:monitoring:amp-iamproxy-query-service-account', 74 | 'system:serviceaccount:monitoring:amp-iamproxy-query-service-account', 75 | ], 76 | }, 77 | }); 78 | 79 | new iam.Role(this, 'QueryRole', { 80 | assumedBy: new iam.PrincipalWithConditions( 81 | new iam.WebIdentityPrincipal( 82 | `arn:aws:iam::${this.account}:oidc-provider/${props.eksCluster.openIdConnectProvider.openIdConnectProviderIssuer}` 83 | ), 84 | { 85 | StringEquals: queryCondition, 86 | } 87 | ), 88 | description: 'Role for querying Prometheus metrics', 89 | inlinePolicies: { 90 | amp: new iam.PolicyDocument({ 91 | statements: [ 92 | new iam.PolicyStatement({ 93 | actions: [ 94 | 'aps:QueryMetrics', 95 | 'aps:GetSeries', 96 | 'aps:GetLabels', 97 | 'aps:GetMetricMetadata', 98 | ], 99 | effect: iam.Effect.ALLOW, 100 | resources: ['*'], 101 | }), 102 | ], 103 | }), 104 | }, 105 | }); 106 | 107 | new eks.HelmChart(this, 'Prometheus', { 108 | cluster: props.eksCluster, 109 | chart: 'kube-prometheus-stack', 110 | repository: 'https://prometheus-community.github.io/helm-charts', 111 | namespace: 'monitoring', 112 | release: 'kube-prometheus', 113 | createNamespace: true, 114 | values: { 115 | prometheus: { 116 | serviceAccount: { 117 | create: true, 118 | name: 'amp-iamproxy-ingest-service-account', 119 | annotations: { 120 | 'eks.amazonaws.com/role-arn': ingestRole.roleArn, 121 | }, 122 | }, 123 | prometheusSpec: { 124 | remoteWrite: [ 125 | { 126 | url: `https://aps-workspaces.${this.region}.amazonaws.com/workspaces/${cfnWorkspace.attrWorkspaceId}/api/v1/remote_write`, 127 | sigv4: { 128 | region: this.region, 129 | }, 130 | queueConfig: { 131 | maxSamplesPerSend: 1000, 132 | maxShards: 200, 133 | capacity: 2500, 134 | }, 135 | }, 136 | ], 137 | }, 138 | }, 139 | alertmaanger: { 140 | enabled: false 141 | }, 142 | grafana: { 143 | enabled: false 144 | }, 145 | promtheusOperator: { 146 | tls: { 147 | enabled: false 148 | }, 149 | admissionWebhooks: { 150 | enabled: false, 151 | patch: { 152 | enabled: false 153 | } 154 | }, 155 | }, 156 | }, 157 | }); 158 | 159 | const grafanaRole = new iam.Role(this, 'GrafanaRole', { 160 | assumedBy: new iam.ServicePrincipal('grafana.amazonaws.com'), 161 | description: 'Role used to administer Grafana workspace for Ethereum', 162 | inlinePolicies: { 163 | 'list-amp': new iam.PolicyDocument({ 164 | statements: [ 165 | new iam.PolicyStatement({ 166 | actions: ['aps:ListWorkspaces'], 167 | effect: iam.Effect.ALLOW, 168 | resources: [ 169 | `arn:aws:aps:${this.region}:${this.account}:/workspaces`, 170 | ], 171 | }), 172 | ], 173 | }), 174 | 'query-amp': new iam.PolicyDocument({ 175 | statements: [ 176 | new iam.PolicyStatement({ 177 | actions: [ 178 | 'aps:GetLabels', 179 | 'aps:GetMetricMetadata', 180 | 'aps:GetSeries', 181 | 'aps:QueryMetrics', 182 | 'aps:DescribeWorkspace', 183 | ], 184 | effect: iam.Effect.ALLOW, 185 | resources: [cfnWorkspace.attrArn], 186 | }), 187 | ], 188 | }), 189 | }, 190 | }); 191 | 192 | const grafanaSg = new ec2.SecurityGroup(this, 'GrafanaSG', { 193 | vpc: props.vpc, 194 | allowAllOutbound: true, 195 | description: 'Amazon Managed Grafana Security Group for Ethereum', 196 | }); 197 | 198 | new grafana.CfnWorkspace(this, 'Grafana', { 199 | accountAccessType: 'CURRENT_ACCOUNT', 200 | description: 'Ethereum Client', 201 | permissionType: 'SERVICE_MANAGED', 202 | roleArn: grafanaRole.roleArn, 203 | authenticationProviders: ['AWS_SSO'], 204 | notificationDestinations: ['SNS'], 205 | vpcConfiguration: { 206 | securityGroupIds: [grafanaSg.securityGroupId], 207 | subnetIds: props.vpc.selectSubnets({ subnetGroupName: 'eks-nodes' }) 208 | .subnetIds, 209 | }, 210 | }); 211 | } 212 | } 213 | -------------------------------------------------------------------------------- /lib/policies/policies.ts: -------------------------------------------------------------------------------- 1 | import { Policy, PolicyStatement, IRole } from 'aws-cdk-lib/aws-iam'; 2 | import { Stack } from 'aws-cdk-lib'; 3 | 4 | // Scope fluentbit to push logs to log-group /aws/containerinsights/CLUSTER_NAME/ only 5 | export function createFluentbitPolicy( 6 | stack: Stack, 7 | clusterName: string, 8 | roleSA: IRole 9 | ): Policy { 10 | const fluentBitSaRoleStatementPolicy = new PolicyStatement({ 11 | resources: [ 12 | `arn:aws:logs:${stack.region}:${stack.account}:log-group:/aws/containerinsights/${clusterName}/*`, 13 | `arn:aws:logs:${stack.region}:${stack.account}:log-group:/aws/containerinsights/${clusterName}/*:log-stream:*`, 14 | ], 15 | actions: [ 16 | 'logs:CreateLogStream', 17 | 'logs:CreateLogGroup', 18 | 'logs:DescribeLogStreams', 19 | 'logs:PutLogEvents', 20 | ], 21 | }); 22 | return new Policy(stack, 'fluentBitSaRolePolicy', { 23 | roles: [roleSA], 24 | statements: [fluentBitSaRoleStatementPolicy], 25 | }); 26 | } 27 | 28 | export function createEBSPolicy( 29 | stack: Stack, 30 | clusterName: string, 31 | roleSA: IRole 32 | ): Policy { 33 | // Scope permissions to describe on all resources, some APIs like ec2:DescribeAvailabilityZones do not Resource types 34 | // https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonec2.html#amazonec2-actions-as-permissions 35 | const readEBSPolicy = new PolicyStatement({ 36 | resources: ['*'], 37 | actions: [ 38 | 'ec2:DescribeAvailabilityZones', 39 | 'ec2:DescribeInstances', 40 | 'ec2:DescribeSnapshots', 41 | 'ec2:DescribeTags', 42 | 'ec2:DescribeVolumes', 43 | 'ec2:DescribeVolumesModifications', 44 | ], 45 | }); 46 | // Scope to createTags only creation of volumes and snapshots 47 | const createTags = new PolicyStatement({ 48 | resources: [ 49 | `arn:aws:ec2:${stack.region}:${stack.account}:volume/*`, 50 | `arn:aws:ec2:${stack.region}:${stack.account}:snapshot/*`, 51 | ], 52 | conditions: { 53 | StringEquals: { 54 | 'ec2:CreateAction': ['CreateVolume', 'CreateSnapshot'], 55 | }, 56 | }, 57 | actions: ['ec2:CreateTags'], 58 | }); 59 | // Scope deletion of tags on volumes and snapshots that already contain eks:cluster-name: MY_CLUSTER_NAME 60 | const deleteTags = new PolicyStatement({ 61 | resources: [ 62 | `arn:aws:ec2:${stack.region}:${stack.account}:volume/*`, 63 | `arn:aws:ec2:${stack.region}:${stack.account}:snapshot/*`, 64 | ], 65 | conditions: { 66 | StringEquals: { 67 | 'aws:ResourceTag/eks:cluster-name': clusterName, 68 | }, 69 | }, 70 | actions: ['ec2:DeleteTags'], 71 | }); 72 | // Scope Attach/Detach/Modify of EBS policies to tags eks:cluster-name': MY_CLUSTER_NAME 73 | const modifyVolume = new PolicyStatement({ 74 | resources: [ 75 | `arn:aws:ec2:${stack.region}:${stack.account}:instance/*`, 76 | `arn:aws:ec2:${stack.region}:${stack.account}:volume/*`, 77 | ], 78 | actions: ['ec2:AttachVolume', 'ec2:DetachVolume', 'ec2:ModifyVolume'], 79 | conditions: { 80 | StringEquals: { 81 | 'aws:ResourceTag/eks:cluster-name': clusterName, 82 | }, 83 | }, 84 | }); 85 | // Scope CreateVolume only when Request contains tags eks:cluster-name: MY_CLUSTER_NAME 86 | const createVolume = new PolicyStatement({ 87 | resources: ['*'], 88 | conditions: { 89 | StringEquals: { 90 | 'aws:RequestTag/eks:cluster-name': clusterName, 91 | }, 92 | }, 93 | actions: ['ec2:CreateVolume'], 94 | }); 95 | // Scope DeleteVolume only when Resource contains tag eks:cluster-name: MY_CLUSTER_NAME 96 | const deleteVolume = new PolicyStatement({ 97 | resources: ['*'], 98 | conditions: { 99 | StringEquals: { 100 | 'aws:ResourceTag/eks:cluster-name': clusterName, 101 | }, 102 | }, 103 | actions: [ 104 | 'ec2:DeleteVolume', 105 | 'ec2:DetachVolume', 106 | 'ec2:AttachVolume', 107 | 'ec2:ModifyVolume', 108 | ], 109 | }); 110 | // Scope Permission to createsnapshot only when Request contains tag eks:cluster-name: MY_CLUSTER_NAME 111 | const createSnapshot = new PolicyStatement({ 112 | resources: ['*'], 113 | conditions: { 114 | StringEquals: { 115 | 'aws:RequestTag/eks:cluster-name': clusterName, 116 | }, 117 | }, 118 | actions: ['ec2:CreateSnapshot'], 119 | }); 120 | // Scope Permission to DeleteSnapshot only when Resource contains tag eks:cluster-name: MY_CLUSTER_NAME 121 | const deleteSnapshot = new PolicyStatement({ 122 | resources: ['*'], 123 | conditions: { 124 | StringEquals: { 125 | 'aws:ResourceTag/eks:cluster-name': clusterName, 126 | }, 127 | }, 128 | actions: ['ec2:DeleteSnapshot'], 129 | }); 130 | return new Policy(stack, 'ebsDriverPolicy', { 131 | roles: [roleSA], 132 | statements: [ 133 | readEBSPolicy, 134 | createTags, 135 | deleteTags, 136 | createVolume, 137 | deleteVolume, 138 | modifyVolume, 139 | createSnapshot, 140 | deleteSnapshot, 141 | ], 142 | }); 143 | } 144 | 145 | export function createAlbIngressPolicy( 146 | stack: Stack, 147 | clusterName: string, 148 | roleSA: IRole 149 | ): Policy { 150 | /* Permissions are board to include all functionality of ALB, Permissions can be removed as fit, refer to annotations to see which actions are needed 151 | https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/ingress/annotations/ 152 | Custom Permission set can be generated by using IAM generated policy 153 | https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_generate-policy.html 154 | */ 155 | 156 | // Permission to create ELB ServiceLinkRole if not already created, scoped to only elasticloadbalancing service role 157 | const serviceLinkedRole = new PolicyStatement({ 158 | actions: ['iam:CreateServiceLinkedRole'], 159 | resources: ['*'], 160 | conditions: { 161 | StringEquals: { 162 | 'iam:AWSServiceName': 'elasticloadbalancing.amazonaws.com', 163 | }, 164 | }, 165 | }); 166 | // Permission needs to self discovery networking attributes 167 | const readPolicy = new PolicyStatement({ 168 | actions: [ 169 | 'ec2:DescribeAccountAttributes', 170 | 'ec2:DescribeAddresses', 171 | 'ec2:DescribeAvailabilityZones', 172 | 'ec2:DescribeInternetGateways', 173 | 'ec2:DescribeVpcs', 174 | 'ec2:DescribeSubnets', 175 | 'ec2:DescribeSecurityGroups', 176 | 'ec2:DescribeInstances', 177 | 'ec2:DescribeNetworkInterfaces', 178 | 'ec2:DescribeTags', 179 | 'ec2:GetCoipPoolUsage', 180 | 'ec2:DescribeCoipPools', 181 | 'elasticloadbalancing:DescribeLoadBalancers', 182 | 'elasticloadbalancing:DescribeLoadBalancerAttributes', 183 | 'elasticloadbalancing:DescribeListeners', 184 | 'elasticloadbalancing:DescribeListenerCertificates', 185 | 'elasticloadbalancing:DescribeSSLPolicies', 186 | 'elasticloadbalancing:DescribeRules', 187 | 'elasticloadbalancing:DescribeTargetGroups', 188 | 'elasticloadbalancing:DescribeTargetGroupAttributes', 189 | 'elasticloadbalancing:DescribeTargetHealth', 190 | 'elasticloadbalancing:DescribeTags', 191 | ], 192 | resources: ['*'], 193 | }); 194 | // Additional Permissions for shield, waf acm and cognito feature set enablement 195 | const readPolicyAdd = new PolicyStatement({ 196 | actions: [ 197 | 'cognito-idp:DescribeUserPoolClient', 198 | 'acm:ListCertificates', 199 | 'acm:DescribeCertificate', 200 | 'iam:ListServerCertificates', 201 | 'iam:GetServerCertificate', 202 | 'waf-regional:GetWebACL', 203 | 'waf-regional:GetWebACLForResource', 204 | 'waf-regional:AssociateWebACL', 205 | 'waf-regional:DisassociateWebACL', 206 | 'wafv2:GetWebACL', 207 | 'wafv2:GetWebACLForResource', 208 | 'wafv2:AssociateWebACL', 209 | 'wafv2:DisassociateWebACL', 210 | 'shield:GetSubscriptionState', 211 | 'shield:DescribeProtection', 212 | 'shield:CreateProtection', 213 | 'shield:DeleteProtection', 214 | ], 215 | resources: ['*'], 216 | }); 217 | // Enable usage of ingress rule for security groups created outside of controller 218 | const writeSG = new PolicyStatement({ 219 | actions: [ 220 | 'ec2:AuthorizeSecurityGroupIngress', 221 | 'ec2:RevokeSecurityGroupIngress', 222 | ], 223 | resources: ['*'], 224 | }); 225 | // Enable controller to automatically create security groups, tags may be added later 226 | const createSG = new PolicyStatement({ 227 | actions: ['ec2:CreateSecurityGroup'], 228 | resources: ['*'], 229 | }); 230 | // Give tagging permission to actions that create the resource, CreateSecurityGroup 231 | const createTags = new PolicyStatement({ 232 | actions: ['ec2:CreateTags'], 233 | resources: ['arn:aws:ec2:*:*:security-group/*'], 234 | conditions: { 235 | StringEquals: { 236 | 'ec2:CreateAction': 'CreateSecurityGroup', 237 | }, 238 | Null: { 239 | 'aws:RequestTag/elbv2.k8s.aws/cluster': 'false', 240 | }, 241 | }, 242 | }); 243 | // Create and Delete Tags for security groups when at time of authorization aws:RequestTag/elbv2.k8s.aws/cluster is null 244 | const createdeleteTags = new PolicyStatement({ 245 | actions: ['ec2:CreateTags', 'ec2:DeleteTags'], 246 | resources: ['arn:aws:ec2:*:*:security-group/*'], 247 | conditions: { 248 | Null: { 249 | 'aws:RequestTag/elbv2.k8s.aws/cluster': 'true', 250 | 'aws:ResourceTag/elbv2.k8s.aws/cluster': 'false', 251 | }, 252 | }, 253 | }); 254 | // Management of SecurityGroup when at time of authorization aws:ResourceTag/elbv2.k8s.aws/cluster is not null 255 | const writeSGIngress = new PolicyStatement({ 256 | actions: [ 257 | 'ec2:AuthorizeSecurityGroupIngress', 258 | 'ec2:RevokeSecurityGroupIngress', 259 | 'ec2:DeleteSecurityGroup', 260 | ], 261 | resources: ['*'], 262 | conditions: { 263 | Null: { 264 | 'aws:ResourceTag/elbv2.k8s.aws/cluster': 'false', 265 | }, 266 | }, 267 | }); 268 | // Allow creation of LoadBalancer/TargetGroup only if Request contains Tags eks:cluster-name: MY_CLUSTER_NAME 269 | const createLoadBalancer = new PolicyStatement({ 270 | actions: [ 271 | 'elasticloadbalancing:CreateLoadBalancer', 272 | 'elasticloadbalancing:CreateTargetGroup', 273 | ], 274 | resources: ['*'], 275 | conditions: { 276 | StringEquals: { 277 | 'aws:RequestTag/eks:cluster-name': clusterName, 278 | }, 279 | }, 280 | }); 281 | // Management of LoadBalancer Listeners and Rules 282 | // TODO Scope to use tags with release of v2.2.0 283 | // https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/1966 284 | const createLoadBalancerAdd = new PolicyStatement({ 285 | actions: [ 286 | 'elasticloadbalancing:CreateListener', 287 | 'elasticloadbalancing:DeleteListener', 288 | 'elasticloadbalancing:CreateRule', 289 | 'elasticloadbalancing:DeleteRule', 290 | ], 291 | resources: ['*'], 292 | }); 293 | // Management of ELB Tags when at authorization time aws:RequestTag/elbv2.k8s.aws/cluster is null 294 | const loadBalancerTags = new PolicyStatement({ 295 | actions: [ 296 | 'elasticloadbalancing:AddTags', 297 | 'elasticloadbalancing:RemoveTags', 298 | ], 299 | resources: [ 300 | 'arn:aws:elasticloadbalancing:*:*:targetgroup/*/*', 301 | 'arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*', 302 | 'arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*', 303 | ], 304 | conditions: { 305 | Null: { 306 | 'aws:RequestTag/elbv2.k8s.aws/cluster': 'true', 307 | 'aws:ResourceTag/elbv2.k8s.aws/cluster': 'false', 308 | }, 309 | }, 310 | }); 311 | // Management of ListenerTags 312 | // TODO Scope using Tags RequestTags for AddTags 313 | // https://docs.aws.amazon.com/service-authorization/latest/reference/list_elasticloadbalancingv2.html#elasticloadbalancingv2-actions-as-permissions 314 | const loadBalancerListenersTags = new PolicyStatement({ 315 | actions: [ 316 | 'elasticloadbalancing:AddTags', 317 | 'elasticloadbalancing:RemoveTags', 318 | ], 319 | resources: [ 320 | 'arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*', 321 | 'arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*', 322 | 'arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*', 323 | 'arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*', 324 | ], 325 | }); 326 | // Management of LoadBalancer Targetgroup and Attributes, scoped to when at authorization time aws:ResourceTag/elbv2.k8s.aws/cluster is not null 327 | const modifyLoadBalancer = new PolicyStatement({ 328 | actions: [ 329 | 'elasticloadbalancing:ModifyLoadBalancerAttributes', 330 | 'elasticloadbalancing:SetIpAddressType', 331 | 'elasticloadbalancing:SetSecurityGroups', 332 | 'elasticloadbalancing:SetSubnets', 333 | 'elasticloadbalancing:ModifyTargetGroup', 334 | 'elasticloadbalancing:ModifyTargetGroupAttributes', 335 | ], 336 | resources: ['*'], 337 | conditions: { 338 | Null: { 339 | 'aws:ResourceTag/elbv2.k8s.aws/cluster': 'false', 340 | }, 341 | }, 342 | }); 343 | // Delete Load Balancer and TargetGroups to tag eks:cluster-name : MY_CLUSTER_NAME 344 | const deleteLoadBalancer = new PolicyStatement({ 345 | resources: ['*'], 346 | actions: [ 347 | 'elasticloadbalancing:DeleteTargetGroup', 348 | 'elasticloadbalancing:DeleteLoadBalancer', 349 | ], 350 | conditions: { 351 | StringEquals: { 352 | 'aws:ResourceTag/eks:cluster-name': clusterName, 353 | }, 354 | }, 355 | }); 356 | // Management of Target scoped to target-groups 357 | const registerTarget = new PolicyStatement({ 358 | actions: [ 359 | 'elasticloadbalancing:RegisterTargets', 360 | 'elasticloadbalancing:DeregisterTargets', 361 | ], 362 | resources: ['arn:aws:elasticloadbalancing:*:*:targetgroup/*/*'], 363 | }); 364 | // Management of LoadBalancer Certs, WebACL and Rules 365 | const modifyLoadBalancerCerts = new PolicyStatement({ 366 | actions: [ 367 | 'elasticloadbalancing:SetWebAcl', 368 | 'elasticloadbalancing:ModifyListener', 369 | 'elasticloadbalancing:AddListenerCertificates', 370 | 'elasticloadbalancing:RemoveListenerCertificates', 371 | 'elasticloadbalancing:ModifyRule', 372 | ], 373 | resources: ['*'], 374 | }); 375 | 376 | return new Policy(stack, 'albIngressPolicy', { 377 | roles: [roleSA], 378 | statements: [ 379 | modifyLoadBalancer, 380 | readPolicy, 381 | writeSG, 382 | createSG, 383 | readPolicyAdd, 384 | createTags, 385 | createdeleteTags, 386 | writeSGIngress, 387 | createLoadBalancer, 388 | loadBalancerTags, 389 | createLoadBalancerAdd, 390 | loadBalancerListenersTags, 391 | registerTarget, 392 | modifyLoadBalancer, 393 | modifyLoadBalancerCerts, 394 | deleteLoadBalancer, 395 | serviceLinkedRole, 396 | ], 397 | }); 398 | } 399 | -------------------------------------------------------------------------------- /lib/storageClass.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: storage.k8s.io/v1 2 | kind: StorageClass 3 | metadata: 4 | name: ebs-gp3-storageclass 5 | provisioner: ebs.csi.aws.com 6 | volumeBindingMode: WaitForFirstConsumer 7 | parameters: 8 | csi.storage.k8s.io/fstype: xfs 9 | type: gp3 10 | iops: "5700" 11 | encrypted: "true" 12 | throughput: "250" 13 | reclaimPolicy: Retain 14 | allowedTopologies: 15 | - matchLabelExpressions: 16 | - key: topology.ebs.csi.aws.com/zone 17 | values: 18 | - us-east-1a 19 | - us-east-1b 20 | - us-east-1c -------------------------------------------------------------------------------- /lib/vpc.ts: -------------------------------------------------------------------------------- 1 | import * as cdk from 'aws-cdk-lib'; 2 | import * as ec2 from 'aws-cdk-lib/aws-ec2'; 3 | import { Construct } from 'constructs'; 4 | 5 | export class VPC extends cdk.Stack { 6 | public readonly vpc: ec2.Vpc; 7 | 8 | constructor(scope: Construct, id: string, props: cdk.StackProps) { 9 | super(scope, id, props); 10 | 11 | const eksUnsupportedAzs: string[] = ['us-east-1e']; 12 | 13 | this.vpc = new ec2.Vpc(this, 'erigon', { 14 | ipAddresses: ec2.IpAddresses.cidr(this.node.tryGetContext('vpcCidr')), 15 | natGateways: 1, 16 | enableDnsSupport: true, 17 | enableDnsHostnames: true, 18 | availabilityZones: this.availabilityZones.filter( 19 | (az) => !eksUnsupportedAzs.includes(az) 20 | ), 21 | gatewayEndpoints: { 22 | S3: { 23 | service: ec2.GatewayVpcEndpointAwsService.S3, 24 | }, 25 | }, 26 | subnetConfiguration: [ 27 | { 28 | cidrMask: 20, 29 | name: 'eks-dmz', 30 | subnetType: ec2.SubnetType.PUBLIC, 31 | }, 32 | { 33 | cidrMask: 21, 34 | name: 'eks-cluster', 35 | subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS, 36 | }, 37 | { 38 | cidrMask: 20, 39 | name: 'eks-nodes', 40 | subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS, 41 | }, 42 | ], 43 | }); 44 | 45 | const ampInterfaceEndpointSG = new ec2.SecurityGroup( 46 | this, 47 | 'AMPInterfaceEndpointSG', 48 | { 49 | vpc: this.vpc, 50 | allowAllOutbound: true, 51 | description: 'AMP Interface Endpoint SG', 52 | // securityGroupName: `amp-interface-endpoint-${this.node.addr}`, 53 | } 54 | ); 55 | 56 | this.vpc 57 | .selectSubnets({ subnetGroupName: 'eks-nodes' }) 58 | .subnets.forEach((subnet) => { 59 | ampInterfaceEndpointSG.addIngressRule( 60 | ec2.Peer.ipv4(subnet.ipv4CidrBlock), 61 | ec2.Port.tcp(443), 62 | 'EKS Nodes' 63 | ); 64 | }); 65 | 66 | this.vpc.addInterfaceEndpoint('PrometheusEndpoint', { 67 | service: new ec2.InterfaceVpcEndpointService( 68 | `com.amazonaws.${this.region}.aps` 69 | ), 70 | securityGroups: [ampInterfaceEndpointSG], 71 | open: false, 72 | subnets: { 73 | subnetGroupName: 'eks-nodes', 74 | }, 75 | }); 76 | 77 | this.vpc.addInterfaceEndpoint('PrometheusWorkspacesEndpoint', { 78 | service: new ec2.InterfaceVpcEndpointService( 79 | `com.amazonaws.${this.region}.aps-workspaces` 80 | ), 81 | securityGroups: [ampInterfaceEndpointSG], 82 | open: false, 83 | subnets: { 84 | subnetGroupName: 'eks-nodes', 85 | }, 86 | }); 87 | 88 | this.vpc.addInterfaceEndpoint('GrafanaEndpoint', { 89 | service: new ec2.InterfaceVpcEndpointService( 90 | `com.amazonaws.${this.region}.grafana` 91 | ), 92 | securityGroups: [ampInterfaceEndpointSG], 93 | open: false, 94 | subnets: { 95 | subnetGroupName: 'eks-nodes', 96 | }, 97 | }); 98 | 99 | this.vpc.addInterfaceEndpoint('GrafanaWorkspacesEndpoint', { 100 | service: new ec2.InterfaceVpcEndpointService( 101 | `com.amazonaws.${this.region}.grafana-workspace` 102 | ), 103 | securityGroups: [ampInterfaceEndpointSG], 104 | open: false, 105 | subnets: { 106 | subnetGroupName: 'eks-nodes', 107 | }, 108 | }); 109 | } 110 | } 111 | -------------------------------------------------------------------------------- /manifests/consoleViewOnlyGroup.yaml: -------------------------------------------------------------------------------- 1 | # Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # From https://github.com/awsdocs/amazon-eks-user-guide/blob/master/doc_source/add-user-role.md 3 | # SPDX-License-Identifier: MIT-0 4 | 5 | apiVersion: rbac.authorization.k8s.io/v1 6 | kind: ClusterRole 7 | metadata: 8 | name: eks-console-dashboard-full-access-clusterrole 9 | rules: 10 | - apiGroups: 11 | - "" 12 | resources: 13 | - nodes 14 | - namespaces 15 | - pods 16 | verbs: 17 | - get 18 | - list 19 | - apiGroups: 20 | - apps 21 | resources: 22 | - deployments 23 | - daemonsets 24 | - statefulsets 25 | - replicasets 26 | verbs: 27 | - get 28 | - list 29 | - apiGroups: 30 | - batch 31 | resources: 32 | - jobs 33 | verbs: 34 | - get 35 | - list 36 | --- 37 | apiVersion: rbac.authorization.k8s.io/v1 38 | kind: ClusterRoleBinding 39 | metadata: 40 | name: eks-console-dashboard-full-access-binding 41 | subjects: 42 | - kind: Group 43 | name: eks-console-dashboard-full-access-group 44 | apiGroup: rbac.authorization.k8s.io 45 | roleRef: 46 | kind: ClusterRole 47 | name: eks-console-dashboard-full-access-clusterrole 48 | apiGroup: rbac.authorization.k8s.io 49 | -------------------------------------------------------------------------------- /manifests/fluentBitSetup.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ServiceAccount 3 | metadata: 4 | name: fluent-bit 5 | namespace: amazon-cloudwatch 6 | --- 7 | apiVersion: rbac.authorization.k8s.io/v1 8 | kind: ClusterRole 9 | metadata: 10 | name: fluent-bit-role 11 | rules: 12 | - nonResourceURLs: 13 | - /metrics 14 | verbs: 15 | - get 16 | - apiGroups: [""] 17 | resources: 18 | - namespaces 19 | - pods 20 | - pods/logs 21 | - nodes 22 | - nodes/proxy 23 | verbs: ["get", "list", "watch"] 24 | --- 25 | apiVersion: rbac.authorization.k8s.io/v1 26 | kind: ClusterRoleBinding 27 | metadata: 28 | name: fluent-bit-role-binding 29 | roleRef: 30 | apiGroup: rbac.authorization.k8s.io 31 | kind: ClusterRole 32 | name: fluent-bit-role 33 | subjects: 34 | - kind: ServiceAccount 35 | name: fluent-bit 36 | namespace: amazon-cloudwatch 37 | --- 38 | apiVersion: v1 39 | kind: ConfigMap 40 | metadata: 41 | name: fluent-bit-config 42 | namespace: amazon-cloudwatch 43 | labels: 44 | k8s-app: fluent-bit 45 | data: 46 | fluent-bit.conf: | 47 | [SERVICE] 48 | Flush 5 49 | Grace 30 50 | Log_Level info 51 | Daemon off 52 | Parsers_File parsers.conf 53 | HTTP_Server ${HTTP_SERVER} 54 | HTTP_Listen 0.0.0.0 55 | HTTP_Port ${HTTP_PORT} 56 | storage.path /var/fluent-bit/state/flb-storage/ 57 | storage.sync normal 58 | storage.checksum off 59 | storage.backlog.mem_limit 5M 60 | 61 | @INCLUDE application-log.conf 62 | @INCLUDE dataplane-log.conf 63 | @INCLUDE host-log.conf 64 | 65 | application-log.conf: | 66 | [INPUT] 67 | Name tail 68 | Tag application.* 69 | Exclude_Path /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy* 70 | Path /var/log/containers/*.log 71 | multiline.parser docker, cri 72 | DB /var/fluent-bit/state/flb_container.db 73 | Mem_Buf_Limit 50MB 74 | Skip_Long_Lines On 75 | Refresh_Interval 10 76 | Rotate_Wait 30 77 | storage.type filesystem 78 | Read_from_Head ${READ_FROM_HEAD} 79 | 80 | [INPUT] 81 | Name tail 82 | Tag application.* 83 | Path /var/log/containers/fluent-bit* 84 | multiline.parser docker, cri 85 | DB /var/fluent-bit/state/flb_log.db 86 | Mem_Buf_Limit 5MB 87 | Skip_Long_Lines On 88 | Refresh_Interval 10 89 | Read_from_Head ${READ_FROM_HEAD} 90 | 91 | [INPUT] 92 | Name tail 93 | Tag application.* 94 | Path /var/log/containers/cloudwatch-agent* 95 | multiline.parser docker, cri 96 | DB /var/fluent-bit/state/flb_cwagent.db 97 | Mem_Buf_Limit 5MB 98 | Skip_Long_Lines On 99 | Refresh_Interval 10 100 | Read_from_Head ${READ_FROM_HEAD} 101 | 102 | [FILTER] 103 | Name kubernetes 104 | Match application.* 105 | Kube_URL https://kubernetes.default.svc:443 106 | Kube_Tag_Prefix application.var.log.containers. 107 | Merge_Log On 108 | Merge_Log_Key log_processed 109 | K8S-Logging.Parser On 110 | K8S-Logging.Exclude Off 111 | Labels Off 112 | Annotations Off 113 | Use_Kubelet On 114 | Kubelet_Port 10250 115 | Buffer_Size 0 116 | 117 | [OUTPUT] 118 | Name cloudwatch_logs 119 | Match application.* 120 | region ${AWS_REGION} 121 | log_group_name /aws/containerinsights/${CLUSTER_NAME}/application 122 | log_stream_prefix ${HOST_NAME}- 123 | auto_create_group true 124 | extra_user_agent container-insights 125 | 126 | dataplane-log.conf: | 127 | [INPUT] 128 | Name systemd 129 | Tag dataplane.systemd.* 130 | Systemd_Filter _SYSTEMD_UNIT=docker.service 131 | Systemd_Filter _SYSTEMD_UNIT=containerd.service 132 | Systemd_Filter _SYSTEMD_UNIT=kubelet.service 133 | DB /var/fluent-bit/state/systemd.db 134 | Path /var/log/journal 135 | Read_From_Tail ${READ_FROM_TAIL} 136 | 137 | [INPUT] 138 | Name tail 139 | Tag dataplane.tail.* 140 | Path /var/log/containers/aws-node*, /var/log/containers/kube-proxy* 141 | multiline.parser docker, cri 142 | DB /var/fluent-bit/state/flb_dataplane_tail.db 143 | Mem_Buf_Limit 50MB 144 | Skip_Long_Lines On 145 | Refresh_Interval 10 146 | Rotate_Wait 30 147 | storage.type filesystem 148 | Read_from_Head ${READ_FROM_HEAD} 149 | 150 | [FILTER] 151 | Name modify 152 | Match dataplane.systemd.* 153 | Rename _HOSTNAME hostname 154 | Rename _SYSTEMD_UNIT systemd_unit 155 | Rename MESSAGE message 156 | Remove_regex ^((?!hostname|systemd_unit|message).)*$ 157 | 158 | [FILTER] 159 | Name aws 160 | Match dataplane.* 161 | imds_version v1 162 | 163 | [OUTPUT] 164 | Name cloudwatch_logs 165 | Match dataplane.* 166 | region ${AWS_REGION} 167 | log_group_name /aws/containerinsights/${CLUSTER_NAME}/dataplane 168 | log_stream_prefix ${HOST_NAME}- 169 | auto_create_group true 170 | extra_user_agent container-insights 171 | 172 | host-log.conf: | 173 | [INPUT] 174 | Name tail 175 | Tag host.dmesg 176 | Path /var/log/dmesg 177 | Key message 178 | DB /var/fluent-bit/state/flb_dmesg.db 179 | Mem_Buf_Limit 5MB 180 | Skip_Long_Lines On 181 | Refresh_Interval 10 182 | Read_from_Head ${READ_FROM_HEAD} 183 | 184 | [INPUT] 185 | Name tail 186 | Tag host.messages 187 | Path /var/log/messages 188 | Parser syslog 189 | DB /var/fluent-bit/state/flb_messages.db 190 | Mem_Buf_Limit 5MB 191 | Skip_Long_Lines On 192 | Refresh_Interval 10 193 | Read_from_Head ${READ_FROM_HEAD} 194 | 195 | [INPUT] 196 | Name tail 197 | Tag host.secure 198 | Path /var/log/secure 199 | Parser syslog 200 | DB /var/fluent-bit/state/flb_secure.db 201 | Mem_Buf_Limit 5MB 202 | Skip_Long_Lines On 203 | Refresh_Interval 10 204 | Read_from_Head ${READ_FROM_HEAD} 205 | 206 | [FILTER] 207 | Name aws 208 | Match host.* 209 | imds_version v1 210 | 211 | [OUTPUT] 212 | Name cloudwatch_logs 213 | Match host.* 214 | region ${AWS_REGION} 215 | log_group_name /aws/containerinsights/${CLUSTER_NAME}/host 216 | log_stream_prefix ${HOST_NAME}. 217 | auto_create_group true 218 | extra_user_agent container-insights 219 | 220 | parsers.conf: | 221 | [PARSER] 222 | Name syslog 223 | Format regex 224 | Regex ^(?