├── .gitignore ├── Dockerfile ├── LICENSE ├── Makefile ├── README.md ├── diagrams └── istio-hpa-overview.png ├── kube-metrics-adapter ├── custom-metrics-apiservice.yaml ├── deployment.yaml ├── external-metrics-apiservice.yaml ├── rbac.yaml └── service.yaml ├── loadtester ├── deployment.yaml └── service.yaml ├── namespaces └── test.yaml └── podinfo ├── deployment.yaml ├── hpa.yaml └── service.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | # Binaries for programs and plugins 2 | *.exe 3 | *.exe~ 4 | *.dll 5 | *.so 6 | *.dylib 7 | 8 | # Test binary, build with `go test -c` 9 | *.test 10 | 11 | # Output of the go coverage tool, specifically when used with LiteIDE 12 | *.out 13 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM golang:1.11 2 | 3 | RUN mkdir -p /zalando-incubator/ 4 | WORKDIR /zalando-incubator 5 | 6 | RUN git clone https://github.com/zalando-incubator/kube-metrics-adapter 7 | WORKDIR /zalando-incubator/kube-metrics-adapter 8 | 9 | RUN rm go.sum 10 | RUN go mod download 11 | RUN GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -a -installsuffix cgo -o kube-metrics-adapter . 12 | 13 | FROM alpine:3.8 14 | 15 | RUN apk --no-cache add ca-certificates 16 | 17 | COPY --from=0 /zalando-incubator/kube-metrics-adapter/kube-metrics-adapter . 18 | 19 | ENTRYPOINT ["./kube-metrics-adapter"] 20 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Stefan Prodan 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | TAG?=master-478c97d 2 | 3 | build: 4 | docker build -t stefanprodan/kube-metrics-adapter:$(TAG) . -f Dockerfile 5 | 6 | push: 7 | docker tag stefanprodan/kube-metrics-adapter:$(TAG) quay.io/stefanprodan/kube-metrics-adapter:$(TAG) 8 | docker push quay.io/stefanprodan/kube-metrics-adapter:$(TAG) 9 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # istio-hpa 2 | 3 | One of the advantages of using a service mesh like Istio is the builtin monitoring capability. You don't have 4 | to instrument your web apps in order to monitor the L7 traffic. The Istio telemetry service collects 5 | stats like HTTP request rate, response status codes and duration form the Envoy sidecars 6 | that are running alongside your apps. 7 | Besides monitoring these metrics can be used to drive autoscaling and 8 | [canary deployments](https://github.com/stefanprodan/flagger). 9 | 10 | ![Istio HPA](https://raw.githubusercontent.com/stefanprodan/istio-hpa/master/diagrams/istio-hpa-overview.png) 11 | 12 | What follows is a step-by-step guide on configuring HPA v2 with metrics provided by Istio Mixer. 13 | When installing Istio make sure that the telemetry service and Prometheus are enabled. 14 | If you're using the GKE Istio add-on, you'll have to deploy Prometheus as described 15 | [here](https://docs.flagger.app/install/flagger-install-on-google-cloud#install-prometheus). 16 | 17 | In order to use the Istio metrics together with the Horizontal Pod Autoscaler you'll need an adapter that 18 | can run Prometheus queries. Zalando made a general purpose metrics adapter for Kubernetes called 19 | [kube-metrics-adapter](https://github.com/zalando-incubator/kube-metrics-adapter). 20 | The Zalando adapter scans the HPA objects, executes promql queries (specified with annotations) and 21 | stores the metrics in memory. 22 | 23 | ### Installing the custom metrics adapter 24 | 25 | Clone the [istio-hpa](https://github.com/stefanprodan/istio-hpa) repository: 26 | 27 | ```bash 28 | git clone https://github.com/stefanprodan/istio-hpa 29 | cd istio-hpa 30 | ``` 31 | 32 | Deploy the metrics adapter in the `kube-system` namespace: 33 | 34 | ```bash 35 | kubectl apply -f ./kube-metrics-adapter/ 36 | ``` 37 | 38 | When the adapter starts, it will generate a self-signed cert and will register itself 39 | under the `custom.metrics.k8s.io` group. 40 | 41 | The adapter is configured to query the Prometheus instance that's running in the `istio-system` namespace. 42 | 43 | Verify the install by checking the adapter logs: 44 | 45 | ```bash 46 | kubectl -n kube-system logs deployment/kube-metrics-adapter 47 | ``` 48 | 49 | ### Installing the demo app 50 | 51 | You will use [podinfo](https://github.com/stefanprodan/k8s-podinfo), 52 | a small Golang-based web app to test the Horizontal Pod Autoscaler. 53 | 54 | First create a `test` namespace with Istio sidecar injection enabled: 55 | 56 | ```bash 57 | kubectl apply -f ./namespaces/ 58 | ``` 59 | 60 | Create the podinfo deployment and ClusterIP service in the `test` namespace: 61 | 62 | ```bash 63 | kubectl apply -f ./podinfo/deployment.yaml,./podinfo/service.yaml 64 | ``` 65 | 66 | In order to trigger the auto scaling, you'll need a tool to generate traffic. 67 | Deploy the load test service in the `test` namespace: 68 | 69 | ```bash 70 | kubectl apply -f ./loadtester/ 71 | ``` 72 | 73 | Verify the install by calling the podinfo API. 74 | Exec into the load tester pod and use `hey` to generate load for a couple of seconds: 75 | 76 | ```bash 77 | export loadtester=$(kubectl -n test get pod -l "app=loadtester" -o jsonpath='{.items[0].metadata.name}') 78 | kubectl -n test exec -it ${loadtester} -- sh 79 | 80 | ~ $ hey -z 5s -c 10 -q 2 http://podinfo.test:9898 81 | 82 | Summary: 83 | Total: 5.0138 secs 84 | Requests/sec: 19.9451 85 | 86 | Status code distribution: 87 | [200] 100 responses 88 | 89 | $ exit 90 | ``` 91 | 92 | The podinfo [ClusterIP service](https://github.com/stefanprodan/istio-hpa/blob/master/podinfo/service.yaml) 93 | exposes port 9898 under the `http` name. When using the http prefix, the Envoy sidecar will 94 | switch to L7 routing and the telemetry service will collect HTTP metrics. 95 | 96 | ### Querying the Istio metrics 97 | 98 | The Istio telemetry service collects metrics from the mesh and stores them in Prometheus. One such metric is 99 | `istio_requests_total`, with it you can determine the rate of requests per second a workload receives. 100 | 101 | This is how you can query Prometheus for the req/sec rate received by podinfo in the last minute, excluding 404s: 102 | 103 | ```sql 104 | sum( 105 | rate( 106 | istio_requests_total{ 107 | destination_workload="podinfo", 108 | destination_workload_namespace="test", 109 | reporter="destination", 110 | response_code!="404" 111 | }[1m] 112 | ) 113 | ) 114 | ``` 115 | 116 | The HPA needs to know the req/sec that each pod receives. You can use the container memory usage metric 117 | from kubelet to count the number of pods and calculate the Istio request rate per pod: 118 | 119 | ```sql 120 | sum( 121 | rate( 122 | istio_requests_total{ 123 | destination_workload="podinfo", 124 | destination_workload_namespace="test" 125 | }[1m] 126 | ) 127 | ) / 128 | count( 129 | count( 130 | container_memory_usage_bytes{ 131 | namespace="test", 132 | pod=~"podinfo.*" 133 | } 134 | ) by (pod) 135 | ) 136 | ``` 137 | 138 | ### Configuring the HPA with Istio metrics 139 | 140 | Using the req/sec query you can define a HPA that will scale the podinfo workload based on the number of requests 141 | per second that each instance receives: 142 | 143 | ```yaml 144 | apiVersion: autoscaling/v2beta1 145 | kind: HorizontalPodAutoscaler 146 | metadata: 147 | name: podinfo 148 | namespace: test 149 | annotations: 150 | metric-config.object.istio-requests-total.prometheus/per-replica: "true" 151 | metric-config.object.istio-requests-total.prometheus/query: | 152 | sum( 153 | rate( 154 | istio_requests_total{ 155 | destination_workload="podinfo", 156 | destination_workload_namespace="test" 157 | }[1m] 158 | ) 159 | ) / 160 | count( 161 | count( 162 | container_memory_usage_bytes{ 163 | namespace="test", 164 | pod=~"podinfo.*" 165 | } 166 | ) by (pod) 167 | ) 168 | spec: 169 | maxReplicas: 10 170 | minReplicas: 1 171 | scaleTargetRef: 172 | apiVersion: apps/v1 173 | kind: Deployment 174 | name: podinfo 175 | metrics: 176 | - type: Object 177 | object: 178 | metricName: istio-requests-total 179 | target: 180 | apiVersion: v1 181 | kind: Pod 182 | name: podinfo 183 | targetValue: 10 184 | ``` 185 | 186 | The above configuration will instruct the Horizontal Pod Autoscaler to scale up the deployment when the average traffic 187 | load goes over 10 req/sec per replica. 188 | 189 | Create the HPA with: 190 | 191 | ```bash 192 | kubectl apply -f ./podinfo/hpa.yaml 193 | ``` 194 | 195 | Start a load test and verify that the adapter computes the metric: 196 | 197 | ``` 198 | kubectl -n kube-system logs deployment/kube-metrics-adapter -f 199 | 200 | Collected 1 new metric(s) 201 | Collected new custom metric 'istio-requests-total' (44m) for Pod test/podinfo 202 | ``` 203 | 204 | List the custom metrics resources: 205 | 206 | ```bash 207 | kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . 208 | ``` 209 | 210 | The Kubernetes API should return a resource list containing the Istio metric: 211 | 212 | ```json 213 | { 214 | "kind": "APIResourceList", 215 | "apiVersion": "v1", 216 | "groupVersion": "custom.metrics.k8s.io/v1beta1", 217 | "resources": [ 218 | { 219 | "name": "pods/istio-requests-total", 220 | "singularName": "", 221 | "namespaced": true, 222 | "kind": "MetricValueList", 223 | "verbs": [ 224 | "get" 225 | ] 226 | } 227 | ] 228 | } 229 | ``` 230 | 231 | After a couple of seconds the HPA will fetch the metric from the adapter: 232 | 233 | ```bash 234 | kubectl -n test get hpa/podinfo 235 | 236 | NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS 237 | podinfo Deployment/podinfo 44m/10 1 10 1 238 | ``` 239 | 240 | ### Autoscaling based on HTTP traffic 241 | 242 | To test the HPA you can use the load tester to trigger a scale up event. 243 | 244 | Exec into the tester pod and use `hey` to generate load for a 5 minutes: 245 | 246 | ```bash 247 | kubectl -n test exec -it ${loadtester} -- sh 248 | 249 | ~ $ hey -z 5m -c 10 -q 2 http://podinfo.test:9898 250 | ``` 251 | Press ctrl+c then exit to get out of load test terminal if you wanna stop prematurely. 252 | 253 | After a minute the HPA will start to scale up the workload until the req/sec per pod drops under the target value: 254 | 255 | ```bash 256 | watch kubectl -n test get hpa/podinfo 257 | 258 | NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS 259 | podinfo Deployment/podinfo 25272m/10 1 10 3 260 | ``` 261 | 262 | When the load test finishes, the number of requests per second will drop to zero and the HPA will 263 | start to scale down the workload. 264 | Note that the HPA has a back off mechanism that prevents rapid scale up/down events, 265 | the number of replicas will go back to one after a couple of minutes. 266 | 267 | By default the metrics sync happens once every 30 seconds and scaling up/down can only happen if there was 268 | no rescaling within the last 3-5 minutes. In this way, the HPA prevents rapid execution of conflicting decisions 269 | and gives time for the Cluster Autoscaler to kick in. 270 | 271 | ### Wrapping up 272 | 273 | Scaling based on traffic is not something new to Kubernetes, an ingress controllers such as 274 | NGINX can expose Prometheus metrics for HPA. The difference in using Istio is that you can autoscale 275 | backend services as well, apps that are accessible only from inside the mesh. 276 | I'm not a big fan of embedding code in Kubernetes yaml but the Zalando metrics adapter is flexible enough 277 | to allow this kind of custom autoscaling. 278 | 279 | If you have any suggestion on improving this guide please submit an issue or PR on GitHub at 280 | [stefanprodan/istio-hpa](https://github.com/stefanprodan/istio-hpa). Contributions are more than welcome! 281 | -------------------------------------------------------------------------------- /diagrams/istio-hpa-overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/stefanprodan/istio-hpa/23a35d2b7345fee91d3a36f3b9886d0e3bbd5558/diagrams/istio-hpa-overview.png -------------------------------------------------------------------------------- /kube-metrics-adapter/custom-metrics-apiservice.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apiregistration.k8s.io/v1beta1 2 | kind: APIService 3 | metadata: 4 | name: v1beta1.custom.metrics.k8s.io 5 | spec: 6 | service: 7 | name: kube-metrics-adapter 8 | namespace: kube-system 9 | group: custom.metrics.k8s.io 10 | version: v1beta1 11 | insecureSkipTLSVerify: true 12 | groupPriorityMinimum: 100 13 | versionPriority: 100 14 | -------------------------------------------------------------------------------- /kube-metrics-adapter/deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: kube-metrics-adapter 5 | namespace: kube-system 6 | labels: 7 | app: kube-metrics-adapter 8 | spec: 9 | replicas: 1 10 | selector: 11 | matchLabels: 12 | app: kube-metrics-adapter 13 | template: 14 | metadata: 15 | labels: 16 | app: kube-metrics-adapter 17 | spec: 18 | serviceAccountName: custom-metrics-apiserver 19 | containers: 20 | - name: kube-metrics-adapter 21 | image: quay.io/stefanprodan/kube-metrics-adapter:master-478c97d 22 | args: 23 | - --prometheus-server=http://prometheus.istio-system.svc.cluster.local:9090 24 | resources: 25 | limits: 26 | cpu: 1000m 27 | memory: 1024Mi 28 | requests: 29 | cpu: 100m 30 | memory: 100Mi 31 | -------------------------------------------------------------------------------- /kube-metrics-adapter/external-metrics-apiservice.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apiregistration.k8s.io/v1beta1 2 | kind: APIService 3 | metadata: 4 | name: v1beta1.external.metrics.k8s.io 5 | spec: 6 | service: 7 | name: kube-metrics-adapter 8 | namespace: kube-system 9 | group: external.metrics.k8s.io 10 | version: v1beta1 11 | insecureSkipTLSVerify: true 12 | groupPriorityMinimum: 100 13 | versionPriority: 100 14 | -------------------------------------------------------------------------------- /kube-metrics-adapter/rbac.yaml: -------------------------------------------------------------------------------- 1 | kind: ServiceAccount 2 | apiVersion: v1 3 | metadata: 4 | name: custom-metrics-apiserver 5 | namespace: kube-system 6 | --- 7 | apiVersion: rbac.authorization.k8s.io/v1beta1 8 | kind: ClusterRole 9 | metadata: 10 | name: custom-metrics-server-resources 11 | rules: 12 | - apiGroups: 13 | - custom.metrics.k8s.io 14 | resources: ["*"] 15 | verbs: ["*"] 16 | --- 17 | apiVersion: rbac.authorization.k8s.io/v1beta1 18 | kind: ClusterRole 19 | metadata: 20 | name: external-metrics-server-resources 21 | rules: 22 | - apiGroups: 23 | - external.metrics.k8s.io 24 | resources: ["*"] 25 | verbs: ["*"] 26 | --- 27 | apiVersion: rbac.authorization.k8s.io/v1beta1 28 | kind: ClusterRole 29 | metadata: 30 | name: custom-metrics-resource-reader 31 | rules: 32 | - apiGroups: 33 | - "" 34 | resources: 35 | - namespaces 36 | - pods 37 | - services 38 | verbs: 39 | - get 40 | - list 41 | --- 42 | apiVersion: rbac.authorization.k8s.io/v1beta1 43 | kind: ClusterRole 44 | metadata: 45 | name: custom-metrics-resource-collector 46 | rules: 47 | - apiGroups: 48 | - "" 49 | resources: 50 | - events 51 | verbs: 52 | - create 53 | - patch 54 | - apiGroups: 55 | - "" 56 | resources: 57 | - pods 58 | verbs: 59 | - list 60 | - apiGroups: 61 | - apps 62 | resources: 63 | - deployments 64 | - statefulsets 65 | verbs: 66 | - get 67 | - apiGroups: 68 | - extensions 69 | resources: 70 | - ingresses 71 | verbs: 72 | - get 73 | - apiGroups: 74 | - autoscaling 75 | resources: 76 | - horizontalpodautoscalers 77 | verbs: 78 | - get 79 | - list 80 | - watch 81 | --- 82 | apiVersion: rbac.authorization.k8s.io/v1beta1 83 | kind: ClusterRoleBinding 84 | metadata: 85 | name: hpa-controller-custom-metrics 86 | roleRef: 87 | apiGroup: rbac.authorization.k8s.io 88 | kind: ClusterRole 89 | name: custom-metrics-server-resources 90 | subjects: 91 | - kind: ServiceAccount 92 | name: horizontal-pod-autoscaler 93 | namespace: kube-system 94 | --- 95 | apiVersion: rbac.authorization.k8s.io/v1beta1 96 | kind: ClusterRoleBinding 97 | metadata: 98 | name: hpa-controller-external-metrics 99 | roleRef: 100 | apiGroup: rbac.authorization.k8s.io 101 | kind: ClusterRole 102 | name: external-metrics-server-resources 103 | subjects: 104 | - kind: ServiceAccount 105 | name: horizontal-pod-autoscaler 106 | namespace: kube-system 107 | --- 108 | apiVersion: rbac.authorization.k8s.io/v1beta1 109 | kind: RoleBinding 110 | metadata: 111 | name: custom-metrics-auth-reader 112 | namespace: kube-system 113 | roleRef: 114 | apiGroup: rbac.authorization.k8s.io 115 | kind: Role 116 | name: extension-apiserver-authentication-reader 117 | subjects: 118 | - kind: ServiceAccount 119 | name: custom-metrics-apiserver 120 | namespace: kube-system 121 | --- 122 | apiVersion: rbac.authorization.k8s.io/v1beta1 123 | kind: ClusterRoleBinding 124 | metadata: 125 | name: custom-metrics:system:auth-delegator 126 | roleRef: 127 | apiGroup: rbac.authorization.k8s.io 128 | kind: ClusterRole 129 | name: system:auth-delegator 130 | subjects: 131 | - kind: ServiceAccount 132 | name: custom-metrics-apiserver 133 | namespace: kube-system 134 | --- 135 | apiVersion: rbac.authorization.k8s.io/v1beta1 136 | kind: ClusterRoleBinding 137 | metadata: 138 | name: custom-metrics-resource-collector 139 | roleRef: 140 | apiGroup: rbac.authorization.k8s.io 141 | kind: ClusterRole 142 | name: custom-metrics-resource-collector 143 | subjects: 144 | - kind: ServiceAccount 145 | name: custom-metrics-apiserver 146 | namespace: kube-system 147 | -------------------------------------------------------------------------------- /kube-metrics-adapter/service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: kube-metrics-adapter 5 | namespace: kube-system 6 | spec: 7 | ports: 8 | - port: 443 9 | targetPort: 443 10 | selector: 11 | app: kube-metrics-adapter 12 | -------------------------------------------------------------------------------- /loadtester/deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: loadtester 5 | namespace: test 6 | labels: 7 | app: loadtester 8 | spec: 9 | selector: 10 | matchLabels: 11 | app: loadtester 12 | template: 13 | metadata: 14 | labels: 15 | app: loadtester 16 | annotations: 17 | prometheus.io/scrape: "true" 18 | spec: 19 | containers: 20 | - name: loadtester 21 | image: quay.io/stefanprodan/flagger-loadtester:0.1.0 22 | imagePullPolicy: IfNotPresent 23 | ports: 24 | - name: http 25 | containerPort: 8080 26 | command: 27 | - ./loadtester 28 | - -port=8080 29 | - -log-level=info 30 | - -timeout=1h 31 | - -log-cmd-output=true 32 | livenessProbe: 33 | exec: 34 | command: 35 | - wget 36 | - --quiet 37 | - --tries=1 38 | - --timeout=4 39 | - --spider 40 | - http://localhost:8080/healthz 41 | timeoutSeconds: 5 42 | readinessProbe: 43 | exec: 44 | command: 45 | - wget 46 | - --quiet 47 | - --tries=1 48 | - --timeout=4 49 | - --spider 50 | - http://localhost:8080/healthz 51 | timeoutSeconds: 5 52 | resources: 53 | limits: 54 | memory: "512Mi" 55 | cpu: "1000m" 56 | requests: 57 | memory: "32Mi" 58 | cpu: "10m" 59 | securityContext: 60 | readOnlyRootFilesystem: true 61 | runAsUser: 10001 62 | -------------------------------------------------------------------------------- /loadtester/service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: loadtester 5 | namespace: test 6 | labels: 7 | app: loadtester 8 | spec: 9 | type: ClusterIP 10 | selector: 11 | app: loadtester 12 | ports: 13 | - name: http 14 | port: 80 15 | protocol: TCP 16 | targetPort: http -------------------------------------------------------------------------------- /namespaces/test.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Namespace 3 | metadata: 4 | name: test 5 | labels: 6 | istio-injection: enabled 7 | -------------------------------------------------------------------------------- /podinfo/deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: podinfo 5 | namespace: test 6 | labels: 7 | app: podinfo 8 | spec: 9 | minReadySeconds: 5 10 | strategy: 11 | rollingUpdate: 12 | maxUnavailable: 0 13 | type: RollingUpdate 14 | selector: 15 | matchLabels: 16 | app: podinfo 17 | template: 18 | metadata: 19 | annotations: 20 | prometheus.io/scrape: "true" 21 | labels: 22 | app: podinfo 23 | spec: 24 | containers: 25 | - name: podinfod 26 | image: quay.io/stefanprodan/podinfo:1.4.0 27 | imagePullPolicy: IfNotPresent 28 | ports: 29 | - containerPort: 9898 30 | name: http 31 | protocol: TCP 32 | command: 33 | - ./podinfo 34 | - --port=9898 35 | - --level=info 36 | livenessProbe: 37 | exec: 38 | command: 39 | - podcli 40 | - check 41 | - http 42 | - localhost:9898/healthz 43 | initialDelaySeconds: 5 44 | timeoutSeconds: 5 45 | readinessProbe: 46 | exec: 47 | command: 48 | - podcli 49 | - check 50 | - http 51 | - localhost:9898/readyz 52 | initialDelaySeconds: 5 53 | timeoutSeconds: 5 54 | resources: 55 | limits: 56 | cpu: 2000m 57 | memory: 512Mi 58 | requests: 59 | cpu: 100m 60 | memory: 64Mi 61 | -------------------------------------------------------------------------------- /podinfo/hpa.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: autoscaling/v2beta1 2 | kind: HorizontalPodAutoscaler 3 | metadata: 4 | name: podinfo 5 | namespace: test 6 | annotations: 7 | metric-config.object.istio-requests-total.prometheus/per-replica: "true" 8 | metric-config.object.istio-requests-total.prometheus/query: | 9 | sum( 10 | rate( 11 | istio_requests_total{ 12 | destination_workload="podinfo", 13 | destination_workload_namespace="test", 14 | reporter="destination" 15 | }[1m] 16 | ) 17 | ) / 18 | count( 19 | count( 20 | container_memory_usage_bytes{ 21 | namespace="test", 22 | pod=~"podinfo.*" 23 | } 24 | ) by (pod) 25 | ) 26 | spec: 27 | maxReplicas: 10 28 | minReplicas: 1 29 | scaleTargetRef: 30 | apiVersion: apps/v1 31 | kind: Deployment 32 | name: podinfo 33 | metrics: 34 | - type: Object 35 | object: 36 | metricName: istio-requests-total 37 | target: 38 | apiVersion: v1 39 | kind: Pod 40 | name: podinfo 41 | targetValue: 10 42 | -------------------------------------------------------------------------------- /podinfo/service.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: Service 4 | metadata: 5 | name: podinfo 6 | namespace: test 7 | labels: 8 | app: podinfo 9 | spec: 10 | type: ClusterIP 11 | ports: 12 | - name: http 13 | port: 9898 14 | targetPort: 9898 15 | protocol: TCP 16 | selector: 17 | app: podinfo 18 | --------------------------------------------------------------------------------