├── .github └── ISSUE_TEMPLATE │ └── bug_report.yaml ├── INSTALLATION.md ├── LICENSE ├── README.md ├── custom-envoy-listener ├── README.md └── envoy-prometheus-metrics-listener.yaml ├── kubernetes-ingress ├── README.md ├── basic-ingress.yaml ├── ca-issuer.yaml ├── grpc-ingress.yaml ├── grpc.md ├── http.md ├── tls-ingress.yaml ├── tls-with-cert-manager.md └── tls.md └── l7-traffic-management ├── README.md └── envoy-test.yaml /.github/ISSUE_TEMPLATE/bug_report.yaml: -------------------------------------------------------------------------------- 1 | name: Bug Report 2 | description: Report a bug encountered while using the Cilium Service Mesh features, currently in beta 3 | labels: ["kind/bug", "needs/triage"] 4 | body: 5 | - type: markdown 6 | attributes: 7 | value: | 8 | Thanks for taking the time to fill out this bug report! If you have usage questions, please try the [slack channel](http://slack.cilium.io/) and see the [FAQ](https://github.com/cilium/cilium/issues?utf8=%E2%9C%93&q=is:issue+label:kind/question+) first. 9 | 10 | Please note, this issue tracker is only for **Cilium Service Mesh** issues. Please report other Cilium issues in the main [Cilium repo](https://github.com/cilium/cilium/issues). 11 | 12 | **Important**: For security related issues: We strongly encourage you to report security vulnerabilities to the private security mailing list: security@cilium.io - first, before disclosing them in any public forums. 13 | - type: checkboxes 14 | attributes: 15 | label: Is there an existing issue for this? 16 | description: Please search to see if an issue already exists for the bug you encountered. 17 | options: 18 | - label: I have searched the existing issues 19 | required: true 20 | - type: textarea 21 | id: what-happened 22 | attributes: 23 | label: What happened? 24 | description: Also tell us, what did you expect to happen? 25 | placeholder: | 26 | 1. In this environment... 27 | 2. With this config... 28 | 3. Run '...' 29 | 4. See error... 30 | value: "A bug happened!" 31 | validations: 32 | required: true 33 | - type: textarea 34 | id: cilium-version 35 | attributes: 36 | label: Cilium Version 37 | description: What version of the software was running when you discovered this issue? (run `cilium version`) 38 | validations: 39 | required: true 40 | - type: textarea 41 | id: kernel-version 42 | attributes: 43 | label: Kernel Version 44 | description: Which kernel version was Cilium running on? (run `uname -a`) 45 | validations: 46 | required: true 47 | - type: textarea 48 | id: k8s-version 49 | attributes: 50 | label: Kubernetes Version 51 | description: Which Kubernetes version are you running? (run `kubectl version`) 52 | validations: 53 | required: true 54 | - type: textarea 55 | id: sysdump 56 | attributes: 57 | label: Sysdump 58 | description: | 59 | - [Install Cilium CLI](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli). 60 | - Run `cilium sysdump` command 61 | placeholder: | 62 | Upload the sysdump file here 63 | - type: textarea 64 | id: logs 65 | attributes: 66 | label: Relevant log output 67 | description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks. 68 | render: shell 69 | - type: textarea 70 | attributes: 71 | label: Anything else? 72 | description: | 73 | Links? References? Anything that will provide more context about the issue you are encountering! 74 | 75 | Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in. 76 | validations: 77 | required: false 78 | - type: checkboxes 79 | id: terms 80 | attributes: 81 | label: Code of Conduct 82 | description: By submitting this issue, you agree to follow our [Code of Conduct](https://github.com/cilium/cilium/blob/master/CODE_OF_CONDUCT.md) 83 | options: 84 | - label: I agree to follow this project's Code of Conduct 85 | required: true 86 | -------------------------------------------------------------------------------- /INSTALLATION.md: -------------------------------------------------------------------------------- 1 | # Installing Cilium Service Mesh 2 | 3 | These instructions should help you get the Cilium Service Mesh-specific images installed in your cluster, with the service mesh features enabled. 4 | 5 | ## Install Cilium and Hubble 6 | 7 | [Install the Cilium CLI](https://docs.cilium.io/en/v1.11/gettingstarted/k8s-install-default/#install-the-cilium-cli) and run `cilium version` to check that you have version 0.10.2 or above. 8 | 9 | Install Cilium with the service mesh builds, and enable Hubble: 10 | 11 | ``` 12 | cilium install --version v1.12.0-rc1 --helm-set enableIngressController=true --kube-proxy-replacement=probe 13 | cilium hubble enable 14 | ``` 15 | 16 | Check that Cilium is running correctly by running `cilium status`. You should see output like this. 17 | 18 | ``` 19 | ❯ cilium status 20 | /¯¯\ 21 | /¯¯\__/¯¯\ Cilium: OK 22 | \__/¯¯\__/ Operator: OK 23 | /¯¯\__/¯¯\ Hubble: OK 24 | \__/¯¯\__/ ClusterMesh: disabled 25 | \__/ 26 | 27 | Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1 28 | DaemonSet cilium Desired: 2, Ready: 2/2, Available: 2/2 29 | Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1 30 | Containers: cilium Running: 2 31 | cilium-operator Running: 1 32 | hubble-relay Running: 1 33 | Cluster Pods: 3/3 managed by Cilium 34 | Image versions cilium quay.io/cilium/cilium:v1.12.0-rc1: 2 35 | cilium-operator quay.io/cilium/operator-generic:v1.12.0-rc1: 1 36 | hubble-relay quay.io/cilium/hubble-relay:v1.12.0-rc1: 1 37 | ``` 38 | 39 | Confirm that the image versions for cilium operator, hubble-relay, and cilium are the [latest beta builds](https://github.com/cilium/cilium-service-mesh-beta#image-tags). 40 | 41 | ### Problems? 42 | 43 | If you see errors during installation, it would be helpful you can try uninstalling, then re-installing the regular release of Cilium to see whether it's a problem specific to the service mesh builds. 44 | 45 | ``` 46 | cilium uninstall 47 | cilium install 48 | cilium hubble enable 49 | ``` 50 | 51 | ## Use Hubble 52 | 53 | In order to use the `hubble` command to observe flows you will want to use port-forwarding: 54 | 55 | ``` 56 | cilium hubble port-forward & 57 | ``` 58 | 59 | [Install the Hubble CLI](https://docs.cilium.io/en/v1.11/gettingstarted/hubble_setup/#install-the-hubble-client), and then you can continuously observe flows in the cluster with commands such as: 60 | 61 | ``` 62 | hubble observe -f 63 | ``` 64 | 65 | Check out `hubble help observe` for command line options to filter the output down to what you are most interested in (e.g., `--to-service`). 66 | 67 | ## (Optional) Install Hubble UI 68 | 69 | If you'd like to visualize flows in the Hubble UI you'll need to install the hubble-ui components: 70 | 71 | ``` 72 | cilium hubble enable --ui 73 | ``` 74 | 75 | The images installed will be the standard images (there are no beta-specific builds for the UI components). 76 | 77 | ``` 78 | ❯ cilium status 79 | /¯¯\ 80 | /¯¯\__/¯¯\ Cilium: OK 81 | \__/¯¯\__/ Operator: OK 82 | /¯¯\__/¯¯\ Hubble: OK 83 | \__/¯¯\__/ ClusterMesh: disabled 84 | \__/ 85 | 86 | DaemonSet cilium Desired: 2, Ready: 2/2, Available: 2/2 87 | Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1 88 | Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1 89 | Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1 90 | Containers: cilium Running: 2 91 | cilium-operator Running: 1 92 | hubble-relay Running: 1 93 | hubble-ui Running: 1 94 | Cluster Pods: 4/4 managed by Cilium 95 | Image versions cilium quay.io/cilium/cilium:v1.12.0-rc1: 2 96 | cilium-operator quay.io/cilium/operator-generic:v1.12.0-rc1: 1 97 | hubble-relay quay.io/cilium/hubble-relay:v1.12.0-rc1: 1 98 | hubble-ui quay.io/cilium/hubble-ui:v0.8.5@sha256:4eaca1ec1741043cfba6066a165b3bf251590cf4ac66371c4f63fbed2224ebb4: 1 99 | hubble-ui quay.io/cilium/hubble-ui-backend:v0.8.5@sha256:2bce50cf6c32719d072706f7ceccad654bfa907b2745a496da99610776fe31ed: 1 100 | hubble-ui docker.io/envoyproxy/envoy:v1.20.2@sha256:eb7d88d5186648049f0a80062120bd45e7557bdff3f6a30e1fc92cbb50916868: 1 101 | ``` 102 | 103 | The following command will open the UI in your browser: 104 | 105 | ``` 106 | cilium hubble ui 107 | ``` 108 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Cilium Service Mesh Beta 2 | 3 | 4 | ## Moved 5 | The beta phase of Cilium Service Mesh has ended. Cilium service mesh is now available in the regular Cilium release (1.12) :tada: 6 | 7 | Please see: 8 | * GitHub: https://github.com/cilium/cilium 9 | * Documentation: https://docs.cilium.io/en/stable/gettingstarted/#service-mesh 10 | -------------------------------------------------------------------------------- /custom-envoy-listener/README.md: -------------------------------------------------------------------------------- 1 | # Example Custom Envoy Listener: Prometheus Metrics 2 | 3 | This example replicates the Prometheus metrics listener which is 4 | already available via Cilium Agent `--proxy-prometheus-port` command 5 | line option. So the point of this example is not to add new 6 | functionality, but to show how a feature that previously required 7 | Cilium Agent code changes can be implemented with the new Cilium Envoy 8 | Config CRD. 9 | 10 | 11 | ## Install Cilium and Hubble with Service Mesh Beta Support 12 | 13 | [Install Cilium to your test cluster](../INSTALLATION.md) and run 14 | `cilium version` to check that you have version 0.10.0 or above. 15 | 16 | 17 | ## Apply Example CRD 18 | 19 | This example adds a new Envoy listener 20 | `envoy-prometheus-metrics-listener` on the standards Prometheus port 21 | (`9090`) to each Cilium node, translating the default Prometheus 22 | metrics path `/metrics` to Envoy's Prometheus metrics path 23 | `/stats/prometheus`. 24 | 25 | Apply this Cilium Envoy Config CRD: 26 | 27 | ``` 28 | kubectl apply -f https://raw.githubusercontent.com/cilium/cilium-service-mesh-beta/main/custom-envoy-listener/envoy-prometheus-metrics-listener.yaml 29 | ``` 30 | 31 | Let's take a look at the Cilium Envoy Config CRD. It begins with this 32 | preamble: 33 | 34 | ``` 35 | apiVersion: cilium.io/v2alpha1 36 | kind: CiliumEnvoyConfig 37 | metadata: 38 | name: envoy-prometheus-metrics-listener 39 | ``` 40 | 41 | This specifies the API version as `cilium.io/v2alpha1`, meaning that 42 | this API is subject to change before it will be added to 43 | `cilium.io/v2`. 44 | 45 | `kind` spells out the new Cilium Envoy Config type (`CiliumEnvoyConfig`). 46 | 47 | The only metadata needed is the name for the resource (here 48 | `envoy-prometheus-metrics-listener`. This version of the CEC CRD is 49 | Cluster-scoped, (i.e., not namespaced), so the name needs to be unique 50 | in the cluster, unless you want to replace a CRD with a new one. 51 | 52 | This CEC CRD `spec` only contains two Envoy resources: 53 | ``` 54 | spec: 55 | resources: 56 | - "@type": type.googleapis.com/envoy.config.listener.v3.Listener 57 | name: envoy-prometheus-metrics-listener 58 | address: 59 | socket_address: 60 | address: "127.0.0.1" 61 | ipv4_compat: true 62 | port_value: 9090 63 | filter_chains: 64 | - filters: 65 | - name: envoy.filters.network.http_connection_manager 66 | typed_config: 67 | "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager 68 | stat_prefix: envoy-prometheus-metrics-listener 69 | rds: 70 | route_config_name: prometheus_route 71 | http_filters: 72 | - name: envoy.filters.http.router 73 | - "@type": type.googleapis.com/envoy.config.route.v3.RouteConfiguration 74 | name: prometheus_route 75 | virtual_hosts: 76 | - name: "prometheus_metrics_route" 77 | domains: ["*"] 78 | routes: 79 | - match: 80 | path: "/metrics" 81 | route: 82 | cluster: "envoy-admin" 83 | prefix_rewrite: "/stats/prometheus" 84 | ``` 85 | 86 | Note that these Envoy resources are not validated by k8s at all, so 87 | any errors in the Envoy resources will only be seen by the Cilium 88 | Agent observing these CRDs. This means that `kubectl apply` will 89 | report success, while parsing and/or installing the resources to the 90 | node-local Envoy instance may have failed. Currently the only way of 91 | verifying this is by observing Cilium Agent logs for errors and 92 | warnings: 93 | 94 | ``` 95 | kubectl logs -n kube-system cilium-xxxxx | grep -E "level=(error|warning)" 96 | ``` 97 | 98 | Please replace `cilium-xxxxx` with one of the Cilium Agent pod names 99 | you see in `kubectl get pods -A`. 100 | 101 | This example contains two Envoy resources, one `Listener` and one 102 | `RouteConfiguration`. Each of the resources needs to have a unique 103 | name. Envoy resource names for different resource types can be the 104 | same, but they are still separated by the resource type. 105 | 106 | Note that same Envoy resource names for a given resource type in 107 | different CEC CRDs will overwrite each other. So for now you need make 108 | sure the resource names are different in different CEC CRDs you may 109 | deploy. This limitation may be lifted in future versions e.g., by 110 | namespacing the resource names with the CEC CRD name. 111 | 112 | 113 | ## Supported Envoy API Versions 114 | 115 | As of now only the Envoy API v3 is supported. 116 | 117 | 118 | ## Supported Envoy Extension Resource Types 119 | 120 | Envoy extensions are resource types that may or may not be built in to 121 | an Envoy build. The standard types referred to in Envoy documetation, 122 | such as `type.googleapis.com/envoy.config.listener.v3.Listener`, and 123 | `type.googleapis.com/envoy.config.route.v3.RouteConfiguration`, are 124 | always available. 125 | 126 | Cilium nodes deploy an Envoy image to support Cilium HTTP policy 127 | enforcement and observability. This build of Envoy has been optimized 128 | for the needs of the Cilium Agent and does not contain many of the 129 | Envoy extensions available in the Envoy code base. 130 | 131 | To see which Envoy extensions are available, please have a look at 132 | [this Envoy extensions configuration 133 | file](https://github.com/cilium/proxy/blob/master/envoy_build_config/extensions_build_config.bzl). 134 | Only the extensions that have not been commented out with `#` are 135 | built in to the Cilium Envoy image. Currently this contains the 136 | following extensions: 137 | 138 | - `envoy.filters.http.router` 139 | - `envoy.filters.listener.tls_inspector` 140 | - `envoy.filters.network.http_connection_manager` 141 | - `envoy.filters.network.mongo_proxy` 142 | - `envoy.filters.network.mysql_proxy` 143 | - `envoy.filters.network.tcp_proxy` 144 | - `envoy.stat_sinks.metrics_service` 145 | - `envoy.transport_sockets.raw_buffer` 146 | 147 | We will evolve the list of built in extensions based on customer 148 | feedback for those extensions to be available to be used from CEC 149 | CRDs. 150 | 151 | ## Test the Listener Port 152 | 153 | Test that the new port is responding to the metrics requests: 154 | 155 | ``` 156 | curl http://:9090/metrics 157 | ``` 158 | 159 | Where `` is the IP address of one of your k8s cluster nodes. 160 | 161 | ## Clean-up 162 | 163 | Remove the prometheus listener with: 164 | 165 | ``` 166 | kubectl delete -f https://raw.githubusercontent.com/cilium/cilium-service-mesh-beta/main/custom-envoy-listener/envoy-prometheus-metrics-listener.yaml 167 | ``` 168 | -------------------------------------------------------------------------------- /custom-envoy-listener/envoy-prometheus-metrics-listener.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: cilium.io/v2alpha1 2 | kind: CiliumEnvoyConfig 3 | metadata: 4 | name: envoy-prometheus-metrics-listener 5 | spec: 6 | resources: 7 | - "@type": type.googleapis.com/envoy.config.listener.v3.Listener 8 | name: envoy-prometheus-metrics-listener 9 | address: 10 | socket_address: 11 | address: "::" 12 | ipv4_compat: true 13 | port_value: 9090 14 | filter_chains: 15 | - filters: 16 | - name: envoy.filters.network.http_connection_manager 17 | typed_config: 18 | "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager 19 | stat_prefix: envoy-prometheus-metrics-listener 20 | rds: 21 | route_config_name: prometheus_route 22 | http_filters: 23 | - name: envoy.filters.http.router 24 | - "@type": type.googleapis.com/envoy.config.route.v3.RouteConfiguration 25 | name: prometheus_route 26 | virtual_hosts: 27 | - name: "prometheus_metrics_route" 28 | domains: ["*"] 29 | routes: 30 | - match: 31 | path: "/metrics" 32 | route: 33 | cluster: "envoy-admin" 34 | prefix_rewrite: "/stats/prometheus" 35 | -------------------------------------------------------------------------------- /kubernetes-ingress/README.md: -------------------------------------------------------------------------------- 1 | # Kubernetes Ingress 2 | 3 | Cilium uses the standard Kubernetes Ingress resource definition, with an `ingressClassName` of `cilium`. This can be used for path-based routing and for TLS termination 4 | 5 | *Note: that the ingress controller creates a service of LoadBalancer type, so [your environment will need to support this](https://github.com/cilium/cilium-service-mesh-beta/issues/3).* 6 | 7 | Examples: 8 | 9 | * [HTTP](http.md) 10 | * [gRPC](grpc.md) 11 | * [TLS Termination](tls.md) 12 | * [TLS Termination using cert-manager](tls-with-cert-manager.md) 13 | -------------------------------------------------------------------------------- /kubernetes-ingress/basic-ingress.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: networking.k8s.io/v1 2 | kind: Ingress 3 | metadata: 4 | name: basic-ingress 5 | namespace: default 6 | spec: 7 | ingressClassName: cilium 8 | rules: 9 | - http: 10 | paths: 11 | - backend: 12 | service: 13 | name: details 14 | port: 15 | number: 9080 16 | path: /details 17 | pathType: Prefix 18 | - backend: 19 | service: 20 | name: productpage 21 | port: 22 | number: 9080 23 | path: / 24 | pathType: Prefix 25 | -------------------------------------------------------------------------------- /kubernetes-ingress/ca-issuer.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: cert-manager.io/v1 2 | kind: Issuer 3 | metadata: 4 | name: self-signed 5 | namespace: default 6 | spec: 7 | selfSigned: {} 8 | --- 9 | apiVersion: cert-manager.io/v1 10 | kind: Certificate 11 | metadata: 12 | name: ca 13 | namespace: default 14 | spec: 15 | isCA: true 16 | privateKey: 17 | algorithm: ECDSA 18 | size: 256 19 | secretName: ca 20 | commonName: ca 21 | issuerRef: 22 | name: self-signed 23 | kind: Issuer 24 | --- 25 | apiVersion: cert-manager.io/v1 26 | kind: Issuer 27 | metadata: 28 | name: ca-issuer 29 | namespace: default 30 | spec: 31 | ca: 32 | secretName: ca 33 | -------------------------------------------------------------------------------- /kubernetes-ingress/grpc-ingress.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: networking.k8s.io/v1 2 | kind: Ingress 3 | metadata: 4 | name: grpc-ingress 5 | namespace: default 6 | spec: 7 | ingressClassName: cilium 8 | rules: 9 | - http: 10 | paths: 11 | - backend: 12 | service: 13 | name: productcatalogservice 14 | port: 15 | number: 3550 16 | path: /hipstershop.ProductCatalogService 17 | pathType: Prefix 18 | - backend: 19 | service: 20 | name: currencyservice 21 | port: 22 | number: 7000 23 | path: /hipstershop.CurrencyService 24 | pathType: Prefix 25 | -------------------------------------------------------------------------------- /kubernetes-ingress/grpc.md: -------------------------------------------------------------------------------- 1 | # Ingress gRPC example 2 | 3 | The example ingress configuration in `grpc-ingress.yaml` shows how to route gRPC traffic to backend 4 | services. 5 | 6 | ## Install grpcurl 7 | 8 | To issue client gRPC requests you can use 9 | [grpcurl](https://github.com/fullstorydev/grpcurl#binaries). 10 | 11 | ## Deploy the demo app 12 | 13 | For this demo we will use [GCP's microservices demo app](https://github.com/GoogleCloudPlatform/microservices-demo). Install the app: 14 | 15 | ``` 16 | kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/master/release/kubernetes-manifests.yaml 17 | ``` 18 | Since gRPC is binary-encoded, you also need the proto definitions for the gRPC 19 | services in order to make gRPC requests. Download this for the demo app: 20 | 21 | ``` 22 | curl -o demo.proto https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/master/pb/demo.proto 23 | ``` 24 | 25 | ## Deploy the ingress 26 | 27 | You'll find the example Ingress definition in `grpc-ingress.yaml`. 28 | 29 | ``` 30 | kubectl apply -f grpc-ingress.yaml 31 | ``` 32 | 33 | This defines paths for requests to be routed to the `productcatalogservice` and 34 | `currencyservice` microservices. 35 | 36 | Just as in the [HTTP ingress demo](http.md) this creates a LoadBalancer 37 | service, and it may take a little while for your cloud provider to provision an 38 | external IP address. 39 | 40 | ``` 41 | ❯ kubectl get ingress 42 | NAME CLASS HOSTS ADDRESS PORTS AGE 43 | grpc-ingress cilium * 80 3d 44 | ``` 45 | 46 | Store that IP address in an environment variable `INGRESS_IP`. 47 | 48 | ## Make gRPC requests to backend services 49 | 50 | To access the currency service: 51 | 52 | ``` 53 | grpcurl -plaintext -proto ./demo.proto $INGRESS_IP:80 hipstershop.CurrencyService/GetSupportedCurrencies 54 | ``` 55 | 56 | To access the product catalog service: 57 | 58 | ``` 59 | grpcurl -plaintext -proto ./demo.proto $INGRESS_IP:80 hipstershop.ProductCatalogService/ListProducts 60 | ``` 61 | 62 | 63 | 64 | 65 | -------------------------------------------------------------------------------- /kubernetes-ingress/http.md: -------------------------------------------------------------------------------- 1 | # Ingress HTTP example 2 | 3 | The example ingress configuration routes traffic to backend services from the `bookinfo` demo microservices app from the Istio project. 4 | 5 | Deploy the demo app: 6 | 7 | ``` 8 | kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/platform/kube/bookinfo.yaml 9 | ``` 10 | 11 | *Note: this is just deploying the demo app, it's not adding any Istio components* 12 | 13 | You can confirm that with Cilium Service Mesh there is no Envoy sidecar created alongside each of the demo app microservices. 14 | 15 | ``` 16 | ❯ kubectl get pods 17 | NAME READY STATUS RESTARTS AGE 18 | details-v1-79f774bdb9-2cnzl 1/1 Running 0 84s 19 | productpage-v1-6b746f74dc-zm59b 1/1 Running 0 83s 20 | ratings-v1-b6994bb9-4zjr7 1/1 Running 0 83s 21 | reviews-v1-545db77b95-fg44w 1/1 Running 0 83s 22 | reviews-v2-7bf8c9648f-rlrmn 1/1 Running 0 83s 23 | reviews-v3-84779c7bbc-9b9tx 1/1 Running 0 83s 24 | ``` 25 | 26 | *Note: With the sidecar implementation the output would show 2/2 READY. One for the microservice and one for the Envoy sidecar.* 27 | 28 | ## Deploy the ingress 29 | 30 | You'll find the example Ingress definition in `basic-ingress.yaml`. 31 | 32 | ``` 33 | kubectl apply -f basic-ingress.yaml 34 | ``` 35 | 36 | This example routes requests for the path `/details` to the `details` service, and `/` to the `productpage` service. 37 | 38 | Getting the list of services, you'll see a LoadBalancer service is automatically created for this ingress. Your cloud provider will automatically provision an external IP address, but it may take around 30 seconds. 39 | 40 | ``` 41 | ❯ kubectl get svc 42 | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 43 | cilium-ingress-basic-ingress LoadBalancer 10.24.11.148 80:30543/TCP 19d 44 | details ClusterIP 10.24.7.97 9080/TCP 19d 45 | kubernetes ClusterIP 10.24.0.1 443/TCP 19d 46 | productpage ClusterIP 10.24.2.38 9080/TCP 19d 47 | ratings ClusterIP 10.24.11.9 9080/TCP 19d 48 | reviews ClusterIP 10.24.13.46 9080/TCP 19d 49 | ``` 50 | 51 | The external IP address should also be populated into the Ingress: 52 | 53 | ``` 54 | ❯ kubectl get ingress 55 | NAME CLASS HOSTS ADDRESS PORTS AGE 56 | basic-ingress cilium * 80 19d 57 | ``` 58 | 59 | *Note: some providers e.g. EKS use a fully-qualified domain name rather than an IP address. In this case, for now you will need to get the domain name from the service as it won't be populated into the Ingress. Connectivity shouldn't be affected. 60 | 61 | ## Make HTTP requests 62 | 63 | Check (with `curl` or in your browser) that you can make HTTP requests to that external address. The `/` path takes you to the home page for the bookinfo application. 64 | 65 | From outside the cluster you can also make requests directly to the `details` service using the path `/details`. But you can't directly access other URL paths that weren't defined in `basic-ingress.yaml`. For example, you can get JSON data from a request to `
/details/1` and get back some data, but you will get a 404 error if you make a request to `
/ratings`. 66 | 67 | -------------------------------------------------------------------------------- /kubernetes-ingress/tls-ingress.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: networking.k8s.io/v1 2 | kind: Ingress 3 | metadata: 4 | name: tls-ingress 5 | namespace: default 6 | spec: 7 | ingressClassName: cilium 8 | rules: 9 | - host: hipstershop.cilium.rocks 10 | http: 11 | paths: 12 | - backend: 13 | service: 14 | name: productcatalogservice 15 | port: 16 | number: 3550 17 | path: /hipstershop.ProductCatalogService 18 | pathType: Prefix 19 | - backend: 20 | service: 21 | name: currencyservice 22 | port: 23 | number: 7000 24 | path: /hipstershop.CurrencyService 25 | pathType: Prefix 26 | - host: bookinfo.cilium.rocks 27 | http: 28 | paths: 29 | - backend: 30 | service: 31 | name: details 32 | port: 33 | number: 9080 34 | path: /details 35 | pathType: Prefix 36 | - backend: 37 | service: 38 | name: productpage 39 | port: 40 | number: 9080 41 | path: / 42 | pathType: Prefix 43 | tls: 44 | - hosts: 45 | - bookinfo.cilium.rocks 46 | - hipstershop.cilium.rocks 47 | secretName: demo-cert 48 | -------------------------------------------------------------------------------- /kubernetes-ingress/tls-with-cert-manager.md: -------------------------------------------------------------------------------- 1 | # Ingress example with TLS termination using cert-manager 2 | 3 | This example builds on the [HTTP](http.md) and [gRPC](grpc.md) ingress 4 | examples, adding TLS termination. This example is similar to the 5 | [TLS](tls.md) ingress example, in which you had to manually create a 6 | self-signed CA using `minica`. In this example, you will be using 7 | [cert-manager](https://cert-manager.io/), a Kubernetes-native operator that 8 | issues and renews TLS certificates for you. 9 | 10 | ## Create TLS certificate and private key 11 | 12 | Let us install cert-manager: 13 | 14 | ```sh 15 | helm repo add jetstack https://charts.jetstack.io 16 | helm install cert-manager jetstack/cert-manager --version v1.7.1 --namespace cert-manager --set installCRDs=true --create-namespace 17 | ``` 18 | 19 | Now, create a CA Issuer: 20 | 21 | ```sh 22 | kubectl apply -f https://raw.githubusercontent.com/cilium/cilium-service-mesh-beta/main/kubernetes-ingress/ca-issuer.yaml 23 | ``` 24 | 25 | ## Deploy the ingress 26 | 27 | Follow the instructions from the [TLS demo](TLS.md) to deploy the ingress. 28 | 29 | These instructions will get you to install a LoadBalancer service, which after 30 | around 30 seconds or so should be populated with an external IP address. 31 | 32 | ```console 33 | $ kubectl get ingress 34 | NAME CLASS HOSTS ADDRESS PORTS AGE 35 | tls-ingress cilium hipstershop.cilium.rocks,bookinfo.cilium.rocks 35.195.24.75 80, 443 6m5s 36 | ``` 37 | 38 | To tell cert-manager that this Ingress needs a certificate, annotate the 39 | Ingress with the name of the CA issuer we previously created: 40 | 41 | ```sh 42 | kubectl annotate ingress tls-ingress cert-manager.io/issuer=ca-issuer 43 | ``` 44 | 45 | This creates a Certificate object along with a Secret containing the TLS 46 | certificate. 47 | 48 | ```console 49 | $ kubectl get certificate,secret demo-cert 50 | NAME READY SECRET AGE 51 | certificate.cert-manager.io/demo-cert True demo-cert 33m 52 | 53 | NAME TYPE DATA AGE 54 | secret/demo-cert kubernetes.io/tls 3 33m 55 | ``` 56 | 57 | _Note: if you previously followed the the basic [TLS example](TLS.md) there was a pre-existing secret called `demo-cert`. This will be properly "updated" by cert-manager but the updated secret won't be picked up by Cilium ([#22](https://github.com/cilium/cilium-service-mesh-beta/issues/22)). You can work around this by deleting and re-creating the service `cilium-ingress-tls-ingress` after checking that cert-manager has properly created the secret. One way to do this is to delete and re-create the ingress config file tls-ingress.yaml_ 58 | 59 | ## Edit /etc/hosts 60 | 61 | Follow the same instructions as in the basic [TLS demo](TLS.md) to edit your 62 | `/etc/hosts` file with the IP address assigned to the ingress. 63 | 64 | ## Make requests 65 | 66 | By specifying the CA's certificate on a curl request, you can say that you trust 67 | certificates signed by that CA. 68 | 69 | This is very similar to the basic TLS demo, except that the CA's secret needs to 70 | be retrieved from the Kubernetes secret where it's stored. 71 | 72 | ```sh 73 | curl --cacert <(kubectl get secret ca -o="jsonpath={.data.ca\.crt}" | base64 -d) \ 74 | -v https://bookinfo.cilium.rocks/details/1 75 | ``` 76 | 77 | If you prefer, instead of supplying the CA you can specify `-k` to tell the curl 78 | client not to validate the server's certificate. Without either, you will get an 79 | error that the certificate was signed by an unknown authority. 80 | 81 | Specifying `-v` on the curl request, you can see that the TLS handshake took 82 | place successfully. 83 | 84 | Similarly you can specify the CA on a gRPC request like this: 85 | 86 | ```sh 87 | grpcurl -proto ./demo.proto -cacert <(kubectl get secret ca -o="jsonpath={.data.ca\.crt}" | base64 -d) \ 88 | hipstershop.cilium.rocks:443 hipstershop.ProductCatalogService/ListProducts 89 | ``` 90 | 91 | (See the [gRPC](grpc.md) example if you don't already have the `demo.proto` file downloaded.) 92 | 93 | You can also visit in your browser. The browser 94 | will warn you that the certificate authority is unknown but if you proceed past 95 | this, you should see the bookstore application home page. 96 | 97 | Note that requests will time out if you don't specify `https://`. 98 | -------------------------------------------------------------------------------- /kubernetes-ingress/tls.md: -------------------------------------------------------------------------------- 1 | # Ingress example with TLS termination 2 | 3 | This example builds on the [HTTP](http.md) and [gRPC](grpc.md) ingress examples, adding TLS 4 | termination. 5 | 6 | ## Create TLS certificate and private key 7 | 8 | For demonstration purposes we will use a TLS certificate signed by a made-up, [self-signed][] 9 | certificate authority (CA). One easy way to do this is with 10 | [`minica`](https://github.com/jsha/minica). We want a certificate that will 11 | validate bookinfo.cilium.rocks and hipstershop.cilium.rocks, as 12 | these are the host names used in this ingress example. 13 | 14 | > If you prefer using [cert-manager][] to automate this part, you may want 15 | > to follow this other example: [Ingress example with TLS termination using 16 | > cert-manager](./tls-with-cert-manager.md). 17 | 18 | [self-signed]: https://cert-manager.io/docs/faq/terminology/#what-does-self-signed-mean-is-my-ca-self-signed 19 | [cert-manager]: https://cert-manager.io/ 20 | 21 | ``` 22 | minica -domains '*.cilium.rocks' 23 | ``` 24 | 25 | On first run, `minica` generates a CA certificate and key (`minica.pem` and 26 | `minica-key.pem`). It also creates a directory called `_.cilium.rocks` 27 | containing a key and certificate file that we will use for the ingress service. 28 | 29 | Create a Kubernetes secret with this demo key and certificate: 30 | 31 | ``` 32 | kubectl create secret tls demo-cert --key=_.cilium.rocks/key.pem --cert=_.cilium.rocks/cert.pem 33 | ``` 34 | 35 | ## Deploy the ingress 36 | 37 | If you previously ran the HTTP and/or gRPC ingress demos, delete the ingresses: 38 | 39 | ``` 40 | kubectl delete ingress basic-ingress 41 | kubectl delete ingress grpc-ingress 42 | ``` 43 | 44 | The ingress configuration for this demo provides the same routing as those demos 45 | but with the addition of TLS termination. Deploy this ingress: 46 | 47 | ``` 48 | kubectl apply -f tls-ingress.yaml 49 | ``` 50 | 51 | This creates a LoadBalancer service, which after around 30 seconds or so should 52 | be populated with an external IP address. 53 | 54 | ``` 55 | ❯ kubectl get ingress 56 | NAME CLASS HOSTS ADDRESS PORTS AGE 57 | tls-ingress cilium hipstershop.cilium.rocks,bookinfo.cilium.rocks 35.195.24.75 80, 443 6m5s 58 | ``` 59 | 60 | ## Edit /etc/hosts 61 | 62 | In this ingress configuration, the host names `hipstershop.cilium.rocks` and 63 | `bookinfo.cilium.rocks` are specified in the path routing rules. The client 64 | needs to specify which host it wants to access. This can be achieved by 65 | editing your local `/etc/hosts` file. (You will almost certainly need to be 66 | superuser to edit this file.) Add entries using the IP address 67 | assigned to the ingress service, so your file looks something like this: 68 | 69 | ``` 70 | ## 71 | # Host Database 72 | # 73 | # localhost is used to configure the loopback interface 74 | # when the system is booting. Do not change this entry. 75 | ## 76 | 127.0.0.1 localhost 77 | 255.255.255.255 broadcasthost 78 | ... 79 | 34.79.55.0 bookinfo.cilium.rocks 80 | 34.79.55.0 hipstershop.cilium.rocks 81 | ``` 82 | 83 | ## Make requests 84 | 85 | By specifying the CA's certificate on a curl request, you can say that you trust certificates 86 | signed by that CA. 87 | 88 | ``` 89 | curl --cacert minica.pem -v https://bookinfo.cilium.rocks/details/1 90 | ``` 91 | 92 | If you prefer, instead of supplying the CA you can specify `-k` to tell the curl client not to validate the 93 | server's certificate. Without either, you will get an error that the certificate 94 | was signed by an unknown authority. 95 | 96 | Specifying -v on the curl request, you can see that the TLS handshake took place successfully. 97 | 98 | Similarly you can specify the CA on a gRPC request like this: 99 | 100 | ``` 101 | grpcurl -proto ./demo.proto -cacert minica.pem hipstershop.cilium.rocks:443 hipstershop.ProductCatalogService/ListProducts 102 | ``` 103 | 104 | (See the [gRPC](grpc.md) example if you don't already have the `demo.proto` file downloaded.) 105 | 106 | You can also visit https://bookinfo.cilium.rocks in your browser. The browser 107 | will warn you that the certificate authority is unknown but if you proceed past 108 | this, you should see the bookstore application home page. 109 | 110 | Note that requests will time out if you don't specify `https://`. 111 | -------------------------------------------------------------------------------- /l7-traffic-management/README.md: -------------------------------------------------------------------------------- 1 | # Layer 7 Traffic Management 2 | 3 | Cilium Service Mesh defines a `CiliumEnvoyConfig` CRD which allows users to set the configuration of the Envoy component built into Cilium agents. 4 | 5 | This feature is enabled using the `--enable-envoy-config` feature flag. 6 | 7 | **_Note: there is currently an [issue](https://github.com/cilium/cilium-service-mesh-beta/issues/9) with Envoy traffic not being subjected to datapath processing properly in direct datapath mode. The workaround is to run Cilium in tunnelling mode, by installing with `--datapath-mode=vxlan` as an option on `cilium install`_** 8 | 9 | This example sets up an Envoy listener which load balances requests between two backend services. 10 | 11 | ## Deploy test applications 12 | 13 | You will need a Kubernetes cluster with at least two nodes for this example. Start by [installing Cilium to your test cluster](../INSTALLATION.md). 14 | 15 | Run `cilium connectivity test` to deploy the test applications used for L7 egress tests: 16 | 17 | ``` 18 | cilium connectivity test --test egress-l7 19 | ``` 20 | 21 | The test workloads run in the namespace `cilium-test` and consist of 22 | * two client deployments, `client` and `client2` 23 | * two backend services, `echo-other-node` and `echo-same-node` 24 | 25 | View information about these pods: 26 | 27 | ``` 28 | kubectl get pods -n cilium-test --show-labels -o wide 29 | ``` 30 | 31 | You can see that 32 | * Only `client2` is labelled with `other=client` - we will use this in a `CiliumNetworkPolicy` definition later in this example 33 | * The pods for `client`, `client2` and `echo-same-node` run on one node, while `echo-other-node` is scheduled to another node 34 | 35 | Make an environment variable with the pod ID for `client2`: 36 | 37 | ``` 38 | export CLIENT2= 39 | ``` 40 | 41 | We are going to use Envoy configuration to load-balance requests between these two backend services `echo-other-node` and `echo-same-node`. 42 | 43 | ### Start observing traffic with Hubble 44 | 45 | Start a second terminal, and observe traffic from the `client2` pod: 46 | 47 | ``` 48 | hubble observe --from-pod cilium-test/$CLIENT2 -f 49 | ``` 50 | 51 | You should be able to get a response from both of the backend services individually from `client2`: 52 | 53 | ``` 54 | kubectl exec -it -n cilium-test $CLIENT2 -- curl -v echo-same-node:8080/ 55 | kubectl exec -it -n cilium-test $CLIENT2 -- curl -v echo-other-node:8080/ 56 | ``` 57 | 58 | Notice that Hubble shows all the flows between these pods as being either `to/from-stack`, `to/from-overlay` or `to/from-endpoint` - there is no traffic marked as flowing to or from the proxy at this stage. (This assumes you don't already have any Layer 7 policies in place affecting this traffic.) 59 | 60 | Verify that you get a 404 error response if you curl to the non-existant URL `/foo` on these services: 61 | 62 | ``` 63 | kubectl exec -it -n cilium-test $CLIENT2 -- curl -v echo-same-node:8080/foo 64 | kubectl exec -it -n cilium-test $CLIENT2 -- curl -v echo-other-node:8080/foo 65 | ``` 66 | 67 | ### Add Layer 7 policy 68 | 69 | Adding Layer 7 policy introduces the Envoy proxy into the path for this traffic. 70 | 71 | ``` 72 | kubectl apply -f https://raw.githubusercontent.com/cilium/cilium-cli/master/connectivity/manifests/client-egress-l7-http.yaml 73 | kubectl apply -f https://raw.githubusercontent.com/cilium/cilium-cli/master/connectivity/manifests/client-egress-only-dns.yaml 74 | ``` 75 | 76 | Make a request to a backend service (either will do): 77 | 78 | ``` 79 | kubectl exec -it -n cilium-test $CLIENT2 -- curl -v echo-same-node:8080/ 80 | ``` 81 | 82 | Adding Layer 7 policy enables Layer 7 visibility. Notice that the Hubble output now includes flows `to-proxy`, and also shows the HTTP protocol information at level 7 (for example `(HTTP/1.1 GET http://echo-same-node:8080/)` 83 | 84 | ### Test Layer 7 policy enforcement 85 | 86 | The policy only permits GET requests to the `/` path, so you will see requests to any other URL being dropped. For example, try: 87 | 88 | ``` 89 | kubectl exec -it -n cilium-test $CLIENT2 -- curl -v echo-same-node:8080/foo 90 | ``` 91 | 92 | The Hubble output will show the HTTP request being dropped, like this: 93 | 94 | ``` 95 | Dec 15 15:10:17.639: cilium-test/client2-5998d566b4-566p5:51312 -> cilium-test/echo-same-node-745bd5c77-xggqw:8080 http-request DROPPED (HTTP/1.1 GET http://echo-same-node:8080/foo) 96 | ``` 97 | 98 | And the curl shoud show a 403 Forbidden response. 99 | 100 | ### Add Envoy load-balancing and URL re-writing 101 | 102 | Apply the `envoy-test.yaml` file, which defines a `CiliumEnvoyConfiguration` 103 | 104 | ``` 105 | kubectl apply -f envoy-test.yaml 106 | ``` 107 | 108 | This configuration listens for traffic intended for either of the two `echo-` services and 109 | * load-balances 50/50 between the two backend `echo-` services 110 | * rewrites the path `/foo` to `/` 111 | 112 | A request to `/foo` should now succeed, because of the path re-writing: 113 | 114 | ``` 115 | kubectl exec -it -n cilium-test $CLIENT2 -- curl -v echo-same-node:8080/foo 116 | ``` 117 | 118 | But the network policy still prevents requests to any path that is not rewritten to `/`. For example, this request will result in a packet being dropped and a `403 Forbidden` response code: 119 | 120 | ``` 121 | kubectl exec -it -n cilium-test $CLIENT2 -- curl -v echo-same-node:8080/bar 122 | ``` 123 | 124 | Try making several requests to one backend service, and you should see in the Hubble output that half the time, they are handled by the other backend. 125 | 126 | Example: 127 | 128 | ``` 129 | Dec 16 13:59:50.430: cilium-test/client2-5998d566b4-j6jms:37748 -> cilium-test/echo-same-node-745bd5c77-5xfmv:8080 http-request FORWARDED (HTTP/1.1 GET http://echo-same-node:8080/) 130 | ... 131 | Dec 16 13:59:53.584: cilium-test/client2-5998d566b4-j6jms:37822 -> cilium-test/echo-other-node-f4d46f75b-l5cgp:8080 http-request FORWARDED (HTTP/1.1 GET http://echo-same-node:8080/) 132 | ... 133 | Dec 16 14:00:13.122: cilium-test/client2-5998d566b4-j6jms:37982 -> cilium-test/echo-same-node-745bd5c77-5xfmv:8080 http-request FORWARDED (HTTP/1.1 GET http://echo-same-node:8080/) 134 | ... 135 | Dec 16 14:00:21.959: cilium-test/client2-5998d566b4-j6jms:37822 -> cilium-test/echo-other-node-f4d46f75b-l5cgp:8080 http-request FORWARDED (HTTP/1.1 GET http://echo-same-node:8080/) 136 | ``` 137 | -------------------------------------------------------------------------------- /l7-traffic-management/envoy-test.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: cilium.io/v2alpha1 2 | kind: CiliumEnvoyConfig 3 | metadata: 4 | name: envoy-lb-listener 5 | spec: 6 | services: 7 | - name: echo-other-node 8 | namespace: cilium-test 9 | - name: echo-same-node 10 | namespace: cilium-test 11 | resources: 12 | - "@type": type.googleapis.com/envoy.config.listener.v3.Listener 13 | name: envoy-lb-listener 14 | filter_chains: 15 | - filters: 16 | - name: envoy.filters.network.http_connection_manager 17 | typed_config: 18 | "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager 19 | stat_prefix: envoy-lb-listener 20 | rds: 21 | route_config_name: lb_route 22 | http_filters: 23 | - name: envoy.filters.http.router 24 | - "@type": type.googleapis.com/envoy.config.route.v3.RouteConfiguration 25 | name: lb_route 26 | virtual_hosts: 27 | - name: "lb_route" 28 | domains: ["*"] 29 | routes: 30 | - match: 31 | prefix: "/" 32 | route: 33 | weighted_clusters: 34 | clusters: 35 | - name: "cilium-test/echo-same-node" 36 | weight: 50 37 | - name: "cilium-test/echo-other-node" 38 | weight: 50 39 | retry_policy: 40 | retry_on: 5xx 41 | num_retries: 3 42 | per_try_timeout: 1s 43 | regex_rewrite: 44 | pattern: 45 | google_re2: {} 46 | regex: "^/foo.*$" 47 | substitution: "/" 48 | - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster 49 | name: "cilium-test/echo-same-node" 50 | connect_timeout: 5s 51 | lb_policy: ROUND_ROBIN 52 | type: EDS 53 | outlier_detection: 54 | split_external_local_origin_errors: true 55 | consecutive_local_origin_failure: 2 56 | - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster 57 | name: "cilium-test/echo-other-node" 58 | connect_timeout: 3s 59 | lb_policy: ROUND_ROBIN 60 | type: EDS 61 | outlier_detection: 62 | split_external_local_origin_errors: true 63 | consecutive_local_origin_failure: 2 64 | --------------------------------------------------------------------------------