├── 10-demystifying.conf ├── Dockerfile ├── Makefile ├── README.md ├── cni-daemonset.png ├── cni-daemonset.yaml ├── cni-high-level.png ├── cni-step-1.png ├── cni-step-2.png ├── cni-step-3.png ├── cni-step-4.png ├── cni-step-5.png ├── cni-step-6.png ├── cni-step-7.png ├── cni-step-8.png ├── demystifying ├── entrypoint.sh ├── kind.yaml ├── node-with-two-container.png └── test.yaml /10-demystifying.conf: -------------------------------------------------------------------------------- 1 | { 2 | "cniVersion": "1.0.0", 3 | "name": "fromScratch", 4 | "type": "demystifying" 5 | } 6 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM alpine:3 2 | 3 | COPY 10-demystifying.conf /cni/10-demystifying.conf 4 | COPY demystifying /cni/demystifying 5 | COPY entrypoint.sh /cni/entrypoint.sh 6 | RUN chmod +x /cni/demystifying 7 | RUN chmod +x /cni/entrypoint.sh 8 | 9 | ENTRYPOINT ["/cni/entrypoint.sh"] 10 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | CLUSTER_NAME?=demystifying-cni 2 | 3 | .PHONY: cluster create init setup start up 4 | cluster create init setup start up: 5 | kind create cluster --config kind.yaml --name ${CLUSTER_NAME} 6 | kubectl delete deploy -n kube-system coredns 7 | kubectl delete deploy -n local-path-storage local-path-provisioner 8 | docker exec demystifying-cni-control-plane crictl pull httpd 9 | 10 | .PHONY: cni cp copy 11 | cni cp copy: 12 | docker cp 10-demystifying.conf demystifying-cni-control-plane:/etc/cni/net.d/10-demystifying.conf 13 | docker cp demystifying demystifying-cni-control-plane:/opt/cni/bin/demystifying 14 | docker exec demystifying-cni-control-plane chmod +x /opt/cni/bin/demystifying 15 | 16 | .PHONY: daemonset ds 17 | daemonset ds: 18 | docker build -t demystifying-cni:0.0.1 . 19 | kind load docker-image demystifying-cni:0.0.1 --name demystifying-cni 20 | kubectl apply -f cni-daemonset.yaml 21 | 22 | .PHONY: test 23 | test: 24 | kubectl apply -f test.yaml 25 | @sleep 5 26 | @echo "\n------\n" 27 | kubectl get pods -o wide 28 | @echo "\n------\n" 29 | docker exec demystifying-cni-control-plane curl -s 10.244.0.20 30 | 31 | .PHONY: clean clear 32 | clean clear: 33 | - kubectl delete -f test.yaml 34 | - docker exec demystifying-cni-control-plane rm /opt/cni/bin/demystifying 35 | - docker exec demystifying-cni-control-plane rm /etc/cni/net.d/10-demystifying.conf 36 | 37 | .PHONY: delete destroy down stop 38 | delete destroy down stop: 39 | kind delete cluster --name ${CLUSTER_NAME} 40 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Introduction 2 | 3 | The Cilium project recently became a graduated CNCF project and is the only graduated project in the CNCF Cloud Native Networking category. 4 | While Cilium can do many things - Ingress, Service Mesh, Observability, Encryption - its popularity initially soared as a pure CNI: a high-performance feature-rich Container Network plugin. 5 | However, we may actually take for granted what CNI actually means. 6 | In this repository, we will demistify what a CNI does and even build a CNI from scratch. 7 | By the end of this repository, you will have built your own homemade alternative to Cilium. 8 | 9 | ## What Is a Container Network Interface (CNI)? 10 | 11 | The Container Network Interface (CNI) is a CNCF project that specifies the relationship between a Container Runtime Interface (CRI), such as containerd, responsible for container creation, and a CNI plugin tasked with configuring network interfaces within the container upon execution. 12 | Ultimately, it's the CNI plugin that performs the substantive tasks of configuring the network, while CNI primarily denotes the interaction framework. 13 | However, it's common practice to simply refer to the CNI plugin as "CNI", a convention we'll adhere to in this repository. 14 | 15 | ## Container Networking Explained 16 | 17 | Containers do not possess their own kernel, instead they rely on the kernel of the host system on which they are running. 18 | This design choice renders containers more lightweight but less isolated compared to Virtual Machines (VMs). 19 | To provide some level of isolation, containers utilize a kernel feature known as namespaces. 20 | These namespaces allocate system resources, such as interfaces, to specific namespaces, preventing those resources from being visible in other namespaces of the same type. 21 | 22 | > Note that namespaces are referring to Linux namespaces, a Linux kernel feature and have nothing to do with Kubernetes namesapces. 23 | 24 | While containers consist of various namespaces, we will concentrate on the network namespace for the purposes of this repository. 25 | Typically each container has its own network namespace. 26 | This isolation ensures that interfaces outside of the container's namespace are not visible within the container's namespace and processes can bind to the same port without conflict. 27 | 28 | To facilitate networking, containers employ a specialized device known as a virtual ethernet device (veth). 29 | Veth devices are always created in interconnected pairs, ensuring that packets reaching one end are transmitted to the other, similar to two systems being linked via a cable. 30 | 31 | To enable communication between a container and the host system one of the veth interfaces resides within the container's network namespace, while the other resides within the host's network namespace. 32 | This configuration allows seamless communication between the container and the host system. 33 | As a result containers on the same node are able to communicate with each other through the host system. 34 | 35 | ![Node with two container](node-with-two-container.png) 36 | 37 | ## How does a CNI work? 38 | 39 | Picture a scenario where a user initiates the creation of a Pod and submits the request to the kube-apiserver. 40 | Following the scheduler's determination of the node where the Pod should be deployed, the kube-apiserver contacts the corresponding kubelet. 41 | The kubelet, rather than directly creating containers, delegates this task to a CRI. 42 | The CRI's responsibility encompasses container creation, including the establishment of a network namespace, as previously discussed. 43 | Once this setup is complete, the CRI calls upon a CNI plugin to generate and configure virtual ethernet devices and necessary routes. 44 | 45 | ![High level dependencies that invoke a CNI](cni-high-level.png) 46 | 47 | > Please note that a that CNIs typically do not handle traffic forwarding or load balancing. 48 | > By default, kube-proxy serves as the default network proxy in Kubernetes which utilizes technologies like iptables or IPVS to direct incoming network traffic to the relevant Pods within the cluster. 49 | > However, Cilium offers a superior alternative by loading eBPF programs directly into the kernel, achieving the same tasks with significantly higher speed. 50 | > For more information on this topic see "[What is Kube-Proxy and why move from iptables to eBPF?](https://isovalent.com/blog/post/why-replace-iptables-with-ebpf/)". 51 | 52 | ## Writing a CNI from scratch 53 | 54 | A common misconception suggests that Kubernetes-related components must be coded exclusively in Go. 55 | However, CNI dispels this notion by being language agnostic, focusing solely on defining the interaction between the CRI and a CNI plugin. 56 | The language used is inconsequential, what matters is that the plugin is executable. 57 | To demonstrate this flexibility, we'll develop a CNI plugin using bash. 58 | 59 | Before delving into the implementation, let's examine the steps in more detail: 60 | 61 | 1. Following the CRI's creation of a network namespace, it will load the first file located in `/etc/cni/net.d/`. Therefore, we'll generate a file named `/etc/cni/net.d/10-demystifying.conf`. This file must adhere to a specific JSON structure outlined in the CNI specification. The line `"type": "demystifying"` indicates the presence of an executable file named demystifying, which the CRI will execute in the next step. 62 | 63 | ![First step](cni-step-1.png) 64 | 65 | 2. The CRI will search in the directory `/opt/cni/bin/` and execute our CNI plugin, `demystifying`. For that reason we will create our bash script at `/opt/cni/bin/demystifying`. When the CRI invokes a CNI plugin, it passes data to the executable: the JSON retrieved from the previous step is conveyed via STDIN, while details about the container, including the relevant network namespace indicated by `CNI_NETNS`, are conveyed as environment variables. 66 | 67 | ![Second step](cni-step-2.png) 68 | 69 | 3. The first task our CNI plugin has to achieve is to create a virtual ethernet device. This action results in the creation of two veth interfaces, which we'll subsequently configure. One of the interfaces will be named `veth_netns` and the other one `veth_host` to make it easier to follow further steps. 70 | 71 | ![Third step](cni-step-3.png) 72 | 73 | 4. Next, we'll move one of the veth interfaces, `veth_netns`, into the container's network namespace. This allows for a connection between the container's network namespace and the host's network namespace. 74 | 75 | ![Fourth step](cni-step-4.png) 76 | 77 | 5. While the veth interfaces are automatically assigned MAC addresses, they lack an IP address. Typically, each node possesses a dedicated CIDR range, from which an IP address is selected. Assigning an IP to the veth interface inside the container network namespace is what is considered to be the Pod IP. For simplicity, we'll statically set `10.244.0.20` as the IP address and rename the interface based on the `CNI_IFNAME` environment variable. Keep in mind that Pod IPs must be unique in order to not create routing issues further down the line. In reality one would therefore keep track of all assigned IPs, a detail that we are skipping for simplicity reasons. 78 | 79 | ![Fifth step](cni-step-5.png) 80 | 81 | 6. The veth interface on the host will receive another IP address, serving as the default gateway within the container's network namespace.We'll statically assign `10.244.0.101` as the IP address. Irrespective of the number of Pods created on the node this IP can stay the same as its sole purpose is to serve as a destination for a route within the container's network namespace. 82 | 83 | ![Sixth step](cni-step-6.png) 84 | 85 | 7. Now it is time to add routes. Inside the container's network namespace, we need to specify that all traffic should be routed through `10.244.0.101`, directing it to the host. On the host side all traffic destined for `10.244.0.20` must be directed through `veth_host`. This configuration achieves bidirectional communication between the container and the host. 86 | 87 | ![Seventh step](cni-step-7.png) 88 | 89 | 8. Finally, we need to inform the CRI of our actions. To accomplish this, we'll print a JSON via STDOUT containing various details about the configuration performed, including the interfaces and IP addresses created. 90 | 91 | ![Eighth step](cni-step-8.png) 92 | 93 | Now it is time to incorporate the above steps. 94 | The easiest way to follow along is by using the kind setup provided in this repository. 95 | Create a test cluster by executing `make cluster`. 96 | 97 | As previously outlined we will now have to create two files on the node. 98 | 99 | The first one will be `/etc/cni/net.d/10-demystifying.conf`: 100 | ``` 101 | { 102 | "cniVersion": "1.0.0", 103 | "name": "fromScratch", 104 | "type": "demystifying" 105 | } 106 | ``` 107 | 108 | The second one being the executable CNI plugin `/opt/cni/bin/demystifying`: 109 | ``` 110 | #!/usr/bin/env bash 111 | 112 | # create veth 113 | VETH_HOST=veth_host 114 | VETH_NETNS=veth_netns 115 | ip link add ${VETH_HOST} type veth peer name ${VETH_NETNS} 116 | 117 | # put one of the veth interfaces into the new network namespace 118 | NETNS=$(basename ${CNI_NETNS}) 119 | ip link set ${VETH_NETNS} netns ${NETNS} 120 | 121 | # assign IP to veth interface inside the new network namespace 122 | IP_VETH_NETNS=10.244.0.20 123 | CIDR_VETH_NETNS=${IP_VETH_NETNS}/32 124 | ip -n ${NETNS} addr add ${CIDR_VETH_NETNS} dev ${VETH_NETNS} 125 | 126 | # assign IP to veth interface on the host 127 | IP_VETH_HOST=10.244.0.101 128 | CIDR_VETH_HOST=${IP_VETH_HOST}/32 129 | ip addr add ${CIDR_VETH_HOST} dev ${VETH_HOST} 130 | 131 | # rename veth interface inside the new network namespace 132 | ip -n ${NETNS} link set ${VETH_NETNS} name ${CNI_IFNAME} 133 | 134 | # ensure all interfaces are up 135 | ip link set ${VETH_HOST} up 136 | ip -n ${NETNS} link set ${CNI_IFNAME} up 137 | 138 | # add routes inside the new network namespace so that it knows how to get to the host 139 | ip -n ${NETNS} route add ${IP_VETH_HOST} dev eth0 140 | ip -n ${NETNS} route add default via ${IP_VETH_HOST} dev eth0 141 | 142 | # add route on the host to let it know how to reach the new network namespace 143 | ip route add ${IP_VETH_NETNS}/32 dev ${VETH_HOST} scope host 144 | 145 | # return a JSON via stdout 146 | RETURN_TEMPLATE=' 147 | { 148 | "cniVersion": "1.0.0", 149 | "interfaces": [ 150 | { 151 | "name": "%s", 152 | "mac": "%s" 153 | }, 154 | { 155 | "name": "%s", 156 | "mac": "%s", 157 | "sandbox": "%s" 158 | } 159 | ], 160 | "ips": [ 161 | { 162 | "address": "%s", 163 | "interface": 1 164 | } 165 | ] 166 | }' 167 | 168 | MAC_HOST_VETH=$(ip link show ${VETH_HOST} | grep link | awk '{print$2}') 169 | MAC_NETNS_VETH=$(ip -netns $nsname link show ${CNI_IFNAME} | grep link | awk '{print$2}') 170 | 171 | RETURN=$(printf "${RETURN_TEMPLATE}" "${VETH_HOST}" "${MAC_HOST_VETH}" "${CNI_IFNAME}" "${mac_netns_veth}" "${CNI_NETNS}" "${CIDR_VETH_NETNS}") 172 | echo ${RETURN} 173 | ``` 174 | 175 | As the CNI plugin must be executable, we'll need to modify the file mode using `chmod +x /opt/cni/bin/demystifying`. 176 | With that, we've constructed an operational CNI. 177 | 178 | These steps can be achieved by executing `make cni`. 179 | 180 | The next time a Pod is created on the node, it will follow the outlined steps, and our CNI will be invoked. 181 | Once a Pod is started we should see the following: 182 | ``` 183 | $ kubectl get pods -o wide 184 | NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 185 | best-app-ever 1/1 Running 0 11s 10.244.0.20 demystifying-cni-control-plane 186 | ``` 187 | 188 | As evident, the Pod is running as expected. 189 | Note that the Pod's IP address is `10.244.0.20`, as set by our CNI. 190 | With everything configured correctly, the node can successfully reach the Pod and receive a response: 191 | ``` 192 | $ curl 10.244.0.20 193 |

It works!

194 | ``` 195 | 196 | To perform such a test we can execute `make test`: 197 | ``` 198 | $ make test 199 | kubectl apply -f test.yaml 200 | pod/best-app-ever created 201 | 202 | ------ 203 | 204 | kubectl get pods -o wide 205 | NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 206 | best-app-ever 1/1 Running 0 6s 10.244.0.20 demystifying-cni-control-plane 207 | 208 | ------ 209 | 210 | docker exec demystifying-cni-control-plane curl -s 10.244.0.20 211 |

It works!

212 | ``` 213 | 214 | So far, this demonstrates the simplest deployment of a CNI. 215 | However, modern CNIs are usually deployed as Pods, enabling centralized management and dynamic scaling across cluster nodes. 216 | To illustrate how this can be done, let’s first remove our existing CNI by running `make clean`. 217 | 218 | Until now, the CNI setup requires manual steps on every node. 219 | This means that each time a new node is added, the necessary CNI files must be copied onto it manually. 220 | A more scalable approach is to use a Kubernetes DaemonSet, which ensures that each node automatically receives one Pod running the CNI. 221 | This Pod will then copy the required files into each node's appropriate directories, making sure the CNI is installed and ready to use. 222 | 223 | ![CNI running as DaemonSet](cni-daemonset.png) 224 | 225 | To deploy the CNI as a DaemonSet, we first need to create a Docker image containing the necessary files: 226 | ``` 227 | FROM alpine:3 228 | 229 | COPY 10-demystifying.conf /cni/10-demystifying.conf 230 | COPY demystifying /cni/demystifying 231 | COPY entrypoint.sh /cni/entrypoint.sh 232 | RUN chmod +x /cni/demystifying 233 | RUN chmod +x /cni/entrypoint.sh 234 | 235 | ENTRYPOINT [/cni/entrypoint.sh] 236 | ``` 237 | 238 | This Dockerfile creates an image with `10-demystifying.conf` and `demystifying` — the same files we used in our previous setup - alongside an `entrypoint.sh` script that will act as the container's entry point. 239 | 240 | The entrypoint.sh script executes the following commands when the container starts: 241 | ``` 242 | #!/usr/bin/env bash 243 | 244 | cp /cni/10-demystifying.conf /etc/cni/net.d/ 245 | cp /cni/demystifying /opt/cni/bin/ 246 | sleep infinity 247 | ``` 248 | 249 | These commands replicate the manual installation steps within the container, copying configuration and executable files to the appropriate directories on each node. 250 | 251 | Next, build the Docker image locally: 252 | ``` 253 | docker build -t demystifying-cni:0.0.1 . 254 | ``` 255 | 256 | In the real world the image would reside in a registry and the node would pull it from there. 257 | Since we built it locally we will have to load the image onto the kind cluster: 258 | ``` 259 | kind load docker-image demystifying-cni:0.0.1 --name demystifying-cni 260 | ``` 261 | 262 | With the image available on the cluster, we can deploy the DaemonSet: 263 | ``` 264 | kubectl apply -f cni-daemonset.yaml 265 | ``` 266 | 267 | To automate these steps, you can simply run `make daemonset`. 268 | 269 | Finally, test that everything works as expected using the DaemonSet architecture: 270 | ``` 271 | $ make test 272 | kubectl apply -f test.yaml 273 | pod/best-app-ever created 274 | 275 | ------ 276 | 277 | kubectl get pods -o wide 278 | NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 279 | best-app-ever 1/1 Running 0 5s 10.244.0.20 demystifying-cni-control-plane 280 | 281 | ------ 282 | 283 | docker exec demystifying-cni-control-plane curl -s 10.244.0.20 284 |

It works!

285 | ``` 286 | 287 | Keep in mind that this setup isn't suitable for production environments and comes with limitations as real-world scenarios require additional steps. 288 | For instance, assigning different IP addresses to the veth interface within the container network namespace and ensuring unique names for the veth interfaces on the host namespace are essential to support multiple Pods. 289 | Additionally, adding network configuration is only one aspect of the tasks a CNI plugin must support. 290 | These tasks are triggered by the CRI setting the `CNI_COMMAND` environment variable to `DEL` or `CHECK` respectively when invoking the CNI plugin. 291 | There are several other tasks, the specific requirements for full compatibility vary across versions and are outlined in the [CNI specification](https://github.com/containernetworking/cni/blob/main/SPEC.md). 292 | Nevertheless, the concepts outlined in this repository hold true regardless of version and offer valuable insights into the workings of a CNI. 293 | 294 | ## Summary 295 | 296 | At the heart of Kubernetes networking lies the Container Network Interface (CNI) specification which defines the exchange between the Container Runtime Interface (CRI) and the executable CNI plugin which resides on every node within the Kubernetes cluster. 297 | While the CRI establishes a container's network namespace, it is the CNI plugin's role to execute intricate network configurations. 298 | These configurations involve creating virtual ethernet interfaces and managing network settings, ensuring seamless connectivity both to and from the newly established container network namespace. 299 | 300 | Cilium is as an advanced networking solution adhering to the CNI specification and further elevating the capabilities of Kubernetes networking within complex environments. 301 | -------------------------------------------------------------------------------- /cni-daemonset.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/f1ko/demystifying-cni/5ae6cdd1253b592eeb3272780b387ecb3f5cf79f/cni-daemonset.png -------------------------------------------------------------------------------- /cni-daemonset.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: DaemonSet 3 | metadata: 4 | name: demystifying-cni 5 | namespace: kube-system 6 | spec: 7 | selector: 8 | matchLabels: 9 | app: demystifying-cni 10 | template: 11 | metadata: 12 | labels: 13 | app: demystifying-cni 14 | spec: 15 | # running in hostNetwork avoids dependency on CNI (solve chicken-egg problem) 16 | hostNetwork: true 17 | tolerations: 18 | # ensure to schedule on nodes that are not ready (nodes without a CNI are not ready) 19 | - key: node.kubernetes.io/not-ready 20 | operator: Exists 21 | effect: NoSchedule 22 | # ensure to schedule on control-plane nodes 23 | - key: node-role.kubernetes.io/control-plane 24 | operator: Exists 25 | effect: NoSchedule 26 | containers: 27 | - name: demystifying-cni 28 | image: demystifying-cni:0.0.1 29 | volumeMounts: 30 | - name: netd 31 | mountPath: /etc/cni/net.d 32 | - name: cnibin 33 | mountPath: /opt/cni/bin 34 | # use hostPath to get access to the node filesystem 35 | volumes: 36 | - name: netd 37 | hostPath: 38 | path: /etc/cni/net.d 39 | - name: cnibin 40 | hostPath: 41 | path: /opt/cni/bin 42 | -------------------------------------------------------------------------------- /cni-high-level.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/f1ko/demystifying-cni/5ae6cdd1253b592eeb3272780b387ecb3f5cf79f/cni-high-level.png -------------------------------------------------------------------------------- /cni-step-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/f1ko/demystifying-cni/5ae6cdd1253b592eeb3272780b387ecb3f5cf79f/cni-step-1.png -------------------------------------------------------------------------------- /cni-step-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/f1ko/demystifying-cni/5ae6cdd1253b592eeb3272780b387ecb3f5cf79f/cni-step-2.png -------------------------------------------------------------------------------- /cni-step-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/f1ko/demystifying-cni/5ae6cdd1253b592eeb3272780b387ecb3f5cf79f/cni-step-3.png -------------------------------------------------------------------------------- /cni-step-4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/f1ko/demystifying-cni/5ae6cdd1253b592eeb3272780b387ecb3f5cf79f/cni-step-4.png -------------------------------------------------------------------------------- /cni-step-5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/f1ko/demystifying-cni/5ae6cdd1253b592eeb3272780b387ecb3f5cf79f/cni-step-5.png -------------------------------------------------------------------------------- /cni-step-6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/f1ko/demystifying-cni/5ae6cdd1253b592eeb3272780b387ecb3f5cf79f/cni-step-6.png -------------------------------------------------------------------------------- /cni-step-7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/f1ko/demystifying-cni/5ae6cdd1253b592eeb3272780b387ecb3f5cf79f/cni-step-7.png -------------------------------------------------------------------------------- /cni-step-8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/f1ko/demystifying-cni/5ae6cdd1253b592eeb3272780b387ecb3f5cf79f/cni-step-8.png -------------------------------------------------------------------------------- /demystifying: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # create veth 4 | VETH_HOST=veth_host 5 | VETH_NETNS=veth_netns 6 | ip link add ${VETH_HOST} type veth peer name ${VETH_NETNS} 7 | 8 | # put one of the veth interfaces into the new network namespace 9 | NETNS=$(basename ${CNI_NETNS}) 10 | ip link set ${VETH_NETNS} netns ${NETNS} 11 | 12 | # assign IP to veth interface inside the new network namespace 13 | IP_VETH_NETNS=10.244.0.20 14 | CIDR_VETH_NETNS=${IP_VETH_NETNS}/32 15 | ip -n ${NETNS} addr add ${CIDR_VETH_NETNS} dev ${VETH_NETNS} 16 | 17 | # assign IP to veth interface on the host 18 | IP_VETH_HOST=10.244.0.101 19 | CIDR_VETH_HOST=${IP_VETH_HOST}/32 20 | ip addr add ${CIDR_VETH_HOST} dev ${VETH_HOST} 21 | 22 | # rename veth interface inside the new network namespace 23 | ip -n ${NETNS} link set ${VETH_NETNS} name ${CNI_IFNAME} 24 | 25 | # ensure all interfaces are up 26 | ip link set ${VETH_HOST} up 27 | ip -n ${NETNS} link set ${CNI_IFNAME} up 28 | 29 | # add routes inside the new network namespace so that it knows how to get to the host 30 | ip -n ${NETNS} route add ${IP_VETH_HOST} dev eth0 31 | ip -n ${NETNS} route add default via ${IP_VETH_HOST} dev eth0 32 | 33 | # add route on the host to let it know how to reach the new network namespace 34 | ip route add ${IP_VETH_NETNS}/32 dev ${VETH_HOST} scope host 35 | 36 | # return a JSON via stdout 37 | RETURN_TEMPLATE=' 38 | { 39 | "cniVersion": "1.0.0", 40 | "interfaces": [ 41 | { 42 | "name": "%s", 43 | "mac": "%s" 44 | }, 45 | { 46 | "name": "%s", 47 | "mac": "%s", 48 | "sandbox": "%s" 49 | } 50 | ], 51 | "ips": [ 52 | { 53 | "address": "%s", 54 | "interface": 1 55 | } 56 | ] 57 | }' 58 | 59 | MAC_HOST_VETH=$(ip link show ${VETH_HOST} | grep link | awk '{print$2}') 60 | MAC_NETNS_VETH=$(ip -netns $nsname link show ${CNI_IFNAME} | grep link | awk '{print$2}') 61 | 62 | RETURN=$(printf "${RETURN_TEMPLATE}" "${VETH_HOST}" "${MAC_HOST_VETH}" "${CNI_IFNAME}" "${mac_netns_veth}" "${CNI_NETNS}" "${CIDR_VETH_NETNS}") 63 | echo ${RETURN} -------------------------------------------------------------------------------- /entrypoint.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | cp /cni/10-demystifying.conf /etc/cni/net.d/ 4 | cp /cni/demystifying /opt/cni/bin/ 5 | sleep infinity 6 | -------------------------------------------------------------------------------- /kind.yaml: -------------------------------------------------------------------------------- 1 | kind: Cluster 2 | apiVersion: kind.x-k8s.io/v1alpha4 3 | networking: 4 | disableDefaultCNI: true 5 | nodes: 6 | - role: control-plane 7 | -------------------------------------------------------------------------------- /node-with-two-container.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/f1ko/demystifying-cni/5ae6cdd1253b592eeb3272780b387ecb3f5cf79f/node-with-two-container.png -------------------------------------------------------------------------------- /test.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | metadata: 4 | name: best-app-ever 5 | spec: 6 | containers: 7 | - name: httpd 8 | image: httpd 9 | --------------------------------------------------------------------------------