├── .gitignore ├── LICENSE ├── README.md ├── README.old.md ├── containerd.md ├── scripts ├── buildPackage.sh ├── prfetch.sh └── run-e2e.sh └── vagrant ├── License.txt ├── Vagrantfile ├── go.sh ├── readme.md └── ubuntu.sh /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | *.sw[a-p] 3 | vagrant/.vagrant/ 4 | vagrant/*.exe 5 | vagrant/*.config 6 | scripts/_output 7 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Patrick Lang 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Guide to developing Kubernetes on Windows 2 | 3 | This repo is archived. If you'd like to see the old page, it's at [README.old.md](README.old.md). When I started this page, the steps needed to build and test Kubernetes and related container infrastructure were still in active development and changing frequently. During 2019, things stabilized, so it's probably better to go directly to each project to get the best documentation. 4 | 5 | Here's the main topics that I covered previously with links to the most current documentation on each topic. 6 | 7 | - How to set up a cluster including Kubernetes nodes 8 | - Using [Azure Kubernetes Service](https://docs.microsoft.com/en-us/azure/aks/windows-container-cli) for a Microsoft-supported cluster 9 | - Using [AKS-Engine](http://aka.ms/windowscontainers/kubernetes) for a self-supported or developent cluster 10 | - How to build Kubernetes binaries (kubelet, kube-proxy, kubectl) for Windows 11 | - The key thing you need to know is `make cross KUBE_BUILD_PLATFORMS=windows/amd64`. There are two ways to get a working development environment 12 | - [Using Docker](https://github.com/kubernetes/kubernetes/blob/master/build/README.md) (easiest & recommended) 13 | - [Using locally installed tools in Linux](https://github.com/kubernetes/community/blob/master/contributors/devel/development.md#building-kubernetes-on-a-local-osshell-environment) 14 | - How to run tests against a Windows Kubernetes node - https://github.com/kubernetes-sigs/windows-testing 15 | - How to [build ContainerD](https://github.com/containerd/containerd/blob/master/BUILDING.md). It's not obvious, but if you run `GOOS=windows make` it will build a Windows binary, even if you're building from a Linux machine. 16 | - How to build and test Linux binaries using a Windows machine - use [Kind](https://kind.sigs.k8s.io/docs/user/using-wsl2/) in WSL2 -------------------------------------------------------------------------------- /README.old.md: -------------------------------------------------------------------------------- 1 | # kubernetes-windows-dev 2 | Guide to developing Kubernetes on Windows 3 | 4 | 5 | 6 | - [How can I contribute to Kubernetes on Windows](#how-can-i-contribute-to-kubernetes-on-windows) 7 | - [Features and Bugs](#features-and-bugs) 8 | - [Testing](#testing) 9 | - [Documentation and samples](#documentation-and-samples) 10 | - [Required tools](#required-tools) 11 | - [Building a cluster](#building-a-cluster) 12 | - [Example aks-engine apimodel](#example-aks-engine-apimodel) 13 | - [Creating Windows pod deployments](#creating-windows-pod-deployments) 14 | - [Connecting to a Windows node](#connecting-to-a-windows-node) 15 | - [Simple method - Remote Desktop](#simple-method---remote-desktop) 16 | - [Scriptable method - SSH](#scriptable-method---ssh) 17 | - [Scriptable method - PowerShell Remoting](#scriptable-method---powershell-remoting) 18 | - [If WinRM isn't enabled](#if-winrm-isnt-enabled) 19 | - [Collecting Logs](#collecting-logs) 20 | - [Hacking AKS-Engine](#hacking-aks-engine) 21 | - [AKS-Engine Enlistment](#aks-engine-enlistment) 22 | - [AKS-Engine Build](#aks-engine-build) 23 | - [Hacking on Kubernetes for Windows](#hacking-on-kubernetes-for-windows) 24 | - [Setting up a dev/build environment](#setting-up-a-devbuild-environment) 25 | - [Kubernetes Enlistment for dev box](#kubernetes-enlistment-for-dev-box) 26 | - [Kubernetes Enlistment for build VM](#kubernetes-enlistment-for-build-vm) 27 | - [Kubernetes Build](#kubernetes-build) 28 | - [Copying files from the build VM](#copying-files-from-the-build-vm) 29 | - [Installing your build](#installing-your-build) 30 | - [Copying binaries using Azure Files](#copying-binaries-using-azure-files) 31 | - [Replacing files on the node](#replacing-files-on-the-node) 32 | - [Testing Kubernetes](#testing-kubernetes) 33 | - [Building Other Components](#building-other-components) 34 | - [Azure-CNI](#azure-cni) 35 | - [Using ContainerD](#using-containerd) 36 | - [Quick tips on Windows administration](#quick-tips-on-windows-administration) 37 | - [If you did this in bash, do this in PowerShell](#if-you-did-this-in-bash-do-this-in-powershell) 38 | - [Credits](#credits) 39 | 40 | 41 | 42 | ## How can I contribute to Kubernetes on Windows 43 | 44 | Windows has it's own Kubernetes special interest group (SIG). The weekly meeting schedule, agenda and past recordings are all on the [SIG-Windows community page](https://github.com/kubernetes/community/tree/master/sig-windows). 45 | 46 | 47 | This [GitHub project board](https://github.com/orgs/kubernetes/projects/8) shows the backlog and work-in progress for the current release. 48 | 49 | ### Features and Bugs 50 | 51 | Major features and bugs are tracked in [GitHub project board](https://github.com/orgs/kubernetes/projects/8), and updated in the weekly SIG-Windows meetings. If the items linked to the cards aren't assigned, feel free to assign them to yourself and get started hacking on them. For more up to date details, you can query for open issues & PRs with the [sig/windows label](https://github.com/kubernetes/kubernetes/labels/sig%2Fwindows) in kubernetes/kubernetes. 52 | 53 | 54 | ### Testing 55 | 56 | The current test results are available on [TestGrid]. These are run by Prow on a periodic basis, using the [SIG-Windows Job Definitions] 57 | 58 | If you want to learn more about the Kubernetes test infrastructure, it's best to start with the [test-infra docs] and this [intro from SIG-Testing]. 59 | 60 | ### Documentation and samples 61 | 62 | User documentation is kept in the [kubernetes/website] repo. The rendered version is at [docs.kubernetes.io]. Here's the [Intro to Windows in Kubernetes]. 63 | 64 | ## Required tools 65 | 66 | - [Git](https://git-scm.com/) 67 | - [Docker Community Edition](https://store.docker.com/search?type=edition&offering=community) 68 | - (optional but awesome) [Visual Studio Code](https://code.visualstudio.com/) 69 | - (for Windows 10 version 1709 or older) [Putty and Pageant](https://github.com/Azure/acs-engine/blob/master/docs/ssh.md#key-management-and-agent-forwarding-with-windows-pageant) for SSH. Windows 10 version 1803 and later have `ssh.exe` built-in which you may use instead. 70 | 71 | If you're using Windows, use "Git Bash" as your command-line environment for building. It can run the same bash scripts as on Linux & Mac, and will run the build containers using Docker for Windows. 72 | 73 | ## Building a cluster 74 | 75 | [AKS-Engine] is what I typically use to deploy clusters. There's a [AKS-Engine Windows Walkthrough] available that I recommend for your first deployment. The rest of this guide assumes you're using acs-engine, but the steps can easily be adapted to the other deployments. 76 | 77 | [Windows Kubernetes The Hard Way] - Peter Hornyack has a guide available showing how to set up everything needed for a mixed Linux + Windows cluster on Google Cloud Platform. 78 | 79 | [Intro to Windows in Kubernetes] - in the Kubernetes documentation also covers how to build an on-premises deployment 80 | 81 | ### Example aks-engine apimodel 82 | 83 | This apimodel.json is a great starting point. It includes two optional settings: 84 | 85 | - Enable winrm for remote management 86 | - Enable the alpha Hyper-V container support 87 | 88 | ```json 89 | { 90 | "apiVersion": "vlabs", 91 | "properties": { 92 | "orchestratorProfile": { 93 | "orchestratorType": "Kubernetes", 94 | "orchestratorVersion": "1.15", 95 | "kubernetesConfig": { 96 | "apiServerConfig" : { 97 | "--feature-gates": "HyperVContainer=true" 98 | }, 99 | "kubeletConfig" : { 100 | "--feature-gates": "HyperVContainer=true" 101 | } 102 | } 103 | }, 104 | "masterProfile": { 105 | "count": 1, 106 | "dnsPrefix": "azwink8s", 107 | "vmSize": "Standard_D2_v3" 108 | }, 109 | "agentPoolProfiles": [ 110 | { 111 | "name": "windowspool", 112 | "count": 2, 113 | "vmSize": "Standard_D2s_v3", 114 | "availabilityProfile": "AvailabilitySet", 115 | "osType": "Windows", 116 | "osDiskSizeGB": 127, 117 | "extensions": [ 118 | { 119 | "name": "winrm" 120 | } 121 | ] 122 | }, 123 | { 124 | "name": "linuxpool", 125 | "count": 2, 126 | "vmSize": "Standard_D2_v2", 127 | "availabilityProfile": "AvailabilitySet", 128 | "osType": "Linux" 129 | } 130 | ], 131 | "windowsProfile": { 132 | "adminUsername": "adminuser", 133 | "adminPassword": "", 134 | "sshEnabled": true 135 | }, 136 | "extensionProfiles": [ 137 | { 138 | "name": "winrm", 139 | "version": "v1" 140 | } 141 | ], 142 | "linuxProfile": { 143 | "adminUsername": "adminuser", 144 | "ssh": { 145 | "publicKeys": [ 146 | { 147 | "keyData": "" 148 | } 149 | ] 150 | } 151 | }, 152 | "servicePrincipalProfile": { 153 | "clientId": "", 154 | "secret": "" 155 | } 156 | } 157 | } 158 | ``` 159 | 160 | 161 | ## Creating Windows pod deployments 162 | 163 | Once you have a cluster up, go ahead and run your first deployment. This assumes you have a Windows Server 2019 (or 1809) node deployed. If you're using a different version, update the image with another tag for [microsoft/iis](https://hub.docker.com/r/microsoft/iis/) as needed. 164 | 165 | ```json 166 | apiVersion: apps/v1 167 | kind: Deployment 168 | metadata: 169 | name: iis-2019 170 | labels: 171 | app: iis-2019 172 | spec: 173 | replicas: 1 174 | template: 175 | metadata: 176 | name: iis-2019 177 | labels: 178 | app: iis-2019 179 | spec: 180 | containers: 181 | - name: iis 182 | image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019 183 | resources: 184 | limits: 185 | cpu: 1 186 | memory: 800m 187 | requests: 188 | cpu: .1 189 | memory: 300m 190 | ports: 191 | - containerPort: 80 192 | nodeSelector: 193 | "beta.kubernetes.io/os": windows 194 | selector: 195 | matchLabels: 196 | app: iis-2019 197 | --- 198 | apiVersion: v1 199 | kind: Service 200 | metadata: 201 | name: iis 202 | spec: 203 | type: LoadBalancer 204 | ports: 205 | - protocol: TCP 206 | port: 80 207 | selector: 208 | app: iis-2019 209 | ``` 210 | 211 | Here are some repos with more samples: 212 | 213 | - [patricklang/Windows-K8s-Samples](https://github.com/PatrickLang/Windows-K8s-Samples) 214 | 215 | 216 | ## Connecting to a Windows node 217 | 218 | This section assumes you deployed with [AKS-Engine]. If you deployed another way, some changes may be needed to access the Windows node. 219 | 220 | ### Simple method - Remote Desktop 221 | 222 | First, get the node's private IP with `kubectl get node` and `kubectl describe node`. Most likely it's in the range of `10.240.0.*` 223 | 224 | You can use SSH port forwarding with the Linux master node to forward a local port such as 5500 to the Windows node's private IP on port 3389. 225 | 226 | `ssh -L 5500::3389 user@linuxmaster.region.cloudapp.azure.com` 227 | 228 | > If you're on Windows 10 version 1709 or older, here's how to [use Putty & Pageant](https://github.com/Azure/acs-engine/blob/master/docs/ssh.md#key-management-and-agent-forwarding-with-windows-pageant) instead 229 | 230 | Once SSH is connected, use an RDP client to connect to `localhost:5500`. Use `mstsc.exe` on Windows, [FreeRDP](http://www.freerdp.com/) on Linux, or [Remote Desktop client](https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/remote-desktop-mac) for Mac. 231 | 232 | When you initially connect, there will be a command prompt open (if using Server Core), otherwise you can use the start menu to open a new PowerShell window. If you're on Server Core - run `start powershell` in that window to open a new PowerShell window. 233 | 234 | ### Scriptable method - SSH 235 | 236 | First, get the node's private IP with `kubectl get node` and `kubectl describe node`. Most likely it's in the range of `10.240.0.*` 237 | 238 | Next, SSH to the Linux master node, then to the node IP. 239 | 240 | ### Scriptable method - PowerShell Remoting 241 | 242 | First, get the node's private IP with `kubectl get node` and `kubectl describe node`. Most likely it's in the range of `10.240.0.*` 243 | 244 | Next, SSH to the Linux master node, and run `docker run -it mcr.microsoft.com/powershell`. Once that's running, use `Get-Credential` and `Enter-PSSession -Credential $cred -Authentication Basic -UseSSL` to connect to the Windows node. 245 | 246 | ```bash 247 | $ docker run -it mcr.microsoft.com/powershell 248 | PowerShell v6.0.2 249 | Copyright (c) Microsoft Corporation. All rights reserved. 250 | 251 | https://aka.ms/pscore6-docs 252 | Type 'help' to get help. 253 | 254 | PS /> $cred = Get-Credential 255 | 256 | PowerShell credential request 257 | Enter your credentials. 258 | User: azureuser 259 | Password for user azureuser: ************ 260 | 261 | PS /> Enter-PSSession 20143k8s9000 -Credential $cred -Authentication Basic -UseSSL 262 | [20143k8s9000]: PS C:\Users\azureuser\Documents> 263 | ``` 264 | 265 | #### If WinRM isn't enabled 266 | 267 | If WinRM wasn't enabled when you did the deployment with ACS-Engine, you can still enable it through the Azure portal. 268 | 269 | 1. Browse to the VM in the Azure portal 270 | 2. In the left pane, browse to Operations -> Run Command 271 | 3. Click "EnableRemotePS", then click Run. 272 | 273 | ## Collecting Logs 274 | 275 | There's a handy script that you can use to collect kubelet, kube-proxy, Docker and Windows HCS and HNS logs from every node all at once - [logslurp](https://github.com/patricklang/logslurp) 276 | 277 | ## Hacking AKS-Engine 278 | 279 | Work items for Windows are tracked in the [AKS-Engine] repo and tagged with Windows. 280 | 281 | ### AKS-Engine Enlistment 282 | 283 | 284 | 1. Fork the [AKS-Engine] repo in GitHub. 285 | 286 | 2. Clone your fork of the [AKS-Engine] repo 287 | 288 | ```bash 289 | # clone your fork as origin 290 | git clone https://github.com//aks-engine.git 291 | ``` 292 | 293 | 3. Set upstream to the main [AKS-Engine] repo. 294 | 295 | ```bash 296 | cd aks-engine 297 | # add upstream repo 298 | git remote add upstream https://github.com/Azure/aks-engine.git 299 | ``` 300 | 301 | > TODO: 4. (optional) Cherry-pick a PR that hasn't been merged. 302 | 303 | ```bash 304 | git remote add ... 305 | git fetch ... 306 | git cherry-pick ... 307 | ``` 308 | 309 | ### AKS-Engine Build 310 | 311 | The AKS-Engine dev environment works great in containers using Docker for Windows or [WSL2]. The repo has scripts to start it up: 312 | 313 | Windows: run `.\makedev.ps1` in PowerShell 314 | 315 | Mac / Linux: run `make dev` 316 | 317 | Once it starts up, just run `make` to build the `aks-engine` binary for Linux. It will be placed at `bin/aks-engine`. This binary works great in Windows using WSL, or you can copy it to a Linux VM. 318 | 319 | If you want to build Windows & Mac binaries, run `make build-cross`. The binaries will be in `_dist/aks-engine---` 320 | 321 | ```bash 322 | ls _dist/ 323 | aks-engine-ee33b2b-darwin-amd64 aks-engine-ee33b2b-linux-amd64 aks-engine-ee33b2b-windows-amd64 324 | ``` 325 | 326 | For more details, see the full [developers guide](https://github.com/Azure/acs-engine/blob/master/docs/developers.md) 327 | 328 | ## Hacking on Kubernetes for Windows 329 | 330 | Windows development is focused on the node (kubelet, kube-proxy), and client (kubectl). So far there hasn't been any push for running the other components (apiserver, controller-manager, scheduler, kube-dns) on Windows. Those components still run on Linux nodes. Most Windows features and bugfixes will not require any changes to the Linux components. 331 | 332 | > **Welcome contribution** - CoreDNS has builds available on Windows. It should work, but a setup guide and deployment template are needed. 333 | 334 | ### Setting up a dev/build environment 335 | 336 | Kubernetes cannot be built on Windows natively, or in a Linux container. For now, the best option is to set up a VM, pull sources, and build there. I typically edit code on my Windows machine with Visual Studio Code (and the Go extension for linting), push to my branch frequently, then pull it in the VM and build. Once I'm done with my changes, I squash the branch, clean up the history, and then submit a PR. 337 | 338 | 339 | 1. Install [Vagrant](https://vagrantup.com) 340 | 2. Copy the Vagrant folder from this repo to a drive with 60GB free 341 | 3. `cd` to that folder, then follow the steps in [vagrant/readme.md](vagrant/readme.md) to start it up. 342 | 343 | 344 | > **Welcome contribution** - update the Kubernetes Makefile to work with Docker for Windows 345 | >The build scripts themselves currently have blocks on OS 346 | > 347 | >```$ make -f build/root/Makefile cross 348 | >!!! [0917 11:44:01] Unsupported host OS. Must be Linux or Mac OS X. 349 | >!!! [0917 11:44:01] Call tree: 350 | >!!! [0917 11:44:01] 1: hack/make-rules/cross.sh:25 source(...) 351 | >make: *** [build/root/Makefile:482: cross] Error 1 352 | 353 | #### Kubernetes Enlistment for dev box 354 | 355 | Since you need to build on a Linux machine, I have found that it's easiest to push all changes to a private branch, pull that in the build VM, then build. 356 | 357 | I use a 3-deep branching strategy: 358 | 359 | 1. Fetch an upstream branch such as master or release-1.15 360 | 2. Branch off that to aggregate open PRs 361 | 3. Use a 3rd branch for my current changes 362 | 363 | As PRs are merged, I rebase all three in order. #1 will never have conflicts. If #2 has a conflict, the open PR will also have a conflict, so wait for it to be fixed there and do a new cherry-pick. #3 - I fix conflicts, squash and update my open PR if needed. 364 | 365 | Steps to create the branches: 366 | 367 | 1. Fork the Kubernetes repo in GitHub. 368 | 369 | 2. Clone your fork of the Kubernetes repo 370 | 371 | ```bash 372 | cd ~ 373 | mkdir -p go/src/k8s.io/ 374 | 375 | # clone your fork as origin 376 | cd ~/go/src/k8s.io 377 | git clone https://github.com//kubernetes.git 378 | ``` 379 | 380 | 3. Set upstream to the main Kubernetes repo. 381 | 382 | ```bash 383 | cd kubernetes 384 | # add upstream repo 385 | git remote add upstream https://github.com/kubernetes/kubernetes.git 386 | ``` 387 | 388 | 4. Fetch an upstream branch, and push it to your origin repo 389 | 390 | ```bash 391 | git fetch upstream release-1.15 392 | git checkout release-1.15 393 | git push -u origin release-1.15 394 | ``` 395 | 396 | 5. (optional) Cherry-pick a PR that hasn't been merged. 397 | 398 | > For an example of how to automate this see [scripts/prfetch.sh](scripts/prfetch.sh) 399 | 400 | Make a cherry-pick branch 401 | 402 | ```bash 403 | git checkout -b 1.15-cherrypick 404 | git push --set-upstream origin 1.15-cherrypick 405 | ``` 406 | 407 | This example uses https://github.com/kubernetes/kubernetes/pull/67435. It merged long ago, so be sure to pick another PR if you want to try this on your own. You need two pieces of information from the PR: 408 | 409 | - The originating user (feiskyer) and branch (dns-cap) 410 | - The commit IDs (0dc1ac03df89295c3bf5ddb7122270febe12eca2 and 3cb62394911261e3d8025d191a3ca80e6a712a67) 411 | 412 | Create a new remote with their username, fetch their branch, cherry-pick the commits, then push to your cherrypick branch. 413 | 414 | ```bash 415 | git remote add feiskyer https://github.com/feiskyer/kubernetes.git 416 | git fetch feiskyer dns-cap 417 | git cherry-pick 0dc1ac03df89295c3bf5ddb7122270febe12eca2 418 | git cherry-pick 3cb62394911261e3d8025d191a3ca80e6a712a67 419 | git push 420 | ``` 421 | 422 | 423 | #### Kubernetes Enlistment for build VM 424 | 425 | You only need to set up a remote to the branch you're building. It's easist to use https, which will avoid needing to log into Git since you only need read access for your public branch. 426 | 427 | ```bash 428 | mkdir -p ~/go/src/k8s.io/ 429 | 430 | # clone your fork as origin 431 | cd ~/go/src/k8s.io 432 | git clone https://github.com//kubernetes.git 433 | cd kubernetes 434 | 435 | # change to your working branch 436 | git fetch origin mybugfix 437 | git checkout mybugfix 438 | ``` 439 | 440 | 441 | ### Kubernetes Build 442 | 443 | Connect to the build VM with `vagrant ssh`. If you haven't already started a `tmux` session, run `tmux` to start one. It will hold the scrollback buffer, and let you disconnect and reconnect if needed without stopping a build. You can detach with _Ctrl-B, d_, and reattach later using `tmux list-sessions` to get the number, then `tmux attach-session -t #` to reconnect to it. 444 | 445 | 446 | From the build VM: 447 | 448 | ```bash 449 | cd ~/go/src/k8s.io/kubernetes 450 | ./build/run.sh make cross KUBE_BUILD_PLATFORMS=windows/amd64 451 | ``` 452 | 453 | It will scroll a lot as the API files are scanned, then eventually start building. Each build target has output similar to this accompanied with a few minutes of waiting. 454 | 455 | ```none 456 | I0917 21:53:22.103338 15972 main.go:75] Completed successfully. 457 | No changes in generated bindata file: test/e2e/generated/bindata.go 458 | No changes in generated bindata file: pkg/generated/bindata.go 459 | Go version: go version go1.10.3 linux/amd64 460 | +++ [0917 21:53:22] Building go targets for windows/amd64: 461 | cmd/kube-proxy 462 | cmd/kubeadm 463 | cmd/kubelet 464 | Env for windows/amd64: GOOS=windows GOARCH=amd64 GOROOT=/usr/local/go CGO_ENABLED= CC= 465 | Coverage is disabled. 466 | ``` 467 | 468 | After the last binary is built, it will copy binaries out of the container into the VM: 469 | 470 | ```none 471 | +++ [0917 21:56:36] Placing binaries 472 | +++ [0917 21:56:58] Syncing out of container 473 | +++ [0917 21:56:58] Stopping any currently running rsyncd container 474 | +++ [0917 21:56:59] Starting rsyncd container 475 | +++ [0917 21:57:00] Running rsync 476 | +++ [0917 21:57:22] Stopping any currently running rsyncd container 477 | ``` 478 | 479 | Run `ls _output/dockerized/bin/windows/amd64/` to see what was built. 480 | 481 | For more details on building, check out the [Building Kubernetes](https://github.com/kubernetes/kubernetes/blob/master/build/README.md) 482 | 483 | #### Copying files from the build VM 484 | 485 | Now, it's time to use SCP to copy the binaries out. 486 | 487 | 1. Get the SSH config with `vagrant ssh-config | Out-File -Encoding ascii k8s-dev.ssh.config` 488 | 2. Copy the files out with SCP `scp -F .\k8s-dev.ssh.config k8s-dev:~/go/src/k8s.io/kubernetes/_output/dockerized/bin/windows/amd64/* .` 489 | 490 | ### Installing your build 491 | 492 | The Windows node binaries (typically in `c:\k`) can easily be replaced for testing or upgrades. The general process is: 493 | 494 | 1. Copy the new binaries to an easily accessible location - Azure Files or Google Cloud Storage work great. 495 | 2. `kubectl drain ` 496 | 3. Connect to the Windows node using PowerShell remoting or Remote Desktop 497 | 4. Run `net stop kubelet` on the Windows node 498 | 5. Replace `kubelet.exe` and `kube-proxy.exe` 499 | 6. Run `net start kubelet` on the Windows node 500 | 7. `kubectl uncordon ` 501 | 502 | 503 | #### Copying binaries using Azure Files 504 | 505 | 1. Create a storage account [full steps](https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?toc=%2Fazure%2Fstorage%2Ffiles%2Ftoc.json&tabs=portal) 506 | 2. Create a file share [full steps](https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-create-file-share) 507 | 3. Mount it on [windows](https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-use-files-windows), [mac](https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-use-files-mac), or [linux](https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-use-files-linux) 508 | 4. Copy `kubelet.exe` and `kube-proxy.exe` to it 509 | 5. From the Azure Portal, browse to the Azure File Share, and click "Connect". It will open a new pane including a command to mount the share on Windows. Save that path for later. 510 | 511 | 512 | #### Replacing files on the node 513 | 514 | First, connect with Remote Desktop or PowerShell remoting - see [Connecting to a Windows node](#connecting-to-a-windows-node) 515 | 516 | Find or open a new PowerShell window, and then paste in that mount command the Azure Portal from step 5 above. It will be something like `net use z: \\myazurefileaccount.file.core.windows.net\myazurefiles /u:AZURE\myazurefileaccount .......` 517 | 518 | Now, stop the `kubelet` service, copy the files over the old ones, and restart it. 519 | 520 | ```powershell 521 | net stop kubelet 522 | cd \k 523 | copy z:\kubelet.exe . 524 | copy z:\kube-proxy.exe . 525 | net start kubelet 526 | ``` 527 | 528 | Now, you can `kubectl uncordon` the node and run pods on it again. 529 | 530 | ## Testing Kubernetes 531 | 532 | For steps to build the Kubernetes E2E tests and run them, see [kubernetes-sigs/windows-testing] . 533 | 534 | [scripts/run-e2e.sh](./scripts/run-e2e.sh) is a convenience script that can make building & running tests a bit easier. 535 | 536 | Examples: 537 | 538 | Delete the existing test binary (if there's one), then run SIG-Windows while skipping SIG-Storage tests using 2 nodes: 539 | 540 | ``` 541 | rm ~/go/src/k8s.io/kubernetes/_output/dockerized/bin/linux/amd64/e2e.test ; nodeCount=2 testArgs='--ginkgo.focus=\[sig-windows\] --ginkgo.skip=\[sig-storage\]' ./run-e2e.sh 542 | ``` 543 | 544 | ## Building Other Components 545 | 546 | ### Azure-CNI 547 | 548 | Azure-CNI source is at [Azure/azure-container-networking](https://github.com/Azure/azure-container-networking/) 549 | 550 | The same dev VM has everything you need to build the Azure CNI repo. Clone it inside the dev VM, then run 551 | 552 | ```bash 553 | ./build/build-all-containerized.sh windows amd64 554 | ``` 555 | 556 | ## Using ContainerD 557 | 558 | This has moved to [containerd.md](containerd.md) 559 | 560 | 561 | ## Quick tips on Windows administration 562 | 563 | ### If you did this in bash, do this in PowerShell 564 | 565 | bash | PowerShell 566 | -----|----------- 567 | `tail ...` | `Get-Content -Last 40 -Wait ...` 568 | `ls | xargs -n1 ...` | `ls | %{ ... $_ }` 569 | 570 | 571 | ## Credits 572 | 573 | Some of the steps were borrowed from [Kubernetes Dev on Azure](https://github.com/khenidak/kubernetes-dev-on-azure). Thanks Kal! 574 | 575 | 576 | 577 | 578 | [TestGrid]: https://testgrid.k8s.io/sig-windows#aks-engine-azure-windows-master 579 | [SIG-Windows Job Definitions]: https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-sigs/sig-windows 580 | [test-infra docs]: https://github.com/kubernetes/test-infra 581 | [intro from SIG-Testing]: https://www.youtube.com/watch?v=7-_O41W3FRU 582 | [kubernetes/website]: https://github.com/kubernetes/website/ 583 | [docs.kubernetes.io]: https://docs.kubernetes.io 584 | [Intro to Windows in Kubernetes]: https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/ 585 | [AKS-Engine Windows Walkthrough]: https://github.com/Azure/aks-engine/blob/master/docs/topics/windows.md 586 | [AKS-Engine]: https://github.com/Azure/aks-engine 587 | [Windows Kubernetes The Hard Way]: https://github.com/pjh/kubernetes-the-hard-way 588 | [WSL2]: https://docs.microsoft.com/en-us/windows/wsl/wsl2-install 589 | [kubernetes-sigs/windows-testing]: https://github.com/kubernetes-sigs/windows-testing 590 | -------------------------------------------------------------------------------- /containerd.md: -------------------------------------------------------------------------------- 1 | # Kubernetes, Windows, and CRI-ContainerD 2 | 3 | This page describes how to build and set up CRI-ContainerD with Kubernetes on Windows. This work is tracked for Kubernetes 1.15 as [enhancement#1001](https://github.com/kubernetes/enhancements/issues/1001). For more background on how this will be used and tested, please review the [kep](https://github.com/kubernetes/enhancements/blob/master/keps/sig-windows/20190424-windows-cri-containerd.md). 4 | 5 | 6 | 7 | 8 | ## Revision History 9 | 10 | Date | Description 11 | -----------|------------ 12 | 2019-05-07 | Split out from README.md 13 | 14 | 15 | 16 | ## Using ContainerD 17 | 18 | [SaswatB](https://github.com/SaswatB) Set up a working environment for testing Kubernetes. It's in the Microsoft SDN repo and is used for the Windows CNI dev/test environments. You can get those scripts here https://github.com/Microsoft/SDN/tree/master/Kubernetes/containerd . I'm aiming to get this better consolidated to clarify how to build and set things up if your setup doesn't match what's prescribed in those scripts. 19 | 20 | ### Building ContainerD 21 | 22 | Getting all the binaries needed will require building from multiple repos. Here's the full list of what's required. 23 | 24 | - For the CRI-ContainerD daemon: 25 | - containerd.exe (built from [jterry75/cri](https://github.com/jterry75/cri/tree/windows_port/cmd/containerd)) 26 | - containerd-shim-runhcs-v1.exe (built from [Microsoft/hcsshim](https://github.com/Microsoft/hcsshim/tree/master/cmd/containerd-shim-runhcs-v1)) 27 | - Containerd & CRI clients: 28 | - ctr.exe - used for managing containers directly with ContainerD (but not CRI). [src](https://github.com/containerd/cri/tree/master/cmd/ctr) 29 | - crictl.exe - used for managing sandboxes(pods) and containers using CRI [src](https://github.com/kubernetes-sigs/cri-tools/) [doc](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md) 30 | - CNI plugin and meta-plugin 31 | - nat.exe - for standalone networking (not Kubernetes) - source:[Microsoft/windows-container-networking](https://github.com/Microsoft/windows-container-networking/tree/master/plugins) 32 | - FlannelD.exe & flannel.exe 33 | - SDNBridge.exe - if using `host-gw` mode - source:[Microsoft/windows-container-networking](https://github.com/Microsoft/windows-container-networking/tree/master/plugins) 34 | - SDNOverlay.exe - if using `overlay` mode - source:[Microsoft/windows-container-networking](https://github.com/Microsoft/windows-container-networking/tree/master/plugins) 35 | 36 | 37 | There's a convenience script you can run from a Linux or Mac with Docker installed. It will create two zip files 38 | with everything you need. 39 | 40 | See [scripts/buildPackage.sh](scripts/buildPackage.sh) 41 | 42 | #### Building containerd-shim-runhcs-v1 43 | 44 | 45 | Getting the source 46 | 47 | ```bash 48 | go get github.com/Microsoft/hcsshim 49 | ``` 50 | 51 | Building it 52 | 53 | ```bash 54 | GOOS=windows go build github.com/Microsoft/hcsshim/cmd/containerd-shim-runhcs-v1 55 | ``` 56 | 57 | 58 | #### Building ContainerD 59 | 60 | There isn't a released version ready for k8s yet, but the master branch of containerd does support CRI for Windows containers. Here's how to build it: 61 | 62 | ```bash 63 | cd $GOPATH 64 | mkdir -p src/github.com/containerd 65 | cd src/github.com/containerd 66 | git clone https://github.com/containerd/containerd.git 67 | cd containerd 68 | GOOS=windows make 69 | ``` 70 | 71 | This will produce `bin/containerd.exe` and `ctr.exe` 72 | 73 | 74 | ### Building CNI meta-plugins compatible with ContainerD 75 | 76 | Clone https://github.com/Microsoft/windows-container-networking on your Linux dev/build machine, then run: 77 | 78 | ```bash 79 | make dev 80 | # in the container 81 | make all 82 | exit 83 | ``` 84 | 85 | That will produce `nat.exe`, `sdnbridge.exe`, and `sdnoverlay.exe` which are needed later. 86 | 87 | 88 | 89 | ### Future: Building from containerd/containerd 90 | 91 | > Don't do this yet. Right now binaries need to be built from jterry75/cri to include CRI support. Skip to the next section 92 | 93 | ``` 94 | user@machine:/> cd $GOPATH/src/github.com/containerd/containerd 95 | user@machine:/> export GOOS=windows 96 | user@machine:/> make 97 | + bin/ctr.exe 98 | + bin/containerd.exe 99 | + bin/containerd-stress.exe 100 | + bin/containerd-release.exe 101 | + bin/containerd-shim-runhcs-v1.exe 102 | + binaries 103 | ``` 104 | 105 | #### Future: Revendoring to get hcsshim changes 106 | 107 | > This is optional, only if you're testing changes to hcsshim. This won't be needed once `containerd-shim-runhcs-v1.exe` is built directly from the hcsshim repo. 108 | 109 | ``` 110 | user@machine:/> go get -u github.com/lk4d4/vndr 111 | user@machine:/> vndr github.com/Microsoft/hcsshim 112 | ``` 113 | 114 | If you intend to include a vendored change in a PR to containerd, be sure to update `vendor.conf` too. 115 | 116 | 117 | 118 | 119 | ## Setting up a node with ContainerD 120 | 121 | >TODO - Testing Windows Server 2019 with ContainerD. VM work started here: https://github.com/patricklang/packer-windows/tree/containerd 122 | 123 | 124 | Configuration Steps 125 | - Register ContainerD as service 126 | - [ ] Enable CRI listener for ContainerD.exe 127 | - [ ] Getting kubelet configured to use CRI endpoint instead of dockershim 128 | 129 | 130 | #### Create ContainerD config 131 | 132 | If you don't already have a config file the daemon can generate one for you: 133 | ``` 134 | C:\> containerd.exe config default > config.toml 135 | ``` 136 | 137 | (Common areas of the config to change) 138 | root - The root where all daemon data is kept 139 | state - The state directory where all plugin data is kept snapshots, images, container bundles, etc. 140 | grpc 141 | address - The address the containerd daemon will serve. Default is: \\.\pipe\containerd-containerd 142 | debug 143 | level - Set to debug for all daemon debugging 144 | 145 | > TODO: Missing CNI step 146 | 147 | #### Test using ContainerD to pull & run an image 148 | 149 | ``` 150 | ctr.exe images pull mcr.microsoft.com/windows/nanoserver:1809 151 | ctr.exe run --rm mcr.microsoft.com/windows/nanoserver:1809 argon-test cmd /c "echo Hello World!" 152 | ``` 153 | 154 | Example Output: 155 | 156 | ```none 157 | PS C:\containerd> ./ctr.exe images pull mcr.microsoft.com/windows/nanoserver:1809 158 | mcr.microsoft.com/windows/nanoserver:1809: resolved |++++++++++++++++++++++++++++++++++++++| 159 | index-sha256:75bae46687f544f139ec57e1925d184fbb2ed70f6e0e5c993a55bd4f8e8e17a8: exists |++++++++++++++++++++++++++++++++++++++| 160 | manifest-sha256:6603a3e57f2d127fbddbc7b0aa3807b02b3c25163a7c6404da1d107ce33549c4: exists |++++++++++++++++++++++++++++++++++++++| 161 | manifest-sha256:5953d8407d58ddc66e3eb426e320e93786a3cb173957cc5af79d46f731f3301c: exists |++++++++++++++++++++++++++++++++++++++| 162 | config-sha256:3601d6edd492515e2f9b352db93b0d67af0d49f1178561b5a5d50e1232c0276a: done |++++++++++++++++++++++++++++++++++++++| 163 | layer-sha256:1046f7eb9dcd29d2478f707dca8726d2ae066a276196e327bd386d50f6448b2a: done |++++++++++++++++++++++++++++++++++++++| 164 | config-sha256:4702b277b15f4ce1a1a3f26092229e7b79f8f6e11450d9d171bcf7367ab96350: done |++++++++++++++++++++++++++++++++++++++| 165 | elapsed: 0.5 s total: 0.0 B (0.0 B/s) 166 | unpacking windows/amd64 sha256:75bae46687f544f139ec57e1925d184fbb2ed70f6e0e5c993a55bd4f8e8e17a8... 167 | done 168 | PS C:\containerd> ./ctr.exe run --rm mcr.microsoft.com/windows/nanoserver:1809 argon-test cmd /c "echo Hello World!" 169 | Hello World! 170 | ``` 171 | 172 | 173 | #### Test using CRI-ContainerD to pull and run an image 174 | 175 | > TODO: this doesn't work yet, section incomplete. ContainerD fails to start a container without a CNI configured. 176 | 177 | First, you need a sandbox/pod configuration. Copy this into a file `pod-sandbox-default.json`. It will create a process-isolated Windows pod. 178 | 179 | ```json 180 | { 181 | "metadata": { 182 | "name": "sandbox", 183 | "namespace": "default", 184 | "attempt": 1 185 | } 186 | } 187 | ``` 188 | 189 | ```none 190 | ./crictl -r npipe:\\\\.\pipe\containerd-containerd pull mcr.microsoft.com/windows/nanoserver:1809 191 | Image is up to date for sha256:4702b277b15f4ce1a1a3f26092229e7b79f8f6e11450d9d171bcf7367ab96350 192 | ``` 193 | 194 | Create the sandbox with: `.\crictl -r npipe:\\\\.\pipe\containerd-containerd runp .\pod-sandbox-default.json` 195 | 196 | 197 | Create a container config, copying this file into `container-config-windows-hello-world.json` 198 | 199 | ```json 200 | 201 | ``` 202 | 203 | `.\crictl.exe create .\container-config-windows-hello-world.json .\pod-sandbox-default.json` 204 | 205 | 206 | #### Viewing running local pods with crictl 207 | 208 | `crictl -r npipe:\\\\.\pipe\containerd-containerd pods` 209 | 210 | `crictl -r npipe:\\\\.\pipe\containerd-containerd inspectp 1c0e277aba1e1` 211 | 212 | -------------------------------------------------------------------------------- /scripts/buildPackage.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e -x -o pipefail 4 | 5 | GOTAG="1.13.7" 6 | DOCKERARGS="--network=host" 7 | 8 | OUTDIR="$(pwd)/_output" 9 | if [ ! -d $OUTDIR ]; then 10 | mkdir $OUTDIR 11 | fi 12 | 13 | cat < $OUTDIR/buildcri.sh 14 | set -e -x -o pipefail 15 | export GOOS=windows 16 | export GOARCH=amd64 17 | go get github.com/Microsoft/hcsshim 18 | cd src/github.com/Microsoft/hcsshim 19 | git rev-parse HEAD > /output/hcsshim-revision.txt 20 | cd \$GOPATH 21 | go build -o /output/containerd-shim-runhcs-v1.exe github.com/Microsoft/hcsshim/cmd/containerd-shim-runhcs-v1 22 | mkdir -p src/github.com/containerd 23 | cd src/github.com/containerd 24 | pwd 25 | git clone https://github.com/containerd/containerd.git 26 | cd containerd 27 | git rev-parse HEAD > /output/containerd-revision.txt 28 | make 29 | # make cri-release # should work, but doesn't 30 | cp bin/ctr.exe /output 31 | #cp bin/containerd.exe /output # missing CRI plugin, so build from containerd/cri 32 | cd \$GOPATH 33 | cd src/github.com/containerd 34 | git clone https://github.com/containerd/cri.git 35 | cd cri 36 | git rev-parse HEAD > /output/cri-revision.txt 37 | make containerd 38 | cp _output/containerd.exe /output 39 | apt update 40 | apt install -y zip 41 | cd /output 42 | zip windows-cri-containerd.zip *.exe *.txt 43 | rm -f /output/*.exe 44 | rm -f /output/*.txt 45 | EOF 46 | chmod +x $OUTDIR/buildcri.sh 47 | 48 | cat < $OUTDIR/buildcni.sh 49 | set -e -x -o pipefail 50 | export GOOS=windows 51 | export GOARCH=amd64 52 | mkdir -p src/github.com/Microsoft 53 | cd src/github.com/Microsoft 54 | git clone https://github.com/Microsoft/windows-container-networking.git 55 | cd windows-container-networking 56 | git rev-parse HEAD > /output/cni-revision.txt 57 | make all 58 | mv out/*.exe /output 59 | apt update 60 | apt install -y zip 61 | cd /output 62 | zip windows-cni-containerd.zip *.exe *.txt 63 | rm -f /output/*.exe 64 | rm -f /output/*.txt 65 | EOF 66 | chmod +x $OUTDIR/buildcni.sh 67 | 68 | 69 | 70 | docker run $DOCKERARGS -v $OUTDIR:/output golang:$GOTAG /bin/bash -c /output/buildcri.sh 71 | docker run $DOCKERARGS -v $OUTDIR:/output golang:$GOTAG /bin/bash -c /output/buildcni.sh 72 | -------------------------------------------------------------------------------- /scripts/prfetch.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -u -x -e 3 | 4 | # This borrowed a lot of ideas from https://raw.githubusercontent.com/kubernetes/kubernetes/master/hack/cherry_pick_pull.sh 5 | # If you're looking for more capabilities like creating new PRs, be sure to start from that script instead of this one 6 | 7 | dir=$(mktemp -d) 8 | 9 | export PRS=$(cat <<-END 10 | END 11 | ) 12 | 13 | export MAIN_REPO_ORG=kubernetes 14 | export MAIN_REPO_NAME=kubernetes 15 | export AGGREGATION_REPO_ORG=azure 16 | export AGGREGATION_REPO_NAME=kubernetes 17 | export AGGREGATION_BRANCH=windows-v1.13-dev 18 | 19 | echo Will fetch PRs: ${PRS} 20 | 21 | if [ -d .git ]; then 22 | echo "Repo already exists. Discarding local changes, and syncing from origin/master" 23 | git am --abort || git rebase --abort || true 24 | git reset HEAD -- . 25 | git checkout -- . 26 | git clean -df 27 | git checkout master 28 | git pull 29 | git checkout ${AGGREGATION_BRANCH} 30 | git reset --hard master 31 | git reset HEAD -- . 32 | else 33 | git clone git@github.com:${AGGREGATION_REPO_ORG}/${AGGREGATION_REPO_NAME}.git . 34 | git remote add upstream https://github.com/${MAIN_REPO_ORG}/${MAIN_REPO_NAME}.git 35 | git fetch upstream 36 | git checkout ${AGGREGATION_BRANCH} 37 | fi 38 | 39 | PR_REGEX="^\#" 40 | for item in ${PRS[@]}; do 41 | if [[ "$item" =~ $PR_REGEX ]]; then 42 | pull="$(echo $item | sed 's/^\#//')" 43 | echo "Cherry-picking https://github.com/${MAIN_REPO_ORG}/${MAIN_REPO_NAME}/pull/${pull}.patch" 44 | curl -o "$dir/${pull}.patch" -sSL "https://github.com/${MAIN_REPO_ORG}/${MAIN_REPO_NAME}/pull/${pull}.patch" 45 | git am -3 "$dir/${pull}.patch" || { 46 | echo "Failed to merge. Stopping now" 47 | exit 1 48 | } 49 | else 50 | echo "Pulling existing change $item" 51 | git merge $item 52 | fi 53 | done 54 | 55 | rm -rf "${dir}" 56 | 57 | echo Success! Now run git push --force-with-lease 58 | exit 0 -------------------------------------------------------------------------------- /scripts/run-e2e.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e -o pipefail #-x 3 | 4 | 5 | function verifyExists { 6 | if ! [ -x "$(command -v $1)" ]; then 7 | echo "$1 is not installed - please install it" >&2 8 | exit 1 9 | fi 10 | } 11 | 12 | # az aks-engine - skipped for now 13 | for i in yq jq curl kubectl go 14 | do 15 | verifyExists $i 16 | done 17 | 18 | # default to getting the latest args from GitHub 19 | if [ -z "$jobUrl" ]; then jobUrl="https://raw.githubusercontent.com/kubernetes/test-infra/master/config/jobs/kubernetes-sigs/sig-windows/sig-windows-config.yaml"; fi 20 | if [ -z "$jobName" ]; then jobName="ci-kubernetes-e2e-aks-engine-azure-master-windows"; fi 21 | if [ -z "$testArgs" ]; then testArgs=$(curl -SsL $jobUrl | yq ".periodics[] | select(.name == \"$jobName\") | .spec.containers[].args[] | match(\"(--ginkgo.*)\";\"g\") | .captures[].string " | sed 's/\"//g' | sed 's/\\\\/\\/g' ); fi 22 | 23 | # require existing cluster, else deploy one 24 | if [ -z "$KUBECONFIG" ]; then echo "Missing KUBECONFIG" >&2; exit 1; fi 25 | # echo Running aks-engine 26 | 27 | # build latest tests 28 | if [ -z "$GOPATH" ]; then GOPATH="$(go env GOPATH)"; fi 29 | testBin="$GOPATH/src/k8s.io/kubernetes/_output/dockerized/bin/linux/amd64/e2e.test" 30 | if [ ! -f $testBin ]; then 31 | echo "e2e.test does not exist, building it" 32 | oldcd=$(pwd) 33 | cd $GOPATH/src/k8s.io/kubernetes 34 | ./build/run.sh make WHAT=test/e2e/e2e.test 35 | cd $oldcd 36 | fi 37 | 38 | # get test prerequisites 39 | if [ -z "$KUBE_TEST_REPO_LIST" ]; then 40 | export KUBE_TEST_REPO_LIST=$(pwd)/repo_list 41 | if [ ! -f $KUBE_TEST_REPO_LIST ]; then 42 | curl -SsL -o repo_list https://raw.githubusercontent.com/kubernetes-sigs/windows-testing/master/images/image-repo-list 43 | fi 44 | fi 45 | 46 | if [ -z "$nodeCount" ]; then 47 | export nodeCount=$(kubectl get node -o wide | grep -e 'Ready.*agent.*Windows' | wc -l) 48 | fi 49 | 50 | fullArgs="--provider=skeleton --num-nodes=$nodeCount --node-os-distro=windows -report-dir logs $testArgs" 51 | 52 | echo Running $testBin $fullArgs 53 | $testBin $fullArgs | tee run-e2e.log 54 | 55 | #echo Getting logs from nodes with CollectLogs.ps1 56 | -------------------------------------------------------------------------------- /vagrant/License.txt: -------------------------------------------------------------------------------- 1 | SDN Sample Scripts v.1.0 2 | 3 | Copyright (c) Microsoft Corporation 4 | 5 | All rights reserved. 6 | 7 | MIT License 8 | 9 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the ""Software""), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | -------------------------------------------------------------------------------- /vagrant/Vagrantfile: -------------------------------------------------------------------------------- 1 | $VirtualSwitchName = "Default Switch" 2 | 3 | LINUX_IMAGE = "bento/ubuntu-16.04" 4 | 5 | Vagrant.configure("2") do |config| 6 | config.vm.provider "hyperv" do |h| 7 | h.enable_virtualization_extensions = true 8 | h.linked_clone = true 9 | h.cpus = 4 10 | h.memory = 6144 11 | h.maxmemory = 6144 12 | h.vmname = "k8s-dev" 13 | end 14 | config.vm.define "k8s-dev" do |subconfig| 15 | subconfig.vm.box = LINUX_IMAGE 16 | subconfig.vm.hostname = "k8s-dev" 17 | subconfig.vm.network :public_network, bridge: $VirtualSwitchName 18 | config.vm.synced_folder ".", "/vagrant", disabled: true 19 | subconfig.vm.provision "shell", path: "ubuntu.sh" 20 | subconfig.vm.provision "shell", path: "go.sh" 21 | end 22 | end -------------------------------------------------------------------------------- /vagrant/go.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | echo Setting up Golang 3 | 4 | $(cd /tmp && wget -q https://storage.googleapis.com/golang/go1.10.linux-amd64.tar.gz) 5 | sudo tar -C /usr/local -xzf /tmp/go1.10.linux-amd64.tar.gz 6 | echo "export PATH=\$PATH:/usr/local/go/bin" | sudo tee --append /etc/profile 7 | echo "export PATH=\$PATH:~/go/bin" | tee --append ~/.bashrc 8 | source /etc/profile 9 | source ~/.bashrc -------------------------------------------------------------------------------- /vagrant/readme.md: -------------------------------------------------------------------------------- 1 | This was originally posted at https://github.com/Microsoft/SDN/tree/master/Kubernetes/linux/vagrant 2 | 3 | # Kubernetes build VM in Vagrant 4 | 5 | This is a Vagrantfile and collection of scripts to set up a Kubernetes build environment. It's designed to make it fast and easy to build Kubernetes binaries for Windows without having to maintain a Linux VM long-term. 6 | 7 | Vagrant uses a simple sequence of commands to handle VM management and running things inside the VM. 8 | 9 | 1. `vagrant up` starts the VM. If it doesn't exist, it will download and create it automatically. 10 | 2. `vagrant ssh` connects to the VM with SSH 11 | 3. `vagrant halt` shuts down the VM, but doesn't destroy it. `vagrant up` will start it again. 12 | 4. `vagrant destroy` stops the VM, and deletes it. All data is lost. 13 | 14 | ## Windows 15 | 16 | First, [Install Hyper-V on Windows 10](https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v), and be sure [Vagrant](https://vagrantup.com) is installed. 17 | 18 | This VM configures a 60GB virtual disk, so make sure you have at least that much free. If you don't have that much free on C:, then copy this folder to another drive and run it there. Vagrant will place the VM in that path. 19 | 20 | 1. From an elevated prompt, run `vagrant up --provider hyperv` 21 | 2. `vagrant ssh` - make sure you can connect. If you get a prompt like `vagrant@k8s-dev:~$`, then you're good to go. Run `exit` to disconnect. 22 | 23 | ## Mac 24 | 25 | > TODO 26 | 27 | ## Linux 28 | 29 | > TODO -------------------------------------------------------------------------------- /vagrant/ubuntu.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # echo "Starting bootstrap" 3 | 4 | # systemctl stop apt-daily.service 5 | # systemctl kill --kill-who=all apt-daily.service 6 | 7 | # # wait until `apt-get updated` has been killed 8 | # while ! (systemctl list-units --all apt-daily.service | fgrep -q dead) 9 | # do 10 | # echo "apt-daily still running" 11 | # sleep 1; 12 | # done 13 | 14 | curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - 15 | add-apt-repository \ 16 | "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ 17 | $(lsb_release -cs) \ 18 | stable" 19 | 20 | apt update 21 | 22 | apt install -y git build-essential docker-ce 23 | sudo usermod -a -G docker vagrant --------------------------------------------------------------------------------