├── Dockerfile ├── LICENSE ├── README.md ├── annotator └── main.go ├── bestprice.go ├── build ├── build-container ├── deployments ├── nginx.yaml └── scheduler.yaml ├── kubernetes.go ├── main.go ├── processor.go └── types.go /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM scratch 2 | MAINTAINER Kelsey Hightower 3 | ADD scheduler /scheduler 4 | ENTRYPOINT ["/scheduler"] 5 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # scheduler 2 | 3 | Toy scheduler for use in Kubernetes demos. 4 | 5 | ## Usage 6 | 7 | Annotate each node using the annotator command: 8 | 9 | ``` 10 | kubectl proxy 11 | ``` 12 | ``` 13 | Starting to serve on 127.0.0.1:8001 14 | ``` 15 | 16 | ``` 17 | go run annotator/main.go 18 | ``` 19 | ``` 20 | gke-k0-default-pool-728d327f-00lq 1.60 21 | gke-k0-default-pool-728d327f-3vzg 0.20 22 | gke-k0-default-pool-728d327f-nmz7 0.80 23 | gke-k0-default-pool-728d327f-pxee 0.05 24 | gke-k0-default-pool-728d327f-xm4i 0.05 25 | gke-k0-default-pool-728d327f-zynj 0.20 26 | ``` 27 | 28 | ### Create a deployment 29 | 30 | ``` 31 | kubectl create -f deployments/nginx.yaml 32 | ``` 33 | ``` 34 | deployment "nginx" created 35 | ``` 36 | 37 | The nginx pod should be in a pending state: 38 | 39 | ``` 40 | kubectl get pods 41 | ``` 42 | ``` 43 | NAME READY STATUS RESTARTS AGE 44 | nginx-1431970305-mwghf 0/1 Pending 0 27s 45 | ``` 46 | 47 | ### Run the Scheduler 48 | 49 | List the nodes and note the price of each node. 50 | 51 | ``` 52 | annotator -l 53 | ``` 54 | ``` 55 | gke-k0-default-pool-728d327f-00lq 0.80 56 | gke-k0-default-pool-728d327f-3vzg 0.40 57 | gke-k0-default-pool-728d327f-nmz7 0.40 58 | gke-k0-default-pool-728d327f-pxee 0.05 59 | gke-k0-default-pool-728d327f-xm4i 1.60 60 | gke-k0-default-pool-728d327f-zynj 0.40 61 | ``` 62 | 63 | Run the best price scheduler: 64 | 65 | ``` 66 | scheduler 67 | ``` 68 | ``` 69 | 2016/08/19 11:16:25 Starting custom scheduler... 70 | 2016/08/19 11:16:28 Successfully assigned nginx-1431970305-mwghf to gke-k0-default-pool-728d327f-pxee 71 | 2016/08/19 11:16:35 Shutdown signal received, exiting... 72 | 2016/08/19 11:16:35 Stopped reconciliation loop. 73 | 2016/08/19 11:16:35 Stopped scheduler. 74 | ``` 75 | 76 | > Notice the pending nginx pod is deployed to the node with the lowest cost. 77 | 78 | ## Run the Scheduler on Kubernetes 79 | 80 | ``` 81 | kubectl create -f deployments/scheduler.yaml 82 | ``` 83 | ``` 84 | deployment "scheduler" created 85 | ``` 86 | -------------------------------------------------------------------------------- /annotator/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "encoding/json" 6 | "flag" 7 | "fmt" 8 | "math/rand" 9 | "net/http" 10 | "os" 11 | "time" 12 | ) 13 | 14 | var listOnly bool 15 | 16 | type NodeList struct { 17 | Items []Node `json:"items"` 18 | } 19 | 20 | type Node struct { 21 | Metadata Metadata `json:"metadata"` 22 | } 23 | 24 | type Metadata struct { 25 | Name string `json:"name,omitempty"` 26 | Annotations map[string]string `json:"annotations"` 27 | } 28 | 29 | func main() { 30 | flag.BoolVar(&listOnly, "l", false, "List current annotations and exist") 31 | flag.Parse() 32 | 33 | prices := []string{"0.05", "0.10", "0.20", "0.40", "0.80", "1.60"} 34 | resp, err := http.Get("http://127.0.0.1:8001/api/v1/nodes") 35 | if err != nil { 36 | fmt.Println(err) 37 | os.Exit(1) 38 | } 39 | if resp.StatusCode != 200 { 40 | fmt.Println("Invalid status code", resp.Status) 41 | os.Exit(1) 42 | } 43 | 44 | var nodes NodeList 45 | decoder := json.NewDecoder(resp.Body) 46 | err = decoder.Decode(&nodes) 47 | if err != nil { 48 | fmt.Println(err) 49 | os.Exit(1) 50 | } 51 | 52 | if listOnly { 53 | for _, node := range nodes.Items { 54 | price := node.Metadata.Annotations["hightower.com/cost"] 55 | fmt.Printf("%s %s\n", node.Metadata.Name, price) 56 | } 57 | os.Exit(0) 58 | } 59 | 60 | rand.Seed(time.Now().Unix()) 61 | for _, node := range nodes.Items { 62 | price := prices[rand.Intn(len(prices))] 63 | annotations := map[string]string{ 64 | "hightower.com/cost": price, 65 | } 66 | patch := Node{ 67 | Metadata{ 68 | Annotations: annotations, 69 | }, 70 | } 71 | 72 | var b []byte 73 | body := bytes.NewBuffer(b) 74 | err := json.NewEncoder(body).Encode(patch) 75 | if err != nil { 76 | fmt.Println(err) 77 | os.Exit(1) 78 | } 79 | 80 | url := "http://127.0.0.1:8001/api/v1/nodes/" + node.Metadata.Name 81 | request, err := http.NewRequest("PATCH", url, body) 82 | if err != nil { 83 | fmt.Println(err) 84 | os.Exit(1) 85 | } 86 | request.Header.Set("Content-Type", "application/strategic-merge-patch+json") 87 | request.Header.Set("Accept", "application/json, */*") 88 | 89 | resp, err := http.DefaultClient.Do(request) 90 | if err != nil { 91 | fmt.Println(err) 92 | os.Exit(1) 93 | } 94 | 95 | if resp.StatusCode != 200 { 96 | fmt.Println(err) 97 | os.Exit(1) 98 | } 99 | fmt.Printf("%s %s\n", node.Metadata.Name, price) 100 | } 101 | } 102 | -------------------------------------------------------------------------------- /bestprice.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All Rights Reserved. 2 | // Licensed under the Apache License, Version 2.0 (the "License"); 3 | // you may not use this file except in compliance with the License. 4 | // You may obtain a copy of the License at 5 | // 6 | // http://www.apache.org/licenses/LICENSE-2.0 7 | // 8 | // Unless required by applicable law or agreed to in writing, software 9 | // distributed under the License is distributed on an "AS IS" BASIS, 10 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 11 | // See the License for the specific language governing permissions and 12 | // limitations under the License. 13 | 14 | package main 15 | 16 | import ( 17 | "strconv" 18 | ) 19 | 20 | func bestPrice(nodes []Node) (Node, error) { 21 | type NodePrice struct { 22 | Node Node 23 | Price float64 24 | } 25 | 26 | var bestNodePrice *NodePrice 27 | for _, n := range nodes { 28 | price, ok := n.Metadata.Annotations["hightower.com/cost"] 29 | if !ok { 30 | continue 31 | } 32 | f, err := strconv.ParseFloat(price, 32) 33 | if err != nil { 34 | return Node{}, err 35 | } 36 | if bestNodePrice == nil { 37 | bestNodePrice = &NodePrice{n, f} 38 | continue 39 | } 40 | if f < bestNodePrice.Price { 41 | bestNodePrice.Node = n 42 | bestNodePrice.Price = f 43 | } 44 | } 45 | 46 | if bestNodePrice == nil { 47 | bestNodePrice = &NodePrice{nodes[0], 0} 48 | } 49 | return bestNodePrice.Node, nil 50 | } 51 | -------------------------------------------------------------------------------- /build: -------------------------------------------------------------------------------- 1 | go build -a --ldflags '-extldflags "-static"' -tags netgo -installsuffix netgo . 2 | -------------------------------------------------------------------------------- /build-container: -------------------------------------------------------------------------------- 1 | GOOS=linux bash build 2 | docker build -t kelseyhightower/scheduler:0.4.0 . 3 | docker push kelseyhightower/scheduler:0.4.0 4 | rm scheduler 5 | -------------------------------------------------------------------------------- /deployments/nginx.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Deployment 3 | metadata: 4 | labels: 5 | app: nginx 6 | name: nginx 7 | spec: 8 | replicas: 1 9 | template: 10 | metadata: 11 | labels: 12 | app: nginx 13 | name: nginx 14 | spec: 15 | schedulerName: hightower 16 | containers: 17 | - name: nginx 18 | image: "nginx:1.11.1-alpine" 19 | resources: 20 | requests: 21 | cpu: "500m" 22 | memory: "128M" 23 | -------------------------------------------------------------------------------- /deployments/scheduler.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Deployment 3 | metadata: 4 | labels: 5 | app: scheduler 6 | name: scheduler 7 | spec: 8 | replicas: 1 9 | template: 10 | metadata: 11 | labels: 12 | app: scheduler 13 | name: scheduler 14 | spec: 15 | containers: 16 | - name: scheduler 17 | image: kelseyhightower/scheduler:0.4.0 18 | - name: kubectl 19 | image: kelseyhightower/kubectl:1.3.4 20 | args: 21 | - "proxy" 22 | -------------------------------------------------------------------------------- /kubernetes.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All Rights Reserved. 2 | // Licensed under the Apache License, Version 2.0 (the "License"); 3 | // you may not use this file except in compliance with the License. 4 | // You may obtain a copy of the License at 5 | // 6 | // http://www.apache.org/licenses/LICENSE-2.0 7 | // 8 | // Unless required by applicable law or agreed to in writing, software 9 | // distributed under the License is distributed on an "AS IS" BASIS, 10 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 11 | // See the License for the specific language governing permissions and 12 | // limitations under the License. 13 | 14 | package main 15 | 16 | import ( 17 | "bytes" 18 | "encoding/json" 19 | "errors" 20 | "fmt" 21 | "io/ioutil" 22 | "log" 23 | "net/http" 24 | "net/url" 25 | "strconv" 26 | "strings" 27 | "time" 28 | ) 29 | 30 | var ( 31 | apiHost = "127.0.0.1:8001" 32 | bindingsEndpoint = "/api/v1/namespaces/default/pods/%s/binding/" 33 | eventsEndpoint = "/api/v1/namespaces/default/events" 34 | nodesEndpoint = "/api/v1/nodes" 35 | podsEndpoint = "/api/v1/pods" 36 | watchPodsEndpoint = "/api/v1/watch/pods" 37 | ) 38 | 39 | func postEvent(event Event) error { 40 | var b []byte 41 | body := bytes.NewBuffer(b) 42 | err := json.NewEncoder(body).Encode(event) 43 | if err != nil { 44 | return err 45 | } 46 | 47 | request := &http.Request{ 48 | Body: ioutil.NopCloser(body), 49 | ContentLength: int64(body.Len()), 50 | Header: make(http.Header), 51 | Method: http.MethodPost, 52 | URL: &url.URL{ 53 | Host: apiHost, 54 | Path: eventsEndpoint, 55 | Scheme: "http", 56 | }, 57 | } 58 | request.Header.Set("Content-Type", "application/json") 59 | 60 | resp, err := http.DefaultClient.Do(request) 61 | if err != nil { 62 | return err 63 | } 64 | if resp.StatusCode != 201 { 65 | return errors.New("Event: Unexpected HTTP status code" + resp.Status) 66 | } 67 | return nil 68 | } 69 | 70 | func getNodes() (*NodeList, error) { 71 | var nodeList NodeList 72 | 73 | request := &http.Request{ 74 | Header: make(http.Header), 75 | Method: http.MethodGet, 76 | URL: &url.URL{ 77 | Host: apiHost, 78 | Path: nodesEndpoint, 79 | Scheme: "http", 80 | }, 81 | } 82 | request.Header.Set("Accept", "application/json, */*") 83 | 84 | resp, err := http.DefaultClient.Do(request) 85 | if err != nil { 86 | return nil, err 87 | } 88 | 89 | err = json.NewDecoder(resp.Body).Decode(&nodeList) 90 | if err != nil { 91 | return nil, err 92 | } 93 | 94 | return &nodeList, nil 95 | } 96 | 97 | func watchUnscheduledPods() (<-chan Pod, <-chan error) { 98 | pods := make(chan Pod) 99 | errc := make(chan error, 1) 100 | 101 | v := url.Values{} 102 | v.Set("fieldSelector", "spec.nodeName=") 103 | 104 | request := &http.Request{ 105 | Header: make(http.Header), 106 | Method: http.MethodGet, 107 | URL: &url.URL{ 108 | Host: apiHost, 109 | Path: watchPodsEndpoint, 110 | RawQuery: v.Encode(), 111 | Scheme: "http", 112 | }, 113 | } 114 | request.Header.Set("Accept", "application/json, */*") 115 | 116 | go func() { 117 | for { 118 | resp, err := http.DefaultClient.Do(request) 119 | if err != nil { 120 | errc <- err 121 | time.Sleep(5 * time.Second) 122 | continue 123 | } 124 | 125 | if resp.StatusCode != 200 { 126 | errc <- errors.New("Invalid status code: " + resp.Status) 127 | time.Sleep(5 * time.Second) 128 | continue 129 | } 130 | 131 | decoder := json.NewDecoder(resp.Body) 132 | for { 133 | var event PodWatchEvent 134 | err = decoder.Decode(&event) 135 | if err != nil { 136 | errc <- err 137 | break 138 | } 139 | 140 | if event.Type == "ADDED" { 141 | pods <- event.Object 142 | } 143 | } 144 | } 145 | }() 146 | 147 | return pods, errc 148 | } 149 | 150 | func getUnscheduledPods() ([]*Pod, error) { 151 | var podList PodList 152 | unscheduledPods := make([]*Pod, 0) 153 | 154 | v := url.Values{} 155 | v.Set("fieldSelector", "spec.nodeName=") 156 | 157 | request := &http.Request{ 158 | Header: make(http.Header), 159 | Method: http.MethodGet, 160 | URL: &url.URL{ 161 | Host: apiHost, 162 | Path: podsEndpoint, 163 | RawQuery: v.Encode(), 164 | Scheme: "http", 165 | }, 166 | } 167 | request.Header.Set("Accept", "application/json, */*") 168 | 169 | resp, err := http.DefaultClient.Do(request) 170 | if err != nil { 171 | return unscheduledPods, err 172 | } 173 | err = json.NewDecoder(resp.Body).Decode(&podList) 174 | if err != nil { 175 | return unscheduledPods, err 176 | } 177 | 178 | for _, pod := range podList.Items { 179 | if pod.Metadata.Annotations["scheduler.alpha.kubernetes.io/name"] == schedulerName { 180 | unscheduledPods = append(unscheduledPods, &pod) 181 | } 182 | } 183 | 184 | return unscheduledPods, nil 185 | } 186 | 187 | func getPods() (*PodList, error) { 188 | var podList PodList 189 | 190 | v := url.Values{} 191 | v.Add("fieldSelector", "status.phase=Running") 192 | v.Add("fieldSelector", "status.phase=Pending") 193 | 194 | request := &http.Request{ 195 | Header: make(http.Header), 196 | Method: http.MethodGet, 197 | URL: &url.URL{ 198 | Host: apiHost, 199 | Path: podsEndpoint, 200 | RawQuery: v.Encode(), 201 | Scheme: "http", 202 | }, 203 | } 204 | request.Header.Set("Accept", "application/json, */*") 205 | 206 | resp, err := http.DefaultClient.Do(request) 207 | if err != nil { 208 | return nil, err 209 | } 210 | err = json.NewDecoder(resp.Body).Decode(&podList) 211 | if err != nil { 212 | return nil, err 213 | } 214 | return &podList, nil 215 | } 216 | 217 | type ResourceUsage struct { 218 | CPU int 219 | } 220 | 221 | func fit(pod *Pod) ([]Node, error) { 222 | nodeList, err := getNodes() 223 | if err != nil { 224 | return nil, err 225 | } 226 | 227 | podList, err := getPods() 228 | if err != nil { 229 | return nil, err 230 | } 231 | 232 | resourceUsage := make(map[string]*ResourceUsage) 233 | for _, node := range nodeList.Items { 234 | resourceUsage[node.Metadata.Name] = &ResourceUsage{} 235 | } 236 | 237 | for _, p := range podList.Items { 238 | if p.Spec.NodeName == "" { 239 | continue 240 | } 241 | for _, c := range p.Spec.Containers { 242 | if strings.HasSuffix(c.Resources.Requests["cpu"], "m") { 243 | milliCores := strings.TrimSuffix(c.Resources.Requests["cpu"], "m") 244 | cores, err := strconv.Atoi(milliCores) 245 | if err != nil { 246 | return nil, err 247 | } 248 | ru := resourceUsage[p.Spec.NodeName] 249 | ru.CPU += cores 250 | } 251 | } 252 | } 253 | 254 | var nodes []Node 255 | fitFailures := make([]string, 0) 256 | 257 | var spaceRequired int 258 | for _, c := range pod.Spec.Containers { 259 | if strings.HasSuffix(c.Resources.Requests["cpu"], "m") { 260 | milliCores := strings.TrimSuffix(c.Resources.Requests["cpu"], "m") 261 | cores, err := strconv.Atoi(milliCores) 262 | if err != nil { 263 | return nil, err 264 | } 265 | spaceRequired += cores 266 | } 267 | } 268 | 269 | for _, node := range nodeList.Items { 270 | var allocatableCores int 271 | var err error 272 | if strings.HasSuffix(node.Status.Allocatable["cpu"], "m") { 273 | milliCores := strings.TrimSuffix(node.Status.Allocatable["cpu"], "m") 274 | allocatableCores, err = strconv.Atoi(milliCores) 275 | if err != nil { 276 | return nil, err 277 | } 278 | } else { 279 | cpu := node.Status.Allocatable["cpu"] 280 | cpuFloat, err := strconv.ParseFloat(cpu, 32) 281 | if err != nil { 282 | return nil, err 283 | } 284 | allocatableCores = int(cpuFloat * 1000) 285 | } 286 | 287 | freeSpace := (allocatableCores - resourceUsage[node.Metadata.Name].CPU) 288 | if freeSpace < spaceRequired { 289 | m := fmt.Sprintf("fit failure on node (%s): Insufficient CPU", node.Metadata.Name) 290 | fitFailures = append(fitFailures, m) 291 | continue 292 | } 293 | nodes = append(nodes, node) 294 | } 295 | 296 | if len(nodes) == 0 { 297 | // Emit a Kubernetes event that the Pod was scheduled successfully. 298 | timestamp := time.Now().UTC().Format(time.RFC3339) 299 | event := Event{ 300 | Count: 1, 301 | Message: fmt.Sprintf("pod (%s) failed to fit in any node\n%s", pod.Metadata.Name, strings.Join(fitFailures, "\n")), 302 | Metadata: Metadata{GenerateName: pod.Metadata.Name + "-"}, 303 | Reason: "FailedScheduling", 304 | LastTimestamp: timestamp, 305 | FirstTimestamp: timestamp, 306 | Type: "Warning", 307 | Source: EventSource{Component: "hightower-scheduler"}, 308 | InvolvedObject: ObjectReference{ 309 | Kind: "Pod", 310 | Name: pod.Metadata.Name, 311 | Namespace: "default", 312 | Uid: pod.Metadata.Uid, 313 | }, 314 | } 315 | 316 | postEvent(event) 317 | } 318 | 319 | return nodes, nil 320 | } 321 | 322 | func bind(pod *Pod, node Node) error { 323 | binding := Binding{ 324 | ApiVersion: "v1", 325 | Kind: "Binding", 326 | Metadata: Metadata{Name: pod.Metadata.Name}, 327 | Target: Target{ 328 | ApiVersion: "v1", 329 | Kind: "Node", 330 | Name: node.Metadata.Name, 331 | }, 332 | } 333 | 334 | var b []byte 335 | body := bytes.NewBuffer(b) 336 | err := json.NewEncoder(body).Encode(binding) 337 | if err != nil { 338 | return err 339 | } 340 | 341 | request := &http.Request{ 342 | Body: ioutil.NopCloser(body), 343 | ContentLength: int64(body.Len()), 344 | Header: make(http.Header), 345 | Method: http.MethodPost, 346 | URL: &url.URL{ 347 | Host: apiHost, 348 | Path: fmt.Sprintf(bindingsEndpoint, pod.Metadata.Name), 349 | Scheme: "http", 350 | }, 351 | } 352 | request.Header.Set("Content-Type", "application/json") 353 | 354 | resp, err := http.DefaultClient.Do(request) 355 | if err != nil { 356 | return err 357 | } 358 | if resp.StatusCode != 201 { 359 | return errors.New("Binding: Unexpected HTTP status code" + resp.Status) 360 | } 361 | 362 | // Emit a Kubernetes event that the Pod was scheduled successfully. 363 | message := fmt.Sprintf("Successfully assigned %s to %s", pod.Metadata.Name, node.Metadata.Name) 364 | timestamp := time.Now().UTC().Format(time.RFC3339) 365 | event := Event{ 366 | Count: 1, 367 | Message: message, 368 | Metadata: Metadata{GenerateName: pod.Metadata.Name + "-"}, 369 | Reason: "Scheduled", 370 | LastTimestamp: timestamp, 371 | FirstTimestamp: timestamp, 372 | Type: "Normal", 373 | Source: EventSource{Component: "hightower-scheduler"}, 374 | InvolvedObject: ObjectReference{ 375 | Kind: "Pod", 376 | Name: pod.Metadata.Name, 377 | Namespace: "default", 378 | Uid: pod.Metadata.Uid, 379 | }, 380 | } 381 | log.Println(message) 382 | return postEvent(event) 383 | } 384 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All Rights Reserved. 2 | // Licensed under the Apache License, Version 2.0 (the "License"); 3 | // you may not use this file except in compliance with the License. 4 | // You may obtain a copy of the License at 5 | // 6 | // http://www.apache.org/licenses/LICENSE-2.0 7 | // 8 | // Unless required by applicable law or agreed to in writing, software 9 | // distributed under the License is distributed on an "AS IS" BASIS, 10 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 11 | // See the License for the specific language governing permissions and 12 | // limitations under the License. 13 | 14 | package main 15 | 16 | import ( 17 | "log" 18 | "os" 19 | "os/signal" 20 | "sync" 21 | "syscall" 22 | ) 23 | 24 | const schedulerName = "hightower" 25 | 26 | func main() { 27 | log.Println("Starting custom scheduler...") 28 | 29 | doneChan := make(chan struct{}) 30 | var wg sync.WaitGroup 31 | 32 | wg.Add(1) 33 | go monitorUnscheduledPods(doneChan, &wg) 34 | 35 | wg.Add(1) 36 | go reconcileUnscheduledPods(30, doneChan, &wg) 37 | 38 | signalChan := make(chan os.Signal, 1) 39 | signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM) 40 | for { 41 | select { 42 | case <-signalChan: 43 | log.Printf("Shutdown signal received, exiting...") 44 | close(doneChan) 45 | wg.Wait() 46 | os.Exit(0) 47 | } 48 | } 49 | } 50 | -------------------------------------------------------------------------------- /processor.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All Rights Reserved. 2 | // Licensed under the Apache License, Version 2.0 (the "License"); 3 | // you may not use this file except in compliance with the License. 4 | // You may obtain a copy of the License at 5 | // 6 | // http://www.apache.org/licenses/LICENSE-2.0 7 | // 8 | // Unless required by applicable law or agreed to in writing, software 9 | // distributed under the License is distributed on an "AS IS" BASIS, 10 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 11 | // See the License for the specific language governing permissions and 12 | // limitations under the License. 13 | 14 | package main 15 | 16 | import ( 17 | "fmt" 18 | "log" 19 | "sync" 20 | "time" 21 | ) 22 | 23 | var processorLock = &sync.Mutex{} 24 | 25 | func reconcileUnscheduledPods(interval int, done chan struct{}, wg *sync.WaitGroup) { 26 | for { 27 | select { 28 | case <-time.After(time.Duration(interval) * time.Second): 29 | err := schedulePods() 30 | if err != nil { 31 | log.Println(err) 32 | } 33 | case <-done: 34 | wg.Done() 35 | log.Println("Stopped reconciliation loop.") 36 | return 37 | } 38 | } 39 | } 40 | 41 | func monitorUnscheduledPods(done chan struct{}, wg *sync.WaitGroup) { 42 | pods, errc := watchUnscheduledPods() 43 | 44 | for { 45 | select { 46 | case err := <-errc: 47 | log.Println(err) 48 | case pod := <-pods: 49 | processorLock.Lock() 50 | time.Sleep(2 * time.Second) 51 | err := schedulePod(&pod) 52 | if err != nil { 53 | log.Println(err) 54 | } 55 | processorLock.Unlock() 56 | case <-done: 57 | wg.Done() 58 | log.Println("Stopped scheduler.") 59 | return 60 | } 61 | } 62 | } 63 | 64 | func schedulePod(pod *Pod) error { 65 | nodes, err := fit(pod) 66 | if err != nil { 67 | return err 68 | } 69 | if len(nodes) == 0 { 70 | return fmt.Errorf("Unable to schedule pod (%s) failed to fit in any node", pod.Metadata.Name) 71 | } 72 | node, err := bestPrice(nodes) 73 | if err != nil { 74 | return err 75 | } 76 | err = bind(pod, node) 77 | if err != nil { 78 | return err 79 | } 80 | return nil 81 | } 82 | 83 | func schedulePods() error { 84 | processorLock.Lock() 85 | defer processorLock.Unlock() 86 | pods, err := getUnscheduledPods() 87 | if err != nil { 88 | return err 89 | } 90 | for _, pod := range pods { 91 | err := schedulePod(pod) 92 | if err != nil { 93 | log.Println(err) 94 | } 95 | } 96 | return nil 97 | } 98 | -------------------------------------------------------------------------------- /types.go: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Google Inc. All Rights Reserved. 2 | // Licensed under the Apache License, Version 2.0 (the "License"); 3 | // you may not use this file except in compliance with the License. 4 | // You may obtain a copy of the License at 5 | // 6 | // http://www.apache.org/licenses/LICENSE-2.0 7 | // 8 | // Unless required by applicable law or agreed to in writing, software 9 | // distributed under the License is distributed on an "AS IS" BASIS, 10 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 11 | // See the License for the specific language governing permissions and 12 | // limitations under the License. 13 | 14 | package main 15 | 16 | // Event is a report of an event somewhere in the cluster. 17 | type Event struct { 18 | ApiVersion string `json:"apiVersion,omitempty"` 19 | Count int64 `json:"count,omitempty"` 20 | FirstTimestamp string `json:"firstTimestamp"` 21 | LastTimestamp string `json:"lastTimestamp"` 22 | InvolvedObject ObjectReference `json:"involvedObject"` 23 | Kind string `json:"kind,omitempty"` 24 | Message string `json:"message,omitempty"` 25 | Metadata Metadata `json:"metadata"` 26 | Reason string `json:"reason,omitempty"` 27 | Source EventSource `json:"source,omitempty"` 28 | Type string `json:"type,omitempty"` 29 | } 30 | 31 | // EventSource contains information for an event. 32 | type EventSource struct { 33 | Component string `json:"component,omitempty"` 34 | Host string `json:"host,omitempty"` 35 | } 36 | 37 | // ObjectReference contains enough information to let you inspect or modify 38 | // the referred object. 39 | type ObjectReference struct { 40 | ApiVersion string `json:"apiVersion,omitempty"` 41 | Kind string `json:"kind,omitempty"` 42 | Name string `json:"name,omitempty"` 43 | Namespace string `json:"namespace,omitempty"` 44 | Uid string `json:"uid"` 45 | } 46 | 47 | // PodList is a list of Pods. 48 | type PodList struct { 49 | ApiVersion string `json:"apiVersion"` 50 | Kind string `json:"kind"` 51 | Metadata ListMetadata `json:"metadata"` 52 | Items []Pod `json:"items"` 53 | } 54 | 55 | type PodWatchEvent struct { 56 | Type string `json:"type"` 57 | Object Pod `json:"object"` 58 | } 59 | 60 | type Pod struct { 61 | Kind string `json:"kind,omitempty"` 62 | Metadata Metadata `json:"metadata"` 63 | Spec PodSpec `json:"spec"` 64 | } 65 | 66 | type PodSpec struct { 67 | NodeName string `json:"nodeName"` 68 | Containers []Container `json:"containers"` 69 | } 70 | 71 | type Container struct { 72 | Name string `json:"name"` 73 | Resources ResourceRequirements `json:"resources"` 74 | } 75 | 76 | type ResourceRequirements struct { 77 | Limits ResourceList `json:"limits"` 78 | Requests ResourceList `json:"requests"` 79 | } 80 | 81 | type ResourceList map[string]string 82 | 83 | type Binding struct { 84 | ApiVersion string `json:"apiVersion"` 85 | Kind string `json:"kind"` 86 | Target Target `json:"target"` 87 | Metadata Metadata `json:"metadata"` 88 | } 89 | 90 | type Target struct { 91 | ApiVersion string `json:"apiVersion"` 92 | Kind string `json:"kind"` 93 | Name string `json:"name"` 94 | } 95 | 96 | type NodeList struct { 97 | ApiVersion string `json:"apiVersion"` 98 | Kind string `json:"kind"` 99 | Items []Node 100 | } 101 | 102 | type Node struct { 103 | Metadata Metadata `json:"metadata"` 104 | Status NodeStatus `json:"status"` 105 | } 106 | 107 | type NodeStatus struct { 108 | Capacity ResourceList `json:"capacity"` 109 | Allocatable ResourceList `json:"allocatable"` 110 | } 111 | 112 | type ListMetadata struct { 113 | ResourceVersion string `json:"resourceVersion"` 114 | } 115 | 116 | type Metadata struct { 117 | Name string `json:"name"` 118 | GenerateName string `json:"generateName"` 119 | ResourceVersion string `json:"resourceVersion"` 120 | Labels map[string]string `json:"labels"` 121 | Annotations map[string]string `json:"annotations"` 122 | Uid string `json:"uid"` 123 | } 124 | --------------------------------------------------------------------------------