├── README.assets ├── image-20200911120003313.png ├── image-20200911120548684.png └── image-20200911120601595.png ├── README.md ├── go.mod ├── go.sum ├── logs └── dynamic-2020-12-14.log ├── main.go ├── nohup.out ├── prom └── prometheus.go ├── rule-template.yaml ├── template ├── deploynginx.yml ├── prometheus-cm.yaml └── push.sh └── utils ├── logger.go ├── type.go └── yaml.go /README.assets/image-20200911120003313.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zhangshunping/dynamicScheduler/1b7030c29a8e0c9faf23e16698f8cc463ba58eb9/README.assets/image-20200911120003313.png -------------------------------------------------------------------------------- /README.assets/image-20200911120548684.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zhangshunping/dynamicScheduler/1b7030c29a8e0c9faf23e16698f8cc463ba58eb9/README.assets/image-20200911120548684.png -------------------------------------------------------------------------------- /README.assets/image-20200911120601595.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zhangshunping/dynamicScheduler/1b7030c29a8e0c9faf23e16698f8cc463ba58eb9/README.assets/image-20200911120601595.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [toc] 2 | 3 | # dynamic-Scheduler 4 | 5 | 6 | 7 | ### 背景 8 | 9 | kubernetes调度策略默认根据node节点的request值（cpu和mem）进行调度，[dynamic-Scheduler](https://github.com/zhangshunping/dynamicScheduler)希望能够解决实际情况过程中遇到如下痛点 10 | 11 | - request设置太小，导致node节点上被分配太多的pod，面对流量高峰时，当前节点的资源使用率过高，导致一些pod 出现oom或者无法执行业务的情况。 12 | 13 | - request接近于limits,导致node节点上分配的pod数量有限，不能够最大容量的分配pod，而本身设置limits值的原因大多是基于高峰资源，而实际业务大多数情况并未处于高峰状态，所以会造成资源浪费。 14 | 15 | - 个性化原因： 16 | 17 | - 我们的业务模型创建的pod是自主性pod，虽然社区针对这样的问题有一些好用的开源的平衡组件[descheduler](https://github.com/kubernetes-sigs/descheduler)，但不满足实际的问题，我们需要的是一个在调度的时候就可以根据资源消耗情况选择调度的组件，而descheduler是通过controller的方式对正在集群内的pod进行平衡，不满足我们目前的需求。 18 | 19 | - kubelet内存驱逐条件（memory.available）,是通过计算cgroups下资源换算得来的，我们的业务是频繁的创建pod环境，然后销毁，可能会有一些pod跑业务的gc出现问题，导致内存无法释放，而此时free -h看到的内存可使用率仍然ok，但实际上/sys/fs/cgroups/memory下的memory.usage_in_bytes的值已经接近整个内存总量了。根据公式 20 | 21 | ```shell 22 | memory.available := node.status.capacity[memory]-node.stats.memory.workingSet 23 | ``` 24 | 25 | 此时memory.available已经低于节点驱逐的available值，就会出现pod被驱逐。因此对于cgroups下memory.available的值也需要做一定的监控和调度策略，kubectl top 命令计算好像不太准确，因此使用pushGateWay的方式，收集节点的memory.available值到prometheus。[根据kubelet计算memory.available脚本](https://kubernetes.io/docs/tasks/administer-cluster/memory-available.sh)进行push。 26 | 27 | ### 解决思路 28 | 29 | 通过监控获取到各个k8s节点的使用率（cpu，mem等），当超过阈值后，给对应节点打上status=presure，根据k8s软亲和性策略，让pod尽量不调度到打上的presure标签的节点。 30 | 31 | - prometheus监控k8s集群的node资源使用率（因为是云原生背景，所以要自动发现弹性节点node，并对弹性节点进行监控）。需注意 32 | 33 | - prometheus.yaml文件中采集node资源时，需要用如下的动态relabel机制，因为，client_golang采集Prometheus的metrics的数据为string，在dynamic-scheduler中使用*ConvertResultDataType*函数将string转换成[]string格式,这里的values为instance的值。 34 | - 需要安装pushgateway，然后crontab的方式运行push.sh脚本，进行memory.available的采集 35 | 36 | ``` 37 | 38 | - job_name: '测试环境k8s资源节点监控' 39 | kubernetes_sd_configs: 40 | - role: node 41 | relabel_configs: 42 | - source_labels: [__meta_kubernetes_node_address_InternalIP] 43 | regex: '(.*)' 44 | replacement: '${1}:9100' 45 | target_label: __address__ 46 | - job_name: 'pushgateway' 47 | honor_labels: true 48 | static_configs: 49 | - targets: ['172.16.95.4:49091'] 50 | 51 | 52 | 53 | ``` 54 | 55 | 56 | 57 | - 判断prometheus的返回得node使用率的metrics是否超过阈值，从而给k8s node节点打标签status=presure,根据k8s default的调度策略的软亲和preferredDuringSchedulingIgnoredDuringExecution策略，调度的时候尽量不调度到被打上标签status=presure的节点及当前资源紧张的节点。 58 | 59 | ```shell 60 | template: 61 | metadata: 62 | labels: 63 | run: nginx 64 | spec: 65 | affinity: 66 | nodeAffinity: 67 | preferredDuringSchedulingIgnoredDuringExecution: 68 | - preference: 69 | matchExpressions: 70 | - key: status 71 | operator: NotIn 72 | values: 73 | - presure 74 | weight: 20 75 | ``` 76 | 77 | - 考虑到apiserver的压力问题，使用client—go 原生informer机制，减少对apiserver的压力。 78 | 79 | - promethues和dynamic-Scheduler 挂了情况，也不能影响到实际业务，因为使用的软亲和的方式，所以最终一定会有node被调度 80 | 81 | 82 | 83 | ### 使用方式 84 | 85 | ```shell 86 | 87 | Usage of ./dynamicScheduler: 88 | -kubeconfig string 89 | 链接k8s kubeconfig的绝对路径 (default "/root/gocode/src/config") 90 | -prom string 91 | prometheus链接地址(-prom http://121.40.XX.XX:49090) (default "http://121.40.XX.XX:49090") 92 | -r string 93 | 查询prometheus 阈值规则 (default "rule.yaml") 94 | -s int 95 | 每次抓取prometheus metrics间隔（-s 10) (default 10) 96 | -webaddr string 97 | 启动服务端口地址(-webaddr :9000) (default ":9000") 98 | 99 | ``` 100 | 101 | - 健康检测 102 | 103 | ``` 104 | curl ip:port/status 105 | ``` 106 | 107 | ![image-20200911120601595](README.assets/image-20200911120601595.png) 108 | 109 | - 查询多少个节点打上了type=presure标签（返回json) 110 | 111 | ```shell 112 | curl ip:port/nodes 113 | ``` 114 | 115 | ![image-20200911120548684](README.assets/image-20200911120548684.png) 116 | 117 | 118 | 119 | 120 | 121 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module dynamicScheduler 2 | 3 | go 1.14 4 | 5 | require ( 6 | github.com/antonfisher/nested-logrus-formatter v1.2.0 7 | github.com/googleapis/gnostic v0.5.3 // indirect 8 | github.com/imdario/mergo v0.3.11 // indirect 9 | github.com/prometheus/client_golang v1.7.1 10 | github.com/sirupsen/logrus v1.4.2 11 | golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d // indirect 12 | golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e // indirect 13 | gopkg.in/yaml.v2 v2.3.0 14 | k8s.io/api v0.18.0 15 | k8s.io/apimachinery v0.19.1 16 | k8s.io/client-go v0.18.0 17 | k8s.io/utils v0.0.0-20200821003339-5e75c0163111 // indirect 18 | 19 | ) 20 | 21 | replace github.com/googleapis/gnostic v0.5.3 => github.com/googleapis/gnostic v0.1.0 22 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= 2 | cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= 3 | cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU= 4 | github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI= 5 | github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0= 6 | github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA= 7 | github.com/Azure/go-autorest/autorest/mocks v0.1.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= 8 | github.com/Azure/go-autorest/autorest/mocks v0.2.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= 9 | github.com/Azure/go-autorest/logger v0.1.0/go.mod h1:oExouG+K6PryycPJfVSxi/koC6LSNgds39diKLz7Vrc= 10 | github.com/Azure/go-autorest/tracing v0.5.0/go.mod h1:r/s2XiOKccPW3HrqB+W0TQzfbtp2fGCgRFtBroKn4Dk= 11 | github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= 12 | github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ= 13 | github.com/PuerkitoBio/purell v1.0.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= 14 | github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= 15 | github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= 16 | github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= 17 | github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= 18 | github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= 19 | github.com/antonfisher/nested-logrus-formatter v1.2.0 h1:bbo1NIVWAspzyQZbzudw4sgntja0Ldx+GM5PoBJ7xgM= 20 | github.com/antonfisher/nested-logrus-formatter v1.2.0/go.mod h1:6WTfyWFkBc9+zyBaKIqRrg/KwMqBbodBjgbHjDz7zjA= 21 | github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= 22 | github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= 23 | github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= 24 | github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= 25 | github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= 26 | github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= 27 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 28 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 29 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 30 | github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= 31 | github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM= 32 | github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE= 33 | github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= 34 | github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= 35 | github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= 36 | github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= 37 | github.com/evanphx/json-patch v4.2.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= 38 | github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= 39 | github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= 40 | github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= 41 | github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= 42 | github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= 43 | github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= 44 | github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE= 45 | github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= 46 | github.com/go-logr/logr v0.1.0 h1:M1Tv3VzNlEHg6uyACnRdtrploV2P7wZqH8BoQMtz0cg= 47 | github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas= 48 | github.com/go-logr/logr v0.2.0 h1:QvGt2nLcHH0WK9orKa+ppBPAxREcH364nPUedEpK0TY= 49 | github.com/go-logr/logr v0.2.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU= 50 | github.com/go-openapi/jsonpointer v0.0.0-20160704185906-46af16f9f7b1/go.mod h1:+35s3my2LFTysnkMfxsJBAMHj/DoqoB9knIWoYG/Vk0= 51 | github.com/go-openapi/jsonreference v0.0.0-20160704190145-13c6e3589ad9/go.mod h1:W3Z9FmVs9qj+KR4zFKmDPGiLdk1D9Rlm7cyMvf57TTg= 52 | github.com/go-openapi/spec v0.0.0-20160808142527-6aced65f8501/go.mod h1:J8+jY1nAiCcj+friV/PDoE1/3eeccG9LYBs0tYvLOWc= 53 | github.com/go-openapi/swag v0.0.0-20160704191624-1d0bd113de87/go.mod h1:DXUve3Dpr1UfpPtxFw+EFuQ41HhCWZfha5jSVRG7C7I= 54 | github.com/go-stack/stack v1.8.0 h1:5SgMzNM5HxrEjV0ww2lTmX6E2Izsfxas4+YHWRs3Lsk= 55 | github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY= 56 | github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= 57 | github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls= 58 | github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= 59 | github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= 60 | github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903 h1:LbsanbbD6LieFkXbj9YNNBupiGHJgFeLpO0j0Fza1h8= 61 | github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= 62 | github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA= 63 | github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= 64 | github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= 65 | github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= 66 | github.com/golang/protobuf v0.0.0-20161109072736-4bd1920723d7/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 67 | github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 68 | github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 69 | github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs= 70 | github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 71 | github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= 72 | github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= 73 | github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= 74 | github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= 75 | github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= 76 | github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= 77 | github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= 78 | github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0= 79 | github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= 80 | github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= 81 | github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= 82 | github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= 83 | github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY= 84 | github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 85 | github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 86 | github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4= 87 | github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 88 | github.com/google/gofuzz v1.0.0 h1:A8PeW59pxE9IoFRqBp37U+mSNaQoZ46F1f0f863XSXw= 89 | github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= 90 | github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g= 91 | github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= 92 | github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= 93 | github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= 94 | github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= 95 | github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= 96 | github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= 97 | github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d h1:7XGaL1e6bYS1yIonGp9761ExpPPV1ui0SAC59Yube9k= 98 | github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY= 99 | github.com/googleapis/gnostic v0.1.0 h1:rVsPeBmXbYv4If/cumu1AzZPwV58q433hvONV1UEZoI= 100 | github.com/googleapis/gnostic v0.1.0/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY= 101 | github.com/googleapis/gnostic v0.4.1 h1:DLJCy1n/vrD4HPjOvYcT8aYQXpPIzoRZONaYwyycI+I= 102 | github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg= 103 | github.com/gophercloud/gophercloud v0.1.0/go.mod h1:vxM41WHh5uqHVBMZHzuwNOHh8XEoIEcSTewFxm1c5g8= 104 | github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= 105 | github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= 106 | github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU= 107 | github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= 108 | github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= 109 | github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= 110 | github.com/imdario/mergo v0.3.11 h1:3tnifQM4i+fbajXKBHXWEH+KvNHqojZ778UH75j3bGA= 111 | github.com/imdario/mergo v0.3.11/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= 112 | github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= 113 | github.com/json-iterator/go v1.1.8 h1:QiWkFLKq0T7mpzwOTu6BzNDbfTE8OLrYhVKYMLF46Ok= 114 | github.com/json-iterator/go v1.1.8/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= 115 | github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68= 116 | github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= 117 | github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= 118 | github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= 119 | github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= 120 | github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= 121 | github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk= 122 | github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= 123 | github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc= 124 | github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= 125 | github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= 126 | github.com/kr/pretty v0.2.0 h1:s5hAObm+yFO5uHYt5dYjxi2rXrsnmRpJx4OYvIWUaQs= 127 | github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= 128 | github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= 129 | github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= 130 | github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= 131 | github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= 132 | github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= 133 | github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= 134 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= 135 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= 136 | github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= 137 | github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI= 138 | github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= 139 | github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= 140 | github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U= 141 | github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw= 142 | github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 143 | github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 144 | github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 145 | github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= 146 | github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= 147 | github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= 148 | github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 149 | github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 150 | github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 151 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 152 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 153 | github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= 154 | github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo= 155 | github.com/prometheus/client_golang v1.7.1 h1:NTGy1Ja9pByO+xAeH/qiWnLrKtr3hJPNjaVUwnjpdpA= 156 | github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M= 157 | github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= 158 | github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= 159 | github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= 160 | github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= 161 | github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= 162 | github.com/prometheus/common v0.10.0 h1:RyRA7RzGXQZiW+tGMr7sxa85G1z0yOpM1qq5c8lNawc= 163 | github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo= 164 | github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= 165 | github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= 166 | github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU= 167 | github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= 168 | github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4= 169 | github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= 170 | github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk= 171 | github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= 172 | github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= 173 | github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= 174 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 175 | github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 176 | github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= 177 | github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 178 | github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= 179 | github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= 180 | go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= 181 | golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= 182 | golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= 183 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2 h1:VklqNMn3ovrHsnt90PveolxSbWFaJdECFbxSq0Mqo2M= 184 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 185 | golang.org/x/crypto v0.0.0-20200220183623-bac4c82f6975 h1:/Tl7pH94bvbAAHBdZJT947M/+gp0+CqQXDtMRC0fseo= 186 | golang.org/x/crypto v0.0.0-20200220183623-bac4c82f6975/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 187 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI= 188 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 189 | golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= 190 | golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= 191 | golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= 192 | golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= 193 | golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 194 | golang.org/x/net v0.0.0-20170114055629-f2499483f923/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 195 | golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 196 | golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 197 | golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 198 | golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 199 | golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 200 | golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 201 | golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 202 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 203 | golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 204 | golang.org/x/net v0.0.0-20191004110552-13f9640d40b9 h1:rjwSpXsdiK0dV8/Naq3kAw9ymfAeJIyd0upUIElB+lI= 205 | golang.org/x/net v0.0.0-20191004110552-13f9640d40b9/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 206 | golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= 207 | golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU= 208 | golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= 209 | golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= 210 | golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= 211 | golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= 212 | golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw= 213 | golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= 214 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 215 | golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 216 | golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 217 | golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 218 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 219 | golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 220 | golang.org/x/sys v0.0.0-20170830134202-bb24a47a89ea/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 221 | golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 222 | golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 223 | golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 224 | golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 225 | golang.org/x/sys v0.0.0-20190209173611-3b5209105503/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 226 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 227 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 228 | golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 229 | golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 230 | golang.org/x/sys v0.0.0-20191022100944-742c48ecaeb7 h1:HmbHVPwrPEKPGLAcHSrMe6+hqSUlvZU0rab6x5EXfGU= 231 | golang.org/x/sys v0.0.0-20191022100944-742c48ecaeb7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 232 | golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 233 | golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 234 | golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1 h1:ogLJMz+qpzav7lGMh10LMvAkM/fAoGlaiiHYiFYdm80= 235 | golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 236 | golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4 h1:5/PjkGUjvEU5Gl6BxmvKRPpqo2uNMv4rcHBMwzk/st8= 237 | golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 238 | golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 239 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 240 | golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 241 | golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs= 242 | golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= 243 | golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k= 244 | golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= 245 | golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= 246 | golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= 247 | golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e h1:EHBhcS0mlXEAVwNyO2dLfjToGsyY4j24pTs2ScHnX7s= 248 | golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= 249 | golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 250 | golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 251 | golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 252 | golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 253 | golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= 254 | golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 255 | golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 256 | golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= 257 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= 258 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 259 | google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= 260 | google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= 261 | google.golang.org/appengine v1.4.0 h1:/wp5JvzpHIxhs/dumFmF7BXTf3Z+dd4uXta4kVyO508= 262 | google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= 263 | google.golang.org/appengine v1.5.0 h1:KxkO13IPW4Lslp2bz+KHP2E3gtFlrIGNThxkZQ3g+4c= 264 | google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= 265 | google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= 266 | google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= 267 | google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= 268 | google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= 269 | google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= 270 | google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= 271 | google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= 272 | google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= 273 | google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= 274 | google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= 275 | google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= 276 | google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= 277 | google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= 278 | google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 279 | google.golang.org/protobuf v1.23.0 h1:4MY060fB1DLGMB/7MBTLnwQUY6+F09GEiz6SsrNqyzM= 280 | google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 281 | google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 282 | google.golang.org/protobuf v1.24.0 h1:UhZDfRO8JRQru4/+LlLE0BRKGF8L+PICnvYZmx/fEGA= 283 | google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4= 284 | gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= 285 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 286 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= 287 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 288 | gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= 289 | gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 290 | gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= 291 | gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= 292 | gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= 293 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= 294 | gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 295 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 296 | gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 297 | gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 298 | gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10= 299 | gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 300 | gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU= 301 | gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 302 | honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 303 | honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 304 | honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 305 | k8s.io/api v0.18.0 h1:lwYk8Vt7rsVTwjRU6pzEsa9YNhThbmbocQlKvNBB4EQ= 306 | k8s.io/api v0.18.0/go.mod h1:q2HRQkfDzHMBZL9l/y9rH63PkQl4vae0xRT+8prbrK8= 307 | k8s.io/apimachinery v0.18.0 h1:fuPfYpk3cs1Okp/515pAf0dNhL66+8zk8RLbSX+EgAE= 308 | k8s.io/apimachinery v0.18.0/go.mod h1:9SnR/e11v5IbyPCGbvJViimtJ0SwHG4nfZFjU77ftcA= 309 | k8s.io/apimachinery v0.19.1 h1:cwsxZazM/LA9aUsBaL4bRS5ygoM6bYp8dFk22DSYQa4= 310 | k8s.io/apimachinery v0.19.1/go.mod h1:DnPGDnARWFvYa3pMHgSxtbZb7gpzzAZ1pTfaUNDVlmA= 311 | k8s.io/client-go v0.18.0 h1:yqKw4cTUQraZK3fcVCMeSa+lqKwcjZ5wtcOIPnxQno4= 312 | k8s.io/client-go v0.18.0/go.mod h1:uQSYDYs4WhVZ9i6AIoEZuwUggLVEF64HOD37boKAtF8= 313 | k8s.io/gengo v0.0.0-20190128074634-0689ccc1d7d6/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0= 314 | k8s.io/gengo v0.0.0-20200413195148-3a45101e95ac/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0= 315 | k8s.io/klog v0.0.0-20181102134211-b9b56d5dfc92/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk= 316 | k8s.io/klog v0.3.0/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk= 317 | k8s.io/klog v1.0.0 h1:Pt+yjF5aB1xDSVbau4VsWe+dQNzA0qv1LlXdC2dF6Q8= 318 | k8s.io/klog v1.0.0/go.mod h1:4Bi6QPql/J/LkTDqv7R/cd3hPo4k2DG6Ptcz060Ez5I= 319 | k8s.io/klog/v2 v2.0.0 h1:Foj74zO6RbjjP4hBEKjnYtjjAhGg4jNynUdYF6fJrok= 320 | k8s.io/klog/v2 v2.0.0/go.mod h1:PBfzABfn139FHAV07az/IF9Wp1bkk3vpT2XSJ76fSDE= 321 | k8s.io/klog/v2 v2.2.0 h1:XRvcwJozkgZ1UQJmfMGpvRthQHOvihEhYtDfAaxMz/A= 322 | k8s.io/klog/v2 v2.2.0/go.mod h1:Od+F08eJP+W3HUb4pSrPpgp9DGU4GzlpG/TmITuYh/Y= 323 | k8s.io/kube-openapi v0.0.0-20200121204235-bf4fb3bd569c/go.mod h1:GRQhZsXIAJ1xR0C9bd8UpWHZ5plfAS9fzPjJuQ6JL3E= 324 | k8s.io/kube-openapi v0.0.0-20200805222855-6aeccd4b50c6/go.mod h1:UuqjUnNftUyPE5H64/qeyjQoUZhGpeFDVdxjTeEVN2o= 325 | k8s.io/utils v0.0.0-20200324210504-a9aa75ae1b89/go.mod h1:sZAwmy6armz5eXlNoLmJcl4F1QuKu7sr+mFQ0byX7Ew= 326 | k8s.io/utils v0.0.0-20200821003339-5e75c0163111 h1:AChSIFe1D4vQ5XkklbH491v1ONSmnt8fnb235DsAw1U= 327 | k8s.io/utils v0.0.0-20200821003339-5e75c0163111/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= 328 | sigs.k8s.io/structured-merge-diff/v3 v3.0.0-20200116222232-67a7b8c61874/go.mod h1:PlARxl6Hbt/+BC80dRLi1qAmnMqwqDg62YvvVkZjemw= 329 | sigs.k8s.io/structured-merge-diff/v3 v3.0.0 h1:dOmIZBMfhcHS09XZkMyUgkq5trg3/jRyJYFZUiaOp8E= 330 | sigs.k8s.io/structured-merge-diff/v3 v3.0.0/go.mod h1:PlARxl6Hbt/+BC80dRLi1qAmnMqwqDg62YvvVkZjemw= 331 | sigs.k8s.io/structured-merge-diff/v4 v4.0.1 h1:YXTMot5Qz/X1iBRJhAt+vI+HVttY0WkSqqhKxQ0xVbA= 332 | sigs.k8s.io/structured-merge-diff/v4 v4.0.1/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw= 333 | sigs.k8s.io/yaml v1.1.0 h1:4A07+ZFc2wgJwo8YNlQpr1rVlgUDlxXHhPJciaPY5gs= 334 | sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o= 335 | sigs.k8s.io/yaml v1.2.0 h1:kr/MCeFWJWTwyaHoR9c8EjH9OumOmoF9YGiZd7lFm/Q= 336 | sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc= 337 | -------------------------------------------------------------------------------- /logs/dynamic-2020-12-14.log: -------------------------------------------------------------------------------- 1 | 1 2 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "context" 5 | "dynamicScheduler/prom" 6 | "dynamicScheduler/utils" 7 | "encoding/json" 8 | "flag" 9 | "fmt" 10 | "github.com/prometheus/client_golang/api" 11 | promv1 "github.com/prometheus/client_golang/api/prometheus/v1" 12 | "io" 13 | v1 "k8s.io/api/core/v1" 14 | metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" 15 | "k8s.io/apimachinery/pkg/labels" 16 | "k8s.io/apimachinery/pkg/selection" 17 | "k8s.io/apimachinery/pkg/types" 18 | "k8s.io/apimachinery/pkg/util/runtime" 19 | "k8s.io/client-go/informers" 20 | "k8s.io/client-go/kubernetes" 21 | "k8s.io/client-go/tools/cache" 22 | "k8s.io/client-go/tools/clientcmd" 23 | "net/http" 24 | "os" 25 | "path/filepath" 26 | "strconv" 27 | "time" 28 | ) 29 | 30 | var ( 31 | FitSelectorAndAlreadyLabelPresureNodes []*v1.Node //符合标签选择器， 32 | FitSelectorAndAlreadyLabelPresureNodeNames []string //集群内已经上压力的node Names 33 | FitSelectorNodes []*v1.Node //根据标签选择器获取的当前node slice 34 | presureNodesNameFromProm []string 35 | count, scrape_interval int 36 | promAddress,webaddr,rulepath string 37 | UpperPresureNodeNames []string 38 | ) 39 | 40 | const ( 41 | CgroupsMemOpen =true 42 | CgroupsMemClose =false 43 | ) 44 | type Response struct { 45 | Type string `json:"type"` 46 | Num int `json:"num"` 47 | NodeNames []string `json:"node_names"` 48 | 49 | } 50 | 51 | func init() { 52 | flag.IntVar(&scrape_interval, "s", 10, "每次抓取prometheus metrics间隔（-s 10)") 53 | flag.StringVar(&promAddress, "prom", "http://121.40.XX.XX:49090", "prometheus链接地址(-prom http://121.40.XX.XX:49090)") 54 | flag.StringVar(&webaddr, "webaddr", ":9000", "启动服务端口地址(-webaddr :9000)") 55 | flag.StringVar(&rulepath,"r","rule.yaml" , "查询prometheus 阈值规则") 56 | } 57 | 58 | func main() { 59 | ctx := context.Background() 60 | stopCh := make(chan struct{}) 61 | //1.k8sClient初始化 62 | var kubeconfig *string 63 | if home := homeDir(); home != "" { 64 | kubeconfig = flag.String("kubeconfig", filepath.Join(home, "src", "config"), "链接k8s kubeconfig的绝对路径") 65 | } else { 66 | kubeconfig = flag.String("kubeconfig", "", "链接k8s kubeconfig绝对路径") 67 | } 68 | flag.Parse() 69 | // use the current context in kubeconfig 70 | config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig) 71 | if err != nil { 72 | panic(err.Error()) 73 | } 74 | 75 | // create the clientset 76 | clientset, err := kubernetes.NewForConfig(config) 77 | if err != nil { 78 | panic(err.Error()) 79 | os.Exit(0) 80 | } 81 | 82 | //2.prometheus 客户都初始化 83 | client, err := api.NewClient(api.Config{ 84 | Address: promAddress, 85 | }) 86 | if err != nil { 87 | utils.Log.Error("Error creating client: %v\n", err) 88 | os.Exit(1) 89 | } 90 | v1api := promv1.NewAPI(client) 91 | //3.k8s NewSharedInfomerFactory 92 | factory := informers.NewSharedInformerFactory(clientset, 30*time.Second) 93 | //4.跟据prometheus获取对应的metrics项目，发现超出的阈值的节点，则给其打上type=presure标签 94 | LabelNodeByPromMetrics(stopCh, factory, ctx, clientset, v1api, scrape_interval) 95 | // 设置多路复用处理函数, 为了健康检查 96 | mux := http.NewServeMux() 97 | mux.HandleFunc("/nodes", ListPresureNodes) 98 | mux.HandleFunc("/status", Healyth) 99 | // 设置服务器 100 | server := &http.Server{ 101 | Addr: webaddr, 102 | Handler: mux, 103 | } 104 | // 设置服务器监听请求端口 105 | server.ListenAndServe() 106 | 107 | <-stopCh 108 | 109 | } 110 | 111 | // 设置多个处理器函数 112 | func ListPresureNodes(w http.ResponseWriter, r *http.Request) { 113 | response := Response{ 114 | Type: "Label type=presure Nodes", 115 | Num: len(FitSelectorAndAlreadyLabelPresureNodeNames), 116 | NodeNames: FitSelectorAndAlreadyLabelPresureNodeNames, 117 | } 118 | jsonbyte, err := json.Marshal(response) 119 | if err != nil { 120 | utils.Log.Error("struct --> json", err) 121 | } 122 | io.WriteString(w, string(jsonbyte)) 123 | 124 | } 125 | 126 | func Healyth(w http.ResponseWriter, r *http.Request) { 127 | io.WriteString(w, "ok") 128 | } 129 | 130 | 131 | func LabelNodeByPromMetrics(stopCh <-chan struct{}, factory informers.SharedInformerFactory, 132 | ctx context.Context, clientset *kubernetes.Clientset, v1api promv1.API, scrape_interval int) { 133 | 134 | //1 node Informer 135 | nodeInformer := factory.Core().V1().Nodes() 136 | //1.1开启node informer 137 | go nodeInformer.Informer().Run(stopCh) 138 | //1.2从k8s中同步list node 139 | if !cache.WaitForCacheSync(stopCh, nodeInformer.Informer().HasSynced) { 140 | runtime.HandleError(fmt.Errorf("Timed out waiting for caches to sync")) 141 | return 142 | } 143 | //2.node selector, 返回打了others和persure节点*[]v1.Node 144 | //nodeOthersSlice := []string{"others"} 145 | nodeOthersSlice2 := []string{"presure"} 146 | //selector, _ := labels.NewRequirement("type", selection.In, nodeOthersSlice) 147 | selector2, _ := labels.NewRequirement("status", selection.In, nodeOthersSlice2) 148 | //3.go程循环监听prometheus,根据metrics跟node加上label 149 | go func() { 150 | for { 151 | count = count + 1 152 | now := time.Now() 153 | utils.Log.Infof("Staring,id=%v", count) 154 | //查询所有节点all 155 | FitSelectorNodes, _ = nodeInformer.Lister().List(labels.NewSelector()) 156 | // 监控使用，查看节点内已经打上presure节点的node 157 | FitSelectorAndAlreadyLabelPresureNodes, _ = nodeInformer.Lister().List(labels.NewSelector().Add(*selector2)) 158 | FitSelectorAndAlreadyLabelPresureNodeNames = []string{} //清空所有的Nodes 159 | for i := 0; i < len(FitSelectorAndAlreadyLabelPresureNodes); i++ { 160 | FitSelectorAndAlreadyLabelPresureNodeNames = append(FitSelectorAndAlreadyLabelPresureNodeNames, FitSelectorAndAlreadyLabelPresureNodes[i].Name) 161 | } 162 | //3.1获取超出阈值的node节点 names 163 | presureNodesNameFromProm = []string{} // 清空presureNodesName nodes 164 | for k, v := range(utils.GetRuleFromYaml(rulepath)) { 165 | m,_:=prom.QueryRebuild(v1api, ctx,v["Promsql"],time.Now()) 166 | th:=v["Threshold"] 167 | floatth,_:=strconv.ParseFloat(th, 64) 168 | UpperPresureNodeNames = CountUpperPresureNodeFromProm(m, floatth,k,CgroupsMemClose) 169 | utils.Log.Infof("%s 超过%v的节点列表：%v", k,floatth,UpperPresureNodeNames) 170 | } 171 | 172 | /* 旧代码逻辑 173 | //3.2计算过去一分钟cpu的使用率 174 | utils.Log.Info("============Cpu==========") 175 | resultFromPromSilceMap, _ := prom.QueryRebuild(v1api, ctx, prom.Node_cpu1, time.Now()) 176 | UpperPresureNodeNames := CountUpperPresureNodeFromProm(resultFromPromSilceMap, cpuThreshold, "cpu",CgroupsMemClose) 177 | utils.Log.Infof("Cpu使用率超过%v的节点列表：%v", cpuThreshold,UpperPresureNodeNames) 178 | //3.3计算mem使用率 179 | utils.Log.Info("============Mem==========") 180 | resultFromPromSilceMapMem, _ := prom.QueryRebuild(v1api, ctx, prom.Node_mem, time.Now()) 181 | UpperPresureNodeNames = CountUpperPresureNodeFromProm(resultFromPromSilceMapMem, memThreshold, "mem",CgroupsMemClose) 182 | utils.Log.Infof("Mem 使用率超过阈值%v的节点列表: %v ", memThreshold, UpperPresureNodeNames) 183 | //计算/sys/fs/cgroup/ 下 memory.avaialbe的值（及kubelet发生驱逐的值） 184 | CgroupMems,_:=prom.QueryRebuild(v1api,ctx,prom.K8_node_cgroups_mem_available,time.Now()) 185 | UpperPresureNodeNames = CountUpperPresureNodeFromProm(CgroupMems, CgroupsMemThreshold, "CgoupsMemAvailable",CgroupsMemOpen) 186 | utils.Log.Infof("节点memory.available低于%v阈值的节点列表： %v ", CgroupsMemThreshold, UpperPresureNodeNames) 187 | utils.Log.Info("============Label==========") 188 | */ 189 | 190 | //3.2清空v1.Node列表(打赏presure标签的Nodes和去掉标签的Nodes) 191 | ReadyForLabelPresure := []*v1.Node{} 192 | ReadyForLabelNil := []*v1.Node{} 193 | //3.3查询需要节点的nodes 194 | for _, v := range FitSelectorNodes { 195 | key := v.Name 196 | if IsExitArray(key, UpperPresureNodeNames) { 197 | ReadyForLabelPresure = append(ReadyForLabelPresure, v) 198 | } else { 199 | ReadyForLabelNil = append(ReadyForLabelNil, v) 200 | } 201 | } 202 | //3.4 label status=presure 和去掉 status=presure 203 | PatchNode(clientset, ctx, "presure", ReadyForLabelPresure) 204 | //3.5需补充，对已经为nil的节点则不重复patch为nil*****-****************** 205 | PatchNode(clientset, ctx, "nil", ReadyForLabelNil) 206 | 207 | utils.Log.Infof("当前有%d个节点已经处于status=presure状态: %v", len(FitSelectorAndAlreadyLabelPresureNodeNames), FitSelectorAndAlreadyLabelPresureNodeNames) 208 | utils.Log.Infof("Ending,id=%v,耗时%v", count, time.Since(now)) 209 | utils.Log.Info("<-------------------------------------->") 210 | 211 | time.Sleep(time.Second * time.Duration(scrape_interval)) 212 | } 213 | }() 214 | 215 | } 216 | 217 | //patch node label 218 | func PatchNode(clientset *kubernetes.Clientset, ctx context.Context, selection string, Nodes []*v1.Node) { 219 | //const selection pressure or nil 220 | var labelvaule interface{} 221 | labelkey := "status" 222 | switch selection { 223 | //去掉标签 224 | case "nil": 225 | labelvaule = nil 226 | //打上标签 227 | case "presure": 228 | labelvaule = fmt.Sprintf("%s", selection) 229 | } 230 | patchTemplate := map[string]interface{}{ 231 | "metadata": map[string]interface{}{ 232 | "labels": map[string]interface{}{ 233 | labelkey: labelvaule, 234 | }, 235 | }, 236 | } 237 | patchdata, _ := json.Marshal(patchTemplate) 238 | for i := 0; i < len(Nodes); i++ { 239 | //master节点不做label处理 240 | if _, ok := Nodes[i].Labels["node-role.kubernetes.io/master"]; !ok { 241 | _, err := clientset.CoreV1().Nodes().Patch(ctx, Nodes[i].Name, types.StrategicMergePatchType, patchdata, metav1.PatchOptions{}) 242 | if err == nil { 243 | utils.Log.Infof("给节点%s打type=%s标签成功", Nodes[i].Name, selection) 244 | } else { 245 | 246 | utils.Log.Errorf("给节点%s打%s标签失败，错误为%v", Nodes[i].Name, selection, err) 247 | } 248 | } else { 249 | utils.Log.Infof("%s节点为master节点，不参与平衡调度标签策略", Nodes[i].Name) 250 | } 251 | 252 | } 253 | } 254 | 255 | // 根据n阈值，筛选超出阈值node节点（master和slave) 256 | func CountUpperPresureNodeFromProm(resultFromPromSilceMap []map[string]string, threshold float64, metricName string,cgroupsbool bool) []string { 257 | for i := 0; i < len(resultFromPromSilceMap); i++ { 258 | currentMetrics, _ := strconv.ParseFloat(resultFromPromSilceMap[i]["value"], 64) 259 | utils.Log.Infof("%s节点当前%s使用率为: %v", resultFromPromSilceMap[i]["instance"], metricName, currentMetrics) 260 | // cgroupsMem判断及 kubelet驱逐指标 memory.available的判断 261 | if cgroupsbool{ 262 | if currentMetrics threshold { 270 | if !IsExitArray(resultFromPromSilceMap[i]["instance"], presureNodesNameFromProm) { 271 | presureNodesNameFromProm = append(presureNodesNameFromProm, resultFromPromSilceMap[i]["instance"]) 272 | } 273 | } 274 | } 275 | return presureNodesNameFromProm 276 | } 277 | 278 | func homeDir() string { 279 | if h := os.Getenv("GOPATH"); h != "" { 280 | return h 281 | } 282 | return os.Getenv("USERPROFILE") 283 | } 284 | 285 | func IsExitArray(value string, arry []string) bool { 286 | for _, v := range arry { 287 | if v == value { 288 | return true 289 | } 290 | } 291 | return false 292 | } 293 | 294 | -------------------------------------------------------------------------------- /nohup.out: -------------------------------------------------------------------------------- 1 | /home/zsp/dynamicScheduler/logs/dynamic-2020-09-11.log 2 | -------------------------------------------------------------------------------- /prom/prometheus.go: -------------------------------------------------------------------------------- 1 | package prom 2 | 3 | import ( 4 | "context" 5 | "dynamicScheduler/utils" 6 | "fmt" 7 | "os" 8 | "strings" 9 | "time" 10 | //"unicode" 11 | 12 | //"github.com/prometheus/client_golang/api" 13 | v1 "github.com/prometheus/client_golang/api/prometheus/v1" 14 | ) 15 | 16 | 17 | // promethues merics 18 | var ( 19 | //PrometheusJob="测试环境k8s资源节点监控" 20 | 21 | PrometheusJob string 22 | Node_load15 string = fmt.Sprintf("node_load15{job='%s'}", PrometheusJob) 23 | Node_load5 string = fmt.Sprintf("node_load5{job='%s'}", PrometheusJob) 24 | Node_load1 string = fmt.Sprintf("node_load1{job='%s'}", PrometheusJob) 25 | //节点过去一分钟cpu 使用率 26 | Node_cpu1 string=fmt.Sprintf("(1-(sum(increase(node_cpu_seconds_total{job='%v' ,mode=\"idle\"}[1m]))by(instance))/(sum(increase(node_cpu_seconds_total{job='%s'}[1m]))by(instance)))*100",PrometheusJob,PrometheusJob) 27 | //查询内存使用使用率 28 | Node_mem string=fmt.Sprintf("(1 - (node_memory_MemAvailable_bytes{job='%v'} / (node_memory_MemTotal_bytes{job='%s'})))* 100",PrometheusJob,PrometheusJob) 29 | //k8s 节点/sys/fs/cgroup/memory Memory.Available 资源，kubelet Evcited memory驱逐根据此选项进行驱逐 30 | K8_node_cgroups_mem_available string=fmt.Sprintf("k8_node_cgroups_mem_available{job='CollectK8sCgroupsMem'}") 31 | ) 32 | 33 | 34 | 35 | 36 | //prometheus query 重构prometheus query 37 | func QueryRebuild(v1api v1.API,ctx context.Context, query string, ts time.Time) ([]map[string]string ,bool){ 38 | result, warnings, err := v1api.Query(ctx, query, ts) 39 | if err != nil { 40 | utils.Log.Error("Error querying Prometheus: %s\n", err) 41 | os.Exit(1) 42 | } 43 | if len(warnings) > 0 { 44 | utils.Log.Error("Warnings: %v\n", warnings) 45 | } 46 | 47 | if result.String() !=""{ 48 | resultslice := strings.Split(result.String(), "\n") 49 | resultSliceMap := ConvertResultDataType(resultslice) 50 | return resultSliceMap ,true 51 | }else{ 52 | utils.Log.Errorf("查询promql有问题,promsql返回结果为:nil,%v",result) 53 | return nil ,false 54 | } 55 | } 56 | 57 | 58 | 59 | // clinet_golang 访问prometheus返回结果为string 60 | // 通过strings的切换，转换为[]map[string]string 类型 61 | func ConvertResultDataType(resultslice []string) []map[string]string { 62 | var resultSilceMap []map[string]string 63 | 64 | for i := 0; i < len(resultslice); i++ { 65 | 66 | s := string(resultslice[i]) 67 | //up{instance="cn-hangzhou.172.16.94.142", job="k8s资源节点监控"} => 1 @[1599043436.489] 68 | // 1.匹配{}中的内容换成数组 69 | start := strings.Index(s, "{") 70 | end := strings.Index(s, "}") 71 | value1 := strings.TrimSpace(s[start+1 : end]) 72 | value1 = strings.Map(mapping, value1) 73 | metadataslice := strings.Fields(value1) 74 | 75 | // 2.匹配=> 之后的值为value 76 | start2 := strings.Index(s, "=>") 77 | end2 := strings.Index(s, "@") 78 | value2 := strings.TrimSpace(s[start2:end2]) 79 | value2 = strings.Map(mapping, value2) 80 | //3. 81 | metadatamap := make(map[string]string) 82 | 83 | for i := 0; i < len(metadataslice); i++ { 84 | if i%2 == 0 { 85 | j := i 86 | key := strings.TrimSpace(metadataslice[j]) 87 | value := strings.TrimSpace(metadataslice[j+1]) 88 | metadatamap[strings.TrimSpace(key)] = strings.TrimSpace(value) 89 | } 90 | } 91 | 92 | metadatamap["value"] = strings.TrimSpace(value2) 93 | resultSilceMap = append(resultSilceMap, metadatamap) 94 | } 95 | return resultSilceMap 96 | } 97 | 98 | func mapping(r rune) rune { 99 | if r == '=' || r == ',' || r == '>'||r =='"' { 100 | return ' ' 101 | } 102 | return r 103 | } 104 | -------------------------------------------------------------------------------- /rule-template.yaml: -------------------------------------------------------------------------------- 1 | PromJob: 2 | Name: "测试环境k8s资源节点监控" 3 | Rulename: 4 | ## 过去一分钟之内的node的cpu使用率 5 | Node_cpu1: 6 | Promsql: "(1-(sum(increase(node_cpu_seconds_total{job='测试环境k8s资源节点监控',mode='idle'}[1m]))by(instance))/(sum(increase(node_cpu_seconds_total{job='测试环境k8s资源节点监控'}[1m]))by(instance)))*100" 7 | Threshold: 70 8 | ## 内存使用率 9 | Node_mem: 10 | Promsql: "(1 - (node_memory_MemAvailable_bytes{job='测试环境k8s资源节点监控'} / (node_memory_MemTotal_bytes{job='测试环境k8s资源节点监控'})))* 100" 11 | Threshold: 70 12 | 13 | 14 | -------------------------------------------------------------------------------- /template/deploynginx.yml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: nginxtest 5 | namespace: default 6 | spec: 7 | replicas: 50 8 | selector: 9 | matchLabels: 10 | run: nginx 11 | strategy: 12 | rollingUpdate: 13 | maxSurge: 25% 14 | maxUnavailable: 25% 15 | type: RollingUpdate 16 | template: 17 | metadata: 18 | labels: 19 | run: nginx 20 | spec: 21 | affinity: 22 | nodeAffinity: 23 | preferredDuringSchedulingIgnoredDuringExecution: 24 | - preference: 25 | matchExpressions: 26 | - key: status 27 | operator: NotIn 28 | values: 29 | - presure 30 | weight: 20 31 | 32 | containers: 33 | - image: nginx 34 | imagePullPolicy: Always 35 | name: nginx 36 | dnsPolicy: ClusterFirst 37 | restartPolicy: Always 38 | -------------------------------------------------------------------------------- /template/prometheus-cm.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: prometheus-config 5 | namespace: kube-ops 6 | data: 7 | prometheus.yml: | 8 | global: 9 | scrape_interval: 15s 10 | evaluation_interval: 15s 11 | 12 | scrape_configs: 13 | - job_name: '测试环境k8s资源节点监控' 14 | kubernetes_sd_configs: 15 | - role: node 16 | relabel_configs: 17 | - source_labels: [__meta_kubernetes_node_address_InternalIP] 18 | regex: '(.*)' 19 | replacement: '${1}:9100' 20 | target_label: __address__ 21 | 22 | 23 | - job_name: 'pushgateway' 24 | honor_labels: true 25 | static_configs: 26 | - targets: ['172.16.95.4:49091'] 27 | 28 | 29 | 30 | - job_name: 'prometheus' 31 | scrape_interval: 10s 32 | static_configs: 33 | - targets: ['localhost:9090'] 34 | 35 | -------------------------------------------------------------------------------- /template/push.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | instance=$(hostname) 4 | ip=$(ifconfig eth0 | grep "inet" |awk '{print $2}'|xargs) 5 | ## 配置pushgateway地址 6 | pushgatewayaddr="XXXXXXXXXXXXXXX" 7 | job="kubeNodeMemoryAvailable" 8 | 9 | 10 | 11 | ## kubelet 计算kubeletr 12 | getMemory(){ 13 | # current memory usage 14 | memory_capacity_in_kb=$(cat /proc/meminfo | grep MemTotal | awk '{print $2}') 15 | memory_capacity_in_bytes=$((memory_capacity_in_kb * 1024)) 16 | memory_usage_in_bytes=$(cat /sys/fs/cgroup/memory/memory.usage_in_bytes) 17 | memory_total_inactive_file=$(cat /sys/fs/cgroup/memory/memory.stat | grep total_inactive_file | awk '{print $2}') 18 | 19 | memory_working_set=${memory_usage_in_bytes} 20 | if [ "$memory_working_set" -lt "$memory_total_inactive_file" ]; 21 | then 22 | memory_working_set=0 23 | else 24 | memory_working_set=$((memory_usage_in_bytes - memory_total_inactive_file)) 25 | fi 26 | 27 | memory_available_in_bytes=$((memory_capacity_in_bytes - memory_working_set)) 28 | memory_available_in_kb=$((memory_available_in_bytes / 1024)) 29 | memory_available_in_mb=$((memory_available_in_kb / 1024)) 30 | 31 | echo $memory_available_in_mb 32 | 33 | 34 | 35 | } 36 | 37 | 38 | pushNodeMem(){ 39 | 40 | cat << EOF |curl --data-binary @- $pushgatewayaddr/metrics/job/CollectK8sCgroupsMem/instance/$instance 41 | k8_node_cgroups_mem_available{ip="$ip"} $(getMemory) 42 | EOF 43 | } 44 | 45 | 46 | pushNodeMem 47 | -------------------------------------------------------------------------------- /utils/logger.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "fmt" 5 | nested "github.com/antonfisher/nested-logrus-formatter" 6 | "github.com/sirupsen/logrus" 7 | "os" 8 | "time" 9 | ) 10 | 11 | var Log = logrus.New() 12 | 13 | func init() { 14 | // 为当前logrus实例设置消息的输出,同样地, 15 | // 可以设置logrus实例的输出到任意io.writer 16 | dir,_:=os.Getwd() 17 | logName:=fmt.Sprintf("%s/logs/dynamic-%v.log",dir,time.Now().Format("2006-01-02")) 18 | fmt.Println(dir) 19 | fmt.Println(logName) 20 | logfile,err:=os.OpenFile(logName, os.O_CREATE|os.O_RDWR|os.O_APPEND, 0644) 21 | if err!=nil{ 22 | panic("打开日志文件失败") 23 | } 24 | Log.SetOutput(logfile) 25 | // 为当前logrus实例设置消息输出格式为json格式. 26 | // 同样地,也可以单独为某个logrus实例设置日志级别和hook,这里不详细叙述. 27 | // Log.Formatter = &logrus.JSONFormatter{} 28 | //Log.Formatter = &logrus.TextFormatter{} 29 | Log.SetFormatter(&nested.Formatter{ 30 | HideKeys: true, 31 | TimestampFormat: time.RFC3339, 32 | TrimMessages: true, 33 | NoColors: true, 34 | }) 35 | // 日志等级 36 | Log.SetLevel(logrus.InfoLevel) 37 | // 日志输出 执行的程序函数名和路径 38 | //Log.SetReportCaller(true) 39 | } 40 | 41 | 42 | 43 | -------------------------------------------------------------------------------- /utils/type.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | 4 | import ( 5 | "sync" 6 | ) 7 | 8 | type void struct{} 9 | 10 | type Set struct{ 11 | sync.RWMutex 12 | m map[string]void 13 | } 14 | 15 | 16 | 17 | // 新建集合对象 18 | func New(items ...string) *Set { 19 | s := &Set{ 20 | m: make(map[string]void, len(items)), 21 | } 22 | 23 | return s 24 | } 25 | 26 | 27 | // 添加元素 28 | func (s *Set) Add(items []string) { 29 | s.Lock() 30 | defer s.Unlock() 31 | for _, v := range items { 32 | s.m[v] =void{} 33 | } 34 | } 35 | 36 | // 删除元素 37 | func (s *Set) Remove(items ...string) { 38 | s.Lock() 39 | defer s.Unlock() 40 | for _, v := range items { 41 | delete(s.m, v) 42 | } 43 | } 44 | 45 | 46 | // 判断元素是否存在 47 | func (s *Set) Has(items ...string) bool { 48 | s.RLock() 49 | defer s.RUnlock() 50 | for _, v := range items { 51 | if _, ok := s.m[v]; !ok { 52 | return false 53 | } 54 | } 55 | return true 56 | } 57 | 58 | 59 | // 元素个数 60 | func (s *Set) Count() int { 61 | return len(s.m) 62 | } 63 | 64 | // 清空集合 65 | func (s *Set) Clear() { 66 | s.Lock() 67 | defer s.Unlock() 68 | s.m = map[string]void{} 69 | } 70 | 71 | // 空集合判断 72 | func (s *Set) Empty() bool { 73 | return len(s.m) == 0 74 | } 75 | 76 | // 无序列表 77 | func (s *Set) List() []string { 78 | s.RLock() 79 | defer s.RUnlock() 80 | list := make([]string, 0, len(s.m)) 81 | for item := range s.m { 82 | list = append(list, item) 83 | } 84 | return list 85 | } 86 | 87 | 88 | // 并集 89 | func (s *Set) Union(sets ...*Set) *Set { 90 | r := New(s.List()...) 91 | for _, set := range sets { 92 | for e := range set.m { 93 | r.m[e] = void{} 94 | } 95 | } 96 | return r 97 | } 98 | 99 | 100 | // 差集 101 | func (s *Set) Minus(sets ...*Set) *Set { 102 | r:=New() 103 | r.Add(s.List()) 104 | for _, set := range sets { 105 | for e := range set.m { 106 | if _, ok := s.m[e]; ok { 107 | delete(r.m, e) 108 | } 109 | } 110 | } 111 | return r 112 | } 113 | 114 | // 交集 115 | func (s *Set) Intersect(sets ...*Set) *Set { 116 | r:=New() 117 | r.Add(s.List()) 118 | for _, set := range sets { 119 | for e := range s.m { 120 | if _, ok := set.m[e]; !ok { 121 | delete(r.m, e) 122 | } 123 | } 124 | } 125 | return r 126 | } 127 | 128 | 129 | 130 | //func main(){ 131 | // //a:=[]string{"a","b","C"} 132 | // seta:=New("seta") 133 | // seta.Add([]string{"a","b","c"}) 134 | // setb:=New("setb") 135 | // setb.Add([]string{"a","b","c","d","E","F"}) 136 | // 137 | // 138 | // a:=setb.Intersect(seta) 139 | // b:=setb.Minus(seta) 140 | // fmt.Println(a.List()) 141 | // fmt.Println(b.List()) 142 | // fmt.Println(setb.List()) 143 | // 144 | // 145 | //} -------------------------------------------------------------------------------- /utils/yaml.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "fmt" 5 | "io/ioutil" 6 | "gopkg.in/yaml.v2" 7 | ) 8 | 9 | func GetRuleFromYaml(rulepath string) map[string]map[string]string { 10 | var c Conf 11 | Y:=c.GetConf(rulepath) 12 | return Y.Rulename 13 | } 14 | 15 | 16 | type Conf struct { 17 | Rulename map[string]map[string]string `yaml:"Rulename"` 18 | } 19 | 20 | func (c *Conf) GetConf(rulepath string) *Conf { 21 | yamlFile, err := ioutil.ReadFile(rulepath) 22 | if err != nil { 23 | fmt.Println(err.Error()) 24 | } 25 | err = yaml.Unmarshal(yamlFile, c) 26 | if err != nil { 27 | fmt.Println(err.Error()) 28 | } 29 | return c 30 | } 31 | --------------------------------------------------------------------------------