├── .gitignore ├── .travis.yml ├── LICENSE ├── README.md ├── async ├── .gitignore ├── project.clj └── src │ └── mesomatic │ └── async │ ├── executor.clj │ └── scheduler.clj ├── dev-resources └── src │ └── mesomatic │ └── dev.clj ├── project.clj ├── src └── mesomatic │ ├── allocator.clj │ ├── executor.clj │ ├── scheduler.clj │ ├── types.clj │ └── utils.clj ├── test └── mesomatic │ └── allocator_test.clj └── ux-resources └── images ├── mesomatic-logo-x1000.png ├── mesomatic-logo-x250.png └── mesomatic-logo.png /.gitignore: -------------------------------------------------------------------------------- 1 | /target 2 | /classes 3 | /checkouts 4 | pom.xml 5 | pom.xml.asc 6 | *.jar 7 | *.class 8 | /.lein-* 9 | /.nrepl-port 10 | .hgignore 11 | .hg/ 12 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: clojure 2 | script: 3 | - lein build 4 | jdk: 5 | - oraclejdk8 6 | - oraclejdk7 7 | - openjdk7 8 | env: 9 | - LEIN_SNAPSHOTS_IN_RELEASE=true 10 | notifications: 11 | email: 12 | - oubiwann@gmail.com 13 | 14 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2015 Pierre-Yves Ritschard 2 | 3 | Permission to use, copy, modify, and distribute this software for any 4 | purpose with or without fee is hereby granted, provided that the above 5 | notice and this permission notice appear in all copies. 6 | 7 | THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 8 | WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 9 | MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 10 | ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 11 | WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 12 | ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 13 | OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 14 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # mesomatic 2 | 3 | [![Build Status][travis-badge]][travis][![Clojars Project][clojars-badge]][clojars][![Clojure version][clojure-v]](project.clj) 4 | 5 | *A simple and idiomatic Clojure facade around the Mesos JAVA API* 6 | 7 | [![][logo]][logo-large] 8 | 9 | **Contents** 10 | 11 | * About 12 | * Resources 13 | * Usage 14 | * Examples 15 | * Namespaces 16 | * Type Conversions 17 | * Release Notes 18 | * Contributor Resources 19 | * Donating 20 | 21 | 22 | ## About 23 | 24 | Mesomatic provides facilities to interact with [Apache Mesos][mesos] from 25 | clojure. It provides a simple and idiomatic facade around the Mesos JAVA API 26 | and facilities to help when writing mesos frameworks. 27 | 28 | Mesomatic versions match the API version they target, a trailing minor 29 | indicates the patch release number, for instance version `1.0.1-r0` will 30 | target mesos `1.0.1`. 31 | 32 | Note that the [clojusc Github org][clojusc] has volunteered to maintain the 33 | library originally created by [pyr][pyr] at `pyr/mesomatic`. The new location, 34 | `clojusc/mesomatic`, is now the offical home for the library. 35 | 36 | 37 | ## Resources 38 | 39 | * An excellent [Mesos presentation][mesos-video] from ApacheCon 2014 40 | * [pyr][pyr]'s Euroclojure 2015 [talk on Mesomatic][mesomatic-video] 41 | 42 | 43 | ## Usage 44 | 45 | Add this to your leiningen profile: 46 | 47 | ```clojure 48 | :dependencies [[clojusc/mesomatic "1.0.1-r1"]] 49 | ``` 50 | 51 | If you want to use the [core.async][core-async] facade, 52 | you will need to pull it in as well: 53 | 54 | ```clojure 55 | :dependencies [[clojusc/mesomatic "1.0.1-r1"] 56 | [clojusc/mesomatic-async "1.0.1-r1"]] 57 | ``` 58 | 59 | 60 | ## Examples 61 | 62 | Be sure to examine the [example frameworks][examples] built with mesomatic. 63 | 64 | 65 | ## Namespaces 66 | 67 | - `mesomatic.types`: contains a facade to and from all protobuf types. 68 | - `mesomatic.scheduler`: facades for schedulers and scheduler-drivers 69 | - `mesomatic.executor`: facades for executors and executor-drivers 70 | - `mesomatic.async.executor`: produce executor callbacks on a channel 71 | - `mesomatic.async.scheduler`: produce scheduler callbacks on a channel 72 | - `mesomatic.helpers`: utility helpers for cluster decisions 73 | 74 | 75 | ## Type Conversions 76 | 77 | To go to and from protobuf types, mesomatic uses two simple functions: 78 | 79 | - `pb->data`: yields a data structure from a mesos type, usually in the form of 80 | a record. 81 | - `data->pb`: converts a data structure to a mesos type. 82 | - `->pb`: convert a plain map to a mesos type hinted at by a keyword 83 | 84 | By yielding records, mesomatic provides elements which are homomorphic to 85 | maps and can easily be converted back to protobuf. 86 | 87 | 88 | ### Special Cases 89 | 90 | A few cases do not yield records: 91 | 92 | - Scalar values (`Protos.Value.Scalar`) yield doubles. 93 | - All enums yield keywords. 94 | - Set values (`Protos.Value.Set`) yield sets. 95 | - Some types containing a single repeated field are unrolled 96 | as a seq of their content, such as `Protos.Value.Ranges`. 97 | 98 | 99 | ## Release Notes 100 | 101 | ### 1.0.1 102 | 103 | - Target mesos 1.0.1 104 | - Support for GPU resources 105 | - Updates for API changes in Java bindings 106 | 107 | 108 | ## Contributor Resources 109 | 110 | - http://mesos.apache.org/documentation/latest/upgrades/ 111 | - https://github.com/ContainerSolutions/minimesos 112 | - https://github.com/ContainerSolutions/minimesos-docker 113 | - https://github.com/katacoda/minimesos-examples 114 | - https://github.com/mesos/elasticsearch/tree/master/system-test/src/systemTest/java/org/apache/mesos/elasticsearch/systemtest 115 | 116 | 117 | ## Donating 118 | 119 | A donation account for supporting development on this project has been set up 120 | on Liberapay here: 121 | 122 | * [https://liberapay.com/clojusc-mesomatic/donate](https://liberapay.com/clojusc-mesomatic/donate) 123 | 124 | You can learn more about Liberapay on its [Wikipedia entry][libera-wiki] or on the 125 | service's ["About" page][libera-about]. 126 | 127 | https://liberapay.com/clojusc-mesomatic/donate 128 | 129 | 130 | 131 | [travis]: https://travis-ci.org/clojusc/mesomatic 132 | [travis-badge]: https://travis-ci.org/clojusc/mesomatic.png?branch=master 133 | [deps]: http://jarkeeper.com/clojusc/mesomatic 134 | [deps-badge]: http://jarkeeper.com/clojusc/mesomatic/status.svg 135 | [logo]: ux-resources/images/mesomatic-logo-x250.png 136 | [logo-large]: ux-resources/images/mesomatic-logo-x1000.png 137 | [tag-badge]: https://img.shields.io/github/tag/clojusc/mesomatic.svg 138 | [tag]: https://github.com/clojusc/mesomatic/tags 139 | [clojure-v]: https://img.shields.io/badge/clojure-1.9.0-blue.svg 140 | [clojars]: https://clojars.org/clojusc/mesomatic 141 | [clojars-badge]: https://img.shields.io/clojars/v/clojusc/mesomatic.svg 142 | [mesos]: http://mesos.apache.org 143 | [pyr]: https://github.com/pyr 144 | [core-async]: https://github.com/clojure/core.async 145 | [examples]: https://github.com/clojusc/mesomatic-examples 146 | [mesos-video]: https://www.youtube.com/watch?v=hTcZGODnyf0 147 | [mesomatic-video]: https://www.youtube.com/watch?v=X-fVA5DxezE 148 | [clojusc]: https://github.com/clojusc 149 | [libera-wiki]: https://en.wikipedia.org/wiki/Liberapay 150 | [libera-about]: https://liberapay.com/about/ 151 | -------------------------------------------------------------------------------- /async/.gitignore: -------------------------------------------------------------------------------- 1 | /target 2 | /classes 3 | /checkouts 4 | pom.xml 5 | pom.xml.asc 6 | *.jar 7 | *.class 8 | /.lein-* 9 | /.nrepl-port 10 | .hgignore 11 | .hg/ 12 | -------------------------------------------------------------------------------- /async/project.clj: -------------------------------------------------------------------------------- 1 | (defproject clojusc/mesomatic-async "1.0.1-r2-SNAPSHOT" 2 | :description "Async support for the Clojure mesomatic Mesos library" 3 | :url "https://github.com/clojusc/mesomatic" 4 | :license {:name "MIT License"} 5 | :dependencies [[clojusc/mesomatic "1.0.1-r2-SNAPSHOT"] 6 | [org.clojure/core.async "0.2.374"] 7 | [org.clojure/clojure "1.8.0"]]) 8 | -------------------------------------------------------------------------------- /async/src/mesomatic/async/executor.clj: -------------------------------------------------------------------------------- 1 | (ns mesomatic.async.executor 2 | (:require [clojure.core.async :refer [put! close! chan]] 3 | [mesomatic.executor :as e])) 4 | 5 | (defn executor 6 | ([ch close?] 7 | (e/executor 8 | (disconnected 9 | [this driver] 10 | (put! ch {:type :disconnected :driver driver})) 11 | (error 12 | [this driver message] 13 | (put! ch {:type :error :driver driver :message message})) 14 | (framework-message 15 | [this driver data] 16 | (put! ch {:type :framework-message :driver driver :data data})) 17 | (kill-task 18 | [this driver task-id] 19 | (put! ch {:type :kill-task :driver driver :task-id task-id})) 20 | (launch-task 21 | [this driver task] 22 | (put! ch {:type :launch-task :driver driver :task task})) 23 | (registered 24 | [this driver executor-info framework-info slave-info] 25 | (put! ch {:type :registered 26 | :driver driver 27 | :executor-info executor-info 28 | :framework-info framework-info 29 | :slave-info slave-info})) 30 | (reregistered 31 | [this driver slave-info] 32 | (put! ch {:type :reregistered :slave-info slave-info})) 33 | (shutdown 34 | [this driver] 35 | (put! ch {:type :shutdown :driver driver}) 36 | (when close? (close! ch))))) 37 | ([ch] 38 | (executor ch true)) 39 | ([] 40 | (let [ch (chan)] 41 | (executor ch true) 42 | ch))) 43 | -------------------------------------------------------------------------------- /async/src/mesomatic/async/scheduler.clj: -------------------------------------------------------------------------------- 1 | (ns mesomatic.async.scheduler 2 | (:require [clojure.core.async :refer [put! close! chan]] 3 | [mesomatic.scheduler :as s])) 4 | 5 | (defn scheduler 6 | ([ch close?] 7 | (s/scheduler 8 | (registered 9 | [this driver framework-id master-info] 10 | (put! ch {:type :registered 11 | :driver driver 12 | :framework-id framework-id 13 | :master-info master-info})) 14 | (reregistered 15 | [this driver master-info] 16 | (put! ch {:type :reregistered 17 | :driver driver 18 | :master-info master-info})) 19 | (disconnected 20 | [this driver] 21 | (put! ch {:type :disconnected :driver driver})) 22 | (resource-offers 23 | [this driver offers] 24 | (put! ch {:type :resource-offers :driver driver :offers offers})) 25 | (offer-rescinded 26 | [this driver offer-id] 27 | (put! ch {:type :offer-rescinded :driver driver :offer-id offer-id})) 28 | (status-update 29 | [this driver status] 30 | (put! ch {:type :status-update :driver driver :status status})) 31 | (framework-message 32 | [this driver executor-id slave-id data] 33 | (put! ch {:type :framework-message 34 | :driver driver 35 | :executor-id executor-id 36 | :slave-id slave-id 37 | :data data})) 38 | (slave-lost 39 | [this driver slave-id] 40 | (put! ch {:type :slave-lost 41 | :driver driver 42 | :slave-id slave-id})) 43 | (executor-lost 44 | [this driver executor-id slave-id status] 45 | (put! ch {:type :executor-lost 46 | :driver driver 47 | :executor-id executor-id 48 | :slave-id slave-id 49 | :status status})) 50 | (error 51 | [this driver message] 52 | (put! ch {:type :error 53 | :driver driver 54 | :message message})))) 55 | ([ch] 56 | (scheduler ch true)) 57 | ([] 58 | (let [ch (chan)] 59 | (scheduler ch true) 60 | ch))) 61 | -------------------------------------------------------------------------------- /dev-resources/src/mesomatic/dev.clj: -------------------------------------------------------------------------------- 1 | (ns mesomatic.dev) 2 | -------------------------------------------------------------------------------- /project.clj: -------------------------------------------------------------------------------- 1 | (defproject clojusc/mesomatic "1.0.1-r2-SNAPSHOT" 2 | :description "A simple and idiomatic Clojure facade around the Mesos JAVA API" 3 | :url "https://github.com/clojusc/mesomatic" 4 | :license {:name "MIT License"} 5 | :dependencies [ 6 | [org.clojure/clojure "1.9.0"] 7 | [org.apache.mesos/mesos "1.0.1"]] 8 | :profiles { 9 | :dev { 10 | :source-paths ["dev-resources/src"] 11 | } 12 | :ubercompile { 13 | :aot :all} 14 | :custom-repl { 15 | :repl-options { 16 | :init-ns mesomatic.dev 17 | :prompt ~#(str "\u001B[35m[\u001B[34m" 18 | % 19 | "\u001B[35m]\u001B[33m λ\u001B[m=> ")}} 20 | :test { 21 | :exclusions [org.clojure/clojure] 22 | :dependencies [ 23 | [clojusc/ltest "0.3.0-SNAPSHOT"]] 24 | :plugins [ 25 | [jonase/eastwood "0.2.5"] 26 | [lein-ancient "0.6.15"] 27 | [lein-bikeshed "0.5.0"] 28 | [lein-kibit "0.1.6"] 29 | [lein-ltest "0.3.0-SNAPSHOT"] 30 | [venantius/yagni "0.1.4"]]}} 31 | :aliases { 32 | "repl" ["with-profile" "+custom-repl,+test" "repl"] 33 | "ubercompile" ["with-profile" "+ubercompile" "compile"] 34 | "check-deps" ["with-profile" "+test" "ancient" "check" ":all"] 35 | "lint" ["with-profile" "+test" "kibit"] 36 | "test" ["with-profile" "+test" "ltest"] 37 | "build" ["with-profile" "+test" "do" 38 | ;["check-deps"] 39 | ["lint"] 40 | ["ubercompile"] 41 | ["clean"] 42 | ["uberjar"] 43 | ["clean"] 44 | ["test"]]}) 45 | -------------------------------------------------------------------------------- /src/mesomatic/allocator.clj: -------------------------------------------------------------------------------- 1 | (ns mesomatic.allocator 2 | "Resource Allocation") 3 | 4 | (defprotocol IAllocator 5 | "Allocator abstraction to allow for pluggable 6 | allocation techniques." 7 | (allocate [this offers tasks] 8 | "Given a list of offers and tasks to start, figure out 9 | the best possible arrangement. Will yield a list of tasks 10 | containing an offerid to select or a nil if no way to fully 11 | fulfill requirements was found. Allocation should not yield partial 12 | results. 13 | 14 | Resources taken into account are: 15 | 16 | - cpus 17 | - mem 18 | - port ranges 19 | 20 | tasks is a list of mesomatic.types/TaskInfo 21 | offers is a list of mesomatic.types/Offer")) 22 | 23 | (defn get-scalar 24 | [rlist name] 25 | (let [rmap (zipmap (map :name rlist) 26 | (map :scalar rlist))] 27 | (get rmap name))) 28 | 29 | (defn get-ranges 30 | [rlist name] 31 | (let [rmap (zipmap (map :name rlist) 32 | (map :ranges rlist))] 33 | (get rmap name))) 34 | 35 | (defn get-ports 36 | [rlist] 37 | (reduce 38 | + 0 39 | (for [{:keys [begin end] :as range} (get-ranges rlist "ports") 40 | :when range] 41 | (inc (- end begin))))) 42 | 43 | (defn resource-factor 44 | "Compute an integral resource factor to help 45 | sort offers" 46 | [{:keys [resources]}] 47 | (+ (* (get-scalar resources "mem") 1000) 48 | (get-scalar resources "cpus"))) 49 | 50 | (defn resource-cmp 51 | "Comparator for resources. Will sort in decreasing order." 52 | [r1 r2] 53 | (pos? 54 | (- (resource-factor r1) 55 | (resource-factor r2)))) 56 | 57 | (defn offer-matches? 58 | "Predicate to validate an offer can satisfy a task's requirements" 59 | [offer task] 60 | (and ;; Check for valid executor 61 | (>= (get-scalar (:resources offer) "cpus") 62 | (get-scalar (:resources task) "cpus")) 63 | (>= (get-scalar (:resources offer) "mem") 64 | (get-scalar (:resources task) "mem")) 65 | (>= (get-ports (:resources offer)) 66 | (get-ports (:resources task))))) 67 | 68 | (defn accept-ports 69 | "Eat up as many ports as specified in the first port 70 | range. Notice that only the first port range is considered 71 | for now." 72 | [resources ports] 73 | (vec 74 | (for [{:keys [name ranges] :as resource} resources] 75 | (if (= name "ports") 76 | (let [offer-begin (-> ports first :begin) 77 | [{:keys [begin end]}] ranges] 78 | (assoc resource :ranges 79 | [{:begin (+ begin offer-begin) 80 | :end (+ end offer-begin)}])) 81 | resource)))) 82 | 83 | (defn map-ports 84 | [port-mappings ports] 85 | (let [begin (-> ports first :begin)] 86 | (vec 87 | (for [[{:keys [host-port container-port protocol]} i] 88 | (partition 2 (interleave port-mappings 89 | (range begin Long/MAX_VALUE)))] 90 | {:host-port (or host-port i) 91 | :container-port container-port 92 | :protocol protocol})))) 93 | 94 | (defn accept-offer 95 | "Associate a task with an offer's corresponding slave. 96 | When allocating multiple instances of a task (for daemons) and 97 | the task-id field is a vector get the appropriate member of the vector" 98 | [offer task pos] 99 | (let [ports (get-ranges (:resources offer) "ports") 100 | get-pos (fn [ids] (if (vector? ids) (nth ids pos) ids))] 101 | (-> task 102 | (assoc :slave-id (:slave-id offer)) 103 | (assoc :offer-id (:id offer)) 104 | (update :task-id get-pos) 105 | (update :resources accept-ports ports) 106 | (cond-> (= (-> task :container :type) :container-type-docker) 107 | (update-in [:container :docker :port-mappings] map-ports ports))))) 108 | 109 | (defn adjust-ports 110 | "Adjust port range. Notice that only the first range 111 | is considered." 112 | [ranges want] 113 | (let [[{:keys [begin end]}] ranges] 114 | [{:begin (+ begin want) :end end}])) 115 | 116 | (defn adjustor 117 | "Yield a function which will adjust a resource record 118 | when appropriate." 119 | [cpus mem ports] 120 | (fn [{:keys [name] :as record}] 121 | (case name 122 | "mem" (update record :scalar - mem) 123 | "cpus" (update record :scalar - cpus) 124 | "ports" (update record :ranges adjust-ports ports) 125 | record))) 126 | 127 | (defn adjust-offer 128 | "If an offer has been accepted, decrease its available resources." 129 | [offer task] 130 | (let [cpus (get-scalar (:resources task) "cpus") 131 | mem (get-scalar (:resources task) "mem") 132 | ports (get-ports (:resources task)) 133 | resources (:resources offer)] 134 | (update offer :resources (partial map (adjustor cpus mem ports))))) 135 | 136 | (defn allocate-task-naively 137 | "Allocate a task's worth of necessary resources. 138 | Tasks may ask for a specific count of instances 139 | and a maximum collocation factor to avoid behind 140 | hosted on a single slave. 141 | 142 | Allocation technique 143 | ==================== 144 | 145 | Let's assume the following workload and available offers: 146 | 147 | [ T1(1,1,1,1) T2(4,8,2,1) T3(1,1,4,2) ] 148 | [ O1(4,8) O2(8,16) O3(2,2) ] 149 | 150 | We first sort our offers: 151 | 152 | [ T2(4,8,2,1) T3(1,1,4,2) T1(1,1,1,1) ] 153 | [ O1(8,16) O2(4,8) O3(2,2) ] 154 | 155 | We can now step through tasks and allocate appropriately: 156 | 157 | [ T3(1,1,4,2) T1(1,1,1,1) ] 158 | [ O1(4,8) O3(2,2) ] 159 | [ T1:(O1,O2) ] 160 | 161 | [ T1(1,1,1,1) ] 162 | [ O1(2,6) ] 163 | [ T2:(O1,O2) T3(O1,O3) ] 164 | 165 | [ ] 166 | [ O1(1,5) ] 167 | [ T2:(O1,O2) T3(O1,O3) T1(O1) ]" 168 | 169 | [acc {:keys [instances] :or {instances 1} :as task}] 170 | (loop [[offer & offers :as untouched] (:offers acc) 171 | payloads [] 172 | adjusted nil 173 | [global local] [0 0]] 174 | (cond 175 | ;; Cannot properly allocate this task 176 | ;; Fail altogether. 177 | (nil? offer) 178 | (reduced (assoc acc :failed? true)) 179 | 180 | ;; We're done with this task. 181 | (>= global instances) 182 | (-> acc 183 | (assoc :offers (into (vec adjusted) untouched)) 184 | (update :payloads into payloads)) 185 | 186 | ;; Too many collocated tasks, skip to next offer. 187 | (and (:colocation task) (>= local (:colocation task))) 188 | (recur offers payloads (conj adjusted offer) [global 0]) 189 | 190 | ;; We have a match, record it. 191 | (offer-matches? offer task) 192 | (recur (conj offers (adjust-offer offer task)) 193 | (conj payloads (accept-offer offer task global)) 194 | adjusted 195 | [(inc global) (inc local)]) 196 | 197 | ;; No match, let's move on. 198 | :else 199 | (recur offers payloads (conj adjusted offer) [global local])))) 200 | 201 | (defn allocate-naively 202 | "Cycle through all tasks, sorted by biggest to smallest 203 | in terms of resource needs and try to satisfy requirements." 204 | [offers tasks] 205 | (let [res (reduce allocate-task-naively 206 | {:offers (sort (comparator resource-cmp) offers)} 207 | (sort (comparator resource-cmp) tasks))] 208 | (when-not (:failed? res) 209 | (:payloads res)))) 210 | 211 | (defn naive-allocator 212 | "Conform to IAllocator by handing off decisions to allocate-naively" 213 | [] 214 | (reify IAllocator 215 | (allocate [this offers tasks] 216 | (allocate-naively offers tasks)))) 217 | -------------------------------------------------------------------------------- /src/mesomatic/executor.clj: -------------------------------------------------------------------------------- 1 | (ns mesomatic.executor 2 | (:require [mesomatic.types :refer [data->pb pb->data ->pb]]) 3 | (:import org.apache.mesos.MesosExecutorDriver)) 4 | 5 | (defprotocol Executor 6 | (disconnected [this driver]) 7 | (error [this driver message]) 8 | (framework-message [this driver data]) 9 | (kill-task [this driver task-id]) 10 | (launch-task [this driver task]) 11 | (registered [this driver executor-info framework-info slave-info]) 12 | (reregistered [this driver slave-info]) 13 | (shutdown [this driver])) 14 | 15 | (defprotocol ExecutorDriver 16 | (abort! [this]) 17 | (join! [this]) 18 | (run-driver! [this]) 19 | (send-framework-message! [this data]) 20 | (send-status-update! [this status]) 21 | (start! [this]) 22 | (stop! [this])) 23 | 24 | (defn wrap-executor 25 | [impl] 26 | (reify 27 | org.apache.mesos.Executor 28 | (disconnected [this driver] 29 | (disconnected impl driver)) 30 | (error [this driver message] 31 | (error impl driver message)) 32 | (frameworkMessage [this driver data] 33 | (framework-message impl driver data)) 34 | (killTask [this driver task-id] 35 | (kill-task impl driver (pb->data task-id))) 36 | (launchTask [this driver task] 37 | (launch-task impl driver (pb->data task))) 38 | (registered [this driver executor-info framework-info slave-info] 39 | (registered impl 40 | driver 41 | (pb->data executor-info) 42 | (pb->data framework-info) 43 | (pb->data slave-info))) 44 | (reregistered [this driver slave-info] 45 | (reregistered impl 46 | driver 47 | (pb->data slave-info))) 48 | (shutdown [this driver] 49 | (shutdown impl driver)))) 50 | 51 | (defmacro executor 52 | [& body] 53 | `(wrap-executor (reify Executor ~@body))) 54 | 55 | (defn executor-driver 56 | [executor] 57 | (let [d (MesosExecutorDriver. executor)] 58 | (reify ExecutorDriver 59 | (abort! [this] 60 | (pb->data (.abort d))) 61 | (join! [this] 62 | (pb->data (.join d))) 63 | (run-driver! [this] 64 | (pb->data (.run d))) 65 | (send-framework-message! [this data] 66 | (pb->data (.sendFrameworkMessage d data))) 67 | (send-status-update! [this status] 68 | (pb->data (.sendStatusUpdate d status))) 69 | (start! [this] 70 | (pb->data (.start d))) 71 | (stop! [this] 72 | (pb->data (.stop d)))))) 73 | -------------------------------------------------------------------------------- /src/mesomatic/scheduler.clj: -------------------------------------------------------------------------------- 1 | (ns mesomatic.scheduler 2 | (:require [mesomatic.types :refer [data->pb pb->data ->pb]]) 3 | (:import org.apache.mesos.MesosSchedulerDriver)) 4 | 5 | (defprotocol Scheduler 6 | (registered [this driver framework-id master-info]) 7 | (reregistered [this driver master-info]) 8 | (disconnected [this driver]) 9 | (resource-offers [this driver offers]) 10 | (offer-rescinded [this driver offer-id]) 11 | (status-update [this driver status]) 12 | (framework-message [this driver executor-id slave-id data]) 13 | (slave-lost [this driver slave-id]) 14 | (executor-lost [this driver executor-id slave-id status]) 15 | (error [this driver message])) 16 | 17 | (defprotocol SchedulerDriver 18 | (abort! [this]) 19 | (acknowledge-status-update [this status]) 20 | (accept-offers [this offer-ids operations] 21 | [this offer-ids operations filters]) 22 | (decline-offer [this offer-id] [this offer-id filters]) 23 | (join! [this]) 24 | (kill-task! [this task-id]) 25 | (launch-tasks! [this offer-id tasks] [this offer-id tasks filters]) 26 | (reconcile-tasks [this statuses]) 27 | (request-resources [this requests]) 28 | (revive-offers [this]) 29 | (run-driver! [this]) 30 | (send-framework-message! [this executor-id slave-id data]) 31 | (start! [this]) 32 | (stop! [this] [this failover?]) 33 | (suppress-offers [this])) 34 | 35 | (defn wrap-driver [d] 36 | (reify SchedulerDriver 37 | (abort! [this] 38 | (pb->data (.abort d))) 39 | (acknowledge-status-update [this status] 40 | (pb->data (.acknowledgeStatusUpdate d (->pb :TaskStatus status)))) 41 | (accept-offers [this offer-ids operations] 42 | (pb->data (.acceptOffers d 43 | (mapv (partial ->pb :OfferID) offer-ids) 44 | (mapv (partial ->pb :Operation) operations) 45 | (->pb :Filters {:refuse-seconds 1})))) 46 | (accept-offers [this offer-ids operations filters] 47 | (pb->data (.acceptOffers d 48 | (mapv (partial ->pb :OfferID) offer-ids) 49 | (mapv (partial ->pb :Operation) operations) 50 | (mapv (partial ->pb :Filters) filters)))) 51 | 52 | (decline-offer [this offer-id] 53 | (pb->data (.declineOffer d (->pb :OfferID offer-id)))) 54 | (decline-offer [this offer-id filters] 55 | (pb->data (.declineOffer d 56 | (->pb :OfferID offer-id) 57 | (->pb :Filters filters)))) 58 | (join! [this] 59 | (pb->data (.join d))) 60 | (kill-task! [this task-id] 61 | (pb->data (.killTask d (->pb :TaskID task-id)))) 62 | (launch-tasks! [this offer-id tasks] 63 | (pb->data (.launchTasks d 64 | (if (sequential? offer-id) 65 | (mapv (partial ->pb :OfferID) offer-id) 66 | (vector (->pb :OfferID offer-id))) 67 | (mapv (partial ->pb :TaskInfo) tasks)))) 68 | (launch-tasks! [this offer-id tasks filters] 69 | (pb->data (.launchTasks d 70 | (if (sequential? offer-id) 71 | (mapv (partial ->pb :OfferID) offer-id) 72 | (vector (->pb :OfferID offer-id))) 73 | (mapv (partial ->pb :TaskInfo) tasks) 74 | (->pb :Filters filters)))) 75 | (reconcile-tasks [this statuses] 76 | (pb->data 77 | (.reconcileTasks d (mapv (partial ->pb :TaskStatus) statuses)))) 78 | (request-resources [this requests] 79 | (pb->data 80 | (.requestResources d (mapv (partial ->pb :Request) requests)))) 81 | (revive-offers [this] 82 | (pb->data (.reviveOffers d))) 83 | (run-driver! [this] 84 | (pb->data (.run d))) 85 | (send-framework-message! [this executor-id slave-id data] 86 | (pb->data (.sendFrameworkMessage d 87 | (->pb :ExecutorID executor-id) 88 | (->pb :SlaveID slave-id) 89 | data))) 90 | (start! [this] 91 | (pb->data (.start d))) 92 | (stop! [this] 93 | (pb->data (.stop d))) 94 | (stop! [this failover?] 95 | (pb->data (.stop this (boolean failover?)))) 96 | (suppress-offers [this] 97 | (pb->data (.suppressOffers d))))) 98 | 99 | (defn wrap-scheduler 100 | [implementation] 101 | (reify 102 | org.apache.mesos.Scheduler 103 | (registered [this driver framework-id master-info] 104 | (registered implementation 105 | (wrap-driver driver) 106 | (pb->data framework-id) 107 | (pb->data master-info))) 108 | (reregistered [this driver master-info] 109 | (reregistered implementation (wrap-driver driver) (pb->data master-info))) 110 | (disconnected [this driver] 111 | (disconnected implementation (wrap-driver driver))) 112 | (resourceOffers [this driver offers] 113 | (resource-offers implementation (wrap-driver driver) (mapv pb->data offers))) 114 | (offerRescinded [this driver offer-id] 115 | (offer-rescinded implementation (wrap-driver driver) (pb->data offer-id))) 116 | (statusUpdate [this driver status] 117 | (status-update implementation (wrap-driver driver) (pb->data status))) 118 | (frameworkMessage [this driver executor-id slave-id data] 119 | (framework-message implementation 120 | (wrap-driver driver) 121 | (pb->data executor-id) 122 | (pb->data slave-id) 123 | data)) 124 | (slaveLost [this driver slave-id] 125 | (slave-lost implementation (wrap-driver driver) (pb->data slave-id))) 126 | (executorLost [this driver executor-id slave-id status] 127 | (executor-lost implementation 128 | (wrap-driver driver) 129 | (pb->data executor-id) 130 | (pb->data slave-id) 131 | (pb->data status))) 132 | (error [this driver message] 133 | (error implementation (wrap-driver driver) message)))) 134 | 135 | (defmacro scheduler 136 | [& body] 137 | `(wrap-scheduler (reify Scheduler ~@body))) 138 | 139 | (defn driver-builder 140 | ([scheduler framework master] 141 | (MesosSchedulerDriver. scheduler 142 | (->pb :FrameworkInfo framework) 143 | master)) 144 | ([scheduler framework master credential] 145 | (MesosSchedulerDriver. scheduler 146 | (->pb :FrameworkInfo framework) 147 | master 148 | (->pb :Credential credential))) 149 | ([scheduler framework master credential implicit-acknowledgements?] 150 | (if (nil? credential) 151 | (MesosSchedulerDriver. scheduler 152 | (->pb :FrameworkInfo framework) 153 | master 154 | implicit-acknowledgements?) 155 | (MesosSchedulerDriver. scheduler 156 | (->pb :FrameworkInfo framework) 157 | master 158 | implicit-acknowledgements? 159 | (->pb :Credential credential))))) 160 | 161 | (defn scheduler-driver 162 | [& args] 163 | (->> args 164 | (apply driver-builder) 165 | (wrap-driver))) 166 | -------------------------------------------------------------------------------- /src/mesomatic/types.clj: -------------------------------------------------------------------------------- 1 | (ns mesomatic.types 2 | "Utility functions to convert to and from mesos types." 3 | (:require [mesomatic.utils :refer [case-enum]]) 4 | (:import org.apache.mesos.Protos$Status 5 | org.apache.mesos.Protos$FrameworkID 6 | org.apache.mesos.Protos$OfferID 7 | org.apache.mesos.Protos$SlaveID 8 | org.apache.mesos.Protos$TaskID 9 | org.apache.mesos.Protos$ExecutorID 10 | org.apache.mesos.Protos$ContainerID 11 | org.apache.mesos.Protos$FrameworkInfo 12 | org.apache.mesos.Protos$FrameworkInfo$Capability 13 | org.apache.mesos.Protos$FrameworkInfo$Capability$Type 14 | org.apache.mesos.Protos$HealthCheck 15 | org.apache.mesos.Protos$HealthCheck$HTTP 16 | org.apache.mesos.Protos$CommandInfo 17 | org.apache.mesos.Protos$CommandInfo$URI 18 | org.apache.mesos.Protos$ExecutorInfo 19 | org.apache.mesos.Protos$MasterInfo 20 | org.apache.mesos.Protos$SlaveInfo 21 | org.apache.mesos.Protos$Value 22 | org.apache.mesos.Protos$Value$Type 23 | org.apache.mesos.Protos$Value$Scalar 24 | org.apache.mesos.Protos$Value$Range 25 | org.apache.mesos.Protos$Value$Ranges 26 | org.apache.mesos.Protos$Value$Set 27 | org.apache.mesos.Protos$Value$Text 28 | org.apache.mesos.Protos$Attribute 29 | org.apache.mesos.Protos$Resource 30 | org.apache.mesos.Protos$ResourceStatistics 31 | org.apache.mesos.Protos$ResourceUsage 32 | org.apache.mesos.Protos$PerfStatistics 33 | org.apache.mesos.Protos$Request 34 | org.apache.mesos.Protos$Offer 35 | org.apache.mesos.Protos$Offer$Operation 36 | org.apache.mesos.Protos$Offer$Operation$Launch 37 | org.apache.mesos.Protos$Offer$Operation$Reserve 38 | org.apache.mesos.Protos$Offer$Operation$Unreserve 39 | org.apache.mesos.Protos$Offer$Operation$Create 40 | org.apache.mesos.Protos$Offer$Operation$Destroy 41 | org.apache.mesos.Protos$Offer$Operation$Type 42 | org.apache.mesos.Protos$TaskInfo 43 | org.apache.mesos.Protos$TaskState 44 | org.apache.mesos.Protos$TaskStatus 45 | org.apache.mesos.Protos$TaskStatus$Source 46 | org.apache.mesos.Protos$TaskStatus$Reason 47 | org.apache.mesos.Protos$Filters 48 | org.apache.mesos.Protos$Environment 49 | org.apache.mesos.Protos$Environment$Variable 50 | org.apache.mesos.Protos$Parameter 51 | org.apache.mesos.Protos$Parameters 52 | org.apache.mesos.Protos$Credential 53 | org.apache.mesos.Protos$Credentials 54 | org.apache.mesos.Protos$RateLimit 55 | org.apache.mesos.Protos$RateLimits 56 | org.apache.mesos.Protos$TimeInfo 57 | org.apache.mesos.Protos$DurationInfo 58 | org.apache.mesos.Protos$Address 59 | org.apache.mesos.Protos$URL 60 | org.apache.mesos.Protos$Unavailability 61 | org.apache.mesos.Protos$MachineID 62 | org.apache.mesos.Protos$MachineInfo 63 | org.apache.mesos.Protos$MachineInfo$Mode 64 | org.apache.mesos.Protos$Volume 65 | org.apache.mesos.Protos$Volume$Mode 66 | org.apache.mesos.Protos$ContainerInfo 67 | org.apache.mesos.Protos$ContainerInfo$Type 68 | org.apache.mesos.Protos$ContainerInfo$DockerInfo 69 | org.apache.mesos.Protos$ContainerInfo$DockerInfo$Network 70 | org.apache.mesos.Protos$ContainerInfo$DockerInfo$PortMapping 71 | org.apache.mesos.Protos$Port 72 | org.apache.mesos.Protos$Ports 73 | org.apache.mesos.Protos$DiscoveryInfo 74 | org.apache.mesos.Protos$DiscoveryInfo$Visibility 75 | org.apache.mesos.Protos$Label 76 | org.apache.mesos.Protos$Labels 77 | )) 78 | 79 | ;; Our two exported signatures: data->pb and pb->data 80 | 81 | (defprotocol Serializable 82 | "Interface to convert from clojure data to mesos protobuf 83 | payloads." 84 | (data->pb [this])) 85 | 86 | (defmulti pb->data 87 | "Open protocol to convert from mesos protobuf to clojure" 88 | class) 89 | 90 | (declare ->pb) 91 | 92 | 93 | ;; Status 94 | ;; ====== 95 | ;; 96 | ;; Enums are a bit of a special case since we don't wrap 97 | ;; them in a record, instead just yield a keyword which 98 | ;; will can be converted back with the extend-protocol 99 | ;; trick later on. 100 | 101 | (defmethod pb->data Protos$Status 102 | [^Protos$Status status] 103 | (case-enum status 104 | Protos$Status/DRIVER_RUNNING :driver-running 105 | Protos$Status/DRIVER_NOT_STARTED :driver-not-started 106 | Protos$Status/DRIVER_ABORTED :driver-aborted 107 | Protos$Status/DRIVER_STOPPED :driver-stopped 108 | status)) 109 | 110 | ;; FrameworkID 111 | ;; =========== 112 | ;; 113 | ;; All payloads will follow this idiom of defining 114 | ;; a record which is serializable and have deserialization 115 | ;; yield an instance of that protocol. 116 | (defrecord FrameworkID [value] 117 | Serializable 118 | (data->pb [this] 119 | (-> (Protos$FrameworkID/newBuilder) 120 | (.setValue (str value)) 121 | (.build)))) 122 | 123 | (defmethod pb->data Protos$FrameworkID 124 | [^Protos$FrameworkID id] 125 | (FrameworkID. (.getValue id))) 126 | 127 | ;; OfferID 128 | ;; ======= 129 | 130 | (defrecord OfferID [value] 131 | Serializable 132 | (data->pb [this] 133 | (-> (Protos$OfferID/newBuilder) 134 | (.setValue (str value)) 135 | (.build)))) 136 | 137 | (defmethod pb->data Protos$OfferID 138 | [^Protos$OfferID id] 139 | (OfferID. (.getValue id))) 140 | 141 | ;; SlaveID 142 | ;; ======= 143 | 144 | (defrecord SlaveID [value] 145 | Serializable 146 | (data->pb [this] 147 | (-> (Protos$SlaveID/newBuilder) 148 | (.setValue (str value)) 149 | (.build)))) 150 | 151 | (defmethod pb->data Protos$SlaveID 152 | [^Protos$SlaveID id] 153 | (SlaveID. (.getValue id))) 154 | 155 | ;; TaskID 156 | ;; ====== 157 | 158 | (defrecord TaskID [value] 159 | Serializable 160 | (data->pb [this] 161 | (-> (Protos$TaskID/newBuilder) 162 | (.setValue (str value)) 163 | (.build)))) 164 | 165 | (defmethod pb->data Protos$TaskID 166 | [^Protos$TaskID id] 167 | (TaskID. (.getValue id))) 168 | 169 | ;; ExecutorID 170 | ;; ========== 171 | 172 | (defrecord ExecutorID [value] 173 | Serializable 174 | (data->pb [this] 175 | (-> (Protos$ExecutorID/newBuilder) 176 | (.setValue (str value)) 177 | (.build)))) 178 | 179 | (defmethod pb->data Protos$ExecutorID 180 | [^Protos$ExecutorID id] 181 | (ExecutorID. (.getValue id))) 182 | 183 | ;; ContainerID 184 | ;; =========== 185 | 186 | (defrecord ContainerID [value] 187 | Serializable 188 | (data->pb [this] 189 | (-> (Protos$ContainerID/newBuilder) 190 | (.setValue (str value)) 191 | (.build)))) 192 | 193 | (defmethod pb->data Protos$ContainerID 194 | [^Protos$ContainerID id] 195 | (ContainerID. (.getValue id))) 196 | 197 | ;; TimeInfo 198 | ;; ======== 199 | 200 | (defrecord TimeInfo [nanoseconds] 201 | Serializable 202 | (data->pb [this] 203 | (-> (Protos$TimeInfo/newBuilder) 204 | (.setNanoseconds (long nanoseconds)) 205 | (.build)))) 206 | 207 | (defmethod pb->data Protos$TimeInfo 208 | [^Protos$TimeInfo time] 209 | (TimeInfo. (.getNanoseconds time))) 210 | 211 | ;; DurationInfo 212 | ;; ============ 213 | 214 | (defrecord DurationInfo [nanoseconds] 215 | Serializable 216 | (data->pb [this] 217 | (-> (Protos$DurationInfo/newBuilder) 218 | (.setNanoseconds (long nanoseconds)) 219 | (.build)))) 220 | 221 | (defmethod pb->data Protos$DurationInfo 222 | [^Protos$DurationInfo duration] 223 | (DurationInfo. (.getNanoseconds duration))) 224 | 225 | ;; Address 226 | ;; ======= 227 | 228 | (defrecord Address [hostname ip port] 229 | Serializable 230 | (data->pb [this] 231 | (-> (Protos$Address/newBuilder) 232 | (.setPort (int port)) 233 | (cond-> 234 | hostname (.setHostname (str hostname)) 235 | ip (.setIp (str ip))) 236 | (.build)))) 237 | 238 | (defmethod pb->data Protos$Address 239 | [^Protos$Address address] 240 | (Address. (.getHostname address) (.getIp address) (.getPort address))) 241 | 242 | ;; URL 243 | ;; === 244 | 245 | (defrecord URL [scheme address path query fragment] 246 | Serializable 247 | (data->pb [this] 248 | (-> (Protos$URL/newBuilder) 249 | (.setScheme (str scheme)) 250 | (.setAddress (->pb :Address address)) 251 | (.addAllQueries (mapv (partial ->pb :Parameter) query)) 252 | (cond-> fragment (.setFragment (str fragment))) 253 | (.build)))) 254 | 255 | (defmethod pb->data Protos$URL 256 | [^Protos$URL url] 257 | (URL. (.getScheme url) 258 | (pb->data (.getAddress url)) 259 | (.getPath url) 260 | (mapv pb->data (.getQueriesList url)) 261 | (.getFragment url))) 262 | 263 | ;; Unavailability 264 | ;; ============== 265 | 266 | (defrecord Unavailability [start duration] 267 | Serializable 268 | (data->pb [this] 269 | (-> (Protos$Unavailability/newBuilder) 270 | (.setStart (->pb :TimeInfo start)) 271 | (cond-> duration (.setDuration (->pb :Duration duration))) 272 | (.build)))) 273 | 274 | (defmethod pb->data Protos$Unavailability 275 | [^Protos$Unavailability una] 276 | (Unavailability. (.getStart una) (.getDuration una))) 277 | 278 | 279 | ;; MachineID 280 | ;; ========= 281 | 282 | (defrecord MachineID [hostname ip] 283 | Serializable 284 | (data->pb [this] 285 | (-> (Protos$MachineID/newBuilder) 286 | (cond-> hostname (.setHostname (str hostname)) 287 | ip (.setIp (str ip))) 288 | (.build)))) 289 | 290 | (defmethod pb->data Protos$MachineID 291 | [^Protos$MachineID mid] 292 | (MachineID. (.getHostname mid) (.getIp mid))) 293 | 294 | ;; MachineInfo 295 | ;; =========== 296 | 297 | (defmethod pb->data Protos$MachineInfo$Mode 298 | [^Protos$MachineInfo$Mode mode] 299 | (case-enum mode 300 | Protos$MachineInfo$Mode/UP :machine-mode-up 301 | Protos$MachineInfo$Mode/DRAINING :machine-mode-draining 302 | Protos$MachineInfo$Mode/DOWN :machine-mode-down)) 303 | 304 | (defrecord MachineInfo [id mode unavailability] 305 | Serializable 306 | (data->pb [this] 307 | (-> (Protos$MachineInfo/newBuilder) 308 | (.setId (->pb :MachineID id)) 309 | (cond-> 310 | mode (.setMode (data->pb mode)) 311 | unavailability (.setUnavailability 312 | (->pb :Unavailability unavailability)))))) 313 | 314 | (defmethod pb->data Protos$MachineInfo 315 | [^Protos$MachineInfo minfo] 316 | (MachineInfo. (pb->data (.getId minfo)) 317 | (pb->data (.getMode minfo)) 318 | (pb->data (.getUnavailability minfo)))) 319 | 320 | ;; FrameworkInfo 321 | ;; ============= 322 | 323 | (defrecord FrameworkCapability [type] 324 | Serializable 325 | (data->pb [this] 326 | (-> (Protos$FrameworkInfo$Capability/newBuilder) 327 | (.setType (data->pb type)) 328 | (.build)))) 329 | 330 | (defmethod pb->data Protos$FrameworkInfo$Capability 331 | [^Protos$FrameworkInfo$Capability capa] 332 | (FrameworkCapability. (pb->data (.getType capa)))) 333 | 334 | (defmethod pb->data Protos$FrameworkInfo$Capability$Type 335 | [^Protos$FrameworkInfo$Capability$Type type] 336 | (case-enum type 337 | Protos$FrameworkInfo$Capability$Type/UNKNOWN 338 | :framework-capability-unknown 339 | Protos$FrameworkInfo$Capability$Type/REVOCABLE_RESOURCES 340 | :framework-capability-revocable-resources 341 | Protos$FrameworkInfo$Capability$Type/TASK_KILLING_STATE 342 | :framework-capability-task-killing-state 343 | Protos$FrameworkInfo$Capability$Type/GPU_RESOURCES 344 | :framework-capability-gpu-resources)) 345 | 346 | (defrecord FrameworkInfo [user name id failover-timeout checkpoint role 347 | hostname principal webui-url capabilities labels] 348 | Serializable 349 | (data->pb [this] 350 | (-> (Protos$FrameworkInfo/newBuilder) 351 | (.setUser (str (or user ""))) 352 | (.setName (str name)) 353 | (cond-> 354 | id (.setId (->pb :FrameworkID id)) 355 | failover-timeout (.setFailoverTimeout (double failover-timeout)) 356 | (not (nil? checkpoint)) (.setCheckpoint (boolean checkpoint)) 357 | role (.setRole (str role)) 358 | hostname (.setHostname (str hostname)) 359 | principal (.setPrincipal (str principal)) 360 | webui-url (.setWebuiUrl (str webui-url)) 361 | labels (.setLabels (->pb :Labels labels))) 362 | (.addAllCapabilities (mapv (partial ->pb :FrameworkCapability) 363 | capabilities)) 364 | (.build)))) 365 | 366 | (defmethod pb->data Protos$FrameworkInfo 367 | [^Protos$FrameworkInfo info] 368 | (FrameworkInfo. 369 | (.getUser info) 370 | (.getName info) 371 | (when-let [id (.getId info)] (pb->data id)) 372 | (.getFailoverTimeout info) 373 | (.getCheckpoint info) 374 | (.getRole info) 375 | (.getHostname info) 376 | (.getPrincipal info) 377 | (.getWebuiUrl info) 378 | (when-let [labels (.getLabels info)] (pb->data labels)) 379 | (mapv pb->data (.getCapabilitiesList info)))) 380 | 381 | ;; HealthCheck 382 | ;; =========== 383 | 384 | (defrecord HealthCheckHTTP [port path statuses] 385 | Serializable 386 | (data->pb [this] 387 | (-> (Protos$HealthCheck$HTTP/newBuilder) 388 | (.setPort (int port)) 389 | (cond-> path (.setPath (str path))) 390 | (.addAllStatuses (mapv int statuses)) 391 | (.build)))) 392 | 393 | (defmethod pb->data Protos$HealthCheck$HTTP 394 | [^Protos$HealthCheck$HTTP http] 395 | (HealthCheckHTTP. 396 | (.getPort http) 397 | (.getPath http) 398 | (.getStatusesList http))) 399 | 400 | (defrecord HealthCheck [http delay-seconds interval-seconds 401 | timeout-seconds consecutive-failures 402 | grace-period-seconds command] 403 | Serializable 404 | (data->pb [this] 405 | (-> (Protos$HealthCheck/newBuilder) 406 | (cond-> 407 | http (.setHttp (->pb :HealthCheckHTTP http)) 408 | delay-seconds (.setDelaySeconds (double delay-seconds)) 409 | interval-seconds (.setIntervalSeconds (double interval-seconds)) 410 | timeout-seconds (.setTimeoutSeconds (double timeout-seconds)) 411 | consecutive-failures (.setConsecutiveFailures 412 | (int consecutive-failures)) 413 | grace-period-seconds (.setGracePeriodSeconds 414 | (double grace-period-seconds)) 415 | command (.setCommand (->pb :CommandInfo command))) 416 | (.build)))) 417 | 418 | (defmethod pb->data Protos$HealthCheck 419 | [^Protos$HealthCheck check] 420 | (HealthCheck. 421 | (when-let [http (.getHttp check)] (pb->data http)) 422 | (.getDelaySeconds check) 423 | (.getIntervalSeconds check) 424 | (.getTimeoutSeconds check) 425 | (.getConsecutiveFailures check) 426 | (.getGracePeriodSeconds check) 427 | (when-let [cmd (.getCommand check)] (pb->data cmd)))) 428 | 429 | ;; CommandInfo 430 | ;; =========== 431 | 432 | (defrecord URI [value executable extract cache] 433 | Serializable 434 | (data->pb [this] 435 | (-> (Protos$CommandInfo$URI/newBuilder) 436 | (.setValue (str value)) 437 | (cond-> 438 | (not (nil? executable)) (.setExecutable (boolean executable)) 439 | (not (nil? extract)) (.setExtract (boolean extract)) 440 | (not (nil? cache)) (.setCache (boolean cache))) 441 | (.build)))) 442 | 443 | (defmethod pb->data Protos$CommandInfo$URI 444 | [^Protos$CommandInfo$URI uri] 445 | (URI. (.getValue uri) (.getExecutable uri) (.getExtract uri) (.getCache uri))) 446 | 447 | (defrecord CommandInfo [uris environment shell value arguments user] 448 | Serializable 449 | (data->pb [this] 450 | (-> (Protos$CommandInfo/newBuilder) 451 | (cond-> 452 | environment (.setEnvironment (->pb :Environment environment)) 453 | (not (nil? shell)) (.setShell (boolean shell)) 454 | value (.setValue (str value)) 455 | user (.setUser (str user))) 456 | (.addAllArguments (map str arguments)) 457 | (.addAllUris (map (partial ->pb :URI) uris)) 458 | (.build)))) 459 | 460 | (defmethod pb->data Protos$CommandInfo 461 | [^Protos$CommandInfo info] 462 | (CommandInfo. 463 | (map pb->data (.getUrisList info)) 464 | (when-let [env (.getEnvironment info)] (pb->data env)) 465 | (.getShell info) 466 | (.getValue info) 467 | (.getArgumentsList info) 468 | (.getUser info))) 469 | 470 | ;; ExecutorInfo 471 | ;; ============ 472 | 473 | (defrecord ExecutorInfo [executor-id framework-id command container 474 | resources name source data discovery] 475 | Serializable 476 | (data->pb [this] 477 | (-> (Protos$ExecutorInfo/newBuilder) 478 | (.setExecutorId (->pb :ExecutorID executor-id)) 479 | (.setCommand (->pb :CommandInfo command)) 480 | (cond-> 481 | framework-id (.setFrameworkId (->pb :FrameworkID framework-id)) 482 | container (.setContainer (->pb :ContainerInfo container)) 483 | name (.setName (str name)) 484 | source (.setSource (str source)) 485 | data (.setData data) 486 | discovery (.setDiscovery discovery)) 487 | (.addAllResources (mapv (partial ->pb :Resource) resources)) 488 | (.build)))) 489 | 490 | (defmethod pb->data Protos$ExecutorInfo 491 | [^Protos$ExecutorInfo info] 492 | (ExecutorInfo. 493 | (pb->data (.getExecutorId info)) 494 | (when-let [id (.getFrameworkId info)] (pb->data id)) 495 | (pb->data (.getCommand info)) 496 | (when-let [cnt (.getContainer info)] (pb->data cnt)) 497 | (map pb->data (.getResourcesList info)) 498 | (.getName info) 499 | (.getSource info) 500 | (.getData info) 501 | (when-let [disco (.getDiscovery info)] (pb->data disco)))) 502 | 503 | ;; MasterInfo 504 | ;; ========== 505 | 506 | (defrecord MasterInfo [id ip port pid hostname version address] 507 | Serializable 508 | (data->pb [this] 509 | (-> (Protos$MasterInfo/newBuilder) 510 | (.setId (str id)) 511 | (.setIp (int ip)) 512 | (.setPort (int port)) 513 | (cond-> 514 | pid (.setPid (str pid)) 515 | hostname (.setHostname (str hostname)) 516 | version (.setVersion (str version)) 517 | address (.setAddress (->pb :Address address))) 518 | (.build)))) 519 | 520 | (defmethod pb->data Protos$MasterInfo 521 | [^Protos$MasterInfo info] 522 | (MasterInfo. 523 | (.getId info) 524 | (.getIp info) 525 | (.getPort info) 526 | (.getPid info) 527 | (.getHostname info) 528 | (.getVersion info) 529 | (when-let [address (.getAddress info)] (pb->data address)))) 530 | 531 | ;; SlaveInfo 532 | ;; ========= 533 | 534 | (defrecord SlaveInfo [hostname port resources attributes id checkpoint] 535 | Serializable 536 | (data->pb [this] 537 | (-> (Protos$SlaveInfo/newBuilder) 538 | (.setHostname (str hostname)) 539 | (cond-> 540 | port (.setPort (int port)) 541 | id (.setId (->pb :SlaveID id)) 542 | (not (nil? checkpoint)) (.setCheckPoint (boolean checkpoint))) 543 | (.addAllResources (mapv (partial ->pb :Resource) resources)) 544 | (.addAllAttributes (mapv (partial ->pb :Attribute) attributes)) 545 | (.build)))) 546 | 547 | (defmethod pb->data Protos$SlaveInfo 548 | [^Protos$SlaveInfo info] 549 | (SlaveInfo. 550 | (.getHostname info) 551 | (.getPort info) 552 | (mapv pb->data (.getResourcesList info)) 553 | (mapv pb->data (.getAttributesList info)) 554 | (when-let [id (.getId info)] (pb->data id)) 555 | (.getCheckpoint info))) 556 | 557 | ;; Value 558 | ;; ===== 559 | 560 | (defmethod pb->data Protos$Value$Type 561 | [^Protos$Value$Type type] 562 | (case-enum type 563 | Protos$Value$Type/SCALAR :value-scalar 564 | Protos$Value$Type/RANGES :value-ranges 565 | Protos$Value$Type/SET :value-set 566 | Protos$Value$Type/TEXT :value-text 567 | type)) 568 | 569 | (defmethod pb->data Protos$Value$Scalar 570 | [^Protos$Value$Scalar scalar] 571 | (.getValue scalar)) 572 | 573 | (defrecord ValueRange [begin end] 574 | Serializable 575 | (data->pb [this] 576 | (-> (Protos$Value$Range/newBuilder) 577 | (.setBegin (long begin)) 578 | (.setEnd (long end)) 579 | (.build)))) 580 | 581 | (defmethod pb->data Protos$Value$Range 582 | [^Protos$Value$Range range] 583 | (ValueRange. (.getBegin range) (.getEnd range))) 584 | 585 | (defrecord ValueRanges [ranges] 586 | Serializable 587 | (data->pb [this] 588 | (-> (Protos$Value$Ranges/newBuilder) 589 | (.addAllRange (mapv (partial ->pb :ValueRange) ranges)) 590 | (.build)))) 591 | 592 | (defmethod pb->data Protos$Value$Ranges 593 | [^Protos$Value$Ranges ranges] 594 | (ValueRanges. (.getRangeList ranges))) 595 | 596 | (defmethod pb->data Protos$Value$Set 597 | [^Protos$Value$Set value-set] 598 | (set (.getItemList value-set))) 599 | 600 | (defmethod pb->data Protos$Value$Text 601 | [^Protos$Value$Text x] 602 | (.getValue x)) 603 | 604 | (defrecord Value [type scalar ranges set text] 605 | Serializable 606 | (data->pb [this] 607 | (-> (Protos$Value/newBuilder) 608 | (.setType (data->pb type)) 609 | (cond-> 610 | scalar (.setScalar (data->pb scalar)) 611 | ranges (.setRanges (data->pb (ValueRanges. ranges))) 612 | set (.setSet (data->pb set)) 613 | text (.setText (data->pb text))) 614 | (.build)))) 615 | 616 | (defmethod pb->data Protos$Value 617 | [^Protos$Value v] 618 | (Value. 619 | (pb->data (.getType v)) 620 | (pb->data (.getScalar v)) 621 | (when-let [ranges (.getRanges v)] 622 | (map pb->data (.getRangeList ranges))) 623 | (when-let [s (.getSet v)] (pb->data s)) 624 | (when-let [t (.getText v)] (pb->data t)))) 625 | 626 | ;; Attribute 627 | ;; ========= 628 | 629 | (defrecord Attribute [name type scalar ranges set text] 630 | Serializable 631 | (data->pb [this] 632 | (-> (Protos$Attribute/newBuilder) 633 | (.setName name) 634 | (.setType (data->pb type)) 635 | (cond-> 636 | scalar (.setScalar (data->pb scalar)) 637 | ranges (.setRanges (data->pb (ValueRanges. ranges))) 638 | set (.setSet (data->pb set)) 639 | text (.setText (data->pb text))) 640 | (.build)))) 641 | 642 | (defmethod pb->data Protos$Attribute 643 | [^Protos$Attribute v] 644 | (Attribute. 645 | (.getName v) 646 | (pb->data (.getType v)) 647 | (pb->data (.getScalar v)) 648 | (when-let [ranges (.getRanges v)] 649 | (map pb->data (.getRangeList ranges))) 650 | (when-let [s (.getSet v)] (pb->data s)) 651 | (when-let [t (.getText v)] (pb->data t)))) 652 | 653 | ;; Resource 654 | ;; ======== 655 | 656 | (defrecord Resource [name type scalar ranges set role] 657 | Serializable 658 | (data->pb [this] 659 | (-> (Protos$Resource/newBuilder) 660 | (.setName name) 661 | (.setType (data->pb type)) 662 | (cond-> 663 | scalar (.setScalar (data->pb scalar)) 664 | ranges (.setRanges (data->pb (ValueRanges. ranges))) 665 | (seq set) (.setSet (data->pb set)) 666 | role (.setRole role)) 667 | (.build)))) 668 | 669 | (defmethod pb->data Protos$Resource 670 | [^Protos$Resource v] 671 | (Resource. 672 | (.getName v) 673 | (pb->data (.getType v)) 674 | (pb->data (.getScalar v)) 675 | (when-let [ranges (.getRanges v)] 676 | (mapv pb->data (.getRangeList ranges))) 677 | (when-let [s (.getSet v)] (pb->data s)) 678 | (.getRole v))) 679 | 680 | ;; ResourceStatistics 681 | ;; ================== 682 | 683 | (defrecord ResourceStatistics [timestamp cpus-user-time-secs 684 | cpus-system-time-secs cpus-limit 685 | cpus-nr-periods cpus-nr-throttled 686 | cpus-throttled-time-secs 687 | mem-rss-bytes mem-limit-bytes 688 | mem-file-bytes mem-anon-bytes 689 | mem-mapped-file-bytes perf 690 | net-rx-packets net-rx-bytes 691 | net-rx-errors net-rx-dropped 692 | net-tx-packets net-tx-bytes 693 | net-tx-errors net-tx-dropped 694 | net-tcp-rtt-microsecs-p50 695 | net-tcp-rtt-microsecs-p90 696 | net-tcp-rtt-microsecs-p95 697 | net-tcp-rtt-microsecs-p99] 698 | Serializable 699 | (data->pb [this] 700 | (-> (Protos$ResourceStatistics/newBuilder) 701 | (.setTimestamp (double timestamp)) 702 | (cond-> 703 | cpus-user-time-secs (.setCpusUserTimeSecs 704 | (double cpus-user-time-secs)) 705 | cpus-system-time-secs (.setCpusSystemTimeSecs 706 | (double cpus-system-time-secs)) 707 | cpus-limit (.setCpusLimit (double cpus-limit)) 708 | cpus-nr-periods (.setCpusNrPeriods 709 | (int cpus-nr-periods)) 710 | cpus-nr-throttled (.setCpusNrThrottled 711 | (int cpus-nr-throttled)) 712 | cpus-throttled-time-secs (.setCpusThrottledTimeSecs 713 | (double cpus-throttled-time-secs)) 714 | mem-rss-bytes (.setMemRssBytes (long mem-rss-bytes)) 715 | mem-limit-bytes (.setMemLimitBytes 716 | (long mem-limit-bytes)) 717 | mem-file-bytes (.setMemFileBytes 718 | (long mem-file-bytes)) 719 | mem-anon-bytes (.setMemAnonBytes 720 | (long mem-anon-bytes)) 721 | mem-mapped-file-bytes (.setMemMappedFileBytes 722 | (long mem-mapped-file-bytes)) 723 | perf (.setPerf (pb->data perf)) 724 | net-rx-packets (.setNetRxPackets (long net-rx-packets)) 725 | net-rx-bytes (.setNetRxBytes (long net-rx-bytes)) 726 | net-rx-errors (.setNetRxErrors (long net-rx-errors)) 727 | net-rx-dropped (.setNetRxDropped (long net-rx-dropped)) 728 | net-tx-packets (.setNetRxPackets (long net-tx-packets)) 729 | net-tx-bytes (.setNetRxBytes (long net-tx-bytes)) 730 | net-tx-errors (.setNetRxErrors (long net-tx-errors)) 731 | net-tx-dropped (.setNetRxDropped (long net-tx-dropped)) 732 | net-tcp-rtt-microsecs-p50 (.setNetTcpRttMicrosecsP50 733 | (double net-tcp-rtt-microsecs-p50)) 734 | net-tcp-rtt-microsecs-p90 (.setNetTcpRttMicrosecsP90 735 | (double net-tcp-rtt-microsecs-p90)) 736 | net-tcp-rtt-microsecs-p95 (.setNetTcpRttMicrosecsP95 737 | (double net-tcp-rtt-microsecs-p95)) 738 | net-tcp-rtt-microsecs-p99 (.setNetTcpRttMicrosecsP99 739 | (double net-tcp-rtt-microsecs-p99))) 740 | (.build)))) 741 | 742 | (defmethod pb->data Protos$ResourceStatistics 743 | [^Protos$ResourceStatistics s] 744 | (ResourceStatistics. 745 | (.getTimestamp s) 746 | (.getCpusUserTimeSecs s) 747 | (.getCpusSystemTimeSecs s) 748 | (.getCpusLimit s) 749 | (.getCpusNrPeriods s) 750 | (.getCpusNrThrottled s) 751 | (.getCpusThrottledTimeSecs s) 752 | (.getMemRssBytes s) 753 | (.getMemLimitBytes s) 754 | (.getMemFileBytes s) 755 | (.getMemAnonbytes s) 756 | (.getMemMappedfileBytes s) 757 | (when-let [p (.getPerf s)] (pb->data p)) 758 | (.getNetRxPackets s) 759 | (.getNetRxBytes s) 760 | (.getNetRxErrors s) 761 | (.getNetRxDropped s) 762 | (.getNetTxPackets s) 763 | (.getNetTxBytes s) 764 | (.getNetTxErrors s) 765 | (.getNetTxDropped s) 766 | (.getNetTcpRttMicrosecsP50 s) 767 | (.getNetTcpRttMicrosecsP90 s) 768 | (.getNetTcpRttMicrosecsP95 s) 769 | (.getNetTcpRttMicrosecsP99 s))) 770 | 771 | ;; ResourceUsage 772 | ;; ============= 773 | 774 | (defrecord ResourceUsage [slave-id framework-id executor-id executor-name 775 | task-id statistics] 776 | Serializable 777 | (data->pb [this] 778 | (-> (Protos$ResourceUsage/newBuilder) 779 | (.setSlaveId (data->pb slave-id)) 780 | (.setFrameworkId (data->pb framework-id)) 781 | (cond-> 782 | executor-id (.setExecutorId (data->pb executor-id)) 783 | executor-name (.setExecutorName (str executor-name)) 784 | task-id (.setTaskid (data->pb task-id)) 785 | statistics (.setStatistics (data->pb statistics))) 786 | (.build)))) 787 | 788 | (defmethod pb->data Protos$ResourceUsage 789 | [^Protos$ResourceUsage usage] 790 | (ResourceUsage. 791 | (pb->data (.getSlaveId usage)) 792 | (pb->data (.getFrameworkId usage)) 793 | (when-let [id (.getExecutorId usage)] (pb->data id)) 794 | (.getExecutorName usage) 795 | (when-let [id (.getTaskId usage)] (pb->data id)) 796 | (when-let [stats (.getResourceStatistics usage)] (pb->data stats)))) 797 | 798 | ;; PerfStatistics 799 | ;; ============== 800 | 801 | (defrecord PerfStatistics [timestamp duration cycles 802 | stalled-cycles-frontend 803 | stalled-cycles-backend 804 | instructions cache-references 805 | cache-misses branches branch-misses 806 | bus-cycles ref-cycles cpu-clock task-clock 807 | page-faults minor-faults major-faults 808 | context-switches cpu-migrations 809 | alignment-faults emulation-faults 810 | l1-dcache-loads l1-dcache-load-misses 811 | l1-dcache-stores l1-dcache-store-misses 812 | l1-dcache-prefetches l1-dcache-prefetch-misses 813 | l1-icache-loads l1-icache-load-misses 814 | l1-icache-prefetches l1-icache-prefetch-misses 815 | llc-loads llc-load-misses 816 | llc-stores llc-store-misses 817 | llc-prefetches llc-prefetch-misses 818 | dtlb-loads dtlb-load-misses 819 | dtlb-stores dtlb-store-misses 820 | dtlb-prefetches dtlb-prefetch-misses 821 | itlb-loads itlb-load-misses 822 | branch-loads branch-load-misses 823 | node-loads node-load-misses 824 | node-stores node-store-misses 825 | node-prefetches node-prefetch-misses] 826 | 827 | Serializable 828 | (data->pb [this] 829 | (let [b (-> (Protos$PerfStatistics/newBuilder) 830 | (.setTimestamp (double timestamp)) 831 | (.setDuration (double duration)))] 832 | 833 | (when cycles 834 | (.setCycles b (long cycles))) 835 | (when stalled-cycles-frontend 836 | (.setStalledCyclesFrontend b (long stalled-cycles-frontend))) 837 | (when stalled-cycles-backend 838 | (.setStalledCyclesBackend b (long stalled-cycles-backend))) 839 | (when instructions 840 | (.setInstructions b (long instructions))) 841 | (when cache-references 842 | (.setCacheReferences b (long cache-references))) 843 | (when cache-misses 844 | (.setCacheMisses b (long cache-misses))) 845 | (when branches 846 | (.setBranches b (long branches))) 847 | (when branch-misses 848 | (.setBranchMisses b (long branch-misses))) 849 | (when bus-cycles 850 | (.setBusCycles b (long bus-cycles))) 851 | (when ref-cycles 852 | (.setRefCycles b (long ref-cycles))) 853 | (when cpu-clock 854 | (.setCpuClock b (double cpu-clock))) 855 | (when task-clock 856 | (.setTaskClock b (double task-clock))) 857 | (when page-faults 858 | (.setPageFaults b (long page-faults))) 859 | (when major-faults 860 | (.setMajorFaults b (long major-faults))) 861 | (when context-switches 862 | (.setContextSwitches b (long context-switches))) 863 | (when cpu-migrations 864 | (.setCpuMigrations b (long cpu-migrations))) 865 | (when alignment-faults 866 | (.setAlignmentFaults b (long alignment-faults))) 867 | (when emulation-faults 868 | (.setEmulationFaults b (long emulation-faults))) 869 | (when l1-dcache-loads 870 | (.setL1DcacheLoads b (long l1-dcache-loads))) 871 | (when l1-dcache-load-misses 872 | (.setL1DcacheLoadMisses b (long l1-dcache-load-misses))) 873 | (when l1-dcache-stores 874 | (.setL1DcacheStores b (long l1-dcache-stores))) 875 | (when l1-dcache-store-misses 876 | (.setL1DcacheStoreMisses b (long l1-dcache-store-misses))) 877 | (when l1-dcache-prefetches 878 | (.setL1DcachePrefetches b (long l1-dcache-prefetches))) 879 | (when l1-dcache-prefetch-misses 880 | (.setL1DcachePrefetchMisses b (long l1-dcache-prefetch-misses))) 881 | (when l1-icache-loads 882 | (.setL1IcacheLoads b (long l1-icache-loads))) 883 | (when l1-icache-load-misses 884 | (.setL1IcacheLoadMisses b (long l1-icache-load-misses))) 885 | (when l1-icache-prefetches 886 | (.setL1IcachePrefetches b (long l1-icache-prefetches))) 887 | (when l1-icache-prefetch-misses 888 | (.setL1IcachePrefetchMisses b (long l1-icache-prefetch-misses))) 889 | (when llc-loads 890 | (.setLlcLoads b (long llc-loads))) 891 | (when llc-load-misses 892 | (.setLlcLoadMisses b (long llc-load-misses))) 893 | (when llc-stores 894 | (.setLlcStores b (long llc-stores))) 895 | (when llc-store-misses 896 | (.setLlcStoreMisses b (long llc-store-misses))) 897 | (when llc-prefetches 898 | (.setLlcPrefetches b (long llc-prefetches))) 899 | (when llc-prefetch-misses 900 | (.setLlcPrefetchMisses b (long llc-prefetch-misses))) 901 | (when dtlb-loads 902 | (.setDtlbLoads b (long dtlb-loads))) 903 | (when dtlb-load-misses 904 | (.setDtlbLoadMisses b (long dtlb-load-misses))) 905 | (when dtlb-stores 906 | (.setDtlbStores b (long dtlb-stores))) 907 | (when dtlb-store-misses 908 | (.setDtlbStoreMisses b (long dtlb-store-misses))) 909 | (when dtlb-prefetches 910 | (.setDtlbPrefetches b (long dtlb-prefetches))) 911 | (when dtlb-prefetch-misses 912 | (.setDtlbPrefetchMisses b (long dtlb-prefetch-misses))) 913 | (when itlb-loads 914 | (.setItlbLoads b (long itlb-loads))) 915 | (when itlb-load-misses 916 | (.setItlbLoadMisses b (long itlb-load-misses))) 917 | (when branch-loads 918 | (.setBranchLoads b (long branch-loads))) 919 | (when branch-load-misses 920 | (.setBranchLoadMisses b (long branch-load-misses))) 921 | (when node-loads 922 | (.setNodeLoads b (long node-loads))) 923 | (when node-load-misses 924 | (.setNodeLoadMisses b (long node-load-misses))) 925 | (when node-stores 926 | (.setNodeStores b (long node-stores))) 927 | (when node-store-misses 928 | (.setNodeStoreMisses b (long node-store-misses))) 929 | (when node-prefetches 930 | (.setNodePrefetches b (long node-prefetches))) 931 | (when node-prefetch-misses 932 | (.setNodePrefetchMisses b (long node-prefetch-misses))) 933 | (.build b)))) 934 | 935 | (defmethod pb->data Protos$PerfStatistics 936 | [^Protos$PerfStatistics s] 937 | (PerfStatistics. 938 | (.getTimestamp s) 939 | (.getDuration s) 940 | (.getCycles s) 941 | (.getStalledCyclesFrontend s) 942 | (.getStalledCyclesBackend s) 943 | (.getInstructions s) 944 | (.getCacheReferences s) 945 | (.getCacheMisses s) 946 | (.getBranches s) 947 | (.getBranchMisses s) 948 | (.getBusCycles s) 949 | (.getRefCycles s) 950 | (.getCpuClock s) 951 | (.getTaskClock s) 952 | (.getPageFaults s) 953 | (.getMajorFaults s) 954 | (.getMinorFaults s) 955 | (.getContextSwitches s) 956 | (.getCpuMigrations s) 957 | (.getAlignmentFaults s) 958 | (.getEmulationFaults s) 959 | (.getL1DcacheLoads s) 960 | (.getL1DcacheLoadMisses s) 961 | (.getL1DcacheStores s) 962 | (.getL1DcacheStoreMisses s) 963 | (.getL1DcacheLoadPrefetches s) 964 | (.getL1DcacheLoadPrefetchMisses s) 965 | (.getL1IcacheLoads s) 966 | (.getL1IcacheLoadMisses s) 967 | (.getL1IcachePrefetches s) 968 | (.getL1IcachePrefetchMisses s) 969 | (.getLlcLoads s) 970 | (.getLlcLoadMisses s) 971 | (.getLlcStores s) 972 | (.getLlcStoreMisses s) 973 | (.getLlcPrefetches s) 974 | (.getLlcPrefetchMisses s) 975 | (.getDtlbLoads s) 976 | (.getDtlbLoadMisses s) 977 | (.getDtlbStores s) 978 | (.getDtlbStoreMisses s) 979 | (.getDtlbPrefetches s) 980 | (.getDtlbPrefetchMisses s) 981 | (.getItlbLoads s) 982 | (.getItlbLoadMisses s) 983 | (.getBranchLoads s) 984 | (.getBranchLoadMisses s) 985 | (.getNodeLoads s) 986 | (.getNodeLoadMisses s) 987 | (.getNodeStores s) 988 | (.getNodeStoreMisses s) 989 | (.getNodePrefetches s) 990 | (.getNodePrefetchMisses s))) 991 | 992 | ;; Request 993 | ;; ======= 994 | 995 | (defrecord Request [slave-id resources] 996 | Serializable 997 | (data->pb [this] 998 | (-> (Protos$Request/newBuilder) 999 | (cond-> slave-id (.setSlaveId (->pb :SlaveID slave-id))) 1000 | (.addAllResources (map (partial ->pb :Resource) resources)) 1001 | (.build)))) 1002 | 1003 | (defmethod pb->data Protos$Request 1004 | [^Protos$Request req] 1005 | (Request. 1006 | (when-let [id (.getSlaveId req)] (pb->data id)) 1007 | (map pb->data (.getResourcesList req)))) 1008 | 1009 | ;; Offer 1010 | ;; ===== 1011 | 1012 | (defrecord Offer [id framework-id slave-id hostname 1013 | resources attributes executor-ids] 1014 | Serializable 1015 | (data->pb [this] 1016 | (-> (Protos$Offer/newBuilder) 1017 | (.setId (->pb :OfferID id)) 1018 | (.setFrameworkId (->pb :FrameworkID framework-id)) 1019 | (.setSlaveId (->pb :SlaveID slave-id)) 1020 | (.setHostname (str hostname)) 1021 | (.addAllResources (mapv (partial ->pb :Resource) resources)) 1022 | (.addAllAttributes (mapv (partial ->pb :Attribute) attributes)) 1023 | (.addAllExecutorIds (mapv (partial ->pb :ExecutorID) attributes)) 1024 | (.build)))) 1025 | 1026 | (defmethod pb->data Protos$Offer 1027 | [^Protos$Offer offer] 1028 | (Offer. 1029 | (pb->data (.getId offer)) 1030 | (pb->data (.getFrameworkId offer)) 1031 | (pb->data (.getSlaveId offer)) 1032 | (.getHostname offer) 1033 | (mapv pb->data (.getResourcesList offer)) 1034 | (mapv pb->data (.getAttributesList offer)) 1035 | (mapv pb->data (.getExecutorIdsList offer)))) 1036 | 1037 | ;; Operation 1038 | ;; ========= 1039 | 1040 | (defmethod pb->data Protos$Offer$Operation$Type 1041 | [^Protos$Offer$Operation$Type type] 1042 | (case-enum type 1043 | Protos$Offer$Operation$Type/LAUNCH :operation-launch 1044 | Protos$Offer$Operation$Type/RESERVE :operation-reserve 1045 | Protos$Offer$Operation$Type/UNRESERVE :operation-unreserve 1046 | Protos$Offer$Operation$Type/CREATE :operation-create 1047 | Protos$Offer$Operation$Type/DESTROY :operation-destroy 1048 | type)) 1049 | 1050 | (defrecord Operation [type tasks resources volumes] 1051 | Serializable 1052 | (data->pb [this] 1053 | (case (:type this) 1054 | :operation-launch 1055 | (let [launch (Protos$Offer$Operation$Launch/newBuilder)] 1056 | (.addAllTaskInfos 1057 | launch (mapv (partial ->pb :TaskInfo) tasks)) 1058 | (-> (Protos$Offer$Operation/newBuilder) 1059 | (.setType (Protos$Offer$Operation$Type/LAUNCH)) 1060 | (.setLaunch launch) 1061 | (.build))) 1062 | :operation-reserve 1063 | (let [reserve (Protos$Offer$Operation$Reserve/newBuilder)] 1064 | (.addAllResrouces 1065 | reserve (mapv (partial ->pb :Resource) resources)) 1066 | (-> (Protos$Offer$Operation/newBuilder) 1067 | (.setType (Protos$Offer$Operation$Type/RESERVE)) 1068 | (.setLaunch reserve) 1069 | (.build))) 1070 | :operation-unreserve 1071 | (let [unreserve (Protos$Offer$Operation$Unreserve/newBuilder)] 1072 | (.addAllResrouces 1073 | unreserve (mapv (partial ->pb :Resource) resources)) 1074 | (-> (Protos$Offer$Operation/newBuilder) 1075 | (.setType (Protos$Offer$Operation$Type/UNRESERVE)) 1076 | (.setLaunch unreserve) 1077 | (.build))) 1078 | :operation-create 1079 | (let [create (Protos$Offer$Operation$Create/newBuilder)] 1080 | (.addAllVolumes 1081 | create (mapv (partial ->pb :Volume) volumes)) 1082 | (-> (Protos$Offer$Operation/newBuilder) 1083 | (.setType (Protos$Offer$Operation$Type/CREATE)) 1084 | (.setLaunch create) 1085 | (.build))) 1086 | :operation-destroy 1087 | (let [destroy (Protos$Offer$Operation$Destroy/newBuilder)] 1088 | (.addAllVolumes 1089 | destroy (mapv (partial ->pb :Volume) volumes)) 1090 | (-> (Protos$Offer$Operation/newBuilder) 1091 | (.setType (Protos$Offer$Operation$Type/DESTROY)) 1092 | (.setLaunch destroy) 1093 | (.build)))))) 1094 | 1095 | (defmethod pb->data Protos$Offer$Operation 1096 | [^Protos$Offer$Operation op] 1097 | (let [type (pb->data (.getType op))] 1098 | (case type 1099 | :operation-launch 1100 | (Operation. type 1101 | (map pb->data (.getTaskInfosList (.getLaunch op))) 1102 | nil 1103 | nil) 1104 | :operation-reserve 1105 | (Operation. type 1106 | nil 1107 | (map pb->data (.getResourcesList (.getReserve op))) 1108 | nil) 1109 | :operation-unreserve 1110 | (Operation. type 1111 | nil 1112 | (map pb->data (.getResourcesList (.getUnreserve op))) 1113 | nil) 1114 | :operation-create 1115 | (Operation. type 1116 | (map pb->data (.getVolumesList (.getCreate op))) 1117 | nil 1118 | nil) 1119 | :operation-destroy 1120 | (Operation. type 1121 | (map pb->data (.getVolumesList (.getDestroy op))) 1122 | nil 1123 | nil)))) 1124 | 1125 | ;; Ports 1126 | ;; ===== 1127 | 1128 | (defrecord Port [number name protocol visibility labels] 1129 | Serializable 1130 | (data->pb [this] 1131 | (-> (Protos$Port/newBuilder) 1132 | (.setNumber (int number)) 1133 | (cond-> name (.setName (str name)) 1134 | protocol (.setProtocol (str protocol)) 1135 | visibility (.setVisibility (pb->data visibility)) 1136 | labels (.setLabels (->pb :Labels labels))) 1137 | (.build)))) 1138 | 1139 | (defmethod pb->data Protos$Port 1140 | [^Protos$Port port] 1141 | (Port. (.getNumber port) 1142 | (.getName port) 1143 | (.getProtocol port) 1144 | (when-let [vis (.getVisibility port)] (pb->data vis)) 1145 | (when-let [labels (.getLabels port)] (pb->data labels)))) 1146 | 1147 | (defrecord Ports [ports] 1148 | Serializable 1149 | (data->pb [this] 1150 | (-> (Protos$Ports/newBuilder) 1151 | (.addAllPorts (mapv (partial ->pb :Port ports))) 1152 | (.build)))) 1153 | 1154 | (defmethod pb->data Protos$Ports 1155 | [^Protos$Ports ports] 1156 | (Ports. (mapv pb->data (.getPortsList ports)))) 1157 | 1158 | ;; DiscoveryInfo 1159 | ;; ============= 1160 | 1161 | (defrecord DiscoveryInfo [visibility name environment location 1162 | version ports labels] 1163 | Serializable 1164 | (data->pb [this] 1165 | (-> (Protos$DiscoveryInfo/newBuilder) 1166 | (.setVisibility visibility) 1167 | (cond-> name (.setName (str name)) 1168 | environment (.setEnvionment (str environment)) 1169 | location (.setLocation (str location)) 1170 | ports (.setPorts (->pb :Ports ports)) 1171 | version (.setVersion (str version)) 1172 | labels (.setLabels (->pb :Labels labels)))))) 1173 | 1174 | (defmethod pb->data Protos$DiscoveryInfo 1175 | [^Protos$DiscoveryInfo di] 1176 | (DiscoveryInfo. (pb->data (.getVisibility di)) 1177 | (.getName di) 1178 | (.getEnvironment di) 1179 | (.getLocation di) 1180 | (.getVersion di) 1181 | (when-let [ports (.getPorts di)] (pb->data ports)) 1182 | (when-let [labels (.getLabels di)] (pb->data labels)))) 1183 | 1184 | (defmethod pb->data Protos$DiscoveryInfo$Visibility 1185 | [^Protos$DiscoveryInfo$Visibility vis] 1186 | (case-enum vis 1187 | Protos$DiscoveryInfo$Visibility/FRAMEWORK 1188 | :discovery-visibility-framework 1189 | Protos$DiscoveryInfo$Visibility/CLUSTER 1190 | :discovery-visibility-cluster 1191 | Protos$DiscoveryInfo$Visibility/EXTERNAL 1192 | :discovery-visibility-external)) 1193 | 1194 | ;; TaskInfo 1195 | ;; ======== 1196 | 1197 | (defrecord TaskInfo [name task-id slave-id resources executor command 1198 | container data health-check labels discovery] 1199 | Serializable 1200 | (data->pb [this] 1201 | (-> (Protos$TaskInfo/newBuilder) 1202 | (.setName (str name)) 1203 | (.setTaskId (->pb :TaskID task-id)) 1204 | (.setSlaveId (->pb :SlaveID slave-id)) 1205 | (cond-> 1206 | executor (.setExecutor (->pb :ExecutorInfo executor)) 1207 | command (.setCommand (->pb :CommandInfo command)) 1208 | container (.setContainer (->pb :ContainerInfo container)) 1209 | data (.setData data) 1210 | health-check (.setHealthCheck (->pb :HealthCheck health-check)) 1211 | labels (.setLabels (->pb :Labels labels)) 1212 | discovery (.setDiscovery (discovery :DiscoveryInfo))) 1213 | (.addAllResources (mapv (partial ->pb :Resource) resources)) 1214 | (.build)))) 1215 | 1216 | (defmethod pb->data Protos$TaskInfo 1217 | [^Protos$TaskInfo info] 1218 | (TaskInfo. 1219 | (.getName info) 1220 | (pb->data (.getTaskId info)) 1221 | (pb->data (.getSlaveId info)) 1222 | (mapv pb->data (.getResourcesList info)) 1223 | (when-let [i (.getExecutor info)] (pb->data i)) 1224 | (when-let [c (.getCommand info)] (pb->data c)) 1225 | (when-let [c (.getContainer info)] (pb->data c)) 1226 | (.getData info) 1227 | (when-let [hc (.getHealthCheck info)] (pb->data hc)) 1228 | (when-let [l (.getLabels info)] (pb->data l)) 1229 | (when-let [d (.getDiscovery info)] (pb->data d)))) 1230 | 1231 | ;; TaskState 1232 | ;; ========= 1233 | 1234 | (defmethod pb->data Protos$TaskState 1235 | [^Protos$TaskState status] 1236 | (case-enum status 1237 | Protos$TaskState/TASK_STAGING :task-staging 1238 | Protos$TaskState/TASK_STARTING :task-starting 1239 | Protos$TaskState/TASK_RUNNING :task-running 1240 | Protos$TaskState/TASK_FINISHED :task-finished 1241 | Protos$TaskState/TASK_FAILED :task-failed 1242 | Protos$TaskState/TASK_KILLED :task-killed 1243 | Protos$TaskState/TASK_LOST :task-lost 1244 | Protos$TaskState/TASK_ERROR :task-error 1245 | status)) 1246 | 1247 | ;; TaskStatus 1248 | ;; ========== 1249 | 1250 | (defmethod pb->data Protos$TaskStatus$Source 1251 | [^Protos$TaskStatus$Source status] 1252 | (case-enum status 1253 | Protos$TaskStatus$Source/SOURCE_MASTER :source-master 1254 | Protos$TaskStatus$Source/SOURCE_SLAVE :source-slave 1255 | Protos$TaskStatus$Source/SOURCE_EXECUTOR :source-executor 1256 | status)) 1257 | 1258 | (defmethod pb->data Protos$TaskStatus$Reason 1259 | [^Protos$TaskStatus$Reason status] 1260 | (case-enum status 1261 | Protos$TaskStatus$Reason/REASON_COMMAND_EXECUTOR_FAILED 1262 | :reason-command-executor-failed 1263 | Protos$TaskStatus$Reason/REASON_CONTAINER_LAUNCH_FAILED 1264 | :reason-container-launch-failed 1265 | Protos$TaskStatus$Reason/REASON_CONTAINER_LIMITATION 1266 | :reason-container-limitation 1267 | Protos$TaskStatus$Reason/REASON_CONTAINER_LIMITATION_DISK 1268 | :reason-container-limitation-disk 1269 | Protos$TaskStatus$Reason/REASON_CONTAINER_LIMITATION_MEMORY 1270 | :reason-container-limitation-memory 1271 | Protos$TaskStatus$Reason/REASON_CONTAINER_PREEMPTED 1272 | :reason-container-preempted 1273 | Protos$TaskStatus$Reason/REASON_CONTAINER_UPDATE_FAILED 1274 | :reason-container-update-failed 1275 | Protos$TaskStatus$Reason/REASON_EXECUTOR_REGISTRATION_TIMEOUT 1276 | :reason-executor-registration-timeout 1277 | Protos$TaskStatus$Reason/REASON_EXECUTOR_REREGISTRATION_TIMEOUT 1278 | :reason-executor-reregistration-timeout 1279 | Protos$TaskStatus$Reason/REASON_EXECUTOR_TERMINATED 1280 | :reason-executor-terminated 1281 | Protos$TaskStatus$Reason/REASON_EXECUTOR_UNREGISTERED 1282 | :reason-executor-unregistered 1283 | Protos$TaskStatus$Reason/REASON_FRAMEWORK_REMOVED 1284 | :reason-framework-removed 1285 | Protos$TaskStatus$Reason/REASON_GC_ERROR 1286 | :reason-gc-error 1287 | Protos$TaskStatus$Reason/REASON_INVALID_FRAMEWORKID 1288 | :reason-invalid-frameworkid 1289 | Protos$TaskStatus$Reason/REASON_INVALID_OFFERS 1290 | :reason-invalid-offers 1291 | Protos$TaskStatus$Reason/REASON_MASTER_DISCONNECTED 1292 | :reason-master-disconnected 1293 | Protos$TaskStatus$Reason/REASON_RECONCILIATION 1294 | :reason-reconciliation 1295 | Protos$TaskStatus$Reason/REASON_RESOURCES_UNKNOWN 1296 | :reason-resources-unknown 1297 | Protos$TaskStatus$Reason/REASON_SLAVE_DISCONNECTED 1298 | :reason-slave-disconnected 1299 | Protos$TaskStatus$Reason/REASON_SLAVE_REMOVED 1300 | :reason-slave-removed 1301 | Protos$TaskStatus$Reason/REASON_SLAVE_RESTARTED 1302 | :reason-slave-restarted 1303 | Protos$TaskStatus$Reason/REASON_SLAVE_UNKNOWN 1304 | :reason-slave-unknown 1305 | Protos$TaskStatus$Reason/REASON_TASK_INVALID 1306 | :reason-task-invalid 1307 | Protos$TaskStatus$Reason/REASON_TASK_UNAUTHORIZED 1308 | :reason-task-unauthorized 1309 | Protos$TaskStatus$Reason/REASON_TASK_UNKNOWN 1310 | :reason-task-unknown 1311 | status)) 1312 | 1313 | (defrecord TaskStatus [task-id state message source reason 1314 | data slave-id executor-id timestamp 1315 | uuid healthy] 1316 | Serializable 1317 | (data->pb [this] 1318 | (-> (Protos$TaskStatus/newBuilder) 1319 | (.setTaskId (->pb :TaskID task-id)) 1320 | (.setState (data->pb state)) 1321 | (cond-> 1322 | message (.setMessage (str message)) 1323 | source (.setSource (data->pb source)) 1324 | reason (.setReason (data->pb reason)) 1325 | data (.setData data) 1326 | slave-id (.setSlaveId (->pb :SlaveID slave-id)) 1327 | executor-id (.setExecutorId (->pb :ExecutorID executor-id)) 1328 | timestamp (.setTimestamp (double timestamp)) 1329 | uuid (.setUuid uuid) 1330 | (not (nil? healthy)) (.setHealthy (boolean healthy))) 1331 | (.build)))) 1332 | 1333 | (defmethod pb->data Protos$TaskStatus 1334 | [^Protos$TaskStatus status] 1335 | (TaskStatus. 1336 | (pb->data (.getTaskId status)) 1337 | (pb->data (.getState status)) 1338 | (.getMessage status) 1339 | (pb->data (.getSource status)) 1340 | (pb->data (.getReason status)) 1341 | (.getData status) 1342 | (pb->data (.getSlaveId status)) 1343 | (pb->data (.getExecutorId status)) 1344 | (.getTimestamp status) 1345 | (.getUuid status) 1346 | (.getHealthy status))) 1347 | 1348 | ;; Filters 1349 | ;; ======= 1350 | 1351 | (defrecord Filters [refuse-seconds] 1352 | Serializable 1353 | (data->pb [this] 1354 | (-> (Protos$Filters/newBuilder) 1355 | (cond-> refuse-seconds (.setRefuseSeconds (double refuse-seconds))) 1356 | (.build)))) 1357 | 1358 | (defmethod pb->data Protos$Filters 1359 | [^Protos$Filters filters] 1360 | (Filters. (.getRefuseSeconds filters))) 1361 | 1362 | ;; Environment 1363 | ;; =========== 1364 | 1365 | (defrecord EnvironmentVariable [name value] 1366 | Serializable 1367 | (data->pb [this] 1368 | (-> (Protos$Environment$Variable/newBuilder) 1369 | (.setName (str name)) 1370 | (.setValue (str value)) 1371 | (.build)))) 1372 | 1373 | (defmethod pb->data Protos$Environment$Variable 1374 | [^Protos$Environment$Variable var] 1375 | (EnvironmentVariable. (.getName var) (.getValue var))) 1376 | 1377 | (defrecord Environment [variables] 1378 | Serializable 1379 | (data->pb [this] 1380 | (-> (Protos$Environment/newBuilder) 1381 | (.addAllVariables (mapv (partial ->pb :EnvironmentVariable) variables)) 1382 | (.build)))) 1383 | 1384 | ;; Parameter 1385 | ;; ========= 1386 | 1387 | (defrecord Parameter [key value] 1388 | Serializable 1389 | (data->pb [this] 1390 | (-> (Protos$Parameter/newBuilder) 1391 | (.setKey (str key)) 1392 | (.setValue (str value)) 1393 | (.build)))) 1394 | 1395 | (defmethod pb->data Protos$Parameter 1396 | [^Protos$Parameter p] 1397 | (Parameter. (.getKey p) (.getValue p))) 1398 | 1399 | (defrecord Parameters [parameter] 1400 | Serializable 1401 | (data->pb [this] 1402 | (-> (Protos$Parameters/newBuilder) 1403 | (.addAllParameters (mapv (partial ->pb :Parameter) parameter)) 1404 | (.build)))) 1405 | 1406 | (defmethod pb->data Protos$Parameters 1407 | [^Protos$Parameters p] 1408 | (Parameters. (mapv pb->data (.getParameterList p)))) 1409 | 1410 | ;; Credential 1411 | ;; ========== 1412 | 1413 | (defrecord Credential [principal secret] 1414 | Serializable 1415 | (data->pb [this] 1416 | (-> (Protos$Credential/newBuilder) 1417 | (.setPrincipal (str principal)) 1418 | (cond-> secret (.setSecret (str secret))) 1419 | (.build)))) 1420 | 1421 | (defmethod pb->data Protos$Credential 1422 | [^Protos$Credential c] 1423 | (Credential. (.getPrincipal c) (.getSecret c))) 1424 | 1425 | (defrecord Credentials [credentials] 1426 | Serializable 1427 | (data->pb [this] 1428 | (-> (Protos$Credentials/newBuilder) 1429 | (.addAllCredentials (mapv (partial ->pb :Credential) credentials)) 1430 | (.build)))) 1431 | 1432 | (defmethod pb->data Protos$Credentials 1433 | [^Protos$Credentials c] 1434 | (Credentials. (mapv pb->data (.getCredentialsList c)))) 1435 | 1436 | ;; RateLimit 1437 | ;; ========= 1438 | 1439 | (defrecord RateLimit [qps principal capacity] 1440 | Serializable 1441 | (data->pb [this] 1442 | (-> (Protos$RateLimit/newBuilder) 1443 | (.setPrincipal (str principal)) 1444 | (cond-> 1445 | qps (.setQps (double qps)) 1446 | capacity (.setCapacity (long capacity))) 1447 | (.build)))) 1448 | 1449 | (defmethod pb->data Protos$RateLimit 1450 | [^Protos$RateLimit rl] 1451 | (RateLimit. 1452 | (.getQps rl) 1453 | (.getPrincipal rl) 1454 | (.getCapacity rl))) 1455 | 1456 | (defrecord RateLimits [limits aggregate-default-qps 1457 | aggregate-default-capacity] 1458 | Serializable 1459 | (data->pb [this] 1460 | (-> (Protos$RateLimits/newBuilder) 1461 | (cond-> 1462 | aggregate-default-qps (.setAggregateDefaultQps 1463 | (double aggregate-default-qps)) 1464 | aggregate-default-capacity (.setAggregateDefaultCapacity 1465 | (long aggregate-default-capacity))) 1466 | (.addAllLimits (mapv (partial ->pb :RateLimit) limits)) 1467 | (.build)))) 1468 | 1469 | (defmethod pb->data Protos$RateLimits 1470 | [^Protos$RateLimits rl] 1471 | (RateLimits. 1472 | (mapv pb->data (.getLimitsList rl)) 1473 | (.getAggregateDefaultQps rl) 1474 | (.getAggregateDefaultCapacity rl))) 1475 | 1476 | ;; Volume 1477 | ;; ====== 1478 | 1479 | (defmethod pb->data Protos$Volume$Mode 1480 | [^Protos$Volume$Mode mode] 1481 | (case-enum mode 1482 | Protos$Volume$Mode/RW :volume-rw 1483 | Protos$Volume$Mode/RO :volume-ro 1484 | mode)) 1485 | 1486 | (defrecord Volume [container-path host-path mode] 1487 | Serializable 1488 | (data->pb [this] 1489 | (-> (Protos$Volume/newBuilder) 1490 | (.setContainerPath (str container-path)) 1491 | (.setMode (data->pb mode)) 1492 | (cond-> host-path (.setHostPath (str host-path))) 1493 | (.build)))) 1494 | 1495 | ;; ContainerInfo 1496 | ;; ============= 1497 | 1498 | (defmethod pb->data Protos$ContainerInfo$Type 1499 | [^Protos$ContainerInfo$Type type] 1500 | (cond type 1501 | Protos$ContainerInfo$Type/DOCKER :container-type-docker 1502 | Protos$ContainerInfo$Type/MESOS :container-type-mesos 1503 | type)) 1504 | 1505 | (defrecord PortMapping [host-port container-port protocol] 1506 | Serializable 1507 | (data->pb [this] 1508 | (-> (Protos$ContainerInfo$DockerInfo$PortMapping/newBuilder) 1509 | (.setHostPort (int host-port)) 1510 | (.setContainerPort (int container-port)) 1511 | (cond-> protocol (.setProtocol protocol)) 1512 | (.build)))) 1513 | 1514 | (defmethod pb->data Protos$ContainerInfo$DockerInfo$PortMapping 1515 | [^Protos$ContainerInfo$DockerInfo$PortMapping pm] 1516 | (PortMapping. 1517 | (.getHostPort pm) 1518 | (.getContainerPort pm) 1519 | (.getProtocol pm))) 1520 | 1521 | (defmethod pb->data Protos$ContainerInfo$DockerInfo$Network 1522 | [^Protos$ContainerInfo$DockerInfo$Network network] 1523 | (case-enum network 1524 | Protos$ContainerInfo$DockerInfo$Network/HOST 1525 | :docker-network-host 1526 | Protos$ContainerInfo$DockerInfo$Network/BRIDGE 1527 | :docker-network-bridge 1528 | Protos$ContainerInfo$DockerInfo$Network/NONE 1529 | :docker-network-none 1530 | network)) 1531 | 1532 | (defrecord DockerInfo [image network force-pull-image port-mappings 1533 | privileged parameters] 1534 | Serializable 1535 | (data->pb [this] 1536 | (-> (Protos$ContainerInfo$DockerInfo/newBuilder) 1537 | (.setImage image) 1538 | (cond-> 1539 | network (.setNetwork (data->pb network)) 1540 | (not (nil? privileged)) (.setForcePullImage (boolean force-pull-image)) 1541 | (not (nil? privileged)) (.setPrivileged (boolean privileged))) 1542 | (.addAllPortMappings (mapv (partial ->pb :PortMapping) port-mappings)) 1543 | (.addAllParameters (mapv (partial ->pb :Parameter) parameters)) 1544 | (.build)))) 1545 | 1546 | (defrecord ContainerInfo [type volumes hostname docker] 1547 | Serializable 1548 | (data->pb [this] 1549 | (-> (Protos$ContainerInfo/newBuilder) 1550 | (.setType (data->pb type)) 1551 | (cond-> 1552 | hostname (.setHostname (str hostname)) 1553 | docker (.setDocker (->pb :DockerInfo docker))) 1554 | (.addAllVolumes (mapv (partial ->pb :Volume) volumes)) 1555 | (.build)))) 1556 | 1557 | (defrecord Label [key value] 1558 | Serializable 1559 | (data->pb [this] 1560 | (-> (Protos$Label/newBuilder) 1561 | (.setKey (str key)) 1562 | (cond-> value (.setValue (str value))) 1563 | (.build)))) 1564 | 1565 | (defmethod pb->data Protos$Label 1566 | [^Protos$Label label] 1567 | (Label. (.getKey label) (.getValue label))) 1568 | 1569 | (defrecord Labels [labels] 1570 | Serializable 1571 | (data->pb [this] 1572 | (-> (Protos$Labels/newBuilder) 1573 | (.addAllLabels (mapv (partial ->pb :Label) labels)) 1574 | (.build)))) 1575 | 1576 | (defmethod pb->data Protos$Labels 1577 | [^Protos$Labels labels] 1578 | (Labels. (mapv pb->data (.getLabelsList labels)))) 1579 | 1580 | 1581 | ;; Safe for the common stuff in extend-protocol, this marks the end of 1582 | ;; messages defined in mesos.proto 1583 | 1584 | ;; Handle serialization of Status, Value.Type and TaskState from 1585 | ;; keywords, as well as Scalars from their value. 1586 | 1587 | (extend-protocol Serializable 1588 | java.lang.Integer 1589 | (data->pb [this] 1590 | (-> (Protos$Value$Scalar/newBuilder) 1591 | (.setValue (double this)) 1592 | (.build))) 1593 | java.lang.Long 1594 | (data->pb [this] 1595 | (-> (Protos$Value$Scalar/newBuilder) 1596 | (.setValue (double this)) 1597 | (.build))) 1598 | java.lang.Double 1599 | (data->pb [this] 1600 | (-> (Protos$Value$Scalar/newBuilder) 1601 | (.setValue this) 1602 | (.build))) 1603 | clojure.lang.PersistentHashSet 1604 | (data->pb [this] 1605 | (-> (Protos$Value$Set/newBuilder) 1606 | (.addAllItem (seq this)) 1607 | (.build))) 1608 | java.lang.String 1609 | (data->pb [this] 1610 | (-> (Protos$Value$Text/newBuilder) (.setValue this) (.build))) 1611 | clojure.lang.Keyword 1612 | (data->pb [this] 1613 | (case this 1614 | :driver-not-started Protos$Status/DRIVER_NOT_STARTED 1615 | :driver-running Protos$Status/DRIVER_RUNNING 1616 | :driver-aborted Protos$Status/DRIVER_ABORTED 1617 | :driver-stopped Protos$Status/DRIVER_STOPPED 1618 | :operation-launch Protos$Offer$Operation$Type/LAUNCH 1619 | :operation-reserve Protos$Offer$Operation$Type/RESERVE 1620 | :operation-unreserve Protos$Offer$Operation$Type/UNRESERVE 1621 | :operation-create Protos$Offer$Operation$Type/CREATE 1622 | :operation-destroy Protos$Offer$Operation$Type/DESTROY 1623 | :task-staging Protos$TaskState/TASK_STAGING 1624 | :task-starting Protos$TaskState/TASK_STARTING 1625 | :task-running Protos$TaskState/TASK_RUNNING 1626 | :task-finished Protos$TaskState/TASK_FINISHED 1627 | :task-failed Protos$TaskState/TASK_FAILED 1628 | :task-killed Protos$TaskState/TASK_KILLED 1629 | :task-lost Protos$TaskState/TASK_LOST 1630 | :task-error Protos$TaskState/TASK_ERROR 1631 | :value-scalar Protos$Value$Type/SCALAR 1632 | :value-ranges Protos$Value$Type/RANGES 1633 | :value-set Protos$Value$Type/SET 1634 | :value-text Protos$Value$Type/TEXT 1635 | :volume-rw Protos$Volume$Mode/RW 1636 | :volume-ro Protos$Volume$Mode/RO 1637 | :docker-network-host Protos$ContainerInfo$DockerInfo$Network/HOST 1638 | :docker-network-bridge Protos$ContainerInfo$DockerInfo$Network/BRIDGE 1639 | :docker-network-none Protos$ContainerInfo$DockerInfo$Network/NONE 1640 | :container-type-docker Protos$ContainerInfo$Type/DOCKER 1641 | :container-type-mesos Protos$ContainerInfo$Type/MESOS 1642 | :machine-mode-up Protos$MachineInfo$Mode/UP 1643 | :machine-mode-draining Protos$MachineInfo$Mode/DRAINING 1644 | :machine-mode-down Protos$MachineInfo$Mode/DOWN 1645 | :source-master Protos$TaskStatus$Source/SOURCE_MASTER 1646 | :source-slave Protos$TaskStatus$Source/SOURCE_SLAVE 1647 | :source-executor Protos$TaskStatus$Source/SOURCE_EXECUTOR 1648 | 1649 | ;; These are too wide and mess up indenting! 1650 | :framework-capability-unknown 1651 | Protos$FrameworkInfo$Capability$Type/UNKNOWN 1652 | :framework-capability-revocable-resources 1653 | Protos$FrameworkInfo$Capability$Type/REVOCABLE_RESOURCES 1654 | :framework-capability-task-killing-state 1655 | Protos$FrameworkInfo$Capability$Type/TASK_KILLING_STATE 1656 | :framework-capability-gpu-resources 1657 | Protos$FrameworkInfo$Capability$Type/GPU_RESOURCES 1658 | 1659 | :discovery-visibility-framework 1660 | Protos$DiscoveryInfo$Visibility/FRAMEWORK 1661 | :discovery-visibility-cluster 1662 | Protos$DiscoveryInfo$Visibility/CLUSTER 1663 | :discovery-visibility-external 1664 | Protos$DiscoveryInfo$Visibility/EXTERNAL 1665 | 1666 | 1667 | :reason-command-executor-failed 1668 | Protos$TaskStatus$Reason/REASON_COMMAND_EXECUTOR_FAILED 1669 | :reason-container-launch-failed 1670 | Protos$TaskStatus$Reason/REASON_CONTAINER_LAUNCH_FAILED 1671 | :reason-container-limitation 1672 | Protos$TaskStatus$Reason/REASON_CONTAINER_LIMITATION 1673 | :reason-container-limitation-disk 1674 | Protos$TaskStatus$Reason/REASON_CONTAINER_LIMITATION_DISK 1675 | :reason-container-limitation-memory 1676 | Protos$TaskStatus$Reason/REASON_CONTAINER_LIMITATION_MEMORY 1677 | :reason-container-preempted 1678 | Protos$TaskStatus$Reason/REASON_CONTAINER_PREEMPTED 1679 | :reason-container-update-failed 1680 | Protos$TaskStatus$Reason/REASON_CONTAINER_UPDATE_FAILED 1681 | :reason-executor-registration-timeout 1682 | Protos$TaskStatus$Reason/REASON_EXECUTOR_REGISTRATION_TIMEOUT 1683 | :reason-executor-reregistration-timeout 1684 | Protos$TaskStatus$Reason/REASON_EXECUTOR_REREGISTRATION_TIMEOUT 1685 | :reason-executor-terminated 1686 | Protos$TaskStatus$Reason/REASON_EXECUTOR_TERMINATED 1687 | :reason-executor-unregistered 1688 | Protos$TaskStatus$Reason/REASON_EXECUTOR_UNREGISTERED 1689 | :reason-framework-removed 1690 | Protos$TaskStatus$Reason/REASON_FRAMEWORK_REMOVED 1691 | :reason-gc-error 1692 | Protos$TaskStatus$Reason/REASON_GC_ERROR 1693 | :reason-invalid-frameworkid 1694 | Protos$TaskStatus$Reason/REASON_INVALID_FRAMEWORKID 1695 | :reason-invalid-offers 1696 | Protos$TaskStatus$Reason/REASON_INVALID_OFFERS 1697 | :reason-master-disconnected 1698 | Protos$TaskStatus$Reason/REASON_MASTER_DISCONNECTED 1699 | :reason-reconciliation 1700 | Protos$TaskStatus$Reason/REASON_RECONCILIATION 1701 | :reason-resources-unknown 1702 | Protos$TaskStatus$Reason/REASON_RESOURCES_UNKNOWN 1703 | :reason-slave-disconnected 1704 | Protos$TaskStatus$Reason/REASON_SLAVE_DISCONNECTED 1705 | :reason-slave-removed 1706 | Protos$TaskStatus$Reason/REASON_SLAVE_REMOVED 1707 | :reason-slave-restarted 1708 | Protos$TaskStatus$Reason/REASON_SLAVE_RESTARTED 1709 | :reason-slave-unknown 1710 | Protos$TaskStatus$Reason/REASON_SLAVE_UNKNOWN 1711 | :reason-task-invalid 1712 | Protos$TaskStatus$Reason/REASON_TASK_INVALID 1713 | :reason-task-unauthorized 1714 | Protos$TaskStatus$Reason/REASON_TASK_UNAUTHORIZED 1715 | :reason-task-unknown 1716 | Protos$TaskStatus$Reason/REASON_TASK_UNKNOWN 1717 | 1718 | ;; default 1719 | nil))) 1720 | 1721 | ;; By default, yield the original payload. 1722 | 1723 | (defmethod pb->data :default 1724 | [this] 1725 | this) 1726 | 1727 | (defn ->pb 1728 | [map-type this] 1729 | (data->pb 1730 | (if (record? this) 1731 | this 1732 | (let [f (case map-type 1733 | :FrameworkID map->FrameworkID 1734 | :FrameworkCapability map->FrameworkCapability 1735 | :OfferID map->OfferID 1736 | :SlaveID map->SlaveID 1737 | :TaskID map->TaskID 1738 | :ExecutorID map->ExecutorID 1739 | :ContainerID map->ContainerID 1740 | :FrameworkInfo map->FrameworkInfo 1741 | :HealthCheckHTTP map->HealthCheckHTTP 1742 | :HealthCheck map->HealthCheck 1743 | :URI map->URI 1744 | :CommandInfo map->CommandInfo 1745 | :ExecutorInfo map->ExecutorInfo 1746 | :MasterInfo map->MasterInfo 1747 | :SlaveInfo map->SlaveInfo 1748 | :ValueRange map->ValueRange 1749 | :ValueRanges map->ValueRanges 1750 | :Value map->Value 1751 | :Attribute map->Attribute 1752 | :Resource map->Resource 1753 | :Labels map->Labels 1754 | :Label map->Label 1755 | :Request map->Request 1756 | :Offer map->Offer 1757 | :Operation map->Operation 1758 | :TaskInfo map->TaskInfo 1759 | :TaskStatus map->TaskStatus 1760 | :Filters map->Filters 1761 | :EnvironmentVariable map->EnvironmentVariable 1762 | :Environment map->Environment 1763 | :Parameter map->Parameter 1764 | :Parameters map->Parameter 1765 | :Credential map->Credential 1766 | :Credentials map->Credentials 1767 | :RateLimit map->RateLimit 1768 | :RateLimits map->RateLimits 1769 | :Volume map->Volume 1770 | :PortMapping map->PortMapping 1771 | :DockerInfo map->DockerInfo 1772 | :ContainerInfo map->ContainerInfo)] 1773 | (f this))))) 1774 | -------------------------------------------------------------------------------- /src/mesomatic/utils.clj: -------------------------------------------------------------------------------- 1 | (ns mesomatic.utils 2 | "Some convenience/utility functions used in the rest of Mesomatic.") 3 | 4 | (defmacro case-enum 5 | "Like `case`, but explicitly dispatch on Java enum ordinals." 6 | [e & clauses] 7 | (letfn [(enum-ordinal [e] `(let [^Enum e# ~e] (.ordinal e#)))] 8 | `(case ~(enum-ordinal e) 9 | ~@(concat 10 | (mapcat (fn [[test result]] 11 | [(eval (enum-ordinal test)) result]) 12 | (partition 2 clauses)) 13 | (when (odd? (count clauses)) 14 | (list (last clauses))))))) 15 | -------------------------------------------------------------------------------- /test/mesomatic/allocator_test.clj: -------------------------------------------------------------------------------- 1 | (ns mesomatic.allocator-test 2 | (:require [clojure.test :refer :all] 3 | [mesomatic.allocator :refer :all])) 4 | 5 | (deftest allocator-test 6 | (let [offers [{:id {:value "20150506-090221-16777343-5050-15248-O36"}, 7 | :framework-id {:value "20150506-090221-16777343-5050-15248-0015"}, 8 | :slave-id {:value "20150505-120153-16777343-5050-32275-S0"}, 9 | :hostname "localhost", 10 | :resources [{:name "mem", :type :value-scalar, :scalar 6918.0} 11 | {:name "cpus", :type :value-scalar, :scalar 8.0} 12 | {:name "disk", :type :value-scalar, :scalar 24989.0} 13 | {:name "ports", 14 | :type :value-ranges, 15 | :ranges [{:begin 31000, :end 32000}]}], 16 | :attributes [], 17 | :executor-ids []}, 18 | 19 | {:id {:value "a4ba1178-6d54-47ca-abec-b7455f372b36"}, 20 | :framework-id {:value "20150506-090221-16777343-5050-15248-0015"}, 21 | :slave-id {:value "8ce8c437-e756-4e17-8711-b3082e3aaec0"}, 22 | :hostname "localhost", 23 | :resources [{:name "mem", :type :value-scalar, :scalar 6918.0} 24 | {:name "cpus", :type :value-scalar, :scalar 8.0} 25 | {:name "disk", :type :value-scalar, :scalar 24989.0} 26 | {:name "ports", 27 | :type :value-ranges, 28 | :ranges [{:begin 31000, :end 32000}]}], 29 | :attributes [], 30 | :executor-ids []}] 31 | 32 | task-info {:name "bundesrat-daemon-bar", 33 | :task-id [{:value "0d6d34f6-0dba-4614-abb9-8e5779aafb7b"} 34 | {:value "65a5e645-b92f-4bcb-9421-1c08eaad1d8f"}], 35 | :resources [{:type :value-scalar, :name "mem", :scalar 512.0} 36 | {:type :value-scalar, :name "cpus", :scalar 0.5} 37 | {:type :value-ranges, :name "ports", :ranges [{:begin 0, :end 0}]}], 38 | :container {:type :container-type-docker, 39 | :docker {:image "dockerfile/nginx", 40 | :port-mappings [{:container-port 80, :protocol nil}], 41 | :network :docker-network-bridge}}, 42 | :command {:shell false}, 43 | :instances 2 44 | :colocation 1}] 45 | 46 | (testing "cannot satisfy workload" 47 | (is (= nil (allocate-naively (take 1 offers) [task-info]))) 48 | (is (= nil (allocate-naively offers (repeat 100 task-info))))) 49 | 50 | (testing "successful allocation" 51 | (is (= 2 (count (allocate-naively offers [task-info])))) 52 | (is (= 2 (->> (allocate-naively offers [task-info]) 53 | (group-by :offer-id) 54 | keys 55 | count))) 56 | (is (= 1 (->> (allocate-naively offers [(assoc task-info :colocation 2)]) 57 | (group-by :offer-id) 58 | keys 59 | count)))))) 60 | -------------------------------------------------------------------------------- /ux-resources/images/mesomatic-logo-x1000.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/clojusc/mesomatic/56a291b94273e78bc4d2ef44e08794b2fb2d1427/ux-resources/images/mesomatic-logo-x1000.png -------------------------------------------------------------------------------- /ux-resources/images/mesomatic-logo-x250.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/clojusc/mesomatic/56a291b94273e78bc4d2ef44e08794b2fb2d1427/ux-resources/images/mesomatic-logo-x250.png -------------------------------------------------------------------------------- /ux-resources/images/mesomatic-logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/clojusc/mesomatic/56a291b94273e78bc4d2ef44e08794b2fb2d1427/ux-resources/images/mesomatic-logo.png --------------------------------------------------------------------------------