├── doc ├── hype.png └── hype-volumes.png ├── hype-gcs ├── src │ ├── test │ │ ├── resources │ │ │ ├── logback-test.xml │ │ │ ├── malformed-manifest.txt │ │ │ ├── example-manifest.txt │ │ │ ├── empty-lines-manifest.txt │ │ │ └── multi-lambda-manifest.txt │ │ └── java │ │ │ └── com │ │ │ └── spotify │ │ │ └── hype │ │ │ └── gcs │ │ │ ├── ManifestUtilTest.java │ │ │ ├── ManifestLoaderTest.java │ │ │ └── StagingUtilTest.java │ └── main │ │ └── java │ │ └── com │ │ └── spotify │ │ └── hype │ │ └── gcs │ │ ├── RunManifest.java │ │ ├── ManifestLoader.java │ │ ├── ManifestUtil.java │ │ └── ZipFiles.java └── pom.xml ├── hype-caplet ├── src │ └── main │ │ ├── resources │ │ ├── logback-test.xml │ │ └── version.properties │ │ └── java │ │ └── Hypelet.java └── pom.xml ├── hype-common ├── src │ ├── test │ │ ├── resources │ │ │ └── logback-test.xml │ │ └── java │ │ │ └── com │ │ │ └── spotify │ │ │ └── hype │ │ │ └── util │ │ │ └── SerializationUtilTest.java │ └── main │ │ └── java │ │ └── com │ │ └── spotify │ │ └── hype │ │ ├── util │ │ ├── Fn.java │ │ ├── Util.java │ │ └── SerializationUtil.java │ │ └── FluentBackoff.java └── pom.xml ├── hype-submitter ├── src │ ├── test │ │ ├── resources │ │ │ ├── logback-test.xml │ │ │ ├── with-image.yaml │ │ │ └── minimal-pod.yaml │ │ └── java │ │ │ └── com │ │ │ └── spotify │ │ │ └── hype │ │ │ ├── model │ │ │ ├── RunEnvironmentTest.java │ │ │ ├── ResourceRequestTests.java │ │ │ └── ResourcesTest.java │ │ │ └── runner │ │ │ ├── VolumeRepositoryTest.java │ │ │ └── KubernetesDockerRunnerTest.java │ └── main │ │ └── java │ │ └── com │ │ └── spotify │ │ └── hype │ │ ├── runner │ │ ├── InvalidExecutionException.java │ │ ├── RunSpec.java │ │ ├── DockerRunner.java │ │ ├── VolumeRepository.java │ │ ├── LocalDockerRunner.java │ │ └── KubernetesDockerRunner.java │ │ ├── model │ │ ├── Secret.java │ │ ├── ContainerEngineCluster.java │ │ ├── StagedContinuation.java │ │ ├── VolumeMount.java │ │ ├── ResourceRequest.java │ │ ├── DockerCluster.java │ │ ├── RunEnvironment.java │ │ └── VolumeRequest.java │ │ ├── ClasspathInspector.java │ │ ├── LocalClasspathInspector.java │ │ └── Submitter.java └── pom.xml ├── hype-testing ├── src │ ├── test │ │ ├── resources │ │ │ ├── logback-test.xml │ │ │ └── test-pod.yaml │ │ └── scala │ │ │ └── com │ │ │ └── spotify │ │ │ └── hype │ │ │ ├── HFnTest.scala │ │ │ ├── LocalSubmitterTest.scala │ │ │ ├── magic │ │ │ └── SubmitterOpsTest.scala │ │ │ └── ScalaSerializationTest.scala │ └── main │ │ ├── resources │ │ └── version.properties │ │ └── java │ │ └── com │ │ └── spotify │ │ └── hype │ │ └── VersionUtil.java └── pom.xml ├── hype-docker ├── Dockerfile └── pom.xml ├── .gitignore ├── circle.yml ├── logback-test.xml ├── hype-submitter_2.11 ├── src │ └── main │ │ └── scala │ │ └── com │ │ └── spotify │ │ └── hype │ │ ├── GkeSubmitter.scala │ │ ├── LocalSubmitter.scala │ │ ├── HypeSubmitter.scala │ │ ├── magic │ │ └── package.scala │ │ ├── HFn.scala │ │ ├── package.scala │ │ └── Example.scala └── pom.xml ├── hype-submitter_2.12 ├── src │ └── main │ │ └── scala │ │ └── com │ │ └── spotify │ │ └── hype │ │ └── package.scala └── pom.xml ├── bin └── install-runner ├── hype-run ├── src │ └── main │ │ └── java │ │ └── com │ │ └── spotify │ │ └── hype │ │ └── stub │ │ ├── Noop.java │ │ └── ContinuationEntryPoint.java └── pom.xml ├── pom.xml ├── LICENSE └── README.md /doc/hype.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spotify/hype/HEAD/doc/hype.png -------------------------------------------------------------------------------- /hype-gcs/src/test/resources/logback-test.xml: -------------------------------------------------------------------------------- 1 | ../../../../logback-test.xml -------------------------------------------------------------------------------- /hype-caplet/src/main/resources/logback-test.xml: -------------------------------------------------------------------------------- 1 | ../../../../logback-test.xml -------------------------------------------------------------------------------- /hype-common/src/test/resources/logback-test.xml: -------------------------------------------------------------------------------- 1 | ../../../../logback-test.xml -------------------------------------------------------------------------------- /hype-submitter/src/test/resources/logback-test.xml: -------------------------------------------------------------------------------- 1 | ../../../../logback-test.xml -------------------------------------------------------------------------------- /hype-testing/src/test/resources/logback-test.xml: -------------------------------------------------------------------------------- 1 | ../../../../logback-test.xml -------------------------------------------------------------------------------- /doc/hype-volumes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spotify/hype/HEAD/doc/hype-volumes.png -------------------------------------------------------------------------------- /hype-caplet/src/main/resources/version.properties: -------------------------------------------------------------------------------- 1 | hypelet.version=${project.version} 2 | -------------------------------------------------------------------------------- /hype-testing/src/main/resources/version.properties: -------------------------------------------------------------------------------- 1 | hypelet.version=${project.version} 2 | -------------------------------------------------------------------------------- /hype-gcs/src/test/resources/malformed-manifest.txt: -------------------------------------------------------------------------------- 1 | l continuation-ce89ba3b.bin 2 | c lib1.jar 3 | clib2.jar 4 | -------------------------------------------------------------------------------- /hype-gcs/src/test/resources/example-manifest.txt: -------------------------------------------------------------------------------- 1 | l continuation-ce89ba3b.bin 2 | c lib1.jar 3 | c lib2.jar 4 | c lib3.jar 5 | f other-file.txt 6 | -------------------------------------------------------------------------------- /hype-gcs/src/test/resources/empty-lines-manifest.txt: -------------------------------------------------------------------------------- 1 | l continuation-ce89ba3b.bin 2 | c lib1.jar 3 | 4 | c lib2.jar 5 | c lib3.jar 6 | f other-file.txt 7 | 8 | -------------------------------------------------------------------------------- /hype-gcs/src/test/resources/multi-lambda-manifest.txt: -------------------------------------------------------------------------------- 1 | l continuation-ce89ba3b.bin 2 | c lib1.jar 3 | c lib2.jar 4 | l continuation-other.bin 5 | c lib3.jar 6 | f other-file.txt 7 | -------------------------------------------------------------------------------- /hype-testing/src/test/resources/test-pod.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | 4 | spec: 5 | restartPolicy: Never 6 | 7 | containers: 8 | - name: hype-run 9 | imagePullPolicy: Always 10 | 11 | env: 12 | - name: HYPE_ENV 13 | value: testing 14 | -------------------------------------------------------------------------------- /hype-submitter/src/test/resources/with-image.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | 4 | spec: 5 | 6 | containers: 7 | - name: hype-run 8 | image: spotify/hype:latest 9 | imagePullPolicy: Always 10 | 11 | env: 12 | - name: EXAMPLE 13 | value: my-env-value -------------------------------------------------------------------------------- /hype-docker/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM openjdk:8-jre 2 | MAINTAINER Rouzbeh Delavari 3 | 4 | ENV GOOGLE_APPLICATION_CREDENTIALS /etc/gcloud/key.json 5 | ENV HYPE_ENV testing 6 | 7 | # Install hype-run command 8 | RUN /bin/sh -c "$(curl -fsSL https://goo.gl/kSogpF)" 9 | ENTRYPOINT ["hype-run"] 10 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ## Artifacts 2 | *.class 3 | 4 | ## Package Files 5 | *.jar 6 | *.war 7 | *.ear 8 | target 9 | 10 | ## IntelliJ 11 | .idea 12 | *.iml 13 | 14 | ## Logs 15 | *.log 16 | 17 | ## Temp 18 | *~ 19 | .#* 20 | 21 | dependency-reduced-pom.xml 22 | .helios/helios-job-ids 23 | created_job_ids 24 | .tmp 25 | -------------------------------------------------------------------------------- /hype-submitter/src/test/resources/minimal-pod.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Pod 3 | 4 | spec: 5 | restartPolicy: Never 6 | 7 | containers: 8 | - name: hype-run 9 | imagePullPolicy: Always 10 | 11 | env: 12 | - name: EXAMPLE 13 | value: my-env-value 14 | 15 | resources: 16 | requests: 17 | cpu: 100m 18 | limits: 19 | memory: 1Gi 20 | -------------------------------------------------------------------------------- /circle.yml: -------------------------------------------------------------------------------- 1 | machine: 2 | java: 3 | version: oraclejdk8 4 | services: 5 | - docker 6 | 7 | dependencies: 8 | cache_directories: 9 | - "~/.m2" 10 | # compile, since dependency:resolve fails for multi-module builds 11 | override: 12 | - mvn -version 13 | 14 | test: 15 | override: 16 | - mvn test -Ddockerfile.build.noCache=true 17 | post: 18 | - bash <(curl -s https://codecov.io/bash) 19 | -------------------------------------------------------------------------------- /logback-test.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | STDOUT 5 | 6 | 7 | %gray(%d{HH:mm:ss.SSS}) %highlight(| %-5level| %-20logger{0}) %green(|>) %msg%n 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | -------------------------------------------------------------------------------- /hype-submitter_2.11/src/main/scala/com/spotify/hype/GkeSubmitter.scala: -------------------------------------------------------------------------------- 1 | package com.spotify.hype 2 | 3 | 4 | case class GkeSubmitter(project: String, 5 | zone: String, 6 | cluster: String, 7 | staging: String 8 | ) extends HypeSubmitter { 9 | private val s = setupShutdown(Submitter.create(staging, 10 | model.ContainerEngineCluster.containerEngineCluster(project, zone, cluster))) 11 | 12 | override protected def submitter: Submitter = s 13 | } 14 | -------------------------------------------------------------------------------- /hype-submitter_2.11/src/main/scala/com/spotify/hype/LocalSubmitter.scala: -------------------------------------------------------------------------------- 1 | package com.spotify.hype 2 | 3 | import com.spotify.hype.model.DockerCluster 4 | 5 | case class LocalSubmitter(keepContainer: Boolean = false, 6 | keepTerminationLog: Boolean = false, 7 | keepVolumes: Boolean = false 8 | ) extends HypeSubmitter { 9 | override protected def submitter: Submitter = setupShutdown(Submitter.createLocal( 10 | DockerCluster.dockerCluster(keepContainer, keepTerminationLog, keepVolumes) 11 | )) 12 | } 13 | -------------------------------------------------------------------------------- /hype-submitter_2.11/src/main/scala/com/spotify/hype/HypeSubmitter.scala: -------------------------------------------------------------------------------- 1 | package com.spotify.hype 2 | 3 | import com.spotify.hype.model.RunEnvironment 4 | 5 | trait HypeSubmitter { 6 | 7 | protected def submitter: Submitter 8 | 9 | def submit[T](hfn: HFn[T], env: RunEnvironment): T = { 10 | submitter.runOnCluster(hfn.run, env, hfn.image) 11 | } 12 | 13 | protected def setupShutdown(submitter: Submitter): Submitter = { 14 | Runtime.getRuntime.addShutdownHook(new Thread(new Runnable { 15 | def run(): Unit = submitter.close() 16 | })) 17 | submitter 18 | } 19 | } 20 | -------------------------------------------------------------------------------- /hype-submitter_2.12/src/main/scala/com/spotify/hype/package.scala: -------------------------------------------------------------------------------- 1 | package com.spotify 2 | 3 | package object hype { 4 | 5 | object RunEnvironment { 6 | def apply(): model.RunEnvironment = model.RunEnvironment.environment() 7 | } 8 | 9 | object EnvironmentFromYaml { 10 | def apply(resourcePath: String): model.RunEnvironment = 11 | model.RunEnvironment.fromYaml(resourcePath) 12 | } 13 | 14 | object VolumeRequest { 15 | def apply(name: String, mountPath: String): model.VolumeRequest = 16 | model.VolumeRequest.volumeRequest(name, mountPath) 17 | } 18 | 19 | } 20 | -------------------------------------------------------------------------------- /hype-submitter_2.11/src/main/scala/com/spotify/hype/magic/package.scala: -------------------------------------------------------------------------------- 1 | package com.spotify.hype 2 | 3 | import com.spotify.hype.model.RunEnvironment 4 | 5 | import scala.language.implicitConversions 6 | 7 | /** 8 | * https://open.spotify.com/track/500h8jAdr7LvzzXlm1qxtK 9 | */ 10 | package object magic { 11 | 12 | implicit class HFnOps[T](val hfn: HFn[T]) extends AnyVal { 13 | def #!(implicit submitter: HypeSubmitter, env: RunEnvironment): T = submitter.submit(hfn, env) 14 | 15 | def #!(env: RunEnvironment)(implicit submitter: HypeSubmitter): T = #!(submitter, env) 16 | } 17 | 18 | implicit def toHfnOps[A](fn: () => A): HFnOps[A] = 19 | HFnOps(toHfn(fn)) 20 | } 21 | -------------------------------------------------------------------------------- /bin/install-runner: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | VERSION=${VERSION:-0.0.16} 4 | DEST=${DEST:-/usr/local/bin} 5 | JAR=${DEST}/hype-run.jar 6 | MAVEN_REPO=https://repo.maven.apache.org/maven2 7 | 8 | echo "=== HYPE RUN INSTALLER (v$VERSION) ===" 9 | set -ex 10 | 11 | # download runner capsule 12 | curl -#fL --output ${JAR} \ 13 | ${MAVEN_REPO}/com/spotify/hype-run/${VERSION}/hype-run-${VERSION}-capsule.jar 14 | 15 | # invoke once in noop mode to download hype caplet 16 | /usr/bin/java -Dcapsule.mode=noop -jar ${JAR} 17 | 18 | 19 | cat > ${DEST}/hype-run < T) = new HFn[T] { 21 | def run: T = f 22 | } 23 | 24 | def withImage[T](img: String)(f: => T) = new HFn[T] { 25 | def run: T = f 26 | 27 | override def image: String = img 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /hype-run/src/main/java/com/spotify/hype/stub/Noop.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-run 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.stub; 22 | 23 | public class Noop { 24 | public static void main(String[] args) { 25 | } 26 | } 27 | -------------------------------------------------------------------------------- /hype-common/src/main/java/com/spotify/hype/util/Fn.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.util; 22 | 23 | import java.io.Serializable; 24 | 25 | @FunctionalInterface 26 | public interface Fn extends Serializable { 27 | T run(); 28 | } 29 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/runner/InvalidExecutionException.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * Spotify Styx Scheduler Service 4 | * -- 5 | * Copyright (C) 2016 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.runner; 22 | 23 | /** 24 | * An exception that can indicate that an invalid execution request was sent to 25 | * {@link KubernetesDockerRunner}. 26 | */ 27 | public class InvalidExecutionException extends RuntimeException { 28 | 29 | public InvalidExecutionException(String message) { 30 | super(message); 31 | } 32 | } 33 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/model/Secret.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.model; 22 | 23 | import io.norberg.automatter.AutoMatter; 24 | 25 | @AutoMatter 26 | public interface Secret { 27 | 28 | String name(); 29 | String mountPath(); 30 | 31 | static Secret secret(String name, String mountPath) { 32 | return new SecretBuilder() 33 | .name(name) 34 | .mountPath(mountPath) 35 | .build(); 36 | } 37 | } 38 | -------------------------------------------------------------------------------- /hype-common/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | hype-root 6 | com.spotify 7 | 0.0.19-SNAPSHOT 8 | 9 | 10 | hype-common 11 | 12 | 13 | 14 | com.google.cloud 15 | google-cloud-storage 16 | 17 | 18 | 19 | com.esotericsoftware 20 | kryo-shaded 21 | 4.0.0 22 | 23 | 24 | 25 | org.slf4j 26 | slf4j-api 27 | 28 | 29 | 30 | junit 31 | junit 32 | test 33 | 34 | 35 | 36 | -------------------------------------------------------------------------------- /hype-docker/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | hype-root 6 | com.spotify 7 | 0.0.19-SNAPSHOT 8 | 9 | 10 | hype-docker 11 | pom 12 | 13 | 14 | 15 | 16 | com.spotify 17 | dockerfile-maven-plugin 18 | 1.3.6 19 | 20 | 21 | build 22 | test-compile 23 | 24 | build 25 | 26 | 27 | 28 | 29 | spotify-hype-testing 30 | ${project.version} 31 | 32 | 33 | 34 | 35 | 36 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/ClasspathInspector.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype; 22 | 23 | import java.nio.file.Path; 24 | import java.util.List; 25 | 26 | public interface ClasspathInspector { 27 | List classpathJars(); 28 | 29 | static ClasspathInspector forClass(Class cls) { 30 | return new LocalClasspathInspector(cls); 31 | } 32 | 33 | static ClasspathInspector forLoader(ClassLoader classLoader) { 34 | return new LocalClasspathInspector(classLoader); 35 | } 36 | } 37 | -------------------------------------------------------------------------------- /hype-submitter_2.11/src/main/scala/com/spotify/hype/package.scala: -------------------------------------------------------------------------------- 1 | package com.spotify 2 | 3 | import scala.language.implicitConversions 4 | 5 | package object hype { 6 | 7 | object RunEnvironment { 8 | def apply(): model.RunEnvironment = model.RunEnvironment.environment() 9 | } 10 | 11 | object RunEnvironmentFromYaml { 12 | def apply(resourcePath: String): model.RunEnvironment = 13 | model.RunEnvironment.fromYaml(resourcePath) 14 | } 15 | 16 | object TransientVolume { 17 | def apply(storageClass: String, size: String): model.VolumeRequest = 18 | model.VolumeRequest.volumeRequest(storageClass, size) 19 | } 20 | 21 | object PersistentVolume { 22 | def apply(name: String, storageClass: String, size: String): model.VolumeRequest = 23 | model.VolumeRequest.createIfNotExists(name, storageClass, size) 24 | } 25 | 26 | implicit def fnToHfn[T](fn: util.Fn[T]): HFn[T] = HFn(fn.run()) 27 | 28 | implicit def toHfn[A](fn: () => A): HFn[A] = HFn(fn()) 29 | 30 | implicit def toFn[A](fn: () => A): util.Fn[A] = new util.Fn[A] { 31 | override def run(): A = fn() 32 | } 33 | 34 | implicit def toFnByName[A](fn: => A): util.Fn[A] = new util.Fn[A] { 35 | override def run(): A = fn 36 | } 37 | } 38 | -------------------------------------------------------------------------------- /hype-testing/src/main/java/com/spotify/hype/VersionUtil.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-testing 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype; 22 | 23 | import java.io.IOException; 24 | import java.util.Properties; 25 | 26 | public class VersionUtil { 27 | 28 | public static String getVersion() { 29 | Properties props = new Properties(); 30 | try { 31 | props.load(VersionUtil.class.getResourceAsStream("/version.properties")); 32 | } catch (IOException e) { 33 | throw new RuntimeException(e); 34 | } 35 | return props.getProperty("hypelet.version"); 36 | } 37 | 38 | } 39 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/model/ContainerEngineCluster.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.model; 22 | 23 | import io.norberg.automatter.AutoMatter; 24 | 25 | @AutoMatter 26 | public interface ContainerEngineCluster { 27 | 28 | String project(); 29 | String zone(); 30 | String cluster(); 31 | 32 | static ContainerEngineCluster containerEngineCluster(String project, String zone, String cluster) { 33 | return new ContainerEngineClusterBuilder() 34 | .project(project) 35 | .zone(zone) 36 | .cluster(cluster) 37 | .build(); 38 | } 39 | } 40 | -------------------------------------------------------------------------------- /hype-submitter_2.11/src/main/scala/com/spotify/hype/Example.scala: -------------------------------------------------------------------------------- 1 | package com.spotify.hype 2 | 3 | import com.spotify.hype.model.ResourceRequest.{CPU, MEMORY} 4 | import com.spotify.hype.model.VolumeRequest._ 5 | 6 | import scala.collection.JavaConverters._ 7 | 8 | private object Example { 9 | 10 | def main(args: Array[String]): Unit = { 11 | 12 | val env = RunEnvironment() 13 | .withSecret("gcp-key", "/etc/gcloud") 14 | .withRequest(CPU.of("200m")) 15 | .withRequest(MEMORY.of("256Mi")) 16 | 17 | val record = ("hello", 42) 18 | 19 | val fn = HFn { 20 | val cwd = System.getProperty("user.dir") 21 | println(s"cwd = $cwd") 22 | println(s"record = $record") 23 | System.getenv().asScala.map { case (foo, bar) => s"$foo=$bar" } 24 | } 25 | 26 | // create a volume request from a predefined storage class with name 'slow' 27 | val slow10Gi = volumeRequest("slow", "10Gi") 28 | 29 | val submitter = GkeSubmitter("datawhere-test", "us-east1-d", "hype-test", "gs://hype-test/staging") 30 | 31 | val ret = submitter.submit(fn, env.withMount(slow10Gi.mountReadWrite("/usr/share/volume"))) 32 | println(ret) 33 | 34 | for (i <- (0 to 10).par) { 35 | submitter.submit(fn, env.withMount(slow10Gi.mountReadOnly("/usr/share/volume"))) 36 | } 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /hype-common/src/main/java/com/spotify/hype/util/Util.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.util; 22 | 23 | public final class Util { 24 | 25 | private static final String ALPHA_NUMERIC_STRING = "abcdefghijklmnopqrstuvwxyz0123456789"; 26 | 27 | private Util() { 28 | } 29 | 30 | public static String randomAlphaNumeric(int count) { 31 | StringBuilder builder = new StringBuilder(); 32 | while (count-- != 0) { 33 | int character = (int)(Math.random() * ALPHA_NUMERIC_STRING.length()); 34 | builder.append(ALPHA_NUMERIC_STRING.charAt(character)); 35 | } 36 | return builder.toString(); 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/model/StagedContinuation.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.model; 22 | 23 | import com.spotify.hype.gcs.RunManifest; 24 | import io.norberg.automatter.AutoMatter; 25 | import java.nio.file.Path; 26 | 27 | @AutoMatter 28 | public interface StagedContinuation { 29 | 30 | Path manifestPath(); 31 | RunManifest manifest(); 32 | 33 | static StagedContinuation stagedContinuation(Path manifestPath, RunManifest manifest) { 34 | return new StagedContinuationBuilder() 35 | .manifestPath(manifestPath) 36 | .manifest(manifest) 37 | .build(); 38 | } 39 | } 40 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/model/VolumeMount.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.model; 22 | 23 | import io.norberg.automatter.AutoMatter; 24 | 25 | /** 26 | * A binding for a {@link VolumeRequest} to be used at a specified mount path. 27 | */ 28 | @AutoMatter 29 | public interface VolumeMount { 30 | 31 | VolumeRequest volumeRequest(); 32 | String mountPath(); 33 | boolean readOnly(); 34 | 35 | static VolumeMount volumeMount( 36 | VolumeRequest volumeRequest, String mountPath, boolean readOnly) { 37 | return new VolumeMountBuilder() 38 | .volumeRequest(volumeRequest) 39 | .mountPath(mountPath) 40 | .readOnly(readOnly) 41 | .build(); 42 | } 43 | } 44 | -------------------------------------------------------------------------------- /hype-testing/src/test/scala/com/spotify/hype/HFnTest.scala: -------------------------------------------------------------------------------- 1 | package com.spotify.hype 2 | 3 | import org.scalatest.{FlatSpec, Matchers} 4 | 5 | object HFnTest { 6 | val testImage = s"spotify-hype-testing:${VersionUtil.getVersion}" 7 | } 8 | 9 | class HFnTest extends FlatSpec with Matchers { 10 | 11 | "HypeModule" should "capture method arguments" in { 12 | def fn(arg: String) = HFn[String] { 13 | arg + " world" 14 | } 15 | 16 | roundtrip(fn("hello")) shouldBe "hello world" 17 | } 18 | 19 | it should "not evaluate on construction" in { 20 | var called = false 21 | def fn(arg: String) = HFn[String] { 22 | called = true 23 | arg + " world" 24 | } 25 | 26 | val result = roundtrip(fn("hello")) 27 | 28 | called shouldBe false 29 | result shouldBe "hello world" 30 | called shouldBe false 31 | } 32 | 33 | it should "convert function0 to module" in { 34 | var called = false 35 | val fn = () => { 36 | called = true 37 | "hello world" 38 | } 39 | 40 | called shouldBe false 41 | roundtrip(fn) shouldBe "hello world" 42 | called shouldBe false 43 | } 44 | 45 | /** 46 | * Do a roundtrip through the serializer before running the module function. This should 47 | * ensure that the syntax used in the tests will survive the closure cleaner. 48 | */ 49 | def roundtrip[T](module: HFn[T]): T = { 50 | Wrapper.roundtrip(module.run).run() 51 | } 52 | } 53 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/runner/RunSpec.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.runner; 22 | 23 | import com.spotify.hype.model.RunEnvironment; 24 | import com.spotify.hype.model.StagedContinuation; 25 | import io.norberg.automatter.AutoMatter; 26 | 27 | @AutoMatter 28 | public interface RunSpec { 29 | 30 | RunEnvironment runEnvironment(); 31 | StagedContinuation stagedContinuation(); 32 | String image(); 33 | 34 | static RunSpec runSpec( 35 | RunEnvironment runEnvironment, 36 | StagedContinuation stagedContinuation, 37 | String image) { 38 | return new RunSpecBuilder() 39 | .runEnvironment(runEnvironment) 40 | .stagedContinuation(stagedContinuation) 41 | .image(image) 42 | .build(); 43 | } 44 | } 45 | -------------------------------------------------------------------------------- /hype-submitter/src/test/java/com/spotify/hype/model/RunEnvironmentTest.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.model; 22 | 23 | import static com.spotify.hype.model.ResourceRequest.CPU; 24 | import static org.hamcrest.Matchers.equalTo; 25 | import static org.hamcrest.Matchers.hasEntry; 26 | import static org.junit.Assert.assertThat; 27 | 28 | import org.junit.Test; 29 | 30 | public class RunEnvironmentTest { 31 | 32 | @Test 33 | public void overrideResourceRequestsForSame() throws Exception { 34 | RunEnvironment env = RunEnvironment.environment() 35 | .withRequest(CPU.of("1")) 36 | .withRequest(CPU.of("7")); 37 | 38 | assertThat(env.resourceRequests().size(), equalTo(1)); 39 | assertThat(env.resourceRequests(), hasEntry("cpu", "7")); 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/model/ResourceRequest.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.model; 22 | 23 | import io.norberg.automatter.AutoMatter; 24 | 25 | /** 26 | * A Kubernetes resource request 27 | */ 28 | @AutoMatter 29 | public interface ResourceRequest { 30 | 31 | ResourceRequestCreator CPU = forResource("cpu"); 32 | ResourceRequestCreator MEMORY = forResource("memory"); 33 | 34 | String resource(); 35 | String amount(); 36 | 37 | static ResourceRequest request(String resource, String amount) { 38 | return new ResourceRequestBuilder() 39 | .resource(resource) 40 | .amount(amount) 41 | .build(); 42 | } 43 | 44 | interface ResourceRequestCreator { 45 | ResourceRequest of(String amount); 46 | } 47 | 48 | static ResourceRequestCreator forResource(String resource) { 49 | return (amount) -> ResourceRequest.request(resource, amount); 50 | } 51 | } 52 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/model/DockerCluster.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.model; 22 | 23 | import io.norberg.automatter.AutoMatter; 24 | 25 | @AutoMatter 26 | public interface DockerCluster { 27 | boolean keepContainer(); 28 | boolean keepTerminationLog(); 29 | boolean keepVolumes(); 30 | 31 | static DockerCluster dockerCluster(final boolean keepContainer, 32 | final boolean keepTerminationLog, 33 | final boolean keepVolumes) { 34 | return new DockerClusterBuilder() 35 | .keepContainer(keepContainer) 36 | .keepTerminationLog(keepTerminationLog) 37 | .keepVolumes(keepVolumes) 38 | .build(); 39 | } 40 | 41 | static DockerCluster dockerCluster() { 42 | return new DockerClusterBuilder() 43 | .keepContainer(false) 44 | .keepTerminationLog(false) 45 | .keepVolumes(false) 46 | .build(); 47 | } 48 | 49 | } 50 | -------------------------------------------------------------------------------- /hype-submitter_2.11/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | hype-root 6 | com.spotify 7 | 0.0.19-SNAPSHOT 8 | 9 | 10 | hype_2.11 11 | 12 | 13 | 2.11 14 | 2.11.9 15 | 16 | 17 | 18 | 19 | com.spotify 20 | hype-submitter 21 | 0.0.19-SNAPSHOT 22 | 23 | 24 | 25 | org.scala-lang 26 | scala-library 27 | ${scala.version} 28 | 29 | 30 | 31 | 32 | 33 | 34 | net.alchim31.maven 35 | scala-maven-plugin 36 | 3.2.1 37 | 38 | 39 | 40 | compile 41 | testCompile 42 | 43 | 44 | 45 | 46 | ${scala.version} 47 | true 48 | 49 | 50 | 51 | 52 | 53 | -------------------------------------------------------------------------------- /hype-submitter_2.12/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | hype-root 6 | com.spotify 7 | 0.0.19-SNAPSHOT 8 | 9 | 10 | hype_2.12 11 | 12 | 13 | 2.12 14 | 2.12.1 15 | 16 | 17 | 18 | 19 | com.spotify 20 | hype-submitter 21 | 0.0.19-SNAPSHOT 22 | 23 | 24 | 25 | org.scala-lang 26 | scala-library 27 | ${scala.version} 28 | 29 | 30 | 31 | 32 | 33 | 34 | net.alchim31.maven 35 | scala-maven-plugin 36 | 3.2.1 37 | 38 | 39 | 40 | compile 41 | testCompile 42 | 43 | 44 | 45 | 46 | ${scala.version} 47 | true 48 | 49 | 50 | 51 | 52 | 53 | -------------------------------------------------------------------------------- /hype-submitter/src/test/java/com/spotify/hype/model/ResourceRequestTests.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.model; 22 | 23 | import static org.hamcrest.Matchers.equalTo; 24 | import static org.junit.Assert.assertThat; 25 | 26 | import org.junit.Test; 27 | 28 | public abstract class ResourceRequestTests { 29 | 30 | private final ResourceRequest.ResourceRequestCreator creator; 31 | 32 | private final String expectedResourceName; 33 | 34 | ResourceRequestTests( 35 | ResourceRequest.ResourceRequestCreator creator, 36 | String expectedResourceName) { 37 | this.creator = creator; 38 | this.expectedResourceName = expectedResourceName; 39 | } 40 | 41 | @Test 42 | public void resourceName() throws Exception { 43 | ResourceRequest request = creator.of("147"); 44 | assertThat(request.resource(), equalTo(expectedResourceName)); 45 | } 46 | 47 | @Test 48 | public void scalarValue() throws Exception { 49 | ResourceRequest request = creator.of("147"); 50 | assertThat(request.amount(), equalTo("147")); 51 | } 52 | } 53 | -------------------------------------------------------------------------------- /hype-gcs/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | hype-root 6 | com.spotify 7 | 0.0.19-SNAPSHOT 8 | 9 | 10 | hype-gcs 11 | 12 | 13 | 14 | com.spotify 15 | hype-common 16 | 0.0.19-SNAPSHOT 17 | 18 | 19 | 20 | com.google.cloud 21 | google-cloud-storage 22 | 23 | 24 | com.google.cloud 25 | google-cloud-nio 26 | 27 | 28 | com.google.cloud.bigdataoss 29 | util 30 | 1.6.0 31 | 32 | 33 | 34 | io.norberg 35 | auto-matter 36 | provided 37 | 38 | 39 | org.slf4j 40 | slf4j-api 41 | 42 | 43 | 44 | junit 45 | junit 46 | test 47 | 48 | 49 | org.hamcrest 50 | hamcrest-all 51 | test 52 | 53 | 54 | 55 | -------------------------------------------------------------------------------- /hype-testing/src/test/scala/com/spotify/hype/LocalSubmitterTest.scala: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017 Spotify AB. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); 5 | * you may not use this file except in compliance with the License. 6 | * You may obtain a copy of the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, 11 | * software distributed under the License is distributed on an 12 | * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 13 | * KIND, either express or implied. See the License for the 14 | * specific language governing permissions and limitations 15 | * under the License. 16 | */ 17 | package com.spotify.hype 18 | 19 | import org.scalatest.{FlatSpec, Matchers} 20 | 21 | import scala.reflect.io.File 22 | 23 | class LocalSubmitterTest extends FlatSpec with Matchers { 24 | 25 | private val submitter = LocalSubmitter() 26 | private val env = RunEnvironment() 27 | 28 | "TestCluster" should "work" in { 29 | val fooHFn = HFn { 30 | "foobar" 31 | } 32 | 33 | submitter.submit(fooHFn, env) shouldBe "foobar" 34 | } 35 | 36 | it should "support volumes write -> read" in { 37 | 38 | val writeHFn = HFn.withImage(HFnTest.testImage) { 39 | // side effect in volume 40 | File("/foo/bar.txt").appendAll("foobar in a file") 41 | "foobar" 42 | } 43 | 44 | val volume = TransientVolume("foo", "10G") 45 | submitter.submit(writeHFn, env.withMount(volume.mountReadWrite("/foo"))) shouldBe "foobar" 46 | 47 | val readHFn = HFn.withImage(HFnTest.testImage) { 48 | File("/readFoo/bar.txt").bufferedReader().readLine() 49 | } 50 | 51 | submitter.submit(readHFn, env.withMount(volume.mountReadOnly("/readFoo"))) shouldBe "foobar in a file" 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /hype-gcs/src/main/java/com/spotify/hype/gcs/RunManifest.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-gcs 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.gcs; 22 | 23 | import io.norberg.automatter.AutoMatter; 24 | import java.io.IOException; 25 | import java.nio.file.Path; 26 | import java.util.List; 27 | 28 | /** 29 | * 
30 |  *   l continuation-ce89ba3b.bin
31 |  *   c lib1.jar
32 |  *   c lib2.jar
33 |  *   c lib3.jar
34 |  *   f other-file.txt
35 |  *
36 | * 37 | *

Where the different classifiers at the beginning of the line stand for 38 | * 39 | * 

40 |  *   l continuation lambda, last one will be picked if several entries exist
41 |  *   c jar file, will be added to the classpath
42 |  *   f regular file, will just be downloaded to the temp location
43 |  *
44 | */ 45 | @AutoMatter 46 | public interface RunManifest { 47 | 48 | String continuation(); 49 | 50 | List classPathFiles(); 51 | 52 | List files(); 53 | 54 | static RunManifest read(Path manifestPath) throws IOException { 55 | return ManifestUtil.read(manifestPath); 56 | } 57 | 58 | static void write(RunManifest manifest, Path manifestPath) throws IOException { 59 | ManifestUtil.write(manifest, manifestPath); 60 | } 61 | } 62 | -------------------------------------------------------------------------------- /hype-submitter/src/test/java/com/spotify/hype/model/ResourcesTest.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.model; 22 | 23 | import static com.spotify.hype.model.ResourceRequest.CPU; 24 | import static com.spotify.hype.model.ResourceRequest.MEMORY; 25 | import static org.hamcrest.Matchers.equalTo; 26 | import static org.junit.Assert.assertThat; 27 | 28 | import org.junit.Test; 29 | import org.junit.experimental.runners.Enclosed; 30 | import org.junit.runner.RunWith; 31 | 32 | @RunWith(Enclosed.class) 33 | public class ResourcesTest { 34 | 35 | @Test 36 | public void request() throws Exception { 37 | ResourceRequest unitRequest = ResourceRequest.request("foo", "7"); 38 | 39 | assertThat(unitRequest.resource(), equalTo("foo")); 40 | assertThat(unitRequest.amount(), equalTo("7")); 41 | } 42 | 43 | public static class CpuResource extends ResourceRequestTests { 44 | public CpuResource() { 45 | super(CPU, "cpu"); 46 | } 47 | } 48 | 49 | public static class MemoryResource extends ResourceRequestTests { 50 | public MemoryResource() { 51 | super(MEMORY, "memory"); 52 | } 53 | } 54 | 55 | public static class CustomResource extends ResourceRequestTests { 56 | public CustomResource() { 57 | super(ResourceRequest.forResource("custom"), "custom"); 58 | } 59 | } 60 | } 61 | -------------------------------------------------------------------------------- /hype-testing/src/test/scala/com/spotify/hype/magic/SubmitterOpsTest.scala: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017 Spotify AB. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); 5 | * you may not use this file except in compliance with the License. 6 | * You may obtain a copy of the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, 11 | * software distributed under the License is distributed on an 12 | * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 13 | * KIND, either express or implied. See the License for the 14 | * specific language governing permissions and limitations 15 | * under the License. 16 | */ 17 | package com.spotify.hype.magic 18 | 19 | import com.spotify.hype.{HFn, HFnTest, LocalSubmitter, RunEnvironment, TransientVolume} 20 | import org.scalatest.{FlatSpec, Matchers} 21 | 22 | import scala.language.postfixOps 23 | import scala.sys.process._ 24 | 25 | class SubmitterOpsTest extends FlatSpec with Matchers { 26 | 27 | private implicit val submitter = LocalSubmitter() 28 | 29 | "Submitter ops" should "do #!" in { 30 | implicit val env = RunEnvironment() 31 | 32 | (getEnv("HYPE_ENV") #!) shouldBe "testing" 33 | } 34 | 35 | def getEnv(name: String): HFn[String] = HFn.withImage(HFnTest.testImage) { 36 | System.getenv(name) 37 | } 38 | 39 | it should "support explicit env" in { 40 | val volume = TransientVolume("slow", "1G") 41 | val explicitEnv = RunEnvironment() 42 | val rwEnv = explicitEnv.withMount(volume.mountReadWrite("/foo")) 43 | val roEnv = explicitEnv.withMount(volume.mountReadOnly("/readFoo")) 44 | 45 | (write("foobar in a file") #! rwEnv) shouldBe "foobar" 46 | (read #! roEnv) shouldBe "foobar in a file" 47 | } 48 | 49 | def write(text: String): HFn[String] = HFn { 50 | // side effect in volume 51 | s"echo -n $text" #> "tee /foo/bar.txt" ! 52 | 53 | "foobar" 54 | } 55 | 56 | def read: HFn[String] = HFn { 57 | ("cat /readFoo/bar.txt" !!).trim 58 | } 59 | } 60 | -------------------------------------------------------------------------------- /hype-common/src/test/java/com/spotify/hype/util/SerializationUtilTest.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-run 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.util; 22 | 23 | import static org.junit.Assert.assertEquals; 24 | 25 | import java.io.Serializable; 26 | import java.nio.file.Path; 27 | import java.util.function.Supplier; 28 | import org.junit.Test; 29 | 30 | @SuppressWarnings("unchecked") 31 | public class SerializationUtilTest { 32 | 33 | private Inner inner = new Inner(); 34 | 35 | @Test 36 | public void roundtripLambda() throws Exception { 37 | Fn fn = () -> inner.sup.get(); 38 | Fn fn1 = roundtrip(fn); 39 | 40 | inner.field = "something else"; 41 | String result = fn1.run(); 42 | 43 | assertEquals("hello", result); 44 | } 45 | 46 | @Test 47 | public void roundtripClosure() throws Exception { 48 | Fn fn = closure("hello"); 49 | Fn fn1 = roundtrip(fn); 50 | 51 | String result = fn1.run(); 52 | assertEquals("hello", result); 53 | } 54 | 55 | private Fn roundtrip(Fn fn) { 56 | Path path = SerializationUtil.serializeContinuation(fn); 57 | return (Fn) SerializationUtil.readContinuation(path); 58 | } 59 | 60 | // this class is intentionally not implementing java.io.Serializable 61 | class Inner { 62 | String field = "hello"; 63 | Supplier sup = (Supplier & Serializable) () -> field; 64 | } 65 | 66 | private static Fn closure(String arg) { 67 | return () -> arg; 68 | } 69 | } 70 | -------------------------------------------------------------------------------- /hype-run/src/main/java/com/spotify/hype/stub/ContinuationEntryPoint.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.stub; 22 | 23 | import com.spotify.hype.util.Fn; 24 | import com.spotify.hype.util.SerializationUtil; 25 | import java.nio.file.Files; 26 | import java.nio.file.Path; 27 | import java.nio.file.Paths; 28 | 29 | /** 30 | * TODO: document. 31 | */ 32 | public class ContinuationEntryPoint { 33 | 34 | public static void main(String[] args) throws Exception { 35 | if (args.length < 2) { 36 | throw new IllegalArgumentException("Usage: "); 37 | } 38 | 39 | final Path continuationPath = Paths.get(args[0], args[1]); 40 | if (!Files.exists(continuationPath)) { 41 | throw new IllegalArgumentException(continuationPath + " does not exist"); 42 | } 43 | 44 | final Path returnValuePath = Paths.get(args[0], args[2]); 45 | if (Files.exists(returnValuePath)) { 46 | throw new IllegalArgumentException(returnValuePath + " already exists"); 47 | } 48 | 49 | System.setProperty("user.dir", args[0]); 50 | final Fn continuation = SerializationUtil.readContinuation(continuationPath); 51 | 52 | Object returnValue; 53 | try { 54 | returnValue = continuation.run(); 55 | } catch (Throwable e) { 56 | e.printStackTrace(); 57 | throw e; 58 | } 59 | 60 | SerializationUtil.serializeObject(returnValue, returnValuePath); 61 | System.out.println("returnValuePath = " + returnValuePath); 62 | } 63 | } 64 | -------------------------------------------------------------------------------- /hype-testing/src/test/scala/com/spotify/hype/ScalaSerializationTest.scala: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-testing 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype 22 | 23 | import java.util.function.Consumer 24 | 25 | import com.spotify.hype.util.Fn 26 | import com.spotify.hype.util.SerializationUtil._ 27 | import io.rouz.flo._ 28 | import org.scalatest._ 29 | 30 | class ScalaSerializationTest extends FlatSpec with Matchers { 31 | 32 | "A flo task closure" can "be serialized and deserialized" in { 33 | val ret = closure("do") 34 | 35 | TaskContext.inmem().evaluate(ret).consume(new Consumer[String] { 36 | def accept(t: String): Unit = { 37 | t shouldBe "do more" 38 | } 39 | }) 40 | } 41 | 42 | def closure(arg: String): Task[String] = defTask($ 43 | in inner 44 | process Wrapper.wrap(ExampleFunction(arg)) 45 | ) 46 | 47 | def inner: Task[String] = task("inner").process("more") 48 | } 49 | 50 | object ExampleFunction { 51 | def apply(arg: String): (String) => String = { 52 | (other) => arg + " " + other 53 | } 54 | } 55 | 56 | object Wrapper { 57 | 58 | def wrap(fn: (String) => String): (String) => String = { 59 | (a) => { 60 | val fnn = fn 61 | val closure = tofn(fnn(a)) 62 | 63 | val deserialized = roundtrip(closure) 64 | deserialized.run() 65 | } 66 | } 67 | 68 | def roundtrip[T](fn: Fn[T]): Fn[T] = 69 | readContinuation(serializeContinuation(fn)).asInstanceOf[Fn[T]] 70 | 71 | def tofn[A](continuation: => A): Fn[A] = new Fn[A] { 72 | def run(): A = continuation 73 | } 74 | } 75 | -------------------------------------------------------------------------------- /hype-gcs/src/main/java/com/spotify/hype/gcs/ManifestLoader.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-gcs 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.gcs; 22 | 23 | import static com.google.common.collect.Iterables.concat; 24 | 25 | import java.io.IOException; 26 | import java.nio.file.Files; 27 | import java.nio.file.Path; 28 | import java.nio.file.StandardCopyOption; 29 | import java.util.LinkedHashSet; 30 | import java.util.Set; 31 | import java.util.concurrent.ExecutionException; 32 | import java.util.concurrent.ForkJoinPool; 33 | 34 | public final class ManifestLoader { 35 | 36 | private static final ForkJoinPool FJP = new ForkJoinPool(32); 37 | 38 | public static RunManifest downloadManifest(Path manifestPath, Path destinationDir) throws IOException { 39 | final RunManifest manifest = ManifestUtil.read(manifestPath); 40 | 41 | final Set manifestEntries = new LinkedHashSet<>(); 42 | manifestEntries.add(manifestPath.resolveSibling(manifest.continuation())); 43 | for (String file : concat(manifest.classPathFiles(), manifest.files())) { 44 | manifestEntries.add(manifestPath.resolveSibling(file)); 45 | } 46 | 47 | try { 48 | FJP.submit(() -> manifestEntries.parallelStream() 49 | .forEach(filePath -> downloadFile(filePath, destinationDir))) 50 | .get(); 51 | } catch (InterruptedException | ExecutionException e) { 52 | throw new RuntimeException(e); 53 | } 54 | 55 | return manifest; 56 | } 57 | 58 | private static void downloadFile(Path filePath, Path destinationDir) { 59 | final Path destinationFile = destinationDir.resolve(filePath.getFileName().toString()); 60 | try { 61 | // todo: retries 62 | Files.copy(filePath, destinationFile, StandardCopyOption.REPLACE_EXISTING); 63 | } catch (IOException e) { 64 | throw new RuntimeException(e); 65 | } 66 | } 67 | } 68 | -------------------------------------------------------------------------------- /hype-gcs/src/main/java/com/spotify/hype/gcs/ManifestUtil.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-gcs 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.gcs; 22 | 23 | import java.io.IOException; 24 | import java.io.PrintWriter; 25 | import java.nio.file.Files; 26 | import java.nio.file.Path; 27 | import java.util.stream.Stream; 28 | import org.slf4j.Logger; 29 | import org.slf4j.LoggerFactory; 30 | 31 | class ManifestUtil { 32 | 33 | private static final Logger LOG = LoggerFactory.getLogger(ManifestUtil.class); 34 | 35 | private static final char LAMBDA = 'l'; 36 | private static final char CLASSPATH_FILE = 'c'; 37 | private static final char REGULAR_FILE = 'f'; 38 | 39 | static RunManifest read(Path manifestPath) throws IOException { 40 | final Stream lines = Files.lines(manifestPath); 41 | 42 | final RunManifestBuilder builder = new RunManifestBuilder(); 43 | lines.forEachOrdered(line -> { 44 | if (line.trim().isEmpty()) { 45 | return; 46 | } 47 | 48 | final String[] split = line.trim().split(" ", 2); 49 | if (split.length != 2) { 50 | throw new IllegalArgumentException("Malformed manifest line '" + line + "'"); 51 | } 52 | 53 | switch (split[0].charAt(0)) { 54 | case LAMBDA: 55 | builder.continuation(split[1]); 56 | break; 57 | 58 | case CLASSPATH_FILE: 59 | builder.addClassPathFile(split[1]); 60 | break; 61 | 62 | case REGULAR_FILE: 63 | builder.addFile(split[1]); 64 | break; 65 | 66 | default: 67 | LOG.warn("Unrecognized manifest entry '" + line + "'"); 68 | } 69 | }); 70 | 71 | return builder.build(); 72 | } 73 | 74 | static void write(RunManifest manifest, Path manifestPath) throws IOException { 75 | try (PrintWriter writer = new PrintWriter(Files.newOutputStream(manifestPath))) { 76 | writer.write(LAMBDA + " " + manifest.continuation() + '\n'); 77 | manifest.classPathFiles().forEach(cpf -> writer.write(CLASSPATH_FILE + " " + cpf + '\n')); 78 | manifest.files().forEach(file -> writer.write(REGULAR_FILE + " " + file + '\n')); 79 | } 80 | } 81 | } 82 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/model/RunEnvironment.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.model; 22 | 23 | import io.norberg.automatter.AutoMatter; 24 | import java.net.URI; 25 | import java.net.URISyntaxException; 26 | import java.nio.file.Path; 27 | import java.nio.file.Paths; 28 | import java.util.List; 29 | import java.util.Map; 30 | import java.util.Optional; 31 | 32 | @AutoMatter 33 | public interface RunEnvironment { 34 | 35 | Optional yamlPath(); 36 | List secretMounts(); 37 | List volumeMounts(); 38 | Map resourceRequests(); 39 | 40 | static RunEnvironment environment() { 41 | return new RunEnvironmentBuilder().build(); 42 | } 43 | 44 | static RunEnvironment fromYaml(String resourcePath) { 45 | final URI resourceUri; 46 | try { 47 | resourceUri = RunEnvironment.class.getResource(resourcePath).toURI(); 48 | } catch (URISyntaxException e) { 49 | throw new RuntimeException(e); 50 | } 51 | return fromYaml(Paths.get(resourceUri)); 52 | } 53 | 54 | static RunEnvironment fromYaml(Path path) { 55 | return new RunEnvironmentBuilder() 56 | .yamlPath(path) 57 | .build(); 58 | } 59 | 60 | default RunEnvironment withSecret(String name, String mountPath) { 61 | return RunEnvironmentBuilder.from(this) 62 | .addSecretMount(Secret.secret(name, mountPath)) 63 | .build(); 64 | } 65 | 66 | default RunEnvironment withSecret(Secret secret) { 67 | return RunEnvironmentBuilder.from(this) 68 | .addSecretMount(secret) 69 | .build(); 70 | } 71 | 72 | default RunEnvironment withMount(VolumeMount volumeMount) { 73 | return RunEnvironmentBuilder.from(this) 74 | .addVolumeMount(volumeMount) 75 | .build(); 76 | } 77 | 78 | default RunEnvironment withRequest(String resource, String amount) { 79 | return RunEnvironmentBuilder.from(this) 80 | .putResourceRequest(resource, amount) 81 | .build(); 82 | } 83 | 84 | default RunEnvironment withRequest(ResourceRequest request) { 85 | return RunEnvironmentBuilder.from(this) 86 | .putResourceRequest(request.resource(), request.amount()) 87 | .build(); 88 | } 89 | } 90 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/model/VolumeRequest.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.model; 22 | 23 | import com.spotify.hype.util.Util; 24 | import io.norberg.automatter.AutoMatter; 25 | 26 | /** 27 | * A request for a volume to be created from the specified storage class, with a specific size. 28 | * 29 | *

see http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html 30 | */ 31 | @AutoMatter 32 | public interface VolumeRequest { 33 | 34 | String VOLUME_REQUEST_PREFIX = "hype-request-"; 35 | 36 | String id(); 37 | boolean keep(); 38 | ClaimRequest spec(); 39 | 40 | @AutoMatter 41 | interface ClaimRequest { 42 | String storageClass(); 43 | String size(); 44 | boolean useExisting(); 45 | } 46 | 47 | static VolumeRequest volumeRequest(String storageClass, String size) { 48 | final String id = VOLUME_REQUEST_PREFIX + Util.randomAlphaNumeric(8); 49 | return new VolumeRequestBuilder() 50 | .id(id) 51 | .keep(false) // new claims are deleted by default 52 | .spec(new ClaimRequestBuilder() 53 | .storageClass(storageClass) 54 | .size(size) 55 | .useExisting(false) 56 | .build()) 57 | .build(); 58 | } 59 | 60 | static VolumeRequest createIfNotExists(String name, String storageClass, String size) { 61 | final String id = String.format("%s-%s-%s", name, storageClass, size); 62 | return new VolumeRequestBuilder() 63 | .id(id) 64 | .keep(true) 65 | .spec(new ClaimRequestBuilder() 66 | .storageClass(storageClass) 67 | .size(size) 68 | .useExisting(true) 69 | .build()) 70 | .build(); 71 | } 72 | 73 | default VolumeRequest keepOnExit() { 74 | return VolumeRequestBuilder.from(this) 75 | .keep(true) 76 | .build(); 77 | } 78 | 79 | /** 80 | * Mount the requested volume in read-only mode at the specified path. 81 | */ 82 | default VolumeMount mountReadOnly(String mountPath) { 83 | return VolumeMount.volumeMount(this, mountPath, true); 84 | } 85 | 86 | /** 87 | * Mount the requested volume in read-write mode at the specified path. 88 | */ 89 | default VolumeMount mountReadWrite(String mountPath) { 90 | return VolumeMount.volumeMount(this, mountPath, false); 91 | } 92 | } 93 | -------------------------------------------------------------------------------- /hype-common/src/main/java/com/spotify/hype/util/SerializationUtil.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-run 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.util; 22 | 23 | import com.esotericsoftware.kryo.Kryo; 24 | import com.esotericsoftware.kryo.io.Input; 25 | import com.esotericsoftware.kryo.io.Output; 26 | import com.esotericsoftware.kryo.serializers.ClosureSerializer; 27 | import java.io.File; 28 | import java.io.FileInputStream; 29 | import java.io.FileOutputStream; 30 | import java.io.IOException; 31 | import java.io.InputStream; 32 | import java.nio.file.Files; 33 | import java.nio.file.Path; 34 | import org.objenesis.strategy.StdInstantiatorStrategy; 35 | 36 | public class SerializationUtil { 37 | 38 | private static final String CONT_FILE = "continuation-"; 39 | private static final String EXT = ".bin"; 40 | 41 | public static Path serializeContinuation(Fn continuation) { 42 | try { 43 | final Path outputPath = Files.createTempFile(CONT_FILE, EXT); 44 | serializeObject(continuation, outputPath); 45 | return outputPath; 46 | } catch (IOException e) { 47 | throw new RuntimeException(e); 48 | } 49 | } 50 | 51 | public static Fn readContinuation(Path continuationPath) { 52 | return (Fn) readObject(continuationPath); 53 | } 54 | 55 | public static void serializeObject(Object obj, Path outputPath) { 56 | Kryo kryo = new Kryo(); 57 | kryo.register(java.lang.invoke.SerializedLambda.class); 58 | kryo.register(ClosureSerializer.Closure.class, new ClosureSerializer()); 59 | 60 | try { 61 | final File file = outputPath.toFile(); 62 | try (Output output = new Output(new FileOutputStream(file))) { 63 | kryo.writeClassAndObject(output, obj); 64 | } 65 | } catch (IOException e) { 66 | throw new RuntimeException(e); 67 | } 68 | } 69 | 70 | public static Object readObject(Path object) { 71 | File file = object.toFile(); 72 | 73 | try (InputStream input = new FileInputStream(file)) { 74 | return readObject(input); 75 | } catch (IOException e) { 76 | throw new RuntimeException(e); 77 | } 78 | } 79 | 80 | public static Object readObject(InputStream inputStream) { 81 | Kryo kryo = new Kryo(); 82 | kryo.register(java.lang.invoke.SerializedLambda.class); 83 | kryo.register(ClosureSerializer.Closure.class, new ClosureSerializer()); 84 | kryo.setInstantiatorStrategy(new Kryo.DefaultInstantiatorStrategy(new StdInstantiatorStrategy())); 85 | 86 | Input input = new Input(inputStream); 87 | return kryo.readClassAndObject(input); 88 | } 89 | } 90 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/LocalClasspathInspector.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype; 22 | 23 | import java.net.URI; 24 | import java.nio.file.Path; 25 | import java.nio.file.Paths; 26 | import java.util.LinkedHashSet; 27 | import java.util.List; 28 | import java.util.Objects; 29 | import java.util.Set; 30 | import java.util.function.UnaryOperator; 31 | import java.util.stream.Collectors; 32 | import java.util.stream.Stream; 33 | import org.jhades.model.ClasspathEntry; 34 | import org.jhades.service.ClasspathScanner; 35 | 36 | class LocalClasspathInspector implements ClasspathInspector { 37 | 38 | private final ClasspathScanner scanner = new ClasspathScanner(); 39 | private final ClassLoader classLoader; 40 | 41 | LocalClasspathInspector(ClassLoader classLoader) { 42 | this.classLoader = Objects.requireNonNull(classLoader); 43 | } 44 | 45 | LocalClasspathInspector(Class clazz) { 46 | this.classLoader = Objects.requireNonNull(clazz).getClassLoader(); 47 | } 48 | 49 | @Override 50 | public List classpathJars() { 51 | return localClasspath().stream() 52 | .map(entry -> Paths.get(URI.create(entry.getUrl())).toAbsolutePath()) 53 | .collect(Collectors.toList()); 54 | } 55 | 56 | private Set localClasspath() { 57 | final String classLoaderName = classLoader.getClass().getName(); 58 | 59 | return scanner.findAllClasspathEntries().stream() 60 | .filter(entry -> classLoaderName.equals(entry.getClassLoaderName())) 61 | .flatMap(LocalClasspathInspector::jarFileEntriesWithExpandedManifest) 62 | .collect(Collectors.toCollection(LinkedHashSet::new)); 63 | } 64 | 65 | private static Stream jarFileEntriesWithExpandedManifest(ClasspathEntry entry) { 66 | if ((!entry.isJar() && !entry.isClassFolder()) || !entry.getUrl().startsWith("file:")) { 67 | return Stream.empty(); 68 | } 69 | 70 | if (entry.findManifestClasspathEntries().isEmpty()) { 71 | return Stream.of(entry); 72 | } else { 73 | final URI uri = URI.create(entry.getUrl()); 74 | Path path = Paths.get(uri).getParent(); 75 | return Stream.concat( 76 | Stream.of(entry), 77 | entry.findManifestClasspathEntries().stream() 78 | .map(normalizerUsingPath(path))); 79 | } 80 | } 81 | 82 | private static UnaryOperator normalizerUsingPath(Path base) { 83 | return entry -> new ClasspathEntry( 84 | entry.getClassLoader(), 85 | base.resolve(entry.getUrl()).toUri().toString()); 86 | } 87 | } 88 | -------------------------------------------------------------------------------- /hype-gcs/src/test/java/com/spotify/hype/gcs/ManifestUtilTest.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-gcs 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.gcs; 22 | 23 | import static org.hamcrest.Matchers.is; 24 | import static org.junit.Assert.assertThat; 25 | 26 | import java.net.URISyntaxException; 27 | import java.net.URL; 28 | import java.nio.file.Files; 29 | import java.nio.file.Path; 30 | import java.nio.file.Paths; 31 | import java.util.List; 32 | import org.junit.Rule; 33 | import org.junit.Test; 34 | import org.junit.rules.ExpectedException; 35 | 36 | public class ManifestUtilTest { 37 | 38 | private static final RunManifest EXAMPLE = new RunManifestBuilder() 39 | .continuation("continuation-ce89ba3b.bin") 40 | .classPathFiles("lib1.jar", "lib2.jar", "lib3.jar") 41 | .files("other-file.txt") 42 | .build(); 43 | 44 | @Rule 45 | public ExpectedException exception = ExpectedException.none(); 46 | 47 | @Test 48 | public void readManifest() throws Exception { 49 | Path manifestPath = load("/example-manifest.txt"); 50 | RunManifest manifest = ManifestUtil.read(manifestPath); 51 | 52 | assertThat(manifest, is(EXAMPLE)); 53 | } 54 | 55 | @Test 56 | public void multipleContinuation() throws Exception { 57 | Path manifestPath = load("/multi-lambda-manifest.txt"); 58 | RunManifest manifest = ManifestUtil.read(manifestPath); 59 | 60 | assertThat(manifest.continuation(), is("continuation-other.bin")); 61 | } 62 | 63 | @Test 64 | public void writeManifest() throws Exception { 65 | Path manifest = Files.createTempFile("manifest", ".txt"); 66 | ManifestUtil.write(EXAMPLE, manifest); 67 | 68 | List expected = Files.readAllLines(load("/example-manifest.txt")); 69 | List strings = Files.readAllLines(manifest); 70 | 71 | assertThat(strings, is(expected)); 72 | } 73 | 74 | @Test 75 | public void skipEmptyLines() throws Exception { 76 | Path manifestPath = load("/empty-lines-manifest.txt"); 77 | RunManifest manifest = ManifestUtil.read(manifestPath); 78 | 79 | assertThat(manifest, is(EXAMPLE)); 80 | } 81 | 82 | @Test 83 | public void malformedManifest() throws Exception { 84 | exception.expect(IllegalArgumentException.class); 85 | exception.expectMessage("Malformed manifest line 'clib2.jar'"); 86 | 87 | Path manifestPath = load("/malformed-manifest.txt"); 88 | ManifestUtil.read(manifestPath); 89 | } 90 | 91 | private Path load(String resourceName) throws URISyntaxException { 92 | URL resource = ManifestUtil.class.getResource(resourceName); 93 | return Paths.get(resource.toURI()); 94 | } 95 | } 96 | -------------------------------------------------------------------------------- /hype-gcs/src/test/java/com/spotify/hype/gcs/ManifestLoaderTest.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-gcs 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.gcs; 22 | 23 | import static java.util.stream.Collectors.toList; 24 | import static org.hamcrest.Matchers.is; 25 | import static org.junit.Assert.assertThat; 26 | 27 | import com.spotify.hype.gcs.StagingUtil.StagedPackage; 28 | import java.io.File; 29 | import java.net.URI; 30 | import java.net.URISyntaxException; 31 | import java.net.URL; 32 | import java.net.URLClassLoader; 33 | import java.nio.file.Files; 34 | import java.nio.file.Path; 35 | import java.util.Arrays; 36 | import java.util.List; 37 | import org.junit.Before; 38 | import org.junit.Test; 39 | 40 | public class ManifestLoaderTest { 41 | 42 | List testFiles; 43 | Path stagingPath; 44 | String stagingLocation; 45 | 46 | @Before 47 | public void setUp() throws Exception { 48 | URLClassLoader cl = (URLClassLoader) StagingUtilTest.class.getClassLoader(); 49 | testFiles = Arrays.stream(cl.getURLs()) 50 | .map(url -> new File(toUri(url))) 51 | .map(File::getAbsolutePath) 52 | .collect(toList()); 53 | 54 | stagingPath = Files.createTempDirectory("unit-test"); 55 | stagingLocation = stagingPath.toUri().toString(); 56 | } 57 | 58 | @Test 59 | public void end2endStaging() throws Exception { 60 | List stagedPackages = 61 | StagingUtil.stageClasspathElements(testFiles, stagingLocation); 62 | 63 | RunManifest manifest = new RunManifestBuilder() 64 | .continuation(stagedPackages.get(0).name()) 65 | .files(stagedPackages.stream().map(StagedPackage::name).collect(toList())) 66 | .build(); 67 | 68 | Path manifestFile = stagingPath.resolve("manifest.txt"); 69 | RunManifest.write(manifest, manifestFile); 70 | 71 | // add one extra file to staging directory 72 | Files.createTempFile(stagingPath, "extra", ""); 73 | List stagedFiles = Files.list(stagingPath).collect(toList()); 74 | assertThat(stagedFiles.size(), is(testFiles.size() + 2)); // extra + manifest 75 | 76 | Path readPath = Files.createTempDirectory("unit-test"); 77 | RunManifest downloadedManifest = ManifestLoader.downloadManifest(manifestFile, readPath); 78 | 79 | List readFiles = Files.list(readPath).collect(toList()); 80 | assertThat(readFiles.size(), is(testFiles.size())); 81 | assertThat(downloadedManifest, is(manifest)); 82 | } 83 | 84 | private static URI toUri(URL url) { 85 | try { 86 | return url.toURI(); 87 | } catch (URISyntaxException e) { 88 | throw new RuntimeException(e); 89 | } 90 | } 91 | } 92 | -------------------------------------------------------------------------------- /hype-testing/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | hype-root 6 | com.spotify 7 | 0.0.19-SNAPSHOT 8 | 9 | 10 | hype-testing 11 | 12 | 13 | 2.11 14 | 2.11.9 15 | 16 | 17 | 18 | 19 | com.spotify 20 | hype_${scala.baseVersion} 21 | 0.0.19-SNAPSHOT 22 | 23 | 24 | io.rouz 25 | flo-scala_${scala.baseVersion} 26 | ${flo.version} 27 | 28 | 29 | 30 | org.scala-lang 31 | scala-library 32 | ${scala.version} 33 | 34 | 35 | 36 | junit 37 | junit 38 | test 39 | 40 | 41 | org.mockito 42 | mockito-all 43 | test 44 | 45 | 46 | org.scalatest 47 | scalatest_${scala.baseVersion} 48 | 3.0.1 49 | test 50 | 51 | 52 | 53 | 54 | 55 | 56 | src/main/resources 57 | true 58 | 59 | 60 | 61 | 62 | 63 | net.alchim31.maven 64 | scala-maven-plugin 65 | 3.2.1 66 | 67 | 68 | 69 | compile 70 | testCompile 71 | 72 | 73 | 74 | 75 | ${scala.version} 76 | true 77 | 78 | 79 | 80 | 81 | 82 | org.scalatest 83 | scalatest-maven-plugin 84 | 1.0 85 | 86 | ${project.build.directory}/surefire-reports 87 | . 88 | WDF TestSuite.txt 89 | 90 | 91 | 92 | test 93 | 94 | test 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | -------------------------------------------------------------------------------- /hype-gcs/src/test/java/com/spotify/hype/gcs/StagingUtilTest.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-gcs 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.gcs; 22 | 23 | import static java.util.Collections.singletonList; 24 | import static java.util.stream.Collectors.toList; 25 | import static org.hamcrest.Matchers.containsString; 26 | import static org.hamcrest.Matchers.hasItem; 27 | import static org.hamcrest.Matchers.hasSize; 28 | import static org.hamcrest.Matchers.is; 29 | import static org.junit.Assert.assertThat; 30 | 31 | import java.io.File; 32 | import java.net.URLClassLoader; 33 | import java.nio.file.Files; 34 | import java.nio.file.Path; 35 | import java.util.List; 36 | import org.junit.Before; 37 | import org.junit.Test; 38 | 39 | public class StagingUtilTest { 40 | 41 | String testFilePath; 42 | String filenameWithoutExtension; 43 | Path tmp; 44 | String stagingPath; 45 | 46 | @Before 47 | public void setUp() throws Exception { 48 | URLClassLoader cl = (URLClassLoader) StagingUtilTest.class.getClassLoader(); 49 | File testFile = new File(cl.getURLs()[0].toURI()); 50 | testFilePath = testFile.getAbsolutePath(); 51 | filenameWithoutExtension = testFile.getName().replaceAll("\\.\\w+$", ""); 52 | 53 | tmp = Files.createTempDirectory("unit-test"); 54 | stagingPath = tmp.toUri().toString(); 55 | } 56 | 57 | @Test 58 | public void stagesFileWithHashedFilename() throws Exception { 59 | StagingUtil.stageClasspathElements(singletonList(testFilePath), stagingPath); 60 | 61 | List list = Files.list(tmp).map(Path::toString).collect(toList()); 62 | assertThat(list, hasSize(1)); 63 | assertThat(list, hasItem(containsString(filenameWithoutExtension))); 64 | 65 | assertThat(list.get(0).matches(".+/" + md5HashPattern()), is(true)); 66 | } 67 | 68 | @Test 69 | public void detectsAlreadyStagedFiles() throws Exception { 70 | StagingUtil.stageClasspathElements(singletonList(testFilePath), stagingPath); 71 | StagingUtil.stageClasspathElements(singletonList(testFilePath), stagingPath); 72 | 73 | List list = Files.list(tmp).map(Path::toString).collect(toList()); 74 | assertThat(list, hasSize(1)); 75 | } 76 | 77 | @Test 78 | public void returnStagedLocations() throws Exception { 79 | List stagedPackages = 80 | StagingUtil.stageClasspathElements(singletonList(testFilePath), stagingPath); 81 | 82 | assertThat(stagedPackages, hasSize(1)); 83 | StagingUtil.StagedPackage stagedPackage = stagedPackages.get(0); 84 | 85 | assertThat(stagedPackage.name().matches(md5HashPattern()), is(true)); 86 | assertThat(stagedPackage.location().matches(".+/" + md5HashPattern()), is(true)); 87 | } 88 | 89 | private String md5HashPattern() { 90 | return filenameWithoutExtension + "-.{22}.+"; 91 | } 92 | } 93 | -------------------------------------------------------------------------------- /hype-caplet/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | hype-root 6 | com.spotify 7 | 0.0.19-SNAPSHOT 8 | 9 | 10 | hype-caplet 11 | 12 | 13 | 0.9.4-beta 14 | 1.0.3 15 | 16 | 17 | 18 | 19 | com.spotify 20 | hype-gcs 21 | 0.0.19-SNAPSHOT 22 | 23 | 24 | com.spotify 25 | hype-common 26 | 0.0.19-SNAPSHOT 27 | 28 | 29 | 30 | co.paralleluniverse 31 | capsule 32 | ${capsule.version} 33 | 34 | 35 | co.paralleluniverse 36 | capsule-util 37 | ${capsule.version} 38 | 39 | 40 | 41 | ch.qos.logback 42 | logback-classic 43 | 44 | 45 | 46 | 47 | 48 | 49 | src/main/resources 50 | true 51 | 52 | 53 | 54 | 55 | 56 | maven-compiler-plugin 57 | 3.5.1 58 | 59 | 1.8 60 | 1.8 61 | 62 | -Xlint:all 63 | 64 | 65 | 66 | 67 | 68 | maven-jar-plugin 69 | 3.0.2 70 | 71 | 72 | 73 | true 74 | true 75 | 76 | 77 | 78 | 79 | 80 | 81 | org.apache.maven.plugins 82 | maven-shade-plugin 83 | 3.0.0 84 | 85 | 86 | package 87 | 88 | shade 89 | 90 | 91 | 92 | 93 | 94 | Capsule 95 | Hypelet 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | -------------------------------------------------------------------------------- /hype-run/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | com.spotify 6 | hype-root 7 | 0.0.19-SNAPSHOT 8 | 9 | 10 | hype-run 11 | 12 | 13 | 14 | com.spotify 15 | hype-common 16 | 0.0.19-SNAPSHOT 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | org.apache.maven.plugins 25 | maven-shade-plugin 26 | 3.0.0 27 | 28 | 29 | package 30 | 31 | shade 32 | 33 | 34 | 35 | 36 | org.scala-lang:scala-library 37 | 38 | 39 | 40 | 41 | com.esotericsoftware 42 | shaded.com.esotericsoftware 43 | 44 | 45 | org.objenesis 46 | shaded.org.objenesis 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | com.github.chrisdchristo 56 | capsule-maven-plugin 57 | 1.5.0 58 | 59 | 60 | 61 | com.spotify.hype.stub.ContinuationEntryPoint 62 | com.spotify:hype-caplet:${project.version} 63 | 64 | true 65 | 66 | 67 | 68 | true 69 | 70 | 71 | 72 | noop 73 | 74 | 75 | Application-Class 76 | com.spotify.hype.stub.Noop 77 | 78 | 79 | 80 | 81 | 82 | 83 | Allow-Snapshots 84 | true 85 | 86 | 87 | Repositories 88 | central local 89 | 90 | 91 | 92 | 93 | 94 | package 95 | 96 | build 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/runner/DockerRunner.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * Spotify Styx Scheduler Service 4 | * -- 5 | * Copyright (C) 2016 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.runner; 22 | 23 | import com.google.api.client.googleapis.auth.oauth2.GoogleCredential; 24 | import com.google.api.services.container.Container; 25 | import com.google.api.services.container.ContainerScopes; 26 | import com.google.api.services.container.model.Cluster; 27 | import com.google.common.base.Throwables; 28 | import com.spotify.docker.client.DefaultDockerClient; 29 | import com.spotify.docker.client.DockerClient; 30 | import com.spotify.docker.client.exceptions.DockerCertificateException; 31 | import com.spotify.hype.model.ContainerEngineCluster; 32 | import com.spotify.hype.model.DockerCluster; 33 | import io.fabric8.kubernetes.client.ConfigBuilder; 34 | import io.fabric8.kubernetes.client.DefaultKubernetesClient; 35 | import io.fabric8.kubernetes.client.KubernetesClient; 36 | import java.io.IOException; 37 | import java.net.URI; 38 | import java.util.Optional; 39 | import org.slf4j.Logger; 40 | import org.slf4j.LoggerFactory; 41 | 42 | /** 43 | * Defines an interface to the Docker execution environment 44 | */ 45 | public interface DockerRunner { 46 | 47 | String NAMESPACE = "default"; 48 | Logger LOG = LoggerFactory.getLogger(DockerRunner.class); 49 | 50 | /** 51 | * Runs a hype execution. Blocks until complete. 52 | * 53 | * @param runSpec Specification of what to run 54 | * @return Optionally a uri pointing to the gcs location of the return value 55 | */ 56 | Optional run(RunSpec runSpec); 57 | 58 | static DockerRunner kubernetes( 59 | KubernetesClient kubernetesClient, 60 | VolumeRepository volumeRepository) { 61 | return new KubernetesDockerRunner(kubernetesClient, volumeRepository); 62 | } 63 | 64 | static DockerRunner local(DockerClient dockerClient, 65 | DockerCluster dockerCluster) { 66 | return new LocalDockerRunner( 67 | dockerClient, 68 | dockerCluster.keepContainer(), 69 | dockerCluster.keepTerminationLog(), 70 | dockerCluster.keepVolumes()); 71 | } 72 | 73 | static KubernetesClient createKubernetesClient(ContainerEngineCluster gkeCluster) { 74 | try { 75 | final GoogleCredential credential = GoogleCredential.getApplicationDefault() 76 | .createScoped(ContainerScopes.all()); 77 | final Container gke = new Container.Builder(credential.getTransport(), credential.getJsonFactory(), credential) 78 | .setApplicationName("hype") 79 | .build(); 80 | 81 | final Cluster cluster = gke.projects().zones().clusters() 82 | .get(gkeCluster.project(), gkeCluster.zone(), gkeCluster.cluster()) 83 | .execute(); 84 | 85 | final io.fabric8.kubernetes.client.Config kubeConfig = new ConfigBuilder() 86 | .withMasterUrl("https://" + cluster.getEndpoint()) 87 | .withCaCertData(cluster.getMasterAuth().getClusterCaCertificate()) 88 | .withClientCertData(cluster.getMasterAuth().getClientCertificate()) 89 | .withClientKeyData(cluster.getMasterAuth().getClientKey()) 90 | .build(); 91 | 92 | return new DefaultKubernetesClient(kubeConfig).inNamespace(NAMESPACE); 93 | } catch (IOException e) { 94 | throw Throwables.propagate(e); 95 | } 96 | } 97 | 98 | static DockerClient createDockerClient() { 99 | try { 100 | return DefaultDockerClient.fromEnv().build(); 101 | } catch (DockerCertificateException e) { 102 | throw Throwables.propagate(e); 103 | } 104 | } 105 | } 106 | -------------------------------------------------------------------------------- /hype-submitter/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | com.spotify 6 | hype-root 7 | 0.0.19-SNAPSHOT 8 | 9 | 10 | hype-submitter 11 | 12 | 13 | 14 | com.spotify 15 | hype-common 16 | 0.0.19-SNAPSHOT 17 | 18 | 19 | com.spotify 20 | hype-gcs 21 | 0.0.19-SNAPSHOT 22 | 23 | 24 | 25 | io.fabric8 26 | kubernetes-client 27 | 28 | 29 | 30 | io.norberg 31 | auto-matter 32 | 33 | 34 | com.google.auto.value 35 | auto-value 36 | 37 | 38 | org.jhades 39 | jhades 40 | 41 | 42 | com.google.apis 43 | google-api-services-container 44 | 45 | 46 | com.google.guava 47 | guava-jdk5 48 | 49 | 50 | 51 | 52 | com.spotify 53 | docker-client 54 | 55 | 56 | 57 | com.google.guava 58 | guava 59 | 60 | 61 | org.slf4j 62 | slf4j-api 63 | 64 | 65 | 66 | junit 67 | junit 68 | test 69 | 70 | 71 | org.hamcrest 72 | hamcrest-all 73 | test 74 | 75 | 76 | org.mockito 77 | mockito-all 78 | test 79 | 80 | 81 | com.github.npathai 82 | hamcrest-optional 83 | test 84 | 85 | 86 | 87 | 88 | 89 | 90 | org.apache.maven.plugins 91 | maven-shade-plugin 92 | 3.0.0 93 | 94 | 95 | package 96 | 97 | shade 98 | 99 | 100 | 101 | 102 | com.google 103 | com.shaded.google 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | *:* 112 | 113 | META-INF/*.SF 114 | META-INF/*.DSA 115 | META-INF/*.RSA 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/runner/VolumeRepository.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.runner; 22 | 23 | import static java.util.stream.Collectors.toList; 24 | 25 | import com.spotify.hype.model.VolumeRequest; 26 | import com.spotify.hype.model.VolumeRequest.ClaimRequest; 27 | import io.fabric8.kubernetes.api.model.PersistentVolumeClaim; 28 | import io.fabric8.kubernetes.api.model.PersistentVolumeClaimBuilder; 29 | import io.fabric8.kubernetes.api.model.Quantity; 30 | import io.fabric8.kubernetes.api.model.ResourceRequirements; 31 | import io.fabric8.kubernetes.api.model.ResourceRequirementsBuilder; 32 | import io.fabric8.kubernetes.client.KubernetesClient; 33 | import java.io.Closeable; 34 | import java.io.IOException; 35 | import java.util.List; 36 | import java.util.Map; 37 | import java.util.Objects; 38 | import java.util.concurrent.ConcurrentHashMap; 39 | import java.util.concurrent.ConcurrentMap; 40 | import org.slf4j.Logger; 41 | import org.slf4j.LoggerFactory; 42 | 43 | /** 44 | * A repository for creating temporary {@link PersistentVolumeClaim}s from {@link VolumeRequest}s. 45 | * 46 | *

The repository will delete all created claims when it is closed. 47 | */ 48 | public class VolumeRepository implements Closeable { 49 | 50 | private static final Logger LOG = LoggerFactory.getLogger(VolumeRepository.class); 51 | 52 | static final String STORAGE_CLASS_ANNOTATION = "volume.beta.kubernetes.io/storage-class"; 53 | static final String READ_WRITE_ONCE = "ReadWriteOnce"; 54 | static final String READ_ONLY_MANY = "ReadOnlyMany"; 55 | 56 | private final KubernetesClient client; 57 | 58 | private final ConcurrentMap claims = 59 | new ConcurrentHashMap<>(); 60 | 61 | public VolumeRepository(KubernetesClient client) { 62 | this.client = Objects.requireNonNull(client); 63 | } 64 | 65 | PersistentVolumeClaim getClaim(VolumeRequest volumeRequest) { 66 | return claims.computeIfAbsent(volumeRequest, this::createClaim); 67 | } 68 | 69 | private PersistentVolumeClaim createClaim(VolumeRequest volumeRequest) { 70 | final ClaimRequest spec = volumeRequest.spec(); 71 | 72 | if (spec.useExisting()) { 73 | final String claimName = volumeRequest.id(); 74 | final PersistentVolumeClaim existingClaim = 75 | client.persistentVolumeClaims().withName(claimName).get(); 76 | 77 | if (existingClaim != null) { 78 | return existingClaim; 79 | } 80 | } 81 | 82 | final ResourceRequirements resources = new ResourceRequirementsBuilder() 83 | .addToRequests("storage", new Quantity(spec.size())) 84 | .build(); 85 | 86 | final PersistentVolumeClaim claimTemplate = new PersistentVolumeClaimBuilder() 87 | .withNewMetadata() 88 | .withName(volumeRequest.id()) 89 | .addToAnnotations(STORAGE_CLASS_ANNOTATION, spec.storageClass()) 90 | .endMetadata() 91 | .withNewSpec() 92 | // todo: storageClassName: // in 1.6 93 | .withAccessModes(READ_WRITE_ONCE, READ_ONLY_MANY) 94 | .withResources(resources) 95 | .endSpec() 96 | .build(); 97 | 98 | final PersistentVolumeClaim claim = client.persistentVolumeClaims().create(claimTemplate); 99 | LOG.info("Created PersistentVolumeClaim {} for {}", 100 | claim.getMetadata().getName(), 101 | volumeRequest); 102 | 103 | return claim; 104 | } 105 | 106 | @Override 107 | public void close() throws IOException { 108 | final List toDelete = claims.entrySet().stream() 109 | .filter(e -> !e.getKey().keep()) 110 | .map(Map.Entry::getValue) 111 | .collect(toList()); 112 | 113 | client.persistentVolumeClaims().delete(toDelete); 114 | } 115 | } 116 | -------------------------------------------------------------------------------- /hype-submitter/src/test/java/com/spotify/hype/runner/VolumeRepositoryTest.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.runner; 22 | 23 | import static org.hamcrest.Matchers.contains; 24 | import static org.hamcrest.Matchers.equalTo; 25 | import static org.hamcrest.Matchers.hasEntry; 26 | import static org.hamcrest.Matchers.is; 27 | import static org.hamcrest.Matchers.not; 28 | import static org.junit.Assert.assertEquals; 29 | import static org.junit.Assert.assertThat; 30 | import static org.mockito.Matchers.any; 31 | import static org.mockito.Mockito.times; 32 | import static org.mockito.Mockito.verify; 33 | import static org.mockito.Mockito.when; 34 | 35 | import com.spotify.hype.model.VolumeRequest; 36 | import io.fabric8.kubernetes.api.model.DoneablePersistentVolumeClaim; 37 | import io.fabric8.kubernetes.api.model.PersistentVolumeClaim; 38 | import io.fabric8.kubernetes.api.model.PersistentVolumeClaimList; 39 | import io.fabric8.kubernetes.api.model.Quantity; 40 | import io.fabric8.kubernetes.client.KubernetesClient; 41 | import io.fabric8.kubernetes.client.dsl.MixedOperation; 42 | import io.fabric8.kubernetes.client.dsl.Resource; 43 | import java.util.List; 44 | import org.junit.Before; 45 | import org.junit.Rule; 46 | import org.junit.Test; 47 | import org.junit.rules.ExpectedException; 48 | import org.junit.runner.RunWith; 49 | import org.mockito.ArgumentCaptor; 50 | import org.mockito.Captor; 51 | import org.mockito.Mock; 52 | import org.mockito.runners.MockitoJUnitRunner; 53 | 54 | @RunWith(MockitoJUnitRunner.class) 55 | public class VolumeRepositoryTest { 56 | 57 | private static final String EXISTING_CLAIM = "name-class-16Gi"; 58 | 59 | @Rule public ExpectedException expect = ExpectedException.none(); 60 | 61 | @Mock KubernetesClient mockClient; 62 | @Mock Resource existingPvcResource; 63 | @Mock Resource nonExistingPvcResource; 64 | @Mock PersistentVolumeClaim mockPvc; 65 | @Mock MixedOperation< // euw 66 | PersistentVolumeClaim, 67 | PersistentVolumeClaimList, 68 | DoneablePersistentVolumeClaim, 69 | Resource< 70 | PersistentVolumeClaim, 71 | DoneablePersistentVolumeClaim>> pvcs; 72 | 73 | @Captor ArgumentCaptor createdPvc; 74 | @Captor ArgumentCaptor> deletedPvcs; 75 | 76 | private VolumeRepository volumeRepository; 77 | 78 | @Before 79 | public void setUp() throws Exception { 80 | volumeRepository = new VolumeRepository(mockClient); 81 | when(mockClient.persistentVolumeClaims()).thenReturn(pvcs); 82 | when(pvcs.withName(any())).thenAnswer(invocation -> 83 | invocation.getArguments()[0].equals(EXISTING_CLAIM) 84 | ? existingPvcResource : nonExistingPvcResource); 85 | when(existingPvcResource.get()).thenReturn(mockPvc); 86 | when(nonExistingPvcResource.get()).thenReturn(null); 87 | when(pvcs.create(createdPvc.capture())).then(invocation -> createdPvc.getValue()); 88 | when(pvcs.delete(deletedPvcs.capture())).thenReturn(true); 89 | } 90 | 91 | @Test 92 | public void createsNewVolumeClaim() throws Exception { 93 | VolumeRequest request = VolumeRequest.volumeRequest("storage-class-name", "16Gi"); 94 | PersistentVolumeClaim claim = volumeRepository.getClaim(request); 95 | 96 | assertThat(claim.getMetadata().getName(), equalTo(request.id())); 97 | assertThat( 98 | claim.getMetadata().getAnnotations(), 99 | hasEntry(VolumeRepository.STORAGE_CLASS_ANNOTATION, "storage-class-name")); 100 | assertThat(claim.getSpec().getAccessModes(), contains("ReadWriteOnce", "ReadOnlyMany")); 101 | assertThat( 102 | claim.getSpec().getResources().getRequests(), 103 | hasEntry("storage", new Quantity("16Gi"))); 104 | } 105 | 106 | @Test 107 | public void returnsExistingClaim() throws Exception { 108 | VolumeRequest request = VolumeRequest.createIfNotExists("name", "class", "16Gi"); 109 | PersistentVolumeClaim claim = volumeRepository.getClaim(request); 110 | 111 | assertThat(claim, is(mockPvc)); 112 | } 113 | 114 | @Test 115 | public void createWhenExistingClaimNotFound() throws Exception { 116 | VolumeRequest request = VolumeRequest.createIfNotExists("new-claim", "class", "16Gi"); 117 | PersistentVolumeClaim claim = volumeRepository.getClaim(request); 118 | 119 | assertThat(claim, not(mockPvc)); 120 | assertThat( 121 | claim.getSpec().getResources().getRequests(), 122 | hasEntry("storage", new Quantity("16Gi"))); 123 | } 124 | 125 | @Test 126 | public void cachesRequestClaims() throws Exception { 127 | VolumeRequest request = VolumeRequest.volumeRequest("storage-class-name", "16Gi"); 128 | volumeRepository.getClaim(request); 129 | volumeRepository.getClaim(request); 130 | 131 | verify(pvcs, times(1)).create(any()); 132 | } 133 | 134 | @Test 135 | public void deletesClaimsOnClose() throws Exception { 136 | VolumeRequest request1 = VolumeRequest.volumeRequest("storage-class-name", "16Gi").keepOnExit(); 137 | VolumeRequest request2 = VolumeRequest.volumeRequest("storage-class-name", "16Gi"); 138 | PersistentVolumeClaim claim1 = volumeRepository.getClaim(request1); 139 | PersistentVolumeClaim claim2 = volumeRepository.getClaim(request2); 140 | 141 | volumeRepository.close(); 142 | assertThat(deletedPvcs.getValue(), contains(claim2)); 143 | } 144 | } 145 | -------------------------------------------------------------------------------- /hype-caplet/src/main/java/Hypelet.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-run 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | import static com.google.cloud.storage.contrib.nio.CloudStorageOptions.withMimeType; 22 | import static com.google.common.base.Charsets.UTF_8; 23 | import static com.spotify.hype.util.Util.randomAlphaNumeric; 24 | import static java.nio.file.StandardOpenOption.CREATE; 25 | import static java.nio.file.StandardOpenOption.CREATE_NEW; 26 | import static java.nio.file.StandardOpenOption.TRUNCATE_EXISTING; 27 | import static java.nio.file.StandardOpenOption.WRITE; 28 | import static java.util.Collections.emptyMap; 29 | 30 | import com.google.common.collect.Sets; 31 | import com.spotify.hype.gcs.ManifestLoader; 32 | import com.spotify.hype.gcs.RunManifest; 33 | import java.io.IOException; 34 | import java.net.URI; 35 | import java.nio.channels.Channels; 36 | import java.nio.channels.WritableByteChannel; 37 | import java.nio.file.FileSystems; 38 | import java.nio.file.Files; 39 | import java.nio.file.OpenOption; 40 | import java.nio.file.Path; 41 | import java.nio.file.Paths; 42 | import java.nio.file.spi.FileSystemProvider; 43 | import java.util.ArrayList; 44 | import java.util.List; 45 | import java.util.Map; 46 | import java.util.Objects; 47 | import java.util.Properties; 48 | import java.util.Set; 49 | 50 | /** 51 | * Capsule caplet that downloads a run-manifest to a temp directory and adds the included files to 52 | * the application JVM classpath. 53 | * 54 | *

It takes one command line arguments: {@code }. 55 | * 56 | *

After staging the manifest contents locally, it invokes the inner JVM by replacing the first 57 | * argument with a path to the local temp directory. It also adds two additional arguments to the 58 | * application JVM, pointing to 1. the continuation file, and 2. to an output file which can be 59 | * written to. If the output file is written, it will be uploaded to the staging uri when the 60 | * application JVM exits. 61 | */ 62 | public class Hypelet extends Capsule { 63 | 64 | private static final String NOOP_MODE = "noop"; 65 | private static final String STAGING_PREFIX = "hype-run-"; 66 | private static final String BINARY = "application/octet-stream"; 67 | private static final String TERMINATION_LOG = "/dev/termination-log"; 68 | private static final String HYPE_EXECUTION_ID = "HYPE_EXECUTION_ID"; 69 | 70 | private final List downloadedJars = new ArrayList<>(); 71 | 72 | private Path manifestPath; 73 | private Path stagingDir; 74 | private String returnFile; 75 | 76 | public Hypelet(Capsule pred) { 77 | super(pred); 78 | } 79 | 80 | @Override 81 | protected ProcessBuilder prelaunch(List jvmArgs, List args) { 82 | if (NOOP_MODE.equals(getMode())) { 83 | return super.prelaunch(jvmArgs, args); 84 | } 85 | 86 | if (args.size() < 1) { 87 | throw new IllegalArgumentException("Usage: "); 88 | } 89 | 90 | System.out.println("=== HYPE RUN CAPSULE (v" + getVersion() + ") ==="); 91 | 92 | try { 93 | final URI uri = URI.create(args.get(0)); 94 | manifestPath = loadFileSystemProvider(uri).getPath(uri); 95 | stagingDir = Files.createTempDirectory(STAGING_PREFIX); 96 | 97 | System.out.println("Downloading files from " + manifestPath.toUri()); 98 | // print manifest 99 | Files.copy(manifestPath, System.out); 100 | final RunManifest manifest = ManifestLoader.downloadManifest(manifestPath, stagingDir); 101 | System.out.println("Done downloading"); 102 | 103 | manifest.classPathFiles().stream() 104 | .map(classPathFile -> stagingDir.resolve(classPathFile)) 105 | .forEach(downloadedJars::add); 106 | 107 | returnFile = manifest.continuation() 108 | .replaceFirst("\\.bin", "-" + getRunId() + "-return.bin"); 109 | 110 | // inner jvm args [tmpDir, continuation-filename, output-filename] 111 | final List stubArgs = new ArrayList<>(args.size()); 112 | stubArgs.add(stagingDir.toString()); 113 | stubArgs.add(manifest.continuation()); 114 | stubArgs.add(returnFile); 115 | return super.prelaunch(jvmArgs, stubArgs); 116 | } catch (Throwable e) { 117 | e.printStackTrace(); 118 | throw new RuntimeException(e); 119 | } 120 | } 121 | 122 | @Override 123 | protected void cleanup() { 124 | if (stagingDir != null && returnFile != null) { 125 | final Path returnFilePath = stagingDir.resolve(returnFile); 126 | if (Files.exists(returnFilePath)) { 127 | try { 128 | final Path uploadPath = manifestPath.resolveSibling(returnFile); 129 | System.out.println("Uploading serialized return value: `" + returnFilePath 130 | + "` to `" + uploadPath.toString() + "`"); 131 | Set options = Sets.newHashSet(WRITE, CREATE_NEW); 132 | if (Objects.equals(uploadPath.toUri().getScheme(), "gs")) { 133 | options.add(withMimeType(BINARY)); 134 | } 135 | try (WritableByteChannel writer = Files.newByteChannel(uploadPath, options)) { 136 | com.google.common.io.Files.asByteSource(returnFilePath.toFile()) 137 | .copyTo(Channels.newOutputStream(writer)); 138 | } 139 | System.out.println("Uploaded to: " + uploadPath.toUri()); 140 | 141 | // write the uploaded uri to the termination log if it exists 142 | final Path terminationLog = Paths.get(TERMINATION_LOG); 143 | if (Files.exists(terminationLog)) { 144 | final byte[] terminationMessage = uploadPath.toUri().toString().getBytes(UTF_8); 145 | try { 146 | Files.write(terminationLog, terminationMessage, CREATE, WRITE, TRUNCATE_EXISTING); 147 | System.out.println("Wrote URI to the termination log `" + terminationLog + "`"); 148 | } catch (IOException e) { 149 | throw new RuntimeException(e); 150 | } 151 | } else { 152 | System.out.println("`" + TERMINATION_LOG + "` does not exist"); 153 | } 154 | } catch (IOException e) { 155 | throw new RuntimeException(e); 156 | } 157 | } 158 | } 159 | 160 | super.cleanup(); 161 | } 162 | 163 | @Override 164 | protected Object lookup0(Object x, String type, 165 | Map.Entry attrContext, 166 | Object context) { 167 | final Object o = super.lookup0(x, type, attrContext, context); 168 | if ("App-Class-Path".equals(attrContext.getKey())) { 169 | final List lookup = new ArrayList<>((List) o); 170 | lookup.addAll(downloadedJars); 171 | return lookup; 172 | } 173 | return o; 174 | } 175 | 176 | private static String getVersion() { 177 | Properties props = new Properties(); 178 | try { 179 | props.load(Hypelet.class.getResourceAsStream("/version.properties")); 180 | } catch (IOException e) { 181 | throw new RuntimeException(e); 182 | } 183 | return props.getProperty("hypelet.version"); 184 | } 185 | 186 | private static String getRunId() { 187 | return System.getenv().containsKey(HYPE_EXECUTION_ID) 188 | ? System.getenv(HYPE_EXECUTION_ID) 189 | : randomAlphaNumeric(8); 190 | } 191 | 192 | private static FileSystemProvider loadFileSystemProvider(URI uri) throws IOException { 193 | if (Objects.equals(uri.getScheme(), "file")) { 194 | return FileSystems.getFileSystem(URI.create("file:///")).provider(); 195 | } else { 196 | return FileSystems.newFileSystem(uri, emptyMap(), Hypelet.class.getClassLoader()).provider(); 197 | } 198 | } 199 | } 200 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/runner/LocalDockerRunner.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.runner; 22 | 23 | import static com.google.common.collect.ImmutableList.of; 24 | 25 | import com.spotify.docker.client.DockerClient; 26 | import com.spotify.docker.client.exceptions.DockerException; 27 | import com.spotify.docker.client.messages.ContainerConfig; 28 | import com.spotify.docker.client.messages.ContainerCreation; 29 | import com.spotify.docker.client.messages.ContainerInfo; 30 | import com.spotify.docker.client.messages.HostConfig; 31 | import com.spotify.docker.client.messages.Image; 32 | import com.spotify.hype.model.RunEnvironment; 33 | import com.spotify.hype.model.StagedContinuation; 34 | import java.io.File; 35 | import java.io.IOException; 36 | import java.net.URI; 37 | import java.nio.file.Files; 38 | import java.nio.file.Path; 39 | import java.nio.file.Paths; 40 | import java.util.List; 41 | import java.util.Objects; 42 | import java.util.Optional; 43 | import java.util.concurrent.TimeUnit; 44 | import org.slf4j.Logger; 45 | import org.slf4j.LoggerFactory; 46 | 47 | public class LocalDockerRunner implements DockerRunner { 48 | 49 | private static final Logger LOG = LoggerFactory.getLogger(LocalDockerRunner.class); 50 | private static final String GCLOUD_CREDENTIALS = "GOOGLE_APPLICATION_CREDENTIALS"; 51 | private static final String STAGING_VOLUME = "/staging"; 52 | private static final String GCLOUD_CREDENTIALS_MOUNT = "/etc/gcloud/key.json"; 53 | private static final int POLL_CONTAINERS_INTERVAL_SECONDS = 1; 54 | 55 | private final DockerClient client; 56 | private final Boolean keepContainer; 57 | private final Boolean keepTerminationLog; 58 | private final Boolean keepVolumes; 59 | 60 | public LocalDockerRunner(final DockerClient client, 61 | final Boolean keepContainer, 62 | final Boolean keepTerminationLog, 63 | final Boolean keepVolumes) { 64 | this.client = client; 65 | this.keepContainer = keepContainer; 66 | this.keepTerminationLog = keepTerminationLog; 67 | this.keepVolumes = keepVolumes; 68 | } 69 | 70 | @Override 71 | public Optional run(final RunSpec runSpec) { 72 | final RunEnvironment env = runSpec.runEnvironment(); 73 | final StagedContinuation stagedContinuation = runSpec.stagedContinuation(); 74 | final String imageWithTag = runSpec.image(); 75 | 76 | final ContainerCreation creation; 77 | try { 78 | // check if it's needed to pull the image 79 | // TODO: figure out authentication with private repos 80 | final List images = client.listImages(); 81 | if (images.stream().noneMatch(i -> 82 | i.repoTags() != null 83 | && i.repoTags().stream().anyMatch(t -> Objects.equals(t, imageWithTag)))) { 84 | LOG.info("Pulling image " + imageWithTag); 85 | try { 86 | client.pull(imageWithTag, System.out::println); // blocking 87 | } catch (DockerException e) { 88 | LOG.error("Could not pull the image " + imageWithTag + ". Try to pull it yourself."); 89 | throw e; 90 | } 91 | } 92 | final HostConfig.Builder hostConfig = HostConfig.builder(); 93 | // Use GOOGLE_APPLICATION_CREDENTIALS environment variable to mount into 94 | final String credentials = System.getenv(GCLOUD_CREDENTIALS); 95 | if (credentials == null) { 96 | LOG.warn(GCLOUD_CREDENTIALS + " not set, won't mount gcloud credentials"); 97 | } else { 98 | LOG.info("Mounting `" + credentials + "` as `" + GCLOUD_CREDENTIALS_MOUNT + "`"); 99 | hostConfig.appendBinds(HostConfig.Bind.from(credentials) 100 | .to(GCLOUD_CREDENTIALS_MOUNT) 101 | .readOnly(true) 102 | .build()); 103 | } 104 | 105 | // Mount temporary file to act as the termination log 106 | // Use user home because Docker Engine daemon has only limited access to on macOS or 107 | // Windows filesystem 108 | final Path localTmp = new File(System.getProperty("user.home")).toPath().resolve(".tmp"); 109 | final Path termLogs = 110 | Files.createDirectories(localTmp.resolve("spotify-hype-termination-logs")); 111 | final Path terminationLog = Files.createTempFile(termLogs, "termination-log", ".txt"); 112 | if (!keepTerminationLog) { 113 | terminationLog.toFile().deleteOnExit(); 114 | } 115 | hostConfig.appendBinds(HostConfig.Bind 116 | .from(terminationLog.toString()) 117 | .to("/dev/termination-log") 118 | .readOnly(false) 119 | .build()); 120 | 121 | hostConfig.appendBinds(HostConfig.Bind 122 | .from(runSpec.stagedContinuation().manifestPath().getParent().toString()) 123 | .to(STAGING_VOLUME) 124 | .readOnly(false) 125 | .build()); 126 | 127 | Path volumes = Files.createDirectories(localTmp.resolve("spotify-hype-volumes")); 128 | env.volumeMounts().forEach(m -> { 129 | String localVolume = volumes.resolve(m.volumeRequest().id()).toString(); 130 | hostConfig.appendBinds(HostConfig.Bind 131 | .from(localVolume) 132 | .to(m.mountPath()) 133 | .build()); 134 | }); 135 | 136 | final File stagingContinuationFile = stagedContinuation.manifestPath().toFile(); 137 | 138 | final ContainerConfig containerConfig = ContainerConfig.builder() 139 | .image(imageWithTag) 140 | .cmd(of("file://" + STAGING_VOLUME + "/" + stagingContinuationFile.getName())) 141 | .hostConfig(hostConfig.build()) 142 | .build(); 143 | creation = client.createContainer(containerConfig); 144 | client.startContainer(creation.id()); 145 | LOG.info("Started container {}", creation.id()); 146 | final Optional uri = blockUntilComplete(creation.id(), terminationLog); 147 | if (!keepContainer) { 148 | if (Objects.equals(System.getenv("CIRCLECI"), "true")) { 149 | LOG.info("Running on CircleCi - won't delete container due to " + 150 | " https://circleci.com/docs/1.0/docker-btrfs-error/"); 151 | } else { 152 | client.removeContainer(creation.id()); 153 | } 154 | } 155 | return uri.map(u -> stagingContinuationFile.toPath() 156 | .resolveSibling(Paths.get(u).toFile().getName()).toUri()); 157 | } catch (DockerException | IOException e) { 158 | throw new RuntimeException("Failed to start docker container", e); 159 | } catch (InterruptedException e) { 160 | throw new RuntimeException("Interrupted while blocking", e); 161 | } 162 | } 163 | 164 | private Optional blockUntilComplete(final String containerId, 165 | final Path terminationLog) throws InterruptedException { 166 | while (true) { 167 | final ContainerInfo containerInfo; 168 | try { 169 | containerInfo = client.inspectContainer(containerId); 170 | 171 | if (!containerInfo.state().running()) { 172 | final int exitCode = containerInfo.state().exitCode(); 173 | LOG.info("Docker container {} exited with exit code {}", containerId, exitCode); 174 | 175 | if (exitCode == 0) { 176 | final File terminationFile = terminationLog.toFile(); 177 | if (terminationFile.exists()) { 178 | final String message = new String(Files.readAllBytes(terminationLog)); 179 | LOG.info("Got termination message: {}", message); 180 | return Optional.of(URI.create(message)); 181 | } 182 | } 183 | return Optional.empty(); 184 | } 185 | } catch (DockerException | InterruptedException | IOException e) { 186 | LOG.error("Error while reading status from docker", e); 187 | return Optional.empty(); 188 | } 189 | 190 | Thread.sleep(TimeUnit.SECONDS.toMillis(POLL_CONTAINERS_INTERVAL_SECONDS)); 191 | } 192 | } 193 | } 194 | -------------------------------------------------------------------------------- /pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | com.spotify 6 | foss-root 7 | 5 8 | 9 | 10 | 11 | 12 | hype-root 13 | 0.0.19-SNAPSHOT 14 | pom 15 | 16 | hype - hyper executor 17 | https://github.com/spotify/hype 18 | 19 | 20 | hype-caplet 21 | hype-common 22 | hype-docker 23 | hype-gcs 24 | hype-run 25 | hype-submitter 26 | hype-testing 27 | hype-submitter_2.11 28 | hype-submitter_2.12 29 | 30 | 31 | 32 | 0.0.9 33 | 1.0.3 34 | 0.11.1-beta 35 | 0.11.1-alpha 36 | 1.0.3 37 | 2.8.4 38 | 39 | 40 | 41 | 42 | The Apache Software License, Version 2.0 43 | http://www.apache.org/licenses/LICENSE-2.0.txt 44 | repo 45 | 46 | 47 | 48 | 49 | 50 | rouz 51 | rouz@spotify.com 52 | Rouzbeh Delavari 53 | 54 | 55 | rgruener 56 | robertg@spotify.com 57 | Robert Gruener 58 | 59 | 60 | yonromai 61 | romain@spotify.com 62 | Romain Yon 63 | 64 | 65 | 66 | 67 | https://github.com/spotify/hype 68 | scm:git:git@github.com:spotify/hype.git 69 | scm:git:git@github.com:spotify/hype.git 70 | HEAD 71 | 72 | 73 | 74 | 75 | 76 | org.jhades 77 | jhades 78 | 1.0.4 79 | 80 | 81 | 82 | co.paralleluniverse 83 | capsule 84 | ${capsule.version} 85 | 86 | 87 | co.paralleluniverse 88 | capsule-util 89 | ${capsule.version} 90 | 91 | 92 | 93 | com.google.cloud 94 | google-cloud-storage 95 | ${gcloud.version} 96 | 97 | 98 | com.google.cloud 99 | google-cloud-nio 100 | ${gcloud-alpha.version} 101 | 102 | 103 | com.google.apis 104 | google-api-services-container 105 | v1-rev10-1.22.0 106 | 107 | 108 | 109 | com.spotify 110 | docker-client 111 | 8.3.1 112 | 113 | 114 | io.fabric8 115 | kubernetes-client 116 | 2.2.13 117 | 118 | 119 | 120 | com.fasterxml.jackson.core 121 | jackson-core 122 | 2.8.3 123 | 124 | 125 | com.fasterxml.jackson.core 126 | jackson-annotations 127 | 2.8.3 128 | 129 | 130 | com.fasterxml.jackson.core 131 | jackson-databind 132 | 2.8.3 133 | 134 | 135 | 136 | com.google.guava 137 | guava 138 | 19.0 139 | 140 | 141 | com.google.auto.value 142 | auto-value 143 | 1.3 144 | provided 145 | 146 | 147 | io.norberg 148 | auto-matter 149 | 0.13.3 150 | provided 151 | 152 | 153 | commons-lang 154 | commons-lang 155 | 2.6 156 | 157 | 158 | org.fusesource.jansi 159 | jansi 160 | 1.11 161 | 162 | 163 | org.slf4j 164 | slf4j-api 165 | 1.7.21 166 | 167 | 168 | ch.qos.logback 169 | logback-classic 170 | 1.2.3 171 | 172 | 173 | 174 | junit 175 | junit 176 | 4.12 177 | test 178 | 179 | 180 | org.hamcrest 181 | hamcrest-all 182 | 1.3 183 | test 184 | 185 | 186 | org.mockito 187 | mockito-all 188 | 1.10.19 189 | test 190 | 191 | 192 | com.github.npathai 193 | hamcrest-optional 194 | 1.0 195 | test 196 | 197 | 198 | 199 | 200 | 201 | 202 | ch.qos.logback 203 | logback-classic 204 | test 205 | 206 | 207 | 208 | 209 | 210 | 211 | maven-failsafe-plugin 212 | 213 | 214 | org.jacoco 215 | jacoco-maven-plugin 216 | 0.7.9 217 | 218 | 219 | 220 | prepare-agent 221 | 222 | 223 | 224 | report 225 | test 226 | 227 | report 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | release 238 | 239 | 240 | 241 | net.alchim31.maven 242 | scala-maven-plugin 243 | 3.2.1 244 | 245 | 246 | attach-docs-and-sources 247 | 248 | add-source 249 | doc-jar 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | -------------------------------------------------------------------------------- /hype-common/src/main/java/com/spotify/hype/FluentBackoff.java: -------------------------------------------------------------------------------- 1 | /* 2 | * -\-\- 3 | * Copyright (C) 2016 Google Inc. 4 | * 5 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 6 | * use this file except in compliance with the License. You may obtain a copy of 7 | * the License at 8 | * 9 | * http://www.apache.org/licenses/LICENSE-2.0 10 | * 11 | * Unless required by applicable law or agreed to in writing, software 12 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 13 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 14 | * License for the specific language governing permissions and limitations under 15 | * the License. 16 | * -/-/- 17 | */ 18 | 19 | package com.spotify.hype; 20 | 21 | import static com.google.common.base.Preconditions.checkArgument; 22 | 23 | import com.google.api.client.util.BackOff; 24 | import com.google.common.base.MoreObjects; 25 | import org.joda.time.Duration; 26 | 27 | /** 28 | * A fluent builder for {@link BackOff} objects that allows customization of the retry algorithm. 29 | * 30 | * @see #DEFAULT for the default configuration parameters. 31 | * 32 | * NOTICE: Copied from from https://github.com/GoogleCloudPlatform/DataflowJavaSDK 33 | */ 34 | public final class FluentBackoff { 35 | 36 | private static final double DEFAULT_EXPONENT = 1.5; 37 | private static final double DEFAULT_RANDOMIZATION_FACTOR = 0.5; 38 | private static final Duration DEFAULT_MIN_BACKOFF = Duration.standardSeconds(1); 39 | private static final Duration DEFAULT_MAX_BACKOFF = Duration.standardDays(1000); 40 | private static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE; 41 | private static final Duration DEFAULT_MAX_CUM_BACKOFF = Duration.standardDays(1000); 42 | 43 | private final double exponent; 44 | private final Duration initialBackoff; 45 | private final Duration maxBackoff; 46 | private final Duration maxCumulativeBackoff; 47 | private final int maxRetries; 48 | 49 | /** 50 | * By default the {@link BackOff} created by this builder will use exponential backoff (base 51 | * exponent 1.5) with an initial backoff of 1 second. These parameters can be overridden with 52 | * {@link #withExponent(double)} and {@link #withInitialBackoff(Duration)}, 53 | * respectively, and the maximum backoff after exponential increase can be capped using {@link 54 | * FluentBackoff#withMaxBackoff(Duration)}. 55 | * 56 | *

The default {@link BackOff} does not limit the number of retries. To limit the backoff, the 57 | * maximum total number of retries can be set using {@link #withMaxRetries(int)}. The 58 | * total time spent in backoff can be time-bounded as well by configuring {@link 59 | * #withMaxCumulativeBackoff(Duration)}. If either of these limits are reached, calls 60 | * to {@link BackOff#nextBackOffMillis()} will return {@link BackOff#STOP} to signal that no more 61 | * retries should continue. 62 | */ 63 | public static final FluentBackoff DEFAULT = new FluentBackoff( 64 | DEFAULT_EXPONENT, 65 | DEFAULT_MIN_BACKOFF, DEFAULT_MAX_BACKOFF, DEFAULT_MAX_CUM_BACKOFF, 66 | DEFAULT_MAX_RETRIES); 67 | 68 | /** 69 | * Instantiates a {@link BackOff} that will obey the current configuration. 70 | * 71 | * @see FluentBackoff 72 | */ 73 | public BackOff backoff() { 74 | return new BackoffImpl(this); 75 | } 76 | 77 | /** 78 | * Returns a copy of this {@link FluentBackoff} that instead uses the specified exponent to 79 | * control the exponential growth of delay. 80 | * 81 | *

Does not modify this object. 82 | * 83 | * @see FluentBackoff 84 | */ 85 | public FluentBackoff withExponent(double exponent) { 86 | checkArgument(exponent > 0, "exponent %s must be greater than 0", exponent); 87 | return new FluentBackoff( 88 | exponent, initialBackoff, maxBackoff, maxCumulativeBackoff, maxRetries); 89 | } 90 | 91 | /** 92 | * Returns a copy of this {@link FluentBackoff} that instead uses the specified initial backoff 93 | * duration. 94 | * 95 | *

Does not modify this object. 96 | * 97 | * @see FluentBackoff 98 | */ 99 | public FluentBackoff withInitialBackoff(Duration initialBackoff) { 100 | checkArgument( 101 | initialBackoff.isLongerThan(Duration.ZERO), 102 | "initialBackoff %s must be at least 1 millisecond", 103 | initialBackoff); 104 | return new FluentBackoff( 105 | exponent, initialBackoff, maxBackoff, maxCumulativeBackoff, maxRetries); 106 | } 107 | 108 | /** 109 | * Returns a copy of this {@link FluentBackoff} that limits the maximum backoff of an individual 110 | * attempt to the specified duration. 111 | * 112 | *

Does not modify this object. 113 | * 114 | * @see FluentBackoff 115 | */ 116 | public FluentBackoff withMaxBackoff(Duration maxBackoff) { 117 | checkArgument( 118 | maxBackoff.getMillis() > 0, 119 | "maxBackoff %s must be at least 1 millisecond", 120 | maxBackoff); 121 | return new FluentBackoff( 122 | exponent, initialBackoff, maxBackoff, maxCumulativeBackoff, maxRetries); 123 | } 124 | 125 | /** 126 | * Returns a copy of this {@link FluentBackoff} that limits the total time spent in backoff 127 | * returned across all calls to {@link BackOff#nextBackOffMillis()}. 128 | * 129 | *

Does not modify this object. 130 | * 131 | * @see FluentBackoff 132 | */ 133 | public FluentBackoff withMaxCumulativeBackoff(Duration maxCumulativeBackoff) { 134 | checkArgument(maxCumulativeBackoff.isLongerThan(Duration.ZERO), 135 | "maxCumulativeBackoff %s must be at least 1 millisecond", maxCumulativeBackoff); 136 | return new FluentBackoff( 137 | exponent, initialBackoff, maxBackoff, maxCumulativeBackoff, maxRetries); 138 | } 139 | 140 | /** 141 | * Returns a copy of this {@link FluentBackoff} that limits the total number of retries, aka 142 | * the total number of calls to {@link BackOff#nextBackOffMillis()} before returning 143 | * {@link BackOff#STOP}. 144 | * 145 | *

Does not modify this object. 146 | * 147 | * @see FluentBackoff 148 | */ 149 | public FluentBackoff withMaxRetries(int maxRetries) { 150 | checkArgument(maxRetries >= 0, "maxRetries %s cannot be negative", maxRetries); 151 | return new FluentBackoff( 152 | exponent, initialBackoff, maxBackoff, maxCumulativeBackoff, maxRetries); 153 | } 154 | 155 | public String toString() { 156 | return MoreObjects.toStringHelper(FluentBackoff.class) 157 | .add("exponent", exponent) 158 | .add("initialBackoff", initialBackoff) 159 | .add("maxBackoff", maxBackoff) 160 | .add("maxRetries", maxRetries) 161 | .add("maxCumulativeBackoff", maxCumulativeBackoff) 162 | .toString(); 163 | } 164 | 165 | private static class BackoffImpl implements BackOff { 166 | 167 | // Customization of this backoff. 168 | private final FluentBackoff backoffConfig; 169 | // Current state 170 | private Duration currentCumulativeBackoff; 171 | private int currentRetry; 172 | 173 | @Override 174 | public void reset() { 175 | currentRetry = 0; 176 | currentCumulativeBackoff = Duration.ZERO; 177 | } 178 | 179 | @Override 180 | public long nextBackOffMillis() { 181 | // Maximum number of retries reached. 182 | if (currentRetry >= backoffConfig.maxRetries) { 183 | return BackOff.STOP; 184 | } 185 | // Maximum cumulative backoff reached. 186 | if (currentCumulativeBackoff.compareTo(backoffConfig.maxCumulativeBackoff) >= 0) { 187 | return BackOff.STOP; 188 | } 189 | 190 | double currentIntervalMillis = 191 | Math.min( 192 | backoffConfig.initialBackoff.getMillis() 193 | * Math.pow(backoffConfig.exponent, currentRetry), 194 | backoffConfig.maxBackoff.getMillis()); 195 | double randomOffset = 196 | (Math.random() * 2 - 1) * DEFAULT_RANDOMIZATION_FACTOR * currentIntervalMillis; 197 | long nextBackoffMillis = Math.round(currentIntervalMillis + randomOffset); 198 | // Cap to limit on cumulative backoff 199 | Duration remainingCumulative = 200 | backoffConfig.maxCumulativeBackoff.minus(currentCumulativeBackoff); 201 | nextBackoffMillis = Math.min(nextBackoffMillis, remainingCumulative.getMillis()); 202 | 203 | // Update state and return backoff. 204 | currentCumulativeBackoff = currentCumulativeBackoff.plus(nextBackoffMillis); 205 | currentRetry += 1; 206 | return nextBackoffMillis; 207 | } 208 | 209 | private BackoffImpl(FluentBackoff backoffConfig) { 210 | this.backoffConfig = backoffConfig; 211 | this.reset(); 212 | } 213 | 214 | public String toString() { 215 | return MoreObjects.toStringHelper(BackoffImpl.class) 216 | .add("backoffConfig", backoffConfig) 217 | .add("currentRetry", currentRetry) 218 | .add("currentCumulativeBackoff", currentCumulativeBackoff) 219 | .toString(); 220 | } 221 | } 222 | 223 | private FluentBackoff( 224 | double exponent, Duration initialBackoff, Duration maxBackoff, Duration maxCumulativeBackoff, 225 | int maxRetries) { 226 | this.exponent = exponent; 227 | this.initialBackoff = initialBackoff; 228 | this.maxBackoff = maxBackoff; 229 | this.maxRetries = maxRetries; 230 | this.maxCumulativeBackoff = maxCumulativeBackoff; 231 | } 232 | } 233 | -------------------------------------------------------------------------------- /hype-submitter/src/test/java/com/spotify/hype/runner/KubernetesDockerRunnerTest.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.runner; 22 | 23 | import static com.spotify.hype.model.ResourceRequest.CPU; 24 | import static com.spotify.hype.model.ResourceRequest.MEMORY; 25 | import static com.spotify.hype.model.RunEnvironment.environment; 26 | import static com.spotify.hype.model.RunEnvironment.fromYaml; 27 | import static com.spotify.hype.runner.KubernetesDockerRunner.EXECUTION_ID; 28 | import static com.spotify.hype.runner.KubernetesDockerRunner.HYPE_RUN; 29 | import static org.hamcrest.Matchers.contains; 30 | import static org.hamcrest.Matchers.hasEntry; 31 | import static org.hamcrest.Matchers.hasItems; 32 | import static org.hamcrest.Matchers.is; 33 | import static org.junit.Assert.assertThat; 34 | 35 | import com.spotify.hype.gcs.RunManifest; 36 | import com.spotify.hype.gcs.RunManifestBuilder; 37 | import com.spotify.hype.model.RunEnvironment; 38 | import com.spotify.hype.model.Secret; 39 | import com.spotify.hype.model.StagedContinuation; 40 | import io.fabric8.kubernetes.api.model.Container; 41 | import io.fabric8.kubernetes.api.model.EnvVar; 42 | import io.fabric8.kubernetes.api.model.EnvVarBuilder; 43 | import io.fabric8.kubernetes.api.model.Pod; 44 | import io.fabric8.kubernetes.api.model.PodSpec; 45 | import io.fabric8.kubernetes.api.model.Quantity; 46 | import io.fabric8.kubernetes.api.model.ResourceRequirements; 47 | import io.fabric8.kubernetes.api.model.VolumeBuilder; 48 | import io.fabric8.kubernetes.api.model.VolumeMountBuilder; 49 | import io.fabric8.kubernetes.client.KubernetesClient; 50 | import java.nio.file.Path; 51 | import java.nio.file.Paths; 52 | import org.junit.Before; 53 | import org.junit.Rule; 54 | import org.junit.Test; 55 | import org.junit.rules.ExpectedException; 56 | import org.mockito.Mockito; 57 | 58 | public class KubernetesDockerRunnerTest { 59 | 60 | private static final Secret SECRET = Secret.secret("keys", "/etc/keys"); 61 | private static final Path MANIFEST_PATH = Paths.get("/etc/manifest.txt"); 62 | private static final RunManifest MANIFEST = new RunManifestBuilder() 63 | .continuation("foobar.bin") 64 | .build(); 65 | 66 | @Rule 67 | public ExpectedException expect = ExpectedException.none(); 68 | 69 | private KubernetesDockerRunner runner; 70 | 71 | @Before 72 | public void setUp() throws Exception { 73 | KubernetesClient client = Mockito.mock(KubernetesClient.class); 74 | VolumeRepository volumeRepository = Mockito.mock(VolumeRepository.class); 75 | runner = new KubernetesDockerRunner(client, volumeRepository); 76 | } 77 | 78 | @Test 79 | public void setsManifestAsArgument() throws Exception { 80 | RunEnvironment env = environment(); 81 | Pod pod = createPod(env); 82 | 83 | Container container = findHypeRunContainer(pod); 84 | assertThat(container.getArgs(), contains(MANIFEST_PATH.toUri().toString())); 85 | } 86 | 87 | @Test 88 | public void setsManifestAsArgumentWhenLoadedFromYaml() throws Exception { 89 | RunEnvironment env = fromYaml("/minimal-pod.yaml"); 90 | Pod pod = createPod(env); 91 | 92 | Container container = findHypeRunContainer(pod); 93 | assertThat(container.getArgs(), contains(MANIFEST_PATH.toUri().toString())); 94 | } 95 | 96 | @Test 97 | public void setsHypeExecIdAsEnvVar() throws Exception { 98 | RunEnvironment env = environment(); 99 | Pod pod = createPod(env); 100 | String name = pod.getMetadata().getName(); 101 | 102 | Container container = findHypeRunContainer(pod); 103 | assertThat(container.getEnv(), hasItems(envVar(EXECUTION_ID, name))); 104 | } 105 | 106 | @Test 107 | public void setsHypeExecIdAsEnvVarWhenLoadedFromYaml() throws Exception { 108 | RunEnvironment env = fromYaml("/minimal-pod.yaml"); 109 | Pod pod = createPod(env); 110 | String name = pod.getMetadata().getName(); 111 | 112 | Container container = findHypeRunContainer(pod); 113 | assertThat(container.getEnv(), hasItems(envVar(EXECUTION_ID, name))); 114 | } 115 | 116 | @Test 117 | public void setsResourceRequests() throws Exception { 118 | RunEnvironment env = environment() 119 | .withRequest(CPU.of("250m")) 120 | .withRequest(MEMORY.of("2Gi")) 121 | .withRequest("gpu", "2"); 122 | Pod pod = createPod(env); 123 | 124 | Container container = findHypeRunContainer(pod); 125 | ResourceRequirements resources = container.getResources(); 126 | assertThat(resources.getRequests(), hasEntry("cpu", new Quantity("250m"))); 127 | assertThat(resources.getRequests(), hasEntry("memory", new Quantity("2Gi"))); 128 | assertThat(resources.getRequests(), hasEntry("gpu", new Quantity("2"))); 129 | } 130 | 131 | @Test 132 | public void addsResourceRequestsWhenLoadedFromYaml() throws Exception { 133 | RunEnvironment env = fromYaml("/minimal-pod.yaml") 134 | .withRequest(MEMORY.of("2Gi")) 135 | .withRequest("gpu", "2"); 136 | Pod pod = createPod(env); 137 | 138 | Container container = findHypeRunContainer(pod); 139 | ResourceRequirements resources = container.getResources(); 140 | assertThat(resources.getRequests(), hasEntry("cpu", new Quantity("100m"))); 141 | assertThat(resources.getRequests(), hasEntry("memory", new Quantity("2Gi"))); 142 | assertThat(resources.getRequests(), hasEntry("gpu", new Quantity("2"))); 143 | } 144 | 145 | @Test 146 | public void overridesResourceRequestsWhenLoadedFromYaml() throws Exception { 147 | RunEnvironment env = fromYaml("/minimal-pod.yaml") 148 | .withRequest(CPU.of("250m")); 149 | Pod pod = createPod(env); 150 | 151 | Container container = findHypeRunContainer(pod); 152 | ResourceRequirements resources = container.getResources(); 153 | assertThat(resources.getRequests(), hasEntry("cpu", new Quantity("250m"))); 154 | } 155 | 156 | @Test(expected=RuntimeException.class) 157 | public void forbidImageInYaml() throws Exception { 158 | RunEnvironment env = fromYaml("/with-image.yaml"); 159 | Pod pod = createPod(env); 160 | } 161 | 162 | @Test 163 | public void mountsSecretVolume() throws Exception { 164 | RunEnvironment env = environment() 165 | .withSecret(SECRET); 166 | Pod pod = createPod(env); 167 | 168 | final PodSpec spec = pod.getSpec(); 169 | assertThat(spec.getVolumes(), hasItems(new VolumeBuilder() 170 | .withName(SECRET.name()) 171 | .withNewSecret() 172 | .withSecretName(SECRET.name()) 173 | .endSecret() 174 | .build())); 175 | 176 | Container container = findHypeRunContainer(pod); 177 | assertThat(container.getVolumeMounts(), hasItems(new VolumeMountBuilder() 178 | .withName(SECRET.name()) 179 | .withMountPath(SECRET.mountPath()) 180 | .withReadOnly(true) 181 | .build())); 182 | } 183 | 184 | @Test 185 | public void mountsSecretVolumeWhenLoadedFromYaml() throws Exception { 186 | RunEnvironment env = fromYaml("/minimal-pod.yaml") 187 | .withSecret(SECRET); 188 | Pod pod = createPod(env); 189 | 190 | final PodSpec spec = pod.getSpec(); 191 | assertThat(spec.getVolumes(), hasItems(new VolumeBuilder() 192 | .withName(SECRET.name()) 193 | .withNewSecret() 194 | .withSecretName(SECRET.name()) 195 | .endSecret() 196 | .build())); 197 | 198 | Container container = findHypeRunContainer(pod); 199 | assertThat(container.getVolumeMounts(), hasItems(new VolumeMountBuilder() 200 | .withName(SECRET.name()) 201 | .withMountPath(SECRET.mountPath()) 202 | .withReadOnly(true) 203 | .build())); 204 | } 205 | 206 | @Test 207 | public void loadsFromYaml() throws Exception { 208 | RunEnvironment env = fromYaml("/minimal-pod.yaml"); 209 | Pod pod = createPod(env); 210 | 211 | assertThat(pod.getSpec().getRestartPolicy(), is("Never")); 212 | 213 | Container container = findHypeRunContainer(pod); 214 | assertThat(container.getImage(), is("busybox:1")); 215 | assertThat(container.getImagePullPolicy(), is("Always")); 216 | assertThat(container.getEnv(), hasItems(envVar("EXAMPLE", "my-env-value"))); 217 | 218 | ResourceRequirements resources = container.getResources(); 219 | assertThat(resources.getRequests(), hasEntry("cpu", new Quantity("100m"))); 220 | assertThat(resources.getLimits(), hasEntry("memory", new Quantity("1Gi"))); 221 | } 222 | 223 | private Pod createPod(RunEnvironment env) { 224 | StagedContinuation cont = StagedContinuation.stagedContinuation(MANIFEST_PATH, MANIFEST); 225 | RunSpec runSpec = RunSpec.runSpec(env, cont, "busybox:1"); 226 | 227 | return runner.createPod(runSpec); 228 | } 229 | 230 | private EnvVar envVar(String name, String value) { 231 | return new EnvVarBuilder() 232 | .withName(name) 233 | .withValue(value) 234 | .build(); 235 | } 236 | 237 | private Container findHypeRunContainer(Pod pod) { 238 | try { 239 | return KubernetesDockerRunner.findHypeRunContainer(pod); 240 | } catch (Exception e) { 241 | throw new AssertionError(HYPE_RUN + " container missing in Pod spec", e); 242 | } 243 | } 244 | } 245 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/Submitter.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * hype-submitter 4 | * -- 5 | * Copyright (C) 2016 - 2017 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype; 22 | 23 | import static com.google.common.base.Preconditions.checkNotNull; 24 | import static com.google.common.base.Preconditions.checkState; 25 | import static com.google.common.io.Files.getNameWithoutExtension; 26 | import static com.spotify.hype.ClasspathInspector.forLoader; 27 | import static com.spotify.hype.model.StagedContinuation.stagedContinuation; 28 | import static com.spotify.hype.runner.RunSpec.runSpec; 29 | import static com.spotify.hype.util.Util.randomAlphaNumeric; 30 | import static java.nio.file.Files.newInputStream; 31 | import static java.util.stream.Collectors.toList; 32 | 33 | import com.spotify.docker.client.DockerClient; 34 | import com.spotify.hype.gcs.RunManifest; 35 | import com.spotify.hype.gcs.RunManifestBuilder; 36 | import com.spotify.hype.gcs.StagingUtil; 37 | import com.spotify.hype.gcs.StagingUtil.StagedPackage; 38 | import com.spotify.hype.model.ContainerEngineCluster; 39 | import com.spotify.hype.model.DockerCluster; 40 | import com.spotify.hype.model.RunEnvironment; 41 | import com.spotify.hype.model.StagedContinuation; 42 | import com.spotify.hype.runner.DockerRunner; 43 | import com.spotify.hype.runner.KubernetesDockerRunner; 44 | import com.spotify.hype.runner.RunSpec; 45 | import com.spotify.hype.runner.VolumeRepository; 46 | import com.spotify.hype.util.Fn; 47 | import com.spotify.hype.util.SerializationUtil; 48 | import io.fabric8.kubernetes.client.KubernetesClient; 49 | import java.io.Closeable; 50 | import java.io.File; 51 | import java.io.IOException; 52 | import java.io.InputStream; 53 | import java.net.URI; 54 | import java.nio.file.Files; 55 | import java.nio.file.Path; 56 | import java.nio.file.Paths; 57 | import java.util.List; 58 | import java.util.Objects; 59 | import java.util.Optional; 60 | import org.slf4j.Logger; 61 | import org.slf4j.LoggerFactory; 62 | 63 | /** 64 | * todo: write explicit file list to gcs (allows for deduped and multi-use staging location) 65 | */ 66 | public class Submitter implements Closeable { 67 | 68 | private static final Logger LOG = LoggerFactory.getLogger(Submitter.class); 69 | 70 | private static final String STAGING_PREFIX = "spotify-hype-staging"; 71 | 72 | private final ClasspathInspector classpathInspector; 73 | private final URI stagingLocation; 74 | 75 | private final VolumeRepository volumeRepository; 76 | private final DockerRunner runner; 77 | 78 | public static Submitter createLocal() throws IOException { 79 | return Submitter.createLocal(DockerCluster.dockerCluster()); 80 | } 81 | 82 | public static Submitter createLocal(DockerCluster cluster) throws IOException { 83 | Path stagingLocation = new File(System.getProperty("user.home")).toPath() 84 | .resolve(".tmp") 85 | .resolve(STAGING_PREFIX); 86 | LOG.info("Local staging location is " + stagingLocation); 87 | Files.createDirectories(stagingLocation); 88 | final ClasspathInspector classpathInspector = forLoader(Submitter.class.getClassLoader()); 89 | return new Submitter(classpathInspector, stagingLocation.toString(), cluster); 90 | } 91 | 92 | public static Submitter create(String stagingLocation, ContainerEngineCluster cluster) { 93 | final ClasspathInspector classpathInspector = forLoader(Submitter.class.getClassLoader()); 94 | return create(classpathInspector, stagingLocation, cluster); 95 | } 96 | 97 | public static Submitter create(ClasspathInspector classpathInspector, 98 | String stagingLocation, 99 | ContainerEngineCluster cluster) { 100 | return new Submitter(classpathInspector, stagingLocation, cluster); 101 | } 102 | 103 | private URI getStagingURI(String stagingLocation) { 104 | checkNotNull(stagingLocation); 105 | URI uri = URI.create(stagingLocation); 106 | if (!uri.isAbsolute()) { 107 | return URI.create("file://" + uri.toString()); 108 | } else { 109 | return uri; 110 | } 111 | } 112 | 113 | private Submitter(ClasspathInspector classpathInspector, 114 | String stagingLocation, 115 | ContainerEngineCluster cluster) { 116 | this.stagingLocation = getStagingURI(stagingLocation); 117 | checkState(Objects.equals(this.stagingLocation.getScheme(), "gs")); 118 | this.classpathInspector = Objects.requireNonNull(classpathInspector); 119 | 120 | final KubernetesClient client = getClient(cluster); 121 | this.volumeRepository = new VolumeRepository(client); 122 | this.runner = DockerRunner.kubernetes(client, volumeRepository); 123 | } 124 | 125 | private Submitter(ClasspathInspector classpathInspector, 126 | String stagingLocation, 127 | DockerCluster cluster) { 128 | this.stagingLocation = getStagingURI(stagingLocation); 129 | if (!Objects.equals(this.stagingLocation.getScheme(), "file")) { 130 | LOG.warn("You are using non local staging location for local cluster"); 131 | } 132 | this.classpathInspector = Objects.requireNonNull(classpathInspector); 133 | 134 | this.volumeRepository = null; 135 | final DockerClient dockerClient = DockerRunner.createDockerClient(); 136 | this.runner = DockerRunner.local(dockerClient, cluster); 137 | } 138 | 139 | public T runOnCluster(Fn fn, RunEnvironment environment, String image) { 140 | // 1. stage 141 | final StagedContinuation stagedContinuation = stageContinuation(fn); 142 | 143 | // 2. submit and wait for k8s pod (returns return value uri, termination log, etc) 144 | final RunSpec runSpec = runSpec(environment, stagedContinuation, image); 145 | 146 | LOG.info("Submitting {} to {}", stagedContinuation.manifestPath().toUri(), environment); 147 | final Optional returnUri = runner.run(runSpec); 148 | 149 | // 3. download serialized return value 150 | if (returnUri.isPresent()) { 151 | final Path path = Paths.get(returnUri.get()); 152 | @SuppressWarnings("unchecked") 153 | final T returnValue; 154 | try (InputStream inputStream = newInputStream(path)) { 155 | // 4. deserialize and return 156 | //noinspection unchecked 157 | returnValue = (T) SerializationUtil.readObject(inputStream); 158 | } catch (IOException e) { 159 | throw new RuntimeException(e); 160 | } 161 | 162 | waitForDetach(environment); 163 | 164 | return returnValue; 165 | } else { 166 | throw new RuntimeException("Failed to get return value"); 167 | } 168 | } 169 | 170 | public StagedContinuation stageContinuation(Fn fn) { 171 | final List files = classpathInspector.classpathJars(); 172 | final Path continuationPath = SerializationUtil.serializeContinuation(fn); 173 | final Path manifestPath = Paths.get(this.stagingLocation) 174 | .resolve("manifest-" + randomAlphaNumeric(8) + ".txt"); 175 | final String continuationFileName = getNameWithoutExtension(continuationPath 176 | .toAbsolutePath().toString()); 177 | 178 | files.add(continuationPath.toAbsolutePath()); 179 | 180 | final List fileStrings = files.stream() 181 | .map(Path::toAbsolutePath) 182 | .map(Path::toString) 183 | .collect(toList()); 184 | 185 | final List stagedPackages = StagingUtil.stageClasspathElements( 186 | fileStrings, 187 | this.stagingLocation.toString()); 188 | 189 | final Optional stagedContinuationPackage = stagedPackages.stream() 190 | .filter(p -> p.name().contains(continuationFileName)) 191 | .findFirst(); 192 | 193 | if (!stagedContinuationPackage.isPresent()) { 194 | throw new RuntimeException(); 195 | } 196 | 197 | final URI uri = URI.create(stagedContinuationPackage.get().location()); 198 | final String cont = Paths.get(uri).getFileName().toString(); 199 | 200 | // todo: move manifest creation into StagingUtil 201 | final RunManifest manifest = new RunManifestBuilder() 202 | .continuation(cont) 203 | .classPathFiles(stagedPackages.stream().map(StagedPackage::name).collect(toList())) 204 | // todo: files 205 | .build(); 206 | try { 207 | RunManifest.write(manifest, manifestPath); 208 | } catch (IOException e) { 209 | throw new RuntimeException(e); 210 | } 211 | 212 | return stagedContinuation(manifestPath, manifest); 213 | } 214 | 215 | @Override 216 | public void close() throws IOException { 217 | if (volumeRepository != null) { 218 | volumeRepository.close(); 219 | } 220 | } 221 | 222 | /** 223 | * Hacky workaround for allowing k8s to detach any ReadWrite volumes from the nodes before 224 | * we continue to submit more pods. 225 | * 226 | *

A volume can be used in ReadOnly mode even when it is attached to a node in ReadWrite 227 | * mode. However, it can only be attached in ReadWrite mode to a single node. 228 | * 229 | *

If several pods requesting the same volume in ReadOnly mode are submitted too quickly 230 | * after it has been used in ReadWrite mode, they'll be able to use it immediately on the node 231 | * that has already has it attached as ReadWrite. All other nodes will have to wait for the 232 | * pods on that node to complete, and the node to detach the volume, before they are able to 233 | * attach the volume in ReadOnly mode. This leads to unnecessary contention between nodes and 234 | * reduces node-parallelism of submitted pods down to one node. 235 | */ 236 | private void waitForDetach(RunEnvironment environment) { 237 | if (runner instanceof KubernetesDockerRunner 238 | && environment.volumeMounts().stream().anyMatch(v -> !v.readOnly())) { 239 | try { 240 | Thread.sleep(10_000); 241 | } catch (InterruptedException ignore) { 242 | } 243 | } 244 | } 245 | 246 | private static KubernetesClient client; 247 | private static synchronized KubernetesClient getClient(ContainerEngineCluster cluster) { 248 | if (client == null) { 249 | client = DockerRunner.createKubernetesClient(cluster); 250 | } 251 | 252 | return client; 253 | } 254 | } 255 | -------------------------------------------------------------------------------- /hype-gcs/src/main/java/com/spotify/hype/gcs/ZipFiles.java: -------------------------------------------------------------------------------- 1 | /* 2 | * -\-\- 3 | * Copyright (C) 2015 Google Inc. 4 | * 5 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 6 | * use this file except in compliance with the License. You may obtain a copy of 7 | * the License at 8 | * 9 | * http://www.apache.org/licenses/LICENSE-2.0 10 | * 11 | * Unless required by applicable law or agreed to in writing, software 12 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 13 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 14 | * License for the specific language governing permissions and limitations under 15 | * the License. 16 | * -/-/- 17 | */ 18 | 19 | package com.spotify.hype.gcs; 20 | 21 | import static com.google.common.base.Preconditions.checkArgument; 22 | import static com.google.common.base.Preconditions.checkNotNull; 23 | 24 | import com.google.common.collect.FluentIterable; 25 | import com.google.common.collect.Iterators; 26 | import com.google.common.io.ByteSource; 27 | import com.google.common.io.CharSource; 28 | import com.google.common.io.Closer; 29 | import com.google.common.io.Files; 30 | import java.io.BufferedOutputStream; 31 | import java.io.File; 32 | import java.io.FileOutputStream; 33 | import java.io.IOException; 34 | import java.io.InputStream; 35 | import java.io.OutputStream; 36 | import java.nio.charset.Charset; 37 | import java.util.Arrays; 38 | import java.util.Iterator; 39 | import java.util.zip.ZipEntry; 40 | import java.util.zip.ZipFile; 41 | import java.util.zip.ZipOutputStream; 42 | 43 | /** 44 | * Functions for zipping a directory (including a subdirectory) into a ZIP-file 45 | * or unzipping it again. 46 | * 47 | * NOTICE: Copied from https://github.com/GoogleCloudPlatform/DataflowJavaSDK 48 | */ 49 | public final class ZipFiles { 50 | private ZipFiles() {} 51 | 52 | /** 53 | * Returns a new {@link ByteSource} for reading the contents of the given 54 | * entry in the given zip file. 55 | */ 56 | static ByteSource asByteSource(ZipFile file, ZipEntry entry) { 57 | return new ZipEntryByteSource(file, entry); 58 | } 59 | 60 | /** 61 | * Returns a new {@link CharSource} for reading the contents of the given 62 | * entry in the given zip file as text using the given charset. 63 | */ 64 | static CharSource asCharSource( 65 | ZipFile file, ZipEntry entry, Charset charset) { 66 | return asByteSource(file, entry).asCharSource(charset); 67 | } 68 | 69 | private static final class ZipEntryByteSource extends ByteSource { 70 | 71 | private final ZipFile file; 72 | private final ZipEntry entry; 73 | 74 | ZipEntryByteSource(ZipFile file, ZipEntry entry) { 75 | this.file = checkNotNull(file); 76 | this.entry = checkNotNull(entry); 77 | } 78 | 79 | @Override 80 | public InputStream openStream() throws IOException { 81 | return file.getInputStream(entry); 82 | } 83 | 84 | // TODO: implement size() to try calling entry.getSize()? 85 | 86 | @Override 87 | public String toString() { 88 | return "ZipFiles.asByteSource(" + file + ", " + entry + ")"; 89 | } 90 | } 91 | 92 | /** 93 | * Returns a {@link FluentIterable} of all the entries in the given zip file. 94 | */ 95 | // unmodifiable Iterator can be safely cast 96 | // to Iterator 97 | @SuppressWarnings("unchecked") 98 | static FluentIterable entries(final ZipFile file) { 99 | checkNotNull(file); 100 | return new FluentIterable() { 101 | @Override 102 | public Iterator iterator() { 103 | return (Iterator) Iterators.forEnumeration(file.entries()); 104 | } 105 | }; 106 | } 107 | 108 | /** 109 | * Unzips the zip file specified by the path and creates the directory structure inside 110 | * the target directory. Refuses to unzip files that refer to a parent directory, for security 111 | * reasons. 112 | * 113 | * @param zipFile the source zip-file to unzip 114 | * @param targetDirectory the directory to unzip to. If the zip-file contains 115 | * any subdirectories, they will be created within our target directory. 116 | * @throws IOException the unzipping failed, e.g. because the output was not writable, the {@code 117 | * zipFile} was not readable, or contains an illegal entry (contains "..", pointing outside 118 | * the target directory) 119 | * @throws IllegalArgumentException the target directory is not a valid directory (e.g. does not 120 | * exist, or is a file instead of a directory) 121 | */ 122 | static void unzipFile( 123 | File zipFile, 124 | File targetDirectory) throws IOException { 125 | checkNotNull(zipFile); 126 | checkNotNull(targetDirectory); 127 | checkArgument( 128 | targetDirectory.isDirectory(), 129 | "%s is not a valid directory", 130 | targetDirectory.getAbsolutePath()); 131 | final ZipFile zipFileObj = new ZipFile(zipFile); 132 | try { 133 | for (ZipEntry entry : entries(zipFileObj)) { 134 | checkName(entry.getName()); 135 | File targetFile = new File(targetDirectory, entry.getName()); 136 | if (entry.isDirectory()) { 137 | if (!targetFile.isDirectory() && !targetFile.mkdirs()) { 138 | throw new IOException( 139 | "Failed to create directory: " + targetFile.getAbsolutePath()); 140 | } 141 | } else { 142 | File parentFile = targetFile.getParentFile(); 143 | if (!parentFile.isDirectory()) { 144 | if (!parentFile.mkdirs()) { 145 | throw new IOException( 146 | "Failed to create directory: " 147 | + parentFile.getAbsolutePath()); 148 | } 149 | } 150 | // Write the file to the destination. 151 | asByteSource(zipFileObj, entry).copyTo(Files.asByteSink(targetFile)); 152 | } 153 | } 154 | } finally { 155 | zipFileObj.close(); 156 | } 157 | } 158 | 159 | /** 160 | * Checks that the given entry name is legal for unzipping: if it contains 161 | * ".." as a name element, it could cause the entry to be unzipped outside 162 | * the directory we're unzipping to. 163 | * 164 | * @throws IOException if the name is illegal 165 | */ 166 | private static void checkName(String name) throws IOException { 167 | // First just check whether the entry name string contains "..". 168 | // This should weed out the the vast majority of entries, which will not 169 | // contain "..". 170 | if (name.contains("..")) { 171 | // If the string does contain "..", break it down into its actual name 172 | // elements to ensure it actually contains ".." as a name, not just a 173 | // name like "foo..bar" or even "foo..", which should be fine. 174 | File file = new File(name); 175 | while (file != null) { 176 | if (file.getName().equals("..")) { 177 | throw new IOException("Cannot unzip file containing an entry with " 178 | + "\"..\" in the name: " + name); 179 | } 180 | file = file.getParentFile(); 181 | } 182 | } 183 | } 184 | 185 | /** 186 | * Zips an entire directory specified by the path. 187 | * 188 | * @param sourceDirectory the directory to read from. This directory and all 189 | * subdirectories will be added to the zip-file. The path within the zip 190 | * file is relative to the directory given as parameter, not absolute. 191 | * @param zipFile the zip-file to write to. 192 | * @throws IOException the zipping failed, e.g. because the input was not 193 | * readable. 194 | */ 195 | static void zipDirectory( 196 | File sourceDirectory, 197 | File zipFile) throws IOException { 198 | checkNotNull(sourceDirectory); 199 | checkNotNull(zipFile); 200 | checkArgument( 201 | sourceDirectory.isDirectory(), 202 | "%s is not a valid directory", 203 | sourceDirectory.getAbsolutePath()); 204 | checkArgument( 205 | !zipFile.exists(), 206 | "%s does already exist, files are not being overwritten", 207 | zipFile.getAbsolutePath()); 208 | Closer closer = Closer.create(); 209 | try { 210 | OutputStream outputStream = closer.register(new BufferedOutputStream( 211 | new FileOutputStream(zipFile))); 212 | zipDirectory(sourceDirectory, outputStream); 213 | } catch (Throwable t) { 214 | throw closer.rethrow(t); 215 | } finally { 216 | closer.close(); 217 | } 218 | } 219 | 220 | /** 221 | * Zips an entire directory specified by the path. 222 | * 223 | * @param sourceDirectory the directory to read from. This directory and all 224 | * subdirectories will be added to the zip-file. The path within the zip 225 | * file is relative to the directory given as parameter, not absolute. 226 | * @param outputStream the stream to write the zip-file to. This method does not close 227 | * outputStream. 228 | * @throws IOException the zipping failed, e.g. because the input was not 229 | * readable. 230 | */ 231 | static void zipDirectory( 232 | File sourceDirectory, 233 | OutputStream outputStream) throws IOException { 234 | checkNotNull(sourceDirectory); 235 | checkNotNull(outputStream); 236 | checkArgument( 237 | sourceDirectory.isDirectory(), 238 | "%s is not a valid directory", 239 | sourceDirectory.getAbsolutePath()); 240 | ZipOutputStream zos = new ZipOutputStream(outputStream); 241 | for (File file : sourceDirectory.listFiles()) { 242 | zipDirectoryInternal(file, "", zos); 243 | } 244 | zos.finish(); 245 | } 246 | 247 | /** 248 | * Private helper function for zipping files. This one goes recursively 249 | * through the input directory and all of its subdirectories and adds the 250 | * single zip entries. 251 | * 252 | * @param inputFile the file or directory to be added to the zip file 253 | * @param directoryName the string-representation of the parent directory 254 | * name. Might be an empty name, or a name containing multiple directory 255 | * names separated by "/". The directory name must be a valid name 256 | * according to the file system limitations. The directory name should be 257 | * empty or should end in "/". 258 | * @param zos the zipstream to write to 259 | * @throws IOException the zipping failed, e.g. because the output was not 260 | * writeable. 261 | */ 262 | private static void zipDirectoryInternal( 263 | File inputFile, 264 | String directoryName, 265 | ZipOutputStream zos) throws IOException { 266 | String entryName = directoryName + inputFile.getName(); 267 | if (inputFile.isDirectory()) { 268 | entryName += "/"; 269 | 270 | // We are hitting a sub-directory. Recursively add children to zip in deterministic, 271 | // sorted order. 272 | File[] childFiles = inputFile.listFiles(); 273 | if (childFiles.length > 0) { 274 | Arrays.sort(childFiles); 275 | // loop through the directory content, and zip the files 276 | for (File file : childFiles) { 277 | zipDirectoryInternal(file, entryName, zos); 278 | } 279 | 280 | // Since this directory has children, exit now without creating a zipentry specific to 281 | // this directory. The entry for a non-entry directory is incompatible with certain 282 | // implementations of unzip. 283 | return; 284 | } 285 | } 286 | 287 | // Put the zip-entry for this file or empty directory into the zipoutputstream. 288 | ZipEntry entry = new ZipEntry(entryName); 289 | entry.setTime(inputFile.lastModified()); 290 | zos.putNextEntry(entry); 291 | 292 | // Copy file contents into zipoutput stream. 293 | if (inputFile.isFile()) { 294 | Files.asByteSource(inputFile).copyTo(zos); 295 | } 296 | } 297 | } 298 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /hype-submitter/src/main/java/com/spotify/hype/runner/KubernetesDockerRunner.java: -------------------------------------------------------------------------------- 1 | /*- 2 | * -\-\- 3 | * Spotify Styx Scheduler Service 4 | * -- 5 | * Copyright (C) 2016 Spotify AB 6 | * -- 7 | * Licensed under the Apache License, Version 2.0 (the "License"); 8 | * you may not use this file except in compliance with the License. 9 | * You may obtain a copy of the License at 10 | * 11 | * http://www.apache.org/licenses/LICENSE-2.0 12 | * 13 | * Unless required by applicable law or agreed to in writing, software 14 | * distributed under the License is distributed on an "AS IS" BASIS, 15 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | * See the License for the specific language governing permissions and 17 | * limitations under the License. 18 | * -/-/- 19 | */ 20 | 21 | package com.spotify.hype.runner; 22 | 23 | import static com.spotify.hype.util.Util.randomAlphaNumeric; 24 | import static java.util.Collections.singletonList; 25 | import static java.util.function.Function.identity; 26 | import static java.util.stream.Collectors.toList; 27 | import static java.util.stream.Collectors.toMap; 28 | 29 | import com.fasterxml.jackson.databind.ObjectMapper; 30 | import com.fasterxml.jackson.dataformat.yaml.YAMLFactory; 31 | import com.google.api.client.util.BackOff; 32 | import com.google.api.client.util.Sleeper; 33 | import com.google.common.annotations.VisibleForTesting; 34 | import com.spotify.hype.FluentBackoff; 35 | import com.spotify.hype.model.RunEnvironment; 36 | import com.spotify.hype.model.Secret; 37 | import com.spotify.hype.model.StagedContinuation; 38 | import com.spotify.hype.model.VolumeMount; 39 | import com.spotify.hype.model.VolumeRequest; 40 | import io.fabric8.kubernetes.api.model.Container; 41 | import io.fabric8.kubernetes.api.model.ContainerBuilder; 42 | import io.fabric8.kubernetes.api.model.ContainerStatus; 43 | import io.fabric8.kubernetes.api.model.DoneablePod; 44 | import io.fabric8.kubernetes.api.model.EnvVarBuilder; 45 | import io.fabric8.kubernetes.api.model.ObjectMeta; 46 | import io.fabric8.kubernetes.api.model.PersistentVolumeClaim; 47 | import io.fabric8.kubernetes.api.model.Pod; 48 | import io.fabric8.kubernetes.api.model.PodBuilder; 49 | import io.fabric8.kubernetes.api.model.PodSpec; 50 | import io.fabric8.kubernetes.api.model.PodSpecBuilder; 51 | import io.fabric8.kubernetes.api.model.PodStatus; 52 | import io.fabric8.kubernetes.api.model.Quantity; 53 | import io.fabric8.kubernetes.api.model.ResourceRequirementsBuilder; 54 | import io.fabric8.kubernetes.api.model.Volume; 55 | import io.fabric8.kubernetes.api.model.VolumeBuilder; 56 | import io.fabric8.kubernetes.api.model.VolumeMountBuilder; 57 | import io.fabric8.kubernetes.client.KubernetesClient; 58 | import io.fabric8.kubernetes.client.KubernetesClientException; 59 | import io.fabric8.kubernetes.client.dsl.PodResource; 60 | import io.norberg.automatter.AutoMatter; 61 | import java.io.IOException; 62 | import java.net.URI; 63 | import java.nio.file.Path; 64 | import java.util.List; 65 | import java.util.Map; 66 | import java.util.Objects; 67 | import java.util.Optional; 68 | import java.util.concurrent.TimeUnit; 69 | 70 | /** 71 | * A {@link DockerRunner} implementation that submits container executions to a Kubernetes cluster. 72 | * 73 | * todo: retry docker operations 74 | * todo: clean up all pods on exit? 75 | */ 76 | public class KubernetesDockerRunner implements DockerRunner { 77 | 78 | static final String HYPE_RUN = "hype-run"; 79 | static final String EXECUTION_ID = "HYPE_EXECUTION_ID"; 80 | 81 | private static final ObjectMapper YAML_MAPPER = new ObjectMapper(new YAMLFactory()); 82 | private static final int POLL_PODS_INTERVAL_SECONDS = 5; 83 | 84 | private final KubernetesClient client; 85 | private final VolumeRepository volumeRepository; 86 | 87 | private static final FluentBackoff BACKOFF_FACTORY = 88 | FluentBackoff.DEFAULT; 89 | 90 | KubernetesDockerRunner(KubernetesClient client, VolumeRepository volumeRepository) { 91 | this.client = Objects.requireNonNull(client); 92 | this.volumeRepository = Objects.requireNonNull(volumeRepository); 93 | } 94 | 95 | @Override 96 | public Optional run(RunSpec runSpec) { 97 | Sleeper retrySleeper = Sleeper.DEFAULT; 98 | BackOff backoff = BACKOFF_FACTORY.backoff(); 99 | 100 | while (true) { 101 | try { 102 | final Pod pod = client.pods().create(createPod(runSpec)); 103 | final String podName = pod.getMetadata().getName(); 104 | LOG.info("Created pod {}", podName); 105 | 106 | Optional uri = blockUntilComplete(podName); 107 | client.pods().withName(podName).delete(); 108 | return uri; 109 | } catch (KubernetesClientException kce) { 110 | try { 111 | long sleep = backoff.nextBackOffMillis(); 112 | if (sleep == BackOff.STOP) { 113 | // Rethrow last error, to be included as a cause in the catch below. 114 | LOG.error("Failed to create Kubernetes pod", kce); 115 | throw new KubernetesClientException("Failed to create Kubernetes pod", kce); 116 | } else { 117 | LOG.warn("Kubernetes creation attempt failed, sleeping before retrying", kce); 118 | retrySleeper.sleep(sleep); 119 | } 120 | } catch (IOException | InterruptedException ioe) { 121 | throw new RuntimeException( 122 | String.format("Failed to create Kubernetes pod when trying to sleep: %s", ioe.getMessage()), ioe); 123 | } 124 | } catch (InterruptedException ie) { 125 | throw new RuntimeException("Interrupted while blocking", ie); 126 | } 127 | } 128 | } 129 | 130 | private Optional blockUntilComplete(final String podName) throws InterruptedException { 131 | LOG.debug("Checking running statuses"); 132 | 133 | boolean nodeAssigned = false; 134 | 135 | while (true) { 136 | final PodResource pod = client.pods().withName(podName); 137 | final PodStatus status = pod.get().getStatus(); 138 | 139 | if (!nodeAssigned && pod.get().getSpec().getNodeName() != null) { 140 | LOG.info("Pod {} assigned to node {}", podName, pod.get().getSpec().getNodeName()); 141 | nodeAssigned = true; 142 | } 143 | 144 | switch (status.getPhase()) { 145 | case "Succeeded": 146 | LOG.info("Kubernetes pod {} exited with status {}", podName, status.getPhase()); 147 | 148 | final Optional containerStatus = status.getContainerStatuses().stream() 149 | .filter(c -> HYPE_RUN.equals(c.getName())) 150 | .findFirst(); 151 | 152 | final Optional terminated = containerStatus 153 | .flatMap(s -> Optional.ofNullable(s.getState().getTerminated())) 154 | .flatMap(t -> Optional.ofNullable(t.getMessage())); 155 | 156 | if (terminated.isPresent()) { 157 | String message = terminated.get(); 158 | LOG.info("Got termination message: {}", message); 159 | return Optional.of(URI.create(message)); 160 | } 161 | break; 162 | 163 | case "Failed": 164 | LOG.info("Kubernetes pod {} failed with status {}", podName, status); 165 | return Optional.empty(); 166 | 167 | default: 168 | break; 169 | } 170 | Thread.sleep(TimeUnit.SECONDS.toMillis(POLL_PODS_INTERVAL_SECONDS)); 171 | } 172 | } 173 | 174 | @VisibleForTesting 175 | Pod createPod(RunSpec runSpec) { 176 | final String podName = HYPE_RUN + "-" + randomAlphaNumeric(8); 177 | final RunEnvironment env = runSpec.runEnvironment(); 178 | final List secrets = env.secretMounts(); 179 | final StagedContinuation stagedContinuation = runSpec.stagedContinuation(); 180 | final List volumeMountInfos = volumeMountInfos(env.volumeMounts()); 181 | 182 | final Pod basePod = getBasePod(env, runSpec.image()); 183 | 184 | // add metadata name 185 | final ObjectMeta metadata = basePod.getMetadata() != null 186 | ? basePod.getMetadata() 187 | : new ObjectMeta(); 188 | metadata.setName(podName); 189 | basePod.setMetadata(metadata); 190 | 191 | final PodSpec spec = basePod.getSpec(); 192 | 193 | // add volumes 194 | secrets.forEach(s -> 195 | spec.getVolumes() 196 | .add(new VolumeBuilder() 197 | .withName(s.name()) 198 | .withNewSecret() 199 | .withSecretName(s.name()) 200 | .endSecret() 201 | .build())); 202 | volumeMountInfos.stream() 203 | .map(VolumeMountInfo::volume) 204 | .forEach(spec.getVolumes()::add); 205 | 206 | final Container container = findHypeRunContainer(basePod); 207 | 208 | // add volume mounts 209 | secrets.forEach(s -> 210 | container.getVolumeMounts() 211 | .add(new VolumeMountBuilder() 212 | .withName(s.name()) 213 | .withMountPath(s.mountPath()) 214 | .withReadOnly(true) 215 | .build())); 216 | volumeMountInfos.stream() 217 | .map(VolumeMountInfo::volumeMount) 218 | .forEach(container.getVolumeMounts()::add); 219 | 220 | // set args 221 | if (container.getArgs().size() > 0) { 222 | LOG.warn("Overriding " + HYPE_RUN + " container args"); 223 | } 224 | container.setArgs(singletonList(stagedContinuation.manifestPath().toUri().toString())); 225 | 226 | // add env var 227 | container.getEnv() 228 | .add(new EnvVarBuilder() 229 | .withName(EXECUTION_ID) 230 | .withValue(podName) 231 | .build()); 232 | 233 | // add resource requests 234 | final ResourceRequirementsBuilder resourceReqsBuilder = container.getResources() != null 235 | ? new ResourceRequirementsBuilder( 236 | container.getResources()) 237 | : new ResourceRequirementsBuilder(); 238 | for (Map.Entry request : env.resourceRequests().entrySet()) { 239 | resourceReqsBuilder.addToRequests(request.getKey(), new Quantity(request.getValue())); 240 | } 241 | container.setResources(resourceReqsBuilder.build()); 242 | 243 | return basePod; 244 | } 245 | 246 | private Pod getBasePod(RunEnvironment env, String image) { 247 | 248 | if (env.yamlPath().isPresent()) { 249 | Path yamlPath = env.yamlPath().get(); 250 | final Pod pod; 251 | try { 252 | pod = YAML_MAPPER.readValue(yamlPath.toFile(), Pod.class); 253 | } catch (IOException e) { 254 | throw new RuntimeException("Failed to parse YAML file " + yamlPath, e); 255 | } 256 | 257 | // ensure image is not set 258 | final Container hypeRunContainer = findHypeRunContainer(pod); 259 | if (hypeRunContainer.getImage() != null) { 260 | throw new RuntimeException("Image on " + HYPE_RUN + " container must not be set"); 261 | } 262 | 263 | hypeRunContainer.setImage(image); 264 | 265 | return pod; 266 | } else { 267 | final Container container = new ContainerBuilder() 268 | .withName(HYPE_RUN) 269 | .withImage(image) 270 | .build(); 271 | 272 | final PodSpec spec = new PodSpecBuilder() 273 | .withRestartPolicy("Never") // todo: enable retries with max retry limit 274 | .addToContainers(container) 275 | .build(); 276 | 277 | return new PodBuilder() 278 | .withSpec(spec) 279 | .build(); 280 | } 281 | } 282 | 283 | private VolumeMountInfo volumeMountInfo(PersistentVolumeClaim claim, VolumeMount volumeMount) { 284 | final String claimName = claim.getMetadata().getName(); 285 | 286 | final Volume volume = new VolumeBuilder() 287 | .withName(claimName) 288 | .withNewPersistentVolumeClaim(claimName, volumeMount.readOnly()) 289 | .build(); 290 | 291 | final io.fabric8.kubernetes.api.model.VolumeMount mount = new VolumeMountBuilder() 292 | .withName(claimName) 293 | .withMountPath(volumeMount.mountPath()) 294 | .withReadOnly(volumeMount.readOnly()) 295 | .build(); 296 | 297 | final String ro = volumeMount.readOnly() ? "readOnly" : "readWrite"; 298 | LOG.info("Mounting {} {} at {}", claimName, ro, volumeMount.mountPath()); 299 | 300 | return new VolumeMountInfoBuilder() 301 | .persistentVolumeClaim(claim) 302 | .volume(volume) 303 | .volumeMount(mount) 304 | .build(); 305 | } 306 | 307 | private List volumeMountInfos(List volumeMounts) { 308 | final Map claims = volumeMounts.stream() 309 | .map(VolumeMount::volumeRequest) 310 | .distinct() 311 | .collect(toMap(identity(), volumeRepository::getClaim)); 312 | 313 | return volumeMounts.stream() 314 | .map(volumeMount -> volumeMountInfo(claims.get(volumeMount.volumeRequest()), volumeMount)) 315 | .collect(toList()); 316 | } 317 | 318 | @VisibleForTesting 319 | static Container findHypeRunContainer(Pod pod) { 320 | final List containers = pod.getSpec().getContainers(); 321 | final Optional hypeRunContainer = containers.stream() 322 | .filter(container -> HYPE_RUN.equals(container.getName())) 323 | .findFirst(); 324 | 325 | if (!hypeRunContainer.isPresent()) { 326 | throw new RuntimeException("Pod spec does not contain a container named '" + HYPE_RUN + "'"); 327 | } 328 | 329 | return hypeRunContainer.get(); 330 | } 331 | 332 | @AutoMatter 333 | interface VolumeMountInfo { 334 | PersistentVolumeClaim persistentVolumeClaim(); 335 | Volume volume(); 336 | io.fabric8.kubernetes.api.model.VolumeMount volumeMount(); 337 | } 338 | } 339 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [DEPRECATED] hype 2 | ==== 3 | PLEASE NOTE: THIS REPO HAS BEEN DEPRECATED BECAUSE IT IS NO LONGER USED BY ANY PROJECTS AT SPOTIFY AND THERE ARE NO PLANS TO CONTINUE DEVELOPMENT. 4 | 5 | IT WILL NOW BE ARCHIVED. 6 | 7 | .ed"""" """$$$$be. 8 | -" ^""**$$$e. 9 | ." '$$$c 10 | / "4$$b 11 | d 3 $$$$ 12 | $ * .$$$$$$ 13 | .$ ^c $$$$$e$$$$$$$$. 14 | d$L 4. 4$$$$$$$$$$$$$$b 15 | $$$$b ^ceeeee. 4$$ECL.F*$$$$$$$ 16 | e$""=. $$$$P d$$$$F $ $$$$$$$$$- $$$$$$ 17 | z$$b. ^c 3$$$F "$$$$b $"$$$$$$$ $$$$*" .=""$c 18 | 4$$$$L \ $$P" "$$b .$ $$$$$...e$$ .= e$$$. 19 | ^*$$$$$c %.. *c .. $$ 3$$$$$$$$$$eF zP d$$$$$ 20 | "**$$$ec "\ %ce"" $$$ $$$$$$$$$$* .r" =$$$$P"" 21 | "*$b. "c *$e. *** d$$$$$"L$$ .d" e$$***" 22 | ^*$$c ^$c $$$ 4J$$$$$% $$$ .e*".eeP" 23 | "$$$$$$"'$=e....$*$$**$cz$$" "..d$*" 24 | "*$$$ *=%4.$ L L$ P3$$$F $$$P" 25 | "$ "%*ebJLzb$e$$$$$b $P" 26 | %.. 4$$$$$$$$$$ " 27 | $$$e z$$$$$$$$$$% 28 | "*$c "$$$$$$$P" 29 | ."""*$$$$$$$$bc 30 | .-" .$***$$$"""*e. 31 | .-" .e$" "*$c ^*b. 32 | .=*"""" .e$*" "*bc "*$e.. 33 | .$" .z*" ^*$e. "*****e. 34 | $$ee$c .d" "*$. 3. 35 | ^*$E")$..$" * .ee==d% 36 | $.d$$$* * J$$$e* 37 | """"" "$$$" 38 | 39 | 40 | 41 | PREVIOUS DOCUMENTATION FOLLOWS 42 | 43 | [![Build Status](https://img.shields.io/circleci/project/github/spotify/hype/master.svg)](https://circleci.com/gh/spotify/hype) 44 | [![codecov.io](https://codecov.io/github/spotify/hype/coverage.svg?branch=master)](https://codecov.io/github/spotify/hype?branch=master) 45 | [![Maven Central](https://img.shields.io/maven-central/v/com.spotify/hype-root.svg)](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.spotify%22%20hype*) 46 | [![GitHub license](https://img.shields.io/github/license/spotify/hype.svg)](./LICENSE) 47 | 48 | A library for seamlessly executing arbitrary JVM closures in [Docker] containers on [Kubernetes]. 49 | 50 | --- 51 | 52 | - [User guide](#user-guide) 53 | * [Dependency](#dependecy) 54 | * [Run functions](#run-functions) 55 | * [Full example](#full-example) 56 | * [Leveraging implicits](#leveraging-implicits) 57 | * [Custom environment images](#custom-environment-images) 58 | - [Process overview](#process-overview) 59 | - [Persistent disk](#persistent-disk) 60 | * [GCE Persistent Disk](#gce-persistent-disk) 61 | + [Volume re-use](#volume-re-use) 62 | - [Environment Pod from YAML](#environment-pod-from-yaml) 63 | 64 | --- 65 | 66 | # User guide 67 | 68 | Hype lets you execute arbitrary JVM code in a distributed environment where different parts 69 | might run concurrently in separate Docker containers, each using different amounts of memory, 70 | CPU and disk. With the help of Kubernetes and a cloud provider such as Google Cloud Platform, 71 | you'll have dynamically scheduled resources available for your code to utilize. 72 | 73 | All this might sound a bit abstract, so let's run through a concrete example. We'll be using Scala 74 | for the examples, but all the core functionality is available from Java as well. 75 | 76 | ## Dependency 77 | 78 | SBT 79 | 80 | ```sbt 81 | "com.spotify" %% "hype" % 82 | ``` 83 | 84 | ## Run functions 85 | 86 | In order to run functions on the cluster, you'll have to set up a `Submitter` value. 87 | The submitter encapsulates "where" to submit your functions. 88 | ```scala 89 | val submitter = GkeSubmitter("gcp-project-id", "gce-zone-id", "gke-cluster-id", "gs://my-staging-bucket") 90 | ``` 91 | 92 | For testing, where you might want to run on a local Docker daemon, use `LocalSubmitter(...)`. 93 | 94 | Writing functions that can be executed with Hype is simple, just wrap them up as an `HFn[T]`. An 95 | `HFn[T]` is a closure that allows Hype to move the actual evaluation into a Docker container. 96 | 97 | ```scala 98 | def example(arg: String) = HFn[String] { 99 | arg + " world!" 100 | } 101 | ``` 102 | 103 | In the previous example, the default Hype Docker image (`spotify/hype`) is used. If you wish to use 104 | your own image, you can easily do so: 105 | 106 | ```scala 107 | def example(arg: String) = HFn.withImage("us.gcr.io/my-image:42") { 108 | arg + " world!" 109 | } 110 | ``` 111 | 112 | Now we'll have to define the environment we want this function to run in. 113 | 114 | ```scala 115 | val env = RunEnvironment() 116 | ``` 117 | 118 | Finally, use use the `Submitter` and `RunEnvironment` to execute an `HFn[T]`. 119 | When execution is complete, it'll return the function value back to your local context. 120 | 121 | ```scala 122 | val result = submitter.submit(example("hello"), env.withRequest("cpu", "750m")) 123 | ``` 124 | 125 | ## Full example 126 | 127 | This is a full example that runs a simple function that executes an arbitrary command and lists all 128 | environment variables. It uses the Scala [sys.process] package to execute commands in the function. 129 | Also see the [docs on how to create k8s secrets](https://kubernetes.io/docs/concepts/configuration/secret/#creating-your-own-secrets) 130 | 131 | ```scala 132 | import sys.process._ 133 | import com.spotify.hype._ 134 | 135 | // A simple model for describing the runtime environment 136 | case class EnvVar(name: String, value: String) 137 | case class Res(cmdOutput: String, mounts: String, vars: List[EnvVar]) 138 | 139 | def extractEnv(cmd: String) = HFn[Res] { 140 | val cmdOutput = cmd !! 141 | val mounts = "df -h" !! 142 | val vars = for ((key, value) <- sys.env.toList) 143 | yield EnvVar(key, value) 144 | 145 | Res(cmdOutput, mounts, vars) 146 | } 147 | 148 | val submitter = GkeSubmitter("gcp-project-id", "gce-zone-id", "gke-cluster-id", "gs://my-staging-bucket") 149 | val env = RunEnvironment() 150 | .withSecret("gcp-key", "/etc/gcloud") // a pre-created k8s secret volume named "gcp-key" 151 | 152 | val res = submitter.submit(extractEnv("uname -a"), env) 153 | 154 | println(res.cmdOutput) 155 | println(res.mounts) 156 | res.vars.foreach(println) 157 | ``` 158 | 159 | The `res.vars` list returned should contain the environment variables that were present in the 160 | docker container while running on the cluster. Here's the output: 161 | 162 | ``` 163 | [info] Running HypeExample 164 | [info] 22:15:14.211 | INFO | StagingUtil |> Uploading 69 files to staging location gs://my-staging-bucket to prepare for execution. 165 | [info] 22:15:51.057 | INFO | StagingUtil |> Uploading complete: 4 files newly uploaded, 65 files cached 166 | [info] 22:15:51.673 | INFO | Submitter |> Submitting gs://my-staging-bucket/manifest-9vhb5u18.txt to RunEnvironment{base=RunEnvironment.SimpleBase{image=gcr.io/gcp-project-id/env-image}, secretMounts=[Secret{name=gcp-key, mountPath=/etc/gcloud}], volumeMounts=[], resourceRequests={}} 167 | [info] 22:15:52.221 | INFO | DockerRunner |> Created pod hype-run-mymlbuw8 168 | [info] 22:15:52.351 | INFO | DockerRunner |> Pod hype-run-mymlbuw8 assigned to node gke-hype-test-default-pool-e1122946-fg9k 169 | [info] 22:16:02.454 | INFO | DockerRunner |> Kubernetes pod hype-run-mymlbuw8 exited with status Succeeded 170 | [info] 22:16:02.455 | INFO | DockerRunner |> Got termination message: gs://my-staging-bucket/continuation-993467547293976140-eUWBfwL9J2tHvWuJw0lU3g-hype-run-mymlbuw8-return.bin 171 | [info] Linux hype-run-mymlbuw8 4.4.21+ #1 SMP Fri Feb 17 15:34:45 PST 2017 x86_64 GNU/Linux 172 | [info] 173 | [info] Filesystem Size Used Avail Use% Mounted on 174 | [info] overlay 95G 4.1G 91G 5% / 175 | [info] tmpfs 7.4G 0 7.4G 0% /dev 176 | [info] tmpfs 7.4G 0 7.4G 0% /sys/fs/cgroup 177 | [info] tmpfs 7.4G 4.0K 7.4G 1% /etc/gcloud 178 | [info] /dev/sda1 95G 4.1G 91G 5% /etc/hosts 179 | [info] tmpfs 7.4G 12K 7.4G 1% /run/secrets/kubernetes.io/serviceaccount 180 | [info] shm 64M 0 64M 0% /dev/shm 181 | [info] 182 | [info] EnvVar(HYPE_EXECUTION_ID,hype-run-mymlbuw8) 183 | [info] EnvVar(GOOGLE_APPLICATION_CREDENTIALS,/etc/gcloud/key.json) 184 | [info] EnvVar(HOSTNAME,hype-run-cv7cln6y) 185 | [info] EnvVar(PATH,/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin) 186 | [info] EnvVar(JAVA_VERSION,8u121) 187 | [info] EnvVar(KUBERNETES_SERVICE_HOST,xx.xx.xx.xx) 188 | ... 189 | ``` 190 | 191 | ## Leveraging implicits 192 | 193 | In order to save some keystrokes, you can use our `implicit` operators: 194 | ```scala 195 | import com.spotify.hype.magic._ 196 | ``` 197 | 198 | Now you can set up an `implicit` `Submitter` value. 199 | ```scala 200 | implicit val submitter = GkeSubmitter("gcp-project-id", "gce-zone-id", "gke-cluster-id", "gs://my-staging-bucket") 201 | ``` 202 | 203 | The environment value can also be declared `implicit`, 204 | but this is not required as it can explicitly be referenced when submitting functions. 205 | 206 | ```scala 207 | implicit val env = RunEnvironment().withSecret("gcp-key", "/etc/gcloud") 208 | ``` 209 | 210 | Finally, use the `#!` (hashbang) operator to execute an `HFn[T]` in a given environment. It will 211 | use the `Submitter` and `RunEnvironment` which should be in scope. 212 | 213 | ```scala 214 | val result = example("hello") #! 215 | ``` 216 | 217 | Using an `implicit` value as we did above works in most cases, but the hashbang (`#!`) 218 | operator also allows you to specify an explicit environment. 219 | 220 | ```scala 221 | val result = example("hello") #! env.withRequest("cpu", "750m") 222 | ``` 223 | ## Custom environment images 224 | 225 | In order for Hype to be able to execute functions in your custom Docker images, you'll have to 226 | install the `hype-run` command by adding the following to your `Dockerfile`: 227 | 228 | ```dockerfile 229 | # Install hype-run command 230 | RUN /bin/sh -c "$(curl -fsSL https://goo.gl/kSogpF)" 231 | ENTRYPOINT ["hype-run"] 232 | ``` 233 | 234 | It is important to have exactly this `ENTRYPOINT` as the Kubernetes Pods will expect to run the 235 | `hype-run` command. 236 | 237 | See example [`Dockerfile`](hype-docker/Dockerfile) 238 | 239 | # Process overview 240 | 241 | This describes what Hype does from a high level point of view. 242 | 243 |

244 | 247 |

248 | 249 | # Persistent disk 250 | 251 | Hype makes it easy to schedule persistent disk volumes across different closures in a workflow. 252 | A typical pattern seen in many use cases is to first use a disk in read-write mode to download and 253 | prepare some data, and then fork out to several parallel tasks that use the disk in read-only mode. 254 | 255 |

256 | 259 |

260 | 261 | ## GCE Persistent Disk 262 | 263 | In this example, we're using a StorageClass for [GCE Persistent Disk] that we've already set up on 264 | our cluster. 265 | 266 | ```yaml 267 | kind: StorageClass 268 | apiVersion: storage.k8s.io/v1beta1 269 | metadata: 270 | name: gce-ssd-pd 271 | provisioner: kubernetes.io/gce-pd 272 | parameters: 273 | type: pd-ssd 274 | ``` 275 | 276 | We can then request volumes from this StorageClass using the Hype API: 277 | 278 | ```scala 279 | import sys.process._ 280 | import com.spotify.hype.magic._ 281 | 282 | implicit val submitter = GkeSubmitter("gcp-project-id", 283 | "gce-zone-id", 284 | "gke-cluster-id", 285 | "gs://my-staging-bucket") 286 | 287 | // Create a 10Gi volume from the 'gce-ssd-pd' storage class 288 | val ssd10Gi = TransientVolume("gce-ssd-pd", "10Gi") 289 | val mount = "/usr/share/volume" 290 | 291 | val env = RunEnvironment() 292 | val readWriteEnv = env.withMount(ssd10Gi.mountReadWrite(mount)) 293 | val readOnlyEnv = env.withMount(ssd10Gi.mountReadOnly(mount)) 294 | 295 | def write = HFn[Int] { 296 | // get a random word and store it in the volume 297 | s"curl -so $mount/word http://www.setgetgo.com/randomword/get.php" ! 298 | } 299 | 300 | def read = HFn[String] { 301 | // read the word file 302 | s"cat $mount/word" !! 303 | } 304 | 305 | // Write to the volume 306 | write #! readWriteEnv 307 | 308 | // Run 10 parallel functions that have read only access to the volume 309 | val results = for (_ <- Range(0, 10).par) 310 | yield read #! readOnlyEnv 311 | ``` 312 | 313 | The submissions from the parallel range will each run concurrently in separate pods and have 314 | read-only access to the `/usr/share/volume` mount. The volume should contain the random word that 315 | was written to it from the `write` function. 316 | 317 | Coordinating metadata and parameters across multiple submissions should be just as trivial as 318 | passing values from function calls as arguments to other functions. 319 | 320 | ### Volume re-use 321 | 322 | By default, the backing claim for a `TransientVolume` on Kubernetes is deleted when the JVM 323 | terminates. 324 | 325 | If you wish to persist the Volume between invocations, you can use: 326 | 327 | ```scala 328 | val disk = PersistentVolume("my-persistent-volume", "gce-ssd-pd", "10Gi") 329 | ``` 330 | 331 | If the volume does not exist, it will be created. Subsequent invocations will return use already 332 | created volume. 333 | 334 | This is useful in use cases with larger volumes that take a significant amount of time to load, 335 | or when there's some sort of workflow orchestration around the Hype code that might run 336 | different parts in separate JVM invocations. 337 | 338 | # Environment Pod from YAML 339 | 340 | Sometimes more control over the Kubernetes Pod is desired. For these cases a regular Pod YAML file 341 | can be used as a base for the `RunEnvironment`. Hype will still manage any used Volume Claims and 342 | mounts, but will leave all other details as you've specified them. 343 | 344 | Hype will expect at least this field to be specified: 345 | 346 | - `spec.containers[name:hype-run]` - There must at least be a container named `hype-run` 347 | 348 | Please note that the image field should *not* bet set (Hype requires each module to define its image). 349 | 350 | _Hype will override the `spec.containers[name:hype-run].args` field, so don't set it._ 351 | 352 | Here's a minimal Pod YAML file with some custom settings, `./src/main/resources/pod.yaml`: 353 | 354 | ```yaml 355 | apiVersion: v1 356 | kind: Pod 357 | 358 | spec: 359 | restartPolicy: Never # do not retry on failure 360 | 361 | containers: 362 | - name: hype-run 363 | imagePullPolicy: Always # pull the image on each run 364 | 365 | env: # additional environment variables 366 | - name: EXAMPLE 367 | value: my-env-value 368 | ``` 369 | 370 | Any resource requests added through the `RunEnvironment` API will merge with, and override the ones 371 | set in the YAML file. 372 | 373 | Then simply load your `RunEnvironment` through 374 | 375 | ```scala 376 | val env = RunEnvironmentFromYaml("/pod.yaml") 377 | ``` 378 | 379 | --- 380 | 381 | _This project is in early development stages, expect anything you see to change._ 382 | 383 | [Docker]: https://www.docker.com 384 | [Kubernetes]: https://kubernetes.io/ 385 | [GCE Persistent Disk]: http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html 386 | [sys.process]: http://www.scala-lang.org/api/rc2/scala/sys/process/package.html 387 | --------------------------------------------------------------------------------