├── .github └── workflows │ └── semgrep.yml ├── README.md ├── buildall.sh ├── druid-base ├── Dockerfile ├── README.md ├── pull.sh └── run.sh ├── mesos-druid-base ├── Dockerfile └── run-mesos.sh └── services ├── README.md ├── broker ├── Dockerfile ├── runtime.properties └── service.properties ├── common.properties ├── coordinator ├── Dockerfile ├── runtime.properties └── service.properties ├── docker-compose.yml ├── historical ├── Dockerfile ├── runtime.properties └── service.properties ├── overlord ├── Dockerfile ├── runtime.properties └── service.properties ├── realtime ├── Dockerfile ├── example-druid.spec.json ├── runtime.properties └── service.properties └── test.query.json /.github/workflows/semgrep.yml: -------------------------------------------------------------------------------- 1 | on: 2 | pull_request: {} 3 | workflow_dispatch: {} 4 | push: 5 | branches: 6 | - main 7 | - master 8 | schedule: 9 | - cron: '0 0 * * *' 10 | name: Semgrep config 11 | jobs: 12 | semgrep: 13 | name: semgrep/ci 14 | runs-on: ubuntu-latest 15 | env: 16 | SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }} 17 | SEMGREP_URL: https://cloudflare.semgrep.dev 18 | SEMGREP_APP_URL: https://cloudflare.semgrep.dev 19 | SEMGREP_VERSION_CHECK_URL: https://cloudflare.semgrep.dev/api/check-version 20 | container: 21 | image: semgrep/semgrep 22 | steps: 23 | - uses: actions/checkout@v4 24 | - run: semgrep ci 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Druid in docker 2 | 3 | ## [Base image](druid-base) 4 | 5 | Base image is used to bake your own configuration. 6 | 7 | ## [Mesos base image](mesos-druid-base) 8 | 9 | Base image that supports running on [marathon](https://mesosphere.github.io/marathon/). 10 | This image should be used as base as well. 11 | 12 | ## [Example cluster](services) 13 | 14 | Example cluster with kafka as realtime input and hdfs as deep storage 15 | to run in boot2docker. 16 | -------------------------------------------------------------------------------- /buildall.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | set -e 4 | 5 | docker build -t bobrik/druid-base:0.8.0 druid-base 6 | docker build -t bobrik/mesos-druid-base:0.8.0 mesos-druid-base 7 | 8 | for service in broker coordinator historical overlord realtime; do 9 | cat services/common.properties services/$service/service.properties > services/$service/runtime.properties 10 | docker build -t bobrik/mesos-druid-$service:0.8.0 services/$service 11 | done 12 | -------------------------------------------------------------------------------- /druid-base/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM java:8-jre 2 | 3 | ENV DRUID_VERSION 0.8.0 4 | 5 | RUN mkdir /opt/druid && \ 6 | curl -L "http://static.druid.io/artifacts/releases/druid-${DRUID_VERSION}-bin.tar.gz" | tar zx --strip-components 1 -C /opt/druid 7 | 8 | ADD ./pull.sh /pull.sh 9 | 10 | ADD ./run.sh /run.sh 11 | 12 | ENTRYPOINT ["/run.sh"] 13 | -------------------------------------------------------------------------------- /druid-base/README.md: -------------------------------------------------------------------------------- 1 | # Druid base image 2 | 3 | ## Usage 4 | 5 | This image expects you to provide configuration file bind-mounted to 6 | `/opt/druid/config/${DRUID_NODE_TYPE}/runtime.properties`. Usage: 7 | 8 | ``` 9 | docker run -d \ 10 | --net host \ 11 | -e DRUID_HOST=10.10.10.10 \ 12 | -e DRUID_PORT=8001 \ 13 | -e DRUID_MEMORY_LIMIT=256m \ 14 | -v /etc/druid/coordinator.properties:/opt/druid/config/coordinator/runtime.properties 15 | --name druid-broker bobrik/druid-base coordinator 16 | ``` 17 | 18 | This would use `/etc/druid/coordinator.properties` to run coordinator node 19 | with 256m of heap for JVM on address `10.10.10.10:8001`. Note that host 20 | networking is required since druid does not support port mapping yet. 21 | 22 | It's a good idea to create your own druid images for each service 23 | with configuration files baked in. 24 | 25 | You might as well overwrite configuration directive instead of overwriting 26 | the whole config. To use it, prefix directive name with `DRUID_CONFIG_` and 27 | replace `.` in directive name with `_` and provide env variable with this 28 | name. For example `druid.storage.type=local` can be passed as 29 | `DRUID_CONFIG_druid_storage_type=local`. 30 | 31 | ## Example image 32 | 33 | Example image for coordinator node: 34 | 35 | ``` 36 | FROM bobrik/druid-base:0.8.0 37 | 38 | ADD ./runtime.properties /opt/druid/config/coordinator/runtime.properties 39 | 40 | ENTRYPOINT ["/run.sh", "coordinator"] 41 | ``` 42 | 43 | Extra script `/pull.sh` is provided to pull druid extensions at build time, 44 | here's how pull hdfs, kafka and mysql extensions: 45 | 46 | ``` 47 | RUN /pull.sh '["io.druid.extensions:druid-hdfs-storage","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage"]' 48 | ``` 49 | 50 | If you don't pull dependencies on at build time, you are going to wait 51 | forever for your nodes to start up. 52 | -------------------------------------------------------------------------------- /druid-base/pull.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | set -e 4 | 5 | exec java -cp "/opt/druid/config/_common:/opt/druid/lib/*" \ 6 | "-Ddruid.extensions.coordinates=$1" \ 7 | -Ddruid.extensions.localRepository=/opt/druid/repository \ 8 | io.druid.cli.Main tools pull-deps 9 | -------------------------------------------------------------------------------- /druid-base/run.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | 5 | DRUID_NODE_TYPE=$1 6 | if [ -z $DRUID_NODE_TYPE ]; then 7 | echo "DRUID_NODE_TYPE (1st image argument) is not set" 8 | exit 1 9 | fi 10 | 11 | if [ -z $DRUID_MEMORY_LIMIT ]; then 12 | echo "DRUID_MEMORY_LIMIT is not set" 13 | exit 1 14 | fi 15 | 16 | if [ -z $DRUID_HOST ]; then 17 | DRUID_HOST=$(ip addr | grep 'eth0' | awk '{print $2}' | cut -f1 -d'/' | tail -1) 18 | fi 19 | 20 | DRUID_PORT=${DRUID_PORT:-8000} 21 | 22 | echo "druid.host=${DRUID_HOST}" > /opt/druid/config/_common/common.runtime.properties 23 | echo "druid.port=${DRUID_PORT}" >> /opt/druid/config/_common/common.runtime.properties 24 | 25 | for VAR in $(env); do 26 | if [[ $VAR =~ ^DRUID_CONFIG_ ]]; then 27 | druid_name=$(echo "$VAR" | sed -r "s/DRUID_CONFIG_(.*)=.*/\1/g" | tr _ .) 28 | env_value=$(echo "$VAR" | sed -r "s/(.*)=.*/\1/g") 29 | if egrep -q "(^|^#)$druid_name=" /opt/druid/config/${DRUID_NODE_TYPE}/runtime.properties; then 30 | sed -r -i "s@(^|^#)($druid_name)=(.*)@\2=${!env_value}@g" "/opt/druid/config/${DRUID_NODE_TYPE}/runtime.properties" 31 | else 32 | echo "$druid_name=${!env_value}" >> "/opt/druid/config/${DRUID_NODE_TYPE}/runtime.properties" 33 | fi 34 | fi 35 | done 36 | 37 | cat "/opt/druid/config/${DRUID_NODE_TYPE}/runtime.properties" 38 | 39 | # workaround for https://github.com/druid-io/druid/pull/1022 40 | DRUID_EXTRA_CLASSPATH=:$(find /opt/druid/repository -type f -name "*.jar" -not -path "*io/druid/extensions*" -printf '%p:' | sed 's/:$//') 41 | 42 | echo CLASSPATH "/opt/druid/config/_common:/opt/druid/config/${DRUID_NODE_TYPE}:/opt/druid/lib/*:${DRUID_EXTRA_CLASSPATH}" 43 | 44 | exec java "-Xmx${DRUID_MEMORY_LIMIT}" \ 45 | -Duser.timezone=UTC \ 46 | -Dfile.encoding=UTF-8 \ 47 | -classpath "/opt/druid/config/_common:/opt/druid/config/${DRUID_NODE_TYPE}:/opt/druid/lib/*:${DRUID_EXTRA_CLASSPATH}" io.druid.cli.Main server "${DRUID_NODE_TYPE}" 48 | -------------------------------------------------------------------------------- /mesos-druid-base/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM bobrik/druid-base:0.8.0 2 | 3 | ADD ./run-mesos.sh /run-mesos.sh 4 | 5 | ENTRYPOINT ["/run-mesos.sh"] 6 | -------------------------------------------------------------------------------- /mesos-druid-base/run-mesos.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | set -e 4 | 5 | export DRUID_HOST=$HOST 6 | export DRUID_PORT=$PORT 7 | 8 | /run.sh $@ 9 | -------------------------------------------------------------------------------- /services/README.md: -------------------------------------------------------------------------------- 1 | # Example service configuration 2 | 3 | This is example configuration to run in boot2docker. 4 | 5 | ## Usage 6 | 7 | Build everything from the root of this repository: 8 | 9 | ``` 10 | ./buildall.sh 11 | ``` 12 | 13 | Run hdfs node: 14 | 15 | ``` 16 | docker run -d -p 9000:9000 --name hdfs sequenceiq/hadoop-docker:2.4.1 17 | ``` 18 | 19 | Then spin up cluster with kafka and mysql: 20 | 21 | ``` 22 | docker-compose up 23 | ``` 24 | 25 | ### Example events 26 | 27 | Events to push in kafka topic `example-druid` should look like this: 28 | 29 | ```json 30 | {"timestamp":"2015-03-31T20:52:08Z","zone_id":2,"requests":23,"bytes_client":1500,"bytes_origin":400} 31 | ``` 32 | 33 | ### Example query 34 | 35 | ``` 36 | curl -X POST -H "Content-type: application/json" "http://192.168.59.103:8031/druid/v2/?pretty" -d @test.query.json 37 | ``` 38 | -------------------------------------------------------------------------------- /services/broker/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM bobrik/mesos-druid-base:0.8.0 2 | 3 | RUN /pull.sh '["io.druid.extensions:druid-hdfs-storage","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage","io.druid.extensions:postgresql-metadata-storage"]' 4 | 5 | ADD ./runtime.properties /opt/druid/config/broker/runtime.properties 6 | 7 | ENTRYPOINT ["/run-mesos.sh", "broker"] 8 | -------------------------------------------------------------------------------- /services/broker/runtime.properties: -------------------------------------------------------------------------------- 1 | # common config 2 | druid.extensions.coordinates=["io.druid.extensions:druid-hdfs-storage","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage","io.druid.extensions:postgresql-metadata-storage"] 3 | druid.extensions.localRepository=/opt/druid/repository 4 | 5 | druid.zk.service.host=boot2docker 6 | 7 | druid.metadata.storage.type=mysql 8 | druid.metadata.storage.connector.connectURI=jdbc:mysql://boot2docker:3306/druid 9 | druid.metadata.storage.connector.user=druid 10 | druid.metadata.storage.connector.password=druid 11 | 12 | # druid.storage.type=s3 13 | druid.s3.accessKey=nope 14 | druid.s3.secretKey=nope-nope 15 | druid.storage.bucket=druid-haha 16 | 17 | druid.storage.type: hdfs 18 | druid.storage.storageDirectory: hdfs://boot2docker:9000/druid 19 | 20 | druid.cache.type=local 21 | druid.cache.sizeInBytes=10000000 22 | 23 | druid.selectors.indexing.serviceName=overlord 24 | 25 | # service specific 26 | druid.service=broker 27 | 28 | druid.broker.cache.useCache=true 29 | druid.broker.cache.populateCache=true 30 | 31 | druid.processing.buffer.sizeBytes=100000000 32 | druid.processing.numThreads=1 33 | -------------------------------------------------------------------------------- /services/broker/service.properties: -------------------------------------------------------------------------------- 1 | druid.service=broker 2 | 3 | druid.broker.cache.useCache=true 4 | druid.broker.cache.populateCache=true 5 | 6 | druid.processing.buffer.sizeBytes=100000000 7 | druid.processing.numThreads=1 8 | -------------------------------------------------------------------------------- /services/common.properties: -------------------------------------------------------------------------------- 1 | # common config 2 | druid.extensions.coordinates=["io.druid.extensions:druid-hdfs-storage","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage","io.druid.extensions:postgresql-metadata-storage"] 3 | druid.extensions.localRepository=/opt/druid/repository 4 | 5 | druid.zk.service.host=boot2docker 6 | 7 | druid.metadata.storage.type=mysql 8 | druid.metadata.storage.connector.connectURI=jdbc:mysql://boot2docker:3306/druid 9 | druid.metadata.storage.connector.user=druid 10 | druid.metadata.storage.connector.password=druid 11 | 12 | # druid.storage.type=s3 13 | druid.s3.accessKey=nope 14 | druid.s3.secretKey=nope-nope 15 | druid.storage.bucket=druid-haha 16 | 17 | druid.storage.type: hdfs 18 | druid.storage.storageDirectory: hdfs://boot2docker:9000/druid 19 | 20 | druid.cache.type=local 21 | druid.cache.sizeInBytes=10000000 22 | 23 | druid.selectors.indexing.serviceName=overlord 24 | 25 | # service specific 26 | -------------------------------------------------------------------------------- /services/coordinator/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM bobrik/mesos-druid-base:0.8.0 2 | 3 | RUN /pull.sh '["io.druid.extensions:druid-hdfs-storage","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage","io.druid.extensions:postgresql-metadata-storage"]' 4 | 5 | ADD ./runtime.properties /opt/druid/config/coordinator/runtime.properties 6 | 7 | ENTRYPOINT ["/run-mesos.sh", "coordinator"] 8 | -------------------------------------------------------------------------------- /services/coordinator/runtime.properties: -------------------------------------------------------------------------------- 1 | # common config 2 | druid.extensions.coordinates=["io.druid.extensions:druid-hdfs-storage","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage","io.druid.extensions:postgresql-metadata-storage"] 3 | druid.extensions.localRepository=/opt/druid/repository 4 | 5 | druid.zk.service.host=boot2docker 6 | 7 | druid.metadata.storage.type=mysql 8 | druid.metadata.storage.connector.connectURI=jdbc:mysql://boot2docker:3306/druid 9 | druid.metadata.storage.connector.user=druid 10 | druid.metadata.storage.connector.password=druid 11 | 12 | # druid.storage.type=s3 13 | druid.s3.accessKey=nope 14 | druid.s3.secretKey=nope-nope 15 | druid.storage.bucket=druid-haha 16 | 17 | druid.storage.type: hdfs 18 | druid.storage.storageDirectory: hdfs://boot2docker:9000/druid 19 | 20 | druid.cache.type=local 21 | druid.cache.sizeInBytes=10000000 22 | 23 | druid.selectors.indexing.serviceName=overlord 24 | 25 | # service specific 26 | druid.service=coordinator 27 | druid.coordinator.startDelay=PT70s 28 | -------------------------------------------------------------------------------- /services/coordinator/service.properties: -------------------------------------------------------------------------------- 1 | druid.service=coordinator 2 | druid.coordinator.startDelay=PT70s 3 | -------------------------------------------------------------------------------- /services/docker-compose.yml: -------------------------------------------------------------------------------- 1 | zk: 2 | image: bobrik/zookeeper 3 | ports: 4 | - 2181:2181 5 | environment: 6 | ZK_ID: 1 7 | ZK_CONFIG: tickTime=2000,initLimit=10,syncLimit=5,maxClientCnxns=1024,forceSync=no,clientPort=2181 8 | volumes: 9 | - /var/lib/zookeeper:/var/lib/zookeper 10 | 11 | mysql: 12 | image: mysql:5.7 13 | ports: 14 | - 3306:3306 15 | environment: 16 | MYSQL_ROOT_PASSWORD: haha 17 | MYSQL_DATABASE: druid 18 | MYSQL_USER: druid 19 | MYSQL_PASSWORD: druid 20 | volumes: 21 | - /var/lib/mysql:/var/lib/mysql 22 | 23 | kafka: 24 | image: wurstmeister/kafka:0.8.2.0 25 | ports: 26 | - 9092:9092 27 | links: 28 | - zk:zk 29 | environment: 30 | KAFKA_ADVERTISED_HOST_NAME: 192.168.59.103 31 | KAFKA_ADVERTISED_PORT: 9092 32 | KAFKA_ZOOKEEPER_CONNECT: 192.168.59.103:2181 33 | volumes: 34 | - /var/lib/kafka:/kafka 35 | 36 | coordinator: 37 | image: bobrik/mesos-druid-coordinator:0.7.0 38 | net: host 39 | environment: 40 | HOST: 192.168.59.103 41 | PORT: 8001 42 | DRUID_MEMORY_LIMIT: 256m 43 | 44 | historical: 45 | image: bobrik/mesos-druid-historical:0.7.0 46 | net: host 47 | environment: 48 | HOST: 192.168.59.103 49 | PORT: 8011 50 | DRUID_MEMORY_LIMIT: 512m 51 | 52 | broker: 53 | image: bobrik/mesos-druid-broker:0.7.0 54 | net: host 55 | environment: 56 | HOST: 192.168.59.103 57 | PORT: 8031 58 | DRUID_MEMORY_LIMIT: 512m 59 | 60 | realtime: 61 | image: bobrik/mesos-druid-realtime:0.7.0 62 | net: host 63 | environment: 64 | HOST: 192.168.59.103 65 | PORT: 8021 66 | DRUID_MEMORY_LIMIT: 512m 67 | 68 | overlord: 69 | image: bobrik/mesos-druid-overlord:0.7.0 70 | net: host 71 | environment: 72 | HOST: 192.168.59.103 73 | PORT: 8041 74 | DRUID_MEMORY_LIMIT: 512m 75 | -------------------------------------------------------------------------------- /services/historical/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM bobrik/mesos-druid-base:0.8.0 2 | 3 | RUN /pull.sh '["io.druid.extensions:druid-hdfs-storage","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage","io.druid.extensions:postgresql-metadata-storage"]' 4 | 5 | ADD ./runtime.properties /opt/druid/config/historical/runtime.properties 6 | 7 | ENTRYPOINT ["/run-mesos.sh", "historical"] 8 | -------------------------------------------------------------------------------- /services/historical/runtime.properties: -------------------------------------------------------------------------------- 1 | # common config 2 | druid.extensions.coordinates=["io.druid.extensions:druid-hdfs-storage","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage","io.druid.extensions:postgresql-metadata-storage"] 3 | druid.extensions.localRepository=/opt/druid/repository 4 | 5 | druid.zk.service.host=boot2docker 6 | 7 | druid.metadata.storage.type=mysql 8 | druid.metadata.storage.connector.connectURI=jdbc:mysql://boot2docker:3306/druid 9 | druid.metadata.storage.connector.user=druid 10 | druid.metadata.storage.connector.password=druid 11 | 12 | # druid.storage.type=s3 13 | druid.s3.accessKey=nope 14 | druid.s3.secretKey=nope-nope 15 | druid.storage.bucket=druid-haha 16 | 17 | druid.storage.type: hdfs 18 | druid.storage.storageDirectory: hdfs://boot2docker:9000/druid 19 | 20 | druid.cache.type=local 21 | druid.cache.sizeInBytes=10000000 22 | 23 | druid.selectors.indexing.serviceName=overlord 24 | 25 | # service specific 26 | druid.service=historical 27 | druid.segmentCache.locations=[{"path": "/tmp/druid/indexCache", "maxSize"\: 10000000000}] 28 | druid.server.maxSize=10000000000 29 | 30 | druid.processing.buffer.sizeBytes=100000000 31 | druid.processing.numThreads=1 32 | -------------------------------------------------------------------------------- /services/historical/service.properties: -------------------------------------------------------------------------------- 1 | druid.service=historical 2 | druid.segmentCache.locations=[{"path": "/tmp/druid/indexCache", "maxSize"\: 10000000000}] 3 | druid.server.maxSize=10000000000 4 | 5 | druid.processing.buffer.sizeBytes=100000000 6 | druid.processing.numThreads=1 7 | -------------------------------------------------------------------------------- /services/overlord/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM bobrik/mesos-druid-base:0.8.0 2 | 3 | RUN /pull.sh '["io.druid.extensions:druid-hdfs-storage","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage","io.druid.extensions:postgresql-metadata-storage"]' 4 | 5 | ADD ./runtime.properties /opt/druid/config/overlord/runtime.properties 6 | 7 | ENTRYPOINT ["/run-mesos.sh", "overlord"] 8 | -------------------------------------------------------------------------------- /services/overlord/runtime.properties: -------------------------------------------------------------------------------- 1 | # common config 2 | druid.extensions.coordinates=["io.druid.extensions:druid-hdfs-storage","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage","io.druid.extensions:postgresql-metadata-storage"] 3 | druid.extensions.localRepository=/opt/druid/repository 4 | 5 | druid.zk.service.host=boot2docker 6 | 7 | druid.metadata.storage.type=mysql 8 | druid.metadata.storage.connector.connectURI=jdbc:mysql://boot2docker:3306/druid 9 | druid.metadata.storage.connector.user=druid 10 | druid.metadata.storage.connector.password=druid 11 | 12 | # druid.storage.type=s3 13 | druid.s3.accessKey=nope 14 | druid.s3.secretKey=nope-nope 15 | druid.storage.bucket=druid-haha 16 | 17 | druid.storage.type: hdfs 18 | druid.storage.storageDirectory: hdfs://boot2docker:9000/druid 19 | 20 | druid.cache.type=local 21 | druid.cache.sizeInBytes=10000000 22 | 23 | druid.selectors.indexing.serviceName=overlord 24 | 25 | # service specific 26 | druid.service=overlord 27 | 28 | druid.indexer.queue.startDelay=PT0M 29 | druid.indexer.runner.javaOpts="-server -Xmx256m" 30 | druid.indexer.fork.property.druid.processing.numThreads=1 31 | druid.indexer.fork.property.druid.computation.buffer.size=100000000 32 | -------------------------------------------------------------------------------- /services/overlord/service.properties: -------------------------------------------------------------------------------- 1 | druid.service=overlord 2 | 3 | druid.indexer.queue.startDelay=PT0M 4 | druid.indexer.runner.javaOpts="-server -Xmx256m" 5 | druid.indexer.fork.property.druid.processing.numThreads=1 6 | druid.indexer.fork.property.druid.computation.buffer.size=100000000 7 | -------------------------------------------------------------------------------- /services/realtime/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM bobrik/mesos-druid-base:0.8.0 2 | 3 | RUN /pull.sh '["io.druid.extensions:druid-hdfs-storage","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage","io.druid.extensions:postgresql-metadata-storage"]' 4 | 5 | ADD ./runtime.properties /opt/druid/config/realtime/runtime.properties 6 | ADD ./example-druid.spec.json /example-druid.spec.json 7 | 8 | ENTRYPOINT ["/run-mesos.sh", "realtime"] 9 | -------------------------------------------------------------------------------- /services/realtime/example-druid.spec.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "dataSchema": { 4 | "dataSource": "example-druid", 5 | "parser": { 6 | "type": "string", 7 | "parseSpec": { 8 | "format": "json", 9 | "timestampSpec": { 10 | "column": "timestamp", 11 | "format": "auto" 12 | }, 13 | "dimensionsSpec": { 14 | "dimensions": [ 15 | "zone_id", 16 | "requests", 17 | "bytes_client", 18 | "bytes_origin" 19 | ], 20 | "dimensionExclusions": [], 21 | "spatialDimensions": [] 22 | } 23 | } 24 | }, 25 | "metricsSpec": [ 26 | { 27 | "type": "longSum", 28 | "name": "requestsSum", 29 | "fieldName": "requests" 30 | }, 31 | { 32 | "type": "longSum", 33 | "name": "bytesClientSum", 34 | "fieldName": "bytes_client" 35 | }, 36 | { 37 | "type": "longSum", 38 | "name": "bytesOriginSum", 39 | "fieldName": "bytes_origin" 40 | } 41 | ], 42 | "granularitySpec": { 43 | "type": "uniform", 44 | "segmentGranularity": "MINUTE", 45 | "queryGranularity": "NONE" 46 | } 47 | }, 48 | "ioConfig": { 49 | "type": "realtime", 50 | "firehose": { 51 | "type": "kafka-0.8", 52 | "consumerProps": { 53 | "zookeeper.connect": "192.168.59.103:2181", 54 | "zookeeper.connection.timeout.ms": "15000", 55 | "zookeeper.session.timeout.ms": "15000", 56 | "zookeeper.sync.time.ms": "5000", 57 | "group.id": "example-druid", 58 | "fetch.message.max.bytes": "1048586", 59 | "auto.offset.reset": "largest", 60 | "auto.commit.enable": "false" 61 | }, 62 | "feed": "example-druid" 63 | }, 64 | "plumber": { 65 | "type": "realtime" 66 | } 67 | }, 68 | "tuningConfig": { 69 | "type": "realtime", 70 | "maxRowsInMemory": 500000, 71 | "intermediatePersistPeriod": "PT1m", 72 | "windowPeriod": "PT2m" 73 | } 74 | } 75 | ] 76 | -------------------------------------------------------------------------------- /services/realtime/runtime.properties: -------------------------------------------------------------------------------- 1 | # common config 2 | druid.extensions.coordinates=["io.druid.extensions:druid-hdfs-storage","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage","io.druid.extensions:postgresql-metadata-storage"] 3 | druid.extensions.localRepository=/opt/druid/repository 4 | 5 | druid.zk.service.host=boot2docker 6 | 7 | druid.metadata.storage.type=mysql 8 | druid.metadata.storage.connector.connectURI=jdbc:mysql://boot2docker:3306/druid 9 | druid.metadata.storage.connector.user=druid 10 | druid.metadata.storage.connector.password=druid 11 | 12 | # druid.storage.type=s3 13 | druid.s3.accessKey=nope 14 | druid.s3.secretKey=nope-nope 15 | druid.storage.bucket=druid-haha 16 | 17 | druid.storage.type: hdfs 18 | druid.storage.storageDirectory: hdfs://boot2docker:9000/druid 19 | 20 | druid.cache.type=local 21 | druid.cache.sizeInBytes=10000000 22 | 23 | druid.selectors.indexing.serviceName=overlord 24 | 25 | # service specific 26 | druid.service=realtime 27 | 28 | druid.realtime.specFile=/example-druid.spec.json 29 | 30 | druid.processing.buffer.sizeBytes=100000000 31 | druid.processing.numThreads=2 32 | -------------------------------------------------------------------------------- /services/realtime/service.properties: -------------------------------------------------------------------------------- 1 | druid.service=realtime 2 | 3 | druid.realtime.specFile=/example-druid.spec.json 4 | 5 | druid.processing.buffer.sizeBytes=100000000 6 | druid.processing.numThreads=2 7 | -------------------------------------------------------------------------------- /services/test.query.json: -------------------------------------------------------------------------------- 1 | { 2 | "queryType": "groupBy", 3 | "dataSource": "example-druid", 4 | "granularity": "minute", 5 | "dimensions": ["zone_id"], 6 | "aggregations": [ 7 | { "type": "count", "name": "rows" }, 8 | { "type": "longSum", "fieldName": "requestsSum", "name": "requests" }, 9 | { "type": "longSum", "fieldName": "bytesClientSum", "name": "bytes_client" }, 10 | { "type": "longSum", "fieldName": "bytesOriginSum", "name": "bytes_origin" } 11 | ], 12 | "intervals": ["2015-01-01T00:00/2020-01-01T00"] 13 | } 14 | --------------------------------------------------------------------------------