├── .gitignore ├── LICENSE ├── README.md ├── examples ├── docker-swarm-mode │ └── Makefile ├── docker │ ├── Makefile │ ├── cluster-master.env_file │ ├── cluster-slave.env_file │ ├── docker-compose.license-master.yml │ ├── docker-compose.yml │ └── shc-member.env_file └── kubernetes-minikube │ └── Makefile └── splunk-cluster ├── .dockerignore ├── .gitignore ├── Dockerfile └── splunk_setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Created by https://www.gitignore.io/api/osx 2 | 3 | ### OSX ### 4 | *.DS_Store 5 | .AppleDouble 6 | .LSOverride 7 | 8 | # Icon must end with two \r 9 | Icon 10 | 11 | 12 | # Thumbnails 13 | ._* 14 | 15 | # Files that might appear in the root of a volume 16 | .DocumentRevisions-V100 17 | .fseventsd 18 | .Spotlight-V100 19 | .TemporaryItems 20 | .Trashes 21 | .VolumeIcon.icns 22 | .com.apple.timemachine.donotpresent 23 | 24 | # Directories potentially created on remote AFP share 25 | .AppleDB 26 | .AppleDesktop 27 | Network Trash Folder 28 | Temporary Items 29 | .apdisk 30 | 31 | 32 | # Splunk licenses 33 | *.lic 34 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Denis Gladkikh 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Table of Contents 2 | 3 | - [Introduction](#introduction) 4 | - [Version](#version) 5 | - [How it works](#how-it-works) 6 | - [Use it](#use-it) 7 | - [Deploy](#deploy) 8 | - [On docker instance](#on-docker-instance) 9 | - [If you do not have a License](#if-you-do-not-have-a-license) 10 | - [If you have a Splunk Enterprise License](#if-you-have-a-splunk-enterprise-license) 11 | - [On docker swarm](#on-docker-swarm) 12 | - [On kubernetes](#on-kubernetes) 13 | - [Examples after setup](#examples-after-setup) 14 | - [Install application on SHC using SHC Deployer](#install-application-on-shc-using-shc-deployer) 15 | - [TODO](#todo) 16 | 17 | 18 | ## Introduction 19 | 20 | > NOTE: I'm working at Splunk, but this is not an official Splunk images. 21 | > I build them in my free time when I'm not at work. I have some knowledge 22 | > about Splunk, but you should think twice before putting them in 23 | > production. I run these images on my own home server just for 24 | > my personal needs. If you have any issues - feel free to open a 25 | > [bug](https://github.com/outcoldman/docker-splunk-cluster/issues). 26 | 27 | > Use for learning purposes. 28 | 29 | This repository contains set of examples how to run Splunk Enterprise cluster in Docker, 30 | including Search Head Cluster and Indexing Cluster. 31 | 32 | The main purpose of this repository is to show how to automate Splunk Cluster deployment. 33 | Below you can find examples how to setup Cluster on Docker, Swarm Mode, Kubernetes (TODO). 34 | 35 | ### Version 36 | 37 | Based on 38 | 39 | * Version: `6.4.3` 40 | * Build: `b03109c2bad4` 41 | 42 | ## How it works 43 | 44 | These examples depend on the custom image, which you can build using `./splunk-cluster/` folder. 45 | This image differs from `outcoldman/splunk` with just one change. 46 | It has special `splunk_setup.py` script, which allows to pre-configure Splunk. 47 | This script supports several commands: 48 | 49 | - `--wait-splunk schema://hostname:mgmt_port [re_server_role1] [re_server_role2] ... [re_server_roleN]`. 50 | This command will wait till specified url will reply and that `/services/server/info` will have 51 | all roles defined in `server_role` list. 52 | - `--configure` - using environment variables append specific configurations. Environment variables can be 53 | defined in format 54 | - `CONF[__{app_location}]__{conf}__{stanza}__{key}={value}` - this will append `key` with `value` under `stanza` in `conf` file 55 | in `local` folder under `app_location`. If `app_location` is not specified will be written in `local` folder under `$SPLUNK_HOME/etc/system/`. 56 | - `CONF[__{app_location}]__meta__{stanza}__{key}={value}` - metadata information will be written in `local.meta` file in 57 | `metadata` folder under `app_location`. If `app_location` is not specified will be written in `metadata` folder under `$SPLUNK_HOME/etc/system/`. 58 | - `--add-licenses {folder}` - all licenses will be added from specified location. If no `*.lic` files can be find in this folder - 59 | script will wait for them. 60 | - `--shc-autobootstrap {number_of_expected_shc_members} {mgmt_uri} {local_user} {local_password} {service_discovery_url} {service_discovery_user} {service_discovery_password}`. 61 | Automatically bootstrap SHC when this will be `{number_of_expected_shc_members}` number of SHC members. Script will use KVStore endpoint 62 | specified under `{service_discovery_url}` to discover other members. `{mgmt_uri}` will be used to bootstrap members. 63 | - `--healthcheck` - using environment variables `SPLUNK_HEALTHCHECK_{health_check_name}=schema://hostname:port` you can define 64 | how container will check that current container is healthy. For example to check that splunkd web server is responsive 65 | you can use `SPLUNK_HEALTHCHECK_SPLUNKD=https://127.0.0.1:8089`. 66 | 67 | ## Use it 68 | 69 | ### Pre-requirement 70 | 71 | You need to have a base Splunk image, you can use [outcoldman/splunk](https://github.com/outcoldman/docker-splunk) 72 | as base image. Build it and tag it as `splunk:latest` on the machine where you will build this image. 73 | 74 | ### Deploy 75 | 76 | #### On docker instance 77 | 78 | > NOTE2: If you are using Docker for Mac - it allocates just 2Gb by default, not enough for this demo. Set more. Maybe 8Gb. 79 | 80 | ``` 81 | cd ./examples/docker 82 | ``` 83 | 84 | This folder has two docker-compose files. One which does not require License Master and Splunk Enterprise License 85 | `docker-compose.yml` and second is an extension for the first one, which adds License Master node. `Makefile` in this folder 86 | deals with how `docker-compose` needs to be invoked. 87 | 88 | If you have Splunk Enterprise License copy it in this folder (make sure that license files have extension `*.lic`) and use 89 | all commands with `-lm` suffix. 90 | 91 | Build image. 92 | 93 | ``` 94 | make build[-lm] 95 | ``` 96 | 97 | Deploy instances. 98 | 99 | ``` 100 | make deploy[-lm] 101 | ``` 102 | 103 | Watch for status of deployment: 104 | - Watch for `docker-compose logs -f shc-member` for the line `Successfully bootstrapped this node as the captain with the given servers.`. 105 | This will mean that SHC is bootstrapped. 106 | - Open Cluster Master web on `http://:9000` and check `Indexer Clustering: Master Node` page 107 | that Indexes are replicated and ready for search. 108 | - Open SHC member and check that you see logs from all instances `index="_internal" | stats count by host`. 109 | 110 | To clean use 111 | 112 | ``` 113 | make clean[-lm] 114 | ``` 115 | 116 | #### On docker swarm 117 | 118 | > NOTE1: Splunk Enterprise License is required 119 | > NOTE2: You have to use docker registry to be sure that each instance will have access to images built by you. 120 | > Or you can publish image on every swarm instance manually. Specify path to your registry with 121 | > environment variable `SPLUNK_CLUSTER_DOCKER_IMAGE_PATH=registry.yourcompany.com/$USER` 122 | 123 | ``` 124 | cd ./examples/docker-swarm-mode 125 | ``` 126 | 127 | Copy Splunk Enterprise license (if you have) in this folder (make sure that license files have extension `*.lic`). 128 | 129 | Prepare swarm. This command will create 5 docker-machine instances in VirtualBox. 3 of them will be used in Docker Swarm 130 | right away, 2 can be added later 131 | 132 | ``` 133 | make setup 134 | ``` 135 | 136 | To use Swarm you need to have access to the Docker registry, specify path to registry and path to image using. 137 | If you will not specify anything it will publish image to hub.docker.com/u/$USER/splunk-cluster (create your repo at 138 | hub.docker.com) 139 | 140 | ``` 141 | export SPLUNK_CLUSTER_DOCKER_IMAGE_PATH=registry.yourcompany.com/$USER 142 | ``` 143 | 144 | Login to your registry (if it is required) 145 | 146 | ``` 147 | docker login registry.yourcompany.com 148 | ``` 149 | 150 | Build image. 151 | 152 | ``` 153 | make build 154 | ``` 155 | 156 | Publish image to your registry 157 | 158 | ``` 159 | make push 160 | ``` 161 | 162 | Deploy cluster. 163 | 164 | ``` 165 | make deploy 166 | ``` 167 | 168 | You can add two more nodes to the Swarm cluster by invoking 169 | 170 | ``` 171 | make setup-add-2 172 | ``` 173 | 174 | To clean splunk cluster (including volumes) use 175 | 176 | ``` 177 | make clean-all 178 | ``` 179 | 180 | To clean images (in case if you want to rebuild) 181 | 182 | ``` 183 | make clean-images 184 | ``` 185 | 186 | To download image on each docker instance 187 | 188 | ``` 189 | make download-image 190 | ``` 191 | 192 | To remove all docker machines use 193 | 194 | ``` 195 | make setup-clean 196 | ``` 197 | 198 | To distribute applications with Cluster Master to the Indexers 199 | 200 | ``` 201 | docker cp my_app $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master):/opt/splunk/etc/master-apps/ 202 | docker exec $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master) entrypoint.sh chown -R splunk:splunk /opt/splunk/etc/master-apps/my_app 203 | docker exec $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master) entrypoint.sh splunk apply cluster-bundle --auth admin:changeme --answer-yes 204 | ``` 205 | 206 | To distribute application with SHC Deployer to SHC Members 207 | 208 | ``` 209 | docker cp my_app $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master):/opt/splunk/etc/shcluster/apps/ 210 | docker exec $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master) entrypoint.sh chown -R splunk:splunk /opt/splunk/etc/shcluster/apps/my_app 211 | docker exec $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master) entrypoint.sh splunk apply shcluster-bundle -restart true --answer-yes -target https://shc-member-01:8089 -auth admin:changeme 212 | ``` 213 | 214 | ``` 215 | docker service create \ 216 | --name cadvisor \ 217 | --mode global \ 218 | --container-label splunk.cluster=cadvisor \ 219 | --label splunk.cluster=cadvisor \ 220 | --network splunk \ 221 | --with-registry-auth \ 222 | --publish 8080:8080 \ 223 | --mount "type=bind,source=/,target=/rootfs,readonly=true" \ 224 | --mount "type=bind,source=/var/run,target=/var/run,readonly=false" \ 225 | --mount "type=bind,source=/sys,target=/sys,readonly=true" \ 226 | --mount "type=bind,source=/var/lib/docker/,target=/var/lib/docker/,readonly=true" \ 227 | $SPLUNK_CLUSTER_DOCKER_IMAGE_PATH/cadvisor \ 228 | -storage_driver=splunk \ 229 | -storage_driver_splunk_insecureskipverify=true \ 230 | -storage_driver_splunk_source=cadvisor \ 231 | -storage_driver_splunk_token=EF211A51-D6AC-4045-8CD6-F730939AC518 \ 232 | -storage_driver_splunk_url=https://cluster-slave:8088 233 | ``` 234 | 235 | #### On kubernetes 236 | 237 | > TODO 238 | 239 | ### Examples after setup 240 | 241 | #### Install application on SHC using SHC Deployer 242 | 243 | ``` 244 | docker cp ~/Downloads/splunk_app_aws shc-deployer:/opt/splunk/etc/shcluster/apps/ 245 | docker exec shc-deployer entrypoint.sh chown -R splunk:splunk /opt/splunk/etc/shcluster/apps/ 246 | docker exec shc-deployer entrypoint.sh splunk apply shcluster-bundle -restart true --answer-yes -target https://$(docker ps --filter=label=splunk.cluster=shc-member -q | head -1):8089 -auth admin:changeme 247 | ``` 248 | -------------------------------------------------------------------------------- /examples/docker-swarm-mode/Makefile: -------------------------------------------------------------------------------- 1 | export SPLUNK_CLUSTER_VERSION ?= 6.4.3 2 | export SPLUNK_CLUSTER_DOCKER_IMAGE_PATH ?= ${USER} 3 | # If you don't want (cannot) delete old volumes, just use SALT to use new one 4 | SALT ?= 5 | 6 | setup-clean: 7 | seq 0 5 | xargs -I @ docker-machine rm -f splunk@ 8 | 9 | setup: 10 | seq 0 5 | xargs -P 6 -I @ bash -c "docker-machine create \ 11 | --driver=virtualbox \ 12 | --virtualbox-disk-size=200000 \ 13 | --virtualbox-memory=1536 \ 14 | --virtualbox-cpu-count=2 \ 15 | splunk@" 16 | seq 0 5 | xargs -P 6 -I @ docker-machine regenerate-certs --force splunk@ 17 | (eval $$(docker-machine env splunk0) && docker swarm init --advertise-addr $$(docker-machine ip splunk0)) 18 | seq 3 | xargs -I @ bash -c 'eval $$(docker-machine env splunk@) && docker swarm join --listen-addr $$(docker-machine ip splunk@) $$(docker-machine ip splunk0):2377 --token $$(eval $$(docker-machine env splunk0) && docker swarm join-token -q worker )' 19 | (eval $$(docker-machine env splunk0) && docker network create --driver=overlay splunk) 20 | seq 3 | xargs -I @ bash -c 'eval $$(docker-machine env splunk0) && docker node update --label-add index=@ splunk@' 21 | 22 | setup-add-2: 23 | seq 4 5 | xargs -I @ bash -c 'eval $$(docker-machine env splunk@) && docker swarm join --listen-addr $$(docker-machine ip splunk@) $$(docker-machine ip splunk0):2377 --token $$(eval $$(docker-machine env splunk0) && docker swarm join-token -q worker )' 24 | seq 4 5 | xargs -I @ bash -c 'eval $$(docker-machine env splunk0) && docker node update --label-add index=@ splunk@' 25 | 26 | setup-remove-2: 27 | -seq 4 5 | xargs -I @ bash -c 'eval $$(docker-machine env splunk@) && docker swarm leave' 28 | -seq 4 5 | xargs -I @ bash -c 'eval $$(docker-machine env splunk0) && docker node rm splunk@' 29 | 30 | build: 31 | cd ./../../splunk-cluster && docker build -t $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:$(SPLUNK_CLUSTER_VERSION) -t $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:latest . 32 | 33 | push: 34 | docker push $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:$(SPLUNK_CLUSTER_VERSION) 35 | docker push $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:latest 36 | 37 | download-image: 38 | seq 0 5 | xargs -P 6 -I @ bash -c 'eval $$(docker-machine env splunk@); docker pull $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:$(SPLUNK_CLUSTER_VERSION)' 39 | 40 | download-image-using-swarm: 41 | docker service create \ 42 | --mode global \ 43 | --name download-image \ 44 | --with-registry-auth \ 45 | $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:$(SPLUNK_CLUSTER_VERSION) entrypoint.sh sleep infinity 46 | sleep 5 47 | while docker service ps download-image | grep Preparing 1>/dev/null 2>&1; do sleep 1; done 48 | docker service rm download-image 49 | 50 | deploy-cluster-master: 51 | ls *.lic 1> /dev/null 2>&1; 52 | docker service create \ 53 | --constraint "node.id == $$(docker node ls | grep '*' | awk '{print($$1)}')" \ 54 | --mode replicated \ 55 | --replicas 1 \ 56 | --name cluster-master \ 57 | --container-label splunk.cluster=cluster-master \ 58 | --label splunk.cluster=cluster-master \ 59 | --network splunk \ 60 | --mount "type=volume,source=cluster-master-etc$(SALT),destination=/opt/splunk/etc" \ 61 | --mount "type=volume,source=cluster-master-var$(SALT),destination=/opt/splunk/var" \ 62 | --with-registry-auth \ 63 | --publish 9000:8000 \ 64 | --env "SPLUNK_START_ARGS=--accept-license --answer-yes --no-prompt" \ 65 | --env "SPLUNK_BEFORE_START_CMD_1=version \$$SPLUNK_START_ARGS" \ 66 | --env "SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --configure" \ 67 | --env "SPLUNK_BEFORE_START_CMD_3=cmd python /opt/splunk/bin/splunk_setup.py --add-licenses /opt/" \ 68 | --env "SPLUNK_ADD_1=role service_discovery_role -auth admin:changeme" \ 69 | --env "SPLUNK_ADD_2=user service_discovery_user -password service_discovery_password -role service_discovery_role -auth admin:changeme" \ 70 | --env "SPLUNK_CMD_1=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://shc-member-01:8089' '(shc_member|shc_captain)'" \ 71 | --env "SPLUNK_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://shc-member-02:8089' '(shc_member|shc_captain)'" \ 72 | --env "SPLUNK_CMD_3=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://shc-member-03:8089' '(shc_member|shc_captain)'" \ 73 | --env "SPLUNK_CMD_4=add search-server shc-member-01:8089 -remoteUsername admin -remotePassword changed -auth admin:changeme" \ 74 | --env "SPLUNK_CMD_5=add search-server shc-member-02:8089 -remoteUsername admin -remotePassword changed -auth admin:changeme" \ 75 | --env "SPLUNK_CMD_6=add search-server shc-member-03:8089 -remoteUsername admin -remotePassword changed -auth admin:changeme" \ 76 | --env "SPLUNK_CMD_7=status" \ 77 | --env "CONF__inputs__default__host=cluster-master" \ 78 | --env "CONF__inputs__http__useDeploymentServer=true" \ 79 | --env "CONF__outputs__indexAndForward__index=false" \ 80 | --env "CONF__outputs__indexer_discovery:search_head_peers__master_uri=https://cluster-master:8089" \ 81 | --env "CONF__outputs__indexer_discovery:search_head_peers__pass4SymmKey=indexdiscovery-changeme" \ 82 | --env "CONF__outputs__tcpout:splunk_search_head_peers_group__autoLB=true" \ 83 | --env "CONF__outputs__tcpout:splunk_search_head_peers_group__indexerDiscovery=search_head_peers" \ 84 | --env "CONF__outputs__tcpout__defaultGroup=splunk_search_head_peers_group" \ 85 | --env "CONF__outputs__tcpout__forwardedindex.filter.disable =true" \ 86 | --env "CONF__outputs__tcpout__indexAndForward=false" \ 87 | --env "CONF__restmap__broker:broker__requireAuthentication=true" \ 88 | --env "CONF__collections__service_discovery__disabled=false" \ 89 | --env "CONF__meta__collections/service_discovery__access=read : [ service_discovery_role ], write : [ service_discovery_role ]" \ 90 | --env "CONF__server__clustering__cluster_label=cluster" \ 91 | --env "CONF__server__clustering__mode=master" \ 92 | --env "CONF__server__clustering__pass4SymmKey=clustering-changeme" \ 93 | --env "CONF__server__clustering__replication_factor=3" \ 94 | --env "CONF__server__clustering__search_factor=2" \ 95 | --env "CONF__server__general__serverName=cluster-master" \ 96 | --env "CONF__server__general__pass4SymmKey=general-changeme" \ 97 | --env "CONF__server__deployment__pass4SymmKey=deployment-changeme" \ 98 | --env "CONF__server__indexer_discovery__pass4SymmKey=indexdiscovery-changeme" \ 99 | --env "CONF__server__shclustering__id=776D2949-B2DA-405E-96B3-B6688C87AB7D" \ 100 | --env "CONF__server__shclustering__pass4SymmKey=shclustering-changeme" \ 101 | --env "CONF__server__shclustering__shcluster_label=shcluster" \ 102 | --env "CONF__serverclass__global__crossServerChecksum=true" \ 103 | --env "CONF__serverclass__serverClass:data-collector-hec:app:splunk_httpinput__restartSplunkd=true" \ 104 | --env "CONF__serverclass__serverClass:data-collector-hec__whitelist.0=data-collector-hec" \ 105 | --env "CONF__web__settings__login_content=Cluster Master" \ 106 | --env "CONF__etc/master-apps/cluster-deployment__indexes__main__repFactor=auto" \ 107 | --env "CONF__etc/master-apps/cluster-deployment__indexes__history__repFactor=auto" \ 108 | --env "CONF__etc/master-apps/cluster-deployment__indexes__summary__repFactor=auto" \ 109 | --env "CONF__etc/master-apps/cluster-deployment__indexes__splunklogger__repFactor=auto" \ 110 | --env "CONF__etc/master-apps/cluster-deployment__indexes___internal__repFactor=auto" \ 111 | --env "CONF__etc/master-apps/cluster-deployment__indexes___audit__repFactor=auto" \ 112 | --env "CONF__etc/master-apps/cluster-deployment__indexes___thefishbucket__repFactor=auto" \ 113 | --env "CONF__etc/deployment-apps/splunk_httpinput__inputs__http__dedicatedIoThreads=2" \ 114 | --env "CONF__etc/deployment-apps/splunk_httpinput__inputs__http__maxThreads=0" \ 115 | --env "CONF__etc/deployment-apps/splunk_httpinput__inputs__http__allowSslRenegotiation=true" \ 116 | --env "CONF__etc/deployment-apps/splunk_httpinput__inputs__http__disabled=false" \ 117 | --env "CONF__etc/deployment-apps/splunk_httpinput__inputs__http__useDeploymentServer=false" \ 118 | --env "CONF__etc/deployment-apps/splunk_httpinput__inputs__http__port=8088" \ 119 | --env "CONF__etc/deployment-apps/splunk_httpinput__inputs__http__allowSslCompression=true" \ 120 | --env "CONF__etc/deployment-apps/splunk_httpinput__inputs__http__sslVersions=*,-ssl2" \ 121 | --env "CONF__etc/deployment-apps/splunk_httpinput__inputs__http__enableSSL=true" \ 122 | --env "CONF__etc/deployment-apps/splunk_httpinput__inputs__http__maxSockets=0" \ 123 | --env "CONF__etc/deployment-apps/splunk_httpinput__inputs__http://default__token=EF211A51-D6AC-4045-8CD6-F730939AC518" \ 124 | --env "CONF__etc/deployment-apps/splunk_httpinput__inputs__http://default__disabled=False" \ 125 | $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:$(SPLUNK_CLUSTER_VERSION) 126 | while [[ "$$(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master)" == "" ]]; do sleep 1; done 127 | ls *.lic | xargs -I @ docker cp @ $$(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master):/opt/ 128 | 129 | deploy-shc-member: 130 | for number in $(shell seq 3); do \ 131 | docker service create \ 132 | --constraint "node.labels.index == $${number}" \ 133 | --name shc-member-0$${number} \ 134 | --mode replicated \ 135 | --replicas 1 \ 136 | --container-label splunk.cluster=shc-member \ 137 | --label splunk.cluster=shc-member \ 138 | --network splunk \ 139 | --mount "type=volume,source=shc-member-0$${number}-etc$(SALT),destination=/opt/splunk/etc" \ 140 | --mount "type=volume,source=shc-member-0$${number}-var$(SALT),destination=/opt/splunk/var" \ 141 | --with-registry-auth \ 142 | --publish 800$${number}:8000 \ 143 | --env "SPLUNK_START_ARGS=--accept-license --answer-yes --no-prompt" \ 144 | --env "SPLUNK_BEFORE_START_CMD_1=version \$$SPLUNK_START_ARGS" \ 145 | --env "SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://cluster-master:8089' cluster_master kv_store license_master" \ 146 | --env "SPLUNK_BEFORE_START_CMD_3=cmd python /opt/splunk/bin/splunk_setup.py --configure" \ 147 | --env "SPLUNK_BEFORE_START_CMD_4=edit user admin -password changed -role admin -auth admin:changeme" \ 148 | --env "SPLUNK_BEFORE_START_CMD_5=cmd python -c 'open(\"/opt/splunk/etc/.ui_login\", \"a\").close()'" \ 149 | --env "SPLUNK_CMD_1=cmd python /opt/splunk/bin/splunk_setup.py --shc-autobootstrap 3 'https://shc-member-0$${number}:8089' admin changed 'https://cluster-master:8089/servicesNS/nobody/system/storage/collections/data/service_discovery' service_discovery_user service_discovery_password" \ 150 | --env "SPLUNK_CMD_2=status" \ 151 | --env "CONF__etc/apps/splunk_management_console__app__install__state=disabled" \ 152 | --env "CONF__outputs__indexAndForward__index=false" \ 153 | --env "CONF__outputs__indexer_discovery:search_head_peers__master_uri=https://cluster-master:8089" \ 154 | --env "CONF__outputs__indexer_discovery:search_head_peers__pass4SymmKey=indexdiscovery-changeme" \ 155 | --env "CONF__outputs__tcpout:splunk_search_head_peers_group__autoLB=true" \ 156 | --env "CONF__outputs__tcpout:splunk_search_head_peers_group__indexerDiscovery=search_head_peers" \ 157 | --env "CONF__outputs__tcpout__defaultGroup=splunk_search_head_peers_group" \ 158 | --env "CONF__outputs__tcpout__forwardedindex.filter.disable =true" \ 159 | --env "CONF__outputs__tcpout__indexAndForward=false" \ 160 | --env "CONF__server__general__pass4SymmKey=general-changeme" \ 161 | --env "CONF__server__general__serverName=shc-member-0$${number}" \ 162 | --env "CONF__server__license__master_uri=https://cluster-master:8089" \ 163 | --env "CONF__server__clustering__cluster_label=cluster" \ 164 | --env "CONF__server__clustering__master_uri=https://cluster-master:8089" \ 165 | --env "CONF__server__clustering__mode=searchhead" \ 166 | --env "CONF__server__clustering__pass4SymmKey=clustering-changeme" \ 167 | --env "CONF__server__replication_port://9889__disabled=false" \ 168 | --env "CONF__server__shclustering__id=776D2949-B2DA-405E-96B3-B6688C87AB7D" \ 169 | --env "CONF__server__shclustering__conf_deploy_fetch_url=https://cluster-master:8089" \ 170 | --env "CONF__server__shclustering__disabled=false" \ 171 | --env "CONF__server__shclustering__election_timeout_ms=10000" \ 172 | --env "CONF__server__shclustering__mgmt_uri=https://shc-member-0$${number}:8089" \ 173 | --env "CONF__server__shclustering__pass4SymmKey=shclustering-changeme" \ 174 | --env "CONF__server__shclustering__replication_factor=3" \ 175 | --env "CONF__server__shclustering__shcluster_label=shcluster" \ 176 | --env "CONF__web__settings__login_content=SHC Member" \ 177 | --env "CONF__inputs__default__host=shc-member-0$${number}" \ 178 | $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:$(SPLUNK_CLUSTER_VERSION); \ 179 | done 180 | 181 | deploy-cluster-slave: 182 | for number in $(shell seq 3); do \ 183 | docker service create \ 184 | --constraint "node.labels.index == $${number}" \ 185 | --name cluster-slave-0$${number} \ 186 | --mode replicated \ 187 | --replicas 1 \ 188 | --container-label splunk.cluster=cluster-slave \ 189 | --label splunk.cluster=cluster-slave \ 190 | --network splunk \ 191 | --mount "type=volume,source=cluster-slave-etc$(SALT),destination=/opt/splunk/etc" \ 192 | --mount "type=volume,source=cluster-slave-var$(SALT),destination=/opt/splunk/var" \ 193 | --with-registry-auth \ 194 | --publish 808$${number}:8088 \ 195 | --env "SPLUNK_START_ARGS=--accept-license --answer-yes --no-prompt" \ 196 | --env "SPLUNK_BEFORE_START_CMD_1=version \$$SPLUNK_START_ARGS" \ 197 | --env "SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://cluster-master:8089' cluster_master license_master" \ 198 | --env "SPLUNK_BEFORE_START_CMD_3=cmd python /opt/splunk/bin/splunk_setup.py --configure" \ 199 | --env "SPLUNK_CMD_1=status" \ 200 | --env "CONF__etc/apps/splunk_management_console__app__install__state=disabled" \ 201 | --env "CONF__server__general__serverName=cluster-slave-0$${number}" \ 202 | --env "CONF__server__general__pass4SymmKey=general-changeme" \ 203 | --env "CONF__server__license__master_uri=https://cluster-master:8089" \ 204 | --env "CONF__server__kvstore__disabled=true" \ 205 | --env "CONF__web__settings__startwebserver=false" \ 206 | --env "CONF__server__clustering__cluster_label=cluster" \ 207 | --env "CONF__server__clustering__mode=slave" \ 208 | --env "CONF__server__clustering__pass4SymmKey=clustering-changeme" \ 209 | --env "CONF__server__clustering__master_uri=https://cluster-master:8089" \ 210 | --env "CONF__server__replication_port://9888__disabled=false" \ 211 | --env "CONF__server__clustering__register_register_replication_address=cluster-slave-0$${number}" \ 212 | --env "CONF__server__clustering__register_replication_address=cluster-slave-0$${number}" \ 213 | --env "CONF__server__clustering__register_search_address=cluster-slave-0$${number}" \ 214 | --env "CONF__inputs__splunktcp://9997__disabled=false" \ 215 | --env "CONF__inputs__default__host=cluster-slave-0$${number}" \ 216 | --env "CONF__server__deployment__pass4SymmKey=deployment-changeme" \ 217 | --env "CONF__deploymentclient__deployment-client__disabled=false" \ 218 | --env "CONF__deploymentclient__deployment-client__clientName=data-collector-hec" \ 219 | --env "CONF__deploymentclient__target-broker:deploymentServer__targetUri=cluster-master:8089" \ 220 | $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:$(SPLUNK_CLUSTER_VERSION); \ 221 | done 222 | 223 | deploy-cluster-slave-2-more: 224 | for number in $(shell seq 4 5); do \ 225 | docker service create \ 226 | --constraint "node.labels.index == $${number}" \ 227 | --name cluster-slave-0$${number} \ 228 | --mode replicated \ 229 | --replicas 1 \ 230 | --container-label splunk.cluster=cluster-slave \ 231 | --label splunk.cluster=cluster-slave \ 232 | --network splunk \ 233 | --mount "type=volume,source=cluster-slave-etc$(SALT),destination=/opt/splunk/etc" \ 234 | --mount "type=volume,source=cluster-slave-var$(SALT),destination=/opt/splunk/var" \ 235 | --with-registry-auth \ 236 | --publish 808$${number}:8088 \ 237 | --env "SPLUNK_START_ARGS=--accept-license --answer-yes --no-prompt" \ 238 | --env "SPLUNK_BEFORE_START_CMD_1=version \$$SPLUNK_START_ARGS" \ 239 | --env "SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://cluster-master:8089' cluster_master license_master" \ 240 | --env "SPLUNK_BEFORE_START_CMD_3=cmd python /opt/splunk/bin/splunk_setup.py --configure" \ 241 | --env "SPLUNK_CMD_1=status" \ 242 | --env "CONF__etc/apps/splunk_management_console__app__install__state=disabled" \ 243 | --env "CONF__server__general__serverName=cluster-slave-0$${number}" \ 244 | --env "CONF__server__general__pass4SymmKey=general-changeme" \ 245 | --env "CONF__server__license__master_uri=https://cluster-master:8089" \ 246 | --env "CONF__server__kvstore__disabled=true" \ 247 | --env "CONF__web__settings__startwebserver=false" \ 248 | --env "CONF__server__clustering__cluster_label=cluster" \ 249 | --env "CONF__server__clustering__mode=slave" \ 250 | --env "CONF__server__clustering__pass4SymmKey=clustering-changeme" \ 251 | --env "CONF__server__clustering__master_uri=https://cluster-master:8089" \ 252 | --env "CONF__server__replication_port://9888__disabled=false" \ 253 | --env "CONF__server__clustering__register_register_replication_address=cluster-slave-0$${number}" \ 254 | --env "CONF__server__clustering__register_replication_address=cluster-slave-0$${number}" \ 255 | --env "CONF__server__clustering__register_search_address=cluster-slave-0$${number}" \ 256 | --env "CONF__inputs__splunktcp://9997__disabled=false" \ 257 | --env "CONF__inputs__default__host=cluster-slave-0$${number}" \ 258 | --env "CONF__server__deployment__pass4SymmKey=deployment-changeme" \ 259 | --env "CONF__deploymentclient__deployment-client__disabled=false" \ 260 | --env "CONF__deploymentclient__deployment-client__clientName=data-collector-hec" \ 261 | --env "CONF__deploymentclient__target-broker:deploymentServer__targetUri=cluster-master:8089" \ 262 | $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:$(SPLUNK_CLUSTER_VERSION); \ 263 | done 264 | 265 | deploy: deploy-cluster-master deploy-shc-member deploy-cluster-slave 266 | 267 | clean: 268 | -docker service rm cluster-master 269 | -seq 3 | xargs -P 3 -I @ docker service rm shc-member-0@ 270 | -seq 3 | xargs -P 3 -I @ docker service rm cluster-slave-0@ 271 | 272 | clean-images: 273 | -seq 0 5 | xargs -P 6 -I @ bash -c 'eval $$(docker-machine env splunk@); docker rmi -f $$(docker images -q);' 274 | 275 | clean-volumes: 276 | -seq 0 5 | xargs -P 6 -I @ bash -c 'eval $$(docker-machine env splunk@); docker kill $$(docker ps -q); docker rm -v $$(docker ps -aq); docker volume rm $$(docker volume ls -q)' 277 | 278 | clean-all: clean clean-volumes 279 | -------------------------------------------------------------------------------- /examples/docker/Makefile: -------------------------------------------------------------------------------- 1 | # With license master 2 | 3 | export SPLUNK_CLUSTER_VERSION ?= 6.4.3 4 | export SPLUNK_CLUSTER_DOCKER_IMAGE_PATH ?= ${USER} 5 | 6 | clean-lm: 7 | docker-compose -f docker-compose.yml -f docker-compose.license-master.yml kill 8 | docker-compose -f docker-compose.yml -f docker-compose.license-master.yml rm -v -f 9 | 10 | bundle: 11 | docker-compose -f docker-compose.yml -f docker-compose.license-master.yml bundle 12 | 13 | build-lm: build 14 | 15 | push-lm: push 16 | 17 | deploy-lm: 18 | #docker-compose -f docker-compose.yml -f docker-compose.license-master.yml pull 19 | @echo "Verifying that at least one *.lic file exist" 20 | ls *.lic 1> /dev/null 2>&1; 21 | docker-compose -f docker-compose.yml -f docker-compose.license-master.yml up -d cluster-master 22 | ls *.lic | xargs -I @ docker cp @ $$(docker ps -qa --filter=label=splunk.cluster=cluster-master):/opt/ 23 | docker-compose -f docker-compose.yml -f docker-compose.license-master.yml up -d 24 | @echo "Use 'docker-compose -f docker-compose.yml -f docker-compose.license-master.yml logs -f cluster-master' to wait for Initialized cluster-master as Cluster Master" 25 | 26 | # Without license master 27 | 28 | clean: 29 | docker-compose -f docker-compose.yml kill 30 | docker-compose -f docker-compose.yml rm -v -f 31 | 32 | build: 33 | cd ./../../splunk-cluster && docker build -t $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:$(SPLUNK_CLUSTER_VERSION) -t $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:latest . 34 | 35 | push: build 36 | docker push $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:$(SPLUNK_CLUSTER_VERSION) 37 | docker push $(SPLUNK_CLUSTER_DOCKER_IMAGE_PATH)/splunk-cluster:latest 38 | 39 | deploy: 40 | docker-compose -f docker-compose.yml -f docker-compose.license-master.yml pull 41 | docker-compose -f docker-compose.yml up -d 42 | @echo "Use 'docker-compose -f docker-compose.yml -f docker-compose.license-master.yml logs -f cluster-master' to wait for Initialized cluster-master as Cluster Master" 43 | -------------------------------------------------------------------------------- /examples/docker/cluster-master.env_file: -------------------------------------------------------------------------------- 1 | SPLUNK_START_ARGS=--accept-license --answer-yes --no-prompt 2 | SPLUNK_BEFORE_START_CMD_1=version $SPLUNK_START_ARGS 3 | SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --configure 4 | SPLUNK_ADD_1=role service_discovery_role -auth admin:changeme 5 | SPLUNK_ADD_2=user service_discovery_user -password service_discovery_password -role service_discovery_role -auth admin:changeme 6 | SPLUNK_CMD_1=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://shc-member-01:8089' '(shc_member|shc_captain)' 7 | SPLUNK_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://shc-member-02:8089' '(shc_member|shc_captain)' 8 | SPLUNK_CMD_3=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://shc-member-03:8089' '(shc_member|shc_captain)' 9 | SPLUNK_CMD_4=add search-server shc-member-01:8089 -remoteUsername admin -remotePassword changed -auth admin:changeme 10 | SPLUNK_CMD_5=add search-server shc-member-02:8089 -remoteUsername admin -remotePassword changed -auth admin:changeme 11 | SPLUNK_CMD_6=add search-server shc-member-03:8089 -remoteUsername admin -remotePassword changed -auth admin:changeme 12 | SPLUNK_CMD_7=status 13 | SPLUNK_HEALTHCHECK_SPLUNKD=https://127.0.0.1:8089 14 | SPLUNK_HEALTHCHECK_SPLUNKWEB=http://127.0.0.1:8000 15 | CONF__inputs__default__host=ENV($HOSTNAME) 16 | CONF__server__general__serverName=ENV($HOSTNAME) 17 | CONF__inputs__http__useDeploymentServer=true 18 | CONF__outputs__indexAndForward__index=false 19 | CONF__outputs__indexer_discovery:search_head_peers__master_uri=ENV(https://$HOSTNAME:8089) 20 | CONF__outputs__indexer_discovery:search_head_peers__pass4SymmKey=indexdiscovery-changeme 21 | CONF__outputs__tcpout:splunk_search_head_peers_group__autoLB=true 22 | CONF__outputs__tcpout:splunk_search_head_peers_group__indexerDiscovery=search_head_peers 23 | CONF__outputs__tcpout__defaultGroup=splunk_search_head_peers_group 24 | CONF__outputs__tcpout__forwardedindex.filter.disable =true 25 | CONF__outputs__tcpout__indexAndForward=false 26 | CONF__restmap__broker:broker__requireAuthentication=true 27 | CONF__collections__service_discovery__disabled=false 28 | CONF__meta__collections/service_discovery__access=read : [ service_discovery_role ], write : [ service_discovery_role ] 29 | CONF__server__clustering__cluster_label=cluster 30 | CONF__server__clustering__mode=master 31 | CONF__server__clustering__pass4SymmKey=clustering-changeme 32 | CONF__server__clustering__replication_factor=3 33 | CONF__server__clustering__search_factor=2 34 | CONF__server__deployment__pass4SymmKey=deployment-changeme 35 | CONF__server__indexer_discovery__pass4SymmKey=indexdiscovery-changeme 36 | CONF__server__shclustering__id=776D2949-B2DA-405E-96B3-B6688C87AB7D 37 | CONF__server__shclustering__pass4SymmKey=shclustering-changeme 38 | CONF__server__shclustering__shcluster_label=shcluster 39 | CONF__serverclass__global__crossServerChecksum=true 40 | CONF__serverclass__serverClass:data-collector-hec:app:splunk_httpinput__restartIfNeeded=true 41 | CONF__serverclass__serverClass:data-collector-hec__whitelist.0=data-collector-hec 42 | CONF__web__settings__login_content=Cluster Master 43 | CONF__etc/master-apps/cluster-deployment__indexes__main__repFactor=auto 44 | CONF__etc/master-apps/cluster-deployment__indexes__history__repFactor=auto 45 | CONF__etc/master-apps/cluster-deployment__indexes__summary__repFactor=auto 46 | CONF__etc/master-apps/cluster-deployment__indexes__splunklogger__repFactor=auto 47 | CONF__etc/master-apps/cluster-deployment__indexes___internal__repFactor=auto 48 | CONF__etc/master-apps/cluster-deployment__indexes___audit__repFactor=auto 49 | CONF__etc/master-apps/cluster-deployment__indexes___thefishbucket__repFactor=auto 50 | CONF__etc/deployment-apps/splunk_httpinput__inputs__http__dedicatedIoThreads=2 51 | CONF__etc/deployment-apps/splunk_httpinput__inputs__http__maxThreads=0 52 | CONF__etc/deployment-apps/splunk_httpinput__inputs__http__allowSslRenegotiation=true 53 | CONF__etc/deployment-apps/splunk_httpinput__inputs__http__disabled=false 54 | CONF__etc/deployment-apps/splunk_httpinput__inputs__http__useDeploymentServer=false 55 | CONF__etc/deployment-apps/splunk_httpinput__inputs__http__port=8088 56 | CONF__etc/deployment-apps/splunk_httpinput__inputs__http__allowSslCompression=true 57 | CONF__etc/deployment-apps/splunk_httpinput__inputs__http__sslVersions=*,-ssl2 58 | CONF__etc/deployment-apps/splunk_httpinput__inputs__http__enableSSL=true 59 | CONF__etc/deployment-apps/splunk_httpinput__inputs__http__maxSockets=0 60 | CONF__etc/deployment-apps/splunk_httpinput__inputs__http://default__token=EF211A51-D6AC-4045-8CD6-F730939AC518 61 | CONF__etc/deployment-apps/splunk_httpinput__inputs__http://default__disabled=False -------------------------------------------------------------------------------- /examples/docker/cluster-slave.env_file: -------------------------------------------------------------------------------- 1 | SPLUNK_START_ARGS=--accept-license --answer-yes --no-prompt 2 | SPLUNK_BEFORE_START_CMD_1=version $SPLUNK_START_ARGS 3 | SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://cluster-master:8089' cluster_master 4 | SPLUNK_BEFORE_START_CMD_3=cmd python /opt/splunk/bin/splunk_setup.py --configure 5 | SPLUNK_CMD_1=status 6 | SPLUNK_HEALTHCHECK_SPLUNKD=https://127.0.0.1:8089 7 | SPLUNK_HEALTHCHECK_HEC=https://127.0.0.1:8088 8 | CONF__etc/apps/splunk_management_console__app__install__state=disabled 9 | CONF__server__kvstore__disabled=true 10 | CONF__web__settings__startwebserver=false 11 | CONF__server__clustering__cluster_label=cluster 12 | CONF__server__clustering__mode=slave 13 | CONF__server__clustering__pass4SymmKey=clustering-changeme 14 | CONF__server__clustering__master_uri=https://cluster-master:8089 15 | CONF__server__replication_port://9888__disabled=false 16 | CONF__inputs__splunktcp://9997__disabled=false 17 | CONF__server__deployment__pass4SymmKey=deployment-changeme 18 | CONF__deploymentclient__deployment-client__disabled=false 19 | CONF__deploymentclient__deployment-client__clientName=data-collector-hec 20 | CONF__deploymentclient__target-broker:deploymentServer__targetUri=cluster-master:8089 21 | CONF__server__clustering__register_replication_address=ENV($HOSTNAME) 22 | CONF__server__clustering__register_forwarder_address=ENV($HOSTNAME) 23 | CONF__server__clustering__register_search_address=ENV($HOSTNAME) 24 | CONF__inputs__default__host=ENV($HOSTNAME) 25 | CONF__server__general__serverName=ENV($HOSTNAME) -------------------------------------------------------------------------------- /examples/docker/docker-compose.license-master.yml: -------------------------------------------------------------------------------- 1 | version: '2' 2 | services: 3 | 4 | cluster-master: 5 | environment: 6 | - "SPLUNK_BEFORE_START_CMD_3=cmd python /opt/splunk/bin/splunk_setup.py --add-licenses /opt/" 7 | - "CONF__server__general__pass4SymmKey=general-changeme" 8 | 9 | shc-member-01: 10 | environment: 11 | - "SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://cluster-master:8089' cluster_master kv_store license_master" 12 | - "CONF__server__general__pass4SymmKey=general-changeme" 13 | - "CONF__server__license__master_uri=https://cluster-master:8089" 14 | 15 | shc-member-02: 16 | environment: 17 | - "SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://cluster-master:8089' cluster_master kv_store license_master" 18 | - "CONF__server__general__pass4SymmKey=general-changeme" 19 | - "CONF__server__license__master_uri=https://cluster-master:8089" 20 | 21 | shc-member-03: 22 | environment: 23 | - "SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://cluster-master:8089' cluster_master kv_store license_master" 24 | - "CONF__server__general__pass4SymmKey=general-changeme" 25 | - "CONF__server__license__master_uri=https://cluster-master:8089" 26 | 27 | cluster-slave-01: 28 | environment: 29 | - "SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://cluster-master:8089' cluster_master license_master" 30 | - "CONF__server__general__pass4SymmKey=general-changeme" 31 | - "CONF__server__license__master_uri=https://cluster-master:8089" 32 | 33 | cluster-slave-02: 34 | environment: 35 | - "SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://cluster-master:8089' cluster_master license_master" 36 | - "CONF__server__general__pass4SymmKey=general-changeme" 37 | - "CONF__server__license__master_uri=https://cluster-master:8089" 38 | cluster-slave-03: 39 | environment: 40 | - "SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://cluster-master:8089' cluster_master license_master" 41 | - "CONF__server__general__pass4SymmKey=general-changeme" 42 | - "CONF__server__license__master_uri=https://cluster-master:8089" -------------------------------------------------------------------------------- /examples/docker/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '2' 2 | services: 3 | 4 | cluster-master: 5 | image: ${SPLUNK_CLUSTER_DOCKER_IMAGE_PATH}/splunk-cluster:${SPLUNK_CLUSTER_VERSION} 6 | container_name: cluster-master 7 | hostname: cluster-master 8 | env_file: 9 | - cluster-master.env_file 10 | ports: 11 | - 9000:8000 12 | labels: 13 | splunk.cluster: "cluster-master" 14 | networks: 15 | splunk: 16 | aliases: 17 | - cluster-master 18 | restart: always 19 | 20 | shc-member-01: 21 | image: ${SPLUNK_CLUSTER_DOCKER_IMAGE_PATH}/splunk-cluster:${SPLUNK_CLUSTER_VERSION} 22 | container_name: shc-member-01 23 | hostname: shc-member-01 24 | depends_on: 25 | - cluster-master 26 | env_file: 27 | - shc-member.env_file 28 | ports: 29 | - 8001:8000 30 | labels: 31 | splunk.cluster: "shc-member" 32 | networks: 33 | splunk: 34 | aliases: 35 | - shc-member-01 36 | restart: always 37 | 38 | shc-member-02: 39 | image: ${SPLUNK_CLUSTER_DOCKER_IMAGE_PATH}/splunk-cluster:${SPLUNK_CLUSTER_VERSION} 40 | container_name: shc-member-02 41 | hostname: shc-member-02 42 | depends_on: 43 | - cluster-master 44 | env_file: 45 | - shc-member.env_file 46 | ports: 47 | - 8002:8000 48 | labels: 49 | splunk.cluster: "shc-member" 50 | networks: 51 | splunk: 52 | aliases: 53 | - shc-member-02 54 | restart: always 55 | 56 | shc-member-03: 57 | image: ${SPLUNK_CLUSTER_DOCKER_IMAGE_PATH}/splunk-cluster:${SPLUNK_CLUSTER_VERSION} 58 | container_name: shc-member-03 59 | hostname: shc-member-03 60 | depends_on: 61 | - cluster-master 62 | env_file: 63 | - shc-member.env_file 64 | ports: 65 | - 8003:8000 66 | labels: 67 | splunk.cluster: "shc-member" 68 | networks: 69 | splunk: 70 | aliases: 71 | - shc-member-03 72 | restart: always 73 | 74 | cluster-slave-01: 75 | image: ${SPLUNK_CLUSTER_DOCKER_IMAGE_PATH}/splunk-cluster:${SPLUNK_CLUSTER_VERSION} 76 | container_name: cluster-slave-01 77 | hostname: cluster-slave-01 78 | depends_on: 79 | - cluster-master 80 | env_file: 81 | - cluster-slave.env_file 82 | ports: 83 | - 8081:8088 84 | labels: 85 | splunk.cluster: "cluster-slave" 86 | networks: 87 | - splunk 88 | restart: always 89 | 90 | cluster-slave-02: 91 | image: ${SPLUNK_CLUSTER_DOCKER_IMAGE_PATH}/splunk-cluster:${SPLUNK_CLUSTER_VERSION} 92 | container_name: cluster-slave-02 93 | hostname: cluster-slave-02 94 | depends_on: 95 | - cluster-master 96 | env_file: 97 | - cluster-slave.env_file 98 | ports: 99 | - 8082:8088 100 | labels: 101 | splunk.cluster: "cluster-slave" 102 | networks: 103 | - splunk 104 | restart: always 105 | 106 | cluster-slave-03: 107 | image: ${SPLUNK_CLUSTER_DOCKER_IMAGE_PATH}/splunk-cluster:${SPLUNK_CLUSTER_VERSION} 108 | container_name: cluster-slave-03 109 | hostname: cluster-slave-03 110 | depends_on: 111 | - cluster-master 112 | env_file: 113 | - cluster-slave.env_file 114 | ports: 115 | - 8083:8088 116 | labels: 117 | splunk.cluster: "cluster-slave" 118 | networks: 119 | - splunk 120 | restart: always 121 | 122 | networks: 123 | splunk: -------------------------------------------------------------------------------- /examples/docker/shc-member.env_file: -------------------------------------------------------------------------------- 1 | SPLUNK_START_ARGS=--accept-license --answer-yes --no-prompt 2 | SPLUNK_BEFORE_START_CMD_1=version $SPLUNK_START_ARGS 3 | SPLUNK_BEFORE_START_CMD_2=cmd python /opt/splunk/bin/splunk_setup.py --wait-splunk 'https://cluster-master:8089' cluster_master kv_store 4 | SPLUNK_BEFORE_START_CMD_3=cmd python /opt/splunk/bin/splunk_setup.py --configure 5 | SPLUNK_BEFORE_START_CMD_4=edit user admin -password changed -role admin -auth admin:changeme 6 | SPLUNK_BEFORE_START_CMD_5=cmd python -c 'open("/opt/splunk/etc/.ui_login", "a").close()' 7 | SPLUNK_CMD_1=cmd python /opt/splunk/bin/splunk_setup.py --shc-autobootstrap 3 https://$HOSTNAME:8089 admin changed 'https://cluster-master:8089/servicesNS/nobody/system/storage/collections/data/service_discovery' service_discovery_user service_discovery_password 8 | SPLUNK_CMD_2=status 9 | SPLUNK_HEALTHCHECK_SPLUNKD=https://127.0.0.1:8089 10 | SPLUNK_HEALTHCHECK_SPLUNKWEB=http://127.0.0.1:8000 11 | CONF__inputs__default__host=ENV($HOSTNAME) 12 | CONF__server__general__serverName=ENV($HOSTNAME) 13 | CONF__etc/apps/splunk_management_console__app__install__state=disabled 14 | CONF__outputs__indexAndForward__index=false 15 | CONF__outputs__indexer_discovery:search_head_peers__master_uri=https://cluster-master:8089 16 | CONF__outputs__indexer_discovery:search_head_peers__pass4SymmKey=indexdiscovery-changeme 17 | CONF__outputs__tcpout:splunk_search_head_peers_group__autoLB=true 18 | CONF__outputs__tcpout:splunk_search_head_peers_group__indexerDiscovery=search_head_peers 19 | CONF__outputs__tcpout__defaultGroup=splunk_search_head_peers_group 20 | CONF__outputs__tcpout__forwardedindex.filter.disable =true 21 | CONF__outputs__tcpout__indexAndForward=false 22 | CONF__server__clustering__cluster_label=cluster 23 | CONF__server__clustering__master_uri=https://cluster-master:8089 24 | CONF__server__clustering__mode=searchhead 25 | CONF__server__clustering__pass4SymmKey=clustering-changeme 26 | CONF__server__replication_port://9889__disabled=false 27 | CONF__server__shclustering__id=776D2949-B2DA-405E-96B3-B6688C87AB7D 28 | CONF__server__shclustering__conf_deploy_fetch_url=https://cluster-master:8089 29 | CONF__server__shclustering__disabled=false 30 | CONF__server__shclustering__election_timeout_ms=10000 31 | CONF__server__shclustering__mgmt_uri=ENV(https://$HOSTNAME:8089) 32 | CONF__server__shclustering__pass4SymmKey=shclustering-changeme 33 | CONF__server__shclustering__replication_factor=3 34 | CONF__server__shclustering__shcluster_label=shcluster 35 | CONF__web__settings__login_content=SHC Member -------------------------------------------------------------------------------- /examples/kubernetes-minikube/Makefile: -------------------------------------------------------------------------------- 1 | setup: 2 | docker-machine create \ 3 | --driver=virtualbox \ 4 | --virtualbox-disk-size=200000 \ 5 | --virtualbox-memory=512 \ 6 | --virtualbox-cpu-count=1 \ 7 | registry 8 | eval $$(docker-machine env registry) && docker run \ 9 | --name registry \ 10 | --publish 80:5000/tcp \ 11 | -d registry:2 12 | minikube start --cpus=6 \ 13 | --disk-size=200g \ 14 | --memory=9216 \ 15 | --vm-driver="virtualbox" \ 16 | --insecure-registry=$$(docker-machine ip registry) 17 | kubectl config set-context minikube -------------------------------------------------------------------------------- /splunk-cluster/.dockerignore: -------------------------------------------------------------------------------- 1 | Dockerfile 2 | .gitignore 3 | .dockerignore -------------------------------------------------------------------------------- /splunk-cluster/.gitignore: -------------------------------------------------------------------------------- 1 | *.lic -------------------------------------------------------------------------------- /splunk-cluster/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM splunk:latest 2 | 3 | MAINTAINER Denis Gladkikh 4 | 5 | COPY splunk_setup.py /opt/splunk/bin/splunk_setup.py -------------------------------------------------------------------------------- /splunk-cluster/splunk_setup.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import json 4 | import time 5 | import socket 6 | import re 7 | import glob 8 | import subprocess 9 | 10 | import requests 11 | 12 | import splunk.clilib.cli_common 13 | import splunk.util 14 | 15 | 16 | var_expandvars_re = re.compile(r'\AENV\((.*)\)$') 17 | var_shell_re = re.compile(r'\ASHELL\((.*)\)$') 18 | 19 | 20 | def main(): 21 | """ 22 | Initialize node. Can run before splunk started and after splunk started 23 | """ 24 | if sys.argv[1] == "--configure": 25 | configure() 26 | elif sys.argv[1] == "--wait-splunk": 27 | wait_splunk(sys.argv[2], sys.argv[3:]) 28 | elif sys.argv[1] == "--add-licenses": 29 | add_licenses(sys.argv[2]) 30 | elif sys.argv[1] == "--shc-autobootstrap": 31 | shc_autobootstrap(int(sys.argv[2]), sys.argv[3], sys.argv[4], sys.argv[5], sys.argv[6], sys.argv[7], sys.argv[8]) 32 | else: 33 | exit(1) 34 | 35 | 36 | def configure(): 37 | """ 38 | using CONF__ notation you can define any configuration, examples 39 | CONF__[{location_under_splunk_home}__]{conf_file}__{stanza}__{key}=value 40 | If location_under_splunk_home is not specified - system is used. 41 | """ 42 | # Allow to set any configurations with this 43 | conf_updates = {} 44 | for env, val in os.environ.iteritems(): 45 | if env.startswith("CONF__"): 46 | parts = env.split("__")[1:] 47 | conf_file_name = None 48 | parent = None 49 | conf_folder = "system" 50 | if len(parts) == 4: 51 | conf_folder = parts[0] 52 | parts = parts[1:] 53 | conf_folder_full = __get_conf_folder_full(conf_folder, parent) 54 | file_name = parts[0] 55 | if file_name == "meta": 56 | file_name = "local.meta" 57 | subfolder = "metadata" 58 | else: 59 | file_name = file_name + ".conf" 60 | subfolder = "local" 61 | conf_file = os.path.join(conf_folder_full, subfolder, file_name) 62 | conf_updates.setdefault(conf_file, {}).setdefault(parts[1], {})[parts[2]] = __get_value(val) 63 | 64 | for conf_file, conf_update in conf_updates.iteritems(): 65 | conf = splunk.clilib.cli_common.readConfFile(conf_file) if os.path.exists(conf_file) else {} 66 | for stanza, values in conf_update.iteritems(): 67 | dest_stanza = conf.setdefault(stanza, {}) 68 | dest_stanza.update(values) 69 | if "default" in conf and not conf["default"]: 70 | del conf["default"] 71 | folder = os.path.dirname(conf_file) 72 | if not os.path.isdir(folder): 73 | os.makedirs(folder) 74 | splunk.clilib.cli_common.writeConfFile(conf_file, conf) 75 | 76 | 77 | def __get_value(val): 78 | var_expand_match = var_expandvars_re.match(val) 79 | if var_expand_match: 80 | return os.path.expandvars(var_expand_match.groups()[0]) 81 | var_shell_match = var_shell_re.match(val) 82 | if var_shell_match: 83 | return subprocess.check_output(var_expand_match.groups()[0], shell=True) 84 | return val 85 | 86 | 87 | def __get_conf_folder_full(conf_folder, parent): 88 | if conf_folder == "system": 89 | return os.path.join(os.environ["SPLUNK_HOME"], "etc", conf_folder) 90 | else: 91 | return os.path.join(os.environ["SPLUNK_HOME"], conf_folder) 92 | 93 | 94 | def wait_splunk(uri, roles): 95 | """ 96 | Wait 5 minutes for dependency 97 | """ 98 | for x in xrange(1, 300): 99 | try: 100 | # This url does not require authentication, ignore certificate 101 | response = requests.get(uri + "/services/server/info?output_mode=json", verify=False) 102 | if response.status_code == 200: 103 | server_roles = response.json()["entry"][0]["content"]["server_roles"] 104 | if not roles or all(any(re.match(role, server_role) for server_role in server_roles) for role in roles): 105 | return 106 | else: 107 | print "Waiting for " + ", ".join(roles) + " in " + uri + " got " + ", ".join(server_roles) + "." 108 | else: 109 | print "Waiting for "+ ", ".join(roles) + " in " + uri + "." 110 | except requests.exceptions.RequestException as exception: 111 | print "Waiting for " + ", ".join(roles) + " in " + uri + ". Exception: " + str(exception) 112 | time.sleep(1) 113 | print "Failed to connect to " + uri + " and check server roles " + ", ".join(roles) 114 | exit(1) 115 | 116 | 117 | def add_licenses(folder): 118 | while True: 119 | if os.path.isdir(folder): 120 | licenses = glob.glob(os.path.join(folder, "*.lic")) 121 | if licenses: 122 | # Adding all licenses one by one and break 123 | for license in licenses: 124 | args = [ 125 | "add", 126 | "licenses", 127 | "-auth", "admin:changeme", 128 | license 129 | ] 130 | __splunk_execute(args) 131 | break 132 | 133 | print "Waiting for license files under " + folder 134 | time.sleep(1) 135 | 136 | 137 | def shc_autobootstrap(autobootstrap, mgmt_uri, local_user, local_password, service_discovery_uri, service_discovery_user, service_discovery_password): 138 | """ 139 | Write current uri to the service discovery URL, if current member has index equal 140 | to INIT_SHCLUSTER_AUTOBOOTSTRAP - bootstrap SHC, if more - add itself to existing SHC 141 | """ 142 | __service_discovery_post(service_discovery_uri, service_discovery_user, service_discovery_password, data=json.dumps({"host": mgmt_uri}), headers={"Content-type": "application/json"}) 143 | all_members = __service_discovery_get(service_discovery_uri, service_discovery_user, service_discovery_password, params={"sort": "_key"}).json() 144 | for index, member in enumerate(all_members): 145 | if member["host"] == mgmt_uri: 146 | if (index + 1) == autobootstrap: 147 | __splunk_execute([ 148 | "bootstrap", 149 | "shcluster-captain", 150 | "-auth", "%s:%s" % (local_user, local_password), 151 | "-servers_list", ",".join(m["host"] for m in all_members[:autobootstrap]) 152 | ]) 153 | elif (index + 1) > autobootstrap: 154 | # We do not check if current list of members already bootstrapped, assuming that autobootstrap is always equal to 155 | # how many instances user creating at beginning 156 | __splunk_execute([ 157 | "add", 158 | "shcluster-member", 159 | "-auth", "%s:%s" % (local_user, local_password), 160 | "-current_member_uri", next(m["host"] for m in all_members[:autobootstrap]) 161 | ]) 162 | 163 | 164 | def __service_discovery_get(service_discovery_uri, service_discovery_user, service_discovery_password, **kwargs): 165 | for x in xrange(1, 300): 166 | 167 | try: 168 | response = requests.get(service_discovery_uri, 169 | verify=False, 170 | auth=(service_discovery_user, service_discovery_password), 171 | **kwargs) 172 | response.raise_for_status() 173 | return response 174 | except requests.exceptions.RequestException as ex: 175 | print "Failed to make GET request to service discovery url. " + str(ex) 176 | sys.stdout.flush() 177 | sys.stderr.flush() 178 | time.sleep(1) 179 | print "FAILED. Could not make GET request to service discovery url." 180 | exit(1) 181 | 182 | 183 | def __service_discovery_post(service_discovery_uri, service_discovery_user, service_discovery_password, **kwargs): 184 | for x in xrange(1, 300): 185 | try: 186 | response = requests.post(service_discovery_uri, 187 | verify=False, 188 | auth=(service_discovery_user, service_discovery_password), 189 | **kwargs) 190 | response.raise_for_status() 191 | return response 192 | except requests.exceptions.RequestException as ex: 193 | print "Failed to make POST request to service discovery url. " + str(ex) 194 | sys.stdout.flush() 195 | sys.stderr.flush() 196 | time.sleep(1) 197 | print "FAILED. Could not make POST request to service discovery url." 198 | exit(1) 199 | 200 | 201 | def __splunk_execute(args): 202 | """ 203 | Execute splunk with arguments 204 | """ 205 | sys.stdout.flush() 206 | sys.stderr.flush() 207 | splunk_args = [os.path.join(os.environ['SPLUNK_HOME'], "bin", "splunk")] 208 | splunk_args.extend(args) 209 | subprocess.check_call(splunk_args) 210 | sys.stdout.flush() 211 | sys.stderr.flush() 212 | 213 | 214 | if __name__ == "__main__": 215 | main() --------------------------------------------------------------------------------