├── .gitignore ├── README.md ├── _config.yml ├── aws-scripts └── test_uploads3.sh ├── install ├── splunk-core │ ├── boot_start-fix.sh │ ├── local.sh │ └── remote.sh ├── splunk-uf │ ├── dep-client-local.sh │ ├── local.sh │ └── remote.sh └── syslog-ng │ ├── rhel_local_install_syslog-ng.sh │ └── rhel_yum_install_syslog-ng.sh ├── rhlinux ├── firewalld │ ├── splunk-core-service.sh │ ├── syslog-ng-service.sh │ └── uba-service.sh ├── kernel │ ├── disable-thp.sh │ ├── increase-ulimit.sh │ ├── optimize_linux.sh │ └── validate-ulimit.sh └── tcp-stack │ └── optimal-teardown.sh ├── ssh_config └── create_authorized_keys.sh ├── ssl-config ├── create-serverpem.sh ├── dod-signed-cert-stripper.sh ├── letsencrypt.sh └── replace-splunk-certs.sh ├── survival-guide ├── firewalking_port_testing │ └── netcat_examples.md ├── hacking_tools │ ├── README.md │ └── decrypt_splunk │ │ ├── bin │ │ └── decrypt.py │ │ └── local │ │ └── app.conf ├── misc_tasks │ ├── loop_through_list_and_cmd.md │ ├── misc_tasks.md │ └── progress_bar.txt ├── open-ssl │ └── open-ssl_cheat_sheet.md ├── splunk_configuration │ ├── create_archive_paths.md │ ├── edit_multiple_files_in_local.md │ ├── install_db_connect.sh │ ├── itsi_installer.sh │ ├── multitenant_appbuilder.sh │ └── multitenant_tabuilder.sh ├── sql_queries-dbx │ └── example_sql_queries.md ├── ssl_troubleshooting │ └── ssl_troubleshooting.md ├── stream_config │ ├── load_pcaps_from_list.md │ └── stream_update.py └── windows_administration │ ├── create_server_list.md │ └── remote_start_stop_splunk.md ├── syslog_ng_configs ├── syslog-ng_ip.conf └── syslog-ng_port.conf └── upgrade ├── splunk-core ├── local.sh └── remote.sh └── splunk-uf ├── local.sh └── remote.sh /.gitignore: -------------------------------------------------------------------------------- 1 | # Borrowed from https://github.com/splunk/splunk-app-splunkgit 2 | 3 | # OSX noise 4 | .DS_Store 5 | 6 | # Compiled python files 7 | *.pyc 8 | 9 | # Eclipse project files 10 | .project 11 | .pydevproject 12 | 13 | # Local stuff 14 | local.meta 15 | local/* 16 | !local/inputs.conf.sample 17 | 18 | # Git repositories folder 19 | git-repositories/ 20 | 21 | # PyCharm/IntelliJ meta-directory 22 | .idea 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Splunk Deployment & Automation 2 | Deployment scripts, playbooks and examples for for configuring Splunk securely. 3 | 4 | ### Any content which ends in .sh is an executable script, read the comments on what it does if you can't tell by the name. Anything that ends in .txt or .md are examples of how to perform various tasks in Linux or windows. They are listed here as commands you might need to do during migration or when you are attempting to automate something you don't want to do manually hundreds of times. These have been mostly been tested on RHEL7 and Ubuntu 16.04 over the last 2 years. 5 | 6 | 7 | ### Splunk Core-UF 8 | These script are used to install or upgrade splunk for linux. Local scripts should be used on the host you are trying to install on, remote scripts expect a list of ips or resolvable hostnames or dns names. 9 | ``` 10 | splunk_automation/install/splunk-core/local.sh 11 | splunk_automation/install/splunk-core/remote.sh 12 | splunk_automation/install/splunk-uf/local.sh 13 | splunk_automation/install/splunk-uf/remote.sh 14 | splunk_automation/upgrade/splunk-core/local.sh 15 | splunk_automation/upgrade/splunk-core/remote.sh 16 | splunk_automation/upgrade/splunk-uf/local.sh 17 | splunk_automation/upgrade/splunk-uf/remote.sh 18 | ``` 19 | 20 | ### OS Firewall Tuning 21 | These scripts are for rhel 7. They open the correct ports needed for splunk core, uba & syslog-ng. The firewalld services are XML based, so you can tweak the scripts as needed. 22 | ``` 23 | splunk_automation/rhlinux/firewalld/splunk-core-service.sh 24 | splunk_automation/rhlinux/firewalld/syslog-ng-service.sh 25 | splunk_automation/rhlinux/firewalld/uba-service.sh 26 | ``` 27 | 28 | ### OS Kernel Tuning 29 | These scripts are used to disable-thp on linux and reconfigure the ulimits. Validate ulimits checks what ulimits the splunkd pid currently has. You may need to restart splunkd for these settings to take effect. 30 | ``` 31 | splunk_automation/linux/kernel/disable-thp.sh 32 | splunk_automation/linux/kernel/increase-ulimit.sh 33 | splunk_automation/linux/kernel/validate-ulimit.sh 34 | ``` 35 | 36 | ## install_syslog-ng 37 | Test scripts for installing syslog-ng on RHEL. The yum install works only if the EPEL is configured upstream. 38 | ``` 39 | ./install_syslog-ng/rhel_local_install_syslog-ng.sh 40 | ./install_syslog-ng/rhel_yum_install_syslog-ng.sh 41 | ``` 42 | 43 | 44 | ## misc_tasks 45 | ``` 46 | ./misc_tasks/loop_through_list_and_cmd.txt 47 | ./misc_tasks/misc_tasks.txt 48 | ./misc_tasks/progress_bar.txt 49 | ``` 50 | 51 | ## splunk_configuration 52 | ``` 53 | ./splunk_configuration/create_archive_paths.txt 54 | ./splunk_configuration/edit_multiple_files_in_local.txt 55 | ./splunk_configuration/install_db_connect.sh 56 | ./splunk_configuration/itsi_installer.sh 57 | ./splunk_configuration/multitenant_appbuilder.sh 58 | ./splunk_configuration/multitenant_tabuilder.sh 59 | ``` 60 | 61 | ## ssh_config 62 | ``` 63 | ./ssh_config/create_authorized_keys.sh 64 | ``` 65 | 66 | ## stream_config 67 | ``` 68 | ./stream_config/load_pcaps_from_list.txt 69 | ./stream_config/stream_update.py 70 | ``` 71 | 72 | ## syslog_ng_configs 73 | ``` 74 | ./syslog_ng_configs/syslog-ng_ip.conf 75 | ./syslog_ng_configs/syslog-ng_port.conf 76 | ``` 77 | 78 | ## windows_administration 79 | ``` 80 | ./windows_administration/create_server_list.txt 81 | ./windows_administration/remote_start_stop_splunk.txt 82 | ``` 83 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-leap-day -------------------------------------------------------------------------------- /aws-scripts/test_uploads3.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: Upload a file to AmazonS3 in order to test connectivity to an environment during a cloud upgrade 5 | # 6 | # 7 | # 8 | # Parameters: ${1} = path to file you wish to upload 9 | # ${2} = s3Key 10 | # ${s3Secret} = s3Secret supplied via interactive session 11 | # ${bucket} = replace this in the script with your own AWS bucket 12 | # 13 | # Example usage: $ bash test_uploads3.sh some-file.tgz SomeKey 14 | # 15 | # Privileges: Curl 16 | # Authors: Amanda Chen, Anthony Tellez 17 | # 18 | # Notes: Script found/developed my Amanda, parameterized by Tellez. 19 | # 20 | file=${1} 21 | bucket=test-bucket-splunk 22 | resource="/${bucket}/${file}" 23 | contentType="application/x-compressed-tar" 24 | dateValue=`date -R` 25 | stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}" 26 | s3Key=${2} 27 | d -s -p "set s3Secret: " s3Secret 28 | printf "%b" "\n" 29 | signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64` 30 | curl -L -X PUT -T "${file}" \ 31 | -H "Host: ${bucket}.s3.amazonaws.com" \ 32 | -H "Date: ${dateValue}" \ 33 | -H "Content-Type: ${contentType}" \ 34 | -H "Authorization: AWS ${s3Key}:${signature}" \ 35 | https://${bucket}.s3.amazonaws.com/${file} 36 | -------------------------------------------------------------------------------- /install/splunk-core/boot_start-fix.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Update ulimit setting correcly during reboot of Splunk in /etc/init.d/splunk 3 | splunk_start() { 4 | echo Starting Splunk... 5 | ulimit -Hn 20240 6 | ulimit -Sn 10240 7 | "/opt/splunk/bin/splunk" start --no-prompt --answer-yes 8 | RETVAL=$? 9 | } 10 | -------------------------------------------------------------------------------- /install/splunk-core/local.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will install splunk and complete some initial setup steps 5 | # Parameters: ${1} = path to splunk install .tgz file 6 | # 7 | # Example usage: $ bash splunkinstall.sh splunk-6.3.2-aaff59bb082c-Linux-x86_64.tgz 8 | # 9 | # Privileges: Must be run as root 10 | # Authors: Chris Tribie, Anthony Tellez 11 | # 12 | # Notes: This script can use customized Splunk install tar or the default from Splunk.com. 13 | # Our custom install comprised the following changes from the base install: 14 | # in ~/splunk/etc/system/local/ 15 | # server.conf - configured master_uri for license server 16 | # authentication.conf - preloaded config for admin access from AD domain 17 | # deploymentclient.conf - preloaded deployment server info 18 | # in ~/splunk/etc/auth/ 19 | # distServerKeys/dmc-hostname/trusted.pem - added the public key for our DMC for search peer configuration 20 | # distServerKeys/ess-hostname/trusted.pem - added the public key for our Enerprise Security Search Head for search peer configuration 21 | # in ~/splunk/etc/ 22 | # splunk-launch.conf - SPLUNK_FIPS=1 - this must be done on first boot to ensure splunk enables the FIPS module 23 | # after untar, splunk is started, the admin password is changed, and 24 | # splunk is set to run at boot time. Since everything up to this point was 25 | # done as the root user, we need to change ownership to the splunk user. 26 | # This is done via the chown command. Last step is to start splunk again. 27 | # 28 | # Revision: Last change: 03/08/2016 by AT :: Increased Security of password entry mechanism 29 | # =========================================================== 30 | # 31 | useradd -d /opt/splunk splunk 32 | tar -zxf "${1}" -C /opt/ 33 | touch /opt/splunk/etc/.ui_login 34 | /opt/splunk/bin/splunk start --accept-license 35 | read -s -p "set password for admin user: " password 36 | printf "%b" "\n" 37 | /opt/splunk/bin/splunk edit user admin -password "${password}" -auth admin:changeme 38 | /opt/splunk/bin/splunk stop 39 | /opt/splunk/bin/splunk enable boot-start -user splunk 40 | chown -R splunk:splunk /opt/splunk 41 | service splunk start 42 | -------------------------------------------------------------------------------- /install/splunk-core/remote.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will remotely install the splunk uf 5 | # Parameters: ${1} = path to splunk install .tgz file 6 | # ${2} = list of hosts to install the uf 7 | # Example usage: $ bash splunkinstall.sh splunk-6.3.2-aaff59bb082c-Linux-x86_64.tgz 8 | # 9 | # Privileges: Must be run as root 10 | # Authors: Anthony Tellez 11 | # 12 | # Notes: This script can use customized Splunk install tar or the default from Splunk.com. 13 | # Our custom install comprised the following changes from the base install: 14 | # in ~/splunk/etc/system/local/ 15 | # deploymentclient.conf - preloaded deployment server info 16 | # Alternatively, ~/splunk/etc/apps/ 17 | # org_all_deploymentclient/local/ 18 | # deploymentclient.conf - preloaded deployment server info 19 | # in ~/splunk/etc/ 20 | # splunk-launch.conf - SPLUNK_FIPS=1 - this must be done on first boot to ensure splunk enables the FIPS module 21 | # after untar, splunk is started, the admin password is changed, and 22 | # splunk is set to run at boot time. Since everything up to this point was 23 | # done as the root user, we need to change ownership to the splunk user. 24 | # This is done via the chown command. Last step is to start splunk again. 25 | # 26 | # Revision: Last change: 03/08/2016 by AT :: Increased Security of password entry mechanism 27 | # =========================================================== 28 | # 29 | createSplunkUser="useradd -d /opt/splunk splunk" 30 | untarSplunk="tar -zxvf /tmp/${1} -C /opt && chown -R splunk:splunk /opt/splunk" 31 | startSplunk="sudo su - splunk -c 'touch /opt/splunk/etc/.ui_login && /opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt'" 32 | bootStart="/opt/splunk/bin/splunk enable boot-start -user splunk" 33 | for HOST in $(< $2); do 34 | scp -r "${1}" $HOST:/tmp 35 | ssh $HOST "${createSplunkUser} && ${untarSplunk}" 36 | ssh $HOST "${startSplunk} && ${bootStart}" 37 | if [ $? -ne 0 ]; then 38 | echo "---- COULD NOT CONNECT TO $HOST ----" 39 | fi 40 | done 41 | -------------------------------------------------------------------------------- /install/splunk-uf/dep-client-local.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will install splunk and complete some initial setup steps 5 | # such as configuring the deployment client & client name based on a list of 6 | # possible values. 7 | # 8 | # Parameters: ${1} = path to splunk install .tgz file 9 | # Clientname (name to set in deploymentclient.conf) 10 | # 11 | # Example usage: $ bash advancedufinstall.sh splunk-6.3.2-aaff59bb082c-Linux-x86_64.tgz 12 | # 13 | # Privileges: Must be run as root 14 | # Authors: Anthony Tellez 15 | # 16 | # Notes: This script can use customized Splunk install tar or the default from Splunk.com. 17 | # Our custom install comprised the following changes from the base install: 18 | # in ~/splunkforwarder/etc/system/local/ 19 | # deploymentclient.conf - preloaded deployment server info 20 | # Alternatively, ~/splunkforwarder/etc/apps/ 21 | # org_all_deploymentclient/local/ 22 | # deploymentclient.conf - preloaded deployment server info 23 | # in ~/splunkforwarder/etc/ 24 | # splunk-launch.conf - SPLUNK_FIPS=1 - this must be done on first boot to ensure splunk enables the FIPS module 25 | # after untar, splunk is started, the admin password is changed, and 26 | # splunk is set to run at boot time. Since everything up to this point was 27 | # done as the root user, we need to change ownership to the splunk user. 28 | # This is done via the chown command. Last step is to start splunk again. 29 | # 30 | # Revision: Last change: 03/08/2016 by AT :: Increased Security of password entry mechanism 31 | # =========================================================== 32 | # 33 | useradd -d /opt/splunkforwarder splunk 34 | tar -zxf "${1}" -C /opt/ 35 | /opt/splunkforwarder/bin/splunk start --accept-license 36 | read -p "set client name for the deployment client " clientname 37 | printf "%b" "\n" 38 | mkdir -p /opt/splunkforwarder/etc/apps/${clientname}_deploymentclient/local 39 | cat >/opt/splunkforwarder/etc/apps/${clientname}_deploymentclient/local/deploymentclient.conf< You should only run this script once. Running it again will append 13 | # to the same file: /etc/firewalld/services/syslog.xml and will break the service! 14 | # 15 | # Revision: Last change: 03/01/2016 by AT :: Added details about script 16 | # =========================================================== 17 | touch /etc/firewalld/services/splunkd.xml 18 | cat >/etc/firewalld/services/splunkd.xml < 20 | 21 | splunkd 22 | Splunkd service for rest and communication. 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | EOF 31 | restorecon /etc/firewalld/services/splunkd.xml 32 | chmod 640 /etc/firewalld/services/splunkd.xml 33 | firewall-cmd --reload 34 | echo "set selinux permissions" 35 | firewall-cmd --permanent --add-service=splunkd 36 | firewall-cmd --reload 37 | firewall-cmd --list-service 38 | echo "done." 39 | -------------------------------------------------------------------------------- /rhlinux/firewalld/syslog-ng-service.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will create a custom syslog service for firewalld 5 | # Parameters: None 6 | # Example usage: $ bash configure_firewalld_syslog.sh 7 | # 8 | # Privileges: Must be run as root 9 | # Author: Anthony Tellez 10 | # 11 | # Notes: You can change the ports/protocol by modifying the XML syntax in the echo for example: 12 | # You should only run this script once. Running it again will append 13 | # to the same file: /etc/firewalld/services/syslog.xml and will break the service! 14 | # 15 | # Revision: Last change: 03/01/2016 by AT :: Added details about script 16 | # =========================================================== 17 | touch /etc/firewalld/services/syslog.xml 18 | cat >/etc/firewalld/services/syslog.xml < 20 | 21 | syslog 22 | Service for syslog communication. 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | EOF 78 | echo "created service" 79 | restorecon /etc/firewalld/services/syslog.xml 80 | chmod 640 /etc/firewalld/services/syslog.xml 81 | firewall-cmd --reload 82 | echo "set selinux permissions" 83 | firewall-cmd --permanent --add-service=syslog 84 | firewall-cmd --reload 85 | firewall-cmd --list-service 86 | echo "done." 87 | -------------------------------------------------------------------------------- /rhlinux/firewalld/uba-service.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will create a custom uba service for firewalld 5 | # Parameters: None 6 | # Example usage: $ bash configure_firewalld_uba.sh 7 | # 8 | # Privileges: Must be run as root 9 | # Author: Anthony Tellez 10 | # 11 | # Notes: You can change the ports/protocol by modifying the XML syntax in the echo for example: 12 | # You should only run this script once. Running it again will append 13 | # to the same file: /etc/firewalld/services/uba.xml and will break the service! 14 | # 15 | # Revision: Last change: 03/01/2016 by AT :: Added details about script 16 | # =========================================================== 17 | touch /etc/firewalld/services/uba.xml 18 | cat >/etc/firewalld/services/uba.xml < 19 | 20 | uba 21 | Service for Splunk UBA communication. 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | EOF 49 | echo "created service" 50 | restorecon /etc/firewalld/services/uba.xml 51 | chmod 640 /etc/firewalld/services/uba.xml 52 | firewall-cmd --reload 53 | echo "set selinux permissions" 54 | firewall-cmd --permanent --add-service=uba 55 | firewall-cmd --reload 56 | firewall-cmd --list-service 57 | echo "done." 58 | -------------------------------------------------------------------------------- /rhlinux/kernel/disable-thp.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: 5 | # Parameters: ${1} = 6 | # ${2} = 7 | # Example usage: $ bash splunkinstall.sh splunk-6.3.2-aaff59bb082c-Linux-x86_64.tgz 8 | # 9 | # Privileges: Must be run as root 10 | # Authors: Anthony Tellez 11 | # 12 | # Notes: 13 | # 14 | # 15 | # Revision: Last change: XX/XX/2017 by AT :: 16 | # =========================================================== 17 | # 18 | mkdir /etc/tuned/custom 19 | touch /etc/tuned/custom/tuned.conf 20 | cat >/etc/tuned/custom/tuned.conf </etc/tuned/custom/script.sh < /sys/kernel/mm/transparent_hugepage/defrag 40 | return 0 41 | } 42 | 43 | stop() { 44 | return 0 45 | } 46 | 47 | process $@ 48 | EOF 49 | tuned-adm profile custom 50 | tuned-adm list 51 | -------------------------------------------------------------------------------- /rhlinux/kernel/increase-ulimit.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: 5 | # Parameters: ${1} = 6 | # ${2} = 7 | # Example usage: $ bash splunkinstall.sh splunk-6.3.2-aaff59bb082c-Linux-x86_64.tgz 8 | # 9 | # Privileges: Must be run as root 10 | # Authors: Anthony Tellez 11 | # 12 | # Notes: 13 | # 14 | # 15 | # Revision: Last change: XX/XX/2017 by AT :: 16 | # =========================================================== 17 | # 18 | cat >/etc/security/limits.d/20-nproc.conf </etc/security/limits.d/90-splunk.conf </etc/sysctl.conf </etc/security/limits.conf <> /etc/security/limits.conf 60 | echo "splunk soft nproc 8192" >> /etc/security/limits.conf 61 | echo "splunk soft nofile 8192" >> /etc/security/limits.conf 62 | echo "splunk hard nofile 8192" >> /etc/security/limits.conf 63 | fi 64 | fi 65 | 66 | #if ubuntu make the following change 67 | if [ "$os_name" == "Ubuntu" ]; then 68 | #backup /etc/pam.d/common-session 69 | cp /etc/pam.d/common-session /etc/pam.d/common-session.orig 70 | #check to see if the pam_limits.so entry exists 71 | test=`grep pam_limits /etc/pam.d/common-session` 72 | if [[ $test =~ "pam_limits" ]]; then 73 | echo "pam settings good,do nothing" 74 | else 75 | echo "session required pam_limits.so" >> /etc/pam.d/common-session 76 | fi 77 | fi 78 | 79 | #disable transparent huge pages 80 | #modify /etc/rc.local to make changes permanent across re-boot 81 | test=`grep hugepage /etc/rc.local` 82 | if [[ $test =~ "hugepage" ]]; then 83 | echo "skip" 84 | else 85 | if [[ ("$os_name" = "CentOS") && ("$os_version" = "6") ]]; then 86 | echo "CentOS Version:$os_version" 87 | disable1=`echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled` 88 | disable2=`echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag` 89 | echo "fix /etc/rc.local" 90 | #backup /etc/rc.local 91 | cp /etc/rc.local /etc/rc.local.orig 92 | #modify /etc/rc.local to turn off THP on reboot 93 | echo "if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then" >> /etc/rc.local 94 | echo " echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled" >> /etc/rc.local 95 | echo "fi" >> /etc/rc.local 96 | echo "if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then" >> /etc/rc.local 97 | echo " echo > never /sys/kernel/mm/redhat_transparent_hugepage/defrag" >> /etc/rc.local 98 | echo "fi" >> /etc/rc.local 99 | elif [[ ("$os_name" = "CentOS") && ("$os_version" = "7") ]]; then 100 | echo "CentOS Version:$os_version" 101 | disable1=`echo never > /sys/kernel/mm/transparent_hugepage/enabled` 102 | disable2=`echo never > /sys/kernel/mm/transparent_hugepage/defrag` 103 | echo "fix /etc/rc.local" 104 | #backup /etc/rc.local 105 | cp /etc/rc.local /etc/rc.local.orig 106 | #modify /etc/rc.local to turn off THP on reboot 107 | echo "if test -f /sys/kernel/mm/transparent_hugepage/enabled; then" >> /etc/rc.local 108 | echo " echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local 109 | echo "fi" >> /etc/rc.local 110 | echo "if test -f /sys/kernel/mm/transparent_hugepage/defrag; then" >> /etc/rc.local 111 | echo " echo > never /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local 112 | echo "fi" >> /etc/rc.local 113 | elif [ "$os_name" = "Ubuntu" ]; then 114 | disable1=`echo never > /sys/kernel/mm/transparent_hugepage/enabled` 115 | disable2=`echo never > /sys/kernel/mm/transparent_hugepage/defrag` 116 | echo "fix /etc/rc.local" 117 | #backup /etc/rc.local 118 | cp /etc/rc.local /etc/rc.local.orig 119 | #modify /etc/rc.local to turn off THP on reboot 120 | echo "if test -f /sys/kernel/mm/transparent_hugepage/enabled; then" >> /etc/rc.local 121 | echo " echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local 122 | echo "fi" >> /etc/rc.local 123 | echo "if test -f /sys/kernel/mm/transparent_hugepage/defrag; then" >> /etc/rc.local 124 | echo " echo > never /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local 125 | echo "fi" >> /etc/rc.local 126 | fi 127 | fi 128 | fi 129 | 130 | fi 131 | -------------------------------------------------------------------------------- /rhlinux/kernel/validate-ulimit.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: 5 | # Parameters: ${1} = 6 | # ${2} = 7 | # Example usage: $ bash splunkinstall.sh splunk-6.3.2-aaff59bb082c-Linux-x86_64.tgz 8 | # 9 | # Privileges: Must be run as root 10 | # Authors: Anthony Tellez 11 | # 12 | # Notes: 13 | # 14 | # 15 | # Revision: Last change: XX/XX/2017 by AT :: 16 | # =========================================================== 17 | # 18 | getprocspl="$(ps aux | grep '[s]plunkd -p 8089' | awk 'NR==1{print $2}')" 19 | ulimitcmd="cat /proc/${getprocspl}/limits" 20 | $ulimitcmd 21 | echo "done." 22 | -------------------------------------------------------------------------------- /rhlinux/tcp-stack/optimal-teardown.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: Optimial teardown for TCP connections. Default is 2 hours, causing ulimit issues in environments 5 | # with lots of forwarders. This reduces it down to 10 minutes. Tweak the integer as need. Value is in seconds 6 | # Parameters: None 7 | # Example usage: $ bash optimal-teardown.sh 8 | # 9 | # Privileges: Must be run as root 10 | # Authors: Anthony Tellez 11 | # 12 | # Notes: 13 | # 14 | # 15 | # Revision: Last change: XX/XX/2017 by AT :: 16 | # =========================================================== 17 | # 18 | echo 600 > /proc/sys/net/ipv4/tcp_keepalive_time 19 | echo '# Persist Updated Keep Alive Setting Afer A Reboot' >> /etc/sysctl.conf 20 | echo 'net.ipv4.tcp_keepalive_time = 600' >> /etc/sysctl.conf 21 | -------------------------------------------------------------------------------- /ssh_config/create_authorized_keys.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will create the .ssh/authorized_keys objects required for passwordless ssh and generate a new 5 | # key for the host machine. 6 | # Example usage: $ bash create_authorized_keys.sh 7 | # 8 | # Privileges: Must be run as root 9 | # Author: Anthony Tellez 10 | # 11 | # Notes: This script only requires root access/user acces 12 | # 13 | # 14 | # Revision: Last change: 03/01/2016 by AT :: Updated for local yum install & added details about script 15 | # =========================================================== 16 | # 17 | mkdir ~/.ssh 18 | chmod 700 ~/.ssh 19 | ssh-keygen -t rsa 20 | read -p "paste your public key to add to the host:" answer 21 | echo "$answer" >> ~/.ssh/authorized_keys 22 | chmod 400 ~/.ssh/authorized_keys -------------------------------------------------------------------------------- /ssl-config/create-serverpem.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | cp /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem.splunk 3 | openssl pkcs8 -topk8 -inform PEM -outform PEM -in /opt/splunk/etc/auth/splunkweb/privkey.pem -out /opt/splunk/etc/auth/encrypted.key -passout pass:"password" 4 | cat /opt/splunk/etc/auth/splunkweb/cert.pem > /opt/splunk/etc/auth/server.pem 5 | cat /opt/splunk/etc/auth/encrypted.key >> /opt/splunk/etc/auth/server.pem 6 | cat /opt/splunk/etc/auth/ca.pem >> /opt/splunk/etc/auth/server.pem 7 | chown -R splunk:splunk /opt/splunk -------------------------------------------------------------------------------- /ssl-config/dod-signed-cert-stripper.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will rip the server certificate and cacert out of the .txt file generated by the DOD webpage 5 | # It will convert the cacaert from p7b to .pem 6 | # three files will be produced: hostname001.dod.mil.cer.txt-server.pem, hostname001.dod.mil.cer.txt-cacert.pem, hostname001.dod.mil.cer.txt-server.pem 7 | # hostname001.dod.mil.cer.txt_only-cert.cert <- which gets deleted 8 | # tar the two .pem files and private key file into one bundle with the hostname ex: tar cvf hostname001.dod.mil.keysandcerts.tar *.pem *key 9 | # Privileges: Must have openssl in path, ownership of certificate txt file 10 | # Author: Anthony Tellez 11 | # 12 | # Notes: Only tested on RHEL7, OSX grep does not have perl support (I believe) 13 | # 14 | # 15 | # Revision: Last change: 05/23/2017 by AT :: Built and tested 16 | # =========================================================== 17 | # 18 | name=${1} 19 | grep -Pzo '(?s)-{5}BEGIN (CERTIFICATE)-{5}.*?-{5}END \1-{5}' ${1} > ${name}_only-cert.cert 20 | cat ${name}_only-cert.cert |awk 'split_after==1{n++;split_after=0} /-----END CERTIFICATE-----/ {split_after=1} {print > "'${name}'-server" n ".cert"}' 21 | mv ${name}-server1.cert ${name}-cacert.p7b 22 | mv ${name}-server.cert ${name}-server.pem 23 | rm -fr ${name}_only-cert.cert 24 | openssl pkcs7 -in ${name}-cacert.p7b -print_certs -out ${name}-cacert.pem 25 | echo "############################## validating server certificate ###########################" 26 | openssl x509 -in ${name}-server.pem -text -noout 27 | echo "############################## validating ca certificate ###########################" 28 | openssl x509 -in ${name}-cacert.pem -text -noout 29 | -------------------------------------------------------------------------------- /ssl-config/letsencrypt.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will replace the default splunk certs with the certificates supplied by letsencypt. 5 | # Example usage: $ bash letsencypt.sh hostname.anthonytellez.com 6 | # 7 | # Privileges: Must have openssl in path, ownership of certificate txt files and rw on /opt/splunk/etc/auth 8 | # Author: Anthony Tellez 9 | # 10 | # Notes: Back up your /opt/splunk/etc/auth directory before running the script! 11 | # 12 | # 13 | # Revision: Last change: 12/12/2017 by AT :: Built and tested 14 | # =========================================================== 15 | # 16 | if [[ $# -eq 0 ]] ; then 17 | echo 'provide the fqdn of the server as argument 1, eg: splunkserver.anthonytellez.com' 18 | exit 0 19 | fi 20 | fqdn="${1}" 21 | cp /opt/splunk/etc/auth/${fqdn}/cert.pem /opt/splunk/etc/auth/splunkweb/cert.pem 22 | cp /opt/splunk/etc/auth/${fqdn}/chain.pem /opt/splunk/etc/auth/ca.pem 23 | cp /opt/splunk/etc/auth/${fqdn}/privkey.pem /opt/splunk/etc/auth/splunkweb/privkey.pem 24 | # create server.pem 25 | openssl pkcs8 -topk8 -inform PEM -outform PEM -in /opt/splunk/etc/auth/${fqdn}/privkey.pem -out /opt/splunk/etc/auth/encrypted.key -passout pass:"password" 26 | cat /opt/splunk/etc/auth/${fqdn}/cert.pem > /opt/splunk/etc/auth/server.pem 27 | cat /opt/splunk/etc/auth/encrypted.key >> /opt/splunk/etc/auth/server.pem 28 | cat /opt/splunk/etc/auth/${fqdn}/chain.pem >> /opt/splunk/etc/auth/server.pem 29 | chown -R splunk:splunk /opt/splunk 30 | -------------------------------------------------------------------------------- /ssl-config/replace-splunk-certs.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will take the two certs from the other script and replace the defualt splunk certs 5 | # It also needs the private key which was used for generating the certificates to create the certificate chain correctly 6 | # Bundled all of these files into a .tar file, the script expects the files to have the same hostname ast the certificate files 7 | # Ex: hostname001.dod.mil.cer.txt-server.pem, hostname001.dod.mil.cer.txt-cacert.pem, hostname001.dod.mil.key 8 | # I suggest backing up /opt/splunk/etc/auth before running this script just incase your keys or certs are wrong. 9 | # Example usage: $ bash replace-splunk-certs.sh hostname001.dod.mil.keysandcerts.tar 10 | # 11 | # Privileges: Must have openssl in path, ownership of certificate txt file 12 | # Author: Anthony Tellez 13 | # 14 | # Notes: Only tested on RHEL7, OSX grep does not have perl support (I believe) 15 | # 16 | # 17 | # Revision: Last change: 05/23/2017 by AT :: Built and tested 18 | # =========================================================== 19 | # 20 | 21 | hostname=`hostname` 22 | mkdir /opt/splunk/certs 23 | tar xvf ${1} -C /opt/splunk/certs 24 | cp /opt/splunk/certs/${hostname}*-server.pem /opt/splunk/etc/auth/splunkweb/cert.pem 25 | cp /opt/splunk/certs/${hostname}*-cacert.pem /opt/splunk/etc/auth/ca.pem 26 | cp /opt/splunk/certs/${hostname}*.key /opt/splunk/etc/auth/splunkweb/privkey.pem 27 | chown -R splunk:splunk /opt/splunk 28 | ## Server Cert Structure: 29 | ### /opt/splunk/etc/auth/splunkweb/cert.pem 30 | ### encrypted /opt/splunk/etc/auth/splunkweb/privkey.pem 31 | ### /opt/splunk/ect/auth/cacert.pem 32 | cp /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem.splunk 33 | openssl pkcs8 -topk8 -inform PEM -outform PEM -in /opt/splunk/etc/auth/splunkweb/privkey.pem -out /opt/splunk/etc/auth/encrypted.key -passout pass:"password" 34 | cat /opt/splunk/etc/auth/splunkweb/cert.pem > /opt/splunk/etc/auth/server.pem 35 | cat /opt/splunk/etc/auth/encrypted.key >> /opt/splunk/etc/auth/server.pem 36 | cat /opt/splunk/etc/auth/ca.pem >> /opt/splunk/etc/auth/server.pem 37 | chown -R splunk:splunk /opt/splunk 38 | -------------------------------------------------------------------------------- /survival-guide/firewalking_port_testing/netcat_examples.md: -------------------------------------------------------------------------------- 1 | ### Using NetCat to test for connectivity 2 | Why NetCat? Um because "telnet is insecure" :unamused: according to security admins, so they don't like to see it installed on the box. 3 | 4 | NetCat also lets you create your own packets and specify protocol to get past firewalls. 5 | ### Installing NetCat 6 | 7 | ``` 8 | yum install nc -y 9 | apt install nc -y 10 | ``` 11 | ### Basic Syntax Sending 12 | $ nc 13 | 14 | $ nc 8.8.8.8 80 15 | 16 | #### Protocols supported 17 | - TCP *DEFAULT* 18 | - UDP -u 19 | - SSL --ssl 20 | 21 | ``` 22 | $ nc --ssl 8.8.8.8 443 23 | $ nc -u 8.8.8.8 514 24 | ``` 25 | 26 | ### Basic Syntax Listening 27 | ``` 28 | $ nc -l 0.0.0.0 29 | $ nc -ul 0.0.0.0 514 30 | ``` 31 | #### Options used: 32 | - l: Listen 33 | - u: udp 34 | - 514: port to listen on 35 | - 0.0.0.0: bind to all interface 36 | 37 | ### Testing/Faking out Syslog 38 | 39 | ``` 40 | $ echo '<14>*sourcehost* message text' | nc -v -u -w 1 *desthost* 514 41 | 42 | $ echo '<14>splunk-src this is a syslog message!' | nc -v -u -w 1 8.8.8.8 514 43 | ``` 44 | #### Options Used: 45 | - v: verbosity level 46 | - u: UDP 47 | - w: Connect Timeout 48 | -------------------------------------------------------------------------------- /survival-guide/hacking_tools/README.md: -------------------------------------------------------------------------------- 1 | # Hacking Tools for Splunk 2 | 3 | ### What 4 | These are known manipulations to Splunk, they require local access to the system in order to perform. They are not considered CVEs or anything of that nature since they require cli access to the host system. 5 | 6 | - Reset Admin Account 7 | - Decrypt Pass4Symmkey 8 | 9 | ### Why 10 | This project is intended to give you access to the Admin account or the Pass4Symmkey in the event these credentials are forgotten. 11 | 12 | ### How 13 | 14 | ### Resetting Admin Account: 15 | By Default, Splunk credentials are stored in $SPLUNK_HOME/etc/passwd: 16 | 17 | ``` 18 | $ cat passwd 19 | :admin:$6$qsBZ3jtfnKCkB3Fq$H14A20UT6617WzFBMPm4YSEnX6jnV7dfgqRf/FX6t1.aqVdDz8VrSdzdoOrHcJ/Ae1FO5XHfnzwpcKl2AZWH90::Administrator:admin:changeme@example.com:: 20 | ``` 21 | 22 | The password can only be viewed by using splunk.secret to decrypt. In order to get around needing to decrypt the password, you can simply rename the passwd file, which will reset the password for the Admin account to "changeme". 23 | 24 | Steps: 25 | * Shut down Splunk Instance 26 | * Delete or Rename the passwd file 27 | * Start Splunk back up 28 | * Access with default credentials admin:changeme 29 | 30 | #### Advanced 31 | - What about when you are on a production system and there are other users in the passwd file? 32 | 33 | ``` 34 | $ cat passwd 35 | :admin:$6$qsBZ3jtfnKCkB3Fq$H14A20UT6617WzFBMPm4YSEnX6jnV7dfgqRf/FX6t1.aqVdDz8VrSdzdoOrHcJ/Ae1FO5XHfnzwpcKl2AZWH90::Administrator:admin:changeme@example.com:: 36 | :atellez:$6$LwY3gVFSV.OMJHjP$SpYEiFGVlqMaPQePE/HqAlnSZtNW3WMdrZkIsbk6/LTeL6JhnEBHnZ0l07SQ/vlQn1QCdJGrq4w.cMqOUxtHh.::Anthony Tellez:admin:atellez@splunk.com:: 37 | ``` 38 | 39 | You can clone the passwd file, and delete file. You need to be careful about any users who are using the local login because their knowledge objects will be temporarily orphaned and they will be unable to login. After a restart the admin user should be back to the default password of changeme. Once you've completed your tasks you can put the original password file back into place. Alternatively, you can merge the user accounts into the bottom of the new passwd file. 40 | 41 | Steps: 42 | * Shut down Splunk Instance 43 | * clone passwd file (Crate a backup) 44 | * Delete passwd file 45 | * Start Splunk Instance 46 | * Access with default credentials admin:changeme 47 | * Make changes needed with the admin user 48 | * Optional Restore or Merge: shutdown splunk, restore original file, start splunk back up 49 | 50 | ### Decrypt credentials: 51 | - Sometimes it isn't possible to just redo all the Pass4Symmkey configurations across many hosts for clustering. This is where decryption can come in handy. 52 | - This technique takes information from a blog by hurricane labs: [https://www.hurricanelabs.com/blog/decrypt-passwords-encrypted-by-splunk/] and packages it as an app located in this repo for easier use without the need to redevelop each part. 53 | 54 | #### Requirements: 55 | * Splunk.Secret from the host you're trying to decrypt Pass4Symmkey 56 | * Clean environment to manipulate 57 | * Pass4Symmkey you need to decrypt 58 | 59 | #### Gotchas: 60 | * Fresh install uses splunk.secret to hash various configurations which will break until you reset them 61 | * Namely SSL configurations for REST & SplunkWeb 62 | 63 | Files to reset: 64 | ``` 65 | $SPLUNK_HOME/etc/system/local/server.conf 66 | $SPLUNK_HOME/etc/passwd 67 | 68 | [general] 69 | serverName = splunk-hacking 70 | pass4SymmKey = $1$cmeZzDfH0mh8 71 | 72 | [sslConfig] 73 | sslPassword = $1$JSvNkHKBmTp8 74 | ``` 75 | sslPassword needs to be reset to password for the REST SSL to work properly. 76 | Admin account needs to be reset (delete passwd file once new splunk.secret is in place) otherwise splunk won't let you login. 77 | 78 | Steps: 79 | * Copy decrypt_splunk to the apps directory 80 | * Update app.conf with the Pass4Symmkey you are trying to decrypt 81 | * Replace the splunk.secret with the matching one from your production host 82 | * Remove the passwd file, update ssl configurations 83 | * Restart splunk so the splunk.secret is used to rehash everything 84 | * Use the python script in the following syntax to decrypt replacing $SPLUNK_HOME with the appropriate directory: 85 | 86 | ``` 87 | $SPLUNK_HOME/bin/splunk cmd python $SPLUNK_HOME/etc/apps/decrypt_splunk/bin/decrypt.py 88 | ``` 89 | -------------------------------------------------------------------------------- /survival-guide/hacking_tools/decrypt_splunk/bin/decrypt.py: -------------------------------------------------------------------------------- 1 | import splunk.entity as entity 2 | import splunk.auth, splunk.search 3 | 4 | def getCredentials(sessionKey): 5 | myapp = 'decrypt_splunk' 6 | try: 7 | # list all credentials 8 | entities = entity.getEntities( 9 | ['admin', 'passwords'], namespace=myapp, 10 | owner='nobody', sessionKey=sessionKey) 11 | except Exception, e: 12 | raise Exception( 13 | "Could not get %s credentials from splunk." 14 | "Error: %s" % (myapp, str(e))) 15 | credentials = [] 16 | # return credentials 17 | for i, c in entities.items(): 18 | credentials.append((c['username'], c['clear_password'])) 19 | return credentials 20 | raise Exception("No credentials have been found") 21 | sessionKey = splunk.auth.getSessionKey('admin','changeme') 22 | credentials = getCredentials(sessionKey) 23 | for username, password in credentials: 24 | print username 25 | print password 26 | -------------------------------------------------------------------------------- /survival-guide/hacking_tools/decrypt_splunk/local/app.conf: -------------------------------------------------------------------------------- 1 | [credential::general] 2 | password = 3 | 4 | [credential::clustering] 5 | password = 6 | 7 | [credential::sslConfigsslPassword] 8 | password = -------------------------------------------------------------------------------- /survival-guide/misc_tasks/loop_through_list_and_cmd.md: -------------------------------------------------------------------------------- 1 | # Create the list of files to iterate through 2 | `ls > all_zipfiles_stdout.txt` 3 | 4 | # iterate through list and run a command 5 | `$ while read line; do unzip "${line}"; done < all_zipfiles_stdout.txt` 6 | -------------------------------------------------------------------------------- /survival-guide/misc_tasks/misc_tasks.md: -------------------------------------------------------------------------------- 1 | # convert all values in a list of a file to lowercase: 2 | tr '[:upper:]' '[:lower:]' < inputfile.txt >> outputfile.txt 3 | 4 | #convert all values in a list of a file to uppercase: 5 | tr '[:lower:]' '[:upper:]' < inputfile.txt >> outputfile.txt 6 | 7 | # create directory & subdirectories 8 | mkdir -p /root/child/grandchild/ -------------------------------------------------------------------------------- /survival-guide/misc_tasks/progress_bar.txt: -------------------------------------------------------------------------------- 1 | i=0 2 | while ((i < 100)); do 3 | printf "\r%3d%% complete" $i 4 | ((i += RANDOM%5+2)) 5 | # Of course, in real life, we'd be getting i from somewhere meaningful. 6 | sleep 1 7 | done 8 | echo 9 | -------------------------------------------------------------------------------- /survival-guide/open-ssl/open-ssl_cheat_sheet.md: -------------------------------------------------------------------------------- 1 | #Certificate Creation 2 | 3 | ## Generate a new private key and Certificate Signing Request 4 | $ openssl req -out CSR.csr -new -newkey rsa:2048 -nodes -keyout privateKey.key 5 | 6 | ## Generate a self-signed certificate 7 | $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout privateKey.key -out certificate.crt 8 | 9 | ## Generate a certificate signing request (CSR) for an existing private key 10 | $ openssl req -out CSR.csr -key privateKey.key -new 11 | 12 | ## Generate a certificate signing request based on an existing certificate 13 | $ openssl x509 -x509toreq -in certificate.crt -out CSR.csr -signkey privateKey.key 14 | 15 | ## Remove a passphrase from a private key 16 | $ openssl rsa -in privateKey.pem -out newPrivateKey.pem 17 | 18 | 19 | # Certificate Checking 20 | ## Check a signing request 21 | $ openssl req -text -noout -verify -in CSR.csr 22 | 23 | # Check a private key 24 | $ openssl rsa -in privateKey.key -check 25 | 26 | # Check a Certificate 27 | $ openssl x509 -in certificate.crt -text -noout 28 | 29 | # Check a PKS#12 file 30 | $ openssl pkcs12 -info -in keyStore.p12 31 | 32 | # SSL Debugging 33 | # Check an MD5 hash of the public key to ensure that it matches with what is in a CSR or private key 34 | $ openssl x509 -noout -modulus -in certificate.crt | openssl md5 35 | $ openssl rsa -noout -modulus -in privateKey.key | openssl md5 36 | $ openssl req -noout -modulus -in CSR.csr | openssl md5 37 | 38 | # Connect to a port/socket using Openssl 39 | $ openssl s_client -connect splunk.com:443 40 | 41 | # SSL Conversion 42 | ## Convert a DER file (.crt .cer .der) to PEM 43 | $ openssl x509 -inform der -in certificate.cer -out certificate.pem 44 | 45 | ## Convert a PEM file to DER 46 | $ openssl x509 -outform der -in certificate.pem -out certificate.der 47 | 48 | ##Convert a PKCS#12 file (.pfx .p12) containing a private key and certificates to PEM 49 | $ openssl pkcs12 -in keyStore.pfx -out keyStore.pem -nodes 50 | 51 | ## Convert a PEM certificate file and a private key to PKCS#12 (.pfx .p12) 52 | $ openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -certfile CACert.crt 53 | -------------------------------------------------------------------------------- /survival-guide/splunk_configuration/create_archive_paths.md: -------------------------------------------------------------------------------- 1 | # How to create a list of index names: 2 | $ cat org_all_indexes/local/indexes.conf | grep -i "\[" >> all_indexes_list.txt 3 | 4 | # edit the list in vi 5 | $ vi all_indexes_list.txt 6 | 7 | # in vi delete the brackets around the index names: 8 | :%s/\[// 9 | :%s/\]// 10 | 11 | # Assuming archive space is /archive: 12 | $ cd /archive 13 | 14 | # Create a bunch of directories for the indexes using the list: 15 | $ while read line; do mkdir "${line}"; done < all_indexes_list.txt 16 | 17 | # rename and remove 3 characters from the left part of a string 18 | $ while read line; do mv "${line}" "${line:3}"; done < broken 19 | 20 | # rename and remove 3 characters from the right part of a string 21 | $ while read line; do mv "${line}" "${line::-3}"; done < broken -------------------------------------------------------------------------------- /survival-guide/splunk_configuration/edit_multiple_files_in_local.md: -------------------------------------------------------------------------------- 1 | # Edit multiple files in vi: Assuming pwd is ~/etc/apps/ 2 | $ vi */local/config.conf 3 | 4 | -------------------------------------------------------------------------------- /survival-guide/splunk_configuration/install_db_connect.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will install oracle jdk for dbconnect and configure the application 5 | # Parameters: None 6 | # Example usage: $ bash install_db_connect.sh 7 | # 8 | # Privileges: Must be run as root 9 | # Author: Anthony Tellez 10 | # 11 | # Notes: This script requires access to the internet in order to grab the latest version of oracle jdk 12 | # 13 | # 14 | # Revision: Last change: 03/08/2016 by AT :: Updated wget configuration/variables 15 | # =========================================================== 16 | # 17 | dl_dbconnect="wget -O splunk-db-connect-2_213.tgz https://splunkbase.splunk.com/app/2686/release/2.1.3/download/?origin=ipb" 18 | dl_jdk='wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u73-b02/jdk-8u73-linux-x64.tar.gz' 19 | 20 | echo "downloading dbconnect v.213" 21 | $dl_dbconnect 22 | 23 | # echo "downloading oracle jdk for dbconnect" 24 | # $dl_jdk 25 | 26 | # tar -zxf splunk-db-connect-2_213.tgz -C /opt/splunk/etc/apps 27 | # tar -zxf jdk-8u73-linux-x64.tar.gz -C /opt 28 | # chown -R splunk:splunk /opt/splunk 29 | # echo "jvm parameters for dbx" 30 | # su -c splunk "/opt/splunk/bin/splunk restart" 31 | -------------------------------------------------------------------------------- /survival-guide/splunk_configuration/itsi_installer.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will install ITSI following the steps from docs.splunk.com 5 | # Parameters: ${1} = path to itsi .spl file 6 | # 7 | # Example usage: $ bash itsi_install.sh splunk-it-service-intelligence_220.spl 8 | # 9 | # Privileges: Run as root 10 | # Authors: Anthony Tellez 11 | # 12 | # Notes: This script will set ownership to a splunk user, it is expected that the splunk_install script was used 13 | # to create the user splunk. 14 | # 15 | # Revision: Last change: 04/27/2016 by AT :: Created script, added documentation 16 | # =========================================================== 17 | # 18 | 19 | service splunk stop 20 | echo "##################################################################" 21 | echo "### Splunk Stopped, Installing IT Service Intelligence ###" 22 | echo "##################################################################" 23 | 24 | tar -xvf ${1} -C /opt/splunk/etc/apps 25 | chown -R splunk:splunk /opt/splunk 26 | 27 | echo "##################################################################" 28 | echo "### IT Service Intelligence Installed, Starting Splunk ###" 29 | echo "##################################################################" 30 | 31 | service splunk start -------------------------------------------------------------------------------- /survival-guide/splunk_configuration/multitenant_appbuilder.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #title :multitenant_appbuilder.sh 3 | #description :This script will take a org name and Splunk TA as input and 4 | # generate a new TA with the orgname embedded in the configs 5 | #author :ctribie, atellez 6 | #date :04Apr2016 7 | #version :1.1 8 | #usage :bash buildapp.sh 9 | #notes : 10 | #============================================================================== 11 | 12 | #Variables 13 | newapps=() 14 | appl=(sourceApps/*) 15 | applist="${appl[@]/sourceApps\/template_/}" 16 | now=`date +"%F_%T"` 17 | 18 | #if [ -z "$1" ] 19 | #then 20 | #capture org name from user input 21 | echo "#################################" 22 | echo "### Org App Builder ###" 23 | echo "#################################" 24 | echo 25 | echo "Please enter the org name you are generating an app for" 26 | echo 27 | read -p 'Orgname: ' orgname 28 | echo 29 | 30 | #else 31 | # orgname=${1[@]} 32 | #fi 33 | 34 | #allow user to choose which app to apply script to 35 | select appname in exit ${applist[@]} 36 | do 37 | if [ $appname = "exit" ] 38 | then 39 | break 40 | fi 41 | 42 | # for j in "${orgname[@]}"; do 43 | #define the final app name and append it to an array 44 | orgapp="${orgname}_${appname}" 45 | newapps+=($orgapp) 46 | # done 47 | 48 | #make a copy of the template directory which includes the org name 49 | copy_command="cp -r sourceApps/template_${appname} ${orgapp}" 50 | $copy_command 51 | 52 | #replace the word ORGNAME from the template with the actual org name 53 | rename="sed -i "s/ORGNAME/$orgname/g" ${orgapp}/*/*.c*";$rename 54 | 55 | echo "App generation complete for ${orgapp}." 56 | done 57 | 58 | #Prompt user to scp files to appropriate server 59 | echo "Would you like to copy this addon to the appropriate Splunk server?" 60 | select yn in "Yes" "No"; do 61 | case $yn in 62 | Yes ) 63 | #make a backup copy of serverclass.conf 64 | # sync="rsync -a /opt/splunk/etc/system/local/serverclass.conf /home/splunk/serverclass-backup/serverclass.conf-$now";$sync 65 | 66 | #iterate through apps processed above 67 | for i in "${newapps[@]}"; do 68 | if [[ $i = *auth ]] 69 | then 70 | destfolder="/opt/splunk/etc/shcluster/apps" 71 | destserver="###deployer.splunk.tld###" 72 | elif [[ $i = *indexes ]] 73 | then 74 | mv_command="mv ${i} usc_${i}";$mv_command 75 | i="usc_${i}" 76 | destfolder="/opt/splunk/etc/master-apps" 77 | destserver="###master.splunk.tld###" 78 | else 79 | destfolder="/opt/splunk/etc/deployment-apps" 80 | destserver="###deployment.splunk.tld###" 81 | fi 82 | #copy files to destination server 83 | echo "Sending files to ${destserver}." 84 | scp="scp -r ${i} ${destserver}:${destfolder}/.";$scp 85 | done 86 | #delete temporary copy 87 | rmscp="rm -rf ${orgname}*";$rmscp 88 | break;; 89 | No ) exit;; 90 | esac 91 | done 92 | -------------------------------------------------------------------------------- /survival-guide/splunk_configuration/multitenant_tabuilder.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #title :tabuild.sh 3 | #description :This script will take a organization name as input and 4 | # generate new TA's with the orgname embedded in the configs 5 | #author :chris tribie, anthony tellez 6 | #date :04APR16 7 | #version :1.1 8 | #usage :bash tabuild.sh 9 | #notes :creates a clientName app which assigns the orgname to 10 | # that organization's systems 11 | # creates an app with outputs.conf for the organization 12 | # creates an app with authorize.conf for the organization and sends it 13 | # to our other deployment server 14 | # edits $SPLUNK_HOME/etc/system/local/serverclass.conf 15 | # adds a new serverclass and associated app mappings 16 | #============================================================================== 17 | 18 | #Variables 19 | now=`date +"%F_%T"` 20 | orgname=$1 21 | destserver="###dmt.splunk.tld###" 22 | destfolder="/opt/splunk/etc/deployment-apps" 23 | serverclass="/opt/splunk/etc/system/local/serverclass.conf" 24 | 25 | #define the final app name 26 | orgapp="domain_${orgname}_clientName" 27 | 28 | #make a copy of the template directory which includes the organization name 29 | copy_command="cp -r /home/splunk/onboarding/sourceApps/domain_template_clientName ${orgapp}";$copy_command 30 | 31 | #replace the word orgname from the template with the actual organization name 32 | rename="sed -i "s/orgname/$orgname/g" ${orgapp}/*/*.c*";$rename 33 | 34 | echo "App generation complete for ${orgapp}. Sending files to ${destfolder}." 35 | 36 | copy_command="cp -r ${orgapp} /opt/splunk/etc/deployment-apps/.";$copy_command 37 | #delete temporary copy 38 | rmcp="rm -rf ${orgapp}";$rmcp 39 | 40 | orgapp="domain_${orgname}_forwarder_outputs" 41 | copy_command="cp -r /home/splunk/onboarding/sourceApps/domain_template_forwarder_outputs ${orgapp}";$copy_command 42 | rename="sed -i "s/orgname/$orgname/g" ${orgapp}/*/*.c*";$rename 43 | 44 | echo "App generation complete for ${orgapp}. Sending files to ${destfolder}." 45 | 46 | copy_command="cp -r ${orgapp} /opt/splunk/etc/deployment-apps/.";$copy_command 47 | rmcp="rm -rf ${orgapp}";$rmcp 48 | 49 | orgapp="domain_auth_${orgname}" 50 | copy_command="cp -r /home/splunk/onboarding/sourceApps/domain_auth_template ${orgapp}";$copy_command 51 | rename="sed -i "s/orgname/$orgname/g" ${orgapp}/*/*.c*";$rename 52 | 53 | echo "App generation complete for ${orgapp}. Sending files to ${destserver}:${destfolder}." 54 | 55 | scp="scp -r ${orgapp} ${destserver}:${destfolder}/.";$scp 56 | rmcp="rm -rf ${orgapp}";$rmcp 57 | 58 | echo "[serverClass:domain_${orgname}_all_clients]" >> ${serverclass} 59 | echo "whitelist.0 = *.${orgname}.(subdomain.gov|subdomain.org)" >> ${serverclass} 60 | echo "" >> ${serverclass} 61 | echo "[serverClass:domain_${orgname}_all_clients:app:domain_${orgname}_clientName]" >> ${serverclass} 62 | echo "restartSplunkWeb = 0" >> ${serverclass} 63 | echo "restartSplunkd = 1" >> ${serverclass} 64 | echo "stateOnClient = enabled" >> ${serverclass} 65 | echo "" >> ${serverclass} 66 | echo "[serverClass:domain_${orgname}_all_clients:app:domain_${orgname}_forwarder_outputs]" >> ${serverclass} 67 | echo "restartSplunkWeb = 0" >> ${serverclass} 68 | echo "restartSplunkd = 1" >> ${serverclass} 69 | echo "stateOnClient = enabled" >> ${serverclass} 70 | echo "" >> ${serverclass} 71 | 72 | echo "" 73 | echo "Completed update of serverclass.conf for ${orgname}." 74 | echo "" 75 | tailcmd="tail -n 12 ${serverclass}";$tailcmd 76 | reloadcmd="/opt/splunk/bin/splunk reload deploy-server -auth admin:password";$reloadcmd 77 | -------------------------------------------------------------------------------- /survival-guide/sql_queries-dbx/example_sql_queries.md: -------------------------------------------------------------------------------- 1 | [epo_inputs] 2 | connection = teledv50 3 | host = teledv50 4 | index = epo 5 | interval = 3600 6 | max_rows = 100000 7 | mode = advanced 8 | query = SELECT [EPOEvents].[ReceivedUTC] AS [timestamp], 9 | [EPOEvents].[AutoID], 10 | [EPOEvents].[ThreatName] AS [signature], 11 | [EPOEvents].[ThreatType] AS [threat_type], 12 | [EPOEvents].[ThreatEventID] AS [signature_id], 13 | [EPOEvents].[ThreatCategory] AS [category], 14 | [EPOEvents].[ThreatSeverity] AS [severity_id], 15 | [EPOEventFilterDesc].[Name] AS [event_description], 16 | [EPOEvents].[DetectedUTC] AS [detected_timestamp], 17 | [EPOEvents].[TargetFileName] AS [file_name], 18 | [EPOEvents].[AnalyzerDetectionMethod] AS [detection_method], 19 | [EPOEvents].[ThreatActionTaken] AS [vendor_action], 20 | CAST([EPOEvents].[ThreatHandled] AS int) AS [threat_handled], 21 | [EPOEvents].[TargetUserName] AS [logon_user], 22 | [EPOComputerProperties].[UserName] AS [user], 23 | [EPOComputerProperties].[DomainName] AS [dest_nt_domain], 24 | [EPOEvents].[TargetHostName] AS [dest_dns], 25 | [EPOEvents].[TargetHostName] AS [dest_nt_host], 26 | [EPOComputerProperties].[IPHostName] AS [fqdn], 27 | [dest_ip] = (convert(varchar(3), 28 | convert(tinyint, 29 | substring(convert(varbinary(4), 30 | convert(bigint, 31 | ([EPOComputerProperties].[IPV4x] + 2147483648))), 32 | 1, 33 | 1)))+'.'+ convert(varchar(3),convert(tinyint,substring(convert(varbinary(4), convert(bigint,([EPOComputerProperties].[IPV4x] + 2147483648))),2,1)))+'.'+ convert(varchar(3),convert(tinyint,substring(convert(varbinary(4), convert(bigint,([EPOComputerProperties].[IPV4x] + 2147483648))),3,1)))+'.'+ convert(varchar(3),convert(tinyint,substring(convert(varbinary(4), convert(bigint,([EPOComputerProperties].[IPV4x] + 2147483648))),4,1))) ), [EPOComputerProperties].[SubnetMask] AS [dest_netmask], [EPOComputerProperties].[NetAddress] AS [dest_mac], [EPOComputerProperties].[OSType] AS [os], [EPOComputerProperties].[OSServicePackVer] AS [sp], [EPOComputerProperties].[OSVersion] AS [os_version], [EPOComputerProperties].[OSBuildNum] AS [os_build], [EPOComputerProperties].[TimeZone] AS [timezone], [EPOEvents].[SourceHostName] AS [src_dns], [src_ip] = ( convert(varchar(3),convert(tinyint,substring(convert(varbinary(4), convert(bigint,([EPOEvents].[SourceIPV4] + 2147483648))),1,1)))+'.'+ convert(varchar(3),convert(tinyint,substring(convert(varbinary(4), convert(bigint,([EPOEvents].[SourceIPV4] + 2147483648))),2,1)))+'.'+ convert(varchar(3),convert(tinyint,substring(convert(varbinary(4), convert(bigint,([EPOEvents].[SourceIPV4] + 2147483648))),3,1)))+'.'+ convert(varchar(3),convert(tinyint,substring(convert(varbinary(4), convert(bigint,([EPOEvents].[SourceIPV4] + 2147483648))),4,1))) ), [EPOEvents].[SourceMAC] AS [src_mac], [EPOEvents].[SourceProcessName] AS [process], [EPOEvents].[SourceURL] AS [url], [EPOEvents].[SourceUserName] AS [source_logon_user], [EPOComputerProperties].[IsPortable] AS [is_laptop], [EPOEvents].[AnalyzerName] AS [product], [EPOEvents].[AnalyzerVersion] AS [product_version], [EPOEvents].[AnalyzerEngineVersion] AS [engine_version], [EPOEvents].[AnalyzerEngineVersion] AS [dat_version], [EPOProdPropsView_VIRUSCAN].[datver] AS [vse_dat_version], [EPOProdPropsView_VIRUSCAN].[enginever64] AS [vse_engine64_version], [EPOProdPropsView_VIRUSCAN].[enginever] AS [vse_engine_version], [EPOProdPropsView_VIRUSCAN].[hotfix] AS [vse_hotfix], [EPOProdPropsView_VIRUSCAN].[productversion] AS [vse_product_version], [EPOProdPropsView_VIRUSCAN].[servicepack] AS [vse_sp] 34 | FROM [ePO_TELEAV438].[dbo].[EPOEvents] EPOEvents 35 | LEFT JOIN [ePO_TELEAV438].[dbo].[EPOLeafNode] EPOLeafNode 36 | ON [EPOEvents].[AgentGUID] = [EPOLeafNode].[AgentGUID] 37 | LEFT JOIN [ePO_TELEAV438].[dbo].[EPOProdPropsView_VIRUSCAN] EPOProdPropsView_VIRUSCAN 38 | ON [EPOLeafNode].[AutoID] = [EPOProdPropsView_VIRUSCAN].[LeafNodeID] 39 | LEFT JOIN [ePO_TELEAV438].[dbo].[EPOComputerProperties] EPOComputerProperties 40 | ON [EPOLeafNode].[AutoID] = [EPOComputerProperties].[ParentID] 41 | LEFT JOIN [ePO_TELEAV438].[dbo].[EPOEventFilterDesc] EPOEventFilterDesc 42 | ON [EPOEvents].[ThreatEventID] = [EPOEventFilterDesc].[EventId] 43 | AND (EPOEventFilterDesc.Language='0409') 44 | WHERE [EPOvents].[AutoID] > ? 45 | ORDER BY [EPOvents].[AutoID] 46 | source = dbx 47 | sourcetype = mcafee:epo 48 | ui_query_catalog = ePO_TELEAV438 49 | ui_query_schema = dbo 50 | ui_query_table = EPOEvents 51 | ui_query_mode = advanced 52 | tail_rising_column_name = AutoID 53 | -------------------------------------------------------------------------------- /survival-guide/ssl_troubleshooting/ssl_troubleshooting.md: -------------------------------------------------------------------------------- 1 | ## SSL Troubleshooting for Splunk 2 | Purpose of this is to provide common errors and how to get to root cause of ssl issues for Splunk to Splunk and intersplunk communication. When configuring Splunk SSL for web or various components talking to each other several configuration files need to modified and parameters need to line up for things to work properly. 3 | 4 | ## Sanity Check: 5 | Information about how certs are generated can be found in /opt/splunk/bin/genRootCA.sh: 6 | 7 | ``` 8 | echo "This script will create a root CA" 9 | echo "It will output two files. ca.pem cacert.pem" 10 | echo "Distribute the cacert.pem to all clients you wish to connect to you." 11 | echo "Keep ca.pem for safe keeping for signing other clients certs" 12 | echo "Remember your password for the ca.pem you will need to later to sign other client certs" 13 | echo "Your root CA will expire in 10 years" 14 | ``` 15 | ### Search Head 16 | 17 | ### Indexer(s) 18 | Inputs.conf 19 | 20 | ``` 21 | [splunktcp-ssl://9997] 22 | 23 | # SSL SETTINGS 24 | [SSL] 25 | serverCert = $SPLUNK_HOME/etc/auth/server.pem 26 | requireClientCert = false 27 | sslPassword = password 28 | ``` 29 | 30 | Server.conf 31 | 32 | ``` 33 | [sslConfig] 34 | sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem 35 | ``` 36 | 37 | ### Checking sslRootCAPath 38 | $ openssl x509 -in /opt/splunk/etc/auth/ca.pem -text -noout 39 | 40 | ``` 41 | Certificate: 42 | Data: 43 | Version: 1 (0x0) 44 | Serial Number: 16551569488170448198 (0xe5b2fcc16997f546) 45 | Signature Algorithm: sha1WithRSAEncryption 46 | Issuer: C=US, ST=CA, L=San Francisco, O=Splunk, CN=SplunkCommonCA/emailAddress=support@splunk.com 47 | Validity 48 | Not Before: May 11 19:51:37 2015 GMT 49 | Not After : May 8 19:51:37 2025 GMT 50 | Subject: C=US, ST=CA, L=San Francisco, O=Splunk, CN=SplunkCommonCA/emailAddress=support@splunk.com 51 | Subject Public Key Info: 52 | Public Key Algorithm: rsaEncryption 53 | Public-Key: (1024 bit) 54 | Modulus: 55 | 00:c9:99:be:79:ca:f6:a6:d4:6a:86:81:32:b4:75: 56 | f1:d7:58:98:81:d0:58:7c:7e:c7:49:15:17:39:77: 57 | 10:49:3c:56:82:fe:49:66:b5:b2:c5:2d:b6:2e:5d: 58 | d0:b6:26:1e:1c:9b:fb:a1:8f:5f:c5:5a:60:34:59: 59 | b8:5b:d3:6a:e8:01:5d:37:67:74:97:d2:91:f2:15: 60 | ad:d4:77:2a:ab:f5:fe:44:44:9d:00:60:50:3e:cb: 61 | 95:21:6c:c9:c3:f7:39:61:b3:b2:7c:b9:cb:9b:dd: 62 | 7b:c0:f2:b9:fb:f5:e8:e4:62:d0:d7:da:b3:10:58: 63 | f3:59:60:f7:2b:c5:41:21:8b 64 | Exponent: 65537 (0x10001) 65 | Signature Algorithm: sha1WithRSAEncryption 66 | 57:7d:77:3c:b2:6f:6c:27:94:3c:b7:b6:51:55:1f:60:54:5d: 67 | d2:59:3c:a2:02:13:75:72:32:c3:d3:36:15:c3:ab:b1:12:55: 68 | 60:4b:25:e5:10:87:ab:89:d4:0d:d0:c8:ba:ed:4e:a1:bf:d6: 69 | 1e:b6:be:f3:fe:53:10:30:e1:31:d9:e2:0d:da:da:2e:b9:dd: 70 | 3d:6a:ef:c7:61:ab:57:0a:9d:e3:ae:13:cd:d3:7b:f7:d1:10: 71 | 7e:78:42:89:33:ae:70:17:a3:3f:af:fd:a1:89:93:38:c4:a5: 72 | 21:30:ad:65:30:2c:0d:64:a0:4f:08:ff:45:c5:13:0c:56:6c: 73 | 46:ed 74 | ``` 75 | 76 | ### Forwarder(s) 77 | Outputs.conf 78 | 79 | ``` 80 | [tcpout:primary_indexers_ssl] 81 | server = jupiter.synapticecho.com:9997 82 | clientCert = $SPLUNK_HOME/etc/auth/server.pem 83 | sslPassword = password 84 | sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem 85 | autoLB = true 86 | # If value is set to true read instructions below: 87 | sslVerifyServerCert = false 88 | ``` 89 | 90 | #### Checking clientCert 91 | $ openssl x509 -in /opt/splunkforwarder/etc/auth/server.pem -text -noout 92 | 93 | ``` 94 | Certificate: 95 | Data: 96 | Version: 1 (0x0) 97 | Serial Number: 15831133880858721752 (0xdbb37c63403c3dd8) 98 | Signature Algorithm: sha1WithRSAEncryption 99 | Issuer: C=US, ST=CA, L=San Francisco, O=Splunk, CN=SplunkCommonCA/emailAddress=support@splunk.com 100 | Validity 101 | Not Before: Mar 3 20:26:48 2017 GMT 102 | Not After : Mar 2 20:26:48 2020 GMT 103 | Subject: CN=SplunkServerDefaultCert, O=SplunkUser 104 | Subject Public Key Info: 105 | Public Key Algorithm: rsaEncryption 106 | Public-Key: (1024 bit) 107 | Modulus: 108 | 00:a7:6b:a8:63:3c:c9:48:2e:9e:fc:4d:b4:26:96: 109 | 4e:83:37:24:9d:cf:9c:b7:a0:a2:d9:30:36:ec:43: 110 | 46:3f:68:b3:04:fb:e4:3f:a4:4a:c3:4b:b4:40:e9: 111 | 1c:be:a9:af:21:3a:5a:87:3f:45:4c:39:64:ef:fc: 112 | c3:64:65:1c:b6:58:c4:0c:9f:71:58:cd:bf:2a:ca: 113 | cf:d9:24:5c:99:ab:f3:2f:16:73:94:cb:62:c2:99: 114 | f2:1a:6c:89:8b:20:d4:7c:8a:86:c9:c4:38:2e:da: 115 | 52:c4:da:ec:db:c0:97:c5:05:31:22:d5:40:87:a9: 116 | 9a:83:a3:1a:93:3a:5c:38:b3 117 | Exponent: 65537 (0x10001) 118 | Signature Algorithm: sha1WithRSAEncryption 119 | 0d:3f:49:04:67:02:f2:68:cd:76:ff:5a:9a:6f:85:51:2f:32: 120 | 87:95:a4:cc:85:1d:4f:2c:f5:93:a3:30:b6:c5:6d:b1:f8:94: 121 | f1:31:39:0f:94:7e:3b:f4:1d:5d:24:f7:c9:ce:02:c6:7a:6e: 122 | 56:40:80:3d:c2:61:3a:08:05:f3:a9:0f:ba:80:cc:78:f5:fa: 123 | 06:4a:fb:9f:df:9b:95:50:a0:c3:b5:1c:cf:f5:a8:ed:ab:0c: 124 | 85:6b:e4:e5:a8:9e:72:5b:67:b7:6d:2a:eb:ff:67:48:7a:35: 125 | 68:76:7b:4c:e5:8c:2d:65:3a:88:8e:f8:b2:62:49:28:b1:73: 126 | 29:19 127 | ``` 128 | 129 | ### Checking sslRootCAPath 130 | 131 | $ openssl x509 -in /opt/splunkforwarder/etc/auth/cacert.pem -text -noout 132 | 133 | ``` 134 | Certificate: 135 | Data: 136 | Version: 1 (0x0) 137 | Serial Number: 16551569488170448198 (0xe5b2fcc16997f546) 138 | Signature Algorithm: sha1WithRSAEncryption 139 | Issuer: C=US, ST=CA, L=San Francisco, O=Splunk, CN=SplunkCommonCA/emailAddress=support@splunk.com 140 | Validity 141 | Not Before: May 11 19:51:37 2015 GMT 142 | Not After : May 8 19:51:37 2025 GMT 143 | Subject: C=US, ST=CA, L=San Francisco, O=Splunk, CN=SplunkCommonCA/emailAddress=support@splunk.com 144 | Subject Public Key Info: 145 | Public Key Algorithm: rsaEncryption 146 | Public-Key: (1024 bit) 147 | Modulus: 148 | 00:c9:99:be:79:ca:f6:a6:d4:6a:86:81:32:b4:75: 149 | f1:d7:58:98:81:d0:58:7c:7e:c7:49:15:17:39:77: 150 | 10:49:3c:56:82:fe:49:66:b5:b2:c5:2d:b6:2e:5d: 151 | d0:b6:26:1e:1c:9b:fb:a1:8f:5f:c5:5a:60:34:59: 152 | b8:5b:d3:6a:e8:01:5d:37:67:74:97:d2:91:f2:15: 153 | ad:d4:77:2a:ab:f5:fe:44:44:9d:00:60:50:3e:cb: 154 | 95:21:6c:c9:c3:f7:39:61:b3:b2:7c:b9:cb:9b:dd: 155 | 7b:c0:f2:b9:fb:f5:e8:e4:62:d0:d7:da:b3:10:58: 156 | f3:59:60:f7:2b:c5:41:21:8b 157 | Exponent: 65537 (0x10001) 158 | Signature Algorithm: sha1WithRSAEncryption 159 | 57:7d:77:3c:b2:6f:6c:27:94:3c:b7:b6:51:55:1f:60:54:5d: 160 | d2:59:3c:a2:02:13:75:72:32:c3:d3:36:15:c3:ab:b1:12:55: 161 | 60:4b:25:e5:10:87:ab:89:d4:0d:d0:c8:ba:ed:4e:a1:bf:d6: 162 | 1e:b6:be:f3:fe:53:10:30:e1:31:d9:e2:0d:da:da:2e:b9:dd: 163 | 3d:6a:ef:c7:61:ab:57:0a:9d:e3:ae:13:cd:d3:7b:f7:d1:10: 164 | 7e:78:42:89:33:ae:70:17:a3:3f:af:fd:a1:89:93:38:c4:a5: 165 | 21:30:ad:65:30:2c:0d:64:a0:4f:08:ff:45:c5:13:0c:56:6c: 166 | 46:ed 167 | ``` 168 | 169 | 170 | ### Whats the difference? 171 | The self signed cert has: 172 | 173 | ``` 174 | Subject: CN=SplunkServerDefaultCert, O=SplunkUser 175 | ``` 176 | 177 | And is derived from the CA cert. 178 | 179 | ## sslPassword set incorrectly 180 | 181 | ### SplunkSSL port Unreachable (Indexer) 182 | Use openssl to connect to the port you configured for recieving data. 183 | 184 | $ openssl s_client -connect 45.55.233.75:9997 185 | 186 | If you cannot connect this means the port is not being opened by the indexer with SSL settings. 187 | 188 | Search for: index=_internal log_level=ERROR component=TcpInputConfig 189 | 190 | If you see messages like the following: 191 | 03-04-2017 20:43:37.701 +0000 ERROR TcpInputConfig - SSL server certificate not found, or password is wrong - SSL ports will not be opened 192 | 03-04-2017 20:43:37.701 +0000 ERROR TcpInputConfig - SSL context not found. Will not open splunk to splunk (SSL) IPv4 port 9997 193 | 194 | The sslPassword in inputs.conf is likely wrong. The default password for the Splunk self-signed cert is "password" 195 | 196 | Messages you might see on your forwarders: 197 | 03-04-2017 20:48:23.148 +0000 ERROR TcpOutputFd - Connection to host=45.55.233.75:9997 failed 198 | 199 | ### Using old configurations 200 | These are issues that will show up in the Splunkd logs, SSL will still work but the settings are deprecated. 201 | 202 | ### Web.conf 203 | 02-06-2017 14:46:31.111 -0500 WARN SSLOptions - web.conf/[settings]/caCertPath: deprecated; use 'serverCert' instead 204 | 205 | ### Inputs.conf 206 | 02-06-2017 14:46:29.516 -0500 WARN SSLOptions - inputs.conf/[SSL]/rootCA: deprecated; use 'sslRootCAPath' instead in server.conf/[sslConfig] 207 | 208 | ### Server.conf 209 | 02-06-2017 14:46:14.374 -0500 WARN SSLOptions - server.conf/[sslConfig]/sslKeysfilePassword: deprecated; use 'sslPassword' instead 210 | 211 | 212 | ### SSL Version issues: 213 | 03-07-2017 15:28:11.721 -0500 ERROR TcpInputProc - Error encountered for connection from src=10.10.184.12:44380. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol 214 | 215 | sslVersions = tls -------------------------------------------------------------------------------- /survival-guide/stream_config/load_pcaps_from_list.md: -------------------------------------------------------------------------------- 1 | # Create the list of pcap files to iterate through 2 | ls */*.pcap > all_pcap_files.txt 3 | 4 | # append absolute path to beginning of each line in vi 5 | :%s/^/\/splunk\//c 6 | 7 | # iterate through list and index using stream 8 | $ while read line; do ./streamfwd "${line}"; done < all_pcap_files.txt 9 | -------------------------------------------------------------------------------- /survival-guide/stream_config/stream_update.py: -------------------------------------------------------------------------------- 1 | # Code written by VladimirShcherbakov, Validated by Anthony Tellez 2 | import urllib2 3 | import urllib 4 | import ssl 5 | import json 6 | from collections import OrderedDict 7 | import base64 8 | import xml.etree.ElementTree as ET 9 | import getopt 10 | import sys 11 | 12 | def usage(): 13 | 14 | """ 15 | ################# 16 | How to run this script 17 | ################# 18 | 19 | ** All arguments are optional ** 20 | -s: splunkd REST API URL, default: https://localhost:8089 21 | -u: user name; admin if not specified 22 | -p: password; changeme if not specified 23 | -i: Splunk index to set all non-aggregated streams to use 24 | Examples: 25 | python stream_update.py 26 | python stream_update.py -s https://mysplunk:8089 -p mypwd -u splunkuser 27 | 28 | """ 29 | print usage.__doc__ 30 | 31 | 32 | # Defaults 33 | SPLUNK_SERVER_URL = "https://localhost:8089" 34 | USER = "admin" 35 | PWD = "changeme" 36 | INDEX="" 37 | 38 | def update(stream): 39 | '''put your bulk stream update code here - return True if stream was updated; False otherwise''' 40 | 41 | # check if the stream is aggregated 42 | is_aggregated = (next((f for f in stream['fields'] if f['aggType'] != 'value'), None) != None) 43 | 44 | retval = False; 45 | 46 | #example: enable 'network_interface' field if it's present 47 | if (not is_aggregated): 48 | field = next((f for f in stream['fields'] if f['name'] == 'network_interface'), None) 49 | if field != None: 50 | field['enabled'] = True 51 | retval = True 52 | 53 | if stream['enabled'] != True: 54 | print "enabling stream " + stream['id'] 55 | stream['enabled'] = True 56 | retval = True 57 | 58 | if stream['statsOnly'] == True: 59 | print "turning off estimate mode for stream " + stream['id'] 60 | stream['statsOnly'] = False 61 | retval = True 62 | 63 | if INDEX != "" and stream.get('index', "") != INDEX: 64 | print "setting stream's splunk index to " + INDEX 65 | stream['index'] = INDEX 66 | retval = True 67 | 68 | return retval # return True if stream was updated by this call 69 | 70 | # this prevents certificate validation issues, comment if not needed 71 | ssl._create_default_https_context = ssl._create_unverified_context 72 | 73 | 74 | """Helper class to issue a PUT request""" 75 | class MethodRequest(urllib2.Request): 76 | def __init__(self, *args, **kwargs): 77 | if 'method' in kwargs: 78 | self._method = kwargs['method'] 79 | del kwargs['method'] 80 | else: 81 | self._method = None 82 | return urllib2.Request.__init__(self, *args, **kwargs) 83 | 84 | def get_method(self, *args, **kwargs): 85 | if self._method is not None: 86 | return self._method 87 | return urllib2.Request.get_method(self, *args, **kwargs) 88 | 89 | 90 | def readAsJson(data): 91 | jsonResource = json.loads(data, object_pairs_hook=OrderedDict) 92 | return jsonResource 93 | 94 | def readStreams(url): 95 | retval = readAsJson(urllib2.urlopen(url + "?output_mode=json").read()) 96 | return retval['entry'][0]['content'] 97 | 98 | 99 | def saveStream(stream, url, sessionKey): 100 | req_url = url + "?output_mode=json&id=" + stream['id'] 101 | 102 | try: 103 | req = MethodRequest(req_url, method='PUT') 104 | req.add_header('Authorization', 'Splunk {0}'.format(sessionKey)) 105 | req.add_header('Content-Type', 'application/json') 106 | req.add_data(json.dumps(stream)) 107 | return urllib2.urlopen(req) 108 | except urllib2.HTTPError, error: 109 | print error.read() 110 | raise error 111 | except Exception, e: 112 | raise e 113 | 114 | 115 | def updateStreams(streams, url): 116 | # run all streams through the update() method 117 | for stream in streams: 118 | #print stream['id'] 119 | if (update(stream)): 120 | print "Saving stream: " + stream['id'] 121 | saveStream(stream, url, sessionKey) 122 | else: 123 | print "Skipping stream: " + stream['id'] + ' - no changes detected' 124 | 125 | def login(url): 126 | req = urllib2.Request(url) 127 | req.add_data("username=%s&password=%s" % (USER, PWD)) 128 | responseXml = ET.fromstring(urllib2.urlopen(req).read()) 129 | return responseXml.find('sessionKey').text 130 | 131 | 132 | if __name__ == '__main__': 133 | 134 | try: 135 | opts, args = getopt.getopt(sys.argv[1:], 'hs:u:p:i:') 136 | except getopt.GetoptError: 137 | print "error" 138 | usage() 139 | sys.exit(2) 140 | 141 | for opt, arg in opts: 142 | if opt in ('-h', '--help'): 143 | usage() 144 | sys.exit(2) 145 | elif opt in ('-s'): 146 | SPLUNK_SERVER_URL = arg 147 | elif opt in ('-p'): 148 | PWD = arg 149 | elif opt in ('-u'): 150 | USER = arg 151 | elif opt in ('-i'): 152 | INDEX = arg 153 | if INDEX != "": 154 | print "Setting Splunk index to " + INDEX 155 | 156 | print "logging to " + SPLUNK_SERVER_URL + " as " + USER 157 | 158 | url_base = SPLUNK_SERVER_URL + "/services/splunk_app_stream/streams" 159 | login_url = SPLUNK_SERVER_URL + "/services/auth/login" 160 | 161 | 162 | sessionKey = login(login_url) 163 | 164 | streams = readStreams(url_base) 165 | 166 | updateStreams(streams, url_base) 167 | -------------------------------------------------------------------------------- /survival-guide/windows_administration/create_server_list.md: -------------------------------------------------------------------------------- 1 | # Run the following command from powershell to create a server list 2 | 3 | (Get-QADComputer -OSName *server*).name > servlist.txt 4 | -------------------------------------------------------------------------------- /survival-guide/windows_administration/remote_start_stop_splunk.md: -------------------------------------------------------------------------------- 1 | # Create an array from your server list to iterate over in powershell: 2 | $hosts=Get-Content .\servlist.txt 3 | 4 | # START SERVICE: 5 | $hosts | %{Get-Service -name SplunkForwarder -ComputerName $_} | Set-Service -verbose -status running 6 | 7 | # STOP SERVICE: 8 | $hosts | %{Get-Service -name SplunkForwarder -ComputerName $_} | Set-Service -verbose -status stopped -------------------------------------------------------------------------------- /syslog_ng_configs/syslog-ng_ip.conf: -------------------------------------------------------------------------------- 1 | @version:3.2 2 | 3 | # =============================================================================================== 4 | # Configuration file for syslog-ng, customized for remote logging 5 | # =============================================================================================== 6 | # Options 7 | # Note about $HOST / HOST 8 | # Description: The name of the source host where the message originates from. 9 | # If the message traverses several hosts and the chain_hostnames() option is on, the first host in the chain is used. 10 | # If the keep_hostname() option is disabled (keep_hostname(no)), the value of the $HOST macro will be the DNS hostname of the host that sent the message to syslog-ng OSE (that is, the DNS hostname of the last hop). In this case the $HOST and $HOST_FROM macros will have the same value. 11 | # If the keep_hostname() option is enabled (keep_hostname(yes)), the value of the $HOST macro will be the hostname retrieved from the log message. That way the name of the original sender host can be used, even if there are log relays between the sender and the server. 12 | # ----------------------------------------------------------------------------------------------- 13 | 14 | options { 15 | 16 | # If the log message is forwarded to the logserver via a relay, and the 17 | # chain_hostnames() option is 'yes', the relay adds its own hostname to 18 | # the hostname of the client, separated with a / character. 19 | chain_hostnames(no); 20 | 21 | # Check client hostnames for valid DNS characters 22 | check_hostname (yes); 23 | 24 | # Specify whether to trust hostname in the log message. 25 | # If "yes", then it is left unchanged, if "no" the server replaces 26 | # it with client's DNS lookup value. 27 | keep_hostname (no); 28 | 29 | # Use DNS fully qualified domain names (FQDN) 30 | # for the names of log file folders 31 | use_fqdn (no); 32 | use_dns (no); 33 | 34 | # Set permissions on newly created 'messages' files 35 | owner("root"); 36 | group("root"); 37 | perm(0755); 38 | 39 | # Set permissions on newly created directories 40 | dir_owner("root"); 41 | dir_group("root"); 42 | dir_perm(0755); 43 | create_dirs(yes); 44 | 45 | # Maximum length of a message in bytes. 46 | log_msg_size(18192); 47 | }; 48 | 49 | 50 | # =============================================================================================== 51 | # Source 52 | # Template: 53 | # Source: s_ { }; 54 | # ANY IP on TCP Port 514: tcp(ip(0.0.0.0) port(514)); 55 | # ANY IP on UDP Port 514: udp(ip(0.0.0.0) port(514)); 56 | # Syslog Localhost logs: internal(); 57 | # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 58 | #source s_snmptrapd { 59 | # file("/var/log/syslog-ng/snmp/wodc-sec-ws-mgt/raw_messages" default-facility(daemon) follow_freq(1) flags(no-parse)); 60 | #}; 61 | source s_remote { 62 | tcp(ip(0.0.0.0) port(514)); 63 | udp(ip(0.0.0.0) port(514)); 64 | }; 65 | source s_local { 66 | internal(); 67 | }; 68 | 69 | # =============================================================================================== 70 | # Filters 71 | # Templates: 72 | # Filter: filter f_ { }; 73 | # Source type 'host();': host("^$"); 74 | # Source type 'message();': message("^$"); 75 | # Source type 'netmask(ip/mask);': netmask(192.168.1.0/24); 76 | # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 77 | 78 | 79 | filter f_blue_coat_proxy { 80 | host("^XX\.XX\.XX\.XX$") or 81 | host("^XX\.XX\.XX\.XX$") or 82 | host("^XX\.XX\.XX\.XX$") or 83 | host("^XX\.XX\.XX\.XX$") or 84 | host("^XX\.XX\.XX\.XX$") or 85 | host("^XX\.XX\.XX\.XX$"); 86 | }; 87 | 88 | filter f_trip_wire { 89 | host("^XX\.XX\.XX\.XX$"); 90 | }; 91 | 92 | filter f_palo_alto { 93 | host("^XX\.XX\.XX\.XX$") or 94 | host("^XX\.XX\.XX\.XX$") or 95 | host("^XX\.XX\.XX\.XX$") or 96 | host("^XX\.XX\.XX\.XX$") or 97 | host("^XX\.XX\.XX\.XX$") or 98 | host("^XX\.XX\.XX\.XX$") or 99 | host("^XX\.XX\.XX\.XX$"); 100 | }; 101 | 102 | filter f_juniper_fw { 103 | host("^XX\.XX\.XX\.XX$") or 104 | host("^XX\.XX\.XX\.XX$") or 105 | host("^XX\.XX\.XX\.XX$") or 106 | host("^XX\.XX\.XX\.XX$") or 107 | host("^XX\.XX\.XX\.XX$") or 108 | host("^XX\.XX\.XX\.XX$") or 109 | host("^XX\.XX\.XX\.XX$") or 110 | host("^XX\.XX\.XX\.XX$"); 111 | }; 112 | 113 | filter f_rsa { 114 | host("^XX\.XX\.XX\.XX$") or 115 | host("^XX\.XX\.XX\.XX$") or 116 | host("^XX\.XX\.XX\.XX$"); 117 | }; 118 | 119 | filter f_rsa_netscout { 120 | host("^XX\.XX\.XX\.XX$"); 121 | }; 122 | 123 | filter f_web_sense { 124 | host("^XX\.XX\.XX\.XX$"); 125 | }; 126 | 127 | filter f_cisco_asa { 128 | host("^XX\.XX\.XX\.XX$") or 129 | host("^XX\.XX\.XX\.XX$") or 130 | host("^XX\.XX\.XX\.XX$") or 131 | host("^XX\.XX\.XX\.XX$") or 132 | host("^XX\.XX\.XX\.XX$") ; 133 | }; 134 | 135 | filter f_avaya_switch { 136 | host("^XX\.XX\.XX\.XX$") or 137 | host("^XX\.XX\.XX\.XX$") or 138 | host("^XX\.XX\.XX\.XX$") or 139 | host("^XX\.XX\.XX\.XX$") or 140 | host("^XX\.XX\.XX\.XX$") ; 141 | }; 142 | 143 | filter f_separatedbyhosts { 144 | host("^$"); 145 | }; 146 | 147 | 148 | # =============================================================================================== 149 | # Destinations 150 | # Template: 151 | # destination d_ {file("/var/log/syslog-ng//$HOST/$YEAR-$MONTH-$DAY/messages");}; 152 | # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 153 | 154 | #destination d_source_syslog {file("/var/log/syslog-ng/unknown/$HOST/$YEAR-$MONTH-$DAY/messages");}; 155 | destination d_loghost {udp("XX.XX.XX.XX"port(514)spoof_source(yes));}; 156 | destination d_separatedbyhosts {file("/var/log/syslog-ng/unknown/$HOST/$YEAR-$MONTH-$DAY/messages");}; 157 | destination d_blue_coat_proxy {file("/var/log/syslog-ng/blue_coat_proxy/$HOST/$YEAR-$MONTH-$DAY/messages");}; 158 | destination d_trip_wire {file("/var/log/syslog-ng/trip_wire/$HOST/$YEAR-$MONTH-$DAY/messages");}; 159 | destination d_palo_alto {file("/var/log/syslog-ng/palo_alto/$HOST/$YEAR-$MONTH-$DAY/messages");}; 160 | destination d_cisco_asa {file("/var/log/syslog-ng/cisco_asa/$HOST/$YEAR-$MONTH-$DAY/messages");}; 161 | destination d_juniper_fw {file("/var/log/syslog-ng/juniper_fw/$HOST/$YEAR-$MONTH-$DAY/messages");}; 162 | destination d_rsa {file("/var/log/syslog-ng/rsa/$HOST/$YEAR-$MONTH-$DAY/messages");}; 163 | destination d_rsa_netscout {file("/var/log/syslog-ng/rsa_netscout/$HOST/$YEAR-$MONTH-$DAY/messages");}; 164 | destination d_web_sense {file("/var/log/syslog-ng/web_sense/$HOST/$YEAR-$MONTH-$DAY/messages");}; 165 | destination d_avaya_switch {file("/var/log/syslog-ng/avaya_switch/$HOST/$YEAR-$MONTH-$DAY/messages");}; 166 | # =============================================================================================== 167 | # Log Action 168 | # Template: 169 | # log{ source( s_); flags();} 170 | # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 171 | #log { source(s_snmptrapd); destination(d_snmp); flags(final); }; 172 | #log { source(s_local); destination(d_source_syslog); flags(final); }; 173 | log { source(s_remote); destination(d_loghost); flags(catchall); }; 174 | log { source(s_remote); filter(f_palo_alto); destination(d_palo_alto); flags(final); }; 175 | log { source(s_remote); filter(f_trip_wire); destination(d_trip_wire); flags(final); }; 176 | log { source(s_remote); filter(f_juniper_fw); destination(d_juniper_fw); flags(final); }; 177 | log { source(s_remote); filter(f_cisco_asa); destination(d_cisco_asa); flags(final); }; 178 | log { source(s_remote); filter(f_rsa); destination(d_rsa); flags(final); }; 179 | log { source(s_remote); filter(f_rsa_netscout); destination(d_rsa_netscout); flags(final); }; 180 | log { source(s_remote); filter(f_web_sense); destination(d_web_sense); flags(final); }; 181 | log { source(s_remote); filter(f_blue_coat_proxy); destination(d_blue_coat_proxy); flags(final); }; 182 | log { source(s_remote); filter(f_trip_wire); destination(d_trip_wire); flags(final); }; 183 | log { source(s_remote); filter(f_avaya_switch); destination(d_avaya_switch); flags(final); }; 184 | log { source(s_remote); destination(d_separatedbyhosts); flags(fallback); }; -------------------------------------------------------------------------------- /syslog_ng_configs/syslog-ng_port.conf: -------------------------------------------------------------------------------- 1 | @version:3.2 2 | 3 | # =============================================================================================== 4 | # Configuration file for syslog-ng, customized for remote logging 5 | # =============================================================================================== 6 | # Options 7 | # Note about $HOST / HOST 8 | # Description: The name of the source host where the message originates from. 9 | # If the message traverses several hosts and the chain_hostnames() option is on, the first host in the chain is used. 10 | # If the keep_hostname() option is disabled (keep_hostname(no)), the value of the $HOST macro will be the DNS hostname of the host that sent the message to syslog-ng OSE (that is, the DNS hostname of the last hop). In this case the $HOST and $HOST_FROM macros will have the same value. 11 | # If the keep_hostname() option is enabled (keep_hostname(yes)), the value of the $HOST macro will be the hostname retrieved from the log message. That way the name of the original sender host can be used, even if there are log relays between the sender and the server. 12 | # ----------------------------------------------------------------------------------------------- 13 | 14 | options { 15 | 16 | # If the log message is forwarded to the logserver via a relay, and the 17 | # chain_hostnames() option is 'yes', the relay adds its own hostname to 18 | # the hostname of the client, separated with a / character. 19 | chain_hostnames(no); 20 | 21 | # Check client hostnames for valid DNS characters 22 | check_hostname (yes); 23 | 24 | # Specify whether to trust hostname in the log message. 25 | # If "yes", then it is left unchanged, if "no" the server replaces 26 | # it with client's DNS lookup value. 27 | keep_hostname (no); 28 | 29 | # Use DNS fully qualified domain names (FQDN) 30 | # for the names of log file folders 31 | use_fqdn (no); 32 | use_dns (no); 33 | 34 | # Set permissions on newly created 'messages' files 35 | owner("root"); 36 | group("root"); 37 | perm(0755); 38 | 39 | # Set permissions on newly created directories 40 | dir_owner("root"); 41 | dir_group("root"); 42 | dir_perm(0755); 43 | create_dirs(yes); 44 | 45 | # Maximum length of a message in bytes. 46 | log_msg_size(18192); 47 | }; 48 | 49 | 50 | # =============================================================================================== 51 | # Source 52 | # Template: 53 | # Source: s_ { }; 54 | # ANY IP on TCP Port 514: tcp(ip(0.0.0.0) port(514)); 55 | # ANY IP on UDP Port 514: udp(ip(0.0.0.0) port(514)); 56 | # Syslog Localhost logs: internal(); 57 | # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 58 | #source s_snmptrapd { 59 | # file("/var/log/syslog-ng/snmp/wodc-sec-ws-mgt/raw_messages" default-facility(daemon) follow_freq(1) flags(no-parse)); 60 | #}; 61 | 62 | source s_remote { 63 | tcp(ip(0.0.0.0) port(514)); 64 | udp(ip(0.0.0.0) port(514)); 65 | }; 66 | source s_blue_coat_proxy { 67 | tcp(ip(0.0.0.0) port(51400)); 68 | udp(ip(0.0.0.0) port(51400)); 69 | }; 70 | 71 | source s_trip_wire { 72 | tcp(ip(0.0.0.0) port(51401)); 73 | udp(ip(0.0.0.0) port(51401)); 74 | }; 75 | 76 | source s_palo_alto { 77 | tcp(ip(0.0.0.0) port(51402)); 78 | udp(ip(0.0.0.0) port(51402)); 79 | }; 80 | 81 | source s_juniper_fw { 82 | tcp(ip(0.0.0.0) port(51403)); 83 | udp(ip(0.0.0.0) port(51403)); 84 | }; 85 | 86 | source s_rsa { 87 | tcp(ip(0.0.0.0) port(51404)); 88 | udp(ip(0.0.0.0) port(51404)); 89 | }; 90 | 91 | source s_rsa_netscout { 92 | tcp(ip(0.0.0.0) port(51405)); 93 | udp(ip(0.0.0.0) port(51405)); 94 | }; 95 | 96 | source s_web_sense { 97 | tcp(ip(0.0.0.0) port(51406)); 98 | udp(ip(0.0.0.0) port(51406)); 99 | }; 100 | 101 | source s_cisco_asa { 102 | tcp(ip(0.0.0.0) port(51407)); 103 | udp(ip(0.0.0.0) port(51407)); 104 | }; 105 | 106 | source s_avaya_switch { 107 | tcp(ip(0.0.0.0) port(51408)); 108 | udp(ip(0.0.0.0) port(51408)); 109 | }; 110 | 111 | 112 | source s_local { 113 | internal(); 114 | }; 115 | # =============================================================================================== 116 | # Filters 117 | # Templates: 118 | # Filter: filter f_ { }; 119 | # Source type 'host();': host("^$"); 120 | # Source type 'message();': message("^$"); 121 | # Source type 'netmask(ip/mask);': netmask(192.168.1.0/24); 122 | # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 123 | 124 | filter f_separatedbyhosts { 125 | host("^$"); 126 | }; 127 | 128 | 129 | # =============================================================================================== 130 | # Destinations 131 | # Template: 132 | # destination d_ {file("/var/log/syslog-ng//$HOST/$YEAR-$MONTH-$DAY/message");}; 133 | # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 134 | 135 | #destination d_source_syslog {file("/var/log/syslog-ng/unknown/$HOST/$YEAR-$MONTH-$DAY/messages");}; 136 | destination d_loghost {udp("XX.XX.XX.XX"port(514)spoof_source(yes));}; 137 | destination d_separatedbyhosts {file("/var/log/syslog-ng/unknown/$HOST/$YEAR-$MONTH-$DAY/messages");}; 138 | destination d_blue_coat_proxy {file("/var/log/syslog-ng/blue_coat_proxy/$HOST/$YEAR-$MONTH-$DAY/messages");}; 139 | destination d_trip_wire {file("/var/log/syslog-ng/trip_wire/$HOST/$YEAR-$MONTH-$DAY/messages");}; 140 | destination d_palo_alto {file("/var/log/syslog-ng/palo_alto/$HOST/$YEAR-$MONTH-$DAY/messages");}; 141 | destination d_cisco_asa {file("/var/log/syslog-ng/cisco_asa/$HOST/$YEAR-$MONTH-$DAY/messages");}; 142 | destination d_juniper_fw {file("/var/log/syslog-ng/juniper_fw/$HOST/$YEAR-$MONTH-$DAY/messages");}; 143 | destination d_rsa {file("/var/log/syslog-ng/rsa/$HOST/$YEAR-$MONTH-$DAY/messages");}; 144 | destination d_rsa_netscout {file("/var/log/syslog-ng/rsa_netscout/$HOST/$YEAR-$MONTH-$DAY/messages");}; 145 | destination d_web_sense {file("/var/log/syslog-ng/web_sense/$HOST/$YEAR-$MONTH-$DAY/messages");}; 146 | destination d_avaya_switch {file("/var/log/syslog-ng/avaya_switch/$HOST/$YEAR-$MONTH-$DAY/messages");}; 147 | 148 | # =============================================================================================== 149 | # Log Action 150 | # Template: 151 | # log{ source( s_); flags();} 152 | # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 153 | #log { source(s_snmptrapd); destination(d_snmp); flags(final); }; 154 | #log { source(s_local); destination(d_source_syslog); flags(final); }; 155 | log { source(s_remote); destination(d_loghost); flags(catchall); }; 156 | log { source(s_palo_alto); destination(d_palo_alto); flags(final); }; 157 | log { source(s_trip_wire); destination(d_trip_wire); flags(final); }; 158 | log { source(s_juniper_fw); destination(d_juniper_fw); flags(final); }; 159 | log { source(s_cisco_asa); destination(d_cisco_asa); flags(final); }; 160 | log { source(s_rsa); destination(d_rsa); flags(final); }; 161 | log { source(s_rsa_netscout); destination(d_rsa_netscout); flags(final); }; 162 | log { source(s_web_sense); destination(d_web_sense); flags(final); }; 163 | log { source(s_blue_coat_proxy); destination(d_blue_coat_proxy); flags(final); }; 164 | log { source(s_trip_wire); destination(d_trip_wire); flags(final); }; 165 | log { source(s_ravaya_switch); destination(d_avaya_switch); flags(final); }; 166 | log { source(s_remote); destination(d_separatedbyhosts); flags(fallback); }; -------------------------------------------------------------------------------- /upgrade/splunk-core/local.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: 5 | # Parameters: ${1} = path to splunk install 6 | # Example usage: $ bash splunkinstall.sh splunk-6.3.2-aaff59bb082c-Linux-x86_64.tgz 7 | # 8 | # Privileges: Must be run as root 9 | # Authors: Anthony Tellez 10 | # 11 | # Notes: 12 | # 13 | # 14 | # Revision: Last change: XX/XX/2017 by AT :: 15 | # =========================================================== 16 | # 17 | sudo su - splunk -c '/opt/splunk/bin/splunk stop' 18 | tar -zxvf ${1} -C /opt && chown -R splunk:splunk /opt/splunk 19 | sudo su - splunk -c '/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt' 20 | -------------------------------------------------------------------------------- /upgrade/splunk-core/remote.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will install splunk and complete some initial setup steps 5 | # Parameters: ${1} = path to splunk install .tgz file 6 | # ${2} = list of hosts to install Splunk Enterprise 7 | # Example usage: $ bash upgrade_splunk.sh splunk-6.3.2-aaff59bb082c-Linux-x86_64.tgz listofhosts.txt 8 | # 9 | # Privileges: Must be run as root 10 | # Authors: Chris Tribie, Anthony Tellez 11 | # 12 | # Notes: This script can use customized Splunk install tar or the default from Splunk.com. 13 | # Our custom install comprised the following changes from the base install: 14 | # in ~/splunkforwarder/etc/system/local/ 15 | # deploymentclient.conf - preloaded deployment server info 16 | # Alternatively, ~/splunkforwarder/etc/apps/ 17 | # org_all_deploymentclient/local/ 18 | # deploymentclient.conf - preloaded deployment server info 19 | # in ~/splunkforwarder/etc/ 20 | # splunk-launch.conf - SPLUNK_FIPS=1 - this must be done on first boot to ensure splunk enables the FIPS module 21 | # after untar, splunk is started, the admin password is changed, and 22 | # splunk is set to run at boot time. Since everything up to this point was 23 | # done as the root user, we need to change ownership to the splunk user. 24 | # This is done via the chown command. Last step is to start splunk again. 25 | # 26 | # Revision: Last change: 03/08/2016 by AT :: Increased Security of password entry mechanism 27 | # =========================================================== 28 | # 29 | stopSplunk="sudo su - splunk -c '/opt/splunk/bin/splunk stop'" 30 | untarSplunk="tar -zxvf /tmp/${1} -C /opt && chown -R splunk:splunk /opt/splunk" 31 | startSplunk="sudo su - splunk -c '/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt'" 32 | for HOST in $(< $2); do 33 | scp -r "${1}" $HOST:/tmp 34 | ssh $HOST "$stopSplunk" 35 | ssh $HOST "$untarSplunk" 36 | ssh $HOST "$startSplunk" 37 | if [ $? -ne 0 ]; then 38 | echo "---- COULD NOT CONNECT TO $HOST ----" 39 | fi 40 | done -------------------------------------------------------------------------------- /upgrade/splunk-uf/local.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: 5 | # Parameters: ${1} = path to splunkforwarder install 6 | # Example usage: $ bash splunkinstall.sh splunk-6.3.2-aaff59bb082c-Linux-x86_64.tgz 7 | # 8 | # Privileges: Must be run as root 9 | # Authors: Anthony Tellez 10 | # 11 | # Notes: 12 | # 13 | # 14 | # Revision: Last change: XX/XX/2017 by AT :: 15 | # =========================================================== 16 | # 17 | sudo su - splunk -c '/opt/splunkforwarder/bin/splunk stop' 18 | tar -zxvf ${1} -C /opt && chown -R splunk:splunk /opt/splunkforwarder 19 | sudo su - splunk -c '/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt' 20 | -------------------------------------------------------------------------------- /upgrade/splunk-uf/remote.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # =========================================================== 4 | # Purpose: This script will install splunk and complete some initial setup steps 5 | # Parameters: ${1} = path to splunk install .tgz file 6 | # ${2} = list of hosts to install the uf 7 | # Example usage: $ bash upgrade_splunk.sh splunk-6.3.2-aaff59bb082c-Linux-x86_64.tgz listofhosts.txt 8 | # 9 | # Privileges: Must be run as root 10 | # Authors: Chris Tribie, Anthony Tellez 11 | # 12 | # Notes: This script can use customized Splunk install tar or the default from Splunk.com. 13 | # Our custom install comprised the following changes from the base install: 14 | # in ~/splunkforwarder/etc/system/local/ 15 | # deploymentclient.conf - preloaded deployment server info 16 | # Alternatively, ~/splunkforwarder/etc/apps/ 17 | # org_all_deploymentclient/local/ 18 | # deploymentclient.conf - preloaded deployment server info 19 | # in ~/splunkforwarder/etc/ 20 | # splunk-launch.conf - SPLUNK_FIPS=1 - this must be done on first boot to ensure splunk enables the FIPS module 21 | # after untar, splunk is started, the admin password is changed, and 22 | # splunk is set to run at boot time. Since everything up to this point was 23 | # done as the root user, we need to change ownership to the splunk user. 24 | # This is done via the chown command. Last step is to start splunk again. 25 | # 26 | # Revision: Last change: 03/08/2016 by AT :: Increased Security of password entry mechanism 27 | # =========================================================== 28 | # 29 | stopSplunk="sudo su - splunk -c '/opt/splunkforwarder/bin/splunk stop'" 30 | untarSplunk="tar -zxvf /tmp/${1} -C /opt && chown -R splunk:splunk /opt/splunkforwarder" 31 | startSplunk="sudo su - splunk -c '/opt/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt'" 32 | for HOST in $(< $2); do 33 | scp -r "${1}" $HOST:/tmp 34 | ssh $HOST "$stopSplunk" 35 | ssh $HOST "$untarSplunk" 36 | ssh $HOST "$startSplunk" 37 | if [ $? -ne 0 ]; then 38 | echo "---- COULD NOT CONNECT TO $HOST ----" 39 | fi 40 | done --------------------------------------------------------------------------------