├── README.md ├── download_lookups_from_splunk.py ├── update_lookups_from_splunk.py └── upload_lookups_to_splunk.py /README.md: -------------------------------------------------------------------------------- 1 | # lookup-editor_scripts 2 | 3 | 4 | ## Upload lookups - [upload_lookups_to_splunk.py](https://github.com/mthcht/lookup-editor_scripts/blob/main/upload_lookups_to_splunk.py) 5 | - Simple script using splunk application [lookup-editor](https://splunkbase.splunk.com/app/1724) (renamed Splunk App for Lookup File Editing 6 | ) endpoint to upload multiple lookups at once: 7 | ![2022-12-24 08_37_55-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209426236-8a713d04-f128-4c52-97c3-0e2b6109aeac.png) 8 | 9 | 10 | ## Update lookups - [update_lookups_from_splunk.py](https://github.com/mthcht/lookup-editor_scripts/blob/main/update_lookups_from_splunk.py) 11 | - script using splunk application [lookup-editor](https://splunkbase.splunk.com/app/1724) endpoint to update part of a lookup or multiple lookups: 12 | 13 | 14 | ### Example of using the -p option (can be done with multiple lookups at once) 15 | 16 | The lookup we have on splunk search app: 17 | ![2022-12-26 23_09_01-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209585956-1eab4858-a727-488b-a065-aad4d6f6540c.png) 18 | 19 | 20 | Execution of the script (script asked for input, we paste the content in the terminal and typed 'ok' to confirm (can be done on multiple lookups at once): 21 | ![2022-12-26 23_10_35-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209585959-931ad7e7-6096-4df9-a538-87a3036698a1.png) 22 | ![2022-12-26 23_11_05-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209585964-ea14a374-1368-4f7d-bdfd-ee7f85522788.png) 23 | 24 | 25 | result: 26 | ![2022-12-26 23_11_27-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209585966-6d410f30-34a7-45a6-ad43-f9d7f40cbe7f.png) 27 | 28 | 29 | 30 | 31 | --- 32 | 33 | ### Example of using the -i option (can be done with multiple lookups at once) 34 | 35 | The lookup we have on splunk search app: 36 | 37 | ![2022-12-26 22_32_47-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209585490-9fe0d7da-0261-45c3-aa71-76623fd36400.png) 38 | 39 | Execution of the script (script asked for each input, can be done on multiple lookups at once): 40 | 41 | ![2022-12-26 22_56_19-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209585499-597ee43b-cac3-44c2-9863-e57b5c312c8f.png) 42 | 43 | result: 44 | ![image](https://user-images.githubusercontent.com/75267080/209585461-11de78f1-fe6c-4fa2-be51-dabeac8ccedb.png) 45 | 46 | 47 | 48 | --- 49 | 50 | ### Example of merging two csv files with option -f 51 | *(not limited to 2 files, we can merge demo_file_to_merge to all the lookups we want on splunk):* 52 | 53 | The lookup we have on splunk search app: 54 | 55 | ![image](https://user-images.githubusercontent.com/75267080/209583591-650e3113-deb9-489e-baf4-5b4e58b5ae25.png) 56 | 57 | The csv file on our desktop we want to merge to the lookup test.csv: 58 | 59 | ![2022-12-26 21_48_00-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209583442-ec5dbba9-8349-41a1-9b03-9b4f9bf32bf7.png) 60 | 61 | 62 | Execution of the script to merge both files: 63 | 64 | ![2022-12-26 21_54_20-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209583455-06c86f30-bea6-4747-9e0c-954ce5c6e353.png) 65 | 66 | 67 | result: 68 | 69 | ![image](https://user-images.githubusercontent.com/75267080/209583519-460a3cdb-edb6-4104-8238-96460117d96f.png) 70 | 71 | --- 72 | 73 | ### Example of merging tree csv files with option -f: 74 | 75 | The lookups we have on splunk search app: 76 | 77 | ![2022-12-26 22_18_27-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209584199-c7cb46b1-f3d2-4583-b13d-46dff03de9e1.png) 78 | ![2022-12-26 22_18_36-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209584203-bcba925e-aaba-4ef6-a722-3c6922ad684c.png) 79 | 80 | The csv file on our desktop we want to merge to the lookup test.csv and test2.csv: 81 | 82 | ![2022-12-26 22_22_57-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209584212-1b126602-1567-484f-bb5c-6d38c8ca9b5f.png) 83 | 84 | 85 | Execution of the script to the files: 86 | 87 | ![2022-12-26 22_25_22-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209584246-759171ee-e9e1-47f5-8be9-324b6346bac6.png) 88 | ![2022-12-26 22_25_57-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209584252-168c482b-7648-4f11-9afb-e9e463c2fed1.png) 89 | ![2022-12-26 22_26_08-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209584255-fbc1609e-255f-44b8-8a0e-44a08ca93ffa.png) 90 | 91 | 92 | result: 93 | 94 | ![2022-12-26 22_27_01-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209584262-72d286f9-e9a9-4e1f-9021-c1268a3f4758.png) 95 | ![2022-12-26 22_27_12-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209584265-8a140ec5-d153-4f4c-a14d-3674b4e7203b.png) 96 | 97 | 98 | ## Download lookups - [download_lookups_from_splunk.py](https://github.com/mthcht/lookup-editor_scripts/blob/main/download_lookups_from_splunk.py) 99 | - Simple script using splunk application [lookup-editor](https://splunkbase.splunk.com/app/1724) endpoint to download lookup(s) from splunk: 100 | 101 | no arguments (use default values declared in the script) 102 | ![2022-12-27 19_04_48-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209706146-825392de-6341-4d20-b75d-bb8a7e4c40f2.png) 103 | 104 | with arguments: 105 | ![2022-12-27 19_10_50-Windows 10 and later x64 - VMware Workstation](https://user-images.githubusercontent.com/75267080/209706214-9166f846-b7db-484b-ab5f-38d78513d69f.png) 106 | 107 | 108 | [update 2023/05] you can also check out the script https://github.com/beckyburwell/splunk_rest_upload_lookups using the same endpoint API 109 | -------------------------------------------------------------------------------- /download_lookups_from_splunk.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # coding: utf-8 3 | # Simple python script using Splunk lookup-editor https://splunkbase.splunk.com/app/1724 rest endpoint to download lookups, can be used by automation processes (wokring on splunk cloud) 4 | 5 | import requests 6 | import csv 7 | import logging 8 | from requests.packages.urllib3.exceptions import InsecureRequestWarning 9 | import time 10 | import argparse 11 | import sys 12 | import pathlib 13 | 14 | # Disable warning for insecure requests 15 | requests.packages.urllib3.disable_warnings(InsecureRequestWarning) 16 | 17 | # Setup logging 18 | logging.basicConfig(level=logging.INFO) 19 | 20 | # Methods 21 | parser = argparse.ArgumentParser(description="Script download_lookups_from_splunk.py: download multiple lookups from Splunk, can be used for automation processes") 22 | parser.add_argument("-app", "--app", help="Specify the Splunk application, default to 'search' app") 23 | parser.add_argument("-f", "--folder", help="Folder path to save the lookups, default to same directory") 24 | parser.add_argument("-l", "--lookups", nargs='+', help="Enter the name of the lookup(s) to download") 25 | args = parser.parse_args() 26 | 27 | # Check the arguments 28 | if len(sys.argv)==1: 29 | # display help message when no args are passed. 30 | parser.print_help() 31 | print("example usages:\n./download_lookups_from_splunk.py\n./download_lookups_from_splunk.py -app search -l Subnet_Scan_Exclusion_List.csv Port_Scan_Exclusion_List.csv\n\ 32 | ./download_lookups_from_splunk.py -app search -f \"/home/mthcht/lookups/\" -l Subnet_Scan_Exclusion_List.csv Port_Scan_Exclusion_List.csv IOC_List.csv\n") 33 | logging.warning("warning: no arguments given to the script, will use default values declared in the script:") 34 | 35 | 36 | # Declare variables (change them) 37 | splunk_management_protocol = "https" # http or https 38 | splunk_management_dest = "FIXME" #example: myserver.local or 127.0.0.1 39 | splunk_management_port = "FIXME" #example default: 8089 40 | splunk_management_service = "/services/data/lookup_edit/lookup_contents" 41 | if args.app: 42 | splunk_app = args.app #splunk app name, example: search 43 | else: 44 | splunk_app = "search" #splunk app default name, example: search 45 | splunk_username = "FIXME" #splunk user with needed permissions 46 | splunk_password = "FIXME" #splunk user password (i know..) 47 | if args.folder: 48 | lookup_folder = args.folder #folder path, it will contain all the lookups you want to download example (do not forget to put the / or \ at the end of the path and escape the \ for windows path) : "C:\\lookups\\" or "/home/mthcht/lookups/" 49 | else: 50 | lookup_folder = pathlib.Path().absolute() #If no folder path, use default current directory of the script 51 | lookup_type = "csv" 52 | date = time.time() 53 | if args.lookups: 54 | lookup_file_names_list=args.lookups 55 | logging.info("list of lookups to download: {}".format(lookup_file_names_list)) 56 | else: 57 | lookup_file_names_list = ["geo_attr_countries.csv","geo_attr_us_states.csv"] # change these values with the lookup(s) you want to download automatically without giving the -l argument 58 | 59 | # download lookups from Splunk 60 | for lookup_file_name in lookup_file_names_list: 61 | # Send GET request to Splunk server 62 | try: 63 | r = requests.get("{}://{}:{}{}?lookup_type={}&namespace={}&lookup_file={}".format( 64 | splunk_management_protocol,splunk_management_dest,splunk_management_port,splunk_management_service,lookup_type,splunk_app,lookup_file_name), 65 | verify=False, 66 | auth=(splunk_username, splunk_password) 67 | ) 68 | except Exception as e: 69 | logging.error("Error sending request to Splunk server: {}".format(e)) 70 | 71 | if r.status_code == 200: 72 | logging.info("[success] file: \'{}\' in {} app has been downloaded from Lookup editor handler {} and will be saved in {}".format(lookup_file_name,splunk_app,r.url,lookup_folder)) 73 | lookup_content_json_backup = r.json() 74 | # save the file downloaded 75 | try: 76 | lookup_file_name_downloaded = "{}_{}.csv".format(lookup_file_name.split('.csv')[0],date) 77 | with open("{}/{}".format(lookup_folder,lookup_file_name_downloaded), 'w', newline='') as myfile: 78 | wr = csv.writer(myfile) 79 | for row in lookup_content_json_backup: 80 | wr.writerow(row) 81 | myfile.close 82 | logging.info("[sucess] File {} saved in folder {}".format(lookup_file_name,lookup_folder)) 83 | except Exception as e: 84 | logging.error("[failed] Error: Cannot save a a backup of the file {} in {}, reason: {}".format(lookup_file_name,lookup_folder,e)) 85 | else: 86 | logging.error("[failed] Error: Downloading file: \'{}\', status:{}, reason:{}, url:{}".format(lookup_file_name,r.status_code,r.reason,r.url)) 87 | -------------------------------------------------------------------------------- /update_lookups_from_splunk.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # coding: utf-8 3 | # Simple python script using Splunk lookup-editor https://splunkbase.splunk.com/app/1724 rest endpoint to update lookups, can be used by automation processes 4 | 5 | import json 6 | import requests 7 | import csv 8 | import logging 9 | from requests.packages.urllib3.exceptions import InsecureRequestWarning 10 | import time 11 | import argparse 12 | import sys 13 | 14 | # Disable warning for insecure requests 15 | requests.packages.urllib3.disable_warnings(InsecureRequestWarning) 16 | 17 | # Setup logging 18 | logging.basicConfig(level=logging.INFO) 19 | 20 | # Methods 21 | parser = argparse.ArgumentParser(description="Script update_lookups_from_splunk.py: Update one or multiple lookups on Splunk, can be used for automation processes") 22 | parser.add_argument("-i", "--ask_input", help="Ask the user the input for each line to add",action="store_true") 23 | parser.add_argument("-p", "--paste_csv_data", help="Paste the values in this format (line separated by \\n):\nfield1,field2,field3 ...\n,value2,value3 ...\nvalue4,,value5 ...\nvalue6,, ...\n",action="store_true") 24 | parser.add_argument("-f", "--csv_file", help="give the full path of a csv files to merge with the lookups (separated by comma)") 25 | parser.add_argument("-l", "--lookups", nargs='+', help="Enter the name of the lookup(s) to update (don't forget the .csv), if the argument is not given, will take the default list declared in the script") 26 | args = parser.parse_args() 27 | 28 | # Check the arguments 29 | if args.csv_file and args.paste_csv_data: 30 | logging.error("Error: you can't use -f and -p as arguments at the same time, please use only one of them.") 31 | parser.print_help() 32 | sys.exit(1) 33 | elif args.ask_input and args.paste_csv_data: 34 | logging.error("Error: you can't use -i and -p as arguments at the same time, please use only one of them.") 35 | parser.print_help() 36 | sys.exit(1) 37 | elif args.ask_input and args.csv_file: 38 | logging.error("Error, you can't use -i and -f as arguments at the same time, please use only one of them.") 39 | parser.print_help() 40 | sys.exit(1) 41 | if len(sys.argv)==1: 42 | # display help message when no args are passed. 43 | parser.print_help() 44 | print("example usages:\n./update_lookups_from_splunk.py -i -l Subnet_Scan_Exclusion_List.csv Port_Scan_Exclusion_List.csv\n./update_lookups_from_splunk.py -p -l Subnet_Scan_Exclusion_List.csv Port_Scan_Exclusion_List.csv\n\ 45 | ./update_lookups_from_splunk.py -i\n./update_lookups_from_splunk.py -p\n./update_lookups_from_splunk.py -f\n\ 46 | ./update_lookups_from_splunk.py -i -l Attackers_List_IOC.csv IPS_Detections_Exclusion_List.csv\n./update_lookups_from_splunk.py -p -l Attackers_List_IOC.csv\n\ 47 | ./update_lookups_from_splunk.py' -f \"C:\\Users\\mthcht\\Documents\\mylist.csv\" -l Port_Scan_Exclusion_List.csv\n\ 48 | ./update_lookups_from_splunk.py' -f \"/home/mthcht/IOC_list.csv\" -l MISP_IOC_list.csv Attackers_List_IOC.csv") 49 | logging.error("Error: you must choose a method, either -i, -p or -f") 50 | sys.exit(1) 51 | if args.ask_input is None and args.paste_csv_data is None and args.csv_file is None: 52 | parser.print_help() 53 | logging.error("Error: you choose a method, use method -i, -p or -f") 54 | sys.exit(1) 55 | 56 | # Declare variables (change them) 57 | splunk_management_protocol = "https" # http or https 58 | splunk_management_dest = "FIXME" #example: myserver.local or 127.0.0.1 59 | splunk_management_port = "FIXME" #example default: 8089 60 | splunk_management_service = "/services/data/lookup_edit/lookup_contents" 61 | splunk_app = "search" #splunk app name, example: search 62 | splunk_username = "FIXME" #splunk user with needed permissions 63 | splunk_password = "FIXME" #splunk user password (i know..) 64 | lookup_folder = "FIXME" #folder path which contains all the lookups you want to download a backup, example (do not forget to put the / or \ at the end of the path and escape the \ for windows path) : "C:\\lookups\\" or "/home/mthcht/lookups/" 65 | lookup_type = "csv" 66 | date = time.time() 67 | if args.lookups: 68 | lookup_file_names_list=args.lookups 69 | logging.info("list of lookups to update: {}".format(lookup_file_names_list)) 70 | else: 71 | lookup_file_names_list = ["test.csv"] # change this value with the lookup(s) you want to update automatically without giving the -l argument 72 | 73 | 74 | def create_dict_field_values_list(): 75 | field_values_list = [] 76 | num_dict = int(input("How many exclusions or inclusions do you want to create in the lookup ?: ")) 77 | for i in range(num_dict): 78 | fields = input("Enter the fields for the exclusion/inclusion separated by comma (example: src_ip,dest_ip,metadata) {}: ".format(i+1)) 79 | values = input("Enter the values for the exclusion/inclusion separated by comma (example: 192.168.1.1,192.168.1.2,mthcht test) {}: ".format(i+1)) 80 | dictionary = dict(zip(fields.split(','), values.split(','))) 81 | field_values_list.append(dictionary) 82 | return field_values_list 83 | 84 | def create_dict_field_values_list_from_text(): 85 | print("Paste the values in this format (line separated by \\n):\n\nfield1,field2,field3 ...\n,value2,value3 ...\nvalue4,,value5 ...\nvalue6,, ...\n\n:") 86 | field_values_list = [] 87 | field_names = None 88 | while True: 89 | data = input("line (or type 'ok' to confirm the list) : ") 90 | if data == 'ok': 91 | break 92 | elif not field_names: 93 | field_names = data.split(',') 94 | else: 95 | dict_ = {} 96 | for i, item in enumerate(data.split(',')): 97 | dict_[field_names[i]] = item 98 | field_values_list.append(dict_) 99 | return field_values_list 100 | 101 | def create_dict_field_values_list_from_csv(file_path_csv_input): 102 | field_names = [] 103 | field_values_list = [] 104 | with open(file_path_csv_input, 'r') as csvfile: 105 | content = csvfile.readlines() 106 | for i, line in enumerate(content): 107 | if i == 0: 108 | field_names = line.strip().split(',') 109 | else: 110 | data = line.strip() 111 | if data: 112 | dict_ = {} 113 | for j, item in enumerate(data.split(',')): 114 | dict_[field_names[j]] = item 115 | field_values_list.append(dict_) 116 | return field_values_list 117 | 118 | # create new inclusions/exclusions lists in field_values_list 119 | if (args.ask_input): 120 | field_values_list = create_dict_field_values_list() 121 | if (args.paste_csv_data): 122 | field_values_list = create_dict_field_values_list_from_text() 123 | if (args.csv_file): 124 | field_values_list = create_dict_field_values_list_from_csv(args.csv_file) 125 | 126 | # Update lookups on Splunk 127 | for lookup_file_name in lookup_file_names_list: 128 | # Send GET request to Splunk server 129 | try: 130 | r = requests.get("{}://{}:{}{}?lookup_type={}&namespace={}&lookup_file={}".format( 131 | splunk_management_protocol,splunk_management_dest,splunk_management_port,splunk_management_service,lookup_type,splunk_app,lookup_file_name), 132 | verify=False, 133 | auth=(splunk_username, splunk_password) 134 | ) 135 | except Exception as e: 136 | logging.error("Error sending request to Splunk server: {}".format(e)) 137 | 138 | if r.status_code == 200: 139 | logging.info("[success] file: \'{}\' in {} app has been downloaded from Lookup editor handler {} and saved in {}".format(lookup_file_name,splunk_app,r.url,lookup_folder)) 140 | lookup_content_json_backup = r.json() 141 | # save the file downloaded 142 | try: 143 | old_lookup_file_name = "{}_{}.csv".format(lookup_file_name.split('.csv')[0],date) 144 | with open("{}{}".format(lookup_folder,old_lookup_file_name), 'w', newline='') as myfile: 145 | wr = csv.writer(myfile) 146 | for row in lookup_content_json_backup: 147 | wr.writerow(row) 148 | myfile.close 149 | logging.info("[sucess] Backup of the File {} saved in folder {}".format(lookup_file_name,lookup_folder)) 150 | except Exception as e: 151 | logging.error("[failed] Error: Cannot save a a backup of the file {} in {}, reason: {}".format(lookup_file_name,lookup_folder,e)) 152 | lookup_content_json = r.json() 153 | lookup_file_name_path = "{}{}".format(lookup_folder,lookup_file_name) 154 | for field_values_dict in field_values_list: 155 | for key, value in field_values_dict.items(): 156 | logging.info("Adding Key: {}, Value: {} from {} to the lookup {}".format(key, value,field_values_dict,lookup_file_name)) 157 | try: 158 | myfield_index = lookup_content_json[0].index(key) 159 | except ValueError as e: 160 | logging.warning("Warning: {}. adding {} to the columns list for the file {}".format(e,key,lookup_file_name)) 161 | lookup_content_json[0].append(key) 162 | for row in lookup_content_json[1:]: 163 | row.append('') 164 | if 'new_row' in locals(): 165 | new_row.append('') 166 | myfield_index = lookup_content_json[0].index(key) 167 | if myfield_index >= 0: 168 | if 'new_row' not in locals(): 169 | new_row = [''] * len(lookup_content_json[0]) 170 | new_row[myfield_index] = value 171 | else: 172 | logging.error("[failed] Error: cannot add {}:{} to the lookup {}, problem defining 'myfield_index' , trace: myfield_index='{}', field_values_dict='{}'".format(key,value,lookup_file_name,myfield_index,field_values_dict)) 173 | try: 174 | if 'new_row' in locals(): 175 | lookup_content_json.append(new_row) 176 | logging.info("new row '{}' has been added to the lookup content of the file {}".format(new_row,lookup_file_name)) 177 | except Exception as e: 178 | logging.error("[failed] Error: cannot add row '{}' to the lookup {}, trace: myfield_index='{}', field_values_dict='{}'".format(new_row,lookup_file_name,myfield_index,field_values_dict)) 179 | del new_row 180 | 181 | # Save a backup of the new file and delete duplicates rows 182 | try: 183 | lookup_content_json_new = [] 184 | for row in lookup_content_json: 185 | if row not in lookup_content_json_new: 186 | lookup_content_json_new.append(row) 187 | new_lookup_file_name = "{}{}_updated_{}.csv".format(lookup_folder,lookup_file_name,date) 188 | with open(new_lookup_file_name, 'w', newline='') as myfile: 189 | wr = csv.writer(myfile) 190 | for row in lookup_content_json_new: 191 | wr.writerow(row) 192 | myfile.close 193 | logging.info("[sucess] Removed duplicated rows and saved as the new file {} in folder {} as {}".format(lookup_file_name,lookup_folder,new_lookup_file_name)) 194 | except Exception as e: 195 | logging.error("[failed] Error: Cannot save the new file {} in {} as {}, reason: {}".format(lookup_file_name,lookup_folder,new_lookup_file_name,e)) 196 | 197 | # Sent the new lookup to Splunk 198 | try: 199 | r = requests.post("{}://{}:{}{}".format(splunk_management_protocol,splunk_management_dest,splunk_management_port,splunk_management_service), 200 | verify=False, 201 | auth=(splunk_username, splunk_password), 202 | data={"output_mode": "json", 203 | "namespace": splunk_app, 204 | "lookup_file": lookup_file_name, 205 | "contents": json.dumps(lookup_content_json_new)}, 206 | timeout=60 207 | ) 208 | if r.status_code == 200: 209 | logging.info("[success] file: \'{}\' updated and uploaded to Lookup editor handler {} and saved in splunk app \'{}\'".format(lookup_file_name,r.url,splunk_app)) 210 | else: 211 | logging.error("[failed] Error: file: \'{}\', status:{}, reason:{}, url:{}".format(lookup_file_name,r.status_code,r.reason,r.url)) 212 | except Exception as e: 213 | logging.error("[failed] Error: Sending updated file {} to Splunk server, reason: {}".format(lookup_file_name,e)) 214 | else: 215 | logging.error("[failed] Error: Downloading file: \'{}\', status:{}, reason:{}, url:{}".format(lookup_file_name,r.status_code,r.reason,r.url)) 216 | -------------------------------------------------------------------------------- /upload_lookups_to_splunk.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # coding: utf-8 3 | 4 | # Simple python script using Splunk lookup-editor https://splunkbase.splunk.com/app/1724 rest endpoint to upload lookups (wiki https://lukemurphey.net/projects/splunk-lookup-editor/wiki/REST_endpoints) 5 | # troubleshoot lookup-editor errors on splunk: index=_internal (sourcetype=lookup_editor_rest_handler OR sourcetype=lookup_backups_rest_handler) 6 | 7 | import json 8 | import requests 9 | import csv 10 | import re 11 | import logging 12 | import pathlib 13 | from requests.packages.urllib3.exceptions import InsecureRequestWarning 14 | 15 | # Disable warning for insecure requests 16 | requests.packages.urllib3.disable_warnings(InsecureRequestWarning) 17 | 18 | # Setup logging 19 | logging.basicConfig(level=logging.INFO) 20 | 21 | # Declare variables (change them) 22 | splunk_management_protocol = "https" # http or https 23 | splunk_management_dest = "FIXME" #example: myserver.local or 127.0.0.1 24 | splunk_management_port = "FIXME" #example default: 8089 25 | 26 | # GET requests to this endpoint will execute get_lookup_contents() and POST requests to this endpoint will execute post_lookup_contents() from the lookup_editor_rest_handler.py in the lookup-editor app 27 | splunk_management_service = "/services/data/lookup_edit/lookup_contents" #endpoint lookup-editor 28 | 29 | splunk_app = "search" #splunk app name, example: search 30 | splunk_username = "FIXME" #splunk user with needed permissions 31 | splunk_password = "FIXME" #splunk user password (i know..) 32 | lookup_folder = "FIXME" #folder path which contains all the lookups you want to upload example: "C:\\lookups\" or "/home/mthcht/lookups/" 33 | 34 | 35 | for lookup_file in pathlib.Path(lookup_folder).iterdir(): 36 | lookup_name = pathlib.Path(lookup_file).name 37 | lookup_name = str(lookup_name.replace(" ","_")) 38 | 39 | # Read data from CSV file 40 | lookup_content = [] 41 | try: 42 | with open(lookup_file, encoding='utf-8', errors='ignore',newline='') as f: 43 | reader = csv.reader(f, delimiter=',') 44 | for row in reader: 45 | lookup_content.append(row) 46 | except Exception as e: 47 | logging.error("Error reading {} : {}".format(lookup_file,e)) 48 | 49 | # Send POST request to Splunk server 50 | try: 51 | r = requests.post("{}://{}:{}{}".format(splunk_management_protocol,splunk_management_dest,splunk_management_port,splunk_management_service) , 52 | verify=False, 53 | auth=(splunk_username, splunk_password), 54 | data={"output_mode": "json", 55 | "namespace": splunk_app, 56 | "lookup_file": lookup_name, 57 | "contents": json.dumps(lookup_content)}, 58 | timeout=60 59 | ) 60 | if r.status_code == 200: 61 | logging.info("[success] file: \'{}\' uploaded to Lookup editor handler {} and saved in splunk app \'{}\'".format(lookup_name,r.url,splunk_app)) 62 | else: 63 | logging.error("[failed] file: \'{}\', status:{}, reason:{}, url:{}".format(lookup_name,r.status_code,r.reason,r.url)) 64 | except Exception as e: 65 | logging.error("Error sending request to Splunk server: {}".format(e)) 66 | --------------------------------------------------------------------------------