├── README.md └── livetail.py /README.md: -------------------------------------------------------------------------------- 1 | # Sumo Logic Live Tail Command Line Interface (CLI) 2 | 3 | [![Join the chat at https://gitter.im/SumoLogic/livetail-cli](https://badges.gitter.im/SumoLogic/livetail-cli.svg)](https://gitter.im/SumoLogic/livetail-cli?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) 4 | 5 | The Live Tail Command Line Interface (CLI) is a standalone application that allows you to start and use a Live Tail session from the command line, similar to `tail -f`. 6 | The output is directed to stdout - so you can pipe the output to commands (grep, awk etc) 7 | 8 | This feature is documented [here.](http://help.sumologic.com/Search/Live_Tail/Live_Tail_CLI) 9 | 10 | ## Installation 11 | 12 | [Platform specific binaries are provided for CLI.](https://github.com/SumoLogic/livetail-cli/releases) 13 | 14 | Simply extract the archive and place the binaries to a location where you have read/write access. 15 | 16 | ## Source Code 17 | 18 | The source code of the Live Tail CLI is included in this repository. 19 | 20 | Prerequisites to use the script: 21 | - Python 3 22 | - the `requests` library https://pypi.org/project/requests/. 23 | 24 | ## Usage 25 | 26 | Like [SumoLogic](https://www.sumologic.com), the Live Tail CLI enables you to tail logs in real time by specifying a filter. 27 | It uses accessId and accessKeys that are used with the SumoLogic API for authentication. 28 | You could either provide the credentials each time using -i and -k command line options, or enter them once when prompted and they would be saved locally in config.json file in the same directory as the CLI. 29 | 30 | Using the provided Access Id and Access Key, the script will be able to automatically determine a deployment where your account exists. 31 | 32 | If you would like to use the Live Tail CLI with a Sumo internal deployment, please specify the deployment in the program arguments using the -d option as it cannot be determined automatically. 33 | 34 | ### Command Line Options 35 | 36 | ``` 37 | usage: livetail [-h] [-i ACCESSID] [-k ACCESSKEY] [-d DEPLOYMENT] [-v] [-c] [filter] 38 | 39 | positional arguments: 40 | filter Live Tail filter 41 | 42 | optional arguments: 43 | -h, --help show this help message and exit 44 | -i ACCESSID Access ID 45 | -k ACCESSKEY Acccess Key 46 | -d DEPLOYMENT Deployment-specific Sumo Logic API URL e.g. api.sumologic.com 47 | -v, --version show program's version number and exit 48 | -c clear Live Tail 49 | ``` 50 | 51 | ## Examples 52 | 53 | Tail all logs from a given Source Host. 54 | 55 | ```sh 56 | ./livetail “_sourceHost = localhost” 57 | ``` 58 | 59 | Tail logs from a sourceCategory, grep for a pattern, write it to a file 60 | 61 | ```sh 62 | ./livetail “_sourceCategory = service” | grep -i “rate limit exceeded” > out.txt 63 | ``` 64 | 65 | Happy Tailing! 66 | -------------------------------------------------------------------------------- /livetail.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # ----------------------------------------------------------- 4 | # Live Tail CLI 5 | # 6 | # This script allows you to start and use a Live Tail session from the command line. 7 | # 8 | # You need to provide your Access Id and Access Key when prompted and the script will be able to automatically determine 9 | # the deployment where your account exists. 10 | # 11 | # If you would like to use the Live Tail CLI with a Sumo internal staging deployment 12 | # please specify the deployment in the program arguments using the -d option 13 | # as it cannot be determined automatically. 14 | # ----------------------------------------------------------- 15 | 16 | import argparse 17 | import getpass 18 | import json 19 | import logging 20 | import requests 21 | import sys 22 | import time 23 | import os 24 | 25 | # Version 2.0 26 | MAJOR_VERSION = 2 27 | MINOR_VERSION = 0 28 | CONFIG_FILE = 'config.json' 29 | HEADERS = {'Content-Type': 'application/json', 30 | 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) ' 31 | 'Chrome/46.0.2490.80 Safari/537.36', 32 | 'Accept': 'application/json'} 33 | LIVE_TAIL_VERSION_ENDPOINT = 'livetail/session/version' 34 | DEFAULT_SUMOLOGIC_DEPLOYMENT = 'api.sumologic.com' 35 | 36 | LOGGER = logging.getLogger('Live Tail CLI') 37 | LOGGER.setLevel(logging.INFO) 38 | logger_handler = logging.StreamHandler() 39 | logger_handler.setLevel(logging.INFO) 40 | LOGGER.addHandler(logger_handler) 41 | 42 | 43 | def get_sumo_logic_api_url(deployment, endpoint): 44 | return f'https://{deployment}/api/v1/{endpoint}' 45 | 46 | 47 | def resolve_deployment(session, access_id, access_key): 48 | """ 49 | Automatically determines a deployment that a user has access to. 50 | 51 | If a specific deployment has not been provided as part of the program args 52 | it is resolved using a redirect method: 53 | 54 | a request is made to a default production API endpoint, 55 | based on a given Access Id, a user gets redirected to a deployment where their account exists 56 | and the deployment URL is retrieved from the response url. 57 | """ 58 | 59 | deployment = DEFAULT_SUMOLOGIC_DEPLOYMENT 60 | url = get_sumo_logic_api_url(deployment, LIVE_TAIL_VERSION_ENDPOINT) 61 | 62 | try: 63 | response = session.get(url, headers=HEADERS, auth=(access_id, access_key)) 64 | deployment = response.url.split('/api/v1/')[0].split('https://')[1] 65 | except Exception as e: 66 | LOGGER.error('### Unable to resolve deployment using the Access Id / Access Key ###', exc_info=e) 67 | sys.exit() 68 | 69 | return deployment 70 | 71 | 72 | def authenticate(session, access_id, access_key, deployment): 73 | """ 74 | Checks if a user is authorized to access the API on a given deployment. 75 | 76 | Authentication is considered successful if a user is able to successfully call into an API endpoint 77 | for a given deployment. 78 | """ 79 | 80 | LOGGER.info('### Authenticating ###') 81 | version_url = get_sumo_logic_api_url(deployment, LIVE_TAIL_VERSION_ENDPOINT) 82 | response = session.get(version_url, headers=HEADERS, auth=(access_id, access_key)) 83 | 84 | if response.status_code != requests.codes.ok: 85 | LOGGER.error('### Authentication failed. Please check the Access ID and Access Key and try again ###') 86 | sys.exit() 87 | 88 | LOGGER.info('### Authentication successful ###') 89 | 90 | check_for_version(response) 91 | 92 | 93 | def check_for_version(version_info): 94 | latest_version = str(version_info.json()['version']) 95 | latest_major_version = float(latest_version.split('.')[0]) 96 | latest_minor_version = float(latest_version.split('.')[1]) 97 | 98 | if latest_major_version > MAJOR_VERSION: 99 | LOGGER.error('### Incompatible version of CLI. Please download the latest version from \n' 100 | 'https://github.com/sumologic/livetail-cli ###') 101 | sys.exit() 102 | 103 | if latest_minor_version > MINOR_VERSION: 104 | LOGGER.warning('### A newer version of Live Tail CLI is available, ' 105 | 'but your current version will still function. \n' 106 | 'If you would like to download the latest version, ' 107 | 'go to https://github.com/sumologic/livetail-cli ###') 108 | 109 | 110 | def create_live_tail_session(session, access_id, access_key, deployment, live_tail_filter): 111 | live_tail_session_url = get_sumo_logic_api_url(deployment, 'livetail/session') 112 | live_tail_session_response = session.post(live_tail_session_url, 113 | json={'filter': live_tail_filter, 'isCLI': True}, 114 | headers=HEADERS, 115 | auth=(access_id, access_key)) 116 | 117 | if live_tail_session_response.status_code != requests.codes.ok: 118 | LOGGER.error('### Unable to create Live Tail session. Please try again ###') 119 | sys.exit() 120 | 121 | if live_tail_session_response.json()['error']: 122 | LOGGER.error('### Failed to create Live Tail session because ' + 123 | live_tail_session_response.json()['errorMessage'] + ' ###') 124 | sys.exit() 125 | 126 | tail_id = live_tail_session_response.json()["id"] 127 | 128 | return tail_id 129 | 130 | 131 | def start_live_tail_session(session, access_id, access_key, deployment, live_tail_filter): 132 | tail_id = None 133 | try: 134 | tail_id = create_live_tail_session(session, access_id, access_key, deployment, live_tail_filter) 135 | time.sleep(2) # Give some time for the engine to be ready 136 | 137 | LOGGER.info('### Starting Live Tail session ###') 138 | offset = 0 139 | 140 | while True: 141 | latest_live_tail_results_url = get_sumo_logic_api_url(deployment, 142 | f'livetail/session/{tail_id}/latest/{offset}') 143 | response = session.get(latest_live_tail_results_url, 144 | headers=HEADERS, 145 | auth=(access_id, access_key)) 146 | 147 | try: 148 | state = response.json()['state'] 149 | except ValueError: 150 | LOGGER.error('### API rate limit exceeded. Ending this Live Tail session ###') 151 | sys.exit() 152 | 153 | if state is not None: 154 | offset = state['currentOffset'] + 1 155 | messages = response.json()['messages'] 156 | user_messages = state['userMessages'] 157 | is_stopped = state['isStopped'] 158 | 159 | for usr_msg in user_messages: 160 | message_type = usr_msg['messageType'] 161 | if 'currentRate' in usr_msg: 162 | if message_type == 'Error': 163 | LOGGER.error('### Your query produced too many messages and caused the session to end. ' 164 | 'Please add additional metadata fields to your query to make it ' 165 | 'more specific. ###') 166 | sys.exit() 167 | elif message_type == 'Warning': 168 | LOGGER.warning('### Your query is producing too many messages and will cause the session ' 169 | 'to end. Please add additional metadata fields to your query to make it ' 170 | 'more specific. ###') 171 | 172 | if 'maxEngineRunningTime' in usr_msg: 173 | if message_type == 'Error': 174 | LOGGER.error('### Your Live Tail session has timed out. ###') 175 | sys.exit() 176 | 177 | if is_stopped: 178 | LOGGER.error('### Your Live Tail session has timed out. ###') 179 | sys.exit() 180 | 181 | for msg in messages: 182 | print(msg['payload']) 183 | 184 | time.sleep(1) 185 | 186 | except Exception as e: 187 | LOGGER.error("### Fatal error has occurred. The Live Tail session will end. ###", exc_info=e) 188 | 189 | finally: 190 | if tail_id is not None: 191 | LOGGER.info('### Ending the Live Tail session ###') 192 | delete_url = get_sumo_logic_api_url(deployment, f'livetail/session/{tail_id}') 193 | session.delete(delete_url, headers=HEADERS, auth=(access_id, access_key)) 194 | sys.exit() 195 | 196 | 197 | def parse_program_args(): 198 | parser = argparse.ArgumentParser() 199 | 200 | parser.add_argument('filter', type=str, help='Live Tail filter', default=None, nargs='?') 201 | parser.add_argument('-i', dest='accessId', type=str, help='Access ID', required=False) 202 | parser.add_argument('-k', dest='accessKey', type=str, help='Access Key', required=False) 203 | parser.add_argument('-d', dest='deployment', type=str, 204 | help='Deployment-specific Sumo Logic API URL e.g. api.sumologic.com', required=False) 205 | 206 | parser.add_argument('-v', '--version', action='version', 207 | version=f'Sumo Logic Live Tail CLI Version ({MAJOR_VERSION}.{MINOR_VERSION})') 208 | 209 | parser.add_argument('-c', dest='clear', action='store_true', help='clear Live Tail') 210 | 211 | args = parser.parse_args() 212 | 213 | if args.accessId is None and args.accessKey is not None: 214 | LOGGER.error('### Please provide the Access ID with the -i argument ###') 215 | sys.exit() 216 | 217 | if args.accessId is not None and args.accessKey is None: 218 | LOGGER.error('### Please provide the Access Key with the -k argument ###') 219 | sys.exit() 220 | 221 | # delete config file on clear 222 | if args.clear: 223 | LOGGER.info('### Clearing Live Tail CLI session ###') 224 | try: 225 | os.remove(CONFIG_FILE) 226 | except OSError: 227 | pass 228 | sys.exit() 229 | 230 | return args 231 | 232 | 233 | def get_access_details(program_args): 234 | if not os.path.exists(CONFIG_FILE): 235 | config_data = {'deployment': '', 'accessId': '', 'accessKey': ''} 236 | else: 237 | with open(CONFIG_FILE) as json_data: 238 | config_data = json.load(json_data) 239 | 240 | if program_args.accessId is None and program_args.accessKey is None: # Args not specified - read from file 241 | access_id = config_data['accessId'] 242 | access_key = config_data['accessKey'] 243 | else: 244 | access_id = program_args.accessId 245 | access_key = program_args.accessKey 246 | 247 | has_asked_for_access_id = False 248 | if access_id is None or access_id == "": 249 | access_id = input('Please enter your Access ID: ') 250 | access_key = getpass.getpass('Please enter your Access Key') # Asks for password without echoing 251 | has_asked_for_access_id = True 252 | 253 | return access_id, access_key, has_asked_for_access_id 254 | 255 | 256 | def launch_live_tail(): 257 | LOGGER.info('### Welcome to Sumo Logic Live Tail Command Line Interface ###') 258 | 259 | args = parse_program_args() 260 | access_id, access_key, has_asked_for_access_id = get_access_details(args) 261 | session = requests.Session() 262 | 263 | # a deployment is determined automatically if it has not been provided in the command line 264 | if args.deployment is None or args.deployment == '': 265 | deployment = resolve_deployment(session, access_id, access_key) 266 | else: 267 | deployment = args.deployment 268 | 269 | # verify whether a user has access to a given deployment 270 | authenticate(session, access_id, access_key, deployment) 271 | 272 | # if a user has been explicitly asked for login details, they are saved in the CONFIG_FILE 273 | if has_asked_for_access_id: 274 | with open(CONFIG_FILE, 'w') as outfile: 275 | json.dump({'deployment': deployment, 'accessId': access_id, 'accessKey': access_key}, outfile) 276 | 277 | start_live_tail_session(session, access_id, access_key, deployment, args.filter) 278 | 279 | 280 | if __name__ == '__main__': 281 | launch_live_tail() --------------------------------------------------------------------------------