├── .gitignore ├── LICENSE.txt ├── README.rst ├── ngxtop ├── __init__.py ├── config_parser.py ├── ngxtop.py └── utils.py ├── setup.cfg ├── setup.py └── tests └── test_config_parser.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[cod] 2 | .idea 3 | .vagrant 4 | Vagrantfile 5 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Binh Le. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | ================================================================ 2 | ``ngxtop`` - **real-time** metrics for nginx server (and others) 3 | ================================================================ 4 | 5 | **ngxtop** parses your nginx access log and outputs useful, ``top``-like, metrics of your nginx server. 6 | So you can tell what is happening with your server in real-time. 7 | 8 | ``ngxtop`` is designed to run in a short-period time just like the ``top`` command for troubleshooting and monitoring 9 | your Nginx server at the moment. If you need a long running monitoring process or storing your webserver stats in external 10 | monitoring / graphing system, you can try `Luameter `_. 11 | 12 | ``ngxtop`` tries to determine the correct location and format of nginx access log file by default, so you can just run 13 | ``ngxtop`` and having a close look at all requests coming to your nginx server. But it does not limit you to nginx 14 | and the default top view. ``ngxtop`` is flexible enough for you to configure and change most of its behaviours. 15 | You can query for different things, specify your log and format, even parse remote Apache common access log with ease. 16 | See sample usages below for some ideas about what you can do with it. 17 | 18 | Installation 19 | ------------ 20 | 21 | :: 22 | 23 | pip install ngxtop 24 | 25 | 26 | Note: ``ngxtop`` is primarily developed and tested with python2 but also supports python3. 27 | 28 | Usage 29 | ----- 30 | 31 | :: 32 | 33 | Usage: 34 | ngxtop [options] 35 | ngxtop [options] (print|top|avg|sum) 36 | ngxtop info 37 | 38 | Options: 39 | -l , --access-log access log file to parse. 40 | -f , --log-format log format as specify in log_format directive. 41 | --no-follow ngxtop default behavior is to ignore current lines in log 42 | and only watch for new lines as they are written to the access log. 43 | Use this flag to tell ngxtop to process the current content of the access log instead. 44 | -t , --interval report interval when running in follow mode [default: 2.0] 45 | 46 | -g , --group-by group by variable [default: request_path] 47 | -w , --having having clause [default: 1] 48 | -o , --order-by order of output for default query [default: count] 49 | -n , --limit limit the number of records included in report for top command [default: 10] 50 | -a ..., --a ... add exp (must be aggregation exp: sum, avg, min, max, etc.) into output 51 | 52 | -v, --verbose more verbose output 53 | -d, --debug print every line and parsed record 54 | -h, --help print this help message. 55 | --version print version information. 56 | 57 | Advanced / experimental options: 58 | -c , --config allow ngxtop to parse nginx config file for log format and location. 59 | -i , --filter filter in, records satisfied given expression are processed. 60 | -p , --pre-filter in-filter expression to check in pre-parsing phase. 61 | 62 | Samples 63 | ------- 64 | 65 | Default output 66 | ~~~~~~~~~~~~~~ 67 | 68 | :: 69 | 70 | $ ngxtop 71 | running for 411 seconds, 64332 records processed: 156.60 req/sec 72 | 73 | Summary: 74 | | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx | 75 | |---------+------------------+-------+-------+-------+-------| 76 | | 64332 | 2775.251 | 61262 | 2994 | 71 | 5 | 77 | 78 | Detailed: 79 | | request_path | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx | 80 | |------------------------------------------+---------+------------------+-------+-------+-------+-------| 81 | | /abc/xyz/xxxx | 20946 | 434.693 | 20935 | 0 | 11 | 0 | 82 | | /xxxxx.json | 5633 | 1483.723 | 5633 | 0 | 0 | 0 | 83 | | /xxxxx/xxx/xxxxxxxxxxxxx | 3629 | 6835.499 | 3626 | 0 | 3 | 0 | 84 | | /xxxxx/xxx/xxxxxxxx | 3627 | 15971.885 | 3623 | 0 | 4 | 0 | 85 | | /xxxxx/xxx/xxxxxxx | 3624 | 7830.236 | 3621 | 0 | 3 | 0 | 86 | | /static/js/minified/utils.min.js | 3031 | 1781.155 | 2104 | 927 | 0 | 0 | 87 | | /static/js/minified/xxxxxxx.min.v1.js | 2889 | 2210.235 | 2068 | 821 | 0 | 0 | 88 | | /static/tracking/js/xxxxxxxx.js | 2594 | 1325.681 | 1927 | 667 | 0 | 0 | 89 | | /xxxxx/xxx.html | 2521 | 573.597 | 2520 | 0 | 1 | 0 | 90 | | /xxxxx/xxxx.json | 1840 | 800.542 | 1839 | 0 | 1 | 0 | 91 | 92 | View top source IPs of clients 93 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 94 | 95 | :: 96 | 97 | $ ngxtop top remote_addr 98 | running for 20 seconds, 3215 records processed: 159.62 req/sec 99 | 100 | top remote_addr 101 | | remote_addr | count | 102 | |-----------------+---------| 103 | | 118.173.177.161 | 20 | 104 | | 110.78.145.3 | 16 | 105 | | 171.7.153.7 | 16 | 106 | | 180.183.67.155 | 16 | 107 | | 183.89.65.9 | 16 | 108 | | 202.28.182.5 | 16 | 109 | | 1.47.170.12 | 15 | 110 | | 119.46.184.2 | 15 | 111 | | 125.26.135.219 | 15 | 112 | | 125.26.213.203 | 15 | 113 | 114 | List 4xx or 5xx responses together with HTTP referer 115 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 116 | 117 | :: 118 | 119 | $ ngxtop -i 'status >= 400' print request status http_referer 120 | running for 2 seconds, 28 records processed: 13.95 req/sec 121 | 122 | request, status, http_referer: 123 | | request | status | http_referer | 124 | |-----------+----------+----------------| 125 | | - | 400 | - | 126 | 127 | Parse apache log from remote server with `common` format 128 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 129 | 130 | :: 131 | 132 | $ ssh user@remote_server tail -f /var/log/apache2/access.log | ngxtop -f common 133 | running for 20 seconds, 1068 records processed: 53.01 req/sec 134 | 135 | Summary: 136 | | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx | 137 | |---------+------------------+-------+-------+-------+-------| 138 | | 1068 | 28026.763 | 1029 | 20 | 19 | 0 | 139 | 140 | Detailed: 141 | | request_path | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx | 142 | |------------------------------------------+---------+------------------+-------+-------+-------+-------| 143 | | /xxxxxxxxxx | 199 | 55150.402 | 199 | 0 | 0 | 0 | 144 | | /xxxxxxxx/xxxxx | 167 | 47591.826 | 167 | 0 | 0 | 0 | 145 | | /xxxxxxxxxxxxx/xxxxxx | 25 | 7432.200 | 25 | 0 | 0 | 0 | 146 | | /xxxx/xxxxx/x/xxxxxxxxxxxxx/xxxxxxx | 22 | 698.727 | 22 | 0 | 0 | 0 | 147 | | /xxxx/xxxxx/x/xxxxxxxxxxxxx/xxxxxx | 19 | 7431.632 | 19 | 0 | 0 | 0 | 148 | | /xxxxx/xxxxx/ | 18 | 7840.889 | 18 | 0 | 0 | 0 | 149 | | /xxxxxxxx/xxxxxxxxxxxxxxxxx | 15 | 7356.000 | 15 | 0 | 0 | 0 | 150 | | /xxxxxxxxxxx/xxxxxxxx | 15 | 9978.800 | 15 | 0 | 0 | 0 | 151 | | /xxxxx/ | 14 | 0.000 | 0 | 14 | 0 | 0 | 152 | | /xxxxxxxxxx/xxxxxxxx/xxxxx | 13 | 20530.154 | 13 | 0 | 0 | 0 | 153 | 154 | -------------------------------------------------------------------------------- /ngxtop/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lebinh/ngxtop/35b3f1e40e87c221b7156300b3611518c1d37745/ngxtop/__init__.py -------------------------------------------------------------------------------- /ngxtop/config_parser.py: -------------------------------------------------------------------------------- 1 | """ 2 | Nginx config parser and pattern builder. 3 | """ 4 | import os 5 | import re 6 | import subprocess 7 | 8 | from pyparsing import Literal, Word, ZeroOrMore, OneOrMore, Group, \ 9 | printables, quotedString, pythonStyleComment, removeQuotes 10 | 11 | from .utils import choose_one, error_exit 12 | 13 | 14 | REGEX_SPECIAL_CHARS = r'([\.\*\+\?\|\(\)\{\}\[\]])' 15 | REGEX_LOG_FORMAT_VARIABLE = r'\$([a-zA-Z0-9\_]+)' 16 | LOG_FORMAT_COMBINED = '$remote_addr - $remote_user [$time_local] ' \ 17 | '"$request" $status $body_bytes_sent ' \ 18 | '"$http_referer" "$http_user_agent"' 19 | LOG_FORMAT_COMMON = '$remote_addr - $remote_user [$time_local] ' \ 20 | '"$request" $status $body_bytes_sent ' \ 21 | '"$http_x_forwarded_for"' 22 | 23 | # common parser element 24 | semicolon = Literal(';').suppress() 25 | # nginx string parameter can contain any character except: { ; " ' 26 | parameter = Word(''.join(c for c in printables if c not in set('{;"\''))) 27 | # which can also be quoted 28 | parameter = parameter | quotedString.setParseAction(removeQuotes) 29 | 30 | 31 | def detect_config_path(): 32 | """ 33 | Get nginx configuration file path based on `nginx -V` output 34 | :return: detected nginx configuration file path 35 | """ 36 | try: 37 | proc = subprocess.Popen(['nginx', '-V'], stderr=subprocess.PIPE) 38 | except OSError: 39 | error_exit('Access log file or format was not set and nginx config file cannot be detected. ' + 40 | 'Perhaps nginx is not in your PATH?') 41 | 42 | stdout, stderr = proc.communicate() 43 | version_output = stderr.decode('utf-8') 44 | conf_path_match = re.search(r'--conf-path=(\S*)', version_output) 45 | if conf_path_match is not None: 46 | return conf_path_match.group(1) 47 | 48 | prefix_match = re.search(r'--prefix=(\S*)', version_output) 49 | if prefix_match is not None: 50 | return prefix_match.group(1) + '/conf/nginx.conf' 51 | return '/etc/nginx/nginx.conf' 52 | 53 | 54 | def get_access_logs(config): 55 | """ 56 | Parse config for access_log directives 57 | :return: iterator over ('path', 'format name') tuple of found directives 58 | """ 59 | access_log = Literal("access_log") + ZeroOrMore(parameter) + semicolon 60 | access_log.ignore(pythonStyleComment) 61 | 62 | for directive in access_log.searchString(config).asList(): 63 | path = directive[1] 64 | if path == 'off' or path.startswith('syslog:'): 65 | # nothing to process here 66 | continue 67 | 68 | format_name = 'combined' 69 | if len(directive) > 2 and '=' not in directive[2]: 70 | format_name = directive[2] 71 | 72 | yield path, format_name 73 | 74 | 75 | def get_log_formats(config): 76 | """ 77 | Parse config for log_format directives 78 | :return: iterator over ('format name', 'format string') tuple of found directives 79 | """ 80 | # log_format name [params] 81 | log_format = Literal('log_format') + parameter + Group(OneOrMore(parameter)) + semicolon 82 | log_format.ignore(pythonStyleComment) 83 | 84 | for directive in log_format.searchString(config).asList(): 85 | name = directive[1] 86 | format_string = ''.join(directive[2]) 87 | yield name, format_string 88 | 89 | 90 | def detect_log_config(arguments): 91 | """ 92 | Detect access log config (path and format) of nginx. Offer user to select if multiple access logs are detected. 93 | :return: path and format of detected / selected access log 94 | """ 95 | config = arguments['--config'] 96 | if config is None: 97 | config = detect_config_path() 98 | if not os.path.exists(config): 99 | error_exit('Nginx config file not found: %s' % config) 100 | 101 | with open(config) as f: 102 | config_str = f.read() 103 | access_logs = dict(get_access_logs(config_str)) 104 | if not access_logs: 105 | error_exit('Access log file is not provided and ngxtop cannot detect it from your config file (%s).' % config) 106 | 107 | log_formats = dict(get_log_formats(config_str)) 108 | if len(access_logs) == 1: 109 | log_path, format_name = list(access_logs.items())[0] 110 | if format_name == 'combined': 111 | return log_path, LOG_FORMAT_COMBINED 112 | if format_name not in log_formats: 113 | error_exit('Incorrect format name set in config for access log file "%s"' % log_path) 114 | return log_path, log_formats[format_name] 115 | 116 | # multiple access logs configured, offer to select one 117 | print('Multiple access logs detected in configuration:') 118 | log_path = choose_one(list(access_logs.keys()), 'Select access log file to process: ') 119 | format_name = access_logs[log_path] 120 | if format_name not in log_formats: 121 | error_exit('Incorrect format name set in config for access log file "%s"' % log_path) 122 | return log_path, log_formats[format_name] 123 | 124 | 125 | def build_pattern(log_format): 126 | """ 127 | Build regular expression to parse given format. 128 | :param log_format: format string to parse 129 | :return: regular expression to parse given format 130 | """ 131 | if log_format == 'combined': 132 | log_format = LOG_FORMAT_COMBINED 133 | elif log_format == 'common': 134 | log_format = LOG_FORMAT_COMMON 135 | pattern = re.sub(REGEX_SPECIAL_CHARS, r'\\\1', log_format) 136 | pattern = re.sub(REGEX_LOG_FORMAT_VARIABLE, '(?P<\\1>.*)', pattern) 137 | return re.compile(pattern) 138 | 139 | 140 | def extract_variables(log_format): 141 | """ 142 | Extract all variables from a log format string. 143 | :param log_format: format string to extract 144 | :return: iterator over all variables in given format string 145 | """ 146 | if log_format == 'combined': 147 | log_format = LOG_FORMAT_COMBINED 148 | for match in re.findall(REGEX_LOG_FORMAT_VARIABLE, log_format): 149 | yield match 150 | 151 | -------------------------------------------------------------------------------- /ngxtop/ngxtop.py: -------------------------------------------------------------------------------- 1 | """ngxtop - ad-hoc query for nginx access log. 2 | 3 | Usage: 4 | ngxtop [options] 5 | ngxtop [options] (print|top|avg|sum) ... 6 | ngxtop info 7 | ngxtop [options] query ... 8 | 9 | Options: 10 | -l , --access-log access log file to parse. 11 | -f , --log-format log format as specify in log_format directive. [default: combined] 12 | --no-follow ngxtop default behavior is to ignore current lines in log 13 | and only watch for new lines as they are written to the access log. 14 | Use this flag to tell ngxtop to process the current content of the access log instead. 15 | -t , --interval report interval when running in follow mode [default: 2.0] 16 | 17 | -g , --group-by group by variable [default: request_path] 18 | -w , --having having clause [default: 1] 19 | -o , --order-by order of output for default query [default: count] 20 | -n , --limit limit the number of records included in report for top command [default: 10] 21 | -a ..., --a ... add exp (must be aggregation exp: sum, avg, min, max, etc.) into output 22 | 23 | -v, --verbose more verbose output 24 | -d, --debug print every line and parsed record 25 | -h, --help print this help message. 26 | --version print version information. 27 | 28 | Advanced / experimental options: 29 | -c , --config allow ngxtop to parse nginx config file for log format and location. 30 | -i , --filter filter in, records satisfied given expression are processed. 31 | -p , --pre-filter in-filter expression to check in pre-parsing phase. 32 | 33 | Examples: 34 | All examples read nginx config file for access log location and format. 35 | If you want to specify the access log file and / or log format, use the -f and -a options. 36 | 37 | "top" like view of nginx requests 38 | $ ngxtop 39 | 40 | Top 10 requested path with status 404: 41 | $ ngxtop top request_path --filter 'status == 404' 42 | 43 | Top 10 requests with highest total bytes sent 44 | $ ngxtop --order-by 'avg(bytes_sent) * count' 45 | 46 | Top 10 remote address, e.g., who's hitting you the most 47 | $ ngxtop --group-by remote_addr 48 | 49 | Print requests with 4xx or 5xx status, together with status and http referer 50 | $ ngxtop -i 'status >= 400' print request status http_referer 51 | 52 | Average body bytes sent of 200 responses of requested path begin with 'foo': 53 | $ ngxtop avg bytes_sent --filter 'status == 200 and request_path.startswith("foo")' 54 | 55 | Analyze apache access log from remote machine using 'common' log format 56 | $ ssh remote tail -f /var/log/apache2/access.log | ngxtop -f common 57 | """ 58 | from __future__ import print_function 59 | import atexit 60 | from contextlib import closing 61 | import curses 62 | import logging 63 | import os 64 | import sqlite3 65 | import time 66 | import sys 67 | import signal 68 | 69 | try: 70 | import urlparse 71 | except ImportError: 72 | import urllib.parse as urlparse 73 | 74 | from docopt import docopt 75 | import tabulate 76 | 77 | from .config_parser import detect_log_config, detect_config_path, extract_variables, build_pattern 78 | from .utils import error_exit 79 | 80 | 81 | DEFAULT_QUERIES = [ 82 | ('Summary:', 83 | '''SELECT 84 | count(1) AS count, 85 | avg(bytes_sent) AS avg_bytes_sent, 86 | count(CASE WHEN status_type = 2 THEN 1 END) AS '2xx', 87 | count(CASE WHEN status_type = 3 THEN 1 END) AS '3xx', 88 | count(CASE WHEN status_type = 4 THEN 1 END) AS '4xx', 89 | count(CASE WHEN status_type = 5 THEN 1 END) AS '5xx' 90 | FROM log 91 | ORDER BY %(--order-by)s DESC 92 | LIMIT %(--limit)s'''), 93 | 94 | ('Detailed:', 95 | '''SELECT 96 | %(--group-by)s, 97 | count(1) AS count, 98 | avg(bytes_sent) AS avg_bytes_sent, 99 | count(CASE WHEN status_type = 2 THEN 1 END) AS '2xx', 100 | count(CASE WHEN status_type = 3 THEN 1 END) AS '3xx', 101 | count(CASE WHEN status_type = 4 THEN 1 END) AS '4xx', 102 | count(CASE WHEN status_type = 5 THEN 1 END) AS '5xx' 103 | FROM log 104 | GROUP BY %(--group-by)s 105 | HAVING %(--having)s 106 | ORDER BY %(--order-by)s DESC 107 | LIMIT %(--limit)s''') 108 | ] 109 | 110 | DEFAULT_FIELDS = set(['status_type', 'bytes_sent']) 111 | 112 | 113 | # ====================== 114 | # generator utilities 115 | # ====================== 116 | def follow(the_file): 117 | """ 118 | Follow a given file and yield new lines when they are available, like `tail -f`. 119 | """ 120 | with open(the_file) as f: 121 | f.seek(0, 2) # seek to eof 122 | while True: 123 | line = f.readline() 124 | if not line: 125 | time.sleep(0.1) # sleep briefly before trying again 126 | continue 127 | yield line 128 | 129 | 130 | def map_field(field, func, dict_sequence): 131 | """ 132 | Apply given function to value of given key in every dictionary in sequence and 133 | set the result as new value for that key. 134 | """ 135 | for item in dict_sequence: 136 | try: 137 | item[field] = func(item.get(field, None)) 138 | yield item 139 | except ValueError: 140 | pass 141 | 142 | 143 | def add_field(field, func, dict_sequence): 144 | """ 145 | Apply given function to the record and store result in given field of current record. 146 | Do nothing if record already contains given field. 147 | """ 148 | for item in dict_sequence: 149 | if field not in item: 150 | item[field] = func(item) 151 | yield item 152 | 153 | 154 | def trace(sequence, phase=''): 155 | for item in sequence: 156 | logging.debug('%s:\n%s', phase, item) 157 | yield item 158 | 159 | 160 | # ====================== 161 | # Access log parsing 162 | # ====================== 163 | def parse_request_path(record): 164 | if 'request_uri' in record: 165 | uri = record['request_uri'] 166 | elif 'request' in record: 167 | uri = ' '.join(record['request'].split(' ')[1:-1]) 168 | else: 169 | uri = None 170 | return urlparse.urlparse(uri).path if uri else None 171 | 172 | 173 | def parse_status_type(record): 174 | return record['status'] // 100 if 'status' in record else None 175 | 176 | 177 | def to_int(value): 178 | return int(value) if value and value != '-' else 0 179 | 180 | 181 | def to_float(value): 182 | return float(value) if value and value != '-' else 0.0 183 | 184 | 185 | def parse_log(lines, pattern): 186 | matches = (pattern.match(l) for l in lines) 187 | records = (m.groupdict() for m in matches if m is not None) 188 | records = map_field('status', to_int, records) 189 | records = add_field('status_type', parse_status_type, records) 190 | records = add_field('bytes_sent', lambda r: r['body_bytes_sent'], records) 191 | records = map_field('bytes_sent', to_int, records) 192 | records = map_field('request_time', to_float, records) 193 | records = add_field('request_path', parse_request_path, records) 194 | return records 195 | 196 | 197 | # ================================= 198 | # Records and statistic processor 199 | # ================================= 200 | class SQLProcessor(object): 201 | def __init__(self, report_queries, fields, index_fields=None): 202 | self.begin = False 203 | self.report_queries = report_queries 204 | self.index_fields = index_fields if index_fields is not None else [] 205 | self.column_list = ','.join(fields) 206 | self.holder_list = ','.join(':%s' % var for var in fields) 207 | self.conn = sqlite3.connect(':memory:') 208 | self.init_db() 209 | 210 | def process(self, records): 211 | self.begin = time.time() 212 | insert = 'insert into log (%s) values (%s)' % (self.column_list, self.holder_list) 213 | logging.info('sqlite insert: %s', insert) 214 | with closing(self.conn.cursor()) as cursor: 215 | for r in records: 216 | cursor.execute(insert, r) 217 | 218 | def report(self): 219 | if not self.begin: 220 | return '' 221 | count = self.count() 222 | duration = time.time() - self.begin 223 | status = 'running for %.0f seconds, %d records processed: %.2f req/sec' 224 | output = [status % (duration, count, count / duration)] 225 | with closing(self.conn.cursor()) as cursor: 226 | for query in self.report_queries: 227 | if isinstance(query, tuple): 228 | label, query = query 229 | else: 230 | label = '' 231 | cursor.execute(query) 232 | columns = (d[0] for d in cursor.description) 233 | result = tabulate.tabulate(cursor.fetchall(), headers=columns, tablefmt='orgtbl', floatfmt='.3f') 234 | output.append('%s\n%s' % (label, result)) 235 | return '\n\n'.join(output) 236 | 237 | def init_db(self): 238 | create_table = 'create table log (%s)' % self.column_list 239 | with closing(self.conn.cursor()) as cursor: 240 | logging.info('sqlite init: %s', create_table) 241 | cursor.execute(create_table) 242 | for idx, field in enumerate(self.index_fields): 243 | sql = 'create index log_idx%d on log (%s)' % (idx, field) 244 | logging.info('sqlite init: %s', sql) 245 | cursor.execute(sql) 246 | 247 | def count(self): 248 | with closing(self.conn.cursor()) as cursor: 249 | cursor.execute('select count(1) from log') 250 | return cursor.fetchone()[0] 251 | 252 | 253 | # =============== 254 | # Log processing 255 | # =============== 256 | def process_log(lines, pattern, processor, arguments): 257 | pre_filer_exp = arguments['--pre-filter'] 258 | if pre_filer_exp: 259 | lines = (line for line in lines if eval(pre_filer_exp, {}, dict(line=line))) 260 | 261 | records = parse_log(lines, pattern) 262 | 263 | filter_exp = arguments['--filter'] 264 | if filter_exp: 265 | records = (r for r in records if eval(filter_exp, {}, r)) 266 | 267 | processor.process(records) 268 | print(processor.report()) # this will only run when start in --no-follow mode 269 | 270 | 271 | def build_processor(arguments): 272 | fields = arguments[''] 273 | if arguments['print']: 274 | label = ', '.join(fields) + ':' 275 | selections = ', '.join(fields) 276 | query = 'select %s from log group by %s' % (selections, selections) 277 | report_queries = [(label, query)] 278 | elif arguments['top']: 279 | limit = int(arguments['--limit']) 280 | report_queries = [] 281 | for var in fields: 282 | label = 'top %s' % var 283 | query = 'select %s, count(1) as count from log group by %s order by count desc limit %d' % (var, var, limit) 284 | report_queries.append((label, query)) 285 | elif arguments['avg']: 286 | label = 'average %s' % fields 287 | selections = ', '.join('avg(%s)' % var for var in fields) 288 | query = 'select %s from log' % selections 289 | report_queries = [(label, query)] 290 | elif arguments['sum']: 291 | label = 'sum %s' % fields 292 | selections = ', '.join('sum(%s)' % var for var in fields) 293 | query = 'select %s from log' % selections 294 | report_queries = [(label, query)] 295 | elif arguments['query']: 296 | report_queries = arguments[''] 297 | fields = arguments[''] 298 | else: 299 | report_queries = [(name, query % arguments) for name, query in DEFAULT_QUERIES] 300 | fields = DEFAULT_FIELDS.union(set([arguments['--group-by']])) 301 | 302 | for label, query in report_queries: 303 | logging.info('query for "%s":\n %s', label, query) 304 | 305 | processor_fields = [] 306 | for field in fields: 307 | processor_fields.extend(field.split(',')) 308 | 309 | processor = SQLProcessor(report_queries, processor_fields) 310 | return processor 311 | 312 | 313 | def build_source(access_log, arguments): 314 | # constructing log source 315 | if access_log == 'stdin': 316 | lines = sys.stdin 317 | elif arguments['--no-follow']: 318 | lines = open(access_log) 319 | else: 320 | lines = follow(access_log) 321 | return lines 322 | 323 | 324 | def setup_reporter(processor, arguments): 325 | if arguments['--no-follow']: 326 | return 327 | 328 | scr = curses.initscr() 329 | atexit.register(curses.endwin) 330 | 331 | def print_report(sig, frame): 332 | output = processor.report() 333 | scr.erase() 334 | try: 335 | scr.addstr(output) 336 | except curses.error: 337 | pass 338 | scr.refresh() 339 | 340 | signal.signal(signal.SIGALRM, print_report) 341 | interval = float(arguments['--interval']) 342 | signal.setitimer(signal.ITIMER_REAL, 0.1, interval) 343 | 344 | 345 | def process(arguments): 346 | access_log = arguments['--access-log'] 347 | log_format = arguments['--log-format'] 348 | if access_log is None and not sys.stdin.isatty(): 349 | # assume logs can be fetched directly from stdin when piped 350 | access_log = 'stdin' 351 | if access_log is None: 352 | access_log, log_format = detect_log_config(arguments) 353 | 354 | logging.info('access_log: %s', access_log) 355 | logging.info('log_format: %s', log_format) 356 | if access_log != 'stdin' and not os.path.exists(access_log): 357 | error_exit('access log file "%s" does not exist' % access_log) 358 | 359 | if arguments['info']: 360 | print('nginx configuration file:\n ', detect_config_path()) 361 | print('access log file:\n ', access_log) 362 | print('access log format:\n ', log_format) 363 | print('available variables:\n ', ', '.join(sorted(extract_variables(log_format)))) 364 | return 365 | 366 | source = build_source(access_log, arguments) 367 | pattern = build_pattern(log_format) 368 | processor = build_processor(arguments) 369 | setup_reporter(processor, arguments) 370 | process_log(source, pattern, processor, arguments) 371 | 372 | 373 | def main(): 374 | args = docopt(__doc__, version='xstat 0.1') 375 | 376 | log_level = logging.WARNING 377 | if args['--verbose']: 378 | log_level = logging.INFO 379 | if args['--debug']: 380 | log_level = logging.DEBUG 381 | logging.basicConfig(level=log_level, format='%(levelname)s: %(message)s') 382 | logging.debug('arguments:\n%s', args) 383 | 384 | try: 385 | process(args) 386 | except KeyboardInterrupt: 387 | sys.exit(0) 388 | 389 | 390 | if __name__ == '__main__': 391 | main() 392 | -------------------------------------------------------------------------------- /ngxtop/utils.py: -------------------------------------------------------------------------------- 1 | import sys 2 | 3 | 4 | def choose_one(choices, prompt): 5 | for idx, choice in enumerate(choices): 6 | print('%d. %s' % (idx + 1, choice)) 7 | selected = None 8 | if sys.version[0] == '3': 9 | raw_input = input 10 | while not selected or selected <= 0 or selected > len(choices): 11 | selected = raw_input(prompt) 12 | try: 13 | selected = int(selected) 14 | except ValueError: 15 | selected = None 16 | return choices[selected - 1] 17 | 18 | 19 | def error_exit(msg, status=1): 20 | sys.stderr.write('Error: %s\n' % msg) 21 | sys.exit(status) 22 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [wheel] 2 | universal = 1 -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | setup( 4 | name='ngxtop', 5 | version='0.0.3', 6 | description='Real-time metrics for nginx server', 7 | long_description=open('README.rst').read(), 8 | license='MIT', 9 | 10 | url='https://github.com/lebinh/ngxtop', 11 | author='Binh Le', 12 | author_email='lebinh.it@gmail.com', 13 | 14 | classifiers=[ 15 | 'Development Status :: 4 - Beta', 16 | 'License :: OSI Approved :: MIT License', 17 | 'Environment :: Console', 18 | 'Intended Audience :: Developers', 19 | 'Intended Audience :: System Administrators', 20 | 'Programming Language :: Python :: 2', 21 | 'Programming Language :: Python :: 2.6', 22 | 'Programming Language :: Python :: 2.7', 23 | 'Programming Language :: Python :: 3', 24 | 'Programming Language :: Python :: 3.2', 25 | 'Programming Language :: Python :: 3.3', 26 | ], 27 | keywords='cli monitoring nginx system', 28 | 29 | packages=['ngxtop'], 30 | install_requires=['docopt', 'tabulate', 'pyparsing'], 31 | 32 | entry_points={ 33 | 'console_scripts': [ 34 | 'ngxtop = ngxtop.ngxtop:main', 35 | ], 36 | }, 37 | ) 38 | -------------------------------------------------------------------------------- /tests/test_config_parser.py: -------------------------------------------------------------------------------- 1 | from ngxtop import config_parser 2 | 3 | 4 | def test_get_log_formats(): 5 | config = ''' 6 | http { 7 | # ubuntu default, log_format on multiple lines 8 | log_format main '$remote_addr - $remote_user [$time_local] "$request" ' 9 | "$status $body_bytes_sent '$http_referer' " 10 | '"$http_user_agent" "$http_x_forwarded_for"'; 11 | 12 | # name can also be quoted, and format don't always have to 13 | log_format 'te st' $remote_addr; 14 | } 15 | ''' 16 | formats = dict(config_parser.get_log_formats(config)) 17 | assert 'main' in formats 18 | assert "'$http_referer'" in formats['main'] 19 | assert 'te st' in formats 20 | 21 | 22 | def test_get_access_logs_no_format(): 23 | config = ''' 24 | http { 25 | # ubuntu default 26 | access_log /var/log/nginx/access.log; 27 | 28 | # syslog is a valid access log, but we can't follow it 29 | access_log syslog:server=address combined; 30 | 31 | # commented 32 | # access_log commented; 33 | 34 | server { 35 | location / { 36 | # has parameter with default format 37 | access_log /path/to/log gzip=1; 38 | } 39 | } 40 | } 41 | ''' 42 | logs = dict(config_parser.get_access_logs(config)) 43 | assert len(logs) == 2 44 | assert logs['/var/log/nginx/access.log'] == 'combined' 45 | assert logs['/path/to/log'] == 'combined' 46 | 47 | 48 | def test_access_logs_with_format_name(): 49 | config = ''' 50 | http { 51 | access_log /path/to/main.log main gzip=5 buffer=32k flush=1m; 52 | server { 53 | access_log /path/to/test.log 'te st'; 54 | } 55 | } 56 | ''' 57 | logs = dict(config_parser.get_access_logs(config)) 58 | assert len(logs) == 2 59 | assert logs['/path/to/main.log'] == 'main' 60 | assert logs['/path/to/test.log'] == 'te st' 61 | --------------------------------------------------------------------------------