├── LICENSE ├── README.md └── redis-tool /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 2-Clause License 2 | 3 | Copyright (c) 2022, hl 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 17 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 18 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 20 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 21 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 22 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 23 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 24 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 25 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # redis-tool 2 | Redis Cluster Daily Maintenance Tool/Redis集群日常运维工具 3 | 4 | ## 工具简介 5 | 6 | **redis-tool**基于原生的redis-cli客户端工具来进行Redis集群的监控、配置、问题分析等运维管理，能够极大降低Redis cluster集群的运维成本。同时作为脚本化工具，下载即可用，即使对于Redis初学者，也能够快速掌握集群的运行状况，完成集群配置管理、性能问题排查，具备Redis集群的基本运维能力。 7 | 8 | 工具主要面向日常运维管理中的常见工作，提升运维效率，简化操作复杂度： 9 | 10 | - **集群监控**：能够获取集群节点的键值数、内存使用量、CPU使用率、客户端连接数、请求响应时延等信息，以表格形式输出，并给出异常告警提示 11 | - **配置管理**：可以按节点属性来统一查询或修改运行参数，并且支持对节点运行参数进行差异对比分析 12 | - **问题分析**：具备慢命令日志查询、热点key分析、TOP命令跟踪、KEY前缀统计等问题分析能力 13 | 14 | ## 安装部署 15 | 16 | redis-tool工具使用shell脚本实现，下载到具有redis-cli工具的主机上即可使用（通常可部署到redis服务主机上）。 17 | 18 | ```bash 19 | # 1. 从github下载redis-tool工具 20 | # 方法一：使用git下载 21 | $ git clone https://github.com/iwhalecloud-platform/redis-tool.git 22 | # 进入下载目录 23 | $ cd redis-tool 24 | # 方法二：使用wget下载 25 | $ wget https://github.com/iwhalecloud-platform/redis-tool/archive/refs/heads/main.zip 26 | # 解压工具包&修改目录名 27 | $ unzip main.zip && mv redis-tool-main redis-tool 28 | # 进入下载目录 29 | $ cd redis-tool 30 | 31 | # 2. 设置REDIS_HOME环境变量为redis-cli工具所在目录（如果PATH环境变量中包含该目录，则该步可省略） 32 | $ echo "export REDIS_HOME=/path/to/redis-cli/" >> ~/.bashrc 33 | $ source ~/.bashrc 34 | ``` 35 | 36 | ## 使用帮助 37 | 38 | 直接运行**redis-tool**可查看到使用帮助： 39 | 40 | ```bash 41 | $ ./redis-tool 42 | Usage: redis-tool [OPTIONS] [command] 43 | OPTIONS: 44 | -h : Redis集群中某个节点IP (默认值: 127.0.0.1). 45 | -p : Redis集群中某个节点端口 (默认值: 6379). 46 | -i : Redis集群节点信息文件(适配codis等基于PROXY构建的集群模式). 47 | 文件格式(每行一个分片)： [SlaveAddr] [SlaveAddr] 48 | -a : Redis访问密码，也可使用'REDISCLI_AUTH'环境变量来传入. 49 | -c : 1. TOP10命令统计滚动展示次数(默认无限次, 'moni'使用). 50 | 2. 从Redis节点中获取的慢命令记录数(默认值: 100, 'slowlog'使用). 51 | 3. 从Redis节点中随机获取的KEY名称数量(默认值: 100000, 'keys'使用). 52 | -d : TOP10命令统计滚动展示等待间隔(单位: 秒, 默认值: 3, 'moni'使用). 53 | -t : 实时命令监控跟踪运行时长(单位: 秒, 默认值: 10, 'trace'使用). 54 | -s: 只处理指定Redis节点('moni' & 'slowlog'使用). 55 | -l: TOP10命令统计滚动展示包含命令执行平均处理耗时('moni'使用). 56 | -f : 1. 基于监控的命令详情文件进行命令监控跟踪处理('trace'使用). 57 | 2. 基于指定的KEY名称列表文件进行前缀统计('keys'使用). 58 | -L : 按KEY前缀统计层级(默认值: 3, 'keys'使用). 59 | -H: 基于监控的命令详情文件进行热点KEY访问分析('trace'使用). 60 | -C: 基于监控的命令详情文件按客户端IP进行命令统计('trace'使用). 61 | -r: 按裸输据形式输出集群节点监控指标,便于存储或上报至第三方监控系统 ('nodes'使用). 62 | -k : 要查询或修改的Redis运行参数名('config'使用). 63 | -v : 要修改的Redis运行参数值('config'使用). 64 | -M: 只访问集群中主节点的运行参数 ('config'使用). 65 | -S: 只访问集群中从节点的运行参数 ('config'使用). 66 | -w: Redis节点运行参数修改后需要重写配置文件('config'使用). 67 | command: 68 | nodes: 集群节点状态监控 (默认执行) 69 | keys: 按KEY前缀层级进行统计分析 70 | moni: TOP10命令周期性滚动统计 71 | trace: 实时命令监控跟踪 & 热点KEY访问分析 72 | slowlog: 集群慢命令日志查询 73 | config: 查询/修改集群运行参数 & 集群运行参数差异检查 74 | ``` 75 | 76 | 77 | 78 | ## 使用示例 79 | 80 | **注意事项**： 81 | 82 | 1. 请提前设置REDIS_HOME环境变量为redis-cli工具所在目录（如果`PATH`环境变量中包含该目录，可省略） 83 | 2. 强烈建议使用REDISCLI_AUTH环境变量来传入Redis访问密码，提升安全性 84 | 85 | ### Redis集群监控 86 | 87 | 使用`nodes`子命令可以汇总集群各节点的主要指标信息，并按照主从关系进行级联展示（当未指定子命令时，默认执行Redis集群监控子命令），可以针对Redis Cluster、主从Redis、自定义集群(codis、ctgcache)等形式集群的监控。 88 | 89 | #### Redis Cluster集群 90 | 91 | 使用`-h`和`-p`参数传入集群中任一节点的IP和PORT，即可获取整个集群节点的监控指标信息： 92 | 93 | ```bash 94 | $ ./redis-tool -h 172.16.18.80 -p 6380 nodes 95 | ================================================== 2022-04-28 18:44:02 ================================================== 96 | SERVER STATUS ROLE VERSION UPTIME(s) KEYS USEMEM %CPU CLIENTS OPS RTT(us) SLOTS 97 | ------------------------------------------------------------------------------------------------------------------------- 98 | 172.16.18.80:6379 OK master 6.2.6 120542 55 21.56M 0.2 1 2 87 0-5460 99 | |-172.16.18.81:6380 OK slave 6.2.6 120538 55 21.56M 0.2 2 0 7702 100 | 172.16.18.81:6379 OK master 6.2.6 120542 56 21.66M 12.7 2 1317 179 5461-10922 101 | |-172.16.18.82:6380 OK slave 6.2.6 120538 56 21.56M 0.2 2 2 424 102 | 172.16.18.82:6379 OK master 6.2.6 120542 45 21.56M 0.3 1 0 151 10923-16383 103 | |-172.16.18.80:6380 OK slave 6.2.6 120539 45 21.57M 0.2 2 0 84 104 | _________________________________________________________________________________________________________________________ 105 | SUM(master/all): NODE=3/6 KEY=156/312 MEM=64.8M/129.5M CLI=4/10 OPS=1319/1321 CPU=13.2/13.8 106 | ========================================================================================================================= 107 | ``` 108 | 109 | #### 主从Redis 110 | 111 | 同样使用`-h`和`-p`参数传入主从Redis中任一节点的IP和PORT，即可获取主从节点的监控指标信息： 112 | 113 | ```bash 114 | $ redis-tool -h 172.16.18.80 -p 7770 115 | ================================================== 2022-10-12 12:26:54 ================================================== 116 | SERVER STATUS ROLE VERSION UPTIME(s) KEYS USEMEM %CPU CLIENTS OPS RTT(us) SLOTS 117 | ------------------------------------------------------------------------------------------------------------------------- 118 | 172.16.18.80:7770 OK master 6.2.6 251 3 1.87M 0.3 1 1 193 single-M 119 | |-172.16.18.80:7772 OK slave 6.2.6 237 3 1.85M 0.3 2 0 140 single-S 120 | |-172.16.18.80:7771 OK slave 6.2.6 243 3 1.85M 0.2 2 2 119 single-S 121 | _________________________________________________________________________________________________________________________ 122 | SUM(master/all): NODE=1/3 KEY=3/9 MEM=1.9M/5.6M CLI=1/5 OPS=1/3 CPU=0.3/0.8 123 | ========================================================================================================================= 124 | ``` 125 | 126 | #### 自定义集群 127 | 128 | 使用`-i`参数传入自定义Redis集群的节点信息文件（适配codis等基于PROXY构建的集群模式） 129 | 130 | 文件格式（每行一个分片）：` [SlaveAddr] [SlaveAddr]` 131 | 132 | ```bash 133 | # 例如：自定义Redis集群中共有4个分片，每个分片包含1主1从节点 134 | $ cat cluster93.conf 135 | 1 172.16.84.94:19100 172.16.84.93:19100 136 | 2 172.16.84.94:19101 172.16.84.93:19101 137 | 3 172.16.84.93:19102 172.16.84.94:19102 138 | 4 172.16.84.93:19103 172.16.84.94:19103 139 | $ redis-tool -i cluster93.conf 140 | ================================================== 2022-10-12 09:16:10 ================================================== 141 | SERVER STATUS ROLE VERSION UPTIME(s) KEYS USEMEM %CPU CLIENTS OPS RTT(us) SLOTS 142 | ------------------------------------------------------------------------------------------------------------------------- 143 | 172.16.84.93:19102 OK master 2.7.0 5270208 773329 578.64M 0.4 314 48 637 3 144 | |-172.16.84.94:19102 OK slave 2.7.0 5270684 773337 572.38M 0.2 2 6 1312 145 | 172.16.84.93:19103 OK master 2.7.0 5270208 775536 577.49M 0.3 202 20 190 4 146 | |-172.16.84.94:19103 OK slave 2.7.0 5270684 775544 573.72M 0.2 2 1 224 147 | 172.16.84.94:19100 OK master 2.7.0 5270699 774345 581.09M 1.3 396 28 301 1 148 | |-172.16.84.93:19100 OK slave 2.7.0 5270223 774345 573.14M 0.2 2 1 1013 149 | 172.16.84.94:19101 OK master 2.7.0 5270699 777694 579.13M 0.3 175 11 409 2 150 | |-172.16.84.93:19101 OK slave 2.7.0 5270223 777694 575.97M 0.2 2 0 243 151 | _________________________________________________________________________________________________________________________ 152 | SUM(master/all): NODE=4/8 KEY=3100904/6201824 MEM=2.3G/4.5G CLI=1087/1095 OPS=107/115 CPU=2.3/3.1 153 | ========================================================================================================================= 154 | ``` 155 | 156 | #### 输出结果说明 157 | 158 | 1. `SERVER`：Redis服务地址 159 | 160 | 2. `STATUS`：可用于判断节点基本状态 161 | 162 | - `OK`：节点正常运行中 163 | 164 | - `FAIL`：节点连接失败 165 | 166 | - `LOAD`：节点正在进行数据加载 167 | 168 | - `NOAUTH`：节点密码认证失败（需指定正确的访问密码） 169 | 170 | 3. `ROLE`：如果节点角色变更，可能是发生过故障切换，需要通过服务日志进一步核查 171 | 172 | - `master`：主节点 173 | - `slave`：从节点 174 | 175 | 4. `VERSION`：对于Redis集群，版本号应该 >=`3.0.0` 176 | 177 | 5. `UPTIME(s)`：节点已经运行的时长（单位：s），**可用于判断节点是否发生重启** 178 | 179 | 6. `KEYS`：节点中存在的键值数 180 | 181 | 7. `USEMEM`：节点使用的内存量，建议不要超过10G，如果过高应该扩大集群规模 182 | 183 | 8. `CLIENTS`：节点当前接入的客户端连接数 184 | 185 | 9. `%CPU`：节点当前的CPU使用率百分比，通常不应该超过70% 186 | 187 | 10. `OPS`：节点当前每秒处理的命令请求量 188 | 189 | 11. `RTT`：节点当前请求响应时延（单位：us），通常不应该超过1000 190 | 191 | 12. `SUM(master/all)`：按主节点和集群维度，汇总节点数、键值数、内存使用量、客户端连接数、OPS、%CPU等指标信息 192 | 193 | **备注**： 194 | 195 | - 主节点分布均匀，能够更好的发挥Redis性能，而且也有利于故障切换时的主从选举，所以对于主节点分布不均匀时，会给出告警信息 196 | - 工具支持`-r`参数，用于将集群节点指标以裸输据输出，方便进行指标文件存储，或上报至第三方监控平台 197 | 198 | 199 | 200 | ### 慢命令分析 201 | 202 | 使用`slowlog`子命令可以汇总集群各节点的慢命令日志，按日志生成时间进行排序展示格式化后的命令详情： 203 | 204 | ```bash 205 | $ redis-tool -p 6380 slowlog 206 | 2022-04-28 18:13:46 [CLUSTER] get slowlog of cluster by the node 127.0.0.1:6380 (COUNT=100) 207 | =========================================== 2022-04-28 18:13:47 ============================================ 208 | Redis ID Time[Asc] Duration(us) Client Command 209 | ------------------------------------------------------------------------------------------------------------ 210 | 172.16.18.80:6379 0 20220428181334 728 172.16.18.80:47026 [ "COMMAND" ] 211 | 172.16.18.81:6379 0 20220428181334 897 172.16.18.80:58702 [ "COMMAND" ] 212 | 172.16.18.82:6379 0 20220428181334 996 172.16.18.80:57426 [ "COMMAND" ] 213 | 172.16.18.80:6379 1 20220428181337 697 172.16.18.80:47044 [ "COMMAND" ] 214 | 172.16.18.81:6379 1 20220428181337 1033 172.16.18.80:58714 [ "COMMAND" ] 215 | 172.16.18.82:6379 1 20220428181337 1024 172.16.18.80:57444 [ "COMMAND" ] 216 | 172.16.18.80:6379 2 20220428181341 716 172.16.18.80:47068 [ "COMMAND" ] 217 | 172.16.18.81:6379 2 20220428181341 1320 172.16.18.80:58742 [ "COMMAND" ] 218 | 172.16.18.82:6379 2 20220428181341 767 172.16.18.80:57470 [ "COMMAND" ] 219 | 172.16.18.80:6379 3 20220428181344 705 172.16.18.80:47104 [ "COMMAND" ] 220 | 172.16.18.81:6379 3 20220428181344 886 172.16.18.80:58780 [ "COMMAND" ] 221 | 172.16.18.82:6379 3 20220428181344 834 172.16.18.80:57504 [ "COMMAND" ] 222 | 172.16.18.82:6379 4 20220428181344 111 172.16.18.80:57504 [ "info" "commandstats" ] 223 | ---------------------------[ 20220428: count = 13 ]------------------------------------------------- 224 | ``` 225 | 226 | **结果说明**： 227 | 228 | - 日志按时间顺序展示，当前页面展示最新的日志信息 229 | - 对于存在慢命令日志的日期会进行日志数量汇总 230 | - 工具支持`-c `参数，来指定查询节点最新的慢命令记录数（默认值：`100`） 231 | 232 | **慢命令日志主要原因**： 233 | 234 | - **大key访问**：key值包含的元素数太大时对Redis服务性能影响较大，建议优化业务逻辑，进行大key拆分或使用替代命令（例如使用`HSCAN`代替`HGETALL`） 235 | - **复杂命令**：`KEYS`、`FLUSHDB`、`HGETALL`等，生产环境通常禁止使用 236 | - **主机内存不足**：当主机因为内存不足，而使用了磁盘交换区时，对于Redis服务性能影响很大，需要进行内存使用限制或主机资源扩容 237 | - **主机CPU繁忙**：主机CPU配置较低，或者部署了其他高CPU占用软件，生产环境建议Redis主机独占部署 238 | - **CPU使用限制**：通过cgroup限制了Redis进程的CPU使用率，导致访问性能下降，生产环境不建议进行限制 239 | 240 | ### 热点key分析 241 | 242 | 当发现某个Redis节点的`%CPU`或`OPS`指标相对于其他节点高很多时，通常是因为存在热点key访问，此时可以使用`trace`子命令实时监控该节点的命令请求，并分析KEY的请求百分比，从而发现热点key： 243 | 244 | **备注**：实时命令监控统计实际使用的时Redis的`monitor`命令，因此建议在节点所在主机上执行，并且不要使用`-t`参数调整监控时间过长。 245 | 246 | ```bash 247 | # 1. 实时监控统计节点每秒处理的命令，并记录命令详情至文件中[172.16.18.81-6379.mon] 248 | # 默认实时跟踪10秒，可通过-t 参数进行修改 249 | $ ./redis-tool -h 172.16.18.81 -p 6379 trace 250 | 2022-04-28 19:07:42 Analyze the command on seconds for the node 172.16.18.81:6379 ... 251 | -------------------------------------------------------------------------------- 252 | 19:07:42> "GET":1584 253 | 19:07:43> "GET":4927 254 | 19:07:44> "GET":3255 255 | 19:07:45> "AUTH":1 "COMMAND":1 "GET":5052 "INFO":3 256 | 19:07:46> "GET":2842 257 | 19:07:47> "GET":5699 258 | 19:07:48> "AUTH":1 "COMMAND":1 "GET":3598 "INFO":3 259 | 19:07:49> "GET":4543 260 | 19:07:50> "GET":3699 261 | 19:07:51> "AUTH":1 "COMMAND":1 "GET":5074 "INFO":3 262 | 19:07:52> "GET":4025 263 | 2022-04-28 19:07:52 You can check the monitor file: [172.16.18.81-6379.mon], or analyze the file by -f again! 264 | # 2. 基于监控的命令详情文件进行热点KEY访问分析 265 | $ ./redis-tool -f 172.16.18.81-6379.mon -H trace 266 | 2022-04-28 19:08:06 Analyze the monitor file 172.16.18.81-6379.mon for hot keys ... 267 | ================================================== ======= ======= 268 | KEY COUNT PERCENT 269 | -------------------------------------------------- ------- ------- 270 | "commandstats" 3 0.01% 271 | "cpu" 3 0.01% 272 | NO-KEY 3 0.01% 273 | "(redacted)" 3 0.01% 274 | "stats" 3 0.01% 275 | "key1" 44298 99.97% 276 | ``` 277 | 278 | 279 | 280 | 热点key的存在通常需要从业务使用侧进行优化，**主要的处理策略**： 281 | 282 | - 将每次业务访问的key进行拆分，避免总是访问同一个key 283 | - 对需要频繁访问的key进行本地缓存，本地缓存数据可以通过定时策略进行更新 284 | - 优化业务流程处理逻辑，减少无效的交互访问次数 285 | 286 | 287 | 288 | ### TOP命令跟踪 289 | 290 | 使用`moni`子命令能够滚动展示周期内TOP10命令执行的执行情况；当存在大量非应用直接调用的命令时（例如`PING`、`CLUSTER`），或者某个命令执行次数或平均处理耗时不符合预期，都可以作为下一步排查的方向。 291 | 292 | **备注**： 293 | 294 | - 默认汇总集群所有主节点的命令执行情况；当使用`-s`参数时，将只展示指定Redis节点的命令执行情况 295 | - 每滚动输出20次命令统计后，会重新计算当前的TOP10命令进行展示 296 | 297 | ```bash 298 | # 1. 每秒滚动展示集群TOP10命令 299 | $ ./redis-tool -h 172.16.18.80 -p 6380 -d 1 moni 300 | 2022-04-28 18:46:11 [CLUSTER] monitor with 3 masters from 172.16.18.80:6380 every 1 seconds 301 | -----------------------TOP10 commands 2022-04-28 18:46:12----------------------- -----all------ -----------netio----- 302 | get info replconf command auth time slowlog set select scan TOTAL %CPU CONNS IN(KB) OUT(KB) 303 | 4445 9 3 3 3 0 0 0 0 0 4463 17.5 7 100 118 304 | 6115 9 4 3 3 0 0 0 0 0 6134 19.7 4 139 139 305 | 4038 9 4 3 3 0 0 0 0 0 4057 16.0 7 91 113 306 | 5647 9 4 3 3 0 0 0 0 0 5666 23.0 6 127 133 307 | 6051 9 3 3 3 0 0 0 0 0 6069 22.0 6 137 137 308 | ... ... 309 | # 2. 每3秒滚动展示集群TOP10命令及平均处理耗时(*数值，对应的即为平均处理耗时，单位：us) 310 | $ ./redis-tool -h 172.16.18.80 -p 6380 -c 5 -l moni 311 | 2022-04-28 18:47:17 [CLUSTER] monitor with 3 masters from 172.16.18.80:6380 every 3 seconds 312 | -----------------------TOP10 commands 2022-04-28 18:47:20----------------------- -----all------ -----------netio----- 313 | get replconf info command auth time slowlog set select scan TOTAL %CPU CONNS IN(KB) OUT(KB) 314 | 14513 10 9 3 3 0 0 0 0 0 14538 21.2 10 327 237 315 | *1 *1 *32 *891 *7 *0 *0 *0 *0 *0 316 | 16470 10 9 3 3 0 0 0 0 0 16495 22.8 11 371 259 317 | *1 *1 *33 *904 *30 *0 *0 *0 *0 *0 318 | 11883 9 9 3 3 0 0 0 0 0 11907 16.9 16 268 206 319 | *1 *1 *25 *882 *8 *0 *0 *0 *0 *0 320 | 15255 10 9 3 3 0 0 0 0 0 15280 19.2 11 343 245 321 | *1 *1 *28 *979 *9 *0 *0 *0 *0 *0 322 | 7974 9 9 3 3 0 0 0 0 0 7998 13.2 9 180 160 323 | *1 *1 *37 *1020 *8 *0 *0 *0 *0 *0 324 | 325 | ``` 326 | 327 | **结果说明**： 328 | 329 | 1. **TOP10 commands**： 330 | - 命令执行次数：周期内命令执行次数 331 | - 平均处理耗时：使用`-l`参数时，会增加一行显示命令平均处理耗时（以`*`开头） 332 | 2. **all**： 333 | - TOTAL：周期内执行的命令总数 334 | - %CPU：当前节点的CPU使用率汇总 335 | 3. **netio**： 336 | - CONNS：周期内新建客户端连接数 337 | - IN(KB)：周期内网络流入数据量汇总 338 | - OUT(KB)：周期内网络流出数据量汇总 339 | 340 | 341 | 342 | ### KEY前缀统计 343 | 344 | 使用`keys`子命令可以快速的抽取一定数量的KEY名称，对KEY按前缀统计键值数和占比信息后按前缀层级进行展示，其中第一层级会采用背景高亮，而键值占比达到一定阈值时会进行字体高亮区分： 345 | 346 | ```bash 347 | # 1. 连接指定redis节点进行KEY前缀统计（默认前缀层级为3） 348 | $ ./redis-tool -p 6380 keys 349 | Time elapsed 0 (s): 52 350 | ============================================= 351 | KEY_PREFIX | COUNT | PERCENT 352 | ------------------------|----------|--------- 353 | hash: | 5 | 9.62% 354 | |- hash.foo | 5 | 9.62% 355 | |- hash.foo.00000 | 5 | 9.62% 356 | key0: | 4 | 7.69% 357 | set: | 4 | 7.69% 358 | |- set.aaa | 4 | 7.69% 359 | |- set.aaa.bbb | 4 | 7.69% 360 | str: | 39 | 75.00% 361 | |- str.abc | 2 | 3.85% 362 | |- str.abc.efg | 1 | 1.92% 363 | |- str.abc.lmn | 1 | 1.92% 364 | |- str.xxx | 37 | 71.15% 365 | |- str.xxx.0 | 5 | 9.62% 366 | |- str.xxx.00 | 32 | 61.54% 367 | ============================================= 368 | # 2. 使用上面生成的键名称文件127.0.0.1-6380.keys进行分析，前缀层级为4 369 | $ ./redis-tool -f 127.0.0.1-6380.keys -L 4 keys 370 | ================================================== 371 | KEY_PREFIX | COUNT | PERCENT 372 | -----------------------------|----------|--------- 373 | hash: | 5 | 9.62% 374 | |- hash.foo | 5 | 9.62% 375 | |- hash.foo.00000 | 5 | 9.62% 376 | key0: | 4 | 7.69% 377 | set: | 4 | 7.69% 378 | |- set.aaa | 4 | 7.69% 379 | |- set.aaa.bbb | 4 | 7.69% 380 | |- set.aaa.bbb.ccc | 4 | 7.69% 381 | str: | 39 | 75.00% 382 | |- str.abc | 2 | 3.85% 383 | |- str.abc.efg | 1 | 1.92% 384 | |- str.abc.efg.000 | 1 | 1.92% 385 | |- str.abc.lmn | 1 | 1.92% 386 | |- str.abc.lmn.00 | 1 | 1.92% 387 | |- str.xxx | 37 | 71.15% 388 | |- str.xxx.0 | 5 | 9.62% 389 | |- str.xxx.00 | 32 | 61.54% 390 | ================================================== 391 | ``` 392 | 393 | **备注**：KEY前缀连接符统一使用`.`进行代替（这样也方便使用`grep`命令来对keys文件按前缀进行检索） 394 | 395 | ### 配置参数管理 396 | 397 | 使用`config`子命令可以方便的检查各节点的配置差异，以及按角色来统一查询/修改指定运行参数。 398 | 399 | #### 检查节点配置差异 400 | 401 | 当未指定参数进行查询或修改时，默认会检查集群中各节点的配置差异，并且按各参数值数量(`UNIQ`)升序展示，其中存在不同值的参数行会高亮显示： 402 | 403 | - CONFIG_NAME：配置参数名 404 | - UNIQ：不同参数值数量，`1`表示各节点的参数值相同 405 | - DIFF_VALUES：参数值详情，不同参数值以逗号（`,`）间隔 406 | 407 | ```bash 408 | $ ./redis-tool -p 6380 config 409 | 2022-04-28 17:52:54 Audit the config with diffrent value on all(count=6) nodes: 410 | ==================================================================================================== 411 | CONFIG_NAME | UNIQ | DIFF_VALUES 412 | ---------------------------------------------------------------------------------------------------- 413 | aclfile | 1 | 414 | acllog-max-len | 1 | 128 415 | acl-pubsub-default | 1 | allchannels 416 | ... ... 417 | zset-max-ziplist-entries | 1 | 128 418 | zset-max-ziplist-value | 1 | 64 419 | dbfilename | 2 | .dump.rdb,dump.rdb 420 | dir | 2 | /home/redis/data/6380,/tmp 421 | logfile | 2 | redis-6379.log,redis-6380.log 422 | pidfile | 2 | /home/redis/data//6379/redis-6379.pid,/home/redis/data/... 423 | port | 2 | 6379,6380 424 | slaveof | 4 | ,172.16.18.80 6379,172.16.18.81 6379,172.16.18.82 6379 425 | ---------------------------------------------------------------------------------------------------- 426 | CONFIG_NAME | UNIQ | DIFF_VALUES 427 | ==================================================================================================== 428 | ``` 429 | 430 | 431 | 432 | #### 查询/修改指定参数 433 | 434 | 集群参数查询/修改范围包括三类：所有节点（默认），所有主节点（`-M`），所有从节点（`-S`） 435 | 436 | ```bash 437 | # 1. 查询集群各节点的slowlog-log-slower-than参数值 438 | $ ./redis-tool -p 6380 -k slowlog-log-slower-than config 439 | 2022-04-28 18:11:23 GET the value of 'slowlog-log-slower-than' on all nodes: 440 | ================================================================================ 441 | REDIS | slowlog-log-slower-than 442 | -------------------------------------------------------------------------------- 443 | 172.16.18.80:6379 | "10000" 444 | 172.16.18.80:6380 | "10000" 445 | 172.16.18.81:6379 | "10000" 446 | 172.16.18.81:6380 | "10000" 447 | 172.16.18.82:6379 | "10000" 448 | 172.16.18.82:6380 | "10000" 449 | ================================================================================ 450 | # 2. 修改集群主节点的slowlog-log-slower-than参数值为100 451 | $ ./redis-tool -p 6380 -k slowlog-log-slower-than -v 100 -M config 452 | 2022-04-28 18:12:55 SET the value of 'slowlog-log-slower-than' on master nodes: 453 | 2022-04-28 18:12:55 [172.16.18.80:6379]: CONFIG SET slowlog-log-slower-than 100 : [OK] 454 | 2022-04-28 18:12:55 [172.16.18.81:6379]: CONFIG SET slowlog-log-slower-than 100 : [OK] 455 | 2022-04-28 18:12:55 [172.16.18.82:6379]: CONFIG SET slowlog-log-slower-than 100 : [OK] 456 | 2022-04-28 18:12:55 Get the value of 'slowlog-log-slower-than' on master nodes after CONFIG SET: 457 | ================================================================================ 458 | REDIS | slowlog-log-slower-than 459 | -------------------------------------------------------------------------------- 460 | 172.16.18.80:6379 | "100" 461 | 172.16.18.81:6379 | "100" 462 | 172.16.18.82:6379 | "100" 463 | ================================================================================ 464 | ``` 465 | 466 | **备注**：使用`-w`参数时，当Redis运行参数修改后，会重写Redis配置文件，这样节点重启后能够继续生效 467 | 468 | **密码参数统一修改示例**： 469 | 470 | ```bash 471 | # 示例一：无密码集群新增密码：redis 472 | redis-tool -h 172.16.18.80 -p 6379 -a '' -k masterauth -v 'redis' config 473 | redis-tool -h 172.16.18.80 -p 6379 -a '' -k requirepass -v 'redis' -w config 474 | redis-tool -h 172.16.18.80 -p 6379 -a 'redis' 475 | 476 | # 示例二：将集群密码redis修改为Red1s@2022 477 | redis-tool -h 172.16.18.80 -p 6379 -a 'redis' -k masterauth -v 'Red1s@2022' config 478 | redis-tool -h 172.16.18.80 -p 6379 -a 'redis' -k requirepass -v 'Red1s@2022' -w config 479 | redis-tool -h 172.16.18.80 -p 6379 -a 'Red1s@2022' 480 | 481 | # 示例三：去掉集群密码设置 482 | redis-tool -h 172.16.18.80 -p 6379 -a 'Red1s@2022' -k masterauth -v '' config 483 | redis-tool -h 172.16.18.80 -p 6379 -a 'Red1s@2022' -k requirepass -v '' -w config 484 | redis-tool -h 172.16.18.80 -p 6379 485 | ``` 486 | 487 | 488 | 489 | 490 | 491 | 492 | 493 | -------------------------------------------------------------------------------- /redis-tool: -------------------------------------------------------------------------------- 1 | #! /bin/bash 2 | # 3 | # redis cluster daily maintenance tool 4 | 5 | 6 | #################################### global ################################### 7 | readonly USAGE="Usage: $(basename $0) [OPTIONS] [command] 8 | OPTIONS: 9 | -h : Redis hostname (default: 127.0.0.1). 10 | -p : Redis port (default: 6379). 11 | -i : Cluster nodes file. File format: 12 | [SlaveAddr] [SlaveAddr] 13 | [SlaveAddr] [SlaveAddr] 14 | -a : Password to use when connecting to redis. 15 | You can also use the REDISCLI_AUTH environment 16 | variable to pass this password more safely. 17 | -c : 1. Rolling times for command statistics (default is inf, used by 'moni'). 18 | 2. Get slowlog number from each node (default: 100, used by 'slowlog'). 19 | 3. Get key number from the node (default: 100000, used by 'keys'). 20 | -d : Waits seconds for monitor (default: 3, used by 'moni'). 21 | -t : The running time(s) on trace (default: 10, used by 'trace'). 22 | -s: Process the specify node only (used by 'moni' & 'slowlog'). 23 | -l: Show average latency for TOP10 commands (used by 'moni'). 24 | -f : 1. Monitor file name to load (used by 'trace'). 25 | 2. Keys file name to statistics (used by 'keys'). 26 | -L : Key prefix level (default: 3, usede by 'keys'). 27 | -H: Analyze the hot keys when use option '-f' (used by 'trace'). 28 | -C: Stat commands by cient host when use option '-f' (used by 'trace'). 29 | -r: Output raw data of cluster nodes (used by 'nodes'). 30 | -k : Config name to get/set (used by 'config'). 31 | -v : Config value to set (used by 'config'). 32 | -M: Only access masters config (used by 'config'). 33 | -S: Only access slaves config (used by 'config'). 34 | -w: Rewrite the config file after set online (used by 'config'). 35 | command: 36 | nodes: Show cluster nodes status (default) 37 | keys: Key prefix statistics 38 | moni: Top10 command rolling statistics 39 | trace: Hotspot key tracing statistics 40 | slowlog: Querying cluster slow command logs 41 | config: Get/Set cluster config 42 | Examples: 43 | # show cluster status by node(127.0.0.1:6379) 44 | $ $(basename $0) nodes 45 | # get cluster slowlog by node(127.0.0.1:7379) 46 | $ $(basename $0) -p 7379 slowlog 47 | # get config of loglevel by node(10.10.10.3:7379) 48 | $ $(basename $0) -h 10.10.10.3 -p 7379 -k loglevel config\n" 49 | 50 | # MYPID: pid of myself 51 | readonly MYPID=$$ 52 | # XXX_PREFIX: temp file prefix name for subcommand 53 | readonly CMDS_PREFIX=".RCTOOL-CMDS-" 54 | readonly NODES_PREFIX=".RCTOOL-NODES-" 55 | readonly INFOS_PREFIX=".RCTOOL-INFOS-" 56 | readonly CLIENTS_PREFIX=".RCTOOL-CLIENTS-" 57 | readonly MONI_PREFIX=".RCTOOL-MONI-" 58 | readonly SLOWLOG_PREFIX=".RCTOOL-SLOWLOG-" 59 | 60 | # redis server ip & password(default: 'REDISCLI_AUTH' environment variable) 61 | REDIS_HOST="127.0.0.1" 62 | REDIS_PORT=6379 63 | REDIS_PASS="${REDISCLI_AUTH}" 64 | # PREFIX_LVL: prefix key level, use option '-l' to set 65 | PREFIX_LVL=3 66 | # TRACE_TIME: how long to trace: default 10s 67 | TRACE_TIME=10 68 | # ROLL_TIME: rolling time for moni: default 3s 69 | ROLL_TIME=3 70 | # MONI_COUNT: moni N times: default infinite 71 | MONI_COUNT=-1 72 | # SLOW_COUNT: count of slowlog to get: default 100 73 | SLOW_COUNT=100 74 | # KEYS_COUNT: count of keys to stat by prefix: default 100000 75 | KEYS_COUNT=100000 76 | # CMD: subcommand: default 'nodes' 77 | CMD="nodes" 78 | # DATABASE: redis database: default 0 79 | DATABASE=0 80 | 81 | ####################### function ######################## 82 | 83 | # clean temp files 84 | function cleanup() { 85 | rm -f ${CMDS_PREFIX}*.${MYPID} \ 86 | ${NODES_PREFIX}*.${MYPID} \ 87 | ${INFOS_PREFIX}*.${MYPID} \ 88 | ${CLIENTS_PREFEX}*.${MYPID} \ 89 | ${MONI_PREFIX}*.${MYPID} \ 90 | ${SLOWLOG_PREFIX}*.${MYPID} || true 91 | } 92 | # when exception: kill self & subprocess 93 | function exitup() { 94 | cleanup 95 | kill -9 0 96 | } 97 | # print info message 98 | function debug() { 99 | echo -e "$(date +'%F %T') \033[37m\033[0m $@" 100 | } 101 | # print warning message 102 | function warn() { 103 | echo -e "$(date +'%F %T') \033[31m\033[0m $@" 104 | } 105 | # check version 106 | function version_ge() { 107 | test "$(echo "$@" | tr " " "\n" | sort -rV | head -n 1)" == "$1" 108 | } 109 | 110 | ####################################### 111 | # check option is integer 112 | # Globals: 113 | # USAGE 114 | # Arguments: 115 | # 1 - option value 116 | # 2 - option type 117 | # Returns: 118 | # None 119 | ####################################### 120 | function assert_int() { 121 | local -r int="${1:?missing value}" 122 | if [[ ! "$int" =~ ^[0-9]+$ ]]; then 123 | warn "Invalid integer for option '-${2}': $int" 124 | echo -e "$USAGE" 125 | exit 1 126 | fi 127 | } 128 | 129 | ####################################### 130 | # check redis-cli to be useful 131 | # Globals: 132 | # REDISCLI_VERSION - redis-cli version 133 | # Arguments: 134 | # None 135 | # Returns: 136 | # None 137 | ####################################### 138 | function check_redis_cli() { 139 | # update PATH environment variable 140 | export PATH=$REDIS_HOME:.:$PATH 141 | local version=$(redis-cli -v 2>&1) 142 | if [[ $? -ne 0 ]]; then 143 | warn "Please use the REDIS_HOME environment variable to pass redis-cli path!" 144 | exit 1 145 | fi 146 | REDISCLI_VERSION=$(echo $version | awk -F" |-" '{print $3}') 147 | } 148 | 149 | ####################################### 150 | # set REDISCLI_AUTH environment variable 151 | # Globals: 152 | # REDIS_PASS - redis password 153 | # REDISCLI_AUTH - password used by redis-cli 154 | # AUTH - password option 155 | # Arguments: 156 | # None 157 | # Returns: 158 | # None 159 | ####################################### 160 | function set_rediscli_auth() { 161 | if [[ -n "$REDIS_PASS" ]]; then 162 | if version_ge "$REDISCLI_VERSION" "5.0.3"; then 163 | export REDISCLI_AUTH="$REDIS_PASS" 164 | else 165 | AUTH="-a $REDIS_PASS" 166 | fi 167 | else 168 | export -n REDISCLI_AUTH 169 | AUTH="" 170 | fi 171 | } 172 | 173 | ####################################### 174 | # cluster cluster nodes list 175 | # Globals: 176 | # REDIS_HOST - redis ip 177 | # REDIS_PORT - redis port 178 | # AUTH - password option 179 | # CLUSTER_NODES_ALL - all nodes info 180 | # MASTER_LIST - master addr list 181 | # Arguments: 182 | # None 183 | # Returns: 184 | # None 185 | ####################################### 186 | function get_cluster_nodes() { 187 | # ID ip:port flags masterID pingtime pongtime configepoch connect slots 188 | local nodes_res="" 189 | if [[ ${#CLUSTER_NODES_ALL} -eq 0 ]]; then 190 | nodes_res=$(redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} ${AUTH} cluster nodes 2>&1) 191 | if [[ $? -eq 0 && "${nodes_res:0:8}" != "LOADING " && "${nodes_res:0:7}" != "NOAUTH " && "${nodes_res:0:4}" != "ERR " ]]; then 192 | # 1.masterID 2.role 3.ID 4.ip:port 5.flags 6.masterID 7.pingtime 8.pongtime 9.configepoch 10.connect 11.slots 193 | CLUSTER_NODES_ALL=$(echo "$nodes_res"|awk '{if($4=="-"){print $1" master "$0}else{print $4" slave "$0}}'|sort) 194 | elif [[ "$nodes_res" =~ "cluster support disabled" ]]; then 195 | # redis isn't in cluster mode 196 | local role_info=$(redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} ${AUTH} --raw role 2>&1) 197 | local role=$(echo "$role_info"|head -1) 198 | if [[ "$role" == "master" || "$role" == "slave" ]]; then 199 | # used master role info: 1 master && >=1 slaves 200 | if [[ "$role" == "slave" ]]; then 201 | local master_ip=$(echo "$role_info"|head -2|tail -1) 202 | local master_port=$(echo "$role_info"|head -3|tail -1) 203 | local master_role_info=$(redis-cli -h ${master_ip} -p ${master_port} ${AUTH} --raw role 2>&1) 204 | local master_role=$(echo "$master_role_info"|head -1) 205 | if [[ "$master_role" == "master" ]]; then 206 | role_info="$master_role_info" 207 | REDIS_HOST=$master_ip 208 | REDIS_PORT=$master_port 209 | fi 210 | fi 211 | local awk_cmd=' 212 | { 213 | idx+=1; 214 | if(role==""){ 215 | role=$1; 216 | if(role=="master") { 217 | print myaddr,"master",myaddr,myaddr,"master","-","pingtime","pongtime","poch","connected","single-M"; 218 | } 219 | } else if(role=="master") { 220 | if(idx>2) { 221 | if((idx-3)%3 == 0) { 222 | slaveip=$1 223 | } else if((idx-3)%3 == 1) { 224 | slave=slaveip":"$1 225 | print myaddr,"slave",slave,slave,"slave",myaddr,"pingtime","pongtime","poch","connected","single-S" 226 | } 227 | } 228 | } else if(role=="slave") { 229 | if(idx==2) { 230 | masterip=$1; 231 | } else if(idx==3) { 232 | master=masterip":"$1 233 | print master,"master",master,master,"master","-","pingtime","pongtime","poch","connected","single-M" 234 | print master,"slave",myaddr,myaddr,"slave",master,"pingtime","pongtime","poch","connected","single-S" 235 | } 236 | } 237 | }' 238 | 239 | CLUSTER_NODES_ALL=$(echo "$role_info" | awk -v myaddr=${REDIS_HOST}:${REDIS_PORT} "$awk_cmd") 240 | fi 241 | fi 242 | if [[ ${#CLUSTER_NODES_ALL} -eq 0 ]]; then 243 | warn "Access redis(${REDIS_HOST}:${REDIS_PORT}) failed: \033[41;37m${nodes_res}\033[0m" 244 | if [ "${nodes_res:0:7}" == "NOAUTH " ]; then 245 | warn "Please input redis password by option '-a '!" 246 | elif [ "${nodes_res:0:8}" == "LOADING " ]; then 247 | warn "Please retry the command after LOADING finished!" 248 | else 249 | warn "Please input active redis by option '-h -p '!" 250 | fi 251 | exit 1 252 | fi 253 | fi 254 | # cut cluster port from node addr 255 | CLUSTER_NODES_ALL=$(echo "$CLUSTER_NODES_ALL"|sed -re 's/@[0-9]*//g') 256 | 257 | # masters addr list 258 | MASTER_LIST=$(echo "${CLUSTER_NODES_ALL}"|grep -w master|grep -v noaddr|awk '{print $4}') 259 | 260 | # all nodes addr list 261 | NODES_LIST=$(echo "${CLUSTER_NODES_ALL}"|grep -v noaddr|awk '{print $4}') 262 | } 263 | 264 | ####################################### 265 | # check redis status by ping 266 | # Globals: 267 | # REDIS_HOST - redis ip 268 | # REDIS_PORT - redis port 269 | # AUTH - password option 270 | # Arguments: 271 | # None 272 | # Returns: 273 | # None 274 | ####################################### 275 | function check_redis_by_ping() { 276 | local ping_res=$(redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} ${AUTH} ping 2>&1) 277 | if [ "$ping_res" != "PONG" ]; then 278 | warn "Access redis(${REDIS_HOST}:${REDIS_PORT}) failed: \033[41;37m${ping_res}\033[0m" 279 | if [ "${ping_res:0:7}" == "NOAUTH " ]; then 280 | warn "Please input redis password by option '-a '!" 281 | elif [ "${ping_res:0:8}" == "LOADING " ]; then 282 | warn "Please retry the command after LOADING finished!" 283 | else 284 | warn "Please input active redis by option '-h -p '!" 285 | fi 286 | exit 1 287 | fi 288 | } 289 | 290 | ####################################### 291 | # print divide line with title 292 | # Arguments: 293 | # 1 - length of line 294 | # 2 - character of line 295 | # 3 - title 296 | # 4 - new line 297 | # Returns: 298 | # divide line 299 | ####################################### 300 | function draw_divide_line() { 301 | local count=$1 302 | local ch=$2 303 | local title=$3 304 | local newline=${4-'\n'} 305 | local title_len=${#title} 306 | local pre_len=$(((count-title_len)/2)) 307 | local str_result="" 308 | for((i=1;i<=$pre_len;i++)) 309 | do 310 | str_result+=$ch 311 | done 312 | str_result+="$title" 313 | local sub_len=$((count-title_len-pre_len)) 314 | for((i=1;i<=$sub_len;i++)) 315 | do 316 | str_result+=$ch 317 | done 318 | echo -en "$str_result$newline" 319 | } 320 | 321 | ####################################### 322 | # show progress bar 323 | # Arguments: 324 | # 1 - length of line 325 | # 2 - character of line 326 | # 3 - title 327 | # 4 - new line 328 | # Returns: 329 | # progress bar 330 | ####################################### 331 | function show_bar() { 332 | local head_msg=$1 333 | local total_num=$2 334 | local proc_num=$3 335 | local progress=0 336 | # length of bar is 50 337 | if [[ $total_num -eq 0 ]]; then 338 | progress=50 339 | else 340 | progress=$((proc_num * 50 / total_num)) 341 | if [[ $progress -gt 50 ]]; then 342 | progress=50 343 | fi 344 | fi 345 | 346 | local progs_info="" 347 | for ((i=0;i<$progress;i++)); do 348 | progs_info="#${progs_info}" 349 | done 350 | if [ "$progs_info" = "" ]; then 351 | progs_info="#" 352 | fi 353 | printf "<%s> | [%-50s]%d\n" $head_msg $progs_info $proc_num 354 | } 355 | 356 | ####################################### 357 | # show progress to figured out 358 | # Arguments: 359 | # 1 - nodes info 360 | # Returns: 361 | # progress bar 362 | ####################################### 363 | function show_progress() { 364 | local node_num=$(echo -e "$1"|wc -l) 365 | local proc_num=0 366 | # show progress with tty, and the cluster nodes > 20 367 | tty >/dev/null 368 | if [[ $? -eq 0 && $node_num -gt 20 ]]; then 369 | while :; 370 | do 371 | proc_num=$(wc -l ${NODES_PREFIX}*.${MYPID} 2>/dev/null|tail -1|awk '{print $1}') 372 | show_bar "TOTAL:$node_num" $node_num $proc_num 373 | # break when figured out 374 | if [[ $node_num -eq $proc_num ]]; then 375 | break 376 | fi 377 | sleep 0.5 378 | # reset cursor 379 | echo -ne "\033[1A\r" 380 | done 381 | fi 382 | } 383 | 384 | ####################################### 385 | # config get/set for cluster 386 | # Globals: 387 | # CLUSTER_NODES_ALL - cluster nodes list 388 | # REDIS_HOST - redis ip 389 | # REDIS_PORT - redis port 390 | # REDIS_PASS - redis password 391 | # AUTH - password option 392 | # CONFIG_SET - config set value 393 | # CONFIG_KEY - config name to get/set 394 | # CONFIG_VALUE - config value to set 395 | # ROLE_TYPE - role type: master,slave,all 396 | # Arguments: 397 | # None 398 | # Returns: 399 | # get/set result 400 | ####################################### 401 | function setget_config() { 402 | # CLUSTER_NODES_ALL: 403 | # 1.masterID 2.role 3.ID 4.ip:port 5.flags 6.masterID 7.pingtime 8.pongtime 9.configepoch 10.connect 11.slots 404 | get_cluster_nodes 405 | 406 | OLDIFS=$IFS 407 | local head_line=$(draw_divide_line 80 "=") 408 | local div_line=$(draw_divide_line 80 "-") 409 | local conf_res="" 410 | if [[ -z "$CONFIG_KEY" ]]; then 411 | # audit config value with diffrent values 412 | declare -A map_config 413 | local conf_key="" 414 | local conf_val="" 415 | local node_count=$(echo "$CLUSTER_NODES_ALL"|grep "$ROLE_TYPE"|wc -l) 416 | debug "Audit the config with diffrent value on \033[37m${ROLE_TYPE:-all}(count=$node_count)\033[0m nodes:" 417 | local proc_count=0 418 | for node in $(echo "$CLUSTER_NODES_ALL"|grep "$ROLE_TYPE"|grep -v noaddr|awk '{print $4}'|sort); do 419 | # ip & port 420 | local node_ip=${node%:*} 421 | local node_port=${node##*:} 422 | if [ "$node_ip" = "" ]; then 423 | node_ip="127.0.0.1" 424 | fi 425 | # get all configs from redis 426 | conf_res=$(redis-cli -h ${node_ip} -p ${node_port} ${AUTH} --raw config get "*"|awk '{if(x==0){x=1;printf("%s=",$1);}else{x=0;print $0}}') 427 | if [[ ${#conf_res} -lt 200 ]]; then 428 | warn "Get config from ${node_ip}:${node_port} failed: \033[41;37m${conf_res}\033[0m" 429 | continue 430 | fi 431 | IFS=$'\n' 432 | for conf in $(echo "$conf_res"); do 433 | conf_key=${conf%%=*} 434 | conf_val=${conf#*=} 435 | map_config[$conf_key]="${map_config[$conf_key]}${conf_val}\n" 436 | done 437 | IFS=$OLDIFS 438 | # show progress barcursor 439 | if [[ $node_count -gt 20 ]]; then 440 | proc_count=$((proc_count+1)) 441 | show_bar "TOTAL:$node_count" $node_count $proc_count 442 | if [[ $node_count -gt $proc_count ]]; then 443 | # reset cursor 444 | echo -ne "\033[1A\r" 445 | fi 446 | fi 447 | done 448 | 449 | # audit the config values 450 | head_line=$(draw_divide_line 100 "=") 451 | div_line=$(draw_divide_line 100 "-") 452 | local title=$(printf " %-32s| %s | %s" "CONFIG_NAME" "UNIQ" "DIFF_VALUES") 453 | echo -e "${head_line}\n${title}\n${div_line}" 454 | local diff_count=0 455 | local diff_info="" 456 | local info="" 457 | for key in ${!map_config[@]}; do 458 | diff_count=$(echo -en "${map_config[$key]}" | sort -u | wc -l) 459 | local values=$(echo -en "${map_config[$key]}" | sort -u | tr '\n' ',') 460 | # delete the last ',' 461 | values=${values%?} 462 | if [[ ${#values} -gt 58 ]]; then 463 | values="${values:0:55}..." 464 | fi 465 | # high light for diffrent values 466 | info=$(printf " %-32s| %-4s | %-58s" "${key}" "$diff_count" "${values}") 467 | if [[ $diff_count -gt 1 ]]; then 468 | info="\033[43;31m${info}\033[0m" 469 | fi 470 | 471 | if [[ ${#diff_info} -gt 0 ]]; then 472 | diff_info="${diff_info}\n${info}" 473 | else 474 | diff_info="${info}" 475 | fi 476 | done 477 | # sort by UNIQ 478 | echo -en "$diff_info" | sort -t '|' -nk 2,2 479 | echo -e "${div_line}\n${title}\n${head_line}" 480 | else 481 | # get/set the config value 482 | local title=$(printf " %-22s| %s" "REDIS" "$CONFIG_KEY") 483 | local last_print="" 484 | if [[ "$CONFIG_SET" = "yes" ]]; then 485 | debug "SET the value of '$CONFIG_KEY' on \033[37m${ROLE_TYPE:-all}\033[0m nodes:" 486 | # print at last for config set 487 | last_print="${head_line}\n${title}\n${div_line}\n" 488 | else 489 | debug "GET the value of '$CONFIG_KEY' on \033[37m${ROLE_TYPE:-all}\033[0m nodes:" 490 | # print immediatly for config get 491 | echo -en "${head_line}\n${title}\n${div_line}\n" 492 | fi 493 | 494 | # Loop nodes to process 495 | local old_pass="$REDIS_PASS" 496 | for node in $(echo "$CLUSTER_NODES_ALL"|grep "$ROLE_TYPE"|grep -v noaddr|awk '{print $4}'|sort); do 497 | # use the origin password 498 | REDIS_PASS="$old_pass" 499 | set_rediscli_auth 500 | # ip & port 501 | local node_ip=${node%:*} 502 | local node_port=${node##*:} 503 | if [ "$node_ip" = "" ]; then 504 | node_ip="127.0.0.1" 505 | fi 506 | 507 | # CONFIG SET 508 | if [[ "$CONFIG_SET" = "yes" ]]; then 509 | conf_res=$(redis-cli -h ${node_ip} -p ${node_port} ${AUTH} config set "$CONFIG_KEY" "$CONFIG_VALUE") 510 | # update the new password 511 | if [[ "$CONFIG_KEY" = "requirepass" && "$conf_res" = "OK" ]]; then 512 | REDIS_PASS="$CONFIG_VALUE" 513 | set_rediscli_auth 514 | fi 515 | # CONFIG REWRITE 516 | if [[ $WRITE_CONFIG = "yes" && "$conf_res" = "OK" ]]; then 517 | conf_res="write config file: "$(redis-cli -h ${node_ip} -p ${node_port} ${AUTH} config rewrite 2>&1) 518 | fi 519 | debug "[${node_ip}:${node_port}]: CONFIG SET $CONFIG_KEY '$CONFIG_VALUE' : [$conf_res]" 520 | fi 521 | 522 | # CONFIG GET 523 | conf_res=$(redis-cli -h ${node_ip} -p ${node_port} ${AUTH} --csv config get "$CONFIG_KEY" 2>&1 | awk -F',' '{print $2}') 524 | conf_res=$(printf " %-22s| %s" "${node_ip}:${node_port}" "$conf_res") 525 | if [[ "$CONFIG_SET" = "yes" ]]; then 526 | # print at last for config set 527 | last_print="${last_print}${conf_res}\n" 528 | else 529 | # print immediatly for config get 530 | echo -e "$conf_res" 531 | fi 532 | done 533 | 534 | if [[ "$CONFIG_SET" = "yes" ]]; then 535 | # print the new value after config set 536 | debug "Get the value of '$CONFIG_KEY' on \033[37m${ROLE_TYPE:-all}\033[0m nodes after CONFIG SET:" 537 | echo -en "${last_print}" 538 | fi 539 | echo -e "${head_line}" 540 | fi 541 | } 542 | 543 | ####################################### 544 | # get nodes info by ip 545 | # Globals: 546 | # CLUSTER_NODES_ALL - cluster nodes list 547 | # REDIS_HOST - redis ip 548 | # NODES_PREFIX - .RCTOOL-NODES- 549 | # INFOS_PREFIX - .RCTOOL-INFOS- 550 | # MYPID - pid 551 | # Arguments: 552 | # 1 - ip 553 | # Returns: 554 | # None 555 | ####################################### 556 | function subprocess_nodes_by_ip() { 557 | local deal_ip=$1 558 | # init result file 559 | local ret_file="${NODES_PREFIX}${deal_ip}.${MYPID}" 560 | >$ret_file 561 | local info_file="${INFOS_PREFIX}${deal_ip}.${MYPID}" 562 | >$info_file 563 | 564 | declare -A map_nodes 565 | OLDIFS=$IFS 566 | IFS=$'\n' 567 | local master_addr="" 568 | for node in $(echo "$CLUSTER_NODES_ALL"); do 569 | IFS=$OLDIFS 570 | # 1.masterID 2.role 3.ID 4.ip:port 5.flags 6.masterID 7.pingtime 8.pongtime 9.configepoch 10.connect 11.slots 571 | local node_addr=$(echo "$node"|awk '{print $4}') 572 | local no_addr=$(echo "$node"|grep -w noaddr|wc -l) 573 | if [[ $no_addr -eq 1 ]]; then 574 | node_addr="noaddr$node_addr" 575 | fi 576 | local role="slave" 577 | local is_master=$(echo "$node"|grep -w master|wc -l) 578 | if [[ $is_master -eq 1 ]]; then 579 | master_addr="$node_addr" 580 | role="master" 581 | fi 582 | 583 | local node_ip=${node_addr%:*} 584 | local node_port=${node_addr##*:} 585 | if [ "$deal_ip" != "" ]; then 586 | if [ "$node_ip" != "$deal_ip" ]; then 587 | continue 588 | fi 589 | else 590 | if [ "$node_ip" != "noaddr" -a "$node_ip" != "" ]; then 591 | continue 592 | fi 593 | fi 594 | if [ "$node_ip" = "" ]; then 595 | node_ip=${REDIS_HOST} 596 | fi 597 | 598 | # ELAPSED: time2 - time1 599 | # $ echo -en "time\r\ntime\r\n"|redis-cli -h 172.16.17.3 -p 7290 --raw 600 | # 1582680185 601 | # 906676 602 | # 1582680185 603 | # 906940 604 | local ustime1=0 605 | local ustime2=0 606 | local elapsed="-" 607 | local status="FAIL" 608 | local total_time=$(echo -en "time\r\ntime\r\n" | redis-cli -h ${node_ip} -p ${node_port} ${AUTH} --raw 2>/dev/null) 609 | if [ "$total_time" != "" ]; then 610 | if [ "${total_time:0:8}" == "LOADING " ];then 611 | status="LOAD" 612 | elif [ "${total_time:0:7}" == "NOAUTH " ];then 613 | status="NOAUTH" 614 | else 615 | status="OK" 616 | ustime1=$(echo $total_time|awk '{printf("%.f", $1*1000000+$2)}') 617 | ustime2=$(echo $total_time|awk '{printf("%.f", $3*1000000+$4)}') 618 | elapsed=$(expr $ustime2 - $ustime1) 619 | fi 620 | fi 621 | 622 | # INFO 623 | local node_info="" 624 | local version="-" 625 | local startime="-" 626 | local usemem="-" 627 | local usemem_byte="-" 628 | local ops="-" 629 | local clients="-" 630 | local keynum="-" 631 | # cpu 632 | local ustime="-" 633 | local cpu="-" 634 | 635 | if [ "$status" == "OK" -o "$status" == "LOAD" ]; then 636 | ustime=$(date +'%s %N'|awk '{printf("%.f", $1*1000000+$2/1000)}') 637 | node_info=$(redis-cli -h ${node_ip} -p ${node_port} ${AUTH} info 2>/dev/null) 638 | if [ "$node_info" != "" -a "${node_info:0:7}" != "NOAUTH " ]; then 639 | version=$(echo "$node_info"|awk -F":|-|\r" '{if($1=="redis_version"){print $(NF-1)}}') 640 | startime=$(echo "$node_info"|awk -F":|\r" '{if($1=="uptime_in_seconds"){print $2}}') 641 | usemem=$(echo "$node_info"|awk -F":|\r" '{if($1=="used_memory_human"){print $2}}') 642 | usemem_byte=$(echo "$node_info"|awk -F":|\r" '{if($1=="used_memory"){print $2}}') 643 | ops=$(echo "$node_info"|awk -F":|\r" '{if($1=="instantaneous_ops_per_sec"){print $2}}') 644 | clients=$(echo "$node_info"|awk -F":|\r" '{if($1=="connected_clients"){print $2}}') 645 | # db0:keys=5001,expires=0,avg_ttl=0 646 | keynum=$(echo "$node_info"|awk -F":|=|,|\r" '{if($1 ~ /^db[0-9]*/){keys+=$3}} END {print keys}') 647 | if [ "$keynum" = "" ]; then 648 | keynum="0" 649 | fi 650 | # cpu info 651 | cpu=0 652 | for N in $(echo "$node_info"|grep -E 'used_cpu_sys:|used_cpu_user:'|awk -F":" '{printf("%.f\n", $2*1000000)}'); 653 | do 654 | cpu=$(($cpu + $N)) 655 | done 656 | fi 657 | fi 658 | local node_id=$(echo "$node"|awk '{print $3}') 659 | # 1.masteraddr 2.role 3.status 4.version 5.uptime 6.keys 7.usemem 8.client 9.ops 10.rtt 11.usemem_byte 12.ustime 13.cpu 660 | map_nodes[$node_id]="$master_addr ${role} ${status} $version $startime $keynum $usemem $clients $ops $elapsed $usemem_byte $ustime $cpu " 661 | echo "1" >>$ret_file 662 | done 663 | # sleep for a while 664 | sleep 0.5 665 | # calc the %cpu 666 | IFS=$'\n' 667 | for node in $(echo "$CLUSTER_NODES_ALL"); do 668 | IFS=$OLDIFS 669 | # 1.masterID 2.role 3.ID 4.ip:port 5.flags 6.masterID 7.pingtime 8.pongtime 9.configepoch 10.connect 11.slots 670 | local node_addr=$(echo "$node"|awk '{print $4}') 671 | local no_addr=$(echo "$node"|grep -w noaddr|wc -l) 672 | if [[ $no_addr -eq 1 ]]; then 673 | node_addr="noaddr$node_addr" 674 | fi 675 | local node_ip=${node_addr%:*} 676 | local node_port=${node_addr##*:} 677 | if [ "$deal_ip" != "" ]; then 678 | if [ "$node_ip" != "$deal_ip" ]; then 679 | continue 680 | fi 681 | else 682 | if [ "$node_ip" != "noaddr" -a "$node_ip" != "" ]; then 683 | continue 684 | fi 685 | fi 686 | if [ "$node_ip" = "" ]; then 687 | node_ip=${REDIS_HOST} 688 | fi 689 | # 1.masteraddr 2.role 3.status 4.version 5.uptime 6.keys 7.usemem 8.client 9.ops 10.rtt 11.usemem_byte 12.ustime 13.cpu 14.%cpu 690 | local node_id=$(echo "$node"|awk '{print $3}') 691 | local node_res="${map_nodes[$node_id]}" 692 | local uslast=$(echo "$node_res"|awk '{print $12}') 693 | local cpulast=$(echo "$node_res"|awk '{print $13}') 694 | if [ "$cpulast" = "-" ];then 695 | echo "$node_res - ${node}" >>$info_file 696 | echo "1" >>$ret_file 697 | continue 698 | fi 699 | local incrcpu="" 700 | local ustime=$(date +'%s %N'|awk '{printf("%.f", $1*1000000+$2/1000)}') 701 | local cpuinfo=$(redis-cli -h ${node_ip} -p ${node_port} ${AUTH} info cpu 2>/dev/null) 702 | if [ "$cpuinfo" != "" -a "${cpuinfo:0:7}" != "NOAUTH " ]; then 703 | local cpu=0 704 | for N in $(echo "$cpuinfo"|grep -E 'used_cpu_sys:|used_cpu_user:'|awk -F":" '{printf("%.f\n", $2*1000000)}'); 705 | do 706 | cpu=$(($cpu + $N)) 707 | done 708 | incrcpu=$(echo "$cpu $cpulast $ustime $uslast"|awk '{if($1>=$2){printf("%.1f",($1-$2)*100/($3-$4)); }else{printf("%.1f",$1*100/($3-$4));}}') 709 | else 710 | incrcpu="-" 711 | fi 712 | echo "$node_res $incrcpu ${node}" >>$info_file 713 | echo "1" >>$ret_file 714 | done 715 | IFS=$OLDIFS 716 | } 717 | 718 | ####################################### 719 | # show cluster nodes status 720 | # Arguments: 721 | # 1 - nodes info list 722 | # Returns: 723 | # cluster nodes status 724 | ####################################### 725 | function show_cluster_status() { 726 | # show cluster nodes 727 | # SERVER STATUS ROLE VERSION UPTIME(s) KEYS USEMEM %CPU CLIENTS OPS RTT(us) SLOTS 728 | awk_cmd=' 729 | BEGIN { 730 | if(coltime <= 0) coltime=systime(); 731 | printf("================================================== %s ==================================================\n", strftime("%F %T", coltime)); 732 | printf("%-24s %-6s %-7s %-8s %-9s %-9s %-8s %-6s %-7s %-6s %-9s %-s \n", 733 | "SERVER","STATUS","ROLE","VERSION","UPTIME(s)","KEYS","USEMEM","%CPU","CLIENTS","OPS","RTT(us)","SLOTS"); 734 | print "-------------------------------------------------------------------------------------------------------------------------" 735 | } 736 | { 737 | mem=substr($8,1,length($8)-1); 738 | unit=substr($8,length($8)); 739 | if(unit=="G") { 740 | mem=mem*1073741824 741 | } else if(unit=="M") { 742 | mem=mem*1048576 743 | } else if(unit=="K") { 744 | mem=mem*1024 745 | } 746 | allkey+=$7; 747 | allmem+=mem; 748 | allcli+=$10; 749 | allops+=$11; 750 | allcpu+=$9; 751 | allnode+=1; 752 | if($4~/master/) { 753 | masterkey+=$7; 754 | mastermem+=mem; 755 | mastercli+=$10; 756 | masterops+=$11; 757 | mastercpu+=$9; 758 | masters+=1; 759 | } 760 | if($3=="OK") { 761 | oknum+=1; 762 | }else if($3=="FAIL") { 763 | failnum+=1; 764 | }else if($3=="NOAUTH") { 765 | noauth+=1; 766 | }else if($3=="LOAD") { 767 | load+=1; 768 | } 769 | 770 | len=length($2); 771 | if(len>21){ 772 | addr=substr($2,0,14)".."substr($2,len-4,5); 773 | } else { 774 | addr=$2; 775 | } 776 | match($2,/:[0-9]+$/); 777 | masterip=substr($2,1,RSTART-1); 778 | if($4~/master/) { 779 | printf("%-24s ", addr); 780 | mapip[masterip]+=1; 781 | } else { 782 | printf(" |-%-21s ", addr); 783 | mapip[masterip]+=0; 784 | } 785 | printf("%-6s %-7s %-8s %-9s %-9s %-8s %-6s %-7s %-6s %-9s",$3,$4,$5,$6,$7,$8,$9,$10,$11,$12); 786 | for(i=13;i<=NF;i++) { 787 | printf(" %s",$i); 788 | } 789 | printf("\n"); 790 | } END { 791 | print "_________________________________________________________________________________________________________________________" 792 | if(allmem<1048576){ 793 | allmem=allmem/1024 794 | allmem=sprintf("%.1fK",allmem) 795 | }else if(allmem<1073741824){ 796 | allmem=allmem/1048576 797 | allmem=sprintf("%.1fM",allmem) 798 | }else{ 799 | allmem=allmem/1073741824 800 | allmem=sprintf("%.1fG",allmem) 801 | } 802 | if(mastermem<1048576){ 803 | mastermem=mastermem/1024 804 | mastermem=sprintf("%.1fK",mastermem) 805 | }else if(mastermem<1073741824){ 806 | mastermem=mastermem/1048576 807 | mastermem=sprintf("%.1fM",mastermem) 808 | }else{ 809 | mastermem=mastermem/1073741824 810 | mastermem=sprintf("%.1fG",mastermem) 811 | } 812 | printf("SUM(master/all): NODE=%s/%s KEY=%s/%s MEM=%s/%s CLI=%s/%s OPS=%s/%s CPU=%s/%s\n", 813 | masters,allnode,masterkey,allkey,mastermem,allmem,mastercli,allcli,masterops,allops,mastercpu,allcpu); 814 | if(noauth>0) { 815 | printf("\033[41;37m!!!!! Access redis for \"NOAUTH\": Please use the option \"-a \" !!!!!\033[0m\n"); 816 | } 817 | if(load>0) { 818 | printf("\033[43;37m!!!!! Please wait the status is \"LOAD\": The nodes are loading data !!!!!\033[0m\n"); 819 | } 820 | if(failnum>0) { 821 | printf("\033[41;37m!!!!! Please check the status is \"FAIL\": The nodes maybe offline !!!!!\033[0m\n"); 822 | } 823 | minnum=masters 824 | maxnum=0 825 | for(ip in mapip) { 826 | if(minnum > mapip[ip]) { 827 | minnum=mapip[ip]; 828 | } 829 | if(maxnum < mapip[ip]) { 830 | maxnum=mapip[ip]; 831 | } 832 | } 833 | if(minnum != maxnum && maxnum > 1) { 834 | printf("\033[41;37m!!!!! Masters unbalance on hosts: min/max = %s/%s !!!!!\033[0m\n", minnum,maxnum); 835 | } 836 | print "=========================================================================================================================" 837 | }' 838 | 839 | # compact format 840 | # 1,SERVER,STATUS,ROLE,VERSION,UPTIME,KEYS,USEMEM,%CPU,CLIENTS,OPS,DELAY,SLOTS& 841 | echo "$1" | tr '&' '\n' | awk -F',' -v coltime=${2:-0} "$awk_cmd" 842 | } 843 | 844 | ####################################### 845 | # get cluster nodes status 846 | # Globals: 847 | # CLUSTER_NODES_ALL - cluster nodes list 848 | # INFOS_PREFIX - .RCTOOL-INFOS- 849 | # MYPID - pid 850 | # OUTPUT_RAW_DATA - print raw data 851 | # Arguments: 852 | # 1 - nodes info list 853 | # Returns: 854 | # cluster nodes status 855 | ####################################### 856 | function get_cluster_status() { 857 | # CLUSTER_NODES_ALL: 858 | # 1.masterID 2.role 3.ID 4.ip:port 5.flags 6.masterID 7.pingtime 8.pongtime 9.configepoch 10.connect 11.slots 859 | get_cluster_nodes 860 | 861 | # get all nodes info by ip 862 | local last_ip="Nil" 863 | for node in $(echo "$CLUSTER_NODES_ALL"|awk '{print $4}'|sort); do 864 | # ip 865 | local ip=${node%:*} 866 | if [ "$last_ip" != "$ip" ]; then 867 | last_ip=$ip 868 | { 869 | subprocess_nodes_by_ip $last_ip 870 | }& 871 | fi 872 | done 873 | # show progress: every node need to access twice in order to calc %cpu 874 | show_progress "${CLUSTER_NODES_ALL}\n${CLUSTER_NODES_ALL}" 875 | # wait subprocess done 876 | wait 877 | 878 | # 14: masteraddr role status version startime keynum usemem clients ops elapsed usemembyte ustime cpu %cpu 879 | # 11: masterID role ID ip:port flags masterID pingtime pongtime configepoch connect slots 880 | local nodes_res=$(cat ${INFOS_PREFIX}*.${MYPID} | sort) 881 | # compact format 882 | # 1,SERVER,STATUS,ROLE,VERSION,UPTIME,KEYS,USEMEM,CPU,CLIENTS,OPS,DELAY,SLOTS& 883 | local awk_cmd=' 884 | { 885 | printf("1,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s",$18,$3,$2,$4,$5,$6,$7,$14,$8,$9,$10); 886 | if(NF > 24) { 887 | for(i=25;i<=NF;i++) { 888 | if(i==25) { 889 | printf(",%s",$i); 890 | } else { 891 | printf(" %s",$i); 892 | } 893 | } 894 | } 895 | printf("&"); 896 | }' 897 | local compres=$(echo "$nodes_res" | awk "$awk_cmd") 898 | # delete the last '&' 899 | compres=${compres:0:-1} 900 | # echo "$compres" 901 | if [ "$OUTPUT_RAW_DATA" = "yes" ]; then 902 | echo "$compres" 903 | else 904 | show_cluster_status "$compres" 905 | fi 906 | } 907 | 908 | ####################################### 909 | # get commands info by ip 910 | # Globals: 911 | # MASTER_LIST - cluster nodes list 912 | # REDIS_HOST - redis ip 913 | # MONI_PREFIX - .RCTOOL-MONI- 914 | # MYPID - pid 915 | # AUTH - auth info 916 | # Arguments: 917 | # 1 - ip 918 | # Returns: 919 | # commands info 920 | ####################################### 921 | function subprocess_monitor_cmds_by_ip() { 922 | local deal_ip=$1 923 | # init result file 924 | local ret_file="${MONI_PREFIX}${deal_ip}.${MYPID}" 925 | >$ret_file 926 | for node_addr in $(echo "$MASTER_LIST"|grep -w $deal_ip); do 927 | # ADDR 928 | local node_ip=${node_addr%:*} 929 | local node_port=${node_addr##*:} 930 | if [ "$node_ip" = "127.0.0.1" -o "$node_ip" = "" ]; then 931 | node_ip=${REDIS_HOST} 932 | fi 933 | 934 | # cmdstat_get:calls=1,usec=1,usec_per_call=1.00 935 | echo -en 'info stats\ninfo commandstats\ninfo cpu\n'|redis-cli -h ${node_ip} -p ${node_port} ${AUTH} 2>/dev/null >>$ret_file 936 | if [ $? -ne 0 ]; then 937 | warn "Can't connect to redis(${node_ip}:${node_port})!\n" 938 | fi 939 | done 940 | # delete character '\r' 941 | sed -i "s/\r//" ${ret_file} 942 | } 943 | 944 | ####################################### 945 | # rolling stats commands of masters 946 | # Globals: 947 | # MASTER_LIST - cluster nodes list 948 | # REDIS_HOST - redis ip 949 | # REDIS_PORT - redis port 950 | # SINGLE_NODE - stat the specify node only 951 | # CMDS_PREFIX - .RCTOOL-CMDS- 952 | # MYPID - pid 953 | # MONI_COUNT - run times 954 | # MONI_LATENCY - show command average latency 955 | # Arguments: 956 | # None 957 | # Returns: 958 | # commands stats 959 | ####################################### 960 | function moni_commands() { 961 | # CLUSTER or SINGLE 962 | if [ "$SINGLE_NODE" != "yes" ]; then 963 | # MASTER_LIST 964 | get_cluster_nodes 965 | local node_num=$(echo "$MASTER_LIST"|wc -l) 966 | debug "[CLUSTER] monitor with $node_num masters from ${REDIS_HOST}:${REDIS_PORT} every $ROLL_TIME seconds" 967 | else 968 | check_redis_by_ping 969 | MASTER_LIST="${REDIS_HOST}:${REDIS_PORT}" 970 | debug "[SINGLE] monitor only the node ${REDIS_HOST}:${REDIS_PORT} every $ROLL_TIME seconds" 971 | fi 972 | 973 | # cmdstat_get:calls=1,usec=1,usec_per_call=1.00 974 | local awk_cmd=' 975 | { 976 | if($1~/cmdstat_/) { 977 | cmdname=substr($1,9); 978 | if(lastcmd=="") { 979 | lastcmd=cmdname; 980 | } 981 | if(lastcmd!=cmdname) { 982 | printf("%s %s %s\n",lastcmd,cmdcalls,cmdusec); 983 | lastcmd=cmdname; 984 | cmdcalls=0; 985 | cmdusec=0; 986 | } 987 | cmdcalls+=$3; 988 | cmdusec+=$5; 989 | } 990 | } END {if(lastcmd!=""){printf("%s %s %s\n",lastcmd,cmdcalls,cmdusec);}}' 991 | 992 | # rolling stats ... 993 | OLDIFS=$IFS 994 | local run_count=0 995 | local base_cmd_stat="" 996 | local last_cmd_stat="" 997 | local last_time=0 998 | local last_total=0 999 | local last_cpu=0 1000 | local last_net_input=0 1001 | local last_net_output=0 1002 | local last_conn_total=0 1003 | local ary_top_cmd_name=() 1004 | while true; do 1005 | # exec 'info' command on all masters 1006 | local last_ip="Nil" 1007 | for node in $(echo "$MASTER_LIST"|sort); do 1008 | # ip 1009 | local ip=${node%:*} 1010 | if [ "$last_ip" != "$ip" ]; then 1011 | last_ip=$ip 1012 | { 1013 | subprocess_monitor_cmds_by_ip $last_ip 1014 | }& 1015 | fi 1016 | done 1017 | wait 1018 | 1019 | # current time 1020 | local curr_time=$(date +'%s %N'|awk '{printf("%.f", $1*1000000+$2/1000)}') 1021 | 1022 | # commands info 1023 | local cmd_stat=$(cat ${MONI_PREFIX}*.${MYPID} 2>/dev/null |grep cmdstat_| sort | awk -F"=|,|:" "$awk_cmd") 1024 | # summary commands 1025 | local curr_total=$(echo -e "$cmd_stat"|awk '{calls+=$2;} END {print calls;}') 1026 | 1027 | # cpu 1028 | local curr_cpu=0 1029 | for N in $(cat ${MONI_PREFIX}*.${MYPID} 2>/dev/null |grep -E 'used_cpu_sys:|used_cpu_user:'|awk -F":" '{printf("%.f\n", $2*1000000)}'); 1030 | do 1031 | curr_cpu=$(($curr_cpu + $N)) 1032 | done 1033 | 1034 | # don't use awk for high precision 1035 | local curr_net_input=0 1036 | for N in $(cat ${MONI_PREFIX}*.${MYPID} 2>/dev/null |grep total_net_input_bytes|awk -F":" '{print $2}'); 1037 | do 1038 | curr_net_input=$(($curr_net_input + $N)) 1039 | done 1040 | curr_net_input=$(expr $curr_net_input / 1024) 1041 | 1042 | # don't use awk for high precision 1043 | local curr_net_output=0 1044 | for N in $(cat ${MONI_PREFIX}*.${MYPID} 2>/dev/null |grep total_net_output_bytes|awk -F":" '{print $2}'); 1045 | do 1046 | curr_net_output=$(($curr_net_output + $N)) 1047 | done 1048 | curr_net_output=$(expr $curr_net_output / 1024) 1049 | 1050 | # connections 1051 | local curr_conn_total=0 1052 | for N in $(cat ${MONI_PREFIX}*.${MYPID} 2>/dev/null |grep total_connections_received|awk -F":" '{print $2}'); 1053 | do 1054 | curr_conn_total=$(($curr_conn_total + $N)) 1055 | done 1056 | 1057 | # show the TITLE: update the TOP10 command 1058 | if [[ $((run_count%20)) -eq 1 ]]; then 1059 | # Get the increment of commands 1060 | local cmd_incr="" 1061 | # calc the increment 1062 | IFS=$'\n' 1063 | for cmd_curr in $(echo "$cmd_stat"); do 1064 | IFS=$OLDIFS 1065 | local curr_info=(${cmd_curr}) 1066 | local last_info=($(echo "$base_cmd_stat"|grep -w ${curr_info[0]})) 1067 | 1068 | cmd_incr+=${curr_info[0]} 1069 | if [ "${last_info[1]}" = "" ]; then 1070 | cmd_incr+=" ${curr_info[1]} ${curr_info[2]}" 1071 | elif [[ ${last_info[1]} -gt ${curr_info[1]} ]]; then 1072 | cmd_incr+=" ${curr_info[1]} ${curr_info[2]}" 1073 | else 1074 | cmd_incr+=" $[curr_info[1]-last_info[1]] $[curr_info[2]-last_info[2]]" 1075 | fi 1076 | cmd_incr+="\n" 1077 | done 1078 | IFS=$OLDIFS 1079 | 1080 | # array of TOP10 commands 1081 | ary_top_cmd_name=() 1082 | local i=0 1083 | IFS=$'\n' 1084 | for cmd_curr in $(echo -e "$cmd_incr"|sort -rnk 2); do 1085 | if [[ $i -lt 10 ]]; then 1086 | ary_top_cmd_name[$i]=${cmd_curr%% *} 1087 | else 1088 | break 1089 | fi 1090 | i=$((i+1)) 1091 | done 1092 | IFS=$OLDIFS 1093 | 1094 | # show the TITLE: --TOP10 commands-- TOTAL net-input net-output 1095 | local title_info="" 1096 | for ((row=0;row<${#ary_top_cmd_name[@]};row++));do 1097 | title_info+=$(printf "%-7s " ${ary_top_cmd_name[$row]}) 1098 | done 1099 | local width=$((${#title_info}-1)) 1100 | local title_head=$(draw_divide_line $width "-" "TOP${#ary_top_cmd_name[@]} commands $(date +'%F %T')" "") 1101 | # length: 8-11-11 1102 | title_head+=" -----all------ -----------netio-----" 1103 | title_info+="TOTAL %CPU CONNS IN(KB) OUT(KB)" 1104 | echo -e "$title_head\n$title_info" 1105 | # save the base cmd_stat 1106 | base_cmd_stat="$cmd_stat" 1107 | fi 1108 | if [[ $run_count -eq 0 ]]; then 1109 | base_cmd_stat="$cmd_stat" 1110 | fi 1111 | 1112 | # show command stats 1113 | if [ "$last_cmd_stat" != "" ]; then 1114 | local output_value="" 1115 | local output_cost="" 1116 | # TOP10 commands 1117 | for ((row=0;row<${#ary_top_cmd_name[@]};row++));do 1118 | local len=7 1119 | local cmd_name=${ary_top_cmd_name[$row]} 1120 | if [[ ${#cmd_name} -gt $len ]]; then 1121 | len=${#cmd_name} 1122 | fi 1123 | # calc the avg cost of the command 1124 | local curr_cost=$(echo -e "$cmd_stat"|grep -w ${cmd_name}|awk '{print $3}') 1125 | local last_cost=$(echo -e "$last_cmd_stat"|grep -w ${cmd_name}|awk '{print $3}') 1126 | # calc the increment of the command 1127 | local curr_value=$(echo -e "$cmd_stat"|grep -w ${cmd_name}|awk '{print $2}') 1128 | local last_value=$(echo -e "$last_cmd_stat"|grep -w ${cmd_name}|awk '{print $2}') 1129 | local cmd_value="" 1130 | local cmd_cost="" 1131 | if [ "$last_value" = "" ];then 1132 | cmd_value=$curr_value 1133 | cmd_cost=$curr_cost 1134 | elif [ "$curr_value" = "" ]; then 1135 | cmd_value="0" 1136 | cmd_cost="0" 1137 | elif [[ ${last_value} -gt ${curr_value} ]]; then 1138 | cmd_value=$curr_value 1139 | cmd_cost=$curr_cost 1140 | else 1141 | cmd_value=$((curr_value-last_value)) 1142 | cmd_cost=$((curr_cost-last_cost)) 1143 | fi 1144 | 1145 | if [ "$cmd_value" = "" ];then 1146 | cmd_value="0" 1147 | fi 1148 | output_value+=$(printf "%-${len}s " "$cmd_value") 1149 | 1150 | if [ "$cmd_value" = "0" ];then 1151 | cmdavgcost="0" 1152 | else 1153 | cmdavgcost=$[cmd_cost/cmd_value] 1154 | fi 1155 | output_cost+=$(printf "*%-$[len-1]s " "$cmdavgcost") 1156 | done 1157 | 1158 | # TOTAL commands 1159 | local value=$(echo "$curr_total $last_total"|awk '{if($1>=$2){print $1-$2}else{print $1}}') 1160 | output_value+=$(printf "%-7s " "$value") 1161 | 1162 | # %CPU 1163 | value=$(echo "$curr_cpu $last_cpu $curr_time $last_time"|awk '{if($1>=$2){printf("%-6.1f ",($1-$2)*100/($3-$4)); }else{printf("%-6.1f ",$1*100/($3-$4));}}') 1164 | output_value+="$value" 1165 | 1166 | # CONNECTIONs add 1167 | value=$(echo "$curr_conn_total $last_conn_total"|awk '{if($1>=$2){print $1-$2}else{print $1}}') 1168 | output_value+=$(printf "%-5s " "$value") 1169 | 1170 | # NET input(KB) 1171 | value=$(echo "$curr_net_input $last_net_input"|awk '{if($1>=$2){print $1-$2}else{print $1}}') 1172 | output_value+=$(printf "%-7s " "$value") 1173 | 1174 | # NET output(KB) 1175 | value=$(echo "$curr_net_output $last_net_output"|awk '{if($1>=$2){print $1-$2}else{print $1}}') 1176 | output_value+=$(printf "%-7s\n" "$value") 1177 | 1178 | echo "$output_value" 1179 | if [ "$MONI_LATENCY" = "yes" ];then 1180 | echo "$output_cost" 1181 | fi 1182 | fi 1183 | 1184 | # save the last cmd_stat 1185 | last_cmd_stat="$cmd_stat" 1186 | last_total=$curr_total 1187 | last_net_input=$curr_net_input 1188 | last_net_output=$curr_net_output 1189 | last_conn_total=$curr_conn_total 1190 | # last cpu 1191 | last_cpu=$curr_cpu 1192 | last_time=$curr_time 1193 | 1194 | if [[ $MONI_COUNT -gt 0 && $run_count -ge $MONI_COUNT ]]; then 1195 | break 1196 | fi 1197 | # delay for a while 1198 | sleep $ROLL_TIME 1199 | run_count=$((run_count+1)) 1200 | done 1201 | } 1202 | 1203 | ####################################### 1204 | # trace command on the sepcify node by 'monitor' 1205 | # Globals: 1206 | # INPUT_FILE - file with 'monitor' result 1207 | # REDIS_HOST - redis ip 1208 | # REDIS_PORT - redis port 1209 | # CMDS_PREFIX - .RCTOOL-CMDS- 1210 | # MYPID - pid 1211 | # AUTH - auth info 1212 | # TRACE_TIME - running time 1213 | # TRACE_HOT - analyze the hot keys 1214 | # TRACE_IP - analyze command by client IP 1215 | # Arguments: 1216 | # 1 - ip 1217 | # Returns: 1218 | # commands info 1219 | ####################################### 1220 | function trace_commands() { 1221 | # monitor result: 1222 | # 1630548595.516774 [0 172.16.17.3:47321] "HGETALL" "REDIS.DBANALYSE.TASK-172.16.17.3" 1223 | # 1645499579.040001 [0 10.18.130.80:42709] "EVAL" "if redis.call('get',KEYS[1]) == ARGV[1] then redis.call('del',KEYS[1]) return 1 else return 0 end" "1" "LOCK_NEW_54_1" "REQUEST_ID_54" 1224 | # 1645499579.040022 [0 lua] "exists" "LOCK_NEW_54_1" 1225 | # 1645499579.040027 [0 lua] "get" "LOCK_NEW_54_1" 1226 | # 1645499579.040033 [0 lua] "del" "LOCK_NEW_54_1" 1227 | 1228 | # stat commands every second: order by command name 1229 | local awk_cmd_stat=' 1230 | BEGIN { 1231 | #PROCINFO["sorted_in"] = "@ind_str_asc" 1232 | PROCINFO["sorted_in"] = "@val_num_desc" 1233 | } 1234 | { 1235 | currsec=substr($1,1,10) 1236 | if(currsec!=lastsec) { 1237 | count=0; 1238 | for(idx in arr) { 1239 | msg=msg" "idx":"arr[idx]; 1240 | count+=arr[idx]; 1241 | } 1242 | if(msg!="") print strftime("%T", lastsec)"> "count""msg; 1243 | msg=""; 1244 | lastsec=currsec; 1245 | delete arr; 1246 | arr[toupper($4)]=1; 1247 | } else { 1248 | ++arr[toupper($4)]; 1249 | } 1250 | } END { 1251 | count=0; 1252 | for(idx in arr) { 1253 | msg=msg" "idx":"arr[idx]; 1254 | count+=arr[idx]; 1255 | } 1256 | print strftime("%T", lastsec)"> "count""msg; 1257 | }' 1258 | 1259 | if [ "${INPUT_FILE}" != "" ]; then 1260 | # process command by input file 1261 | if [ ! -f "${INPUT_FILE}" ]; then 1262 | warn "Can't load the monitor file: ${INPUT_FILE}!" 1263 | exit 1 1264 | fi 1265 | 1266 | if [ "${TRACE_HOT}" = "yes" ]; then 1267 | debug "Analyze the monitor file ${INPUT_FILE} for hot keys ..." 1268 | # ignore 'EVAL' command: will get KEY from lua subcommand 1269 | ( echo "================================================== ======= ======="; 1270 | echo "KEY COUNT PERCENT"; 1271 | echo "-------------------------------------------------- ------- -------"; 1272 | (tail -c +4 ${INPUT_FILE} | grep -iv '"EVAL"' | awk ' 1273 | { 1274 | arrTmp[$5] += 1; 1275 | arrKey[$5] += 1; 1276 | count += 1; 1277 | } END { 1278 | len=asort(arrTmp,arrNum) 1279 | if(len>=100) { 1280 | top=arrNum[len-100] 1281 | } else { 1282 | top=arrNum[1] 1283 | } 1284 | for(i in arrKey) { 1285 | if(arrKey[i]>=top){ 1286 | if(i!=""){ 1287 | printf("%s %s %0.2f%\n", i, arrKey[i], arrKey[i]*100/count) 1288 | } else { 1289 | printf("NO-KEY %s %0.2f%\n", arrKey[i], arrKey[i]*100/count) 1290 | } 1291 | } 1292 | } 1293 | }' | sort -k2n,2) 1294 | ) | column -t 1295 | elif [ "${TRACE_IP}" = "yes" ]; then 1296 | debug "Analyze the monitor file ${INPUT_FILE} for client host ..." 1297 | # ignore lua subcommand 1298 | ( echo "========== =============== ======="; 1299 | echo "COMMAND CLIENTIP COUNT"; 1300 | echo "---------- --------------- -------"; 1301 | (tail -c +4 ${INPUT_FILE} | awk -F'[: ]' ' 1302 | { 1303 | if($3!="lua]") arr[$5,$3] += 1; 1304 | } END { 1305 | for (k in arr) { 1306 | split(k,idx,SUBSEP); 1307 | printf("%s %s %s\n",idx[1], idx[2],arr[idx[1],idx[2]]); 1308 | } 1309 | }' | sort -k1,1 -k3n,3) 1310 | ) | column -t 1311 | else 1312 | debug "Analyze the monitor file ${INPUT_FILE} on seconds ..." 1313 | echo "--------------------------------------------------------------------------------" 1314 | tail -c +4 ${INPUT_FILE} | awk "$awk_cmd_stat" 1315 | fi 1316 | else 1317 | # check redis status 1318 | check_redis_by_ping 1319 | 1320 | INPUT_FILE="${REDIS_HOST}-${REDIS_PORT}.mon" 1321 | rm -rf ${INPUT_FILE} 1322 | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} $AUTH monitor >${INPUT_FILE} & 1323 | local cli_pid=$! 1324 | local cli_exist=$(ps -p $cli_pid | grep redis-cli |wc -l) 1325 | if [ "$cli_exist" = "0" ]; then 1326 | warn "Can't connect to the node ${REDIS_HOST}:${REDIS_PORT}!" 1327 | exit 1 1328 | fi 1329 | 1330 | debug "Analyze the command on seconds for the node ${REDIS_HOST}:${REDIS_PORT} ..." 1331 | echo "--------------------------------------------------------------------------------" 1332 | # wait for ${INPUT_FILE} created 1333 | for i in {1..50} 1334 | do 1335 | sleep 0.1 1336 | if [ -f "${INPUT_FILE}" ]; then 1337 | tail -c +4 -f ${INPUT_FILE} | awk "$awk_cmd_stat" & 1338 | # wait for TRACE_TIME to exit. 1339 | sleep ${TRACE_TIME} 1340 | break; 1341 | fi 1342 | done 1343 | 1344 | # Kill all sub processes 1345 | local all_pids=$(ps --ppid $$ | awk '{if($4~/redis-cli|tail/) print $1}') 1346 | if [ "$all_pids" != "" ]; then 1347 | kill $all_pids 1348 | fi 1349 | debug "You can check the monitor file: [${INPUT_FILE}], or analyze the file by -f again!" 1350 | fi 1351 | } 1352 | 1353 | ####################################### 1354 | # parse node slowlog by ip 1355 | # Globals: 1356 | # NODES_LIST - cluster nodes list 1357 | # REDIS_HOST - redis ip 1358 | # NODES_PREFIX - .RCTOOL-NODES- 1359 | # SLOWLOG_PREFIX - .RCTOOL-SLOWLOG- 1360 | # MYPID - pid 1361 | # AUTH - auth info 1362 | # SLOW_COUNT - count of slowlog 1363 | # Arguments: 1364 | # 1 - ip 1365 | # Returns: 1366 | # slowlog 1367 | ####################################### 1368 | function subprocess_slowlog_by_ip() { 1369 | local deal_ip=$1 1370 | 1371 | # awk scripts: parse slowlog 1372 | local awk_slowlog=' 1373 | BEGIN { 1374 | id=-1; 1375 | time=0; 1376 | duration=0; 1377 | cmd=""; 1378 | client="\"\""; 1379 | name="\"\""; 1380 | step=0; 1381 | splen=0; 1382 | spprefix=""; 1383 | } 1384 | { 1385 | if ($3 == "(integer)") { 1386 | if (id >= 0) { 1387 | print id,time,duration,client,name,cmd; 1388 | } else { 1389 | splen = index($0, "("); 1390 | for(i=1;i= 0) { 1422 | print id,time,duration,client,name,cmd; 1423 | } 1424 | }' 1425 | # init result file 1426 | local ret_file="${SLOWLOG_PREFIX}${deal_ip}.${MYPID}" 1427 | local prog_file="${NODES_PREFIX}${deal_ip}.${MYPID}" 1428 | >$ret_file 1429 | >$prog_file 1430 | for node_addr in $(echo "$NODES_LIST"|grep -w $deal_ip); do 1431 | # ADDR 1432 | local node_ip=${node_addr%:*} 1433 | local node_port=${node_addr##*:} 1434 | if [ "$node_ip" = "127.0.0.1" -o "$node_ip" = "" ]; then 1435 | node_ip=${REDIS_HOST} 1436 | fi 1437 | 1438 | local node_slowlog=$(redis-cli -h ${node_ip} -p ${node_port} ${AUTH} --no-raw slowlog get $SLOW_COUNT 2>/dev/null) 1439 | if [ $? -ne 0 ]; then 1440 | warn "Can't connect to redis(${node_ip}:${node_port})!\n" 1441 | elif [ "$node_slowlog" != "(empty list or set)" ]; then 1442 | echo "$node_slowlog"|awk "$awk_slowlog"|awk '{print "'$node_ip:$node_port' "$0}' >> ${ret_file} 1443 | fi 1444 | 1445 | echo 1 >>$prog_file 1446 | done 1447 | } 1448 | 1449 | ####################################### 1450 | # get cluster slowlog 1451 | # Globals: 1452 | # NODES_LIST - cluster nodes list 1453 | # REDIS_HOST - redis ip 1454 | # REDIS_PORT - redis port 1455 | # SLOWLOG_PREFIX - .RCTOOL-SLOWLOG- 1456 | # MYPID - pid 1457 | # AUTH - auth info 1458 | # SLOW_COUNT - count of slowlog 1459 | # SINGLE_NODE - process the specify node only 1460 | # Arguments: 1461 | # 1 - ip 1462 | # Returns: 1463 | # slowlog 1464 | ####################################### 1465 | function get_slowlogs() { 1466 | # CLUSTER or SINGLE 1467 | if [ "$SINGLE_NODE" != "yes" ]; then 1468 | # NODES_LIST 1469 | get_cluster_nodes 1470 | debug "[CLUSTER] get slowlog of cluster by the node ${REDIS_HOST}:${REDIS_PORT} (COUNT=${SLOW_COUNT})" 1471 | else 1472 | check_redis_by_ping 1473 | NODES_LIST="${REDIS_HOST}:${REDIS_PORT}" 1474 | debug "[SINGLE] get slowlog of node ${REDIS_HOST}:${REDIS_PORT} (COUNT=${SLOW_COUNT})" 1475 | fi 1476 | 1477 | local last_ip="Nil" 1478 | for node in $(echo "$NODES_LIST"|sort); do 1479 | # ip 1480 | local ip=${node%:*} 1481 | if [ "$last_ip" != "$ip" ]; then 1482 | last_ip=$ip 1483 | { 1484 | subprocess_slowlog_by_ip $last_ip 1485 | }& 1486 | fi 1487 | done 1488 | # show progress 1489 | show_progress "$NODES_LIST" 1490 | # wait subprocess done 1491 | wait 1492 | 1493 | local width=108 1494 | draw_divide_line $width "=" "$(date +' %F %T ')" 1495 | printf "%-21s %-6s %-15s %-12s %-21s %s\n" "Redis" "ID" "Time[Asc]" "Duration(us)" "Client" "Command" 1496 | draw_divide_line $width "-" 1497 | # show the slowlogs order by time 1498 | local awk_slog_show=' 1499 | BEGIN { 1500 | lastdate=""; 1501 | count=0; 1502 | } 1503 | { 1504 | currdate = substr($3,1,8) 1505 | if(lastdate != currdate) 1506 | { 1507 | if(lastdate != "") 1508 | { 1509 | printf("---------------------------[ %s: count = %s ]-------------------------------------------------\n", lastdate, count); 1510 | } 1511 | lastdate=currdate 1512 | count=0; 1513 | } 1514 | printf("%-21s %-6s %-15s %-12s %-21s [ ",$1,$2,$3,$4,substr($5,2,length($5)-2)); 1515 | for(i=7;i<=NF;i++) 1516 | { 1517 | printf("%s ",$i); 1518 | } 1519 | printf("]\n"); 1520 | count+=1; 1521 | } END { 1522 | if(lastdate != "") 1523 | { 1524 | printf("---------------------------[ %s: count = %s ]-------------------------------------------------\n", lastdate, count); 1525 | } else { 1526 | print "(no slow log)" 1527 | } 1528 | }' 1529 | # 172.16.17.3:7290 27 20201202084309 50 "10.45.80.253:33788" "JavaClient" "INFO" "commandstats" 1530 | cat ${SLOWLOG_PREFIX}*.${MYPID} 2>/dev/null | sort -k 3n -k 2n | awk "$awk_slog_show" 1531 | } 1532 | 1533 | ####################################### 1534 | # key prefix stat by file 1535 | # Globals: 1536 | # INPUT_FILE - key file name 1537 | # PREFIX_LVL - prefix level 1538 | # Arguments: 1539 | # None 1540 | # Returns: 1541 | # None 1542 | ####################################### 1543 | function keyfile_stat() { 1544 | if [ ! -f "$INPUT_FILE" ]; then 1545 | warn "File not exist: $INPUT_FILE" 1546 | exit 1 1547 | fi 1548 | # 需要设置export LC_ALL=C，这样sort才能正确的按ASCII字符排序 1549 | # 统一使用点号(.)作为连接符(可方便使用grep在key文件中按前缀来过滤查询) 1550 | # 排序完成后再使用awk命令按前缀树状层级展示，然后通过column命令进行表格化对齐 1551 | # 最后使用awk命令来美化TITLE字段以及高亮显示部分层级 1552 | (echo -e "==========,========,========\n KEY_PREFIX,COUNT,PERCENT\n----------,--------,--------"; 1553 | cat $INPUT_FILE | awk '{print $1}' | awk -v level=$PREFIX_LVL -F'[/_:.$|-]' ' 1554 | { 1555 | count+=1; 1556 | if($0 ~ /^[\x20-\x7E]+$/) { 1557 | gsub(/[\x30-\x39]/,"0"); 1558 | for ( i = 1; i <= level && NF >=i; i++ ) { 1559 | if (i == 1) 1560 | prefix=$i; 1561 | else 1562 | prefix=prefix"."$i; 1563 | switch(i) { 1564 | case 1: 1565 | x1[prefix]+=1; 1566 | break; 1567 | case 2: 1568 | x2[prefix]+=1; 1569 | break; 1570 | case 3: 1571 | x3[prefix]+=1; 1572 | break; 1573 | case 4: 1574 | x4[prefix]+=1; 1575 | break; 1576 | case 5: 1577 | x5[prefix]+=1; 1578 | break; 1579 | case 6: 1580 | x6[prefix]+=1; 1581 | break; 1582 | } 1583 | } 1584 | } 1585 | } END { 1586 | for ( value in x1 ) { 1587 | perc=100*x1[value]/count; 1588 | if(perc>=70) { 1589 | printf(" %s,%s,\033[31;4m%.2f%%\033[0m,1\n", value, x1[value], perc); 1590 | } else { 1591 | printf(" %s,%s,%.2f%%,1\n", value, x1[value], perc); 1592 | } 1593 | } 1594 | for ( value in x2 ) { 1595 | perc=100*x2[value]/count; 1596 | if(perc>=60) { 1597 | printf(" %s,%s,\033[31;4m%.2f%%\033[0m,2\n", value, x2[value], perc); 1598 | } else { 1599 | printf(" %s,%s,%.2f%%,2\n", value, x2[value], perc); 1600 | } 1601 | } 1602 | for ( value in x3 ) { 1603 | perc=100*x3[value]/count; 1604 | if(perc>=50) { 1605 | printf(" %s,%s,\033[31;4m%.2f%%\033[0m,3\n", value, x3[value], perc); 1606 | } else { 1607 | printf(" %s,%s,%.2f%%,3\n", value, x3[value], perc); 1608 | } 1609 | } 1610 | for ( value in x4 ) { 1611 | perc=100*x4[value]/count; 1612 | if(perc>=40) { 1613 | printf(" %s,%s,\033[31;4m%.2f%%\033[0m,4\n", value, x4[value], perc); 1614 | } else { 1615 | printf(" %s,%s,%.2f%%,4\n", value, x4[value], perc); 1616 | } 1617 | } 1618 | for ( value in x5 ) { 1619 | perc=100*x5[value]/count; 1620 | if(perc>=30) { 1621 | printf(" %s,%s,\033[31;4m%.2f%%\033[0m,5\n", value, x5[value], perc); 1622 | } else { 1623 | printf(" %s,%s,%.2f%%,5\n", value, x5[value], perc); 1624 | } 1625 | } 1626 | for ( value in x6 ) { 1627 | perc=100*x6[value]/count; 1628 | if(perc>=20) { 1629 | printf(" %s,%s,\033[31;4m%.2f%%\033[0m,6\n", value, x6[value], perc); 1630 | } else { 1631 | printf(" %s,%s,%.2f%%,6\n", value, x6[value], perc); 1632 | } 1633 | } 1634 | }' | sort -t ',' -k1,1 | awk -F',' -v OFS=',' ' 1635 | { 1636 | for(i=$4;i>0;i--) { 1637 | if(i==1) { 1638 | printf("%s:,%s,%s\n", $1, $2, $3) 1639 | } else if(i==2) { 1640 | printf(" |-") 1641 | } else { 1642 | printf(" ") 1643 | } 1644 | } 1645 | }' ) | column -t -s ',' -o ' | ' | awk -F'|' ' 1646 | { 1647 | if(NR==1) { 1648 | gsub(/[ |]/,"="); 1649 | len=length($0); 1650 | print 1651 | } else if(NR==3) { 1652 | gsub(/ /,"-"); 1653 | print 1654 | } else { 1655 | if(NR==2 || substr($0,1,2)==" ") { 1656 | print 1657 | } else { 1658 | printf("\033[42;30m%s\033[0m\n",$0) 1659 | } 1660 | } 1661 | } END { 1662 | for(i=1;i<=len;i++) { 1663 | printf("="); 1664 | } 1665 | printf("\n"); 1666 | }' 1667 | } 1668 | 1669 | ####################################### 1670 | # get key list from redis server 1671 | # Globals: 1672 | # INPUT_FILE - key file name 1673 | # REDIS_HOST - redis server ip 1674 | # REDIS_PORT - redis server port 1675 | # KEYS_COUNT - max keys get from server 1676 | # AUTH - password option 1677 | # Arguments: 1678 | # None 1679 | # Returns: 1680 | # None 1681 | ####################################### 1682 | function get_keys() { 1683 | if [ "$INPUT_FILE" = "" ]; then 1684 | # check redis status 1685 | check_redis_by_ping 1686 | 1687 | INPUT_FILE="${REDIS_HOST}-${REDIS_PORT}.keys" 1688 | 1689 | touch ${INPUT_FILE} 1690 | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} ${AUTH} -n ${DATABASE} --scan >${INPUT_FILE} & 1691 | local subpid=$! 1692 | local loguser=$(id -un) 1693 | local loop=0 1694 | local count=0 1695 | local subexist=1 1696 | while [ 1 ]; do 1697 | sleep 0.5 1698 | subexist=$(ps -fu $loguser | grep redis-cli | awk -v pid=$subpid '{if($2==pid){print "1"}}' | wc -l) 1699 | if [[ $subexist -eq 0 ]]; then 1700 | break 1701 | fi 1702 | # check keys enough 1703 | count=$(cat ${INPUT_FILE} | wc -l) 1704 | if [[ $KEYS_COUNT -gt 0 && $KEYS_COUNT -le $count ]]; then 1705 | kill $subpid 1706 | break 1707 | fi 1708 | loop=$((loop+1)) 1709 | printf "Time elapsed $((loop/2)) (s): $count\r" 1710 | done 1711 | wait 1712 | count=$(cat ${INPUT_FILE} | wc -l) 1713 | printf "Time elapsed $((loop/2)) (s): $count\n" 1714 | fi 1715 | } 1716 | 1717 | ####################################### 1718 | # get cluster nodes from file 1719 | # Globals: 1720 | # CLUSTER_NODES_ALL - cluster nodes list 1721 | # REDIS_PORT - redis port 1722 | # REDIS_PASS - redis password 1723 | # Arguments: 1724 | # None 1725 | # Returns: 1726 | # None 1727 | ####################################### 1728 | function get_nodes_by_file() { 1729 | if [[ -n "${NODES_FILE}" ]]; then 1730 | # load cluster nodes file 1731 | if [ ! -f "${NODES_FILE}" ]; then 1732 | warn "Can't load the cluster nodes file: ${NODES_FILE}!" 1733 | exit 1 1734 | fi 1735 | local awk_cmd=' 1736 | { 1737 | slots=$1 1738 | master=$2 1739 | if(master!=""){ 1740 | print master,"master",master,master,"master","-","pingtime","pongtime","poch","connected",slots 1741 | for(i=3;i<=NF;i++){ 1742 | print master,"slave",$i,$i,"slave",master,"pingtime","pongtime","poch","connected" 1743 | } 1744 | } 1745 | }' 1746 | CLUSTER_NODES_ALL=$(cat $NODES_FILE | awk "$awk_cmd") 1747 | if [[ -z "$CLUSTER_NODES_ALL" ]]; then 1748 | warn "Invalid nodes file: '${NODES_FILE}'\n"\ 1749 | "Format:\n"\ 1750 | " [SlaveAddr]\n"\ 1751 | "Example:\n"\ 1752 | " 1 10.10.10.1:6379 10.10.10.2:6379\n"\ 1753 | " 2 10.10.10.2:6380 10.10.10.1:6380" 1754 | exit 2 1755 | fi 1756 | local master 1757 | master=$(echo "$CLUSTER_NODES_ALL" | head -1 | awk '{print $1}') 1758 | REDIS_HOST=${master%:*} 1759 | REDIS_PORT=${master##*:} 1760 | fi 1761 | } 1762 | 1763 | 1764 | ####################################### 1765 | # parse options 1766 | # Globals: 1767 | # REDIS_HOST - redis server ip 1768 | # REDIS_PORT - redis server port 1769 | # REDIS_PASS - redis server password 1770 | # KEYS_COUNT - max keys get from server 1771 | # MONI_COUNT - times of monitor updates 1772 | # SLOW_COUNT - count of slowlog to get 1773 | # ROLL_TIME - monitor delay time in seconds 1774 | # SINGLE_NODE - single node only 1775 | # MONI_LATENCY - show average latency on moni 1776 | # TRACE_TIME - the running time on trace 1777 | # INPUT_FILE - input file name 1778 | # PREFIX_LVL - prefix level 1779 | # TRACE_HOT - analyze the hot keys 1780 | # TRACE_IP - stat commands by cient host 1781 | # OUTPUT_RAW_DATA - output raw data of nodes 1782 | # CONFIG_KEY - config name to get/set 1783 | # CONFIG_VALUE - config value to get/set 1784 | # CONFIG_SET - config set value 1785 | # ROLE_TYPE - node role type: master,slave,all 1786 | # WRITE_CONFIG - rewrite config file after set 1787 | # USAGE - help message 1788 | # CMD - sub command 1789 | # AUTH - auth parameter 1790 | # Arguments: 1791 | # command line 1792 | # Returns: 1793 | # None 1794 | ####################################### 1795 | function main() { 1796 | trap "exitup" INT TERM 1797 | trap "cleanup" EXIT 1798 | # use nawk by default 1799 | if [ -f /usr/bin/nawk ]; then alias awk=nawk ;fi 1800 | shopt -s expand_aliases 1801 | 1802 | # sort --help 1803 | # *** WARNING *** 1804 | # The locale specified by the environment affects sort order. 1805 | # Set LC_ALL=C to get the traditional sort order that uses 1806 | # native byte values. 1807 | export LC_ALL=C 1808 | 1809 | if [[ $# -eq 0 ]]; then 1810 | echo -e "$USAGE" 1811 | exit 2 1812 | fi 1813 | 1814 | while getopts "h:p:n:i:a:d:c:t:f:L:k:v:HCslrMSw" arg 1815 | do 1816 | case $arg in 1817 | h) # redis ip 1818 | REDIS_HOST="$OPTARG" 1819 | ;; 1820 | p) # redis port 1821 | REDIS_PORT="$OPTARG" 1822 | ;; 1823 | i) # input cluster nodes file 1824 | NODES_FILE="$OPTARG" 1825 | ;; 1826 | n) # database: used by 'keys' command 1827 | assert_int "$OPTARG" "n" 1828 | DATABASE="$OPTARG" 1829 | ;; 1830 | a) # redis password 1831 | REDIS_PASS="$OPTARG" 1832 | ;; 1833 | c) # number of monitor/slowlog/keys 1834 | assert_int "$OPTARG" "c" 1835 | MONI_COUNT="$OPTARG" 1836 | SLOW_COUNT="$OPTARG" 1837 | KEYS_COUNT="$OPTARG" 1838 | ;; 1839 | d) # monitor delay time in seconds 1840 | ROLL_TIME="$OPTARG" 1841 | ;; 1842 | s) # single only 1843 | SINGLE_NODE="yes" 1844 | ;; 1845 | l) # show average latency 1846 | MONI_LATENCY="yes" 1847 | ;; 1848 | t) # the running time on trace 1849 | assert_int "$OPTARG" "t" 1850 | TRACE_TIME="$OPTARG" 1851 | # running max time = 600 seconds 1852 | if [[ ${TRACE_TIME} -gt 600 ]];then 1853 | TRACE_TIME=600 1854 | fi 1855 | ;; 1856 | f) # monitor/keys file 1857 | INPUT_FILE="$OPTARG" 1858 | ;; 1859 | L) # prefix level 1860 | assert_int "$OPTARG" "L" 1861 | PREFIX_LVL="$OPTARG" 1862 | if [[ $PREFIX_LVL -le 0 ]];then 1863 | PREFIX_LVL=1; 1864 | fi 1865 | ;; 1866 | H) # analyze the hot keys 1867 | TRACE_HOT="yes" 1868 | ;; 1869 | C) # stat commands by cient host 1870 | TRACE_IP="yes" 1871 | ;; 1872 | r) # output raw data of nodes 1873 | OUTPUT_RAW_DATA="yes" 1874 | ;; 1875 | k) # config name to get/set 1876 | CONFIG_KEY="$OPTARG" 1877 | ;; 1878 | v) # config value to set 1879 | CONFIG_VALUE="$OPTARG" 1880 | CONFIG_SET="yes" 1881 | ;; 1882 | M) # master type 1883 | ROLE_TYPE="master" 1884 | ;; 1885 | S) # master type 1886 | ROLE_TYPE="slave" 1887 | ;; 1888 | w) # rewrite config file 1889 | WRITE_CONFIG="yes" 1890 | ;; 1891 | ?) # unknown options, show usage 1892 | echo -e "$USAGE" 1893 | exit 1 1894 | ;; 1895 | esac 1896 | done 1897 | # remove options dealed 1898 | shift $((OPTIND-1)) 1899 | 1900 | # check options 1901 | if [[ $# -gt 2 ]]; then 1902 | echo -e "$USAGE" 1903 | exit 2 1904 | elif [[ $# -eq 1 ]]; then 1905 | CMD="$1" 1906 | fi 1907 | 1908 | # check redis-cli to be useful 1909 | check_redis_cli 1910 | # set REDISCLI_AUTH 1911 | set_rediscli_auth 1912 | # get cluster nodes from $NODES_FILE 1913 | get_nodes_by_file 1914 | 1915 | case $CMD in 1916 | config) 1917 | setget_config 1918 | ;; 1919 | keys) 1920 | # get keys from server 1921 | get_keys 1922 | # stat keys in file 1923 | keyfile_stat 1924 | ;; 1925 | moni) 1926 | moni_commands 1927 | ;; 1928 | nodes) 1929 | get_cluster_status 1930 | ;; 1931 | slowlog) 1932 | get_slowlogs 1933 | ;; 1934 | trace) 1935 | trace_commands 1936 | ;; 1937 | *) 1938 | warn "Invalid subcommand '$CMD'" 1939 | echo -e "$USAGE" 1940 | exit 1 1941 | esac 1942 | } 1943 | 1944 | ############################################ main ################################################# 1945 | main "$@" 1946 | --------------------------------------------------------------------------------