├── README.md ├── go.mod ├── go.sum └── slow_log_summary.go /README.md: -------------------------------------------------------------------------------- 1 | # slow-log-summary 2 | 在分析慢日志时，DBA 常用的一个工具是`pt-query-digest`（Percona Toolkit 中的一个工具，具体什么是 Percona Toolkit 及 Percona Toolkit 如何安装，可参考：[MySQL 中如何归档数据](http://mp.weixin.qq.com/s?__biz=Mzg5OTY2MjU5MQ==&mid=2247485439&idx=1&sn=943f1bb046a3ef3ac724615dcb4a8fd1&chksm=c04e906ff7391979bfdfd7b35b7d84cda9888b2259d5d606a9aaac4a50608e182b2febab42c8#rd)）。 3 | 4 | 该工具可对慢日志进行汇总分析，生成一个慢查询汇总报告（报告的具体内容，后面有展示）。 5 | 6 | 虽然这个报告很实用，但这种方式其实也有不少的痛点： 7 | 8 | - 工具如何使用和怎么读懂报告有一定的学习成本。 9 | 10 | - DBA 需要时间来分析这个报告，才能告诉开发童鞋哪些 SQL 需要优化。 11 | 12 | 虽然报告的可读性已经非常好了，但离开发童鞋的要求（直接告诉我哪些 SQL 需要优化，当然，最好也能给出优化建议）还是有一定的差距。 13 | 14 | - 如果开发童鞋想要进行分析呢？对不起，他们一般没有服务器登录权限，拿不到慢日志。 15 | 16 | 下面分享的工具（`slow-log-summary`）能有效解决这些痛点： 17 | 18 | - 该工具会生成一个慢查询汇总报告。每一类 SQL 是一行。执行总耗时越久的 SQL，排名樾靠前，也最值得优化。 19 | 20 | - 报告是 HTML 格式，方便发送日报。 21 | 22 | - 对于习惯使用慢日志和`pt-query-digest`的童鞋（主要是DBA），该工具会直接基于`pt-query-digest`分析后的结果生成一个 HTML 报告，省去了分析的时间。 23 | 24 | - 对于不方便获取慢日志的童鞋（主要是开发），可直接从 performance_schema 中获取 SQL 的执行耗时分布情况。 25 | 26 | 27 | 28 | # 工具地址 29 | 30 | 项目地址：https://github.com/slowtech/slow-log-summary 31 | 32 | 可直接下载二进制包，也可进行源码编译。 33 | 34 | ### 直接下载二进制包 35 | 36 | ```bash 37 | # wget https://github.com/slowtech/slow-log-summary/releases/download/v1.0.0/slow-log-summary-linux-amd64.tar.gz 38 | # tar xvf slow-log-summary-linux-amd64.tar.gz 39 | ``` 40 | 41 | 解压后，会在当前目录生成一个名为 `slow-log-summary` 的可执行文件。 42 | 43 | ### 源码编译 44 | 45 | ```bash 46 | # wget https://github.com/slowtech/slow-log-summary/archive/refs/tags/v1.0.0.tar.gz 47 | # tar xvf v1.0.0.tar.gz 48 | # cd slow-log-summary-1.0.0 49 | # go build -o slow-log-summary slow_log_summary.go 50 | ``` 51 | 52 | 编译完成后，会在当前目录生成一个名为 `slow_log_summary `的可执行文件。 53 | 54 | 55 | 56 | # 参数解析 57 | 58 | ```bash 59 | # ./slow-log-summary --help 60 | slow-log-summary version: 1.0.0 61 | Usage: 62 | slow-log-summary -source -r [other options] 63 | 64 | Example: 65 | ./slow-log-summary -source perf -h 10.0.0.168 -P 3306 -u root -p '123456' 66 | ./slow-log-summary -source slowlog -pt /usr/local/bin/pt-query-digest -slowlog /data/mysql/3306/data/n1-slow.log 67 | 68 | Common Options: 69 | -help 70 | Display usage 71 | 72 | Source Type Options: 73 | -source string 74 | Slow log source: 'perf' or 'slowlog' (default "perf") 75 | 76 | Output File Options: 77 | -r string 78 | Direct output to a given file (default "/tmp/slow-log-summary-20060102-150405.html") 79 | 80 | Options when source is 'perf': 81 | -h string 82 | MySQL host (default "localhost") 83 | -P int 84 | MySQL port (default 3306) 85 | -u string 86 | MySQL username (default "root") 87 | -p string 88 | MySQL password 89 | -D string 90 | MySQL database (default "performance_schema") 91 | 92 | Options when source is 'slowlog': 93 | -pt string 94 | Absolute path for pt-query-digest. Example: /usr/local/bin/pt-query-digest 95 | -slowlog string 96 | Absolute path for slowlog. Example: /var/log/mysql/node1-slow.log 97 | -since string 98 | Parse only queries newer than this value, YYYY-MM-DD [HH:MM:SS] 99 | -until string 100 | Parse only queries older than this value, YYYY-MM-DD [HH:MM:SS] 101 | -yday 102 | Parse yesterday's slowlog 103 | ``` 104 | 105 | 其中， 106 | 107 | - -source：指定慢查询汇总报告的来源。可设置 perf（performance_schema），也可设置 slowlog（慢日志 + pt-query-digest）。 108 | - -r：慢查询汇总报告文件名。如果不指定，则默认为 "/tmp/slow-log-summary-当前时间.html"，例如 `/tmp/slow-log-summary-20060102-150405.html`。 109 | 110 | 当 source 设置为 perf 时，我们需要设置实例的连接信息，包括： 111 | 112 | - -h：主机名，默认是 localhost。 113 | - -P：端口，默认是 3306。 114 | - -u：用户名，默认是 root。 115 | - -p：密码。 116 | - -D：库名，默认是 performance_schema。 117 | 118 | 当 source 设置为 slowlog 时，因为该工具是基于`pt-query-digest`对慢日志进行分析，所以需要通过 -pt 设置`pt-query-digest`的绝对路径和 -slowlog 设置慢日志的绝对路径。 119 | 120 | 除此之外，还可指定分析的时间范围，默认是分析整个慢日志，也可指定 -yday 只分析昨天的慢日志，或者通过 -since 和 -until 指定具体的开始时间和结束时间。 121 | 122 | 123 | 124 | # 常见用法 125 | 126 | ### 基于 performance_schema 生成慢查询报告 127 | 128 | ```bash 129 | # ./slow-log-summary -source perf -h 10.0.0.137 -P 3306 -u root -p 123456 -r slow-log-summary.html 130 | Output written to file slow-log-summary.html 131 | 132 | 不在命令行中指定密码 133 | # ./slow-log-summary -source perf -h 10.0.0.137 -P 3306 -u root 134 | Enter MySQL password: 135 | Output written to file /tmp/slow-log-summary-20231113-204738.html 136 | ``` 137 | 138 | 139 | 140 | ### 基于慢日志和 pt-query-digest 生成慢查询报告 141 | 142 | **分析整个慢日志** 143 | 144 | ```bash 145 | # ./slow-log-summary -source slowlog -pt /usr/local/bin/pt-query-digest -slowlog /data/mysql/3306/data/n1-slow.log 146 | ``` 147 | 148 | 149 | 150 | **分析昨天的慢日志** 151 | 152 | ```bash 153 | # ./slow-log-summary -source slowlog -pt /usr/local/bin/pt-query-digest -slowlog /data/mysql/3306/data/n1-slow.log -yday 154 | ``` 155 | 156 | 157 | 158 | **分析指定时间段的慢日志** 159 | 160 | ```bash 161 | # ./slow-log-summary -source slowlog -pt /usr/local/bin/pt-query-digest -slowlog /data/mysql/3306/data/n1-slow.log -since '2023-11-04 08:01:00' -until '2023-11-04 08:10:00' 162 | ``` 163 | 164 | 165 | 166 | # 工具效果 167 | 168 | ### performance_schema 169 | 170 | 下面是基于 performance_schema 生成的查询耗时汇总报告： 171 | 172 | ![MySQL实战](https://images.cnblogs.com/cnblogs_com/ivictor/2359774/o_8188c401.jpg) 173 | 174 | 注意， 175 | 176 | 1. 报告右上角的“生成时间”是中国时区时间，而报告中 SQL 的“第一次出现时间”和“最近一次出现时间”是原样输出，没有进行时区转换，具体是什么时区下的时间取决于实例的 time_zone。 177 | 2. Digest Text 的长度由 performance_schema_max_digest_length 参数控制，默认是 1024。 178 | 3. Sample SQL 的长度由 performance_schema_max_sql_text_length 参数控制，默认是 1024。 179 | 4. 对于 MySQL 8.0 之前的版本，报告中只会显示 Digest Text（规范化语句摘要），不会显示 Sample SQL（一类 SQL 中一个具体的 SQL 语句）。Sample SQL 是 MySQL 8.0 才开始支持的。 180 | 5. 对于 MySQL 8.0.31 开始的版本，报告中还会显示语句的最大内存使用量。 181 | 6. 对于 MySQL 8.0.28 开始的版本，如果 performance_schema.setup_consumers 中 CPU 相关的配置开启了（默认没有开启），报告中还会显示语句的CPU平均耗时。 182 | ```sql 183 | UPDATE performance_schema.setup_consumers SET ENABLED = 'YES' WHERE NAME='events_statements_cpu'; 184 | ``` 185 | 7. DigestPrefix 是 Digest 的前缀，我们可以基于该值从 performance_schema.events_statements_summary_by_digest 获取该类 SQL 更详细的统计信息。 186 | 187 | 188 | ### 慢日志 + pt-query-digest 189 | 190 | 下面是基于慢日志 + pt-query-digest 生成的慢查询汇总报告： 191 | 192 | ![MySQL实战](https://images.cnblogs.com/cnblogs_com/ivictor/2359774/o_2f8af7c9.jpg) 193 | 194 | 报告中的 Digest 是 Digest Text（规范化语句摘要）对应的哈希值。 195 | 196 | 如果我们想看某一类 SQL 更详细的统计信息，可通过该类 SQL 的 Digest，去`pt-query-digest`的原始报告中找。 197 | 198 | 注意，因为 MySQL 5.6，5.7 的`pt-query-digest`的原始报告中不会输出示例 SQL 的数据库名，所以，如果分析的是 MySQL 5.6，5.7 当慢日志，数据库名这一列会为空。 199 | 200 | 201 | # 实现思路 202 | 203 | ### performance_schema 204 | 205 | performance_schema 这种方式是直接读取`performance_schema.events_statements_summary_by_digest`表中的数据。 206 | 207 | 该表按照 SCHEMA_NAME 和 DIGEST 对 MySQL 服务端执行的 SQL 进行分类统计，会统计这类 SQL 性能相关的指标，例如总耗时、执行次数、扫描行数、发送行数等，具体指标如下所示。 208 | 209 | ```sql 210 | mysql> select * from performance_schema.events_statements_summary_by_digest limit 1\G 211 | *************************** 1. row *************************** 212 | SCHEMA_NAME: NULL 213 | DIGEST: 44e35cee979ba420eb49a8471f852bbe15b403c89742704817dfbaace0d99dbb 214 | DIGEST_TEXT: SELECT @@`version_comment` LIMIT ? 215 | COUNT_STAR: 18 216 | SUM_TIMER_WAIT: 59547311000 217 | MIN_TIMER_WAIT: 153661000 218 | AVG_TIMER_WAIT: 3308183000 219 | MAX_TIMER_WAIT: 18769205000 220 | SUM_LOCK_TIME: 0 221 | SUM_ERRORS: 0 222 | SUM_WARNINGS: 0 223 | SUM_ROWS_AFFECTED: 0 224 | SUM_ROWS_SENT: 18 225 | SUM_ROWS_EXAMINED: 18 226 | SUM_CREATED_TMP_DISK_TABLES: 0 227 | SUM_CREATED_TMP_TABLES: 0 228 | SUM_SELECT_FULL_JOIN: 0 229 | SUM_SELECT_FULL_RANGE_JOIN: 0 230 | SUM_SELECT_RANGE: 0 231 | SUM_SELECT_RANGE_CHECK: 0 232 | SUM_SELECT_SCAN: 0 233 | SUM_SORT_MERGE_PASSES: 0 234 | SUM_SORT_RANGE: 0 235 | SUM_SORT_ROWS: 0 236 | SUM_SORT_SCAN: 0 237 | SUM_NO_INDEX_USED: 0 238 | SUM_NO_GOOD_INDEX_USED: 0 239 | SUM_CPU_TIME: 355275000 240 | MAX_CONTROLLED_MEMORY: 16720 241 | MAX_TOTAL_MEMORY: 63457 242 | COUNT_SECONDARY: 0 243 | FIRST_SEEN: 2023-11-04 08:01:11.564304 244 | LAST_SEEN: 2023-11-13 12:57:20.357740 245 | QUANTILE_95: 19054607179 246 | QUANTILE_99: 19054607179 247 | QUANTILE_999: 19054607179 248 | QUERY_SAMPLE_TEXT: select @@version_comment limit 1 249 | QUERY_SAMPLE_SEEN: 2023-11-13 12:57:20.357740 250 | QUERY_SAMPLE_TIMER_WAIT: 275751000 251 | 1 row in set (0.00 sec) 252 | ``` 253 | 254 | 该表是 MySQL 5.6 引入的，5.7 和 5.6 中的表结构基本一致，这里重点说说 MySQL 8.0 中新增的列： 255 | 256 | - QUANTILE_95，QUANTILE_99，QUANTILE_999：QUANTILE_95 表示 95% 的语句的执行耗时低于或等于这个值。相对于平均值，这些指标对业务更有参考意义。 257 | 258 | - QUERY_SAMPLE_TEXT，QUERY_SAMPLE_SEEN，QUERY_SAMPLE_TIMER_WAIT：MySQL 8.0 中引入的，给出了一个具体的 SQL、该 SQL 出现的时间和执行耗时情况。 259 | 260 | 在 MySQL 8.0 之前，没有具体的 SQL，只有这类 SQL 的 DIGEST_TEXT，有时候想找开发童鞋理论，却总有种证据不够充分的感觉。 261 | 262 | - MAX_CONTROLLED_MEMORY，MAX_TOTAL_MEMORY ：MySQL 8.0.31 中引入的，表示语句在执行过程中使用的最大受控内存量（由 connection_memory_limit 参数控制的内存）、最大内存量。 263 | 264 | - SUM_CPU_TIME：CPU_TIME 是 MySQL 8.0.28 中引入的，表示当前线程在 CPU 上消耗的时间，单位皮秒（picoseconds）。 265 | 266 | 注意，events_statements_summary_by_digest 表的记录数由 performance_schema_digests_size 参数控制。 267 | 268 | 该参数虽然是基于 max_connections、table_definition_cache、table_open_cache 的值动态生成，但对于生产系统，一般默认是 10000。 269 | 270 | 如果记录数满了，Performance Schema 会将新的 SQL 的 SCHEMA_NAME 和 DIGEST 设置为 NULL，同时增加 Performance_schema_digest_lost 变量的值。 271 | 272 | 如果我们看到表中 SCHEMA_NAME 和 DIGEST 为 NULL 的记录中的 COUNT_STAR（执行次数）的值比较大，可适当调大 performance_schema_digests_size 的值。 273 | 274 | 275 | 276 | ### 慢日志 + pt-query-digest 277 | 278 | pt-query-digest 对慢日志进行分析后，生成的汇总报告的内容如下： 279 | 280 | \# pt-query-digest /data/mysql/3306/data/n1-slow.log 281 | 282 | ```bash 283 | # 220.8s user time, 260ms system time, 20.53M rss, 175.02M vsz 284 | # Current date: Mon Nov 13 13:27:12 2023 285 | # Hostname: n1 286 | # Files: /data/mysql/3306/data/n1-slow.log 287 | # Overall: 2.41M total, 53 unique, 3.03 QPS, 0.00x concurrency ___________ 288 | # Time range: 2023-11-04T08:01:23 to 2023-11-13T13:12:25 289 | # Attribute total min max avg 95% stddev median 290 | # ============ ======= ======= ======= ======= ======= ======= ======= 291 | # Exec time 568s 1us 643ms 235us 424us 3ms 144us 292 | # Lock time 4s 0 176ms 1us 1us 168us 1us 293 | # Rows sent 35.77M 0 128 15.55 97.36 34.54 0.99 294 | # Rows examine 70.51M 0 256 30.65 192.76 61.52 0.99 295 | # Query size 126.30M 5 1.41k 54.91 234.30 50.86 34.95 296 | 297 | -- 汇总部分 -- 298 | # Profile 299 | # Rank Query ID Response time Calls R/Call V 300 | # ==== =================================== ============== ======= ====== = 301 | # 1 0xE81D0B3DB4FB31BC558CAEF5F387E929 155.4159 27.3% 1205900 0.0001 0.00 SELECT sbtest? 302 | # 2 0xFFFCA4D67EA0A788813031B8BBC3B329 99.7578 17.6% 120591 0.0008 0.10 COMMIT 303 | # 3 0xF0C5AE75A52E847D737F39F04B198EF6 56.9487 10.0% 120590 0.0005 0.00 SELECT sbtest? 304 | # 4 0xB2249CB854EE3C2AD30AD7E3079ABCE7 49.0592 8.6% 120590 0.0004 0.07 UPDATE sbtest? 305 | # 5 0x9934EF6887CC7A6384D1DEE77FA8D4C3 45.1779 7.9% 120590 0.0004 0.00 SELECT sbtest? 306 | # 6 0xDDBF88031795EC65EAB8A8A8BEEFF705 36.8558 6.5% 120590 0.0003 0.06 DELETE sbtest? 307 | # 7 0xA729E7889F57828D3821AE1F716D5205 34.0835 6.0% 120590 0.0003 0.00 SELECT sbtest? 308 | # 8 0xFF7C69F51BBD3A736EEB1BFDCCF4EBCD 30.8271 5.4% 120590 0.0003 0.00 SELECT sbtest? 309 | # 9 0x410C2605CF6B250BE96B374065B13356 27.7122 4.9% 120590 0.0002 0.02 UPDATE sbtest? 310 | # 10 0x6C545CFB55365122F1256A27240AEFC7 26.4530 4.7% 120590 0.0002 0.01 INSERT sbtest? 311 | # MISC 0xMISC 6.1028 1.1% 120849 0.0001 0.0 <43 ITEMS> 312 | 313 | -- 明细部分 -- 314 | # Query 1: 4.02k QPS, 0.52x concurrency, ID 0xE81D0B3DB4FB31BC558CAEF5F387E929 at byte 493647221 315 | # Scores: V/M = 0.00 316 | # Time range: 2023-11-04T08:02:09 to 2023-11-04T08:07:09 317 | # Attribute pct total min max avg 95% stddev median 318 | # ============ === ======= ======= ======= ======= ======= ======= ======= 319 | # Count 49 1205900 320 | # Exec time 27 155s 47us 317ms 128us 194us 356us 108us 321 | # Lock time 40 2s 0 4ms 1us 1us 7us 1us 322 | # Rows sent 3 1.15M 1 1 1 1 0 1 323 | # Rows examine 1 1.15M 1 1 1 1 0 1 324 | # Query size 32 41.52M 36 37 36.10 36.69 0.50 34.95 325 | # String: 326 | # Databases sbtest 327 | # Hosts 10.0.0.198 328 | # Users root 329 | # Query_time distribution 330 | # 1us 331 | # 10us ############################ 332 | # 100us ################################################################ 333 | # 1ms # 334 | # 10ms # 335 | # 100ms # 336 | # 1s 337 | # 10s+ 338 | # Tables 339 | # SHOW TABLE STATUS FROM `sbtest` LIKE 'sbtest6'\G 340 | # SHOW CREATE TABLE `sbtest`.`sbtest6`\G 341 | # EXPLAIN /*!50100 PARTITIONS*/ 342 | SELECT c FROM sbtest6 WHERE id=20570\G 343 | ... 344 | ``` 345 | 346 | 报告主要包括两大部分：Profile （汇总）部分和明细部分。 347 | 348 | 汇总部分对每一类 SQL 按照总的执行时间进行了排序。 349 | 350 | 通过汇总部分，我们可以直观地看到哪类 SQL 执行耗时最久、执行次数最多。 351 | 352 | 而明细部分则提供了每一类 SQL 具体的执行信息。 353 | 354 | 明细部分最后，会给出这类 SQL 执行时间最久的那个 SQL 作为示例 SQL。 355 | 356 | 如果示例 SQL 是 DELETE、UPDATE 语句，`pt-query-digest` 还会将其转化为 SELECT 语句一并输出，如 357 | 358 | ```sql 359 | UPDATE sbtest6 SET k=k+1 WHERE id=44622\G 360 | # Converted for EXPLAIN 361 | # EXPLAIN /*!50100 PARTITIONS*/ 362 | select k=k+1 from sbtest6 where id=44622\G 363 | ``` 364 | 365 | 之所以会这么处理，是因为在 MySQL 5.6 之前，`EXPLAIN` 命令不支持 DML 语句。 366 | 367 | 所以，如果我们在`slow-log-summary`的报告中，看到类似下面的示例 SQL，不要奇怪。实际上，它就是一条 UPDATE 语句。 368 | 369 | ```sql 370 | UPDATE sbtest6 SET k=k+1 WHERE id=44622; 371 | select k=k+1 from sbtest6 where id=44622; 372 | ``` 373 | 374 | 说回`slow-log-summary`。 375 | 376 | `slow-log-summary`报告中的排名、总耗时、耗时占比、总执行次数、Digest 实际上取的就是 Profile 部分的 Rank、Response time、Calls 和 Query ID。 377 | 378 | 接着，我们会基于 Query ID 从明细部分拿到各类 SQL 的 Exec time(avg)、Rows sent(avg) 、Rows examine(avg) 和示例 SQL。 379 | 380 | # 总结 381 | 382 | 最后，我们对比下这两种采集方式的优缺点： 383 | 384 | ### performance_schema 385 | 386 | 优点： 387 | 388 | - 简单方便，可远程获取。 389 | 390 | - 对数据库性能影响较小。 391 | 392 | - 性能相关的指标比较全面。 393 | 394 | HTML 报告中只展示了一部分，如果需要其它指标，大家可留言或自行修改源码。 395 | 396 | 缺点： 397 | 398 | - 实例关闭，events_statements_summary_by_digest 中的数据就会被清空。 399 | 400 | - events_statements_summary_by_digest 不会记录 Prepared Statement。 401 | 402 | - 有限的明细数据。 403 | 404 | 虽然 mysql 执行过的 SQL 会存储在 events_statements_xxx 表中，但这些表的容量毕竟有限。一旦超过限制，之前的记录就会被覆盖。所以如果要基于明细数据来定位问题，很可能记录就不存在。 405 | 406 | 407 | 408 | ### 慢日志 + pt-query-digest 409 | 410 | 优点： 411 | 412 | - 会记录每条慢日志，方便我们定位问题。 413 | 414 | 缺点： 415 | 416 | - 开启慢日志，对数据库性能会有一定的影响。 417 | - 性能相关的指标较少。 418 | - 慢日志存储在服务器上。如果没有服务器登录权限，分析起来就会比较麻烦。 419 | - 需要安装`pt-query-digest`。而很多对安全比较敏感的环境禁止在服务器上下载和安装第三方工具。 420 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module slow-log-summary 2 | 3 | go 1.18 4 | 5 | require ( 6 | github.com/go-sql-driver/mysql v1.7.0 7 | github.com/jmoiron/sqlx v1.3.5 8 | golang.org/x/crypto v0.7.0 9 | ) 10 | 11 | require ( 12 | github.com/howeyc/gopass v0.0.0-20210920133722-c8aef6fb66ef // indirect 13 | golang.org/x/sys v0.6.0 // indirect 14 | golang.org/x/term v0.6.0 // indirect 15 | ) 16 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/go-sql-driver/mysql v1.6.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg= 2 | github.com/go-sql-driver/mysql v1.7.0 h1:ueSltNNllEqE3qcWBTD0iQd3IpL/6U+mJxLkazJ7YPc= 3 | github.com/go-sql-driver/mysql v1.7.0/go.mod h1:OXbVy3sEdcQ2Doequ6Z5BW6fXNQTmx+9S1MCJN5yJMI= 4 | github.com/howeyc/gopass v0.0.0-20210920133722-c8aef6fb66ef/go.mod h1:lADxMC39cJJqL93Duh1xhAs4I2Zs8mKS89XWXFGp9cs= 5 | github.com/jmoiron/sqlx v1.3.5 h1:vFFPA71p1o5gAeqtEAwLU4dnX2napprKtHr7PYIcN3g= 6 | github.com/jmoiron/sqlx v1.3.5/go.mod h1:nRVWtLre0KfCLJvgxzCsLVMogSvQ1zNJtpYr2Ccp0mQ= 7 | github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo= 8 | github.com/mattn/go-sqlite3 v1.14.6/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU= 9 | golang.org/x/crypto v0.7.0 h1:AvwMYaRytfdeVt3u6mLaxYtErKYjxA2OXjJ1HHq6t3A= 10 | golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU= 11 | golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ= 12 | golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 13 | golang.org/x/term v0.6.0 h1:clScbb1cHjoCkyRbWwBEUZ5H/tIFu5TAXIqaZD0Gcjw= 14 | golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U= 15 | -------------------------------------------------------------------------------- /slow_log_summary.go: -------------------------------------------------------------------------------- 1 | // Author: Chen Chen 2 | // Created: 2023-10-24 3 | // Tool Description: Generate a summary report of MySQL slow queries. 4 | 5 | package main 6 | 7 | import ( 8 | "bytes" 9 | "context" 10 | "flag" 11 | "fmt" 12 | _ "github.com/go-sql-driver/mysql" 13 | "github.com/jmoiron/sqlx" 14 | "golang.org/x/crypto/ssh/terminal" 15 | "html/template" 16 | "io/ioutil" 17 | "log" 18 | "math" 19 | "os" 20 | "os/exec" 21 | "regexp" 22 | "strconv" 23 | "strings" 24 | "time" 25 | ) 26 | 27 | // 定义HTML模板用于生成HTML报告 28 | const temp = ` 29 | 30 | 31 | 32 | 33 | Slow Log 34 | 125 | 126 | 127 |
128 |
129 |
130 |

Slow Log Summary

131 | {{if eq .slowLogSource "performance_schema"}} 132 | 慢日志来源：performance_schema 实例地址：{{.instanceAddr}} 实例版本：{{.mysqlVersion}} 生成时间：{{.now}} 133 | {{else}} 134 | 慢日志来源：{{.slowLogSource}} 分析时间范围：{{.timeRangeStart}} ~ {{.timeRangeEnd}} 生成时间：{{.now}} 135 | {{end}} 136 |
137 | 138 | 139 | 140 | {{if eq .slowLogSource "performance_schema"}} 141 | 142 | 143 | 144 | 145 | 146 | 147 | {{with index .slowLogSummary 0}} 148 | {{if .CpuTimeAvg}} 149 | 150 | {{end}} 151 | {{if .MaxTotalMemory}} 152 | 153 | {{end}} 154 | {{end}} 155 | 156 | 157 | 158 | 159 | 160 | {{with index .slowLogSummary 0}} 161 | {{if .SampleSQL}} 162 | 163 | {{end}} 164 | {{end}} 165 | {{else}} 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | {{end}} 177 | 178 | 179 | 180 | {{if eq .slowLogSource "performance_schema"}} 181 | {{range .slowLogSummary}} 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | {{if .CpuTimeAvg}} 190 | 191 | {{end}} 192 | {{if .MaxTotalMemory}} 193 | 194 | {{end}} 195 | 196 | 197 | 198 | 199 | 200 | {{if .SampleSQL}} 201 | 202 | {{end}} 203 | 204 | {{end}} 205 | {{else}} 206 | {{range .slowLogSummary}} 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | {{end}} 220 | {{end}} 221 | 222 |
排名总耗时总执行次数平均耗时平均扫描行数平均发送行数CPU平均耗时最大内存使用量第一次出现时间最近一次出现时间数据库名DigestPrefixDigest TextSample SQL排名总耗时耗时占比总执行次数平均耗时平均扫描行数平均发送行数数据库名DigestSample SQL
{{ .RowNum}}{{ .TotalLatency}}{{ .ExecutionCount}}{{ .AvgLatency}}{{ .RowsExaminedAvg}}{{ .RowsSentAvg}}{{ .CpuTimeAvg }}{{ .MaxTotalMemory }}{{ .FirstSeen}}{{ .LastSeen}}{{ .Database}}{{ .Digest}}{{ .DigestText}}{{ .SampleSQL}}
{{ .Rank}}{{ .ResponseTime}}{{ .ResponseRatio}}{{ .Calls}}{{ .AvgExecTime}}{{ .RowsExamine}}{{ .RowsSent}}{{ .Database}}{{ .Digest}}{{range .SampleSQL}}{{.}};
{{end}}
223 |
224 |
225 |
226 |
227 | 228 | 229 | ` 230 | 231 | var currentTime time.Time 232 | 233 | // Config结构用于存储命令行参数 234 | type Config struct { 235 | Help bool 236 | Source string 237 | Host string 238 | Username string 239 | Password string 240 | Database string 241 | Port int 242 | PtCmd string 243 | Slowlog string 244 | Since string 245 | Until string 246 | Yday bool 247 | ResultFile string 248 | } 249 | 250 | // 自定义命令行参数帮助信息 251 | func customUsage() { 252 | fmt.Fprintf(os.Stdout, `slow-log-summary version: 1.0.0 253 | Usage: 254 | slow-log-summary -source -r [other options] 255 | 256 | Example: 257 | ./slow-log-summary -source perf -h 10.0.0.168 -P 3306 -u root -p '123456' 258 | ./slow-log-summary -source slowlog -pt /usr/local/bin/pt-query-digest -slowlog /data/mysql/3306/data/n1-slow.log 259 | 260 | Common Options: 261 | -help 262 | Display usage 263 | 264 | Source Type Options: 265 | -source string 266 | Slow log source: 'perf' or 'slowlog' (default "perf") 267 | 268 | Output File Options: 269 | -r string 270 | Direct output to a given file (default "/tmp/slow-log-summary-20060102-150405.html") 271 | 272 | Options when source is 'perf': 273 | -h string 274 | MySQL host (default "localhost") 275 | -P int 276 | MySQL port (default 3306) 277 | -u string 278 | MySQL username (default "root") 279 | -p string 280 | MySQL password 281 | -D string 282 | MySQL database (default "performance_schema") 283 | 284 | Options when source is 'slowlog': 285 | -pt string 286 | Absolute path for pt-query-digest. Example: /usr/local/bin/pt-query-digest 287 | -slowlog string 288 | Absolute path for slowlog. Example: /var/log/mysql/node1-slow.log 289 | -since string 290 | Parse only queries newer than this value, YYYY-MM-DD [HH:MM:SS] 291 | -until string 292 | Parse only queries older than this value, YYYY-MM-DD [HH:MM:SS] 293 | -yday 294 | Parse yesterday's slowlog 295 | `) 296 | } 297 | 298 | // 解析命令行参数 299 | func (c *Config) ParseFlags() { 300 | resultFileName := fmt.Sprintf("/tmp/slow-log-summary-%s.html", currentTime.Format("20060102-150405")) 301 | f := flag.NewFlagSet(os.Args[0], flag.ExitOnError) 302 | f.BoolVar(&c.Help, "help", false, "Display usage") 303 | f.StringVar(&c.Source, "source", "", "Slow log source") 304 | f.StringVar(&c.Host, "h", "localhost", "MySQL host") 305 | f.IntVar(&c.Port, "P", 3306, "MySQL port") 306 | f.StringVar(&c.Username, "u", "root", "MySQL username") 307 | f.StringVar(&c.Password, "p", "", "MySQL password") 308 | f.StringVar(&c.Database, "D", "performance_schema", "MySQL database") 309 | f.StringVar(&c.PtCmd, "pt", "", "Absolute path for pt-query-digest. Example:/usr/local/bin/pt-query-digest") 310 | f.StringVar(&c.Slowlog, "slowlog", "", "Absolute path for slowlog. Example:/var/log/mysql/node1-slow.log") 311 | f.StringVar(&c.Since, "since", "", "Parse only queries newer than this value,YYYY-MM-DD [HH:MM:SS]") 312 | f.StringVar(&c.Until, "until", "", "Parse only queries older than this value,YYYY-MM-DD [HH:MM:SS]") 313 | f.BoolVar(&c.Yday, "yday", false, "Parse yesterday's slowlog") 314 | f.StringVar(&c.ResultFile, "r", resultFileName, "Direct output to a given file") 315 | f.Parse(os.Args[1:]) 316 | if c.Help { 317 | customUsage() 318 | os.Exit(0) 319 | } 320 | } 321 | 322 | // 执行系统命令 323 | func executeCommand(command string, args []string) (string, error) { 324 | cmd := exec.Command(command, args...) 325 | 326 | var stdout bytes.Buffer 327 | var stderr bytes.Buffer 328 | 329 | cmd.Stdout = &stdout 330 | cmd.Stderr = &stderr 331 | 332 | err := cmd.Run() 333 | if err != nil { 334 | return "", fmt.Errorf("command %s %v failed with %s", command, args, stderr.String()) 335 | } 336 | 337 | return stdout.String(), nil 338 | } 339 | 340 | // 获取通过pt-query-digest解析的慢日志汇总报告 341 | func getSlowLogSummaryByPtQueryDigest(ptQueryDigestCmd []string, slowlogFile string, now string) map[string]interface{} { 342 | cmdTimeWithQuotes := make([]string, len(ptQueryDigestCmd)) 343 | for i, arg := range ptQueryDigestCmd { 344 | if strings.Contains(arg, " ") { 345 | cmdTimeWithQuotes[i] = `"` + arg + `"` 346 | } else { 347 | cmdTimeWithQuotes[i] = arg 348 | } 349 | } 350 | fmt.Printf("Executing pt-query-digest command: %s\n", strings.Join(cmdTimeWithQuotes, " ")) 351 | slowLog, err := executeCommand("perl", ptQueryDigestCmd) 352 | if err != nil { 353 | log.Fatalf("Error: Failed to execute the Perl command for pt-query-digest: %v", err) 354 | } 355 | if strings.Contains(string(slowLog), "# No events processed") { 356 | log.Println("Warning: No events processed") 357 | } 358 | lines := strings.Split(slowLog, "\n") 359 | ptQueryDigestReportPath := fmt.Sprintf("/tmp/pt-query-digest-report-%s.txt", currentTime.Format("20060102-150405")) 360 | // Write slowLog content to the file 361 | err = ioutil.WriteFile(ptQueryDigestReportPath, []byte(slowLog), 0644) 362 | if err != nil { 363 | log.Printf("Warning: Failed to write slowLog content to file: %v", err) 364 | } 365 | fmt.Println(fmt.Sprintf("pt-query-digest report written to file %s", ptQueryDigestReportPath)) 366 | linesNums := len(lines) 367 | timeRangeFlag := false 368 | var timeRangeStart string 369 | var timeRangeEnd string 370 | profileFlag := false 371 | sampleFlag := false 372 | sampleSQL := []string{} 373 | sampleSQLInfo := make(map[string]string) 374 | slowLogProfile := [][]string{} 375 | sampleSQLs := make(map[string]map[string]string) 376 | var queryID string 377 | for k, line := range lines { 378 | if strings.Contains(line, "# Overall") { 379 | timeRangeFlag = true 380 | continue 381 | } 382 | if timeRangeFlag { 383 | if strings.HasPrefix(line, "# Time range") { 384 | re, _ := regexp.Compile(" +") 385 | rowToArray := re.Split(line, -1) 386 | timeRangeStart, timeRangeEnd = rowToArray[3], rowToArray[5] 387 | timeRangeStart = strings.Replace(timeRangeStart, "T", " ", 1) 388 | timeRangeEnd = strings.Replace(timeRangeEnd, "T", " ", 1) 389 | timeRangeFlag = false 390 | } 391 | } 392 | if strings.Contains(line, "# Profile") { 393 | profileFlag = true 394 | continue 395 | } else if profileFlag && (len(line) == 0 || strings.HasPrefix(line, "# MISC 0xMISC")) { 396 | profileFlag = false 397 | continue 398 | } 399 | if profileFlag { 400 | if strings.HasPrefix(line, "# Rank") || strings.HasPrefix(line, "# ====") { 401 | continue 402 | } 403 | re, _ := regexp.Compile(" +") 404 | rowToArray := re.Split(line, 9) 405 | slowLogProfile = append(slowLogProfile, rowToArray) 406 | } else if strings.Contains(line, "concurrency, ID 0x") { 407 | re := regexp.MustCompile(`(?U)ID (0x.*) `) 408 | queryID = re.FindStringSubmatch(line)[1] 409 | sampleFlag = true 410 | sampleSQL = []string{} 411 | sampleSQLInfo = make(map[string]string) 412 | } else if sampleFlag && (strings.HasPrefix(line, "# Exec time") || strings.HasPrefix(line, "# Lock time")) { 413 | re, _ := regexp.Compile(" +") 414 | rowToArray := re.Split(line, -1) 415 | sampleSQLInfo[rowToArray[1]] = rowToArray[7] 416 | } else if sampleFlag && (strings.HasPrefix(line, "# Rows sent") || strings.HasPrefix(line, "# Rows examine")) { 417 | re, _ := regexp.Compile(" +") 418 | rowToArray := re.Split(line, -1) 419 | sampleSQLInfo[rowToArray[2]] = rowToArray[7] 420 | } else if sampleFlag && (strings.HasPrefix(line, "# Databases")) { 421 | re, _ := regexp.Compile(" +") 422 | rowToArray := re.Split(line, 3) 423 | sampleSQLInfo["Databases"] = rowToArray[2] 424 | } else if sampleFlag && (!strings.HasPrefix(line, "#")) && len(line) != 0 { 425 | sampleSQL = append(sampleSQL, line) 426 | } else if sampleFlag && (len(line) == 0 || k == (linesNums-1)) { 427 | sampleFlag = false 428 | sampleSQLText := strings.Join(sampleSQL, "\n") 429 | sampleSQLText = strings.TrimRight(sampleSQLText, "\\G") 430 | sampleSQLInfo["sampleSQL"] = sampleSQLText 431 | sampleSQLs[queryID] = sampleSQLInfo 432 | } 433 | } 434 | 435 | for i, v := range slowLogProfile { 436 | for key := range sampleSQLs { 437 | miniQueryID := strings.Trim(v[2], ".") 438 | if strings.Contains(key, miniQueryID) { 439 | v[8] = sampleSQLs[key]["sampleSQL"] 440 | v[2] = key 441 | v = append(v, sampleSQLs[key]["Exec"], sampleSQLs[key]["Lock"], sampleSQLs[key]["sent"], sampleSQLs[key]["examine"], sampleSQLs[key]["Databases"]) 442 | slowLogProfile[i] = v 443 | break 444 | } 445 | } 446 | } 447 | 448 | type slowlog struct { 449 | Rank string 450 | ResponseTime string 451 | ResponseRatio string 452 | Calls string 453 | AvgExecTime string 454 | AvgLockTime string 455 | RowsSent string 456 | RowsExamine string 457 | Database string 458 | Digest string 459 | SampleSQL []string 460 | } 461 | 462 | slowlogs := []slowlog{} 463 | for _, value := range slowLogProfile { 464 | // 之所以要将字符串分隔为切片，主要是为了模板渲染时生成换行符
465 | sampleSQLSplitResult := strings.Split(value[8], "\\G") 466 | slowlogrecord := slowlog{value[1], value[3], value[4], value[5], value[9], value[10], value[11], value[12], value[13], value[2], sampleSQLSplitResult} 467 | slowlogs = append(slowlogs, slowlogrecord) 468 | } 469 | return map[string]interface{}{"slowLogSource": slowlogFile, "slowLogSummary": slowlogs, "now": now, "timeRangeStart": timeRangeStart, "timeRangeEnd": timeRangeEnd} 470 | } 471 | 472 | // 将纳秒时间戳格式化为日期时间字符串 473 | func formatPicoTime(val string) string { 474 | timeVal, err := strconv.ParseFloat(val, 64) 475 | if err != nil { 476 | log.Fatalf("Error: Conversion failed: %v\n", err) 477 | } 478 | 479 | if math.IsNaN(timeVal) { 480 | return "null" 481 | } 482 | 483 | const ( 484 | nano = 1000 485 | micro = 1000 * nano 486 | milli = 1000 * micro 487 | sec = 1000 * milli 488 | min = 60 * sec 489 | hour = 60 * min 490 | day = 24 * hour 491 | ) 492 | 493 | var divisor uint64 494 | var unit string 495 | 496 | timeAbs := math.Abs(timeVal) 497 | 498 | if timeAbs >= float64(day) { 499 | divisor = day 500 | unit = "d" 501 | } else if timeAbs >= float64(hour) { 502 | divisor = hour 503 | unit = "h" 504 | } else if timeAbs >= float64(min) { 505 | divisor = min 506 | unit = "min" 507 | } else if timeAbs >= float64(sec) { 508 | divisor = sec 509 | unit = "s" 510 | } else if timeAbs >= float64(milli) { 511 | divisor = milli 512 | unit = "ms" 513 | } else if timeAbs >= float64(micro) { 514 | divisor = micro 515 | unit = "us" 516 | } else if timeAbs >= float64(nano) { 517 | divisor = nano 518 | unit = "ns" 519 | } else { 520 | divisor = 1 521 | unit = "ps" 522 | } 523 | 524 | var result string 525 | 526 | if divisor == 1 { 527 | result = fmt.Sprintf("%.3f %s", timeVal, unit) 528 | } else { 529 | value := timeVal / float64(divisor) 530 | if math.Abs(value) >= 100000.0 { 531 | result = fmt.Sprintf("%.2e %s", value, unit) 532 | } else { 533 | result = fmt.Sprintf("%.2f %s", value, unit) 534 | } 535 | } 536 | 537 | return result 538 | } 539 | 540 | func IsMySQLVersionGreaterOrEqual(version1, version2 string) bool { 541 | // 对于 5.7.44-log 之类的版本号，首先会去掉中划线及中划线之后的字符 542 | version1 = strings.Split(version1, "-")[0] 543 | version2 = strings.Split(version2, "-")[0] 544 | parts1 := strings.Split(version1, ".") 545 | parts2 := strings.Split(version2, ".") 546 | 547 | for i := 0; i < len(parts1) && i < len(parts2); i++ { 548 | v1, _ := strconv.Atoi(parts1[i]) 549 | v2, _ := strconv.Atoi(parts2[i]) 550 | 551 | if v1 > v2 { 552 | return true 553 | } else if v1 < v2 { 554 | return false 555 | } 556 | } 557 | 558 | return true 559 | } 560 | 561 | // 从performance_schema中获取慢日志汇总报告 562 | func getSlowLogSummaryFromPerformanceSchema(username string, password string, host string, database string, port int, now string) map[string]interface{} { 563 | // 创建数据库连接 564 | dsn := fmt.Sprintf("%s:%s@tcp(%s:%d)/%s", username, password, host, port, database) 565 | db, err := sqlx.Connect("mysql", dsn) 566 | if err != nil { 567 | log.Fatalf("Error: Failed to establish a connection to the database: %v", err) 568 | } 569 | defer db.Close() 570 | var statementAnalysisSQL string 571 | 572 | var mysqlVersion string 573 | err = db.Get(&mysqlVersion, "SELECT VERSION()") 574 | 575 | if err != nil { 576 | log.Fatalf("Error: Failed to retrieve MySQL version: %v", err) 577 | } 578 | 579 | // 检查 MySQL 版本 580 | var queryColumn string 581 | if IsMySQLVersionGreaterOrEqual(mysqlVersion, "8.0.0") { 582 | queryColumn = ", IFNULL(QUERY_SAMPLE_TEXT,'NULL') AS sample_query" 583 | if IsMySQLVersionGreaterOrEqual(mysqlVersion, "8.0.28") { 584 | // 查询 performance_schema.setup_consumers 中 'events_statements_cpu' 记录的 enabled 列 585 | var eventsStatementsCPUEnabled string 586 | err = db.Get(&eventsStatementsCPUEnabled, "SELECT enabled FROM performance_schema.setup_consumers WHERE name='events_statements_cpu'") 587 | if err != nil { 588 | log.Fatalf("Error: Failed to retrieve events_statements_cpu enabled status: %v", err) 589 | } 590 | if eventsStatementsCPUEnabled == "YES" { 591 | queryColumn += ", ROUND(IFNULL(SUM_CPU_TIME / NULLIF(COUNT_STAR, 0), 0), 0) AS cup_time_avg" 592 | } 593 | } 594 | if IsMySQLVersionGreaterOrEqual(mysqlVersion, "8.0.31") { 595 | queryColumn += ", FORMAT_BYTES(MAX_TOTAL_MEMORY) AS max_total_memory" 596 | } 597 | } else if !IsMySQLVersionGreaterOrEqual(mysqlVersion, "5.6.0") { 598 | log.Fatalf("Error: MySQL version %s is not supported. This tool only supports MySQL 5.6 and above.", mysqlVersion) 599 | } 600 | 601 | statementAnalysisSQL = fmt.Sprintf(` 602 | SELECT 603 | CASE WHEN SCHEMA_NAME IS NULL AND DIGEST IS NULL THEN 'NULL' 604 | WHEN SCHEMA_NAME IS NULL THEN '' 605 | ELSE SCHEMA_NAME 606 | END AS db, 607 | IF(SUM_NO_GOOD_INDEX_USED > 0 OR SUM_NO_INDEX_USED > 0, 'Y', 'N') AS full_scan, 608 | COUNT_STAR AS exec_count, 609 | SUM_ERRORS AS err_count, 610 | SUM_WARNINGS AS warn_count, 611 | SUM_TIMER_WAIT AS total_latency, 612 | MAX_TIMER_WAIT AS max_latency, 613 | AVG_TIMER_WAIT AS avg_latency, 614 | SUM_LOCK_TIME AS lock_latency, 615 | SUM_ROWS_SENT AS rows_sent, 616 | ROUND(IFNULL(SUM_ROWS_SENT / NULLIF(COUNT_STAR, 0), 0), 0) AS rows_sent_avg, 617 | SUM_ROWS_EXAMINED AS rows_examined, 618 | ROUND(IFNULL(SUM_ROWS_EXAMINED / NULLIF(COUNT_STAR, 0), 0), 0) AS rows_examined_avg, 619 | SUM_ROWS_AFFECTED AS rows_affected, 620 | ROUND(IFNULL(SUM_ROWS_AFFECTED / NULLIF(COUNT_STAR, 0), 0), 0) AS rows_affected_avg, 621 | SUM_CREATED_TMP_TABLES AS tmp_tables, 622 | SUM_CREATED_TMP_DISK_TABLES AS tmp_disk_tables, 623 | SUM_SORT_ROWS AS rows_sorted, 624 | SUM_SORT_MERGE_PASSES AS sort_merge_passes, 625 | CASE 626 | WHEN DIGEST IS NULL THEN 'NULL' 627 | ELSE LEFT(DIGEST,10) 628 | END AS digest, 629 | IFNULL(DIGEST_TEXT,'NULL') AS digest_text, 630 | DATE_FORMAT(FIRST_SEEN, '%%Y-%%m-%%d %%H:%%i:%%s') AS first_seen, 631 | DATE_FORMAT(LAST_SEEN, '%%Y-%%m-%%d %%H:%%i:%%s') AS last_seen 632 | %s 633 | FROM performance_schema.events_statements_summary_by_digest 634 | ORDER BY total_latency DESC 635 | `, queryColumn) 636 | type QuerySummary struct { 637 | RowNum int 638 | SampleSQL string `db:"sample_query"` 639 | Database string `db:"db"` 640 | FullScan string `db:"full_scan"` 641 | ExecutionCount int `db:"exec_count"` 642 | ErrorCount int `db:"err_count"` 643 | WarningCount int `db:"warn_count"` 644 | TotalLatency string `db:"total_latency"` 645 | MaxLatency string `db:"max_latency"` 646 | AvgLatency string `db:"avg_latency"` 647 | LockLatency string `db:"lock_latency"` 648 | RowsSent int `db:"rows_sent"` 649 | RowsSentAvg int `db:"rows_sent_avg"` 650 | RowsExamined int `db:"rows_examined"` 651 | RowsExaminedAvg int `db:"rows_examined_avg"` 652 | RowsAffected int `db:"rows_affected"` 653 | RowsAffectedAvg int `db:"rows_affected_avg"` 654 | TmpTables int `db:"tmp_tables"` 655 | TmpDiskTables int `db:"tmp_disk_tables"` 656 | RowsSorted int `db:"rows_sorted"` 657 | SortMergePasses int `db:"sort_merge_passes"` 658 | Digest string `db:"digest"` 659 | DigestText string `db:"digest_text"` 660 | FirstSeen string `db:"first_seen"` 661 | LastSeen string `db:"last_seen"` 662 | CpuTimeAvg string `db:"cup_time_avg"` 663 | MaxTotalMemory string `db:"max_total_memory"` 664 | } 665 | var QuerySummaries []QuerySummary 666 | ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) 667 | defer cancel() 668 | err = db.SelectContext(ctx, &QuerySummaries, statementAnalysisSQL) 669 | if err != nil { 670 | log.Fatalf("Error: Failed to retrieve query summaries from the database: %v", err) 671 | } 672 | for i, summary := range QuerySummaries { 673 | summary.RowNum = i + 1 674 | summary.TotalLatency = formatPicoTime(summary.TotalLatency) 675 | summary.MaxLatency = formatPicoTime(summary.MaxLatency) 676 | summary.AvgLatency = formatPicoTime(summary.AvgLatency) 677 | summary.LockLatency = formatPicoTime(summary.LockLatency) 678 | if summary.CpuTimeAvg != "" { 679 | summary.CpuTimeAvg = formatPicoTime(summary.CpuTimeAvg) 680 | } 681 | QuerySummaries[i] = summary 682 | } 683 | return map[string]interface{}{"slowLogSource": "performance_schema", "slowLogSummary": QuerySummaries, "now": now, "instanceAddr": fmt.Sprintf("%s:%d", host, port), "mysqlVersion": mysqlVersion} 684 | } 685 | 686 | func validateAndConstructCmd(pt, slowlog, since, until string, yday bool) []string { 687 | 688 | if len(pt) == 0 || len(slowlog) == 0 { 689 | log.Fatalf("Error: Both -pt and -slowlog parameters are required.") 690 | } 691 | 692 | today := time.Now().Format("2006-01-02") 693 | yesterday := time.Now().AddDate(0, 0, -1).Format("2006-01-02") 694 | 695 | var ptQueryDigestCmd []string 696 | ptQueryDigestCmd = append(ptQueryDigestCmd, pt) 697 | ptQueryDigestCmd = append(ptQueryDigestCmd, slowlog) 698 | if len(since) != 0 || yday { 699 | ptQueryDigestCmd = append(ptQueryDigestCmd, "--since") 700 | if len(since) != 0 { 701 | ptQueryDigestCmd = append(ptQueryDigestCmd, since) 702 | } else { 703 | ptQueryDigestCmd = append(ptQueryDigestCmd, yesterday) 704 | } 705 | 706 | } 707 | if len(until) != 0 || yday { 708 | ptQueryDigestCmd = append(ptQueryDigestCmd, "--until") 709 | if len(until) != 0 { 710 | ptQueryDigestCmd = append(ptQueryDigestCmd, until) 711 | } else { 712 | ptQueryDigestCmd = append(ptQueryDigestCmd, today) 713 | } 714 | 715 | } 716 | return ptQueryDigestCmd 717 | } 718 | 719 | func main() { 720 | cst := time.FixedZone("CST", 8*60*60) 721 | currentTime = time.Now().In(cst) 722 | conf := Config{} 723 | conf.ParseFlags() 724 | report_content := make(map[string]interface{}) 725 | now := currentTime.Format("2006-01-02 15:04:05") 726 | log.SetFlags(log.Lshortfile) 727 | if len(conf.Source) == 0 { 728 | log.Fatalf("Error: The --source parameter is required.") 729 | } 730 | if conf.Source == "perf" { 731 | if conf.Since != "" || conf.Until != "" || conf.Yday { 732 | log.Fatalf("Error: Parameters 'since', 'until', and 'yday' are not allowed when source is set to 'perf'.") 733 | } 734 | } 735 | if conf.Source == "perf" { 736 | if conf.Password == "" { 737 | fmt.Print("Enter MySQL password: ") 738 | bytePassword, err := terminal.ReadPassword(int(os.Stdin.Fd())) 739 | fmt.Println() 740 | if err != nil { 741 | log.Fatalf("Error: Failed to read the password - %v", err) 742 | } 743 | conf.Password = string(bytePassword) 744 | } 745 | report_content = getSlowLogSummaryFromPerformanceSchema(conf.Username, conf.Password, conf.Host, conf.Database, conf.Port, now) 746 | } 747 | 748 | if conf.Source == "slowlog" { 749 | ptQueryDigestCmd := validateAndConstructCmd(conf.PtCmd, conf.Slowlog, conf.Since, conf.Until, conf.Yday) 750 | report_content = getSlowLogSummaryByPtQueryDigest(ptQueryDigestCmd, conf.Slowlog, now) 751 | } 752 | // 创建并写入HTML报告 753 | var report = template.Must(template.New("slowlog").Parse(temp)) 754 | file, err := os.Create(conf.ResultFile) 755 | if err != nil { 756 | log.Fatal(err) 757 | } 758 | defer file.Close() 759 | report.Execute(file, report_content) 760 | fmt.Println(fmt.Sprintf("Output written to file %s", conf.ResultFile)) 761 | } 762 | --------------------------------------------------------------------------------