├── README.md ├── mysql_sniffer ├── mysql_sniffer.py └── mysql_sniffer_centos6 /README.md: -------------------------------------------------------------------------------- 1 | # mysql_sniffer 是一个基于 MySQL 协议的抓包工具，用来实时抓取 MySQL 服务端的请求，并格式化输出，输出内容包括访问用户、访问时间、来源 IP、执行的SQL语句。 2 | 3 | ``` 4 | mysql_sniffer is a packet capture tool based on the MySQL protocol, 5 | used to capture real-time requests from the MySQL server and format the output. 6 | The output includes user access, access time, source IP, and executed SQL statements. 7 | ``` 8 | 9 | https://www.oschina.net/p/hcymysql_mysql_sniffer 10 | 11 | 在进行MySQL 8.0升级时，了解新版本对SQL语法的改变和新增的功能是非常重要的。通过使用mysql_sniffer，DBA可以在升级之前对现有的SQL语句进行抓取和分析，以确保在新版本中能够正常运行。 12 | 13 | 使用mysql_sniffer工具可以带来以下几点好处： 14 | 15 | 1) 对SQL语法的改变有更深入的了解：MySQL 8.0引入了一些新的SQL语法，也对一些旧的语法进行了修改或弃用。通过mysql_sniffer，DBA可以抓取并分析现有的SQL语句，以确定它们是否会受到这些改变的影响。 16 | 17 | 2) 发现并解决潜在的问题：如果在新版本中，某些SQL语句无法正常运行，那么通过mysql_sniffer，DBA可以提前发现这些问题，并在升级之前进行修复。 18 | 19 | 3) 数据库割接、迁移时，使用 mysql_sniffer 可以方便地判断原主库是否还有业务访问。通过 mysql_sniffer ，你可以截获数据库的查询语句、事务操作等信息，并进行分析。如果在割接或迁移过程中，没有新的业务请求经过原主库，那么可以判断原主库可以正常下线。 20 | 21 | ``` 22 | When upgrading to MySQL 8.0, it is crucial to understand the changes and additions to SQL syntax in the new version. 23 | By using mysql_sniffer, DBAs can capture and analyze existing SQL statements before upgrading to ensure they will function properly in the new version. 24 | 25 | The benefits of using mysql_sniffer include: 26 | 27 | Deeper understanding of changes to SQL syntax: MySQL 8.0 introduces new SQL syntax and modifies or deprecates some old syntax. 28 | With mysql_sniffer, DBAs can capture and analyze existing SQL statements to determine if they will be affected by these changes. 29 | 30 | Identification and resolution of potential issues: If certain SQL statements cannot function properly in the new version, 31 | DBAs can identify and resolve these issues earlier by using mysql_sniffer before upgrading. 32 | ``` 33 | 34 | ------------------------------------------------- 35 | 有一些已知SQL语法与MySQL 8.0不兼容，例如： 36 | ``` 37 | select NVL(id/0,'YES') from test.t1 where id = 1; 38 | select user_id,sum(amount) from test.user group by user_id DESC limit 10; 39 | ``` 40 | 41 | 第一条语句，NVL函数是MariaDB特有的，在MySQL 8.0中，要改成： 42 | ``` 43 | select IFNULL(id/0,'YES') from test.t1 where id = 1; 44 | ``` 45 | 46 | 第二条语句，在MySQL 8.0中group by 字段 ASC/DESC 失效，要改成： 47 | ``` 48 | select user_id,sum(amount) from test.user group by user_id order by user_id DESC limit 10; 49 | ``` 50 | 51 | ### 那么，如何判断业务上的未知SQL是否与MySQL 8.0兼容呢？ 52 | 53 | ### mysql_sniffer工具可以帮助你 54 | 55 | # 介绍 56 | ``` 57 | usage: mysql_sniffer [-h] -p PORT [-t TABLES [TABLES ...]] [-l LOG] [-c] [-r RUNTIME] [-v] 58 | 59 | MySQL packet sniffer 60 | 61 | options: 62 | -h, --help show this help message and exit 63 | -p PORT, --port PORT MySQL server port 64 | -t TABLES [TABLES ...], --tables TABLES [TABLES ...] 65 | Table names to capture 66 | -l LOG, --log LOG Log file path 67 | -c, --console Print log to console 68 | -r RUNTIME, --runtime RUNTIME 69 | Runtime of packet sniffing in seconds 70 | -v, --version show program's version number and exit 71 | ``` 72 | 73 | # 参数解释 74 | ``` 75 | -p 指定端口，MySQL默认3306 76 | -c 是把抓取到的SQL打印到终端 77 | -t 指定具体的表名，例如只抓取t1，t2，t3这三张表， -t t1 t2 t3 （不支持正则表达式，请写具体的表名） 78 | -l 抓取的SQL保存在哪个文件里，不指定默认保存在mysql_packet.sql文件里 79 | -r 抓取多长时间，单位秒 80 | ``` 81 | 82 | # 演示视频 83 | https://www.douyin.com/video/7295397864006536502 84 | 85 | # 使用 86 | ``` 87 | shell> chmod 755 mysql_sniffer 88 | ``` 89 | 90 | 在 MySQL 5.7 或者 MariaDB 机器上执行（SSH的ROOT权限） 91 | ``` 92 | shell> ./mysql_sniffer -p 3306 -r 60 93 | ``` 94 | 将会抓取60秒数据（-r 代表抓取的时间，单位秒），默认会把线上的SQL语句（select/insert/update/delete/call）存入mysql_packet.sql文件里。 95 | 96 | ![image](https://github.com/hcymysql/mysql_sniffer/assets/19261879/7ed20afb-db0e-4e7a-9892-f03ccb34e5aa) 97 | 98 | ![image](https://github.com/hcymysql/mysql_sniffer/assets/19261879/9a7177ea-3af5-49da-a2f3-c86ad4fb5a89) 99 | 100 | #### 抓取1-10分钟数据，然后把mysql_packet.sql文件拷贝到MySQL 8.0测试环境里，然后执行下面的命令： 101 | ``` 102 | mysql -S /tmp/mysql_mysql8_1.sock yourDB -f < mysql_packet.sql > /dev/null 103 | ``` 104 | #### 看报错信息，是否含有（You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax）。 105 | #### 没有语法错误，就代表SQL是兼容的。 106 | 107 | #### 注：请确保生产环境和测试环境的表结构一致，测试环境不需要任何数据。 108 | 109 | -------------------------------------------- 110 | # 测试 111 | 1） 假定 192.168.1.1 是 MySQL 5.7 / MariaDB，在该机器上运行./mysql_sniffer -p 3306 -c 112 | 113 | 2） 在 192.168.1.2 机器上运行sysbench，模拟出生产环境数据读写。 114 | ``` 115 | shell> sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --oltp_tables_count=1 116 | --mysql-table-engine=innodb --oltp-table-size=100000 --max-requests=0 --max-time=120 --num-threads=12 117 | --mysql-host=192.168.1.1 --mysql-port=3306 --mysql-user=admin 118 | --mysql-password=hechunyang --mysql-db=test --db-driver=mysql run 119 | ``` 120 | 121 | 3） mysql_sniffer会实时打印出目前运行的SQL语句。 122 | 123 | -------------------------------------------- 124 | 注：工具适用于 Centos6 和 Centos7 系统。 125 | 126 | mysql_sniffer（Centos7） 127 | 128 | mysql_sniffer_centos6（Centos6） 129 | 130 | 131 | 132 | -------------------------------------------------------------------------------- /mysql_sniffer: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hcymysql/mysql_sniffer/f085843f37a76623a14d24ba0eec102c1e819e4c/mysql_sniffer -------------------------------------------------------------------------------- /mysql_sniffer.py: -------------------------------------------------------------------------------- 1 | import re 2 | import sys 3 | import argparse 4 | import logging 5 | from scapy.all import * 6 | from datetime import datetime 7 | import time 8 | 9 | # Set Scapy's TCP reassembly limit to unlimited 10 | conf.contribs['TCPSession'] = {'reassembler': 'nosack'} 11 | 12 | query = b'' 13 | 14 | def parse_mysql_packet(packet, table_names): 15 | global query 16 | queries = ['select', 'insert', 'update', 'delete', 'call'] 17 | if packet.haslayer(TCP) and packet.haslayer(Raw) and packet[TCP].dport == port: 18 | payload = bytes(packet[Raw].load) 19 | #print(payload) 20 | 21 | if payload[4] == 0x03: 22 | query = payload[5:] 23 | else: 24 | #query += payload[5:] 25 | match_user = re.search(b'\x00(\w+)\x00', payload) 26 | if match_user: 27 | logger.info(f"-- Connect User：{match_user.group(1).decode()}") 28 | 29 | """ 30 | regex = rb'\x00(\w+)\x00' 31 | match = re.search(regex, payload[5:]) 32 | if match: 33 | index = match.start() 34 | query += payload[5:index] 35 | """ 36 | try: 37 | data = query.decode('utf-8') 38 | except UnicodeDecodeError: 39 | try: 40 | data = query.decode('latin1') 41 | except UnicodeDecodeError: 42 | # If both decodings fail, log the error and return 43 | logger.error("Failed to decode query") 44 | return 45 | 46 | words = data.strip().split() 47 | if words and words[0].lower() in queries: 48 | current_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S") 49 | source_ip = packet[IP].src 50 | logger.info(f"-- Time: {current_time}") 51 | logger.info(f"-- Source IP: {source_ip}") 52 | 53 | if table_names: 54 | for table_name in table_names: 55 | regex_pattern = r'FROM\s+\b{}\b'.format(table_name) 56 | match = re.search(regex_pattern, data, re.IGNORECASE) 57 | if match: 58 | logger.info(f"{data};") 59 | else: 60 | logger.info(f"{data};") 61 | 62 | logger.info("-- --------------------") 63 | 64 | def sniff_mysql_packets(port, table_names, runtime=0): 65 | # 创建一个TCPSession对象 66 | tcp_session = TCPSession() 67 | try: 68 | sniff(filter=f"tcp port {port} and tcp[13] == 24", prn=lambda pkt: parse_mysql_packet(pkt, table_names), session=tcp_session, timeout=runtime) 69 | except KeyboardInterrupt: 70 | print("\nSniffing operation stopped") 71 | sys.exit(0) 72 | 73 | # 解析命令行参数 74 | parser = argparse.ArgumentParser(description='MySQL packet sniffer') 75 | parser.add_argument('-p', '--port', type=int, help='MySQL server port', required=True) 76 | parser.add_argument('-t', '--tables', nargs='+', help='Table names to capture') 77 | parser.add_argument('-l', '--log', type=str, default='mysql_packet.sql', help='Log file path') 78 | parser.add_argument('-c', '--console', action='store_true', help='Print log to console') 79 | parser.add_argument('-r', '--runtime', type=int, help='Runtime of packet sniffing in seconds') 80 | parser.add_argument('-v', '--version', action='version', version='mysql_sniffer工具版本号: 1.0.4，更新日期：2023-10-28') 81 | args = parser.parse_args() 82 | 83 | port = args.port 84 | table_names = args.tables 85 | log_file = args.log 86 | 87 | # 创建日志记录器 88 | logger = logging.getLogger('mysql_packet_logger') 89 | logger.setLevel(logging.INFO) 90 | 91 | # 创建文件输出处理器 92 | file_handler = logging.FileHandler(log_file, 'a') 93 | file_handler.setLevel(logging.INFO) 94 | logger.addHandler(file_handler) 95 | 96 | # 如果设置了-c参数，创建并添加终端输出处理器 97 | if args.console: 98 | console_handler = logging.StreamHandler() 99 | console_handler.setLevel(logging.INFO) 100 | logger.addHandler(console_handler) 101 | 102 | start_time = time.time() # 记录抓取开始时间 103 | 104 | if table_names: 105 | try: 106 | sniff_mysql_packets(port, table_names, args.runtime) 107 | except KeyboardInterrupt: 108 | print("\nSniffing operation stopped") 109 | sys.exit(0) 110 | else: 111 | print("No table names specified. Capturing all tables...") 112 | try: 113 | sniff_mysql_packets(port, [], args.runtime) 114 | except KeyboardInterrupt: 115 | print("\nSniffing operation stopped") 116 | sys.exit(0) 117 | 118 | end_time = time.time() # 记录抓取结束时间 119 | total_time = end_time - start_time # 计算抓取总时间 120 | 121 | print(f"Packet sniffing completed. Total time: {total_time:.2f} seconds.") 122 | 123 | -------------------------------------------------------------------------------- /mysql_sniffer_centos6: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hcymysql/mysql_sniffer/f085843f37a76623a14d24ba0eec102c1e819e4c/mysql_sniffer_centos6 --------------------------------------------------------------------------------