├── .gitattributes ├── .gitignore ├── LICENSE ├── README.md ├── README_IMGS ├── 30万分数.png ├── 命名约定.jpg ├── 多项式回归(degree=7).png ├── 投影距离.jpg └── 线性回归.png ├── assests ├── center_black.png ├── center_white.png └── piece.png ├── requirements.txt ├── run.py ├── src ├── __init__.py ├── adb.py ├── jump.py └── model.py └── training.txt /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | MANIFEST 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | .pytest_cache/ 49 | 50 | # Translations 51 | *.mo 52 | *.pot 53 | 54 | # Django stuff: 55 | *.log 56 | local_settings.py 57 | db.sqlite3 58 | 59 | # Flask stuff: 60 | instance/ 61 | .webassets-cache 62 | 63 | # Scrapy stuff: 64 | .scrapy 65 | 66 | # Sphinx documentation 67 | docs/_build/ 68 | 69 | # PyBuilder 70 | target/ 71 | 72 | # Jupyter Notebook 73 | .ipynb_checkpoints 74 | 75 | # pyenv 76 | .python-version 77 | 78 | # celery beat schedule file 79 | celerybeat-schedule 80 | 81 | # SageMath parsed files 82 | *.sage.py 83 | 84 | # Environments 85 | .env 86 | .venv 87 | env/ 88 | venv/ 89 | ENV/ 90 | env.bak/ 91 | venv.bak/ 92 | 93 | # Spyder project settings 94 | .spyderproject 95 | .spyproject 96 | 97 | # Rope project settings 98 | .ropeproject 99 | 100 | # mkdocs documentation 101 | /site 102 | 103 | # mypy 104 | .mypy_cache/ 105 | 106 | .DS_Store 107 | tmp/ 108 | assests/font.ttf 109 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Invoker 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # WechatJump 2 | 3 | adb + pillow + opencv + sklearn 实现的微信跳一跳机器人，分数可达 30 万以上。 4 | 5 | **反对一切使用外挂的行为，该项目只为学习目的，没有加入反作弊代码，以后也不会有：）** 6 | 7 | ![30万分数](./README_IMGS/30万分数.png) 8 | 9 | [TOC] 10 | 11 | # 命名约定 12 | 13 | 为便于描述，做以下约定： 14 | 15 | ![命名约定](./README_IMGS/命名约定.jpg) 16 | 17 | # 特性 18 | 19 | 与年初的跳一跳机器人相比，进行了以下改进： 20 | 21 | - adb 截图后图像直接传输给上位机，不经过手机存储和 SD 卡；上位机一直在内存中处理图像，减少 IO 耗时，提高效率。 22 | - 使用上一次截图的目标棋盘（也是本次截图的起始棋盘）作为模版，使用 OpenCV 模版匹配寻找起始棋盘的中心坐标，根据中心坐标和棋子中心坐标的相对位置、上次计算的跳跃距离计算实际跳跃距离。替代其他项目中在“跳跃后视角平移前”进行截图的方法，不需要反复尝试调整合适的 “Magic Number”，且每次跳跃只需截图一次。简而言之，省力✌️，提高准确性，提高效率； 23 | - 为加快运行速度，只在程序初始化时训练模型，但程序运行时输出的数据可用于回归模型训练。 24 | - 计算两点间距离时，使用两点在跳跃方向（与水平线夹角 30 度）上的投影距离，与欧式距离相比更加精确。即下图中 OB 的距离（图片来源：[腾讯WeTest团队](http://wetest.qq.com/lab/view/364.html)）： 25 | 26 | ![投影距离](./README_IMGS/投影距离.jpg) 27 | 28 | # 使用的技术和库 29 | 30 | - OpenCV —— 模版匹配，Canny 边缘检测 31 | - Pillow —— ImageDraw 模块 32 | - sklearn —— 线性回归，多项式回归 33 | 34 | # 使用说明 35 | 36 | ## 环境准备 37 | 38 | - 安装 OpenCV 和 adb 工具，MacOS 可以使用 brew 安装，其他系统请参考 [OpenCV 文档](https://docs.opencv.org/3.4.3/da/df6/tutorial_py_table_of_contents_setup.html) 和 [Android 调试桥](https://developer.android.com/studio/command-line/adb?hl=zh-cn)： 39 | 40 | ``` 41 | brew install opencv 42 | brew cask install android-platform-tools 43 | ``` 44 | 45 | - 安装依赖 46 | 47 | ```shell 48 | pip install -r requirements.txt 49 | ``` 50 | 51 | ## 设置项 52 | 53 | 在项目根目录 `run.py` 中，有几个配置项需要修改： 54 | 55 | - `ADB_DEVICE_SERIAL` 通过 adb devices 命令得到的手机序列号； 56 | - `TRAINING_DATASET` 训练数据文件路径； 57 | - `TRAINING_MODEL` “LR” 线性回归模型，“PR” 一元多项式回归模型； 58 | - `TRAINING_PR_DEGREE` 一元多项式回归模型时，变量的最高指数； 59 | - `JUMP_DELAY` 跳跃后与下次截图之间的时间间隔，单位秒。设置太小会在下次截图时引入中心连跳的涟漪特效，会对边缘检测造成干扰，不放心的话就往大了设置； 60 | - `SHOW_MARKED_IMG` 是否在跳跃时展示标注数据的 RGB 图像。 61 | 62 | 另外，如果选择展示标注数据的 RGB 图像，还需要向 `assests` 目录中放入 TTF 字体文件（需支持中英文），并命名为 `font.ttf`，用于标注图像。由于版权问题，项目中不包含该字体文件。 63 | 64 | ## 分辨率 65 | 66 | 由于模版匹配的特性（踩过的坑第 2 点），不同分辨率手机可能需要重新采集模版图片，模版图片在 `assests` 目录下。 67 | 68 | 本项目在分辨率为 2160*1080 的小米 6x 上运行正常，理论上相同分辨率的手机都可以直接使用项目自带的模版图片，无需重新采集。 69 | 70 | ## 运行 71 | 72 | 将跳一跳小程序打开，并点击开始游戏按钮进入游戏界面。运行： 73 | 74 | ``` 75 | python3 run.py 76 | ``` 77 | 78 | 输出数据格式为： 79 | 80 | ``` 81 | [上次跳跃实际距离 pixel] [上次跳跃按压时间 ms] [上次跳跃是否跳中棋盘中心] 82 | ``` 83 | 84 | 例如： 85 | 86 | ``` 87 | 293.0696181049753 445 True 88 | ``` 89 | 90 | 表示上次跳跃按压屏幕 445 ms，实际跳跃距离为 293.0696181049753 像素，且跳中了棋盘中心。 91 | 92 | **该数据格式与训练数据格式相同**，因此程序输出数据可保存用作下次启动程序时的模型训练数据。 93 | 94 | 建议运行时将其保存到文件中： 95 | 96 | ``` 97 | # -u 使 stdout 直接输出，不缓存 98 | python3 -u run.py > run.log 99 | ``` 100 | 101 | # Show Time 102 | 103 | ## 最好成绩 104 | 105 | 目前运行到最高的成绩已经超过 30 万（见上图），跑了一晚上没有要停的意思，只能 Ctrl + C 了，理论上分数无上限。 106 | 107 | 后续再测试每小时大概能跳 23000~24000 分。连跳中心的连击中断造成了一定的分数损失。 108 | 109 | ## 线性回归拟合 110 | 111 | ![线性回归](./README_IMGS/线性回归.png) 112 | 113 | ## 多项式回归拟合(degree=7) 114 | 115 | ![多项式回归](./README_IMGS/多项式回归(degree=7).png) 116 | 117 | # 踩过的坑 118 | 119 | 1. 由于图像坐标系从左上角开始，Y轴在向下的方向上是递增的，所以斜率与普通直角坐标系相反。也就是说，从左下到右上方向的直线实际为负斜率。 120 | 2. OpenCV 模版匹配对缩放图片处理效果不好，导致从小程序文件解包出来的棋子图像无法使用，必须获取当前分辨率下的棋子图像和中心点图像作为模版； 121 | 122 | # 还可以改进的地方 123 | 124 | 1. 使用并发同时进行上次跳跃评估和本次跳跃计算，提高运行速度； 125 | 2. 自动适配不同分辨率的手机屏幕； -------------------------------------------------------------------------------- /README_IMGS/30万分数.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mrchi/WechatJump/6468f52270d1e6fd7591f79fcc68c66978aa9f54/README_IMGS/30万分数.png -------------------------------------------------------------------------------- /README_IMGS/命名约定.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mrchi/WechatJump/6468f52270d1e6fd7591f79fcc68c66978aa9f54/README_IMGS/命名约定.jpg -------------------------------------------------------------------------------- /README_IMGS/多项式回归(degree=7).png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mrchi/WechatJump/6468f52270d1e6fd7591f79fcc68c66978aa9f54/README_IMGS/多项式回归(degree=7).png -------------------------------------------------------------------------------- /README_IMGS/投影距离.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mrchi/WechatJump/6468f52270d1e6fd7591f79fcc68c66978aa9f54/README_IMGS/投影距离.jpg -------------------------------------------------------------------------------- /README_IMGS/线性回归.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mrchi/WechatJump/6468f52270d1e6fd7591f79fcc68c66978aa9f54/README_IMGS/线性回归.png -------------------------------------------------------------------------------- /assests/center_black.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mrchi/WechatJump/6468f52270d1e6fd7591f79fcc68c66978aa9f54/assests/center_black.png -------------------------------------------------------------------------------- /assests/center_white.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mrchi/WechatJump/6468f52270d1e6fd7591f79fcc68c66978aa9f54/assests/center_white.png -------------------------------------------------------------------------------- /assests/piece.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mrchi/WechatJump/6468f52270d1e6fd7591f79fcc68c66978aa9f54/assests/piece.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | cycler==0.10.0 2 | kiwisolver==1.0.1 3 | matplotlib==2.2.3 4 | nose==1.3.7 5 | numpy==1.15.1 6 | Pillow==5.2.0 7 | pyparsing==2.2.0 8 | python-dateutil==2.7.3 9 | pytz==2018.5 10 | scikit-learn==0.19.2 11 | scipy==1.1.0 12 | six==1.11.0 13 | -------------------------------------------------------------------------------- /run.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # coding=utf-8 3 | 4 | from src.model import MachineLearningModel 5 | from src.jump import WechatJump 6 | from src.adb import PyADB 7 | 8 | # 请根据实际情况修改以下配置项 9 | # ------------------------------------------------------------------------------ 10 | ADB_DEVICE_SERIAL = "48a666d9" # adb devices 命令得到的手机序列号 11 | 12 | TRAINING_DATASET = "./training.txt" # 训练数据文件路径 13 | TRAINING_MODEL = "LR" # LR 线性回归模型，PR 一元多项式回归模型 14 | TRAINING_PR_DEGREE = 3 # 一元多项式回归模型时，变量的最高指数 15 | 16 | JUMP_DELAY = 1.1 # 跳跃后与下次截图之间的时间间隔，单位秒 17 | SHOW_MARKED_IMG = False # 展示标注数据的 RGB 图像 18 | # ------------------------------------------------------------------------------ 19 | 20 | 21 | if __name__ == '__main__': 22 | # adb 初始化 23 | adb = PyADB(ADB_DEVICE_SERIAL) 24 | 25 | # 训练回归模型 26 | model = MachineLearningModel(TRAINING_DATASET, only_center=True) 27 | if TRAINING_MODEL == "LR": 28 | model.train_linear_regression_model() 29 | elif TRAINING_MODEL == "PR": 30 | model.train_polynomial_regression_model(degree=TRAINING_PR_DEGREE) 31 | 32 | # 跳一跳类初始化 33 | wj = WechatJump(adb, model) 34 | 35 | # 运行 36 | wj.run(jump_delay=JUMP_DELAY, show_img=SHOW_MARKED_IMG) 37 | -------------------------------------------------------------------------------- /src/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mrchi/WechatJump/6468f52270d1e6fd7591f79fcc68c66978aa9f54/src/__init__.py -------------------------------------------------------------------------------- /src/adb.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # coding=utf-8 3 | 4 | import functools 5 | import subprocess 6 | from io import BytesIO 7 | 8 | from PIL import Image 9 | 10 | __all__ = ["PyADB"] 11 | 12 | _sysrun = functools.partial( 13 | subprocess.run, 14 | stdout=subprocess.PIPE, 15 | stderr=subprocess.PIPE, 16 | ) 17 | 18 | 19 | class ADBError(Exception): 20 | pass 21 | 22 | 23 | class ConnectionError(ADBError): 24 | pass 25 | 26 | 27 | class LongTapError(ADBError): 28 | pass 29 | 30 | 31 | class ShortTapError(ADBError): 32 | pass 33 | 34 | 35 | class PyADB: 36 | 37 | def __init__(self, device_serial): 38 | self.device_serial = device_serial 39 | 40 | def connect(self, ip, port=5555): 41 | """连接网络adb调试设备""" 42 | cmd = f"adb -s {self.device_serial} connect {ip}:{port}".split() # noqa 43 | try: 44 | result = _sysrun(cmd, timeout=2) 45 | returncode = result.returncode 46 | except subprocess.TimeoutExpired as e: 47 | errmsg = f"Connect {ip}:{port} timeout." 48 | else: 49 | if result.returncode != 0: 50 | raise ConnectionError(result.stderr.decode.strip()) 51 | return "connected" in output, output 52 | raise ConnectionError(errmsg) 53 | 54 | def get_resolution(self): 55 | """获取屏幕分辨率""" 56 | cmd = f"adb -s {self.device_serial} exec-out wm size".split() 57 | result = _sysrun(cmd) 58 | w, h = result.stdout.decode().split("Physical size: ")[-1].split("x") 59 | return (int(w), int(h)) 60 | 61 | def screencap(self): 62 | """截图，输出为 Pillow.Image 对象""" 63 | cmd = f"adb -s {self.device_serial} exec-out screencap -p".split() 64 | result = _sysrun(cmd) 65 | img = Image.open(BytesIO(result.stdout)) 66 | return img 67 | 68 | def short_tap(self, cord): 69 | """短按点击，坐标为 (x, y) 格式""" 70 | cmd = f"adb -s {self.device_serial} exec-out input tap {cord[0]} {cord[1]}".split() 71 | result = _sysrun(cmd) 72 | if result.returncode != 0: 73 | raise ShortTapError(result.stderr.decode.strip()) 74 | 75 | def long_tap(self, cord, duration): 76 | """长按, duration单位为ms，坐标为 (x, y) 格式""" 77 | cmd = f"adb -s {self.device_serial} exec-out input swipe {cord[0]} {cord[1]} {cord[0]} {cord[1]} {duration}".split() 78 | result = _sysrun(cmd) 79 | if result.returncode != 0: 80 | raise LongTapError(result.stderr.decode.strip()) 81 | -------------------------------------------------------------------------------- /src/jump.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | import os 4 | import time 5 | import math 6 | 7 | import cv2 8 | import numpy as np 9 | from PIL import ImageDraw, ImageFont 10 | 11 | from .adb import PyADB 12 | from .model import MachineLearningModel 13 | 14 | __all__ = ["WechatJump"] 15 | 16 | ASSESTS_DIR = os.path.join(os.path.dirname(__file__), "../assests") # assests文件目录 17 | PIECE_IMG = os.path.join(ASSESTS_DIR, "piece.png") # 棋子图片 18 | CENTER_BLACK_IMG = os.path.join(ASSESTS_DIR, "center_black.png") # 黑色背景中心点 19 | CENTER_WHITE_IMG = os.path.join(ASSESTS_DIR, "center_white.png") # 白色背景中心点 20 | TTF_FONT_FILE = os.path.join(ASSESTS_DIR, "font.ttf") # 图片标注使用的ttf字体 21 | 22 | NULL_POS = np.array([0, 0]) # 无法确定坐标时的默认坐标 23 | 24 | 25 | class WechatJump: 26 | """ 所有的坐标都是 (x, y) 格式的，但是在 opencv 的数组中是 (y, x) 格式的""" 27 | def __init__(self, adb, model): 28 | self.adb = adb 29 | self.model = model 30 | # 获取屏幕分辨率，计算 开始按钮、再玩一局按钮和好友排行榜界面返回按钮 的坐标 31 | self.resolution = np.array(self.adb.get_resolution()) 32 | self.start_btn = self.resolution * np.array([0.5, 0.67]) 33 | self.again_btn = self.resolution * np.array([0.62, 0.79]) 34 | self.top_chart_back_btn = self.resolution * np.array([0.07, 0.87]) 35 | # 读取棋子图片和中心点图片 36 | self.piece = cv2.imread(PIECE_IMG, cv2.IMREAD_GRAYSCALE) 37 | self.center_black = cv2.imread(CENTER_BLACK_IMG, cv2.IMREAD_GRAYSCALE) 38 | self.center_white = cv2.imread(CENTER_WHITE_IMG, cv2.IMREAD_GRAYSCALE) 39 | # 设置偏移量，模版匹配时得到的坐标是模版左上角像素的坐标，计算棋子位置和中心点位置时要进行偏移 40 | self.piece_delta = np.array([38, 186]) 41 | self.center_delta = np.array([19, 15]) 42 | self.init_attrs() 43 | 44 | def start_game(self): 45 | """点击开始游戏按钮""" 46 | self.adb.short_tap(self.start_btn) 47 | 48 | def another_game(self): 49 | """点击再玩一局按钮""" 50 | self.adb.short_tap(self.top_chart_back_btn) 51 | self.adb.short_tap(self.again_btn) 52 | 53 | @staticmethod 54 | def match_template(img, tpl, threshold=0.8): 55 | """opencv模版匹配，传入图像都为灰度图像。 56 | 要使用当前分辨率下的图片模版（棋子、中心点图片等），否则匹配结果很差。 57 | """ 58 | result = cv2.matchTemplate(img, tpl, cv2.TM_CCOEFF_NORMED) 59 | _, maxVal, _, maxLoc = cv2.minMaxLoc(result) 60 | return np.array(maxLoc) if maxVal >= threshold else NULL_POS 61 | 62 | @staticmethod 63 | def calc_distance(a, b, jump_right): 64 | """两点在跳跃方向（认为与水平线夹角为 30 度）上投影的距离，比欧式距离精确。""" 65 | if jump_right: 66 | distance = abs((a[1]-b[1]) - (a[0]-b[0]) / math.sqrt(3)) 67 | else: 68 | distance = abs((a[1]-b[1]) + (a[0]-b[0]) / math.sqrt(3)) 69 | # # 欧式距离 70 | # distance = np.sqrt(np.sum(np.square(a-b))) 71 | return distance 72 | 73 | def match_center_tpl(self, img): 74 | """使用模版匹配寻找中心点坐标，中心点在上次跳中棋盘中心后出现在目标棋盘中心。""" 75 | black_match_pos = self.match_template(img, self.center_black) 76 | white_match_pos = self.match_template(img, self.center_white) 77 | if black_match_pos.any(): 78 | self.target_pos = black_match_pos + self.center_delta 79 | self.on_center = True 80 | elif white_match_pos.any(): 81 | self.target_pos = white_match_pos + self.center_delta 82 | self.on_center = True 83 | else: 84 | self.target_pos = NULL_POS 85 | self.on_center = False 86 | return self.target_pos 87 | 88 | def init_attrs(self): 89 | """初始化变量""" 90 | # 测量跳跃距离 91 | self.last_distance = self.distance if hasattr(self, "distance") else None 92 | self.distance = None 93 | # 跳跃时间 94 | self.last_duration = self.duration if hasattr(self, "duration") else None 95 | self.duration = None 96 | # 向右跳跃标志位 97 | self.last_jump_right = self.jump_right if hasattr(self, "jump_right") else None 98 | self.jump_right = None 99 | # 目标棋盘的图像 100 | self.last_target_img = self.target_img if hasattr(self, "target_img") else NULL_POS 101 | self.target_img = NULL_POS 102 | # 棋子坐标、目标棋盘中心坐标，当前棋盘中心坐标，当前棋盘上顶点坐标 103 | self.piece_pos = NULL_POS 104 | self.target_pos = NULL_POS 105 | self.start_pos = NULL_POS 106 | self.top_pos = NULL_POS 107 | # 上次跳跃实际距离 108 | self.last_actual_distance = None 109 | # 上次跳跃跳中中心标志位 110 | self.on_center = None 111 | 112 | def get_piece_pos(self, img): 113 | """使用模版匹配寻找棋子位置，同时判断跳跃方向。""" 114 | match_pos = self.match_template(img, self.piece, 0.7) 115 | if not match_pos.any(): 116 | raise ValueError("无法定位棋子") 117 | self.piece_pos = match_pos + self.piece_delta 118 | 119 | # 计算跳跃方向。若棋子在图片左侧半区，则向右跳跃（不存在在中线上的情况） 120 | self.jump_right = self.piece_pos[0] < self.resolution[0] // 2 121 | 122 | return self.piece_pos 123 | 124 | def get_target_pos(self, img): 125 | """ 126 | 获取目标棋盘中心点坐标、上顶点坐标。步骤： 127 | 128 | 1. 使用模版匹配寻找中心点。 129 | 2. 使用边缘检测寻找目标棋盘上顶点坐标，灰度图像 -> 高斯模糊 -> Canny边缘图像 -> 逐行遍历。 130 | 3. 如果第1步模版匹配没有找到中心点，则寻找目标棋盘下顶点并计算中心点。 131 | """ 132 | self.match_center_tpl(img) 133 | 134 | # 高斯模糊后，处理成Canny边缘图像 135 | img = cv2.GaussianBlur(img, (5, 5), 0) 136 | img = cv2.Canny(img, 1, 10) 137 | 138 | # 有时棋子图像的高度高于目标棋盘，为去掉棋子对判断的影响，抹掉棋子的边缘图像（将像素值置为0） 139 | # 图像数组的索引是先y后x，即 img[y1:y2, x1:x2] 的形式 140 | # +2 -2 扩大抹除范围，保证完全抹掉棋子 141 | img[ 142 | self.piece_pos[1]-self.piece_delta[1]-2: self.piece_pos[1]+2, 143 | self.piece_pos[0]-self.piece_delta[0]-2: self.piece_pos[0]+self.piece_delta[0]+2, 144 | ] = 0 145 | 146 | # 为避免屏幕上半部分分数和小程序按钮的影响 147 | # 从 1/3*H 的位置开始向下逐行遍历到 2/3*H，寻找目标棋盘的上顶点 148 | y_start = self.resolution[1] // 3 149 | y_stop = self.resolution[1] * 2 // 3 150 | 151 | # 上顶点的 y 坐标 152 | for y in range(y_start, y_stop): 153 | if img[y].any(): 154 | y_top = y 155 | break 156 | else: 157 | raise ValueError("无法定位目标棋盘上顶点") 158 | 159 | # 上顶点的 x 坐标，也是中心点的 x 坐标 160 | x = int(round(np.mean(np.nonzero(img[y_top])))) 161 | self.top_pos = np.array([x, y_top]) 162 | 163 | # 如果模版匹配已经找到了目标棋盘中心点，就不需要再继续寻找下顶点继而确定中心点 164 | if self.target_pos.any(): 165 | return self.target_pos 166 | 167 | # 下顶点的 y 坐标，+40是为了消除多圆环类棋盘的干扰 168 | for y in range(y_top+40, y_stop): 169 | if img[y, x] or img[y, x-1]: 170 | y_bottom = y 171 | break 172 | else: 173 | raise ValueError("无法定位目标棋盘下顶点") 174 | 175 | # 由上下顶点 y 坐标取中点获得中心点 y 坐标 176 | self.target_pos = np.array([x, (y_top + y_bottom) // 2]) 177 | return self.target_pos 178 | 179 | def get_start_pos(self, img): 180 | """通过模版匹配，获取起始棋盘中心坐标""" 181 | # 上次跳跃截图的目标棋盘，就是本次跳跃的起始棋盘（棋子所在的棋盘） 182 | if self.last_target_img.any(): 183 | match_pos = self.match_template(img, self.last_target_img, 0.7) 184 | if match_pos.any(): 185 | shape = self.last_target_img.shape 186 | start_pos = match_pos + np.array([shape[1]//2, 0]) 187 | # 如果起始棋盘坐标与当前棋子坐标差距过大，则认为有问题，丢弃 188 | if (np.abs(start_pos-self.piece_pos) < np.array([100, 100])).all(): 189 | self.start_pos = start_pos 190 | return self.start_pos 191 | 192 | def review_last_jump(self): 193 | """评估上次跳跃参数，计算实际跳跃距离。""" 194 | # 如果这些属性不存在，就无法进行评估 195 | if self.last_distance \ 196 | and self.last_duration \ 197 | and self.start_pos.any() \ 198 | and self.last_jump_right is not None: 199 | pass 200 | else: 201 | return 202 | 203 | # 计算棋子和起始棋盘中心的偏差距离 204 | d = self.calc_distance(self.start_pos, self.piece_pos, self.last_jump_right) 205 | 206 | # 计算实际跳跃距离，这里要分情况讨论跳过头和没跳到两种情况讨论 207 | k = 1 / math.sqrt(3) if self.last_jump_right else -1 / math.sqrt(3) 208 | # 没跳到，实际距离 = 上次测量距离 - 偏差距离 209 | if self.piece_pos[1] > k*(self.piece_pos[0]-self.start_pos[0]) + self.start_pos[1]: 210 | self.last_actual_distance = self.last_distance - d 211 | # 跳过头，实际距离 = 上次测量距离 + 偏差距离 212 | elif self.piece_pos[1] < k*(self.piece_pos[0]-self.start_pos[0]) + self.start_pos[1]: 213 | self.last_actual_distance = self.last_distance + d 214 | # 刚刚好 215 | else: 216 | self.last_actual_distance = self.last_distance 217 | 218 | print(self.last_actual_distance, self.last_duration, self.on_center) 219 | 220 | def get_target_img(self, img): 221 | """获取当前目标棋盘的图像。""" 222 | half_height = self.target_pos[1] - self.top_pos[1] 223 | half_width = int(round(half_height * math.sqrt(3))) 224 | self.target_img = img[ 225 | self.target_pos[1]: self.target_pos[1]+half_height+100, 226 | self.target_pos[0]-half_width-3: self.target_pos[0]+half_width+3, 227 | ] 228 | return self.target_img 229 | 230 | def jump(self): 231 | """跳跃，并存储本次目标跳跃距离和按压时间""" 232 | # 计算棋子坐标和目标棋盘中心坐标之间距离 233 | self.distance = self.calc_distance(self.piece_pos, self.target_pos, self.jump_right) 234 | # 计算长按时间 235 | self.duration = int(round(self.model.predict(self.distance))) 236 | # 跳！ 237 | self.adb.long_tap(self.resolution // 2, self.duration) 238 | 239 | def mark_img(self, img_rgb): 240 | """在 RGB 图像上标注数据，会改变传入的 img_rgb""" 241 | draw = ImageDraw.Draw(img_rgb) 242 | # 棋子中心点，红色 243 | draw.line((0, self.piece_pos[1], self.resolution[0], self.piece_pos[1]), "#ff0000") 244 | draw.line((self.piece_pos[0], 0, self.piece_pos[0], self.resolution[1]), "#ff0000") 245 | # 目标棋盘中心点，蓝色 246 | draw.line((0, self.target_pos[1], self.resolution[0], self.target_pos[1]), "#0000ff") 247 | draw.line((self.target_pos[0], 0, self.target_pos[0], self.resolution[1]), "#0000ff") 248 | # 当前棋盘中心点，黑色 249 | draw.line((0, self.start_pos[1], self.resolution[0], self.start_pos[1]), "#000000") 250 | draw.line((self.start_pos[0], 0, self.start_pos[0], self.resolution[1]), "#000000") 251 | 252 | draw.multiline_text( 253 | (20, 20), 254 | "\n".join([ 255 | "[上次跳跃数据]", 256 | f"向右跳跃: {self.last_jump_right}", # noqa 257 | f"落点在中心: {self.on_center}", # noqa 258 | f"应跳跃距离: {self.last_distance}", # noqa 259 | f"实跳跃距离: {self.last_actual_distance}", # noqa 260 | f"按压时间: {self.last_duration}", # noqa 261 | ]), 262 | fill='#000000', 263 | font=ImageFont.truetype(TTF_FONT_FILE, 45) 264 | ) 265 | 266 | def single_run(self): 267 | """单次运行""" 268 | img_rgb = self.adb.screencap() 269 | img = cv2.cvtColor(np.asarray(img_rgb), cv2.COLOR_RGB2GRAY) 270 | self.init_attrs() 271 | self.get_piece_pos(img) 272 | self.get_target_pos(img) 273 | self.get_start_pos(img) 274 | self.get_target_img(img) 275 | self.review_last_jump() 276 | self.jump() 277 | return img_rgb 278 | 279 | def run(self, jump_delay=1.1, show_img=False): 280 | """循环运行""" 281 | while True: 282 | img_rgb = self.single_run() 283 | time.sleep(self.duration/5000 + jump_delay) 284 | if show_img: 285 | self.mark_img(img_rgb) 286 | img_rgb.show() 287 | -------------------------------------------------------------------------------- /src/model.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | 3 | import numpy as np 4 | from sklearn.linear_model import LinearRegression 5 | from sklearn.preprocessing import PolynomialFeatures 6 | 7 | __all__ = ["MachineLearningModel"] 8 | 9 | 10 | class MachineLearningModel: 11 | """机器学习回归分析""" 12 | def __init__(self, dataset_file, only_center=False): 13 | self.read_training_datasets(dataset_file, only_center) 14 | 15 | def read_training_datasets(self, dataset_file, only_center=False): 16 | """读取训练数据""" 17 | with open(dataset_file, "r") as f: 18 | data = f.readlines() 19 | 20 | dataset_X = [] 21 | dataset_Y = [] 22 | 23 | for line in data: 24 | line = line.strip() 25 | if not line: 26 | continue 27 | x, y, z = line.split() 28 | # 只取跳中了中心的数据 29 | if only_center and z == "False": 30 | continue 31 | dataset_X.append(float(x)) 32 | dataset_Y.append(int(y)) 33 | 34 | self.dataset_X = np.array(dataset_X).reshape([len(dataset_X), 1]) 35 | self.dataset_Y = np.array(dataset_Y) 36 | 37 | def train_linear_regression_model(self): 38 | """训练线性回归模型""" 39 | linear = LinearRegression() 40 | linear.fit(self.dataset_X, self.dataset_Y) 41 | self.predict = lambda x: linear.predict(x)[0] 42 | 43 | def train_polynomial_regression_model(self, degree): 44 | """训练多项式回归模型""" 45 | poly_feat = PolynomialFeatures(degree=degree) 46 | x_tranformed = poly_feat.fit_transform(self.dataset_X) 47 | linear = LinearRegression() 48 | linear.fit(x_tranformed, self.dataset_Y) 49 | self.predict = lambda x: linear.predict(poly_feat.transform(x))[0] 50 | -------------------------------------------------------------------------------- /training.txt: -------------------------------------------------------------------------------- 1 | 293.0696181049753 445 True 2 | 383.2576396401454 561 True 3 | 235.20210464549405 342 True 4 | 614.7276934780705 868 True 5 | 639.0785464391564 877 True 6 | 530.3435195201938 750 True 7 | 554.1281292110203 764 True 8 | 491.41531628991834 709 True 9 | 577.6751345948129 799 True 10 | 270.09996299037243 399 True 11 | 316.19397375795745 474 True 12 | 639.4293994002422 876 True 13 | 615.1917950932083 869 True 14 | 650.625551822949 904 True 15 | 590.0259875558987 830 True 16 | 502.7247173666769 733 False 17 | 490.13626009855693 700 True 18 | 603.3768405169847 835 True 19 | 410.07259421636905 581 True 20 | 601.7994902477951 828 True 21 | 443.5892547147644 645 True 22 | 431.5781477158342 613 True 23 | 478.6003617136947 697 True 24 | 268.5226127211828 414 True 25 | 491.37386440559095 705 True 26 | 638.8520491310527 897 True 27 | 588.4486372867092 824 True 28 | 268.21321164442435 390 True 29 | 395.453792062852 578 True 30 | 259.77945491468364 373 True 31 | 240.3568051838733 347 True 32 | 238.2850084141488 370 True 33 | 613.6862415937431 848 True 34 | 327.6995272574226 492 True 35 | 663.3990545148451 925 True 36 | 565.0148805569685 805 True 37 | 408.9896904477143 560 True 38 | 421.7328482542134 606 True 39 | 638.8520491310526 881 True 40 | 601.799490247795 832 True 41 | 540.1999259807449 759 True 42 | 639.2746988618632 910 False 43 | 314.37901918173384 477 True 44 | 360.55593371797363 531 True 45 | 651.2029020921385 903 True 46 | 615.1503432088809 868 True 47 | 640.0785464391565 890 True 48 | 570.7883832488648 807 True 49 | 292.02816622064796 437 True 50 | 589.0259875558988 835 True 51 | 502.14736709748723 703 True 52 | 394.2576396401454 562 True 53 | 589.1806880942779 824 True 54 | 395.4537920628521 561 True 55 | 552.8601800185893 761 True 56 | 479.6003617136947 687 True 57 | 564.2828297493996 801 True 58 | 246.77945491468367 382 True 59 | 314.61662348876786 468 True 60 | 578.2524848640026 823 True 61 | 603.3768405169847 837 True 62 | 602.9541907861743 835 True 63 | 359.7106342563529 522 True 64 | 280.1414148746999 427 True 65 | 553.5507789418307 776 True 66 | 540.7772762499344 760 True 67 | 479.60036171369467 694 True 68 | 247.85125168440817 360 True 69 | 627.0785464391564 884 True 70 | 347.7824310260774 516 True 71 | 466.36275740666065 681 True 72 | 303.7602170282168 462 True 73 | 316.61662348876786 464 True 74 | 258.1717597600969 407 True 75 | 192.4174949546675 295 True 76 | 347.78243102607735 503 True 77 | 203.61364737737415 314 False 78 | 639.8520491310527 886 False 79 | 327.96747644985373 491 True 80 | 420.15549798502377 605 True 81 | 554.8187281342618 769 True 82 | 491.9512146747806 716 False 83 | 602.3768405169848 830 True 84 | 180.108093877909 256 False 85 | 227.70765814495917 335 True 86 | 552.1695810953477 753 True 87 | 468.82685902179844 689 False 88 | --------------------------------------------------------------------------------