├── requirements.txt ├── images ├── logo.jpg ├── opencv.png ├── python.png ├── sign.jpg ├── examples.jpg ├── captcha-1.jpg ├── captcha-2.jpg ├── captcha-3.jpg ├── first-cap.jpg ├── huggingface.png ├── left_number.jpg ├── other-type.png ├── persian-cap.jpg ├── math-captcha2.jpg ├── other-types1.jpg ├── other-types2.png ├── other-types3.jpg ├── other-types4.jpg ├── other-types5.jpg ├── other-types6.jpg ├── right_number.jpg ├── captcha-example.jpg └── captcha-example4.jpg ├── LICENSE ├── MathCaptchaSolver.py ├── .gitignore └── README.md /requirements.txt: -------------------------------------------------------------------------------- 1 | transformers==4.36.2 2 | opencv-python==4.8.0.74 3 | numpy 4 | -------------------------------------------------------------------------------- /images/logo.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/logo.jpg -------------------------------------------------------------------------------- /images/opencv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/opencv.png -------------------------------------------------------------------------------- /images/python.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/python.png -------------------------------------------------------------------------------- /images/sign.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/sign.jpg -------------------------------------------------------------------------------- /images/examples.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/examples.jpg -------------------------------------------------------------------------------- /images/captcha-1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/captcha-1.jpg -------------------------------------------------------------------------------- /images/captcha-2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/captcha-2.jpg -------------------------------------------------------------------------------- /images/captcha-3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/captcha-3.jpg -------------------------------------------------------------------------------- /images/first-cap.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/first-cap.jpg -------------------------------------------------------------------------------- /images/huggingface.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/huggingface.png -------------------------------------------------------------------------------- /images/left_number.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/left_number.jpg -------------------------------------------------------------------------------- /images/other-type.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/other-type.png -------------------------------------------------------------------------------- /images/persian-cap.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/persian-cap.jpg -------------------------------------------------------------------------------- /images/math-captcha2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/math-captcha2.jpg -------------------------------------------------------------------------------- /images/other-types1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/other-types1.jpg -------------------------------------------------------------------------------- /images/other-types2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/other-types2.png -------------------------------------------------------------------------------- /images/other-types3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/other-types3.jpg -------------------------------------------------------------------------------- /images/other-types4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/other-types4.jpg -------------------------------------------------------------------------------- /images/other-types5.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/other-types5.jpg -------------------------------------------------------------------------------- /images/other-types6.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/other-types6.jpg -------------------------------------------------------------------------------- /images/right_number.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/right_number.jpg -------------------------------------------------------------------------------- /images/captcha-example.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/captcha-example.jpg -------------------------------------------------------------------------------- /images/captcha-example4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AmireNoori/MathCaptchaSolver/HEAD/images/captcha-example4.jpg -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 Amir Noori 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /MathCaptchaSolver.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import numpy as np 3 | from transformers import pipeline 4 | 5 | pipe = pipeline("image-to-text", model="microsoft/trocr-large-printed") 6 | 7 | class CaptchaSolver: 8 | def __init__(self, image_path): 9 | self.image = cv2.imread(image_path) 10 | self.kernel = np.ones((2, 2), np.uint8) 11 | 12 | def enhance_legibility(self, cropped_image): 13 | gray = cv2.cvtColor(cropped_image, cv2.COLOR_BGR2GRAY) 14 | _, mask = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU) 15 | return cv2.erode(cv2.blur(mask, (2, 2)), self.kernel, iterations=1) 16 | 17 | def math_operation(self, left_number, right_number, operation='+'): 18 | if left_number.isdigit() and right_number.isdigit(): 19 | return eval(f"{left_number} {operation} {right_number}") 20 | else: 21 | return None 22 | def math_operation_for_both_signs(self, left_number, right_number): 23 | if left_number.isdigit() and right_number.isdigit(): 24 | left_number = int(left_number) 25 | right_number = int(right_number) 26 | return [left_number - right_number, left_number + right_number] 27 | else: 28 | return None 29 | def resolve(self, left_image, right_image, sign_image, negative_sign_right_image): 30 | sign = pipe(sign_image)[0]['generated_text'] 31 | left_number = pipe(left_image)[0]['generated_text'] 32 | 33 | if sign in {'+', '@', '4','*'}: 34 | right_number = pipe(right_image)[0]['generated_text'] 35 | return self.math_operation(left_number, right_number) 36 | elif sign in {'-', '='}: 37 | right_number = pipe(negative_sign_right_image)[0]['generated_text'] 38 | return self.math_operation(left_number, right_number, '-') 39 | else: 40 | unfixed_right_number = ''.join(char for char in pipe(right_image)[0]['generated_text'] if char.isdigit()) 41 | return self.math_operation_for_both_signs(left_number, unfixed_right_number) 42 | 43 | def solve_captcha(self): 44 | positions = {'left': 5, 'right': 60, 'sign': 39, 'negative_sign_right': 56} 45 | dimensions = {'width': 25, 'height': 20, 'width_sign': 15, 'height_sign': 15, 'width_negative_sign': 18} 46 | 47 | left_image = self.image[7:27, positions['left']:positions['left']+dimensions['width']] 48 | right_image = self.image[7:27, positions['right']:positions['right']+dimensions['width']] 49 | sign_image = self.image[10:25, positions['sign']:positions['sign']+dimensions['width_sign']] 50 | negative_sign_right_image = self.image[7:27, positions['negative_sign_right']:positions['negative_sign_right']+dimensions['width_negative_sign']] 51 | 52 | left_enhanced = self.enhance_legibility(left_image) 53 | right_enhanced = self.enhance_legibility(right_image) 54 | negative_sign_right_enhanced = self.enhance_legibility(negative_sign_right_image) 55 | 56 | cv2.imwrite('left_number.jpg', left_enhanced) 57 | cv2.imwrite('right_number.jpg', right_enhanced) 58 | cv2.imwrite('sign.jpg', sign_image) 59 | cv2.imwrite('negative_sign_right_number.jpg', negative_sign_right_enhanced) 60 | 61 | return self.resolve('left_number.jpg', 'right_number.jpg', 'sign.jpg', 'negative_sign_right_number.jpg') -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | share/python-wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | MANIFEST 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .nox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | nosetests.xml 47 | coverage.xml 48 | *.cover 49 | *.py,cover 50 | .hypothesis/ 51 | .pytest_cache/ 52 | cover/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | .pybuilder/ 76 | target/ 77 | 78 | # Jupyter Notebook 79 | .ipynb_checkpoints 80 | 81 | # IPython 82 | profile_default/ 83 | ipython_config.py 84 | 85 | # pyenv 86 | # For a library or package, you might want to ignore these files since the code is 87 | # intended to run in multiple environments; otherwise, check them in: 88 | # .python-version 89 | 90 | # pipenv 91 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 92 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 93 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 94 | # install all needed dependencies. 95 | #Pipfile.lock 96 | 97 | # poetry 98 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 99 | # This is especially recommended for binary packages to ensure reproducibility, and is more 100 | # commonly ignored for libraries. 101 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 102 | #poetry.lock 103 | 104 | # pdm 105 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 106 | #pdm.lock 107 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 108 | # in version control. 109 | # https://pdm.fming.dev/#use-with-ide 110 | .pdm.toml 111 | 112 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 113 | __pypackages__/ 114 | 115 | # Celery stuff 116 | celerybeat-schedule 117 | celerybeat.pid 118 | 119 | # SageMath parsed files 120 | *.sage.py 121 | 122 | # Environments 123 | .env 124 | .venv 125 | env/ 126 | venv/ 127 | ENV/ 128 | env.bak/ 129 | venv.bak/ 130 | 131 | # Spyder project settings 132 | .spyderproject 133 | .spyproject 134 | 135 | # Rope project settings 136 | .ropeproject 137 | 138 | # mkdocs documentation 139 | /site 140 | 141 | # mypy 142 | .mypy_cache/ 143 | .dmypy.json 144 | dmypy.json 145 | 146 | # Pyre type checker 147 | .pyre/ 148 | 149 | # pytype static type analyzer 150 | .pytype/ 151 | 152 | # Cython debug symbols 153 | cython_debug/ 154 | 155 | # PyCharm 156 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 157 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 158 | # and can be added to the global gitignore or merged into this file. For a more nuclear 159 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 160 | #.idea/ 161 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |
2 | 3 | Logo 4 | 5 | 6 |

Math Captcha Solver

7 | 8 |

9 | A great tool for solving math captchas ! 10 |

11 |
12 | 13 | 14 | 15 |
16 | Table of Contents 17 |
    18 |
  1. 19 | About The Project 20 | 23 |
    2. 24 |
  2. 25 | How it processes 26 | 29 |
    3. 30 |
  3. Usage
    4. 31 |
  4. License
    5. 32 |
  5. Contact
    6. 33 |
  6. Kindness
    7. 34 |
  7. Collections
    8. 35 | 36 |
37 |
38 | 39 | 40 | ## About The Project 41 | 42 | This project is a tool to process mathematical captcha images and calculate the answer using deep learning models and image processing using Python language. 43 | 44 | An example of captchas used in this project: 45 | 46 | ![captcha example](images/captcha-example.jpg) 47 | 48 | If you are looking for solving captchas of the following types: 49 | 50 | captcha example 51 | 52 | visit this webpage : [API Captcha Solver](http://163.5.182.121:4848/) 53 | 54 | also you can message me on Telegram : [@AmireNoori1](https://t.me/AmireNoori1) 55 | 56 | If you are looking for solving captchas of the following types: 57 | 58 | ![captcha example](images/other-type.png) 59 | 60 | go to this branch: 61 | 62 | [MathCaptchaSolver-v2](https://github.com/AmireNoori/MathCaptchaSolver/tree/v2) 63 | 64 | If you are looking for solving captchas of the following types: 65 | 66 | ![captcha example](images/captcha-example4.jpg) 67 | 68 | go to this repo: 69 | 70 | [SVG Captcha Solver](https://github.com/AmireNoori/SVG-Captcha-Solver) 71 | 72 | If you are looking for solving captchas of the following types: 73 | 74 | ![captcha example](images/persian-cap.jpg) 75 | 76 | go to this repo: 77 | 78 | [Persian Captcha Solver](https://github.com/AmireNoori/PersianCaptchaSolver) 79 | 80 | 81 | 82 | go to this repo: 83 | 84 | [Persian Captcha Solver](https://github.com/AmireNoori/PersianCaptchaSolver) 85 | 86 | ### Built With 87 | 88 | The deep learning model used in this tool: It is `TrOCR (large-sized model, fine-tuned on SROIE)`, which you can download from the link [https://huggingface.co/microsoft/trocr-large-printed](https://huggingface.co/microsoft/trocr-large-printed) and read about it. 89 | The TrOCR model is an encoder-decoder model, consisting of an image Transformer as encoder, and a text Transformer as decoder. 90 | Python language and opecv library 91 | * https://huggingface.co 92 | * https://opencv.org 93 | * https://www.python.org 94 | 95 | ## How it processes 96 | 97 | The processing of numbers and signs in the image is performed by the powerful model that I introduced above. But to process this type of captcha images, this model alone was not able to correctly extract the numbers and calculate the operation. So we had to use image processing to improve the result. 98 | 99 | * First, let's see what will be returned if we give the complete captcha image to the model: 100 | 101 | ![captcha example](images/first-cap.jpg) 102 | 103 | * Output 104 | ```sh 105 | [{'generated_text': '40 % & @'}] 106 | ``` 107 | 108 | As you can see, the output returned to us is `40 % & @`, which is not a very good result. 109 | 110 | So we first used morphological operations to erode the image and erase the extra lines to some extent, then we gave the image a more normal state using the blur method. 111 | And finally, we used the crop technique to save two numbers separately with the sign between them with fixed coordinates of each number in all captcha images. 112 | 113 | Now we have three pictures, two numbers and a math symbol : 114 | 115 | ![left number](images/left_number.jpg) ![sign](images/sign.jpg) ![right number](images/right_number.jpg) 116 | 117 | 118 | Next, according to the extracted numbers in the image, the function performs the subtraction or addition operation and returns the answer to us. 119 | 120 | The output of the tool is in 3 different types. 121 | 122 | * 1-The first type of output is an `integer` that is the sum or subtraction of two numbers.this output is displayed when the math sign is correctly recognized 123 | 124 | Example: 125 | 126 |
127 | captcha example 128 |
129 | 130 | 131 | * Output 132 | ```sh 133 | 99 134 | ``` 135 | 136 | * 2-The second type of output is a `list` containing two numbers. 137 | 138 | This output is displayed when the tool is able to recognize the numbers, but despite the possible filters that have been set for the sign, it is not able to recognize the sign and is forced to add or subtract the numbers together, and the list it displays is the total result.you can test both answers in the input according to the output of the tool 139 | 140 | Example: 141 | 142 |
143 | captcha example 144 |
145 | 146 | 147 | * Output 148 | ```sh 149 | [3, 19] 150 | ``` 151 | 152 | * 3-The third type of output is `None`. 153 | 154 | This output is displayed when the tool is not able to recognize one of the two numbers and as a result could not calculate the numbers together. 155 | This case happens very rarely and the tool detects the numbers correctly in most cases, but it may still show you such an output. 156 | 157 | 158 |
159 | captcha example 160 |
161 | 162 | 163 | * Output 164 | ```sh 165 | None 166 | ``` 167 | 168 | * Execution Time 169 | Captcha processing and calculation time in a system without GPU is between `16` seconds and `20` seconds from the time of execution to the end time, which is much less in systems with GPU and the processing speed is faster. 170 | 171 | ### Installation 172 | 173 | 1. Clone the repo 174 | ```sh 175 | git clone https://github.com/AmireNoori/MathCaptchaSolver/tree/main 176 | ``` 177 | 178 | You must have installed `transformers`, `opencv-python` and `numpy` libraries. For this, you can install each of them separately or install this tool using the `requirements.txt` file by entering the following command in the terminal. 179 | 180 | 2. Install packages 181 | ```sh 182 | pip install -r requirements.txt 183 | ``` 184 | 185 | ## Usage 186 | 187 | This tool is written in the form of a library that you can easily import and use in your robot or program. 188 | 189 | * An example of how to use : 190 | 191 | _Create a `main.py` file and enter the following codes in it._ 192 | 193 | * Example 194 | ```sh 195 | from MathCaptchaSolver import CaptchaSolver 196 | solver = CaptchaSolver('Your captcha Image path') # Enter captcha Image path 197 | result = solver.solve_captcha() 198 | print(result) 199 | ``` 200 | * Terminal 201 | ```sh 202 | python main.py 203 | ``` 204 | 205 | _Note: When you run the program for the first time, your system must be connected to the Internet to download the model from the Hugging Face website and it may take some time to run the program for the first time._ 206 | ## Examples 207 | 208 | This screenshot is an example of the results obtained from the 20 captcha images shared for you 209 |
210 | screenshot 211 |
212 | 213 | 214 | ## Collections 215 | If the captcha you are looking for is not among the pictures above, below are other examples of captchas that I designed as a captcha solver. To get any of the captcha solvers, you can send a message to my pv in Telegram. 216 | 217 | captcha example captcha example captcha example 218 | captcha examplecaptcha example 219 | 220 | ## License 221 | 222 | This project is licensed under the [MIT License](LICENSE). 223 | 224 | ### MIT License 225 | 226 | MIT License 227 | 228 | Copyright (c) [2024] [Amir Noori] 229 | 230 | Permission is hereby granted, free of charge, to any person obtaining a copy 231 | of this software and associated documentation files (the "Software"), to deal 232 | in the Software without restriction, including without limitation the rights 233 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 234 | copies of the Software, and to permit persons to whom the Software is 235 | furnished to do so, subject to the following conditions: 236 | 237 | The above copyright notice and this permission notice shall be included in all 238 | copies or substantial portions of the Software. 239 | 240 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 241 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 242 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 243 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 244 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 245 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 246 | SOFTWARE. 247 | 248 | © [Amir Noori](https://github.com/AmireNoori) 249 | 250 | ## Contact 251 | 252 | Amir Noori - [@AmireNoori1](https://t.me/AmireNoori1) - noorifardam@gmail.com 253 | 254 | Project Link: [https://github.com/AmireNoori/MathCapthaSolver](https://github.com/AmireNoori/MathCaptchaSolver) 255 | 256 | ## Kindness 257 | 258 | Thank you if this tool was useful for you and you used it, give it a star ⭐ and make me happy by following my profile❤️. 259 | --------------------------------------------------------------------------------