├── .gitignore ├── LICENSE ├── readme.md ├── readme_images ├── dataset_example.png ├── overlay_lines.png ├── prediction.png ├── random_fen.png └── weight_KQR.png ├── tensorflow_compvision.ipynb ├── tensorflow_generate_training_data.ipynb ├── tensorflow_learn.ipynb └── tensorflow_learn_cnn.ipynb /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints 2 | *.pyc 3 | *.png 4 | *.jpg 5 | *.gif 6 | 7 | # Ignore saved models 8 | saved_models/ 9 | 10 | # Ignore chessboard input images and tile outputs 11 | chessboards/ 12 | tiles/ 13 | 14 | # Ignore reddit username/password config file 15 | auth_config.py 16 | praw.ini 17 | 18 | # Ignore tracking files 19 | *.txt 20 | 21 | !readme_images/* 22 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Sameer Ansari 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | TensorFlow Chessbot - [/u/ChessFenBot](https://www.reddit.com/user/ChessFenBot) [◕ _ ◕]\* *I make FENs* 2 | --- 3 | 4 | *CLI and Bot code lives on the [chessfenbot branch](https://github.com/Elucidation/tensorflow_chessbot/tree/chessfenbot)* 5 | 6 | [Live demo here](http://elucidation.github.io/ChessboardFenTensorflowJs/) using [TensorflowJs](https://js.tensorflow.org/) 7 | 8 | **TL;DR**: 9 | > Turn http://i.imgur.com/HnWYt8A.png → [1nkr4/1p3q1p/pP4pn/P1r5/3N1p2/2b2B1P/5PPB/2RQ1RK1](https://lichess.org/analysis/1nkr4/1p3q1p/pP4pn/P1r5/3N1p2/2b2B1P/5PPB/2RQ1RK1_w) 10 | 11 | ![Prediction](readme_images/prediction.png) 12 | 13 | A [TensorFlow](www.tensorflow.org) [Convolutional Neural Network](https://en.wikipedia.org/wiki/Convolutional_neural_network) algorithm trained on 32x32 grayscale chess tiles predicts chessboards layouts from online chessboard screenshots. 14 | 15 | * 5x5x32 Input Convolution layer 16 | * 5x5x64 Convolution layer 17 | * 8x8x1024 Dense Fully Connected layer 18 | * 1024x13 Dropout + Softmax Readout layer 19 | 20 | Yes, using a CNN is kinda overkill, but it is *exciting*. 21 | 22 | ### Running Tensorflow_Chessbot on images locally and via URLs 23 | 24 | On a linux machine which has Tensorflow and SciPy installed 25 | 26 | * [Tensorflow Install instructions](https://www.tensorflow.org/versions/r0.9/get_started/os_setup.html) 27 | * [SciPy Install Instructions](https://www.scipy.org/install.html), possibly as easy as this on Ubuntu 28 | 29 | sudo apt-get install python-numpy python-scipy python-matplotlib ipython ipython-notebook python-pandas python-sympy python-nose 30 | 31 | Download the chessfenbot branch somewhere : [tensorflow_chessbot/chessfenbot](https://github.com/Elucidation/tensorflow_chessbot/archive/chessfenbot.zip) 32 | 33 | *Alternatively clone the [chessfenbot branch](https://github.com/Elucidation/tensorflow_chessbot/tree/chessfenbot)* 34 | 35 | Now, to run pass the following arguments to `tensorflow_chessbot.py` 36 | 37 | $ ./tensorflow_chessbot.py -h 38 | usage: tensorflow_chessbot.py [-h] [--url URL] [--filepath FILEPATH] 39 | 40 | Predict a chessboard FEN from supplied local image link or URL 41 | 42 | optional arguments: 43 | -h, --help show this help message and exit 44 | --url URL URL of image (ex. http://imgur.com/u4zF5Hj.png) 45 | --filepath FILEPATH filepath to image (ex. u4zF5Hj.png) 46 | 47 | By default, it will try and load the URL `http://imgur.com/u4zF5Hj.png` and make a prediction in it, otherwise, you could pass a local file like so (for example with an image file `u4zF5Hj.png` located in the same directory): 48 | 49 | ./tensorflow_chessbot.py --filepath u4zF5Hj.png 50 | 51 | Which should output something like: 52 | 53 | ./tensorflow_chessbot.py --filepath ./u4zF5Hj.png 54 | Setting up CNN TensorFlow graph... 55 | I tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 2 56 | I tensorflow/core/common_runtime/direct_session.cc:58] Direct session inter op parallelism threads: 2 57 | Loading model 'saved_models/model_10000.ckpt' 58 | Model restored. 59 | Certainty range [0.999545 - 1], Avg: 0.999977, Overall: 0.998546 60 | Predicted FEN: 11111111/11111p11/11111k1P/11p1111P/1p1p1111/11111111/111K1111/11111111 61 | Certainty: 99.9% 62 | Done 63 | 64 | 65 | Similarly, a URL can be tested by calling with a URL: 66 | 67 | $ ./tensorflow_chessbot.py --url http://imgur.com/u4zF5Hj.png 68 | 69 | ### ~~Reddit Bot~~ *Deprecated* 70 | 71 | *Code lives on the [chessfenbot branch](https://github.com/Elucidation/tensorflow_chessbot/tree/chessfenbot)* 72 | 73 | [/u/ChessFenBot](https://www.reddit.com/user/ChessFenBot) used to (~2015-2016) automatically reply to [reddit /r/chess](https://www.reddit.com/r/) new topic image posts that contain detectable online chessboard screenshots. A screenshot either ends in `.png`, `.jpg`, `.gif`, or is an `imgur` link. 74 | 75 | It replies with a [lichess](https://lichess.org) analysis link for that layout and a predicted [FEN](https://en.wikipedia.org/wiki/Forsyth%E2%80%93Edwards_Notation). 76 | 77 | ```py 78 | predictor = ChessboardPredictor() 79 | fen, certainty = predictor.makePrediction('http://imgur.com/u4zF5Hj.png') 80 | print "Predicted FEN: %s" % fen 81 | print "Certainty: %.1f%%" % (certainty*100) 82 | ``` 83 | 84 | ``` 85 | Certainty range [0.999545 - 1], Avg: 0.999977, Overall: 0.998546 86 | Predicted FEN: 8/5p2/5k1P/2p4P/1p1p4/8/3K4/8 87 | Certainty: 99.9% 88 | Done 89 | [Finished in 1.8s] 90 | ``` 91 | 92 | ChessFenBot automatically replied to [this reddit post](https://www.reddit.com/r/chess/comments/45osos/very_difficult_find_the_best_move_for_white/d004cg6?context=3), it processed the [screenshot link url](http://i.imgur.com/HnWYt8A.png) and responded with: 93 | 94 | > ChessFenBot [◕ _ ◕]\* *I make FENs* 95 | > 96 | > --- 97 | > 98 | > I attempted to generate a chessboard layout from the posted image, with an overall certainty of **99.9916%**. 99 | > 100 | > FEN: [1nkr4/1p3q1p/pP4pn/P1r5/3N1p2/2b2B1P/5PPB/2RQ1RK1](http://www.fen-to-image.com/image/30/1nkr1111/1p111q1p/pP1111pn/P1r11111/111N1p11/11b11B1P/11111PPB/11RQ1RK1.png) 101 | > 102 | > Here is a link to a [Lichess Analysis](https://lichess.org/analysis/1nkr4/1p3q1p/pP4pn/P1r5/3N1p2/2b2B1P/5PPB/2RQ1RK1_w) - White to play 103 | > 104 | > --- 105 | > 106 | > Yes I am a machine learning bot | [`How I work`](https://github.com/Elucidation/tensorflow_chessbot 'Must go deeper') | Reply with a corrected FEN or [Editor link)](https://lichess.org/editor/r1b1r1k1/5pp1/p1pR1nNp/8/2B5/2q5/P1P1Q1PP/5R1K) to add to my next training dataset 107 | 108 | ## Workflow 109 | 110 | There are three ipython notebooks which show the workflow from turning a screenshot of a chessboard into a set of 32x32 grayscale tiles, to generating those tiles for training and testing, and then the actual training and learning of the neural network from those trials using [TensorFlow](http://www.tensorflow.org). 111 | 112 | 1. [tensorflow_compvision.ipynb](tensorflow_compvision.ipynb) - Computer Vision 113 | 1. [tensorflow_generate_training_data.ipynb](tensorflow_generate_training_data.ipynb) - Generating a dataset from set of screenshots of chessboards in known configurations 114 | 1. [tensorflow_learn.ipynb](tensorflow_learn.ipynb) - **TensorFlow Neural Network Training & Prediction** Basic Regression classifier, works for more common lichess.org and chess.com screenshots 115 | 1. [tensorflow_learn_cnn.ipynb](tensorflow_learn_cnn.ipynb) - **TensorFlow Convolutional Neural Network Training & Prediction** tested with ~73% success rate on 71 chess subreddit posts 116 | 117 | --- 118 | 119 | #### [tensorflow_compvision.ipynb](tensorflow_compvision.ipynb) - 1. Computer Vision 120 | 121 | Here is a screenshot with the detected lines of the chessboard overlaid, showing where we'll cut the image into tiles. 122 | 123 | ![overlay lines](readme_images/overlay_lines.png) 124 | 125 | --- 126 | 127 | #### [tensorflow_generate_training_data.ipynb](tensorflow_generate_training_data.ipynb) - 2. Generating a dataset 128 | 129 | [Lichess.org](lichess.org) provides a URL interface with a FEN string that loads a webpage with that board arrayed. A nice repo called [pythonwebkit2png](https://github.com/adamn/python-webkit2png) provides a way to render webpages programmatically, allowing us to generate several (80 in thise) random FEN strings, load the URL and take a screenshot all automatically. 130 | 131 | ![random fen](readme_images/random_fen.png) 132 | 133 | Here is 5 example tiles and their associated label, a 13 length one-hot vector corresponding to 6 white pieces, 6 black pieces, and 1 empty space. 134 | 135 | ![dataset example](readme_images/dataset_example.png) 136 | 137 | --- 138 | 139 | #### [tensorflow_learn.ipynb](tensorflow_learn.ipynb) - 3. TensorFlow Neural Network Training & Prediction 140 | 141 | We train the neural network on generated data from 80 lichess.org screenshots, which is 5120 tiles. We test it with 5 screenshots (320 tiles) as a quick sanity check. Here is a visualization of the weights for the white King, Queen and Rook. 142 | 143 | ![Some weights](readme_images/weight_KQR.png) 144 | 145 | Finally we can make predictions on images passed by URL, the ones from lichess and visually similar boards work well, the ones that are too different from what we trained for don't work, suggesting that getting more data is in order. Here is a prediction on the image for [this reddit post](https://www.reddit.com/r/chess/comments/45inab/moderate_black_to_play_and_win/) 146 | 147 | ![Prediction](readme_images/prediction.png) 148 | 149 | --- 150 | 151 | #### [tensorflow_learn_cnn.ipynb](tensorflow_learn_cnn.ipynb) - TensorFlow Convolutional Neural Network Training & Prediction 152 | 153 | Built a slightly larger dataset of ~150 screenshots which is around 9600 tiles which includes randomized FEN diagrams from lichess.org, chess.com, and 2 FEN generated diagram sites. 154 | 155 | Tested with ~73% success rate on 71 chess subreddit posts, good enough to make a first draft Reddit bot. 156 | 157 | --- 158 | 159 | ### Ideation 160 | Reddit post has an image link (perhaps as well as a statement "white/black to play"). 161 | 162 | Bot takes the image, uses some CV to find a chessboard on it, splits up into 163 | a set of images of squares. These are the inputs to the tensorflow CNN 164 | which will return probability of which piece is on it (or empty) 165 | 166 | Dataset will include chessboard squares from chess.com, lichess 167 | Different styles of each, all the pieces 168 | 169 | Generate synthetic data via added noise: 170 | * change in coloration 171 | * highlighting 172 | * occlusion from lines etc. 173 | 174 | Take most probable set from TF response, use that to generate a FEN of the 175 | board, and bot comments on thread with FEN and link to lichess analysis 176 | -------------------------------------------------------------------------------- /readme_images/dataset_example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Elucidation/tensorflow_chessbot/db893cf50c68307b2494cc0e92c26f70aca3ab5a/readme_images/dataset_example.png -------------------------------------------------------------------------------- /readme_images/overlay_lines.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Elucidation/tensorflow_chessbot/db893cf50c68307b2494cc0e92c26f70aca3ab5a/readme_images/overlay_lines.png -------------------------------------------------------------------------------- /readme_images/prediction.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Elucidation/tensorflow_chessbot/db893cf50c68307b2494cc0e92c26f70aca3ab5a/readme_images/prediction.png -------------------------------------------------------------------------------- /readme_images/random_fen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Elucidation/tensorflow_chessbot/db893cf50c68307b2494cc0e92c26f70aca3ab5a/readme_images/random_fen.png -------------------------------------------------------------------------------- /readme_images/weight_KQR.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Elucidation/tensorflow_chessbot/db893cf50c68307b2494cc0e92c26f70aca3ab5a/readme_images/weight_KQR.png --------------------------------------------------------------------------------