├── .circleci └── config.yml ├── .gitignore ├── 100_class_traning_note.md ├── LICENSE.md ├── README.md ├── detect.rb ├── models └── download_model.sh ├── screenshots ├── altered_2_characters.png ├── from_photo.png ├── full_image_2_characters.png ├── iqdb_from_photo.png ├── iqdb_status.png ├── saucenao_from_altered_2_characters.png ├── saucenao_from_full_image_2_characters.png └── saucenao_from_photo.png ├── setup.cfg ├── setup.py ├── src └── moeflow │ ├── __init__.py │ ├── classify.py │ ├── cmds │ ├── __init__.py │ └── main.py │ ├── face_detect.py │ ├── jinja2_env.py │ ├── static │ └── main.css │ ├── templates │ └── main.html │ └── util.py ├── test_accuracy_vs_num_of_categories.png └── tests ├── __init__.py └── test_example.py /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | jobs: 3 | build: 4 | working_directory: ~/circleci-moeflow 5 | docker: 6 | - image: circleci/python:3.5 7 | steps: 8 | - checkout 9 | - restore_cache: 10 | key: deps1-{{ .Branch }} 11 | - run: 12 | command: | 13 | python3 -m venv venv 14 | . venv/bin/activate 15 | pip install tensorflow==1.4.0 16 | pip install -e .[develop] 17 | - save_cache: 18 | key: deps1-{{ .Branch }}-{{ checksum "setup.py" }} 19 | paths: 20 | - "venv" 21 | - run: 22 | command: | 23 | . venv/bin/activate 24 | pip install -e .[tests] 25 | mkdir -p "${CIRCLE_WORKING_DIRECTORY}/results/junit/" 26 | pytest --junitxml="results/junit/results-${CIRCLE_NODE_INDEX}.xml" --verbose --cov=moeflow --cov-report term-missing tests 27 | - store_artifacts: 28 | path: "results" 29 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # MoeFlow models 2 | models/output_* 3 | 4 | # Image serving directory 5 | src/moeflow/static/images/* 6 | 7 | # Python 8 | *.pyc 9 | .cache/ 10 | /build 11 | /cover 12 | /dist 13 | /venv 14 | 15 | # Packages 16 | *.egg 17 | *.egg-info 18 | .eggs/ 19 | 20 | # Unit test / coverage reports 21 | .coverage 22 | coverage.xml 23 | .tox 24 | nosetests.xml 25 | 26 | # Others 27 | *~ 28 | *.swp 29 | *.DS_Store 30 | *.idea/ 31 | 32 | # Sphinx docs 33 | docs/_build* 34 | -------------------------------------------------------------------------------- /100_class_traning_note.md: -------------------------------------------------------------------------------- 1 | ## List of characters 2 | 3 | Currently, 100 characters are picked from 25 animation series. 4 | 5 | 1. Angel Beats!: Tachibana Kanade 6 | 2. Charlotte: Tomori Nao 7 | 3. Chuunibyou demo Koi ga Shitai!: Dekomori Sanae, Nibutani Shinka, Takanashi Rikka 8 | 4. Date A Live: Itsuka Kotori, Tobiichi Origami, Tokisaki Kurumi, Yatogami Tohka, Yoshino 9 | 5. Eromanga-sensei: Izumi Sagiri, Yamada Elf 10 | 6. Evangelion: Ayanami Rei, Souryuu Asuka Langley 11 | 7. Fate Series: Illyasviel von Einzbern, Matou Sakura, Miyu Edelfelt, Saber, Tohsaka Rin 12 | 8. Gochuumon wa Usagi desu ka?: Hoto Cocoa, Jouga Maya, Kafuu Chino, Kirima Sharo, Natsu Megumi, Tedeza Rize, Ujimatsu Chiya 13 | 9. K-On!: Akiyama Mio, Asahina Mikuru, Hirasawa Yui, Kotobuki Tsumugi, Nakano Azusa, Tainaka Ritsu 14 | 10. Kancolle: Hibiki, Kashima, Kongou, Shigure 15 | 11. Kiniro Mosaic: Alice Cartelet, Inokuma Youko, Komichi Aya, Kujou Karen, Oomiya Shinobu 16 | 12. Kono Subarashii Sekai ni Shukufuku wo!: Aqua, Dustiness Ford Lalatina, Megumin 17 | 13. Love Live!: Ayase Eli, Hoshizora Rin, Koizumi Hanayo, Kousaka Honoka, Minami Kotori, Nishikino Maki, Sonoda Umi, Toujou Nozomi, Yazawa Nico 18 | 14. Love Live! Sunshine!!: Kunikida Hanamaru, Kurosawa Dia, Kurosawa Ruby, Matsuura Kanan, Ohara Mari, Sakurauchi Riko, Takami Chika, Tsushima Yoshiko, Watanabe You 19 | 15. Madoka Magica: Akemi Homura, Kaname Madoka, Miki Sayaka, Sakura Kyouko, Tomoe Mami 20 | 16. New Game!: Sakura Nene, Suzukaze Aoba, Takimoto Hifumi, Yagami Kou 21 | 17. Nisekoi: Kirisaki Chitoge, Onodera Kosaki 22 | 18. Ore no Imouto ga Konnani Kawaii Wake ga Nai: Aragaki Ayase, Gokou Ruri, Kousaka Kirino 23 | 19. Re:Zero kara Hajimeru Isekai Seikatsu: Emilia, Ram, Rem 24 | 20. Saenai Heroine no Sodatekata: Hyoudou Michiru, Kasumigaoka Utaha, Katou Megumi, Sawamura Spencer Eriri 25 | 21. Steins;Gate: Makise Kurisu, Shiina Mayuri 26 | 22. Suzumiya Haruhi no Yuuutsu: Nagato Yuki, Suzumiya Haruhi 27 | 23. Sword Art Online: Asuna, Kirigaya Suguha, Leafa, Lisbeth, Silica, Sinon, Yui 28 | 24. Toaru Kagaku no Railgun: Misaka Mikoto, Saten Ruiko, Shirai Kuroko, Uiharu Kazari 29 | 25. Yahari Ore no Seishun Love Comedy wa Machigatteiru: Yuigahama Yui, Yukinoshita Yukino 30 | 31 | ## Test results with 100 class 32 | 33 | Each class only contains around ~~30~~* 60 images. 34 | 35 | (*) The number of dataset is increased from ~30 to ~60 and the overall accuracy is increased by 5% to 10% (from 60% - 65% to 70%). 36 | 37 | Since we are using a very small number of images (in order to be as realistic as possible), we need to modify `retrain.py`. 38 | If we don't modify it, validation list will be empty for several characters, which will cause division by zero exception. 39 | 40 | https://github.com/tensorflow/tensorflow/blob/r1.4/tensorflow/examples/image_retraining/retrain.py#L193-L198: 41 | 42 | ``` 43 | if len(validation_images) == 0: 44 | validation_images.append(base_name) 45 | elif len(testing_images) == 0: 46 | testing_images.append(base_name) 47 | elif percentage_hash < validation_percentage: 48 | validation_images.append(base_name) 49 | elif percentage_hash < (testing_percentage + validation_percentage): 50 | testing_images.append(base_name) 51 | else: 52 | training_images.append(base_name) 53 | ``` 54 | 55 | This mechanism allows validation & testing percentage somewhere between 10% - 15%, but it guarantees at least 1 image exists in each category. 56 | 57 | ## Results 58 | 59 | The experiment is done twice, with ~30 images per category and ~60 images per category later on. 60 | 61 | **Results of ~30 images per category**: 62 | 63 | |Number of Categories|Learning Rate|Training Steps|Final Train Accuracy|Final Test Accuracy| 64 | | --- | --- | --- | --- | --- | 65 | | 100 | 0.01 | 4000 | 89.0% | 53.0% (N=385) | 66 | | 100 | 0.02 | 4000 | 98.0% | 56.9% (N=385) | 67 | | 100 | 0.01 | 8000 | 99.0% | 59.2% (N=385) | 68 | | 100 | 0.02 | 8000 | 100.0% | 59.7% (N=385) | 69 | | 100 | 0.005 | 16000 | 97.0% | 59.0% (N=385) | 70 | | 100 | 0.02 | 16000 | 100.0% | **60.3% (N=385)** | 71 | 72 | The initial model accuracy is **60.3%**. 73 | 74 | Tested stuffs: 75 | - At several occasions, the number of training steps was increased to 40000 with no avail (no significant changes) 76 | - Hyperparameter tuning was experimented (random crop 5% random brightness 5%), but the result only has 1% difference and it took the entire day to run 77 | 78 | **Result of ~60 images per category**: 79 | 80 | |Number of Categories|Learning Rate|Training Steps|Final Train Accuracy|Final Test Accuracy| 81 | | --- | --- | --- | --- | --- | 82 | | 3 | 0.02 | 4000 | 100.0% | **85.0% (N=20)** | 83 | | 25 | 0.02 | 4000 | 100.0% | **80.6% (N=175)** | 84 | | 25 | 0.01 | 4000 | 100.0% | 80.6% (N=175) | 85 | | 35 | 0.02 | 4000 | 100.0% | **81.7% (N=219)** | 86 | | 35 | 0.01 | 4000 | 100.0% | 78.1% (N=219) | 87 | | 50 | 0.02 | 4000 | 99.0% | 77.4% (N=318) | 88 | | 50 | 0.02 | 8000 | 100.0% | **78.9% (N=318)** | 89 | | 65 | 0.02 | 16000 | 100.0% | **74.2% (N=438)** | 90 | | 75 | 0.02 | 4000 | 93.0% | 71.1% (N=477) | 91 | | 75 | 0.02 | 8000 | 98.0% | 74.2% (N=477) | 92 | | 75 | 0.02 | 16000 | 98.0% | **76.7% (N=477)** | 93 | | 100 | 0.02 | 4000 | 85.0% | 64.1% (N=682) | 94 | | 100 | 0.02 | 8000 | 94.0% | 67.9% (N=682) | 95 | | 100 | 0.02 | 16000 | 94.0% | **70.1% (N=682)** | 96 | 97 | ![](test_accuracy_vs_num_of_categories.png) 98 | 99 | TODO: 100 | - Investigate the correlation between number of categories vs final test accuracy (diminishing rate) 101 | - Improve character detection: rotation / axis 102 | - Improve character recognition: image noise, brightness / contrast, border, facial expression (closed eyes, etc), characters with more than 1 form (DAL, SAO, ...) 103 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Copyright (c) 2017 Iskandar Setiadi 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 4 | 5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 6 | 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MoeFlow 2 | 3 | [![CircleCI](https://circleci.com/gh/freedomofkeima/MoeFlow/tree/master.svg?style=shield)](https://circleci.com/gh/freedomofkeima/MoeFlow/tree/master) 4 | 5 | Repository for anime characters recognition website, powered by TensorFlow. 6 | 7 | Demonstration page (Alpha version): [MoeFlow Website](https://freedomofkeima.com/moeflow/). 8 | 9 | MoeFlow is featured in: 10 | - PyCon ID 2017 | [Presentation](https://freedomofkeima.com/pyconid2017.pdf) 11 | - [PyCon JP 2018](https://pycon.jp/2018/en/event/sessions) | [Presentation](https://freedomofkeima.com/pyconjp2018.pdf) | [Video](https://www.youtube.com/watch?v=Oh-raRnQoUA) 12 | 13 | ## Project Introduction 14 | 15 | This project is related to [freedomofkeima/transfer-learning-anime](https://github.com/freedomofkeima/transfer-learning-anime). 16 | 17 | ### Background 18 | 19 | This project is heavily inspired from characters indexing website such as [saucenao](https://saucenao.com/) and [iqdb](https://www.iqdb.org/). In general, character indexing websites work well since character arts are generally limited in terms of number compared to real-life photos. 20 | 21 | iqdb_status 22 | 23 | However, there are cases where character indexing websites will not work well, e.g.: the image is cropped or altered. 24 | 25 | **Full Image** (Top: Saucenao, Bottom: MoeFlow) 26 | 27 | saucenao_full_image 28 | 29 | moeflow_full_image 30 | 31 | **Altered Image** 32 | 33 | saucenao_full_image 34 | 35 | moeflow_full_image 36 | 37 | Or, there are cases where you want to recognize a character from a photo. 38 | 39 | (Top: Saucenao, Middle: iqdb, Bottom: MoeFlow) 40 | 41 | saucenao_photo 42 | 43 | iqdb_photo 44 | 45 | moeflow_photo 46 | 47 | ### Transfer Learning 48 | 49 | This project only uses [~~30~~* 60 images per character](100_class_traning_note.md) for learning, which are very low in number for image recognition learning. However, this number is chosen since the majority of characters has a limited number of arts. 50 | 51 | In [yande.re](https://yande.re/tag?name=&order=count&type=4), there are around 35000 registered character tags. However, top 1000 characters only have 70+ images while top 2000 characters only have 40+ images. 52 | 53 | (*) The number of dataset is increased from ~30 to ~60 and the overall accuracy is increased by 5% to 10% (from 60% - 65% to 70%). 54 | 55 | ## Requirements 56 | 57 | - TensorFlow 1.4.0 (`pip install tensorflow==1.4.0` first) 58 | - [nagadomi/animeface-2009](https://github.com/nagadomi/animeface-2009) 59 | 60 | ## How to create initial environment 61 | 62 | Python Environment: 63 | 64 | ``` 65 | $ virtualenv -p python3 venv # Ensure python3 version is 3.5, otherwise TensorFlow might not work 66 | $ . venv/bin/activate 67 | $ pip install tensorflow==1.4.0 68 | ``` 69 | 70 | Since `nagadomi/animeface-2009` is an independent project, you need to clone it somewhere in your local directory. Note that the project requires Ruby, ImageMagick, and gcc to run. 71 | 72 | After you finish installing it, go to `detect.rb` and update the `require` part (line 4) accordingly. 73 | 74 | After that, you need to download MoeFlow model via `models/download_model.sh` (~ 100 MB). 75 | 76 | ## How to run 77 | 78 | After running steps above, you can simply run it by: 79 | 80 | ``` 81 | $ export MOEFLOW_MODEL_PATH='/path/to/MoeFlow/models' 82 | $ pip install -e . 83 | $ app 84 | ``` 85 | 86 | If your application is configured to run in a relative path, e.g.: [https://freedomofkeima.com/moeflow/](https://freedomofkeima.com/moeflow/), then you can set static URL path via `export MOEFLOW_RELATIVE_URL_PATH='/moeflow/'`. 87 | 88 | ## License 89 | 90 | This project itself is licensed under MIT License. 91 | 92 | Face recognition feature is developed by [nagadomi](https://github.com/nagadomi). 93 | 94 | All images are owned by their respective creators. 95 | -------------------------------------------------------------------------------- /detect.rb: -------------------------------------------------------------------------------- 1 | require "pp" 2 | require "rmagick" 3 | # Put nagadomi/animeface-2009 path here 4 | require "/home/ec2-user/animeface-2009/animeface-ruby/AnimeFace" 5 | 6 | if ARGV.size == 0 7 | warn "Usage: #{$0} " 8 | exit(-1) 9 | end 10 | 11 | image = Magick::ImageList.new(ARGV[0]) 12 | faces = AnimeFace::detect(image) 13 | counter = 0 14 | pp faces 15 | faces.each do |ctx| 16 | next if ctx["likelihood"] < 0.8 17 | counter = counter + 1 18 | filename = File.basename(ARGV[0]).split(".").first + "_out_" + counter.to_s + ".jpg" 19 | output = File.join(ARGV[1], filename) 20 | face = ctx["face"] 21 | if face["width"] < 125 or face["height"] < 125 22 | margin = ([face["width"], face["height"]].min / 5).ceil 23 | else 24 | margin = 25 25 | end 26 | x = [face["x"] - margin, 0].max 27 | y = [face["y"] - margin, 0].max 28 | width = [face["width"] + 2 * margin, image.columns - x].min 29 | height = [face["height"] + 2 * margin, image.rows - y].min 30 | gc = image.crop(x, y, width, height) 31 | gc.write(output) 32 | end 33 | 34 | -------------------------------------------------------------------------------- /models/download_model.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Download label 4 | wget https://freedomofkeima.com/transfer-learning-model/output_labels_2.txt 5 | 6 | # Download model 7 | wget https://freedomofkeima.com/transfer-learning-model/output_graph_2.pb 8 | 9 | -------------------------------------------------------------------------------- /screenshots/altered_2_characters.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/freedomofkeima/MoeFlow/5c76fd260102bf06db39b2cc36ef501960d60d3c/screenshots/altered_2_characters.png -------------------------------------------------------------------------------- /screenshots/from_photo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/freedomofkeima/MoeFlow/5c76fd260102bf06db39b2cc36ef501960d60d3c/screenshots/from_photo.png -------------------------------------------------------------------------------- /screenshots/full_image_2_characters.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/freedomofkeima/MoeFlow/5c76fd260102bf06db39b2cc36ef501960d60d3c/screenshots/full_image_2_characters.png -------------------------------------------------------------------------------- /screenshots/iqdb_from_photo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/freedomofkeima/MoeFlow/5c76fd260102bf06db39b2cc36ef501960d60d3c/screenshots/iqdb_from_photo.png -------------------------------------------------------------------------------- /screenshots/iqdb_status.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/freedomofkeima/MoeFlow/5c76fd260102bf06db39b2cc36ef501960d60d3c/screenshots/iqdb_status.png -------------------------------------------------------------------------------- /screenshots/saucenao_from_altered_2_characters.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/freedomofkeima/MoeFlow/5c76fd260102bf06db39b2cc36ef501960d60d3c/screenshots/saucenao_from_altered_2_characters.png -------------------------------------------------------------------------------- /screenshots/saucenao_from_full_image_2_characters.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/freedomofkeima/MoeFlow/5c76fd260102bf06db39b2cc36ef501960d60d3c/screenshots/saucenao_from_full_image_2_characters.png -------------------------------------------------------------------------------- /screenshots/saucenao_from_photo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/freedomofkeima/MoeFlow/5c76fd260102bf06db39b2cc36ef501960d60d3c/screenshots/saucenao_from_photo.png -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | description-file = README.md 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from setuptools import find_packages, setup 3 | 4 | requires = [ 5 | 'aiofiles==0.3.2', 6 | 'attrs==17.4.0', 7 | 'httptools==0.0.9', 8 | 'jinja2==2.10', 9 | 'MarkupSafe==1.0', 10 | 'numpy==1.13.3', 11 | 'opencv-python==3.3.0.10', 12 | 'python-magic==0.4.13', 13 | 'sanic==0.6.0', 14 | 'ujson==1.35', 15 | 'uvloop==0.8.1', 16 | 'websockets==4.0.1', 17 | ] 18 | 19 | console_scripts = [ 20 | 'app = moeflow.cmds.main:main', 21 | ] 22 | 23 | setup( 24 | name='MoeFlow', 25 | version='0.0.1', 26 | author='Iskandar Setiadi', 27 | author_email='iskandarsetiadi@gmail.com', 28 | url='https://github.com/freedomofkeima/MoeFlow', 29 | description='Anime characters recognition website, powered by TensorFlow', 30 | license='MIT', 31 | packages=find_packages(where='src'), 32 | package_dir={ 33 | '': 'src' 34 | }, 35 | install_requires=requires, 36 | include_package_data=True, 37 | classifiers=[ 38 | "Programming Language :: Python :: 3.5", 39 | "Environment :: Web Environment" 40 | ], 41 | entry_points={'console_scripts': console_scripts}, 42 | extras_require={ 43 | 'tests': ['pytest', 'pytest-cov', 'pytest-sugar'] 44 | }, 45 | zip_safe=False 46 | ) 47 | -------------------------------------------------------------------------------- /src/moeflow/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/freedomofkeima/MoeFlow/5c76fd260102bf06db39b2cc36ef501960d60d3c/src/moeflow/__init__.py -------------------------------------------------------------------------------- /src/moeflow/classify.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import logging 3 | import tensorflow as tf 4 | 5 | 6 | def read_tensor_from_image_file(file_name, input_height=299, input_width=299, 7 | input_mean=0, input_std=255): 8 | input_name = "file_reader" 9 | output_name = "normalized" 10 | file_reader = tf.read_file(file_name, input_name) 11 | image_reader = tf.image.decode_jpeg( 12 | file_reader, 13 | channels=3, 14 | name='jpeg_reader' 15 | ) 16 | float_caster = tf.cast(image_reader, tf.float32) 17 | dims_expander = tf.expand_dims(float_caster, 0) 18 | resized = tf.image.resize_bilinear( 19 | dims_expander, 20 | [input_height, input_width] 21 | ) 22 | normalized = tf.divide(tf.subtract(resized, [input_mean]), [input_std]) 23 | sess = tf.Session() 24 | result = sess.run(normalized) 25 | 26 | return result 27 | 28 | 29 | def classify_resized_face(file_name, label_lines, graph): 30 | results = [] 31 | logging.info('Processing classification') 32 | with tf.Session(graph=graph) as sess: 33 | # Feed the image data as input to the graph and get first prediction 34 | softmax_tensor = sess.graph.get_tensor_by_name('final_result:0') 35 | input_operation = sess.graph.get_operation_by_name("Mul") 36 | t = read_tensor_from_image_file(file_name) 37 | predictions = sess.run( 38 | softmax_tensor, 39 | {input_operation.outputs[0]: t} 40 | ) 41 | # Sort to show labels of first prediction in order of confidence 42 | top_k = predictions[0].argsort()[-3:][::-1] 43 | 44 | for node_id in top_k: 45 | human_string = label_lines[node_id] 46 | score = predictions[0][node_id] 47 | results.append((human_string, score)) 48 | return results 49 | 50 | -------------------------------------------------------------------------------- /src/moeflow/cmds/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/freedomofkeima/MoeFlow/5c76fd260102bf06db39b2cc36ef501960d60d3c/src/moeflow/cmds/__init__.py -------------------------------------------------------------------------------- /src/moeflow/cmds/main.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import cv2 3 | import logging 4 | import magic 5 | import os 6 | import shutil 7 | import tempfile 8 | import tensorflow as tf 9 | import uuid 10 | from sanic import Sanic, response 11 | from moeflow.classify import classify_resized_face 12 | from moeflow.face_detect import run_face_detection 13 | from moeflow.jinja2_env import render 14 | from moeflow.util import resize_large_image, resize_faces, cleanup_image_cache 15 | 16 | app = Sanic(__name__) 17 | dir_path = os.path.dirname(os.path.realpath(__file__)) 18 | static_path = os.path.join(dir_path, '..', 'static') 19 | app.static('/static', static_path) 20 | 21 | ALLOWED_MIMETYPE = ['image/jpeg', 'image/png'] 22 | 23 | 24 | @app.route("/", methods=['GET', 'POST']) 25 | async def main_app(request): 26 | if request.method == "POST": 27 | uploaded_image = request.files.get('uploaded_image') 28 | mime_type = magic.from_buffer(uploaded_image.body, mime=True) 29 | if mime_type not in ALLOWED_MIMETYPE: 30 | return response.html(render("main.html")) 31 | # Scale down input image to ~800 px 32 | image = resize_large_image(uploaded_image.body) 33 | with tempfile.NamedTemporaryFile(mode="wb", suffix='.jpg') as input_jpg: 34 | filename = input_jpg.name 35 | logging.info("Input file is created at {}".format(filename)) 36 | cv2.imwrite(filename, image) 37 | # Copy to image directory 38 | ori_name = uuid.uuid4().hex + ".jpg" 39 | shutil.copyfile(filename, os.path.join( 40 | static_path, 'images', ori_name 41 | )) 42 | results = [] 43 | # Run face detection with animeface-2009 44 | detected_faces = run_face_detection(filename) 45 | # This operation will rewrite detected faces to 96 x 96 px 46 | resize_faces(detected_faces) 47 | # Classify with TensorFlow 48 | if not detected_faces: # Use overall image as default 49 | detected_faces = [filename] 50 | for face in detected_faces: 51 | predictions = classify_resized_face( 52 | face, 53 | app.label_lines, 54 | app.graph 55 | ) 56 | face_name = uuid.uuid4().hex + ".jpg" 57 | shutil.copyfile(face, os.path.join( 58 | static_path, 'images', face_name 59 | )) 60 | results.append({ 61 | "image_name": face_name, 62 | "prediction": predictions 63 | }) 64 | logging.info(predictions) 65 | # Cleanup 66 | cleanup_image_cache(os.path.join(static_path, 'images')) 67 | for faces in detected_faces: 68 | if faces != filename: 69 | os.remove(faces) 70 | return response.html( 71 | render( 72 | "main.html", 73 | ori_name=ori_name, 74 | results=results 75 | ) 76 | ) 77 | return response.html(render("main.html")) 78 | 79 | 80 | @app.route("/hello_world") 81 | async def hello_world(request): 82 | return response.text("Hello world!") 83 | 84 | 85 | @app.listener('before_server_start') 86 | async def initialize(app, loop): 87 | moeflow_path = os.environ.get('MOEFLOW_MODEL_PATH') 88 | label_path = os.path.join(os.sep, moeflow_path, "output_labels_2.txt") 89 | model_path = os.path.join(os.sep, moeflow_path, "output_graph_2.pb") 90 | app.label_lines = [ 91 | line.strip() for line in tf.gfile.GFile(label_path) 92 | ] 93 | graph = tf.Graph() 94 | graph_def = tf.GraphDef() 95 | with tf.gfile.FastGFile(model_path, 'rb') as f: 96 | graph_def.ParseFromString(f.read()) 97 | with graph.as_default(): 98 | tf.import_graph_def(graph_def, name='') 99 | app.graph = graph 100 | logging.info("MoeFlow model is now initialized!") 101 | 102 | 103 | def main(): 104 | # Set logger 105 | logging.basicConfig(level=logging.INFO) 106 | app.run(host="0.0.0.0", port=8888) 107 | 108 | if __name__ == '__main__': 109 | main_app() 110 | 111 | -------------------------------------------------------------------------------- /src/moeflow/face_detect.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import logging 3 | import os 4 | import subprocess 5 | 6 | 7 | def run_face_detection(input_image_path): 8 | """ 9 | Receives input image path 10 | Return list of path (detected faces in /tmp directory) 11 | """ 12 | args = [ 13 | 'ruby', 14 | 'detect.rb', 15 | input_image_path, 16 | '/tmp' 17 | ] 18 | results = [] 19 | # Execute 20 | try: 21 | output = subprocess.check_output( 22 | args, 23 | shell=False, 24 | timeout=30 25 | ) 26 | except Exception: 27 | logging.exception("Face detection failed!") 28 | return [] 29 | input_name_base = os.path.basename(input_image_path) 30 | input_name_without_ext = os.path.splitext(input_name_base)[0] 31 | for filename in os.listdir('/tmp'): 32 | if filename.startswith(input_name_without_ext + '_out'): 33 | results.append("/tmp/{}".format(filename)) 34 | return results 35 | 36 | -------------------------------------------------------------------------------- /src/moeflow/jinja2_env.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import os 3 | 4 | import jinja2 5 | 6 | dir_path = os.path.dirname(os.path.realpath(__file__)) 7 | 8 | 9 | def render(tpl_path, **context): 10 | relative_url_path = os.environ.get("MOEFLOW_RELATIVE_URL_PATH") 11 | if not relative_url_path: 12 | relative_url_path = "" 13 | return jinja2.Environment( 14 | loader=jinja2.FileSystemLoader( 15 | os.path.join(dir_path, 'templates') 16 | ) 17 | ).get_template(tpl_path).render(url_path=relative_url_path, **context) 18 | -------------------------------------------------------------------------------- /src/moeflow/static/main.css: -------------------------------------------------------------------------------- 1 | .container { 2 | max-width: 960px; 3 | } 4 | 5 | -------------------------------------------------------------------------------- /src/moeflow/templates/main.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | MoeFlow: Anime Characters Recognition 13 | 14 | 15 |
16 |
17 |

MoeFlow: Anime Characters Recognition (Alpha Ver.)

18 |

For more information, see freedomofkeima/MoeFlow and freedomofkeima/transfer-learning-anime at Github.

19 |

List of 100 supported characters can be accessed here.

20 |

Note: This operation will be very slow (around 15 seconds) if there are a lot of characters in a single image!

21 |
22 | 23 |
24 |

Upload Image

25 |
26 |
27 | 28 | Accepted mime-type: image/jpeg, image/png 29 |
30 |
31 | 32 |
33 |
34 | 35 | {% if ori_name %} 36 |
37 |
38 |
39 |

Input:

40 | 41 |
42 | 43 |
44 |

Output:

45 | 46 | 47 | 48 | 49 | 50 | {% for result in results %} 51 | 52 | 53 | 60 | 61 | {% endfor %} 62 |
CharacterPrediction (Top-3)
54 |
    55 | {% for prediction in result['prediction'] %} 56 |
  • {{ prediction[0] }}
    • 57 | {% endfor %} 58 |
59 |
63 |
64 |
65 | {% endif %} 66 |
67 |

Freedomofkeima Zone, 2017 - 2018.

68 |
69 |
70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | -------------------------------------------------------------------------------- /src/moeflow/util.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import cv2 3 | import logging 4 | import math 5 | import os 6 | import time 7 | 8 | import numpy as np 9 | 10 | WIDTH_HEIGHT_LIMIT = 1600 # in pixel 11 | 12 | 13 | def resize_large_image(image_data): 14 | img_array = np.fromstring(image_data, dtype=np.uint8) 15 | image = cv2.imdecode(img_array, 1) 16 | height, width = image.shape[:2] 17 | logging.info("Height: {}, Width: {}".format(height, width)) 18 | if height > width and height > WIDTH_HEIGHT_LIMIT: 19 | ratio = float(WIDTH_HEIGHT_LIMIT) / float(height) 20 | new_width = int((width * ratio) + 0.5) 21 | return cv2.resize( 22 | image, 23 | (new_width, WIDTH_HEIGHT_LIMIT), 24 | interpolation=cv2.INTER_AREA 25 | ) 26 | elif width > WIDTH_HEIGHT_LIMIT: 27 | ratio = float(WIDTH_HEIGHT_LIMIT) / float(width) 28 | new_height = int((height * ratio) + 0.5) 29 | return cv2.resize( 30 | image, 31 | (WIDTH_HEIGHT_LIMIT, new_height), 32 | interpolation=cv2.INTER_AREA 33 | ) 34 | else: 35 | return image 36 | 37 | 38 | def resize_faces(image_files, width=96, height=96): 39 | for image_file in image_files: 40 | image = cv2.imread(image_file) 41 | resized_image = cv2.resize( 42 | image, 43 | (width, height), 44 | interpolation=cv2.INTER_AREA 45 | ) 46 | cv2.imwrite(image_file, resized_image) 47 | 48 | 49 | def cleanup_image_cache(image_dir, expire=3600): # Expire in 1 hour 50 | now = time.time() 51 | for f in os.listdir(image_dir): 52 | f = os.path.join(image_dir, f) 53 | if os.stat(f).st_mtime < now - expire: 54 | if os.path.isfile(f): 55 | os.remove(f) 56 | 57 | -------------------------------------------------------------------------------- /test_accuracy_vs_num_of_categories.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/freedomofkeima/MoeFlow/5c76fd260102bf06db39b2cc36ef501960d60d3c/test_accuracy_vs_num_of_categories.png -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/freedomofkeima/MoeFlow/5c76fd260102bf06db39b2cc36ef501960d60d3c/tests/__init__.py -------------------------------------------------------------------------------- /tests/test_example.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from moeflow.cmds.main import hello_world 3 | 4 | 5 | def test_hello_world(): 6 | hello_world(object()) 7 | assert True 8 | --------------------------------------------------------------------------------