├── .gitignore ├── README.md ├── behavior └── behavior.xar ├── deep_nao.pml ├── dialog └── dialog-en.top ├── doc ├── application_manager_add_current.png ├── disabled.png ├── nao.jpg ├── shell.png ├── upload_model.png └── use_custom_model.png ├── html ├── img │ ├── .empty │ └── deep.jpg ├── index.html ├── index.js ├── robotutils.js ├── style.css ├── upload.html └── upload.js ├── install.sh ├── lib └── python2.7 │ └── site-packages │ └── cv2.so ├── manifest.xml ├── models ├── ssd_mobilenet_v2_oid_v4 │ ├── frozen_inference_graph.pb │ ├── graph.pbtxt │ ├── objects.names.en │ └── objects.names.fr └── yolo_v4_tiny_custom │ └── .empty ├── notebook └── NAO_Deep_Learning_Custom_Model_Training.ipynb └── scripts ├── deep_nao.py ├── explorer.py ├── models.py ├── stk ├── __init__.py ├── events.py ├── logging.py ├── runner.py ├── services.py └── worker.py ├── stop_behavior.py └── translations.py /.gitignore: -------------------------------------------------------------------------------- 1 | *pyc 2 | *pkg 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Object recognition on NAO with Deep Learning 2 | 3 | This application demonstrate how the NAO robot can recognize objects using deep learning algorithms. 4 | It comes with a pre-trained deep learning model that allows NAO to recognize up to 600 different objects with its camera. 5 | 6 | You can also train your own deep learning model on your own object image dataset and upload the model to be used in this application. 7 | 8 | * [1. Hardware & software compatibility](#1-hardware-&-software-compatibility) 9 | * [2. Supported language](#2-supported-language) 10 | * [3. Install the app](#3-install-the-app) 11 | * [3.1. Dependancies](#31-dependancies) 12 | * [3.1.1. Language french](#311-language-french) 13 | * [3.2. Install the app using Choregraphe](#32-install-the-app-using-choregraphe) 14 | * [4. Use the app](#4-use-the-app) 15 | * [4.1. Start the app](#41-start-the-app) 16 | * [4.2. Recognized objects](#42-recognized-objects) 17 | * [4.3. Remote control the app via its web page](#43-remote-control-the-app-via-its-web-page) 18 | * [4.3.1. Open the page in your internet browser](#431-open-the-page-in-your-internet-browser) 19 | * [4.3.2. Application disabled](#432-application-disabled) 20 | * [4.4. Interact with the app via the web page](#44-interact-with-the-app-via-the-web-page) 21 | * [5. Training your custom model](#5-training-your-custom-model) 22 | * [6. Uploading your custom model to the app](#6-uploading-your-custom-model-to-the-app) 23 | 24 | # Video 25 | 26 | Please see the following video on YouTube for some more information about how this project works in practice: https://youtu.be/NwHXtjw7TG0 27 | 28 | ## 1. Hardware & software compatibility 29 | 30 | This app is compatible with NAO V6, and [Naoqi 2.8.6](https://developer.softbankrobotics.com/blog/nao6/naoqi-developer-guide/nao6-naoqi-28-286-release-note) 31 | 32 | ## 2. Supported language 33 | 34 | This app is available in: 35 | 36 | * French 37 | * English 38 | 39 | The application will automatically switch to English or French language when you [change the language on NAO](https://developer.softbankrobotics.com/nao6/nao-documentation/nao-user-guide/configuring-and-settings/setting-nao-s-preferred-language#-modifying-the-preferred-language-) 40 | 41 | ## 3. Install the app 42 | 43 | ### 3.1. Dependancies 44 | 45 | #### 3.1.1. Language french 46 | 47 | If you want to use the Deep NAO app in French, you need to have the French language installed on your robot. If it is not installed by default, [you can order the French language by contacting customer support](https://developer.softbankrobotics.com/nao6/nao-documentation/nao-user-guide/configuring-and-settings/setting-nao-s-preferred-language#-ordering-additional-languages-). 48 | 49 | ### 3.2. Install the app using Choregraphe 50 | 51 | To install this application on your NAO, you need to use the [Choregraphe suite](https://developer.softbankrobotics.com/nao6/naoqi-developer-guide/choregraphe-suite/what-choregraphe). 52 | 53 | First launch Choregraphe. Click on **`File` > `Open Project`**, select the `deep_nao.pml` file located in this project folder, then click **`Open`**. 54 | 55 | Then [connect Choregraphe to your NAO robot](https://developer.softbankrobotics.com/nao6/naoqi-developer-guide/choregraphe-suite/connecting-choregraphe-robot#chore-howto-connect). 56 | 57 | Finally open the [Robot applications panel](https://developer.softbankrobotics.com/nao6/naoqi-developer-guide/choregraphe-suite/main-panels/robot-applications-panel#-what-is-the-robot-applications-panel-), and click on the `Package and install current project to the robot` icon ![](doc/application_manager_add_current.png), located in the [Robot application panel toolbar](https://developer.softbankrobotics.com/nao6/naoqi-developer-guide/choregraphe-suite/main-panels/robot-applications-panel#app-man-toolbar). This will package and install the Deep NAO application to your NAO robot. 58 | 59 | Once this is done, you can close Choregraphe, the app has been installed on your NAO. 60 | 61 | ## 4. Use the app 62 | 63 | ### 4.1. Start the app 64 | 65 | The app start automatically when the robot is in [solitary](https://developer.softbankrobotics.com/nao6/naoqi-developer-guide/programming-living-robot/state-machine-management#-states-) or [interactive](https://developer.softbankrobotics.com/nao6/naoqi-developer-guide/programming-living-robot/state-machine-management#-states-) state. It is stopped when the robot is in [disabled](https://developer.softbankrobotics.com/nao6/naoqi-developer-guide/programming-living-robot/state-machine-management#-states-) or [safeguard](https://developer.softbankrobotics.com/nao6/naoqi-developer-guide/programming-living-robot/state-machine-management#-states-) state. To enable the application when the robot is in [disabled](https://developer.softbankrobotics.com/nao6/naoqi-developer-guide/programming-living-robot/state-machine-management#-states-) state, [double click on NAO chest button](https://developer.softbankrobotics.com/nao6/nao-documentation/nao-user-guide/daily-use/switching-autonomous-life-and#nao-freeze). 66 | 67 | ### 4.2. Recognized objects 68 | 69 | We provide a default deep learning model that was trained on [Google Open Image Dataset](https://storage.googleapis.com/openimages/web/index.html). Using this model, NAO can recognize up to 600 different objects. 70 | 71 | You can view the complete list of recognized object in [this file](models/ssd_mobilenet_v2_oid_v4/objects.names.en) (open it with a text editor). 72 | 73 | ### 4.3. Remote control the app via its web page 74 | 75 | #### 4.3.1. Open the page in your internet browser 76 | 77 | A web page is available that allows you to remote control the application. 78 | You can access it at the url: `http:///apps/deep_nao/` 79 | 80 | Replace `` by the [real ip of the NAO robot](https://developer.softbankrobotics.com/nao6/nao-documentation/nao-user-guide/daily-use/requesting-technical-information#-reading-ip-address-and-notifications-) (for instance something like 192.168.0.2). 81 | 82 | #### 4.3.2. Application disabled 83 | 84 | If the application is disabled, the web page will display the following message: 85 | 86 |
87 | 88 |
89 | 90 | In that case, see [4.1. Start the app](#41-start-the-app) to find out how to start the application. 91 | 92 | ### 4.4. Interact with the app via the web page 93 | 94 | When the Deep NAO application is running, the remote control webpage looks as follow: 95 | 96 |
97 | 98 |
99 | 100 | When the app is active, the web page will display NAO camera stream, so that you can see what objects NAO sees and recognizes. 101 | When NAO recognizes an object, it will display a yellow square around the object in the camera stream image. The name of the object will also be displayed. 102 | 103 | You can ask NAO to say the name of the objects it recognizes. To do so you can either: 104 | 105 | * click on the `Describe what you see` button: NAO will say what object it recognizes at the moment 106 | * toggle the `Describe continuously` switch button: Whenever NAO recognizes a new object, it will say its name. It will continue doing so until you toggle back the switch to off. 107 | * toggle the `Explore continuously` switch button: NAO will start to walk randomly in the room to search for objects to describe. As with the `Describe continuously` button, whenever it recognize a new object it will say its name. 108 | 109 | When you have uploaded a custom model to the app, a `Use custom model` switch will appear. Toggle it if you want NAO to use your custom model. 110 | 111 | ## 5. Training your custom model 112 | 113 | You can train your own custom model so that NAO recognizes the objects you taught it to learn. 114 | 115 | We provide a tutorial in the form of a [Google Colab notebook tutorial](https://colab.research.google.com/github/softbankrobotics-labs/nao-deep-learning/blob/master/notebook/NAO_Deep_Learning_Custom_Model_Training.ipynb) that allow you to train a custom network to recognize two different objects. You can pick these two objects from the 600 objects available in the [Google Open Image dataset](https://storage.googleapis.com/openimages/web/index.html). 116 | 117 | 118 | Now go to the [Colab tutorial](https://colab.research.google.com/github/softbankrobotics-labs/nao-deep-learning/blob/master/notebook/NAO_Deep_Learning_Custom_Model_Training.ipynb), and follow the instructions! 119 | 120 | 121 | ## 6. Uploading your custom model to the app 122 | 123 | Once you have trained your custom model using the [Colab tutorial](https://colab.research.google.com/github/softbankrobotics-labs/nao-deep-learning/blob/master/notebook/NAO_Deep_Learning_Custom_Model_Training.ipynb), download the resulting files from your Google Drive. 124 | The tutorial should have created a folder `NAO_DeepLearning` in your Google drive, with a `result` folder in it. Download all the files from the `NAO_DeepLearning/result` folder. There should be the following files: 125 | 126 | * `model.cfg` 127 | * `model.weights` 128 | * `objects.names.en` 129 | * `objects.names.fr` 130 | 131 | Once you have these four files, you need to upload them to the Deep NAO app. To do so, open the special url: `http:///apps/deep_nao/upload.html` in your browser. 132 | 133 | Replace `` by the [real ip of the NAO robot](https://developer.softbankrobotics.com/nao6/nao-documentation/nao-user-guide/daily-use/requesting-technical-information#-reading-ip-address-and-notifications-) (for instance something like 192.168.0.2). 134 | 135 | Your browser will display: 136 | 137 |
138 | 139 |
140 | 141 | Click on each browse button to select the four files you just donwloaded. Then click on upload files. 142 | Once the upload is done, you will be redirected to the Deep NAO main web page. Toggle the `Use custom model` ![](doc/use_custom_model.png) switch and NAO will start using your custom model instead of the default one. 143 | 144 | If you want to revert to the default model, toggle back the `Use custom model` ![](doc/use_custom_model.png) switch. 145 | -------------------------------------------------------------------------------- /behavior/behavior.xar: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | media/images/box/root.png 5 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | -------------------------------------------------------------------------------- /deep_nao.pml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | -------------------------------------------------------------------------------- /dialog/dialog-en.top: -------------------------------------------------------------------------------- 1 | topic: ~main () 2 | language: English 3 | 4 | u:(What do you see) ^pCall(DeepNao.getVisibleObjects()) 5 | c1:(_*) I see $1 6 | e1: I don't see any object at the moment 7 | 8 | #u:(Did you see _*) ^pCall() 9 | -------------------------------------------------------------------------------- /doc/application_manager_add_current.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softbankrobotics-labs/nao-deep-learning/7bb0226041b4636207ae6b2346b581c8a147b0a6/doc/application_manager_add_current.png -------------------------------------------------------------------------------- /doc/disabled.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softbankrobotics-labs/nao-deep-learning/7bb0226041b4636207ae6b2346b581c8a147b0a6/doc/disabled.png -------------------------------------------------------------------------------- /doc/nao.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softbankrobotics-labs/nao-deep-learning/7bb0226041b4636207ae6b2346b581c8a147b0a6/doc/nao.jpg -------------------------------------------------------------------------------- /doc/shell.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softbankrobotics-labs/nao-deep-learning/7bb0226041b4636207ae6b2346b581c8a147b0a6/doc/shell.png -------------------------------------------------------------------------------- /doc/upload_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softbankrobotics-labs/nao-deep-learning/7bb0226041b4636207ae6b2346b581c8a147b0a6/doc/upload_model.png -------------------------------------------------------------------------------- /doc/use_custom_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softbankrobotics-labs/nao-deep-learning/7bb0226041b4636207ae6b2346b581c8a147b0a6/doc/use_custom_model.png -------------------------------------------------------------------------------- /html/img/.empty: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softbankrobotics-labs/nao-deep-learning/7bb0226041b4636207ae6b2346b581c8a147b0a6/html/img/.empty -------------------------------------------------------------------------------- /html/img/deep.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softbankrobotics-labs/nao-deep-learning/7bb0226041b4636207ae6b2346b581c8a147b0a6/html/img/deep.jpg -------------------------------------------------------------------------------- /html/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | NAO Deep Learning 5 | 6 | 7 | 8 | 9 | 10 | 11 |
12 |
13 | 14 |
15 |
16 |
17 | 18 |
19 |
20 | Describe continuously: 21 | 25 |
26 |
27 | Explore continuously: 28 | 32 |
33 |
34 | Use custom model: 35 | 39 |
40 |
41 |
42 |
43 | DeepNao service is not running. Double click on NAO chest button. 44 |
45 | 46 | -------------------------------------------------------------------------------- /html/index.js: -------------------------------------------------------------------------------- 1 | 2 | var img; 3 | var preloadImage = new Image(); 4 | preloadImage.onload = function() { 5 | document.getElementById("img").src = preloadImage.src; 6 | } 7 | 8 | function updateImageSrc() { 9 | let dt = new Date(); 10 | let baseUrl = "img/deep.jpg"; 11 | img = baseUrl + "?t=" + dt.getTime(); 12 | preloadImage.src = img; 13 | } 14 | 15 | setInterval(updateImageSrc, 300); 16 | 17 | function setImageClickListener() { 18 | document.querySelector('#img').onclick = function (e) { 19 | var rect = e.target.getBoundingClientRect(); 20 | var x = (e.clientX - rect.left) / (rect.width / 2) - 1; //x position within the element. 21 | var y = -((e.clientY - rect.top) / (rect.height / 2) - 1); //y position within the element. 22 | DeepNaoService.webLookAt(x, y); 23 | console.log(x); 24 | console.log(y); 25 | } 26 | } 27 | 28 | var DeepNaoService = null; 29 | 30 | function setContinuousDescritionCheckboxClickListener() { 31 | const checkbox = document.getElementById('continuous_description_checkbox') 32 | checkbox.addEventListener('change', (event) => { 33 | if (event.currentTarget.checked) { 34 | DeepNaoService.continuous_description.setValue(true); 35 | } else { 36 | DeepNaoService.continuous_description.setValue(false); 37 | } 38 | console.log("Checkbox checked: " + event.currentTarget.checked); 39 | }); 40 | } 41 | 42 | function setContinuousExplorationCheckboxClickListener() { 43 | const checkbox = document.getElementById('continuous_exploration_checkbox') 44 | checkbox.addEventListener('change', (event) => { 45 | if (event.currentTarget.checked) { 46 | DeepNaoService.continuous_exploration.setValue(true); 47 | } else { 48 | DeepNaoService.continuous_exploration.setValue(false); 49 | } 50 | console.log("Checkbox checked: " + event.currentTarget.checked); 51 | }); 52 | } 53 | 54 | function setCustomModelCheckboxClickListener() { 55 | const checkbox = document.getElementById('use_custom_model_checkbox') 56 | checkbox.addEventListener('change', (event) => { 57 | if (event.currentTarget.checked) { 58 | DeepNaoService.use_custom_model.setValue(true); 59 | } else { 60 | DeepNaoService.use_custom_model.setValue(false); 61 | } 62 | console.log("Checkbox checked: " + event.currentTarget.checked); 63 | }); 64 | } 65 | 66 | function setDescribeButtonClickListener() { 67 | document.getElementById("describe").addEventListener("click", function() { 68 | DeepNaoService.describeVisibleObjects(); 69 | }); 70 | } 71 | 72 | function onContinuousDescriptionChange(value) { 73 | const checkbox = document.getElementById('continuous_description_checkbox') 74 | checkbox.checked = value; 75 | console.log("Checkbox checked: " + value); 76 | } 77 | 78 | function onContinuousExplorationChange(value) { 79 | const checkbox = document.getElementById('continuous_exploration_checkbox') 80 | checkbox.checked = value; 81 | console.log("Checkbox checked: " + value); 82 | } 83 | 84 | function onUseCustomModelChange(value) { 85 | const checkbox = document.getElementById('use_custom_model_checkbox') 86 | checkbox.checked = value; 87 | console.log("Checkbox checked: " + value); 88 | } 89 | 90 | function onCustomModelAvailable(value) { 91 | if (value) { 92 | document.getElementById('use_custom_model_line').style.display = "flex"; 93 | } 94 | } 95 | 96 | function onDeepNaoRunningChange(isRunning) { 97 | if (isRunning) { 98 | document.getElementById('main').style.display = "flex"; 99 | document.getElementById('not_running').style.display = "none"; 100 | } else { 101 | document.getElementById('main').style.display = "none"; 102 | document.getElementById('not_running').style.display = "block"; 103 | } 104 | } 105 | 106 | function onLanguageChange(value) { 107 | console.log("onLanguageChange: " + value) 108 | if ((value == "French") || (value == "fr_FR")) { 109 | document.getElementById('describe').textContent = "Décris ce que tu vois"; 110 | document.getElementById('continuous_describe_label').textContent = "Décris en continue:"; 111 | document.getElementById('continuous_explore_label').textContent = "Explore en continue:"; 112 | document.getElementById('use_custom_model_label').textContent = "Utilise un modèle personnalisé:"; 113 | document.getElementById('not_running').textContent = "Le service DeepNao n'est pas actif. Veuillez mettre NAO debout ou assis puis double cliquer sur le bouton torse de NAO."; 114 | } else { 115 | document.getElementById('describe').textContent = "Describe what you see"; 116 | document.getElementById('continuous_describe_label').textContent = "Describe continuously:"; 117 | document.getElementById('continuous_explore_label').textContent = "Explore continuously:"; 118 | document.getElementById('use_custom_model_label').textContent = "Use custom model:"; 119 | document.getElementById('not_running').textContent = "DeepNao service is not running. Please sit or stand NAO then double click on NAO chest button."; 120 | } 121 | } 122 | 123 | RobotUtils.onService(function(DeepNao) { 124 | console.log("Service DeepNao available"); 125 | DeepNaoService = DeepNao; 126 | setImageClickListener(); 127 | setContinuousDescritionCheckboxClickListener(); 128 | setContinuousExplorationCheckboxClickListener(); 129 | setCustomModelCheckboxClickListener(); 130 | setDescribeButtonClickListener(); 131 | DeepNao.continuous_description.value().then(onContinuousDescriptionChange); 132 | DeepNao.continuous_exploration.value().then(onContinuousExplorationChange); 133 | DeepNao.use_custom_model.value().then(onUseCustomModelChange); 134 | DeepNao.use_custom_model.connect(onUseCustomModelChange); 135 | DeepNao.isCustomModelAvailable().then(onCustomModelAvailable); 136 | DeepNao.running.value().then(onDeepNaoRunningChange); 137 | DeepNao.running.connect(onDeepNaoRunningChange); 138 | }); 139 | 140 | RobotUtils.onService(function(ALTextToSpeech) { 141 | console.log("Service ALTextToSpeech available"); 142 | ALTextToSpeech.getLanguage().then(onLanguageChange); 143 | ALTextToSpeech.languageTTS.connect(onLanguageChange); 144 | }); 145 | -------------------------------------------------------------------------------- /html/robotutils.js: -------------------------------------------------------------------------------- 1 | /* 2 | * robotutils.js version 0.2 3 | * 4 | * A utility library for naoqi; 5 | * 6 | * This library is a wrapper over qimessaging.js. Some advantages: 7 | * - debugging and iterating are made easier by support for a 8 | * ?robot= query parameter in the URL, that allows 9 | * you to open a local file and connect it to a remote robot. 10 | * - there is some syntactic sugar over common calls that 11 | * allows you to keep your logic simple without too much nesting 12 | * 13 | * You can of course directly use qimessaging.js instead. 14 | * 15 | * See the method documentations below for sample usage. 16 | * 17 | * Copyright Aldebaran Robotics 18 | * Authors: ekroeger@aldebaran.com, jjeannin@aldebaran.com 19 | */ 20 | 21 | RobotUtils = (function(self) { 22 | 23 | /*--------------------------------------------- 24 | * Public API 25 | */ 26 | 27 | /* RobotUtils.onServices(servicesCallback, errorCallback) 28 | * 29 | * A function for using NAOqi services. 30 | * 31 | * "servicesCallback" should be a function whose arguments are the 32 | * names of NAOqi services; the callback will be called 33 | * with those services as parameters (or the errorCallback 34 | * will be called with a reason). 35 | * 36 | * Sample usage: 37 | * 38 | * RobotUtils.onServices(function(ALLeds, ALTextToSpeech) { 39 | * ALLeds.randomEyes(2.0); 40 | * ALTextToSpeech.say("I can speak"); 41 | * }); 42 | * 43 | * This is actually syntactic sugar over RobotUtils.connect() and 44 | * some basic QiSession functions, so that the code stays simple. 45 | */ 46 | self.onServices = function(servicesCallback, errorCallback) { 47 | self.connect(function(session) { 48 | var wantedServices = getParamNames(servicesCallback); 49 | var pendingServices = wantedServices.length; 50 | var services = new Array(wantedServices.length); 51 | var i; 52 | for (i = 0; i < wantedServices.length; i++) { 53 | (function (i){ 54 | session.service(wantedServices[i]).then(function(service) { 55 | services[i] = service; 56 | pendingServices -= 1; 57 | if (pendingServices == 0) { 58 | servicesCallback.apply(undefined, services); 59 | } 60 | }, function() { 61 | var reason = "Failed getting a NaoQi Module: " + wantedServices[i] 62 | console.log(reason); 63 | if (errorCallback) { 64 | errorCallback(reason); 65 | } 66 | }); 67 | })(i); 68 | } 69 | }, errorCallback); 70 | } 71 | 72 | // alias, so that the code looks natural when there is only one service. 73 | self.onService = self.onServices; 74 | 75 | /* RobotUtils.subscribeToALMemoryEvent(event, eventCallback, subscribeDoneCallback) 76 | * 77 | * connects a callback to an ALMemory event. Returns a MemoryEventSubscription. 78 | * 79 | * This is just syntactic sugar over calls to the ALMemory service, which you can 80 | * do yourself if you want finer control. 81 | */ 82 | self.subscribeToALMemoryEvent = function(event, eventCallback, subscribeDoneCallback) { 83 | var evt = new MemoryEventSubscription(event); 84 | self.onServices(function(ALMemory) { 85 | ALMemory.subscriber(event).then(function (sub) { 86 | evt.setSubscriber(sub) 87 | sub.signal.connect(eventCallback).then(function(id) { 88 | evt.setId(id); 89 | if (subscribeDoneCallback) subscribeDoneCallback(id) 90 | }); 91 | }, 92 | onALMemoryError); 93 | }); 94 | return evt; 95 | } 96 | 97 | /* RobotUtils.connect(connectedCallback, failureCallback) 98 | * 99 | * connectedCallback should take a single argument, a NAOqi session object 100 | * 101 | * This function is mostly meant for internal use, for your app you 102 | * should probably use the more specific RobotUtils.onServices or 103 | * RobotUtils.subscribeToALMemoryEvent. 104 | * 105 | * There can be several calls to .connect() in parallel, only one 106 | * session will be created. 107 | */ 108 | self.connect = function(connectedCallback, failureCallback) { 109 | if (self.session) { 110 | // We already have a session, don't create a new one 111 | connectedCallback(self.session); 112 | return; 113 | } 114 | else if (pendingConnectionCallbacks.length > 0) { 115 | // A connection attempt is in progress, just add this callback to the queue 116 | pendingConnectionCallbacks.push(connectedCallback); 117 | return; 118 | } 119 | else { 120 | // Add self to the queue, but create a new connection. 121 | pendingConnectionCallbacks.push(connectedCallback); 122 | } 123 | 124 | var qimAddress = null; 125 | var robotlibs = '/libs/'; 126 | if (self.robotIp) { 127 | // Special case: we're doing remote debugging on a robot. 128 | robotlibs = "http://" + self.robotIp + "/libs/"; 129 | qimAddress = self.robotIp + ":80"; 130 | } 131 | 132 | function onConnected(session) { 133 | self.session = session; 134 | var numCallbacks = pendingConnectionCallbacks.length; 135 | for (var i = 0; i < numCallbacks; i++) { 136 | pendingConnectionCallbacks[i](session); 137 | } 138 | } 139 | 140 | getScript(robotlibs + 'qimessaging/2/qimessaging.js', function() { 141 | QiSession( 142 | onConnected, 143 | failureCallback, 144 | qimAddress 145 | ) 146 | }, function() { 147 | if (self.robotIp) { 148 | console.error("Failed to get qimessaging.js from robot: " + self.robotIp); 149 | } else { 150 | console.error("Failed to get qimessaging.js from this domain; host this app on a robot or add a ?robot=MY-ROBOT-IP to the URL."); 151 | } 152 | failureCallback(); 153 | }); 154 | } 155 | 156 | // public variables that can be useful. 157 | self.robotIp = _getRobotIp(); 158 | self.session = null; 159 | 160 | /*--------------------------------------------- 161 | * Internal helper functions 162 | */ 163 | 164 | // Replacement for jQuery's getScript function 165 | function getScript(source, successCallback, failureCallback) { 166 | var script = document.createElement('script'); 167 | var prior = document.getElementsByTagName('script')[0]; 168 | script.async = 1; 169 | prior.parentNode.insertBefore(script, prior); 170 | 171 | script.onload = script.onreadystatechange = function( _, isAbort ) { 172 | if(isAbort || !script.readyState || /loaded|complete/.test(script.readyState) ) { 173 | script.onload = script.onreadystatechange = null; 174 | script = undefined; 175 | 176 | if(isAbort) { 177 | if (failureCallback) failureCallback(); 178 | } else { 179 | // Success! 180 | if (successCallback) successCallback(); 181 | } 182 | } 183 | }; 184 | script.src = source; 185 | } 186 | 187 | function _getRobotIp() { 188 | var regex = new RegExp("[\\?&]robot=([^&#]*)"); 189 | var results = regex.exec(location.search); 190 | return results == null ? "" : decodeURIComponent(results[1].replace(/\+/g, " ").replace("/", "")); 191 | } 192 | 193 | // Helper for getting the parameters from a function. 194 | var STRIP_COMMENTS = /((\/\/.*$)|(\/\*[\s\S]*?\*\/))/mg; 195 | function getParamNames(func) { 196 | var fnStr = func.toString().replace(STRIP_COMMENTS, ''); 197 | var result = fnStr.slice(fnStr.indexOf('(')+1, fnStr.indexOf(')')).match(/([^\s,]+)/g); 198 | if(result === null) 199 | result = []; 200 | return result; 201 | }; 202 | 203 | // ALMemory helpers (event subscription requires a lot of boilerplate) 204 | 205 | function MemoryEventSubscription(event) { 206 | this._event = event; 207 | this._internalId = null; 208 | this._sub = null; 209 | this._unsubscribe = false; 210 | } 211 | 212 | MemoryEventSubscription.prototype.setId = function(id) { 213 | this._internalId = id; 214 | // as id can be receveid after unsubscribe call, defere 215 | if (this._unsubscribe) this.unsubscribe(this._unsubscribeCallback); 216 | } 217 | 218 | MemoryEventSubscription.prototype.setSubscriber = function(sub) { 219 | this._sub = sub; 220 | // as sub can be receveid after unsubscribe call, defere 221 | if (this._unsubscribe) this.unsubscribe(this._unsubscribeCallback); 222 | } 223 | 224 | MemoryEventSubscription.prototype.unsubscribe = function(unsubscribeDoneCallback) 225 | { 226 | if (this._internalId != null && this._sub != null) { 227 | evtSubscription = this; 228 | evtSubscription._sub.signal.disconnect(evtSubscription._internalId).then(function() { 229 | if (unsubscribeDoneCallback) unsubscribeDoneCallback(); 230 | }).fail(onALMemoryError); 231 | } 232 | else 233 | { 234 | this._unsubscribe = true; 235 | this._unsubscribeCallback = unsubscribeDoneCallback; 236 | } 237 | } 238 | 239 | var onALMemoryError = function(errMsg) { 240 | console.log("ALMemory error: " + errMsg); 241 | } 242 | 243 | var pendingConnectionCallbacks = []; 244 | 245 | return self; 246 | 247 | })(window.RobotUtils || {}); 248 | -------------------------------------------------------------------------------- /html/style.css: -------------------------------------------------------------------------------- 1 | body, html, #main { 2 | height: 100%; 3 | width: 100%; 4 | margin: 0; 5 | background-color: #eee; 6 | overflow: hidden; 7 | font-family: 'Brush Script MT', cursive; 8 | font-size: 30px; 9 | } 10 | 11 | #main { 12 | display: none; 13 | } 14 | 15 | #img_div { 16 | float: left; 17 | height: 100%; 18 | display: block; 19 | } 20 | 21 | #img { 22 | border-radius: 20px; 23 | border: 5px solid #aaa; 24 | height: 99%; 25 | border-radius: 20px; 26 | } 27 | 28 | #controls { 29 | padding: 20px; 30 | position: relative; 31 | display: block; 32 | float: left; 33 | flex-grow: 1; 34 | } 35 | 36 | .line { 37 | height: 100px; 38 | color: #333; 39 | display: flex; 40 | } 41 | 42 | button:active { 43 | background-color: #0176d3; 44 | } 45 | 46 | 47 | button:disabled, 48 | button[disabled]{ 49 | background-color: #ccc; 50 | color: #888; 51 | } 52 | 53 | button { 54 | border-radius: 8px; 55 | background-color: #2196F3; 56 | border: none; 57 | text-align: center; 58 | text-decoration: none; 59 | display: inline-block; 60 | width: 100%; 61 | height: 80px; 62 | font-family: 'Brush Script MT', cursive; 63 | font-size: 30px; 64 | color: white; 65 | } 66 | .label { 67 | flex-grow: 1; 68 | } 69 | 70 | .switch { 71 | position: relative; 72 | display: inline-block; 73 | width: 60px; 74 | height: 34px; 75 | } 76 | 77 | .switch input { 78 | opacity: 0; 79 | width: 0; 80 | height: 0; 81 | } 82 | 83 | .slider { 84 | position: absolute; 85 | cursor: pointer; 86 | top: 0; 87 | left: 0; 88 | right: 0; 89 | bottom: 0; 90 | background-color: #ccc; 91 | -webkit-transition: .4s; 92 | transition: .4s; 93 | } 94 | 95 | .slider:before { 96 | position: absolute; 97 | content: ""; 98 | height: 26px; 99 | width: 26px; 100 | left: 4px; 101 | bottom: 4px; 102 | background-color: white; 103 | -webkit-transition: .4s; 104 | transition: .4s; 105 | } 106 | 107 | input:checked + .slider { 108 | background-color: #2196F3; 109 | } 110 | 111 | input:focus + .slider { 112 | box-shadow: 0 0 1px #2196F3; 113 | } 114 | 115 | input:checked + .slider:before { 116 | -webkit-transform: translateX(26px); 117 | -ms-transform: translateX(26px); 118 | transform: translateX(26px); 119 | } 120 | 121 | /* Rounded sliders */ 122 | .slider.round { 123 | border-radius: 34px; 124 | } 125 | 126 | .slider.round:before { 127 | border-radius: 50%; 128 | } 129 | 130 | #use_custom_model_line { 131 | display: none; 132 | } 133 | 134 | /* Upload */ 135 | 136 | #upload_model { 137 | margin: auto; 138 | height: 100%; 139 | display: flex; 140 | flex-direction: column; 141 | } 142 | 143 | label { 144 | font-size: 17px; 145 | width: 50%; 146 | display: inline-block; 147 | } 148 | 149 | .space { 150 | flex-grow: 1; 151 | } 152 | 153 | #upload { 154 | margin-bottom: 40px; 155 | } 156 | 157 | 158 | 159 | /* Loading */ 160 | 161 | @keyframes spin { 162 | 0% { 163 | transform: rotate(0deg); 164 | } 165 | 100% { 166 | transform: rotate(360deg); 167 | } 168 | } 169 | 170 | 171 | .loadingicon { 172 | border-radius: 50%; 173 | width: 200px; 174 | height: 200px; 175 | border: 30px solid white; 176 | /*animation: spin 1.5s infinite linear;*/ 177 | margin: 0 auto; 178 | text-align: center; 179 | display: flex; 180 | justify-content: center; 181 | align-content: center; 182 | flex-direction: column; 183 | position: relative; 184 | top: 30%; 185 | } 186 | 187 | 188 | @keyframes blinker { 189 | 50% { 190 | opacity: 0; 191 | } 192 | } 193 | 194 | .loadingtext { 195 | display: inline; 196 | color: white; 197 | animation: blinker 2s linear infinite; 198 | } 199 | 200 | #blackout { 201 | display: none; 202 | background:rgba(0,0,0,.7); 203 | width:100%; 204 | height:100%; 205 | position: absolute; 206 | left:0; 207 | top:0; 208 | z-index:10 209 | } 210 | 211 | #not_running { 212 | display: block; 213 | margin: auto; 214 | text-align: center; 215 | width: 50%; 216 | padding: 100px; 217 | } 218 | -------------------------------------------------------------------------------- /html/upload.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | NAO Deep Learning - Upload a custom model 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
13 |
14 |

Upload a custom model

15 |
16 | 18 | 19 |
20 |
21 | 23 | 24 |
25 |
26 | 28 | 29 |
30 |
31 | 33 | 34 |
35 |
36 | 37 |
38 |
39 |
40 |
41 | Uploading... 42 |
43 |
44 | 45 | 46 | 47 | -------------------------------------------------------------------------------- /html/upload.js: -------------------------------------------------------------------------------- 1 | 2 | 3 | function _arrayBufferToBase64(bytes) { 4 | var binary = ''; 5 | var len = bytes.byteLength; 6 | for (var i = 0; i < len; i++) { 7 | binary += String.fromCharCode( bytes[ i ] ); 8 | } 9 | return window.btoa( binary ); 10 | } 11 | 12 | function parseFile(file, callback) { 13 | return new Promise(function (resolve, reject) { 14 | var fileSize = file.size; 15 | var chunkSize = 64 * 1024; // bytes 16 | var offset = 0; 17 | var self = this; // we need a reference to the current object 18 | var chunkReaderBlock = null; 19 | 20 | var readEventHandler = function(evt) { 21 | if (evt.target.error == null) { 22 | var bytes = new Uint8Array(evt.target.result) 23 | console.log(bytes.byteLength); 24 | callback(_arrayBufferToBase64(bytes), offset); // callback for handling read chunk 25 | offset += bytes.byteLength; 26 | } else { 27 | console.log("Read error: " + evt.target.error); 28 | reject(); 29 | return; 30 | } 31 | if (offset >= fileSize) { 32 | resolve(fileSize); 33 | console.log("Done reading file"); 34 | return; 35 | } 36 | 37 | // of to the next chunk 38 | chunkReaderBlock(offset, chunkSize, file); 39 | } 40 | 41 | chunkReaderBlock = function(_offset, length, _file) { 42 | var r = new FileReader(); 43 | var blob = _file.slice(_offset, length + _offset); 44 | r.onload = readEventHandler; 45 | r.readAsArrayBuffer(blob); 46 | } 47 | 48 | // now let's start the read with the first block 49 | chunkReaderBlock(offset, chunkSize, file); 50 | }); 51 | } 52 | 53 | var DeepNaoService = null; 54 | 55 | function uploadFile(file_type, input_id) { 56 | console.log("Upload file") 57 | return new Promise(function (resolve, reject) { 58 | DeepNaoService.startUpload(file_type); 59 | parseFile(document.getElementById(input_id).files[0], function(chunk, offset) { 60 | DeepNaoService.uploadChunk(chunk, offset, file_type); 61 | }).then(function (size) { 62 | DeepNaoService.finishUpload(size, file_type).then(function () { 63 | resolve(); 64 | }) 65 | }, function (error) { 66 | reject(error); 67 | }); 68 | }); 69 | 70 | } 71 | 72 | function displayLoading() { 73 | document.getElementById("blackout").style.display = "block"; 74 | } 75 | 76 | function hideLoading() { 77 | document.getElementById("blackout").style.display = "none"; 78 | document.getElementById("loadingicon").style.borderColor = "white"; 79 | } 80 | 81 | function uploadError(e) { 82 | alert("Error uploading file"); 83 | hideLoading(); 84 | } 85 | 86 | function upload() { 87 | color = "#0176d3"; 88 | displayLoading(); 89 | document.getElementById("loadingicon").style.borderLeftColor = color; 90 | uploadFile("weights", 'weights-input').then(function () { 91 | document.getElementById("loadingicon").style.borderTopColor = color; 92 | uploadFile("config", 'config-input').then(function () { 93 | document.getElementById("loadingicon").style.borderRightColor = color; 94 | uploadFile("names_fr", 'names-fr-input').then(function () { 95 | document.getElementById("loadingicon").style.borderBottomColor = color; 96 | uploadFile("names_en", 'names-en-input').then(function () { 97 | hideLoading(); 98 | window.location.href = "index.html" 99 | }, uploadError); 100 | }, uploadError); 101 | }, uploadError); 102 | }, uploadError); 103 | } 104 | 105 | function enableUpload() { 106 | document.getElementById('upload').disabled = !Boolean( 107 | document.getElementById('config-input').value && 108 | document.getElementById('weights-input').value && 109 | document.getElementById('names-fr-input').value && 110 | document.getElementById('names-en-input').value 111 | ); 112 | } 113 | 114 | document.getElementById('config-input').addEventListener('change', enableUpload, false); 115 | document.getElementById('weights-input').addEventListener('change', enableUpload, false); 116 | document.getElementById('names-fr-input').addEventListener('change', enableUpload, false); 117 | document.getElementById('names-en-input').addEventListener('change', enableUpload, false); 118 | 119 | 120 | function setUploadClickListener() { 121 | document.querySelector('#upload').onclick = function (e) { 122 | upload(); 123 | } 124 | } 125 | 126 | function onLanguageChange(value) { 127 | console.log("onLanguageChange: " + value) 128 | if ((value == "French") || (value == "fr_FR")) { 129 | document.getElementById('title').textContent = "Télécharger un modèle personnalisé"; 130 | document.getElementById('config-input-label').textContent 131 | = "Fichier de configuration du modèle (model.cfg)"; 132 | document.getElementById('weight-input-label').textContent 133 | = "Fichier de poids du modèle (model.weights)"; 134 | document.getElementById('names-fr-input-label').textContent 135 | = "Noms des objets en Francais (objects.names.fr)"; 136 | document.getElementById('names-en-input-label').textContent 137 | = "Noms des objets en Anglais (objects.names.en)"; 138 | document.getElementById('upload').textContent = "Télécharger les fichiers"; 139 | document.getElementById('loadingtext').textContent = "Téléchargement..."; 140 | } else { 141 | document.getElementById('title').textContent = "Upload a custom model"; 142 | document.getElementById('config-input-label').textContent 143 | = "Model config file (model.cfg):"; 144 | document.getElementById('weight-input-label').textContent 145 | = "Model weight file (model.weights):"; 146 | document.getElementById('names-fr-input-label').textContent 147 | = "Objects names in French (objects.names.fr):"; 148 | document.getElementById('names-en-input-label').textContent 149 | = "Objects names in English (objects.names.en):"; 150 | document.getElementById('upload').textContent = "Upload files"; 151 | document.getElementById('loadingtext').textContent = "Uploading..."; 152 | } 153 | } 154 | 155 | RobotUtils.onService(function(DeepNao) { 156 | document.getElementById('main').style.display = "flex"; 157 | console.log("Service DeepNao available"); 158 | DeepNaoService = DeepNao; 159 | setUploadClickListener(); 160 | }); 161 | 162 | RobotUtils.onService(function(ALTextToSpeech) { 163 | console.log("Service ALTextToSpeech available"); 164 | ALTextToSpeech.getLanguage().then(onLanguageChange); 165 | ALTextToSpeech.languageTTS.connect(onLanguageChange); 166 | }); 167 | -------------------------------------------------------------------------------- /install.sh: -------------------------------------------------------------------------------- 1 | 2 | #!/bin/sh 3 | 4 | if [ $# -ne 1 ]; then 5 | echo "Usage: $0 " 6 | exit 1 7 | fi 8 | 9 | qipkg make-package deep_nao.pml 10 | robot=$1 11 | rsync -arv deep_nao-1.0.0.pkg nao@$robot: 12 | ssh nao@$robot qicli call PackageManager.remove deep_nao 13 | ssh nao@$robot qicli call PackageManager.install /home/nao/deep_nao-1.0.0.pkg 14 | -------------------------------------------------------------------------------- /lib/python2.7/site-packages/cv2.so: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softbankrobotics-labs/nao-deep-learning/7bb0226041b4636207ae6b2346b581c8a147b0a6/lib/python2.7/site-packages/cv2.so -------------------------------------------------------------------------------- /manifest.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | deep_nao 5 | deep_nao 6 | 7 | 8 | Deep learning for NAO, applied to object recognition. 9 | Apprentissage profond pour NAO, applique a la reconnaissance d'objets. 10 | 11 | 12 | fr_FR 13 | en_US 14 | 15 | 16 | fr_FR 17 | en_US 18 | 19 | 20 | 21 | interactive 22 | 23 | 24 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | -------------------------------------------------------------------------------- /models/ssd_mobilenet_v2_oid_v4/frozen_inference_graph.pb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softbankrobotics-labs/nao-deep-learning/7bb0226041b4636207ae6b2346b581c8a147b0a6/models/ssd_mobilenet_v2_oid_v4/frozen_inference_graph.pb -------------------------------------------------------------------------------- /models/ssd_mobilenet_v2_oid_v4/objects.names.en: -------------------------------------------------------------------------------- 1 | Tortoise 2 | Container 3 | Magpie 4 | Sea turtle 5 | Football 6 | Ambulance 7 | Ladder 8 | Toothbrush 9 | Syringe 10 | Sink 11 | Toy 12 | Organ 13 | Cassette deck 14 | Apple 15 | Human eye 16 | Cosmetics 17 | Paddle 18 | Snowman 19 | Beer 20 | Chopsticks 21 | Human beard 22 | Bird 23 | Parking meter 24 | Traffic light 25 | Croissant 26 | Cucumber 27 | Radish 28 | Towel 29 | Doll 30 | Skull 31 | Washing machine 32 | Glove 33 | Tick 34 | Belt 35 | Sunglasses 36 | Banjo 37 | Cart 38 | Ball 39 | Backpack 40 | Bicycle 41 | Home appliance 42 | Centipede 43 | Boat 44 | Surfboard 45 | Boot 46 | Headphones 47 | Hot dog 48 | Shorts 49 | Fast food 50 | Bus 51 | Boy 52 | Screwdriver 53 | Bicycle wheel 54 | Barge 55 | Laptop 56 | Miniskirt 57 | Drill 58 | Dress 59 | Bear 60 | Waffle 61 | Pancake 62 | Brown bear 63 | Woodpecker 64 | Blue jay 65 | Pretzel 66 | Bagel 67 | Tower 68 | Teapot 69 | Person 70 | Bow and arrow 71 | Swimwear 72 | Beehive 73 | Brassiere 74 | Bee 75 | Bat 76 | Starfish 77 | Popcorn 78 | Burrito 79 | Chainsaw 80 | Balloon 81 | Wrench 82 | Tent 83 | Vehicle registration plate 84 | Lantern 85 | Toaster 86 | Flashlight 87 | Billboard 88 | Tiara 89 | Limousine 90 | Necklace 91 | Carnivore 92 | Scissors 93 | Stairs 94 | Computer keyboard 95 | Printer 96 | Traffic sign 97 | Chair 98 | Shirt 99 | Poster 100 | Cheese 101 | Sock 102 | Fire hydrant 103 | Land vehicle 104 | Earrings 105 | Tie 106 | Watercraft 107 | Cabinetry 108 | Suitcase 109 | Muffin 110 | Bidet 111 | Snack 112 | Snowmobile 113 | Clock 114 | Medical equipment 115 | Cattle 116 | Cello 117 | Jet ski 118 | Camel 119 | Coat 120 | Suit 121 | Desk 122 | Cat 123 | Bronze sculpture 124 | Juice 125 | Gondola 126 | Beetle 127 | Cannon 128 | Computer mouse 129 | Cookie 130 | Office building 131 | Fountain 132 | Coin 133 | Calculator 134 | Cocktail 135 | Computer monitor 136 | Box 137 | Stapler 138 | Christmas tree 139 | Cowboy hat 140 | Hiking equipment 141 | Studio couch 142 | Drum 143 | Dessert 144 | Wine rack 145 | Drink 146 | Zucchini 147 | Ladle 148 | Human mouth 149 | Dairy 150 | Dice 151 | Oven 152 | Dinosaur 153 | Ratchet 154 | Couch 155 | Cricket ball 156 | Winter melon 157 | Spatula 158 | Whiteboard 159 | Pencil sharpener 160 | Door 161 | Hat 162 | Shower 163 | Eraser 164 | Fedora 165 | Guacamole 166 | Dagger 167 | Scarf 168 | Dolphin 169 | Sombrero 170 | Tin can 171 | Mug 172 | Tap 173 | Harbor seal 174 | Stretcher 175 | Can opener 176 | Goggles 177 | Human body 178 | Roller skates 179 | Coffee cup 180 | Cutting board 181 | Blender 182 | Plumbing fixture 183 | Stop sign 184 | Office supplies 185 | Volleyball 186 | Vase 187 | Slow cooker 188 | Wardrobe 189 | Coffee 190 | Whisk 191 | Paper towel 192 | Personal care 193 | Food 194 | Sun hat 195 | Tree house 196 | Flying disc 197 | Skirt 198 | Gas stove 199 | Salt and pepper shakers 200 | Mechanical fan 201 | Face powder 202 | Fax 203 | Fruit 204 | French fries 205 | Nightstand 206 | Barrel 207 | Kite 208 | Tart 209 | Treadmill 210 | Fox 211 | Flag 212 | Horn 213 | Window blind 214 | Human foot 215 | Golf cart 216 | Jacket 217 | Egg 218 | Street light 219 | Guitar 220 | Pillow 221 | Human leg 222 | Isopod 223 | Grape 224 | Human ear 225 | Power plugs and sockets 226 | Panda 227 | Giraffe 228 | Woman 229 | Door handle 230 | Rhinoceros 231 | Bathtub 232 | Goldfish 233 | Houseplant 234 | Goat 235 | Baseball bat 236 | Baseball glove 237 | Mixing bowl 238 | Marine invertebrates 239 | Kitchen utensil 240 | Light switch 241 | House 242 | Horse 243 | Stationary bicycle 244 | Hammer 245 | Ceiling fan 246 | Sofa bed 247 | Adhesive tape 248 | Harp 249 | Sandal 250 | Bicycle helmet 251 | Saucer 252 | Harpsichord 253 | Human hair 254 | Heater 255 | Harmonica 256 | Hamster 257 | Curtain 258 | Bed 259 | Kettle 260 | Fireplace 261 | Scale 262 | Drinking straw 263 | Insect 264 | Hair dryer 265 | Kitchenware 266 | Indoor rower 267 | Invertebrate 268 | Food processor 269 | Bookcase 270 | Refrigerator 271 | Wood-burning stove 272 | Punching bag 273 | Common fig 274 | Cocktail shaker 275 | Jaguar 276 | Golf ball 277 | Fashion accessory 278 | Alarm clock 279 | Filing cabinet 280 | Artichoke 281 | Table 282 | Tableware 283 | Kangaroo 284 | Koala 285 | Knife 286 | Bottle 287 | Bottle opener 288 | Lynx 289 | Lavender 290 | Lighthouse 291 | Dumbbell 292 | Human head 293 | Bowl 294 | Humidifier 295 | Porch 296 | Lizard 297 | Billiard table 298 | Mammal 299 | Mouse 300 | Motorcycle 301 | Musical instrument 302 | Swim cap 303 | Frying pan 304 | Snowplow 305 | Bathroom cabinet 306 | Missile 307 | Bust 308 | Man 309 | Waffle iron 310 | Milk 311 | Ring binder 312 | Plate 313 | Mobile phone 314 | Baked goods 315 | Mushroom 316 | Crutch 317 | Pitcher 318 | Mirror 319 | Lifejacket 320 | Table tennis racket 321 | Pencil case 322 | Musical keyboard 323 | Scoreboard 324 | Briefcase 325 | Kitchen knife 326 | Nail 327 | Tennis ball 328 | Plastic bag 329 | Oboe 330 | Chest of drawers 331 | Ostrich 332 | Piano 333 | Girl 334 | Plant 335 | Potato 336 | Hair spray 337 | Sports equipment 338 | Pasta 339 | Penguin 340 | Pumpkin 341 | Pear 342 | Infant bed 343 | Polar bear 344 | Mixer 345 | Cupboard 346 | Jacuzzi 347 | Pizza 348 | Digital clock 349 | Pig 350 | Reptile 351 | Rifle 352 | Lipstick 353 | Skateboard 354 | Raven 355 | High heels 356 | Red panda 357 | Rose 358 | Rabbit 359 | Sculpture 360 | Saxophone 361 | Shotgun 362 | Seafood 363 | Submarine sandwich 364 | Snowboard 365 | Sword 366 | Picture frame 367 | Sushi 368 | Loveseat 369 | Ski 370 | Squirrel 371 | Tripod 372 | Stethoscope 373 | Submarine 374 | Scorpion 375 | Segway 376 | Training bench 377 | Snake 378 | Coffee table 379 | Skyscraper 380 | Sheep 381 | Television 382 | Trombone 383 | Tea 384 | Tank 385 | Taco 386 | Telephone 387 | Torch 388 | Tiger 389 | Strawberry 390 | Trumpet 391 | Tree 392 | Tomato 393 | Train 394 | Tool 395 | Picnic basket 396 | Cooking spray 397 | Trousers 398 | Bowling equipment 399 | Football helmet 400 | Truck 401 | Measuring cup 402 | Coffeemaker 403 | Violin 404 | Vehicle 405 | Handbag 406 | Paper cutter 407 | Wine 408 | Weapon 409 | Wheel 410 | Worm 411 | Wok 412 | Whale 413 | Zebra 414 | Auto part 415 | Jug 416 | Pizza cutter 417 | Cream 418 | Monkey 419 | Lion 420 | Bread 421 | Platter 422 | Chicken 423 | Eagle 424 | Helicopter 425 | Owl 426 | Duck 427 | Turtle 428 | Hippopotamus 429 | Crocodile 430 | Toilet 431 | Toilet paper 432 | Squid 433 | Clothing 434 | Footwear 435 | Lemon 436 | Spider 437 | Deer 438 | Frog 439 | Banana 440 | Rocket 441 | Wine glass 442 | Countertop 443 | Tablet computer 444 | Waste container 445 | Swimming pool 446 | Dog 447 | Book 448 | Elephant 449 | Shark 450 | Candle 451 | Leopard 452 | Axe 453 | Hand dryer 454 | Soap dispenser 455 | Porcupine 456 | Flower 457 | Canary 458 | Cheetah 459 | Palm tree 460 | Hamburger 461 | Maple 462 | Building 463 | Fish 464 | Lobster 465 | Asparagus 466 | Furniture 467 | Hedgehog 468 | Airplane 469 | Spoon 470 | Otter 471 | Bull 472 | Oyster 473 | Horizontal bar 474 | Convenience store 475 | Bomb 476 | Bench 477 | Ice cream 478 | Caterpillar 479 | Butterfly 480 | Parachute 481 | Orange 482 | Antelope 483 | Beaker 484 | Moths and butterflies 485 | Window 486 | Closet 487 | Castle 488 | Jellyfish 489 | Goose 490 | Mule 491 | Swan 492 | Peach 493 | Coconut 494 | Seat belt 495 | Raccoon 496 | Chisel 497 | Fork 498 | Lamp 499 | Camera 500 | Squash 501 | Racket 502 | Human face 503 | Human arm 504 | Vegetable 505 | Diaper 506 | Unicycle 507 | Falcon 508 | Chime 509 | Snail 510 | Shellfish 511 | Cabbage 512 | Carrot 513 | Mango 514 | Jeans 515 | Flowerpot 516 | Pineapple 517 | Drawer 518 | Stool 519 | Envelope 520 | Cake 521 | Dragonfly 522 | Sunflower 523 | Microwave oven 524 | Honeycomb 525 | Marine mammal 526 | Sea lion 527 | Ladybug 528 | Shelf 529 | Watch 530 | Candy 531 | Salad 532 | Parrot 533 | Handgun 534 | Sparrow 535 | Van 536 | Grinder 537 | Spice rack 538 | Light bulb 539 | Corded phone 540 | Sports uniform 541 | Tennis racket 542 | Wall clock 543 | Serving tray 544 | Kitchen & dining room table 545 | Dog bed 546 | Cake stand 547 | Cat furniture 548 | Bathroom accessory 549 | Facial tissue holder 550 | Pressure cooker 551 | Kitchen appliance 552 | Tire 553 | Ruler 554 | Luggage and bags 555 | Microphone 556 | Broccoli 557 | Umbrella 558 | Pastry 559 | Grapefruit 560 | Band-aid 561 | Animal 562 | Bell pepper 563 | Turkey 564 | Lily 565 | Pomegranate 566 | Doughnut 567 | Glasses 568 | Human nose 569 | Pen 570 | Ant 571 | Car 572 | Aircraft 573 | Human hand 574 | Skunk 575 | Teddy bear 576 | Watermelon 577 | Cantaloupe 578 | Dishwasher 579 | Flute 580 | Balance beam 581 | Sandwich 582 | Shrimp 583 | Sewing machine 584 | Binoculars 585 | Rays and skates 586 | Ipod 587 | Accordion 588 | Willow 589 | Crab 590 | Crown 591 | Seahorse 592 | Perfume 593 | Alpaca 594 | Taxi 595 | Canoe 596 | Remote control 597 | Wheelchair 598 | Rugby ball 599 | Armadillo 600 | Maracas 601 | Helmet 602 | -------------------------------------------------------------------------------- /models/ssd_mobilenet_v2_oid_v4/objects.names.fr: -------------------------------------------------------------------------------- 1 | Tortue 2 | Récipient 3 | Pie 4 | Tortue de mer 5 | Football 6 | Ambulance 7 | Échelle 8 | Brosse à dents 9 | Seringue 10 | Évier 11 | Jouet 12 | Organe 13 | Platine cassette 14 | Pomme 15 | Œil humain 16 | Produits de beauté 17 | Pagayer 18 | Bonhomme de neige 19 | Bière 20 | Baguettes 21 | Barbe humaine 22 | Oiseau 23 | Parcmètre 24 | Feu de circulation 25 | Croissant 26 | Concombre 27 | Un radis 28 | Serviette 29 | Poupée 30 | Le crâne 31 | Machine à laver 32 | Gant 33 | Cocher 34 | Courroie 35 | Des lunettes de soleil 36 | Banjo 37 | Chariot 38 | Balle 39 | Sac à dos 40 | Bicyclette 41 | Électroménager 42 | Mille-Pattes 43 | Bateau 44 | Planche de surf 45 | Démarrage 46 | Écouteurs 47 | Hot-dog 48 | Shorts 49 | Fast food 50 | Bus 51 | Garçon 52 | Tournevis 53 | Roue de vélo 54 | Barge 55 | Portable 56 | Mini jupe 57 | Percer 58 | Robe 59 | Ours 60 | Gaufre 61 | Crêpe 62 | ours brun 63 | Pivert 64 | Geai bleu 65 | Bretzel 66 | Beignet 67 | La tour 68 | Théière 69 | Une personne 70 | Arc et des flèches 71 | Maillots de bain 72 | Ruche 73 | Soutien-gorge 74 | abeille 75 | Chauve souris 76 | Étoile de mer 77 | Pop corn 78 | Burrito 79 | Tronçonneuse 80 | Ballon 81 | Clé 82 | Tente 83 | Plaque d'immatriculation du véhicule 84 | Lanterne 85 | Grille-pain 86 | Lampe de poche 87 | Panneau d'affichage 88 | Tiare 89 | Limousine 90 | Collier 91 | Carnivore 92 | Les ciseaux 93 | Escaliers 94 | Clavier d'ordinateur 95 | Imprimante 96 | Panneau de signalisation 97 | Chaise 98 | Chemise 99 | Affiche 100 | Du fromage 101 | Chaussette 102 | Bouche d'incendie 103 | Véhicule terrestre 104 | Des boucles d'oreilles 105 | Attacher 106 | Motomarine 107 | Ébénisterie 108 | Valise 109 | Muffin 110 | Bidet 111 | Goûter 112 | Motoneige 113 | L'horloge 114 | Équipement médical 115 | Bétail 116 | Violoncelle 117 | Jet ski 118 | chameau 119 | Manteau 120 | Combinaison 121 | Pupitre 122 | Chat 123 | Sculpture en bronze 124 | Jus 125 | Gondole 126 | Scarabée 127 | Canon 128 | Souris d'ordinateur 129 | Biscuit 130 | Immeuble de bureaux 131 | Fontaine 132 | Pièce de monnaie 133 | Calculatrice 134 | Cocktail 135 | Moniteur d'ordinateur 136 | Boîte 137 | Agrafeuse 138 | Sapin de Noël 139 | Chapeau de cowboy 140 | Matériel de randonnée 141 | Canapé Studio 142 | Tambouriner 143 | Dessert 144 | Casier à vin 145 | Boire 146 | Courgette 147 | Louche 148 | Bouche humaine 149 | Laitier 150 | Dé 151 | Four 152 | Dinosaure 153 | Rochet 154 | Canapé 155 | Balle de cricket 156 | Melon d'hiver 157 | Spatule 158 | Tableau blanc 159 | Taille-crayon 160 | Porte 161 | Chapeau 162 | Douche 163 | La gomme 164 | Feutre 165 | Guacamole 166 | Dague 167 | Foulard 168 | Dauphin 169 | Sombrero 170 | Boîte de conserve 171 | Agresser 172 | Robinet 173 | Phoque commun 174 | Tendeur 175 | Ouvre-boîte 176 | Des lunettes de protection 177 | Corps humain 178 | Patins à roulettes 179 | Tasse à café 180 | Planche à découper 181 | Mixeur 182 | Appareil de plomberie 183 | Panneau stop 184 | Fournitures de bureau 185 | Volley-ball 186 | Vase 187 | Mijoteuse 188 | Penderie 189 | Café 190 | Fouet 191 | Essuie-tout 192 | Soins personnels 193 | Aliments 194 | chapeau de soleil 195 | Cabane dans les arbres 196 | Disque volant 197 | Jupe 198 | Cuisinière à gaz 199 | La salière et la poivrière 200 | Ventilateur mécanique 201 | Poudre pour le visage 202 | Fax 203 | Fruit 204 | frites 205 | La table de nuit 206 | Baril 207 | cerf-volant 208 | Tarte 209 | Tapis roulant 210 | Renard 211 | Drapeau 212 | Corne 213 | Store de fenêtre 214 | Pied humain 215 | Voiturette de golf 216 | Veste 217 | Œuf 218 | éclairage public 219 | Guitare 220 | Oreiller 221 | Jambe humaine 222 | Isopode 223 | Grain de raisin 224 | Oreille humaine 225 | Fiches et prises de courant 226 | Panda 227 | Girafe 228 | Femme 229 | Poignée de porte 230 | Rhinocéros 231 | Baignoire 232 | Poisson rouge 233 | Plante d'appartement 234 | Chèvre 235 | Batte de baseball 236 | Gant de baseball 237 | Bol mélangeur 238 | Invertébrés marins 239 | Ustensile de cuisine 240 | Interrupteur 241 | loger 242 | Cheval 243 | Vélo stationnaire 244 | Marteau 245 | Ventilateur de plafond 246 | Canapé-lit 247 | Ruban adhésif 248 | Harpe 249 | Sandale 250 | Casque de vélo 251 | Soucoupe 252 | Clavecin 253 | Cheveux humains 254 | Chauffe-eau 255 | Harmonica 256 | Hamster 257 | Rideau 258 | Lit 259 | Bouilloire 260 | Cheminée 261 | Escalader 262 | Paille 263 | Insecte 264 | Sèche-cheveux 265 | Ustensiles de cuisine 266 | Rameur d'intérieur 267 | Invertébré 268 | Robot culinaire 269 | Bibliothèque 270 | Réfrigérateur 271 | Poêle à bois 272 | Sac de boxe 273 | Figue commune 274 | Shaker à cocktail 275 | Jaguar 276 | Balle de golf 277 | Accessoire de mode 278 | Réveil 279 | Classeur 280 | Artichaut 281 | Table 282 | Vaisselle 283 | Kangourou 284 | Koala 285 | Couteau 286 | Bouteille 287 | Ouvre-bouteille 288 | Lynx 289 | Lavande 290 | Phare 291 | Haltère 292 | Tête humaine 293 | bol 294 | Humidificateur 295 | Porche 296 | Lézard 297 | Table de billard 298 | Mammifère 299 | Souris 300 | Moto 301 | Instrument de musique 302 | Bonnet de bain 303 | Poêle à frire 304 | Chasse-neige 305 | Armoire de toilette 306 | Missile 307 | Bousiller 308 | Homme 309 | Gaufrier 310 | Lait 311 | Reliure à anneaux 312 | Plaque 313 | Téléphone mobile 314 | Produits de boulangerie 315 | Champignon 316 | Béquille 317 | Lanceur 318 | Miroir 319 | Gilet de sauvetage 320 | Raquette de tennis de table 321 | Trousse 322 | Clavier musical 323 | Tableau de bord 324 | Mallette 325 | Couteau de cuisine 326 | Clou 327 | Balle de tennis 328 | Sac plastique 329 | Hautbois 330 | Commode 331 | Autruche 332 | Piano 333 | Fille 334 | Usine 335 | Pomme de terre 336 | Laque pour les cheveux 337 | Équipement sportif 338 | Pâtes 339 | manchot 340 | Citrouille 341 | Poire 342 | Lit bébé 343 | Ours polaire 344 | Mixer 345 | Armoire 346 | Jacuzzi 347 | Pizza 348 | Horloge digitale 349 | Cochon 350 | Reptile 351 | Fusil 352 | Rouge à lèvres 353 | planche à roulette 354 | corbeau 355 | Talons hauts 356 | Panda rouge 357 | Rose 358 | Lapin 359 | Sculpture 360 | Saxophone 361 | Fusil à pompe 362 | Fruits de mer 363 | Sandwich sous-marin 364 | Snowboard 365 | Épée 366 | Cadre de l'image 367 | Sushi 368 | Causeuse 369 | Ski 370 | Écureuil 371 | Trépied 372 | Stéthoscope 373 | Sous-marin 374 | Scorpion 375 | Segway 376 | Banc d'entraînement 377 | Serpent 378 | Table basse 379 | Gratte-ciel 380 | Mouton 381 | Télévision 382 | Trombone 383 | Thé 384 | Réservoir 385 | Taco 386 | Téléphone 387 | Torche 388 | tigre 389 | fraise 390 | Trompette 391 | Arbre 392 | Tomate 393 | Former 394 | Outil 395 | Panier pique-nique 396 | Aérosol de cuisson 397 | Pantalon 398 | Matériel de bowling 399 | Casque de football américain 400 | Camion 401 | Tasse à mesurer 402 | Machine à café 403 | Violon 404 | Véhicule 405 | Sac à main 406 | Coupe-papier 407 | Vin 408 | Arme 409 | Roue 410 | Ver 411 | Wok 412 | Baleine 413 | Zèbre 414 | Pièce auto 415 | Cruche 416 | couteau à pizza 417 | Crème 418 | Singe 419 | Lion 420 | Pain 421 | Plat 422 | Poulet 423 | Aigle 424 | Hélicoptère 425 | Chouette 426 | Canard 427 | Tortue 428 | Hippopotame 429 | Crocodile 430 | Toilette 431 | Papier toilette 432 | Calmar 433 | Vêtements 434 | Chaussure 435 | Citron 436 | Araignée 437 | Cerf 438 | La grenouille 439 | Banane 440 | Fusée 441 | Verre de vin 442 | Comptoir 443 | Tablette 444 | Conteneur à déchets 445 | Piscine 446 | Chien 447 | Livre 448 | l'éléphant 449 | Requin 450 | Bougie 451 | Léopard 452 | Hache 453 | Sèche mains 454 | Distributeur de savon 455 | Porc-épic 456 | Fleur 457 | Canari 458 | guépard 459 | palmier 460 | Hamburger 461 | Érable 462 | Bâtiment 463 | Poisson 464 | Homard 465 | Asperges 466 | Meubles 467 | Hérisson 468 | Avion 469 | Cuillère 470 | loutre 471 | Taureau 472 | huître 473 | Barre horizontale 474 | Épicerie 475 | Bombe 476 | Banc 477 | La crème glacée 478 | chenille 479 | Papillon 480 | Parachute 481 | Orange 482 | Antilope 483 | Gobelet 484 | Papillons et papillons 485 | La fenêtre 486 | Toilettes 487 | château 488 | Méduse 489 | OIE 490 | Mule 491 | cygne 492 | Pêche 493 | Noix de coco 494 | Ceinture de sécurité 495 | Raton laveur 496 | Ciseau 497 | Fourchette 498 | Lampe 499 | Caméra 500 | Écraser 501 | Raquette 502 | Visage humain 503 | Bras humain 504 | Légume 505 | Couche 506 | Monocycle 507 | Faucon 508 | Carillon 509 | Escargot 510 | Fruits de mer 511 | Choux 512 | Carotte 513 | Mangue 514 | jeans 515 | Pot de fleur 516 | L'ananas 517 | Tiroir 518 | Tabouret 519 | Enveloppe 520 | Gâteau 521 | Libellule 522 | Tournesol 523 | Four micro-onde 524 | Rayon de miel 525 | Mammifère marin 526 | Lion de mer 527 | Coccinelle 528 | Étagère 529 | Regardez 530 | Bonbons 531 | salade 532 | Perroquet 533 | Pistolet 534 | moineau 535 | Van 536 | Broyeur 537 | étagère à épices 538 | Ampoule 539 | Téléphone filaire 540 | Uniforme de sport 541 | Raquette de tennis 542 | horloge murale 543 | Plateau 544 | Table de cuisine et salle à manger 545 | Lit pour chien 546 | Stand de gâteaux 547 | Meubles pour chats 548 | Accessoire de salle de bain 549 | Porte-mouchoirs en papier 550 | Cocotte minute 551 | Appareil de cuisine 552 | Pneu 553 | Règle 554 | Bagages et sacs 555 | Microphone 556 | Brocoli 557 | Parapluie 558 | Pâtisserie 559 | Pamplemousse 560 | Pansement 561 | Animal 562 | poivron 563 | dinde 564 | Lis 565 | Grenade 566 | Donut 567 | Lunettes 568 | Nez humain 569 | Stylo 570 | Fourmi 571 | Auto 572 | Avion 573 | Main humaine 574 | Moufette 575 | ours en peluche 576 | Pastèque 577 | Cantaloup 578 | Lave-vaisselle 579 | Flûte 580 | Poutre d'équilibre 581 | Sandwich 582 | Crevette 583 | Machine à coudre 584 | Jumelles 585 | Rayons et patins 586 | Ipod 587 | Accordéon 588 | saule 589 | Crabe 590 | couronner 591 | hippocampe 592 | Parfum 593 | Alpaga 594 | Taxi 595 | Canoë 596 | Télécommande 597 | Fauteuil roulant 598 | Ballon de rugby 599 | Tatou 600 | Maracas 601 | Casque 602 | -------------------------------------------------------------------------------- /models/yolo_v4_tiny_custom/.empty: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/softbankrobotics-labs/nao-deep-learning/7bb0226041b4636207ae6b2346b581c8a147b0a6/models/yolo_v4_tiny_custom/.empty -------------------------------------------------------------------------------- /notebook/NAO_Deep_Learning_Custom_Model_Training.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "NAO Deep Learning Custom Model Training", 7 | "provenance": [], 8 | "collapsed_sections": [ 9 | "JboccXyAZCNw" 10 | ] 11 | }, 12 | "kernelspec": { 13 | "name": "python3", 14 | "display_name": "Python 3" 15 | } 16 | }, 17 | "cells": [ 18 | { 19 | "cell_type": "markdown", 20 | "metadata": { 21 | "id": "vjV8M1kdmtgg" 22 | }, 23 | "source": [ 24 | "# NAO Deep Learning Custom model training (based on Tiny YOLOv4)\n", 25 | "\n", 26 | "Welcome to the Deep NAO custom model training tutorial. In this notebook we demonstrate how to train your custom object detection deep learning model to use it in the Deep NAO application.\n", 27 | "\n", 28 | "The model you will train uses the Tiny YOLOv4 architecture. This Colab takes around 1h30 to complete. " 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": { 34 | "id": "JboccXyAZCNw" 35 | }, 36 | "source": [ 37 | "## Choose GPU for your Colab instance \n", 38 | "\n", 39 | "To execute this notebook, we recommend that you Click on `Edit` > `Notebook settings` in the toolbar of this page, and select `GPU` in the field `Hardware`" 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": { 45 | "id": "_ewvAUg_bG9g" 46 | }, 47 | "source": [ 48 | "## Retrieve and compile the Darknet framework\n", 49 | "\n", 50 | "[Darknet](https://pjreddie.com/darknet/) is the Deep Learning framework that you will use to train your custom model. Your custom model will use a Tiny YOLOv4 architecture.\n", 51 | "\n", 52 | "Download darknet and uncompress it." 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "metadata": { 58 | "id": "T7r0RMRIZ5ej" 59 | }, 60 | "source": [ 61 | "# Download Darknet source code from the Github repository\n", 62 | "!wget https://github.com/AlexeyAB/darknet/archive/darknet_yolo_v4_pre.zip\n", 63 | "# Extract the archive\n", 64 | "!unzip darknet_yolo_v4_pre.zip\n", 65 | "# Rename the folder as 'darknet'\n", 66 | "%mv darknet-darknet_yolo_v4_pre darknet\n" 67 | ], 68 | "execution_count": null, 69 | "outputs": [] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": { 74 | "id": "W_YH5Vclcm7X" 75 | }, 76 | "source": [ 77 | "Modify the Makefile to enable GPU (this will run faster) and OpenCV. Then compile:" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "metadata": { 83 | "id": "CwftKNapbRxv" 84 | }, 85 | "source": [ 86 | "# Go to 'darknet' folder\n", 87 | "%cd darknet\n", 88 | "# Edit the Makefile to enable OpenCV, GPU, CUDNN and CUDNN_HALF\n", 89 | "!sed -i 's/OPENCV=0/OPENCV=1/' Makefile\n", 90 | "!sed -i 's/GPU=0/GPU=1/' Makefile\n", 91 | "!sed -i 's/CUDNN=0/CUDNN=1/' Makefile\n", 92 | "!sed -i 's/CUDNN_HALF=0/CUDNN_HALF=1/' Makefile\n", 93 | "!sed -i 's/LIBSO=0/LIBSO=1/' Makefile\n", 94 | "# Compile the Darknet framework code\n", 95 | "!make -j2" 96 | ], 97 | "execution_count": null, 98 | "outputs": [] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": { 103 | "id": "BI2BeSEIbYHu" 104 | }, 105 | "source": [ 106 | "Now that the Darknet framework is installed, let's get some training dataset." 107 | ] 108 | }, 109 | { 110 | "cell_type": "markdown", 111 | "metadata": { 112 | "id": "aRLGiiZvukRg" 113 | }, 114 | "source": [ 115 | "## Prepare your training data\n", 116 | "\n", 117 | "### Google Open Image Dataset\n", 118 | "\n", 119 | "To train our deep learning model, we need a dataset of images containing the objects we want to detect. The images must be labeled. Which means for each image, if it contains the object to be detected, we need to provide to the training algorithm the coordinates of the object in the image. \n", 120 | "\n", 121 | "Fortunately for us, there exists datasets of already labelled images out there on the internet. Google provides such a dataset: the [Open Image Dataset](https://storage.googleapis.com/openimages/web/index.html), that contains thousand of labelled images for 600 classes of objects.\n", 122 | "\n", 123 | "You can explore the dataset using the [Visualizer](https://storage.googleapis.com/openimages/web/visualizer/index.html?set=train&type=detection&c=%2Fm%2F01lsmm) they provide.\n", 124 | "\n", 125 | "\n", 126 | "### Download labeled object images from Google Open Image Dataset\n", 127 | "\n", 128 | "In this tutorial we train our model to recognize 2 different objects out of the 600 objects available in the Google Image dataset. Choose below which objects you want nao to learn to recognize, and launch the download of labeled images of these objects." 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "metadata": { 134 | "id": "ZSmcdrMo2oeY" 135 | }, 136 | "source": [ 137 | "#@title Choose the objects NAO will learn to recognize\n", 138 | "\n", 139 | "first_object = 'Glasses' #@param [\"Accordion\", \"Adhesive tape\", \"Aircraft\", \"Airplane\", \"Alarm clock\", \"Alpaca\", \"Ambulance\", \"Animal\", \"Ant\", \"Antelope\", \"Apple\", \"Armadillo\", \"Artichoke\", \"Asparagus\", \"Auto part\", \"Axe\", \"Backpack\", \"Bagel\", \"Baked goods\", \"Balance beam\", \"Ball\", \"Balloon\", \"Banana\", \"Band-aid\", \"Banjo\", \"Barge\", \"Barrel\", \"Baseball bat\", \"Baseball glove\", \"Bat\", \"Bathroom accessory\", \"Bathroom cabinet\", \"Bathtub\", \"Beaker\", \"Bear\", \"Bed\", \"Bee\", \"Beehive\", \"Beer\", \"Beetle\", \"Bell pepper\", \"Belt\", \"Bench\", \"Bicycle\", \"Bicycle helmet\", \"Bicycle wheel\", \"Bidet\", \"Billboard\", \"Billiard table\", \"Binoculars\", \"Bird\", \"Blender\", \"Blue jay\", \"Boat\", \"Bomb\", \"Book\", \"Bookcase\", \"Boot\", \"Bottle\", \"Bottle opener\", \"Bow and arrow\", \"Bowl\", \"Bowling equipment\", \"Box\", \"Boy\", \"Brassiere\", \"Bread\", \"Briefcase\", \"Broccoli\", \"Bronze sculpture\", \"Brown bear\", \"Building\", \"Bull\", \"Burrito\", \"Bus\", \"Bust\", \"Butterfly\", \"Cabbage\", \"Cabinetry\", \"Cake\", \"Cake stand\", \"Calculator\", \"Camel\", \"Camera\", \"Canary\", \"Candle\", \"Candy\", \"Cannon\", \"Canoe\", \"Can opener\", \"Cantaloupe\", \"Car\", \"Carnivore\", \"Carrot\", \"Cart\", \"Cassette deck\", \"Castle\", \"Cat\", \"Caterpillar\", \"Cat furniture\", \"Cattle\", \"Ceiling fan\", \"Cello\", \"Centipede\", \"Chainsaw\", \"Chair\", \"Cheese\", \"Cheetah\", \"Chest of drawers\", \"Chicken\", \"Chime\", \"Chisel\", \"Chopsticks\", \"Christmas tree\", \"Clock\", \"Closet\", \"Clothing\", \"Coat\", \"Cocktail\", \"Cocktail shaker\", \"Coconut\", \"Coffee\", \"Coffee cup\", \"Coffeemaker\", \"Coffee table\", \"Coin\", \"Common fig\", \"Computer keyboard\", \"Computer monitor\", \"Computer mouse\", \"Container\", \"Convenience store\", \"Cookie\", \"Cooking spray\", \"Corded phone\", \"Cosmetics\", \"Couch\", \"Countertop\", \"Cowboy hat\", \"Crab\", \"Cream\", \"Cricket ball\", \"Crocodile\", \"Croissant\", \"Crown\", \"Crutch\", \"Cucumber\", \"Cupboard\", \"Curtain\", \"Cutting board\", \"Dagger\", \"Dairy\", \"Deer\", \"Desk\", \"Dessert\", \"Diaper\", \"Dice\", \"Digital clock\", \"Dinosaur\", \"Dishwasher\", \"Dog\", \"Dog bed\", \"Doll\", \"Dolphin\", \"Door\", \"Door handle\", \"Doughnut\", \"Dragonfly\", \"Drawer\", \"Dress\", \"Drill\", \"Drink\", \"Drinking straw\", \"Drum\", \"Duck\", \"Dumbbell\", \"Eagle\", \"Earrings\", \"Egg\", \"Elephant\", \"Envelope\", \"Eraser\", \"Face powder\", \"Facial tissue holder\", \"Falcon\", \"Fashion accessory\", \"Fast food\", \"Fax\", \"Fedora\", \"Filing cabinet\", \"Fire hydrant\", \"Fireplace\", \"Fish\", \"Flag\", \"Flashlight\", \"Flower\", \"Flowerpot\", \"Flute\", \"Flying disc\", \"Food\", \"Food processor\", \"Football\", \"Football helmet\", \"Footwear\", \"Fork\", \"Fountain\", \"Fox\", \"French fries\", \"Frog\", \"Fruit\", \"Frying pan\", \"Furniture\", \"Gas stove\", \"Giraffe\", \"Girl\", \"Glasses\", \"Glove\", \"Goat\", \"Goggles\", \"Goldfish\", \"Golf ball\", \"Golf cart\", \"Gondola\", \"Goose\", \"Grape\", \"Grapefruit\", \"Grinder\", \"Guacamole\", \"Guitar\", \"Hair dryer\", \"Hair spray\", \"Hamburger\", \"Hammer\", \"Hamster\", \"Handbag\", \"Hand dryer\", \"Handgun\", \"Harbor seal\", \"Harmonica\", \"Harp\", \"Harpsichord\", \"Hat\", \"Headphones\", \"Heater\", \"Hedgehog\", \"Helicopter\", \"Helmet\", \"High heels\", \"Hiking equipment\", \"Hippopotamus\", \"Home appliance\", \"Honeycomb\", \"Horizontal bar\", \"Horn\", \"Horse\", \"Hot dog\", \"House\", \"Houseplant\", \"Human arm\", \"Human beard\", \"Human body\", \"Human ear\", \"Human eye\", \"Human face\", \"Human foot\", \"Human hair\", \"Human hand\", \"Human head\", \"Human leg\", \"Human mouth\", \"Human nose\", \"Humidifier\", \"Ice cream\", \"Indoor rower\", \"Infant bed\", \"Insect\", \"Invertebrate\", \"Ipod\", \"Isopod\", \"Jacket\", \"Jacuzzi\", \"Jaguar\", \"Jeans\", \"Jellyfish\", \"Jet ski\", \"Jug\", \"Juice\", \"Kangaroo\", \"Kettle\", \"Kitchen appliance\", \"Kitchen & dining room table\", \"Kitchen knife\", \"Kitchen utensil\", \"Kitchenware\", \"Kite\", \"Knife\", \"Koala\", \"Ladder\", \"Ladle\", \"Ladybug\", \"Lamp\", \"Land vehicle\", \"Lantern\", \"Laptop\", \"Lavender\", \"Lemon\", \"Leopard\", \"Lifejacket\", \"Light bulb\", \"Lighthouse\", \"Light switch\", \"Lily\", \"Limousine\", \"Lion\", \"Lipstick\", \"Lizard\", \"Lobster\", \"Loveseat\", \"Luggage and bags\", \"Lynx\", \"Magpie\", \"Mammal\", \"Man\", \"Mango\", \"Maple\", \"Maracas\", \"Marine invertebrates\", \"Marine mammal\", \"Measuring cup\", \"Mechanical fan\", \"Medical equipment\", \"Microphone\", \"Microwave oven\", \"Milk\", \"Miniskirt\", \"Mirror\", \"Missile\", \"Mixer\", \"Mixing bowl\", \"Mobile phone\", \"Monkey\", \"Moths and butterflies\", \"Motorcycle\", \"Mouse\", \"Muffin\", \"Mug\", \"Mule\", \"Mushroom\", \"Musical instrument\", \"Musical keyboard\", \"Nail\", \"Necklace\", \"Nightstand\", \"Oboe\", \"Office building\", \"Office supplies\", \"Orange\", \"Organ\", \"Ostrich\", \"Otter\", \"Oven\", \"Owl\", \"Oyster\", \"Paddle\", \"Palm tree\", \"Pancake\", \"Panda\", \"Paper cutter\", \"Paper towel\", \"Parachute\", \"Parking meter\", \"Parrot\", \"Pasta\", \"Pastry\", \"Peach\", \"Pear\", \"Pen\", \"Pencil case\", \"Pencil sharpener\", \"Penguin\", \"Perfume\", \"Person\", \"Personal care\", \"Piano\", \"Picnic basket\", \"Picture frame\", \"Pig\", \"Pillow\", \"Pineapple\", \"Pitcher\", \"Pizza\", \"Pizza cutter\", \"Plant\", \"Plastic bag\", \"Plate\", \"Platter\", \"Plumbing fixture\", \"Polar bear\", \"Pomegranate\", \"Popcorn\", \"Porch\", \"Porcupine\", \"Poster\", \"Potato\", \"Power plugs and sockets\", \"Pressure cooker\", \"Pretzel\", \"Printer\", \"Pumpkin\", \"Punching bag\", \"Rabbit\", \"Raccoon\", \"Racket\", \"Radish\", \"Ratchet\", \"Raven\", \"Rays and skates\", \"Red panda\", \"Refrigerator\", \"Remote control\", \"Reptile\", \"Rhinoceros\", \"Rifle\", \"Ring binder\", \"Rocket\", \"Roller skates\", \"Rose\", \"Rugby ball\", \"Ruler\", \"Salad\", \"Salt and pepper shakers\", \"Sandal\", \"Sandwich\", \"Saucer\", \"Saxophone\", \"Scale\", \"Scarf\", \"Scissors\", \"Scoreboard\", \"Scorpion\", \"Screwdriver\", \"Sculpture\", \"Seafood\", \"Seahorse\", \"Sea lion\", \"Seat belt\", \"Sea turtle\", \"Segway\", \"Serving tray\", \"Sewing machine\", \"Shark\", \"Sheep\", \"Shelf\", \"Shellfish\", \"Shirt\", \"Shorts\", \"Shotgun\", \"Shower\", \"Shrimp\", \"Sink\", \"Skateboard\", \"Ski\", \"Skirt\", \"Skull\", \"Skunk\", \"Skyscraper\", \"Slow cooker\", \"Snack\", \"Snail\", \"Snake\", \"Snowboard\", \"Snowman\", \"Snowmobile\", \"Snowplow\", \"Soap dispenser\", \"Sock\", \"Sofa bed\", \"Sombrero\", \"Sparrow\", \"Spatula\", \"Spice rack\", \"Spider\", \"Spoon\", \"Sports equipment\", \"Sports uniform\", \"Squash\", \"Squid\", \"Squirrel\", \"Stairs\", \"Stapler\", \"Starfish\", \"Stationary bicycle\", \"Stethoscope\", \"Stool\", \"Stop sign\", \"Strawberry\", \"Street light\", \"Stretcher\", \"Studio couch\", \"Submarine\", \"Submarine sandwich\", \"Suit\", \"Suitcase\", \"Sunflower\", \"Sunglasses\", \"Sun hat\", \"Surfboard\", \"Sushi\", \"Swan\", \"Swim cap\", \"Swimming pool\", \"Swimwear\", \"Sword\", \"Syringe\", \"Table\", \"Tablet computer\", \"Table tennis racket\", \"Tableware\", \"Taco\", \"Tank\", \"Tap\", \"Tart\", \"Taxi\", \"Tea\", \"Teapot\", \"Teddy bear\", \"Telephone\", \"Television\", \"Tennis ball\", \"Tennis racket\", \"Tent\", \"Tiara\", \"Tick\", \"Tie\", \"Tiger\", \"Tin can\", \"Tire\", \"Toaster\", \"Toilet\", \"Toilet paper\", \"Tomato\", \"Tool\", \"Toothbrush\", \"Torch\", \"Tortoise\", \"Towel\", \"Tower\", \"Toy\", \"Traffic light\", \"Traffic sign\", \"Train\", \"Training bench\", \"Treadmill\", \"Tree\", \"Tree house\", \"Tripod\", \"Trombone\", \"Trousers\", \"Truck\", \"Trumpet\", \"Turkey\", \"Turtle\", \"Umbrella\", \"Unicycle\", \"Van\", \"Vase\", \"Vegetable\", \"Vehicle\", \"Vehicle registration plate\", \"Violin\", \"Volleyball\", \"Waffle\", \"Waffle iron\", \"Wall clock\", \"Wardrobe\", \"Washing machine\", \"Waste container\", \"Watch\", \"Watercraft\", \"Watermelon\", \"Weapon\", \"Whale\", \"Wheel\", \"Wheelchair\", \"Whisk\", \"Whiteboard\", \"Willow\", \"Window\", \"Window blind\", \"Wine\", \"Wine glass\", \"Wine rack\", \"Winter melon\", \"Wok\", \"Woman\", \"Wood-burning stove\", \"Woodpecker\", \"Worm\", \"Wrench\", \"Zebra\", \"Zucchini\"]\n", 140 | "second_object = 'Mobile phone' #@param [\"Accordion\", \"Adhesive tape\", \"Aircraft\", \"Airplane\", \"Alarm clock\", \"Alpaca\", \"Ambulance\", \"Animal\", \"Ant\", \"Antelope\", \"Apple\", \"Armadillo\", \"Artichoke\", \"Asparagus\", \"Auto part\", \"Axe\", \"Backpack\", \"Bagel\", \"Baked goods\", \"Balance beam\", \"Ball\", \"Balloon\", \"Banana\", \"Band-aid\", \"Banjo\", \"Barge\", \"Barrel\", \"Baseball bat\", \"Baseball glove\", \"Bat\", \"Bathroom accessory\", \"Bathroom cabinet\", \"Bathtub\", \"Beaker\", \"Bear\", \"Bed\", \"Bee\", \"Beehive\", \"Beer\", \"Beetle\", \"Bell pepper\", \"Belt\", \"Bench\", \"Bicycle\", \"Bicycle helmet\", \"Bicycle wheel\", \"Bidet\", \"Billboard\", \"Billiard table\", \"Binoculars\", \"Bird\", \"Blender\", \"Blue jay\", \"Boat\", \"Bomb\", \"Book\", \"Bookcase\", \"Boot\", \"Bottle\", \"Bottle opener\", \"Bow and arrow\", \"Bowl\", \"Bowling equipment\", \"Box\", \"Boy\", \"Brassiere\", \"Bread\", \"Briefcase\", \"Broccoli\", \"Bronze sculpture\", \"Brown bear\", \"Building\", \"Bull\", \"Burrito\", \"Bus\", \"Bust\", \"Butterfly\", \"Cabbage\", \"Cabinetry\", \"Cake\", \"Cake stand\", \"Calculator\", \"Camel\", \"Camera\", \"Canary\", \"Candle\", \"Candy\", \"Cannon\", \"Canoe\", \"Can opener\", \"Cantaloupe\", \"Car\", \"Carnivore\", \"Carrot\", \"Cart\", \"Cassette deck\", \"Castle\", \"Cat\", \"Caterpillar\", \"Cat furniture\", \"Cattle\", \"Ceiling fan\", \"Cello\", \"Centipede\", \"Chainsaw\", \"Chair\", \"Cheese\", \"Cheetah\", \"Chest of drawers\", \"Chicken\", \"Chime\", \"Chisel\", \"Chopsticks\", \"Christmas tree\", \"Clock\", \"Closet\", \"Clothing\", \"Coat\", \"Cocktail\", \"Cocktail shaker\", \"Coconut\", \"Coffee\", \"Coffee cup\", \"Coffeemaker\", \"Coffee table\", \"Coin\", \"Common fig\", \"Computer keyboard\", \"Computer monitor\", \"Computer mouse\", \"Container\", \"Convenience store\", \"Cookie\", \"Cooking spray\", \"Corded phone\", \"Cosmetics\", \"Couch\", \"Countertop\", \"Cowboy hat\", \"Crab\", \"Cream\", \"Cricket ball\", \"Crocodile\", \"Croissant\", \"Crown\", \"Crutch\", \"Cucumber\", \"Cupboard\", \"Curtain\", \"Cutting board\", \"Dagger\", \"Dairy\", \"Deer\", \"Desk\", \"Dessert\", \"Diaper\", \"Dice\", \"Digital clock\", \"Dinosaur\", \"Dishwasher\", \"Dog\", \"Dog bed\", \"Doll\", \"Dolphin\", \"Door\", \"Door handle\", \"Doughnut\", \"Dragonfly\", \"Drawer\", \"Dress\", \"Drill\", \"Drink\", \"Drinking straw\", \"Drum\", \"Duck\", \"Dumbbell\", \"Eagle\", \"Earrings\", \"Egg\", \"Elephant\", \"Envelope\", \"Eraser\", \"Face powder\", \"Facial tissue holder\", \"Falcon\", \"Fashion accessory\", \"Fast food\", \"Fax\", \"Fedora\", \"Filing cabinet\", \"Fire hydrant\", \"Fireplace\", \"Fish\", \"Flag\", \"Flashlight\", \"Flower\", \"Flowerpot\", \"Flute\", \"Flying disc\", \"Food\", \"Food processor\", \"Football\", \"Football helmet\", \"Footwear\", \"Fork\", \"Fountain\", \"Fox\", \"French fries\", \"Frog\", \"Fruit\", \"Frying pan\", \"Furniture\", \"Gas stove\", \"Giraffe\", \"Girl\", \"Glasses\", \"Glove\", \"Goat\", \"Goggles\", \"Goldfish\", \"Golf ball\", \"Golf cart\", \"Gondola\", \"Goose\", \"Grape\", \"Grapefruit\", \"Grinder\", \"Guacamole\", \"Guitar\", \"Hair dryer\", \"Hair spray\", \"Hamburger\", \"Hammer\", \"Hamster\", \"Handbag\", \"Hand dryer\", \"Handgun\", \"Harbor seal\", \"Harmonica\", \"Harp\", \"Harpsichord\", \"Hat\", \"Headphones\", \"Heater\", \"Hedgehog\", \"Helicopter\", \"Helmet\", \"High heels\", \"Hiking equipment\", \"Hippopotamus\", \"Home appliance\", \"Honeycomb\", \"Horizontal bar\", \"Horn\", \"Horse\", \"Hot dog\", \"House\", \"Houseplant\", \"Human arm\", \"Human beard\", \"Human body\", \"Human ear\", \"Human eye\", \"Human face\", \"Human foot\", \"Human hair\", \"Human hand\", \"Human head\", \"Human leg\", \"Human mouth\", \"Human nose\", \"Humidifier\", \"Ice cream\", \"Indoor rower\", \"Infant bed\", \"Insect\", \"Invertebrate\", \"Ipod\", \"Isopod\", \"Jacket\", \"Jacuzzi\", \"Jaguar\", \"Jeans\", \"Jellyfish\", \"Jet ski\", \"Jug\", \"Juice\", \"Kangaroo\", \"Kettle\", \"Kitchen appliance\", \"Kitchen & dining room table\", \"Kitchen knife\", \"Kitchen utensil\", \"Kitchenware\", \"Kite\", \"Knife\", \"Koala\", \"Ladder\", \"Ladle\", \"Ladybug\", \"Lamp\", \"Land vehicle\", \"Lantern\", \"Laptop\", \"Lavender\", \"Lemon\", \"Leopard\", \"Lifejacket\", \"Light bulb\", \"Lighthouse\", \"Light switch\", \"Lily\", \"Limousine\", \"Lion\", \"Lipstick\", \"Lizard\", \"Lobster\", \"Loveseat\", \"Luggage and bags\", \"Lynx\", \"Magpie\", \"Mammal\", \"Man\", \"Mango\", \"Maple\", \"Maracas\", \"Marine invertebrates\", \"Marine mammal\", \"Measuring cup\", \"Mechanical fan\", \"Medical equipment\", \"Microphone\", \"Microwave oven\", \"Milk\", \"Miniskirt\", \"Mirror\", \"Missile\", \"Mixer\", \"Mixing bowl\", \"Mobile phone\", \"Monkey\", \"Moths and butterflies\", \"Motorcycle\", \"Mouse\", \"Muffin\", \"Mug\", \"Mule\", \"Mushroom\", \"Musical instrument\", \"Musical keyboard\", \"Nail\", \"Necklace\", \"Nightstand\", \"Oboe\", \"Office building\", \"Office supplies\", \"Orange\", \"Organ\", \"Ostrich\", \"Otter\", \"Oven\", \"Owl\", \"Oyster\", \"Paddle\", \"Palm tree\", \"Pancake\", \"Panda\", \"Paper cutter\", \"Paper towel\", \"Parachute\", \"Parking meter\", \"Parrot\", \"Pasta\", \"Pastry\", \"Peach\", \"Pear\", \"Pen\", \"Pencil case\", \"Pencil sharpener\", \"Penguin\", \"Perfume\", \"Person\", \"Personal care\", \"Piano\", \"Picnic basket\", \"Picture frame\", \"Pig\", \"Pillow\", \"Pineapple\", \"Pitcher\", \"Pizza\", \"Pizza cutter\", \"Plant\", \"Plastic bag\", \"Plate\", \"Platter\", \"Plumbing fixture\", \"Polar bear\", \"Pomegranate\", \"Popcorn\", \"Porch\", \"Porcupine\", \"Poster\", \"Potato\", \"Power plugs and sockets\", \"Pressure cooker\", \"Pretzel\", \"Printer\", \"Pumpkin\", \"Punching bag\", \"Rabbit\", \"Raccoon\", \"Racket\", \"Radish\", \"Ratchet\", \"Raven\", \"Rays and skates\", \"Red panda\", \"Refrigerator\", \"Remote control\", \"Reptile\", \"Rhinoceros\", \"Rifle\", \"Ring binder\", \"Rocket\", \"Roller skates\", \"Rose\", \"Rugby ball\", \"Ruler\", \"Salad\", \"Salt and pepper shakers\", \"Sandal\", \"Sandwich\", \"Saucer\", \"Saxophone\", \"Scale\", \"Scarf\", \"Scissors\", \"Scoreboard\", \"Scorpion\", \"Screwdriver\", \"Sculpture\", \"Seafood\", \"Seahorse\", \"Sea lion\", \"Seat belt\", \"Sea turtle\", \"Segway\", \"Serving tray\", \"Sewing machine\", \"Shark\", \"Sheep\", \"Shelf\", \"Shellfish\", \"Shirt\", \"Shorts\", \"Shotgun\", \"Shower\", \"Shrimp\", \"Sink\", \"Skateboard\", \"Ski\", \"Skirt\", \"Skull\", \"Skunk\", \"Skyscraper\", \"Slow cooker\", \"Snack\", \"Snail\", \"Snake\", \"Snowboard\", \"Snowman\", \"Snowmobile\", \"Snowplow\", \"Soap dispenser\", \"Sock\", \"Sofa bed\", \"Sombrero\", \"Sparrow\", \"Spatula\", \"Spice rack\", \"Spider\", \"Spoon\", \"Sports equipment\", \"Sports uniform\", \"Squash\", \"Squid\", \"Squirrel\", \"Stairs\", \"Stapler\", \"Starfish\", \"Stationary bicycle\", \"Stethoscope\", \"Stool\", \"Stop sign\", \"Strawberry\", \"Street light\", \"Stretcher\", \"Studio couch\", \"Submarine\", \"Submarine sandwich\", \"Suit\", \"Suitcase\", \"Sunflower\", \"Sunglasses\", \"Sun hat\", \"Surfboard\", \"Sushi\", \"Swan\", \"Swim cap\", \"Swimming pool\", \"Swimwear\", \"Sword\", \"Syringe\", \"Table\", \"Tablet computer\", \"Table tennis racket\", \"Tableware\", \"Taco\", \"Tank\", \"Tap\", \"Tart\", \"Taxi\", \"Tea\", \"Teapot\", \"Teddy bear\", \"Telephone\", \"Television\", \"Tennis ball\", \"Tennis racket\", \"Tent\", \"Tiara\", \"Tick\", \"Tie\", \"Tiger\", \"Tin can\", \"Tire\", \"Toaster\", \"Toilet\", \"Toilet paper\", \"Tomato\", \"Tool\", \"Toothbrush\", \"Torch\", \"Tortoise\", \"Towel\", \"Tower\", \"Toy\", \"Traffic light\", \"Traffic sign\", \"Train\", \"Training bench\", \"Treadmill\", \"Tree\", \"Tree house\", \"Tripod\", \"Trombone\", \"Trousers\", \"Truck\", \"Trumpet\", \"Turkey\", \"Turtle\", \"Umbrella\", \"Unicycle\", \"Van\", \"Vase\", \"Vegetable\", \"Vehicle\", \"Vehicle registration plate\", \"Violin\", \"Volleyball\", \"Waffle\", \"Waffle iron\", \"Wall clock\", \"Wardrobe\", \"Washing machine\", \"Waste container\", \"Watch\", \"Watercraft\", \"Watermelon\", \"Weapon\", \"Whale\", \"Wheel\", \"Wheelchair\", \"Whisk\", \"Whiteboard\", \"Willow\", \"Window\", \"Window blind\", \"Wine\", \"Wine glass\", \"Wine rack\", \"Winter melon\", \"Wok\", \"Woman\", \"Wood-burning stove\", \"Woodpecker\", \"Worm\", \"Wrench\", \"Zebra\", \"Zucchini\"]\n", 141 | "\n", 142 | "# Download the OIDv4 toolkit from the Github repository. The OIDv4 toolkit \n", 143 | "# contains scripts to automate the download of images from the Google Open Image \n", 144 | "# Dataset\n", 145 | "!git clone https://github.com/theAIGuysCode/OIDv4_ToolKit.git\n", 146 | "# Go to the toolkit folder\n", 147 | "%cd OIDv4_ToolKit\n", 148 | "\n", 149 | "# Install python requirements\n", 150 | "!pip install -r requirements.txt \n", 151 | "\n", 152 | "#Choose how many image of each object to use for the training (400 should be enough)\n", 153 | "training_image_number = 400 #@param {type:\"integer\"}\n", 154 | "\n", 155 | "# Download images of the objects you want to recognize from the Google Open Image Dataset\n", 156 | "!python main.py downloader --classes \"$first_object\" \"$second_object\" --type_csv train --limit $training_image_number --yes\n", 157 | "\n", 158 | "# Download 40 images for validation\n", 159 | "!python main.py downloader --classes \"$first_object\" \"$second_object\" --type_csv validation --limit 40 --yes\n", 160 | "\n", 161 | "# Convert the images annotation files to YoloV4 format\n", 162 | "!echo $first_object > classes.txt\n", 163 | "!echo $second_object >> classes.txt \n", 164 | "!cat classes.txt\n", 165 | "!python convert_annotations.py\n" 166 | ], 167 | "execution_count": null, 168 | "outputs": [] 169 | }, 170 | { 171 | "cell_type": "markdown", 172 | "metadata": { 173 | "id": "WZxWgULJ6DFL" 174 | }, 175 | "source": [ 176 | "Prepare the dataset for the training" 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "metadata": { 182 | "id": "evFr0BFx8IlV" 183 | }, 184 | "source": [ 185 | "%%shell\n", 186 | "# Create a data folder, containing an obj folder\n", 187 | "mkdir -p data/obj\n", 188 | "# Create the 'obj.names' file, that contains the names of the objects to recognize\n", 189 | "cp classes.txt data/obj.names\n", 190 | "# Create the 'obj.data' configuration file, used by Darknet\n", 191 | "cat > data/obj.data < data/train.txt\n", 202 | "# Remove Darknet data folder \n", 203 | "rm -rf ../data\n", 204 | "# And replace the Darknet data folder by ours\n", 205 | "mv data .." 206 | ], 207 | "execution_count": null, 208 | "outputs": [] 209 | }, 210 | { 211 | "cell_type": "markdown", 212 | "metadata": { 213 | "id": "2I5GIc39xmcX" 214 | }, 215 | "source": [ 216 | "Resize all images to 416x416, the input size for the Tiny YOLOv4 model." 217 | ] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "metadata": { 222 | "id": "TlPrmGIRw6Md" 223 | }, 224 | "source": [ 225 | "# Install imagemagick\n", 226 | "!sudo apt install imagemagick\n", 227 | "# Resize all images to 416x416\n", 228 | "!mogrify -verbose -strip -resize 416x416! -quality 75 data/obj/**/*jpg" 229 | ], 230 | "execution_count": null, 231 | "outputs": [] 232 | }, 233 | { 234 | "cell_type": "markdown", 235 | "metadata": { 236 | "id": "xyhbN1-nb8UO" 237 | }, 238 | "source": [ 239 | "That's it, our dataset is ready. Time to prepare the training of our model." 240 | ] 241 | }, 242 | { 243 | "cell_type": "markdown", 244 | "metadata": { 245 | "id": "3U8jF9LW8jcg" 246 | }, 247 | "source": [ 248 | "## Retrieve the deep learning model and setup the training\n", 249 | "\n", 250 | "There are several architecture of deep learning networks that exists. We chose to use the Tiny YOLOv4 architecture, as it is fast enough and have a good enough acuracy. \n", 251 | "\n", 252 | "Retrieve the deep learning model from the Darknet github repository, and configure it to fit our objective to recognize two objects." 253 | ] 254 | }, 255 | { 256 | "cell_type": "code", 257 | "metadata": { 258 | "id": "tVqjk64p4i07" 259 | }, 260 | "source": [ 261 | "# Go to Darknet folder\n", 262 | "%cd /content/darknet\n", 263 | "# Remove default config file \n", 264 | "%rm -f cfg/yolov4-tiny-custom.cfg\n", 265 | "# Download custom config file\n", 266 | "!wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4-tiny-custom.cfg -P cfg\n", 267 | "# Rename the config file to create ours\n", 268 | "%cp cfg/yolov4-tiny-custom.cfg cfg/yolov4-tiny-tutorial.cfg\n", 269 | "# Edit the config file as per darknet README recommendations, to train to recognize 2 classes\n", 270 | "!sed -i '7s/subdivisions=1/subdivisions=16/' cfg/yolov4-tiny-tutorial.cfg\n", 271 | "!sed -i '20s/max_batches = 500200/max_batches = 6001/' cfg/yolov4-tiny-tutorial.cfg\n", 272 | "!sed -i '22s/steps=400000,450000/steps=4800,5400/' cfg/yolov4-tiny-tutorial.cfg\n", 273 | "!sed -i '212s/filters=255/filters=21/' cfg/yolov4-tiny-tutorial.cfg\n", 274 | "!sed -i '220s/classes=80/classes=2/' cfg/yolov4-tiny-tutorial.cfg\n", 275 | "!sed -i '263s/filters=255/filters=21/' cfg/yolov4-tiny-tutorial.cfg\n", 276 | "!sed -i '269s/classes=80/classes=2/' cfg/yolov4-tiny-tutorial.cfg\n", 277 | "\n", 278 | "# Verify edition worked\n", 279 | "!diff -u cfg/yolov4-tiny-custom.cfg cfg/yolov4-tiny-tutorial.cfg" 280 | ], 281 | "execution_count": null, 282 | "outputs": [] 283 | }, 284 | { 285 | "cell_type": "markdown", 286 | "metadata": { 287 | "id": "EpoJXNCSd64U" 288 | }, 289 | "source": [ 290 | "We won't train the deep learning model weights from scratch. We will start with pretrained weights in order to speed up the training. We use weights pre-trained on the [Coco dataset](https://cocodataset.org/). Let's download them.\n", 291 | "\n", 292 | "\n", 293 | "\n" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "metadata": { 299 | "id": "3XX_ln6zdrP_" 300 | }, 301 | "source": [ 302 | "# Remove default weights\n", 303 | "%rm -f cfg/yolov4-tiny.conv.29 \n", 304 | "# Download the pretrained weight\n", 305 | "!wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v4_pre/yolov4-tiny.conv.29 -P cfg" 306 | ], 307 | "execution_count": null, 308 | "outputs": [] 309 | }, 310 | { 311 | "cell_type": "markdown", 312 | "metadata": { 313 | "id": "B22NDfL1AB_B" 314 | }, 315 | "source": [ 316 | "## Mount your google drive account to backup weights\n", 317 | "\n", 318 | "We use your google drive folder to save the model weights during the training. This will allow you to retrieve the weights at the end of the training. \n", 319 | "\n", 320 | "Execute the code below: it will tell you to open an url, log in your google drive, and copy paste a secret code in the window below to allow to mount your google drive account. " 321 | ] 322 | }, 323 | { 324 | "cell_type": "code", 325 | "metadata": { 326 | "id": "nyXwXIvhq-Vr" 327 | }, 328 | "source": [ 329 | "# Mount google drive\n", 330 | "from google.colab import drive\n", 331 | "drive.mount(\"drive\")\n", 332 | "\n", 333 | "# Create a backup directory in Google drive\n", 334 | "%mkdir -p drive/MyDrive/NAO_DeepLearning/backup\n", 335 | "# Remove the default 'backup' directory in Darknet\n", 336 | "%rm -rf backup/\n", 337 | "# Replace by ours: create a link to the backup directory in the Google drive\n", 338 | "!ln -fs drive/MyDrive/NAO_DeepLearning/backup" 339 | ], 340 | "execution_count": null, 341 | "outputs": [] 342 | }, 343 | { 344 | "cell_type": "markdown", 345 | "metadata": { 346 | "id": "yEW9GRt_rfdT" 347 | }, 348 | "source": [ 349 | "## Run the training\n", 350 | "\n", 351 | "At last let's run the training !\n", 352 | "\n", 353 | "You will run it for 6000 iterations. This will take approximately one hour.\n", 354 | "Every 100 iteration, the darknet framework will display the mAP (mean average precision) which is a measure of how well the network recognise the objects.\n", 355 | "Eventually you will see lines like:\n", 356 | "\n", 357 | "`mean average precision (mAP@0.50) = 0.751946, or 75.19 %`\n", 358 | "\n", 359 | "(here 75.19% is already a very good mAP). Usually the mAP will increase during the training. If it stop increasing, it's time to stop the training.\n", 360 | "During the training, the computed weights are periodically saved to the `NAO_DeepLearning/backup` folder we created in your google drive." 361 | ] 362 | }, 363 | { 364 | "cell_type": "code", 365 | "metadata": { 366 | "id": "lMIRHT66rnHm" 367 | }, 368 | "source": [ 369 | "# Run the training on our dataset, this takes about 1 hour when using a Colab \n", 370 | "# with a GPU\n", 371 | "!./darknet detector train data/obj.data cfg/yolov4-tiny-tutorial.cfg cfg/yolov4-tiny.conv.29 -map -dont_show 2>&1 | tee backup/darknet.log | grep -vE '(^v3|Loaded|Tensor|^$|Last accuracy|next mAP)'" 372 | ], 373 | "execution_count": null, 374 | "outputs": [] 375 | }, 376 | { 377 | "cell_type": "markdown", 378 | "metadata": { 379 | "id": "ocVk9lt1Y19f" 380 | }, 381 | "source": [ 382 | "## Validate our model\n", 383 | "\n", 384 | "You can validate that your model has been correctly trained by running it on images it has never seen. For this purpose we previously downloaded 80 images (40 for each objects) that we did not use during the training. Let's run our model on it and calculate the mAP (mean average precision).\n", 385 | "\n", 386 | "First put the images in the correct folder, and resize them." 387 | ] 388 | }, 389 | { 390 | "cell_type": "code", 391 | "metadata": { 392 | "id": "D62xl2XZlUhT" 393 | }, 394 | "source": [ 395 | "%mkdir -p data/valid/\n", 396 | "%cp -r OIDv4_ToolKit/OID/Dataset/validation/* data/valid/\n", 397 | "!find data/valid/ -name \"*jpg\" > data/test.txt\n", 398 | "!mogrify -verbose -strip -resize 416x416! -quality 75 data/valid/**/*jpg\n" 399 | ], 400 | "execution_count": null, 401 | "outputs": [] 402 | }, 403 | { 404 | "cell_type": "markdown", 405 | "metadata": { 406 | "id": "21Z7Hl_BhUxX" 407 | }, 408 | "source": [ 409 | "Then let's run our model on them.\n", 410 | "\n", 411 | "It will output something like:\n", 412 | "\n", 413 | "```\n", 414 | "class_id = 0, name = Glasses, ap = 66.72% \t (TP = 31, FP = 7) \n", 415 | "class_id = 1, name = Mobile phone, ap = 85.36% \t (TP = 50, FP = 10) \n", 416 | "\n", 417 | " IoU threshold = 50 %, used Area-Under-Curve for each unique Recall \n", 418 | " mean average precision (mAP@0.50) = 0.760398, or 76.04 % \n", 419 | "```\n", 420 | "\n", 421 | "If you chose different objects than Glasses and Mobile phone, it will use your objects names. We obtained the log above when we prepared this tutorial. In our case, our model had a mAP of 66.72% on Glasses, and 85.36% on Mobile Phone, and a global mAP of 76.04%. This is very good! \n", 422 | "\n", 423 | "How much do you obtain ? Find out with the code below." 424 | ] 425 | }, 426 | { 427 | "cell_type": "code", 428 | "metadata": { 429 | "id": "5PZoz_Sger-0" 430 | }, 431 | "source": [ 432 | "!./darknet detector map data/obj.data cfg/yolov4-tiny-tutorial.cfg backup/yolov4-tiny-tutorial_best.weights\n" 433 | ], 434 | "execution_count": null, 435 | "outputs": [] 436 | }, 437 | { 438 | "cell_type": "markdown", 439 | "metadata": { 440 | "id": "-nTYbgnvXE-h" 441 | }, 442 | "source": [ 443 | "## Gather all files in Google Drive\n", 444 | "\n", 445 | "Finally gather all the required files you will need to upload to the NAO Deep Learning application.\n", 446 | "\n", 447 | "First generate translations for class names in french, so that you can use the NAO app in French if you want." 448 | ] 449 | }, 450 | { 451 | "cell_type": "code", 452 | "metadata": { 453 | "id": "oD_M43fPTgaw" 454 | }, 455 | "source": [ 456 | "translations = {\"Tortoise\": \"Tortue\", \"Container\": \"Récipient\", \"Magpie\": \"Pie\", \"Sea turtle\": \"Tortue de mer\", \"Football\": \"Football\", \"Ambulance\": \"Ambulance\", \"Ladder\": \"Échelle\", \"Toothbrush\": \"Brosse à dents\", \"Syringe\": \"Seringue\", \"Sink\": \"Évier\", \"Toy\": \"Jouet\", \"Organ\": \"Organe\", \"Cassette deck\": \"Platine cassette\", \"Apple\": \"Pomme\", \"Human eye\": \"Œil humain\", \"Cosmetics\": \"Produits de beauté\", \"Paddle\": \"Pagayer\", \"Snowman\": \"Bonhomme de neige\", \"Beer\": \"Bière\", \"Chopsticks\": \"Baguettes\", \"Human beard\": \"Barbe humaine\", \"Bird\": \"Oiseau\", \"Parking meter\": \"Parcmètre\", \"Traffic light\": \"Feu de circulation\", \"Croissant\": \"Croissant\", \"Cucumber\": \"Concombre\", \"Radish\": \"Un radis\", \"Towel\": \"Serviette\", \"Doll\": \"Poupée\", \"Skull\": \"Le crâne\", \"Washing machine\": \"Machine à laver\", \"Glove\": \"Gant\", \"Tick\": \"Cocher\", \"Belt\": \"Courroie\", \"Sunglasses\": \"Des lunettes de soleil\", \"Banjo\": \"Banjo\", \"Cart\": \"Chariot\", \"Ball\": \"Balle\", \"Backpack\": \"Sac à dos\", \"Bicycle\": \"Bicyclette\", \"Home appliance\": \"Électroménager\", \"Centipede\": \"Mille-Pattes\", \"Boat\": \"Bateau\", \"Surfboard\": \"Planche de surf\", \"Boot\": \"Démarrage\", \"Headphones\": \"Écouteurs\", \"Hot dog\": \"Hot-dog\", \"Shorts\": \"Shorts\", \"Fast food\": \"Fast food\", \"Bus\": \"Bus\", \"Boy\": \"Garçon\", \"Screwdriver\": \"Tournevis\", \"Bicycle wheel\": \"Roue de vélo\", \"Barge\": \"Barge\", \"Laptop\": \"Portable\", \"Miniskirt\": \"Mini jupe\", \"Drill\": \"Percer\", \"Dress\": \"Robe\", \"Bear\": \"Ours\", \"Waffle\": \"Gaufre\", \"Pancake\": \"Crêpe\", \"Brown bear\": \"ours brun\", \"Woodpecker\": \"Pivert\", \"Blue jay\": \"Geai bleu\", \"Pretzel\": \"Bretzel\", \"Bagel\": \"Beignet\", \"Tower\": \"La tour\", \"Teapot\": \"Théière\", \"Person\": \"Une personne\", \"Bow and arrow\": \"Arc et des flèches\", \"Swimwear\": \"Maillots de bain\", \"Beehive\": \"Ruche\", \"Brassiere\": \"Soutien-gorge\", \"Bee\": \"abeille\", \"Bat\": \"Chauve souris\", \"Starfish\": \"Étoile de mer\", \"Popcorn\": \"Pop corn\", \"Burrito\": \"Burrito\", \"Chainsaw\": \"Tronçonneuse\", \"Balloon\": \"Ballon\", \"Wrench\": \"Clé\", \"Tent\": \"Tente\", \"Vehicle registration plate\": \"Plaque d'immatriculation du véhicule\", \"Lantern\": \"Lanterne\", \"Toaster\": \"Grille-pain\", \"Flashlight\": \"Lampe de poche\", \"Billboard\": \"Panneau d'affichage\", \"Tiara\": \"Tiare\", \"Limousine\": \"Limousine\", \"Necklace\": \"Collier\", \"Carnivore\": \"Carnivore\", \"Scissors\": \"Les ciseaux\", \"Stairs\": \"Escaliers\", \"Computer keyboard\": \"Clavier d'ordinateur\", \"Printer\": \"Imprimante\", \"Traffic sign\": \"Panneau de signalisation\", \"Chair\": \"Chaise\", \"Shirt\": \"Chemise\", \"Poster\": \"Affiche\", \"Cheese\": \"Du fromage\", \"Sock\": \"Chaussette\", \"Fire hydrant\": \"Bouche d'incendie\", \"Land vehicle\": \"Véhicule terrestre\", \"Earrings\": \"Des boucles d'oreilles\", \"Tie\": \"Attacher\", \"Watercraft\": \"Motomarine\", \"Cabinetry\": \"Ébénisterie\", \"Suitcase\": \"Valise\", \"Muffin\": \"Muffin\", \"Bidet\": \"Bidet\", \"Snack\": \"Goûter\", \"Snowmobile\": \"Motoneige\", \"Clock\": \"L'horloge\", \"Medical equipment\": \"Équipement médical\", \"Cattle\": \"Bétail\", \"Cello\": \"Violoncelle\", \"Jet ski\": \"Jet ski\", \"Camel\": \"chameau\", \"Coat\": \"Manteau\", \"Suit\": \"Combinaison\", \"Desk\": \"Pupitre\", \"Cat\": \"Chat\", \"Bronze sculpture\": \"Sculpture en bronze\", \"Juice\": \"Jus\", \"Gondola\": \"Gondole\", \"Beetle\": \"Scarabée\", \"Cannon\": \"Canon\", \"Computer mouse\": \"Souris d'ordinateur\", \"Cookie\": \"Biscuit\", \"Office building\": \"Immeuble de bureaux\", \"Fountain\": \"Fontaine\", \"Coin\": \"Pièce de monnaie\", \"Calculator\": \"Calculatrice\", \"Cocktail\": \"Cocktail\", \"Computer monitor\": \"Moniteur d'ordinateur\", \"Box\": \"Boîte\", \"Stapler\": \"Agrafeuse\", \"Christmas tree\": \"Sapin de Noël\", \"Cowboy hat\": \"Chapeau de cowboy\", \"Hiking equipment\": \"Matériel de randonnée\", \"Studio couch\": \"Canapé Studio\", \"Drum\": \"Tambouriner\", \"Dessert\": \"Dessert\", \"Wine rack\": \"Casier à vin\", \"Drink\": \"Boire\", \"Zucchini\": \"Courgette\", \"Ladle\": \"Louche\", \"Human mouth\": \"Bouche humaine\", \"Dairy\": \"Laitier\", \"Dice\": \"Dé\", \"Oven\": \"Four\", \"Dinosaur\": \"Dinosaure\", \"Ratchet\": \"Rochet\", \"Couch\": \"Canapé\", \"Cricket ball\": \"Balle de cricket\", \"Winter melon\": \"Melon d'hiver\", \"Spatula\": \"Spatule\", \"Whiteboard\": \"Tableau blanc\", \"Pencil sharpener\": \"Taille-crayon\", \"Door\": \"Porte\", \"Hat\": \"Chapeau\", \"Shower\": \"Douche\", \"Eraser\": \"La gomme\", \"Fedora\": \"Feutre\", \"Guacamole\": \"Guacamole\", \"Dagger\": \"Dague\", \"Scarf\": \"Foulard\", \"Dolphin\": \"Dauphin\", \"Sombrero\": \"Sombrero\", \"Tin can\": \"Boîte de conserve\", \"Mug\": \"Agresser\", \"Tap\": \"Robinet\", \"Harbor seal\": \"Phoque commun\", \"Stretcher\": \"Tendeur\", \"Can opener\": \"Ouvre-boîte\", \"Goggles\": \"Des lunettes de protection\", \"Human body\": \"Corps humain\", \"Roller skates\": \"Patins à roulettes\", \"Coffee cup\": \"Tasse à café\", \"Cutting board\": \"Planche à découper\", \"Blender\": \"Mixeur\", \"Plumbing fixture\": \"Appareil de plomberie\", \"Stop sign\": \"Panneau stop\", \"Office supplies\": \"Fournitures de bureau\", \"Volleyball\": \"Volley-ball\", \"Vase\": \"Vase\", \"Slow cooker\": \"Mijoteuse\", \"Wardrobe\": \"Penderie\", \"Coffee\": \"Café\", \"Whisk\": \"Fouet\", \"Paper towel\": \"Essuie-tout\", \"Personal care\": \"Soins personnels\", \"Food\": \"Aliments\", \"Sun hat\": \"chapeau de soleil\", \"Tree house\": \"Cabane dans les arbres\", \"Flying disc\": \"Disque volant\", \"Skirt\": \"Jupe\", \"Gas stove\": \"Cuisinière à gaz\", \"Salt and pepper shakers\": \"La salière et la poivrière\", \"Mechanical fan\": \"Ventilateur mécanique\", \"Face powder\": \"Poudre pour le visage\", \"Fax\": \"Fax\", \"Fruit\": \"Fruit\", \"French fries\": \"frites\", \"Nightstand\": \"La table de nuit\", \"Barrel\": \"Baril\", \"Kite\": \"cerf-volant\", \"Tart\": \"Tarte\", \"Treadmill\": \"Tapis roulant\", \"Fox\": \"Renard\", \"Flag\": \"Drapeau\", \"Horn\": \"Corne\", \"Window blind\": \"Store de fenêtre\", \"Human foot\": \"Pied humain\", \"Golf cart\": \"Voiturette de golf\", \"Jacket\": \"Veste\", \"Egg\": \"Œuf\", \"Street light\": \"éclairage public\", \"Guitar\": \"Guitare\", \"Pillow\": \"Oreiller\", \"Human leg\": \"Jambe humaine\", \"Isopod\": \"Isopode\", \"Grape\": \"Grain de raisin\", \"Human ear\": \"Oreille humaine\", \"Power plugs and sockets\": \"Fiches et prises de courant\", \"Panda\": \"Panda\", \"Giraffe\": \"Girafe\", \"Woman\": \"Femme\", \"Door handle\": \"Poignée de porte\", \"Rhinoceros\": \"Rhinocéros\", \"Bathtub\": \"Baignoire\", \"Goldfish\": \"Poisson rouge\", \"Houseplant\": \"Plante d'appartement\", \"Goat\": \"Chèvre\", \"Baseball bat\": \"Batte de baseball\", \"Baseball glove\": \"Gant de baseball\", \"Mixing bowl\": \"Bol mélangeur\", \"Marine invertebrates\": \"Invertébrés marins\", \"Kitchen utensil\": \"Ustensile de cuisine\", \"Light switch\": \"Interrupteur\", \"House\": \"loger\", \"Horse\": \"Cheval\", \"Stationary bicycle\": \"Vélo stationnaire\", \"Hammer\": \"Marteau\", \"Ceiling fan\": \"Ventilateur de plafond\", \"Sofa bed\": \"Canapé-lit\", \"Adhesive tape\": \"Ruban adhésif\", \"Harp\": \"Harpe\", \"Sandal\": \"Sandale\", \"Bicycle helmet\": \"Casque de vélo\", \"Saucer\": \"Soucoupe\", \"Harpsichord\": \"Clavecin\", \"Human hair\": \"Cheveux humains\", \"Heater\": \"Chauffe-eau\", \"Harmonica\": \"Harmonica\", \"Hamster\": \"Hamster\", \"Curtain\": \"Rideau\", \"Bed\": \"Lit\", \"Kettle\": \"Bouilloire\", \"Fireplace\": \"Cheminée\", \"Scale\": \"Escalader\", \"Drinking straw\": \"Paille\", \"Insect\": \"Insecte\", \"Hair dryer\": \"Sèche-cheveux\", \"Kitchenware\": \"Ustensiles de cuisine\", \"Indoor rower\": \"Rameur d'intérieur\", \"Invertebrate\": \"Invertébré\", \"Food processor\": \"Robot culinaire\", \"Bookcase\": \"Bibliothèque\", \"Refrigerator\": \"Réfrigérateur\", \"Wood-burning stove\": \"Poêle à bois\", \"Punching bag\": \"Sac de boxe\", \"Common fig\": \"Figue commune\", \"Cocktail shaker\": \"Shaker à cocktail\", \"Jaguar\": \"Jaguar\", \"Golf ball\": \"Balle de golf\", \"Fashion accessory\": \"Accessoire de mode\", \"Alarm clock\": \"Réveil\", \"Filing cabinet\": \"Classeur\", \"Artichoke\": \"Artichaut\", \"Table\": \"Table\", \"Tableware\": \"Vaisselle\", \"Kangaroo\": \"Kangourou\", \"Koala\": \"Koala\", \"Knife\": \"Couteau\", \"Bottle\": \"Bouteille\", \"Bottle opener\": \"Ouvre-bouteille\", \"Lynx\": \"Lynx\", \"Lavender\": \"Lavande\", \"Lighthouse\": \"Phare\", \"Dumbbell\": \"Haltère\", \"Human head\": \"Tête humaine\", \"Bowl\": \"bol\", \"Humidifier\": \"Humidificateur\", \"Porch\": \"Porche\", \"Lizard\": \"Lézard\", \"Billiard table\": \"Table de billard\", \"Mammal\": \"Mammifère\", \"Mouse\": \"Souris\", \"Motorcycle\": \"Moto\", \"Musical instrument\": \"Instrument de musique\", \"Swim cap\": \"Bonnet de bain\", \"Frying pan\": \"Poêle à frire\", \"Snowplow\": \"Chasse-neige\", \"Bathroom cabinet\": \"Armoire de toilette\", \"Missile\": \"Missile\", \"Bust\": \"Bousiller\", \"Man\": \"Homme\", \"Waffle iron\": \"Gaufrier\", \"Milk\": \"Lait\", \"Ring binder\": \"Reliure à anneaux\", \"Plate\": \"Plaque\", \"Mobile phone\": \"Téléphone mobile\", \"Baked goods\": \"Produits de boulangerie\", \"Mushroom\": \"Champignon\", \"Crutch\": \"Béquille\", \"Pitcher\": \"Lanceur\", \"Mirror\": \"Miroir\", \"Lifejacket\": \"Gilet de sauvetage\", \"Table tennis racket\": \"Raquette de tennis de table\", \"Pencil case\": \"Trousse\", \"Musical keyboard\": \"Clavier musical\", \"Scoreboard\": \"Tableau de bord\", \"Briefcase\": \"Mallette\", \"Kitchen knife\": \"Couteau de cuisine\", \"Nail\": \"Clou\", \"Tennis ball\": \"Balle de tennis\", \"Plastic bag\": \"Sac plastique\", \"Oboe\": \"Hautbois\", \"Chest of drawers\": \"Commode\", \"Ostrich\": \"Autruche\", \"Piano\": \"Piano\", \"Girl\": \"Fille\", \"Plant\": \"Usine\", \"Potato\": \"Pomme de terre\", \"Hair spray\": \"Laque pour les cheveux\", \"Sports equipment\": \"Équipement sportif\", \"Pasta\": \"Pâtes\", \"Penguin\": \"manchot\", \"Pumpkin\": \"Citrouille\", \"Pear\": \"Poire\", \"Infant bed\": \"Lit bébé\", \"Polar bear\": \"Ours polaire\", \"Mixer\": \"Mixer\", \"Cupboard\": \"Armoire\", \"Jacuzzi\": \"Jacuzzi\", \"Pizza\": \"Pizza\", \"Digital clock\": \"Horloge digitale\", \"Pig\": \"Cochon\", \"Reptile\": \"Reptile\", \"Rifle\": \"Fusil\", \"Lipstick\": \"Rouge à lèvres\", \"Skateboard\": \"planche à roulette\", \"Raven\": \"corbeau\", \"High heels\": \"Talons hauts\", \"Red panda\": \"Panda rouge\", \"Rose\": \"Rose\", \"Rabbit\": \"Lapin\", \"Sculpture\": \"Sculpture\", \"Saxophone\": \"Saxophone\", \"Shotgun\": \"Fusil à pompe\", \"Seafood\": \"Fruits de mer\", \"Submarine sandwich\": \"Sandwich sous-marin\", \"Snowboard\": \"Snowboard\", \"Sword\": \"Épée\", \"Picture frame\": \"Cadre de l'image\", \"Sushi\": \"Sushi\", \"Loveseat\": \"Causeuse\", \"Ski\": \"Ski\", \"Squirrel\": \"Écureuil\", \"Tripod\": \"Trépied\", \"Stethoscope\": \"Stéthoscope\", \"Submarine\": \"Sous-marin\", \"Scorpion\": \"Scorpion\", \"Segway\": \"Segway\", \"Training bench\": \"Banc d'entraînement\", \"Snake\": \"Serpent\", \"Coffee table\": \"Table basse\", \"Skyscraper\": \"Gratte-ciel\", \"Sheep\": \"Mouton\", \"Television\": \"Télévision\", \"Trombone\": \"Trombone\", \"Tea\": \"Thé\", \"Tank\": \"Réservoir\", \"Taco\": \"Taco\", \"Telephone\": \"Téléphone\", \"Torch\": \"Torche\", \"Tiger\": \"tigre\", \"Strawberry\": \"fraise\", \"Trumpet\": \"Trompette\", \"Tree\": \"Arbre\", \"Tomato\": \"Tomate\", \"Train\": \"Former\", \"Tool\": \"Outil\", \"Picnic basket\": \"Panier pique-nique\", \"Cooking spray\": \"Aérosol de cuisson\", \"Trousers\": \"Pantalon\", \"Bowling equipment\": \"Matériel de bowling\", \"Football helmet\": \"Casque de football américain\", \"Truck\": \"Camion\", \"Measuring cup\": \"Tasse à mesurer\", \"Coffeemaker\": \"Machine à café\", \"Violin\": \"Violon\", \"Vehicle\": \"Véhicule\", \"Handbag\": \"Sac à main\", \"Paper cutter\": \"Coupe-papier\", \"Wine\": \"Vin\", \"Weapon\": \"Arme\", \"Wheel\": \"Roue\", \"Worm\": \"Ver\", \"Wok\": \"Wok\", \"Whale\": \"Baleine\", \"Zebra\": \"Zèbre\", \"Auto part\": \"Pièce auto\", \"Jug\": \"Cruche\", \"Pizza cutter\": \"couteau à pizza\", \"Cream\": \"Crème\", \"Monkey\": \"Singe\", \"Lion\": \"Lion\", \"Bread\": \"Pain\", \"Platter\": \"Plat\", \"Chicken\": \"Poulet\", \"Eagle\": \"Aigle\", \"Helicopter\": \"Hélicoptère\", \"Owl\": \"Chouette\", \"Duck\": \"Canard\", \"Turtle\": \"Tortue\", \"Hippopotamus\": \"Hippopotame\", \"Crocodile\": \"Crocodile\", \"Toilet\": \"Toilette\", \"Toilet paper\": \"Papier toilette\", \"Squid\": \"Calmar\", \"Clothing\": \"Vêtements\", \"Footwear\": \"Chaussure\", \"Lemon\": \"Citron\", \"Spider\": \"Araignée\", \"Deer\": \"Cerf\", \"Frog\": \"La grenouille\", \"Banana\": \"Banane\", \"Rocket\": \"Fusée\", \"Wine glass\": \"Verre de vin\", \"Countertop\": \"Comptoir\", \"Tablet computer\": \"Tablette\", \"Waste container\": \"Conteneur à déchets\", \"Swimming pool\": \"Piscine\", \"Dog\": \"Chien\", \"Book\": \"Livre\", \"Elephant\": \"l'éléphant\", \"Shark\": \"Requin\", \"Candle\": \"Bougie\", \"Leopard\": \"Léopard\", \"Axe\": \"Hache\", \"Hand dryer\": \"Sèche mains\", \"Soap dispenser\": \"Distributeur de savon\", \"Porcupine\": \"Porc-épic\", \"Flower\": \"Fleur\", \"Canary\": \"Canari\", \"Cheetah\": \"guépard\", \"Palm tree\": \"palmier\", \"Hamburger\": \"Hamburger\", \"Maple\": \"Érable\", \"Building\": \"Bâtiment\", \"Fish\": \"Poisson\", \"Lobster\": \"Homard\", \"Asparagus\": \"Asperges\", \"Furniture\": \"Meubles\", \"Hedgehog\": \"Hérisson\", \"Airplane\": \"Avion\", \"Spoon\": \"Cuillère\", \"Otter\": \"loutre\", \"Bull\": \"Taureau\", \"Oyster\": \"huître\", \"Horizontal bar\": \"Barre horizontale\", \"Convenience store\": \"Épicerie\", \"Bomb\": \"Bombe\", \"Bench\": \"Banc\", \"Ice cream\": \"La crème glacée\", \"Caterpillar\": \"chenille\", \"Butterfly\": \"Papillon\", \"Parachute\": \"Parachute\", \"Orange\": \"Orange\", \"Antelope\": \"Antilope\", \"Beaker\": \"Gobelet\", \"Moths and butterflies\": \"Papillons et papillons\", \"Window\": \"La fenêtre\", \"Closet\": \"Toilettes\", \"Castle\": \"château\", \"Jellyfish\": \"Méduse\", \"Goose\": \"OIE\", \"Mule\": \"Mule\", \"Swan\": \"cygne\", \"Peach\": \"Pêche\", \"Coconut\": \"Noix de coco\", \"Seat belt\": \"Ceinture de sécurité\", \"Raccoon\": \"Raton laveur\", \"Chisel\": \"Ciseau\", \"Fork\": \"Fourchette\", \"Lamp\": \"Lampe\", \"Camera\": \"Caméra\", \"Squash\": \"Écraser\", \"Racket\": \"Raquette\", \"Human face\": \"Visage humain\", \"Human arm\": \"Bras humain\", \"Vegetable\": \"Légume\", \"Diaper\": \"Couche\", \"Unicycle\": \"Monocycle\", \"Falcon\": \"Faucon\", \"Chime\": \"Carillon\", \"Snail\": \"Escargot\", \"Shellfish\": \"Fruits de mer\", \"Cabbage\": \"Choux\", \"Carrot\": \"Carotte\", \"Mango\": \"Mangue\", \"Jeans\": \"jeans\", \"Flowerpot\": \"Pot de fleur\", \"Pineapple\": \"L'ananas\", \"Drawer\": \"Tiroir\", \"Stool\": \"Tabouret\", \"Envelope\": \"Enveloppe\", \"Cake\": \"Gâteau\", \"Dragonfly\": \"Libellule\", \"Sunflower\": \"Tournesol\", \"Microwave oven\": \"Four micro-onde\", \"Honeycomb\": \"Rayon de miel\", \"Marine mammal\": \"Mammifère marin\", \"Sea lion\": \"Lion de mer\", \"Ladybug\": \"Coccinelle\", \"Shelf\": \"Étagère\", \"Watch\": \"Regardez\", \"Candy\": \"Bonbons\", \"Salad\": \"salade\", \"Parrot\": \"Perroquet\", \"Handgun\": \"Pistolet\", \"Sparrow\": \"moineau\", \"Van\": \"Van\", \"Grinder\": \"Broyeur\", \"Spice rack\": \"étagère à épices\", \"Light bulb\": \"Ampoule\", \"Corded phone\": \"Téléphone filaire\", \"Sports uniform\": \"Uniforme de sport\", \"Tennis racket\": \"Raquette de tennis\", \"Wall clock\": \"horloge murale\", \"Serving tray\": \"Plateau\", \"Kitchen & dining room table\": \"Table de cuisine et salle à manger\", \"Dog bed\": \"Lit pour chien\", \"Cake stand\": \"Stand de gâteaux\", \"Cat furniture\": \"Meubles pour chats\", \"Bathroom accessory\": \"Accessoire de salle de bain\", \"Facial tissue holder\": \"Porte-mouchoirs en papier\", \"Pressure cooker\": \"Cocotte minute\", \"Kitchen appliance\": \"Appareil de cuisine\", \"Tire\": \"Pneu\", \"Ruler\": \"Règle\", \"Luggage and bags\": \"Bagages et sacs\", \"Microphone\": \"Microphone\", \"Broccoli\": \"Brocoli\", \"Umbrella\": \"Parapluie\", \"Pastry\": \"Pâtisserie\", \"Grapefruit\": \"Pamplemousse\", \"Band-aid\": \"Pansement\", \"Animal\": \"Animal\", \"Bell pepper\": \"poivron\", \"Turkey\": \"dinde\", \"Lily\": \"Lis\", \"Pomegranate\": \"Grenade\", \"Doughnut\": \"Donut\", \"Glasses\": \"Lunettes\", \"Human nose\": \"Nez humain\", \"Pen\": \"Stylo\", \"Ant\": \"Fourmi\", \"Car\": \"Auto\", \"Aircraft\": \"Avion\", \"Human hand\": \"Main humaine\", \"Skunk\": \"Moufette\", \"Teddy bear\": \"ours en peluche\", \"Watermelon\": \"Pastèque\", \"Cantaloupe\": \"Cantaloup\", \"Dishwasher\": \"Lave-vaisselle\", \"Flute\": \"Flûte\", \"Balance beam\": \"Poutre d'équilibre\", \"Sandwich\": \"Sandwich\", \"Shrimp\": \"Crevette\", \"Sewing machine\": \"Machine à coudre\", \"Binoculars\": \"Jumelles\", \"Rays and skates\": \"Rayons et patins\", \"Ipod\": \"Ipod\", \"Accordion\": \"Accordéon\", \"Willow\": \"saule\", \"Crab\": \"Crabe\", \"Crown\": \"couronner\", \"Seahorse\": \"hippocampe\", \"Perfume\": \"Parfum\", \"Alpaca\": \"Alpaga\", \"Taxi\": \"Taxi\", \"Canoe\": \"Canoë\", \"Remote control\": \"Télécommande\", \"Wheelchair\": \"Fauteuil roulant\", \"Rugby ball\": \"Ballon de rugby\", \"Armadillo\": \"Tatou\", \"Maracas\": \"Maracas\", \"Helmet\": \"Casque\"}\n", 457 | "with open(\"data/obj.names\") as file:\n", 458 | " lines = file.readlines()\n", 459 | " with open(\"data/obj.names.fr\", \"w\") as fr:\n", 460 | " for line in lines:\n", 461 | " fr.write(\"%s\\n\" % translations[line.rstrip()])" 462 | ], 463 | "execution_count": null, 464 | "outputs": [] 465 | }, 466 | { 467 | "cell_type": "markdown", 468 | "metadata": { 469 | "id": "0ziytKFgYauT" 470 | }, 471 | "source": [ 472 | "Then copy all result files to folder `NAO_DeepLearning/backup/result`" 473 | ] 474 | }, 475 | { 476 | "cell_type": "code", 477 | "metadata": { 478 | "id": "_ER1LwUORGvV" 479 | }, 480 | "source": [ 481 | "%mkdir -p backup/result/\n", 482 | "%cp backup/yolov4-tiny-tutorial_best.weights backup/result/model.weights \n", 483 | "%cp cfg/yolov4-tiny-tutorial.cfg backup/result/model.cfg\n", 484 | "%cp data/obj.names backup/result/objects.names.en\n", 485 | "%cp data/obj.names.fr backup/result/objects.names.fr" 486 | ], 487 | "execution_count": null, 488 | "outputs": [] 489 | }, 490 | { 491 | "cell_type": "markdown", 492 | "metadata": { 493 | "id": "anzEk61JWOMg" 494 | }, 495 | "source": [ 496 | "And that's it ! You can go to your Google Drive `NAO_DeepLearning/backup/result` folder and download its content. You will have to upload it to the NAO Deep Learning application." 497 | ] 498 | } 499 | ] 500 | } -------------------------------------------------------------------------------- /scripts/deep_nao.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -* 2 | """ 3 | 4 | """ 5 | 6 | __version__ = "0.0.1" 7 | 8 | __copyright__ = "Copyright 2021, Aldebaran Robotics" 9 | __author__ = 'Remi HUMBERT' 10 | __email__ = 'rhumbert@softbankrobotics.com' 11 | 12 | from functools import wraps 13 | import stk.runner 14 | import stk.events 15 | import stk.services 16 | import stk.worker 17 | from stk.logging import get_logger, log_exceptions_and_return, log_exceptions 18 | import qi 19 | import sys 20 | import motion 21 | import almath 22 | import time 23 | import collections 24 | import numpy 25 | import vision_definitions 26 | import base64 27 | import os 28 | from models import MobilenetV2SSD, TinyYoloV4 29 | from translations import sentences 30 | 31 | SERVICE_NAME="DeepNao" 32 | 33 | BEHAVIOR="deep_nao/behavior" 34 | 35 | if stk.runner.is_on_robot(): 36 | # Special: for NAOqi OS, pre-compiled cv2 is packaged 37 | # patch path for local cv2, should only be run when on robot 38 | sys.path.insert(0, "lib/python2.7/site-packages") 39 | 40 | import cv2 41 | 42 | 43 | @qi.singleThreaded() 44 | class Activity(object): 45 | 46 | APP_ID = "com.softbankrobotics.deep_nao" 47 | FOV_V = 0.7731809 48 | FOV_H = 0.963421747 49 | 50 | # When continuous describing the scene, do describe the same object only 51 | # after 60 seconds have passed. 52 | CONTINUOUS_DESCRIPTION_REPEAT_TIME = 60 53 | 54 | def __init__(self, qiapp): 55 | self.qiapp = qiapp 56 | self.logger = get_logger(qiapp.session, self.APP_ID) 57 | self.running = qi.Property("b") 58 | self.running.setValue(False) 59 | self.wait_for_dependency_services() 60 | self.events = stk.events.EventHelper(qiapp.session) 61 | self.services = stk.services.ServiceCache(qiapp.session) 62 | 63 | self.periodic_task = qi.PeriodicTask() 64 | self.periodic_task.setCallback(self.process_camera_image) 65 | self.periodic_task.compensateCallbackTime(False) 66 | # Run as fast as possible, network inference will slow it down 67 | self.periodic_task.setUsPeriod(1) 68 | self.subscriber_id = None 69 | 70 | self.topic_name = None 71 | self.uploaded_size = {} 72 | # Continuous description 73 | self.continuous_description = qi.Property("b") 74 | self.continuous_exploration = qi.Property("b") 75 | self.use_custom_model = qi.Property("b") 76 | 77 | 78 | 79 | @qi.nobind 80 | @log_exceptions 81 | def on_continuous_exploration_change(self, exploration_value): 82 | if exploration_value: 83 | self.services.Explorer.start() 84 | else: 85 | self.services.Explorer.stop() 86 | self.services.ALRobotPosture.goToPosture("Crouch", 1.0) 87 | self.services.ALMotion.rest() 88 | 89 | ## Handling loading of dependencies 90 | 91 | @qi.nobind 92 | @log_exceptions 93 | def wait_for_dependency_services(self): 94 | dependencies = [ 95 | "ALDialog", 96 | "ALAutonomousLife", 97 | "ALTextToSpeech", 98 | "ALVideoDevice", 99 | "ALRobotPosture", 100 | "ALMotion", 101 | "Explorer", 102 | "StopBehaviorOnRobotFall", 103 | ] 104 | for service in dependencies: 105 | self.logger.info("Waiting for service %s" % service) 106 | self.qiapp.session.waitForService(service) 107 | 108 | ##################################### 109 | ## Service start / stop management ## 110 | ##################################### 111 | 112 | @qi.nobind 113 | @stk.events.on("AutonomousLife/FocusedActivity") 114 | @log_exceptions 115 | def on_focus_change(self, activity): 116 | #if not self.robot_has_fallen.value(): 117 | if activity == BEHAVIOR: 118 | self.start_behavior() 119 | else: 120 | self.stop_behavior() 121 | 122 | @qi.nobind 123 | @log_exceptions 124 | @stk.events.on("ALTextToSpeech.languageTTS") 125 | def on_language_change(self, value): 126 | if value == "fr_FR": 127 | value = "French" 128 | else: 129 | value = "English" 130 | self.language = value 131 | self.net.reload_classes(value) 132 | 133 | @qi.nobind 134 | @log_exceptions 135 | def on_start(self): 136 | self.events.connect_decorators(self) 137 | # clean any previous subscribtion (on re-install or restart) 138 | self.unsubscribe_all() 139 | use_model_pref = self.services.ALPreferenceManager.getValue( 140 | "com.softbankrobotics.deep_nao", "use_custom_model") 141 | try: 142 | use_model_pref = bool(int(use_model_pref)) 143 | except: 144 | use_model_pref = False 145 | self.logger.info("Use model pref: %s" % use_model_pref) 146 | use_model_pref = use_model_pref and self.isCustomModelAvailable() 147 | self.logger.info("Use model pref with custom model: %s" % use_model_pref) 148 | self.use_custom_model.setValue(bool(use_model_pref)) 149 | self.use_custom_model.connect(self.use_custom_model_changes) 150 | 151 | self.services.ALMemory.raiseEvent("DeepNao/RobotFallen", "False") 152 | 153 | if self.services.ALAutonomousLife.focusedActivity() == BEHAVIOR: 154 | self.start_behavior() 155 | 156 | 157 | @qi.nobind 158 | @log_exceptions 159 | def on_stop(self): 160 | "Cleanup" 161 | self.stop_behavior() 162 | self.logger.info("Application finished.") 163 | self.events.clear() 164 | 165 | @qi.nobind 166 | @log_exceptions 167 | def start_behavior(self): 168 | if self.running.value(): 169 | return 170 | self.logger.info("Start behavior.") 171 | self.running.setValue(True) 172 | try: 173 | self.described_objects_time = {} 174 | self.move_head_state = 0 175 | self.continuous_description.setValue(False) 176 | self.continuous_exploration.setValue(False) 177 | self.explore_link = self.continuous_exploration.connect( 178 | self.on_continuous_exploration_change) 179 | 180 | self.language = self.services.ALTextToSpeech.getLanguage() 181 | 182 | if self.use_custom_model.value(): 183 | if not self.load_custom_yolov4_network(): 184 | self.load_mobilenetv2_ssd_network() 185 | else: 186 | self.load_mobilenetv2_ssd_network() 187 | 188 | self.services.ALAutonomousLife.setAutonomousAbilityEnabled("BackgroundMovement", False) 189 | self.services.ALAutonomousLife.setAutonomousAbilityEnabled("BasicAwareness", False) 190 | self.services.ALRobotPosture.goToPosture("Crouch", 1.0) 191 | self.services.ALMotion.rest() 192 | self.services.ALTextToSpeech.say(sentences[self.language]["ready"]) 193 | self.subscribe_to_camera() 194 | self.periodic_task.start(True) 195 | except Exception, e: 196 | self.logger.error(e) 197 | self.stop_behavior() 198 | self.running.setValue(False) 199 | 200 | 201 | @log_exceptions 202 | def stop_behavior(self): 203 | self.logger.info("Stop behavior.") 204 | try: 205 | self.services.Explorer.stop() 206 | self.continuous_exploration.disconnect(self.explore_link) 207 | self.continuous_exploration.setValue(False) 208 | self.continuous_description.setValue(False) 209 | self.periodic_task.stop() 210 | self.unsubscribe_from_camera() 211 | finally: 212 | self.running.setValue(False) 213 | 214 | 215 | ####################### 216 | # Deep Models loading # 217 | ####################### 218 | 219 | @qi.nobind 220 | @log_exceptions 221 | def load_mobilenetv2_ssd_network(self): 222 | classes = { 223 | "French": "models/ssd_mobilenet_v2_oid_v4/objects.names.fr", 224 | "English": "models/ssd_mobilenet_v2_oid_v4/objects.names.en" 225 | } 226 | self.net = MobilenetV2SSD( 227 | "models/ssd_mobilenet_v2_oid_v4/frozen_inference_graph.pb", 228 | "models/ssd_mobilenet_v2_oid_v4/graph.pbtxt", 229 | classes, 230 | self.language) 231 | self.logger.info("Net loaded, found classes: %s" % self.net.classes) 232 | 233 | 234 | 235 | CUSTOM_MODEL_CONFIG = "models/yolo_v4_tiny_custom/model.cfg" 236 | CUSTOM_MODEL_WEIGHT = "models/yolo_v4_tiny_custom/model.weights" 237 | CUSTOM_MODEL_NAMES_FR = "models/yolo_v4_tiny_custom/objects.names.fr" 238 | CUSTOM_MODEL_NAMES_EN = "models/yolo_v4_tiny_custom/objects.names.en" 239 | 240 | @log_exceptions 241 | def isCustomModelAvailable(self): 242 | return os.path.exists(self.CUSTOM_MODEL_CONFIG) \ 243 | and os.path.exists(self.CUSTOM_MODEL_WEIGHT) \ 244 | and os.path.exists(self.CUSTOM_MODEL_NAMES_FR) \ 245 | and os.path.exists(self.CUSTOM_MODEL_NAMES_EN) 246 | 247 | @qi.nobind 248 | @log_exceptions 249 | def use_custom_model_changes(self, value): 250 | self.services.ALPreferenceManager.setValue( 251 | "com.softbankrobotics.deep_nao", "use_custom_model", value) 252 | if self.running.value(): 253 | if self.use_custom_model.value(): 254 | if not self.load_custom_yolov4_network(): 255 | self.use_custom_model.setValue(False) 256 | return 257 | else: 258 | self.load_mobilenetv2_ssd_network() 259 | self.services.ALTextToSpeech.say(sentences[self.language]["ready"]) 260 | 261 | @qi.nobind 262 | @log_exceptions 263 | def load_custom_yolov4_network(self): 264 | classes = { 265 | "French": self.CUSTOM_MODEL_NAMES_FR, 266 | "English": self.CUSTOM_MODEL_NAMES_EN, 267 | } 268 | try: 269 | self.net = TinyYoloV4( 270 | self.CUSTOM_MODEL_WEIGHT, 271 | self.CUSTOM_MODEL_CONFIG, 272 | classes, 273 | self.language 274 | ) 275 | except Exception, e: 276 | self.logger.error("Error loading custom model: %s" % (e)) 277 | self.services.ALTextToSpeech.say(sentences[self.language]["error_custom"]) 278 | self.remove_custom_model() 279 | return False 280 | 281 | self.logger.info("Net loaded, found classes: %s" % self.net.classes) 282 | return True 283 | 284 | def remove_custom_model(self): 285 | if os.path.exists(self.CUSTOM_MODEL_CONFIG): 286 | os.remove(self.CUSTOM_MODEL_CONFIG) 287 | if os.path.exists(self.CUSTOM_MODEL_WEIGHT): 288 | os.remove(self.CUSTOM_MODEL_WEIGHT) 289 | if os.path.exists(self.CUSTOM_MODEL_NAMES_FR): 290 | os.remove(self.CUSTOM_MODEL_NAMES_FR) 291 | if os.path.exists(self.CUSTOM_MODEL_NAMES_EN): 292 | os.remove(self.CUSTOM_MODEL_NAMES_EN) 293 | 294 | 295 | ########################### 296 | # Camera image processing # 297 | ########################### 298 | 299 | @qi.nobind 300 | @log_exceptions 301 | def subscribe_to_camera(self): 302 | fps = 2 303 | self.subscriber_id = self.services.ALVideoDevice.subscribeCamera( 304 | self.APP_ID, 305 | vision_definitions.kTopCamera, 306 | vision_definitions.kVGA, 307 | vision_definitions.kBGRColorSpace, 308 | fps 309 | ) 310 | 311 | @qi.nobind 312 | @log_exceptions 313 | def unsubscribe_from_camera(self): 314 | self.logger.info("Unsubscribe from camera") 315 | if self.subscriber_id is not None: 316 | self.services.ALVideoDevice.unsubscribe(self.subscriber_id) 317 | self.subscriber_id = None 318 | self.logger.info("Unsubscribe done") 319 | else: 320 | self.logger.info("Nothing to unsubscribe") 321 | 322 | @qi.nobind 323 | @log_exceptions 324 | def process_camera_image(self): 325 | #self.logger.info("Process camera image") 326 | 327 | image_remote = self.services.ALVideoDevice.getImageRemote(self.subscriber_id) 328 | if not image_remote: 329 | raise Exception("No data in image") 330 | 331 | width = image_remote[0] 332 | height = image_remote[1] 333 | channels = image_remote[2] 334 | colorspace = image_remote[3] 335 | data = image_remote[6] 336 | 337 | #self.logger.info("Got image %sx%sx%s %s" % (width, height, channels, colorspace)) 338 | 339 | imageBGR = numpy.frombuffer(data, dtype = numpy.uint8).reshape(height, width, channels) 340 | 341 | self.services.ALVideoDevice.releaseImage(self.subscriber_id) 342 | 343 | start_time = time.time() 344 | imageBGR = self.net.label_image_content(imageBGR) 345 | end_time = time.time() 346 | self.logger.info("Inference time: %s" % (end_time - start_time)) 347 | 348 | if self.continuous_description.value() or self.continuous_exploration.value(): 349 | self.continuousDescribeObjects() 350 | 351 | path = "/home/nao/.local/share/PackageManager/apps/deep_nao/html/img/deep.jpg" 352 | cv2.imwrite(path, imageBGR, [cv2.IMWRITE_JPEG_QUALITY, 55]) 353 | 354 | 355 | @qi.nobind 356 | @log_exceptions 357 | def unsubscribe_all(self): 358 | self.logger.info("_unsubscribe_all...") 359 | for subscriber_id in self.services.ALVideoDevice.getSubscribers(): 360 | if subscriber_id.startswith(self.APP_ID): 361 | self.services.ALVideoDevice.unsubscribe(subscriber_id) 362 | self.logger.info("_unsubscribe_all done") 363 | 364 | 365 | ########################## 366 | # Web interface commands # 367 | ########################## 368 | 369 | @qi.nobind 370 | @log_exceptions 371 | def remove_head_stiffness(self): 372 | self.services.ALMotion.setStiffnesses(['HeadYaw', 'HeadPitch'], 0.0) 373 | 374 | @stk.worker.async("move_head", 2, 0) 375 | @log_exceptions 376 | def webLookAt(self, x, y): 377 | if hasattr(self, "remove_stiffness_future") \ 378 | and self.remove_stiffness_future.isRunning(): 379 | self.remove_stiffness_future.cancel() 380 | delta_yaw = -x * self.FOV_H / 2 381 | delta_pitch = -y * self.FOV_V / 2 382 | self.last_lookat_time = time.time() 383 | self.services.ALMotion.setStiffnesses(['HeadYaw', 'HeadPitch'], 0.5) 384 | self.services.ALMotion.angleInterpolation( 385 | ['HeadYaw', 'HeadPitch'], [delta_yaw, delta_pitch], [1, 1], False) 386 | self.remove_stiffness_future = qi.async(self.remove_head_stiffness, delay=2000*1000) 387 | 388 | 389 | @stk.worker.async("describe_objects", 2, 0) 390 | @log_exceptions 391 | def describeVisibleObjects(self): 392 | if self.net.last_visible_objects: 393 | sentence = sentences[self.language]["I_see"] 394 | for obj, num in self.net.last_visible_objects.iteritems(): 395 | sentence += "%s %s" % (num, obj) 396 | else: 397 | sentence = sentences[self.language]["see_no_objects"] 398 | 399 | self.services.ALTextToSpeech.say(sentence) 400 | 401 | 402 | ########################## 403 | # Continuous description # 404 | ########################## 405 | 406 | @qi.nobind 407 | @log_exceptions 408 | def look_front(self): 409 | self.services.ALMotion.setStiffnesses(['HeadYaw'], 1.0) 410 | self.services.ALMotion.angleInterpolation("HeadYaw", 0, 1, True) 411 | self.services.ALMotion.setStiffnesses(['HeadYaw'], 0.0) 412 | 413 | @qi.nobind 414 | @log_exceptions 415 | def look_left(self): 416 | self.services.ALMotion.setStiffnesses(['HeadYaw'], 1.0) 417 | self.services.ALMotion.angleInterpolation("HeadYaw", 0.45, 1, True) 418 | self.services.ALMotion.setStiffnesses(['HeadYaw'], 0.0) 419 | 420 | @qi.nobind 421 | @log_exceptions 422 | def look_right(self): 423 | self.services.ALMotion.setStiffnesses(['HeadYaw'], 1.0) 424 | self.services.ALMotion.angleInterpolation("HeadYaw", -0.45, 1, True) 425 | self.services.ALMotion.setStiffnesses(['HeadYaw'], 0.0) 426 | 427 | 428 | @qi.nobind 429 | @log_exceptions 430 | def move_head_to_search_for_objects(self): 431 | current_time = time.time() 432 | 433 | if not hasattr(self, "last_described_time"): 434 | # Init last_described_time if it does not exists 435 | self.last_described_time = current_time 436 | return 437 | 438 | elapsed_time = current_time - self.last_described_time 439 | self.logger.info("Move head: %s, head state: %s %s" 440 | % (elapsed_time, self.move_head_state, self.last_described_time)) 441 | 442 | if elapsed_time > 12: 443 | self.look_front() 444 | self.move_head_state = 0 445 | self.last_described_time = current_time 446 | elif self.move_head_state == 0 and elapsed_time > 3: 447 | self.look_left() 448 | self.move_head_state = 1 449 | elif self.move_head_state == 1 and elapsed_time > 6: 450 | self.look_front() 451 | self.move_head_state = 2 452 | elif self.move_head_state == 2 and elapsed_time > 9: 453 | self.look_right() 454 | self.move_head_state = 3 455 | 456 | 457 | @stk.worker.async("describe_objects", 2, 0) 458 | @log_exceptions 459 | def continuousDescribeObjects(self): 460 | # If no objects to describe, return 461 | if not self.net.last_visible_objects: 462 | self.move_head_to_search_for_objects() 463 | return 464 | 465 | # Remove objects already describe recently 466 | objets_to_describe = self.net.last_visible_objects.copy() 467 | current_time = time.time() 468 | self.logger.info(self.described_objects_time) 469 | for obj, t in self.described_objects_time.items(): 470 | if current_time - t < self.CONTINUOUS_DESCRIPTION_REPEAT_TIME: 471 | if obj in objets_to_describe: 472 | del objets_to_describe[obj] 473 | else: 474 | del self.described_objects_time[obj] 475 | 476 | # If no objects to describe, return 477 | if not objets_to_describe: 478 | self.move_head_to_search_for_objects() 479 | return 480 | 481 | # Generate the sentence with objects 482 | sentence = sentences[self.language]["I_see"] 483 | for obj, num in objets_to_describe.iteritems(): 484 | sentence += "%s %s" % (num, obj) 485 | 486 | # Describe 487 | self.services.ALTextToSpeech.say(sentence) 488 | 489 | current_time = time.time() 490 | 491 | # Record time when objects were described 492 | for obj, n in objets_to_describe.iteritems(): 493 | self.described_objects_time[obj] = current_time 494 | 495 | self.last_described_time = current_time 496 | self.move_head_state = 0 497 | 498 | 499 | ############################## 500 | # Web upload of custom model # 501 | ############################## 502 | 503 | FILE_TYPE_TO_FILENAME = { 504 | "config": CUSTOM_MODEL_CONFIG, 505 | "weights": CUSTOM_MODEL_WEIGHT, 506 | "names_fr": CUSTOM_MODEL_NAMES_FR, 507 | "names_en": CUSTOM_MODEL_NAMES_EN, 508 | } 509 | 510 | @log_exceptions 511 | def startUpload(self, file_type): 512 | if not file_type in self.FILE_TYPE_TO_FILENAME: 513 | raise RuntimeError("Unknown filetype") 514 | with open(self.FILE_TYPE_TO_FILENAME[file_type], "w") as out: 515 | out.truncate(0) 516 | self.uploaded_size[file_type] = 0 517 | 518 | @log_exceptions 519 | def uploadChunk(self, chunk, offset, file_type): 520 | if not file_type in self.FILE_TYPE_TO_FILENAME: 521 | raise RuntimeError("Unknown filetype") 522 | data = base64.b64decode(chunk) 523 | with open(self.FILE_TYPE_TO_FILENAME[file_type], "r+") as out: 524 | out.seek(offset) 525 | out.write(data) 526 | self.uploaded_size[file_type] += len(data) 527 | 528 | @log_exceptions 529 | def finishUpload(self, filesize, file_type): 530 | if not file_type in self.uploaded_size: 531 | raise RuntimeError("Unknown filetype") 532 | if filesize != self.uploaded_size[file_type]: 533 | self.logger.error("Uploaded file size differ: %s %s" 534 | % (filesize, self.uploaded_size[file_type])) 535 | if os.path.exists(self.FILE_TYPE_TO_FILENAME[file_type]): 536 | os.remove(self.FILE_TYPE_TO_FILENAME[file_type]) 537 | raise RuntimeError("Bad size") 538 | else: 539 | self.logger.info("Upload of %s is done" % file_type) 540 | 541 | 542 | if __name__ == "__main__": 543 | stk.runner.run_activity(Activity, SERVICE_NAME) 544 | -------------------------------------------------------------------------------- /scripts/explorer.py: -------------------------------------------------------------------------------- 1 | import random 2 | import time 3 | import math 4 | import qi 5 | import stk.runner 6 | import stk.events 7 | import stk.services 8 | import stk.worker 9 | from stk.logging import get_logger, log_exceptions_and_return, log_exceptions 10 | 11 | logger = qi.logging.Logger('com.softbankrobotics.deep_nao.explorer') 12 | 13 | # Sleep time at the end of the sonar loop 14 | SONAR_PERIOD = 0.1 15 | 16 | # Distance at which to stop in fron of obstacles 17 | OBSTACLE_DISTANCE = 0.4 18 | 19 | EXPLORATION_WALK_SPEED = 1.0 20 | 21 | NOWHERE, LEFT, RIGHT, FRONT = range(4) 22 | 23 | SERVICE_NAME="Explorer" 24 | 25 | @qi.singleThreaded() 26 | class Explorer(object): 27 | 28 | def __init__(self, qiapp): 29 | self.qiapp = qiapp 30 | self.wait_for_dependency_services() 31 | self.services = stk.services.ServiceCache(qiapp.session) 32 | self.logger = logger 33 | 34 | # We keep track of a very rough estimate of the current deviation from the 35 | # original direction to avoid the robot to try to turn right, then turn left, 36 | # then turn right etc. (i.e. to seem stuck in a corner of the room) 37 | self.deviation = 0 38 | self.runningFuture = None 39 | self.isRunning = False 40 | 41 | ## Handling loading of dependencies 42 | 43 | @qi.nobind 44 | @log_exceptions 45 | def wait_for_dependency_services(self): 46 | dependencies = [ 47 | "ALRobotPosture", 48 | "ALMemory", 49 | "ALMotion", 50 | ] 51 | for service in dependencies: 52 | logger.info("Waiting for service %s" % service) 53 | self.qiapp.session.waitForService(service) 54 | 55 | @qi.nobind 56 | @log_exceptions 57 | def get_obstacle_position(self, sonarLeft, sonarRight): 58 | left = sonarLeft < OBSTACLE_DISTANCE 59 | right = sonarRight < OBSTACLE_DISTANCE 60 | if left and right: 61 | return FRONT 62 | if left: 63 | return LEFT 64 | if right: 65 | return RIGHT 66 | return NOWHERE 67 | 68 | @log_exceptions 69 | def start(self): 70 | if self.isRunning: 71 | return 72 | self.mustStop = False 73 | self.runningFuture = qi.async(lambda: self.run()) 74 | 75 | 76 | @log_exceptions 77 | def stop(self): 78 | if not self.isRunning: 79 | return 80 | self.mustStop = True 81 | self.services.ALMotion.stopMove() 82 | if self.runningFuture is not None: 83 | self.runningFuture.wait() 84 | self.runningFuture = None 85 | 86 | @qi.nobind 87 | @log_exceptions 88 | def run(self): 89 | self.isRunning = True 90 | try: 91 | # To make sure arms are not in front of the sonar 92 | self.services.ALRobotPosture.goToPosture('StandInit', 0.75) 93 | 94 | while not self.mustStop: 95 | 96 | if not self.services.ALMotion.moveIsActive(): 97 | self.services.ALMotion.moveToward(EXPLORATION_WALK_SPEED, 0, 0) 98 | 99 | # Look ahead 100 | self.services.ALMotion.setAngles(['HeadYaw', 'HeadPitch'], [0.0, 0.0], 0.25) 101 | 102 | sonarLeft = self.services.ALMemory.getData("Device/SubDeviceList/US/Left/Sensor/Value") 103 | sonarRight = self.services.ALMemory.getData("Device/SubDeviceList/US/Right/Sensor/Value") 104 | 105 | p = self.get_obstacle_position(sonarLeft, sonarRight) 106 | if p != NOWHERE: 107 | self.on_obstacle(p) 108 | 109 | time.sleep(0.1) 110 | 111 | self.services.ALMotion.stopMove() 112 | finally: 113 | self.isRunning = False 114 | 115 | 116 | @qi.nobind 117 | @log_exceptions 118 | def on_obstacle(self, p): 119 | #time.sleep(WAIT_BEFORE_HANDLING) 120 | 121 | if p == LEFT: 122 | x, y, theta = 0, 0, -self.angle_side() 123 | elif p == RIGHT: 124 | x, y, theta = 0, 0, self.angle_side() 125 | elif p == FRONT: 126 | if self.deviation == 0: 127 | sign = random.choice([-1 ,1]) 128 | else: 129 | sign = math.copysign(1, self.deviation) 130 | x, y, theta = 0, 0, sign * random.uniform(math.pi/2, 3*math.pi/4) 131 | 132 | if self.mustStop: 133 | return 134 | 135 | headAngle = math.copysign(1, theta) * math.pi/4 136 | qi.async(self.services.ALMotion.angleInterpolationWithSpeed, 'HeadYaw', 137 | [headAngle, -headAngle], 0.25) 138 | 139 | if self.mustStop: 140 | return 141 | 142 | self.services.ALMotion.moveTo([(-0.1, 0, 0), (x, y, theta)]) 143 | 144 | self.deviation += math.copysign(1, theta) 145 | if abs(self.deviation) >= 4: 146 | self.deviation = 0 147 | 148 | 149 | # Function used to pick a random angle when an obstacle is detected on 150 | # the side. (We often want small angles, hence the weird triangular 151 | # distribution) 152 | @qi.nobind 153 | @log_exceptions 154 | def angle_side(self): 155 | return random.triangular(math.pi/6, math.pi/2, math.pi/4) 156 | 157 | 158 | if __name__ == "__main__": 159 | stk.runner.run_activity(Explorer, SERVICE_NAME) 160 | -------------------------------------------------------------------------------- /scripts/models.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import stk.runner 3 | if stk.runner.is_on_robot(): 4 | # Special: for NAOqi OS, pre-compiled cv2 is packaged 5 | # patch path for local cv2, should only be run when on robot 6 | sys.path.insert(0, "lib/python2.7/site-packages") 7 | 8 | import cv2 9 | import unicodedata 10 | import numpy 11 | 12 | def strip_accents(unicode_text): 13 | text = unicodedata.normalize('NFD', unicode_text).encode('ascii', 'ignore').decode("utf-8") 14 | return str(text) 15 | 16 | 17 | class MobilenetV2SSD(): 18 | 19 | def __init__(self, model_pbtxt, model_pb, object_names, language): 20 | self.net = cv2.dnn.readNetFromTensorflow(model_pbtxt, model_pb) 21 | self.classes = [] 22 | self.object_names = object_names 23 | with open(object_names[language]) as names: 24 | for line in names: 25 | self.classes.append(line.strip('\n').decode("utf-8")) 26 | self.last_visible_objects = {} 27 | 28 | def reload_classes(self, language): 29 | self.classes = [] 30 | with open(self.object_names[language]) as names: 31 | for line in names: 32 | self.classes.append(line.strip('\n').decode("utf-8")) 33 | 34 | def label_image_content(self, imageBGR): 35 | blob = cv2.dnn.blobFromImage(imageBGR, size=(300, 300), swapRB=False, crop=False) 36 | self.net.setInput(blob) 37 | outputs = self.net.forward() 38 | width = imageBGR.shape[1] 39 | height = imageBGR.shape[0] 40 | objects = {} 41 | for detection in outputs[0,0,:,:]: 42 | score = float(detection[2]) 43 | if score > 0.3: 44 | classID = int(detection[1]) 45 | left = int(detection[3] * width) 46 | top = int(detection[4] * height) 47 | right = int(detection[5] * width) 48 | bottom = int(detection[6] * height) 49 | cv2.rectangle(imageBGR, (left, top), (right, bottom), (23, 230, 210), thickness=2) 50 | classe = self.classes[classID - 1] 51 | if classe in objects: 52 | objects[classe] += 1 53 | else: 54 | objects[classe] = 1 55 | text = "{}: {:.4f}".format(strip_accents(classe), score) 56 | cv2.putText(imageBGR, text, (left, top + 15), 57 | cv2.FONT_HERSHEY_SIMPLEX, 0.5, (23, 230, 210), 1) 58 | self.last_visible_objects = objects 59 | return imageBGR 60 | 61 | 62 | class TinyYoloV4(): 63 | 64 | def __init__(self, model_cfg, model_weights, object_names, language): 65 | self.net = cv2.dnn.readNet(model_weights, model_cfg) 66 | self.classes = [] 67 | self.object_names = object_names 68 | self.layer_names = self.net.getUnconnectedOutLayers() 69 | print(self.layer_names) 70 | self.layer_names = ["yolo_30"] 71 | with open(object_names[language]) as names: 72 | for line in names: 73 | self.classes.append(line.strip('\n').decode("utf-8")) 74 | self.last_visible_objects = {} 75 | 76 | def reload_classes(language): 77 | self.classes = [] 78 | with open(self.object_names[language]) as names: 79 | for line in names: 80 | self.classes.append(line.strip('\n').decode("utf-8")) 81 | 82 | def label_image_content(self, imageBGR): 83 | blob = cv2.dnn.blobFromImage(imageBGR, 1/255.0, (416, 416), swapRB=False, crop=False) 84 | self.net.setInput(blob) 85 | outputs = self.net.forward(self.layer_names) 86 | color = (23, 230, 210) 87 | width = imageBGR.shape[1] 88 | height = imageBGR.shape[0] 89 | save_image = True 90 | objects = {} 91 | for output in outputs: 92 | for detection in output: 93 | scores = detection[5:] 94 | classID = numpy.argmax(scores) 95 | confidence = scores[classID] 96 | if confidence > 0.5: 97 | save_image = True 98 | #self.logger.info("Class: %s (%s)" % (classID, confidence)) 99 | box = detection[:4] * numpy.array([width, height, width, height]) 100 | (centerX, centerY, bwidth, bheight) = box.astype("int") 101 | x = int(centerX - (bwidth / 2)) 102 | y = int(centerY - (bheight / 2)) 103 | cv2.rectangle(imageBGR, (x, y), (x + bwidth, y + bheight), color, 2) 104 | classe = self.classes[classID] 105 | if classe in objects: 106 | objects[classe] += 1 107 | else: 108 | objects[classe] = 1 109 | text = "{}: {:.4f}".format(strip_accents(classe), confidence) 110 | cv2.putText(imageBGR, text, (x, y - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 1) 111 | 112 | self.last_visible_objects = objects 113 | return imageBGR 114 | -------------------------------------------------------------------------------- /scripts/stk/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | STK - A collection of libraries useful for making apps with NAOqi. 3 | """ 4 | -------------------------------------------------------------------------------- /scripts/stk/events.py: -------------------------------------------------------------------------------- 1 | """ 2 | stk.events.py 3 | 4 | Provides misc. wrappers for ALMemory and Signals (using the same syntax for 5 | handling both). 6 | """ 7 | 8 | __version__ = "0.1.1" 9 | 10 | __copyright__ = "Copyright 2015, Aldebaran Robotics" 11 | __author__ = 'ekroeger' 12 | __email__ = 'ekroeger@aldebaran.com' 13 | 14 | import qi 15 | 16 | 17 | def on(*keys): 18 | """Decorator for connecting a callback to one or several events. 19 | 20 | Usage: 21 | 22 | class O: 23 | @on("MyMemoryKey") 24 | def my_callback(self,value): 25 | print "I was called!", value 26 | 27 | o = O() 28 | events = EventsHelper() 29 | events.connect_decorators(o) 30 | 31 | After that, whenever MyMemoryKey is raised, o.my_callback will be called 32 | with the value. 33 | """ 34 | def decorator(func): 35 | func.__event_keys__ = keys 36 | return func 37 | return decorator 38 | 39 | 40 | class EventHelper(object): 41 | "Helper for ALMemory; takes care of event connections so you don't have to" 42 | 43 | def __init__(self, session=None): 44 | self.session = None 45 | self.almemory = None 46 | if session: 47 | self.init(session) 48 | self.handlers = {} # a handler is (subscriber, connections) 49 | self.subscriber_names = {} 50 | self.wait_value = None 51 | self.wait_promise = None 52 | 53 | def init(self, session): 54 | "Sets the NAOqi session, if it wasn't passed to the constructor" 55 | self.session = session 56 | self.almemory = session.service("ALMemory") 57 | 58 | def connect_decorators(self, obj): 59 | "Connects all decorated methods of target object." 60 | for membername in dir(obj): 61 | member = getattr(obj, membername) 62 | if hasattr(member, "__event_keys__"): 63 | for event in member.__event_keys__: 64 | self.connect(event, member) 65 | 66 | def connect(self, event, callback): 67 | """Connects an ALMemory event or signal to a callback. 68 | 69 | Note that some events trigger side effects in services when someone 70 | subscribes to them (such as WordRecognized). Those will *not* be 71 | triggered by this function, for those, use .subscribe(). 72 | """ 73 | if event not in self.handlers: 74 | if "." in event: 75 | # if we have more than one ".": 76 | service_name, signal_name = event.split(".") 77 | service = self.session.service(service_name) 78 | self.handlers[event] = (getattr(service, signal_name), []) 79 | else: 80 | # It's a "normal" ALMemory event. 81 | self.handlers[event] = ( 82 | self.almemory.subscriber(event).signal, []) 83 | signal, connections = self.handlers[event] 84 | connection_id = signal.connect(callback) 85 | connections.append(connection_id) 86 | return connection_id 87 | 88 | def subscribe(self, event, attachedname, callback): 89 | """Subscribes to an ALMemory event so as to notify providers. 90 | 91 | This is necessary for things like WordRecognized.""" 92 | connection_id = self.connect(event, callback) 93 | dummyname = "on_" + event.replace("/", "") 94 | self.almemory.subscribeToEvent(event, attachedname, dummyname) 95 | self.subscriber_names[event] = attachedname 96 | return connection_id 97 | 98 | def disconnect(self, event, connection_id=None): 99 | "Disconnects a connection, or all if no connection is specified." 100 | if event in self.handlers: 101 | signal, connections = self.handlers[event] 102 | if connection_id: 103 | if connection_id in connections: 104 | signal.disconnect(connection_id) 105 | connections.remove(connection_id) 106 | else: 107 | # Didn't specify a connection ID: remove all 108 | for connection_id in connections: 109 | signal.disconnect(connection_id) 110 | del connections[:] 111 | if event in self.subscriber_names: 112 | name = self.subscriber_names[event] 113 | self.almemory.unsubscribeToEvent(event, name) 114 | del self.subscriber_names[event] 115 | 116 | def clear(self): 117 | "Disconnect all connections" 118 | for event in list(self.handlers): 119 | self.disconnect(event) 120 | 121 | def get(self, key): 122 | "Gets ALMemory value." 123 | return self.almemory.getData(key) 124 | 125 | def get_int(self, key): 126 | "Gets ALMemory value, cast as int." 127 | try: 128 | return int(self.get(key)) 129 | except RuntimeError: 130 | # Key doesn't exist 131 | return 0 132 | except ValueError: 133 | # Key exists, but can't be parsed to int 134 | return 0 135 | 136 | def set(self, key, value): 137 | "Sets value of ALMemory key." 138 | return self.almemory.raiseEvent(key, value) 139 | 140 | def remove(self, key): 141 | "Remove key from ALMemory." 142 | try: 143 | self.almemory.removeData(key) 144 | except RuntimeError: 145 | pass 146 | 147 | def _on_wait_event(self, value): 148 | "Internal - callback for an event." 149 | if self.wait_promise: 150 | self.wait_promise.setValue(value) 151 | self.wait_promise = None 152 | 153 | def _on_wait_signal(self, *args): 154 | "Internal - callback for a signal." 155 | if self.wait_promise: 156 | self.wait_promise.setValue(args) 157 | self.wait_promise = None 158 | 159 | def cancel_wait(self): 160 | "Cancel the current wait (raises an exception in the waiting thread)" 161 | if self.wait_promise: 162 | self.wait_promise.setCanceled() 163 | self.wait_promise = None 164 | 165 | def wait_for(self, event, subscribe=False): 166 | """Block until a certain event is raised, and returns it's value. 167 | 168 | If you pass subscribe=True, ALMemory.subscribeToEvent will be called 169 | (sometimes necessary for side effects, i.e. WordRecognized). 170 | 171 | This will block a thread so you should avoid doing this too often! 172 | """ 173 | if self.wait_promise: 174 | # there was already a wait in progress, cancel it! 175 | self.wait_promise.setCanceled() 176 | self.wait_promise = qi.Promise() 177 | if subscribe: 178 | connection_id = self.subscribe(event, "EVENTHELPER", 179 | self._on_wait_event) 180 | elif "." in event: # it's a signal 181 | connection_id = self.connect(event, self._on_wait_signal) 182 | else: 183 | connection_id = self.connect(event, self._on_wait_event) 184 | try: 185 | result = self.wait_promise.future().value() 186 | finally: 187 | self.disconnect(event, connection_id) 188 | return result 189 | -------------------------------------------------------------------------------- /scripts/stk/logging.py: -------------------------------------------------------------------------------- 1 | """ 2 | stk.logging.py 3 | 4 | Utility library for logging with qi. 5 | """ 6 | 7 | __version__ = "0.1.2" 8 | 9 | __copyright__ = "Copyright 2015, Aldebaran Robotics" 10 | __author__ = 'ekroeger' 11 | __email__ = 'ekroeger@aldebaran.com' 12 | 13 | import functools 14 | import traceback 15 | 16 | import qi 17 | 18 | 19 | def get_logger(session, app_id): 20 | """Returns a qi logger object.""" 21 | logger = qi.logging.Logger(app_id) 22 | try: 23 | qicore = qi.module("qicore") 24 | log_manager = session.service("LogManager") 25 | provider = qicore.createObject("LogProvider", log_manager) 26 | log_manager.addProvider(provider) 27 | except RuntimeError: 28 | # no qicore, we're not running on a robot, it doesn't matter 29 | pass 30 | except AttributeError: 31 | # old version of NAOqi - logging will probably not work. 32 | pass 33 | return logger 34 | 35 | 36 | def log_exceptions(func): 37 | """Catches all exceptions in decorated method, and prints them. 38 | 39 | Attached function must be on an object with a "logger" member. 40 | """ 41 | @functools.wraps(func) 42 | def wrapped(self, *args): 43 | try: 44 | return func(self, *args) 45 | except Exception as exc: 46 | self.logger.error(traceback.format_exc()) 47 | raise exc 48 | return wrapped 49 | 50 | 51 | def log_exceptions_and_return(default_value): 52 | """If an exception occurs, print it and return default_value. 53 | 54 | Attached function must be on an object with a "logger" member. 55 | """ 56 | def decorator(func): 57 | @functools.wraps(func) 58 | def wrapped(self, *args): 59 | try: 60 | return func(self, *args) 61 | except Exception: 62 | self.logger.error(traceback.format_exc()) 63 | return default_value 64 | return wrapped 65 | return decorator 66 | -------------------------------------------------------------------------------- /scripts/stk/runner.py: -------------------------------------------------------------------------------- 1 | """ 2 | stk.runner.py 3 | 4 | A helper library for making simple standalone python scripts as apps. 5 | 6 | Wraps some NAOqi and system stuff, you could do all this by directly using the 7 | Python SDK, these helper functions just isolate some frequently used/hairy 8 | bits so you don't have them mixed in your logic. 9 | """ 10 | 11 | __version__ = "0.1.3" 12 | 13 | __copyright__ = "Copyright 2015, Aldebaran Robotics" 14 | __author__ = 'ekroeger' 15 | __email__ = 'ekroeger@aldebaran.com' 16 | 17 | import sys 18 | import qi 19 | from distutils.version import LooseVersion 20 | 21 | # 22 | # Helpers for making sure we have a robot to connect to 23 | # 24 | 25 | 26 | def check_commandline_args(description): 27 | "Checks whether command-line parameters are enough" 28 | import argparse 29 | parser = argparse.ArgumentParser(description=description) 30 | parser.add_argument('--qi-url', help='connect to specific NAOqi instance') 31 | 32 | args = parser.parse_args() 33 | return args 34 | 35 | 36 | def is_on_robot(): 37 | "Returns whether this is being executed on an Aldebaran robot." 38 | import platform 39 | return "aldebaran" in platform.platform() 40 | 41 | 42 | def get_debug_robot(): 43 | "Returns IP address of debug robot, complaining if not found" 44 | try: 45 | import qiq.config 46 | qiqrobot = qiq.config.defaultHost() 47 | if qiqrobot: 48 | robot = raw_input( 49 | "connect to which robot? (default is {0}) ".format(qiqrobot)) 50 | if robot: 51 | return robot 52 | else: 53 | return qiqrobot 54 | else: 55 | print "qiq found, but it has no default robot configured." 56 | except ImportError: 57 | # qiq not installed 58 | print "qiq not installed (you can use it to set a default robot)." 59 | return raw_input("connect to which robot? ") 60 | 61 | 62 | def init(qi_url=None): 63 | "Returns a QiApplication object, possibly with interactive input." 64 | if qi_url: 65 | sys.argv.extend(["--qi-url", qi_url]) 66 | else: 67 | args = check_commandline_args('Run the app.') 68 | if bool(args.qi_url): 69 | qi_url = args.qi_url 70 | elif not is_on_robot(): 71 | print "no --qi-url parameter given; interactively getting debug robot." 72 | debug_robot = get_debug_robot() 73 | if debug_robot: 74 | sys.argv.extend(["--qi-url", debug_robot]) 75 | qi_url = debug_robot 76 | else: 77 | raise RuntimeError("No robot, not running.") 78 | 79 | qiapp = None 80 | sys.argv[0] = str(sys.argv[0]) 81 | 82 | # In versions bellow 2.3, look for --qi-url in the arguemnts and call accordingly the Application 83 | if qi_url and LooseVersion(qi.__version__) < LooseVersion("2.3"): 84 | position = 0 85 | qiapp = qi.Application(url="tcp://"+qi_url+":9559") 86 | # In versions greater than 2.3 the ip can simply be passed through argv[0] 87 | else: 88 | # In some environments sys.argv[0] has unicode, which qi rejects 89 | qiapp = qi.Application() 90 | 91 | qiapp.start() 92 | return qiapp 93 | 94 | 95 | # Main runner 96 | 97 | def run_activity(activity_class, service_name=None): 98 | """Instantiate the given class, and runs it. 99 | 100 | The given class must take a qiapplication object as parameter, and may also 101 | have on_start and on_stop methods, that will be called before and after 102 | running it.""" 103 | qiapp = init() 104 | activity = activity_class(qiapp) 105 | service_id = None 106 | 107 | try: 108 | # if it's a service, register it 109 | if service_name: 110 | # Note: this will fail if there is already a service. Unregistering 111 | # it would not be a good practice, because it's process would still 112 | # be running. 113 | service_id = qiapp.session.registerService(service_name, activity) 114 | 115 | if hasattr(activity, "on_start"): 116 | def handle_on_start_done(on_start_future): 117 | "Custom callback, for checking errors" 118 | if on_start_future.hasError(): 119 | try: 120 | msg = "Error in on_start(), stopping application: %s" \ 121 | % on_start_future.error() 122 | if hasattr(activity, "logger"): 123 | activity.logger.error(msg) 124 | else: 125 | print msg 126 | finally: 127 | qiapp.stop() 128 | qi.async(activity.on_start).addCallback(handle_on_start_done) 129 | 130 | # Run the QiApplication, which runs until someone calls qiapp.stop() 131 | qiapp.run() 132 | 133 | finally: 134 | # Cleanup 135 | if hasattr(activity, "on_stop"): 136 | # We need a qi.async call so that if the class is single threaded, 137 | # it will wait for callbacks to be finished. 138 | qi.async(activity.on_stop).wait() 139 | if service_id: 140 | qiapp.session.unregisterService(service_id) 141 | 142 | 143 | def run_service(service_class, service_name=None): 144 | """Instantiate the given class, and registers it as a NAOqi service. 145 | 146 | The given class must take a qiapplication object as parameter, and may also 147 | have on_start and on_stop methods, that will be called before and after 148 | running it. 149 | 150 | If the service_name parameter is not given, the classes' name will be used. 151 | """ 152 | if not service_name: 153 | service_name = service_class.__name__ 154 | run_activity(service_class, service_name) 155 | -------------------------------------------------------------------------------- /scripts/stk/services.py: -------------------------------------------------------------------------------- 1 | """ 2 | stk.services.py 3 | 4 | Syntactic sugar for accessing NAOqi services. 5 | """ 6 | 7 | __version__ = "0.1.2" 8 | 9 | __copyright__ = "Copyright 2015, Aldebaran Robotics" 10 | __author__ = 'ekroeger' 11 | __email__ = 'ekroeger@aldebaran.com' 12 | 13 | 14 | class ServiceCache(object): 15 | "A helper for accessing NAOqi services." 16 | 17 | def __init__(self, session=None): 18 | self.session = None 19 | self.services = {} 20 | if session: 21 | self.init(session) 22 | 23 | def init(self, session): 24 | "Sets the session object, if it wasn't passed to constructor." 25 | self.session = session 26 | 27 | def __getattr__(self, servicename): 28 | "We overload this so (instance).ALMotion returns the service, or None." 29 | if (not servicename in self.services) or ( 30 | servicename == "ALTabletService"): 31 | # ugly hack: never cache ALtabletService, always ask for a new one 32 | if servicename.startswith("__"): 33 | # Behave like a normal python object for those 34 | raise AttributeError 35 | try: 36 | self.services[servicename] = self.session.service(servicename) 37 | except RuntimeError: # Cannot find service 38 | self.services[servicename] = None 39 | return self.services[servicename] 40 | -------------------------------------------------------------------------------- /scripts/stk/worker.py: -------------------------------------------------------------------------------- 1 | import time 2 | import functools 3 | import Queue 4 | import threading 5 | import qi 6 | 7 | 8 | logger = qi.logging.Logger('deep_nao.worker') 9 | 10 | 11 | class Worker: 12 | """Worker allows you to queue tasks on its list of tasks, and then it 13 | execute them one after another until its task list is empty. 14 | 15 | """ 16 | 17 | def __init__(self, name): 18 | self.tasks = Queue.Queue(0) 19 | self.running = False 20 | self.lock = threading.Lock() 21 | self.name = name 22 | 23 | def __repr__(self): 24 | return "Worker(%s)" % self.name 25 | 26 | def queue_size(self): 27 | return self.tasks.qsize() 28 | 29 | def add(self, function): 30 | """Add a new task on the task list""" 31 | logger.verbose("%s add task." % self) 32 | self.tasks.put(function) 33 | with self.lock: 34 | if not self.running: 35 | self.running = True 36 | qi.async(self._run) 37 | 38 | def _run(self): 39 | """Excecute the tasks on the task list""" 40 | logger.verbose("%s start running." % self) 41 | try: 42 | while True: 43 | with self.lock: 44 | try: 45 | function = self.tasks.get_nowait() 46 | except Queue.Empty: 47 | self.running = False 48 | break 49 | try: 50 | function() 51 | finally: 52 | self.tasks.task_done() 53 | logger.verbose("%s done running." % self) 54 | except: 55 | logger.error(traceback.format_exc()) 56 | 57 | workers = {} 58 | 59 | @classmethod 60 | def get_worker(cls, worker_id): 61 | """Get or create a Worker indentified with worker_id.""" 62 | if worker_id not in cls.workers: 63 | cls.workers[worker_id] = cls(worker_id) 64 | return cls.workers[worker_id] 65 | 66 | 67 | def _cancel(promise): 68 | promise.setCanceled() 69 | 70 | 71 | def async(worker_id, expire_delay=-1, max_queue=-1): 72 | def wrap(method): 73 | @functools.wraps(method) 74 | def wrapper(self, *args, **kwds): 75 | promise = qi.Promise(_cancel) 76 | future = promise.future() 77 | queued_tmstp = time.time() 78 | 79 | def runner(): 80 | if not promise.future().isCanceled(): 81 | try: 82 | exec_tmstp = time.time() 83 | if expire_delay != -1 and exec_tmstp - queued_tmstp > expire_delay: 84 | logger.info("Expired %s%s, do not exec" % (method.__name__, args)) 85 | promise.setError("Could not exec method, it expired") 86 | return 87 | res = method(self, *args, **kwds) 88 | if not promise.future().isCanceled(): 89 | promise.setValue(res) 90 | except Exception, e: 91 | logger.error(traceback.format_exc()) 92 | if not promise.future().isCanceled(): 93 | promise.setError(str(e)) 94 | worker = Worker.get_worker(worker_id) 95 | if max_queue == -1 or max_queue >= worker.queue_size(): 96 | worker.add(runner) 97 | else: 98 | promise.setError("Could not perform task. Queue is full.") 99 | if not hasattr(self, "_future"): 100 | self._futures = [] 101 | self._futures.append(future) 102 | return future 103 | return wrapper 104 | return wrap 105 | -------------------------------------------------------------------------------- /scripts/stop_behavior.py: -------------------------------------------------------------------------------- 1 | import qi 2 | import stk.runner 3 | import stk.events 4 | import stk.services 5 | from stk.logging import get_logger, log_exceptions_and_return, log_exceptions 6 | 7 | SERVICE_NAME="StopBehaviorOnRobotFall" 8 | 9 | @qi.singleThreaded() 10 | class Activity(object): 11 | 12 | APP_ID = "com.sotfbankrobotics.stop_behavior_on_fall" 13 | 14 | 15 | def __init__(self, qiapp): 16 | self.qiapp = qiapp 17 | self.events = stk.events.EventHelper(qiapp.session) 18 | self.s = stk.services.ServiceCache(qiapp.session) 19 | self.logger = get_logger(qiapp.session, self.APP_ID) 20 | self.wait_for_dependency_services() 21 | self.behavior_running = set([]) 22 | self.behavior_list = [ 23 | "deep_nao/behavior", 24 | ] 25 | self.robot_posture_check_task = qi.PeriodicTask() 26 | self.robot_posture_check_task.setCallback(self.check_robot_posture_after_fall) 27 | self.robot_posture_check_task.compensateCallbackTime(False) 28 | self.robot_posture_check_task.setUsPeriod(500) 29 | self.s.ALMemory.raiseEvent("DeepNao/RobotFallen", "False") 30 | 31 | @qi.nobind 32 | def wait_for_dependency_services(self): 33 | dependencies = [ 34 | "ALBehaviorManager", 35 | "ALMemory", 36 | ] 37 | for service in dependencies: 38 | self.logger.info("Waiting for service %s" % service) 39 | self.qiapp.session.waitForService(service) 40 | 41 | def add_behavior_to_stop(self, behavior): 42 | self.behavior_list.append(behavior) 43 | 44 | @qi.nobind 45 | def on_start(self): 46 | self.events.connect_decorators(self) 47 | self.started_link = self.s.ALBehaviorManager.behaviorStarted.connect(self.on_behavior_started) 48 | self.stopped_link = self.s.ALBehaviorManager.behaviorStopped.connect(self.on_behavior_stopped) 49 | 50 | @qi.nobind 51 | def on_stop(self): 52 | "Cleanup" 53 | self.logger.info("Application finished.") 54 | self.events.clear() 55 | self.s.ALBehaviorManager.behaviorStarted.disconnect(self.started_link) 56 | self.s.ALBehaviorManager.behaviorStopped.disconnect(self.stopped_link) 57 | 58 | @stk.events.on("robotHasFallen") 59 | def on_robot_has_fallen(self, unused): 60 | self.s.ALMemory.raiseEvent("DeepNao/RobotFallen", "True") 61 | for behavior in self.behavior_running: 62 | self.logger.info("STOPPING: %s" % behavior) 63 | self.s.ALBehaviorManager.stopBehavior(behavior) 64 | self.robot_posture_check_task.start(True) 65 | 66 | def on_behavior_started(self, behavior): 67 | self.logger.info("start %s" % behavior) 68 | if behavior in self.behavior_list: 69 | self.behavior_running.add(behavior) 70 | 71 | def on_behavior_stopped(self, behavior): 72 | self.logger.info("stop %s" % behavior) 73 | if behavior in self.behavior_list: 74 | self.behavior_running.remove(behavior) 75 | 76 | @qi.nobind 77 | @log_exceptions 78 | def check_robot_posture_after_fall(self): 79 | posture = self.s.ALRobotPosture.getPosture() 80 | if posture in [ 81 | "Crouch", 82 | "Sit", 83 | "SitRelax", 84 | "Stand", 85 | "StandInit", 86 | "StandZero" 87 | ]: 88 | self.robot_posture_check_task.stop() 89 | self.s.ALMemory.raiseEvent("DeepNao/RobotFallen", "False") 90 | 91 | 92 | if __name__ == "__main__": 93 | stk.runner.run_activity(Activity, SERVICE_NAME) 94 | -------------------------------------------------------------------------------- /scripts/translations.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -* 2 | 3 | sentences = { 4 | "English": { 5 | "ready": "I'm ready, show me objects to recognize.", 6 | "see_no_objects": "I don't see any objects", 7 | "I_see": "I see ", 8 | "error_custom": "I can't load the custom model" 9 | }, 10 | "French" : { 11 | "ready": "Je suis pret, montre moi des objets à reconnaitre.", 12 | "see_no_objects": "Je ne vois aucun objet", 13 | "I_see": "Je vois ", 14 | "error_custom": "Je ne peux pas utiliser le modèle personnalisé" 15 | } 16 | } 17 | --------------------------------------------------------------------------------