├── .gitattributes ├── .gitignore ├── LICENSE.md ├── README.md ├── Single_Tello_Test ├── README.md ├── command.txt ├── stats.py ├── tello.py └── tello_test.py ├── TelloPython_FAQ(CH).txt ├── TelloPython_FAQ.txt ├── Tello_Video ├── LICENSE.md ├── README.md ├── h264decoder │ ├── CMakeLists.txt │ ├── Linux │ │ └── libh264decoder.so │ ├── Mac │ │ └── libh264decoder.so │ ├── Windows │ │ ├── x64 │ │ │ └── libh264decoder.pyd │ │ └── x86 │ │ │ └── libh264decoder.pyd │ ├── h264decoder.cpp │ ├── h264decoder.hpp │ ├── h264decoder_python.cpp │ └── readme.md ├── img │ └── readme.md ├── install │ ├── Linux │ │ ├── linux_install.sh │ │ └── linux_uninstall.sh │ ├── Mac │ │ ├── mac_install.sh │ │ └── mac_uninstall.sh │ └── Windows │ │ ├── install.bat │ │ └── uninstall.bat ├── main.py ├── tello.py └── tello_control_ui.py ├── Tello_Video_With_Pose_Recognition ├── LICENSE.md ├── README.md ├── h264decoder │ ├── CMakeLists.txt │ ├── Linux │ │ └── libh264decoder.so │ ├── Mac │ │ └── libh264decoder.so │ ├── Windows │ │ ├── x64 │ │ │ └── libh264decoder.pyd │ │ └── x86 │ │ │ └── libh264decoder.pyd │ ├── h264decoder.cpp │ ├── h264decoder.hpp │ ├── h264decoder_python.cpp │ └── readme.md ├── img │ └── readme.md ├── install │ ├── Linux │ │ ├── linux_install.sh │ │ └── linux_uninstall.sh │ ├── Mac │ │ ├── mac_install.sh │ │ └── mac_uninstall.sh │ └── Windows │ │ ├── install.bat │ │ └── uninstall.bat ├── main.py ├── model │ ├── getModels.bat │ ├── getModels.sh │ └── pose │ │ └── mpi │ │ └── pose_deploy_linevec_faster_4_stages.prototxt ├── tello.py ├── tello_control_ui.py └── tello_pose.py ├── doc ├── readme.pdf └── ╦╡├≈.pdf ├── tello_state.py └── tello_video_dll(ForWin64).zip /.gitattributes: -------------------------------------------------------------------------------- 1 | E:\181115\Tello-Python 2 | $ cat ./.gitattributes 3 | *.bat -text -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.DS_Store 3 | *.dat 4 | *.caffemodel 5 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Sample code is offered under MIT License (See below). 2 | 3 | Copyright 2018 Ryze 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a 6 | copy of this software and associated documentation files (the "Software"), 7 | to deal in the Software without restriction, including without limitation 8 | the rights to use, copy, modify, merge, publish, distribute, sublicense, 9 | and/or sell copies of the Software, and to permit persons to whom the 10 | Software is furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included 13 | in all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS 16 | OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 18 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 19 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 20 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 21 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Tello-Python 2 | 3 | ## Introduction 4 | 5 | This is a collection of python-based sample code that interact with the Ryze Tello drone. 6 | 7 | ## Project Description 8 | 9 | This toolkit contains three sample programs based on tello sdk and python2.7,including Single_Tello_Test, Tello_Video, and Tello_Video (With_Pose_Recognition). There is also a program file named tello_state.py. 10 | 11 | - **Single_Tello_Test** 12 | 13 | In Single_Tello_Test,You can design a series of command combinations by writing a txt script to let tello execute a series of actions you have designed. This program can also be used as a command set test tool for tello. 14 | 15 | - **Tello_Video** 16 | 17 | In Tello_Video，You can receive the video stream data from tello, decode the video through the h264 decoding library, and display it on a GUI interface based on Tkinter and PIL. In addition, it also supports a control panel that can operate tello. This sample code provides an example of receiving and processing and getting the correct video data. The source code of the h264 decoding library is also provided in the package, which can be used for your reference. 18 | 19 | - **Tello_Video(With_Pose_Recognition)** 20 | 21 | Tello_Video_With_Pose_Recognition is an application version modified from Tello_Video.It uses the decoded video data，and everytime extract a single frame image for pose recognition operation , and binds the specific posture and aircraft control commands to realize the pose control of Tello.This code is mainly used as an application case for utilizing the decoded video data of tello for image processing. 22 | 23 | - **Tello_state.py** 24 | 25 | Tello_state.py can read the various status data of tello, and can be used as a tool to debug and view the status of tello. 26 | 27 | ## Environmental configuration 28 | 29 | The sample codes above are based on python2.7.There is no need to install additional third-party libraries for running Single_Tello_Test and tello_state.py.For Tello_Video and Tello_Video (With_Pose_Recognition), you need to install a series of third-party libraries. Therefore, in these two folders, a one-click installation script (based on windows32/64, linux and macos) is provided, which can facilitate you with installing all relevant dependencies. 30 | 31 | Specific to the content and description of each package, you can refer to the readme file in the related folder. 32 | 33 | ## Contact Information 34 | 35 | If you have any questions about this sample code and the installation, please feel free to contact me. You can communicate with me by sending e-mail to sdk@ryzerobotics.com. 36 | And recently we have committed a new FAQ file under the 'Tello-Python'.If you have any questions,you can firstly refer to it . 37 | 38 | ## About Multi-Tello-Formation 39 | 40 | Please refer to github repository https://github.com/TelloSDK/Multi-Tello-Formation. 41 | This is a python program that enable the function of multi-tello swarms. 42 | 43 | 44 | 45 | -------------------------------------------------------------------------------- /Single_Tello_Test/README.md: -------------------------------------------------------------------------------- 1 | # Single_Tello_Test 2 | ## Step1 3 | Write the command set to be run in command.txt, for example:： 4 | ``` 5 | command 6 | takeoff 7 | land 8 | ``` 9 | ## Step2 10 | The script will automatically send a command to Tello. After receiving the reply from the previous command, the next command will be sent immediately. 11 | To add a delay, you can use the Delay command and the script will automatically delay. The unit of delay is seconds, which can be given to decimals. 12 | ``` 13 | delay 5 14 | ``` 15 | ## Step3 16 | Run the script 17 | ``` 18 | python tello_test.py command.txt 19 | ``` 20 | The command window will type each instruction and its reply. After the execution is finished, the commands will be stored in the log folder to name the test end time. 21 | -------------------------------------------------------------------------------- /Single_Tello_Test/command.txt: -------------------------------------------------------------------------------- 1 | command 2 | takeoff 3 | delay 5 4 | land 5 | -------------------------------------------------------------------------------- /Single_Tello_Test/stats.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | 3 | class Stats: 4 | def __init__(self, command, id): 5 | self.command = command 6 | self.response = None 7 | self.id = id 8 | 9 | self.start_time = datetime.now() 10 | self.end_time = None 11 | self.duration = None 12 | 13 | def add_response(self, response): 14 | self.response = response 15 | self.end_time = datetime.now() 16 | self.duration = self.get_duration() 17 | # self.print_stats() 18 | 19 | def get_duration(self): 20 | diff = self.end_time - self.start_time 21 | return diff.total_seconds() 22 | 23 | def print_stats(self): 24 | print '\nid: %s' % self.id 25 | print 'command: %s' % self.command 26 | print 'response: %s' % self.response 27 | print 'start time: %s' % self.start_time 28 | print 'end_time: %s' % self.end_time 29 | print 'duration: %s\n' % self.duration 30 | 31 | def got_response(self): 32 | if self.response is None: 33 | return False 34 | else: 35 | return True 36 | 37 | def return_stats(self): 38 | str = '' 39 | str += '\nid: %s\n' % self.id 40 | str += 'command: %s\n' % self.command 41 | str += 'response: %s\n' % self.response 42 | str += 'start time: %s\n' % self.start_time 43 | str += 'end_time: %s\n' % self.end_time 44 | str += 'duration: %s\n' % self.duration 45 | return str -------------------------------------------------------------------------------- /Single_Tello_Test/tello.py: -------------------------------------------------------------------------------- 1 | import socket 2 | import threading 3 | import time 4 | from stats import Stats 5 | 6 | class Tello: 7 | def __init__(self): 8 | self.local_ip = '' 9 | self.local_port = 8889 10 | self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # socket for sending cmd 11 | self.socket.bind((self.local_ip, self.local_port)) 12 | 13 | # thread for receiving cmd ack 14 | self.receive_thread = threading.Thread(target=self._receive_thread) 15 | self.receive_thread.daemon = True 16 | self.receive_thread.start() 17 | 18 | self.tello_ip = '192.168.10.1' 19 | self.tello_port = 8889 20 | self.tello_adderss = (self.tello_ip, self.tello_port) 21 | self.log = [] 22 | 23 | self.MAX_TIME_OUT = 15.0 24 | 25 | def send_command(self, command): 26 | """ 27 | Send a command to the ip address. Will be blocked until 28 | the last command receives an 'OK'. 29 | If the command fails (either b/c time out or error), 30 | will try to resend the command 31 | :param command: (str) the command to send 32 | :param ip: (str) the ip of Tello 33 | :return: The latest command response 34 | """ 35 | self.log.append(Stats(command, len(self.log))) 36 | 37 | self.socket.sendto(command.encode('utf-8'), self.tello_adderss) 38 | print 'sending command: %s to %s' % (command, self.tello_ip) 39 | 40 | start = time.time() 41 | while not self.log[-1].got_response(): 42 | now = time.time() 43 | diff = now - start 44 | if diff > self.MAX_TIME_OUT: 45 | print 'Max timeout exceeded... command %s' % command 46 | # TODO: is timeout considered failure or next command still get executed 47 | # now, next one got executed 48 | return 49 | print 'Done!!! sent command: %s to %s' % (command, self.tello_ip) 50 | 51 | def _receive_thread(self): 52 | """Listen to responses from the Tello. 53 | 54 | Runs as a thread, sets self.response to whatever the Tello last returned. 55 | 56 | """ 57 | while True: 58 | try: 59 | self.response, ip = self.socket.recvfrom(1024) 60 | print('from %s: %s' % (ip, self.response)) 61 | 62 | self.log[-1].add_response(self.response) 63 | except socket.error, exc: 64 | print "Caught exception socket.error : %s" % exc 65 | 66 | def on_close(self): 67 | pass 68 | # for ip in self.tello_ip_list: 69 | # self.socket.sendto('land'.encode('utf-8'), (ip, 8889)) 70 | # self.socket.close() 71 | 72 | def get_log(self): 73 | return self.log 74 | 75 | -------------------------------------------------------------------------------- /Single_Tello_Test/tello_test.py: -------------------------------------------------------------------------------- 1 | from tello import Tello 2 | import sys 3 | from datetime import datetime 4 | import time 5 | 6 | start_time = str(datetime.now()) 7 | 8 | file_name = sys.argv[1] 9 | 10 | f = open(file_name, "r") 11 | commands = f.readlines() 12 | 13 | tello = Tello() 14 | for command in commands: 15 | if command != '' and command != '\n': 16 | command = command.rstrip() 17 | 18 | if command.find('delay') != -1: 19 | sec = float(command.partition('delay')[2]) 20 | print 'delay %s' % sec 21 | time.sleep(sec) 22 | pass 23 | else: 24 | tello.send_command(command) 25 | 26 | log = tello.get_log() 27 | 28 | out = open('log/' + start_time + '.txt', 'w') 29 | for stat in log: 30 | stat.print_stats() 31 | str = stat.return_stats() 32 | out.write(str) 33 | -------------------------------------------------------------------------------- /TelloPython_FAQ(CH).txt: -------------------------------------------------------------------------------- 1 | FAQ about Tello-Python (常见问题解答) 2 | 3 | Q1:在windows上，运行tello_video需要哪些依赖项/怎么手动？ 4 | 5 | A1:在windows上，运行tello_video，首先，需要下载python2.7， 6 | a.python2.7下载链接： 7 | win64: https://www.python.org/ftp/python/2.7.15/python-2.7.15.amd64.msi 8 | win32: https://www.python.org/ftp/python/2.7.15/python-2.7.15.msi 9 | 10 | b.然后可以通过pip安装四个依赖项，分别是： （pip的安装方式请自行google） 11 | numpy,opencv-python,pillow,matplotib。安装指令分别为: 12 | numpy: python -m pip install numpy 13 | matplotlib: python -m pip install matplotlib 14 | opencv-python: python -m pip install -v opencv-python==3.4.2.17 15 | pillow: python -m pip install pillow 16 | 17 | c.接下来，需要下载并安装boost，目的是获得一个叫boost_python27-vc120-mt-x**-1_68.dll的动态链接库 18 | boost下载链接: 19 | win64: https://nchc.dl.sourceforge.net/project/boost/boost-binaries/1.68.0/boost_1_68_0-msvc-12.0-64.exe 20 | win32: https://excellmedia.dl.sourceforge.net/project/boost/boost-binaries/1.68.0/boost_1_68_0-msvc-12.0-32.exe 21 | 下载完毕之后双击安装包进行安装，安装完成后，找到boost_python27-vc120-mt-x**-1_68.dll文件('**'部分取决你的电脑位数),放置到.\Python27\Lib\site-packages路径下。 22 | 23 | d.接下来，需要下载ffmpeg，目的是得到包括avcodec-58.dll在内的多个dll文件。 24 | ffmpeg下载链接： 25 | win64: https://ffmpeg.zeranoe.com/builds/win64/shared/ffmpeg-20180825-844ff49-win64-shared.zip 26 | win32: https://ffmpeg.zeranoe.com/builds/win32/shared/ffmpeg-20180825-844ff49-win32-shared.zip 27 | 下载完毕之后，解压文件，并将./bin 路径下的所有.dll文件都放置到.\Python27\Lib\site-packages路径下。 28 | 29 | e.接下来，需要下载vs2013编译器， 30 | vs2013编译器下载链接： 31 | win64: https://download.microsoft.com/download/2/E/6/2E61CFA4-993B-4DD4-91DA-3737CD5CD6E3/vcredist_x64.exe 32 | win32: https://download.microsoft.com/download/2/E/6/2E61CFA4-993B-4DD4-91DA-3737CD5CD6E3/vcredist_x86.exe 33 | 下载完毕后，运行安装包，安装程序。 34 | 35 | f.将Tello-Python程序包中的libh264decoder复制到.\Python27\Lib\site-packages路径下。 36 | h264decoder.pyd路径： 37 | win64: .\Tello-Python\Tello_Video\h264decoder\Windows\x64\libh264decoder.pyd 38 | win32: .\Tello-Python\Tello_Video\h264decoder\Windows\x86\libh264decoder.pyd 39 | 40 | 最后，在最新提交的commit中,提供了一个tello_video_dll.zip的压缩包（仅适用于win64），里面包括所需的与boost，ffmpeg，libh264decoder，vs2013编译器相关的文件。 41 | 首先，双击运行vcredist_x64.exe（vs2013编译器的安装包）进行安装。其次，将文件夹中的剩余文件都复制到.\Python27\Lib\site-packages路径下。 42 | （通过该zip,你可以省去在网上下载和部分安装的环节，但注意这个zip里面的文件仅适用于win64!另外,像numpy,matplotlib,pillow和opencv-python你仍需要用pip进行安装。） 43 | 44 | 45 | Q2:windows上的一键安装脚本(install.bat)在我的电脑上无法正常运行。 46 | 47 | A2:目前提供的install.bat由于包含MD5校验等环节，对于部分用户电脑不是很兼容，后续会提供一个稳定安装的版本。如果无法正常运行自动安装脚本，请参考 48 | Q1中的手动安装依赖项的方式。 49 | 50 | 51 | Q3:install_manual.bat有什么作用？ 52 | 53 | A3:该脚本与install.bat在完成的任务上并无区别，区别主要在于会在每一个安装项目前进行中断，并提示用户是否安装。 54 | 目前该脚本并不稳定，建议您适用install.bat或者采用Q1中的手动安装的方式。 55 | 56 | 57 | Q4:在windows/linux/mac中，我的所有依赖项均已经全部安装完成，运行python main.py之后也没有发生报错，但是运行程序后GUI并没有弹出视频流的画面，也无法使用 58 | Snapshot按钮进行拍照。 59 | 60 | A4:正常情况下，在已经连接tello wifi的情况下，运行程序后GUI会弹出视频流的播放窗口，并且可以正常使用snapshot按钮进行拍照，如果没有正常运行(实际效果参照https://www.youtube.com/watch?v=kcXN7CYgQ0g)， 61 | 请考虑并检查以下几种情况： 62 | a.你的PC设备是否已经正常连接到你的tello 的wifi； 63 | b.检查你的PC的网络防火墙，可能防火墙会对视频流数据进行拦截； 64 | c.进一步检查你的网络设置，因为tello是对PC的11111端口进行视频流数据的推送的，所以请检查你的网络设置中是否禁用了11111UDP端口，或者对11111端口的UDP数据进行了拦截。 65 | 如果有，请取消禁用； 66 | d.极少数情况下，可能是PC发送的'streamon'指令发生丢包，没有被tello接收，针对这种情况，请尝试重新运行程序。 67 | 68 | 69 | Q5:运行程序后，无法正常使用command panel中的按键响应功能。 70 | 71 | A5:请确认鼠标不要点击command panel外的地方，如果点击了，请用鼠标重新点击command panel界面，再尝试按键响应功能。 72 | 73 | 74 | Q6:tello EDU版本是否提供python 版本的多机编队例程？ 75 | 76 | A6:请参考https://github.com/TelloSDK/Multi-Tello-Formation，并阅读其README.md文档，此外，该例程内提供了可直接执行文件版本(.exe)， 77 | 请参照Tello-Swarm中的userguide使用。 78 | 79 | Q7:libh264decoder的作用是什么？ 80 | 81 | A7:libh264decoder是根据.\Tello-Python\Tello_Video\h264decoder文件夹中的h264decoder.cpp，h264decoder.hpp和h264decoder_python.cpp生成的 82 | 动态链接库，其作用是对tello向PC设备传输的h264编码格式的视频流进行解码。在windows上是以libh264decoder.pyd的形式体现，在linux和macos 83 | 上以libh264decoder.so的形式体现。 84 | Windows/Linux/Mac上的libh264decoder动态链接库均提供在.\Tello-Python\Tello_Video\h264decoder路径下。 85 | 在Linux和Mac的一键脚本安装过程中，libh264decoder.so通过源码实时编译生成。 86 | 在Windows的一键脚本安装过程中，libh264decoder.pyd直接从\Tello_Video\h264decoder\Windows中复制，并被放置到.\Python27\Lib\site-packages中。 87 | 88 | Q8:Tello EDU能否在station模式下对外传输视频流(对Tello EDU 使用了ap 'ssid' 'password'指令)？ 89 | 90 | A8:目前该功能并不被支持。 91 | 92 | Q9:tello SDK接口所支持的UDP通信的频段和信道？ 93 | 94 | A9:目前只支持2.4GHz,支持的信道包括0-11。 95 | 96 | 如果有其他问题，欢迎提问或补充! -------------------------------------------------------------------------------- /TelloPython_FAQ.txt: -------------------------------------------------------------------------------- 1 | FAQ about Tello-Python 2 | 3 | Q1:On windows, what dependencies are needed to run tello_video/how to install it manually? 4 | 5 | A1:On windows, to run tello_video, first, you need to download python2.7, 6 | a.Python2.7 download link: 7 | win64: https://www.python.org/ftp/python/2.7.15/python-2.7.15.amd64.msi 8 | win32: https://www.python.org/ftp/python/2.7.15/python-2.7.15.msi 9 | 10 | b.Then you can install these four dependencies through pip, ('way to install pip' please google by yourself) 11 | numpy,opencv-python,pillow,matplotib。The installation instructions are: 12 | numpy: python -m pip install numpy 13 | matplotlib: python -m pip install matplotlib 14 | opencv-python: python -m pip install -v opencv-python==3.4.2.17 15 | pillow: python -m pip install pillow 16 | 17 | c.Next, you need to download and install boost in order to get a dynamic link library called boost_python27-vc120-mt-x**-1_68.dll 18 | boost download link: 19 | win64: https://nchc.dl.sourceforge.net/project/boost/boost-binaries/1.68.0/boost_1_68_0-msvc-12.0-64.exe 20 | win32: https://excellmedia.dl.sourceforge.net/project/boost/boost-binaries/1.68.0/boost_1_68_0-msvc-12.0-32.exe 21 | After the download is complete, double-click the installation package to install. After the installation is complete, find the 22 | boost_python27-vc120-mt-x**-1_68.dll file (the '**' depends on the number of your windows system) and place it in 23 | .\Python27\Lib\site-packages path. 24 | 25 | d.Next, you need to download ffmpeg, the purpose is to get multiple dll files including avcodec-58.dll,etc. 26 | ffmpeg download link: 27 | win64: https://ffmpeg.zeranoe.com/builds/win64/shared/ffmpeg-20180825-844ff49-win64-shared.zip 28 | win32: https://ffmpeg.zeranoe.com/builds/win32/shared/ffmpeg-20180825-844ff49-win32-shared.zip 29 | After the download is complete, extract the files,enter ./bin directory and place all the .dll files under the .\Python27\Lib\site-packages path. 30 | 31 | e.Next, you need to download the vs2013 compiler. 32 | vs2013 compiler download link： 33 | win64: https://download.microsoft.com/download/2/E/6/2E61CFA4-993B-4DD4-91DA-3737CD5CD6E3/vcredist_x64.exe 34 | win32: https://download.microsoft.com/download/2/E/6/2E61CFA4-993B-4DD4-91DA-3737CD5CD6E3/vcredist_x86.exe 35 | After the download is complete, run the installation package and install. 36 | 37 | f.Finally Copy the libh264decoder from the Tello-Python package to the .\Python27\Lib\site-packages directory. 38 | h264decoder.pyd path: 39 | win64: .\Tello-Python\Tello_Video\h264decoder\Windows\x64\libh264decoder.pyd 40 | win32: .\Tello-Python\Tello_Video\h264decoder\Windows\x86\libh264decoder.pyd 41 | 42 | In the latest commit, a zip named 'tello_video_dll.zip' is provided (for win64 only), which includes the required files related 43 | to boost, ffmpeg, libh264decoder, vs2013 compiler.If you want to install it: 44 | First, double-click to run vcredist_x64.exe (vs2013 compiler installation package) to install.Second, copy the remaining files in the folder 45 | to the C:\Python27\Lib\site-packages path. 46 | (With this zip, you can avoid the online download and partial installation, but note that the files in this zip are only available for win64! 47 | Also, like numpy, matplotlib, pillow and opencv-python you still need to install with pip by yourself.） 48 | 49 | 50 | Q2:The one-click installation script (install.bat) on windows does not work properly on my computer. 51 | 52 | A2:At present, the install.bat that we provide contains steps such as MD5 checksum, which is not very compatible with some users' computers. A stable installation version will be provided later. 53 | If the one-click installation script does not work properly, please refer to the manual installation of dependencies in Q1&A1. 54 | 55 | 56 | Q3:What does install_manual.bat do? What is the difference with install.bat? 57 | 58 | A3:This script is no different from install.bat in the completed tasks. The main difference is that it will be interrupted before each installation project, 59 | and prompt the user to install. 60 | Currently the script is not stable. It is recommended that you apply install.bat or use the manual installation method in Q1&A1. 61 | 62 | 63 | Q4:In windows/linux/mac, all my dependencies have been installed, and no error has occurred after running python main.py.However, after running 64 | the program, the GUI does not pop up the video stream, and I cannot use the Snapshot button to take a photo. 65 | 66 | A4:Under normal circumstances, when tello wifi is connected, the GUI will pop up the playback window of the video stream after running the program, and the snapshot button can 67 | be used to take pictures.If it is not working properly (see https://www.youtube.com/watch?v=kcXN7CYgQ0g for actual results), 68 | Please consider and check the following situations: 69 | a.Whether your PC device is properly connected to your tello wifi; 70 | b.Check your PC's network firewall, which may block video stream data; 71 | c.Further check your network settings, because tello is pushing the video stream data to the 11111 port of your PC, so please check if the 11111 UDP port is disabled in 72 | your PC network settings, or the UDP data of port 11111 is blocked. If there is, please cancel it; 73 | d.In rare cases, it is possible that the 'streamon' command sent by the PC has been lost and not received by the tello. In this case, try to run the program again. 74 | 75 | 76 | Q5:After running the program, the keyboard control function in the command panel cannot be used normally. 77 | 78 | A5:Please make sure that the mouse does not click outside the command panel. If you click it, please click the command panel again with the mouse and 79 | retry the keyboard control function. 80 | 81 | 82 | Q6:Will you provide a python version sample code of the multi-tello swarm for tello EDU version? 83 | 84 | A6:Please refer to https://github.com/TelloSDK/Multi-Tello-Formation and read its README.md documentation.In addition, the program provides a direct executable file version (.exe). 85 | Please refer to the userguide in Tello-Swarm.zip for use. 86 | 87 | Q7:What is the role of libh264decoder? 88 | 89 | A7:Libh264decoder is a dynamic link library (dll)compiled from h264decoder.cpp, h264decoder.hpp and h264decoder_python.cpp in the .\Tello-Python\Tello_Video\h264decoder folder. 90 | ,its function is to decode the video stream of the h264 encoded format transmitted by the tello to the PC device.On windows, it exists in the form of libh264decoder.pyd, 91 | and in linux and macos,it exists in the form of libh264decoder.so.The libh264decoder dynamic link library on Windows,Linux and Mac is available in the 92 | .\Tello-Python\Tello_Video\h264decoder path. In the one-click script installation process for Linux and Mac, libh264decoder.so is compiled and generated in real time through 93 | source code.During the one-click script installation process for Windows, libh264decoder.pyd is copied directly from \Tello_Video\h264decoder\Windows and placed in 94 | .\Python27\Lib\site-packages. 95 | 96 | Q8:Can Tello EDU transmit video streams externally in station mode (send ap 'ssid' 'password' command for Tello EDU)? 97 | 98 | A8:This feature is currently not supported. 99 | 100 | Q9:What are the bands and channels of wifi UDP communication supported by the tello SDK interface? 101 | 102 | A9:Currently only supports 2.4GHz, and the supported channels include 0-11. 103 | 104 | If you have other questions, please ask questions or add! -------------------------------------------------------------------------------- /Tello_Video/LICENSE.md: -------------------------------------------------------------------------------- 1 | Sample code is offered under MIT License (See below). 2 | 3 | Copyright 2018 Ryze 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a 6 | copy of this software and associated documentation files (the "Software"), 7 | to deal in the Software without restriction, including without limitation 8 | the rights to use, copy, modify, merge, publish, distribute, sublicense, 9 | and/or sell copies of the Software, and to permit persons to whom the 10 | Software is furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included 13 | in all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS 16 | OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 18 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 19 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 20 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 21 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /Tello_Video/README.md: -------------------------------------------------------------------------------- 1 | # Tello-Video 2 | 3 | This is an example using the Tello SDK v1.3.0.0 and above to receive video stream from Tello camera,decode the video stream and show the image by GUI. 4 | 5 | - Written in Python 2.7 6 | - Tello SDK v1.3.0.0 and above(with h.264 video streaming) 7 | - This example includes a simple UI build with Tkinter to interact with Tello 8 | - Interactive control of Tello based on human movement is achieved via body pose recognition module. 9 | 10 | ## Prerequisites 11 | 12 | - Python2.7 13 | - pip 14 | - Python OpenCV 15 | - Numpy 16 | - PIL 17 | - libboost-python 18 | - Tkinter 19 | - homebrew(for mac) 20 | - Python h264 decoder 21 | - 22 | 23 | ## Installation 24 | 25 | In order to facilitate you to install python2.7 and various dependencies, we have written a one-click installation script for windows, Linux and macos. You can choose to run this script for the one-click installation, or you can download python2.7 and related libraries and dependencies online. If you have questions about the actions that the script performs, you can open the script with an editor and look up the comments for each instruction in the script. In addition, we have additionally written an uninstall script that cleans and restores all downloaded and configured content from the one-click installation script. 26 | 27 | - **Windows** 28 | 29 | Go to the "install\Windows" folder,select and run the correct "windows_install.bat" according to your computer operating system bits. 30 | 31 | - **Linux (Ubuntu 14.04 and above)** 32 | 33 | Go to the "install\Linux" folder in command line, run 34 | 35 | ``` 36 | chmod +x linux_install.sh 37 | ./linux_install.sh 38 | ``` 39 | 40 | - **Mac** 41 | 42 | 1. Make sure you have the latest Xcode command line tools installed. If not, you might need to update your OS X and XCode to the latest version in order to compile the h264 decoder module 43 | 2. Go to the "install\Mac" folder folder in command line, run 44 | 45 | ``` 46 | chmod a+x ./mac_install.sh 47 | ./mac_install.sh 48 | ``` 49 | 50 | If you see no errors during installation, you are good to go! 51 | 52 | ## Run the project 53 | - **Step1**. Turn on Tello and connect your computer device to Tello via wifi. 54 | 55 | 56 | - **Step2**. Open project folder in terminal. Run: 57 | 58 | ``` 59 | python main.py 60 | ``` 61 | 62 | - **Step3**. A UI will show up, you can now: 63 | 64 | - Watch live video stream from the Tello camera; 65 | - Take snapshot and save jpg to local folder; 66 | - Open Command Panel, which allows you to: 67 | - Take Off 68 | - Land 69 | - Flip (in forward, backward, left and right direction) 70 | - Control Tello using keyboard inputs: 71 | - **[key-Up]** move forward 20cm 72 | - **[key-Down]** move backward 20cm 73 | - **[key-Left]** move left 20 cm 74 | - **[key-Right]** move right 20 cm 75 | - **[key-w]** move up 20cm 76 | - **[key-s]** move down 20cm 77 | - **[key-a]** rotate counter-clockwise by 30 degree 78 | - **[key-d]** rotate clockwise by 30 degree 79 | - You can also adjust the **distance** and **degree** via the trackbar and hit the "reset distance" or "reset degree" button to customize your own control. 80 | 81 | ## Project Description 82 | 83 | ### tello.py - class Tello 84 | 85 | Wrapper class to interact with Tello drone. 86 | Modified from 87 | 88 | The object starts 2 threads: 89 | 90 | 1. thread for receiving command response from Tello 91 | 2. thread for receiving video stream 92 | 93 | You can use **read()** to read the last frame from Tello camera, and pause the video by setting **video_freeze(is_freeze=True)**. 94 | 95 | ### tello_control_ui.py - class TelloUI 96 | 97 | Modified from: https://www.pyimagesearch.com/2016/05/30/displaying-a-video-feed-with-opencv-and-tkinter/ 98 | 99 | Build with Tkinter. Display video, control video play/pause and control Tello using buttons and arrow keys. 100 | 101 | ### h264decoder - class libh264decoder 102 | 103 | From . 104 | 105 | A c++ based class that decodes raw h264 data. This module interacts with python language via python-libboost library, and its decoding functionality is based on ffmpeg library. 106 | 107 | After compilation, a libh264decoder.so or libh264decoder.pyd file will be placed in the working directory so that the main python file can reference it. 108 | 109 | If you have to compile it from source,with Linux or Mac,you can: 110 | 111 | ``` 112 | cd h264decoder 113 | mkdir build 114 | cd build 115 | cmake .. 116 | make 117 | cp libh264decoder.so ../../ 118 | ``` 119 | With Windows,you can create a project through visual studio, add files in h264decoder and dependencies such as ffmpeg and libboost, compile the project and generate a libh264decoder.pyd file.We have generated a libh264decoder.pyd and put it in the "\h264decoder\Windows"foleder so that you can copy put it to "python/site-package". 120 | 121 | ##Contact Information 122 | 123 | If you have any questions about this sample code and the installation, please feel free to contact me. You can communicate with me by sending e-mail to sdk@ryzerobotics.com. 124 | -------------------------------------------------------------------------------- /Tello_Video/h264decoder/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8) 2 | project(python_h264decoder) 3 | 4 | set(Python_ADDITIONAL_VERSIONS 2.7) 5 | 6 | if(UNIX AND NOT APPLE) 7 | set(LINUX TRUE) 8 | endif() 9 | 10 | if(APPLE) 11 | set(CMAKE_SHARED_LIBRARY_SUFFIX ".so") 12 | find_package(Boost REQUIRED COMPONENTS python27) 13 | elseif(LINUX) 14 | find_package(Boost REQUIRED COMPONENTS "python") 15 | endif(APPLE) 16 | 17 | 18 | 19 | find_package(PythonInterp 2.7 REQUIRED) 20 | find_package(PythonLibs 2.7 REQUIRED ) 21 | 22 | 23 | include_directories(${PYTHON_INCLUDE_DIRS}) 24 | include_directories(${Boost_INCLUDE_DIRS}) 25 | link_directories(${Boost_LIBRARY_DIRS}) 26 | 27 | add_compile_options ("-std=c++0x") 28 | 29 | add_library(h264decoder SHARED h264decoder.cpp h264decoder_python.cpp) 30 | 31 | if(APPLE) 32 | target_link_libraries(h264decoder avcodec swscale avutil ${Boost_LIBRARIES} ${Boost_PYTHON_LIBRARY_RELEASE} ${PYTHON_LIBRARIES}) 33 | elseif(LINUX) 34 | target_link_libraries(h264decoder avcodec swscale avutil ${Boost_PYTHON_LIBRARY_RELEASE} ${PYTHON_LIBRARIES}) 35 | endif(APPLE) 36 | 37 | add_custom_command(TARGET h264decoder POST_BUILD 38 | COMMAND ${CMAKE_COMMAND} -E create_symlink 39 | ${CMAKE_BINARY_DIR}/libh264decoder.so ${CMAKE_SOURCE_DIR}/libh264decoder.so) 40 | install(TARGETS h264decoder LIBRARY DESTINATION .) 41 | -------------------------------------------------------------------------------- /Tello_Video/h264decoder/Linux/libh264decoder.so: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/Tello_Video/h264decoder/Linux/libh264decoder.so -------------------------------------------------------------------------------- /Tello_Video/h264decoder/Mac/libh264decoder.so: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/Tello_Video/h264decoder/Mac/libh264decoder.so -------------------------------------------------------------------------------- /Tello_Video/h264decoder/Windows/x64/libh264decoder.pyd: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/Tello_Video/h264decoder/Windows/x64/libh264decoder.pyd -------------------------------------------------------------------------------- /Tello_Video/h264decoder/Windows/x86/libh264decoder.pyd: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/Tello_Video/h264decoder/Windows/x86/libh264decoder.pyd -------------------------------------------------------------------------------- /Tello_Video/h264decoder/h264decoder.cpp: -------------------------------------------------------------------------------- 1 | extern "C" { 2 | #include 3 | #include 4 | #include 5 | #include 6 | } 7 | 8 | #ifndef PIX_FMT_RGB24 9 | #define PIX_FMT_RGB24 AV_PIX_FMT_RGB24 10 | #endif 11 | 12 | #ifndef CODEC_CAP_TRUNCATED 13 | #define CODEC_CAP_TRUNCATED AV_CODEC_CAP_TRUNCATED 14 | #endif 15 | 16 | #ifndef CODEC_FLAG_TRUNCATED 17 | #define CODEC_FLAG_TRUNCATED AV_CODEC_FLAG_TRUNCATED 18 | #endif 19 | 20 | #include "h264decoder.hpp" 21 | #include 22 | 23 | typedef unsigned char ubyte; 24 | 25 | /* For backward compatibility with release 9 or so of libav */ 26 | #if (LIBAVCODEC_VERSION_MAJOR <= 54) 27 | # define av_frame_alloc avcodec_alloc_frame 28 | # define av_frame_free avcodec_free_frame 29 | #endif 30 | 31 | 32 | H264Decoder::H264Decoder() 33 | { 34 | avcodec_register_all(); 35 | 36 | codec = avcodec_find_decoder(AV_CODEC_ID_H264); 37 | if (!codec) 38 | throw H264InitFailure("cannot find decoder"); 39 | 40 | context = avcodec_alloc_context3(codec); 41 | if (!context) 42 | throw H264InitFailure("cannot allocate context"); 43 | 44 | if(codec->capabilities & CODEC_CAP_TRUNCATED) { 45 | context->flags |= CODEC_FLAG_TRUNCATED; 46 | } 47 | 48 | int err = avcodec_open2(context, codec, nullptr); 49 | if (err < 0) 50 | throw H264InitFailure("cannot open context"); 51 | 52 | parser = av_parser_init(AV_CODEC_ID_H264); 53 | if (!parser) 54 | throw H264InitFailure("cannot init parser"); 55 | 56 | frame = av_frame_alloc(); 57 | if (!frame) 58 | throw H264InitFailure("cannot allocate frame"); 59 | 60 | #if 1 61 | pkt = new AVPacket; 62 | if (!pkt) 63 | throw H264InitFailure("cannot allocate packet"); 64 | av_init_packet(pkt); 65 | #endif 66 | } 67 | 68 | 69 | H264Decoder::~H264Decoder() 70 | { 71 | av_parser_close(parser); 72 | avcodec_close(context); 73 | av_free(context); 74 | av_frame_free(&frame); 75 | #if 1 76 | delete pkt; 77 | #endif 78 | } 79 | 80 | 81 | ssize_t H264Decoder::parse(const ubyte* in_data, ssize_t in_size) 82 | { 83 | auto nread = av_parser_parse2(parser, context, &pkt->data, &pkt->size, 84 | in_data, in_size, 85 | 0, 0, AV_NOPTS_VALUE); 86 | return nread; 87 | } 88 | 89 | 90 | bool H264Decoder::is_frame_available() const 91 | { 92 | return pkt->size > 0; 93 | } 94 | 95 | 96 | const AVFrame& H264Decoder::decode_frame() 97 | { 98 | int got_picture = 0; 99 | int nread = avcodec_decode_video2(context, frame, &got_picture, pkt); 100 | if (nread < 0 || got_picture == 0) 101 | throw H264DecodeFailure("error decoding frame\n"); 102 | return *frame; 103 | } 104 | 105 | 106 | ConverterRGB24::ConverterRGB24() 107 | { 108 | framergb = av_frame_alloc(); 109 | if (!framergb) 110 | throw H264DecodeFailure("cannot allocate frame"); 111 | context = nullptr; 112 | } 113 | 114 | ConverterRGB24::~ConverterRGB24() 115 | { 116 | sws_freeContext(context); 117 | av_frame_free(&framergb); 118 | } 119 | 120 | 121 | const AVFrame& ConverterRGB24::convert(const AVFrame &frame, ubyte* out_rgb) 122 | { 123 | int w = frame.width; 124 | int h = frame.height; 125 | int pix_fmt = frame.format; 126 | 127 | context = sws_getCachedContext(context, 128 | w, h, (AVPixelFormat)pix_fmt, 129 | w, h, PIX_FMT_RGB24, SWS_BILINEAR, 130 | nullptr, nullptr, nullptr); 131 | if (!context) 132 | throw H264DecodeFailure("cannot allocate context"); 133 | 134 | // Setup framergb with out_rgb as external buffer. Also say that we want RGB24 output. 135 | avpicture_fill((AVPicture*)framergb, out_rgb, PIX_FMT_RGB24, w, h); 136 | // Do the conversion. 137 | sws_scale(context, frame.data, frame.linesize, 0, h, 138 | framergb->data, framergb->linesize); 139 | framergb->width = w; 140 | framergb->height = h; 141 | return *framergb; 142 | } 143 | 144 | /* 145 | Determine required size of framebuffer. 146 | 147 | avpicture_get_size is used in http://dranger.com/ffmpeg/tutorial01.html 148 | to do this. However, avpicture_get_size returns the size of a compact 149 | representation, without padding bytes. Since we use avpicture_fill to 150 | fill the buffer we should also use it to determine the required size. 151 | */ 152 | int ConverterRGB24::predict_size(int w, int h) 153 | { 154 | return avpicture_fill((AVPicture*)framergb, nullptr, PIX_FMT_RGB24, w, h); 155 | } 156 | 157 | 158 | 159 | std::pair width_height(const AVFrame& f) 160 | { 161 | return std::make_pair(f.width, f.height); 162 | } 163 | 164 | int row_size(const AVFrame& f) 165 | { 166 | return f.linesize[0]; 167 | } 168 | 169 | 170 | void disable_logging() 171 | { 172 | av_log_set_level(AV_LOG_QUIET); 173 | } 174 | -------------------------------------------------------------------------------- /Tello_Video/h264decoder/h264decoder.hpp: -------------------------------------------------------------------------------- 1 | #pragma once 2 | /* 3 | This h264 decoder class is just a thin wrapper around libav 4 | functions to decode h264 videos. It would have been easy to use 5 | libav directly in the python module code but I like to keep these 6 | things separate. 7 | 8 | It is mostly based on roxlu's code. See 9 | http://roxlu.com/2014/039/decoding-h264-and-yuv420p-playback 10 | 11 | However, in contrast to roxlu's code the color space conversion is 12 | done by libav functions - so on the CPU, I suppose. 13 | 14 | Most functions/members throw exceptions. This way, error states are 15 | conveniently forwarded to python via the exception translation 16 | mechanisms of boost::python. 17 | */ 18 | 19 | // for ssize_t (signed int type as large as pointer type) 20 | #include 21 | #include 22 | #include 23 | 24 | struct AVCodecContext; 25 | struct AVFrame; 26 | struct AVCodec; 27 | struct AVCodecParserContext; 28 | struct SwsContext; 29 | struct AVPacket; 30 | 31 | 32 | class H264Exception : public std::runtime_error 33 | { 34 | public: 35 | H264Exception(const char* s) : std::runtime_error(s) {} 36 | }; 37 | 38 | class H264InitFailure : public H264Exception 39 | { 40 | public: 41 | H264InitFailure(const char* s) : H264Exception(s) {} 42 | }; 43 | 44 | class H264DecodeFailure : public H264Exception 45 | { 46 | public: 47 | H264DecodeFailure(const char* s) : H264Exception(s) {} 48 | }; 49 | 50 | 51 | class H264Decoder 52 | { 53 | /* Persistent things here, using RAII for cleanup. */ 54 | AVCodecContext *context; 55 | AVFrame *frame; 56 | AVCodec *codec; 57 | AVCodecParserContext *parser; 58 | /* In the documentation example on the github master branch, the 59 | packet is put on the heap. This is done here to store the pointers 60 | to the encoded data, which must be kept around between calls to 61 | parse- and decode frame. In release 11 it is put on the stack, too. 62 | */ 63 | AVPacket *pkt; 64 | public: 65 | H264Decoder(); 66 | ~H264Decoder(); 67 | /* First, parse a continuous data stream, dividing it into 68 | packets. When there is enough data to form a new frame, decode 69 | the data and return the frame. parse returns the number 70 | of consumed bytes of the input stream. It stops consuming 71 | bytes at frame boundaries. 72 | */ 73 | ssize_t parse(const unsigned char* in_data, ssize_t in_size); 74 | bool is_frame_available() const; 75 | const AVFrame& decode_frame(); 76 | }; 77 | 78 | // TODO: Rename to OutputStage or so?! 79 | class ConverterRGB24 80 | { 81 | SwsContext *context; 82 | AVFrame *framergb; 83 | 84 | public: 85 | ConverterRGB24(); 86 | ~ConverterRGB24(); 87 | 88 | /* Returns, given a width and height, 89 | how many bytes the frame buffer is going to need. */ 90 | int predict_size(int w, int h); 91 | /* Given a decoded frame, convert it to RGB format and fill 92 | out_rgb with the result. Returns a AVFrame structure holding 93 | additional information about the RGB frame, such as the number of 94 | bytes in a row and so on. */ 95 | const AVFrame& convert(const AVFrame &frame, unsigned char* out_rgb); 96 | }; 97 | 98 | void disable_logging(); 99 | 100 | /* Wrappers, so we don't have to include libav headers. */ 101 | std::pair width_height(const AVFrame&); 102 | int row_size(const AVFrame&); 103 | 104 | /* all the documentation links 105 | * My version of libav on ubuntu 16 appears to be from the release/11 branch on github 106 | * Video decoding example: https://libav.org/documentation/doxygen/release/11/avcodec_8c_source.html#l00455 107 | * https://libav.org/documentation/doxygen/release/9/group__lavc__decoding.html 108 | * https://libav.org/documentation/doxygen/release/11/group__lavc__parsing.html 109 | * https://libav.org/documentation/doxygen/release/9/swscale_8h.html 110 | * https://libav.org/documentation/doxygen/release/9/group__lavu.html 111 | * https://libav.org/documentation/doxygen/release/9/group__lavc__picture.html 112 | * http://dranger.com/ffmpeg/tutorial01.html 113 | */ -------------------------------------------------------------------------------- /Tello_Video/h264decoder/h264decoder_python.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | 6 | // python string api, see 7 | // https://docs.python.org/2/c-api/string.html 8 | extern "C" { 9 | #include 10 | } 11 | 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | namespace py = boost::python; 18 | 19 | #include "h264decoder.hpp" 20 | 21 | using ubyte = unsigned char; 22 | 23 | 24 | class GILScopedReverseLock 25 | { 26 | // see https://docs.python.org/2/c-api/init.html (Releasing the GIL ...) 27 | public: 28 | GILScopedReverseLock() 29 | : state(nullptr) 30 | { 31 | unlock(); 32 | } 33 | 34 | ~GILScopedReverseLock() 35 | { 36 | lock(); 37 | } 38 | 39 | void lock() 40 | { 41 | // Allow successive calls to lock. 42 | // E.g. lock() followed by destructor. 43 | if (state != nullptr) 44 | { 45 | PyEval_RestoreThread(state); 46 | state = nullptr; 47 | } 48 | } 49 | 50 | void unlock() 51 | { 52 | assert (state == nullptr); 53 | state = PyEval_SaveThread(); 54 | } 55 | 56 | GILScopedReverseLock(const GILScopedReverseLock &) = delete; 57 | GILScopedReverseLock(const GILScopedReverseLock &&) = delete; 58 | GILScopedReverseLock operator=(const GILScopedReverseLock &) = delete; 59 | GILScopedReverseLock operator=(const GILScopedReverseLock &&) = delete; 60 | private: 61 | PyThreadState *state; 62 | }; 63 | 64 | 65 | /* The class wrapped in python via boost::python */ 66 | class PyH264Decoder 67 | { 68 | H264Decoder decoder; 69 | ConverterRGB24 converter; 70 | 71 | /* Extract frames from input stream. Stops at frame boundaries and returns the number of consumed bytes 72 | * in num_consumed. 73 | * 74 | * If a frame is completed, is_frame_available is set to true, and the returned python tuple contains 75 | * formation about the frame as well as the frame buffer memory. 76 | * 77 | * Else, i.e. all data in the buffer is consumed, is_frame_available is set to false. The returned tuple 78 | * contains dummy data. 79 | */ 80 | py::tuple decode_frame_impl(const ubyte *data, ssize_t num, ssize_t &num_consumed, bool &is_frame_available); 81 | 82 | public: 83 | /* Decoding style analogous to c/c++ way. Stop at frame boundaries. 84 | * Return tuple containing frame data as above as nested tuple, and an integer telling how many bytes were consumed. */ 85 | py::tuple decode_frame(const py::str &data_in_str); 86 | /* Process all the input data and return a list of all contained frames. */ 87 | py::list decode(const py::str &data_in_str); 88 | }; 89 | 90 | 91 | py::tuple PyH264Decoder::decode_frame_impl(const ubyte *data_in, ssize_t len, ssize_t &num_consumed, bool &is_frame_available) 92 | { 93 | GILScopedReverseLock gilguard; 94 | num_consumed = decoder.parse((ubyte*)data_in, len); 95 | 96 | if (is_frame_available = decoder.is_frame_available()) 97 | { 98 | const auto &frame = decoder.decode_frame(); 99 | int w, h; std::tie(w,h) = width_height(frame); 100 | Py_ssize_t out_size = converter.predict_size(w,h); 101 | 102 | gilguard.lock(); 103 | // Construction of py::handle causes ... TODO: WHAT? No increase of ref count ?! 104 | py::object py_out_str(py::handle<>(PyString_FromStringAndSize(NULL, out_size))); 105 | char* out_buffer = PyString_AsString(py_out_str.ptr()); 106 | 107 | gilguard.unlock(); 108 | const auto &rgbframe = converter.convert(frame, (ubyte*)out_buffer); 109 | 110 | gilguard.lock(); 111 | return py::make_tuple(py_out_str, w, h, row_size(rgbframe)); 112 | } 113 | else 114 | { 115 | gilguard.lock(); 116 | return py::make_tuple(py::object(), 0, 0, 0); 117 | } 118 | } 119 | 120 | 121 | py::tuple PyH264Decoder::decode_frame(const py::str &data_in_str) 122 | { 123 | ssize_t len = PyString_Size(data_in_str.ptr()); 124 | const ubyte* data_in = (const ubyte*)(PyString_AsString(data_in_str.ptr())); 125 | 126 | ssize_t num_consumed = 0; 127 | bool is_frame_available = false; 128 | auto frame = decode_frame_impl(data_in, len, num_consumed, is_frame_available); 129 | 130 | return py::make_tuple(frame, num_consumed); 131 | } 132 | 133 | 134 | py::list PyH264Decoder::decode(const py::str &data_in_str) 135 | { 136 | ssize_t len = PyString_Size(data_in_str.ptr()); 137 | const ubyte* data_in = (const ubyte*)(PyString_AsString(data_in_str.ptr())); 138 | 139 | py::list out; 140 | 141 | try 142 | { 143 | while (len > 0) 144 | { 145 | ssize_t num_consumed = 0; 146 | bool is_frame_available = false; 147 | 148 | try 149 | { 150 | auto frame = decode_frame_impl(data_in, len, num_consumed, is_frame_available); 151 | if (is_frame_available) 152 | { 153 | out.append(frame); 154 | } 155 | } 156 | catch (const H264DecodeFailure &e) 157 | { 158 | if (num_consumed <= 0) 159 | // This case is fatal because we cannot continue to move ahead in the stream. 160 | throw e; 161 | } 162 | 163 | len -= num_consumed; 164 | data_in += num_consumed; 165 | } 166 | } 167 | catch (const H264DecodeFailure &e) 168 | { 169 | } 170 | 171 | return out; 172 | } 173 | 174 | 175 | BOOST_PYTHON_MODULE(libh264decoder) 176 | { 177 | PyEval_InitThreads(); // need for release of the GIL (http://stackoverflow.com/questions/8009613/boost-python-not-supporting-parallelism) 178 | py::class_("H264Decoder") 179 | .def("decode_frame", &PyH264Decoder::decode_frame) 180 | .def("decode", &PyH264Decoder::decode); 181 | py::def("disable_logging", disable_logging); 182 | } -------------------------------------------------------------------------------- /Tello_Video/h264decoder/readme.md: -------------------------------------------------------------------------------- 1 | H264 Decoder Python Module 2 | ========================== 3 | 4 | The aim of this project is to provide a simple decoder for video 5 | captured by a Raspberry Pi camera. At the time of this writing I only 6 | need H264 decoding, since a H264 stream is what the RPi software 7 | delivers. Furthermore flexibility to incorporate the decoder in larger 8 | python programs in various ways is desirable. 9 | 10 | The code might also serve as example for libav and boost python usage. 11 | 12 | 13 | Files 14 | ----- 15 | * `h264decoder.hpp`, `h264decoder.cpp` and `h264decoder_python.cpp` contain the module code. 16 | 17 | * Other source files are tests and demos. 18 | 19 | 20 | Requirements 21 | ------------ 22 | * cmake for building 23 | * libav 24 | * boost python 25 | 26 | 27 | Todo 28 | ---- 29 | 30 | * Add a video clip for testing and remove hard coded file names in demos/tests. 31 | 32 | 33 | License 34 | ------- 35 | The code is published under the Mozilla Public License v. 2.0. 36 | -------------------------------------------------------------------------------- /Tello_Video/img/readme.md: -------------------------------------------------------------------------------- 1 | This folder mainly stores photos taken by the tello front view camera. -------------------------------------------------------------------------------- /Tello_Video/install/Linux/linux_install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | echo 'Compiling and Installing the Tello Video Stream module' 4 | echo 'You might need to enter your password' 5 | 6 | cd .. 7 | cd .. 8 | sudo apt-get update -y 9 | 10 | # install python 2.7 11 | sudo apt-get install python2.7 python-pip -y 12 | sudo pip install --upgrade pip 13 | 14 | #switch to python2.7 15 | sudo update-alternatives --install /usr/bin/python python /usr/bin/python2 150 16 | sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 100 17 | 18 | sudo apt-get update -y 19 | 20 | # install cmake 21 | #sudo apt-get install cmake -y 22 | sudo pip install cmake 23 | 24 | # install dependencies 25 | sudo apt-get install libboost-all-dev -y 26 | sudo apt-get install libavcodec-dev -y 27 | sudo apt-get install libswscale-dev -y 28 | sudo apt-get install python-numpy -y 29 | sudo apt-get install python-matplotlib -y 30 | sudo pip install opencv-python 31 | sudo apt-get install python-imaging-tk 32 | 33 | # pull and build h264 decoder library 34 | cd h264decoder 35 | mkdir build 36 | cd build 37 | cmake .. 38 | make 39 | 40 | # copy source .so file to tello.py directory 41 | cp libh264decoder.so ../../ 42 | 43 | echo 'Compilation and Installation Done!' 44 | -------------------------------------------------------------------------------- /Tello_Video/install/Linux/linux_uninstall.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | echo 'Uninstalling the Tello Video Stream module' 4 | 5 | cd .. 6 | cd .. 7 | sudo pip uninstall opencv-python -y 8 | sudo pip uninstall cmake -y 9 | sudo pip uninstall pip -y 10 | 11 | sudo apt-get remove libboost-all-dev -y 12 | sudo apt-get remove libavcodec-dev -y 13 | sudo apt-get remove libswscale-dev -y 14 | sudo apt-get remove python-numpy -y 15 | sudo apt-get remove python-matplotlib -y 16 | sudo apt-get remove python-pil.imagetk -y 17 | 18 | sudo apt-get remove python-pip -y 19 | sudo apt-get remove python2.7 -y 20 | 21 | sudo apt-get update -y 22 | 23 | rm ./libh264decoder.so 24 | rm -r ./h264decoder/build 25 | echo 'Uninstallation Done!' 26 | -------------------------------------------------------------------------------- /Tello_Video/install/Mac/mac_install.sh: -------------------------------------------------------------------------------- 1 | 2 | #!/bin/sh 3 | 4 | 5 | 6 | echo 'Compiling and Installing the Tello Video Stream module' 7 | 8 | echo 'You might need to enter your password' 9 | 10 | # go to /sample_code folder 11 | cd .. 12 | cd .. 13 | 14 | # install Homebrew 15 | 16 | /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" 17 | 18 | brew update 19 | 20 | 21 | 22 | # install pip 23 | 24 | #sudo easy_install pip 25 | 26 | curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py 27 | sudo python get-pip.py 28 | 29 | # install cmake 30 | 31 | brew install cmake 32 | 33 | 34 | 35 | # install dependencies 36 | 37 | brew install boost 38 | 39 | brew install boost-python 40 | 41 | brew install ffmpeg 42 | 43 | brew install tcl-tk 44 | 45 | sudo pip install numpy --ignore-installed 46 | 47 | sudo pip install matplotlib --ignore-installed 48 | 49 | sudo pip install pillow --ignore-installed 50 | 51 | sudo pip install opencv-python --ignore-installed 52 | 53 | 54 | 55 | # pull and build h264 decoder library 56 | 57 | cd h264decoder 58 | 59 | 60 | mkdir build 61 | 62 | cd build 63 | 64 | cmake .. 65 | 66 | make 67 | 68 | 69 | 70 | # copy source .so file to tello.py directory 71 | 72 | cp libh264decoder.so ../../ 73 | 74 | 75 | 76 | echo 'Compilation and Installation Done!' 77 | -------------------------------------------------------------------------------- /Tello_Video/install/Mac/mac_uninstall.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | echo 'Uninstalling the Tello Video Stream module' 3 | 4 | echo 'You might need to enter your password' 5 | 6 | # go to /sample_code folder 7 | cd .. 8 | cd .. 9 | 10 | brew update 11 | 12 | sudo pip uninstall matplotlib -y 13 | sudo pip uninstall numpy -y 14 | sudo pip install pillow -y 15 | sudo pip install opencv-python -y 16 | sudo pip uninstall pip -y 17 | 18 | brew uninstall tcl-tk 19 | brew uninstall ffmpeg 20 | brew uninstall boost-python 21 | brew uninstall boost 22 | brew uninstall cmake 23 | 24 | rm -f ./libh264decoder.so 25 | rm -rf ./h264decoder/build 26 | 27 | echo 'Uninstallation Done!' 28 | -------------------------------------------------------------------------------- /Tello_Video/install/Windows/install.bat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/Tello_Video/install/Windows/install.bat -------------------------------------------------------------------------------- /Tello_Video/install/Windows/uninstall.bat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/Tello_Video/install/Windows/uninstall.bat -------------------------------------------------------------------------------- /Tello_Video/main.py: -------------------------------------------------------------------------------- 1 | import tello 2 | from tello_control_ui import TelloUI 3 | 4 | 5 | def main(): 6 | 7 | drone = tello.Tello('', 8889) 8 | vplayer = TelloUI(drone,"./img/") 9 | 10 | # start the Tkinter mainloop 11 | vplayer.root.mainloop() 12 | 13 | if __name__ == "__main__": 14 | main() 15 | -------------------------------------------------------------------------------- /Tello_Video/tello.py: -------------------------------------------------------------------------------- 1 | import socket 2 | import threading 3 | import time 4 | import numpy as np 5 | import libh264decoder 6 | 7 | class Tello: 8 | """Wrapper class to interact with the Tello drone.""" 9 | 10 | def __init__(self, local_ip, local_port, imperial=False, command_timeout=.3, tello_ip='192.168.10.1', 11 | tello_port=8889): 12 | """ 13 | Binds to the local IP/port and puts the Tello into command mode. 14 | 15 | :param local_ip (str): Local IP address to bind. 16 | :param local_port (int): Local port to bind. 17 | :param imperial (bool): If True, speed is MPH and distance is feet. 18 | If False, speed is KPH and distance is meters. 19 | :param command_timeout (int|float): Number of seconds to wait for a response to a command. 20 | :param tello_ip (str): Tello IP. 21 | :param tello_port (int): Tello port. 22 | """ 23 | 24 | self.abort_flag = False 25 | self.decoder = libh264decoder.H264Decoder() 26 | self.command_timeout = command_timeout 27 | self.imperial = imperial 28 | self.response = None 29 | self.frame = None # numpy array BGR -- current camera output frame 30 | self.is_freeze = False # freeze current camera output 31 | self.last_frame = None 32 | self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # socket for sending cmd 33 | self.socket_video = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # socket for receiving video stream 34 | self.tello_address = (tello_ip, tello_port) 35 | self.local_video_port = 11111 # port for receiving video stream 36 | self.last_height = 0 37 | self.socket.bind((local_ip, local_port)) 38 | 39 | # thread for receiving cmd ack 40 | self.receive_thread = threading.Thread(target=self._receive_thread) 41 | self.receive_thread.daemon = True 42 | 43 | self.receive_thread.start() 44 | 45 | # to receive video -- send cmd: command, streamon 46 | self.socket.sendto(b'command', self.tello_address) 47 | print ('sent: command') 48 | self.socket.sendto(b'streamon', self.tello_address) 49 | print ('sent: streamon') 50 | 51 | self.socket_video.bind((local_ip, self.local_video_port)) 52 | 53 | # thread for receiving video 54 | self.receive_video_thread = threading.Thread(target=self._receive_video_thread) 55 | self.receive_video_thread.daemon = True 56 | 57 | self.receive_video_thread.start() 58 | 59 | def __del__(self): 60 | """Closes the local socket.""" 61 | 62 | self.socket.close() 63 | self.socket_video.close() 64 | 65 | def read(self): 66 | """Return the last frame from camera.""" 67 | if self.is_freeze: 68 | return self.last_frame 69 | else: 70 | return self.frame 71 | 72 | def video_freeze(self, is_freeze=True): 73 | """Pause video output -- set is_freeze to True""" 74 | self.is_freeze = is_freeze 75 | if is_freeze: 76 | self.last_frame = self.frame 77 | 78 | def _receive_thread(self): 79 | """Listen to responses from the Tello. 80 | 81 | Runs as a thread, sets self.response to whatever the Tello last returned. 82 | 83 | """ 84 | while True: 85 | try: 86 | self.response, ip = self.socket.recvfrom(3000) 87 | #print(self.response) 88 | except socket.error as exc: 89 | print ("Caught exception socket.error : %s" % exc) 90 | 91 | def _receive_video_thread(self): 92 | """ 93 | Listens for video streaming (raw h264) from the Tello. 94 | 95 | Runs as a thread, sets self.frame to the most recent frame Tello captured. 96 | 97 | """ 98 | packet_data = "" 99 | while True: 100 | try: 101 | res_string, ip = self.socket_video.recvfrom(2048) 102 | packet_data += res_string 103 | # end of frame 104 | if len(res_string) != 1460: 105 | for frame in self._h264_decode(packet_data): 106 | self.frame = frame 107 | packet_data = "" 108 | 109 | except socket.error as exc: 110 | print ("Caught exception socket.error : %s" % exc) 111 | 112 | def _h264_decode(self, packet_data): 113 | """ 114 | decode raw h264 format data from Tello 115 | 116 | :param packet_data: raw h264 data array 117 | 118 | :return: a list of decoded frame 119 | """ 120 | res_frame_list = [] 121 | frames = self.decoder.decode(packet_data) 122 | for framedata in frames: 123 | (frame, w, h, ls) = framedata 124 | if frame is not None: 125 | # print 'frame size %i bytes, w %i, h %i, linesize %i' % (len(frame), w, h, ls) 126 | 127 | frame = np.fromstring(frame, dtype=np.ubyte, count=len(frame), sep='') 128 | frame = (frame.reshape((h, ls / 3, 3))) 129 | frame = frame[:, :w, :] 130 | res_frame_list.append(frame) 131 | 132 | return res_frame_list 133 | 134 | def send_command(self, command): 135 | """ 136 | Send a command to the Tello and wait for a response. 137 | 138 | :param command: Command to send. 139 | :return (str): Response from Tello. 140 | 141 | """ 142 | 143 | print (">> send cmd: {}".format(command)) 144 | self.abort_flag = False 145 | timer = threading.Timer(self.command_timeout, self.set_abort_flag) 146 | 147 | self.socket.sendto(command.encode('utf-8'), self.tello_address) 148 | 149 | timer.start() 150 | while self.response is None: 151 | if self.abort_flag is True: 152 | break 153 | timer.cancel() 154 | 155 | if self.response is None: 156 | response = 'none_response' 157 | else: 158 | response = self.response.decode('utf-8') 159 | 160 | self.response = None 161 | 162 | return response 163 | 164 | def set_abort_flag(self): 165 | """ 166 | Sets self.abort_flag to True. 167 | 168 | Used by the timer in Tello.send_command() to indicate to that a response 169 | 170 | timeout has occurred. 171 | 172 | """ 173 | 174 | self.abort_flag = True 175 | 176 | def takeoff(self): 177 | """ 178 | Initiates take-off. 179 | 180 | Returns: 181 | str: Response from Tello, 'OK' or 'FALSE'. 182 | 183 | """ 184 | 185 | return self.send_command('takeoff') 186 | 187 | def set_speed(self, speed): 188 | """ 189 | Sets speed. 190 | 191 | This method expects KPH or MPH. The Tello API expects speeds from 192 | 1 to 100 centimeters/second. 193 | 194 | Metric: .1 to 3.6 KPH 195 | Imperial: .1 to 2.2 MPH 196 | 197 | Args: 198 | speed (int|float): Speed. 199 | 200 | Returns: 201 | str: Response from Tello, 'OK' or 'FALSE'. 202 | 203 | """ 204 | 205 | speed = float(speed) 206 | 207 | if self.imperial is True: 208 | speed = int(round(speed * 44.704)) 209 | else: 210 | speed = int(round(speed * 27.7778)) 211 | 212 | return self.send_command('speed %s' % speed) 213 | 214 | def rotate_cw(self, degrees): 215 | """ 216 | Rotates clockwise. 217 | 218 | Args: 219 | degrees (int): Degrees to rotate, 1 to 360. 220 | 221 | Returns: 222 | str: Response from Tello, 'OK' or 'FALSE'. 223 | 224 | """ 225 | 226 | return self.send_command('cw %s' % degrees) 227 | 228 | def rotate_ccw(self, degrees): 229 | """ 230 | Rotates counter-clockwise. 231 | 232 | Args: 233 | degrees (int): Degrees to rotate, 1 to 360. 234 | 235 | Returns: 236 | str: Response from Tello, 'OK' or 'FALSE'. 237 | 238 | """ 239 | return self.send_command('ccw %s' % degrees) 240 | 241 | def flip(self, direction): 242 | """ 243 | Flips. 244 | 245 | Args: 246 | direction (str): Direction to flip, 'l', 'r', 'f', 'b'. 247 | 248 | Returns: 249 | str: Response from Tello, 'OK' or 'FALSE'. 250 | 251 | """ 252 | 253 | return self.send_command('flip %s' % direction) 254 | 255 | def get_response(self): 256 | """ 257 | Returns response of tello. 258 | 259 | Returns: 260 | int: response of tello. 261 | 262 | """ 263 | response = self.response 264 | return response 265 | 266 | def get_height(self): 267 | """Returns height(dm) of tello. 268 | 269 | Returns: 270 | int: Height(dm) of tello. 271 | 272 | """ 273 | height = self.send_command('height?') 274 | height = str(height) 275 | height = filter(str.isdigit, height) 276 | try: 277 | height = int(height) 278 | self.last_height = height 279 | except: 280 | height = self.last_height 281 | pass 282 | return height 283 | 284 | def get_battery(self): 285 | """Returns percent battery life remaining. 286 | 287 | Returns: 288 | int: Percent battery life remaining. 289 | 290 | """ 291 | 292 | battery = self.send_command('battery?') 293 | 294 | try: 295 | battery = int(battery) 296 | except: 297 | pass 298 | 299 | return battery 300 | 301 | def get_flight_time(self): 302 | """Returns the number of seconds elapsed during flight. 303 | 304 | Returns: 305 | int: Seconds elapsed during flight. 306 | 307 | """ 308 | 309 | flight_time = self.send_command('time?') 310 | 311 | try: 312 | flight_time = int(flight_time) 313 | except: 314 | pass 315 | 316 | return flight_time 317 | 318 | def get_speed(self): 319 | """Returns the current speed. 320 | 321 | Returns: 322 | int: Current speed in KPH or MPH. 323 | 324 | """ 325 | 326 | speed = self.send_command('speed?') 327 | 328 | try: 329 | speed = float(speed) 330 | 331 | if self.imperial is True: 332 | speed = round((speed / 44.704), 1) 333 | else: 334 | speed = round((speed / 27.7778), 1) 335 | except: 336 | pass 337 | 338 | return speed 339 | 340 | def land(self): 341 | """Initiates landing. 342 | 343 | Returns: 344 | str: Response from Tello, 'OK' or 'FALSE'. 345 | 346 | """ 347 | 348 | return self.send_command('land') 349 | 350 | def move(self, direction, distance): 351 | """Moves in a direction for a distance. 352 | 353 | This method expects meters or feet. The Tello API expects distances 354 | from 20 to 500 centimeters. 355 | 356 | Metric: .02 to 5 meters 357 | Imperial: .7 to 16.4 feet 358 | 359 | Args: 360 | direction (str): Direction to move, 'forward', 'back', 'right' or 'left'. 361 | distance (int|float): Distance to move. 362 | 363 | Returns: 364 | str: Response from Tello, 'OK' or 'FALSE'. 365 | 366 | """ 367 | 368 | distance = float(distance) 369 | 370 | if self.imperial is True: 371 | distance = int(round(distance * 30.48)) 372 | else: 373 | distance = int(round(distance * 100)) 374 | 375 | return self.send_command('%s %s' % (direction, distance)) 376 | 377 | def move_backward(self, distance): 378 | """Moves backward for a distance. 379 | 380 | See comments for Tello.move(). 381 | 382 | Args: 383 | distance (int): Distance to move. 384 | 385 | Returns: 386 | str: Response from Tello, 'OK' or 'FALSE'. 387 | 388 | """ 389 | 390 | return self.move('back', distance) 391 | 392 | def move_down(self, distance): 393 | """Moves down for a distance. 394 | 395 | See comments for Tello.move(). 396 | 397 | Args: 398 | distance (int): Distance to move. 399 | 400 | Returns: 401 | str: Response from Tello, 'OK' or 'FALSE'. 402 | 403 | """ 404 | 405 | return self.move('down', distance) 406 | 407 | def move_forward(self, distance): 408 | """Moves forward for a distance. 409 | 410 | See comments for Tello.move(). 411 | 412 | Args: 413 | distance (int): Distance to move. 414 | 415 | Returns: 416 | str: Response from Tello, 'OK' or 'FALSE'. 417 | 418 | """ 419 | return self.move('forward', distance) 420 | 421 | def move_left(self, distance): 422 | """Moves left for a distance. 423 | 424 | See comments for Tello.move(). 425 | 426 | Args: 427 | distance (int): Distance to move. 428 | 429 | Returns: 430 | str: Response from Tello, 'OK' or 'FALSE'. 431 | 432 | """ 433 | return self.move('left', distance) 434 | 435 | def move_right(self, distance): 436 | """Moves right for a distance. 437 | 438 | See comments for Tello.move(). 439 | 440 | Args: 441 | distance (int): Distance to move. 442 | 443 | """ 444 | return self.move('right', distance) 445 | 446 | def move_up(self, distance): 447 | """Moves up for a distance. 448 | 449 | See comments for Tello.move(). 450 | 451 | Args: 452 | distance (int): Distance to move. 453 | 454 | Returns: 455 | str: Response from Tello, 'OK' or 'FALSE'. 456 | 457 | """ 458 | 459 | return self.move('up', distance) 460 | -------------------------------------------------------------------------------- /Tello_Video/tello_control_ui.py: -------------------------------------------------------------------------------- 1 | from PIL import Image 2 | from PIL import ImageTk 3 | import Tkinter as tki 4 | from Tkinter import Toplevel, Scale 5 | import threading 6 | import datetime 7 | import cv2 8 | import os 9 | import time 10 | import platform 11 | 12 | class TelloUI: 13 | """Wrapper class to enable the GUI.""" 14 | 15 | def __init__(self,tello,outputpath): 16 | """ 17 | Initial all the element of the GUI,support by Tkinter 18 | 19 | :param tello: class interacts with the Tello drone. 20 | 21 | Raises: 22 | RuntimeError: If the Tello rejects the attempt to enter command mode. 23 | """ 24 | 25 | self.tello = tello # videostream device 26 | self.outputPath = outputpath # the path that save pictures created by clicking the takeSnapshot button 27 | self.frame = None # frame read from h264decoder and used for pose recognition 28 | self.thread = None # thread of the Tkinter mainloop 29 | self.stopEvent = None 30 | 31 | # control variables 32 | self.distance = 0.1 # default distance for 'move' cmd 33 | self.degree = 30 # default degree for 'cw' or 'ccw' cmd 34 | 35 | # if the flag is TRUE,the auto-takeoff thread will stop waiting for the response from tello 36 | self.quit_waiting_flag = False 37 | 38 | # initialize the root window and image panel 39 | self.root = tki.Tk() 40 | self.panel = None 41 | 42 | # create buttons 43 | self.btn_snapshot = tki.Button(self.root, text="Snapshot!", 44 | command=self.takeSnapshot) 45 | self.btn_snapshot.pack(side="bottom", fill="both", 46 | expand="yes", padx=10, pady=5) 47 | 48 | self.btn_pause = tki.Button(self.root, text="Pause", relief="raised", command=self.pauseVideo) 49 | self.btn_pause.pack(side="bottom", fill="both", 50 | expand="yes", padx=10, pady=5) 51 | 52 | self.btn_landing = tki.Button( 53 | self.root, text="Open Command Panel", relief="raised", command=self.openCmdWindow) 54 | self.btn_landing.pack(side="bottom", fill="both", 55 | expand="yes", padx=10, pady=5) 56 | 57 | # start a thread that constantly pools the video sensor for 58 | # the most recently read frame 59 | self.stopEvent = threading.Event() 60 | self.thread = threading.Thread(target=self.videoLoop, args=()) 61 | self.thread.start() 62 | 63 | # set a callback to handle when the window is closed 64 | self.root.wm_title("TELLO Controller") 65 | self.root.wm_protocol("WM_DELETE_WINDOW", self.onClose) 66 | 67 | # the sending_command will send command to tello every 5 seconds 68 | self.sending_command_thread = threading.Thread(target = self._sendingCommand) 69 | def videoLoop(self): 70 | """ 71 | The mainloop thread of Tkinter 72 | Raises: 73 | RuntimeError: To get around a RunTime error that Tkinter throws due to threading. 74 | """ 75 | try: 76 | # start the thread that get GUI image and drwa skeleton 77 | time.sleep(0.5) 78 | self.sending_command_thread.start() 79 | while not self.stopEvent.is_set(): 80 | system = platform.system() 81 | 82 | # read the frame for GUI show 83 | self.frame = self.tello.read() 84 | if self.frame is None or self.frame.size == 0: 85 | continue 86 | 87 | # transfer the format from frame to image 88 | image = Image.fromarray(self.frame) 89 | 90 | # we found compatibility problem between Tkinter,PIL and Macos,and it will 91 | # sometimes result the very long preriod of the "ImageTk.PhotoImage" function, 92 | # so for Macos,we start a new thread to execute the _updateGUIImage function. 93 | if system =="Windows" or system =="Linux": 94 | self._updateGUIImage(image) 95 | 96 | else: 97 | thread_tmp = threading.Thread(target=self._updateGUIImage,args=(image,)) 98 | thread_tmp.start() 99 | time.sleep(0.03) 100 | except RuntimeError, e: 101 | print("[INFO] caught a RuntimeError") 102 | 103 | 104 | def _updateGUIImage(self,image): 105 | """ 106 | Main operation to initial the object of image,and update the GUI panel 107 | """ 108 | image = ImageTk.PhotoImage(image) 109 | # if the panel none ,we need to initial it 110 | if self.panel is None: 111 | self.panel = tki.Label(image=image) 112 | self.panel.image = image 113 | self.panel.pack(side="left", padx=10, pady=10) 114 | # otherwise, simply update the panel 115 | else: 116 | self.panel.configure(image=image) 117 | self.panel.image = image 118 | 119 | 120 | def _sendingCommand(self): 121 | """ 122 | start a while loop that sends 'command' to tello every 5 second 123 | """ 124 | 125 | while True: 126 | self.tello.send_command('command') 127 | time.sleep(5) 128 | 129 | def _setQuitWaitingFlag(self): 130 | """ 131 | set the variable as TRUE,it will stop computer waiting for response from tello 132 | """ 133 | self.quit_waiting_flag = True 134 | 135 | def openCmdWindow(self): 136 | """ 137 | open the cmd window and initial all the button and text 138 | """ 139 | panel = Toplevel(self.root) 140 | panel.wm_title("Command Panel") 141 | 142 | # create text input entry 143 | text0 = tki.Label(panel, 144 | text='This Controller map keyboard inputs to Tello control commands\n' 145 | 'Adjust the trackbar to reset distance and degree parameter', 146 | font='Helvetica 10 bold' 147 | ) 148 | text0.pack(side='top') 149 | 150 | text1 = tki.Label(panel, text= 151 | 'W - Move Tello Up\t\t\tArrow Up - Move Tello Forward\n' 152 | 'S - Move Tello Down\t\t\tArrow Down - Move Tello Backward\n' 153 | 'A - Rotate Tello Counter-Clockwise\tArrow Left - Move Tello Left\n' 154 | 'D - Rotate Tello Clockwise\t\tArrow Right - Move Tello Right', 155 | justify="left") 156 | text1.pack(side="top") 157 | 158 | self.btn_landing = tki.Button( 159 | panel, text="Land", relief="raised", command=self.telloLanding) 160 | self.btn_landing.pack(side="bottom", fill="both", 161 | expand="yes", padx=10, pady=5) 162 | 163 | self.btn_takeoff = tki.Button( 164 | panel, text="Takeoff", relief="raised", command=self.telloTakeOff) 165 | self.btn_takeoff.pack(side="bottom", fill="both", 166 | expand="yes", padx=10, pady=5) 167 | 168 | # binding arrow keys to drone control 169 | self.tmp_f = tki.Frame(panel, width=100, height=2) 170 | self.tmp_f.bind('', self.on_keypress_w) 171 | self.tmp_f.bind('', self.on_keypress_s) 172 | self.tmp_f.bind('', self.on_keypress_a) 173 | self.tmp_f.bind('', self.on_keypress_d) 174 | self.tmp_f.bind('', self.on_keypress_up) 175 | self.tmp_f.bind('', self.on_keypress_down) 176 | self.tmp_f.bind('', self.on_keypress_left) 177 | self.tmp_f.bind('', self.on_keypress_right) 178 | self.tmp_f.pack(side="bottom") 179 | self.tmp_f.focus_set() 180 | 181 | self.btn_landing = tki.Button( 182 | panel, text="Flip", relief="raised", command=self.openFlipWindow) 183 | self.btn_landing.pack(side="bottom", fill="both", 184 | expand="yes", padx=10, pady=5) 185 | 186 | self.distance_bar = Scale(panel, from_=0.02, to=5, tickinterval=0.01, digits=3, label='Distance(m)', 187 | resolution=0.01) 188 | self.distance_bar.set(0.2) 189 | self.distance_bar.pack(side="left") 190 | 191 | self.btn_distance = tki.Button(panel, text="Reset Distance", relief="raised", 192 | command=self.updateDistancebar, 193 | ) 194 | self.btn_distance.pack(side="left", fill="both", 195 | expand="yes", padx=10, pady=5) 196 | 197 | self.degree_bar = Scale(panel, from_=1, to=360, tickinterval=10, label='Degree') 198 | self.degree_bar.set(30) 199 | self.degree_bar.pack(side="right") 200 | 201 | self.btn_distance = tki.Button(panel, text="Reset Degree", relief="raised", command=self.updateDegreebar) 202 | self.btn_distance.pack(side="right", fill="both", 203 | expand="yes", padx=10, pady=5) 204 | 205 | def openFlipWindow(self): 206 | """ 207 | open the flip window and initial all the button and text 208 | """ 209 | 210 | panel = Toplevel(self.root) 211 | panel.wm_title("Gesture Recognition") 212 | 213 | self.btn_flipl = tki.Button( 214 | panel, text="Flip Left", relief="raised", command=self.telloFlip_l) 215 | self.btn_flipl.pack(side="bottom", fill="both", 216 | expand="yes", padx=10, pady=5) 217 | 218 | self.btn_flipr = tki.Button( 219 | panel, text="Flip Right", relief="raised", command=self.telloFlip_r) 220 | self.btn_flipr.pack(side="bottom", fill="both", 221 | expand="yes", padx=10, pady=5) 222 | 223 | self.btn_flipf = tki.Button( 224 | panel, text="Flip Forward", relief="raised", command=self.telloFlip_f) 225 | self.btn_flipf.pack(side="bottom", fill="both", 226 | expand="yes", padx=10, pady=5) 227 | 228 | self.btn_flipb = tki.Button( 229 | panel, text="Flip Backward", relief="raised", command=self.telloFlip_b) 230 | self.btn_flipb.pack(side="bottom", fill="both", 231 | expand="yes", padx=10, pady=5) 232 | 233 | def takeSnapshot(self): 234 | """ 235 | save the current frame of the video as a jpg file and put it into outputpath 236 | """ 237 | 238 | # grab the current timestamp and use it to construct the filename 239 | ts = datetime.datetime.now() 240 | filename = "{}.jpg".format(ts.strftime("%Y-%m-%d_%H-%M-%S")) 241 | 242 | p = os.path.sep.join((self.outputPath, filename)) 243 | 244 | # save the file 245 | cv2.imwrite(p, cv2.cvtColor(self.frame, cv2.COLOR_RGB2BGR)) 246 | print("[INFO] saved {}".format(filename)) 247 | 248 | 249 | def pauseVideo(self): 250 | """ 251 | Toggle the freeze/unfreze of video 252 | """ 253 | if self.btn_pause.config('relief')[-1] == 'sunken': 254 | self.btn_pause.config(relief="raised") 255 | self.tello.video_freeze(False) 256 | else: 257 | self.btn_pause.config(relief="sunken") 258 | self.tello.video_freeze(True) 259 | 260 | def telloTakeOff(self): 261 | return self.tello.takeoff() 262 | 263 | def telloLanding(self): 264 | return self.tello.land() 265 | 266 | def telloFlip_l(self): 267 | return self.tello.flip('l') 268 | 269 | def telloFlip_r(self): 270 | return self.tello.flip('r') 271 | 272 | def telloFlip_f(self): 273 | return self.tello.flip('f') 274 | 275 | def telloFlip_b(self): 276 | return self.tello.flip('b') 277 | 278 | def telloCW(self, degree): 279 | return self.tello.rotate_cw(degree) 280 | 281 | def telloCCW(self, degree): 282 | return self.tello.rotate_ccw(degree) 283 | 284 | def telloMoveForward(self, distance): 285 | return self.tello.move_forward(distance) 286 | 287 | def telloMoveBackward(self, distance): 288 | return self.tello.move_backward(distance) 289 | 290 | def telloMoveLeft(self, distance): 291 | return self.tello.move_left(distance) 292 | 293 | def telloMoveRight(self, distance): 294 | return self.tello.move_right(distance) 295 | 296 | def telloUp(self, dist): 297 | return self.tello.move_up(dist) 298 | 299 | def telloDown(self, dist): 300 | return self.tello.move_down(dist) 301 | 302 | def updateTrackBar(self): 303 | self.my_tello_hand.setThr(self.hand_thr_bar.get()) 304 | 305 | def updateDistancebar(self): 306 | self.distance = self.distance_bar.get() 307 | print 'reset distance to %.1f' % self.distance 308 | 309 | def updateDegreebar(self): 310 | self.degree = self.degree_bar.get() 311 | print 'reset distance to %d' % self.degree 312 | 313 | def on_keypress_w(self, event): 314 | print "up %d m" % self.distance 315 | self.telloUp(self.distance) 316 | 317 | def on_keypress_s(self, event): 318 | print "down %d m" % self.distance 319 | self.telloDown(self.distance) 320 | 321 | def on_keypress_a(self, event): 322 | print "ccw %d degree" % self.degree 323 | self.tello.rotate_ccw(self.degree) 324 | 325 | def on_keypress_d(self, event): 326 | print "cw %d m" % self.degree 327 | self.tello.rotate_cw(self.degree) 328 | 329 | def on_keypress_up(self, event): 330 | print "forward %d m" % self.distance 331 | self.telloMoveForward(self.distance) 332 | 333 | def on_keypress_down(self, event): 334 | print "backward %d m" % self.distance 335 | self.telloMoveBackward(self.distance) 336 | 337 | def on_keypress_left(self, event): 338 | print "left %d m" % self.distance 339 | self.telloMoveLeft(self.distance) 340 | 341 | def on_keypress_right(self, event): 342 | print "right %d m" % self.distance 343 | self.telloMoveRight(self.distance) 344 | 345 | def on_keypress_enter(self, event): 346 | if self.frame is not None: 347 | self.registerFace() 348 | self.tmp_f.focus_set() 349 | 350 | def onClose(self): 351 | """ 352 | set the stop event, cleanup the camera, and allow the rest of 353 | 354 | the quit process to continue 355 | """ 356 | print("[INFO] closing...") 357 | self.stopEvent.set() 358 | del self.tello 359 | self.root.quit() 360 | 361 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/LICENSE.md: -------------------------------------------------------------------------------- 1 | Sample code is offered under MIT License (See below). 2 | 3 | Copyright 2018 Ryze 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a 6 | copy of this software and associated documentation files (the "Software"), 7 | to deal in the Software without restriction, including without limitation 8 | the rights to use, copy, modify, merge, publish, distribute, sublicense, 9 | and/or sell copies of the Software, and to permit persons to whom the 10 | Software is furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included 13 | in all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS 16 | OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 18 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 19 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 20 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 21 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/README.md: -------------------------------------------------------------------------------- 1 | # Tello-SimplePoseRecognition 2 | 3 | This is an example using the Tello SDK v1.3.0.0 and above to receive video stream from Tello camera and do real-time body pose recognition processing on PC. You're welcome to fork/clone/copy this example and let Tello fly creatively. 4 | 5 | - Written in Python 2.7 6 | - Tello SDK v1.3.0.0 and above(with h.264 video streaming) 7 | - This example includes a simple UI build with Tkinter to interact with Tello 8 | - Interactive control of Tello based on human movement is achieved via body pose recognition module. 9 | 10 | ## Prerequisites 11 | 12 | - Python2.7 13 | - pip 14 | - Python OpenCV 15 | - Numpy 16 | - PIL 17 | - libboost-python 18 | - Tkinter 19 | - homebrew(for mac) 20 | - Python h264 decoder 21 | - 22 | 23 | ## Installation 24 | 25 | In order to facilitate you to install python2.7 and various dependencies, we have written a one-click installation script for windows, Linux and macos. You can choose to run this script for the one-click installation, or you can download python2.7 and related libraries and dependencies online. If you have questions about the actions that the script performs, you can open the script with an editor and look up the comments for each instruction in the script. In addition, we have additionally written an uninstall script that cleans and restores all downloaded and configured content from the one-click installation script. 26 | 27 | - **Windows** 28 | 29 | Go to the "install\Windows" folder,select and run the correct "windows_install.bat" according to your computer operating system bits. 30 | 31 | - **Linux (Ubuntu 14.04 and above)** 32 | 33 | Go to the "install\Linux" folder in command line, run 34 | 35 | ``` 36 | chmod +x linux_install.sh 37 | ./linux_install.sh 38 | ``` 39 | 40 | - **Mac** 41 | 42 | 1. Make sure you have the latest Xcode command line tools installed. If not, you might need to update your OS X and XCode to the latest version in order to compile the h264 decoder module 43 | 2. Go to the "install\Mac" folder folder in command line, run 44 | 45 | ``` 46 | chmod a+x ./mac_install.sh 47 | ./mac_install.sh 48 | ``` 49 | 50 | If you see no errors during installation, you are good to go! 51 | 52 | ## Get the Model 53 | 54 | You can get the pose recognition model by run the script named "getModels.bat" or "getModels.bat"(according to your os type) under the path of "./model/".And it will take some time to download the model. 55 | 56 | 57 | ## Run the project 58 | - **Step1**. Turn on Tello and connect your computer device to Tello via wifi. 59 | 60 | 61 | - **Step2**. Open project folder in terminal. Run: 62 | 63 | ``` 64 | python main.py 65 | ``` 66 | 67 | - **Step3**. A UI will show up, you can now: 68 | 69 | - Watch live video stream from the Tello camera; 70 | - Take snapshot and save jpg to local folder; 71 | - Open Command Panel, which allows you to: 72 | - Take Off 73 | - After you click this button, tello will receive the takeoff command from computer. If the battery is low, it will not take off and will give a corresponding reminder. When the tello completes the takeoff command (about 1.2 meters), in order to make the arms of the person controlling the tello can enter the field of the tello's front camera completely, the tello will fly up again for about 0.5 meters to compensate for the height. In this process,if the wifi communication between the tello and the computer is interrupted or lost, the tello will execute the land command.If tello move up again sucessfully,the thread that sends command of 'command' every 5 seconds will be started to prevent tello from landing. In order to let tello fly up to a really proper height,you should put the tello on the ground at 74 | the beginning. 75 | - Land 76 | - Flip (in forward, backward, left and right direction) 77 | - Control Tello using keyboard inputs: 78 | - **[key-Up]** move forward 20cm 79 | - **[key-Down]** move backward 20cm 80 | - **[key-Left]** move left 20 cm 81 | - **[key-Right]** move right 20 cm 82 | - **[key-w]** move up 20cm 83 | - **[key-s]** move down 20cm 84 | - **[key-a]** rotate counter-clockwise by 30 degree 85 | - **[key-d]** rotate clockwise by 30 degree 86 | - You can also adjust the **distance** and **degree** via the trackbar and hit the "reset distance" or "reset degree" button to customize your own control. 87 | 88 | - Turn on **Pose Recognition** mode. A 17-joints skeleton based on your body will appear on the screen. Raise your arm UP or FLAT (like a "Y" or "T"), Tello will move forward 0.5 meters. Raise both your arm DOWN (your body be like '/|\'), Tello will move back 0.5 meters.Raise your arm bending(your body be like 'v|v'),Tello will land. 89 | 90 | 91 | ## Project Description 92 | 93 | ### tello.py - class Tello 94 | 95 | Wrapper class to interact with Tello drone. 96 | Modified from 97 | 98 | The object starts 2 threads: 99 | 100 | 1. thread for receiving command response from Tello 101 | 2. thread for receiving video stream 102 | 103 | You can use **read()** to read the last frame from Tello camera, and pause the video by setting **video_freeze(is_freeze=True)**. 104 | 105 | ### tello_control_ui.py - class TelloUI 106 | 107 | Modified from: https://www.pyimagesearch.com/2016/05/30/displaying-a-video-feed-with-opencv-and-tkinter/ 108 | 109 | Build with Tkinter. Display video, control video play/pause and control Tello using buttons and arrow keys. 110 | The object starts 4 threads: 111 | 112 | 1. thread for sending command 'command' to Tello every 5s; Starts after take off. This prevents Tello from landing automatically if no commands are sent within 15s. 113 | 2. thread for starting a Tello automatically takeoff process.After open the Command Panel and click the Takeoff button,this 114 | thread will be started.Computer will send the 'takeoff' command and check a series of response from Tello.If the corresponding response is received by computer,the thread will send a 'moveup'command to control Tello to a suitable height.Finally,the thread 115 | will start the sending commang thread to preventing Tello from landing automatically. 116 | 3. thread for displaying video.This thread will read the video stream from the h264decoder software module,with the format of frame.If the pose recognition mode are open,a 17-joints skeleton will be draw on the image. 117 | 4. thread for initializing the photo image object.This thread is only enabled on Macos.It's a short-term solution to some compatibility problems between Macos and some plugins such as Tkinter and PIL. 118 | 119 | ### tello_pose.py - class Tello_Pose 120 | Code modifed from:https://github.com/spmallick/learnopencv/tree/master/OpenPose 121 | Using pre-trained caffe model from . 122 | 123 | Detect Body Pose and draw a 17-joints skeleton. Analyze pose by calculating angles between joints. 124 | 125 | ### h264decoder - class libh264decoder 126 | 127 | From . 128 | 129 | A c++ based class that decodes raw h264 data. This module interacts with python language via python-libboost library, and its decoding functionality is based on ffmpeg library. 130 | 131 | After compilation, a libh264decoder.so or libh264decoder.pyd file will be placed in the working directory so that the main python file can reference it. 132 | 133 | If you have to compile it from source,with Linux or Mac,you can: 134 | 135 | ``` 136 | cd h264decoder 137 | mkdir build 138 | cd build 139 | cmake .. 140 | make 141 | cp libh264decoder.so ../../ 142 | ``` 143 | With Windows,you can create a project through visual studio, add files in h264decoder and dependencies such as ffmpeg and libboost, compile the project and generate a libh264decoder.pyd file.We have generated a libh264decoder.pyd and put it in the "\h264decoder\Windows"foleder so that you can copy put it to "python/site-package". 144 | 145 | 146 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/h264decoder/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8) 2 | project(python_h264decoder) 3 | 4 | set(Python_ADDITIONAL_VERSIONS 2.7) 5 | 6 | if(UNIX AND NOT APPLE) 7 | set(LINUX TRUE) 8 | endif() 9 | 10 | if(APPLE) 11 | set(CMAKE_SHARED_LIBRARY_SUFFIX ".so") 12 | find_package(Boost REQUIRED COMPONENTS python27) 13 | elseif(LINUX) 14 | find_package(Boost REQUIRED COMPONENTS "python") 15 | endif(APPLE) 16 | 17 | 18 | 19 | find_package(PythonInterp 2.7 REQUIRED) 20 | find_package(PythonLibs 2.7 REQUIRED ) 21 | 22 | 23 | include_directories(${PYTHON_INCLUDE_DIRS}) 24 | include_directories(${Boost_INCLUDE_DIRS}) 25 | link_directories(${Boost_LIBRARY_DIRS}) 26 | 27 | add_compile_options ("-std=c++0x") 28 | 29 | add_library(h264decoder SHARED h264decoder.cpp h264decoder_python.cpp) 30 | 31 | if(APPLE) 32 | target_link_libraries(h264decoder avcodec swscale avutil ${Boost_LIBRARIES} ${Boost_PYTHON_LIBRARY_RELEASE} ${PYTHON_LIBRARIES}) 33 | elseif(LINUX) 34 | target_link_libraries(h264decoder avcodec swscale avutil ${Boost_PYTHON_LIBRARY_RELEASE} ${PYTHON_LIBRARIES}) 35 | endif(APPLE) 36 | 37 | add_custom_command(TARGET h264decoder POST_BUILD 38 | COMMAND ${CMAKE_COMMAND} -E create_symlink 39 | ${CMAKE_BINARY_DIR}/libh264decoder.so ${CMAKE_SOURCE_DIR}/libh264decoder.so) 40 | install(TARGETS h264decoder LIBRARY DESTINATION .) 41 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/h264decoder/Linux/libh264decoder.so: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/Tello_Video_With_Pose_Recognition/h264decoder/Linux/libh264decoder.so -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/h264decoder/Mac/libh264decoder.so: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/Tello_Video_With_Pose_Recognition/h264decoder/Mac/libh264decoder.so -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/h264decoder/Windows/x64/libh264decoder.pyd: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/Tello_Video_With_Pose_Recognition/h264decoder/Windows/x64/libh264decoder.pyd -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/h264decoder/Windows/x86/libh264decoder.pyd: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/Tello_Video_With_Pose_Recognition/h264decoder/Windows/x86/libh264decoder.pyd -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/h264decoder/h264decoder.cpp: -------------------------------------------------------------------------------- 1 | extern "C" { 2 | #include 3 | #include 4 | #include 5 | #include 6 | } 7 | 8 | #ifndef PIX_FMT_RGB24 9 | #define PIX_FMT_RGB24 AV_PIX_FMT_RGB24 10 | #endif 11 | 12 | #ifndef CODEC_CAP_TRUNCATED 13 | #define CODEC_CAP_TRUNCATED AV_CODEC_CAP_TRUNCATED 14 | #endif 15 | 16 | #ifndef CODEC_FLAG_TRUNCATED 17 | #define CODEC_FLAG_TRUNCATED AV_CODEC_FLAG_TRUNCATED 18 | #endif 19 | 20 | #include "h264decoder.hpp" 21 | #include 22 | 23 | typedef unsigned char ubyte; 24 | 25 | /* For backward compatibility with release 9 or so of libav */ 26 | #if (LIBAVCODEC_VERSION_MAJOR <= 54) 27 | # define av_frame_alloc avcodec_alloc_frame 28 | # define av_frame_free avcodec_free_frame 29 | #endif 30 | 31 | 32 | H264Decoder::H264Decoder() 33 | { 34 | avcodec_register_all(); 35 | 36 | codec = avcodec_find_decoder(AV_CODEC_ID_H264); 37 | if (!codec) 38 | throw H264InitFailure("cannot find decoder"); 39 | 40 | context = avcodec_alloc_context3(codec); 41 | if (!context) 42 | throw H264InitFailure("cannot allocate context"); 43 | 44 | if(codec->capabilities & CODEC_CAP_TRUNCATED) { 45 | context->flags |= CODEC_FLAG_TRUNCATED; 46 | } 47 | 48 | int err = avcodec_open2(context, codec, nullptr); 49 | if (err < 0) 50 | throw H264InitFailure("cannot open context"); 51 | 52 | parser = av_parser_init(AV_CODEC_ID_H264); 53 | if (!parser) 54 | throw H264InitFailure("cannot init parser"); 55 | 56 | frame = av_frame_alloc(); 57 | if (!frame) 58 | throw H264InitFailure("cannot allocate frame"); 59 | 60 | #if 1 61 | pkt = new AVPacket; 62 | if (!pkt) 63 | throw H264InitFailure("cannot allocate packet"); 64 | av_init_packet(pkt); 65 | #endif 66 | } 67 | 68 | 69 | H264Decoder::~H264Decoder() 70 | { 71 | av_parser_close(parser); 72 | avcodec_close(context); 73 | av_free(context); 74 | av_frame_free(&frame); 75 | #if 1 76 | delete pkt; 77 | #endif 78 | } 79 | 80 | 81 | ssize_t H264Decoder::parse(const ubyte* in_data, ssize_t in_size) 82 | { 83 | auto nread = av_parser_parse2(parser, context, &pkt->data, &pkt->size, 84 | in_data, in_size, 85 | 0, 0, AV_NOPTS_VALUE); 86 | return nread; 87 | } 88 | 89 | 90 | bool H264Decoder::is_frame_available() const 91 | { 92 | return pkt->size > 0; 93 | } 94 | 95 | 96 | const AVFrame& H264Decoder::decode_frame() 97 | { 98 | int got_picture = 0; 99 | int nread = avcodec_decode_video2(context, frame, &got_picture, pkt); 100 | if (nread < 0 || got_picture == 0) 101 | throw H264DecodeFailure("error decoding frame\n"); 102 | return *frame; 103 | } 104 | 105 | 106 | ConverterRGB24::ConverterRGB24() 107 | { 108 | framergb = av_frame_alloc(); 109 | if (!framergb) 110 | throw H264DecodeFailure("cannot allocate frame"); 111 | context = nullptr; 112 | } 113 | 114 | ConverterRGB24::~ConverterRGB24() 115 | { 116 | sws_freeContext(context); 117 | av_frame_free(&framergb); 118 | } 119 | 120 | 121 | const AVFrame& ConverterRGB24::convert(const AVFrame &frame, ubyte* out_rgb) 122 | { 123 | int w = frame.width; 124 | int h = frame.height; 125 | int pix_fmt = frame.format; 126 | 127 | context = sws_getCachedContext(context, 128 | w, h, (AVPixelFormat)pix_fmt, 129 | w, h, PIX_FMT_RGB24, SWS_BILINEAR, 130 | nullptr, nullptr, nullptr); 131 | if (!context) 132 | throw H264DecodeFailure("cannot allocate context"); 133 | 134 | // Setup framergb with out_rgb as external buffer. Also say that we want RGB24 output. 135 | avpicture_fill((AVPicture*)framergb, out_rgb, PIX_FMT_RGB24, w, h); 136 | // Do the conversion. 137 | sws_scale(context, frame.data, frame.linesize, 0, h, 138 | framergb->data, framergb->linesize); 139 | framergb->width = w; 140 | framergb->height = h; 141 | return *framergb; 142 | } 143 | 144 | /* 145 | Determine required size of framebuffer. 146 | 147 | avpicture_get_size is used in http://dranger.com/ffmpeg/tutorial01.html 148 | to do this. However, avpicture_get_size returns the size of a compact 149 | representation, without padding bytes. Since we use avpicture_fill to 150 | fill the buffer we should also use it to determine the required size. 151 | */ 152 | int ConverterRGB24::predict_size(int w, int h) 153 | { 154 | return avpicture_fill((AVPicture*)framergb, nullptr, PIX_FMT_RGB24, w, h); 155 | } 156 | 157 | 158 | 159 | std::pair width_height(const AVFrame& f) 160 | { 161 | return std::make_pair(f.width, f.height); 162 | } 163 | 164 | int row_size(const AVFrame& f) 165 | { 166 | return f.linesize[0]; 167 | } 168 | 169 | 170 | void disable_logging() 171 | { 172 | av_log_set_level(AV_LOG_QUIET); 173 | } 174 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/h264decoder/h264decoder.hpp: -------------------------------------------------------------------------------- 1 | #pragma once 2 | /* 3 | This h264 decoder class is just a thin wrapper around libav 4 | functions to decode h264 videos. It would have been easy to use 5 | libav directly in the python module code but I like to keep these 6 | things separate. 7 | 8 | It is mostly based on roxlu's code. See 9 | http://roxlu.com/2014/039/decoding-h264-and-yuv420p-playback 10 | 11 | However, in contrast to roxlu's code the color space conversion is 12 | done by libav functions - so on the CPU, I suppose. 13 | 14 | Most functions/members throw exceptions. This way, error states are 15 | conveniently forwarded to python via the exception translation 16 | mechanisms of boost::python. 17 | */ 18 | 19 | // for ssize_t (signed int type as large as pointer type) 20 | #include 21 | #include 22 | 23 | struct AVCodecContext; 24 | struct AVFrame; 25 | struct AVCodec; 26 | struct AVCodecParserContext; 27 | struct SwsContext; 28 | struct AVPacket; 29 | 30 | 31 | class H264Exception : public std::runtime_error 32 | { 33 | public: 34 | H264Exception(const char* s) : std::runtime_error(s) {} 35 | }; 36 | 37 | class H264InitFailure : public H264Exception 38 | { 39 | public: 40 | H264InitFailure(const char* s) : H264Exception(s) {} 41 | }; 42 | 43 | class H264DecodeFailure : public H264Exception 44 | { 45 | public: 46 | H264DecodeFailure(const char* s) : H264Exception(s) {} 47 | }; 48 | 49 | 50 | class H264Decoder 51 | { 52 | /* Persistent things here, using RAII for cleanup. */ 53 | AVCodecContext *context; 54 | AVFrame *frame; 55 | AVCodec *codec; 56 | AVCodecParserContext *parser; 57 | /* In the documentation example on the github master branch, the 58 | packet is put on the heap. This is done here to store the pointers 59 | to the encoded data, which must be kept around between calls to 60 | parse- and decode frame. In release 11 it is put on the stack, too. 61 | */ 62 | AVPacket *pkt; 63 | public: 64 | H264Decoder(); 65 | ~H264Decoder(); 66 | /* First, parse a continuous data stream, dividing it into 67 | packets. When there is enough data to form a new frame, decode 68 | the data and return the frame. parse returns the number 69 | of consumed bytes of the input stream. It stops consuming 70 | bytes at frame boundaries. 71 | */ 72 | ssize_t parse(const unsigned char* in_data, ssize_t in_size); 73 | bool is_frame_available() const; 74 | const AVFrame& decode_frame(); 75 | }; 76 | 77 | // TODO: Rename to OutputStage or so?! 78 | class ConverterRGB24 79 | { 80 | SwsContext *context; 81 | AVFrame *framergb; 82 | 83 | public: 84 | ConverterRGB24(); 85 | ~ConverterRGB24(); 86 | 87 | /* Returns, given a width and height, 88 | how many bytes the frame buffer is going to need. */ 89 | int predict_size(int w, int h); 90 | /* Given a decoded frame, convert it to RGB format and fill 91 | out_rgb with the result. Returns a AVFrame structure holding 92 | additional information about the RGB frame, such as the number of 93 | bytes in a row and so on. */ 94 | const AVFrame& convert(const AVFrame &frame, unsigned char* out_rgb); 95 | }; 96 | 97 | void disable_logging(); 98 | 99 | /* Wrappers, so we don't have to include libav headers. */ 100 | std::pair width_height(const AVFrame&); 101 | int row_size(const AVFrame&); 102 | 103 | /* all the documentation links 104 | * My version of libav on ubuntu 16 appears to be from the release/11 branch on github 105 | * Video decoding example: https://libav.org/documentation/doxygen/release/11/avcodec_8c_source.html#l00455 106 | * https://libav.org/documentation/doxygen/release/9/group__lavc__decoding.html 107 | * https://libav.org/documentation/doxygen/release/11/group__lavc__parsing.html 108 | * https://libav.org/documentation/doxygen/release/9/swscale_8h.html 109 | * https://libav.org/documentation/doxygen/release/9/group__lavu.html 110 | * https://libav.org/documentation/doxygen/release/9/group__lavc__picture.html 111 | * http://dranger.com/ffmpeg/tutorial01.html 112 | */ -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/h264decoder/h264decoder_python.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | 6 | // python string api, see 7 | // https://docs.python.org/2/c-api/string.html 8 | extern "C" { 9 | #include 10 | } 11 | 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | namespace py = boost::python; 18 | 19 | #include "h264decoder.hpp" 20 | 21 | using ubyte = unsigned char; 22 | 23 | 24 | class GILScopedReverseLock 25 | { 26 | // see https://docs.python.org/2/c-api/init.html (Releasing the GIL ...) 27 | public: 28 | GILScopedReverseLock() 29 | : state(nullptr) 30 | { 31 | unlock(); 32 | } 33 | 34 | ~GILScopedReverseLock() 35 | { 36 | lock(); 37 | } 38 | 39 | void lock() 40 | { 41 | // Allow successive calls to lock. 42 | // E.g. lock() followed by destructor. 43 | if (state != nullptr) 44 | { 45 | PyEval_RestoreThread(state); 46 | state = nullptr; 47 | } 48 | } 49 | 50 | void unlock() 51 | { 52 | assert (state == nullptr); 53 | state = PyEval_SaveThread(); 54 | } 55 | 56 | GILScopedReverseLock(const GILScopedReverseLock &) = delete; 57 | GILScopedReverseLock(const GILScopedReverseLock &&) = delete; 58 | GILScopedReverseLock operator=(const GILScopedReverseLock &) = delete; 59 | GILScopedReverseLock operator=(const GILScopedReverseLock &&) = delete; 60 | private: 61 | PyThreadState *state; 62 | }; 63 | 64 | 65 | /* The class wrapped in python via boost::python */ 66 | class PyH264Decoder 67 | { 68 | H264Decoder decoder; 69 | ConverterRGB24 converter; 70 | 71 | /* Extract frames from input stream. Stops at frame boundaries and returns the number of consumed bytes 72 | * in num_consumed. 73 | * 74 | * If a frame is completed, is_frame_available is set to true, and the returned python tuple contains 75 | * formation about the frame as well as the frame buffer memory. 76 | * 77 | * Else, i.e. all data in the buffer is consumed, is_frame_available is set to false. The returned tuple 78 | * contains dummy data. 79 | */ 80 | py::tuple decode_frame_impl(const ubyte *data, ssize_t num, ssize_t &num_consumed, bool &is_frame_available); 81 | 82 | public: 83 | /* Decoding style analogous to c/c++ way. Stop at frame boundaries. 84 | * Return tuple containing frame data as above as nested tuple, and an integer telling how many bytes were consumed. */ 85 | py::tuple decode_frame(const py::str &data_in_str); 86 | /* Process all the input data and return a list of all contained frames. */ 87 | py::list decode(const py::str &data_in_str); 88 | }; 89 | 90 | 91 | py::tuple PyH264Decoder::decode_frame_impl(const ubyte *data_in, ssize_t len, ssize_t &num_consumed, bool &is_frame_available) 92 | { 93 | GILScopedReverseLock gilguard; 94 | num_consumed = decoder.parse((ubyte*)data_in, len); 95 | 96 | if (is_frame_available = decoder.is_frame_available()) 97 | { 98 | const auto &frame = decoder.decode_frame(); 99 | int w, h; std::tie(w,h) = width_height(frame); 100 | Py_ssize_t out_size = converter.predict_size(w,h); 101 | 102 | gilguard.lock(); 103 | // Construction of py::handle causes ... TODO: WHAT? No increase of ref count ?! 104 | py::object py_out_str(py::handle<>(PyString_FromStringAndSize(NULL, out_size))); 105 | char* out_buffer = PyString_AsString(py_out_str.ptr()); 106 | 107 | gilguard.unlock(); 108 | const auto &rgbframe = converter.convert(frame, (ubyte*)out_buffer); 109 | 110 | gilguard.lock(); 111 | return py::make_tuple(py_out_str, w, h, row_size(rgbframe)); 112 | } 113 | else 114 | { 115 | gilguard.lock(); 116 | return py::make_tuple(py::object(), 0, 0, 0); 117 | } 118 | } 119 | 120 | 121 | py::tuple PyH264Decoder::decode_frame(const py::str &data_in_str) 122 | { 123 | ssize_t len = PyString_Size(data_in_str.ptr()); 124 | const ubyte* data_in = (const ubyte*)(PyString_AsString(data_in_str.ptr())); 125 | 126 | ssize_t num_consumed = 0; 127 | bool is_frame_available = false; 128 | auto frame = decode_frame_impl(data_in, len, num_consumed, is_frame_available); 129 | 130 | return py::make_tuple(frame, num_consumed); 131 | } 132 | 133 | 134 | py::list PyH264Decoder::decode(const py::str &data_in_str) 135 | { 136 | ssize_t len = PyString_Size(data_in_str.ptr()); 137 | const ubyte* data_in = (const ubyte*)(PyString_AsString(data_in_str.ptr())); 138 | 139 | py::list out; 140 | 141 | try 142 | { 143 | while (len > 0) 144 | { 145 | ssize_t num_consumed = 0; 146 | bool is_frame_available = false; 147 | 148 | try 149 | { 150 | auto frame = decode_frame_impl(data_in, len, num_consumed, is_frame_available); 151 | if (is_frame_available) 152 | { 153 | out.append(frame); 154 | } 155 | } 156 | catch (const H264DecodeFailure &e) 157 | { 158 | if (num_consumed <= 0) 159 | // This case is fatal because we cannot continue to move ahead in the stream. 160 | throw e; 161 | } 162 | 163 | len -= num_consumed; 164 | data_in += num_consumed; 165 | } 166 | } 167 | catch (const H264DecodeFailure &e) 168 | { 169 | } 170 | 171 | return out; 172 | } 173 | 174 | 175 | BOOST_PYTHON_MODULE(libh264decoder) 176 | { 177 | PyEval_InitThreads(); // need for release of the GIL (http://stackoverflow.com/questions/8009613/boost-python-not-supporting-parallelism) 178 | py::class_("H264Decoder") 179 | .def("decode_frame", &PyH264Decoder::decode_frame) 180 | .def("decode", &PyH264Decoder::decode); 181 | py::def("disable_logging", disable_logging); 182 | } -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/h264decoder/readme.md: -------------------------------------------------------------------------------- 1 | H264 Decoder Python Module 2 | ========================== 3 | 4 | The aim of this project is to provide a simple decoder for video 5 | captured by a Raspberry Pi camera. At the time of this writing I only 6 | need H264 decoding, since a H264 stream is what the RPi software 7 | delivers. Furthermore flexibility to incorporate the decoder in larger 8 | python programs in various ways is desirable. 9 | 10 | The code might also serve as example for libav and boost python usage. 11 | 12 | 13 | Files 14 | ----- 15 | * `h264decoder.hpp`, `h264decoder.cpp` and `h264decoder_python.cpp` contain the module code. 16 | 17 | * Other source files are tests and demos. 18 | 19 | 20 | Requirements 21 | ------------ 22 | * cmake for building 23 | * libav 24 | * boost python 25 | 26 | 27 | Todo 28 | ---- 29 | 30 | * Add a video clip for testing and remove hard coded file names in demos/tests. 31 | 32 | 33 | License 34 | ------- 35 | The code is published under the Mozilla Public License v. 2.0. 36 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/img/readme.md: -------------------------------------------------------------------------------- 1 | This folder mainly stores photos taken by the tello front view camera. -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/install/Linux/linux_install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | echo 'Compiling and Installing the Tello Video Stream module' 4 | echo 'You might need to enter your password' 5 | 6 | cd .. 7 | cd .. 8 | sudo apt-get update -y 9 | 10 | # install python 2.7 11 | sudo apt-get install python2.7 python-pip -y 12 | sudo pip install --upgrade pip 13 | 14 | #switch to python2.7 15 | sudo update-alternatives --install /usr/bin/python python /usr/bin/python2 150 16 | sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 100 17 | 18 | sudo apt-get update -y 19 | 20 | # install cmake 21 | #sudo apt-get install cmake -y 22 | sudo pip install cmake 23 | 24 | # install dependencies 25 | sudo apt-get install libboost-all-dev -y 26 | sudo apt-get install libavcodec-dev -y 27 | sudo apt-get install libswscale-dev -y 28 | sudo apt-get install python-numpy -y 29 | sudo apt-get install python-matplotlib -y 30 | sudo pip install opencv-python 31 | sudo apt-get install python-imaging-tk 32 | 33 | # pull and build h264 decoder library 34 | cd h264decoder 35 | mkdir build 36 | cd build 37 | cmake .. 38 | make 39 | 40 | # copy source .so file to tello.py directory 41 | cp libh264decoder.so ../../ 42 | 43 | echo 'Compilation and Installation Done!' 44 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/install/Linux/linux_uninstall.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | echo 'Uninstalling the Tello Video Stream module' 4 | 5 | cd .. 6 | cd .. 7 | sudo pip uninstall opencv-python -y 8 | sudo pip uninstall cmake -y 9 | sudo pip uninstall pip -y 10 | 11 | sudo apt-get remove libboost-all-dev -y 12 | sudo apt-get remove libavcodec-dev -y 13 | sudo apt-get remove libswscale-dev -y 14 | sudo apt-get remove python-numpy -y 15 | sudo apt-get remove python-matplotlib -y 16 | sudo apt-get remove python-pil.imagetk -y 17 | 18 | sudo apt-get remove python-pip -y 19 | sudo apt-get remove python2.7 -y 20 | 21 | sudo apt-get update -y 22 | 23 | rm ./libh264decoder.so 24 | rm -r ./h264decoder/build 25 | echo 'Uninstallation Done!' 26 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/install/Mac/mac_install.sh: -------------------------------------------------------------------------------- 1 | 2 | #!/bin/sh 3 | 4 | 5 | 6 | echo 'Compiling and Installing the Tello Video Stream module' 7 | 8 | echo 'You might need to enter your password' 9 | 10 | # go to /sample_code folder 11 | cd .. 12 | cd .. 13 | 14 | # install Homebrew 15 | 16 | /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" 17 | 18 | brew update 19 | 20 | 21 | 22 | # install pip 23 | 24 | #sudo easy_install pip 25 | 26 | curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py 27 | sudo python get-pip.py 28 | 29 | # install cmake 30 | 31 | brew install cmake 32 | 33 | 34 | 35 | # install dependencies 36 | 37 | brew install boost 38 | 39 | brew install boost-python 40 | 41 | brew install ffmpeg 42 | 43 | brew install tcl-tk 44 | 45 | sudo pip install numpy --ignore-installed 46 | 47 | sudo pip install matplotlib --ignore-installed 48 | 49 | sudo pip install pillow --ignore-installed 50 | 51 | sudo pip install opencv-python --ignore-installed 52 | 53 | 54 | 55 | # pull and build h264 decoder library 56 | 57 | cd h264decoder 58 | 59 | 60 | mkdir build 61 | 62 | cd build 63 | 64 | cmake .. 65 | 66 | make 67 | 68 | 69 | 70 | # copy source .so file to tello.py directory 71 | 72 | cp libh264decoder.so ../../ 73 | 74 | 75 | 76 | echo 'Compilation and Installation Done!' 77 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/install/Mac/mac_uninstall.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | echo 'Uninstalling the Tello Video Stream module' 3 | 4 | echo 'You might need to enter your password' 5 | 6 | # go to /sample_code folder 7 | cd .. 8 | cd .. 9 | 10 | brew update 11 | 12 | sudo pip uninstall matplotlib -y 13 | sudo pip uninstall numpy -y 14 | sudo pip install pillow -y 15 | sudo pip install opencv-python -y 16 | sudo pip uninstall pip -y 17 | 18 | brew uninstall tcl-tk 19 | brew uninstall ffmpeg 20 | brew uninstall boost-python 21 | brew uninstall boost 22 | brew uninstall cmake 23 | 24 | rm -f ./libh264decoder.so 25 | rm -rf ./h264decoder/build 26 | 27 | echo 'Uninstallation Done!' 28 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/install/Windows/install.bat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/Tello_Video_With_Pose_Recognition/install/Windows/install.bat -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/install/Windows/uninstall.bat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/Tello_Video_With_Pose_Recognition/install/Windows/uninstall.bat -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/main.py: -------------------------------------------------------------------------------- 1 | import tello 2 | from tello_control_ui import TelloUI 3 | 4 | 5 | def main(): 6 | 7 | drone = tello.Tello('', 8889) 8 | vplayer = TelloUI(drone,"./img/") 9 | 10 | # start the Tkinter mainloop 11 | vplayer.root.mainloop() 12 | 13 | if __name__ == "__main__": 14 | main() 15 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/model/getModels.bat: -------------------------------------------------------------------------------- 1 | echo=1/*>nul&@cls 2 | :: ------------------------- BODY, FACE AND HAND MODELS ------------------------- 3 | :: Downloading body pose (COCO and MPI), face and hand models 4 | @echo off 5 | set OPENPOSE_URL=http://posefs1.perception.cs.cmu.edu/OpenPose/models/ 6 | set POSE_FOLDER=pose/ 7 | 8 | set MPI_FOLDER=%POSE_FOLDER%mpi/ 9 | set MPI_MODEL=%MPI_FOLDER%pose_iter_160000.caffemodel 10 | call :down %OPENPOSE_URL%%MPI_MODEL% %MPI_MODEL% 11 | goto :eof 12 | 13 | :down 14 | echo Source: "%~1" 15 | echo Destination: "%~f2" 16 | echo Start downloading "%~2"... 17 | cscript -nologo -e:jscript "%~f0" "download" "%~1" "%~2" 18 | echo ------------------------------------------------------ 19 | goto :eof 20 | 21 | */ 22 | 23 | function download(DownSource, DownDestination) 24 | { 25 | var DownPost 26 | ,DownGet; 27 | 28 | var DownPost=null; 29 | try{ 30 | DownPost=new XMLHttpRequest(); 31 | }catch(e){ 32 | try{ 33 | DownPost=new ActiveXObject("Msxml2.XMLHTTP"); 34 | DownPost.setOption(2, 13056); 35 | }catch(ex){ 36 | try{ 37 | DownPost=new ActiveXObject("Microsoft.XMLHTTP"); 38 | }catch(e3){ 39 | DownPost=null; 40 | } 41 | } 42 | } 43 | DownPost.open("GET",DownSource,0); 44 | DownPost.send(); 45 | DownGet = new ActiveXObject("ADODB"+String.fromCharCode(0x2e)+"Stream"); 46 | DownGet.Mode = 3; 47 | DownGet.Type = 1; 48 | DownGet.Open(); 49 | DownGet.Write(DownPost.responseBody); 50 | DownGet.SaveToFile(DownDestination,2); 51 | } 52 | 53 | function unpack(PackedFileSource, UnpackFileDestination, ParentFolder) 54 | { 55 | var FileSysObject = new Object 56 | ,ShellObject = new ActiveXObject("Shell.Application") 57 | ,intOptions = 4 + 16 58 | ,DestinationObj 59 | ,SourceObj; 60 | 61 | if (!UnpackFileDestination) UnpackFileDestination = '.'; 62 | var FolderTest = ShellObject.NameSpace(ParentFolder + UnpackFileDestination); 63 | FileSysObject = ShellObject.NameSpace(ParentFolder); 64 | while (!FolderTest) 65 | { 66 | WSH.Echo ('Unpack Destination Folder Not Exist, Creating...'); 67 | FileSysObject.NewFolder(UnpackFileDestination); 68 | FolderTest = ShellObject.NameSpace(ParentFolder + UnpackFileDestination); 69 | if (FolderTest) 70 | WSH.Echo('Unpack Destination Folder Created.'); 71 | } 72 | DestinationObj = ShellObject.NameSpace(ParentFolder + UnpackFileDestination); 73 | SourceObj = ShellObject.NameSpace(ParentFolder + PackedFileSource); 74 | for (var i = 0; i < SourceObj.Items().Count; i++) 75 | { 76 | try { 77 | if (SourceObj) { 78 | WSH.Echo('Unpacking ' + SourceObj.Items().Item(i) + '... '); 79 | DestinationObj.CopyHere(SourceObj.Items().Item(i), intOptions); 80 | WSH.Echo('Unpack ' + SourceObj.Items().Item(i) + ' Done.'); 81 | } 82 | } 83 | catch(e) { 84 | WSH.Echo('Failed: ' + e); 85 | } 86 | } 87 | } 88 | 89 | switch (WScript.Arguments(0)){ 90 | case "download": 91 | download(WScript.Arguments(1), WScript.Arguments(2)); 92 | break; 93 | case "unpack": 94 | unpack(WScript.Arguments(1), WScript.Arguments(2), WScript.Arguments(3)); 95 | break; 96 | default: 97 | } -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/model/getModels.sh: -------------------------------------------------------------------------------- 1 | # ------------------------- BODY, FACE AND HAND MODELS ------------------------- 2 | # Downloading body pose (COCO and MPI), face and hand models 3 | OPENPOSE_URL="http://posefs1.perception.cs.cmu.edu/OpenPose/models/" 4 | POSE_FOLDER="pose/" 5 | 6 | # Body (MPI) 7 | MPI_FOLDER=${POSE_FOLDER}"mpi/" 8 | MPI_MODEL=${MPI_FOLDER}"pose_iter_160000.caffemodel" 9 | wget -c ${OPENPOSE_URL}${MPI_MODEL} -P ${MPI_FOLDER} 10 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/model/pose/mpi/pose_deploy_linevec_faster_4_stages.prototxt: -------------------------------------------------------------------------------- 1 | input: "image" 2 | input_dim: 1 3 | input_dim: 3 4 | input_dim: 1 # This value will be defined at runtime 5 | input_dim: 1 # This value will be defined at runtime 6 | layer { 7 | name: "conv1_1" 8 | type: "Convolution" 9 | bottom: "image" 10 | top: "conv1_1" 11 | param { 12 | lr_mult: 1.0 13 | decay_mult: 1 14 | } 15 | param { 16 | lr_mult: 2.0 17 | decay_mult: 0 18 | } 19 | convolution_param { 20 | num_output: 64 21 | pad: 1 22 | kernel_size: 3 23 | weight_filler { 24 | type: "gaussian" 25 | std: 0.01 26 | } 27 | bias_filler { 28 | type: "constant" 29 | } 30 | } 31 | } 32 | layer { 33 | name: "relu1_1" 34 | type: "ReLU" 35 | bottom: "conv1_1" 36 | top: "conv1_1" 37 | } 38 | layer { 39 | name: "conv1_2" 40 | type: "Convolution" 41 | bottom: "conv1_1" 42 | top: "conv1_2" 43 | param { 44 | lr_mult: 1.0 45 | decay_mult: 1 46 | } 47 | param { 48 | lr_mult: 2.0 49 | decay_mult: 0 50 | } 51 | convolution_param { 52 | num_output: 64 53 | pad: 1 54 | kernel_size: 3 55 | weight_filler { 56 | type: "gaussian" 57 | std: 0.01 58 | } 59 | bias_filler { 60 | type: "constant" 61 | } 62 | } 63 | } 64 | layer { 65 | name: "relu1_2" 66 | type: "ReLU" 67 | bottom: "conv1_2" 68 | top: "conv1_2" 69 | } 70 | layer { 71 | name: "pool1_stage1" 72 | type: "Pooling" 73 | bottom: "conv1_2" 74 | top: "pool1_stage1" 75 | pooling_param { 76 | pool: MAX 77 | kernel_size: 2 78 | stride: 2 79 | } 80 | } 81 | layer { 82 | name: "conv2_1" 83 | type: "Convolution" 84 | bottom: "pool1_stage1" 85 | top: "conv2_1" 86 | param { 87 | lr_mult: 1.0 88 | decay_mult: 1 89 | } 90 | param { 91 | lr_mult: 2.0 92 | decay_mult: 0 93 | } 94 | convolution_param { 95 | num_output: 128 96 | pad: 1 97 | kernel_size: 3 98 | weight_filler { 99 | type: "gaussian" 100 | std: 0.01 101 | } 102 | bias_filler { 103 | type: "constant" 104 | } 105 | } 106 | } 107 | layer { 108 | name: "relu2_1" 109 | type: "ReLU" 110 | bottom: "conv2_1" 111 | top: "conv2_1" 112 | } 113 | layer { 114 | name: "conv2_2" 115 | type: "Convolution" 116 | bottom: "conv2_1" 117 | top: "conv2_2" 118 | param { 119 | lr_mult: 1.0 120 | decay_mult: 1 121 | } 122 | param { 123 | lr_mult: 2.0 124 | decay_mult: 0 125 | } 126 | convolution_param { 127 | num_output: 128 128 | pad: 1 129 | kernel_size: 3 130 | weight_filler { 131 | type: "gaussian" 132 | std: 0.01 133 | } 134 | bias_filler { 135 | type: "constant" 136 | } 137 | } 138 | } 139 | layer { 140 | name: "relu2_2" 141 | type: "ReLU" 142 | bottom: "conv2_2" 143 | top: "conv2_2" 144 | } 145 | layer { 146 | name: "pool2_stage1" 147 | type: "Pooling" 148 | bottom: "conv2_2" 149 | top: "pool2_stage1" 150 | pooling_param { 151 | pool: MAX 152 | kernel_size: 2 153 | stride: 2 154 | } 155 | } 156 | layer { 157 | name: "conv3_1" 158 | type: "Convolution" 159 | bottom: "pool2_stage1" 160 | top: "conv3_1" 161 | param { 162 | lr_mult: 1.0 163 | decay_mult: 1 164 | } 165 | param { 166 | lr_mult: 2.0 167 | decay_mult: 0 168 | } 169 | convolution_param { 170 | num_output: 256 171 | pad: 1 172 | kernel_size: 3 173 | weight_filler { 174 | type: "gaussian" 175 | std: 0.01 176 | } 177 | bias_filler { 178 | type: "constant" 179 | } 180 | } 181 | } 182 | layer { 183 | name: "relu3_1" 184 | type: "ReLU" 185 | bottom: "conv3_1" 186 | top: "conv3_1" 187 | } 188 | layer { 189 | name: "conv3_2" 190 | type: "Convolution" 191 | bottom: "conv3_1" 192 | top: "conv3_2" 193 | param { 194 | lr_mult: 1.0 195 | decay_mult: 1 196 | } 197 | param { 198 | lr_mult: 2.0 199 | decay_mult: 0 200 | } 201 | convolution_param { 202 | num_output: 256 203 | pad: 1 204 | kernel_size: 3 205 | weight_filler { 206 | type: "gaussian" 207 | std: 0.01 208 | } 209 | bias_filler { 210 | type: "constant" 211 | } 212 | } 213 | } 214 | layer { 215 | name: "relu3_2" 216 | type: "ReLU" 217 | bottom: "conv3_2" 218 | top: "conv3_2" 219 | } 220 | layer { 221 | name: "conv3_3" 222 | type: "Convolution" 223 | bottom: "conv3_2" 224 | top: "conv3_3" 225 | param { 226 | lr_mult: 1.0 227 | decay_mult: 1 228 | } 229 | param { 230 | lr_mult: 2.0 231 | decay_mult: 0 232 | } 233 | convolution_param { 234 | num_output: 256 235 | pad: 1 236 | kernel_size: 3 237 | weight_filler { 238 | type: "gaussian" 239 | std: 0.01 240 | } 241 | bias_filler { 242 | type: "constant" 243 | } 244 | } 245 | } 246 | layer { 247 | name: "relu3_3" 248 | type: "ReLU" 249 | bottom: "conv3_3" 250 | top: "conv3_3" 251 | } 252 | layer { 253 | name: "conv3_4" 254 | type: "Convolution" 255 | bottom: "conv3_3" 256 | top: "conv3_4" 257 | param { 258 | lr_mult: 1.0 259 | decay_mult: 1 260 | } 261 | param { 262 | lr_mult: 2.0 263 | decay_mult: 0 264 | } 265 | convolution_param { 266 | num_output: 256 267 | pad: 1 268 | kernel_size: 3 269 | weight_filler { 270 | type: "gaussian" 271 | std: 0.01 272 | } 273 | bias_filler { 274 | type: "constant" 275 | } 276 | } 277 | } 278 | layer { 279 | name: "relu3_4" 280 | type: "ReLU" 281 | bottom: "conv3_4" 282 | top: "conv3_4" 283 | } 284 | layer { 285 | name: "pool3_stage1" 286 | type: "Pooling" 287 | bottom: "conv3_4" 288 | top: "pool3_stage1" 289 | pooling_param { 290 | pool: MAX 291 | kernel_size: 2 292 | stride: 2 293 | } 294 | } 295 | layer { 296 | name: "conv4_1" 297 | type: "Convolution" 298 | bottom: "pool3_stage1" 299 | top: "conv4_1" 300 | param { 301 | lr_mult: 1.0 302 | decay_mult: 1 303 | } 304 | param { 305 | lr_mult: 2.0 306 | decay_mult: 0 307 | } 308 | convolution_param { 309 | num_output: 512 310 | pad: 1 311 | kernel_size: 3 312 | weight_filler { 313 | type: "gaussian" 314 | std: 0.01 315 | } 316 | bias_filler { 317 | type: "constant" 318 | } 319 | } 320 | } 321 | layer { 322 | name: "relu4_1" 323 | type: "ReLU" 324 | bottom: "conv4_1" 325 | top: "conv4_1" 326 | } 327 | layer { 328 | name: "conv4_2" 329 | type: "Convolution" 330 | bottom: "conv4_1" 331 | top: "conv4_2" 332 | param { 333 | lr_mult: 1.0 334 | decay_mult: 1 335 | } 336 | param { 337 | lr_mult: 2.0 338 | decay_mult: 0 339 | } 340 | convolution_param { 341 | num_output: 512 342 | pad: 1 343 | kernel_size: 3 344 | weight_filler { 345 | type: "gaussian" 346 | std: 0.01 347 | } 348 | bias_filler { 349 | type: "constant" 350 | } 351 | } 352 | } 353 | layer { 354 | name: "relu4_2" 355 | type: "ReLU" 356 | bottom: "conv4_2" 357 | top: "conv4_2" 358 | } 359 | layer { 360 | name: "conv4_3_CPM" 361 | type: "Convolution" 362 | bottom: "conv4_2" 363 | top: "conv4_3_CPM" 364 | param { 365 | lr_mult: 1.0 366 | decay_mult: 1 367 | } 368 | param { 369 | lr_mult: 2.0 370 | decay_mult: 0 371 | } 372 | convolution_param { 373 | num_output: 256 374 | pad: 1 375 | kernel_size: 3 376 | weight_filler { 377 | type: "gaussian" 378 | std: 0.01 379 | } 380 | bias_filler { 381 | type: "constant" 382 | } 383 | } 384 | } 385 | layer { 386 | name: "relu4_3_CPM" 387 | type: "ReLU" 388 | bottom: "conv4_3_CPM" 389 | top: "conv4_3_CPM" 390 | } 391 | layer { 392 | name: "conv4_4_CPM" 393 | type: "Convolution" 394 | bottom: "conv4_3_CPM" 395 | top: "conv4_4_CPM" 396 | param { 397 | lr_mult: 1.0 398 | decay_mult: 1 399 | } 400 | param { 401 | lr_mult: 2.0 402 | decay_mult: 0 403 | } 404 | convolution_param { 405 | num_output: 128 406 | pad: 1 407 | kernel_size: 3 408 | weight_filler { 409 | type: "gaussian" 410 | std: 0.01 411 | } 412 | bias_filler { 413 | type: "constant" 414 | } 415 | } 416 | } 417 | layer { 418 | name: "relu4_4_CPM" 419 | type: "ReLU" 420 | bottom: "conv4_4_CPM" 421 | top: "conv4_4_CPM" 422 | } 423 | layer { 424 | name: "conv5_1_CPM_L1" 425 | type: "Convolution" 426 | bottom: "conv4_4_CPM" 427 | top: "conv5_1_CPM_L1" 428 | param { 429 | lr_mult: 1.0 430 | decay_mult: 1 431 | } 432 | param { 433 | lr_mult: 2.0 434 | decay_mult: 0 435 | } 436 | convolution_param { 437 | num_output: 128 438 | pad: 1 439 | kernel_size: 3 440 | weight_filler { 441 | type: "gaussian" 442 | std: 0.01 443 | } 444 | bias_filler { 445 | type: "constant" 446 | } 447 | } 448 | } 449 | layer { 450 | name: "relu5_1_CPM_L1" 451 | type: "ReLU" 452 | bottom: "conv5_1_CPM_L1" 453 | top: "conv5_1_CPM_L1" 454 | } 455 | layer { 456 | name: "conv5_1_CPM_L2" 457 | type: "Convolution" 458 | bottom: "conv4_4_CPM" 459 | top: "conv5_1_CPM_L2" 460 | param { 461 | lr_mult: 1.0 462 | decay_mult: 1 463 | } 464 | param { 465 | lr_mult: 2.0 466 | decay_mult: 0 467 | } 468 | convolution_param { 469 | num_output: 128 470 | pad: 1 471 | kernel_size: 3 472 | weight_filler { 473 | type: "gaussian" 474 | std: 0.01 475 | } 476 | bias_filler { 477 | type: "constant" 478 | } 479 | } 480 | } 481 | layer { 482 | name: "relu5_1_CPM_L2" 483 | type: "ReLU" 484 | bottom: "conv5_1_CPM_L2" 485 | top: "conv5_1_CPM_L2" 486 | } 487 | layer { 488 | name: "conv5_2_CPM_L1" 489 | type: "Convolution" 490 | bottom: "conv5_1_CPM_L1" 491 | top: "conv5_2_CPM_L1" 492 | param { 493 | lr_mult: 1.0 494 | decay_mult: 1 495 | } 496 | param { 497 | lr_mult: 2.0 498 | decay_mult: 0 499 | } 500 | convolution_param { 501 | num_output: 128 502 | pad: 1 503 | kernel_size: 3 504 | weight_filler { 505 | type: "gaussian" 506 | std: 0.01 507 | } 508 | bias_filler { 509 | type: "constant" 510 | } 511 | } 512 | } 513 | layer { 514 | name: "relu5_2_CPM_L1" 515 | type: "ReLU" 516 | bottom: "conv5_2_CPM_L1" 517 | top: "conv5_2_CPM_L1" 518 | } 519 | layer { 520 | name: "conv5_2_CPM_L2" 521 | type: "Convolution" 522 | bottom: "conv5_1_CPM_L2" 523 | top: "conv5_2_CPM_L2" 524 | param { 525 | lr_mult: 1.0 526 | decay_mult: 1 527 | } 528 | param { 529 | lr_mult: 2.0 530 | decay_mult: 0 531 | } 532 | convolution_param { 533 | num_output: 128 534 | pad: 1 535 | kernel_size: 3 536 | weight_filler { 537 | type: "gaussian" 538 | std: 0.01 539 | } 540 | bias_filler { 541 | type: "constant" 542 | } 543 | } 544 | } 545 | layer { 546 | name: "relu5_2_CPM_L2" 547 | type: "ReLU" 548 | bottom: "conv5_2_CPM_L2" 549 | top: "conv5_2_CPM_L2" 550 | } 551 | layer { 552 | name: "conv5_3_CPM_L1" 553 | type: "Convolution" 554 | bottom: "conv5_2_CPM_L1" 555 | top: "conv5_3_CPM_L1" 556 | param { 557 | lr_mult: 1.0 558 | decay_mult: 1 559 | } 560 | param { 561 | lr_mult: 2.0 562 | decay_mult: 0 563 | } 564 | convolution_param { 565 | num_output: 128 566 | pad: 1 567 | kernel_size: 3 568 | weight_filler { 569 | type: "gaussian" 570 | std: 0.01 571 | } 572 | bias_filler { 573 | type: "constant" 574 | } 575 | } 576 | } 577 | layer { 578 | name: "relu5_3_CPM_L1" 579 | type: "ReLU" 580 | bottom: "conv5_3_CPM_L1" 581 | top: "conv5_3_CPM_L1" 582 | } 583 | layer { 584 | name: "conv5_3_CPM_L2" 585 | type: "Convolution" 586 | bottom: "conv5_2_CPM_L2" 587 | top: "conv5_3_CPM_L2" 588 | param { 589 | lr_mult: 1.0 590 | decay_mult: 1 591 | } 592 | param { 593 | lr_mult: 2.0 594 | decay_mult: 0 595 | } 596 | convolution_param { 597 | num_output: 128 598 | pad: 1 599 | kernel_size: 3 600 | weight_filler { 601 | type: "gaussian" 602 | std: 0.01 603 | } 604 | bias_filler { 605 | type: "constant" 606 | } 607 | } 608 | } 609 | layer { 610 | name: "relu5_3_CPM_L2" 611 | type: "ReLU" 612 | bottom: "conv5_3_CPM_L2" 613 | top: "conv5_3_CPM_L2" 614 | } 615 | layer { 616 | name: "conv5_4_CPM_L1" 617 | type: "Convolution" 618 | bottom: "conv5_3_CPM_L1" 619 | top: "conv5_4_CPM_L1" 620 | param { 621 | lr_mult: 1.0 622 | decay_mult: 1 623 | } 624 | param { 625 | lr_mult: 2.0 626 | decay_mult: 0 627 | } 628 | convolution_param { 629 | num_output: 512 630 | pad: 0 631 | kernel_size: 1 632 | weight_filler { 633 | type: "gaussian" 634 | std: 0.01 635 | } 636 | bias_filler { 637 | type: "constant" 638 | } 639 | } 640 | } 641 | layer { 642 | name: "relu5_4_CPM_L1" 643 | type: "ReLU" 644 | bottom: "conv5_4_CPM_L1" 645 | top: "conv5_4_CPM_L1" 646 | } 647 | layer { 648 | name: "conv5_4_CPM_L2" 649 | type: "Convolution" 650 | bottom: "conv5_3_CPM_L2" 651 | top: "conv5_4_CPM_L2" 652 | param { 653 | lr_mult: 1.0 654 | decay_mult: 1 655 | } 656 | param { 657 | lr_mult: 2.0 658 | decay_mult: 0 659 | } 660 | convolution_param { 661 | num_output: 512 662 | pad: 0 663 | kernel_size: 1 664 | weight_filler { 665 | type: "gaussian" 666 | std: 0.01 667 | } 668 | bias_filler { 669 | type: "constant" 670 | } 671 | } 672 | } 673 | layer { 674 | name: "relu5_4_CPM_L2" 675 | type: "ReLU" 676 | bottom: "conv5_4_CPM_L2" 677 | top: "conv5_4_CPM_L2" 678 | } 679 | layer { 680 | name: "conv5_5_CPM_L1" 681 | type: "Convolution" 682 | bottom: "conv5_4_CPM_L1" 683 | top: "conv5_5_CPM_L1" 684 | param { 685 | lr_mult: 1.0 686 | decay_mult: 1 687 | } 688 | param { 689 | lr_mult: 2.0 690 | decay_mult: 0 691 | } 692 | convolution_param { 693 | num_output: 28 694 | pad: 0 695 | kernel_size: 1 696 | weight_filler { 697 | type: "gaussian" 698 | std: 0.01 699 | } 700 | bias_filler { 701 | type: "constant" 702 | } 703 | } 704 | } 705 | layer { 706 | name: "conv5_5_CPM_L2" 707 | type: "Convolution" 708 | bottom: "conv5_4_CPM_L2" 709 | top: "conv5_5_CPM_L2" 710 | param { 711 | lr_mult: 1.0 712 | decay_mult: 1 713 | } 714 | param { 715 | lr_mult: 2.0 716 | decay_mult: 0 717 | } 718 | convolution_param { 719 | num_output: 16 720 | pad: 0 721 | kernel_size: 1 722 | weight_filler { 723 | type: "gaussian" 724 | std: 0.01 725 | } 726 | bias_filler { 727 | type: "constant" 728 | } 729 | } 730 | } 731 | layer { 732 | name: "concat_stage2" 733 | type: "Concat" 734 | bottom: "conv5_5_CPM_L1" 735 | bottom: "conv5_5_CPM_L2" 736 | bottom: "conv4_4_CPM" 737 | top: "concat_stage2" 738 | concat_param { 739 | axis: 1 740 | } 741 | } 742 | layer { 743 | name: "Mconv1_stage2_L1" 744 | type: "Convolution" 745 | bottom: "concat_stage2" 746 | top: "Mconv1_stage2_L1" 747 | param { 748 | lr_mult: 4.0 749 | decay_mult: 1 750 | } 751 | param { 752 | lr_mult: 8.0 753 | decay_mult: 0 754 | } 755 | convolution_param { 756 | num_output: 128 757 | pad: 3 758 | kernel_size: 7 759 | weight_filler { 760 | type: "gaussian" 761 | std: 0.01 762 | } 763 | bias_filler { 764 | type: "constant" 765 | } 766 | } 767 | } 768 | layer { 769 | name: "Mrelu1_stage2_L1" 770 | type: "ReLU" 771 | bottom: "Mconv1_stage2_L1" 772 | top: "Mconv1_stage2_L1" 773 | } 774 | layer { 775 | name: "Mconv1_stage2_L2" 776 | type: "Convolution" 777 | bottom: "concat_stage2" 778 | top: "Mconv1_stage2_L2" 779 | param { 780 | lr_mult: 4.0 781 | decay_mult: 1 782 | } 783 | param { 784 | lr_mult: 8.0 785 | decay_mult: 0 786 | } 787 | convolution_param { 788 | num_output: 128 789 | pad: 3 790 | kernel_size: 7 791 | weight_filler { 792 | type: "gaussian" 793 | std: 0.01 794 | } 795 | bias_filler { 796 | type: "constant" 797 | } 798 | } 799 | } 800 | layer { 801 | name: "Mrelu1_stage2_L2" 802 | type: "ReLU" 803 | bottom: "Mconv1_stage2_L2" 804 | top: "Mconv1_stage2_L2" 805 | } 806 | layer { 807 | name: "Mconv2_stage2_L1" 808 | type: "Convolution" 809 | bottom: "Mconv1_stage2_L1" 810 | top: "Mconv2_stage2_L1" 811 | param { 812 | lr_mult: 4.0 813 | decay_mult: 1 814 | } 815 | param { 816 | lr_mult: 8.0 817 | decay_mult: 0 818 | } 819 | convolution_param { 820 | num_output: 128 821 | pad: 3 822 | kernel_size: 7 823 | weight_filler { 824 | type: "gaussian" 825 | std: 0.01 826 | } 827 | bias_filler { 828 | type: "constant" 829 | } 830 | } 831 | } 832 | layer { 833 | name: "Mrelu2_stage2_L1" 834 | type: "ReLU" 835 | bottom: "Mconv2_stage2_L1" 836 | top: "Mconv2_stage2_L1" 837 | } 838 | layer { 839 | name: "Mconv2_stage2_L2" 840 | type: "Convolution" 841 | bottom: "Mconv1_stage2_L2" 842 | top: "Mconv2_stage2_L2" 843 | param { 844 | lr_mult: 4.0 845 | decay_mult: 1 846 | } 847 | param { 848 | lr_mult: 8.0 849 | decay_mult: 0 850 | } 851 | convolution_param { 852 | num_output: 128 853 | pad: 3 854 | kernel_size: 7 855 | weight_filler { 856 | type: "gaussian" 857 | std: 0.01 858 | } 859 | bias_filler { 860 | type: "constant" 861 | } 862 | } 863 | } 864 | layer { 865 | name: "Mrelu2_stage2_L2" 866 | type: "ReLU" 867 | bottom: "Mconv2_stage2_L2" 868 | top: "Mconv2_stage2_L2" 869 | } 870 | layer { 871 | name: "Mconv3_stage2_L1" 872 | type: "Convolution" 873 | bottom: "Mconv2_stage2_L1" 874 | top: "Mconv3_stage2_L1" 875 | param { 876 | lr_mult: 4.0 877 | decay_mult: 1 878 | } 879 | param { 880 | lr_mult: 8.0 881 | decay_mult: 0 882 | } 883 | convolution_param { 884 | num_output: 128 885 | pad: 3 886 | kernel_size: 7 887 | weight_filler { 888 | type: "gaussian" 889 | std: 0.01 890 | } 891 | bias_filler { 892 | type: "constant" 893 | } 894 | } 895 | } 896 | layer { 897 | name: "Mrelu3_stage2_L1" 898 | type: "ReLU" 899 | bottom: "Mconv3_stage2_L1" 900 | top: "Mconv3_stage2_L1" 901 | } 902 | layer { 903 | name: "Mconv3_stage2_L2" 904 | type: "Convolution" 905 | bottom: "Mconv2_stage2_L2" 906 | top: "Mconv3_stage2_L2" 907 | param { 908 | lr_mult: 4.0 909 | decay_mult: 1 910 | } 911 | param { 912 | lr_mult: 8.0 913 | decay_mult: 0 914 | } 915 | convolution_param { 916 | num_output: 128 917 | pad: 3 918 | kernel_size: 7 919 | weight_filler { 920 | type: "gaussian" 921 | std: 0.01 922 | } 923 | bias_filler { 924 | type: "constant" 925 | } 926 | } 927 | } 928 | layer { 929 | name: "Mrelu3_stage2_L2" 930 | type: "ReLU" 931 | bottom: "Mconv3_stage2_L2" 932 | top: "Mconv3_stage2_L2" 933 | } 934 | layer { 935 | name: "Mconv4_stage2_L1" 936 | type: "Convolution" 937 | bottom: "Mconv3_stage2_L1" 938 | top: "Mconv4_stage2_L1" 939 | param { 940 | lr_mult: 4.0 941 | decay_mult: 1 942 | } 943 | param { 944 | lr_mult: 8.0 945 | decay_mult: 0 946 | } 947 | convolution_param { 948 | num_output: 128 949 | pad: 3 950 | kernel_size: 7 951 | weight_filler { 952 | type: "gaussian" 953 | std: 0.01 954 | } 955 | bias_filler { 956 | type: "constant" 957 | } 958 | } 959 | } 960 | layer { 961 | name: "Mrelu4_stage2_L1" 962 | type: "ReLU" 963 | bottom: "Mconv4_stage2_L1" 964 | top: "Mconv4_stage2_L1" 965 | } 966 | layer { 967 | name: "Mconv4_stage2_L2" 968 | type: "Convolution" 969 | bottom: "Mconv3_stage2_L2" 970 | top: "Mconv4_stage2_L2" 971 | param { 972 | lr_mult: 4.0 973 | decay_mult: 1 974 | } 975 | param { 976 | lr_mult: 8.0 977 | decay_mult: 0 978 | } 979 | convolution_param { 980 | num_output: 128 981 | pad: 3 982 | kernel_size: 7 983 | weight_filler { 984 | type: "gaussian" 985 | std: 0.01 986 | } 987 | bias_filler { 988 | type: "constant" 989 | } 990 | } 991 | } 992 | layer { 993 | name: "Mrelu4_stage2_L2" 994 | type: "ReLU" 995 | bottom: "Mconv4_stage2_L2" 996 | top: "Mconv4_stage2_L2" 997 | } 998 | layer { 999 | name: "Mconv5_stage2_L1" 1000 | type: "Convolution" 1001 | bottom: "Mconv4_stage2_L1" 1002 | top: "Mconv5_stage2_L1" 1003 | param { 1004 | lr_mult: 4.0 1005 | decay_mult: 1 1006 | } 1007 | param { 1008 | lr_mult: 8.0 1009 | decay_mult: 0 1010 | } 1011 | convolution_param { 1012 | num_output: 128 1013 | pad: 3 1014 | kernel_size: 7 1015 | weight_filler { 1016 | type: "gaussian" 1017 | std: 0.01 1018 | } 1019 | bias_filler { 1020 | type: "constant" 1021 | } 1022 | } 1023 | } 1024 | layer { 1025 | name: "Mrelu5_stage2_L1" 1026 | type: "ReLU" 1027 | bottom: "Mconv5_stage2_L1" 1028 | top: "Mconv5_stage2_L1" 1029 | } 1030 | layer { 1031 | name: "Mconv5_stage2_L2" 1032 | type: "Convolution" 1033 | bottom: "Mconv4_stage2_L2" 1034 | top: "Mconv5_stage2_L2" 1035 | param { 1036 | lr_mult: 4.0 1037 | decay_mult: 1 1038 | } 1039 | param { 1040 | lr_mult: 8.0 1041 | decay_mult: 0 1042 | } 1043 | convolution_param { 1044 | num_output: 128 1045 | pad: 3 1046 | kernel_size: 7 1047 | weight_filler { 1048 | type: "gaussian" 1049 | std: 0.01 1050 | } 1051 | bias_filler { 1052 | type: "constant" 1053 | } 1054 | } 1055 | } 1056 | layer { 1057 | name: "Mrelu5_stage2_L2" 1058 | type: "ReLU" 1059 | bottom: "Mconv5_stage2_L2" 1060 | top: "Mconv5_stage2_L2" 1061 | } 1062 | layer { 1063 | name: "Mconv6_stage2_L1" 1064 | type: "Convolution" 1065 | bottom: "Mconv5_stage2_L1" 1066 | top: "Mconv6_stage2_L1" 1067 | param { 1068 | lr_mult: 4.0 1069 | decay_mult: 1 1070 | } 1071 | param { 1072 | lr_mult: 8.0 1073 | decay_mult: 0 1074 | } 1075 | convolution_param { 1076 | num_output: 128 1077 | pad: 0 1078 | kernel_size: 1 1079 | weight_filler { 1080 | type: "gaussian" 1081 | std: 0.01 1082 | } 1083 | bias_filler { 1084 | type: "constant" 1085 | } 1086 | } 1087 | } 1088 | layer { 1089 | name: "Mrelu6_stage2_L1" 1090 | type: "ReLU" 1091 | bottom: "Mconv6_stage2_L1" 1092 | top: "Mconv6_stage2_L1" 1093 | } 1094 | layer { 1095 | name: "Mconv6_stage2_L2" 1096 | type: "Convolution" 1097 | bottom: "Mconv5_stage2_L2" 1098 | top: "Mconv6_stage2_L2" 1099 | param { 1100 | lr_mult: 4.0 1101 | decay_mult: 1 1102 | } 1103 | param { 1104 | lr_mult: 8.0 1105 | decay_mult: 0 1106 | } 1107 | convolution_param { 1108 | num_output: 128 1109 | pad: 0 1110 | kernel_size: 1 1111 | weight_filler { 1112 | type: "gaussian" 1113 | std: 0.01 1114 | } 1115 | bias_filler { 1116 | type: "constant" 1117 | } 1118 | } 1119 | } 1120 | layer { 1121 | name: "Mrelu6_stage2_L2" 1122 | type: "ReLU" 1123 | bottom: "Mconv6_stage2_L2" 1124 | top: "Mconv6_stage2_L2" 1125 | } 1126 | layer { 1127 | name: "Mconv7_stage2_L1" 1128 | type: "Convolution" 1129 | bottom: "Mconv6_stage2_L1" 1130 | top: "Mconv7_stage2_L1" 1131 | param { 1132 | lr_mult: 4.0 1133 | decay_mult: 1 1134 | } 1135 | param { 1136 | lr_mult: 8.0 1137 | decay_mult: 0 1138 | } 1139 | convolution_param { 1140 | num_output: 28 1141 | pad: 0 1142 | kernel_size: 1 1143 | weight_filler { 1144 | type: "gaussian" 1145 | std: 0.01 1146 | } 1147 | bias_filler { 1148 | type: "constant" 1149 | } 1150 | } 1151 | } 1152 | layer { 1153 | name: "Mconv7_stage2_L2" 1154 | type: "Convolution" 1155 | bottom: "Mconv6_stage2_L2" 1156 | top: "Mconv7_stage2_L2" 1157 | param { 1158 | lr_mult: 4.0 1159 | decay_mult: 1 1160 | } 1161 | param { 1162 | lr_mult: 8.0 1163 | decay_mult: 0 1164 | } 1165 | convolution_param { 1166 | num_output: 16 1167 | pad: 0 1168 | kernel_size: 1 1169 | weight_filler { 1170 | type: "gaussian" 1171 | std: 0.01 1172 | } 1173 | bias_filler { 1174 | type: "constant" 1175 | } 1176 | } 1177 | } 1178 | layer { 1179 | name: "concat_stage3" 1180 | type: "Concat" 1181 | bottom: "Mconv7_stage2_L1" 1182 | bottom: "Mconv7_stage2_L2" 1183 | bottom: "conv4_4_CPM" 1184 | top: "concat_stage3" 1185 | concat_param { 1186 | axis: 1 1187 | } 1188 | } 1189 | layer { 1190 | name: "Mconv1_stage3_L1" 1191 | type: "Convolution" 1192 | bottom: "concat_stage3" 1193 | top: "Mconv1_stage3_L1" 1194 | param { 1195 | lr_mult: 4.0 1196 | decay_mult: 1 1197 | } 1198 | param { 1199 | lr_mult: 8.0 1200 | decay_mult: 0 1201 | } 1202 | convolution_param { 1203 | num_output: 128 1204 | pad: 3 1205 | kernel_size: 7 1206 | weight_filler { 1207 | type: "gaussian" 1208 | std: 0.01 1209 | } 1210 | bias_filler { 1211 | type: "constant" 1212 | } 1213 | } 1214 | } 1215 | layer { 1216 | name: "Mrelu1_stage3_L1" 1217 | type: "ReLU" 1218 | bottom: "Mconv1_stage3_L1" 1219 | top: "Mconv1_stage3_L1" 1220 | } 1221 | layer { 1222 | name: "Mconv1_stage3_L2" 1223 | type: "Convolution" 1224 | bottom: "concat_stage3" 1225 | top: "Mconv1_stage3_L2" 1226 | param { 1227 | lr_mult: 4.0 1228 | decay_mult: 1 1229 | } 1230 | param { 1231 | lr_mult: 8.0 1232 | decay_mult: 0 1233 | } 1234 | convolution_param { 1235 | num_output: 128 1236 | pad: 3 1237 | kernel_size: 7 1238 | weight_filler { 1239 | type: "gaussian" 1240 | std: 0.01 1241 | } 1242 | bias_filler { 1243 | type: "constant" 1244 | } 1245 | } 1246 | } 1247 | layer { 1248 | name: "Mrelu1_stage3_L2" 1249 | type: "ReLU" 1250 | bottom: "Mconv1_stage3_L2" 1251 | top: "Mconv1_stage3_L2" 1252 | } 1253 | layer { 1254 | name: "Mconv2_stage3_L1" 1255 | type: "Convolution" 1256 | bottom: "Mconv1_stage3_L1" 1257 | top: "Mconv2_stage3_L1" 1258 | param { 1259 | lr_mult: 4.0 1260 | decay_mult: 1 1261 | } 1262 | param { 1263 | lr_mult: 8.0 1264 | decay_mult: 0 1265 | } 1266 | convolution_param { 1267 | num_output: 128 1268 | pad: 3 1269 | kernel_size: 7 1270 | weight_filler { 1271 | type: "gaussian" 1272 | std: 0.01 1273 | } 1274 | bias_filler { 1275 | type: "constant" 1276 | } 1277 | } 1278 | } 1279 | layer { 1280 | name: "Mrelu2_stage3_L1" 1281 | type: "ReLU" 1282 | bottom: "Mconv2_stage3_L1" 1283 | top: "Mconv2_stage3_L1" 1284 | } 1285 | layer { 1286 | name: "Mconv2_stage3_L2" 1287 | type: "Convolution" 1288 | bottom: "Mconv1_stage3_L2" 1289 | top: "Mconv2_stage3_L2" 1290 | param { 1291 | lr_mult: 4.0 1292 | decay_mult: 1 1293 | } 1294 | param { 1295 | lr_mult: 8.0 1296 | decay_mult: 0 1297 | } 1298 | convolution_param { 1299 | num_output: 128 1300 | pad: 3 1301 | kernel_size: 7 1302 | weight_filler { 1303 | type: "gaussian" 1304 | std: 0.01 1305 | } 1306 | bias_filler { 1307 | type: "constant" 1308 | } 1309 | } 1310 | } 1311 | layer { 1312 | name: "Mrelu2_stage3_L2" 1313 | type: "ReLU" 1314 | bottom: "Mconv2_stage3_L2" 1315 | top: "Mconv2_stage3_L2" 1316 | } 1317 | layer { 1318 | name: "Mconv3_stage3_L1" 1319 | type: "Convolution" 1320 | bottom: "Mconv2_stage3_L1" 1321 | top: "Mconv3_stage3_L1" 1322 | param { 1323 | lr_mult: 4.0 1324 | decay_mult: 1 1325 | } 1326 | param { 1327 | lr_mult: 8.0 1328 | decay_mult: 0 1329 | } 1330 | convolution_param { 1331 | num_output: 128 1332 | pad: 3 1333 | kernel_size: 7 1334 | weight_filler { 1335 | type: "gaussian" 1336 | std: 0.01 1337 | } 1338 | bias_filler { 1339 | type: "constant" 1340 | } 1341 | } 1342 | } 1343 | layer { 1344 | name: "Mrelu3_stage3_L1" 1345 | type: "ReLU" 1346 | bottom: "Mconv3_stage3_L1" 1347 | top: "Mconv3_stage3_L1" 1348 | } 1349 | layer { 1350 | name: "Mconv3_stage3_L2" 1351 | type: "Convolution" 1352 | bottom: "Mconv2_stage3_L2" 1353 | top: "Mconv3_stage3_L2" 1354 | param { 1355 | lr_mult: 4.0 1356 | decay_mult: 1 1357 | } 1358 | param { 1359 | lr_mult: 8.0 1360 | decay_mult: 0 1361 | } 1362 | convolution_param { 1363 | num_output: 128 1364 | pad: 3 1365 | kernel_size: 7 1366 | weight_filler { 1367 | type: "gaussian" 1368 | std: 0.01 1369 | } 1370 | bias_filler { 1371 | type: "constant" 1372 | } 1373 | } 1374 | } 1375 | layer { 1376 | name: "Mrelu3_stage3_L2" 1377 | type: "ReLU" 1378 | bottom: "Mconv3_stage3_L2" 1379 | top: "Mconv3_stage3_L2" 1380 | } 1381 | layer { 1382 | name: "Mconv4_stage3_L1" 1383 | type: "Convolution" 1384 | bottom: "Mconv3_stage3_L1" 1385 | top: "Mconv4_stage3_L1" 1386 | param { 1387 | lr_mult: 4.0 1388 | decay_mult: 1 1389 | } 1390 | param { 1391 | lr_mult: 8.0 1392 | decay_mult: 0 1393 | } 1394 | convolution_param { 1395 | num_output: 128 1396 | pad: 3 1397 | kernel_size: 7 1398 | weight_filler { 1399 | type: "gaussian" 1400 | std: 0.01 1401 | } 1402 | bias_filler { 1403 | type: "constant" 1404 | } 1405 | } 1406 | } 1407 | layer { 1408 | name: "Mrelu4_stage3_L1" 1409 | type: "ReLU" 1410 | bottom: "Mconv4_stage3_L1" 1411 | top: "Mconv4_stage3_L1" 1412 | } 1413 | layer { 1414 | name: "Mconv4_stage3_L2" 1415 | type: "Convolution" 1416 | bottom: "Mconv3_stage3_L2" 1417 | top: "Mconv4_stage3_L2" 1418 | param { 1419 | lr_mult: 4.0 1420 | decay_mult: 1 1421 | } 1422 | param { 1423 | lr_mult: 8.0 1424 | decay_mult: 0 1425 | } 1426 | convolution_param { 1427 | num_output: 128 1428 | pad: 3 1429 | kernel_size: 7 1430 | weight_filler { 1431 | type: "gaussian" 1432 | std: 0.01 1433 | } 1434 | bias_filler { 1435 | type: "constant" 1436 | } 1437 | } 1438 | } 1439 | layer { 1440 | name: "Mrelu4_stage3_L2" 1441 | type: "ReLU" 1442 | bottom: "Mconv4_stage3_L2" 1443 | top: "Mconv4_stage3_L2" 1444 | } 1445 | layer { 1446 | name: "Mconv5_stage3_L1" 1447 | type: "Convolution" 1448 | bottom: "Mconv4_stage3_L1" 1449 | top: "Mconv5_stage3_L1" 1450 | param { 1451 | lr_mult: 4.0 1452 | decay_mult: 1 1453 | } 1454 | param { 1455 | lr_mult: 8.0 1456 | decay_mult: 0 1457 | } 1458 | convolution_param { 1459 | num_output: 128 1460 | pad: 3 1461 | kernel_size: 7 1462 | weight_filler { 1463 | type: "gaussian" 1464 | std: 0.01 1465 | } 1466 | bias_filler { 1467 | type: "constant" 1468 | } 1469 | } 1470 | } 1471 | layer { 1472 | name: "Mrelu5_stage3_L1" 1473 | type: "ReLU" 1474 | bottom: "Mconv5_stage3_L1" 1475 | top: "Mconv5_stage3_L1" 1476 | } 1477 | layer { 1478 | name: "Mconv5_stage3_L2" 1479 | type: "Convolution" 1480 | bottom: "Mconv4_stage3_L2" 1481 | top: "Mconv5_stage3_L2" 1482 | param { 1483 | lr_mult: 4.0 1484 | decay_mult: 1 1485 | } 1486 | param { 1487 | lr_mult: 8.0 1488 | decay_mult: 0 1489 | } 1490 | convolution_param { 1491 | num_output: 128 1492 | pad: 3 1493 | kernel_size: 7 1494 | weight_filler { 1495 | type: "gaussian" 1496 | std: 0.01 1497 | } 1498 | bias_filler { 1499 | type: "constant" 1500 | } 1501 | } 1502 | } 1503 | layer { 1504 | name: "Mrelu5_stage3_L2" 1505 | type: "ReLU" 1506 | bottom: "Mconv5_stage3_L2" 1507 | top: "Mconv5_stage3_L2" 1508 | } 1509 | layer { 1510 | name: "Mconv6_stage3_L1" 1511 | type: "Convolution" 1512 | bottom: "Mconv5_stage3_L1" 1513 | top: "Mconv6_stage3_L1" 1514 | param { 1515 | lr_mult: 4.0 1516 | decay_mult: 1 1517 | } 1518 | param { 1519 | lr_mult: 8.0 1520 | decay_mult: 0 1521 | } 1522 | convolution_param { 1523 | num_output: 128 1524 | pad: 0 1525 | kernel_size: 1 1526 | weight_filler { 1527 | type: "gaussian" 1528 | std: 0.01 1529 | } 1530 | bias_filler { 1531 | type: "constant" 1532 | } 1533 | } 1534 | } 1535 | layer { 1536 | name: "Mrelu6_stage3_L1" 1537 | type: "ReLU" 1538 | bottom: "Mconv6_stage3_L1" 1539 | top: "Mconv6_stage3_L1" 1540 | } 1541 | layer { 1542 | name: "Mconv6_stage3_L2" 1543 | type: "Convolution" 1544 | bottom: "Mconv5_stage3_L2" 1545 | top: "Mconv6_stage3_L2" 1546 | param { 1547 | lr_mult: 4.0 1548 | decay_mult: 1 1549 | } 1550 | param { 1551 | lr_mult: 8.0 1552 | decay_mult: 0 1553 | } 1554 | convolution_param { 1555 | num_output: 128 1556 | pad: 0 1557 | kernel_size: 1 1558 | weight_filler { 1559 | type: "gaussian" 1560 | std: 0.01 1561 | } 1562 | bias_filler { 1563 | type: "constant" 1564 | } 1565 | } 1566 | } 1567 | layer { 1568 | name: "Mrelu6_stage3_L2" 1569 | type: "ReLU" 1570 | bottom: "Mconv6_stage3_L2" 1571 | top: "Mconv6_stage3_L2" 1572 | } 1573 | layer { 1574 | name: "Mconv7_stage3_L1" 1575 | type: "Convolution" 1576 | bottom: "Mconv6_stage3_L1" 1577 | top: "Mconv7_stage3_L1" 1578 | param { 1579 | lr_mult: 4.0 1580 | decay_mult: 1 1581 | } 1582 | param { 1583 | lr_mult: 8.0 1584 | decay_mult: 0 1585 | } 1586 | convolution_param { 1587 | num_output: 28 1588 | pad: 0 1589 | kernel_size: 1 1590 | weight_filler { 1591 | type: "gaussian" 1592 | std: 0.01 1593 | } 1594 | bias_filler { 1595 | type: "constant" 1596 | } 1597 | } 1598 | } 1599 | layer { 1600 | name: "Mconv7_stage3_L2" 1601 | type: "Convolution" 1602 | bottom: "Mconv6_stage3_L2" 1603 | top: "Mconv7_stage3_L2" 1604 | param { 1605 | lr_mult: 4.0 1606 | decay_mult: 1 1607 | } 1608 | param { 1609 | lr_mult: 8.0 1610 | decay_mult: 0 1611 | } 1612 | convolution_param { 1613 | num_output: 16 1614 | pad: 0 1615 | kernel_size: 1 1616 | weight_filler { 1617 | type: "gaussian" 1618 | std: 0.01 1619 | } 1620 | bias_filler { 1621 | type: "constant" 1622 | } 1623 | } 1624 | } 1625 | layer { 1626 | name: "concat_stage4" 1627 | type: "Concat" 1628 | bottom: "Mconv7_stage3_L1" 1629 | bottom: "Mconv7_stage3_L2" 1630 | bottom: "conv4_4_CPM" 1631 | top: "concat_stage4" 1632 | concat_param { 1633 | axis: 1 1634 | } 1635 | } 1636 | layer { 1637 | name: "Mconv1_stage4_L1" 1638 | type: "Convolution" 1639 | bottom: "concat_stage4" 1640 | top: "Mconv1_stage4_L1" 1641 | param { 1642 | lr_mult: 4.0 1643 | decay_mult: 1 1644 | } 1645 | param { 1646 | lr_mult: 8.0 1647 | decay_mult: 0 1648 | } 1649 | convolution_param { 1650 | num_output: 128 1651 | pad: 3 1652 | kernel_size: 7 1653 | weight_filler { 1654 | type: "gaussian" 1655 | std: 0.01 1656 | } 1657 | bias_filler { 1658 | type: "constant" 1659 | } 1660 | } 1661 | } 1662 | layer { 1663 | name: "Mrelu1_stage4_L1" 1664 | type: "ReLU" 1665 | bottom: "Mconv1_stage4_L1" 1666 | top: "Mconv1_stage4_L1" 1667 | } 1668 | layer { 1669 | name: "Mconv1_stage4_L2" 1670 | type: "Convolution" 1671 | bottom: "concat_stage4" 1672 | top: "Mconv1_stage4_L2" 1673 | param { 1674 | lr_mult: 4.0 1675 | decay_mult: 1 1676 | } 1677 | param { 1678 | lr_mult: 8.0 1679 | decay_mult: 0 1680 | } 1681 | convolution_param { 1682 | num_output: 128 1683 | pad: 3 1684 | kernel_size: 7 1685 | weight_filler { 1686 | type: "gaussian" 1687 | std: 0.01 1688 | } 1689 | bias_filler { 1690 | type: "constant" 1691 | } 1692 | } 1693 | } 1694 | layer { 1695 | name: "Mrelu1_stage4_L2" 1696 | type: "ReLU" 1697 | bottom: "Mconv1_stage4_L2" 1698 | top: "Mconv1_stage4_L2" 1699 | } 1700 | layer { 1701 | name: "Mconv2_stage4_L1" 1702 | type: "Convolution" 1703 | bottom: "Mconv1_stage4_L1" 1704 | top: "Mconv2_stage4_L1" 1705 | param { 1706 | lr_mult: 4.0 1707 | decay_mult: 1 1708 | } 1709 | param { 1710 | lr_mult: 8.0 1711 | decay_mult: 0 1712 | } 1713 | convolution_param { 1714 | num_output: 128 1715 | pad: 3 1716 | kernel_size: 7 1717 | weight_filler { 1718 | type: "gaussian" 1719 | std: 0.01 1720 | } 1721 | bias_filler { 1722 | type: "constant" 1723 | } 1724 | } 1725 | } 1726 | layer { 1727 | name: "Mrelu2_stage4_L1" 1728 | type: "ReLU" 1729 | bottom: "Mconv2_stage4_L1" 1730 | top: "Mconv2_stage4_L1" 1731 | } 1732 | layer { 1733 | name: "Mconv2_stage4_L2" 1734 | type: "Convolution" 1735 | bottom: "Mconv1_stage4_L2" 1736 | top: "Mconv2_stage4_L2" 1737 | param { 1738 | lr_mult: 4.0 1739 | decay_mult: 1 1740 | } 1741 | param { 1742 | lr_mult: 8.0 1743 | decay_mult: 0 1744 | } 1745 | convolution_param { 1746 | num_output: 128 1747 | pad: 3 1748 | kernel_size: 7 1749 | weight_filler { 1750 | type: "gaussian" 1751 | std: 0.01 1752 | } 1753 | bias_filler { 1754 | type: "constant" 1755 | } 1756 | } 1757 | } 1758 | layer { 1759 | name: "Mrelu2_stage4_L2" 1760 | type: "ReLU" 1761 | bottom: "Mconv2_stage4_L2" 1762 | top: "Mconv2_stage4_L2" 1763 | } 1764 | layer { 1765 | name: "Mconv3_stage4_L1" 1766 | type: "Convolution" 1767 | bottom: "Mconv2_stage4_L1" 1768 | top: "Mconv3_stage4_L1" 1769 | param { 1770 | lr_mult: 4.0 1771 | decay_mult: 1 1772 | } 1773 | param { 1774 | lr_mult: 8.0 1775 | decay_mult: 0 1776 | } 1777 | convolution_param { 1778 | num_output: 128 1779 | pad: 3 1780 | kernel_size: 7 1781 | weight_filler { 1782 | type: "gaussian" 1783 | std: 0.01 1784 | } 1785 | bias_filler { 1786 | type: "constant" 1787 | } 1788 | } 1789 | } 1790 | layer { 1791 | name: "Mrelu3_stage4_L1" 1792 | type: "ReLU" 1793 | bottom: "Mconv3_stage4_L1" 1794 | top: "Mconv3_stage4_L1" 1795 | } 1796 | layer { 1797 | name: "Mconv3_stage4_L2" 1798 | type: "Convolution" 1799 | bottom: "Mconv2_stage4_L2" 1800 | top: "Mconv3_stage4_L2" 1801 | param { 1802 | lr_mult: 4.0 1803 | decay_mult: 1 1804 | } 1805 | param { 1806 | lr_mult: 8.0 1807 | decay_mult: 0 1808 | } 1809 | convolution_param { 1810 | num_output: 128 1811 | pad: 3 1812 | kernel_size: 7 1813 | weight_filler { 1814 | type: "gaussian" 1815 | std: 0.01 1816 | } 1817 | bias_filler { 1818 | type: "constant" 1819 | } 1820 | } 1821 | } 1822 | layer { 1823 | name: "Mrelu3_stage4_L2" 1824 | type: "ReLU" 1825 | bottom: "Mconv3_stage4_L2" 1826 | top: "Mconv3_stage4_L2" 1827 | } 1828 | layer { 1829 | name: "Mconv4_stage4_L1" 1830 | type: "Convolution" 1831 | bottom: "Mconv3_stage4_L1" 1832 | top: "Mconv4_stage4_L1" 1833 | param { 1834 | lr_mult: 4.0 1835 | decay_mult: 1 1836 | } 1837 | param { 1838 | lr_mult: 8.0 1839 | decay_mult: 0 1840 | } 1841 | convolution_param { 1842 | num_output: 128 1843 | pad: 3 1844 | kernel_size: 7 1845 | weight_filler { 1846 | type: "gaussian" 1847 | std: 0.01 1848 | } 1849 | bias_filler { 1850 | type: "constant" 1851 | } 1852 | } 1853 | } 1854 | layer { 1855 | name: "Mrelu4_stage4_L1" 1856 | type: "ReLU" 1857 | bottom: "Mconv4_stage4_L1" 1858 | top: "Mconv4_stage4_L1" 1859 | } 1860 | layer { 1861 | name: "Mconv4_stage4_L2" 1862 | type: "Convolution" 1863 | bottom: "Mconv3_stage4_L2" 1864 | top: "Mconv4_stage4_L2" 1865 | param { 1866 | lr_mult: 4.0 1867 | decay_mult: 1 1868 | } 1869 | param { 1870 | lr_mult: 8.0 1871 | decay_mult: 0 1872 | } 1873 | convolution_param { 1874 | num_output: 128 1875 | pad: 3 1876 | kernel_size: 7 1877 | weight_filler { 1878 | type: "gaussian" 1879 | std: 0.01 1880 | } 1881 | bias_filler { 1882 | type: "constant" 1883 | } 1884 | } 1885 | } 1886 | layer { 1887 | name: "Mrelu4_stage4_L2" 1888 | type: "ReLU" 1889 | bottom: "Mconv4_stage4_L2" 1890 | top: "Mconv4_stage4_L2" 1891 | } 1892 | layer { 1893 | name: "Mconv5_stage4_L1" 1894 | type: "Convolution" 1895 | bottom: "Mconv4_stage4_L1" 1896 | top: "Mconv5_stage4_L1" 1897 | param { 1898 | lr_mult: 4.0 1899 | decay_mult: 1 1900 | } 1901 | param { 1902 | lr_mult: 8.0 1903 | decay_mult: 0 1904 | } 1905 | convolution_param { 1906 | num_output: 128 1907 | pad: 3 1908 | kernel_size: 7 1909 | weight_filler { 1910 | type: "gaussian" 1911 | std: 0.01 1912 | } 1913 | bias_filler { 1914 | type: "constant" 1915 | } 1916 | } 1917 | } 1918 | layer { 1919 | name: "Mrelu5_stage4_L1" 1920 | type: "ReLU" 1921 | bottom: "Mconv5_stage4_L1" 1922 | top: "Mconv5_stage4_L1" 1923 | } 1924 | layer { 1925 | name: "Mconv5_stage4_L2" 1926 | type: "Convolution" 1927 | bottom: "Mconv4_stage4_L2" 1928 | top: "Mconv5_stage4_L2" 1929 | param { 1930 | lr_mult: 4.0 1931 | decay_mult: 1 1932 | } 1933 | param { 1934 | lr_mult: 8.0 1935 | decay_mult: 0 1936 | } 1937 | convolution_param { 1938 | num_output: 128 1939 | pad: 3 1940 | kernel_size: 7 1941 | weight_filler { 1942 | type: "gaussian" 1943 | std: 0.01 1944 | } 1945 | bias_filler { 1946 | type: "constant" 1947 | } 1948 | } 1949 | } 1950 | layer { 1951 | name: "Mrelu5_stage4_L2" 1952 | type: "ReLU" 1953 | bottom: "Mconv5_stage4_L2" 1954 | top: "Mconv5_stage4_L2" 1955 | } 1956 | layer { 1957 | name: "Mconv6_stage4_L1" 1958 | type: "Convolution" 1959 | bottom: "Mconv5_stage4_L1" 1960 | top: "Mconv6_stage4_L1" 1961 | param { 1962 | lr_mult: 4.0 1963 | decay_mult: 1 1964 | } 1965 | param { 1966 | lr_mult: 8.0 1967 | decay_mult: 0 1968 | } 1969 | convolution_param { 1970 | num_output: 128 1971 | pad: 0 1972 | kernel_size: 1 1973 | weight_filler { 1974 | type: "gaussian" 1975 | std: 0.01 1976 | } 1977 | bias_filler { 1978 | type: "constant" 1979 | } 1980 | } 1981 | } 1982 | layer { 1983 | name: "Mrelu6_stage4_L1" 1984 | type: "ReLU" 1985 | bottom: "Mconv6_stage4_L1" 1986 | top: "Mconv6_stage4_L1" 1987 | } 1988 | layer { 1989 | name: "Mconv6_stage4_L2" 1990 | type: "Convolution" 1991 | bottom: "Mconv5_stage4_L2" 1992 | top: "Mconv6_stage4_L2" 1993 | param { 1994 | lr_mult: 4.0 1995 | decay_mult: 1 1996 | } 1997 | param { 1998 | lr_mult: 8.0 1999 | decay_mult: 0 2000 | } 2001 | convolution_param { 2002 | num_output: 128 2003 | pad: 0 2004 | kernel_size: 1 2005 | weight_filler { 2006 | type: "gaussian" 2007 | std: 0.01 2008 | } 2009 | bias_filler { 2010 | type: "constant" 2011 | } 2012 | } 2013 | } 2014 | layer { 2015 | name: "Mrelu6_stage4_L2" 2016 | type: "ReLU" 2017 | bottom: "Mconv6_stage4_L2" 2018 | top: "Mconv6_stage4_L2" 2019 | } 2020 | layer { 2021 | name: "Mconv7_stage4_L1" 2022 | type: "Convolution" 2023 | bottom: "Mconv6_stage4_L1" 2024 | top: "Mconv7_stage4_L1" 2025 | param { 2026 | lr_mult: 4.0 2027 | decay_mult: 1 2028 | } 2029 | param { 2030 | lr_mult: 8.0 2031 | decay_mult: 0 2032 | } 2033 | convolution_param { 2034 | num_output: 28 2035 | pad: 0 2036 | kernel_size: 1 2037 | weight_filler { 2038 | type: "gaussian" 2039 | std: 0.01 2040 | } 2041 | bias_filler { 2042 | type: "constant" 2043 | } 2044 | } 2045 | } 2046 | layer { 2047 | name: "Mconv7_stage4_L2" 2048 | type: "Convolution" 2049 | bottom: "Mconv6_stage4_L2" 2050 | top: "Mconv7_stage4_L2" 2051 | param { 2052 | lr_mult: 4.0 2053 | decay_mult: 1 2054 | } 2055 | param { 2056 | lr_mult: 8.0 2057 | decay_mult: 0 2058 | } 2059 | convolution_param { 2060 | num_output: 16 2061 | pad: 0 2062 | kernel_size: 1 2063 | weight_filler { 2064 | type: "gaussian" 2065 | std: 0.01 2066 | } 2067 | bias_filler { 2068 | type: "constant" 2069 | } 2070 | } 2071 | } 2072 | layer { 2073 | name: "concat_stage7" 2074 | type: "Concat" 2075 | bottom: "Mconv7_stage4_L2" 2076 | bottom: "Mconv7_stage4_L1" 2077 | top: "net_output" 2078 | concat_param { 2079 | axis: 1 2080 | } 2081 | } 2082 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/tello.py: -------------------------------------------------------------------------------- 1 | import socket 2 | import threading 3 | import time 4 | import numpy as np 5 | import libh264decoder 6 | 7 | class Tello: 8 | """Wrapper class to interact with the Tello drone.""" 9 | 10 | def __init__(self, local_ip, local_port, imperial=False, command_timeout=.3, tello_ip='192.168.10.1', 11 | tello_port=8889): 12 | """ 13 | Binds to the local IP/port and puts the Tello into command mode. 14 | 15 | :param local_ip (str): Local IP address to bind. 16 | :param local_port (int): Local port to bind. 17 | :param imperial (bool): If True, speed is MPH and distance is feet. 18 | If False, speed is KPH and distance is meters. 19 | :param command_timeout (int|float): Number of seconds to wait for a response to a command. 20 | :param tello_ip (str): Tello IP. 21 | :param tello_port (int): Tello port. 22 | """ 23 | 24 | self.abort_flag = False 25 | self.decoder = libh264decoder.H264Decoder() 26 | self.command_timeout = command_timeout 27 | self.imperial = imperial 28 | self.response = None 29 | self.frame = None # numpy array BGR -- current camera output frame 30 | self.is_freeze = False # freeze current camera output 31 | self.last_frame = None 32 | self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # socket for sending cmd 33 | self.socket_video = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # socket for receiving video stream 34 | self.tello_address = (tello_ip, tello_port) 35 | self.local_video_port = 11111 # port for receiving video stream 36 | self.last_height = 0 37 | self.socket.bind((local_ip, local_port)) 38 | 39 | # thread for receiving cmd ack 40 | self.receive_thread = threading.Thread(target=self._receive_thread) 41 | self.receive_thread.daemon = True 42 | 43 | self.receive_thread.start() 44 | 45 | # to receive video -- send cmd: command, streamon 46 | self.socket.sendto(b'command', self.tello_address) 47 | print ('sent: command') 48 | self.socket.sendto(b'streamon', self.tello_address) 49 | print ('sent: streamon') 50 | 51 | self.socket_video.bind((local_ip, self.local_video_port)) 52 | 53 | # thread for receiving video 54 | self.receive_video_thread = threading.Thread(target=self._receive_video_thread) 55 | self.receive_video_thread.daemon = True 56 | 57 | self.receive_video_thread.start() 58 | 59 | def __del__(self): 60 | """Closes the local socket.""" 61 | 62 | self.socket.close() 63 | self.socket_video.close() 64 | 65 | def read(self): 66 | """Return the last frame from camera.""" 67 | if self.is_freeze: 68 | return self.last_frame 69 | else: 70 | return self.frame 71 | 72 | def video_freeze(self, is_freeze=True): 73 | """Pause video output -- set is_freeze to True""" 74 | self.is_freeze = is_freeze 75 | if is_freeze: 76 | self.last_frame = self.frame 77 | 78 | def _receive_thread(self): 79 | """Listen to responses from the Tello. 80 | 81 | Runs as a thread, sets self.response to whatever the Tello last returned. 82 | 83 | """ 84 | while True: 85 | try: 86 | self.response, ip = self.socket.recvfrom(3000) 87 | #print(self.response) 88 | except socket.error as exc: 89 | print ("Caught exception socket.error : %s" % exc) 90 | 91 | def _receive_video_thread(self): 92 | """ 93 | Listens for video streaming (raw h264) from the Tello. 94 | 95 | Runs as a thread, sets self.frame to the most recent frame Tello captured. 96 | 97 | """ 98 | packet_data = "" 99 | while True: 100 | try: 101 | res_string, ip = self.socket_video.recvfrom(2048) 102 | packet_data += res_string 103 | # end of frame 104 | if len(res_string) != 1460: 105 | for frame in self._h264_decode(packet_data): 106 | self.frame = frame 107 | packet_data = "" 108 | 109 | except socket.error as exc: 110 | print ("Caught exception socket.error : %s" % exc) 111 | 112 | def _h264_decode(self, packet_data): 113 | """ 114 | decode raw h264 format data from Tello 115 | 116 | :param packet_data: raw h264 data array 117 | 118 | :return: a list of decoded frame 119 | """ 120 | res_frame_list = [] 121 | frames = self.decoder.decode(packet_data) 122 | for framedata in frames: 123 | (frame, w, h, ls) = framedata 124 | if frame is not None: 125 | # print 'frame size %i bytes, w %i, h %i, linesize %i' % (len(frame), w, h, ls) 126 | 127 | frame = np.fromstring(frame, dtype=np.ubyte, count=len(frame), sep='') 128 | frame = (frame.reshape((h, ls / 3, 3))) 129 | frame = frame[:, :w, :] 130 | res_frame_list.append(frame) 131 | 132 | return res_frame_list 133 | 134 | def send_command(self, command): 135 | """ 136 | Send a command to the Tello and wait for a response. 137 | 138 | :param command: Command to send. 139 | :return (str): Response from Tello. 140 | 141 | """ 142 | 143 | print (">> send cmd: {}".format(command)) 144 | self.abort_flag = False 145 | timer = threading.Timer(self.command_timeout, self.set_abort_flag) 146 | 147 | self.socket.sendto(command.encode('utf-8'), self.tello_address) 148 | 149 | timer.start() 150 | while self.response is None: 151 | if self.abort_flag is True: 152 | break 153 | timer.cancel() 154 | 155 | if self.response is None: 156 | response = 'none_response' 157 | else: 158 | response = self.response.decode('utf-8') 159 | 160 | self.response = None 161 | 162 | return response 163 | 164 | def set_abort_flag(self): 165 | """ 166 | Sets self.abort_flag to True. 167 | 168 | Used by the timer in Tello.send_command() to indicate to that a response 169 | 170 | timeout has occurred. 171 | 172 | """ 173 | 174 | self.abort_flag = True 175 | 176 | def takeoff(self): 177 | """ 178 | Initiates take-off. 179 | 180 | Returns: 181 | str: Response from Tello, 'OK' or 'FALSE'. 182 | 183 | """ 184 | 185 | return self.send_command('takeoff') 186 | 187 | def set_speed(self, speed): 188 | """ 189 | Sets speed. 190 | 191 | This method expects KPH or MPH. The Tello API expects speeds from 192 | 1 to 100 centimeters/second. 193 | 194 | Metric: .1 to 3.6 KPH 195 | Imperial: .1 to 2.2 MPH 196 | 197 | Args: 198 | speed (int|float): Speed. 199 | 200 | Returns: 201 | str: Response from Tello, 'OK' or 'FALSE'. 202 | 203 | """ 204 | 205 | speed = float(speed) 206 | 207 | if self.imperial is True: 208 | speed = int(round(speed * 44.704)) 209 | else: 210 | speed = int(round(speed * 27.7778)) 211 | 212 | return self.send_command('speed %s' % speed) 213 | 214 | def rotate_cw(self, degrees): 215 | """ 216 | Rotates clockwise. 217 | 218 | Args: 219 | degrees (int): Degrees to rotate, 1 to 360. 220 | 221 | Returns: 222 | str: Response from Tello, 'OK' or 'FALSE'. 223 | 224 | """ 225 | 226 | return self.send_command('cw %s' % degrees) 227 | 228 | def rotate_ccw(self, degrees): 229 | """ 230 | Rotates counter-clockwise. 231 | 232 | Args: 233 | degrees (int): Degrees to rotate, 1 to 360. 234 | 235 | Returns: 236 | str: Response from Tello, 'OK' or 'FALSE'. 237 | 238 | """ 239 | return self.send_command('ccw %s' % degrees) 240 | 241 | def flip(self, direction): 242 | """ 243 | Flips. 244 | 245 | Args: 246 | direction (str): Direction to flip, 'l', 'r', 'f', 'b'. 247 | 248 | Returns: 249 | str: Response from Tello, 'OK' or 'FALSE'. 250 | 251 | """ 252 | 253 | return self.send_command('flip %s' % direction) 254 | 255 | def get_response(self): 256 | """ 257 | Returns response of tello. 258 | 259 | Returns: 260 | int: response of tello. 261 | 262 | """ 263 | response = self.response 264 | return response 265 | 266 | def get_height(self): 267 | """Returns height(dm) of tello. 268 | 269 | Returns: 270 | int: Height(dm) of tello. 271 | 272 | """ 273 | height = self.send_command('height?') 274 | height = str(height) 275 | height = filter(str.isdigit, height) 276 | try: 277 | height = int(height) 278 | self.last_height = height 279 | except: 280 | height = self.last_height 281 | pass 282 | return height 283 | 284 | def get_battery(self): 285 | """Returns percent battery life remaining. 286 | 287 | Returns: 288 | int: Percent battery life remaining. 289 | 290 | """ 291 | 292 | battery = self.send_command('battery?') 293 | 294 | try: 295 | battery = int(battery) 296 | except: 297 | pass 298 | 299 | return battery 300 | 301 | def get_flight_time(self): 302 | """Returns the number of seconds elapsed during flight. 303 | 304 | Returns: 305 | int: Seconds elapsed during flight. 306 | 307 | """ 308 | 309 | flight_time = self.send_command('time?') 310 | 311 | try: 312 | flight_time = int(flight_time) 313 | except: 314 | pass 315 | 316 | return flight_time 317 | 318 | def get_speed(self): 319 | """Returns the current speed. 320 | 321 | Returns: 322 | int: Current speed in KPH or MPH. 323 | 324 | """ 325 | 326 | speed = self.send_command('speed?') 327 | 328 | try: 329 | speed = float(speed) 330 | 331 | if self.imperial is True: 332 | speed = round((speed / 44.704), 1) 333 | else: 334 | speed = round((speed / 27.7778), 1) 335 | except: 336 | pass 337 | 338 | return speed 339 | 340 | def land(self): 341 | """Initiates landing. 342 | 343 | Returns: 344 | str: Response from Tello, 'OK' or 'FALSE'. 345 | 346 | """ 347 | 348 | return self.send_command('land') 349 | 350 | def move(self, direction, distance): 351 | """Moves in a direction for a distance. 352 | 353 | This method expects meters or feet. The Tello API expects distances 354 | from 20 to 500 centimeters. 355 | 356 | Metric: .02 to 5 meters 357 | Imperial: .7 to 16.4 feet 358 | 359 | Args: 360 | direction (str): Direction to move, 'forward', 'back', 'right' or 'left'. 361 | distance (int|float): Distance to move. 362 | 363 | Returns: 364 | str: Response from Tello, 'OK' or 'FALSE'. 365 | 366 | """ 367 | 368 | distance = float(distance) 369 | 370 | if self.imperial is True: 371 | distance = int(round(distance * 30.48)) 372 | else: 373 | distance = int(round(distance * 100)) 374 | 375 | return self.send_command('%s %s' % (direction, distance)) 376 | 377 | def move_backward(self, distance): 378 | """Moves backward for a distance. 379 | 380 | See comments for Tello.move(). 381 | 382 | Args: 383 | distance (int): Distance to move. 384 | 385 | Returns: 386 | str: Response from Tello, 'OK' or 'FALSE'. 387 | 388 | """ 389 | 390 | return self.move('back', distance) 391 | 392 | def move_down(self, distance): 393 | """Moves down for a distance. 394 | 395 | See comments for Tello.move(). 396 | 397 | Args: 398 | distance (int): Distance to move. 399 | 400 | Returns: 401 | str: Response from Tello, 'OK' or 'FALSE'. 402 | 403 | """ 404 | 405 | return self.move('down', distance) 406 | 407 | def move_forward(self, distance): 408 | """Moves forward for a distance. 409 | 410 | See comments for Tello.move(). 411 | 412 | Args: 413 | distance (int): Distance to move. 414 | 415 | Returns: 416 | str: Response from Tello, 'OK' or 'FALSE'. 417 | 418 | """ 419 | return self.move('forward', distance) 420 | 421 | def move_left(self, distance): 422 | """Moves left for a distance. 423 | 424 | See comments for Tello.move(). 425 | 426 | Args: 427 | distance (int): Distance to move. 428 | 429 | Returns: 430 | str: Response from Tello, 'OK' or 'FALSE'. 431 | 432 | """ 433 | return self.move('left', distance) 434 | 435 | def move_right(self, distance): 436 | """Moves right for a distance. 437 | 438 | See comments for Tello.move(). 439 | 440 | Args: 441 | distance (int): Distance to move. 442 | 443 | """ 444 | return self.move('right', distance) 445 | 446 | def move_up(self, distance): 447 | """Moves up for a distance. 448 | 449 | See comments for Tello.move(). 450 | 451 | Args: 452 | distance (int): Distance to move. 453 | 454 | Returns: 455 | str: Response from Tello, 'OK' or 'FALSE'. 456 | 457 | """ 458 | 459 | return self.move('up', distance) 460 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/tello_control_ui.py: -------------------------------------------------------------------------------- 1 | from PIL import Image 2 | from PIL import ImageTk 3 | import Tkinter as tki 4 | from Tkinter import Toplevel, Scale 5 | import threading 6 | import datetime 7 | import cv2 8 | import os 9 | import time 10 | from tello_pose import Tello_Pose 11 | import platform 12 | 13 | class TelloUI: 14 | """Wrapper class to enable the GUI.""" 15 | 16 | def __init__(self,tello,outputpath): 17 | """ 18 | Initial all the element of the GUI,support by Tkinter 19 | 20 | :param tello: class interacts with the Tello drone. 21 | 22 | Raises: 23 | RuntimeError: If the Tello rejects the attempt to enter command mode. 24 | """ 25 | 26 | self.tello = tello # videostream device 27 | self.outputPath = outputpath # the path that save pictures created by clicking the takeSnapshot button 28 | self.frame = None # frame read from h264decoder and used for pose recognition 29 | self.thread = None # thread of the Tkinter mainloop 30 | self.stopEvent = None 31 | 32 | # control variables 33 | self.distance = 0.1 # default distance for 'move' cmd 34 | self.degree = 30 # default degree for 'cw' or 'ccw' cmd 35 | # if the pose recognition mode is opened 36 | self.pose_mode = False 37 | # if the flag is TRUE,the auto-takeoff thread will stop waiting for the response from tello 38 | self.quit_waiting_flag = False 39 | 40 | # if the flag is TRUE,the pose recognition skeleton will be drawn on the GUI picture 41 | self.draw_skeleton_flag = False 42 | # pose recognition 43 | self.my_tello_pose = Tello_Pose() 44 | 45 | # record the coordinates of the nodes in the pose recognition skeleton 46 | self.points = [] 47 | #list of all the possible connections between skeleton nodes 48 | self.POSE_PAIRS = [[0,1], [1,2], [2,3], [3,4], [1,5], [5,6], [6,7], [1,14], [14,8], [8,9], [9,10], [14,11], [11,12], [12,13] ] 49 | 50 | # initialize the root window and image panel 51 | self.root = tki.Tk() 52 | self.panel = None 53 | # self.panel_for_pose_handle_show = None 54 | 55 | # create buttons 56 | self.btn_snapshot = tki.Button(self.root, text="Snapshot!", 57 | command=self.takeSnapshot) 58 | self.btn_snapshot.pack(side="bottom", fill="both", 59 | expand="yes", padx=10, pady=5) 60 | 61 | self.btn_pose = tki.Button(self.root, text="Pose Recognition Status: Off", 62 | command=self.setPoseMode) 63 | self.btn_pose.pack(side="bottom", fill="both", 64 | expand="yes", padx=10, pady=5) 65 | 66 | self.btn_pause = tki.Button(self.root, text="Pause", relief="raised", command=self.pauseVideo) 67 | self.btn_pause.pack(side="bottom", fill="both", 68 | expand="yes", padx=10, pady=5) 69 | 70 | self.btn_landing = tki.Button( 71 | self.root, text="Open Command Panel", relief="raised", command=self.openCmdWindow) 72 | self.btn_landing.pack(side="bottom", fill="both", 73 | expand="yes", padx=10, pady=5) 74 | 75 | # start a thread that constantly pools the video sensor for 76 | # the most recently read frame 77 | self.stopEvent = threading.Event() 78 | self.thread = threading.Thread(target=self.videoLoop, args=()) 79 | self.thread.start() 80 | 81 | # set a callback to handle when the window is closed 82 | self.root.wm_title("TELLO Controller") 83 | self.root.wm_protocol("WM_DELETE_WINDOW", self.onClose) 84 | 85 | # the auto-takeoff thread will start if the 'takeoff' button on command window is clicked 86 | self.auto_takeoff_thread = threading.Thread(target=self._autoTakeoff) 87 | # the sending_command will send command to tello every 5 seconds 88 | self.sending_command_thread = threading.Thread(target = self._sendingCommand) 89 | self.get_GUI_Image_thread = threading.Thread(target = self._getGUIImage) 90 | def videoLoop(self): 91 | """ 92 | The mainloop thread of Tkinter 93 | Raises: 94 | RuntimeError: To get around a RunTime error that Tkinter throws due to threading. 95 | """ 96 | try: 97 | # start the thread that get GUI image and drwa skeleton 98 | time.sleep(0.5) 99 | self.get_GUI_Image_thread.start() 100 | while not self.stopEvent.is_set(): 101 | # read the frame for pose recognition 102 | self.frame = self.tello.read() 103 | if self.frame is None or self.frame.size == 0: 104 | continue 105 | # smoothing filter 106 | self.frame = cv2.bilateralFilter(self.frame, 5, 50, 100) 107 | 108 | cmd = '' 109 | self.points.append(None) 110 | # process pose-recognition 111 | if self.pose_mode: 112 | cmd,self.draw_skeleton_flag,self.points = self.my_tello_pose.detect(self.frame) 113 | 114 | # process command - map your motion to whatever Tello movement you want! 115 | if cmd == 'moveback': 116 | self.telloMoveBackward(0.50) 117 | elif cmd == 'moveforward': 118 | self.telloMoveForward(0.50) 119 | elif cmd == 'land': 120 | self.telloLanding() 121 | 122 | except RuntimeError, e: 123 | print("[INFO] caught a RuntimeError") 124 | 125 | def _getGUIImage(self): 126 | """ 127 | Main operation to read frames from h264decoder and draw skeleton on 128 | frames if the pose mode is opened 129 | """ 130 | # read the system of your computer 131 | system = platform.system() 132 | while not self.stopEvent.is_set(): 133 | # read the frame for GUI show 134 | frame = self.tello.read() 135 | if frame is None or frame.size == 0: 136 | continue 137 | if self.pose_mode: 138 | # Draw the detected skeleton points 139 | for i in range(15): 140 | if self.draw_skeleton_flag == True: 141 | cv2.circle(frame, self.points[i], 8, (0, 255, 255), thickness=-1, lineType=cv2.FILLED) 142 | cv2.putText(frame, "{}".format(i), self.points[i], cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2,lineType=cv2.LINE_AA) 143 | # Draw Skeleton 144 | for pair in self.POSE_PAIRS: 145 | partA = pair[0] 146 | partB = pair[1] 147 | if self.points[partA] and self.points[partB]: 148 | cv2.line(frame, self.points[partA], self.points[partB], (0, 255, 255), 2) 149 | cv2.circle(frame, self.points[partA], 8, (0, 0, 255), thickness=-1, lineType=cv2.FILLED) 150 | 151 | # transfer the format from frame to image 152 | image = Image.fromarray(frame) 153 | 154 | # we found compatibility problem between Tkinter,PIL and Macos,and it will 155 | # sometimes result the very long preriod of the "ImageTk.PhotoImage" function, 156 | # so for Macos,we start a new thread to execute the _updateGUIImage function. 157 | if system =="Windows" or system =="Linux": 158 | self._updateGUIImage(image) 159 | 160 | else: 161 | thread_tmp = threading.Thread(target=self._updateGUIImage,args=(image,)) 162 | thread_tmp.start() 163 | time.sleep(0.03) 164 | 165 | def _updateGUIImage(self,image): 166 | """ 167 | Main operation to initial the object of image,and update the GUI panel 168 | """ 169 | image = ImageTk.PhotoImage(image) 170 | # if the panel none ,we need to initial it 171 | if self.panel is None: 172 | self.panel = tki.Label(image=image) 173 | self.panel.image = image 174 | self.panel.pack(side="left", padx=10, pady=10) 175 | # otherwise, simply update the panel 176 | else: 177 | self.panel.configure(image=image) 178 | self.panel.image = image 179 | 180 | def _autoTakeoff(self): 181 | """ 182 | Firstly,it will waiting for the response that will be sent by Tello if Tello 183 | 184 | finish the takeoff command.If computer doesn't receive the response,it may be 185 | 186 | because tello doesn't takeoff normally,or because the UDP pack of response is 187 | 188 | lost.So in order to confirm the reason,computer will send 'height?'command to 189 | 190 | get several real-time height datas and get a average value.If the height is in 191 | 192 | normal range,tello will execute the moveup command.Otherwise,tello will land. 193 | 194 | Finally,the sending-command thread will start. 195 | """ 196 | response = None 197 | height_tmp = 0 # temp variable to content value of height 198 | height_val = 0 # average value of height 199 | cnt = 0 # effective number of height reading 200 | timeout = 6 # max waiting time of tello's response 201 | 202 | timer = threading.Timer(timeout, self._setQuitWaitingFlag) 203 | timer.start() 204 | 205 | # waiting for the response from tello 206 | while response != 'ok' : 207 | if self.quit_waiting_flag is True: 208 | break 209 | response = self.tello.get_response() 210 | print "ack:%s"%response 211 | timer.cancel() 212 | 213 | # receive the correct response 214 | if response == 'ok': 215 | self.tello.move_up(0.5) 216 | 217 | # calculate the height of tello 218 | else: 219 | for i in range(0,50): 220 | height_tmp = self.tello.get_height() 221 | try: 222 | height_val = height_val + height_tmp 223 | cnt = cnt + 1 224 | print height_tmp,cnt 225 | except: 226 | height_val = height_val 227 | 228 | height_val = height_val/cnt 229 | 230 | # if the height value is in normal range 231 | if height_val == 9 or height_val == 10 or height_val == 11: 232 | self.tello.move_up(0.5) 233 | else: 234 | self.tello.land() 235 | # start the sendingCmd thread 236 | self.sending_command_thread.start() 237 | 238 | def _sendingCommand(self): 239 | """ 240 | start a while loop that sends 'command' to tello every 5 second 241 | """ 242 | 243 | while True: 244 | self.tello.send_command('command') 245 | time.sleep(5) 246 | 247 | def _setQuitWaitingFlag(self): 248 | """ 249 | set the variable as TRUE,it will stop computer waiting for response from tello 250 | """ 251 | self.quit_waiting_flag = True 252 | 253 | def openCmdWindow(self): 254 | """ 255 | open the cmd window and initial all the button and text 256 | """ 257 | panel = Toplevel(self.root) 258 | panel.wm_title("Command Panel") 259 | 260 | # create text input entry 261 | text0 = tki.Label(panel, 262 | text='This Controller map keyboard inputs to Tello control commands\n' 263 | 'Adjust the trackbar to reset distance and degree parameter', 264 | font='Helvetica 10 bold' 265 | ) 266 | text0.pack(side='top') 267 | 268 | text1 = tki.Label(panel, text= 269 | 'W - Move Tello Up\t\t\tArrow Up - Move Tello Forward\n' 270 | 'S - Move Tello Down\t\t\tArrow Down - Move Tello Backward\n' 271 | 'A - Rotate Tello Counter-Clockwise\tArrow Left - Move Tello Left\n' 272 | 'D - Rotate Tello Clockwise\t\tArrow Right - Move Tello Right', 273 | justify="left") 274 | text1.pack(side="top") 275 | 276 | self.btn_landing = tki.Button( 277 | panel, text="Land", relief="raised", command=self.telloLanding) 278 | self.btn_landing.pack(side="bottom", fill="both", 279 | expand="yes", padx=10, pady=5) 280 | 281 | self.btn_takeoff = tki.Button( 282 | panel, text="Takeoff", relief="raised", command=self.telloTakeOff) 283 | self.btn_takeoff.pack(side="bottom", fill="both", 284 | expand="yes", padx=10, pady=5) 285 | 286 | # binding arrow keys to drone control 287 | self.tmp_f = tki.Frame(panel, width=100, height=2) 288 | self.tmp_f.bind('', self.on_keypress_w) 289 | self.tmp_f.bind('', self.on_keypress_s) 290 | self.tmp_f.bind('', self.on_keypress_a) 291 | self.tmp_f.bind('', self.on_keypress_d) 292 | self.tmp_f.bind('', self.on_keypress_up) 293 | self.tmp_f.bind('', self.on_keypress_down) 294 | self.tmp_f.bind('', self.on_keypress_left) 295 | self.tmp_f.bind('', self.on_keypress_right) 296 | self.tmp_f.pack(side="bottom") 297 | self.tmp_f.focus_set() 298 | 299 | self.btn_landing = tki.Button( 300 | panel, text="Flip", relief="raised", command=self.openFlipWindow) 301 | self.btn_landing.pack(side="bottom", fill="both", 302 | expand="yes", padx=10, pady=5) 303 | 304 | self.distance_bar = Scale(panel, from_=0.02, to=5, tickinterval=0.01, digits=3, label='Distance(m)', 305 | resolution=0.01) 306 | self.distance_bar.set(0.2) 307 | self.distance_bar.pack(side="left") 308 | 309 | self.btn_distance = tki.Button(panel, text="Reset Distance", relief="raised", 310 | command=self.updateDistancebar, 311 | ) 312 | self.btn_distance.pack(side="left", fill="both", 313 | expand="yes", padx=10, pady=5) 314 | 315 | self.degree_bar = Scale(panel, from_=1, to=360, tickinterval=10, label='Degree') 316 | self.degree_bar.set(30) 317 | self.degree_bar.pack(side="right") 318 | 319 | self.btn_distance = tki.Button(panel, text="Reset Degree", relief="raised", command=self.updateDegreebar) 320 | self.btn_distance.pack(side="right", fill="both", 321 | expand="yes", padx=10, pady=5) 322 | 323 | def openFlipWindow(self): 324 | """ 325 | open the flip window and initial all the button and text 326 | """ 327 | 328 | panel = Toplevel(self.root) 329 | panel.wm_title("Gesture Recognition") 330 | 331 | self.btn_flipl = tki.Button( 332 | panel, text="Flip Left", relief="raised", command=self.telloFlip_l) 333 | self.btn_flipl.pack(side="bottom", fill="both", 334 | expand="yes", padx=10, pady=5) 335 | 336 | self.btn_flipr = tki.Button( 337 | panel, text="Flip Right", relief="raised", command=self.telloFlip_r) 338 | self.btn_flipr.pack(side="bottom", fill="both", 339 | expand="yes", padx=10, pady=5) 340 | 341 | self.btn_flipf = tki.Button( 342 | panel, text="Flip Forward", relief="raised", command=self.telloFlip_f) 343 | self.btn_flipf.pack(side="bottom", fill="both", 344 | expand="yes", padx=10, pady=5) 345 | 346 | self.btn_flipb = tki.Button( 347 | panel, text="Flip Backward", relief="raised", command=self.telloFlip_b) 348 | self.btn_flipb.pack(side="bottom", fill="both", 349 | expand="yes", padx=10, pady=5) 350 | 351 | def takeSnapshot(self): 352 | """ 353 | save the current frame of the video as a jpg file and put it into outputpath 354 | """ 355 | 356 | # grab the current timestamp and use it to construct the filename 357 | ts = datetime.datetime.now() 358 | filename = "{}.jpg".format(ts.strftime("%Y-%m-%d_%H-%M-%S")) 359 | 360 | p = os.path.sep.join((self.outputPath, filename)) 361 | 362 | # save the file 363 | cv2.imwrite(p, cv2.cvtColor(self.frame, cv2.COLOR_RGB2BGR)) 364 | print("[INFO] saved {}".format(filename)) 365 | 366 | def setPoseMode(self): 367 | """ 368 | Toggle the open/close of pose recognition mode 369 | """ 370 | if self.pose_mode is False: 371 | self.pose_mode = True 372 | self.btn_pose.config(text='Pose Recognition Status: On') 373 | else: 374 | self.pose_mode = False 375 | self.btn_pose.config(text='Pose Recognition Status: Off') 376 | 377 | def pauseVideo(self): 378 | """ 379 | Toggle the freeze/unfreze of video 380 | """ 381 | if self.btn_pause.config('relief')[-1] == 'sunken': 382 | self.btn_pause.config(relief="raised") 383 | self.tello.video_freeze(False) 384 | else: 385 | self.btn_pause.config(relief="sunken") 386 | self.tello.video_freeze(True) 387 | 388 | def telloTakeOff(self): 389 | """ 390 | send the takeoff command to tello,and wait for the first response, 391 | 392 | if get the 'error'response,remind the "battery low" warning.Otherwise, 393 | 394 | start the auto-takeoff thread 395 | """ 396 | takeoff_response = None 397 | 398 | self.tello.takeoff() 399 | time.sleep(0.2) 400 | 401 | takeoff_response = self.tello.get_response() 402 | 403 | if takeoff_response != 'error': 404 | self.auto_takeoff_thread.start() 405 | else: 406 | print "battery low,please repalce with a new one" 407 | 408 | def telloLanding(self): 409 | return self.tello.land() 410 | 411 | def telloFlip_l(self): 412 | return self.tello.flip('l') 413 | 414 | def telloFlip_r(self): 415 | return self.tello.flip('r') 416 | 417 | def telloFlip_f(self): 418 | return self.tello.flip('f') 419 | 420 | def telloFlip_b(self): 421 | return self.tello.flip('b') 422 | 423 | def telloCW(self, degree): 424 | return self.tello.rotate_cw(degree) 425 | 426 | def telloCCW(self, degree): 427 | return self.tello.rotate_ccw(degree) 428 | 429 | def telloMoveForward(self, distance): 430 | return self.tello.move_forward(distance) 431 | 432 | def telloMoveBackward(self, distance): 433 | return self.tello.move_backward(distance) 434 | 435 | def telloMoveLeft(self, distance): 436 | return self.tello.move_left(distance) 437 | 438 | def telloMoveRight(self, distance): 439 | return self.tello.move_right(distance) 440 | 441 | def telloUp(self, dist): 442 | return self.tello.move_up(dist) 443 | 444 | def telloDown(self, dist): 445 | return self.tello.move_down(dist) 446 | 447 | def updateTrackBar(self): 448 | self.my_tello_hand.setThr(self.hand_thr_bar.get()) 449 | 450 | def updateDistancebar(self): 451 | self.distance = self.distance_bar.get() 452 | print 'reset distance to %.1f' % self.distance 453 | 454 | def updateDegreebar(self): 455 | self.degree = self.degree_bar.get() 456 | print 'reset distance to %d' % self.degree 457 | 458 | def on_keypress_w(self, event): 459 | print "up %d m" % self.distance 460 | self.telloUp(self.distance) 461 | 462 | def on_keypress_s(self, event): 463 | print "down %d m" % self.distance 464 | self.telloDown(self.distance) 465 | 466 | def on_keypress_a(self, event): 467 | print "ccw %d degree" % self.degree 468 | self.tello.rotate_ccw(self.degree) 469 | 470 | def on_keypress_d(self, event): 471 | print "cw %d m" % self.degree 472 | self.tello.rotate_cw(self.degree) 473 | 474 | def on_keypress_up(self, event): 475 | print "forward %d m" % self.distance 476 | self.telloMoveForward(self.distance) 477 | 478 | def on_keypress_down(self, event): 479 | print "backward %d m" % self.distance 480 | self.telloMoveBackward(self.distance) 481 | 482 | def on_keypress_left(self, event): 483 | print "left %d m" % self.distance 484 | self.telloMoveLeft(self.distance) 485 | 486 | def on_keypress_right(self, event): 487 | print "right %d m" % self.distance 488 | self.telloMoveRight(self.distance) 489 | 490 | def on_keypress_enter(self, event): 491 | if self.frame is not None: 492 | self.registerFace() 493 | self.tmp_f.focus_set() 494 | 495 | def onClose(self): 496 | """ 497 | set the stop event, cleanup the camera, and allow the rest of 498 | 499 | the quit process to continue 500 | """ 501 | print("[INFO] closing...") 502 | self.stopEvent.set() 503 | del self.tello 504 | self.root.quit() 505 | 506 | -------------------------------------------------------------------------------- /Tello_Video_With_Pose_Recognition/tello_pose.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import time 3 | import math 4 | import numpy as np 5 | import datetime 6 | import os 7 | 8 | 9 | ''' 10 | Correspondence between the number of the skeleton node and 11 | the human body: 12 | Head - 0, Neck - 1, Right Shoulder - 2, Right Elbow - 3, Right Wrist - 4, 13 | Left Shoulder - 5, Left Elbow - 6, Left Wrist - 7, Right Hip - 8, 14 | Right Knee - 9, Right Ankle - 10, Left Hip - 11, Left Knee - 12, 15 | Left Ankle - 13, Chest - 14, Background - 15 16 | ''' 17 | 18 | class Tello_Pose: 19 | def __init__(self): 20 | 21 | # read the path of the trained model of the neural network for pose recognition 22 | self.protoFile = "model/pose/mpi/pose_deploy_linevec_faster_4_stages.prototxt" 23 | self.weightsFile = "model/pose/mpi/pose_iter_160000.caffemodel" 24 | 25 | # total number of the skeleton nodes 26 | self.nPoints = 15 27 | 28 | # read the neural network of the pose recognition 29 | self.net = cv2.dnn.readNetFromCaffe(self.protoFile, self.weightsFile) 30 | 31 | # count the number of frames,and after every certain number of frames 32 | # is read, frame_cnt will be cleared and recounted. 33 | self.frame_cnt = 0 34 | self.arm_down_45_cnt = 0 # count numbers of the arm_dowm_45 captured in every certain number of frames 35 | self.arm_flat_cnt = 0 # count numbers of the arm_flat captured in every certain number of frames 36 | self.arm_V_cnt = 0 # count numbers of the arm_V captured in every certain number of frames 37 | 38 | # the period of pose reconigtion,it depends on your computer performance 39 | self.period = 0 40 | # record how many times the period of pose reconigtion is calculated. 41 | self.period_calculate_cnt =0 42 | 43 | 44 | def getAngle(self, start, end): 45 | """ 46 | Calculate the angle between start and end 47 | 48 | :param start: start point [x, y] 49 | :param end: end point [x, y] 50 | :return: the clockwise angle from start to end 51 | """ 52 | angle = int(math.atan2((start[1] - end[1]), (start[0] - end[0])) * 180 / math.pi) 53 | return angle 54 | 55 | def is_arms_down_45(self, points): 56 | """ 57 | Determine if the person is holding the arms 58 | like: 59 | | 60 | / | \ 61 | / \ 62 | 63 | 64 | :param points: set of body key points 65 | :return: if the person detected moves both of his arms down for about 45 degrees 66 | """ 67 | right = False 68 | if points[2] and points[3] and points[4]: 69 | # calculate the shoulder angle 70 | shoulder_angle = self.getAngle(points[2], points[3]) 71 | 72 | if -60 < shoulder_angle < -20: 73 | elbow_angle = self.getAngle(points[3], points[4]) 74 | # if arm is straight 75 | if abs(elbow_angle - shoulder_angle) < 25: 76 | right = True 77 | 78 | left = False 79 | if points[5] and points[6] and points[7]: 80 | shoulder_angle = self.getAngle(points[5], points[6]) 81 | # correct the dimension 82 | if shoulder_angle < 0: 83 | shoulder_angle = shoulder_angle + 360 84 | 85 | if 200 < shoulder_angle < 240: 86 | elbow_angle = self.getAngle(points[6], points[7]) 87 | if elbow_angle < 0: 88 | elbow_angle = elbow_angle + 360 89 | # if arm is straight 90 | if abs(elbow_angle - shoulder_angle) < 25: 91 | left = True 92 | # If at least one arm meets the requirements, it is considered a successful capture 93 | if left or right: 94 | return True 95 | else: 96 | return False 97 | 98 | def is_arms_flat(self, points): 99 | """ 100 | Determine if the person moves his arm flat 101 | like: _ _|_ _ 102 | | 103 | / \ 104 | :param points: set of body key points 105 | :return: if the person detected moves both of his arms flat 106 | """ 107 | right = False 108 | if points[2] and points[3] and points[4]: 109 | 110 | shoulder_angle = self.getAngle(points[2], points[3]) 111 | # if arm is flat 112 | if -10 < shoulder_angle < 40: 113 | elbow_angle = self.getAngle(points[3], points[4]) 114 | # if arm is straight 115 | if abs(elbow_angle - shoulder_angle) < 30: 116 | right = True 117 | 118 | left = False 119 | if points[5] and points[6] and points[7]: 120 | shoulder_angle = self.getAngle(points[5], points[6]) 121 | # correct the dimension 122 | if shoulder_angle < 0: 123 | shoulder_angle = shoulder_angle + 360 124 | 125 | # if arm is flat 126 | if 140 < shoulder_angle < 190: 127 | elbow_angle = self.getAngle(points[6], points[7]) 128 | if elbow_angle < 0: 129 | elbow_angle = elbow_angle + 360 130 | # if arm is straight 131 | if abs(elbow_angle - shoulder_angle) < 30: 132 | left = True 133 | # If at least one arm meets the requirements, it is considered a successful capture 134 | if left or right: 135 | return True 136 | else: 137 | return False 138 | 139 | def is_arms_V(self, points): 140 | """ 141 | Determine if the person has his/her shoulder and elbow to a certain degree 142 | like: | 143 | \/|\/ 144 | / \ 145 | 146 | :param points: set of body key pointss 147 | """ 148 | right = False 149 | 150 | if points[2] and points[3] and points[4]: 151 | shoulder_angle = self.getAngle(points[2], points[3]) 152 | 153 | if -60 < shoulder_angle < -20: 154 | elbow_angle = self.getAngle(points[3], points[4]) 155 | if 0 < elbow_angle < 90 : 156 | right = True 157 | 158 | left = False 159 | if points[5] and points[6] and points[7]: 160 | shoulder_angle = self.getAngle(points[5], points[6]) 161 | # correct the dimension 162 | if shoulder_angle < 0: 163 | shoulder_angle = shoulder_angle + 360 164 | if 200 < shoulder_angle < 240: 165 | elbow_angle = self.getAngle(points[6], points[7]) 166 | if 90 < elbow_angle < 180: 167 | left = True 168 | 169 | # If at least one arm meets the requirements, it is considered a successful capture 170 | if left or right: 171 | return True 172 | else: 173 | return False 174 | 175 | 176 | def detect(self, frame): 177 | """ 178 | Main operation to recognize body pose using a trained model 179 | 180 | :param frame: raw h264 decoded frame 181 | :return: 182 | draw_skeleton_flag: the flag that indicates if the skeleton 183 | are detected and depend if the skeleton is drawn on the pic 184 | cmd: the command to be received by Tello 185 | points:the coordinates of the skeleton nodes 186 | """ 187 | frameWidth = frame.shape[1] 188 | frameHeight = frame.shape[0] 189 | 190 | frame_cnt_threshold = 0 191 | prob_threshold = 0.05 192 | pose_captured_threshold = 0 193 | 194 | draw_skeleton_flag = False 195 | 196 | # input image dimensions for the network 197 | # IMPORTANT: 198 | # Greater inWidth and inHeight will result in higher accuracy but longer process time 199 | # Smaller inWidth and inHeight will result in lower accuracy but shorter process time 200 | # Play around it by yourself to get best result! 201 | inWidth = 168 202 | inHeight = 168 203 | inpBlob = cv2.dnn.blobFromImage(frame, 1.0 / 255, (inWidth, inHeight), 204 | (0, 0, 0), swapRB=False, crop=False) 205 | self.net.setInput(inpBlob) 206 | 207 | # get the output of the neural network and calculate the period of the process 208 | period_starttime = time.time() 209 | output = self.net.forward() 210 | period_endtime = time.time() 211 | 212 | # calculation the period of pose reconigtion for 6 times,and get the average value 213 | if self.period_calculate_cnt <= 5 : 214 | self.period = self.period + period_endtime - period_starttime 215 | self.period_calculate_cnt = self.period_calculate_cnt + 1 216 | if self.period_calculate_cnt == 6 : 217 | self.period = self.period / 6 218 | 219 | H = output.shape[2] 220 | W = output.shape[3] 221 | 222 | # Empty list to store the detected keypoints 223 | points = [] 224 | 225 | 226 | for i in range(self.nPoints): 227 | # confidence map of corresponding body's part. 228 | probMap = output[0, i, :, :] 229 | 230 | # Find global maxima of the probMap. 231 | minVal, prob, minLoc, point = cv2.minMaxLoc(probMap) 232 | 233 | # Scale the point to fit on the original image 234 | x = (frameWidth * point[0]) / W 235 | y = (frameHeight * point[1]) / H 236 | 237 | if prob > prob_threshold: 238 | points.append((int(x), int(y))) 239 | draw_skeleton_flag = True 240 | else: 241 | points.append(None) 242 | draw_skeleton_flag = False 243 | 244 | # check the captured pose 245 | if self.is_arms_down_45(points): 246 | self.arm_down_45_cnt += 1 247 | print "%d:arm down captured"%self.frame_cnt 248 | 249 | if self.is_arms_flat(points): 250 | self.arm_flat_cnt += 1 251 | print "%d:arm up captured"%self.frame_cnt 252 | 253 | if self.is_arms_V(points): 254 | self.arm_V_cnt += 1 255 | print '%d:arm V captured'%self.frame_cnt 256 | 257 | self.frame_cnt += 1 258 | 259 | # set the frame_cnt_threshold and pose_captured_threshold according to 260 | # the period of the pose recognition 261 | cmd = '' 262 | if self.period < 0.3: 263 | frame_cnt_threshold = 5 264 | pose_captured_threshold = 4 265 | elif self.period >= 0.3 and self.period <0.6: 266 | frame_cnt_threshold = 4 267 | pose_captured_threshold = 3 268 | elif self.period >= 0.6: 269 | frame_cnt_threshold = 2 270 | pose_captured_threshold = 2 271 | 272 | # check whether pose control command are generated once for 273 | # certain times of pose recognition 274 | if self.frame_cnt == frame_cnt_threshold: 275 | if self.arm_down_45_cnt >= pose_captured_threshold: 276 | print '!!!arm up,move back!!!' 277 | cmd = 'moveback' 278 | if self.arm_flat_cnt >= pose_captured_threshold: 279 | print '!!!arm down,moveforward!!!' 280 | cmd = 'moveforward' 281 | if self.arm_V_cnt == self.frame_cnt : 282 | print '!!!arm V,land!!!' 283 | cmd = 'land' 284 | self.frame_cnt = 0 285 | self.arm_down_45_cnt = 0 286 | self.arm_flat_cnt = 0 287 | self.arm_V_cnt = 0 288 | 289 | return cmd,draw_skeleton_flag,points 290 | -------------------------------------------------------------------------------- /doc/readme.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/doc/readme.pdf -------------------------------------------------------------------------------- /doc/╦╡├≈.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/doc/╦╡├≈.pdf -------------------------------------------------------------------------------- /tello_state.py: -------------------------------------------------------------------------------- 1 | import socket 2 | from time import sleep 3 | import curses 4 | 5 | INTERVAL = 0.2 6 | 7 | 8 | 9 | def report(str): 10 | stdscr.addstr(0, 0, str) 11 | stdscr.refresh() 12 | 13 | if __name__ == "__main__": 14 | stdscr = curses.initscr() 15 | curses.noecho() 16 | curses.cbreak() 17 | 18 | local_ip = '' 19 | local_port = 8890 20 | socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # socket for sending cmd 21 | socket.bind((local_ip, local_port)) 22 | 23 | tello_ip = '192.168.10.1' 24 | tello_port = 8889 25 | tello_adderss = (tello_ip, tello_port) 26 | 27 | socket.sendto('command'.encode('utf-8'), tello_adderss) 28 | 29 | try: 30 | index = 0 31 | while True: 32 | index += 1 33 | response, ip = socket.recvfrom(1024) 34 | if response == 'ok': 35 | continue 36 | out = response.replace(';', ';\n') 37 | out = 'Tello State:\n' + out 38 | report(out) 39 | sleep(INTERVAL) 40 | except KeyboardInterrupt: 41 | curses.echo() 42 | curses.nocbreak() 43 | curses.endwin() 44 | 45 | 46 | -------------------------------------------------------------------------------- /tello_video_dll(ForWin64).zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dji-sdk/Tello-Python/693776d65b691525773adf4f320f034946fb15b2/tello_video_dll(ForWin64).zip --------------------------------------------------------------------------------