├── .gitattributes ├── .gitignore ├── Dockerfile ├── README.md └── app ├── config.json ├── labelling_engine.py ├── main.py ├── main_engine.py ├── mqtt_engine.py ├── requirements.txt ├── video_capture.py └── weights └── checker_model.pb /.gitattributes: -------------------------------------------------------------------------------- 1 | *.h5 filter=lfs diff=lfs merge=lfs -text 2 | *.pb filter=lfs diff=lfs merge=lfs -text -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ################ 2 | ## .gitignore ## 3 | ################ 4 | *.pt 5 | images/ 6 | crop_noise/ 7 | crop_num/ 8 | .vscode/ 9 | venv/ 10 | 11 | ########################### 12 | ## .gitignore for VSCode ## 13 | ########################### 14 | .vscode/* 15 | !.vscode/settings.json 16 | !.vscode/tasks.json 17 | !.vscode/launch.json 18 | !.vscode/extensions.json 19 | *.code-workspace 20 | 21 | ### VisualStudioCode Patch ### 22 | # Ignore all local history of files 23 | .history 24 | 25 | ########################### 26 | ## .gitignore for GoLang ## 27 | ########################### 28 | 29 | # Binaries for programs and plugins 30 | *.exe 31 | *.exe~ 32 | *.dll 33 | *.so 34 | *.dylib 35 | 36 | # Test binary, built with `go test -c` 37 | *.test 38 | 39 | # Output of the go coverage tool, specifically when used with LiteIDE 40 | *.out 41 | 42 | # Dependency directories (remove the comment below to include it) 43 | # vendor/ 44 | 45 | ########################### 46 | ## .gitignore for Python ## 47 | ########################### 48 | 49 | # Byte-compiled / optimized / DLL files 50 | __pycache__/ 51 | *.py[cod] 52 | *$py.class 53 | 54 | # C extensions 55 | *.so 56 | 57 | # Distribution / packaging 58 | .Python 59 | build/ 60 | develop-eggs/ 61 | dist/ 62 | downloads/ 63 | eggs/ 64 | .eggs/ 65 | lib/ 66 | lib64/ 67 | parts/ 68 | sdist/ 69 | var/ 70 | wheels/ 71 | share/python-wheels/ 72 | *.egg-info/ 73 | .installed.cfg 74 | *.egg 75 | MANIFEST 76 | 77 | # PyInstaller 78 | # Usually these files are written by a python script from a template 79 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 80 | *.manifest 81 | *.spec 82 | 83 | # Installer logs 84 | pip-log.txt 85 | pip-delete-this-directory.txt 86 | 87 | # Unit test / coverage reports 88 | htmlcov/ 89 | .tox/ 90 | .nox/ 91 | .coverage 92 | .coverage.* 93 | .cache 94 | nosetests.xml 95 | coverage.xml 96 | *.cover 97 | *.py,cover 98 | .hypothesis/ 99 | .pytest_cache/ 100 | cover/ 101 | 102 | # Translations 103 | *.mo 104 | *.pot 105 | 106 | # Django stuff: 107 | *.log 108 | local_settings.py 109 | db.sqlite3 110 | db.sqlite3-journal 111 | 112 | # Flask stuff: 113 | instance/ 114 | .webassets-cache 115 | 116 | # Scrapy stuff: 117 | .scrapy 118 | 119 | # Sphinx documentation 120 | docs/_build/ 121 | 122 | # PyBuilder 123 | .pybuilder/ 124 | target/ 125 | 126 | # Jupyter Notebook 127 | .ipynb_checkpoints 128 | 129 | # IPython 130 | profile_default/ 131 | ipython_config.py 132 | 133 | # pyenv 134 | # For a library or package, you might want to ignore these files since the code is 135 | # intended to run in multiple environments; otherwise, check them in: 136 | # .python-version 137 | 138 | # pipenv 139 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 140 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 141 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 142 | # install all needed dependencies. 143 | #Pipfile.lock 144 | 145 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 146 | __pypackages__/ 147 | 148 | # Celery stuff 149 | celerybeat-schedule 150 | celerybeat.pid 151 | 152 | # SageMath parsed files 153 | *.sage.py 154 | 155 | # Environments 156 | .env 157 | .venv 158 | env/ 159 | venv/ 160 | ENV/ 161 | env.bak/ 162 | venv.bak/ 163 | 164 | # Spyder project settings 165 | .spyderproject 166 | .spyproject 167 | 168 | # Rope project settings 169 | .ropeproject 170 | 171 | # mkdocs documentation 172 | /site 173 | 174 | # mypy 175 | .mypy_cache/ 176 | .dmypy.json 177 | dmypy.json 178 | 179 | # Pyre type checker 180 | .pyre/ 181 | 182 | # pytype static type analyzer 183 | .pytype/ 184 | 185 | # Cython debug symbols 186 | cython_debug/ 187 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM freckie/rpi-opencv-pytorch-python:3.8-buster 2 | 3 | WORKDIR /app 4 | RUN git clone https://github.com/icns-distributed-cloud/room-number-recognition 5 | 6 | WORKDIR /app/room-number-recognition/app 7 | RUN pip3 install -r requirements.txt 8 | RUN pip3 uninstall pillow && pip3 install pillow 9 | 10 | RUN python3 main.py --config config.json -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Room-Number-Recognition 2 | Room number recognition service for *indoor self-driving car*s. This service analyzes all contours in every single frame, finds the *Room Number Plate*, and extracts the number from the plate. 3 | 4 | ## Branches 5 | - `master`: stable version for release (with tag named like v1.7.4) 6 | - `develop`: branch for development 7 | - `hotfix`: fix bugs occurred at a stable version 8 | 9 | ## Training 10 | ### Checker Model 11 | . 12 | 13 | ### SVHN Model 14 | SVHN Model is a CNN model based on [YOLOv5-small](https://github.com/ultralytics/yolov5) model, which is trained with [SVHN Dataset](http://ufldl.stanford.edu/housenumbers/)(format 2). 15 | For more information, visit this repository: [icns-distributed-cloud/YOLOv5-SVHN](https://github.com/icns-distributed-cloud/YOLOv5-SVHN) 16 | 17 | ## Project Structure 18 | ``` 19 | / 20 | ├── docs/ 21 | ├── app/ 22 | └── README.md 23 | ``` 24 | - `docs`: documentation files 25 | - `app`: main service 26 | 27 | ## Environment 28 | - Training 29 | - CentOS 7.5 30 | - Nvidia GTX 1080 Ti * 8ea 31 | - Python 3.8 32 | - PyTorch 1.7.0 33 | 34 | - Service 35 | - Raspberry Pi 4 Model B 36 | - Raspbian Buster 37 | - Python 3.8 38 | - OpenCV 4.4.0 39 | 40 | ## Quickstart 41 | ``` 42 | $ git clone https://github.com/icns-distributed-cloud/Room-Number-Recognition 43 | $ cd Room-Number-Recognition/app 44 | $ pip3 install -r requirements.txt 45 | $ python3 main.py --config=config.json 46 | ``` -------------------------------------------------------------------------------- /app/config.json: -------------------------------------------------------------------------------- 1 | { 2 | "main_engine": { 3 | "device_number": 0, 4 | "window_horizontal_size": 640, 5 | "window_vertical_size": 480, 6 | "padding_size": 5, 7 | "noise_counter_threshold": 90, 8 | "fps_queue_capacity": 20, 9 | "show_on_gui": true 10 | }, 11 | "labelling_engine": { 12 | "model1": { 13 | "path": "weights/checker_model.pb", 14 | "input_layer": "input_layer_1", 15 | "output_layers": ["output_layer_1/Softmax"] 16 | }, 17 | "model2": { 18 | "repository": "icns-distributed-cloud/yolov5-svhn", 19 | "function": "svhn" 20 | }, 21 | "output_queue_capacity": 20, 22 | "flag_for_save_img": false, 23 | "path_for_noise": "./crop_noise", 24 | "path_for_num": "./crop_num" 25 | }, 26 | "mqtt_engine": { 27 | "broker_ip": "13.209.49.61", 28 | "broker_port": 1883, 29 | "pub_topic": "/room" 30 | } 31 | } -------------------------------------------------------------------------------- /app/labelling_engine.py: -------------------------------------------------------------------------------- 1 | import os 2 | import glob 3 | import logging 4 | import datetime 5 | from queue import Full, Empty 6 | from collections import Counter 7 | from multiprocessing import Process, Queue 8 | 9 | import torch 10 | import numpy as np 11 | import cv2 12 | 13 | class CheckerModel: 14 | ''' 15 | CheckeModel is a wrapper for Checker Model, which is made with tensorflow. 16 | @param cfg: cfg['labelling_engine']['model1'] 17 | ''' 18 | def __init__(self, cfg): 19 | self.model = cv2.dnn.readNetFromTensorflow(cfg['path']) 20 | self.input_layer = cfg['input_layer'] 21 | self.output_layers = cfg['output_layers'] 22 | 23 | def predict(self, img): 24 | ''' 25 | Returns true if the img contains image. 26 | @param img: image 27 | @return: result 28 | ''' 29 | gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY) 30 | gray = np.reshape(gray, (48, 48, 1)) 31 | blob = cv2.dnn.blobFromImage(gray, 1./255, (48, 48), (0, 0, 0), False, False) 32 | self.model.setInput(blob, self.input_layer) 33 | output = self.model.forward() 34 | is_target = np.argmax(output[0]) 35 | return True if is_target == 0 else False 36 | 37 | class SVHNModel: 38 | ''' 39 | CheckeModel is a wrapper for SVHN Model, which is made with YOLO darknet. 40 | @param cfg: cfg['labelling_engine']['model2'] 41 | ''' 42 | def __init__(self, cfg): 43 | self.repo = cfg['repository'] 44 | self.func = cfg['function'] 45 | self.model = torch.hub.load(self.repo, self.func).fuse().eval() 46 | self.model = self.model.autoshape() 47 | 48 | def predict(self, img): 49 | ''' 50 | Returns label. 51 | @param img: rgb image 52 | @return: label 53 | ''' 54 | _img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) 55 | with torch.no_grad(): 56 | prediction = self.model(_img, size=640) 57 | if prediction[0] is None: 58 | return None 59 | 60 | boxes = list() 61 | for pred in prediction: 62 | for x1, y1, x2, y2, conf, clas in pred: # xyxy, confidence, class 63 | boxes.append({ 64 | 'x1': x1, 65 | 'y1': y1, 66 | 'x2': x2, 67 | 'y2': y2, 68 | 'conf': conf, 69 | 'class': int(clas) 70 | }) 71 | 72 | boxes = sorted(boxes, key=lambda x: x['x1']) 73 | if len(boxes) <= 2: 74 | return None 75 | label = self.make_label(boxes) 76 | return label 77 | 78 | def make_label(self, boxes): 79 | ''' 80 | Convert list of boxes to label string. 81 | @param boxes: list of boxes 82 | @return: label string 83 | ''' 84 | chars = [] 85 | for it in [str(box['class']) for box in boxes]: 86 | if it == '10': 87 | chars.append('0') 88 | else: 89 | chars.append(it) 90 | 91 | if len(chars) == 3: 92 | return ''.join(chars) 93 | elif len(chars) >= 4: 94 | return ''.join(chars[:3]) + '-' + chars[3] 95 | else: 96 | return '' 97 | 98 | 99 | class LabellingEngine: 100 | ''' 101 | LabellingEngine is a class for managing models, queues and subprocess. 102 | @param cfg: cfg['labelling_engine'] 103 | ''' 104 | def __init__(self, cfg): 105 | # Logging 106 | self.init_logger() 107 | 108 | # Variables 109 | self.model1_cfg = cfg['model1'] 110 | self.model2_cfg = cfg['model2'] 111 | self.flag_for_save_img = cfg['flag_for_save_img'] 112 | self.path_for_noise = cfg['path_for_noise'] 113 | self.path_for_num = cfg['path_for_num'] 114 | self.output_queue = [] 115 | self.output_queue_cap = cfg['output_queue_capacity'] 116 | self.most_frequent_label = '' 117 | 118 | # Init models 119 | self.model1 = CheckerModel(self.model1_cfg) 120 | self.model2 = SVHNModel(self.model2_cfg) 121 | self.logger.info('Imported all models successfully.') 122 | self.idx = 0 123 | 124 | # Path 125 | if self.flag_for_save_img: 126 | if not os.path.exists(self.path_for_num): 127 | os.makedirs(self.path_for_num) 128 | else: 129 | files = glob.glob(self.path_for_num + '/*') 130 | for f in files: 131 | os.remove(f) 132 | 133 | if not os.path.exists(self.path_for_noise): 134 | os.makedirs(self.path_for_noise) 135 | else: 136 | files = glob.glob(self.path_for_noise + '/*') 137 | for f in files: 138 | os.remove(f) 139 | 140 | def init_logger(self): 141 | ''' 142 | Initiate a logger for LabellingEngine. 143 | ''' 144 | logger = logging.getLogger("Main.LabellingEngine") 145 | logger.setLevel(logging.INFO) 146 | self.logger = logger 147 | 148 | def predict(self, img, bigimg): 149 | ''' 150 | Predict the image. 151 | @param img: 48*48 rgb image 152 | @param bigimg: 153 | @return: string of the doorplate number. 154 | 'Noise' when it is noise, 155 | 'NaN' when the SVHN model failed to predict. 156 | @return: the most frequent label 157 | @return: flag that the image contains numbers 158 | ''' 159 | is_noise = self.model1.predict(img) 160 | 161 | if self.flag_for_save_img: 162 | path = self.path_for_noise if is_noise else self.path_for_num 163 | path = (path + '/{}.png').format(self.idx) 164 | save_img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR) 165 | cv2.imwrite(path, save_img) 166 | self.idx += 1 167 | 168 | if is_noise: 169 | return 'Noise', self.most_frequent_label, False 170 | 171 | now = self.model2.predict(bigimg) 172 | if now is None: 173 | return 'NaN', self.most_frequent_label, False 174 | 175 | self.most_frequent_label = self.get_most_frequent_label(now) 176 | return now, self.most_frequent_label, True 177 | 178 | def get_most_frequent_label(self, now=None): 179 | ''' 180 | Get the most frequent predicted label from output queue. 181 | @param now: new label 182 | @return: the most frequent label string 183 | ''' 184 | func = lambda q: Counter(q).most_common(1)[0][0] 185 | if now is None: 186 | if len(self.output_queue) == 0: 187 | return '' 188 | return func(self.output_queue) 189 | 190 | self.output_queue.append(now) 191 | if len(self.output_queue) > self.output_queue_cap: 192 | self.output_queue = self.output_queue[1:] 193 | return func(self.output_queue) 194 | 195 | def clear_most_frequent_label(self): 196 | ''' 197 | Clear the most frequent label variable and the output queue. 198 | ''' 199 | self.most_frequent_label = '' 200 | self.output_queue.clear() 201 | 202 | def close(self): 203 | ''' 204 | Close all models. 205 | ''' 206 | pass 207 | -------------------------------------------------------------------------------- /app/main.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | import argparse 4 | 5 | from main_engine import MainEngine 6 | 7 | def init_logger(): 8 | ''' 9 | Initiate main logger. 10 | ''' 11 | _logger = logging.getLogger('Main') 12 | _logger.setLevel(logging.INFO) 13 | stream_handler = logging.StreamHandler() 14 | formatter = logging.Formatter(fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s") 15 | stream_handler.setLevel(logging.INFO) 16 | stream_handler.setFormatter(formatter) 17 | _logger.addHandler(stream_handler) 18 | return _logger 19 | 20 | def load_config(filename): 21 | ''' 22 | Load and parse config file. 23 | @params filename: config file path 24 | ''' 25 | cfg: dict 26 | with open(filename, 'r') as f: 27 | cfg = json.load(f) 28 | return cfg 29 | 30 | if __name__ == "__main__": 31 | # argparse 32 | parser = argparse.ArgumentParser() 33 | parser.add_argument('--config', type=str, required=True, \ 34 | default='config.json', help='config file') 35 | args = parser.parse_args() 36 | 37 | # Load config 38 | cfg = load_config(args.config) 39 | 40 | # Logging 41 | logger = init_logger() 42 | 43 | # main(logger, args) 44 | me = MainEngine(cfg) 45 | me.run() -------------------------------------------------------------------------------- /app/main_engine.py: -------------------------------------------------------------------------------- 1 | import time 2 | import logging 3 | 4 | import cv2 5 | from labelling_engine import LabellingEngine 6 | from mqtt_engine import MQTTEngine 7 | from video_capture import BufferlessVideoCapture 8 | 9 | class MainEngine: 10 | ''' 11 | MainEngine is a class for managing core functions. 12 | @param cfg: cfg 13 | ''' 14 | def __init__(self, cfg): 15 | # Logging 16 | self.init_logger() 17 | 18 | # Variables 19 | self.device_number = cfg['main_engine']['device_number'] 20 | self.padding_size = cfg['main_engine']['padding_size'] 21 | self.window_horizontal_size = cfg['main_engine']['window_horizontal_size'] 22 | self.window_vertical_size = cfg['main_engine']['window_vertical_size'] 23 | self.fps_queue = [] 24 | self.fps_queue_cap = cfg['main_engine']['fps_queue_capacity'] 25 | self.most_frequent_label = '' 26 | self.noise_counter = 0 27 | self.noise_counter_threshold = cfg['main_engine']['noise_counter_threshold'] 28 | self.show_on_gui = cfg['main_engine']['show_on_gui'] 29 | 30 | # Labelling Engine 31 | self.le = LabellingEngine(cfg['labelling_engine']) 32 | # self.se = SerialEngine('/dev/ttyS0', 9600) 33 | self.mqtt = MQTTEngine(cfg['mqtt_engine']) 34 | self.mqtt.connect() 35 | 36 | def init_logger(self): 37 | ''' 38 | Initiate a logger for MainEngine. 39 | ''' 40 | logger = logging.getLogger("Main.MainEngine") 41 | logger.setLevel(logging.INFO) 42 | self.logger = logger 43 | 44 | def crop(self, image, x, y, w, h, padding): 45 | ''' 46 | Return cropped image with padding. 47 | @param image: image 2D array 48 | @param x: coordinate for x-axis 49 | @param y: coordinate for y-axis 50 | @param w: width 51 | @param h: height 52 | @param padding: padding size 53 | @return: cropped image array 54 | ''' 55 | copied = image.copy() 56 | if x > padding: 57 | x -= padding 58 | if y > padding: 59 | y -= padding 60 | if (w + padding * 2) < self.window_horizontal_size: 61 | w += padding * 2 62 | else: 63 | w = self.window_horizontal_size 64 | if (h + padding * 2) < self.window_vertical_size: 65 | h += padding * 2 66 | else: 67 | h = self.window_vertical_size 68 | 69 | return copied[y:(y + h), x:(x + w)] 70 | 71 | def draw_bbox(self, frame, prev_time): 72 | ''' 73 | draw_bbox function analyzes contours in this frame, 74 | detects doortag image, gets cropped image and send 75 | it to LabellingEngine. 76 | @param frame: one frame from VideoCapture (BGR image) 77 | @param prev_time: time when the previous draw_bbox job finished 78 | @return: current time 79 | @return: True if the frame contains number 80 | ''' 81 | original_img = frame 82 | canny = cv2.Canny(cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY), 50, 150) 83 | 84 | # Get all contour points 85 | try: 86 | _, coutours, _ = cv2.findContours(canny, cv2.RETR_TREE, cv2.CHAIN_APPROX_TC89_L1) 87 | except: 88 | coutours, _ = cv2.findContours(canny, cv2.RETR_TREE, cv2.CHAIN_APPROX_TC89_L1) 89 | 90 | # Analyze each contour 91 | found_number = False 92 | for contour in coutours: 93 | # Get rectangle bounding contour 94 | [x, y, w, h] = cv2.boundingRect(contour) 95 | 96 | if self.filter_noise(x, y, w, h): 97 | continue 98 | 99 | # Send cropped RGB image to Labelling Engine 100 | cropped = self.crop(original_img, x, y, w, h, self.padding_size) 101 | cropped = cv2.cvtColor(cropped, cv2.COLOR_BGR2RGB) 102 | cropped = cv2.resize(cropped, (48, 48), interpolation=cv2.INTER_LINEAR) 103 | bigcrop = self.crop(original_img, x, y, w, h, 100) 104 | bigcrop = cv2.cvtColor(bigcrop, cv2.COLOR_BGR2RGB) 105 | label, self.most_frequent_label, ok = self.le.predict(cropped, bigcrop) 106 | label_string = label 107 | cv2.putText(original_img, label_string, (x, y - 8), cv2.FONT_HERSHEY_COMPLEX_SMALL, \ 108 | 0.9, (0, 0, 255), 1) 109 | 110 | # Draw rectangle bbox on original image 111 | cv2.rectangle(original_img, (x, y), (x + w, y + h), (0, 0, 255), 2) 112 | 113 | if not ok: 114 | continue 115 | 116 | found_number = True 117 | print('label: {} / most: {}'.format(label, self.most_frequent_label)) 118 | 119 | # Calculate FPS and show FPS string on frame 120 | current_time = time.time() 121 | sec = current_time - prev_time 122 | fps = self.calc_fps(sec) 123 | fps_string = 'FPS : {}'.format(fps) 124 | cv2.putText(original_img, fps_string, (5, 20), cv2.FONT_HERSHEY_COMPLEX_SMALL, \ 125 | 0.8, (0, 255, 0), 1) 126 | 127 | # Show the most frequent label 128 | freq_string = 'Most Frequent Label : {}'.format(self.most_frequent_label) 129 | cv2.putText(original_img, freq_string, (5, 40), cv2.FONT_HERSHEY_COMPLEX_SMALL, \ 130 | 0.8, (0, 255, 0), 1) 131 | 132 | if self.show_on_gui: 133 | cv2.imshow('Room Number Recognition', original_img) 134 | 135 | return current_time, found_number 136 | 137 | def clear_most_frequent_label(self): 138 | ''' 139 | Clear noise counter and the most frequent label. 140 | ''' 141 | self.noise_counter = 0 142 | self.most_frequent_label = '' 143 | self.le.clear_most_frequent_label() 144 | 145 | def filter_noise(self, x: int, y: int, w: int, h: int): 146 | ''' 147 | Returns true if the contour is noise. 148 | @param x, y, w, h: size of coutour 149 | ''' 150 | winH, winW = self.window_horizontal_size, self.window_vertical_size 151 | ratio = float(h) / float(w) 152 | 153 | if w > 70 or h > 40 or w < 15: 154 | return True 155 | # if y > 150 or x < 200 or x > 500: 156 | # return True 157 | if float(h) < float(winH) * 0.03: 158 | return True 159 | if float(w) < float(winW) * 0.05: 160 | return True 161 | if ratio < 0.45 or ratio > 0.55: 162 | return True 163 | 164 | return False 165 | 166 | def calc_fps(self, now_fps=None): 167 | ''' 168 | Returns average FPS. 169 | @param now_fps: elapsed time 170 | @return: average FPS 171 | ''' 172 | func = lambda q: round(len(q) / sum(q), 1) 173 | if now_fps is None: 174 | if len(self.fps_queue) == 0: 175 | return 1 176 | return func(self.fps_queue) 177 | 178 | self.fps_queue.append(now_fps) 179 | if len(self.fps_queue) > self.fps_queue_cap: 180 | self.fps_queue = self.fps_queue[1:] 181 | return func(self.fps_queue) 182 | 183 | def run(self): 184 | ''' 185 | Main loop for capturing video and draw bbox. 186 | ''' 187 | # Start and wait for predicting subprocess 188 | self.logger.info('now accessing to camera device..') 189 | 190 | # cv2 window 191 | if self.show_on_gui: 192 | cv2.namedWindow('Room Number Recognition', cv2.WINDOW_NORMAL) 193 | cv2.resizeWindow('Room Number Recognition', self.window_horizontal_size, self.window_vertical_size) 194 | 195 | # Main loop 196 | # cap = cv2.VideoCapture(self.device_number) 197 | cap = BufferlessVideoCapture(self.device_number) 198 | prev_time = 1 199 | while cap.isOpened(): 200 | try: 201 | # Get frame 202 | ret, inp = cap.read() 203 | if ret: 204 | prev_time, found_number = self.draw_bbox(inp, prev_time) 205 | if not found_number: 206 | self.noise_counter += 1 207 | if self.noise_counter > self.noise_counter_threshold: 208 | self.clear_most_frequent_label() 209 | else: 210 | self.mqtt.publish({ 211 | 'label': str(self.most_frequent_label) 212 | }) 213 | 214 | # Wait for quit signal 215 | if cv2.waitKey(1) & 0xFF == ord('q'): 216 | self.logger.info('terminating process..') 217 | break 218 | except KeyboardInterrupt: 219 | self.logger.info('terminating process.. (occured by KeyboardInterrupt)') 220 | break 221 | 222 | # Release device and wait for closing the subprocess 223 | cap.release() 224 | self.logger.info('released camera.') 225 | self.le.close() 226 | self.logger.info('closed LabellingEngine.') 227 | cv2.destroyAllWindows() 228 | self.logger.info('closed all windows.') 229 | -------------------------------------------------------------------------------- /app/mqtt_engine.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | 4 | import paho.mqtt.client as mqtt 5 | 6 | class MQTTEngine: 7 | ''' 8 | MQTTEngine is a class for MQTT communication. 9 | @param cfg: cfg['mqtt_engine'] 10 | ''' 11 | def __init__(self, cfg): 12 | # Logging 13 | self.init_logger() 14 | 15 | # Variables 16 | self.broker_ip = cfg['broker_ip'] 17 | self.broker_port = cfg['broker_port'] 18 | self.pub_topic = cfg['pub_topic'] 19 | 20 | # MQTT Client 21 | self.client = mqtt.Client() 22 | 23 | def init_logger(self): 24 | ''' 25 | Initiate a logger for MainEngine. 26 | ''' 27 | logger = logging.getLogger('Main.MQTTEngine') 28 | logger.setLevel(logging.INFO) 29 | self.logger = logger 30 | 31 | def connect(self): 32 | ''' 33 | Connect to MQTT Broker. 34 | ''' 35 | self.client.connect_async(self.broker_ip, self.broker_port) 36 | self.client.on_connect = self._on_connect 37 | self.client.on_disconnect = self._on_disconnect 38 | self.client.loop_start() 39 | 40 | def close(self): 41 | ''' 42 | Close the MQTT connection. 43 | ''' 44 | self.client.loop_stop() 45 | self.client.disconnect() 46 | 47 | def publish(self, body): 48 | ''' 49 | Publish message as the specific topic. 50 | ''' 51 | self.client.publish(self.pub_topic, json.dumps(body), 1) 52 | 53 | def _on_connect(self, client, userdata, flags, rc): 54 | if rc == 0: 55 | self.logger.info('made connection with MQTT broker successfully.') 56 | else: 57 | self.logger.info('MQTT connection failed. Code =' + str(rc)) 58 | 59 | def _on_disconnect(self, client, userdata, flags, rc=0): 60 | self.logger.info('MQTT connection disconnected.') 61 | -------------------------------------------------------------------------------- /app/requirements.txt: -------------------------------------------------------------------------------- 1 | Cython 2 | matplotlib>=3.2.2 3 | numpy>=1.18.5 4 | #opencv-python>=4.1.2 5 | pillow 6 | PyYAML>=5.3 7 | scipy>=1.4.1 8 | tensorboard>=2.2 9 | #torch>=1.7.0 10 | #torchvision>=0.7.0 11 | tqdm>=4.41.0 12 | paho-mqtt>=1.5.1 -------------------------------------------------------------------------------- /app/video_capture.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import queue 3 | import threading 4 | 5 | class BufferlessVideoCapture: 6 | ''' 7 | BufferlessVideoCapture is a wrapper for cv2.VideoCapture, 8 | which doesn't have frame buffer. 9 | @param name: videocapture name 10 | ''' 11 | def __init__(self, name): 12 | self.cap = cv2.VideoCapture(name) 13 | self.q = queue.Queue() 14 | self.thr = threading.Thread(target=self._reader) 15 | self.thr.daemon = True 16 | self.thr.start() 17 | 18 | def _reader(self): 19 | ''' 20 | Main loop for thread. 21 | ''' 22 | while True: 23 | ret, frame = self.cap.read() 24 | if not ret: 25 | break 26 | if not self.q.empty(): 27 | try: 28 | self.q.get_nowait() # discard previous (unprocessed) frame 29 | except queue.Empty: 30 | pass 31 | if self.q.qsize() > 2: 32 | print(self.q.qsize()) 33 | self.q.put(frame) 34 | 35 | def isOpened(self): 36 | return self.cap.isOpened() 37 | 38 | def release(self): 39 | self.cap.release() 40 | 41 | def read(self): 42 | ''' 43 | Read current frame. 44 | ''' 45 | return True, self.q.get() 46 | 47 | def close(self): 48 | pass 49 | -------------------------------------------------------------------------------- /app/weights/checker_model.pb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/icns-distributed-cloud/Room-Number-Recognition/3a3b7b2e5d9c17e1fc068d957029a74138d75203/app/weights/checker_model.pb --------------------------------------------------------------------------------