7 |
8 | ## Executive Summary
9 |
10 | Since the pandemic is finally softening and nearing its end, many companies are returning to the office. However, we encounter some forgotten problems as we stay in the office. As an employee, overtime hours are usually not reflected in our salary because we cannot track and justify how long we have worked in a day. On the other hand, as an employer, we are not sure how long an employee works in a day, primarily when the company supports a flexible schedule. Some employee could easily cheat their time which happened to our small business before. This project aims to answer. How might we ensure fairness in tracking hours worked for employees?
11 | 12 |The solution I implemented in this project is to use deep learning techniques to detect an employee and monitor the time the employee is seated in their station. Another feature of this project is to count the number of people inside the office. To be able to track if there are some violations of social distancing.
13 | 14 |I extracted 200 sample frames from an office's closed-circuit television (CCTV) footage, then trained a custom YOLO model to detect employees with the manually labeled sampled images as a training and validation set. I set a threshold depending on the Intersect Over Union (IoU) of the employee to each station to determine if the seat is taken or vacant. To measure the time the employee is seated at the station, I performed Optical Character Recognition (OCR) to the time of the CCTV footage presented on the screen to have a more accurate result than using frames per second of the camera might vary.
15 | 16 |The application of transfer learning on YOLO increased the average precision from 69% trained on the COCO dataset to 98% in our custom dataset. This allows for robust and real-time detection when deployed as a real-time employee tracker. Thresholding for rules values depends on the camera angle and distinguishes between employees and edge cases.
17 | 18 |Recommendations for this project involve additional features outside of the current scope, like detection of the actual action of the employee if they are talking, or working, among others. Further recommendations involve alternative use cases for the system. With modifications to the implementation, this system may be redeployed for parking management, security systems, and traffic management.
19 | 20 | https://user-images.githubusercontent.com/67182415/177245750-7488612b-f820-4b33-95b0-74e9ad26776e.mov 21 | -------------------------------------------------------------------------------- /md/Individual Project Report_La Rosa.md: -------------------------------------------------------------------------------- 1 | ```python 2 | display(Image(filename="cover.png")) 3 | ``` 4 | 5 | 6 |  7 | 8 | 9 | ## Highlights 10 | 11 | - Employee monitoring to ensure fairness in tracking hours worked for employees 12 | - Use of YOLO for counting and detecting persons in a room 13 | - Fine tuning of YOLO in a custom dataset to improve the result by 30% 14 | - Use of Intersection over Union (IoU) as a threshold to determine if the station is taken or vacant 15 | - Use of image processing techniques and use of pytesseract to perform OCR to get the time of the cctv camera 16 | 17 | ## Introduction 18 | 19 |Since the pandemic is finally softening and nearing its end, many companies are returning to the office. However, we encounter some forgotten problems as we stay in the office. As an employee, overtime hours are usually not reflected in our salary because we cannot track and justify how long we have worked in a day. On the other hand, as an employer, we are not sure how long an employee works in a day, primarily when the company supports a flexible schedule. Some employee could easily cheat their time which happened to our small business before. This project aims to answer How might we ensure fairness in tracking hours worked for employees?
20 |The solution that I implemented in this project is to use deep learning techniques to detect an employee and monitor the time the employee is seated in its station. An additional feature in this project is to count the number of person inside the office. To be able to track if there are some violations on social distancing.
21 | 22 | 23 | ```python 24 | # python standard libraries 25 | import os 26 | import time 27 | import glob 28 | import re 29 | import shutil 30 | import pickle 31 | from datetime import datetime, timedelta 32 | from base64 import b64decode, b64encode 33 | 34 | # google colab/notebook libraries 35 | from IPython.display import display, Javascript, Image 36 | from google.colab.output import eval_js 37 | from google.colab.patches import cv2_imshow 38 | 39 | # external libraries 40 | import cv2 41 | import numpy as np 42 | import PIL 43 | import io 44 | import html 45 | import matplotlib.pyplot as plt 46 | from tqdm import tqdm 47 | from skimage.morphology import erosion 48 | 49 | # define color constants 50 | person_color = (220, 155, 58) 51 | vacant_color = (0, 0, 200) 52 | taken_color = (0, 200, 0) 53 | station_color = (0, 100, 0) 54 | 55 | # constant coordinates 56 | coordinates = { 57 | 'station_1' : {'x1':1600, 'x2':1800, 58 | 'y1':575, 'y2':780}, 59 | 'station_2' : {'x1':1287, 'x2':1472, 60 | 'y1':310, 'y2':425}, 61 | 'station_3' : {'x1':1145, 'x2':1287, 62 | 'y1':197, 'y2':268}, 63 | 64 | 'station_4' : {'x1':561, 'x2':764, 65 | 'y1':424, 'y2':578} 66 | } 67 | 68 | coordinates_ocr= [(1256, 39), (1885, 101)] 69 | 70 | %matplotlib inline 71 | ``` 72 | 73 | 74 | ```python 75 | from IPython.display import HTML 76 | from IPython.display import Image 77 | 78 | HTML(''' 90 | 104 | ''') 106 | ``` 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | ```python 115 | # install if working on colab 116 | !sudo apt install tesseract-ocr 117 | !pip -qq install pytesseract 118 | 119 | # import and install separately to avoid dependency issues later 120 | import pytesseract 121 | ``` 122 | 123 | 124 | ```python 125 | # clone darknet repo if you are using original repository 126 | !git clone https://github.com/AlexeyAB/darknet 127 | ``` 128 | 129 | 130 | ```python 131 | # connect to google drive 132 | from google.colab import drive 133 | drive.mount('/content/drive') 134 | ``` 135 | 136 | 137 | ```python 138 | # copy whole dataset 139 | !cp /content/drive/MyDrive/MSDS/ML3/final_project/XVR_ch5_main*.mp4 /content/ 140 | 141 | # clone of darknet github, extracted dataset, configurations, and custom weights 142 | !cp /content/drive/MyDrive/MSDS/ML3/final_project/darknet_best.zip /content/ 143 | 144 | # trained weights from custom dataset 145 | !cp /content/drive/MyDrive/MSDS/ML3/final_project/yolov4-obj_best.weights /content/ 146 | 147 | # unzip darknet directory 148 | !unzip -qq darknet_best.zip 149 | 150 | # clone original darknet repo (use if you want default settings) 151 | !git clone https://github.com/AlexeyAB/darknet 152 | 153 | # change makefile to have GPU, OPENCV and LIBSO enabled 154 | %cd /content/darknet 155 | !sed -i 's/OPENCV=0/OPENCV=1/' Makefile 156 | !sed -i 's/GPU=0/GPU=1/' Makefile 157 | !sed -i 's/CUDNN=0/CUDNN=1/' Makefile 158 | !sed -i 's/CUDNN_HALF=0/CUDNN_HALF=1/' Makefile 159 | !sed -i 's/LIBSO=0/LIBSO=1/' Makefile 160 | 161 | # make darknet (builds darknet so that you can then use the darknet.py file 162 | # and have its dependencies) 163 | !make 164 | ``` 165 | 166 |Since it is just a single camera each of the frames would have the same size and channels. For exploratory data analysis, let us check the dimesions of one frame of the CCTV feed.
169 | 170 | 171 | ```python 172 | # print sample image 173 | vidcap = cv2.VideoCapture('../XVR_ch5_main_20220214100004_20220214110005.mp4') 174 | success,frame = vidcap.read() 175 | cv2_imshow(frame) 176 | ``` 177 | 178 | 179 |  180 | 181 | 182 | The frame have dimensions of: 183 | 184 | 185 | ```python 186 | # Get dimensions of image 187 | width, height, channels = frame.shape 188 | print(f'width: {width}') 189 | print(f'height: {height}') 190 | print(f'channels: {channels}') 191 | ``` 192 | 193 | width: 1080 194 | height: 1920 195 | channels: 3 196 | 197 | 198 | ## Methodology 199 | 200 | 201 | ```python 202 | display(Image(filename="methodology.png")) 203 | ``` 204 | 205 | 206 |  207 | 208 | 209 | a.) Dataset 210 |The dataset used for this project is a personal video surveillance of our small business with total length of 1 hour in February 14.
211 | 212 | b.) Extract images 213 |Extracted 200 sample images to be used for training and validation set.
214 | 215 | c.) Fine tune YOLOv4 model 216 |Performed transfer learning using pretrained weights trained on COCO dataset and fine tune using custom dataset.
217 | 218 | d.) Perform non-max suppression 219 |Performed non-max suppression to remove multiple bounding boxes in a single object and get the maximum confidence among those bounding boxes that overlaps.
220 | 221 | e.) Set-up work station 222 |Set up work station to get and create a rule based on IoU of the detected person and the workstation to identify what are the stations that are taken and vacant.
223 | 224 | f.) Compute for the time of the employee in work station 225 |Used the information provided by the DVR which can be found on the upper right of the image and performed image processing techniques and optical character recognition to parse the image to a datetime python object.
226 | 227 | ## Results and Discussion 228 | 229 |YOLO has pre trained model on COCO dataset which could classify 80 objects. COCO dataset is a large scale object detection, segmentation, and captioning of over 330k images. One of the objects that the YOLO trained on this dataset could classify is person which is what we need for this study. Let us try to use if it would work well on our dataset.
232 | 233 | 234 | ```python 235 | # use if you want to use default settings 236 | # get bthe scaled yolov4 weights file that is pre-trained to detect 80 classes (objects) from shared google drive 237 | !wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1V3vsIaxAlGWvK4Aar9bAiK5U0QFttKwq' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1V3vsIaxAlGWvK4Aar9bAiK5U0QFttKwq" -O yolov4-csp.weights && rm -rf /tmp/cookies.txt 238 | ``` 239 | 240 | 241 | ```python 242 | # import darknet functions to perform object detections 243 | from darknet import * 244 | # load in our YOLOv4 architecture network 245 | network, class_names, class_colors = load_network("cfg/yolov4-csp.cfg", "cfg/coco.data", "yolov4-csp.weights") 246 | width = network_width(network) 247 | height = network_height(network) 248 | 249 | # darknet helper function to run detection on image 250 | def darknet_helper(img, width, height): 251 | darknet_image = make_image(width, height, 3) 252 | img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) 253 | img_resized = cv2.resize(img_rgb, (width, height), 254 | interpolation=cv2.INTER_LINEAR) 255 | 256 | # get image ratios to convert bounding boxes to proper size 257 | img_height, img_width, _ = img.shape 258 | width_ratio = img_width/width 259 | height_ratio = img_height/height 260 | 261 | # run model on darknet style image to get detections 262 | copy_image_from_bytes(darknet_image, img_resized.tobytes()) 263 | detections = detect_image(network, class_names, darknet_image) 264 | free_image(darknet_image) 265 | return detections, width_ratio, height_ratio 266 | ``` 267 | 268 | 269 | ```python 270 | # evaluate in validation set 271 | %cd /content/darknet 272 | !./darknet detector map data/obj.data cfg/yolov4-csp.cfg ../yolov4-csp.weights -points 0 273 | ``` 274 | 275 | 276 | ```python 277 | display(Image(filename="../pretrained.png")) 278 | ``` 279 | 280 | 281 |  282 | 283 | 284 |It turns out that for some of the frames it would perfectly classify the person in the image. However, there are some frames that the model misclassify the person like as we can see in the lower right of the figure. Additionally, there are sometimes multiple bounding boxes on a single object and some objects that are not needed in this project are being classified like tv monitor. Although the model is trained on thousand of images, it was not really trained on this type of environment and probably not to all angles of a person.
285 | 286 | ## ii. Train YOLO on custom dataset 287 | 288 |To remedy the issues found in using pretrained model on COCO dataset, we can perform training on custom dataset. The detailed explanation on how to train your YOLO on custom dataset can be found in their documentation.
289 | 290 |Training YOLO on custom dataset would require images, so we need to transform our video and sample it into images. 50 to 100 images are usually enough to train a single object but for this project I sampled the data into 200 images to ensure more correctness of the data.
293 | 294 | 295 | 296 | ```python 297 | # extract data from video to 700 sampled images 298 | cap = cv2.VideoCapture('XVR_ch5_main_20220214100004_20220214110005.mp4') 299 | length = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) 300 | img_array =[] 301 | counter = 0 302 | image_count = 0 303 | while ret: 304 | # Capture frame-by-frame 305 | ret, frame = cap.read() 306 | 307 | if counter % int(length/700) == 0: 308 | fname = f'{image_count}.jpg' 309 | image_count += 1 310 | cv2.imwrite(fname, frame) 311 | 312 | counter += 1 313 | if cv2.waitKey(1) & 0xFF == ord('q'): 314 | break 315 | 316 | # When everything done, release the capture 317 | cap.release() 318 | cv2.destroyAllWindows() 319 | ``` 320 | 321 |Now that we have our 200 images, we need to manually label the images in a format expected by the YOLO algorithm which is {object-class} {x_center} {y_center} {width} {height}. I used LabelImg to manually label my data into bounding boxes. It will produce text files of bounding boxes with the same filename as the image file and a text file that contains the class names. We then split the data into train and 10 percent validation set.
The model training can be performed by using the detector train command which expects at least three parameters: data, configurations, and initial weights. Initially I used yolov4.conv.137 as my pre trained weights which is trained in COCO dataset. Then, I retrained my weights to further improve the results. Overall it took me about six hours of training to get a good result.
The non max supression created by darknet prioritizes the bottom right bounding box and removes the overlap. I have updated the code of darknet to get the maximum confidence of the bounding boxes and remove the overlap. The solution could be slower than darknet implementation but it yielded into better accuracy which is more important in this project. I chose 65% threshold for the non-max suppression as it provided optimal result.
395 | 396 | 397 | ```python 398 | def non_max_suppression_fast(detections, overlap_thresh): 399 | """ modified non max suppression from darknet to get the overlap 400 | with max confidence 401 | 402 | Parameters 403 | ========== 404 | detections : tuple 405 | class_name, confidence, and coordinates 406 | overlap_thresh : float 407 | IOU threshold 408 | 409 | Returns 410 | ========== 411 | non_max_suppression_fast : tuple 412 | detections without high overlap 413 | """ 414 | boxes = [] 415 | confs = [] 416 | 417 | for detection in detections: 418 | class_name, conf, (x, y, w, h) = detection 419 | x1 = x - w / 2 420 | y1 = y - h / 2 421 | x2 = x + w / 2 422 | y2 = y + h / 2 423 | boxes.append(np.array([x1, y1, x2, y2])) 424 | confs.append(conf) 425 | 426 | boxes_array = np.array(boxes) 427 | 428 | # initialize the list of picked indexes 429 | pick = [] 430 | # grab the coordinates of the bounding boxes 431 | x1 = boxes_array[:, 0] 432 | y1 = boxes_array[:, 1] 433 | x2 = boxes_array[:, 2] 434 | y2 = boxes_array[:, 3] 435 | # compute the area of the bounding boxes and sort the bounding 436 | # boxes by the bottom-right y-coordinate of the bounding box 437 | area = (x2 - x1 + 1) * (y2 - y1 + 1) 438 | idxs = np.argsort(y2) 439 | confs = np.array(confs) 440 | # keep looping while some indexes still remain in the indexes 441 | # list 442 | 443 | while len(idxs) > 0: 444 | # grab the last index in the indexes list and add the 445 | # index value to the list of picked indexes 446 | last = len(idxs) - 1 447 | i = idxs[last] 448 | # find the largest (x, y) coordinates for the start of 449 | # the bounding box and the smallest (x, y) coordinates 450 | # for the end of the bounding box 451 | 452 | xx1 = np.maximum(x1[i], x1[idxs[:last]]) 453 | yy1 = np.maximum(y1[i], y1[idxs[:last]]) 454 | xx2 = np.minimum(x2[i], x2[idxs[:last]]) 455 | yy2 = np.minimum(y2[i], y2[idxs[:last]]) 456 | # compute the width and height of the bounding box 457 | w = np.maximum(0, xx2 - xx1 + 1) 458 | h = np.maximum(0, yy2 - yy1 + 1) 459 | # compute the ratio of overlap 460 | overlap = (w * h) / area[idxs[:last]] 461 | 462 | # choose the highest confidence among overlaps 463 | overlap_args = np.where(overlap > overlap_thresh)[0] 464 | overlap_indices = idxs[overlap_args].tolist() + [i] 465 | confidence_list = confs[idxs[overlap_args]].tolist() + [confs[i]] 466 | confidence_list = list(map(float, confidence_list)) 467 | highest_confidence = np.argmax(confidence_list) 468 | pick.append(overlap_indices[highest_confidence]) 469 | 470 | # delete indices that overlaps 471 | idxs = np.delete(idxs, np.concatenate(([last], overlap_args))) 472 | 473 | return [detections[i] for i in pick] 474 | ``` 475 | 476 |Now that we have trained our custom YOLO model and created a custom non-max suppression, we can now try if the model would work on a test set. The average precision increased from 69% trained on coco dataset to 98% in our custom dataset. You can evaluate using mean average precision of the object. It can be calculated by adjusting the threshold of confidence and get the average precision score on 50% IoU score. It can be calculated by using detector map command of YOLOv4 repository.
We now get the coordinates of the four work stations. I used paint to manually get the coordinates of each of the stations and plotted inte figure below.
573 | 574 | 575 | ```python 576 | # run custom yolo on a sample image 577 | vidcap = cv2.VideoCapture('/content/XVR_ch5_main_20220214100004_20220214110005.mp4') 578 | success,frame = vidcap.read() 579 | 580 | 581 | for stations, coordinate in coordinates.items(): 582 | cv2.rectangle(frame, (coordinate['x1'], coordinate['y1']), 583 | (coordinate['x2'], coordinate['y2']), station_color, 2) 584 | cv2.putText(frame, f"{stations}", 585 | (coordinate['x1'], coordinate['y1'] - 5), 586 | cv2.FONT_HERSHEY_SIMPLEX, 0.5, 587 | station_color, 2) 588 | 589 | cv2_imshow(frame) 590 | ``` 591 | 592 | 593 |  594 | 595 | 596 |To integrate work stations in our custom network, we need to set up a rule to determine if the work station is taken or vacant. I used the Intersect Over Union (IoU) of 0.3 as a threshold if they overlap with the person to determine if the station is taken.
599 | 600 | 601 | ```python 602 | def get_iou(bb1, bb2): 603 | """ 604 | Calculate the Intersection over Union (IoU) of two bounding boxes. 605 | 606 | Parameters 607 | ---------- 608 | bb1 : dict 609 | Keys: {'x1', 'x2', 'y1', 'y2'} 610 | The (x1, y1) position is at the top left corner, 611 | the (x2, y2) position is at the bottom right corner 612 | bb2 : dict 613 | Keys: {'x1', 'x2', 'y1', 'y2'} 614 | The (x, y) position is at the top left corner, 615 | the (x2, y2) position is at the bottom right corner 616 | 617 | Returns 618 | ------- 619 | float 620 | in [0, 1] 621 | """ 622 | # determine the coordinates of the intersection rectangle 623 | x_left = max(bb1['x1'], bb2['x1']) 624 | y_top = max(bb1['y1'], bb2['y1']) 625 | x_right = min(bb1['x2'], bb2['x2']) 626 | y_bottom = min(bb1['y2'], bb2['y2']) 627 | 628 | if x_right < x_left or y_bottom < y_top: 629 | return 0.0 630 | 631 | # The intersection of two axis-aligned bounding boxes is always an 632 | # axis-aligned bounding box 633 | intersection_area = (x_right - x_left) * (y_bottom - y_top) 634 | 635 | # compute the area of both AABBs 636 | bb1_area = (bb1['x2'] - bb1['x1']) * (bb1['y2'] - bb1['y1']) 637 | bb2_area = (bb2['x2'] - bb2['x1']) * (bb2['y2'] - bb2['y1']) 638 | # compute the intersection over union by taking the intersection 639 | # area and dividing it by the sum of prediction + ground-truth 640 | # areas - the interesection area 641 | iou = intersection_area / float(bb1_area + bb2_area - intersection_area) 642 | 643 | return iou 644 | ``` 645 | 646 | 647 | ```python 648 | # run custom yolo on a sample image 649 | vidcap = cv2.VideoCapture('/content/XVR_ch5_main_20220214100004_20220214110005.mp4') 650 | success,frame = vidcap.read() 651 | 652 | # get the predicted detections of the trained custom yolo 653 | detections, width_ratio, height_ratio = darknet_helper(frame, width, height) 654 | 655 | # apply non max suppression to eliminate multiple predictions 656 | # on same person 657 | detections = non_max_suppression_fast(detections, 0.65) 658 | detections_bb = [] 659 | for label, confidence, bbox in detections: 660 | left, top, right, bottom = bbox2points(bbox) 661 | left, top, right, bottom = (int(left * width_ratio), 662 | int(top * height_ratio), 663 | int(right * width_ratio), 664 | int(bottom * height_ratio)) 665 | 666 | cv2.rectangle(frame, (left, top), (right, bottom), person_color, 2) 667 | cv2.putText(frame, "{} [{:.2f}]".format(label, float(confidence)), 668 | (left, top - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, 669 | person_color, 2) 670 | 671 | detections_bb.append({ 672 | 'x1' : left, 673 | 'y1' : top, 674 | 'x2' : right, 675 | 'y2' : bottom 676 | }) 677 | 678 | thresh = 0.3 679 | for stations, coordinate in coordinates.items(): 680 | taken = False 681 | for detection in detections_bb: 682 | iou = get_iou(coordinate, detection) 683 | if iou >= thresh: 684 | taken = True 685 | break 686 | color = taken_color if taken else vacant_color 687 | 688 | cv2.rectangle(frame, (coordinate['x1'], coordinate['y1']), 689 | (coordinate['x2'], coordinate['y2']), color, 2) 690 | 691 | cv2.putText(frame, f"{stations}", 692 | (coordinate['x1'], coordinate['y1'] - 5), 693 | cv2.FONT_HERSHEY_SIMPLEX, 0.5, 694 | color, 2) 695 | frame = cv2.resize(frame, (1080, 720), 696 | interpolation=cv2.INTER_AREA) 697 | 698 | cv2_imshow(frame) 699 | ``` 700 | 701 | 702 |  703 | 704 | 705 |To get the time the employee stays in its work station, we can perform it in two ways: We can use the information of the image presented by the DVR which can be seen in the upper right of the image, or used the information of the camera by knowing the frames per second of the camera. In this project I used both method but the former is more appropriate since based on my experience, CCTV cameras are often changed compared to the DVR. Cameras could have multiple frames per second, and using that information could yield to incorrect classification of time. Hence, extracting the information produced by the DVR would have longer usability of the model that we would create. Here is the sample of the original image:
708 | 709 | 710 | ```python 711 | if not imgs: 712 | vidcap = cv2.VideoCapture('../XVR_ch5_main_20220214100004_20220214110005.mp4') 713 | success,imgs = vidcap.read() 714 | 715 | cv2_imshow(imgs[1]) 716 | ``` 717 | 718 | 719 |  720 | 721 | 722 | ### a. Datetime information using OCR 723 | 724 |First, I performed image preprocessing in order for the OCR model to understand the text in the image more accurately. The image processing techniques performed are extract the image to get the area with the date time only, convert the image into BGR to Gray, adjust the contrast and brightness to make the background brighter before thresholding, perform adaptive thresholding to remove the background, and lasty use morphological operations such as multiple erosion to make the text bolder.
725 | 726 | 727 | ```python 728 | def multi_ero(im, num): 729 | """ Perform multiple erosion on the image 730 | 731 | Parameters 732 | ========== 733 | im : np.array 734 | image file 735 | num : int 736 | number of times to apply erosion 737 | """ 738 | for i in range(num): 739 | im = erosion(im) 740 | return im 741 | 742 | imgs = [] 743 | # get images for testing 744 | vidcap = cv2.VideoCapture('../XVR_ch5_main_20220214100004_20220214110005.mp4') 745 | success,frame = vidcap.read() 746 | for i in tqdm(range(8000)): 747 | # Capture frame-by-frame 748 | success, frame = vidcap.read() 749 | if not i % 50: 750 | if frame is not None: 751 | imgs.append(frame) 752 | else: 753 | pass 754 | invalid = [] 755 | valid = [] 756 | datetime_clean = [] 757 | 758 | 759 | for img in tqdm(imgs): 760 | img = cv2.cvtColor(img.copy(), cv2.COLOR_BGR2GRAY) 761 | 762 | contrast = 3 763 | contrast = max(contrast, 1.0); contrast = min(contrast, 3.0) 764 | 765 | brightness = 60 766 | brightness = max(brightness, 0.0); brightness = min(brightness, 100.0) 767 | 768 | img = np.clip(contrast * img.astype('float32') 769 | + brightness, 0.0, 255.0) 770 | 771 | img = img.astype('uint8') 772 | 773 | img = cv2.adaptiveThreshold(img, 774 | 255, 775 | cv2.ADAPTIVE_THRESH_GAUSSIAN_C, 776 | cv2.THRESH_BINARY, 777 | 21, 2) 778 | 779 | img = img[coordinates_ocr[0][1]:coordinates_ocr[1][1], 780 | coordinates_ocr[0][0]:coordinates_ocr[1][0]] 781 | 782 | img = multi_ero(img, 2) 783 | datetime_clean.append(img) 784 | text = pytesseract.image_to_string(img, lang='eng', 785 | config='--psm 10 --oem 3 -c tessedit_char_whitelist=0123456789:-') 786 | 787 | time_format = r'[0-5]\d:[0-5]\d:[0-5]\d' 788 | date_format = r'\d{4}-(?:0\d|1[12])-(?:[0-2]\d|3[01])' 789 | datetime_format = date_format + time_format 790 | text = text.replace(' ', '') 791 | try: 792 | timestamp_string = re.sub('(\d{4}-(?:0\d|1[12])-(?:[0-2]\d|3[01]))', 793 | r'\1' + r' ', 794 | re.findall(datetime_format, text)[0]) 795 | except: 796 | invalid.append(text) 797 | continue 798 | 799 | 800 | if len(text) != 20: 801 | invalid.append(text) 802 | 803 | else: 804 | valid.append(text) 805 | ``` 806 | 807 | 808 | ```python 809 | cv2_imshow(datetime_clean[1]) 810 | ``` 811 | 812 | 813 |  814 | 815 | 816 |I used Pytesseract to read the text in the image and convert it to datetime object of python. Pytesseract is an optical character recognition (OCR) tool for python made by Google that uses Deep Learning and in particular LSTM to predict on the text of the image. I used the following configurations:psm=10 so that it it will classify per character, and tessedit_char_whitelist=0123456789:- so that the model would be forced to classify between these whitelist characters which are expected in our date and time element
I checked the format and found out that 70% of the results are correctly classified in a correct datetime format by the model. In particular, it experienced difficulties in classifying number eight when it is near six. However there are about 15 frames per second so there will be a lot of chance for the model to get the correct time. Here is a sample of the output of the pytesseract that is passed to be converted to Python datetime.
898 | 899 | 900 | ```python 901 | get_ocr_datetime(imgs[1]) 902 | ``` 903 | 904 | 905 | datetime.datetime(2022, 2, 14, 10, 0, 7) 906 | 907 | 908 | ### b. Datetime information using fps 909 | 910 |The camera that I am using is a 15 fps which is one of the standard of a CCTV. Using that information, we count the number of frames an employee is sitting on its station then for every 15 frames we count that as 1 second.
911 | 912 |Save frames as video as mp4 and compress it in order to be compatible with Google Colab. The demo video will be submitted separately so it will not blow up the size of the notebook.
1115 | 1116 | 1117 | ```python 1118 | cap = cv2.VideoCapture('resized_cctv_full.mp4') 1119 | 1120 | img_array =[] 1121 | success = True 1122 | while success: 1123 | success,frame = cap.read() 1124 | # Capture frame-by-frame 1125 | img_array.append(frame) 1126 | ``` 1127 | 1128 | 1129 | ```python 1130 | %cd /content/ 1131 | fname = 'resized_cctv.mp4' 1132 | if not resize: 1133 | width = 1920 1134 | height = 1080 1135 | 1136 | if any([True if fname in f else False for f in os.listdir()]): 1137 | !rm resized_cctv.mp4 1138 | 1139 | out = cv2.VideoWriter('/content/resized_cctv.mp4', 1140 | cv2.VideoWriter_fourcc(*'MP4V'), 1141 | 20, (1600, 900)) 1142 | 1143 | for i in tqdm(range(len(img_array))): 1144 | out.write(img_array[i]) 1145 | out.release() 1146 | ``` 1147 | 1148 | 1149 | ```python 1150 | %cd darknet 1151 | 1152 | from IPython.display import HTML 1153 | from base64 import b64encode 1154 | import os 1155 | 1156 | # Input video path 1157 | save_path = "/content/resized_cctv_full.mp4" 1158 | 1159 | # Compressed video path 1160 | compressed_path = "/content/resized_cctv_compressed.mp4" 1161 | 1162 | os.system(f"ffmpeg -i {save_path} -vcodec libx264 {compressed_path}") 1163 | 1164 | # Show video 1165 | mp4 = open(compressed_path,'rb').read() 1166 | data_url = "data:video/mp4;base64," + b64encode(mp4).decode() 1167 | HTML(""" 1168 | 1171 | """ % data_url) 1172 | 1173 | ``` 1174 | 1175 | 1176 | ```python 1177 | !cp /content/resized_cctv.mp4 /content/drive/MyDrive/MSDS/ML3/final_project/resized_cctv_full_fast.mp4 1178 | ``` 1179 | -------------------------------------------------------------------------------- /md/output_0_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/md/output_0_0.png -------------------------------------------------------------------------------- /md/output_13_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/md/output_13_0.png -------------------------------------------------------------------------------- /md/output_17_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/md/output_17_0.png -------------------------------------------------------------------------------- /md/output_25_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/md/output_25_0.png -------------------------------------------------------------------------------- /md/output_47_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/md/output_47_0.png -------------------------------------------------------------------------------- /md/output_51_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/md/output_51_0.png -------------------------------------------------------------------------------- /md/output_55_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/md/output_55_0.png -------------------------------------------------------------------------------- /md/output_58_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/md/output_58_0.png -------------------------------------------------------------------------------- /md/output_62_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/md/output_62_0.png -------------------------------------------------------------------------------- /outputs/cleaned_timer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/outputs/cleaned_timer.png -------------------------------------------------------------------------------- /outputs/custom_yolo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/outputs/custom_yolo.png -------------------------------------------------------------------------------- /outputs/header.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/outputs/header.png -------------------------------------------------------------------------------- /outputs/iou_threshold.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/outputs/iou_threshold.png -------------------------------------------------------------------------------- /outputs/methodology.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/outputs/methodology.png -------------------------------------------------------------------------------- /outputs/pre_trained_yolo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/outputs/pre_trained_yolo.png -------------------------------------------------------------------------------- /outputs/resized_cctv_full_fast.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/outputs/resized_cctv_full_fast.mp4 -------------------------------------------------------------------------------- /outputs/resized_cctv_full_fast_cut.mov: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/outputs/resized_cctv_full_fast_cut.mov -------------------------------------------------------------------------------- /outputs/resized_cctv_full_fast_cut.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/outputs/resized_cctv_full_fast_cut.mp4 -------------------------------------------------------------------------------- /outputs/sample_frame.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/outputs/sample_frame.png -------------------------------------------------------------------------------- /outputs/stations.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pgplarosa/Employee-Monitoring-Using-Object-Detection/84165a72930d785d7489e4a6300d4b4b69812957/outputs/stations.png --------------------------------------------------------------------------------