├── .gitattributes
├── .gitignore
├── README.md
├── download.py
├── download_imagenet.py
├── images
    ├── ex1.png
    ├── ex2.png
    ├── model_plot.png
    └── result.png
├── infer.py
├── infer_cam.py
└── requirements.txt


/.gitattributes:
--------------------------------------------------------------------------------
1 | # Auto detect text files and perform LF normalization
2 | * text=auto
3 | 


--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
 1 | *.zip
 2 | *.wmv
 3 | *.mpg
 4 | *.mpeg
 5 | *.mp4
 6 | *.mov
 7 | *.flv
 8 | *.avi
 9 | *.ogv
10 | *.ogg
11 | *.webm


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # Violence Detection using CNN LSTM
 2 | 
 3 | ## Introduction
 4 | 
 5 | Today, the amount of public violence has increased dramatically. As much in high schools as in the street. This has resulted in the ubiquitous use of surveillance cameras. This has helped the authorities to identify these events and take the necessary measures. But almost all systems today require the human-inspection of these videos to identify such events, which is virtually inefficient. It is therefore necessary to have such a practical system that can automatically monitor and identify the surveillance videos. The development of various deep learning techniques, thanks to the availability of large data sets and computational resources, has resulted in a historic change in the community of computer vision. Various techniques have been developed to address problems such as object detection, recognition, tracking, action recognition, legend generation, etc.
 6 | 
 7 | ## Architecture 
 8 | 
 9 | ![Decoder model architecture](images/model_plot.png)  
10 | 
11 | **Fig: Decoder Network Architecture**
12 | 
13 | ## Dataset
14 | 
15 | data path are defined as follows:
16 | 
17 | - hocky - [Data_Link](http://academictorrents.com/details/38d9ed996a5a75a039b84cf8a137be794e7cee89/tech)
18 | - violentflow - [Data_Link](https://www.openu.ac.il/home/hassner/data/violentflows/)
19 | - movies - [Data_Link](http://academictorrents.com/details/70e0794e2292fc051a13f05ea6f5b6c16f3d3635)
20 | 
21 | ## Usage of pretrained model
22 | 
23 | - Clone this repository and change directory to this repository
24 | 
25 | ```bash
26 | git clone https://github.com/manncodes/Violence-Detection-CNNLSTM.git
27 | cd Violence-Detection-CNNLSTM
28 | ```
29 | 
30 | - Download Inbuilt Keras VGG16 model by running the following script:
31 | 
32 | ```bash
33 | python download_imagenet.py
34 | ```
35 | 
36 | - Now download the pretrained model H5 file from [here](https://drive.google.com/drive/folders/1SYD0dbfOLRBcidaACw5aiQsbtN4ySXLh?usp=sharing) and put it inside the model folder so it'll be "model/vlstm_92.h5".
37 | 
38 | - Install Dependencies
39 | 
40 |  ```bash
41 |  pip install -r requirements.txt
42 |  ```
43 | 
44 | - to run inference on a video existing in 'data' folder run following command:
45 | 
46 | ```bash
47 | python infer.py <Video name existing in data folder>
48 | # Example
49 | python infer.py fi3_xvid.avi
50 | ```
51 | 
52 | - to run inference live:
53 | 
54 | ```bash
55 | python infer_cam.py
56 | ```
57 | 
58 | Example of 2 consecutive inference of a violent video.
59 | ![Example](images/ex1.png)
60 | 
61 | ![Example](images/ex2.png)
62 | 
63 | *to be noted better version of live inference is made by us in a different repository. Do contact me to get access to it. That repository also contains an android app for the violence detection UI.*
64 | 
65 | ## Results
66 | 
67 | **Results on Hocket Dataset after hypertuning of parameters**
68 | ![results](images/result.png)
69 | 


--------------------------------------------------------------------------------
/download.py:
--------------------------------------------------------------------------------
 1 | import sys
 2 | import os
 3 | import urllib.request
 4 | import tarfile
 5 | import zipfile
 6 | 
 7 | def _print_download_progress(count, block_size, total_size):
 8 |     # Percentage completion.
 9 |     pct_complete = float(count * block_size) / total_size
10 | 
11 |     # Status-message. Note the \r which means the line should overwrite itself.
12 |     msg = "\r- Download progress: {0:.1%}".format(pct_complete)
13 | 
14 |     # Print it.
15 |     sys.stdout.write(msg)
16 |     sys.stdout.flush()
17 | 
18 | 
19 | 
20 | 
21 | 
22 | def maybe_download_and_extract(url, download_dir):
23 | 
24 |     # Filename for saving the file downloaded from the internet.
25 |     # Use the filename from the URL and add it to the download_dir.
26 |     filename = url.split('/')[-1]
27 |     file_path = os.path.join(download_dir, filename)
28 | 
29 |     if not os.path.exists(file_path):
30 |         # Check if the download directory exists, otherwise create it.
31 |         if not os.path.exists(download_dir):
32 |             os.makedirs(download_dir)
33 | 
34 |         # Download the file from the internet.
35 |         file_path, _ = urllib.request.urlretrieve(url=url,
36 |                                                   filename=file_path,
37 |                                                   reporthook=_print_download_progress)
38 | 
39 |         print()
40 |         print("Download finished. Extracting files.")
41 | 
42 |         if file_path.endswith(".zip"):
43 |             # Unpack the zip-file.
44 |             zipfile.ZipFile(file=file_path, mode="r").extractall(download_dir)
45 |         elif file_path.endswith((".tar.gz", ".tgz")):
46 |             # Unpack the tar-ball.
47 |             tarfile.open(name=file_path, mode="r:gz").extractall(download_dir)
48 | 
49 |         print("Done.")
50 |     else:
51 |         print("Data has apparently already been downloaded and unpacked.")
52 | 
53 | 
54 | 


--------------------------------------------------------------------------------
/download_imagenet.py:
--------------------------------------------------------------------------------
1 | from keras.applications import VGG16
2 | '''
3 | This will download VGG16 Model locally.
4 | This script must be run before running rest of the scripts.
5 | '''
6 | image_model = VGG16(include_top=True, weights='imagenet')  
7 | image_model.summary()


--------------------------------------------------------------------------------
/images/ex1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/manncodes/Violence-Detection-CNNLSTM/c53ab926d42f8fdc8a7a9d2a1681b3abb5c59bed/images/ex1.png


--------------------------------------------------------------------------------
/images/ex2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/manncodes/Violence-Detection-CNNLSTM/c53ab926d42f8fdc8a7a9d2a1681b3abb5c59bed/images/ex2.png


--------------------------------------------------------------------------------
/images/model_plot.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/manncodes/Violence-Detection-CNNLSTM/c53ab926d42f8fdc8a7a9d2a1681b3abb5c59bed/images/model_plot.png


--------------------------------------------------------------------------------
/images/result.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/manncodes/Violence-Detection-CNNLSTM/c53ab926d42f8fdc8a7a9d2a1681b3abb5c59bed/images/result.png


--------------------------------------------------------------------------------
/infer.py:
--------------------------------------------------------------------------------
  1 | import cv2
  2 | import os
  3 | import numpy as np
  4 | import keras
  5 | from keras.utils.vis_utils import plot_model
  6 | from keras.applications import VGG16
  7 | from keras import backend as K
  8 | from keras.models import Model
  9 | import sys
 10 | import time
 11 | import multiprocessing
 12 | from termcolor import colored
 13 | os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
 14 | model = keras.models.load_model('model/vlstm_92.h5')
 15 | image_model = VGG16(include_top=True, weights='imagenet')  
 16 | model.summary()  
 17 | # plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)
 18 | #We will use the output of the layer prior to the final
 19 | # classification-layer which is named fc2. This is a fully-connected (or dense) layer.
 20 | transfer_layer = image_model.get_layer('fc2')
 21 | image_model_transfer = Model(inputs=image_model.input,outputs=transfer_layer.output)
 22 | transfer_values_size = K.int_shape(transfer_layer.output)[1]
 23 | 
 24 | # Frame size  
 25 | img_size = 224
 26 | 
 27 | img_size_touple = (img_size, img_size)
 28 | 
 29 | # Number of channels (RGB)
 30 | num_channels = 3
 31 | 
 32 | # Flat frame size
 33 | img_size_flat = img_size * img_size * num_channels
 34 | 
 35 | # Number of classes for classification (Violence-No Violence)
 36 | num_classes = 2
 37 | 
 38 | # Number of files to train
 39 | _num_files_train = 1
 40 | 
 41 | # Number of frames per video
 42 | _images_per_file = 20
 43 | 
 44 | # Number of frames per training set
 45 | _num_images_train = _num_files_train * _images_per_file
 46 | 
 47 | # Video extension
 48 | video_exts = ".avi"
 49 | 
 50 | in_dir = "data"
 51 | 
 52 | def get_frames(current_dir, file_name):
 53 |     in_file = os.path.join(current_dir, file_name)
 54 |     images = []
 55 |     vidcap = cv2.VideoCapture(in_file)
 56 |     success,image = vidcap.read()
 57 |     count = 0
 58 |     while count<_images_per_file:
 59 |         RGB_img = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
 60 |         res = cv2.resize(RGB_img, dsize=(img_size, img_size),interpolation=cv2.INTER_CUBIC) 
 61 |         images.append(res)
 62 |         success,image = vidcap.read()
 63 |         count += 1
 64 |     resul = np.array(images)
 65 |     resul = (resul / 255.).astype(np.float16)
 66 |     return resul
 67 | 
 68 | 
 69 | def get_transfer_values(current_dir, file_name):
 70 |     
 71 |     # Pre-allocate input-batch-array for images.
 72 |     shape = (_images_per_file,) + img_size_touple + (3,)
 73 |     image_batch = np.zeros(shape=shape, dtype=np.float16)
 74 |     image_batch = get_frames(current_dir, file_name)
 75 |     # Pre-allocate output-array for transfer-values.
 76 |     # Note that we use 16-bit floating-points to save memory.
 77 |     shape = (_images_per_file, transfer_values_size)
 78 |     transfer_values = np.zeros(shape=shape, dtype=np.float16)
 79 | 
 80 |     transfer_values = \
 81 |             image_model_transfer.predict(image_batch)
 82 |             
 83 |     return transfer_values
 84 | 
 85 | 
 86 | def infer(curr_dir,file_name):
 87 |     tr = get_transfer_values(curr_dir,file_name)
 88 |     print('inp shape = ',tr.shape)
 89 |     tr = tr[np.newaxis,...]
 90 |     inp = np.array(tr)
 91 |     print('inp shape = ',inp.shape)
 92 |     pred = model.predict(inp)
 93 |     print(pred)
 94 |     res = np.argmax(pred[0])
 95 |     if res == 0:
 96 |         print("\n\n"+ colored('VIOLENT','red')+" Video with confidence: "+str(round(pred[0][res]*100,2))+" %")
 97 |         return 'Violent',str(round(pred[0][res]*100,2))
 98 |     else:
 99 |         print("\n\n" + colored('NON-VIOLENT','green') +" Video with confidence: "+str(round(pred[0][res]*100,2))+" %")
100 |         return 'Non-Violent',str(round(pred[0][res]*100,2))
101 | 
102 | 
103 | if __name__ == "__main__":
104 |     arg = sys.argv
105 |     video_name = arg[1]
106 |     start_time = time.time()
107 |     result,confidence =infer(in_dir,video_name)
108 |     if result=='Violent':
109 |         color = (0,0,255)
110 |     else:
111 |         color = (0,255,0)
112 | 
113 |     end_time = time.time()
114 |     # url = 'http://192.168.43.1:8080/video'
115 |     # cap = cv2.VideoCapture(url)
116 |     '''
117 |     for the live stream:
118 |     '''
119 |     # while(True):
120 |     #     ret, frame = cap.read()
121 |     #     if frame is not None:
122 |     #         cv2.imshow('frame',frame)
123 |     #     q = cv2.waitKey(1)
124 |     #     if q == ord("q"):
125 |     #         break
126 |     # cv2.destroyAllWindows()
127 |     
128 |     '''
129 |     for the prerecorded video
130 |     ''' 
131 |     delta = round(end_time-start_time,2)
132 |     fps = round(20/(delta*2),2)
133 |     print("Inferrence time: "+str(delta)+" s")
134 |     print(str(fps)+" fps ^_^")
135 |     vpath = in_dir + '\\' + video_name
136 |     cap = cv2.VideoCapture(vpath)
137 |     count = 0
138 |     while(cap.isOpened()):
139 |         ret, frame = cap.read()
140 |         time.sleep(0.05)
141 |         try:
142 |             frame = cv2.putText(frame,result + ' detected', (40,40), cv2.FONT_HERSHEY_SIMPLEX, 1,color, 2, cv2.LINE_AA)
143 |             # frame = cv2.putText(frame,confidence + ' %',(100,40),cv2.FONT_HERSHEY_SIMPLEX, 0.4,color , 1, cv2.LINE_AA)
144 |             cv2.imshow('frame', frame)
145 |             cv2.imwrite('result/'+str(result)+str(count)+'.jpg',frame)
146 |             time.sleep(0.05)
147 |             count+=1
148 |         except:
149 |             break
150 |         if cv2.waitKey(1) & 0xFF == ord('q'):
151 |             break
152 |     cap.release()
153 |     cv2.destroyAllWindows()
154 | 


--------------------------------------------------------------------------------
/infer_cam.py:
--------------------------------------------------------------------------------
  1 | import cv2
  2 | import os
  3 | import numpy as np
  4 | import keras
  5 | from keras.applications import VGG16
  6 | from keras import backend as K
  7 | from keras.models import Model
  8 | import sys
  9 | import time
 10 | import multiprocessing
 11 | from termcolor import colored
 12 | os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
 13 | model = keras.models.load_model('model/vlstm_92.h5')
 14 | image_model = VGG16(include_top=True, weights='imagenet')  
 15 | model.summary()  
 16 | #We will use the output of the layer prior to the final
 17 | # classification-layer which is named fc2. This is a fully-connected (or dense) layer.
 18 | transfer_layer = image_model.get_layer('fc2')
 19 | image_model_transfer = Model(inputs=image_model.input,outputs=transfer_layer.output)
 20 | transfer_values_size = K.int_shape(transfer_layer.output)[1]
 21 | 
 22 | # Frame size  
 23 | img_size = 224
 24 | 
 25 | img_size_touple = (img_size, img_size)
 26 | 
 27 | # Number of channels (RGB)
 28 | num_channels = 3
 29 | 
 30 | # Flat frame size
 31 | img_size_flat = img_size * img_size * num_channels
 32 | 
 33 | # Number of classes for classification (Violence-No Violence)
 34 | num_classes = 2
 35 | 
 36 | # Number of files to train
 37 | _num_files_train = 1
 38 | 
 39 | # Number of frames per video
 40 | _images_per_file = 20
 41 | 
 42 | # Number of frames per training set
 43 | _num_images_train = _num_files_train * _images_per_file
 44 | 
 45 | # Video extension
 46 | video_exts = ".avi"
 47 | 
 48 | in_dir = "data"
 49 | 
 50 | #url of video stream
 51 | url = 'http://192.168.43.1:8080/video'
 52 | 
 53 | 
 54 | if __name__ == "__main__":
 55 |     cap = cv2.VideoCapture(url)
 56 |     time.sleep(0.5)
 57 |     count = 0 
 58 |     images=[]
 59 |     shape = (_images_per_file,) + img_size_touple + (3,)
 60 |     image_batch = np.zeros(shape=shape, dtype=np.float16)
 61 |     while(True):
 62 |         ret, frame = cap.read()
 63 |         count+=1
 64 |         
 65 |         if count <= _images_per_file:
 66 |             RGB_img = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
 67 |             res = cv2.resize(RGB_img, dsize=(img_size, img_size),interpolation=cv2.INTER_CUBIC) 
 68 |             images.append(res)
 69 |         else:
 70 |             resul = np.array(images)
 71 |             resul = (resul / 255.).astype(np.float16)
 72 |             transfer_shape = (_images_per_file, transfer_values_size)
 73 |             transfer_values = np.zeros(shape=transfer_shape, dtype=np.float16)
 74 |             transfer_values = image_model_transfer.predict(image_batch)
 75 |             print(transfer_values.shape)
 76 |             inp = np.array(transfer_values)
 77 |             inp = inp[np.newaxis,...]
 78 |             print(inp.shape)
 79 |             pred = model.predict(inp)
 80 |             res = np.argmax(pred[0])
 81 |             count = 0
 82 |             images = []
 83 |             shape = (_images_per_file,) + img_size_touple + (3,)
 84 |             image_batch = np.zeros(shape=shape, dtype=np.float16)
 85 | 
 86 |             #print result
 87 |             if res == 0:
 88 |                 print("\n\n"+ colored('VIOLENT','red')+" Video with confidence: "+str(round(pred[0][res]*100,2))+" %")
 89 |             else:
 90 |                 print("\n\n" + colored('NON-VIOLENT','green') +" Video with confidence: "+str(round(pred[0][res]*100,2))+" %")
 91 |         # showing the video stream
 92 |         if frame is not None:
 93 |             cv2.imshow('frame',frame)
 94 |         q = cv2.waitKey(1)
 95 |         if q == ord("q"):
 96 |             break
 97 |     cv2.destroyAllWindows()
 98 | 
 99 | 
100 | 


--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | opencv_python==4.4.0.44
2 | termcolor==1.1.0
3 | numpy==1.19.2
4 | Keras==2.4.3
5 | 


--------------------------------------------------------------------------------