├── Body Language Detection.ipynb ├── Emotion Graph ├── Bar plot.png ├── Horizontal bar plot in decreasing order of sizes.png ├── Horizontal bar plot with increasing order of sizes.png ├── Horizontal bar plot.png └── Pie plot.png ├── LICENSE ├── README.md ├── Video Decoder ├── angry.mp4 ├── body_language.pkl ├── confused.mp4 ├── depressed.mp4 ├── excited.mp4 ├── happy.mp4 ├── sad.mp4 ├── scared.mp4 ├── serious.mp4 ├── surprised.mp4 └── tension.mp4 ├── body_language.pkl ├── body_language.tflite └── emotion_data.xlsx /Emotion Graph/Bar plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Emotion Graph/Bar plot.png -------------------------------------------------------------------------------- /Emotion Graph/Horizontal bar plot in decreasing order of sizes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Emotion Graph/Horizontal bar plot in decreasing order of sizes.png -------------------------------------------------------------------------------- /Emotion Graph/Horizontal bar plot with increasing order of sizes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Emotion Graph/Horizontal bar plot with increasing order of sizes.png -------------------------------------------------------------------------------- /Emotion Graph/Horizontal bar plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Emotion Graph/Horizontal bar plot.png -------------------------------------------------------------------------------- /Emotion Graph/Pie plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Emotion Graph/Pie plot.png -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Baivab Sarkar 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Body-Language-Detection-with-MediaPipe-and-OpenCV 2 | https://github.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/assets/109382325/231cfaf5-f1e1-410d-998c-caa984f03c52 3 | 4 | 5 | **This Jupyter Notebook (IPython Notebook) provides the code and instructions for implementing body language detection using [MediaPipe](https://github.com/google/mediapipe) and [OpenCV](https://github.com/opencv/opencv). This innovative tool incorporates two distinct models to achieve its functionality, providing users with a comprehensive approach to body language analysis.** 6 | * [Scikit-Learn (.pkl)](https://github.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/blob/main/README.md#1-scikit-learn-pkl) 7 | * [TensorFlow-Keras (.tflite)](https://github.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV#2-tensorflow-keras-tflite) 8 | * [Features ⭐](https://github.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV#features-) 9 | 10 | ## 1. Scikit-Learn (.pkl) 11 | The first model is built using **Scikit-Learn** and is stored in a **.pkl (Python Pickle) format**. 12 | 1. It employs pipelines to encapsulate preprocessing and modeling steps for multiple algorithms. 13 | ```python 14 | pipelines = { 15 | 'lr':make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)), 16 | 'rc':make_pipeline(StandardScaler(), RidgeClassifier()), 17 | 'rf':make_pipeline(StandardScaler(), RandomForestClassifier()), 18 | 'gb':make_pipeline(StandardScaler(), GradientBoostingClassifier()), 19 | } 20 | ``` 21 | 22 | 2. It systematically trains and evaluates different models using accuracy as a metric. 23 | ```output 24 | lr 0.995260663507109 25 | rc 0.985781990521327 26 | rf 0.9881516587677726 27 | gb 0.9928909952606635 28 | ``` 29 | 3. It saves the best-performing model for later use using pickle. 30 | ```python 31 | with open('body_language.pkl', 'wb') as f: 32 | pickle.dump(fit_models['rf'], f) 33 | ``` 34 | ## 2. TensorFlow-Keras (.tflite) 35 | 36 | The second model is built using **TensorFlow-Keras** and is stored in a **TensorFlow Lite (.tflite) format**. 37 | 1. It Builds and compiles a neural network model for classification. 38 | ```python 39 | model.compile(optimizer='adam', 40 | loss='sparse_categorical_crossentropy', 41 | metrics=['accuracy']) 42 | ``` 43 | 2. It trains the model with relevant metrics. 44 | 3. It converts and saves the model in TensorFlow Lite format for mobile deployment. 45 | ```python 46 | converter = tf.lite.TFLiteConverter.from_keras_model(model) 47 | tflite_model = converter.convert() 48 | open("body_language.tflite", "wb").write(tflite_model) 49 | ``` 50 | ## Features ⭐ 51 | ### 1. Create the training dataset using both a **Webcam** and **recording video data (.mp4)**, extracting relevant frames, and annotating those frames with corresponding labels. 52 | #### View Folder: [Video Decoder](https://github.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/tree/main/Video%20Decoder) 53 | #### Modify the codef for MP4: 54 | ```python 55 | class_name = "Happy" 56 | # Replace 'path_to_your_video_file' with the actual path to your video file 57 | cap = cv2.VideoCapture('path_to_your_video_file') 58 | # Initiate holistic model 59 | with mp_holistic.Holistic(min_detection_confidence=0.5, min_tracking_confidence=0.5) as holistic: 60 | 61 | while cap.isOpened(): 62 | ret, frame = cap.read() 63 | ``` 64 | ### 2. Trained models to recognize 10 distinct body language and facial expression categories, enabling the automated recognition of emotions and gestures in videos. 65 | #### Class Labels 66 | 1. Happy 67 | 2. Sad 68 | 3. Angry 69 | 4. Surprised 70 | 5. Confused 71 | 6. Tension 72 | 7. Surprised 73 | 8. Excited 74 | 9. Pain 75 | 10. Depressed 76 | ### 3. Visual representation of different emotional expressions, with each expression depicted in a separate chart or plot using the Matplotlib library in Python. 77 | #### Pie plot 78 | ![image](https://github.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/assets/109382325/9562c572-fe44-4982-aa8b-9a9e7a241f52) 79 | #### Bar plot 80 | ![image](https://github.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/assets/109382325/b1882a70-f2f8-4f38-ae06-8acfbce5e30f) 81 | #### Horizontal bar plot 82 | ![image](https://github.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/assets/109382325/7585391a-8c02-4c8a-87ea-cc4771f6fc09) 83 | #### Horizontal bar plot in creasing order of sizes 84 | ![image](https://github.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/assets/109382325/a1776f7f-994d-4fca-97c2-22e03a4f4465) 85 | #### Horizontal bar plot with increasing order of sizes 86 | ![image](https://github.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/assets/109382325/4f24a433-daac-4024-90ff-34f00f9c2825) 87 | 88 | 89 | -------------------------------------------------------------------------------- /Video Decoder/angry.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Video Decoder/angry.mp4 -------------------------------------------------------------------------------- /Video Decoder/body_language.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Video Decoder/body_language.pkl -------------------------------------------------------------------------------- /Video Decoder/confused.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Video Decoder/confused.mp4 -------------------------------------------------------------------------------- /Video Decoder/depressed.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Video Decoder/depressed.mp4 -------------------------------------------------------------------------------- /Video Decoder/excited.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Video Decoder/excited.mp4 -------------------------------------------------------------------------------- /Video Decoder/happy.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Video Decoder/happy.mp4 -------------------------------------------------------------------------------- /Video Decoder/sad.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Video Decoder/sad.mp4 -------------------------------------------------------------------------------- /Video Decoder/scared.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Video Decoder/scared.mp4 -------------------------------------------------------------------------------- /Video Decoder/serious.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Video Decoder/serious.mp4 -------------------------------------------------------------------------------- /Video Decoder/surprised.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Video Decoder/surprised.mp4 -------------------------------------------------------------------------------- /Video Decoder/tension.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/Video Decoder/tension.mp4 -------------------------------------------------------------------------------- /body_language.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/body_language.pkl -------------------------------------------------------------------------------- /body_language.tflite: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/body_language.tflite -------------------------------------------------------------------------------- /emotion_data.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ThisIs-Developer/Body-Language-Detection-with-MediaPipe-and-OpenCV/07df7b5aeaa48f2a30d433ee0f275cdff6bcb1bc/emotion_data.xlsx --------------------------------------------------------------------------------