├── .gitignore ├── LICENSE.txt ├── README.md ├── models └── model.py ├── piano_keys ├── key01.mp3 ├── key02.mp3 ├── key03.mp3 ├── key04.mp3 ├── key05.mp3 ├── key06.mp3 ├── key07.mp3 ├── key08.mp3 ├── key09.mp3 ├── key10.mp3 ├── key11.mp3 ├── key12.mp3 ├── key13.mp3 ├── key14.mp3 ├── key15.mp3 ├── key16.mp3 ├── key17.mp3 ├── key18.mp3 ├── key19.mp3 ├── key20.mp3 ├── key21.mp3 ├── key22.mp3 ├── key23.mp3 └── key24.mp3 ├── requirements.txt ├── run.py └── src ├── GLOBAL.py ├── fetch_data.py ├── mapping.py ├── piano.py ├── train_model.py └── training_data ├── touched └── .gitkeep └── untouched └── .gitkeep /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | .idea 3 | temp 4 | tp.py 5 | demo -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 Mayuresh1611 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # PAPER PIANO 2 | 3 | ### Now we don't need to buy a Piano if we want to play music. We can play piano on paper, although it may not give a feeling of pressing keys on the piano but it gets work done! 4 | 5 | https://github.com/Mayuresh1611/Paper-Piano/assets/103099867/c2b0b4f1-3b6b-4eed-927d-91c5b5c91708 6 | 7 | **Currently it only supports a maximum of 2 fingers** (1 finger of both hands). Support for more fingers and a highly susceptible training model is on the way. 8 | 9 | 1. [Setting up project](#setting-up-project) 10 | 2. [How to Use](#how-to-use) 11 | 3. [Training and Adjusting ](#training-and-adjusting) 12 | 4. [Contributing to this project](#contributing-to-this-project) 13 | 14 | ## SETTING UP PROJECT 15 | Python version 3.11 and above 16 | 1. Clone the repository ```git clone https://github.com/Mayuresh1611/Paper-Piano.git``` 17 | 2. run command ```pip install -r requirements.txt``` in the command line. 18 | 3. Execute ```run.py``` file 19 | 20 | ## HOW TO USE 21 | This is a little trickier part as the project requires you to set up a webcam in a specific angle at a specific height and distance. Also stronger the light, the better the performance. 22 | #### STUFF YOU WILL REQUIRE 23 | 1. webcam or you can use third-party tools for webcam. 24 | 2. Two A4-sized white paper, horizontally joined together. 2 rectangles need to be drawn at both ends of the paper with a black marker, thicker lines yield better results. 25 | 3. The recommended position for the webcam will be such that it can capture the finger and shadow beneath the finger and should have both boxes we drew on joined paper in the FOV of the camera. 26 | Just like shown in the demo video. 27 | 4. A light source in front, ie. behind the camera would be preferred. Casting sharp shadows. 28 | 4. Hand with all fingers. 29 | 30 | #### TRAINING AND ADJUSTING 31 | * During training the model on your finger, the first window will appear where the box will be drawn around the tip of the finger. If the box does not cover the complete finger and little surrounding of the finger. Adjust the camera accordingly. 32 | * In the training stage, do not move your finger furiously, move it slowly giving out every angle. 33 | * While training, during the touched finger state, do press the finger but not too hard. In an untouched finger state, do not touch paper, you can get near paper but not too close. Lift the finger high like you would normally do. 34 | * CNN has been used to train over the data, it distinguishes the touched and untouched finger, if the results are not satisfactory, retrain the model. 35 | ## CONTRIBUTING TO THIS PROJECT 36 | If we make this project into a complete working piano on paper. It would give access to pianos and instruments for those who cannot afford them. ```like me :)``` 37 | 38 | * There are no set rules or code of conduct as of now. Anyone with better ideas and improvement is welcomed. 39 | * The only rules are 40 | 1. The name of the function should match the functionality it is doing. 41 | 2. Use comments only when required. 42 | 3. Share with the ones who can improve this project even more. 43 | 44 | -------------------------------------------------------------------------------- /models/model.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | from tensorflow.keras.models import load_model 3 | import numpy as np 4 | import sys 5 | sys.path.append("src") 6 | import GLOBAL 7 | import os 8 | 9 | 10 | model_list = os.listdir("models") 11 | if "touch_detection_model.h5" not in model_list: 12 | print("We need to train model on your finger's data") 13 | else: 14 | model = load_model("models/touch_detection_model.h5") 15 | 16 | def Predict(img): 17 | resized_img = cv2.resize(img, GLOBAL.TRACKING_BOX_RESOLUTION) 18 | # Add the batch dimension and normalize pixel values 19 | data = np.expand_dims(resized_img/255, axis=0) 20 | # Make the prediction 21 | prediction = model.predict(data) 22 | print(prediction) 23 | if prediction > 0.502: 24 | return 0 25 | else: 26 | return 1 -------------------------------------------------------------------------------- /piano_keys/key01.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key01.mp3 -------------------------------------------------------------------------------- /piano_keys/key02.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key02.mp3 -------------------------------------------------------------------------------- /piano_keys/key03.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key03.mp3 -------------------------------------------------------------------------------- /piano_keys/key04.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key04.mp3 -------------------------------------------------------------------------------- /piano_keys/key05.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key05.mp3 -------------------------------------------------------------------------------- /piano_keys/key06.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key06.mp3 -------------------------------------------------------------------------------- /piano_keys/key07.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key07.mp3 -------------------------------------------------------------------------------- /piano_keys/key08.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key08.mp3 -------------------------------------------------------------------------------- /piano_keys/key09.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key09.mp3 -------------------------------------------------------------------------------- /piano_keys/key10.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key10.mp3 -------------------------------------------------------------------------------- /piano_keys/key11.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key11.mp3 -------------------------------------------------------------------------------- /piano_keys/key12.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key12.mp3 -------------------------------------------------------------------------------- /piano_keys/key13.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key13.mp3 -------------------------------------------------------------------------------- /piano_keys/key14.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key14.mp3 -------------------------------------------------------------------------------- /piano_keys/key15.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key15.mp3 -------------------------------------------------------------------------------- /piano_keys/key16.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key16.mp3 -------------------------------------------------------------------------------- /piano_keys/key17.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key17.mp3 -------------------------------------------------------------------------------- /piano_keys/key18.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key18.mp3 -------------------------------------------------------------------------------- /piano_keys/key19.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key19.mp3 -------------------------------------------------------------------------------- /piano_keys/key20.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key20.mp3 -------------------------------------------------------------------------------- /piano_keys/key21.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key21.mp3 -------------------------------------------------------------------------------- /piano_keys/key22.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key22.mp3 -------------------------------------------------------------------------------- /piano_keys/key23.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key23.mp3 -------------------------------------------------------------------------------- /piano_keys/key24.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/piano_keys/key24.mp3 -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy 2 | matplotlib 3 | tensorflow 4 | opencv-python 5 | mediapipe 6 | scipy 7 | pygame 8 | shapely 9 | -------------------------------------------------------------------------------- /run.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import mediapipe as mp 3 | import os 4 | import time 5 | from src import fetch_data , train_model, GLOBAL , piano 6 | from models import model 7 | 8 | FINGER = [GLOBAL.INDEX_FINGER] 9 | 10 | def RUN(finger): 11 | mp_drawing = mp.solutions.drawing_utils 12 | mp_hands = mp.solutions.hands 13 | 14 | cap = cv2.VideoCapture(GLOBAL.WEB_CAM) 15 | iter = 0 16 | with mp_hands.Hands( 17 | min_detection_confidence=0.5, 18 | min_tracking_confidence=0.5) as hands: 19 | while cap.isOpened(): 20 | success, image = cap.read() 21 | if not success: 22 | print("Ignoring empty camera frame.") 23 | continue 24 | 25 | image = cv2.cvtColor(image , cv2.COLOR_BGR2RGB) 26 | copy_image = image.copy() 27 | 28 | image.flags.writeable = False 29 | results = hands.process(image) 30 | image.flags.writeable = True 31 | image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) 32 | 33 | finger_tracking_frame = None # initializing region of interest 34 | 35 | if results.multi_hand_landmarks: 36 | for hand_landmarks in results.multi_hand_landmarks: 37 | mp_drawing.draw_landmarks( 38 | image, hand_landmarks, mp_hands.HAND_CONNECTIONS) 39 | for finger_tip_id in [finger]: # Landmark IDs for all five fingers' tips 40 | finger_tip = hand_landmarks.landmark[finger_tip_id] 41 | height, width, _ = image.shape 42 | tip_x, tip_y, tip_z = int(finger_tip.x * width), int(finger_tip.y * height), finger_tip.z 43 | 44 | box_size = int(GLOBAL.BOX_SIZE // 2) # Adjust the size of the box as needed 45 | box_color = (0, 255, 0) # Green color 46 | 47 | # Coordinates of the rectangle 48 | x1, y1 = tip_x - box_size, tip_y - box_size 49 | x2, y2 = tip_x + box_size, tip_y + box_size 50 | 51 | # Draw a square box around the finger tip 52 | cv2.rectangle(image, (x1, y1), (x2, y2), box_color, 2) 53 | 54 | # Crop the region of interest (ROI) 55 | finger_tracking_frame = copy_image[y1:y2, x1:x2] 56 | color = (0, 0, 255) 57 | if finger_tracking_frame is not None and finger_tracking_frame.shape[0] > 0 and finger_tracking_frame.shape[1] > 0: 58 | finger_tracking_frame = cv2.cvtColor(finger_tracking_frame , cv2.COLOR_BGR2RGB) 59 | pred = model.Predict(finger_tracking_frame) 60 | if pred: 61 | color=(0, 255, 0) 62 | else: 63 | if color == (0, 255, 0): 64 | pass 65 | else: 66 | color = (0, 0, 255) 67 | image = cv2.circle(image, (250 , 300), 2, color, 20) 68 | cv2.imshow('Tocuh tracking', image) 69 | key = cv2.waitKey(5) 70 | if key == ord('q'): 71 | cap.release() 72 | cv2.destroyAllWindows() 73 | break 74 | 75 | 76 | def fetch_train_data(): 77 | print("The First window is try window so that you can adjust the finger position, adjust hand position so that box will cover finger tip and finger tip\n1. Window after try will be touch train window\n\t do not lift any finger, move fingers slowly on the paper to get all angles\n2. After this window untouch train window will pop up\n\t lift fingers so that it can take pics of finger tips for training\n\t Then model will be trained and you should see the prediction window for Index finger") 78 | print("Press Y to move for training stage") 79 | while 1: 80 | key = input(">> ") 81 | if key.lower() == 'y': 82 | break 83 | time.sleep(2) 84 | fetch_data.Try(FINGER) 85 | time.sleep(2) 86 | fetch_data.Capture(GLOBAL.TOUCH_FOLDER , "touched" , FINGER) 87 | time.sleep(2) 88 | fetch_data.Capture(GLOBAL.UNTOUCH_FOLDER , "untouched" , FINGER) 89 | 90 | train_model.start_training() 91 | 92 | print("Model Training Complete") 93 | time.sleep(3) 94 | 95 | RUN(GLOBAL.INDEX_FINGER) 96 | 97 | print("welcome to Paper Piano") 98 | # fetch_data.delete_model() 99 | run = True 100 | 101 | fetch_data.clear_training_data() 102 | 103 | while run: 104 | model_list = os.listdir("models") 105 | if "touch_detection_model.h5" not in model_list: 106 | print("We need to train model on your finger's data") 107 | fetch_data.clear_training_data() 108 | fetch_train_data() 109 | 110 | else: 111 | 112 | print("-------------*MENU*-------------\n[1] Retrain model\n[2] Start Paper Piano\n[3] Exit") 113 | check = True 114 | while check: 115 | opt = int(input()) 116 | if opt == 1: 117 | check = False 118 | fetch_data.clear_training_data() 119 | fetch_train_data() 120 | elif opt == 2: 121 | check = False 122 | print("Adjust paper accordingly until you see mesh of keys and press 'q'") 123 | time.sleep(3) 124 | piano.start_piano(GLOBAL.INDEX_FINGER) 125 | elif opt==3: 126 | check = False 127 | run = False 128 | -------------------------------------------------------------------------------- /src/GLOBAL.py: -------------------------------------------------------------------------------- 1 | WEB_CAM = 0 # by default 0 2 | 3 | BOX_SIZE = 40 4 | TRACKING_BOX_RESOLUTION = (BOX_SIZE , BOX_SIZE) 5 | THUMB = 4 6 | 7 | UNTOUCH_FOLDER = "src/training_data/untouched" 8 | TOUCH_FOLDER = "src/training_data/touched" 9 | SAMPLES = 200 10 | # FINGER INDEX FOR mediapipe hand mesh 11 | INDEX_FINGER = 8 12 | MIDDLE_FINGER = 12 13 | RING_FINGER = 16 14 | PINKY_FINGER = 20 15 | -------------------------------------------------------------------------------- /src/fetch_data.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import numpy as np 3 | import mediapipe as mp 4 | import os 5 | from src import GLOBAL 6 | 7 | 8 | 9 | # CAPTURE TOUCHED n UNTOUCHED 10 | # track finger 11 | # save images 12 | 13 | def Capture(save_folder , finger_state , finger): 14 | mp_drawing = mp.solutions.drawing_utils 15 | mp_hands = mp.solutions.hands 16 | 17 | cap = cv2.VideoCapture(GLOBAL.WEB_CAM) 18 | iter = 0 19 | with mp_hands.Hands( 20 | min_detection_confidence=0.5, 21 | min_tracking_confidence=0.5) as hands: 22 | while cap.isOpened(): 23 | success, image = cap.read() 24 | if not success: 25 | print("Ignoring empty camera frame.") 26 | continue 27 | 28 | image = cv2.cvtColor(image , cv2.COLOR_BGR2RGB) 29 | copy_image = image.copy() 30 | 31 | image.flags.writeable = False 32 | results = hands.process(image) 33 | image.flags.writeable = True 34 | image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) 35 | 36 | finger_tracking_frame = None # initializing region of interest 37 | 38 | if results.multi_hand_landmarks: 39 | for hand_landmarks in results.multi_hand_landmarks: 40 | mp_drawing.draw_landmarks( 41 | image, hand_landmarks, mp_hands.HAND_CONNECTIONS) 42 | for finger_tip_id in finger: # Landmark IDs for all five fingers' tips 43 | finger_tip = hand_landmarks.landmark[finger_tip_id] 44 | height, width, _ = image.shape 45 | tip_x, tip_y, tip_z = int(finger_tip.x * width), int(finger_tip.y * height), finger_tip.z 46 | 47 | box_size = int(GLOBAL.BOX_SIZE // 2) # Adjust the size of the box as needed 48 | box_color = (0, 255, 0) # Green color 49 | 50 | # Coordinates of the rectangle 51 | x1, y1 = tip_x - box_size, tip_y - box_size 52 | x2, y2 = tip_x + box_size, tip_y + box_size 53 | 54 | # Draw a square box around the finger tip 55 | cv2.rectangle(image, (x1, y1), (x2, y2), box_color, 2) 56 | 57 | # Crop the region of interest (ROI) 58 | finger_tracking_frame = copy_image[y1:y2, x1:x2] 59 | 60 | if finger_tracking_frame is not None and finger_tracking_frame.shape[0] > 0 and finger_tracking_frame.shape[1] > 0: 61 | finger_tracking_frame = cv2.cvtColor(finger_tracking_frame , cv2.COLOR_BGR2RGB) 62 | filename = os.path.join(save_folder, f'finger-{finger_state}{iter}.png') 63 | cv2.imwrite(filename, finger_tracking_frame) 64 | print(f'Saved touched image: {filename}') 65 | iter += 1 66 | if iter >= GLOBAL.SAMPLES: 67 | cv2.destroyAllWindows() 68 | return 69 | 70 | cv2.imshow(f'{finger_state} SAVING', image) 71 | key = cv2.waitKey(5) 72 | if key == ord('q'): 73 | cap.release() 74 | cv2.destroyAllWindows() 75 | break 76 | 77 | 78 | 79 | 80 | def clear_training_data(): 81 | for file in os.listdir(GLOBAL.TOUCH_FOLDER): 82 | os.remove(f'{GLOBAL.TOUCH_FOLDER}/{file}') 83 | 84 | for file in os.listdir(GLOBAL.UNTOUCH_FOLDER): 85 | os.remove(f'{GLOBAL.UNTOUCH_FOLDER}/{file}') 86 | print("TRAINING DATA CLEARED") 87 | 88 | def delete_model(): 89 | model = os.listdir("models") 90 | if "touch_detection_model.h5" in model: 91 | model.remove("touch_detection_model.h5") 92 | print("model removed") 93 | else: 94 | print("model not present") 95 | 96 | def Try(finger): 97 | mp_drawing = mp.solutions.drawing_utils 98 | mp_hands = mp.solutions.hands 99 | 100 | cap = cv2.VideoCapture(GLOBAL.WEB_CAM) 101 | with mp_hands.Hands( 102 | min_detection_confidence=0.5, 103 | min_tracking_confidence=0.5) as hands: 104 | while cap.isOpened(): 105 | success, image = cap.read() 106 | if not success: 107 | print("Ignoring empty camera frame.") 108 | continue 109 | 110 | image = cv2.cvtColor(image , cv2.COLOR_BGR2RGB) 111 | image.flags.writeable = False 112 | results = hands.process(image) 113 | image.flags.writeable = True 114 | image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) 115 | 116 | 117 | if results.multi_hand_landmarks: 118 | for hand_landmarks in results.multi_hand_landmarks: 119 | mp_drawing.draw_landmarks( 120 | image, hand_landmarks, mp_hands.HAND_CONNECTIONS) 121 | for finger_tip_id in finger: # Landmark IDs for all five fingers' tips 122 | finger_tip = hand_landmarks.landmark[finger_tip_id] 123 | height, width, _ = image.shape 124 | tip_x, tip_y, tip_z = int(finger_tip.x * width), int(finger_tip.y * height), finger_tip.z 125 | 126 | box_size = int(GLOBAL.BOX_SIZE // 2) # Adjust the size of the box as needed 127 | box_color = (0, 255, 0) # Green color 128 | 129 | # Coordinates of the rectangle 130 | x1, y1 = tip_x - box_size, tip_y - box_size 131 | x2, y2 = tip_x + box_size, tip_y + box_size 132 | 133 | # Draw a square box around the finger tip 134 | cv2.rectangle(image, (x1, y1), (x2, y2), box_color, 2) 135 | cv2.imshow('Tocuh tracking', image) 136 | key = cv2.waitKey(5) 137 | if key == ord('q'): 138 | break 139 | cap.release() 140 | cv2.destroyAllWindows() 141 | 142 | 143 | 144 | -------------------------------------------------------------------------------- /src/mapping.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import numpy as np 3 | from scipy.spatial import distance 4 | 5 | 6 | 7 | """section formula""" 8 | def get_sections(line , points): 9 | 10 | sected_coords= [] 11 | 12 | x1= line[0][0] 13 | y1= line[0][1] 14 | x2= line[1][0] 15 | y2= line[1][1] 16 | 17 | for m in range(1, points+1): 18 | n = points+1 - m 19 | x = int((x2*m + x1*n) / (m+n)) 20 | y = int((y2*m + y1*n) / (m+n)) 21 | 22 | sected_coords.append([x, y]) 23 | return sected_coords 24 | 25 | 26 | """Drawing mesh inside those rectangles""" 27 | def draw_mesh(verts, image): 28 | # Define the polygon vertices 29 | polygon_vertices = np.array(verts, np.int32) 30 | 31 | # Reshape the array for OpenCV's fillPoly function 32 | polygon_vertices = polygon_vertices.reshape((-1, 1, 2)) 33 | 34 | 35 | # Draw the polygon 36 | cv2.line(image, verts[0], verts[1], (0, 255, 0), 2) 37 | cv2.line(image, verts[2], verts[3], (0, 255, 0), 2) 38 | 39 | # Define the number of rows and columns for the mesh 40 | rows, cols = 1, 24 41 | 42 | # columns 43 | line_ad = get_sections([verts[0] , verts[1]] , cols-1) 44 | line_bc = get_sections([verts[2] , verts[3]] , cols-1) 45 | 46 | 47 | 48 | # rows are currently not used 49 | line_ab = get_sections([verts[0] , verts[2]] , rows-1) 50 | line_dc = get_sections([verts[1] , verts[3]] , rows-1) 51 | 52 | line_ad.insert(0 , verts[0]) 53 | line_ad.append(verts[1]) 54 | 55 | line_bc.insert(0 , verts[2]) 56 | line_bc.append(verts[3]) 57 | 58 | 59 | polygons = [] 60 | for pnt in range(cols): 61 | polygon = [line_ad[pnt] , line_ad[pnt + 1] , line_bc[pnt + 1] , line_bc[pnt]] 62 | polygons.append(polygon) 63 | 64 | 65 | return cols , polygons 66 | 67 | 68 | def draw_over_image(image , cols , polygons): 69 | colors = [(255 , 0 , 0 ) , (0 , 255 , 0) , (0 , 0 , 255) , (0 , 0 ,0)] 70 | for pnt in range(cols): 71 | polygon = np.array(polygons[pnt], dtype=np.int32) 72 | cv2.polylines(image, [polygon], True, colors[1], 2) 73 | 74 | return image 75 | 76 | 77 | def analyse(image): 78 | gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) 79 | 80 | # Apply Gaussian blur and adaptive thresholding to obtain binary image 81 | blur = cv2.GaussianBlur(gray, (7, 7), 1) 82 | canny = cv2.Canny(blur , 10, 150) 83 | # cv2.imshow("canny" , canny) 84 | thresh = cv2.adaptiveThreshold(canny, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, 5, 2) 85 | 86 | # cv2.imshow("thresh" , thresh) 87 | # Find contours in the binary image 88 | contours, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) 89 | 90 | # Sort contours by area and keep the largest ones 91 | contours = sorted(contours, key=cv2.contourArea, reverse=True) 92 | 93 | shapes = [] 94 | 95 | # Loop over the contours 96 | for contour in contours: 97 | # Approximate the contour to a polygon 98 | peri = cv2.arcLength(contour, True) 99 | approx = cv2.approxPolyDP(contour, 0.024 * peri, True) 100 | # If the contour has 4 corners (a rectangle) 101 | if len(approx) == 4: 102 | shapes.append(approx) 103 | for point in approx: 104 | x, y = point.ravel() 105 | 106 | # Display the result 107 | try: 108 | count = 1 109 | mesh_coords = [] 110 | rect1 = [list(arr.flatten()) for arr in shapes[0]] 111 | rect2 = [list(arr.flatten()) for arr in shapes[1]] 112 | rect1.sort() 113 | rect2.sort() 114 | 115 | coords_n_dist = [] 116 | 117 | for i in rect1: 118 | for j in rect2: 119 | dist = distance.euclidean(i , j) 120 | coords_n_dist.append([dist , [i , j]]) 121 | coords_n_dist.sort() 122 | mesh_coords.extend(coords_n_dist[0][1]) 123 | for i in coords_n_dist[1:]: 124 | # threshold of 10 over y values of points 125 | if coords_n_dist[0][1][0][1] + 10 < i[1][0][1] and coords_n_dist[0][1][1][1] + 10< i[1][1][1] : 126 | max = i[1] 127 | break 128 | mesh_coords.extend(max) 129 | if rect1[0][0] > rect2[3][0]: 130 | rect1 , rect2 = rect2 , rect1 131 | 132 | for i in range(2): 133 | for j in range(4): 134 | cv2.putText(image,str(j) + str( shapes[i][j][0]) , shapes[i][j][0], cv2.FONT_HERSHEY_SIMPLEX, 0.4, (0, 0, 255), 2) 135 | except: 136 | # print("could not mesh") 137 | return 0 , [] 138 | else: 139 | return draw_mesh(mesh_coords, image) 140 | 141 | -------------------------------------------------------------------------------- /src/piano.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import mediapipe as mp 3 | mp_drawing = mp.solutions.drawing_utils 4 | mp_hands = mp.solutions.hands 5 | from src.mapping import analyse, draw_over_image 6 | import os 7 | from shapely.geometry import Point, Polygon 8 | import threading 9 | import queue 10 | import pygame 11 | from src import GLOBAL 12 | 13 | from models.model import Predict 14 | 15 | 16 | cols = 0 17 | polys = [] 18 | 19 | snd_list = os.listdir("piano_keys") 20 | sounds = [f"piano_keys/{sound}" for sound in snd_list] 21 | 22 | 23 | 24 | def play_sound(sound_path): 25 | pygame.mixer.init() 26 | pygame.mixer.music.load(sound_path) 27 | pygame.mixer.music.play() 28 | while pygame.mixer.music.get_busy(): 29 | continue 30 | 31 | def predict_worker(img_queue, result_queue, stop_event): 32 | 33 | while not stop_event.is_set(): 34 | img = img_queue.get() 35 | if img is None: 36 | break 37 | 38 | prediction = Predict(img) 39 | result_queue.put(prediction) 40 | 41 | print("Worker thread stopped.") 42 | 43 | def start_piano(finger): 44 | stop_event = threading.Event() 45 | img_queue = queue.Queue() 46 | result_queue = queue.Queue() 47 | predict_thread = threading.Thread(target=predict_worker, args=(img_queue, result_queue, stop_event)) 48 | predict_thread.start() 49 | 50 | cap = cv2.VideoCapture(0) 51 | with mp_hands.Hands( 52 | min_detection_confidence=0.5, 53 | min_tracking_confidence=0.5) as hands: 54 | while cap.isOpened(): 55 | success, image = cap.read() 56 | if not success: 57 | print("Ignoring empty camera frame.") 58 | continue 59 | image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB) 60 | cols , polys = analyse(image) 61 | image = draw_over_image(image , cols , polys) 62 | cv2.imshow('analyse', image) 63 | key = cv2.waitKey(5) 64 | if key == ord('q'): 65 | cap.release() 66 | cv2.destroyAllWindows() 67 | break 68 | 69 | print(cols , polys) 70 | cnvrt_poly = [Polygon(polygon_coords) for polygon_coords in polys] 71 | prev = None 72 | 73 | cap = cv2.VideoCapture(0) 74 | with mp_hands.Hands( 75 | min_detection_confidence=0.5, 76 | min_tracking_confidence=0.5) as hands: 77 | while cap.isOpened(): 78 | success, image = cap.read() 79 | 80 | if not success: 81 | print("Ignoring empty camera frame.") 82 | continue 83 | image = cv2.cvtColor(image , cv2.COLOR_BGR2RGB) 84 | frame = image.copy() 85 | frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) 86 | image.flags.writeable = False 87 | results = hands.process(image) 88 | image.flags.writeable = True 89 | image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) 90 | 91 | roi = None # Initialize roi outside the loop 92 | 93 | if results.multi_hand_landmarks: 94 | for hand_landmarks in results.multi_hand_landmarks: 95 | mp_drawing.draw_landmarks( 96 | image, hand_landmarks, mp_hands.HAND_CONNECTIONS) 97 | 98 | 99 | for finger_tip_id in [finger]: # Landmark IDs for all five fingers' tips 100 | finger_tip = hand_landmarks.landmark[finger_tip_id] 101 | height, width, _ = image.shape 102 | tip_x, tip_y, tip_z = int(finger_tip.x * width), int(finger_tip.y * height), finger_tip.z 103 | 104 | box_size = int(GLOBAL.BOX_SIZE // 2) # Adjust the size of the box as needed 105 | box_color = (0, 255, 0) # Green color 106 | 107 | # Coordinates of the rectangle 108 | x1, y1 = tip_x - box_size, tip_y - box_size 109 | x2, y2 = tip_x + box_size, tip_y + box_size 110 | 111 | # Draw a square box around the finger tip 112 | cv2.rectangle(image, (x1, y1), (x2, y2), box_color, 2) 113 | 114 | # Crop the region of interest (ROI) 115 | roi = frame[y1:y2, x1:x2] 116 | 117 | # Save the cropped image to the 'touched' folder 118 | color = (0, 0, 255) 119 | touched = False 120 | if roi is not None and roi.shape[0] > 0 and roi.shape[1] > 0: 121 | 122 | img_queue.put(roi) 123 | 124 | try: 125 | prediction = result_queue.get(timeout=0.1) 126 | 127 | except queue.Empty: 128 | prediction = None 129 | 130 | if prediction is not None: 131 | if prediction > 0.5: 132 | color = (0, 255, 0) 133 | touched = True 134 | else: 135 | if touched: 136 | pass 137 | else: 138 | color = (0, 0, 255) 139 | image = cv2.circle(image, (50, 50), 10, color, 20) 140 | 141 | 142 | point = Point(tip_x , tip_y) 143 | for poly in cnvrt_poly: 144 | is_inside = point.within(poly) 145 | 146 | if is_inside: 147 | 148 | text = cnvrt_poly.index(poly) + 1 149 | 150 | cv2.putText(image , str(text) , (100 , 50) , cv2.FONT_HERSHEY_SIMPLEX, 2, color, 2) 151 | if touched: 152 | if text != prev: 153 | sound_path = sounds[text - 1] 154 | sound_thread = threading.Thread(target=play_sound, args=(sound_path,)) 155 | sound_thread.start() 156 | prev = text 157 | else: 158 | prev = None 159 | 160 | break 161 | # Your remaining code 162 | 163 | image = draw_over_image(image , cols , polys) 164 | image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB) 165 | image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) 166 | cv2.imshow('MediaPipe Hands', image) 167 | cv2.imshow('Original', frame) 168 | key = cv2.waitKey(5) 169 | if key == ord('q'): 170 | img_queue.put(None) 171 | stop_event.set() 172 | predict_thread.join() 173 | cap.release() 174 | cv2.destroyAllWindows() 175 | break 176 | 177 | -------------------------------------------------------------------------------- /src/train_model.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from matplotlib import pyplot as plt 3 | import tensorflow as tf 4 | import os 5 | import cv2 6 | from src import GLOBAL 7 | 8 | import tensorflow 9 | from tensorflow.keras.models import Sequential 10 | from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense, Flatten, Dropout 11 | 12 | def start_training(): 13 | gpus = tf.config.experimental.list_physical_devices('GPU') 14 | for gpu in gpus: 15 | tf.config.experimental.set_memory_growth(gpu, True) 16 | 17 | total_samples = len(os.listdir(GLOBAL.TOUCH_FOLDER) + os.listdir(GLOBAL.UNTOUCH_FOLDER)) 18 | data = tf.keras.utils.image_dataset_from_directory("src/training_data" , image_size=(40,40), batch_size=total_samples // 5) 19 | 20 | data_iterator = data.as_numpy_iterator() 21 | batch = data_iterator.next() 22 | 23 | train_size = int(len(data)*.7) 24 | val_size = int(len(data)*.2) 25 | test_size = int(len(data)*.1)+1 26 | 27 | train = data.take(train_size) 28 | val = data.skip(train_size).take(val_size) 29 | test = data.skip(train_size+val_size).take(test_size) 30 | 31 | model = Sequential() 32 | 33 | model.add(Conv2D(32, (3,3), 1, activation='relu', input_shape=(40,40,3))) 34 | model.add(MaxPooling2D()) 35 | 36 | model.add(Conv2D(16, (3,3), 1, activation='relu')) 37 | model.add(MaxPooling2D()) 38 | 39 | model.add(Flatten()) 40 | 41 | model.add(Dense(128, activation='relu')) 42 | model.add(Dense(256, activation='relu')) 43 | model.add(Dense(1, activation='sigmoid')) 44 | 45 | 46 | model.compile('adam', loss=tf.losses.BinaryCrossentropy(), metrics=['accuracy']) 47 | print(model.summary()) 48 | 49 | hist = model.fit(train, epochs=23, validation_data=val) 50 | 51 | fig = plt.figure() 52 | plt.plot(hist.history['loss'], color='teal', label='loss') 53 | plt.plot(hist.history['val_loss'], color='orange', label='val_loss') 54 | fig.suptitle('Loss', fontsize=20) 55 | plt.legend(loc="upper left") 56 | plt.show() 57 | 58 | fig = plt.figure() 59 | plt.plot(hist.history['accuracy'], color='teal', label='accuracy') 60 | plt.plot(hist.history['val_accuracy'], color='orange', label='val_accuracy') 61 | fig.suptitle('Accuracy', fontsize=20) 62 | plt.legend(loc="upper left") 63 | plt.show() 64 | 65 | 66 | model.save('models/touch_detection_model.h5') 67 | -------------------------------------------------------------------------------- /src/training_data/touched/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/src/training_data/touched/.gitkeep -------------------------------------------------------------------------------- /src/training_data/untouched/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Mayuresh1611/Paper-Piano/eafdcfd979865749ad57ac04fa605f1e5efc2a22/src/training_data/untouched/.gitkeep --------------------------------------------------------------------------------