├── .gitignore ├── LICENSE ├── README.md ├── assets └── Screenshot 2024-08-28 164725.png ├── main.py ├── mlforkidsimages.py └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | env/ 2 | __pycache__/ 3 | 4 | *.jpg 5 | 6 | api.py 7 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | # MIT License 2 | 3 | Copyright (c) 2024 Halip26 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | 7 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Finger Languange Detector using MLforkid API 2 | 3 | The project is able to recognize finger gestures from camera captures. 4 | 5 | ## Preview 6 | 7 | ![Screenshot](assets/Screenshot%202024-08-28%20164725.png) 8 | 9 | ## **Machine Learning: A Comprehensive Explanation** 10 | 11 | **Introduction** 12 | Machine Learning is a subset of artificial intelligence that allows systems to learn from data without being explicitly programmed. In simpler terms, it's the ability of machines to learn from experience, identify patterns in data, and make decisions with minimal human intervention. 13 | 14 | ## **Why is Machine Learning Important?** 15 | 16 | Machine Learning has revolutionized many industries, from healthcare to finance. Its ability to analyze vast amounts of data and uncover complex patterns makes it invaluable for tasks such as: 17 | 18 | * **Healthcare:** Disease diagnosis, drug discovery 19 | * **Finance:** Fraud detection, stock market prediction 20 | * **Marketing:** Customer segmentation, product recommendation 21 | * **Transportation:** Self-driving cars 22 | * 23 | 24 | ## **Core Concepts** 25 | 26 | * **Data** 27 | Data is the fuel for Machine Learning algorithms. The quality of data directly impacts the performance of a model. Data preprocessing involves tasks like: 28 | ***Cleaning:** Handling missing values, outliers, and inconsistencies. 29 | * **Transformation:** Converting data into a suitable format for the chosen algorithm. 30 | * **Feature scaling:** Scaling numerical features to a common range. 31 | * **Algorithms** 32 | Algorithms are the instructions that a machine learning system follows to learn from data. There are two main types of learning: 33 | ***Supervised Learning:** Algorithms learn from labeled data (e.g., images of cats and dogs with labels "cat" or "dog"). Examples include: Linear Regression, Logistic Regression, Decision Trees, Random Forest, Support Vector Machines. 34 | * **Unsupervised Learning:** Algorithms find patterns in unlabeled data. Examples include: K-Means Clustering, Hierarchical Clustering, PCA. 35 | * **Model** 36 | A model is a mathematical representation of the data that has been learned. Models are used to make predictions or classifications on new, unseen data. 37 | * **Libraries and Tools** 38 | Python is the most popular programming language for Machine Learning. Popular libraries include: 39 | ***Scikit-learn:** Provides various machine learning algorithms. 40 | * **Pandas:** For data manipulation. 41 | ***NumPy:** For numerical computations. 42 | * **Matplotlib, Seaborn:** For data visualization. 43 | * **TensorFlow, Keras:** For deep learning. 44 | -------------------------------------------------------------------------------- /assets/Screenshot 2024-08-28 164725.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RKNITH/finger-gesture-python/1d402958350f07fe0ae420d654a7e0956770631f/assets/Screenshot 2024-08-28 164725.png -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | from api import API_TOKEN 2 | from mlforkidsimages import MLforKidsImageProject 3 | 4 | # this one for capture photo from camera 5 | import cv2 6 | 7 | # to call the image file in console/terminal 8 | import sys 9 | 10 | # Treat this key like a password & keep it secret 11 | key = API_TOKEN 12 | 13 | # Train your model 14 | myproject = MLforKidsImageProject(key) 15 | myproject.train_model() 16 | 17 | 18 | # Capture an image using the camera 19 | def capture_image(): 20 | cam = cv2.VideoCapture(0) 21 | cv2.namedWindow("Capture Image") 22 | 23 | while True: 24 | ret, frame = cam.read() 25 | if not ret: 26 | print("Failed to grab frame") 27 | break 28 | cv2.imshow("Capture Image", frame) 29 | 30 | # Press 'SPACE' to capture the image 31 | if cv2.waitKey(1) & 0xFF == ord(" "): 32 | image_path = "assets/captured_image.jpg" 33 | cv2.imwrite(image_path, frame) 34 | print(f"Image saved to {image_path}") 35 | break 36 | 37 | cam.release() 38 | cv2.destroyAllWindows() 39 | return image_path 40 | 41 | 42 | image_path = capture_image() 43 | 44 | demo = myproject.prediction(image_path) 45 | 46 | label = demo["class_name"] 47 | confidence = demo["confidence"] 48 | 49 | # show the result on console 50 | print("Result: '%s' with %d%% confidence" % (label, confidence)) 51 | -------------------------------------------------------------------------------- /mlforkidsimages.py: -------------------------------------------------------------------------------- 1 | import os 2 | os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' 3 | os.environ['TF_USE_LEGACY_KERAS'] = '1' 4 | import tensorflow as tf 5 | tf.get_logger().setLevel('ERROR') 6 | 7 | import tensorflow_hub as hub 8 | from tensorflow.keras.preprocessing import image 9 | from tensorflow.keras.preprocessing.image import ImageDataGenerator 10 | from tensorflow.keras import Sequential 11 | from tensorflow.keras.layers import Dropout, Dense, Rescaling 12 | 13 | import numpy as np 14 | import urllib.request, urllib.error, json 15 | from time import sleep 16 | 17 | # 18 | # Helper class for training an image classifier using training data 19 | # from the Machine Learning for Kids website. 20 | # 21 | class MLforKidsImageProject: 22 | 23 | IMAGESIZE=(224,224) 24 | INPUTLAYERSIZE=IMAGESIZE + (3,) 25 | 26 | # scratchkey is the secret API key that allows access to training 27 | # data from a single project on the MLforKids website 28 | def __init__(self, scratchkey: str): 29 | # register custom HTTP handler 30 | opener = urllib.request.build_opener(MLforKidsHTTP()) 31 | urllib.request.install_opener(opener) 32 | 33 | print("MLFORKIDS: Downloading information about your machine learning project") 34 | self.scratchkey = scratchkey 35 | try: 36 | apiurl = self.__switchToTemporarySite("https://machinelearningforkids.co.uk/api/scratch/" + scratchkey + "/train") 37 | with urllib.request.urlopen(apiurl) as url: 38 | self.__downloaded_training_images_list = json.loads(url.read().decode()) 39 | except urllib.error.HTTPError: 40 | raise RuntimeError("Unable to retrieve machine learning project - please check that the key is correct") 41 | 42 | # Generates a name for the local cache file where the downloaded training 43 | # image is saved. An image file extension is required, otherwise it will 44 | # be ignored by ImageDataGenerator. 45 | def __get_fname(self, trainingitem): 46 | extension = ".png" if trainingitem["imageurl"].lower().endswith(".png") else ".jpg" 47 | return trainingitem["id"] + extension 48 | 49 | # Downloads all of the training images for this project, and sets up an 50 | # ImageDataGenerator against the folder where they have been downloaded 51 | def __get_training_images_generator(self): 52 | print("MLFORKIDS: Getting your training images to use to train your machine learning model") 53 | cachedir = "~/.keras/" 54 | cachelocation = os.path.join("datasets", "mlforkids", self.scratchkey) 55 | projectcachedir = str(os.path.expanduser(os.path.join(cachedir, cachelocation))) 56 | for trainingitem in self.__downloaded_training_images_list: 57 | try: 58 | tf.keras.utils.get_file(origin=self.__switchToTemporarySite(trainingitem["imageurl"]), 59 | cache_dir=cachedir, 60 | cache_subdir=os.path.join(cachelocation, trainingitem["label"]), 61 | fname=self.__get_fname(trainingitem)) 62 | # avoid common rate-limiting errors by pausing 63 | # for a quarter-second between each download 64 | sleep(0.25) 65 | except Exception as downloaderr: 66 | print("ERROR: Unable to download training image from", trainingitem["imageurl"]) 67 | print(downloaderr) 68 | print("ERROR: Skipping training image and continuing without it", trainingitem["imageurl"]) 69 | return ImageDataGenerator().flow_from_directory(str(projectcachedir), 70 | target_size=MLforKidsImageProject.IMAGESIZE) 71 | 72 | # Creates a lookup table for the classes that this project is being trained 73 | # to recognize. 74 | # TODO : dumb implementation - should rewrite 75 | def __get_class_lookup(self, training_image_data): 76 | class_labels = [None]*training_image_data.num_classes 77 | class_names = training_image_data.class_indices.keys() 78 | for classname in class_names: 79 | class_labels[training_image_data.class_indices[classname]] = classname 80 | return class_labels 81 | 82 | # Defines a simple image classifier based on a mobilenet model from TensorFlow hub 83 | def __define_model(self): 84 | print("MLFORKIDS: Defining the layers to include in your neural network") 85 | model = Sequential([ 86 | # input layer is resizing all images to save having to do that in a manual pre-processing step 87 | Rescaling(1/127, input_shape=MLforKidsImageProject.INPUTLAYERSIZE), 88 | # using an existing pre-trained model as an untrainable main layer 89 | hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_140_224/classification/5"), 90 | # 91 | Dropout(rate=0.2), 92 | # 93 | Dense(self.num_classes) 94 | ]) 95 | model.build((None,) + MLforKidsImageProject.INPUTLAYERSIZE) 96 | 97 | # model compile parameters copied from tutorial at https://www.tensorflow.org/hub/tutorials/tf2_image_retraining 98 | model.compile( 99 | optimizer=tf.keras.optimizers.SGD(learning_rate=0.005, momentum=0.9), 100 | loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True, label_smoothing=0.1), 101 | metrics=['accuracy']) 102 | 103 | return model 104 | 105 | # Runs the model fit function to train the tl model 106 | def __train_model(self, trainingimagesdata): 107 | print("MLFORKIDS: Starting the training of your machine learning model") 108 | if trainingimagesdata.batch_size > trainingimagesdata.samples: 109 | trainingimagesdata.batch_size = trainingimagesdata.samples 110 | steps_per_epoch = trainingimagesdata.samples // trainingimagesdata.batch_size 111 | epochs = 8 112 | if trainingimagesdata.samples > 55: 113 | epochs = 15 114 | self.ml_model.fit(trainingimagesdata, epochs=epochs, steps_per_epoch=steps_per_epoch, verbose=0) 115 | print("MLFORKIDS: Model training complete") 116 | 117 | 118 | # Cloudflare is currently blocking access to the Machine Learning for Kids API 119 | # from non-browser user agents 120 | # While I raise this with them to get this unblocked, switching to this 121 | # temporary URL should avoid the problem 122 | # 123 | # TODO: remove this function as soon as Cloudflare have 124 | # stopped breaking Python apps 125 | # 126 | def __switchToTemporarySite(self, url): 127 | return url.replace("https://machinelearningforkids.co.uk/api/scratch/", 128 | "https://mlforkids-api.j8clybxvjr0.us-south.codeengine.appdomain.cloud/api/scratch/") 129 | 130 | 131 | # 132 | # public methods 133 | # 134 | 135 | # Fetches the training data for this project, and uses it to train a machine learning model 136 | def train_model(self): 137 | training_images = self.__get_training_images_generator() 138 | self.num_classes = training_images.num_classes 139 | self.ml_class_names = self.__get_class_lookup(training_images) 140 | self.ml_model = self.__define_model() 141 | self.__train_model(training_images) 142 | 143 | # Returns a prediction for the image at the specified location 144 | def prediction(self, image_location: str): 145 | if hasattr(self, "ml_model") == False: 146 | raise RuntimeError("Machine learning model has not been trained for this project") 147 | testimg = image.load_img(image_location, target_size=MLforKidsImageProject.IMAGESIZE) 148 | testimg = image.img_to_array(testimg) 149 | testimg = np.expand_dims(testimg, axis=0) 150 | predictions = self.ml_model.predict(testimg) 151 | topprediction = predictions[0] 152 | topanswer = np.argmax(topprediction) 153 | return { 154 | "class_name": self.ml_class_names[topanswer], 155 | "confidence": 100 * np.max(tf.nn.softmax(topprediction)) 156 | } 157 | 158 | 159 | # 160 | # Helper class for making HTTP requests to fetch training images 161 | # for machine learning projects 162 | # 163 | # It adds a user-agent header so that when scraping images from 164 | # third-party websites, the Python code correctly identifies 165 | # itself, so that appropriate rate-limiting can be applied. 166 | # 167 | class MLforKidsHTTP(urllib.request.HTTPHandler): 168 | def http_request(self, req): 169 | req.headers["User-Agent"] = "MachineLearningForKidsPythonBot/1.0" 170 | return super().http_request(req) 171 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy==1.26.4 2 | pillow==10.3.0 3 | scipy==1.13.0 4 | tensorflow==2.16.1 5 | tensorflow-hub==0.16.1 6 | tf_keras==2.16.0 7 | opencv-python --------------------------------------------------------------------------------