├── repo └── .gitkeep ├── requirements.txt ├── .gitignore ├── images ├── installation-web-ui.jpg └── deepfacelive-scripts-tab.jpg ├── README.md ├── install.py └── scripts ├── ddetailerutils.py ├── dflutils.py ├── deepfacelab.py ├── deepfacelive.py └── command.py /repo/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /repo/dflive/ 2 | /repo/dflab/ 3 | /notebooks/ 4 | __pycache__ 5 | .idea 6 | .ipynb_checkpoints -------------------------------------------------------------------------------- /images/installation-web-ui.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/idinkov/sd-deepface-1111/HEAD/images/installation-web-ui.jpg -------------------------------------------------------------------------------- /images/deepfacelive-scripts-tab.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/idinkov/sd-deepface-1111/HEAD/images/deepfacelive-scripts-tab.jpg -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## DeepFaceLab and DeepFaceLive on Stable Diffusion 2 | 3 | This is an implementation of iperov's DeepFaceLab and DeepFaceLive in Stable Diffusion Web UI by AUTOMATIC1111. 4 | 5 | Note: DeepFaceLive is currently fully implemented while DeepFaceLab is in the works. 6 | 7 | This can be installed as a plugin for https://github.com/AUTOMATIC1111/stable-diffusion-webui 8 | 9 | DeepFaceLive is implemented as a script which can be accessed on txt2img and img2img tabs. 10 | 11 | ![](images/deepfacelive-scripts-tab.jpg) 12 | 13 | DeepFaceLab has a separate tab and controls to manage workspaces and train custom models. 14 | 15 | ## Use cases 16 | 17 | - You can easily face swap any face in stable diffusion with the one that you want, with a combination of DeepFaceLab to create your model and DeepFaceLive to implement the model to be used in stable diffusion generating process. 18 | - Enhance and make more stable and person specific the output of faces in stable diffusion. 19 | - You can also use DeepFaceLab to train your own models and use them in DeepFaceLive to generate faces in stable diffusion. 20 | 21 | ## Examples 22 | 23 | 24 | ## Table of contents 25 | 26 | 1) [Introduction](#1-introduction) 27 | 2) [Installation](#2-installation) 28 | 3) [Usage of DeepFaceLive](#3-usage-of-deepfacelive) 29 | 4) [Developers Documentation](#4-developers-documentation) 30 | 5) [Checkpoints Suggestions](#5-checkpoints-suggestions) 31 | 6) [Credits](#6-credits) 32 | 7) [Training your own model with DeepFaceLab](#7-training-your-own-model-with-deepfacelab) 33 | 8) [Q&A](#8-qa) 34 | 9) [License](#9-license) 35 | 36 | ## 1. Introduction 37 | 38 | This is implementation of DeepFaceLab and DeepFaceLive in Stable Diffusion Web UI by AUTOMATIC1111. 39 | 40 | DeepFaceLab is a tool that utilizes machine learning to replace faces in videos and photos. With DeepFaceLab you can train your own face model and use it in DeepFaceLive to generate faces in stable diffusion. 41 | 42 | DeepFaceLab will output a SAEHD model which is then exported to ~(700MB) .dfm file which contains the trained face needed for DeepFaceLive to work as a script. 43 | 44 | Note: It is not required to train your own model, you can use the pre-trained models provided by iperov. They will be seen in the dropdown in the UI for model selection and will be downloaded when used. List with the available faces can be seen in the readme of iperov repo here: https://github.com/iperov/DeepFaceLive 45 | 46 | ## 2. Installation 47 | 48 | ### 2.1. Requirements 49 | 50 | - Python 3.10 51 | - [Stable Diffusion Web UI by AUTOMATIC1111](https://github.com/AUTOMATIC1111/stable-diffusion-webui) 52 | 53 | ### 2.2. Installation through Web UI 54 | 55 | After you have installed and launched AUTOMATIC1111 in the WebUI you need to select "Extensions" tab and click on "Install from URL" tab. 56 | 57 | There in the URL for extension's git repository you need to paste the current repo URL: 58 | 59 | `https://github.com/idinkov/sd-deepface-1111` 60 | 61 | And click on "Install" button. 62 | 63 | After that you need to refresh the UI or restart it and you should see the new tab for DeepFaceLab. And the new script for DeepFaceLive. 64 | 65 | ![](images/installation-web-ui.jpg) 66 | 67 | ## 3. Usage of DeepFaceLive 68 | 69 | ## 4. Developers Documentation 70 | 71 | ### 4.1. Python packages 72 | - tqdm 73 | - numpy 74 | - opencv-python 75 | - opencv-contrib-python 76 | - numexpr 77 | - h5py 78 | - ffmpeg-python 79 | - scikit-image 80 | - scipy 81 | - colorama 82 | - tensorflow 83 | - pyqt5 84 | - tf2onnx 85 | - onnxruntime or onnxruntime-gpu 86 | - protobuf==3.20.3 87 | 88 | ## 4.1.1. Developer Notes 89 | 90 | In order to implement DeepFaceLab which at the moment of writing uses Python 3.6 and DeepFaceLive which uses Python 3.7 the following modifications had to be made in order to modernize it to run on Python 3.10 91 | 92 | - Implemented xlib in DeepFaceLive had been updated for Collections import to work 93 | 94 | 95 | ## 5. Checkpoints Suggestions 96 | 97 | I have found the following stable diffusion checkpoints to produce good results rendering humans: 98 | * [Dreamlike Photoreal 2.0](https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0) 99 | 100 | ## 6. Credits 101 | 102 | - Stable Diffusion Web UI by AUTOMATIC1111: https://github.com/AUTOMATIC1111/stable-diffusion-webui 103 | - DeepFaceLab by iperov: https://github.com/iperov/DeepFaceLab 104 | - DeepFaceLive by iperov: https://github.com/iperov/DeepFaceLive 105 | - Detection Detailer by dustysys: https://github.com/dustysys/ddetailer 106 | 107 | ## 7. Training your own model with DeepFaceLab 108 | 109 | ## 8. Q&A 110 | 111 | ## 9. License 112 | 113 | -------------------------------------------------------------------------------- /install.py: -------------------------------------------------------------------------------- 1 | import os 2 | import subprocess 3 | import sys 4 | from pathlib import Path 5 | from typing import Tuple, Optional 6 | from packaging import version 7 | import importlib.metadata 8 | 9 | import launch 10 | from launch import is_installed, run, run_pip 11 | 12 | # Determine whether to skip installation based on command-line arguments 13 | try: 14 | skip_install = getattr(launch.args, "skip_install", False) 15 | except AttributeError: 16 | skip_install = getattr(launch, "skip_install", False) 17 | 18 | python = sys.executable 19 | 20 | def comparable_version(version_str: str) -> Tuple[int, ...]: 21 | """Convert a version string into a tuple of integers for comparison.""" 22 | return tuple(map(int, version_str.split("."))) 23 | 24 | def get_installed_version(package: str) -> Optional[str]: 25 | """Retrieve the installed version of a package, if available.""" 26 | try: 27 | return importlib.metadata.version(package) 28 | except importlib.metadata.PackageNotFoundError: 29 | return None 30 | 31 | def install_uddetailer(): 32 | """Install and manage dependencies for the 'uddetailer' component.""" 33 | if not is_installed("mim"): 34 | run_pip("install -U openmim", desc="Installing openmim") 35 | 36 | # Ensure minimum requirements are met 37 | if not is_installed("mediapipe"): 38 | run_pip('install protobuf>=3.20', desc="Installing protobuf") 39 | run_pip('install mediapipe>=0.10.3', desc="Installing mediapipe") 40 | 41 | torch_version = get_installed_version("torch") 42 | legacy = torch_version and comparable_version(torch_version)[0] < 2 43 | 44 | # Check versions and manage installations for mmdet and mmcv 45 | mmdet_version = get_installed_version("mmdet") 46 | mmdet_v3 = mmdet_version and version.parse(mmdet_version) >= version.parse("3.0.0") 47 | 48 | if not is_installed("mmdet") or (legacy and mmdet_v3) or (not legacy and not mmdet_v3): 49 | if legacy and mmdet_v3: 50 | print("Uninstalling mmdet and mmengine...") 51 | run(f'"{python}" -m pip uninstall -y mmdet mmengine', live=True) 52 | run(f'"{python}" -m mim install mmcv-full', desc="Installing mmcv-full") 53 | run_pip("install mmdet==2.28.2", desc="Installing mmdet") 54 | else: 55 | if not mmdet_v3: 56 | print("Uninstalling mmdet, mmcv, and mmcv-full...") 57 | run(f'"{python}" -m pip uninstall -y mmdet mmcv mmcv-full', live=True) 58 | print("Installing mmcv, mmdet, and mmengine...") 59 | if not is_installed("mmengine"): 60 | run_pip("install mmengine==0.8.5", desc="Installing mmengine") 61 | if version.parse(torch_version) >= version.parse("2.1.0"): 62 | run(f'"{python}" -m mim install mmcv~=2.1.0', desc="Installing mmcv 2.1.0") 63 | else: 64 | run(f'"{python}" -m mim install mmcv~=2.0.0', desc="Installing mmcv") 65 | run(f'"{python}" -m mim install -U mmdet>=3.0.0', desc="Installing mmdet") 66 | run_pip("install mmdet>=3", desc="Installing mmdet") 67 | 68 | # Verify mmcv and mmengine versions 69 | mmcv_version = get_installed_version("mmcv") 70 | if mmcv_version and version.parse(mmcv_version) >= version.parse("2.0.1"): 71 | print(f"Your mmcv version {mmcv_version} may not work with mmyolo.") 72 | print("Consider fixing the version restriction manually.") 73 | 74 | mmengine_version = get_installed_version("mmengine") 75 | if mmengine_version: 76 | if version.parse(mmengine_version) >= version.parse("0.9.0"): 77 | print(f"Your mmengine version {mmengine_version} may not work on Windows.") 78 | print("Install mmengine 0.8.5 manually or use an updated version of bitsandbytes.") 79 | else: 80 | print(f"Your mmengine version is {mmengine_version}") 81 | 82 | if not legacy and not is_installed("mmyolo"): 83 | run(f'"{python}" -m pip install mmyolo', desc="Installing mmyolo") 84 | 85 | # Install additional requirements from requirements.txt 86 | req_file = os.path.join(os.path.dirname(os.path.realpath(__file__)), "requirements.txt") 87 | if os.path.exists(req_file): 88 | mainpackage = 'sd-deepface-1111' 89 | with open(req_file) as file: 90 | for package in file: 91 | package = package.strip() 92 | try: 93 | if '==' in package: 94 | package_name, package_version = package.split('==') 95 | installed_version = get_installed_version(package_name) 96 | if installed_version != package_version: 97 | run_pip(f"install -U {package}", desc=f"{mainpackage} requirement: updating {package_name} to {package_version}") 98 | elif '>=' in package: 99 | package_name, package_version = package.split('>=') 100 | installed_version = get_installed_version(package_name) 101 | if not installed_version or comparable_version(installed_version) < comparable_version(package_version): 102 | run_pip(f"install -U {package}", desc=f"{mainpackage} requirement: updating {package_name} to {package_version}") 103 | elif not is_installed(package): 104 | run_pip(f"install {package}", desc=f"{mainpackage} requirement: {package}") 105 | except Exception as e: 106 | print(f"Error installing {package}: {e}") 107 | 108 | def install(): 109 | """Install essential packages for DeepFaceLab and related tools.""" 110 | packages = { 111 | "tqdm": "requirements for DeepFaceLab - tqdm", 112 | "numpy": "requirements for DeepFaceLab - numpy", 113 | "numexpr": "requirements for DeepFaceLab - numexpr", 114 | "h5py": "requirements for DeepFaceLab - h5py", 115 | "opencv-python": "requirements for DeepFaceLab - opencv-python", 116 | "opencv-contrib-python": "requirements for DeepFaceLab - opencv-contrib-python", 117 | "ffmpeg-python": "requirements for DeepFaceLab - ffmpeg-python", 118 | "scikit-image": "requirements for DeepFaceLab - scikit-image", 119 | "scipy": "requirements for DeepFaceLab - scipy", 120 | "colorama": "requirements for DeepFaceLab - colorama", 121 | "tensorflow": "requirements for DeepFaceLab - tensorflow", 122 | "pyqt5": "requirements for DeepFaceLab - pyqt5", 123 | "tf2onnx": "requirements for DeepFaceLab - tf2onnx", 124 | "onnxruntime": "requirements for DeepFaceLab - onnxruntime", 125 | "onnxruntime-gpu": "requirements for DeepFaceLab - onnxruntime-gpu==1.12.1", 126 | "protobuf": "requirements for DeepFaceLab - protobuf==3.20.3", 127 | } 128 | 129 | for package, desc in packages.items(): 130 | if not is_installed(package) or (package == "onnxruntime-gpu" and get_installed_version(package) != '1.12.1') or (package == "protobuf" and get_installed_version(package) != '3.20.3'): 131 | version_specifier = "" if package != "onnxruntime-gpu" and package != "protobuf" else "==1.12.1" if package == "onnxruntime-gpu" else "==3.20.3" 132 | run_pip(f"install {package}{version_specifier}", desc=desc) 133 | 134 | def checkout_git_commit(repo_name: str, commit: str, output_folder: str): 135 | """Clone a GitHub repository and check out a specific commit.""" 136 | if not os.path.isdir(output_folder): 137 | subprocess.run(['git', 'clone', f'https://github.com/{repo_name}.git', output_folder]) 138 | 139 | os.chdir(output_folder) 140 | subprocess.run(['git', 'checkout', commit]) 141 | 142 | # Main script execution 143 | if not skip_install: 144 | install() 145 | install_uddetailer() 146 | 147 | script_path = Path(os.path.dirname(os.path.abspath(__file__))) / "repo/" 148 | checkout_git_commit('idinkov/DeepFaceLive', 'af8396925cccc1d3f02867e16b8929060c3ebc5f', str(script_path / "dflive")) 149 | -------------------------------------------------------------------------------- /scripts/ddetailerutils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import cv2 4 | from PIL import Image 5 | import numpy as np 6 | import gradio as gr 7 | 8 | from modules import processing, images 9 | from modules import scripts, script_callbacks, shared, devices, modelloader 10 | from modules.processing import Processed, StableDiffusionProcessingTxt2Img, StableDiffusionProcessingImg2Img 11 | from modules.shared import opts, cmd_opts, state 12 | from modules.sd_models import model_hash 13 | from modules.paths import models_path 14 | from basicsr.utils.download_util import load_file_from_url 15 | 16 | dd_models_path = os.path.join(models_path, "mmdet") 17 | 18 | def list_models(model_path): 19 | model_list = modelloader.load_models(model_path=model_path, ext_filter=[".pth"]) 20 | 21 | def modeltitle(path, shorthash): 22 | abspath = os.path.abspath(path) 23 | 24 | if abspath.startswith(model_path): 25 | name = abspath.replace(model_path, '') 26 | else: 27 | name = os.path.basename(path) 28 | 29 | if name.startswith("\\") or name.startswith("/"): 30 | name = name[1:] 31 | 32 | shortname = os.path.splitext(name.replace("/", "_").replace("\\", "_"))[0] 33 | 34 | return f'{name} [{shorthash}]', shortname 35 | 36 | models = [] 37 | for filename in model_list: 38 | h = model_hash(filename) 39 | title, short_model_name = modeltitle(filename, h) 40 | models.append(title) 41 | 42 | return models 43 | 44 | class DetectionDetailerScript(): 45 | def run(self, 46 | p, 47 | model, 48 | model_name, 49 | init_image, 50 | dd_conf_a = 30, 51 | dd_dilation_factor_a = 4, 52 | dd_offset_x_a = 0, 53 | dd_offset_y_a = 0): 54 | 55 | new_image = p 56 | results_a = inference(init_image, model, model_name, dd_conf_a / 100.0) 57 | masks_a = create_segmasks(results_a) 58 | masks_a = dilate_masks(masks_a, dd_dilation_factor_a, 1) 59 | masks_a = offset_masks(masks_a, dd_offset_x_a, dd_offset_y_a) 60 | output_image = init_image 61 | gen_count = len(masks_a) 62 | if (gen_count > 0): 63 | #state.job_count += gen_count 64 | new_image.init_images = [init_image] 65 | new_image.batch_size = 1 66 | new_image.n_iter = 1 67 | for i in range(gen_count): 68 | new_image.image_mask = masks_a[i] 69 | processed = processing.process_images(p) 70 | new_image.seed = processed.seed + 1 71 | new_image.init_images = processed.images 72 | 73 | if (gen_count > 0): 74 | output_image = processed.images[0] 75 | 76 | return output_image 77 | 78 | import mmcv 79 | from mmdet.apis import (inference_detector, 80 | init_detector) 81 | 82 | 83 | def modeldataset(model_shortname): 84 | path = modelpath(model_shortname) 85 | if ("mmdet" in path and "segm" in path): 86 | dataset = 'coco' 87 | else: 88 | dataset = 'bbox' 89 | return dataset 90 | 91 | def modelpath(model_shortname): 92 | return dd_models_path + "/" + model_shortname 93 | 94 | def is_allblack(mask): 95 | cv2_mask = np.array(mask) 96 | return cv2.countNonZero(cv2_mask) == 0 97 | 98 | def bitwise_and_masks(mask1, mask2): 99 | cv2_mask1 = np.array(mask1) 100 | cv2_mask2 = np.array(mask2) 101 | cv2_mask = cv2.bitwise_and(cv2_mask1, cv2_mask2) 102 | mask = Image.fromarray(cv2_mask) 103 | return mask 104 | 105 | def subtract_masks(mask1, mask2): 106 | cv2_mask1 = np.array(mask1) 107 | cv2_mask2 = np.array(mask2) 108 | cv2_mask = cv2.subtract(cv2_mask1, cv2_mask2) 109 | mask = Image.fromarray(cv2_mask) 110 | return mask 111 | 112 | def dilate_masks(masks, dilation_factor, iter=1): 113 | if dilation_factor == 0: 114 | return masks 115 | dilated_masks = [] 116 | kernel = np.ones((dilation_factor, dilation_factor), np.uint8) 117 | for i in range(len(masks)): 118 | cv2_mask = np.array(masks[i]) 119 | dilated_mask = cv2.dilate(cv2_mask, kernel, iter) 120 | dilated_masks.append(Image.fromarray(dilated_mask)) 121 | return dilated_masks 122 | 123 | def offset_masks(masks, offset_x, offset_y): 124 | if (offset_x == 0 and offset_y == 0): 125 | return masks 126 | offset_masks = [] 127 | for i in range(len(masks)): 128 | cv2_mask = np.array(masks[i]) 129 | offset_mask = cv2_mask.copy() 130 | offset_mask = np.roll(offset_mask, -offset_y, axis=0) 131 | offset_mask = np.roll(offset_mask, offset_x, axis=1) 132 | 133 | offset_masks.append(Image.fromarray(offset_mask)) 134 | return offset_masks 135 | 136 | def combine_masks(masks): 137 | initial_cv2_mask = np.array(masks[0]) 138 | combined_cv2_mask = initial_cv2_mask 139 | for i in range(1, len(masks)): 140 | cv2_mask = np.array(masks[i]) 141 | combined_cv2_mask = cv2.bitwise_or(combined_cv2_mask, cv2_mask) 142 | 143 | combined_mask = Image.fromarray(combined_cv2_mask) 144 | return combined_mask 145 | 146 | def create_segmasks(results): 147 | segms = results[1] 148 | segmasks = [] 149 | for i in range(len(segms)): 150 | cv2_mask = segms[i].astype(np.uint8) * 255 151 | mask = Image.fromarray(cv2_mask) 152 | segmasks.append(mask) 153 | 154 | return segmasks 155 | 156 | def get_device(): 157 | device_id = shared.cmd_opts.device_id 158 | if device_id is not None: 159 | cuda_device = f"cuda:{device_id}" 160 | else: 161 | cuda_device = "cpu" 162 | return cuda_device 163 | 164 | def inference(image, model, modelname, conf_thres): 165 | path = modelpath(modelname) 166 | if ("mmdet" in path and "bbox" in path): 167 | results = inference_mmdet_bbox(image, model, modelname, conf_thres) 168 | elif ("mmdet" in path and "segm" in path): 169 | results = inference_mmdet_segm(image, model, modelname, conf_thres) 170 | return results 171 | 172 | def preload_ddetailer_model(modelname): 173 | model_checkpoint = modelpath(modelname) 174 | model_config = os.path.splitext(model_checkpoint)[0] + ".py" 175 | model_device = get_device() 176 | return init_detector(model_config, model_checkpoint, device=model_device) 177 | 178 | def inference_mmdet_segm(image, model, modelname, conf_thres): 179 | mmdet_results = inference_detector(model, np.array(image)) 180 | bbox_results, segm_results = mmdet_results 181 | dataset = modeldataset(modelname) 182 | labels = [ 183 | np.full(bbox.shape[0], i, dtype=np.int32) 184 | for i, bbox in enumerate(bbox_results) 185 | ] 186 | n, m = bbox_results[0].shape 187 | if (n == 0): 188 | return [[], []] 189 | labels = np.concatenate(labels) 190 | bboxes = np.vstack(bbox_results) 191 | segms = mmcv.concat_list(segm_results) 192 | filter_inds = np.where(bboxes[:, -1] > conf_thres)[0] 193 | results = [[], []] 194 | for i in filter_inds: 195 | results[0].append(bboxes[i]) 196 | results[1].append(segms[i]) 197 | 198 | return results 199 | 200 | def inference_mmdet_bbox(image, model, modelname, conf_thres): 201 | results = inference_detector(model, np.array(image)) 202 | cv2_image = np.array(image) 203 | cv2_image = cv2_image[:, :, ::-1].copy() 204 | cv2_gray = cv2.cvtColor(cv2_image, cv2.COLOR_BGR2GRAY) 205 | 206 | segms = [] 207 | for (x0, y0, x1, y1, conf) in results[0]: 208 | cv2_mask = np.zeros((cv2_gray.shape), np.uint8) 209 | cv2.rectangle(cv2_mask, (int(x0), int(y0)), (int(x1), int(y1)), 255, -1) 210 | cv2_mask_bool = cv2_mask.astype(bool) 211 | segms.append(cv2_mask_bool) 212 | 213 | n, m = results[0].shape 214 | if (n == 0): 215 | return [[], []] 216 | bboxes = np.vstack(results[0]) 217 | filter_inds = np.where(bboxes[:, -1] > conf_thres)[0] 218 | results = [[], []] 219 | for i in filter_inds: 220 | results[0].append(bboxes[i]) 221 | results[1].append(segms[i]) 222 | 223 | return results -------------------------------------------------------------------------------- /scripts/dflutils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import shutil 3 | from pathlib import Path 4 | from modules import shared, paths, script_callbacks 5 | 6 | class DflFiles: 7 | 8 | @staticmethod 9 | def folder_exists(dir_path): 10 | """ 11 | Check whether a directory exists. 12 | 13 | Returns True if the directory exists, False otherwise. 14 | """ 15 | return os.path.exists(dir_path) and os.path.isdir(dir_path) 16 | 17 | @staticmethod 18 | def create_folder(path): 19 | """Create a folder at the specified path""" 20 | os.makedirs(path, exist_ok=True) 21 | 22 | @staticmethod 23 | def delete_folder(path): 24 | """Delete a folder and all its contents at the specified path""" 25 | os.removedirs(path) 26 | 27 | @staticmethod 28 | def empty_folder(path): 29 | """Clear the contents of a folder at the specified path""" 30 | for file_name in os.listdir(path): 31 | file_path = os.path.join(path, file_name) 32 | if os.path.isfile(file_path): 33 | os.remove(file_path) 34 | 35 | @staticmethod 36 | def create_empty_file(path): 37 | """Create an empty file at the specified path""" 38 | open(path, 'a').close() 39 | 40 | @staticmethod 41 | def delete_file(path): 42 | """Delete a file at the specified path""" 43 | os.remove(path) 44 | 45 | @staticmethod 46 | def move_file(src, dst): 47 | """Move a file from the source path to the destination path""" 48 | os.replace(src, dst) 49 | 50 | @staticmethod 51 | def get_files_from_dir(base_dir, extension_list, two_dimensions=False): 52 | """Return a list of file names in a directory with a matching file extension""" 53 | files = [] 54 | for v in Path(base_dir).iterdir(): 55 | if v.suffix in extension_list and not v.name.startswith('.ipynb'): 56 | if two_dimensions: 57 | files.append([v.name, v.name]) 58 | else: 59 | files.append(v.name) 60 | return files 61 | 62 | @staticmethod 63 | def get_folder_names_in_dir(base_dir): 64 | """Return a list of folder names in a directory""" 65 | folders = [] 66 | for v in Path(base_dir).iterdir(): 67 | if v.is_dir() and not v.name.startswith('.ipynb'): 68 | folders.append(v.name) 69 | return folders 70 | 71 | @staticmethod 72 | def extract_archive(archive_path, dest_dir, dir_name): 73 | """ 74 | Extract an archive file to a directory with a specified name. 75 | 76 | The extracted directory will be created inside the destination directory with the specified name. 77 | If the name is taken, a suffix will be added to create a unique name. 78 | 79 | Returns the full path of the directory where the contents were extracted. 80 | """ 81 | suffix = '' 82 | extracted_dir_name = dir_name 83 | while os.path.exists(os.path.join(dest_dir, extracted_dir_name + suffix)): 84 | if not suffix: 85 | suffix = 1 86 | else: 87 | suffix += 1 88 | extracted_dir_name = f'{dir_name}_{suffix}' 89 | 90 | extracted_dir_path = os.path.join(dest_dir, extracted_dir_name) 91 | os.makedirs(extracted_dir_path, exist_ok=True) 92 | 93 | # Define a dictionary that maps file extensions to archive types 94 | archive_types = { 95 | '.zip': 'zip', 96 | '.rar': 'rar', 97 | '.7z': '7z', 98 | '.tar': 'tar', 99 | '.tar.gz': 'gztar', 100 | '.tgz': 'gztar' 101 | } 102 | 103 | # Determine the type of archive file based on the file extension 104 | file_ext = os.path.splitext(archive_path)[1].lower() 105 | 106 | # Use the appropriate function from the `shutil` library to extract the archive 107 | shutil.unpack_archive(archive_path, extracted_dir_path, archive_types[file_ext]) 108 | 109 | return extracted_dir_path 110 | 111 | @staticmethod 112 | def copy_file_to_dest_dir(temp_file_path, file_original_name, dest_dir): 113 | """ 114 | Copy a file to a destination directory using the original file name. 115 | 116 | If a file with the same name already exists in the destination directory, a suffix 117 | will be added to the file name until a unique name is found. 118 | 119 | Returns the full path of the copied file. 120 | """ 121 | suffix = '' 122 | dest_file_name = file_original_name 123 | while os.path.exists(os.path.join(dest_dir, dest_file_name)): 124 | if not suffix: 125 | suffix = 1 126 | else: 127 | suffix += 1 128 | dest_file_name = f'{os.path.splitext(file_original_name)[0]}_{suffix}{os.path.splitext(file_original_name)[1]}' 129 | 130 | dest_file_path = os.path.join(dest_dir, dest_file_name) 131 | shutil.copyfile(temp_file_path, dest_file_path) 132 | 133 | return dest_file_path 134 | 135 | 136 | class DflOptions: 137 | def __init__(self, opts): 138 | dfl_path = Path(paths.script_path) / "deepfacelab" 139 | scripts_path = Path(os.path.dirname(os.path.abspath(__file__))) / "../" 140 | print("Scripts path:" + str(scripts_path)) 141 | 142 | self.dfl_path = Path(str(dfl_path / "dfl-files")) 143 | self.workspaces_path = Path(str(dfl_path / "workspaces")) 144 | self.pak_path = Path(str(dfl_path / "pak-files")) 145 | self.xseg_path = Path(str(dfl_path / "xseg-models")) 146 | self.saehd_path = Path(str(dfl_path / "saehd-models")) 147 | self.videos_path = Path(str(dfl_path / "videos")) 148 | self.videos_frames_path = Path(str(dfl_path / "videos-frames")) 149 | self.tmp_path = Path(str(dfl_path / "tmp")) 150 | self.dflab_repo = Path(str(dfl_path / "dflab")) 151 | self.dflive_repo = Path(str(dfl_path / "dflive")) 152 | 153 | if hasattr(opts, "opts.deepfacelab_dfl_path"): 154 | self.dfl_path = Path(opts.deepfacelab_dfl_path) 155 | 156 | if hasattr(opts, "deepfacelab_workspaces_path"): 157 | self.workspaces_path = Path(opts.deepfacelab_workspaces_path) 158 | 159 | if hasattr(opts, "deepfacelab_pak_path"): 160 | self.pak_path = Path(opts.deepfacelab_pak_path) 161 | 162 | if hasattr(opts, "deepfacelab_xseg_path"): 163 | self.xseg_path = Path(opts.deepfacelab_xseg_path) 164 | 165 | if hasattr(opts, "deepfacelab_saehd_path"): 166 | self.saehd_path = Path(opts.deepfacelab_saehd_path) 167 | 168 | if hasattr(opts, "deepfacelab_videos_path"): 169 | self.videos_path = Path(opts.deepfacelab_videos_path) 170 | 171 | if hasattr(opts, "deepfacelab_videos_frames_path"): 172 | self.videos_frames_path = Path(opts.deepfacelab_videos_frames_path) 173 | 174 | if hasattr(opts, "deepfacelab_tmp_path"): 175 | self.tmp_path = Path(opts.deepfacelab_tmp_path) 176 | 177 | if hasattr(opts, "deepfacelab_dflab_repo_path"): 178 | self.dflab_repo = Path(opts.deepfacelab_dflab_repo_path) 179 | 180 | if hasattr(opts, "deepfacelab_dflive_repo_path"): 181 | self.dflive_repo = Path(opts.deepfacelab_dflive_repo_path) 182 | 183 | # Create dirs if not existing 184 | for p in [self.dfl_path, self.workspaces_path, self.pak_path, self.xseg_path, self.saehd_path, self.videos_path, self.videos_frames_path, self.tmp_path]: 185 | DflFiles.create_folder(p) 186 | 187 | 188 | # Getters start here 189 | 190 | def get_dfl_path(self): 191 | return self.dfl_path 192 | 193 | def get_workspaces_path(self): 194 | return self.workspaces_path 195 | 196 | def get_pak_path(self): 197 | return self.pak_path 198 | 199 | def get_xseg_path(self): 200 | return self.xseg_path 201 | 202 | def get_saehd_path(self): 203 | return self.saehd_path 204 | 205 | def get_videos_path(self): 206 | return self.videos_path 207 | 208 | def get_videos_frames_path(self): 209 | return self.videos_frames_path 210 | 211 | def get_tmp_path(self): 212 | return self.tmp_path 213 | 214 | def get_dflab_repo_path(self): 215 | return self.dflab_repo 216 | 217 | def get_dflive_repo_path(self): 218 | return self.dflive_repo 219 | 220 | # Lists start here 221 | 222 | def get_dfl_list(self, include_downloadable=True): 223 | dir_files = DflFiles.get_files_from_dir(self.dfl_path, [".dfm"], True) 224 | dir_files_one = DflFiles.get_files_from_dir(self.dfl_path, [".dfm"]) 225 | if include_downloadable: 226 | downloable_files = self.get_downloadable_models(dir_files_one) 227 | tmp_files = [] 228 | for f in downloable_files: 229 | # Get basename from url 230 | basename = os.path.basename(f[1]) 231 | tmp_files.append([basename, f[1]]) 232 | return dir_files + tmp_files 233 | return dir_files 234 | 235 | def get_downloadable_models(self, available_models): 236 | from scripts.command import get_downloadable_models 237 | return get_downloadable_models(self.dfl_path, available_models) 238 | 239 | def get_pak_list(self): 240 | return DflFiles.get_files_from_dir(self.pak_path, [".pak"]) 241 | 242 | def get_videos_list(self): 243 | return DflFiles.get_files_from_dir(self.videos_path, [".mp4", ".mkv"]) 244 | 245 | def get_videos_list_full_path(self): 246 | return list(self.videos_path + "/" + v for v in self.get_videos_list()) 247 | 248 | def get_workspaces_list(self): 249 | return DflFiles.get_folder_names_in_dir(self.workspaces_path) 250 | 251 | def get_saehd_models_list(self): 252 | return DflFiles.get_folder_names_in_dir(self.saehd_path) 253 | 254 | def get_xseg_models_list(self): 255 | return DflFiles.get_folder_names_in_dir(self.xseg_path) -------------------------------------------------------------------------------- /scripts/deepfacelab.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | import platform 3 | import math 4 | import json 5 | import sys 6 | import os 7 | import re 8 | from pathlib import Path 9 | import shutil 10 | 11 | import gradio as gr 12 | import numpy as np 13 | from tqdm import tqdm 14 | from PIL import Image, ImageFilter 15 | import cv2 16 | 17 | from modules.ui import create_refresh_button 18 | from modules.ui_common import folder_symbol 19 | from modules.shared import opts, OptionInfo 20 | from modules import shared, paths, script_callbacks 21 | 22 | current_frame_set = [] 23 | current_frame_set_index = 0 24 | 25 | 26 | def on_ui_tabs(): 27 | dflFiles = DflFiles() 28 | dfl_options = DflOptions(opts) 29 | 30 | dfl_path = dfl_options.get_dfl_path() 31 | workspaces_path = dfl_options.get_workspaces_path() 32 | pak_path = dfl_options.get_pak_path() 33 | xseg_path = dfl_options.get_xseg_path() 34 | saehd_path = dfl_options.get_saehd_path() 35 | videos_path = dfl_options.get_videos_path() 36 | videos_frames_path = dfl_options.get_videos_frames_path() 37 | tmp_path = dfl_options.get_tmp_path() 38 | 39 | def get_dfl_list(): 40 | return dflFiles.get_files_from_dir(dfl_path, [".dfl"]) 41 | 42 | def get_pak_list(): 43 | return dflFiles.get_files_from_dir(pak_path, [".pak"]) 44 | 45 | def get_videos_list(): 46 | return dflFiles.get_files_from_dir(videos_path, [".mp4", ".mkv"]) 47 | 48 | def get_videos_list_full_path(): 49 | return list(videos_path + "/" + v for v in get_videos_list()) 50 | 51 | def get_workspaces_list(): 52 | return dflFiles.get_folder_names_in_dir(workspaces_path) 53 | 54 | def get_saehd_models_list(): 55 | return dflFiles.get_folder_names_in_dir(saehd_path) 56 | 57 | def get_xseg_models_list(): 58 | return dflFiles.get_folder_names_in_dir(xseg_path) 59 | 60 | def render_train_saehd(gr): 61 | with gr.Row(): 62 | model_saehd_dropdown = gr.Dropdown(choices=get_saehd_models_list(), elem_id="saehd_model_dropdown", label="SAEHD Model:", interactive=True) 63 | create_refresh_button(model_saehd_dropdown, lambda: None, lambda: {"choices": get_saehd_models_list()}, "refresh_saehd_model_list") 64 | model_saehd_create_new_model = gr.Checkbox(label="Create new model") 65 | with gr.Row(): 66 | model_xseg_dropdown = gr.Dropdown(choices=get_xseg_models_list(), elem_id="xseg_model_dropdown", label="XSEG Model:", interactive=True) 67 | create_refresh_button(model_xseg_dropdown, lambda: None, lambda: {"choices": get_xseg_models_list()}, "refresh_xseg_model_list") 68 | 69 | with gr.Row(): 70 | src_pak_dropdown = gr.Dropdown(choices=get_pak_list(), elem_id="src_pak_dropdown", label="Src Faceset:", interactive=True) 71 | create_refresh_button(src_pak_dropdown, lambda: None, lambda: {"choices": get_pak_list()}, "refresh_src_pak_dropdown") 72 | 73 | with gr.Row(): 74 | dst_pak_dropdown = gr.Dropdown(choices=get_pak_list(), elem_id="dst_pak_dropdown", label="Dst Faceset:", interactive=True) 75 | create_refresh_button(dst_pak_dropdown, lambda: None, lambda: {"choices": get_pak_list()}, "refresh_dst_pak_dropdown") 76 | 77 | train_saehd = gr.Button(value="Train SAEHD", variant="primary") 78 | log_output = gr.HTML(value="") 79 | 80 | def render_faceset_extract(gr): 81 | with gr.Row(): 82 | videos_dropdown = gr.Dropdown(choices=get_videos_list(), elem_id="videos_dropdown", label="Videos", 83 | interactive=True) 84 | create_refresh_button(videos_dropdown, lambda: None, lambda: {"choices": get_videos_list()}, 85 | "refresh_videos_dropdown") 86 | faceset_output_facetype_dropdown = gr.Dropdown(choices=['half_face', 'full_face', 'whole_face', 'head', 'mark_only'], value="whole_face", label="Output face type", interactive=True) 87 | faceset_output_resolution_dropdown = gr.Dropdown(choices=["256x256","512x512","768x768","1024x1024"], value="512x512", label="Output resolution", interactive=True) 88 | faceset_output_type_dropdown = gr.Dropdown(choices=["jpg","png"], value="jpg", label="Output filetype",interactive=True) 89 | faceset_output_quality_dropdown = gr.Dropdown(choices=[90,100], value=100, label="Output quality",interactive=True) 90 | faceset_output_debug_dropdown = gr.Checkbox(value=False, label="Generate debug frames") 91 | faceset_extract_frames_button = gr.Button(value="Extract Frames Only", variant="primary") 92 | faceset_extract_button = gr.Button(value="Faceset Extract", variant="primary") 93 | faceset_extract_output = gr.Markdown() 94 | faceset_extract_button.click(DflAction.extract_frames, [videos_dropdown], faceset_extract_output) 95 | faceset_extract_button.click(DflAction.extract_faceset, [videos_dropdown, 96 | faceset_output_facetype_dropdown, 97 | faceset_output_resolution_dropdown, 98 | faceset_output_type_dropdown, 99 | faceset_output_quality_dropdown, 100 | faceset_output_debug_dropdown], faceset_extract_output) 101 | faceset_extract_frames_button.click(DflAction.extract_frames,videos_dropdown, faceset_extract_output) 102 | 103 | 104 | 105 | def render_create_dfl(gr): 106 | train_saehd = gr.Button(value="Create DFL", variant="primary") 107 | 108 | def click_create_workspace(text): 109 | if text == "": 110 | return f"Error!" 111 | return f"Workspace " + text + " created" 112 | 113 | def render_classic_tabs(): 114 | with gr.Row(): 115 | workspace_dropdown = gr.Dropdown(choices=get_workspaces_list(), elem_id="workspace_dropdown", label="Current workspace:", interactive=True) 116 | create_refresh_button(workspace_dropdown, lambda: None, lambda: {"choices": get_workspaces_list()}, "refresh_workspace_list") 117 | with gr.Tab("Faceset Extract"): 118 | render_faceset_extract(gr) 119 | with gr.Tab("Train"): 120 | render_train_saehd(gr) 121 | with gr.Tab("Create DFL"): 122 | render_create_dfl(gr) 123 | 124 | def upload_files_videos(files): 125 | file_paths = [file.name for file in files] 126 | for file in files: 127 | DflFiles.copy_file_to_dest_dir(file.name, "video.mp4", videos_path) 128 | 129 | return file_paths 130 | 131 | def get_current_video_path(videos_dropdown): 132 | return gr.Textbox.update(value=get_current_video_path_only(videos_dropdown), visible=True) 133 | 134 | def get_current_video_path_only(videos_dropdown): 135 | return str(videos_path) + "/" + str(videos_dropdown) 136 | 137 | def action_delete_video(videos_dropdown): 138 | filePath = get_current_video_path_only(videos_dropdown) 139 | DflFiles.delete_file(str(filePath)) 140 | return gr.Textbox.update(visible=False) 141 | 142 | def reset_video_dropdown(videos_dropdown): 143 | return gr.Dropdown.update(value="") 144 | 145 | def render_dataset_videos(): 146 | upload_button = gr.UploadButton("Click to Upload Video/s", file_types=["video"], file_count="multiple") 147 | file_output = gr.File(label="Uploaded Video/s") 148 | upload_button.upload(upload_files_videos, upload_button, file_output, scroll_to_output=True) 149 | 150 | def render_dataset_xseg(): 151 | upload_button_xseg = gr.UploadButton("Click to Upload XSEG model/s", file_types=["video"], file_count="multiple") 152 | file_output_xseg = gr.File(label="Uploaded XSEG model/s") 153 | upload_button_xseg.upload(upload_files_videos, upload_button_xseg, file_output_xseg, scroll_to_output=True) 154 | 155 | def render_dataset_saehd(): 156 | upload_button_saehd = gr.UploadButton("Click to Upload SAEHD model/s", file_types=["video"], file_count="multiple") 157 | file_output_saehd = gr.File(label="Uploaded SAEHD model/s") 158 | upload_button_saehd.upload(upload_files_videos, upload_button_saehd, file_output_saehd, scroll_to_output=True) 159 | 160 | def render_dataset_facesets(): 161 | upload_button_datasets = gr.UploadButton("Click to Upload Faceset/s", file_types=["video"], file_count="multiple") 162 | file_output_datasets = gr.File(label="Uploaded Faceset/s") 163 | upload_button_datasets.upload(upload_files_videos, upload_button_datasets, file_output_datasets, scroll_to_output=True) 164 | 165 | def render_dataset_dfl(): 166 | upload_button_dfl = gr.UploadButton("Click to Upload DFL/s", file_types=["video"], file_count="multiple") 167 | file_output_dfl = gr.File(label="Uploaded DFL/s") 168 | upload_button_dfl.upload(upload_files_videos, upload_button_dfl, file_output_dfl, scroll_to_output=True) 169 | 170 | # Display contents in main tab "DeepFaceLab" in SD1111 UI 171 | with gr.Blocks(analytics_enabled=False) as training_picker: 172 | with gr.Row(): 173 | with gr.Column(scale=1): 174 | gr.Markdown("DeepFaceLab") 175 | render_classic_tabs() 176 | gr.Markdown("Creator") 177 | with gr.Tab("Workspace"): 178 | text = gr.Textbox(value="", label="Name") 179 | button = gr.Button(value="Create Workspace", variant="primary") 180 | output1 = gr.Textbox(label="Status") 181 | button.click(click_create_workspace, [text], output1) 182 | with gr.Tab("SAEHD Model"): 183 | with gr.Tab("New Model"): 184 | nothing = 0 185 | with gr.Tab("Clone Existing Model"): 186 | with gr.Row(): 187 | model_saehd_dropdown = gr.Dropdown(choices=get_saehd_models_list(), elem_id="saehd_model_dropdown", label="SAEHD Model:", interactive=True) 188 | create_refresh_button(model_saehd_dropdown, lambda: None, lambda: {"choices": get_saehd_models_list()}, "refresh_saehd_model_list") 189 | 190 | text = gr.Textbox(value="", label="Name") 191 | button = gr.Button(value="Create SAEHD Model", variant="primary") 192 | output1 = gr.Textbox(label="Status") 193 | button.click(click_create_workspace, [text], output1) 194 | 195 | gr.Markdown("Fast tools") 196 | with gr.Tab("DFL Creator"): 197 | nothing = 0 198 | 199 | with gr.Column(scale=2): 200 | gr.Markdown("Preview/Browser") 201 | with gr.Tabs() as tabs_preview: 202 | with gr.TabItem("Status", id=0): 203 | nothing = 0 204 | with gr.TabItem("Workspaces", id=1): 205 | nothing = 0 206 | with gr.TabItem("Videos", id=2): 207 | with gr.Tabs() as tabs_preview_videos: 208 | with gr.TabItem("Browser", id=0): 209 | browser_videos_gallery = gr.Gallery(fn=get_videos_list_full_path) 210 | browser_videos_gallery.style(grid=4, height=4, container=True) 211 | with gr.TabItem("Preview", id=1): 212 | with gr.Row(): 213 | videos_dropdown = gr.Dropdown(choices=get_videos_list(), elem_id="videos_dropdown", label="Videos:", interactive=True) 214 | create_refresh_button(videos_dropdown, lambda: None, lambda: {"choices": get_videos_list()}, "refresh_videos_dropdown") 215 | with gr.Row(): 216 | download_video = gr.Button(value="Download Video", variant="gray") 217 | delete_video = gr.Button(value="Delete Video", variant="red") 218 | main_video_preview = gr.Video(interactive=None) 219 | download_video.click(reset_video_dropdown, videos_dropdown, videos_dropdown) 220 | delete_video.click(action_delete_video, videos_dropdown, main_video_preview) 221 | videos_dropdown.change(get_current_video_path, videos_dropdown, main_video_preview) 222 | with gr.TabItem("XSEG", id=3): 223 | with gr.Tabs() as tabs_preview_xseg: 224 | with gr.TabItem("Browser", id=0): 225 | nothing = 0 226 | with gr.TabItem("Viewer", id=1): 227 | with gr.Row(): 228 | xseg_dropdown = gr.Dropdown(choices=get_xseg_models_list(), elem_id="xseg_dropdown", label="XSEG Model:", interactive=True) 229 | create_refresh_button(xseg_dropdown, lambda: None, lambda: {"choices": get_xseg_models_list()}, "refresh_xseg_list") 230 | 231 | with gr.TabItem("SAEHD", id=4): 232 | with gr.Tabs() as tabs_preview_saehd: 233 | with gr.TabItem("Browser", id=0): 234 | nothing = 0 235 | with gr.TabItem("Viewer", id=1): 236 | with gr.Row(): 237 | saehd_dropdown = gr.Dropdown(choices=get_saehd_models_list(), elem_id="saehd_dropdown", label="SAEHD Model:", interactive=True) 238 | create_refresh_button(saehd_dropdown, lambda: None, lambda: {"choices": get_saehd_models_list()}, "refresh_dfl_list") 239 | 240 | with gr.TabItem("Facesets", id=5): 241 | with gr.Tabs() as tabs_preview_facesets: 242 | with gr.TabItem("Browser", id=0): 243 | nothing = 0 244 | with gr.TabItem("Viewer", id=1): 245 | with gr.Row(): 246 | faceset_dropdown = gr.Dropdown(choices=get_pak_list(), elem_id="faceset_dropdown", label="Faceset:", interactive=True) 247 | create_refresh_button(faceset_dropdown, lambda: None, lambda: {"choices": get_pak_list()}, "refresh_faceset_list") 248 | with gr.TabItem("DFL", id=6): 249 | with gr.Tabs() as tabs_preview_dfl: 250 | with gr.TabItem("Browser", id=0): 251 | nothing = 0 252 | with gr.TabItem("Viewer", id=1): 253 | with gr.Row(): 254 | dfl_dropdown = gr.Dropdown(choices=get_dfl_list(), elem_id="dfl_dropdown", label="DFL:", interactive=True) 255 | create_refresh_button(dfl_dropdown, lambda: None, lambda: {"choices": get_dfl_list()}, "refresh_dfl_list") 256 | with gr.Column(scale=1): 257 | gr.Markdown("Upload") 258 | with gr.Tab("Videos"): 259 | render_dataset_videos() 260 | with gr.Tab("XSEG"): 261 | render_dataset_xseg() 262 | with gr.Tab("SAEHD"): 263 | render_dataset_saehd() 264 | with gr.Tab("Facesets"): 265 | render_dataset_facesets() 266 | with gr.Tab("DFL"): 267 | render_dataset_dfl() 268 | 269 | return (training_picker, "DeepFaceLab", "deepfacelab"), 270 | 271 | 272 | def on_ui_settings(): 273 | dfl_path = Path(paths.script_path) / "deepfacelab" 274 | section = ('deepfacelab', "DeepFaceLab") 275 | opts.add_option("deepfacelab_dflab_repo_path", OptionInfo(str(dfl_path / "dflab-repo"), "Path to DeepFaceLab repo are located", section=section)) 276 | opts.add_option("deepfacelab_dflive_repo_path", OptionInfo(str(dfl_path / "dflive-repo"), "Path to DeepFaceLive repo are located", section=section)) 277 | opts.add_option("deepfacelab_workspaces_path", OptionInfo(str(dfl_path / "workspaces"), "Path to dir where DeepFaceLab workspaces are located", section=section)) 278 | opts.add_option("deepfacelab_dfl_path", OptionInfo(str(dfl_path / "dfl-files"), "Path to read/write .dfl files from", section=section)) 279 | opts.add_option("deepfacelab_pak_path", OptionInfo(str(dfl_path / "pak-files"), "Default facesets .pak image directory", section=section)) 280 | opts.add_option("deepfacelab_xseg_path", OptionInfo(str(dfl_path / "xseg-models"), "Default XSeg path for XSeg models directory", section=section)) 281 | opts.add_option("deepfacelab_saehd_path", OptionInfo(str(dfl_path / "saehd-models"), "Default path for SAEHD models directory", section=section)) 282 | opts.add_option("deepfacelab_videos_path", OptionInfo(str(dfl_path / "videos"), "Default path for Videos for deepfacelab", section=section)) 283 | opts.add_option("deepfacelab_videos_frames_path", OptionInfo(str(dfl_path / "videos-frames"), "Default path for Video Frames for deepfacelab", section=section)) 284 | opts.add_option("deepfacelab_tmp_path", OptionInfo(str(dfl_path / "tmp"), "Default path for tmp actions for deepfacelab", section=section)) 285 | 286 | 287 | script_callbacks.on_ui_settings(on_ui_settings) 288 | script_callbacks.on_ui_tabs(on_ui_tabs) 289 | 290 | 291 | 292 | import os 293 | from pathlib import Path 294 | 295 | class DflCommunicator: 296 | @staticmethod 297 | def extract_frames(video_path, frames_dir, file_format_output='jpg', file_format_output_quality=100): 298 | # Create the frames directory if it doesn't exist 299 | if not os.path.exists(frames_dir): 300 | os.makedirs(frames_dir) 301 | 302 | # Open the video file 303 | cap = cv2.VideoCapture(video_path) 304 | 305 | # Get the frames per second (FPS) of the video 306 | fps = cap.get(cv2.CAP_PROP_FPS) 307 | 308 | # Initialize a counter for the frames 309 | frame_count = 0 310 | 311 | # Loop through the frames of the video 312 | while cap.isOpened(): 313 | # Read a frame from the video 314 | ret, frame = cap.read() 315 | 316 | # If there are no more frames, break out of the loop 317 | if not ret: 318 | break 319 | 320 | # Save the frame as a file in the frames directory 321 | file_extension = '.' + file_format_output 322 | frame_file = os.path.join(frames_dir, f"{frame_count:06d}{file_extension}") 323 | if file_format_output == 'jpg': 324 | cv2.imwrite(frame_file, frame, [int(cv2.IMWRITE_JPEG_QUALITY), file_format_output_quality]) 325 | elif file_format_output == 'png': 326 | cv2.imwrite(frame_file, frame, [int(cv2.IMWRITE_PNG_COMPRESSION), file_format_output_quality]) 327 | 328 | # Increment the frame counter 329 | frame_count += 1 330 | 331 | # Release the video capture object 332 | cap.release() 333 | 334 | # Return the number of frames extracted 335 | return frame_count 336 | 337 | class DflFiles: 338 | 339 | @staticmethod 340 | def folder_exists(dir_path): 341 | """ 342 | Check whether a directory exists. 343 | 344 | Returns True if the directory exists, False otherwise. 345 | """ 346 | return os.path.exists(dir_path) and os.path.isdir(dir_path) 347 | 348 | @staticmethod 349 | def create_folder(path): 350 | """Create a folder at the specified path""" 351 | os.makedirs(path, exist_ok=True) 352 | 353 | @staticmethod 354 | def delete_folder(path): 355 | """Delete a folder and all its contents at the specified path""" 356 | os.removedirs(path) 357 | 358 | @staticmethod 359 | def empty_folder(path): 360 | """Clear the contents of a folder at the specified path""" 361 | for file_name in os.listdir(path): 362 | file_path = os.path.join(path, file_name) 363 | if os.path.isfile(file_path): 364 | os.remove(file_path) 365 | 366 | @staticmethod 367 | def create_empty_file(path): 368 | """Create an empty file at the specified path""" 369 | open(path, 'a').close() 370 | 371 | @staticmethod 372 | def delete_file(path): 373 | """Delete a file at the specified path""" 374 | os.remove(path) 375 | 376 | @staticmethod 377 | def move_file(src, dst): 378 | """Move a file from the source path to the destination path""" 379 | os.replace(src, dst) 380 | 381 | @staticmethod 382 | def get_files_from_dir(base_dir, extension_list): 383 | """Return a list of file names in a directory with a matching file extension""" 384 | files = [] 385 | for v in Path(base_dir).iterdir(): 386 | if v.suffix in extension_list and not v.name.startswith('.ipynb'): 387 | files.append(v.name) 388 | return files 389 | 390 | @staticmethod 391 | def get_folder_names_in_dir(base_dir): 392 | """Return a list of folder names in a directory""" 393 | folders = [] 394 | for v in Path(base_dir).iterdir(): 395 | if v.is_dir() and not v.name.startswith('.ipynb'): 396 | folders.append(v.name) 397 | return folders 398 | 399 | @staticmethod 400 | def extract_archive(archive_path, dest_dir, dir_name): 401 | """ 402 | Extract an archive file to a directory with a specified name. 403 | 404 | The extracted directory will be created inside the destination directory with the specified name. 405 | If the name is taken, a suffix will be added to create a unique name. 406 | 407 | Returns the full path of the directory where the contents were extracted. 408 | """ 409 | suffix = '' 410 | extracted_dir_name = dir_name 411 | while os.path.exists(os.path.join(dest_dir, extracted_dir_name + suffix)): 412 | if not suffix: 413 | suffix = 1 414 | else: 415 | suffix += 1 416 | extracted_dir_name = f'{dir_name}_{suffix}' 417 | 418 | extracted_dir_path = os.path.join(dest_dir, extracted_dir_name) 419 | os.makedirs(extracted_dir_path, exist_ok=True) 420 | 421 | # Define a dictionary that maps file extensions to archive types 422 | archive_types = { 423 | '.zip': 'zip', 424 | '.rar': 'rar', 425 | '.7z': '7z', 426 | '.tar': 'tar', 427 | '.tar.gz': 'gztar', 428 | '.tgz': 'gztar' 429 | } 430 | 431 | # Determine the type of archive file based on the file extension 432 | file_ext = os.path.splitext(archive_path)[1].lower() 433 | 434 | # Use the appropriate function from the `shutil` library to extract the archive 435 | shutil.unpack_archive(archive_path, extracted_dir_path, archive_types[file_ext]) 436 | 437 | return extracted_dir_path 438 | 439 | @staticmethod 440 | def copy_file_to_dest_dir(temp_file_path, file_original_name, dest_dir): 441 | """ 442 | Copy a file to a destination directory using the original file name. 443 | 444 | If a file with the same name already exists in the destination directory, a suffix 445 | will be added to the file name until a unique name is found. 446 | 447 | Returns the full path of the copied file. 448 | """ 449 | suffix = '' 450 | dest_file_name = file_original_name 451 | while os.path.exists(os.path.join(dest_dir, dest_file_name)): 452 | if not suffix: 453 | suffix = 1 454 | else: 455 | suffix += 1 456 | dest_file_name = f'{os.path.splitext(file_original_name)[0]}_{suffix}{os.path.splitext(file_original_name)[1]}' 457 | 458 | dest_file_path = os.path.join(dest_dir, dest_file_name) 459 | shutil.copyfile(temp_file_path, dest_file_path) 460 | 461 | return dest_file_path 462 | 463 | class DflOptions: 464 | def __init__(self, opts): 465 | self.dfl_files = DflFiles() 466 | self.dfl_path = Path(opts.deepfacelab_dfl_path) 467 | self.workspaces_path = Path(opts.deepfacelab_workspaces_path) 468 | self.pak_path = Path(opts.deepfacelab_pak_path) 469 | self.xseg_path = Path(opts.deepfacelab_xseg_path) 470 | self.saehd_path = Path(opts.deepfacelab_saehd_path) 471 | self.videos_path = Path(opts.deepfacelab_videos_path) 472 | self.videos_frames_path = Path(opts.deepfacelab_videos_frames_path) 473 | self.tmp_path = Path(opts.deepfacelab_tmp_path) 474 | self.dflab_repo = Path(opts.deepfacelab_dflab_repo_path) 475 | self.dflive_repo = Path(opts.deepfacelab_dflive_repo_path) 476 | 477 | # Create dirs if not existing 478 | for p in [self.dfl_path, self.workspaces_path, self.pak_path, self.xseg_path, self.saehd_path, self.videos_path, self.videos_frames_path, self.tmp_path]: 479 | self.dfl_files.create_folder(p) 480 | 481 | def get_dfl_path(self): 482 | return self.dfl_path 483 | 484 | def get_workspaces_path(self): 485 | return self.workspaces_path 486 | 487 | def get_pak_path(self): 488 | return self.pak_path 489 | 490 | def get_xseg_path(self): 491 | return self.xseg_path 492 | 493 | def get_saehd_path(self): 494 | return self.saehd_path 495 | 496 | def get_videos_path(self): 497 | return self.videos_path 498 | 499 | def get_videos_frames_path(self): 500 | return self.videos_frames_path 501 | 502 | def get_tmp_path(self): 503 | return self.tmp_path 504 | 505 | def get_dflab_repo_path(self): 506 | return self.dflab_repo 507 | 508 | def get_dflive_repo_path(self): 509 | return self.dflive_repo 510 | 511 | class DflAction: 512 | dflFiles = DflFiles() 513 | 514 | @staticmethod 515 | def extract_frames(videos_dropdown): 516 | return "Video: " + str(videos_dropdown) 517 | 518 | @staticmethod 519 | def extract_faceset(videos_dropdown, 520 | faceset_output_facetype_dropdown, 521 | faceset_output_resolution_dropdown, 522 | faceset_output_type_dropdown, 523 | faceset_output_quality_dropdown, 524 | faceset_output_debug_dropdown): 525 | return "Video: " + str(videos_dropdown) + " Output Face Type: " + faceset_output_facetype_dropdown -------------------------------------------------------------------------------- /scripts/deepfacelive.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | 4 | # Get the absolute path of the directory containing the importing file 5 | current_dir = os.path.dirname(os.path.abspath(__file__)) 6 | 7 | # Append the relative path to the importing file to the system path 8 | sys.path.append(os.path.join(current_dir, '../repo/dflive/')) 9 | 10 | import cv2 11 | from PIL import Image 12 | import numpy as np 13 | import gradio as gr 14 | from pathlib import Path 15 | 16 | from modules import processing, images 17 | from modules import scripts, script_callbacks, shared, devices, modelloader 18 | from modules.processing import Processed, StableDiffusionProcessingTxt2Img, StableDiffusionProcessingImg2Img 19 | from modules.shared import opts, cmd_opts, state 20 | from modules.sd_models import model_hash 21 | from modules.paths import models_path 22 | from basicsr.utils.download_util import load_file_from_url 23 | from modules.ui import create_refresh_button 24 | from scripts.ddetailerutils import DetectionDetailerScript, preload_ddetailer_model 25 | 26 | from scripts.command import execute_deep_face_live_multiple 27 | dd_models_path = os.path.join(models_path, "mmdet") 28 | 29 | from scripts.dflutils import DflOptions, DflFiles 30 | dfl_options = DflOptions(opts) 31 | 32 | def list_models(include_downloadable=True): 33 | dfl_dropdown = [] 34 | dfl_dropdown.append("None") 35 | #dfl_dropdown.append("Automatic") 36 | dfl_list = dfl_options.get_dfl_list(include_downloadable) 37 | for dfl in dfl_list: 38 | dfl_dropdown.append(dfl[0]) 39 | return dfl_dropdown 40 | 41 | def download_url(url, filename): 42 | load_file_from_url(url, filename) 43 | 44 | from urllib.parse import urlparse 45 | 46 | def is_url(string): 47 | try: 48 | result = urlparse(string) 49 | # Check if the URL has a scheme (e.g., http or https) and a network location (e.g., www.example.com) 50 | return all([result.scheme, result.netloc]) 51 | except ValueError: 52 | return False 53 | 54 | def get_model_url(model_name): 55 | dfl_list = dfl_options.get_dfl_list(True) 56 | for dfl in dfl_list: 57 | if dfl[0] == model_name and is_url(dfl[1]): 58 | return dfl[1] 59 | return None 60 | 61 | def list_detectors(): 62 | return ['CenterFace', 'S3FD', 'YoloV5'] 63 | 64 | def list_markers(): 65 | return ['OpenCV LBF', 'Google FaceMesh', 'InsightFace_2D106'] 66 | 67 | def list_align_modes(): 68 | return ['From rectangle', 'From points', 'From static rect'] 69 | 70 | def startup(): 71 | if (DflFiles.folder_exists(dd_models_path) == False): 72 | print("No detection models found, downloading...") 73 | bbox_path = os.path.join(dd_models_path, "bbox") 74 | load_file_from_url("https://huggingface.co/dustysys/ddetailer/resolve/main/mmdet/bbox/mmdet_anime-face_yolov3.pth", bbox_path) 75 | load_file_from_url("https://huggingface.co/dustysys/ddetailer/raw/main/mmdet/bbox/mmdet_anime-face_yolov3.py", bbox_path) 76 | 77 | 78 | startup() 79 | 80 | def gr_show(visible=True): 81 | return {"visible": visible, "__type__": "update"} 82 | 83 | 84 | class DeepFaceLive(scripts.Script): 85 | def title(self): 86 | return "DeepFaceLive - AI Face Swap/Recovery" 87 | 88 | def show(self, is_img2img): 89 | return True 90 | 91 | 92 | def ui(self, is_img2img): 93 | import modules.ui 94 | 95 | gr.HTML("") 96 | with gr.Group(): 97 | with gr.Row(): 98 | gr.HTML("

You can put the models into " + str(dfl_options.dfl_path) + "


") 99 | with gr.Row(): 100 | dfm_model_dropdown = gr.Dropdown(label="DFM Model", choices=list_models(True), value="None", visible=True, type="value") 101 | create_refresh_button(dfm_model_dropdown, lambda: None, lambda: {"choices": list_models(True)}, "refresh_dfm_model_list") 102 | 103 | 104 | with gr.Row(): 105 | image_return_original_checkbox = gr.Checkbox(label="Return original image") 106 | enable_detection_detailer_face_checkbox = gr.Checkbox(label="Enable Detection Detailer for face", value=False) 107 | save_detection_detailer_image_checkbox = gr.Checkbox(label="Return Detection Detailer Image") 108 | 109 | with gr.Group(elem_id="dfl_settings"): 110 | with gr.Tab("Face Detector"): 111 | gr.HTML("
") 112 | with gr.Row(): 113 | step_1_detector_input = gr.Dropdown(label="Detector", choices=list_detectors(), value="YoloV5", visible=True, type="value") 114 | step_1_window_size_input = gr.Dropdown(label="Window size", choices=["Auto", "512", "320", "160"], value="Auto", type="value") 115 | step_1_threshold_input = gr.Slider(minimum=0.0, maximum=1.0, step=0.1, label="Threshold ", value=0.5) 116 | with gr.Row(): 117 | step_1_max_faces_input = gr.Number(label="Max Faces", min_value=1, max_value=None, value=3) 118 | step_1_sort_by_input = gr.Dropdown(label="Sort By", choices=["Largest", "Dist from center", "From left to right", "From top to bottom", "From bottom to top"], value="Largest") 119 | 120 | with gr.Tab("Marker"): 121 | gr.HTML("
") 122 | with gr.Row(): 123 | step_2_marker_input = gr.Dropdown(label="Marker", choices=list_markers(), value="InsightFace_2D106", visible=True, type="value") 124 | step_2_marker_coverage_input = gr.Slider(minimum=0.0, maximum=3.0, step=0.1, label="Marker coverage", value=1.6) 125 | 126 | with gr.Tab("Aligner"): 127 | gr.HTML("
") 128 | with gr.Row(): 129 | step_3_align_mode_input = gr.Dropdown(label="Align mode", choices=list_align_modes(), value="From points", visible=True, type="value") 130 | step_3_face_coverage_input = gr.Slider(label="Face coverage ", minimum=0.0, maximum=3.0, step=0.1, value=2.2) 131 | step_3_resolution_input = gr.Slider(label="Resolution ", minimum=16, maximum=640, step=16, value=320) 132 | with gr.Row(): 133 | step_3_exclude_moving_parts_input = gr.Checkbox(label="Exclude moving parts", value=True) 134 | step_3_head_mode_input = gr.Checkbox(label="Head mode", value=False) 135 | step_3_freeze_z_rotation_input = gr.Checkbox(label="Freeze Z rotation", value=False) 136 | with gr.Row(): 137 | step_3_x_offset_input = gr.Number(label="X offset", min_value=0, max_value=200, step=0.1, value=0) 138 | step_3_y_offset_input = gr.Number(label="Y offset", min_value=0, max_value=200, step=0.1, value=0) 139 | 140 | with gr.Tab("Swapper"): 141 | gr.HTML("
") 142 | with gr.Row(): 143 | step_4_swap_all_faces_input = gr.Checkbox(label="Swap all faces", value=True) 144 | step_4_face_id_input = gr.Number(label="Face ID", min_value=0, max_value=24, step=1, value=0) 145 | step_4_two_pass_input = gr.Checkbox(label="Two Pass", value=False) 146 | step_4_pre_sharpen_input = gr.Slider(label="Pre-sharpen ", minimum=0.0, maximum=8.0, step=0.1, value=4.0) 147 | with gr.Row(): 148 | step_4_pre_gamma_red_input = gr.Number(label="Pre-gamma Red", min_value=1, max_value=100, step=1, value=1) 149 | step_4_pre_gamma_green_input = gr.Number(label="Pre-gamma Green", min_value=1, max_value=100, step=1, value=1) 150 | step_4_pre_gamma_blue_input = gr.Number(label="Pre-gamma Blue", min_value=1, max_value=100, step=1, value=1) 151 | with gr.Row(): 152 | step_4_post_gamma_red_input = gr.Number(label="Post-gamma Red", min_value=1, max_value=100, step=1, value=1) 153 | step_4_post_gamma_green_input = gr.Number(label="Post-gamma Green", min_value=1, max_value=100, step=1, value=1) 154 | step_4_post_gamma_blue_input = gr.Number(label="Post-gamma Blue", min_value=1, max_value=100, step=1, value=1) 155 | 156 | 157 | with gr.Tab("Merger"): 158 | gr.HTML("
") 159 | with gr.Row(): 160 | step_5_x_offset_input = gr.Slider(label="Face X offset ", minimum=0, maximum=10, step=1, value=0) 161 | step_5_y_offset_input = gr.Slider(label="Face Y offset ", minimum=0, maximum=10, step=1, value=0) 162 | step_5_face_scale_input = gr.Slider(label="Face scale ", minimum=0.0, maximum=2.0, step=0.1, value=1.0) 163 | with gr.Row(): 164 | step_5_face_mask_src_input = gr.Checkbox(label="Face mask (SRC)", value=True) 165 | step_5_face_celeb_input = gr.Checkbox(label="Face mask (CELEB)", value=True) 166 | step_5_face_lmrks_input = gr.Checkbox(label="Face mask (LMRKS)", value=False) 167 | with gr.Row(): 168 | step_5_face_mask_erode_input = gr.Number(label="Face mask erode", min_value=0, max_value=100, step=1, value=5) 169 | step_5_face_mask_blur_input = gr.Number(label="Face mask blur", min_value=0, max_value=100, step=1, value=25) 170 | step_5_color_transfer_input = gr.Dropdown(label="Color transfer", choices=["none", "rct"], value="rct", visible=True, type="value") 171 | with gr.Row(): 172 | step_5_color_compression_input = gr.Slider(label="Color compression ", minimum=0, maximum=100, step=1, value=0) 173 | step_5_face_opacity_input = gr.Slider(label="Face opacity ", minimum=0.00, maximum=1.00, step=0.01, value=1.00) 174 | #step_5_interpolation_input = gr.Dropdown(label="Interpolation", choices=["bilinear", "bicubic", "lanczos4"], value="bilinear", visible=True, type="value") 175 | step_5_interpolation_input = gr.Dropdown(label="Interpolation", choices=["bilinear"], value="bilinear", visible=True, type="value") 176 | 177 | return [dfm_model_dropdown, 178 | image_return_original_checkbox, 179 | enable_detection_detailer_face_checkbox, 180 | save_detection_detailer_image_checkbox, 181 | step_1_detector_input, 182 | step_1_window_size_input, 183 | step_1_threshold_input, 184 | step_1_max_faces_input, 185 | step_1_sort_by_input, 186 | step_2_marker_input, 187 | step_2_marker_coverage_input, 188 | step_3_align_mode_input, 189 | step_3_face_coverage_input, 190 | step_3_resolution_input, 191 | step_3_exclude_moving_parts_input, 192 | step_3_head_mode_input, 193 | step_3_freeze_z_rotation_input, 194 | step_3_x_offset_input, 195 | step_3_y_offset_input, 196 | step_4_swap_all_faces_input, 197 | step_4_face_id_input, 198 | step_4_two_pass_input, 199 | step_4_pre_sharpen_input, 200 | step_4_pre_gamma_red_input, 201 | step_4_pre_gamma_green_input, 202 | step_4_pre_gamma_blue_input, 203 | step_4_post_gamma_red_input, 204 | step_4_post_gamma_green_input, 205 | step_4_post_gamma_blue_input, 206 | step_5_x_offset_input, 207 | step_5_y_offset_input, 208 | step_5_face_scale_input, 209 | step_5_face_mask_src_input, 210 | step_5_face_celeb_input, 211 | step_5_face_lmrks_input, 212 | step_5_face_mask_erode_input, 213 | step_5_face_mask_blur_input, 214 | step_5_color_transfer_input, 215 | step_5_interpolation_input, 216 | step_5_color_compression_input, 217 | step_5_face_opacity_input] 218 | 219 | def process_frames(self, images, dfm_model_dropdown, step_1_detector_input, 220 | step_1_window_size_input, 221 | step_1_threshold_input, 222 | step_1_max_faces_input, 223 | step_1_sort_by_input, 224 | step_2_marker_input, 225 | step_2_marker_coverage_input, 226 | step_3_align_mode_input, 227 | step_3_face_coverage_input, 228 | step_3_resolution_input, 229 | step_3_exclude_moving_parts_input, 230 | step_3_head_mode_input, 231 | step_3_freeze_z_rotation_input, 232 | step_3_x_offset_input, 233 | step_3_y_offset_input, 234 | step_4_swap_all_faces_input, 235 | step_4_face_id_input, 236 | step_4_two_pass_input, 237 | step_4_pre_sharpen_input, 238 | step_4_pre_gamma_red_input, 239 | step_4_pre_gamma_green_input, 240 | step_4_pre_gamma_blue_input, 241 | step_4_post_gamma_red_input, 242 | step_4_post_gamma_green_input, 243 | step_4_post_gamma_blue_input, 244 | step_5_x_offset_input, 245 | step_5_y_offset_input, 246 | step_5_face_scale_input, 247 | step_5_face_mask_src_input, 248 | step_5_face_celeb_input, 249 | step_5_face_lmrks_input, 250 | step_5_face_mask_erode_input, 251 | step_5_face_mask_blur_input, 252 | step_5_color_transfer_input, 253 | step_5_interpolation_input, 254 | step_5_color_compression_input, 255 | step_5_face_opacity_input 256 | ): 257 | 258 | from scripts.command import DetectorType, FaceSortBy, MarkerType, AlignMode 259 | print("Processing " + dfm_model_dropdown) 260 | img_array = [] 261 | for orig_image in images: 262 | img_array.append(np.array(orig_image)) 263 | 264 | dfm_path = str(Path(str(dfl_options.dfl_path) + "/" + str(dfm_model_dropdown))) 265 | 266 | step_1_detector = DetectorType.YOLOV5 267 | if step_1_detector_input == "CenterFace": 268 | step_1_detector = DetectorType.CENTER_FACE 269 | elif step_1_detector_input == "S3FD": 270 | step_1_detector = DetectorType.S3FD 271 | 272 | step_1_sort_by = FaceSortBy.LARGEST 273 | if step_1_sort_by_input == "Dist from center": 274 | step_1_sort_by = FaceSortBy.DIST_FROM_CENTER 275 | if step_1_sort_by_input == "From left to right": 276 | step_1_sort_by = FaceSortBy.LEFT_RIGHT 277 | if step_1_sort_by_input == "From top to bottom": 278 | step_1_sort_by = FaceSortBy.TOP_BOTTOM 279 | if step_1_sort_by_input == "From bottom to top": 280 | step_1_sort_by = FaceSortBy.BOTTOM_TOP 281 | 282 | step_1_window_size = step_1_window_size_input 283 | if step_1_window_size == "Auto": 284 | step_1_window_size = 0 285 | 286 | step_1_threshold = str(step_1_threshold_input) 287 | 288 | step_2_marker = MarkerType.INSIGHT_2D106 289 | if step_2_marker_input == "OpenCV LBF": 290 | step_2_marker = MarkerType.OPENCV_LBF 291 | elif step_2_marker_input == "Google FaceMesh": 292 | step_2_marker = MarkerType.GOOGLE_FACEMESH 293 | 294 | step_3_align_mode = AlignMode.FROM_POINTS 295 | if step_3_align_mode_input == "From rectangle": 296 | step_3_align_mode = AlignMode.FROM_RECT 297 | elif step_3_align_mode_input == "From static rect": 298 | step_3_align_mode = AlignMode.FROM_STATIC_RECT 299 | 300 | return execute_deep_face_live_multiple(numpy_images=img_array, 301 | dfm_path=dfm_path, 302 | device_id=0, 303 | step_1_detector=step_1_detector, 304 | step_1_window_size=step_1_window_size, 305 | step_1_threshold=step_1_threshold, 306 | step_1_max_faces=step_1_max_faces_input, 307 | step_1_sort_by=step_1_sort_by, 308 | step_2_marker=step_2_marker, 309 | step_2_marker_coverage=step_2_marker_coverage_input, 310 | step_3_align_mode=step_3_align_mode, 311 | step_3_face_coverage=step_3_face_coverage_input, 312 | step_3_resolution=int(step_3_resolution_input), 313 | step_3_exclude_moving_parts=step_3_exclude_moving_parts_input, 314 | step_3_head_mode=step_3_head_mode_input, 315 | step_3_freeze_z_rotation=step_3_freeze_z_rotation_input, 316 | step_3_x_offset=step_3_x_offset_input, 317 | step_3_y_offset=step_3_y_offset_input, 318 | step_4_swap_all_faces=step_4_swap_all_faces_input, 319 | step_4_face_id=step_4_face_id_input, 320 | step_4_two_pass=step_4_two_pass_input, 321 | step_4_pre_sharpen=step_4_pre_sharpen_input, 322 | step_4_pre_gamma_red=step_4_pre_gamma_red_input, 323 | step_4_pre_gamma_green=step_4_pre_gamma_green_input, 324 | step_4_pre_gamma_blue=step_4_pre_gamma_blue_input, 325 | step_4_post_gamma_red=step_4_post_gamma_red_input, 326 | step_4_post_gamma_green=step_4_post_gamma_green_input, 327 | step_4_post_gamma_blue=step_4_post_gamma_blue_input, 328 | step_5_x_offset=step_5_x_offset_input, 329 | step_5_y_offset=step_5_y_offset_input, 330 | step_5_face_scale=step_5_face_scale_input, 331 | step_5_face_mask_src=step_5_face_mask_src_input, 332 | step_5_face_celeb=step_5_face_celeb_input, 333 | step_5_face_lmrks=step_5_face_lmrks_input, 334 | step_5_face_mask_erode=step_5_face_mask_erode_input, 335 | step_5_face_mask_blur=step_5_face_mask_blur_input, 336 | step_5_color_transfer=step_5_color_transfer_input, 337 | step_5_interpolation=step_5_interpolation_input, 338 | step_5_color_compression=step_5_color_compression_input, 339 | step_5_face_opacity=step_5_face_opacity_input) 340 | 341 | def generate_batches(self, final_images, batch_size): 342 | """ 343 | Generate batches of data based on a given batch size. 344 | 345 | Args: 346 | final_images (list): List of images to be batched. 347 | batch_size (int): The size of each batch. 348 | 349 | Returns: 350 | list: A list of batches, where each batch is a list of images. 351 | """ 352 | num_batches = (len(final_images) + batch_size - 1) // batch_size 353 | batches = [] 354 | for i in range(num_batches): 355 | start = i * batch_size 356 | end = min((i + 1) * batch_size, len(final_images)) 357 | batches.append(final_images[start:end]) 358 | return batches 359 | 360 | def run(self, p, 361 | dfm_model_dropdown, 362 | image_return_original_checkbox, 363 | enable_detection_detailer_face_checkbox, 364 | save_detection_detailer_image_checkbox, 365 | step_1_detector_input, 366 | step_1_window_size_input, 367 | step_1_threshold_input, 368 | step_1_max_faces_input, 369 | step_1_sort_by_input, 370 | step_2_marker_input, 371 | step_2_marker_coverage_input, 372 | step_3_align_mode_input, 373 | step_3_face_coverage_input, 374 | step_3_resolution_input, 375 | step_3_exclude_moving_parts_input, 376 | step_3_head_mode_input, 377 | step_3_freeze_z_rotation_input, 378 | step_3_x_offset_input, 379 | step_3_y_offset_input, 380 | step_4_swap_all_faces_input, 381 | step_4_face_id_input, 382 | step_4_two_pass_input, 383 | step_4_pre_sharpen_input, 384 | step_4_pre_gamma_red_input, 385 | step_4_pre_gamma_green_input, 386 | step_4_pre_gamma_blue_input, 387 | step_4_post_gamma_red_input, 388 | step_4_post_gamma_green_input, 389 | step_4_post_gamma_blue_input, 390 | step_5_x_offset_input, 391 | step_5_y_offset_input, 392 | step_5_face_scale_input, 393 | step_5_face_mask_src_input, 394 | step_5_face_celeb_input, 395 | step_5_face_lmrks_input, 396 | step_5_face_mask_erode_input, 397 | step_5_face_mask_blur_input, 398 | step_5_color_transfer_input, 399 | step_5_interpolation_input, 400 | step_5_color_compression_input, 401 | step_5_face_opacity_input 402 | ): 403 | processing.fix_seed(p) 404 | seed = p.seed 405 | initial_info = None 406 | p.do_not_save_grid = True 407 | p.do_not_save_samples = True 408 | is_txt2img = isinstance(p, StableDiffusionProcessingTxt2Img) 409 | print(step_1_threshold_input) 410 | if (not is_txt2img): 411 | orig_image = p.init_images[0] 412 | else: 413 | p_txt = p 414 | p = StableDiffusionProcessingImg2Img( 415 | batch_size=p_txt.batch_size, 416 | init_images=None, 417 | resize_mode=0, 418 | denoising_strength=0.4, 419 | mask=None, 420 | mask_blur=4, 421 | inpainting_fill=1, 422 | inpaint_full_res=1, 423 | inpaint_full_res_padding=32, 424 | inpainting_mask_invert=0, 425 | sd_model=p_txt.sd_model, 426 | outpath_samples=p_txt.outpath_samples, 427 | outpath_grids=p_txt.outpath_grids, 428 | prompt=p_txt.prompt, 429 | negative_prompt=p_txt.negative_prompt, 430 | styles=p_txt.styles, 431 | seed=p_txt.seed, 432 | subseed=p_txt.subseed, 433 | subseed_strength=p_txt.subseed_strength, 434 | seed_resize_from_h=p_txt.seed_resize_from_h, 435 | seed_resize_from_w=p_txt.seed_resize_from_w, 436 | sampler_name=p_txt.sampler_name, 437 | n_iter=p_txt.n_iter, 438 | steps=p_txt.steps, 439 | cfg_scale=p_txt.cfg_scale, 440 | width=p_txt.width, 441 | height=p_txt.height, 442 | tiling=p_txt.tiling, 443 | ) 444 | p.do_not_save_grid = True 445 | p.do_not_save_samples = True 446 | p.prompt = p_txt.prompt 447 | 448 | import math 449 | output_images = [] 450 | factor_jobs = 0 451 | add_job_count = 0 452 | batch_size_dfl = 40 453 | if dfm_model_dropdown != "None": 454 | if get_model_url(dfm_model_dropdown) is not None: 455 | add_job_count += 1 456 | add_job_count = math.ceil((p.n_iter * p.batch_size)/batch_size_dfl) 457 | if enable_detection_detailer_face_checkbox: 458 | factor_jobs += 1 459 | state.job_count = p.n_iter + (p.n_iter * p.batch_size)*factor_jobs + add_job_count 460 | processed = processing.process_images(p_txt if is_txt2img else p) 461 | if initial_info is None: 462 | initial_info = processed.info 463 | final_images = [] 464 | images_count = len(processed.images) 465 | 466 | if enable_detection_detailer_face_checkbox: 467 | ddetailer_model = preload_ddetailer_model("bbox/mmdet_anime-face_yolov3.pth") 468 | 469 | model_location = dfm_model_dropdown 470 | print(dfm_model_dropdown) 471 | print(get_model_url(dfm_model_dropdown)) 472 | if get_model_url(dfm_model_dropdown) is not None: 473 | download_url_name = get_model_url(dfm_model_dropdown) 474 | download_location = str(dfl_options.get_dfl_path()) + "/" + os.path.basename(download_url_name) 475 | state.job = f"Downloading model " + dfm_model_dropdown + " from " + download_url_name + " to " + download_location 476 | download_url(download_url_name, str(dfl_options.get_dfl_path())) 477 | model_location = download_url_name.split("/")[-1] 478 | state.job_no += 1 479 | 480 | for image_n, current_image in enumerate(processed.images): 481 | text_generation = f"Generation {(image_n+1)} out of {images_count}" 482 | print(text_generation) 483 | state.job = text_generation 484 | devices.torch_gc() 485 | if is_txt2img: 486 | init_image = current_image 487 | else: 488 | init_image = orig_image 489 | 490 | if image_return_original_checkbox or (dfm_model_dropdown == "None" and not enable_detection_detailer_face_checkbox): 491 | output_images.append(init_image) 492 | 493 | if enable_detection_detailer_face_checkbox: 494 | ddscript = DetectionDetailerScript() 495 | last_no = state.job_no 496 | 497 | init_image = ddscript.run(p=p, model=ddetailer_model, model_name="bbox/mmdet_anime-face_yolov3.pth", init_image=init_image) 498 | if last_no == state.job_no: 499 | state.job_no += 1 500 | if save_detection_detailer_image_checkbox or dfm_model_dropdown == "None": 501 | output_images.append(init_image) 502 | 503 | final_images.append(init_image) 504 | 505 | if dfm_model_dropdown != "None": 506 | batches = self.generate_batches(final_images, batch_size_dfl) 507 | for batch in batches: 508 | output_images_itteration = self.process_frames(images=batch, 509 | dfm_model_dropdown=model_location, 510 | step_1_detector_input=step_1_detector_input, 511 | step_1_window_size_input=step_1_window_size_input, 512 | step_1_threshold_input=step_1_threshold_input, 513 | step_1_max_faces_input=step_1_max_faces_input, 514 | step_1_sort_by_input=step_1_sort_by_input, 515 | step_2_marker_input=step_2_marker_input, 516 | step_2_marker_coverage_input=step_2_marker_coverage_input, 517 | step_3_align_mode_input=step_3_align_mode_input, 518 | step_3_face_coverage_input=step_3_face_coverage_input, 519 | step_3_resolution_input=step_3_resolution_input, 520 | step_3_exclude_moving_parts_input=step_3_exclude_moving_parts_input, 521 | step_3_head_mode_input=step_3_head_mode_input, 522 | step_3_freeze_z_rotation_input=step_3_freeze_z_rotation_input, 523 | step_3_x_offset_input=step_3_x_offset_input, 524 | step_3_y_offset_input=step_3_y_offset_input, 525 | step_4_swap_all_faces_input=step_4_swap_all_faces_input, 526 | step_4_face_id_input=step_4_face_id_input, 527 | step_4_two_pass_input=step_4_two_pass_input, 528 | step_4_pre_sharpen_input=step_4_pre_sharpen_input, 529 | step_4_pre_gamma_red_input=step_4_pre_gamma_red_input, 530 | step_4_pre_gamma_green_input=step_4_pre_gamma_green_input, 531 | step_4_pre_gamma_blue_input=step_4_pre_gamma_blue_input, 532 | step_4_post_gamma_red_input=step_4_post_gamma_red_input, 533 | step_4_post_gamma_green_input=step_4_post_gamma_green_input, 534 | step_4_post_gamma_blue_input=step_4_post_gamma_blue_input, 535 | step_5_x_offset_input=step_5_x_offset_input, 536 | step_5_y_offset_input=step_5_y_offset_input, 537 | step_5_face_scale_input=step_5_face_scale_input, 538 | step_5_face_mask_src_input=step_5_face_mask_src_input, 539 | step_5_face_celeb_input=step_5_face_celeb_input, 540 | step_5_face_lmrks_input=step_5_face_lmrks_input, 541 | step_5_face_mask_erode_input=step_5_face_mask_erode_input, 542 | step_5_face_mask_blur_input=step_5_face_mask_blur_input, 543 | step_5_color_transfer_input=step_5_color_transfer_input, 544 | step_5_interpolation_input=step_5_interpolation_input, 545 | step_5_color_compression_input=step_5_color_compression_input, 546 | step_5_face_opacity_input=step_5_face_opacity_input) 547 | 548 | for output_image in output_images_itteration: 549 | output_images.append(output_image) 550 | 551 | state.job_no += 1 552 | 553 | if initial_info is None: 554 | initial_info = "No initial info" 555 | 556 | return Processed(p, output_images, seed, initial_info) 557 | 558 | 559 | 560 | def get_device(): 561 | device_id = shared.cmd_opts.device_id 562 | if device_id is not None: 563 | cuda_device = f"cuda:{device_id}" 564 | else: 565 | cuda_device = "cpu" 566 | return cuda_device -------------------------------------------------------------------------------- /scripts/command.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | 4 | # Get the absolute path of the directory containing the importing file 5 | current_dir = os.path.dirname(os.path.abspath(__file__)) 6 | 7 | # Append the relative path to the importing file to the system path 8 | sys.path.append(os.path.join(current_dir, '../repo/dflive/')) 9 | 10 | import numpy as np 11 | from enum import IntEnum 12 | from xlib import avecl as lib_cl 13 | from xlib.image import ImageProcessor 14 | from xlib.face import FRect, ELandmarks2D, FLandmarks2D, FPose 15 | from xlib.python import all_is_not_None 16 | from modelhub import onnx as onnx_models 17 | from modelhub import cv as cv_models 18 | from apps.DeepFaceLive.backend.BackendBase import (BackendFaceSwapInfo) 19 | from pathlib import Path 20 | from modelhub.DFLive.DFMModel import DFMModel, get_available_models_info 21 | 22 | class DetectorType(IntEnum): 23 | CENTER_FACE = 0 24 | S3FD = 1 25 | YOLOV5 = 2 26 | 27 | class FaceSortBy(IntEnum): 28 | LARGEST = 0 29 | DIST_FROM_CENTER = 1 30 | LEFT_RIGHT = 2 31 | RIGHT_LEFT = 3 32 | TOP_BOTTOM = 4 33 | BOTTOM_TOP = 5 34 | 35 | def step_1_face_detector(frame_image, 36 | device_info, 37 | detector_type=DetectorType.YOLOV5, 38 | threshold=0.5, 39 | fixed_window_size=320, 40 | sort_by=FaceSortBy.LARGEST, 41 | max_faces=3): 42 | _, H, W, _ = ImageProcessor(frame_image).get_dims() 43 | 44 | rects = [] 45 | if detector_type == DetectorType.CENTER_FACE: 46 | centerFace = onnx_models.CenterFace(device_info) 47 | rects = centerFace.extract(frame_image, threshold=threshold, fixed_window=fixed_window_size)[0] 48 | elif detector_type == DetectorType.S3FD: 49 | sd3d = onnx_models.S3FD(device_info) 50 | rects = sd3d.extract(frame_image, threshold=threshold, fixed_window=fixed_window_size)[0] 51 | elif detector_type == DetectorType.YOLOV5: 52 | yoloV5Face = onnx_models.YoloV5Face(device_info) 53 | rects = yoloV5Face.extract(frame_image, threshold=threshold, fixed_window=fixed_window_size)[0] 54 | 55 | # to list of FaceURect 56 | rects = [FRect.from_ltrb((l / W, t / H, r / W, b / H)) for l, t, r, b in rects] 57 | 58 | # sort 59 | if sort_by == FaceSortBy.LARGEST: 60 | rects = FRect.sort_by_area_size(rects) 61 | elif sort_by == FaceSortBy.DIST_FROM_CENTER: 62 | rects = FRect.sort_by_dist_from_2D_point(rects, 0.5, 0.5) 63 | elif sort_by == FaceSortBy.LEFT_RIGHT: 64 | rects = FRect.sort_by_dist_from_horizontal_point(rects, 0) 65 | elif sort_by == FaceSortBy.RIGHT_LEFT: 66 | rects = FRect.sort_by_dist_from_horizontal_point(rects, 1) 67 | elif sort_by == FaceSortBy.TOP_BOTTOM: 68 | rects = FRect.sort_by_dist_from_vertical_point(rects, 0) 69 | elif sort_by == FaceSortBy.BOTTOM_TOP: 70 | rects = FRect.sort_by_dist_from_vertical_point(rects, 1) 71 | 72 | return_faces = [] 73 | if len(rects) != 0: 74 | if max_faces != 0 and len(rects) > max_faces: 75 | rects = rects[:max_faces] 76 | 77 | for face_id, face_urect in enumerate(rects): 78 | if face_urect.get_area() != 0: 79 | fsi = BackendFaceSwapInfo() 80 | fsi.image_name = "" 81 | fsi.face_urect = face_urect 82 | return_faces.append(fsi) 83 | 84 | return return_faces 85 | 86 | class MarkerType(IntEnum): 87 | OPENCV_LBF = 0 88 | GOOGLE_FACEMESH = 1 89 | INSIGHT_2D106 = 2 90 | 91 | def step_2_face_marker(frame_image, 92 | faces, 93 | device_info, 94 | marker_type=MarkerType.INSIGHT_2D106, 95 | marker_state_coverage=1.6): 96 | if marker_type == MarkerType.OPENCV_LBF: 97 | opencv_lbf = cv_models.FaceMarkerLBF() 98 | elif marker_type == MarkerType.GOOGLE_FACEMESH: 99 | google_facemesh = onnx_models.FaceMesh(device_info) 100 | elif marker_type == MarkerType.INSIGHT_2D106: 101 | insightface_2d106 = onnx_models.InsightFace2D106(device_info) 102 | 103 | marker_coverage = marker_state_coverage 104 | if marker_coverage is None: 105 | if marker_type == MarkerType.OPENCV_LBF: 106 | marker_coverage = 1.1 107 | elif marker_type == MarkerType.GOOGLE_FACEMESH: 108 | marker_coverage = 1.4 109 | elif marker_type == MarkerType.INSIGHT_2D106: 110 | marker_coverage = 1.6 111 | 112 | is_opencv_lbf = marker_type == MarkerType.OPENCV_LBF and opencv_lbf is not None 113 | is_google_facemesh = marker_type == MarkerType.GOOGLE_FACEMESH and google_facemesh is not None 114 | is_insightface_2d106 = marker_type == MarkerType.INSIGHT_2D106 and insightface_2d106 is not None 115 | 116 | for face_iterration in enumerate(faces): 117 | face_id = face_iterration[0] 118 | fsi = face_iterration[1] 119 | if fsi.face_urect is not None: 120 | # Cut the face to feed to the face marker 121 | face_image, face_uni_mat = fsi.face_urect.cut(frame_image, marker_coverage, 122 | 256 if is_opencv_lbf else \ 123 | 192 if is_google_facemesh else \ 124 | 192 if is_insightface_2d106 else 0) 125 | _, H, W, _ = ImageProcessor(face_image).get_dims() 126 | if is_opencv_lbf: 127 | lmrks = opencv_lbf.extract(face_image)[0] 128 | elif is_google_facemesh: 129 | lmrks = google_facemesh.extract(face_image)[0] 130 | elif is_insightface_2d106: 131 | lmrks = insightface_2d106.extract(face_image)[0] 132 | 133 | if is_google_facemesh: 134 | fsi.face_pose = FPose.from_3D_468_landmarks(lmrks) 135 | 136 | if is_opencv_lbf: 137 | lmrks /= (W, H) 138 | elif is_google_facemesh: 139 | lmrks = lmrks[..., 0:2] / (W, H) 140 | elif is_insightface_2d106: 141 | lmrks = lmrks[..., 0:2] / (W, H) 142 | 143 | face_ulmrks = FLandmarks2D.create(ELandmarks2D.L68 if is_opencv_lbf else \ 144 | ELandmarks2D.L468 if is_google_facemesh else \ 145 | ELandmarks2D.L106 if is_insightface_2d106 else None, 146 | lmrks) 147 | face_ulmrks = face_ulmrks.transform(face_uni_mat, invert=True) 148 | fsi.face_ulmrks = face_ulmrks 149 | 150 | return faces 151 | 152 | class AlignMode(IntEnum): 153 | FROM_RECT = 0 154 | FROM_POINTS = 1 155 | FROM_STATIC_RECT = 2 156 | 157 | def step_3_face_aligner(frame_image, 158 | faces, 159 | align_mode=AlignMode.FROM_POINTS, 160 | head_mode=False, 161 | freeze_z_rotation=False, 162 | resolution=320, 163 | face_coverage=2.2, 164 | x_offset=0, 165 | y_offset=0, 166 | exclude_moving_parts=False): 167 | for face_iterration in enumerate(faces): 168 | face_id = face_iterration[0] 169 | fsi = face_iterration[1] 170 | head_yaw = None 171 | if head_mode or freeze_z_rotation: 172 | if fsi.face_pose is not None: 173 | head_yaw = fsi.face_pose.as_radians()[1] 174 | 175 | face_ulmrks = fsi.face_ulmrks 176 | if face_ulmrks is not None: 177 | fsi.face_resolution = resolution 178 | 179 | H, W = frame_image.shape[:2] 180 | if align_mode == AlignMode.FROM_RECT: 181 | face_align_img, uni_mat = fsi.face_urect.cut(frame_image, coverage=face_coverage, 182 | output_size=resolution, 183 | x_offset=x_offset, y_offset=y_offset) 184 | 185 | elif align_mode == AlignMode.FROM_POINTS: 186 | face_align_img, uni_mat = face_ulmrks.cut(frame_image, 187 | face_coverage + (1.0 if head_mode else 0.0), 188 | resolution, 189 | exclude_moving_parts=exclude_moving_parts, 190 | head_yaw=head_yaw, 191 | x_offset=x_offset, 192 | y_offset=y_offset - 0.08 + ( 193 | -0.50 if head_mode else 0.0), 194 | freeze_z_rotation=freeze_z_rotation) 195 | elif align_mode == AlignMode.FROM_STATIC_RECT: 196 | rect = FRect.from_ltrb([0.5 - (fsi.face_resolution / W) / 2, 0.5 - (fsi.face_resolution / H) / 2, 197 | 0.5 + (fsi.face_resolution / W) / 2, 0.5 + (fsi.face_resolution / H) / 2, ]) 198 | face_align_img, uni_mat = rect.cut(frame_image, coverage=face_coverage, 199 | output_size=resolution, 200 | x_offset=x_offset, y_offset=y_offset) 201 | 202 | fsi.face_align_image_name = f'{face_id}_aligned' 203 | fsi.image_to_align_uni_mat = uni_mat 204 | fsi.face_align_ulmrks = face_ulmrks.transform(uni_mat) 205 | 206 | # Due to FaceAligner is not well loaded, we can make lmrks mask here 207 | face_align_lmrks_mask_img = fsi.face_align_ulmrks.get_convexhull_mask(face_align_img.shape[:2], 208 | color=(255,), dtype=np.uint8) 209 | fsi.face_align_lmrks_mask_name = f'{face_id}_aligned_lmrks_mask' 210 | fsi.aligned_image = face_align_img 211 | fsi.aligned_lmrks_mask_image = face_align_lmrks_mask_img 212 | 213 | return faces 214 | 215 | def step_4_face_swapper(face_align_images, 216 | dfm_model, 217 | device_info, 218 | swap_all_faces=True, 219 | selected_face_id=1, 220 | pre_gamma=[1,1,1], 221 | post_gamma=[1,1,1], 222 | presharpen_amount=2, 223 | two_pass=False, 224 | morph_factor=0): 225 | 226 | face_output = [] 227 | 228 | for image_id, faces in enumerate(face_align_images): 229 | face_iterration_output = [] 230 | for face_align_image in faces: 231 | pre_gamma_red, pre_gamma_green, pre_gamma_blue = list(pre_gamma) 232 | post_gamma_red, post_gamma_green, post_gamma_blue = list(post_gamma) 233 | 234 | fai_ip = ImageProcessor(face_align_image) 235 | if presharpen_amount != 0: 236 | fai_ip.gaussian_sharpen(sigma=1.0, power=presharpen_amount) 237 | 238 | if pre_gamma_red != 1.0 or pre_gamma_green != 1.0 or pre_gamma_blue != 1.0: 239 | fai_ip.gamma(pre_gamma_red, pre_gamma_green, pre_gamma_blue) 240 | face_align_image = fai_ip.get_image('NHWC') 241 | 242 | celeb_face, celeb_face_mask_img, face_align_mask_img = dfm_model.convert(face_align_image, 243 | morph_factor=morph_factor) 244 | celeb_face, celeb_face_mask_img, face_align_mask_img = celeb_face[0], celeb_face_mask_img[0], \ 245 | face_align_mask_img[0] 246 | 247 | if two_pass: 248 | celeb_face, celeb_face_mask_img, _ = dfm_model.convert(celeb_face, 249 | morph_factor=morph_factor) 250 | celeb_face, celeb_face_mask_img = celeb_face[0], celeb_face_mask_img[0] 251 | 252 | if post_gamma_red != 1.0 or post_gamma_blue != 1.0 or post_gamma_green != 1.0: 253 | celeb_face = ImageProcessor(celeb_face).gamma(post_gamma_red, post_gamma_blue, 254 | post_gamma_green).get_image('HWC') 255 | 256 | face_iterration_output.append([face_align_mask_img, celeb_face, celeb_face_mask_img]) 257 | face_output.append(face_iterration_output) 258 | 259 | return face_output 260 | 261 | _cpu_interp = {'bilinear' : ImageProcessor.Interpolation.LINEAR, 262 | 'bicubic' : ImageProcessor.Interpolation.CUBIC, 263 | 'lanczos4' : ImageProcessor.Interpolation.LANCZOS4} 264 | 265 | def step_5_merge_on_cpu(frame_image, face_resolution, face_align_img, face_align_mask_img, face_align_lmrks_mask_img, face_swap_img, face_swap_mask_img, aligned_to_source_uni_mat, frame_width, frame_height, do_color_compression, interpolation, face_mask_source, face_mask_celeb, face_mask_lmrks, face_mask_erode, face_mask_blur, color_transfer, face_opacity, color_compression ): 266 | import numexpr as ne 267 | interpolation = _cpu_interp[interpolation] 268 | 269 | frame_image = ImageProcessor(frame_image).to_ufloat32().get_image('HWC') 270 | 271 | masks = [] 272 | if face_mask_source: 273 | masks.append( ImageProcessor(face_align_mask_img).to_ufloat32().get_image('HW') ) 274 | if face_mask_celeb: 275 | masks.append( ImageProcessor(face_swap_mask_img).to_ufloat32().get_image('HW') ) 276 | if face_mask_lmrks: 277 | masks.append( ImageProcessor(face_align_lmrks_mask_img).to_ufloat32().get_image('HW') ) 278 | 279 | masks_count = len(masks) 280 | if masks_count == 0: 281 | face_mask = np.ones(shape=(face_resolution, face_resolution), dtype=np.float32) 282 | else: 283 | face_mask = masks[0] 284 | for i in range(1, masks_count): 285 | face_mask *= masks[i] 286 | 287 | # Combine face mask 288 | face_mask = ImageProcessor(face_mask).erode_blur(face_mask_erode, face_mask_blur, fade_to_border=True).get_image('HWC') 289 | frame_face_mask = ImageProcessor(face_mask).warp_affine(aligned_to_source_uni_mat, frame_width, frame_height).clip2( (1.0/255.0), 0.0, 1.0, 1.0).get_image('HWC') 290 | 291 | face_swap_ip = ImageProcessor(face_swap_img).to_ufloat32() 292 | 293 | if color_transfer == 'rct': 294 | face_swap_img = face_swap_ip.rct(like=face_align_img, mask=face_mask, like_mask=face_mask) 295 | 296 | frame_face_swap_img = face_swap_ip.warp_affine(aligned_to_source_uni_mat, frame_width, frame_height, interpolation=interpolation).get_image('HWC') 297 | 298 | # Combine final frame 299 | opacity = np.float32(face_opacity) 300 | one_f = np.float32(1.0) 301 | if opacity == 1.0: 302 | out_merged_frame = ne.evaluate('frame_image*(one_f-frame_face_mask) + frame_face_swap_img*frame_face_mask') 303 | else: 304 | out_merged_frame = ne.evaluate('frame_image*(one_f-frame_face_mask) + frame_image*frame_face_mask*(one_f-opacity) + frame_face_swap_img*frame_face_mask*opacity') 305 | 306 | if do_color_compression and color_compression != 0: 307 | color_compression = max(4, (127.0 - color_compression) ) 308 | out_merged_frame *= color_compression 309 | np.floor(out_merged_frame, out=out_merged_frame) 310 | out_merged_frame /= color_compression 311 | out_merged_frame += 2.0 / color_compression 312 | 313 | return out_merged_frame 314 | 315 | _gpu_interp = {'bilinear' : lib_cl.EInterpolation.LINEAR, 316 | 'bicubic' : lib_cl.EInterpolation.CUBIC, 317 | 'lanczos4' : lib_cl.EInterpolation.LANCZOS4} 318 | 319 | _n_mask_multiply_op_text = [ f"float X = {'*'.join([f'(((float)I{i}) / 255.0)' for i in range(n)])}; O = (X <= 0.5 ? 0 : 1);" for n in range(5) ] 320 | 321 | def step_5_merge_on_gpu(frame_image, face_resolution, face_align_img, face_align_mask_img, face_align_lmrks_mask_img, face_swap_img, face_swap_mask_img, aligned_to_source_uni_mat, frame_width, frame_height, do_color_compression, interpolation, face_mask_source, face_mask_celeb, face_mask_lmrks, face_mask_erode, face_mask_blur, color_transfer, face_opacity, color_compression ): 322 | interpolation = _gpu_interp[interpolation] 323 | 324 | masks = [] 325 | if face_mask_source: 326 | masks.append( lib_cl.Tensor.from_value(face_align_mask_img) ) 327 | if face_mask_celeb: 328 | masks.append( lib_cl.Tensor.from_value(face_swap_mask_img) ) 329 | if face_mask_lmrks: 330 | masks.append( lib_cl.Tensor.from_value(face_align_lmrks_mask_img) ) 331 | 332 | masks_count = len(masks) 333 | if masks_count == 0: 334 | face_mask_t = lib_cl.Tensor(shape=(face_resolution, face_resolution), dtype=np.float32, initializer=lib_cl.InitConst(1.0)) 335 | else: 336 | face_mask_t = lib_cl.any_wise(_n_mask_multiply_op_text[masks_count], *masks, dtype=np.uint8).transpose( (2,0,1) ) 337 | 338 | face_mask_t = lib_cl.binary_morph(face_mask_t, face_mask_erode, face_mask_blur, fade_to_border=True, dtype=np.float32) 339 | face_swap_img_t = lib_cl.Tensor.from_value(face_swap_img ).transpose( (2,0,1), op_text='O = ((O_TYPE)I) / 255.0', dtype=np.float32) 340 | 341 | if color_transfer == 'rct': 342 | face_align_img_t = lib_cl.Tensor.from_value(face_align_img).transpose( (2,0,1), op_text='O = ((O_TYPE)I) / 255.0', dtype=np.float32) 343 | face_swap_img_t = lib_cl.rct(face_swap_img_t, face_align_img_t, target_mask_t=face_mask_t, source_mask_t=face_mask_t) 344 | 345 | frame_face_mask_t = lib_cl.remap_np_affine(face_mask_t, aligned_to_source_uni_mat, interpolation=lib_cl.EInterpolation.LINEAR, output_size=(frame_height, frame_width), post_op_text='O = (O <= (1.0/255.0) ? 0.0 : O > 1.0 ? 1.0 : O);' ) 346 | frame_face_swap_img_t = lib_cl.remap_np_affine(face_swap_img_t, aligned_to_source_uni_mat, interpolation=interpolation, output_size=(frame_height, frame_width), post_op_text='O = clamp(O, 0.0, 1.0);' ) 347 | 348 | frame_image_t = lib_cl.Tensor.from_value(frame_image).transpose( (2,0,1), op_text='O = ((float)I) / 255.0;' if frame_image.dtype == np.uint8 else None, 349 | dtype=np.float32 if frame_image.dtype == np.uint8 else None) 350 | 351 | opacity = face_opacity 352 | if opacity == 1.0: 353 | frame_final_t = lib_cl.any_wise('O = I0*(1.0-I1) + I2*I1', frame_image_t, frame_face_mask_t, frame_face_swap_img_t, dtype=np.float32) 354 | else: 355 | frame_final_t = lib_cl.any_wise('O = I0*(1.0-I1) + I0*I1*(1.0-I3) + I2*I1*I3', frame_image_t, frame_face_mask_t, frame_face_swap_img_t, np.float32(opacity), dtype=np.float32) 356 | 357 | if do_color_compression and color_compression != 0: 358 | color_compression = max(4, (127.0 - color_compression) ) 359 | frame_final_t = lib_cl.any_wise('O = ( floor(I0 * I1) / I1 ) + (2.0 / I1);', frame_final_t, np.float32(color_compression)) 360 | 361 | return frame_final_t.transpose( (1,2,0) ).np() 362 | 363 | def step_5_face_merger(frame_image, 364 | faces, 365 | device_info, 366 | face_x_offset=0, 367 | face_y_offset=0, 368 | face_scale=1, 369 | interpolation='bilinear', 370 | face_mask_source=True, 371 | face_mask_celeb=True, 372 | face_mask_lmrks=False, 373 | face_mask_erode=5, 374 | face_mask_blur=25, 375 | color_transfer=False, 376 | face_opacity=1, 377 | color_compression=0): 378 | 379 | merged_frame = frame_image 380 | 381 | if merged_frame is not None: 382 | fsi_list = faces 383 | fsi_list_len = len(fsi_list) 384 | has_merged_faces = False 385 | 386 | for face_iterration in enumerate(faces): 387 | fsi_id = face_iterration[0] 388 | fsi = face_iterration[1] 389 | 390 | image_to_align_uni_mat = fsi.image_to_align_uni_mat 391 | face_resolution = fsi.face_resolution 392 | 393 | face_align_img = fsi.aligned_image 394 | face_align_lmrks_mask_img = fsi.aligned_lmrks_mask_image 395 | face_align_mask_img = fsi.face_align_mask_image 396 | face_swap_img = fsi.face_swap_image_image 397 | face_swap_mask_img = fsi.face_swap_mask_image 398 | 399 | if all_is_not_None(face_resolution, face_align_img, face_align_mask_img, face_swap_img, 400 | face_swap_mask_img, image_to_align_uni_mat): 401 | face_height, face_width = face_align_img.shape[:2] 402 | frame_height, frame_width = merged_frame.shape[:2] 403 | aligned_to_source_uni_mat = image_to_align_uni_mat.invert() 404 | aligned_to_source_uni_mat = aligned_to_source_uni_mat.source_translated(-face_x_offset, 405 | -face_y_offset) 406 | aligned_to_source_uni_mat = aligned_to_source_uni_mat.source_scaled_around_center(face_scale, 407 | face_scale) 408 | aligned_to_source_uni_mat = aligned_to_source_uni_mat.to_exact_mat(face_width, face_height, 409 | frame_width, frame_height) 410 | 411 | do_color_compression = fsi_id == fsi_list_len - 1 412 | if str(device_info) == 'CPU' or True: 413 | merged_frame = step_5_merge_on_cpu(merged_frame, face_resolution, face_align_img, 414 | face_align_mask_img, face_align_lmrks_mask_img, face_swap_img, 415 | face_swap_mask_img, aligned_to_source_uni_mat, frame_width, 416 | frame_height, do_color_compression, interpolation, face_mask_source, face_mask_celeb, face_mask_lmrks, face_mask_erode, face_mask_blur, color_transfer, face_opacity, color_compression) 417 | else: 418 | merged_frame = step_5_merge_on_gpu(merged_frame, face_resolution, face_align_img, 419 | face_align_mask_img, face_align_lmrks_mask_img, face_swap_img, 420 | face_swap_mask_img, aligned_to_source_uni_mat, frame_width, 421 | frame_height, do_color_compression, interpolation, face_mask_source, face_mask_celeb, face_mask_lmrks, face_mask_erode, face_mask_blur, color_transfer, face_opacity, color_compression) 422 | 423 | return merged_frame 424 | 425 | from xlib.onnxruntime import (get_available_devices_info) 426 | 427 | def execute_deep_face_live_multiple( numpy_images, 428 | dfm_path, 429 | device_id = 0, 430 | step_1_detector = DetectorType.YOLOV5, 431 | step_1_window_size = 320, 432 | step_1_threshold = 0.5, 433 | step_1_max_faces = 3, 434 | step_1_sort_by = FaceSortBy.LARGEST, 435 | step_2_marker = MarkerType.INSIGHT_2D106, 436 | step_2_marker_coverage = 1.6, 437 | step_3_align_mode = AlignMode.FROM_POINTS, 438 | step_3_face_coverage = 2.2, 439 | step_3_resolution = 320, 440 | step_3_exclude_moving_parts = True, 441 | step_3_head_mode = False, 442 | step_3_freeze_z_rotation = False, 443 | step_3_x_offset = 0, 444 | step_3_y_offset = 0, 445 | step_4_swap_all_faces = True, 446 | step_4_face_id = 0, 447 | step_4_two_pass = False, 448 | step_4_pre_sharpen = 0.5, 449 | step_4_pre_gamma_red = 1, 450 | step_4_pre_gamma_green = 1, 451 | step_4_pre_gamma_blue = 1, 452 | step_4_post_gamma_red = 1, 453 | step_4_post_gamma_green = 1, 454 | step_4_post_gamma_blue = 1, 455 | step_5_x_offset=0, 456 | step_5_y_offset=0, 457 | step_5_face_scale=1, 458 | step_5_face_mask_src=True, 459 | step_5_face_celeb=True, 460 | step_5_face_lmrks=False, 461 | step_5_face_mask_erode=5, 462 | step_5_face_mask_blur=25, 463 | step_5_color_transfer='rct', 464 | step_5_interpolation='bilinear', 465 | step_5_color_compression=0, 466 | step_5_face_opacity=1 467 | ): 468 | faces_array = [] 469 | for image_id, numpy_image in enumerate(numpy_images): 470 | print("Numpy array shape:", numpy_image.shape) 471 | available_devices = get_available_devices_info() 472 | device_info = available_devices[int(device_id)] 473 | 474 | print("Step 1. Face Detector") 475 | faces = step_1_face_detector(frame_image=numpy_image, 476 | device_info=device_info, 477 | detector_type=int(step_1_detector), 478 | threshold=float(step_1_threshold), 479 | fixed_window_size=int(step_1_window_size), 480 | sort_by=int(step_1_sort_by), 481 | max_faces=int(step_1_max_faces)) 482 | 483 | print("Step 2. Face Marker") 484 | faces = step_2_face_marker(frame_image=numpy_image, 485 | faces=faces, 486 | device_info=device_info, 487 | marker_type=step_2_marker, 488 | marker_state_coverage=step_2_marker_coverage) 489 | 490 | print("Step 3. Face Aligner") 491 | faces = step_3_face_aligner(frame_image=numpy_image, 492 | faces=faces, 493 | align_mode=step_3_align_mode, 494 | head_mode=step_3_head_mode, 495 | freeze_z_rotation=step_3_freeze_z_rotation, 496 | resolution=step_3_resolution, 497 | face_coverage=step_3_face_coverage, 498 | x_offset=step_3_x_offset, 499 | y_offset=step_3_y_offset, 500 | exclude_moving_parts=step_3_exclude_moving_parts) 501 | 502 | faces_array.append(faces) 503 | 504 | # Note: There is a very weird issue with Step 4, when run through web ui, faces come out bluish and the issue is 505 | # not present when run through the command line. This is a workaround to fix the issue. The issue itself is in the 506 | # tensorflow library, and it is not present in the onnx version of the library. 507 | 508 | print("Step 4. Face Swapper") 509 | faces_array = step_4_face_swapper_remote(faces_array=faces_array, 510 | dfm_model=dfm_path, 511 | device_id=1, 512 | swap_all_faces=step_4_swap_all_faces, 513 | selected_face_id=int(step_4_face_id), 514 | pre_gamma=[step_4_pre_gamma_red, step_4_pre_gamma_green, step_4_pre_gamma_blue], 515 | post_gamma=[step_4_post_gamma_red, step_4_post_gamma_green, step_4_post_gamma_blue], 516 | presharpen_amount=step_4_pre_sharpen, 517 | two_pass=step_4_two_pass) 518 | 519 | # print("Step 4. Face Swapper") 520 | # faces = step_4_face_swapper(faces=faces, 521 | # dfm_model=dfm_path, 522 | # device_info=device_info, 523 | # swap_all_faces=step_4_swap_all_faces, 524 | # selected_face_id=int(step_4_face_id), 525 | # pre_gamma=[step_4_pre_gamma_red, step_4_pre_gamma_green, step_4_pre_gamma_blue], 526 | # post_gamma=[step_4_post_gamma_red, step_4_post_gamma_green, step_4_post_gamma_blue], 527 | # presharpen_amount=step_4_pre_sharpen, 528 | # two_pass=step_4_two_pass) 529 | 530 | # for face_iterration in enumerate(faces): 531 | # i = face_iterration[0] 532 | # fsi = face_iterration[1] 533 | # return fsi.face_swap_image_image 534 | # return ImageProcessor(fsi.aligned_image).get_image('HWC') 535 | 536 | output_images = [] 537 | for image_id, faces in enumerate(faces_array): 538 | print("Step 5. Face Merger") 539 | output_images.append(step_5_face_merger(frame_image=numpy_images[image_id], 540 | faces=faces, 541 | device_info=device_info, 542 | face_x_offset=int(step_5_x_offset), 543 | face_y_offset=int(step_5_y_offset), 544 | face_scale=int(step_5_face_scale), 545 | interpolation=str(step_5_interpolation), 546 | face_mask_source=bool(step_5_face_mask_src), 547 | face_mask_celeb=bool(step_5_face_celeb), 548 | face_mask_lmrks=bool(step_5_face_lmrks), 549 | face_mask_erode=int(step_5_face_mask_erode), 550 | face_mask_blur=int(step_5_face_mask_blur), 551 | color_transfer=str(step_5_color_transfer), 552 | face_opacity=int(step_5_face_opacity), 553 | color_compression=int(step_5_color_compression))) 554 | 555 | return output_images 556 | 557 | def step_4_face_swapper_remote(faces_array, 558 | dfm_model, 559 | device_id, 560 | swap_all_faces=True, 561 | selected_face_id=1, 562 | pre_gamma=[1,1,1], 563 | post_gamma=[1,1,1], 564 | presharpen_amount=5, 565 | two_pass=False, 566 | morph_factor=0): 567 | import tempfile 568 | import subprocess 569 | import os 570 | import json 571 | from PIL import Image 572 | import base64 573 | import cv2 574 | 575 | faces_tmp_images = [] 576 | 577 | for image_id, faces in enumerate(faces_array): 578 | face_iterration_images = [] 579 | for face_iterration in enumerate(faces): 580 | face_id = face_iterration[0] 581 | fsi = face_iterration[1] 582 | 583 | # Create a temporary file to write the PNG image data to 584 | with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as tmp_file: 585 | # Use the Pillow library to write the NumPy array to the temporary file as a PNG image 586 | img = Image.fromarray(fsi.aligned_image) 587 | img.save(tmp_file, "PNG") 588 | 589 | # Get the filename of the temporary file 590 | tmp_filename = tmp_file.name 591 | face_iterration_images.append(tmp_filename) 592 | faces_tmp_images.append(face_iterration_images) 593 | 594 | faces_tmp_images_json = base64.b64encode(json.dumps(faces_tmp_images).encode("utf-8")).decode("utf-8") 595 | current_file = os.path.abspath(__file__) 596 | 597 | python_call = ['python', 598 | current_file, 599 | faces_tmp_images_json, 600 | dfm_model, 601 | str(device_id), 602 | str(swap_all_faces), 603 | str(selected_face_id), 604 | str(presharpen_amount), 605 | str(two_pass)] 606 | 607 | print("Calling: " + " ".join(str(x) for x in python_call)) 608 | 609 | result = subprocess.run(python_call, capture_output=True, text=True) 610 | 611 | try: 612 | # Try to parse the JSON output 613 | result = json.loads(result.stdout) 614 | except json.JSONDecodeError: 615 | # Handle the case where the output is not valid JSON 616 | print("Output is not valid JSON") 617 | return 618 | 619 | for image_id, faces_output in enumerate(result): 620 | for face_id in enumerate(faces_output): 621 | print(face_id) 622 | fsi = faces_array[image_id][face_id[0]] 623 | swap_swap_image = cv2.imread(face_id[1][1]) 624 | swap_swap_image = cv2.cvtColor(swap_swap_image, cv2.COLOR_BGR2RGB) 625 | fsi.face_swap_image_image = swap_swap_image 626 | fsi.face_align_mask_image = np.array(cv2.imread(face_id[1][0])) 627 | fsi.face_swap_mask_image = np.array(cv2.imread(face_id[1][2])) 628 | 629 | return faces_array 630 | 631 | def get_downloadable_models(models_path, available_models): 632 | models_info = get_available_models_info(models_path) 633 | return_models = [] 634 | model_names = [] 635 | for available_model in available_models: 636 | model_names.append(os.path.basename(available_model)) 637 | 638 | for model in models_info: 639 | url = model.get_url() 640 | if url is not None and os.path.basename(model.get_model_path()) not in model_names: 641 | return_models.append([model.get_name(), url]) 642 | 643 | return return_models 644 | 645 | # Here is step 4 called through the remote function, which is called from the web UI 646 | if __name__ == '__main__': 647 | import cv2 648 | import json 649 | import tempfile 650 | from PIL import Image 651 | import base64 652 | 653 | # Get the image file path and DFL path from command line arguments 654 | faces_tmp_images_json = sys.argv[1] 655 | dfm_model = sys.argv[2] 656 | device_id = int(sys.argv[3]) 657 | swap_all_faces = bool(sys.argv[4]) 658 | selected_face_id = int(sys.argv[5]) 659 | presharpen_amount = float(sys.argv[6]) 660 | two_pass = bool(sys.argv[7]) 661 | 662 | available_devices = get_available_devices_info() 663 | device_info = available_devices[int(device_id)] 664 | try: 665 | # Decode the output to array 666 | decoded_string = base64.b64decode(faces_tmp_images_json).decode("utf-8") 667 | faces_tmp_images = json.loads(decoded_string) 668 | except json.JSONDecodeError: 669 | error_dict = {"error": "faces_tmp_images_json is not readable. Invalid JSON."} 670 | error_json = json.dumps(error_dict) 671 | print(error_json) 672 | exit() 673 | 674 | faces_numpy = [] 675 | for image_id, faces in enumerate(faces_tmp_images): 676 | face_iterration_images = [] 677 | for face in faces: 678 | # Load the image using OpenCV 679 | image = cv2.imread(face) 680 | 681 | # Convert the image to a numpy array 682 | numpy_array = np.array(image) 683 | face_iterration_images.append(numpy_array) 684 | faces_numpy.append(face_iterration_images) 685 | 686 | path = Path(dfm_model) 687 | dfm_model = DFMModel(path, device_info) 688 | 689 | output = step_4_face_swapper(face_align_images=faces_numpy, 690 | dfm_model=dfm_model, 691 | device_info=device_info, 692 | swap_all_faces=swap_all_faces, 693 | selected_face_id=selected_face_id, 694 | presharpen_amount=presharpen_amount, 695 | two_pass=two_pass) 696 | output_files = [] 697 | for image_id, output_data in enumerate(output): 698 | output_itteration = [] 699 | for face in output_data: 700 | # Create a temporary file to write the PNG image data to 701 | with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as tmp_file: 702 | # Write the numpy array to an image file using OpenCV 703 | cv2.imwrite(tmp_file.name, face[0]) 704 | 705 | # Get the filename of the temporary file 706 | face_align_mask_img = tmp_file.name 707 | 708 | with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as tmp_file2: 709 | # Use the Pillow library to write the NumPy array to the temporary file as a PNG image 710 | cv2.imwrite(tmp_file2.name, face[1]) 711 | 712 | # Get the filename of the temporary file 713 | celeb_face = tmp_file2.name 714 | 715 | with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as tmp_file3: 716 | # Use the Pillow library to write the NumPy array to the temporary file as a PNG image 717 | cv2.imwrite(tmp_file3.name, face[2]) 718 | 719 | # Get the filename of the temporary file 720 | celeb_face_mask_img = tmp_file3.name 721 | 722 | output_itteration.append([face_align_mask_img, celeb_face, celeb_face_mask_img]) 723 | output_files.append(output_itteration) 724 | 725 | output_json = json.dumps(output_files) 726 | print(output_json) 727 | 728 | # if __name__ == '__main__': 729 | # # Get the image file path and DFL path from command line arguments 730 | # image_path = sys.argv[1] 731 | # dfl_path = sys.argv[2] 732 | # device_id = sys.argv[3] 733 | # 734 | # # Load the image using OpenCV 735 | # image = cv2.imread(image_path) 736 | # 737 | # # Convert the image to a numpy array 738 | # numpy_array = np.array(image) 739 | # 740 | # final_image = execute_deep_face_live(numpy_array, dfl_path, device_id) 741 | # img = ImageProcessor(final_image, copy=True).to_uint8().get_image('HWC') 742 | # cv2.imwrite('output.jpg', img) 743 | 744 | --------------------------------------------------------------------------------