├── Colab ├── AdelaiDepth │ ├── AdelaiDepth-demo-image.PNG │ ├── AdelaiDepth_ResNet101_Depth_Estimation.ipynb │ └── README.md ├── BioGPT │ ├── BioGPT_HF_Transformers.ipynb │ ├── README.md │ └── images │ │ └── BioGPT_example.png ├── C-GPT │ ├── C-ChatGPT-Colab-Example.PNG │ ├── C_ChatGPT.ipynb │ └── README.md ├── DFR │ ├── Deep_Feature_Rotation_Simple.ipynb │ └── README.md ├── DRPreter │ ├── DRPreter_Evaluation.ipynb │ └── README.md ├── DieSitCom │ ├── DieSitCom_Colab.ipynb │ ├── README.md │ └── demo-image.PNG ├── DriveGAN │ ├── DriveGAN_Playground.ipynb │ ├── README.md │ └── Simulator-in-Colab.PNG ├── FLAVR │ ├── Fast_Frame_Interpolation_with_FLAVR.ipynb │ └── README.md ├── FSGAN │ ├── FSGAN_demo_image.PNG │ ├── Few_shot_GAN_TensorFlow_Inference.ipynb │ └── README.md ├── GLIDE │ ├── GLIDE - Inpainting.ipynb │ ├── README.md │ ├── lion.png │ └── source_image.jpg ├── GPNN │ ├── Efficient_GPNN.ipynb │ └── README.md ├── MiniCLIP │ ├── MiniCLIP_Notebook.ipynb │ └── README.md ├── MobileFaceSwap │ ├── MobileFaceSwap.ipynb │ ├── README.md │ └── demo_images │ │ ├── result.jpg │ │ ├── source.jpg │ │ └── target.jpg ├── OWL-ViT │ ├── OWL_ViT_Evaluation.ipynb │ ├── README.md │ └── images │ │ └── OWL-ViT-Example.png ├── PackNet-SfM │ ├── PackNet_SfM_Inference.ipynb │ ├── README.md │ └── demo-image.png ├── Pix2Pix │ ├── Pix2Pix_for_Satellite_to_Map_Image_Translation_Data_Preparation.ipynb │ ├── Pix2Pix_for_Satellite_to_Map_Image_Translation_Model_Training.ipynb │ └── README.md ├── ReplitLM │ ├── Code_Generation_with_ReplitLM.ipynb │ └── README.md ├── SDImageVariations │ ├── README.md │ ├── Stable_Diffusion_Image_Variations.ipynb │ └── images │ │ └── sample_image.jpg ├── SPADE-TF │ ├── README.md │ ├── SPADE_in_TensorFlow_Inference.ipynb │ └── SPADE_in_TensorFlow_Training.ipynb ├── SinGAN │ ├── README.md │ └── SinGAN_TensorFlow.ipynb ├── StyleCariGAN │ ├── Caricatures.PNG │ ├── README.md │ └── StyleCariGAN_Evaluation.ipynb ├── TBEFN │ ├── README.md │ ├── TBEFN_TF_2.ipynb │ └── TBEFN_demo_image.PNG ├── VersatileDiffusion │ ├── README.md │ ├── Versatile_Diffusion.ipynb │ └── images │ │ ├── dual_guided_example_0.png │ │ ├── dual_guided_example_1.png │ │ ├── dual_guided_example_2.png │ │ ├── image_variation_example_1.png │ │ ├── image_variation_example_2.png │ │ ├── source_image_0.jpg │ │ ├── source_image_1.jpg │ │ └── text2img_example.png ├── ViT │ ├── README.md │ └── Vision Transformer - Pytorch.ipynb └── Zero-shot-image │ ├── Few-shot-image_demo.PNG │ ├── README.md │ └── Zero_shot_Image_to_text.ipynb ├── Jupyter └── CLIP │ ├── CLIP - Semantic Search on Unseen Images - Local.ipynb │ ├── README.md │ └── requirements.txt ├── LICENSE └── README.md /Colab/AdelaiDepth/AdelaiDepth-demo-image.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/AdelaiDepth/AdelaiDepth-demo-image.PNG -------------------------------------------------------------------------------- /Colab/AdelaiDepth/README.md: -------------------------------------------------------------------------------- 1 | # AdelaiDepth ResNet101 Test 2 | A notebook to test monocular depth estimation and 3D scene reconstruction from single RGB images using the Open Source [AdelaiDepth](https://github.com/aim-uofa/AdelaiDepth) toolbox. This notebook is derived from the original one available in the official GitHub repo for the toolbox and other than some code refactoring, it includes also the possibility to perform 3D scene reconstruction and visualize 3D Point Cloud. 3 | ![Demo Image](https://github.com/virtualramblas/python-notebooks-repo/blob/main/Colab/AdelaiDepth/AdelaiDepth-demo-image.PNG) 4 | -------------------------------------------------------------------------------- /Colab/BioGPT/BioGPT_HF_Transformers.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "provenance": [] 7 | }, 8 | "kernelspec": { 9 | "name": "python3", 10 | "display_name": "Python 3" 11 | }, 12 | "language_info": { 13 | "name": "python" 14 | } 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "source": [ 20 | "# BioGPT with HF Transformers\n", 21 | "A notebook to evaluate [BioGPT](https://academic.oup.com/bib/article/23/6/bbac409/6713511?guestAccessKey=a66d9b5d-4f83-4017-bb52-405815c907b9), Microsoft's domain-specific generative Transformer language model pre-trained on large-scale biomedical literature. \n", 22 | "No hardware acceleration needed to execute the code in this notebook." 23 | ], 24 | "metadata": { 25 | "id": "RrqxgftIonuE" 26 | } 27 | }, 28 | { 29 | "cell_type": "markdown", 30 | "source": [ 31 | "### Settings" 32 | ], 33 | "metadata": { 34 | "id": "pkXi_eRNf4AJ" 35 | } 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "source": [ 40 | "Install the missing requirements in the Colab VM (Hugging Face's Transformer and sacremoses)." 41 | ], 42 | "metadata": { 43 | "id": "q1fjMgBVk3zk" 44 | } 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": { 50 | "id": "-jRokZ8Okq99" 51 | }, 52 | "outputs": [], 53 | "source": [ 54 | "!pip install transformers sacremoses" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "source": [ 60 | "Import the necessary packages/classes." 61 | ], 62 | "metadata": { 63 | "id": "o8E0DnnSk8gF" 64 | } 65 | }, 66 | { 67 | "cell_type": "code", 68 | "source": [ 69 | "import torch\n", 70 | "from transformers import pipeline, set_seed\n", 71 | "from transformers import BioGptTokenizer, BioGptForCausalLM" 72 | ], 73 | "metadata": { 74 | "id": "lx0NSUeWlDoy" 75 | }, 76 | "execution_count": null, 77 | "outputs": [] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "source": [ 82 | "Load a pretrained model." 83 | ], 84 | "metadata": { 85 | "id": "ynv_4OCilEba" 86 | } 87 | }, 88 | { 89 | "cell_type": "code", 90 | "source": [ 91 | "model = BioGptForCausalLM.from_pretrained(\"microsoft/biogpt\")\n", 92 | "tokenizer = BioGptTokenizer.from_pretrained(\"microsoft/biogpt\")" 93 | ], 94 | "metadata": { 95 | "id": "xqyEyCLZlIRg" 96 | }, 97 | "execution_count": null, 98 | "outputs": [] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "source": [ 103 | "### Text Generation" 104 | ], 105 | "metadata": { 106 | "id": "urarK2o8FY-l" 107 | } 108 | }, 109 | { 110 | "cell_type": "markdown", 111 | "source": [ 112 | "Create the generation pipeline." 113 | ], 114 | "metadata": { 115 | "id": "SkjhNVn4lIxB" 116 | } 117 | }, 118 | { 119 | "cell_type": "code", 120 | "source": [ 121 | "generator = pipeline('text-generation', model=model, tokenizer=tokenizer)\n", 122 | "set_seed(42)" 123 | ], 124 | "metadata": { 125 | "id": "OFFsh7VGlQDc" 126 | }, 127 | "execution_count": null, 128 | "outputs": [] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "source": [ 133 | "Set the prompt for the model, the maximum lenght of each generated text sequence and the maximum number of sequences to generate." 134 | ], 135 | "metadata": { 136 | "id": "2-V2RmTXgNSa" 137 | } 138 | }, 139 | { 140 | "cell_type": "code", 141 | "source": [ 142 | "prompt = \"Psoralen is\" #@param {type: \"string\"}\n", 143 | "generated_sequence_max_length = 60 #@param {type:\"slider\", min:10, max:200, step:1}\n", 144 | "num_return_sequences = 3 #@param {type:\"slider\", min:1, max:20, step:1}" 145 | ], 146 | "metadata": { 147 | "id": "sD0UbcHoBmSW" 148 | }, 149 | "execution_count": null, 150 | "outputs": [] 151 | }, 152 | { 153 | "cell_type": "markdown", 154 | "source": [ 155 | "Generate text. The generated sequences are printed to the code cell output." 156 | ], 157 | "metadata": { 158 | "id": "DzxLj-l5lQ2r" 159 | } 160 | }, 161 | { 162 | "cell_type": "code", 163 | "source": [ 164 | "generator(prompt, \n", 165 | " max_length=generated_sequence_max_length, \n", 166 | " num_return_sequences=num_return_sequences, \n", 167 | " do_sample=True)" 168 | ], 169 | "metadata": { 170 | "id": "TzNpoYvBlUrO" 171 | }, 172 | "execution_count": null, 173 | "outputs": [] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "source": [ 178 | "### Beam-Search Decoding" 179 | ], 180 | "metadata": { 181 | "id": "2MWYQvHMBI9y" 182 | } 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "source": [ 187 | "Set the minimum and max lenght of the generated text and the number of beams. The prompt for the model is the same set as for previous form for text generation." 188 | ], 189 | "metadata": { 190 | "id": "7bt3atDxgpQR" 191 | } 192 | }, 193 | { 194 | "cell_type": "code", 195 | "source": [ 196 | "generated_text_min_length = 100 #@param {type:\"slider\", min:10, max:200, step:1}\n", 197 | "generated_text_max_length = 1024 #@param {type:\"slider\", min:300, max:1200, step:1}\n", 198 | "num_beams = 5 #@param {type:\"slider\", min:1, max:10, step:1}" 199 | ], 200 | "metadata": { 201 | "id": "TgZjfnFhDwti" 202 | }, 203 | "execution_count": null, 204 | "outputs": [] 205 | }, 206 | { 207 | "cell_type": "markdown", 208 | "source": [ 209 | "Get the feature of the given prompt in PyTorch format." 210 | ], 211 | "metadata": { 212 | "id": "IbP7ngGDhJKb" 213 | } 214 | }, 215 | { 216 | "cell_type": "code", 217 | "source": [ 218 | "inputs = tokenizer(prompt, return_tensors=\"pt\")" 219 | ], 220 | "metadata": { 221 | "id": "8x2sfCELBPP8" 222 | }, 223 | "execution_count": null, 224 | "outputs": [] 225 | }, 226 | { 227 | "cell_type": "markdown", 228 | "source": [ 229 | "Do beam-search decoding. The generated text is printed to the code cell output." 230 | ], 231 | "metadata": { 232 | "id": "keoSgGc7hk1u" 233 | } 234 | }, 235 | { 236 | "cell_type": "code", 237 | "source": [ 238 | "with torch.no_grad():\n", 239 | " beam_output = model.generate(**inputs,\n", 240 | " min_length=generated_text_min_length,\n", 241 | " max_length=generated_text_max_length,\n", 242 | " num_beams=num_beams,\n", 243 | " early_stopping=True\n", 244 | " )\n", 245 | "tokenizer.decode(beam_output[0], skip_special_tokens=True)" 246 | ], 247 | "metadata": { 248 | "id": "nRfQnx3PDlf8" 249 | }, 250 | "execution_count": null, 251 | "outputs": [] 252 | } 253 | ] 254 | } -------------------------------------------------------------------------------- /Colab/BioGPT/README.md: -------------------------------------------------------------------------------- 1 | # BioGPT with HF Transformers 2 | A notebook to evaluate [BioGPT](https://academic.oup.com/bib/article/23/6/bbac409/6713511?guestAccessKey=a66d9b5d-4f83-4017-bb52-405815c907b9), Microsoft's domain-specific generative Transformer language model pre-trained on large-scale biomedical literature, through the Hugging Face's [Transformers](https://huggingface.co/docs/transformers/index) library. 3 | The code in this notebooks executes on CPU. 4 | 5 | ![Generated](images/BioGPT_example.png) -------------------------------------------------------------------------------- /Colab/BioGPT/images/BioGPT_example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/BioGPT/images/BioGPT_example.png -------------------------------------------------------------------------------- /Colab/C-GPT/C-ChatGPT-Colab-Example.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/C-GPT/C-ChatGPT-Colab-Example.PNG -------------------------------------------------------------------------------- /Colab/C-GPT/C_ChatGPT.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "provenance": [] 7 | }, 8 | "kernelspec": { 9 | "name": "python3", 10 | "display_name": "Python 3" 11 | }, 12 | "language_info": { 13 | "name": "python" 14 | } 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "source": [ 20 | "A notebook to compile Nicholas Carlini's C-ChatGPT and start a chat with the model within a Colab VM. \n", 21 | "More explanations about the code [here](https://nicholas.carlini.com/writing/2023/chat-gpt-2-in-c.html). \n", 22 | "No hardware acceleration needed to execute the code in this notebook: a regualr free Colab GPU VM is enough." 23 | ], 24 | "metadata": { 25 | "id": "HVaQl09wwl-L" 26 | } 27 | }, 28 | { 29 | "cell_type": "markdown", 30 | "source": [ 31 | "## Settings" 32 | ], 33 | "metadata": { 34 | "id": "1foFnaNzxMpe" 35 | } 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "source": [ 40 | "Clone the official GitHub repository." 41 | ], 42 | "metadata": { 43 | "id": "mB4mWQ2KxOHU" 44 | } 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": { 50 | "id": "FYToAlwbDHfR" 51 | }, 52 | "outputs": [], 53 | "source": [ 54 | "!git clone https://github.com/carlini/c-chat-gpt-2.git\n", 55 | "%cd c-chat-gpt-2" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "source": [ 61 | "Download the pretrained GPT-2 Small model weights. A bash script has been provided to manage the downlaod, we need just to change the permissions on it to allow execution." 62 | ], 63 | "metadata": { 64 | "id": "R8M4Sn1ixTzt" 65 | } 66 | }, 67 | { 68 | "cell_type": "code", 69 | "source": [ 70 | "!chmod 744 ./download.sh\n", 71 | "!./download.sh" 72 | ], 73 | "metadata": { 74 | "id": "CKZzORsWFhlc" 75 | }, 76 | "execution_count": null, 77 | "outputs": [] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "source": [ 82 | "Install the *colab-xterm* Python package, to allow opening a terminal in a Colab notebook code cell (it works on the Colab free tier too) and starting a chat with the model." 83 | ], 84 | "metadata": { 85 | "id": "cDzTuvL0xxdX" 86 | } 87 | }, 88 | { 89 | "cell_type": "code", 90 | "source": [ 91 | "!pip install colab-xterm\n", 92 | "%load_ext colabxterm" 93 | ], 94 | "metadata": { 95 | "id": "Cg117Qelx8qC" 96 | }, 97 | "execution_count": null, 98 | "outputs": [] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "source": [ 103 | "## Compilation" 104 | ], 105 | "metadata": { 106 | "id": "i-fZbV11yDfb" 107 | } 108 | }, 109 | { 110 | "cell_type": "markdown", 111 | "source": [ 112 | "Compile the C code using *gcc*. The *-lm* option to the command to compile the model is mandatory here as, even if in the source code the math.h header is included, in the Linux env for the Colab VMs that library isn't automatically linked, but you need to do it explicitely. \n", 113 | "You can ignore few warnings showing in the cell output at compilation time. \n", 114 | "At the end of a successful compilation an executable file named *a.out* will be created in the current directory." 115 | ], 116 | "metadata": { 117 | "id": "mrQKX3BkJoHP" 118 | } 119 | }, 120 | { 121 | "cell_type": "code", 122 | "source": [ 123 | "!gcc -O3 ./c_chat_gpt_2.c -lm" 124 | ], 125 | "metadata": { 126 | "id": "x43keqeEEeM1" 127 | }, 128 | "execution_count": null, 129 | "outputs": [] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "source": [ 134 | "## Inference" 135 | ], 136 | "metadata": { 137 | "id": "nUZzbmLv0EM0" 138 | } 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "source": [ 143 | "Launch a terminal and from there execute the compiled a.out file to start a chat session with the model. The command syntax is to use in the terminal is as follows: \n", 144 | "*./a.out chekpoints_file_name vocab_file_name chat_history maximum_sequence_length* \n", 145 | "Example:\n", 146 | "\n", 147 | "\n", 148 | "````\n", 149 | "./a.out gpt2-124M.ckpt vocab.bpe \"$(echo -e \"\\nAlice: Hello, how are you doing today?\\nBob: I am doing well. I am a language model trained by OpenAI. How can I assist you?\\nAlice: Can you answer my questions?\\nBob: Yes I will answer your questions. What do you want to know?\\nAlice: What is your name?\\nBob: My name is Bob.\\nAlice: Nice to meet you Bob. I'm alice.\\nBob: How can I help you?\")\" 512````\n", 150 | "\n", 151 | "Press Ctrl+C in the terminal to end a chat session." 152 | ], 153 | "metadata": { 154 | "id": "qhF0qyOw82o7" 155 | } 156 | }, 157 | { 158 | "cell_type": "code", 159 | "source": [ 160 | "%xterm" 161 | ], 162 | "metadata": { 163 | "id": "zCZtbAobrCty" 164 | }, 165 | "execution_count": null, 166 | "outputs": [] 167 | } 168 | ] 169 | } -------------------------------------------------------------------------------- /Colab/C-GPT/README.md: -------------------------------------------------------------------------------- 1 | A notebook to compile Nicholas Carlini's C-ChatGPT (a ChatGPT clone, in 3000 bytes of C, backed by GPT-2) and start a chat with the model within a Colab VM. 2 | More explanations about the C code [here](https://nicholas.carlini.com/writing/2023/chat-gpt-2-in-c.html). 3 | No hardware acceleration needed to execute the code in this notebook: a regualr free Colab GPU VM is enough. 4 | 5 | ![Generated](C-ChatGPT-Colab-Example.PNG) -------------------------------------------------------------------------------- /Colab/DFR/Deep_Feature_Rotation_Simple.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Deep Feature Rotation - Simple.ipynb", 7 | "provenance": [] 8 | }, 9 | "kernelspec": { 10 | "name": "python3", 11 | "display_name": "Python 3" 12 | }, 13 | "language_info": { 14 | "name": "python" 15 | }, 16 | "accelerator": "GPU" 17 | }, 18 | "cells": [ 19 | { 20 | "cell_type": "markdown", 21 | "source": [ 22 | "# Deep Feature Rotation (DFR)\n", 23 | "A notebook to do image style transfer by using the official [TensorFlow](https://www.tensorflow.org/) 2 implementation of the Deep Feature Rotation (DFR) [paper](https://arxiv.org/abs/2202.04426). While an official Colab notebook has been provided by the paper authors, this one has been implemented to stay in synch with any original code changes as it starts by cloning the official implementation of the paper from GitHub. Also, the possibility to upload content and style images, a form to set up training parameters and code to display the generated images have been provided here. \n", 24 | "A GPU runtime is needed to execute the code in this notebook. \n", 25 | "Credits for the DFR paper: \n", 26 | "\n", 27 | "\n", 28 | "```\n", 29 | "@INPROCEEDINGS{9701465, \n", 30 | " author={Nguyen, Son Truong and Tuyen, Nguyen Quang and Phuc, Nguyen Hong}, \n", 31 | " booktitle={2021 8th NAFOSTED Conference on Information and Computer Science (NICS)}, \n", 32 | " title={Deep Feature Rotation for Multimodal Image Style Transfer}, \n", 33 | " year={2021}, \n", 34 | " pages={260-265}, \n", 35 | " doi={10.1109/NICS54270.2021.9701465}\n", 36 | "}\n", 37 | "```\n", 38 | "\n" 39 | ], 40 | "metadata": { 41 | "id": "BB8VWUXZelEm" 42 | } 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "source": [ 47 | "# Settings" 48 | ], 49 | "metadata": { 50 | "id": "x71v4MCQhMV5" 51 | } 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "source": [ 56 | "Clone the official GitHub repository." 57 | ], 58 | "metadata": { 59 | "id": "QKTIh4EJjvBo" 60 | } 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": { 66 | "id": "xZZI-AEujphn" 67 | }, 68 | "outputs": [], 69 | "source": [ 70 | "!git clone https://github.com/sonnguyen129/deep-feature-rotation.git\n", 71 | "%cd deep-feature-rotation" 72 | ] 73 | }, 74 | { 75 | "cell_type": "markdown", 76 | "source": [ 77 | "Upload content and style images." 78 | ], 79 | "metadata": { 80 | "id": "eB7sigy4j8BS" 81 | } 82 | }, 83 | { 84 | "cell_type": "code", 85 | "source": [ 86 | "from google.colab import files\n", 87 | "\n", 88 | "def upload_files():\n", 89 | " uploaded = files.upload()\n", 90 | " for k, v in uploaded.items():\n", 91 | " open(k, 'wb').write(v)\n", 92 | " return list(uploaded.keys())" 93 | ], 94 | "metadata": { 95 | "id": "N8Zc_-Tsp6v8" 96 | }, 97 | "execution_count": null, 98 | "outputs": [] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "source": [ 103 | "uploaded_image_list = upload_files()" 104 | ], 105 | "metadata": { 106 | "id": "7PNGWEPRqMUx" 107 | }, 108 | "execution_count": null, 109 | "outputs": [] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "source": [ 114 | "Add the *dfr* directory to the Python Path." 115 | ], 116 | "metadata": { 117 | "id": "QnB2B2sWhXDL" 118 | } 119 | }, 120 | { 121 | "cell_type": "code", 122 | "source": [ 123 | "import sys\n", 124 | "import os\n", 125 | "\n", 126 | "dfr_path = '/content/deep-feature-rotation/dfr'\n", 127 | "sys.path.append(dfr_path)\n", 128 | "os.environ[\"PYTHONPATH\"] += (\":\" + dfr_path)" 129 | ], 130 | "metadata": { 131 | "id": "ZauuMgYbx61E" 132 | }, 133 | "execution_count": null, 134 | "outputs": [] 135 | }, 136 | { 137 | "cell_type": "markdown", 138 | "source": [ 139 | "Set up arguments for the training script." 140 | ], 141 | "metadata": { 142 | "id": "mSgVIF4PrlOL" 143 | } 144 | }, 145 | { 146 | "cell_type": "code", 147 | "source": [ 148 | "#@title Training Options\n", 149 | "\n", 150 | "content_path = \"\" #@param uploaded_image_list\n", 151 | "style_path = \"\" #@param uploaded_image_list\n", 152 | "total_variation_weight = 30 #@param {type: \"number\"}\n", 153 | "snapshot_interval = 100 #@param {type: \"number\"}\n", 154 | "\n", 155 | "if content_path in uploaded_image_list and style_path in uploaded_image_list:\n", 156 | " os.environ['CONTENT_PATH'] = content_path\n", 157 | " os.environ['STYLE_PATH'] = style_path\n", 158 | " os.environ['TOTAL_VARIATION_WEIGHT'] = str(total_variation_weight)\n", 159 | " os.environ['SNAPSHOT_INTERVAL'] = str(snapshot_interval)\n", 160 | "else:\n", 161 | " print('*** Please select content and style from the uploaded images ***')" 162 | ], 163 | "metadata": { 164 | "id": "-C2z-fdSrpQF" 165 | }, 166 | "execution_count": null, 167 | "outputs": [] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "source": [ 172 | "Start the training." 173 | ], 174 | "metadata": { 175 | "id": "czW92G4AqW6f" 176 | } 177 | }, 178 | { 179 | "cell_type": "code", 180 | "source": [ 181 | "!python ./train.py --content-path $CONTENT_PATH --style-path $STYLE_PATH --snapshot_interval $SNAPSHOT_INTERVAL --total-variation-weight $TOTAL_VARIATION_WEIGHT" 182 | ], 183 | "metadata": { 184 | "id": "FvhMuD7qqY2q" 185 | }, 186 | "execution_count": null, 187 | "outputs": [] 188 | }, 189 | { 190 | "cell_type": "markdown", 191 | "source": [ 192 | "Show the results." 193 | ], 194 | "metadata": { 195 | "id": "Won2kN-1q1ZZ" 196 | } 197 | }, 198 | { 199 | "cell_type": "code", 200 | "source": [ 201 | "import cv2\n", 202 | "import matplotlib.pyplot as plt\n", 203 | "\n", 204 | "image_output_dir = os.path.join('./results', content_path.split('.')[0] + '_' + style_path.split('.')[0] + '_1.0')\n", 205 | "items = os.listdir(image_output_dir)\n", 206 | "items.sort() \n", 207 | "\n", 208 | "generated_image_count = len(items)\n", 209 | "image_index = 0\n", 210 | "rows = total_variation_weight\n", 211 | "cols = 4\n", 212 | "fig, axes = plt.subplots(nrows=rows, ncols=cols, figsize=(60,60))\n", 213 | "\n", 214 | "for i in range(rows):\n", 215 | " for j in range(cols): \n", 216 | " if image_index < generated_image_count:\n", 217 | " full_path = image_output_dir + '/' + items[image_index]\n", 218 | " image = cv2.imread(full_path)\n", 219 | " image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB) \n", 220 | " axes[i, j].imshow(image)\n", 221 | " axes[i, j].set_title(items[image_index])\n", 222 | " image_index+=1" 223 | ], 224 | "metadata": { 225 | "id": "LAlTorXJq2_T" 226 | }, 227 | "execution_count": null, 228 | "outputs": [] 229 | } 230 | ] 231 | } 232 | -------------------------------------------------------------------------------- /Colab/DFR/README.md: -------------------------------------------------------------------------------- 1 | # Image Style Transfer with Deep Feature Rotation (DFR) 2 | A notebook to do image style transfer by using the official [TensorFlow](https://www.tensorflow.org/) 2 implementation of the Deep Feature Rotation (DFR) [paper](https://arxiv.org/abs/2202.04426). While an official Colab notebook has been provided by the paper authors, this one has been implemented to stay in synch with any original code changes as it starts by cloning the official implementation of the paper from GitHub. Also, the possibility to upload custom content and style images, a form to set up training parameters and code to display the generated images have been provided here. 3 | A GPU runtime is needed to execute the code in this notebook. 4 | ![DFR Architecture](https://github.com/sonnguyen129/deep-feature-rotation/raw/main/doc/model.png) -------------------------------------------------------------------------------- /Colab/DRPreter/DRPreter_Evaluation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "provenance": [] 7 | }, 8 | "kernelspec": { 9 | "name": "python3", 10 | "display_name": "Python 3" 11 | }, 12 | "language_info": { 13 | "name": "python" 14 | }, 15 | "accelerator": "GPU", 16 | "gpuClass": "standard" 17 | }, 18 | "cells": [ 19 | { 20 | "cell_type": "markdown", 21 | "source": [ 22 | "# DRPreter Paper Evaluation\n", 23 | "This notebook contains code to evaluate (training and test) the **DRPreter: Interpretable Anticancer Drug Response Prediction Using Knowledge-Guided Graph Neural Networks and Transformer** [paper](https://www.mdpi.com/1422-0067/23/22/13919). \n", 24 | "![DRPreter](https://user-images.githubusercontent.com/68269057/198502117-785291dd-af73-40d3-8fed-0e8881404119.png) \n", 25 | "DRPreter learns cell line and drug information with graph neural networks; the cell-line graph is further divided into multiple subgraphs with domain knowledge on biological pathways. A type-aware transformer in DRPreter helps detect relationships between pathways and a drug, highlighting important pathways that are involved in the drug response. \n", 26 | "A GPU runtime is required to execute the code in this notebook." 27 | ], 28 | "metadata": { 29 | "id": "jtjVaxBsS1Bq" 30 | } 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "source": [ 35 | "## Settings" 36 | ], 37 | "metadata": { 38 | "id": "-uxa3bWQF_YA" 39 | } 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "source": [ 44 | "Clone the official GitHub repository." 45 | ], 46 | "metadata": { 47 | "id": "ZjRfS90p-PUH" 48 | } 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "metadata": { 54 | "id": "_DOaNrqm9_IS" 55 | }, 56 | "outputs": [], 57 | "source": [ 58 | "!git clone https://github.com/babaling/DRPreter.git\n", 59 | "%cd DRPreter" 60 | ] 61 | }, 62 | { 63 | "cell_type": "markdown", 64 | "source": [ 65 | "No need to install all of the Python packages indicated in the provided *geometric.yaml* file: only few of the DRPtreter required dependencies (*torch-geometric*, *torch-sparse*, *torch-scatter*, *rdkit*, *dgllife* and *dgl*) are't available by default in the Colab VMs. So, these need to be installed before proceeding with the paper evaluation. \n", 66 | "First, identify the current version of PyTorch and CUDA. These info are needed to identify the proper version of *torch-sparse* and *torch-scatter* to install and run properly in Colab." 67 | ], 68 | "metadata": { 69 | "id": "dIkPs75bAWQm" 70 | } 71 | }, 72 | { 73 | "cell_type": "code", 74 | "source": [ 75 | "import torch\n", 76 | "\n", 77 | "def format_pytorch_version(version):\n", 78 | " return version.split('+')[0]\n", 79 | "\n", 80 | "TORCH_version = torch.__version__\n", 81 | "TORCH = format_pytorch_version(TORCH_version)\n", 82 | "\n", 83 | "def format_cuda_version(version):\n", 84 | " return 'cu' + version.replace('.', '')\n", 85 | "\n", 86 | "CUDA_version = torch.version.cuda\n", 87 | "CUDA = format_cuda_version(CUDA_version)" 88 | ], 89 | "metadata": { 90 | "id": "PQJksdDxCGDF" 91 | }, 92 | "execution_count": null, 93 | "outputs": [] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "source": [ 98 | "Then, install the missing packages. Please note the strict version requirement for torch-geometric: release 1.7.1 has been chosen, as indicated in the repo. Anyway it should be < 2.0." 99 | ], 100 | "metadata": { 101 | "id": "cvs2vThvUi8H" 102 | } 103 | }, 104 | { 105 | "cell_type": "code", 106 | "source": [ 107 | "!pip install pyg-lib -f https://pytorch-geometric.com/whl/torch-{TORCH}+{CUDA}.html\n", 108 | "!pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-{TORCH}+{CUDA}.html\n", 109 | "!pip install torch-sparse -f https://pytorch-geometric.com/whl/torch-{TORCH}+{CUDA}.html\n", 110 | "!pip install torch-geometric==1.7.1\n", 111 | "!pip install rdkit dgllife dgl" 112 | ], 113 | "metadata": { 114 | "id": "JiQro1o2j1sv" 115 | }, 116 | "execution_count": null, 117 | "outputs": [] 118 | }, 119 | { 120 | "cell_type": "markdown", 121 | "source": [ 122 | "Create the cell-line data." 123 | ], 124 | "metadata": { 125 | "id": "t5pQcck6Wy4a" 126 | } 127 | }, 128 | { 129 | "cell_type": "code", 130 | "source": [ 131 | "!python ./cellline_graph.py" 132 | ], 133 | "metadata": { 134 | "id": "hKjmvDc2eNrc" 135 | }, 136 | "execution_count": null, 137 | "outputs": [] 138 | }, 139 | { 140 | "cell_type": "markdown", 141 | "source": [ 142 | "Create the drug data." 143 | ], 144 | "metadata": { 145 | "id": "Ddli5q3wmP1n" 146 | } 147 | }, 148 | { 149 | "cell_type": "code", 150 | "source": [ 151 | "!python ./drug_graph.py" 152 | ], 153 | "metadata": { 154 | "id": "wPRzQ-P-mXUn" 155 | }, 156 | "execution_count": null, 157 | "outputs": [] 158 | }, 159 | { 160 | "cell_type": "markdown", 161 | "source": [ 162 | "## Training" 163 | ], 164 | "metadata": { 165 | "id": "-_gtPJVWGFT0" 166 | } 167 | }, 168 | { 169 | "cell_type": "markdown", 170 | "source": [ 171 | "You can skip this section and move straight to the **Test** section if you don't want to train a new model. " 172 | ], 173 | "metadata": { 174 | "id": "4IYeI78kXBm9" 175 | } 176 | }, 177 | { 178 | "cell_type": "markdown", 179 | "source": [ 180 | "Create the directory where to store the results, as it isn't automatically created by the provided training script." 181 | ], 182 | "metadata": { 183 | "id": "3XHQdHPTXTIi" 184 | } 185 | }, 186 | { 187 | "cell_type": "code", 188 | "source": [ 189 | "!mkdir ./Result" 190 | ], 191 | "metadata": { 192 | "id": "SDZC5AIlpaVK" 193 | }, 194 | "execution_count": null, 195 | "outputs": [] 196 | }, 197 | { 198 | "cell_type": "markdown", 199 | "source": [ 200 | "Set the number of epochs." 201 | ], 202 | "metadata": { 203 | "id": "Oi70UROEXpYS" 204 | } 205 | }, 206 | { 207 | "cell_type": "code", 208 | "source": [ 209 | "%env EPOCHS_NUM = 10" 210 | ], 211 | "metadata": { 212 | "id": "UlxyUAdNXtJE" 213 | }, 214 | "execution_count": null, 215 | "outputs": [] 216 | }, 217 | { 218 | "cell_type": "markdown", 219 | "source": [ 220 | "Start the training." 221 | ], 222 | "metadata": { 223 | "id": "cvw31TkyXen0" 224 | } 225 | }, 226 | { 227 | "cell_type": "code", 228 | "source": [ 229 | "!python ./main.py --mode train --epochs $EPOCHS_NUM" 230 | ], 231 | "metadata": { 232 | "id": "Z7iufe2_-deS" 233 | }, 234 | "execution_count": null, 235 | "outputs": [] 236 | }, 237 | { 238 | "cell_type": "markdown", 239 | "source": [ 240 | "A results summary is printed to the output of the previous cell, but full result data are saved to files, so it is possible to have a look at them too. \n", 241 | "Load the results data first." 242 | ], 243 | "metadata": { 244 | "id": "3dLNxecBI7gE" 245 | } 246 | }, 247 | { 248 | "cell_type": "code", 249 | "source": [ 250 | "import pandas as pd\n", 251 | "\n", 252 | "result_txt_df = pd.read_csv('./Result/results_seed42.txt', sep=\"\\t\") \n", 253 | "result_csv_df = pd.read_csv('./Result/results_df_seed42.csv', sep=\"\\t\")" 254 | ], 255 | "metadata": { 256 | "id": "pPpDs34QIkMe" 257 | }, 258 | "execution_count": null, 259 | "outputs": [] 260 | }, 261 | { 262 | "cell_type": "markdown", 263 | "source": [ 264 | "Then display the performance metrics..." 265 | ], 266 | "metadata": { 267 | "id": "uwf8ts-mKNNH" 268 | } 269 | }, 270 | { 271 | "cell_type": "code", 272 | "source": [ 273 | "result_txt_df" 274 | ], 275 | "metadata": { 276 | "id": "aO40gc8vJoPa" 277 | }, 278 | "execution_count": null, 279 | "outputs": [] 280 | }, 281 | { 282 | "cell_type": "markdown", 283 | "source": [ 284 | "... and the predictions versus true values too." 285 | ], 286 | "metadata": { 287 | "id": "0N9H5PDYKSvn" 288 | } 289 | }, 290 | { 291 | "cell_type": "code", 292 | "source": [ 293 | "result_csv_df" 294 | ], 295 | "metadata": { 296 | "id": "VeYGQrT8Jwx9" 297 | }, 298 | "execution_count": null, 299 | "outputs": [] 300 | }, 301 | { 302 | "cell_type": "markdown", 303 | "source": [ 304 | "## Test" 305 | ], 306 | "metadata": { 307 | "id": "S0FsVCKaGK1a" 308 | } 309 | }, 310 | { 311 | "cell_type": "markdown", 312 | "source": [ 313 | "This section is in case you want only test the pretrained models released by the paper's authors." 314 | ], 315 | "metadata": { 316 | "id": "8eY_ARu8hukD" 317 | } 318 | }, 319 | { 320 | "cell_type": "markdown", 321 | "source": [ 322 | "Select a seed." 323 | ], 324 | "metadata": { 325 | "id": "ylm_aIHUh6Tw" 326 | } 327 | }, 328 | { 329 | "cell_type": "code", 330 | "source": [ 331 | "#@title Test Options\n", 332 | "\n", 333 | "seed_to_test = \"653\" #@param [\"2\", \"16\", \"33\", \"61\", \"79\", \"100\", \"220\", \"653\", \"1004\", \"4001\"]\n", 334 | "\n", 335 | "import os\n", 336 | "\n", 337 | "os.environ['SEED_TO_TEST'] = seed_to_test" 338 | ], 339 | "metadata": { 340 | "id": "vYVCtVxdijS4" 341 | }, 342 | "execution_count": null, 343 | "outputs": [] 344 | }, 345 | { 346 | "cell_type": "markdown", 347 | "source": [ 348 | "Start testing the selected pretrained model. A test results summary is printed to the output of the code cell." 349 | ], 350 | "metadata": { 351 | "id": "JamOPJ7XkwJB" 352 | } 353 | }, 354 | { 355 | "cell_type": "code", 356 | "source": [ 357 | "!python main.py --mode test --seed $SEED_TO_TEST" 358 | ], 359 | "metadata": { 360 | "id": "yhUf2u6W-irF" 361 | }, 362 | "execution_count": null, 363 | "outputs": [] 364 | } 365 | ] 366 | } -------------------------------------------------------------------------------- /Colab/DRPreter/README.md: -------------------------------------------------------------------------------- 1 | # DRPreter Paper Evaluation 2 | A notebook to evaluate (training and test) the **DRPreter: Interpretable Anticancer Drug Response Prediction Using Knowledge-Guided Graph Neural Networks and Transformer** [paper](https://www.mdpi.com/1422-0067/23/22/13919). 3 | ![DRPreter](https://user-images.githubusercontent.com/68269057/198502117-785291dd-af73-40d3-8fed-0e8881404119.png) 4 | DRPreter learns cell line and drug information with graph neural networks; the cell-line graph is further divided into multiple subgraphs with domain knowledge on biological pathways. A type-aware transformer in DRPreter helps detect relationships between pathways and a drug, highlighting important pathways that are involved in the drug response. 5 | A GPU runtime is required to execute the code in this notebook. -------------------------------------------------------------------------------- /Colab/DieSitCom/DieSitCom_Colab.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "DieSitCom Colab.ipynb", 7 | "provenance": [], 8 | "toc_visible": true 9 | }, 10 | "kernelspec": { 11 | "name": "python3", 12 | "display_name": "Python 3" 13 | }, 14 | "language_info": { 15 | "name": "python" 16 | } 17 | }, 18 | "cells": [ 19 | { 20 | "cell_type": "markdown", 21 | "metadata": { 22 | "id": "nXJu35zLMDeZ" 23 | }, 24 | "source": [ 25 | "#DieSitCom Colab\n", 26 | "A notebook to run the minimal [Streamlit](https://streamlit.io/)-based viewer for [DICOM](https://www.dicomstandard.org/) data by [Matteo Mancini](https://neurosnippets.com/posts/diesitcom/). \n", 27 | "This notebook is also a good reference to understand how to run Streamlit apps in Colab." 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": { 33 | "id": "uQ2q9VEoNxt4" 34 | }, 35 | "source": [ 36 | "Install the missing requirements for the Streamlit app." 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "metadata": { 42 | "id": "hJDbBu30L4U3" 43 | }, 44 | "source": [ 45 | "!pip install SimpleITK==2.0.2" 46 | ], 47 | "execution_count": null, 48 | "outputs": [] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "metadata": { 53 | "id": "xk3-lPVyOldx" 54 | }, 55 | "source": [ 56 | "!pip install streamlit" 57 | ], 58 | "execution_count": null, 59 | "outputs": [] 60 | }, 61 | { 62 | "cell_type": "markdown", 63 | "metadata": { 64 | "id": "pfbhSeCzS09l" 65 | }, 66 | "source": [ 67 | "Get the DieSitCom app script." 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "metadata": { 73 | "id": "tPzKn_nBS113" 74 | }, 75 | "source": [ 76 | "!wget https://raw.githubusercontent.com/matteomancini/neurosnippets/master/brainviz/diesitcom/diesitcom.py" 77 | ], 78 | "execution_count": null, 79 | "outputs": [] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "metadata": { 84 | "id": "_WOgKbjfUlpl" 85 | }, 86 | "source": [ 87 | "Download some DICOM (MRI brain scanner) sample data. Replace the code in the cells below in order to download your own DICOM data." 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "metadata": { 93 | "id": "NbdMiY35YwyV" 94 | }, 95 | "source": [ 96 | "!mkdir data" 97 | ], 98 | "execution_count": null, 99 | "outputs": [] 100 | }, 101 | { 102 | "cell_type": "code", 103 | "metadata": { 104 | "id": "4UwJzg4FY21P" 105 | }, 106 | "source": [ 107 | "%cd ./data/" 108 | ], 109 | "execution_count": null, 110 | "outputs": [] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "metadata": { 115 | "id": "oO_hHTlEUmV4" 116 | }, 117 | "source": [ 118 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-01.dcm\n", 119 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-02.dcm\n", 120 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-03.dcm\n", 121 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-04.dcm\n", 122 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-05.dcm\n", 123 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-06.dcm\n", 124 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-07.dcm\n", 125 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-08.dcm\n", 126 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-09.dcm\n", 127 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-10.dcm\n", 128 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-11.dcm\n", 129 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-12.dcm\n", 130 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-13.dcm\n", 131 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-14.dcm\n", 132 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-15.dcm\n", 133 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-16.dcm\n", 134 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-17.dcm\n", 135 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-18.dcm\n", 136 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-19.dcm\n", 137 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-20.dcm\n", 138 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-21.dcm\n", 139 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-22.dcm\n", 140 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-23.dcm\n", 141 | "!wget https://github.com/plotly/dash-vtk/raw/master/demos/data/mri_brain/1-24.dcm" 142 | ], 143 | "execution_count": null, 144 | "outputs": [] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "metadata": { 149 | "id": "fRh-bWZaZLov" 150 | }, 151 | "source": [ 152 | "%cd .." 153 | ], 154 | "execution_count": null, 155 | "outputs": [] 156 | }, 157 | { 158 | "cell_type": "markdown", 159 | "metadata": { 160 | "id": "O3ClQe8-Oy5H" 161 | }, 162 | "source": [ 163 | "Install [ngrok](https://github.com/inconshreveable/ngrok). It is an Open Source reverse proxy that creates a secure tunnel from a public endpoint to a locally running web service. This is needed to access the Streamlit app UI running within this notebook." 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "metadata": { 169 | "id": "XHfz-nEzOzVy" 170 | }, 171 | "source": [ 172 | "!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip" 173 | ], 174 | "execution_count": null, 175 | "outputs": [] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "metadata": { 180 | "id": "OFVhwwnaQXOE" 181 | }, 182 | "source": [ 183 | "!unzip ngrok-stable-linux-amd64.zip" 184 | ], 185 | "execution_count": null, 186 | "outputs": [] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "metadata": { 191 | "id": "ZJLjjJ6xZSmt" 192 | }, 193 | "source": [ 194 | "!rm ./ngrok-stable-linux-amd64.zip" 195 | ], 196 | "execution_count": null, 197 | "outputs": [] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": { 202 | "id": "eo0ffRSAQln5" 203 | }, 204 | "source": [ 205 | "Create a tunnel to port 8501 (the default port for Streamlit apps). Once the tunnel has been created, follow the instruction that will be sent to the output of the next cells in order to start the DICOM viewer app and access its UI from the web browser." 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "metadata": { 211 | "id": "ZZgreyNiQerP" 212 | }, 213 | "source": [ 214 | "get_ipython().system_raw('./ngrok http 8501 &')" 215 | ], 216 | "execution_count": null, 217 | "outputs": [] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "metadata": { 222 | "id": "PVbQYFWsTR0l" 223 | }, 224 | "source": [ 225 | "!curl -s http://localhost:4040/api/tunnels | python3 -c \\\n", 226 | " 'import sys, json; print(\"Execute the next cell and the go to the following URL: \" +json.load(sys.stdin)[\"tunnels\"][0][\"public_url\"])'" 227 | ], 228 | "execution_count": null, 229 | "outputs": [] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "metadata": { 234 | "id": "udfBBOhKTVK8" 235 | }, 236 | "source": [ 237 | "!streamlit run ./diesitcom.py" 238 | ], 239 | "execution_count": null, 240 | "outputs": [] 241 | } 242 | ] 243 | } -------------------------------------------------------------------------------- /Colab/DieSitCom/README.md: -------------------------------------------------------------------------------- 1 | # DieSitCom 2 | A notebook to run the minimal [Streamlit](https://streamlit.io/)-based viewer for [DICOM](https://www.dicomstandard.org/) data by [Matteo Mancini](https://neurosnippets.com/posts/diesitcom/) in Colab. 3 | This notebook is also a good reference to learn how to run Streamlit apps in Colab. 4 | 5 | ![The Viewer UI](https://github.com/virtualramblas/python-notebooks-repo/raw/main/Colab/DieSitCom/demo-image.PNG) -------------------------------------------------------------------------------- /Colab/DieSitCom/demo-image.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/DieSitCom/demo-image.PNG -------------------------------------------------------------------------------- /Colab/DriveGAN/DriveGAN_Playground.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "DriveGAN Playground.ipynb", 7 | "provenance": [] 8 | }, 9 | "kernelspec": { 10 | "name": "python3", 11 | "display_name": "Python 3" 12 | }, 13 | "language_info": { 14 | "name": "python" 15 | }, 16 | "accelerator": "GPU" 17 | }, 18 | "cells": [ 19 | { 20 | "cell_type": "markdown", 21 | "metadata": { 22 | "id": "0QO-0HXJ-zma" 23 | }, 24 | "source": [ 25 | "## DriveGAN Playground\n", 26 | "A notebook to play with the [DriveGAN](https://nv-tlabs.github.io/DriveGAN/) simulator within Colab. \n", 27 | "**A GPU runtime is required in order to run the simulator.**" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": { 33 | "id": "6CnP4Qw1yJeA" 34 | }, 35 | "source": [ 36 | "### Setup" 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": { 42 | "id": "Az73HoLN_g1C" 43 | }, 44 | "source": [ 45 | "Clone the official GitHub repo." 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "metadata": { 51 | "id": "RMquJIHR_MoM" 52 | }, 53 | "source": [ 54 | "!git clone https://github.com/nv-tlabs/DriveGAN_code.git\n", 55 | "%cd DriveGAN_code" 56 | ], 57 | "execution_count": null, 58 | "outputs": [] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "metadata": { 63 | "id": "duRl4SpVxO0S" 64 | }, 65 | "source": [ 66 | "Install the required dependencies. All those specified in the *requirements.txt* file are already pre-installed in the Colab environment but *simplejson*." 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "metadata": { 72 | "id": "Yk_yXjbfFMPy" 73 | }, 74 | "source": [ 75 | "!pip install simplejson" 76 | ], 77 | "execution_count": null, 78 | "outputs": [] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "metadata": { 83 | "id": "vHTsewMw3S9f" 84 | }, 85 | "source": [ 86 | "Install the [Ninja](https://ninja-build.org/) build system in order for the simulator to load the C++ extensions." 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "metadata": { 92 | "id": "VY7EMZX53XyV" 93 | }, 94 | "source": [ 95 | "!wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip\n", 96 | "!sudo unzip ninja-linux.zip -d /usr/local/bin/\n", 97 | "!sudo update-alternatives --install /usr/bin/ninja ninja /usr/local/bin/ninja 1 --force\n", 98 | "!rm -f ninja-linux.zip" 99 | ], 100 | "execution_count": null, 101 | "outputs": [] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "metadata": { 106 | "id": "R_Db6WuJxv6T" 107 | }, 108 | "source": [ 109 | "### Playing with the trained model" 110 | ] 111 | }, 112 | { 113 | "cell_type": "markdown", 114 | "metadata": { 115 | "id": "YBwRN9woxynl" 116 | }, 117 | "source": [ 118 | "Download the dynamics engine and the VAE-GAN model checkpoints." 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "metadata": { 124 | "id": "-BWDkGolzUeh" 125 | }, 126 | "source": [ 127 | "!gdown --id 1weGqehYmgotgFqGgCORa88rJ8_AC-lXr" 128 | ], 129 | "execution_count": null, 130 | "outputs": [] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "metadata": { 135 | "id": "M3OBIhoQ0E7C" 136 | }, 137 | "source": [ 138 | "!gdown --id 19mwH8sgNDNRl9FvJitABu19HLuxmvRFm" 139 | ], 140 | "execution_count": null, 141 | "outputs": [] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "metadata": { 146 | "id": "gk9lN-fQ0v9Q" 147 | }, 148 | "source": [ 149 | "Install [ngrok](https://github.com/inconshreveable/ngrok). It is an Open Source reverse proxy that creates a secure tunnel from a public endpoint to a locally running web service. This is needed to access the simulator web app UI running within this notebook." 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "metadata": { 155 | "id": "80n1Gisz05La" 156 | }, 157 | "source": [ 158 | "!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip" 159 | ], 160 | "execution_count": null, 161 | "outputs": [] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "metadata": { 166 | "id": "mTfjwjtT1lXv" 167 | }, 168 | "source": [ 169 | "!unzip ngrok-stable-linux-amd64.zip\n", 170 | "!rm ./ngrok-stable-linux-amd64.zip" 171 | ], 172 | "execution_count": null, 173 | "outputs": [] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": { 178 | "id": "2p90HFQW1tMo" 179 | }, 180 | "source": [ 181 | "Create a tunnel to port 8888 (select a different port number, depending on what you will set in the cell code that starts the simulator server). Once the tunnel has been created, follow the instruction that will be sent to the output of the next cells in order to start the simulator web app and access its UI from the web browser." 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "metadata": { 187 | "id": "yqN60fMK2Oup" 188 | }, 189 | "source": [ 190 | "get_ipython().system_raw('./ngrok http 8888 &')" 191 | ], 192 | "execution_count": null, 193 | "outputs": [] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "metadata": { 198 | "id": "JM37hlgP2P8v" 199 | }, 200 | "source": [ 201 | "!curl -s http://localhost:4040/api/tunnels | python3 -c \\\n", 202 | " 'import sys, json; print(\"Execute the next cell and the go to the following URL: \" +json.load(sys.stdin)[\"tunnels\"][0][\"public_url\"])'" 203 | ], 204 | "execution_count": null, 205 | "outputs": [] 206 | }, 207 | { 208 | "cell_type": "markdown", 209 | "metadata": { 210 | "id": "99Yqo9EZ06dt" 211 | }, 212 | "source": [ 213 | "Start the simulator to play with the pre-trained model. \n", 214 | "The startup would require some time (between 10 and 20 seconds) to complete. It is ready to play when the '**view @ http://localhost:8888**' message is printed to the cell output. \n", 215 | "The keys to control the car are: \n", 216 | "- 'w': speed up, \n", 217 | "- 's': slow down, \n", 218 | "- 'a': steer left, \n", 219 | "- 'd': steer right \n", 220 | "\n", 221 | "The UI provides additional buttons for changing contents. If the simulator freezes, simply refresh the web page. \n", 222 | "To stop the simulator, simply interrupt the execution of the following cell." 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "metadata": { 228 | "id": "IXF0vLo705rX" 229 | }, 230 | "source": [ 231 | "!./scripts/play/server.sh ./simulator_epoch1020.pt 8888 ./vaegan_iter210000.pt" 232 | ], 233 | "execution_count": null, 234 | "outputs": [] 235 | } 236 | ] 237 | } -------------------------------------------------------------------------------- /Colab/DriveGAN/README.md: -------------------------------------------------------------------------------- 1 | ## Playing with the DriveGAN simulator in Colab 2 | ![Demo Image](https://github.com/virtualramblas/python-notebooks-repo/blob/main/Colab/DriveGAN/Simulator-in-Colab.PNG) 3 | A notebook for a rapid set up and to play with the [DriveGAN](https://nv-tlabs.github.io/DriveGAN/) simulator in [Colab](https://colab.research.google.com). 4 | DriveGAN is a fully differentiable simulator that allows for re-simulation of a given video sequence, offering an agent to drive through a recorded scene again, possibly taking different actions. 5 | Upload the *DriveGAN_Playground.ipynb* notebook to Colab and execute it. A GPU runtime is required: you can switch to this kind of runtime from the *Runtime -> Change runtime type* Colab menu. 6 | -------------------------------------------------------------------------------- /Colab/DriveGAN/Simulator-in-Colab.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/DriveGAN/Simulator-in-Colab.PNG -------------------------------------------------------------------------------- /Colab/FLAVR/Fast_Frame_Interpolation_with_FLAVR.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Fast Frame Interpolation with FLAVR.ipynb", 7 | "provenance": [] 8 | }, 9 | "kernelspec": { 10 | "name": "python3", 11 | "display_name": "Python 3" 12 | }, 13 | "language_info": { 14 | "name": "python" 15 | }, 16 | "accelerator": "GPU" 17 | }, 18 | "cells": [ 19 | { 20 | "cell_type": "markdown", 21 | "source": [ 22 | "# Fast Frame Interpolation with FLAVR\n", 23 | "FLAVR is a fast, flow-free frame interpolation method capable of single shot multi-frame prediction. It uses a customized encoder decoder architecture with spatio-temporal convolutions and channel gating to capture and interpolate complex motion trajectories between frames to generate realistic high frame rate videos. This notebook is to apply slow-motion filtering on your own videos. \n", 24 | "A GPU runtime is suggested to execute the code in this notebook. \n", 25 | " \n", 26 | "Credits for the original FLAVR work:\n", 27 | "\n", 28 | "\n", 29 | "```\n", 30 | "@article{kalluri2021flavr,\n", 31 | " title={FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation},\n", 32 | " author={Kalluri, Tarun and Pathak, Deepak and Chandraker, Manmohan and Tran, Du},\n", 33 | " booktitle={arxiv},\n", 34 | " year={2021}\n", 35 | "}\n", 36 | "```\n", 37 | "\n" 38 | ], 39 | "metadata": { 40 | "id": "GtNm2bt5m__t" 41 | } 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "source": [ 46 | "### Settings" 47 | ], 48 | "metadata": { 49 | "id": "Cer3xI_vC8AX" 50 | } 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "source": [ 55 | "Clone the official GitHub repository." 56 | ], 57 | "metadata": { 58 | "id": "L25AZqD1aqYy" 59 | } 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": null, 64 | "metadata": { 65 | "id": "5geYqIv5ah8_" 66 | }, 67 | "outputs": [], 68 | "source": [ 69 | "!git clone https://github.com/tarun005/FLAVR.git\n", 70 | "%cd FLAVR" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "source": [ 76 | "Install the missing requirements. Almost all the required Python packages for the code in this notebook are available by default in a Colab runtime. Only *PyAV*, a Pythonic binding for the FFmpeg libraries, to be installed really." 77 | ], 78 | "metadata": { 79 | "id": "VZ69AA375uby" 80 | } 81 | }, 82 | { 83 | "cell_type": "code", 84 | "source": [ 85 | "!pip install av" 86 | ], 87 | "metadata": { 88 | "id": "L1Bd6U5H5x8X" 89 | }, 90 | "execution_count": null, 91 | "outputs": [] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "source": [ 96 | "Download a pretrained model. The Colab GPU runtime specs allow full completion only for 2X interpolation." 97 | ], 98 | "metadata": { 99 | "id": "3idcRJmwa0ss" 100 | } 101 | }, 102 | { 103 | "cell_type": "code", 104 | "source": [ 105 | "!gdown --id 1XFk9YZP9llTeporF-t_Au1dI-VhDQppG" 106 | ], 107 | "metadata": { 108 | "id": "eAjOsOhCbCXB" 109 | }, 110 | "execution_count": null, 111 | "outputs": [] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "source": [ 116 | "If the code cell above doesn't work, please copy the pre-trained model manually to your Google Drive space and then follow the instructions for the next 3 code cells." 117 | ], 118 | "metadata": { 119 | "id": "nspERdHKiilc" 120 | } 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "source": [ 125 | "Mount your Google Drive. After executing the code in the cell below, a URL will be shown in the cell output. Click on it and follow the instructions that would appear online." 126 | ], 127 | "metadata": { 128 | "id": "t1W1cafV0RRB" 129 | } 130 | }, 131 | { 132 | "cell_type": "code", 133 | "source": [ 134 | "from google.colab import drive\n", 135 | "\n", 136 | "drive.mount('/content/gdrive')" 137 | ], 138 | "metadata": { 139 | "id": "DPiPftbD0SWC" 140 | }, 141 | "execution_count": null, 142 | "outputs": [] 143 | }, 144 | { 145 | "cell_type": "markdown", 146 | "source": [ 147 | "Copy the pre-trained model to this runtime filesystem." 148 | ], 149 | "metadata": { 150 | "id": "-PibNYIlpu4K" 151 | } 152 | }, 153 | { 154 | "cell_type": "code", 155 | "source": [ 156 | "!cp -av '/content/gdrive/My Drive/FLAVR_2x.pth' './FLAVR_2x.pth'" 157 | ], 158 | "metadata": { 159 | "id": "cEcpbDyW0axe" 160 | }, 161 | "execution_count": null, 162 | "outputs": [] 163 | }, 164 | { 165 | "cell_type": "markdown", 166 | "source": [ 167 | "Unmount your Google Drive when done with the pre-trained model copy." 168 | ], 169 | "metadata": { 170 | "id": "ZLD3eO790bLP" 171 | } 172 | }, 173 | { 174 | "cell_type": "code", 175 | "source": [ 176 | "drive.flush_and_unmount()" 177 | ], 178 | "metadata": { 179 | "id": "8Meb4kd90eFF" 180 | }, 181 | "execution_count": null, 182 | "outputs": [] 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "source": [ 187 | "Define a function to upload videos." 188 | ], 189 | "metadata": { 190 | "id": "_-ll5UukbWE_" 191 | } 192 | }, 193 | { 194 | "cell_type": "code", 195 | "source": [ 196 | "import os\n", 197 | "import shutil\n", 198 | "from google.colab import files\n", 199 | "\n", 200 | "def upload_files(upload_path):\n", 201 | " uploaded = files.upload()\n", 202 | " for filename, content in uploaded.items():\n", 203 | " dst_path = os.path.join(upload_path, filename)\n", 204 | " shutil.move(filename, dst_path)\n", 205 | " return list(uploaded.keys())" 206 | ], 207 | "metadata": { 208 | "id": "1R442tTXbcT9" 209 | }, 210 | "execution_count": null, 211 | "outputs": [] 212 | }, 213 | { 214 | "cell_type": "markdown", 215 | "source": [ 216 | "Create a directory for uploaded videos." 217 | ], 218 | "metadata": { 219 | "id": "4edydmfoceHc" 220 | } 221 | }, 222 | { 223 | "cell_type": "code", 224 | "source": [ 225 | "!mkdir ./test_videos\n", 226 | "image_input_dir = '/content/FLAVR/test_videos/'" 227 | ], 228 | "metadata": { 229 | "id": "Zl3_EauGcjqE" 230 | }, 231 | "execution_count": null, 232 | "outputs": [] 233 | }, 234 | { 235 | "cell_type": "markdown", 236 | "source": [ 237 | "### Slow-Motion Filtering" 238 | ], 239 | "metadata": { 240 | "id": "zJieST7OoVEV" 241 | } 242 | }, 243 | { 244 | "cell_type": "markdown", 245 | "source": [ 246 | "Upload your own video." 247 | ], 248 | "metadata": { 249 | "id": "5tko37fpczcH" 250 | } 251 | }, 252 | { 253 | "cell_type": "code", 254 | "source": [ 255 | "uploaded_videos = upload_files(image_input_dir)" 256 | ], 257 | "metadata": { 258 | "id": "btKmYZb3ciJt" 259 | }, 260 | "execution_count": null, 261 | "outputs": [] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "source": [ 266 | "os.environ['UPLOADED_VIDEO_FILENAME'] = os.path.join(image_input_dir, uploaded_videos[0])" 267 | ], 268 | "metadata": { 269 | "id": "KtbfV4g24LI7" 270 | }, 271 | "execution_count": null, 272 | "outputs": [] 273 | }, 274 | { 275 | "cell_type": "markdown", 276 | "source": [ 277 | "Execute interpolation on the uploaded video." 278 | ], 279 | "metadata": { 280 | "id": "B8oX93NIc60B" 281 | } 282 | }, 283 | { 284 | "cell_type": "code", 285 | "source": [ 286 | "!python ./interpolate.py --input_video $UPLOADED_VIDEO_FILENAME --factor 2 --load_model ./FLAVR_2x.pth" 287 | ], 288 | "metadata": { 289 | "id": "I94Xy1e8dEW_" 290 | }, 291 | "execution_count": null, 292 | "outputs": [] 293 | }, 294 | { 295 | "cell_type": "markdown", 296 | "source": [ 297 | "Display the result." 298 | ], 299 | "metadata": { 300 | "id": "LsmXsAVI-c3d" 301 | } 302 | }, 303 | { 304 | "cell_type": "code", 305 | "source": [ 306 | "from moviepy.editor import VideoFileClip\n", 307 | "\n", 308 | "uploaded_video_filename_tokens = uploaded_videos[0].split('.')\n", 309 | "result_video_path = uploaded_video_filename_tokens[0] + '_2x.' + uploaded_video_filename_tokens[1]\n", 310 | "\n", 311 | "clip = VideoFileClip(result_video_path)\n", 312 | "clip.ipython_display(width=280)" 313 | ], 314 | "metadata": { 315 | "id": "Z9CeJL-Dd-Ul" 316 | }, 317 | "execution_count": null, 318 | "outputs": [] 319 | } 320 | ] 321 | } -------------------------------------------------------------------------------- /Colab/FLAVR/README.md: -------------------------------------------------------------------------------- 1 | # Fast Frame Interpolation with FLAVR 2 | [FLAVR](https://tarun005.github.io/FLAVR/) is a fast, flow-free frame interpolation method capable of single shot multi-frame prediction. It uses a customized encoder decoder architecture with spatio-temporal convolutions and channel gating to capture and interpolate complex motion trajectories between frames to generate realistic high frame rate videos. 3 | 4 | ![FLAVR Model's Architecture.](https://github.com/tarun005/FLAVR/blob/main/figures/arch_dia.png) 5 | 6 | This repo hosts a notebook to apply 2x slow-motion filtering using the official [PyTorch](https://pytorch.org/) FLAVR implementation on your own videos. -------------------------------------------------------------------------------- /Colab/FSGAN/FSGAN_demo_image.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/FSGAN/FSGAN_demo_image.PNG -------------------------------------------------------------------------------- /Colab/FSGAN/Few_shot_GAN_TensorFlow_Inference.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Few-shot GAN TensorFlow - Inference.ipynb", 7 | "provenance": [] 8 | }, 9 | "kernelspec": { 10 | "name": "python3", 11 | "display_name": "Python 3" 12 | }, 13 | "language_info": { 14 | "name": "python" 15 | }, 16 | "accelerator": "GPU" 17 | }, 18 | "cells": [ 19 | { 20 | "cell_type": "markdown", 21 | "source": [ 22 | "# Few-shot Adaptation GAN in TensorFlow - Image Generation\n", 23 | "This notebook performs image generation using a Few-Shot GAN (FSGAN). The official code is availble on GitHub and related to the following [paper](https://arxiv.org/pdf/2010.11943.pdf). Credits: \n", 24 | "\n", 25 | "\n", 26 | "```\n", 27 | "@article{robb2020fsgan,\n", 28 | " title = {Few-Shot Adaptation of Generative Adversarial Networks},\n", 29 | " author = {Robb, Esther and Chu, Wen-Sheng and Kumar, Abhishek and Huang, Jia-Bin},\n", 30 | " journal = {arXiv preprint arXiv:2010.11943},\n", 31 | " year = {2020}\n", 32 | "}\n", 33 | "``` \n", 34 | "A GPU runtime is required to execute the code in this notebook.\n", 35 | "\n" 36 | ], 37 | "metadata": { 38 | "id": "NBy_EAIxaGYu" 39 | } 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "source": [ 44 | "### Settings" 45 | ], 46 | "metadata": { 47 | "id": "9b5St56YbmGC" 48 | } 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "source": [ 53 | "Clone the official GitHub repository." 54 | ], 55 | "metadata": { 56 | "id": "IIKEt0t2FYYJ" 57 | } 58 | }, 59 | { 60 | "cell_type": "code", 61 | "source": [ 62 | "!git clone https://github.com/e-271/few-shot-gan.git\n", 63 | "%cd few-shot-gan" 64 | ], 65 | "metadata": { 66 | "id": "X_UCLKbvFcs3" 67 | }, 68 | "execution_count": null, 69 | "outputs": [] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "source": [ 74 | "Switch to TensorFlow 1.x, as the official implementation of this code isn't compatible with TensorFlow 2.x." 75 | ], 76 | "metadata": { 77 | "id": "aprZYAdCHm6M" 78 | } 79 | }, 80 | { 81 | "cell_type": "code", 82 | "source": [ 83 | "%tensorflow_version 1.x\n", 84 | "\n", 85 | "import tensorflow\n", 86 | "print(tensorflow.__version__)" 87 | ], 88 | "metadata": { 89 | "id": "1liD8nxRHq23" 90 | }, 91 | "execution_count": null, 92 | "outputs": [] 93 | }, 94 | { 95 | "cell_type": "markdown", 96 | "source": [ 97 | "### Inference" 98 | ], 99 | "metadata": { 100 | "id": "Jmm9oA-DDL_G" 101 | } 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "source": [ 106 | "Download pre-trained checkpoint. Multiple pre-trained models are available from the paper's authors. In this notebook we are going to use a model pre-trained on Vincent van Gogh paintings. Future releases will allow selection of different checkpoints." 107 | ], 108 | "metadata": { 109 | "id": "xD4PpWKIEu4i" 110 | } 111 | }, 112 | { 113 | "cell_type": "code", 114 | "source": [ 115 | "!mkdir ./checkpoints" 116 | ], 117 | "metadata": { 118 | "id": "iszu9CrEFpRp" 119 | }, 120 | "execution_count": null, 121 | "outputs": [] 122 | }, 123 | { 124 | "cell_type": "code", 125 | "execution_count": null, 126 | "metadata": { 127 | "id": "gkRaAwp3C3Mz" 128 | }, 129 | "outputs": [], 130 | "source": [ 131 | "%cd ./checkpoints\n", 132 | "!gdown --id 1uObQa4Q0EkSJYOALZSnb96U_t9wMFMBe\n", 133 | "%cd .." 134 | ] 135 | }, 136 | { 137 | "cell_type": "markdown", 138 | "source": [ 139 | "Generate images." 140 | ], 141 | "metadata": { 142 | "id": "wO_t0u8wF0aW" 143 | } 144 | }, 145 | { 146 | "cell_type": "code", 147 | "source": [ 148 | "!python run_generator.py generate-images --network=./checkpoints/gogh-pca-000020.pkl --seeds=0-100" 149 | ], 150 | "metadata": { 151 | "id": "aN75RiZTGmzz" 152 | }, 153 | "execution_count": null, 154 | "outputs": [] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "source": [ 159 | "Browse the generated images." 160 | ], 161 | "metadata": { 162 | "id": "ZyvqtAINIa9x" 163 | } 164 | }, 165 | { 166 | "cell_type": "code", 167 | "source": [ 168 | "import os\n", 169 | "from IPython.display import Image, display\n", 170 | "from IPython.html.widgets import interact\n", 171 | "\n", 172 | "input_images_path = './results/00000-generate-images'\n", 173 | "imgs = list(filter(lambda fn:fn.lower().endswith('.jpg'), os.listdir(input_images_path)))\n", 174 | "def loadimg(k):\n", 175 | " display(Image(open(os.path.join(input_images_path, imgs[k]), 'rb').read()))\n", 176 | "interact(loadimg, k=(0, len(imgs)-1))" 177 | ], 178 | "metadata": { 179 | "id": "RPqzwHWiuSWV" 180 | }, 181 | "execution_count": null, 182 | "outputs": [] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "source": [ 187 | "" 188 | ], 189 | "metadata": { 190 | "id": "PyCSEuRuu1a5" 191 | }, 192 | "execution_count": null, 193 | "outputs": [] 194 | } 195 | ] 196 | } -------------------------------------------------------------------------------- /Colab/FSGAN/README.md: -------------------------------------------------------------------------------- 1 | # Few-Shot Adaptation of Generative Adversarial Networks 2 | Notebooks for few-shot adaptation of a GAN and image generation. Credits for the original work: 3 | ``` 4 | @article{robb2020fsgan, 5 | title = {Few-Shot Adaptation of Generative Adversarial Networks}, 6 | author = {Robb, Esther and Chu, Wen-Sheng and Kumar, Abhishek and Huang, Jia-Bin}, 7 | journal = {arXiv preprint arXiv:2010.11943}, 8 | year = {2020} 9 | } 10 | ``` 11 | Features: 12 | - [ ] Training 13 | - [ ] Original checkpoints multiple selection 14 | - [x] Image generation -------------------------------------------------------------------------------- /Colab/GLIDE/README.md: -------------------------------------------------------------------------------- 1 | # Image Inpainting Using GLIDE 2 | A more user friendly notebook derived from the original in the **glide-text2im** GitHub [repo](https://github.com/openai/glide-text2im), that shows how to use filtered **GLIDE** to fill in a masked region of an image, conditioned on a text prompt. Here's the [link](https://arxiv.org/abs/2112.10741) to the related paper. 3 | 4 | Source image: 5 | 6 | 7 | An oil painting of a lion: 8 | 9 | ![Final result](./lion.png) -------------------------------------------------------------------------------- /Colab/GLIDE/lion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/GLIDE/lion.png -------------------------------------------------------------------------------- /Colab/GLIDE/source_image.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/GLIDE/source_image.jpg -------------------------------------------------------------------------------- /Colab/GPNN/Efficient_GPNN.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Efficient-GPNN.ipynb", 7 | "provenance": [] 8 | }, 9 | "kernelspec": { 10 | "name": "python3", 11 | "display_name": "Python 3" 12 | }, 13 | "language_info": { 14 | "name": "python" 15 | }, 16 | "accelerator": "GPU" 17 | }, 18 | "cells": [ 19 | { 20 | "cell_type": "markdown", 21 | "source": [ 22 | "# Efficient-GPNN\n", 23 | "A notebook to test the tasks (image reshuffling, retargeting and style transfer) available for the efficient [PyTorch](https://pytorch.org) implementation of the [GPNN](https://arxiv.org/abs/2103.15545) paper. \n", 24 | "A GPU runtime is needed to execute the code in this notebook. \n" 25 | ], 26 | "metadata": { 27 | "id": "F18CvTUtQMqG" 28 | } 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "source": [ 33 | "## Settings" 34 | ], 35 | "metadata": { 36 | "id": "NDdhI7jwRx9N" 37 | } 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "source": [ 42 | "Clone the official GitHub repository." 43 | ], 44 | "metadata": { 45 | "id": "fBziJ7bSqXbJ" 46 | } 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "metadata": { 52 | "id": "vZLu7tdCqPWt" 53 | }, 54 | "outputs": [], 55 | "source": [ 56 | "!git clone https://github.com/ariel415el/Efficient-GPNN.git\n", 57 | "%cd Efficient-GPNN" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "source": [ 63 | "Install FAISS for GPU. It is a library for efficient similarity search and clustering of dense vectors, written in C++ with complete wrappers for Python." 64 | ], 65 | "metadata": { 66 | "id": "OoVjUWd9qnfd" 67 | } 68 | }, 69 | { 70 | "cell_type": "code", 71 | "source": [ 72 | "!pip install faiss-gpu" 73 | ], 74 | "metadata": { 75 | "id": "py_1gXhNtvxz" 76 | }, 77 | "execution_count": null, 78 | "outputs": [] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "source": [ 83 | "Define a function to browse the generated images. It will be used for retargeting and style transfer results." 84 | ], 85 | "metadata": { 86 | "id": "fPQrwSrGUsZ3" 87 | } 88 | }, 89 | { 90 | "cell_type": "code", 91 | "source": [ 92 | "import os\n", 93 | "from IPython.display import Image, display\n", 94 | "\n", 95 | "def loadimg(k, images_path):\n", 96 | " display(Image(open(os.path.join(images_path, imgs[k]), 'rb').read()))" 97 | ], 98 | "metadata": { 99 | "id": "xv3aOaOLUv5j" 100 | }, 101 | "execution_count": null, 102 | "outputs": [] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "source": [ 107 | "## Image Generation" 108 | ], 109 | "metadata": { 110 | "id": "jQdqvV9FU97l" 111 | } 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "source": [ 116 | "Do image reshuffling." 117 | ], 118 | "metadata": { 119 | "id": "15ZVslN9tw8_" 120 | } 121 | }, 122 | { 123 | "cell_type": "code", 124 | "source": [ 125 | "!python ./scripts/reshuffle.py" 126 | ], 127 | "metadata": { 128 | "id": "c1GrVxSwt_Tg" 129 | }, 130 | "execution_count": null, 131 | "outputs": [] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "source": [ 136 | "The generated images are saved in the */content/Efficient-GPNN/outputs/reshuffling* directory." 137 | ], 138 | "metadata": { 139 | "id": "Iu4xkZ4wVDqM" 140 | } 141 | }, 142 | { 143 | "cell_type": "markdown", 144 | "source": [ 145 | "Do image retargeting." 146 | ], 147 | "metadata": { 148 | "id": "lYc7QSVxwtxd" 149 | } 150 | }, 151 | { 152 | "cell_type": "code", 153 | "source": [ 154 | "!python ./scripts/retarget.py" 155 | ], 156 | "metadata": { 157 | "id": "IY3i2O_Mt_Vk" 158 | }, 159 | "execution_count": null, 160 | "outputs": [] 161 | }, 162 | { 163 | "cell_type": "markdown", 164 | "source": [ 165 | "Browse the generated images." 166 | ], 167 | "metadata": { 168 | "id": "k1C4mW1LUPqB" 169 | } 170 | }, 171 | { 172 | "cell_type": "code", 173 | "source": [ 174 | "from IPython.html.widgets import interact\n", 175 | "\n", 176 | "output_images_path = './outputs/retarget/SIGD16/NN-PytorchNNLowMem(GPU,alpha=0.005)_R-256_S-0.85->14+I(0,1.5)'\n", 177 | "imgs = list(filter(lambda fn:fn.lower().endswith('.jpg'), os.listdir(output_images_path)))\n", 178 | "interact(loadimg, k=(0, len(imgs)-1), images_path=output_images_path)" 179 | ], 180 | "metadata": { 181 | "id": "ouhNMkLKUTVd" 182 | }, 183 | "execution_count": null, 184 | "outputs": [] 185 | }, 186 | { 187 | "cell_type": "markdown", 188 | "source": [ 189 | "Do image style transfer." 190 | ], 191 | "metadata": { 192 | "id": "biH-4BzyxkLa" 193 | } 194 | }, 195 | { 196 | "cell_type": "code", 197 | "source": [ 198 | "!python ./scripts/style_transfer.py" 199 | ], 200 | "metadata": { 201 | "id": "P0B0V84fx6y2" 202 | }, 203 | "execution_count": null, 204 | "outputs": [] 205 | }, 206 | { 207 | "cell_type": "markdown", 208 | "source": [ 209 | "Browse the generated images." 210 | ], 211 | "metadata": { 212 | "id": "gGLu2Lb_W6wy" 213 | } 214 | }, 215 | { 216 | "cell_type": "code", 217 | "source": [ 218 | "output_images_path = './outputs/style_transfer/FaisIVFPQ(GPU)_1024x0.75->1024'\n", 219 | "imgs = list(filter(lambda fn:fn.lower().endswith('.jpg'), os.listdir(output_images_path)))\n", 220 | "interact(loadimg, k=(0, len(imgs)-1), images_path=output_images_path)" 221 | ], 222 | "metadata": { 223 | "id": "E7Y0AuqoMH0v" 224 | }, 225 | "execution_count": null, 226 | "outputs": [] 227 | } 228 | ] 229 | } -------------------------------------------------------------------------------- /Colab/GPNN/README.md: -------------------------------------------------------------------------------- 1 | # Efficient-GPNN 2 | A notebook to test the tasks (image reshuffling, retargeting and style transfer) available for the efficient [PyTorch](https://pytorch.org) implementation of the [GPNN](https://arxiv.org/abs/2103.15545) paper. 3 | Single image generative models perform synthesis and manipulation tasks by capturing the distribution of patches within a single image. Despite their impressiveness, single image GANs require long training time (usually hours) for each image and each task. They often suffer from artifacts and are prone to optimization issues such as mode collapse. This paper shows that all of these tasks can be performed without any training, within minutes, in a unified, surprisingly simple framework. 4 | A GPU runtime is needed to execute the code in this notebook. -------------------------------------------------------------------------------- /Colab/MiniCLIP/MiniCLIP_Notebook.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "MiniCLIP Notebook.ipynb", 7 | "provenance": [] 8 | }, 9 | "kernelspec": { 10 | "name": "python3", 11 | "display_name": "Python 3" 12 | } 13 | }, 14 | "cells": [ 15 | { 16 | "cell_type": "markdown", 17 | "metadata": { 18 | "id": "3LnzFjFIIHB-" 19 | }, 20 | "source": [ 21 | "## MiniCLIP to Colab\r\n", 22 | "A Colab notebook porting of the MiniCLIP demo (https://github.com/HendrikStrobelt/miniClip) from Hendrik Strobelt." 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": { 28 | "id": "Lpme4g6XIbQG" 29 | }, 30 | "source": [ 31 | "Import the missing dependencies.\r\n" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "metadata": { 37 | "id": "vRtdXt99H_k9" 38 | }, 39 | "source": [ 40 | "!pip install torchray " 41 | ], 42 | "execution_count": null, 43 | "outputs": [] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": { 48 | "id": "Fqxxk4tXJYrZ" 49 | }, 50 | "source": [ 51 | "Install CLIP." 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "metadata": { 57 | "id": "OuvoziLfJbPc" 58 | }, 59 | "source": [ 60 | "!pip install git+https://github.com/openai/CLIP.git" 61 | ], 62 | "execution_count": null, 63 | "outputs": [] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": { 68 | "id": "z-xkdCpZKeJF" 69 | }, 70 | "source": [ 71 | "Add the MiniCLIP user defined functions." 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "metadata": { 77 | "id": "0GmuXS4hKid9" 78 | }, 79 | "source": [ 80 | "from PIL import Image\r\n", 81 | "import numpy as np\r\n", 82 | "import torch\r\n", 83 | "from matplotlib import cm\r\n", 84 | "\r\n", 85 | "def min_max_norm(array):\r\n", 86 | " lim = [array.min(), array.max()]\r\n", 87 | " array = array - lim[0] \r\n", 88 | " array.mul_(1 / (1.e-10+ (lim[1] - lim[0])))\r\n", 89 | " # array = torch.clamp(array, min=0, max=1)\r\n", 90 | " return array\r\n", 91 | "\r\n", 92 | "def torch_to_rgba(img):\r\n", 93 | " img = min_max_norm(img)\r\n", 94 | " rgba_im = img.permute(1, 2, 0).cpu()\r\n", 95 | " if rgba_im.shape[2] == 3:\r\n", 96 | " rgba_im = torch.cat((rgba_im, torch.ones(*rgba_im.shape[:2], 1)), dim=2)\r\n", 97 | " assert rgba_im.shape[2] == 4\r\n", 98 | " return rgba_im\r\n", 99 | "\r\n", 100 | "def numpy_to_image(img, size):\r\n", 101 | " \"\"\"\r\n", 102 | " takes a [0..1] normalized rgba input and returns resized image as [0...255] rgba image\r\n", 103 | " \"\"\"\r\n", 104 | " resized = Image.fromarray((img*255.).astype(np.uint8)).resize((size, size))\r\n", 105 | " return resized\r\n", 106 | "\r\n", 107 | "def upscale_pytorch(img:np.array, size):\r\n", 108 | " torch_img = torch.from_numpy(img).unsqueeze(0).permute(0,3,1,2)\r\n", 109 | " print(torch_img)\r\n", 110 | " upsampler = torch.nn.Upsample(size=size) \r\n", 111 | " return upsampler(torch_img)[0].permute(1,2,0).cpu().numpy()\r\n", 112 | "\r\n", 113 | "\r\n", 114 | "def heatmap(image:torch.Tensor, heatmap: torch.Tensor, size=None, alpha=.6):\r\n", 115 | " if not size:\r\n", 116 | " size = image.shape[1]\r\n", 117 | " # print(heatmap)\r\n", 118 | " # print(min_max_norm(heatmap))\r\n", 119 | "\r\n", 120 | " img = torch_to_rgba(image).numpy() # [0...1] rgba numpy \"image\"\r\n", 121 | " hm = cm.hot(min_max_norm(heatmap).numpy()) # [0...1] rgba numpy \"image\"\r\n", 122 | "\r\n", 123 | " # print(hm.shape, hm)\r\n", 124 | " #\r\n", 125 | "\r\n", 126 | " img = np.array(numpy_to_image(img,size))\r\n", 127 | " hm = np.array(numpy_to_image(hm, size))\r\n", 128 | " # hm = upscale_pytorch(hm, size)\r\n", 129 | " # print (hm) \r\n", 130 | "\r\n", 131 | " #return Image.fromarray((alpha * hm + (1-alpha)*img).astype(np.uint8))\r\n", 132 | " return Image.fromarray(hm)" 133 | ], 134 | "execution_count": null, 135 | "outputs": [] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": { 140 | "id": "Ou2jNi3uKKUQ" 141 | }, 142 | "source": [ 143 | "Import other necessary dependencies." 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "metadata": { 149 | "id": "TVaWm5fKKMwg" 150 | }, 151 | "source": [ 152 | "import clip\r\n", 153 | "from torchray.attribution.grad_cam import grad_cam" 154 | ], 155 | "execution_count": null, 156 | "outputs": [] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": { 161 | "id": "88EACKVtLH_u" 162 | }, 163 | "source": [ 164 | "User defined function to get the CLIP ResNet50 model." 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "metadata": { 170 | "id": "thqra1WYLNW2" 171 | }, 172 | "source": [ 173 | "def get_model():\r\n", 174 | " return clip.load(\"RN50\", device=device, jit=False)" 175 | ], 176 | "execution_count": null, 177 | "outputs": [] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "metadata": { 182 | "id": "lLUJNcH9N22I" 183 | }, 184 | "source": [ 185 | "User defined function to upload an image." 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "metadata": { 191 | "id": "O2TXJnWHOAGn" 192 | }, 193 | "source": [ 194 | "def upload_files():\r\n", 195 | " from google.colab import files\r\n", 196 | " uploaded = files.upload()\r\n", 197 | " for k, v in uploaded.items():\r\n", 198 | " open(k, 'wb').write(v)\r\n", 199 | " return list(uploaded.keys())" 200 | ], 201 | "execution_count": null, 202 | "outputs": [] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "metadata": { 207 | "id": "6ui3gxPDLPK5" 208 | }, 209 | "source": [ 210 | "Set the device to use." 211 | ] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "metadata": { 216 | "id": "zxHwcOHwLR_Q" 217 | }, 218 | "source": [ 219 | "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"" 220 | ], 221 | "execution_count": null, 222 | "outputs": [] 223 | }, 224 | { 225 | "cell_type": "markdown", 226 | "metadata": { 227 | "id": "oQaxvnefLXQj" 228 | }, 229 | "source": [ 230 | "Build the Form." 231 | ] 232 | }, 233 | { 234 | "cell_type": "code", 235 | "metadata": { 236 | "id": "7P_lWct7LYCP" 237 | }, 238 | "source": [ 239 | "#@title Options\r\n", 240 | "\r\n", 241 | "alpha = '0.5' #@param [\"0.5\", \"0,7\", \"0.8\"]\r\n", 242 | "layer = 'layer4.2.relu' #@param [\"layer4.2.relu\"]" 243 | ], 244 | "execution_count": null, 245 | "outputs": [] 246 | }, 247 | { 248 | "cell_type": "markdown", 249 | "metadata": { 250 | "id": "HjnEQ-y_OHLh" 251 | }, 252 | "source": [ 253 | "Upload an image." 254 | ] 255 | }, 256 | { 257 | "cell_type": "code", 258 | "metadata": { 259 | "id": "p4nOmE7lOJN2" 260 | }, 261 | "source": [ 262 | "uploaded_image_list = upload_files()" 263 | ], 264 | "execution_count": null, 265 | "outputs": [] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": { 270 | "id": "gzpiM6nkPLwm" 271 | }, 272 | "source": [ 273 | "Enter some descriptive text." 274 | ] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "metadata": { 279 | "id": "qktYYW6KPP4q" 280 | }, 281 | "source": [ 282 | "#@title Enter some descriptive texts\r\n", 283 | "\r\n", 284 | "textarea = 'a pizza; a beer' #@param {type:\"string\"}\r\n", 285 | "prefix = 'an image of' #@param {type:\"string\"}" 286 | ], 287 | "execution_count": null, 288 | "outputs": [] 289 | }, 290 | { 291 | "cell_type": "markdown", 292 | "metadata": { 293 | "id": "-jmJkNPKOchM" 294 | }, 295 | "source": [ 296 | "Read and preprocess the uploaded image." 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "metadata": { 302 | "id": "5byv-IjbOnx0" 303 | }, 304 | "source": [ 305 | "image_raw = Image.open(uploaded_image_list[0])" 306 | ], 307 | "execution_count": null, 308 | "outputs": [] 309 | }, 310 | { 311 | "cell_type": "code", 312 | "metadata": { 313 | "id": "bM4SPqA3Ovw-" 314 | }, 315 | "source": [ 316 | "model, preprocess = get_model()" 317 | ], 318 | "execution_count": null, 319 | "outputs": [] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "metadata": { 324 | "id": "Ko55OULhO6W-" 325 | }, 326 | "source": [ 327 | "image = preprocess(image_raw).unsqueeze(0).to(device)" 328 | ], 329 | "execution_count": null, 330 | "outputs": [] 331 | }, 332 | { 333 | "cell_type": "markdown", 334 | "metadata": { 335 | "id": "NcHqXCSvO9OS" 336 | }, 337 | "source": [ 338 | "Preprocess text." 339 | ] 340 | }, 341 | { 342 | "cell_type": "code", 343 | "metadata": { 344 | "id": "rCS-Hjx0PnZN" 345 | }, 346 | "source": [ 347 | "prefix = prefix.strip()\r\n", 348 | "if len(prefix) > 0:\r\n", 349 | " categories = [f\"{prefix} {x.strip()}\" for x in textarea.split(';')]\r\n", 350 | "else:\r\n", 351 | " categories = [x.strip() for x in textarea.split(';')]\r\n", 352 | "text = clip.tokenize(categories).to(device)" 353 | ], 354 | "execution_count": null, 355 | "outputs": [] 356 | }, 357 | { 358 | "cell_type": "markdown", 359 | "metadata": { 360 | "id": "p-ajGOLKQeb4" 361 | }, 362 | "source": [ 363 | "Calculate the saliency map." 364 | ] 365 | }, 366 | { 367 | "cell_type": "code", 368 | "metadata": { 369 | "id": "H_aJ8yvkQe52" 370 | }, 371 | "source": [ 372 | "with torch.no_grad():\r\n", 373 | " image_features = model.encode_image(image)\r\n", 374 | " text_features = model.encode_text(text)\r\n", 375 | " image_features_norm = image_features.norm(dim=-1, keepdim=True)\r\n", 376 | " image_features_new = image_features / image_features_norm\r\n", 377 | " text_features_norm = text_features.norm(dim=-1, keepdim=True)\r\n", 378 | " text_features_new = text_features / text_features_norm\r\n", 379 | " logit_scale = model.logit_scale.exp()\r\n", 380 | " logits_per_image = logit_scale * image_features_new @ text_features_new.t()\r\n", 381 | " probs = logits_per_image.softmax(dim=-1).cpu().numpy().tolist()\r\n", 382 | "\r\n", 383 | "saliency = grad_cam(model.visual, image.type(model.dtype), image_features, saliency_layer=layer)" 384 | ], 385 | "execution_count": null, 386 | "outputs": [] 387 | }, 388 | { 389 | "cell_type": "code", 390 | "metadata": { 391 | "id": "3ErqI-wbQ50Q" 392 | }, 393 | "source": [ 394 | "hm = heatmap(image[0], saliency[0][0,].detach().type(torch.float32).cpu(), alpha=alpha)" 395 | ], 396 | "execution_count": null, 397 | "outputs": [] 398 | }, 399 | { 400 | "cell_type": "markdown", 401 | "metadata": { 402 | "id": "NwsP1Mf5R_j6" 403 | }, 404 | "source": [ 405 | "Collect the images." 406 | ] 407 | }, 408 | { 409 | "cell_type": "code", 410 | "metadata": { 411 | "id": "DTr5uD1dR-dl" 412 | }, 413 | "source": [ 414 | "collect_images = []\r\n", 415 | "for i in range(len(categories)):\r\n", 416 | " # mutliply the normalized text embedding with image norm to get approx image embedding\r\n", 417 | " text_prediction = (text_features_new[[i]] * image_features_norm)\r\n", 418 | " saliency = grad_cam(model.visual, image.type(model.dtype), text_prediction, saliency_layer=layer)\r\n", 419 | " hm = heatmap(image[0], saliency[0][0,].detach().type(torch.float32).cpu(), alpha=alpha)\r\n", 420 | " collect_images.append(hm)\r\n", 421 | "logits = logits_per_image.cpu().numpy().tolist()[0]" 422 | ], 423 | "execution_count": null, 424 | "outputs": [] 425 | }, 426 | { 427 | "cell_type": "markdown", 428 | "metadata": { 429 | "id": "2iLjHdGITFd6" 430 | }, 431 | "source": [ 432 | "Show the Grad Cam for text embeddings." 433 | ] 434 | }, 435 | { 436 | "cell_type": "code", 437 | "metadata": { 438 | "id": "0V9JDAEkSSkT" 439 | }, 440 | "source": [ 441 | "text_embeddings = [f\"{x} - {str(round(y, 3))}/{str(round(l, 2))}\" for (x, y, l) in\r\n", 442 | " zip(categories, probs[0], logits)]\r\n", 443 | "for image_idx in (0, len(collect_images) - 1):\r\n", 444 | " display(collect_images[image_idx])\r\n", 445 | " print(text_embeddings[image_idx])\r\n" 446 | ], 447 | "execution_count": null, 448 | "outputs": [] 449 | }, 450 | { 451 | "cell_type": "markdown", 452 | "metadata": { 453 | "id": "v9k5klM9Znl9" 454 | }, 455 | "source": [ 456 | "Show the original image and Grad Cam for image embedding." 457 | ] 458 | }, 459 | { 460 | "cell_type": "code", 461 | "metadata": { 462 | "id": "Vxx7DdusaeXg" 463 | }, 464 | "source": [ 465 | "display(Image.fromarray((torch_to_rgba(image[0]).numpy() * 255.).astype(np.uint8)), hm)" 466 | ], 467 | "execution_count": null, 468 | "outputs": [] 469 | } 470 | ] 471 | } -------------------------------------------------------------------------------- /Colab/MiniCLIP/README.md: -------------------------------------------------------------------------------- 1 | # MiniCLIP: demo to explore text descriptions and saliency maps for Clip models 2 | Porting to Colab notebook of a Streamlit [demo](https://github.com/HendrikStrobelt/miniClip) from [Hendrik Strobelt](http://hendrik.strobelt.com/). It uses OpenAI's [CLIP](https://github.com/openai/CLIP), ResNet50, to observe how similar are the embeddings of given descriptions to the image embeddings under the Clip model and what is the salience map when using the text-embeddings of given descriptions as logits for the image model part of Clip. 3 | A CPU runtime is enough to run this notebook, but it works with a GPU runtime too. -------------------------------------------------------------------------------- /Colab/MobileFaceSwap/MobileFaceSwap.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "NSlkZYbj4ivr" 7 | }, 8 | "source": [ 9 | "# MobileFaceSwap - Inference on CPU\n", 10 | "A notebook to execute face swap on your images or videos using the [MobileFaceSwap framework](https://arxiv.org/abs/2201.03808), using only CPUs. No need for hardware acceleration in order to execute the code in this notebook cells. \n", 11 | "Credits for the MobileFaceSwap paper: \n", 12 | "\n", 13 | "\n", 14 | "```\n", 15 | "@inproceedings{xu2022MobileFaceSwap,\n", 16 | " title={MobileFaceSwap: A Lightweight Framework for Video Face Swapping},\n", 17 | " author={Xu, Zhiliang and Hong, Zhibin and Ding, Changxing and Zhu, Zhen and Han, Junyu and Liu, Jingtuo and Ding, Errui},\n", 18 | " booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},\n", 19 | " year={2022}\n", 20 | "}\n", 21 | "```\n", 22 | "\n" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": { 28 | "id": "IE2o1s5m5c5S" 29 | }, 30 | "source": [ 31 | "## Settings" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": { 37 | "id": "TmIBWT-e5fkq" 38 | }, 39 | "source": [ 40 | "Clone the official GitHub repository." 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "metadata": { 47 | "id": "x7E3Ax7ejd9p" 48 | }, 49 | "outputs": [], 50 | "source": [ 51 | "!git clone https://github.com/Seanseattle/MobileFaceSwap.git\n", 52 | "%cd MobileFaceSwap" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": { 58 | "id": "23T02Hzo5kV_" 59 | }, 60 | "source": [ 61 | "Install the missing dependencies in the Colab runtime (PaddlePaddle, InsightFace and ONNX Runtime)." 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "metadata": { 68 | "id": "snI4jaJhjpLY" 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "!pip install paddlepaddle==2.1.2" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": { 79 | "id": "O9GhC4jBj3J7" 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "!pip install insightface==0.2.1" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": { 90 | "id": "PbcTfVWXm6bY" 91 | }, 92 | "outputs": [], 93 | "source": [ 94 | "!pip install onnxruntime" 95 | ] 96 | }, 97 | { 98 | "cell_type": "markdown", 99 | "metadata": { 100 | "id": "VvEyrNiY6agR" 101 | }, 102 | "source": [ 103 | "Download the pretrained model." 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": { 110 | "id": "TutCOLSCj8Ch" 111 | }, 112 | "outputs": [], 113 | "source": [ 114 | "!gdown --id 1ZIzGLDB15GRAZAbkfNR0hNWdgQpxeA_r\n", 115 | "!tar -xvf ./checkpoints.tar\n", 116 | "!rm ./checkpoints.tar" 117 | ] 118 | }, 119 | { 120 | "cell_type": "markdown", 121 | "metadata": { 122 | "id": "RhffyxrN6yV6" 123 | }, 124 | "source": [ 125 | "Create a directory where to store the uploaded images and/or videos." 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "metadata": { 132 | "id": "F_sgOyDClkPZ" 133 | }, 134 | "outputs": [], 135 | "source": [ 136 | "!mkdir ./data" 137 | ] 138 | }, 139 | { 140 | "cell_type": "markdown", 141 | "metadata": { 142 | "id": "tkbYUhit7zra" 143 | }, 144 | "source": [ 145 | "Define a function to upload files to Colab." 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": null, 151 | "metadata": { 152 | "id": "04npK1Wg79N5" 153 | }, 154 | "outputs": [], 155 | "source": [ 156 | "import os\n", 157 | "import shutil\n", 158 | "from google.colab import files\n", 159 | "\n", 160 | "def upload_files(upload_path):\n", 161 | " uploaded = files.upload()\n", 162 | " for filename, content in uploaded.items():\n", 163 | " dst_path = os.path.join(upload_path, filename)\n", 164 | " shutil.move(filename, dst_path)\n", 165 | " return list(uploaded.keys())" 166 | ] 167 | }, 168 | { 169 | "cell_type": "markdown", 170 | "metadata": { 171 | "id": "OUrmuphT66tF" 172 | }, 173 | "source": [ 174 | "## Inference" 175 | ] 176 | }, 177 | { 178 | "cell_type": "markdown", 179 | "metadata": { 180 | "id": "EOrWbQ-h7C3V" 181 | }, 182 | "source": [ 183 | "#### Image Swap" 184 | ] 185 | }, 186 | { 187 | "cell_type": "markdown", 188 | "metadata": { 189 | "id": "yW_ZOVHc68oU" 190 | }, 191 | "source": [ 192 | "Upload your own images (source and target)." 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": null, 198 | "metadata": { 199 | "id": "NTK3uwKN8DIo" 200 | }, 201 | "outputs": [], 202 | "source": [ 203 | "image_input_dir = '/content/MobileFaceSwap/data/'\n", 204 | "uploaded_images = upload_files(image_input_dir)" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": null, 210 | "metadata": { 211 | "id": "tv9SAGBwBpwL" 212 | }, 213 | "outputs": [], 214 | "source": [ 215 | "os.environ['TARGET_IMAGE'] = uploaded_images[0]\n", 216 | "os.environ['SOURCE_IMAGE'] = uploaded_images[1]" 217 | ] 218 | }, 219 | { 220 | "cell_type": "markdown", 221 | "metadata": { 222 | "id": "QeV71G23IuCc" 223 | }, 224 | "source": [ 225 | "Run inference." 226 | ] 227 | }, 228 | { 229 | "cell_type": "code", 230 | "execution_count": null, 231 | "metadata": { 232 | "id": "ei3aesMSkYhc" 233 | }, 234 | "outputs": [], 235 | "source": [ 236 | "!python ./image_test.py --target_img_path data/$TARGET_IMAGE --source_img_path data/$SOURCE_IMAGE --output_dir results" 237 | ] 238 | }, 239 | { 240 | "cell_type": "markdown", 241 | "metadata": { 242 | "id": "eGmUO0r1Cdnm" 243 | }, 244 | "source": [ 245 | "Display the result." 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": null, 251 | "metadata": { 252 | "id": "WO19QQjGCf3L" 253 | }, 254 | "outputs": [], 255 | "source": [ 256 | "import cv2\n", 257 | "from google.colab.patches import cv2_imshow\n", 258 | "\n", 259 | "img = cv2.imread(os.path.join('./results', uploaded_images[0]))\n", 260 | "cv2_imshow(img)" 261 | ] 262 | }, 263 | { 264 | "cell_type": "markdown", 265 | "metadata": { 266 | "id": "yj-c77qyE7qw" 267 | }, 268 | "source": [ 269 | "### Video Swap" 270 | ] 271 | }, 272 | { 273 | "cell_type": "markdown", 274 | "metadata": { 275 | "id": "z6K89WXxFK8u" 276 | }, 277 | "source": [ 278 | "Upload your own source image and target video." 279 | ] 280 | }, 281 | { 282 | "cell_type": "code", 283 | "execution_count": null, 284 | "metadata": { 285 | "id": "hK4waac1FRYT" 286 | }, 287 | "outputs": [], 288 | "source": [ 289 | "uploaded_image_and_video = upload_files(image_input_dir)" 290 | ] 291 | }, 292 | { 293 | "cell_type": "code", 294 | "execution_count": null, 295 | "metadata": { 296 | "id": "RLtsEXFOGSkM" 297 | }, 298 | "outputs": [], 299 | "source": [ 300 | "import fnmatch\n", 301 | "\n", 302 | "uploaded_target_video = fnmatch.filter(uploaded_image_and_video, '*mp4')[0]\n", 303 | "os.environ['TARGET_VIDEO'] = uploaded_target_video\n", 304 | "os.environ['SOURCE_IMAGE'] = fnmatch.filter(uploaded_image_and_video, '*jpg')[0]" 305 | ] 306 | }, 307 | { 308 | "cell_type": "markdown", 309 | "metadata": { 310 | "id": "OC5Hzg1cIyDE" 311 | }, 312 | "source": [ 313 | "Run inference." 314 | ] 315 | }, 316 | { 317 | "cell_type": "code", 318 | "execution_count": null, 319 | "metadata": { 320 | "id": "kjy2iupskk85" 321 | }, 322 | "outputs": [], 323 | "source": [ 324 | "!python ./video_test.py --target_video_path data/$TARGET_VIDEO --source_img_path data/$SOURCE_IMAGE --output_path results " 325 | ] 326 | }, 327 | { 328 | "cell_type": "markdown", 329 | "metadata": { 330 | "id": "Ko766eLJJ0Sl" 331 | }, 332 | "source": [ 333 | "Display the result video." 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": null, 339 | "metadata": { 340 | "id": "k9aea2vu5DHH" 341 | }, 342 | "outputs": [], 343 | "source": [ 344 | "from moviepy.editor import VideoFileClip\n", 345 | "\n", 346 | "result_video_path = os.path.join('./results', uploaded_target_video)\n", 347 | "\n", 348 | "clip = VideoFileClip(result_video_path)\n", 349 | "clip.ipython_display(width=280)" 350 | ] 351 | } 352 | ], 353 | "metadata": { 354 | "colab": { 355 | "name": "MobileFaceSwap.ipynb", 356 | "provenance": [] 357 | }, 358 | "kernelspec": { 359 | "display_name": "Python 3", 360 | "name": "python3" 361 | }, 362 | "language_info": { 363 | "name": "python" 364 | } 365 | }, 366 | "nbformat": 4, 367 | "nbformat_minor": 0 368 | } 369 | -------------------------------------------------------------------------------- /Colab/MobileFaceSwap/README.md: -------------------------------------------------------------------------------- 1 | # MobileFaceSwap - Inference on CPU 2 | A notebook to execute face swap on your images or videos using the [MobileFaceSwap framework](https://arxiv.org/abs/2201.03808), using only CPUs. No need for hardware acceleration in order to execute the code in this notebook cells. 3 | 4 | **Source**: 5 | ![](demo_images/source.jpg) 6 | **Target**: 7 | ![](demo_images/target.jpg) 8 | **Swap result**: 9 | ![](demo_images/result.jpg) -------------------------------------------------------------------------------- /Colab/MobileFaceSwap/demo_images/result.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/MobileFaceSwap/demo_images/result.jpg -------------------------------------------------------------------------------- /Colab/MobileFaceSwap/demo_images/source.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/MobileFaceSwap/demo_images/source.jpg -------------------------------------------------------------------------------- /Colab/MobileFaceSwap/demo_images/target.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/MobileFaceSwap/demo_images/target.jpg -------------------------------------------------------------------------------- /Colab/OWL-ViT/OWL_ViT_Evaluation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "provenance": [] 7 | }, 8 | "kernelspec": { 9 | "name": "python3", 10 | "display_name": "Python 3" 11 | }, 12 | "language_info": { 13 | "name": "python" 14 | } 15 | }, 16 | "cells": [ 17 | { 18 | "cell_type": "markdown", 19 | "source": [ 20 | "# Object detection with Vision Transformer for Open-World Localization (OWL-ViT)\n", 21 | "The **[OWL-ViT](https://arxiv.org/abs/2205.06230)** model is an **open-vocabulary object detection model** that uses the standard Vision Transformer to perform detection. \n", 22 | "![OWL-ViT](https://github.com/google-research/scenic/raw/main/scenic/projects/owl_vit/data/owl_vit_schematic.png) \n", 23 | "Given an image and a free-text query, it finds objects matching that query in the image. It can also do one-shot object detection, i.e. detect objects based on a single example image. This notebook is to evaluate both tasks through the pre-trained model available in the Hugging Face's *transformers* library. \n", 24 | "As a result of my evaluation of this model, it looks like it is robust with reference to the open-vocabulary object detection task, but not yet robust enough to perform accurate one-shot object detection. \n", 25 | "A free Colab VM without hardware acceleration is enough to execute the code in this notebook." 26 | ], 27 | "metadata": { 28 | "id": "88X-tLr8gM8I" 29 | } 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "source": [ 34 | "## Settings" 35 | ], 36 | "metadata": { 37 | "id": "pJfqB0HLRo7h" 38 | } 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "source": [ 43 | "Install the missing requirements (actually only *transformers* not present in the Colab VM)." 44 | ], 45 | "metadata": { 46 | "id": "qsXIa1hhRs_j" 47 | } 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": null, 52 | "metadata": { 53 | "id": "hYwEzTmdz2IZ" 54 | }, 55 | "outputs": [], 56 | "source": [ 57 | "!pip install transformers" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "source": [ 63 | "Import the general packages/classes to use." 64 | ], 65 | "metadata": { 66 | "id": "EX9kQ9D2SKBe" 67 | } 68 | }, 69 | { 70 | "cell_type": "code", 71 | "source": [ 72 | "import torch\n", 73 | "from PIL import Image\n", 74 | "from transformers import OwlViTProcessor, OwlViTForObjectDetection" 75 | ], 76 | "metadata": { 77 | "id": "LTIQIjYD0AP9" 78 | }, 79 | "execution_count": null, 80 | "outputs": [] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "source": [ 85 | "Load the pre-trained ViT processor and model." 86 | ], 87 | "metadata": { 88 | "id": "XukaunmjSj3i" 89 | } 90 | }, 91 | { 92 | "cell_type": "code", 93 | "source": [ 94 | "processor = OwlViTProcessor.from_pretrained(\"google/owlvit-base-patch32\")\n", 95 | "model = OwlViTForObjectDetection.from_pretrained(\"google/owlvit-base-patch32\")" 96 | ], 97 | "metadata": { 98 | "id": "wv-DwebY0Fel" 99 | }, 100 | "execution_count": null, 101 | "outputs": [] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "source": [ 106 | "Define a function to upload images." 107 | ], 108 | "metadata": { 109 | "id": "jSgwJjz70L3H" 110 | } 111 | }, 112 | { 113 | "cell_type": "code", 114 | "source": [ 115 | "from google.colab import files\n", 116 | "\n", 117 | "def upload_files():\n", 118 | " uploaded = files.upload()\n", 119 | " for k, v in uploaded.items():\n", 120 | " open(k, 'wb').write(v)\n", 121 | " return list(uploaded.keys())" 122 | ], 123 | "metadata": { 124 | "id": "CFEjxYkJ0Na0" 125 | }, 126 | "execution_count": null, 127 | "outputs": [] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "source": [ 132 | "## Open-vocabulary Object Detection" 133 | ], 134 | "metadata": { 135 | "id": "EJva2nfHTI54" 136 | } 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "source": [ 141 | "Define a function to add bounding boxes to the input image." 142 | ], 143 | "metadata": { 144 | "id": "Jqi466-oUDKs" 145 | } 146 | }, 147 | { 148 | "cell_type": "code", 149 | "source": [ 150 | "import cv2\n", 151 | "import numpy as np\n", 152 | "\n", 153 | "def add_bounding_boxes(boxes, scores, labels, text_queries, img):\n", 154 | " img = np.array(img)\n", 155 | "\n", 156 | " font = cv2.FONT_HERSHEY_SIMPLEX\n", 157 | "\n", 158 | " for box, score, label in zip(boxes, scores, labels):\n", 159 | " box = [int(i) for i in box.tolist()]\n", 160 | "\n", 161 | " if score >= score_threshold:\n", 162 | " img = cv2.rectangle(img, box[:2], box[2:], (255,0,0), 5)\n", 163 | " if box[3] + 25 > 768:\n", 164 | " y = box[3] - 10\n", 165 | " else:\n", 166 | " y = box[3] + 25\n", 167 | " \n", 168 | " img = cv2.putText(\n", 169 | " img, text_queries[label], (box[0], y), font, 1, (255,0,0), 2, cv2.LINE_AA\n", 170 | " )\n", 171 | "\n", 172 | " return Image.fromarray(img, 'RGB')" 173 | ], 174 | "metadata": { 175 | "id": "4xqM5s5t2N-H" 176 | }, 177 | "execution_count": null, 178 | "outputs": [] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "source": [ 183 | "Upload an image and display it." 184 | ], 185 | "metadata": { 186 | "id": "FM-tsjZGTOzx" 187 | } 188 | }, 189 | { 190 | "cell_type": "code", 191 | "source": [ 192 | "uploaded_image_list = upload_files()\n", 193 | "image = Image.open(uploaded_image_list[0])\n", 194 | "display(image)" 195 | ], 196 | "metadata": { 197 | "id": "OCbe4i4V0rfO" 198 | }, 199 | "execution_count": null, 200 | "outputs": [] 201 | }, 202 | { 203 | "cell_type": "markdown", 204 | "source": [ 205 | "Indicate a list of free texts for querying and the minimal score threshold for the object detection." 206 | ], 207 | "metadata": { 208 | "id": "BzEZjYnCTS5h" 209 | } 210 | }, 211 | { 212 | "cell_type": "code", 213 | "source": [ 214 | "texts = [[\"tuxedo cat\", \"tabby cat\"]]\n", 215 | "score_threshold = 0.08" 216 | ], 217 | "metadata": { 218 | "id": "b0krqHxm0zeE" 219 | }, 220 | "execution_count": null, 221 | "outputs": [] 222 | }, 223 | { 224 | "cell_type": "markdown", 225 | "source": [ 226 | "Do object detection on the uploaded image, querying by the list of choosen free texts." 227 | ], 228 | "metadata": { 229 | "id": "Lh0MJtshU72D" 230 | } 231 | }, 232 | { 233 | "cell_type": "code", 234 | "source": [ 235 | "inputs = processor(text=texts, images=image, return_tensors=\"pt\")\n", 236 | "outputs = model(**inputs)" 237 | ], 238 | "metadata": { 239 | "id": "V4ovVPUu020a" 240 | }, 241 | "execution_count": null, 242 | "outputs": [] 243 | }, 244 | { 245 | "cell_type": "markdown", 246 | "source": [ 247 | "Target image height and width to rescale box predictions and then convert the outputs (bounding boxes and class logits) to the COCO API format." 248 | ], 249 | "metadata": { 250 | "id": "LPhQzXKCVKH0" 251 | } 252 | }, 253 | { 254 | "cell_type": "code", 255 | "source": [ 256 | "target_sizes = torch.Tensor([image.size[::-1]])\n", 257 | "\n", 258 | "results = processor.post_process(outputs=outputs, target_sizes=target_sizes)" 259 | ], 260 | "metadata": { 261 | "id": "kPGozPnu05Ax" 262 | }, 263 | "execution_count": null, 264 | "outputs": [] 265 | }, 266 | { 267 | "cell_type": "markdown", 268 | "source": [ 269 | "Retrieve the predictions for the input image by corresponding text queries." 270 | ], 271 | "metadata": { 272 | "id": "CIe_Twr8VwxA" 273 | } 274 | }, 275 | { 276 | "cell_type": "code", 277 | "source": [ 278 | "i = 0\n", 279 | "text = texts[i]\n", 280 | "boxes, scores, labels = results[i][\"boxes\"], results[i][\"scores\"], results[i][\"labels\"]\n", 281 | "\n", 282 | "for box, score, label in zip(boxes, scores, labels):\n", 283 | " box = [round(i, 2) for i in box.tolist()]\n", 284 | " if score >= score_threshold:\n", 285 | " print(f\"Detected {text[label]} with confidence {round(score.item(), 3)} at location {box}\")" 286 | ], 287 | "metadata": { 288 | "id": "fkeXJSOU0-B8" 289 | }, 290 | "execution_count": null, 291 | "outputs": [] 292 | }, 293 | { 294 | "cell_type": "markdown", 295 | "source": [ 296 | "Generate the bounding boxes for each prediction, overlay them to the input image and finally display it to the next code cell output." 297 | ], 298 | "metadata": { 299 | "id": "uMffKUJnWTzX" 300 | } 301 | }, 302 | { 303 | "cell_type": "code", 304 | "source": [ 305 | "labelled_image = add_bounding_boxes(boxes, scores, labels, text, image)\n", 306 | "display(labelled_image)" 307 | ], 308 | "metadata": { 309 | "id": "KaJxHJ4y2Oxl" 310 | }, 311 | "execution_count": null, 312 | "outputs": [] 313 | }, 314 | { 315 | "cell_type": "markdown", 316 | "source": [ 317 | "## Image-guided Object Detection" 318 | ], 319 | "metadata": { 320 | "id": "mbTTa_r17bfu" 321 | } 322 | }, 323 | { 324 | "cell_type": "markdown", 325 | "source": [ 326 | "Upload the target and query images and display them." 327 | ], 328 | "metadata": { 329 | "id": "n00M9rOqWp2Q" 330 | } 331 | }, 332 | { 333 | "cell_type": "code", 334 | "source": [ 335 | "uploaded_image_guided_list = upload_files()\n", 336 | "\n", 337 | "target_image = Image.open(uploaded_image_guided_list[0])\n", 338 | "target_sizes = torch.Tensor([target_image.size[::-1]])\n", 339 | "\n", 340 | "query_image = Image.open(uploaded_image_guided_list[1])\n", 341 | "\n", 342 | "display(target_image)\n", 343 | "display(query_image)" 344 | ], 345 | "metadata": { 346 | "id": "ZHvsRkeD46Y2" 347 | }, 348 | "execution_count": null, 349 | "outputs": [] 350 | }, 351 | { 352 | "cell_type": "markdown", 353 | "source": [ 354 | "Process the target and query images." 355 | ], 356 | "metadata": { 357 | "id": "BPre-sQXW_Dw" 358 | } 359 | }, 360 | { 361 | "cell_type": "code", 362 | "source": [ 363 | "device = 'cpu'\n", 364 | "inputs = processor(images=target_image, query_images=query_image, return_tensors=\"pt\").to(device)\n", 365 | "\n", 366 | "for key, val in inputs.items():\n", 367 | " print(f\"{key}: {val.shape}\")" 368 | ], 369 | "metadata": { 370 | "id": "OL14nbPj-lkn" 371 | }, 372 | "execution_count": null, 373 | "outputs": [] 374 | }, 375 | { 376 | "cell_type": "markdown", 377 | "source": [ 378 | "Get the predictions." 379 | ], 380 | "metadata": { 381 | "id": "FMAvA6F9XHmS" 382 | } 383 | }, 384 | { 385 | "cell_type": "code", 386 | "source": [ 387 | "with torch.no_grad():\n", 388 | " outputs = model.image_guided_detection(**inputs)\n", 389 | "\n", 390 | "for k, val in outputs.items():\n", 391 | " if k not in {\"text_model_output\", \"vision_model_output\"}:\n", 392 | " print(f\"{k}: shape of {val.shape}\")\n", 393 | "\n", 394 | "print(\"\\nVision model outputs\")\n", 395 | "for k, val in outputs.vision_model_output.items():\n", 396 | " print(f\"{k}: shape of {val.shape}\") " 397 | ], 398 | "metadata": { 399 | "id": "U15CFFT--pJD" 400 | }, 401 | "execution_count": null, 402 | "outputs": [] 403 | }, 404 | { 405 | "cell_type": "markdown", 406 | "source": [ 407 | "Generate the bounding boxes for each prediction, overlay them to the target image and finally display it to the next code cell output." 408 | ], 409 | "metadata": { 410 | "id": "e0dyBtkNXRxh" 411 | } 412 | }, 413 | { 414 | "cell_type": "code", 415 | "source": [ 416 | "img = cv2.cvtColor(np.array(target_image), cv2.COLOR_BGR2RGB)\n", 417 | "outputs.logits = outputs.logits.cpu()\n", 418 | "outputs.target_pred_boxes = outputs.target_pred_boxes.cpu() \n", 419 | "\n", 420 | "results = processor.post_process_image_guided_detection(outputs=outputs, threshold=0.6, nms_threshold=0.3, target_sizes=target_sizes)\n", 421 | "boxes, scores = results[0][\"boxes\"], results[0][\"scores\"]\n", 422 | "\n", 423 | "for box, score in zip(boxes, scores):\n", 424 | " box = [int(i) for i in box.tolist()]\n", 425 | "\n", 426 | " img = cv2.rectangle(img, box[:2], box[2:], (255,0,0), 5)\n", 427 | " if box[3] + 25 > 768:\n", 428 | " y = box[3] - 10\n", 429 | " else:\n", 430 | " y = box[3] + 25 \n", 431 | " \n", 432 | "display(Image.fromarray(img[:,:,::-1]))" 433 | ], 434 | "metadata": { 435 | "id": "rmsPUwje_PDF" 436 | }, 437 | "execution_count": null, 438 | "outputs": [] 439 | } 440 | ] 441 | } -------------------------------------------------------------------------------- /Colab/OWL-ViT/README.md: -------------------------------------------------------------------------------- 1 | # Object detection with Vision Transformer for Open-World Localization (OWL-ViT) 2 | The **[OWL-ViT](https://arxiv.org/abs/2205.06230)** model is an **open-vocabulary object detection model** that uses the standard Vision Transformer to perform detection. 3 | ![OWL-ViT](https://github.com/google-research/scenic/raw/main/scenic/projects/owl_vit/data/owl_vit_schematic.png) 4 | Given an image and a free-text query, it finds objects matching that query in the image. It can also do one-shot object detection, i.e. detect objects based on a single example image. This notebook is to evaluate both tasks through the pre-trained model available in the Hugging Face's *transformers* library. 5 | As a result of my evaluation of this model, it looks like it is robust with reference to the open-vocabulary object detection task, but not yet robust enough to perform accurate one-shot object detection. 6 | Execution of the code in this notebook is fast on CPU: no need for hardware acceleration. 7 | ### Example 8 | Search texts: "tuxedo cat", "tabby cat" 9 | ![Example](images/OWL-ViT-Example.png) -------------------------------------------------------------------------------- /Colab/OWL-ViT/images/OWL-ViT-Example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/OWL-ViT/images/OWL-ViT-Example.png -------------------------------------------------------------------------------- /Colab/PackNet-SfM/README.md: -------------------------------------------------------------------------------- 1 | ## Monocular Depth Estimation using the Toyota's PackNet-SfM. 2 | A notebook for a rapid setup and to do inference with the Toyota's [PackNet-SfM](https://github.com/virtualramblas/packnet-sfm) in [Colab](https://colab.research.google.com). 3 | PackNet-SfM is the official PyTorch implementation of the Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion [paper](https://arxiv.org/abs/1905.02693), a generalized self-supervised depth and pose estimation beyond the pinhole model to all central cameras, which allows the learning of meaningful depth and pose on non-pinhole cameras such as fisheye and catadioptric. 4 | ![Demo Image](https://github.com/virtualramblas/python-notebooks-repo/blob/main/Colab/PackNet-SfM/demo-image.png) 5 | -------------------------------------------------------------------------------- /Colab/PackNet-SfM/demo-image.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/PackNet-SfM/demo-image.png -------------------------------------------------------------------------------- /Colab/Pix2Pix/README.md: -------------------------------------------------------------------------------- 1 | ## Pix2Pix for Satellite-to-Map Image Translation in Keras 2 | Notebooks for the data preparation and training of a pix2pix GAN [Keras](https://keras.io/) implementation from [Jason Brownlee](https://machinelearningmastery.com/how-to-develop-a-pix2pix-gan-for-image-to-image-translation/) following [this paper](https://arxiv.org/abs/1611.07004) (the official paper implementation is in PyTorch). The purpose of this Keras model is satellite images translation into Google Maps images. 3 | This repo includes 2 notebooks: 4 | * [Data Preparation](https://github.com/virtualramblas/python-notebooks-repo/blob/main/Colab/Pix2Pix/Pix2Pix_for_Satellite_to_Map_Image_Translation_Data_Preparation.ipynb) 5 | * [Training](https://github.com/virtualramblas/python-notebooks-repo/blob/main/Colab/Pix2Pix/Pix2Pix_for_Satellite_to_Map_Image_Translation_Model_Training.ipynb) 6 | 7 | Execute the Data Preparation notebook first. The prepared training data would be saved in Numpy compressed array format in your mounted Google Drive (follow the instructions in the notebook itself for this task). Then execute the training: the preliminary saved data would be loaded from your Google Drive. At the end of the training you can programmatically save model checkpoint to your mounted Google Drive within the notebook itself. 8 | -------------------------------------------------------------------------------- /Colab/ReplitLM/Code_Generation_with_ReplitLM.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "provenance": [], 7 | "gpuType": "T4" 8 | }, 9 | "kernelspec": { 10 | "name": "python3", 11 | "display_name": "Python 3" 12 | }, 13 | "language_info": { 14 | "name": "python" 15 | }, 16 | "gpuClass": "standard" 17 | }, 18 | "cells": [ 19 | { 20 | "cell_type": "markdown", 21 | "source": [ 22 | "# Code Generation with ReplitLM\n", 23 | "A notebook to generate code in any of the programming languages for which the [ReplitLM](https://github.com/replit/ReplitLM) model has been trained. \n", 24 | "No hardware acceleration required. " 25 | ], 26 | "metadata": { 27 | "id": "-3mn1T4ZySgq" 28 | } 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "source": [ 33 | "Install the missing requirements in the Colab VMs." 34 | ], 35 | "metadata": { 36 | "id": "xx79u_4lj_IY" 37 | } 38 | }, 39 | { 40 | "cell_type": "code", 41 | "source": [ 42 | "!pip install transformers sentencepiece einops accelerate" 43 | ], 44 | "metadata": { 45 | "id": "9U6ItlZ7kDG6" 46 | }, 47 | "execution_count": null, 48 | "outputs": [] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "source": [ 53 | "Download model and tokenizer using the Hugging Face's Transformer library." 54 | ], 55 | "metadata": { 56 | "id": "4mi9WYGSj50n" 57 | } 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": { 63 | "id": "M9MOpgmjjmR2" 64 | }, 65 | "outputs": [], 66 | "source": [ 67 | "import torch\n", 68 | "from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed\n", 69 | "\n", 70 | "model_id = \"replit/replit-code-v1-3b\"\n", 71 | "PAD_TOKEN = \"<|pad|>\"\n", 72 | "EOS_TOKEN = \"<|endoftext|>\"\n", 73 | "UNK_TOKEN = \"<|unk|>\"\n", 74 | "MAX_INPUT_TOKENS = 1024\n", 75 | "\n", 76 | "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", 77 | "\n", 78 | "tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)\n", 79 | "tokenizer.truncation_side = \"left\" # ensures if truncate, then keep the last N tokens of the prompt going L -> R\n", 80 | "if device == \"cuda\":\n", 81 | " model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, low_cpu_mem_usage=True).to(device, dtype=torch.bfloat16)\n", 82 | "else:\n", 83 | " model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, low_cpu_mem_usage=True)\n", 84 | "model.eval()" 85 | ] 86 | }, 87 | { 88 | "cell_type": "markdown", 89 | "source": [ 90 | "Set the prompt and the values of other model parameters." 91 | ], 92 | "metadata": { 93 | "id": "vDOmfexBk1nn" 94 | } 95 | }, 96 | { 97 | "cell_type": "code", 98 | "source": [ 99 | "prompt = 'def sieve_eratosthenes(n):' #@param {type:\"string\"}\n", 100 | "max_tokens = 112 #@param {type:\"slider\", min:10, max:1024, step:1}\n", 101 | "temperature = 0.2 #@param {type:\"slider\", min:0.1, max:2.5, step:0.1}\n", 102 | "seed = 42 #@param {type:\"slider\", min:0, max:1000, step:1}\n", 103 | "top_p = 0.9 #@param {type:\"slider\", min:0, max:1, step:0.1}\n", 104 | "top_k = 4 #@param {type:\"slider\", min:1, max:64, step:1}\n", 105 | "repetition_penalty = 1.0 #@param {type:\"slider\", min:1.0, max:1.9, step:0.1}\n", 106 | "use_cache = True #@param {type:\"boolean\"}\n" 107 | ], 108 | "metadata": { 109 | "id": "HA4ztsPhk8Jv" 110 | }, 111 | "execution_count": null, 112 | "outputs": [] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "source": [ 117 | "Generate code." 118 | ], 119 | "metadata": { 120 | "id": "cffzlKHVlFno" 121 | } 122 | }, 123 | { 124 | "cell_type": "code", 125 | "source": [ 126 | "model_input = tokenizer.encode(prompt, return_tensors=\"pt\", max_length=MAX_INPUT_TOKENS, truncation=True).to(device)\n", 127 | "set_seed(seed)\n", 128 | "model_output = model.generate(model_input, \n", 129 | " max_length=max_tokens, \n", 130 | " do_sample=True, \n", 131 | " top_p=top_p, \n", 132 | " top_k=top_k, \n", 133 | " temperature=temperature, \n", 134 | " num_return_sequences=1, \n", 135 | " repetition_penalty = repetition_penalty,\n", 136 | " pad_token_id=tokenizer.pad_token_id,\n", 137 | " eos_token_id=tokenizer.eos_token_id,\n", 138 | " use_cache=use_cache,)" 139 | ], 140 | "metadata": { 141 | "id": "O4qwOPHvlH5O" 142 | }, 143 | "execution_count": null, 144 | "outputs": [] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "source": [ 149 | "Decode the generated output." 150 | ], 151 | "metadata": { 152 | "id": "Eeede5DdlPyF" 153 | } 154 | }, 155 | { 156 | "cell_type": "code", 157 | "source": [ 158 | "generated_code = tokenizer.decode(model_output[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)\n", 159 | "print(generated_code)\n", 160 | "prompt = generated_code" 161 | ], 162 | "metadata": { 163 | "id": "7woibwtKlTDm" 164 | }, 165 | "execution_count": null, 166 | "outputs": [] 167 | } 168 | ] 169 | } -------------------------------------------------------------------------------- /Colab/ReplitLM/README.md: -------------------------------------------------------------------------------- 1 | # Code Generation with ReplitLM 2 | A notebook to generate code in any of the programming languages for which the [ReplitLM](https://github.com/replit/ReplitLM) model has been trained. 3 | No hardware acceleration required. -------------------------------------------------------------------------------- /Colab/SDImageVariations/README.md: -------------------------------------------------------------------------------- 1 | # Stable Diffusion Image Variations 2 | A user-friendly notebook for image-to-image generation using a fine-tuned version of Stable Diffusion conditioned on CLIP image embeddings. It uses the code from the [lambad-diffusers](https://github.com/LambdaLabsML/lambda-diffusers) repo. Execution details within the notebook text cells. 3 | GPU acceleration is needed to execute the code in this notebook. 4 | 5 | ![Example](images/sample_image.jpg) -------------------------------------------------------------------------------- /Colab/SDImageVariations/Stable_Diffusion_Image_Variations.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "metadata": { 7 | "id": "qXhK8zq6tLLD" 8 | }, 9 | "source": [ 10 | "# Stable Diffusion Image Variations\n", 11 | "A user-friendly notebook for image-to-image generation using a fine-tuned version of Stable Diffusion conditioned on CLIP image embeddings. It uses the code from the [lambad-diffusers](https://github.com/LambdaLabsML/lambda-diffusers) repo. \n", 12 | "GPU acceleration is needed to execute the code in this notebook." 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": { 18 | "id": "KXvZKYK3lWQm" 19 | }, 20 | "source": [ 21 | "## Settings" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": { 27 | "id": "tDN-QZk6x_lY" 28 | }, 29 | "source": [ 30 | "Clone the lambda diffusers GitHub repo." 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": { 37 | "id": "AmYjRilax662" 38 | }, 39 | "outputs": [], 40 | "source": [ 41 | "!git clone https://github.com/LambdaLabsML/lambda-diffusers.git\n", 42 | "%cd lambda-diffusers" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": { 48 | "id": "wXhrjlxyyMsY" 49 | }, 50 | "source": [ 51 | "Install the requirements. Please refer to the code cell below and don't use the provided requirements.txt file, as it doesn't reflect the latest changes." 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": null, 57 | "metadata": { 58 | "id": "o6nGRW4QyY5_" 59 | }, 60 | "outputs": [], 61 | "source": [ 62 | "!pip install transformers==4.25\n", 63 | "!pip install ftfy==6.1.1\n", 64 | "!pip instal lPillow==9.2.0\n", 65 | "!pip install onnxruntime==1.12.1\n", 66 | "!pip installscikit-image==0.19.3\n", 67 | "!pip install -e .\n", 68 | "!pip install accelerate" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": { 74 | "id": "WkFBCyIHyZ-U" 75 | }, 76 | "source": [ 77 | "Define a function to upload images." 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": null, 83 | "metadata": { 84 | "id": "H9w8y_Nvyfnb" 85 | }, 86 | "outputs": [], 87 | "source": [ 88 | "from google.colab import files\n", 89 | "\n", 90 | "def upload_files():\n", 91 | " uploaded = files.upload()\n", 92 | " for k, v in uploaded.items():\n", 93 | " open(k, 'wb').write(v)\n", 94 | " return list(uploaded.keys())" 95 | ] 96 | }, 97 | { 98 | "cell_type": "markdown", 99 | "metadata": { 100 | "id": "hK8ss2yklaY0" 101 | }, 102 | "source": [ 103 | "## Image Generation" 104 | ] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": { 109 | "id": "O_IOv-mtyjIh" 110 | }, 111 | "source": [ 112 | "Import the dependencies to be used." 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": null, 118 | "metadata": { 119 | "id": "nFD0eHSeynU7" 120 | }, 121 | "outputs": [], 122 | "source": [ 123 | "import torchvision.transforms as transforms\n", 124 | "from diffusers import StableDiffusionImageVariationPipeline\n", 125 | "from PIL import Image" 126 | ] 127 | }, 128 | { 129 | "cell_type": "markdown", 130 | "metadata": { 131 | "id": "lYXKh_jAyfBF" 132 | }, 133 | "source": [ 134 | "Upload an image to be used as input." 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": null, 140 | "metadata": { 141 | "id": "73hILxyDyiPM" 142 | }, 143 | "outputs": [], 144 | "source": [ 145 | "uploaded_image_list = upload_files()\n", 146 | "input_im = Image.open(uploaded_image_list[0])\n", 147 | "display(input_im)" 148 | ] 149 | }, 150 | { 151 | "cell_type": "markdown", 152 | "metadata": { 153 | "id": "VJjXQKuNypAC" 154 | }, 155 | "source": [ 156 | "Create the pipeline. The code cell below will take few minutes (depending on the network traffic) as it must download the pretrained SD Image Variations model components." 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": null, 162 | "metadata": { 163 | "id": "EXD2Ne3FyoUc" 164 | }, 165 | "outputs": [], 166 | "source": [ 167 | "device = \"cuda:0\"\n", 168 | "sd_pipe = StableDiffusionImageVariationPipeline.from_pretrained(\n", 169 | " \"lambdalabs/sd-image-variations-diffusers\",\n", 170 | " revision=\"v2.0\",\n", 171 | " )\n", 172 | "sd_pipe = sd_pipe.to(device)" 173 | ] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": { 178 | "id": "xflqm5gGljP6" 179 | }, 180 | "source": [ 181 | "Set some Image Variation options." 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": null, 187 | "metadata": { 188 | "id": "NAqq-5XvlpLx" 189 | }, 190 | "outputs": [], 191 | "source": [ 192 | "guidance_scale = 10 #@param {type:\"slider\", min:1, max:20, step:0.5}\n", 193 | "steps = 50 #@param {type:\"slider\", min:1, max:100, step:1}\n", 194 | "output_filename = \"result.jpg\" #@param {type: \"string\"}" 195 | ] 196 | }, 197 | { 198 | "cell_type": "markdown", 199 | "metadata": { 200 | "id": "GBqwkfDVyv5y" 201 | }, 202 | "source": [ 203 | "Resize, normalize and finally transform the input image into a PyTorch tensor. " 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": null, 209 | "metadata": { 210 | "id": "aMQnP-VDy31i" 211 | }, 212 | "outputs": [], 213 | "source": [ 214 | "im = Image.open(uploaded_image_list[0])\n", 215 | "tform = transforms.Compose([\n", 216 | " transforms.ToTensor(),\n", 217 | " transforms.Resize(\n", 218 | " (224, 224),\n", 219 | " interpolation=transforms.InterpolationMode.BICUBIC,\n", 220 | " antialias=False,\n", 221 | " ),\n", 222 | " transforms.Normalize(\n", 223 | " [0.48145466, 0.4578275, 0.40821073],\n", 224 | " [0.26862954, 0.26130258, 0.27577711]),\n", 225 | "])\n", 226 | "inp = tform(im).to(device).unsqueeze(0)" 227 | ] 228 | }, 229 | { 230 | "cell_type": "markdown", 231 | "metadata": { 232 | "id": "5vELGZacwNRV" 233 | }, 234 | "source": [ 235 | "Start image-to-image generation." 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": null, 241 | "metadata": { 242 | "id": "a4HWdUfDy7tW" 243 | }, 244 | "outputs": [], 245 | "source": [ 246 | "out = sd_pipe(inp, guidance_scale=guidance_scale, num_inference_steps=steps)\n", 247 | "out[\"images\"][0].save(output_filename)" 248 | ] 249 | }, 250 | { 251 | "cell_type": "markdown", 252 | "metadata": { 253 | "id": "BFnBXcfry4mS" 254 | }, 255 | "source": [ 256 | "Display the generated image." 257 | ] 258 | }, 259 | { 260 | "cell_type": "code", 261 | "execution_count": null, 262 | "metadata": { 263 | "id": "zXOCWunty_1k" 264 | }, 265 | "outputs": [], 266 | "source": [ 267 | "output_im = Image.open(output_filename)\n", 268 | "display(output_im)" 269 | ] 270 | } 271 | ], 272 | "metadata": { 273 | "accelerator": "GPU", 274 | "colab": { 275 | "provenance": [] 276 | }, 277 | "gpuClass": "standard", 278 | "kernelspec": { 279 | "display_name": "Python 3", 280 | "name": "python3" 281 | }, 282 | "language_info": { 283 | "name": "python" 284 | } 285 | }, 286 | "nbformat": 4, 287 | "nbformat_minor": 0 288 | } 289 | -------------------------------------------------------------------------------- /Colab/SDImageVariations/images/sample_image.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/SDImageVariations/images/sample_image.jpg -------------------------------------------------------------------------------- /Colab/SPADE-TF/README.md: -------------------------------------------------------------------------------- 1 | # Semantic Image Synthesis with SPADE in TensorFlow 2 | A porting to [TensorFlow](https://www.tensorflow.org/) 1 of the Semantic Image Synthesis from the NVIDIA's SPADE [paper](https://arxiv.org/abs/1903.07291). Credits for the porting to TF go to [taki0112](https://github.com/taki0112). 3 | -------------------------------------------------------------------------------- /Colab/SPADE-TF/SPADE_in_TensorFlow_Inference.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "uPgoceONA6MF" 7 | }, 8 | "source": [ 9 | "# Semantic Image Synthesis with SPADE in TensorFlow\n", 10 | "The code in this notebook is to do Semantic Image Synthesis using a [TensorFlow](https://www.tensorflow.org/) implementation of the NVIDIA's SPADE [paper](https://arxiv.org/abs/1903.07291). Credits for TF porting go to [taki0112](https://github.com/taki0112). \n", 11 | "Tests are performed using a pre-trained model on the [CelebAMask-HQ](https://github.com/switchablenorms/CelebAMask-HQ) dataset. \n", 12 | "Please switch to a GPU runtime before executing the code in this notebook. " 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": { 18 | "id": "-UiI6CHoCVED" 19 | }, 20 | "source": [ 21 | "### Settings" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": { 27 | "id": "qghgwm3zLkC7" 28 | }, 29 | "source": [ 30 | "Clone the GitHub repository from taki0112." 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": { 37 | "id": "N1z1K5yVLfJO" 38 | }, 39 | "outputs": [], 40 | "source": [ 41 | "!git clone https://github.com/taki0112/SPADE-Tensorflow.git\n", 42 | "%cd ./SPADE-Tensorflow" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": { 48 | "id": "Sf-ntRrSQGTG" 49 | }, 50 | "source": [ 51 | "Switch to TensorFlow 1.x, as the implementation of this code is in TensorFlow 1." 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": null, 57 | "metadata": { 58 | "id": "dsI6kfnBQLQb" 59 | }, 60 | "outputs": [], 61 | "source": [ 62 | "%tensorflow_version 1.x" 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": { 68 | "id": "gTHrHPTzky_c" 69 | }, 70 | "source": [ 71 | "Downgrade scipy to release 1.2.0." 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": null, 77 | "metadata": { 78 | "id": "0ie8dq0Mk36T" 79 | }, 80 | "outputs": [], 81 | "source": [ 82 | "!pip install scipy==1.2.0" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": { 88 | "id": "MCuIMJe6MdyN" 89 | }, 90 | "source": [ 91 | "Download the model checkpoint (pre-trained on the CelebAMask-HQ dataset). \n", 92 | "The checkpoint size is 1.76 GB." 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": null, 98 | "metadata": { 99 | "id": "GWJ6KrIPNBf5" 100 | }, 101 | "outputs": [], 102 | "source": [ 103 | "!gdown --id 1UIj7eRJeNWrDS-3odyaoLhcqk0tNcEez\n", 104 | "!unzip ./checkpoint.zip\n", 105 | "!rm -f ./checkpoint.zip" 106 | ] 107 | }, 108 | { 109 | "cell_type": "markdown", 110 | "metadata": { 111 | "id": "7xMQ2didkNGR" 112 | }, 113 | "source": [ 114 | "Create the segmap label for the test dataset as explainded in https://github.com/taki0112/SPADE-Tensorflow/issues/15" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": null, 120 | "metadata": { 121 | "id": "R5RGrO6skMa5" 122 | }, 123 | "outputs": [], 124 | "source": [ 125 | "segmap_label_content = '{(0, 0, 0): 0, (0, 0, 255): 1, (255, 0, 0): 2, (150, 30, 150): 3, (255, 65, 255): 4, (150, 80, 0): 5, (170, 120, 65): 6, (125, 125, 125): 7, (255, 255, 0): 8, (0, 255, 255): 9, (255, 150, 0): 10, (255, 225, 120): 11, (255, 125, 125): 12, (200, 100, 100): 13, (0, 255, 0): 14, (0, 150, 80): 15, (215, 175, 125): 16, (220, 180, 210): 17, (125, 125, 255): 18}'\n", 126 | "with open('./dataset/spade_celebA/segmap_label.txt', 'w') as f:\n", 127 | " f.write(segmap_label_content)" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "metadata": { 133 | "id": "T2Hp9dYSDvt3" 134 | }, 135 | "source": [ 136 | "### Semantic Image Synthesis Tests" 137 | ] 138 | }, 139 | { 140 | "cell_type": "markdown", 141 | "metadata": { 142 | "id": "LwqYleFRNNGx" 143 | }, 144 | "source": [ 145 | "Random test using the pre-trained model." 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": null, 151 | "metadata": { 152 | "id": "wgUwMCSvNR8A" 153 | }, 154 | "outputs": [], 155 | "source": [ 156 | "!python main.py --checkpoint_dir . --dataset spade_celebA --segmap_ch 3 --phase random" 157 | ] 158 | }, 159 | { 160 | "cell_type": "markdown", 161 | "metadata": { 162 | "id": "8zihjuUJNaCa" 163 | }, 164 | "source": [ 165 | "Display the results." 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": null, 171 | "metadata": { 172 | "id": "tNUrjDDMMPxx" 173 | }, 174 | "outputs": [], 175 | "source": [ 176 | "!rm -f ./results/SPADE_spade_celebA_hinge_2multi_4dis_1_1_10_10_0.05_sn_TTUR_more/index.html" 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": null, 182 | "metadata": { 183 | "id": "gQGLdzmnHr_Q" 184 | }, 185 | "outputs": [], 186 | "source": [ 187 | "import cv2\n", 188 | "import os\n", 189 | "import matplotlib.pyplot as plt\n", 190 | "\n", 191 | "image_output_dir = './results/SPADE_spade_celebA_hinge_2multi_4dis_1_1_10_10_0.05_sn_TTUR_more'\n", 192 | "items = os.listdir(image_output_dir)\n", 193 | "items.sort() \n", 194 | "\n", 195 | "generated_image_count = len(items)\n", 196 | "image_index = 0\n", 197 | "rows = 5\n", 198 | "cols = 3\n", 199 | "fig, axes = plt.subplots(nrows=rows, ncols=cols, figsize=(15,15))\n", 200 | "\n", 201 | "for i in range(rows):\n", 202 | " for j in range(cols): \n", 203 | " if image_index < generated_image_count:\n", 204 | " full_path = image_output_dir + '/' + items[image_index]\n", 205 | " image = cv2.imread(full_path)\n", 206 | " image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB) \n", 207 | " axes[i, j].imshow(image)\n", 208 | " axes[i, j].set_title(items[image_index])\n", 209 | " image_index+=1" 210 | ] 211 | } 212 | ], 213 | "metadata": { 214 | "accelerator": "GPU", 215 | "colab": { 216 | "collapsed_sections": [], 217 | "name": "SPADE in TensorFlow - Inference.ipynb", 218 | "provenance": [] 219 | }, 220 | "kernelspec": { 221 | "display_name": "Python 3", 222 | "name": "python3" 223 | }, 224 | "language_info": { 225 | "name": "python" 226 | } 227 | }, 228 | "nbformat": 4, 229 | "nbformat_minor": 0 230 | } 231 | -------------------------------------------------------------------------------- /Colab/SPADE-TF/SPADE_in_TensorFlow_Training.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "aq93kTTSJmtR" 7 | }, 8 | "source": [ 9 | "### Settings" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": { 15 | "id": "aGMy3OGlRmPQ" 16 | }, 17 | "source": [ 18 | "Clone the GitHub repository from taki0112." 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "id": "QAODRNncRpuI" 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "!git clone https://github.com/taki0112/SPADE-Tensorflow.git\n", 30 | "%cd ./SPADE-Tensorflow" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": { 36 | "id": "GuwPmynVRucS" 37 | }, 38 | "source": [ 39 | "Switch to TensorFlow 1.x, as the implementation of this code is in TensorFlow 1." 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": null, 45 | "metadata": { 46 | "id": "zpNhD4LHRvl4" 47 | }, 48 | "outputs": [], 49 | "source": [ 50 | "%tensorflow_version 1.x" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": { 56 | "id": "LeENv3t_JqiG" 57 | }, 58 | "source": [ 59 | "### Training" 60 | ] 61 | }, 62 | { 63 | "cell_type": "markdown", 64 | "metadata": { 65 | "id": "tBVfoO0797FV" 66 | }, 67 | "source": [ 68 | "Start training." 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": { 75 | "id": "vcI76Z909pWJ" 76 | }, 77 | "outputs": [], 78 | "source": [ 79 | "!python ./main.py --dataset spade_celebA --img_ch 3 --segmap_ch 3 --epoch 10 --iteration 100 --phase train" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "metadata": { 85 | "id": "byic54PrEgUL" 86 | }, 87 | "source": [ 88 | "Analyze the training logs in TensorBoard offline." 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": null, 94 | "metadata": { 95 | "id": "rIQ6MzmREn55" 96 | }, 97 | "outputs": [], 98 | "source": [ 99 | "%load_ext tensorboard" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": null, 105 | "metadata": { 106 | "id": "SeQbjFkVEsFs" 107 | }, 108 | "outputs": [], 109 | "source": [ 110 | "%tensorboard --logdir ./logs" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": { 116 | "id": "gashfRdXIsrH" 117 | }, 118 | "source": [ 119 | "### Testing" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": { 125 | "id": "7aIa8XqBIxdg" 126 | }, 127 | "source": [ 128 | "Downgrade scipy to release 1.2.0." 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": null, 134 | "metadata": { 135 | "id": "g2qHUkWIIwds" 136 | }, 137 | "outputs": [], 138 | "source": [ 139 | "!pip install scipy==1.2.0" 140 | ] 141 | }, 142 | { 143 | "cell_type": "markdown", 144 | "metadata": { 145 | "id": "1DxtNzogAdBW" 146 | }, 147 | "source": [ 148 | "Do a random test with the pre-trained model." 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": null, 154 | "metadata": { 155 | "id": "kviecJ22AjRW" 156 | }, 157 | "outputs": [], 158 | "source": [ 159 | "!python main.py --checkpoint_dir ./checkpoint/ --dataset spade_celebA --segmap_ch 3 --phase random" 160 | ] 161 | }, 162 | { 163 | "cell_type": "markdown", 164 | "metadata": { 165 | "id": "39qviJqqDgQa" 166 | }, 167 | "source": [ 168 | "Display the results." 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": null, 174 | "metadata": { 175 | "id": "yp6OMMr2Dj80" 176 | }, 177 | "outputs": [], 178 | "source": [ 179 | "!rm -f ./results/SPADE_spade_celebA_hinge_2multi_4dis_1_1_10_10_0.05_sn_TTUR_more/index.html" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": null, 185 | "metadata": { 186 | "id": "Yz1WYK7sDmrN" 187 | }, 188 | "outputs": [], 189 | "source": [ 190 | "import cv2\n", 191 | "import os\n", 192 | "import matplotlib.pyplot as plt\n", 193 | "\n", 194 | "image_output_dir = './results/SPADE_spade_celebA_hinge_2multi_4dis_1_1_10_10_0.05_sn_TTUR_more'\n", 195 | "items = os.listdir(image_output_dir)\n", 196 | "items.sort() \n", 197 | "\n", 198 | "generated_image_count = len(items)\n", 199 | "image_index = 0\n", 200 | "rows = 5\n", 201 | "cols = 3\n", 202 | "fig, axes = plt.subplots(nrows=rows, ncols=cols, figsize=(15,15))\n", 203 | "\n", 204 | "for i in range(rows):\n", 205 | " for j in range(cols): \n", 206 | " if image_index < generated_image_count:\n", 207 | " full_path = image_output_dir + '/' + items[image_index]\n", 208 | " image = cv2.imread(full_path)\n", 209 | " image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB) \n", 210 | " axes[i, j].imshow(image)\n", 211 | " axes[i, j].set_title(items[image_index])\n", 212 | " image_index+=1" 213 | ] 214 | } 215 | ], 216 | "metadata": { 217 | "accelerator": "GPU", 218 | "colab": { 219 | "collapsed_sections": [], 220 | "name": "SPADE in TensorFlow - Training.ipynb", 221 | "provenance": [] 222 | }, 223 | "kernelspec": { 224 | "display_name": "Python 3", 225 | "name": "python3" 226 | }, 227 | "language_info": { 228 | "name": "python" 229 | } 230 | }, 231 | "nbformat": 4, 232 | "nbformat_minor": 0 233 | } 234 | -------------------------------------------------------------------------------- /Colab/SinGAN/README.md: -------------------------------------------------------------------------------- 1 | # SinGAN 2 | This repository hosts notebooks to train and/or perform other generative tasks with SinGAN, an unconditional generative model that can be learned from a single natural image. You can read about the original SinGAN paper [here](https://arxiv.org/abs/1905.01164). The official source code associated to this paper is implemented in Python 3 and the Deep Learning framework of reference is PyTorch. In this repo you can find Colab notebooks releated to the official [PyTorch](https://pytorch.org/) implementation and a community port to [TensorFlow](https://www.tensorflow.org/) 2. 3 | A GPU runtime is needed to execute the code in any of these notebooks. 4 | ## List of notebooks and tasks in this repo 5 | * Official PyTorch implementation (to be added). 6 | * [cryu854](https://github.com/cryu854)'s TensorFlow 2 port. 7 | * Training 8 | * Random Sampling 9 | * Harmonization -------------------------------------------------------------------------------- /Colab/SinGAN/SinGAN_TensorFlow.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "Us3HvlhMMmlp" 7 | }, 8 | "source": [ 9 | "# SinGAN Port to TensorFlow\n", 10 | "This notebook allows training and some tasks such as random sampling and harmonization using [cryu854](https://github.com/cryu854)'s [TensorFlow](https://www.tensorflow.org/) 2 port of SinGAN, an unconditional generative model that can be learned from a single natural image. You can read about the original SinGAN paper [here](https://arxiv.org/abs/1905.01164). \n", 11 | "A GPU runtime is needed to execute the code in this notebook." 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": { 17 | "id": "L_gbhfZZN0Le" 18 | }, 19 | "source": [ 20 | "## Settings" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": { 26 | "id": "j9KCE3wDsaj9" 27 | }, 28 | "source": [ 29 | "Clone the GitHub repository." 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": { 36 | "id": "Wp_CvaWmsONl" 37 | }, 38 | "outputs": [], 39 | "source": [ 40 | "!git clone https://github.com/cryu854/SinGAN.git\n", 41 | "%cd ./SinGAN" 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": { 47 | "id": "4pYO9gXBN5N4" 48 | }, 49 | "source": [ 50 | "## Training" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": { 56 | "id": "BhZmyZt0ski_" 57 | }, 58 | "source": [ 59 | "Upload your own image for training." 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": { 66 | "id": "MIOeC2JUss5K" 67 | }, 68 | "outputs": [], 69 | "source": [ 70 | "from google.colab import files\n", 71 | "\n", 72 | "uploaded = files.upload()" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": { 79 | "id": "HruIjK2Qs7gJ" 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "import os\n", 84 | "\n", 85 | "for key in uploaded.keys():\n", 86 | " img_filename = key\n", 87 | "\n", 88 | "os.environ['UPLOADED_IMAGE_FILENAME'] = img_filename" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "metadata": { 94 | "id": "ShyDo0JMs-04" 95 | }, 96 | "source": [ 97 | "Train a SinGAN model on the uploaded image." 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "metadata": { 104 | "id": "4wVO-oOBujSF" 105 | }, 106 | "outputs": [], 107 | "source": [ 108 | "!python main.py train --image $UPLOADED_IMAGE_FILENAME --num_scales 8" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": { 114 | "id": "E-drxVMGMOt0" 115 | }, 116 | "source": [ 117 | "Compress the trained model, so that it would be easier to download." 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": { 124 | "id": "RYaX4rMg9pwS" 125 | }, 126 | "outputs": [], 127 | "source": [ 128 | "!tar -zcvf training_checkpoints.tar.gz ./training_checkpoints/" 129 | ] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "metadata": { 134 | "id": "eGePd65ZvHj1" 135 | }, 136 | "source": [ 137 | "## Random Sampling" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": null, 143 | "metadata": { 144 | "id": "iMFcfkIHtEr-" 145 | }, 146 | "outputs": [], 147 | "source": [ 148 | "!python main.py inference --image $UPLOADED_IMAGE_FILENAME --dir ./training_checkpoints --mode random_sample --inject_scale 1" 149 | ] 150 | }, 151 | { 152 | "cell_type": "markdown", 153 | "metadata": { 154 | "id": "00nQSJG1vcxV" 155 | }, 156 | "source": [ 157 | "Display the generated images." 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": null, 163 | "metadata": { 164 | "id": "uSLik9pFMuLX" 165 | }, 166 | "outputs": [], 167 | "source": [ 168 | "import matplotlib.pyplot as plt\n", 169 | "import cv2\n", 170 | "\n", 171 | "input_images_path = './results/random_sample'\n", 172 | "items = os.listdir(input_images_path) \n", 173 | "\n", 174 | "for each_image in items:\n", 175 | " if each_image.endswith(\".jpg\"):\n", 176 | " full_path = input_images_path + '/' + each_image\n", 177 | " image = cv2.imread(full_path)\n", 178 | " image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB)\n", 179 | " plt.figure()\n", 180 | " plt.imshow(image)\n", 181 | " plt.grid(False)\n", 182 | " plt.title(each_image)" 183 | ] 184 | }, 185 | { 186 | "cell_type": "markdown", 187 | "metadata": { 188 | "id": "cMBEaFpsv6wO" 189 | }, 190 | "source": [ 191 | "## Harmonization" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": { 197 | "id": "nmXmWIqOLhQZ" 198 | }, 199 | "source": [ 200 | "Upload an image for harmonization." 201 | ] 202 | }, 203 | { 204 | "cell_type": "code", 205 | "execution_count": null, 206 | "metadata": { 207 | "id": "_lrdmgtYMG9-" 208 | }, 209 | "outputs": [], 210 | "source": [ 211 | "uploaded = files.upload()" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": null, 217 | "metadata": { 218 | "id": "mIZR4Kc-MLdT" 219 | }, 220 | "outputs": [], 221 | "source": [ 222 | "for key in uploaded.keys():\n", 223 | " img_filename = key\n", 224 | "\n", 225 | "os.environ['UPLOADED_HARMONIZATION_IMAGE_FILENAME'] = img_filename" 226 | ] 227 | }, 228 | { 229 | "cell_type": "markdown", 230 | "metadata": { 231 | "id": "qWesBghWL7RY" 232 | }, 233 | "source": [ 234 | "Do harmonization." 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": null, 240 | "metadata": { 241 | "id": "uOLmUBcZwCoI" 242 | }, 243 | "outputs": [], 244 | "source": [ 245 | "!python main.py inference --image $UPLOADED_HARMONIZATION_IMAGE_FILENAME --dir ./training_checkpoints --mode harmonization --inject_scale 5" 246 | ] 247 | }, 248 | { 249 | "cell_type": "markdown", 250 | "metadata": {}, 251 | "source": [ 252 | "The generated image is save into the *./results/harmonization/* directory" 253 | ] 254 | } 255 | ], 256 | "metadata": { 257 | "accelerator": "GPU", 258 | "colab": { 259 | "collapsed_sections": [], 260 | "name": "SinGAN TensorFlow.ipynb", 261 | "provenance": [] 262 | }, 263 | "kernelspec": { 264 | "display_name": "Python 3", 265 | "name": "python3" 266 | }, 267 | "language_info": { 268 | "name": "python" 269 | } 270 | }, 271 | "nbformat": 4, 272 | "nbformat_minor": 0 273 | } 274 | -------------------------------------------------------------------------------- /Colab/StyleCariGAN/Caricatures.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/StyleCariGAN/Caricatures.PNG -------------------------------------------------------------------------------- /Colab/StyleCariGAN/README.md: -------------------------------------------------------------------------------- 1 | ## StyleCariGAN Evaluation 2 | A notebook to evaluate the *StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation* [paper](https://arxiv.org/abs/2107.04331) on your own images. 3 | StyleCariGAN is a caricature generation framework based on shape and style manipulation using StyleGAN. It automatically creates a realistic and detailed caricature from an input photo with optional controls on shape exaggeration degree and color stylization type. 4 | A GPU runtime is needed to run this code successfully. 5 | ![Some generated caricatures.](https://github.com/virtualramblas/python-notebooks-repo/raw/main/Colab/StyleCariGAN/Caricatures.PNG) 6 | -------------------------------------------------------------------------------- /Colab/StyleCariGAN/StyleCariGAN_Evaluation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "StyleCariGAN Evaluation.ipynb", 7 | "provenance": [] 8 | }, 9 | "kernelspec": { 10 | "name": "python3", 11 | "display_name": "Python 3" 12 | }, 13 | "language_info": { 14 | "name": "python" 15 | }, 16 | "accelerator": "GPU" 17 | }, 18 | "cells": [ 19 | { 20 | "cell_type": "markdown", 21 | "metadata": { 22 | "id": "DJFfIdHvSLrT" 23 | }, 24 | "source": [ 25 | "## StyleCariGAN Evaluation\n", 26 | "Notebook to evaluate the StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation [paper](https://arxiv.org/abs/2107.04331) on your own images. \n", 27 | "StyleCariGAN is a caricature generation framework based on shape and style manipulation using StyleGAN. It automatically creates a realistic and detailed caricature from an input photo with optional controls on shape exaggeration degree and color stylization type. \n", 28 | "-----> **A GPU runtime is needed to run this code successfully.** <----- \n", 29 | " \n", 30 | "@article{Jang2021StyleCari, \n", 31 | " author = {Wonjong Jang and Gwangjin Ju and Yucheol Jung and Jiaolong Yang and Xin Tong and Seungyong Lee}, \n", 32 | " title = {StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation}, \n", 33 | " booktitle = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH)}, \n", 34 | " publisher = {ACM}, \n", 35 | " volume = {40}, \n", 36 | " number = {4}, \n", 37 | " year = {2021} \n", 38 | "}" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": { 44 | "id": "pNZN9j_gNqBG" 45 | }, 46 | "source": [ 47 | "### Setup" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": { 53 | "id": "37d6AHu2PBsm" 54 | }, 55 | "source": [ 56 | "Clone the official GitHub repo." 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "metadata": { 62 | "id": "Ji67mImeO6ZY" 63 | }, 64 | "source": [ 65 | "!git clone https://github.com/wonjongg/StyleCariGAN.git" 66 | ], 67 | "execution_count": null, 68 | "outputs": [] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "metadata": { 73 | "id": "PLXxCynaPN_J" 74 | }, 75 | "source": [ 76 | "%cd ./StyleCariGAN" 77 | ], 78 | "execution_count": null, 79 | "outputs": [] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "metadata": { 84 | "id": "MniI1XxcPafz" 85 | }, 86 | "source": [ 87 | "Download pre-trained model weights." 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "metadata": { 93 | "id": "feA9Kpv3Pcyu" 94 | }, 95 | "source": [ 96 | "!bash ./download.sh" 97 | ], 98 | "execution_count": null, 99 | "outputs": [] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "metadata": { 104 | "id": "IR-1sTiZQhsd" 105 | }, 106 | "source": [ 107 | "If the downolad from the Dropbox link above doesn't work, please try from this other location. Otherwise ignore the following cell and move to the Ninja installation." 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "metadata": { 113 | "id": "0lnjzykUQvC6" 114 | }, 115 | "source": [ 116 | "!gdown --id 1hBAWJQjbB-B_EA2xsz1XGUh6gDNb-GKO" 117 | ], 118 | "execution_count": null, 119 | "outputs": [] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": { 124 | "id": "ZPN0BE-xQuJe" 125 | }, 126 | "source": [ 127 | "Install [Ninja](https://ninja-build.org/) in order to load the C++ extensions." 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "metadata": { 133 | "id": "St_S9OwsQ0s1" 134 | }, 135 | "source": [ 136 | "!wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip\n", 137 | "!sudo unzip ninja-linux.zip -d /usr/local/bin/\n", 138 | "!sudo update-alternatives --install /usr/bin/ninja ninja /usr/local/bin/ninja 1 --force " 139 | ], 140 | "execution_count": null, 141 | "outputs": [] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "metadata": { 146 | "id": "mAlNRcVQQ4ec" 147 | }, 148 | "source": [ 149 | "!rm -f ninja-linux.zip" 150 | ], 151 | "execution_count": null, 152 | "outputs": [] 153 | }, 154 | { 155 | "cell_type": "markdown", 156 | "metadata": { 157 | "id": "O_pk0yUBFdKQ" 158 | }, 159 | "source": [ 160 | "Create the input and output directories. Uploaded images will be saved into the input directory. Generated caricatures will be saved into sub-directories of the output dir." 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "metadata": { 166 | "id": "sm73Moc7Fdzs" 167 | }, 168 | "source": [ 169 | "!mkdir ./examples/input\n", 170 | "!mkdir ./examples/output/" 171 | ], 172 | "execution_count": null, 173 | "outputs": [] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": { 178 | "id": "NNor5J41F-8u" 179 | }, 180 | "source": [ 181 | "User defined function to upload images." 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "metadata": { 187 | "id": "YtQjQKwUGFq3" 188 | }, 189 | "source": [ 190 | "def upload_files():\n", 191 | " from google.colab import files\n", 192 | " uploaded = files.upload()\n", 193 | " for k, v in uploaded.items():\n", 194 | " open(k, 'wb').write(v)\n", 195 | " return list(uploaded.keys())" 196 | ], 197 | "execution_count": null, 198 | "outputs": [] 199 | }, 200 | { 201 | "cell_type": "markdown", 202 | "metadata": { 203 | "id": "waVhjCIMNwuz" 204 | }, 205 | "source": [ 206 | "### Evaluation" 207 | ] 208 | }, 209 | { 210 | "cell_type": "markdown", 211 | "metadata": { 212 | "id": "55gHehV5Gf2X" 213 | }, 214 | "source": [ 215 | "Upload your own images." 216 | ] 217 | }, 218 | { 219 | "cell_type": "code", 220 | "metadata": { 221 | "id": "2gmYmxFFGi3W" 222 | }, 223 | "source": [ 224 | "%cd ./examples/input/\n", 225 | "uploaded_image_list = upload_files()\n", 226 | "%cd ../.." 227 | ], 228 | "execution_count": null, 229 | "outputs": [] 230 | }, 231 | { 232 | "cell_type": "markdown", 233 | "metadata": { 234 | "id": "52Vpx4k9Iw4x" 235 | }, 236 | "source": [ 237 | "Display the uploaded images." 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "metadata": { 243 | "id": "hsqmpj5VI0zx" 244 | }, 245 | "source": [ 246 | "import matplotlib.pyplot as plt\n", 247 | "import os\n", 248 | "import cv2\n", 249 | "\n", 250 | "input_images_path = './examples/input'\n", 251 | "items = os.listdir(input_images_path)\n", 252 | "print (items) \n", 253 | "\n", 254 | "for each_image in items:\n", 255 | " if each_image.endswith(\".jpg\") or each_image.endswith(\".png\"):\n", 256 | " print(each_image)\n", 257 | " full_path = input_images_path + '/' + each_image\n", 258 | " image = cv2.imread(full_path)\n", 259 | " image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB)\n", 260 | " plt.figure()\n", 261 | " plt.imshow(image)\n", 262 | " plt.grid(False)" 263 | ], 264 | "execution_count": null, 265 | "outputs": [] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": { 270 | "id": "KAxIIIc9PrM-" 271 | }, 272 | "source": [ 273 | "Test the model." 274 | ] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "metadata": { 279 | "id": "WXg7cGdnPyHR" 280 | }, 281 | "source": [ 282 | "!python test.py --ckpt ./checkpoint/StyleCariGAN/001000.pt --input_dir ./examples/input/ --output_dir ./examples/output/ --invert_images" 283 | ], 284 | "execution_count": null, 285 | "outputs": [] 286 | }, 287 | { 288 | "cell_type": "markdown", 289 | "metadata": { 290 | "id": "Gb_IjAwgIlR9" 291 | }, 292 | "source": [ 293 | "The generated caricatures are available in sub-directories of the */content/StyleCAriGAN/examples/output* directory." 294 | ] 295 | } 296 | ] 297 | } -------------------------------------------------------------------------------- /Colab/TBEFN/README.md: -------------------------------------------------------------------------------- 1 | ## TBEFN: A Two-branch Exposure-fusion Network for Low-light Image Enhancement 2 | Unofficial TensorFlow 2.x implementation of this [paper](https://ieeexplore.ieee.org/document/9261119). The official code is compatible with TensorFlow 1.x only. 3 | **Update October 10th 2021: the TF 2 code has been now merged to the official GitHub repo for this paper.** 4 | This is the proposed architecture: 5 | ![The TBEFN architecture](https://github.com/lukun199/TBEFN/raw/master/demo_img/Structure.png) 6 | No GPU runtime needed to execute the code in this notebook. 7 | Some examples (on the left the original low-light images, on the right the TBEFN outputs): 8 | ![Low-light image enhancements done through TBEFN.](https://github.com/virtualramblas/python-notebooks-repo/raw/main/Colab/TBEFN/TBEFN_demo_image.PNG) 9 | -------------------------------------------------------------------------------- /Colab/TBEFN/TBEFN_demo_image.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/TBEFN/TBEFN_demo_image.PNG -------------------------------------------------------------------------------- /Colab/VersatileDiffusion/README.md: -------------------------------------------------------------------------------- 1 | # Versatile Diffusion 2 | The [Versatile Diffusion](https://arxiv.org/abs/2211.08332) paper expands the existing single-flow diffusion pipeline into a multi-flow network that handles diverse generation tasks in one unified model. Here's a notebook to evaluate some of the VD tasks: text-to-image, image-variation and dual-guided generation. These tasks are executed here through the Hugging Face's [Diffusers](https://github.com/huggingface/diffusers) library. 3 | ![The Versatile Diffusion structure](https://github.com/SHI-Labs/Versatile-Diffusion/raw/master/assets/figures/vd_combined.png) 4 | 5 | ## Examples 6 | ### Text 2 image 7 | Prompt: A portrait of Spider-man, high definition, comics style. Seed = 0 8 | ![Generated](images/text2img_example.png) 9 | ### Image Variation 10 | Source: 11 | ![Source image](images/source_image_1.jpg) 12 | Variations: 13 | ![Generated](images/image_variation_example_1.png) ![Generated](images/image_variation_example_2.png) 14 | ### Dual-guided generation 15 | Source: 16 | ![Source](images/source_image_0.jpg) 17 | Prompt: Spider man. cyberpunk assassin. blade runner 2 0 4 9 concept painting. epic painting by craig mullins and alphonso mucha. artstationhq. painting with vivid color. ( rb 6 s, cyberpunk 2 0 7 7, matrix ) 18 | Generated: 19 | ![Generated](images/dual_guided_example_2.png) 20 | 21 | Source: 22 | ![Source](images/source_image_1.jpg) 23 | Prompt: A portrait of a horse, oil painting. 24 | Generated: 25 | ![Generated](images/dual_guided_example_0.png) ![Generated](images/dual_guided_example_1.png) -------------------------------------------------------------------------------- /Colab/VersatileDiffusion/Versatile_Diffusion.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "pV1bJqWLryNz" 7 | }, 8 | "source": [ 9 | "# Versatile Diffusion\n", 10 | "The [Versatile Diffusion](https://arxiv.org/abs/2211.08332) paper expands the existing single-flow diffusion pipeline into a multi-flow network that handles diverse generation tasks in one unified model. This notebook is to evaluate some of these tasks: text-to-image, image-variation and dual-guided generation. These tasks are executed here through the Hugging Face's [Diffusers](https://github.com/huggingface/diffusers) library. \n", 11 | "![The Versatile Diffusion structure](https://github.com/SHI-Labs/Versatile-Diffusion/raw/master/assets/figures/vd_combined.png) \n", 12 | "An hardware accelerated runtime (GPU) is required to execute the code in this notebook. \n", 13 | "No need to execute the three tasks in this notebook in sequence: once the code cells in the *Settings* section have been successfully executed, you can then jump directly to the section(s) of interest. " 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": { 19 | "id": "Bp4WAONNWW3z" 20 | }, 21 | "source": [ 22 | "## Settings" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": { 28 | "id": "nUbrzoUsr7p8" 29 | }, 30 | "source": [ 31 | "Install any missing requirement in the Colab VM. Only *diffusers* (for PyTorch) and *transformers* need to be installed. Their installation will automaticall install also *huggingface-hub*, *accelerate* and *tokenizers*." 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": null, 37 | "metadata": { 38 | "id": "V8Lh31HPsDPU" 39 | }, 40 | "outputs": [], 41 | "source": [ 42 | "!pip install diffusers[torch]\n", 43 | "!pip install transformers" 44 | ] 45 | }, 46 | { 47 | "cell_type": "markdown", 48 | "metadata": { 49 | "id": "OxcQQPCoXmtW" 50 | }, 51 | "source": [ 52 | "Define a function to upload images to the Colab VM." 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": { 59 | "id": "ptyyoIfsXqoG" 60 | }, 61 | "outputs": [], 62 | "source": [ 63 | "from google.colab import files\n", 64 | "\n", 65 | "def upload_files():\n", 66 | " uploaded = files.upload()\n", 67 | " for k, v in uploaded.items():\n", 68 | " open(k, 'wb').write(v)\n", 69 | " return list(uploaded.keys())" 70 | ] 71 | }, 72 | { 73 | "cell_type": "markdown", 74 | "metadata": { 75 | "id": "2jOZUNKQVltN" 76 | }, 77 | "source": [ 78 | "Import the general dependencies across multiple tasks." 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "metadata": { 85 | "id": "d2HjAcZwVr8b" 86 | }, 87 | "outputs": [], 88 | "source": [ 89 | "import torch\n", 90 | "from PIL import Image" 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": { 96 | "id": "Di-Ib-isX6mj" 97 | }, 98 | "source": [ 99 | "## Text to Image" 100 | ] 101 | }, 102 | { 103 | "cell_type": "markdown", 104 | "metadata": { 105 | "id": "X1moW9pEaA5r" 106 | }, 107 | "source": [ 108 | "Create the Versatile Diffusion pipeline for this task. For all the pipelines in this notebook, the float 16 version of the pre-trained models are used, as their size is half of the the same for the float 32 models." 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": null, 114 | "metadata": { 115 | "id": "TnXhXGdsaBcX" 116 | }, 117 | "outputs": [], 118 | "source": [ 119 | "from diffusers import VersatileDiffusionTextToImagePipeline\n", 120 | "\n", 121 | "pipe = VersatileDiffusionTextToImagePipeline.from_pretrained(\"shi-labs/versatile-diffusion\", torch_dtype=torch.float16)\n", 122 | "pipe.remove_unused_weights()\n", 123 | "pipe = pipe.to(\"cuda\")" 124 | ] 125 | }, 126 | { 127 | "cell_type": "markdown", 128 | "metadata": { 129 | "id": "h6Mhv07nzfDz" 130 | }, 131 | "source": [ 132 | "Setup prompt, seed and strength to perform a text to image task." 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": null, 138 | "metadata": { 139 | "id": "ECA_ll0BYFpe" 140 | }, 141 | "outputs": [], 142 | "source": [ 143 | "text2img_prompt = \"Sticker of a cute spider, white border, die cut, head, cute, trending on artstation\" #@param {type: \"string\"}\n", 144 | "text2img_seed = 0 #@param {type: \"number\"}\n", 145 | "text2img_strength = 0.75 #@param {type:\"slider\", min:0, max:1, step:0.05}" 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "metadata": { 151 | "id": "Scb2CvHPaSvd" 152 | }, 153 | "source": [ 154 | "Do text to image." 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": null, 160 | "metadata": { 161 | "id": "F2-2_XqJzhYA" 162 | }, 163 | "outputs": [], 164 | "source": [ 165 | "generator = torch.Generator(device=\"cuda\").manual_seed(text2img_seed)\n", 166 | "image = pipe(text2img_prompt, \n", 167 | " generator=generator,\n", 168 | " strength=text2img_strength).images[0]\n", 169 | "display(image)" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": { 175 | "id": "XSd65C8e0uSl" 176 | }, 177 | "source": [ 178 | "## Image Variation" 179 | ] 180 | }, 181 | { 182 | "cell_type": "markdown", 183 | "metadata": { 184 | "id": "8MfqXZbUaROh" 185 | }, 186 | "source": [ 187 | "Create the Versatile Diffusion pipeline for this task." 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": null, 193 | "metadata": { 194 | "id": "xm9NqQTVaScN" 195 | }, 196 | "outputs": [], 197 | "source": [ 198 | "from diffusers import VersatileDiffusionImageVariationPipeline\n", 199 | "\n", 200 | "pipe = VersatileDiffusionImageVariationPipeline.from_pretrained(\"shi-labs/versatile-diffusion\", torch_dtype=torch.float16)\n", 201 | "pipe = pipe.to(\"cuda\")" 202 | ] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "metadata": { 207 | "id": "2UHsBnEnayUr" 208 | }, 209 | "source": [ 210 | "Upload source image(s)." 211 | ] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "execution_count": null, 216 | "metadata": { 217 | "id": "N8ejo_9Ya43E" 218 | }, 219 | "outputs": [], 220 | "source": [ 221 | "uploaded_image_variation_list = upload_files()\n", 222 | "uploaded_image_variation_list" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "metadata": { 228 | "id": "x_XePNHMaoeF" 229 | }, 230 | "source": [ 231 | "Select a source image and set seed and strength. Because the Colab forms dropdown doesn't support variables, the only way to specify the source image filename is to copy and paste it from the previous code cell output." 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": null, 237 | "metadata": { 238 | "id": "B5G1VnUfaEFC" 239 | }, 240 | "outputs": [], 241 | "source": [ 242 | "image_variation_source = \"pexels-pixabay-48785.jpg\" #@param {type: \"string\"}\n", 243 | "image_variation_seed = 222222 #@param {type: \"number\"}\n", 244 | "image_variation_strength = 0.75 #@param {type:\"slider\", min:0, max:1, step:0.05}" 245 | ] 246 | }, 247 | { 248 | "cell_type": "markdown", 249 | "metadata": { 250 | "id": "r9DvN4iApohl" 251 | }, 252 | "source": [ 253 | "Do image variation." 254 | ] 255 | }, 256 | { 257 | "cell_type": "code", 258 | "execution_count": null, 259 | "metadata": { 260 | "id": "tMESviAh0xCn" 261 | }, 262 | "outputs": [], 263 | "source": [ 264 | "image = Image.open(image_variation_source)\n", 265 | "display(image)\n", 266 | "\n", 267 | "generator = torch.Generator(device=\"cuda\").manual_seed(image_variation_seed)\n", 268 | "image = pipe(image, \n", 269 | " generator=generator,\n", 270 | " strength=image_variation_strength).images[0]\n", 271 | "display(image)" 272 | ] 273 | }, 274 | { 275 | "cell_type": "markdown", 276 | "metadata": { 277 | "id": "4m2TMJras8rT" 278 | }, 279 | "source": [ 280 | "## Dual-guided Generation." 281 | ] 282 | }, 283 | { 284 | "cell_type": "markdown", 285 | "metadata": { 286 | "id": "_9i-W9T6yLSD" 287 | }, 288 | "source": [ 289 | "Create the Versatile Diffusion pipeline for this task." 290 | ] 291 | }, 292 | { 293 | "cell_type": "code", 294 | "execution_count": null, 295 | "metadata": { 296 | "id": "HSJ4MrcU1D3R" 297 | }, 298 | "outputs": [], 299 | "source": [ 300 | "from diffusers import VersatileDiffusionDualGuidedPipeline\n", 301 | "\n", 302 | "pipe = VersatileDiffusionDualGuidedPipeline.from_pretrained(\"shi-labs/versatile-diffusion\", torch_dtype=torch.float16)\n", 303 | "pipe.remove_unused_weights()\n", 304 | "pipe = pipe.to(\"cuda\")" 305 | ] 306 | }, 307 | { 308 | "cell_type": "markdown", 309 | "metadata": { 310 | "id": "qK7fkai_hjaG" 311 | }, 312 | "source": [ 313 | "Upload some image(s)." 314 | ] 315 | }, 316 | { 317 | "cell_type": "code", 318 | "execution_count": null, 319 | "metadata": { 320 | "id": "A2ImYIBYvQHy" 321 | }, 322 | "outputs": [], 323 | "source": [ 324 | "uploaded_dual_guided_list = upload_files()\n", 325 | "uploaded_dual_guided_list" 326 | ] 327 | }, 328 | { 329 | "cell_type": "markdown", 330 | "metadata": { 331 | "id": "IX1cxxLKgJbX" 332 | }, 333 | "source": [ 334 | "Select the source image and set the prompt, the seed and the strength value. Again, the only way to specify the source image filename is to copy and paste it from the previous code cell output." 335 | ] 336 | }, 337 | { 338 | "cell_type": "code", 339 | "execution_count": null, 340 | "metadata": { 341 | "id": "LdMafLA4so1f" 342 | }, 343 | "outputs": [], 344 | "source": [ 345 | "dual_guided_source = \"marvel-spiderman-i15585.jpg\" #@param {type: \"string\"}\n", 346 | "dual_guided_prompt = \"Spider man. blade runner 2049 concept painting. painting with vivid color.\" #@param {type: \"string\"}\n", 347 | "dual_guided_seed = 555557 #@param {type: \"number\"}\n", 348 | "dual_guided_strength = 0.5 #@param {type:\"slider\", min:0, max:1, step:0.05}" 349 | ] 350 | }, 351 | { 352 | "cell_type": "markdown", 353 | "metadata": { 354 | "id": "91tkpCmX1KEK" 355 | }, 356 | "source": [ 357 | "Do dual-guided generation." 358 | ] 359 | }, 360 | { 361 | "cell_type": "code", 362 | "execution_count": null, 363 | "metadata": { 364 | "id": "Q1nIHppbx_yg" 365 | }, 366 | "outputs": [], 367 | "source": [ 368 | "image = Image.open(dual_guided_source)\n", 369 | "display(image)\n", 370 | "\n", 371 | "generator = torch.Generator(device=\"cuda\").manual_seed(dual_guided_seed)\n", 372 | "\n", 373 | "image = pipe(prompt=dual_guided_prompt, \n", 374 | " image=image, \n", 375 | " text_to_image_strength=dual_guided_strength, \n", 376 | " generator=generator).images[0]\n", 377 | "display(image)" 378 | ] 379 | } 380 | ], 381 | "metadata": { 382 | "accelerator": "GPU", 383 | "colab": { 384 | "provenance": [] 385 | }, 386 | "gpuClass": "standard", 387 | "kernelspec": { 388 | "display_name": "Python 3", 389 | "name": "python3" 390 | }, 391 | "language_info": { 392 | "name": "python" 393 | } 394 | }, 395 | "nbformat": 4, 396 | "nbformat_minor": 0 397 | } 398 | -------------------------------------------------------------------------------- /Colab/VersatileDiffusion/images/dual_guided_example_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/VersatileDiffusion/images/dual_guided_example_0.png -------------------------------------------------------------------------------- /Colab/VersatileDiffusion/images/dual_guided_example_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/VersatileDiffusion/images/dual_guided_example_1.png -------------------------------------------------------------------------------- /Colab/VersatileDiffusion/images/dual_guided_example_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/VersatileDiffusion/images/dual_guided_example_2.png -------------------------------------------------------------------------------- /Colab/VersatileDiffusion/images/image_variation_example_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/VersatileDiffusion/images/image_variation_example_1.png -------------------------------------------------------------------------------- /Colab/VersatileDiffusion/images/image_variation_example_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/VersatileDiffusion/images/image_variation_example_2.png -------------------------------------------------------------------------------- /Colab/VersatileDiffusion/images/source_image_0.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/VersatileDiffusion/images/source_image_0.jpg -------------------------------------------------------------------------------- /Colab/VersatileDiffusion/images/source_image_1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/VersatileDiffusion/images/source_image_1.jpg -------------------------------------------------------------------------------- /Colab/VersatileDiffusion/images/text2img_example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/VersatileDiffusion/images/text2img_example.png -------------------------------------------------------------------------------- /Colab/ViT/README.md: -------------------------------------------------------------------------------- 1 | # ViT (Vision Transformer) PyTorch 2 | A notebook to train a Visual Transformer for binary image classification in Google Colab (Python 3 and PyTorch). Started from a Python package and a notebook in a lucidrains [repo](https://github.com/lucidrains/vit-pytorch). 3 | Please refer to [this paper](https://openreview.net/pdf?id=YicbFdNTTy) and [this other one](https://arxiv.org/abs/2006.04768) for implementation details. 4 | A GPU runtime is needed to train the model in this notebook. -------------------------------------------------------------------------------- /Colab/Zero-shot-image/Few-shot-image_demo.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/virtualramblas/python-notebooks-repo/69a922c51e0e842aad4e9a0ddea1ddebab393a64/Colab/Zero-shot-image/Few-shot-image_demo.PNG -------------------------------------------------------------------------------- /Colab/Zero-shot-image/README.md: -------------------------------------------------------------------------------- 1 | # Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic 2 | A notebook to explore the capabilities of the [Pytorch](https://pytorch.org/) implementation of the zero-Shot Image-to-Text generation for visual-semantic arithmetic [paper](https://arxiv.org/abs/2111.14447). 3 | 4 | ![Architecture](https://github.com/YoadTew/zero-shot-image-to-text/raw/main/git_images/Architecture.jpg) 5 | 6 | ![Real-world Knowledge Demo](https://github.com/virtualramblas/python-notebooks-repo/raw/main/Colab/Zero-shot-image/Few-shot-image_demo.PNG) -------------------------------------------------------------------------------- /Jupyter/CLIP/CLIP - Semantic Search on Unseen Images - Local.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "Import section." 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": null, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "import gzip\n", 17 | "import html\n", 18 | "import os\n", 19 | "from functools import lru_cache\n", 20 | "from io import BytesIO\n", 21 | "\n", 22 | "import ftfy\n", 23 | "import ipyplot\n", 24 | "import numpy as np\n", 25 | "import regex as re\n", 26 | "import requests\n", 27 | "import PIL\n", 28 | "import torch\n", 29 | "from torchvision.transforms import Compose, Resize, CenterCrop, ToTensor, Normalize\n", 30 | "from PIL import Image" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": {}, 36 | "source": [ 37 | "Load the CLIP model previously downloaded from [OpenAI](https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt). Replace the placeholder with the download location of the pre-trained model on your local machine." 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": null, 43 | "metadata": {}, 44 | "outputs": [], 45 | "source": [ 46 | "model = torch.jit.load(\"\").cuda().eval()" 47 | ] 48 | }, 49 | { 50 | "cell_type": "markdown", 51 | "metadata": {}, 52 | "source": [ 53 | "Details about the model:" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "metadata": {}, 60 | "outputs": [], 61 | "source": [ 62 | "input_resolution = model.input_resolution.item()\n", 63 | "context_length = model.context_length.item()\n", 64 | "vocab_size = model.vocab_size.item()\n", 65 | "\n", 66 | "print(\"Model parameters:\", f\"{np.sum([int(np.prod(p.shape)) for p in model.parameters()]):,}\")\n", 67 | "print(\"Input resolution:\", input_resolution)\n", 68 | "print(\"Context length:\", context_length)\n", 69 | "print(\"Vocab size:\", vocab_size)" 70 | ] 71 | }, 72 | { 73 | "cell_type": "markdown", 74 | "metadata": {}, 75 | "source": [ 76 | "Normalize the pixel intensity using the data set mean and standard deviation." 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "metadata": {}, 83 | "outputs": [], 84 | "source": [ 85 | "image_mean = torch.tensor([0.48145466, 0.4578275, 0.40821073]).cuda()\n", 86 | "image_std = torch.tensor([0.26862954, 0.26130258, 0.27577711]).cuda()" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "Define the image preprocessing: center-crop the input images so that their resolution is conformed to what the model expects." 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": {}, 100 | "outputs": [], 101 | "source": [ 102 | "preprocess = Compose([\n", 103 | " Resize(input_resolution, interpolation=Image.BICUBIC),\n", 104 | " CenterCrop(input_resolution),\n", 105 | " ToTensor()\n", 106 | "])" 107 | ] 108 | }, 109 | { 110 | "cell_type": "markdown", 111 | "metadata": {}, 112 | "source": [ 113 | "Also, text need to be preprocessed. Let's use a case-insensitive tokenizer (to be preliminarily downloaded from [OpenAI](https://openaipublic.azureedge.net/clip/bpe_simple_vocab_16e6.txt.gz)). Replace the placeholder in the *__init__* function of the class *SimpleTokenizer* below with the download location of the zipped pre-trained model on your local machine." 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": null, 119 | "metadata": {}, 120 | "outputs": [], 121 | "source": [ 122 | "@lru_cache()\n", 123 | "def bytes_to_unicode():\n", 124 | " \"\"\"\n", 125 | " Returns list of utf-8 byte and a corresponding list of unicode strings.\n", 126 | " The reversible bpe codes work on unicode strings.\n", 127 | " This means you need a large # of unicode characters in your vocab if you want to avoid UNKs.\n", 128 | " When you're at something like a 10B token dataset you end up needing around 5K for decent coverage.\n", 129 | " This is a signficant percentage of your normal, say, 32K bpe vocab.\n", 130 | " To avoid that, we want lookup tables between utf-8 bytes and unicode strings.\n", 131 | " And avoids mapping to whitespace/control characters the bpe code barfs on.\n", 132 | " \"\"\"\n", 133 | " bs = list(range(ord(\"!\"), ord(\"~\")+1))+list(range(ord(\"¡\"), ord(\"¬\")+1))+list(range(ord(\"®\"), ord(\"ÿ\")+1))\n", 134 | " cs = bs[:]\n", 135 | " n = 0\n", 136 | " for b in range(2**8):\n", 137 | " if b not in bs:\n", 138 | " bs.append(b)\n", 139 | " cs.append(2**8+n)\n", 140 | " n += 1\n", 141 | " cs = [chr(n) for n in cs]\n", 142 | " return dict(zip(bs, cs))\n", 143 | "\n", 144 | "\n", 145 | "def get_pairs(word):\n", 146 | " \"\"\"Return set of symbol pairs in a word.\n", 147 | " Word is represented as tuple of symbols (symbols being variable-length strings).\n", 148 | " \"\"\"\n", 149 | " pairs = set()\n", 150 | " prev_char = word[0]\n", 151 | " for char in word[1:]:\n", 152 | " pairs.add((prev_char, char))\n", 153 | " prev_char = char\n", 154 | " return pairs\n", 155 | "\n", 156 | "\n", 157 | "def basic_clean(text):\n", 158 | " text = ftfy.fix_text(text)\n", 159 | " text = html.unescape(html.unescape(text))\n", 160 | " return text.strip()\n", 161 | "\n", 162 | "\n", 163 | "def whitespace_clean(text):\n", 164 | " text = re.sub(r'\\s+', ' ', text)\n", 165 | " text = text.strip()\n", 166 | " return text\n", 167 | "\n", 168 | "\n", 169 | "class SimpleTokenizer(object):\n", 170 | " def __init__(self, bpe_path: str = \"\"):\n", 171 | " self.byte_encoder = bytes_to_unicode()\n", 172 | " self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}\n", 173 | " merges = gzip.open(bpe_path).read().decode(\"utf-8\").split('\\n')\n", 174 | " merges = merges[1:49152-256-2+1]\n", 175 | " merges = [tuple(merge.split()) for merge in merges]\n", 176 | " vocab = list(bytes_to_unicode().values())\n", 177 | " vocab = vocab + [v+'' for v in vocab]\n", 178 | " for merge in merges:\n", 179 | " vocab.append(''.join(merge))\n", 180 | " vocab.extend(['<|startoftext|>', '<|endoftext|>'])\n", 181 | " self.encoder = dict(zip(vocab, range(len(vocab))))\n", 182 | " self.decoder = {v: k for k, v in self.encoder.items()}\n", 183 | " self.bpe_ranks = dict(zip(merges, range(len(merges))))\n", 184 | " self.cache = {'<|startoftext|>': '<|startoftext|>', '<|endoftext|>': '<|endoftext|>'}\n", 185 | " self.pat = re.compile(r\"\"\"<\\|startoftext\\|>|<\\|endoftext\\|>|'s|'t|'re|'ve|'m|'ll|'d|[\\p{L}]+|[\\p{N}]|[^\\s\\p{L}\\p{N}]+\"\"\", re.IGNORECASE)\n", 186 | "\n", 187 | " def bpe(self, token):\n", 188 | " if token in self.cache:\n", 189 | " return self.cache[token]\n", 190 | " word = tuple(token[:-1]) + ( token[-1] + '',)\n", 191 | " pairs = get_pairs(word)\n", 192 | "\n", 193 | " if not pairs:\n", 194 | " return token+''\n", 195 | "\n", 196 | " while True:\n", 197 | " bigram = min(pairs, key = lambda pair: self.bpe_ranks.get(pair, float('inf')))\n", 198 | " if bigram not in self.bpe_ranks:\n", 199 | " break\n", 200 | " first, second = bigram\n", 201 | " new_word = []\n", 202 | " i = 0\n", 203 | " while i < len(word):\n", 204 | " try:\n", 205 | " j = word.index(first, i)\n", 206 | " new_word.extend(word[i:j])\n", 207 | " i = j\n", 208 | " except:\n", 209 | " new_word.extend(word[i:])\n", 210 | " break\n", 211 | "\n", 212 | " if word[i] == first and i < len(word)-1 and word[i+1] == second:\n", 213 | " new_word.append(first+second)\n", 214 | " i += 2\n", 215 | " else:\n", 216 | " new_word.append(word[i])\n", 217 | " i += 1\n", 218 | " new_word = tuple(new_word)\n", 219 | " word = new_word\n", 220 | " if len(word) == 1:\n", 221 | " break\n", 222 | " else:\n", 223 | " pairs = get_pairs(word)\n", 224 | " word = ' '.join(word)\n", 225 | " self.cache[token] = word\n", 226 | " return word\n", 227 | "\n", 228 | " def encode(self, text):\n", 229 | " bpe_tokens = []\n", 230 | " text = whitespace_clean(basic_clean(text)).lower()\n", 231 | " for token in re.findall(self.pat, text):\n", 232 | " token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))\n", 233 | " bpe_tokens.extend(self.encoder[bpe_token] for bpe_token in self.bpe(token).split(' '))\n", 234 | " return bpe_tokens\n", 235 | "\n", 236 | " def decode(self, tokens):\n", 237 | " text = ''.join([self.decoder[token] for token in tokens])\n", 238 | " text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', errors=\"replace\").replace('', ' ')\n", 239 | " return text\n" 240 | ] 241 | }, 242 | { 243 | "cell_type": "markdown", 244 | "metadata": {}, 245 | "source": [ 246 | "Retrieve some local images. Set the *path* variable with a local directory path." 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": null, 252 | "metadata": {}, 253 | "outputs": [], 254 | "source": [ 255 | "path = ''\n", 256 | "\n", 257 | "files = os.listdir(path)\n", 258 | "\n", 259 | "all_images =[]\n", 260 | "for f in files:\n", 261 | " image = PIL.Image.open(path + '\\\\' + f).convert(\"RGB\")\n", 262 | " all_images.append(image)\n", 263 | "\n", 264 | "print(\"Total images retrived: \", len(all_images))" 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": {}, 270 | "source": [ 271 | "Plot the retrived images within the notebook using [ipyplot](https://github.com/karolzak/ipyplot)." 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": null, 277 | "metadata": {}, 278 | "outputs": [], 279 | "source": [ 280 | "max_images = 105\n", 281 | "img_width = 130\n", 282 | "ipyplot.plot_images(all_images, max_images=max_images, img_width=img_width)" 283 | ] 284 | }, 285 | { 286 | "cell_type": "markdown", 287 | "metadata": {}, 288 | "source": [ 289 | "Preprocess the retrived images." 290 | ] 291 | }, 292 | { 293 | "cell_type": "code", 294 | "execution_count": null, 295 | "metadata": {}, 296 | "outputs": [], 297 | "source": [ 298 | "images = [preprocess(single_image) for single_image in all_images]\n", 299 | "\n", 300 | "image_input = torch.tensor(np.stack(images)).cuda()\n", 301 | "image_input -= image_mean[:, None, None]\n", 302 | "image_input /= image_std[:, None, None]\n", 303 | "with torch.no_grad():\n", 304 | " image_features = model.encode_image(image_input).float()\n", 305 | "image_features /= image_features.norm(dim=-1, keepdim=True)" 306 | ] 307 | }, 308 | { 309 | "cell_type": "markdown", 310 | "metadata": {}, 311 | "source": [ 312 | "Create the tokenizer and then define function two functions for semantic search, one to extract the text features from the search sentence and the second one to get top N images having semantic similarity." 313 | ] 314 | }, 315 | { 316 | "cell_type": "code", 317 | "execution_count": null, 318 | "metadata": {}, 319 | "outputs": [], 320 | "source": [ 321 | "tokenizer = SimpleTokenizer()\n", 322 | "\n", 323 | "'''\n", 324 | "Given a sentence, this fuctions returns its text features. \n", 325 | "'''\n", 326 | "def get_text_features_from_search_phrase(phrase):\n", 327 | " sentence_tokens = [tokenizer.encode(\"%s \"%(phrase) + \"<|endoftext|>\")]\n", 328 | " text_input = torch.zeros(len(sentence_tokens), model.context_length, dtype=torch.long)\n", 329 | " for idx, tokens in enumerate(sentence_tokens):\n", 330 | " text_input[idx, :len(tokens)] = torch.tensor(tokens)\n", 331 | " \n", 332 | " text_input = text_input.cuda()\n", 333 | " with torch.no_grad():\n", 334 | " features = model.encode_text(text_input).float()\n", 335 | " features /= features.norm(dim=-1, keepdim=True)\n", 336 | "\n", 337 | " return features\n", 338 | "\n", 339 | "'''\n", 340 | "Given a list of images and their similarity list, this function returns \n", 341 | "the similarity scores for the top images (along with the images themselves).\n", 342 | "'''\n", 343 | "def get_top_semantic_similarity(similarity_list, image_list, max_number_of_results):\n", 344 | " results = zip(range(len(similarity_list)), similarity_list)\n", 345 | " results = sorted(results, key=lambda x: x[1],reverse= True)\n", 346 | " top_images = []\n", 347 | " scores=[]\n", 348 | " for index,score in results[:max_number_of_results]:\n", 349 | " scores.append(score)\n", 350 | " top_images.append(image_list[index])\n", 351 | " return scores, top_images" 352 | ] 353 | }, 354 | { 355 | "cell_type": "markdown", 356 | "metadata": {}, 357 | "source": [ 358 | "Set a search phrase." 359 | ] 360 | }, 361 | { 362 | "cell_type": "code", 363 | "execution_count": null, 364 | "metadata": {}, 365 | "outputs": [], 366 | "source": [ 367 | "search_phrase = \"set a search sentence here\"" 368 | ] 369 | }, 370 | { 371 | "cell_type": "markdown", 372 | "metadata": {}, 373 | "source": [ 374 | "Extract the text features from the search phrase." 375 | ] 376 | }, 377 | { 378 | "cell_type": "code", 379 | "execution_count": null, 380 | "metadata": {}, 381 | "outputs": [], 382 | "source": [ 383 | "text_features = get_text_features_from_search_phrase(search_phrase)" 384 | ] 385 | }, 386 | { 387 | "cell_type": "markdown", 388 | "metadata": {}, 389 | "source": [ 390 | "Calculate similarities." 391 | ] 392 | }, 393 | { 394 | "cell_type": "code", 395 | "execution_count": null, 396 | "metadata": {}, 397 | "outputs": [], 398 | "source": [ 399 | "similarity = text_features.cpu().numpy() @ image_features.cpu().numpy().T" 400 | ] 401 | }, 402 | { 403 | "cell_type": "markdown", 404 | "metadata": {}, 405 | "source": [ 406 | "Get the top images having semantic similarity." 407 | ] 408 | }, 409 | { 410 | "cell_type": "code", 411 | "execution_count": null, 412 | "metadata": {}, 413 | "outputs": [], 414 | "source": [ 415 | "similarity = similarity[0]\n", 416 | "max_number_of_results = 3\n", 417 | "scores, imgs = get_top_semantic_similarity(similarity, all_images, max_number_of_results)" 418 | ] 419 | }, 420 | { 421 | "cell_type": "markdown", 422 | "metadata": {}, 423 | "source": [ 424 | "Print the scores and plot the top images having semantic similarity." 425 | ] 426 | }, 427 | { 428 | "cell_type": "code", 429 | "execution_count": null, 430 | "metadata": {}, 431 | "outputs": [], 432 | "source": [ 433 | "print (\"Similarity scores:\", scores)\n", 434 | "ipyplot.plot_images(imgs, img_width=300)" 435 | ] 436 | }, 437 | { 438 | "cell_type": "code", 439 | "execution_count": null, 440 | "metadata": {}, 441 | "outputs": [], 442 | "source": [] 443 | } 444 | ], 445 | "metadata": { 446 | "kernelspec": { 447 | "display_name": "Python 3", 448 | "language": "python", 449 | "name": "python3" 450 | }, 451 | "language_info": { 452 | "codemirror_mode": { 453 | "name": "ipython", 454 | "version": 3 455 | }, 456 | "file_extension": ".py", 457 | "mimetype": "text/x-python", 458 | "name": "python", 459 | "nbconvert_exporter": "python", 460 | "pygments_lexer": "ipython3", 461 | "version": "3.7.9" 462 | } 463 | }, 464 | "nbformat": 4, 465 | "nbformat_minor": 4 466 | } 467 | -------------------------------------------------------------------------------- /Jupyter/CLIP/README.md: -------------------------------------------------------------------------------- 1 | # Semantic search on unseen images with OpenAI's CLIP 2 | A Jupyter notebook to do semantic search on unseen and unlabelled images using [OpenAI's CLIP](https://openai.com/blog/clip/). 3 | This notebook has been created after reading the related [paper](https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language_Supervision.pdf) and it has been partially derived from the following [Colab example](https://github.com/openai/CLIP/blob/main/notebooks/Interacting_with_CLIP.ipynb), but to be used on a local Jupyter Notebook server. 4 | 5 | #### Setup 6 | - Create a virtual environment for Python 3. 7 | - Install Jupyter for it. 8 | - Import the dependencies listed in the associated requirements file. 9 | - Download the pre-trained CLIP model from [here](https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt). 10 | - Download an OpenAI case insensitive tokenizer from [here](https://openaipublic.azureedge.net/clip/bpe_simple_vocab_16e6.txt.gz) (don't extract the archive content). 11 | - Run Jupyter. 12 | - Import the notebook. 13 | 14 | It has been tested on Windows machines with an NVIDIA GPU. GPU is **required** in order to execute the code in this notebook. 15 | 16 | #### Usage 17 | Follow the description for each cell of the notebook. 18 | 19 | -------------------------------------------------------------------------------- /Jupyter/CLIP/requirements.txt: -------------------------------------------------------------------------------- 1 | ftfy 2 | ipyplot 3 | numpy 4 | pillow 5 | regex 6 | requests 7 | torch 8 | torchvision 9 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # python-notebooks-repo 2 | A repo to share Jupyter and Colab notebooks. 3 | 4 | List of notebooks: 5 | Jupyter: 6 | - Semantic search on unseen images with OpenAI's CLIP 7 | 8 | Colab: 9 | - Training of a Visual Transformer for image classification 10 | - MiniCLIP: demo to explore text descriptions and saliency maps for Clip models 11 | - DieSitCom: a minimal Streamlit-based DICOM data viewer 12 | - Evaluation of StyleCariGAN to generate caricatures from input photos 13 | - Unofficial TensorFlow 2.x implementation of low-light image enhancement through TBEFN 14 | - Pix2Pix data preparation and training for Satellite-to-Map Image Translation in Keras (TF 2) 15 | - DriveGAN simulator playground 16 | - Monocular Depth Estimation using the Toyota's PackNet-SfM 17 | - Monocular Depth Estimation and 3D scene reconstruction using the AdelaiDepth toolbox. 18 | - Few-shot Adaptation GAN in TensorFlow 19 | - Zero-Shot image-to-text generation for visual-semantic arithmetic 20 | - Image inpainting using GLIDE 21 | - Semantic Image synthesis with SPADE in TensorFlow 22 | - SinGAN: Learning a generative model from a single natural image (PyTorch and TensorFlow 2) 23 | - Image Style Transfer with Deep Feature Rotation (DFR) 24 | - 2x slow-motion video filtering using FLAVR 25 | - Image reshuffling, retargeting and style transfer with Efficient GPNN 26 | - Mobile Face Swap (images and videos) on CPU 27 | - DRPreter (Drug Response PREdictor and interpreTER) model training and test 28 | - Versatile Diffusion text to image, image variation and dual-guided generation 29 | - Object detection with Vision Transformer for Open-World Localization (OWL-ViT) on CPU 30 | - Stable Diffusion Image Variations 31 | - Microsoft's BioGPT text generation on CPU 32 | - C-ChatGPT compilation and inference in Colab 33 | - Code generation through ReplitLM --------------------------------------------------------------------------------