├── README.md ├── SECURITY.md ├── bernie.jpg ├── circle.png ├── circle_animation.gif ├── differentiable_rendering.m ├── differentiable_rendering_example.mlx ├── example_animation.gif └── license.txt /README.md: -------------------------------------------------------------------------------- 1 | # 2D Differentiable Rendering Example 2 | 3 | [![Open in MATLAB Online](https://www.mathworks.com/images/responsive/global/open-in-matlab-online.svg)](https://matlab.mathworks.com/open/github/v1?repo=matlab-deep-learning/2D-Differentiable-Rendering-Example) 4 | 5 | *Keywords: differentiable rendering, autodifferentiation, differentiable rasterization, inverse rendering* 6 | 7 | This repository provides an example of performing differentiable vector graphics rendering using MATLAB. 8 | 9 | The example demonstrates how to use the differentiable rendering pipeline to approximate a target image with multiple vector shapes. By differentiating an image-based loss function with respect to the shapes' parameters—such as size, location, and color—you can iteratively adjust the shapes to closely match the target image. 10 | 11 | ![animation](example_animation.gif) 12 | 13 | ## What is Differentiable Rendering? 14 | Differentiable rendering is a process that allows the computation of gradients through the rasterization pipeline. In traditional rasterization, vector graphics are converted into a raster image (a grid of pixels), but this process is typically non-differentiable due to the discontinuities in the rasterization process. Differentiable rasterization, on the other hand, introduces a smooth approximation to this process, enabling the computation of gradients with respect to the input parameters of the vector shapes. 15 | 16 | Differentiable rasterization opens up new possibilities in various fields, such as computer graphics, machine learning, and computer vision. Some of the key benefits include: 17 | 18 | - Optimization: By enabling gradient-based optimization techniques, differentiable rasterization allows for fine-tuning vector graphics to achieve specific visual goals. 19 | - Inverse Graphics: It facilitates the reconstruction of vector graphics from raster images, which is useful in applications like image vectorization and graphic design. 20 | - Learning-Based Methods: Differentiable rasterization can be integrated into neural networks, enabling end-to-end training for tasks that involve rendering, such as image synthesis and style transfer. 21 | 22 | ## Setup and Getting Started 23 | To Run: 24 | 1. Open `differentiable_rendering_example.mlx` live script in MATLAB® 25 | 2. Run the file. 26 | 27 | ### MathWorks Products (https://www.mathworks.com) 28 | 29 | 1. Requires MATLAB® release R2024a or newer 30 | 2. [Deep Learning Toolbox™](https://www.mathworks.com/products/deep-learning.html) 31 | 32 | ## License 33 | 34 | The license is available in the License.txt file in this GitHub repository. 35 | 36 | ## Community Support 37 | [MATLAB Central](https://www.mathworks.com/matlabcentral) 38 | 39 | Copyright 2024 The MathWorks, Inc. 40 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | # Reporting Security Vulnerabilities 2 | 3 | If you believe you have discovered a security vulnerability, please report it to 4 | [security@mathworks.com](mailto:security@mathworks.com). Please see 5 | [MathWorks Vulnerability Disclosure Policy for Security Researchers](https://www.mathworks.com/company/aboutus/policies_statements/vulnerability-disclosure-policy.html) 6 | for additional information. -------------------------------------------------------------------------------- /bernie.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/matlab-deep-learning/2D-Differentiable-Rendering-Example/60097b516259f39c417ad3d0158527c98a69b887/bernie.jpg -------------------------------------------------------------------------------- /circle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/matlab-deep-learning/2D-Differentiable-Rendering-Example/60097b516259f39c417ad3d0158527c98a69b887/circle.png -------------------------------------------------------------------------------- /circle_animation.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/matlab-deep-learning/2D-Differentiable-Rendering-Example/60097b516259f39c417ad3d0158527c98a69b887/circle_animation.gif -------------------------------------------------------------------------------- /differentiable_rendering.m: -------------------------------------------------------------------------------- 1 | %% 2D Differentiable Rendering in MATLAB 2 | 3 | % This example shows how to perform differentiable vector graphics rendering 4 | % in MATLAB®. It makes use of dlarray for auto-differentiation, and employs 5 | % signed distance functions and sigmoid smoothing to ensure the rasterization 6 | % pipeline is continuous and differentiable. First, we look at how to perform 7 | % differentiable rendering. Then, we use the differentiable rendering pipeline 8 | % to optimize shape parameters to fit a target image. 9 | 10 | % Copyright 2024 The MathWorks, Inc 11 | 12 | %% Optimizing rasterized shapes to approximate an image 13 | % Choose one of the following examples to see how we can optimize shape parameters 14 | % to render an image as similar as possible to a target image. 15 | 16 | % In circles example, we optimize the position, size and color of two circles 17 | % to perfectly match a target image. 18 | % 19 | % In gaussians example, we optimize the mean, sigma, and color of a large number 20 | % of Gaussian functions to approximate a target image. 21 | % 22 | % In rectangles example, we optimize the position, size and color of many rectangles 23 | % to approximate a target image. 24 | 25 | example = "circles"; 26 | % example = "gaussians"; 27 | % example = "rectangles"; 28 | 29 | %% Load a target image to approximate 30 | 31 | % Here we load the image we want to approximate with our shapes. We also pre-compute 32 | % a grid of normalized pixel coordinates to be used in the rendering function. 33 | 34 | % Load target image 35 | if strcmp(example, "rectangles") || strcmp(example, "gaussians") 36 | target_image = im2double(imread('bernie.jpg')); 37 | elseif strcmp(example, "circles") 38 | target_image = im2double(imread('circle.png')); 39 | end 40 | 41 | % Convert to dlarray 42 | target_image = dlarray(target_image, 'SSCB'); 43 | 44 | % Precompute a pixel grid 45 | [image_height, image_width, ~, ~] = size(target_image); 46 | [X, Y] = ndgrid(linspace(0,1,image_height), linspace(0,1,image_width)); 47 | X = dlarray(X, 'SSCB'); 48 | Y = dlarray(Y, 'SSCB'); 49 | 50 | %% Define the main hyperparameters 51 | 52 | % The hyperparameters are the type of shape to use (rectangle, circle or Gaussian), 53 | % and the number of them to use. 54 | 55 | switch example 56 | case "rectangles" 57 | % Select the type of shape and number of them to use 58 | shape_type = "rectangle"; 59 | num_shapes = 75; 60 | case "circles" 61 | shape_type = "circle"; 62 | num_shapes = 2; 63 | case "gaussians" 64 | shape_type = "gaussians"; 65 | num_shapes = 150; 66 | end 67 | 68 | %% Initialize the parameters 69 | 70 | % Start by choosing a random initialization of the shape parameters. _Note: 71 | % To find a particularly good set of initial values one could repeat the random 72 | % initialization process multiple times and choose the set that result in the 73 | % lowest loss function. This is not implemented here_. 74 | % 75 | % *The shapes are each parameterized by three variables:* 76 | % 77 | % For *rectangles*, _center_ is the coordinate of the center of the rectangle, 78 | % _dimensions_ is the width and height of the rectangle, and _color_ is the RGB 79 | % color. 80 | % 81 | % For circle*s*, _center_ is the coordinate of the center of the circle, _dimensions_ 82 | % is the width and height of the circle, and _color_ is the RGB color. 83 | % 84 | % For *Gaussians*, _center_ is the coordinate of the center (i.e. the mean), 85 | % _dimensions_ is the standard deviation in the x and y directions, and _color_ 86 | % is the RGB color 87 | 88 | switch example 89 | case {"rectangles", "gaussians"} 90 | % Make a random initialization for the shape parameters 91 | % Values are chosen to be in a reasonable range 92 | center = dlarray(rand(2, num_shapes), 'CB'); 93 | dimensions = dlarray(0.1 + rand(2, num_shapes) * 0.5, 'CB'); 94 | color = dlarray(rand(3, num_shapes), 'CB'); 95 | 96 | case "circles" 97 | % Make a random initialization for the shape parameters 98 | % Values are chosen to be in a reasonable range. 99 | 100 | % We set the two circles positions such that one will appear somewhere in the left side of 101 | % the image and one somewhere on the right side. 102 | 103 | center = dlarray([0.25, 0.25; 0, 0.5] + 0.5*rand(2, num_shapes), 'CB'); 104 | dimensions = dlarray(0.5 + rand(2, num_shapes) * 0.5, 'CB'); 105 | color = dlarray(rand(3, num_shapes), 'CB'); 106 | end 107 | 108 | % Because the shapes colors are added to one another, the final 109 | % render can have pixel values much greater than 1. 110 | % We normalize the initial colours to ensure the rendered image has the same 111 | % mean as the target image to help the optimization converge. 112 | rendered_image = render(center, dimensions, color, X, Y, shape_type); 113 | color = color .* (mean(target_image(:))/mean(rendered_image(:))); 114 | 115 | % Plot the initialization render alongside the target image 116 | figure 117 | rendered_image = render(center, dimensions, color, X, Y, shape_type); 118 | 119 | subplot(1, 2, 1); 120 | imshow(extractdata(rendered_image)); 121 | title('Initialization'); 122 | 123 | subplot(1, 2, 2); 124 | imshow(extractdata(target_image)) 125 | title('Target Image'); 126 | 127 | %% Optimize 128 | 129 | % Set the optimization hyperparameters 130 | 131 | % Set the learning rate. We use a slower learning rate for the circles 132 | % example to more clearly visualize how it optimizes over time 133 | switch example 134 | case {"rectangles", "gaussians"} 135 | learning_rate = 0.75; 136 | case "circles" 137 | learning_rate = 0.05; 138 | end 139 | 140 | % Multiplier for the learning rate to reduce it over time 141 | decay_rate = 0.99; 142 | 143 | % Total number of iterations 144 | num_iterations = 125; 145 | 146 | % Variables to store the loss function value over time, and the momentum 147 | % vectors used by the sgdmupdate function 148 | losses = zeros(num_iterations,1); 149 | v_center = []; 150 | v_dimensions = []; 151 | v_color = []; 152 | 153 | figure 154 | 155 | for iter = 1:num_iterations 156 | % Compute gradients and loss value 157 | [gradients, loss_value] = dlfeval(@computeGradients, center, dimensions, color, target_image, X, Y, shape_type); 158 | 159 | % Extract the gradients 160 | grad_center = gradients{1}; 161 | grad_dimensions = gradients{2}; 162 | grad_color = gradients{3}; 163 | 164 | % Update shape parameters using the sgdm optimizer 165 | [center, v_center] = sgdmupdate(center, grad_center, v_center, learning_rate); 166 | [dimensions, v_dimensions] = sgdmupdate(dimensions, grad_dimensions, v_dimensions, learning_rate); 167 | [color, v_color] = sgdmupdate(color, grad_color, v_color, learning_rate); 168 | 169 | % Clip the parameters to ensure they stay within reasonable bounds 170 | center = clip(center, 0.01, 0.99); 171 | dimensions = clip(dimensions, 0.01, 1); 172 | 173 | % Reduce the learning rate over time to make smaller adjustments at 174 | % later iterations 175 | learning_rate = learning_rate * decay_rate; 176 | 177 | % Store the cost function 178 | losses(iter) = extractdata(loss_value); 179 | 180 | % Plot the rendered image alongside the target image 181 | rendered_image = render(center, dimensions, color, X, Y, shape_type); 182 | 183 | subplot(2, 2, 1); 184 | imshow(extractdata(rendered_image)); 185 | title(['Iteration: ', num2str(iter)]); 186 | 187 | subplot(2, 2, 2); 188 | imshow(extractdata(target_image)); 189 | title('Target Image'); 190 | 191 | subplot(2, 2, [3, 4]); 192 | plot(losses); 193 | xlabel('Iteration'); 194 | ylabel('Loss'); 195 | title("Learning rate " + string(learning_rate) + " Loss: "+string(extractdata(loss_value))); 196 | 197 | drawnow; 198 | end 199 | 200 | %% Supporting functions 201 | 202 | % Loss function 203 | function loss = computeLoss(center, dimensions, color, target_image, X, Y, shape_type) 204 | % Render the image given the current shape parameters, and then 205 | % calculate the mean-square-error of the render compared to the target 206 | % image. 207 | rendered_image = render(center, dimensions, color, X, Y, shape_type); 208 | loss = mean((rendered_image(:) - target_image(:)).^2); 209 | end 210 | 211 | %% 212 | 213 | % Function to compute gradients with respect to shape parameters 214 | function [gradients, loss_value] = computeGradients(center, dimensions, color, target_image, X, Y, shape_type) 215 | % Evaluate the loss function and the gradient with respect to the shape 216 | % parameters 217 | loss_value = computeLoss(center, dimensions, color, target_image, X, Y, shape_type); 218 | gradients = dlgradient(loss_value, {center, dimensions, color}); 219 | end 220 | 221 | %% 222 | 223 | % The differentiable rendering function 224 | % 225 | % The edges of the shapes are blurred using a sigmoid function which makes the 226 | % rasterization continuous and differentiable. Different shapes are defined using 227 | % different signed distance functions. 228 | function rendered_image = render(center, dimensions, color, X, Y, shape_type) 229 | % This function takes the shape parameters, pixel grids, and shape type 230 | % and then outputs the final rendered image. 231 | 232 | % It uses the same process as detailed in the 'Differentiable rendering introduce' 233 | % section, but the code is vectorized to perform all the steps for all 234 | % shapes simultaneously for speed. 235 | 236 | switch shape_type 237 | case "rectangle" 238 | % Render each rectangle using the signed distance function followed 239 | % by the sigmoid function. 240 | dist_to_center = max(abs(X - reshape(center(1,:), 1, 1,1, [])) ./ (0.5 * reshape(dimensions(1,:), 1, 1,1, [])), ... 241 | abs(Y - reshape(center(2,:), 1, 1,1, [])) ./ (0.5 * reshape(dimensions(2,:), 1, 1,1, []))); 242 | mask = stableSigmoid(dist_to_center); 243 | case "circle" 244 | % Render each circle using the signed distance function followed 245 | % by the sigmoid function 246 | dist_to_center = (X - reshape(center(1,:), 1, 1,1, [])).^2 ./ ((0.5 * reshape(dimensions(1,:), 1, 1,1, [])).^2 + 1e-8) ... 247 | + (Y - reshape(center(2,:), 1, 1,1, [])).^2 ./ ((0.5 * reshape(dimensions(2,:), 1, 1,1, [])).^2 + 1e-8); 248 | mask = stableSigmoid(dist_to_center); 249 | case "gaussians" 250 | % Render each gaussian. This does not require signed distance 251 | % functions or sigmoids -- we just use the standard formula for a 252 | % 2D Gaussian. 253 | mask = exp(-((X - reshape(center(1,:), 1, 1,1, [])).^2 ./ (0.02 * reshape(dimensions(1,:), 1, 1,1, [])) + ... 254 | (Y - reshape(center(2,:), 1, 1,1, [])).^2 ./ (0.02 * reshape(dimensions(2,:), 1, 1,1, [])))); 255 | end 256 | 257 | % Combine each of the shapes into one image with additive blending 258 | % Different blending types will give different results! 259 | rendered_image = sum(mask .* reshape(color, 1, 1, size(color, 1), []), 4); 260 | end 261 | 262 | %% 263 | 264 | % Stable sigmoid function for smooth edge approximation 265 | function y = stableSigmoid(x) 266 | y = 1 ./ (1 + exp(-clip(150 * (1-x),-20,20))); 267 | y = clip(y, 1e-6, 1-1e-6); 268 | end 269 | -------------------------------------------------------------------------------- /differentiable_rendering_example.mlx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/matlab-deep-learning/2D-Differentiable-Rendering-Example/60097b516259f39c417ad3d0158527c98a69b887/differentiable_rendering_example.mlx -------------------------------------------------------------------------------- /example_animation.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/matlab-deep-learning/2D-Differentiable-Rendering-Example/60097b516259f39c417ad3d0158527c98a69b887/example_animation.gif -------------------------------------------------------------------------------- /license.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2024, The MathWorks, Inc. 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are 6 | met: 7 | 8 | * Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | * Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in 12 | the documentation and/or other materials provided with the distribution 13 | * Neither the name of the The MathWorks, Inc. nor the names 14 | of its contributors may be used to endorse or promote products derived 15 | from this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 27 | POSSIBILITY OF SUCH DAMAGE. 28 | --------------------------------------------------------------------------------