├── .gitignore ├── Assets ├── ExampleInput.jpg ├── ExampleMask.jpg ├── ExampleOutput.jpg ├── ExampleWithMaskInput.jpg ├── ExampleWithMaskOutput.jpg └── None.png └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | -------------------------------------------------------------------------------- /Assets/ExampleInput.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/YuAo/HexagonalBokehBlur/6945e73c674f527b87c075fd9be1a379905e3a99/Assets/ExampleInput.jpg -------------------------------------------------------------------------------- /Assets/ExampleMask.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/YuAo/HexagonalBokehBlur/6945e73c674f527b87c075fd9be1a379905e3a99/Assets/ExampleMask.jpg -------------------------------------------------------------------------------- /Assets/ExampleOutput.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/YuAo/HexagonalBokehBlur/6945e73c674f527b87c075fd9be1a379905e3a99/Assets/ExampleOutput.jpg -------------------------------------------------------------------------------- /Assets/ExampleWithMaskInput.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/YuAo/HexagonalBokehBlur/6945e73c674f527b87c075fd9be1a379905e3a99/Assets/ExampleWithMaskInput.jpg -------------------------------------------------------------------------------- /Assets/ExampleWithMaskOutput.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/YuAo/HexagonalBokehBlur/6945e73c674f527b87c075fd9be1a379905e3a99/Assets/ExampleWithMaskOutput.jpg -------------------------------------------------------------------------------- /Assets/None.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/YuAo/HexagonalBokehBlur/6945e73c674f527b87c075fd9be1a379905e3a99/Assets/None.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Hexagonal Bokeh Blur Filter 2 | 3 | This is an implementation note for the "[Hexagonal Bokeh Blur Filter](https://github.com/MetalPetal/MetalPetal/blob/master/Frameworks/MetalPetal/Filters/MTIHexagonalBokehBlurFilter.h)" in [MetalPetal](https://github.com/MetalPetal/MetalPetal). 4 | 5 | ## Previews 6 | 7 | | Input | Mask | Output | 8 | | ------------- | ------------- | ------------- | 9 | | ![Example 1](Assets/ExampleInput.jpg) | ![Example 1](Assets/None.png) | ![Example 1 Output](Assets/ExampleOutput.jpg) | 10 | | ![Example 2](Assets/ExampleWithMaskInput.jpg) | ![Example 2 Mask](Assets/ExampleMask.jpg) | ![Example 2 Output](Assets/ExampleWithMaskOutput.jpg) | 11 | 12 | ## Implementation 13 | 14 | ### Files 15 | 16 | [MTIHexagonalBokehBlurFilter.h](https://github.com/MetalPetal/MetalPetal/blob/master/Frameworks/MetalPetal/Filters/MTIHexagonalBokehBlurFilter.h) 17 | 18 | [MTIHexagonalBokehBlurFilter.m](https://github.com/MetalPetal/MetalPetal/blob/master/Frameworks/MetalPetal/Filters/MTIHexagonalBokehBlurFilter.m) 19 | 20 | [LensBlur.metal](https://github.com/MetalPetal/MetalPetal/blob/master/Frameworks/MetalPetal/Shaders/LensBlur.metal) 21 | 22 | ### Overview 23 | 24 | The implementation is based on the concept described in **WHITE, John, and BARRÉ-BRISEBOIS, Colin. More Performance! Five Rendering Ideas From Battlefield 3 and Need For Speed: The Run, Advances in Real-Time Rendering in Games, SIGGRAPH 2011. [[Slides](https://www.slideshare.net/DICEStudio/five-rendering-ideas-from-battlefield-3-need-for-speed-the-run)]**, as well as the excellent blog post of Colin Barré-Brisebois [ 25 | Hexagonal Bokeh Blur Revisited](https://colinbarrebrisebois.com/2017/04/18/hexagonal-bokeh-blur-revisited/). 26 | 27 | The implementation primarily follows the improved 2-pass version described in [ 28 | Hexagonal Bokeh Blur Revisited](https://colinbarrebrisebois.com/2017/04/18/hexagonal-bokeh-blur-revisited/) with the following modifications. 29 | 30 | ### Mask and CoC (Circle of Confusion) 31 | 32 | The per pixel circle of confusion (CoC) should be calculated from real world camera parameters (focal length, aperture, focal plane, etc). Since we are building an image processing framework, and we do not have the full information of the camera or the depth map of the image, we leave this part to the framework users. 33 | 34 | You need to provide an image as the source of per pixel CoC (the `inputMask` parameter). This mask image along with the `radius` parameter determine the per pixel blur amount. 35 | 36 | The mask image can be generated in many ways from many sources depending on your application and needs: 37 | 38 | - Using an image semantic segmentation neual network to create a foreground/background segementation mask. 39 | 40 | - Using the depth map from the dual camera system or the TrueDepth camera to generate a mask. [WWDC 17 - Session 508](https://developer.apple.com/videos/play/wwdc2017/508/) / [WWDC 17 - Session 507](https://developer.apple.com/videos/play/wwdc2017/507/) 41 | 42 | - Using the portrait segmentation API of iOS 12. [WWDC 18 - Session 503](https://developer.apple.com/videos/play/wwdc2018/503/) 43 | 44 | - Don't provide a mask, make the full image blur. 45 | 46 | - Let the user draw a mask. 47 | 48 | ### HDR and Lightness 49 | 50 | `MTIHexagonalBokehBlurFilter` uses a `pow(n)` and `pow(1/n)` approach to simulate highlights. The input texture is first powered with the `lightness_factor` to make the lighter colors pop out. After the blur effect is applied, the texture is powered with the `1.0/lightness_factor` to restore the color of the image. (Inspired by [@evanw](https://github.com/evanw) https://github.com/evanw/glfx.js/blob/master/src/filters/blur/lensblur.js#L18) 51 | 52 | We also did a little tweak to the original pixel shader to make the overall lightness of the output image to be consistent with the input image. 53 | 54 | The original second pass pixel shader: 55 | 56 | ``` 57 | // Get the center to determine the radius of the blur 58 | float coc = tex2D(verticalBlurTexture, uv).a; 59 | float coc2 = tex2D(diagonalBlurTexture, uv).a; 60 | 61 | // Sample the vertical blur (1st MRT) texture with this new blur direction 62 | float2 blurDir = coc * invViewDims * float2(cos(-PI/6), sin(-PI/6)); 63 | float4 color = BlurTexture(verticalBlurTexture, uv, blurDir) * coc; 64 | 65 | // Sample the diagonal blur (2nd MRT) texture with this new blur direction 66 | float2 blurDir2 = coc2 * invViewDims * float2(cos(-5*PI/6), sin(-5*PI/6)); 67 | float4 color2 = BlurTexture(diagonalBlurTexture, uv, blurDir2) * coc2; 68 | 69 | float3 output = (color.rgb + color2.rgb) * 0.5f; 70 | ``` 71 | 72 | Ours: 73 | 74 | ``` 75 | ... 76 | float coc = verticalTexture.sample(verticalSampler, vertexIn.textureCoordinate).a; 77 | float coc2 = diagonalTexture.sample(diagonalSampler, vertexIn.textureCoordinate).a; 78 | float4 color = (sampleWithDelta(verticalTexture, verticalSampler, vertexIn.textureCoordinate, delta0 * coc) + 79 | sampleWithDelta(diagonalTexture, diagonalSampler, vertexIn.textureCoordinate, delta1 * coc2)) 80 | * (1.0/3.0); 81 | ... 82 | 83 | ``` 84 | 85 | Notice the final `* (1.0/3.0)` vs `* 0.5`. 86 | 87 | The `verticalTexture` contains one color sample, while, the `diagonalTexture` contains two color samples: 88 | ``` 89 | // Output to MRT 90 | PSOUTPUT output; 91 | output.vertical = float4(color.rgb, coc); 92 | output.diagonal = float4(color2.rgb + output.vertical.xyz, coc); 93 | ``` 94 | It is reasonable to `* (1.0 / 3.0)` in the end to maintain the lightness consistance. 95 | 96 | ### Multiple Render Targets in MetalPetal 97 | 98 | MRT(Multiple Render Targets) is a feature of modern GPUs that allows the programmable rendering pipeline to render images to multiple render target textures at once. The 2-passs version of the filter requires MRT. 99 | 100 | It is quite simple to do MRT with [MetalPetal](https://github.com/MetalPetal/MetalPetal). 101 | 102 | - Create a render pipeline with the `colorAttachmentCount` set to the number of render targets you'd like to use. 103 | 104 | ``` 105 | // MTIHexagonalBokehBlurFilter.m 106 | 107 | kernel = [[MTIRenderPipelineKernel alloc] initWithVertexFunctionDescriptor:[[MTIFunctionDescriptor alloc] initWithName:MTIFilterPassthroughVertexFunctionName] 108 | fragmentFunctionDescriptor:[[MTIFunctionDescriptor alloc] initWithName:@"hexagonalBokehBlurAlpha"] 109 | vertexDescriptor:nil 110 | colorAttachmentCount:2 111 | alphaTypeHandlingRule:MTIAlphaTypeHandlingRule.generalAlphaTypeHandlingRule]; 112 | 113 | ``` 114 | 115 | - Mark the fragment shader output struct member with `color(n)` attributes. 116 | 117 | ``` 118 | // LensBlur.metal 119 | 120 | typedef struct { 121 | float4 vertical [[color(0)]]; 122 | float4 diagonal [[color(1)]]; 123 | } HexagonalBokehBlurAlphaPassOutput; 124 | 125 | ``` 126 | 127 | - Get the output images. 128 | 129 | ``` 130 | // MTIHexagonalBokehBlurFilter.m 131 | 132 | NSArray *outputs = [[MTIHexagonalBokehBlurFilter alphaPassKernel] applyToInputImages:@[...] 133 | parameters:@{...} 134 | outputDescriptors:@[firstOutputDescriptor,secondOutputDescriptor]; 135 | ``` 136 | 137 | ## Acknowledgements 138 | 139 | Thanks to Jackie ([@fsjack](https://github.com/fsjack)) for the mask and coc releated optimzations. 140 | 141 | Thanks to Martin ([@obs1dium](https://github.com/obs1dium)) for fixing the sample issue [#47](https://github.com/MetalPetal/MetalPetal/issues/47) and [pointing out the brightness problem](https://github.com/MetalPetal/MetalPetal/commit/3edb4c838d567410584aff22bc9067089a2ed2ea#comments). 142 | 143 | [@evanw](https://github.com/evanw/) for his [JavaScript/WebGL implementation](https://github.com/evanw/glfx.js). 144 | 145 | --- 146 | 147 | © 2019 Yu Ao 148 | 149 | Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License. 150 | 151 | 152 | --------------------------------------------------------------------------------