├── .gitignore ├── LICENSE ├── README.md ├── Tiramisu.xcodeproj ├── project.pbxproj └── project.xcworkspace │ ├── contents.xcworkspacedata │ └── xcshareddata │ └── IDEWorkspaceChecks.plist ├── Tiramisu ├── AppDelegate.swift ├── Assets.xcassets │ ├── AppIcon.appiconset │ │ └── Contents.json │ └── Contents.json ├── Base.lproj │ ├── LaunchScreen.storyboard │ └── Main.storyboard ├── CoreMLHelpers │ ├── Math.swift │ ├── MultiArray.swift │ ├── Probs.swift │ └── UIImage+CVPixelBuffer.swift ├── Info.plist ├── Models │ ├── Tiramisu45.h5 │ ├── Tiramisu45.mlmodel │ └── convert.py ├── UIKitHelpers │ └── Popup.swift └── ViewController.swift └── img ├── cmap.png └── example.png /.gitignore: -------------------------------------------------------------------------------- 1 | ## Build generated 2 | build/ 3 | DerivedData/ 4 | 5 | ## Various settings 6 | *.pbxuser 7 | !default.pbxuser 8 | *.mode1v3 9 | !default.mode1v3 10 | *.mode2v3 11 | !default.mode2v3 12 | *.perspectivev3 13 | !default.perspectivev3 14 | xcuserdata/ 15 | 16 | ## Other 17 | *.moved-aside 18 | *.xccheckout 19 | *.xcscmblueprint 20 | 21 | ## Obj-C/Swift specific 22 | *.hmap 23 | *.ipa 24 | *.dSYM.zip 25 | *.dSYM 26 | 27 | ## Playgrounds 28 | timeline.xctimeline 29 | playground.xcworkspace 30 | 31 | # Swift Package Manager 32 | # 33 | # Add this line if you want to avoid checking in source code from Swift Package Manager dependencies. 34 | # Packages/ 35 | # Package.pins 36 | .build/ 37 | 38 | # Carthage 39 | # 40 | # Add this line if you want to avoid checking in source code from Carthage dependencies. 41 | # Carthage/Checkouts 42 | Carthage/Build 43 | *.bcsymbolmap 44 | 45 | # MacOS 46 | # General 47 | *.DS_Store 48 | .AppleDouble 49 | .LSOverride 50 | 51 | # Icon must end with two \r 52 | Icon 53 | 54 | # Thumbnails 55 | ._* 56 | 57 | # Files that might appear in the root of a volume 58 | .DocumentRevisions-V100 59 | .fseventsd 60 | .Spotlight-V100 61 | .TemporaryItems 62 | .Trashes 63 | .VolumeIcon.icns 64 | .com.apple.timemachine.donotpresent 65 | 66 | # Directories potentially created on remote AFP share 67 | .AppleDB 68 | .AppleDesktop 69 | Network Trash Folder 70 | Temporary Items 71 | .apdisk 72 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Christian Kauten 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # iOS Semantic Segmentation 2 | 3 | An example of semantic segmentation on iOS using CoreML and Keras. Trained 4 | Tiramisu 45 weights come from [here][sem-seg]. A device with a camera is 5 | required, preferably a newer one to maintain an acceptable frame-rate from 6 | the model. 7 | 8 | [sem-seg]: https://github.com/Kautenja/neural-semantic-segmentation 9 | 10 |

11 | Predictions from Tiramisu 45 on iPhone XS Video Stream. 12 | 13 | 14 | 17 | 20 | 21 |
15 | Segmentation Demonstration 16 | 18 | Color Legend 19 |
22 |

23 | 24 | - note that the 1280 × 720 input image is scaled (fill) to 480 × 352, 25 | explaining the discrepancy in size between the camera stream and 26 | segmentation outputs 27 | 28 | ## Requirements 29 | 30 | - iOS >= 12.x 31 | - The Metal Performance Shader for ArgMax feature channel reduction is 32 | only available from iOS 12 onward. An iterative CPU implementation of 33 | ArgMax results in a _3x_ slowdown compared to the vectorized GPU one 34 | on Metal (on iPhone XS). 35 | 36 | ## Model 37 | 38 | The original Keras model file can be found in [Tiramisu/Models][models] as 39 | [Tiramisu45.h5][model-h5]. An accompanying python file, [convert.py][convert], 40 | handles the conversion from the Keras model into a CoreML model as 41 | [Tiramisu45.mlmodel][model-mlmodel] using [coremltools][coremltools]. The 42 | model is trained first on CamVid, then on CityScapes using similar 43 | hyperparameters as reported in the original paper. Additional augmentation 44 | is performed (brightness adjustment, random rotations) during training to 45 | promote a model that is robust against variations in lighting and angle 46 | from the camera. 47 | 48 | [models]: ./Tiramisu/Models 49 | [convert]: ./Tiramisu/Models/convert.py 50 | [model-h5]: ./Tiramisu/Models/Tiramisu45.h5 51 | [model-mlmodel]: ./Tiramisu/Models/Tiramisu45.mlmodel 52 | [coremltools]: https://github.com/apple/coremltools 53 | 54 | ## Frame Rate 55 | 56 | Tiramisu 45 is heavy weight despite few (≈800,000) parameters due to the 57 | skip connections in dense blocks and between the encoder and decoder. As a 58 | result, the frame-rate suffers. The values reported here are averaged over 59 | 30 seconds of runtime after application initialization. Note that because of 60 | intense computation, the devices will get hot quickly and begin thermal 61 | throttling. The iPhone XS frame-rate drops to ≈6 when this throttling occurs. 62 | 63 | | Device | Frame Rate | 64 | |:----------|:-----------| 65 | | iPhone XS | ≈ 12 | 66 | | iPhone 7 | ≈ 2 | 67 | | iPad Air | < 1 | 68 | -------------------------------------------------------------------------------- /Tiramisu.xcodeproj/project.pbxproj: -------------------------------------------------------------------------------- 1 | // !$*UTF8*$! 2 | { 3 | archiveVersion = 1; 4 | classes = { 5 | }; 6 | objectVersion = 50; 7 | objects = { 8 | 9 | /* Begin PBXBuildFile section */ 10 | 0945E3AA21755E6300B2AB60 /* AppDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0945E3A921755E6300B2AB60 /* AppDelegate.swift */; }; 11 | 0945E3AC21755E6300B2AB60 /* ViewController.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0945E3AB21755E6300B2AB60 /* ViewController.swift */; }; 12 | 0945E3AF21755E6300B2AB60 /* Main.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 0945E3AD21755E6300B2AB60 /* Main.storyboard */; }; 13 | 0945E3B121755E6400B2AB60 /* Assets.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 0945E3B021755E6400B2AB60 /* Assets.xcassets */; }; 14 | 0945E3B421755E6400B2AB60 /* LaunchScreen.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 0945E3B221755E6400B2AB60 /* LaunchScreen.storyboard */; }; 15 | 0945E4172175811C00B2AB60 /* Math.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0945E40F2175811C00B2AB60 /* Math.swift */; }; 16 | 0945E4182175811C00B2AB60 /* UIImage+CVPixelBuffer.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0945E4102175811C00B2AB60 /* UIImage+CVPixelBuffer.swift */; }; 17 | 0945E41B2175811C00B2AB60 /* MultiArray.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0945E4132175811C00B2AB60 /* MultiArray.swift */; }; 18 | 09A102052176932300220459 /* Tiramisu45.mlmodel in Sources */ = {isa = PBXBuildFile; fileRef = 09A102042176932300220459 /* Tiramisu45.mlmodel */; }; 19 | 09DC0B562175B4090044D811 /* Probs.swift in Sources */ = {isa = PBXBuildFile; fileRef = 09DC0B552175B4090044D811 /* Probs.swift */; }; 20 | 09DC0B592175B5690044D811 /* Popup.swift in Sources */ = {isa = PBXBuildFile; fileRef = 09DC0B582175B5690044D811 /* Popup.swift */; }; 21 | /* End PBXBuildFile section */ 22 | 23 | /* Begin PBXFileReference section */ 24 | 0945E3A621755E6300B2AB60 /* Tiramisu.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = Tiramisu.app; sourceTree = BUILT_PRODUCTS_DIR; }; 25 | 0945E3A921755E6300B2AB60 /* AppDelegate.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AppDelegate.swift; sourceTree = ""; }; 26 | 0945E3AB21755E6300B2AB60 /* ViewController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ViewController.swift; sourceTree = ""; }; 27 | 0945E3AE21755E6300B2AB60 /* Base */ = {isa = PBXFileReference; lastKnownFileType = file.storyboard; name = Base; path = Base.lproj/Main.storyboard; sourceTree = ""; }; 28 | 0945E3B021755E6400B2AB60 /* Assets.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = Assets.xcassets; sourceTree = ""; }; 29 | 0945E3B321755E6400B2AB60 /* Base */ = {isa = PBXFileReference; lastKnownFileType = file.storyboard; name = Base; path = Base.lproj/LaunchScreen.storyboard; sourceTree = ""; }; 30 | 0945E3B521755E6400B2AB60 /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = ""; }; 31 | 0945E40F2175811C00B2AB60 /* Math.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = Math.swift; sourceTree = ""; }; 32 | 0945E4102175811C00B2AB60 /* UIImage+CVPixelBuffer.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = "UIImage+CVPixelBuffer.swift"; sourceTree = ""; }; 33 | 0945E4132175811C00B2AB60 /* MultiArray.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = MultiArray.swift; sourceTree = ""; }; 34 | 09A102042176932300220459 /* Tiramisu45.mlmodel */ = {isa = PBXFileReference; lastKnownFileType = file.mlmodel; path = Tiramisu45.mlmodel; sourceTree = ""; }; 35 | 09DC0B552175B4090044D811 /* Probs.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Probs.swift; sourceTree = ""; }; 36 | 09DC0B582175B5690044D811 /* Popup.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Popup.swift; sourceTree = ""; }; 37 | /* End PBXFileReference section */ 38 | 39 | /* Begin PBXFrameworksBuildPhase section */ 40 | 0945E3A321755E6300B2AB60 /* Frameworks */ = { 41 | isa = PBXFrameworksBuildPhase; 42 | buildActionMask = 2147483647; 43 | files = ( 44 | ); 45 | runOnlyForDeploymentPostprocessing = 0; 46 | }; 47 | /* End PBXFrameworksBuildPhase section */ 48 | 49 | /* Begin PBXGroup section */ 50 | 0945E39D21755E6300B2AB60 = { 51 | isa = PBXGroup; 52 | children = ( 53 | 0945E3A821755E6300B2AB60 /* Tiramisu */, 54 | 0945E3A721755E6300B2AB60 /* Products */, 55 | ); 56 | sourceTree = ""; 57 | }; 58 | 0945E3A721755E6300B2AB60 /* Products */ = { 59 | isa = PBXGroup; 60 | children = ( 61 | 0945E3A621755E6300B2AB60 /* Tiramisu.app */, 62 | ); 63 | name = Products; 64 | sourceTree = ""; 65 | }; 66 | 0945E3A821755E6300B2AB60 /* Tiramisu */ = { 67 | isa = PBXGroup; 68 | children = ( 69 | 09DC0B572175B5550044D811 /* UIKitHelpers */, 70 | 0945E40E2175811C00B2AB60 /* CoreMLHelpers */, 71 | 0945E3BB21755EF500B2AB60 /* Models */, 72 | 0945E3A921755E6300B2AB60 /* AppDelegate.swift */, 73 | 0945E3AB21755E6300B2AB60 /* ViewController.swift */, 74 | 0945E3AD21755E6300B2AB60 /* Main.storyboard */, 75 | 0945E3B021755E6400B2AB60 /* Assets.xcassets */, 76 | 0945E3B221755E6400B2AB60 /* LaunchScreen.storyboard */, 77 | 0945E3B521755E6400B2AB60 /* Info.plist */, 78 | ); 79 | path = Tiramisu; 80 | sourceTree = ""; 81 | }; 82 | 0945E3BB21755EF500B2AB60 /* Models */ = { 83 | isa = PBXGroup; 84 | children = ( 85 | 09A102042176932300220459 /* Tiramisu45.mlmodel */, 86 | ); 87 | path = Models; 88 | sourceTree = ""; 89 | }; 90 | 0945E40E2175811C00B2AB60 /* CoreMLHelpers */ = { 91 | isa = PBXGroup; 92 | children = ( 93 | 0945E40F2175811C00B2AB60 /* Math.swift */, 94 | 0945E4102175811C00B2AB60 /* UIImage+CVPixelBuffer.swift */, 95 | 0945E4132175811C00B2AB60 /* MultiArray.swift */, 96 | 09DC0B552175B4090044D811 /* Probs.swift */, 97 | ); 98 | path = CoreMLHelpers; 99 | sourceTree = ""; 100 | }; 101 | 09DC0B572175B5550044D811 /* UIKitHelpers */ = { 102 | isa = PBXGroup; 103 | children = ( 104 | 09DC0B582175B5690044D811 /* Popup.swift */, 105 | ); 106 | path = UIKitHelpers; 107 | sourceTree = ""; 108 | }; 109 | /* End PBXGroup section */ 110 | 111 | /* Begin PBXNativeTarget section */ 112 | 0945E3A521755E6300B2AB60 /* Tiramisu */ = { 113 | isa = PBXNativeTarget; 114 | buildConfigurationList = 0945E3B821755E6400B2AB60 /* Build configuration list for PBXNativeTarget "Tiramisu" */; 115 | buildPhases = ( 116 | 0945E3A221755E6300B2AB60 /* Sources */, 117 | 0945E3A321755E6300B2AB60 /* Frameworks */, 118 | 0945E3A421755E6300B2AB60 /* Resources */, 119 | ); 120 | buildRules = ( 121 | ); 122 | dependencies = ( 123 | ); 124 | name = Tiramisu; 125 | productName = Tiramisu; 126 | productReference = 0945E3A621755E6300B2AB60 /* Tiramisu.app */; 127 | productType = "com.apple.product-type.application"; 128 | }; 129 | /* End PBXNativeTarget section */ 130 | 131 | /* Begin PBXProject section */ 132 | 0945E39E21755E6300B2AB60 /* Project object */ = { 133 | isa = PBXProject; 134 | attributes = { 135 | LastSwiftUpdateCheck = 1000; 136 | LastUpgradeCheck = 1000; 137 | ORGANIZATIONNAME = Kautenja; 138 | TargetAttributes = { 139 | 0945E3A521755E6300B2AB60 = { 140 | CreatedOnToolsVersion = 10.0; 141 | }; 142 | }; 143 | }; 144 | buildConfigurationList = 0945E3A121755E6300B2AB60 /* Build configuration list for PBXProject "Tiramisu" */; 145 | compatibilityVersion = "Xcode 9.3"; 146 | developmentRegion = en; 147 | hasScannedForEncodings = 0; 148 | knownRegions = ( 149 | en, 150 | Base, 151 | ); 152 | mainGroup = 0945E39D21755E6300B2AB60; 153 | productRefGroup = 0945E3A721755E6300B2AB60 /* Products */; 154 | projectDirPath = ""; 155 | projectRoot = ""; 156 | targets = ( 157 | 0945E3A521755E6300B2AB60 /* Tiramisu */, 158 | ); 159 | }; 160 | /* End PBXProject section */ 161 | 162 | /* Begin PBXResourcesBuildPhase section */ 163 | 0945E3A421755E6300B2AB60 /* Resources */ = { 164 | isa = PBXResourcesBuildPhase; 165 | buildActionMask = 2147483647; 166 | files = ( 167 | 0945E3B421755E6400B2AB60 /* LaunchScreen.storyboard in Resources */, 168 | 0945E3B121755E6400B2AB60 /* Assets.xcassets in Resources */, 169 | 0945E3AF21755E6300B2AB60 /* Main.storyboard in Resources */, 170 | ); 171 | runOnlyForDeploymentPostprocessing = 0; 172 | }; 173 | /* End PBXResourcesBuildPhase section */ 174 | 175 | /* Begin PBXSourcesBuildPhase section */ 176 | 0945E3A221755E6300B2AB60 /* Sources */ = { 177 | isa = PBXSourcesBuildPhase; 178 | buildActionMask = 2147483647; 179 | files = ( 180 | 0945E3AC21755E6300B2AB60 /* ViewController.swift in Sources */, 181 | 09A102052176932300220459 /* Tiramisu45.mlmodel in Sources */, 182 | 0945E4182175811C00B2AB60 /* UIImage+CVPixelBuffer.swift in Sources */, 183 | 09DC0B592175B5690044D811 /* Popup.swift in Sources */, 184 | 0945E41B2175811C00B2AB60 /* MultiArray.swift in Sources */, 185 | 09DC0B562175B4090044D811 /* Probs.swift in Sources */, 186 | 0945E3AA21755E6300B2AB60 /* AppDelegate.swift in Sources */, 187 | 0945E4172175811C00B2AB60 /* Math.swift in Sources */, 188 | ); 189 | runOnlyForDeploymentPostprocessing = 0; 190 | }; 191 | /* End PBXSourcesBuildPhase section */ 192 | 193 | /* Begin PBXVariantGroup section */ 194 | 0945E3AD21755E6300B2AB60 /* Main.storyboard */ = { 195 | isa = PBXVariantGroup; 196 | children = ( 197 | 0945E3AE21755E6300B2AB60 /* Base */, 198 | ); 199 | name = Main.storyboard; 200 | sourceTree = ""; 201 | }; 202 | 0945E3B221755E6400B2AB60 /* LaunchScreen.storyboard */ = { 203 | isa = PBXVariantGroup; 204 | children = ( 205 | 0945E3B321755E6400B2AB60 /* Base */, 206 | ); 207 | name = LaunchScreen.storyboard; 208 | sourceTree = ""; 209 | }; 210 | /* End PBXVariantGroup section */ 211 | 212 | /* Begin XCBuildConfiguration section */ 213 | 0945E3B621755E6400B2AB60 /* Debug */ = { 214 | isa = XCBuildConfiguration; 215 | buildSettings = { 216 | ALWAYS_SEARCH_USER_PATHS = NO; 217 | CLANG_ANALYZER_NONNULL = YES; 218 | CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE; 219 | CLANG_CXX_LANGUAGE_STANDARD = "gnu++14"; 220 | CLANG_CXX_LIBRARY = "libc++"; 221 | CLANG_ENABLE_MODULES = YES; 222 | CLANG_ENABLE_OBJC_ARC = YES; 223 | CLANG_ENABLE_OBJC_WEAK = YES; 224 | CLANG_WARN_BLOCK_CAPTURE_AUTORELEASING = YES; 225 | CLANG_WARN_BOOL_CONVERSION = YES; 226 | CLANG_WARN_COMMA = YES; 227 | CLANG_WARN_CONSTANT_CONVERSION = YES; 228 | CLANG_WARN_DEPRECATED_OBJC_IMPLEMENTATIONS = YES; 229 | CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; 230 | CLANG_WARN_DOCUMENTATION_COMMENTS = YES; 231 | CLANG_WARN_EMPTY_BODY = YES; 232 | CLANG_WARN_ENUM_CONVERSION = YES; 233 | CLANG_WARN_INFINITE_RECURSION = YES; 234 | CLANG_WARN_INT_CONVERSION = YES; 235 | CLANG_WARN_NON_LITERAL_NULL_CONVERSION = YES; 236 | CLANG_WARN_OBJC_IMPLICIT_RETAIN_SELF = YES; 237 | CLANG_WARN_OBJC_LITERAL_CONVERSION = YES; 238 | CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; 239 | CLANG_WARN_RANGE_LOOP_ANALYSIS = YES; 240 | CLANG_WARN_STRICT_PROTOTYPES = YES; 241 | CLANG_WARN_SUSPICIOUS_MOVE = YES; 242 | CLANG_WARN_UNGUARDED_AVAILABILITY = YES_AGGRESSIVE; 243 | CLANG_WARN_UNREACHABLE_CODE = YES; 244 | CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; 245 | CODE_SIGN_IDENTITY = "iPhone Developer"; 246 | COPY_PHASE_STRIP = NO; 247 | DEBUG_INFORMATION_FORMAT = dwarf; 248 | ENABLE_STRICT_OBJC_MSGSEND = YES; 249 | ENABLE_TESTABILITY = YES; 250 | GCC_C_LANGUAGE_STANDARD = gnu11; 251 | GCC_DYNAMIC_NO_PIC = NO; 252 | GCC_NO_COMMON_BLOCKS = YES; 253 | GCC_OPTIMIZATION_LEVEL = 0; 254 | GCC_PREPROCESSOR_DEFINITIONS = ( 255 | "DEBUG=1", 256 | "$(inherited)", 257 | ); 258 | GCC_WARN_64_TO_32_BIT_CONVERSION = YES; 259 | GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; 260 | GCC_WARN_UNDECLARED_SELECTOR = YES; 261 | GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; 262 | GCC_WARN_UNUSED_FUNCTION = YES; 263 | GCC_WARN_UNUSED_VARIABLE = YES; 264 | IPHONEOS_DEPLOYMENT_TARGET = 12.0; 265 | MTL_ENABLE_DEBUG_INFO = INCLUDE_SOURCE; 266 | MTL_FAST_MATH = YES; 267 | ONLY_ACTIVE_ARCH = YES; 268 | SDKROOT = iphoneos; 269 | SWIFT_ACTIVE_COMPILATION_CONDITIONS = DEBUG; 270 | SWIFT_OPTIMIZATION_LEVEL = "-Onone"; 271 | }; 272 | name = Debug; 273 | }; 274 | 0945E3B721755E6400B2AB60 /* Release */ = { 275 | isa = XCBuildConfiguration; 276 | buildSettings = { 277 | ALWAYS_SEARCH_USER_PATHS = NO; 278 | CLANG_ANALYZER_NONNULL = YES; 279 | CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE; 280 | CLANG_CXX_LANGUAGE_STANDARD = "gnu++14"; 281 | CLANG_CXX_LIBRARY = "libc++"; 282 | CLANG_ENABLE_MODULES = YES; 283 | CLANG_ENABLE_OBJC_ARC = YES; 284 | CLANG_ENABLE_OBJC_WEAK = YES; 285 | CLANG_WARN_BLOCK_CAPTURE_AUTORELEASING = YES; 286 | CLANG_WARN_BOOL_CONVERSION = YES; 287 | CLANG_WARN_COMMA = YES; 288 | CLANG_WARN_CONSTANT_CONVERSION = YES; 289 | CLANG_WARN_DEPRECATED_OBJC_IMPLEMENTATIONS = YES; 290 | CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; 291 | CLANG_WARN_DOCUMENTATION_COMMENTS = YES; 292 | CLANG_WARN_EMPTY_BODY = YES; 293 | CLANG_WARN_ENUM_CONVERSION = YES; 294 | CLANG_WARN_INFINITE_RECURSION = YES; 295 | CLANG_WARN_INT_CONVERSION = YES; 296 | CLANG_WARN_NON_LITERAL_NULL_CONVERSION = YES; 297 | CLANG_WARN_OBJC_IMPLICIT_RETAIN_SELF = YES; 298 | CLANG_WARN_OBJC_LITERAL_CONVERSION = YES; 299 | CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; 300 | CLANG_WARN_RANGE_LOOP_ANALYSIS = YES; 301 | CLANG_WARN_STRICT_PROTOTYPES = YES; 302 | CLANG_WARN_SUSPICIOUS_MOVE = YES; 303 | CLANG_WARN_UNGUARDED_AVAILABILITY = YES_AGGRESSIVE; 304 | CLANG_WARN_UNREACHABLE_CODE = YES; 305 | CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; 306 | CODE_SIGN_IDENTITY = "iPhone Developer"; 307 | COPY_PHASE_STRIP = NO; 308 | DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym"; 309 | ENABLE_NS_ASSERTIONS = NO; 310 | ENABLE_STRICT_OBJC_MSGSEND = YES; 311 | GCC_C_LANGUAGE_STANDARD = gnu11; 312 | GCC_NO_COMMON_BLOCKS = YES; 313 | GCC_WARN_64_TO_32_BIT_CONVERSION = YES; 314 | GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; 315 | GCC_WARN_UNDECLARED_SELECTOR = YES; 316 | GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; 317 | GCC_WARN_UNUSED_FUNCTION = YES; 318 | GCC_WARN_UNUSED_VARIABLE = YES; 319 | IPHONEOS_DEPLOYMENT_TARGET = 12.0; 320 | MTL_ENABLE_DEBUG_INFO = NO; 321 | MTL_FAST_MATH = YES; 322 | SDKROOT = iphoneos; 323 | SWIFT_COMPILATION_MODE = wholemodule; 324 | SWIFT_OPTIMIZATION_LEVEL = "-O"; 325 | VALIDATE_PRODUCT = YES; 326 | }; 327 | name = Release; 328 | }; 329 | 0945E3B921755E6400B2AB60 /* Debug */ = { 330 | isa = XCBuildConfiguration; 331 | buildSettings = { 332 | ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; 333 | CODE_SIGN_STYLE = Automatic; 334 | DEVELOPMENT_TEAM = YLBWSHF35Z; 335 | INFOPLIST_FILE = Tiramisu/Info.plist; 336 | LD_RUNPATH_SEARCH_PATHS = ( 337 | "$(inherited)", 338 | "@executable_path/Frameworks", 339 | ); 340 | PRODUCT_BUNDLE_IDENTIFIER = com.github.kautenja.Tiramisu.Tiramisu; 341 | PRODUCT_NAME = "$(TARGET_NAME)"; 342 | SWIFT_VERSION = 4.2; 343 | TARGETED_DEVICE_FAMILY = "1,2"; 344 | }; 345 | name = Debug; 346 | }; 347 | 0945E3BA21755E6400B2AB60 /* Release */ = { 348 | isa = XCBuildConfiguration; 349 | buildSettings = { 350 | ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; 351 | CODE_SIGN_STYLE = Automatic; 352 | DEVELOPMENT_TEAM = YLBWSHF35Z; 353 | INFOPLIST_FILE = Tiramisu/Info.plist; 354 | LD_RUNPATH_SEARCH_PATHS = ( 355 | "$(inherited)", 356 | "@executable_path/Frameworks", 357 | ); 358 | PRODUCT_BUNDLE_IDENTIFIER = com.github.kautenja.Tiramisu.Tiramisu; 359 | PRODUCT_NAME = "$(TARGET_NAME)"; 360 | SWIFT_VERSION = 4.2; 361 | TARGETED_DEVICE_FAMILY = "1,2"; 362 | }; 363 | name = Release; 364 | }; 365 | /* End XCBuildConfiguration section */ 366 | 367 | /* Begin XCConfigurationList section */ 368 | 0945E3A121755E6300B2AB60 /* Build configuration list for PBXProject "Tiramisu" */ = { 369 | isa = XCConfigurationList; 370 | buildConfigurations = ( 371 | 0945E3B621755E6400B2AB60 /* Debug */, 372 | 0945E3B721755E6400B2AB60 /* Release */, 373 | ); 374 | defaultConfigurationIsVisible = 0; 375 | defaultConfigurationName = Release; 376 | }; 377 | 0945E3B821755E6400B2AB60 /* Build configuration list for PBXNativeTarget "Tiramisu" */ = { 378 | isa = XCConfigurationList; 379 | buildConfigurations = ( 380 | 0945E3B921755E6400B2AB60 /* Debug */, 381 | 0945E3BA21755E6400B2AB60 /* Release */, 382 | ); 383 | defaultConfigurationIsVisible = 0; 384 | defaultConfigurationName = Release; 385 | }; 386 | /* End XCConfigurationList section */ 387 | }; 388 | rootObject = 0945E39E21755E6300B2AB60 /* Project object */; 389 | } 390 | -------------------------------------------------------------------------------- /Tiramisu.xcodeproj/project.xcworkspace/contents.xcworkspacedata: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 8 | -------------------------------------------------------------------------------- /Tiramisu.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | IDEDidComputeMac32BitWarning 6 | 7 | 8 | 9 | -------------------------------------------------------------------------------- /Tiramisu/AppDelegate.swift: -------------------------------------------------------------------------------- 1 | // 2 | // AppDelegate.swift 3 | // Tiramisu 4 | // 5 | // Created by James Kauten on 10/15/18. 6 | // Copyright © 2018 Kautenja. All rights reserved. 7 | // 8 | 9 | import UIKit 10 | 11 | @UIApplicationMain 12 | class AppDelegate: UIResponder, UIApplicationDelegate { 13 | 14 | var window: UIWindow? 15 | 16 | 17 | func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool { 18 | // Override point for customization after application launch. 19 | return true 20 | } 21 | 22 | func applicationWillResignActive(_ application: UIApplication) { 23 | // Sent when the application is about to move from active to inactive state. This can occur for certain types of temporary interruptions (such as an incoming phone call or SMS message) or when the user quits the application and it begins the transition to the background state. 24 | // Use this method to pause ongoing tasks, disable timers, and invalidate graphics rendering callbacks. Games should use this method to pause the game. 25 | } 26 | 27 | func applicationDidEnterBackground(_ application: UIApplication) { 28 | // Use this method to release shared resources, save user data, invalidate timers, and store enough application state information to restore your application to its current state in case it is terminated later. 29 | // If your application supports background execution, this method is called instead of applicationWillTerminate: when the user quits. 30 | } 31 | 32 | func applicationWillEnterForeground(_ application: UIApplication) { 33 | // Called as part of the transition from the background to the active state; here you can undo many of the changes made on entering the background. 34 | } 35 | 36 | func applicationDidBecomeActive(_ application: UIApplication) { 37 | // Restart any tasks that were paused (or not yet started) while the application was inactive. If the application was previously in the background, optionally refresh the user interface. 38 | } 39 | 40 | func applicationWillTerminate(_ application: UIApplication) { 41 | // Called when the application is about to terminate. Save data if appropriate. See also applicationDidEnterBackground:. 42 | } 43 | 44 | 45 | } 46 | 47 | -------------------------------------------------------------------------------- /Tiramisu/Assets.xcassets/AppIcon.appiconset/Contents.json: -------------------------------------------------------------------------------- 1 | { 2 | "images" : [ 3 | { 4 | "idiom" : "iphone", 5 | "size" : "20x20", 6 | "scale" : "2x" 7 | }, 8 | { 9 | "idiom" : "iphone", 10 | "size" : "20x20", 11 | "scale" : "3x" 12 | }, 13 | { 14 | "idiom" : "iphone", 15 | "size" : "29x29", 16 | "scale" : "2x" 17 | }, 18 | { 19 | "idiom" : "iphone", 20 | "size" : "29x29", 21 | "scale" : "3x" 22 | }, 23 | { 24 | "idiom" : "iphone", 25 | "size" : "40x40", 26 | "scale" : "2x" 27 | }, 28 | { 29 | "idiom" : "iphone", 30 | "size" : "40x40", 31 | "scale" : "3x" 32 | }, 33 | { 34 | "idiom" : "iphone", 35 | "size" : "60x60", 36 | "scale" : "2x" 37 | }, 38 | { 39 | "idiom" : "iphone", 40 | "size" : "60x60", 41 | "scale" : "3x" 42 | }, 43 | { 44 | "idiom" : "ipad", 45 | "size" : "20x20", 46 | "scale" : "1x" 47 | }, 48 | { 49 | "idiom" : "ipad", 50 | "size" : "20x20", 51 | "scale" : "2x" 52 | }, 53 | { 54 | "idiom" : "ipad", 55 | "size" : "29x29", 56 | "scale" : "1x" 57 | }, 58 | { 59 | "idiom" : "ipad", 60 | "size" : "29x29", 61 | "scale" : "2x" 62 | }, 63 | { 64 | "idiom" : "ipad", 65 | "size" : "40x40", 66 | "scale" : "1x" 67 | }, 68 | { 69 | "idiom" : "ipad", 70 | "size" : "40x40", 71 | "scale" : "2x" 72 | }, 73 | { 74 | "idiom" : "ipad", 75 | "size" : "76x76", 76 | "scale" : "1x" 77 | }, 78 | { 79 | "idiom" : "ipad", 80 | "size" : "76x76", 81 | "scale" : "2x" 82 | }, 83 | { 84 | "idiom" : "ipad", 85 | "size" : "83.5x83.5", 86 | "scale" : "2x" 87 | }, 88 | { 89 | "idiom" : "ios-marketing", 90 | "size" : "1024x1024", 91 | "scale" : "1x" 92 | } 93 | ], 94 | "info" : { 95 | "version" : 1, 96 | "author" : "xcode" 97 | } 98 | } -------------------------------------------------------------------------------- /Tiramisu/Assets.xcassets/Contents.json: -------------------------------------------------------------------------------- 1 | { 2 | "info" : { 3 | "version" : 1, 4 | "author" : "xcode" 5 | } 6 | } -------------------------------------------------------------------------------- /Tiramisu/Base.lproj/LaunchScreen.storyboard: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | -------------------------------------------------------------------------------- /Tiramisu/Base.lproj/Main.storyboard: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 43 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | -------------------------------------------------------------------------------- /Tiramisu/CoreMLHelpers/Math.swift: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2017 M.I. Hollemans 3 | 4 | Permission is hereby granted, free of charge, to any person obtaining a copy 5 | of this software and associated documentation files (the "Software"), to 6 | deal in the Software without restriction, including without limitation the 7 | rights to use, copy, modify, merge, publish, distribute, sublicense, and/or 8 | sell copies of the Software, and to permit persons to whom the Software is 9 | furnished to do so, subject to the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be included in 12 | all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 15 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 16 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 17 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 18 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 19 | FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 20 | IN THE SOFTWARE. 21 | */ 22 | 23 | import Foundation 24 | 25 | public func clamp(_ x: T, min: T, max: T) -> T { 26 | if x < min { return min } 27 | if x > max { return max } 28 | return x 29 | } 30 | -------------------------------------------------------------------------------- /Tiramisu/CoreMLHelpers/MultiArray.swift: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2017 M.I. Hollemans 3 | 4 | Permission is hereby granted, free of charge, to any person obtaining a copy 5 | of this software and associated documentation files (the "Software"), to 6 | deal in the Software without restriction, including without limitation the 7 | rights to use, copy, modify, merge, publish, distribute, sublicense, and/or 8 | sell copies of the Software, and to permit persons to whom the Software is 9 | furnished to do so, subject to the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be included in 12 | all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 15 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 16 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 17 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 18 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 19 | FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 20 | IN THE SOFTWARE. 21 | */ 22 | 23 | import Foundation 24 | import UIKit 25 | import CoreML 26 | import Swift 27 | 28 | public protocol MultiArrayType: Comparable { 29 | static var multiArrayDataType: MLMultiArrayDataType { get } 30 | static func +(lhs: Self, rhs: Self) -> Self 31 | static func *(lhs: Self, rhs: Self) -> Self 32 | init(_: Int) 33 | var toUInt8: UInt8 { get } 34 | } 35 | 36 | extension Double: MultiArrayType { 37 | public static var multiArrayDataType: MLMultiArrayDataType { return .double } 38 | public var toUInt8: UInt8 { return UInt8(self) } 39 | } 40 | 41 | extension Float: MultiArrayType { 42 | public static var multiArrayDataType: MLMultiArrayDataType { return .float32 } 43 | public var toUInt8: UInt8 { return UInt8(self) } 44 | } 45 | 46 | extension Int32: MultiArrayType { 47 | public static var multiArrayDataType: MLMultiArrayDataType { return .int32 } 48 | public var toUInt8: UInt8 { return UInt8(self) } 49 | } 50 | 51 | /** 52 | Wrapper around MLMultiArray to make it more Swifty. 53 | */ 54 | public struct MultiArray { 55 | public let array: MLMultiArray 56 | public let pointer: UnsafeMutablePointer 57 | 58 | private(set) public var strides: [Int] 59 | private(set) public var shape: [Int] 60 | 61 | /** 62 | Creates a new multi-array filled with all zeros. 63 | */ 64 | public init(shape: [Int]) { 65 | let m = try! MLMultiArray(shape: shape as [NSNumber], dataType: T.multiArrayDataType) 66 | self.init(m) 67 | memset(pointer, 0, MemoryLayout.stride * count) 68 | } 69 | 70 | /** 71 | Creates a new multi-array initialized with the specified value. 72 | */ 73 | public init(shape: [Int], initial: T) { 74 | self.init(shape: shape) 75 | for i in 0..(OpaquePointer(array.dataPointer)) 92 | } 93 | 94 | /** 95 | Returns the number of elements in the entire array. 96 | */ 97 | public var count: Int { 98 | return shape.reduce(1, *) 99 | } 100 | 101 | public subscript(a: Int) -> T { 102 | get { return pointer[a] } 103 | set { pointer[a] = newValue } 104 | } 105 | 106 | public subscript(a: Int, b: Int) -> T { 107 | get { return pointer[a*strides[0] + b*strides[1]] } 108 | set { pointer[a*strides[0] + b*strides[1]] = newValue } 109 | } 110 | 111 | public subscript(a: Int, b: Int, c: Int) -> T { 112 | get { return pointer[a*strides[0] + b*strides[1] + c*strides[2]] } 113 | set { pointer[a*strides[0] + b*strides[1] + c*strides[2]] = newValue } 114 | } 115 | 116 | public subscript(a: Int, b: Int, c: Int, d: Int) -> T { 117 | get { return pointer[a*strides[0] + b*strides[1] + c*strides[2] + d*strides[3]] } 118 | set { pointer[a*strides[0] + b*strides[1] + c*strides[2] + d*strides[3]] = newValue } 119 | } 120 | 121 | public subscript(a: Int, b: Int, c: Int, d: Int, e: Int) -> T { 122 | get { return pointer[a*strides[0] + b*strides[1] + c*strides[2] + d*strides[3] + e*strides[4]] } 123 | set { pointer[a*strides[0] + b*strides[1] + c*strides[2] + d*strides[3] + e*strides[4]] = newValue } 124 | } 125 | 126 | public subscript(indices: [Int]) -> T { 127 | get { return pointer[offset(for: indices)] } 128 | set { pointer[offset(for: indices)] = newValue } 129 | } 130 | 131 | func offset(for indices: [Int]) -> Int { 132 | var offset = 0 133 | for i in 0.. MultiArray { 144 | precondition(order.count == strides.count) 145 | var newShape = shape 146 | var newStrides = strides 147 | for i in 0.. MultiArray { 158 | let newCount = dimensions.reduce(1, *) 159 | precondition(newCount == count, "Cannot reshape \(shape) to \(dimensions)") 160 | 161 | var newStrides = [Int](repeating: 0, count: dimensions.count) 162 | newStrides[dimensions.count - 1] = 1 163 | for i in stride(from: dimensions.count - 1, to: 0, by: -1) { 164 | newStrides[i - 1] = newStrides[i] * dimensions[i] 165 | } 166 | 167 | return MultiArray(array, dimensions, newStrides) 168 | } 169 | } 170 | 171 | extension MultiArray: CustomStringConvertible { 172 | public var description: String { 173 | return description([]) 174 | } 175 | 176 | func description(_ indices: [Int]) -> String { 177 | func indent(_ x: Int) -> String { 178 | return String(repeating: " ", count: x) 179 | } 180 | 181 | // This function is called recursively for every dimension. 182 | // Add an entry for this dimension to the end of the array. 183 | var indices = indices + [0] 184 | 185 | let d = indices.count - 1 // the current dimension 186 | let N = shape[d] // how many elements in this dimension 187 | 188 | var s = "[" 189 | if indices.count < shape.count { // not last dimension yet? 190 | for i in 0.. UIImage? { 227 | if shape.count == 3, let (b, w, h) = toRawBytesRGBA(offset: offset, scale: scale) { 228 | return UIImage.fromByteArrayRGBA(b, width: w, height: h) 229 | } else if shape.count == 2, let (b, w, h) = toRawBytesGray(offset: offset, scale: scale) { 230 | return UIImage.fromByteArrayGray(b, width: w, height: h) 231 | } else { 232 | return nil 233 | } 234 | } 235 | 236 | /** 237 | Converts the multi-array into an array of RGBA pixels. 238 | 239 | - Note: The multi-array must have shape (3, height, width). If your array 240 | has a different shape, use `reshape()` or `transpose()` first. 241 | */ 242 | public func toRawBytesRGBA(offset: T, scale: T) 243 | -> (bytes: [UInt8], width: Int, height: Int)? { 244 | guard shape.count == 3 else { 245 | print("Expected a multi-array with 3 dimensions, got \(shape)") 246 | return nil 247 | } 248 | guard shape[0] == 3 else { 249 | print("Expected first dimension to have 3 channels, got \(shape[0])") 250 | return nil 251 | } 252 | 253 | let height = shape[1] 254 | let width = shape[2] 255 | var bytes = [UInt8](repeating: 0, count: height * width * 4) 256 | 257 | for h in 0.. (bytes: [UInt8], width: Int, height: Int)? { 281 | guard shape.count == 2 else { 282 | print("Expected a multi-array with 2 dimensions, got \(shape)") 283 | return nil 284 | } 285 | 286 | let height = shape[0] 287 | let width = shape[1] 288 | var bytes = [UInt8](repeating: 0, count: height * width) 289 | 290 | for h in 0.. UIImage? { 307 | guard shape.count == 3 else { 308 | print("Expected a multi-array with 3 dimensions, got \(shape)") 309 | return nil 310 | } 311 | guard channel >= 0 && channel < shape[0] else { 312 | print("Channel must be between 0 and \(shape[0] - 1)") 313 | return nil 314 | } 315 | 316 | let height = shape[1] 317 | let width = shape[2] 318 | var a = MultiArray(shape: [height, width]) 319 | for y in 0.. UIImage? { 15 | // TODO: dynamically load a label map instead of hard coding 16 | // can this bonus data be included in the model file? 17 | let label_map = [ 18 | 0: [255, 0, 0], 19 | 1: [70, 70, 70], 20 | 2: [0, 0, 142], 21 | 3: [153, 153, 153], 22 | 4: [190, 153, 153], 23 | 5: [220, 20, 60], 24 | 6: [128, 64, 128], 25 | 7: [244, 35, 232], 26 | 8: [220, 220, 0], 27 | 9: [70, 130, 180], 28 | 10: [107, 142, 35], 29 | 11: [0, 0, 0] 30 | ] 31 | // convert the MLMultiArray to a MultiArray 32 | var codes = MultiArray(_probs) 33 | // get the shape information from the probs 34 | let height = codes.shape[1] 35 | let width = codes.shape[2] 36 | // initialize some bytes to store the image in 37 | var bytes = [UInt8](repeating: 255, count: height * width * 4) 38 | // iterate over the pixels in the output probs 39 | for h in 0 ..< height { 40 | for w in 0 ..< width { 41 | // get the array offset for this word 42 | let offset = h * width * 4 + w * 4 43 | // get the RGB value for the highest probability class 44 | let rgb = label_map[Int(codes[0, h, w])] 45 | // set the bytes to the RGB value and alpha of 1.0 (255) 46 | bytes[offset + 0] = UInt8(rgb![0]) 47 | bytes[offset + 1] = UInt8(rgb![1]) 48 | bytes[offset + 2] = UInt8(rgb![2]) 49 | } 50 | } 51 | // create a UIImage from the byte array 52 | return UIImage.fromByteArray(bytes, width: width, height: height, 53 | scale: 0, orientation: .up, 54 | bytesPerRow: width * 4, 55 | colorSpace: CGColorSpaceCreateDeviceRGB(), 56 | alphaInfo: .premultipliedLast) 57 | } 58 | -------------------------------------------------------------------------------- /Tiramisu/CoreMLHelpers/UIImage+CVPixelBuffer.swift: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2017 M.I. Hollemans 3 | 4 | Permission is hereby granted, free of charge, to any person obtaining a copy 5 | of this software and associated documentation files (the "Software"), to 6 | deal in the Software without restriction, including without limitation the 7 | rights to use, copy, modify, merge, publish, distribute, sublicense, and/or 8 | sell copies of the Software, and to permit persons to whom the Software is 9 | furnished to do so, subject to the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be included in 12 | all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 15 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 16 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 17 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 18 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 19 | FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 20 | IN THE SOFTWARE. 21 | */ 22 | 23 | import UIKit 24 | import VideoToolbox 25 | 26 | extension UIImage { 27 | /** 28 | Resizes the image to width x height and converts it to an RGB CVPixelBuffer. 29 | */ 30 | public func pixelBuffer(width: Int, height: Int) -> CVPixelBuffer? { 31 | return pixelBuffer(width: width, height: height, 32 | pixelFormatType: kCVPixelFormatType_32ARGB, 33 | colorSpace: CGColorSpaceCreateDeviceRGB(), 34 | alphaInfo: .noneSkipFirst) 35 | } 36 | 37 | /** 38 | Resizes the image to width x height and converts it to a grayscale CVPixelBuffer. 39 | */ 40 | public func pixelBufferGray(width: Int, height: Int) -> CVPixelBuffer? { 41 | return pixelBuffer(width: width, height: height, 42 | pixelFormatType: kCVPixelFormatType_OneComponent8, 43 | colorSpace: CGColorSpaceCreateDeviceGray(), 44 | alphaInfo: .none) 45 | } 46 | 47 | func pixelBuffer(width: Int, height: Int, pixelFormatType: OSType, 48 | colorSpace: CGColorSpace, alphaInfo: CGImageAlphaInfo) -> CVPixelBuffer? { 49 | var maybePixelBuffer: CVPixelBuffer? 50 | let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue, 51 | kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] 52 | let status = CVPixelBufferCreate(kCFAllocatorDefault, 53 | width, 54 | height, 55 | pixelFormatType, 56 | attrs as CFDictionary, 57 | &maybePixelBuffer) 58 | 59 | guard status == kCVReturnSuccess, let pixelBuffer = maybePixelBuffer else { 60 | return nil 61 | } 62 | 63 | CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0)) 64 | let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer) 65 | 66 | guard let context = CGContext(data: pixelData, 67 | width: width, 68 | height: height, 69 | bitsPerComponent: 8, 70 | bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer), 71 | space: colorSpace, 72 | bitmapInfo: alphaInfo.rawValue) 73 | else { 74 | return nil 75 | } 76 | 77 | UIGraphicsPushContext(context) 78 | context.translateBy(x: 0, y: CGFloat(height)) 79 | context.scaleBy(x: 1, y: -1) 80 | self.draw(in: CGRect(x: 0, y: 0, width: width, height: height)) 81 | UIGraphicsPopContext() 82 | 83 | CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0)) 84 | return pixelBuffer 85 | } 86 | } 87 | 88 | extension UIImage { 89 | /** 90 | Creates a new UIImage from a CVPixelBuffer. 91 | NOTE: This only works for RGB pixel buffers, not for grayscale. 92 | */ 93 | public convenience init?(pixelBuffer: CVPixelBuffer) { 94 | var cgImage: CGImage? 95 | VTCreateCGImageFromCVPixelBuffer(pixelBuffer, options: nil, imageOut: &cgImage) 96 | 97 | if let cgImage = cgImage { 98 | self.init(cgImage: cgImage) 99 | } else { 100 | return nil 101 | } 102 | } 103 | 104 | /** 105 | Creates a new UIImage from a CVPixelBuffer, using Core Image. 106 | */ 107 | public convenience init?(pixelBuffer: CVPixelBuffer, context: CIContext) { 108 | let ciImage = CIImage(cvPixelBuffer: pixelBuffer) 109 | let rect = CGRect(x: 0, y: 0, width: CVPixelBufferGetWidth(pixelBuffer), 110 | height: CVPixelBufferGetHeight(pixelBuffer)) 111 | if let cgImage = context.createCGImage(ciImage, from: rect) { 112 | self.init(cgImage: cgImage) 113 | } else { 114 | return nil 115 | } 116 | } 117 | } 118 | 119 | extension UIImage { 120 | /** 121 | Creates a new UIImage from an array of RGBA bytes. 122 | */ 123 | @nonobjc public class func fromByteArrayRGBA(_ bytes: [UInt8], 124 | width: Int, 125 | height: Int, 126 | scale: CGFloat = 0, 127 | orientation: UIImage.Orientation = .up) -> UIImage? { 128 | return fromByteArray(bytes, width: width, height: height, 129 | scale: scale, orientation: orientation, 130 | bytesPerRow: width * 4, 131 | colorSpace: CGColorSpaceCreateDeviceRGB(), 132 | alphaInfo: .premultipliedLast) 133 | } 134 | 135 | /** 136 | Creates a new UIImage from an array of grayscale bytes. 137 | */ 138 | @nonobjc public class func fromByteArrayGray(_ bytes: [UInt8], 139 | width: Int, 140 | height: Int, 141 | scale: CGFloat = 0, 142 | orientation: UIImage.Orientation = .up) -> UIImage? { 143 | return fromByteArray(bytes, width: width, height: height, 144 | scale: scale, orientation: orientation, 145 | bytesPerRow: width, 146 | colorSpace: CGColorSpaceCreateDeviceGray(), 147 | alphaInfo: .none) 148 | } 149 | 150 | @nonobjc class func fromByteArray(_ bytes: [UInt8], 151 | width: Int, 152 | height: Int, 153 | scale: CGFloat, 154 | orientation: UIImage.Orientation, 155 | bytesPerRow: Int, 156 | colorSpace: CGColorSpace, 157 | alphaInfo: CGImageAlphaInfo) -> UIImage? { 158 | var image: UIImage? 159 | bytes.withUnsafeBytes { ptr in 160 | if let context = CGContext(data: UnsafeMutableRawPointer(mutating: ptr.baseAddress!), 161 | width: width, 162 | height: height, 163 | bitsPerComponent: 8, 164 | bytesPerRow: bytesPerRow, 165 | space: colorSpace, 166 | bitmapInfo: alphaInfo.rawValue), 167 | let cgImage = context.makeImage() { 168 | image = UIImage(cgImage: cgImage, scale: scale, orientation: orientation) 169 | } 170 | } 171 | return image 172 | } 173 | } 174 | -------------------------------------------------------------------------------- /Tiramisu/Info.plist: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | CFBundleDevelopmentRegion 6 | $(DEVELOPMENT_LANGUAGE) 7 | CFBundleExecutable 8 | $(EXECUTABLE_NAME) 9 | CFBundleIdentifier 10 | $(PRODUCT_BUNDLE_IDENTIFIER) 11 | CFBundleInfoDictionaryVersion 12 | 6.0 13 | CFBundleName 14 | $(PRODUCT_NAME) 15 | CFBundlePackageType 16 | APPL 17 | CFBundleShortVersionString 18 | 1.0 19 | CFBundleVersion 20 | 1 21 | LSRequiresIPhoneOS 22 | 23 | NSCameraUsageDescription 24 | Tiramisu needs to use the camera to segment images. 25 | UILaunchStoryboardName 26 | LaunchScreen 27 | UIMainStoryboardFile 28 | Main 29 | UIRequiredDeviceCapabilities 30 | 31 | armv7 32 | 33 | UIStatusBarStyle 34 | UIStatusBarStyleDefault 35 | UISupportedInterfaceOrientations 36 | 37 | UIInterfaceOrientationLandscapeRight 38 | 39 | UISupportedInterfaceOrientations~ipad 40 | 41 | UIInterfaceOrientationPortrait 42 | UIInterfaceOrientationPortraitUpsideDown 43 | UIInterfaceOrientationLandscapeLeft 44 | UIInterfaceOrientationLandscapeRight 45 | 46 | 47 | 48 | -------------------------------------------------------------------------------- /Tiramisu/Models/Tiramisu45.h5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Kautenja/ios-semantic-segmentation/5bd69ed63ce2be2b52b66ba1aa3a43c26f3f9900/Tiramisu/Models/Tiramisu45.h5 -------------------------------------------------------------------------------- /Tiramisu/Models/Tiramisu45.mlmodel: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Kautenja/ios-semantic-segmentation/5bd69ed63ce2be2b52b66ba1aa3a43c26f3f9900/Tiramisu/Models/Tiramisu45.mlmodel -------------------------------------------------------------------------------- /Tiramisu/Models/convert.py: -------------------------------------------------------------------------------- 1 | """A script to convert a Keras vision model to a CoreML model.""" 2 | import sys 3 | import os 4 | import coremltools 5 | 6 | 7 | # try to unwrap the path to the weights 8 | try: 9 | weights = sys.argv[1] 10 | # create an output file using the same name as input with new extension 11 | output_file = weights.replace('.h5', '.mlmodel') 12 | except IndexError: 13 | print(__doc__) 14 | 15 | 16 | # load the CoreML model from the Keras model 17 | coreml_model = coremltools.converters.keras.convert(weights, 18 | input_names='image', 19 | image_input_names='image', 20 | output_names='segmentation', 21 | image_scale=1/255.0, 22 | ) 23 | 24 | 25 | # setup the attribution meta-data for the model 26 | coreml_model.author = 'Kautenja' 27 | coreml_model.license = 'MIT' 28 | coreml_model.short_description = '45 Layers Tiramisu Semantic Segmentation Model trained on CamVid & CityScapes.' 29 | coreml_model.input_description['image'] = 'An input image in RGB order' 30 | coreml_model.output_description['segmentation'] = 'The segmentation map as the Softmax output' 31 | 32 | 33 | # get the spec from the model 34 | spec = coreml_model.get_spec() 35 | # create a local reference to the Float32 type 36 | Float32 = coremltools.proto.FeatureTypes_pb2.ArrayFeatureType.FLOAT32 37 | # set the output shape for the segmentation to Float32 38 | spec.description.output[0].type.multiArrayType.dataType = Float32 39 | # save the spec to disk 40 | coremltools.utils.save_spec(spec, output_file) 41 | -------------------------------------------------------------------------------- /Tiramisu/UIKitHelpers/Popup.swift: -------------------------------------------------------------------------------- 1 | // 2 | // Popup.swift 3 | // Tiramisu 4 | // 5 | // Created by James Kauten on 10/16/18. 6 | // Copyright © 2018 Kautenja. All rights reserved. 7 | // 8 | 9 | import Foundation 10 | import UIKit 11 | 12 | /// Display a popup on an input view controller with title and message. 13 | /// Args: 14 | /// vc: the view controller to display the popup on 15 | /// title: the title of the popup to display 16 | /// message: the message for the popup alert 17 | /// 18 | func popup_alert(_ vc: ViewController, title: String, message: String) { 19 | // create an alert view controller with given title and message 20 | let alert = UIAlertController(title: title, message: message, preferredStyle: .alert) 21 | // create the acknowledgement action for the popup 22 | let alertAction = UIAlertAction(title: "OK", style: .default) 23 | // add the action to the popup view controller 24 | alert.addAction(alertAction) 25 | // present the popup on the input view controller 26 | vc.present(alert, animated: true) 27 | } 28 | -------------------------------------------------------------------------------- /Tiramisu/ViewController.swift: -------------------------------------------------------------------------------- 1 | // 2 | // ViewController.swift 3 | // Tiramisu 4 | // 5 | // Created by James Kauten on 10/15/18. 6 | // Copyright © 2018 Kautenja. All rights reserved. 7 | // 8 | 9 | import UIKit 10 | import AVFoundation 11 | import Vision 12 | import Metal 13 | import MetalPerformanceShaders 14 | 15 | /// A view controller to pass camera inputs through a vision model 16 | class ViewController: UIViewController, AVCaptureVideoDataOutputSampleBufferDelegate { 17 | 18 | /// a local reference to time to update the framerate 19 | var time = Date() 20 | 21 | var ready: Bool = true 22 | 23 | /// the view to preview raw RGB data from the camera 24 | @IBOutlet weak var preview: UIView! 25 | /// the view for showing the segmentation 26 | @IBOutlet weak var segmentation: UIImageView! 27 | /// a label to show the framerate of the model 28 | @IBOutlet weak var framerate: UILabel! 29 | 30 | /// the camera session for streaming data from the camera 31 | var captureSession: AVCaptureSession! 32 | /// the video preview layer 33 | var videoPreviewLayer: AVCaptureVideoPreviewLayer! 34 | 35 | /// TODO: 36 | private var _device: MTLDevice? 37 | /// TODO: 38 | var device: MTLDevice! { 39 | get { 40 | // try to unwrap the private device instance 41 | if let device = _device { 42 | return device 43 | } 44 | _device = MTLCreateSystemDefaultDevice() 45 | return _device 46 | } 47 | } 48 | 49 | var _queue: MTLCommandQueue? 50 | 51 | var queue: MTLCommandQueue! { 52 | get { 53 | // try to unwrap the private queue instance 54 | if let queue = _queue { 55 | return queue 56 | } 57 | _queue = device.makeCommandQueue() 58 | return _queue 59 | } 60 | } 61 | 62 | /// the model for the view controller to apss camera data through 63 | private var _model: VNCoreMLModel? 64 | /// the model for the view controller to apss camera data through 65 | var model: VNCoreMLModel! { 66 | get { 67 | // try to unwrap the private model instance 68 | if let model = _model { 69 | return model 70 | } 71 | // try to create a new model and fail gracefully 72 | do { 73 | _model = try VNCoreMLModel(for: Tiramisu45().model) 74 | } catch let error { 75 | let message = "failed to load model: \(error.localizedDescription)" 76 | popup_alert(self, title: "Model Error", message: message) 77 | } 78 | return _model 79 | } 80 | } 81 | 82 | /// the request and handler for the model 83 | private var _request: VNCoreMLRequest? 84 | /// the request and handler for the model 85 | var request: VNCoreMLRequest! { 86 | get { 87 | // try to unwrap the private request instance 88 | if let request = _request { 89 | return request 90 | } 91 | // create the request 92 | _request = VNCoreMLRequest(model: model) { (finishedRequest, error) in 93 | // handle an error from the inference engine 94 | if let error = error { 95 | print("inference error: \(error.localizedDescription)") 96 | return 97 | } 98 | // make sure the UI is ready for another frame 99 | guard self.ready else { return } 100 | // get the outputs from the model 101 | let outputs = finishedRequest.results as? [VNCoreMLFeatureValueObservation] 102 | // get the probabilities as the first output of the model 103 | guard let softmax = outputs?[0].featureValue.multiArrayValue else { 104 | print("failed to extract output from model") 105 | return 106 | } 107 | // get the dimensions of the probability tensor 108 | let channels = softmax.shape[0].intValue 109 | let height = softmax.shape[1].intValue 110 | let width = softmax.shape[2].intValue 111 | 112 | // create an image for the softmax outputs 113 | let desc = MPSImageDescriptor(channelFormat: .float32, 114 | width: width, 115 | height: height, 116 | featureChannels: channels) 117 | let probs = MPSImage(device: self.device, imageDescriptor: desc) 118 | probs.writeBytes(softmax.dataPointer, 119 | dataLayout: .featureChannelsxHeightxWidth, 120 | imageIndex: 0) 121 | 122 | // create an output image for the Arg Max output 123 | let desc1 = MPSImageDescriptor(channelFormat: .float32, 124 | width: width, 125 | height: height, 126 | featureChannels: 1) 127 | let classes = MPSImage(device: self.device, imageDescriptor: desc1) 128 | 129 | // create a buffer and pass the inputs through the filter to the outputs 130 | let buffer = self.queue.makeCommandBuffer() 131 | let filter = MPSNNReduceFeatureChannelsArgumentMax(device: self.device) 132 | filter.encode(commandBuffer: buffer!, sourceImage: probs, destinationImage: classes) 133 | 134 | // add a callback to handle the buffer's completion and commit the buffer 135 | buffer?.addCompletedHandler({ (_buffer) in 136 | let argmax = try! MLMultiArray(shape: [1, softmax.shape[1], softmax.shape[2]], dataType: .float32) 137 | classes.readBytes(argmax.dataPointer, 138 | dataLayout: .featureChannelsxHeightxWidth, 139 | imageIndex: 0) 140 | 141 | // unmap the discrete segmentation to RGB pixels 142 | let image = codesToImage(argmax) 143 | // update the image on the UI thread 144 | DispatchQueue.main.async { 145 | self.segmentation.image = image 146 | let fps = -1 / self.time.timeIntervalSinceNow 147 | self.time = Date() 148 | self.framerate.text = "\(fps)" 149 | } 150 | self.ready = true 151 | }) 152 | self.ready = false 153 | buffer?.commit() 154 | 155 | } 156 | // set the input image size to be a scaled version 157 | // of the image 158 | _request?.imageCropAndScaleOption = .scaleFill 159 | return _request 160 | } 161 | } 162 | 163 | /// Respond to a memory warning from the OS 164 | override func didReceiveMemoryWarning() { 165 | super.didReceiveMemoryWarning() 166 | popup_alert(self, title: "Memory Warning", message: "received memory warning") 167 | } 168 | 169 | /// Handle the view appearing 170 | override func viewDidAppear(_ animated: Bool) { 171 | super.viewDidAppear(animated) 172 | // setup the AV session 173 | captureSession = AVCaptureSession() 174 | captureSession.sessionPreset = .hd1280x720 175 | // get a handle on the back camera 176 | guard let camera = AVCaptureDevice.default(for: AVMediaType.video) else { 177 | let message = "Unable to access the back camera!" 178 | popup_alert(self, title: "Camera Error", message: message) 179 | return 180 | } 181 | // create an input device from the back camera and handle 182 | // any errors (i.e., privacy request denied) 183 | do { 184 | // setup the camera input and video output 185 | let input = try AVCaptureDeviceInput(device: camera) 186 | let videoOutput = AVCaptureVideoDataOutput() 187 | videoOutput.setSampleBufferDelegate(self, queue: DispatchQueue(label: "videoQueue")) 188 | // add the inputs and ouptuts to the sessionr and start the preview 189 | if captureSession.canAddInput(input) && captureSession.canAddOutput(videoOutput) { 190 | captureSession.addInput(input) 191 | captureSession.addOutput(videoOutput) 192 | setupCameraPreview() 193 | } 194 | } 195 | catch let error { 196 | let message = "failed to intialize camera: \(error.localizedDescription)" 197 | popup_alert(self, title: "Camera Error", message: message) 198 | return 199 | } 200 | } 201 | 202 | /// Setup the live preview from the camera 203 | func setupCameraPreview() { 204 | // create a video preview layer for the view controller 205 | videoPreviewLayer = AVCaptureVideoPreviewLayer(session: captureSession) 206 | // set the metadata of the video preview 207 | videoPreviewLayer.videoGravity = .resizeAspect 208 | videoPreviewLayer.connection?.videoOrientation = .landscapeRight 209 | // add the preview layer as a sublayer of the preview view 210 | preview.layer.addSublayer(videoPreviewLayer) 211 | // start the capture session asyncrhonously 212 | DispatchQueue.global(qos: .userInitiated).async { 213 | // start the capture session in the background thread 214 | self.captureSession.startRunning() 215 | // set the frame of the video preview to the bounds of the 216 | // preview view 217 | DispatchQueue.main.async { 218 | self.videoPreviewLayer.frame = self.preview.bounds 219 | } 220 | } 221 | } 222 | 223 | /// Handle a frame from the camera video stream 224 | func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { 225 | // create a Core Video pixel buffer which is an image buffer that holds pixels in main memory 226 | // Applications generating frames, compressing or decompressing video, or using Core Image 227 | // can all make use of Core Video pixel buffers 228 | guard let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { 229 | let message = "failed to create pixel buffer from video input" 230 | popup_alert(self, title: "Inference Error", message: message) 231 | return 232 | } 233 | // execute the request 234 | do { 235 | try VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:]).perform([request]) 236 | } catch let error { 237 | let message = "failed to perform inference: \(error.localizedDescription)" 238 | popup_alert(self, title: "Inference Error", message: message) 239 | } 240 | } 241 | 242 | } 243 | -------------------------------------------------------------------------------- /img/cmap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Kautenja/ios-semantic-segmentation/5bd69ed63ce2be2b52b66ba1aa3a43c26f3f9900/img/cmap.png -------------------------------------------------------------------------------- /img/example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Kautenja/ios-semantic-segmentation/5bd69ed63ce2be2b52b66ba1aa3a43c26f3f9900/img/example.png --------------------------------------------------------------------------------