2 |
2 |
26 |
27 | [Reality-UI](https://github.com/maxxfrazer/RealityUI) - a framework for building user interfaces in Augmented Reality
28 |
29 | ----
30 |
31 | 
32 |
33 |
34 | [ARKit-CoreLocation framework](https://github.com/ProjectDent/ARKit-CoreLocation)
35 |
36 | ----
37 |
38 | 
39 |
40 | [CoreML-ARKit](https://github.com/hanleyweng/CoreML-in-ARKit) - an example project using machine learning and ARKit to display labels above objects
41 |
42 |
43 | # Comparision: ARKit, SceneKit, RealityKit - What's the difference?
44 |
45 | Apple’s ARKit platform was originally built on top of SceneKit — a 3D graphics framework developed for mobile games. This gave a solid foundation to leverage SceneKit’s rendering technology to create an early platform for augmented reality applications. **RealityKit** was introduced after **ARKit and SceneKit,** as a way to simplify the workflow of creating an augmented reality application. RealityKit offers many features out of the box, such as: handling gestures, collisions, and applying real world lighting effects to objects.
46 |
47 | **ARKit** (iOS 11)
48 |
49 | **ARKit** captures devices data that is used to render objects in a realistic method - it combines device motion tracking, camera scene capture, and advanced scene processing. SceneKit offers a way to render AR content using the data provided from ARKit.
50 |
51 | **SceneKit** (iOS 8)
52 |
53 | **SceneKit** was originally developed for mobile games. In an augmented reality application, SceneKit uses the data from ARKit to render objects in a realistic method. SceneKit provides a way to drill down and manage these objects, but many of the features were not created for an augmented reality application. This is where RealityKit comes in.
54 |
55 |
56 | **RealityKit** (iOS 13)
57 |
58 | **RealityKit** was created as a way to simplify the creation of building augmented reality applications. Out of the box, RealityKit provides many of the features that would traditionally require manual calculations and rendering.
59 |
60 | 
61 |
62 | _RealityKit mapping of virtual object on table_
63 |
64 |
65 |
66 | -----
67 |
68 |
69 |
70 | # ARKit
71 |
72 | ARKit is an augmented reality framework for iOS, available on both iPhone and iPad. ARKit lets developers place digital objects in the real world by blending the camera on the screen with virtual objects, allowing users to interact with these objects in a real space.
73 |
74 | It does this by using the camera on iOS devices to create a map of the area, detecting things like tabletops, floors and the location of the device in the physical space using CoreMotion data. No calibration is required from the user.
75 |
76 | ARKit can run on most modern iPhones and iPads, utilizing SceneKit integration to render the objects. ARKit also makes it possible to integrate with third-party tools such as Unity and Unreal Engine, to use their rendering capabilities.
77 |
78 | ARKit Features
79 | - [Face Tracking](https://developer.apple.com/documentation/arkit/content_anchors/tracking_and_visualizing_faces)
80 | - Front + Back Camera data collection
81 | - [People / Object Occlusion](https://developer.apple.com/documentation/arkit/camera_lighting_and_effects/occluding_virtual_content_with_people)
82 | - Lighting & Shadows estimation
83 | - Motion Capture (body tracking)
84 | - Scene Understanding (Surface detection)
85 | - [World Tracking](https://developer.apple.com/documentation/arkit/arworldtrackingconfiguration)
86 | - [ARWorldMaps](https://developer.apple.com/documentation/arkit/arworldmap) (Save and relaunch a scene as a world map)
87 | - [Location Anchors](https://developer.apple.com/documentation/arkit/content_anchors/tracking_geographic_locations_in_ar)
88 | - Multiplayer Connectivity
89 |
90 |
91 | 
92 | -
93 | 
94 |
100 |
101 | # RealityKit
102 |
103 | The RealityKit framework was built from the ground up specifically for augmented reality with photo-realistic rendering, camera effects, animations, physics, and more. With native Swift APIs, ARKit integration, incredibly realistic physics-based rendering, transform and skeletal animations, spatial audio, and rigid body physics, RealityKit makes AR development faster and easier than ever before.
104 |
105 |
106 | RealityKit Features
107 |
108 | - Leverages data from ARKit for realistic rendering of augmented reality experiences
109 | - Gesture handling (tap to move objects)
110 | - Animations (start or stop multiple animations on different objects)
111 | - Shaders (allows for complex rendering of objects)
112 | - Physics (Object movement and collision)
113 | - Lighting & Shadows estimation
114 |
115 | Materials
116 |
117 | - SimpleMaterial
118 | - UnlitMaterial
119 | - OcclusionMaterial
120 | - VideoMaterial
121 | - PhysicallyBasedMaterial
122 | - CustomMaterial
123 |
124 |
125 | _____
126 |
127 |
128 | # Quick Start - RealityKit Implementation
129 |
130 | Quickly add a .usdz file to an ```ARView``` using ```Entity.load ```
131 | ```swift
132 | import RealityKit
133 |
134 | //Initialize an ARView programically
135 | let arView = ARView(frame: CGRect(x: 0, y: 0, width: 0, height: 0))
136 |
137 | //Or connect via IBOutlet
138 | @IBOutlet weak var arView: ARView!
139 |
140 | //Add a .usdz object to the ARView
141 | let anchor = AnchorEntity()
142 | let modelEntity = try! Entity.loadModel(named: "name_of_usdz_file")
143 | anchor.addChild(modelEntity)
144 | arView.scene.addAnchor(anchor)
145 |
146 |
147 | ```
148 |
149 |
150 | _Note: Make sure to enable the **Privacy - Camera Usage Description** setting in info.plist_
151 |
152 |
153 | # Common Components in an Augmented Reality Application
154 |
155 |
156 | 
157 |
158 |
159 | - [ARView](https://developer.apple.com/documentation/realitykit/arview)
160 | - _A view that displays an augmented reality experience that incorporates content from RealityKit._
161 | - This is what allows us to see 3d models placed into the environment, using the camera view.
162 |
163 | - [Scene](https://developer.apple.com/documentation/realitykit/scene)
164 | - A container that holds the collection of entities rendered by an AR view.
165 | - You don’t create a Scene instance directly. Instead, you get the one and only scene associated with a view from the scene property of an ARView instance.
166 | - To add content to the view’s scene, you first create and add one or more AnchorEntity instances to the scene’s anchors collection.
167 |
168 | - [AnchorEntity](https://developer.apple.com/documentation/realitykit/anchorentity)
169 | - Anchors tell RealityKit how to pin virtual content to real world objects, like flat surfaces or images. You then add a hierarchy of other Entity instances to each anchor to indicate the geometry and behaviors that RealityKit should render at a given anchor point.
170 |
171 | - [Entity](https://developer.apple.com/documentation/realitykit/entity)
172 | - An element of a RealityKit scene to which you attach components that provide appearance and behavior characteristics for the entity.
173 | - You create and configure entities to embody objects that you want to place in the real world in an AR app. You do this by adding Entity instances to the Scene instance associated with an ARView.
174 |
175 | ----------------
176 |
177 | # Useful Snippets
178 |
179 | ```swift
180 |
181 | // create an entity from local .usdz file
182 |
183 | func addEntity() {
184 | let anchor = AnchorEntity()
185 | let modelEntity = try! Entity.loadModel(named: "Local_Filename_Here")
186 |
187 | anchor.addChild(modelEntity)
188 | arView.scene.addAnchor(anchor)
189 | }
190 |
191 | //create a cube entity with texture from remote image url
192 |
193 | func createCubeWithRemoteTexture(remoteUrl: URL) {
194 | // First create a local temporary file URL to store the image at the remote URL.
195 | let fileURL = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
196 | // Download contents of imageURL as Data. Use a URLSession if you want to do this asynchronously.
197 | let data = try! Data(contentsOf: remoteURL)
198 |
199 | // Write the image Data to the file URL.
200 | try! data.write(to: fileURL)
201 | do {
202 | // Create a TextureResource by loading the contents of the file URL.
203 | let texture = try TextureResource.load(contentsOf: fileURL)
204 | var material = SimpleMaterial()
205 | material.baseColor = MaterialColorParameter.texture(texture)
206 |
207 | let entity = ModelEntity(mesh: .generateBox(size: 0.5), materials: [material])
208 | let anchor = AnchorEntity(.plane(.any, classification: .any, minimumBounds: .zero))
209 |
210 | entity.generateCollisionShapes(recursive: true)
211 | arView.installGestures(for: entity)
212 |
213 | anchor.addChild(entity)
214 | arView.scene.addAnchor(anchor)
215 | } catch {
216 | print(error.localizedDescription)
217 | }
218 | }
219 |
220 |
221 |
222 | //save an AR world map to url
223 |
224 | func writeWorldMap(_ worldMap: ARWorldMap, to url: URL) throws {
225 | let data = try NSKeyedArchiver.archivedData(withRootObject: worldMap, requiringSecureCoding: true)
226 | try data.write(to: url)
227 | }
228 |
229 | //load an AR worldmap from url
230 |
231 | func loadWorldMap(from url: URL) throws -> ARWorldMap {
232 | let mapData = try Data(contentsOf: url)
233 | guard let worldMap = try NSKeyedUnarchiver.unarchivedObject(ofClass: ARWorldMap.self, from: mapData)
234 | else { throw ARError(.invalidWorldMap) }
235 | return worldMap
236 | }
237 |
238 | ```
239 |
240 | -----------------
241 |
242 | # Media:
243 | 
244 |
245 | _Apple Measure App_
246 |
247 | 
248 |
249 | _Lowes Envision App_
250 |
251 | 
252 |
253 | _Face Tracking_
254 |
255 | Veep - Augmented Reality Social Network: https://youtu.be/HDHnbd8QHGU
256 |
257 |
258 | # Resources:
259 |
260 | **Apple Tools**
261 |
262 | 
263 |
264 | https://developer.apple.com/augmented-reality/tools/
265 |
266 | - Reality Composer - found within Xcode developer menu
267 |
268 | - Reality Converter - Download from Apple website, allows for quick conversion of common 3d media formats into .usdz or .reality for ARKit compatibility. Common filetypes include: .fbx and .obj
269 |
270 | - Object Capture - https://developer.apple.com/augmented-reality/object-capture/
271 | - Turn photos from your iPhone or iPad into high‑quality 3D models that are optimized for AR using the new Object Capture API on macOS Monterey. Object Capture uses photogrammetry to turn a series of pictures taken on your iPhone or iPad into USDZ files that can be viewed in AR
272 |
273 | **Useful Links (Apple.com)**
274 |
275 | - Apple Quick Look Gallery (AR previews on Safari): https://developer.apple.com/augmented-reality/quick-look/
276 | - Apple Developer Augmented Reality Homepage - https://developer.apple.com/augmented-reality/
277 |
278 | **Apple Example Projects:**
279 |
280 | - WWDC 2021 Fish Scene: https://developer.apple.com/documentation/realitykit/building_an_immersive_experience_with_realitykit
281 | - WWDC 2021 Reality Kit 2 Example Project: https://developer.apple.com/documentation/realitykit/building_an_immersive_experience_with_realitykit
282 | - Saving and Loading World Data: https://developer.apple.com/documentation/arkit/data_management/saving_and_loading_world_data#see-also
283 | - Tracking and Visualizing Planes: https://developer.apple.com/documentation/arkit/content_anchors/tracking_and_visualizing_planes
284 | - Tracking and Visualizing Faces: https://developer.apple.com/documentation/arkit/content_anchors/tracking_and_visualizing_faces
285 |
286 | **Other Example Projects & Frameworks:**
287 |
288 | - [ARKit Sampler](https://github.com/shu223/ARKit-Sampler) - a sample implementation of many ARKit features.
289 | - [RealityUI](https://github.com/maxxfrazer/RealityUI) - a framework for building user interfaces in AR
290 |
291 |
292 | **3D Model Marketplaces**
293 |
294 | - https://sketchfab.com/
295 | - https://www.turbosquid.com
296 | - https://assetstore.unity.com
297 | - https://free3d.com/
298 |
299 | **Augmented Reality iOS Timeline**
300 |
301 | - 2012 Vuforia Engine (3rd party AR framework)
302 | - 2017 ARKit Introduction
303 | - 2018 ARKit 2
304 | - 2019 ARKit 3
305 | - 2019 RealityKit
306 | - 2020 ARKit 4
307 | - 2021 RealityKit 2
308 | - 2021 ARKit 5
309 |
310 |
311 | Sources:
312 | - TechRepublic ARKit Cheat Sheet: https://www.techrepublic.com/article/apples-arkit-everything-the-pros-need-to-know/
313 | - Lowes AR App: https://www.marketingdive.com/news/lowes-debuts-2-ar-apps-to-help-people-furnish-their-homes/505798/
314 | - Shaders: https://maxxfrazer.medium.com/getting-started-with-realitykit-shaders-468c55738f8e
315 | - ARKit Sample Project: https://github.com/shu223/ARKit-Sampler
316 |
317 |
318 | # Conclusion / Future of AR Apps on iOS
319 |
320 | - RealityKit + SwiftUI + Apple Silicon = Apple Glasses
321 |
322 | With such a strong augmented reality platform, it certainly seems Apple is preparing for their launch of Apple Glasses. These tools and frameworks have been quietly brewing in the background - the major bugs, flaws, and quirks, have all been worked out. Apple Glasses might spark a consumer revolution similar to the first iPhone, so developers and anyone involved in mobile app development should pay attention to this space.
323 |
324 |
325 |
326 |
327 |
--------------------------------------------------------------------------------