Updated: August 30, 2021.
TL;DR
RealityKit 2.0
RealityKit is the youngest SDK in Apple family of rendering technologies. This high-level framework was released in 2019. RealityKit is made for AR / VR projects, has simplified settings for multi-user experience and can be used on iOS / macOS. Performs multithreaded rendering.
There's no Objective-C legacy, RealityKit supports only Swift, and rather declarative syntax (like in SwiftUI). The main advantage of RealityKit – it can complement / change / customise scenes coming from Reality Composer app and can be a powerful extension for ARKit – although it shines as a standalone AR SDK as well. In RealityKit the main units are entities (ModelEntity, AnchorEntity, TriggerVolume, BodyTrackedEntity, PointLight, SpotLight, DirectionalLight and PerspectiveCamera) that has components and can be created from resources like ModelEntity. The framework runs an Entity Component System (ECS) on the CPU to manage tasks like physics, animations, audio processing, and network synchronization. But it relies on the Metal and GPU hardware to perform multithreaded rendering. RealityKit has six materials: UnlitMaterial, SimpleMaterial, PhysicallyBasedMaterial (with 18 AOVs for controlling material's look), OcclusionMaterial, VideoMaterial and, of course, CustomMaterial.
Pay particular attention to shadows on iOS – devices up to A11 chipset produce projective (a.k.a. depth map) shadows, but on devices with A12 and higher we can see raytraced shadows. Maybe your solution in this case could be fake shadows. Many contemporary render features in RealityKit are On by default: camera's depth of field, face/person occlusion, grounding shadows, motion blur, camera grain, etc.
Sample code:
@IBOutlet weak var arView: ARView!
let box = MeshResource.generateBox(size: 0.5)
let material = PhysicallyBasedMaterial()
let model = ModelEntity(mesh: box, materials: [material])
let anchor = AnchorEntity(world: [0, 0,-1])
anchor.addChild(model)
arView.scene.anchors.append(anchor)
RealityKit reads in .usdz, .rcproject and .reality file formats. Supports asset animation, dynamics, PBR materials, HDR Image Based Lighting and spatial audio. All scene models must be tethered with anchors (AnchorEntity class). Framework automatically generates and uses mipmaps, which are a series of progressively low-rez variants of objects' texture that improve render times when applied to distant objects. RealityKit works with a polygonal mesh generated using Scene Reconstruction feature. I wanna add a few words about AR Quick Look - a zero-settings framework that's built on the RealityKit engine and it's conceived for fast AR visualization.
Conclusion: RealityKit gives you a high-quality render technology and up-to-date AR capabilities out-of-the-box. Supports LiDAR Scanner. Supports Photogrammetry tools. Plays Reality Composer's behaviours through its notification API. RealityKit can be used as standalone framework, or as a partner of ARKit and MetalKit. Starting from iOS 15 we have access to fragment/pixel shaders and geometry modifiers via Metal scripting and CustomMaterials.
RealityKit works with UIKit storyboards or with SwiftUI interfaces. It has a minimum of a boilerplate code. For example, RealityKit has a very simple setup for models' collision and for gestures (pan, rotate, pinch). And there's composition over inheritance, so it's rather a Protocol Oriented Programming framework – tight coupling in most cases is no longer a problem in your code. RealityKit fits perfectly with the Combine reactive paradigm, which helps handling publishers, subscribers and asynchronous events.
RealityKit's native view is ARView.
@available(OSX 10.15, iOS 13.0, *)
@objc open class ARView : ARViewBase
SceneKit
SceneKit is a high-level framework as well. The oldest one in Apple family of rendering technologies. It was released in 2012. SceneKit was conceived for VR and can be run on iOS / macOS. For AR projects you can use it only in conjunction with ARKit. SceneKit supports both Objective-C and Swift. In SceneKit the main unit is a node (SCNNode class) that has its own hierarchy and can store a light (SCNLight), or a camera (SCNCamera), or a geometry (SCNGeometry), or a particle system (SCNParticleSystem), or audio players (SCNAudioPlayer). The main advantage of SceneKit – it's highly customisable, it can change geometry and materials at runtime, it has morphers, skinners and constraints, it renders a scene up to 120 fps and it has an advanced setup for particle system. There are Blinn, Constant, Lambert, Phong, ShadowOnly and PBR shaders.
Occlusion shader is also available for us in SceneKit but in a custom form (there's no out-of-the-box occlusion material here like we can find in RealityKit). In case you need a video material in SCNScene you should implement SpriteKit's SKVideoNode.
Also we can use an SCNProgram object to perform custom rendering. It's a complete Metal or OpenGL shader program that replaces SceneKit's rendering of a material or even geometry. SceneKit's reliable companion is a Model I/O library that carries out import, export, and models' manipulation using a common infrastructure.
Sample code:
@IBOutlet weak var sceneView: SCNView!
sceneView.scene = SCNScene()
sceneView.autoenablesDefaultLighting = true
let boxNode = SCNNode()
boxNode.geometry = SCNBox(width: 0.5, height: 0.5, length: 0.5, chamferRadius: 0)
boxNode.geometry?.firstMaterial?.lightingModel = .physicallyBased
boxNode.geometry?.firstMaterial?.diffuse.contents = UIColor.red
boxNode.geometry?.firstMaterial?.metalness.contents = 1.0
sceneView.scene?.rootNode.addChildNode(boxNode)
SceneKit reads in .usdz, .dae and .scn file formats. Supports nested asset animation, dynamics, particles, PBR materials, HDR IBL and spatial audio. For implicit and explicit transform animation of any node you can use SCNAction, SCNTransaction and CAAnimation classes. Though a collisions' setup in SceneKit is a little bit complicated. To create a modular and scalable game architecture with SceneKit we need to implement GameplayKit’s entity-component pattern.
Conclusion: SceneKit gives you a high-quality render technology (but at first you need to setup physicallyBased shaders), although for AR projects you can use it only with ARKit. SceneKit is highly customisable and can be used with Swift and Objective-C, and it gives you a set of useful renderer(...) instance methods coming from ARSCNViewDelegate protocol that allows you update AR models and tracked anchors at 60 fps. Works with UIKit and SwiftUI (despite the fact that there is no SceneKit+SwiftUI template in Xcode). There are obvious reasons that Apple might make this framework deprecated during the next 3 years – SceneKit hasn't been updated since 2017 (excluding minor changes, like clearCoat material property). But SceneKit still has several basic advantages over RealityKit 2.0.
SceneKit's native view is SCNView.
@available(iOS 8.0, tvOS 9.0, *)
open class SCNView : UIView, SCNSceneRenderer, SCNTechniqueSupport
@available(OSX 10.8, *)
open class SCNView : NSView, SCNSceneRenderer, SCNTechniqueSupport
Metal and MetalKit
To be precise, Metal is not a rendering technology but rather the GPU accelerator with the ability of using a rich shading language. It was released in 2014. It's a low-level framework. Metal is implemented everywhere – in RealityKit, SceneKit, ARKit, CoreML, Vision, AVFoundation, etc. Metal combines functions similar to OpenGL and OpenCL under the hood of just one API. Of course, Metal can be used as a renderer for advanced 3D graphics. Metal renders not only reflections but also refractions and subsurface scattering phenomena.
According to Apple documentation: "Metal is a C++ based programming language that developers can use to write code that is executed on the GPU for graphics and general-purpose data-parall
