This post takes a look at ARKit Face Tracking on iPhone X, XS, XR and iPad Pro 2018. It is based on my WWDC 2019 Scholarship Submission. I am not an artist. All models I created for this post are just for illustration purposes. But I want to look into the code required to create your own Animoji. This post is divided into four steps:
ARKit Face Tracking is easy to setup.
Simply start an ARFaceTrackingConfiguration
where it is supported and ARKit sets everything up for your.
When a face is detected by the True Depth Camera ARKit creates an ARFaceAnchor
.
Then, make your ViewController
confirm to ARSCNViewDelegate
and implement
renderer(_:didAdd:for:)
where you add and load your face.
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
guard anchor is ARFaceAnchor else {
return
}
node.addChildNode(characterNode)
}
To receive updated facial expressions also implement the renderer(_didUpdate:for:)
method as well.
func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
DispatchQueue.main.async {
guard let face = anchor as? ARFaceAnchor else {
return
}
self.process(face.blendShapes)
}
}
To extract the facial expressions from a face use the .blendShapes
property.
ARKit automatically assigns certain facial expressions a weight from 0 to 1.
Examples for BlendShapeLocations
are for example .eyeBlinkLeft
or .jawOpen
.
The meaning if the weight for each blend shape can be found in the documentation.
Now that we now about the basics of Face Tracking let’s take a look into how to create an animatable character in Blender.
Let’s assume you have a simple face model in blender with a neutral facial expression.
Blender supports shape keys.
These are modification you can do on the base geometry.
Press the plus button (you need to be in object mode).
This creates the neutral Basis.
Now modify your character, for example by closing the left eye
(only move vertices, all shapes need to be topologically identical, that is,
have the same number of vertices, edges and faces).
Then press the plus button again and name the key e.g. CloseLeftEye.
As you can see, you can assign a weight between 0 and 1 for the shape
(sounds familiar?).
Now, create a shape key for every ShapeBlendLocation
you want to support in ARKit.
As you can see you can then individually assign an arbitary weight to each shape.
Blender than applies all shape modifications at the same time.
Export the model as an .dae
file (check that Include Shape Keys is set in the options).
For SceneKit to read this file you need to fix a few properties in the file.
Thankfully, there is a great tool by JonAlle on GitHub that does this for us.
Then add the file to your .scnassets
where you can convert it to an .scn
file.
In the Xcode editor you can also confirm that the shape keys are working as expected.
In SceneKit all shape keys are referenced by a number in which is was defined in Blender. To make working with those easier we can create a simple enum that makes the reference explicit.
enum Modifier: Int {
case closeLeftEye = 0
case closeRightEye
case openMouth
case smileLeft
case smileRight
case liftLeftBrow
case liftRightBrow
case tongueOut
}
We than use the SCNMorpherClass
to apply the differnt keys with their weights.
func changeModifiers(_ modifiers: [Modifier: CGFloat]) {
for (modifier, value) in modifiers {
characterNode.morpher?.setWeight(value, forTargetAt: modifier.rawValue)
}
}
All we have to do now is to change the weights when we get a new update for our face anchor.
func process(_ blendShapes: [ARFaceAnchor.BlendShapeLocation : NSNumber]) {
let modifiers = blendShapesToModifiers(blendShapes)
changeModifiers(modifiers)
}
func blendShapesToModifiers(_ blendShapes: [ARFaceAnchor.BlendShapeLocation : NSNumber]) -> [CharacterController.Modifier: CGFloat] {
var modifiers: [CharacterController.Modifier: CGFloat] = [:]
if let leftEye = blendShapes[.eyeBlinkRight] {
modifiers[.closeLeftEye] = CGFloat(truncating: leftEye)
}
if let rightEye = blendShapes[.eyeBlinkLeft] {
modifiers[.closeRightEye] = CGFloat(truncating: rightEye)
}
if let mouthOpen = blendShapes[.jawOpen] {
modifiers[.openMouth] = CGFloat(truncating: mouthOpen)
}
if let smileLeft = blendShapes[.mouthSmileRight] {
modifiers[.smileLeft] = CGFloat(truncating: smileLeft)
}
if let smileRight = blendShapes[.mouthSmileLeft] {
modifiers[.smileRight] = CGFloat(truncating: smileRight)
}
if let browLeft = blendShapes[.browDownRight] {
modifiers[.liftLeftBrow] = 1 - CGFloat(truncating: browLeft)
}
if let browRight = blendShapes[.browDownLeft] {
modifiers[.liftRightBrow] = 1 - CGFloat(truncating: browRight)
}
if let tongue = blendShapes[.tongueOut] {
modifiers[.tongueOut] = CGFloat(truncating: tongue)
}
return modifiers
}
That’s it.
Using the SCNMorpher
together with Blender’s shape keys and ARKit blend shapes is super easy to setup.
If you (like me) are completely untalented in creating 3D models you can simply download one from the internet. If you are a student you can use the Autodesk Character Generator, where you can create character models in an RPG-like fashion, for free. They even support shape keys!