App Development

Lessons Learned from Developing an ARKit App

Blogggg-07

Recently we finished development on an app that heavily utilizes ARKit and SceneKit. We had a short timeline, so we had to make mistakes and learn fast. Here’s an abbreviated list of some of those lessons.

Have your developers and designers work beside each other

When dealing with apps that are primarily visually-rich content, we found that there were many benefits to having our developers and designers work directly next to each other. We realized very early on that this kind of app required frequent iteration for the designers to feel comfortable with what was being built. By having our developers work with the designers on the project, we were able to prototype ideas that the designers would come up with as well as tweak things like particle effects and 3-D model generation on the fly. This allowed us to be more agile in our approach to taking features from idea to implementation and allowed our designers to have finer control over the final aesthetic.

Embrace protocol-oriented programming from the beginning and break up your controllers

This ARKit app is a game and in the beginning, our architecture primarily used one GameController that was handling our “game logic.” This was a fine idea, but what ended up happening was our interactions with ARKit, logic around how the world should be modified, UIKit interactions with our heads-up display (HUD), and instructions being sent to the multipeer framework—among other disparate logics—were all getting very tightly coupled together. This made unit testing a nightmare before we refactored. We decided to split apart the GameController logic into tinier controllers that had more focused purposes. One of the most valuable controllers to break out was an ARWorldController. This let us section off any interactions with ARKit into its own realm. Similar benefits were found with creating a HUDViewController and a MultipeerController. We still maintained the idea of a GameController, but it became more of a central hub for all the separate controllers to communicate with each other. This change allowed us to test internal business logic around how the game rules should function and separately test the interaction points with the various Apple frameworks we were utilizing. It also meant that we could remove most references to ARKit objects (e.g. ARAnchor) from our unit tests when all we really cared about was the logic backing the interactions.

Learn to live with NSSecureCoding

One of the initial designs for the project involved using Multipeer Connectivity (https://developer.apple.com/documentation/multipeerconnectivity) to give the users the ability to see their AR world in a shared space with friends. A MultipeerSession can share Data objects to other connected peers. If you encode objects in one user’s AR world as Data, that can be sent to other peers, where it’s decoded and placed in the other peers’ AR worlds.

This sounds like a perfect place for Swift’s Codable protocol (https://developer.apple.com/documentation/swift/codable). Unfortunately, most of the objects that ARKit uses to map out the AR world around a user do not conform to Codable. We could extend these types to conform to Codable or make our own “dirty” objects that are codable that are just used for transferring the data needed to instantiate the ARKit objects. What we found was there were a lot of different object types about which we wanted to be sending information (e.g. ARAnchor, ARPlaneAnchor, SCNNode, etc.).

To override all of these or come up with our own “dirty” objects felt both cumbersome and brittle. However, these objects do conform to NSSecureCoding. This means that we could use NSCoding practices for encoding/decoding our multipeer instructions. Unfortunately, this approach gives up all the advantages that Codable gives us, not least of which all the free compiler-generated encoding/decoding code. Hopefully, Apple will implement Codable versions of these AR objects that can be used, but for now using NSSecureCoding is the path of least resistance.

Use enums for all the things

From analytics calls to multipeer instructions to resource loading to localized strings to rows in a table view, Swift’s enums are an amazing tool. I often joke that if I could write my entire app with enums, I would. This is more of a general lesson than an AR-specific one. The more you use enums, the more you’re going to get compile time help with areas of code you’re not thinking about. You’re going to figure out exactly what cases you need to worry about testing and which you don’t. The biggest tip here is to avoid using default cases in switch statements unless you feel you really need it. Let the compiler help you next time you add a case!

Worry about performance

AR is resource intensive—like, VERY intensive. It can be easy to fall into a pattern of not worrying too much about resources when you’re just putting labels on a screen. However, once you have to start mapping out the world around you and painting objects on top of every single frame and do it quickly enough that it doesn’t cause latency for the user, you’ll realize quickly that resource management still matters. We found ourselves looking for any way to make our code even slightly more performant. Some of these tricks included loading our resources upfront on background threads, condensing model nodes down to just the nodes that we needed for mapping in the world, and working with our designer to reduce the number of polygons in our models. Sometimes, however, we found that a less performant option was warranted given what we were getting in return. For a time, our shadows had a jagged nature to them, and we ended up opting for increasing the shadow sampling count on our lighting to give the shadows a bit more fidelity. For others, this is an area where you could potentially find performance increases if you don’t need shadows or don’t need high-fidelity shadowing. It’s going to be a game of give and take between development and design, but it’s something you should be thinking about from the very beginning.

This is an abridged list of lessons. Working with ARKit makes for a different development experience from more typical data-driven apps and we definitely learned more than can fit in a blog post. We found it best to work in a flexible manner that allowed us to make changes frequently to account for unanticipated hurdles. Hopefully this list helps jumpstart your development process.

Join our team to work with Fortune 500 companies in solving real-world product strategy, design, and technical problems.

Find Your Role
Quickstart-Guide-to-Kotlin-Multiplatform

A Quick Start Guide to Kotlin Multiplatform

Kotlin Multiplatform, though still experimental, is a great up-and-coming solution...

Read the article