iOS 13 & RealityKit support

[jerryrt]

Seems Apple introduced a new render path for the coming iOS 13 release with the RealityKit, but it is not supported currently by ARVideoKit.

My attempt is to record the official "CapturingBodyMotionIn3D" demo project with a modified branch but failed.

Here is my branch:
https://github.com/jerryrt/ARVideoKit/tree/ios_13_reality_kit

Now it only crashes the app, with the following log. Any help/suggestion deeply appreciated.

2019-06-26 19:47:17.128271+0800 BodyDetection[16813:3466900] [Assert] COMPATIBILITY BUG IN CLIENT OF UIKIT: Moving method canvasToolbar from UICanvas to UIWindowScene 2019-06-26 19:47:17.128726+0800 BodyDetection[16813:3466900] [Assert] Registering Canvas Component Class _UICanvasUserActivityManager, please update to Frame Components 2019-06-26 19:47:17.377301+0800 BodyDetection[16813:3466900] [Assert] Calling deprecated SPI on UICanvas. This method and class will dissapear prior to GM 2019-06-26 19:47:17.377458+0800 BodyDetection[16813:3466900] [Assert] Calling deprecated SPI on UICanvas. This method and class will dissapear prior to GM 2019-06-26 19:47:17.378419+0800 BodyDetection[16813:3466900] [Assert] Calling deprecated SPI on UICanvas. This method and class will dissapear prior to GM 2019-06-26 19:47:17.405362+0800 BodyDetection[16813:3466900] Metal API Validation Enabled 2019-06-26 19:47:18.102950+0800 BodyDetection[16813:3466900] [TraitCollection] Class _UISheetPresentationController overrides the -traitCollection getter, which is not supported. If you're trying to override traits, you must use the appropriate API. 2019-06-26 19:47:18.103005+0800 BodyDetection[16813:3466900] [TraitCollection] Class _UIRootPresentationController overrides the -traitCollection getter, which is not supported. If you're trying to override traits, you must use the appropriate API. 2019-06-26 19:47:18.185617+0800 BodyDetection[16813:3466900] [TraitCollection] Class UIPopoverPresentationController overrides the -traitCollection getter, which is not supported. If you're trying to override traits, you must use the appropriate API. 2019-06-26 19:47:18.278773+0800 BodyDetection[16813:3466900] [API] Requesting unregistered service named Interaction Manager - 0x103411850 2019-06-26 19:47:18.278826+0800 BodyDetection[16813:3466900] [API] Requesting unregistered service named Interaction Manager - 0x103411850 2019-06-26 19:47:18.512311+0800 BodyDetection[16813:3466900] Compiler failed to build request 2019-06-26 19:47:18.512471+0800 BodyDetection[16813:3466900] [Graphics] makeRenderPipelineState failed [output of type ushort is not compatible with a MTLPixelFormatR16Float color attachement.]. 2019-06-26 19:47:18.512502+0800 BodyDetection[16813:3466900] [Graphics] makeRenderPipelineState failed. 2019-06-26 19:47:18.780749+0800 BodyDetection[16813:3466900] [Common] _BSMachError: port 14c03; (os/kern) invalid capability (0x14) "Unable to insert COPY_SEND" 2019-06-26 19:47:18.791973+0800 BodyDetection[16813:3466947] Compiler failed to build request 2019-06-26 19:47:18.793396+0800 BodyDetection[16813:3466947] Compiler failed to build request 2019-06-26 19:47:18.794085+0800 BodyDetection[16813:3466947] [Graphics] makeRenderPipelineState failed [Vertex shader writes point size but inputPrimitiveTopology is MTLPrimitiveTopologyClassTriangle]. 2019-06-26 19:47:18.794522+0800 BodyDetection[16813:3466947] [Graphics] makeRenderPipelineState failed. -[MTLDebugRenderCommandEncoder setRenderPipelineState:]:1294: failed assertion 'renderPipelineState must not be nil.'

[yuwen-lu]

I met with a similar issue simply by trying to get the Apple CapturingBodyDetectionIn3D demo running. I haven’t found any solution yet. Is it with the demo app itself or with this framework?🤔

[jerryrt]

The above crash is largely caused by the demo project. I have crafted a clean demo project to ease the dev&test during my last effort.

https://github.com/jerryrt/ARVideoKit/blob/ios_13_reality_kit/Examples/Swift/ARVideoKit-Example/View%20Controllers/RKViewController.swift

This is my working branch, at least the RealityKit is running, but the recording is not working(too slow). Currently, my workaround is to capture the content by using UIView's drawing method, but it is too slow for video.

jerryrt@5f221a1

In the original design of this module, it uses a GPU based render engine underhood to render the content, I have no luck in setting up RealityKit to work with this approach, just FYI.