Apple’s mysterious fisheye projection

If you’ve read my first post about Spatial Video, the second about Encoding Spatial Video, or if you’ve used my command-line tool, you may recall a mention of Apple’s mysterious “fisheye” projection format. Mysterious because they’ve documented a CMProjectionType.fisheye enumeration with no elaboration, they stream their immersive Apple TV+ videos in this format, yet they’ve provided no method to produce or playback third-party content using this projection type.

Additionally, the format is undocumented, they haven’t responded to an open question on the Apple Discussion Forums asking for more detail, and they didn’t cover it in their WWDC23 sessions. As someone who has experience in this area – and a relentless curiosity – I’ve spent time digging-in to Apple’s fisheye projection format, and this post shares what I’ve learned.
↫ Mike Swanson

There is just so much cool technology crammed into the Vision Pro, from the crazy displays down to, apparently, the encoding format for spatial video. Too bad Apple seems to have forgotten that a technology is not a product, as even the most ardent Apple supporterts – like John Gruber, or the hosts of ATP – have stated their Vision Pro devices are lying unused, collecting dust, just months after launch.