Apple Glass leak just revealed a killer feature

Apple Glass concept
(Image credit: Techweartrend)

During the pandemic, everyone has got a lot of mileage out of changing Zoom backgrounds to liven up dull meetings. But it looks like Apple is assessing the same kind of technology for its upcoming AR specs, the Apple Glass.

First spotted by Patently Apple, the patent describes how a headset could perform chroma keying – which is replacing a solid color background with something else. This may not sound too taxing, but the trick is to make it a hardware solution on the headset, because even slight delays could otherwise prompt motion sickness. Add to that the fact that a headset is constantly moving unlike a Zoom-enabled laptop, and you can see why this could prove quite a challenge. 

First filed back in February, but only published publicly this week, Apple’s patent is about creating a low latency 3D stereoscopic background for Apple AR. What this comes down to is “a wearable immersive head-mounted display (HMD) doing real-time color-keying using embedded stereo RGB cameras to capture the environment and a display system to show real-world images augmented with virtual content.”

It seems like Apple has had some success with this challenge. “The system runs at a high frame rate (typically 75 fps or more) and it achieves a latency of less than one frame by capturing images and formatting them for display in the HMD itself," the patent says.

A patent for the chroma keying on Apple Glass

(Image credit: Apple patent)

How has Apple done this? By offloading the processing to the headset, which can, according to the patent, “format camera images, detect the selected color range and make a composite with the virtual content. “ Clever stuff.

Along with the fact that people tend to move their heads when wearing a headset, there’s another issue Apple has to contend with: all of this has to be done twice, once for each eye. “The system achieves mixed reality by having a user look at a display through wide angle lenses (FOV is typically 110 degree or more)," the patent reads. "Two cameras located behind the HMD display capture the environment from viewpoints located a few centimetres in front of each eye."

All very nice, but what’s the point? As you might expect, the patent doesn’t go into massive amounts of detail here, keeping things vague: “games, videos and virtual objects.”

We won’t have concrete answers any time soon, either. People in the know differ about when we’ll see Apple Glass for the first time, but the earliest estimate is March next year, with well connected analyst Ming-Chi Kuo saying 2022 at the earliest. A report from last year suggested 2023 was more likely.

It’s also worth remembering that Apple patents an awful lot of things – over 2,500 bits and pieces in the last year alone – and not everything gets used. So while you may well be applying funny backgrounds to your Apple Glass in 2023, it’s equally possible that that’s another feature left on the cutting room floor.  

Alan Martin

Freelance contributor Alan has been writing about tech for over a decade, covering phones, drones and everything in between. Previously Deputy Editor of tech site Alphr, his words are found all over the web and in the occasional magazine too. When not weighing up the pros and cons of the latest smartwatch, you'll probably find him tackling his ever-growing games backlog. Or, more likely, playing Spelunky for the millionth time.

  • Hardware Geek
    The biggest question for me is whether these will run independently or if hey will require an iPhone to work. I don't want to switch to an iPhone but if these run without one, I may pick up a pair.