Microsoft HoloLens is the first self-contained Augmented Reality (AR) mobile computer. Hololens projects images (holograms) onto the headset, enabling you to engage with your digital content and interact with holograms in the world around you. The first version, available as a developer kit is sold for $3,000 with no timeline for a consumer version. However, Microsoft is busy working away on their second version of Hololens, and some details were discussed in a recent Microsoft Blog entry.
Deep Neural Networks (DNNs)
Deep neural networks, or DNNs. Require large amounts of data and processing. Often, this requires processing which exceeds that of local computer power. Especially on a battery powered mobile device. Processing the information in the cloud allows to off load some of the processing but introduces too much latency for AR devices. Hence the need for Hololens to develop its custom silicon, the Holographic Processing Unit (HPU)
Holographic Processing Unit (HPU) v2.0
HoloLens contains a custom multiprocessor called the Holographic Processing Unit, or HPU. It is responsible for processing the information coming from all of the onboard sensors, including Microsoft’s custom time-of-flight depth sensor, head-tracking cameras, the inertial measurement unit (IMU), and the infrared camera.
The HPU, among many other things, allows for full hand and finger tracking in real time.
Harry Shum, executive vice president of Microsoft Artificial Intelligence and Research Group, announced in a keynote speech at CVPR 2017:
"...the second version of the HPU, currently under development, will incorporate an AI coprocessor to natively and flexibly implement DNNs. The chip supports a wide variety of layer types, fully programmable by us."
Harry showed an early version of the HPU v2.0 running live code implementing hand segmentation at CVPR 2017. No word yet on the availability of Hololens 2.0.