By Kushan Vyas, Tech Lead – Mali-D77 DPU, Arm
At May’s SID Display Week, Arm formally launched the Arm Mali -D77 DPU, a display processor that significantly improves the VR user experience on head-mounted display (HMD). The headline benefit was all about the elimination of motion sickness for VR users, but there are also a number of compelling performance gains to improve the overall VR user experience. These are made possible by brand new dedicated hardware functions that are part of Mali-D77 – Lens Distortion Correction (LDC), Chromatic Aberration Correction (CAC) and Asynchronous Timewarp (ATW).
HMDs
Re-projection in real time
In most VR solutions, the graphics processor or the video decoder generate the content that the user is ultimately going to interact with in the VR scene. This content is produced originally as standard rectangular frames, with no reference to the user. As the user moves around, and interacts with the VR scene, these rectangular frames need to be translated or transposed to the point of view of the head position of the user. This is known as the process of re-projection.
Traditionally, re-projection is undertaken by the graphics processor, which would read in the standard (unprocessed) frame from external memory, apply re-projection to it and write it back to external memory or to a cache shared with the display processor. However, this additional read and write operation incurs latency from the user’s perspective. This means that any changes in the head position of the user would take a frame or two before they get applied.
In a solution that has Mali-D77, the graphics processor or the video decoder are focused purely on generating the standard frames, while the DPU reads the frames from external memory and re-projects these in real-time right before transmission to the panel. By performing this operation on the last stage of the multimedia pipeline, right before transmission to the panel, the perceived latency of the user is nearly zero. This process of re-projecting frames to the latest head position of the user is known as timewarp.
Generating artificial frames to reduce perceived latency
Re-projecting in the real time path of Mali-D77 has an additional implication for reducing perceived latency. In scenarios where the graphics processor needs to render complex content, it is not always capable of keeping up with the frame rate of the display processor, and so on occasion it will not be able to generate a re-projected frame in time for the display processor to read. In these instances, the display processor will re-fetch and display the last frame re-projected by the graphics processor. From the user’s perspective, however, this comes across as jitter because they’re moving around in the VR scene, but for a split-second the scene doesn’t move with the user. This breaks the immersion of the experience.
By re-projecting on Mali-D77, the DPU will re-fetch the previously generated frame, but re-project it to the latest head position of the user. From the user’s perspective this means that as they move around, the scene continues to move with them. However, the content doesn’t change as rapidly – which is less imperceptible and thus maintains the feeling of immersion for the user. This ability to generate artificial frames and then re-projected to the latest head position of the user is one of the key features of Mali-D77, setting it apart over existing VR solutions. The process of generating artificial frames, decoupled from the rate of generating the original frames is known as Asynchronous Timewarp.
Lens Pre-Distortion
VR headsets tend to have large lenses within them. The main goal is to artificially push the scene on the screen further away, in order to create the illusion of immersion. However, due to the properties of light going through lenses, the images tend to get warped. This is particularly noticeable around the edges.
Mali-D77 addresses this issue by applying the inverse distortion to the images, before they are sent out through the lens. The amount of distortion – and the form of distortion – is software configurable. This configurability allows Mali-D77 to be used in a wide range of headsets.
Chromatic Aberration Correction
One of the other challenges of light going through a glass is that the colour channels separate. This phenomenon is called Chromatic Aberration. Mali-D77 addresses this by applying the inverse chromatic aberration to the content before sending them out of the display. The amount of colour channel separation varies based on the lens, and so this too is software configurable.
Three functions as a single pass
As part of Mali-D77, all three operations of LDC, CAC and ATW are implemented as a single pass through the real-time part of the display processor. This ensures a near zero latency perception and enables a rich and immersive experience for the user.
Figure 1 Dedicated hardware functions of the Mali-D77 DPU – LDC
More VR Functions
In addition to the three core VR functions that Mali-D77 is capable of, it also performs additional functions that enhance the overall viewing experience for the user – Global Vignetting and Clamp to Edge.
Global Vignetting
On occasions when the lens distortion is on the extreme end of what Mali-D77 can support, artefacts around the edges of the VR layer can be accentuated. In order to mask these, a vignette layer can be applied on top of the VR layer to smooth these out. The global vignette can be used additionally to focus the attention of the user on the relevant parts of the screen.
Clamp to Edge
When re-projecting a layer, it is possible that the view rotates far enough that the visible area “falls off” the edge of the buffer. In these instances, Mali-D77 can replicate pixels from the edge of the buffer to fill the gaps. This is typically considered less distracting than the black flashes that would otherwise appear.
VR Processing on the display processor
Traditionally, most functions for VR processing that have been described in this article are currently solved via the graphics processor. The innovation with Mali-D77 is being able to complete VR processing actions on the real time path of the display processor. The outcome is system bandwidth and power savings, while also enabling new zero latency. In addition, each hardware function on the DPU provides in-line processing to address the technical challenges that are facing VR HMDs, such as color and image distortion. The end result is no more motion sickness and higher-quality processing for a more immersive VR experience.