Augmented Reality, or AR, has already proven its revolutionary worth for the entertainment industry. It is used in gaming, movies, live events, theme parks, museums and exhibitions, theatre and performances, television, advertising, social media, e-sports, tourism, fashion, art, music and dance, and even comics and graphic novels or magic shows.
One market estimate suggests the AR & VR market for the entertainment industry may reach US$30 billion by 2030, growing at a CAGR of nearly 19% during the forecast period 2024-2030.
The report identified several driving factors that led to growth for AR in entertainment. For instance, deploying AR helped content creators craft immersive narratives that blurred the lines between reality and fiction, inspiring audiences to become active participants in storytelling and enhancing engagement and emotional connection. AR & VR also helped build personalized experiences tailored to individual preferences.
Leveraging data analytics in combination with AR, entertainment companies could customize content, recommendations, and even advertisements. But, these all could be called only the tips of the iceberg.
AR could accomplish much more with specific technological innovations. In the coming segment, we discuss one such groundbreaking innovation that involves a technology called DFPM, Dynamic Facial Projection Mapping.
DFPM: What is it?
It is a novel augmented reality technique where images are projected onto human faces, altering their real-time appearances. In other words, the method involves projecting dynamic visuals onto a person’s face in real time, using advanced facial tracking to ensure projections adapt seamlessly to movements and expressions.
Source: Disney Research
While the technology holds immense potential and could help take long leaps as far as artistic imagination goes, the technique suffers from technical challenges. In short, this problem could be seen as the problem of misalignment.
A new study published in the journal of the Institute of Electrical and Electronics Engineers1 deals with the issue of misalignment. But, before delving deeper into the solution, let us look at what the problem is.
Click here to learn about investing in future of virtual reality.
The Issue of Misalignment in Deploying DFPM
Projecting visuals onto a moving face requires the DFPM system to detect the user’s facial features, such as the eyes, nose, and mouth, within less than a millisecond. Even slight delays in processing or minuscule misalignments between the camera’s and projector’s image coordinates can result in projection errors—or “misalignment artifacts.” These are noticeable errors and can hamper the quality of immersion. It can go up to the point of ruining the immersion.
A team of researchers from the Institute of Science Tokyo, Japan, set out to find solutions to existing challenges in DFPM. Led by Associate Professor Yoshihiro Watanabe, the team also included graduate student Mr. Hao-Lun Peng. In their study, they proposed concepts to reduce the ‘misalignment artifacts.’
Two Proposals to Reduce ‘Misalignment Artifacts‘
The paper by the two Tokyo-based researchers has a long title, which is the case for such papers. But, the problem it deals with is simple to comprehend. It – as cited before – looks to reduce the misalignment artifacts between projected images and target faces, which is a persistent challenge for DFPM’, researchers believe.
Researchers made two proposals to achieve their objective. The first concept proposed a high-speed face-tracking method that would exploit temporal information. In their efforts, the researchers first introduced a cropped-area-limited inter/extrapolation-based face detection framework, which allowed parallel execution with facial landmark detection.
The researchers then proposed a novel hybrid facial landmark detection method that combined fast Ensemble of Regression Trees (ERT)-based detections and an auxiliary detection. ERT-based detections produced fast results in 0.107 ms using temporal information with the support of auxiliary detection to recover from detection errors.
To train the facial landmark detection method, the researchers proposed an innovative method for simulating high-frame-rate video annotations to address the lack of publicly available high-frame-rate annotated datasets.
The second concept involved a lens-shift co-axial projector-camera setup that could maintain a high optical alignment with only a 1.274-pixel error between 1 m and 2 m depth, reducing misalignment by applying the same optical designs to the projector and camera without causing large misalignment as in conventional methods.
The deployment of these concepts could lead researchers to develop a novel high-speed DFPM system that achieves nearly perfect alignment with human visual perception.
Click here for a list of companies redefining the AR/VR frontier.
The Achievements Summarized
If we shift our focus away from the highly technological aspects of the research to try to understand what it could achieve from an application perspective, the findings would be as follows:
The proposed setup could lead to exceptionally high optical alignment of less than 1.3 pixel error at a depth of 1-2 m. The setup offered faster processing and led to high accuracy in dynamic scenarios.
Additionally, during the process, the researchers developed a method for simulating high-speed video annotations to train the models.
Altogether, the results should help build more compelling and hyper-realistic effects for live performances, fashion shows, and artistic presentations.
While researchers would continue to bring more precision to the technology and its application in the entertainment industry, businesses are also making improvements they deem fit and necessary for improving their offerings.
In the coming segments, we look into two such companies that have made significant strides in the space where AR and entertainment converge.
1. Walt Disney (DIS +1.21%)
Reports published in November last year suggested that Disney had created a new strategic group to manage and coordinate how the company developed and deployed next-generation tech — like artificial intelligence and mixed reality. It was called the Office of Technology Enablement.
Jamie Voris, who had served as CTO of Walt Disney Studios for 14 years, was chosen to head up the newly formed division. Voris, in the new role, was to report to Disney Entertainment co-chairman Alan Bergman.
In an internal memo – one that was circulated before the launch of the new division – Bergman had the following to say:
“Our ability to remain at the forefront of technological advances will only be more critical as we move forward — making it all the more important to understand and embrace new technological shifts in ways that enable our people, creativity, and business.”
In that same memo, Bergman said that The Office of Technology Enablement was tasked with ensuring Disney was “a progressive, innovative, and responsible leader“ in AI and mixed reality.
More importantly, Bergman admitted that the pace and scope of advances in extended reality, which involves AR as one of its most significant components, are profound and will continue to impact Disney’s consumer experiences, creative endeavors, and business for years to come.
Experts and industry analysts believe that Disney’s leveraging of these technologies will be multifaceted. The exploration will span its various business divisions.
In the theme park sector, for instance, the company is assembling a dedicated team to investigate how AR and VR can enhance visitor experiences, aligning with broader industry trends. In a suggestive leadership move, Kyle Laughlin has also recently returned to Disney.
As the new senior vice president of research and development for Walt Disney Imagineering, experts believe that Kyle’s background in AR, VR, and AI positions him well to drive innovation in theme park attractions.
As far as specific uses of facial recognition technology are concerned, reports from December 2021 suggested that Disney Corporation was trialing the use of facial recognition software to check visitors arriving at its Disney World theme park in Florida – known as the Magic Kingdom.
The automated tagging pipeline created for Disney’s content repository came equipped with face detection and recognition modules that were applied to Disney’s library of content (shows, films, etc.).
The Walt Disney Company (DIS +1.21%)
On September 28, 2024, The Walt Disney Company (NYSE: DIS) reported its fourth quarter and full-year earnings. Revenues increased 6% for Q4 to $22.6 billion from $21.2 billion in the prior-year quarter and 3% for the year to $91.4 billion from $88.9 billion in the prior year.
2. Electronic Art Inc. (EA -0.41%)
Another company in the entertainment space that has been doing amazing work with augmented reality technologies and improving facial recognition substantially is Electronic Arts or EA. The company presented at SIGGRAPH Asia 2024, held in Tokyo, Japan, on the theory of stabilization by skull carving.
The company believed accurately stabilizing facial motion was essential for creating photo-real avatars for 3D games, virtual reality, movies, and machine-learning training data. Especially in the last case, the company believed facial stabilization needed to work quickly and automatically when the source data was a population of people with varying morphology.
The presenters of EA believed it was critical to distinguish rigid skull motions from facial expressions since misalignment between skull motion and facial expressions could result in an animation model that was hard to control and unsuitable for natural-looking motion.
However, all these were hard to achieve because existing stabilization methods had some drawbacks. These methods struggled to work with sparse sets of very different expressions, such as when combining multiple units from a facial action coding system (FACS). While there were other methods, the EA team found them to be not sufficiently robust.
As a solution, the EA team leveraged the latest advances in neural signed distance fields and differentiable isosurface meshing to compute skull stabilization rigid transforms directly on unstructured triangle meshes or point clouds. Their approach helped significantly enhance accuracy and robustness.
The team did not stop there. It went ahead to introduce the concept of a “stable hull“ as the surface of the boolean intersection of stabilized scans, analogous to the “visual hull“ (in shape-from-silhouette) and the “photo hull“ (in space carving). The stable hull resembles a skull overlaid with minimal soft tissue thickness, and upper teeth are automatically included.
The team claimed that their skull carving algorithm simultaneously optimizes the stable hull shape and rigid transforms to get accurate stabilization of complex expressions for a diverse set of people, outperforming existing methods.
EA has long been in favor of leveraging AR judiciously. Back in 2017, the EA CEO termed AR as ‘more interesting.’ It introduced Cranium Technology in EA SPORTS FC™ 25. The technology offered precise control over many aspects of the gaming character’s head model.
Users could use this technology for sculpting and textures to design a character’s head model with the spirit of a creator. It offered users more than just shaping—it allowed animating the user’s face to move naturally, dialing up immersion and realism.
Electronic Arts Inc. (EA -0.41%)
According to the latest available financial report, EA’s net bookings for Q3 FY25 totaled US$2.215 billion.
“The record success of our EA SPORTS FC 25 Team of the Year event demonstrates our creative teams’ ability to adapt, innovate, and execute at scale,” said Andrew Wilson, CEO of Electronic Arts.
Facial Projection Mapping: Continuing Research, Consistent Improvement
The transformative facial projection mapping technology research we started our article with was not achieved in a day. It carries forward a long tradition of research in this field.
For instance, in 2021, the School of Engineering, Tokyo Institute of Technology researchers published a paper titled ‘High-Speed Dynamic Projection Mapping onto Human Arm with Realistic Skin Deformation.‘ 2
In their opening remarks, the researchers admitted that dynamic projection mapping for a moving object according to its position and shape was fundamental for augmented reality to resemble changes on a target surface.
They specifically dealt with augmenting the human arm surface via dynamic projection mapping that could enhance applications in fashion, user interfaces, prototyping, education, medical assistance, and other fields. However, they identified some drawbacks.
They found that conventional methods neglected skin deformation and had a high latency between motion and projection, causing noticeable misalignment between the target arm surface and projected images.
As a solution, the researchers proposed a system for high-speed dynamic projection mapping onto a rapidly moving human arm with realistic skin deformation. With the developed system, the user did not perceive any misalignment between the arm surface and projected images.
To accomplish their task, the researchers first combined a state-of-the-art parametric deformable surface model with efficient regression-based accuracy compensation to represent skin deformation and modified the texture coordinates to achieve fast and accurate image generation for projection mapping based on joint tracking.
As a second step, the researchers developed a high-speed system that provided a latency between motion and projection below 10 ms, which was generally imperceptible by human vision.
Similar research was carried out by a team of researchers in 2017. The team had collaborators from Disney Research, Princeton University, Chalmers University, and Osaka University. The research was titled ‘Makeup Lamps: Live Augmentation of Human Faces via Projection.’3
In the paper, the researchers proposed the first system for live dynamic augmentation of human faces. Using projector-based illumination, they altered the appearance of human performers during novel performances. The key challenge of live augmentation, however, was latency.
An image was generated according to a specific pose but was displayed on a different facial configuration by the time it was projected. The system proposed by the researchers aimed at reducing latency during every step of the process, from capture through processing to projection.
Using infrared illumination, an optically and computationally aligned high-speed camera detected facial orientation as well as expression. The estimated expression blend shapes were mapped onto a lower dimensional space, and the facial motion and non-rigid deformation were estimated, smoothed, and predicted through adaptive Kalman filtering. Finally, the desired appearance was generated by interpolating precomputed offset textures according to time, global position, and expression.
The researchers evaluated their system through an optimized CPU and GPU prototype and demonstrated successful low-latency augmentation for different performers and performances with varying facial play and motion speed.
The researchers claimed, in contrast to existing methods, that their presented system was the first method that fully supported dynamic facial projection mapping without the requirement of any physical tracking markers and incorporated facial expressions.
Facial projection mapping and research to achieve stabilized output without latency and misalignment are not new. Researchers have been rigorously pursuing this path rigorously for its transformative capabilities. The latest DFPM technique is certainly a breakthrough and will help reimagine the entertainment with enhanced performance aspects.
Click here for a list of top augmented reality & virtual reality stocks.
Study Reference:
1. Peng, H.-L., Sato, K., Nakagawa, S., & Watanabe, Y. (2025). Perceptually-Aligned Dynamic Facial Projection Mapping by High-Speed Face-Tracking Method and Lens-Shift Co-Axial Setup. IEEE Transactions on Visualization and Computer Graphics, Early Access, 1-15. https://doi.org/10.1109/TVCG.2025.3527203
2. Peng, H.-L., & Watanabe, Y. (2021). High-Speed Dynamic Projection Mapping onto Human Arm with Realistic Skin Deformation. Applied Sciences, 11(9), 3753. https://doi.org/10.3390/app11093753
3. Bermano, A. H., Billeter, M., Iwai, D., & Grundhöfer, A. (2017). Makeup Lamps: Live Augmentation of Human Faces via Projection. Computer Graphics Forum, 36(2), 311-323. https://doi.org/10.1111/cgf.13128