发布日期: 2025-02-22

更新日期: 2025-02-22

文章字数: 1.3k

阅读时长: 5 分

阅读次数:

⚠️ 以下所有内容总结都来自于大语言模型的能力，如有错误，仅供参考，谨慎使用
🔴 请注意：千万不要用于严肃的学术场景，只能用于论文阅读前的初筛！
💗 如果您觉得我们的项目对您有帮助 ChatPaperFree ，还请您给我们一些鼓励！⭐️ HuggingFace免费体验

2025-02-22 更新

NeRF-3DTalker: Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis

Authors:Xiaoxing Liu, Zhilei Liu, Chongke Bi

Talking head synthesis is to synthesize a lip-synchronized talking head video using audio. Recently, the capability of NeRF to enhance the realism and texture details of synthesized talking heads has attracted the attention of researchers. However, most current NeRF methods based on audio are exclusively concerned with the rendering of frontal faces. These methods are unable to generate clear talking heads in novel views. Another prevalent challenge in current 3D talking head synthesis is the difficulty in aligning acoustic and visual spaces, which often results in suboptimal lip-syncing of the generated talking heads. To address these issues, we propose Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis (NeRF-3DTalker). Specifically, the proposed method employs 3D prior information to synthesize clear talking heads with free views. Additionally, we propose a 3D Prior Aided Audio Disentanglement module, which is designed to disentangle the audio into two distinct categories: features related to 3D awarded speech movements and features related to speaking style. Moreover, to reposition the generated frames that are distant from the speaker’s motion space in the real space, we have devised a local-global Standardized Space. This method normalizes the irregular positions in the generated frames from both global and local semantic perspectives. Through comprehensive qualitative and quantitative experiments, it has been demonstrated that our NeRF-3DTalker outperforms state-of-the-art in synthesizing realistic talking head videos, exhibiting superior image quality and lip synchronization. Project page: https://nerf-3dtalker.github.io/NeRF-3Dtalker.

说话人头部合成是通过音频合成一个唇同步的说话人头部视频。最近，NeRF技术因其增强合成说话人头部现实感和纹理细节的能力而引起了研究人员的关注。然而，当前大多数基于音频的NeRF方法主要关注正面人脸的呈现。这些方法无法在新视角生成清晰的说话人头部。当前3D说话人头部合成的另一个普遍挑战是声音和视觉空间的对齐困难，这通常导致生成的说话人头部唇同步不佳。为了解决这些问题，我们提出了用于说话人头部合成的带有3D先验辅助音频分离的神经网络辐射场（NeRF-3DTalker）。具体而言，该方法采用3D先验信息来合成具有自由视角的清晰说话人头部。此外，我们提出了3D先验辅助音频分离模块，旨在将音频分为两类：与3D获奖语音动作相关的特征和与演讲风格相关的特征。而且，为了重新定位在真实空间中远离说话者动作空间所产生的帧，我们从全局和局部语义角度设计了一个局部-全局标准化空间。该方法对生成帧中的不规则位置进行了归一化。通过全面的定性和定量实验，证明我们的NeRF-3DTalker在合成逼真的说话人头部视频方面优于现有技术，具有卓越的图像质量和唇同步效果。项目页面：https://nerf-3dtalker.github.io/NeRF-3Dtalker。

论文及项目相关链接

PDF Accepted by ICASSP 2025

Summary

本文介绍了基于音频合成说话人头部的技术，并指出当前面临的挑战。为此，提出了一种基于神经辐射场和三维先验辅助音频分离的方法（NeRF-3DTalker）。该方法利用三维先验信息合成清晰的可视化头部，并提出一种三维先验辅助音频分离模块，将音频分为与三维获奖语音动作相关的特征和与说话风格相关的特征。此外，为了调整生成帧的位置，开发了一种局部全局标准化空间。实验证明，NeRF-3DTalker在合成逼真的说话人头视频方面优于现有技术，具有优越的图像质量和唇同步效果。

Key Takeaways