1/2
@CVPR HighLight Paper Alert
Paper Title: DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis
Few pointers from the paper
In this paper authors have presented “DiffPortrait3D”, a conditional diffusion model that is capable of synthesizing 3D-consistent photo-realistic novel views from as few as a single in-the-wild portrait.
Specifically, given a single RGB input, authors aimed to synthesize plausible but consistent facial details rendered from novel camera views with retained both identity and facial expression.
In lieu of time-consuming optimization and fine-tuning, their zero-shot method generalizes well to arbitrary face portraits with unposed camera views, extreme facial expressions, and diverse artistic depictions.
At its core, they leveraged the generative prior of 2D diffusion models pre-trained on large-scale image datasets as their rendering backbone, while the denoising is guided with disentangled attentive control of appearance and camera pose.
To achieve this, they first inject the appearance context from the reference image into the self-attention layers of the frozen UNets. The rendering view is then manipulated with a novel conditional control module that interprets the camera pose by watching a condition image of a crossed subject from the same view.
Furthermore, they inserted a trainable cross- view attention module to enhance view consistency, which is further strengthened with a novel 3D-aware noise generation process during inference.
Organization: @USC , @BytedanceTalk Inc
Paper Authors: Yuming Gu, You Xie, Hongyi Xu, Guoxian Song, Yichun Shi, @DiChang10 , @jingyangcarl , @linjieluo_t
Read the Full Paper here: [2312.13016] DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis
Project Page: DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis
Code: GitHub - FreedomGu/DiffPortrait3D: Official Repository of [CVPR'24 Highlight Diffportrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis]
Be sure to watch the attached Demo Video-Sound on
Music by Mike Kripak from @pixabay
Find this Valuable ?
QT and teach your network something new
Follow me , @NaveenManwani17 , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements.
/search?q=#CVPR2024Highlight
2/2
@yuming_gu
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
@CVPR HighLight Paper Alert
Paper Title: DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis
Few pointers from the paper
In this paper authors have presented “DiffPortrait3D”, a conditional diffusion model that is capable of synthesizing 3D-consistent photo-realistic novel views from as few as a single in-the-wild portrait.
Specifically, given a single RGB input, authors aimed to synthesize plausible but consistent facial details rendered from novel camera views with retained both identity and facial expression.
In lieu of time-consuming optimization and fine-tuning, their zero-shot method generalizes well to arbitrary face portraits with unposed camera views, extreme facial expressions, and diverse artistic depictions.
At its core, they leveraged the generative prior of 2D diffusion models pre-trained on large-scale image datasets as their rendering backbone, while the denoising is guided with disentangled attentive control of appearance and camera pose.
To achieve this, they first inject the appearance context from the reference image into the self-attention layers of the frozen UNets. The rendering view is then manipulated with a novel conditional control module that interprets the camera pose by watching a condition image of a crossed subject from the same view.
Furthermore, they inserted a trainable cross- view attention module to enhance view consistency, which is further strengthened with a novel 3D-aware noise generation process during inference.
Organization: @USC , @BytedanceTalk Inc
Paper Authors: Yuming Gu, You Xie, Hongyi Xu, Guoxian Song, Yichun Shi, @DiChang10 , @jingyangcarl , @linjieluo_t
Read the Full Paper here: [2312.13016] DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis
Project Page: DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis
Code: GitHub - FreedomGu/DiffPortrait3D: Official Repository of [CVPR'24 Highlight Diffportrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis]
Be sure to watch the attached Demo Video-Sound on
Music by Mike Kripak from @pixabay
Find this Valuable ?
QT and teach your network something new
Follow me , @NaveenManwani17 , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements.
/search?q=#CVPR2024Highlight
2/2
@yuming_gu
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196