1/2
@CVPR HighLight Paper Alert
Paper Title: DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis
Few pointers from the paper
In this paper authors have presented “DiffPortrait3D”, a conditional diffusion model that is capable of synthesizing 3D-consistent photo-realistic novel views from as few as a single in-the-wild portrait.
Specifically, given a single RGB input, authors aimed to synthesize plausible but consistent facial details rendered from novel camera views with retained both identity and facial expression.
In lieu of time-consuming optimization and fine-tuning, their zero-shot method generalizes well to arbitrary face portraits with unposed camera views, extreme facial expressions, and diverse artistic depictions.
At its core, they leveraged the generative prior of 2D diffusion models pre-trained on large-scale image datasets as their rendering backbone, while the denoising is guided with disentangled attentive control of appearance and camera pose.
To achieve this, they first inject the appearance context from the reference image into the self-attention layers of the frozen UNets. The rendering view is then manipulated with a novel conditional control module that interprets the camera pose by watching a condition image of a crossed subject from the same view.
Furthermore, they inserted a trainable cross- view attention module to enhance view consistency, which is further strengthened with a novel 3D-aware noise generation process during inference.
Organization: @USC , @BytedanceTalk Inc
Paper Authors: Yuming Gu, You Xie, Hongyi Xu, Guoxian Song, Yichun Shi, @DiChang10 , @jingyangcarl , @linjieluo_t
Read the Full Paper here: [2312.13016] DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis
Project Page: DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis
Code: GitHub - FreedomGu/DiffPortrait3D: Official Repository of [CVPR'24 Highlight Diffportrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis]
Be sure to watch the attached Demo Video-Sound on 
Music by Mike Kripak from @pixabay
Find this Valuable
?
QT and teach your network something new
Follow me
, @NaveenManwani17 , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements.
/search?q=#CVPR2024Highlight
2/2
@yuming_gu
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
@CVPR HighLight Paper Alert Paper Title: DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View SynthesisFew pointers from the paperIn this paper authors have presented “DiffPortrait3D”, a conditional diffusion model that is capable of synthesizing 3D-consistent photo-realistic novel views from as few as a single in-the-wild portrait.Specifically, given a single RGB input, authors aimed to synthesize plausible but consistent facial details rendered from novel camera views with retained both identity and facial expression.In lieu of time-consuming optimization and fine-tuning, their zero-shot method generalizes well to arbitrary face portraits with unposed camera views, extreme facial expressions, and diverse artistic depictions.At its core, they leveraged the generative prior of 2D diffusion models pre-trained on large-scale image datasets as their rendering backbone, while the denoising is guided with disentangled attentive control of appearance and camera pose.To achieve this, they first inject the appearance context from the reference image into the self-attention layers of the frozen UNets. The rendering view is then manipulated with a novel conditional control module that interprets the camera pose by watching a condition image of a crossed subject from the same view.Furthermore, they inserted a trainable cross- view attention module to enhance view consistency, which is further strengthened with a novel 3D-aware noise generation process during inference.Organization: @USC , @BytedanceTalk IncPaper Authors: Yuming Gu, You Xie, Hongyi Xu, Guoxian Song, Yichun Shi, @DiChang10 , @jingyangcarl , @linjieluo_t Read the Full Paper here: [2312.13016] DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View SynthesisProject Page: DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View SynthesisCode: GitHub - FreedomGu/DiffPortrait3D: Official Repository of [CVPR'24 Highlight Diffportrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis] Be sure to watch the attached Demo Video-Sound on Music by Mike Kripak from @pixabay Find this Valuable
?QT and teach your network something newFollow me
, @NaveenManwani17 , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements./search?q=#CVPR2024Highlight
2/2
@yuming_gu
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
These shortcomings hinder performance in scenarios with occlusion, dynamic objects, and
Grouping is inherently ambiguous due to the multiple levels of granularity in which one can decompose a scene— should the wheels of an excavator be considered separate or part of the whole?
Data: 
the difficulty of accurately estimating eye poses
In this paper authors have presented “SimXR”, a method for controlling a simulated avatar from information (headset pose and cameras) obtained from AR/VR headsets.
️ Hardware Tutorial
Product Update 