STERR-GAN: Spatio-temporal Re-rendering for Antique Facial Video Restoration

Manh-Khanh Ngo Huu*, Vinh Quang Ngo*, Duc-Tuan Luu, Bich-Nga Pham, Vinh Tiep Nguyen,
1University of Information Technology, HCM-VNU
Paper arXiv Video Code Data

STERR-GAN restores old facial films with various deterioration, such as low-resolution, noise, blur, compression artifacts, etc. to high-quality videos. The results are comparable to the state-of-the-art methods.

Abstract

Facial old films are an essential part of our cultural heritage and can provide valuable imagination of historical figures. However, over time, films can become degraded and damaged, making them difficult or impossible to watch. Film restoration is the process of repairing and preserving old films so that future generations can enjoy them. Not limited to preserving historical video footage, video restoration can apply to our life, such as security, crime prevention, and investigation.

Despite the fact that there are some research on facial image restoration, the work on facial video restoration still inadequate. In this thesis, we focus on facial video restoration, with the input is a video of faces that suffer from various deterioration issues, such as scratches, film grain noise, monochrome, and the output is its color high-quality video.

From our observation, recent facial image restoration models can recover faces from old images. Still, they are designed for image restoration and do not leverage temporal information, as a result, it struggles with flickering problems. We propose Spatio-temporal Re-rendering for Antique Facial Video Restoration (STERR-GAN), which (1) adopts a generative prior for facial restoration ("Re-rendering" term) and (2) spatial information ("Spatio-temporal" term) to avoid flickering artifact. Besides proposing a new design, we also utilize stable loss to enhance the illumance stability of result videos.

In addition, we find out that there are many datasets for video restoration task or facial image restoration task, but the dataset for facial video restoration is insufficient. Hence, we introduce Video dataset for Antique Restoration (VAR). We believe that it will be a valuable resource for measuring the performance of future models and advancing research in this study area.

More results

( Input - DeepRemaster [1] - GPF-GAN [2] - RTN [3] - STERR-GAN (Our) )

Related Links

[1] "DeepRemaster: temporal source-reference attention networks for comprehensive video enhancement". Iizuka, S. and Simo-Serra, E. ACM Transactions on Graphics (TOG), 38(6):1–13. 2019

[2] "GPF-GAN: Towards real-world blind face restoration with generative facial prior". Wang, X., Li, Y., Zhang, H., and Shan, Y. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2021

[3] "Bringing Old Films Back to Life". Wan, Ziyu and Zhang, Bo and Chen, Dongdong and Liao, Jing. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022

BibTeX

@article{sterrgan,
  author    = {Manh-Khanh Ngo Huu, Vinh Quang Ngo, Vinh Tiep Nguyen},
  title     = {STERR-GAN: Spatio-temporal Re-rendering for Antique Facial Video Restoration},
  year      = {2022}
}