Code for Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid, accepted by Pattern Recognition'23.

This project is for our new inpainting method SPN which is under peer review. This work is an extension version of our previous work SPL (IJCAI'21). If you have any questions, feel free to make issues. Thanks for your interests!

Update 2023.4 We have added the function ``Kmeans_map(x)'' in main.py file in both setups. This function is used to draw the K-means cluster results in our experiments part.

Briefly speaking, in this work, we still focus on the key insight that learning semantic priors from specific pretext tasks can benefit image inpainting, and we further strengthen the modeling of the learned priors in this work from the following aspects:

1. We exploit multi-scale semantic priors in a feature pyramid manner to achieve consistent understanding of both gloabl and local context. The image generator is also improved to incorporate the prior pyramid.
2. We extend our prior learned in a probabilistic manner which enables our method to handle probabilistic image inpainting problem.
3. Besides, more analyses of the learned prior pyramid and the choices of the semantic supervision are provided in our experiment part.

• Python 3.7
• PyTorch 1.8 (1.6+ may also work)
• NVIDIA GPU + CUDA cuDNN
• Inplace_Abn (only needed for training our model, used in ASL_TRresNet model)
• torchlight (We only use it to record the printed information. You can change it as you want.)

We use Places2, CelebA and Paris Street-View datasets for determinstic image inpainting which is same with SPL, and CelebA-HQ dataset is used for probabilistic image inpainting. We also used the irregular mask provided by Liu et al. which can be downloaded from their website. For the detailed processes of these datasets please refer to SPL and our paper.

Since our approach can be applied for both deterministic and probabilistic image inpainting, so we seperate the codes under these two setups in different files and each file contains corresponding training and testing commonds.

For all setups, the common pre-preparations are list as follows:

1. Download the pre-trained models and copy them under ./checkpoints directory.

2. (For training) Make another directory, e.g ./pretrained_ASL, and download the weights of TResNet_L pretrained on OpenImage dataset to this directory.

3. Install torchlight

cd ./torchlight
python setup.py install