Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

Last update: Dec 30, 2022

Related tags

Deep Learning Sync2Gen

Overview

Sync2Gen

Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

0. Environment

Environment: python 3.6 and cuda 10.0 on Ubuntu 18.04

Pytorch 1.4.0
tensorflow 1.14.0 (for tensorboard)

1. Dataset

├──dataset_3dfront/
    ├──data
        ├── bedroom
            ├── 0_abs.npy
            ├── 0_rel.pkl
            ├── ...
        ├── living
            ├── 0_abs.npy
            ├── 0_rel.pkl
            ├── ...
        ├── train_bedroom.txt
        ├── train_living.txt
        ├── val_bedroom.txt
        └── val_living.txt

See 3D-FRONT Dataset for dataset generation.

2. VAE

2.1 Generate scenes from random noises

Download the pretrained model from https://drive.google.com/file/d/1VKNlEdUj1RBUOjBaBxE5xQvfsZodVjam/view?usp=sharing

Sync2Gen
└── log
    └── 3dfront
        ├── bedroom
        │   └── vaef_lr0001_w00001_B64
        │       ├── checkpoint_eval799.tar
        │       └── pairs
        └── living
            └── vaef_lr0001_w00001_B64
                ├── checkpoint_eval799.tar
                └── pairs

type='bedroom'; # or living
CUDA_VISIBLE_DEVICES=0 python ./test_sparse.py  --type $type  --log_dir ./log/3dfront/$type/vaef_lr0001_w00001_B64 --model_dict=model_scene_forward --max_parts=80 --num_class=20 --num_each_class=4 --batch_size=32 --variational --latent_dim 20 --abs_dim 16  --weight_kld 0.0001  --learning_rate 0.001 --use_dumped_pairs --dump_results --gen_from_noise --num_gen_from_noise 100

The predictions are dumped in ./dump/$type/vaef_lr0001_w00001_B64

2.2 Training

To train the network:

type='bedroom'; # or living
CUDA_VISIBLE_DEVICES=0 python ./train_sparse.py --data_path ./dataset_3dfront/data  --type $type  --log_dir ./log/3dfront/$type/vaef_lr0001_w00001_B64  --model_dict=model_scene_forward --max_parts=80 --num_class=20 --num_each_class=4 --batch_size=64 --variational --latent_dim 20 --abs_dim 16  --weight_kld 0.0001  --learning_rate 0.001

3. Bayesian optimization

cd optimization

3.1 Prior generation

See Prior generation.

3.2 Optimization

type=bedroom # or living;
bash opt.sh $type vaef_lr0001_w00001_B64  EXP_NAME

We use Pytorch-LBFGS for optimization.

3.3 Visualization

There is a simple visualization tool:

type=bedroom # or living
bash vis.sh $type vaef_lr0001_w00001_B64 EXP_NAME

The visualization is in ./vis. {i:04d}_2(3)d_pred.png is the initial prediction from VAE. {i:04d}_2(3)d_sync.png is the optimized layout after synchronization.

Acknowledgements

The repo is built based on:

We thank the authors for their great job.

Contact

If you have any questions, you can contact Haitao Yang (yanghtr [AT] outlook [DOT] com).

Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

Related tags

Overview

Sync2Gen

0. Environment

1. Dataset

2. VAE

2.1 Generate scenes from random noises

2.2 Training

3. Bayesian optimization

3.1 Prior generation

3.2 Optimization

3.3 Visualization

Acknowledgements

Contact

Owner

Haitao Yang

Code for the CVPR 2021 paper: Understanding Failures of Deep Networks via Robust Feature Extraction

Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch

StackRec: Efficient Training of Very Deep Sequential Recommender Models by Iterative Stacking

Python script that takes an Impulse response .wav and a input .wav to demonstrate audio convolution.

Orthogonal Over-Parameterized Training

MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images

Code for "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clouds", CVPR 2021

Official Implementation of CVPR 2022 paper: "Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning"

This is the official source code of "BiCAT: Bi-Chronological Augmentation of Transformer for Sequential Recommendation".

Using pretrained GROVER to extract the atomic fingerprints from molecule

Summary of related papers on visual attention

kullanışlı ve işinizi kolaylaştıracak bir araç

Non-Homogeneous Poisson Process Intensity Modeling and Estimation using Measure Transport

Official Implementation of LARGE: Latent-Based Regression through GAN Semantics

Official implementation for (Refine Myself by Teaching Myself : Feature Refinement via Self-Knowledge Distillation, CVPR-2021)

Official PyTorch Implementation of Rank & Sort Loss [ICCV2021]

Does Oversizing Improve Prosumer Profitability in a Flexibility Market? - A Sensitivity Analysis using PV-battery System

Draw like Bob Ross using the power of Neural Networks (With PyTorch)!

RGBD-Net - This repository contains a pytorch lightning implementation for the 3DV 2021 RGBD-Net paper.

MoViNets PyTorch implementation: Mobile Video Networks for Efficient Video Recognition;