Pixel Consensus Voting for Panoptic Segmentation (CVPR 2020)

Implementation for Pixel Consensus Voting (CVPR 2020).

This codebase contains

• the essential ingredients of PCV, including various spatial discretization schemes and convolutional backprojection inference. The network backbone is a simple FPN on ResNet.
• Visualzier 1 (src/vis.py): loads a single image into a dynamic, interacive interface that allows users to click on pixels to inspect model prediction. It is built on matplotlib interactive API and jupyter widgets. Under the hood it's React.
• Visualizer 2 (src/pan_vis.py): A global inspector that take panoptic segmentation prediction and displays prediction segments against ground truth. Useful to track down which images make the most serious error and how.

Quick walkthrough

• The core of PCV is contained in src/pcv. The results reported in the paper uses src/pcv/pcv_basic. There are also a few modification ideas that didn't work out e.g. "inner grid collapse" (src/pcv/pcv_igc), erasing boundary loss src/pcv/pcv_boundless, smoothened gt assignment src/pcv/pcv_smooth.

• The deconv voting filter weight intializaiton is in src/pcv/components/ballot.py. Different deconv discretization schemes can be found in src/pcv/components/grid_specs.py. src/pcv/components/snake.py manages the generation of snake grid on which pcv operates.

• The backprojection code is in src/pcv/inference/mask_from_vote.py. Since this is a non-standard procedure of convolving a filter to do equality comparison, I implemented a simple conv using advanced indexing. See the function src/pcv/inference/mask_from_vote.py:unroll_img_inds.

• The main entry point is run.py and src/entry.py

• The rest of the codebase are pretty self explanatory.

Current status

• Many of the modules are self-contained, but the code does need repairing to run properly. I don't plan to maintain it at the moment, but feel free to email me if you have any questions.