Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers

beyond masking

Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers

The code is coming

Figure 1: Pipeline of token-based pre-training.

Figure 2: The visualization of the proposed 5 tasks.

main results

All the results are pre-trained for 300 epochs using Vit-base as default.

Effiencicy

Figure 3: Efficiency of the integrated task.