Generic Event Boundary Detection: A Benchmark for Event Segmentation

Related tags

Deep LearningGEBD
Overview

Generic Event Boundary Detection: A Benchmark for Event Segmentation

We release our data annotation & baseline codes for detecting generic event boundaries in video.

Links: [Arxiv] [LOVEU Challenge]

Contributors: Mike Zheng Shou, Stan Lei, Deepti Ghadiyaram, Weiyao Wang, Matt Feiszli.

Overview

This repo has the following structure:

./
│   LICENSE
│   README.md
│   INSTRUCTIONS.md
│
├───BdyDet
│   ├───k400
│   │       detect_event_boundary.py
│   │       run_multiprocess_detect_event_boundary.py
│   │
│   └───TAPOS
│           detect_event_boundary.py
│           run_multiprocess_detect_event_boundary.py
│
├───Challenge_eval_Code
│       eval.py
│       README.md
│
├───data
│   ├───export
│   │       prepare_hmdb_release.ipynb
│   │       prepare_k400_release.ipynb
│   │
│   ├───exp_k400
│   │   │   classInd.txt
│   │   │   val_set.csv
│   │   │
│   │   ├───detect_seg
│   │   └───pred_err
│   └───exp_TAPOS
│       │   train_set.csv
│       │   val_set.csv
│       │
│       ├───detect_seg
│       └───pred_err
├───eval
│       eval_GEBD_k400.ipynb
│       eval_GEBD_TAPOS.ipynb
│
├───PA_DPC
│   │   LICENSE
│   │   README.md
│   │
│   ├───asset
│   │       arch.png
│   │
│   ├───backbone
│   │       convrnn.py
│   │       resnet_2d.py
│   │       resnet_2d3d.py
│   │       select_backbone.py
│   │
│   ├───dpc
│   │       dataset_3d_infer_pred_error.py
│   │       main_infer_pred_error.py
│   │       model_3d.py
│   │
│   └───utils
│           augmentation.py
│           utils.py
│
└───PC
    │   PC_test.py
    │   PC_train.py
    │   README.md
    │
    ├───DataAssets
    ├───datasets
    │       augmentation.py
    │       MultiFDataset.py
    │
    ├───modeling
    │       resnetGEBD.py
    │
    ├───run
    │       pc_k400_dist.sh
    │       pc_tapos_dist.sh
    │
    └───utils
            augmentation.py
            checkpoint_saver.py
            augmentation.py
            checkpoint_saver.py
            clip_grad.py
            cuda.py
            getter.py
            helper.py
            log.py
            metric.py
            model_ema.py
            optim_factory.py
            sampler.py
            scheduler.py

Note that we release codes on Github. Annotations are available on GoogleDrive. Run the code by yourself to generate the output files. Refer to INSTRUCTIONS for preparing data and generating submission files.

  • data/:

    • export/ folder stores temporal boundary annotations of our Kinetics-GEBD and HMDB-GEBD datasets; download our raw annotations and put them under this folder.
    • exp_k400/ and exp_TAPOS/ store intermediate experimental data and final results.
  • Challenge_eval_Code/: codes for evaluation in LOVEU Challenge Track 1.

  • BdyDet/: codes for detecting boundary positions based on predictability sequence.

  • eval/: codes for evaluating the performance of boundary detection.

  • PA_DPC/: codes for computing the predictability sequence using various methods.

  • PC/: codes for supervised baseline on GEBD.

data/

  • In data/export/:

    • *_raw_annotation.pkl stores the raw annotations; download raw annotations here.
    • we further filter out videos that receives <3 annotations and conduct pre-processing e.g. merge very close boundaries - we use notebook prepare_*_release.ipynb and the output is stored in *_mr345_min_change_duration0.3.pkl.
  • Some fields in *_raw_annotation.pkl:

    • fps: frames per second.
    • video_duration:video duration in second.
    • f1_consis: a list of consistency scores, each score corresponds to a specific annotator’s score as compared to other annotators.
    • substages_timestamps: a list of annotations from each annotator; each annotator’s annotation is again a list of boundaries. time in second; for 'label', it is of format A: B.
      • If A is “ShotChangeGradualRange”, B could be “Change due to Cut”, “Change from/to slow motion”, “Change from/to fast motion”, or “N/A”.
      • If A is “ShotChangeImmediateTimestamp”, B could be “Change due to Pan”, “Change due to Zoom”, “Change due to Fade/Dissolve/Gradual”, “Multiple”, or “N/A”.
      • If A is “EventChange (Timestamp/Range)”, B could be “Change of Color”, “Change of Actor/Subject”, “Change of Object Being Interacted”, “Change of Action”, “Multiple”, or “N/A”.
  • Some fields in *_mr345_min_change_duration0.3.pkl:

    • substages_myframeidx: the term of substage is following the convention in TAPOS to refer to boundary position -- here we store each boundary’s frame index which starts at 0.
  • In data/exp_k400/ and data/exp_TAPOS/:

    • pred_err/ stores output of PA_DPC/ i.e. the predictability sequence;
    • detect_seg/ stores output of BdyDet/ i.e. detected boundary positions.

Challenge_eval_Code/

  • We use eval.py for evaluation in our competition.

  • Although one can use frame_index and number of total frames to measure the Rel.Dis, as we implemented in eval/, you should represent the detected boundaries with timestamps (in seconds). For example:

    {
    ‘6Tz5xfnFl4c’: [5.9, 9.4], # boundaries detected at 5.9s, 9.4s of this video
    ‘zJki61RMxcg’: [0.6, 1.5, 2.7] # boundaries detected at 0.6s, 1.5s, 2.7s of this video
    ...
    }
  • Refer to this file to generate GT files from raw annotations.

BdyDet/

Change to directory ./BdyDet/k400 or ./BdyDet/TAPOS and run the following command, which will launch multiple processes of detect_event_boundary.py.

(Note to set the number of processes according to your server in detect_event_boundary.py.)

python run_multiprocess_detect_event_boundary.py

PA_DPC/

Our implementation is based on the [DPC] framework. Please refer to their README or website to learn installation and usage. In the below, we only explain how to run our scripts and what are our modifications.

  • Modifications at a glance

    • main_infer_pred_error.py runs in only inference mode to assess predictability over time.
    • dataset_3d_infer_pred_error.py contains loaders for two datasets i.e. videos from Kinetics and truncated instances from TAPOS.
    • model_3d.py adds several classes for feature extraction purposes, e.g. ResNet feature before the pooling layer, to enable computing feature difference directly based on ImageNet pretrained model, in contrast to the predictive model in DPC.
  • How to run?

    change to directory ./PA_DPC/dpc/

    • Kinetics-GEBD:
    python main_infer_pred_error.py --gpu 0 --model resnet50-beforepool --num_seq 2 --pred_step 1 --seq_len 5 --dataset k400 --batch_size 160 --img_dim 224 --mode val --ds 3 --pred_task featdiff_rgb
    • TAPOS:
    python main_infer_pred_error.py --gpu 0 --model resnet50-beforepool --num_seq 2 --pred_step 1 --seq_len 5 --dataset tapos_instances --batch_size 160 --img_dim 224 --mode val --ds 3 --pred_task featdiff_rgb
    • Notes:

      • ds=3 means we downsample the video frames by 3 to reduce computation.

      • seq_len=5 means 5 frames for each step (concept used in DPC).

      • num_seq=2and pred_step=1 so that at a certain time position t, we use 1 step before and 1 step after to assess the predictability at time t; thus the predictability at time t is the feature difference between the average feature of 5 sampled frames before t and the average feature of 5 sampled frames after t.

      • we slide such model over time to obtain predictability at different temporal positions. More details can be found in dataset_3d_infer_pred_error.py. Note that window_lists.pkl stores the index of frames to be sampled for every window - since it takes a long time to compute (should be able to optimize in the future), we store its value and just load this pre-computed value in the future runs on the same dataset during experimental explorations.

PC/

PC is a supervised baseline for GEBD task. In the PC framework, for each frame f in a video, we take T frames preceding f and T frames succeeding f as inputs, and then build a binary classifier to predict if f is boundary or background.

  • Get Started

    • Check PC/datasets/MultiFDataset.py to generate GT files for training. Note that you should prepare k400_mr345_*SPLIT*_min_change_duration0.3.pkl for Kinetics-GEBD and TAPOS_*SPLIT*_anno.pkl (this should be organized as k400_mr345_*SPLIT*_min_change_duration0.3.pkl for convenience) for TAPOS before running our code.
    • You should accordingly change PATH_TO in our codes to your data/frames path as needed.
  • How to run?

    Change directory to ./PC.

    • Train on Kinetics-GEBD:

      CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 PC_train.py \
      --dataset kinetics_multiframes \
      --train-split train \
      --val-split val \
      --num-classes 2 \
      --batch-size 32 \
      --n-sample-classes 2 \
      --n-samples 16 \
      --lr 0.01 \
      --warmup-epochs 0 \
      --epochs 30 \
      --decay-epochs 10 \
      --model multiframes_resnet \
      --pin-memory \
      --balance-batch \
      --sync-bn \
      --amp \
      --native-amp \
      --eval-metric loss \
      --log-interval 50
    • Train on TAPOS:

      CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 PC_train.py \
      --dataset tapos_multiframes \
      --train-split train \
      --val-split val \
      --num-classes 2 \
      --batch-size 32 \
      --n-sample-classes 2 \
      --n-samples 16 \
      --lr 0.01 \
      --warmup-epochs 0 \
      --epochs 30 \
      --decay-epochs 10 \
      --model multiframes_resnet \
      --pin-memory \
      --balance-batch \
      --sync-bn \
      --amp \
      --native-amp \
      --eval-metric loss \
      --log-interval 50 
    • Generate scores sequence on Kinetics-GEBD Validation Set:

      CUDA_VISIBLE_DEVICES=0 python PC_test.py \ 
      --dataset kinetics_multiframes \
      --val-split val \
      --resume path_to/checkpoint
    • Generate scores sequence on TAPOS:

      CUDA_VISIBLE_DEVICES=0 python PC_test.py \ 
      --dataset tapos_multiframes \
      --val-split val \
      --resume path_to/checkpoint
  • Models

Misc

  • Download datasets from Kinetics-400 and TAPOS. (Note that some of the videos can not be downloaded from YouTube for some reason, you can go ahead with those available.)

  • Note that for TAPOS, you need to cut out each action instance by yourself first and then can use our following codes to process each instance's video separately.

  • Extract frames of videos in the dataset.

  • To reproduce PA_DPC, generate your own data/exp_*/val_set.csv, in which path to the folder of video frames and the number of frames in that specific video should be contained.

Q&A

For any questions, welcome to create an issue or email Mike ([email protected]) and Stan ([email protected]). Thank you for helping us improve our data & codes.

An NVDA add-on to split screen reader and audio from other programs to different sound channels

An NVDA add-on to split screen reader and audio from other programs to different sound channels (add-on idea credit: Tony Malykh)

Joseph Lee 7 Dec 25, 2022
Code accompanying the paper Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs (Chen et al., CVPR 2020, Oral).

Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs This repository contains PyTorch implementation of our pa

Shizhe Chen 178 Dec 29, 2022
The repository contain code for building compiler using puthon.

Building Compiler This is a python implementation of JamieBuild's "Super Tiny Compiler" Overview JamieBuilds developed a wonderfully educative compile

Shyam Das Shrestha 1 Nov 21, 2021
Code samples for my book "Neural Networks and Deep Learning"

Code samples for "Neural Networks and Deep Learning" This repository contains code samples for my book on "Neural Networks and Deep Learning". The cod

Michael Nielsen 13.9k Dec 26, 2022
PASSL包含 SimCLR,MoCo,BYOL,CLIP等基于对比学习的图像自监督算法以及 Vision-Transformer,Swin-Transformer,BEiT,CVT,T2T,MLP_Mixer等视觉Transformer算法

PASSL Introduction PASSL is a Paddle based vision library for state-of-the-art Self-Supervised Learning research with PaddlePaddle. PASSL aims to acce

186 Dec 29, 2022
Federated learning on graph, especially on graph neural networks (GNNs), knowledge graph, and private GNN.

Federated learning on graph, especially on graph neural networks (GNNs), knowledge graph, and private GNN.

keven 198 Dec 20, 2022
Evaluating Cross-lingual Sentence Representations

XNLI: The Cross-Lingual NLI Corpus XNLI is an evaluation corpus for language transfer and cross-lingual sentence classification in 15 languages. New:

Meta Research 395 Dec 19, 2022
Pytorch implementation AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

AttnGAN Pytorch implementation for reproducing AttnGAN results in the paper AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative

Tao Xu 1.2k Dec 26, 2022
Multiview Dataset Toolkit

Multiview Dataset Toolkit Using multi-view cameras is a natural way to obtain a complete point cloud. However, there is to date only one multi-view 3D

11 Dec 22, 2022
The MATH Dataset

Measuring Mathematical Problem Solving With the MATH Dataset This is the repository for Measuring Mathematical Problem Solving With the MATH Dataset b

Dan Hendrycks 267 Dec 26, 2022
Tensor-Based Quantum Machine Learning

TensorLy_Quantum TensorLy-Quantum is a Python library for Tensor-Based Quantum Machine Learning that builds on top of TensorLy and PyTorch. Website: h

TensorLy 85 Dec 03, 2022
Implementation of paper "Towards a Unified View of Parameter-Efficient Transfer Learning"

A Unified Framework for Parameter-Efficient Transfer Learning This is the official implementation of the paper: Towards a Unified View of Parameter-Ef

Junxian He 216 Dec 29, 2022
A repo with study material, exercises, examples, etc for Devnet SPAUTO

MPLS in the SDN Era -- DevNet SPAUTO Get right to the study material: Checkout the Wiki! A lab topology based on MPLS in the SDN era book used for 30

Hugo Tinoco 67 Nov 16, 2022
Scientific Computation Methods in C and Python (Open for Hacktoberfest 2021)

Sci - cpy README is a stub. Do expand it. Objective This repository is meant to be a ready reference for scientific computation methods. Do ⭐ it if yo

Sandip Dutta 7 Oct 12, 2022
A playable implementation of Fully Convolutional Networks with Keras.

keras-fcn A re-implementation of Fully Convolutional Networks with Keras Installation Dependencies keras tensorflow Install with pip $ pip install git

JihongJu 202 Sep 07, 2022
Sibur challange 2021 competition - 6 place

sibur challange 2021 Решение на 6 место: https://sibur.ai-community.com/competitions/5/tasks/13 Скор 1.4066/1.4159 public/private. Архитектура - однос

Ivan 5 Jan 11, 2022
A collection of Reinforcement Learning algorithms from Sutton and Barto's book and other research papers implemented in Python.

Reinforcement-Learning-Notebooks A collection of Reinforcement Learning algorithms from Sutton and Barto's book and other research papers implemented

Pulkit Khandelwal 1k Dec 28, 2022
This is the official Pytorch-version code of FlatGCN (Flattened Graph Convolutional Networks for Recommendation).

FlatGCN This is the official Pytorch-version code of FlatGCN (Flattened Graph Convolutional Networks for Recommendation, submitted to ICASSP2022). Req

Dreamer 2 Aug 09, 2022
Storage-optimizer - Identify potintial optimizations on the cloud storage accounts

Storage Optimizer Identify potintial optimizations on the cloud storage accounts

Zaher Mousa 1 Feb 13, 2022
Speech-Emotion-Analyzer - The neural network model is capable of detecting five different male/female emotions from audio speeches. (Deep Learning, NLP, Python)

Speech Emotion Analyzer The idea behind creating this project was to build a machine learning model that could detect emotions from the speech we have

Mitesh Puthran 965 Dec 24, 2022