"Video Moment Retrieval from Text Queries via Single Frame Annotation" in SIGIR 2022.

Overview

ViGA: Video moment retrieval via Glance Annotation

This is the official repository of the paper "Video Moment Retrieval from Text Queries via Single Frame Annotation" published in SIGIR 2022.

https://arxiv.org/abs/2204.09409

Dependencies

This project has been tested on the following conda environment.

$ conda create --name viga python=3.7
$ source activate viga
(viga)$ conda install pytorch=1.10.0 cudatoolkit=11.3.1
(viga)$ pip install numpy scipy pyyaml tqdm 

Data preparation

This repository contains our glance annotations already. To replicate our work, one should prepare extra data and finally get the following structure.

ckpt/                                 our pre-trained model, available at https://drive.google.com/file/d/1S4e8XmIpiVFJKSSJ4Tig4qN0yaCwiVLs/view?usp=sharing
data/
+-- activitynetcaptions/
|   +-- c3d/                    
|   +-- annotations/
|   |   +-- glance/
|   |   |   +-- train.json                
|   |   |   +-- val_1.json                
|   |   |   +-- val_2.json   
|   |   +-- train.json                downloaded
|   |   +-- val_1.json                downloaded
|   |   +-- val_2.json                downloaded
+-- charadessta/
|   +-- i3d/                     
|   +-- c3d/ 
|   +-- vgg/
|   +-- annotations/
|   |   +-- glance/
|   |   |   +-- charades_sta_train.txt
|   |   |   +-- charades_sta_test.txt
|   |   +-- charades_sta_train.txt    downloaded
|   |   +-- charades_sta_test.txt     downloaded
|   |   +-- Charades_v1_train.csv     downloaded
|   |   +-- Charades_v1_test.csv      downloaded
+-- tacos/
|   +-- c3d/ 
|   +-- annotations/
|   |   +-- glance/
|   |   |   +-- train.json                
|   |   |   +-- test.json                 
|   |   |   +-- val.json
|   |   +-- train.json                downloaded
|   |   +-- test.json                 downloaded
|   |   +-- val.json                  downloaded
glove.840B.300d.txt                   downloaded from https://nlp.stanford.edu/data/glove.840B.300d.zip

1. ActivityNet Captions

c3d feature

Downloaded from http://activity-net.org/challenges/2016/download.html. We extracted the features from sub_activitynet_v1-3.c3d.hdf5 as individual files.

Folder contains 19994 vid.npys, each of shape (T, 500).

annotation

Downloaded from https://cs.stanford.edu/people/ranjaykrishna/densevid/

2. Charades-STA

c3d feature

We extracted this by ourselves, due to the lack of storage resource we are currently not able to make this feature publicly available.

Folder contains 9848 vid.npys, each of shape (T, 4096).

i3d feature

Downloaded from https://github.com/JonghwanMun/LGI4temporalgrounding. This is the features extracted from I3D (finetuned on Charades). We processed them by trimming off unnecessary dimensions.

Folder contains 9848 vid.npys, each of shape (T, 1024).

vgg feature

Downloaded from https://github.com/microsoft/2D-TAN. We processed the data by converting the downloaded version vgg_rgb_features.hdf5 into numpy arrays.

Folder contains 6672 vid.npys, each of shape (T, 4096).

annotation

Downloaded from https://github.com/jiyanggao/TALL

3. TACoS

c3dfeature

Downloaded from https://github.com/microsoft/2D-TAN. We extracted the features from tall_c3d_features.hdf5 as individual files.

Folder contains 127 vid.npys, each of shape (T, 4096).

annotation

Downloaded from https://github.com/microsoft/2D-TAN

2. Run

Our models were trained using the following commands.

(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.train --task activitynetcaptions
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.train --task charadessta
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.train --task tacos

Our trained models were evaluated using the following commands.

(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/activitynetcaptions
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/charadessta_c3d
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/charadessta_i3d
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/charadessta_vgg
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/tacos
Owner
Ran Cui
Ran Cui
Community Repository for Unofficial Saltbox Add-ons

Saltbox Sandbox Repo Community Repository for Unofficial Saltbox Add-ons Requirements Saltbox Documentation Undetermined Roles List of roles can be fo

Salty Organization 31 Dec 19, 2022
A python tool capable of creating HUGE wordlists. Has the ability to add custom words for concatenation in any way you see fit.

A python tool capable of creating HUGE wordlists. Has the ability to add custom words for concatenation in any way you see fit.

Codex 9 Oct 05, 2022
Website OSINT untuk mencari informasi dari email dan nomor telepon. Dibuat dengan React dan Flask.

Inspektur Cari informasi mengenai email dan nomor telepon dengan mudah. Inspektur adalah aplikasi OSINT yang berguna untuk mencari informasi berdasark

Bagas Wastu 36 Dec 04, 2022
Shell hunter for AF

AF-ShellHunter AF-ShellHunter: Auto shell lookup AF-ShellHunter its a script designed to automate the search of WebShell's in AF Team How to pip3 ins

Eduardo 34 May 13, 2022
Log4Shell Proof of Concept (CVE-2021-44228)

CVE-2021-44228 Log4Shell Proof of Concept (CVE-2021-44228) Make sure to use Java 8 JDK. Java 8 Download Images Credits Casey Dunham - Java Reverse She

Kr0ff 3 Jul 23, 2022
Script Crack Facebook Yang Kaya Akan Teh Hijau 🚶‍♂

r-mbf Script Crack Facebook 🚶‍♂ Bukti Recode [•] Install Script $ pkg update && pkg upgrade $ pkg install python $ pkg install git $ pip install requ

O'Hayo Smrn 3 Apr 02, 2022
Anti Supercookie - Confusing the ISP & Escaping the Supercookie

Confusing the ISP & Escaping the Supercookie

Baris Dincer 2 Nov 22, 2022
Lite version of my Gatekeeper backdoor for public use.

Gatekeeper Lite Backdoor Fully functioning bind-type backdoor This backdoor is a fully functioning bind shell and lite version of my full functioning

Joe Helle 56 Mar 25, 2022
A Python r2pipe script to automatically create a Frida hook to intercept TLS traffic for Flutter based apps

boring-flutter A Python r2pipe script to automatically create a Frida hook to intercept TLS traffic for Flutter based apps. Currently only supporting

Hamza 64 Oct 18, 2022
🐝 ℹ️ Honeybee extension for export to IES-VE gem file format

honeybee-ies Honeybee extension for export a HBJSON file to IES-VE GEM file format Installation pip install honeybee-ies QuickStart import pathlib fro

Ladybug Tools 4 Jul 12, 2022
大宝剑-信息收集和资产梳理工具(红队、蓝队、企业组织架构、子域名、Web资产梳理、Web指纹识别、ICON_Hash资产匹配)

大宝剑-信息收集和资产梳理工具(红队、蓝队、企业组织架构、子域名、Web资产梳理、Web指纹识别、ICON_Hash资产匹配)

Wolf Group Security Team 835 Jan 05, 2023
pwncat module that automatically exploits CVE-2021-4034 (pwnkit)

pwncat_pwnkit Introduction The purpose of this module is to attempt to exploit CVE-2021-4034 (pwnkit) on a target when using pwncat. There is no need

Dana Epp 33 Jul 01, 2022
md5 hash cracking with python.

Python-Md5-Cracker- md5 hash cracking with python. Original files added First create a file called word.txt then run the wordCreate.py script The task

Nebil Sharifi 0 Aug 31, 2022
DoSer.py - Simple DoSer in Python

DoSer.py - Simple DoSer in Python What is DoSer? DoSer is basically an HTTP Denial of Service attack that affects threaded servers. It works like this

1 Oct 12, 2021
A tool to crack a wifi password with a help of wordlist

A tool to crack a wifi password with a help of wordlist. This may take long to crack a wifi depending upon number of passwords your wordlist contains. Also it is slower as compared to social media ac

Saad 144 Dec 29, 2022
阿里云accesskey利用工具

aliyun-accesskey-Tools 此工具用于查询ALIYUN_ACCESSKEY的主机,并且远程执行命令。 对于ALIYUN_ACCESSKEY利用方式可参考文章:记一次阿里云主机泄露Access Key到Getshell 工具截图 安装模块 pip install -r require

一灯老和尚 826 Jan 01, 2023
macOS Initial Access Payload Generator

Mystikal macOS Initial Access Payload Generator Related Blog Post: https://posts.specterops.io/introducing-mystikal-4fbd2f7ae520 Usage: Install Xcode

Leo Pitt 206 Dec 31, 2022
Script Crack Facebook Elite 🚶‍♂

elite Script Crack Facebook Elite 🚶‍♂ Install Script $ pkg update && pkg upgrade $ termux-setup-storage $ pkg install git $ pkg install python $ pip

Yumasaa 1 Jan 02, 2022
BETA: Layla - recon tool for bug bounty

WELCOME TO LAYLA Layla is a python script that automatically performs recon on a

Matheus Faria 68 Jan 04, 2023
Burp Extensions

Burp Extensions This is a collection of extensions to Burp Suite that I have written. getAllParams.py - Version 1.2 This is a python extension that ru

/XNL-h4ck3r 364 Dec 30, 2022