Pytorch implementation of CVPR2020 paper “VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation”

Overview

VectorNet Re-implementation

This is the unofficial pytorch implementation of CVPR2020 paper "VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation". (And it's a part of test of the summer camp 2020 organized by IIIS, Tsinghua University.)

  1. 运行环境

    python 3.7, Pytorch1.1.0, torchvision0.3.0, cuda9.0

  2. 文件说明

    ----- VectorNet

    +--- ArgoverseDataset.py 数据集读取、预处理、转换为tensor

    +--- subgraph_net.py polyline subgraph相关类实现

    +--- gnn.py 带Attention机制的GCN,因为图是全连接,所以没有用dgl

    +--- vectornet.py 把subgraph和GNN合并起来的model,loss计算

    +--- train.py 网络训练入口,会保存checkpoint

    +--- test.py 网络测试入口,同时实现了评估函数,会保存inference结果

    +--- Visualization.ipynb 可视化vectorize的HD map

  3. 运行准备

    • 安装argoverse-api且按照说明,将HD map数据放置到指定位置
    • 下载forecast数据集,将train.py和test.py中cfg['data_locate']修改为解压位置
  4. 代码函数解读

    • ArgoverseDataset.py

      定义了类class ArgoverseForecastDataset(torch.utils.data.Dataset)

      • def __init__(self, cfg) 类初始化,主要步骤有

        self.axis_range = self.get_map_range(self.am) #用于normalize坐标
        self.city_halluc_bbox_table, self.city_halluc_tableidx_to_laneid_map = self.am.build_hallucinated_lane_bbox_index()
        self.vector_map, self.extra_map = self.generate_vector_map()

        调用argoverse api读取HD map数据,重点是generate_vector_map函数

      • def generate_vector_map(self) 读取HD map并转换成vector

        利用argoverse api的get_lane_segment_polygon(key, city_name) 获取道路边沿的采样点,以论文指定的vector的方式拼接,该api是得到polygon,而我们只要两个边沿,因此做了一些处理

        同时将相关semantic label获取,返回至extra_map,待后续组装进vector内

      • def __getitem__(self, index) 迭代获取数据函数,在该函数中读取了trajectory数据,同时对坐标进行了一系列预处理,最后转换为tensor

        获取trajectory同样利用argoverse api,数据预处理主要分为3个步骤

        (1)平移坐标使last_observe移到中心

        (2)rotate利用齐次坐标旋转矩阵实现,夹角利用向量内积获得

        (3)normalize这里通过线性变换把坐标normalize到一定范围,这里认为last_observe的位置就是数据集分布的中心,即 $$ x = \frac{x}{max-min} $$

      • __getitem__返回

             self.traj_feature, self.map_feature

        其中self.traj_feature 是$N\times feature$ 维的tensor指示轨迹polyline的vector集合 self.map_feature 是一个有三个key的dict, map_feature['PIT']和map_feature['MIA'] 是list,分别是两座城市道路的polyline的list,即list的每一个元素是一个$N\times feature$ 维的tensor,指示一条道路的polyline,map_feature['city_name']保存该trajectory所在的城市 def get_trajectory(self, index)generate_vector_map 类似,区别在于trajectory是针对timestamp进行轨迹拼接,同时需要将timestamp装入向量中作为semantic label的信息

    • subgraph_net.py

      定义了类class SubgraphNet(nn.Module) class SubgraphNet_Layer(nn.Module)

      • class SubgraphNet_Layer

        输入:$N\times feature$ 维的单polyline tensor

        输出:$N\times (feature+global\ feature)$ 维的单polyline tensor

        实现了单层的SubgraphNet,按照文章叙述,encoder是一个MLP,具体由一个全连接层、一个layer_norm 和一个RELU激发层组成,随后是max_pool提取全局信息,最后concatenate将信息整合,与Point R-CNN相似

      • class SubgraphNet

        输入:$N\times feature$ 维的单polyline tensor

        输出:$1\times (feature+global\ feature)$ 维的单polyline tensor

        3 层SubgraphNet_Layer组合,最后max_pool提取代表性信息

    • gnn.py

      定义了类class GraphAttentionNet(nn.Module)

      • class GraphAttentionNet

        输入:$K\times (feature+global\ feature)$ 维的全图特征信息

        输出:$K\times value\ dims$ 维的传播后全图特征信息

        因为在本论文中,将邻接矩阵定义为全连接矩阵,因此没有建图实现消息传播的必要性。Attention机制在本类中加以实现,公式即为 $$ GNN(P)=softmax(P_QP_K^T)P_V $$ 注意:这里进行的都是矩阵计算。$P_Q$是查询,$P_K$是key,$P_V$是值,softmax一步是获得各value的权重

        具体的实现参考了论文Attention is All you need

    • vectornet.py

      定义了类class VectorNet(nn.Module)

      • class VectorNet 本类的 forwardtrainevaluate 两种情况

        输入:trajectory_batch, mapfeature_batch

        输出:train时输出loss,evaluate时输出预测结果predictions和真值label

        • 由于不同道路的polyline采样点数不同,因此在dataset数据读取时把它放入了list中,因此在本类中会首先完成对数据的拆包
        • 然后构造两个SubgraphNet类,traj_subgraphnet,和map_subgraphnet将不同polyline的信息,都处理为$1\times (feature+global\ feature)$ 维的polyline信息,然后concatenate起来
        • 此后会进行L2 normalize以有效训练后面的GNN,正则化后直接传入GNN,并得到传播后的vector信息 $1\times value\ dims$ 维,decoder使用了MLP与subgraph_net参数相似,但多加了一层全连接网络以生成回归坐标
        • 如果是train则使用torch.nn.MSEloss计算损失,可以证明在误差服从标准高斯分布时,Gaussian Negative Likelihood Loss就是MSEloss,它们本质上是等价的。如果是evaluate则把prediction和label一起输出,在test.py中实现Average Displacement Error的计算
    • train.py

      网络训练入口

      • def main()

        首先初始化一些参数,为代码简便,这里把配置(cfg)直接编码在代码中,更合适的做法应是利用 argparse 通过命令行传入。然后实例化dataset,利用dataloader打包为minibatch,初始化model,设置优化器,和步长自调节器

        另外这里使用tensorboard可视化损失,文件保存在 ./run/文件夹下,因此需要初始化SummaryWriter

      • def do_train(model, cfg, train_loader, optimizer, scheduler, writer)

        较为常见的主训练循环,每 5 个epoch调节一次步长,每10个epoch保存一次模型参数,训练结束保存一次模型参数,输出每2个iteration(minibatch)输出一次信息,采用logger保存日志文件

    • test.py

      网络推断入口

      • def main()

        与train.py几乎相同,注意cfg['model_path']模型参数文件路径和cfg['save_path']推理结果存储路径两个参数

      • def inference(model, cfg, val_loader)

        较do_train有所简化,因为无需再处理vector_map数据,已经被编码进网络里(只使用了一层的GNN),将输出的result和label用list保存起来,调用evaluate()函数计算ADE指标

      • def evaluate(dataset, predictions, labels)

        传入dataset是因为需要把预处理过的数据,变换回原始坐标,即先反归一化,然后逆向旋转,最后平移,ADE loss即是预测点和真值点间欧氏距离的平均,inference的结果保存在路径cfg['save_path']下

  5. 一些可视化的结果(详见visualization.ipynb)

    • loss 收敛(150组数据,训练了25个epoch,adadelta优化器,有点过拟合) img1
      img2
    • baseline的结果(150组数据,训练了10个epoch,9步预测) img3
    • 地图矢量化
      img1
      img4
    • 轨迹预测(蓝色的是label,红色是预测,十字路口场景呈现回归现象)
      img2
Demonstrates iterative FGSM on Apple's NeuralHash model.

apple-neuralhash-attack Demonstrates iterative FGSM on Apple's NeuralHash model. TL;DR: It is possible to apply noise to CSAM images and make them loo

Lim Swee Kiat 11 Jun 23, 2022
Machine Learning Models were applied to predict the mass of the brain based on gender, age ranges, and head size.

Brain Weight in Humans Variations of head sizes and brain weights in humans Kaggle dataset obtained from this link by Anubhab Swain. Image obtained fr

Anne Livia 1 Feb 02, 2022
A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection

Confluence: A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection 1. 介绍 用以替代 NMS,在所有 bbox 中挑选出最优的集合。 NMS 仅考虑了 bbox 的得分,然后根据 IOU 来

44 Sep 15, 2022
Anomaly detection analysis and labeling tool, specifically for multiple time series (one time series per category)

taganomaly Anomaly detection labeling tool, specifically for multiple time series (one time series per category). Taganomaly is a tool for creating la

Microsoft 272 Dec 17, 2022
A toolkit for making real world machine learning and data analysis applications in C++

dlib C++ library Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real worl

Davis E. King 11.6k Jan 01, 2023
A Python package for time series augmentation

tsaug tsaug is a Python package for time series augmentation. It offers a set of augmentation methods for time series, as well as a simple API to conn

Arundo Analytics 278 Jan 01, 2023
RoMA: Robust Model Adaptation for Offline Model-based Optimization

RoMA: Robust Model Adaptation for Offline Model-based Optimization Implementation of RoMA: Robust Model Adaptation for Offline Model-based Optimizatio

9 Oct 31, 2022
A Vision Transformer approach that uses concatenated query and reference images to learn the relationship between query and reference images directly.

A Vision Transformer approach that uses concatenated query and reference images to learn the relationship between query and reference images directly.

24 Dec 13, 2022
MassiveSumm: a very large-scale, very multilingual, news summarisation dataset

MassiveSumm: a very large-scale, very multilingual, news summarisation dataset This repository contains links to data and code to fetch and reproduce

Daniel Varab 19 Dec 16, 2022
DCGAN LSGAN WGAN-GP DRAGAN PyTorch

Recommendation Our GAN based work for facial attribute editing - AttGAN. News 8 April 2019: We re-implement these GANs by Tensorflow 2! The old versio

Zhenliang He 408 Nov 30, 2022
Tackling Obstacle Tower Challenge using PPO & A2C combined with ICM.

Obstacle Tower Challenge using Deep Reinforcement Learning Unity Obstacle Tower is a challenging realistic 3D, third person perspective and procedural

Zhuoyu Feng 5 Feb 10, 2022
📖 Deep Attentional Guided Image Filtering

📖 Deep Attentional Guided Image Filtering [Paper] Zhiwei Zhong, Xianming Liu, Junjun Jiang, Debin Zhao ,Xiangyang Ji Harbin Institute of Technology,

9 Dec 23, 2022
TEA: A Sequential Recommendation Framework via Temporally Evolving Aggregations

TEA: A Sequential Recommendation Framework via Temporally Evolving Aggregations Requirements python 3.6 torch 1.9 numpy 1.19 Quick Start The experimen

DMIRLAB 4 Oct 16, 2022
Cortex-compatible model server for Python and TensorFlow

Nucleus model server Nucleus is a model server for TensorFlow and generic Python models. It is compatible with Cortex clusters, Kubernetes clusters, a

Cortex Labs 14 Nov 27, 2022
LaneAF: Robust Multi-Lane Detection with Affinity Fields

LaneAF: Robust Multi-Lane Detection with Affinity Fields This repository contains Pytorch code for training and testing LaneAF lane detection models i

155 Dec 17, 2022
Convnet transfer - Code for paper How transferable are features in deep neural networks?

How transferable are features in deep neural networks? This repository contains source code necessary to reproduce the results presented in the follow

Jason Yosinski 143 Sep 13, 2022
A modular, open and non-proprietary toolkit for core robotic functionalities by harnessing deep learning

A modular, open and non-proprietary toolkit for core robotic functionalities by harnessing deep learning Website • About • Installation • Using OpenDR

OpenDR 304 Dec 28, 2022
Implementation of the Triangle Multiplicative module, used in Alphafold2 as an efficient way to mix rows or columns of a 2d feature map, as a standalone package for Pytorch

Triangle Multiplicative Module - Pytorch Implementation of the Triangle Multiplicative module, used in Alphafold2 as an efficient way to mix rows or c

Phil Wang 22 Oct 28, 2022
Pytorch Geometric Tutorials

Pytorch Geometric Tutorials

Antonio Longa 648 Jan 08, 2023
Pretrained Pytorch face detection (MTCNN) and recognition (InceptionResnet) models

Face Recognition Using Pytorch Python 3.7 3.6 3.5 Status This is a repository for Inception Resnet (V1) models in pytorch, pretrained on VGGFace2 and

Tim Esler 3.3k Jan 04, 2023