VOneNet: CNNs with a Primary Visual Cortex Front-End

Related tags

Deep Learningvonenet
Overview

VOneNet: CNNs with a Primary Visual Cortex Front-End

A family of biologically-inspired Convolutional Neural Networks (CNNs). VOneNets have the following features:

  • Fixed-weight neural network model of the primate primary visual cortex (V1) as the front-end.
  • Robust to image perturbations
  • Brain-mapped
  • Flexible: can be adapted to different back-end architectures

read more...

Available Models

(Click on model names to download the weights of ImageNet-trained models. Alternatively, you can use the function get_model in the vonenet package to download the weights.)

Name Description
VOneResNet50 Our best performing VOneNet with a ResNet50 back-end
VOneCORnet-S VOneNet with a recurrent neural network back-end based on the CORnet-S
VOneAlexNet VOneNet with a back-end based on AlexNet

Quick Start

VOneNets was trained with images normalized with mean=[0.5,0.5,0.5] and std=[0.5,0.5,0.5]

More information coming soon...

Longer Motivation

Current state-of-the-art object recognition models are largely based on convolutional neural network (CNN) architectures, which are loosely inspired by the primate visual system. However, these CNNs can be fooled by imperceptibly small, explicitly crafted perturbations, and struggle to recognize objects in corrupted images that are easily recognized by humans. Recently, we observed that CNN models with a neural hidden layer that better matches primate primary visual cortex (V1) are also more robust to adversarial attacks. Inspired by this observation, we developed VOneNets, a new class of hybrid CNN vision models. Each VOneNet contains a fixed weight neural network front-end that simulates primate V1, called the VOneBlock, followed by a neural network back-end adapted from current CNN vision models. The VOneBlock is based on a classical neuroscientific model of V1: the linear-nonlinear-Poisson model, consisting of a biologically-constrained Gabor filter bank, simple and complex cell nonlinearities, and a V1 neuronal stochasticity generator. After training, VOneNets retain high ImageNet performance, but each is substantially more robust, outperforming the base CNNs and state-of-the-art methods by 18% and 3%, respectively, on a conglomerate benchmark of perturbations comprised of white box adversarial attacks and common image corruptions. Additionally, all components of the VOneBlock work in synergy to improve robustness. Read more: Dapello*, Marques*, et al. (biorxiv, 2020)

Requirements

  • Python 3.6+
  • PyTorch 0.4.1+
  • numpy
  • pandas
  • tqdm
  • scipy

Citation

Dapello, J., Marques, T., Schrimpf, M., Geiger, F., Cox, D.D., DiCarlo, J.J. (2020) Simulating a Primary Visual Cortex at the Front of CNNs Improves Robustness to Image Perturbations. biorxiv. doi.org/10.1101/2020.06.16.154542

License

GNU GPL 3+

FAQ

Soon...

Setup and Run

  1. You need to clone it in your local repository $ git clone https://github.com/dicarlolab/vonenet.git

  2. And when you setup its codes, you must need 'val' directory. so here is link. this link is from Korean's blog I refered as below https://seongkyun.github.io/others/2019/03/06/imagenet_dn/

    ** Download link**
    

https://academictorrents.com/collection/imagenet-2012

Once you download that large tar files, you must unzip that files -- all instructions below are refered above link, I only translate it

Unzip training dataset

$ mkdir train && mb ILSVRC2012_img_train.tar train/ && cd train $ tar -xvf ILSVRC2012_img_train.tar $ rm -f ILSVRC2012_img_train.tar (If you want to remove zipped file(tar)) $ find . -name "*.tar" | while read NAME ; do mkdir -p "${NAME%.tar}"; tar -xvf "${NAME}" -C "${NAME%.tar}"; rm -f "${NAME}"; done $ cd ..

Unzip validation dataset

$ mkdir val && mv ILSVRC2012_img_val.tar val/ && cd val && tar -xvf ILSVRC2012_img_val.tar $ wget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash

when it's finished, you can see train directory, val directory that 'val' directory is needed when setting up

Caution!!!!

after all execution above, must remove directory or file not having name n0000 -> there will be fault in training -> ex) 'ILSVRC2012_img_train' in train directory, 'ILSVRC2012_img_val.tar' in val directory

  1. if you've done getting data, then we can setting up go to local repository which into you cloned and open terminal (you must check your versions of python, pytorch, cudatoolkit if okay then,) $ python3 setup.py install $ python3 run.py --in_path {directory including above dataset, 'val' directory must be in!}

If you see any GPU related problem especially 'GPU is not available' although you already got

$ python3 run.py --in_path {directory including above dataset, 'val' directory must be in!} --ngpus 0

ngpus is 1 as default. if you don't care running on CPU you do so

Comments
  • GPU requirements

    GPU requirements

    Hi! Thank you so much for releasing the code!

    If I wanted to train the VOneResNet50 on a NVIDIA GeForce RTX 2070 how long should I expect it to take? I'm new to training neural networks this big and am working on a small project for a course, so it would be good to have an estimate.

    Thank you so much!

    Maria Inês

    opened by mariainescravo 4
  • k_exc parameter

    k_exc parameter

    Hi,

    Thanks for releasing your code! Quick question- what is the significance of the k_exc parameter used in the V1 block?

    https://github.com/dicarlolab/vonenet/blob/master/vonenet/modules.py#L91

    Norman

    opened by normster 4
  • Robust Accuracy results not matching

    Robust Accuracy results not matching

    Firstly, thank you for open sourcing the code for your paper. It has been really helpful !!

    I had a small query regarding the robust evaluation of models. I tried to evaluate the pretrained VoneResNet50 model with standard PGD with EOT and I get the following results:

    robust accuracy (top1):0.3666
    robust accuracy (top5):0.635
    

    My PGD parameters were as follows :

    iterations : 64
    norm : L inifity
    epsilon: 0.0009803921569 (= 1/1020)
    eot_iterations : 8
    Library: advertorch 
    

    I used the code in this PR and also checked with another library

    It seems like the top-5 accuracy is closer to the accuracy mentioned in the paper. I'm confused since the paper mentions that the accuracy is always top-1?

    opened by code-Assasin 3
  • Can you provide the trained VOneNet model file onto google drive?

    Can you provide the trained VOneNet model file onto google drive?

    Can you provide the trained VOneNet model file onto google drive so that I can download for my experiments. CIFAR-10, CIFAR-100, ImageNet datasets, do you have the trained model file??

    opened by machanic 2
  • Update README.md

    Update README.md

    There are problems in line 17, 18, 19 README.md. Because When I finished download, system tells me this is wrong extension.

    and add setup and run instructions. please check it and if there some error, please correct it

    opened by comeeasy 1
  • explaining neural variances

    explaining neural variances

    Thank you for the code for the V1Block. Interesting work!

    I was wondering how you exactly compared regular convolutional features and the ones from VOneNet to explain the Neural Variances.

    Since the paper stresses that this model is SoTA in explaining these, I would be really glad if you can include the code for that too / or if you could point me to existing repositories that do that (if you are aware of any), that'd be great too!

    Thanks again!

    opened by vinbhaskara 1
  • fix: added missing argument for restoring model training

    fix: added missing argument for restoring model training

    For restoring the model training, the code already provided the logic but forgot to add the argument to the parser. Now it is able to restore the model training providing the epoch number and the path containing those files.

    opened by ALLIESXO 0
  • How to test the top-scoring Brain Score model - vonenet-resnet50-non-stochastic?

    How to test the top-scoring Brain Score model - vonenet-resnet50-non-stochastic?

    Hi, I am trying to understand what's the correct way to test (using the pretrained model trained on ImageNet) the voneresnet-50-non_stochastic model that is currently scoring two on Brain Score.

    I want the model to be pretrained on ImageNet. When loading the model through net = vonenet.get_model(model_arch='resnet50', pretrained=True) a state_dict file that already contains the noise_level, noise_scale and noise_mode parameter gets loaded (in vonenet/__init__.py line 38. Do the pretrained model performance depends on these values to be fixed at 'neuronal', 0.35 and 0.07? Or can set one of these to 0 (which one?) and just keep using the same pretrained model for testing?

    Thanks, Valerio

    opened by ValerioB88 0
  • Alignment of quadrutre pairs (q0 and q1) in terms of input channels?

    Alignment of quadrutre pairs (q0 and q1) in terms of input channels?

    Hi Tiago and Joel, this is a very cool project.

    The initialize method of the GFB class doesn't set the random seed of randint:

        def initialize(self, sf, theta, sigx, sigy, phase):
            random_channel = torch.randint(0, self.in_channels, (self.out_channels,))
    

    Doesn't this cause the filters of simple_conv_q0 and simple_conv_q1 to be misaligned in terms of input channels?

    opened by Tal-Golan 1
  • add example of adversarial evaluation

    add example of adversarial evaluation

    check out my attack example and let me know what you think.

    I made it entirely self contained in adv_evaluate.py, and I added an example to the README.md

    opened by dapello 0
Owner
The DiCarlo Lab at MIT
Working to discover the neuronal algorithms underlying visual object recognition
The DiCarlo Lab at MIT
A machine learning project which can detect and predict the skin disease through image recognition.

ML-Project-2021 A machine learning project which can detect and predict the skin disease through image recognition. The dataset used for this is the H

Debshishu Ghosh 1 Jan 13, 2022
A lightweight library designed to accelerate the process of training PyTorch models by providing a minimal

A lightweight library designed to accelerate the process of training PyTorch models by providing a minimal, but extensible training loop which is flexible enough to handle the majority of use cases,

Chris Hughes 110 Dec 23, 2022
MADE (Masked Autoencoder Density Estimation) implementation in PyTorch

pytorch-made This code is an implementation of "Masked AutoEncoder for Density Estimation" by Germain et al., 2015. The core idea is that you can turn

Andrej 498 Dec 30, 2022
PyTorch implementations of the beta divergence loss.

Beta Divergence Loss - PyTorch Implementation This repository contains code for a PyTorch implementation of the beta divergence loss. Dependencies Thi

Billy Carson 7 Nov 09, 2022
Face recognition project by matching the features extracted using SIFT.

MV_FaceDetectionWithSIFT Face recognition project by matching the features extracted using SIFT. By : Aria Radmehr Professor : Ali Amiri Dependencies

Aria Radmehr 4 May 31, 2022
DualGAN-tensorflow: tensorflow implementation of DualGAN

ICCV paper of DualGAN DualGAN: unsupervised dual learning for image-to-image translation please cite the paper, if the codes has been used for your re

Jack Yi 252 Nov 10, 2022
Relative Uncertainty Learning for Facial Expression Recognition

Relative Uncertainty Learning for Facial Expression Recognition The official implementation of the following paper at NeurIPS2021: Title: Relative Unc

35 Dec 28, 2022
Strongly local p-norm-cut algorithms for semi-supervised learning and local graph clustering

Strongly local p-norm-cut algorithms for semi-supervised learning and local graph clustering

Meng Liu 2 Jul 19, 2022
YOLO-v5 기반 단안 카메라의 영상을 활용해 차간 거리를 일정하게 유지하며 주행하는 Adaptive Cruise Control 기능 구현

자율 주행차의 영상 기반 차간거리 유지 개발 Table of Contents 프로젝트 소개 주요 기능 시스템 구조 디렉토리 구조 결과 실행 방법 참조 팀원 프로젝트 소개 YOLO-v5 기반으로 단안 카메라의 영상을 활용해 차간 거리를 일정하게 유지하며 주행하는 Adap

14 Jun 29, 2022
Code Repo for the ACL21 paper "Common Sense Beyond English: Evaluating and Improving Multilingual LMs for Commonsense Reasoning"

Common Sense Beyond English: Evaluating and Improving Multilingual LMs for Commonsense Reasoning This is the Github repository of our paper, "Common S

INK Lab @ USC 19 Nov 30, 2022
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning

AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (NeurIPS 2020) Introduction AdaShare is a novel and differentiable approach fo

94 Dec 22, 2022
Official Implementation and Dataset of "PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask and Group-Level Consistency", CVPR 2021

Portrait Photo Retouching with PPR10K Paper | Supplementary Material PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask an

184 Dec 11, 2022
deep learning model with only python and numpy with test accuracy 99 % on mnist dataset and different optimization choices

deep_nn_model_with_only_python_100%_test_accuracy deep learning model with only python and numpy with test accuracy 99 % on mnist dataset and differen

0 Aug 28, 2022
Collision risk estimation using stochastic motion models

collision_risk_estimation Collision risk estimation using stochastic motion models. This is a new approach, based on stochastic models, to predict the

Unmesh 7 Jun 26, 2022
A spherical CNN for weather forecasting

DeepSphere-Weather - Deep Learning on the sphere for weather/climate applications. The code in this repository provides a scalable and flexible framew

DeepSphere 47 Dec 25, 2022
Official implementation of NPMs: Neural Parametric Models for 3D Deformable Shapes - ICCV 2021

NPMs: Neural Parametric Models Project Page | Paper | ArXiv | Video NPMs: Neural Parametric Models for 3D Deformable Shapes Pablo Palafox, Aljaz Bozic

PabloPalafox 109 Nov 22, 2022
Rayvens makes it possible for data scientists to access hundreds of data services within Ray with little effort.

Rayvens augments Ray with events. With Rayvens, Ray applications can subscribe to event streams, process and produce events. Rayvens leverages Apache

CodeFlare 32 Dec 25, 2022
TAUFE: Task-Agnostic Undesirable Feature DeactivationUsing Out-of-Distribution Data

A deep neural network (DNN) has achieved great success in many machine learning tasks by virtue of its high expressive power. However, its prediction can be easily biased to undesirable features, whi

KAIST Data Mining Lab 8 Dec 07, 2022
The implementation code for "DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction"

DAGAN This is the official implementation code for DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruct

TensorLayer Community 159 Nov 22, 2022
[arXiv22] Disentangled Representation Learning for Text-Video Retrieval

Disentangled Representation Learning for Text-Video Retrieval This is a PyTorch implementation of the paper Disentangled Representation Learning for T

Qiang Wang 49 Dec 18, 2022