Reverse engineer your pytorch vision models, in style

Related tags

Deep Learningrover
Overview

🔍 Rover

Reverse engineer your CNNs, in style

Open In Colab

Rover will help you break down your CNN and visualize the features from within the model. No need to write weirdly abstract code to visualize your model's features anymore.

💻 Usage

git clone https://github.com/Mayukhdeb/rover.git; cd rover

install requirements:

pip install -r requirements.txt
from rover import core
from rover.default_models import models_dict

core.run(models_dict = models_dict)

and then run the script with streamlit as:

$ streamlit run your_script.py

if everything goes right, you'll see something like:

You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501

🧙 Custom models

rover supports pretty much any PyTorch model with an input of shape [N, 3, H, W] (even segmentation models/VAEs and all that fancy stuff) with imagenet normalization on input.

import torchvision.models as models 
model = models.resnet34(pretrained= True)  ## or any other model (need not be from torchvision.models)

models_dict = {
    'my model': model,  ## add in any number of models :)
}

core.run(
    models_dict = models_dict
)

🖼️ Channel objective

Optimizes a single channel from one of the layer(s) selected.

  • layer index: specifies which layer you want to use out of the layers selected.
  • channel index: specifies the exact channel which needs to be visualized.

🧙‍♂️ Writing your own objective

This is for the smarties who like to write their own objective function. The only constraint is that the function should be named custom_func.

Here's an example:

def custom_func(layer_outputs):
    '''
    layer_outputs is a list containing 
    the outputs (torch.tensor) of each layer you selected

    In this example we'll try to optimize the following:
    * the entire first layer -> layer_outputs[0].mean()
    * 20th channel of the 2nd layer -> layer_outputs[1][20].mean()
    '''
    loss = layer_outputs[0].mean() + layer_outputs[1][20].mean()
    return -loss

Running on google colab

Check out this notebook. I'll also include the instructions here just in case.

Clone the repo + install dependencies

!git clone https://github.com/Mayukhdeb/rover.git
!pip install torch-dreams --quiet
!pip install streamlit --quiet

Navigate into the repo

import os 
os.chdir('rover')

Write your file into a script from a cell. Here I wrote it into test.py

%%writefile  test.py

from rover import core
from rover.default_models import models_dict

core.run(models_dict = models_dict)

Run script on a thread

import threading

proc = threading.Thread(target= os.system, args=['streamlit run test.py'])
proc.start()

Download ngrok:

!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
!unzip -o ngrok-stable-linux-amd64.zi

More ngrok stuff

get_ipython().system_raw('./ngrok http 8501 &')

Get your URL where rover is hosted

!curl -s http://localhost:4040/api/tunnels | python3 -c \
    "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"

💻 Args

  • width (int, optional): Width of image to be optimized
  • height (int, optional): Height of image to be optimized
  • iters (int, optional): Number of iterations, higher -> stronger visualization
  • lr (float, optional): Learning rate
  • rotate (deg) (int, optional): Max rotation in default transforms
  • scale max (float, optional): Max image size factor.
  • scale min (float, optional): Minimum image size factor.
  • translate (x) (float, optional): Maximum translation factor in x direction
  • translate (y) (float, optional): Maximum translation factor in y direction
  • weight decay (float, optional): Weight decay for default optimizer. Helps prevent high frequency noise.
  • gradient clip (float, optional): Maximum value of the norm of gradient.

Run locally

Clone the repo

git clone https://github.com/Mayukhdeb/rover.git

install requirements

pip install -r requirements.txt

showtime

streamlit run test.py
Owner
Mayukh Deb
Learning about life, one epoch at a time
Mayukh Deb
VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition

VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition Usage First, install PyTorch 1.7.1+, torchvision 0.8.2

40 Dec 12, 2022
A complete end-to-end demonstration in which we collect training data in Unity and use that data to train a deep neural network to predict the pose of a cube. This model is then deployed in a simulated robotic pick-and-place task.

Object Pose Estimation Demo This tutorial will go through the steps necessary to perform pose estimation with a UR3 robotic arm in Unity. You’ll gain

Unity Technologies 187 Dec 24, 2022
PyTorch implementation of Munchausen Reinforcement Learning based on DQN and SAC. Handles discrete and continuous action spaces

Exploring Munchausen Reinforcement Learning This is the project repository of my team in the "Advanced Deep Learning for Robotics" course at TUM. Our

Mohamed Amine Ketata 10 Mar 10, 2022
Code for "ShineOn: Illuminating Design Choices for Practical Video-based Virtual Clothing Try-on", accepted at WACV 2021 Generation of Human Behavior Workshop.

ShineOn: Illuminating Design Choices for Practical Video-based Virtual Clothing Try-on [ Paper ] [ Project Page ] This repository contains the code fo

Andrew Jong 97 Dec 13, 2022
Neural Architecture Search Powered by Swarm Intelligence 🐜

Neural Architecture Search Powered by Swarm Intelligence 🐜 DeepSwarm DeepSwarm is an open-source library which uses Ant Colony Optimization to tackle

288 Oct 28, 2022
Package for working with hypernetworks in PyTorch.

Package for working with hypernetworks in PyTorch.

Christian Henning 71 Jan 05, 2023
masscan + nmap + Finger

说明 个人根据使用习惯修改masnmap而来的一个小工具。调用masscan做全端口扫描,再调用nmap做服务识别,最后调用Finger做Web指纹识别。工具使用场景适合风险探测排查、众测等。 使用方法 安装依赖 pip3 install -r requirements.txt -i https:/

Ryan 3 Mar 25, 2022
On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification

Understanding Bayesian Classification This repository hosts the code to reproduce the results presented in the paper On Uncertainty, Tempering, and Da

Sanyam Kapoor 18 Nov 17, 2022
An unreferenced image captioning metric (ACL-21)

UMIC This repository provides an unferenced image captioning metric from our ACL 2021 paper UMIC: An Unreferenced Metric for Image Captioning via Cont

hwanheelee 14 Nov 20, 2022
The official implementation of the IEEE S&P`22 paper "SoK: How Robust is Deep Neural Network Image Classification Watermarking".

Watermark-Robustness-Toolbox - Official PyTorch Implementation This repository contains the official PyTorch implementation of the following paper to

49 Dec 19, 2022
Code and data of the EMNLP 2021 paper "Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer"

StyleAttack Code and data of the EMNLP 2021 paper "Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer" Prepare Pois

THUNLP 19 Nov 20, 2022
PyTorch code of my ICDAR 2021 paper Vision Transformer for Fast and Efficient Scene Text Recognition (ViTSTR)

Vision Transformer for Fast and Efficient Scene Text Recognition (ICDAR 2021) ViTSTR is a simple single-stage model that uses a pre-trained Vision Tra

Rowel Atienza 198 Dec 27, 2022
The Fundamental Clustering Problems Suite (FCPS) summaries 54 state-of-the-art clustering algorithms, common cluster challenges and estimations of the number of clusters as well as the testing for cluster tendency.

FCPS Fundamental Clustering Problems Suite The package provides over sixty state-of-the-art clustering algorithms for unsupervised machine learning pu

9 Nov 27, 2022
ShuttleNet: Position-aware Fusion of Rally Progress and Player Styles for Stroke Forecasting in Badminton (AAAI 2022)

ShuttleNet: Position-aware Rally Progress and Player Styles Fusion for Stroke Forecasting in Badminton (AAAI 2022) Official code of the paper ShuttleN

Wei-Yao Wang 11 Nov 30, 2022
BaseCls BaseCls 是一个基于 MegEngine 的预训练模型库,帮助大家挑选或训练出更适合自己科研或者业务的模型结构

BaseCls BaseCls 是一个基于 MegEngine 的预训练模型库,帮助大家挑选或训练出更适合自己科研或者业务的模型结构。 文档地址:https://basecls.readthedocs.io 安装 安装环境 BaseCls 需要 Python = 3.6。 BaseCls 依赖 M

MEGVII Research 28 Dec 23, 2022
VM3000 Microphones

VM3000-Microphones This project was completed by Ricky Leman under the supervision of Dr Ben Travaglione and Professor Melinda Hodkiewicz as part of t

UWA System Health Lab 0 Jun 04, 2021
QAHOI: Query-Based Anchors for Human-Object Interaction Detection (paper)

QAHOI QAHOI: Query-Based Anchors for Human-Object Interaction Detection (paper) Requirements PyTorch = 1.5.1 torchvision = 0.6.1 pip install -r requ

38 Dec 29, 2022
A Broader Picture of Random-walk Based Graph Embedding

Random-walk Embedding Framework This repository is a reference implementation of the random-walk embedding framework as described in the paper: A Broa

Zexi Huang 23 Dec 13, 2022
StyleGAN2 Webtoon / Anime Style Toonify

StyleGAN2 Webtoon / Anime Style Toonify Korea Webtoon or Japanese Anime Character Stylegan2 base high Quality 1024x1024 / 512x512 Generate and Transfe

121 Dec 21, 2022
Neural Cellular Automata + CLIP

🧠 Text-2-Cellular Automata Using Neural Cellular Automata + OpenAI CLIP (Work in progress) Examples Text Prompt: Cthulu is watching cthulu_is_watchin

Mainak Deb 21 Dec 19, 2022