Adam with minor modifications which give significant improvement

Related tags

MiscellaneousBAdam
Overview

BAdam

Modification of Adam [1] optimizer with increased stability and better performance. Tricks used:

  1. Decoupled weight decay as in AdamW [2]. Such decoupling allows easier tuning of weight decay and learning rate. This implementation follows PyTorch and multiplies weight decay by learning rate, allowing simultaneous scheduling. Due to this typical values passed to optimizer should be much higher that with SGD, if you use 1e-4 with SGD, good start would be to use 1e-2 with BAdam.
  2. Epsilon is inside sqrt to avoid NaN in mixed precision. Default value is much larger than in Adam to reduce 'adaptivity' it leads to better and wider optimums [3]. Large epsilon also works better than amsgrad version of Adam [5]
  3. exp_avg_sq inits with large value, rather than with zeros. This removes the need for lr warmup and does the same thing as all the tricks from RAdam [4], while being much simpler.
  4. Removed bias correction. It's not needed if exp_avg_sq is correctly initialized

Practical Tips

Default values for this optimizer were tuned on Imagenet and work as good baseline for other computer vision tasks. Try them as is, before further tuning.

Installation

pip install git+https://github.com/bonlime/[email protected]

Reference:

[1] Adam: A Method for Stochastic Optimization
[2] Decoupled Weight Decay Regularization
[3] On the Convergence of Adam and Beyond
[4] On the Variance of the Adaptive Learning Rate and Beyond [5] Adaptive Methods for Non-convex Optimization

Owner
Emil Zakirov. MIPT & Skoltech. Computer Vision Engineer.
Official repository for the BPF Performance Tools book

BPF Performance Tools This is the official repository of BPF (eBPF) tools from the book BPF Performance Tools: Linux and Application Observability. Th

Brendan Gregg 1.2k Dec 28, 2022
Time tracking program that will format output to be easily put into Gitlab

time_tracker Time tracking program that will format output to be easily put into Gitlab. Feel free to branch and use it yourself! Getting Started Clon

Jake Strasler 2 Oct 13, 2022
This interactive script demonstrates the Menezes-Vanstone-EC-Cryptosystem

Menezes-Vanstone-EC-Cryptosystem This interactive script demonstrates the Meneze

Nishaant Goswamy 1 Jan 02, 2022
Xbox-Flood is for flood anything

Intruduction Installation Usage Installing Python 3 Wiki Getting Started Creating a Key Intruduction Xbox-Flood is for flooding messages (invitations

kayake 4 Feb 18, 2022
Show Public IP Information In Linux Taskbar

IP Information In Linux Taskbar 📍 How Use IP Script? 🤔 Download ip.py script and save somewhere in your system. Add command applet in your taskbar a

HOP 2 Jan 25, 2022
token vesting escrow with cliff and clawback

Yearn Vesting Escrow A modified version of Curve Vesting Escrow contracts with added functionality: An escrow can have a start_date in the past.

62 Dec 08, 2022
Check is a integer is even

Is Even Check if interger is even using isevenapi. https://isevenapi.xyz/ Main features: cache memoization api retry handler hide ads Install pip inst

Rosiney Gomes Pereira 45 Dec 19, 2022
This Python3 script will monitor Upwork RSS feed and then email you the results.

Upwork RSS Parser This Python3 script will monitor Upwork RSS feed and then email you the results. Table of Contents General Info Technologies Used Fe

Chris 5 Nov 29, 2021
Python module used to generate random facts

Randfacts is a python library that generates random facts. You can use randfacts.get_fact() to return a random fun fact. Disclaimer: Facts are not gua

Tabulate 14 Dec 14, 2022
All exercises done during the Python 3 course in the Video Course (World 1, 2 and 3)

Python3-cursoemvideo-exercises - All exercises done during the Python 3 course in the Video Course (World 1, 2 and 3)

Renan Barbosa 3 Jan 17, 2022
Jarvis Python BOT acts like Google-assistance

Jarvis-Python-BOT Jarvis Python BOT acts like Google-assistance Setup Add Mail ID (Gmail) in the file at line no 82.

Ishan Jogalekar 1 Jan 08, 2022
Let's pretend you want to create a AWS Lambda project called "sns-processor".

Usage Let's pretend you want to create a AWS Lambda project called "sns-processor". Rather than using lambda and then editing the results to include y

1 Dec 31, 2021
Werkzeug has a debug console that requires a pin. It's possible to bypass this with an LFI vulnerability or use it as a local privilege escalation vector.

Werkzeug Debug Console Pin Bypass Werkzeug has a debug console that requires a pin by default. It's possible to bypass this with an LFI vulnerability

Wyatt Dahlenburg 23 Dec 17, 2022
A python script that automatically joins a zoom meeting based on your timetable.

Zoom Automation A python script that automatically joins a zoom meeting based on your timetable. What does it do? It performs the following processes:

Shourya Gupta 3 Jan 01, 2022
NFT-Image-Generator - Utility to generate a large collection of unique images

NFT-Image-Generator Utility for creating a generative art collection from suppli

Sem Moolenschot 60 Dec 15, 2022
A patch and keygen tools for typora.

A patch and keygen tools for typora.

Mason Shi 1.4k Apr 12, 2022
About Python's multithreading and GIL

About Python's multithreading and GIL

Souvik Ghosh 3 Mar 01, 2022
An optional component handler for hikari, inspired by discord.py's views.

hikari-miru An optional component handler for hikari, inspired by discord.py's views.

43 Dec 26, 2022
A simple hash system.

PBH-Hash-System A simple hash system. Usage You could use it like this: from pbh import pbh print(pbh("Hey", True)) Output: 2feae2471698cfcdcbd6b98ca

Karim 3 Mar 24, 2022
Very Simple Zoom Spam Pinger!

Very Simple Zoom Spam Pinger!

Syntax. 2 Mar 05, 2022