Distributed Grid Descent: an algorithm for hyperparameter tuning guided by Bayesian inference, designed to run on multiple processes and potentially many machines with no central point of control

Overview

Distributed Grid Descent

An implementation of Distributed Grid Descent: an algorithm for hyperparameter tuning guided by Bayesian inference, designed to run on multiple processes and potentially many machines with no central point of control as described in Appendix B of Working Memory Graphs [Loynd et al., 2019].

Note: This project is a work in progress. Please contact me if you like to contribute and help to develop a fully fledged python library out of it.

Usage

import numpy as np
from dgd import DistributedGridDescent

model = ... # model wrapper
data = {
    "train_data": ...
}

param_grid = {
    "learning_rate":[3e-3, 1e-3, 3e-4, 1e-4, 3e-5, 1e-5],
    "optimizer":["adam", "rmsprop"],
    "lr_annealing":[False, 0.95, 0.99],
    "batch_size":[32, 64, 128, 256, 1024],
    "num_linear_layers":[1, 2, 4, 8, 16],
    "num_neurons":[512, 256, 128, 64, 32, 16],
    "dropout":[0.0, 0.1, 0.3, 0.5],
    "l2":[0.0, 0.01, 0.1]
}

dgd = DistributedGridDescent(model, param_grid, metric=np.mean, n_jobs=-1)
dgd.run(data)

print(dgd.best_params_)
df = pd.DataFrame(dgd.results_).set_index("ID").sort_values(by=["metric"],ascending=False)

Examples and Tutorials

See sklearn_example.py, pytorch_example.py, rosenbrock_example.py and tensorflow_example.py in the examples folder for examples of basic usage of dgd.
See rosenbrock_server_example.py for an example of distributed usage.

Strong and weak scaling analysis

scaling_analysis

Algorithm

Input: Set of hyperparameters H, each having a discrete, ordered set of possible values.  
Input: Maximum number of training steps N per run.  
repeat  
    Download any new results.  
    if no results so far then
        Choose a random configuration C from the grid defined by H.
    else
        Identify the run set S with the highest metric.
        Initialize neighborhood B to contain only S.
        Expand B by adding all possible sets whose configurations differ from that of S by one step in exactly one hyperparameter setting.
        Calculate a ceiling M = Count(B) + 1.
        Weight each run set x in B M - Count(x).
        Sample a random run set S' from B according to run set weights.
        Choose configuration C from S'.
    end if
    Perform one training run of N steps using C.
    Calculate the runs Metric.
    Log the result on shared storage.
until terminated by user.

See Appendix B of Loynd et al., 2019 for details.

Owner
Martin
Machine Learning Engineer at heart MSc Student in Computational Science & Engineering :computer: :books: :wrench: @ ETH Zürich :switzerland:
Martin
N Queen Problem using Genetic Algorithm

The N Queen is the problem of placing N chess queens on an N×N chessboard so that no two queens attack each other.

Mahdi Hassanzadeh 2 Nov 11, 2022
Minimal examples of data structures and algorithms in Python

Pythonic Data Structures and Algorithms Minimal and clean example implementations of data structures and algorithms in Python 3. Contributing Thanks f

Keon 22k Jan 09, 2023
Rover. Finding the shortest pass by Dijkstra’s shortest path algorithm

rover Rover. Finding the shortest path by Dijkstra’s shortest path algorithm Задача Вы — инженер, проектирующий роверы-беспилотники. Вам надо спроекти

1 Nov 11, 2021
Our implementation of Gillespie's Stochastic Simulation Algorithm (SSA)

SSA Our implementation of Gillespie's Stochastic Simulation Algorithm (SSA) Requirements python =3.7 numpy pandas matplotlib pyyaml Command line usag

Anoop Lab 1 Jan 27, 2022
This is an implementation of the QuickHull algorithm in Python. I

QuickHull This is an implementation of the QuickHull algorithm in Python. It randomly generates a set of points and finds the convex hull of this set

Anant Joshi 4 Dec 04, 2022
Path tracing obj - (taichi course final project) a path tracing renderer that can import and render obj files

Path tracing obj - (taichi course final project) a path tracing renderer that can import and render obj files

5 Sep 10, 2022
iAWE is a wonderful dataset for those of us who work on Non-Intrusive Load Monitoring (NILM) algorithms.

iAWE is a wonderful dataset for those of us who work on Non-Intrusive Load Monitoring (NILM) algorithms. You can find its main page and description via this link. If you are familiar with NILM-TK API

Mozaffar Etezadifar 3 Mar 19, 2022
Algorithms implemented in Python

Python Algorithms Library Laurent Luce Description The purpose of this library is to help you with common algorithms like: A* path finding. String Mat

Laurent Luce 264 Dec 06, 2022
A GUI visualization of QuickSort algorithm

QQuickSort A simple GUI visualization of QuickSort algorithm. It only uses PySide6, it does not have any other external dependency. How to run Install

Jaime R. 2 Dec 24, 2021
Algorithms-in-Python - Programs related to DSA in Python for placement practice

Algorithms-in-Python Programs related to DSA in Python for placement practice CO

MAINAK CHAUDHURI 2 Feb 02, 2022
Using A * search algorithm and GBFS search algorithm to solve the Romanian problem

Romanian-problem-using-Astar-and-GBFS Using A * search algorithm and GBFS search algorithm to solve the Romanian problem Romanian problem: The agent i

Mahdi Hassanzadeh 6 Nov 22, 2022
A Python program to easily solve the n-queens problem using min-conflicts algorithm

QueensProblem A program to easily solve the n-queens problem using min-conflicts algorithm Performances estimated with a sample of 1000 different rand

0 Oct 21, 2022
Pathfinding algorithm based on A*

Pathfinding V1 What is pathfindingV1 ? This program is my very first path finding program, using python and turtle for graphic rendering. How is it wo

Yan'D 6 May 26, 2022
All algorithms implemented in Python for education

The Algorithms - Python All algorithms implemented in Python - for education Implementations are for learning purposes only. As they may be less effic

1 Oct 20, 2021
A library for benchmarking, developing and deploying deep learning anomaly detection algorithms

A library for benchmarking, developing and deploying deep learning anomaly detection algorithms Key Features • Getting Started • Docs • License Introd

OpenVINO Toolkit 1.5k Jan 04, 2023
Implementation of core NuPIC algorithms in C++

NuPIC Core This repository contains the C++ source code for the Numenta Platform for Intelligent Computing (NuPIC)

Numenta 270 Nov 19, 2022
Algorithmic Trading with Python

Source code for Algorithmic Trading with Python (2020) by Chris Conlan

Chris Conlan 1.3k Jan 03, 2023
This project consists of a collaborative filtering algorithm to predict movie reviews ratings from a dataset of Netflix ratings.

Collaborative Filtering - Netflix movie reviews Description This project consists of a collaborative filtering algorithm to predict movie reviews rati

Shashank Kumar 1 Dec 21, 2021
A raw implementation of the nearest insertion algorithm to resolve TSP problems in a TXT format.

TSP-Nearest-Insertion A raw implementation of the nearest insertion algorithm to resolve TSP problems in a TXT format. Instructions Load a txt file wi

sjas_Phantom 1 Dec 02, 2021