Distributed Grid Descent: an algorithm for hyperparameter tuning guided by Bayesian inference, designed to run on multiple processes and potentially many machines with no central point of control

Related tags

AlgorithmsDistributedGridDescent
Overview

Distributed Grid Descent

An implementation of Distributed Grid Descent: an algorithm for hyperparameter tuning guided by Bayesian inference, designed to run on multiple processes and potentially many machines with no central point of control as described in Appendix B of Working Memory Graphs [Loynd et al., 2019].

Note: This project is a work in progress. Please contact me if you like to contribute and help to develop a fully fledged python library out of it.

Usage

import numpy as np
from dgd import DistributedGridDescent

model = ... # model wrapper
data = {
    "train_data": ...
}

param_grid = {
    "learning_rate":[3e-3, 1e-3, 3e-4, 1e-4, 3e-5, 1e-5],
    "optimizer":["adam", "rmsprop"],
    "lr_annealing":[False, 0.95, 0.99],
    "batch_size":[32, 64, 128, 256, 1024],
    "num_linear_layers":[1, 2, 4, 8, 16],
    "num_neurons":[512, 256, 128, 64, 32, 16],
    "dropout":[0.0, 0.1, 0.3, 0.5],
    "l2":[0.0, 0.01, 0.1]
}

dgd = DistributedGridDescent(model, param_grid, metric=np.mean, n_jobs=-1)
dgd.run(data)

print(dgd.best_params_)
df = pd.DataFrame(dgd.results_).set_index("ID").sort_values(by=["metric"],ascending=False)

Examples and Tutorials

See sklearn_example.py, pytorch_example.py, rosenbrock_example.py and tensorflow_example.py in the examples folder for examples of basic usage of dgd.
See rosenbrock_server_example.py for an example of distributed usage.

Strong and weak scaling analysis

scaling_analysis

Algorithm

Input: Set of hyperparameters H, each having a discrete, ordered set of possible values.  
Input: Maximum number of training steps N per run.  
repeat  
    Download any new results.  
    if no results so far then
        Choose a random configuration C from the grid defined by H.
    else
        Identify the run set S with the highest metric.
        Initialize neighborhood B to contain only S.
        Expand B by adding all possible sets whose configurations differ from that of S by one step in exactly one hyperparameter setting.
        Calculate a ceiling M = Count(B) + 1.
        Weight each run set x in B M - Count(x).
        Sample a random run set S' from B according to run set weights.
        Choose configuration C from S'.
    end if
    Perform one training run of N steps using C.
    Calculate the runs Metric.
    Log the result on shared storage.
until terminated by user.

See Appendix B of Loynd et al., 2019 for details.

Owner
Martin
Machine Learning Engineer at heart MSc Student in Computational Science & Engineering :computer: :books: :wrench: @ ETH Zürich :switzerland:
Martin
GitHub Repository
A Python library for simulating finite automata, pushdown automata, and Turing machines

Automata Copyright 2016-2021 Caleb Evans Released under the MIT license Automata is a Python 3 library which implements the structures and algorithms

Caleb Evans 219 Dec 12, 2022
This repository explores an implementation of Grover's Algorithm for knights on a chessboard.

Grover Knights Welcome to my Knights project! Project Description: I explore an implementation of a quantum oracle for knights on a chessboard.

Will Sun 8 Feb 22, 2022
Machine Learning algorithms implementation.

Machine Learning Algorithms Machine Learning algorithms implementation. What can I find here? ML Algorithms KNN K-Means-Clustering SVM (MultiClass) Pe

David Levin 1 Dec 10, 2021
HashDB is a community-sourced library of hashing algorithms used in malware.

HashDB HashDB is a community-sourced library of hashing algorithms used in malware. How To Use HashDB HashDB can be used as a stand alone hashing libr

OALabs 216 Jan 06, 2023
A Python project for optimizing the 8 Queens Puzzle using the Genetic Algorithm implemented in PyGAD.

8QueensGenetic A Python project for optimizing the 8 Queens Puzzle using the Genetic Algorithm implemented in PyGAD. The project uses the Kivy cross-p

Ahmed Gad 16 Nov 13, 2022
An NUS timetable generator which uses a genetic algorithm to optimise timetables to suit the needs of NUS students.

A timetable optimiser for NUS which uses an evolutionary algorithm to "breed" a timetable suited to your needs.

Nicholas Lee 3 Jan 09, 2022
Exact algorithm for computing two-sided statistical tolerance intervals under a normal distribution assumption using Python.

norm-tol-int Exact algorithm for computing two-sided statistical tolerance intervals under a normal distribution assumption using Python. Methods The

Jed Ludlow 1 Jan 06, 2022
Implements (high-dimenstional) clustering algorithm

Description Implements (high-dimenstional) clustering algorithm described in https://arxiv.org/pdf/1804.02624.pdf Dependencies python3 pytorch (=0.4)

Eric Elmoznino 5 Dec 27, 2022
Algorithmic Trading with Python

Source code for Algorithmic Trading with Python (2020) by Chris Conlan

Chris Conlan 1.3k Jan 03, 2023
Python-Strongest-Encrypter - Transform your text into encrypted symbols using their dictionary

How does the encrypter works? Transform your text into encrypted symbols using t

1 Jul 10, 2022
ROS Basics and TurtleSim

Homework 1: Turtle Control Package Anna Garverick This package draws given waypoints, then waits for a service call with a start position to send the

Anna Garverick 1 Nov 22, 2021
N Queen Problem using Genetic Algorithm

The N Queen is the problem of placing N chess queens on an N×N chessboard so that no two queens attack each other.

Mahdi Hassanzadeh 2 Nov 11, 2022
Implementation of Apriori algorithms via Python

Installing run bellow command for installing all packages pip install -r requirements.txt Data Put csv data under this directory "infrastructure/data

Mahdi Rezaei 0 Jul 25, 2022
Supplementary Data for Evolving Reinforcement Learning Algorithms

evolvingrl Supplementary Data for Evolving Reinforcement Learning Algorithms This dataset contains 1000 loss graphs from two experiments: 500 unique g

John Co-Reyes 42 Sep 21, 2022
Implemented page rank program

Page Rank Implemented page rank program based on fact that a website is more important if it is linked to by other important websites using recursive

Vaibhaw 6 Aug 24, 2022
This repository provides some codes to demonstrate several variants of Markov-Chain-Monte-Carlo (MCMC) Algorithms.

Demo-of-MCMC These files are based on the class materials of AEROSP 567 taught by Prof. Alex Gorodetsky at University of Michigan. Author: Hung-Hsiang

Sean 1 Feb 05, 2022
Better control of your asyncio tasks

quattro: task control for asyncio quattro is an Apache 2 licensed library, written in Python, for task control in asyncio applications. quattro is inf

Tin Tvrtković 37 Dec 28, 2022
An implementation of ordered dithering algorithm in python as multimedia course project

One way of minimizing the size of an image is to simply reduce the number of bits you use to represent each pixel.

7 Dec 02, 2022
Apriori - An algorithm for frequent item set mining and association rule learning over relational databases

Apriori Apriori is an algorithm for frequent item set mining and association rul

Mohammad Nazari 8 Jan 10, 2022
frePPLe - open source supply chain planning

frePPLe Open source supply chain planning FrePPLe is an easy-to-use and easy-to-implement open source advanced planning and scheduling tool for manufa

frePPLe 385 Jan 06, 2023