Sequential model-based optimization with a `scipy.optimize` interface

Overview

Logo

pypi conda Travis Status CircleCI Status binder gitter Zenodo DOI

Scikit-Optimize

Scikit-Optimize, or skopt, is a simple and efficient library to minimize (very) expensive and noisy black-box functions. It implements several methods for sequential model-based optimization. skopt aims to be accessible and easy to use in many contexts.

The library is built on top of NumPy, SciPy and Scikit-Learn.

We do not perform gradient-based optimization. For gradient-based optimization algorithms look at scipy.optimize here.

Approximated objective

Approximated objective function after 50 iterations of gp_minimize. Plot made using skopt.plots.plot_objective.

Important links

Install

scikit-optimize requires

  • Python >= 3.6
  • NumPy (>= 1.13.3)
  • SciPy (>= 0.19.1)
  • joblib (>= 0.11)
  • scikit-learn >= 0.20
  • matplotlib >= 2.0.0

You can install the latest release with:

pip install scikit-optimize

This installs an essential version of scikit-optimize. To install scikit-optimize with plotting functionality, you can instead do:

pip install 'scikit-optimize[plots]'

This will install matplotlib along with scikit-optimize.

In addition there is a conda-forge package of scikit-optimize:

conda install -c conda-forge scikit-optimize

Using conda-forge is probably the easiest way to install scikit-optimize on Windows.

Getting started

Find the minimum of the noisy function f(x) over the range -2 < x < 2 with skopt:

import numpy as np
from skopt import gp_minimize

def f(x):
    return (np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2)) +
            np.random.randn() * 0.1)

res = gp_minimize(f, [(-2.0, 2.0)])

For more control over the optimization loop you can use the skopt.Optimizer class:

from skopt import Optimizer

opt = Optimizer([(-2.0, 2.0)])

for i in range(20):
    suggested = opt.ask()
    y = f(suggested)
    opt.tell(suggested, y)
    print('iteration:', i, suggested, y)

Read our introduction to bayesian optimization and the other examples.

Development

The library is still experimental and under heavy development. Checkout the next milestone for the plans for the next release or look at some easy issues to get started contributing.

The development version can be installed through:

git clone https://github.com/scikit-optimize/scikit-optimize.git
cd scikit-optimize
pip install -e.

Run all tests by executing pytest in the top level directory.

To only run the subset of tests with short run time, you can use pytest -m 'fast_test' (pytest -m 'slow_test' is also possible). To exclude all slow running tests try pytest -m 'not slow_test'.

This is implemented using pytest attributes. If a tests runs longer than 1 second, it is marked as slow, else as fast.

All contributors are welcome!

Making a Release

The release procedure is almost completely automated. By tagging a new release travis will build all required packages and push them to PyPI. To make a release create a new issue and work through the following checklist:

  • update the version tag in __init__.py
  • update the version tag mentioned in the README
  • check if the dependencies in setup.py are valid or need unpinning
  • check that the doc/whats_new/v0.X.rst is up to date
  • did the last build of master succeed?
  • create a new release
  • ping conda-forge

Before making a release we usually create a release candidate. If the next release is v0.X then the release candidate should be tagged v0.Xrc1 in __init__.py. Mark a release candidate as a "pre-release" on GitHub when you tag it.

Commercial support

Feel free to get in touch if you need commercial support or would like to sponsor development. Resources go towards paying for additional work by seasoned engineers and researchers.

Made possible by

The scikit-optimize project was made possible with the support of

Wild Tree Tech NYU Center for Data Science NSF Northrop Grumman

If your employer allows you to work on scikit-optimize during the day and would like recognition, feel free to add them to the "Made possible by" list.

Comments
  • [MRG+1] Parallel optimization with skopt

    [MRG+1] Parallel optimization with skopt

    A typical use of Bayesian optimization is in sequential mode, where one does only one experiment (function evaluation) at a time. However, one could hope to reduce the overall wall clock time by running a few function evaluations / experiments at a time.

    Approaches that try such parallelisation typically yield speed up of convergence wrt wall clock time somewhere proportional to the number of experiments run in parallel, eg:

    https://arxiv.org/pdf/1602.05149.pdf (see page 13) https://people.orie.cornell.edu/pfrazier/Presentations/2015.02.CMU.pdf (see slide 32)

    Currently such feature is not supported by skopt explicitly (correct me if I am wrong), but one could nonetheless attempt some simple approaches already. One such approach is to sample multiple points that minimize current surrogate, more precisely something like calling the below function multiple times, and evaluating results in parallel:

    def get_point(X,Y):
        """
        :param X: All points evaluated so far
        :param Y: Objective values at corresponding points
        :return: next point to evaluate
        """
        opt = Optimizer(dimensions=dimensions,base_estimator=some_surrogate)
        opt.tell(X, Y)
        return opt.ask()
    

    Assume for simplicity that at a single sequential step I can run multiple experiments in parallel. Even with simplistic approach above, I seem to be able to achieve speedups proportional to what is shown in the literature when I run multiple experiments in parallel. Consider example of optimizing 2 parameters of decision tree, averaged over 64 repeats (green: 1 experiment / step, blue 2, red: 4):

    figure_1-7

    You can reproduce these results with /examples/skopt_parallel_example.py.

    I have more examples with similar results, for more dimensions (up to 11), other surrogates, other toy and “real” optimization problems etc. I also tried a bit more complicated approaches for obtaining points to evaluate in parallel, but they did not perform much better.

    In light of this, I propose to make a minimalistic ipynb example describing such approach. While I cannot make theoretical argument that it should always yield speedup (probably there are counterexamples), this seems to be useful in practice and at least running multiple experiments instead of one should not do harm. Let me know if you would be interested or if someone is already working on something like this.

    opened by iaroslav-ai 71
  • Remove BayesSearchCV(iid=) parameter deprecated in sklearn 0.24

    Remove BayesSearchCV(iid=) parameter deprecated in sklearn 0.24

    Fixes https://github.com/scikit-optimize/scikit-optimize/issues/978 Fixes https://github.com/scikit-optimize/scikit-optimize/issues/987 Fixes https://github.com/scikit-optimize/scikit-optimize/issues/927 Fixes https://github.com/scikit-optimize/scikit-optimize/issues/990 Fixes https://github.com/scikit-optimize/scikit-optimize/issues/981 via updated scikit-learn

    Fixes https://github.com/scikit-optimize/scikit-optimize/issues/1006

    Fixes https://github.com/scikit-optimize/scikit-optimize/issues/718

    Closes https://github.com/scikit-optimize/scikit-optimize/pull/951

    opened by kernc 40
  • [MRG] Rewrote most of plots.py. Added 2 more plotting functions. Better support for named dimensions.

    [MRG] Rewrote most of plots.py. Added 2 more plotting functions. Better support for named dimensions.

    This pull-request partially solves issue #576.

    I have added some more support for named dimensions in the search-space, as well as two plotting functions. I don't have time to fix your existing plotting functions but I have made TODO comments for what needs to be done in plots.py

    Before I start, let me repeat that I really like your library! So please take the following as friendly and constructive feedback. I spent the whole Saturday from morning to evening working on this and half of this Sunday. A large part of yesterday was just trying to understand your code because of the severe lack of comments. I spent this much effort because I think your library is important and I hope you will polish it some more.

    Please note that I have to travel tomorrow so it might be a while before I can respond if you have questions / comments to this PR.

    How to test this pull-request

    The following code uses the additions of this PR by simulating an objective function for tuning the hyper-parameters of a simple neural network. It will save a few plots to disk and print various info. It should work if you copy this to a python-file and run it.

    (And yes! My code is also ugly when I'm writing it. I polish it when it is done and works. But this code is not for release :-)

    import numpy as np
    from math import exp
    
    from skopt import gp_minimize
    from skopt.space import Real, Categorical, Integer
    from skopt.plots import plot_histogram, plot_contour, plot_objective
    
    dim_learning_rate = Real(name='learning_rate', low=1e-6, high=1e-2, prior='log-uniform')
    dim_num_dense_layers = Integer(name='num_dense_layers', low=1, high=5)
    dim_num_dense_nodes = Integer(name='num_dense_nodes', low=5, high=512)
    dim_activation = Categorical(name='activation', categories=['relu', 'sigmoid'])
    
    dimensions = [dim_learning_rate,
                  dim_num_dense_layers,
                  dim_num_dense_nodes,
                  dim_activation]
    
    default_parameters = [1e-4, 1, 64, 'relu']
    
    
    def model_fitness(x):
        learning_rate, num_dense_layers, num_dense_nodes, activation = x
    
        fitness = ((exp(learning_rate) - 1.0) * 1000) ** 2 + \
                  (num_dense_layers) ** 2 + \
                  (num_dense_nodes/100) ** 2
    
        fitness *= 1.0 + 0.1 * np.random.rand()
    
        if activation == 'sigmoid':
            fitness += 10
    
        return fitness
    
    print(model_fitness(x=default_parameters))
    
    search_result = gp_minimize(func=model_fitness,
                                dimensions=dimensions,
                                n_calls=40,
                                x0=default_parameters)
    
    print(search_result.x)
    print(search_result.fun)
    
    for fitness, x in sorted(zip(search_result.func_vals, search_result.x_iters)):
        print(fitness, x)
    
    space = search_result.space
    
    print(search_result.x_iters)
    
    print(space.to_dict(x=default_parameters))
    
    print("Plotting now ...")
    
    # fig = plot_histogram(result=search_result, dimension_id='activation')
    fig = plot_histogram(result=search_result, dimension_id='learning_rate', bins=20)
    # fig = plot_histogram(result=search_result, dimension_id='num_dense_layers', bins=20)
    # fig = plot_histogram(result=search_result, dimension_id='num_dense_nodes', bins=20)
    fig.savefig('histogram.png')
    
    fig = plot_contour(result=search_result,
                         dimension_id1='learning_rate',
                         dimension_id2='num_dense_nodes')
    fig.savefig('contour_learning_rate_vs_num_dense_nodes.png')
    
    fig = plot_contour(result=search_result,
                         dimension_id1='num_dense_layers',
                         dimension_id2='num_dense_nodes')
    fig.savefig('contour_num_dense_layers_vs_num_dense_nodes.png')
    
    print("Done plotting!")
    

    Comments about this PR

    • I am completely new to your library so it is possible I have misunderstood how something works. Please feel free to make changes to the code I have added. But please keep the good commenting and coding style as it will help people in the future who have to understand and maintain this code.

    • I am not familiar with the syntax you use for doc-strings. I have tried to imitate it. I normally use PyCharm which has a different syntax for doc-strings. It is one of the things I really don't like about Python, that you don't have to specify data-types in the function-declaration, but then you often have to do it in the doc-strings using a strange syntax. It's a really poor language-design. But anyway, please check if my docstrings are correct.

    • Some of the functionality I have added such as space.get_dimensions() is not actually used by my code. It is intended to be helpful for when you fix the other plotting functions to support named dimensions and search-spaces with categories, etc.

    • I added the function space.to_dict(). There is a somewhat related function in utils.point_asdict(). But I prefer the name to_dict() which is also used by Pandas. I realize it is bad to have two naming conventions in one library. Could I convince you to change the name of asdict()? :-)

    Comments about your existing code

    You REALLY need to make better comments in your code! It is very difficult for others to understand what you want to do and how everything fits together. When there is a bug we have to guess what your intentions are with the code. If you don't have time to maintain this library in the future, it will probably become abandoned because others will find it easier to just start a new project rather than to try and understand and build on yours. That would be a pity, because you have some outstanding ideas in your library!

    For example, this is one of the few comments in plots.py:

    # plots on the diagonal are special, like Texas. They have
    # their own range so do not mess with them.
    

    I had to read this numerous times to finally understand that it was probably a joke about Texas USA, although I'm still not 100% sure. You really have to be a master at writing comments in order to pull off jokes, otherwise it will be confusing and harmful to the people who try to understand your code. I personally don't write jokes in code and I am very reluctant to make jokes in my video tutorials as well, for the same reason.

    Please look at the coding and commenting style in the functions I have added. You may think that I write too many comments, but what I am aiming for is to explain as much as possible in plain English. Every time you have a function call into skopt or matplotlib or some other package, the reader needs to look up the semantics of that function. If that is also poorly documented, reading your code becomes like solving a giant puzzle. It takes immense effort and bugs are allowed to propagate very easily. When everything is commented in plain English, you can read and understand the code very quickly.

    Another issue is your actual coding style. I found bugs as well as code that was correct but not very good. For example, these nested loops are typical in plots.py:

    for i in range(space.n_dims):  # rows
        for j in range(space.n_dims):  # columns
            # ...
            if i != j:
                # ...
                if j > 0:
                    # ...
    

    Firstly, you should have a local variable called n_dims instead of always referring to space.n_dims. Secondly, your nesting depth is sometimes 4 or 5 inside these for-loops, because you haven't been very precise in your loop-conditions. You can rewrite the for-loops to something like the following. It drastically reduces the nesting-depth and also only executes the loops that are necessary:

    for i in range(n_dims):
        # Do something for the diagonal case.
    
        for j in range(i):
            # Do something for the case where j<i.
    
        for j in range(i+1, n_dims):
            # Do something for the case where j>i.
    

    You also have this code in plots.py which is incredibly difficult to understand because of the multiple nestings and list-comprehension:

    diagonal_ylim = (np.min([ax[i, i].get_ylim()[0]
                             for i in range(space.n_dims)]),
                     np.max([ax[i, i].get_ylim()[1]
                             for i in range(space.n_dims)]))
    

    It is better to split code like this into several lines to avoid the complicated nesting and also make it easier to step-debug. I added a revised version to the code, although I've commented it out for now so you can test it properly.

    Another thing is that it is quite confusing how the Space and Dimension objects interact and wrap each other, e.g. regarding the transformation. It appears that the transformation and its inverse may be carried out repeatedly to nullify each other, although I'm not sure.

    opened by Hvass-Labs 39
  • [MRG+1] Fix compatibility with sklearn

    [MRG+1] Fix compatibility with sklearn

    Doing anything in the __init__ method of a sklearn estimator will most of the time break his compatibility with sklearn's clone, get_params and set_params, or the ability to fit it several times. It is in fact the case here, it would break if used in a Pipeline or other meta-objects in sklearn.

    This PR is a quick (for now, dirty) proposal to fix the compatibility issues.

    I've basically moved every lines of code that are not self.foo = foo, where foo is an arg, inside the fit() and add_space() methods.

    Fixes #550 #551

    opened by fcharras 38
  • [MRG] Cleanup skopt/parameter.py with docs and some minor changes

    [MRG] Cleanup skopt/parameter.py with docs and some minor changes

    Added docs and some minor cosmetics. Removed prior for now so that we can compare our results with a randomized search where we have some prior knowledge about the candidates. (I understand it might be useful but YAGNI)

    opened by MechCoder 37
  • [WIP] Sphinx based documentation

    [WIP] Sphinx based documentation

    Fixes #428

    Getting started with a documentation marathon.

    • [ ] module doc strings
    • [ ] adapt doc building so sphinx docs are pushed to github.io
    • [ ] structure the API docs
    • [ ] integrate example notebooks
    • [ ] extend quickstart material
    • [ ] add two paragraph summary of when skopt is the right tool
    opened by betatim 33
  • 0.1 release

    0.1 release

    I would like to get the 0.1 release out before school starts again (i.e September). This is just a parent issue to track the blockers.

    • [x] Consistent and backward-compatible API. Addressed by https://github.com/scikit-optimize/scikit-optimize/pull/75
    • [x] SMAC https://github.com/scikit-optimize/scikit-optimize/issues/57
    • (Local search technique that performs better than random sampling on piecewise constant predict functions (https://github.com/scikit-optimize/scikit-optimize/issues/74), postponed till we have a conclusion in #109)
    • [x] Examples (https://github.com/scikit-optimize/scikit-optimize/issues/65)
    • [x] Support for Python 2.7 (#87)
    • [x] Consistent return types #86
    • [x] Name collision #76 (punting for now)
    • [x] Need a logo #107 (code speaks louder than images, no logo required)
    • [ ] release mechanics #133
    • [x] better defaults #166
    • [x] merge #145
    • [x] merge #169
    • [x] maybe merge #162 (nice to have but don't hold the 🚄 for it)
    • [x] stop this list from getting ever longer 📋

    Is there anything else?

    Major 
    opened by MechCoder 32
  • [MRG + 1] Tuning hyperparameters in a pipeline

    [MRG + 1] Tuning hyperparameters in a pipeline

    I didn't see an example of optimizing over the hyperparams of several sklearn objects assembled into a pipeline (i.e. a pipeline generalization examples/hyperparameter-optimization.py). I took a couple hours and made a notebook showing how to do this, if that example seems useful.

    It's kinda drafty and terse compared to the other (really nice!) example notebooks. I'm happy to add/modify if you can think of more stuff that might be interesting to show here.

    opened by cmmalone 31
  • [MRG] Scatter plot matrix

    [MRG] Scatter plot matrix

    Create a scatter plot from a OptimizeResult.

    Do you know how to make the top triangle vanish or something useful we could display there?

    This is the output from a run on hart6: download 7

    opened by betatim 30
  • [WIP] Sphinxing

    [WIP] Sphinxing

    First part of #640.

    Scope

    • [x] Tooling and prerequisites: Cherry pick as much as possible from betatim's existing setup.
      • [x] Building the documentation and creating a html version should behave correctly.
      • [x] Include the napoleon plugin so that we can stay with the Numpy docstring style.
      • [x] Have the first examples show up correctly in the new sphinx docs (to verify that everything is set up correctly). (*)

    (*): Restrict this to the section Top level minimization functions, mainly dummy_minimize.

    Update: Almost finished. I need to look over all the changes again, then i set it to MRG. All aesthetical, content-related or structural changes are not yet done and will be processed in further PRs.

    opened by chschroeder 28
  • [MRG+1] Scikit - Optimize based GridSearchCV plug-in replacement

    [MRG+1] Scikit - Optimize based GridSearchCV plug-in replacement

    A class as discussed in #78 under the tentative name of SkoptSearchCV (discussion on what might be a better name is welcome :)

    Minimalist usage example as of right now:

    from skopt.wrappers import SkoptSearchCV
    from skopt.space import Real, Categorical, Integer
    
    opt = SkoptSearchCV(
        SVC(),
        [{
            'C': Real(1e-6, 1e+6, prior='log-uniform'),
            'gamma': Real(1e-6, 1e+1, prior='log-uniform'),
            'degree': Integer(1, 8),
            'kernel': Categorical(['linear', 'poly', 'rbf']),
        }],
        n_jobs=1, n_iter=32,
    )
    
    opt.fit(X_train, y_train)
    print(opt.score(X_test, y_test))
    

    If something is missing in todos let me know.

    • [x] add skopt.wrappers to setup.py
    • [x] implement basic wrapper similar to RandomizedSearchCV using BaseSearchCV
    • [x] support multiple search spaces as list of dicts, similar to GridSearchCV
    • [x] support parallel search
    • [x] add draft of example usage in sklearn-wrapper
    • [x] add all necessary data to cv_results_
    • [x] add tests
    • [x] fix the docstrings and add comments
    • [x] review and improve example usage
    New feature 
    opened by iaroslav-ai 28
  • AttributeError when using skopt.gp_minimize (numpy 1.24.0)

    AttributeError when using skopt.gp_minimize (numpy 1.24.0)

    I frequently use skopt.gp_minimize. But, since numpy version 1.24.0, it raises an exception

    AttributeError: module 'numpy' has no attribute 'int'
    

    This issue has been anticipated for some time, in the form of a deprecation warning, and it is targeted by PR #1093 and #1123. I think it is redundant for me to contribute with my own PR, since #1123 seems to solve the problem, but fails validation because of seemingly independent issues with the circleci automation.

    In the meantime I will just have to manually add a constraint on the numpy version for my own pipelines, so I have no personal urgency. But I guess it could be a problem for new users, trying to figure out why the function is not working. I hope this issue will help them fix it.

    opened by ChromaticIsobar 2
  • InvalidParameterError when using ExtraTreesRegressor

    InvalidParameterError when using ExtraTreesRegressor

    As per the title, when attempting to use the ExtraTreesRegressor class an sklearn.utils._param_validation.InvalidParameterError exception is raised as below.

    sklearn.utils._param_validation.InvalidParameterError: The 'criterion' parameter of ExtraTreesRegressor must be a str among {'squared_error', 'poisson', 'friedman_mse', 'absolute_error'}. Got 'mse' instead.
    

    The ExtraTreesRegressor.criterion attribute is set using a default optional argument to the constructor with the value mse. It looks like this criterion value was removed in sklearn 1.2 as per this commit.

    It should be an easy fix. Just requires changing the default argument for criterion to either friedman_mse or squared_error I guess... Not sure which one of these would be most appropriate as the default...

    opened by chrisk314 1
  • Optimizing with given initial points

    Optimizing with given initial points

    Hi everyone.

    I would like to apply the command "skopt.optimizer()" to minimize a function with given initial points. However, I could not find any option for inserting given initial values for parameters as can be seen in the following: class skopt.optimizer.Optimizer(dimensions, base_estimator='gp', n_random_starts=None, n_initial_points=10, initial_point_generator='random', n_jobs=1, acq_func='gp_hedge', acq_optimizer='auto', random_state=None, model_queue_size=None, acq_func_kwargs=None, acq_optimizer_kwargs=None) Would you please let me know how to insert initial values using the command. Thanks

    opened by shadibeh 1
  • gp_minimize() does not work in parallel

    gp_minimize() does not work in parallel

    According to the documentation, setting n_jobs should set the number of cores which are used by gp_minimize. I was trying to do that with the following code:

    from skopt import gp_minimize
    
    def J(paramlist):
     	x = paramlist[0]
     	z=0.0
     	for i in range(1,500000000):
     		z+=1.0/i
     	print(z)
     	return z*x**2
      
     res = gp_minimize(J,                 
                       [(-1.0, 1.0)],
                       acq_func="EI",    
                       n_calls=1000,        
                       n_random_starts=1,  
                       random_state=1234,
                       acq_optimizer="lbfgs",
                       n_jobs=5,
                       )
    

    However, regardless of the value of n_jobs, it only runs on a single core.

    opened by btyukodi 0
  • sampler returns a list, not array

    sampler returns a list, not array

    scikit-optimize version: 0.8.0, 0.8.1, 0.9.0

    For all samplers in skopt/sampler/, includes Lhs, Sobol, Halton, Hammersly, Grid, the return type of sampler.generate() is list, but should it be numpy.ndarray according to the class description?

    The reason is that h = space.inverse_transform(h) is called, and inverse_transform converts np.ndarray to list in the end. Add code like h = np.asarray(h) will help.

    Reproduction Code:

    from skopt.sampler import Lhs, Sobol, Halton, Hammersly, Grid
    
    dimensions = [(0.0, 1.0) for i in range(2)]
    size = 4
    for sampler in [Lhs, Sobol, Halton, Hammersly, Grid]:
        print(sampler)
        X = sampler().generate(dimensions, size)
        print(type(X), X)
    
    opened by jhj0411jhj 0
  • Slow Performance

    Slow Performance

    I am seeing very slow performance when using skopt.Optimizer. Here is a reproducible example, that takes ~22.5s on my laptop:

    os.environ["OMP_NUM_THREADS"] = "1"
    import skopt
    import numpy as np
    import time
    noise_level =0.1
    def f(x, noise_level=noise_level):
        return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2))\
               + np.random.randn() * noise_level
    
    def optimize(n_trials=16, n_batches=3):
        opt = skopt.Optimizer([(-2.0, 2.0)], "GP", n_initial_points=n_trials)
        for b in range(n_batches):
            xs = opt.ask(n_trials)
            ys = [f(x) for x in xs]
            opt.tell(xs, ys)
    
    def timeit(fn):
        start = time.time()
        fn()
        end = time.time()
        print(f"Took {end-start:.2f}s")
    
    
    timeit(optimize)
    

    I understand that GPs time complexity is O(n^3), but, this still seems too slow for such a small number of trials (my real problem involves a 6 dimensional space and hundreds of trials). Note that changing the base_estimator from GP to GBRT cuts the excution time to ~6s for this toy example. What am I missing?

    opened by nickos556 0
Releases(v0.9.0)
  • v0.9.0(Oct 12, 2021)

    What's Changed

    • [MRG]Add hollow iterations early stopper by @ludaavics in https://github.com/scikit-optimize/scikit-optimize/pull/917
    • Doc line, logspace call, set_transformer default parameter misstype, … by @GCidd in https://github.com/scikit-optimize/scikit-optimize/pull/921
    • Revert change to plot_histogramm by @holgern in https://github.com/scikit-optimize/scikit-optimize/pull/942
    • [MRG] Fixes GaussianProcessRegressor for sklearn >= 0.23 by @holgern in https://github.com/scikit-optimize/scikit-optimize/pull/943
    • [MRG] Refactor Sobol' API by @tupui in https://github.com/scikit-optimize/scikit-optimize/pull/955
    • docs: fix simple typo, stategies -> strategies by @timgates42 in https://github.com/scikit-optimize/scikit-optimize/pull/979
    • [MRG] Correct documented default is_int for Normalize. by @gcp in https://github.com/scikit-optimize/scikit-optimize/pull/1019
    • [MRG] add ThresholdStopper() by @GuillaumeSimo in https://github.com/scikit-optimize/scikit-optimize/pull/1000
    • [MRG] Check whether prior values are valid. by @gcp in https://github.com/scikit-optimize/scikit-optimize/pull/1020
    • Fix numpy Deprecation errors by @xmatthias in https://github.com/scikit-optimize/scikit-optimize/pull/1023
    • Remove BayesSearchCV(iid=) parameter deprecated in sklearn 0.24 by @kernc in https://github.com/scikit-optimize/scikit-optimize/pull/988
    • Re-add BayesSearchCV.best_score_ needed by some examples by @kernc in https://github.com/scikit-optimize/scikit-optimize/pull/1031
    • Revert "Re-add BayesSearchCV.best_score_ needed by some examples" by @kernc in https://github.com/scikit-optimize/scikit-optimize/pull/1032
    • Use deepcopy to prevent reference cycles in Optimizer by @freddyaboulton in https://github.com/scikit-optimize/scikit-optimize/pull/1029
    • [MRG] Fixed grammatical error in docs: 'less' vs 'fewer' by @samuelstevens in https://github.com/scikit-optimize/scikit-optimize/pull/1040
    • [MRG] update packges in circleci deps by @QuentinSoubeyran in https://github.com/scikit-optimize/scikit-optimize/pull/1069
    • [MRG] Fix GaussianProcessRegressor for scikit-learn 1.0 by @QuentinSoubeyran in https://github.com/scikit-optimize/scikit-optimize/pull/1063
    • Fix remaining sklearn 1.0 return_X_y= error by @kernc in https://github.com/scikit-optimize/scikit-optimize/pull/1078
    • MNT: Fix CircleCI builds by unpinning + full docs for PRs by @kernc in https://github.com/scikit-optimize/scikit-optimize/pull/1079
    • MNT: Fix "LaTeX Error: File 'tgtermes.sty' not found." on CircleCI by @kernc in https://github.com/scikit-optimize/scikit-optimize/pull/1080

    New Contributors

    • @ludaavics made their first contribution in https://github.com/scikit-optimize/scikit-optimize/pull/917
    • @GCidd made their first contribution in https://github.com/scikit-optimize/scikit-optimize/pull/921
    • @tupui made their first contribution in https://github.com/scikit-optimize/scikit-optimize/pull/955
    • @timgates42 made their first contribution in https://github.com/scikit-optimize/scikit-optimize/pull/979
    • @GuillaumeSimo made their first contribution in https://github.com/scikit-optimize/scikit-optimize/pull/1000
    • @xmatthias made their first contribution in https://github.com/scikit-optimize/scikit-optimize/pull/1023
    • @kernc made their first contribution in https://github.com/scikit-optimize/scikit-optimize/pull/988
    • @freddyaboulton made their first contribution in https://github.com/scikit-optimize/scikit-optimize/pull/1029
    • @samuelstevens made their first contribution in https://github.com/scikit-optimize/scikit-optimize/pull/1040
    • @QuentinSoubeyran made their first contribution in https://github.com/scikit-optimize/scikit-optimize/pull/1069

    Full Changelog: https://github.com/scikit-optimize/scikit-optimize/compare/v0.8.1...v0.9.0

    Source code(tar.gz)
    Source code(zip)
  • v0.8.1(Sep 4, 2020)

    • GaussianProcessRegressor from sklearn >= 0.23 normalizes the variance to 1, which needs to be taken into account. This release handles this and returns the same results as on sklearn <= 0.23
    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Sep 3, 2020)

    • n_jobs support was added to Optimizer and fixed for forest_minimize #884

    • Allow dimension selection for plot_objective and plot_evaluations and add plot_histogram and plot_objective_2D. Plot code has been refactored. #848

    • Initial sampling generation from latin hypercube, sobol, hammersly and halton is possible and can be set in all optimizers #835

    • Improve sampler and add grid sampler #851

    • Fix library for scikit-learn >= 0.23. numpy MaskArray is replaced by numpy.ma.array. y_normalize=False has been added and initial runs has been increased. #939

    • Integer transform and inverse_transform for normalize #880

    • Add is_constant property to dimension and n_constant_dimensions property to Space #883

    • Skip constant dimensions for plot_objective and plot_evaluations to allow plots using BayesSearchCV #888

    • Fix Fix Optimizer for full categorical spaces #874

    • Improve circle ci #852

    • Add project toml and adapt minimal numpy, scipy, pyyaml and joblib version in setup.py #850

    Source code(tar.gz)
    Source code(zip)
  • v0.7.4(Feb 23, 2020)

  • v0.7.3(Feb 23, 2020)

  • v0.7.2(Feb 11, 2020)

    Version 0.7.2

    New features

    • Add expected_minimum_random_sampling
    • New plot examples
    • Add more parameter to plot_objective
    • Return ordereddict in point_asdict
    • update_next() and get_results() added to Optimize

    Bug fixes

    • Fix searchcv rank (issue #831)
    • Fix random forest regressor (issue #766)
    • Fix doc examples
    • Fix integer normalize by using round()
    • Fix random forest regressor (Add missing min_impurity_decrease)

    Maintenance

    • Fix license detection in github
    • Add doctest to CI
    Source code(tar.gz)
    Source code(zip)
  • v0.7.1(Feb 2, 2020)

    Version 0.7.1

    New features

    • Sphinx documentation
    • notebooks are replaced by sphinx-gallery
    • New StringEncoder, can be used in Categoricals
    • Remove string conversion in Identity
    • dtype can be set in Integer and Real

    Bug fixes

    • Fix categorical space (issue #821)
    • int can be set as dtype to fix issue #790

    Maintenance

    • Old pdoc scripts are removed and replaced by sphinx
    Source code(tar.gz)
    Source code(zip)
  • v0.7(Jan 29, 2020)

    Version 0.7

    New features

    • Models queue has now a customizable size (model_queue_size).
    • Add log-uniform prior to Integer space
    • Support for plotting categorical dimensions

    Bug fixes

    • Allow BayesSearchCV to work with sklearn 0.21
    • Reduce the amount of deprecation warnings in unit tests

    Maintenance

    • joblib instead of sklearn.externals.joblib
    • Improve travis CI unit tests (Different sklearn version are checked)
    • Added versioneer support, to keep things simple and to fix pypi deploy
    Source code(tar.gz)
    Source code(zip)
  • v0.7rc4(Jan 29, 2020)

  • v0.7rc3(Jan 29, 2020)

  • v0.7rc2(Jan 29, 2020)

  • v0.7rc1(Jan 29, 2020)

  • v0.6(Apr 21, 2019)

    Version 0.6

    Highly composite six.

    New features

    • plot_regret function for plotting the cumulative regret; The purpose of such plot is to access how much an optimizer is effective at picking good points.
    • CheckpointSaver that can be used to save a checkpoint after each iteration with skopt.dump
    • Space.from_yaml() to allow for external file to define Space parameters

    Bug fixes

    • Fixed numpy broadcasting issues in gaussian_ei, gaussian_pi
    • Fixed build with newest scikit-learn
    • Use native python types inside BayesSearchCV
    • Include fit_params in BayesSearchCV refit

    Maintenance

    • Added versioneer support, to reduce changes with new version of the skopt
    Source code(tar.gz)
    Source code(zip)
  • v0.6rc1(Apr 21, 2019)

  • v0.5.2(Mar 25, 2018)

    Version 0.5.2

    Bug fixes

    • Separated n_points from n_jobs in BayesSearchCV.
    • Dimensions now support boolean np.arrays.

    Maintenance

    • matplotlib is now an optional requirement (install with pip install 'scikit-optimize[plots]')
    Source code(tar.gz)
    Source code(zip)
  • v0.5.1(Feb 9, 2018)

  • v0.5(Feb 5, 2018)

    Version 0.5

    High five!

    New features

    • Single element dimension definition, which can be used to fix the value of a dimension during optimization.
    • total_iterations property of BayesSearchCV that counts total iterations needed to explore all subspaces.
    • Add iteration event handler for BayesSearchCV, useful for early stopping inside BayesSearchCV search loop.
    • Added utils.use_named_args decorator to help with unpacking named dimensions when calling an objective function.

    Bug fixes

    • Removed redundant estimator fitting inside BayesSearchCV.
    • Fixed the log10 transform for Real dimensions that would lead to values being out of bounds.
    Source code(tar.gz)
    Source code(zip)
  • v0.5rc1(Jan 23, 2018)

  • v0.4(Aug 21, 2017)

    Version 0.4

    Go forth!

    New features

    • Support early stopping of optimization loop.
    • Benchmarking scripts to evaluate performance of different surrogate models.
    • Support for parallel evaluations of the objective function via several constant liar stategies.
    • BayesSearchCV as a drop in replacement for scikit-learn's GridSearchCV.
    • New acquisition functions "EIps" and "PIps" that takes into account function compute time.

    Bug fixes

    • Fixed inference of dimensions of type Real.

    API changes

    • Change interface of GradientBoostingQuantileRegressor's predict method to match return type of other regressors
    • Dimensions of type Real are now inclusive of upper bound.
    Source code(tar.gz)
    Source code(zip)
  • v0.4rc1(Aug 21, 2017)

  • v0.3(Feb 13, 2017)

    Third time's a charm.

    New features

    • Accuracy improvements of the optimization of the acquisition function by pre-selecting good candidates as starting points when using acq_optimizer='lbfgs'.
    • Support a ask-and-tell interface. Check out the Optimizer class if you need fine grained control over the iterations.
    • Parallelize L-BFGS minimization runs over the acquisition function.
    • Implement weighted hamming distance kernel for problems with only categorical dimensions.
    • New acquisition function gp_hedge that probabilistically chooses one of EI, PI or LCB at every iteration depending upon the cumulative gain.

    Bug fixes

    • Warnings are now raised if a point is chosen as the candidate optimum multiple times.
    • Infinite gradients that were raised in the kernel gradient computation are now fixed.
    • Integer dimensions are now normalized to [0, 1] internally in gp_minimize.

    API Changes.

    • The default acq_optimizer function has changed from "auto" to "lbfgs" in gp_minimize.
    Source code(tar.gz)
    Source code(zip)
  • v0.2(Oct 28, 2016)

  • v0.1-fix(Aug 10, 2016)

An integration of several popular automatic augmentation methods, including OHL (Online Hyper-Parameter Learning for Auto-Augmentation Strategy) and AWS (Improving Auto Augment via Augmentation Wise Weight Sharing) by Sensetime Research.

An integration of several popular automatic augmentation methods, including OHL (Online Hyper-Parameter Learning for Auto-Augmentation Strategy) and AWS (Improving Auto Augment via Augmentation Wise

45 Dec 08, 2022
Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks

Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks This is a Pytorch-Lightning implementation of the paper "Self-s

Photogrammetry & Robotics Bonn 111 Dec 06, 2022
[NeurIPS-2021] Slow Learning and Fast Inference: Efficient Graph Similarity Computation via Knowledge Distillation

Efficient Graph Similarity Computation - (EGSC) This repo contains the source code and dataset for our paper: Slow Learning and Fast Inference: Effici

23 Nov 11, 2022
Implementation of the paper titled "Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guarantees"

Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guarantees Implementation of the paper titled "Using Sampling to

MIDAS, IIIT Delhi 2 Aug 29, 2022
scikit-learn inspired API for CRFsuite

sklearn-crfsuite sklearn-crfsuite is a thin CRFsuite (python-crfsuite) wrapper which provides interface simlar to scikit-learn. sklearn_crfsuite.CRF i

417 Dec 20, 2022
Alpha-Zero - Telegram Group Manager Bot Written In Python Using Pyrogram

✨ Alpha Zero Bot ✨ Telegram Group Manager Bot + Userbot Written In Python Using

1 Feb 17, 2022
PyTorch implementation of Glow

glow-pytorch PyTorch implementation of Glow, Generative Flow with Invertible 1x1 Convolutions (https://arxiv.org/abs/1807.03039) Usage: python train.p

Kim Seonghyeon 433 Dec 27, 2022
UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation

UNION Automatic Evaluation Metric described in the paper UNION: An UNreferenced MetrIc for Evaluating Open-eNded Story Generation (EMNLP 2020). Please

50 Dec 30, 2022
Code of the paper "Shaping Visual Representations with Attributes for Few-Shot Learning (ASL)".

Shaping Visual Representations with Attributes for Few-Shot Learning This code implements the Shaping Visual Representations with Attributes for Few-S

chx_nju 9 Sep 01, 2022
[CVPR 2021] Region-aware Adaptive Instance Normalization for Image Harmonization

RainNet — Official Pytorch Implementation Region-aware Adaptive Instance Normalization for Image Harmonization Jun Ling, Han Xue, Li Song*, Rong Xie,

130 Dec 11, 2022
Implementation of CSRL from the AAAI2022 paper: Constraint Sampling Reinforcement Learning: Incorporating Expertise For Faster Learning

CSRL Implementation of CSRL from the AAAI2022 paper: Constraint Sampling Reinforcement Learning: Incorporating Expertise For Faster Learning Python: 3

4 Apr 14, 2022
Implementation of Segformer, Attention + MLP neural network for segmentation, in Pytorch

Segformer - Pytorch Implementation of Segformer, Attention + MLP neural network for segmentation, in Pytorch. Install $ pip install segformer-pytorch

Phil Wang 208 Dec 25, 2022
Tensorflow implementation of soft-attention mechanism for video caption generation.

SA-tensorflow Tensorflow implementation of soft-attention mechanism for video caption generation. An example of soft-attention mechanism. The attentio

Paul Chen 153 Nov 14, 2022
App customer segmentation cohort rfm clustering

CUSTOMER SEGMENTATION COHORT RFM CLUSTERING TỔNG QUAN VỀ HỆ THỐNG DỮ LIỆU Nên chuyển qua theme màu dark thì sẽ nhìn đẹp hơn https://customer-segmentat

hieulmsc 3 Dec 18, 2021
The AugNet Python module contains functions for the fast computation of image similarity.

AugNet AugNet: End-to-End Unsupervised Visual Representation Learning with Image Augmentation arxiv link In our work, we propose AugNet, a new deep le

Ming 74 Dec 28, 2022
EMNLP 2021 Findings' paper, SCICAP: Generating Captions for Scientific Figures

SCICAP: Scientific Figures Dataset This is the Github repo of the EMNLP 2021 Findings' paper, SCICAP: Generating Captions for Scientific Figures (Hsu

Edward 26 Nov 21, 2022
an Evolutionary Algorithm assisted GAN

EvoGAN an Evolutionary Algorithm assisted GAN ckpts

3 Oct 09, 2022
GLODISMO: Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery

GLODISMO: Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery This is the code to the paper: Gradient-Based Learn

3 Feb 15, 2022
Classifying audio using Wavelet transform and deep learning

Audio Classification using Wavelet Transform and Deep Learning A step-by-step tutorial to classify audio signals using continuous wavelet transform (C

Aditya Dutt 17 Nov 29, 2022
[AAAI-2022] Official implementations of MCL: Mutual Contrastive Learning for Visual Representation Learning

Mutual Contrastive Learning for Visual Representation Learning This project provides source code for our Mutual Contrastive Learning for Visual Repres

winycg 48 Jan 02, 2023