Model search is a framework that implements AutoML algorithms for model architecture search at scale

Overview

Model Search

header

Model search (MS) is a framework that implements AutoML algorithms for model architecture search at scale. It aims to help researchers speed up their exploration process for finding the right model architecture for their classification problems (i.e., DNNs with different types of layers).

The library enables you to:

  • Run many AutoML algorithms out of the box on your data - including automatically searching for the right model architecture, the right ensemble of models and the best distilled models.

  • Compare many different models that are found during the search.

  • Create you own search space to customize the types of layers in your neural networks.

The technical description of the capabilities of this framework are found in InterSpeech paper.

While this framework can potentially be used for regression problems, the current version supports classification problems only. Let's start by looking at some classic classification problems and see how the framework can automatically find competitive model architectures.

Getting Started

Let us start with the simplest case. You have a csv file where the features are numbers and you would like to run let AutoML find the best model architecture for you.

Below is a code snippet for doing so:

import model_search
from model_search import constants
from model_search import single_trainer
from model_search.data import csv_data

trainer = single_trainer.SingleTrainer(
    data=csv_data.Provider(
        label_index=0,
        logits_dimension=2,
        record_defaults=[0, 0, 0, 0],
        filename="model_search/data/testdata/csv_random_data.csv"),
    spec=constants.DEFAULT_DNN)

trainer.try_models(
    number_models=200,
    train_steps=1000,
    eval_steps=100,
    root_dir="/tmp/run_example",
    batch_size=32,
    experiment_name="example",
    experiment_owner="model_search_user")

The above code will try 200 different models - all binary classification models, as the logits_dimension is 2. The root directory will have a subdirectory of all models, all of which will be already evaluated. You can open the directory with tensorboard and see all the models with the evaluation metrics.

The search will be performed according to the default specification. That can be found in: model_search/configs/dnn_config.pbtxt.

For more details about the fields and if you want to create your own specification, you can look at: model_search/proto/phoenix_spec.proto.

Now, what if you don't have a csv with the features? The next section shows how to run without a csv.

Non-csv data

To run with non-csv data, you will have to implement a class inherited from the abstract class model_search.data.Provider. This enables us to define our own input_fn and hence customize the feature columns and the task (i.e., the number of classes in the classification task).

class Provider(object, metaclass=abc.ABCMeta):
  """A data provider interface.

  The Provider abstract class that defines three function for Estimator related
  training that return the following:
    * An input function for training and test input functions that return
      features and label batch tensors. It is responsible for parsing the
      dataset and buffering data.
    * The feature_columns for this dataset.
    * problem statement.
  """

  def get_input_fn(self, hparams, mode, batch_size: int):
    """Returns an `input_fn` for train and evaluation.

    Args:
      hparams: tf.HParams for the experiment.
      mode: Defines whether this is training or evaluation. See
        `estimator.ModeKeys`.
      batch_size: the batch size for training and eval.

    Returns:
      Returns an `input_fn` for train or evaluation.
    """

  def get_serving_input_fn(self, hparams):
    """Returns an `input_fn` for serving in an exported SavedModel.

    Args:
      hparams: tf.HParams for the experiment.

    Returns:
      Returns an `input_fn` that takes no arguments and returns a
        `ServingInputReceiver`.
    """

  @abc.abstractmethod
  def number_of_classes(self) -> int:
    """Returns the number of classes. Logits dim for regression."""

  def get_feature_columns(
      self
  ) -> List[Union[feature_column._FeatureColumn,
                  feature_column_v2.FeatureColumn]]:
    """Returns a `List` of feature columns."""

An example of an implementation can be found in model_search/data/csv_data.py.

Once you have this class, you can pass it to model_search.single_trainer.SingleTrainer and your single trainer can now read your data.

Adding your models and architectures to a search space

You can use our platform to test your own existing models.

Our system searches over what we call blocks. We have created an abstract API for an object that resembles a layer in a DNN. All that needs to be implemented for this class is two functions:

class Block(object, metaclass=abc.ABCMeta):
  """Block api for creating a new block."""

  @abc.abstractmethod
  def build(self, input_tensors, is_training, lengths=None):
    """Builds a block for phoenix.

    Args:
      input_tensors: A list of input tensors.
      is_training: Whether we are training. Used for regularization.
      lengths: The lengths of the input sequences in the batch.

    Returns:
      output_tensors: A list of the output tensors.
    """

  @abc.abstractproperty
  def is_input_order_important(self):
    """Is the order of the entries in the input tensor important.

    Returns:
      A bool specifying if the order of the entries in the input is important.
      Examples where the order is important: Input for a cnn layer.
      (e.g., pixels an image). Examples when the order is not important:
      Input for a dense layer.
    """

Once you have implemented your own blocks (i.e., layers), you need to register them with a decorator. Example:

@register_block(
    lookup_name='AVERAGE_POOL_2X2', init_args={'kernel_size': 2}, enum_id=8)
@register_block(
    lookup_name='AVERAGE_POOL_4X4', init_args={'kernel_size': 4}, enum_id=9)
class AveragePoolBlock(Block):
  """Average Pooling layer."""

  def __init__(self, kernel_size=2):
    self._kernel_size = kernel_size

  def build(self, input_tensors, is_training, lengths=None):

(All code above can be found in model_search/blocks.py). Once registered, you can tell the system to search over these blocks by supplying them in blocks_to_use in PhoenixSpec in model_search/proto/phoenix_spec.proto. Namely, if you look at the default specification for dnn found in model_search/configs/dnn_config.pbtxt, you can change the repeated field blocks_to_use and add you own registered blocks.

Note: Our system stacks blocks one on top of each other to create tower architectures that are then going to be ensembled. You can set the minimal and maximal depth allowed in the config to 1 which will change the system to search over which block perform best for the problem - I.e., your blocks can be now an implementation of full classifiers and the system will choose the best one.

Creating a training stand alone binary without writing a main

Now, let's assume you have the data class, but you don't want to write a main function to run it.

We created a simple way to create a main that will just train a dataset and is configurable via flags.

To create it, you need to follow two steps:

  1. You need to register your data provider.

  2. You need to call a help function to create a build rule.

Example: Suppose you have a provider, then you need to register it via a decorator we define it as follows:

@data.register_provider(lookup_name='csv_data_provider', init_args={})
class Provider(data.Provider):
  """A csv data provider."""

  def __init__(self):

The above code can be found in model_search/data/csv_data_for_binary.py.

Next, once you have such library (data provider defined in a .py file and registered), you can supply this library to a help build function an it will create a binary rule as follows:

model_search_oss_binary(
    name = "csv_data_binary",
    dataset_dep = ":csv_data_for_binary",
)

You can also add a test automatically to test integration of your provider with the system as follows:

model_search_oss_test(
    name = "csv_data_for_binary_test",
    dataset_dep = ":csv_data_for_binary",
    problem_type = "dnn",
    extra_args = [
        "--filename=$${TEST_SRCDIR}/model_search/data/testdata/csv_random_data.csv",
    ],
    test_data = [
        "//model_search/data/testdata:csv_random_data",
    ],
)

The above function will create a runable binary. The snippets are taken from the following file: model_search/data/BUILD. The binary is configurable by the flags in model_search/oss_trainer_lib.py.

Distributed Runs

Our system can run a distributed search - I.e., run many search trainer in parallel.

How does it work?

You need to run your binary on multiple machines. Additionally, you need to make one change to configure the bookkeeping of the search.

On a single machine, the bookkeeping is done via a file. For a distributed system however, we need a database.

In order to point our system to the database, you need to set the flags in the file:

model_search/metadata/ml_metadata_db.py

to point to your database.

Once you have done so, the binaries created from the previous section will connect to this database and an async search will begin.

Cloud AutoML

Want to try higher performance AutoML without writing code? Try: https://cloud.google.com/automl-tables

Issues
  • absl.flags._exceptions.UnparsedFlagAccessError: Trying to access flag --mlmd_default_sqllite_filename before flags were parsed

    absl.flags._exceptions.UnparsedFlagAccessError: Trying to access flag --mlmd_default_sqllite_filename before flags were parsed

    I get the error when I run the code snippet from the project page:

    Traceback (most recent call last): File "btc.py", line 14, in <module> trainer.try_models( File "/home/lasse/Development/projects/btcpred/model_search/model_search/single_trainer.py", line 56, in try_models phoenix_instance = phoenix.Phoenix( File "/home/lasse/Development/projects/btcpred/model_search/model_search/phoenix.py", line 239, in __init__ self._metadata = ml_metadata_db.MLMetaData(phoenix_spec, study_name, File "/home/lasse/Development/projects/btcpred/model_search/model_search/metadata/ml_metadata_db.py", line 84, in __init__ if FLAGS.mlmd_default_sqllite_filename: File "/home/lasse/Development/projects/btcpred/.env/lib/python3.8/site-packages/absl/flags/_flagvalues.py", line 498, in __getattr__ raise _exceptions.UnparsedFlagAccessError(error_message) absl.flags._exceptions.UnparsedFlagAccessError: Trying to access flag --mlmd_default_sqllite_filename before flags were parsed.

    opened by lsiem 6
  • Cannot connect sqlite3 database

    Cannot connect sqlite3 database

    Hi,

    I cannot connect to database while running the “Getting started” example. How can I fix this? Thank you.

    Here is the full trace:

    Traceback (most recent call last):
    
    
    
     File "C:\Users\e173196\Anaconda projects\model_search\load_flags.py", line 32, in <module>
    
       trainer.try_models(
    
    
    
     File "C:\Users\e173196\Anaconda projects\model_search\model_search\single_trainer.py", line 57, in try_models
    
       phoenix_instance = phoenix.Phoenix(
    
    
    
     File "C:\Users\e173196\Anaconda projects\model_search\model_search\phoenix.py", line 239, in __init__
    
       self._metadata = ml_metadata_db.MLMetaData(phoenix_spec, study_name,
    
    
    
     File "C:\Users\e173196\Anaconda projects\model_search\model_search\metadata\ml_metadata_db.py", line 100, in __init__
    
       self._store = metadata_store.MetadataStore(self._connection_config)
    
    
    
     File "C:\Users\e173196\Anaconda3\envs\Model_search\lib\site-packages\ml_metadata\metadata_store\metadata_store.py", line 91, in __init__
    
       self._metadata_store = metadata_store_serialized.CreateMetadataStore(
    
    
    
    RuntimeError: Cannot connect sqlite3 database: unable to open database file
    
    
    opened by j850613enna 6
  • What does

    What does "experiment_owner" mean? I don’t understand how to fill in this parameter???

    error:UnparsedFlagAccessError: Trying to access flag --mlmd_default_sqllite_filename before flags were parsed.

    opened by Mrmdzz 5
  • setup.py

    setup.py

    Hello, It setup.py missing, or I'm missing something?

    Thanks, Vic

    opened by VictorT787 4
  • model_search not available yet on pip

    model_search not available yet on pip

    When is it going to be available as a pip package?

    image

    Thanks a million!

    opened by GermanCM 4
  • How do I train my own data for image classification by this code?

    How do I train my own data for image classification by this code?

    How do I train my own data for image classification by this code?

    opened by KaichiehKJ 4
  • UnparsedFlagAccessError

    UnparsedFlagAccessError

    Hi - What could be causing this?


    UnparsedFlagAccessError Traceback (most recent call last) in () 14 batch_size=32, 15 experiment_name="animalfaces", ---> 16 experiment_owner="myname")

    3 frames /usr/local/lib/python3.6/dist-packages/absl/flags/_flagvalues.py in getattr(self, name) 496 # get too much noise. 497 logging.error(error_message) --> 498 raise _exceptions.UnparsedFlagAccessError(error_message) 499 500 def setattr(self, name, value):

    UnparsedFlagAccessError: Trying to access flag --mlmd_default_sqllite_filename before flags were parsed.

    opened by iandoriath 3
  • Project import generated by Copybara.

    Project import generated by Copybara.

    PiperOrigin-RevId: 352621578

    opened by hanna-maz 3
  • NewRandomAccessFile failed to Create/Open

    NewRandomAccessFile failed to Create/Open

    NotFoundError: NewRandomAccessFile failed to Create/Open: model_search/model_search/configs/dnn_config.pbtxt : The system cannot find the path specified. ; No such process

    It seems like the path "model_search" was imported twice...Any idea how to solve it? Thanks in advance!

    opened by Shoot-to-root 2
  • sqlite default filename is not Windows compatible

    sqlite default filename is not Windows compatible

    File model_search/metadata/ml_metadata_db.py, lines 102-103 are:

            self._connection_config.sqlite.filename_uri = (
                "/tmp/filedb-%d" % random.randint(0, 1000000))
    

    /tmp/... is of course not a valid path for Windows. Can you update this path such that it chooses a valid path when running on Windows?

    opened by DeanIsMe 0
  • Update README.md

    Update README.md

    Fix invalid syntax

    opened by ggirou 0
  • How to know which model is the best and what is its accuracy?

    How to know which model is the best and what is its accuracy?

    I have trained 200 models using trainer.try_models() but now I need to know which model performed the best and its accuracy?

    Anyone knows how to get the details such as: what evaluation score was used? what is the best model? and how can we use it later for prediction?

    Thank you.

    opened by meriemferdjouni 4
  • How to predict from model_search saved_model.pb

    How to predict from model_search saved_model.pb

    Hi, I have tried simple classification dataset "German credit data" to make predictions. I load the model_search saved_model.pb model. importedModel = tf.saved_model.load(saved_model_dir)

    But, when I try to predict or make summary() from the model ,I get the following error,

    AttributeError: 'AutoTrackable' object has no attribute 'summary'

    When I print print (gs_model.signatures), I got "['serving_default']".

    How can I predict, get metrics like f1_score, accuracy, R2,etc from the model.

    Is it possible to store the model_search model as tenorflow v2 compatible in oss_trainer_lib.py instead of "estimator"model.

    Pls suggest on this. I am using tensorflow 2.4.0 How can I make the model to work like keras model.

    Thanks, SJRam

    opened by jayarams79 3
  • Training on Multiclass Image Dataset

    Training on Multiclass Image Dataset

    Is there already a way to use the code for multiclass image datasets? Documentation shows only for binary image datasets. Tried changing the "label_mode" variable in image_data.py to "categorical" and change the return value of the "number_of_classes" function to the number of classes. Still an error.

    opened by Monkeydion 1
  • Possible beta for  Regression Modeling?

    Possible beta for Regression Modeling?

    The current task that I'm trying to run requires regression modeling, and I'm wondering if there is any progress for creating a model for regression analysis? For example, if there is some beta code or something similar that I could use, it would be greatly appreciated.

    opened by solomon-lo 1
  • Documentation for Utilizing Output of Model Search?

    Documentation for Utilizing Output of Model Search?

    Is there anywhere where we can find additional documentation? I've run the Google Model Search and obtained a saved_model.pb, but I'm not sure how to load this back into Python.

    Here is some code that I've found for converting the output, but it doesn't seem quite right. So, documentation for how to actually implement this would be appreciated.

    image

    opened by solomon-lo 0
  • Neural Network Architecture Details

    Neural Network Architecture Details

    I ran the default setting on my dataset and I am trying to figure out what the architecture of the model looks like (i.e how many layers, its size) and what Hyperparameter values did it choose. Where could I find this information?

    opened by gnaven 3
  • I'm confused by distributed search, any suggestions are welcomed!

    I'm confused by distributed search, any suggestions are welcomed!

    Any detailed method for single-machine multi-card distributed search?

    opened by CyFeng16 0
  • Is it feasible to use MirroredStrategy for training?

    Is it feasible to use MirroredStrategy for training?

    Thank you for making this work open source, and let us glimpse some of Google's achievements on the NAS platform. , I have successfully run feature data, binary and multi-classes image network search after checking materials and issues.

    The current problem: model_search is only trained on a single GPU card, and the computational efficiency is not satisfactory. Is there a way to modify model_search (via distributing configs?) for multi-GPU card synchronous/asynchronous training?

    opened by CyFeng16 0
Owner
Google
Google ❤️ Open Source
Google
code for paper "Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?"

Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? Code for paper: Does Unsupervised Architecture Representation

null 33 Dec 22, 2021
MMRazor: a model compression toolkit for model slimming and AutoML

Documentation: https://mmrazor.readthedocs.io/ English | 简体中文 Introduction MMRazor is a model compression toolkit for model slimming and AutoML, which

OpenMMLab 419 Jan 24, 2022
Implements MLP-Mixer: An all-MLP Architecture for Vision.

MLP-Mixer-CIFAR10 This repository implements MLP-Mixer as proposed in MLP-Mixer: An all-MLP Architecture for Vision. The paper introduces an all MLP (

Sayak Paul 42 Jan 21, 2022
Code release to accompany paper "Geometry-Aware Gradient Algorithms for Neural Architecture Search."

Geometry-Aware Gradient Algorithms for Neural Architecture Search This repository contains the code required to run the experiments for the DARTS sear

null 16 Dec 14, 2021
This implements one of result networks from Large-scale evolution of image classifiers

Exotic structured image classifier This implements one of result networks from Large-scale evolution of image classifiers by Esteban Real, et. al. Req

null 54 Dec 21, 2021
FastReID is a research platform that implements state-of-the-art re-identification algorithms.

FastReID is a research platform that implements state-of-the-art re-identification algorithms.

JDAI-CV 2.4k Jan 27, 2022
deep-table implements various state-of-the-art deep learning and self-supervised learning algorithms for tabular data using PyTorch.

deep-table implements various state-of-the-art deep learning and self-supervised learning algorithms for tabular data using PyTorch.

null 45 Jan 19, 2022
Densely Connected Search Space for More Flexible Neural Architecture Search (CVPR2020)

DenseNAS The code of the CVPR2020 paper Densely Connected Search Space for More Flexible Neural Architecture Search. Neural architecture search (NAS)

Jamin Fong 284 Jan 18, 2022
Clairvoyance: a Unified, End-to-End AutoML Pipeline for Medical Time Series

Clairvoyance: A Pipeline Toolkit for Medical Time Series Authors: van der Schaar Lab This repository contains implementations of Clairvoyance: A Pipel

van_der_Schaar \LAB 66 Jan 18, 2022
An AutoML Library made with Optuna and PyTorch Lightning

An AutoML Library made with Optuna and PyTorch Lightning Installation Recommended pip install -U gradsflow From source pip install git+https://github.

GradsFlow 266 Jan 29, 2022
Neural networks applied in recognizing guitar chords using python, AutoML.NET with C# and .NET Core

Chord Recognition Demo application The demo application is written in C# with .NETCore. As of July 9, 2020, the only version available is for windows

Andres Mauricio Rondon Patiño 19 Jan 20, 2022
AutoDeeplab / auto-deeplab / AutoML for semantic segmentation, implemented in Pytorch

AutoML for Image Semantic Segmentation Currently this repo contains the only working open-source implementation of Auto-Deeplab which, by the way out-

AI Necromancer 285 Jan 14, 2022
(ImageNet pretrained models) The official pytorch implemention of the TPAMI paper "Res2Net: A New Multi-scale Backbone Architecture"

Res2Net The official pytorch implemention of the paper "Res2Net: A New Multi-scale Backbone Architecture" Our paper is accepted by IEEE Transactions o

Res2Net Applications 838 Jan 24, 2022
Compute descriptors for 3D point cloud registration using a multi scale sparse voxel architecture

MS-SVConv : 3D Point Cloud Registration with Multi-Scale Architecture and Self-supervised Fine-tuning Compute features for 3D point cloud registration

null 30 Jan 19, 2022
This framework implements the data poisoning method found in the paper Adversarial Examples Make Strong Poisons

Adversarial poison generation and evaluation. This framework implements the data poisoning method found in the paper Adversarial Examples Make Strong

null 19 Jan 9, 2022
Reinforcement learning library(framework) designed for PyTorch, implements DQN, DDPG, A2C, PPO, SAC, MADDPG, A3C, APEX, IMPALA ...

Automatic, Readable, Reusable, Extendable Machin is a reinforcement library designed for pytorch. Build status Platform Status Linux Windows Supported

Iffi 262 Jan 18, 2022
SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems

The SLIDE package contains the source code for reproducing the main experiments in this paper. Dataset The Datasets can be downloaded in Amazon-

Intel Labs 67 Jan 7, 2022
This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of Coordinate Independent Convolutional Networks.

Orientation independent Möbius CNNs This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of

Maurice Weiler 57 Dec 29, 2021