CARL provides highly configurable contextual extensions to several well-known RL environments.

Last update: Dec 28, 2022

Related tags

Overview

The CARL Benchmark Library

CARL (context adaptive RL) provides highly configurable contextual extensions to several well-known RL environments. It's designed to test your agent's generalization capabilities in all scenarios where intra-task generalization is important.

Benchmarks include:

OpenAI gym classic control suite extended with several physics context features like gravity or friction
OpenAI gym Box2D BipedalWalker, LunarLander and CarRacing, each with their own modification possibilities like new vehicles to race
All Brax locomotion environments with exposed internal features like joint strength or torso mass
Super Mario (TOAD-GAN), a procedurally generated jump'n'run game with control over level similarity
RNADesign, an environment for RNA design given structure constraints with structures from different datasets to choose from

Installation

We recommend you use a virtual environment (e.g. Anaconda) to install CARL and its dependencies. We recommend and test with python 3.9 under Linux.

First, clone our repository and install the basic requirements:

git clone https://github.com/automl/CARL.git --recursive
cd CARL
pip install .

This will only install the basic classic control environments, which should run on most operating systems. For the full set of environments, use the install options:

pip install -e .[box2d, brax, rna, mario]

These may not be compatible with Windows systems. Box2D environment may need to be installed via conda on MacOS systems:

conda install -c conda-forge gym-box2d

In general, we test on Linux systems, but aim to keep the benchmark compatible with MacOS as much as possible. Mario at this point, however, will not run on any operation system besides Linux

To install the additional requirements for ToadGAN:

javac src/envs/mario/Mario-AI-Framework/**/*.java

If you want to use the RNA design environment:

cd src/envs/rna/learna
make requirements
make data

In case you want to run our experiments or use our training files, also install the experiment dependencies:

pip install -e .[experiments]

Train an Agent

To get started with CARL, you can use our 'train.py' script. It will train a PPO agent on the environment of your choice with custom context variations that are sampled from a standard deviation.

To use MetaCartPole with variations in gravity and friction by 20% compared to the default, run:

python train.py 
--env CARLCartPoleEnv 
--context_args gravity friction
--default_sample_std_percentage 0.2
--outdir <result_location>

You can use the plotting scripts in src/eval to view the results.

CARL's Contextual Extension

CARL contextually extends the environment by making the context visible and configurable. During training we therefore can encounter different contexts and train for generalization. We exemplarily show how Brax' Fetch is extended and embedded by CARL. Different instiations can be achieved by setting the context features to different values.

Cite Us

@misc{CARL,
  author    = {C. Benjamins and 
               T. Eimer and 
               F. Schubert and 
               A. Biedenkapp and 
               B. Rosenhahn and 
               F. Hutter and 
               M. Lindauer},
  title     = {CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning},
  howpublished = {https://github.com/automl/CARL},
  year      = {2021},
  month     = aug,
}

References

OpenAI gym, Brockman et al., 2016. arXiv preprint arXiv:1606.01540

Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation, Freeman et al., NeurIPS 2021 (Dataset & Benchmarking Track)

TOAD-GAN: Coherent Style Level Generation from a Single Example, Awiszus et al., AIIDE 2020

Learning to Design RNA, Runge et al., ICRL 2019

License

CARL falls under the Apache License 2.0 (see file 'LICENSE') as is permitted by all work that we use. This includes CARLMario, which is not based on the Nintendo Game, but on TOAD-GAN and TOAD-GUI running under an MIT license. They in turn make use of the Mario AI framework (https://github.com/amidos2006/Mario-AI-Framework). This is not the original game but a replica, explicitly built for research purposes and includes a copyright notice (https://github.com/amidos2006/Mario-AI-Framework#copyrights ).

Comments

Rna fixup

RNA is now better documented and more easily runnable. There's also an option to subsample the datasets instead of always using all instances per context.

The thing that's missing right now are more context options like filtering by solvers or GC-content, but those aren't easily extractable from our data right now, so that's a separate work package all together.

opened by TheEimer 6
Gym 0.22.0
update required minimum gym version number

added pygame as a requirement because it is not picked up by the gym requirements

getting rid of CustomBipedalWalkerEnv because the functionality of changing the gravity is covered by CARLEnv (same for CustomLunarLanderEnv)

add high game over penalty for LunarLander by a wrapper
opened by benjamc 6
Instance selection

Instance selection now is a class. Default is still roundrobin selection. An instance is only selected when env.reset() (or to be more specific, _progress_instance() is called.

opened by benjamc 4
Added Encoders

Context encoders have been added as a folder and an experiment for running the encoder added in the experiments folders. Since the working directory is the experiment one, I had to add an absolute path for the saved weights. This might need to be changed in the config file

opened by amsks 4
Update References with correct conference

Thanks for the pointer to the survey, but it hasn't been published anywhere, so that detail is incorrect (I wouldn't want to claim that it's published somewhere when it isn't).

opened by RobertKirk 3
Performance Deviations in Brax

Comparing HalfCheetah in Brax (via gym.make and then wrapped as here: https://github.com/google/brax/blob/main/notebooks/training_torch.ipynb) vs in CARL makes a big difference in return even when the context is kept static. Do we do any unexpected reward normalization? Does the way we reset the env make a difference compared to theirs (as we actually update the simluation)?
bug

opened by TheEimer 2
Integrate DM Control
[ ] (convert test file to jupyter notebook. I would like to keep that)

[ ] check tests / write more to increase coverage

[x] update README.md

[x] update documentation

[x] add dm_control to requirements

[x] support dict observation space

documentation tests
opened by benjamc 2
Fix gym version

Gym released a new version where the signature of the step function has changed. This affects our code and requires a separate PR. For now, fix the gym version.

opened by benjamc 1
Initial statedistrs #48
#48 Make initial state distribution configurable. So far, only uniform distributions are used and the bounds can be adjusted.

Classic control:

[x] Acrobot

[x] Pendulum

[x] MountainCar (normal distribution instead of uniform)

[x] MountainCarContinuous (uniform distribution)

[x] CartPole

Box2d

[x] LunarLander

[ ] (maybe/later) Make distributions fully configurable by passing the distribution class and its parameters.

[x] Update documentation: Contexts are automatically filled with the default context if underspecified.
opened by benjamc 1
Integrate dmcontrol

Add support for dm control environments. Integrated walker, quadruped and fish.

In dmc environments there is an additional setting for the context, namely the context mask, which can reduce the amount of context features.

opened by sebidoe 1
use appropriate library for building states

So far, when we do not hide the context, we concatenate the context to the state. For jax based environments (brax) this means that the state is converted from a jax to a numpy array. Now, the state builder checks which library to use and keeps jax states as jax arrays and numpy states as numpy arrays.

Noticed in #42.

opened by benjamc 1
AttributeError: 'System' object has no attribute 'body_idx' in brax

when running test/test_all_envs.py, there is AttributeError: 'System' object has no attribute 'body_idx' in carl_fetch and carl_humanoid environments.

opened by andy-james0310 3

Releases(v0.2.0)

v0.2.0(Jul 12, 2022)
Integrate dm control environments (#55)

Add context masks to only append those to the state (#54)

Extend classic control environments to parametrize initial state distributions (#52)

Remove RNA environment for maintenance (#61)

Fixed pre-commit (mypy, black, flake8, isort) (#62)

Source code(tar.gz)
Source code(zip)

Owner

AutoML-Freiburg-Hannover

GitHub Repository

FinRL-Meta: A Universe for Data-Driven Financial Reinforcement Learning. 🔥

FinRL-Meta: A Universe of Market Environments. FinRL-Meta is a universe of market environments for data-driven financial reinforcement learning. Users

543 Jan 08, 2023

CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes Implementation of CoSMA: Convolutional Semi-Regular Mesh Autoencoder arXiv p

10 Oct 11, 2022

Rank 3 : Source code for OPPO 6G Data Generation Challenge

OPPO 6G Data Generation with an E2E Framework Homepage of OPPO 6G Data Generation Challenge Datasets H1_32T4R.mat H2_32T4R.mat Please put the original

97 Jan 07, 2023

This code is part of the reproducibility package for the SANER 2022 paper "Generating Clarifying Questions for Query Refinement in Source Code Search".

Clarifying Questions for Query Refinement in Source Code Search This code is part of the reproducibility package for the SANER 2022 paper "Generating

0 Dec 04, 2021

Code for the paper Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations (AKBC 2021).

Relation Prediction as an Auxiliary Training Objective for Knowledge Base Completion This repo provides the code for the paper Relation Prediction as

85 Jan 02, 2023

A New Open-Source Off-road Environment for Benchmark Generalization of Autonomous Driving

A New Open-Source Off-road Environment for Benchmark Generalization of Autonomous Driving Isaac Han, Dong-Hyeok Park, and Kyung-Joong Kim IEEE Access

13 Dec 27, 2022

Detecting drunk people through thermal images using Deep Learning (CNN)

Drunk Detection CNN Detecting drunk people through thermal images using Deep Learning (CNN) Dataset We used thermal images provided by Electronics Lab

3 Oct 27, 2022

Simple tutorials using Google's TensorFlow Framework

TensorFlow-Tutorials Introduction to deep learning based on Google's TensorFlow framework. These tutorials are direct ports of Newmu's Theano Tutorial

6k Jan 06, 2023

ISNAS-DIP: Image Specific Neural Architecture Search for Deep Image Prior [CVPR 2022]

ISNAS-DIP: Image-Specific Neural Architecture Search for Deep Image Prior (CVPR 2022) Metin Ersin Arican*, Ozgur Kara*, Gustav Bredell, Ender Konukogl

24 Dec 18, 2022

Apply our monocular depth boosting to your own network!

MergeNet - Boost Your Own Depth Boost custom or edited monocular depth maps using MergeNet Input Original result After manual editing of base You can

142 Dec 17, 2022

Class-Attentive Diffusion Network for Semi-Supervised Classification [AAAI'21] (official implementation)

Class-Attentive Diffusion Network for Semi-Supervised Classification Official Implementation of AAAI 2021 paper Class-Attentive Diffusion Network for

7 Sep 20, 2022

General Multi-label Image Classification with Transformers

General Multi-label Image Classification with Transformers Jack Lanchantin, Tianlu Wang, Vicente Ordóñez Román, Yanjun Qi Conference on Computer Visio

154 Dec 21, 2022

Awesome Transformers in Medical Imaging

This repo supplements our Survey on Transformers in Medical Imaging Fahad Shamshad, Salman Khan, Syed Waqas Zamir, Muhammad Haris Khan, Munawar Hayat,

666 Jan 06, 2023

Time Series Forecasting with Temporal Fusion Transformer in Pytorch

Forecasting with the Temporal Fusion Transformer Multi-horizon forecasting often contains a complex mix of inputs – including static (i.e. time-invari

6 Jan 24, 2022

This is the official implementation for "Do Transformers Really Perform Bad for Graph Representation?".

Graphormer By Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng*, Guolin Ke, Di He*, Yanming Shen and Tie-Yan Liu. This repo is the official impl

1.3k Dec 29, 2022

This repository contains the code for our paper VDA (public in EMNLP2021 main conference)

Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models This repository contains the code for our paper VDA (publ

13 Aug 06, 2022

Official repository for Jia, Raghunathan, Göksel, and Liang, "Certified Robustness to Adversarial Word Substitutions" (EMNLP 2019)

Certified Robustness to Adversarial Word Substitutions This is the official GitHub repository for the following paper: Certified Robustness to Adversa

38 Oct 16, 2022