An OpenAI-Gym Package for Training and Testing Reinforcement Learning algorithms with OpenSim Models

Overview

Logo

Authors: Utkarsh A. Mishra and Dr. Dimitar Stanev

Advisors: Dr. Dimitar Stanev and Prof. Auke Ijspeert, Biorobotics Laboratory (BioRob), EPFL

Video Playlist: https://www.youtube.com/playlist?list=PLDvnH871wUkFPOcCKcsTN6ZzzjNZOVlt_

The bioimiitation-gym package is a python package that provides a gym environment for training and testing OpenSim models. The gym environment is based on the OpenAI gym package.

This work is towards a framework aimed towards learning to imitate human gaits. Humans exhibit movements like walking, running, and jumping in the most efficient manner, which served as the source of motivation for this project. Skeletal and Musculoskeletal human models were considered for motions in the sagittal and frontal plane, and results from both were compared exhaustively. While skeletal models are driven with motor actuation, musculoskeletal models perform through muscle-tendon actuation.

Baseline Architecture

Model-free reinforcement learning algorithms were used to optimize inverse dynamics control actions to satisfy the objective of imitating a reference motion along with secondary objectives of minimizing effort in terms of power spent by motors and metabolic energy consumed by the muscles. On the one hand, the control actions for the motor actuated model is the target joint angles converted into joint torques through a Proportional-Differential controller. While on the other hand, the control actions for the muscle-tendon actuated model is the muscle excitations converted implicitly to muscle activations and then to muscle forces which apply moments on joints. Muscle-tendon actuated models were found to have superiority over motor actuation as they are inherently smooth due to muscle activation dynamics and don't need any external regularizers.

Results

All the results and analysis are presented in an illustrative, qualitative, and quantitative manner.

Installation

Please follow the instructions in the installation.md file to install the package.

Environment in the bioimitation-gym package

All environments in the bioimitation-gym package are provided in the biomitation_envs/imitation_envs/envs directory. They are majorly divided into two categories:

  • muscle environments: These are the environments that are used for training the muscle tendon unit actuated model.
  • torque environments: These are the environments that are used for training the torque actuate model.

Further, 2D / planar and 3D / spatial environments are provided for each category. The tasks covered in each of the sub-categories are as follows:

  • Walking
  • Running
  • Jumping
  • Prosthetic Walking with a locked knee joint for the left leg
  • Walking with a typical Cerebel Palsy defect

The following 2D muscle actuated environment names can be used based on the package:

  • MuscleWalkingImitation2D-v0
  • MuscleRunningImitation2D-v0
  • MuscleJumpingImitation2D-v0
  • MuscleLockedKneeImitation2D-v0

The following 3D muscle actuated environment names can be used based on the package:

  • MuscleWalkingImitation3D-v0
  • MuscleRunningImitation3D-v0
  • MuscleJumpingImitation3D-v0
  • MuscleLockedKneeImitation3D-v0
  • MusclePalsyImitation3D-v0

The following 2D torque actuated environment names can be used based on the package:

  • TorqueWalkingImitation2D-v0
  • TorqueRunningImitation2D-v0
  • TorqueJumpingImitation2D-v0
  • TorqueLockedKneeImitation2D-v0

The following 3D torque actuated environment names can be used based on the package:

  • TorqueWalkingImitation3D-v0
  • TorqueRunningImitation3D-v0
  • TorqueJumpingImitation3D-v0
  • TorqueLockedKneeImitation3D-v0

Usage Instructions

The complete bioimitation directory consists of the following sub-directories:

  • imitation_envs: This directory contains the data and environments associated with the package.
  • learning_algorithm: This directory contains the learning algorithm used for several experiments. The code is the modified version of original SAC algorithm and is taken from the open source implementation of ikostrikov/jaxrl.

More information on the subdirectories can be found in their respective README files (if any).

The package is mostly based on the highly scalable and distributed reinforcement learning framework Ray RLLIB. The template scipts to train and test the models are provided in the tests directory.

To run a RLLIB training script, run the following command:

python tests/sample_rllib_training.py  --env_name MuscleWalkingImitation2D-v0

You can change the algorithm configurations in the configs directory. The configs/train_default.py file contains the default configuration for the train script and the configs/test_default.py file contains the default configuration for the test script which is:

python tests/sample_rllib_testing.py

The default environment configuration is provided in the configs/env_default.py file. Feel free to change the default configuration as per your needs. A typical script to test the environment is provided in the biomitation_envs/imitation_envs/envs directory is:

import os
from absl import app, flags
from ml_collections import config_flags
import gym
import bioimitation

FLAGS = flags.FLAGS

flags.DEFINE_string('env_name', 'MuscleWalkingImitation2D-v0', 'Name of the environment.')

config_flags.DEFINE_config_file(
    'config',
    'configs/env_default.py',
    'File path to the environment configuration.',
    lock_config=False)

def main(_):

    example_config = dict(FLAGS.config)

    env = gym.make(FLAGS.env_name, config=example_config)

    env.reset()

    for i in range(1000):
        _, _, done, _ = env.step(env.action_space.sample())
        if done:
            env.reset()

if __name__ == '__main__':
    app.run(main)

Don't forget to import the bioimitation package before running the script.

Citation

If you use this work in your research, please cite the following as:

@misc{
    mishra2021bioimitation,
    title = {BioImitation-Gym: A OpenAI-Gym Package for Training and Testing Reinforcement Learning algorithms with OpenSim Models},
    author = {Utkarsh A. Mishra and Dimitar Stanev and Auke Ijspeert},
    year = {2021},
    url = {https://github.com/UtkarshMishra/bioimitation-gym}
}
@article{mishra2021learning,
  title={Learning Control Policies for Imitating Human Gaits},
  author={Utkarsh A. Mishra},
  journal={arXiv preprint arXiv:2106.15273},
  year={2021}
}

References

[1] OsimRL project: https://osim-rl.kidzinski.com/

[2] OpenSim: https://github.com/opensim-org/opensim-core and https://opensim.stanford.edu/

[3] OpenAI Gym: https://gym.openai.com/

[4] Ray RLLIB: https://ray.readthedocs.io/en/latest/

[6] ikostrikov/jaxrl: https://github.com/ikostrikov/jaxrl

Owner
Utkarsh Mishra
Graduate from @iitroorkee (Batch of 2021), programming enthusiast. Reinforcement Learning, Robotics & Self-Driving interests me.
Utkarsh Mishra
Website which uses Deep Learning to generate horror stories.

Creepypasta - Text Generator Website which uses Deep Learning to generate horror stories. View Demo · View Website Repo · Report Bug · Request Feature

Dhairya Sharma 5 Oct 14, 2022
People movement type classifier with YOLOv4 detection and SORT tracking.

Movement classification The goal of this project would be movement classification of people, in other words, walking (normal and fast) and running. Yo

4 Sep 21, 2021
A JAX-based research framework for writing differentiable numerical simulators with arbitrary discretizations

jaxdf - JAX-based Discretization Framework Overview | Example | Installation | Documentation ⚠️ This library is still in development. Breaking changes

UCL Biomedical Ultrasound Group 65 Dec 23, 2022
Prototypical Networks for Few shot Learning in PyTorch

Prototypical Networks for Few shot Learning in PyTorch Simple alternative Implementation of Prototypical Networks for Few Shot Learning (paper, code)

Orobix 835 Jan 08, 2023
A general-purpose encoder-decoder framework for Tensorflow

READ THE DOCUMENTATION CONTRIBUTING A general-purpose encoder-decoder framework for Tensorflow that can be used for Machine Translation, Text Summariz

Google 5.5k Jan 07, 2023
PyTorch IPFS Dataset

PyTorch IPFS Dataset IPFSDataset(Dataset) See the jupyter notepad to see how it works and how it interacts with a standard pytorch DataLoader You need

Jake Kalstad 2 Apr 13, 2022
Sequence modeling benchmarks and temporal convolutional networks

Sequence Modeling Benchmarks and Temporal Convolutional Networks (TCN) This repository contains the experiments done in the work An Empirical Evaluati

CMU Locus Lab 3.5k Jan 01, 2023
Betafold - AlphaFold with tunings

BetaFold We (hegelab.org) craeted this standalone AlphaFold (AlphaFold-Multimer,

2 Aug 11, 2022
Pytorch implementation of FlowNet by Dosovitskiy et al.

FlowNetPytorch Pytorch implementation of FlowNet by Dosovitskiy et al. This repository is a torch implementation of FlowNet, by Alexey Dosovitskiy et

Clément Pinard 762 Jan 02, 2023
Pytorch implementation of XRD spectral identification from COD database

XRDidentifier Pytorch implementation of XRD spectral identification from COD database. Details will be explained in the paper to be submitted to NeurI

Masaki Adachi 4 Jan 07, 2023
RLBot Python bindings for the Rust crate rl_ball_sym

RLBot Python bindings for rl_ball_sym 0.6 Prerequisites: Rust & Cargo Build Tools for Visual Studio RLBot - Verify that the file %localappdata%\RLBotG

Eric Veilleux 2 Nov 25, 2022
Using knowledge-informed machine learning on the PRONOSTIA (FEMTO) and IMS bearing data sets. Predict remaining-useful-life (RUL).

Knowledge Informed Machine Learning using a Weibull-based Loss Function Exploring the concept of knowledge-informed machine learning with the use of a

Tim 43 Dec 14, 2022
PyTorch image models, scripts, pretrained weights -- ResNet, ResNeXT, EfficientNet, EfficientNetV2, NFNet, Vision Transformer, MixNet, MobileNet-V3/V2, RegNet, DPN, CSPNet, and more

PyTorch Image Models Sponsors What's New Introduction Models Features Results Getting Started (Documentation) Train, Validation, Inference Scripts Awe

Ross Wightman 22.9k Jan 09, 2023
A light-weight image labelling tool for Python designed for creating segmentation data sets.

An image labelling tool for creating segmentation data sets, for Django and Flask.

117 Nov 21, 2022
[SIGGRAPH'22] StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets

[Project] [PDF] This repository contains code for our SIGGRAPH'22 paper "StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets" by Axel Sauer, Katja

742 Jan 04, 2023
Code for ACL2021 long paper: Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases

LANKA This is the source code for paper: Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases (ACL 2021, long paper) Referen

Boxi Cao 30 Oct 24, 2022
Cowsay - A rewrite of cowsay in python

Python Cowsay A rewrite of cowsay in python. Allows for parsing of existing .cow

James Ansley 3 Jun 27, 2022
Ganilla - Official Pytorch implementation of GANILLA

GANILLA We provide PyTorch implementation for: GANILLA: Generative Adversarial Networks for Image to Illustration Translation. Paper Arxiv Updates (Fe

Samet Hi 462 Dec 05, 2022
Convert ONNX model graph to Keras model format.

Convert ONNX model graph to Keras model format.

Grigory Malivenko 175 Dec 28, 2022
The fastai book, published as Jupyter Notebooks

English / Spanish / Korean / Chinese / Bengali / Indonesian The fastai book These notebooks cover an introduction to deep learning, fastai, and PyTorc

fast.ai 17k Jan 07, 2023