2021 Real Robot Challenge Phase2 attemp

Overview

Real_Robot_Challenge_Phase2_AE_attemp

We(team name:thriftysnipe) are the first place winner of Phase1 in 2021 Real Robot Challenge.
Please see this page for more details: https://real-robot-challenge.com/leaderboard
To see more details about out Phase1 works: https://github.com/wq13552463699/Real_Robot_challenge
We were granted the access to Phase 2.

I am sorry, the project is too complex with too much large files, It is too hard to upload them all on Github. I just attached a part of the core code here for you to take a quick lreview. If you think my attempts is approriate, you can go to this Google Drive to download the full project file(all codes, results, trained models, environmental files,.etc):
https://drive.google.com/file/d/14vjCrWU6vzMdXxVSR2FeskMvuQpgqWqM/view?usp=sharing

RRC phase2 task description:

Randomly place 25 dices with the size of 0.01x0.01x0.01m in the environment. Use own controller to drive the three-finger robot to rearrange the dice to a specific pattern. Unfortunately, due to the set task is too difficult, no team could complete the task on the actual robot, so all teams with record are awarded third place in this phase. But I think our attempt has a reference value, if later scholars conduct related research, our method may be useful.

Our considerations:

We consider using a reinforcement learning algorithm as the controller in this phase. However, in this phase, information that can play as observations, such as coordinates and orientation of the dices, cannot be obtained from the environment directly but they are crucial for RL to run.
The alternative observations we can use are the images of the three cameras set in 3 different angles in the environment and their segmentation masks. We picked segmentation masks rather than the raw images since the attendance of noise and redundancy in the raw images were too much. Please see the following segmentation mask example(RGB's 3 channels represent segmentation masks from 3 different angles).

The segmentation masks have the dimension of 270x270x3, if directly passing it to the RL agent, which would lead to computational explosion and hard to converge. Hence, we planned to use some means to extract the principal components that can play as observations from it. In addition, the observation value also includes readable read-robot data(joint angle of the robot arm, end effector position, end effector speed, etc.).

Segmentation mask dimensionality reduction

This is the most important part of this task. We tried different methods, such as GAN, VAE, AE, to extract the principal conponents from the images. The quality of data dimensionality reduction can be easily seem from the discripency of reconstructed and oringinal images or the loss curves. After many trials(adjusting hyperparameters, network structure, depth, etc.), we got different trained VAE, GAN and AE models. We conducted offline tests on the obtained model and compared the results, we were surprised to find that the AE performed the best. When the latent of AE is 384, the quality of the reconstructed image is the best. The result is shown in the figure below.

The loss function also converges to an acceptable range:

Build up observation and trian RL agent.

We use the best AE encoder to deal with the segmentation masks to generate the observation and stitch with the readable data. The structure of the overall obervation is shown as follow:
We fed the above observations to several current cutting-edge model based and model free reinforcement learning algorithms, including DDPG+HER, PPO, SLAC, PlaNet and Dreamer. We thought it would work and enable the agent to learn for somewhat anyway. But it is a pity that after many attempts, the model still didn't have any trend to converge. Due to time limited, our attempts were over here.

Some reasons might lead to fail

  1. We used AE as the observation model. Although the AE's dimensionality reduction capability were the best, the latent space of AE were disordered and didn't make sense to RL agent. The observations passed to the RL must be fixed and orderly. Continuous delivery of unfixed data caused a dimensional disaster. For example, the third number in the observation vector passed at t1 represents 'infos of the 1st dice', and the number on the same position at t2 represents the 'infos of the 3rd dice'. This disorderly change with time makes RL very confused.
  2. The extracted latent space from segmentation mask dominates the observations, making RL ignore the existence of robots. The latent space size is 384, but which for the robot data is 27. The two are far apart, and there is a big data bias.
  3. Robot arm blocked the dices, segmentation masks can only represent a part of the dice. This problem cannot be avoided and can only be solved by more powerful image processing technology. This is also a major challenge in the current Image-based RL industry

Contribution

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. Please make sure to update tests as appropriate.

Owner
Qiang Wang
PhD at UCD. Research interest: Reinforcement Learning; Computer vision&Touch; Representation learning
Qiang Wang
The PicoEMP is a low-cost Electromagnetic Fault Injection (EMFI) tool,

ChipSHOUTER-PicoEMP The PicoEMP is a low-cost Electromagnetic Fault Injection (EMFI) tool, designed specifically for self-study and hobbiest research.

NewAE Technology Inc. 312 Jan 07, 2023
Examples to accompany the

Examples to accompany the "Raspberry Pi Pico Python SDK" book published by Raspberry Pi Trading, which forms part of the technical documentation in support of Raspberry Pi Pico and the MicroPython po

Raspberry Pi 589 Jan 08, 2023
Python application, displaying currently played track from Spotify on OLED display connected via I2C

RaspberryPi Spotify OLED Display This application will display currently played track on SSD1306 OLED display connected to RaspberryPi. Displayed stuf

Wojciech Olech 2 Dec 30, 2021
A python module for interacting with rolimon's, a roblox value site.

rpi - rolimon's python interaction rpi is an open source python-based rolimon's api wrapper. It provides an end-to-end pipeline in which each componen

Acier 11 Nov 08, 2022
Fener ROS2 package version 2

Fener's ROS2 codes that runs on the vehicle. This node contains basic sensing and actuation nodes for vehicle control. Also example applications will be added.

Muhammed Sezer 1 Jan 18, 2022
从零开始打造一个智能家居系统

SweetHome 从零开始打造智能家居系统的尝试,主要的实现有 可以扫码添加设备并控制设备的Android App 可以控制亮灭的灯,并可以设置在Android App连接到指定Wifi后自动亮起 可以控制开关的窗帘,机械结构部分自己设计并3D打印出来 树莓派主控,实现Http请求接口和ZigBe

金榜 5 May 01, 2022
uOTA - OTA updater for MicroPython

Update your device firmware written in MicroPython over the air. Suitable for private and/or larger projects with many files.

Martin Komon 25 Dec 19, 2022
How to configure IOMMU device for nested Proxmox hypervisor (PVE) VM - PCIe Passthrough

Configuring PCIe Passthrough for Nested Virtualization on Proxmox Summary: If you are running bare-metal L0 (level 0) Proxmox (PVE) hypervisor with ne

Travis Johnson 6 Aug 30, 2022
FHEM Connector for FHT Heating devices

home-assistant-fht from: https://github.com/Rsclub22 FHEM Connector for FHT Heating devices (connected via FHEM) Requires FHEM to work You can find FH

5 Dec 01, 2022
Workshop for student hackathons focused on IoT dev

Scenario: The Mutt Matcher (IoT version) According to the World Health Organization there are more than 200 million stray dogs worldwide. The American

Microsoft 15 Aug 10, 2022
ENC28J60 Ethernet chip driver for MicroPython (RP2)

micropy-ENC28J60 ENC28J60 Ethernet chip driver for MicroPython v1.17 (RP2) Rationale ENC28J60 is a popular and cheap module for DIY projects. At the m

11 Nov 16, 2022
Example code and projects for FeatherS2 and FeatherS2 Neo

FeatherS2 & FeatherS2 Neo This repo is a collection of code, firmware, and files

Unexpected Maker 5 Jan 01, 2023
A Simple Python KeyLogger App

✨ Kurulum Uygulamayı bilgisayarınızda kullana bilmek için bazı işlemler yapmanız gerekiyor. Aşağıdaki yönlendirmeleri takip ederek bunu yapabilirsiniz

VorteX 7 Jun 11, 2022
Run this code to blink your ThinkPad LED with a hidden mysterious Morse code! ;)

TMorse Run this code to blink your ThinkPad LED with a hidden mysterious Morse code! ;) Compatible with python3.9+. No third-party library is required

Mahyar 2 Jul 11, 2022
Simple python3 implementation of microKanren with lots of type annotations for clarity

MicroKanren-py This is (yet another) python implementation of microKanren. It's a reasonably 1:1 translation of the code provided in the paper, but ev

Erik Derohanian 3 Dec 10, 2022
Raspberry Pi Power Button - Wake/Power Off/Restart(Double Press)

Control Raspberry pi with physically attached button. Wake, Power Off, and Restart (Double Press) . Python3 script runs as a service with easy installation.

Stas Yakobov 16 Oct 22, 2022
Simples Keylogger para Windows com um autoboot implementado no sistema

MKW Keylogger Keylogger simples para Windos com um autoboot implementado no sistema, o malware irá capturar pressionamentos de tecla e armazená-lo em

3 Jul 03, 2021
OPNsense integration with Home Assistant

hass-opnsense Join OPNsense with home-assistant! hass-opnsense uses the built-in xmlrpc service of OPNsense for all interactions. This project is curr

Travis Glenn Hansen 54 Jan 03, 2023
Add filters (background blur, etc) to your webcam on Linux.

webcam-filters Add filters (background blur, etc) to your webcam on Linux. Video conferencing applications tend to either lack video effects altogethe

Jashandeep Sohi 480 Dec 14, 2022
Simple Python script to decode and verify an European Health Certificate QR-code

A simple Python script to decode and verify an European Health Certificate QR-code.

Mathias Panzenböck 61 Oct 05, 2022