2021 Real Robot Challenge Phase2 attemp

Overview

Real_Robot_Challenge_Phase2_AE_attemp

We(team name:thriftysnipe) are the first place winner of Phase1 in 2021 Real Robot Challenge.
Please see this page for more details: https://real-robot-challenge.com/leaderboard
To see more details about out Phase1 works: https://github.com/wq13552463699/Real_Robot_challenge
We were granted the access to Phase 2.

I am sorry, the project is too complex with too much large files, It is too hard to upload them all on Github. I just attached a part of the core code here for you to take a quick lreview. If you think my attempts is approriate, you can go to this Google Drive to download the full project file(all codes, results, trained models, environmental files,.etc):
https://drive.google.com/file/d/14vjCrWU6vzMdXxVSR2FeskMvuQpgqWqM/view?usp=sharing

RRC phase2 task description:

Randomly place 25 dices with the size of 0.01x0.01x0.01m in the environment. Use own controller to drive the three-finger robot to rearrange the dice to a specific pattern. Unfortunately, due to the set task is too difficult, no team could complete the task on the actual robot, so all teams with record are awarded third place in this phase. But I think our attempt has a reference value, if later scholars conduct related research, our method may be useful.

Our considerations:

We consider using a reinforcement learning algorithm as the controller in this phase. However, in this phase, information that can play as observations, such as coordinates and orientation of the dices, cannot be obtained from the environment directly but they are crucial for RL to run.
The alternative observations we can use are the images of the three cameras set in 3 different angles in the environment and their segmentation masks. We picked segmentation masks rather than the raw images since the attendance of noise and redundancy in the raw images were too much. Please see the following segmentation mask example(RGB's 3 channels represent segmentation masks from 3 different angles).

The segmentation masks have the dimension of 270x270x3, if directly passing it to the RL agent, which would lead to computational explosion and hard to converge. Hence, we planned to use some means to extract the principal components that can play as observations from it. In addition, the observation value also includes readable read-robot data(joint angle of the robot arm, end effector position, end effector speed, etc.).

Segmentation mask dimensionality reduction

This is the most important part of this task. We tried different methods, such as GAN, VAE, AE, to extract the principal conponents from the images. The quality of data dimensionality reduction can be easily seem from the discripency of reconstructed and oringinal images or the loss curves. After many trials(adjusting hyperparameters, network structure, depth, etc.), we got different trained VAE, GAN and AE models. We conducted offline tests on the obtained model and compared the results, we were surprised to find that the AE performed the best. When the latent of AE is 384, the quality of the reconstructed image is the best. The result is shown in the figure below.

The loss function also converges to an acceptable range:

Build up observation and trian RL agent.

We use the best AE encoder to deal with the segmentation masks to generate the observation and stitch with the readable data. The structure of the overall obervation is shown as follow:
We fed the above observations to several current cutting-edge model based and model free reinforcement learning algorithms, including DDPG+HER, PPO, SLAC, PlaNet and Dreamer. We thought it would work and enable the agent to learn for somewhat anyway. But it is a pity that after many attempts, the model still didn't have any trend to converge. Due to time limited, our attempts were over here.

Some reasons might lead to fail

  1. We used AE as the observation model. Although the AE's dimensionality reduction capability were the best, the latent space of AE were disordered and didn't make sense to RL agent. The observations passed to the RL must be fixed and orderly. Continuous delivery of unfixed data caused a dimensional disaster. For example, the third number in the observation vector passed at t1 represents 'infos of the 1st dice', and the number on the same position at t2 represents the 'infos of the 3rd dice'. This disorderly change with time makes RL very confused.
  2. The extracted latent space from segmentation mask dominates the observations, making RL ignore the existence of robots. The latent space size is 384, but which for the robot data is 27. The two are far apart, and there is a big data bias.
  3. Robot arm blocked the dices, segmentation masks can only represent a part of the dice. This problem cannot be avoided and can only be solved by more powerful image processing technology. This is also a major challenge in the current Image-based RL industry

Contribution

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. Please make sure to update tests as appropriate.

Owner
Qiang Wang
PhD at UCD. Research interest: Reinforcement Learning; Computer vision&Touch; Representation learning
Qiang Wang
This allows you to record keyboard and mouse input, and play it back using pynput.

Record and Play with Python! This allows you to record keyboard and mouse input, and play it back (with looping) using pynput. It allows for automatio

George Jensen 45 Jan 02, 2023
Home Assistant custom component to help ev-chargers stay below peak hourly energy levels.

Peaqev ev-charging Peaqev ev-charging is an attempt of charging an ev without breaching a preset monthly max-peak energy level. In order for this inte

Magnus Eldén 35 Dec 24, 2022
This OctoPrint plugin will make the initial connection to 3D Hub a breeze

3D Hub Connector This OctoPrint plugin will make the initial connection to 3D Hub a breeze. In future it will help in setting up a tunnel connection a

3D Hub 2 Aug 03, 2022
Beam designs for infinite Z 3D printers

A 3D printed beam that is as stiff as steel A while ago Naomi Wu 机械妖姬 very kindly sent us one of Creality's infinite-Z belt printers. Lots of people h

RepRap Ltd 105 Oct 22, 2022
E-Ink Magic Calendar that automatically syncs to Google Calendar and runs off a battery powered Raspberry Pi Zero

E-Ink Magic Calendar that automatically syncs to Google Calendar and runs off a battery powered Raspberry Pi Zero

2.8k Dec 30, 2022
A iot Bike sytem based on RaspberryPi, Ardiuino

Cyclic 's Kernel ---- A iot Bike sytem based on RaspberryPi, Ardiuino, etc 0x1 What is This? Cyclic 's Kernel is an independent System With self-produ

Retr0mous 2 Oct 09, 2022
A LiteX project which builds a SoC with DRAM / HDIM output via the GPDI SYZYGY addon.

ButterStick GPDI LiteX demo A LiteX project which builds a SoC with DRAM / HDIM output via the GPDI SYZYGY addon. Getting started Connect GPDI board t

4 Nov 21, 2021
Home assiatant Custom component: Camera Archiver

Camera archiver Archive your ftp camera meadia files on other ftp with files renaming and event creation. Event can be used for send information to el

1 Jan 06, 2022
Example Python code for building RPi-controlled robotic systems

RPi Example Code Example Python code for building RPi-controlled robotic systems These python files have been compiled / developed by the Neurobionics

Elliott Rouse 2 Feb 04, 2022
Get input from OLED Joystick, Runs command, Displays output on OLED Screen (Great for P4wnP1)

p4wnsolo-joyterm Gets text input from OLED Joystick Runs the command you typed Displays output on OLED Screen (Great for P4wnP1 - even better on Raspb

PawnSolo 7 Dec 19, 2022
ESP32 recording button presses, and serving webpage that graphs the numbers over time.

ESP32-IoT-button-graph-test ESP32 recording button presses, and serving webpage via webSockets in order to graph the responses. The objective was to t

f-caro 1 Nov 30, 2021
A DUCO (Duino-Coin) miner for GigaDevice ARM boards.

GD32 Duino-Coin Miner Description Contains the firmware and miner software for mining DUCO (Duino-Coin) on GigaDevice GD32 chips. Supported boards GD3

Maximilian Gerhardt 2 Feb 20, 2022
Cow Feeder is a bot automatically execute trade on cowswap

Cow Feeder is a bot automatically execute trade on cowswap, includes functions: Monitoring Ethereum network gas price and execute trade whe

6 Apr 20, 2022
Testing additional addon devices, and their working scripts

ESP32-addon-devices-potpurri Testing additional addon devices, and their micropython working scripts 📑 List of device addons tested so far Ethernet P

f-caro 0 Nov 26, 2022
[unmaintained] WiFi tools for linux

Note: This project is unmaintained. While I would love to keep up the development on this project, it is difficult for me for several reasons: I don't

Rocky Meza 288 Dec 13, 2022
A modular sequencer based on Pi Pico & EuroPi

PicoSequencer A modular sequencer based on Pi Pico & EuroPi by Zeno Van Moerkerke / Keurslager Kurt For now it is 'only' a trigger sequencer, but I si

5 Oct 27, 2022
ModbusTCP2MQTT - Sungrow & SMA Solar Inverter addon for Home Assistant

ModbusTCP2MQTT Sungrow & SMA Solar Inverter addon for Home Assistant This addon will connect directly to your Inverter using Modbus TCP. Support model

Teny Smart 40 Dec 21, 2022
Custom component for Home Assistant that integrates Candy/Haier Wi-Fi washing machines (also known as Simply-Fi).

Candy Home Assistant component Custom component for Home Assistant that integrates Candy/Haier Wi-Fi washing machines (also known as Simply-Fi). This

Olivér Falvai 61 Dec 29, 2022
Custom component for MPC-HC for home-assistant

mpc_hc The current mpchc integration in homeassistant violates ADR0004, so it will be deleted from core. This is just the existing integration copied

3 Dec 15, 2022
Python module for controlling Broadlink RM2/3 (Pro) remote controls, A1 sensor platforms and SP2/3 smartplugs

Python module for controlling Broadlink RM2/3 (Pro) remote controls, A1 sensor platforms and SP2/3 smartplugs

Matthew Garrett 1.2k Jan 04, 2023