Offical code for the paper: "Growing 3D Artefacts and Functional Machines with Neural Cellular Automata" https://arxiv.org/abs/2103.08737

Last update: Jan 01, 2023

Related tags

Deep Learning 3d-artefacts-nca

Overview

Growing 3D Artefacts and Functional Machines with Neural Cellular Automata

Video of more results: https://www.youtube.com/watch?v=-EzztzKoPeo

Requirements

python3.8
This package automatically install test-evocraft-py (https://github.com/shyamsn97/test-evocraft-py), but for further functionality follow installation steps here: https://github.com/real-itu/Evocraft-py

Installation

For general installation

python setup.py install

For ray tune + mlflow

python -m pip install -r ray-requirements.txt
python setup.py install

Usage

Make sure an evocraft-py server is running, either with test-evocraft-py --interactive or by following the steps in https://github.com/real-itu/Evocraft-py.

Configs

Each nca is trained on a specific structure w/ hyperparams and configurations defined in yaml config, which we use with hydra to create the NCA trainer class.

Example Config for generating a "PlainBlacksmith" Minecraft Structure:

trainer:
    name: PlainBlacksmith
    min_steps: 48
    max_steps: 64
    visualize_output: true
    device_id: 0
    use_cuda: true
    num_hidden_channels: 10
    epochs: 20000
    batch_size: 5
    model_config:
        normal_std: 0.1
        update_net_channel_dims: [32, 32]
    optimizer_config:
        lr: 0.002
    dataset_config:
        nbt_path: artefact_nca/data/structs_dataset/nbts/village/plain_village_blacksmith.nbt

defaults:
  - voxel

Generation and Training

See generation notebook for ways to load in a pretrained nca and generate a structure in minecraft

See training notebook for ways to train an nca

CLI training

python artefact_nca/train.py config={path to yaml config} trainer.dataset_config.nbt_path={absolute path to nbt file to use}

Example:

python artefact_nca/train.py config=pretrained_models/PlainBlacksmith/plain_blacksmith.yaml trainer.dataset_config.nbt_path=/home/shyam/Code/3d-artefacts-nca/artefact_nca/data/structs_dataset/nbts/village/plain_village_blacksmith.nbt

Spawning in minecraft

See generation notebook for more details

Example spawning the oak tree

Load in a trainer

from artefact_nca.trainer.voxel_ca_trainer import VoxelCATrainer

nbt_path = {path to repo}/artefact_nca/data/structs_dataset/nbts/village/Extra_dark_oak.nbt
ct = VoxelCATrainer.from_config(
                    "{path to repo}/pretrained_models/Extra_dark_oak/extra_dark_oak.yaml",
                    config={
                        "pretrained_path":"{path to repo}/pretrained_models/Extra_dark_oak/Extra_dark_oak.pt",
                        "dataset_config":{"nbt_path":nbt_path},
                        "use_cuda":False
                    }
                )

Create MinecraftClient to view the growth of the structure in Minecraft at position (-10, 10, 10) (x, y, z)

from artefact_nca.utils.minecraft import MinecraftClient
m = MinecraftClient(ct, (-10, 10, 10))

Spawn 100 iterations and display progress every 5 time steps

m.spawn(100)

Output should look like this:

Structures

see data directory. To view structures and spawn in minecraft see generation notebook. An example of spawning and viewing the Tree:

import matplotlib.pyplot as plt
from artefact_nca.utils.minecraft import MinecraftClient

base_nbt_path = {path to nbts}
nbt_path = "{}/village/Extra_dark_oak.nbt".format(base_nbt_path)

 # spawn at coords (50, 10, 10)
blocks, unique_vals, target, color_dict, unique_val_dict = MinecraftClient.load_entity("Extra_dark_oak", nbt_path=nbt_path, load_coord=(50,10,10))

color_arr = convert_to_color(target, color_dict)

fig = plt.figure()
ax = fig.gca(projection='3d')
ax.voxels(color_arr, facecolors=color_arr, edgecolor='k')

plt.show()

This should spawn and display:

Authors

Shyam Sudhakaran [email protected], https://github.com/shyamsn97

Djordje Grbic [email protected], https://github.com/djole

Siyan Li [email protected], https://github.com/sli613

Adam Katona [email protected], https://github.com/adam-katona

Elias Najarro https://github.com/enajx

Claire Glanois https://github.com/claireaoi

Sebastian Risi [email protected], https://github.com/sebastianrisi

Citation

If you use the code for academic or commecial use, please cite the associated paper:

@inproceedings{Sudhakaran2021,
   title = {Growing 3D Artefacts and Functional Machines with Neural Cellular Automata}, 
   author = {Shyam Sudhakaran and Djordje Grbic and Siyan Li and Adam Katona and Elias Najarro and Claire Glanois and Sebastian Risi},
   booktitle = {2021 Conference on Artificial Life},
   year = {2021},
   url = {https://arxiv.org/abs/2103.08737}
}

Offical code for the paper: "Growing 3D Artefacts and Functional Machines with Neural Cellular Automata" https://arxiv.org/abs/2103.08737

Related tags

Overview

Growing 3D Artefacts and Functional Machines with Neural Cellular Automata

Video of more results: https://www.youtube.com/watch?v=-EzztzKoPeo

Requirements

Installation

For general installation

For ray tune + mlflow

Usage

Configs

Generation and Training

See generation notebook for ways to load in a pretrained nca and generate a structure in minecraft

See training notebook for ways to train an nca

CLI training

Spawning in minecraft

Example spawning the oak tree

Structures

Authors

Citation

Owner

Robotics Evolution and Art Lab

Adversarial Framework for (non-) Parametric Image Stylisation Mosaics

Tool for live presentations using manim

Self-Supervised Pillar Motion Learning for Autonomous Driving (CVPR 2021)

Discord bot-CTFD-Thread-Parser - Discord bot CTFD-Thread-Parser

Open-source codebase for EfficientZero, from "Mastering Atari Games with Limited Data" at NeurIPS 2021.

A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines

NeuPy is a Tensorflow based python library for prototyping and building neural networks

Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

PyTorch implementation of the Value Iteration Networks (VIN) (NIPS '16 best paper)

A Kernel fuzzer focusing on race bugs

KSAI Lite is a deep learning inference framework of kingsoft, based on tensorflow lite

[CoRL 2021] A robotics benchmark for cross-embodiment imitation.

Simple Pixelbot for Diablo 2 Resurrected written in python and opencv.

Tutorials, assignments, and competitions for MIT Deep Learning related courses.

A library for researching neural networks compression and acceleration methods.

Luminaire is a python package that provides ML driven solutions for monitoring time series data.

Instance-wise Feature Importance in Time (FIT)

MoCoPnet - Deformable 3D Convolution for Video Super-Resolution

Code for PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning

Code for ICCV 2021 paper "Distilling Holistic Knowledge with Graph Neural Networks"