Offical code for the paper: "Growing 3D Artefacts and Functional Machines with Neural Cellular Automata" https://arxiv.org/abs/2103.08737

Last update: Jan 01, 2023

Related tags

Deep Learning 3d-artefacts-nca

Overview

Growing 3D Artefacts and Functional Machines with Neural Cellular Automata

Video of more results: https://www.youtube.com/watch?v=-EzztzKoPeo

Requirements

python3.8
This package automatically install test-evocraft-py (https://github.com/shyamsn97/test-evocraft-py), but for further functionality follow installation steps here: https://github.com/real-itu/Evocraft-py

Installation

For general installation

python setup.py install

For ray tune + mlflow

python -m pip install -r ray-requirements.txt
python setup.py install

Usage

Make sure an evocraft-py server is running, either with test-evocraft-py --interactive or by following the steps in https://github.com/real-itu/Evocraft-py.

Configs

Each nca is trained on a specific structure w/ hyperparams and configurations defined in yaml config, which we use with hydra to create the NCA trainer class.

Example Config for generating a "PlainBlacksmith" Minecraft Structure:

trainer:
    name: PlainBlacksmith
    min_steps: 48
    max_steps: 64
    visualize_output: true
    device_id: 0
    use_cuda: true
    num_hidden_channels: 10
    epochs: 20000
    batch_size: 5
    model_config:
        normal_std: 0.1
        update_net_channel_dims: [32, 32]
    optimizer_config:
        lr: 0.002
    dataset_config:
        nbt_path: artefact_nca/data/structs_dataset/nbts/village/plain_village_blacksmith.nbt

defaults:
  - voxel

Generation and Training

See generation notebook for ways to load in a pretrained nca and generate a structure in minecraft

See training notebook for ways to train an nca

CLI training

python artefact_nca/train.py config={path to yaml config} trainer.dataset_config.nbt_path={absolute path to nbt file to use}

Example:

python artefact_nca/train.py config=pretrained_models/PlainBlacksmith/plain_blacksmith.yaml trainer.dataset_config.nbt_path=/home/shyam/Code/3d-artefacts-nca/artefact_nca/data/structs_dataset/nbts/village/plain_village_blacksmith.nbt

Spawning in minecraft

See generation notebook for more details

Example spawning the oak tree

Load in a trainer

from artefact_nca.trainer.voxel_ca_trainer import VoxelCATrainer

nbt_path = {path to repo}/artefact_nca/data/structs_dataset/nbts/village/Extra_dark_oak.nbt
ct = VoxelCATrainer.from_config(
                    "{path to repo}/pretrained_models/Extra_dark_oak/extra_dark_oak.yaml",
                    config={
                        "pretrained_path":"{path to repo}/pretrained_models/Extra_dark_oak/Extra_dark_oak.pt",
                        "dataset_config":{"nbt_path":nbt_path},
                        "use_cuda":False
                    }
                )

Create MinecraftClient to view the growth of the structure in Minecraft at position (-10, 10, 10) (x, y, z)

from artefact_nca.utils.minecraft import MinecraftClient
m = MinecraftClient(ct, (-10, 10, 10))

Spawn 100 iterations and display progress every 5 time steps

m.spawn(100)

Output should look like this:

Structures

see data directory. To view structures and spawn in minecraft see generation notebook. An example of spawning and viewing the Tree:

import matplotlib.pyplot as plt
from artefact_nca.utils.minecraft import MinecraftClient

base_nbt_path = {path to nbts}
nbt_path = "{}/village/Extra_dark_oak.nbt".format(base_nbt_path)

 # spawn at coords (50, 10, 10)
blocks, unique_vals, target, color_dict, unique_val_dict = MinecraftClient.load_entity("Extra_dark_oak", nbt_path=nbt_path, load_coord=(50,10,10))

color_arr = convert_to_color(target, color_dict)

fig = plt.figure()
ax = fig.gca(projection='3d')
ax.voxels(color_arr, facecolors=color_arr, edgecolor='k')

plt.show()

This should spawn and display:

Authors

Shyam Sudhakaran [email protected], https://github.com/shyamsn97

Djordje Grbic [email protected], https://github.com/djole

Siyan Li [email protected], https://github.com/sli613

Adam Katona [email protected], https://github.com/adam-katona

Elias Najarro https://github.com/enajx

Claire Glanois https://github.com/claireaoi

Sebastian Risi [email protected], https://github.com/sebastianrisi

Citation

If you use the code for academic or commecial use, please cite the associated paper:

@inproceedings{Sudhakaran2021,
   title = {Growing 3D Artefacts and Functional Machines with Neural Cellular Automata}, 
   author = {Shyam Sudhakaran and Djordje Grbic and Siyan Li and Adam Katona and Elias Najarro and Claire Glanois and Sebastian Risi},
   booktitle = {2021 Conference on Artificial Life},
   year = {2021},
   url = {https://arxiv.org/abs/2103.08737}
}

Offical code for the paper: "Growing 3D Artefacts and Functional Machines with Neural Cellular Automata" https://arxiv.org/abs/2103.08737

Related tags

Overview

Growing 3D Artefacts and Functional Machines with Neural Cellular Automata

Video of more results: https://www.youtube.com/watch?v=-EzztzKoPeo

Requirements

Installation

For general installation

For ray tune + mlflow

Usage

Configs

Generation and Training

See generation notebook for ways to load in a pretrained nca and generate a structure in minecraft

See training notebook for ways to train an nca

CLI training

Spawning in minecraft

Example spawning the oak tree

Structures

Authors

Citation

Owner

Robotics Evolution and Art Lab

Cerberus Transformer: Joint Semantic, Affordance and Attribute Parsing

The Pytorch code of "Joint Distribution Matters: Deep Brownian Distance Covariance for Few-Shot Classification", CVPR 2022 (Oral).

Numerai tournament example scripts using NN and optuna

Build a medical knowledge graph based on Unified Language Medical System (UMLS)

Adversarial Reweighting for Partial Domain Adaptation

Tensorflow 2 implementations of the C-SimCLR and C-BYOL self-supervised visual representation methods from "Compressive Visual Representations" (NeurIPS 2021)

Pytorch modules for paralel models with same architecture. Ideal for multi agent-based systems

Capsule endoscopy detection DACON challenge

A Python library for adversarial machine learning focusing on benchmarking adversarial robustness.

Codebase for Attentive Neural Hawkes Process (A-NHP) and Attentive Neural Datalog Through Time (A-NDTT)

Covid19-Forecasting - An interactive website that tracks, models and predicts COVID-19 Cases

Super Pix Adv - Offical implemention of Robust Superpixel-Guided Attentional Adversarial Attack (CVPR2020)

Novel and high-performance medical image classification pipelines are heavily utilizing ensemble learning strategies

Library extending Jupyter notebooks to integrate with Apache TinkerPop and RDF SPARQL.

K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce (EMNLP Founding 2021)

Official pytorch implementation for Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion (CVPR 2022)

Implementation of Perceiver, General Perception with Iterative Attention in TensorFlow

ICS 4u HD project, start before-wards. A curtain shooting game using python.

Code for the paper "JANUS: Parallel Tempered Genetic Algorithm Guided by Deep Neural Networks for Inverse Molecular Design"

Code for 'Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning' (AAAI 2022)