Official code for our EMNLP2021 Outstanding Paper MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks

Overview

MindCraft

Authors: Cristian-Paul Bara*, Sky CH-Wang*, Joyce Chai

This is the official code repository for the paper (arXiv link):

Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai. 2021. MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).

@inproceedings{bara2021mindcraft,
  title={MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks},
  author={Bara, Cristian-Paul and CH-Wang, Sky and Chai, Joyce},
  booktitle={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year={2021}
}

Installation Instructions

This README assumes that the user is about to set up the MindCraft task on a to-be-newly-created Ubuntu-based AWS EC2 Server. If not, some commands may be invalid (e.g. apt-get vs. apt). This has been tested on both Ubuntu versions 18 and 20.

Server Setup & Port Forwarding (for reference, AWS port forwarding guidelines are adapted from here):

  1. Launch an EC2 Instance, choosing an Ubuntu-based x86 Amazon Machine Image such as Ubuntu Server 20.04 LTS (HVM), SSD Volume Type.
  2. Choose an instance type. Minimum resource requirements are relatively high, seeing as we are going to run web & game servers concurrently on the same machine. Testing has indicated that instances like t2.xlarge with at least 16 gigs of RAM work fine.
  3. Leave the options specified in 3. Configure Instance, 4. Add Storage, and 5. Add Tags as default.
  4. On 6. Configure Security Group, Add Rule of type Custom TCP Rule, with options Port Range: 22565 and Source: Anywhere. This ensures that players can access the game server with just the IP address.
  5. Add another rule, this time with option Port Range: 8080, keeping all other options the same as above. This ensures that players can access the web server with just the IP address.
  6. Review and launch, specifying your EC2 KeyPair for remote admin access.

Required Depenencies (install these sshed into the EC2 machine):

  1. Java (for reference, Java-Spigot guidelines are adapted from here)
    1. First, update your local package lists with sudo apt update; this updates the URL locations for all the required dependencies you're going to install later. On a newly-created EC2 server, these lists at startup are going to be horribly out of date.
    2. Next, install Java Runtime Environment 8 with sudo apt install openjdk-8-jre-headless.
  2. MySQL (for reference, MySQL guidelines are adapted from here)
    1. To install MySQL server, run sudo apt install mysql-server.
    2. Run setup script sudo mysql_secure_installation to configure your newly installed server with authentication permissions. Go through the setup instructions, setting up a password for the root-user, and confirming other security settings.
    3. To confirm that your newly-installed MySQL server is running, run systemctl status mysql.service to manually check.
    4. Now, the password just created in step 2.2 doesn't actually enable remote connections as root (which we want for our game server & web server); here, execute:
      1. sudo mysql
      2. In the MYSQL commandline, run ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password'; (replacing password with a password of your choosing).
      3. And finally FLUSH PRIVILEGES; to write changes to disk.
      4. To exit the MySQL commandline, exit.
    5. To create your MySQL database (where all MindCraft user activity is recorded), enter the MySQL commandline this time with mysql -u root -p, and then entering the password you just created.
    6. In the commandline interface, CREATE DATABASE minecraft; to create, and then SHOW DATABASES; to manually confirm the creation of the minecraft database. Before you exit, run SHOW GLOBAL VARIABLES LIKE 'PORT'; to confirm the port that the MySQL server is running on (by default, it should be 3306).
    7. Finally, set up game server plugin authentication details! Before you proceed: if this is your first time setting this up, the files listed below may not have been created yet. To create these files, go ahead and start the server once with bash startServer.sh. You're going to see a ton of errors appear, but hold out for now! When the server is done spinning up, enter stop to stop the server, and then proceed onto these sub-steps.
      1. Open spigot/plugins/situatedDialogue/config.yml, and change:
        1. mysql_password to what you defined in 2.4.2.
        2. mysql_port to what you just confirmed in 2.4.6 (default is 3306).
      2. Open spigot/plugins/LogBlock/config.yml, and change the same lines as above, this time under section mysql. Remember to also change the MySQL user here to root (default initialization has it as something else)!
      3. Open mean/server.js and do the same for MYSQL authentication credentials in the first few lines.
      4. Open spigot/plugins/AdvancedReplay and do the same for mysql.yml.
  3. NPM (for reference, Node.JS guidelines are adapted from here)
    1. Package lists should already be up-to-date, so running sudo apt install nodejs and sudo apt install npm will suffice; this will install the latest versions.
  4. MongoDB (for reference, Mongo guidelines are adapted from here) These steps are necessary mainly because I have used a cookie cutter MEAN stack setup, even though the underlying web server doesn't really use Mongo at all.
    1. Run wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -
    2. Run echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list
    3. Run sudo apt update and sudo apt install -y mongodb-org to install Mongo.
    4. Run sudo systemctl start mongod to start the service and sudo systemctl status mongod to check service status.
    5. Finally, run sudo systemctl enable mongod to enable Mongo to start on every server reboot.

File Structure

The MindCraft environment is split into three main modules: (1) initialization, by which task variables (e.g. number of games, complexity of games, and more) are set to customized or default values; (2) the game server, a Bukkit/Spigot-made Minecraft multiplayer server that hosts the game itself and records all in-game interactions; and (3) the web server, a MEAN-stack (mainly, Node.JS) web server that records webpage user interactions, where currently recording of player mental states takes place.

All recording of user data -both game server and through the web server- are consolidated into a local MySQL database, the authentication details of which are to be specifiied in initialization files (or left as default).

Execution

Both the game server and web server are designed to be run concurrently in parallel. To achieve this, set up two tmuxes and run the following sections in separate muxes.

Game Server: Run bash startServer.sh. Edit the parameters passed to the plan generator python script if desired before running the server. If it's your first time running the server, you may have to go the newly-generated spigot/eula.txt and change eula=false to eula=true.
Web Server: Navigate to the mean folder and run npm start. Do this after the game server has been successfully spun up.

Replay

For replaying a specific previous game, make sure that the correct plan file of the logged game, indicated in the format logs/logs.XXXXXXX.plan.json (where XXX is the UNIX time stamp of when the game was played), is copied to the following folder and named as plan_generator/plan.json. After this has been done, start the server with bash spigot/start.command instead of the usual bash file.

Owner
Situated Language and Embodied Dialogue (SLED) Research Group
SLED Research Group @ University of Michigan
Situated Language and Embodied Dialogue (SLED) Research Group
Remote sensing change detection tool based on PaddlePaddle

PdRSCD PdRSCD(PaddlePaddle Remote Sensing Change Detection)是一个基于飞桨PaddlePaddle的遥感变化检测的项目,pypi包名为ppcd。目前0.2版本,最新支持图像列表输入的训练和预测,如多期影像、多源影像甚至多期多源影像。可以快速完

38 Aug 31, 2022
Reinforcement Learning via Supervised Learning

Reinforcement Learning via Supervised Learning Installation Run pip install -e . in an environment with Python = 3.7.0, 3.9. The code depends on MuJ

Scott Emmons 49 Nov 28, 2022
Code for DeepXML: A Deep Extreme Multi-Label Learning Framework Applied to Short Text Documents

DeepXML Code for DeepXML: A Deep Extreme Multi-Label Learning Framework Applied to Short Text Documents Architectures and algorithms DeepXML supports

Extreme Classification 49 Nov 06, 2022
Official implementation of the ICCV 2021 paper "Conditional DETR for Fast Training Convergence".

The DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergen

281 Dec 30, 2022
PyTorch implementation of Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction (ICCV 2021).

Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction Introduction This is official PyTorch implementation of Towards Accurate Alignment

TANG Xiao 96 Dec 27, 2022
Jingju baseline - A baseline model of our project of Beijing opera script generation

Jingju Baseline It is a baseline of our project about Beijing opera script gener

midon 1 Jan 14, 2022
Use CLIP to represent video for Retrieval Task

A Straightforward Framework For Video Retrieval Using CLIP This repository contains the basic code for feature extraction and replication of results.

Jesus Andres Portillo Quintero 54 Dec 22, 2022
Codes for AAAI22 paper "Learning to Solve Travelling Salesman Problem with Hardness-Adaptive Curriculum"

Paper For more details, please see our paper Learning to Solve Travelling Salesman Problem with Hardness-Adaptive Curriculum which has been accepted a

14 Sep 30, 2022
CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching(CVPR2021)

CFNet(CVPR 2021) This is the implementation of the paper CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching, CVPR 2021, Zhelun Shen, Yuch

106 Dec 28, 2022
Official code for "Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer. ICCV2021".

Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer. ICCV2021. Introduction We proposed a novel model training paradi

Lucas 103 Dec 14, 2022
Geometric Deep Learning Extension Library for PyTorch

Documentation | Paper | Colab Notebooks | External Resources | OGB Examples PyTorch Geometric (PyG) is a geometric deep learning extension library for

Matthias Fey 16.5k Jan 08, 2023
A Multi-modal Perception Tracker (MPT) for speaker tracking using both audio and visual modalities

MPT A Multi-modal Perception Tracker (MPT) for speaker tracking using both audio and visual modalities. Implementation for our AAAI 2022 paper: Multi-

yidiLi 4 May 08, 2022
Lightweight stereo matching network based on MobileNetV1 and MobileNetV2

MobileStereoNet: Towards Lightweight Deep Networks for Stereo Matching

Cognitive Systems Research Group 139 Nov 30, 2022
Test-Time Personalization with a Transformer for Human Pose Estimation, NeurIPS 2021

Transforming Self-Supervision in Test Time for Personalizing Human Pose Estimation This is an official implementation of the NeurIPS 2021 paper: Trans

41 Nov 28, 2022
Source code for paper "Deep Diffusion Models for Robust Channel Estimation", TBA.

diffusion-channels Source code for paper "Deep Diffusion Models for Robust Channel Estimation". Generic flow: Use 'matlab/main.mat' to generate traini

The University of Texas Computational Sensing and Imaging Lab 15 Dec 22, 2022
Pytorch implementation of 'Fingerprint Presentation Attack Detector Using Global-Local Model'

RTK-PAD This is an official pytorch implementation of 'Fingerprint Presentation Attack Detector Using Global-Local Model', which is accepted by IEEE T

6 Aug 01, 2022
Efficient and Scalable Physics-Informed Deep Learning and Scientific Machine Learning on top of Tensorflow for multi-worker distributed computing

Notice: Support for Python 3.6 will be dropped in v.0.2.1, please plan accordingly! Efficient and Scalable Physics-Informed Deep Learning Collocation-

tensordiffeq 74 Dec 09, 2022
PyTorch implementation of MulMON

MulMON This repository contains a PyTorch implementation of the paper: Learning Object-Centric Representations of Multi-object Scenes from Multiple Vi

NanboLi 16 Nov 03, 2022
CompilerGym is a library of easy to use and performant reinforcement learning environments for compiler tasks

CompilerGym is a library of easy to use and performant reinforcement learning environments for compiler tasks

Facebook Research 721 Jan 03, 2023
A very simple baseline to estimate 2D & 3D SMPL-compatible keypoints from a single color image.

Minimal Body A very simple baseline to estimate 2D & 3D SMPL-compatible keypoints from a single color image. The model file is only 51.2 MB and runs a

Yuxiao Zhou 49 Dec 05, 2022