TLaunch: Launch Programs on Multiple Hosts

Overview

TLaunch: Launch Programs on Multiple Hosts

Introduction

Deepmind launchpad is a library that helps writing distributed program in a simple way. But currently it only supports (or has only open-sourced) launching programs on a single host, either multi-threaded or multi-processed. This library provides a way of launching existing launchpad programs on multiple nodes. Only some simple modification to your program is needed.

Installation

First, let's clone this repository. Then cd into the repository, and execute:

pip install -r requirements.txt
pip install .

Usage

1. Launch programs on multiple hosts without communication

from absl import app
import logging
from tlaunch import lp_ssh

class Worker:
  def __init__(self, worker_id):
    self.worker_id = worker_id

  def run(self):
    logging.info('Worker {}:{}'.format(self.worker_id, i))
    lp_ssh.stop()

def make_program():
  program = lp_ssh.Program('worker')
  worker_num = 2
  current_num = 0
  for host in ['host1','host2']:
    for i in range(worker_num):
      ssh_node  = lp_ssh.SSHNode(Worker, current_num).to_host(host)
      current_num += 1
      program.add_node(ssh_node, label=host+'_worker')
  lp_ssh.launch(program, terminal='ssh_tmux_session')
def main(_):
  make_program()

if __name__ == '__main__':
  app.run(main)

In this code, we place Worker on host1 and host2 via to_host() function. With lp_ssh.launch(), Each Worker will start to run on its corresponding hosts. Besides, examples/mnist/run.sh shows an example of how to train MNIST dataset on multiple hosts.

2. Launch programs on multiple hosts with communication

examples/commands/run_cmd.py gives an example of how to check GPU status of remote hosts. The information can be transferred via defining a TransmitNode.

3. Different data-transfer types

[Optional] Kubernetes Support

If you want to use TLaunch with Kubernetes:

  1. go installed on host machine to run kustomize.
  2. A running kubernetes cluster.
  3. Volcano scheduler installed to enable gang scheduling. This will tell you how to install volcano for your kubernetes cluster.

Install and run lp-operator on your kubernetes cluster

cd lp-operator
make deploy

Then lp-operator should be running in namespace lp-operator-system. You can use kubectl get all -n lp-operator-system to check the status of the running operator.

[Optional] Kubernetes Usage

Check here for more information.

Citing TLaunch

If you use TLaunch in your work, please cite us:

@article{tartrl2021tlaunch,
    title={TLaunch: Launch Programs on Multiple Hosts},
    author={Shiyu Huang, Sen Na, Shizhen Xu, Ting Chen, Jun Zhu},
    year={2021},
    howpublished={\url{https://github.com/TARTRL/TLaunch}},
}
Owner
Tsinghua AI Research Team for Reinforcement Learning
Tsinghua AI Research Team for Reinforcement Learning (Creativity, Practicality and Optimist)
Tsinghua AI Research Team for Reinforcement Learning
Exploiting CVE-2021-42278 and CVE-2021-42287 to impersonate DA from standard domain user

About Exploiting CVE-2021-42278 and CVE-2021-42287 to impersonate DA from standard domain user Changed from sam-the-admin. Usage SAM THE ADMIN CVE-202

Evi1cg 500 Jan 06, 2023
This is a simple Port Flooder written in Python 3.

This is a simple Port Flooder written in Python 3. Use this tool to quickly stress test your network devices and measure your router's or server's load.

Júlio Carneiro 4 Feb 20, 2022
Wireguard VPN Server Installer for: on Ubuntu, Debian, Arch, Fedora and CentOS

XGuard (Wireguard Server Installer) This Python script should make the installation of a Wireguard VPN server as easy as possible. Wireguard is a mode

Johann 3 Nov 04, 2022
Passphrase-wordlist - Shameless clone of passphrase wordlist

This repository is NOT official -- the original repository is located on GitLab

Jeff McJunkin 2 Feb 05, 2022
A Radare2 based Python module for Binary Analysis and Reverse Engineering.

Zepu1chr3 A Radare2 based Python module for Binary Analysis and Reverse Engineering. Installation You can simply run this command. pip3 install zepu1c

Mehmet Ali KERİMOĞLU 5 Aug 25, 2022
VMware vCenter earlier v(7.0.2.00100) unauthorized arbitrary file read

vcenter_fileread_exploit VMware vCenter earlier v(7.0.2.00100) unauthorized arbitrary file read Usage python3 vCenter_fileread.py http(s)://ip Referen

Ashish Kunwar 4 Sep 23, 2022
解密哥斯拉webshell管理工具流量

kingkong 解密哥斯拉Godzilla-V2.96 webshell管理工具流量 目前只支持jsp类型的webshell流量解密 Usage 获取攻击者上传到服务器的webshell样本 获取wireshark之类的流量包,一般甲方有科来之类的全流量镜像设备,联系运维人员获取,这里以test.

h4ck for fun 46 Dec 21, 2022
DoSer.py - Simple DoSer in Python

DoSer.py - Simple DoSer in Python What is DoSer? DoSer is basically an HTTP Denial of Service attack that affects threaded servers. It works like this

8 Sep 02, 2022
Confluence OGNL injection

CVE-2021-26084 Confluence OGNL injection CVE-2021-26084 is an Object-Graph Navigation Language (OGNL) injection vulnerability in the Atlassian Conflue

Ashish Kunwar 15 Sep 23, 2022
Facebook Fast Cracking Tool With Python

Pro-Crack Facebook Fast Cracking Tool This is a multi-password‌ cracking tool that can help you hack facebook accounts very quickly Installation On Te

ReD H4CkeR 5 Feb 19, 2022
Uncover the full name of a target on Linkedin.

Revealin Uncover the full name of a target on Linkedin. It's just a little PoC exploiting a design flaw. Useful for OSINT. Screenshot Usage $ git clon

mxrch 129 Dec 21, 2022
This is a keylogger in python for Windows, Mac and Linux!

Python-Keylogger This is a keylogger in python for Windows, Mac and Linux! #How to use it by downloading the zip file? Download the zip file first The

Zeus_Dxvxm 2 Nov 12, 2021
Tools for investigating Log4j CVE-2021-44228

Log4jTools Tools for investigating Log4j CVE-2021-44228 FetchPayload.py (Get java payload from ldap path provided in JNDI lookup). Example command: Re

MalwareTech 91 Dec 29, 2022
JS Deobfuscation is a Python script that deobfuscate JS code and it's time saver for you.

JS Deobfuscation is a Python script that deobfuscate JS code and it's time saver for you. Although it may not work with high degrees of obfuscation, it's a pretty nice tool to help you even if it's j

Quatrecentquatre 3 May 01, 2022
Blinder is a tool that will help you simplify the exploitation of blind SQL injection

Blinder Have you found a blind SQL injection? Great! Now you need to export it, but are you too lazy to sort through the values? Most likely,

10 Dec 06, 2022
web指纹识别工具

前言 一直苦于没有用的顺手的web指纹识别工具,学习前辈s7ckTeam的Glass和broken5的WebAliveScan优秀开源程序开发的轻量型web指纹工具。

EASY 966 Dec 26, 2022
Fuck - Multi Brute Force 🚶‍♂

f-mbf Fuck - Multi Brute Force 🚶‍♂ Install Script $ pkg update && pkg upgrade $ pkg install python2 $ pkg install git $ pip2 install requests $ pip2

Yumasaa 1 Dec 03, 2021
A Superfast SMS & Call bomber for Linux And Termux !

A Superfast SMS & Call bomber for Linux And Termux !

Anubhav Kashyap 15 Feb 21, 2022
pybotnet - A Python Library for building Botnet , Trojan or BackDoor for windows and linux with Telegram control panel

pybotnet A Python Library for building botnet , trojan or backdoor for windows and linux with Telegram control panel Disclaimer: Please note that this

</oNion 181 Jan 02, 2023
Android Malware (Analysis | Scoring) System

An Obfuscation-Neglect Android Malware Scoring System Quark-Engine is also bundled with Kali Linux, BlackArch. A trust-worthy, practical tool that's r

Quark-Engine 1k Jan 04, 2023