TLaunch: Launch Programs on Multiple Hosts

Overview

TLaunch: Launch Programs on Multiple Hosts

Introduction

Deepmind launchpad is a library that helps writing distributed program in a simple way. But currently it only supports (or has only open-sourced) launching programs on a single host, either multi-threaded or multi-processed. This library provides a way of launching existing launchpad programs on multiple nodes. Only some simple modification to your program is needed.

Installation

First, let's clone this repository. Then cd into the repository, and execute:

pip install -r requirements.txt
pip install .

Usage

1. Launch programs on multiple hosts without communication

from absl import app
import logging
from tlaunch import lp_ssh

class Worker:
  def __init__(self, worker_id):
    self.worker_id = worker_id

  def run(self):
    logging.info('Worker {}:{}'.format(self.worker_id, i))
    lp_ssh.stop()

def make_program():
  program = lp_ssh.Program('worker')
  worker_num = 2
  current_num = 0
  for host in ['host1','host2']:
    for i in range(worker_num):
      ssh_node  = lp_ssh.SSHNode(Worker, current_num).to_host(host)
      current_num += 1
      program.add_node(ssh_node, label=host+'_worker')
  lp_ssh.launch(program, terminal='ssh_tmux_session')
def main(_):
  make_program()

if __name__ == '__main__':
  app.run(main)

In this code, we place Worker on host1 and host2 via to_host() function. With lp_ssh.launch(), Each Worker will start to run on its corresponding hosts. Besides, examples/mnist/run.sh shows an example of how to train MNIST dataset on multiple hosts.

2. Launch programs on multiple hosts with communication

examples/commands/run_cmd.py gives an example of how to check GPU status of remote hosts. The information can be transferred via defining a TransmitNode.

3. Different data-transfer types

[Optional] Kubernetes Support

If you want to use TLaunch with Kubernetes:

  1. go installed on host machine to run kustomize.
  2. A running kubernetes cluster.
  3. Volcano scheduler installed to enable gang scheduling. This will tell you how to install volcano for your kubernetes cluster.

Install and run lp-operator on your kubernetes cluster

cd lp-operator
make deploy

Then lp-operator should be running in namespace lp-operator-system. You can use kubectl get all -n lp-operator-system to check the status of the running operator.

[Optional] Kubernetes Usage

Check here for more information.

Citing TLaunch

If you use TLaunch in your work, please cite us:

@article{tartrl2021tlaunch,
    title={TLaunch: Launch Programs on Multiple Hosts},
    author={Shiyu Huang, Sen Na, Shizhen Xu, Ting Chen, Jun Zhu},
    year={2021},
    howpublished={\url{https://github.com/TARTRL/TLaunch}},
}
Owner
Tsinghua AI Research Team for Reinforcement Learning
Tsinghua AI Research Team for Reinforcement Learning (Creativity, Practicality and Optimist)
Tsinghua AI Research Team for Reinforcement Learning
MD5-CRACKER - A gmail brute force app created with python3

MD5-CRACKER So this is my first app i created with python3 . if you guys downloa

2 Nov 10, 2022
Reusable Lightweight Pythonic Dependency Injection Library

Vacuna Inject everything! Vacuna is a little library to provide dependency management for your python code. Install pip install vacuna Usage import va

Fernando Martínez González 16 Sep 15, 2021
Encrypted Python Password Manager

PyPassKeep Encrypted Python Password Manager About PyPassKeep (PPK for short) is an encrypted python password manager used to secure your passwords fr

KrisIsHere 1 Nov 17, 2021
一款辅助探测Orderby注入漏洞的BurpSuite插件,Python3编写,适用于上xray等扫描器被ban的场景

OrderbyHunter 一款辅助探测Orderby注入漏洞的BurpSuite插件,Python3编写,适用于上xray等扫描器被ban的场景 1. 支持Get/Post型请求参数的探测,被动探测,对于存在Orderby注入的请求将会在HTTP Histroy里标红 2. 自定义排序参数list

Automne 21 Aug 12, 2022
Check for breached passwords with k-anonymity

passwnd Check for breached passwords with k-anonymity Usage To get prompted to enter the password securely, simply run: passwnd.py Alternatively, you

Nat 1 Feb 08, 2022
Burp Extensions

Burp Extensions This is a collection of extensions to Burp Suite that I have written. getAllParams.py - Version 1.2 This is a python extension that ru

/XNL-h4ck3r 364 Dec 30, 2022
Simple script to have LDAP authentication in Home Assistant Docker, using NGINX's ldap-auth container

Home Assistant LDAP Auth Simple script to have LDAP authentication in Home Assistant Docker, using NGINX's ldap-auth container. Usage Deploy NGINX's l

Erik 1 Sep 21, 2022
Python Password Generator

This is a console-based version of a password generator written with Python. The program generates a password based on numbers of letters, numbers, and symbols specified by the user. This is a simple

p.katekomol 1 Jan 24, 2022
CVE-2021-43798Exp多线程批量验证脚本

Grafana V8.*任意文件读取Exp--多线程批量验证脚本 漏洞描述 Grafana是一个开源的度量分析与可视化套件。经常被用作基础设施的时间序列数据和应用程序分析的可视化,它在其他领域也被广泛的使用包括工业传感器、家庭自动化、天气和过程控制等。其 8.*版本任意文件读取漏洞,该漏洞目前为0d

2 Dec 16, 2021
Click-Jack - Automatic tool to find Clickjacking Vulnerability in various Web applications

CLICK-Jack It is a automatic tool to find Clickjacking Vulnerability in various

Prince Prafull 4 Jan 10, 2022
C++ fully undetected shellcode launcher

charlotte c++ fully undetected shellcode launcher ;) releasing this to celebrate the birth of my newborn description 13/05/2021: c++ shellcode launche

894 Dec 25, 2022
Extensive Python3 network scanner, simplified.

Snake Map Extensive Python3 network scanner, simplified. _,.--. --..,_ .'`__ o `;__, `'.'. .'.'` '---'` '

Miss Bliss 4 Apr 16, 2022
Generate malicious files using recently published homoglyphic-attack (CVE-2021-42694)

CVE-2021-42694 Generate malicious files using recently published homoglyph-attack vulnerability, which was discovered at least in C, C++, C#, Go, Pyth

js-on 17 Dec 11, 2022
Safe Policy Optimization with Local Features

Safe Policy Optimization with Local Feature (SPO-LF) This is the source-code for implementing the algorithms in the paper "Safe Policy Optimization wi

Akifumi Wachi 6 Jun 05, 2022
Spray365 is a password spraying tool that identifies valid credentials for Microsoft accounts (Office 365 / Azure AD).

What is Spray365? Spray365 is a password spraying tool that identifies valid credentials for Microsoft accounts (Office 365 / Azure AD). How is Spray3

Mark Hedrick 246 Dec 28, 2022
A Radare2 based Python module for Binary Analysis and Reverse Engineering.

Zepu1chr3 A Radare2 based Python module for Binary Analysis and Reverse Engineering. Installation You can simply run this command. pip3 install zepu1c

Mehmet Ali KERİMOĞLU 5 Aug 25, 2022
Virus-Builder - This tool will generate a virus that can only destroy Windows computer

Virus-Builder - This tool will generate a virus that can only destroy Windows computer. You can also configure to auto run in usb drive

Saad 16 Dec 30, 2022
S2-061 的payload,以及对应简单的PoC/Exp

S2-061 脚本皆根据vulhub的struts2-059/061漏洞测试环境来写的,不具普遍性,还望大佬多多指教 struts2-061-poc.py(可执行简单系统命令) 用法:python struts2-061-poc.py http://ip:port command 例子:python

dreamer 46 Oct 20, 2022
Attack SQL Server through gopher protocol

Attack SQL Server through gopher protocol

hack2fun 17 Nov 30, 2022