Augmented Traffic Control: A tool to simulate network conditions

Overview

Augmented Traffic Control

build-status-image pypi-version

Full documentation for the project is available at http://facebook.github.io/augmented-traffic-control/.

Overview

Augmented Traffic Control (ATC) is a tool to simulate network conditions. It allows controlling the connection that a device has to the internet. Developers can use ATC to test their application across varying network conditions, easily emulating high speed, mobile, and even severely impaired networks. Aspects of the connection that can be controlled include:

  • bandwidth
  • latency
  • packet loss
  • corrupted packets
  • packets ordering

In order to be able to shape the network traffic, ATC must be running on a device that routes the traffic and sees the real IP address of the device, like your network gateway for instance. This also allows any devices that route through ATC to be able to shape their traffic. Traffic can be shaped/unshaped using a web interface allowing any devices with a web browser to use ATC without the need for a client application.

ATC is made of multiple components that interact together:

  • atcd: The ATC daemon which is responsible for setting/unsetting traffic shaping. atcd exposes a Thrift interface to interact with it.
  • django-atc-api: A Django app based on Django Rest Framework that provides a RESTful interface to atcd.
  • django-atc-demo-ui: A Django app that provides a simple Web UI to use atc from a mobile phone.
  • django-atc-profile-storage: A Django app that can be used to save shaping profiles, making it easier to re-use them later without manually re-entering those settings.

By splitting ATC in sub-components, it make it easier to hack on it or build on top of it. While django-atc-demo-ui is shipped as part of ATC's main repository to allow people to be able to use ATC out of the box, by providing a REST API to atcd, it makes it relatively easy to interact with atcd via the command line and opens the path for the community to be able to build creative command line tools, web UI or mobile apps that interact with ATC.

ATC architecture

Requirements

Most requirements are handled automatically by pip, the packaging system used by ATC, and each ATC package may have different requirements and the README.md files of the respective packages should be checked for more details. Anyhow, some requirements apply to the overall codebase:

  • Python 2.7: Currently, ATC is only supported on python version 2.7.
  • Django 1.10: Currently, ATC is only supported using django version 1.10.

Installing ATC

The fact that ATC is splitted in multiple packages allows for multiple deployment scenarii. However, deploying all the packages on the same host is the simplest and most likely fitting most use cases.

To get more details on how to install/configure each packages, please refer to the packages' respective READMEs.

Packages

The easiest way to install ATC is by using pip.

pip install atc_thrift atcd django-atc-api django-atc-demo-ui django-atc-profile-storage

Django

Now that we have all the packages installed, we need to create a new Django project in which we will use our Django app.

django-admin startproject atcui
cd atcui

Now that we have our django project, we need to configure it to use our apps and we need to tell it how to route to our apps.

Open atcui/settings.py and enable the ATC apps by adding to INSTALLED_APPS:

INSTALLED_APPS = (
    ...
    # Django ATC API
    'rest_framework',
    'atc_api',
    # Django ATC Demo UI
    'bootstrap_themes',
    'django_static_jquery',
    'atc_demo_ui',
    # Django ATC Profile Storage
    'atc_profile_storage',
)

Now, open atcui/urls.py and enable routing to the ATC apps by adding the routes to urlpatterns:

...
...
from django.views.generic.base import RedirectView
from django.conf.urls import include

urlpatterns = [
    ...
    # Django ATC API
    url(r'^api/v1/', include('atc_api.urls')),
    # Django ATC Demo UI
    url(r'^atc_demo_ui/', include('atc_demo_ui.urls')),
    # Django ATC profile storage
    url(r'^api/v1/profiles/', include('atc_profile_storage.urls')),
    url(r'^$', RedirectView.as_view(url='/atc_demo_ui/', permanent=False)),
]

Finally, let's update the Django DB:

python manage.py migrate

Running ATC

All require packages should now be installed and configured. We now need to run the daemon and the UI interface. While we will run ATC straight from the command line in this example, you can refer to example sysvinit and upstart scripts.

atcd

atcd modifies network related settings and as such needs to run in privileged mode:

sudo atcd

Supposing eth0 is your interface to connect to the internet and eth1, your interface to connect to your lan, this should just work. If your setting is slightly different, use the command line arguments --atcd-wan and --atcd-lan to adapt to your configuration.

ATC UI

The UI on the other hand is a standard Django Web app and can be run as a normal user. Make sure you are in the directory that was created when you ran django-admin startproject atcui and run:

python manage.py runserver 0.0.0.0:8000

You should now be able to access the web UI at http://localhost:8000

ATC Code Structure

ATC source code is available under the atc directory, it is currently composed of:

  • atc_thrift the thrift interface's library
  • atcd the ATC daemon that runs on the router doing the traffic shaping
  • django-atc-api A django app that provides a RESTful interface to atcd
  • django-atc-demo-ui A django app that provides a simple demo UI leveraging the RESTful API
  • django-atc-profile-storage A django app that allows saving shaping profiles to DB allowing users to select their favorite profile from a list instead of re-entering all the profile details every time.

The chef directory contains 2 chef cookbooks:

  • atc A cookbook to deploy ATC. It also allows to deploy ATC in a Virtual Box VM in order to develop on ATC.
  • atclient Set up a Linux Desktop VM that can be used to test shaping end to end.

atcd

atcd is the daemon that runs on the router that does the shaping. Interaction with the daemon is done using thrift. The interface definition can be found in atc_thrift.thrift.

atc_thrift

atc_thrift defines the thrift interface to communicate with the atcd daemon.

django-atc-api

django-atc-api is a django app that provide a REST API to the atcd daemon. Web applications, command line tools can use the API in order to shape/unshape traffic.

django-atc-demo-ui

django-atc-demo-ui is a simple Web UI to enable/disable traffic shaping. The UI is mostly written in React

django-atc-profile-storage

django-atc-profile-storage allows saving profiles to DB. A typical use case will be to save a list of predefined/often used shaping settings that you want to be able to accessing in just a few clicks/taps.

Developing on ATC

To make ATC development easier, we use Virtual Box and Vagrant to provision and run a VM that will run the ATC daemon and the ATC UI from your git checkout.

Interacting with ATC will only shape the traffic within the VM and not on the host.

Setting up the environment

Note: vagrant is an easy way to set up a test environment, but virtualization will produce different results than a setup on bare-metal. We recommend using vagrant only for testing/development and using bare-metal for setups which require realistic shaping settings.

You will need to install VirtualBox, Vagrant and a couple of plugins:

  • VirtualBox
  • Vagrant
  • Chef DK
  • Install some vagrant plugins:
  • vagrant plugin install vagrant-berkshelf --plugin-version '>= 2.0.1'
  • vagrant plugin install vagrant-omnibus
  • Clone this repo: git clone [email protected]:facebook/augmented-traffic-control.git atc

Running ATC

Once in the repo, go to the chef/atc directory and run:

vagrant up trusty

This will take some time before it completes, once the VM is provision, SSH into it:

vagrant ssh trusty

You should now be able to access ATC at: http://localhost:8080/

Using the Sample Profiles

Once you've got ATC up and running, you can run the script utils/restore-profiles.sh to setup the set of default profiles.

The script needs to be passed a hostname:port with the location of your ATC instance:

utils/restore-profiles.sh localhost:8080

After doing this, you should see the 10 sample profiles listed below in your ATC instance:

  • 2G - Developing Rural
  • 2G - Developing Urban
  • 3G - Average
  • 3G - Good
  • Cable
  • DSL
  • Edge - Average
  • Edge - Good
  • Edge - Lossy
  • No Connectivity

Naturally, you cannot improve your natural network speed by selecting a faster profile than your service. For example, selecting the Cable profile will not make your network faster if your natural connection speed resembles DSL more closely.

Hacking on the code

Hacking on ATC is done from the host and tested in the VM. In order to reflect the changes, you will need to start the services manually.

Both atcd and atcui have their python libraries installed in a python virtualenv so you will need to activate the environment in order to be able to run the services.

The virtualenv is installed in /usr/local/atc/venv/bin/activate .

source /usr/local/atc/venv/bin/activate

Running the daemon

The atcd daemon is running under the root user privileges, all operations below needs to be done as root.

To run the daemon manually, first make sure it is not running in the background:

service atcd stop

And run the daemon:

atcd

Once you are happy with your changes and you want to test them, you will need to kill the daemon and restart it in order to apply the changes.

Running the API/UI

This is a django project and, when running the django built-in HTTP server, will detect code changes and reload automatically.

To run the HTTP REST API and UI:

cd /var/django && python manage.py runserver 0.0.0.0:8000
Owner
Meta Archive
These projects have been archived and are generally unsupported, but are still available to view and use
Meta Archive
[IEEE Transactions on Computational Imaging] Self-Gated Memory Recurrent Network for Efficient Scalable HDR Deghosting

Few-shot Deep HDR Deghosting This repository contains code and pretrained models for our paper: Self-Gated Memory Recurrent Network for Efficient Scal

Susmit Agrawal 4 Dec 29, 2021
SafePicking: Learning Safe Object Extraction via Object-Level Mapping, ICRA 2022

SafePicking Learning Safe Object Extraction via Object-Level Mapping Kentaro Wad

Kentaro Wada 49 Oct 24, 2022
Structured Edge Detection Toolbox

################################################################### # # # Structure

Piotr Dollar 779 Jan 02, 2023
CLIP (Contrastive Language–Image Pre-training) trained on Indonesian data

CLIP-Indonesian CLIP (Radford et al., 2021) is a multimodal model that can connect images and text by training a vision encoder and a text encoder joi

Galuh 17 Mar 10, 2022
ECCV18 Workshops - Enhanced SRGAN. Champion PIRM Challenge on Perceptual Super-Resolution. The training codes are in BasicSR.

ESRGAN (Enhanced SRGAN) [ 🚀 BasicSR] [Real-ESRGAN] ✨ New Updates. We have extended ESRGAN to Real-ESRGAN, which is a more practical algorithm for rea

Xintao 4.7k Jan 02, 2023
E-RAFT: Dense Optical Flow from Event Cameras

E-RAFT: Dense Optical Flow from Event Cameras This is the code for the paper E-RAFT: Dense Optical Flow from Event Cameras by Mathias Gehrig, Mario Mi

Robotics and Perception Group 71 Dec 12, 2022
Deep Learning Theory

Deep Learning Theory 整理了一些深度学习的理论相关内容,持续更新。 Overview Recent advances in deep learning theory 总结了目前深度学习理论研究的六个方向的一些结果,概述型,没做深入探讨(2021)。 1.1 complexity

fq 103 Jan 04, 2023
A MNIST-like fashion product database. Benchmark

Fashion-MNIST Table of Contents Why we made Fashion-MNIST Get the Data Usage Benchmark Visualization Contributing Contact Citing Fashion-MNIST License

Zalando Research 10.5k Jan 08, 2023
The code repository for EMNLP 2021 paper "Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization".

Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization [Paper] accepted at the EMNLP 2021: Vision Guided Genera

CAiRE 42 Jan 07, 2023
Scene-Text-Detection-and-Recognition (Pytorch)

Scene-Text-Detection-and-Recognition (Pytorch) Competition URL: https://tbrain.t

Gi-Luen Huang 9 Jan 02, 2023
Implementation of ConvMixer-Patches Are All You Need? in TensorFlow and Keras

Patches Are All You Need? - ConvMixer ConvMixer, an extremely simple model that is similar in spirit to the ViT and the even-more-basic MLP-Mixer in t

Sayan Nath 8 Oct 03, 2022
Official implementation of the paper "Topographic VAEs learn Equivariant Capsules"

Topographic Variational Autoencoder Paper: https://arxiv.org/abs/2109.01394 Getting Started Install requirements with Anaconda: conda env create -f en

T. Andy Keller 69 Dec 12, 2022
Neural Module Network for VQA in Pytorch

Neural Module Network (NMN) for VQA in Pytorch Note: This is NOT an official repository for Neural Module Networks. NMN is a network that is assembled

Harsh Trivedi 111 Nov 24, 2022
A full-fledged version of Pix2Seq

Stable-Pix2Seq A full-fledged version of Pix2Seq What it is. This is a full-fledged version of Pix2Seq. Compared with unofficial-pix2seq, stable-pix2s

peng gao 205 Dec 27, 2022
Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021)

Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021) Kranti Kumar Parida, Siddharth Srivastava, Gaurav Sharma. We address the pr

Kranti Kumar Parida 33 Jun 27, 2022
TensorFlow for Raspberry Pi

TensorFlow on Raspberry Pi It's officially supported! As of TensorFlow 1.9, Python wheels for TensorFlow are being officially supported. As such, this

Sam Abrahams 2.2k Dec 16, 2022
Off-policy continuous control in PyTorch, with RDPG, RTD3 & RSAC

arXiv technical report soon available. we are updating the readme to be as comprehensive as possible Please ask any questions in Issues, thanks. Intro

Zhihan 31 Dec 30, 2022
Convert Mission Planner (ArduCopter) Waypoint Missions to Litchi CSV Format to execute on DJI Drones

Mission Planner to Litchi Convert Mission Planner (ArduCopter) Waypoint Surveys to Litchi CSV Format to execute on DJI Drones Litchi doesn't support S

Yaros 24 Dec 09, 2022
A Python library for Deep Probabilistic Modeling

Abstract DeeProb-kit is a Python library that implements deep probabilistic models such as various kinds of Sum-Product Networks, Normalizing Flows an

DeeProb-org 46 Dec 26, 2022
Get a Grip! - A robotic system for remote clinical environments.

Get a Grip! Within clinical environments, sterilization is an essential procedure for disinfecting surgical and medical instruments. For our engineeri

Jay Sharma 1 Jan 05, 2022