ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete

Overview

ShadowClone

ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete.

You can make full use of the Free Tiers provided by cloud providers and supercharge your mundane cli tools with shadow clone jutsu (Naruto style)!

Installation

Please visit the wiki for installation and intial configuration instructions

Usage

⚡ python shadowclone.py -h
usage: shadowclone.py [-h] -i INPUT [-s SPLITNUM] [-o OUTPUT] -c COMMAND

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
  -s SPLITNUM, --split SPLITNUM
                        Number of lines per chunk of file
  -o OUTPUT, --output OUTPUT
  -c COMMAND, --command COMMAND
                        command to execute

How it works

We create a container image during the initial setup and register it as a runtime for our function in AWS/GCP/Azure whatever. When you execute ShadowClone on your computer, instances of that container are activated automatically and are only active for the duration of its execution. How many instances to activate is dynamically decided at runtime depending on the size of the input file provided and the split factor. The input is then split into chunks and equally distributed between all the instances to execute in parallel. For example, if your input file has 10,000 lines and you set the split factor to 100 lines, then it will be split into 100 chunks of 100 lines each and 100 instances will be run in parallel!

Features

  • Extremely fast
  • No need to maintain a VPS (or a fleet of it :))
  • Costs almost nothing per month
    • Compatible with free tiers of most cloud services
  • Cloud agnostic
    • Same script works with AWS, GCP, Azure etc.
  • Supports upto 1000 parallel invocations
  • Dynamically decide the number of invocations
  • Run any tool in parallel on the cloud
  • Pipe output to other tools

Comparison

This tool was inspired by the awesome Axiom and Fleex projects and goes beyond the concept of VPS for running the tools by using serverless functions and containers.

Features Axiom/Fleex ShadowClone
Instances 10-100s* 1000s
Cost Per instance/per minute Mostly Free**
Startup Time 4-5 minutes 2-3 seconds
Idle Cost $++ Free
On Demand Scalability No

*Most cloud providers do not allow spinning up too many instances by default, so you are limited to around 10-15 instances at max. You have to make a request to the support to increase this number.

** AWS & Azure allow 1 million invocations per month for free. Google allows 2 million invocations per month for free. You will be charged only if you go above these limits

Demo

DNS Bruteforcing using a 43mb file - 34 seconds

asciicast

Running httpx on 94K subdomains in 1 min

asciicast

References

Lithops documentation

Free Tiers

Cloud Provider Free Allowance Link
Google Functions 2 Million invocations, 400,000 GB-seconds per month Google Cloud Free Program
AWS Lambda 1 Million invocations, Up to 3.2 million seconds of compute time per month Free Cloud Computing Services - AWS Free Tier
Azure Functions 1 Million invocations Microsoft Azure Free Services

Obviously, you can make any number of function invocations per month. The table above only shows how many invocations are free.

Similar Tools

Disclaimer

This tool is designed as a proof-of-concept implementation and it's usage is intended for educational purpose only. Usage for attacking targets without prior mutual consent is illegal. It's the end user's responsibility to obey all applicable local, state and federal laws. Developers assume no liability and are not responsible for any misuse or damage caused by this program.

A badge generator service to count visitors of your markdown file.

Github Visitors Badge A badge generator service to count visitors of your markdown file. Hello every one! In this post, I will tell you the story of m

Kɪꜱᴀʀᴀ Pᴇꜱᴀɴᴊɪᴛʜ Pᴇʀᴇʀᴀ 〄 1 Feb 06, 2022
HackerNews and Reddit in one placce

EDIT: this project is 3.5 years old. I found it sad it's just laying around, so I did some minimal fixes and deployed it. Hope you enjoy! (PR's welcom

Hugo Montenegro 1 Nov 13, 2021
Manage Proxmox KVM Virtual Machines via Slack bot.

proxmox-slack-bot Create KVM Virtual Machines on Proxmox, the easy way. Not much works works here yet... Setup dev environment Setup fully editable st

Plenus Pyramis 3 Mar 20, 2022
Senditapp.com bot spammer, spam your friends

Sendit Spammer Python ⚠️ I am not responsible for how you use this tool. This tool is against "Sendit" ToS and shall not be used in a production envir

Glaukio 1 Dec 31, 2021
A telegram bot to track whales activities on multiple blockchains.

Telegram Bot : Whale Watcher A straightforward telegram bot written in python to track whales activity on multiple blockchains, using whale-alert API

Laurenz Bougan 1 Dec 10, 2021
Reads and prints information from the website MalAPI.io

MalAPIReader Reads and prints information from the website MalAPI.io optional arguments:

Squiblydoo 16 Nov 10, 2022
Python wrapper for eBay API

python-ebay - Python Wrapper for eBay API This project intends to create a simple python wrapper around eBay APIs. Development and Download Sites The

Roopesh 99 Nov 16, 2022
Generate visualizations of GitHub user and repository statistics using GitHubActions

GitHub Stats Visualization Generate visualizations of GitHub user and repository

Jun Shi 3 Dec 15, 2022
A Telegram Bot to manage your music channel with some cool features.

Music Channel Manager V2 A Telegram Bot to manage your music channel with some cool features like appending your predefined username to the musics tag

11 Oct 21, 2022
Twitter Analysis of MIUUL CEO

Twitter Analysis of MIUUL CEO Business Problem I got last @mvahitkeskin 184 twee

Çağrı Karadeniz 6 Mar 12, 2022
Another secured and Yet Fastest telegram userbot

Vision-UserBot A stable, simple Telegram UserBot in Pyrogram! Support Variables ➨ TG_APP_ID - Your Telegram Api id. ➨ TG_API_HASH - Your Telegram Api

TeamVision 40 Oct 24, 2022
Technical Test - Python Programmer Grupo Nexxera

Technical Test Nexxera Group Teste Técnico - Grupo de Programadores Python Nexxera 📍 Prepare-se para usar a API Nossa API de transações de contas dig

Raul dos Santos Moraes 1 Mar 16, 2022
Hydro Quebec API wrapper.

HydroQC Hydro Quebec API wrapper. This is a package to access some functionalities of Hydro Quebec API that are not documented. Documentation https://

Olivier BEAU 9 Dec 02, 2022
Token-gate Notion pages

This is a Next.js project bootstrapped with create-next-app. Getting Started First, run the development server: npm run dev # or yarn dev Open http://

John 8 Oct 13, 2022
Telegram 隨機色圖,支援每日自動爬取

Telegram 隨機色圖機器人 使用此原始碼的Bot 開放的隨機色圖機器人: @katonei_bot 已實現的功能 爬取每日R18排行榜 不夠色!再來一張 Tag 索引,指定Tag色圖 將爬取到的色圖轉為 WebP 格式儲存,節省空間 需要注意的事件 好久之前的怪東西,代碼質量不保證 請在使用A

cluckbird 15 Oct 18, 2021
LoL API is a Python application made to serve League of Legends data.

LoL API is a Python application made to serve League of Legends data.

Caique Cunha Pereira 1 Nov 06, 2021
This is a okay that is okay that means none is okay

Owner: Masterolic 🇮🇳 CatUB A Powerful, Smart And Simple Userbot In Telethon. Credits This is A Remix Bot Of Many UserBot. DARKCOBRA FridayUserBot Ja

Masterolic 1 Nov 28, 2021
Administration Panel for Control FiveM Servers From Discord

FiveM Discord Administration Panel Version 1.0.0 If you would like to report an issue or request a feature. Join our Discord or create an issue. Contr

NIma 9 Jun 17, 2022
Open API to list Viet Nam administrative divisions

Viet Nam province API Homepage: https://provinces.open-api.vn This is online tool to let my VietnamProvinces library reach more users. VietnamProvince

Nguyễn Hồng Quân 52 Dec 05, 2022
Based on nonebot, a common bot framework for maimai.

mai bot 使用指南 此 README 提供了最低程度的 mai bot 教程与支持。 Step 1. 安装 Python 请自行前往 https://www.python.org/ 下载 Python 3 版本( 3.7)并将其添加到环境变量(在安装过程中勾选 Add to system P

Diving-Fish 150 Jan 01, 2023