Write maintainable, production-ready pipelines using Jupyter or your favorite text editor. Develop locally, deploy to the cloud. ☁️

Overview

CI Linux CI macOS CI Windows Documentation Status PyPI Conda (channel only) Conda Coverage Twitter Binder

Join our community | Newsletter | Contact us | Docs | Blog

Notebooks are hard to maintain. Teams often prototype projects in notebooks, but maintaining them is an error-prone process that slows progress down. Ploomber overcomes the challenges of working with .ipynb files allowing teams to develop collaborative, production-ready pipelines using JupyterLab or any text editor.

Main Features

  1. Scripts as notebooks. Open .py files as notebooks, then execute them from the terminal and generate an output notebook to review results.
  2. Dependency resolution. Quickly build a DAG by referring to previous tasks in your code; Ploomber infers execution order and orchestrates execution.
  3. Incremental builds. Speed up iterations by skipping tasks whose source code hasn't changed since the last execution.
  4. Production-ready. Deploy to Kubernetes (via Argo Workflows), Airflow, and AWS Batch without code changes.
  5. Parallelization. Run independent tasks in parallel.
  6. Testing. Import pipelines in any testing frameworks and test them with any CI service (e.g. GitHub Actions).
  7. Flexible. Use Jupyter notebooks, Python scripts, R scripts, SQL scripts, Python functions, or a combination of them as pipeline tasks. Write pipelines using a pipeline.yaml file or with Python.

repo-lab-example

Community

Resources

Installation

Compatible with Python 3.6 and higher.

Install with pip:

pip install ploomber

Or with conda:

conda install ploomber -c conda-forge

Getting started

Use Binder to try out Ploomber without setting up an environment:

image

Or run an example locally:

# ML pipeline example
ploomber examples --name ml-basic
cd ml-basic

# if using pip
pip install -r requirements.txt

# if using conda
conda env create --file environment.yml
conda activate ml-basic

# run pipeline
ploomber build

Pipeline output saved in the output/ folder. Check out the pipeline definition in the pipeline.yaml file.

To get a list of examples, run ploomber examples.

Click here to go to our examples repository.

Owner
Ploomber
A community dedicated to promote and discuss best practices for Data Science software
Ploomber
Jupyter Interactive Notebook

Jupyter Notebook The Jupyter notebook is a web-based notebook environment for interactive computing. Notice Please note that this repository is curren

Project Jupyter 9.7k Jan 02, 2023
Official repository for Spyder - The Scientific Python Development Environment

Copyright © 2009–2021 Spyder Project Contributors Some source files and icons may be under other authorship/licenses; see NOTICE.txt. Project status B

Spyder IDE 7.3k Dec 31, 2022
Vim plugin for executing Python's optional static type checker MyPy

Vim plugin for executing Python's optional static type checker MyPy

Mark McDonnell 91 Dec 04, 2022
A simple Notepad-like editor written in Python

monkepad A simple Notepad-like editor written in Python Since MonkePad is written in one file, all your customization can be done without much trouble

5 Dec 28, 2021
Write maintainable, production-ready pipelines using Jupyter or your favorite text editor. Develop locally, deploy to the cloud. ☁️

Write maintainable, production-ready pipelines using Jupyter or your favorite text editor. Develop locally, deploy to the cloud. ☁️

Ploomber 2.9k Jan 06, 2023
cross-editor syntax highlighter for Lua, showing some merit of Typed BNF

Cross-editor contextual syntax highlighter via Typed BNF Do you like "one grammar, syntax highlighters everywhere?" 喜欢我一个文法,到处高亮吗? PS: NOTE that paren

Taine Zhao 14 Feb 09, 2022
openBrowsser is a Sublime Text plug-in, which allows you to add a keyboard shortcut, to directly access a website from a selection.

openBrowsser is a Sublime Text plug-in, which allows you to add a keyboard shortcut, to directly access a website from a selection. Instal

Florian COLLIN 1 Dec 14, 2021
Python IDE for beginners

Thonny Thonny is a Python IDE meant for learning programming. End users See https://thonny.org and wiki for more info. Contributors Contributions are

Thonny, Python IDE for beginners 2.1k Jan 02, 2023
A code-completion engine for Vim

YouCompleteMe: a code-completion engine for Vim NOTE: Minimum Requirements Have Changed Our policy is to support the Vim version that's in the latest

24.5k Dec 30, 2022
A powerful text editor for MATE

Pluma - The MATE text editor General Information Pluma (pluma) is a small and lightweight UTF-8 text editor for the MATE environment. It started as a

MATE Desktop 134 Dec 31, 2022
Python Tools for Visual Studio

Python tools for Visual Studio Python support for Visual Studio is developed and managed here. Visit our product overview and downloads page, read our

Microsoft 2.4k Dec 30, 2022
A JupyterLab extension for version control using Git

A JupyterLab extension for version control using Git

JupyterLab 1.2k Jan 03, 2023
A very simple Editor.js parser written in pure Python

pyEditor.js A very simple Editor.js parser written in pure Python. Soon-to-be published on PyPI. Features: Automatically convert Editor.js's JSON outp

Kevo 7 Nov 01, 2022
A WYSIWYG layout editor for Jupyter widgets

Based on the React library FlexLayout, ipyflex allows you to compose the sophisticated dashboard layouts from existing Jupyter widgets without coding. It supports multiple tabs, resizable cards, drag

Duc Trung LE 93 Nov 24, 2022
A small, simple editor for beginner Python programmers. Written in Python and Qt5.

Mu - A Simple Python Code Editor Mu is a simple code editor for beginner programmers based on extensive feedback from teachers and learners. Having sa

Mu 1.2k Jan 03, 2023
awesome Python autocompletion with SublimeText

SublimeJEDI SublimeJEDI is a Sublime Text 3 and Sublime Text 2 and plugin to the awesome autocomplete library Jedi Python Version Support Sublime Jedi

Serhii Ruskykh 933 Dec 08, 2022
Python-Text-editor: a simple text editor on Python and Tkinter

Python-Text-editor This is a simple text editor on Python and Tkinter. The proje

Innokentie 1 Jan 03, 2022
A free Python source code editor and Notepad replacement for Windows

Website Download Features Toolbar Wide array of view options Syntax highlighting support for Python Usable accelerator keys for each function (Ctrl+N,

Mohamed Ahmed 7 Feb 16, 2022
Emacs Python Development Environment

Elpy, the Emacs Python IDE Elpy is an Emacs package to bring powerful Python editing to Emacs. It combines and configures a number of other packages,

Jorgen Schäfer 1.8k Jan 02, 2023
PlugNik is a simple implementation of plugin repository for JetBrains Application.

PlugNik is a simple implementation of plugin repository for JetBrains Application.

roy reznik 11 Jun 30, 2022