Import Python modules from dicts and JSON formatted documents.

Overview

Paker

Build Version Version

Paker is module for importing Python packages/modules from dictionaries and JSON formatted documents. It was inspired by httpimporter.

Important: Since v0.6.0 paker supports importing .pyd and .dll modules directly from memory. This was achieved by using _memimporter from py2exe project. Importing .so files on Linux still requires writing them to disk.

Installation

From PyPI

pip install paker -U

From source

git clone https://github.com/desty2k/paker.git
cd paker
pip install .

Usage

In Python script

You can import Python modules directly from string, dict or bytes (without disk IO).

import paker
import logging

MODULE = {"somemodule": {"type": "module", "extension": "py", "code": "fun = lambda x: x**2"}}
logging.basicConfig(level=logging.NOTSET)

if __name__ == '__main__':
    with paker.loads(MODULE) as loader:
        # somemodule will be available only in this context
        from somemodule import fun
        assert fun(2), 4
        assert fun(5), 25
        print("6**2 is {}".format(fun(6)))
        print("It works!")

To import modules from .json files use load function. In this example paker will serialize and import mss package.

import paker
import logging

file = "mss.json"
logging.basicConfig(level=logging.NOTSET)

# install mss using `pip install mss`
# serialize module
with open(file, "w+") as f:
    paker.dump("mss", f, indent=4)

# now you can uninstall mss using `pip uninstall mss -y`
# load package back from dump file
with open(file, "r") as f:
    loader = paker.load(f)

import mss
with mss.mss() as sct:
    sct.shot()

# remove loader and clean the cache
loader.unload()

try:
    # this will throw error
    import mss
except ImportError:
    print("mss unloaded successfully!")

CLI

Paker can also work as a standalone script. To dump module to JSON dict use dump command:

paker dump mss

To recreate module from JSON dict use load:

paker load mss.json

Show all modules and packages in .json file

paker list mss.json

How it works

When importing modules or packages Python iterates over importers in sys.meta_path and calls find_module method on each object. If the importer returns self, it means that the module can be imported and None means that importer did not find searched package. If any importer has confirmed the ability to import module, Python executes another method on it - load_module. Paker implements its own importer called jsonimporter, which instead of searching for modules in directories, looks for them in Python dictionaries

To dump module or package to JSON document, Paker recursively iterates over modules and creates dict with code and type of each module and submodules if object is package.

You might also like...
An executor that loads ONNX models and embeds documents using the ONNX runtime.

ONNXEncoder An executor that loads ONNX models and embeds documents using the ONNX runtime. Usage via Docker image (recommended) from jina import Flow

Implementation of self-attention mechanisms for general purpose. Focused on computer vision modules. Ongoing repository.
Implementation of self-attention mechanisms for general purpose. Focused on computer vision modules. Ongoing repository.

Self-attention building blocks for computer vision applications in PyTorch Implementation of self attention mechanisms for computer vision in PyTorch

Turning SymPy expressions into PyTorch modules.

sympytorch A micro-library as a convenience for turning SymPy expressions into PyTorch Modules. All SymPy floats become trainable parameters. All SymP

DI-HPC is an acceleration operator component for general algorithm modules in reinforcement learning algorithms

DI-HPC: Decision Intelligence - High Performance Computation DI-HPC is an acceleration operator component for general algorithm modules in reinforceme

Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules
Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules

DCSR: Dual Camera Super-Resolution Implementation for our ICCV 2021 oral paper: Dual-Camera Super-Resolution with Aligned Attention Modules paper | pr

Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules
Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules

DCSR: Dual Camera Super-Resolution Implementation for our ICCV 2021 oral paper: Dual-Camera Super-Resolution with Aligned Attention Modules paper | pr

Weight initialization schemes for PyTorch nn.Modules

nninit Weight initialization schemes for PyTorch nn.Modules. This is a port of the popular nninit for Torch7 by @kaixhin. ##Update This repo has been

Pytorch modules for paralel models with same architecture. Ideal for multi agent-based systems
Pytorch modules for paralel models with same architecture. Ideal for multi agent-based systems

WideLinears Pytorch parallel Neural Networks A package of pytorch modules for fast paralellization of separate deep neural networks. Ideal for agent-b

Stacs-ci - A set of modules to enable integration of STACS with commonly used CI / CD systems
Stacs-ci - A set of modules to enable integration of STACS with commonly used CI / CD systems

Static Token And Credential Scanner CI Integrations What is it? STACS is a YARA

Comments
  • psutil example exits with module not found when using _memimporter

    psutil example exits with module not found when using _memimporter

    I pulled latest releases zip file, ran python setup.py build and attempted to run the psutil example with the compiled pyd. This resulted in the following error:

    DEBUG:jsonimporter:searching for pwd
    DEBUG:jsonimporter:searching for psutil._common
    INFO:jsonimporter:psutil._common has been imported successfully
    DEBUG:jsonimporter:searching for psutil._compat
    INFO:jsonimporter:psutil._compat has been imported successfully
    DEBUG:jsonimporter:searching for psutil._pswindows
    DEBUG:jsonimporter:searching for psutil._psutil_windows
    DEBUG:jsonimporter:searching for psutil._psutil_windows
    INFO:jsonimporter:using _memimporter to load '.pyd' file
    INFO:jsonimporter:unloaded all modules
    Traceback (most recent call last):
      File "c:\Users\User\Desktop\paker-0.7.1\paker-0.7.1\build\lib.win-amd64-cpython-310\psutil_example.py", line 20, in <module>
        import psutil
      File "c:\Users\User\Desktop\paker-0.7.1\paker-0.7.1\build\lib.win-amd64-cpython-310\paker\importers\jsonimporter.py", line 115, in load_module
        exec(jsonmod["code"], mod.__dict__)
      File "<string>", line 107, in <module>
      File "c:\Users\User\Desktop\paker-0.7.1\paker-0.7.1\build\lib.win-amd64-cpython-310\paker\importers\jsonimporter.py", line 115, in load_module
        exec(jsonmod["code"], mod.__dict__)
      File "<string>", line 35, in <module>
      File "c:\Users\User\Desktop\paker-0.7.1\paker-0.7.1\build\lib.win-amd64-cpython-310\paker\importers\jsonimporter.py", line 134, in load_module
        mod = _memimporter.import_module(fullname, path, initname, self._get_data, spec)
    ImportError: MemoryLoadLibrary failed loading psutil\_psutil_windows.pyd: The specified module could not be found. (126)
    

    Is this an issue with how I compiled memimporter, or something else?

    opened by rkbennett 1
Releases(v0.7.1)
Owner
Wojciech Wentland
Wojciech Wentland
A PyTorch Implementation of "Watch Your Step: Learning Node Embeddings via Graph Attention" (NeurIPS 2018).

Attention Walk ⠀⠀ A PyTorch Implementation of Watch Your Step: Learning Node Embeddings via Graph Attention (NIPS 2018). Abstract Graph embedding meth

Benedek Rozemberczki 303 Dec 09, 2022
Denoising Diffusion Implicit Models

Denoising Diffusion Implicit Models (DDIM) Jiaming Song, Chenlin Meng and Stefano Ermon, Stanford Implements sampling from an implicit model that is t

465 Jan 05, 2023
A highly efficient and modular implementation of Gaussian Processes in PyTorch

GPyTorch GPyTorch is a Gaussian process library implemented using PyTorch. GPyTorch is designed for creating scalable, flexible, and modular Gaussian

3k Jan 02, 2023
functorch is a prototype of JAX-like composable function transforms for PyTorch.

functorch is a prototype of JAX-like composable function transforms for PyTorch.

Facebook Research 1.2k Jan 09, 2023
R interface to fast.ai

R interface to fastai The fastai package provides R wrappers to fastai. The fastai library simplifies training fast and accurate neural nets using mod

113 Dec 20, 2022
nanodet_plus,yolov5_v6.0

OAK_Detection OAK设备上适配nanodet_plus,yolov5_v6.0 Environment pytorch = 1.7.0

炼丹去了 1 Feb 18, 2022
LaBERT - A length-controllable and non-autoregressive image captioning model.

Length-Controllable Image Captioning (ECCV2020) This repo provides the implemetation of the paper Length-Controllable Image Captioning. Install conda

bearcatt 53 Nov 13, 2022
A Pytorch implementation of the multi agent deep deterministic policy gradients (MADDPG) algorithm

Multi-Agent-Deep-Deterministic-Policy-Gradients A Pytorch implementation of the multi agent deep deterministic policy gradients(MADDPG) algorithm This

Phil Tabor 159 Dec 28, 2022
基于Paddlepaddle复现yolov5,支持PaddleDetection接口

PaddleDetection yolov5 https://github.com/Sharpiless/PaddleDetection-Yolov5 简介 PaddleDetection飞桨目标检测开发套件,旨在帮助开发者更快更好地完成检测模型的组建、训练、优化及部署等全开发流程。 PaddleD

36 Jan 07, 2023
Machine learning, in numpy

numpy-ml Ever wish you had an inefficient but somewhat legible collection of machine learning algorithms implemented exclusively in NumPy? No? Install

David Bourgin 11.6k Dec 30, 2022
EigenGAN Tensorflow, EigenGAN: Layer-Wise Eigen-Learning for GANs

Gender Bangs Body Side Pose (Yaw) Lighting Smile Face Shape Lipstick Color Painting Style Pose (Yaw) Pose (Pitch) Zoom & Rotate Flush & Eye Color Mout

Zhenliang He 321 Dec 01, 2022
Biomarker identification for COVID-19 Severity in BALF cells Single-cell RNA-seq data

scBALF Covid-19 dataset Analysis Here is the Github page that has the codes for the bioinformatics pipeline described in the paper COVID-Datathon: Bio

Nami Niyakan 2 May 21, 2022
This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis, accepted at EMNLP 2021.

MultiModal-InfoMax This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Informa

Deep Cognition and Language Research (DeCLaRe) Lab 89 Dec 26, 2022
Tensorboard for pytorch (and chainer, mxnet, numpy, ...)

tensorboardX Write TensorBoard events with simple function call. The current release (v2.3) is tested on anaconda3, with PyTorch 1.8.1 / torchvision 0

Tzu-Wei Huang 7.5k Dec 28, 2022
Tensorflow 2 Object Detection API kurulumu, GPU desteği, custom model hazırlama

Tensorflow 2 Object Detection API Bu tutorial, TensorFlow 2.x'in kararlı sürümü olan TensorFlow 2.3'ye yöneliktir. Bu, görüntülerde / videoda nesne a

46 Nov 20, 2022
Creating predictive checklists from data using integer programming.

Learning Optimal Predictive Checklists A Python package to learn simple predictive checklists from data subject to customizable constraints. For more

Healthy ML 5 Apr 19, 2022
[ICCV 2021] Encoder-decoder with Multi-level Attention for 3D Human Shape and Pose Estimation

MAED: Encoder-decoder with Multi-level Attention for 3D Human Shape and Pose Estimation Getting Started Our codes are implemented and tested with pyth

ZiNiU WaN 176 Dec 15, 2022
Research code for the paper "How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models"

Introduction This repository contains research code for the ACL 2021 paper "How Good is Your Tokenizer? On the Monolingual Performance of Multilingual

AdapterHub 20 Aug 04, 2022
Many Class Activation Map methods implemented in Pytorch for CNNs and Vision Transformers. Including Grad-CAM, Grad-CAM++, Score-CAM, Ablation-CAM and XGrad-CAM

Class Activation Map methods implemented in Pytorch pip install grad-cam ⭐ Tested on many Common CNN Networks and Vision Transformers. ⭐ Includes smoo

Jacob Gildenblat 6.6k Jan 06, 2023
A Pytorch implementation of MoveNet from Google. Include training code and pre-train model.

Movenet.Pytorch Intro MoveNet is an ultra fast and accurate model that detects 17 keypoints of a body. This is A Pytorch implementation of MoveNet fro

Mr.Fire 241 Dec 26, 2022