Import Python modules from dicts and JSON formatted documents.

Overview

Paker

Build Version Version

Paker is module for importing Python packages/modules from dictionaries and JSON formatted documents. It was inspired by httpimporter.

Important: Since v0.6.0 paker supports importing .pyd and .dll modules directly from memory. This was achieved by using _memimporter from py2exe project. Importing .so files on Linux still requires writing them to disk.

Installation

From PyPI

pip install paker -U

From source

git clone https://github.com/desty2k/paker.git
cd paker
pip install .

Usage

In Python script

You can import Python modules directly from string, dict or bytes (without disk IO).

import paker
import logging

MODULE = {"somemodule": {"type": "module", "extension": "py", "code": "fun = lambda x: x**2"}}
logging.basicConfig(level=logging.NOTSET)

if __name__ == '__main__':
    with paker.loads(MODULE) as loader:
        # somemodule will be available only in this context
        from somemodule import fun
        assert fun(2), 4
        assert fun(5), 25
        print("6**2 is {}".format(fun(6)))
        print("It works!")

To import modules from .json files use load function. In this example paker will serialize and import mss package.

import paker
import logging

file = "mss.json"
logging.basicConfig(level=logging.NOTSET)

# install mss using `pip install mss`
# serialize module
with open(file, "w+") as f:
    paker.dump("mss", f, indent=4)

# now you can uninstall mss using `pip uninstall mss -y`
# load package back from dump file
with open(file, "r") as f:
    loader = paker.load(f)

import mss
with mss.mss() as sct:
    sct.shot()

# remove loader and clean the cache
loader.unload()

try:
    # this will throw error
    import mss
except ImportError:
    print("mss unloaded successfully!")

CLI

Paker can also work as a standalone script. To dump module to JSON dict use dump command:

paker dump mss

To recreate module from JSON dict use load:

paker load mss.json

Show all modules and packages in .json file

paker list mss.json

How it works

When importing modules or packages Python iterates over importers in sys.meta_path and calls find_module method on each object. If the importer returns self, it means that the module can be imported and None means that importer did not find searched package. If any importer has confirmed the ability to import module, Python executes another method on it - load_module. Paker implements its own importer called jsonimporter, which instead of searching for modules in directories, looks for them in Python dictionaries

To dump module or package to JSON document, Paker recursively iterates over modules and creates dict with code and type of each module and submodules if object is package.

You might also like...
An executor that loads ONNX models and embeds documents using the ONNX runtime.

ONNXEncoder An executor that loads ONNX models and embeds documents using the ONNX runtime. Usage via Docker image (recommended) from jina import Flow

Implementation of self-attention mechanisms for general purpose. Focused on computer vision modules. Ongoing repository.
Implementation of self-attention mechanisms for general purpose. Focused on computer vision modules. Ongoing repository.

Self-attention building blocks for computer vision applications in PyTorch Implementation of self attention mechanisms for computer vision in PyTorch

Turning SymPy expressions into PyTorch modules.

sympytorch A micro-library as a convenience for turning SymPy expressions into PyTorch Modules. All SymPy floats become trainable parameters. All SymP

DI-HPC is an acceleration operator component for general algorithm modules in reinforcement learning algorithms

DI-HPC: Decision Intelligence - High Performance Computation DI-HPC is an acceleration operator component for general algorithm modules in reinforceme

Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules
Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules

DCSR: Dual Camera Super-Resolution Implementation for our ICCV 2021 oral paper: Dual-Camera Super-Resolution with Aligned Attention Modules paper | pr

Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules
Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules

DCSR: Dual Camera Super-Resolution Implementation for our ICCV 2021 oral paper: Dual-Camera Super-Resolution with Aligned Attention Modules paper | pr

Weight initialization schemes for PyTorch nn.Modules

nninit Weight initialization schemes for PyTorch nn.Modules. This is a port of the popular nninit for Torch7 by @kaixhin. ##Update This repo has been

Pytorch modules for paralel models with same architecture. Ideal for multi agent-based systems
Pytorch modules for paralel models with same architecture. Ideal for multi agent-based systems

WideLinears Pytorch parallel Neural Networks A package of pytorch modules for fast paralellization of separate deep neural networks. Ideal for agent-b

Stacs-ci - A set of modules to enable integration of STACS with commonly used CI / CD systems
Stacs-ci - A set of modules to enable integration of STACS with commonly used CI / CD systems

Static Token And Credential Scanner CI Integrations What is it? STACS is a YARA

Comments
  • psutil example exits with module not found when using _memimporter

    psutil example exits with module not found when using _memimporter

    I pulled latest releases zip file, ran python setup.py build and attempted to run the psutil example with the compiled pyd. This resulted in the following error:

    DEBUG:jsonimporter:searching for pwd
    DEBUG:jsonimporter:searching for psutil._common
    INFO:jsonimporter:psutil._common has been imported successfully
    DEBUG:jsonimporter:searching for psutil._compat
    INFO:jsonimporter:psutil._compat has been imported successfully
    DEBUG:jsonimporter:searching for psutil._pswindows
    DEBUG:jsonimporter:searching for psutil._psutil_windows
    DEBUG:jsonimporter:searching for psutil._psutil_windows
    INFO:jsonimporter:using _memimporter to load '.pyd' file
    INFO:jsonimporter:unloaded all modules
    Traceback (most recent call last):
      File "c:\Users\User\Desktop\paker-0.7.1\paker-0.7.1\build\lib.win-amd64-cpython-310\psutil_example.py", line 20, in <module>
        import psutil
      File "c:\Users\User\Desktop\paker-0.7.1\paker-0.7.1\build\lib.win-amd64-cpython-310\paker\importers\jsonimporter.py", line 115, in load_module
        exec(jsonmod["code"], mod.__dict__)
      File "<string>", line 107, in <module>
      File "c:\Users\User\Desktop\paker-0.7.1\paker-0.7.1\build\lib.win-amd64-cpython-310\paker\importers\jsonimporter.py", line 115, in load_module
        exec(jsonmod["code"], mod.__dict__)
      File "<string>", line 35, in <module>
      File "c:\Users\User\Desktop\paker-0.7.1\paker-0.7.1\build\lib.win-amd64-cpython-310\paker\importers\jsonimporter.py", line 134, in load_module
        mod = _memimporter.import_module(fullname, path, initname, self._get_data, spec)
    ImportError: MemoryLoadLibrary failed loading psutil\_psutil_windows.pyd: The specified module could not be found. (126)
    

    Is this an issue with how I compiled memimporter, or something else?

    opened by rkbennett 1
Releases(v0.7.1)
Owner
Wojciech Wentland
Wojciech Wentland
Chainer Implementation of Semantic Segmentation using Adversarial Networks

Semantic Segmentation using Adversarial Networks Requirements Chainer (1.23.0) Differences Use of FCN-VGG16 instead of Dilated8 as Segmentor. Caution

Taiki Oyama 99 Jun 28, 2022
the code for our CVPR 2021 paper Bilateral Grid Learning for Stereo Matching Network [BGNet]

BGNet This repository contains the code for our CVPR 2021 paper Bilateral Grid Learning for Stereo Matching Network [BGNet] Environment Python 3.6.* C

3DCV developer 87 Nov 29, 2022
Differentiable Neural Computers, Sparse Access Memory and Sparse Differentiable Neural Computers, for Pytorch

Differentiable Neural Computers and family, for Pytorch Includes: Differentiable Neural Computers (DNC) Sparse Access Memory (SAM) Sparse Differentiab

ixaxaar 302 Dec 14, 2022
This repository contains part of the code used to make the images visible in the article "How does an AI Imagine the Universe?" published on Towards Data Science.

Generative Adversarial Network - Generating Universe This repository contains part of the code used to make the images visible in the article "How doe

Davide Coccomini 9 Dec 18, 2022
Self-supervised learning algorithms provide a way to train Deep Neural Networks in an unsupervised way using contrastive losses

Self-supervised learning Self-supervised learning algorithms provide a way to train Deep Neural Networks in an unsupervised way using contrastive loss

Arijit Das 2 Mar 26, 2022
Real-Time Seizure Detection using EEG: A Comprehensive Comparison of Recent Approaches under a Realistic Setting

Real-Time Seizure Detection using Electroencephalogram (EEG) This is the repository for "Real-Time Seizure Detection using EEG: A Comprehensive Compar

AITRICS 30 Dec 17, 2022
Code for "On Memorization in Probabilistic Deep Generative Models"

On Memorization in Probabilistic Deep Generative Models This repository contains the code necessary to reproduce the experiments in On Memorization in

The Alan Turing Institute 3 Jun 09, 2022
Solver for Large-Scale Rank-One Semidefinite Relaxations

STRIDE: spectrahedral proximal gradient descent along vertices A Solver for Large-Scale Rank-One Semidefinite Relaxations About STRIDE is designed for

48 Dec 20, 2022
This is the pytorch implementation for the paper: *Learning Accurate Performance Predictors for Ultrafast Automated Model Compression*, which is in submission to TPAMI

SeerNet This is the pytorch implementation for the paper: Learning Accurate Performance Predictors for Ultrafast Automated Model Compression, which is

3 May 01, 2022
U-2-Net: U Square Net - Modified for paired image training of style transfer

U2-Net: U Square Net Modified for paired image training of style transfer This is an unofficial repo making use of the code which was made available b

Doron Adler 43 Oct 03, 2022
PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

Amin Rezaei 157 Dec 11, 2022
A variational Bayesian method for similarity learning in non-rigid image registration (CVPR 2022)

A variational Bayesian method for similarity learning in non-rigid image registration We provide the source code and the trained models used in the re

daniel grzech 14 Nov 21, 2022
SpinalNet: Deep Neural Network with Gradual Input

SpinalNet: Deep Neural Network with Gradual Input This repository contains scripts for training different variations of the SpinalNet and its counterp

H M Dipu Kabir 142 Dec 30, 2022
CBREN: Convolutional Neural Networks for Constant Bit Rate Video Quality Enhancement

CBREN This is the Pytorch implementation for our IEEE TCSVT paper : CBREN: Convolutional Neural Networks for Constant Bit Rate Video Quality Enhanceme

Zhao Hengrun 3 Nov 04, 2022
This is a model to classify Vietnamese sign language using Motion history image (MHI) algorithm and CNN.

Vietnamese sign lagnuage recognition using MHI and CNN This is a model to classify Vietnamese sign language using Motion history image (MHI) algorithm

Phat Pham 3 Feb 24, 2022
An addernet CUDA version

Training addernet accelerated by CUDA Usage cd adder_cuda python setup.py install cd .. python main.py Environment pytorch 1.10.0 CUDA 11.3 benchmark

LingXY 4 Jun 20, 2022
Pretty Tensor - Fluent Neural Networks in TensorFlow

Pretty Tensor provides a high level builder API for TensorFlow. It provides thin wrappers on Tensors so that you can easily build multi-layer neural networks.

Google 1.2k Dec 29, 2022
Adversarial-autoencoders - Tensorflow implementation of Adversarial Autoencoders

Adversarial Autoencoders (AAE) Tensorflow implementation of Adversarial Autoencoders (ICLR 2016) Similar to variational autoencoder (VAE), AAE imposes

Qian Ge 236 Nov 13, 2022
Using VapourSynth with super resolution models and speeding them up with TensorRT.

VSGAN-tensorrt-docker Using image super resolution models with vapoursynth and speeding them up with TensorRT. Using NVIDIA/Torch-TensorRT combined wi

111 Jan 05, 2023
Low-code/No-code approach for deep learning inference on devices

EzEdgeAI A concept project that uses a low-code/no-code approach to implement deep learning inference on devices. It provides a componentized framewor

On-Device AI Co., Ltd. 7 Apr 05, 2022