Python package for reference counting native pointers

Overview

refcount

license status Documentation Status master: Build status - master testing: Build status - devel

Reference counted native handles

This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boils down to "simply" maintaining a set of counters, it is deceptively complicated to do so properly and not end up with memory leaks or crashes. This package offers structured options for reliably managing external native resources. Surprisingly I could not locate an existing package doing just what I needed. Other use cases requiring reference counting, aside from native library resources, may benefit from reusing and extending classes in refcount.

refcount from version 0.7 includes classes using cffi. Other low-level interoperability mechanisms may well be added in the future.

License

MIT (see License.txt)

Documentation

Hosted at pyrefcount.readthedocs.io

Source code

The code repository is on GitHub.

Installation

pip install refcount

From source:

pip install -r requirements.txt
python setup.py install

Sample use

The following example is based on one of the unit tests.

Say we have a C++ library with objects and a C API:

#define TEST_DOG_PTR  testnative::dog*
#define TEST_OWNER_PTR  testnative::owner*
#define TEST_COUNTED_PTR  testnative::reference_counter*

testnative::dog* create_dog();
testnative::owner* create_owner(testnative::dog* d);
void say_walk(testnative::owner* owner);
void release(testnative::reference_counter* obj);
// etc.

From the outside of the library the API is exported with opaque pointers void* (C structs pointers and native C99 types could be handled too).

void* create_dog();
void* create_owner(void* d);
void say_walk(void* owner);
void release(void* obj);
// etc.

Starting with the end in mind, from Python we want an API hiding the low level details close to the C API, in particular avoiding managing native memory via release C API calls, piggybacking the python GC instead.

dog = Dog()
owner = DogOwner(dog)
owner.say_walk()
print(dog.position)
dog = None # the "native dog" is still alive though, as the owner incremented the ref count
owner = None

This is doable with refcount and the cffi package. One possible design is:

ut_ffi = cffi.FFI()

ut_ffi.cdef('extern void* create_dog();')
ut_ffi.cdef('extern void* create_owner( void* d);')
ut_ffi.cdef('extern void say_walk( void* owner);')
ut_ffi.cdef('extern void release( void* obj);')
# etc.

ut_dll = ut_ffi.dlopen('c:/path/to/test_native_library.dll', 1) # Lazy loading

class CustomCffiNativeHandle(CffiNativeHandle):
    def __init__(self, pointer, prior_ref_count = 0):
        super(CustomCffiNativeHandle, self).__init__(pointer, type_id='', prior_ref_count = prior_ref_count)

    def _release_handle(self) -> bool:
        ut_dll.release(self.get_handle())
        return True

class Dog(CustomCffiNativeHandle):
    def __init__(self, pointer = None):
        if pointer is None:
            pointer = ut_dll.create_dog()
        super(Dog, self).__init__(pointer)
    # etc.

class DogOwner(CustomCffiNativeHandle):

    def __init__(self, dog):
        super(DogOwner, self).__init__(None)
        self._set_handle(ut_dll.create_owner(dog.get_handle()))
        self.dog = dog
        self.dog.add_ref() # Do note this important reference increment

    def say_walk(self):
        ut_dll.say_walk(self.get_handle())

    def _release_handle(self) -> bool:
        super(DogOwner, self)._release_handle()
        # super(DogOwner, self)._release_handle()
        self.dog.release()
        return True

Related work

Ancestry, acknowledgements

This python package refcount actually spawned from prior work for interoperability between C++, R and .NET (R.NET)

refcount features using cffi were also significantly informed by Kevin Plastow's work while he was at the Australian Bureau of Meteorology; this contribution is gratefully acknowledged.

In you have native interop needs you may also want to look at:

Other python packages

refcount was created in part because no existing prior (Python) work could quite fit the need. There are however packages that may better address your particular need:

You might also like...
A test repository to build a python package and publish the package to Artifact Registry using GCB

A test repository to build a python package and publish the package to Artifact Registry using GCB. Then have the package be a dependency in a GCF function.

PSP (Python Starter Package) is meant for those who want to start coding in python but are new to the coding scene.

Python Starter Package PSP (Python Starter Package) is meant for those who want to start coding in python, but are new to the coding scene. We include

PyPIContents is an application that generates a Module Index from the Python Package Index (PyPI) and also from various versions of the Python Standard Library.

PyPIContents is an application that generates a Module Index from the Python Package Index (PyPI) and also from various versions of the Python Standar

Example python package with pybind11 cpp extension

Developing C++ extension in Python using pybind11 This is a summary of the commands used in the tutorial.

qecsim is a Python 3 package for simulating quantum error correction using stabilizer codes.

qecsim qecsim is a Python 3 package for simulating quantum error correction using stabilizer codes.

A command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, VueJS projects.
A command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, VueJS projects.

Cookiecutter A command-line utility that creates projects from cookiecutters (project templates), e.g. creating a Python package project from a Python

A toy repo illustrating a minimal installable Python package

MyToy: a minimal Python package This repository contains a minimal, toy Python package with a few files as illustration for students of how to lay out

Python package that mirrors the original Nodejs ReplAPI-It.

Python-ReplAPI-It Python package that mirrors the original Nodejs ReplAPI-It. Contributing First fork the repo: $ git clone https://github.com/ReplAPI

This project recreates the R-based RCy3 Cytoscape Automation library as a Python package.

Python library for calling Cytoscape Automation via CyREST

Comments
  • Roadmap for reference counting around pybind11

    Roadmap for reference counting around pybind11

    opened by jmp75 1
  • CFFI wrappers helper functions for use in codegen

    CFFI wrappers helper functions for use in codegen

    Basically, need to write the equivalent of wrapping functions in an the R package cinterop. While a cinterop Python package will be warranted, the refcount wrapping/unwrapping helper functions should probably live in refcount.

    enhancement 
    opened by jmp75 1
  • Bump certifi from 2022.6.15 to 2022.12.7

    Bump certifi from 2022.6.15 to 2022.12.7

    Bumps certifi from 2022.6.15 to 2022.12.7.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
Releases(v1.1.0)
  • v1.1.0(Aug 19, 2022)

    Main changes

    • Expand some features to cater for macos
    • Tidy up and reengineer some of the legacy functions in the platform utilities putils to facilitate library loading. Minor breaking changes, but probably for no-one but the author in effect.
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Aug 13, 2022)

    Main changes

    • Improve documentation, and use the mkdocs-material theme
    • mkdocs.yml changes. Enable dark/light modes from mkdocs-material
    • Improved type hints, and add static type checks (mypy)
    • Improve unit tests and add unit test code coverage reporting
    • Manage the package using poetry; phase out setuptools.
    • Apply some of the approaches in https://py-pkgs.org.
    • Restore appveyor CI to a working state
    Source code(tar.gz)
    Source code(zip)
  • 0.9.3(Aug 13, 2022)

Owner
CSIRO Hydroinformatics
CSIRO - hydroinformatics repositories
CSIRO Hydroinformatics
Keyboard Layout Change - Extension for Ulauncher

Keyboard Layout Change - Extension for Ulauncher

Marco Borchi 4 Aug 26, 2022
A general purpose low level programming language written in Python.

A general purpose low level programming language written in Python. Basal is an easy mid level programming language compiling to C. It has an easy syntax, similar to Python, Rust etc.

Snm Logic 6 Mar 30, 2022
An easy way to access the Scratch API!

The majority of people are likely here because they want to easily access the Scratch API!

rgantzos 0 May 04, 2022
XAC HID Gamepad implementation for CircuitPython 7 or above.

CircuitPython_XAC_Gamepad Setup process Install CircuitPython 7 or above in your board. Add the init.py file under \lib\adafruit_hid directory of CIRC

5 Dec 19, 2022
A web-based chat application that enables multiple users to interact with one another

A web-based chat application that enables multiple users to interact with one another, in the same chat room or different ones according to their choosing.

3 Apr 22, 2022
From "fixed RAnDom CRashes" to "[FIX] Fixed random crashes."

Clean Commit From fixed RAnDom CRashes to [FIX] Fixed random crashes. Clean commit helps you by auto-formating your commits to make your repos better

Mathias 3 Dec 26, 2021
A(Sync) Interface for internal Audible API written in pure Python.

Audible Audible is a Python low-level interface to communicate with the non-publicly Audible API. It enables Python developers to create there own Aud

mkb79 192 Jan 03, 2023
A frontend to ease the use of pulseaudio's routing capabilities, mimicking voicemeeter's workflow

Pulsemeeter A frontend to ease the use of pulseaudio's routing capabilities, mimicking voicemeeter's workflow Features Create virtual inputs and outpu

Gabriel Carneiro 164 Jan 04, 2023
A general-purpose wallet generator, for supported coins only

2gen A general-purpose generator for keys. Designed for all cryptocurrencies supporting the Bitcoin format of keys and addresses. Functions To enable

Vlad Usatii 1 Jan 12, 2022
a url shortener with fastapi and tortoise-orm

fastapi-tortoise-orm-url-shortener a url shortener with fastapi and tortoise-orm

19 Aug 12, 2022
an opensourced roblox group finder writen in python 100% free and virus-free

Roblox-Group-Finder an opensourced roblox group finder writen in python 100% free and virus-free note : if you don't want install python or just use w

mollomm1 1 Nov 11, 2021
A Way to Use Python, Easier.

PyTools A Way to Use Python, Easier. How to Install Just copy this code, then make a new file in your project directory called PyTools.py, then paste

Kamran 2 Aug 15, 2022
Dev-meme - A repository that contains memes just for people like us

A repository that contains memes just for people like us. Coders are constantly

Padmashree Jha 4 Oct 31, 2022
A fancy and practical functional tools

Funcy A collection of fancy functional tools focused on practicality. Inspired by clojure, underscore and my own abstractions. Keep reading to get an

Alexander Schepanovski 2.9k Dec 29, 2022
A simple tool made in Python language

Simple tool Uma simples ferramenta feita 100% em linguagem Python 💻 Requisitos: Python3 instalado em seu dispositivo Clonagem e acesso 📳 git clone h

josh washington 4 Dec 07, 2021
Pampy: The Pattern Matching for Python you always dreamed of.

Pampy: Pattern Matching for Python Pampy is pretty small (150 lines), reasonably fast, and often makes your code more readable and hence easier to rea

Claudio Santini 3.5k Dec 30, 2022
Flask html response minifier

Flask-HTMLmin Minify flask text/html mime type responses. Just add MINIFY_HTML = True to your deployment config to minify HTML and text responses of y

Hamid Feizabadi 85 Dec 07, 2022
Birthday program - A program that lookups a birthday txt file and compares to the current date to check for birthdays

Birthday Program This is a program that lookups a birthday txt file and compares

Daquiver 4 Feb 02, 2022
Batch Python Program Verify

Batch Python Program Verify About As a TA(teaching assistant) of Programming Class, it is very annoying to test students' homework assignments one by

Han-Wei Li 7 Dec 20, 2022
A totally unrealistic cell growth/reproduction simulation.

A totally unrealistic cell growth/reproduction simulation.

Andrien Wiandyano 1 Oct 24, 2021