A Python wrapper around the libmemcached interface from TangentOrg.

Related tags

Cachingpylibmc
Overview

pylibmc is a Python client for memcached written in C.

See the documentation at sendapatch.se/projects/pylibmc/ for more information.

https://travis-ci.org/lericson/pylibmc.png?branch=master

New in version 1.6.0

Though no major feature overhauls have taken place, this release is partially incompatible with 1.5.0. This stems from the fact that python-memcached is now using a flag that pylibmc has been using for some years. python-memcached uses it for a different purpose, and an incompatible one. We deemed that it would be better to support this interoperability. The change also means that Unicode strings are now stored as UTF-8 rather than pickled, which may or may not result in a slight performance improvement for this type of data.

We have also introduced a pickle_protocol behavior to enable seamless interoperability between Python 2.x and 3.x. Also, this release introduces a ManyLinux wheel, making installation a breeze on ManyLinux systems (which I suppose is many linuxes.)

New in version 1.5.0

This release fixes critical memory leaks in common code paths introduced in 1.4.2. Also fixes a critical bug in a corner of the zlib inflation code, where prior memory errors would trigger a double free. Thank you to everybody involved in the making of this release, and especially Eau de Web, without their contributions, this release and the bug fixes it contains wouldn't have been so expedient.

New in version 1.4.0

Brace yourself, Python 3.x support has come!

Thanks to everybody involved in this project; this release involves less authors but a lot more work per person. Thanks especially to Harvey Falcic for the work he put in, without which there wouldn't be any Python 3.x support. Also thanks to Sergey Pashinin for the initial stab at the problem.

Other than that, we had miscellaneous bug fixes, testing improvements, and documentation updates.

Last but not least I would like to ask for your support in this project, either by helping out with development, testing, documentation or anything at all; or simply by donating some magic internet money to the project's Bitcoin address 12dveKhqiJWCY8zXT4kaHdHELXPeGAUo9h.

License

Released under the BSD 3-clause license; see LICENSE for details.

Maintainer


http://www.smbc-comics.com/comics/20110908.gif
Comments
  • Enable zlib compression with Memcached check and set

    Enable zlib compression with Memcached check and set

    I was trying to enable zlib compression with memcached check-and-set and it appears that this is functionality that hasn't yet been built out: https://github.com/lericson/pylibmc/blob/78138d33c4156111294269a2a8f0cfcc66ac5c5c/src/_pylibmcmodule.c#L1041-L1043

    I'm not an expert in C but would be happy to try to create a PR if it would be welcomed and useful!

    opened by YPCrumble 1
  • please add to the documentation behaviors section some information regarding: tcp_keepalive, retry_timeout.

    please add to the documentation behaviors section some information regarding: tcp_keepalive, retry_timeout.

    in the documentation there is no mention of the behavior "tcp_keepalive". this behavior is important and necessary when running in a kubernetes or docker-compose scenario because the connection goes through something that monitors for dead connections and the thing that monitors for dead connections has a timeout of like 15 minutes or something.

    so i set the memcached behavior "tcp_keepalive" to 300 to ensure that my connecitons do not get cut by the networking thing.

    for more better description of the "networking thing" see https://github.com/zulip/zulip/pull/12711 which describes some about pika and also mentions the pylibmc.

    my request: please add to the documentation some information regarding the "tcp_keepalive".

    also i noticed there was missing a mention of "retry_timeout". please add that to the documentation too. (if you do not set this value to something like 5... does the client just error out on the first error?)


    for the tcp keepalive to work you also need to set the "tcp_keepidle" value. unfortunately your python code does not expose this feature from the underlying c library. so for now i have a patch that i wrote.

    diff --git a/src/_pylibmcmodule.h b/src/_pylibmcmodule.h
    index cee9772..36a28bb 100644
    --- a/src/_pylibmcmodule.h
    +++ b/src/_pylibmcmodule.h
    @@ -202,6 +202,7 @@ static PylibMC_Behavior PylibMC_behaviors[] = {
         { MEMCACHED_BEHAVIOR_TCP_NODELAY, "tcp_nodelay" },
     #if LIBMEMCACHED_VERSION_HEX >= 0x00044000
         { MEMCACHED_BEHAVIOR_TCP_KEEPALIVE, "tcp_keepalive" },
    +    { MEMCACHED_BEHAVIOR_TCP_KEEPIDLE, "tcp_keepidle" },
     #endif
         { MEMCACHED_BEHAVIOR_HASH, "hash" },
         { MEMCACHED_BEHAVIOR_KETAMA_HASH, "ketama_hash" },
    

    and i do a pip download --no-binary=pylibmc pylibmc. extract, cd, patch, build, install.

    tcp_keepalive is boolean value. see https://man7.org/linux/man-pages/man7/socket.7.html section SO_KEEPALIVE.

    tcp_keepidle is integer time in seconds. see https://man7.org/linux/man-pages/man7/tcp.7.html section on TCP_KEEPIDLE.

    opened by trevorboydsmith 0
  • bools stored on 1.5.2 come out as strings that evaluate to True on 1.6.1

    bools stored on 1.5.2 come out as strings that evaluate to True on 1.6.1

    I believe this commit results in bools written with 1.5.2 to strings in 1.6.1, which causes the value to actually switch if you stored False. This makes upgrading from 1.5.2 to 1.6.1 (and getting the pickle_protocol behavior, thank you for adding that!) difficult. Unfortunately tracking down all the places we might be setting False is also difficult (and tedious). Any ideas about how we can work around this? The only thing I've come up with is forking 1.6 to have the write path set an extra flag for anything with FLAG_TEXT, and then checking for that extra flag on read, monitoring what fraction of our reads come back without that flag, and hoping that eventually it goes to 0.

    (pylibmctest) ✔ 12:16 ~ $ python
    Python 3.7.3 | packaged by conda-forge | (default, Dec  6 2019, 08:36:57)
    [Clang 9.0.0 (tags/RELEASE_900/final)] :: Anaconda, Inc. on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pylibmc
    >>> cl = pylibmc.Client(['localhost:11211'])
    >>> cl.set('foo', False)
    True
    >>> cl.get('foo')
    False
    >>>
    (pylibmctest) ✔ 12:44 ~ $ pip freeze | grep pylibmc
    pylibmc==1.5.2
    (pylibmctest) ✔ 12:45 ~ $ pip install pylibmc==1.6.1
    Collecting pylibmc==1.6.1
      Using cached https://files.pythonhosted.org/packages/a7/0c/f7a3af34b05c167a69ed1fc330b06b658dac4ab25b8632c52d1022dd5337/pylibmc-1.6.1.tar.gz
    Building wheels for collected packages: pylibmc
      Building wheel for pylibmc (setup.py) ... done
      Created wheel for pylibmc: filename=pylibmc-1.6.1-cp37-cp37m-macosx_10_9_x86_64.whl size=32878 sha256=5bde92da27513e7024d29f73dd0a9837ebb4c7024d2b553eac62db835d504fdf
      Stored in directory: /Users/aaronwebber/Library/Caches/pip/wheels/71/5e/41/6796b369874b1b0345bccf31449e162eadcda85b104922f2de
    Successfully built pylibmc
    Installing collected packages: pylibmc
      Found existing installation: pylibmc 1.5.2
        Uninstalling pylibmc-1.5.2:
          Successfully uninstalled pylibmc-1.5.2
    Successfully installed pylibmc-1.6.1
    (pylibmctest) ✔ 12:45 ~ $ python
    Python 3.7.3 | packaged by conda-forge | (default, Dec  6 2019, 08:36:57)
    [Clang 9.0.0 (tags/RELEASE_900/final)] :: Anaconda, Inc. on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pylibmc
    >>> cl = pylibmc.Client(['localhost:11211'])
    >>> cl.get('foo')
    '0'
    >>> bool(cl.get('foo'))    # This is the real problem  :(
    True
    
    opened by sabw8217 0
  • NumPy arrays and instances of datetime, date and time pickled in Python 2 cannot be deserialized in Python 3

    NumPy arrays and instances of datetime, date and time pickled in Python 2 cannot be deserialized in Python 3

    I realize that I am very late with the python3 migration, so I understand if this ticket is closed right away. However others running into the same issue can hopefully benefit from this report.

    According to the docs for pickle inPython 3:

    Using encoding='latin1' is required for unpickling NumPy arrays and instances of datetime, date and time pickled by Python 2.

    However the default encoding for unpickling is US-ASCII, thus one cannot deserialize datetime, date and time objects as well as NumPy arrays if they have been serialized in Python 2. Many migrate from Python 2 to Python 3 using a path were both versions run in parallel for some time.

    Allowing to configure the encoding used for pickle.load()/_PylibMC_pickle_loads would resolve the issue. I also added a workaround below.

    Environment

    pylibmc==1.6.1 memcached 1.6.6 python 2.7.18 and 3.6.11

    Steps to reproduce

    In the Python 2 environment:

    import sys
    import pylibmc
    from pickle import load as pickle_load
    from io import BytesIO
    from datetime import datetime
                
    
    client = pylibmc.Client(["localhost:11211"], behaviors={"pickle_protocol": 2}, binary=True)
    
    client.set("testText", "asdf")
    client.set("testDict", {"a": 123})
    client.set("testDatetime", datetime.now())
    

    In the Python 3 environment:

    import sys
    import pylibmc
    from pickle import load as pickle_load
    from io import BytesIO
    from datetime import datetime
                
    
    client = pylibmc.Client(["localhost:11211"], behaviors={"pickle_protocol": 2}, binary=True)
    
    # this works
    client.get("testText")
    client.get("testDict")
    
    # this fails with: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)
    client.get("testDatetime")
    

    Expected behaviour

    The datetime object (as well as the others) is correctly retrieved from the cache

    Actual Behaviour

    The datetime object cannot be deserialized

    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)

    Workaround

    import sys
    import pylibmc
    from pickle import load as pickle_load
    from io import BytesIO
    
    class MemcachedClient(pylibmc.Client):
    
         if sys.version_info.major >= 3:        
            PYLIBMC_FLAG_PICKLE = 1
         
            def deserialize(self, value, flags):
                if flags & self.PYLIBMC_FLAG_PICKLE:
                    return pickle_load(BytesIO(value), encoding=u"latin1")
                return super(MemcachedClient, self).deserialize(value, flags)
                
    
    client = MemcachedClient(["localhost:11211"], behaviors={"pickle_protocol": 2}, binary=True)
    client.get("testText")
    client.get("testDict")
    client.get("testDatetime")
    
    
    opened by LaughInJar 1
  • Iterator over multi get

    Iterator over multi get

    An idea left as a comment in the code is to provide an iterable interface for multi getting, for example

    mc = …
    
    for key, val in mc.iter_multi_get(set_of_keys):
        use(key, val)
    

    However, it is not clear if this would be a good idea as it may incur a very large overhead, switching between Python and C.

    feature 
    opened by lericson 0
Releases(1.6.3)
Owner
Ludvig Ericson
Software engineer and PhD student specializing in robotics
Ludvig Ericson
WSGI middleware for sessions and caching

Cache and Session Library About Beaker is a web session and general caching library that includes WSGI middleware for use in web applications. As a ge

Ben Bangert 500 Dec 29, 2022
RecRoom Library Cache Tool

RecRoom Library Cache Tool A handy tool to deal with the Library cache file. Features Parse Library cache Remove Library cache Parsing The script pars

Jesse 5 Jul 09, 2022
Automatic Flask cache configuration on Heroku.

flask-heroku-cacheify Automatic Flask cache configuration on Heroku. Purpose Configuring your cache on Heroku can be a time sink. There are lots of di

Randall Degges 39 Jun 05, 2022
Python disk-backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python.

DiskCache is an Apache2 licensed disk and file backed cache library, written in pure-Python, and compatible with Django.

Grant Jenks 1.7k Jan 05, 2023
An ORM cache for Django.

Django ORMCache A cache manager mixin that provides some caching of objects for the ORM. Installation / Setup / Usage TODO Testing Run the tests with:

Educreations, Inc 15 Nov 27, 2022
An implementation of memoization technique for Django

django-memoize django-memoize is an implementation of memoization technique for Django. You can think of it as a cache for function or method results.

Unhaggle 118 Dec 09, 2022
No effort, no worry, maximum performance.

Django Cachalot Caches your Django ORM queries and automatically invalidates them. Documentation: http://django-cachalot.readthedocs.io Table of Conte

NoriPyt 976 Dec 28, 2022
Peerix is a peer-to-peer binary cache for nix derivations

Peerix Peerix is a peer-to-peer binary cache for nix derivations. Every participating node can pull derivations from each other instances' respective

92 Dec 13, 2022
A caching extension for Flask

Flask-Caching Adds easy cache support to Flask. This is a fork of the Flask-Cache extension. Flask-Caching also includes the cache module from werkzeu

Peter Justin 774 Jan 02, 2023
CacheControl is a port of the caching algorithms in httplib2 for use with requests session object.

CacheControl CacheControl is a port of the caching algorithms in httplib2 for use with requests session object. It was written because httplib2's bett

Eric Larson 409 Dec 04, 2022
Aircache is an open-source caching and security solution that can be integrated with most decoupled apps that use REST APIs for communicating.

AirCache Aircache is an open-source caching and security solution that can be integrated with most decoupled apps that use REST APIs for communicating

AirCache 2 Dec 22, 2021
Simple caching transport for httpx

httpx-cache is yet another implementation/port is a port of the caching algorithms in httplib2 for use with httpx Transport object.

Ouail 28 Jan 01, 2023
Asynchronous cache manager designed for horizontally scaled web servers.

Introduction Asynchronous cache manager designed for horizontally scaled web applications. NOTE: Currently has implementation only for FastAPI using R

Serghei 23 Dec 01, 2022
A Redis cache backend for django

Redis Django Cache Backend A Redis cache backend for Django Docs can be found at http://django-redis-cache.readthedocs.org/en/latest/. Changelog 3.0.0

Sean Bleier 1k Dec 15, 2022
A Python wrapper around the libmemcached interface from TangentOrg.

pylibmc is a Python client for memcached written in C. See the documentation at sendapatch.se/projects/pylibmc/ for more information. New in version 1

Ludvig Ericson 458 Dec 30, 2022
A caching extension for Flask

A fork of the Flask-cache extension which adds easy cache support to Flask.

Pallets Community 773 Jan 08, 2023
A slick ORM cache with automatic granular event-driven invalidation.

Cacheops A slick app that supports automatic or manual queryset caching and automatic granular event-driven invalidation. It uses redis as backend for

Alexander Schepanovski 1.7k Dec 30, 2022
Automatic caching and invalidation for Django models through the ORM.

Cache Machine Cache Machine provides automatic caching and invalidation for Django models through the ORM. For full docs, see https://cache-machine.re

846 Nov 26, 2022
Asyncio cache manager for redis, memcached and memory

aiocache Asyncio cache supporting multiple backends (memory, redis and memcached). This library aims for simplicity over specialization. All caches co

aio-libs 764 Jan 02, 2023
A decorator for caching properties in classes.

cached-property A decorator for caching properties in classes. Why? Makes caching of time or computational expensive properties quick and easy. Becaus

Daniel Roy Greenfeld 658 Dec 01, 2022