🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.

Overview

Boltons

boltons should be builtins.

Boltons is a set of over 230 BSD-licensed, pure-Python utilities in the same spirit as — and yet conspicuously missing from — the standard library, including:

Full and extensive docs are available on Read The Docs. See what's new by checking the CHANGELOG.

Boltons is tested against Python 2.6, 2.7, 3.4, 3.5, 3.6, 3.7, 3.8 and 3.9, as well as CPython nightly and PyPy/PyPy3.

Installation

Boltons can be added to a project in a few ways. There's the obvious one:

pip install boltons

On macOS, it can also be installed via MacPorts:

sudo port install py-boltons

Then, thanks to PyPI, dozens of boltons are just an import away:

from boltons.cacheutils import LRU
my_cache = LRU()

However, due to the nature of utilities, application developers might want to consider other options, including vendorization of individual modules into a project. Boltons is pure-Python and has no dependencies. If the whole project is too big, each module is independent, and can be copied directly into a project. See the Integration section of the docs for more details.

Third-party packages

The majority of boltons strive to be "good enough" for a wide range of basic uses, leaving advanced use cases to Python's myriad specialized 3rd-party libraries. In many cases the respective boltons module will describe 3rd-party alternatives worth investigating when use cases outgrow boltons. If you've found a natural "next-step" library worth mentioning, see the next section!

Gaps

Found something missing in the standard library that should be in boltons? Found something missing in boltons? First, take a moment to read the very brief architecture statement to make sure the functionality would be a good fit.

Then, if you are very motivated, submit a Pull Request. Otherwise, submit a short feature request on the Issues page, and we will figure something out.

Issues
  • Add ioutils module

    Add ioutils module

    Spooled Temporary Files are file-like objects that start out mapped to in-memory objects, but automatically roll over to a temporary file once they reach a certain (configurable) threshhold. Unfortunately the built-in SpooledTemporaryFile class in Python does not implement the exact API that some common classes like StringIO do. SpooledTemporaryFile also spools all of it's in-memory files as cStringIO instances. cStringIO instances cannot be deep-copied, and they don't work with the zip library either. This along with the incompatible api makes it useless for several use-cases.

    To combat this but still gain the memory savings and usefulness of a true spooled file-like-object, two custom classes have been implemented which have a compatible API.

    SpooledBytesIO is a spooled file-like-object that only accepts bytes. On Python 2.x this means the 'str' type; on Python 3.x this means the 'bytes' type. Bytes are written in and retrieved exactly as given, but it will raise TypeErrors if something other than bytes are written.

    SpooledStringIO is a spooled file-like-object that only accepts unicode values. On Python 2.x this means the 'unicode' type and on Python 3.x this means the 'str' type. Values are accepted as unicode and then coerced into utf-8 encoded bytes for storage. On retrieval, the values are returned as unicode.

    opened by induane 20
  • Functions for time since UNIX epoch

    Functions for time since UNIX epoch

    I've always thought it was weird these were absent from the standard library, even though they're so simple. I added two functions to timeutils.py to transform between seconds since the UNIX epoch and datetime objects. Currently, they ignore any timezone information, but if you think these functions are a good fit for the package, I might be able to work on adding that kind of functionality.

    opened by tsupinie 18
  • Non-decorator version of funcutils.wraps and option to hide wrapped function.

    Non-decorator version of funcutils.wraps and option to hide wrapped function.

    Hi!

    As as discussed in #242, here is my suggestion to add an edge case to the usage of funcutils.wraps. The situation is that if the wrapped and wrapper function have the same arguments but differ as to which are keyword-only or positional-only, the function returned by wraps will fail as the wrong "invocation string" was inserted in the process. A solution to this is to create the internal FunctionBuilderinstance from the wrapper instead of the wrapped. As it didn't make sense to simply add an argument to wraps, I decided to replicate the structure of the built-in functools and add a update_wrapper function, of which wraps is only a partial call.

    This solves a problem that originated from using partials. Code that now works to wrap those is as follow:

    import functools
    from boltons import funcutils
    
    def func(a, b=1, c=1):  # Has the signature "a, b=1, c=1"
        return a, b, c
    
    wrapper = partial(func, b=2)
    functools.update_wrapper(wapper, func)  # Needed as FunctionBuilder will look for some attributes like __name__ that are not updated directly with `partial`.
    # wrapper has the signature; "a, *, b=2, c=1"
    
    new_func = funcutils.update_wrapper(wrapper, func, build_from=wrapper, injected='b')
    # new_func now has the signature : "a, *, c=1"
    new_func(1)  # Prints : 1, 2, 1, 
    

    Also, another small improvement I made here is to add an option to completely hide the wrapped function in the new one returned. Most IDEs use inspect.signature to extract the signature functions that they display to the user. However, by default, signature follows all wrapped functions until the innermost, so the user doesn't see the updated signature. One could use the follow_wrapped=False option, but I believe offering the possibility here could be nice.

    Closes #242

    opened by aulemahal 12
  • funcutils.wraps() with injected keyword does not work for keyword-only arguments.

    funcutils.wraps() with injected keyword does not work for keyword-only arguments.

    Given the following decorator:

    def inject_loop(func):
          sig = inspect.signature(func)
          loop_param = sig.parameters['loop'].replace(default=None)
          sig = sig.replace(parameters=[loop_param])
    
          def add_loop(
              args: typing.Tuple[typing.Any, ...],
              kwargs: typing.Dict[str, typing.Any]
          ) -> collections.OrderedDict:
              bargs = sig.bind(*args, **kwargs)
              bargs.apply_defaults()
              if bargs.arguments['loop'] is None:
                  bargs.arguments['loop'] = asyncio.get_event_loop()
    
              return bargs.arguments
    
          if inspect.isasyncgenfunction(func):
              async def async_gen_loop_wrapper(*args, **kwargs):
                  async for elem in func(**add_loop(args, kwargs)):
    >>                yield elem
              ret = async_gen_loop_wrapper
    
          elif inspect.iscoroutinefunction(func):
              async def async_loop_wrapper(*args, **kwargs):
                  return await func(**add_loop(args, kwargs))
              ret = async_loop_wrapper
    
          elif inspect.isgeneratorfunction(func):
              def gen_loop_wrapper(*args, **kwargs):
                  yield from func(**add_loop(args, kwargs))
              ret = gen_loop_wrapper
    
          else:
              def func_loop_wrapper(*args, **kwargs):
                  return func(**add_loop(args, kwargs))
              ret = func_loop_wrapper
    
          return boltons.funcutils.wraps(func, injected=['loop'])(ret)
    

    The following code fails:

    @inject_loop
    def example(*, loop):  # loop is a keyword-only argument
        return loop
    

    with the following stacktrace:

    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    .../python3.6/site-packages/boltons/funcutils.py in remove_arg(self, arg_name)
        565         try:
    --> 566             self.args.remove(arg_name)
        567         except ValueError:
    
    ValueError: list.remove(x): x not in list
    
    During handling of the above exception, another exception occurred:
    
    MissingArgument                           Traceback (most recent call last)
    <ipython-input-3-6ef3f4f5dc19> in <module>()
    ----> 1 @lib.inject_loop
          2 def example(*, loop):
          3     return loop
    
    ... in inject_loop(func)
        169         ret = func_loop_wrapper
        170
    --> 171     return boltons.funcutils.wraps(func, injected=['loop'])(ret)
        172
        173
    
    .../python3.6/site-packages/boltons/funcutils.py in wraps(func, injected, **kw)
        295     for arg in injected:
        296         try:
    --> 297             fb.remove_arg(arg)
        298         except MissingArgument:
        299             if inject_to_varkw and fb.varkw is not None:
    
    .../python3.6/site-packages/boltons/funcutils.py in remove_arg(self, arg_name)
        569                                   % (arg_name, self.name, self.args))
        570             exc.arg_name = arg_name
    --> 571             raise exc
        572         d_dict.pop(arg_name, None)
        573         self.defaults = tuple([d_dict[a] for a in self.args if a in d_dict])
    
    MissingArgument: arg 'loop' not found in example argument list: []
    

    However, it works if I do this:

    @inject_loop
    def example(loop):  # Not a keyword-only argument
        return loop
    

    I believe the solution should also check the kwonly args when attempting to mark a parameter as "injected", so that keyword only arguments can be listed in the injected parameter to wraps().

    opened by xlorepdarkhelm 12
  • Add FrozenDict type to dictutils

    Add FrozenDict type to dictutils

    This is another tool I have built a number of times in various permutations; I don't know if it's all that interesting but I've found it quite useful in a number of situations including immutable constant maps without heinous amounts of namedtuple boilerplate, and some cases with threading.

    A FrozenDict is a dictionary whose values are set during instance creation and are fixed from that point on. Setting and altering values is not allowed. This can be useful in a number of scenarios, including setting up mapping constants without worrying that the values will get mutated by a misbehaving function.

    There are a lot of online recipes for creating a FrozenDict class, but most still rely on a real dictionary under the hood for storage. They also tend to be extra weighty because they have an underlying object dict.

    One common solution is to use a named tuple, but this requires setting up boilerplate for every type or relying on factory functions. The FrozenDict utilizes a named tuple for storage and then is further made lighter by utilizing slots to eliminate the underlying object dict.

    opened by induane 12
  • Why IndexedSet not update the index of items?

    Why IndexedSet not update the index of items?

    What is the motivation of not updating the current index of items when they are removed? It's this a bug? Because i starting using IndexedSet to pop items from a list that i have and i start get index of range errors.

    https://github.com/mahmoud/boltons/blob/bf8a65942cd8078dba7dc45543ae0923fe2fabbc/boltons/setutils.py#L201-L209

    opened by Urahara 11
  • PermissionError with AtomicSaver on Windows

    PermissionError with AtomicSaver on Windows

    If I try this little code on Windows 7:

    from boltons.fileutils import AtomicSaver
    with AtomicSaver('foo.txt') as f:
        f.write('whatever')
    

    I get the following Exception:

    Traceback (most recent call last):
      File "C:\Python34\lib\site-packages\boltons\fileutils.py", line 272, in setup
        overwrite=self.overwrite_part)
      File "C:\Python34\lib\site-packages\boltons\fileutils.py", line 194, in _atomic_rename
        os.rename(path, new_path)
    PermissionError: [WinError 32] Der Prozess kann nicht auf die Datei zugreifen, da sie von einem ande
    ren Prozess verwendet wird: 'C:\\Windows\\System32\\tmphia0tzzm' -> 'foo.txt.part'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\Python34\lib\site-packages\boltons\fileutils.py", line 281, in __enter__
        self.setup()
      File "C:\Python34\lib\site-packages\boltons\fileutils.py", line 274, in setup
        os.unlink(tmp_part_path)
    PermissionError: [WinError 32] Der Prozess kann nicht auf die Datei zugreifen, da sie von einem ande
    ren Prozess verwendet wird: 'C:\\Windows\\System32\\tmphia0tzzm'
    
    opened by stlehmann 11
  • add new file procutils, for process-related utilities

    add new file procutils, for process-related utilities

    Add a more flexible interface to subprocess, in the style of subprocess.call() but better.

    I find subprocess.call(), check_call() and check_output() really badly lacking, but I end up using them anyway due to the considerable extra complexity in using my own Popen object.

    For example:

    • check_* does not report an exit code or stderr in the exception, so you may know "something failed" but it's hard to tell what or why.
    • *_call use the process's stdout, resulting in often unwanted extra output
    • with any of these calls, it is impossible to interact with stdin

    I have endeavoured to create a function that is very flexible but with excellent defaults, in the style of requests.get(), where the 90% use case is very simple, but the 99% use case is still possible. In getting into specifics (such as having None redirect output to /dev/null), I'm pretty sure I broke windows support. It's probably possible to add with minor changes in interface though.

    opened by ekimekim 10
  • strip() that works with iterables

    strip() that works with iterables

    I'd like strip / rstrip / lstrip that works with arbitrary iterables:

    def iter_lstrip(li, it): 
        pos = 0
        while li[pos] == it:
            pos += 1
            if pos == len(li):
                return []
        return li[pos:]
    
    
    def iter_rstrip(li, it):
        return list(reversed(iter_lstrip(list(reversed(li)), it)))
    
    
    def iter_strip(li, it):
        return iter_lstrip(iter_rstrip(li, it), it)
    
    iter_strip([0,0,0,1,2,3,4,0,0], 0)
    # [1,2,3,4]
    
    opened by kurtbrose 9
  • Fix issues with LRI Cache #155

    Fix issues with LRI Cache #155

    Adds failing tests to exercise issues with LRI

    To fix the issue, I have a trivial solution that has LRI inherit from LRU. After a couple hours of thinking about it I don't think there's a faster solution that using the linked list that LRU uses without introducing an unbounded memory size of the underlying datastructure stored.

    That being said LRU has a bunch of thread locking logic that I'm uncertain you would want to add to the LRI cache. So the current solution is more of a conversation piece. Open to feedback on how to go forward on this:

    1. Extract common logic from LRU and have LRI and LRU utilize that? (lets us omit thread safety from LRI if we want)
    2. Keep naive solution (though thread locking might introduce unwanted overhead).
    3. Maybe make thread safety an optional feature on both LRU and LRI. Would be tricky not to break existing features. Probably would require names like: LRU, ThreadUnsafeLRU, LRI, ThreadSafeLRI.
    opened by CameronCairns 8
  • complement set

    complement set

    the idea is complements of sets -- like inverse sets -- they support all of the set API with each other and with regular sets with the exception of iteration and len (b/c they contain "everything")

    I really badly wanted one of these to be able to pass in a "universal set" as a null value rather than None -- would have saved a bunch of code like

    for item in things:
        if filter is not None:
            if item not in filter:
                continue
    

    instead, complement(set()) would simply return True for everything (or everything that is hashable)

    the next day a co-worker wanted one of these for a different purpose, so I took a couple hours to bang out some code and the implementation actually seems pretty straight forward

    still need to implement the operator overloads, more docs and better tests

    opening the PR early to allow for feedback ASAP

    opened by kurtbrose 8
  • Feature request: intersection and union of lists of ranges

    Feature request: intersection and union of lists of ranges

    Let a range be a pair of sortable objects, e.g. a tuple of numbers, strings, date objects or anything implementing the __lt__ family operators (or maybe a sorting key function should be accepted in the mood of the sorted builtin!)

    Given an iterable of ranges, I find myself rewriting functions to compute the intersection (usually of 2 ranges) and the union of those.

    For example

    >>> ranges = [(1,5), (2,6), (1,4)]
    >>> intersection(ranges)
    (2, 4)
    >>> union(ranges)
    [(1,6)]
    >>> ranges = [(1,5), (6,8), (7,9)]
    >>> intersection(ranges)
    None
    >>> union(ranges)
    [(1,5),(6,9)]
    

    Input data might be not necessarily ordered.

    E.g see https://stackoverflow.com/questions/52073609/get-unions-and-intersections-of-list-of-datetime-ranges-python

    I think the boltons library should have a place for functions of this kind!

    opened by n1ngu 0
  • windowed with overlap

    windowed with overlap

    Hello, thank you for this super useful and high-quality library! I'm wondering if there is any support for windowed with overlap, e.g. by default there is overlap of the window size - 1

    >>> g = windowed_iter(itertools.count(), 3)
    >>> next(g)
    (0, 1, 2)
    >>> next(g)
    (1, 2, 3)
    >>> next(g)
    (2, 3, 4)
    

    But what if you want (0, 1, 2), (2, 3, 4), (4, 5, 7)... ? Any easy solution in boltons?

    opened by wimglenn 0
  • boltons.urlutils.URL cannot handle some characters in credentials

    boltons.urlutils.URL cannot handle some characters in credentials

    When trying to parse a URL with special characters in the credentials part it is not working as expected, I found an issue with ? and /.

    If it is in the password there is an exception

    In [52]: URL("http://username:[email protected]:443")
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    ~/tests/venv/lib/python3.7/site-packages/boltons/urlutils.py in parse_url(url_text)
        938             try:
    --> 939                 port = int(port_str)
        940             except ValueError:
    
    ValueError: invalid literal for int() with base 10: 'password'
    
    During handling of the above exception, another exception occurred:
    
    URLParseError                             Traceback (most recent call last)
    <ipython-input-52-4078cba2692c> in <module>
    ----> 1 URL("http://username:[email protected]:443")
    
    ~/tests/venv/lib/python3.7/site-packages/boltons/urlutils.py in __init__(self, url)
        496                                         ' passing the result. (got: %s)'
        497                                         % (DEFAULT_ENCODING, ude))
    --> 498             ud = parse_url(url)
        499
        500         _e = u''
    
    ~/tests/venv/lib/python3.7/site-packages/boltons/urlutils.py in parse_url(url_text)
        941                 if port_str:  # empty ports ok according to RFC 3986 6.2.3
        942                     raise URLParseError('expected integer for port, not %r'
    --> 943                                         % port_str)
        944                 port = None
        945
    
    URLParseError: expected integer for port, not 'password'
    

    If it is in the username it is not raising an exception but the pairing is incorrect

    In [56]: print(URL("http://username?:[email protected]:443").port)
    None
    

    This seems to be an issue with urlparse so I'm not sure it is boltons to blame.

    Trying to use a regex pattern is working for me

    pattern = re.compile(
    r"""
        (?P<schema>[\w\+]+)://
        (?:
            (?P<username>[^:/]*)
            (?::(?P<password>.*))?
        @)?
        (?:
            (?:
                (?P<host>[^/:]+)
            )?
            (?::(?P<port>[^/]*))?
        )?
        """,
    re.X,
    )
    

    (This still not working if the username contain /)

    opened by arossert 0
  • Added py310 to gh actions

    Added py310 to gh actions

    • Updated packages in requirements-test.txt to latest versions: the current version is incompatible with python 3.10
    • Added a separate requirements-test-legacy.txt for py27, py34, py36, pypy, pypy3
    opened by suimong 0
  • Fixes and improvements to SpooledBytesIO & SpooledStringIO

    Fixes and improvements to SpooledBytesIO & SpooledStringIO

    • Inherit from io.IOBase so instance & type checks are simpler
    • __eq__ and __ne__ no longer need to load both files completely into memory
    • __eq__ and __ne__ reset read-head in comparison errors
    • Some methods now raise a ValueError if the file is closed. This brings the implementations closer to StringIO and implementations from the io module
    • Added some additional unittests
    • Removed some Python 2.6 style format strings
    opened by bwatson-dk 0
Releases(20.0.0)
Owner
Mahmoud Hashemi
Code goes here, but also @hatnote & @SimpleLegal. @paypal alumnus. Belaboring the finer points of software. glomming, but mostly 2020ing atm.
Mahmoud Hashemi
Runes - Simple Cookies You Can Extend (similar to Macaroons)

Runes - Simple Cookies You Can Extend (similar to Macaroons) is a paper called "Macaroons: Cookies with Context

Rusty Russell 17 Apr 12, 2022
A functional standard library for Python.

Toolz A set of utility functions for iterators, functions, and dictionaries. See the PyToolz documentation at https://toolz.readthedocs.io LICENSE New

null 3.9k Jun 4, 2022
Standard implementations of FedLab and its provided benchmarks.

FedLab-benchmarks This repo contains standard implementations of FedLab and its provided benchmarks. Currently, following algorithms or benchrmarks ar

SMILELab-FL 62 Jun 6, 2022
Cleaning-utils - a collection of small Python functions and classes which make cleaning pipelines shorter and easier

cleaning-utils [] [] [] cleaning-utils is a collection of small Python functions

null 3 Feb 17, 2022
SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance .

SysInfo SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance . Installation Download t

null 5 Nov 8, 2021
A python app which aggregates and splits costs from multiple public cloud providers into a csv

Cloud Billing This project aggregates the costs public cloud resources by accounts, services and tags by importing the invoices from public cloud prov

null 1 Dec 17, 2021
A utility that makes it easy to work with Python projects containing lots of packages, of which you only want to develop some.

Mixed development source packages on top of stable constraints using pip mxdev [mɪks dɛv] is a utility that makes it easy to work with Python projects

BlueDynamics Alliance 6 May 28, 2022
A Python utility belt containing simple tools, a stdlib like feel, and extra batteries. Hashing, Caching, Timing, Progress, and more made easy!

Ubelt is a small library of robust, tested, documented, and simple functions that extend the Python standard library. It has a flat API that all behav

Jon Crall 601 Jun 4, 2022
Manage your exceptions in Python like a PRO

A linter to manage all your python exceptions and try/except blocks (limited only for those who like dinosaurs).

Guilherme Latrova 284 May 26, 2022
Astvuln is a simple AST scanner which recursively scans a directory, parses each file as AST and runs specified method.

Astvuln Astvuln is a simple AST scanner which recursively scans a directory, parses each file as AST and runs specified method. Some search methods ar

Bitstamp Security 7 May 29, 2022
A clock app, which helps you with routine tasks.

Clock This app helps you with routine tasks. Alarm Clock Timer Stop Watch World Time (Which city you want) About me Full name: Matin Ardestani Age: 14

Matin Ardestani 12 Mar 29, 2022
one_click_kag_server is a program which tries to fully automate the creation of a King Arthur's Gold server.

one_click_kag_server is a program which tries to fully automate the creation of a King Arthur's Gold server.

Benjamin Gorman 4 Jan 5, 2022
New time-based UUID formats which are suited for use as a database key

uuid6 New time-based UUID formats which are suited for use as a database key. This module extends immutable UUID objects (the UUID class) with the fun

null 7 May 28, 2022
UUID version 7, which are time-sortable (following the Peabody RFC4122 draft)

uuid7 - time-sortable UUIDs This module implements the version 7 UUIDs, proposed by Peabody and Davis in https://www.ietf.org/id/draft-peabody-dispatc

Steve Simmons 9 Jun 6, 2022
glip is a module for retrieve ip address like local-ip, global-ip, external-ip as string.

gle_ip_info glip is a module for retrieve ip address like local-ip, global-ip, external-ip as string.

Fatin Shadab 3 Nov 21, 2021
ticktock is a minimalist library to view Python time performance of Python code.

ticktock is a minimalist library to view Python time performance of Python code.

Victor Benichoux 27 Feb 10, 2022
isort is a Python utility / library to sort imports alphabetically, and automatically separated into sections and by type.

isort is a Python utility / library to sort imports alphabetically, and automatically separated into sections and by type. It provides a command line utility, Python library and plugins for various editors to quickly sort all your imports.

Python Code Quality Authority 4.9k Jun 3, 2022
A Python library for reading, writing and visualizing the OMEGA Format

A Python library for reading, writing and visualizing the OMEGA Format, targeted towards storing reference and perception data in the automotive context on an object list basis with a focus on an urban use case.

Institut für Kraftfahrzeuge, RWTH Aachen, ika 11 May 19, 2022
RapidFuzz is a fast string matching library for Python and C++

RapidFuzz is a fast string matching library for Python and C++, which is using the string similarity calculations from FuzzyWuzzy

Max Bachmann 1.3k May 26, 2022