Powerful Python library for atomic file writes.

Overview

python-atomicwrites

https://travis-ci.com/untitaker/python-atomicwrites.svg?branch=master https://ci.appveyor.com/api/projects/status/vadc4le3c27to59x/branch/master?svg=true Documentation Status

Atomic file writes.

from atomicwrites import atomic_write

with atomic_write('foo.txt', overwrite=True) as f:
    f.write('Hello world.')
    # "foo.txt" doesn't exist yet.

# Now it does.

See API documentation for more low-level interfaces.

Features that distinguish it from other similar libraries (see Alternatives and Credit):

  • Race-free assertion that the target file doesn't yet exist. This can be controlled with the overwrite parameter.

  • Windows support, although not well-tested. The MSDN resources are not very explicit about which operations are atomic. I'm basing my assumptions off a comment by Doug Crook, who appears to be a Microsoft employee:

    Question: Is MoveFileEx atomic if the existing and new files are both on the same drive?

    The simple answer is "usually, but in some cases it will silently fall-back to a non-atomic method, so don't count on it".

    The implementation of MoveFileEx looks something like this: [...]

    The problem is if the rename fails, you might end up with a CopyFile, which is definitely not atomic.

    If you really need atomic-or-nothing, you can try calling NtSetInformationFile, which is unsupported but is much more likely to be atomic.

  • Simple high-level API that wraps a very flexible class-based API.

  • Consistent error handling across platforms.

How it works

It uses a temporary file in the same directory as the given path. This ensures that the temporary file resides on the same filesystem.

The temporary file will then be atomically moved to the target location: On POSIX, it will use rename if files should be overwritten, otherwise a combination of link and unlink. On Windows, it uses MoveFileEx through stdlib's ctypes with the appropriate flags.

Note that with link and unlink, there's a timewindow where the file might be available under two entries in the filesystem: The name of the temporary file, and the name of the target file.

Also note that the permissions of the target file may change this way. In some situations a chmod can be issued without any concurrency problems, but since that is not always the case, this library doesn't do it by itself.

fsync

On POSIX, fsync is invoked on the temporary file after it is written (to flush file content and metadata), and on the parent directory after the file is moved (to flush filename).

fsync does not take care of disks' internal buffers, but there don't seem to be any standard POSIX APIs for that. On OS X, fcntl is used with F_FULLFSYNC instead of fsync for that reason.

On Windows, _commit is used, but there are no guarantees about disk internal buffers.

Alternatives and Credit

Atomicwrites is directly inspired by the following libraries (and shares a minimal amount of code):

Other alternatives to atomicwrites include:

  • sashka/atomicfile. Originally I considered using that, but at the time it was lacking a lot of features I needed (Windows support, overwrite-parameter, overriding behavior through subclassing).
  • The Boltons library collection features a class for atomic file writes, which seems to have a very similar overwrite parameter. It is lacking Windows support though.

License

Licensed under the MIT, see LICENSE.

Owner
Markus Unterwaditzer
"Do not even think of telephoning me about this program. Send cash first!" --Author of the UNIX file command.
Markus Unterwaditzer
Extract an archive file (zip file or tar file) stored on AWS S3

S3 Extract Extract an archive file (zip file or tar file) stored on AWS S3. Details Downloads archive from S3 into memory, then extract and re-upload

Evan 1 Dec 14, 2021
File support for asyncio

aiofiles: file support for asyncio aiofiles is an Apache2 licensed library, written in Python, for handling local disk files in asyncio applications.

Tin Tvrtković 2.1k Jan 01, 2023
Better directory iterator and faster os.walk(), now in the Python 3.5 stdlib

scandir, a better directory iterator and faster os.walk() scandir() is a directory iteration function like os.listdir(), except that instead of return

Ben Hoyt 506 Dec 29, 2022
fast change directory with python and ruby

fcdir fast change directory with python and ruby run run python script , chose drirectoy and change your directory need you need python and ruby deskt

XCO 2 Jun 20, 2022
CSV-Handler written in Python3

CSVHandler This code allows you to work intelligently with CSV files. A file in CSV syntax is converted into several lists, which are combined in a to

Max Tischberger 1 Jan 13, 2022
Python package to read and display segregated file names present in a directory based on type of the file

tpyfilestructure Python package to read and display segregated file names present in a directory based on type of the file. Installation You can insta

Tharun Kumar T 2 Nov 28, 2021
This is a junk file creator tool which creates junk files in Internal Storage

This is a junk file creator tool which creates junk files in Internal Storage

KiLL3R_xRO 3 Jun 20, 2021
An object-oriented approach to Python file/directory operations.

Unipath An object-oriented approach to file/directory operations Version: 1.1 Home page: https://github.com/mikeorr/Unipath Docs: https://github.com/m

Mike Orr 506 Dec 29, 2022
Uproot is a library for reading and writing ROOT files in pure Python and NumPy.

Uproot is a library for reading and writing ROOT files in pure Python and NumPy. Unlike the standard C++ ROOT implementation, Uproot is only an I/O li

Scikit-HEP Project 164 Dec 31, 2022
A simple file sharing tool written in python

Share it A simple file sharing tool written in python Installation If you are using Windows os you can directly Run .exe file -- download If you are

Sachit Yadav 7 Dec 16, 2022
A python script to pull the transactions of an Algorand wallet and put them into a CSV file.

AlgoCSV A python script to pull the transactions of an Algorand wallet and put them into a CSV file. Dependancies: Requests Main features: Groups: Com

21 Jun 25, 2022
Utils for streaming large files (S3, HDFS, gzip, bz2...)

smart_open — utils for streaming large files in Python What? smart_open is a Python 3 library for efficient streaming of very large files from/to stor

RARE Technologies 2.7k Jan 06, 2023
Simple Python File Manager

This script lets you automatically relocate files based on their extensions. Very useful from the downloads folder !

Aimé Risson 22 Dec 27, 2022
Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series. The Fourier series can be animated and visualized, the function can be output as a tw

Alexander 12 Jan 01, 2023
A simple library for temporary storage of small files

TemporaryStorage An simple library for temporary storage of small files. Navigation Install Usage In Python console As a standalone application List o

2 Apr 17, 2022
Maltego transforms to pivot between PE files based on their VirusTotal codeblocks

VirusTotal Codeblocks Maltego Transforms Introduction These Maltego transforms allow you to pivot between different PE files based on codeblocks they

Ariel Jungheit 18 Feb 03, 2022
ValveVMF - A python library to parse Valve's VMF files

ValveVMF ValveVMF is a Python library for parsing .vmf files for the Source Engi

pySourceSDK 2 Jan 02, 2022
Pti-file-format - Reverse engineering the Polyend Tracker instrument file format

pti-file-format Reverse engineering the Polyend Tracker instrument file format.

Jaap Roes 14 Dec 30, 2022
Provides a convenient way to append numpy arrays to a file.

Provides a convenient way to append numpy arrays to a file. The NpendWriter and NpendReader classes are used to write and read numpy arrays respective

3 May 14, 2022
Small Python script to generate a calendar (.ics) file from SIMASTER courses schedule.

simaster.ics Small Python script to generate a calendar (.ics) file from SIMASTER courses schedule. Usage Getting the events.json file from SIMASTER O

Faiz Jazadi 8 Nov 02, 2022