ZipFly is a zip archive generator based on zipfile.py

Overview

Build Status GitHub release (latest by date) Downloads

Buzon - ZipFly

ZipFly is a zip archive generator based on zipfile.py. It was created by Buzon.io to generate very large ZIP archives for immediate sending out to clients, or for writing large ZIP archives without memory inflation.

Requirements

Python 3.6+

Install

pip3 install zipfly

Basic usage, compress on-the-fly during writes

Using this library will save you from having to write the Zip to disk. Some data will be buffered by the zipfile deflater, but memory inflation is going to be very constrained. Data will be written to destination by default at regular 32KB intervals.

ZipFly defaults attributes:

  • paths: [ ]
  • mode: (write) w
  • chunksize: (bytes) 32768
  • compression: Stored
  • allowZip64: True
  • compresslevel: None
  • storesize: (bytes) 0
  • encode: utf-8

paths list of dictionaries:

.
fs Should be the path to a file on the filesystem
n (Optional) Is the name which it will have within the archive
(by default, this will be the same as fs)

    import zipfly

    paths = [
        {
            'fs': '/path/to/large/file'
        },
    ]

    zfly = zipfly.ZipFly(paths = paths)

    generator = zfly.generator()
    print (generator)
    # 
   


    with open("large.zip", "wb") as f:
        for i in generator:
            f.write(i)

Examples

Streaming multiple files in a zip with Django or Flask Send forth large files to clients with the most popular frameworks

Create paths Easy way to create the array paths from a parent folder.

Predict the size of the zip file before creating it Use the BufferPredictionSize to compute the correct size of the resulting archive before creating it.

Streaming a large file Efficient way to read a single very large binary file in python

Set a comment Your own comment in the zip file

Maintainer

Santiago Debus (@santiagodebus.com)

License

This library was created by Buzon.io and is released under the MIT. Copyright 2021 Cardallot, Inc.

Comments
  • Cannot include empty folders

    Cannot include empty folders

    Python's zipfile module supports writing empty folders to zip archive.

    when trying with zipfly i get:

    File "/.../zipfly.py", line 211, in generator
        with open( path[self.filesystem], 'rb' ) as e:
    IsADirectoryError: [Errno 21] Is a directory: '/path/to/dir'
    

    Is there an undocumented way for writing empty folders?

    opened by rnixx 2
  • Is compression possible?

    Is compression possible?

    Hi,

    I see that only uncompressed ZIP files are can be created with zipfly. What is the reason for that? I don't see any compression level specific code in ZipFly.generator, except file size estimation (which doesn't work for ZIP64 achives).

    So, is compression just an unimplemented feature? Or are there any aspects which don't allow using compression for streaming made this way?

    I understand that compression is often useless in streaming scenarios but it's not true in our case. We're looking for a replacement for zipstream-new which seems to be dead. ZipFly looks nice for us...

    opened by dezhin 2
  • ASGI breaks functionability in HttpStreamingResponse

    ASGI breaks functionability in HttpStreamingResponse

    Running Django 4.1 in async ASGI mode (channels & websockets needed)

    Zipfly functionability breaks as it tries to compile the entire zip to memory before sending zip to client, filling up memory and causing a crash

    opened by T-101 1
  • Release master branch to pypi

    Release master branch to pypi

    It looks like the last version on PyPI has been published 2 years ago and there have been many changes in the code since.

    It would be helpful if you could release the latest changes.

    PS since the project is already using GH action for testing, it may be helpful to set up GH actions for release as well.

    https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/

    https://github.com/marketplace/actions/pypi-publish

    opened by 1oglop1 1
  • Stream into stream

    Stream into stream

    Hi, thank you for the inspirational code.

    I'm wondering if it's possible to adapt your code and make it compatible with S3 storage.

    I use django-storages and boto3 client returns a streaming body (already open fp) and all metadata.

    I need a zipfile (or any other archive) to be created from the set of files during the GET request (not great but it has to be).

    So to save the memory I have 2 options.

    1. Use zipfly as is and download files into a temporary location (and remove it after the operation)
    2. Or better solution that doesn't require intermediate storage so I could pipe the content of streaming body into zipfly and return that as a streaming response.

    streaming body (many of them) -> zipfly(zip file) -> streaming response

    Do you think that option two is possible? If so, could you please point out what pieces of code I should focus on to adapt zipfly?

    Thank you

    opened by 1oglop1 1
  • Add functionality to provide file streams and zip them on the fly

    Add functionality to provide file streams and zip them on the fly

    It is a common use case to store large files remotely and not next to the code. See e.g. for django https://github.com/jschneier/django-storages. If many (large) files must be shipped synchronuosly as a zip file, it saves memory and storage to pass them through the web worker as a stream without saving anything to disk. To archive that, file buffers as input may be supported.

    I would appreciate if we could add this functionality. This pull request contains a tested initial attempt. Feel free to improve!

    This pull request is based on https://github.com/BuzonIO/zipfly/pull/75 and https://github.com/BuzonIO/zipfly/pull/76

    opened by beckedorf 0
  • Using bytes instead of actual files

    Using bytes instead of actual files

    Hi, thanks for the great work. I have some bytes generators and I want to make a zip on the fly out of them. Is there an option to do this? Sorry for my poor python knowledge. Thanks :)

    opened by TaToTanWeb 2
Owner
Buzon
Buzon.io is a Cardallot Inc. service
Buzon
A wrapper for DVD file structure and ISO files.

vs-parsedvd DVDs were an error. A wrapper for DVD file structure and ISO files. You can find me in the IEW Discord server

7 Nov 17, 2022
A simple library for temporary storage of small files

TemporaryStorage An simple library for temporary storage of small files. Navigation Install Usage In Python console As a standalone application List o

2 Apr 17, 2022
Test app for importing contact information in CSV files.

Contact Import TestApp Test app for importing contact information in CSV files. Explore the docs » · Report Bug · Request Feature Table of Contents Ab

1 Feb 06, 2022
A simple file module for creating, editing and saving files.

A simple file module for creating, editing and saving files.

1 Nov 25, 2021
A python script to convert an ucompressed Gnucash XML file to a text file for Ledger and hledger.

README 1 gnucash2ledger gnucash2ledger is a Python script based on the Github Gist by nonducor (nonducor/gcash2ledger.py). This Python script will tak

Thomas Freeman 0 Jan 28, 2022
ValveVMF - A python library to parse Valve's VMF files

ValveVMF ValveVMF is a Python library for parsing .vmf files for the Source Engi

pySourceSDK 2 Jan 02, 2022
Kartothek - a Python library to manage large amounts of tabular data in a blob store

Kartothek - a Python library to manage (create, read, update, delete) large amounts of tabular data in a blob store

15 Dec 25, 2022
Simple archive format designed for quickly reading some files without extracting the entire archive

Simple archive format designed for quickly reading some files without extracting the entire archive

Jarred Sumner 336 Dec 30, 2022
An universal file format tool kit. At present will handle the ico format problem.

An universal file format tool kit. At present will handle the ico format problem.

Sadam·Sadik 1 Dec 26, 2021
csv2ir is a script to convert ir .csv files to .ir files for the flipper.

csv2ir csv2ir is a script to convert ir .csv files to .ir files for the flipper. For a repo of .ir files, please see https://github.com/logickworkshop

Alex 38 Dec 31, 2022
Object-oriented file system path manipulation

path (aka path pie, formerly path.py) implements path objects as first-class entities, allowing common operations on files to be invoked on those path

Jason R. Coombs 1k Dec 28, 2022
The best way to convert files on your computer, be it .pdf to .png, .pdf to .docx, .png to .ico, or anything you can imagine.

The best way to convert files on your computer, be it .pdf to .png, .pdf to .docx, .png to .ico, or anything you can imagine.

JareBear 2 Nov 20, 2021
A platform independent file lock for Python

py-filelock This package contains a single module, which implements a platform independent file lock in Python, which provides a simple way of inter-p

Benedikt Schmitt 497 Jan 05, 2023
Singer is an open source standard for moving data between databases, web APIs, files, queues, and just about anything else you can think of.

Singer is an open source standard for moving data between databases, web APIs, files, queues, and just about anything else you can think of. Th

Singer 1.1k Jan 05, 2023
Python codes for the server and client end that facilitates file transfers. (Using AWS EC2 instance as the server)

Server-and-Client-File-Transfer Python codes for the server and client end that facilitates file transfers. I will be using an AWS EC2 instance as the

Amal Farhad Shaji 2 Oct 13, 2021
Python module that parse power builder file (PBD) and analyze code

PowerBuilder-decompile Python module that parse power builder file (PBD) and analyze code (Incomplete) this tool is composed of: pbd_dump.py pbd file

Samy Sultan 8 Dec 15, 2022
MetaMove is written in Python3 and aims at easing batch renaming operations based on file meta data.

MetaMove MetaMove is written in Python3 and aims at easing batch renaming operations based on file meta data. MetaMove abuses eval combined with f-str

Jan Philippi 2 Dec 28, 2021
File support for asyncio

aiofiles: file support for asyncio aiofiles is an Apache2 licensed library, written in Python, for handling local disk files in asyncio applications.

Tin Tvrtković 2.1k Jan 01, 2023
Automatically generates a TypeQL script for doing entity and relationship insertions from a .csv file, so you don't have to mess with writing TypeQL.

Automatically generates a TypeQL script for doing entity and relationship insertions from a .csv file, so you don't have to mess with writing TypeQL.

3 Feb 09, 2022
Pure Python tools for reading and writing all TIFF IFDs, sub-IFDs, and tags.

Tiff Tools Pure Python tools for reading and writing all TIFF IFDs, sub-IFDs, and tags. Developed by Kitware, Inc. with funding from The National Canc

Digital Slide Archive 32 Dec 14, 2022