Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
Python Client for Instagram API

This project is not actively maintained. Proceed at your own risk! python-instagram A Python 2/3 client for the Instagram REST and Search APIs Install

Facebook Archive 2.9k Dec 30, 2022
SUPPORTS 500 GROUPS NO NEED OF BOT πŸ˜‰

LOVELY RADIO SUPPORTS 500 GROUPS NO NEED OF BOT πŸ˜‰ Requirements Telegram API_ID , API_HASH and SESSION_NAME HEROKU Get YouTube live stream link instal

6 Nov 24, 2021
Client to allow skytrack to be used with GSPro Golf simulator application

Skytrack Interface for GSPro A Basic Interface connection from Skytrack Launch Monitors to be able to play simulator golf via GSPro About The Project

James Peruggia 2 Oct 24, 2021
πŸ” Google Search unofficial API for Python with no external dependencies

Python Google Search API Unofficial Google Search API for Python. It uses web scraping in the background and is compatible with both Python 2 and 3. W

Avi Aryan 204 Dec 28, 2022
Free TradingView webhook alert for basic plan users.

TradingView-Free-Webhook-Alerts Project start on 01-02-2022 Providing the free webhook service to the basic plan users in TradingView. Portal β†  Instal

Freeman 31 Dec 25, 2022
Python library for interacting with the Wunderlist 2 REST API

Overview Wunderpy2 is a thin Python library for accessing the official Wunderlist 2 API. What does a thin library mean here? Only the bare minimum of

mieubrisse 24 Dec 29, 2020
New developed moderation discord bot by archisha

Monitor42 New developed moderation discord bot by Ξ±rchΞΉshΞ±#5518. Details Prefix: 42! Commands: Moderation Use 42!help to get command list. Invite http

Kamilla Youver 0 Jun 29, 2022
A Telegram Bot written in Python for mirroring files on the Internet to Google Drive

No support is going to be provided of any kind, only maintaining this for vps user on request. This is a Telegram Bot written in Python for mirroring

0 Dec 26, 2021
Converts a text file of songs to a playlist on your Spotify account.

Playlist Converter Convert a text file of songs to a playlist on your Spotify account. Create your playlists faster instead of manually searching for

Priya Aggarwal 18 Dec 21, 2022
Ulaavi for nuke, helps to keep our stocl elements organised.

Ulaavi Ulaavi for nuke, helps to keep our stock elements organised. Installation Downlaod ffmpeg from ffmpeg.org linux : https://johnvansickle.com/ffm

Arun Subramaniyam 17 Aug 24, 2022
A simple Discord Mass Dm with Scraper

Python-Mass-DM A simple Discord Mass Dm with Scraper If Member Scraper in Taliban.py doesn't work. You can DM me cuz that scraper is for tokens that g

RyanzSantos 4 Sep 02, 2022
The WhatsApp lib

yowsup WARNING It seems that recently yowsup gets detected during registration resulting in an instant ban for your number right after registering wit

Tarek 6.8k Jan 04, 2023
Grape - A webbrowser with its own Search Engine

Grape πŸ”Ž A Web Browser made entirely in python. Search Engine πŸ”Ž Installation: F

Grape 2 Sep 06, 2022
Python wrapper for Stanford CoreNLP.

stanfordcorenlp stanfordcorenlp is a Python wrapper for Stanford CoreNLP. It provides a simple API for text processing tasks such as Tokenization, Par

884 Dec 25, 2022
Credit Card And SK Checker Written In Python

Credit Card And SK Checker Written In Python

Rimuru Tempest 57 Jan 08, 2023
This repository will be a draft of a package about the latest total marine fish production in Indonesia. Data will be collected from PIPP (Pusat Informasi Pelabuhan Perikanan).

indomarinefish This package will give us information about the latest total marine fish production in Indonesia. The Name of the fish is written in In

1 Oct 13, 2021
Python script that extract data via YouTube Api and manipulates it.

UNLIMITED README for the Unlimited game [Mining game] Explore the docs Β» View Demo Β· Report Bug Β· Request Feature Table of Contents About The Project

Serban Chisca 1 Dec 12, 2021
Want to play What Would Rather on your Server? Invite the bot now!😏

What is this Bot? πŸ‘€ What You Would Rather? is a Guessing game where you guess one thing. Long Description short Take this example: You typed r!rather

δΈ‚γ„šδΉˆδΉ™γƒ„ 2 Nov 17, 2021
The official command-line client for spyse.com

Spyse CLI The official command-line client for spyse.com. NOTE: This tool is currently in the early stage beta and shouldn't be used in production. Yo

Spyse 43 Dec 08, 2022
M3U Playlist for free TV channels

Free TV This is an M3U playlist for free TV channels around the World. Either free locally (over the air): Or free on the Internet: Plex TV Pluto TV P

Free TV 964 Jan 08, 2023