Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
Script to post multiple status(posts) on twitter

Script to post multiple status on twitter (i.e. TWITTER STORM) This program can post upto maximum limit of twitter(around 300 tweets) within seconds.

Sandeep Kumar 4 Sep 09, 2021
The successor of GeoSnipe, a pythonic Minecraft username sniper based on AsyncIO.

OneSnipe The successor of GeoSnipe, a pythonic Minecraft username sniper based on AsyncIO. Documentation View Documentation Features • Mojang & Micros

1 Jan 14, 2022
VoiceMaster-Discord-Bot - Fork from original Discord bot with max channel limit, staff role and more

VoiceMaster VoiceMaster is a discord bot created to change the way servers work,

2 Feb 28, 2022
A Telegram Bot written in Python for mirroring files on the Internet to your Google Drive or Telegram

Original Repo mirror-leech-telegram-bot This is a Telegram Bot written in Python for mirroring files on the Internet to your Google Drive or Telegram.

0 Jan 03, 2022
Apps related to Odoo it's calendar features

calendar Apps related to Odoo it's calendar/appointments features: online_appointment_locations: allow setting an online URL per employee online_appoi

Yenthe Van Ginneken 3 Oct 27, 2022
Token Manager written in Python

Discord-API-Token-Entrance Description This is a Token Manager that allows your token to enter your discord server, written in python. Packages Requir

Tootle 1 Apr 15, 2022
A python script to send sms anonymously with SMS Gateway API. Works on command line terminal.

incognito-sms-sender A python script to send sms anonymously with SMS Gateway API. Works on command line terminal. Download and run script Go to API S

ʀᴇxɪɴᴀᴢᴏʀ 1 Oct 25, 2021
Simple screen recorder

Kooha Simple screen recorder Description Kooha is a simple screen recorder built with GTK. It allows you to record your screen and also audio from you

Dave Patrick 1.2k Jan 03, 2023
This Server Cloner can clone the server you want with all the perms of roles in every particular channel.

Server-Cloner-with-perms 🚀 This Server Cloner can clone the server you want with all the perms of roles in every particular channel. Features Clone C

Gripz 0 Feb 17, 2022
The Python SDK for the Rackspace Cloud

pyrax Python SDK for OpenStack/Rackspace APIs DEPRECATED: Pyrax is no longer being developed or supported. See openstacksdk and the rackspacesdk plugi

PyContribs 238 Sep 21, 2022
A file-based quote bot written in Python

Let's Write a Python Quote Bot! This repository will get you started with building a quote bot in Python. It's meant to be used along with the Learnin

1 Dec 07, 2021
A Discord Bot - has a few commands. Built using python - Discord.py - RIP.

Discord_Bot A Discord Bot has been built here. It is capable of running a few commands. The below present screenshot should suffice in terms of explai

Manab Kumar Biswas 1 May 22, 2022
Automatically Message From Discord Account

Discord-AutoMessage A robust and versatile solution for automated social interactions HOW TO INSTALL Open cmd cd into your project directory Run the f

13 Jul 11, 2022
Fast and small Discord-Toolset.

Mooncord 🌙 Discord server: https://discord.gg/frnpk2rg Fast and small Discord-Toolset. Enjoy? Star this repo ⭐ (Main file in Mooncord/Moon-1.0.1/vers

7ua 9 Dec 11, 2021
Generates a coverage badge using coverage.py and the shields.io service.

Welcome to README Coverage Badger 👋 Generates a coverage badge using coverage.py and the shields.io service. Your README file is then updated with th

Victor Miti 10 Dec 06, 2022
Tiktok 2 Instagram With Python

Tiktok2Instagram 📸 About The Project What it does: Download the source video from a user inputted Tiktok URL. 📙 Add audio to the Tiktok video from a

Carter Belisle 4 Feb 06, 2022
A simple Python app to provide RPC for iTunes and the Music app. MacOS exclusive.

Ongaku You know, ongaku. A port of Ongaku to Python. Why? I don't know. A simple application providing the now playing state from iTunes (or the Music

Deltaion Lee 4 Oct 22, 2022
Easy and simple, Telegram Bot to Show alert when some edits a message in Group

Edit-Message-Alert Just a simple bot to show alert when someone edits a message sent by them, Just 17 Lines of Code These codes are for those who incu

Nuhman Pk 6 Dec 15, 2021
Python client for Midea dhumidifier

This is a library that allows communication with Midea dehumidifier appliances via the local area network. midea-beautiful-dehumidifier This library a

Nenad Bogojevic 42 Dec 22, 2022
This repository contains the best Data Science free hand-picked resources to equip you with all the industry-driven skills and interview preparation kit.

Best Data Science Resources Hey, Data Enthusiasts out there! Finally, after lots of requests from the community I finally came up with the best free D

Mohit Kumar 415 Dec 31, 2022