Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
Univerity-student oriented (lithuanian) discord bot

Univerity-student oriented (lithuanian) discord bot

3 Nov 30, 2021
Polars-fun - Example notebooks for how to use pola.rs

polars-fun Polars is an awesome Rust DataFrame library with Python language bindings. This repo makes it easy to run Polars code on your local machine

Matthew Powers 2 Jun 28, 2022
A Really Simple and Advanced Google Colab NoteBook to Setup and start using Rclone on Google Colab easily.

Rclone on Google Colab (Advanced!) πŸ”₯ 1.Setup and Start using Rclone on Google Colab and Create/Edit/View and delete your Rclone config file and keep

Dr.Caduceus 14 May 24, 2022
Code to help me strengthen my bot army

discord-bot-manager an api to help you manage your other bots auth lazy: using the browser dev tools, capture a post call and view the Authorization h

Riley Snyder 2 Mar 18, 2022
Notion API Database Python Implementation

Python Notion Database Notion API Database Python Implementation created only by database from the official Notion API. Installing / Getting started p

minwook 78 Dec 19, 2022
Stock market bot that will be used to learn about API calls and database connections.

Stock market bot that will be used to learn about API calls and database connections.

1 Dec 24, 2021
Kodi script for proper Australian weather data

Kodi Oz Weather weather.ozweather Script for Kodi for high quality Australian weather data sourced directly from the BOM. Available from the Kodi offi

Jeremy Daalder 5 Nov 24, 2022
A Simple Telegram Inline Torrent Search Bot by @infotechIT

Torrent-Search-RoBot A Simple Telegram Inline Torrent Search Bot by @infotechIT. Torrent API Using api.infotech.wtf API Host Bot Deploy to Heroku Clic

InfoTech 0 May 05, 2022
Tinkoff social pulse api wrapper

Tinkoff social pulse api wrapper

Semenov Artur 9 Dec 20, 2022
Telegram bot to check availability of vaccination slots in India.

cowincheckbot Telegram bot to check availability of vaccination slots in India. Setup Install requirements using pip3 install -r requirements.txt Crea

Muhammed Shameem 10 Jun 11, 2022
A simple program to display current playing from Spotify app on your desktop

WallSpot A simple program to display current playing from Spotify app on your desktop How to Use: Linux: Currently Supports GNOME and KDE. If you want

Nannan 4 Feb 19, 2022
A pre-attack hacker tool which aims to find out sensitives comments in HTML comment tag and to help on reconnaissance process

Find Out in Comment Find sensetive comment out in HTML ⚈ About This is a pre-attack hacker tool that searches for sensitives words in HTML comments ta

Pablo EmΓ­dio S.S 8 Dec 31, 2022
Simple web-based hcaptcha bypass

Hcaptcha-Bypass !!! If you found this useful, please click the STAR button !!! Simple web-based hcaptcha bypass Just a demonstration right now, and yo

Kieronia 4 Oct 06, 2021
DISCORD script to automate sending messages to a particular server

discord discord script This script sends random quotes to an discord server and tags random users on the server in the process MADE WITH LOVE BY SACS

Solomon ogu 1 Nov 06, 2021
Some Discord bot block bad words, with this simple hacking tool you will be able to bypass blacklisted words

DISCORD-BAD-WORD-BYPASS-2022 DISCORD BLACKLISTED WORDS HACKING/BYPASS (EDUCATIONAL PURPOSES ONLY) bypass discord blacklisted words. Description Some D

6 Nov 20, 2022
WeChat SDK for Python

___ __ _______ ________ ___ ___ ________ _________ ________ ___ ___ |\ \ |\ \|\ ___ \ |\ ____\|\ \|\ \|\ __ \|\___

wechatpy 3.3k Dec 26, 2022
Powerful Telegram bot to countdown to your important events in any group chat.

Powerful Telegram bot to countdown to your important events in any group chat. Live countdown timer.

118 Dec 30, 2022
OAN Music - Highly advanced User Music Bot

ΰ½§αœ°κ™°κ¦Ώβž’πŽπ€πΰΌ’β˜› 🎧 Advanced πŽπ€π Music bot. πŸ”— 𝐏𝐨𝐰𝐞𝐫𝐞𝐝 𝐛𝐲 : βž’π€ttitude

Attitude king 5 Feb 25, 2022
A smart tool to backup members πŸ“ˆ So you even after a raid/ ban you can easily restore them in seconds 🎲

πŸ€‘ Discord-backer πŸ€‘ A open-source Discord member backup and restore tool for your server. This can help you get all your members in 5 Seconds back af

John 29 Dec 21, 2022