Simple helper library to convert a collection of numpy data to tfrecord, and build a tensorflow dataset from the tfrecord.

Last update: Jan 16, 2022

Related tags

Overview

numpy2tfrecord

Simple helper library to convert a collection of numpy data to tfrecord, and build a tensorflow dataset from the tfrecord.

Installation

$ git clone [email protected]:yonetaniryo/numpy2tfrecord.git
$ cd numpy2tfrecord
$ pip install .

Test

$ pytest -v

How to use

Convert a collection of numpy data to tfrecord

import numpy as np
from numpy2tfrecord import Numpy2Tfrecord

converter = Numpy2Tfrecord()
x = np.arange(100).reshape(10, 10).astype(np.float32)  # float array
y = np.arange(100).reshape(10, 10).astype(np.int64)  # int array
a = 5  # int
b = 0.3  # float
entry = {"x": x, "y": y, "a": a, "b": b}
converter.add(entry)  # add data entry
...

converter.export_to_tfrecord("test.tfrecord")  # export to tfrecord

Build a tensorflow dataset from tfrecord

dataset = build_dataset_from_tfrecord("test.tfrecord")  # load tfrecord and build tf.data.Dataset

You might also like...

MMdnn is a set of tools to help users inter-operate among different deep learning frameworks. E.g. model conversion and visualization. Convert models between Caffe, Keras, MXNet, Tensorflow, CNTK, PyTorch Onnx and CoreML.

MMdnn MMdnn is a comprehensive and cross-framework tool to convert, visualize and diagnose deep learning (DL) models. The "MM" stands for model manage

5.7k Jan 9, 2023

A tutorial showing how to train, convert, and run TensorFlow Lite object detection models on Android devices, the Raspberry Pi, and more!

1.3k Jan 2, 2023

tf2onnx - Convert TensorFlow, Keras and Tflite models to ONNX.

tf2onnx converts TensorFlow (tf-1.x or tf-2.x), tf.keras and tflite models to ONNX via command line or python api.

1.8k Jan 8, 2023

TensorFlow Ranking is a library for Learning-to-Rank (LTR) techniques on the TensorFlow platform

2.6k Jan 4, 2023

Minimal deep learning library written from scratch in Python, using NumPy/CuPy.

SmallPebble Project status: experimental, unstable. SmallPebble is a minimal/toy automatic differentiation/deep learning library written from scratch

92 Dec 30, 2022

Technical Indicators implemented in Python only using Numpy-Pandas as Magic - Very Very Fast! Very tiny! Stock Market Financial Technical Analysis Python library . Quant Trading automation or cryptocoin exchange

MyTT Technical Indicators implemented in Python only using Numpy-Pandas as Magic - Very Very Fast! to Stock Market Financial Technical Analysis Python

34 Dec 27, 2022

Calculates JMA (Japan Meteorological Agency) seismic intensity (shindo) scale from acceleration data recorded in NumPy array

shindo.py Calculates JMA (Japan Meteorological Agency) seismic intensity (shindo) scale from acceleration data stored in NumPy array Introduction Japa

3 Sep 23, 2022

Build tensorflow keras model pipelines in a single line of code. Created by Ram Seshadri. Collaborators welcome. Permission granted upon request.

deep_autoviml Build keras pipelines and models in a single line of code! Table of Contents Motivation How it works Technology Install Usage API Image

102 Dec 17, 2022

A hue shift helper for OBS

obs-hue-shift A hue shift helper for OBS This is a repo based on the really nice script Hegemege made. The original script can be found https://gist.g

1 Jan 10, 2022

Comments

Add some more choices for adding samples
add -> add_sample

add_batch: takes a dictionary whose elements each have the batch axis

add_list_of_samples: takes a list of dictionaries as input

add_list_of_batches

enhancement
opened by yonetaniryo 0

Releases(0.0.2)

0.0.2(Jan 26, 2022)

Source code(tar.gz)
Source code(zip)

Owner

Ryo Yonetani

GitHub Repository

Lab Materials for MIT 6.S191: Introduction to Deep Learning

This repository contains all of the code and software labs for MIT 6.S191: Introduction to Deep Learning! All lecture slides and videos are available

5.6k Dec 26, 2022

Recurrent Neural Network Tutorial, Part 2 - Implementing a RNN in Python and Theano

Please read the blog post that goes with this code! Jupyter Notebook Setup System Requirements: Python, pip (Optional) virtualenv To start the Jupyter

863 Dec 15, 2022

Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Official repository of OFA. Paper: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

1.4k Jan 08, 2023

A PaddlePaddle version of Neural Renderer, refer to its PyTorch version

Neural 3D Mesh Renderer in PadddlePaddle A PaddlePaddle version of Neural Renderer, refer to its PyTorch version Install Run: pip install neural-rende

13 Jul 12, 2022

The spiritual successor to knockknock for PyTorch Lightning, get notified when your training ends

Who's there? The spiritual successor to knockknock for PyTorch Lightning, to get a notification when your training is complete or when it crashes duri

70 Oct 06, 2022

High-performance moving least squares material point method (MLS-MPM) solver.

High-Performance MLS-MPM Solver with Cutting and Coupling (CPIC) (MIT License) A Moving Least Squares Material Point Method with Displacement Disconti

2.2k Dec 31, 2022

PyTorch implementation of the supervised learning experiments from the paper Model-Agnostic Meta-Learning (MAML)

pytorch-maml This is a PyTorch implementation of the supervised learning experiments from the paper Model-Agnostic Meta-Learning (MAML): https://arxiv

516 Jan 05, 2023

Lowest memory consumption and second shortest runtime in NTIRE 2022 challenge on Efficient Super-Resolution

FMEN Lowest memory consumption and second shortest runtime in NTIRE 2022 on Efficient Super-Resolution. Our paper: Fast and Memory-Efficient Network T

33 Dec 01, 2022

ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction

ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction. NeurIPS 2021.

59 Nov 25, 2022

Official git repo for the CHIRP project

CHIRP Project This is the official git repository for the CHIRP project. Pull requests are accepted here, but for the moment, the main repository is s

77 Jan 08, 2023

Official PyTorch implementation of GDWCT (CVPR 2019, oral)

This repository provides the official code of GDWCT, and it is written in PyTorch. Paper Image-to-Image Translation via Group-wise Deep Whitening-and-

135 Dec 02, 2022

The final project for "Applying AI to Wearable Device Data" course from "AI for Healthcare" - Udacity.

Motion Compensated Pulse Rate Estimation Overview This project has 2 main parts. Develop a Pulse Rate Algorithm on the given training data. Then Test

2 Oct 25, 2022

Official implementation of "Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets" (CVPR2021)

Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets This is the official implementation of "Towards Good Pract

52 Nov 22, 2022

Simple helper library to convert a collection of numpy data to tfrecord, and build a tensorflow dataset from the tfrecord.

Related tags

Overview

numpy2tfrecord

Installation

Test

How to use

Convert a collection of numpy data to tfrecord

Build a tensorflow dataset from tfrecord

You might also like...

MMdnn is a set of tools to help users inter-operate among different deep learning frameworks. E.g. model conversion and visualization. Convert models between Caffe, Keras, MXNet, Tensorflow, CNTK, PyTorch Onnx and CoreML.

A tutorial showing how to train, convert, and run TensorFlow Lite object detection models on Android devices, the Raspberry Pi, and more!

tf2onnx - Convert TensorFlow, Keras and Tflite models to ONNX.

TensorFlow Ranking is a library for Learning-to-Rank (LTR) techniques on the TensorFlow platform

Minimal deep learning library written from scratch in Python, using NumPy/CuPy.

Technical Indicators implemented in Python only using Numpy-Pandas as Magic - Very Very Fast! Very tiny! Stock Market Financial Technical Analysis Python library . Quant Trading automation or cryptocoin exchange

Calculates JMA (Japan Meteorological Agency) seismic intensity (shindo) scale from acceleration data recorded in NumPy array

Build tensorflow keras model pipelines in a single line of code. Created by Ram Seshadri. Collaborators welcome. Permission granted upon request.

A hue shift helper for OBS

Comments

Add some more choices for adding samples

Releases(0.0.2)

0.0.2(Jan 26, 2022)

Owner

Ryo Yonetani

Lab Materials for MIT 6.S191: Introduction to Deep Learning

Recurrent Neural Network Tutorial, Part 2 - Implementing a RNN in Python and Theano

Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

A PaddlePaddle version of Neural Renderer, refer to its PyTorch version

The spiritual successor to knockknock for PyTorch Lightning, get notified when your training ends

High-performance moving least squares material point method (MLS-MPM) solver.

PyTorch implementation of the supervised learning experiments from the paper Model-Agnostic Meta-Learning (MAML)

Lowest memory consumption and second shortest runtime in NTIRE 2022 challenge on Efficient Super-Resolution

ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction

Official git repo for the CHIRP project

Official PyTorch implementation of GDWCT (CVPR 2019, oral)

The final project for "Applying AI to Wearable Device Data" course from "AI for Healthcare" - Udacity.

Official implementation of "Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets" (CVPR2021)

A demo of how to use JAX to create a simple gravity simulation

Task-related Saliency Network For Few-shot learning

This script runs neural style transfer against the provided content image.

OBBDetection: an oriented object detection toolbox modified from MMdetection

A rule-based log analyzer & filter

A small demonstration of using WebDataset with ImageNet and PyTorch Lightning

Human Dynamics from Monocular Video with Dynamic Camera Movements