S3 Extract

Extract an archive file (zip file or tar file) stored on AWS S3.

Details

Downloads archive from S3 into memory, then extract and re-upload to given destination.

The following S3 information is expected to be given as Environment Variables:

AWS_ENDPOINT_URL
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
CERT_PATH (optional)
CERT_DL_URL (optional)
SIGNATURE_VERSION (optional, defaults to "s3v4")
REGION_NAME (optional, defaults to "us-east-1")

Additionally, these information needed for ClearML remote execution can also be given as Env Variable (optional, args will override env var):

DEFAULT_DOCKER_IMG
DEFAULT_QUEUE

For those not familiar, environment variables can be set through various ways, some being:

export = in terminal
can be set in ~/.bashrc as well for more permanence

Iteratively extracting a folder of zips/tars is supported as well, through --src-is-dir flag.

Usage

usage: run.py [-h] [--src-is-dir] [--dst-bucket DST_BUCKET] [--verbose] [--remote] [--clml-proj CLML_PROJ] [--clml-task-name CLML_TASK_NAME] [--clml-task-type CLML_TASK_TYPE]
              [--docker-img DOCKER_IMG] [--queue QUEUE]
              src_bucket src_path dst_path

positional arguments:
  src_bucket            Source bucket
  src_path              Source path
  dst_path              Destination path

optional arguments:
  -h, --help            show this help message and exit
  --src-is-dir          Flag to indicate that given src path is a directory. Will iteratively extract any files in it ending with .zip or .tar.
  --dst-bucket DST_BUCKET
                        Destination bucket (optional), will default to Source bucket.
  --verbose             print out current upload filename as it progresses
  --remote              use clearml to remotely run job
  --clml-proj CLML_PROJ
                        ClearML Project Name
  --clml-task-name CLML_TASK_NAME
                        ClearML Task Name
  --clml-task-type CLML_TASK_TYPE
                        ClearML Task Type, e.g. training, testing, inference, etc
  --docker-img DOCKER_IMG
                        Base docker image to pull for ClearML remote execution
  --queue QUEUE         ClearML remote execution queue

Example usage:

python run.py my-bucket dataset/coco/images.tar dataset/coco/ --verbose --remote --clml-proj coco --clml-task-name coco_extraction --docker-img ubuntu/20.04 --queue 1xGPU

Extract an archive file (zip file or tar file) stored on AWS S3

Related tags

Overview

S3 Extract

Details

Usage

Owner

Evan

ValveVMF - A python library to parse Valve's VMF files

Extract longest transcript or longest CDS transcript from GTF annotation file or gencode transcripts fasta file.

pydicom - Read, modify and write DICOM files with python code

A file utility for accessing both local and remote files through a unified interface.

Python file organizer application

Remove [x]_ from StudIP zip Archives and archive_filelist.csv completely

This is just a GUI that detects your file's real extension using the filetype module.

Python function to stream unzip all the files in a ZIP archive: without loading the entire ZIP file or any of its files into memory at once

ZipFly is a zip archive generator based on zipfile.py

Measure file similarity in a many-to-many fashion

This program can help you to move and rename many files at once

A python script to pull the transactions of an Algorand wallet and put them into a CSV file.

A Certificate renaming tool made for IEEE CS SBC, SJCE.

Object-oriented file system path manipulation

Generates a clean .txt file of contents of a 3 lined csv file

Python's Filesystem abstraction layer

Simple archive format designed for quickly reading some files without extracting the entire archive

Python script for converting figma produced SVG files into C++ JUCE framework source code

A simple Python code that takes input from a csv file and makes it into a vcf file.

Yadl - it is a simple library for working with both dotenv files and environment variables.