Optimus's transformation plugins are implementations of Task and Hook interfaces that allows execution of arbitrary jobs in optimus.
brew tap odpf/taps
brew install optimus-plugins-odpf
curl -sL ${PLUGIN_RELEASE_URL} | tar xvz
chmod +x optimus-*
mv optimus-* /usr/bin/
Hello, Currently, for any query, we try to find the dependancy and ignoredependancy with FindDependenciesWithRegex and then we again pull the Refereced table with big query dry run.
If query contains view which is marked with /* @ignoreupstream */ helper, then ignoredependancy will contain the view name but not the table referenced by view.
The change here is to revise ignoredependancy list with table referenced by view.
I kept the loop execution in sequential manner, please let me know if should add concurrency here
enhancementWe have a query referencing to table as well as view.
select * from proj.dataset.table t1 left join proj.dataset.view v1 on t1.date = v1.date and t1.id = v1.id
proj.dataset.table then it correctly ignores to create upstream dependancy for this table.proj.dataset.view ( note the view query refers to 2 more tables ) then it does not ignore view or table referenced by view.As per Optimus PR, the executor boot process is standardised and maintained at optimus. Plugin devs need no longer have to wrap the executor image. closes odpf/optimus#405
Hi team,
I have a bq2bq job with window configuration
window:
size: 720h
offset: -48h
truncate_to: M
I expect to have transformation for date 01 to last day of the month, e.g on April, I expect got transformation from date 01 - 30. but currently only got transformation from date 01 - 29
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-26 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-27 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-28 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-29 00:00:00+00:00
[2022-06-13 15:08:12,324] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: start transformation job
[2022-06-13 15:08:12,324] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: sql transformation query:
after checking, I suspect the logic may related to this line, where the last day generated by windows class not included as the transformation partition.
https://github.com/odpf/transformers/blob/ea1de4f0de3d17d9be7ccefb1e2f3beab1a685f1/task/bq2bq/executor/bumblebee/transformation.py#L393
please kindly check it, and release the fix. thank you
With this we are adding support for using secrets in macros, we do not want to print the env vars in the logs, so exporting them as a separate file from optimus.
Plugins can export this extra file to get env vars.
As per https://github.com/odpf/optimus/pull/425, the executor boot process is standardised and maintained at optimus. Plugin devs need no longer have to wrap the executor image. closes https://github.com/odpf/optimus/issues/405
The most intuitive way is to parse the query and hit the metadata apis instead of going through the dry run which should be definitly costly then the metadata fetch apis.
enhancement performanceOptions
aa14f19 feat: load method for partition replace using aggregation queries (#15)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.9cb26e8f feat: add destination type when generating destination of bq2bq task to support backup feature (#7)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.882d75fb build(deps): bump sqlparse from 0.4.1 to 0.4.2 in /task/bq2bq/executor (#6) 7b6ea2d feat: add execution_project configuration (#9)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.7047c3ba fix(bq2bq): handle BQ dry run error caused by self table SQL reference (#5)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.6631527a refactor: implement plugin mods 621dc7d refactor: update help for bigquery prompts to ask for destination (#3)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.52122157 build(deps): bump rsa from 4.6 to 4.7 in /task/bq2bq/executor (#1) 5f4d0d2 build(deps): bump urllib3 from 1.25.9 to 1.26.5 in /task/bq2bq/executor (#2) db8307c feat(bq2bq): support default values inferred from job name
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.4365796e fix: dataset/table name passed during config gen
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.3
Voicegain Speech-to-Text Python SDK Python SDK for the Voicegain Speech-to-Text API. This API allows for large vocabulary speech-to-text transcription
textlesslib Textless NLP is an active area of research that aims to extend NLP t
Translators - is a library which aims to bring free, multiple, enjoyable translation to individuals and students in Python
Data and code for the paper "Tracing Origins: Coreference-aware Machine Reading Comprehension" at ACL2022. Dataset There are three folders for our thr
The revolutionary IDE for all NLP (Natural language processing) stuffs on the internet.
Transformers: from NLP to CV This is a practical introduction to Transformers from Natural Language Processing (NLP) to Computer Vision (CV) Introduct
it's about time Code repository for "It's About Time: Analog clock Reading in the Wild" Packages required: pytorch (used 1.9, any reasonable version s
Creating a python chatbot that Starbucks users can text to place an order + help cut wait time of a normal coffee.
Parrot is a paraphrase based utterance augmentation framework purpose built to accelerate training NLU models. A paraphrase framework is more than just a paraphrasing model.
TearingNet: Point Cloud Autoencoder to Learn Topology-Friendly Representations Created by Jiahao Pang, Duanshun Li, and Dong Tian from InterDigital In
IMS-Toucan is a toolkit to train state-of-the-art Speech Synthesis models. Everything is pure Python and PyTorch based to keep it as simple and beginner-friendly, yet powerful as possible.
nlpcommon nlpcommon, Python Text Tool. Guide Feature Install Usage Dataset Contact Cite Reference Feature nlpcommon is a python Open Source
This repo is to provide a list of literature regarding Deep Learning on Graphs for NLP
chaii - hindi & tamil question answering This is the solution for rank 5th in Kaggle competition: chaii - Hindi and Tamil Question Answering. The comp
B-MRC MRC approach for Aspect-based Sentiment Analysis (ABSA) Paper: Bidirectional Machine Reading Comprehension for Aspect Sentiment Triplet Extracti
VAD-SLI-ASR Python scripts for a speech processing pipeline with Voice Activity
InferSent InferSent is a sentence embeddings method that provides semantic representations for English sentences. It is trained on natural language in
Predicting Yelp Review Quality Table of Contents Introduction Motivation Goal and Central Questions The Data Data Storage and ETL EDA Data Pipeline Da
Code for EmBERT, a transformer model for embodied, language-guided visual task completion.
Tacotron-pytorch A pytorch implementation of Tacotron: A Fully End-to-End Text-To-Speech Synthesis Model. Requirements Install python 3 Install pytorc
3 Dec 14, 2022
379 Dec 27, 2022
907 Dec 27, 2022
11 Sep 29, 2022
3 Aug 08, 2021
138 Dec 27, 2022
52 Nov 10, 2022
2 Jan 20, 2022
681 Jan 01, 2023
21 Dec 29, 2022
247 Jan 05, 2023
3 May 29, 2022
230 Nov 22, 2022
33 Dec 18, 2022
1 Apr 05, 2022
14 Dec 09, 2022
3 Nov 27, 2022
41 Jan 03, 2023
203 Dec 02, 2022