Automatizando a criação de DAGs usando Jinja e YAML

Overview

Automatizando a criação de DAGs no Airflow usando Jinja e YAML

Arquitetura do Repo:

  • Pastas por contexto de negócio (ex: Marketing, Analytics, HR, etc), dentro da pasta contexts/;
    • Dentro de cada pasta, temos os YAMLs com as regras da criação da DAG, sendo 1 YAML = 1 DAG;
  • Pasta de templates, uma vez que possuímos diversos operators, gerando a necessidade de diversos templates
  • Pasta com os scripts para criação de DAGs

Como o código funciona

  • A pessoa adiciona/atualiza o arquivo YAML com todas as informações necessárias para criar a DAG ( para isso veja o CONTEXTS_README presente na pasta contexts/ )
  • O script scripts/convert_yaml_in_dag.py precisa ser executado, passando qual o nome do YAML, e ele fará a conversão para uma DAG.

Sobre as pastas:

  • contexts/: Pasta onde ficam armazenados os YAMLs por contexto da empresa.
    Dentro dessa pasta há um README explicando os padrões de criação dos YAMLs

  • templates/: Pasta onde ficam os arquivos .template, que são as bases para o Jinja criar as DAGs.

    • temos o arquivo dag.template
    • temos os operators já adicionados via template
    • temos um README com os operators já adicionados
  • scripts/ : Scripts tanto para transformar YAML em DAG, quanto para validar se o YAML está correto

    • yml_converter.py: Responsável por criar DAGs a partir do YAML
  • dags/: Pasta padrão do airflow, pode ficar nesse repositório ou em outro, ou até em outra branch.

Referencias:

TBD - Próximos passos

  • Automatizar a execução do script convert_yaml_in_dag.py
  • Generalizar o script acima, para mais opções de contextos, criando um input recebendo isto
  • Implementar testes na estrutura dos YAMLs. Uma sugestão é usar a lib cerberus
  • Implementar CI para os testes. Uma sugestão é usar o proprio github_actions
Owner
Arthur Henrique Dell' Antonia
Arthur Henrique Dell' Antonia
Easy and secure implementation of Azure AD for your FastAPI APIs 🔒 Single- and multi-tenant support.

Easy and secure implementation of Azure AD for your FastAPI APIs 🔒 Single- and multi-tenant support.

Intility 220 Jan 05, 2023
Python One-Time Password Library

PyOTP - The Python One-Time Password Library PyOTP is a Python library for generating and verifying one-time passwords. It can be used to implement tw

PyAuth 2.2k Dec 26, 2022
Django Admin Two-Factor Authentication, allows you to login django admin with google authenticator.

Django Admin Two-Factor Authentication Django Admin Two-Factor Authentication, allows you to login django admin with google authenticator. Why Django

Iman Karimi 9 Dec 07, 2022
An extension of django rest framework, providing a configurable password reset strategy

Django Rest Password Reset This python package provides a simple password reset strategy for django rest framework, where users can request password r

Anexia 363 Dec 24, 2022
Simple Login - Login Extension for Flask - maintainer @cuducos

Login Extension for Flask The simplest way to add login to flask! How it works First, install it from PyPI: $ pip install flask_simplelogin Then, use

Flask Extensions 181 Jan 01, 2023
User Authentication in Flask using Flask-Login

User-Authentication-in-Flask Set up & Installation. 1 .Clone/Fork the git repo and create an environment Windows git clone https://github.com/Dev-Elie

ONDIEK ELIJAH OCHIENG 31 Dec 11, 2022
JSON Web Token Authentication support for Django REST Framework

REST framework JWT Auth JSON Web Token Authentication support for Django REST Framework Overview This package provides JSON Web Token Authentication s

Styria Digital Development 178 Jan 02, 2023
Alisue 299 Dec 06, 2022
A JSON Web Token authentication plugin for the Django REST Framework.

Simple JWT Abstract Simple JWT is a JSON Web Token authentication plugin for the Django REST Framework. For full documentation, visit django-rest-fram

Jazzband 3.2k Dec 28, 2022
A generic, spec-compliant, thorough implementation of the OAuth request-signing logic

OAuthLib - Python Framework for OAuth1 & OAuth2 *A generic, spec-compliant, thorough implementation of the OAuth request-signing logic for Python 3.5+

OAuthlib 2.5k Jan 02, 2023
Out-of-the-box support register, sign in, email verification and password recovery workflows for websites based on Django and MongoDB

Using djmongoauth What is it? djmongoauth provides out-of-the-box support for basic user management and additional operations including user registrat

hao 3 Oct 21, 2021
Google Auth Python Library

Google Auth Python Library This library simplifies using Google's various server-to-server authentication mechanisms to access Google APIs. Installing

Google APIs 598 Jan 07, 2023
Authentication with fastapi and jwt cd realistic

Authentication with fastapi and jwt cd realistic Dependencies bcrypt==3.1.7 data

Fredh Macau 1 Jan 04, 2022
A JSON Web Token authentication plugin for the Django REST Framework.

Simple JWT Abstract Simple JWT is a JSON Web Token authentication plugin for the Django REST Framework. For full documentation, visit django-rest-fram

Jazzband 3.2k Dec 29, 2022
The ultimate Python library in building OAuth, OpenID Connect clients and servers. JWS,JWE,JWK,JWA,JWT included.

Authlib The ultimate Python library in building OAuth and OpenID Connect servers. JWS, JWK, JWA, JWT are included. Authlib is compatible with Python2.

Hsiaoming Yang 3.4k Jan 04, 2023
A simple model based API maker written in Python and based on Django and Django REST Framework

Fast DRF Fast DRF is a small library for making API faster with Django and Django REST Framework. It's easy and configurable. Full Documentation here

Mohammad Ashraful Islam 18 Oct 05, 2022
Django-registration (redux) provides user registration functionality for Django websites.

Description: Django-registration provides user registration functionality for Django websites. maintainers: Macropin, DiCato, and joshblum contributor

Andrew Cutler 920 Jan 08, 2023
A Python package, that allows you to acquire your RecNet authorization bearer token with your account credentials!

RecNet-Login This is a Python package, that allows you to acquire your RecNet bearer token with your account credentials! Installation Done via git: p

Jesse 6 Aug 18, 2022
JSON Web Token Authentication support for Django REST Framework

REST framework JWT Auth Notice This project is currently unmaintained. Check #484 for more details and suggested alternatives. JSON Web Token Authenti

José Padilla 3.2k Dec 31, 2022
Skit-auth - Authorization for skit.ai's platform

skit-auth This is a simple authentication library for Skit's platform. Provides

Skit 3 Jan 08, 2022