Know your customer pipeline in apache air flow

Overview

KYC_pipline

Know your customer pipeline in apache air flow

For a successful pipeline run take these steps:

  1. Run you Airflow server
  2. Admin -> connection -> create
  3. Triger the input_dag
  4. Before triggering the File_ process dag, move one of the JSON files into the \tmp folder (for example the request_1411.json). In the program we must give the right name of the JSON file to load.
  5. Triger the File_process dag

Input:

For our data, when I read the assignment and what I understood was that I had to create a bunch of dummy data as JSON files with random dates. This can be done with this main.py (Fourthline\Python codes\Generating data) which I turned into a Dag in Airflow as Input_dag.py.

Transformation:

These where the next steps I took for transforming my data: • Read JSON file from a file • Transform the data into an integer • Check the fraud of the request • Agent checks the request • We create the SQLite database • Fill the database with the decision • Create a decision json file with the final output

Read JSON file: ...

Check Fraud: ...

Agent Check: ...

Owner
saeed
Computer Engineer !
saeed
Manage your exceptions in Python like a PRO

A linter to manage all your python exceptions and try/except blocks (limited only for those who like dinosaurs).

Guilherme Latrova 353 Dec 31, 2022
jfc is an utility to make reviewing ArXiv papers for your Journal Club easier.

jfc is an utility to make reviewing ArXiv papers for your Journal Club easier.

Miguel M. 3 Dec 20, 2021
A random cats photos python module

A random cats photos python module

Fayas Noushad 6 Dec 01, 2021
Minimal Windows system information tool written in Python

wfetch wfetch is a Minimal Windows system information tool written in Python (Only works on Windows) Installation First of all have python installed.

zJairO 3 Jan 24, 2022
Animation retargeting tool for Autodesk Maya. Retargets mocap to a custom rig with a few clicks.

Animation Retargeting Tool for Maya A tool for transferring animation data and mocap from a skeleton to a custom rig in Autodesk Maya. Installation: A

Joaen 63 Jan 06, 2023
Generate random german words

Generate random german words / Generiere zufällige deutsche Wörter Getting Started Pip install with pip install zufallsworte Install the library with

Maximilian Freitag 5 Mar 24, 2022
A way to write regex with objects instead of strings.

Py Idiomatic Regex (AKA iregex) Documentation Available Here An easier way to write regex in Python using OOP instead of strings. Makes the code much

Ryan Peach 18 Nov 15, 2021
This script allows you to retrieve all functions / variables names of a Python code, and the variables values.

Memory Extractor This script allows you to retrieve all functions / variables names of a Python code, and the variables values. How to use it ? The si

Venax 2 Dec 26, 2021
iOS Snapchat parser for chats and cached files

ParseSnapchat iOS Snapchat parser for chats and cached files Tested on Windows and Linux install required libraries: pip install -r requirements.txt c

11 Dec 05, 2022
HeadHunter parser

HHparser Description Program for finding work at HeadHunter service Features Find job Parse vacancies Dependencies python pip geckodriver firefox Inst

memphisboy 1 Oct 30, 2021
Etherium unit conversation and arithmetic library

etherunit Etherium unit conversation and arithmetic library Install pip install -u etherunit Usage from etherunit import Ether, Gwei, Wei, E Creat

Yasin Özel 1 Nov 10, 2021
Run async workflows using pytest-fixtures-style dependency injection

Run async workflows using pytest-fixtures-style dependency injection

Simon Willison 26 Jun 26, 2022
Pampy: The Pattern Matching for Python you always dreamed of.

Pampy: Pattern Matching for Python Pampy is pretty small (150 lines), reasonably fast, and often makes your code more readable and hence easier to rea

Claudio Santini 3.5k Jan 06, 2023
Cleaning-utils - a collection of small Python functions and classes which make cleaning pipelines shorter and easier

cleaning-utils [] [] [] cleaning-utils is a collection of small Python functions

4 Aug 31, 2022
Obsidian tools - a Python package for analysing an Obsidian.md vault

obsidiantools is a Python package for getting structured metadata about your Obsidian.md notes and analysing your vault.

Mark Farragher 153 Jan 04, 2023
Simple code to generate a password for your account!

Password-Generator Simple code to generate a password for your account! Password Generator for passwords for your accounts or anything else! This code

DEEM 1 Jun 05, 2022
A program will generate a eth key pair that has the public key that starts with a defined amount of 0

ETHAdressGenerator This short program will generate a eth key pair that has the public key that starts with a defined amount of 0 Requirements Python

3 Nov 19, 2021
Finger is a function symbol recognition engine for binary programs

Finger is a function symbol recognition engine for binary programs

332 Jan 01, 2023
A tiny Python library for generating public IDs from integers

pids Create short public identifiers based on integer IDs. Installation pip install pids Usage from pids import pid public_id = pid.from_int(1234) #

Simon Willison 7 Nov 11, 2021
A script to parse and display buy_tag and sell_reason for freqtrade backtesting trades

freqtrade-buyreasons A script to parse and display buy_tag and sell_reason for freqtrade backtesting trades Usage Copy the buy_reasons.py script into

Robert Davey 31 Jan 01, 2023