Know your customer pipeline in apache air flow

Overview

KYC_pipline

Know your customer pipeline in apache air flow

For a successful pipeline run take these steps:

  1. Run you Airflow server
  2. Admin -> connection -> create
  3. Triger the input_dag
  4. Before triggering the File_ process dag, move one of the JSON files into the \tmp folder (for example the request_1411.json). In the program we must give the right name of the JSON file to load.
  5. Triger the File_process dag

Input:

For our data, when I read the assignment and what I understood was that I had to create a bunch of dummy data as JSON files with random dates. This can be done with this main.py (Fourthline\Python codes\Generating data) which I turned into a Dag in Airflow as Input_dag.py.

Transformation:

These where the next steps I took for transforming my data: • Read JSON file from a file • Transform the data into an integer • Check the fraud of the request • Agent checks the request • We create the SQLite database • Fill the database with the decision • Create a decision json file with the final output

Read JSON file: ...

Check Fraud: ...

Agent Check: ...

Owner
saeed
Computer Engineer !
saeed
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.

Boltons boltons should be builtins. Boltons is a set of over 230 BSD-licensed, pure-Python utilities in the same spirit as — and yet conspicuously mis

Mahmoud Hashemi 6k Jan 04, 2023
Multipurpose Growtopia Server tools, can be used for newbie to learn things.

Information Multipurpose Growtopia Server tools, can be used for newbie to learn things. Requirements - Python 3.x - Operating System (Recommended : W

Morphias 2 Oct 29, 2021
Python @deprecat decorator to deprecate old python classes, functions or methods.

deprecat Decorator Python @deprecat decorator to deprecate old python classes, functions or methods. Installation pip install deprecat Usage To use th

12 Dec 12, 2022
Adding two matrix from scratch using python.

Adding-two-matrix-from-scratch-using-python. Here, I have take two matrix from user and add it without using any library. I made this program from scr

Sachin Vinayak Dabhade 4 Sep 24, 2021
Numbers-parser - Python module for parsing Apple Numbers .numbers files

numbers-parser numbers-parser is a Python module for parsing Apple Numbers .numbers files. It supports Numbers files generated by Numbers version 10.3

Jon Connell 154 Jan 05, 2023
Tools to connect to and interact with the Mila cluster

milatools The milatools package provides the mila command, which is meant to help with connecting to and interacting with the Mila cluster. Install Re

Mila 32 Dec 01, 2022
A python module for extract domains

A python module for extract domains

Fayas Noushad 4 Aug 10, 2022
A clock app, which helps you with routine tasks.

Clock This app helps you with routine tasks. Alarm Clock Timer Stop Watch World Time (Which city you want) About me Full name: Matin Ardestani Age: 14

Matin Ardestani 13 Jul 30, 2022
This is Cool Utility tools that you can use in python.

This is Cool Utility tools that you can use in python. There are a few tools that you might find very useful, you can use this on pretty much any project and some utils might help you a lot and save

Senarc Studios 6 Apr 18, 2022
Random Number Generator Analysis With Python

Random-Number-Generator-Analysis Governor's Honors Program Project to determine

Jack Prewitt 2 Jan 23, 2022
A python app which aggregates and splits costs from multiple public cloud providers into a csv

Cloud Billing This project aggregates the costs public cloud resources by accounts, services and tags by importing the invoices from public cloud prov

1 Oct 04, 2022
JeNot - A tool to notify you when Jenkins builds are done.

JeNot - Jenkins Notifications NOTE: under construction, buggy, and not production-ready What A tool to notify you when Jenkins builds are done. Why Je

1 Jun 24, 2022
Raganarok X: Next Generation Data Dump

Raganarok X Data Dump Raganarok X: Next Generation Data Dump More interesting Files File Name Contains en_langs All the variables you need in English

14 Jul 15, 2022
Blender 2.93 addon for loading Quake II MD2 files

io_mesh_md2 is a Blender 2.93 addon for importing Quake II MD2 files.

Joshua Skelton 11 Aug 31, 2022
Simple RGB to HEX game made in python

Simple RGB to HEX game made in python

5 Aug 26, 2022
Simple Python tool that generates a pseudo-random password with numbers, letters, and special characters in accordance with password policy best practices.

Simple Python tool that generates a pseudo-random password with numbers, letters, and special characters in accordance with password policy best practices.

Joe Helle 7 Mar 25, 2022
Backman is a random/fixed background image setter for wlroots based compositors

backman Backman is a random/fixed background image setter for wlroots based compositors Dependencies: The program depends on swaybg, python3-toml (or

Hemish 3 Mar 09, 2022
A thing to simplify listening for PG notifications with asyncpg

asyncpg-listen This library simplifies usage of listen/notify with asyncpg: Handles loss of a connection Simplifies notifications processing from mult

ANNA 18 Dec 23, 2022
A python program to find binary, octal and hexadecimal of a decimal.

decimal-converter This little python program can convert a decimal in to, Binary Octal Hexadecimal Needed Python 3 or later or a online python compile

Chandula Janith 0 Nov 27, 2021
Spacegit is a .git exposed finder

Spacegit Spacegit is a basic .git exposed finder Usage: You need python3 installed to run spacegit use: python3 spacegit.py (url) Disclaimer: **This i

2 Nov 30, 2021