This is a practice on Airflow, which is building virtual env, installing Airflow and constructing data pipeline (DAGs)

Overview

airflow-test

This is a practice on Airflow, which is

First of all, build virtual env using virtualbox

This is optional but highly recommended

  • It is very important to make sure that development env is independent and has no conflict with other env setting

Next, make python virtual env on proper directory

python3 -m venv sandbox
source ~/sandbox/bin/activate

Let start to install Airflow and initial setting

Install Airflow

pip3 install apache-airflow==2.1.0 \ ## airflow version depends on when you install and constraint 
  --constraint 'git address' ## constraint is ???. It is optional. refer to https://gist.github.com/marclamberti

Airflow Inital Setting

airflow db init ## airflow metadata initializing
airflow users create -u admin -p admin -f jaeyoung -l jang -r Admin -e [email protected] ## create admin user

airflow webserver ## launch web server. after that we can access airflow web server through localhost:8080
airflow scheduler ## activate airfow scheduler to run DAGs

Construct data pipeline with Airflow

One example of data pipeline (one DAGs) - let follow below steps and make data pipeline

  • creating table -> is-api-available -> extending -> processing-use ->
  • what is it? when new user comes in, sense that and process and store user info to destination (?)
  1. create table
  • make python script user_processing.py in dags folder
  • let create table using sqlite operator in user_processing.py
from airflow.models import DAG
from airflow.providers.sqlite.operators.sqlite import SqliteOperator

from datetime import datetime

default_args = {
    'start_date': datetime(2020, 1, 1)
}

with DAG('user_processing', schedule_interval='@daily', default_args=default_args, catchup=False) as dag:
    # Define tasks/operators

    creating_talbe = SqliteOperator(
        task_id='creating_table',
        sqlite_conn_id='db_sqlite',
        sql='''
            CREATE TABLE users (
                firstname TEXT NOT NULL,
                lastname TEXT NOT NULL,
                country TEXT NOT NULL,
                username TEXT NOT NULL,
                password TEXT NOT NULL,
                email TEXT NOT NULL PRIMARY KEY
            );
            '''
    )
  • set sqlite connection on Airflow Web UI
pip install apache-airflow-providers-sqlite ## install airflow provider of sqlite. refer airflow doc. may already be installed from intial installing Airflow..
# next, make a new sqltie connection named 'db_sqlite' in Airflow
  • after that, do test airflow tasks!

airflow tasks test user_processing creating_table 2020-01-01

  • one more things to do just for check again
sqlite3 airflow.db ## connect to sqlite3 in airflow.db
sqlite>.tables; ## see the tables in airflow.dbb
sqlite>select * form users; ## test query
Owner
Jaeyoung
Jaeyoung
a package that provides a marketstrategy for whitelisting on golem

filterms a package that provides a marketstrategy for whitelisting on golem watching requestor logs distribute 10 tasks asynchronously is fun. but you

KJM 3 Aug 03, 2022
Lags valorant servers by rapidly picking up and throwing shorties.

Lags valorant servers by rapidly picking up and throwing shorties.

Eric Still 9 Dec 30, 2021
A tool for removing PUPs using signatures

Unwanted program removal tool A tool for removing PUPs using signatures What is the unwanted program removal tool? The unwanted program removal tool i

4 Sep 20, 2022
An Airflow operator to call the main function from the dbt-core Python package

airflow-dbt-python An Airflow operator to call the main function from the dbt-core Python package Motivation Airflow running in a managed environment

Tomás Farías Santana 93 Jan 08, 2023
This app converts an pdf file into the audio file.

PDF-to-Audio This app takes an pdf as an input and convert it into audio, and the library text-to-speech starts speaking the preffered page given in t

Ojas Barawal 3 Aug 04, 2021
Aim of the project is to reduce phishing victims. 😇

Sites: For more details visit our Blog. How to use 😀 : You just have to paste the url in the ENTER THE SUSPECTED URL section and SELECT THE RESEMBELI

0 May 19, 2022
The program calculates the BMI of people

Programmieren Einleitung: Das Programm berechnet den BMI von Menschen. Es ist sehr einfach zu handhaben, so können alle Menschen ihren BMI berechnen.

2 Dec 16, 2021
An Advent calendar of small programming puzzles for a variety of skill sets and skill levels.

Advent of Code 2021 The Advent of Code is an Advent calendar of small programming puzzles for a variety of skill sets and skill levels that can be sol

Evan Cope 0 Feb 13, 2022
Class XII computer science project.

Computer Science Project — Class XII Kshitij Srivastava (XI – A) Introduction The aim of this project is to create a fully operational system for a me

Kshitij Srivastava 2 Jul 21, 2022
Syarat.ID Source Code - Syarat.ID is a content aggregator website

Syarat.ID is a content aggregator website that gathering all informations with the specific keyword: "syarat" from the internet.

Syarat.ID 2 Oct 15, 2021
Python solution of advent-of-code 2021

Advent of code 2021 Python solutions of Advent of Code 2021 written by Eric Bouteillon Requirements The solutions were developed and tested using Pyth

Eric Bouteillon 3 Oct 25, 2022
Insights in greek football league 2020-2021 and bookmaker's accuracy

Greek_Football_League_Analysis_2020_2021 Aim of Project: This project aims in deriving useful insights from greek football league 2020-2021 by mean st

2 Jan 16, 2022
Linux Security and Monitoring Scripts

Linux Security and Monitoring Scripts These are a collection of security and monitoring scripts you can use to monitor your Linux installation for sec

Andre Pawlowski 65 Aug 27, 2022
Expense Tracker is a very good tool to keep track of your expenseditures and the total money you saved.

Expense Tracker is a very good tool to keep track of your expenseditures and the total money you saved.

Shreejan Dolai 9 Dec 31, 2022
Grade 8 Version of Space Invaders

Space-Invaders Grade 8 Version of Space Invaders Compatability This program is Python 3 Compatable, and not Python 2 Compatable because i haven't test

Space64 0 Feb 16, 2022
APRS Track Direct is a collection of tools that can be used to run an APRS website

APRS Track Direct APRS Track Direct is a collection of tools that can be used to run an APRS website. You can use data from APRS-IS, CWOP-IS, OGN, HUB

Per Qvarforth 42 Dec 29, 2022
Prototype application for GCM bias-correction and downscaling

dodola Prototype application for GCM bias-correction and downscaling This is an unstable prototype. This is under heavy development. Features Nothing!

Climate Impact Lab 9 Dec 27, 2022
msImpersonate - User account impersonation written in pure Python3

msImpersonate v1.0 msImpersonate is a Python-native user impersonation tool that is capable of impersonating local or network user accounts with valid

Joe Helle 90 Dec 16, 2022
Web站点选优工具 - 优化GitHub的打开速度、高效Clone

QWebSiteOptimizer - Web站点速度选优工具 在访问GitHub等网站时,DNS解析到的IP地址可能并不是最快,过慢的节点会严重影响我们的访问情况,故制作出这样的工具来进一步优化网络质量。 由于该方案并非为VPN等方式进行的速度优化,以下几点需要您注意: 后续访问对应网站时仍可能需

QPT Family 15 May 01, 2022
Alternative StdLib for Nim for Python targets

Alternative StdLib for Nim for Python targets, hijacks Python StdLib for Nim

Juan Carlos 100 Jan 01, 2023