Make dbt docs and Apache Superset talk to one another

Overview

dbt-superset-lineage

License: MIT PyPI GitHub last commit PyPI - Python Version PyPI - Format

dbt-superset-lineage

Make dbt docs and Apache Superset talk to one another

Why do I need something like this?

Odds are rather high that you use dbt together with a visualisation tool. If so, these questions might have popped into your head time to time:

  • "Could I get rid of this model? Does it get used for some dashboards? And in which ones, if yes?"
  • "It would be so handy to see all these well-maintained column descriptions when exploring and creating charts."

In case your visualisation tool of choice is Supserset, you are in luck!

Using dbt-superset-lineage, you can:

  • Add dependencies of Superset dashboards to your dbt sources and models
  • Sync column descriptions from dbt docs to Superset

This will help you:

  • Avoid broken dashboards because of deprecated or changed models
  • Choosing the right attributes without navigating back and forth between chart and documentation

Installation

pip install dbt-superset-lineage

Usage

dbt-superset-lineage comes with two basic commands: pull-dashboards and push-descriptions. The documentation for the individual commands can be shown by using the --help option.

It includes a wrapper for Superset API, one only needs to provide SUPERSET_ACCESS_TOKEN/SUPERSET_REFRESH_TOKEN (obtained via /security/login) as environment variable or through --superset-access-token/superset-refresh-token option.

N.B.

  • Make sure to run dbt compile (or dbt run) against the production profile, not your development profile
  • In case more databases are used within dbt and/or Superset and there are duplicate names (schema + table) across them, specify the database through --dbt-db-name and/or --superset-db-id options
  • Currently, PUT requests are only supported if CSRF tokens are disabled in Superset (WTF_CSRF_ENABLED=False).
  • Tested on dbt v0.20.0 and Apache Superset v1.3.0. Other versions, esp. those newer of Superset, might face errors due to different underlying code and API.

Pull dashboards

Pull dashboards from Superset and add them as exposures to dbt docs with references to dbt sources and models, making them visible both separately and as dependencies.

N.B.

  • Only published dashboards are extracted.
$ cd jaffle_shop
$ dbt compile  # Compile project to create manifest.json
$ export SUPERSET_ACCESS_TOKEN=<TOKEN>
$ dbt-superset-lineage pull-dashboards https://mysuperset.mycompany.com  # Pull dashboards from Superset to /models/exposures/superset_dashboards.yml
$ dbt docs generate # Generate dbt docs
$ dbt docs serve # Serve dbt docs

Separate exposure in dbt docs

Referenced exposure in dbt docs

Push descriptions

Push column descriptions from your dbt docs to Superset as plain text so that they could be viewed in Superset when creating charts.

N.B.:

  • Run carefully as this rewrites your datasets using merged column metadata from Superset and dbt docs.
  • Descriptions are rendered as plain text, hence no markdown syntax, incl. links, will be displayed.
  • Avoid special characters and strings in your dbt docs, e.g. or <null>.
$ cd jaffle_shop
$ dbt compile  # Compile project to create manifest.json
$ export SUPERSET_ACCESS_TOKEN=<TOKEN>
$ dbt-superset-lineage push-descriptions https://mysuperset.mycompany.com  # Push descrptions from dbt docs to Superset

Column descriptions in Superset

License

Licensed under the MIT license (see LICENSE.md file for more details).

Owner
Slido
Slido is an audience interaction platform for meetings and events. Public repositories
Slido
Insights in greek football league 2020-2021 and bookmaker's accuracy

Greek_Football_League_Analysis_2020_2021 Aim of Project: This project aims in deriving useful insights from greek football league 2020-2021 by mean st

2 Jan 16, 2022
A Desktop application for the signalum python library

Signalum Desktop A Desktop application on the Signalum Python Library/CLI Tool. The Signalum Desktop application is an attempt to develop a single too

BISOHNS 35 Feb 15, 2021
Sample microservices application demo

Development mode docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d or export COMPOSE_FILE='docker-compose.yml:docker-compose.dev.ym

Konstantinos Bairaktaris 1 Nov 14, 2021
Short, introductory guide for the Python programming language

100 Page Python Intro This book is a short, introductory guide for the Python programming language.

Sundeep Agarwal 185 Dec 26, 2022
The purpose of this code base is to add a specified signal-to-noise ratio noise from MUSAN dataset to a pure speech signal and to generate far-field speech data using room impulse response data from BUT [email protected] Reverb Database.

Add_noise_and_rir_to_speech The purpose of this code base is to add a specified signal-to-noise ratio noise from MUSAN dataset to a pure speech signal

Yunqi Chen 7 Oct 30, 2022
This is a small compiler to demonstrate how compilers work.

This is a small compiler to demonstrate how compilers work. It compiles our own dialect to C, while being written in Python.

Md. Tonoy Akando 2 Jul 19, 2022
Remote execution of a simple function on the server

FunFetch Remote execution of a simple function on the server All types of Python support objects.

Decave 4 Jun 30, 2022
VacationCycleLogicBackEnd - Vacation Cycle Logic BackEnd With Python

Vacation Cycle Logic BackEnd Getting Started Existing virtualenv If your project

Mohamed Gamal 0 Jan 03, 2022
Incident Response Process and Playbooks | Goal: Playbooks to be Mapped to MITRE Attack Techniques

PURPOSE OF PROJECT That this project will be created by the SOC/Incident Response Community Develop a Catalog of Incident Response Playbook for every

Austin Songer 987 Jan 02, 2023
A simple Programming Language

R.S.O.C. A custom built programming language About The Project R.S.O.C. is a custom built programming language very similar to a low-level 8085 progra

Ravi Maurya 17 Sep 13, 2022
Expose multicam options in the Blender VSE headers.

Multicam Expose multicam options in the Blender VSE headers. Install Download space_sequencer.py and swap it with the one that comes with the Blender

4 Feb 27, 2022
Function Plotter✨

Function-Plotter Build With : Python PyQt5 unittest matplotlib Getting Started This is an list of needed instructions to set up your project locally,

Ahmed Lotfy 3 Jan 06, 2022
A docker container (Docker Desktop) for a simple python Web app few unit tested

Short web app using Flask, tested with unittest on making massive requests, responses of the website, containerized

Omar 1 Dec 13, 2021
Myrepo - A tool to create your own Arch Linux repository

myrepo A (experimental) tool to create your own Arch Linux repository Example We

Anton Hvornum 5 Feb 19, 2022
Navigate to your directory of choice the proceed as follows

Installation 🚀 Navigate to your directory of choice the proceed as follows; 1 .Clone the git repo and create a virtual environment Depending on your

Ondiek Elijah Ochieng 2 Jan 31, 2022
*考研学习利器,玩电脑控制不住自己时,可以使用该程序定日期锁屏,同时有精美壁纸锁屏显示,也不会枯燥。

LockscreenbyTime_win10 A python program in win10. You can set the time to lock the computer(by setting year, month, day), Fullscreen pictures will sho

PixianDouban 4 Jul 10, 2022
Powerful Assistant

Delta-Assistant Hi I'm Phoenix This project is a smart assistant This is the 1.0 version of this project I am currently working on the third version o

1 Nov 17, 2021
CoreSE - basic of social Engineering tool

Core Social Engineering basic of social Engineering tool. just for fun :) About First of all, I must say that I wrote such a project because of my int

Hamed Mohammadvand 7 Jun 10, 2022
A performant state estimator for power system

A state estimator for power system. Turbocharged with sparse matrix support, JIT, SIMD and improved ordering.

9 Dec 12, 2022
Python Create Your Own Tool Series

Python Create Your Own Tool Series Hey there! This is an additional Github repository that contains the final product files for each video in my Youtu

Joe Helle 21 Dec 02, 2022