Make dbt docs and Apache Superset talk to one another

Overview

dbt-superset-lineage

License: MIT PyPI GitHub last commit PyPI - Python Version PyPI - Format

dbt-superset-lineage

Make dbt docs and Apache Superset talk to one another

Why do I need something like this?

Odds are rather high that you use dbt together with a visualisation tool. If so, these questions might have popped into your head time to time:

  • "Could I get rid of this model? Does it get used for some dashboards? And in which ones, if yes?"
  • "It would be so handy to see all these well-maintained column descriptions when exploring and creating charts."

In case your visualisation tool of choice is Supserset, you are in luck!

Using dbt-superset-lineage, you can:

  • Add dependencies of Superset dashboards to your dbt sources and models
  • Sync column descriptions from dbt docs to Superset

This will help you:

  • Avoid broken dashboards because of deprecated or changed models
  • Choosing the right attributes without navigating back and forth between chart and documentation

Installation

pip install dbt-superset-lineage

Usage

dbt-superset-lineage comes with two basic commands: pull-dashboards and push-descriptions. The documentation for the individual commands can be shown by using the --help option.

It includes a wrapper for Superset API, one only needs to provide SUPERSET_ACCESS_TOKEN/SUPERSET_REFRESH_TOKEN (obtained via /security/login) as environment variable or through --superset-access-token/superset-refresh-token option.

N.B.

  • Make sure to run dbt compile (or dbt run) against the production profile, not your development profile
  • In case more databases are used within dbt and/or Superset and there are duplicate names (schema + table) across them, specify the database through --dbt-db-name and/or --superset-db-id options
  • Currently, PUT requests are only supported if CSRF tokens are disabled in Superset (WTF_CSRF_ENABLED=False).
  • Tested on dbt v0.20.0 and Apache Superset v1.3.0. Other versions, esp. those newer of Superset, might face errors due to different underlying code and API.

Pull dashboards

Pull dashboards from Superset and add them as exposures to dbt docs with references to dbt sources and models, making them visible both separately and as dependencies.

N.B.

  • Only published dashboards are extracted.
$ cd jaffle_shop
$ dbt compile  # Compile project to create manifest.json
$ export SUPERSET_ACCESS_TOKEN=<TOKEN>
$ dbt-superset-lineage pull-dashboards https://mysuperset.mycompany.com  # Pull dashboards from Superset to /models/exposures/superset_dashboards.yml
$ dbt docs generate # Generate dbt docs
$ dbt docs serve # Serve dbt docs

Separate exposure in dbt docs

Referenced exposure in dbt docs

Push descriptions

Push column descriptions from your dbt docs to Superset as plain text so that they could be viewed in Superset when creating charts.

N.B.:

  • Run carefully as this rewrites your datasets using merged column metadata from Superset and dbt docs.
  • Descriptions are rendered as plain text, hence no markdown syntax, incl. links, will be displayed.
  • Avoid special characters and strings in your dbt docs, e.g. or <null>.
$ cd jaffle_shop
$ dbt compile  # Compile project to create manifest.json
$ export SUPERSET_ACCESS_TOKEN=<TOKEN>
$ dbt-superset-lineage push-descriptions https://mysuperset.mycompany.com  # Push descrptions from dbt docs to Superset

Column descriptions in Superset

License

Licensed under the MIT license (see LICENSE.md file for more details).

Owner
Slido
Slido is an audience interaction platform for meetings and events. Public repositories
Slido
Data Structures and Algorithms Python - Practice data structures and algorithms in python with few small projects

Data Structures and Algorithms All the essential resources and template code nee

Hesham 13 Dec 01, 2022
Python Excuse Generator

Excuse Generator Python Excuse Generator This project is an excuse generator that provides the user with an excuse as to why they weren't paying atten

Collin Sanders 5 Jul 07, 2022
A promo calculator for sports betting odds.

Sportbetter Calculation Toolkit Parlay Calculator This is a quick parlay calculator that considers some of the common promos offered. It is used to id

Luke Bhan 1 Sep 08, 2022
In this repo, I will put all the code related to data science using python libraries like Numpy, Pandas, Matplotlib, Seaborn and many more.

Python-for-DS In this repo, I will put all the code related to data science using python libraries like Numpy, Pandas, Matplotlib, Seaborn and many mo

1 Jan 10, 2022
Python calculator made with tkinter package

Python-Calculator Python calculator made with tkinter package. works both on Visual Studio Code Or Any Other Ide Or You Just Copy paste The Same Thing

Pro_Gamer_711 1 Nov 11, 2021
3x+1 recreated in Python

3x-1 3x+1 recreated in Python If a number is odd it is multiplied by 3 and 1 is added to the product. If a number is even it is divided by 2. These ru

4 Aug 19, 2022
Annotates sequences with Eggnog-mapper and hhblits against PDB70

Annotating "hypothetical" proteins with the PDB See config/ for configuration information. This workflow takes as input a set of protein sequences. It

1 Apr 05, 2022
Flight Reservation App With Python

Flight Reservation App With Python

victor-h. 1 Nov 21, 2021
GWCelery is a simple and reliable package for annotating and orchestrating LIGO/Virgo alerts

GWCelery is a simple and reliable package for annotating and orchestrating LIGO/Virgo alerts, built from widely used open source components.

Min-A Cho Zeno 1 Nov 02, 2021
Get a link to the web version of a git-tracked file or directory

githyperlink Get a link to the web version of a git-tracked file or directory. Applies to GitHub and GitLab remotes (and maybe others but those are no

Tomas Fiers 2 Nov 08, 2022
Up to date simple useragent faker with real world database

fake-useragent info: Up to date simple useragent faker with real world database Features grabs up to date useragent from useragentstring.com randomize

Victor K. 2.9k Jan 04, 2023
Penelope Shell Handler

penelope Penelope is an advanced shell handler. Its main aim is to replace netcat as shell catcher during exploiting RCE vulnerabilities. It works on

293 Dec 30, 2022
Anki Addon idea by gbrl.sc to see previous ratings of a card in the reviewer

Card History At A Glance Stop having to press card browser and ctrl+i for every card and then WINCING to see it's history of reviews FEATURES Visualiz

Jerry Zhou 11 Dec 19, 2022
A python package that computes an optimal motion plan for approaching a red light

redlight_approach redlight_approach is a Python package that computes an optimal motion plan during traffic light approach. RLA_demo.mov Given the par

Jonathan Roy 4 Oct 27, 2022
100 Days of Python Programming

100 days of Python Following the initiative of my friend Helber Belmiro, who is almost done with his 100 days of Java, I have decided to start my 100

Henrique Pereira 19 Nov 08, 2021
NBT-Project: This is a APP for building NBT's

NBT-Project This is an APP for building NBT's When using this you select a box on kit maker You input the name and enchant in there related boxes Then

1 Jan 21, 2022
Odoo. Open Source Apps To Grow Your Business.

Odoo Odoo is a suite of web based open source business apps. The main Odoo Apps include an Open Source CRM, Website Builder, eCommerce, Warehouse Mana

Odoo 27.6k Jan 09, 2023
Custom python interface to xstan (a modified (cmd)stan)

Custom python interface to xstan (a modified (cmd)stan) Use at your own risk, currently everything is very brittle and will probably be changed in the

2 Dec 16, 2021
Blender Add-on to Add Metal Materials to Your Scene

Blender QMM (Quick Metal Materials) Blender Addon to Add Metal Materials to Your Scene Installation Download the latest ZIP from Releases. Usage This

Don Schnitzius 27 Dec 26, 2022
Collection of system-wide scripts that I use on my Gentoo

linux-scripts Collection of scripts that I use on my Gentoo machine. I tend to put all scripts in /scripts directory. It is not likely that you would

Xoores 1 Jan 09, 2022