tap-dbt-artifacts is a Singer Tap for artifacts generated by running dbt commands built with the Meltano SDK for Singer Taps.

⚠️ It currently supports the v1 schemas introduced in dbt==0.18.1 for each of the 4 main artifacts listed below and is not compatible with artifacts generated by versions of dbt older than 0.18.1 ⚠️

The easiest way to run this tap is with Meltano - you can install it in your Meltano project as a "custom extractor" (full instructions here)

meltano add --custom extractor tap-dbt-artifacts

Alternatively, to use this tap in a Singer pipeline without Meltano you can install it using pipx

pipx install tap-dbt-artifacts

or pip

pip install tap-dbt-artifacts

This tap takes only a single config option dbt_target_dir, the full path to your dbt target directory. A full list of supported settings and capabilities for this tap is also available by running:

tap-dbt-artifacts --about

You can configure the tap when using Meltano using

meltano config tap-dbt-artifacts set dbt_target_dir '/path/to/dbt/target'

Note that since the tap is simply loading a local file to a Singer Target of your choice, it does not require any authentication.

You can easily run tap-dbt-artifacts by itself or in a pipeline using Meltano.

tap-dbt-artifacts --version
tap-dbt-artifacts --help
tap-dbt-artifacts --config CONFIG --discover > ./catalog.json

meltano elt tap-dbt-artifacts <loader> --job_id=<pipeline-name>

pipx install poetry
poetry install

Create tests within the tap_dbt_artifacts/tests subfolder and then run:

poetry run pytest

You can also test the tap-dbt-artifacts CLI interface directly using poetry run:

poetry run tap-dbt-artifacts --help

Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.

This project comes with a custom meltano.yml project file already created. Install Meltano (if you haven't already) and any needed plugins:

# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-dbt-artifacts
meltano install

Now you can test and orchestrate using Meltano:

# Test invocation:
meltano invoke tap-dbt-artifacts --version
# OR run a test `elt` pipeline:
meltano elt tap-dbt-artifacts target-jsonl

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.

1. Include meltano.yml file with project to simplify testing for local dev
2. Support config option for dbt schema version
3. Write tests to ensure top level keys for each stream are correct