Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Info

DBT Core is the self managed version of DBT. Artifacts that generated by DBT core need to be updated to include their project name and then pushed to the K landing directory.

Files to be loaded

DBT generates manifest, catalog and run_result files that can be used to load metadata and logs into K.

...

<project_name>_manifest.json

generated by dbt run / compile

<project_name>_catalog.json

generated by dbt run / compile

<project_name>_run_results_YYYYMMDDHHmmss.json

generated from dbt run

Optional but required for usage information in KADA.

mapping.json

manually created for your environment.

See Generating-the-mapping.json

Generating the mapping.json

A mapping file is required to map dbt projects to KADA sources. Example

...

Note

Update the mapping file when you create a new project in dbt.

Onboarding dbt source in KADA

  1. Create a DBT source by going to Platform Settings > Integration / Sources






    Click Add Source and choose Dbt

  2. Select Load DBT File

  3. Fill in the following details about your DBT instance

    1. Name: Give your DBT source a name

    2. Host: Add the host name of your DBT server

    3. DBT Server: Same as your host name of your DBT Server

    4. Ignore DBT Cloud Account ID

  4. Click Finish Setup

  5. Obtain credentials to the KADA Platform landing directory by contacting the KADA Support for keys

  6. Manually create the mapping.json file and upload to the landing directory.

  7. Configure your orchestration tool (eg Airflow) that executes dbt to:

    1. Rename files as per Key Artefacts

    2. Send the files to the landing directory.

      1. The landing directory can be found by going to Platform Settings > Sources > Select Edit on the DBT source you created



        You will see the landing directory at the bottom



  8. Go back to Sources and Select Edit Schedule

  9. Select a load frequency that aligns with your pipeline