dbt Cloud
This page will walkthrough the setup of dbt Cloud in K using the direct connect method
Each dbt Cloud account will need to be added as a separate dbt source.
Integration details
Scope | Included | Comments |
---|---|---|
Metadata | YES |
|
Tests | YES |
|
Lineage | YES |
|
Usage | YES |
|
Sensitive Data Scanner | N/A |
|
Step 1) Create a Service Token
Log into your dbt cloud account and go to Account Settings > Service Tokens
Click New Token.
Give the Token a name
Add Read Only Access to All Projects
Copy the Token for use in the next step
Step 2) Add dbt Cloud as a New Source
Select Platform Settings in the side bar
In the pop-out side panel, under Integrations click on Sources
Click Add Source and select dbt CLOUD
Select Direct Connect and add your dbt Cloud details and click Next
Fill in the Source Settings and click Save & Next
Name: Give the dbt source a name in K
Host: add your
cloud.getdbt.com.[account id]
Account ID: You can find your Account ID in the URL when you log into your dbt cloud console. e.g. https://cloud.getdbt.com/#/accounts/ [account ID]/projects/[project ID]/dashboard/
Environment ID: Add the ID for each Environment you want to include. Add the Environment ID’s as a comma separated list
Each Project may have more than one environments. You can find the ID by going to the Environments page for a project and looking at the URL.
The value must be the ID, not the name of the environment.
Project mapping: Each DBT project is associated with a source. Add a mapping between the DBT project and the source by following the instructions here Host / Database Mapping. You will need to add a mapping for each project.
NOTE: You can find the ID by going to the Project page in your DBT cloud account and looking at the URL
Add your Token copied from Step 1 and click Save
Test your connection and click Save
Click Finish Setup
Step 3) Schedule dbt Cloud source load
Select Platform Settings in the side bar
In the pop-out side panel, under Integrations click on Sources
Locate your new dbt Cloud Source and click on the Schedule Settings (clock) icon to set the schedule
Note that scheduling a source can take up to 15 minutes to propagate the change.
Step 4) Manually run an ad hoc load to test dbt Cloud setup
Next to your new Source, click on the Run manual load icon
Confirm how your want the source to be loaded
After the source load is triggered, a pop up bar will appear taking you to the Monitor tab in the Batch Manager page. This is the usual page you visit to view the progress of source loads
A manual source load will also require a manual run of
DAILY
GATHER_METRICS_AND_STATS
To load all metrics and indexes with the manually loaded metadata. These can be found in the Batch Manager page
Troubleshooting failed loads
If the job failed at the extraction step
Check the error. Contact KADA Support if required.
Rerun the source job
If the job failed at the load step, the landing folder failed directory will contain the file with issues.
Find the bad record and fix the file
Rerun the source job