| Scroll ignore | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
About Collectors
| Insert excerpt | ||||||
|---|---|---|---|---|---|---|
|
...
Pre-requisites
Python 3.6 8 - 3.10
Access to the KADA Collector repository that contains the DBT Cloud whl
The repository is currently hosted in KADA’s Azure Blob Storage. You will be given a SAS token to access the repository. Reach out to KADA Support (support@kada.ai) if you do not have access.
Download the DBT whl (e.g. kada_collectors_extractors_dbt-#.#.#-py3-none-any.whl)
Access to 11
Access to K landing directory
Access toDBT Cloud
...
Some python packages also have dependencies on the OS level packages, so you may be required to install additional OS packages if the below fails to install.
You can download the Latest Core Library and whl via Platform Settings → Sources → Download Collectors
...
Run the following command to install the collector
| Code Block |
|---|
pip install kada_collectors_extractors_dbt-3.1.0-py3-<version>-none-any.whl |
You will also need to install the common library kada_collectors_lib -1.0.0 for this collector to function properly.
| Code Block |
|---|
pip install kada_collectors_lib-1.0.1-py3<version>-none-any.whl |
| Info |
|---|
These are some known possible packages you may require depending on your OS, this is not exhaustive and only serves as a guide. |
...
Step 4: Configure the Collector
The collector requires a set of parameters to connect to and extract metadata from DBT Cloud
...
FIELD
...
FIELD TYPE
...
DESCRIPTION
...
FIELD | FIELD TYPE | DESCRIPTION | EXAMPLE | ||
|---|---|---|---|---|---|
account_id | string | DBT cloud account Id | “xxxxx.australia-east.azure” | ||
environment_ids | list<string> | List of environment Ids to extract | 12345,234234 | ||
token | string | Generated token from the DBT console |
| ||
output_path | string | Absolute path to the output location where files are to be written | “/tmp/output” | ||
timeout | integer | By default we allow 20 seconds for the API to respond, for slower connections it may take longer, so adjust accordingly. | 20 | ||
mapping | JSON | Mapping between DBT project ids and their corresponding database host value in K. | The keys are DBT project ids where as the host is corresponding onboarded host in K
| ||
dry_run | boolean | If you enable dry run, the extractor will simply produce the mapping.json file only which helps you map all your projects to a corresponding database host. | false | ||
compress | boolean | To gzip the output or not | true |
These parameters can be added directly into the run or you can use pass the parameters in via a JSON file. The following is an example you can use that is included in the example run code below.
...
| Code Block |
|---|
import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.dbt import Extractor
get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger
_type = 'dbt'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))
parser = argparse.ArgumentParser(description='KADA DBT Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()
start_hwm, end_hwm = get_hwm(_typeargs.name)
ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})
publish_hwm(_type, end_hwm) |
...
| Code Block |
|---|
class Extractor(token: str = None, account_id: str = None, environment_ids: list=[], \
mapping: dict = {}, timeout: int = 10, dry_run: bool = False, \
output_path: str = './output', compress: bool = False) -> None |
ktokentoken: DBT Cloud Read Only API Token.
account_id: account ID DBT Cloud, should be a numeric ID.
environment_ids: environment ID DBT Cloud, should be a numeric ID.
mapping: Dict of project ids to corresponding database hosts
timeout: Timeout for the API call
dry_run: Run the extractor for the purpose of producing
output_path: full or relative path to where the outputs should go
compress: To gzip output files or not
...