| Scroll ignore | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
About Collectors
| Insert excerpt | ||||||
|---|---|---|---|---|---|---|
|
...
You will also need to install the common library kada_collectors_lib-1.0.0 1 for this collector to function properly.
...
The collector requires a set of parameters to connect to and extract metadata from DBT Cloud
FIELD | FIELD TYPE | DESCRIPTION | EXAMPLE |
|---|
FIELD | FIELD TYPE | DESCRIPTION | EXAMPLE | ||
|---|---|---|---|---|---|
account_id | string | DBT cloud account Id | “xxxxx.australia-east.azure” | ||
environment_ids | list<string> | List of environment Ids to extract | 12345,234234 | ||
token | string | Generated token from the DBT console |
| ||
output_path | string | Absolute path to the output location where files are to be written | “/tmp/output” | ||
timeout | integer | By default we allow 20 seconds for the API to respond, for slower connections it may take longer, so adjust accordingly. | 20 | ||
mapping | JSON | Mapping between DBT project ids and their corresponding database host value in K. | The keys are DBT project ids where as the host is corresponding onboarded host in K
| ||
dry_run | boolean | If you enable dry run, the extractor will simply produce the mapping.json file only which helps you map all your projects to a corresponding database host. | false | ||
compress | boolean | To gzip the output or not | true |
...
| Code Block |
|---|
class Extractor(token: str = None, account_id: str = None, environment_ids: list=[], \
mapping: dict = {}, timeout: int = 10, dry_run: bool = False, \
output_path: str = './output', compress: bool = False) -> None |
ktokentoken: DBT Cloud Read Only API Token.
account_id: account ID DBT Cloud, should be a numeric ID.
environment_ids: environment ID DBT Cloud, should be a numeric ID.
mapping: Dict of project ids to corresponding database hosts
timeout: Timeout for the API call
dry_run: Run the extractor for the purpose of producing
output_path: full or relative path to where the outputs should go
compress: To gzip output files or not
...