| Scroll ignore | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
About Collectors
| Insert excerpt | ||||||
|---|---|---|---|---|---|---|
|
...
Python 3.6 - 3.9
Access to the KADA Collector repository that contains the Snowflake whl
The repository is currently hosted in KADA’s Azure Blob Storage. You will be given a SAS token to access the repository. Reach out to KADA Support (support@kada.ai) if you do not have access.
Download the Snowflake whl (e.g. kada_collectors_extractors_snowflake-#.#.#-py3-none-any.whl)
Access to K landing directory
Access toSnowflake (see section below)
...
From the above record down the following to be used for the setup
User name / Password
Role
Warehouse
Snowflake account (found in the URL of your Snowflake instance - between https:// and .snowflakecomputing.com/…)
Snowflake integration uses username/password. Using keys will be supported in an upcoming release
...
It is recommended to use a python environment such as pyenv or pipenv if you are not intending to install this package at the system level.
You can download the latest Core Library and whl via Platform Settings → Sources → Download Collectors
...
Run the following command to install the collector
| Code Block |
|---|
pip install kada_collectors_extractors_snowflake-3.0.0-py3-<version>-none-any.whl |
You will also need to install the latest common library kada_collectors_lib -1.0.0 for this collector to function properly.
| Code Block |
|---|
pip install kada_collectors_lib-1.0.0-py3<version>-none-any.whl |
Some python packages also have dependencies on the OS level packages, so you may be required to install additional OS packages if the below fails to install. These are some known possible packages you may require depending on your OS, this is not exhaustive and only serves as a guide.
...
The collector requires a set of parameters to connect to and extract metadata from Snowflake
FIELD | FIELD TYPE | DESCRIPTION | EXAMPLE |
|---|---|---|---|
account | string | Snowflake account | “abc123.australia-east.azure” |
username | string | Username to log into the snowflake account | |
password | string | Password to log into the snowflake account | |
information_database | string | Database where all the required tables are located, generally this is snowflake | “snowflake” |
role | string | The role to access the required account_usage tables, generally this is accountadmin | “accountadmin” |
warehouse | string | The warehouse to execute the queries against | “xs_analytics” |
databases | list<string> | A list of databases to extract from Snowflake | [“dwh”, “adw”] |
login_timeout | integer | The max amount of time in seconds allowed for the extractor to establish and authenticate a connection, generally 5 is sufficient but if you have a slow network you can increase this up to 20 | 5 |
output_path | string | Absolute path to the output location where files are to be written | “/tmp/output” |
mask | boolean | To enable masking or not | true |
compress | boolean | To gzip the output or not | true |
host | string | The host value for snowflake that was onboarded in K | “abc123.australia-east.azure.snowflakecomputing.com” |
These parameters can be added directly into the run or you can use pass the parameters in via a JSON file. The following is an example you can use that is included in the example run code below.
...
| Code Block | ||
|---|---|---|
| ||
{
"account": "",
"username": "",
"password": "",
"information_database": "",
"role": "",
"warehouse": "",
"databases": [],
"login_timeout": 5,
"output_path": "/tmp/output",
"mask": true,
"compress": true
"host": ""
}
|
...
Step 5: Run the Collector
...