This page will guide you through the setup of Snowflake in K using the direct connect method.
Integration details
Scope | Included | Comments |
---|---|---|
Metadata | YES | See below for know limitations |
Lineage | YES |
|
Usage | YES | |
Sensitive Data Scanner | YES |
Known limitations
Streams & Tasks are not currently included in the metadata extracted from Snowflake. Currently under review.
Access roles only consider grants to users. Access roles do not consider grants to other roles
Step 1) Establish Snowflake Access
K only needs access to metadata tables in Snowflake. These tables are stored in the “Snowflake” database. You do not need to provide K access to any data stored in your Snowflake database.
Create a Snowflake user with read access to following tables in the Snowflake database.
account_usage.history
account_usage.views
account_usage.tables
account_usage.columns
account_usage.copy_history
account_usage.grants_to_roles
account_usage.grants_to_users
account_usage.schemata
account_usage.databases
--Log in with a user that has the permissions to create a role/user --Create a new role for the Catalog user Create role CATALOG_READ_ONLY; --Grant the role access to the Account usage schema grant imported privileges on database Snowflake to CATALOG_READ_ONLY; grant select on all tables in schema SNOWFLAKE.ACCOUNT_USAGE to CATALOG_READ_ONLY; grant monitor on account to role CATALOG_READ_ONLY; --Create a new user for K and grant it the role (remove the []) create user [kada_user] password=['abc123!@#'] default_role = CATALOG_READ_ONLY default_warehouse = [warehouse];
From the above record down the following to be used for the setup
User name / Password
Role
Warehouse
Snowflake account (found in the URL of your Snowflake instance - between https:// and .snowflakecomputing.com/…)
Snowflake integration uses username/password. Using keys will be supported in an upcoming release
Step 2) Connecting K to Snowflake
Select Platform Settings in the side bar
In the pop-out side panel, under Integrations click on Sources
Click Add Source and select Snowflake
Select Direct Connect and add your Snowflake details and click Next
Fill in the Source Settings and click Next
Name: The name you wish to give your Snowflake DB in K
Host: Add your Snowflake Account (found in your Snowflake URL)
Omit the https:// from the URL
Information Database Role: Add the role the user should use
Warehouse: Add the warehouse the user should use
Add the Connection details and click Save & Next when connection is successful
Snowflake account: Use the same details (Snowflake account) you previously added in the Host setting
Username: Add the Snowflake user name
Password: Add the Snowflake user password
Test your connection and click Save
Select the Databases you wish to load into K and click Finish Setup
All databases will be listed. If you have a lot of databases this may take a few seconds to load
Return to the Sources page and locate the new Snowflake source that you loaded
Click on the clock icon to select Edit Schedule and set your preferred schedule for the Snowflake load
Note that scheduling a source can take up to 15 minutes to propagate the change.
Step 3) Manually run an ad hoc load to test Snowflake setup
Next to your new Source, click on the Run manual load icon
Confirm how you want the source to be loaded
After the source load is triggered, a pop up bar will appear taking you to the Monitor tab in the Batch Manager page. This is the usual page you visit to view the progress of source loads
A manual source load will also require a manual run of
DAILY
GATHER_METRICS_AND_STATS
To load all metrics and indexes with the manually loaded metadata. These can be found in the Batch Manager page
Troubleshooting failed loads
If the job failed at the extraction step
Check the error. Contact KADA Support if required.
Rerun the source job
If the job failed at the load step, the landing folder failed directory will contain the file with issues.
Find the bad record and fix the file
Rerun the source job