| Scroll ignore | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
About Collectors
| Insert excerpt | ||||||
|---|---|---|---|---|---|---|
|
...
Pre-requisites
...
Python 3.6 - 3.10
...
Access to the KADA Collector repository that contains the Hevo whl
The repository is currently hosted in KADA’s Azure Blob Storage. You will be given a SAS token to access the repository. Reach out to KADA Support (support@kada.ai) if you do not have access.
Download the Hevo whl (e.g. kada_collectors_extractors_hevo-#.#.#-py3-none-any.whl)
...
Collector Server Minimum Requirements
| Insert excerpt | ||||||||
|---|---|---|---|---|---|---|---|---|
|
Hevo Requirements
Access toHevo
...
Step 1: Create Hevo API key/Secret
...
Some python packages also have dependencies on the OS level packages, so you may be required to install additional OS packages if the below fails to install.
You can download the Latest latest Core Library and Athena whl via Platform Settings → Sources → Download Collectors
...
You will also need to install the common library kada_collectors_lib -1.0.0 for this collector to function properly.
...
The collector requires a set of parameters to connect to and extract metadata from Hevo.
FIELD | FIELD TYPE | DESCRIPTION | EXAMPLE | ||
|---|---|---|---|---|---|
api_key | string | API Key for Hevo |
| ||
api_secret | string | Secret for the API Key |
| ||
region | string | Region prefix as per https://docs.hevodata.com/getting-started/creating-your-hevo-account/regions/ | au | ||
output_path | string | Absolute path to the output location where files are to be written | “/tmp/output” | ||
mask | boolean | To enable masking or not | true | ||
timeout | integer | Timeout in seconds allowed against the Fivetran APIs | 20 | ||
mapping | JSON | Mapping file of data source names against the onboarded host and database name in K | Assuming I have a “myDSN” data source name in powerbi, I’ll map it to host “myhost” and database “mydatabase” onboarded in K, snowflake type references are handled automatically
| ||
compress | boolean | To gzip the output or not | true |
These parameters can be added directly into the run or you can use pass the parameters in via a JSON file.
...
| Code Block |
|---|
import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.hevo import Extractor
get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger
_type = 'hevo'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))
parser = argparse.ArgumentParser(description='KADA Hevo Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
aparser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()
start_hwm, end_hwm = get_hwm(_typeargs.name)
ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})
publish_hwm(_type, end_hwm) |
...