| Scroll ignore | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
About Collectors
| Insert excerpt | ||||||
|---|---|---|---|---|---|---|
|
...
Pre-requisites
...
Python 3.6 - 3.10
...
Access to the KADA Collector repository that contains the BigQuery whl
The repository is currently hosted in KADA’s Azure Blob Storage. You will be given a SAS token to access the repository. Reach out to KADA Support (support@kada.ai) if you do not have access.
Download the BigQuery whl (e.g. kada_collectors_extractors_bigquery-#.#.#-py3-none-any.whl)
...
Collector Server Minimum Requirements
| Insert excerpt | ||||||||
|---|---|---|---|---|---|---|---|---|
|
BigQuery Requirements
Access toBiqQuery
...
Step 1: Establish BigQuery Access
...
Some python packages also have dependencies on the OS level packages, so you may be required to install additional OS packages if the below fails to install.
You can download the Latest Core Library and whl via Platform Settings → Sources → Download Collectors
...
Run the following command to install the collector
| Code Block |
|---|
pip install kada_collectors_extractors_bigquery-3.0.0-py3<version>-none-any.whl |
You will also need to install the common library kada_collectors_lib -1.0.0 for this collector to function properly.
| Code Block |
|---|
pip install kada_collectors_lib-1.0.1-py3-<version>-none-any.whl |
| Info |
|---|
Under the covers this uses the BigQuery Client API and may have OS dependencies see https://cloud.google.com/bigquery/docs/reference/libraries |
...
Step 5: Configure the Collector
The collector requires a set of parameters to connect to and extract metadata from BigQuery
FIELD | FIELD TYPE | DESCRIPTION | EXAMPLE | ||
|---|---|---|---|---|---|
regions | list<string> | List of valid regions to inspect against for data, see https://cloud.google.com/bigquery/docs/locations for list of valid regions | “us” | ||
projects | list<string> | List of project ids to inspect across the regions specified | “kada-data” | ||
host | string | This is the host that was onboarded in K for BigQuery | “bigquery” | ||
json_credentials | JSON | See permissions section on how to download the credentials json to assign to this value |
| ||
output_path | string | Absolute path to the output location where files are to be written | “/tmp/output” | ||
mask | boolean | To enable masking or not | true | ||
compress | boolean | To gzip the output or not | true |
These parameters can be added directly into the run or you can use pass the parameters in via a JSON file.
...
| Code Block |
|---|
import os import argparse from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger from kada_collectors.extractors.athenabigquery import Extractor get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger _type = 'bigquery' dirname = os.path.dirname(__file__) filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type)) parser = argparse.ArgumentParser(description='KADA BigQuery Extractor.') parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.') parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.') args = parser.parse_args() start_hwm, end_hwm = get_hwm(_typeargs.name) ext = Extractor(**load_config(args.config)) ext.test_connection() ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm}) publish_hwm(_type, end_hwm) |
...
If you want prefer file managed hwm, you can edit the location of the hwn by following these instructions https://kadaai.atlassian.net/wiki/spaces/DATKSL/pages/18943181521902411777/Notes+v2.0.0#Storing-HWM-in-another-locationAdditional+Notes#Storing-High-Water-Marks-(HWM)
...
Step 8: Push the Extracts to K
...