Page Comparison

About Collectors

...

• Python 3.6 - 3.10

• Access to the KADA Collector repository that contains the BigQuery Hevo whl

  • The repository is currently hosted in KADA’s Azure Blob Storage. You will be given a SAS token to access the repository. Reach out to KADA Support (support@kada.ai) if you do not have access.

  • Download the BigQuery Hevo whl (e.g. kada_collectors_extractors_bigqueryhevo-#.#.#-py3-none-any.whl)

• Access to K landing directory

• Access to BiqQuery Hevo

...

Step 1:

...

Create Hevo API key/Secret

• Create a Service Account by going to the Google Cloud Admin or clicking on this link

  • Give the Service Account a name (e.g. KADA BQ Integration)

  • Select the Projects that include the BigQuery instance(s) that you want to catalog

  • Click Save
    

• Create a Service Token

  • Click on the Service Account

    Image Removed

  • Select the Keys tab. Click on Create new key

    Image Removed

  • Select the JSON option. After clicking ‘CREATE’, the JSON file will automatically download to your device. Provide this to the user(s) that will complete the next steps

    Image Removed

• Add permission grants on the Service Account by going to IAM page or clicking on this link

  • Click on ADD

    Image Removed

  • Add the Service Account to the ‘New principals’ field.

    Image Removed

  • Grant the following roles this principal as shown in the following screenshot.

    • BigQuery Job User

    • BigQuery Metadata Viewer

    • BigQuery Read Session User

    • BigQuery Resource Viewer

  • Image Removed

    Click SAVE

Step 2: Create the Source in K

...

• Login to Hevo

• Click on your Avatar in the top right hand corner and select Account in the drop down menu

• Select API Keys in the side panel and click Generate a New API Key.
  

  Image Added

• Copy the Access Key and Secret Key

...

Step 2: Create the Source in K

• Go to Settings, Select Sources and click Add Source

• Select Hevo as the Source Type

• Select “Load from File system” option

  Image RemovedImage Added

• Give the source a Name - e.g. BigQuery Hevo Production

• Add the Host name for the BigQuery Hevo Server

• Click Finish Setup

...

Step 3: Getting Access to the Source Landing Directory

...

Run the following command to install the collector.

pip install kada_collectors_extractors_bigqueryhevo-3.0.0-py3-none-any.whl

You will also need to install the common library kada_collectors_lib-1.0.1 for this collector to function properly.

pip install kada_collectors_lib-1.0.10-py3-none-any.whl

...

Step 5: Configure the Collector

The collector requires a set of parameters to connect to and extract metadata from BigQueryHevo.

{
        "

myDSN": 

{
      

     "

host": "

myhost",
    

 "database": "

mydatabase"
     

  }
  

 }

These parameters can be added directly into the run or you can use pass the parameters in via a JSON file.

KADA provides an out of the box script that reads a configuration JSON file and runs the extractor. Below is the configuration file.

kada_bigqueryhevo_extractor_config.json

{
    "regionsapi_key": []"",
    "projectsapi_secret": []"",
    "hostregion": "",
    "json_credentialsoutput_path": "/tmp/output",
    "mask": true,
    "timeout": 20,
    "mapping": {
        "myDSN": {},
            "output_pathhost": "/tmp/outputmyhost",
            "maskdatabase": true "mydatabase"
        }
    },
    "compress": true
}

...

Step 6: Run the Collector

...

This can be executed in any python environment where the whl has been installed. It will produce and read a high water mark file from the same directory as the execution called bigqueryhevo_hwm.txt and produce files according to the configuration JSON.

This is the wrapper script: kada_bigqueryhevo_extractor.py

import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.athenahevo import Extractor

get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger

_type = 'bigqueryhevo'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

parser = argparse.ArgumentParser(description='KADA BigQueryHevo Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
args = parser.parse_args()

start_hwm, end_hwm = get_hwm(_type)

ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

publish_hwm(_type, end_hwm)

...

from kada_collectors.extractors.bigqueryhevo import Extractor

kwargs = {my args} # However you choose to construct your args
hwm_kwrgs = {"start_hwm": "end_hwm": } # The hwm values

ext = Extractor(**kwargs)
ext.run(**hwm_kwrgs)

...

class Extractor(regionsapi_key: liststr = []None, projectsapi_secret: liststr = []None, hostregion: str = 'bigquery'None, \
  json_credentialsmapping: dict = {}, timeout: int = 10, output_path: str = './output', \
  mask: bool = False, \
  compress: bool = False) -> None

...

api_key: The API Key for the registered application for access to Hevo APIs
api_secret: The API secret for the registered application for access to Hevo APIs
region: The region prefix for your Hevo
mapping: Dict of DNS to database and hostnames
timeout: Timeout for the API call
output_path: full or relative path to where the outputs should go
mask: To mask the META/DATABASE_LOG files or not
compress: To gzip output files or not

...

A high water mark file is created in the same directory as the execution called bigqueryhevo_hwm.txt and produce files according to the configuration JSON. This file is only produced if you call the publish_hwm method.

...