Versions Compared

Old Version 6

changes.mady.by.user Chichi

Saved on

compared with

New Version Current

changes.mady.by.user Chichi

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Scroll ignore
scroll-viewporttrue
scroll-pdftrue
scroll-officetrue
scroll-chmtrue
scroll-docbooktrue
scroll-eclipsehelptrue
scroll-htmltrue
scroll-epubtrue

Open in new tab

About Collectors

Insert excerpt
Collector Method
Collector Method
nameabout

...

Pre-requisites

...

Python 3.6 - 3.10

...

Collector Server Minimum Requirements

Insert excerpt
Collector Method
Collector Method
nameCollectorServerSpec
nopaneltrue

BigQuery Requirements

  • Access toBiqQuery

...

Step 1: Establish BigQuery Access

...

You can download the Latest Core Library and Athena whl via Platform Settings → Sources → Download Collectors

...

You will also need to install the common library kada_collectors_lib -1.0.0 for this collector to function properly.

...

The collector requires a set of parameters to connect to and extract metadata from BigQuery

FIELD

FIELD TYPE

DESCRIPTION

EXAMPLE

regions

list<string>

List of valid regions to inspect against for data, see https://cloud.google.com/bigquery/docs/locations for list of valid regions

“us”

projects

list<string>

List of project ids to inspect across the regions specified

“kada-data”

host

string

This is the host that was onboarded in K for BigQuery

“bigquery”

json_credentials

JSON

See permissions section on how to download the credentials json to assign to this value

Code Block
{
    "type": "service_account",
    "project_id": "kada-data",
    "private_key_id": "",
    "private_key": "",
    "client_email": "kada.iam.gserviceaccount.com",
    "client_id": "1234",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/kada.iam.gserviceaccount.com"
}

output_path

string

Absolute path to the output location where files are to be written

“/tmp/output”

mask

boolean

To enable masking or not

true

compress

boolean

To gzip the output or not

true

These parameters can be added directly into the run or you can use pass the parameters in via a JSON file.

...

Code Block
import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.athenabigquery import Extractor

get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger

_type = 'bigquery'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

parser = argparse.ArgumentParser(description='KADA BigQuery Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()

start_hwm, end_hwm = get_hwm(_typeargs.name)

ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

publish_hwm(_type, end_hwm)

...