Versions Compared

Old Version 8

changes.mady.by.user Chichi

Saved on

compared with

New Version 9

changes.mady.by.user Sidney Chen

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Scroll ignore
scroll-viewporttrue
scroll-pdftrue
scroll-officetrue
scroll-chmtrue
scroll-docbooktrue
scroll-eclipsehelptrue
scroll-htmltrue
scroll-epubtrue

Open in new tab

About Collectors

Insert excerpt
Collector Method
Collector Method
nameabout

...

The collector requires a set of parameters to connect to and extract metadata from BigQuery

FIELD

FIELD TYPE

DESCRIPTION

EXAMPLE

regions

list<string>

List of valid regions to inspect against for data, see https://cloud.google.com/bigquery/docs/locations for list of valid regions

“us”

projects

list<string>

List of project ids to inspect across the regions specified

“kada-data”

host

string

This is the host that was onboarded in K for BigQuery

“bigquery”

json_credentials

JSON

See permissions section on how to download the credentials json to assign to this value

Code Block
{
    "type": "service_account",
    "project_id": "kada-data",
    "private_key_id": "",
    "private_key": "",
    "client_email": "kada.iam.gserviceaccount.com",
    "client_id": "1234",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/kada.iam.gserviceaccount.com"
}

output_path

string

Absolute path to the output location where files are to be written

“/tmp/output”

mask

boolean

To enable masking or not

true

compress

boolean

To gzip the output or not

true

These parameters can be added directly into the run or you can use pass the parameters in via a JSON file.

...

Code Block
import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.athenabigquery import Extractor

get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger

_type = 'bigquery'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

parser = argparse.ArgumentParser(description='KADA BigQuery Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()

start_hwm, end_hwm = get_hwm(_typeargs.name)

ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

publish_hwm(_type, end_hwm)

...