Versions Compared

Old Version 2

changes.mady.by.user Chichi

Saved on

compared with

New Version Current

changes.mady.by.user Sidney Chen

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Scroll ignore
scroll-viewporttrue
scroll-pdftrue
scroll-officetrue
scroll-chmtrue
scroll-docbooktrue
scroll-eclipsehelptrue
scroll-htmltrue
scroll-epubtrue

Open in new tab

About Collectors

Insert excerpt
Collector Method
Collector Method
nameabout

...

Note

This collector is a metadata only collector and no usage information can be extracted at this stage. So use set Ensure that the meta_only property value is set to true.

...

Step 1: Create the Source in K

...

Some python packages also have dependencies on the OS level packages, so you may be required to install additional OS packages if the below fails to install.

You can download the Latest latest Core Library and ThoughtSpot whl via Platform Settings → Sources → Download Collectors

...

You will also need to install the common library kada_collectors_lib -1.0.0 for this collector to function properly.

...

FIELD

FIELD TYPE

DESCRIPTION

EXAMPLE

username

string

Username to log into the Thoughtspot account

“myuser”

password

string

Password to log into the Thoughtspot account

 

org_ids

list<integer>

A list of organisation ids you wish to extract from

If you are unable to locate the Organisation ID through the Admin console, you may use this python script instead to list the IDs

Code Block
import requests
import json
BASE_URL = 'REPLACE BASE URL' # e.g. https://kada.thoughtspot.cloud
LOGIN_URL = BASE_URL + '/callosum/v1/tspublic/v1/session/login'  
ORG_URL = BASE_URL + '/callosum/v1/tspublic/v1/session/orgs'  
data = {
    'username': "REPLACE USERNAME",
    'password': "REPLACE PASSWORD"
}
with requests.Session() as session:
    session.post(LOGIN_URL, verify=True, data=data)

    res = session.get(ORG_URL, verify=True)
    res_json = json.loads(res.text)
    for org in res_json.get('orgs', []):
        print("Org Name: {}, Org Id: {}".format(org['orgName'], org['orgId']))

[123, 34]

output_path

string

Absolute path to the output location where files are to be written

“/tmp/output”

mask

boolean

To enable masking or not

true

timeout

integer

Timeout in seconds allowed against the powerbi APIs, for slower connections we recommend 30, default is 20

20

mapping

JSON

Mapping file of data source names against the onboarded host and database name in K

Assuming I have a “myDSN” data source name in powerbi, I’ll map it to host “myhost” and database “mydatabase” onboarded in K, snowflake type references are handled automatically

Code Block
{
        "myDSN": {
            "host": "myhost",
            "database": "mydatabase"
        }
    }

compress

boolean

To gzip the output or not

true

meta_only

trueboolean

To only extract metadata or not, we currently only support this to be truetrue. Only true is supported for this source

true

base_api_url

string

Should be in the form https://x.thoughtspot.cloud depending on your account

https://kada.thoughtspot.cloud

These parameters can be added directly into the run or you can use pass the parameters in via a JSON file. The following is an example you can use that is included in the example run code below.

...

Code Block
{
    "username": "",
    "password": "",
    "org_ids": [],
    "output_path": "/tmp/output",
    "timeout": 20,
    "mapping": {},
    "compress": true,
    "meta_only": true,
    "base_api_url": "https://sample.thoughtspot.cloud"
}

...

Step 6: Run the Collector

...

Code Block
import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.thoughtspot import Extractor

get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger

_type = 'teradatathoughtspot'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

parser = argparse.ArgumentParser(description='KADA TeradataThoughtspot Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
args = parser.parse_args()

start_hwm, end_hwm = get_hwm(_type)

ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

publish_hwm(_type, end_hwm)

...