| Scroll ignore | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
About Collectors
Collectors are extractors that are developed and managed by you (A customer of K).
...
Deploying and orchestrating the extract code
Managing a high water mark so the extract only pull the latest metadata
Storing and pushing the extracts to your K instance.
...
Pre-requisites
...
Python 3.7 - 3.11
...
Collector Server Minimum Requirements
| Insert excerpt | ||||||||
|---|---|---|---|---|---|---|---|---|
|
Cognos Requirements
Cognos access
Cognos Analytics user that has the ability to read all objects in Cognos
A SQL Authenticated user Database User for the underlying Audit Database configured for Cognos
Cognos auditing must be enabled (Log level - Basic)
...
FIELD | FIELD TYPE | DESCRIPTION | EXAMPLE | ||
|---|---|---|---|---|---|
server_url | string | Cognos server address domain including the protocol (e.g. | “https://10.1.19.15:9300” | ||
username | string | Username to log into Cognos server created in Step 1 | “cognos” | ||
password | string | Password to log into Cognos server for the user created in Step 1 |
| ||
namespace | string | The user namespace which the user will log into. By default the namespace is | “CognosEx” | ||
timeout | boolean | API timeout for Cognos APIs in seconds. | 20 | ||
db_host | string | IP address or address of the Audit database. | “10.1.19.15” | ||
db_username | string | Username for the Audit database created in Step 2 | “kada” | ||
db_password | list<string> | Password for the database user created in Step 2 |
| ||
db_port | integer | Default is usually 1433 for SQLServer | 1433 | ||
db_name | string | Database name where the audit tables are stored | “Audit” | ||
db_schema | string | Schema name where the audit tables are stored | dbo | ||
db_driver | string | Driver name must match the one installed on the collector machine | “ | ||
db_use_kerberos | boolean | Does the database request impersonation, e.g. Kerberos | false | ||
meta_only | boolean | For meta only set this to true otherwise leave it as false. If you do not have access to the Audit database then set this to true | false | ||
output_path | string | Absolute path to the output location where files are to be written | “/tmp/output” | ||
mask | boolean | To enable masking or not | true | ||
mapping | json | This should be populate with the mapping.json output where each data source name mentioned is mapped to an onboarded K host Leave this empty ( | Where analytics.adw is the onboarded database in K
| ||
compress | boolean | To gzip the output or not | true |
...
| Code Block | ||
|---|---|---|
| ||
{
"server_url": "http://xxx:9300",
"username": "",
"password": "",
"namespace": "",
"timeout": 20,
"db_host": "",
"db_username": "",
"db_password": "",
"db_port": 8060,
"db_name": "",
"db_schema": "",
"meta_db_use_kerberos": false,
"meta_only": false,
"output_path": "/tmp/output",
"mask": false,
"mapping": {},
"compress": false
} |
...
| Code Block | ||
|---|---|---|
| ||
import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.cognos import Extractor
get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger
_type = 'cognos'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))
parser = argparse.ArgumentParser(description='KADA Cognos Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()
start_hwm, end_hwm = get_hwm(_typeargs.name)
ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})
publish_hwm(_type, end_hwm) |
...
| Code Block |
|---|
class Extractor(
server_url: str = None,
username: str = None,
password: str = None,
namespace: str = None,
timeout: int = 20,
db_host: str = None,
db_username: str = None,
db_password: str = None,
db_port: int = None,
db_name: str = None,
db_schema: str = None,
db_driver: str = None,
db_use_kerberos: bool = None,
meta_only: bool = False,
mapping: dict = {},
output_path: str = './output',
mask: bool = False,
compress: bool = False
) -> None |
server_url: Cognos API URL
username: Cognos API Username
password: Cognos API Password
namespace: Cognos API Namespace
timeout: Cognos API Timeout
db_host: Database host
db_username: Database username
db_password: Database password
db_port: Database port
db_name: Database name
db_schema: Database schema
db_use_kerberos: Database impersonation required
meta_only: Only extract metadata
mapping: Mapping for the metadata
output_path: Output path for the files
mask: Mask the data
compress: Compress the data
...