| Scroll ignore | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
About Collectors
| Insert excerpt | ||||||
|---|---|---|---|---|---|---|
|
...
Pre-requisites
Python 3.6 - 3.10
Access to K landing directory
Access to SQL Server (see section below)
...
Collector server minimum requirements
| Insert excerpt | ||||||||
|---|---|---|---|---|---|---|---|---|
|
SQL Server Requirements
Setting up SQL Server for metadata extraction is a 2 step process.
...
| Code Block |
|---|
CREATE LOGIN kadauser WITH password='PASSWORD'; CREATE USER kadauser FROM LOGIN kadauser; |
| Note |
|---|
Apply per database in scope for metadata collection. |
| Code Block |
|---|
CREATE USER kadauser FROM LOGIN kadauser;
GRANT VIEW DEFINITION TO kadauser;
GRANT VIEW DATABASE STATE to kadauser;
GRANT CONTROL to kadauser; -- required for extended events sys.fn_xe_file_target_read_file |
The following table should also be available to SELECT by the user created in each database
...
The collector requires a set of parameters to connect to and extract metadata from SQLServer Azure.
FIELD | FIELD TYPE | DESCRIPTION | EXAMPLE |
|---|---|---|---|
server | string | SQLServer Azure server. If using a custom port append with comma | “mydatabase.database.windows.net” |
host | string | The onboarded host value in K, generally this would be the same as the server value, depending on what you onboard it as. | |
username | string | Username to log into the SQLServer Azure account | “myuser” |
password | string | Password to log into the SQLServer Azure account |
|
databases | list<string> | A list of databases to extract from SQLServer Azure | [“dwh”, “adw”] |
driver | string | This is the ODBC driver, generally its ODBC Driver 17 for SQL Server, if you another driver installed please use that instead. | “ODBC Driver 17 for SQL Server” |
meta_only | boolean | Do you want to extract metadata only without enabling extended events? We currently only support true | true |
output_path | string | Absolute path to the output location where files are to be written | “/tmp/output” |
mask | boolean | To enable masking or not | true |
compress | boolean | To gzip the output or not | true |
events_name | string | The created extended event session name for each database, the event name should be exactly the same per database. This needs to be specified when meta_only is false | KADA |
These parameters can be added directly into the run or you can use pass the parameters in via a JSON file. The following is an example you can use that is included in the example run code below.
...
| Code Block | ||
|---|---|---|
| ||
import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.sqlserver_azure import Extractor
get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger
_type = 'sqlserver_azure'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))
parser = argparse.ArgumentParser(description='KADA SqlServer Azure Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()
start_hwm, end_hwm = get_hwm(_typeargs.name)
ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})
publish_hwm(_type, end_hwm) |
...