Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Scroll ignore
scroll-viewporttrue
scroll-pdftrue
scroll-officetrue
scroll-chmtrue
scroll-docbooktrue
scroll-eclipsehelptrue
scroll-htmltrue
scroll-epubtrue

Open in new tab

About Collectors

Insert excerpt
Collector Method
Collector Method
nameabout

...

Pre-requisites

  • Python 3.6 8 - 3.1011

  • Access to K landing directory

  • Access to SQL Server (see section below)

  • Check the SQLServer instance port

    • Run the following query and note the local tcp port.

      Code Block
      SELECT local_tcp_port
      FROM   sys.dm_exec_connections
      WHERE  session_id = @@SPID
      GO

...

  • INFORMATION_SCHEMA.ROUTINES

  • INFORMATION_SCHEMA.VIEWS

  • INFORMATION_SCHEMA.TABLE_CONSTRAINTS

  • INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE

  • INFORMATION_SCHEMA.TABLES

  • INFORMATION_SCHEMA.COLUMNS

  • INFORMATION_SCHEMA.VIEWS

  • sys.foreign_key_columns

  • sys.objectobjects

  • sys.tables

  • sys.schemas

  • sys.columns

  • VIEW SERVER STATE permission on the server

    • Required for Extended Event log

  • VIEW Definition

    • All databases

      Code Block
      USE master 
      GO 
      GRANT VIEW ANY DEFINITION TO Kadauser
    • Selected databases. Repeat for each database

      Code Block
      USE <REPLACE WITH A DATABASE> 
      GO 
      GRANT VIEW ANY DEFINITION TO Kadauser

...

Code Block
breakoutModewide
--Query To Create Extended Events Session
CREATE EVENT SESSION [KADA] ON SERVER ADD EVENT sqlserver.sp_statement_completed (
	ACTION(package0.collect_system_time, package0.event_sequence, sqlos.task_time, sqlserver.client_app_name, sqlserver.client_hostname, sqlserver.database_id, sqlserver.database_name, sqlserver.nt_username, sqlserver.query_hash, sqlserver.server_instance_name, sqlserver.server_principal_name, sqlserver.server_principal_sid, sqlserver.session_id, sqlserver.session_nt_username, sqlserver.transaction_id, sqlserver.username) WHERE (
		(
			[statement] LIKE '%CREATE %'
			OR [statement] LIKE '%DROP %'
			OR [statement] LIKE '%MERGE %'
			OR [statement] LIKE '%FROM %'
			)
		AND [sqlserver].[server_principal_name] <> N'USERS_TO_EXCLUDE'
		AND [sqlserver].[is_system] = (0)
		AND NOT [statement] LIKE 'Insert into % Values %'
		AND [sqlserver].[Query_hash] <> (0)
		)
	), ADD EVENT sqlserver.sql_statement_completed (
	SET collect_statement = (1) ACTION(package0.collect_system_time, package0.event_sequence, sqlos.task_time, sqlserver.client_app_name, sqlserver.client_hostname, sqlserver.database_id, sqlserver.database_name, sqlserver.nt_username, sqlserver.query_hash, sqlserver.server_instance_name, sqlserver.server_principal_name, sqlserver.session_id, sqlserver.session_nt_username, sqlserver.transaction_id, sqlserver.username) WHERE (
		(
			[statement] LIKE '%CREATE %'
			OR [statement] LIKE '%DROP %'
			OR [statement] LIKE '%MERGE %'
			OR [statement] LIKE '%FROM %'
			)
		AND [sqlserver].[server_principal_name] <> N'N'USERS_TO_EXCLUDE'
		AND [sqlserver].[is_system] = (0)
		AND NOT [statement] LIKE 'Insert into % Values %'
		AND [sqlserver].[Query_hash] <> (0)
		)
	) ADD TARGET package0.event_file (SET filename = N'G:\extended events\Extendedevents.xel', max_file_size = (20), max_rollover_files = (100))
	WITH (MAX_MEMORY = 4096 KB, EVENT_RETENTION_MODE = ALLOW_MULTIPLE_EVENT_LOSS, MAX_DISPATCH_LATENCY = 30 SECONDS, MAX_EVENT_SIZE = 0 KB, MEMORY_PARTITION_MODE = NONE, TRACK_CAUSALITY = ON, STARTUP_STATE = ON)
GO


-- Check if the session is dropping events and see other data about the session
-- https://sqlperformance.com/2019/10/extended-events/understanding-event-loss-with-extended-events
SELECT
   s.name, 
   s.total_regular_buffers,
   s.regular_buffer_size,
   s.total_large_buffers,
   s.large_buffer_size,
   s.dropped_event_count,
   s.dropped_buffer_count,
   s.largest_event_dropped_size
FROM sys.dm_xe_sessions AS s;


-- Also check log growth rate. Apply filters to remove noise.
-- some filters:
-- [sqlserver].[server_principal_name] = N'name of principal'
-- [sqlserver].[is_system] = (0)
-- [sqlserver].[client_app_name] = N'name of app'

...

Some python packages also have dependencies on the OS level packages, so you may be required to install additional OS packages if the below fails to install.

You can download the latest Latest Core Library and SQL Server whl via Platform Settings → SourcesDownload Collectors

...

Code Block
pip install kada_collectors_extractors_<version>sqlserver-3.0.0-py3-none-any.whl

You will also need to install the common library kada_collectors_lib-1.0.0 for this collector to function properly.

Code Block
pip install kada_collectors_lib-<version>1.0.0-py3-none-any.whl
Info

Note that you will also need an ODBC package installed at the OS level for pyodbc to use as well as a SQLServer ODBC driver, refer to https://docs.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver15

...

The collector requires a set of parameters to connect to and extract metadata from SQL Server.

FIELD

FIELD TYPE

DESCRIPTION

EXAMPLE

server

string

SQLServer server.

If using a custom port append with comma Example: 10.1.1.23,5678

“10.1.18.19”

host

string

The onboarded host value in K, generally this would be the same as the server value, depending on what you onboard it as.

“mysqlserver”

username

string

Username to log into the SQLServer account

“myuser”

password

string

Password to log into the SQLServer account

 

databases

list<string>

A list of databases to extract from SQLServer

[“dwh”, “adw”]

sqlserver_version

string

Version of SQLServer release name, supported is 2012, 2016, 2017, 2019

2016

driver

string

This is the ODBC driver, generally its ODBC Driver 17 for SQL Server, if you another driver installed please use that instead

“ODBC Driver 17 for SQL Server”

meta_only

boolean

Do you want to extract metadata only without enabling extended events?

true

events_path

string

This is the extended events file pattern configuration for SQLServer. Only required if meta_only is false.

“/tmp/eevents*.xel”

output_path

string

Absolute path to the output location where files are to be written

“/tmp/output”

mask

boolean

To enable masking or not

true

compress

boolean

To gzip the output or not

true

These parameters can be added directly into the run or you can use pass the parameters in via a JSON file. The following is an example you can use that is included in the example run code below.

...

Code Block
languagejson
{
    "server": "",
    "usernamehost": "",
    "passwordusername": "",
    "databasespassword": [""],
    "hostdatabases": [""],
    "sqlserver_version": "2016",
    "driver": "ODBC Driver 17 for SQL Server",
    "events_path": "/tmp/Extendedevents*.xel",
    "output_path": "/tmp/output",
    "mask": true,
    "compress": true
}

...

,
    "meta_only": true
}

...

Step 5: Run the Collector

The following code is an example of how to run the extractor. You may need to uplift this code to meet any code standards at your organisation.

...

Code Block
languagepy
import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.sqlserver import Extractor

get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger

_type = 'sqlserver'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

parser = argparse.ArgumentParser(description='KADA SqlServer Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
args =
parser.parseadd_args()

start_hwm, end_hwm = get_hwm(_type)

ext = argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()

start_hwm, end_hwm = get_hwm(args.name)

ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

publish_hwm(_type, end_hwm)

Note

In some scenarios, you may receive an error message : pyodbc.OperationalError: ('08001', '[08001] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x2746 (10054) (SQLDriverConnect)')about the SSL settings.

This error can be resolved via the Open SSL settings. Refer to: https://github.com/mkleehammer/pyodbc/issues/610#issuecomment-534920201

  • Code Block
    Edited /etc/ssl/openssl.cnf 
    
    # Change or add
    
    MinProtocol = TLSv1.0
    
    CipherString = DEFAULT@SECLEVEL=1

...

Code Block
languagepy
class Extractor(username: str = None, password: str = None, server: str = None, \
 
  host: str = None, driver: str = None, events_path: str = None, databases: list = [], \
    sqlserver_version: str = None, output_path: str = './output', mask: bool = False, \
    compress: bool = False, meta_only: bool = False) -> None

username: username to sign into sqlserver
password: password to sign into sqlserver
server: sqlserver host
host: the onboarded host value in K, generally it will be the same as the server
driver: sqlserver driver name
events_path: regex location of the events files on the server
databases: list of databases to extract
sqlserver_version: Release name for the SQLServer supported is 2012, 2016, 2017, 2019
output_path: full or relative path to where the outputs should go
mask: To mask the META/DATABASE_LOG files or not
compress: To gzip output files or not
meta_only: To extract without extended events enabled

...