Page Comparison

...

1. Deploying and orchestrating the extract code

2. Managing a high water mark so the extract only pull the latest metadata

3. Storing and pushing the extracts to your K instance.

...

Pre-requisites

...

Collector Server Minimum Requirements

SSIS Requirements

MSDB database / SQLServer DB access

• if using SSISDB to store package you will need to download each project via SQL Server Management Studio. See https://kadaai.atlassian.net/wiki/spaces/KSL/pages/1960706049/SSIS+Collector+method+-+v3.1.0#Downloading-SSISDB-packages

  • The collector will need access to the underlying SQLServer Database with permissions to read the following tables is the SSIS main databases:

    • MSDB.DBO.SYSSSISPACKAGES<SSIS Logging Database>.DBO.SYSSSISLOG where <SSIS Logging Database> is the database configured for SSIS logging

• Access to K landing directory

• Access to the KADA Collector repository that contains the SSIS whl

  • The repository is currently hosted in KADA’s Azure Blob Storage. You will be given a SAS token to access the repository. Reach out to KADA Support (support@kada.ai) if you do not have access.

  • Download the SSIS whl (e.g. kada_collectors_extractors_ssis-#.#.#-py3-none-any.whl)

    • .SYSSSISPACKAGES

    • <SSIS Logging Database>.DBO.SYSSSISLOG where <SSIS Logging Database> is the database configured for SSIS logging

• Check your SSIS instance port

  • Run the following query and note the local tcp port.

    Code Block
    SELECT local_tcp_port FROM sys.dm_exec_connections WHERE session_id = @@SPID GO

Downloading SSISDB packages

1. In SQL Server Studio Manager Go to Integration Services Catalog > Projects.

   

2. Right click on a Project > Export…
   

   

3. Save project file

   

4. Change the file extraction from .ispac to .zip

   

5. Repeat for each project.

...

Some python packages also have dependencies on the OS level packages, so you may be required to install additional OS packages if the below fails to install.

You can download the latest Core Library and whl via Platform Settings → Sources → Download Collectors

...

Run the following command to install the collector

pip install kada_collectors_extractors_ssis-3.1.0-py3<version>-none-any.whl

You will also need to install the common library kada_collectors_lib-1.0.0 lib for this collector to function properly.

pip install kada_collectors_lib-1.0.0-py3<version>-none-any.whl

...

The collector requires a set of parameters to connect to and extract metadata from SSIS.

{
        "myDSN": {
            "host": "myhost",
            "database": "mydatabase"
        }
    }

These parameters can be added directly into the run or you can use pass the parameters in via a JSON file. The following is an example you can use that is included in the example run code below.

...

import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.ssis import Extractor

get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger

_type = 'ssis'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

parser = argparse.ArgumentParser(description='KADA SSIS Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()

start_hwm, end_hwm = get_hwm(_typeargs.name)

ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

publish_hwm(_type, end_hwm)

...