Page Comparison

About Collectors

...

Pre-requisites

...

Python 3.6 - 3.10

...

Access to the KADA Collector repository that contains the MySQL whl

• The repository is currently hosted in KADA’s Azure Blob Storage. You will be given a SAS token to access the repository. Reach out to KADA Support (support@kada.ai) if you do not have access.

• Download the MySQL whl (e.g. kada_collectors_extractors_mySQL-#.#.#-py3-none-any.whl)

...

Access to K landing directory

...

Collector Server Minimum Requirements

MySQL Requirements

• Access to MySQL. Supported MySQL versions include:

  • 5.7x

  • 8.0x

• Access to the following Information Schema tables

  1. INFORMATION_SCHEMA.VIEWS

  2. INFORMATION_SCHEMA.TABLES

  3. INFORMATION_SCHEMA.COLUMNS

  4. INFORMATION_SCHEMA.KEY_COLUMN_USAGE

...

Step 1: Enabling logging (if desired)

Enable global logging

SET GLOBAL log_output = 'TABLE';
SET GLOBAL general_log = 'ON';

...

SET @old_log_state = @@GLOBAL.general_log;
SET GLOBAL general_log = 'OFF';
ALTER TABLE mysql.general_log ENGINE = MyISAM;
ALTER TABLE mysql.slow_log ENGINE = MyISAM;
ALTER TABLE mysql.general_log ADD INDEX (event_time);
ALTER TABLE mysql.slow_log ADD INDEX (start_time);
SET GLOBAL general_log = @old_log_state;

Note this study Impact of General Query Log on MySQL Performance and the performance impact of enabling logging.

...

Step 2: Create the Source in K

...

Some python packages also have dependencies on the OS level packages, so you may be required to install additional OS packages if the below fails to install.

You can download the latest Core Library and whl via Platform Settings → Sources → Download Collectors

...

Run the following command to install the collector

pip install kada_collectors_extractors_mysql-3.0.0-py3-<version>-none-any.whl

You will also need to install the common library kada_collectors_lib -1.0.0 for this collector to function properly.

pip install kada_collectors_lib-1.0.0-py3<version>-none-any.whl

...

Step 5: Configure the Collector

...

The MySQL collector only extracts metadata and does not extract or process query usage on the database.

These parameters can be added directly into the run or you can use pass the parameters in via a JSON file. The following is an example you can use that is included in the example run code below.

...

This is the wrapper script: kada_mysql_analytics_extractor.py

import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.oracle_analyticsmysql import Extractor

get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger

_type = 'mysql'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

parser = argparse.ArgumentParser(description='KADA MySQL Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()

start_hwm, end_hwm = get_hwm(_typeargs.name)

ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

publish_hwm(_type, end_hwm)

...

If you wish to maintain your own high water mark files elsewhere you can use the above section’s script as a guide on how to call the extractor. The configuration file is simply the keyword arguments in JSON format. Refer to this document for more information https://kadaai.atlassian.net/wiki/spaces/KSL/pages/1902411777/AdditionalCollector+Integration+General+Notes#Storing-HWM-in-another-location

If you are handling external arguments of the runner yourself, you’ll need to consider additional items for the run method. Refer to this document for more information https://kadaai.atlassian.net/wiki/spaces/KSL/pages/1902411777/AdditionalCollector+Integration+General+Notes#The-run-method

from kada_collectors.extractors.mysql import Extractor

kwargs = {my args} # However you choose to construct your args
hwm_kwrgs = {"start_hwm": "end_hwm": } # The hwm values

ext = Extractor(**kwargs)
ext.run(**hwm_kwrgs)

...

If you want prefer file managed hwm, you can edit the location of the hwn by following these instructions Additional Collector Integration General Notes

...

Step 8: Push the Extracts to K

...