Page Comparison

About Collectors

...

Pre-requisites

...

Python 3.6 - 3.10

...

Collector Server Minimum Requirements

MySQL Requirements

• Access to MySQL. Supported MySQL versions include:

  • 5.7x

  • 8.0x

• Access to the following Information Schema tables

  1. INFORMATION_SCHEMA.VIEWS

  2. INFORMATION_SCHEMA.TABLES

  3. INFORMATION_SCHEMA.COLUMNS

  4. INFORMATION_SCHEMA.KEY_COLUMN_USAGE

...

Step 1: Enabling logging (if desired)

Enable global logging

SET GLOBAL log_output = 'TABLE';
SET GLOBAL general_log = 'ON';

...

SET @old_log_state = @@GLOBAL.general_log;
SET GLOBAL general_log = 'OFF';
ALTER TABLE mysql.general_log ENGINE = MyISAM;
ALTER TABLE mysql.slow_log ENGINE = MyISAM;
ALTER TABLE mysql.general_log ADD INDEX (event_time);
ALTER TABLE mysql.slow_log ADD INDEX (start_time);
SET GLOBAL general_log = @old_log_state;

Note this study Impact of General Query Log on MySQL Performance and the performance impact of enabling logging.

...

Step 2: Create the Source in K

...

The MySQL collector only extracts metadata and does not extract or process query usage on the database.

These parameters can be added directly into the run or you can use pass the parameters in via a JSON file. The following is an example you can use that is included in the example run code below.

...

This is the wrapper script: kada_mysql_analytics_extractor.py

import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.oracle_analyticsmysql import Extractor

get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger

_type = 'mysql'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

parser = argparse.ArgumentParser(description='KADA MySQL Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()

start_hwm, end_hwm = get_hwm(_typeargs.name)

ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

publish_hwm(_type, end_hwm)

...

If you wish to maintain your own high water mark files elsewhere you can use the above section’s script as a guide on how to call the extractor. The configuration file is simply the keyword arguments in JSON format. Refer to this document for more information https://kadaai.atlassian.net/wiki/spaces/KSL/pages/1902411777/AdditionalCollector+Integration+General+Notes#Storing-HWM-in-another-location

If you are handling external arguments of the runner yourself, you’ll need to consider additional items for the run method. Refer to this document for more information https://kadaai.atlassian.net/wiki/spaces/KSL/pages/1902411777/AdditionalCollector+Integration+General+Notes#The-run-method

from kada_collectors.extractors.mysql import Extractor

kwargs = {my args} # However you choose to construct your args
hwm_kwrgs = {"start_hwm": "end_hwm": } # The hwm values

ext = Extractor(**kwargs)
ext.run(**hwm_kwrgs)

...

If you want prefer file managed hwm, you can edit the location of the hwn by following these instructions Additional Collector Integration General Notes

...

Step 8: Push the Extracts to K

...