| Scroll ignore | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
About Collectors
| Insert excerpt | ||||||
|---|---|---|---|---|---|---|
|
...
This is the wrapper script: kada_mysql_analytics_extractor.py
| Code Block |
|---|
import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.mysql import Extractor
get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger
_type = 'mysql'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))
parser = argparse.ArgumentParser(description='KADA MySQL Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()
start_hwm, end_hwm = get_hwm(args.name)
ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})
publish_hwm(_type, end_hwm) |
...
If you wish to maintain your own high water mark files elsewhere you can use the above section’s script as a guide on how to call the extractor. The configuration file is simply the keyword arguments in JSON format. Refer to this document for more information https://kadaai.atlassian.net/wiki/spaces/KSL/pages/1902411777/AdditionalCollector+Integration+General+Notes#Storing-HWM-in-another-location
If you are handling external arguments of the runner yourself, you’ll need to consider additional items for the run method. Refer to this document for more information https://kadaai.atlassian.net/wiki/spaces/KSL/pages/1902411777/AdditionalCollector+Integration+General+Notes#The-run-method
| Code Block |
|---|
from kada_collectors.extractors.mysql import Extractor
kwargs = {my args} # However you choose to construct your args
hwm_kwrgs = {"start_hwm": "end_hwm": } # The hwm values
ext = Extractor(**kwargs)
ext.run(**hwm_kwrgs) |
...
If you want prefer file managed hwm, you can edit the location of the hwn by following these instructions Additional Collector Integration General Notes
...
Step 8: Push the Extracts to K
...