K provides context and trust to your data items by helping you understand the lineage and relationships between data items. You can see where your data item sources data from, where it flows to, how it joins to other data items.
As part of the Oct 23 release of 5.33, K can now automatically detect column level lineage and help you understand how column level data is used, transformed, and derived throughout data processes.
We are in the process of refining the parsing process to reduce the time taken for the job to run and increase the accuracy.
During the beta release, there are the following known limitations:
?
Enabling column level lineage
K detects column level lineage by parsing code. Depending on the amount of code that needs to be reviewed by K, this process can take time.
To trigger this column lineage job, go to Settings → Batch Manager and select QUERY_LINEAGE
We recommend running the QUERY_LINEAGE job after hours with there are less jobs running. If your platform has ~X queries, the job may take between X - X hrs