Column Lineage limitations
- Dean Nguyen
- Ben
- Sidney Chen
K processes lineage across numerous sources however there are some scenarios where lineage and the grain of lineage is limited.
This page will cover known limitations and any remediation available or planned.
| Issue | Example | Fix | Workaround |
|---|---|---|---|---|
1 | Nested SELECT * from a CTE not supported | CREATE TABLE MAIN.FOOBAR AS (
WITH FOO AS (
SELECT COL1, COL2 FROM FROM MAIN.BAR
)
SELECT * FROM FOO
) | Under investigation | Use Bulk Lineage Mapper to map columns between source and target tables. |
2 | SELECT * with Unions not supported | CREATE TABLE MAIN.FOOBAR AS (
SELECT * FROM MAIN.BAR
UNION
SELECT * FROM MAIN.BAR2
UNION
SELECT * FROM MAIN.BAR3
) | Under investigation |
|
3 | Insert into statements currently skipped in processing | INSERT INTO FOOBAR (
COL1, COL2 )
SELECT COL1, COL2 FROM FOO
) | Investigation completed. Fix pending. Target release in Q1 25 |
|
4 | Copy grants in statements currently causes statement to be skipped in processing | Investigation completed Fix pending Target release in Q1 25 |
| |
5 | Code that is not currently wrapped in a CREATE statement e.g. DBT code from manifest (represented as a select statement) |
| Investigation completed Fix pending Target release in Q1 25 |
|