dbt - Automatic Data Lineage

What to Expect

dbt models

Dataedo shows column-level lineage for every model that dbt materializes (table, view, incremental, ...). Each model is treated as a processor that links its upstream objects to the objects it creates in the warehouse (see Processors and processes).

• Ephemeral models do not appear in lineage. When an ephemeral model feeds another model, lineage is created from upstream sources of the ephemeral model.
• source objects themselves do not appear in lineage even as processors; their materialized counterparts do.
• Column-level links come from SQL parsing.
• Additional object-level links come from dbt metadata (in case SQL parsing is not possible).

Same flow in dbt Docs

How it looks inside warehouse documentation (not dbt documentation)

• If you hide the processors (so-called business view), you will see a clean source-to-target flow without dbt objects.

• If you enable the processors (technical view), you will see how each object was created.

.csv file → dbt seed (processor) → materialized table/view

Dataedo creates object-level data lineage from a .csv file to its materialization in the warehouse with dbt seed (structure) as a processor.

Limitations

• Compiled SQL must be in the manifest – column-level lineage comes from parsing the compiled SQL that dbt stores in manifest.json. Artifacts from a command that only parses the project carry none of it, so only object-level lineage is created.
• Warehouse must be documented – automatic lineage appears only if the warehouse targeted by your dbt project has also been imported into Dataedo.
• Column-level lineage depends on SQL dialect – Dataedo can break lineage down to columns only when the Dataedo parser supports the warehouse's SQL dialect. See the SQL parsing overview for the current list of dialects.
• Materialized objects only – lineage is generated for models that dbt materializes. Raw source tables and ephemeral models themselves do not get lineage, although flows feeding into an ephemeral model are still traced.
• Linked sources must be mapped manually – any linked source left unassigned will produce no lineage.

Troubleshooting

Lineage missing right after the first import

See “After the first import — activate automatic lineage” in the dbt Cloud guide or the dbt Core guide. After mapping the linked sources, remember to run Import changes on the dbt documentation to populate the lineage.

No object-level data lineage

1. Check dbt Docs – If object-level lineage is absent in dbt Docs, it cannot be imported. See the dbt guide on viewing lineage.
2. Automatic lineage – Confirm the Automatic lineage checkbox is ticked in the connection form.
3. Warehouse imported – The target warehouse must also be in Dataedo.
4. Linked source mapped – Map the warehouse to its linked source (see Map each linked source to the right database in the Cloud/Core guides).
5. Run Import changes – Once mapping is correct, run Import changes to refresh lineage.
6. Contact support – If lineage is still missing, open a support ticket.

how to verify lineage is present

Open any non-ephemeral model and switch to the Data lineage tab.

Desktop: make sure you are viewing the dbt object, not the warehouse table.
Web: if you open a warehouse object, confirm it belongs to your dbt project and enable Show processors in the lineage tab.

Column-level lineage missing for some columns/objects

Some models have column-level lineage and others do not, or a model has it for only part of its columns.

1. Supported SQL features – check whether Dataedo's parser supports the SQL features used in those models. See the SQL parsing overview.
2. Contact support – if the features are supported yet lineage is still missing, open a support ticket and attach the compiled SQL of the problematic object.

No column-level data lineage

You see object-level lineage created from dbt metadata, but no column-level lineage at all.

1. Compiled SQL in the manifest – column-level lineage is built by parsing the compiled SQL that dbt writes into manifest.json. Artifacts from a command that only parses the project (dbt parse, dbt ls, dbt docs generate --no-compile) do not carry that SQL, so only object-level lineage can be created. Regenerate them with dbt docs generate, then run Import changes.
2. Parse SQL – make sure the Parse SQL checkbox is ticked in the connection form.
3. Dialect match – verify that each warehouse-linked source uses the correct SQL dialect. If Dataedo auto-assigns a similar but not identical dialect (e.g., PL/SQL for Db2), column-level lineage may be incomplete. See the SQL parsing overview for supported dialects.
4. Default schema – confirm the Default schema is set on the warehouse connections mapped to linked sources.
5. Contact support – if column-level lineage is still missing, open a support ticket, attach manifest.json and name one model that has no column-level lineage.

how to check the manifest for compiled SQL

Open manifest.json in a text editor and search for "compiled_code". If the phrase is not there at all, or every occurrence is null, the artifacts were not compiled.