Processes in Data Lineage
A process is a part of the script, filling a purpose of data transformation or transit. Imagine a procedure that moves data from one table to another and logs the amount of moved rows. We can distinguish two processes here: one is moving the data, and the second one is logging it. You can, however, still decide to treat it as a single process that handles both actions at once.
In certain cases, especially with automatic processes concerned with multiple ways of representing the same data flow, one lineage connection can be represented by multiplied processes.
What is a Processor?
A processor is simply an object that can process data (perform a process). One processor can perform multiple processes, or just one. In Dataedo, you can designate any object as a processors, however some objects are better fit for the role then others. For example, since tables themselves only store data and don't process it, they usually should not be marked as processors.
Processor suggestions in Dataedo
The table below shows our suggestions regarding which objects to use as processors, and what are their uses:
| Object Type | Processes and Flows Limitations |
|---|---|
| View | Can be a processor, but with only one process. The view pulls the data itself, so it should have only one outflow, being the view itself. |
| Function | Functions can be processors, with an unlimited number of processes. |
| Procedure | Procedures can be processors, with an unlimited number of processes, each statement counting as one process. |
Processors and processes in Lineage
Processor are often integral to data flow and transformations. As such, it is possible to view them on Object's Lineage Diagrams. To do so, open Lineage Diagram's settings and switch to the via processors lineage view.
In this view, processors will be displayed as intermediaries between objects they process. Expanding a processor's view, will show which of its processes are involved in the data flow.
Lineage of processors
Since Data can flow through processors, the processors themselves have their own Lineage Diagrams.
If you expand a processor's view, you will be able to see all of the processes contained within it, and the data that flows in and out of each process.
Clicking on a process highlights all the objects that it processes.
Processor's Lineage is generated only if both its data inflow and outflow objects exist in Dataedo. If one of them is missing, the processors will still be visible in the Database lineage however the processor lineage will not be generated.
Processors and Manual Lineage
You can add processes and processors when defining additional lineage links by hand.
In Portal, when mapping two objects in relation to each other, you can select one of objects in your repository to act as a processor. This can be done by using the Add processor button in the object-mapping popup.
The processors you select in this view, will apply to all column pair in the column mapping popup. This means that the processors you select should be responsible for transforming all mapped columns.
If you add a processor this way, the processor's lineage will be adjusted to display the dataflows you established as manual lineage.
