The data processing module is structured around 3 objects, from the broadest container down to the most detailed link.
Which objects make up the module?
- Dataflow: used to hierarchize data processing. It cannot link elements of the dictionary together, which is why it's not possible to describe data processing in it.
- Data processing: used to describe links between one or many objects from the dictionary (databases, table, view...). Links are described with data processing units.
- Data processing element: the most detailed element of the module. It describes the link operated between different objects of the dictionary (one to x bases, containers, tables, fields...). It can be a simple link, a transformation, a filter...
How do the objects fit together?
Here is a data processing ("product load") that loads information (Bill of material, product category...) from Azure Datalake into the Enterprise Datawarehouse. As we can see:
- One data processing can contain many data processing elements, linking as many inputs to as many outputs. Each data processing element can explain one transformation.
- A data flow (here "Azure to EDWH") can contain many data processings.
