From MIKE2.0 Methodology
Overview
Data lineage is a foundation capability of metadata management which enables and/or enhances all the other capabilities. It provides the functionality to determine where data comes from, how it is transformed, and where it is going. Data Lineage metadata traces the lifecycle of information between systems, including the operations that are performed upon the data. Enabling lineage tracing in a data integration environment has several benefits and applications invaluable to business and IT users:
- To meet compliance targets such as Basel II
- In-depth data analysis
- Authorization management
- Impact assessment support
- Conduct risk assessment
- Ensure data governance principles
- Enables reusability and standardisation
The need to establish a comprehensive view of Data Lineage has grown in importance over the past few years, particularly with renewed compliance requirements. The ability to trace lineage of data from producers to consumers will be an importance feature of the SAFE Architecture.
Image: Data Lineage across Systems in the Architecture
Impact analysis is a subset of data lineage which records and traces the movement of data and data relationships. Impact analysis is at a more technical level of detail than data lineage. Metadata management provides a data lineage view for both business and technical users. The data lineage function provides a mechanism to connect the business with technical terms by attaching the business rules to the technical transformations performed along the processing. The data lineage report provides the users with the ability to drill up and down through different levels of detail according to the desired degree of technicality.
In summary, Data Lineage:
- Shows the data flow from the data source, through various systems and staging areas, to the data destination.
- Determines where data comes from, how it is transformed, and where it is going.
Example: data lineage shows how data in a database table column eventually feeds a metric in a business intelligence report.
In summary, Impact Analysis:
- Shows all instances where an object is used within and across repositories.
- Provides more technical information than data lineage; useful for technical people
- Used before making changes to source repositories, such as business intelligence, data integration, or database repositories.
Example: if you want to drop a database table, you can use where-used analysis to determine what other objects in any repository use the information provided by the database table.
Data Lineage
In a data integration environment, data lineage provides the origin and subsequent processing history; this implies two forms of navigating the data lineage: moving backward to discover the ancestor items starting with the source system or moving forward to discover descendant items up to the target system.
Benefits of Data Lineage
Compliance initiatives such as Basel II Accord and Sarbanes-Oxley Act require companies to “collect, process and disclose the information required in the company's [financial] reports.” Audits ensure that data used in calculations are in the correct fields. When audits reveal any mistakes or lack of quality data, regulators must be able to track down (or trace) the exact path the data followed from its source of origin and understand how a data element is calculated. However, many different intermediate systems within the integration environment might call a piece of data by different terms and data lineage is a very useful concept in this context. Data lineage helps auditors discover sources and targets of data, and how the data came to have the value shown by documenting data elements, business rules (logic), and source to target mappings
Data Lineage Report
The data lineage report displays details at the field level enabling the auditing of data integration environment. For example, data lineage shows how data in a staging table column eventually feeds another column in a data warehouse or a BI report and the method used to calculate the resulting column.
Another view of the data lineage report shows the journey or path taken by a data element across various stages; each topic heading is technical metadata from various systems that let an auditor track the data's path.
Impact Analysis
Impact analysis enables the effective assessment of the impacts of changes on interdependent systems into a data integration environment. It shows all instances where an object is used within and across metadata sources and is a valuable capability for technical users. Because it provides a way to track how different data elements fit together, metadata enables the prediction the potential impacts caused by changes in one area of the system. Those changes might include the deletion or change to a specific standard Basel II data element, changes to a system of record or poor data quality measures in one of the subsystems.
Benefits of Impact Analysis
Impact analysis can be used in a number of different scenarios as it follows:
- Impact analysis can also be used for data placement strategy (for example, when a new source of data is available, impact analysis can be used to decide if the data should be placed in its own database, shared on the data warehouse, etc).
- Impact analysis can also facilitate the estimation of the effects of replacing one application with another or changing business definitions, with a clearer view of what is involved and what can be optimized.
- Implementing impact analysis aids the development groups by providing a mechanism for development and testing of systems and by understanding the effect of proposed changes into an integrated environment.
| BUSINESS/TECHNICAL VALUE
| BENEFIT MEASURES
|
| Reduction of IT related problems
| IT staff much less likely to make programming errors when making system enhancements, since all affected programs, tables/files, and fields are identified
|
| Reduce IT development lifecycles and costs
| IT development lifecycles are greatly reduced, since all affected programs, tables/files, and fields are identified
|
| Reduce redundant data
| IT staff can identify redundant data in systems, and reduce the likelihood of building redundant systems or populating systems with redundant data
|
| Reduce redundant processes
| IT staff can identify redundant processes in systems, and reduce the likelihood of building redundant system processes in the first place
|
| Reduce impact of employee turnover
| Documents the knowledge that may otherwise be known only by the developer who built the programs, and makes it available to the entire IT staff
|
| Improved system performance
| Removes redundant data and processes, thereby improving system performance
|
Impact Analysis Report
Impact analysis reporting minimizes the costs of the system enhancement and helps to reduce the propensity of new development errors.
The information of the impact analysis can be reported at different levels of granularity as it follows: application system, application system table, application system field and table type. It is important to many people within the organization, in general technical staff including project managers, database administrators, programmers, data modelers, and business analysts.
In an integrated environment data is collected from the operational systems of a business. It is quite common for these operational systems to undergo changes in their business rules and data structures, and these changes can directly impact the integrated environments that they feed. Impact analysis reports help to control the effect of these changes.
