Metadata Management Solution Offering
From MIKE2.0 Methodology
-> You are here: Metadata Management Solution Offering
Metadata Management is a key aspect of the MIKE2.0 approach to Information Development. Without managing your metadata, you cannot manage your information.
Many organisations fail to recognise the importance of metadata management (MM) . As a result, most have, to be blunt, traditionally done a very poor job at it. One of the challenges is that MM is often presented in different contexts. Many organisations have typically seen it as an afterthought, even though activities such as building a data dictionary have always been upfront activities.
On the positive side, recent business and technology changes have moved MM to the forefront. Regulatory requirements such as Basel II, for example, have required organisations to understand the lineage of data as it flows across the environment. Solutions that provide a comprehensive analytical view of an organisation’s metadata environment are referred to in MIKE2.0 as a Metadata Warehouse.
From a technology perspective, many software and IT vendors have re-architected their products over the last few years to take a metadata-driven approach. While initiated by some forward-thinking smaller players, is now a core part of the approach of the largest IT providers. That is, they have made MM more of an integrated by-product of the software development process. No longer an afterthought and a separate activity, this approach can provide a fundamental improvement in software development practices. In MIKE2.0, this set of collective set of techniques and technology is referred to as Active Metadata Integration.
Organisations that want to take a focus on Information Development (ID) need to make MM one of their key focus points. Considering all the changes over the past few years, before defining a solution, it is best to have a clear definition on the subject.
Solution Offering Purpose
Foundational Solutions are the lowest assets within MIKE2.0 of a comprehensive nature. They may tie together multiple Supporting Assets. What's more, they are referenced by the Overall Implementation Guide and other Solution Offerings.
Solution Offering Relationship Overview
MIKE2.0 Solutions provide a detailed and holistic way of addressing specific problems. They can be mapped directly to the Phases and Activities of the MIKE2.0 Overall Implementation Guide (OIG), providing additional content to help understand the overall approach.
The MIKE2.0 Overall Implementation Guide explains the relationships between the Phases, Activities, and Tasks of the overall methodology. The OIG also shows how the Supporting Assets tie to the overall Methodology and MIKE2.0 Solutions.
Users of the MIKE2.0 Methodology should always start with the OIG and the MIKE2.0 Usage Model as a starting point for projects.
Solution Offering Definition
What is Metadata
Metadata is data and information about data. It provides the intelligence to be able to link pieces of data, and also facilitates the ability to trace the path of data through the different stages of the supply chain – from source to publication - both internally as well as externally.
Metadata describes the data and the transformation rules that it undergoes at every step –in business terms as well as technical terms.
Business Metadata consists of “definitional” requirements and rules. It includes:
Technical Metadata can come in many forms. It includes:
Most business metadata has to be manually captured as a starting point whereas some technical metadata is generated (sometimes referred to as operational metadata). Technical Metadata is generally in a number of sources across the information environment. Some metadata, such as data quality metrics of attributes could arguably be classified as business or technical metadata.
What is Metadata Management?
Metadata management is the mechanism for business and technical metadata to be properly defined, shared and integrated across the environment. The MIKE2.0 Methodology presents two comprehensive solutions to manage metadata:
Solutions may involve centralised or distributed metadata architecture. Metadata-Driven Integration is the more advanced of the two techniques and would also include delivery of a Metadata Warehouse.
What are the Benefits?
A piece of data is meaningless unless it is defined and put into context. Metadata helps to put a context around data and transform it to information. Some of the key benefits include:
A more detailed list of business drivers for better metadata management helps explain this concept further.
What is the Approach?
Better Metadata Management will typically require a shift in development practices. For the Metadata Warehouse, the shift is somewhat gradual. For Active Metadata Integration, it will typically be a significant shift to how an organisation is developing software. The MIKE2.0 Solution for Metadata Management presents a set of activities and techniques for both.
For either approach it is important to remember that metadata management is not just about technology, applications and tools. Metadata management requires a cultural shift and a methodology that defines processes, roles and responsibilities. Where possible, technology will be used to enforce or guide the methodology, but technology alone will not provide a solution.
It is also important to remember that metadata management is an evolutionary process. A complete enterprise metadata management framework cannot be deployed in one single motion. Metadata existed before the term became popular among information and data management professionals. An information architecture always uses metadata, but the simple existence of metadata does not mean that all metadata needs to be managed. Metadata management should start with immediate needs and be implemented based on real business and technical requirements.
How it fits into the Overall MIKE2.0 Methodology
The MIKE2.0 Solution for Metadata Management defines an approach that can be used to manage business and technical metadata across the complete enterprise environment. It provides a comparison of techniques and key activities as they relate to the Overall Implementation Guide.
Comparison of Techniques
The key distinction between the approaches is that Active Metadata Integration automates the outputs from the analysis and design process into the build. Even with Active Metadata Integration solutions will involve some form of manual integration. With a manual process the probability of integration errors are much greater and any single error can be very costly to trace and recover from. Therefore, the techniques employed should progressively automate the metadata management process as much as possible.
The Metadata Warehouse
When building a metadata warehouse, the goal is to deliver a solution through a pragmatic approach to relieve immediate and medium-term metadata management pain points while keeping focused on long-term metadata management objectives. Just like a Data Warehouse, the solution should be designed to evolve into an infrastructure with wider and more strategic reach across the organisation. The design should be open so that interfaces, components and databases can be replaced and immediate priorities quickly resolved.
Active Metadata-Driven Approach
Active Metadata Integration extends the concept of being able to link metadata flows into and out of a managed metadata environment in an analytical fashion by making it an integrated part of the design process. This active approach to metadata management makes use of a model-driven approach that improves consistency, quality and reuse utility, by automating the integration between the sources of record for metadata. Metadata Integration itself is made model-driven by interfacing with Model Management capabilities in the architecture to provide a framework for repeatable development processes and reusable components to integrate metadata.
Relationship to Solution Capabilities
Relationship to Enterprise Views
Metadata Management is a key enabler for Information Development that goes across people, process, organisation and technology. Best practices in metadata management typically require significant changes to the competencies that are employed within an organisation and the skills required to implement these competencies. New roles may be required specifically to take ownership of metadata management, such as the Metadata Manager within an Information Development Organisation.
Significant technology changes are also typically required; metadata management is one of the areas where vendor technologies have made their most significant advances in recent years.
The approach is driven by an overall strategy, defined in the Business and Technology Blueprint.
Mapping to the Information Governance Framework
Mapping to the SAFE Architecture Framework
The SAFE Architecture provides a number of components that relate specifically to metadata management. The aim is to eventually move to a metadata-driven approach across the architecture, but this Active Metadata Integration may not be followed from the onset.
Metadata management is first introduced as one of the Foundation Capabilities for Information Development within the architecture, introduced through the concept of a Data Dictionary. More advanced capabilities within the architecture include Active Metadata Integration. As metadata is distributed across technology in the application, infrastructure and information environment a number of component capabilities are required to manage the metadata environment. The key architectural component for the Metadata-driven approach is some form of repository for storing artifacts and supporting modelling tools to create data models, meta-models and to report on this information. There are vendor offerings in this space and many organisations build bespoke repository-based solutions.
Mapping to the Overall Implementation Guide
Building a comprehensive solution for metadata management requires activities across all 5 phases of the MIKE2.0 Methodology. Oftentimes, it is done in the context of a larger project as opposed to a standalone project to build a metadata repository. For purposes of explanation in relation to the Overall Implementation Guide, however, it has been explained below as a project in isolation.
Taking a metadata-driven approach is the more sophisticated of the two techniques and has a significant impact on the software development process. The metadata-driven approach is explained in relation to defining a metadata warehouse.
Business Assessment and Strategy Definition Blueprint (Phase 1)
All activities from the Business Blueprint phase of MIKE2.0 will be required to define the strategy for a comprehensive metadata management solution. In a large, complex organisation it will require engagement with a high number of stakeholders and may result in significant organisational change. The key activities from the Business Blueprint are as follows:
Enterprise Information Management Awareness
Building awareness on the need for metadata management will often be important, particularly if it is in the context of another project. This activity should be undertaken with key stakeholders and can use training materials from MIKE2.0 to conduct workshops.
Overall Business Strategy for Information Development
In the Business Blueprint phase, the focus is on developing an initial Information Development strategy that is aligned to the specific set of business requirements, many of which are driven from the Application Development stream. The Overall Business Strategy for Information Development should also involve the initial gathering of business metadata as part of defining the conceptual model.
Organisational QuickScan for Information Development
IM QuickScan provides a comprehensive assessment of information management maturity, which includes specific questions related to metadata management. To complement results from the QuickScan assessment, MIKE2.0 also provides a summary view of metadata management maturity that shows the key criteria for comparing different maturity levels across organisations.
Future-State Vision for Information Management
The SAFE Architecture is used as a starting point to define the strategic conceptual architecture at the component level and a set of high level solution architecture options. The architectural overlay to the SAFE Architecture that focuses on metadata management can be used to frame the key set of capabilities required as part of the architecture.
Initial Data Governance Organisation
Most organisations will require a large degree of structural change to move to a mature Information Development organisation. It is starting in Phase 1 that we want to begin to “re-balance” the Enterprise to take more of a focus on the development of information. As part of the MIKE2.0 methodology, the aim is develop an [[ Information Development Organisation to complement more traditional organisational models around application development and infrastructure. This organisation should have roles and responsibilities specifically aligned to Metadata Management and this information stored in a repository.
Business Blueprint Completion
When formulating the overall business case and programme plan as part of Business Blueprint Completion, it is important to emphasize the need for a foundation of metadata management capabilities to be delivered in the early stages of the programme. Much of the financial business case related to improved operational efficiency will derive from improved metadata management practices. Making metadata management a mandatory part of fulfilling the business case helps to ensure it will not be short-cut during technology selection.
Technology Assessment and Selection Blueprint (Phase 2)
All activities from the Technology Blueprint phase of MIKE2.0 will be required to define a comprehensive approach to metadata management, which will typically involve the implementation of new technologies and establishing standards for how technology will be implemented. In phase 2 of MIKE2.0, a diligent approach is applied to establish the technology requirements at the level needed to make strategic product decisions. The key activities are as follows:
Strategic Requirements for Technology Backplane Development
Strategic Requirements for Technology Backplane Development not only leads to the selection of technologies, but also to how it will be implemented. Technology Selection QuickScan can be used as s starting point for establishing the metadata management requirements as part of the environment.
Future-State Physical Architecture and Vendor Selection
Once again, the Technology Selection QuickScan can hep form a starting point to compare and contrast vendors. The requirements for Active Metadata Integration are much more sophisticated then building a Metadata Warehouse.
Data Governance Policies
Data Governance Policies are derived from the Principles and Guidelines developed in Phase 1. Metadata Policies should equally be defined at this stage.
Metadata Standards and Data Standards should be established before the modelling team begins any detailed work. Some form of existing standards can often be used as a starting point. Metadata Standards should equally be defined at this stage.
Software Development Lifecycle Preparation
Taking a comprehensive approach to metadata management will typically involve significant changes to the software development lifecycle. In particular, Active Metadata Integration will fundamentally change the design process that is followed by streamlining the outputs from analysis and design into development.
The Metadata Driven Architecture is enabled from early in project and the implementation of the approach first starts within the Technology Blueprint. As this approach will be formally designed and implemented in the Continuous Implementation Phases, it is important to get some metadata management practices in place from the onset. Therefore, it can be valuable to build a prototype Metadata Repository and define an initial set of metadata management standards. This can then help drive the SDLC standards that will be developed.
Roadmap and Foundation Activities (Phase 3)
The Roadmap and Foundation Activities is where the metadata store is formally designed, developed and populated. The key activities for this approach can be difficult to follow as they cross aspects of the MIKE2.0 Overall Implementation Guide is a non-sequential order. In summary, first build a data dictionary, then to store other metadata artifacts and then continue to populate this model with different artifacts.
Defining a Data Dictionary should be a standard aspect of any Database Design. In reality, it is a practice that is often not strictly followed. Building a Metadata Warehouse involves re-visiting existing data models and the enhancing database design with a more robust data dictionary encompassing domain values, term definitions and relevant physicalisation information.
Enterprise Information Architecture
MIKE2.0 takes the approach of building out the Enterprise Information Architecture over time for each new increment that is implemented as part of the overall programme. The scope for building the Enterprise Information Architecture is defined by the in-scope Key Data Elements (KDEs). The Data Mastering Model that is developed at this point is the cornerstone of defining the metadata architecture across a federated enterprise.
Data Governance Metrics
Data Governance Metrics provide the information quality objectives that the organisation plans to achieve. These metrics should be populated into the metadata model. Not all metrics can be determined through profiling; some will require an interviewing based assessment and architectural review.
Data Profiling focuses on conducting an assessment of actual data and data structures and is focused on measuring data integrity, consistency, completeness and validity. Metadata such as business rules and mapping rules are identified as a by-product of this process.
Message Modelling generally refers the modelling of data structures as it flows between systems, as opposed to simply at data rest. A category of business messages and its data elements, their structures and headers and their relationships to system interfaces are important metadata to manage in the environment.
Data Re-Engineering aims to progressively address quality issues with existing data. Quality metrics should be stored into metadata repository in a fashion that it is made available to business users.
Solution Architecture Definition/Revision
Solution Architecture Definition/Revision involves the more formal design of the metadata repository that was initially developed as a prototype in Phase 2.
Design Increment (Phase 4)
Metadata Management Solutions would apply across multiple Design Increment activities, the most critical activities are described below.
Business Intelligence Design
When designing the Business Intelligence environment, business metadata can help users better understand their analytical environment through more user-friendly attribute names and term definitions. The design should include an approach to integrate metadata between different representations of the same data elements.
In addition to providing user access, metadata reports can be designed as part of this activity. For example, standard reports on Data Lineage can be helpful in meeting business regulatory requirements. In an Active Metadata Integration approach, source and target system data models and identified business are often directly used to build the analytical metadata environment.
ETL Logical Design
The ETL Logical Design process should include specific activities to integration metadata across systems. In an Active Metadata Integration approach outputs from Data Modelling, Data Profiling and Data Re-Engineering feed directly into the ETL Design process and through to the implementation of code.
Services Oriented Architecture Design
Services Oriented Architecture Design techniques can be applied to defined Model Services and Metadata Management Services. Metadata Services provide fine and coarse grained services to build a reusable platform of independent metadata capabilities to drive a Model Driven Architecture.
Incremental Development, Testing, Deployment and Improvement (Phase 5)
From a metadata management perspective, phase 5 is focused on development of the metadata integration interfaces, testing the solution through Functional Testing, SIT Testing, End-to-End Testing and UAT and then deploying the solution into production. Continuous Improvement activities are also important from a metadata management perspective.
Technology Backplane Development
Technology Backplane Development includes the development of metadata artefacts and their integration across systems in the environment. In an Active Metadata Integration approach the design process feeds more directly into the development process.
BI Application Development
BI Application Development builds on the BI Application Design to ensure BI metadata is properly defined, integrated between systems and reported on specifically in the Metadata Warehouse analytical environment.
Continuous Improvement - Infrastructure
Continuous Improvement of Infrastructure is a key part of moving to Active Metadata Integration approach. In this activity, organisations are pro-actively trying to improve existing infrastructure as opposed to reacting to issues as they occur. As moving to this architectural model will take long periods of time, continuous improvement through re-factoring existing software assets is a valuable and pragmatic approach.
Mapping to Supporting Assets
Logical Architecture, Design and Development Best Practices
A number of artifacts help support the MIKE2.0 Solution for Metadata Management:
Product-Specific Implementation Techniques
Relationships to other Solution Offerings
Extending the Open Methodology through Solution Offerings
This solution would benefit from the following enhancements to the core approach:
Wiki asset search