Open Framework, Information Management Strategy & Collaborative Governance | Data & Social Methodology - MIKE2.0 Methodology
Wiki Home
Collapse Expand Close

Members
Collapse Expand Close

To join, please contact us.

Improve MIKE 2.0
Collapse Expand Close
Need somewhere to start? How about the most wanted pages; or the pages we know need more work; or even the stub that somebody else has started, but hasn't been able to finish. Or create a ticket for any issues you have found.

Metadata Management Solution Offering

From MIKE2.0 Methodology

Share/Save/Bookmark
Jump to: navigation, search
Hv3.jpg This Solution Offering currently receives Major Coverage in the MIKE2.0 Methodology. Most required activities are provided through the Overall Implementation Guide and SAFE Architecture, but some Activities are still missing and there are only a few Supporting Assets. In summary, aspects of the Solution Offering can be used but it cannot be used as a whole.

Contents

Introduction


The Metadata, Taxonomy, Cataloging and Classification Solution Offering provides the necessary descriptors to turn data and content into information--and ultimately knowledge. Metadata management is a core focus of Information Management. This offering provides techniques for managing metadata with the same diligence as core data and content. Techniques for building taxonomies, catalogs, and classification schemes are also key enablers. This offering provides them with the intent of helping organizations define their own information models. The result for the end user: data and content are much easier to understand. The process of moving to a more “active” approach to Metadata Integration is also described within this offering.

Executive Summary

Metadata Management is a key aspect of the MIKE2.0 approach to Information Development. Without managing your metadata, you cannot manage your information.

Many organisations fail to recognise the importance of metadata management (MM) . As a result, most have, to be blunt, traditionally done a very poor job at it. One of the challenges is that MM is often presented in different contexts. Many organisations have typically seen it as an afterthought, even though activities such as building a data dictionary have always been upfront activities.

On the positive side, recent business and technology changes have moved MM to the forefront. Regulatory requirements such as Basel II, for example, have required organisations to understand the lineage of data as it flows across the environment. Solutions that provide a comprehensive analytical view of an organisation’s metadata environment are referred to in MIKE2.0 as a Metadata Warehouse.

From a technology perspective, many software and IT vendors have re-architected their products over the last few years to take a metadata-driven approach. While initiated by some forward-thinking smaller players, is now a core part of the approach of the largest IT providers. That is, they have made MM more of an integrated by-product of the software development process. No longer an afterthought and a separate activity, this approach can provide a fundamental improvement in software development practices. In MIKE2.0, this set of collective set of techniques and technology is referred to as Active Metadata Integration.

Organisations that want to take a focus on Information Development (ID) need to make MM one of their key focus points. Considering all the changes over the past few years, before defining a solution, it is best to have a clear definition on the subject.

Solution Offering Purpose

This is a Foundational Solution. Foundational Solutions are "background" solutions that support to Core Solution Offerings of the MIKE2.0 Methodology.

Foundational Solutions are the lowest assets within MIKE2.0 of a comprehensive nature. They may tie together multiple Supporting Assets. What's more, they are referenced by the Overall Implementation Guide and other Solution Offerings.


The Metadata_Management_Solution_Offering is also Core Solution Offering (CSO). CSOs bring together all assets in MIKE2.0 relevant to solving a specific business and technology problem. Many of these assets may already exist. As the suite is built out over time, assets can be progressively added to an Offering.

Solution Offering Relationship Overview

Mike2 solution groups iam.jpg
The MIKE2.0 Solution for Metadata Management describes how the Activities and Supporting Assets of the MIKE2.0 Methodology can be used to deliver a comprehensive strategy for moving to a metadata-driven architecture and building out a metadata warehouse.

MIKE2.0 Solutions provide a detailed and holistic way of addressing specific problems. They can be mapped directly to the Phases and Activities of the MIKE2.0 Overall Implementation Guide (OIG), providing additional content to help understand the overall approach.

The MIKE2.0 Overall Implementation Guide explains the relationships between the Phases, Activities, and Tasks of the overall methodology. The OIG also shows how the Supporting Assets tie to the overall Methodology and MIKE2.0 Solutions.

Users of the MIKE2.0 Methodology should always start with the OIG and the MIKE2.0 Usage Model as a starting point for projects.

Solution Offering Definition

What is Metadata

Metadata is data and information about data. It provides the intelligence to be able to link pieces of data, and also facilitates the ability to trace the path of data through the different stages of the supply chain – from source to publication - both internally as well as externally.

Metadata describes the data and the transformation rules that it undergoes at every step –in business terms as well as technical terms.

Business Metadata consists of “definitional” requirements and rules. It includes:

  • Definitions of attributes in business terms
  • Allowable values of attributes
  • Business rules used in the transformation of data
  • Identification of source / destination tables and attributes
  • Data Quality levels
  • Organisational model and roles and responsibilities
  • Programme delivery metrics

Technical Metadata can come in many forms. It includes:

  • Data integration configuration information
  • Data model schema definitions
  • Message schema definitions
  • Technical transformation rules
  • Data load timings
  • Security information
  • Archiving and logging metadata
  • Application identifiers
  • Header information
  • Environment-specific information

Most business metadata has to be manually captured as a starting point whereas some technical metadata is generated (sometimes referred to as operational metadata). Technical Metadata is generally in a number of sources across the information environment. Some metadata, such as data quality metrics of attributes could arguably be classified as business or technical metadata.

What is Metadata Management?

Metadata management is the mechanism for business and technical metadata to be properly defined, shared and integrated across the environment. The MIKE2.0 Methodology presents two comprehensive solutions to manage metadata:

  • A Metadata Warehouse provides an integrated store of business and technical metadata that users can analyse through a standard set of reports or direct query access.
  • Metadata Driven Integration involves using business and technical metadata to specifically drive the software development process, i.e. using a data model as input to drive a set of ETL mapping rules.

Solutions may involve centralised or distributed metadata architecture. Metadata-Driven Integration is the more advanced of the two techniques and would also include delivery of a Metadata Warehouse.

What are the Benefits?

A piece of data is meaningless unless it is defined and put into context. Metadata helps to put a context around data and transform it to information. Some of the key benefits include:

Value Description
Improved Productivity and Availability Downstream developers can achieve greater productivity and responsiveness by retrieving physical table and file definitions for the development of data movement and transformation processes. Information about data can be accessed on demand.
Better Consistency The same language and understanding of data can be shared across business and technical users; better consistency can lead to reduced complexity
Quality Improvement A consistent language for data that combines business and technical metadata will lead to more reliable interpretations of data.
Smoother Development Techniques Change management can be aided by providing metadata inventory and impact analysis reports. Taking a metadata-driven approach can smooth the transition from design to the development.
Impact Analysis Impact Analysis allows business and technical uses to see the potential results of a change. In business-driven environments where changes occur frequently, this is instrumental is managing the impact to a complex, federated environment.

A more detailed list of business drivers for better metadata management helps explain this concept further.

What is the Approach?

Better Metadata Management will typically require a shift in development practices. For the Metadata Warehouse, the shift is somewhat gradual. For Active Metadata Integration, it will typically be a significant shift to how an organisation is developing software. The MIKE2.0 Solution for Metadata Management presents a set of activities and techniques for both.

For either approach it is important to remember that metadata management is not just about technology, applications and tools. Metadata management requires a cultural shift and a methodology that defines processes, roles and responsibilities. Where possible, technology will be used to enforce or guide the methodology, but technology alone will not provide a solution.

It is also important to remember that metadata management is an evolutionary process. A complete enterprise metadata management framework cannot be deployed in one single motion. Metadata existed before the term became popular among information and data management professionals. An information architecture always uses metadata, but the simple existence of metadata does not mean that all metadata needs to be managed. Metadata management should start with immediate needs and be implemented based on real business and technical requirements.

How it fits into the Overall MIKE2.0 Methodology

The MIKE2.0 Solution for Metadata Management defines an approach that can be used to manage business and technical metadata across the complete enterprise environment. It provides a comparison of techniques and key activities as they relate to the Overall Implementation Guide.

Comparison of Techniques

The key distinction between the approaches is that Active Metadata Integration automates the outputs from the analysis and design process into the build. Even with Active Metadata Integration solutions will involve some form of manual integration. With a manual process the probability of integration errors are much greater and any single error can be very costly to trace and recover from. Therefore, the techniques employed should progressively automate the metadata management process as much as possible.

The Metadata Warehouse

When building a metadata warehouse, the goal is to deliver a solution through a pragmatic approach to relieve immediate and medium-term metadata management pain points while keeping focused on long-term metadata management objectives. Just like a Data Warehouse, the solution should be designed to evolve into an infrastructure with wider and more strategic reach across the organisation. The design should be open so that interfaces, components and databases can be replaced and immediate priorities quickly resolved.

The Metadata Warehouse

Active Metadata-Driven Approach

Active Metadata Integration extends the concept of being able to link metadata flows into and out of a managed metadata environment in an analytical fashion by making it an integrated part of the design process. This active approach to metadata management makes use of a model-driven approach that improves consistency, quality and reuse utility, by automating the integration between the sources of record for metadata. Metadata Integration itself is made model-driven by interfacing with Model Management capabilities in the architecture to provide a framework for repeatable development processes and reusable components to integrate metadata.

Active Metadata Integration Development Process

Relationship to Solution Capabilities

Relationship to Enterprise Views

Metadata Management is a key enabler for Information Development that goes across people, process, organisation and technology. Best practices in metadata management typically require significant changes to the competencies that are employed within an organisation and the skills required to implement these competencies. New roles may be required specifically to take ownership of metadata management, such as the Metadata Manager within an Information Development Organisation.

Significant technology changes are also typically required; metadata management is one of the areas where vendor technologies have made their most significant advances in recent years.

The approach is driven by an overall strategy, defined in the Business and Technology Blueprint.

Mapping to the Information Governance Framework

Mapping to the SAFE Architecture Framework

Metadata Management Overlay across the SAFE Architecture

The SAFE Architecture provides a number of components that relate specifically to metadata management. The aim is to eventually move to a metadata-driven approach across the architecture, but this Active Metadata Integration may not be followed from the onset.

Metadata management is first introduced as one of the Foundation Capabilities for Information Development within the architecture, introduced through the concept of a Data Dictionary. More advanced capabilities within the architecture include Active Metadata Integration. As metadata is distributed across technology in the application, infrastructure and information environment a number of component capabilities are required to manage the metadata environment. The key architectural component for the Metadata-driven approach is some form of repository for storing artifacts and supporting modelling tools to create data models, meta-models and to report on this information. There are vendor offerings in this space and many organisations build bespoke repository-based solutions.

Mapping to the Overall Implementation Guide

Building a comprehensive solution for metadata management requires activities across all 5 phases of the MIKE2.0 Methodology. Oftentimes, it is done in the context of a larger project as opposed to a standalone project to build a metadata repository. For purposes of explanation in relation to the Overall Implementation Guide, however, it has been explained below as a project in isolation.

Taking a metadata-driven approach is the more sophisticated of the two techniques and has a significant impact on the software development process. The metadata-driven approach is explained in relation to defining a metadata warehouse.

At this stage, there are arguably some gaps in the Overall Implementation Guide with relation to metadata management. This is discussed in extension section below.

Business Assessment and Strategy Definition Blueprint (Phase 1)

All activities from the Business Blueprint phase of MIKE2.0 will be required to define the strategy for a comprehensive metadata management solution. In a large, complex organisation it will require engagement with a high number of stakeholders and may result in significant organisational change. The key activities from the Business Blueprint are as follows:

Enterprise Information Management Awareness

Building awareness on the need for metadata management will often be important, particularly if it is in the context of another project. This activity should be undertaken with key stakeholders and can use training materials from MIKE2.0 to conduct workshops.

Overall Business Strategy for Information Development

In the Business Blueprint phase, the focus is on developing an initial Information Development strategy that is aligned to the specific set of business requirements, many of which are driven from the Application Development stream. The Overall Business Strategy for Information Development should also involve the initial gathering of business metadata as part of defining the conceptual model.

Organisational QuickScan for Information Development

IM QuickScan provides a comprehensive assessment of information management maturity, which includes specific questions related to metadata management. To complement results from the QuickScan assessment, MIKE2.0 also provides a summary view of metadata management maturity that shows the key criteria for comparing different maturity levels across organisations.

Future-State Vision for Information Management

The SAFE Architecture is used as a starting point to define the strategic conceptual architecture at the component level and a set of high level solution architecture options. The architectural overlay to the SAFE Architecture that focuses on metadata management can be used to frame the key set of capabilities required as part of the architecture.

Initial Data Governance Organisation

Most organisations will require a large degree of structural change to move to a mature Information Development organisation. It is starting in Phase 1 that we want to begin to “re-balance” the Enterprise to take more of a focus on the development of information. As part of the MIKE2.0 methodology, the aim is develop an [[ Information Development Organisation to complement more traditional organisational models around application development and infrastructure. This organisation should have roles and responsibilities specifically aligned to Metadata Management and this information stored in a repository.

Business Blueprint Completion

When formulating the overall business case and programme plan as part of Business Blueprint Completion, it is important to emphasize the need for a foundation of metadata management capabilities to be delivered in the early stages of the programme. Much of the financial business case related to improved operational efficiency will derive from improved metadata management practices. Making metadata management a mandatory part of fulfilling the business case helps to ensure it will not be short-cut during technology selection.

Technology Assessment and Selection Blueprint (Phase 2)

All activities from the Technology Blueprint phase of MIKE2.0 will be required to define a comprehensive approach to metadata management, which will typically involve the implementation of new technologies and establishing standards for how technology will be implemented. In phase 2 of MIKE2.0, a diligent approach is applied to establish the technology requirements at the level needed to make strategic product decisions. The key activities are as follows:

Strategic Requirements for Technology Backplane Development

Strategic Requirements for Technology Backplane Development not only leads to the selection of technologies, but also to how it will be implemented. Technology Selection QuickScan can be used as s starting point for establishing the metadata management requirements as part of the environment.

Future-State Physical Architecture and Vendor Selection

Once again, the Technology Selection QuickScan can hep form a starting point to compare and contrast vendors. The requirements for Active Metadata Integration are much more sophisticated then building a Metadata Warehouse.

Data Governance Policies

Data Governance Policies are derived from the Principles and Guidelines developed in Phase 1. Metadata Policies should equally be defined at this stage.

Data Standards

Metadata Standards and Data Standards should be established before the modelling team begins any detailed work. Some form of existing standards can often be used as a starting point. Metadata Standards should equally be defined at this stage.

Software Development Lifecycle Preparation

Taking a comprehensive approach to metadata management will typically involve significant changes to the software development lifecycle. In particular, Active Metadata Integration will fundamentally change the design process that is followed by streamlining the outputs from analysis and design into development.

Metadata-Driven Architecture

The Metadata Driven Architecture is enabled from early in project and the implementation of the approach first starts within the Technology Blueprint. As this approach will be formally designed and implemented in the Continuous Implementation Phases, it is important to get some metadata management practices in place from the onset. Therefore, it can be valuable to build a prototype Metadata Repository and define an initial set of metadata management standards. This can then help drive the SDLC standards that will be developed.

Roadmap and Foundation Activities (Phase 3)

The Roadmap and Foundation Activities is where the metadata store is formally designed, developed and populated. The key activities for this approach can be difficult to follow as they cross aspects of the MIKE2.0 Overall Implementation Guide is a non-sequential order. In summary, first build a data dictionary, then to store other metadata artifacts and then continue to populate this model with different artifacts.

Database Design

Defining a Data Dictionary should be a standard aspect of any Database Design. In reality, it is a practice that is often not strictly followed. Building a Metadata Warehouse involves re-visiting existing data models and the enhancing database design with a more robust data dictionary encompassing domain values, term definitions and relevant physicalisation information.

Enterprise Information Architecture

MIKE2.0 takes the approach of building out the Enterprise Information Architecture over time for each new increment that is implemented as part of the overall programme. The scope for building the Enterprise Information Architecture is defined by the in-scope Key Data Elements (KDEs). The Data Mastering Model that is developed at this point is the cornerstone of defining the metadata architecture across a federated enterprise.

Data Governance Metrics

Data Governance Metrics provide the information quality objectives that the organisation plans to achieve. These metrics should be populated into the metadata model. Not all metrics can be determined through profiling; some will require an interviewing based assessment and architectural review.

Data Profiling

Data Profiling focuses on conducting an assessment of actual data and data structures and is focused on measuring data integrity, consistency, completeness and validity. Metadata such as business rules and mapping rules are identified as a by-product of this process.

Message Modelling

Message Modelling generally refers the modelling of data structures as it flows between systems, as opposed to simply at data rest. A category of business messages and its data elements, their structures and headers and their relationships to system interfaces are important metadata to manage in the environment.

Data Re-Engineering

Data Re-Engineering aims to progressively address quality issues with existing data. Quality metrics should be stored into metadata repository in a fashion that it is made available to business users.

Solution Architecture Definition/Revision

Solution Architecture Definition/Revision involves the more formal design of the metadata repository that was initially developed as a prototype in Phase 2.

Design Increment (Phase 4)

Metadata Management Solutions would apply across multiple Design Increment activities, the most critical activities are described below.

Business Intelligence Design

When designing the Business Intelligence environment, business metadata can help users better understand their analytical environment through more user-friendly attribute names and term definitions. The design should include an approach to integrate metadata between different representations of the same data elements.

In addition to providing user access, metadata reports can be designed as part of this activity. For example, standard reports on Data Lineage can be helpful in meeting business regulatory requirements. In an Active Metadata Integration approach, source and target system data models and identified business are often directly used to build the analytical metadata environment.

ETL Logical Design

The ETL Logical Design process should include specific activities to integration metadata across systems. In an Active Metadata Integration approach outputs from Data Modelling, Data Profiling and Data Re-Engineering feed directly into the ETL Design process and through to the implementation of code.

Services Oriented Architecture Design

Services Oriented Architecture Design techniques can be applied to defined Model Services and Metadata Management Services. Metadata Services provide fine and coarse grained services to build a reusable platform of independent metadata capabilities to drive a Model Driven Architecture.

Incremental Development, Testing, Deployment and Improvement (Phase 5)

From a metadata management perspective, phase 5 is focused on development of the metadata integration interfaces, testing the solution through Functional Testing, SIT Testing, End-to-End Testing and UAT and then deploying the solution into production. Continuous Improvement activities are also important from a metadata management perspective.

Technology Backplane Development

Technology Backplane Development includes the development of metadata artefacts and their integration across systems in the environment. In an Active Metadata Integration approach the design process feeds more directly into the development process.

BI Application Development

BI Application Development builds on the BI Application Design to ensure BI metadata is properly defined, integrated between systems and reported on specifically in the Metadata Warehouse analytical environment.

Continuous Improvement - Infrastructure

Continuous Improvement of Infrastructure is a key part of moving to Active Metadata Integration approach. In this activity, organisations are pro-actively trying to improve existing infrastructure as opposed to reacting to issues as they occur. As moving to this architectural model will take long periods of time, continuous improvement through re-factoring existing software assets is a valuable and pragmatic approach.

Mapping to Supporting Assets

Logical Architecture, Design and Development Best Practices

A number of artifacts help support the MIKE2.0 Solution for Metadata Management:

Product-Specific Implementation Techniques

Relationships to other Solution Offerings

Extending the Open Methodology through Solution Offerings

This solution would benefit from the following enhancements to the core approach:

  • A specific Activity should be added to the Foundation Activities to extend the prototype metadata repository built in Phase 2 to a more robust model. This should include determination of the underlying meta-model, integration interfaces and definition of a end-user access interface
  • Development techniques should be added for Active Metadata Integration in the form of Supporting Assets and there should be a specific task under the Software Development Lifecycle Preparation Activity that references these techniques
  • There may need to be specific techniques added for Active Metadata Integration that serve as variations to existing tasks, similar to what is done with the Services Oriented Architecture Design Activity
  • Specific activities are required for Metadata Integration for other component areas, such as is done with ETL Logical Design
Wiki Contributors
Collapse Expand Close

View more contributors