Open Framework, Information Management Strategy & Collaborative Governance | Data & Social Methodology - MIKE2.0 Methodology
Wiki Home
Collapse Expand Close

Members
Collapse Expand Close

To join, please contact us.

Improve MIKE 2.0
Collapse Expand Close
Need somewhere to start? How about the most wanted pages; or the pages we know need more work; or even the stub that somebody else has started, but hasn't been able to finish. Or create a ticket for any issues you have found.

Data Warehousing Solution Offering

From MIKE2.0 Methodology

Share/Save/Bookmark
Jump to: navigation, search

Hv3.jpg This Solution Offering currently receives Major Coverage in the MIKE2.0 Methodology. Most required activities are provided through the Overall Implementation Guide and SAFE Architecture, but some Activities are still missing and there are only a few Supporting Assets. In summary, aspects of the Solution Offering can be used but it cannot be used as a whole.
A Creation Guide exists that can be used to help complete this article. Contributors should reference this guide to help complete the article.

Contents

Introduction


The Data Warehousing Solution Offering provides an approach for delivering a Data Mart, an Enterprise Data Warehouse or variants of these systems that are more departmentally-focused, operational in nature or application-specific. The focus of this solution offering is around the “back end” of Data Warehousing and the overall delivery process as opposed the “front end” Business Intelligence aspect that is provided through other offerings. Therefore, the enablers for this offering are techniques related to data modelling, data integration, metadata management and data quality management as well as Data Warehouse strategy and architectural techniques.

Executive Summary

When it comes to providing a better approach to Data Warehousing and Business Intelligence (BI), it is often important to start with definitions. This is because there are varying definitions of the technologies and techniques that are required for a contemporary BI environment.

In the MIKE2.0 Methodology, the Business Intelligence components in the SAFE Architecture and their complementary implementation techniques relate to the “front end” reporting and analytical tools.

These techniques may be applied to a Data Mart, an Enterprise Data Warehouse or variants of these systems that are more departmentally-focused, operational in nature or application-specific. These systems enable users to access data in a repository brought together from many different systems across the organisation through the “back-end”. As an alternative to a single data store, there has also been a recent trend towards virtual/federated Data Warehousing that involves accessing large amounts of data across systems. In MIKE2.0, the collective set of back-end systems is referred to as the Technology Backplane.

To build any of these systems the front-end and back-end are required. Whether both of these areas should be classified as Business Intelligence relates more to marketing and project funding than a technology or project delivery decision. Delivering these projects should follow a “journey approach” that is somewhat different than the techniques for functional or infrastructure-related projects.

In the past, many Data Warehousing and Business Intelligence solutions have failed to meet the expectations of its user community. Whilst useability is critical and Reporting and Analytical technologies and techniques vary, addressing issues in this area tends to be manageable. Most major failures were typically due to the back-end or the delivery process.

Back-end issues have primarily been related to:

  1. Data Integration
  2. Metadata Management
  3. Data Quality Management
  4. Data Modelling

Delivery approach issues were primarily related to:

  1. Lack of a strategic vision that allowed for incremental delivery
  2. Poorly defined requirements
  3. Inadequate testing
  4. Architectural inflexibility

Despite these failures, organisations want a better Business Intelligence environment more than ever – and the capabilities they need today are even more sophisticated. In order to move to a reliable and effective Business Intelligence environment, the focus must be on getting these areas right and taking an Information Development approach. The MIKE2.0 Solution for Data Warehousing brings together key Activities and Supporting Assets for a better approach to this problem.

Solution Offering Purpose

This is a Core Solution Offering. Core Solution Offerings bring together all assets in MIKE2.0 relevant to solving a specific business and technology problem. Many of these assets may already exist and as the suite is built out over time, assets can be progressively added to an Offering.

A Core Solution Offering contains all the elements required to define and deliver a go-to-market offering. It can use a combination of open, shared and private assets.

Solution Offering Relationship Overview

The MIKE2.0 Data Warehousing Solution Offering is part of the EDM Solution Group

The MIKE2.0 Solution Offering for Data Warehousing describes how the Activities and Supporting Assets of the MIKE2.0 Methodology can be used to deliver a better environment for Reporting and Analytics.

MIKE2.0 Solution Offerings provide a detailed and holistic way of addressing specific problems. MIKE2.0 Solution Offerings can be mapped directly to the Phases and Activities of the MIKE2.0 Overall Implementation Guide, providing additional content to help understand the overall approach.

The MIKE2.0 Overall Implementation Guide explains the relationships between the Phases, Activities and Tasks of the overall methodology as well as how the Supporting Assets tie to the overall methodology and MIKE2.0 Solutions. Users of the MIKE2.0 Methodology should always start with the Overall Implementation Guide and the MIKE2.0 Usage Model as a starting point for projects.

Solution Offering Definition

Relationship to Solution Capabilities

The MIKE2.0 Solution Offering for Data Warehousing goes across multiple Enterprise Views and uses a number of components from the SAFE Architecture. For any significant implementation, the great majority of activities will be required from the Overall Implementation Guide.

Relationship to Enterprise Views

Building a better Data Warehousing and Business Intelligence environment means taking an approach more focused on Information Development. Therefore, work across the Information Development worksteam is the primary focus on this solution.

It should be noted that in addition to the integration work that is needed, significant work will also typically be required for other areas of Infrastructure Development. This work includes platform implementation, security, archiving, storage and backup and recovery. Although this is critical to the system design and may account for a large percentage of project resources and technology budget, the technologies and techniques in this area are more mature and problems in Infrastructure Development (with the exception of integration issues) tend not to be ultimate cause of most Data Warehousing project failures.

Mapping to the Information Governance Framework

Mapping to the SAFE Architecture Framework

Components Required from the SAFE Architecture for BI

Development of a comprehensive Data Warehousing and Business Intelligence environment will include varying components from the SAFE Architecture, depending on the scope of the initiative viagra australia online. Whereas other components also may play a role in the move to more advanced capabilities, the key enablers to the approach are listed below.

This first step is getting Foundation Capabilities for Information Development and Infrastructure Development in place. This reduces delivery risk and helps to build a solution that meets user requirements for usability, quality and performance.

Business Intelligence components for integrated reporting, visually-rich dashboards and detailed analytics allow users to get at the data delivered through the Technology Backplane into a Common or Shared Repository. From an Information Formats perspective, the focus is typically on integration of structured data.


Some organisations may move to a more sophisticated capability of a Services Oriented Architecture for greater reuse and improved flexibility. Moving away from a batch-only environment will also enable them to have more advanced capabilities for Enterprise Business Management such as Business Activity Monitoring It is important that Foundation Capabilities and some Enabling Technologies are in place as part of this transition.

Finally, an Active Metadata Integration approach may provide a complete overlay of metadata management capabilities on top of the overall architecture.

Mapping to the Overall Implementation Guide

In many cases, most of the Activities from the Overall Implementation Guide will be required for building out the new Data Warehousing and Business Intelligence environment. Users of MIKE2.0 should review each activity as a starting point to see if they are required based on the scope of the project requirements.

Shown below are the most important activities for building a better Business Intelligence environment and how they relate to the overall approach.

Business Assessment and Strategy Definition (Phase 1)

A comprehensive Data Warehousing and Business Intelligence environment means developing a vision that impacts people, process, organisation and technology. All activities from the Business Blueprint phase of MIKE2.0 will be required to define this strategic approach although it is possible that other initiatives may cover some activities (i.e. there may be another project underway to define a Data Governance team).

Overall Business Strategy for Information Development

The Overall Business Strategy for Information Development activity establishes the overall set of strategic business requirements that then translate into a set of high-level information requirements. The strategic business requirements are then translated into the Detailed Business Requirements at the beginning of Phase 3 as part of the continuous implementation approach to building the Data Warehousing and Business Intelligence environment.

Future State Vision for Information Management

The Future State Vision for Information Management presents the strategic architecture at a conceptual level, based on the initial set of strategic business requirements, the current-state environment and industry best practices. Whilst still in the very early stages (the focus is on options), it is here that the guiding principles, initial set of architectural components and conceptual data model for the Business Intelligence environment begin to be formulated. By the end of the activity, there will be some idea of what the future-state Data Warehousing and Business Intelligence environment will look like from a technology perspective.

Business Blueprint Completion

The Business Blueprint Completion activity is one of the final activities in building the overall Business and Technology Blueprint. It brings together output from the Technology Blueprint and makes revisions to the Business Blueprint. As Data Warehousing initiatives are often implemented through a long-running, increment-based approach, the planning steps defined in this activity are particularly important. In addition, it is in this activity that the Business Case is re-formulated to make a clear case for investment and the benefits it will provide. This activity is completed in conjunction with the Technology Blueprint Completion activity, which is also important for the Data Warehousing Solution.

Technology Assessment and Selection Blueprint (Phase 2)

All activities from the Technology Blueprint phase of MIKE2.0 will typically be required to define the strategic approach to building a Data Warehousing and Business Intelligence Environment. Some activities (such as the definition of Data Standards) may only require a review of existing artifacts as they may already be in place.

A relational, row-based data warehouse may not always be the correct selection. [1] speaks to some of the typical arguments against using Hadoop for "big data" and some reasons why a Hadoop approach may be the best environment selection.

Strategic Requirements for BI Application Development

Strategic Requirements for BI Application Development translates the strategic business requirements into strategic requirements for the Business Intelligence system. At a high-level, requirements are captured for key business functions, analytical methods, calculations and how users will get access to the data. Along with the Strategic Business Requirements, these requirements should also act as supplementary input into the Detailed Business Requirements activity the beginning of Phase 3.

Strategic Requirements for Technology Backplane Development

Strategic Requirements for Technology Backplane Development map the functional capabilities that are required for Information Development and Infrastructure Development against the conceptual architecture capabilities as defined by the SAFE framework. This is a key activity as the back-end capabilities required for developing the Business Intelligence systems are the area where many initiatives tend to run into issues.

Metadata Driven Architecture

MIKE2.0 aims to move to a Metadata Driven Architecture by getting metadata management practices in place from the onset. As part of the initial strategy it can valuable to build a prototype metadata repository and define an initial set of metadata management standards. For a Data Warehousing solution, improved techniques for metadata management:

  • Improve business reporting and management decisioning
  • Help meet statutory reporting requirements by providing a mechanism for enterprise wide timely reporting
  • Facilitate future development of analytical applications
  • Improve maintenance of the Business Intelligence systems
  • Offer better visibility to the lineage of information between systems, including the operations that are performed upon the data

In order to provide these capabilities, SAFE provides a metadata architecture overlay across all components.

Roadmap and Foundation Activities

This phase is one of key features of MIKE2.0 in that it involves getting capabilities that tend to be the biggest issues on many Business Intelligence initiatives and getting them “out in front” of other design and development activities.

Detailed Business Requirements

The Detailed Business Requirements activity refines the requirements for a particular increment. For a Data Warehousing solution it will involve validating, refining, categorizing and prioritising business requirements for this particular increment. The strategic requirements established during the Blueprint are used as input.

Database Design

The Database Design of the target environment is the cornerstone of the Business Intelligence solution. In this activity, the target data model is refined to a detailed level (building off the conceptual design) to meet the more detailed business requirements. The Data Standards defined in Phase 2 provide the guidelines for the modeling process and must be in place first. As part of continuous implementation, it often involves making extensions to an existing model.

Data Profiling

Data Profiling focuses on conducting an assessment of actual data and data structures and is focused on measuring data integrity, consistency, completeness and validity. In MIKE2.0, Data Profiling is seen as a key technique for minimising delivery issues in Data Warehousing projects that are the result of data quality issues only being discovered at a late stage.

As part of this process, data quality issues are identified at the individual attribute level, at the table-level and between tables. Metadata such as business rules and mapping rules are identified as a by-product of this process.

Business Intelligence Initial Design and Prototype

The Business Intelligence Initial Design and Prototype is the first step in building the part of the Business Intelligence system that business users will interact with. The initial design standards for reports are developed in this activity, along with the data requirements for reports. The report designer then begins development of the reports through prototyping and works with users to refine the product. This conceptual design can be considered part of the Solution Architecture.

Solution Architecture Definition/Revision

The Solution Architecture Definition/Revision for the Data Warehousing environment translates the strategic architecture from the Blueprint into a more detailed architecture that frames the detailed design. A Solution Architecture may already be in place - as part of the continuous implementation approach, the focus may be on its revision. The Solution Architecture will cover the conceptual design of all major components in the Business Intelligence system, including infrastructural components and SDLC process design steps for source control and automation of testing.

Incremental Design

Phase 4 finalises the design of the overall Data Warehousing solution by building on the architecture, modelling and data re-engineering work from the Foundation Activities. It also “hardens” the design in areas where prototyping and initial design has occurred. All activities are required to build the Data Warehousing environment; the key activities are shown below.

Business Intelligence Design

The Business Intelligence Design activity is focused on “hardening” the design of the BI prototype so that it can be implemented into a production system – it may involve standard reports or ad-hoc access design. This activity may also consist of designing certain functions using an OLAP or data mining tool. The detailed design builds on the standards and conceptual design established in prior phases.

ETL Logical Design

The ETL Logical Design provides the key design approach for integrating data from the source to target Business Intelligence environment. The logical design is technology-independent and builds on the integration standards established in earlier activities. It is often one of the most complex activities of the Business Intelligence implementation.

ETL Physical Design

The ETL Physical Design provides the key design approach for integrating data from the source to target Business Intelligence environment. It is specific to a technology and how it will be implemented. The physical design complements the ETL Logical Design and also builds on standards established in earlier activities.

Incremental Development, Testing, Deployment and Improvement

The implementation activities in phase 5 provide a mechanism for continuous implementation and improvement of the Data Warehousing environment.

BI Application Development

BI Application Development can begin once the target database is place. It builds on the prototyping and detailed design activities from the prior phases.

Technology Backplane Development

Getting the Technology Backplane available as soon as possible is critical for the development of Business Intelligence Applications.

Testing Activities

Testing activities for a Data Warehousing solution includes multiple cycles that are largely executed in a serial fashion. It involves Functional Testing, System Integration Testing, End-to-End Testing, Stress and Volume Testing, UAT and PVT. All activities are required for building a complex Data Warehousing solution. Functional Testing, SVT and UAT are the most important for the front-end of Business Intelligence. System Integration Testing, End-to-End Testing and Stress and Volume Testing are the most important for testing the Technology Backplane.

Continuous Improvement Activities

The Continuous Improvement activities are focused on delivering incremental improvements to existing functionality. For a Data Warehousing solution, it is important to factor in these continuous activities to improve the quality of data, the infrastructure and to improve organisational efficiency.

Mapping to Supporting Assets

Logical Architecture, Design and Development Best Practices

A number of artefacts help support the MIKE2.0 Solution for Data Warehousing:

In addition, there are MIKE2.0 Solutions that are specifically focused on each of the back-end issues that tend to be problematic in building a better Data Waerhousing environment:

Product-Specific Implementation Techniques

Product Selection Criteria

Relationships to other Solution Offerings

The MIKE2.0 Data Warehousing Solution Offering delivers the back-end for a comprehensive Business Intelligence environment. Each Data Warehousing solution will be followed or accompanied by a Business Intelligence Solution that will provide the front-end capabilities for the business users.

Extending the Open Methodology through Solution Offerings

Wiki Contributors
Collapse Expand Close

View more contributors