Enterprise Search Solution Offering

From MIKE2.0 Methodology

This Solution Offering currently receives Major Coverage in the MIKE2.0 Methodology. Most required activities are provided through the Overall Implementation Guide and SAFE Architecture, but some Activities are still missing and there are only a few Supporting Assets. In summary, aspects of the Solution Offering can be used but it cannot be used as a whole.

Introduction

The Enterprise Search Solution Offering provides an approach for accessing information held in a variety of sources across the enterprise through a simple text query and easy-to-use web interface. Techniques are provided that ensure users can only see the content to which they have access rights; techniques are also provided to allow this information to be accessed from a variety of sources, not just web pages. Techniques for optimising the search process are also included within this offering. Techniques are included to associate metadata for content or reference data that may be related and which users may formally request access to.

Executive Summary

Search is not a new concept from a technology standpoint, but it is a new capability from a business perspective - a most users are desperate for it. What users want is something like Google - easy to use, quick and powerful. Unfortunately, searching across an Enterprise isn't quite so easy. First of all, there is security to worry about - we can't have all our information assets available to anyone. Second, the decisions that business users make are different than those made by users surfing the web. The impact of accuracy issues is higher and getting different results (often a result on the public web) can be a disaster for an Enterprise. Finally, there is a relevancy ranking and information delivery issue. If a single search turns up thousands of results, users won't want to sift through all of it. Without some of the mechanisms available on the web to prioritise which results show up first, how can we do this in an Enterprise?

Enterprise Search helps to address these issues by enhancing the techniques that are available through public web search. Ultimately, its aims to provide an enhanced Business Intelligence environment that makes federated access to information easy enough that:

• A line manager can ask a question like "Should I lend to this customer?", pulling in information from a loan application, credit systems and a Data Warehouse.

• An executive can ask "what do I need to do to increase profitability?", bringing in external market data, historical product information, operational costs and customer segment data.

• A customs official ask "What's the chance this person is a security threat?", using internal and external information that is both historical and transactional.

• A designer can perform non-text based searches on media content to find the sound byte they are looking for on an audio recording.

Although some of these capabilities are emerging, a growing number of organisations are using Search to replace or augment their traditional mechanisms for data access. In conjunction with other technologies, Search is a key enabler for accessing the information available in a Federated Enterprise. As Search evolves, this Solution Offering will continue to evolve inline with the technologies and techniques.

Solution Offering Purpose

This is a Core Solution Offering. Core Solution Offerings bring together all assets in MIKE2.0 relevant to solving a specific business and technology problem. Many of these assets may already exist and as the suite is built out over time, assets can be progressively added to an Offering.

A Core Solution Offering contains all the elements required to define and deliver a go-to-market offering. It can use a combination of open, shared and private assets.

Solution Offering Relationship Overview

This Solution Offering is part of the Access, Search and Information Delivery Offering Group

MIKE2.0 Solution Offerings provide a detailed and holistic way of addressing specific problems. MIKE2.0 Solution Offerings can be mapped directly to the Phases and Activities of the MIKE2.0 Overall Implementation Guide, providing additional content to help understand the overall approach. The MIKE2.0 Overall Implementation Guide explains the relationships between the Phases, Activities and Tasks of the overall methodology as well as how the Supporting Assets tie to the overall methodology and MIKE2.0 Solutions. Users of the MIKE2.0 Methodology should always start with the Overall Implementation Guide and the MIKE2.0 Usage Model as a starting point for projects.

Solution Offering Definition

Due to users' familiarity with search on the web or their desktop, there is often a misconception that search projects are simple and only focused on data access. While this should be the goal from a user perspective, the programmes of work are often quite complex when applied on an enteprise scale. Enterprise Search may include:

• Access across many federated sources, holding structured, semi-structured and unstructured content
• Significant Data Quality Improvement activities such as Data Investigation and Data Re-Engineering
• The aggregation of data from many sources into a single repository, similar to an Integrated Operational Data Store architecture
• Enabling a Business Intelligence environment in the fashion more traditionally implemented through a Data Warehouse
• A significant change to the traditional user experience for information-based business decisions

Therefore this Solution Offering provides a large number of activities and required capabilities across the MIKE2.0 Methodology.

Relationship to Solution Capabilities

The MIKE2.0 Solution Offering for Data Warehousing goes across multiple Enterprise Views and uses a number of components from the SAFE Architecture. For any significant implementation, the great majority of activities will be required from the Overall Implementation Guide.

Relationship to Enterprise Views

Search is primarily about enabling Information Development. It provides the ability to access structured and unstructured content and also to bring this information together in a single, integrated view.

Mapping to the Information Governance Framework

The Information Governance Solution Offering is required across all Solution Offerings. For Enterprise Search it is particularly important that the governance models focus on security for accessing information as this can be an area of significant complexity. The scope should cover structured data and unstructured content, with appropriate roles related to taxonomic development, standard lexicons and data/content quality.

Mapping to the SAFE Architecture Framework

For an Enterprise Search programme, a number of component may be required from the SAFE Architecture. In addition to a Search capability that allows access to federated systems. Enterprise Search may also involve components related to:

• Business Intelligence
• Data Investigation
• Data Re-Engineering

These components may be implemented through a Services Oriented Architecture. Increasingly, search results are being directly used in applications to be make business decisions and search vendors are creating derivative applications that make use of search results to make composite decisions.

Mapping to the Overall Implementation Guide

In most cases an Enterprise Search engagement will involve the majority of the Activities from the Overall Implementation Guide. Users of MIKE2.0 should review each activity as a starting point to see if they are required based on the scope of the project requirements.

The Usage Model for Access, Search and Content Delivery provides a list of all the required activities for Enterprise Search. Shown below are the most important activities for an Enterprise Search programme and how they relate to the overall approach.

Phase 1 - Business Assessment and Strategy Definition Blueprint

For a comprehensive, top-down programme generally all the activities are required from this phase to define the overall Business Strategy.

Overall Business Strategy for Information Development

The Overall Business Strategy for Information Development is important for defining the strategic requirements for information that users want to access. Its not just BI metrics for Search that are required, some additional activities are needed for unstructured content that more accurately describe the user experience through Search (currently proposed as an extensions).

Organisational QuickScan for Information Development

A number of tasks from the Organisational QuickScan for Information Development are important for Search. These include:

• Defining the Application Portfolio, which covers the candidate sources to be searched.
• Use of Information Maturity QuickScan, which should be extended to cover to specifically cover search
• Enhancements to IM QuickScan should feed into the Economic Value of Information model. Search could also be useful for doing this in a more quantitative fashion.
• Infrastructure Maturity QuickScan, which should be enhanced to better cover search

Future-State Vision for Information Management

The Future State Vision for Information Management is an important activity for defining the strategic conceptual architecture for Enterprise Search. As this typically involves a number of different conceptual components, a systematic approach to architecture is important. High Solution Architecture Options should also be defined at this stage.

Some tasks, such defining the future-state conceptual data model may be required depending on the architectural options to define a target storage repository when search results are used in more of a Business Intelligence fashion.

Business Blueprint Completion

Business Blueprint Completion is an important activity for defining how the solution will be implemented over time as well as the immediate priorities. The business case is also important, at it is critical for making the case on how Enterprise Search will bring value to the organisation and how the investments will explicitly lead to improved economic or capability returns.

Phase 2 - Technology Assessment and Selection Blueprint

For a comprehensive, top-down programme generally all the activities are required from this phase to define the overall Technology Strategy.

Strategic Requirements for BI Application Development

Strategic Requirements for BI Application Development is an activity used to define the user analysis system. For some Enterprise Search engagements, this capability allows users to make complex, federated queries to make mission-critical business decisions.

Strategic Requirements for Technology Backplane Development

An Enterprise Search engagement should involve the definition of Strategic Requirements for Technology Backplane Development. It should include foundational and advanced capabilities for Infrastructure Development and Information Development.

Some aspects of search are more infrastructure oriented and go across these capabilities. For example, there are significant integration aspects to search. By providing federated query mechanism, search provides an advanced capability for infrastructure integration.

Data Standards

The Data Standards activity includes a number of tasks that would apply for Search. This activity should also be extended from its current form to have complete coverage. Relevant tasks include:

• Define Data Specification Standards - applies in particular if building a content store to hold the search results. Should be extended to cover defining a standard lexicon.
• Define Data Modelling Standards - applies if building a content store to hold the search results.
• Define Data Security Standards - security is particularly important from a search perspective as it relates to what information can be shown to the user.
• Define Data Integration Standards - for search, this relates to interfaces between systems and techniques to be applied

The activity should be expanded to include:

• Define Content Modelling Standards - A task may be added to cover terminology more familiar to the building of unstructured content.
• Specific standards for search would be helpful to users as well as the overall approach for how search will be applied. Also applies to how information will be presented back to a user that is search specific. This overlaps with the BI standards task.

Phase 3 - Information Management Roadmap and Foundation Activities

Many of the activities from Phase 3 would be required as part of an Enterprise Search engagement. Whereas all of the Roadmap activities will be needed, not all Foundation Activities are required.

Information Management Roadmap Overview

The Information Management Roadmap Overview would show how the project will be implemented for a specific increment.

Enterprise Information Architecture

Building out the Enterprise Information Architecture can be a relevant part of a search project; search can also be an enabler in identifying some of the current-state information in different information store. This activity is useful for enterprise search scenarios that involve integration of overlapping data sets.

Database Design

Database Design is performed for Enterprise Search if a persistent data store is built to hold data. A persistent store is particularly important if a Business Intelligence repository will be built or if there will be significant aggregation of search results.

Data Profiling

Data Profiling is generally applied and is particularly important if the scope of work includes automated integration of search results as opposed to simpling user-driven access.

Data Re-Engineering

Data Re-Engineering is applied to Search to bring results together as well as through an interactive process with the user. A few tasks may need to be added to the standard Data Re-Engineering process that are more commonly applied for Search, such as lemmatisation and spell checking that are applied through a user-driven interface described below.

Business Intelligence Initial Design and Prototype

Business Intelligence Initial Design and Prototype applies for search if it is being used for user-driven analytics and reporting. The prototype may be a less formal process as the search BI interface may be less complex.

Phase 4 - Design Increment

For Enterprise Search, a number of design activities are required.

Business Intelligence Design

The Business Intelligence Design activity may apply as part of an Enterprise Search project as extension to analyse the data that has been returned.

Information Security Design

Information Security Design may apply as an important part of any Enterprise Search engagement to ensure proper access controls to information. It may employ the use of cryptographical controls and there should be an ongoing monitoring process of search queries.

Infrastructure Management Process Design

The Infrastructure Management Process Design activity provides a set of requirements for the physical implementation of the information stores, the index used for search and how this information is archived, stored and destroyed over time.

Data Integration Logical Design

Data Integration Logical Design would be just as applicable for Search as for other types of integration. Within this activity, logical adapter interfaces are defined that may go through an application interface or the back-end of a system.

Data Integration Physical Design

Data Integration Physical Design would be just as applicable for Search as for other types of integration. Physical design of interfaces would include technologies to be used, performance requirements and security.

User Interface Design

User Interface Design activity defines the front-end interface in which in user will interact with the information system. From a search perspective, the simplicity of this interface design is essential. Simple data re-engineering steps such as lemmatisation and spell checking may be interactively provided through this interface.

Technical it is important to separate the logic and backend from the front-end interface to assure the flexibility and sustainability of the solution.

Also critical is the ability for users to be able to make quick decisions about the relevance of a result from the large amounts of information. Based on the document type or the source system of a document, the preview "snippets" of the result set should be customized. This means the displayed fields within the result set should be aligned to the criterias to decide whether a result is interesting or not.

Users need to be able to visualise information results through graphical views, categorisation of results and personalisation.

Another requirement for users can be the ability to save and organize their personal search results. This feature can be provided by a "shopping" cart in which the user can save his favorite hits of various search queries.

Services Oriented Architecture Design

Services Oriented Architecture Design would be just as applicable for Search as for other types of integration. Reusable interface services can be built, as well as those specific to mediation or data quality management.

Phase 5 - Develop, Test & Deploy Increment

Technology Backplane Development

The Technology Backplane Development activity covers the development of integration and information management components as part of an Enterprise Search engagement. Integration interfaces, aggregation services and operationalisation of data re-engineering process are all included within this activity.

BI Application Development

BI Application Development may be required if the search solution requires a BI component to it. Most solutions won't require all of the tasks listed within this activity.

Testing Activities

Depending on the complexity of the Search engagement and the amount of automation required, different testing activities will be required. In complex engagements all activities may be needed. Functional Testing and some level of System Integration Testing will be required at a minimum; SVT and UAT are also important.

Mapping to Supporting Assets

Logical Architecture, Design and Development Best Practices

Key logical assets include:

• Models for Computing Based on Information Search
• Architecture Design Patterns for Search - Data Access
• Architecture Design Patterns for Search - Business Intelligence
• Architecture Design Patterns for Search - Content Aggregation
• Techniques for non-text Search
• Lemmatization techniques may be built into a product, but some logical best practices may also apply
• Architectural components such as an Integrated Operational Data Store can be used to bring search results together from federated systems; a number of Search Aggregation Techniques are applied as part of this process
• Search Engine Optimisation Techniques for Enterprise content
• Relevancy Ranking Techniques that can be applied within the Enterprise
• Techniques for Linking Internal Search Results to Public Search

Product-Specific Implementation Techniques

Specific techniques referenced in this section would include content from organisations such as:

• Autonomy
• Fast
• Google
• IBM, covering Omnifind and UIMA open source offering
• Microsoft
• Lucene

Product Selection Criteria

Key criteria for selection include:

• Breadth of offering into areas such as Data Investigation and Re-Engineering, Data Integration and taxonomic development
• Performance results
• Market positioning

The Technology Assessment QuickScan model should be extended to cover Enterprise Search

Relationships to other Solution Offerings

This Solution Offering is closely related to the following:

• Data Warehousing Solution Offering
• Data Investigation and Re-Engineering Solution Offering
• Information Value Assessment Solution Offering

Extending the Open Methodology through Solution Offerings

Listed below are proposed extensions to the Overall Implementation Guide to meet the requirements for Enterprise Search:

Potential Activity Changes

Overall Business Strategy for Information Development

Its not just BI metrics for Search that are required, some additional activities are needed for unstructured content and that more accurately describe the user experience through Search

Organisational QuickScan for Information Development

Should explicitly extend the QuickScan tools to cover search and more broadly the management of unstructured content.

Strategic Requirements for BI Application Development

This activity applies to enterprise search but some tasks should be added and/or generalised.

Current-State Logical Architecture

The tasks for future-state and current-state logical architecture may be revised to make it more inclusive an better aligned with search.

Data Standards

Should be extended to include Content Modelling Standards and Search Standards specifications

Message Modelling

Data Classification

If this activity is completed and includes the development of taxonomies than it would be more relevant for Enterprise Search

Data Profiling

Some changes to this activity may be necessary to make it work better with the profiling approach for search, but generally all steps are the same.

Data Re-Engineering

This activity may be expanded to cover the dynamic, user oriented re-engineering tasks specific to search.

Solution Architecture

This activity will likely be generalised to better cover solution offerings such as Search.

Infrastructure Management Process Design

This activity may have 1 - 2 tasks added that specific to building an index to support Search or some tasks may be generlised to provide converage.

Information Security Design

It is still under development. It may have some more detail added to it in relation to web security.