Informatica integration solution with custom metadata sources

From MIKE2 Methodology

Informatica Metadata Manager uses XConnects to load metadata from different source repositories into the Metadata Manger Warehouse for analysis and integration. The custom XConnect is used to load metadata, for tools or processes for which Informatica does not provide any out-of-the-box metadata solution. The figure below provides information about Metadata Manager Architecture highlighting the role of XConnect among the other components.

To integrate custom metadata, the following tasks have to be completed:

• Design the metamodel
• Implement the metamodel design
• Create the custom XConnect
• Setup and run the custom XConnect
• Configure the reports and schema

Prerequisites for Integrating Custom Metadata

The custom metadata integration process assumes knowledge of following topics:

• Common Warehouse Metamodel (CWM) and Informatica-defined metamodels. The CWM metamodel includes industry-standard packages, classes, and class associations. The Informatica-defined metamodel supplements the CWM metamodel by providing repository-specific packages, classes, and class associations. For more information about CWM, [1]. For more information about the Informatica-defined metamodel components run and review the metamodel reports.

• PowerCenter functionality. Metadata integration process can require configuring and running PowerCenter workflows that extract custom metadata from source repository database and loading it into the Metadata Manger warehouse.

• Data Analyzer functionality. Metadata Manger embeds Data Analyzer functionality to create, run, and maintain a metadata reporting environment. Knowledge of creating, modifying, and deleting reports, dashboards, and analytic workflows in Data Analyzer is required. Also knowledge of creating, modifying, and deleting table definitions, metrics, and attributes is required to update the schema with new or changed objects.

Contents 1 Task 1: Design the Metamodel 2 Task 2: Implement the Metamodel Design 3 Task 3: Create an XConnect using the Custom Metadata Configurator 4 Task 4: Setting up the XConnect 5 Task 5: Configure the Reports and Schema

Task 1: Design the Metamodel

The objective of this phase is to design the metamodel. This task consists of the following steps:

• Identify Custom Classes. To identify classes, determine the various types of metadata in the source repository that needs to be loaded into the Metadata Manger warehouse. Each type of metadata corresponds to one class.

• Identify Custom Class Properties. For each class identified in step 1, identify all class properties that need to be tracked in the Metadata Manger warehouse.

• Map Custom Classes to CWM Classes. Metadata Manger prepackages all CWM classes, class properties, and class associations. To quickly develop a custom metamodel and reduce redundancy, reuse the predefined class properties and associations instead of recreating them. To determine which custom classes can inherit properties from CWM classes, map custom classes to the packaged CWM classes. For all properties that cannot be inherited, define them in Metadata Manger.

• Determine the Metadata Tree Structure. Configure the way the metadata tree displays objects. Configure the metadata tree structure for a class when defining the class in the next task "Implement the Metamodel Design". Configure classes of objects to display in the metadata tree along with folders and the objects they contain.

• Identify Custom Class Associations. The metadata browser uses class associations to display metadata. For each identified class association, determine if a predefined association from a CWM base class can be reused or if an association needs to be defined manually in Metadata Manger.

• Identify Custom Packages. A package contains related classes and class associations. Import and export packages of classes and class associations from Metadata Manger. Assign packages to repository types to define the structure of the contained metadata. In this step, identify packages to group the custom classes and associations you identified in previous steps.

Task 2: Implement the Metamodel Design

This task includes the following steps.

• Create Custom Metamodel Originator. The Metadata Manger warehouse may contain many metamodels that store metadata from a variety of source systems. When creating a new metamodel, enter the originator of each metamodel. An originator is the organization that creates and owns the metamodel. The types of Originators include: Informatica, Standards Organization, Third Party, and Customer. Select ‘Customer’ as the new custom originator in Metadata Manager.

• Create Custom Packages . Define the packages to which custom classes and associations are assigned. Packages contain classes and their class associations. Packages have a hierarchical structure, where one package can be the parent of another package. Parent packages are generally used to group child packages together.

• Create Custom Classes. In this step, create custom classes identified in the metamodel design task.

• Create Custom Class Associations. In this step, implement the custom class associations identified in the metamodel design phase. In the previous step, CWM classes are added as base classes. Any of the class associations from the CWM base classes can be reused. Define those custom class associations that cannot be reused.

• Create Custom Repository Type. Each type of repository contains unique metadata. For example, a PowerCenter data integration repository type contains workflows and mappings, but a Data Analyzer business intelligence repository type does not.

• Associate packages to Custom Repository Type. To maintain the uniqueness of each repository type, define repository types in Metadata Manger, and for each repository type, assign packages of classes and class associations to it.

Task 3: Create an XConnect using the Custom Metadata Configurator

The XConnect extracts source repository metadata and reformats it to conform to the structure expected by the PowerCenter workflow. Then, the PowerCenter workflow loads the reformatted metadata into the Metadata Manager warehouse. Each XConnect extracts, transforms, and loads metadata for a particular repository type.

To create a custom XConnect using the Custom Metadata Configurator, map the metadata to the metamodel classes and associations identified previously in the Design the Metamodel Task. Provide the source metadata in one or more files in a .csv or .txt format and use the Custom Metadata Configurator to map the metadata in the files to the metamodel defined in Metadata Manager.

Another alternative to metadata files is o extract the metadata directly from the source repository database. In this case the PowerCenter mappings, generated automatically by the Custom Metadata Configurator after the completion of the custom XConnect development have to be modified to point to the source metadata repository instead of the supplied files and to select the desired fields.

The Configurator uses the following two metadata files for the metamodel map:

• Element metadata file. This file contains the metadata object names and attributes, in fact the attribute names of the fields from the source metadata extract file. The metadata objects are instances of the classes defined in the metamodel. Using the Custom Metadata Configurator, which exposes the information in the element metadata file, the object attributes in the element metadata file are mapped to the class attributes defined in the metamodel.

• Association metadata file. This file includes the relationships between metadata objects, which are instances of the class associations defined in the metamodel. The following information must be specified to set the class association between two metadata objects:
  • The repository and class of each object in the association.
  • The source and target objects that participate in the association.

Note:

To uniquely identify each record in the association or element metadata file specify a class rule. A class rule filters record in the file that does not apply to the class attribute or association map. A class rule must always be present even in the case when no rule is necessary to uniquely identify each record in the file, specify a dummy rule.

The following steps are taken to create the custom XConnect using the Custom Metadata Configurator:

• Create the templates. A template maps the metadata to the corresponding metamodel.
• Generate the PowerCenter mappings. PowerCenter mappings are generated to extract the custom source repository metadata and load it into the Metadata Manager warehouse.

After the XConnect is created using the Custom Metadata Configurator, it has to be set up in the Metadata Manager Console.

Create the templates

The purpose of this task is to create a template which will store information about how to map metadata object attributes to the class attributes and can also store the class associations between metadata objects.

To create a template, the following information needs to be provided:

• Template name
• Element and association metadata files. The element and association metadata files that provide the metadata objects and associations between them.
• Repository name. The source repository that stores the custom metadata.
• Classes. The metamodel classes to be included in the map with the element metadata objects from the file.

Generate the PowerCenter mappings

Once the template is completed, the Custom Metadata Configurator creates the PowerCenter mappings to extract the metadata from the element and association metadata files and load it into the Metadata Manager Warehouse.

When you generate the PowerCenter mappings, the Custom Metadata Configurator creates the following PowerCenter objects:

• PowerCenter mappings. For each template, creates one mapping for the class attribute map and another for the association map.
• PowerCenter sessions. Creates one session for each PowerCenter mapping. It includes all sessions in a PowerCenter workflow.
• PowerCenter workflows. Creates one workflow for each template.

The Custom Metadata Configurator stores all of these generated PowerCenter objects in an XML file, and then uses pmrep utility to import the XML file into the PowerCenter repository.

Task 4: Setting up the XConnect

Once the custom XConnect is created, it is set up and run from the Metadata Console like any out-of-the-box XConnect.

Task 5: Configure the Reports and Schema

The objective of this task is to set up the reporting environment, which needs to run reports on the metadata from different sources integrated and stored in the Metadata Manger warehouse. How you set up the reporting environment depends on the reporting requirements. The following options are available for creating reports:

• Use the existing schema and reports. Metadata Manger contains packaged reports that can be used to analyze business intelligence metadata, data integration metadata, data modeling tool metadata, and database catalog metadata. Metadata Manger also provides impact analysis and lineage reports that provide information on any type of metadata.

• Create new reports using the existing schema. Build new reports using the existing Metadata Manger metrics and attributes.

• Create new Metadata Manger warehouse tables and views to support the schema and reports. If the packaged Metadata Manger schema does not meet the reporting requirements, create new Metadata Manger warehouse tables and views.

• If you build new Metadata Manger warehouse tables or views, register the tables in the Metadata Manger schema and create new metrics/attributes in the Metadata Manger schema. Note that Metadata Manger schema is built on the Metadata Manger views.

After the environment setup is complete, test all schema objects, such as dashboards, analytic workflows, reports, metrics, attributes, and alerts.