While much of the discussion about information management centres on things that are new and exciting, it is easy to neglect some of the basic principles that the profession has learnt over the last decade.  Here are just five things that I think are among the most important to consider if your project is to be a success.

First, use a standard project plan.  MIKE2.0 has been available for some years now and provides a work breakdown structure which is comprehensive.  Such an approach allows you to involve contractors and multiple service providers without being locked into anyone’s proprietary method.

Second, use data models that have been published.  There are many of them around ranging from low cost publications by authors such as Len Silverston through to enterprise models provided by the major software vendors.  Even the most expensive model is typically much cheaper than the labour cost that it can save.

Third, borrow from Don Rumsfeld: “There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know.”  The data warehouse is trying to manage the complexity of the entire business.  You can’t possibly know everything and hence requirements analysis should focus on the fundamental principles of the organisation and those things that are hard to undo later.

Fourth, the foundation of tomorrow’s enterprise data warehouse is unlikely to be today’s tactical solution.  Avoid the temptation to make the first iteration self-funding, the organisation has to be prepared to make an investment otherwise there are always cheaper short term solutions.

Finally, ask yourself whether your organisation is really as unique as your stakeholders think it is.  One of the most common reasons given for the use of unusual architectures or data models that don’t borrow from published materials is that the business is unique.  Everyone is looking for a point of differentiation but that doesn’t mean that you shouldn’t adopt standards where possible.  It is unlikely that the use of an unusual data warehouse architecture is going to enable a store to sell more toothpaste.  That same store, might, however, gain a real edge by combining consumer and supplier data in a new and novel way building on existing approaches to modelling the data.

Our profession is not a large one globally.  In fact, a couple of years ago I did a back-of-the-envelope calculation that there are about 50,000 to 100,000 people around the world who would regard themselves as primarily working in data or information management.  We worked this out by estimating that to specialise you would have to be working in one of the 5,000 largest organisations (private and government) and a reasonably constrained number of professional services organisations.

Regardless of whether the numbers are right, the numbers are not large given the impact that the profession has.  Most of us are connected in some way through the conferences we attend or the communities that we participate in (such as this one).  Looking at who are in the roles, anecdotally there appears to be a disproportionately large number of people with a data warehousing background.  I include myself in this category.

Over the past five or even ten years the field has broadened out to apply standardised approaches to data management.  The scope of this includes data quality, master data management, data integration, linking metadata to taxonomies and so on.  With a reach and impact that touches every aspect of the organisation, why is it that so many people in the field still come from a data warehousing background?

What is even more interesting, when you pull groups together to talk about data management, they almost always end-up referring back to the role that the data warehouse plays in the enterprise.  A discussion on master data management will almost always include reference to the standardisation that has happened within the data warehouse.  A discussion on data quality generally refers back to the process to cleanse the data warehouse.  A discussion on data integration seems to include a debate on the operational role of the data warehouse.

I am very interested in the view of our community.  Do we refer back to data warehouses because that is where so many of us came from, or is it that an architecture which has a data warehouse playing an important role is conducive to good data management?

When Ralph Kimball wrote “The Data Warehouse Toolkit” (published 1996) it defined Dimensional Modelling in a way that immediately demanded attention by data warehouse practitioners worldwide. The book and the techniques it described were not new and were common the approach we had used for the better part of a decade, what the book did do that was foundational was to describe the approach in a consistent and considered with a terminology that could be used by everyone.

There are many similar challenges that data warehouse designers face on every project. For instance two challenges we are often called upon to decide how to handle changes to source system models and the proper handling of changes to reference and master data.

The former is usually handled by splitting logical entities when creating physical tables separating attributes and relationships that have a higher probability of changing. The latter is commonly handled in one of three ways. Method one sees non volatile and volatile attributes are split into two tables (with a one to many relationship) Method two has the current attribute values are held in one table with changes over time maintained in a second table (again one to many). Finally, method three has changes across a number of concepts tracked in an audit table which is only intended for forensic purposes.

On recent data warehouse projects, we are using a variant of method one that has been formalised as “The Data Vault”.  The Data Vault techniques put forward by Dan Linstedt formalises both of these issues and makes sensible design recommendations. In particular, it adopts an approach using “hub”, “link” and “satellite” tables.

Originally, Linstedt attempted to patent these concepts, but this application was rejected and he has now adopted a free approach and is promoting his concepts through books, training and his web site: http://www.danlinstedt.com/

Well thought through online strategies can do so much more than deliver high quality web sites for internal and external users. They can dramatically improve some of your business fundamentals. There are few things more fundamental than the quality of your data.

When people think of data quality they often focus first on customer data. One of the best ways to ensure that customer data is right is to provide a way for your own customers to update their details online. On its own, this is an important capability, but to be really effective it needs to be linked to something that the customer regularly does on the web, such as reviewing their accounts, orders or other interactions with your organisation. Truly effective businesses make updating customer details part of every interaction and available to all stakeholders in the customer, effectively building a Facebook-like facility for their customers identifying relationships (friends), preferences and activities.

Apart from enhanced customer service, it is worth remembering that it is much harder to maintain a fraudulent identify when you are connected through multiple relationships and you have to maintain an exponential number of fronts.

Business data includes much more than just customer details. Online collaboration both inside and outside the enterprise can enhance almost all data in some way. One of the most common problems businesses face is maintaining an accurate understanding of the definition of complex business terminology. Every organisation develops their own language and expects staff, customers and business partners to understand it. Worse, few maintain a dictionary of this language.

Consider creating such a dictionary, with components that are visible internally, other parts to business partners and a relevant subset to the world in general. To really leverage the power of the web, make this dictionary readily updatable (even using a wiki). While open to misuse, it is unlikely that internal staff or business partners who are easily traced will deliberately abuse the privilege. Online communities have shown that complex topics attract genuinely interested contributors who can often provide a better explanation to their peers that you could hope to publish either from an insight or simple labour perspective.

Finally having learnt to use the web to better maintain customer data and your data dictionary, it rapidly becomes obvious that many datasets would be candidates to be open to a wider community for monitoring, comment or even enhancement. Consider lists of branches, community contacts and products. In the last case, suppliers sometimes make changes which flow through your supply chain without being updated in online catalogues.

If there is one thing we’ve learnt, the fear that we feel about opening our content up for collaboration is often disproportionate to the real risk of misuse. If you succumb to this fear without carefully considering what you are worried about, then you’ll miss out on the power that the crowd can bring to our business.

Readers interested in these concepts should read further about the intersection of Enterprise 2.0 and Information Management in MIKE2.0, in particular the MIKE2.0 Enterprise 2.0 Solution Offering.

Information Theory Approach to Data Quality for MDM

Introduction

Over the past decade data quality has been a major focus for data management professionals, data government organizations, and other data quality stakeholders across the enterprise. Still the quality of data remains low for many organizations. To a considerable extent this is caused by a lack of scientifically or at least consistently defined data quality metrics. Data professionals are still lacking a common methodology that would enable them to measure data quality objectively in terms of scientifically defined metrics and compare data sets in terms of their quality across systems, departments and corporations.  

Even though many data profiling metrics exist, their usage is not scientifically justified. Consequently enterprises and their departments apply their own standards or apply no standards at all.

As a result, regulatory agencies, executive management and data governance organizations are lacking a standard, objective and scientifically defined way to articulate data quality requirements and measure data quality improvement progress. An elusiveness of data quality results in that job performance of the enterprise roles responsible for data quality lacks consistently defined criteria, which ultimately causes limited progress in data quality improvements.

A quantitative approach to data quality, if developed and adopted by data management community, would enable data professionals to better prioritize data quality issues and take corrective actions proactively and efficiently.

In this article we will discuss a scientific approach to data quality for MDM based on Information Theory. This approach seems to be a good candidate to address the aforementioned problem.

Approaches to Data Quality

At a high level there are two well-known and broadly used approaches to data quality. Typically both of them are used to a certain degree by every enterprise.

The first approach is mostly application driven and oftentimes referred to as a “fit-for-purpose” approach. Oftentimes business users determine that certain application queries or reports do not return the right data. For instance if a query that is supposed to fetch top 10 Q2 customers does not return some of the customers the business expects to see, in depth data analysis follows. The data analysis may determine that some customer records are duplicated and some transaction records have incorrect or missing transaction dates. This type of finding can trigger some activities aimed at understanding of the data issues and corrective actions.

An advantage of this approach to data quality is that it is aligned with tactical needs of business functions, groups and departments. A disadvantage of this approach is that it addresses data quality issues re-actively based upon business request or even complaint. Some data quality issues may not be easy to discover and business users cannot decide which report is right and which one is wrong. The organization may eventually draw a conclusion that their data is bad but would not be able to indicate what exactly needs to be fixed in the data, which limits the IT’s abilities to fix the issues. When multiple LOB’s and functions across the enterprise struggle with their specific data quality issues separately, it is difficult to quantify the overall state of data quality and define priorities with which data quality problems are to be addressed by the enterprise.

The second approach is based on data profiling. Data profiling tools are intended to make a data quality improvement process more pro-active and measurable. A number of data profiling metrics is typically introduced to screen for missing and invalid attributes, duplicate records, duplicate attribute values that are supposed to be unique, frequency of attributes, cardinality of attributes and their allowed values, standardization and validation of certain data formats for simple and complex attribute types, violations of referential integrity, etc. A limitation of the data profiling techniques is in that an additional analysis is required to understand which of the metrics are most important for the business and why. It may not be easy to come up with a definitive answer and translate it into a data quality improvement action plan. The variety of data profiling metrics is not based on science but rather driven by the variety of ways relational database technology can report on data quality issues.

Each of the two approaches above has its niche and significance. When the quality of master data is in question an alternative and more strategic approach can be considered by data governance organizations. This approach avoids detailed analysis of business applications while providing a solid scientific foundation for its metrics.

Information Theory Approach to Data Quality for MDM  

Master data are those data which are foundational to business processes, are usually widely distributed, which, when well managed, are directly contributing
to the success of an organization, and when not well managed pose the most risk. Customer, Patient, Citizen, Member, Client, Member, Broker, Product, Financial Instrument, Drug are the entities oftentimes referred to as master data entities while company specific selection of master entities is driven by the enterprise business and focus.

Master Data Service (MDS) defines its primary function as the creation of the “golden view” of the master entities. We will assume that MDS has successfully created and maintains the “golden view” of entity F in the data hub. This “golden record” can be dynamic or persistent. There exist a number of data sources across the enterprise with the data corresponding to domain F. This includes the source systems that feed the data hub and other data sources that may be not integrated with the data hub. We will define an external dataset f which data quality is to be quantified with respect to F. For the purpose of this discussion f can represent any data set such as a single data source or multiple sources.

Our goal is to compare the source data set f with the entity data set F. The data quality of the data set f will be characterized by how well it represents the benchmark entity F defined as the “golden view” for the data in domain F. We are making an assumption here that the “golden view” was created algorithmically and then validated by the data stewards.

In Information Theory the information quantity associated with the entity F is expressed in terms of the entropy:

                    H(F) = – ∑ P_k log P_k,                                                                                            (1)   

where P_k are the probabilities of the attribute (token) values in the “golden” data set F. Index “K” runs over all records in F and all attributes. The base in the log function is 2.

H(F) represents the quantity of information in the “golden” representation of entity F.

Similarly for the comparison data set f

                    H(f) = – ∑ p_i log p_i,                                                                                            (2)   

We will use small “p” for the probabilities associated with f while capital letter “P” is used for the probabilities characterizing the “golden” entity record.

Mutual entropy J(f,F) characterizes how well f represents F.

J(f,F) = H(f) + H(F) – H(f,F)                                                                        (3)   

In (3) H(f,F) is the joint entropy of f and F. It is expressed in terms of probabilities of combined events, e.g. the probability that the name = “Smith” in “the golden record” F and name = “Schmidt” in the source record linked to the same entity. The behavior of J qualifies this function as a good candidate quantifying the data quality of f with respect to F. When the data quality is low, the correlation between f and F is low. In an extreme case of a very low data quality f doesn’t correlate with F and these variables are independent. Then

                    H(f,F) = H(f) + H(F)                                                                                      (4)   

and

                    J(f,F) = 0                                                                                                       (5)   

If f represents F extremely well, e.g. f = F, then H(f) = H(F) = H(f,F) and

                    J(f,F) = H(F)                                                                                                  (6)   

We define Data Quality of f with respect to F by the following equation:

                    DQ(f,F) = J(f,F)/H(F)                                                                                      (7)   

With this definition of data quality DQ changes from 0 to 1, where 0 indicates the data quality of f is minimal; f does not represent F.  When DQ = 1 f perfectly represents F and the data quality of f with respect to F is 100%, and therefore f represents F perfectly well.

The approach can also be used to determine partial attribute/token level data quality. This will provide additional insights into what causes most significant data quality issues.

The data quality improvement should be done iteratively. Changes in the data source data may impact the “golden record”. Then equations (1) and (7) are applied again to recalculate the data quantity and data quality characteristics.

Conclusion

The article offers an Information Theory based method for quantifying Information Assets and the Data Quality of the Assets through equations (1) and (7). The proposed method leverages the notion of a “golden record” created and maintained in the data hub. The “golden record” is used as the benchmark against which the data quality of other sources is measured.

Organizations can leverage this approach to augment its data governance offerings for MDM and make our data governance approach truly unique. The quantitative approach to data quality will ultimately help data governance organizations develop policies based on scientifically defined data quality and quantity metrics.

By applying this approach consistently on a number of engagements, over time we will accumulate valuable insights into how metrics (1) and (7) apply to real world data characteristics and scenarios. We will develop good practices defining acceptable data quality thresholds, e.g. it might be a future industry policy for P&C insurance business to keep the quality of Customer data above the 92% mark, which sets clearly articulated data governance policy based on scientifically sound approach to data quality metrics.

The developed approach can be incorporated in the future products to enable data governance and provide data governance organizations with new tooling. Data governance will be able to select information sources and assets to be measured, quantify them according to (1) and (7), set the target metrics for data stewards, measure the progress on an on-going basis and report on the data quality improvement progress.

Even though we are mainly focusing on data quality, the quantity of data in equation (1) characterizes the overall significance of a corporate data set from the Information Theory perspective. For M&A the method can be used to measure an additional amount of information that the joint enterprise will have compared to the information owned by the companies separately. The approach developed above will measure both the information acquired due to the difference in the customer bases and the information quantity increment due to better and more precise and useful information about the existing customers.

  Simple Illustrative Examples

In this Appendix we will apply the theory developed above to two simple illustrative cases. We will define the “golden” data set F as follows:

The probabilities of attributes values in F are:

Scenario 1

Dataset f is the same as the “golden” data set. Then

                                                f = F, H(f) = H(F) = 3.5.

The probability matrix for combined values:

and

                                                         H(F) = H(f) = H(f,F) = 3.5

                                             J(f,F) = H(F) + H(f) – H(f,F) = H(F) = 3.5

Equation (7) yields

                                                           DQ = J(f,F)/H(F) = 1

As expected the data quality of f when f = F yields 1 or 100%

Scenario 2

Dataset for the “golden record” F remains the same as in scenario 1.

                                                              H(F) = 3.5

We will change dataset f by adding a new record: “Larry, CA”. We will assume that the new record for “Larry” represent the same individual as “Larry, NJ”. Therefore records “Larry, NJ” and “Larry, CA” will have the same EID = 1. Data stewards determined that “NJ” is the right value for the attribute State.  Dataset f is as follows:

 The probability matrix for f is:

The probability matrix for combined values:

Substituting the values for H(F), H(f) and H(f,F) into 7 we will obtain:

                                 J(f,F) = 3.5 + 3.292878689 – 3.84385619 = 2.949022499

                                  DQ = J(f,F)/H(F) = 2.949022499/3.5 = 0.842577857 or ~ 84%

As I described in my last post, the quantity of information being generated globally and within each of our organisations is absolutely overwhelming.  All good managers facing a large problem start by trying to break the task down into manageable pieces.  The question information managers face is what is the right starting point for breaking enterprise information into such manageable pieces.  I’ve seen organisations start with technology (structured database, records, documents, email, HTML etc.).  I’ve seen others start by the subject being covered (customer, finance, human resources, product etc.).

A better approach is to ask how the information is used by the business.  Over many years, I have come to the conclusion that there are four ways that information is used.

The first use is the measurement of performance from executive to operations (for example the Balanced Scorecard).  The second use is to navigate the organisation via location, product, staff, customer or other common concepts (for example Master Data or Dimensional Models).  The third is to describe the business in an abstract or atomic way (for example third normal form data models in the data warehouse or the Enterprise Content Management repository).  Finally, the fourth is the operational system data which sits in front of the customer or production-line process.

Readers who are interested in exploring these ideas further can read a more detailed article on the Four Layers of Information.

Data quality (DQ) and data governance are key topics on the information management agenda at most large organsations. Lots of literature and guidance on how to move data quality forward from a technical, IT-level issue to become a business-owned issue. Once it’s a business-owned issue, organisations can push forward with improvements in the areas that matter most to the bottom line and also top line.

In addition to assets for data quality improvement, MIKE2.0 also offers the IM Quickscan tool . Read here how it helped a real customer raise the profile of data quality within the organisation: http://www.dataqualitypro.com/data-quality-home/how-to-create-a-data-quality-survey-for-your-organisation.html

I feel the QuickScan survey was a great success in gaining insight into the perception of Data Quality within our organization as well as securing more involvement from all concerned parties. Further, the people who participated that were not previously interested in Data Quality seemed to take great interest in the results. Finally, the survey helped cement the business cases for a Data Quality Solution and greater Data Governance.

The UK Cabinet Office just completed an innovative competition called ‘Show Us A Better Way’. The government produces massive amounts of data on crime, on health, on education. This competition is looking for innovative ways to use this information, e.g. in mashups, and to release more value to the public.

The MIKE2.0 community also submitted a proposal, based on our experience with setting up MIKE2.0 and using omCollab:

MyGov Personal Government

Tell us about your ideas (leave a comment…)!

The EDM council has recently published a piece of primary research on the executive view of data and its relationship to current operational practices. The report is based on interviews with heads of data management at 20 global financial institutions. The key message is that while data is being seen as a critical player in meeting business, operational and compliance objectives, the majority of organisations are still in a “clean and consolidate” mode, trying to deliver the initial migration of content from their multiple business silos into centralised authoritative source systems.

The report then touches on the executive view of data management, the business view, funding and governance issues and implementation. Key actions to consider are:

• Build your credibility on execution capability within the organisation
• Be transparent to the business with operational metrics and SLA’s
• Get metadata management right
• Federated centralisation of business processes and operating model
• Prove each component of EDM with a ROI

For a full copy of the report click here.

As part of MIKE2.0, we believe we are presenting a unique perspective in the area of standards development. Our approach is to create a collaborative community for the development of standards for Information Management, including those that apply to Capital Markets.

Some interesting work around open source and open standards is developing in relation to market data:

• Market Data Definition Language (MDDL) is an extensible Markup Language (XML) derived specification, which facilitates the interchange of information about financial instruments used throughout the world’s markets. A community is build around MDDL, including a wiki-based development environment.

• FAST (FIX Adapted for Streaming data) protocol is emerging standard used for the distribution of market data. Some of the steps being discussed with the open sourcing of FAST could be seen be particularly beneficial to this standard as it continues to evolve.

With open content and collaborative technologies, it’s easy for these projects to work together and we’ve starting doing this through MIKE2.0 with references to these projects.