I recently had some customer service issues with a few telecommunication companies. When calling the 800 numbers and speaking with reps about my issues (and there were quite a few), one thing kept frustrating me: receiving different answers to my questions from different people from the same company.

In large part, this stemmed from the fact that different customer service reps at each company had access to different data. Oh, how I loved the lines “I don’t have access to that information” and “That’s not my department, sir.” I don’t attribute this to issues related to permissions, rights, or security. Rather, I strongly suspect that each department kept its data in different systems. Because these systems didn’t talk to each other (or receive real-time updates from each other via some type of ETL tool), each rep would tell me the “current” information in his/her own system.

Now, it’s easy for me—and people like me—to view the world exclusively through our normal lenses. In my case, I’m a technology guy. People like me know more than a little about information management (IM). We understand things that some people don’t. (Of course, we often don’t understand things that we probably should, but that’s a conversation over beers sometime.)

IM types understand all too well why many large organizations struggle keeping tabs on their data. The usual suspects are:

• An eye chart of disparate systems
• End user inattention to detail
• Organizational charts nowhere near transparent
• Bifurcation of responsibilities among different departments
• An inattention to data quality and data governance

John Q. Public

To most people reading this post (and similar posts on data- and management-oriented sites), this is old hat. But consider John Q. Public. He is no data management expert. He doesn’t know the difference between a house key and a primary key. He probably wouldn’t be able to define data quality or referential integrity.

But here’s the rub: It doesn’t matter.

John’s no idiot. He knows bad customer service when he experiences it. He may not know why a company is giving him the runaround, but he sure as heck doesn’t like it. What’s more, he now has new options for expressing his disgruntlement.

The social web has shifted the pendulum to the customer, empowering people in ways simply not possible even fifteen years ago. As Maria Ogneva writes, this “social customer” is not terribly tolerant of old school customer service practices. They don’t bite their tongue when things go awry. Ogneva writes:

With easy publishing tools, creating content is easy. With social sharing tools, it’s even easier to share this content with as many people as are in your network, and with platforms as good for discovery as Twitter is, you can even expose your message to anyone tracking that topic. So now, this same customer, who’s had opinions of her own about any and every product she’s ever used, can publish and share her thoughts with a mouse click. And if you are looking for information on that product, you can find what other people have already said.

Simon Says

Today customers have at their disposal many tools to communicate their happiness—or lack thereof. This has enormous implications for data management. Small businesses get a bit of a free pass here because they often know their customers via personal relationship. By contrast, few corporations can make that claim. When it comes to more effectively managing their data, large organizations need to get their acts together. The risk of not doing so: losing current and future customers.

Feedback

What do you think?

It looks like 2010 will be the year of Analytics with advertising from most of the major players and tie-ups between IBM and SPSS as well as Accenture and SAS just to name a couple of high profile examples.  Read a little deeper though and the real business activity gets a whole lot more interesting.

It’s true that most of the focus has recently been on analytical techniques, with books such as Competing on Analytics (Davenport and Harris) popularising terms like “Analytics Competitor”.  Improvements in computing power have also made complex analytics practical on almost every dataset.  However, what is really driving the trend?  Davenport and Harris themselves provide some of the evidence in their book when they identify that one of the most important factors in the take-up of analytics is the presence of a leader (typically the CEO) who believes in the power of analytics and data.  They use in their book a quote, original from W. Edwards Deming): “In god we trust, all others bring data”.

What does this all mean?  Fundamentally the pinnacle of extracting value from data is interpreting it, to do this you need very clever analytics.  To do the analytics, you need great data that is not biased by any management interpretation – that sounds a lot like Enterprise Data Management.  To leverage the analytics, you need an organisation that is committed to using information, learning from the past and applying it to the future – that sounds a lot like Information Management.

2010 is going to be very interesting with the hope that we can bring data, information and analytics to the fore of business thinking.

Jason Sharpe (Telstra General Manager for Collaboration, Knowledge, and Intelligence) and I met this week to discuss an area that we both feel strongly about: the need for standards within and across organizations for the development of reports.  We think this isn’t really an area that any commercial business looks to use as a point of differentiation, rather it is a “ticket to play” for good information practice.

Jason and I are looking for collaborators to develop vendor-neutral reporting templates (building on the existing MIKE2.0 Deliverable Templates) which could then be extended to vendor-specific templates and cross referenced to the MIKE2.0 WBS.  By doing this within MIKE2.0, client organizations can effectively engage their vendors and service providers to finalize these documents for their specific needs which reduces the vendor’s cost of sale while improve the speed and quality of outcome for their clients.

Please let Jason and myself know if you are interested in taking part in this initiative.

While much of the discussion about information management centres on things that are new and exciting, it is easy to neglect some of the basic principles that the profession has learnt over the last decade.  Here are just five things that I think are among the most important to consider if your project is to be a success.

First, use a standard project plan.  MIKE2.0 has been available for some years now and provides a work breakdown structure which is comprehensive.  Such an approach allows you to involve contractors and multiple service providers without being locked into anyone’s proprietary method.

Second, use data models that have been published.  There are many of them around ranging from low cost publications by authors such as Len Silverston through to enterprise models provided by the major software vendors.  Even the most expensive model is typically much cheaper than the labour cost that it can save.

Third, borrow from Don Rumsfeld: “There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know.”  The data warehouse is trying to manage the complexity of the entire business.  You can’t possibly know everything and hence requirements analysis should focus on the fundamental principles of the organisation and those things that are hard to undo later.

Fourth, the foundation of tomorrow’s enterprise data warehouse is unlikely to be today’s tactical solution.  Avoid the temptation to make the first iteration self-funding, the organisation has to be prepared to make an investment otherwise there are always cheaper short term solutions.

Finally, ask yourself whether your organisation is really as unique as your stakeholders think it is.  One of the most common reasons given for the use of unusual architectures or data models that don’t borrow from published materials is that the business is unique.  Everyone is looking for a point of differentiation but that doesn’t mean that you shouldn’t adopt standards where possible.  It is unlikely that the use of an unusual data warehouse architecture is going to enable a store to sell more toothpaste.  That same store, might, however, gain a real edge by combining consumer and supplier data in a new and novel way building on existing approaches to modelling the data.

Our profession is not a large one globally.  In fact, a couple of years ago I did a back-of-the-envelope calculation that there are about 50,000 to 100,000 people around the world who would regard themselves as primarily working in data or information management.  We worked this out by estimating that to specialise you would have to be working in one of the 5,000 largest organisations (private and government) and a reasonably constrained number of professional services organisations.

Regardless of whether the numbers are right, the numbers are not large given the impact that the profession has.  Most of us are connected in some way through the conferences we attend or the communities that we participate in (such as this one).  Looking at who are in the roles, anecdotally there appears to be a disproportionately large number of people with a data warehousing background.  I include myself in this category.

Over the past five or even ten years the field has broadened out to apply standardised approaches to data management.  The scope of this includes data quality, master data management, data integration, linking metadata to taxonomies and so on.  With a reach and impact that touches every aspect of the organisation, why is it that so many people in the field still come from a data warehousing background?

What is even more interesting, when you pull groups together to talk about data management, they almost always end-up referring back to the role that the data warehouse plays in the enterprise.  A discussion on master data management will almost always include reference to the standardisation that has happened within the data warehouse.  A discussion on data quality generally refers back to the process to cleanse the data warehouse.  A discussion on data integration seems to include a debate on the operational role of the data warehouse.

I am very interested in the view of our community.  Do we refer back to data warehouses because that is where so many of us came from, or is it that an architecture which has a data warehouse playing an important role is conducive to good data management?

When Ralph Kimball wrote “The Data Warehouse Toolkit” (published 1996) it defined Dimensional Modelling in a way that immediately demanded attention by data warehouse practitioners worldwide. The book and the techniques it described were not new and were common the approach we had used for the better part of a decade, what the book did do that was foundational was to describe the approach in a consistent and considered with a terminology that could be used by everyone.

There are many similar challenges that data warehouse designers face on every project. For instance two challenges we are often called upon to decide how to handle changes to source system models and the proper handling of changes to reference and master data.

The former is usually handled by splitting logical entities when creating physical tables separating attributes and relationships that have a higher probability of changing. The latter is commonly handled in one of three ways. Method one sees non volatile and volatile attributes are split into two tables (with a one to many relationship) Method two has the current attribute values are held in one table with changes over time maintained in a second table (again one to many). Finally, method three has changes across a number of concepts tracked in an audit table which is only intended for forensic purposes.

On recent data warehouse projects, we are using a variant of method one that has been formalised as “The Data Vault”.  The Data Vault techniques put forward by Dan Linstedt formalises both of these issues and makes sensible design recommendations. In particular, it adopts an approach using “hub”, “link” and “satellite” tables.

Originally, Linstedt attempted to patent these concepts, but this application was rejected and he has now adopted a free approach and is promoting his concepts through books, training and his web site: http://www.danlinstedt.com/

Well thought through online strategies can do so much more than deliver high quality web sites for internal and external users. They can dramatically improve some of your business fundamentals. There are few things more fundamental than the quality of your data.

When people think of data quality they often focus first on customer data. One of the best ways to ensure that customer data is right is to provide a way for your own customers to update their details online. On its own, this is an important capability, but to be really effective it needs to be linked to something that the customer regularly does on the web, such as reviewing their accounts, orders or other interactions with your organisation. Truly effective businesses make updating customer details part of every interaction and available to all stakeholders in the customer, effectively building a Facebook-like facility for their customers identifying relationships (friends), preferences and activities.

Apart from enhanced customer service, it is worth remembering that it is much harder to maintain a fraudulent identify when you are connected through multiple relationships and you have to maintain an exponential number of fronts.

Business data includes much more than just customer details. Online collaboration both inside and outside the enterprise can enhance almost all data in some way. One of the most common problems businesses face is maintaining an accurate understanding of the definition of complex business terminology. Every organisation develops their own language and expects staff, customers and business partners to understand it. Worse, few maintain a dictionary of this language.

Consider creating such a dictionary, with components that are visible internally, other parts to business partners and a relevant subset to the world in general. To really leverage the power of the web, make this dictionary readily updatable (even using a wiki). While open to misuse, it is unlikely that internal staff or business partners who are easily traced will deliberately abuse the privilege. Online communities have shown that complex topics attract genuinely interested contributors who can often provide a better explanation to their peers that you could hope to publish either from an insight or simple labour perspective.

Finally having learnt to use the web to better maintain customer data and your data dictionary, it rapidly becomes obvious that many datasets would be candidates to be open to a wider community for monitoring, comment or even enhancement. Consider lists of branches, community contacts and products. In the last case, suppliers sometimes make changes which flow through your supply chain without being updated in online catalogues.

If there is one thing we’ve learnt, the fear that we feel about opening our content up for collaboration is often disproportionate to the real risk of misuse. If you succumb to this fear without carefully considering what you are worried about, then you’ll miss out on the power that the crowd can bring to our business.

Readers interested in these concepts should read further about the intersection of Enterprise 2.0 and Information Management in MIKE2.0, in particular the MIKE2.0 Enterprise 2.0 Solution Offering.

Information Theory Approach to Data Quality for MDM

Introduction

Over the past decade data quality has been a major focus for data management professionals, data government organizations, and other data quality stakeholders across the enterprise. Still the quality of data remains low for many organizations. To a considerable extent this is caused by a lack of scientifically or at least consistently defined data quality metrics. Data professionals are still lacking a common methodology that would enable them to measure data quality objectively in terms of scientifically defined metrics and compare data sets in terms of their quality across systems, departments and corporations.  

Even though many data profiling metrics exist, their usage is not scientifically justified. Consequently enterprises and their departments apply their own standards or apply no standards at all.

As a result, regulatory agencies, executive management and data governance organizations are lacking a standard, objective and scientifically defined way to articulate data quality requirements and measure data quality improvement progress. An elusiveness of data quality results in that job performance of the enterprise roles responsible for data quality lacks consistently defined criteria, which ultimately causes limited progress in data quality improvements.

A quantitative approach to data quality, if developed and adopted by data management community, would enable data professionals to better prioritize data quality issues and take corrective actions proactively and efficiently.

In this article we will discuss a scientific approach to data quality for MDM based on Information Theory. This approach seems to be a good candidate to address the aforementioned problem.

Approaches to Data Quality

At a high level there are two well-known and broadly used approaches to data quality. Typically both of them are used to a certain degree by every enterprise.

The first approach is mostly application driven and oftentimes referred to as a “fit-for-purpose” approach. Oftentimes business users determine that certain application queries or reports do not return the right data. For instance if a query that is supposed to fetch top 10 Q2 customers does not return some of the customers the business expects to see, in depth data analysis follows. The data analysis may determine that some customer records are duplicated and some transaction records have incorrect or missing transaction dates. This type of finding can trigger some activities aimed at understanding of the data issues and corrective actions.

An advantage of this approach to data quality is that it is aligned with tactical needs of business functions, groups and departments. A disadvantage of this approach is that it addresses data quality issues re-actively based upon business request or even complaint. Some data quality issues may not be easy to discover and business users cannot decide which report is right and which one is wrong. The organization may eventually draw a conclusion that their data is bad but would not be able to indicate what exactly needs to be fixed in the data, which limits the IT’s abilities to fix the issues. When multiple LOB’s and functions across the enterprise struggle with their specific data quality issues separately, it is difficult to quantify the overall state of data quality and define priorities with which data quality problems are to be addressed by the enterprise.

The second approach is based on data profiling. Data profiling tools are intended to make a data quality improvement process more pro-active and measurable. A number of data profiling metrics is typically introduced to screen for missing and invalid attributes, duplicate records, duplicate attribute values that are supposed to be unique, frequency of attributes, cardinality of attributes and their allowed values, standardization and validation of certain data formats for simple and complex attribute types, violations of referential integrity, etc. A limitation of the data profiling techniques is in that an additional analysis is required to understand which of the metrics are most important for the business and why. It may not be easy to come up with a definitive answer and translate it into a data quality improvement action plan. The variety of data profiling metrics is not based on science but rather driven by the variety of ways relational database technology can report on data quality issues.

Each of the two approaches above has its niche and significance. When the quality of master data is in question an alternative and more strategic approach can be considered by data governance organizations. This approach avoids detailed analysis of business applications while providing a solid scientific foundation for its metrics.

Information Theory Approach to Data Quality for MDM  

Master data are those data which are foundational to business processes, are usually widely distributed, which, when well managed, are directly contributing
to the success of an organization, and when not well managed pose the most risk. Customer, Patient, Citizen, Member, Client, Member, Broker, Product, Financial Instrument, Drug are the entities oftentimes referred to as master data entities while company specific selection of master entities is driven by the enterprise business and focus.

Master Data Service (MDS) defines its primary function as the creation of the “golden view” of the master entities. We will assume that MDS has successfully created and maintains the “golden view” of entity F in the data hub. This “golden record” can be dynamic or persistent. There exist a number of data sources across the enterprise with the data corresponding to domain F. This includes the source systems that feed the data hub and other data sources that may be not integrated with the data hub. We will define an external dataset f which data quality is to be quantified with respect to F. For the purpose of this discussion f can represent any data set such as a single data source or multiple sources.

Our goal is to compare the source data set f with the entity data set F. The data quality of the data set f will be characterized by how well it represents the benchmark entity F defined as the “golden view” for the data in domain F. We are making an assumption here that the “golden view” was created algorithmically and then validated by the data stewards.

In Information Theory the information quantity associated with the entity F is expressed in terms of the entropy:

                    H(F) = – ∑ P_k log P_k,                                                                                            (1)   

where P_k are the probabilities of the attribute (token) values in the “golden” data set F. Index “K” runs over all records in F and all attributes. The base in the log function is 2.

H(F) represents the quantity of information in the “golden” representation of entity F.

Similarly for the comparison data set f

                    H(f) = – ∑ p_i log p_i,                                                                                            (2)   

We will use small “p” for the probabilities associated with f while capital letter “P” is used for the probabilities characterizing the “golden” entity record.

Mutual entropy J(f,F) characterizes how well f represents F.

J(f,F) = H(f) + H(F) – H(f,F)                                                                        (3)   

In (3) H(f,F) is the joint entropy of f and F. It is expressed in terms of probabilities of combined events, e.g. the probability that the name = “Smith” in “the golden record” F and name = “Schmidt” in the source record linked to the same entity. The behavior of J qualifies this function as a good candidate quantifying the data quality of f with respect to F. When the data quality is low, the correlation between f and F is low. In an extreme case of a very low data quality f doesn’t correlate with F and these variables are independent. Then

                    H(f,F) = H(f) + H(F)                                                                                      (4)   

and

                    J(f,F) = 0                                                                                                       (5)   

If f represents F extremely well, e.g. f = F, then H(f) = H(F) = H(f,F) and

                    J(f,F) = H(F)                                                                                                  (6)   

We define Data Quality of f with respect to F by the following equation:

                    DQ(f,F) = J(f,F)/H(F)                                                                                      (7)   

With this definition of data quality DQ changes from 0 to 1, where 0 indicates the data quality of f is minimal; f does not represent F.  When DQ = 1 f perfectly represents F and the data quality of f with respect to F is 100%, and therefore f represents F perfectly well.

The approach can also be used to determine partial attribute/token level data quality. This will provide additional insights into what causes most significant data quality issues.

The data quality improvement should be done iteratively. Changes in the data source data may impact the “golden record”. Then equations (1) and (7) are applied again to recalculate the data quantity and data quality characteristics.

Conclusion

The article offers an Information Theory based method for quantifying Information Assets and the Data Quality of the Assets through equations (1) and (7). The proposed method leverages the notion of a “golden record” created and maintained in the data hub. The “golden record” is used as the benchmark against which the data quality of other sources is measured.

Organizations can leverage this approach to augment its data governance offerings for MDM and make our data governance approach truly unique. The quantitative approach to data quality will ultimately help data governance organizations develop policies based on scientifically defined data quality and quantity metrics.

By applying this approach consistently on a number of engagements, over time we will accumulate valuable insights into how metrics (1) and (7) apply to real world data characteristics and scenarios. We will develop good practices defining acceptable data quality thresholds, e.g. it might be a future industry policy for P&C insurance business to keep the quality of Customer data above the 92% mark, which sets clearly articulated data governance policy based on scientifically sound approach to data quality metrics.

The developed approach can be incorporated in the future products to enable data governance and provide data governance organizations with new tooling. Data governance will be able to select information sources and assets to be measured, quantify them according to (1) and (7), set the target metrics for data stewards, measure the progress on an on-going basis and report on the data quality improvement progress.

Even though we are mainly focusing on data quality, the quantity of data in equation (1) characterizes the overall significance of a corporate data set from the Information Theory perspective. For M&A the method can be used to measure an additional amount of information that the joint enterprise will have compared to the information owned by the companies separately. The approach developed above will measure both the information acquired due to the difference in the customer bases and the information quantity increment due to better and more precise and useful information about the existing customers.

  Simple Illustrative Examples

In this Appendix we will apply the theory developed above to two simple illustrative cases. We will define the “golden” data set F as follows:

The probabilities of attributes values in F are:

Scenario 1

Dataset f is the same as the “golden” data set. Then

                                                f = F, H(f) = H(F) = 3.5.

The probability matrix for combined values:

and

                                                         H(F) = H(f) = H(f,F) = 3.5

                                             J(f,F) = H(F) + H(f) – H(f,F) = H(F) = 3.5

Equation (7) yields

                                                           DQ = J(f,F)/H(F) = 1

As expected the data quality of f when f = F yields 1 or 100%

Scenario 2

Dataset for the “golden record” F remains the same as in scenario 1.

                                                              H(F) = 3.5

We will change dataset f by adding a new record: “Larry, CA”. We will assume that the new record for “Larry” represent the same individual as “Larry, NJ”. Therefore records “Larry, NJ” and “Larry, CA” will have the same EID = 1. Data stewards determined that “NJ” is the right value for the attribute State.  Dataset f is as follows:

 The probability matrix for f is:

The probability matrix for combined values:

Substituting the values for H(F), H(f) and H(f,F) into 7 we will obtain:

                                 J(f,F) = 3.5 + 3.292878689 – 3.84385619 = 2.949022499

                                  DQ = J(f,F)/H(F) = 2.949022499/3.5 = 0.842577857 or ~ 84%

As I described in my last post, the quantity of information being generated globally and within each of our organisations is absolutely overwhelming.  All good managers facing a large problem start by trying to break the task down into manageable pieces.  The question information managers face is what is the right starting point for breaking enterprise information into such manageable pieces.  I’ve seen organisations start with technology (structured database, records, documents, email, HTML etc.).  I’ve seen others start by the subject being covered (customer, finance, human resources, product etc.).

A better approach is to ask how the information is used by the business.  Over many years, I have come to the conclusion that there are four ways that information is used.

The first use is the measurement of performance from executive to operations (for example the Balanced Scorecard).  The second use is to navigate the organisation via location, product, staff, customer or other common concepts (for example Master Data or Dimensional Models).  The third is to describe the business in an abstract or atomic way (for example third normal form data models in the data warehouse or the Enterprise Content Management repository).  Finally, the fourth is the operational system data which sits in front of the customer or production-line process.

Readers who are interested in exploring these ideas further can read a more detailed article on the Four Layers of Information.

Data quality (DQ) and data governance are key topics on the information management agenda at most large organsations. Lots of literature and guidance on how to move data quality forward from a technical, IT-level issue to become a business-owned issue. Once it’s a business-owned issue, organisations can push forward with improvements in the areas that matter most to the bottom line and also top line.

In addition to assets for data quality improvement, MIKE2.0 also offers the IM Quickscan tool . Read here how it helped a real customer raise the profile of data quality within the organisation: http://www.dataqualitypro.com/data-quality-home/how-to-create-a-data-quality-survey-for-your-organisation.html
I feel the QuickScan survey was a great success in gaining insight into the perception of Data Quality within our organization as well as securing more involvement from all concerned parties. Further, the people who participated that were not previously interested in Data Quality seemed to take great interest in the results. Finally, the survey helped cement the business cases for a Data Quality Solution and greater Data Governance.