The weakness of using relative rankings instead of absolute standards was exemplified in the TED Book by Jim Hornthal, A Haystack Full of Needles, using the humorous story of how Little Rock, Arkansas was initially rated as one of the world’s premier destinations for museums, romance, and fine dining by Triporati.com, a travel site intended to offer personalized destination recommendations powered by expert travel advice.

During alpha testing of their recommendation engine, Triporati discovered that the expert assigned to rate Little Rock had awarded the city’s best museums and restaurants with the highest score (5) because they were “the best that Little Rock has.”

No offense intended to Little Rock, Arkansas, but this is why you need data quality standards.

As Hornthal explained, “relative ranking alone is deadly.”  Triporati’s experts had to be re-trained to understand that the goal wasn’t to determine “What is the best that a city has to offer?” but rather, “How do these attributes compare to the best?”

To put their ratings into context for the experts, Triporati explained the rating system used by the recommendation engine for attractions (e.g., museums, restaurants) should represent the following:

• 5 — People would come from all over the world to visit the attraction
• 4 — People would come from all over the country to visit the attraction
• 3 — People would come from all across a state or region to visit the attraction
• 2 — People could be enticed to cross a city to visit the attraction
• 1 — People might be enticed to cross the street to visit the attraction

In the business scenario of the Triporati recommendation engine, improving data quality required the establishment of an absolute standard.  Most data quality professionals would say that this is an example of defining data quality as real-world alignment.

An alternative definition of data quality is fitness for the purpose of use, which is more akin to a relative ranking.  A common challenge is that data often has multiple business uses, each with its own fitness requirements, which is why applying data investigation to different business problems reveals different data quality tolerances that an absolute standard cannot reflect.

Furthermore, since some fine folks in the Southern United States might argue that Little Rock, Arkansas being a premier destination for museums and fine dining is more aligned with their real world, and since fine dining that is fit for the purpose of my use would be any restaurant that serves bacon double cheeseburgers and local micro-brewed beers, there is simply no accounting for taste.

But there needs to be an accounting for data quality, which is why you need data quality standards.

Fifteen years ago, the presentation of data typically fell under the purview of analysts and IT professionals. Quarterly or annual meetings entailed rolling data up into now quaint diagrams, graphs, and charts.

My, how times have changed. Today, data is everywhere. We have entered the era of Big Data and, as I write in Too Big to Ignore, many things are changing.

Big Data: Enterprise Shifts

In the workplace, let’s focus on two major shifts. First, today it’s becoming incumbent upon just about every member of a team, group, department, and organization to effectively present data in a compelling manner. Hidden in the petabytes of structured and unstructured data are key consumer, employee, and organizational insights that, if unleashed, would invariably move the needle.

Second, data no longer needs be presented on an occasional or periodic basis. Many employees are routinely looking at data of all types, a trend that will only intensify in the coming years.

The proliferation of effective data visualization tools like Ease.ly and Tableau provides tremendous opportunity. (The latter just went public with the übercool stock symbol $DATA.) Sadly, though, not enough employees—and, by extension, organizations—maximize the massive opportunity presented by data visualization. Of course, notable exceptions exist, but far too many professionals ignore DV tools. The result: they fail to present data in visually compelling ways. Far too many of us rely upon old standbys: bar charts, simple graphs, and the ubiquitous Excel spreadsheet. One of the biggest challenges to date with Big Data: Getting more people actually use the data–and the tools that make that data dance.

This begs the question: Why the lack of adoption? I’d posit that two factors are at play here:

• Lack of knowledge that such tools exist among end users.
• Many end users who know of these tools are unwilling to use them.

Simon Says: Make the Data Dance

Big Data in and of itself guarantees nothing. Presenting findings to senior management should involve more than pouring over thousands of records. Yes, the ability to drill down is essential. But starting with a compelling visual represents a strong start in gaining their attention.

Big Data is impossible to leverage with traditional tools (read: relational databases, SQL statements, Excel spreadsheets, and the like.)  Fortunately, increasingly powerful tools allow us to interpret and act upon previously unimaginable amounts of data. But we have to decide to use them.

Feedback

What say you?

It seems rather obvious to state that choosing between a dictatorship and a democracy is such an easy choice it doesn’t even need to be discussed.  However, when it comes to data, it seems like the choice is not so obvious, since for as long as I can remember the data management industry has been infatuated with the notion of instituting some form of a Data Dictatorship.

Providing the organization with a single system of record, a single version of the truth, a single view, a golden copy, or a consolidated repository of trusted data has long been the rallying cry and siren song of data warehousing, and more recently, of master data management.

I admit a data dictatorship has its appeal, especially since most information development concepts are easier to manage and govern when you only have to deal with one official source of enterprise data.

Of course, the reality is most organizations are a Data Democracy, which means that other data sources, both internal and external, will be used.  During a recent Twitter chat, one discussion thread noted how the democratization of data has lead to the consumerization of data, which has made the silo-ification of data easier than ever.  Cloud-based services (and other consumerization of IT trends) make rolling your own data silo simple and inexpensive, at least in terms of financial cost, but arguably expensive in terms of splintering the enterprise’s data asset.

Forging one big data silo in the cloud to rule them all might soon be pitched as a new form of data dictatorship.  For this to happen, users must surrender the freedoms consumerization brought them, but history has shown reverting back to dictatorship after democracy is difficult, if not impossible.

As Winston Churchill famously said, “no one pretends that democracy is perfect or all-wise.  Indeed, it has been said that democracy is the worst form of government except all those other forms that have been tried from time to time.”

No one pretends that data democracy is perfect or all-wise.  Indeed, most data professionals would say that data democracy is the worst form of data governance except all those other forms that have been tried from time to time.  But perhaps it’s simply time to stop pursuing any form of data dictatorship.

I’ve written before on this site on how Netflix uses data in fascinating ways. The company’s knowledge of its customers–and their viewing habits–is beyond impressive. Moreover, it’s instructive for companies attempting to navigate the era of Big Data.

Consider this Wired article explaining how Netflix operates and utilizes its data:

But Net­flix doesn’t only know what its audience likes to watch; it also knows how viewers like to watch it—beyond taste, the company understands habits. It doesn’t just know that we like to watch Breaking Bad; it knows that we like to watch four episodes of Breaking Bad in a row instead of going to sleep. It knows, in other words, that we like to binge.

Think about the astonishing level of knowledge that Netflix has developed on its customers. Not just the what, but the how, the when and (increasingly) the why. Talk about the Holy Grail. This should be the goal of every for-profit enterprise. Period.

Minimizing Risk

Equipped with information like this, Netflix can make more informed business decisions. Case in point: Its decision to resurrect the very popular cult classic Arrested Development. While by no means an inexpensive or risk-free venture, Netflix management is reasonably confident that its bet will pay off for one simple reason: its data supports the decision.

Note that each move Netflix makes is hardly guaranteed. No business decision even resembles complete certainty. But, by virtue of its exceptional data management, Netflix has moved the needle considerably. Its odds of success are without question much higher because it has minimized risk.

Think about the way in which far too many organizations operate these days. Forget Big Data and effectively harnessing its power. I’ve personally seen many enterprises manage their data so poorly that a comprehensive and accurate list of customers could not be produced in a reasonable period of time. Without this information, questions like how, when, and why could not be answered. The downstream effect: Decisions were based upon conjecture, rank, culture, and policy. This type of scenario is hardly ideal.

Simon Says

Rome, as they say, was not built in a day–and neither was Netflix. Rather than fret over the state of your data, take the steps now to improve your ability to analyze data in a few years.

Feedback

What say you?

Think for a minute about how much we spend on healthcare. In the United States, the numbers break down as follows:

• Roughly $3 trillion spent annually, a number rising at 6-7% per year
• This represents about 17% of US Gross Domestic Product (GDP)
• Some estimates put the number wasted annually on healthcare at a mind-boggling $2 trillion
• There’s at least $60B in annual Medicare fraud alone each year (and some estimates put that number at $250B) in fraud

For more astonishing data on healthcare, click here. The stats are frightening. With so much waste and opportunity, it should be no surprise that quite a few software vendors are focusing on Big Data–and not just behemoths like IBM. Start-ups like Explorys, Humedica, Apixio, and scores of others have entered the space.

Where’s the Data?

With so much action surrounding Big Data and healthcare, you’d think that there would be a tremendous number of examples. You’d expect there to be more statistics on how Big Data has helped organizations save lives, reduce costs, and increase revenue.

And you’d be wrong.

I’ve worked in hospitals a great deal over my career, and the term risk aversion is entirely apropos. Forget for a minute the significant difficulty in isolating cause and effect. (It’s not easy to accurately claim that deploying Hadoop throughout the organization saved 187 lives in 2012.)

Say for a minute that you’re the CIO of a healthcare organization and you make such a claim. Think about the potential ramifications from lawsuit-happy attorneys. Imagine having to respond to inquiries from lawyers about why you waited so long to deploy software that would have saved so many lives. What were you waiting for? How much will you pay my clients to drop their suit?

This isn’t to say that you can’t find data on, well, Big Data and healthcare. You can. You just have to look really hard–and you’ll more than likely be less than satisfied with the results. For example, this Humedica case study shows increased diagnosis of patients with diabetes who fell between the cracks.

Simon Says

Large organizations are conservative by their nature. Toss in potential lawsuits and it’s easy to understand the paucity results-oriented Big Data healthcare studies. What’s more, we’re still in the early innnings. Expect more data on Big Data in healthcare over the coming years.

Feedback

What say you?

In her book Being Wrong: Adventures in the Margin of Error, Kathryn Schulz explained “the pivotal insight of the Scientific Revolution was that the advancement of knowledge depends on current theories collapsing in the face of new insights and discoveries.  In this model of progress, errors do not lead us away from the truth. Instead, they edge us incrementally toward it.”

In his book Ignorance: How It Drives Science, Stuart Firestein explained “questions are more relevant than answers.  Questions are bigger than answers.  One good question can give rise to several layers of answers, inspire decades-long searches for solutions, generate whole new fields of inquiry, and prompt changes in entrenched thinking.  Answers, on the other hand, often end the process.”

Unfortunately, some people seem to misunderstand the goal of big data and data science to be the pursuit to provide the answers to all of our questions.  Some go so far as to claim that eventually we will know everything, that soon we will be able to foretell the future with absolute certainty.

These were a few of the misunderstandings addressed by Andrew McAfee in his recent Harvard Business Review blog post Pundits: Stop Sounding Ignorant About Data. “I’ve been talking and hanging out with a lot of data geeks over the past months and even though they’re highly ambitious people,” McAfee concluded, “they’re very circumspect when they talk about their work.  They know that the universe is a ridiculously messy and complex place and that all we can do is chip away at its mysteries with whatever tools are available, our brains always first and foremost among them.  The geeks are excited these days because in the current era of Big Data the tools just got a whole lot better.”

“The right question asked in the right way, rather than the accumulation of more data,” Firestein concluded, “allows a field to progress.  Scientists don’t just design an experiment based on what they don’t know.  The truly successful strategy is one that provides them even a glimpse of what’s on the other side of their ignorance and an opportunity to see if they can’t get the question to be bigger.  Ignorance works as the engine of science because it is virtually unbounded, and that makes science much more expansive.”

Science has always been about bigger questions, not bigger data.

In virtually every country there is a debate around privacy, driven most recently by the rise of Big Data, social networks and technology more generally.  Without doubt, the major technology companies recognise the value of the information they are collecting from individuals and are working very hard to minimise the impact of privacy changes by putting as much control as possible in the hands of the individual.

This debate will only intensify as the social networks push to also own the identity space.  Just look at the number of websites that encourage you to use Facebook, Google or similar services to log in.  In doing so, yet more of your data is available to these businesses which you are providing in exchange for the service of having your identity conveniently confirmed.

Citizens and consumers are becoming increasingly aware of the value of their personal information and hence are looking for more privacy options.  The default position of governments has been to revisit their privacy regulations.  This approach, though, is doomed from the start as no regulation can possibly keep up with this rapidly developing information economy.

In fact, it is likely to be economic forces that will provide the solution to the privacy risks of Big Data.  Both government and businesses are beginning to realise that they can make individuals much more comfortable to share their data if rather than providing an almost infinite (and incomprehensible) set of privacy settings they simply renounce any attempt at taking over ownership of the data and simply borrow or lease it with the permission of the true owner – you.

This approach is referred to as personally controlled records and is made possible by the databases that support the growth of Big Data.  No longer is it necessary to extract every piece of information and replicate it many times over in order to support complex analytics.  Rather it can be packaged as a neat record, kept in the control of the individual and purged upon expiry or the revocation of the lease that has been provided.

Rather than reducing the value to business and government, this approach actually opens up a huge array of new possibilities and leaves the individual in control.  The evidence is growing that when people feel confident that they can withdraw their information at any time and are not at risk of unintended consequences they are much more willing to submit their data for a wide array of purposes.

The protection of the individual in a world of Big Data is not through better privacy, but rather through clarifying who actually owns the data and ensuring that their rights are maintained.

Hear more in a Sky News interview I provided recently on trends in technology.