Information, data, and metadata are three interrelated words we hear a lot in the enterprise information management industry.  An example of the difference, and relationship, between data and information is grapes and wine, where data is to grapes as information is to wine, meaning that information is created from data.  And metadata is essential to understanding data, information, and the business and technical aspects of the processes that transform data into information.

In fact, the importance of metadata adding context all along the journey from data to information can not be overstated.  As David Weinberger explained in his book Too Big to Know, “the atoms of data hook together only because they share metadata.”

Although it has always played an essential role in information development, metadata management has an even bigger role play in the era of big data and information overload.

“The solution to the information overload problem,” according to Weinberger, “is to create more information: metadata.  When you put a label on a folder, you’re using metadata so that you can find the papers within it . . . just as a caption helps us make sense of a photo.”

Photos in need of captions and videos in need of categories are great examples of the growing rise of unstructured data, which is deepening our dependence on metadata.  And the semi-structured data of social media (e.g., tweets with hashtags) is another example of how data without the context provided by metadata will never be able to complete its journey to information.

Of course, the journey doesn’t end with information.  In 1988, Russell Ackoff, as Weinberger explained, “sketched a pyramid that has probably been redrawn on a white board somewhere in the world every hour since.  The largest layer at the bottom of the pyramid represents data, followed by successively narrower layers of information, knowledge, understanding, and wisdom.  The drawing makes perfect visual sense: There’s obviously plenty of data in the world, but not a lot of wisdom.  Starting from mere ones and zeroes, up through what they stand for, what they mean, what sense they make, and what insight they provide, each layer acquires value from the ones below it.”  Furthermore, I would argue that metadata provides the footholds allowing us to scale from one layer of the pyramid to the next.

Metadata is our guide on the journey from data to information, enabling us to understand the often complex business and technical contexts surrounding enterprise information management, and allowing us to journey further toward meaningful knowledge and actionable insight.

A lot of the fast-moving large volumes of various data swimming in the big data primordial soup are unstructured or semi-structured.  Without metadata, the amino acids of data won’t combine into the protein chains of information, the building blocks of meaningful knowledge and actionable insight.

Data has always needed metadata, but as you make the business case for big data in your organization, you’d better remember the bigger your data, the better your metadata needs to be.  In other words, bigger data needs better metadata.

We’re hearing a great deal these days about Big Data and related terms, one of which is Big Data analytics. There are many definitions of this term and here’s one as good as any:

Big Data analytics is the process of examining large amounts of data of a variety of types (Big Data) to uncover hidden patterns, unknown correlations, and other useful information. Such information can provide competitive advantages over rival organizations and result in business benefits, such as more effective marketing and increased revenue.

You’ll get no argument from me on the importance of defining key terms, be it Big Data, analytics, platforms, etc. Many blown IT projects or corporate initiatives can trace their failures to people not being on the same page from day one.

And this is why I’m a bit skeptical of the term Big Data analytics. Is the focus on Big Data? Analytics? Both?

Where’s the Focus?

I’d actually argue that it should be neither. That is, “BDA” is just a means towards the normal business end. To me, the entire point of capturing, storing, and analyzing any data (Big or Small) is to move the needle. Period. Or, if you like, consider the simple diagram below:

How many of us take the chain to the end? Or do things stop prematurely? I worry that the focus on either analytics or Big Data is misplaced. They are all merely means to the traditional business ends: increasing sales, decreasing expenses, etc.

I’ve written thousands of reports in my consulting career and, lamentably, far too many of my clients would want the report for the sake of wanting the report. I can recall several occasions in which I’ve stumped my clients by asking a simple question like, “What do you do with this information?”

Simon Says: Don’t Forget the Endgame

I have no doubt that the analytics available from unstructured data can augment our understanding of customers, users, employees, and just about everyone else. At the same time, though, data for the sake of data is meaningless. Consider two organizations, A and B. The former effectively utilizes Small Data and routinely makes decisions based on analysis, tested hypothesis, and fact. The latter doesn’t touch the vast troves of data at its disposal–both big and small.

All else equal, I’ll bet on Organization A any day of the week and twice on Sunday.

Feedback

What say you?

Few companies do data management better than Amazon–and I’m not just talking about their internal practices. Regardless of how well the company’s analytic systems generate über-accurate recommendations, it’s not perfect. Nor, for that matter, is it a substitute for human intuition.

To the Amazon’s credit, it recognizes the inherent limitations of relying exclusively upon its sophisticated algorithms and machines. Why not let customers refine, customize, and even remove their own recommendations, à la Netflix? In fact, Amazon does just that. Just look at the image below:

When perusing books, Amazon lets each customer override its own algorithm-generated recommendations. In this case, I can tell Amazon that I have no desire to read Guy Kawasaki’s book. (This was random. I have no bone to pick with Apple’s former chief evangelizer.)

It’s evident to me that machines can spawn remarkable recommendations. Collaborative filtering is nothing short of amazing–and I’m more than willing to consider Netflix gentle suggestions. But organizations adept at Big Data realize the inherent limitations of a computer- or data-only method to data management. In fact, there are typically legitimate reasons to ignore the results of even very accurate algorithms. Brass tacks: they’re not always right.

Even mighty Google–another Big Data stalwart–isn’t batting 1.000 vis-à-vis algorithm accuracy. Consider the recent Nature.com story on Google Flu that “drastically overestimated peak flu levels.”

The Limits of Democratized Data

No one is saying that all data should be democratic. I can’t imagine allowing employees to update their own pay rates or companies letting vendors tweak their own invoices. (In fact, ERP self-service tools have been with us for more than a decade, although many organizations refuse to use them for a cauldron of reasons).

Still, it’s hard to see the downside of Amazon’s move here. After all, don’t customers know what they like better than some machine? What’s the real harm in allowing them to remove items from their search or browsing history that they have no intention of buying? I’d argue that the benefits of this type of move far exceed their costs.

Simon Says

Organizations ought to learn from the examples set by Big Data leaders such as Amazon, Apple, Facebook, and Google. No CTO, CIO, or individual employee should be so enamored with his or her algorithm or technology that common sense is ignored. As a starting point, yes, emerging technologies and fancy algorithms can do amazing things and tap into heretofore unknown insights. By the same token, though, a user-override can often improve a good but imperfect result.

Feedback

What say you?

We’ve just released the 12th episode of our Open MIKE Podcast series!

Episode 12: “Information Development Using MIKE2.0” features content from our new book and key aspects of the following MIKE2.0 articles:

• Information Development Book: openmethodology.org/wiki/Information_Development_Book
• Information Development: openmethodology.org/wiki/Information_Development

Check it out:

Pre-order your copy of ”Information Development Using MIKE2.0” online today.

Want to get involved? Step up to the “MIKE”

We kindly invite any existing MIKE contributors to contact us if they’d like to contribute any audio or video segments for future episodes.

On Twitter? Contribute and follow the discussion via the #MIKEPodcast hashtag.

You can also find the videos and blog post summaries for every episode of the Open MIKE Podcast at: ocdqblog.com/MIKE

More than most people, I wonder about our relationship with data and technology. More and more of us are almost always connected, generating and consuming ever-increasing amounts of data. But do we use this data as much as we should? And when should we shut off intuition and start analyzing?

Those questions were on my mind as I read this recent Forbes’ article. From the piece:

Recently, the Corporate Executive Board developed a tool it calls Insight IQ and used it to assess the tendency of managers to rely on intuition versus analysis.  They found that 19% of over 5,000 managers in major global corporations are “Visceral decision makers” that rely almost exclusively on intuition.  (I suspect this figure is actually too low, based on other research and questions I have about the validity of the test.  But let’s not get hung up on this point; there is far more to be gleaned from this field of inquiry.)  Insight IQ proceeds to split roughly in half the remaining managers between “Unquestioning empiricists” who rely entirely on analysis and “Informed skeptics” (clearly the right answer to the test) who find some way to balance intuition and analysis.

Fellow MIKE2.0 blogger Jim Harris made a similar point. Most learned folks know (I’d hope, at least) that there are times to use both intuition and data. That is, you can’t get away with relying exclusively on your gut–nor is data the answer to all problems.

The Benefits of Intuition

Intuition is great as a starting point for solving big problems. For instance, in my years consulting on large-scale IT projects, often business issues would manifest themselves. Sure, sometimes the resolution involved merely the click of a box or the reversal of a journal entry. More often than not, though, thornier issues required a starting point–aka, a working theory or an initial hypothesis.

Equipped with that working theory, I would take the logical next step: to test it. At this point, data and analysis became invaluable. How else would you prove (or disprove) your hypothesis? Test environments were especially useful since enterprise systems rife with business rules (and software bugs) meant that isolating cause and effect could take time.

Without intuition, many problems would not have been solved. I can think of one woman with whom I worked who lacked knowledge of the system we were implementing. When a problem arose, she didn’t know where to start. Her intentions were benign, but she just couldn’t help. (It didn’t help that she lacked data analysis skills to boot.)

The Limits of Intuition

A few years later, I worked on an extremely complex data issue for a large healthcare organization. (The details of this project aren’t terribly relevant here. Suffice it to say that intuition could only get me so far.) Very quickly, I realized that the problem would require 90 percent analysis, but that ten percent for intuition served me well. Some very expensive bigwig consultants on the project had neither the data nor the intuitive skills to assist the client, and they wound up doing more harm than good.

Simon Says: It’s Not a Binary

Data–especially the big kind–can complement our understanding of events and trends, especially when used right. However, make no mistake: data does in no way supplant the need for intuition.

Maybe in 20 years a Terminator-like device will make us obsolete, but were a long way from that. In the meantime, ensure that your employees understand the benefits of Big Data, the limitations of intuition, and when to use each.

Feedback

What say you?

In this blog I often talk of the value of information.  Information is a valuable asset and companies increasingly place great store in identifying new sources of data about their products and customers.

Individually, we are also quickly assembling a mass of personal information through our social networks.  Professionally, the most popular social network is LinkedIn.

When LinkedIn launched a decade ago we enthusiastically started to build our portfolio of contacts.  Each contact has a value to us in our career.  While that value is intangible it motivates us to maintain the contact as a relationship that enhances our network.

However the volume of our contacts is starting to become overwhelming.  In many cases the accepted invitations can be counted in the thousands.  Projecting forward a decade, it is easy to see that many of us could be facing a set of contacts in the tens of thousands.

While a list of several hundred people reflected a set of connections that was meaningful in a business context, the accepted invitations that are spilling over are starting to resemble a mailing list more than a premium set of relationships.

The social networks are trying to help by adding data to the mix, enabling us to find out who values our status updates, how our connections are progressing through their careers and who is interested in our profile.  However there is an argument that this is perhaps the information equivalent of “quantitative easing”.  That is, when we are concerned that the information economy is stalling we publish more information.

The inevitable consequence of quantitative easing is, of course, inflation.

The social network that finds the solution to the inevitable inflation, that is the need to add more information just to maintain a static real value, will have a huge edge over its competitors and perhaps find a new role in the information economy.

Maybe the professional network of the future will allow contacts to degrade and eventually disappear if they are not maintained.  It could be that we will augment our relationships with other data about our interactions and hence score their real relevance to us.  At the very least we will develop better ways to mutually identify the bonds that have the greatest potential value.

What is most exciting is that the most effective solutions to managing the overwhelming number of contacts that we are accumulating probably haven’t been invented yet.  Let’s hope the solutions appear before the networks lose their value through sheer volume.

In his excellent series on data profiling, Jim Harris details a number of basic steps that organizations can take to assess the completeness and validity of their data, a process called data profiling. Among the many lessons that Harris imparts is the need to assess key data according to the following attributes:

• NULL – count of the number of records with a NULL value
• Missing – count of the number of records with a missing value (i.e., non-NULL absence of data, e.g., character spaces)
• Actual – count of the number of records with an actual value (i.e., non-NULL and non-Missing)
• Completeness – percentage calculated as Actual divided by the total number of records
• Cardinality – count of the number of distinct actual values
• Uniqueness – percentage calculated as Cardinality divided by the total number of records
• Distinctness – percentage calculated as Cardinality divided by Actual

Admittedly, few companies do comprehensive diagnoses of key data elements prior to undertaking a massive system integration project.

A few years ago, I worked on a project for a company implementing a new ERP system (call it XYZ, Inc. here). To be sure, management was no exception to this rule. With regard to employee, customer, and vendor information, XYZ did not profile its data prior commencing the project. Instead, it opted instead for a reactive approach. The predictable result: its data quality suffered during and after the system activation.

Simon Says: Data Profiling Is No Luxury

Do not begin a major system endeavor with both a tight timeline and the assumption that all data can be cleansed during the project. Better yet, make data clean up a separate project before beginning the project in earnest.

It’s a myth to think of data profiling as a luxury. Think of it it’s an investment. The time and money spent profiling data will pay off in spades, forcing the organization to decide which data elements are essential and which are not. If there’s any part of a project that lends itself to milestone consulting, it’s data profiling. Consultants can identify data-oriented issues but should not be expected to resolve them.

Feedback

What say you?

A few years ago, I was looking at PR firms that could help me promote my third book, The New Small. I had worked with a small firm before a few years before on book number one and was less than pleased with the results. This time, I vowed, I was going to do a better job vetting prospective agencies.

In the course of my research, I stumbled across an interesting firm (that I’ll call ABC here) and I set up an appointment with its president and founder (let’s call her Alice).

Strike 1: Forgetting the Appointment

Normally, when someone wants my business, he or she initiates the first call. A few minutes after our call was supposed to start, my phone hadn’t rang yet. Puzzled, I dialed Alice up and spoke to her assistant, explaining that we were on for a 10 a.m. discussion. A few minutes later, I was transferred to Alice.

Here’s where things really go downhill. Alice clearly didn’t know who I was or what I had book written. It might as well had been a cold call on my end.

If Alice had apologized and politely asked to reschedule the call in a day or two, I would have gladly said yes. After all, things happen, right? I fancy myself a pretty forgiving soul–at least the first time that someone drops the ball.

Her next actions, though, sealed the deal: I would not consider working with Alice or her agency.

Alice asked for my website address while on the phone. Yes, she could have covered by googling it, but in retrospect maybe she hadn’t even written down my name. I gave her the URL and directed her to the page that had the most recent examples of my media placements. She clicked on an appearance of mine on a local New Jersey TV show (shamelessly linked).

Strike 2: Rushing to Judgment

While on the phone with talking with me, she “watched” (and I use that term very loosely) the first 15 seconds of that TV appearance. In her infinite wisdom, she quickly concluded that I needed media training. Coincidentally, her firm offered that service for people like me. What luck!

Now, I can take constructive criticism. No, really. What’s more, I may not be the most media-savvy person on the planet. However, I’m certainly so awful that someone–even a self-anointed PR expert–can size me up with half of her concentration focused on a video in under a minute. If I had slurred the first 20 words out of my mouth, refused to make eye contact with the host of the show, and wore a mustard-stained tie to the filming, maybe Alice would have been right. I told Alice that there was no way that she could make that determination so quickly (while on the phone with me).

Strike 3: No Data

At this point, I was pretty annoyed. But, if Alice knew what she was doing–and had the data to back it up–I would have still considered working with her firm. I asked for data on her clients’ success rate. I wanted to know things like the percentage of Alice’s clients appeared on national TV or public radio and how many bookings per client Alice was able to procure for her clients. She seemed puzzled that I would even ask.

That was the unpardonable sin. I would not be retaining her firm.

Simon Says

Mistakes happen. I get it. When you make one, though, own up to it and don’t compound matters. Admit that you screwed up. Don’t throw salt into the wound of your prospective client. And, most important, keep your data handy in case someone asks for it.

Feedback

What say you?