I just read a great post by Daniel Mathai of Infosys, Why Social CRM Can Wait.  He brings an interesting perspective into the decision to add social features to your CRM, which I think is valid for many other business intelligence projects as well.

Why are we adding more features to our applications?  Because the capability is there, or because we need it?

It is easy to make a decision to pursue additional intelligence based on the fact that it will give you a competitive advantage or the ability to learn more about your customer and prospect base.   However, more often than not, managers are quick to jump into the latest functionality without thinking of how it will impact the system as a whole.  From my experience, the system (whether it be CRM or ERP, custom built or software) needs to be audited properly to ensure that it is well managed and integrated before embarking on the latest functionality to enhance it.

It sounds simple: Make sure you can fit the new car in the garage before you bring it home.  But it happens, time and time again.  In your opinion, what are some of the reasons why companies fail to audit and prep their information systems before adding functionality that will significantly alter it?

It looks like 2010 will be the year of Analytics with advertising from most of the major players and tie-ups between IBM and SPSS as well as Accenture and SAS just to name a couple of high profile examples.  Read a little deeper though and the real business activity gets a whole lot more interesting.

It’s true that most of the focus has recently been on analytical techniques, with books such as Competing on Analytics (Davenport and Harris) popularising terms like “Analytics Competitor”.  Improvements in computing power have also made complex analytics practical on almost every dataset.  However, what is really driving the trend?  Davenport and Harris themselves provide some of the evidence in their book when they identify that one of the most important factors in the take-up of analytics is the presence of a leader (typically the CEO) who believes in the power of analytics and data.  They use in their book a quote, original from W. Edwards Deming): “In god we trust, all others bring data”.

What does this all mean?  Fundamentally the pinnacle of extracting value from data is interpreting it, to do this you need very clever analytics.  To do the analytics, you need great data that is not biased by any management interpretation – that sounds a lot like Enterprise Data Management.  To leverage the analytics, you need an organisation that is committed to using information, learning from the past and applying it to the future – that sounds a lot like Information Management.

2010 is going to be very interesting with the hope that we can bring data, information and analytics to the fore of business thinking.

While much of the discussion about information management centres on things that are new and exciting, it is easy to neglect some of the basic principles that the profession has learnt over the last decade.  Here are just five things that I think are among the most important to consider if your project is to be a success.

First, use a standard project plan.  MIKE2.0 has been available for some years now and provides a work breakdown structure which is comprehensive.  Such an approach allows you to involve contractors and multiple service providers without being locked into anyone’s proprietary method.

Second, use data models that have been published.  There are many of them around ranging from low cost publications by authors such as Len Silverston through to enterprise models provided by the major software vendors.  Even the most expensive model is typically much cheaper than the labour cost that it can save.

Third, borrow from Don Rumsfeld: “There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know.”  The data warehouse is trying to manage the complexity of the entire business.  You can’t possibly know everything and hence requirements analysis should focus on the fundamental principles of the organisation and those things that are hard to undo later.

Fourth, the foundation of tomorrow’s enterprise data warehouse is unlikely to be today’s tactical solution.  Avoid the temptation to make the first iteration self-funding, the organisation has to be prepared to make an investment otherwise there are always cheaper short term solutions.

Finally, ask yourself whether your organisation is really as unique as your stakeholders think it is.  One of the most common reasons given for the use of unusual architectures or data models that don’t borrow from published materials is that the business is unique.  Everyone is looking for a point of differentiation but that doesn’t mean that you shouldn’t adopt standards where possible.  It is unlikely that the use of an unusual data warehouse architecture is going to enable a store to sell more toothpaste.  That same store, might, however, gain a real edge by combining consumer and supplier data in a new and novel way building on existing approaches to modelling the data.

When Ralph Kimball wrote “The Data Warehouse Toolkit” (published 1996) it defined Dimensional Modelling in a way that immediately demanded attention by data warehouse practitioners worldwide. The book and the techniques it described were not new and were common the approach we had used for the better part of a decade, what the book did do that was foundational was to describe the approach in a consistent and considered with a terminology that could be used by everyone.

There are many similar challenges that data warehouse designers face on every project. For instance two challenges we are often called upon to decide how to handle changes to source system models and the proper handling of changes to reference and master data.

The former is usually handled by splitting logical entities when creating physical tables separating attributes and relationships that have a higher probability of changing. The latter is commonly handled in one of three ways. Method one sees non volatile and volatile attributes are split into two tables (with a one to many relationship) Method two has the current attribute values are held in one table with changes over time maintained in a second table (again one to many). Finally, method three has changes across a number of concepts tracked in an audit table which is only intended for forensic purposes.

On recent data warehouse projects, we are using a variant of method one that has been formalised as “The Data Vault”.  The Data Vault techniques put forward by Dan Linstedt formalises both of these issues and makes sensible design recommendations. In particular, it adopts an approach using “hub”, “link” and “satellite” tables.

Originally, Linstedt attempted to patent these concepts, but this application was rejected and he has now adopted a free approach and is promoting his concepts through books, training and his web site: http://www.danlinstedt.com/

Jason Kolb provides a great post on predictive analytics. Its powerful stuff, but technology is values-neutral and can cause issues just as it can help solve them. To see analytics applied for good, check out this presentation by Hans Rosling.

As with any complex system, its easy to get things wrong and you need to be really smart to really screw up. But investment banks won’t try and stop predicting the future and people won’t stop playing on the edge. The predictive analytics genie is definitely out of the bottle, the goal is make sure these predictions have some controls, and understanding of data lineage and regulations around risk. That’s one of the reasons why I think Information Development is so important.