It’s been a lucrative five years for consultants in information management with new work being as easy to win as saying the word “compliance”.  Executives are more than willing to sign-up new consulting engagements based the need to meet their compliance and regulatory requirements.  The trouble is, this type of information management engagement breeds a defensive rather than a confident enterprise.

A defensive organization believes that data needs to be locked-down, that risks need to be taken out and the analysis resulting from any dataset should be predictable.  Of course, any regular reader of this blog would know that we view data contained in large enterprises as complex and displaying all of the attributes of chaos mathematics which means any attempt to remove surprises from data is a fruitless endeavor.

A confident organization, on the other hand, recognizes that data is complex and chaotic but seeks to gain benefit from that complexity.  Rather than be afraid of randomness, they use the techniques of MIKE2.0 to identify the risks and then focus on monitoring and measuring.  In general, I observe a strong correlation between the confident enterprise and the adoption of Web 2.0 techniques and principles.  The confident organization believes that there is more value in collaboration and is willing to sponsor individual innovation.

A good example of why this is so important can be seen in social networking sites such as Facebook.  With the rapid growth in their use by a new generation of consumers, service providers ranging from telecommunication and financial services right through to government, need to come to grips with both the technology and the cultural drivers behind them.  Consumers are becoming more confident in sharing quite detailed information about themselves in a way that they expect others to pick-up.  Increasingly it will make no sense for providers to ask individuals to provide data about their relationships, locale or other details when those are already available in the public web.

In fact, one of the reasons why Facebook is so powerful is its ability to interface into custom applications.  Imagine the impact if you wanted to sell these consumers a new financial or telecommunications product and you made it possible to apply online from within Facebook!  More importantly, you can give the individual a sense of control by allowing them to privately share critical information with you and then maintain it in a form with which they are comfortable – perhaps for a multitude of providers.

Obviously there are challenges in this type of initiative, but good use of data measurement, reconciliation and parsing approaches allow it to be done.  The question is whether your enterprise has even considered whether it’s worth doing?  You can bet it won’t be long before your competitors do!

Michael zur Muehlen from the Stevens Institute of Technology was recently interviewed as part of the AT&T Techchannel roundtable on Web 2.0. During the interview he used BearingPoint’s Information Management initiative as an example of Web 2.0 in action and directed viewers to our open source site. You can view the program on YouTube (part 1: http://www.youtube.com/watch?v=WIm5txBm1YA, part 2: http://www.youtube.com/watch?v=5it8oBMbPrg).

Even if you have already used MIKE2.0 resources, it is always worth re-visiting as new materials are being constantly loaded. Some examples of new additions to MIKE2.0 include material supporting data mart consolidation, role and skill mapping, eDiscovery, understanding the information implications of hosted CRM, protecting your enterprise content and XBRL.

There is huge interest from clients in enterprise search, with the focus being how to create useful applications that go beyond documents or web pages.  Increasingly, we’re seeing organizations that have invested in metadata for regulatory compliance discovering the value of this asset using search technologies and techniques.

The original web experience was intended to be click-based navigating via a number of hubs to any point in the internet, but the last five years has seen the majority of users move to a language-based approach starting with a site like Google or Yahoo.  The example I often use is the rain radar, often when setting out to a meeting in a city I’ll check to see if rain is coming.  In Melbourne I can navigate from the www.bom.gov.au website to the radar but it’s faster for me to type “Melbourne weather radar” into Google, with the added benefit that I can use the same interface when I’m in Auckland, Singapore, New York etc..

At work, users are still in the late 90’s relying on incomplete intranets and a poorly maintained web of links.  The problem is primarily access to the structured repositories and even more importantly access to the structures of those repositories (ie., the metadata.

In many cases, banks have been the early adopters of metadata repositories followed by insurers and then the very large government departments.  The main driver for these repositories has been compliance and (for banks) risk (Basel II).  These repositories are enormously rich in content, but extremely difficult to interface to the rest of the organization’s information.  Search can be the solution and I recommend the following three steps:

1. Interface to metadata repositories
In a bank, a user should be able to search for “Risk Weighted Asset” and find not only the relevant documents but also a list of the systems and databases that contain relevant data as well as appropriate controls, processes and business rules.  It isn’t difficult to build interfaces between structured metadata and the search tools.

2. Interface to master data
The next step is to build an interface that allows the user to type “Assets Walmart 2005″ and find, via the metadata, appropriate queries which can then be launched in a BI tool (eg., Business Objects or Cognos).  This is part of my view that search should be the kick-off point for all information analysis.  Again, this sounds difficult but really isn’t, you can use the metadata repository to define the dimensions of search and emulate hints (ie., “Did you mean xyz”) to help if the user is almost on target.

3. Better analysis of the quality of search
The search index increasingly becomes an asset in its own right.  Using the techniques in MIKE2.0, we can use do constant health checks on the usability and relevance of the search index itself.

It took about 25 years for the ARPA initiative to evolve into the web and about 10 years before the advent of techniques and technologies arose that make up web 2.0. Although it’s a bit early, we’ll probably start to see the momentum around web 3.0 before the end of the decade. Web 2.0 was a bottom-up movement and Enterprise 2.0 is about making use of these capabilities. We may see Enterprise 3.0 connect with Web 3.0 earlier and even help drive the need for new technologies. Besides the typical “vice drivers”, what would drive demand for Enterprise 3.0 from the business side?

• Fixing healthcare – an avalanche of costs is breaking current systems, whether in the countries like the US or in those applying more socialized models.
• Enabling virtual shoring – organizations will want to improve their use of offshoring, outsourcing and physically separated staff. “Virtual shoring” in virtual worlds can help provide the answer through better collaboration and greatly reduced travel costs.
• Bank of the Future – firms and individuals will continue to strive for new ways to raise capital, manage liquidity and hedge risk. Retail consumers will want a richer and interactive experience provided that simplifies their lives through use of technology, not a branch. Institutions will want a richer technology experience too – as well as the ability to bring in information from all types of sources in the decisioning process.

On the supply side, the fundamental change will be open source, globalization and technology exponentiation factors.

In areas such as health care, we’ll see these factors work together. A virtual treatment room may be a way to reduce cost by providing access to. Increasingly individuals don’t have pensions and they will look to have innovative product options that make long term health care affordable.2 technology areas look to stand out that will support these new models:

• User Interactivity – the current interface needs a major upgrade. Visualization will become much, much richer and collaboration easier.
• Information Development – information currency will continue to get more valuable. This will be driven by more systems and types content, greater abstraction levels and that increasingly decisions will be made in an automated fashion independent of human intervention.

So what might if look like? Think of an enhanced version of Second Life combined with everything Google is doing on steroids. Second Life for individuals to act in new virtual communities for commerce, healthcare and education; Google to link it back to the real world. If this sounds far off, its not. Look at what organizations such as IBM, Dell and ABN Amro are doing in the virtual world. Some of its very early (and some would argue PR oriented) but its happening.

In the wake of a recent adverse court finding for a major media company in Australia, I was interviewed by The Age on the business dangers of email.  I argued that communication within and between companies should be embraced rather than feared, but proper governance inevitably meant that one-on-one emails were not the best way to manage this unstructured content.

You can read the full article at http://www.theage.com.au/news/business/teamwork-avoids-dangers-of-oneonone-emails/2007/09/09/1189276544211.html or listen the podcast at the same site.

In Small Worlds Data Transformation Measures, Rob wrote about the challenges of data modelling in today’s complex, federated enterprise. This is explained through an overview on Graph Theory, which provides the foundation for the relational modelling techniques first developed by Ted Cod over 30 years ago.

Relational Theory has been one of the stalwarts of software engineering. It is governed by a Codd’s rules, which have fundamentally stayed intact despite the rapid advances in other areas of software engineering – a testament to their effectiveness and simplicity.

While evolutions have taken place over time and there have been some variations to approach (e.g. dimensional modelling), the changes have built on the relational theory foundation and abided by its design principles.

But is it time for a change? Are some of the issues we are seeing today the result of the foundation starting to crumble due to complexity? Or is it that there are so many violations of Codd’s Rules? While the latter is certainly a contributing factor, it may be that relational theory is starting to wear under the weight of our modern systems infrastructures – and the issues will continue to get worse. Whereas there does not appear to be an equivalent approach to relational theory that will address the issues we see today, we think Small Worlds Theory and Web 2.0 may provide some ideas for a new approach.

Small Worlds Theory helps provide rationale for a different approach to modelling information. Small Worlds Theory tells us that for a complex system to be manageable it must be designed as an efficient network and that many systems (biological, social or technological) follow this approach. Although the information across organizations is highly federated, it does not inter-relate through an efficient network. As opposed to building a single enterprise data model, it is the services model that includes the modelling of “data in motion” that should be incopoporated into the comprehensive approach.

In addition to better modelling of federated data, new techniques should also to bring in unstructured content. This includes the information from the “informal network” such as that developed in wikis and blogs. While there are standards to add structure to unstructured content, their uptake has been slow. People prefer a quick and easy approach to classification, especially for content that is more informal in nature.

Therefore, the approach may involve the use of categories and taxonomies to bring together collaborative forms of communications and link it to the formal network. Both Andi Rindler and Jeremy Thomas have discussed some work we are doing in their area on the MIKE2.0 project on their blog posts. We’re also starting to see the implementation of some very cool ideas for dynamically bringing together tagging concepts such as the Tagline Generator.

In summary, whereas an approach based on a mathematical foundation is a required to provide a solution equivalent to Codd’s and there is a grand vision for a “semantic web”, we may chip away at the problem through a variety of techniques. Just as Search is already providing a common for mechanism for data access, other techniques may help with information federation and unstructured content.