MIKE2.0 uses “small world” measures to assess data models (see Small Worlds Data Transformation Measure). The challenge many users have found is that for large models, it is very difficult to calculate these metrics. Thomas Isaksson has created an open source tool and provided the code on SourceForge which automates the process , supporting any RDBMS for which you have a JDBC driver or ERWin (via CSV output). 

We hope that this initiative will further the encourage the adoption of these data model metrics and help demystify the traditional modelling process. Please take the time to use, test and extend this beta code.

Michael zur Muehlen from the Stevens Institute of Technology was recently interviewed as part of the AT&T Techchannel roundtable on Web 2.0. During the interview he used BearingPoint’s Information Management initiative as an example of Web 2.0 in action and directed viewers to our open source site. You can view the program on YouTube (part 1: http://www.youtube.com/watch?v=WIm5txBm1YA, part 2: http://www.youtube.com/watch?v=5it8oBMbPrg).

Even if you have already used MIKE2.0 resources, it is always worth re-visiting as new materials are being constantly loaded. Some examples of new additions to MIKE2.0 include material supporting data mart consolidation, role and skill mapping, eDiscovery, understanding the information implications of hosted CRM, protecting your enterprise content and XBRL.

There is huge interest from clients in enterprise search, with the focus being how to create useful applications that go beyond documents or web pages.  Increasingly, we’re seeing organizations that have invested in metadata for regulatory compliance discovering the value of this asset using search technologies and techniques.

The original web experience was intended to be click-based navigating via a number of hubs to any point in the internet, but the last five years has seen the majority of users move to a language-based approach starting with a site like Google or Yahoo.  The example I often use is the rain radar, often when setting out to a meeting in a city I’ll check to see if rain is coming.  In Melbourne I can navigate from the www.bom.gov.au website to the radar but it’s faster for me to type “Melbourne weather radar” into Google, with the added benefit that I can use the same interface when I’m in Auckland, Singapore, New York etc..

At work, users are still in the late 90’s relying on incomplete intranets and a poorly maintained web of links.  The problem is primarily access to the structured repositories and even more importantly access to the structures of those repositories (ie., the metadata.

In many cases, banks have been the early adopters of metadata repositories followed by insurers and then the very large government departments.  The main driver for these repositories has been compliance and (for banks) risk (Basel II).  These repositories are enormously rich in content, but extremely difficult to interface to the rest of the organization’s information.  Search can be the solution and I recommend the following three steps:

1. Interface to metadata repositories
In a bank, a user should be able to search for “Risk Weighted Asset” and find not only the relevant documents but also a list of the systems and databases that contain relevant data as well as appropriate controls, processes and business rules.  It isn’t difficult to build interfaces between structured metadata and the search tools.

2. Interface to master data
The next step is to build an interface that allows the user to type “Assets Walmart 2005″ and find, via the metadata, appropriate queries which can then be launched in a BI tool (eg., Business Objects or Cognos).  This is part of my view that search should be the kick-off point for all information analysis.  Again, this sounds difficult but really isn’t, you can use the metadata repository to define the dimensions of search and emulate hints (ie., “Did you mean xyz”) to help if the user is almost on target.

3. Better analysis of the quality of search
The search index increasingly becomes an asset in its own right.  Using the techniques in MIKE2.0, we can use do constant health checks on the usability and relevance of the search index itself.

The last 5 years have seen a rapid advancement in open source software development. The reason it works so well is that open source isn’t just about freely available code, it’s a different approach to developing software to can leverage a huge resource pool of available talent. In some cases, an open source product may be of higher quality than a commercially developed product.

Open Source Information Development
Open source content development is the other area that has exploded. Wikipedia is the best example of collaborative development of content where authors build upon the work of others to release a product that anyone can edit.

What about open source data? We’re starting to see it too. A great example is the OpenStreetMap project, where individuals are “mapping the world” by building a repository of geodata and point of interest overlay in a wiki. OpenStreetMap looks to form a credible competitor to the government geodata providers – with the costly ones in Europe providing a particularly attractive target. Some of the ideas about what it may achieve in the future are even more exciting.

Software Development vs. Information Development
When developing open source code or information there are many similarities. The biggest differences have to do with commit rights and release cycles. Whereas code is released in cycles, wikis tend to have content changing all the time. This provides maximum value in terms of encouraging contributions but results in instability.

The other issue is that of authorization to contribute. In a code model there are typically controls around commit rights and a test process to ensure the code developed matches the planned specification.

To get more open source Information Development, we think a hybrid model makes sense in some scenarios. This is what we’ve done with the open methodology framework in MIKE2.0 in an attempt to add stability and reliability. While there are certainly some downsides, we believe that in some cases it can provide the best model for development.

Where Else Might it Apply?
The best case for the approach seems to be related to the development of standards as a means for effective collaboration. I can imagine a few nightmare scenarios for open source information development related to personal privacy, but by looking at the geodata providers we may see other candidates - such as credit agencies - that will face competition through a variation of the open model.

We’re often asked to compare approaches to managing structured and unstructured data and attempts to bridge the gap between the two.  Traditionally, technology practitioners who worried about unstructured data have been entirely different group to those that worried about structured data.

In fact, there are three types of data, structured, unstructured and a hybrid (records-oriented) grouping of semi-structured.  They have much in common and are all part of the enterprise information landscape.  In order to look at ways to leverage the relative strengths of the different types of data, it is important to first understand how they are used.

There are three primary applications of data within most enterprises.

The first is in support of operational processes.  In the case of structured data, these processes are usually complex from a system perspective but often quite transactional from a human perspective.  In the case of semi-structured and unstructured data, there is often less system intervention or interpretation of the data with a heavy reliance on human interpretation.

Secondly, each of the three is used for analysis.  In the case of structured, it is easy to understand how the analysis is undertaken.  With semi-structured/record data, analysis can be divided into aggregation of the structured components and a manual analysis of the free-text.  With unstructured, analysis is usually restricted to searching for like terms and manually evaluating the documents.

Finally, all three types of data are used as a reference to back-up decisions and provide an audit trail for operational processes.

MIKE2.0 recommends approaches to governance, architecture and integration which are independent of the structure of the data itself.

The majority of effort associated with all data, regardless of its form, is gaining access to it at the time when it’s needed.  In all three cases, there are processes to lookup or search the data.  SQL for structured data, lookups for semi-structured and tree-oriented folders for unstructured.  Increasing, the techniques for finding all three types are converging in one set of processes called Enterprise Search.

Ironically, despite the power of search, successful implementations are really mandating the implementation of common metadata and the use of a single enterprise metadata model.  Again, MIKE2.0 takes the information architect through these requirements in a lot of detail.

In the future, organisations can expect to keep all three forms of data (structured, semi-structured records and unstructured documents) in the same repositories.  However, there is no need to wait for this future utopia to begin leveraging all three in the same applications and managing them in a common way.

Why would an executive care?  There are two main reasons why every business and technology executive should consider the quality of data modelling to be core to their success.

The first is that information is a valuable economic asset (as argued in MIKE2.0 in the article the Economic Value of Information).  Customer data, performance data, analytical information all combine to be an asset that is often worth multiples of billions of dollars.  If a company had billions of dollars worth of gold, I’d expect business executives to want to review and understand how such a valuable asset was housed!  Given that the data model is usually the main home for the information asset, the same should also be true.  The data model cannot be delegated to junior technical staff!

Increasingly there is another reason for elevating the data model.  Legacy information is becoming an obstacle to business transformation.  As the price of storage dropped during the 1990s, new systems began also storing ancillary data about the parties involved in each transaction and substantially more context for the event.  Context could, for example, include the whole sales relationship tracking leading up to a transaction, or the staff contract changes that led to a salary change.  With the context as part of the legal record, there are operational, regulatory and strategic reasons requiring that any new or transforming business function do nothing to corrupt the existing detail.  The data model is the only tool we have to map new business requirements to old data.

Given the complexity of data modelling, it’s not surprising that executives have shied away from speaking to technologists about the detail of individual models.  A discussion on normalization principles would be enough to put most decision makers off!

In the Small Worlds Data Transformation Measure article MIKE2.0 introduces a set of simple metrics to indicate whether on average the data models of an organization or doing a good job of managing the information asset.  Using the principle that information makes most sense in the context of the enterprise, it measures the level of connectivity and the degree of separation on average across a subset or all of the data models housing the information assets.

One of the concepts we introduce in MIKE2.0 is that of Networked Information Governance – essentially Information Governance combined with Enterprise 2.0. What we found when it comes to governance is the importance of the “informal network” in solving problems – the emails, hallway discussions and phone calls that place on a daily basis or in a crisis.

When it comes to bringing the information network together with a formal approach, technologies and techniques from Enterprise 2.0 are a great fit: search, tagging and aggregation are the keys to bridging the gap.

Can this approach be extended beyond Information Governance? We believe it can. Governance techniques can generally benefit from this approach – from a corporate board discussion to managing compliance with environmental regulations. Does it mean we remove the verbal conversations? Of course not (although it might have them recorded, tagged and indexed). Is security important? Definitely. The proposed solution isn’t trivial, but the Networked Information Governance Solution Offering helps define the steps required.

Various motives can drive a company to undergo a large-scale “transformation” of its systems – upgrade technology, improve business processes, change an operating model. Whatever the goal, transformation efforts typically focus on the process and infrastructure changes required to improve system functionality and expand organizational capabilities. Often overlooked, however, is another key element of effective transformation – data. Without complete, accurate and useful data, transformation efforts aren’t likely to reach their goals. A data-driven approach to transformation provides the foundation to support process and infrastructure changes and meet ongoing data requirements.

In the MIKE2.0 Methodology we provided some ideas around how organizations can take a data-driven approach to mitigate risk and provide greater information-based functionality. Some good areas to go to find out more include:

• The Solution Offering
• Guiding Principles of Data Driven IT Transformation

The Exec Summary provides a presentation that summarizes the approach. We”ve used this approach effectively and would love to hear any feedback on similar experiences or the approach.

I am continually struck by the lack of formal valuation models to information. Considerng how much organizations spend on building and maintaining information assets and how valuable they are to the health of the business, you would think it would be an area that would receive more focus.

While I’ve seen a number of academic papers on assessing the Economic Value of Information, the practically implemented cases are few and far between.  I have done some development on “Assessment-oriented” models that can be value in formulating a strategy, such as the Economic Value of Information model in MIKE2.0.

An Information Value Assessment should provide a mechanism to assign an economic value to the information assets an organization holds and  the resulting impacts of Information Governance practices on this value.   It could also measure whether the return outweighs the cost and the time required to attain this return.

Governance models are just one way of assessing value.  Other simple techniques could include:

• Mastering - how many systems hold this common data?

• Latency - if I load this data into a warehouse in an hourly fashion as opposed to weekly what are the gains?
• Quality - RI issues, accuracy issues
• Reach - how many people read my blog?  who are the readers?

I think this is an area where industry models will greatly improve, similar to what has occurred in the past 10 years in the infrastructure space.  The lack of model points to the immaturity of information management as a competency and the strict building of information with technology.   I would welcome any other opinions on technqiues.

Welcome to Information Development - a blog dedicated to the subject of Information Management. Complementary to the MIKE2.0 Methodology which is provides a structured competency for Information Development, this is a collection of perspectives on how the management of information has tremendous impacts on business, technology and society.