Agile Challenges and The Great Divide



Agile software development is a funny thing. Most of what I know about formal Agile methodologies comes from my friend Roland Cuellar. He wrote the chapter on Agile in my second book. While I’m relatively new to formal Agile processes, in fact I have been using them for the vast majority of my career. In this post, I’ll be discussing some of the limitations of Agile within the context of data management.

On my current assignment, my client is a large financial institution in the midst of some M&A activity. I am working closely with a director named Tommy to build an ETL tool. (This is not his real name; I just happen to be listening to Styx right now). Tommy is getting pulled in a bunch of different directions and, using more tact that I have, is trying to successfully navigate his organization’s political waters. As soon as we make the changes to file formats for his internal clients (read: very senior folks), he almost just as quickly comes back with changes. Some are minor; many are not.

The Limitations of Agile

The benefits of Agile–and there are many–are not unequivocal. While my little ETL tool is nowhere near as complex as many enterprise systems, building anything in an iterative process introduces risks such as rework and related time and expense overruns.

Now, I wouldn’t consider myself a hard core software developer. I can conceptualize as well as the next guy but sometimes I just have to see the aftereffects of a change. No one can think of everything conceptually. While versioning can partially control rework, I’d be lying if I claimed that all of my work is linear. At least my client understands this.

Implementing an ostensibly simple change request in a production environment can have one or more of the following ramifications:

• Break something else down the line.
• Corrupt or change data.
• Introduce inefficiencies–i.e., batch processes involve superfluous tables and views that, had the end result been known, could have easily been averted.
• Lead to excessive dependence on a resource with intimate knowledge of the data and development process. As a result, it might be difficult for an organization to rid itself of an expensive external resource.

What’s more, all of these changes may not be immediately evident. Dealing with tens of thousands of records (or more) means that it’s easy to miss a few. For example, because of some data manipulation, I had to derive a number of fields in a translate table. It would have been easy for me to forget that those fields needed to be updated in several related tables. Fortunately, I’m pretty diligent. Because of the complexity of my task, however, it would have been understandable for me to have missed a few of these interdependencies.

The Great Divide

Agile challenges are exacerbated by the IT-business disconnect, something which most experienced data management professionals have seen over their careers. “The line” wants what they want when they want it, irrespective of the development and data ramifications of their requests. Most business folks could care less about data models, referential integrity, and normalized tables. Changes often come from the top down and IT just needs to make it happen. This breeds frustration in IT folks, put in the unfortunate position of “just doing their job” if they successfully make the changes and “screwing things up” if they don’t.

Simon Says

Bottom line: Agile isn’t a panacea to The Great Divide between IT and the business. It may be better for many projects than a Waterfall methodology and, to be sure, not understanding fundamental business requirements will pose major challenges for any software development process. Ensure that IT is at the table during major development efforts, especially if business end users are unfamiliar with software, technical, and data issues.

Feedback

What say you?