From MIKE2.0 Methodology

Abstract
The aim of this article is to present and analyse the main agile practices that, according to the author and perhaps the wider agile community, can be successfully used to transform a business organisation into a more agile one and hence make it more competitive. This transformation is referred to as the Agile Business Transformation (ABT).

The article also introduces the main benefits an organisation will achieve if it switches to an agile software development methodology from a waterfall type software development methodology while also contrasts the main manifestations of these methodologies in organisations.

Finally, the article describes the people required to drive the ABT and the changement management aspects of the transformation, ending with a summary overview of the ABT and the main message about agile which is that it's an adaptive methodology and not a prescriptive one hence organisations that undergo an ABT should further continue to adapt to the most efficient way of working.

A request has been made to Peer Review this article in order to receive a broader perspective on how it may be improved. Please make any edits you see fit to improve the quality of this article.

Introduction

Today most organisations that even want to stay and operate with minimal profits in a market need to constantly change, evolve and improve in response to external market changes. To be a competitive market organisation or a market leader, an organisation needs to be in front of the market changes and with its own transforming changes become the cause of the market changes and set the rules of the game. Therefore, competitive market organisations need to innovate and change strategic directions quickly, with IT is at the core of most strategic operationalisations (implementations). The importance of constant organisational change and the importance of its supporting IT change is clear.

The question then is, how will an organisation's business software support the strategy if the organisation follows a rigid waterfall type software process that has a delivery cycle of a year or typically more? In that time the business environment would have changed again and not be even recognisable (e.g. consider the effect of the current “credit crunch”) and then all the effort on the project started will have been wasted.

If there is agility at the strategic level in the organisation, then the software development function needs to find a way to respond to this change and deliver benefits sooner rather than later. Developing software is certainly not like constructing buildings which need long analysis phases. The way to deliver benefits sooners is by using the agile software development methodology and undergoing an ABT.

There are a few agile software development methodologies such as Extreme Programming and Scrum. All these share the values of the Agile Manifesto and have many common practies. The key with all of them is that agile is not a prescriptive but an adaptive methodology that involves many feedback mechanisms that allow for continuous adaptable, process improvement and cost reduction.

This article will effectivelly describe the ABT, which involves the description and analysis of the main transforming agile practices, the people required to lead ABT and change management. These agile practices of the ABT are common in most agile software development methodologies and in author’s opinion, are the key to a successful ABT.

The structure of this article is as following: section 2 analyses the benefits of agile while contrasting the waterfall type software developments to the agile way; section 3 analyses the main transforming practices; section 4 looks at the people required for the ABT; section 5 considers the change management aspects and finally in section 7 the article is summarised.

This article approaches agile from a process and business perspective. The analysis, quality assurance and development practices of agile teams such as pair-programming, test-driven development, continuous integration, automated testing, domain-driven design and others are in other articles on Mike2.0.

The Benefits of the Agile Business Transformation

The ABT is the process of transforming the business from having a waterfall type development approach to having an agile one. The author is not suggesting the waterfall type approach, used in many engineering disciplines like civil and electrical engineering is bad as such, the author however is of the opinion that the waterfall type approach is good but not for developing business software.

A waterfall development approach compromises of one-off, discrete and sequential project phases (analysis – development – testing – deployment). These phases usually don’t overlap only involve communication between teams that execute the different phases during next phase handover periods. Waterfall usually has a lengthy analysis phase which often paralyses the development process as so much detailed analysis was done that the development teams need a long time to comprehend the analysis artefacts. What’s also an issue is that in software development problems don’t come up in the design until implementation starts and a lot of analysis already done needs to be redone and many… many… documents need to be changed too! And maybe the business also came back after a few months when the analysis was finally complete with changes as their competitors have done something to which they have to respond… all these changes create sunk costs and overheads. Many organisations will not fit in this perhaps extreme waterfall description but many might not be far from it. 

The development phase of a waterfall project is similarly, like the analysis phase, very costly when things change and the software code is not agile.

On the other hand the agile development approach has an initial short analysis phase typically of 1-4 weeks (depending on complexity and size of the domain) and accepts that any longer periods of analysis are not very useful as things will change so the analysis done will become waste in the future. The initial analysis in the agile approach is done to cover the breath of the scope and detailed analysis is done later. The software is then delivered in short iterations which involve overlapping analysis – development – testing – deployment phases which are continuous and there is no handover between functional streams but they all work together till the end of the project. As soon as new changes and higher priority requirements come about (e.g. in response to competitors) they are analysed “just in time” and scheduled to be developed in the next or even current iteration. Of course such agility also needs a different approach to implementing code – that is left for other article on Mike2.0; but since not a lot of upfront analysis was done, there were very little sunk costs and the business is getting now higher value for money. The illustration in the diagram below visually shows the differences between the two approaches.

The top group of blocks on the diagram show how the different work streams (analysis – development – testing – deployment) work in the agile development approach and the long blocks below show how they work in the waterfall way. From the diagram above note that the 6 blocks of “A” equate to the “W-A” block and the same relation persists for all the other functional blocks. The diagram above also hypothetically shows that all work streams in the development process take the same time for the purpose of simplicity in illustration. Notice how long the business needs to wait in the waterfall approach to see any benefits. If each block from the diagram above is equated the same cost then the diagram shows that both will incur the same cost but this is not the case as will be analysed in this article. Cost in the agile approach is not only reduced due to lower cost of change (as mentioned) but also due to its focus on continuous improvement and incorporation of some lean principles from the Toyota Production System.

Further contrast between the methodologies is probably better explained elsewhere but what was summarised is enough to allow one to derive the following benefits for an organisation developing software in the agile way (and these are the benefits of the ABT) :

The Practices of the Agile Business Transformation

The six main agile practices that can transform the business to becoming “more welcoming to change” (lower cost of change), more lean and hence more competitive are: developing software iteratively, using user stories to capture business requirements, using story walls for team feedback on progress and mid-iteration constraints, having regular stand-ups and actionable team retrospectives to allow for continuous improvement from the productivity perspective, showcasing the achievements to the business on frequent basis and developing software features in priority order.

The practices together with the people of the ABT i.e. the agile champion are the two key aspects of success of the ABT. Professionals, who have worked with agile methodologies, have lead the agile functional streams during projects, can adapt the practices to different environments and thoroughly understand why agile works are to whom the author will refer to as the “agile champions” in this article. These people are the key to the ABT and the people change is discussed in the chapter to follow. The agile champions together with the practices will change mindset and then benefits of the transformation will be realized. The people in the ABT process are the essential ingredient to success and agile values “individuals and interactions over processes and tools” (http://agilemanifesto.org/).

Agile champions have hence accumulated the tacit knowledge that can’t easily be conveyed as it is tacit and implicit. Therefore the practices that will be explained are the explicit knowledge and full benefits might not reached without the agile champions at least initially in process of the ABT.

Iterations

Iterations are full development cycles (analysis-development-testing-deployment) of typically between a week and a month in duration. The agile practice, delivery in iterations, has the objective to deliver the software by producing many fully working versions of the final system but with only a subset of features at a time.

Usually the bigger and more capable the development team, the shorter the iterations, as the length of the iteration should be defined by the value it delivers to the business i.e. the iteration should deliver significant business benefits. For example, if there are only two developers developing code and in a week they can create layout changes on the data entry screen, then the iteration should be longer than a week as the business benefit delivered with layout changes might not be significant enough – what is of significant value is what the business says it is and it should also have a financial figure attached to it.

Here below is the analysis of the advantages and the disadvantages of developing software with iterations.

Main Benefits of Iterations

- Provide rapid feedback on true team progress, code development, testing, team dynamics and how well business requirements were captured and communicated - issues can be identified almost immediately and acted upon.

- Allow early realisation of business benefits as with every iteration completed software features can be released to live systems or to update software products so revenue can start flowing in or productivity benefits achieved quickly.

- Iterations support process, code and other innovation as with iterations one can try a new idea and the iteration will provide quick feedback on its value – hence the “fail quickly principle applies”. Innovations are the source of competitive advantages and many times of sustainable ones.

- Minimise sunk costs associated with unpredictability of business software requirements as there is a need for having stable requirements only for the duration of the current iteration and just a visionary roadmap for the future instead of the full requirements specification. As explained earlier, if requirements are changed the sunk costs are low. Other code change costs are also low because of the way agile teams develop code (outside the scope of this document).

- Allow identification of business needs as opposed to wants – since working software is delivered quickly, the end-users, after using it, can provide feedback on their real needs. To elaborate on this important point – the business who demand software features usually know what they want at the start of the project but since new software is being build, the business stakeholders and users might not be sure what they really need since it is hard to imagine what one needs, especially the details, until one has used or tried to use it.

Possible Drawbacks of Iterations

- Iterations can be a substantial overhead if the deployment cost are large, as with iterations deployment to live systems is very frequent.

- There is a need for continuous involvement of the software customer and the user as during a current iteration the business requirements or changes to requirements for the next iteration have to be detailed (just in time analysis).

User Stories

User stories are an agile practice used for capturing business requirements and serve as a reminder for conversations to happen allowing for just in time analysis. User stories are normally written on cards to facilitate their collaborative creation by the business stakeholders and the business analysts. What follows in this section is a brief description of what the user story’s contents, the process of how they are created and broken down from high level to low level user stories to allow for just in time analysis, and finally the advantages and disadvantages of using them to note business requirements.

The user story itself has two parts: the user story’s description and the acceptance criteria by which the business will accept this requirement as being met. The acceptance criteria can be seen as a “soft” contract of what should be developed. It’s “soft” because it can always be changed if the change gives the business more value. Guides to writing acceptance criteria, user stories and creating story trees (to manage the analysis structure) are described in other articles on Mike2.0. Acceptance criteria are no more than assertive statements in the form of Pre Condition – Action – Post Condition. Here below is an example of the user story description:As an Investment Banker<Who>,  I want to have a dashboard showing all my investment positions <What>,  So that I can see the changes in my positions at one place <Why>.This description should say “who” the requirement is for, “what” is this requirement and “why” is the requirement beneficial (i.e. what value does it create).

The simple description gives the development team and everyone a lot of information to think about and most importantly will prompt conversations and from constructive conversations creative solutions to problems will arise. “Who” tells everyone who is the main beneficiary of the feature and this is important as the team must always keep in mind the end user in order to deliver quality features. The key here is that the developers usually spend many hours developing the feature, and so many hours of thought while keeping the end user in mind means that they could come up with creative alternative and less costly ways if the know exactly for who get feature is and why it is needed.The “what” part of the user story is self explanatory; it is a statement of what the user or business stakeholder wants and it should be written in business (domain) language. It shouldn’t include implementation specifics; e.g. the want shouldn’t be “I want a popup to warn me of deletion” as maybe a popup might not be the best or cheapest way to satisfy the “why” which is a “warning of deletion” and the people who know the best how to satisfy the wants are the developers and user interface specialists.

Finally the “why” is perhaps the most important part of the user story as when the why is missing or attention is not put in specifying exactly why the requirement is needed then the development team will be confused and ask themselves the familiar question: “but why does the business need this?” When the development team is expecting a “why” but there is no justification (“why”) of the feature, this will prompt a conversation with the business and hence it might be discovered that feature might not be needed. Further, business analysts that notice the business not clearly able to convey why something is needed should question the business value of the feature. And finally, of course, what might have been a good set of requirements before two weeks when the user story was written

Further, the user stories are lightweight, easy to create and discard and hence enable just in time analysis. At a start of the project user stories are created to capture requirements at a high level and later when the software development is planned the user stories are used to capture lower (implementation) level business requirements. In fact, just before the development iteration begins the agile analyst team will take the high level user stories and analyse them to create the low level user stories, but until then the user stories capture a reminder for the conversation to happen – hence this is a just in time analysis process. Not doing just in time analysis creates a larger inventory of work items not being used to create revenue yet and further the inventory might loose all of its value if the requirements change and invalidate them (i.e. if the market changes).

The diagram below shows a hierarchy of business’s needs – starting from the project’s objectives, to high level user stories (breakdown of the objective) and the low level user stories which can be implemented. From the diagram, the stories in the yellow ellipse shows the initial project scope, the purple and the pink ellipses shows the stories developed in iterations 1 and 2, and the remaining blue boxes are stories to be developed. Notice there is very little upfront analysis - very few “tree level 3 stories”. Low level user stories or iteration level user stories are developed and analysis is done when needed – just in time for development. Notice the clear link between the objective and user stories that are being delivered. The stories that are analysed and hence developed first are the highest priority stories first – in agile the scope is not defined because requirements are always changing but the team is always working on the highest priority features.

Main Benefits of User Stories

- Put the business in the driving seat – the user story is focused around the business needs and the whole user story is written in business language so it’s easy for the business to enter the relevant conversations and make decisions.

- Act as a reminder for a conversation - and prompt conversations because of the format of the description (who-what-why) and also since the acceptance criteria are never captured to full completeness in order to allow creative input during the development.

- Enable just in time analysis – user stories are reminders for a conversation to happen, hence once written as high level user stories (breakdown of business objectives), they are not analysed further until needed but capture enough detail for a useful conversation. This avoids unnecessary upfront analysis of features that might never be developed if the requirements change.

Possible Drawbacks of User Stories

- Possible dependencies - It can be sometimes complex to partition business requirements into independent low level user stories and dependencies will be introduced. These dependencies can cause iteration planning difficulties and cause lower priority stories to have to be delivered first.

Story Walls

A story wall is a way of openly tracking and managing the progress of the user stories in an iteration. The story wall can either be drawn on a physical whiteboard and stories stuck and moved along it or it can be an electronic version when the team members are geographically distributed. Details of exactly what story wall columns, or story states, should be used on the story wall is up to the team to decide. The story wall diagram below is an illustration of a story wall showing the progress status of the team at a particular point in time in the iteration. The red rectangular blocks are user stories.

Stories that are planned to be developed in the iteration are in the first column in the diagram above and these are stories that are analysed to sufficient level of detail for implementation. Nothing is complete until the business is happy and says the user story is complete, therefore each user story requires a business “sign-off”, hence the column.

Apart from communicating to the whole team the story progress, the story wall can show the lack of resource capacities of sub-teams involved (testers-analysts-developers) and this can quickly illuminate both permanent resource constraints and the effects of temporary constraints (e.g. one person from a particular team ill for a day). Permanent resource constraint problems can be minimised by careful work project planning, however even then productivity of individuals will depend on their experience and many other factors. Hence due to factors that can’t be accurately accounted for in planning, resource constraints could persist but these will be obvious from the piling up of user stories on the story wall in particular columns reflecting some sub-team as a constraint. Further, the story wall, allows for visual manifestation of temporary constraints (e.g. how someone’s day off maybe from work affects the team’s overall workflow rate). From the story wall above one potential constraint could hypothetically be that the business is not available for sign-off of the stories completely developed and tested; hence the team should investigate this hypothetical constraint and act on it.

Main Benefits of Story Walls

- Allow for quick resource constraints identification - these constraints will be acted upon and waste eliminated, and as a new constraint becomes obvious then that one will be acted upon too and efficiency will constantly be improved (Goldratt & Cox, 2004). Resources can be moved around to help with the temporary constraints e.g. developers can help with some testing.
- Open and cheap progress tracking– when team members finish work on a story they just move it along to the next column and progress is almost tracked at no cost and it’s openly communicated.
- Show the iteration’s bigger picture – the bigger picture is conveyed through the story wall; since user stories are discrete the development teams working on the specific user story can loose the sight of the overall goal or the “theme” of the iteration and it is important for everyone to see the bigger picture and the goal of the collective work during an iteration.

Possible Drawbacks of Story Walls

There are no possible drawbacks of using the story walls known to the author.

Retrospectives and Stand-ups

In some projects managers impose a software development process but in author’s opinion, an opinion shared by many with whom he has worked before, is that the process should mainly be “grown from bottom up” i.e. the team, lead by agile champions, should pick the process and practices that work best for the current project team and are also economically efficient. The process and hence the practices used shouldn’t and usually can’t be completely tuned before the project begins and once the project begins one should expect to have to adapt the practices or select completely new practices that work best given the team. This is the most important aspect of the agile software development ideology - agile is an adaptive methodology and not a prescriptive one.

Quantitative feedback (efficiency metrics) on project’s progress and process will be made clear by the iteration metrics that the project managers record, but the qualitative feedback on the process is fed back through the team retrospectives and stand-ups and hence this is why this qualitative feedback practice is so important and complementary. This open feedback is necessary to prompt adaptation of the process and practices.

Team retrospectives are informal but structured team meetings that should happen at the end of every iteration allowing enough time for the practices chosen at the beginning of the iteration to take effect. The whole team should be involved in the retrospective, including managers, analysts, quality assurance analysts and all the developers themselves. The business stakeholders are also encouraged to participate as the communication with the business in an agile team is taking place all the time hence the business stakeholder’s feedback is important.

During a retrospective the team should talk about and note what has went well, less well, what puzzles the team (e.g. some decision) and create an action list of how to improve going forward. Ideas on improvements are discussed openly with the whole team and can then later be checked for economic impact as well before introduced. If the environment is not safe then retrospectives are not very useful, the idea is that anyone can openly say what they think and contribute constructively.

Daily stand-ups complement the retrospectives. Whereas a retrospective is a weekly, biweekly or monthly activity (depending on the duration of the iteration), the daily stand-ups provide an opportunity for much quicker feedback, on daily basis. Daily stand-ups should involve the whole team standing by the story wall and each member should say what they have done yesterday, what they plan to do today and any difficulties they experienced. Development difficulties should be mentioned as other team members could be experts in that area and can help or do the task in entirety - e.g. a developer is having problems with javascript but another is an expert who is working on something different, they can switch if the switching costs are low or advice can be given. Also any other problems noted with the process or practice should be mentioned during stand-ups so that this feedback can be acted upon.

Main Benefits of Retrospectives and Stand-ups

- Provide qualitative feedback on the project – some of these issues can sometimes burn the project if not known earlier. E.g. the analysts realise that the business is not clear in what they really want and cannot decide on the priorities.
- Allow for continuous improvement - as feedback is collected and acted upon; this allows for continuous adaptations of practices used and the process for maximum efficiency.
- Support knowledge sharing between the team – especially during stand-ups where team members make the team aware of what they are currently working on.

Possible Drawbacks of Retrospectives and Stand-ups

- Need to be managed – as conversations during these two feedback practices shouldn’t go off topic and outputs must be captured and be acted upon.

Showcasing

Showcasing is the agile practice where the development teams show and demonstrate to the business what they have done in the last iteration. This demonstration is usually not targeted at the users of the system since they will in every iteration get to use the new features and are aware of the progress but showcasing is aimed at demonstrations to the higher level business stakeholders.

Showcasing creates a strong sense of ownership in the developers of their work and they will not be thinking they are developing for “some business users somewhere”. The development team will also feel a strong sense of accomplishment also by meeting the business needs and feeling the business’s gratitude for what was delivered and the value it created. Showcasing will also form important bonds that will enable more free flowing communication between the business departments or teams and the development teams, this lack of communication leads to poorly specified or understood business requirements and can cause projects to fail.

Showcasing will also create buy in from business line managers and this is extremely important in the ABT and also in any change projects. In classical waterfall projects the higher level business stakeholders wait for so long for the functionality to be delivered that all they know of during the project is the costs and see no benefits for a long time.

Main Benefits of Showcasing

- Creates business buy-in with higher level business stakeholders – as they get to see the regular benefits they are getting for their money.
- Increases the sense of ownership with developers – and this creates a stronger sense of responsibility which yields more dedication towards one’s work and drives out quality.

Possible Drawbacks of Showcasing

- Can be a significant overhead – as many senior executives will need to take time off to see the showcase and also many project team members will be taken away from project work, the costs of running a showcase can be significant. It should go ahead but maybe the senior executives can rotate on visits to the showcase.

Priority Driven Development

The priority driven development practice ensures that the project team is always working on the highest priority features identified by the business. The highest priority features normally should be the ones that deliver the highest business value. There should be no technical-only developments done unless they deliver higher business value than all other software features remaining.

Ensuring that the team is always developing the highest priority software features (or user stories) is essential so that the customer and the business gets the most out of their money spend and hence are satisfied with the deliverable. Since the software features are deployed to live systems iteratively, revenue could potentially be able to come in early and this will reduce the payback period and a capital expenditure project can become a self-sustainable one and be a revenue generator. Further, in case for some reason the project runs out of budget due to external market factors, the team will have delivered the highest value software with the budget they had.

Priority driven development therefore means that:

1) The business should decide on the priority of all the software features to be developed - therefore prioritisation sessions should be held for the business to decide on priorities of both high level user stories and later on the low level user stories as when higher level stories are broken down there could be low level stories that are not as important,

2) The team delivering the software is always working on the highest priority features - therefore the user stories that go into an iteration should be managed to be the highest priority stories available.

Regarding the first point, the business should make a decision based on the benefits of the feature will give them and the cost of development of that feature. Exceptions to this rule are when a software feature is a technical necessity or when some features must be delivered before to enable the delivery of other features. Finally, all technical features must be also prioritised by the businesses based on the cost/benefit trade-off.

Main Benefits of Priority Driven Development

- Aligns team and organisation’s goals – this prioritisation ensures the team’s focus is on what matters most and that the whole team is aligned to creating value for the organisation which is the end goal.
- Allows for quick wins – high business value (high benefits and low development costs) will be delivered first and hence quick wins achieved.
- Minimises risk of project's tangible outcomes – if the project is stopped for whatever reason the highest value features would have been delivered early and fully functional.

Possible Drawbacks of Priority Driven Development

- Sometimes it can be hard to quantify a business value of a software feature at a lower level of granularity - but the relative priorities of features can be used instead as a way of classifying the one of higher and lower value.

Relation to the other Agile Practices

The diagram below best explains how the main practices of the ABT are related to all the other practices used in the agile teams around the world. The practices at the inner circle are the core of the ABT but never the less, for a full transition to agile an organisation should adopt other agile practices used in the project and iteration management, business analysis, quality assurance and code development functions.

In author's opinion, for an organisation to have agility in their software development process and to be able to be flagged as "agile" it needs to at minimum be exercising the main practices of the ABT and described in this article. Other practices, that can be grouped according to project roles (management-business analysis-quality assurance-code development) are extremely useful and are (will be) covered in other articles on agile on Mike2.0.

The Agile Champions in the Agile Business Transformation

All business transformations need champions and the ABT needs agile champions who will lead ABT practices and the functional aspects (or roles) of the project i.e. they should lead the project management, analysis, development and the testing. Although some of the benefits of the ABT practices can be achieved without having agile champions, their full potential in author’s opinion will not be reached without them.

Apart from the main practices of the ABT there are numerous other beneficial practices that these agile champions can institutionalise and lead for a full transition of an organisation into working the agile way. For example the developers should lead pair-programming, continuous code integration, test driven development and automated testing. All the specific practices of these roles are outside the scope of this document and (will) feature in related articles on Mike2.0.

The diagram below shows the main project roles of an agile project team and the main activities all these roles normally perform. Notice the amount of overlapping activities between the roles, especially in validating requirements. One of the main reasons why software projects fail is because they don't meet the requirements or meet the wrong requirements; hence in the agile project team it’s the whole team’s responsibility to always question why they are developing a particular feature.

Change Management

Change management is an important part of the ABT and it must involve managing the stakeholders’ buy-in, communication about the change, the change roadmap/plan and agile champions to drive the change. It also goes without saying that ABT should be driven by a business case and progress should be tracked. The responsibility of change management is with all the whole agile team but ownership is with the project manager usually.

The practices of the ABT mentioned in this article involve a lot of tacit knowledge and this is one of the reasons why it’s hard to learn agile out of the book and why the agile champions are needed to lead by example in all agile team roles during the change.

The diagram below shows the mindset change that is likely to happen as the business is transformed to work in the agile way; this new mindset reflects the agile values from the agile manifesto.

Summary and Conclusion

ABT's aim is to increase the agility of software development and delivery teams while at the same time de-risking software projects, delivering benefits early and making the software development teams think like the business with one goal – creating business value.

The main ABT practices that, together with agile champions, will lead the transformation have been explained and analysed. ABT practices' fit to the overall agile practices landscape (all the practices related to different roles) has also been illustrated.

Finally, all agile practices are very useful, but the core practices, the practices of the ABT, are the main turning wheels of the transformation that introduces agility into the software development process and teams.

Direct and Indirect References

C, M. (2004), "User Stories Applied", The Addison Wesley Signature Series - A Kent Black

Early, A. (2008), "Introduction Change Management", BearingPoint presentation.

Fowler, M. (2005), "The New Methodology", http://martinfowler.com/articles/newMethodology.html

Goldratt, E.M. & Cox, J. (2004), "The Goal", Gower Publishing Ltd; 3rd edition

Poppendieck, M. & Poppendieck, T. (2003), "Lean Software Development", The Addison Wesley