Agile Information Development Solution Offering
From MIKE2.0 Methodology
-> You are here: Agile Information Development Solution Offering
Caretakers of the information asset are very busy these days and under quite a bit of pressure to deliver a variety of platforms, working together, that deliver the right information at the right time to the right people and processes using the right medium. It can be a daunting responsibility.
The pace of change is accelerating as well and it is increasingly difficult to nail down requirements and plan precisely for tasks many months out, given changing corporate priorities and the often unknown unknowns in the projects.
Traditional approaches are incompatible with the modern information world. They entail too much cost, rework and risk and are too inefficient. They get no better with time and have caused innumerable disconnects regarding delivery dates for technology-related projects over the years. In reality, what is really in use today are approaches that are either well overly rigorous or there is a complete lack of a thought-out approach. Few have achieved a productive balance. Without agile approaches, that is.
An overly rigorous approach has an inordinate amount of project management, change management, time management, documentation, testing, budgeting, planning and standards. Where an approach is not defined, there is a lot of room for creativity and fast response. It’s flexible, reactive and often able to handle new technology well, but it can also be undisciplined and directionless.
Agile approaches are thought by many to lean too heavy towards creativity and openness. However, they provide just as much of that as is necessary to balance with the rigor of traditional approaches.
Solution Offering Purpose
The purpose of the MIKE2.0 Agile Information Development Solution Offering can be found in the now somewhat famous Agile Manifesto. It has applicability to information management projects. It calls for Individuals and Interactions over Process and Tools, Working Software over Comprehensive Documentation, Customer Collaboration over Contract Negotiation and Responding to Change over Following a Plan.
There are several principles behind the solution. Early and continuous delivery of valuable software is one of them. Delivering this working software frequently is another. Having business people and self-organized developer teams working together in the development of the software is a principle, as is building the projects around motivated individuals.
The solution keeps development as simple as it can be, while still being effective. Rather than being rigid up front about establishing a pattern for the long haul, the solution demands regular reflection and tuning.
Solution Offering Relationship Overview
The solution applies to all the other Solution Offerings, all of which deliver solutions in a specific domain. The solution is not about the “what” you are developing. It’s about the “how” you do it. Therefore, with a big data project, you would take the Mapping to the Overall Implementation Guide and deliver the phases using this solution, the agile approach.
Solution Offering Definition
Please see Guidelines_for_adopting_SCRUM_in_Information_Management_Projects for more information on the applicability of agile methodologies to information management.
This Solution Offering is expanding practitioners capabilities from traditional approaches to agile. As such, we present a minimum of agile terms that are minimally applicable to the approach. If you are already an agile practitioner, there may be some things that need to be unlearned.
Those terms are Stories (and themes and epics, related to the story), Impediments, Product Roadmap, Release Vision, Team Capacity, Product and Sprint Backlogs, Sprint Burndown and Sprint Retrospective.
Two- to four-week time boxed efforts are referred to as sprints which are comprised of Stories and roll up to Releases, which roll up to Products. Since all development is centered around stories, this Solution describes what makes a good user story.
A good user story is Independent, Negotiable, Valuable, Estimatable, Sized Appropriately and Testable. The story must be valuable to the business by providing value to the business sponsor. The story must be small enough, free of dependencies, and understood well enough such that it can reasonably be completed during the sprint. The story can be independently verified by the business sponsor or his or her proxy. The story is written in a language that can be understood by the business sponsor and the development team. Ideally they use the business language as well.
All stories are maintained in the Product Backlog and pulled into sprints in a Sprint Planning meeting. The Product Backlog is owned by the Product Owner but anyone can contribute to it. It captures product requirements (features to be developed or changes to be made).
This Solution recommends that stories follow the “As a ____, I want _____ so that _____.” For example, “As a…System Administrator I want to ensure all “code 2” projects’ databases are backed up once a week so that I can maintain a data center compliant with corporate policies”. Make sure stories are focused on “doing”. Stories that say “I want to have” probably are not focused on the value proposition.
Every two weeks or so on an agile project, the team will go through a variety of exercises. One is a retrospective on the sprint that is ending. We want to know what went right, what went wrong and challenge our guiding principles for the project. Another exercise is planning the next sprint. A project owner will have already done extensive prework for this, reviewing and adding to the backlog and putting the backlog in priority order.
One of the self-organizing aspects of agile is the story point (as a metric of value) estimation. Once established, teams can consider how many points have been delivered and consider new factors like vacations, new team members or ill-fit assignments on the way to consider for the new sprint, we would look to deliver more or less story points in the next sprint.
Another important goal of agile is to deliver to the business and not just to IT or infrastructure. If the sprint does not include shippable components, they need to be “potentially shippable.” You would need to decide what this means in context of your project, but it should be a high bar.
Once we have our goal story points in hand, it is time for the team and the sponsor representation to estimate over the prioritized stories until we reach the target. This is a meeting to discuss and assign the stories, check with team members on omissions, etc. and a second meeting or just the expectation that by end of day of the estimates to solidify the sprint.
Philosophically, the total story points are what the collective TEAM needs to deliver. The team will win or lose together based on this. If, during the sprint, team members need to pick up for one another – usually because of slight under- and over- estimation in the story points, that is expected. So, there are significant TEAM aspects to the estimation.
“Story points” is also an agile concept that may be unfamiliar to someone coming from waterfall approaches. Many use the Fibonacci sequence of numbers (1, 2, 3, 5, 8, 13, 21, etc.). This works because it’s easier to estimate the short stories so there’s more granularity possible there (1, 2, 3, 5 points). As the stories get harder and longer, they are naturally more difficult to estimate so they are estimated at only certain intervals. There should be an upper limit on story points assigned (i.e., 21, 34). Over that, stories should be broken up. The agile guideline for this in terms of hours is 16 hours estimated maximum per story.
Others use shirt sizes (S, M, L, XL, XXL), etc. or dog breeds (from Chihuahua to Terrier to Great Dane).
Judgment is essential for agile success. If you go into sprint planning looking for 29 story points and the points land you at 27 and the smallest story point count you have left is 3, should you add it or not? Judgment.
2 weeks is the best sprint cadence to work scope around. However, too high a focus on delivering a sprint could cause good methodology compromise. If you compromise data quality, metadata, architecture and other forms of essential value, you may win the sprint battle, but lose the business intelligence war. This Solution will talk about the meetings and milestones in the agile methodology in Mapping to the Information Guide, below.
Relationship to Solution Capabilities
The MIKE2.0 Agile Information Development Solution Offering goes across multiple Enterprise Views and uses a number of components from the SAFE Architecture. For any significant implementation, the great majority of activities will be required from the Overall Implementation Guide. The MIKE2.0 Agile Information Development Solution Offering is a framework for HOW those activities should be pursued by the development team.
Relationship to Enterprise Views
Utilizing the MIKE2.0 Agile Information Development Solution Offering means taking an approach focused on Information Development, but with an understanding of the significant differences from legacy approaches.
Mapping to the Overall Implementation Guide
In many cases, most of the Activities from the Overall Implementation Guide will be required for building out an environment. Users of MIKE2.0 should review each activity as a starting point to see if they are required based on the scope of the project requirements.
Shown below are the most important activities for building an environment with an agile approach.
At the Sprint Planning meeting, at the beginning of the sprint, the Product Owner presents the Product Backlog in priority order to the team and answers any questions about the order. The Product Owner defines the goal of the sprint. Many times, a sprint will be focused around a given, sizable deliverable. Other times, it will not be and the goal may be harder to define.
The team members and the Product Owner work through the Product Backlog, in priority order, accepting stories until the predetermined target story point limit is reached. These stories are firmly committed to for the sprint.
Immediately after the meeting, the team members will then define the tasks for each accepted Product Backlog story. These stories now form the Story Backlog. Hours are assigned to the tasks and the tasks will now be tracked throughout the sprint.
During the Sprint, the project team makes daily progress against the accepted tasks – the sprint backlog. There is no shortchanging of necessary rigor in the development cycle. The many development tasks in the sprint backlog will go through the steps of requirements, design, development, QA and production. Every story will not necessarily make it through all steps during a sprint, but as a rule, at least one should be getting to production during every sprint.
The idea behind getting to production is that it be “potentially shippable”. If you were working in a software house, this would mean the product can be shipped out. Going to production is a similar concept since the work has shipped to the users. Please note it may take a few initial sprints to get a work product to the production stage. However, once you get there, there should be no turning back. Every sprint thereafter must contain a potentially shippable, production story.
If QA is a separate organization, it often has its own sprints and manages its own tasks. The project team could keep an overarching story for QA in its sprint backlog for this.
Keep in mind that not all work done by a unit, such as a business intelligence team, belongs inside of sprints. Queue-based work such as production support or work that cannot wait up to 2 weeks to be prioritized into a sprint should be isolated to team members not assigned to the sprint. Team members assigned to the sprint should apply 5-6 hours per day to tasks, allowing 2-3 hours per day for overhead, the underestimated and the unexpected.
There are reciprocal agreements at work during a sprint. The sprint team commits to delivering the sprint backlog and the rest of the organization agrees to leave the priorities alone until after the sprint.
The most common sprint length is 2 weeks. It is not unusual to start at a longer length, like 4 weeks or a month, until the overhead of getting in and out of a sprint is smoother and abilities of the team are understood better, to where shorter sprints are predictable.
Items to consider in determining sprint length:
• How long the business can go without requesting a significant change
• Ability to reliably predict effort on tasks
• Length of the overall release
• Amount of uncertainty on the project
• The overhead of iterating
The daily meeting should be held at the same time every day. These are usually held in the morning to set the work schedule for the day, but they can be held any time of day.
These should be held in the same room, with dedicated, unerased whiteboards that allow for continuity. The meeting should last for no more than 15 minutes. After some opening comments on progress from the sprint master, the floor is given to every team member in a clockwise direction. Each team members answers the three basic question:
• What did you do?
• What will you do?
• Are there any blockers to progress?
Naturally, conversations can be taken up, but the sprint master must keep the formal part of the meeting to 15 minutes and push detailed task conversations to separate meetings.
The sprint master may or may not be an expert in the domain, say data warehousing. The sprint master’s rudimentary understanding of progress is hours logged against the expected hours of the story. So if a story was estimated to be 10 hours and 5 hours have been logged, it is considered 50% complete. The hours should be logged daily, preferably in the business hour before the meeting so that the meeting will have up to date status information.
At the end of the sprint, a sprint review meeting is held where the sprint team presents to relevant stakeholders what was accomplished during the sprint. This is referred to as a demo, although different forms of presentation may be appropriate depending on the nature of the story.
The hard part of a successful Sprint Review is working with the stakeholders ahead of time to craft successful story delivery. This makes the review a mere formality. Team members should be working with the Sprint Review stakeholders throughout the sprint to help ensure a successful review. Inevitably, some aspect of the story is left for the review. As a rule, no more than 2 hours should be dedicated to preparation for the review. This connotes the proper formality of the review. Also as a rule, no slides should be developed or presented. It’s about demonstration.
Naturally, some aspects of stories will have been missed or misfired in the development. This creates new stories which go into the prioritization process in Sprint Planning. These stories tend to be high priority because the stakeholders were already expecting it.
One sprint’s end is another sprint’s beginning. The three meetings that support the transition are the Sprint Review, the Retrospective and Sprint Planning.
The Sprint Retrospective is within the team – team members, Scrum Master and Product Owner. This is time to inspect and adapt the process – to take a look at what is and what is not working. It’s more “open forum” discussion around “what worked” and “what didn’t work”.
Many teams skip the Retrospective, but that is not recommended. Agile methods are meant to not only be dynamic when it comes to doing priority tasks, the approach itself should be dynamic, adjusting to the realities that the team is experiencing.
Agile Roles and Responsibilities
The Product Owner comes from the business. The Product Owner is either the sponsor or a surrogate of the sponsor and will contribute pretty heavily to the sprints. Ultimately, the Product Owner owns the scope and the vision for the deliveries of the sprint team. Ideally, the Product Owner has not only business experience but also technical domain experience from the business side.
The Product Owner’s responsibilities include:
• Establish the vision
• Manage the return on investment (ROI)
• Build the right system / product in the right sequence
• Call for Releases
• Write Stories (Product Backlog Items)
• Show up at critical events
I’m using SCRUM methodology nomenclature here, but it is a common role across the agile methodologies to have someone in charge of the methodology itself. The Scrum Master is a team-oriented leader and is ultimately responsible for the productivity of the team. The Scrum Master may or may not have domain expertise and may or is probably not dedicated to a single sprint (unless the Scrum Master is playing other roles as well.)
The Scrum Master’s roles include:
• Responsible for enforcing values and practices of process and team
• Remove impediments from Team
• Remove barriers between Team and others
• Educate outside groups about how Team is working
• Improve productivity in any way possible
• Guardian of the Scrum process
• Not a command-and-control manager
The team of 5-8 is a self-organized multi-capable team dedicated to forging and delivering the sprint backlog. Team members should be open to doing ANY work deemed necessary to accomplish sprint goals. This may include some work which is not in their primary job description, albeit at a slower pace than their core competency. Sometimes it is about timing and not competency. Team members are expected to “pick up” for each other during the sprint, again – to ensure success.
The sprint team will succeed or fail as a team. Well-written stories in the sprint backlog make success clear.
Characteristics of a sprint team include:
• Self organizing - Team decides how best to organize to meet the Sprint goals
• Self Managing - Every member of the Team is responsible for “managing” the Team
• Cross-functionally skilled - No ‘roles,’ but all necessary skills (e.g., QA, Programming, UI Design, etc.) necessary to go from Product Backlog to potentially shippable product increment
• Right sized - 5-8
• Empowered - Authority to do whatever is needed to meet commitment
• Committed - Committed to delivering Sprint features
• Focused - Members should be full-time
• Immutable - During a Sprint, Team structure does not change; membership can change between Sprints
To make it all work harmoniously, the team needs to be guided by a set of principles. Though these may sound like things we learned as school children, without some acknowledgement, these are some of the things that anchor down agile progress.
• Never use the word “you” because the other person may feel on the spot and defensive
• Never refer to history (e.g., “three months ago, you said…!”)
• Be on time for meetings; if you are late, apologize and pay a late “penalty”
• If everyone is talking at once, use a pen to determine who talks; whoever is holding the pen talks, everyone else listens
• Everyone’s opinion is important and needs to be understood and taken into account
• No name calling
These all factor into the Relationships of Agile to other Solution Offerings.
Relationships to other Solution Offerings
This Solution Offering relates to nearly every other Offering here since it relates to the methodology for delivering the Offerings. There are several differences in approach between an agile approach and a legacy, waterfall approach. Agile teams would be wise to acknowledge the differences and monitor them as being catalysts for issues that arise during agile rollouts, especially with team members new to agile.
Going to production will become a normal part of near-daily life, not the light at the end of the tunnel. Like a lot of things, once you begin to do something regularly, it becomes comfortable. Going to production will become like this.
Actually, for many shops, agile is a veneer for a lack of documentation so agile doesn’t necessarily mean less documentation. However, in the place of documentation are more meetings and verbal communications.
Secondary talents may be called on to deliver a sprint. For example, an ETL professional may need to work on the business intelligence side of things for a sprint or two.
It’s no more marathon where there are serious periods of coasting. It’s now all short sprints. The team needs to get used to delivering more – as well as correcting more - as a result. It’s the best way to arrive at the same destination, only faster.
Work group members are supposed to size their own work efforts in the sprint planning meeting. This is different from being given a deadline. However, honest assessments are vital to overall planning and if tasks are seriously under- or over-estimated, it’s likely to be viewed poorly by the team.
Quality Assurance is still usually a different environment and still requires code migration. However, as with going to production more frequently with less code, the same will be true of QA. End-to-end and regression testing will have to be in place and exercised at discretion, not with every change.
This is a deficiency in many agile teams, which is the lack of that long-term plan (i.e., “the analytic system will be in production in the fall”). While we advocate having a flexible long-term map of anticipated sprints, many agile teams continue sprint-to-sprint without one. At the same time, get used to much more frequent planning meetings.
The centralization of the leadership roles on an agile project gives way to a distribution of the responsibilities to people with roles of Project Owner and Scrum Master. The Scrum Master, in particular, may not be a business intelligence domain expert, yet plays a crucial role on the project by owning the development process.
The new forms and lines of communication in agile come with new terminology that must be used often and very specifically.
Most of the team is suddenly thrust into an “equal” position – a team member. Senior leaders are mixed with junior developers and performance is based moreso on delivery than seniority. Replanning of work efforts can be done very frequently and with the variety of work comes a commitment to teamwork.
Extending the Open Methodology through Solution Offerings
The Agile Information Development Solution Offering has numerous implications for the Open Methodology.
These include a greater focus on agile readiness in the Organisational QuickScan for Information Development, an agile perspective on The_5_Phases_of_MIKE2 , and strong support from the Data Integration Solution and the Data_Modelling_Solution which creates agility.
Wiki asset search