The end of waterfall?

I was reflecting on why we don’t all simply do things in a lean-agile way by default. Why are short iterations, fast feedback and a collaborative working style not the natural way of doing things? What is so attractive about organising the development process around a single pass, stage gated approach (typically referred to as “waterfall”) that we often need a whole change programme to address it?

H. L. Mencken quipped, “For every complex problem, there is a solution that is simple, neat, and wrong.” One root cause of waterfall’s historical popularity could be that software projects have been inappropriately associated with a predictable manufacturing paradigm (such as manufacturing TVs or cars). In this paradigm, things can be predictably specified and planned, in contrast to the the chaos and uncertainty associated with a new product development paradigm. Since predictable manufacturing is the wrong paradigm for software, practices and values rooted in it are not helpful. Two assumptions in particular is are a source of amusement or dismay to developers of software solutions:

  • There is a single set of requirements and delivery options that describe what the business wants to do (requirements) and how it will be delivered (analysis and design).
  • The specifications do not change, or only change minimally (and perhaps predictably) during the construction and test phases of the project, subject to a strong change management process.

The deep appreciation—that building software is complex, new product development with high change rates, and not predictable manufacturing—is at the heart of the motivation for lean-agile methods.

Another possible factor is that single-pass waterfall gives the illusion of an orderly, predictable, accountable, and measurable process, with simple document-driven milestones (such as “requirements complete”). There is a special irony in choosing a simple-to-track process that yields higher levels of risk since the high risk activities – testing, integration and most of all, finding out whether the solution is valuable, are pushed to the end.

Software development is a young field, so it is no surprise the simple formula of “requirements, design, implementation” have dominated during the first attempts to create a skilful development process. The single-pass waterfall has the simplicity of explanation and recall (“do the requirements, then design, and then implement”);

So is the IT industry abandoning waterfall as it matures? Last month Gartner (in a research note entitled “The End of the Waterfall as We Know It” – requires a Gartner subscription) confirmed that waterfall is still the dominant approach across the industry (52% of development projects). They go on to say…

“Quite simply, waterfall methods, when used in the traditional, project-based manner, are inconsistent and risky. Since there are other choices available that have the potential to be more consistent and less risky, it’s time to begin the transition to these methods.”

Seems to me that we are reaching the tipping point as the IT industry moves to adopting lean and agile approaches!

Background Reading

What are good metrics for IT development?


How should our development activities be measured? What should matter? Mary Poppendieck, inventor of the term “lean software development” has a view:

“The right measurements for software development are delivered business value, cycle time to deliver that value, and customer satisfaction*

I like this because its a small number of metrics and it promotes a focus on value, speed and customers.  Two of the 12 principles behind the Agile Manifesto are:

  • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  • Working software is the primary measure of progress.

So it seems like these metrics will promote these principles. There is no standard on how to measure these things practically, or who the customer is, or when to start the clock on cycle time, or how to quantify business value, or how to avoid nasty behavioural side-effects. One particularly difficult area is to decide when business value is delivered. Is this when a feature is actually been used or is when the feature is ready to launch in production but is held back because of a business decision  (know as earned business value in IT industry jargon)?

What else? All experts say it’s better to have a small number of metrics. Even so, there are things other lean-agile practitioners like to add on top of this. Perhaps these are of primary interest to the delivery/portfolio manager/delivery team but not beyond that:

  • Velocity –  This is equivalent to throughput – i.e. how much functionality is delivered per month. It’s hard to measure if one doesn’t have a stable way of estimating the size/complexity of requirements and it’s impossible to compare across delivery streams.
  • Defects/Errors – Both open and fixed.
  • Unit test coverage – For all common programming languages there are tools which can calculate what percentage of the lines of code are executed by the unit tests. Perhaps this does need more focus as coverage in many of the teams I have worked with is 0% (depending on the language/feature, min 80% coverage is normal) and there are very few teams who actually know what their coverage is.
  • Technical debt – How much has been added and how much has been removed per month. Most of the teams I have worked with don’t track technical debt so that makes this an interesting one to work on.
  • Work in progress – This can be visualised on the team kanban board. Work-in-progress and queuing times are leading indicator for cycle time (whereas cycle time is always a lagging indicator) which allows action earlier.
  • Lean-agile practice adoption maturity

Perhaps this gives some inspiration to adjusting key performance indicators next time round?

More reading:


Prioritise my stuff first!

hand-upSeeking inspiration in what to say to a project which desperately wants it’s stuff prioritised first? Try this…

Hi project!

Our delivery team is committed to managing the capacity of our development pipeline in a way that provides the best outcome for the company as a whole. We use  CD3(cost-of-delay divided by duration) as a framework for prioritisation since this maximises the company’s overall return-on-investment.

Using this approach, if the work you need us to do is not prioritised straight away, we will explore the following options together with you:

  1. Wait. The project shouldn’t start any related work until the bottleneck resource is finished with higher priority work.
  2. Redesign the solution to reduce the scope of the work required. Smaller pieces of work block the pipeline for shorter periods and increase the CD3 priority score (if the cost-of-delay remains the same).
  3. Redesign the solution such that the work is not is not needed. This can either be by pursuing a tactical solution which later will be replaced or by finding some way of not needing that functionality after all.
  4. Increase capacity at the bottleneck. This could be temporary or permanent. If we see it as a temporary bottleneck, the best would be that other project members go help with the bottleneck for a while if they have the skills (T-shaped individuals, please!)
  5. Abandon project. It may be that the project simply isn’t as valuable as other projects we could work on and we don’t see this changing any time soon.

We believe these options create value for the company. There are two further options which generally don’t add value for the company:

  1. Start more work in parallel. It’s finishing work that makes a difference, not starting it. Starting too much at once means that we lose focus, overhead/switching/coordination costs increase and some value is delivered later than would otherwise have happened.
  2. Complain to senior management to get the priority score of the work increased. This can be by increasing the cost-of-delay for “strategic reasons” or because we have “suddenly” found some extra $ benefits of the project. If this kind “tampering” really really can’t be avoided, we insist that it is the cost-of-delay which is manually adjusted such that the increased urgency of this work is communicated to all teams that are involved in producing the feature. This way we avoid a situation whereby one team considers a feature very urgent and another team considers it less urgent.

In the interest of the company as a whole, we want to steer you away from these last two options.  “Removing roadblocks” is often perceived as a core skill in our management culture but exercising either of these option is not normally removing a roadblock to value creation.

We look forward to exploring this further with you to get the best outcome for the company as a whole!

Best wishes

The delivery team


Increase personal productivity?

tomatoThe Pomodoro technique is used by a lot of lean-agile practitioners to improve their personal productivity and apparently it works. I’m gonna get going with it and I challenge readers of this blog to try it out. Anybody who lets me know that they have become a Certified Pomodoro Master, I will publicise it here on this blog!

The basics of the technique

  • Put all the activities you have to accomplish on the Activity Inventory Sheet.
  • At the beginning of each day select the tasks you need to complete and copy them on the To Do Sheet
  • Start working:
    • Choose the topmost task from the list
    • Set the Pomodoro to 25 minutes (the Pomodoro is the timer)
    • Work until the Pomodoro rings
    • Mark the task with an x on the To Do Sheet
    • Take a short break (3–5 minutes)
  • Keep on working, Pomodoro after Pomodoro, until the task at hand is finished, then cross it out on the To Do Sheet.
  • Every 4 Pomodoros take a longer break (15–30 m)


There’s a load more info. on why it works/goals and how to deal with stuff (managing interruptions etc.) on the Pomodoro website:

Why wouldn’t you give it a go?


Have you got the “lean-agile mindset”?

brainI spent some time this week evaluating the Value, Flow, Quality training course developed by  the agile consultancy Emergn. The course provoked some reflections on what adopting a lean-agile mindset means…

Intuitive Decision Making

The course asserts that way we make decisions is largely based on our intuition. Studies of how CEO’s make decisions shows that, although they often believe they are making rational fact-based decisions, it’s actually coming “from the gut.”  This kind of expert intuition is powerful stuff and can almost be magical – fire-fighters who run out of the house seconds before the floor collapses, chess masters who can glance at a board and declare that white can win in 3 moves, etc.

It was argued in the course that good intuitions take a long time to build up. Within IT development, the high variation both in the type of work, the length of time of most IT implementations and the short lifetime of most teams makes it hard to spot patterns. In a lot of cases, the feedback actually comes after the project is finished.

Lousy Rules of Thumb

Consequently, our intuition about IT development isn’t that great. So we fall back on more general rules of thumb. Which of these do you like?

  • Minimising cost is good
  • A standard process makes things easier and more efficient
  • If the customer needs something, then it must be done
  • Detailed plans allow us to monitor progress and stay on track
  • Ensuring clear handover points normally gives the best outcome

The course contended that these are fairly lousy rules of thumb for a product development context. It’s not that these are never true in a product development context. It’s more that applying them without knowing why rarely gives a good outcome – and often has the opposite effect to that intended.

So this is what a Mindset Change looks like

This made me think about what a lean-agile “mindset” really is. What we mean is replacing these general rules of thumb with some more powerful ones which are more relevant for product development. For us to accept this, we need to understand the “why” of these new rules of thumb (which is what the training course tries to address).

Let’s try some out. Which of these lean-agile oriented rules of thumb do you like?

  • The customer doesn’t know what he/she wants
  • Reducing batch size is good
  • Speed is normally more important than predictability
  • Maximising total value is good
  • A collaborative approach normally gives the best outcome

If you like these better than the first set, then you are already travelling the lean-agile path!

“Leaning” our customers – lean consumption

Piloting the first module of an upcoming Value Flow Quality course has provoked me to reflect on different ways of putting the external customer at the heart of what we do.

As part of the exercises for the course, I reviewed the dynamic priority list (DPL) of a team I have worked with (the list of requirements which have a priority score but work has not started yet) to see how much focus the external customer gets. Top marks to this team – all the items in the DPL on the day I checked had a value proposition with benefits to the company clearly identified! Some of the items in the DPL were error fixes. For most of these, the GCSS team had calculated the time that would be saved by company staff if the error was fixed and used this to work out the cost saving to the company in dollars.

So far so good – this is way,way better than many (or even most) development teams. Yet this way of evaluating errors really underestimates the value created by addressing the error (and hence affects prioritisation). By fixing an error we may also reduce the costs that our customers incur. It costs us money to answer the phone but it costs them money to ring us up. Reducing their costs in ring us up surely is worth something to us too?

This opens up the whole world of lean consumption i.e. lean applied to customers. The basic principles of lean consumption are (and I quote…):

  • Solve the customer’s problem completely, by insuring that everything works the first time. No customer wants to call a help line, so turn your help lines into kaizen opportunities to identify and eliminate the root cause of customer calls.
  • Don’t waste the consumer’s time. For example, challenge the need for queues of any sort. You will discover that queues always waste both the customer’s time and the provider’s money.
  • Provide exactly what the customer wants. The level of out-of-stocks of the right items and overstocks of the wrong items is remarkably high in almost every aspect of business. These consumer frustrations are almost completely avoidable with lean replenishment systems utilizing pull principles.
  • Provide value where the customer wants. Most providers secretly want the customer to come to them. For example, the best pricing is available in a Wal-Mart style big-box retail format that customers must drive miles to access. Yet most customers want just the opposite, with attractively priced goods conveniently available nearby. The application of lean principles can provide most value where it is wanted at lower cost.
  • Provide value when the customer wants. Most current-day sales and production systems encourage customers to place orders at the last moment with no warning. This makes level loading of production systems impossible. Yet most of us actually plan ahead, particularly for big-ticket items like computers, cars, and white goods. Some simple lean principles can turn strangers into partners who plan ahead with their providers, dramatically reducing costs for customers and providers.
  • Reduce the number of problems customers need to solve. Most of us would like to deal with only a few providers to solve our big problems – computing and communication, mobility, healthcare, financial management, shelter, personal logistics (better known as “shopping”.) Yet with the web we have been going in the opposite direction from industry. Firms following Toyota’s lead are asking a much smaller number of suppliers to solve much larger problems, even as consumers are asking ever larger numbers of strangers to solve tiny problems on a one-off basis, wasting time and creating frustration. Lean principles show a way to do much better.

Sounds great, doesn’t it? It would be interesting to start applying these principles. When you zoom out, the customer is operating a supply chain and he/she ultimately doesn’t mind where a cost reduction happens – either in the part of the process which he/she operates or the part he/she has outsourced to the company in question.

So back to my example with the error fixing. If making a change (in this case an error fix) reduces our operating costs by $1 per year then its clear that what it is worth to the company is simply $1 per year since this goes direct to our bottom line. If instead, making a change reduces the customer’s operating costs by $1 per year then what is this worth to the company?

A tricky question! Clearly for large changes which affect our customer’s operating cost, we can afford do market research into this to come up with a number. Maybe the change will increase our market share, allow us to charge higher prices or increase loyalty (or at least prevent a decline in any/all of these). We can talk to customers and make some assumptions and get a number. But for tiny error fixes? I don’t think so. We need a rule of thumb. What should it be? Of the $1 value that is created by reducing the customer’s operating cost, it’s unlikely that the company will capture it all (i.e. $1). It’s also unlikely that the company will capture none of the value ($0). So we need a number between $0 and $1.

If we were to standardise on an approach like this, we would be able to effectively evaluate and prioritise small initiatives that improve our customer’s operations. We would also be more careful about pushing cost from our process into the customer’s process. We would not just have the four benefit types (increase/protect revenue, decrease/avoid costs) but we would have an additional one “improve customer’s business.” Something to think about?

Read more on lean consumption