I was reflecting on why we don’t all simply do things in a lean-agile way by default. Why are short iterations, fast feedback and a collaborative working style not the natural way of doing things? What is so attractive about organising the development process around a single pass, stage gated approach (typically referred to as “waterfall”) that we often need a whole change programme to address it?
H. L. Mencken quipped, “For every complex problem, there is a solution that is simple, neat, and wrong.” One root cause of waterfall’s historical popularity could be that software projects have been inappropriately associated with a predictable manufacturing paradigm (such as manufacturing TVs or cars). In this paradigm, things can be predictably specified and planned, in contrast to the the chaos and uncertainty associated with a new product development paradigm. Since predictable manufacturing is the wrong paradigm for software, practices and values rooted in it are not helpful. Two assumptions in particular is are a source of amusement or dismay to developers of software solutions:
- There is a single set of requirements and delivery options that describe what the business wants to do (requirements) and how it will be delivered (analysis and design).
- The specifications do not change, or only change minimally (and perhaps predictably) during the construction and test phases of the project, subject to a strong change management process.
The deep appreciation—that building software is complex, new product development with high change rates, and not predictable manufacturing—is at the heart of the motivation for lean-agile methods.
Another possible factor is that single-pass waterfall gives the illusion of an orderly, predictable, accountable, and measurable process, with simple document-driven milestones (such as “requirements complete”). There is a special irony in choosing a simple-to-track process that yields higher levels of risk since the high risk activities – testing, integration and most of all, finding out whether the solution is valuable, are pushed to the end.
Software development is a young field, so it is no surprise the simple formula of “requirements, design, implementation” have dominated during the first attempts to create a skilful development process. The single-pass waterfall has the simplicity of explanation and recall (“do the requirements, then design, and then implement”);
So is the IT industry abandoning waterfall as it matures? Last month Gartner (in a research note entitled “The End of the Waterfall as We Know It” – requires a Gartner subscription) confirmed that waterfall is still the dominant approach across the industry (52% of development projects). They go on to say…
“Quite simply, waterfall methods, when used in the traditional, project-based manner, are inconsistent and risky. Since there are other choices available that have the potential to be more consistent and less risky, it’s time to begin the transition to these methods.”
Seems to me that we are reaching the tipping point as the IT industry moves to adopting lean and agile approaches!