What is lean & agile development and why do we do it?

I learned the elements of lean operations, as it came out of Toyota, before I ever heard of “agile” as represented in the Manifesto and before scrum was popularized. As agile and scrum became popular I saw important elements of lean operations reflected, such as the one-piece flow around which scrum is based, the focus on technical excellence, in-person communication, and continuous improvement and reflection. However, I found agile and scrum missing some key elements that I found in lean product development: the role of the chief engineer, scaling via module development teams, light project management and integrating events (the scrum demo is a partial implementation of the broader events concept), and effective communication via tools such as A3. In essence, I found agile and scrum to be partial implementations for software of the more comprehensive lean framework. Above is the summary framework I created integrating all of these concepts into the model I explain and demonstrate in Tale of Two Systems.

So the answer to the first question, what is lean and agile software development, is summarized above. It’s a set of processes, tools, and people concepts and practices that are proven to work well in the field. Next question, is why? Why do these work well? Why do we use lean and agile techniques?

The answer is not very satisfying: we use these techniques because we have to. Here is a foundational insight into software development activities I learned from lean. There are two archetypal process types, and they call for vastly different process control regimes. These are represented in the Manufacturing vs Software Development table.

Manufacturing represents a predictable process. One can lay out all the steps, reliably cost them, and deal with change as an unusual event. Complex large-scale software projects are not so predictable - at the inception it is rarely possible to lay out all the steps reliably and change rates in many elements are high. The now too-discredited waterfall methods made the fundamental error of seeking to make a process calling for adaptive approaches and controls into a predictive manufacturing-like process.

It was highly attractive to lay out manufacturing-like processes for software. Standardize process so it can optimized. What could possibly be wrong with articulating technology-neutral requirements completely, designing the technology to meet them, building the solution, testing the solution against requirements, and happily implementing when bugs were mostly eliminated? The only problem was that it didn’t work for anything of material complexity and innovation. Accordingly, we do adaptive process control, which has become formalize as lean and agile development, because we have to. And only when we have to: if we can reliably standardize we must.

There is much more detail on these ideas and an engaging elaboration in two fictional projects, both complex and calling for lean and agile techniques but only one of them practicing them, in Tale of Two Systems. In People over Process I discuss at some length the leadership responsibility to fit approach to problem, and dive deeply into the facilitative leadership for agility approach that helps people work together in an efficient and rigorous way.