According to the Standish Chaos report, 13% of Waterfall software development projects succeed. This compares with 42% of Agile projects. It’s no wonder so many people want to jump on the Agile bandwagon.
The Standish Group have been producing their Chaos report on the efficacy of projects since 1994. Sadly, a project failure rate of over 70% hasn’t budged much. And much of the seeming improvement is due to gaming the system rather than anything more substantive. For the purposes of this segment, I am going to take the numbers at face value. Anecdotally, these results correspond well to my own experiences. Among these experiences are companies claiming to do Agile but are really still doing Waterfall. Another trick is to use Wagile, so-called blended Agile. Finally, projects are deconstructed into such small fragments that they may have succeeded, but they were so small that project management was not responsible for the success. In fact, all it did was to add to the cost of the project and the success was a vapid vanity victory. Hooray for participation awards. You’ve made the grade.
But if Agile software development projects outperform Waterfall, how do I hop on that bandwagon?
Agile Transformation
Asking to transition from Waterfall to Agile is the wrong question. It lacks context. The real question is when to use Waterfall and when to use Agile. And then you need to ask which flavour.
Some projects are better suited for Waterfall and others for Agile. And, it’s not always about the project. Culture is a massive consideration. In fact, expecting a good outcome from Agile in the wrong environment is a recipe for failure, so I’ll start there.
There are several preconditions for Agile. Miss one, and you’ve already failed. They are not mentioned in the Agile Manifesto or even in the Twelve Principles. I’ll also discuss why this may have happened.
Trust
Agile requires an atmosphere of trust. Management and executive leadership needs to trust the process to facilitate functional outcome. This trust isn’t limited to the Agile team. Unless the project is too minuscule for anyone of merit to even notice, it needs to extend to the enterprise at large. There are ways around this, but even they require trust. More on this later.
Developer Competence
Your developers need to be experienced, top-notch developers. Diminishing marginal returns set in fast. Just because you have one high performing team—chemistry aside—, it doesn’t mean that the next team will be successful.
You can attempt to mitigate this competence by seeding the teams with less competent developers and perhaps pair programming, but this also diminishes the throughput. It’s been argued that throughput and velocity are not explicitly mentioned in the foundations of Agile—and the argument is valid—, but time-to-value is a meta aspect that at least implies velocity.
This developer competence also ties back to Trust. If you don’t have top performing developers on the team, trust would be diminished. And, it’s not enough to just have your best developers, they need to excel among developers categorically—not just be the tallest kid in your third grade organisation.
Domain Competence
Given a base of trust and competent developers, domain expertise comes into play. It’s not that competent developers in a trusting environment can’t rise to the occasion and learn the domain; it’s just that this is an impediment to the aforementioned velocity. When discussing domain competence, one needs to consider domains for the customers, users, enterprise, marketplace, as well as any infrastructure and dependencies. On balance, if among the team this domain knowledge is retained, they’ll be in a good place.
If you are employing Agile as a development framework, then there should be some unknowns—both known and unknown; this is a separate consideration.
Once you’ve got trust and competency, you are at least setting yourself up for—yet never guaranteeing—success.
Software Projects
So, what about the type of project? As I am not concerned with non-software projects, I don’t have a horse in that race, and I’ll leave it alone. Following the advice of Wittgenstein, ‘Whereof one cannot speak, thereof one must be silent‘.
Taking a short diversion, when people think of Agile, they are typically thinking of scrum. There are other scalable versions, which I consider to be AINO—Agile in name only—, and there is Kanban. To be fair, Kanban preceded Agile by half a century or so, so for the purposes of this segment, Agile should be considered to be synonymous with scrum.
In the realm of software, there are types of projects that should be categorically and summarily excluded from Agile. These are maintenance and bug-fixing; configuration projects; projects that would be completed in a couple of weeks or less; projects with known outcomes or part of a repeating process. Cross these off your list.
This leaves software projects expected to take several months or more and having an unknown yet hypothesised feature set. The reason for taking an Agile approach is that the hypotheses may well be wrong, so one wants to limit time, effort, and budget, reserving these for the next attempt. This is where the time-to-value and velocity come into play.
Project Management
Agile is a software development framework, not a project management framework. Agile operates organically in an unstructured, self-forming fashion. Project management is a discipline that explicitly imposes structure. To employ structure to an organic process is to miss the point. Organisations appear to require a sense of certainty. Agile promises no certainty. Applying structure is akin to killing the golden goose that lays the golden eggs. It’s expecting a captive animal to thrive.
I’m going to come right out and say it: Project management has no place in Agile. None. Nada. Zilch. Rien.
Agile and Project management are another anti-pattern. In fact, when I’ve argued previously that Blended Agile is an anti-pattern, the root cause can be traced back to the project manager; rather, to the notion of project management. Let’s unwind this.
There is nothing inherently wrong with project management, just as there is nothing wrong with nuclear fission technology. The problem lies in the context and application.
Given that 70% of project managed projects fail, it should be a no-brainer that this doesn’t work. In baseball, hitting the ball 30% of the time and failing 70% would put you in a good place; catching the ball at the same rate would be grounds to consider you to instead be a designated hitter.
And who are these project managers? Are project managers inherently masochists? Are they Sisyphus incarnate? Are they insane, presuming the next time they do the same thing, it will produce a different outcome? Are they delusional? Perhaps, they love to root for the underdogs. Or perhaps they can’t quite grasp how probability operates. Maybe they just need the money. Any port in a storm.
With such an abysmal track record, why would someone even consider staffing a software project with a project manager?
Here are some ideas:
Lack of Trust
We’ve already discussed that trust is a necessary precondition for Agile. If you don’t trust the team or have a general distrust that employees will do the right thing, project management sends a clear signal: We don’t trust that this team can produce useful output, so we are going to assign a taskmaster to keep them in check. This is not a sweeping statement about project managers’ ability to trust or to argue that there is never a need for project managers, but in an Agile context, it’s a kick in the teeth.
Weak Teams
What about compensating for weak teams? Maybe the organisation trusts people generally, but it faces a reality of immature dev resources. Again, let’s reflect on preconditions: developer competence. Yet again, you’ve entered the game with a weak hand. Better to fold (relative to Agile), than to play your losing hand. The project manager won’t be enough to compensate.
Illusion of Control
Humans have a cognitive need to feel they are in control. Executives and managers seem to suffer from this more than others. A project manager is a person serving in a role to manage projects. It’s what they do. It’s in the name. What could go wrong? Clearly, 70-odd percent of projects is what could go wrong. It’s not even a good illusion. This is closer to delusion.
Scapegoat
Perhaps project managers are sacrificial lambs. Ritual human sacrifices. If 70% of projects fail and projects with project managers fail at a higher rate, perhaps management just needs a scapegoat to deflect blame, to be the throat to choke. But framed correctly, it was management who decided to staff the project with the project manager in the first place knowing that they were operating with these odds. Whilst they might have convinced themselves that they would be the lottery winners holding the 30% ticket or that some golden boy would lock them in that victory. They had the Tom Brady of project managers, and he’d come through—if only he got the chance to play. Put me in, coach! I’ll take one for the team.
End Game
Bringing this home, we aren’t really any closer to understanding why someone would come to the conclusion that putting a project manager on an Agile team would yield a positive outcome. At the top of this segment I discuss trust and competency, yet the founders don’t mention these. Why not?
First, it may come down to the curse of knowledge. The people who formulated or codified the Agile approach were top performers frustrated at the state of software development at the time. They may have lost track of the fact that not everyone else had the depth and breadth of experience that they had. They left unsaid what might have only been implied.
Second, given their competence and competencies, they were trusted; they instilled trust. They were in that place and assumed that place, but that place is rare in the corporate world.
In closing, you don’t need some massive large scale Agile transformation. What you need is to understand the benefits and detriments of waterfall and Agile and create a framework to help you determine which to employ and when. To double down on the prerequisites, if you don’t have the ingredients of trust and competency in place, the best you can hope for is stone soup.