re: engineering management
notes on software development, leading teams and changing the way we work

Is Agile or Lean Enough?

At the end of the year, I tend to reflect on the progress my team has made and what directions we should take in the new year. Actually, I think about this all the time. However, since the company shuts down I have extra time to be introspective. This year given the snowstorms in Seattle, I’ve had even more time.

Generally, I have been a proponent of agile techniques. They’re basically codifications of common sense best practices. For the past few years, I’ve been particularly attracted to Scrum for managing my development projects. While the basics are pretty easy, Scrum (like most agile methodologies) actually requires a lot of discipline. So, my team and I spent a lot of time increasing our discipline and paying attention to the issues that Scrum surfaced in our development process.

While that approach definitely delivered some benefits, I constantly felt like there were huge gaps in what agile in general and Scrum in particular were bringing to the table. A recent blog post by James Shore, The Decline and Fall of Agile, captured some of my concerns:

It says that the team should be cross-functional and recommends co-locating the team in a shared workspace. It says the team should deliver a valuable, shippable product at the end of every Sprint, and that the team should self-organize, discover impediments, and remove them.
Oh, and it also has a few mechanical things about a monthly Sprint and daily Scrum. Trivial stuff compared to the rest. But guess which part people adopt? That’s right—Sprints and Scrums. Rapid cycles, but none of the good stuff that makes rapid cycles sustainable.

Taking part in many agile discussions at work as well as practicing it on my project, I recognized this as one of the bigger problems. As much as my team and I may want to work in an agile fashion, there were other factors conspiring against us. Co-location was the biggest one and getting a fully cross-functional team in place is not always possible. Shore also states:

It’s human nature to only do the stuff that’s familiar and fun, and that’s what has happened with Agile. People look at agile methods as a chinese menu of practices, choose the few that look cool, and ditch the rest. Unfortunately, the parts they leave out are the parts that make Agile work. Scrum makes it worse by ignoring important (but hard) agile engineering practices, and the Scrum Alliance makes it worse still with their armies of trainers some good, some not issuing dubious “ScrumMaster” certificates to people who demonstrated competence in connecting butt to chair for two days.

I’ll be honest, our company offered Scrum training with Ken Schwaber and I took it (Hey, it was FREE!) And yes, I am now a Certified Scrum Master. And what enlightenment did I receive from this training. Not a whole heckuva lot. In fact, at the conclusion of this training I painfully recalled an enjoyable post from Steve Yegge, Good Agile, Bad Agile:

Up until maybe a year ago, I had a pretty one-dimensional view of so-called “Agile” programming, namely that it’s an idiotic fad-diet of a marketing scam making the rounds as yet another technological virus implanting itself in naive programmers who’ve never read “No Silver Bullet”, the kinds of programmers who buy extended warranties and self-help books and believe their bosses genuinely care about them as people, the kinds of programmers who attend conferences to make friends and who don’t know how to avoid eye contact with leaflet-waving fanatics in airports and who believe writing shit on index cards will suddenly make software development easier.
You know. Chumps. That’s the word I’m looking for. My bad-cholesterol view was that Agile Methodologies are for chumps.

And if you have taken this particular Scrum course and recall the “secret handshake” at the end, the chump title probably seems like a better name for the certification you received. Sure, there were a few useful bits in terms of clarifying a few ambiguous parts of the standard Scrum literature. However, the course offered no practical help with day-to-day development issues. Specifically, the disconnect from one or more development practices that pair up well with Scrum as Shore notes.

So, reflecting on the past year I see two fundamental issues with our adoption of agile practices. The first is the lack of top-down adoption of agile in our company. While there are many teams adopting agile (specifically Scrum), there is still the cross-functional gap. That leads to a lot of wasted time spent context switching between Scrum and the “traditional” development process. I’ve witnessed a lot of teams say that they’re doing Scrum but they’re just re-labeling portions of the legacy process with terms they learned from Scrum. More fragile than agile. It seems as if there is not an organizational drive towards continuous improvement that is driven from the top, then it is easy to regress to past behavioral patterns. In Scaling Lean and Agile Development, Craig Larman summarizes this nicely:

There are better ways to build large systems than with many developers in many places. Rather, build a small group of great developers and other talents that can work together in teams, pay them well, and keep them together in one place with product management or whoever acts as the voice of the customer.
But of course you are still going to do large, multisite, or offshore development. This is because your existing system is already structured that way, or because in the case of large groups there is the mindset that “big systems need lots of people.”

THAT’S what our current situation feels like. And that’s not something that can be addressed from the bottom up alone.

And my second issue? Exactly what Shore mentioned:

Scrum makes it worse by ignoring important (but hard) agile engineering practices

Scrum does not prescribe any development practice to use alongside it. That’s OK, but one would think that by now the Scrum practitioners would have strong recommendations based on real-world experience. The common wisdom is Extreme Programming (XP), but I have several issues with XP. It seems to be even more of a take-it-all-or-leave-it-all religion than Scrum is. And XP doesn’t work for all teams. And most agile processes over-focus on the development team, but gloss over the other cross-functional team members.

Now that the Scrum backlash is in full swing, people are starting to embrace Lean. What they fail to realize is that Scrum is just the gateway drug to Lean. Scrum is not a pure pull system, which is what you would have once you Got Lean. After reading Corey Ladas’ blog post and book some of the ideas resonated with me. But, this was only because I had years before been a fan of the Theory of Constraints and Eliyahu Goldratt’s books (sidebar: his books are great due to his ability to communicate dry, academic topics via engaging novels). And again, the pragmatic issues around cross-functional product development processes are still largely ignored.

So, in 2009, how will I evolve my team’s processes? We’ll probably continue with Scrum or even “evolve” towards Lean. But our focus will be on sharpening our engineering practices. We’ve been pretty good about constant communication and unit testing, but we can probably tighten up our feature designs and code reviews. It’s all about balance.