The Real Challenge For Agile Development Isn't In Development It's Everyone Else

I think that within the agile community we focus on trying to produce the most value with the least amount of resources and in the shortest amount of time. We don't care about anything else. We look at conventional software development practices as a wasteful and antiquated set of rules that were established when the biggest logjam in software development was waiting in line at the punch card reader.

As software people, delivering value is what is important to us. It is logical. We are continuously refining our processes and practices to find what delivers the most value with the lowest amount of wasted effort possible.

The agile movement in software development is a wonderful effort that is making fantastic strides to improve the process of delivering value through software.

I love smart people. I love to be around them. There's nothing that pleases me more than to feel that I am the dumbest guy in the room. The atmosphere and culture around the agile software movement effervesces me with wonderful positive emotions. I feel that it is a veritable School at Athens--not the Bulldogs.

Did I mention that I believe that unless major changes occur, agile development will fail to uproot waterfall as the methodology of choice for commercial software development? Is it because commercial software developers are stupid? Well, not all of them. Not even most of them. Stupid doesn't come into play.

The reason that I believe that agile software development as it is today will never gain wide scale acceptance in the industry is because the people who make the decisions, the deciders if you will.
Deciders are the people like the people in finance. Don't they really have the ultimate say on whether projects or practices go or don't. They are the ones who say whether the company will pay for something. I'd say that these are the people you want to sell.

What do people in finance have to do with software development. Everything and nothing. Everything because they pay for it. Nothing, because, unless they are participating in developing new financial software, they aren't going to recognize agile methodologies as viable. To them, XP and agility sounds like some crazy ponzi scheme.

XP, great practices, bad name. If you put Extreme Programming's name into context it makes sense. Extreme Programming is called extreme because it is a collection of the best practices in software development concentrated and cranked up to 11. It's an extreme set of excellent practices.

It does not help the cause that XP people tend to be zealots. Why shouldn't they be, it can work very well at delivering value with little waste. If I found something radically different from the norm and wanted to share it with the world, I'd be enthusiastic about it too.

Do you XP people know what it sounds like when you try to tell people in the other parts of the company about XP? You sound like an idiot from the start. eXtreme Programming, woo hoo! Yeow. Let's put a half pipe up in the break room and replace the coffee machine with a Mountain Dew and Red Bull fountain.

To them you look like these guys. Doing crazy stunts, bucking the system. Sticking it to the man. You guys look like a bunch of fly by the seat of your pants irresponsible risk taking idiots to the members of the more conservative areas of the business.

Let's get back to punch cards, sadly, I'm not talking about the promotional cards that I get at my favorite pizza place.

Software development back then was rigid and structured. It had to be. Computers were expensive and time on them was scarce. In those days people planned out their work upfront. They physically had to.

To take a concept and turn it into software, a person would need to turn that concept into a set of commands. Then enter those commands into a machine that would punch holes in a cardboard card. Those punch cards, oh I'd really like a Toto right now, took some time to fill out. Then they would need to be fed into a card reader that turned the instructions on the punch cards into instructions that the big fancy computer could read. In a few hours, you might get a result from the computer that would tell you if your program works or not.

By whether or not it works I mean, it will tell you that your program failed and send you back to figuring out what went wrong. You then would spend several iterations of analysis and fixing until the software finally worked. And then if a change order came in, a big chunk of that process would need to be repeated. That was expensive and the finance people knew it.

No CFO worth his salt would have allowed software to be developed in a way that allowed iterative processes to preside. It would be prohibitively expensive and our grandchildren wouldn't live to see it reach completion. The best way to make sure that development time didn't spiral out of control was to do all of the design and product definition up front. Writing plans out on paper is relatively cheap.

The other benefit of this process was that it was more predictable than not having a structured plan. It was relatively easy to turn the Big Up Front Designs into a task list with milestones. Oh, what could be more predictable? Is the project on schedule? Let me check the task list?

Finance people love predictability. Wall street rewards it. If a public corporation delivers within their quarterly estimates, investors buy more of their stock. If they fail to meet their numbers, even if they outperform them, the perception of the company by investors is that the corporation is volatile and not a good investment. If you deliver on your estimate, you are rewarded. If you fail to deliver on your estimate, even if you outperform it, you are punished.

To our friends in finance, the deciders, predictability is one of the most favored traits to a business unit. If they can count on an area of the business to do exactly as they said they would, the financial people will favor them, i.e., the predictable business unit's funding is approved.

Unpredictable business units are terrible to finance people. They have no idea what kind of return on investment they will get. Maybe it will be a home run. Maybe it will be a total waste of money. It's usually easy enough for the financial people to predict though, deny or cut off the funding and the outcome of the project is pretty well guaranteed.

This brings me back to agile software development. It's great that we're working on ways to deliver more value with less resources in less time. Nobody is going to argue that it is a bad thing. Agility's greatest weakness and its greatest strength is its instability. Agile methodologies are unstable by design in the same way that fighter jets are unstable. Cargo planes are stable, they lumber through the sky, fighter planes are so unstable that many of them cannot fly without fly-by-wire computer controls.

Agile methodologies are modeled against the real world. The real world has change. Lots of change. Change all the time. That's why agile software development makes sense to developers. It's a way for people to produce the most valuable software by adapting to changes. The great strength of controlled instability is, you can turn on a dime. If a priority were to take precedence over the current plan, the agile team will waste far less energy changing directions than the team that has their task list broken out.

The big challenge for IT professionals who want to adopt agile practices is to placate finance's need to perceive projects as being predictable. That is, they want to know that on a certain date, and not well before that date, the project that they are paying for will be finished. Better yet, they'd like to be able to check in on a project from time to time and ask how it's going.

Using agile methodologies, it isn't that difficult to deliver a product on a given date. The challenge is convincing the people with the money that it will happen. Edward Tufte says that "Details give you credibility." Providing lots of details upfront is difficult if the designers are postponing all irreversible decisions until the last responsible moment. I would submit that teams should defer providing details about the implementation itself and instead provide a detailed plan for the project schedule.

If the project is defined as having a minimum set of features to be considered functional or acceptable as quickly as possible, then the other details are just gravy. Hit the minimum features so they are functional and then prioritize them and improve on as many aspects of the project as the schedule will allow.

Instead of culling feature specifications out by name within a document, why not just name them as a feature set. Be very specific on what is the minimum acceptable set of features, but only be specific in what they will achieve. Go no further than specifying the minimum feature descriptions upfront. Beyond the minimum features, give the least amount of description possible. Leave those decisions until when the context of the application and stakeholders' feedback can shape those decisions.

The biggest key to getting other areas of an organization to adopt agile methodologies is to use language and descriptions that don't discredit the practices right away. Instead of letting the zealots describe agility, have people who understand agility, but also understand the people who ultimately decide whether the company is going to pay for agile projects.

That's really the big hurdle is communicating what agility is without making it seem like some sort of trick.

Getting buy in from the rest of an organization is going to be the greatest hurdle. Everything else is trivial in comparison. We can refine our techniques to make developing higher quality software more efficiently but it really doesn't matter unless agility can be sold as a superior option to the rest of an organization.

One of the most damaging things that can happen to a group that wants to be agile is for their company to half support it, one of those ok, we'll let you do iterations and we'll let you guys have stand up meetings, but we're going to require detailed documentation up front... What happens is the organization says it's letting the team be agile, but it isn't. You get a team that has some practices that came from agile methodologies, but the project itself is still very waterfall.

I believe this type of half adoption is very damaging to organizational adoption of agile methodologies. I believe it is, because that project is perceived as being 'agile' when in fact it will be a mix of many practices. It may work, but it probably won't work dramatically better than the old methodology, because in many ways it still is the old methodology.

If the agile project fails, as these types of 'agile' projects failed at a former employer of mine, the blame was really easy to place. People blamed those crazy agile ideas on the project's failure. The blamestorm was that company's retrospective practice. The damage was done though. The people who made decisions there only knew one thing about agile projects, they fail--it's not like the non-agile projects don't fail. If one wanted to get that company to be more receptive to adopting agile methodologies they could do themselves a favor by calling all of their projects the salmon filled waterfall project. Lay the blame on the old methodology and the decision makers will rush to the extreme opposite of waterfall.

The big problem with getting a company to embrace agility is getting them to embrace the whole ideal. Agile development is too often sold in pieces, but usually the pieces that aren't essentially agile, e.g., continuous integration, TDD, stand up meetings. Those things are good practices, but they don't essentially change the methodology.

The essential piece that is so hard for non-believers to adopt is, as Mary Poppendieck says, "deferring all [commitments] until the last responsible moment." For agility to succeed, I believe that an organization's culture must embrace this ideal of deferred commitment. Anything less is not agility and should not be sold as such.



That is the tough pill to swallow in my opinion. To me it has always made sense that decisions made today are often reversed tomorrow. I've never been comfortable making decisions before they need to be made. It's in my own personal wiring to want to defer decisions. In the same way that I am very uncomfortable making a decision today, there are people who are very uncomfortable not making a decision today. People want all of the details to be known upfront.

Getting those people to be comfortable with allowing decisions to be deferred is the only way to get an organization to adopt agility. If the organization's culture is not receptive to deferred commitment, I don't know what will make them comfortable. If I were a developer in a culture that does not accept deferred commitment who wants to work in an agile environment I think it would be easier to go somewhere else or start my own company.