What is irreducible complexity?
A common critique levelled at the agile philosophy is that of irreducible complexity. The argument is very much like the argument of the same name often levelled at evolutionists. The argument typically given against evolution is this:
If evolution predicts gradual changes, then what use is half an eye?
This article isn’t about evolution, and I’ll leave the rebuttal of that to people like Richard Dawkins (if you’re interested then read the Blind Watchmaker - it’s that book that made me start writing this post). The following is a paraphrase of a concern that I’ve heard a few times:
Using the principle of gradual improvement, what if I ask for a table? I can split the story up into four separate “leg stories” and a “table top story”, but I can’t release a table with two legs.
Oddly, the concern often uses the same metaphor; so maybe there’s an article out there that I’ve missed!
A table with two legs
Certainly the above statement is true (as far as it goes), but let’s forget about tables (as we can’t create one using software - or at least not without a 3D printer). What about the following requirement:
As a sales order entry clerk I want a new sales order entry client So that my efficiency can be improved
Clearly we can’t have “half of a sales order entry system” any more than we can have half a table.
The first thing here is that we need some answers. Forget about the system that’s been asked for, for a minute, and consider why it is being asked. *
So that my efficiency can be improved
What are they using now, and why is it not efficient? Since there is no-one to answer this hypothetical question, I will do so myself (after going to speak with the imaginary clerk):
The sales order clerk is currently using an existing piece of software, written sometime in the eighties. It’s character based (1. Place sales order, 2. Find sales order, etc..).
I’ve just watched him (his name is Brian) create a hypothetical sales order. It took around 5 minutes, and most of the time was taken up looking up product codes using a sheet of A4 that has them printed on.
Do we really need the table
The first thing to ask here is what exactly is the minimum viable product (MVP). That is: what does the client actually need. Do they really need a new sales order system, or do they just need a quick look-up for the product?
Let me just clarify what I’m saying here: I am not saying that when a client comes and asks for a table that they are told they can have a stool with a plank of wood lashed to it, and they can like it or lump it. We are at a point now (and have been for some time) in IT where we can build anything, but it takes time; and, at this moment (2017) development time is likely more expensive than any other single part of the IT infrastructure. Furthermore, it should always be kept in mind that, during the software development process, requirements change.
But we really need the table
Whether it’s an internal or external client, they are well within their rights to ask for a complete product, and that can be quoted for and produced. The process under waterfall would be that a Business Requirement document would be produced, followed by a functional specification, which details how the stated business requirement would be solved, and finally, the software would be written to address the functional specification.
There’s a lot of room for misinterpretation here. The person writing the code can not only be a different person to the one talking to the customer, but can be three or four layers removed. Even if you ask the client to sign off documents at every stage of the process, you have to assume that they have read and understood what you have placed in front of them. There are essentially three things that can go wrong here: 1. The requirements are misunderstood, or wrongly stated, the stated functional specification doesn’t fulfil the business requirement, the code doesn’t correctly represent the functional specification, or one or more of these documents is misleading; in summary, the people producing the software get it wrong somewhere. 2. The client changes their requirement due to budget constraints, or due to changing business requirements, or external factors, or because they realise that they don’t need what has been requested anymore, or need something different, or extra; in summary: the client has changed the scope. 3. The required software is more, or less complex than initially anticipated, a new software development tool is introduced; in summary, the quoted price for the job becomes invalid.
These are not mutually exclusive: all three can happen at the same time. (The same is true for the table: the blueprints can be wrong, the carpenter can fail to follow them correctly, and the person asking for the table can realise that they need a bigger or smaller one).
What does this mean for the sales order system? What if we identified that, rather that replace the sales order system entirely, we could start by replacing the product lookup; maybe we’d raise a few user stories that asked for a simple application or web page with a look-up facility. We would identify all the parts of the system that need replacing, but we would order them, and we would arrange to show the client at regular intervals.
Again, let’s reduce that complexity further:
- Create a simple CSV text document that contains all the product codes and descriptions.
- Create a RESTful call that accepts a string and searches the product description in the database (in our case, the CSV).
- Create a console app that calls this.Now the clerk has more than he had before! It doesn’t look much better, but he isn’t trying to find a product on a sheet of paper anymore… so the next story might be:
- Create a webpage that allows the entry of a product description and returns a product codeThe point being that, at every stage, the product is getting slightly better. We can show this to the clerk, and after a few days development, he can identify any issues. We can have something that provides tangible benefit in days.
This takes more time, though!
Yes - yes it does. If you compare it to the waterfall process on paper, I think it probably does take more time, especially if you start getting embroiled in all the scrum ceremonies. But let’s consider what that means:
1. The business requirement document is right, first time.
2. The functional specification is right, first time.
3. The software is right, first time.Because the three points above will never** be true, they get iterated. I actually read an interesting article some time ago that argued that, for this reason, waterfall doesn’t even exist as a system. Unfortunately, I couldn’t find a link to it.
So, whilst on paper it might seem time hungry, what about the following scenario: you complete the product look-up page, and the client gets a big order; they need to get a web-site to enable customers to place orders directly themselves. The requirements have changed. Thinking a few months down the line, should the web-site work, the old sales order system may no longer be required at all. In the old way of a detailed spec, sign-off and months of development, they would effectively have to cancel the order and lose the time spent thus far. Most clients these days will appreciate that, by seeing the software in an iterative development cycle, they have opportunities to change the scope that they didn’t previously have.
It’s worth remembering what the agile manifesto actually says - because it’s not a big long document. It basically says that you should talk to people - your colleagues, your clients and their customers - whoever you can: show the software as often as you possibly can; talk to the client and check that they still have the same requirements; release as often as possible and check it’s really improving things.
Agile doesn’t always work, though
There are some times when this approach absolutely doesn’t work. For example, when NASA build a new spaceship; it’s unlikely that the people that program the guidance system will be using an iterative approach to development; whilst a guidance system that sends the ship in broadly the right direction may (or may not) be better than none at all, $100B worth of kit needs a complete product. It also has a long shelf life.
However, if you consider the principle of agile, rather than the specific implementations, such as scrum, you can take some of the advice (even into NASA***).
Summary
In summary, and to blatantly plagiarise Richard Dawkins, half an eye is useful, and so is half a sales order system, and so is half a table; the key thing is to ensure that it’s the correct half, and it fulfils some of the requirements.
Footnotes
*I realise that this requirement for such a large and complex system would not be sufficient, but it serves to illustrate a point.
**Okay, I can’t prove this, but I’ve never worked on a project where these have all been right first time. It is, of course, theoretically possible.
***I don’t, and never have, worked for NASA, so I actually have no idea what the processes are. I know nothing, whatsoever, about space travel, either - this is entirely based on guesswork and conjecture. For all I know, they launch the thing, and there’s a guy sat in Houston with a joystick, controlling it.
References
http://www.scrumguides.org/docs/scrumguide/v2016/2016-Scrum-Guide-US.pdf
http://www.first55tech.com/agile-requirements-blog/irreducible-complexity