The problem: software design degrades over time
In my last few years as a software developer I have been running into the same problems over again. I've been fortunate enough to be able to be involved from the start in the architecture of a couple of systems, and spent a good deal of time planning each system up-front.
However, I have found that despite starting out well, the codebase of each project has gradually become messier and less maintainable, and as a consequence, each new task seems to take longer and longer.
One contributing factor is that we developers sometimes just have to write "quick and dirty" code, due to time constraints, or an urgent bug fix. We hate doing it, but sometimes it has to be done. We tell ourselves we'll come back and fix it later, but how often does that actually happen? Usually, "later" means "once we get this release out", which means by definition we will have to come back and change code that's now in production.
Unsurprisingly, with all the risks and testing overhead involved in changing production code, the temporary "quick and dirty" code ends up becoming a part of the system. This only has to happen a few times to create a maintainability nightmare. Developers tend to write in the existing design idiom of the codebase, and "hacky" code tends to obscure this idiom, resulting in inconsistencies and harder-to-understand code, serving to increase the temptation to write even more "hacky" code.
Another large part of the problem I think is caused by the requirements of the project changing, and new features being added that weren't anticipated. I used to think that this was a design problem: the architecture wasn't extensible enough, we didn't think far enough ahead in trying to predict what features might be needed. This is true; however I've since come to appreciate that no matter how hard you try or how deeply you think, you can't anticipate all the changes that might be requested for any project.
In the past I have tried to build extensibility points everywhere, but this has resulted in a lot of complexity that simply was never needed.
Rather than trying to plan and anticipate every future development, the design should instead be able to change throughout the lifetime of the project as the requirements change. This would then free up the initial design process to concentrate only on the bare minimum of functionality in order to get the job done - using the YAGNI principle. The design can then change to accommodate new functionality when it's actually needed.
But changing the design of production code is fraught with difficulties. Any change to the code must be tested, and testing manually is expensive, and still is not guaranteed to find all the potential bugs that may have been introduced by the change. Developers are therefore discouraged psychologically and economically from making design changes to existing code.
Unit tests create confidence to change the design
It seems that the way to remove this inertia towards making design changes is to cover the code with unit tests. All the existing behaviour is then captured by the tests, which will provide immediate feedback whenever a bug is introduced.
The problem is that writing unit tests is a big undertaking, and as code tends not to be written with testability in mind, this means that the tests end up being far more complex and much more effort to write than they should be.
What usually happens is that developers become discouraged by how hard the tests are to write, and end up "forgetting" to write them., or putting them off for a later date. Which of course then defeats the entire point of having unit tests at all, as if there are sections of the code that are not covered by tests, then the developers cannot be confident that the tests will pick up any bugs introduced by changes to the code.
Writing testable code
So, we have a new problem: how to write easy to test code? There are techniques we can use to make our code easier to test, such as decoupling, inversion of control, the Law of Demeter etc, and these will certainly help a great deal in removing the physical barriers towards unit testing. In a previous post I linked to a great series of talks by Miško Hevery on writing testable code, and there are many other resources out there.
Nevertheless, there still remains the psychological barrier: unit testing is not fun, at least not as much fun as writing code that "does stuff", and there is still the temptation there to leave writing the tests till later.
In addition, it is only when trying to write a test for a piece of code that we can truly see how easy or difficult it is to test, and we may end up having to rewrite code we have already written in order to make it amenable to testing.
Write the tests first!
Don't write a single line of code without first writing a test that covers the behaviour of the code. The tests then serve as a form of specification of the behaviour of the code, and ensure that testability of the code is guaranteed from the outset. Every line of code then will then end up being covered by tests, and the safety net is in place. The code can then be changed as much as you like without fear of breaking anything as the tests will immediately flag up any bugs that are introduced.
This is the thinking behind Test Driven Development (TDD). It's an iterative process, with each iteration divided into 3 steps:
- Write a failing test: write a test describing what the code should do, that it doesn't do already. Then run the test and watch it fail.
- Make the test pass: write the least possible amount of code that satisfies the test.
- Refactor to a clean design: make any changes to the design needed as a result of adding the new code. Any changes should be covered by existing tests. Run the tests as you refactor to check you haven't broken anything.
Obviously this method will take longer than simply just writing the code, but we are looking for a sustainable solution. Although it will take longer at the outset to write the same amount of code, the benefits will show in the long term. Changes to the code are easy and relatively risk-free, and as a consequence, the code can be redesigned and refactored as needed by changes to requirements. The software then becomes flexible and adaptable, but also reliable, well-designed and easy to maintain.
For me, TDD is the future of software development, and I think it's time for me to bite the bullet and embrace it. I for one have had enough of producing sub-standard software, and I intend to travel as far down the TDD path as I can. Maybe it will work, maybe the grass won't be as green as I thought in the other side, but there is only one way to find out.