I listen to the Maintainable podcast a lot. At the end of every episode, the host Robbie asks each guest if they are team rewrite or team refactor. That is to say, do they find themselves more often leaning towards rewriting old code or refactoring it. It's a question a lot of developers find themselves facing. Obviously, the answer is very situationally dependent, but most of us tend to have a bias one way or the other. I thought I would outline my thought process when facing that question.
Questions To Ask Yourself
If we start with curiosity, there are some questions we can ask to help us gain more information. These are litmus tests that can point us in one direction or the other.
Does it work?
The first question I usually ask is "Does it work?". If the customer says yes that doesn't tell us a lot. If the customer says no, that is more interesting. I follow that up with "Has it ever worked?"
If the software has never worked, then that immediately points me towards a rewrite. I've had several occasions where a customer came to me with a system that has never worked. They hired someone for a couple months and they were never able to get the system to work. Part of the reason to refactor is to preserve the value that is already present in the software. If the software never worked, then there is no value to preserve. At that point any discussion of refactoring stops.
If the software doesn't work now but did work at some point in the past, that is a different scenario. If we can get back to that working state, then I would probably roll back to that as my starting point. Then I can decide whether to rewrite or refactor based on that new baseline. This is often only possible if the original developer was using source code control and was good about tagging the release commit and documenting the build environment. That should be the standard nowadays, but should be isn't the same as is.
Does it have tests?
The next question to ask, after you ask "Does it work?", is "Can you prove it?". That usually comes in the form of "Do you have any tests?" Just because the customer tells you the software works, doesn't mean it actually works. There should be some sort of end-to-end test. At the very minimum, if you are testing widgets, you need a known good widget and at least one known bad widget that you can run through the tester both before and after you make any changes. Without this, refactoring and rewriting are both very difficult. At a bare minimum, you could always just run the 2 machines in parallel and compare the results. That's workable but less than ideal.
In addition to the end-to-end testing, I also ask about unit tests. If the code has well-maintained unit tests with good coverage, then refactoring becomes a more appealing option. If not, then you have to look at how easy it would be to add tests. If there are no tests and it seems difficult to add them or the code has poorly maintained tests, that pushes me more towards rewriting the code.
If we rewrite, can we freeze the existing version?
The last question to ask if you are considering a rewrite is "Can we freeze the existing version and for how long?" The problem most rewrites face is that no one will adopt the new version until it achieves feature parity with the old version. The new version starts with zero features. If we don't add any features to the old version, then feature parity may be achievable. If we start adding features to the old version it gets much harder to achieve feature parity. If we can't freeze the old version or if we can only freeze it for a very short period of time, that makes a rewrite very difficult. Are your users willing to go 6 months or a year or however long your rewrite is going to take (let's be conservative here) to get new features? If the answer is no, then refactoring is the way to go. It will let you improve the code base while at the same time letting you add new features.
Every Legacy Code Base Deserves a Good Refactoring Attempt
The last thing I would say is that before you rewrite, every code base deserves a good refactoring attempt. My only exception might be if the code never worked at all. In that case, I might just skip directly to a rewrite. Definitely timebox this refactoring attempt so you don't go too far down the rabbit hole.
One reason every legacy code base deserves a good refactoring attempt is that it just might work. You might find that it's not as hard to refactor as you thought. Even if the codebase is beyond saving and you determine that you need to do a rewrite, it's not a wasted effort. While you are refactoring, you are learning about the code and what it does. That has value and will make the effort of rewriting it a lot easier.
Obviously, I am slightly biased towards refactoring. That's changed over time. In the beginning, I was very poor at reading and understanding others' code and very poor at refactoring. In those days I was always on team rewrite. I will say I did get bitten a few times there. There were a few times when I ended up wishing I had refactored instead. As I've grown older and wiser the pendulum has swung the other way towards refactoring. The only real advice I can give is to ask the questions above and make your decision based on that. If you find yourself always wanting to rewrite, perhaps consider working on improving your code reading and refactoring skills and see if that changes the calculus. That and always give refactoring a try. You never know when it might work and it's a good way to gain knowledge about the codebase.