Approval Tests
Unit testing for legacy code can be really hard. Approval testing can make it easier.
I recently put together and released on VIPM an approval testing library. What is approval testing? Well, let's start by discussing traditional unit testing.
Traditional Unit Testing
For traditional unit testing, let's start with some simple examples.
A Pure Function.
Let's start with a subvi that is a pure function in functional programming terms. All that really means is we have some inputs on the connector pane and we do some calculations on those inputs and produce an output through the connector pane. The key is the only thing that affects the output is the inputs. Every time we run the same inputs through we get the same output. Oh, and there are no side effects or outputs other than the connector pane. So no global variables, no touching database, file system, serial port, etc. Think of the add primitive in LabVIEW - we'll use integers to keep it simple (doubles muddy the water a little). We'll use the add primitive in these examples as the code under test to keep things simple, but in real life, you would replace that with your own subvi that you want to test.
A Simple Example
So to start with, at its most basic level, a unit test is nothing more than a subvi that calls the code you want to test. In this example, we'll just test the add function. So we create a test subvi and drop our add function on the block diagram. We wire some constants up to the inputs and wire the output to an equals. Because the inputs are constant, we know what the answer should be, so we can wire a constant with the correct value. If the equals output is true the test passes.
Getting Fancy
The above VI simply checks one set of inputs, but often we want to check several. We could create a separate test for each set of input conditions, but that can get tedious. So often we'll wrap the code we want to test (in this case the add function) in a for loop and iterate over an array of input values to generate an array of output values. Very straightforward.
Where it works well
This approach works really well in a few scenarios. One is doing Test-Driven Development or TDD. In TDD you start with a problem statement: I want a function or set of functions such that when I give it a certain set of inputs I get a certain set of outputs. That is known ahead of time, so it is easy to write a test. Then implement the code and you know you are done when the test passes and you can't think of any more tests to write. The traditional approach to unit testing also works really well on small functions with only a handful of inputs and outputs where you can easily calculate the desired result by hand.
Where it falls apart
If the traditional unit testing approach works well, why do we need approval testing? What problem does it solve?
Large Objects
The first place you might consider using approval testing is for complicated functions that produce large amounts of output. If a function outputs a large cluster or object, then creating that constant to represent the desired output can be difficult. There are a lot of elements to calculate. That is tedious in and of itself, but often some of those elements will be very difficult to calculate by hand. So this whole process is very time-consuming and tedious, particularly if you have multiple test cases.
Legacy Code
Traditional unit testing is very hard to implement for legacy code. Often the system requirements aren't defined and if they are, often the software does something different and the users are relying on that different behavior. I've written about this before. Another challenge in testing each individual function is that it can be difficult to tell what the desired output is. Functions and inputs often aren't named well and tracing through the execution to determine what is going on can be difficult. This all makes it hard to formulate what the desired outputs are.
Also, often with Legacy code, you want to change the structure which means you aren't as interested in testing individual small VIs - because they may go away. Instead, you are testing larger chunks of code that produce large objects or clusters as results. That puts you in the previous case of having large results and wanting to verify the result as a whole without calculating values for each individual piece.
Approval Tests as a Software Vise
A vise is something that you stick a block of wood into. You can then cut/saw/drill the wood and the wood won't move. We can use approval tests as a software vise. Before we start refactoring, we exercise the code we are testing with a set of inputs. We capture the output. Then as we are doing our refactoring we constantly run the approval tests to make sure our outputs haven't changed. To the outside world, our code behaves the same, but inside, when we are done, it is much easier to read and understand.
Using Approval Testing as a software vise can solve the two problems mentioned above. For large outputs, with approval testing, you don't have to generate the constants, you simply run the code and record what gets output. If it looks good you approve it and you are done. It doesn't work so well for TDD, but for existing code it can work well. For legacy code, approval testing also works well as a vise even for smaller functions. We don't necessarily know or care what the software is currently putting out other than we know that the users find it acceptable. In this case, we aren't using the approval tests to ensure the output is correct necessarily, but simply that it doesn't change as we refactor. The goal is to be able to refactor with confidence that you are not affecting the outputs.
Example - Processing DataFiles
The first time I actually used approval tests was in Python. I was taking a Python Class from Enthought. There was an exercise on refactoring, but much to my chagrin, there was no mention of unit tests (we hadn't covered that yet in the class). I remembered talking to Llewellyn Falco about his approval testing framework a while ago, so I decided to give it a try.
The exercise was rather simple. A Python developer had created a script to read a csv file. The script would then output some report text that had a header with the name of the input file and a bunch of statistics. It printed that out a as a string. There was a csv file provided as an example. The goal of the exercise was to clean up the code so it would be easier to maintain long-term. It was given as part of the problem statement that the output was correct, the goal being simply to refactor the code to make it more understandable and easier to maintain.
Because the script output simply text, using approval tests was easy. I just used pip to install the framework and imported Approvals in my Python Test script. My test had one line. Approvals.verify(generate_report(sample_file)). That's it. I didn't have to figure out what the output should be. As part of the scenario, it was given that running the generate_report function on the sample_file output exactly the output they (the customer) wanted. The approval test wasn't comprehensive. It was just a single file. It didn't account for every possible input condition, but it did give me some assurance when I refactored that I wasn't breaking things. If you were doing something more serious, you might ask for more sample data files, particularly any containing known edge cases. Even just a single data file was useful in my case. It did help me catch myself making some simple mistakes as I was doing my refactoring.
Example - Serial Driver
Recently I used this idea on a LabVIEW project. This is what lead to the creation of the library I put on VIPM. I am working on a data logging project for a company. They want to add a 3-axis stage. They have another program that uses a 3-axis stage for which they have the source code. They want to take that stage and attach it to this new data logger that I am working on. I don't have access to the stage or know anything about it.
The existing code is very much in need of some major refactoring. Lots of nested stacked sequence structures. I was able to track down the subVIs that were interacting with the stage by tracing the VISA Ref through the program. Luckily there were only 3 of them and that was the only place the VISA Ref was used. The code inside those subVIs also needed some work. Each subVI was opening, flushing, and then closing the VISA ref. That's probably not necessary. Also, there were a bunch of magic numbers floating around and a few other issues.
I wanted to be able to refactor these VIs' and put them in a driver class. However, I wanted to make sure that once I pulled them out of the existing program the code still worked. The problem was that I didn't have a stage to test with and I didn't have a manual or anything to know what the serial commands were supposed to be. I didn't even have a model number for the stage. What to do?
I started by using approval tests. I took the existing code and dropped it into a test VI. Then I used a program called com0com that Enrique Noe Arias put me onto. It creates a pair of virtual serial ports that form a virtual loopback. In my test, I connected one side to the existing instrument driver code and on the other side I just did a serial read and passed that into my Approval Testing method. I ran it and used the approval testing framework to log the results. Since I had no idea what the code was supposed to output over the serial port other than the fact that according to the customer the current code works, I just approved it.
I did that for each of the 3 original serial VIs. Once that was done, I had my vice setup, so I could go to work. I was able to slowly refactor the code into a nice class-based driver all the while making sure that the code still sent the same commands over the serial port.
Getting Fancy - Printers
In both of these cases, the output was text, so doing the approval testing was very easy. The approval testing framework specifically stores the approved and actual results in text files so it works well with Git and so they are easy to diff. When you have non-text results then you need some sort of printer function to turn whatever type of data you have into text.
The simplest printer is just to flatten whatever data you have to xml or json. I much prefer json. It's probably a good idea to pretty print it, as that will likely make the diffs more meaningful. You just take the pretty-printed json and feed it into the approval method and that is it.
You can get fancy with printers. Sometimes you might be testing some code where the output changes in some "predictable" way every run. If you are testing a report generator, it might include today's date. Your printer can filter that out. Find today's date and replace it with some unchanging token like TODAY. This also works for things like file references. After converting to json, find the element corresponding to the reference and check to see if it is valid and open, and replace the ref with an appropriate text value. If it is always supposed to point to the same file, you could also replace the reference with the path it points to.
Where To Learn More
To start learning about approval testing, you might check out Llewellyn's website.
Here is a link to download the package I put together from VIPM.
And here is the repository in case you want to see the source code.
Need Help With Legacy Code?
If you need help with Legacy Code and you want someone who is not just immediately going to tell you to write it, then let's chat. We'll take a look at your code and give you our recommendation. Rewriting may still be an option, but often there are things that can be done to avoid rewriting.
