Approval Testing For LabVIEW 2.0 Released

Work has been a little slow lately, so I've had lots of time to work on side projects. One project that I have been working on a lot lately is Approval Testing for LabVIEW. Approval Testing is an established thing created by Llewellyn Falco. You can find out more about approval testing at https://approvaltests.com/. You'll see that LabVIEW is now an officially supported language! I encourage to you to visit that site and read up on approval testing in general. You can also check out my Software Engineering Radio conversation with Llewellyn about Approval Testing.

Version 1.0 of Approval Testing in LabVIEW was just a really simple proof of concept. There was just one method. It took a string input. It was up to the person writing the test to convert whatever they wanted to test into a string. The VI simply compared the string to a .approved file. If it matched, then great! The VI output a true for passed. If they didn't match the data got stored in an actual file and VI output false for failed. It was up to the user to connect that output up to their favorite Unit Test Framework. It was also up to the user to figure out how to approve the results. It was the simplest thing that could possibly work.

Here is a 7x7 presentation at GDevCon NA where introduce version 1.0.

Working with Llewellyn

Shortly after GDevCon NA last year, I began working as a guest host on Software Engineering Radio. I needed to find guests to interview, so I reached out to Llewellyn to talk about Approval Testing. After our interview, we started talking and we set up a weekly meeting to work on Approval Testing in LabVIEW. Working with Llewellyn definitely opened my eyes to what is possible. He's helped me polish the interface and make Approvals in LabVIEW really easy to use and user-friendly. I got to benefit from his many years of experience with Approvals. There were a lot of little usability items that he suggested that I hadn't thought of. They made a huge difference.

What is new in 2.0?

First of all version 2.0 is a breaking change. See the upgrading section below for guidance on upgrading. I did my version 1.0 prototype without talking to Llewellyn, so there were definitely some that I wanted to change to maintain compatibility/similarity with Approval Tests in other languages. That way I could benefit from all the features Llewellyn (and other contributors) had already implemented in other languages. I'll try to highlight the major changes here.

Upgrading TLDR

If you are upgrading from version 1.0 - here is the tldr

• do a find and replace of "Pass If Approved.vi" with "Verify.vim" Consider using the extension for your unit test framework (note this includes the assertion so requires a little more editing on your part)
• find all your ".approved" files and change the extension to ".approved.txt"
• Update your .gitignore to ignore *.received.*
• We recommend you install Beyond Compare (paid, but well worth it) or WinMerge (free alternative) as a diff viewer. This will automatically pop up if your test fails. It is not required, yet it makes reviewing failures and approving results much easier.

Move to GitHub

This is more or less transparent to users. I was using GitLab. Llewellyn uses GitHub for all the other versions of Approvals. Llewellyn suggested I move to GitHub. I didn't really have a good reason not to. Moving over had the benefit that I got to learn how to use GitHub Actions, so that was a nice bonus. The repository is here now: https://github.com/approvals/ApprovalTests.LabVIEW

Pass if Approved.vi renamed

The first major change you will notice when you upgrade is that "Pass if Approved.vi" no longer exists. When you upgrade all your existing approval tests will complain about not being able to find this VI. It has been renamed and turned into Verify.vim. The immediate fix is to do a find and replace. You may want to use one of the extensions - see below for more on that.

Verify.vim is a major improvement. You can now pass in any datatype. It does it's best to flatten it to a string. As a fallback for complicated cases, it just flattens it to JSON. It also now has an options input - see below for more on that.

Actual changed to receive

Version 1.0 used .actual to denote the results of the test when it failed. Now we use .received to match Approval Tests in other languages. We also added an additional extensions. So now your result files will be *.approved.txt and *.received.txt or *.approved.json or if verifying a file, we just keep the original extension at the end, whatever it is.

New Pallette

The next thing you will probably notice is that the Pallette got redesigned.

Here is the original:

And here are the new palletes.

Extensions

The extensions are featured prominently at the top of the main pallette. These pallette items will drop all the code necessary to create a new verify. If you have ever used Caraya and opened Quick Drop and typed in New Test, it is the same idea. You just start with a blank VI. Drop one of these VIs off the pallettes and then wire up the connector pane and follow the comments.

The advanced pallette contains the bare verify methods without any framework specific wrappers. If you use a different framework, you'll want to use these. Maybe I should have called this pallette generic instead.

Different Verifies

Immediately you've probably recognized that there are multiple verifies now.

• Verify - This is the replacement for "Pass If Approved.vi" from version 1.0. It is now a VIM so it takes in anything and attempts to intelligently convert it to a string. If it can't figure out what to do with your input, as a fallback it converts it to JSON.
• Verify as JSON - This takes a variant input and flattens it to JSON.
• Verify as Parameterized - This takes a VI reference - this the VI with the behavior you want to verify. It also takes in a set of maps. Each map provides the input value for each input to the VI. The Verify captures the output of the VI for each set of inputs you provide.
• Verify as Combinations - Similar to Verify as Parameterized except you provide a single map that lists all possible values for each input. The verify then calculates all possible combinations and capture the output for each combination of inputs.
• Verify CLI Output - This takes a CLI command in and captures the output of system exec - ie. STD output, STD ERR and exit code.
• Verify File - This takes in a path to a file. It verifies the file hash. If it detects that the file is a VI, instead of a file hash it uses the LVCompare function. This is very useful for testing scripting code. You have a base VI. You create a temp copy of it. Run the scripting code on the temp copy and then verify the temp copy. NOTE: the LVCompare functionality is not perfect. It can be somewhat flaky particularly in LV2020. Apparently it's gotten better in more recent versions.

Here is an example video showing how to use Combination Approvals.

Difftools

Another thing you may notice is that we have implemented difftools. If you have Beyond Compare, or WinMerge installed, Approvals will automatically detect that and launch either one when your test fails (it prefers Beyond Compare if it finds both).

This makes reviewing and approving changes much quicker. The window on the left is the received data and the window on the right is the approved data. In Beyond Compare to approve the changes you just simply select all on the left file (ctl+a) then click the arrow to move all the changes to the approved file and then save it.

On your CI machine, assuming you don't have Beyond Compare or WinMerge installed then they won't get displayed so it won't hang your CI. It will just generate the received file quietly and fail the test.

Options

We added an options class input. This is for future expansion opportunities. For now, it is not very stable so you should avoid using it except for Scrubbers as it may have breaking changes in future versions.

Scrubbers

Scrubbers are really useful. They allow you to deal with variable/unpredictable inputs like GUIDs or Dates. There are 2 built in. It is classed based so you can easily build your own. There are 2 included scrubbers:

• Regex - This one simply does a simple regex replace. Anything that matches the regex gets replaced with a single token. For example you might replace all dates with "DATE" if you don't care about the value of the dates.
• Regex count unique - This is similar to the above except that it counts unique values. So you end up with "DATE_01", "DATE_02", etc. If a date repeats then the number repeats as well. This is for cases where you don't care about the exact date, but you care that it repeats.

GLA Summit Preview

I am planning on doing a presentation at the GLA Summit showing how to use Combination Testing for refactoring Legacy Code. It is a good practical example of where approval tests shine. Be sure to check it out.