Spawning VMs like Docker Containers

So I have been doing CI in LabVIEW for several years, something like 5 or more. All that time I've been using Virtual Machines. For each project, I would create 2 Virtual Machines, 1 for development and 1 as a CI runner. I would then just have that CI runner always running in the background waiting for jobs. It was not an ideal setup yet it worked.

The Problem

That setup worked and it had problems. For starters, having a bunch of VMs running and spending most of their time simply waiting for jobs is a bit wasteful. Being able to simply spin one up as needed would be nice.

Secondly, the more important problem was the shared state between jobs. If I ran a job and it installed a VIPM package on the VM, when that job ended that package was still there waiting for the next job. With one VM runner per project, that wasn't terrible. At some point, I migrated to one VM runner per LabVIEW version because I had a bunch of projects and I was running out of disk space. Switching from one project to another did sometimes cause problems.

A last problem is that with one VM per project or LabVIEW Version, I couldn't run jobs in parallel. I mean I guess I could have 4 different VMs for each project or version and that would require a massive server and wouldn't be very efficient.

Docker

At first glance, Docker seems to solve all these problems. You can spin up Docker images on demand. They start from a fresh slate every time, so you have no shared state between jobs. You can also spawn multiple instances of the same image. It works really well with text-based languages. Sounds great, right?

The problems with Docker

LabVIEW does not play well with Docker. I've always liked the idea of Docker and setting up a LabVIEW Docker image always seemed a bit much. I know Felipe, Christian, and Chris Stryker have made strides in that area, and yet it still eluded me.

The first thing that makes using LabVIEW in a Docker container hard is its reliance on the GUI. Installing and licensing it without a GUI can be challenging. NIPM makes this easier.

A second issue is installing hardware drivers inside a Docker Container. NI Drivers don't like being installed inside a Docker container. I know Christian has found some workarounds and it sounded painful.

A third issue is simply troubleshooting. Because Docker doesn't have a GUI troubleshooting can be hard. How can you tell that VIPM or LabVIEW are hung up on some licensing dialog box?

What if I could spawn VMs like you can spawn Docker Containers?

If you could spawn a VM on demand, it would solve a lot of these problems with Docker. You could set up the VM using a traditional GUI. You could install hardware drivers easily. You can also leave the VM disk for the spawned VM around for troubleshooting if needed (by modifying the cleanup script).

Spinning up VMs on demand still provides many of the benefits of Docker. Mainly that I can spin them up on demand, that they always start with a fresh slate, and that I can launch many in parallel. Wins all around!

Enter KVM Executor

I had given a presentation a few years ago at NI Week about trying to make this all work with VMWare and I could never quite figure it out. In the end, I was on the right track as far as launching VMs using the CLI and bash script, I was just doing it in the wrong spot. I was trying to launch them as part of the job script and not as part of the runner itself. I gave up. At some point, I stopped using VMWare and switched to KVM. Then a few months ago I stumbled upon the libvirt/KVM executor.

Using libvirt with the Custom executor | GitLab

GitLab product documentation.

Docs

The scripts

This executor is a set of scripts. The way you set it up is the following: You have a server with KVM installed on it. That is your runner. You set it up as a custom runner and point gitlab-runner to this set of scripts. The scripts handle launching, communicating over SSH (to run the job script), and shutting down/cleaning up the VM.

I had to modify the scripts slightly to work on my machine. Here is a GitLab Repo showing my modifications.
https://gitlab.com/sas-gcli-tools/kvm-executor

Setting up the Base VM

This system relies on having a Base VM that is cloned for each job. To start make sure you install all the programs you need, like LabVIEW, VIPM, DAQmx, maybe some base VIPM packages, etc. - anything you don't want to install every time you run a job.

A couple notes:

• The runner communicates with the base VM via ssh. So you have to set that up in the guest. Make sure that the user running gitlab-runner on the host can login to the guest with an ssh key. Also set the ssh connection to use Git Bash (or your preferred shell).
• On the base VM you'll want to install Git for Windows You will want to run the following in an admin powershell prompt git config --system --unset credential.helper That will turn off Git credential caching. If you don't you'll get an error in your console output and it will still continue. Also, get rid of any .bashrc file in your home directory as that can cause problems with the ssh connection.
• You'll also want to make sure you have gitlab-runner on the base VM and on the PATH, so it can be found. You don't have to install it as a service or register it or anything. Gitlab-runner is only used to upload artifacts. If you aren't uploading artifacts then you don't need it.
• If you are using any variables, either user-defined or predefined in the custom executor scripts on your host, you'll need to prefix them with CUSTOM_ENV_

Shrinking our base VM

Once you have the base VM set up, it is a good idea to shrink your base VM. I have instructions here. The difference can be dramatic and will save you a ton of disk space. Performance-wise with a 12-15GB compressed VM disk, it only takes 1-2 minutes to spin up a new clone, which works well for me.

Using the KVM runner

Using the KVM runner is pretty easy. I simply tagged it with KVM, so your job needs that tag. Then there is a matter of telling the KVM runner which BASE_VM to use, so you set a BASE_VM variable either at the file or job level and that's it. Otherwise, your .gitlab-ci.yml file looks the same as it did before.

Running Multiple Jobs in Parallel

To run multiple jobs in parallel, all you have to do is change the concurrent setting at the top of the gitlab-runner config.toml file.

Troubleshooting

If you are having issues with a job, you can edit the cleanup script and comment out the code to delete the disk image. Then just run the job again and the disk image will stick around afterwards. Create a new VM with that disk image and then you can inspect it. You could also skip creating a new VM and just load the disk image directly following instructions like this: https://gist.github.com/shamil/62935d9b456a6f9877b5 If you do either of these, make sure to reset the cleanup script and purge any unneeded VMs afterward.

Housekeeping

Occassionally if things fail, depending on how they fail, you may end up with leftover disk images. The names have a very specific format and you can edit where they are stored - for example to throw them all in a clones folder. It might be useful to have a cron job check for cloned VM disks that haven't been touched recently and delete them.

Managing multiple Base VMs

The initial SSH setup takes a little time. I would do that first and have a base Windows image without LabVIEW installed. Then I would copy that disk to create a new base VM for each LabVIEW version. Then if you have a special project, you can copy one of the LabVIEW versions and modify it further. That way you don't have to do the SSH setup each time.

Need Help?

Using a custom executor is a great way to improve your CI process. It's made my pipelines much faster and more robust. I just outlined how to use it for KVM, but you could easily apply the same idea to VMWare, HyperV or any other VM solution. You could also use it with a cloud provider to spin up cloud images on the fly (assuming the cloud provider has a CLI tool - I assume most do). If you want to explore those options, we'd be happy to help. Set up a call using the button below.