Git SubModules and GitLab CI

One common challenge for LabVIEW developers is how to share code across projects. You write some piece of code that could potentially be useful in some other project. How do you set yourself up so that it can be reused in the future? Generally, my advice on that has been to use VIPM to package it up and distribute it, but recently I discovered a use case for Git Submodules. Git Submodules are a little tricky to use with GitLab CI, so let's talk a little about how to set that up.

Before we get into how to set up submodules though, it is good to talk about when to choose submodules over VIPM. For code that is reused across projects, VIPM is probably the better choice. It works for low-level APIs, frameworks, and development tools. Going through the process of updating a package and re-releasing it is kind of a pain, so I like VIPM for things whose APIs are fairly stable and not under active development. Within a project, I've found that submodules can be a good way to break up a project into various repositories that can evolve independently. It can be a good lightweight alternative to VIPM. Generally, I've found I use submodules for things that have reuse potential, but I don't have a specific reuse case for them yet. Generally, I'm just using the code in this one project at the time, but I still want to keep it separate in case I find another use case later. VIPM fits better for things where I already have multiple use cases already defined.

The Challenge

I was working on a project that was basically a datalogger. The client wanted to add some motion with a 3 axis stage. To keep things truly separate, I started by working on the motion code in isolation. I created a separate repository for the motion code. I wrote a driver for the hardware controller. Then I wrapped it in a DQMH module that could provide all the scanning capability needed. I then created a demo so the customers could test it out. It went through a few iterations, but I finally had a standalone app that did everything we needed from a motion point of view.

Decision Point

At that point, I had a choice: How do I incorporate this motion code into the existing project?  I basically saw 3 choices:

1. Just copy the code from the separate motion repository into the main repository for the project, check it in, and continue development in that repository, essentially abandoning the motion repository. This felt like not quite the right answer.
2. Create a VIPM package. This code felt like something that had the potential to be reused outside this project, but right now this is the only project I am working on for this customer, and it is the only place I've encountered this particular motion controller, so I have no immediate reuse plans for this code. The API also felt unstable. VIPM just seemed a little overkill in this case.
3. Submodules just seemed to fit the bill. Keep things modular and independent, yet still allow for changes to the motion code if needed without jumping through the hoop of rebuilding a package.

So I chose number 3. However, that leads to some problems when using GitLab CI. So let's walk through how to do this.

Creating a submodule

First, we have to create the submodule. That is pretty easy. We go into our main repository and run the following command.

git submodule add url_to_submodule_repo

That command creates a .gitmodules file which looks something like this:

[submodule "motion"]
 path = motion
 url = https://gitlab.com/sas/motion.git

It also clones our submodule repository, so now we have a working copy that we can call from our main code. Of course, we should commit our changes at this point.

Everything works nicely until you make some changes to your main code and push them to your GitLab server and the CI fails.

GitLab CI setup

There are 3 things  I usually do to set up GitLab CI to work with submodules. Step 1 and 2 may not be strictly required, but I like to set things up this way anyway. Step 3 is absolutely required. This process assumes the main repository and submodule repository are on the same GitLab instance. They don't have to be in the same group.

1. I like to change the absolute path in the .gitmodules file to a relative path. In this case, it would be ../motion.git since the repository is in the same group.
2. I also like to add the  GIT_SUBMODULE_STRATEGY : recursive variable to our .gitlab-ci.yml file. This is to make sure that all the submodules get cloned when the runner clones the main repository.
3. The last step is to grant permission to allow our runner to clone the repository. In the submodule repository, Go to Settings > CI/CD. Scroll down to token access and in the box that says "Allow CI job tokens from the following projects to access this repository" enter the group/repository for the main repository. Once we hit the "add project" button, it should show up in the list of allowed repositories.

Once we do all that, our CI should continue to work as normal.

Need Help?

If you need help with using git, submodules, or GitLab CI, let's set up a call and talk about how we can help you.