Avoiding Duplication in GitLab yaml.

DRY - Don't Repeat Yourself. This is a pretty common mantra in software development. I have seen it overused and abused, yet it is generally a good principle to follow. Aside from saving on typing, following DRY makes it easier to make changes to our code and keep it consistent. It allows us to make changes in one spot instead of many. I want to highlight a couple features of GitLab CI that help us keep our YAML files DRY.

Default Section

An easy way to avoid duplication is to put common elements into a default section. This changes the default values for every job. You can override elements for specific jobs if you need to, by simply defining them again inside that job.
Let's start with a simple real-world example. Below is some YAML I pulled from my Blue AutoFormatter.

default:
  artifacts:
    expire_in: 7 days
    when: always
  retry:
    max: 2
    when: job_execution_timeout
    

This particular piece of YAML solved 2 problems I was running into. The first problem was that I was running out of space in my GitLab account. I had too many artifacts. By default, GitLab was storing old artifacts, and they weren't expiring. I had to figure out how to go and delete them all. This was a few years ago, so I don't remember exactly, but I think I eventually stumbled upon some GitLab API call that allowed me to do that, and it was painful. Now I add a default artifacts section to all my projects to ensure everything expires. Now, I don't worry. Regardless of this setting, GitLab will keep the most recent artifact for you.

The second issue involved VIPM on my runner machines. For some reason, VIPM would occasionally hang, causing my job to timeout. I tried, but I could not figure out why. However, simply retrying the job often worked. Adding this default retry section to retry on timeout solved the problem. Now every time a job times out, whether due to VIPM or some other issue, it will get retried. However, jobs that simply fail for some other reason - like the unit tests failed - will be unaffected. They will continue to fail without retrying. This means I will still detect problems like flaky tests.

Extends

The default section works great, but sometimes you want to duplicate a whole job. In this particular case (again pulled from Blue), I wanted to duplicate my test installation job. I wanted to install and test Blue, not just in LV2020, but in the latest version of LabVIEW as well. That involves simply changing one variable: the BASE_VM.

Duplicating the entire job to simply change one variable seemed like overkill and seems to violate dry. That's when I discovered extends.

Here is the yaml:

test_installation_W10_LV20:
  stage: install
  variables:
    BASE_VM: "W10_LV20"
  tags:
    - "kvm"
  script:
    - ./scripts/test_install.sh $CI_PIPELINE_IID
  artifacts:
    paths:
      - pytest/*.received.*
    reports:
      junit: 
        - reports/pytest_report.$BASE_VM.xml
        - reports/InstallLUnitReport.$BASE_VM.xml
        - reports/PreBuildTestReport.PostInstall.$BASE_VM.xml
        - reports/ScriptingTestReport.PostInstall.$BASE_VM.xml


test_installation_W11_LV25Q1:
  extends: test_installation_W10_LV20
  variables:
    BASE_VM: "W11_LV25Q1"

When you use extends in one job, it inherits all the elements from another job. Then you can override and change any that you need to. There are some rules around exactly how this works. The YAML gets merged in a very specific way. In some cases, things get completely overridden. In other cases, things get merged. You should look carefully at the GitLab help to be sure. In this particular case, I am simply overriding the value of the BASE_VM variable.

You can also use extends with hidden jobs to create templates. If you start a job name with a period (.), then that job is not executed. You can though, use it in another job via the extends keyword, which turns it into a template.

Need Help?

Need help managing or creating a CI/CD pipeline? We can help. Let's talk. Use the button below to set up a conversation.