Gitlab CI/CD

Integrated directly into GitLab since version 8.0, the FB3 offers the associated Continuous Integration and Delivery service GitLab-CI/CD.

Runners

GitLab does not execute builds itself but delegates this to so-called Runners. A Runner is a process that can run on any machine and polls the GitLab server to fetch and process pending jobs. There is a distinction between "instance," "group," and "project" runners. A project runner serves only specific projects, while a group runner serves all projects within certain groups. Instance runners handle all jobs for which no project or group runner exists and that are enabled for the use of instance runners (see Settings -> CI/CD -> Runners in the project settings).

Since not all jobs work in all environments, tags can be used to control which jobs can be executed on which runners. If a job has tags assigned, it will only run on a runner with the matching tags. If a job has no tags assigned, it will only be executed by runners that are explicitly configured to handle untagged jobs. If multiple runners are eligible to execute a job, one will be randomly selected.

Instance Runners

The FB3 provides a number of instance runners:

Platform	Tags	Host names
Mac OS X	`darwin`	`m{01..06}`
Kubernetes	`linux, docker, kubernetes`	`fb3-k8s-{1..4}`

The macOS runners run on the pool computers in the E0 in MZH. Before running jobs on them, it should be checked whether all necessary tools and dependencies are installed. Missing dependencies can be installed on request (serviceinformatik.uni-bremen.de).

The Kubernetes runner is installed on a Kubernetes cluster and executes jobs in OCI containers. The Kubernetes instance runner also handles untagged jobs.

A snippet of the list of available instance runners can be found under Settings -> CI/CD -> Runners in the project view. You can only determine which instance runner a job was executed on by checking the build log afterward.

Group/Project Runners

You can also create your own runners in the context of a group or a project, which will then serve only your own projects. The advantage is that you can customize the development environment or simply continue using an existing environment directly. A prerequisite for operating a runner is that it can reach the GitLab server.

Further information on creating your own runners can be found in the project view in GitLab under Settings -> CI/CD -> Runners.

Security

This section applies only to runners that use the shell executor (currently the macOS runners among the provided instance runners).

When using and setting up runners, several points regarding security and the protection of confidential data should be considered:

Jobs are arbitrary shell scripts executed with the permissions of the user under which the respective runner process runs. This means that any GitLab user with the appropriate access rights to a project can execute any commands with the permissions of the runners serving that project.

Particularly when setting up your own runners, it is important to ensure that no confidential information is accessible and that the runner does not have more permissions on the underlying system than necessary for processing its jobs.

Particularly with instance runners, it should be noted that the private authentication token of the runner could be read, allowing the runner to be impersonated from another machine. Additionally, it is possible to directly access the code of the projects previously handled by the respective runner as long as they remain in the runner's working directory.

Docker

As described above, runners tagged with linux, docker, kubernetes execute their jobs in a Docker container. The default image is debian:latest, but this can be configured according to the official documentation. Dependencies can be installed in the Docker image either using the service notation or by calling the package manager of the image in the before_script section in .gitlab-ci.yml.

Alternatively, custom-built Docker images can be used through the integrated container registry.

Tips and Tricks

Building container images

There are currently two ways to build container images with the Kubernetes instance runners: Kaniko and BuildKit. Kaniko should be preferred whenever possible, as BuildKit requires a separate service with elevated privileges, of which only a few instances are running, which may lead to bottlenecks under high usage.

Here is an example job in .gitlab-ci.yml to build a container image from a Dockerfile in the root of the repository and upload it to the project's container registry under the default image name:

Kaniko

The latest Kaniko image should be used. (The image currently doesn't have a latest tag.)

build:docker:
  stage: build
  image:
    name: ghcr.io/kaniko-build/dist/chainguard-dev-kaniko/executor:v1.25.1-debug
    entrypoint: [""]
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:latest"

BuildKit

The network address for the BuildKit service is defined in the environment variable BUILDKIT_HOST, which is set by the Kubernetes runners. If for any reason one wants to use a self-hosted BuildKit service, this variable can simply be overridden in the job settings.

build:docker:
  stage: build
  image: moby/buildkit:latest
  before_script:
    - mkdir -p ~/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > ~/.docker/config.json
  script:
    - buildctl build
        --frontend dockerfile.v0
        --local context="${CI_PROJECT_DIR}"
        --local dockerfile="${CI_PROJECT_DIR}"
        --output type=image,name=${CI_REGISTRY_IMAGE}:latest,push=true

Accessing resources of other projects

You can grant a job permission to access the resources of other projects, for example, to build a container image only once. To do this, in the project that is to be accessed, go to Settings -> CI/CD -> Job Token Permissions and add the project that runs the job to the list of authorized projects.

Caching artifacts and dependencies

The Kubernetes instance runners are configured with a cache that can be used to transfer build artifacts or dependencies installed by the job between individual jobs.

Gitlab Pages

Gitlab can be used to host static websites. Dynamic content, such as with PHP, is not possible at this point.

To use this feature, in addition to CI/CD, the "Pages" option must be enabled in the project settings under Settings -> General -> Visibility, project features, permissions . After that, you can follow the instructions under Deploy -> Pages to create a CI/CD job that generates the content and make further settings.

See the above-linked Gitlab documentation for more options and examples.