Gitlab CI

Enabling device testing in GitLab

Note

This section assumes that the devices under test you want to use are already exposed on postmarketOS gitlab by other community members, or you have the capabilities to do it yourself.

The official documentation on how to control and add devices to GitLab forges from CI-tron is documented upstream, and forms the basis of our implementation.

There are, however, a couple of differences compared to the official documentation:

  • The provided template files are vendored as-is in our GitLab instance to reduce our reliance on external projects.

  • Common patterns across postmarketOS devices have been moved to hidden jobs that any job can extend from in order to reduce code duplication.

The common patterns can be found under ci-common, and can be included by adding the following excerpt at the top of your projects’ .gitlab-ci.yml:

variables:
  # WARNING: it is strongly advised to use a commit hash instead of a branch name:
  CI_TRON_TEMPLATE_PROJECT: &ci-tron-template-project postmarketOS/ci-common
  CI_TRON_JOB_TEMPLATE_PROJECT_URL: $CI_SERVER_URL/$CI_TRON_TEMPLATE_PROJECT
  CI_TRON_JOB_TEMPLATE_COMMIT: &ci-tron-template-commit 7c95b5f2d53533e8722abf57c73e558168e811f3

include:
  - project: *ci-tron-template-project
    ref: *ci-tron-template-commit
    file: '/ci-tron/common.yml'

You can now use extends to inherit from any of the hidden jobs defined in common.yml.

Generate and run the boot artifacts

To generate the boot artifacts needed by the devices under test, you may use the .pmos-ci-tron-build-boot-artifacts job from the ci-common repo (ci-tron/common.yml). The job requires you to set 3 variables:

  • DEVICE_NAME: The device name, in the wiki format (eg. qemu-amd64)

  • INSTALL_PACKAGES: The list of packages that should be installed

  • KERNEL_VARIANT: The kernel name (eg. lts, or virt), if applicable. Leave empty if only one kernel is supported.

The job also makes it possible to use packages or binaries built in previous CI steps into the generated kernel an initramfs. For that it considers 3 locations:

  • keys: Anything located under this directory will be copied to pmbootstrap local keys.

  • packages: Any package located under this directory will be copied to pmbootstrap local package repository. Together with keys, this can be used to use new packages instead of the ones in the binary repo.

  • rootfs: Any file located under this directory will be copied to the pmbootstrap rootfs chroot under the same relative path. This can be used to run custom binaries (e.g: mkinitfs, boot-deploy) instead of the ones provided by packages.

To boot the kernel and initramfs generated in the previous stage, you may use the .pmos-ci-tron-initramfs-test job from the ci-common repo (ci-tron/common.yml). The job requires to set the following elements:

  • tags:: The list of tags the Gitlab runner should have to run the job

  • variables:: All the CI-tron variables needed to specify which kernel and initramfs to load

Example

The following example will build the boot artifacts for the lts kernel of the qemu-amd64, then boot them on the device.

prepare-qemu-amd64:
  stage: hardware tests
  extends:
    - .pmos-ci-tron-build-boot-artifacts
  variables:
    DEVICE_NAME: qemu-amd64
    KERNEL_VARIANT: lts
    INSTALL_PACKAGES: device-${DEVICE_NAME} device-${DEVICE_NAME}-kernel-${KERNEL_VARIANT} postmarketos-mkinitfs-hook-ci

test-qemu-amd64:
  stage: hardware tests
  extends:
    - .pmos-ci-tron-initramfs-test

  # List of tags the device/runner should have to be picked by GitLab for testing.
  # Alternatively you can use .pmos-ci-tron-runner-qemu-amd64 from common.yml
  tags:
    - cpu:arch:x86_64
    - virtio:codename:VIRTIO

  # Wait for the preapre-qemu-amd64 job to be over before executing this job
  dependencies: []
  needs:
    - job: 'prepare-qemu-amd64'
      artifacts: false

  # Specify how to get the kernel and initramfs to use for testing
  variables:
    CI_TRON_KERNEL__URL: "glartifact://build-qemu-amd64/${CI_TRON__PMB_EXPORT_PATH}/vmlinuz-${KERNEL_VARIANT}"
    CI_TRON_INITRAMFS__INITRAMFS__URL: "glartifact://build-qemu-amd64/${CI_TRON__PMB_EXPORT_PATH}/initramfs"

Add a new device to pmaports

See the correspoding pmaports documentation

Debugging

If something goes wrong on device testing in CI, it might be needed to go do some debugging related to the actual devices.

Hardware tests always run at the end of the pipelines, and often depend on previous building and testing jobs to be successful. This guide does not cover how to debug those previous steps, like building packages in pmaports from MR changes, or compiling mkinitfs. However, once those are successful, there are 3 possible thing that can go wrong:

Preparing ci-tron artifacts for booting on devices

These artifacts should be generated using the .pmos-ci-tron-build-boot-artifacts hidden job from ci-common’s ci-tron/common.yml, which in turn calls the build-boot-artifacts.sh script in the same repository.

If something goes wrong in this step, it should be possible to:

  • Run the build-boot-artifacts.sh script locally, making sure the required variables and paths contain the right data.

  • Push your changes to a branch in the ci-common repository (or a fork of it), and modify the following variables at the top of the .gitlab-ci.yml file to test your changes:

    • CI_TRON_TEMPLATE_PROJECT: &ci-tron-template-project my-user/ci-common

    • CI_TRON_JOB_TEMPLATE_COMMIT: &ci-tron-template-commit my-branch-name-or-commit-sha

Testing the artifacts on the real devices

If the testing itself fails, it might be necessary to look into the devices themselves. In this case, the test jobs should upload job_script.sh as an artifact. The script contains the details on the code run on the ci-tron gateway (see the architecture on top of this page). Therefore, anybody with access to the gateway can download that script, modify variables at wish (e.g: maybe you want to test on a different device in the same farm, or build and pass your own initramfs), and run on the device. For example, it might be desired to build a custom initramfs with the debug-shell mkinitfs hook installed. Then it can be built locally, and fetched in the device by providing a different value in CI_TRON_INITRAMFS__INITRAMFS__URL. For further details on variables to use, refer to the official documentation.

To be able to access the gateway, you must have access to it, preferably through wireguard. Giving access depends on the administrator. For farms managed by the postmarketOS-team, any team member can request access to the infra team. The details can be found in the private infra repo.