Hardware CI

Hardware CI includes a lot of moving parts, and as debugging is a lot harder than development, it can be hard to understand which part of the whole system is resulting in failures. This aims to explain the CI setup in pmaports to make it easier to understand the different moving parts.

Overall architecture

The hardware CI consists of the following building blocks:

  • Dynamic generation of jobs: the .ci/lib/generate_build_jobs.py is run on CI to detect what changed from the target branch. If there are packages to be built, or changes that should prompt testing on hardware, a complete GitLab pipeline definition is created based on the input template. This generated pipeline is consumed by GitLab to create a downstream pipeline and contains all the details about what to build, and the specific process to test each of the relevant devices. If there is nothing to build or test, a dummy pipeline is created instead (see https://gitlab.com/gitlab-org/gitlab/-/issues/368248)

  • Building changed packages: those packages that are changed are built by CI, together with a repo that allows to use mrtest add and mrtest upgrade. This is independent of any work related to hardware.

  • Preparation of test artifacts: if any of the packages that were built is a dependency of a DUT, then CI proceeds to create boot artifacts for the DUT using those newly-built packages. This step often makes use of helper functions in ci-common.

  • Definition of tests: how and what to tests for a device is defined by a gitlab-ci.yml.j2 jinja template under the device package folder. Those templates often include jobs from other places, since tests are usually generic and shared across multiple devices.

Add a new device to pmaports

Adding a new device for testing under pmaports can be done by adding a gitlab-ci.yml.j2 file in your device package. This file should define all the test jobs for your device and will get included when generating the final pipeline by generate_build_jobs.py.

Your jobs are expected to set all the necessary attributes to run jobs on a CI-tron-provided runner, but the following template variables and hidden jobs are provided to you as helpers:

  • Jinja template variables (defined in .ci/lib/generate_build_jobs.py):

    • device: Instance of the Device class which represents the device under test.

  • Hidden jobs (defined in .ci/build-jobs.yaml.j2):

    • .test-{{ device.name }}: Defines all the common parameters for testing this device (;

    • .device-boot-flow-separate-artifacts: Sets the kernel and initramfs URLs when the prepare job generates separate kernel and initramfs files as artifacts.

    • .device-boot-flow-fastboot-image: Sets the full fastboot boot image when the prepare job generates a boot image ready for booting into the DUT.

Note

Rather than duplicating the gitlab-ci.yml.j2 file for all the devices that can make use of it, you may save the file in .gitlab-ci/packages/ then link to it in the device folder.

Tests

There are 3 kind of tests in the postmarketOS hardware CI setup:

Kernel tests

One important part of the hardware CI is to be able to test all the low-level integration and logic. That includes, first and foremost, the kernel and some initramfs functionality.

The package postmarketos-mkinitfs-hook-ci takes care of that. If installed, it will:

  • Run in the initramfs any scripts installed under /usr/libexec/pmos-tests-initramfs.

  • Prints a success or fail so it can be detected by whatever is orchestrating the tests.

  • Print a message that the execution is done, and halt the boot.

This allows for any package to provide arbitrary scripts in that location, and have them execute during hardware-CI testing on the initramfs.

However, most test routines are pretty generic. So instead of every device creating there own, most of them are aggregated as subpackages of the postmarketos-test package. Then, if a DUT wants to enable a specific initramfs test, everything it needs is a subpackage that depends on the corresponding tests from postmarketos-test, and that gets installed if postmarketos-mkinitfs-hook-ci is also installed. This subpackage is, as a convention, named pmtest. This is how device package’s APKBUILD that enables tests for unl0kr and suspend would look like:

subpackages="$pkgname-pmtest"

pmtest() {
    install_if="$pkgname=$pkgver-r$pkgrel postmarketos-mkinitfs-hook-ci"
    depends="postmarketos-test-suspend postmarketos-test-unl0kr"

    mkdir -p "$subpkgdir"
}

Boot tests

Note

The infrastructure for these tests has not been developed yet.

Boot tests are expected to test the boot sequence. For that, they should run the exact same boot sequence that the DUTs they run on, including an unmodified initramfs. These tests have not yet been developed. These tests could test things like checking that the partition detection is correct, or that FDE runs successfully.

System tests

Note

The infrastructure for these tests has not been developed yet.

Anything that requires complex user-space integration or deamons should be done on top of the rootfs, not the initramfs. These tests could cover things like checking WiFi or Bluetooth connectivity, or making calls.

Debugging

Dynamic pipeline generation

Testing pipelines in pmaports are generated dynamically, so that changing an armv7 devices only builds for that architecture, and does not run ci-tron tests for any other device. The pipeline is generated by the .ci/lib/generate_build_jobs.py script. The script requires jinja2 and pmbootstrap to be installed, but once they are it can be run passing a jinja template (unless doing changes, it should be .ci/build-jobs.yaml.j2) as a first argument, and the output file as a second argument. This way, it is possible to run the code generation locally, and inspect the resulting pipeline.