Added a clause to the implementation plan to alert Alpine developers to our efforts when we start work downstream.
Reworded the motivation section to be clearer about the issues faced without hermetic-usr.
Clarified the configuration order example to state that the filenames are generic examples and not representative of an actual layout.
Explained the addition of systemd components to aports/main and referenced the contingency plan in the requirements section of the implementation.
Explained why we won’t use /usr/etc/ like OpenSUSE.

Vocabulary¶

distributor - The top-level distribution providing the system. (e.g. Alpine)

FHS - Filesystem Hierarchy Standard

host - The device running the operating system. Handled by the sysadmin.

stateful - Having a state that is changed from the original system.

stateless - Having no device-specific state. Same as the original system.

TLD - Top level directory. A directory under root (/).

UAPI - The Linux Userspace API Group.

UAPI Configuration Specification - The UAPI specification for configuration handling and override. Sometimes called the hermetic-usr configuration style. Docs

Summary¶

Hermetic /usr (from now on referred to as hermetic-usr) is a file hierarchy and configuration management system in which /usr/ is the sole source of truth for the operating system, containing all distributor provided files. This prevents core system files from becoming stateful, limiting configuration drift and allowing for the system to be restored to distributor configuration.

Configuration drift is limited by following the UAPI Configuration Specification. Splitting reading of config files based on distributor, system overrides (e.g. downstream configs), and sysadmin configuration, with each overriding one another.

Motivation¶

We have 2 problems:

Impossible to fall back to the packaged set of files after user modification.
- Files added by system administrators can mess with the core system state.
  - Almost always these files are not overwritten on a config change.
- Some files for system functionality are not provided automatically. (e.g. /etc/fstab/)
Impossible to differentiate files shipped by Alpine, postmarketOS, and modified by system administrators.
- Current model requires either overriding Alpine packages or using the rare program-specific override directories.

We want to solve those issues to improve the reliability, consistency, and reproducibility of our system.

Filesystem Layout¶

Below is are examples of a non-hermetic-usr filesystem tree and a hermetic-usr layout. The tree is only expanded to show the relevant directories to the proposal.

The hermetic tree is only shown at image build-time, as the runtime tree is similar to the non-hermetic-usr tree.

Non-Hermetic /usr¶

/
├── dev
├── home
├── media
├── opt
├── root
├── sys
├── usr
│   ├── bin
│   ├── include
│   ├── lib
│   ├── libexec
│   ├── local
│   ├── sbin
│   └── share

Hermetic /usr (Build Time)¶

/
├── usr
│   ├── bin
│   ├── include
│   ├── lib
│   ├── libexec
│   ├── local
│   ├── sbin
│   ├── share
│   │   ├── factory
│   │   │   ├── etc
│   │   │   ├── opt
│   │   │   └── var
│   │   └── [pkg specific dir]

Configuration Hierarchy¶

The UAPI Config Spec follows a system of configuration hierarchies. A sustainable way to implement this is to define it in the values set in build systems and implementing a value called “distconfdir”. Some systems already have distconfdir, but others will need to add it.

For example, in pmOS we would set: prefix=/usr, distconfdir=/usr/share/"$pkgname", sysconfdir=/etc/pkgname

This would correlate to this reading order for configs. (The filenames are not representative of the actual filenames. Configs can be named anything except will need the numbers in the override folder to determine override order.)

/usr/share/pkgname/default_config.conf
/usr/share/pkgname/conf.d/10_distributor_override.conf
/usr/share/pkgname/conf.d/99_downstream_override.conf
/etc/pkgname/sysadmin_override.conf
/etc/pkgname/conf.d/10_first_sysadmin_override.conf
/etc/pkgname/conf.d/99_second_sysadmin_override.conf
/run/pkgname/temp_sysadmin_override.conf

Effects¶

Filesystem¶

Benefits
- Simplify base by separating host and distributor files.
- Allow for “factory resets” similar to Android by wiping all directories other than /usr/ and having logic to automatically repopulate directories.
Challenges
- Current Alpine packaging does not align with hermetic-usr.
  - Shifting this will take a long time and a lot of work.
- Most programs read directly from /etc/ instead of following the UAPI spec.
  - A non-exhaustive list of these can be found in this issue.
  - This cannot be fixed distro-side, but would have to be a collaboration with the UAPI people and other distributions to properly implement this in upstreams.

Configuration¶

Benefits
- Alignment with the UAPI Configuration Files Specification.
- Allow easier overriding of Alpine configs for packages using /usr/share/pkgname/conf.d/.
  - Stops relying on “replaces” functionality.
- Allow sysadmins to modify configuration files using /etc/pkgname/conf.d/ without modifying the core system.
Challenges
- Many programs do not follow the UAPI Config Spec yet.
  - See the Filesystem section.
- UAPI Config Spec directories are not standardized, with each distribution acting slightly differently.

Implementation¶

Requirements¶

These are the requirements for implementation of hermetic-usr. They are systemd components, meaning that they will most likely face some backlash against including them in aports/main. If the backlash it too strong, this is not an issue for implementation. (See Contingency Plan.)

systemd-sysusers
- Declaratively create users and groups on boot.
systemd-tmpfiles
- Declaratively handle symlinking or copying configurations.

Plan¶

The implementation plan is split into stages, with each having a summary in bold and a requirement for moving to the next stage also in bold.

The design of this plan is a loop, working on core and essential packages before proposing a simple POC to Alpine and looping back to progress with less essential packages.

All work before the final stage should take place in branches titled “hermetic-usr”. This way, testing can occur without modifying the master branches of our repositories.

Stage 0 (Asynchronous Tasks)¶

Stage 0 is defined as all tasks that can be done while working on later stages. These include anything that will require waiting for a long time or does not make sense to depend on anything.

Define /usr/share/factory/ and its subdirectories in the FHS.

Future stages can be moved to freely, with the exception of the final stage.

Stage 1 (Filesystem Tree and Unofficial Alpine Talks)¶

Stage 1 is defined by the modification of the filesystem tree to support hermetic-usr and unofficially (not through TSC) alerting Alpine to our intentions. This means moving /etc/, /opt/, and /var/ into /usr/share/factory/ and creating tmpfile configs to repopulate them under / at runtime. This should be done using pmbootstrap, with logic implemented for moving the directories at image build-time.

tmpfiles configs will be stored in a package named postmarketos-tmpfiles-configs, with logic similar to the existing systemd-services package as it will create $pkgname-tmpfiles subpackages for Alpine packages containing tmpfiles configuration files.

Programs should not need to be patched yet, as the tmpfile configs should make sure that they find their configurations in the expected TLDs at runtime.

Along with this, we will begin talks with some Alpine developers about hermetic-usr and our plans for it. This does not include a full proposal to the TSC, but instead just speaking with some trusted Alpine developers about the feasibility of Alpine accepting such a proposal and what we would need to change.

The next stage can begin once pmbootstrap logic is working and tmpfiles configs have been made for core packages.

Stage 2 (User and Group Declaration)¶

Stage 2 is defined by creating systemd-sysusers configs to allow simple factory resetting. These configs will be stored in the packages themselves in the “hermetic-usr” pmaports branch.

Alpine packages will be handled by creating a postmarketos-sysusers-configs package, which acts similarly to the previously created tmpfiles config package and creates $pkgname-sysusers subpackages for Alpine packages containing sysusers configuration files.

The next stage can begin once sysusers configs have been created for core packages.

Stage 3 (Patching Programs)¶

Stage 3 is defined by creating a patch-set that implements the UAPI Config Spec for major build-systems and core packages. We will collaborate with AerynOS (formerly Serpent OS) with this as they have a plan to do so and currently implement hermetic-usr.

The patches should be proposed to the respective programs, making sure that we keep good faith and do not push projects that do not want or are unable to accept the UAPI Config Spec. It would be best to ask in a chat room or mailing list about the project’s willingness to accept such patches before work or proposing. If a project is not willing and is an extremely prominent tool, it should be considered if we are willing to maintain a patch set.

This stage does not include creating downstream configs or extensively modifying existing package configs, but instead sowing the seeds for doing so.

The next stage can begin once major build systems and core packages have patches created that we can apply in downstream forks, and these patches have been proposed to the respective upstreams.

Stage 4 (UAPI Config and hermetic-usr Compliance)¶

Stage 4 is characterized by implementing UAPI Config Spec and hermetic-usr compliance into the important packages in pmaports. All core packages will install their files under /usr/ and, if applicable, read their configurations from /usr/share/pkgname/.

These packages should also leverage the patched build-systems to properly set the distconfdir and sysconfdir values. Important aports packages will be forked to follow this pattern as well. (e.g. abuild forked to modify abuild-meson values.)

The next stage can begin once everything from the previous stages is completed.

Stage 5 (Proposing to Alpine)¶

Stage 5 is characterized as the proposal of hermetic-usr to Alpine Linux. This proposal will be used as a source for writing a proposal on the TSC Issue Tracker, citing the work we have done downstream and the patches accepted to various build-systems and programs.

The proposal should include asking Alpine to accept systemd-tmpfiles and systemd-sysusers into aports/main, but should not be a blocker to acceptance. (See the Contingency Plan section for more information.)

The next stage can begin if Alpine accepts the proposal. If not, see the Contingency Plan section.

Stage 6 (Implementing in Alpine and Merging Into pmOS Edge)¶

Stage 6 is characterized by upstreaming packages forked from aports, upstreaming sysusers and tmpfiles configs into aports (if accepted), helping with patching Alpine packages to support hermetic-usr, and merging the hermetic-usr branches into the master branches of our repositories. The usr-merge proposal is a good source for what needs to be done while modifying the filesystem tree of Alpine.

Workarounds, such as the manual moving of directories in pmbootstrap, will be deleted in favor of slowly modifying packages upstream.

This section is purposfully sparse, as a true implementation plan will be drafted for the proposal to Alpine, and anything written here will become outdated during upstream implementation.

Contingency Plan¶

If Alpine does not accept hermetic-usr, work towards achieving this PMCR should halt unless it is decided that working without Alpine’s support is found to be feasible and worth the effort.

If Alpine does not accept systemd-sysusers and systemd-tmpfiles, there are a few things we can do.

If they reject on the grounds of it being systemd, we can propose using sd-tools instead, which are decoupled tools from systemd. If Alpine still rejects those, we can propose creating a custom script that symlinks as tmpfiles would. To make up for losing sysusers, we can create sysusers configs downstream and continue using our custom subpackages.

An alternative if we decide to continue without Alpine would be to add a mode to apk-tools similar to the interactive mode that switches apk-tools from installing outside /usr/ to installing TLDs to their respective factory directories.

FAQ¶

Why package specific directories instead of using `/usr/etc/` like OpenSUSE?¶

While /usr/etc/ is considered a valid directory for configs with hermetic-usr, it loses out on all of the configuration overlay potential stated in the UAPI Config Spec. Along with this, OpenSUSE implemented /usr/etc/ to solve an issue with RPM packaging, not to abide by hermetic-usr.

The Proposer¶

This proposal is a collaboration between Aster Boese (@JustSoup321) and Casey Connolly (@kcxt). Aster is a Trusted Contributor while Casey is a Core Contributor.

This was first proposed by Aster in postmarketOS issue 62 with help from Clayton Craft (@craftyguy) and Casey.

Blocking issues¶

This is a non-exhaustive list of major blocking issues.

Completing the usr-merge in Alpine.

0002: Implement Hermetic /usr¶

Version History¶

1¶

2¶

3¶

4¶

5¶

6¶

7¶

8¶

9¶

10¶

11¶

12¶

13¶