All Systems Go! 2025

A welcome session for All Systems Go!

Linux lacks a coherent security model, and by extension we never defined one for the systemd project either.

In this talk I'd like to start changing this, and begin defining some general security design guidelines that we so far mostly followed implicitly, and make them more explicit. After all, systemd to a large degree is involved in security subsystems, from SecureBoot, Measured Boot & TPM, to its service sandboxing, dm-verity/dm-crypt support, its FIDO2/PKCS#11 hookups, its many security boundaries, secure parameterization, Linux Security Module initialization and more.

While this distributions & applications consuming systemd might follow different security models I think it's important to talk about a unified vision from the systemd upstream perspective, even if various downstreams then make modifications or only deploy a subset of it.

Introduced in kernel v6.7, the Netkit device is an eBPF-programmable network device designed with containers in mind. In this talk, I will go over the the basics of the Netkit device, and discuss the performance gains we have realized and challenges we faced when rolling out Netkit across millions of containers at Meta.

I'll use these 20 minutes to explain why and why contributing to systemd is a great experience. We'll avoid beating dead horses by not discussing git forges and email, but instead focus on the development experience, from building systemd distribution packages from git main, running integration tests against those distribution packages, debugging failures, writing new tests, and installing the distribution packages on real hardware to debug issues.

Running BPF programs today requires CAP_BPF capability, which is an all or nothing BPF capability, and it's ignored in containers anyway.
But BPF nowadays spans a large area, from simple monitoring to potentially invasive fields like network or tracing.

BPF Tokens aims to add fine grained BPF capabilities to systemd units and containers, avoiding to give the whole CAP_BPF capability or even worse running the service as privileged user.

References:
https://lwn.net/Articles/947173/
https://github.com/systemd/systemd/pull/36134

Systing helps you solve problems in minutes rather than days. Out of the box it gives you everything you could possibly need, combined with perfetto’s visualization you will never be confused again.

Communication is paramount in modern application development. This applies equally well to the process of writing applications and to the code itself. The complexity of the tasks ahead of us calls for a distributed and coordinated development effort, and this often manifests in our code: We design distributed, communicating systems to split complexity and responsibility among many people and teams, and at the same time meet the demand for ever faster systems.

The last decade showed significantly increased popularity in API design, network protocols, and distributed computations. At the same time some of the most intriguing language research improves how multi-threaded applications synchronize and exchange information without sacrificing safety or performance. Between these two lies an almost forgotten world: Linux Inter-process communication (IPC) has lost ground to thread-communication and networking protocols.

Let us look at how other operating systems have evolved their IPC layers, what new systems decide to go with, and why Linux IPC has not seen any major changes since the 90s. And finally, can we lift Linux IPC out of stagnation and catch up with everyone else?

Same as every year, a lot has happened in the systemd project since last year's
ASG. We released multiple versions, packed with new components and features.
This talk will provide an overview of these changes, commenting on successes and
challenges, and a sneak peak at what lies ahead.

Haven’t you ever wanted to find ways to make your Rust code the most optimal in the world? I know how you feel. This is a talk, where I’d tell you how easy it is to profile your Rust software and how most often the solutions are trivial.

Let's have an open discussion with systemd developers who are at ASG and users in the audience. We will open with the developers saying what they plan to work on in the near future, and then allow questions / comments from the audience.

Provide a varlink service to access /etc/passwd and /etc/shadow so that no setuid and setgid binaries are necessary for this task.

systemd-confext is a lightweight overlay mechanism for /etc, allowing you to drop in a configuration extension ("confext") bundle and let systemd make it visible to your service as though it was already shipped with the base image. Building on the same extension magic as systemd-sysext, confext also introduces extra features tailored for the /etc use case, such as vpick-ing the newest version and the ability to pick up config revisions with a systemctl reload.

This talk presents the changes to systemd-confext since its debut at All Systems Go! 2023, the lessons learned along the way to make it work, and how we leverage this capability at Microsoft already to deliver configuration payloads in production.

On image based desktops distributions such as Fedora Atomic desktops and Universal Blue, users are expected to run their graphical applications using Flatpaks and their command line ones using containers. But that approach does not work well for some applications that require more privileges, direct access to devices or kernel interfaces.

With systemd system extensions (sysexts), it is possible to extend an image based system on demand. Sysexts come with a lot of advantages: they can be created out of arbitrary content (not only packages), are quickly enabled or disabled and can be built and shared independently of the main distribution channels.

We will demonstrate how the Atomic Desktops can take benefit of sysexts to provide extensions such as virtual machine management (libvirt), alternative container runtimes (moby-engine or docker), IDE (VS Code) or debugging (gdb).

We will also look at important details when building sysexts, the current limits when deploying them (SELinux policy modules, service management, RPM database update), what is currently blocking us from using it for more complex cases (kernel modules) and what we would need to properly manage and update them.

All the big cloud providers provide your machines with a unique cryptographic identity that can be used to talk to their cloud services securely without having to manage or rotate any cryptographic secrets yourself. For example GCP has Service accounts and AWS has IAM roles. This ubiquity of cloud identity and the seamless integration with all the the services of these cloud providers is one of the reasons why they are so successful.

SPIFFE (Secure Production Identity Framework For Everyone) tries to unify these concepts of workload identity in a vendor neutral framework. But how do we bootstrap our cryptographic identity securely when we are running things on our own hardware as opposed to on cloud? What is our bottom turtle?

In this talk, I will show how I use Nix in combination with the swiss-army knife of tools provided by systemd (ukify, systemd-measure, systemd-repart, systemd-veritysetup-generator) to create reproducible images for which we can predict TPM measurements.

Paired with a custom attestation plugin for SPIRE (the reference CA server for SPIFFE) that uses TPM remote attestation I can give each of my servers a unique identity encoded in a TLS certificate if and only if they were booted up with the software that I intended them to boot up with.

This then allows me to have workloads talk to each other with mutual TLS without having to manage any keys or certificates myself.

systemd-sysext provides a nice way to enhance a distribution with a read-only root filesystem without the need to reboot. But there is additional tooling necessary to manage the sysext images:
* install an image which is compatible with the installed OS version
* update installed images to the newest compatible version
* rollback images in case of an OS rollback
* cleanup unneeded images

In this presentation I will talk about which tooling systemd itself provides for this (importctl, updatectl, ...) and what the benefits and disadvantages of this tools are compared with real world use cases. In the end I created an own, generic and distribution independent tool for this using systemd tools in the backend. Using openSUSE MicroOS as example I will demonstrate how we solved the problems with it and how we integrated it in our update stack.

Landlock is an unprivileged kernel feature that enables all Linux users to sandbox their processes. Complementary to seccomp, developers can leverage Landlock to restrict their programs in a fine-grained way. While Landlock can be used by end users through sandboxer tools, there is currently no well-integrated solution to define security policies tailored to system services. Although AppArmor and seccomp security policies can already be tied to a system unit, we aim to provide a more dynamic, standalone, and unprivileged option with Landlock.

In this talk, we'll briefly explain what Landlock is and highlight its differences from other Linux security features (e.g., namespaces, seccomp, other LSMs). We'll then focus on the new configuration format we are designing for Landlock security policies, its characteristics, and how it could extend systemd units by taking into account runtime context (e.g., XDG variables).

See https://github.com/systemd/systemd/pull/39174

For many years, firewall solutions on Linux have grown and evolved, without any major change, until eBPF. While eBPF can allow very fast and efficient packet filtering, the learning curve doesn't make it easily accessible to non-developers. bpfilter aims to bridge the gap between existing tools (nftables, iptables) and modern technologies such as eBPF.

By translating filtering rules into native code, bpfilter abstracts the complexity behind cutting-edge kernel technologies while maintaining backward compatibility with existing solutions. Let's discuss about bpfilter and see it in action!

More than six months have passed since Daan tried to shame gently peer pressure Lennart to actually use the stuff he builds, via a FOSDEM talk:

https://fosdem.org/2025/schedule/event/fosdem-2025-4057-particleos-can-we-make-lennart-poettering-run-an-image-based-distribution-/

Did he succeed? Is dogfooding standard practice now in the systemd development process? Or do things like randomly breaking logging in GNOME (cough) still happen from time to time? Join us for this talk to find out, and to apply yet more peer pressure.

We will also spend some time talking about more boring and mundane topics, such as giving an overview of the current status of ParticleOS, and how we build it as a ready-to-consume and secure-by-default signed and self-enrolling appliance on the SUSE Open Build Service.

https://github.com/systemd/particleos
https://build.opensuse.org/package/show/system:systemd/particleos-fedora
https://build.opensuse.org/package/show/system:systemd/particleos-debian

Many OS artifacts today are still verified using proprietary, stateful keyring formats.
With the "File Hierarchy for the Verification of OS Artifacts (VOA)" an attempt is made to rid the ecosystem of this limitation by implementing a generic lookup directory.
With extensibility in mind, this unifying hierarchy currently provides integration for OpenPGP, with further integrations in planning.

Simplify systemd management with isd! isd is a TUI offering fuzzy search for units, auto-refreshing previews, smart sudo handling, and a fully customizable interface for power-users and newcomers alike.

If you ever became frustrated while typing:

• systemctl start --user unit-A.service (manually starting a unit)
• systemctl status --user unit-A.service (seeing that it failed)
• journalctl -xe --user -u unit-A.service (checking the logs)
• systemctl edit --user unit-A.service (updating the unit)
• (repeat until problem is solved)

isd could help.

In this presentation, we will discuss the features that isd currently supports, the features that are planned for the future, and the experience of developing a TUI for systemd commands.

At Meta, we've been looking to revamp our internal container runtime (Twine). Instead of maintaining all the low level container runtime code ourselves, we'd much prefer having more of this managed by systemd. This talk will go over what we did to make systemd transient units a suitable environment for running system containers (pid namespace support, cgroup namespace support, namespace delegation, ...), and why we went this route instead of reusing systemd-nspawn.

Marco will review the features available in the initramfs-tools ecosystem, the initrd generator used by Debian and Ubuntu, and how they can be implemented (or not) by adopting mkosi-initrd.

oo7-daemon is the new D-Bus Secret Service provider that aims to fully replace gnome-keyring. In this followup (continuation of my 2024 talk) lightning talk, I will go through the progress made, the challenges faced and the status of systemd credentials integration.

The social event will take place, once again, at Mein Haus am See from 19:00-23:00. Food will be served and drink tokens will be handed out at the door.

19:00-23:00 - Food and drinks on the ground floor with full venue access.

You can stay after 23:00 but from that point there is no official All Systems Go! event and you're on your own. ;)

Flatpak is the de-facto standard for distributing desktop applications across various Linux based systems. It also offers other advantages such as sandboxing. It is particularly useful for image based systems as it installs the applications into a separate location and doesn't try to modify the system.

GNOME OS is GNOME's development, testing and QA operating system. It builds the latest and greatest in-development versions of the GNOME desktop and core applications. It is also Linux based system that tries to fully embrace the systemd ecosystem.

The applications are however built into the system. While this might be great for testing the apps as they would be in most distros, we also want to build our Flatpak applications from the same build definitions and our users (or more correctly early adopters) prefer to use Flatpak for various reasons.

In this talk we'll explore what other image based distributions do to provide Flatpak applications to their users, what users expect from "Flatpak applications" and the various proposals for implementing that in GNOME OS. We hope to be able to present our end result by the time of the conference.

Many traditional container engines make use of the "subuid" concept and the "newuidmap" tool to implement a concept of "unprivileged" user-namespace containers on Linux. This approach has many shortcomings in my PoV, from both a security and scalability standpoint.

Recent systemd versions provide a more powerful, more secure, mor scalable alternative, via systemd-nsresourced, systemd-mountfsd and other components.

In this talk I want to shed some light on the problems with the "old ways", and in particular focus on what the "new ways" bring to the table, and how to make use of them in container runtimes.

GNOME OS is a distribution based around systemd-sysupdate. This year, we finally created a live installer image using the same /usr partition as the installed OS. The main innovation however is the ability to install without the need to reboot. The user can start working while the installation is happening.

This live image is built using systemd-repart. And the installer itself also uses systemd-repart. But systemd-repart is not the complete solution and we had to solve some challenges.

It has been 10 years since Axis Communications had a presentation at the systemd conference. Back then, we have shown how we have increased our product quality, stability and boot times by porting our platform to systemd. 10 years later, we had different challenges to keep the resource usages and boot times under control. We have started from bottom up and sliced our software for this purpose. This work also got us inspired to create virtual versions of our hardware products that we cluster deploy using systemd's nspawn.

GNOME has collected some very old code over the years. During the recent GNOME 49 release, we've made some drastic cleanups. Most visibly, we've dropped support for X11 and gained many dependencies on systemd. Let's explore some of the what and why for these changes!

Coredumping on Linux has long been a nightmare. Currently two modes are supported:

(1) Dumping directly into a file somewhere on the filesystem.
(2) Dumping into a pipe connected to a usermode helper process spawned as a child of the system_unbound_wq or kthreadd.

For simplicity I'm mostly ignoring (1). There's probably still some users of (1) out there but processing coredumps in this way can be considered adventurous especially in the face of set*id binaries.

The most common option should be (2) by now. It works by allowing userspace to put a string into /proc/sys/kernel/core_pattern like:

    |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h

The "|" at the beginning indicates to the kernel that a pipe must be used. The path following the pipe indicator is a path to a binary that will be spawned as a usermode helper process. Any additional parameters pass information about the task that is generating the coredump to the binary that processes the coredump.

In the example core_pattern shown above systemd-coredump is spawned as a usermode helper. There's various conceptual consequences of this (non-exhaustive list):

• systemd-coredump is spawned with file descriptor number 0 (stdin) connected to the read-end of the pipe. All other file descriptors are closed. That specifically includes 1 (stdout) and 2 (stderr). This has already caused bugs because userspace assumed that this cannot happen (Whether or not this is a sane assumption is irrelevant.).

• systemd-coredump will be spawned as a child of system_unbound_wq. So it is not a child of any userspace process and specifically not a child of PID 1. It cannot be waited upon and is in a weird hybrid upcall which are difficult for userspace to control correctly.

• systemd-coredump is spawned with full kernel privileges. This necessitates all kinds of weird privilege dropping excercises in userspace to make this safe.

• A new usermode helper has to be spawned for each crashing process.

On recent kernels a new mode has been added making use of AF_UNIX sockets. This talk will talk about the new modes, how they can be used, what advantages they have in comparison to the other modes, and look at technical implementation details.

CentOS Stream is especially suited for production deployments. In these environments it's often common to develop improvements to distribution packages and want to contribute them upstream. Unfortunately, until very recently that required one to then maintain their own build and deployment pipeline for the packages, at least until the changes made their way into the distribution.

CentOS Proposed Updates (CPU) SIG aims to bridge this gap - changes that have been submitted as merge requests can be built in this SIG, providing those who run Stream in production with access to needed updates while they make their way into CentOS Stream. We hope this will help increase collaboration between RHEL engineers, CentOS Stream contributors, and the rebuild community as well, especially those that have distributions derived from CentOS Stream directly (such as AlmaLinux with AlmaLinux OS Kitten), as everyone can focus on making improvements without reinventing their own build pipelines.

Going for minimal containers with restricted system calls and unprivileged users is the usual Kubernetes approach these days, and it works great for most web apps. However, the development of more complex infrastructure extensions frequently hinders application functionality.

While looking for a solution to deploy virtiofsd in an unprivileged container for KubeVirt, we stumbled on seccomp notifiers. Seccomp notifiers are a kernel feature which monitors syscalls and get notifications to a userspace application when a syscall is executed.

Alternative options involved either the use of a custom protocol using UNIX sockets or the deployment of virtiofs as a privileged component alongside the unprivileged VM.

After our evaluation, the seccomp notifier turned out to be the simplest solution among all the choices. Unfortunately, the main constraint is the monitor's resilience after a restart, such as after a crash or an upgrade. This limitation forced us to back up to one of the less elegant approaches. But there is hope how this could be solved!

The session will explain why seccomp notifiers are a lean solution to avoid extra userspace communication and synchronization, the current limitations and possible future solutions to overcome today’s challenges.

Last year, Lennart Poettering of the systemd fame, gave a presentation at this very same conference, where he introduced Varlink, a modern yet simple IPC mechanism. He presented a case for Varlink, rather than D-Bus to be the future of Inter-process communication on Linux. As someone who works on D-Bus, I took upon myself to prove him wrong, only to find out that I achieved exactly the opposite.

It didn't take long before I got convinced of his vision. Since I was largely responsible for giving the world an easy to use D-Bus Rust library, I thought it's only fitting that I do the same for Varlink. This talk will be the story of the creation of such a library, the challenges I faced, where Varlink fits the Rust idioms really well and where it does not and how all of this affected the development and the API.

File descriptors for processes on Linux have been available for quite some time now. Userspace has adapted them widely.

Over the last two years or so we've extended the abilities of pidfds significantly. This talk will go over all the new features and deep dive into their implementation and usage.

In the Linux world there are several tools and technologies to encrypt data on a hard drive, most falling into one of two categories: block device encryption (like LUKS) or stacked filesystem encryption (like EncFs or gocryptfs). This presentation will introduce Dirlock, a new tool that belongs to a third category: native filesystem encryption, using the kernel's fscrypt API. Dirlock is currently being developed and its aim is to provide a flexible way to encrypt files, suitable for both user accounts and arbitrary directories, with full PAM integration, support for hardware-backed mechanisms such as FIDO2 or TPM and with a D-Bus API for easy management.

Traditional package updates using tools like RPM or Zypper can introduce risks, such as incomplete updates or accidentally breaking the running system. To overcome these challenges, we developed container-snap, a prototype plugin designed to deliver atomic OS updates—updates that are fully applied or rolled back without compromising the system's state.

container-snap leverages OCI images as the source for updates and integrates seamlessly with openSUSE’s tukit to enable transactional OS updates. By utilizing Podman’s btrfs storage driver, it creates btrfs subvolumes directly from OCI images, allowing systems to boot from the OCI image. This approach empowers users to construct their own OS images using familiar container image-building tools, like Docker or Buildah.

In this session, we’ll dive into:
- The architecture and technical implementation of container-snap
- Challenges encountered during development and how we resolved them
- Key lessons learned along the way
- A live demo showcasing container-snap in action

Come and join this session to learn more about how to boot from an OCI image without bricking your system!

Introduce ue-rs, a fresh project that aims to be a drop-in reimplementation of update engine, written in Rust.

In last year's ASG!, bootc and bootable containers were introduced. In this talk, we'll go over what changed since last year, and how Fedora CoreOS and RHEL CoreOS are leveraging bootable containers to reduce maintenance and increase sharing.

S3NS is a trusted cloud operator that self-hosts Google Cloud infrastructure in France, targeting the SecNumCloud certification, the most stringent Cloud certification framework. SecNumCloud includes strict legal and operational constraints.

To manage these systems securely and reproducibly, we’ve built a family of dedicated administration terminals based on the image based philosophy.

These terminals rely on NixOS semantics and draw from the ParticleOS ecosystem: systemd-repart, and dm-verity, ensuring atomic updates, full immutability of the Nix store, and verifiable integrity of the boot chain and runtime system (measured boot), while using remote attestations by TPM2 when connecting to production assets.

We will present the purpose of these terminals and what needs they serve along with their high level characteristics: partition layouts, provisioning and connection flow to the production assets.

This talk will show an application of many of the concepts that were presented in the NixOS ecosystem and in All Systems Go itself by the systemd community.

With Bootable Containers (bootc), we can place the operating system files inside a standard OCI container. This lets users modify the content of the operating system using familiar container tools and the Containerfile pattern. They can then share those container images using container registries and sign them using cosign.

Using composefs and fs-verity, we can link a UKI to a complete read only filesystem tree, guaranteeing that every system file is verified on load. We integrate this in bootc by creating a reliable way to turn container images into composefs filesystem trees, and then including the UKI in the container image.

We will share the progress on the integration of UKI and composefs in bootc and how we are going to enable remote attestation for those systems using trustee, notably for Confidential Computing use cases.

test.thing is a VM runner which targets guests using an API defined by systemd. It started after a conversation at devconf about turning mkosi qemu into a library. A quick intro.

composefs is an approach to image-mode systems without the disk images. Files are stored in a de-duplicated content-addressed storage with integrity guaranteed through fs-verity. The last year has seen an acceleration of development on composefs-rs, a pure Rust implementation of the ideas behind composefs. Our goal is unification of the storage of bootable system images (via bootc), application Flatpaks, and traditional OCI container environments, bringing deduplication and integrity guarantees to all three. An overview.

Updates are a critical piece of managing your fleet of devices. Nowadays, Yocto-based distributions can utilize layers for well-established update mechanisms. But, did you know that recent releases of Yocto already come with a simple update mechanism?

Enter systemd-sysupdate: a mechanism capable of automatically discovering, downloading, and installing A/B-style updates. By combining it with tools like systemd-boot, we can turn it into a
comprehensive alternative for common scenarios.

In this talk, we will briefly introduce systemd-sysupdate, show how it can be integrated with your Yocto distribution, and share thoughts on how it can be improved further.

I overview how OS management is done at Meta. We run millions of Linux servers and we have to make sure that OS gets updated on all of them in a given period of time. To do that we developed several products: MetalOS (Image based version of CentOS), Antlir (image builder) and Rolling OS Update (a service that keeps a set of DNF repos in sync with upstream repos and uses them to update OS )

The UAPI Boot Loader Specification defines conventions that let multiple operating systems and bootloaders share boot config files. So far, only systemd-boot implements it - and it’s UEFI-only by design.

As a result, hybrid UEFI/BIOS images require maintaining (and keeping in sync) two sets of bootloader configs: one for systemd-boot, and one for a legacy bootloader such as syslinux.

I set out to fix that by building a BIOS bootloader that uses the UAPI Boot Loader Specification - allowing both UEFI and legacy boot to use a single shared set of config files. This talk is about why that matters, how I built it, and what comes next.

Closing session of All Systems Go! 2025