To see our schedule with full functionality, like timezone conversion and personal scheduling, please enable JavaScript and go here.
09:25
09:25
5min
Opening session of All Systems Go! 2024

A welcome session for All Systems Go!

Main Hall
09:30
09:30
40min
Reproducible and Immutable OS Images with NixOS
Moritz Sanft

Many consider NixOS a great tool for declarative definition of their OS, but only few know about its capabilities for Image-based Linux. NixOS offers the tools to combine modern technologies such as discoverable disk images (DDIs), unified kernel images (UKIs), and TPM-based measured boot for transforming declarative configurations into security-focused and immutable OS images for both the server and the desktop.

This talk showcases how we build such reproducible and immutable DDIs with NixOS, and how ukify, systemd-repart, dm-verity and measured boot are involved in that process. We will also briefly cover the support of SecureBoot in NixOS through the Lanzaboote project, and what else is yet to come for image-based NixOS.

Main Hall
10:15
10:15
25min
Introducing azure-init, a minimal provisioning agent written in Rust
Dongsu Park

Azure-init is a fresh open source reference implementation for provisioning Linux virtual machines in Azure. In contrast to existing systems like cloud-init, azure-init aims to be minimal, focusing on basic instance initialization from Azure metadata. Azure-init also consists of a flexible structure to enable its use by other provisioning agents like Fedora CoreOS’ Afterburn. Finally, azure-init aims to be fast and secure, being written in Rust. In this talk we will review the motivations for the creation of azure-init, the current status of the project, and vision for its future development.

Dome
10:15
25min
Using Trusted Platform Modules (TPMs) at scale for protecting keys
Jonathan McDowell

Meta runs a large production fleet of servers, all making extensive use of TLS for inter-host communication. As part of a general approach of securing keys against exfiltration a project has been undertaken to make use of existing TPM chips to provide secure storage for high privilege private keys. This talk will touch upon the approach taken to allow for the use of a hardware backed key without compromising performance, but mostly focus on the software infrastructure that needed to be built to provision and monitor TPM health across the fleet (a prerequisite for confirmation of viability).

Main Hall
10:45
10:45
25min
Enhancing Security with Systemd: Secure Web Tokens and TPM 2.0
Philipp Deppenwiese

As the digital landscape evolves, ensuring robust security measures becomes paramount. In this talk, we will explore the implementation of a new systemd service designed to enhance secure web token management through TPM 2.0 and FIDO2 support. This integration facilitates seamless interaction with the xdg-credentials-portal, aiming to provide a straightforward and secure approach to handling credentials.

Main Hall
10:45
25min
Improving bpftrace reliability
Daniel Xu

bpftrace is a popular and powerful dynamic tracer for Linux systems. In the vast majority of uses cases, bpftrace does its job quickly, efficiently, and accurately. However with the rapid increase of users, use cases, and features, the bpftrace community has started to feel (technical) growing pains. In particular, we've started to uncover various reliability issues. In this talk, we will cover what is already done as well as what is currently broken and how we will systematically fix and prevent these issues from re-occuring.

Dome
11:15
11:15
40min
Full Disk Encryption in openSUSE MicroOS and Tumbleweed
Alberto Planas Dominguez

The openSUSE project has been looking for a Full Disk Encryption (FDE) solution since long ago. After some iterations we are converging in a systemd based solution. This talks will present the alternatives and will focus in the current proposed solution based on systemd-pcrlock.

Main Hall
11:15
40min
eBPF Data Collection for Everyone – empowering the community to obtain Linux insights using Inspektor Gadget
Alban Crequy, Michael Friese

In this presentation we show how eBPF programmers can easily distribute their programs using Inspektor Gadget, a tool designed for the creation, deployment, and execution of eBPF programs (gadgets) across Kubernetes and Linux environments. Inspektor Gadget encapsulates eBPF programs into OCI containers, providing well-understood and easily distributable units. We then detail how an end user can use Inspektor Gadget to easily derive valuable systems insights.

Dome
11:55
11:55
40min
Fixing an old Linux process memory security bug
Adrian Ratiu

There is a well-known trade-off between security lockdowns and a user's abiliy to
debug/inspect a system. The Linux kernel is finally fixing an old proc/mem security
bug which illustrates this trade-off nicely. The kernel will provide a mechanism,
so distros need to implement a policy according to their own security needs, to
restrict proc/mem access (it gives userspace RW access to processes memory).

This talk goes into the what, why and how of getting this bug fixed, with some policies
for plugging the long-standing hole for different use-cases, without breaking
debuggers or container supervisors.

Dome
11:55
40min
The road to a trusted and measured boot chain in Bootable Containers
Timothée Ravier, JB Trystram

Fedora image based variants (CoreOS, Atomic Desktops, IoT) are currently built using ostree and rpm-ostree. This enables an hybrid approach where the system is managed like an image but modifications are still possible using RPMs.

But this approach has limits:
- It is difficult for users to customize their operating system and share those customizations.
- The integrity of the boot chain is not guarenteed and it is costly to validate the system content at runtime.

To address those shortcomings, we are introducing the bootable containers (bootc) project. With bootable containers, the content of the operating system, including the kernel and initrd (or a UKI) is shipped in a container image alongside its corresponding base userspace root filesystem. This image can then be modified using container native tools and shared via a container registry.

To chain from platform Secure Boot to a verified root filesystem, the ostree project has integrated support for composefs. It combines multiple Linux kernel features (overlayfs, EROFS and fs-verity) to provide read-only mountable filesystem trees stacking on top of an underlying "lower" Linux filesystem.

We will detail how we are integrating composefs and UKI support in Bootable Containers to enable a trusted and measured boot chain while letting users customize and re-sign their images to fit their needs.

Main Hall
12:40
12:40
25min
Lessons learned from migrating AI accelerator drivers from the kernel to userspace
George Utsin

The kernel driver is dead; long live the userspace driver! In this talk, we’ll discuss the motivation, challenges and outcomes of migrating drivers for Meta’s AI accelerator chips from the kernel to userspace.

Dome
12:40
25min
systemd 255 in Tizen, or how we have paid our technical debt and took another one
Łukasz Stelmach

In this talk we present the story of upgrading systemd in Tizen by eleven releases. We share both the lessons we've learnt during the most recent upgrade as well as decade long experience of the maintenance and development of key packages in the only GNU/Linux distribution that uses kdbus. We describe our day-to-day git workflow as well as upgrade procedures we came up with over the years.

Main Hall
13:05
13:05
90min
Lunch break
Main Hall
13:05
90min
Lunch break
Dome
14:35
14:35
40min
Fort Kairos: A New Dawn for Secure Linux in Untrusted Environments
Mauro Morales

At the edge, there's one thing we know for sure: it's not to be trusted. But imagine if Kairos could change that, letting you sleep soundly knowing your intellectual property is secure.

Kairos is a fully open source project to run kubernetes at the edge. As such, we have put Trusted Boot into action. Inspired by Lennard Pottering, the mind behind Systemd, we've leveraged Secure Boot, Trusted Boot, TPM, and disk encryption. The result? A Linux OS that's built tough against the challenges of untrusted environments.

Dome
14:35
40min
Platform security in NixOS
Ryan Lahfa, Niklas Sturm

You may have heard about this weird distribution, NixOS, that breaks compatibility with /usr.
This talk explores the properties inherent to NixOS, focusing on its distinct approach to package management and system configuration. Learn how these principles combine with general upstream efforts at bringing TPM2, Secure Boot and more to your Linux distribution.

Main Hall
15:20
15:20
40min
bootc: Generating an ecosystem around bootable OCI containers
Ben Breard, Colin Walters

Containers have become the de facto choice for deploying most applications, and all of us benefit from the isolation, portability, and the surrounding ecosystem. In this talk we’ll take a deep dive into the world of bootable containers, using the same ideas, goals and technology for the host system (whether virtualized or bare metal). We’ll look at the bootc project under the github.com/containers umbrella and its current flagship distribution usage in the new Fedora/CentOS bootc project and initiative. We hope you are as excited as we are by taking cloud-native approaches down to the operating system level, and a key goal is finding points that can be shared with other components of the ecosystem, from the uapi-group.org to other container-based OSes.

Dome
15:20
40min
systemd & TPM in 2024
Lennart Poettering

An update on systemd's TPM features, i.e. what happened since last year, i.e. systemd-pcrlock, NvPCRs, and Varlink APIs.

Main Hall
16:00
16:00
30min
Coffee break
Main Hall
16:00
30min
Coffee break
Dome
16:30
16:30
25min
An extendable and securely signed image-based OS with updates
Marius Schiffer

With systemd tooling, including mkosi, it is possible to build an OS image that fulfills all checkmarks a modern image-based OS should have, but with a standard off-the-shelf distribution!
This talk gives an overview for a possible workflow, including A/B updates and offline signed images and updates, in real-use. As a bonus, it is also self-replicating and uses as little configuration as possible, leveraging built-in systemd auto detection.

Dome
16:30
25min
Portable software bills of materials with Nix and systemd portable services
Julien Malka

While software bills of materials become of increasing value to further trust in the software supply chain, generating high quality SBOMs still poses some challenges in some ecosystems due to the lack of proper tooling or accessible build metadata. In this talk, I'll explain and demonstrate how we can leverage the static dependency graph of functional package managers like Nix to generate very precise SBOMs, that can be relevant for running a service on any linux distribution thanks to systemd portable services.

Main Hall
17:00
17:00
25min
Creating Arch Linux images using mkosi
Jelle van der Waa

Arch Linux creates 2 cloud images, 2 vagrant images every month using custom bash scripts and requiring root for building. This talk will look at how these images can be created using mkosi, building them in CI, testing the build images and as a bonus; build reproducible?

Project link: https://gitlab.archlinux.org/archlinux/arch-boxes

Dome
17:00
25min
Waiter, an OS please, with some sysext sprinkled on top
Mathieu Tortuyaux, Timothée Ravier

On general purpose image based systems such as Flatcar and Fedora CoreOS, users are encouraged to run all their applications using containers. To make updates safe and predictable, the system is mounted as read only and local modifications are discouraged.

While containers offer a lot of flexibility on Linux, there are still cases where installing binaries or running applications directly on the host operating system is preferred. For example to add kernel modules, use an alternative container runtime version, add more udev rules, etc.

Some of those use cases could be addressed with statically linked binaries, but their management is manual and their usage creates new issues around updates, versionning, memory footprint and not everything can be statically compiled. Alternatively, one can build its own image but at non-negligeable maintenance costs.

Systemd's system extensions (sys-ext) provide a mechanism to extend the content of the host while preserving the safety guarentees around updates. We will demonstrate how Flatcar, Fedora CoreOS and Atomic Desktops are leveraging sysext images to securely extend the OS. With practical examples and usecases (e.g Cluster API) learn how to install Python, Podman, Kubernetes, ZFS, everything at the same time, by composing your very own image with systemd-sysext.

Main Hall
17:30
17:30
25min
Rediscovering systemd Portable Services
Luca Boccassi

systemd introduced Portable Services support in 2018, as part of v239. This feature was covered at ASG 2018 and in a blog post published at the time: https://0pointer.net/blog/walkthrough-for-portable-services.html
But a lot has changed in the past 6 years, and very crucial new features have been introduced, so it is time to have another look at this topic and see what has happened in the meanwhile, what new use cases have opened up, and what is coming in the near future.

https://systemd.io/PORTABLE_SERVICES/

Main Hall
17:30
25min
interacting with systemd from high level languages
Alvaro Leiva Geisse

Probably the way systemd is thought of and used is mostly as a service manager, and a collection of tools built around the idea of “low level user space”. We rarely think of it as a library that can be used as part of any high level language or application. This talk will cover this aspect of systemd, and through the lens of pystemd, explore how applications can use (and abuse) systemd.

Dome
18:00
18:00
5min
Efficient RAUC Updates using composefs
Jan Lübbe

A quick overview of how RAUC uses libcomposefs to handle new use-cases.

Main Hall
18:05
18:05
5min
oo7-daemon + systemd per-user credentials
Dhanuka Warusadura

oo7-daemon (a temporary name based on the oo7 client library) project aims to provide a replacement for the gnome-keyring-daemon as the new D-Bus Secret Service provider in the GNOME desktop environment. In this talk I will go through the latest development plans and the progress made to integrate TPM backed credentials support to oo7-daemon using systemd per-user credentials as a backend.

Main Hall
18:10
18:10
5min
Debian, empty /var/, empty /etc/ and factory reset
Marco d'Itri

This presentation will review how far Debian (and more generally, traditional distributions) is from supporting factory reset: what can work, what is missing and possible hacks^Wways to do it without starting a distribution-wide effort.

Main Hall
18:15
18:15
5min
Integration testing environment for mixed HPC and cloud workloads

Integration testing environment for mixed HPC and cloud workloads

Main Hall
18:20
18:20
5min
A new way to develop on immutable Linux

A new way to develop on immutable Linux

Main Hall
18:25
18:25
5min
Ideas for improving systemd-boot

Ideas for improving systemd-boot

Main Hall
09:30
09:30
25min
Reproducible Builds at Sidero Labs: Tools and Techniques
Utku Özdemir

Ensuring consistent and secure software builds is crucial in today's cloud-native environments. At Sidero Labs, we've developed a comprehensive approach to reproducible builds for Talos Linux using a variety of tools and techniques. This talk will explore our use of Docker Buildx, Kres, and other key components that contribute to our build system. We'll share insights into our methods, challenges faced, and solutions implemented, providing practical guidance for developers aiming to achieve reproducibility in their own projects.

Dome
09:30
25min
systemd: state of the project
Luca Boccassi, Zbigniew Jędrzejewski-Szmek

Same as every year, a lot has happened in the systemd project since last year's
ASG. We released multiple versions, packed with new components and features.
This talk will provide an overview of these changes, commenting on successes and
challenges, and a sneak peak at what lies ahead.

Main Hall
09:55
09:55
25min
systemd: round table
Zbigniew Jędrzejewski-Szmek, Luca Boccassi, Lennart Poettering, Mike Yuan, Yu Watanabe

Let's have an open discussion with systemd developers who are at ASG and users in the audience. We will open with the developers saying what they plan to work on in the near future, and then allow questions / comments from the audience.

Main Hall
09:55
25min
using io_uring for storage
Nils Goroll

A brief report about how we use io_uring in SLASH/fellow https://gitlab.com/uplex/varnish/slash, an always consistent, eventually persistent storage engine for Varnish-Cache. (FOSS, LGPL)

Dome
10:20
10:20
25min
Varlink Now!
Lennart Poettering

Why bother with Varlink IPC, and why now?

Main Hall
10:20
25min
libpathrs: securing path operations for system tools
Aleksa Sarai

Container runtimes and other privileged system management tools have historically struggled with safely operating on a path within a directory tree controlled by a malicious user. libpathrs is a library which makes it easy to do said path operations, as well as providing some other safe path-related utilities such as providing safe wrappers to operate on procfs files in a safe way.

Dome
10:50
10:50
40min
Booting an embedded system like a PC
Manuel Traut

This shows how to boot an mkosi generated arm64 Debian Image with UKI and systemd-boot on a u-boot based EFI firmware with a fTPM as a Trusted-Application in OP-TEE

Dome
10:50
40min
systemd-ifying postmarketOS, our immutable future, and why Alpine is cooler than you thought
Caleb Connolly, Clayton Craft

postmarketOS was started with the lofty goal of enabling long term support for
mobile phones and other devices with traditionally short lifespans, and doing so
outside of the Android walled garden. This has inevitably resulted in a lot of
upstream focused hardware bringup and development. Join us and learn what
our community have been building, how we're running systemd on Alpine Linux
and what we see in the future for postmarketOS.

Main Hall
11:35
11:35
40min
Building Secure Container Images for the Cloud with Yocto
Frank Vasquez

Yocto is a tool for building custom Linux distros. When you think about it, a container image is just a custom Linux distro. The distro (e.g. Alpine) is your base image and the customizations are the rest of your application or microservice. Like Podman, Yocto can generate a complete root filesystem in the form of an OCI container image. Originally targeted at bare metal, the Yocto configuration and build process seems complex when compared to the Containerfile approach of cloud native tools. Yocto's OpenEmbedded origins also mean that reduced image size, SBOM generation, license compliance, and reproducible builds were concerns early on in the project rather than afterthoughts. With security and risk of litigation now top of mind, this talk explains Yocto's uniquely layered and ultimately monolithic approach to solving these real-world software problems.

Dome
11:35
40min
Integrating systemd soft-reboot into a distribution and surviving it
Thorsten Kukuk

In this talk, I will discuss how Linux distributions can integrate and benefit from using systemd soft-reboot. Using openSUSE Tumbleweed as an example, I will show where and how it makes sense for traditional Linux distributions to use it and where the pitfalls are. With openSUSE MicroOS, we have a distribution with a read-only root file system that particularly benefits from a soft-reboot because a reboot is necessary after every update in order to change the root file system. However, this also requires special measures to ensure that it always functions smoothly.

Afterwards I will talk about the requirements and solutions for services to survive a soft reboot and what's necessary to make the whole thing supportable.

Main Hall
12:20
12:20
40min
Removing Cloud Providers From the Zero Trust Equation
Fabian Kammel

This presentation introduces a novel approach to enhance the trust in SPIFFE by leveraging confidential computing technologies, specifically Confidential Virtual Machines.
The presentation will provide an introduction to the realm of confidential computing, as well as an overview of SPIFFE/SPIRE. Armed with this knowledge we will demonstrate a practical example that integrates the AWS Instance Identity Document plugin with AMD SEV-SNP, showcasing the implementation challenges and solutions.

Dome
12:20
40min
What's your PID 1 up to?
Cooper Ry Lees

How do you continually test and release new versions of systemd with confidence? Also, once released, how do you monitor PID 1 itself and your PID 1 usage across your server fleet? This talk dives into Meta’s way of answering these questions so we can minimize the risk of breaking changes and fun each systemd release brings us. Some of the technology in the talk is OSS, so you too, can join in on the fun knowing how your systemd usage is across your own infrastructure!

Main Hall
13:00
13:00
90min
Lunch break
Main Hall
13:00
90min
Lunch break
Dome
14:30
14:30
40min
Avocado Linux: Highly Secure Accelerated Embedded Development Platform for (A)IoT
Justin Schneck

Developing embedded products often involves a trade-off between robust security and accelerated development. Production environments, while offering high security and immutability, can inhibit rapid development cycles. Conversely, sandbox environments provide the flexibility and integration needed for fast development but are not suitable for production deployment. The transition between these two environments is typically fraught with challenges, consuming significant time and effort.

This talk introduces Avocado Linux, a highly secure, image-based operating system and layer repository with deeply integrated developer tools. Avocado strikes a perfect balance between flexibility and immutability, combining the best of both worlds, accelerating time to market without compromising on security.

By leveraging innovative systemd features like System Extensions, Configuration Extensions, and Portable Services, Avocado Linux provides a robust framework for service management, process isolation, and secure, atomic updates. Its design ensures robust security and system integrity, with comprehensive use of dm-verity and mechanisms for recovery and factory reset, safeguarding device data integrity even in the face of unexpected failures.

Join us to explore how Avocado can transform your embedded systems development with faster integration, enhanced reliability, and seamless composability. Discover how this distribution delivers significant business value by enabling rapid deployment, maintaining security, and ensuring system integrity. Learn how Avocado abstracts away the complexities of system development, allowing your team and applications to thrive and your embedded product to scale and succeed.

Main Hall
14:30
40min
busd: There is a new D-Bus broker in town
Zeeshan Ali Khan

D-Bus is an IPC mechanism that is very ubiquitous on Linux systems everywhere (desktop, cloud and embedded). It is the mechanism you'd use to communicate with many of the core Linux userspace subsystems, such as systemd, NetworkManager etc. Traditionally, most of these services have been written in C, a language known for its lack of safety and expressiveness.

In the past years, Zeeshan has developed a library, called zbus for enabling implementation of D-Bus services and clients in a programming language designed for safety: Rust. zbus has become the go-to library for writing D-Bus code in Rust. While that is major step forward, the communication typically still happens through a broker and the two major broker implementation are both are written in C and have been stagnating for years.

This is why Zeeshan has recently started working on writing a D-Bus broker based on zbus, called busd, which not only aims provide a drop-in replacement for existing brokers, but also modernize the D-Bus space by providing new features needed by apps and services, such as systemd.

In this talk, Zeeshan will walk us through a summary of his journey so far, the current state of busd and his plans and dreams for the future of D-Bus.

Dome
15:15
15:15
40min
Boring infrastructure: Building a secure signing environment
David Runge

Many Linux distributions rely on cryptographic signatures for their packages and release artifacts. However, most of the used signing solutions either do not rely on hardware backed private key material or are run in untrusted environments.

This presentation will provide a general overview of the Signstar project, which is currently under development by Arch Linux to provide a generic signing solution based on a Hardware Security Module (HSM).

Main Hall
15:15
40min
SSH authentication using user and machine identities
Morten Linderud

Strong authentication requires multiple signals: identity claims proves that identity of the person, while device attestation proves possession of a given machine, and device bound keys prevent the key from being stolen.

In this presentation we will take a look at how the TPM provides device attestation and device bound keys. We will connect this with identity claims from SSO providers to provide a centrally managed short-lived SSH certificates for users and their devices. This is implemented as an open-source project called “ssh-tpm-ca-authority”.

Dome
16:00
16:00
30min
Coffee break
Main Hall
16:00
30min
Coffee break
Dome
16:30
16:30
25min
Improving systemd’s integration testing infrastructure
Sam Leonard

The Sovereign Tech Fund paid Codethink to help improve the integration testing infrastructure of systemd. This talk covers how the integration test suite used to work and what it does now.

Main Hall
16:30
25min
Installing your OS with systemd-repart
Adrian Vovk

There's a new installer for GNOME OS, and it's built on top of systemd-repart. Here's how and why we did it

Dome
17:00
17:00
25min
GNOME OS + systemd-sysupdate
Abderrahim Kitouni, Jude Onyenegecha

As a reference for developers and testers, GNOME OS is an experimental Linux distribution that ships the latest in-development GNOME desktop, core applications, and stack.

GNOME OS is currently using OSTree, this talk covers the ongoing work to add features to systemd-sysupdate and transition to it. Features like optional transfers, delta updates, and major version upgrades.

Dome
17:00
25min
Successes and struggles using the systemd user instance in developer environments
Colin Chan

This talk will explore several of the ways we've leveraged the systemd user instance in our developer environments at Meta, challenges we faced while doing so, and how we worked around those challenges.

Main Hall
17:30
17:30
25min
Home Directory Encryption in GNOME
Adrian Vovk

Thanks to work made possible by the STF grant, all the pieces are there for GNOME to integrate with systemd-homed. This talk describes what it took to get here, what new features it gives us, what still remains to be done

Dome
17:30
25min
mkosi-initrd: initrds built from system packages
Zbigniew Jędrzejewski-Szmek

mkosi-initrd is a project to build initrds from normal system packages (rpms, debs). Initially separate, it now is part of mkosi — just another build stage. systemd uses mkosi for automated tests, and this now includes building an initrd and booting a VM with it, so such initrds are getting fairly wide testing, albeit in fairly narrow circumstances. The process of adoption of mkosi-initrd in distributions has been slow, but with an implementation natively in mkosi, the technical base is really good. What remains to be done to make this the default approach? Can Fedora 41 finally make this an option for users?

Main Hall
18:00
18:00
25min
Can systemd-resolved replace Avahi?
Abderrahim Kitouni

Multicast DNS (mDNS) and DNS Service Discovery (DNS-SD), collectively know as zeroconf, are technologies used for devices to find each other and advertise services on the local network.

There are two widely used FOSS implementations: mDNSResponder is used by Apple and Android, while Avahi is used by most GNU/Linux distributions. However, there is a third one in systemd-resolved -- widely installed but rarely used.

In this talk, I will explain how mDNS and DNS-SD work individually and together, and explore how to use them with resolvectl. I'll also try to go over the deficiencies in the systemd-resolved and have a discussion about the ways that it can be improved to replace Avahi as the default implementation on GNU/Linux systems.

Dome
18:00
25min
initrd performance improvements
Benjamin Drung

Every second spent on waiting for a system to boot is wasted time. In this talk I present the steps we took in Ubuntu to speed up the boot and the initrd generation time. The presented improvements are not specific to Ubuntu and can be ported to other implementations (like dracut) to benefit other distributions as well. The talk will present further speed improvements that can/will be implemented in the future. That includes rewriting parts in modern languages like Rust.

Main Hall
18:25
18:25
5min
Closing session of All Systems Go! 2024

Closing session of All Systems Go! 2024

Main Hall