0.8 ASG2019 2019-09-20 2019-09-22 3 00:05 https://cfp.all-systems-go.io https://cfp.all-systems-go.io/media/ASG2019/img/asg-logo-ondark.svg Europe/Berlin Loft Lightning talk 2019-09-20T09:30:00+02:00 09:30 00:10 Opening of All Systems Go! ASG2019-170-opening Chris Kuehl en false https://cfp.all-systems-go.io/ASG2019/talk/A3KZGD/ https://cfp.all-systems-go.io/ASG2019/talk/A3KZGD/feedback/ Loft 35 min talk + 5 min Q&A 2019-09-20T09:45:00+02:00 09:45 00:40 In this talk David will show Grafana's advanced features to manage a fleet of Linux hosts. He will also show relevant metrics and logging datasources and how they can be combined to get a full picture of what is going on. ASG2019-162-effective-infrastructure-monitoring-with-grafana David Kaltschmidt en false https://cfp.all-systems-go.io/ASG2019/talk/XJAWA7/ https://cfp.all-systems-go.io/ASG2019/talk/XJAWA7/feedback/ Loft 35 min talk + 5 min Q&A 2019-09-20T10:30:00+02:00 10:30 00:40 Presenting [traceloop](https://github.com/kinvolk/traceloop), a “time travel” tracing tool to trace system calls in cgroups using BPF and overwritable ring buffers. ASG2019-159-traceloop-for-systemd-and-kubernetes-inspektor-gadget Alban Crequy en Many people use the “strace” tool to synchronously trace system calls using ptrace. [Traceloop](https://github.com/kinvolk/traceloop) similarly traces system calls but asynchronously in the background, using BPF and tracing per cgroup. I’ll show how it can be integrated with systemd and with Kubernetes via [Inspektor Gadget](https://github.com/kinvolk/inspektor-gadget). Traceloop's traces are recorded in a fast, in-memory, overwritable ring buffer like a flight recorder. As opposed to “strace”, the tracing could be permanently enabled on systemd services or Kubernetes pods and inspected in case of a crash. This is like a always-on “strace in the past”. Traceloop uses BPF through the gobpf library. Several new features have been added in gobpf for the needs of traceloop: support for overwritable ring buffers and swapping buffers when the userspace utility dumps the buffer. https://github.com/kinvolk/traceloop https://github.com/kinvolk/inspektor-gadget https://github.com/iovisor/gobpf Slides: https://docs.google.com/presentation/d/1zIZUrTrD7FkS9pHnWz87ZmoLTrO1g9-J_lDMD7E5kdo/edit false https://cfp.all-systems-go.io/ASG2019/talk/98A9LW/ https://cfp.all-systems-go.io/ASG2019/talk/98A9LW/feedback/ Loft 35 min talk + 5 min Q&A 2019-09-20T11:30:00+02:00 11:30 00:40 How can we build hostile and untrusted code in containers? There are many options available, but not all of them are as safe as they claim to be... ASG2019-146-rootless-reproducible-hermetic-secure-container-build-showdown Andrew Martin en Rootless container image builds (as distinct from rootless container runtimes) have crept ever closer with orca-build, BuildKit, and img proving the concept. They are desperately needed: a build pipeline with an exposed Docker socket can be used by a malicious actor to escalate privilege - and is probably a backdoor into most Kubernetes-based CI build farms. With a slew of new rootless tooling emerging including Red Hat’s buildah, Google’s Kaniko, and Uber’s Makisu, we will see build systems that support building untrusted Dockerfiles? How are traditional build and packaging requirements like reproducibility and hermetic isolation being approached? In this talk we: - Detail attacks on container image builds - Compare the strengths and weaknesses of modern container build tooling - Chart the history and future of container build projects - Explore the safety of untrusted builds false https://cfp.all-systems-go.io/ASG2019/talk/PVYETJ/ https://cfp.all-systems-go.io/ASG2019/talk/PVYETJ/feedback/ Loft 35 min talk + 5 min Q&A 2019-09-20T12:15:00+02:00 12:15 00:40 Let's bring the UNIX concept of Home Directories into the 21st century. ASG2019-164-reinventing-home-directories Lennart Poettering en The concept of home directories on Linux/UNIX has little changed in the last 39 years. It's time to have a closer look, and bring them up to today's standards, regarding encryption, storage, authentication, user records, and more. In this talk we'll talk about "systemd-homed", a new component for systemd, that reworks how we do home directories on Linux, adds strong encryption that makes sense, supports automatic enumeration and hot-plugged home directories and more. false https://cfp.all-systems-go.io/ASG2019/talk/VSQRXA/ https://cfp.all-systems-go.io/ASG2019/talk/VSQRXA/feedback/ Loft 35 min talk + 5 min Q&A 2019-09-20T14:05:00+02:00 14:05 00:40 How did Microsoft made SQL Server available on Linux, Containers and ARM CPUs? Come hear the story from the SQL Server engineering team. ASG2019-131-how-microsoft-sql-server-went-multi-platform-sqlpal Argenis FernandezBrian GianforcaroEugene Birukov en We'd love to tell the story on how we made SQL Server available to ecosystems outside of Windows in this talk. It's a great story that involves quite a bit of interesting technologies and we'd like to share that with everyone! false https://cfp.all-systems-go.io/ASG2019/talk/GTYJFV/ https://cfp.all-systems-go.io/ASG2019/talk/GTYJFV/feedback/ Loft 35 min talk + 5 min Q&A 2019-09-20T14:50:00+02:00 14:50 00:40 Resource control is reaching feature completeness and the focus at facebook is shifting towards productionizing. Let's go over what feature completeness means and the productionizing efforts. ASG2019-133-resource-control-facebook-2019 Tejun HeoDan Schatzberg en Until recently, we never had all the kernel and system features needed to implement work-conserving comprehensive resource control. With the recent additions of senpai, io.weight and cpu.headroom and others, we now have all pieces to implement protection, stacking and side-loading. Our focus at facebook is gradually shifting towards productionizing resource control so that service owners can obtain high resource reliability and utilization without worrying about the details. Let's go over how resource control features come together to form the basic resource profiles and how we're trying to productionize them. false https://cfp.all-systems-go.io/ASG2019/talk/KEK3MD/ https://cfp.all-systems-go.io/ASG2019/talk/KEK3MD/feedback/ Loft 20 min talk + 5 min Q&A 2019-09-20T15:35:00+02:00 15:35 00:25 The difficult task to checkpoint and restore a process is used in many container runtimes to implement container live migration. This talk will give details how CRIU is able to checkpoint and restore processes, how it is integrated in different container runtimes and which optimizations CRIU offers to decrease the downtime during container migration. ASG2019-120-container-live-migration Adrian Reber en In this talk I want to provide details how CRIU checkpoints and restores a process. Starting from ptrace() to pause the process, how parasite code is injected into the process to checkpoint the process from its own address space. How CRIU transforms itself to the restored process during restore. How SELinux and seccomp is restored. I also want to give an overview how CRIU uses userfaultfd for lazy migration and dirty page tracking for pre-copy migration. I want to end this talk with an overview about how CRIU is integrated in different container runtimes to implement container live migration. false https://cfp.all-systems-go.io/ASG2019/talk/E88Z7V/ https://cfp.all-systems-go.io/ASG2019/talk/E88Z7V/feedback/ Loft 20 min talk + 5 min Q&A 2019-09-20T16:20:00+02:00 16:20 00:25 In this talk, I'll go through my efforts to revamp libcontainer's systemd driver, in particular to support the unified cgroup hierarchy. ASG2019-151-revamping-libcontainer-s-systemd-driver Filipe Brandenburger en libcontainer is part of runc (opencontainers/runc in GitHub) and is used by the Docker and containerd ecosystem to spawn containers. This work is trying to bridge the gap between the Docker/containerd/Kubernetes ecosystem and cgroup2 through the unified hierarchy, using systemd as an authoritative container manager. I'll also touch on alternative approaches (such as crun and systemd-nspawn) and briefly talk about the OCI standard and the need for it to evolve to properly support cgroup2 semantics. false https://cfp.all-systems-go.io/ASG2019/talk/YPU3HL/ https://cfp.all-systems-go.io/ASG2019/talk/YPU3HL/feedback/ Loft 20 min talk + 5 min Q&A 2019-09-20T16:50:00+02:00 16:50 00:25 The primary focus is to gather feedback from systemd community regarding ongoing and future work to introduce custom cgroup-bpf programs to systemd. The motivation is to give a user a capability to attach their own cgroup-bpf programs to systemd containers. This is a continuation of <a href="https://github.com/systemd/systemd/issues/10227" title="discussion"> started at ASG2018 and followed by <a href="https://github.com/systemd/systemd/pull/12151" title="PR12151"> and <a href="https://github.com/systemd/systemd/pull/12419" title="PR12419">. ASG2019-144-custom-cgroup-bpf-programs-in-systemd Julia Kartseva en Currently systemd utilizes BPF macro-assembly which is poorly extendable and maintainable, so the 1st iteration would be introducing `libbpf` library to systemd. The first attempt was made and it raised valid questions about `libbpf` testability and dependencies it introduces. We’d like to address that. Another topic of focus may be implementation details, such as how to store libbpf programs: either as bytecode or as restricted C which compiles with the rest of systemd. For attendees with no context a brief intro to eBPF will be made including new initiatives which may be of use to systemd, e.g. “Compile once, run everywhere”. Since this is ongoing work the agenda may vary depending on activity in PRs. false https://cfp.all-systems-go.io/ASG2019/talk/M8DVWG/ https://cfp.all-systems-go.io/ASG2019/talk/M8DVWG/feedback/ Loft Lightning talk 2019-09-20T17:20:00+02:00 17:20 00:05 How Endless are implementing time-limited scopes in systemd, using that to implement time-limited login sessions, and then using that to implement parental controls on the desktop. ASG2019-132-time-limited-login-sessions Philip Withnall en false https://cfp.all-systems-go.io/ASG2019/talk/8RB73U/ https://cfp.all-systems-go.io/ASG2019/talk/8RB73U/feedback/ Loft Lightning talk 2019-09-20T17:25:00+02:00 17:25 00:05 Zstandard (zstd) is a new lossless compression algorithm with a very attractive compression rate and performance. In production environments it comes with some quantifiable benefits but also with some surprising issues. ASG2019-167-impact-of-zstd Oskari SaarenmaaVille Tainio en false https://cfp.all-systems-go.io/ASG2019/talk/DG3YDE/ https://cfp.all-systems-go.io/ASG2019/talk/DG3YDE/feedback/ Loft Lightning talk 2019-09-20T17:30:00+02:00 17:30 00:05 Several of the standard tools like `grep` and `find` have rewritten alternatives, performing the tasks much quicker and have a more intuitive interface. Present some of them. ASG2019-157-alternatives-to-standard-utilities Paul Menzel en false https://cfp.all-systems-go.io/ASG2019/talk/JFC7VC/ https://cfp.all-systems-go.io/ASG2019/talk/JFC7VC/feedback/ Loft Lightning talk 2019-09-20T17:35:00+02:00 17:35 00:05 Using RPMs can be very advantageous during development of systemd on Fedora. In order to make that viable, we need to build them from a git checkout and have the ability to use incremental builds. ASG2019-149-using-rpms-for-systemd-development Filipe Brandenburger en I will explore tooling I've been using and building to use RPMs during systemd development. I'll quickly cover the motivation and advantages while I manage to build one during a lightning demo. false https://cfp.all-systems-go.io/ASG2019/talk/JM7GDN/ https://cfp.all-systems-go.io/ASG2019/talk/JM7GDN/feedback/ Cage 35 min talk + 5 min Q&A 2019-09-20T09:45:00+02:00 09:45 00:40 Atomic Updates and user modified configuration files in /etc often lead to hard to resolve conflicts. In this talk, I want to show the most common and biggest problems and possible solutions. ASG2019-119-atomic-updates-and-configuration-files-in-etc Ignaz Forster en More and more Linux Distributors have a Distribution using atomic updates to update the system. They all have the problem of updating the files in /etc, as an admin could do changes after the update but before the reboot to activate the updates. But everybody come up with another solution which solves their usecase, but is not generic useable. Additional there is the "Factory Reset" of systemd, which no big distribution has really fully implemented today. A unique handling of /etc for atomic updates could also help to convince upstream developers to add support to their applications, while currently they hesitate to add distribution specific patches and support. During this talk, I will describe the different areas of problems and possible solutions. The goal is to provide a concept working for all Linux Distributors (like the FHS). My dream is, that no package installs anything in /etc, it should only contain changes made by the system administrator or configuration files managed by the system administrator. For some problems, it would be already enough today if Linux distributors would adjust the configuration of applications or use all features of them. Other requires minimal to intrusive changes to packages, and for the last kind complete new concepts are necessary. false https://cfp.all-systems-go.io/ASG2019/talk/KYTCJV/ https://cfp.all-systems-go.io/ASG2019/talk/KYTCJV/feedback/ Cage 35 min talk + 5 min Q&A 2019-09-20T10:30:00+02:00 10:30 00:40 Whether to support users, ensure their security, or meet compliance goals, organizations need to deploy monitoring of their desktop machines. Yet, many approaches overreach by effectively being rootkits. In this presentation, we'll examine: * What data a monitoring system needs to collect * Where the data we need lives on a modern Linux desktop * Which data sources expose sandbox-friendly API access * Sandboxing the monitoring daemon itself ASG2019-175-privacy-respecting-linux-desktop-monitoring David Strauss en false https://cfp.all-systems-go.io/ASG2019/talk/3ZKVWF/ https://cfp.all-systems-go.io/ASG2019/talk/3ZKVWF/feedback/ Cage 35 min talk + 5 min Q&A 2019-09-20T11:30:00+02:00 11:30 00:40 Have you ever encountered a transient performance issue, that was hard to investigate only from the database point of view? On top of how many layers of abstraction your database is working? What is the difference between running your database on a bare metal, VM or inside a container? PostgreSQL does not work in the vacuum, it heavily relies on functionality provided by an underlying platform. And sometimes to answer these questions above one needs to step back and look at a problem not only from a database point of view. In this talk we will discuss how to achieve that, how to tame such tools as strace, perf or eBPF to troubleshoot intricate issues and stay curious. ASG2019-117-postgresql-at-low-level-stay-curious- Dmitrii Dolgov en Have you ever encountered a transient performance issue, that was hard to investigate only from the database point of view? On top of how many layers of abstraction your database is working? What is the difference between running your database on a bare metal, VM or inside a container? PostgreSQL does not work in the vacuum, it heavily relies on functionality provided by an underlying platform. And sometimes to answer these questions above one needs to step back and look at a problem not only from a database point of view. In this talk we will discuss how to achieve that, how to tame such tools as strace, perf or eBPF to troubleshoot intricate issues and stay curious. false https://cfp.all-systems-go.io/ASG2019/talk/AXPVZ3/ https://cfp.all-systems-go.io/ASG2019/talk/AXPVZ3/feedback/ Cage 35 min talk + 5 min Q&A 2019-09-20T12:15:00+02:00 12:15 00:40 Learn how a Service Mesh can secure your bare-metal (non-virtualized) workloads quickly without any code modifications to improve your security posture. ASG2019-136-securing-bare-metal-micro-services-service-mesh John Studarus en Zero Trust is an information security mantra to not implicitly trust any the underlying infrastructure (hardware, network, software, etc). For many organizations, this extends into the cloud where this philosophy is applied to workloads running in public, virtualized clouds. We'll be taking this philosophy to protect an insecure application, the Fortune Cookie Micro Service, running atop a bare metal cloud with a Service Mesh to provide authentication and encryption of data in motion without the complexities of virtualization or containerization. This walkthrough uses all open source software (Terraform for the deployment atop the Packet bare metal cloud and Consul for the service mesh) atop Ubuntu physical nodes. false https://cfp.all-systems-go.io/ASG2019/talk/H3YZZM/ https://cfp.all-systems-go.io/ASG2019/talk/H3YZZM/feedback/ Cage 35 min talk + 5 min Q&A 2019-09-20T14:05:00+02:00 14:05 00:40 GNU poke is a new interactive editor for binary data. Not limited to editing basic entities such as bits and bytes, it provides a full-fledged procedural, interactive programming language designed to describe data structures and to operate on them. ASG2019-127-gnu-poke-an-extensible-editor-for-structured-binary-data Jose E. Marchesi en GNU poke is a new interactive editor for binary data. Not limited to editing basic entities such as bits and bytes, it provides a full-fledged procedural, interactive programming language designed to describe data structures and to operate on them. Once a user has defined a structure for binary data (usually matching some file format) she can search, inspect, create, shuffle and modify abstract entities such as ELF relocations, MP3 tags, DWARF expressions, partition table entries, and so on, with primitives resembling simple editing of bits and bytes. The program comes with a library of already written descriptions (or "pickles" in poke parlance) for many binary formats. GNU poke is useful in many domains. It is very well suited to aid in the development of programs that operate on binary files, such as assemblers and linkers. This was in fact the primary inspiration that brought me to write it: easily injecting flaws into ELF files in order to reproduce toolchain bugs. Also, due to its flexibility, poke is also very useful for reverse engineering, where the real structure of the data being edited is discovered by experiment, interactively. It is also good for the fast development of prototypes for programs like linkers, compressors or filters, and it provides a convenient foundation to write other utilities such as diff and patch tools for binary files. This talk (unlike Gaul) is divided into four parts. First I will introduce the program and show what it does: from simple bits/bytes editing to user-defined structures. Then I will show some of the internals, and how poke is implemented. The third block will cover the way of using Poke to describe user data, which is to say the art of writing "pickles". The presentation ends with a status of the project, a call for hackers, and a hint at future works. false https://cfp.all-systems-go.io/ASG2019/talk/BKXVJQ/ https://cfp.all-systems-go.io/ASG2019/talk/BKXVJQ/feedback/ Cage 35 min talk + 5 min Q&A 2019-09-20T14:50:00+02:00 14:50 00:40 Transactional updates (also called atomic updates) are a way to update a system without interfering with the currently running system - making this a rock-solid way to update any machine, from embedded systems to cluster nodes. ASG2019-128-transactional-updates-with-btrfs Ignaz Forster en What do openSUSE MicroOS, Fedora CoreOS, Chrome OS, Ubuntu Core and Android have in common? All of them are using a *read-only root file system* and so called *transactional / atomic updates* to update a system safely - without having to worry that a broken update could leave your system in some undefined state. This talk will focus on how to use *btrfs*' snapshot feature to implement such a transactional system and explain where the pitfalls of implementing such a system compared to a traditional read-write system are. false https://cfp.all-systems-go.io/ASG2019/talk/SXENPK/ https://cfp.all-systems-go.io/ASG2019/talk/SXENPK/feedback/ Cage 20 min talk + 5 min Q&A 2019-09-20T15:35:00+02:00 15:35 00:25 Follow a journey of writing STM32 microcontroller firmware from scratch, using open-source tools. ASG2019-161-microcontroller-firmware-from-scratch Nikolai Kondrashov en Follow Nikolay Kondrashov's journey of learning to write firmware for an STM32 microcontroller (the Blue Pill one) from scratch, using only open-source tools. From blinking LEDs, to controlling a toy car, without the complicated, and license-restricted manufacturer's libraries, or the comfortable crutches of the Arduino stack. Learn where to look for information, which tools you might need, and how to do it yourself with a similar or a different microcontroller. See the slides at https://slides.com/spbnick/microcontroller-firmware-from-scratch/ false https://cfp.all-systems-go.io/ASG2019/talk/JDCVYP/ https://cfp.all-systems-go.io/ASG2019/talk/JDCVYP/feedback/ Cage 20 min talk + 5 min Q&A 2019-09-20T16:20:00+02:00 16:20 00:25 What happened in the coreboot based firmware world since last year? How to get started? ASG2019-156-news-from-the-coreboot-land Paul Menzel en In September, coreboot 4.10 will have been released, and the Open Source Firmware Conference took place. Take this opportunity to present the latest news and changes in the coreboot based firmware world. AMD devices are available with coreboot, and after Google and Puri.sm more vendors like System76 ship their devices with coreboot. While at it, give a quick introduction how to get started. false https://cfp.all-systems-go.io/ASG2019/talk/UUYNXW/ https://cfp.all-systems-go.io/ASG2019/talk/UUYNXW/feedback/ Cage 20 min talk + 5 min Q&A 2019-09-20T16:50:00+02:00 16:50 00:25 The embedded world has dealt with image creation for decades. Why not use those decade of experience to reliably create image for the datacenter world ? ASG2019-124-buildroot-using-embedded-tools-to-build-container-images Jérémy Rosen en Building an OS image in a reliable, reproducible, tracable and archivable way is a hard problem, but it is a problem that the embedded world has been working on for decades and where mature and easy to use tools exist Nowdays, the world of containers is rediscovering these problems and most tools do not provide the level of tracability and reliability needed to be able to properly track the content of an image in every detail and be confident that it is possible to report what changes are local and what licenses are used. Buildroot is one of the tools the embedded world provides to solve that problem. It is robust, mature, deadly simple to use and can really help getting back the control on container images. false https://cfp.all-systems-go.io/ASG2019/talk/B7D7BC/ https://cfp.all-systems-go.io/ASG2019/talk/B7D7BC/feedback/ Loft 20 min talk + 5 min Q&A 2019-09-21T09:30:00+02:00 09:30 00:25 BuildStream is used to build Freedesktop SDK for different deployment systems allowing applications based on it to be distributed at once to multiple systems. ASG2019-145-distributing-freedesktop-sdk-applications-to-flatpak-snapd-and-docker Valentin David en Flatpak, Snapd and Docker are similar. They are all used for deployment and applications use their own runtime. Each system has its own tools for development. Flatpak uses Flatpak Builder. Snapd uses Snapcraft. Docker development is based on `Dockerfile`s. Freedesktop SDK was developed to be the runtime of Flatpak. It used to be partly built with Flatpak Builder. It has since changed to be built with a deployment system agnostic tool: BuildStream. For this reason we can export the Freedesktop SDK to multiple formats. We will show how it is possible to build an application for the three systems at once. false https://cfp.all-systems-go.io/ASG2019/talk/CF7FSX/ https://cfp.all-systems-go.io/ASG2019/talk/CF7FSX/feedback/ Loft 20 min talk + 5 min Q&A 2019-09-21T10:00:00+02:00 10:00 00:25 oomd is a userspace out-of-memory killer. This talk covers past, present, and future development along with possible plans for systemd integration. ASG2019-135-oomd2-and-beyond-a-year-of-improvements Daniel XuAnita Zhang en Running out of memory on a host is a particularly nasty scenario. In the Linux kernel, if memory is being overcommitted, it results in the kernel out-of-memory (OOM) killer kicking in. Perhaps surprisingly, the kernel does not often handle this well. oomd builds on top of recent kernel development to effectively implement OOM killing in userspace. This results in a faster, more predictable, and more accurate handling of OOM scenarios. oomd has gained a number of new features and interesting deployments in the last year. The most notable feature is a complete redesign of the control plane which enables arbitrary but "gotcha"-free configurations. In this talk, Daniel Xu will cover past, present, future, and path-not-taken development plans along with experiences gained from overseeing large deployments of oomd. Anita Zhang will close the talk with a discussion of why oomd would be a great addition to systemd. false https://cfp.all-systems-go.io/ASG2019/talk/DQX3DH/ https://cfp.all-systems-go.io/ASG2019/talk/DQX3DH/feedback/ Loft 20 min talk + 5 min Q&A 2019-09-21T10:30:00+02:00 10:30 00:25 Buck is an opensource build system. At Facebook, we’ve taught it to build container images that work with systemd. ASG2019-143-building-portable-service-images-with-buck Lindsay Salisbury en At Facebook we use an open-source build system called Buck. Buck is a build system designed to provide more strong guarantees of incremental builds, reproducibility, and dependency management. Open-source Buck can now be used to construct fully described and fully self-contained container images that work with systemd! I will show how we use this tool internally at Facebook and how it can be used externally (It’s open-source!) to build service containers for use by systemd. I will dive into the the details of how these builds are performed with systemd-nspawn, how we use the Buck system to define the systemd services and their dependencies, and how these images work at runtime. false https://cfp.all-systems-go.io/ASG2019/talk/K7E7T7/ https://cfp.all-systems-go.io/ASG2019/talk/K7E7T7/feedback/ Loft 35 min talk + 5 min Q&A 2019-09-21T11:00:00+02:00 11:00 00:40 Traditionally processes are identified globally via process identifiers (PIDs). Due to how pid allocation works the kernel is free to recycle PIDs once a process has been reaped. As such, PIDs do not allow another process to maintain a private, stable reference on a process. On systems under pressure it is thus possible that a PID is recycled without other (non-parent) processes being aware of it. This becomes rather problematic when (non-parent) processes are in charge of managing other processes as is the case for system managers or userspace implementations of OOM killers. Over the last months we have been working on solving these and other problems by introducing pidfds – process file descriptors. Among other nice properties, the allow callers to maintain a private, stable reference on a process. In this talk we will look at challenges we faced and the different approaches people pushed for. We will see what already has been implement and pushed upstream, look into various implementation details and outline what we have planned for the future. ASG2019-172-pidfds-process-file-descriptors-on-linux Christian Brauner en false https://cfp.all-systems-go.io/ASG2019/talk/TPS8TS/ https://cfp.all-systems-go.io/ASG2019/talk/TPS8TS/feedback/ Loft 20 min talk + 5 min Q&A 2019-09-21T11:55:00+02:00 11:55 00:25 Despite of it's old age, ksh still remains one of the most popular shells. In 2013, David Korn and others who worked on ksh were laid off from AT&T Bell Labs. This lead to speculations of death of ksh. In 2017, Siteshwar Vashisht and Kurtis Rader resumed it's development on GitHub. This talk will be about what makes ksh so challenging to maintain and how new developers are trying to revive it. ASG2019-121-squeezing-water-from-stone-kornshell-in-2019 Siteshwar Vashisht en false https://cfp.all-systems-go.io/ASG2019/talk/CV9R3N/ https://cfp.all-systems-go.io/ASG2019/talk/CV9R3N/feedback/ Loft 35 min talk + 5 min Q&A 2019-09-21T12:25:00+02:00 12:25 00:40 Most modern container image formats use tar-based linear archives to represent root filesystems, which results in many issues when using modern container images. In this talk, we will demonstrate a solution to this problem that we plan to propose for standardisation within the Open Container Initiative (code-named "OCIv2 images"). ASG2019-123-ociv2-container-images-considered-harmful Aleksa Sarai en This talk is specific to the Open Container Initiative's image specification, but the same techniques could be applied to other systems (though we'd obviously recommend using OCI). In order to avoid the [numerous issues with tar archives](https://www.cyphar.com/blog/post/ociv2-images-i-tar) it is necessary to come up with a different format. In addition, layer representations result in needless wasted space for storage of files which are no longer relevant to running containers. Massive amounts of duplication are also rampant within OCI images because tar archives are completely opaque to OCI's content-addressable store. Luckily the problem of representing a container root filesystem for distribution is very similar to existing problems within backup systems, and we can take advantage of prior art such as [restic](https://restic.net/) to show us how we can get significant space-savings and possibly efficiency savings. However, we also must ensure that the runtime cost of using this new system is equivalent to existing container images. Container images are efficient at runtime because they map directly to how overlay filesystems represent change-sets as layers, but with some tricks it is possible for us to obtain most of the improvements we also gained in distribution with de-duplication. Our proposed solution to all of these problems will be laid out, with opportunities for feedback and discussion. false https://cfp.all-systems-go.io/ASG2019/talk/VMTEPT/ https://cfp.all-systems-go.io/ASG2019/talk/VMTEPT/feedback/ Loft 20 min talk + 5 min Q&A 2019-09-21T14:05:00+02:00 14:05 00:25 We'll be covering happenings, learnings and new challenges running and supporting systemd in production on the Facebook fleet throughout the past year. ASG2019-142-systemd-facebook-in-2019 Davide Cavalca en This talk is a followup to [State of systemd @ Facebook](https://cfp.all-systems-go.io/ASG2018/talk/192/) that was presented last year. We'll cover the latest developments, how we're leveraging new systemd features, the design of our CI/CD pipeline for systemd, and finally discuss a number of interesting case studies. false https://cfp.all-systems-go.io/ASG2019/talk/983XHL/ https://cfp.all-systems-go.io/ASG2019/talk/983XHL/feedback/ Loft 35 min talk + 5 min Q&A 2019-09-21T14:35:00+02:00 14:35 00:40 The boot loader specification defines a generic drop-in based solution for defining boot targets. sd-boot is a boot loader for UEFI systems, and included in the systemd source tree. In this talk we’ll have a closer look on the what, the why and the how of the specification and the boot loader. ASG2019-163-boot-loader-specification-sd-boot Lennart Poettering en false https://cfp.all-systems-go.io/ASG2019/talk/HFJMLU/ https://cfp.all-systems-go.io/ASG2019/talk/HFJMLU/feedback/ Loft 35 min talk + 5 min Q&A 2019-09-21T15:20:00+02:00 15:20 00:40 This talk covers the ongoing effort about adding eBPF support to the GNU Toolchain. eBPF is a virtual machine running within the Linux kernel; initially intended for user-level packet capture and filtering, eBPF has since been generalized to also serve as a general-purpose infrastructure for non-networking purposes. ASG2019-126-ebpf-support-in-the-gnu-toolchain Jose E. Marchesi en This talk covers the ongoing effort about adding eBPF support to the GNU Toolchain. eBPF is a virtual machine running within the Linux kernel; initially intended for user-level packet capture and filtering, eBPF has since been generalized to also serve as a general-purpose infrastructure for non-networking purposes. Binutils support is already upstream [1]. This includes a CGEN cpu description, assembler, disassembler and linker. By the time of the conference a simulator will be available as well, along with GDB support. A GCC backend will be submitted for inclusion upstream before September. The first part of the talk will be a brief general description of the project, its components, what motivated us to start working on it, and an update on the project's status at the time of the conference. Then we will discuss the particular challenges of supporting a target like eBPF: On one hand, the kernel virtual machine has some unique characteristics that have a definitive impact on the tooling, like the in-kernel validator and the specialized contexts in which eBPF programs run. We will show how the tools can help improving the eBPF programmer's experience. On the other hand, the exact shape of compiled eBPF code is still subject to change, and is in fact rapidly changing and evolving. Initially quite simple in terms of toolchain needs (single compilation units, no linking) this is changing as more kernel systems are being changed/written to be based on eBPF, and as the in-kernel validator is becoming more and more sophisticated. Along with bigger and more complex programs comes the need for more abstraction, hence modularity and code reuse. Kernel hackers are already discussing about bpf-to-bpf calls, run-time linking, and so on. This increased level of ambition and sophistication imposes additional requirements on the tools. Finally, interoperability with clang/llvm (the other available toolchain supporting eBPF) will be also discussed, in the more general context of ABI and conventions for compiled eBPF, which are still to be (well) defined and documented. [1] https://sourceware.org/ml/binutils/2019-05/msg00306.html false https://cfp.all-systems-go.io/ASG2019/talk/MAYDS8/ https://cfp.all-systems-go.io/ASG2019/talk/MAYDS8/feedback/ Loft 35 min talk + 5 min Q&A 2019-09-21T16:30:00+02:00 16:30 00:40 Come and learn about packit: tooling which enables you to integrate your upstream project into Fedora linux. ASG2019-125-linux-distro-should-be-an-upstream-contributor-too Martin Sehnoutka en Imagine a world where Linux distributions provide feedback about using your upstream project back to the project. So that when you are working on a change, you'll know right away: * if it builds or a project Z changed API again * if it works or that your change doesn't work with older systemd which this distro has * or if your change breaks components which depend on your project That's not all! If we have a service which can do all of this, why not propose a new upstream release automatically as a change to the linux distro once the release is done? Wouldn't it be awesome if upstream developers could control and track in which version their software is in Fedora 30? Sounds interesting? Please join us in this session and learn more about the packit tool and the packit service: tooling which makes your dream come true. false https://cfp.all-systems-go.io/ASG2019/talk/US8XA9/ https://cfp.all-systems-go.io/ASG2019/talk/US8XA9/feedback/ Loft 20 min talk + 5 min Q&A 2019-09-21T17:15:00+02:00 17:15 00:25 A summary of the current state of Thunderbolt, kernel as well as user space, including the latest development where the the input–output memory management unit (IOMMU) is used to prevent Direct Memory Access (DMA) attacks. A brief explanation and discussion of such such an attack, the recent Thunderclap attacks, will be given including with a focus on how it is related to the IOMMU feature on Linux. ASG2019-150-the-state-of-thunderbolt-on-gnu-linux Christian Kellner en false https://cfp.all-systems-go.io/ASG2019/talk/HXLJNF/ https://cfp.all-systems-go.io/ASG2019/talk/HXLJNF/feedback/ Loft Lightning talk 2019-09-21T17:45:00+02:00 17:45 00:15 Closing of All Systems Go! 2019 ASG2019-171-closing Chris Kuehl en false https://cfp.all-systems-go.io/ASG2019/talk/WB9TFT/ https://cfp.all-systems-go.io/ASG2019/talk/WB9TFT/feedback/ Cage 20 min talk + 5 min Q&A 2019-09-21T09:30:00+02:00 09:30 00:25 How to run clusters for GPU computing based blockchain hashing diskless on cost effective commodity hardware. ASG2019-141-coinboot-cost-effective-diskless-gpu-clusters-for-blockchain-hashing-and-beyond Gunter Miegel en Running the nodes of a cluster diskless is quite common in HPC environments. The challenges to run diskless in the context of blockchain hashing for cryptocurrencies are different. There are constraints like to run sufficiently on hundreds of machines with commodity 1 Gbit/s network hardware or the modest RAM size of 4 Gigabyte. This talk will provide insights in the technical approaches that made it possible to run GPU-clusters for blockchain hashing diskless and provide an outlook to other potential GPU-based use cases beyond blockchain hashing. I will discuss like how some early userspace trickery and state of the art RAM compression is used. How to handle the modest given RAM size and how a neat toolset based on container-runtimes helps to easily build boot images and plug-in packages. And how to use plug-in packages as an elegant way for adding further software like proprietary GPU drivers to the computing nodes of the clusters. false https://cfp.all-systems-go.io/ASG2019/talk/XNU7NE/ https://cfp.all-systems-go.io/ASG2019/talk/XNU7NE/feedback/ Cage 20 min talk + 5 min Q&A 2019-09-21T10:00:00+02:00 10:00 00:25 During development and testing it is often needed to test different kernels or run various sets of unit tests quickly. With lrun it is possible to do exactly that. It utilizes existing technology including UML, KVM and Namespaces to facility different environments. It has been in active use for testing Bluetooth and Wi-Fi features on Linux and can be easily extended to other technologies in the future. This presentation will introduce lrun and its design. It will also show demos of its current use cases. ASG2019-148-development-and-testing-with-lrun Marcel Holtmann en false https://cfp.all-systems-go.io/ASG2019/talk/N8YRKX/ https://cfp.all-systems-go.io/ASG2019/talk/N8YRKX/feedback/ Cage 35 min talk + 5 min Q&A 2019-09-21T10:30:00+02:00 10:30 00:40 Testing the effectiveness of Kubernetes Network Policies can be done in different approaches. In this talk we will show you the benefits and drawbacks of different approaches and what solution we finally chose. ASG2019-138-trust-is-good-control-is-better-a-short-story-about-network-policies Maximilian BischoffJohannes Scheuermann en Probably everybody who uses Kubernetes in a productive environment with multiple users possibly has looked at policies. Often the operators of the cluster(s) just trust the policies but in some cases it might be useful to control if the policies actually have taken action and often there are just to many Policies in the cluster setup to manually test them all (and obviously you don’t want to do this). Testing the effectiveness of the Network Policies can be done in different approaches. In this talk we will show you the benefits and drawbacks of different approaches and what solution we finally chose. Also we will show you some other tools and how they complement our solution. As a takeaway you will get an overview of different testing strategies for policies, as well as understanding challenges in testing policies in general and the Kubernetes ecosystem. We will get a feeling that it’s not always the best idea to just trust other plugins to implement the policies correctly. Our solution is open-sourced under https://github.com/inovex/illuminatio/ false https://cfp.all-systems-go.io/ASG2019/talk/QXMUUW/ https://cfp.all-systems-go.io/ASG2019/talk/QXMUUW/feedback/ Cage 20 min talk + 5 min Q&A 2019-09-21T11:55:00+02:00 11:55 00:25 The open source wireless daemon iwd has been introduced about 5 years ago and has seen an active development since its inception. The last year has been focused on behind the scenes work for new Wi-Fi standards that make connection setup faster, make roaming smoother and also introduce new security standards including WPA3. This presentation will demonstrate the new advances in Wi-Fi support for Linux and show how they improve the usage from within Network Manager and other connection managers. ASG2019-147-iwd-state-of-the-union Marcel Holtmann en false https://cfp.all-systems-go.io/ASG2019/talk/WBJNQQ/ https://cfp.all-systems-go.io/ASG2019/talk/WBJNQQ/feedback/ Cage 35 min talk + 5 min Q&A 2019-09-21T12:25:00+02:00 12:25 00:40 This talk is about the bmc-toolbox, an open-source project that leverages the _Baseboard Management Controller_ (iLOs iDracs and similar) to help manage a large fleet (>50K) of bare metal servers at Booking.com [bmc-toolbox.github.io](https://bmc-toolbox.github.io/) Its goal is to provide vendor agnostic tooling to manage the lifecycle of bare metal servers, this talk describes the tools part of bmc-toolbox and various aspects of managing a large fleet of bare metal servers. ASG2019-165-bmc-management-with-bmc-toolbox /media/ASG2019/images/7WMKLH/bmc-toolbox.png Joel RebelloJuliano Martinez en The bmc-toolbox leverages the _Baseboard Management Controller_ to help manage the lifecycle of datacenter bare metal. It provides vendor agnostic tools and a library in Go lang to *inventorize*, *configure*, *manage**, **update* a large fleet of bare metal assets with the help of the BMC. - *bmclib* - A Go lang library that provides a consistent set of methods to interface with BMCs. - *dora* - tool to **inventorize** a fleet of bare metal servers and chassis assets. - *bmcbutler* - tool to handle **configuration management** for a fleet of bare metal server and chassis BMCs. - *actor* - A single **API webservice** endpoint to interact with a fleet of bare metal BMCs. - *bmcldap* - LDAP based **authentication/authorization** service/proxy for BMCs. - *bmcfwupd* - tool to **update** the BMC firmware. This talk covers, - The challenges managing the provisioning and lifecycle of a *not yet hyperscale* size set of bare metal servers. - The purpose of the tools included of bmc-toolbox, how they help make our lives easier - How the tooling interacts with the BMCs (vendor specific APIs, Redfish) - The current state of Redfish in the wild false https://cfp.all-systems-go.io/ASG2019/talk/7WMKLH/ https://cfp.all-systems-go.io/ASG2019/talk/7WMKLH/feedback/ Cage 20 min talk + 5 min Q&A 2019-09-21T14:05:00+02:00 14:05 00:25 Currently everyone uses the same seccomp rules for running their containers. This tool allows us to generate seccomp rules based on what the container actually requires and allows us to lock down the container. ASG2019-140-generating-seccomp-profiles-for-containers-using-podman-and-ebpf Dan Walsh en We had a GSOC student this summer who instrumented podman to allow it to run containers and then genrate the seccomp rules for the container based on the syscalls that the container actually made. Once you have this newly generate seccomp file and are satisfied that you have thoroughly tested the container, you can run the container inproduction using the seccomp.json file. This talk will explain how the tool works and demonstrate it in action. false https://cfp.all-systems-go.io/ASG2019/talk/ACEWHG/ https://cfp.all-systems-go.io/ASG2019/talk/ACEWHG/feedback/ Cage 35 min talk + 5 min Q&A 2019-09-21T14:35:00+02:00 14:35 00:40 We will present [Yomi](https://github.com/openSUSE/yomi), a new proposal for installing Linux using [SaltStack](https://github.com/saltstack/salt). This installer is designed to be used in heterogeneous clusters, where you need a bit of intelligence during the installation and be integrated as one more step in the provisioning process. ASG2019-118-yomi-an-opensuse-installer-based-on-saltstack Alberto Planas Dominguez en [Yomi](https://github.com/openSUSE/yomi) is a new kind of installer for the [open]SUSE family based on SaltStack and independent of AutoYaST. The goal of this project is to make the installation of Linux (currently openSUSE) when: * You have a cluster of heterogeneous nodes (different profiles of memory, storage, CPU and network configurations) * The installation needs to be unattended * The installer needs to make decisions based on local profiles and external data * The installation process needs to be integrated, as one step more, into a more complicated provisioning workflow. The dependencies of Yomi are minimal, as only Salt and a very few CLI tools are required, which make it ideal to be deployed a booted from PXE Boot. false https://cfp.all-systems-go.io/ASG2019/talk/KDEYJZ/ https://cfp.all-systems-go.io/ASG2019/talk/KDEYJZ/feedback/ Cage 35 min talk + 5 min Q&A 2019-09-21T15:20:00+02:00 15:20 00:40 Ever experienced a broken system by simply upgrading packages? No more! This talk introduces the purely functional package manager Nix and the advancements all software distributions can benefit from - with some of those already implemented in mainstream package managers like snap. ASG2019-155-purely-functional-package-management /media/ASG2019/images/AD8VYE/nixos-logo-only-hires.png Franz Pletz en false https://cfp.all-systems-go.io/ASG2019/talk/AD8VYE/ https://cfp.all-systems-go.io/ASG2019/talk/AD8VYE/feedback/ Cage 35 min talk + 5 min Q&A 2019-09-21T16:30:00+02:00 16:30 00:40 Lessons learned operating thousands of stateful production clusters on top of Fedora and systemd-nspawn. ASG2019-166-stateful-systems-on-immutable-infrastructure Hannu Valtonen en Aiven is a cloud data platform operating thousands of production clusters on top of different cloud infrastructure providers (e.g. AWS, GCP). We offer the latest open source database & streaming engines to our users around the world, and implement most of our platform using the latest open source software including Fedora and systemd-nspawn. We wanted to base our platform on a fast moving Linux distribution like Fedora to gain quick access to new technology and avoid having to backport a lot of things. Fast moving distributions are typically not supported for a long time, but implementing an immutable infrastructure where deployed machines are not touched afterwards makes it possible to use them in production. In this talk we’ll share the details of our architecture and the lessons we’ve learned as well as problems we’ve faced over the years operating hundreds of thousands of virtual machines and containers with it on top of six different public clouds. false https://cfp.all-systems-go.io/ASG2019/talk/RLCDFS/ https://cfp.all-systems-go.io/ASG2019/talk/RLCDFS/feedback/ Cage 20 min talk + 5 min Q&A 2019-09-21T17:15:00+02:00 17:15 00:25 Senpai is a userspace tool to auto-tune cgroup memory limits. ASG2019-134-senpai-automatic-memory-sizing-for-containers Johannes Weiner en Due to virtual memory and optimistic caching strategies, true memory consumption of an application, and true utilization of a system's RAM, are mostly unknowns on modern operating systems. This has always made memory provisioning a tough and error-prone trial-and-error task, but it's aggravated with containerization, where the stated goal is thinner margins and higher resource efficiency. Senpai is a userspace tool that harnesses recently developed Linux kernel features to automatically shrink cgroups to their smallest possible memory size without notably affecting the performance of the contained applications. This talk goes over the motivation to develop senpai, how it works, and success stories from the Facebook fleet. false https://cfp.all-systems-go.io/ASG2019/talk/TCBLRG/ https://cfp.all-systems-go.io/ASG2019/talk/TCBLRG/feedback/