Traceloop for systemd and Kubernetes + Inspektor Gadget
2019-09-20, 10:30–11:10, Loft

Presenting traceloop, a “time travel” tracing tool to trace system calls in cgroups using BPF and overwritable ring buffers.


Many people use the “strace” tool to synchronously trace system calls using ptrace. Traceloop similarly traces system calls but asynchronously in the background, using BPF and tracing per cgroup. I’ll show how it can be integrated with systemd and with Kubernetes via Inspektor Gadget.

Traceloop's traces are recorded in a fast, in-memory, overwritable ring buffer like a flight recorder. As opposed to “strace”, the tracing could be permanently enabled on systemd services or Kubernetes pods and inspected in case of a crash. This is like a always-on “strace in the past”.

Traceloop uses BPF through the gobpf library. Several new features have been added in gobpf for the needs of traceloop: support for overwritable ring buffers and swapping buffers when the userspace utility dumps the buffer.

https://github.com/kinvolk/traceloop
https://github.com/kinvolk/inspektor-gadget
https://github.com/iovisor/gobpf

Slides: https://docs.google.com/presentation/d/1zIZUrTrD7FkS9pHnWz87ZmoLTrO1g9-J_lDMD7E5kdo/edit