Development of a specialized eBPF-based profiler for file accesses in HPC environments

The file access profiler fsnoop tracks file accesses on specified processes and helps identify inefficient patterns. It is specifically designed to be used in High-Performance Computing (HPC) environments, running on different nodes across multiple clusters.

Project logo (own design)

Initial situation

In High-Performance Computing (HPC), file access efficiency is a critical factor for running scientific applications. In a pipeline-model, executables communicate using files, i.e. one executable might read the output file of the preceding one. Having inefficient file access patterns can lead to a prolonged execution time or even file system breakdowns. ESA's Euclid mission for example experienced situations where inefficient access to the file system led to the fact that a simple file listing command (ls) could take up to multiple hours! A profiler can help identify issues by gathering data during execution time and then analyzing it to detect patterns or inefficiencies.

Problem statement

Application profilers and even file access profilers have existed for a long time to improve performance of applications and their data access. However, none of the existing solutions on the market met the requirements of this project. The goal of this bachelor's thesis was to develop an eBPF-based file access profiler that could run in HPC environments. It should help identify issues with inefficient file access as well as possible congestions at the shared file system. Visualizing the gathered information about file access is another main goal of the project. Finally, the solution should help identify bottlenecks and thereby improve efficiency of file access in HPC environments.

Execution of fsnoop
                            
                        
This image shows how to execute fsnoop in tracing mode as to capture data on processes that enable IO-Tracing.
                            A test program is ran shortly after the profiler has been started, in order to generate some tracing data.
                            In order for it to be traced, the test program needs the environment variable ENABLE_IO_PROFILING set to the value 1.
Modular architecture
                            
                        
The architecture of fsnoop contains the executable fsnoop as well as the CoreCoordinator and PatternCoordinator.
                            Main reponsibility of the CoreCoordinator is loading the modules and their probes to profile applications and gather the required data.
                            Once that data is retrieved, the PatternCoordinator can be used to process and analyze the traces in order to find patterns and/or provide new metrics.
                            Both the initial data, as well as the metrics created from the PatternInterpreters can be visualized and analyzed in Grafana.
                        
Results from Euclid-Pipeline
                            
                        
The visualizations shown in the image above are created using the results gathered during our test run of a Euclid pipeline.
                            This pipeline ran on the DEV-Cluster of the Royal Observatory, Edinburgh (ROE) and during the run,
                            11,960,219 traces were collected!
                            Grafna was used to create those visualizations using standard graphs and tables with a subset of the collected data.
                        

Solution developed and its benefits

The developed file access profiler fsnoop is eBPF-based, runs on HPC clusters and helps identify file access patterns. Not only does it interpret the gathered information but by visualizing and analyzing it in Grafana, one can perform countless evaluations with the data at hand.

data retrieval using eBPF probes
no compile-time changes needed (any program can be profiled using fsnoop)
modular design to allow for future extensions
interpretation of low cache usage and overlapping file access
data visualization of traces in Grafana
easy-to-use command line tool to start tracing or pattern interpretation
low performance overhead (see box below)

What is eBPF?

The Extended Berkeley Packet Filter is a revolutionary technology in kernel-near programming. Traditionally, adding features to the Linux kernel was difficult and it could take years until every system included the new feature. eBPF can be compared to Javascript on the web browser as with it you can extend the capabilities of your system kernel. Using so-called kprobes you can hook into any kernel function and gather or even alter data in a safe and efficient way. You can hook into both the entry point as well as the exit point, allowing you to store e.g. execution time, argument values and the return value of a function.

Why Grafana?

While the project required a custom profiler to be developed, the goal was to be as compatible as possible with existing solutions and infrastructure. Therefore, the format of choice for collecting the aggregated file access data is the standardized OpenTelemetry Trace format. Grafana Tempo is used to store those traces and then essentially any visualization platform such as Grafana can be used to view this data. This allows for much more interpretation than just the abovementioned patterns as any user analyzing the data can create new visualizations, filters and aggregations.

Performance Overhead
Running in HPC environments, performance overhead of the profiler is critical. Therefore it was essential to measure it, even if it was just in some simulated use cases. The benchmark tried to simulate the behavior of pipelines as they appear in HPC-environments. Using fio, two workers are generated that construct files and write content and two workers are generated that consume this data by reading the generated files. This mimics somewhat the behavior we can expect when one executable in the pipeline writes an output file that is then used by the subsequent executable.

Metric Without fsnoop With fsnoop Overhead

Read bandwidth 1,095 MiB/s 1,071 MiB/s 2.19%

Read IOPS 8,762 8,566 2.24%

Read latency 199.6 µs 205.2 µs 2.81%

Write bandwidth 2,544 MiB/s 2,442 MiB/s 4.01%

Write IOPS 20,350 19,540 3.98%

Write latency 97.3 µs 101.9 µs 4.73%

Measurements made locally on a student's notebook

Metric	Without fsnoop	With fsnoop	Overhead
Read bandwidth	1,095 MiB/s	1,071 MiB/s	2.19%
Read IOPS	8,762	8,566	2.24%
Read latency	199.6 µs	205.2 µs	2.81%
Write bandwidth	2,544 MiB/s	2,442 MiB/s	4.01%
Write IOPS	20,350	19,540	3.98%
Write latency	97.3 µs	101.9 µs	4.73%

Key terms

eBPFExtended Berkeley Packet Filter (read more)
ProfilingMeasure operations, duration etc. during program execution
File Access PatternsPatterns in file access such as low cache usage, overlapping file access
ESAEuropean Space Agency (ESA), their mission Euclid jumpstarted this project
EuclidESA's Euclid mission explores the composition and evolution of the dark Universe (read more)

Customer

FHNW, I4DS
Prof. Dr. André Csillaghy
Bahnhofstrasse 6
5210 Windisch
more about I4DS

Team

Philippe Krüttli
Steven Meier

Advisor
Simon Marcin