Optimisation of scientific code using eBPF profilers in HPC clusters

By leveraging the experimental profiling signal from OpenTelemetry and enhancing the open-source OpenTelemetry eBPF profiler, this thesis allows for enhanced observability in HPC clusters for their operators and science code developers.

Official mission patch for the ESA Euclid mission and project logo

Initial situation

Optimizing science code running on large scale High-Performance Computing (HPC) environments, often having thousands of CPU cores, is a difficult task. With limited observability (mainly through logs of the executed code and system monitoring tools) and a high degree of concurrency spread over multiple worker nodes it is often difficult to identify the sources of performance bottlenecks.

Problem statement / Project goal

For HPC cluster operators, many times having multiple missions / customers exuting code on their cluster at the same time, it is very difficult to find the cause of a bottleneck in their systems. At the same time, for the developers of the science code, it is often very difficult to predict code behaviour due to the large difference in concurrency of a local development machine (~16 CPU cores) and a HPC environment (thousands of CPU cores).
The goal of this bachelor thesis is to enhance the monitoring and profiling capabilities of HPC systems for both operators and developers.

Solution developed and its benefits

This thesis enhances HPC performance monitoring by extending the OpenTelemetry eBPF profiler, enabling more efficient profiling and real-time observability for large-scale scientific computing environments like the ESA Euclid mission. By modifying the profiler to capture environment variables, this solution allows developers and administrators to filter and group profiling data more effectively, making it easier to analyze complex, parallelized workloads.

Enhanced Data Filtering & Organization

With the ability to capture environment variables, profiling samples can now be grouped by job, process, or worker node. This makes performance insights more actionable for both developers optimizing workloads and administrators monitoring system health.

Real-Time HPC Monitoring

Cluster administrators can now better track system load across worker nodes, improving resource allocation and troubleshooting potential bottlenecks in processing pipelines.

Integration into Open Source

The modifications have been successfully merged into the upstream OpenTelemetry repository, ensuring long-term maintainability, compatibility with new updates, and accessibility to the broader HPC community.

Laying the Groundwork for Future Improvements

This solution opens the door to further enhancements, such as:

• Off-CPU Profiling: Identifying performance bottlenecks beyond CPU usage, like file system and network slowdowns.
• User Group-Based Profiling: Expanding beyond environment variables to support systems that categorize processes by user groups.
• Better Symbolization Support: Improved stack trace readability in future Grafana Pyroscope releases.
• Multi-Metric Profiling: Extending profiling to cover memory, disk, and network usage, giving a more holistic view of HPC performance.

By addressing key profiling challenges, this work significantly improves the observability of HPC workloads, ensuring more efficient debugging, optimisation, and system monitoring for high-performance computing environments worldwide.

Key terms

• eBPFExtended Berkeley Packet Filter
• OpenTelemetryAn open source standard for telemetry infrastructure
• GrafanaTelemetry visualiation frontend
• FlamegraphGraph used to display profiling samples
• HPCHigh Performance Computing
• ESAEuropean Space Agency
• EuclidA space telescope used to survey dark energy and dark matter

Project sponsor

FHNW, i4DS
Prof. Dr. André Csillaghy
Bahnhofstrasse 6
5210 Windisch
more about i4DS

Team

Tino Heuberger

Advisor
Simon Marcin