Export profiles using async-profiler via OTLP

[kamphaus]

Component(s)

No response

Is your feature request related to a problem? Please describe.

I'd like to profile Java applications while linking traces and profiles.

Describe the solution you'd like

I'd like to add a profiling component which uses async-profiler that emits profiles from the SDK via OTLP when profiling is enabled (eg. by setting OTEL_PROFILES_EXPORTER=otlp).

Describe alternatives you've considered

I considered using Grafana's Pyroscope integration with the Java Otel SDK https://github.com/grafana/otel-profiling-java. It uses async-profiler to export profiles and a SpanProcessor extension to link spans and profiles. However Pyroscope does not export profiles via OTLP.

I also considered using inferred-spans component which also uses async-profiler v4.
However inferred-spans exports traces and not profiles.

Additional context

Async-profiler v4 supports exporting profiles in OTLP format: https://github.com/async-profiler/async-profiler/blob/master/docs/OutputFormats.md

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[SylvainJuge]

Being able to use async-profiler for this would definitely be interesting, as it would allow to get profiling data without having to use ebpf to capture data. Here are a few thoughts on what kind of challenges we can expect when implementing this:

I think the biggest challenge here is correlation:

• the correlation between traces/spans and prifiling data is not defined yet: 🏆 Epic: OTel API/SDK Integration opentelemetry-ebpf-instrumentation#1140
• once such correlation is defined, then we could modify/extend async-profiler to automatically capture the current span into the profiling data (it requires a known shared structure, for example in-memory).
• if async-profiler is running from within the same JVM as the application (for example if it's included in a java agent), then we could maybe access the current trace/span ID by relying on SDK/instrumentation implementation details.

Another challenge would be to make this feature work alongside the inferred-spans without interfering with it.

[jack-berg]

cc @jhalliday

[jhalliday]

@kamphaus async-profiler doesn't currently 'export' OTLP in the sense that the SDK does. It's capable of writing the profiling data OTLP message format to disk, i.e. 'exporting' from its in-memory structure to the filesystem. It's not network aware. The OTel Java SDK writes OTLP, but doesn't read it. So this would currently require reading the OTLP using e.g. code generated by google's protoc, then reserializing it through the SDK to wrap it in the expected grpc envelope message type. (It's theoretically possible to directly inline bytes of one serialized protobuf message into another, but the API's don't support doing that) Ironically it's closer to being possible, or at least requiring fewer new dependencies, to go via the JFR format, which async-profiler writes and the SDK reads via the JDK's JFR event API. I prioritized that path because it gets JFR and async-profiler support with one step of work instead of two.

For signal correlation, async-profiler is working on its own metadata context API, which the SDK could use to attach trace ids to the profile. Long term the vision is some form of plugin for the OTel SDK that's trace and profiler-aware and will abstract over the chosen profiler's own API, intercepting SDK trace context storage writes and pushing a copy down to the profiler.

Notes on OpenTelemetry profiling for Java has a more verbose explanation of this and other use cases I'm tracking.

cc @fandreuz