Background

AsyncGetCallTrace (AGCT) is a non-standard extension of HotSpot JVM to obtain Java stack traces outside safepoints. async-profiler has been relying on AGCT heavily, it even got its name after this function.

In the meantime, being non-API, AsyncGetCallTrace was never supported in OpenJDK well enough: it did not receive enough testing, it was broken several times even in minor JDK updates, e.g. JDK-8307549.

AsyncGetCallTrace is notorious for inability to walk Java stack in different corner cases. There is a long-standing bug JDK-8178287 with several examples. But the worst thing is that AsyncGetCallTrace can crash JVM, and there is no reliable way to get around this outside the JVM.

Furthermore, AsyncGetCallTrace has limited capabilities. It does not provide information about Java frame type (interpreted/compiled/inlined). It skips all non-Java frames, so it's impossible to see JNI or JVM internal frames in the middle of the call stack.

It was previously proposed in #66 to re-implement stack walking in async-profiler without the problematic API. However, as async-profiler got workarounds for different shortcomings of AsyncGetCallTrace, the re-implementation was postponed. But after years, we still see issues with AGCT from time to time, including random crashes and missing stack traces.

Proposal

Get rid of the dependency on AsyncGetCallTrace entirely, at least, for HotSpot JVM. Leverage VMStructs for getting offsets of the HotSpot structures essential for stack walking. Implement stack walking externally like Serviceability Agent does.

Requirements

1. Reliability. External stack walker must be protected against JVM crashes.
2. Quality. Collected stack traces should be at least as accurate as with AGCT; the amount of "unknown" states should be minimal.
3. Completeness. New implementation should display all frames: Java, native and JVM stubs throughout the whole stack.
4. Details. Stack walker should provide additional information on each frame, like JIT compilation type.
5. Performance. While the performance of the external implementation may be inferior to the JVM built-in one, it should be fast enough for production profiling.

Target platforms

The implementation targets HotSpot JVM on Linux and macOS, both x64 and ARM64.
It is not required to provide new stack walker for other platforms and other JVM implementations.

Example

This is how mixed stack traces may look on a flame graph: