It is not obvious to me how the fact that profile is dumped for the first time affects aggregation_temporality in the output data. IMO, these two things are not quite related; they are on different layers, and one depending on another seems to be a leaky abstraction.

Example 1:

User starts profiling (e.g. using an agent, but this does not matter), then, after a few seconds, runs asprof dump -f test.html just to verify that profiler is working, then after one more minute an agent gets OTLP data. In this case, OTLP profile will have aggregation_temporality=cumulative. If a user did not run asprof dump, the same profile would have aggregation_temporality=delta, even though the rest of the data would be identical to the first case.

Example 2:

Consider an agent that restarts profiling session and after 30 seconds calls dumpOtlp. Since it's a new chunk of data after restart, it will have aggregation_temporality=delta.
In an alternative implementation, an agents restarts profiling session and calls dumpOtlp immediately. It will get an empty profile with aggregation_temporality=delta. Then after 30 seconds the agent calls dumpOtlp again and gets a meaningful profile with aggregation_temporality=cumulative. Note that profile data itself will not be any different from a case when no immediate dumpOtlp call is made.
If both implementations are valid according to the proposed solution, then an agent could just emit an empty delta profile after every restart and let async-profiler always dump cumulative profile. I'm not saying this will be the right solution, this example is just to demonstrate that the logic of _first_dump dependency is likely flawed.

Yeah, I thought about such a case. I'm not sure there's a clean solution to this problem. You could look things from this perspective: dump formats may support the possibility to tell the consumer whether it's a cumulative or a delta dump. OTLP does, so we set it. But if the first dump happens to use a format that does not support it, the information is simply lost. This seems more like misuse on the user side, which could also happen in other ways: what if the user forgets to export the packet containing the first dump after a restart?

Regarding your second example, I see just one flaw in the reasoning: profiles produced by Async-Profiler itself won't adhere to the protocol. The existence of an external agent handling the flow is implicitly assumed, which makes this solution a bit less convenient.

Other options we considered:

• set first_dump=false only in dumpOtlp: I didn't particularly like this, but it would solve the problem you highlighted in the first example
• no flag in Async-Profiler, the agent is responsible for restarting the profiler and setting DELTA in post-processing as necessary: this implies implementing something like loop in the exporter, which felt a bit like reinventing what is already there.

My point is, delta/cumulative is a higher level concept, it is meaningless without baseline.
Async-profiler does not submit data itself, it does not know what was submitted previously.

If I stop profiler and then run it again after 1 hour - why it should be treated as "delta" from the previous report, if there was a long gap when no data was collected at all? Or if the agent submits 3 packets: cumulative, delta, cumulative - how will receiver recognize that the last cumulative report is counted from the last delta and not from the beginning? (without a profile timestamp which you don't currently report)

To me, a scenario which you've described as "misuse" is completely valid. Imagine I've configured an agent to report profiles to a remote server, but don't see data in the UI. To check if the issue is on the collection side, I run asprof dump to see if profiler is running and gathering data.

I believe the following approach makes up a simple and unambiguous solution:

1. dumpOtlp always produces messages with AGGREGATION_TEMPORALITY_CUMULATIVE;
2. Profile should include time_nanos and duration_nanos fields, where time_nanos is the beginning of the profiling session.
3. Consecutive dumps will have the same time_nanos, but when profiler restarts, time_nanos will change to represent the beginning of the new session. This way, we'll disambiguate when cumulative is aggregated from.

The example in profiles.proto describes exactly our case, if we replace "fault" with "restart".

Yes, I see now that time_nanos + duration_nanos is a less ambiguous way to express the same concept. We'll amend this PR with the required changes.

I'll also check with the OTEL profiling community, maybe this aggregation temporality thing could be removed from the protocol altogether, or its semantics clarified. The eBPF profiler does not use it.

@apangin are you ok with changing _start_time to contain nanoseconds in profiler.cpp? Otherwise, we could have a separate field.

All good now, thanks