Summary:

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:
Pull Request resolved: pytorch#161209

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:
Pull Request resolved: pytorch#161209

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:
Pull Request resolved: pytorch#161209

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:
Pull Request resolved: pytorch#161209

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:
Pull Request resolved: pytorch#161209

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:

Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Future Work: Add more metadata to the GC event

Test Plan:
Ran trace with GC induced and saw it on trace. https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2Fdevgpu003.rva5.facebook.com%2Frank-0.Aug_21_13_52_27.3127794.pt.trace.json.gz&bucket=gpu_traces

Also added a test

Rollback Plan:

Reviewed By: ngimel

Differential Revision: D80491146

Summary:
Adds Python Garbage Collection to Kineto Traces and Profiler FunctionEvents. Create custom cpp callback in profiler_python.cpp. Then define a python function with cpp and register that callback for all python garbage collection. We don't worry about thread safety in this case because we are only doing init/teardown for main thread while holding GIL.

Currently we are hiding this behind experimental config because python tracing tends to be unstable especially when adding any new feature. If this is found to not add too much overhead we can set this to on by default. NOTE: To enable this you need both with_stack=True and the experimental config on!

Test Plan:
Ran trace with GC induced and saw it on trace

Also added a test

Rollback Plan:

Differential Revision: D80491146

Pull Request resolved: pytorch#161209
Approved by: https://github.com/ngimel