…NVIDIA#379)

Closes: NVIDIA#378

This PR introduces a complete overhaul of the NeMo Agent toolkit's observability infrastructure, providing a robust, scalable, and extensible framework for monitoring and tracing workflow execution.

## Key Improvements

- **Backwards Compatibility**: Maintains full backward compatibility while adding significant improvements in extensibility
- **Modular Architecture**: Plugin-based system supporting multiple telemetry backends
- **Reduced Dependencies**: Core observability no longer requires OpenTelemetry - now available as optional plugin
- **High Performance**: Copy-on-write isolation enables efficient concurrent execution
- **Extensible**: Type-safe processor pipelines for flexible data transformation
- **Multiple Backends**: Built-in support for OpenTelemetry, Phoenix, and Weave

## New Architecture

### Class/Plugin Hierarchy
```
Exporter (abstract interface)
└── BaseExporter (abstract)
    └── ProcessingExporter (abstract)
        ├── SpanExporter (abstract)
        │   ├── OtelSpanExporter (abstract)
        │   │   ├── PhoenixOtelExporter (concrete)
        │   │   │   └── phoenix (plugin)
        │   │   ├── OTLPSpanAdapterExporter (concrete)
        │   │   │   ├── langfuse (plugin)
        │   │   │   ├── langsmith (plugin)
        │   │   │   ├── otelcollector (plugin)
        │   │   │   ├── patronus (plugin)
        │   │   │   └── galileo (plugin)
        │   │   └── RagaAICatalystExporter (concrete)
        │   │       └── catalyst (plugin)
        │   └── WeaveExporter (concrete)
        │       └── weave (plugin)
        └── RawExporter (abstract)
            └── FileExporter (concrete)
                └── file (plugin)
```

### Core Components

#### ExporterManager

- Manages exporter lifecycle and concurrent execution
- Copy-on-write isolation for concurrent workflows
- Automatic cleanup and resource management

#### Processing Pipeline

Processing pipelines enable flexible data transformation chains where events can be modified, enriched, or reformatted before export. This allows exporters to adapt data for different backends without duplicating transformation logic.

- `Processor[InputT, OutputT]` generic transformation interface
- `BatchingProcessor` for efficient dynamic batching of traces
- Type-safe processor chains

#### Copy-on-Write Isolation

```
ExporterManager
├── Original Exporters (Shared Registry)
│   ├── Shared: HTTP Clients, Auth, Configuration
│   └── create_isolated_exporters()
│       ├── Workflow 1 → Isolated Instance (private tasks, events, buffers)
│       ├── Workflow 2 → Isolated Instance (private tasks, events, buffers)
│       └── Workflow N → Isolated Instance (private tasks, events, buffers)
└── Benefits: Fast (shared resources) + Safe (isolated state)
```

#### Performance Benefits

- **Fast**: Shares expensive resources (HTTP clients, auth)
- **Safe**: Isolates mutable state (tasks, events, buffers)
- **Memory Efficient**: Minimal overhead for isolation
- **Concurrent**: Enables parallel workflow execution


#### Core Plugin Exporters

- **FileExporter**: Local file export for debugging

### Plugin Exporters

- **OpenTelemetry**: Standard OTEL span export with OTLP adapter
- **Phoenix**: AI observability platform integration
- **Weave**: Weights & Biases integration

## Integration

The system integrates seamlessly with existing NAT runtime:

```python
# In AIQRunner
async with self._exporter_manager.start(context_state=self._context_state):
    result = await self._entry_fn.ainvoke(self._input_message, to_type=to_type)

# In WorkflowBuilder
for key, exporter_config in telemetry_config.tracing.items():
    await self.add_exporter(key, exporter_config)
```

Also included are minor bug fixes:

- Fixing `Function.astream` to set final output of `ActiveFunctionContextManager` for streaming requests
- Fixing `Function.astream` to use function runtime instance name to remove ambiguity in traces (previously the config type was used)

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Matthew Penn (https://github.com/mpenn)
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: NVIDIA#379

…NVIDIA#379)

Closes: NVIDIA#378

This PR introduces a complete overhaul of the NeMo Agent toolkit's observability infrastructure, providing a robust, scalable, and extensible framework for monitoring and tracing workflow execution.

## Key Improvements

- **Backwards Compatibility**: Maintains full backward compatibility while adding significant improvements in extensibility
- **Modular Architecture**: Plugin-based system supporting multiple telemetry backends
- **Reduced Dependencies**: Core observability no longer requires OpenTelemetry - now available as optional plugin
- **High Performance**: Copy-on-write isolation enables efficient concurrent execution
- **Extensible**: Type-safe processor pipelines for flexible data transformation
- **Multiple Backends**: Built-in support for OpenTelemetry, Phoenix, and Weave

## New Architecture

### Class/Plugin Hierarchy
```
Exporter (abstract interface)
└── BaseExporter (abstract)
    └── ProcessingExporter (abstract)
        ├── SpanExporter (abstract)
        │   ├── OtelSpanExporter (abstract)
        │   │   ├── PhoenixOtelExporter (concrete)
        │   │   │   └── phoenix (plugin)
        │   │   ├── OTLPSpanAdapterExporter (concrete)
        │   │   │   ├── langfuse (plugin)
        │   │   │   ├── langsmith (plugin)
        │   │   │   ├── otelcollector (plugin)
        │   │   │   ├── patronus (plugin)
        │   │   │   └── galileo (plugin)
        │   │   └── RagaAICatalystExporter (concrete)
        │   │       └── catalyst (plugin)
        │   └── WeaveExporter (concrete)
        │       └── weave (plugin)
        └── RawExporter (abstract)
            └── FileExporter (concrete)
                └── file (plugin)
```

### Core Components

#### ExporterManager

- Manages exporter lifecycle and concurrent execution
- Copy-on-write isolation for concurrent workflows
- Automatic cleanup and resource management

#### Processing Pipeline

Processing pipelines enable flexible data transformation chains where events can be modified, enriched, or reformatted before export. This allows exporters to adapt data for different backends without duplicating transformation logic.

- `Processor[InputT, OutputT]` generic transformation interface
- `BatchingProcessor` for efficient dynamic batching of traces
- Type-safe processor chains

#### Copy-on-Write Isolation

```
ExporterManager
├── Original Exporters (Shared Registry)
│   ├── Shared: HTTP Clients, Auth, Configuration
│   └── create_isolated_exporters()
│       ├── Workflow 1 → Isolated Instance (private tasks, events, buffers)
│       ├── Workflow 2 → Isolated Instance (private tasks, events, buffers)
│       └── Workflow N → Isolated Instance (private tasks, events, buffers)
└── Benefits: Fast (shared resources) + Safe (isolated state)
```

#### Performance Benefits

- **Fast**: Shares expensive resources (HTTP clients, auth)
- **Safe**: Isolates mutable state (tasks, events, buffers)
- **Memory Efficient**: Minimal overhead for isolation
- **Concurrent**: Enables parallel workflow execution


#### Core Plugin Exporters

- **FileExporter**: Local file export for debugging

### Plugin Exporters

- **OpenTelemetry**: Standard OTEL span export with OTLP adapter
- **Phoenix**: AI observability platform integration
- **Weave**: Weights & Biases integration

## Integration

The system integrates seamlessly with existing NAT runtime:

```python
# In AIQRunner
async with self._exporter_manager.start(context_state=self._context_state):
    result = await self._entry_fn.ainvoke(self._input_message, to_type=to_type)

# In WorkflowBuilder
for key, exporter_config in telemetry_config.tracing.items():
    await self.add_exporter(key, exporter_config)
```

Also included are minor bug fixes:

- Fixing `Function.astream` to set final output of `ActiveFunctionContextManager` for streaming requests
- Fixing `Function.astream` to use function runtime instance name to remove ambiguity in traces (previously the config type was used)

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Matthew Penn (https://github.com/mpenn)
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: NVIDIA#379

…NVIDIA#379)

Closes: NVIDIA#378

This PR introduces a complete overhaul of the NeMo Agent toolkit's observability infrastructure, providing a robust, scalable, and extensible framework for monitoring and tracing workflow execution.

- **Backwards Compatibility**: Maintains full backward compatibility while adding significant improvements in extensibility
- **Modular Architecture**: Plugin-based system supporting multiple telemetry backends
- **Reduced Dependencies**: Core observability no longer requires OpenTelemetry - now available as optional plugin
- **High Performance**: Copy-on-write isolation enables efficient concurrent execution
- **Extensible**: Type-safe processor pipelines for flexible data transformation
- **Multiple Backends**: Built-in support for OpenTelemetry, Phoenix, and Weave

```
Exporter (abstract interface)
└── BaseExporter (abstract)
    └── ProcessingExporter (abstract)
        ├── SpanExporter (abstract)
        │   ├── OtelSpanExporter (abstract)
        │   │   ├── PhoenixOtelExporter (concrete)
        │   │   │   └── phoenix (plugin)
        │   │   ├── OTLPSpanAdapterExporter (concrete)
        │   │   │   ├── langfuse (plugin)
        │   │   │   ├── langsmith (plugin)
        │   │   │   ├── otelcollector (plugin)
        │   │   │   ├── patronus (plugin)
        │   │   │   └── galileo (plugin)
        │   │   └── RagaAICatalystExporter (concrete)
        │   │       └── catalyst (plugin)
        │   └── WeaveExporter (concrete)
        │       └── weave (plugin)
        └── RawExporter (abstract)
            └── FileExporter (concrete)
                └── file (plugin)
```

- Manages exporter lifecycle and concurrent execution
- Copy-on-write isolation for concurrent workflows
- Automatic cleanup and resource management

Processing pipelines enable flexible data transformation chains where events can be modified, enriched, or reformatted before export. This allows exporters to adapt data for different backends without duplicating transformation logic.

- `Processor[InputT, OutputT]` generic transformation interface
- `BatchingProcessor` for efficient dynamic batching of traces
- Type-safe processor chains

```
ExporterManager
├── Original Exporters (Shared Registry)
│   ├── Shared: HTTP Clients, Auth, Configuration
│   └── create_isolated_exporters()
│       ├── Workflow 1 → Isolated Instance (private tasks, events, buffers)
│       ├── Workflow 2 → Isolated Instance (private tasks, events, buffers)
│       └── Workflow N → Isolated Instance (private tasks, events, buffers)
└── Benefits: Fast (shared resources) + Safe (isolated state)
```

- **Fast**: Shares expensive resources (HTTP clients, auth)
- **Safe**: Isolates mutable state (tasks, events, buffers)
- **Memory Efficient**: Minimal overhead for isolation
- **Concurrent**: Enables parallel workflow execution

- **FileExporter**: Local file export for debugging

- **OpenTelemetry**: Standard OTEL span export with OTLP adapter
- **Phoenix**: AI observability platform integration
- **Weave**: Weights & Biases integration

The system integrates seamlessly with existing NAT runtime:

```python
async with self._exporter_manager.start(context_state=self._context_state):
    result = await self._entry_fn.ainvoke(self._input_message, to_type=to_type)

for key, exporter_config in telemetry_config.tracing.items():
    await self.add_exporter(key, exporter_config)
```

Also included are minor bug fixes:

- Fixing `Function.astream` to set final output of `ActiveFunctionContextManager` for streaming requests
- Fixing `Function.astream` to use function runtime instance name to remove ambiguity in traces (previously the config type was used)

- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Matthew Penn (https://github.com/mpenn)
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: NVIDIA#379