Add support for customizing output and uploading it to remote storage (S3 bucket) #71

AnuradhaKaruppiah · 2025-04-02T22:49:37Z

Description

Workflow, profiling and evaluation results can be customized and uploaded using S3 credentials.
Sample Usage for customizing and uploading:

eval:
  general:
    output:
      dir: ./.tmp/aiq/examples/simple_output/
      # Whether to cleanup the output directory before running the workflow
      cleanup: true
      custom_scripts:
        convert_workflow_to_csv:
          script: examples/simple/src/aiq_simple/scripts/workflow_to_csv.py
          kwargs:
            input: ./.tmp/aiq/examples/simple_output/workflow_output.json
            output: ./.tmp/aiq/examples/simple_output/workflow.csv
      # Upload contents of output directory to remote storage using S3 credentials
      remote_dir: output
      s3:
        endpoint_url: http://10.185.X.X:9000
        bucket: aiq-simple-bucket
        access_key: fake-access-key
        secret_key: fake-secret-key
    dataset:
      _type: json
      # Download dataset from remote storage using S3 credentials
      remote_file_path: input/langsmith.json
      file_path: ./.tmp/aiq/examples/simple_input/langsmith.json
      s3:
        endpoint_url: http://10.185.X.X:9000
        bucket: aiq-simple-bucket
        access_key: fake-access-key
        secret_key: fake-access-key

Sample Output:

2025-04-04 09:23:38,412 - aiq.profiler.profile_runner - INFO - Wrote combined data to: .tmp/aiq/examples/simple_output/all_requests_profiler_traces.json
2025-04-04 09:23:38,542 - aiq.profiler.profile_runner - INFO - Wrote merged standardized DataFrame to .tmp/aiq/examples/simple_output/standardized_data_all.csv
2025-04-04 09:23:38,819 - aiq.profiler.profile_runner - INFO - Wrote inference optimization results to: .tmp/aiq/examples/simple_output/inference_optimization.json███████████████████████████████████████████████████████████| 3/3 [00:07<00:00,  2.04s/it]
2025-04-04 09:23:39,825 - aiq.profiler.profile_runner - INFO - Nested stack analysis complete
2025-04-04 09:23:39,864 - aiq.profiler.profile_runner - INFO - Concurrency spike analysis complete
2025-04-04 09:23:39,865 - aiq.profiler.profile_runner - INFO - Wrote workflow profiling report to: .tmp/aiq/examples/simple_output/workflow_profiling_report.txt
2025-04-04 09:23:39,866 - aiq.profiler.profile_runner - INFO - Wrote workflow profiling metrics to: .tmp/aiq/examples/simple_output/workflow_profiling_metrics.json
2025-04-04 09:23:39,870 - aiq.eval.evaluate - INFO - Workflow output written to .tmp/aiq/examples/simple_output/workflow_output.json
2025-04-04 09:23:39,871 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/rag_relevance_output.json
2025-04-04 09:23:39,871 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/rag_groundedness_output.json
2025-04-04 09:23:39,872 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/rag_accuracy_output.json
2025-04-04 09:23:39,872 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/trajectory_accuracy_output.json
2025-04-04 09:23:39,872 - aiq.eval.utils.output_uploader - INFO - Running custom script: examples/simple/src/aiq_simple/scripts/workflow_to_csv.py examples/simple/src/aiq_simple/scripts/workflow_to_csv.py --input ./.tmp/aiq/examples/simple_output/workflow_output.json --output ./.tmp/aiq/examples/simple_output/workflow.csv
✅ Converted workflow_output.json to workflow.csv
2025-04-04 09:23:39,922 - aiq.eval.utils.output_uploader - INFO - Custom script examples/simple/src/aiq_simple/scripts/workflow_to_csv.py completed successfully.
Uploading files to S3:   0%|                                                                                                                                                                                                         | 0/12 [00:00<?, ?it/s]2025-04-04 09:23:40,492 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/inference_optimization.json to s3://aiq-simple-bucket/output/inference_optimization.json
Uploading files to S3:   8%|████████████████                                                                                                                                                                                 | 1/12 [00:00<00:02,  3.94it/s]2025-04-04 09:23:40,518 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow_profiling_metrics.json to s3://aiq-simple-bucket/output/workflow_profiling_metrics.json
2025-04-04 09:23:40,518 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow.csv to s3://aiq-simple-bucket/output/workflow.csv
2025-04-04 09:23:40,520 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow_profiling_report.txt to s3://aiq-simple-bucket/output/workflow_profiling_report.txt
2025-04-04 09:23:40,600 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/trajectory_accuracy_output.json to s3://aiq-simple-bucket/output/trajectory_accuracy_output.json
Uploading files to S3:  42%|████████████████████████████████████████████████████████████████████████████████▍                                                                                                                | 5/12 [00:00<00:00, 16.44it/s]2025-04-04 09:23:40,614 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/rag_accuracy_output.json to s3://aiq-simple-bucket/output/rag_accuracy_output.json
2025-04-04 09:23:40,627 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/rag_groundedness_output.json to s3://aiq-simple-bucket/output/rag_groundedness_output.json
2025-04-04 09:23:40,867 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/rag_relevance_output.json to s3://aiq-simple-bucket/output/rag_relevance_output.json
Uploading files to S3:  67%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                                                | 8/12 [00:00<00:00, 13.45it/s]2025-04-04 09:23:40,882 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/gantt_chart.png to s3://aiq-simple-bucket/output/gantt_chart.png
2025-04-04 09:23:40,885 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/standardized_data_all.csv to s3://aiq-simple-bucket/output/standardized_data_all.csv
2025-04-04 09:23:40,909 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow_output.json to s3://aiq-simple-bucket/output/workflow_output.json
2025-04-04 09:23:42,068 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/all_requests_profiler_traces.json to s3://aiq-simple-bucket/output/all_requests_profiler_traces.json
Uploading files to S3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:01<00:00,  6.55it/s]

Closes #65, #66

By Submitting this PR I confirm:

I am familiar with the Contributing Guidelines.
We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
- Any contribution which contains commits that are not Signed-Off will not be accepted.
When the PR is ready for review, new or existing tests cover these changes.
When the PR is ready for review, the documentation is up to date with these changes.

Workflow, profiling and evaluation results can be customized and uploaded using S3 credentials. Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

And provide a sample script Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Copilot

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

examples/simple/src/aiq_simple/scripts/worflow_to_csv.py

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

dnandakumar-nv

Everything looks good to me! Just two minor suggestions about the README.

docs/source/concepts/evaluate.md

AnuradhaKaruppiah · 2025-04-04T16:03:15Z

Everything looks good to me! Just two minor suggestions about the README.

Thanks for the review

To clearly state that the contents of the output directory are deleted. Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

AnuradhaKaruppiah · 2025-04-04T16:32:06Z

/merge

… (S3 bucket) (NVIDIA#71) Workflow, profiling and evaluation results can be customized and uploaded using S3 credentials. Sample Usage for customizing and uploading: ``` eval: general: output: dir: ./.tmp/aiq/examples/simple_output/ # Whether to cleanup the output directory before running the workflow cleanup: true custom_scripts: convert_workflow_to_csv: script: examples/simple/src/aiq_simple/scripts/workflow_to_csv.py kwargs: input: ./.tmp/aiq/examples/simple_output/workflow_output.json output: ./.tmp/aiq/examples/simple_output/workflow.csv # Upload contents of output directory to remote storage using S3 credentials remote_dir: output s3: endpoint_url: http://10.185.X.X:9000 bucket: aiq-simple-bucket access_key: fake-access-key secret_key: fake-secret-key dataset: _type: json # Download dataset from remote storage using S3 credentials remote_file_path: input/langsmith.json file_path: ./.tmp/aiq/examples/simple_input/langsmith.json s3: endpoint_url: http://10.185.X.X:9000 bucket: aiq-simple-bucket access_key: fake-access-key secret_key: fake-access-key ``` Sample Output: ``` 2025-04-04 09:23:38,412 - aiq.profiler.profile_runner - INFO - Wrote combined data to: .tmp/aiq/examples/simple_output/all_requests_profiler_traces.json 2025-04-04 09:23:38,542 - aiq.profiler.profile_runner - INFO - Wrote merged standardized DataFrame to .tmp/aiq/examples/simple_output/standardized_data_all.csv 2025-04-04 09:23:38,819 - aiq.profiler.profile_runner - INFO - Wrote inference optimization results to: .tmp/aiq/examples/simple_output/inference_optimization.json███████████████████████████████████████████████████████████| 3/3 [00:07<00:00, 2.04s/it] 2025-04-04 09:23:39,825 - aiq.profiler.profile_runner - INFO - Nested stack analysis complete 2025-04-04 09:23:39,864 - aiq.profiler.profile_runner - INFO - Concurrency spike analysis complete 2025-04-04 09:23:39,865 - aiq.profiler.profile_runner - INFO - Wrote workflow profiling report to: .tmp/aiq/examples/simple_output/workflow_profiling_report.txt 2025-04-04 09:23:39,866 - aiq.profiler.profile_runner - INFO - Wrote workflow profiling metrics to: .tmp/aiq/examples/simple_output/workflow_profiling_metrics.json 2025-04-04 09:23:39,870 - aiq.eval.evaluate - INFO - Workflow output written to .tmp/aiq/examples/simple_output/workflow_output.json 2025-04-04 09:23:39,871 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/rag_relevance_output.json 2025-04-04 09:23:39,871 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/rag_groundedness_output.json 2025-04-04 09:23:39,872 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/rag_accuracy_output.json 2025-04-04 09:23:39,872 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/trajectory_accuracy_output.json 2025-04-04 09:23:39,872 - aiq.eval.utils.output_uploader - INFO - Running custom script: examples/simple/src/aiq_simple/scripts/workflow_to_csv.py examples/simple/src/aiq_simple/scripts/workflow_to_csv.py --input ./.tmp/aiq/examples/simple_output/workflow_output.json --output ./.tmp/aiq/examples/simple_output/workflow.csv ✅ Converted workflow_output.json to workflow.csv 2025-04-04 09:23:39,922 - aiq.eval.utils.output_uploader - INFO - Custom script examples/simple/src/aiq_simple/scripts/workflow_to_csv.py completed successfully. Uploading files to S3: 0%| | 0/12 [00:00<?, ?it/s]2025-04-04 09:23:40,492 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/inference_optimization.json to s3://aiq-simple-bucket/output/inference_optimization.json Uploading files to S3: 8%|████████████████ | 1/12 [00:00<00:02, 3.94it/s]2025-04-04 09:23:40,518 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow_profiling_metrics.json to s3://aiq-simple-bucket/output/workflow_profiling_metrics.json 2025-04-04 09:23:40,518 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow.csv to s3://aiq-simple-bucket/output/workflow.csv 2025-04-04 09:23:40,520 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow_profiling_report.txt to s3://aiq-simple-bucket/output/workflow_profiling_report.txt 2025-04-04 09:23:40,600 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/trajectory_accuracy_output.json to s3://aiq-simple-bucket/output/trajectory_accuracy_output.json Uploading files to S3: 42%|████████████████████████████████████████████████████████████████████████████████▍ | 5/12 [00:00<00:00, 16.44it/s]2025-04-04 09:23:40,614 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/rag_accuracy_output.json to s3://aiq-simple-bucket/output/rag_accuracy_output.json 2025-04-04 09:23:40,627 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/rag_groundedness_output.json to s3://aiq-simple-bucket/output/rag_groundedness_output.json 2025-04-04 09:23:40,867 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/rag_relevance_output.json to s3://aiq-simple-bucket/output/rag_relevance_output.json Uploading files to S3: 67%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 8/12 [00:00<00:00, 13.45it/s]2025-04-04 09:23:40,882 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/gantt_chart.png to s3://aiq-simple-bucket/output/gantt_chart.png 2025-04-04 09:23:40,885 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/standardized_data_all.csv to s3://aiq-simple-bucket/output/standardized_data_all.csv 2025-04-04 09:23:40,909 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow_output.json to s3://aiq-simple-bucket/output/workflow_output.json 2025-04-04 09:23:42,068 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/all_requests_profiler_traces.json to s3://aiq-simple-bucket/output/all_requests_profiler_traces.json Uploading files to S3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:01<00:00, 6.55it/s] ``` Closes NVIDIA#65, NVIDIA#66 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) URL: NVIDIA#71

… (S3 bucket) (NVIDIA#71) Workflow, profiling and evaluation results can be customized and uploaded using S3 credentials. Sample Usage for customizing and uploading: ``` eval: general: output: dir: ./.tmp/aiq/examples/simple_output/ # Whether to cleanup the output directory before running the workflow cleanup: true custom_scripts: convert_workflow_to_csv: script: examples/simple/src/aiq_simple/scripts/workflow_to_csv.py kwargs: input: ./.tmp/aiq/examples/simple_output/workflow_output.json output: ./.tmp/aiq/examples/simple_output/workflow.csv # Upload contents of output directory to remote storage using S3 credentials remote_dir: output s3: endpoint_url: http://10.185.X.X:9000 bucket: aiq-simple-bucket access_key: fake-access-key secret_key: fake-secret-key dataset: _type: json # Download dataset from remote storage using S3 credentials remote_file_path: input/langsmith.json file_path: ./.tmp/aiq/examples/simple_input/langsmith.json s3: endpoint_url: http://10.185.X.X:9000 bucket: aiq-simple-bucket access_key: fake-access-key secret_key: fake-access-key ``` Sample Output: ``` 2025-04-04 09:23:38,412 - aiq.profiler.profile_runner - INFO - Wrote combined data to: .tmp/aiq/examples/simple_output/all_requests_profiler_traces.json 2025-04-04 09:23:38,542 - aiq.profiler.profile_runner - INFO - Wrote merged standardized DataFrame to .tmp/aiq/examples/simple_output/standardized_data_all.csv 2025-04-04 09:23:38,819 - aiq.profiler.profile_runner - INFO - Wrote inference optimization results to: .tmp/aiq/examples/simple_output/inference_optimization.json███████████████████████████████████████████████████████████| 3/3 [00:07<00:00, 2.04s/it] 2025-04-04 09:23:39,825 - aiq.profiler.profile_runner - INFO - Nested stack analysis complete 2025-04-04 09:23:39,864 - aiq.profiler.profile_runner - INFO - Concurrency spike analysis complete 2025-04-04 09:23:39,865 - aiq.profiler.profile_runner - INFO - Wrote workflow profiling report to: .tmp/aiq/examples/simple_output/workflow_profiling_report.txt 2025-04-04 09:23:39,866 - aiq.profiler.profile_runner - INFO - Wrote workflow profiling metrics to: .tmp/aiq/examples/simple_output/workflow_profiling_metrics.json 2025-04-04 09:23:39,870 - aiq.eval.evaluate - INFO - Workflow output written to .tmp/aiq/examples/simple_output/workflow_output.json 2025-04-04 09:23:39,871 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/rag_relevance_output.json 2025-04-04 09:23:39,871 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/rag_groundedness_output.json 2025-04-04 09:23:39,872 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/rag_accuracy_output.json 2025-04-04 09:23:39,872 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/trajectory_accuracy_output.json 2025-04-04 09:23:39,872 - aiq.eval.utils.output_uploader - INFO - Running custom script: examples/simple/src/aiq_simple/scripts/workflow_to_csv.py examples/simple/src/aiq_simple/scripts/workflow_to_csv.py --input ./.tmp/aiq/examples/simple_output/workflow_output.json --output ./.tmp/aiq/examples/simple_output/workflow.csv ✅ Converted workflow_output.json to workflow.csv 2025-04-04 09:23:39,922 - aiq.eval.utils.output_uploader - INFO - Custom script examples/simple/src/aiq_simple/scripts/workflow_to_csv.py completed successfully. Uploading files to S3: 0%| | 0/12 [00:00<?, ?it/s]2025-04-04 09:23:40,492 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/inference_optimization.json to s3://aiq-simple-bucket/output/inference_optimization.json Uploading files to S3: 8%|████████████████ | 1/12 [00:00<00:02, 3.94it/s]2025-04-04 09:23:40,518 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow_profiling_metrics.json to s3://aiq-simple-bucket/output/workflow_profiling_metrics.json 2025-04-04 09:23:40,518 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow.csv to s3://aiq-simple-bucket/output/workflow.csv 2025-04-04 09:23:40,520 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow_profiling_report.txt to s3://aiq-simple-bucket/output/workflow_profiling_report.txt 2025-04-04 09:23:40,600 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/trajectory_accuracy_output.json to s3://aiq-simple-bucket/output/trajectory_accuracy_output.json Uploading files to S3: 42%|████████████████████████████████████████████████████████████████████████████████▍ | 5/12 [00:00<00:00, 16.44it/s]2025-04-04 09:23:40,614 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/rag_accuracy_output.json to s3://aiq-simple-bucket/output/rag_accuracy_output.json 2025-04-04 09:23:40,627 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/rag_groundedness_output.json to s3://aiq-simple-bucket/output/rag_groundedness_output.json 2025-04-04 09:23:40,867 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/rag_relevance_output.json to s3://aiq-simple-bucket/output/rag_relevance_output.json Uploading files to S3: 67%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 8/12 [00:00<00:00, 13.45it/s]2025-04-04 09:23:40,882 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/gantt_chart.png to s3://aiq-simple-bucket/output/gantt_chart.png 2025-04-04 09:23:40,885 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/standardized_data_all.csv to s3://aiq-simple-bucket/output/standardized_data_all.csv 2025-04-04 09:23:40,909 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow_output.json to s3://aiq-simple-bucket/output/workflow_output.json 2025-04-04 09:23:42,068 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/all_requests_profiler_traces.json to s3://aiq-simple-bucket/output/all_requests_profiler_traces.json Uploading files to S3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:01<00:00, 6.55it/s] ``` Closes NVIDIA#65, NVIDIA#66 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) URL: NVIDIA#71 Signed-off-by: Yuchen Zhang <134643420+yczhang-nv@users.noreply.github.com>

… (S3 bucket) (NVIDIA#71) Workflow, profiling and evaluation results can be customized and uploaded using S3 credentials. Sample Usage for customizing and uploading: ``` eval: general: output: dir: ./.tmp/aiq/examples/simple_output/ # Whether to cleanup the output directory before running the workflow cleanup: true custom_scripts: convert_workflow_to_csv: script: examples/simple/src/aiq_simple/scripts/workflow_to_csv.py kwargs: input: ./.tmp/aiq/examples/simple_output/workflow_output.json output: ./.tmp/aiq/examples/simple_output/workflow.csv # Upload contents of output directory to remote storage using S3 credentials remote_dir: output s3: endpoint_url: http://10.185.X.X:9000 bucket: aiq-simple-bucket access_key: fake-access-key secret_key: fake-secret-key dataset: _type: json # Download dataset from remote storage using S3 credentials remote_file_path: input/langsmith.json file_path: ./.tmp/aiq/examples/simple_input/langsmith.json s3: endpoint_url: http://10.185.X.X:9000 bucket: aiq-simple-bucket access_key: fake-access-key secret_key: fake-access-key ``` Sample Output: ``` 2025-04-04 09:23:38,412 - aiq.profiler.profile_runner - INFO - Wrote combined data to: .tmp/aiq/examples/simple_output/all_requests_profiler_traces.json 2025-04-04 09:23:38,542 - aiq.profiler.profile_runner - INFO - Wrote merged standardized DataFrame to .tmp/aiq/examples/simple_output/standardized_data_all.csv 2025-04-04 09:23:38,819 - aiq.profiler.profile_runner - INFO - Wrote inference optimization results to: .tmp/aiq/examples/simple_output/inference_optimization.json███████████████████████████████████████████████████████████| 3/3 [00:07<00:00, 2.04s/it] 2025-04-04 09:23:39,825 - aiq.profiler.profile_runner - INFO - Nested stack analysis complete 2025-04-04 09:23:39,864 - aiq.profiler.profile_runner - INFO - Concurrency spike analysis complete 2025-04-04 09:23:39,865 - aiq.profiler.profile_runner - INFO - Wrote workflow profiling report to: .tmp/aiq/examples/simple_output/workflow_profiling_report.txt 2025-04-04 09:23:39,866 - aiq.profiler.profile_runner - INFO - Wrote workflow profiling metrics to: .tmp/aiq/examples/simple_output/workflow_profiling_metrics.json 2025-04-04 09:23:39,870 - aiq.eval.evaluate - INFO - Workflow output written to .tmp/aiq/examples/simple_output/workflow_output.json 2025-04-04 09:23:39,871 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/rag_relevance_output.json 2025-04-04 09:23:39,871 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/rag_groundedness_output.json 2025-04-04 09:23:39,872 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/rag_accuracy_output.json 2025-04-04 09:23:39,872 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple_output/trajectory_accuracy_output.json 2025-04-04 09:23:39,872 - aiq.eval.utils.output_uploader - INFO - Running custom script: examples/simple/src/aiq_simple/scripts/workflow_to_csv.py examples/simple/src/aiq_simple/scripts/workflow_to_csv.py --input ./.tmp/aiq/examples/simple_output/workflow_output.json --output ./.tmp/aiq/examples/simple_output/workflow.csv ✅ Converted workflow_output.json to workflow.csv 2025-04-04 09:23:39,922 - aiq.eval.utils.output_uploader - INFO - Custom script examples/simple/src/aiq_simple/scripts/workflow_to_csv.py completed successfully. Uploading files to S3: 0%| | 0/12 [00:00<?, ?it/s]2025-04-04 09:23:40,492 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/inference_optimization.json to s3://aiq-simple-bucket/output/inference_optimization.json Uploading files to S3: 8%|████████████████ | 1/12 [00:00<00:02, 3.94it/s]2025-04-04 09:23:40,518 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow_profiling_metrics.json to s3://aiq-simple-bucket/output/workflow_profiling_metrics.json 2025-04-04 09:23:40,518 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow.csv to s3://aiq-simple-bucket/output/workflow.csv 2025-04-04 09:23:40,520 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow_profiling_report.txt to s3://aiq-simple-bucket/output/workflow_profiling_report.txt 2025-04-04 09:23:40,600 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/trajectory_accuracy_output.json to s3://aiq-simple-bucket/output/trajectory_accuracy_output.json Uploading files to S3: 42%|████████████████████████████████████████████████████████████████████████████████▍ | 5/12 [00:00<00:00, 16.44it/s]2025-04-04 09:23:40,614 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/rag_accuracy_output.json to s3://aiq-simple-bucket/output/rag_accuracy_output.json 2025-04-04 09:23:40,627 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/rag_groundedness_output.json to s3://aiq-simple-bucket/output/rag_groundedness_output.json 2025-04-04 09:23:40,867 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/rag_relevance_output.json to s3://aiq-simple-bucket/output/rag_relevance_output.json Uploading files to S3: 67%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 8/12 [00:00<00:00, 13.45it/s]2025-04-04 09:23:40,882 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/gantt_chart.png to s3://aiq-simple-bucket/output/gantt_chart.png 2025-04-04 09:23:40,885 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/standardized_data_all.csv to s3://aiq-simple-bucket/output/standardized_data_all.csv 2025-04-04 09:23:40,909 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/workflow_output.json to s3://aiq-simple-bucket/output/workflow_output.json 2025-04-04 09:23:42,068 - aiq.eval.utils.output_uploader - INFO - Uploaded .tmp/aiq/examples/simple_output/all_requests_profiler_traces.json to s3://aiq-simple-bucket/output/all_requests_profiler_traces.json Uploading files to S3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:01<00:00, 6.55it/s] ``` Closes NVIDIA#65, NVIDIA#66 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) URL: NVIDIA#71

Add config for customizing output and uploading to a S3 bucket

b34aa87

Workflow, profiling and evaluation results can be customized and uploaded using S3 credentials. Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

AnuradhaKaruppiah requested a review from a team as a code owner April 2, 2025 22:49

Overwrite the output_dir with output.dir if present

8911188

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

AnuradhaKaruppiah added feature request New feature or request non-breaking Non-breaking change labels Apr 2, 2025

AnuradhaKaruppiah added 12 commits April 2, 2025 16:41

Fix import

7b6247e

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Fix autoformatting done by editor

232b91d

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Miscellaneous fixes for upload to s3 bucket

e972b9e

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Make upload optional based on config

39b74b4

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Fixes for output customizing

fe90928

And provide a sample script Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Merge remote-tracking branch 'upstream/develop' into s3-upload

dcec813

Unit tests for uploading the output directory via s3

44e5673

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Move the output upload to aioboto3

9f7a8d1

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Update tests for aioboto

e72c165

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Miscellaneous test fixes

a7f6ad9

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Update READMEs with info on s3 upload and output customization

0039757

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Provide sample config file for using remote storage

b1c76aa

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

AnuradhaKaruppiah requested a review from Copilot April 4, 2025 15:32

Copilot AI reviewed Apr 4, 2025

View reviewed changes

examples/simple/src/aiq_simple/scripts/worflow_to_csv.py Show resolved Hide resolved

Fix a spelling mistake in the sample script file name

8eb2571

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

AnuradhaKaruppiah changed the title ~~Add config for customizing output and uploading it to a S3 bucket~~ Add support for customizing output and uploading it to remote storage (S3 bucket) Apr 4, 2025

AnuradhaKaruppiah requested a review from dnandakumar-nv April 4, 2025 15:53

dnandakumar-nv approved these changes Apr 4, 2025

View reviewed changes

docs/source/concepts/evaluate.md Show resolved Hide resolved

docs/source/concepts/evaluate.md Show resolved Hide resolved

AnuradhaKaruppiah added 3 commits April 4, 2025 09:09

Updated the cleanup description

9c8451f

To clearly state that the contents of the output directory are deleted. Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Update doc for output customization

55e05a7

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Fix location of remote_dir in the sample config

fce20c8

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

rapids-bot bot merged commit 238a3a8 into NVIDIA:develop Apr 4, 2025
8 of 9 checks passed

AnuradhaKaruppiah mentioned this pull request Apr 4, 2025

[FEA]: Provide an output customizer #66

Closed

2 tasks

AnuradhaKaruppiah deleted the s3-upload branch May 6, 2025 00:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for customizing output and uploading it to remote storage (S3 bucket) #71

Add support for customizing output and uploading it to remote storage (S3 bucket) #71

AnuradhaKaruppiah commented Apr 2, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

dnandakumar-nv left a comment

Uh oh!

Uh oh!

Uh oh!

AnuradhaKaruppiah commented Apr 4, 2025

Uh oh!

AnuradhaKaruppiah commented Apr 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Add support for customizing output and uploading it to remote storage (S3 bucket) #71

Add support for customizing output and uploading it to remote storage (S3 bucket) #71

Conversation

AnuradhaKaruppiah commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

By Submitting this PR I confirm:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dnandakumar-nv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AnuradhaKaruppiah commented Apr 4, 2025

Uh oh!

AnuradhaKaruppiah commented Apr 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AnuradhaKaruppiah commented Apr 2, 2025 •

edited

Loading