Add an async /evaluate endpoint to trigger evaluation jobs on a remote cluster #109

AnuradhaKaruppiah · 2025-04-09T22:20:13Z

Description

Sample Usage:

Start the workflow service on any cluster via aiq serve-

aiq serve --config_file=examples/simple/configs/config.yml

Send evaluation request to it using a separate (or same) eval_config file -
That will return immediately with a job id. The job is run as a background task

curl --request POST   --url http://localhost:8000/evaluate   --header 'Content-Type: application/json'   --data '{
    "config_file": "examples/simple/configs/eval_config.yml"
}'
{"job_id":"551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d","status":"submitted"}

You can then query the status by using the job-id or my asking for the status of the last (created) job -

curl --request GET --url http://localhost:8000/evaluate/job/last |jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   242  100   242    0     0    204      0  0:00:01  0:00:01 --:--:--   204
{
  "job_id": "551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d",
  "status": "running",
  "config_file": "examples/simple/configs/eval_config.yml",
  "error": null,
  "output_path": "None",
  "created_at": "2025-04-10T03:56:43.931778",
  "updated_at": "2025-04-10T03:56:43.933886"
}

curl --request GET --url http://localhost:8000/evaluate/job/551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d |jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   283  100   283    0     0     74      0  0:00:03  0:00:03 --:--:--    74
{
  "job_id": "551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d",
  "status": "success",
  "config_file": "examples/simple/configs/eval_config.yml",
  "error": null,
  "output_path": ".tmp/aiq/examples/simple/workflow_output.json",
  "created_at": "2025-04-10T03:56:43.931778",
  "updated_at": "2025-04-10T03:57:11.722188"
}

You can get a complete list of jobs by running -

curl --request GET --url http://localhost:8000/evaluate/jobs | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   569  100   569    0     0  47535      0 --:--:-- --:--:-- --:--:-- 56900
[
  {
    "job_id": "a62086d3-3373-485f-95b1-80e0477fac6b",
    "status": "success",
    "config_file": "examples/simple/configs/eval_config.yml",
    "error": null,
    "output_path": ".tmp/aiq/examples/simple/workflow_output.json",
    "created_at": "2025-04-10T04:43:29.741201",
    "updated_at": "2025-04-10T04:43:58.451151"
  },
  {
    "job_id": "a5a11719-cd8d-468d-b1c2-9855ab2e9927",
    "status": "success",
    "config_file": "examples/simple/configs/eval_config.yml",
    "error": null,
    "output_path": ".tmp/aiq/examples/simple/workflow_output.json",
    "created_at": "2025-04-10T04:43:35.432887",
    "updated_at": "2025-04-10T04:44:04.429978"
  }
]

This endpoint allows you to filter by status -

curl http://localGET --url http://localhost:8000/evaluate/jobs?status=running |jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   244  100   244    0     0   3870      0 --:--:-- --:--:-- --:--:--  3935
[
  {
    "job_id": "1cff39fe-bedb-4089-9034-50377abc8f6b",
    "status": "running",
    "config_file": "examples/simple/configs/eval_config.yml",
    "error": null,
    "output_path": "None",
    "created_at": "2025-04-10T04:45:06.708431",
    "updated_at": "2025-04-10T04:45:06.710928"
  }
]

Closes: #100

By Submitting this PR I confirm:

I am familiar with the Contributing Guidelines.
We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
- Any contribution which contains commits that are not Signed-Off will not be accepted.
When the PR is ready for review, new or existing tests cover these changes.
When the PR is ready for review, the documentation is up to date with these changes.

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Also update status Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Also allow user to specify the job id Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

This is a minor optimization to prevent the periodic check. Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

AnuradhaKaruppiah · 2025-04-11T22:25:01Z

/merge

…e cluster (NVIDIA#109) Sample Usage: 1. Start the workflow service on any cluster via `aiq serve`- ``` aiq serve --config_file=examples/simple/configs/config.yml ``` 2. Send evaluation request to it using a separate (or same) eval_config file - That will return immediately with a job id. The job is run as a background task ``` curl --request POST --url http://localhost:8000/evaluate --header 'Content-Type: application/json' --data '{ "config_file": "examples/simple/configs/eval_config.yml" }' {"job_id":"551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d","status":"submitted"} ``` 3. You can then query the status by using the job-id or my asking for the status of the last (created) job - ``` curl --request GET --url http://localhost:8000/evaluate/job/last |jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 242 100 242 0 0 204 0 0:00:01 0:00:01 --:--:-- 204 { "job_id": "551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d", "status": "running", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": "None", "created_at": "2025-04-10T03:56:43.931778", "updated_at": "2025-04-10T03:56:43.933886" } ``` ``` curl --request GET --url http://localhost:8000/evaluate/job/551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d |jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 283 100 283 0 0 74 0 0:00:03 0:00:03 --:--:-- 74 { "job_id": "551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d", "status": "success", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": ".tmp/aiq/examples/simple/workflow_output.json", "created_at": "2025-04-10T03:56:43.931778", "updated_at": "2025-04-10T03:57:11.722188" } ``` 4. You can get a complete list of jobs by running - ``` curl --request GET --url http://localhost:8000/evaluate/jobs | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 569 100 569 0 0 47535 0 --:--:-- --:--:-- --:--:-- 56900 [ { "job_id": "a62086d3-3373-485f-95b1-80e0477fac6b", "status": "success", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": ".tmp/aiq/examples/simple/workflow_output.json", "created_at": "2025-04-10T04:43:29.741201", "updated_at": "2025-04-10T04:43:58.451151" }, { "job_id": "a5a11719-cd8d-468d-b1c2-9855ab2e9927", "status": "success", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": ".tmp/aiq/examples/simple/workflow_output.json", "created_at": "2025-04-10T04:43:35.432887", "updated_at": "2025-04-10T04:44:04.429978" } ] ``` This endpoint allows you to filter by status - ``` curl http://localGET --url http://localhost:8000/evaluate/jobs?status=running |jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 244 100 244 0 0 3870 0 --:--:-- --:--:-- --:--:-- 3935 [ { "job_id": "1cff39fe-bedb-4089-9034-50377abc8f6b", "status": "running", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": "None", "created_at": "2025-04-10T04:45:06.708431", "updated_at": "2025-04-10T04:45:06.710928" } ] ``` Closes: NVIDIA#100 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Eric Evans II (https://github.com/ericevans-nv) - David Gardner (https://github.com/dagardner-nv) URL: NVIDIA#109 Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com>

…e cluster (NVIDIA#109) Sample Usage: 1. Start the workflow service on any cluster via `aiq serve`- ``` aiq serve --config_file=examples/simple/configs/config.yml ``` 2. Send evaluation request to it using a separate (or same) eval_config file - That will return immediately with a job id. The job is run as a background task ``` curl --request POST --url http://localhost:8000/evaluate --header 'Content-Type: application/json' --data '{ "config_file": "examples/simple/configs/eval_config.yml" }' {"job_id":"551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d","status":"submitted"} ``` 3. You can then query the status by using the job-id or my asking for the status of the last (created) job - ``` curl --request GET --url http://localhost:8000/evaluate/job/last |jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 242 100 242 0 0 204 0 0:00:01 0:00:01 --:--:-- 204 { "job_id": "551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d", "status": "running", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": "None", "created_at": "2025-04-10T03:56:43.931778", "updated_at": "2025-04-10T03:56:43.933886" } ``` ``` curl --request GET --url http://localhost:8000/evaluate/job/551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d |jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 283 100 283 0 0 74 0 0:00:03 0:00:03 --:--:-- 74 { "job_id": "551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d", "status": "success", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": ".tmp/aiq/examples/simple/workflow_output.json", "created_at": "2025-04-10T03:56:43.931778", "updated_at": "2025-04-10T03:57:11.722188" } ``` 4. You can get a complete list of jobs by running - ``` curl --request GET --url http://localhost:8000/evaluate/jobs | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 569 100 569 0 0 47535 0 --:--:-- --:--:-- --:--:-- 56900 [ { "job_id": "a62086d3-3373-485f-95b1-80e0477fac6b", "status": "success", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": ".tmp/aiq/examples/simple/workflow_output.json", "created_at": "2025-04-10T04:43:29.741201", "updated_at": "2025-04-10T04:43:58.451151" }, { "job_id": "a5a11719-cd8d-468d-b1c2-9855ab2e9927", "status": "success", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": ".tmp/aiq/examples/simple/workflow_output.json", "created_at": "2025-04-10T04:43:35.432887", "updated_at": "2025-04-10T04:44:04.429978" } ] ``` This endpoint allows you to filter by status - ``` curl http://localGET --url http://localhost:8000/evaluate/jobs?status=running |jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 244 100 244 0 0 3870 0 --:--:-- --:--:-- --:--:-- 3935 [ { "job_id": "1cff39fe-bedb-4089-9034-50377abc8f6b", "status": "running", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": "None", "created_at": "2025-04-10T04:45:06.708431", "updated_at": "2025-04-10T04:45:06.710928" } ] ``` Closes: NVIDIA#100 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Eric Evans II (https://github.com/ericevans-nv) - David Gardner (https://github.com/dagardner-nv) URL: NVIDIA#109 Signed-off-by: Yuchen Zhang <134643420+yczhang-nv@users.noreply.github.com>

…e cluster (NVIDIA#109) Sample Usage: 1. Start the workflow service on any cluster via `aiq serve`- ``` aiq serve --config_file=examples/simple/configs/config.yml ``` 2. Send evaluation request to it using a separate (or same) eval_config file - That will return immediately with a job id. The job is run as a background task ``` curl --request POST --url http://localhost:8000/evaluate --header 'Content-Type: application/json' --data '{ "config_file": "examples/simple/configs/eval_config.yml" }' {"job_id":"551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d","status":"submitted"} ``` 3. You can then query the status by using the job-id or my asking for the status of the last (created) job - ``` curl --request GET --url http://localhost:8000/evaluate/job/last |jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 242 100 242 0 0 204 0 0:00:01 0:00:01 --:--:-- 204 { "job_id": "551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d", "status": "running", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": "None", "created_at": "2025-04-10T03:56:43.931778", "updated_at": "2025-04-10T03:56:43.933886" } ``` ``` curl --request GET --url http://localhost:8000/evaluate/job/551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d |jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 283 100 283 0 0 74 0 0:00:03 0:00:03 --:--:-- 74 { "job_id": "551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d", "status": "success", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": ".tmp/aiq/examples/simple/workflow_output.json", "created_at": "2025-04-10T03:56:43.931778", "updated_at": "2025-04-10T03:57:11.722188" } ``` 4. You can get a complete list of jobs by running - ``` curl --request GET --url http://localhost:8000/evaluate/jobs | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 569 100 569 0 0 47535 0 --:--:-- --:--:-- --:--:-- 56900 [ { "job_id": "a62086d3-3373-485f-95b1-80e0477fac6b", "status": "success", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": ".tmp/aiq/examples/simple/workflow_output.json", "created_at": "2025-04-10T04:43:29.741201", "updated_at": "2025-04-10T04:43:58.451151" }, { "job_id": "a5a11719-cd8d-468d-b1c2-9855ab2e9927", "status": "success", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": ".tmp/aiq/examples/simple/workflow_output.json", "created_at": "2025-04-10T04:43:35.432887", "updated_at": "2025-04-10T04:44:04.429978" } ] ``` This endpoint allows you to filter by status - ``` curl http://localGET --url http://localhost:8000/evaluate/jobs?status=running |jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 244 100 244 0 0 3870 0 --:--:-- --:--:-- --:--:-- 3935 [ { "job_id": "1cff39fe-bedb-4089-9034-50377abc8f6b", "status": "running", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": "None", "created_at": "2025-04-10T04:45:06.708431", "updated_at": "2025-04-10T04:45:06.710928" } ] ``` Closes: NVIDIA#100 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Eric Evans II (https://github.com/ericevans-nv) - David Gardner (https://github.com/dagardner-nv) URL: NVIDIA#109

AnuradhaKaruppiah added 2 commits April 8, 2025 19:53

Initial code for /evaluate endpoint

58c1281

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Use the existing session manager for running the workflow.

9ce6648

Also update status Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

AnuradhaKaruppiah requested a review from a team as a code owner April 9, 2025 22:20

AnuradhaKaruppiah marked this pull request as draft April 9, 2025 22:20

AnuradhaKaruppiah added 6 commits April 9, 2025 16:17

Add an evaluate status endpoint to query the status of the job

18c0ed4

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Update the job status to a pydantic model

47aaf1d

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Add an endpoint to query the last created evaluation job

6c1d508

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Miscellaneous fixes

943a7ee

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Rename functions to provide a better short description

1da4dca

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Rename routes and descriptions for clarity

4fcc2c8

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

ericevans-nv approved these changes Apr 10, 2025

View reviewed changes

AnuradhaKaruppiah added 9 commits April 10, 2025 10:06

Add a lock to prevent them one evaluation running at a time

6f9fadf

Also allow user to specify the job id Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Default EvaluationRunConfig to match CLI defaults

17e5946

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Fix issue with user specified job-id handling

2675850

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Allow configurable repitions for the evaluate endpoint

9253e07

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Add a mechanism to expire jobs

4770837

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Display the time at which a job is expected to expire

fd1b2fa

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

If a job already exists return its status

e29b396

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Fix filling expires_at

8e55812

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Unit tests for the evaluate endpoint

c022da3

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

AnuradhaKaruppiah added feature request New feature or request non-breaking Non-breaking change labels Apr 10, 2025

AnuradhaKaruppiah added 3 commits April 10, 2025 15:50

Fix cleanup handling

5a87b80

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Fix output dir handling for output customization

1af79bd

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Create cleanup task on first evaluation job

1c3b29f

This is a minor optimization to prevent the periodic check. Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

AnuradhaKaruppiah marked this pull request as ready for review April 11, 2025 02:30

AnuradhaKaruppiah added 4 commits April 11, 2025 11:09

Add documentation for the evaluation endpoint

9aa8009

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Fix mermaid block

7a7cb8d

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Reduce label size

30907ef

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Fix typos in the documentation

1a0ad4a

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

AnuradhaKaruppiah added 4 commits April 11, 2025 11:36

Rename evaluate_api.md to evaluate-api.md for consistency

d8479c6

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Fix unit test breakage because of adding an arg to the custom scripts

218dc07

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Fix mermaid usage in docs to work with sphinx

2c31979

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

Add sphinx-mermaid to allow mermaid flow diagrams

da3be73

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>

dagardner-nv approved these changes Apr 11, 2025

View reviewed changes

rapids-bot bot merged commit af05dc9 into NVIDIA:develop Apr 11, 2025
10 checks passed

AnuradhaKaruppiah deleted the evaluate-endpoint branch May 6, 2025 00:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add an async /evaluate endpoint to trigger evaluation jobs on a remote cluster #109

Add an async /evaluate endpoint to trigger evaluation jobs on a remote cluster #109

AnuradhaKaruppiah commented Apr 9, 2025 •

edited

Loading

Uh oh!

AnuradhaKaruppiah commented Apr 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add an async /evaluate endpoint to trigger evaluation jobs on a remote cluster #109

Add an async /evaluate endpoint to trigger evaluation jobs on a remote cluster #109

Conversation

AnuradhaKaruppiah commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

By Submitting this PR I confirm:

Uh oh!

AnuradhaKaruppiah commented Apr 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AnuradhaKaruppiah commented Apr 9, 2025 •

edited

Loading