Allow non-json string outputs for workflows that use unstructured datasets #396
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…asets Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
There was a problem hiding this comment.
Pull Request Overview
This PR removes the JSON-only restriction on unstructured dataset outputs by adding a helper to parse plain strings, JSON strings, and model objects, and updates tests accordingly.
- Added
parse_if_json_stringtopublish_eval_inputto handle multiple output types. - Switched the unstructured case to use the new parser instead of always calling
json.loads. - Introduced new tests in
test_dataset_handler.pyto validate plain strings, JSON strings, Pydantic models, and numeric outputs.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| src/aiq/eval/dataset_handler/dataset_handler.py | Added parse_if_json_string helper and updated unstructured JSON serialization to use it. |
| tests/aiq/eval/dataset_handler/test_dataset_handler.py | Imported json, EvalInputItem, EvalOutputItem; added make_eval_input_item helper and new test for unstructured outputs. |
Comments suppressed due to low confidence (1)
src/aiq/eval/dataset_handler/dataset_handler.py:155
- The
jsonmodule isn't imported in this file but is used inparse_if_json_string. Please addimport jsonat the top.
def parse_if_json_string(value):
mdemoret-nv
added
improvement
mdemoret-nv
approved these changes
Jun 27, 2025
|
/merge |
mpenn
pushed a commit
to mpenn/NeMo-Agent-Toolkit
that referenced
this pull request
Jul 23, 2025
…asets (NVIDIA#396) Unstructured datasets were expecting json outputs (overfitted to swe-bench datasets). This change removes the restrictions and allows workflows that use unstructured datasets to return any output object. ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/develop/docs/source/resources/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: NVIDIA#396 Signed-off-by: Matthew Penn <mpenn@nvidia.com>
AnuradhaKaruppiah
added a commit
to AnuradhaKaruppiah/oss-agentiq
that referenced
this pull request
Aug 4, 2025
…asets (NVIDIA#396) Unstructured datasets were expecting json outputs (overfitted to swe-bench datasets). This change removes the restrictions and allows workflows that use unstructured datasets to return any output object. ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/develop/docs/source/resources/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: NVIDIA#396
scheckerNV
pushed a commit
to scheckerNV/aiq-factory-reset
that referenced
this pull request
Aug 22, 2025
…asets (NVIDIA#396) Unstructured datasets were expecting json outputs (overfitted to swe-bench datasets). This change removes the restrictions and allows workflows that use unstructured datasets to return any output object. ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/develop/docs/source/resources/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: NVIDIA#396
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Unstructured datasets were expecting json outputs (overfitted to swe-bench datasets). This change removes the restrictions and allows workflows that use unstructured datasets to return any output object.
By Submitting this PR I confirm: