KEMBAR78
Eval doc updates by AnuradhaKaruppiah · Pull Request #322 · NVIDIA/NeMo-Agent-Toolkit · GitHub
Skip to content

Eval doc updates #322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
May 29, 2025
Merged

Eval doc updates #322

merged 5 commits into from
May 29, 2025

Conversation

@AnuradhaKaruppiah
Copy link
Contributor

@AnuradhaKaruppiah AnuradhaKaruppiah commented May 29, 2025 •
edited
Loading

Description

  1. Add a link to the RAGAS nv metrics prompts
  2. Add a note that the RAGAS nv metrics prompts are not tunable. The user can instead use the "Tunable RAG Evaluator" or add their own "Custom Evaluator"
  3. Add a note on the recommended max_tokens for Trajectory Evluator.

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
    • Any contribution which contains commits that are not Signed-Off will not be accepted.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

@AnuradhaKaruppiah
Eval doc updates
1551e42 
1. Add a link to the RAGAS nv metrics prompts
2. Add a note that the RAGSA nv metrics prompts are not tunable. The
   user can instead use the "Tunable RAG Evaluator" or add their
   own "Custom Evaluator"
3. Add a note on the recommended max_tokens for Trajectory Evluator.

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah AnuradhaKaruppiah requested a review from Copilot May 29, 2025 02:04
@AnuradhaKaruppiah AnuradhaKaruppiah added doc Improvements or additions to documentation non-breaking Non-breaking change labels May 29, 2025
Copilot
Copilot AI reviewed May 29, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the evaluation documentation to provide clearer guidelines on judge LLM configurations and evaluator settings.

  • Adds detailed explanations on the judge LLM's score and max_tokens configuration.
  • Includes new links to the RAGAS NV metrics leadership board and prompt documentation.
  • Clarifies the trajectory evaluator's output and configuration instructions.

AnuradhaKaruppiah and others added 3 commits May 28, 2025 19:05
@AnuradhaKaruppiah @Copilot
Update docs/source/workflows/evaluate.md
ab3a330 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Anuradha Karuppiah <anuradha.karuppiah@gmail.com>
@AnuradhaKaruppiah
Drop unnecessary white space
137c7a3 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Fix CI failures
7273531 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
dagardner-nv
dagardner-nv approved these changes May 29, 2025
View reviewed changes
Copy link
Contributor

@dagardner-nv dagardner-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please merge develop into this branch so that the links are validated in CI

@AnuradhaKaruppiah
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 1b7be46 into NVIDIA:develop May 29, 2025
12 checks passed
@AnuradhaKaruppiah AnuradhaKaruppiah deleted the ak-eval-docs branch May 29, 2025 22:57
gfreeman-nvidia pushed a commit to gfreeman-nvidia/AIQToolkit that referenced this pull request May 30, 2025
@AnuradhaKaruppiah @gfreeman-nvidia
Eval doc updates (NVIDIA#322)
85e7db0 
1. Add a link to the RAGAS nv metrics prompts
2. Add a note that the RAGAS nv metrics prompts are not tunable. The user can instead use the "Tunable RAG Evaluator" or add their own "Custom Evaluator"
3. Add a note on the recommended max_tokens for Trajectory Evluator.

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AIQToolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: NVIDIA#322
Signed-off-by: Greg Freeman <gfreeman@nvidia.com>
ericevans-nv pushed a commit to ericevans-nv/agent-iq that referenced this pull request Jun 3, 2025
@AnuradhaKaruppiah @ericevans-nv
Eval doc updates (NVIDIA#322)
0205fe3 
1. Add a link to the RAGAS nv metrics prompts
2. Add a note that the RAGAS nv metrics prompts are not tunable. The user can instead use the "Tunable RAG Evaluator" or add their own "Custom Evaluator"
3. Add a note on the recommended max_tokens for Trajectory Evluator.

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AIQToolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: NVIDIA#322
Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com>
ericevans-nv pushed a commit to ericevans-nv/agent-iq that referenced this pull request Jun 3, 2025
@AnuradhaKaruppiah @ericevans-nv
Eval doc updates (NVIDIA#322)
43ef48a 
1. Add a link to the RAGAS nv metrics prompts
2. Add a note that the RAGAS nv metrics prompts are not tunable. The user can instead use the "Tunable RAG Evaluator" or add their own "Custom Evaluator"
3. Add a note on the recommended max_tokens for Trajectory Evluator.

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AIQToolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: NVIDIA#322
Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com>
ericevans-nv pushed a commit to ericevans-nv/agent-iq that referenced this pull request Jun 3, 2025
@AnuradhaKaruppiah @ericevans-nv
Eval doc updates (NVIDIA#322)
5c0fb7e 
1. Add a link to the RAGAS nv metrics prompts
2. Add a note that the RAGAS nv metrics prompts are not tunable. The user can instead use the "Tunable RAG Evaluator" or add their own "Custom Evaluator"
3. Add a note on the recommended max_tokens for Trajectory Evluator.

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AIQToolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: NVIDIA#322
Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com>
AnuradhaKaruppiah added a commit to AnuradhaKaruppiah/oss-agentiq that referenced this pull request Aug 4, 2025
@AnuradhaKaruppiah
Eval doc updates (NVIDIA#322)
c55d624 
1. Add a link to the RAGAS nv metrics prompts
2. Add a note that the RAGAS nv metrics prompts are not tunable. The user can instead use the "Tunable RAG Evaluator" or add their own "Custom Evaluator"
3. Add a note on the recommended max_tokens for Trajectory Evluator.

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AIQToolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: NVIDIA#322
scheckerNV pushed a commit to scheckerNV/aiq-factory-reset that referenced this pull request Aug 22, 2025
@AnuradhaKaruppiah
Eval doc updates (NVIDIA#322)
50d8d31 
1. Add a link to the RAGAS nv metrics prompts
2. Add a note that the RAGAS nv metrics prompts are not tunable. The user can instead use the "Tunable RAG Evaluator" or add their own "Custom Evaluator"
3. Add a note on the recommended max_tokens for Trajectory Evluator.

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AIQToolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: NVIDIA#322
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@dagardner-nv dagardner-nv dagardner-nv approved these changes

Assignees

No one assigned

Labels

doc Improvements or additions to documentation non-breaking Non-breaking change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@AnuradhaKaruppiah @dagardner-nv