KEMBAR78
Document using local/self-hosted models by dagardner-nv · Pull Request #101 · NVIDIA/NeMo-Agent-Toolkit · GitHub
Skip to content

Document using local/self-hosted models #101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Apr 15, 2025
Merged

Document using local/self-hosted models #101

merged 12 commits into from
Apr 15, 2025

Conversation

@dagardner-nv
Copy link
Contributor

Description

  • Document using locally hosted models using NIMs and using vLLM
  • Adds an OpenAI LangChain embedding client (required for the vLLM).

Closes #34
Closes #37

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
    • Any contribution which contains commits that are not Signed-Off will not be accepted.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

@dagardner-nv
WIP
4f25ab9 
Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv
Add an openai langchain embedder
091d495 
Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv
Register openai embedder
6e101d6 
Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv
Fix type-o in docstring
9bc25e5 
Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv
First pass at a vLLM config
96d6c5a 
Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv
WIP
f8d4629 
Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv
Lint fix
9723d67 
Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv
Add an api_key value
e19e1a1 
Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv
Document vLLM
b080296 
Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv dagardner-nv added doc Improvements or additions to documentation non-breaking Non-breaking change labels Apr 8, 2025
@dagardner-nv dagardner-nv self-assigned this Apr 8, 2025
@dagardner-nv dagardner-nv requested a review from a team as a code owner April 8, 2025 17:03
@dagardner-nv dagardner-nv marked this pull request as draft April 8, 2025 17:03
@dagardner-nv
Add CR header
995db87 
Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv dagardner-nv marked this pull request as ready for review April 9, 2025 16:25
@dagardner-nv dagardner-nv marked this pull request as draft April 9, 2025 20:22
@dagardner-nv
Switch to using the microsoft/Phi-3-mini-4k-instruct LLM model as it …
41f1a46 
…was available for both NIM and vLLM

Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv
Merge branch 'develop' of github.com:NVIDIA/AgentIQ into david-using-…
f157cad 
…local-llms-35

Signed-off-by: David Gardner <dagardner@nvidia.com>
@dagardner-nv dagardner-nv marked this pull request as ready for review April 9, 2025 22:26
@dagardner-nv dagardner-nv marked this pull request as draft April 10, 2025 15:55
@dagardner-nv dagardner-nv marked this pull request as ready for review April 10, 2025 17:05
@mdemoret-nv
Copy link
Collaborator

/merge

@rapids-bot rapids-bot bot merged commit 926d618 into NVIDIA:develop Apr 15, 2025
10 checks passed
yczhang-nv pushed a commit to yczhang-nv/NeMo-Agent-Toolkit that referenced this pull request Apr 21, 2025
@dagardner-nv @yczhang-nv
Document using local/self-hosted models (NVIDIA#101)
1efba2b 
* Document using locally hosted models using NIMs and using vLLM
* Adds an OpenAI LangChain embedding client (required for the vLLM).

Closes NVIDIA#34
Closes NVIDIA#37

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: NVIDIA#101
Signed-off-by: Yuchen Zhang <134643420+yczhang-nv@users.noreply.github.com>
ericevans-nv pushed a commit to ericevans-nv/agent-iq that referenced this pull request Apr 23, 2025
@dagardner-nv @ericevans-nv
Document using local/self-hosted models (NVIDIA#101)
b9120a9 
* Document using locally hosted models using NIMs and using vLLM
* Adds an OpenAI LangChain embedding client (required for the vLLM).

Closes NVIDIA#34
Closes NVIDIA#37

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: NVIDIA#101
Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com>
ericevans-nv pushed a commit to ericevans-nv/agent-iq that referenced this pull request Apr 23, 2025
@dagardner-nv @ericevans-nv
Document using local/self-hosted models (NVIDIA#101)
8cad7a0 
* Document using locally hosted models using NIMs and using vLLM
* Adds an OpenAI LangChain embedding client (required for the vLLM).

Closes NVIDIA#34
Closes NVIDIA#37

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: NVIDIA#101
Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com>
yczhang-nv pushed a commit to yczhang-nv/NeMo-Agent-Toolkit that referenced this pull request May 8, 2025
@dagardner-nv @yczhang-nv
Document using local/self-hosted models (NVIDIA#101)
e5d591e 
* Document using locally hosted models using NIMs and using vLLM
* Adds an OpenAI LangChain embedding client (required for the vLLM).

Closes NVIDIA#34
Closes NVIDIA#37

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: NVIDIA#101
Signed-off-by: Yuchen Zhang <134643420+yczhang-nv@users.noreply.github.com>
AnuradhaKaruppiah pushed a commit to AnuradhaKaruppiah/oss-agentiq that referenced this pull request Aug 4, 2025
@dagardner-nv
Document using local/self-hosted models (NVIDIA#101)
de4be9f 
* Document using locally hosted models using NIMs and using vLLM
* Adds an OpenAI LangChain embedding client (required for the vLLM).

Closes NVIDIA#34
Closes NVIDIA#37

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: NVIDIA#101
scheckerNV pushed a commit to scheckerNV/aiq-factory-reset that referenced this pull request Aug 22, 2025
@dagardner-nv
Document using local/self-hosted models (NVIDIA#101)
650e6a1 
* Document using locally hosted models using NIMs and using vLLM
* Adds an OpenAI LangChain embedding client (required for the vLLM).

Closes NVIDIA#34
Closes NVIDIA#37

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: NVIDIA#101
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees

@dagardner-nv dagardner-nv

Labels

doc Improvements or additions to documentation non-breaking Non-breaking change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Feature Request: Add Support for vLLM-Based Model Hosting Provide an example for using local llm in config

2 participants

@dagardner-nv @mdemoret-nv