Scripts for running CI locally #59

dagardner-nv · 2025-04-02T15:28:48Z

Description

Adds a ci/scripts/run_ci_local.sh script allowing for executing CI scripts locally mirroring the GHA CI pipeline as closely as possible.
uv and pre-commit packages are cached in .tmp/local_ci_tmp/cache
uv's venv is located in .tmp/local_ci_tmp/local_ci_workspace/.venv
By default the script will perform a git clone and checkout the latest commit. This requires the latest commit to be pushed.
Alternately setting the environment variable USE_HOST_GIT=1 the host's repo will be mounted inside the CI container, avoiding the need to commit/push changes. (Note: Doing so requires all git-lfs files to be checked out first).
Fix escape sequences in ci/scripts/gitutils.py

Usage:

Run the entire CI pipeline:

./ci/scripts/run_ci_local.sh all

Run just the checks stage:

./ci/scripts/run_ci_local.sh checks

Debug a CI issue with the bash pseudo-stage. This will perform a git clone & checkout, and then drop you in a bash shell with all of the CI variables set.

./ci/scripts/run_ci_local.sh bash

By Submitting this PR I confirm:

I am familiar with the Contributing Guidelines.
We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
- Any contribution which contains commits that are not Signed-Off will not be accepted.
When the PR is ready for review, new or existing tests cover these changes.
When the PR is ready for review, the documentation is up to date with these changes.

Signed-off-by: David Gardner <dagardner@nvidia.com>

…-local Signed-off-by: David Gardner <dagardner@nvidia.com>

Signed-off-by: David Gardner <dagardner@nvidia.com>

…t repo, just assume the user has already done this Signed-off-by: David Gardner <dagardner@nvidia.com>

Signed-off-by: David Gardner <dagardner@nvidia.com>

AnuradhaKaruppiah · 2025-04-03T18:34:52Z

/merge

* Run the test CI stage on both Amd64 and Arm64. * Document installing QEMU to run CI for Arm64 on an Amd64 system. * Disable `examples/simple/tests/test_web_query_tool.py` test on Arm due to a known issue with Faiss #72 Requires #59 to be merged first ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #73

* Adds a `ci/scripts/run_ci_local.sh` script allowing for executing CI scripts locally mirroring the GHA CI pipeline as closely as possible. * uv and pre-commit packages are cached in `.tmp/local_ci_tmp/cache` * uv's venv is located in `.tmp/local_ci_tmp/local_ci_workspace/.venv` * By default the script will perform a `git clone` and checkout the latest commit. This requires the latest commit to be pushed. * Alternately setting the environment variable `USE_HOST_GIT=1` the host's repo will be mounted inside the CI container, avoiding the need to commit/push changes. (Note: Doing so requires all git-lfs files to be checked out first). * Fix escape sequences in `ci/scripts/gitutils.py` Usage: * Run the entire CI pipeline: ```bash ./ci/scripts/run_ci_local.sh all ``` * Run just the checks stage: ```bash ./ci/scripts/run_ci_local.sh checks ``` * Debug a CI issue with the `bash` pseudo-stage. This will perform a git clone & checkout, and then drop you in a bash shell with all of the CI variables set. ```bash ./ci/scripts/run_ci_local.sh bash ``` ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#59

* Run the test CI stage on both Amd64 and Arm64. * Document installing QEMU to run CI for Arm64 on an Amd64 system. * Disable `examples/simple/tests/test_web_query_tool.py` test on Arm due to a known issue with Faiss NVIDIA#72 Requires NVIDIA#59 to be merged first ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#73

* Adds a `ci/scripts/run_ci_local.sh` script allowing for executing CI scripts locally mirroring the GHA CI pipeline as closely as possible. * uv and pre-commit packages are cached in `.tmp/local_ci_tmp/cache` * uv's venv is located in `.tmp/local_ci_tmp/local_ci_workspace/.venv` * By default the script will perform a `git clone` and checkout the latest commit. This requires the latest commit to be pushed. * Alternately setting the environment variable `USE_HOST_GIT=1` the host's repo will be mounted inside the CI container, avoiding the need to commit/push changes. (Note: Doing so requires all git-lfs files to be checked out first). * Fix escape sequences in `ci/scripts/gitutils.py` Usage: * Run the entire CI pipeline: ```bash ./ci/scripts/run_ci_local.sh all ``` * Run just the checks stage: ```bash ./ci/scripts/run_ci_local.sh checks ``` * Debug a CI issue with the `bash` pseudo-stage. This will perform a git clone & checkout, and then drop you in a bash shell with all of the CI variables set. ```bash ./ci/scripts/run_ci_local.sh bash ``` ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#59 Signed-off-by: Yuchen Zhang <134643420+yczhang-nv@users.noreply.github.com>

* Run the test CI stage on both Amd64 and Arm64. * Document installing QEMU to run CI for Arm64 on an Amd64 system. * Disable `examples/simple/tests/test_web_query_tool.py` test on Arm due to a known issue with Faiss NVIDIA#72 Requires NVIDIA#59 to be merged first ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#73 Signed-off-by: Yuchen Zhang <134643420+yczhang-nv@users.noreply.github.com>

* Adds a `ci/scripts/run_ci_local.sh` script allowing for executing CI scripts locally mirroring the GHA CI pipeline as closely as possible. * uv and pre-commit packages are cached in `.tmp/local_ci_tmp/cache` * uv's venv is located in `.tmp/local_ci_tmp/local_ci_workspace/.venv` * By default the script will perform a `git clone` and checkout the latest commit. This requires the latest commit to be pushed. * Alternately setting the environment variable `USE_HOST_GIT=1` the host's repo will be mounted inside the CI container, avoiding the need to commit/push changes. (Note: Doing so requires all git-lfs files to be checked out first). * Fix escape sequences in `ci/scripts/gitutils.py` Usage: * Run the entire CI pipeline: ```bash ./ci/scripts/run_ci_local.sh all ``` * Run just the checks stage: ```bash ./ci/scripts/run_ci_local.sh checks ``` * Debug a CI issue with the `bash` pseudo-stage. This will perform a git clone & checkout, and then drop you in a bash shell with all of the CI variables set. ```bash ./ci/scripts/run_ci_local.sh bash ``` ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#59 Signed-off-by: Yuchen Zhang <134643420+yczhang-nv@users.noreply.github.com>

* Run the test CI stage on both Amd64 and Arm64. * Document installing QEMU to run CI for Arm64 on an Amd64 system. * Disable `examples/simple/tests/test_web_query_tool.py` test on Arm due to a known issue with Faiss NVIDIA#72 Requires NVIDIA#59 to be merged first ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#73 Signed-off-by: Yuchen Zhang <134643420+yczhang-nv@users.noreply.github.com>

* Adds a `ci/scripts/run_ci_local.sh` script allowing for executing CI scripts locally mirroring the GHA CI pipeline as closely as possible. * uv and pre-commit packages are cached in `.tmp/local_ci_tmp/cache` * uv's venv is located in `.tmp/local_ci_tmp/local_ci_workspace/.venv` * By default the script will perform a `git clone` and checkout the latest commit. This requires the latest commit to be pushed. * Alternately setting the environment variable `USE_HOST_GIT=1` the host's repo will be mounted inside the CI container, avoiding the need to commit/push changes. (Note: Doing so requires all git-lfs files to be checked out first). * Fix escape sequences in `ci/scripts/gitutils.py` Usage: * Run the entire CI pipeline: ```bash ./ci/scripts/run_ci_local.sh all ``` * Run just the checks stage: ```bash ./ci/scripts/run_ci_local.sh checks ``` * Debug a CI issue with the `bash` pseudo-stage. This will perform a git clone & checkout, and then drop you in a bash shell with all of the CI variables set. ```bash ./ci/scripts/run_ci_local.sh bash ``` ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#59

* Run the test CI stage on both Amd64 and Arm64. * Document installing QEMU to run CI for Arm64 on an Amd64 system. * Disable `examples/simple/tests/test_web_query_tool.py` test on Arm due to a known issue with Faiss NVIDIA#72 Requires NVIDIA#59 to be merged first ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#73

* Adds a `ci/scripts/run_ci_local.sh` script allowing for executing CI scripts locally mirroring the GHA CI pipeline as closely as possible. * uv and pre-commit packages are cached in `.tmp/local_ci_tmp/cache` * uv's venv is located in `.tmp/local_ci_tmp/local_ci_workspace/.venv` * By default the script will perform a `git clone` and checkout the latest commit. This requires the latest commit to be pushed. * Alternately setting the environment variable `USE_HOST_GIT=1` the host's repo will be mounted inside the CI container, avoiding the need to commit/push changes. (Note: Doing so requires all git-lfs files to be checked out first). * Fix escape sequences in `ci/scripts/gitutils.py` Usage: * Run the entire CI pipeline: ```bash ./ci/scripts/run_ci_local.sh all ``` * Run just the checks stage: ```bash ./ci/scripts/run_ci_local.sh checks ``` * Debug a CI issue with the `bash` pseudo-stage. This will perform a git clone & checkout, and then drop you in a bash shell with all of the CI variables set. ```bash ./ci/scripts/run_ci_local.sh bash ``` ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#59

* Run the test CI stage on both Amd64 and Arm64. * Document installing QEMU to run CI for Arm64 on an Amd64 system. * Disable `examples/simple/tests/test_web_query_tool.py` test on Arm due to a known issue with Faiss NVIDIA#72 Requires NVIDIA#59 to be merged first ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#73

dagardner-nv added 13 commits April 1, 2025 15:48

Fix escape sequences

632d904

Signed-off-by: David Gardner <dagardner@nvidia.com>

Fix handling of github/gitlab and local git variables

53a5a76

Signed-off-by: David Gardner <dagardner@nvidia.com>

Change location of the uv venv to not clash when running locally

b1568da

Signed-off-by: David Gardner <dagardner@nvidia.com>

temp

2b86009

Signed-off-by: David Gardner <dagardner@nvidia.com>

WIP

864970b

Signed-off-by: David Gardner <dagardner@nvidia.com>

WIP

9b77b6d

Signed-off-by: David Gardner <dagardner@nvidia.com>

WIP

2497062

Signed-off-by: David Gardner <dagardner@nvidia.com>

WIP

a73a30d

Signed-off-by: David Gardner <dagardner@nvidia.com>

Remove debug logging

cfe382a

Signed-off-by: David Gardner <dagardner@nvidia.com>

Use the --active flag

2831460

Signed-off-by: David Gardner <dagardner@nvidia.com>

Adding local CI scripts

6b7b518

Signed-off-by: David Gardner <dagardner@nvidia.com>

WIP

73f5080

Signed-off-by: David Gardner <dagardner@nvidia.com>

Change venv name

ab1fd6a

Signed-off-by: David Gardner <dagardner@nvidia.com>

dagardner-nv self-assigned this Apr 2, 2025

dagardner-nv requested a review from a team as a code owner April 2, 2025 15:28

dagardner-nv added improvement Improvement to existing functionality non-breaking Non-breaking change labels Apr 2, 2025

dagardner-nv added 2 commits April 2, 2025 09:32

Log completion

16368e4

Signed-off-by: David Gardner <dagardner@nvidia.com>

Fix location of workspace_tmp allowing artifact storage

e258920

Signed-off-by: David Gardner <dagardner@nvidia.com>

dagardner-nv marked this pull request as draft April 2, 2025 16:33

dagardner-nv added the skip-ci Optionally Skip CI for this PR label Apr 2, 2025

dagardner-nv added 5 commits April 2, 2025 09:34

Merge branch 'develop' of github.com:NVIDIA/AgentIQ into david-run-ci…

e40653c

…-local Signed-off-by: David Gardner <dagardner@nvidia.com>

Move the venv allowing for local caching

85347f3

Signed-off-by: David Gardner <dagardner@nvidia.com>

Cache uv and pre-commit packages locally

e6bc0ff

Signed-off-by: David Gardner <dagardner@nvidia.com>

Don't fetch lfs files when using host git as this will modify the hos…

00f8136

…t repo, just assume the user has already done this Signed-off-by: David Gardner <dagardner@nvidia.com>

Avoid ssh verification by trusting github.com

eeee4a9

Signed-off-by: David Gardner <dagardner@nvidia.com>

dagardner-nv removed the skip-ci Optionally Skip CI for this PR label Apr 2, 2025

Cleanup CI locations

13b7f7b

Signed-off-by: David Gardner <dagardner@nvidia.com>

dagardner-nv marked this pull request as ready for review April 2, 2025 17:50

dagardner-nv marked this pull request as draft April 2, 2025 18:09

dagardner-nv added 2 commits April 2, 2025 11:40

Add new documentation to index

826402d

Signed-off-by: David Gardner <dagardner@nvidia.com>

Add CR header

780a89e

Signed-off-by: David Gardner <dagardner@nvidia.com>

dagardner-nv marked this pull request as ready for review April 2, 2025 18:47

dagardner-nv added 2 commits April 2, 2025 11:53

Fix CR Year

3e91312

Signed-off-by: David Gardner <dagardner@nvidia.com>

Fix GitHub casing

4b0f5bb

Signed-off-by: David Gardner <dagardner@nvidia.com>

dagardner-nv mentioned this pull request Apr 2, 2025

Support ARM in CI #73

Merged

AnuradhaKaruppiah approved these changes Apr 3, 2025

View reviewed changes

rapids-bot bot merged commit c151736 into NVIDIA:develop Apr 3, 2025
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Scripts for running CI locally #59

Scripts for running CI locally #59

Uh oh!

dagardner-nv commented Apr 2, 2025 •

edited

Loading

Uh oh!

AnuradhaKaruppiah commented Apr 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Scripts for running CI locally #59

Scripts for running CI locally #59

Uh oh!

Conversation

dagardner-nv commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

By Submitting this PR I confirm:

Uh oh!

AnuradhaKaruppiah commented Apr 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dagardner-nv commented Apr 2, 2025 •

edited

Loading