-
Notifications
You must be signed in to change notification settings - Fork 3.1k
Add object_storage_cache_path to PreTrainingDataModule #14103
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
318b38e to
703a202
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thank you!
Signed-off-by: Shunjia Ding <shunjiad@nvidia.com>
703a202 to
51c8d28
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks!
…4103) Signed-off-by: Shunjia Ding <shunjiad@nvidia.com> Signed-off-by: Amir Hussein <amhussein@nvidia.com>
|
hey so while running the code i had a few errors in trying to use the validate_dataset_asset_accessibility function for msc path, as it created an msc object without suffices which turned out to be an issue, and morever even with providing the actual path with suffix to try it out, I found out that the if statement in line 109 was being triggered which checked it the provided file was readable or not using os and as it was a msc object it did not work and threw error. I was using ai store on local host to test things and I created a bucket and directly added my .bin and .idx file to it. |
…4103) Signed-off-by: Shunjia Ding <shunjiad@nvidia.com> Signed-off-by: Amir Hussein <amhussein@nvidia.com>
…4103) Signed-off-by: Shunjia Ding <shunjiad@nvidia.com> Signed-off-by: Amir Hussein <amhussein@nvidia.com>
…4103) Signed-off-by: Shunjia Ding <shunjiad@nvidia.com>
…4103) Signed-off-by: Shunjia Ding <shunjiad@nvidia.com> Signed-off-by: Guyue Huang <guyueh@nvidia.com>
Important
The
Update branchbutton must only be pressed in very rare occassions.An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.
What does this PR do ?
Introduce
object_storage_cache_path(from Megatron-LM code) into thePreTrainingDataModuleto enable reading data from object storage.See Training from Object Storage for more details.
Collection: [Note which collection this PR will affect]
Changelog
Usage
GitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information