KEMBAR78
Add object_storage_cache_path to PreTrainingDataModule by shunjiad · Pull Request #14103 · NVIDIA-NeMo/NeMo · GitHub
Skip to content

Conversation

@shunjiad
Copy link
Contributor

@shunjiad shunjiad commented Jul 2, 2025

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Introduce object_storage_cache_path (from Megatron-LM code) into the PreTrainingDataModule to enable reading data from object storage.

See Training from Object Storage for more details.

Collection: [Note which collection this PR will affect]

Changelog

  • Add object_storage_cache_path to PreTrainingDataModule

Usage

data = PreTrainingDataModule(
    paths=["msc://pretrain/data/simple_wiki_gpt_text_document"],
    seq_length=512,
    micro_batch_size=2,
    global_batch_size=2,
    tokenizer=tokenizer,
    object_storage_cache_path="/tmp/object_storage_cache_path",
)

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

dimapihtar
dimapihtar previously approved these changes Jul 3, 2025
Copy link
Collaborator

@dimapihtar dimapihtar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you!

Signed-off-by: Shunjia Ding <shunjiad@nvidia.com>
Copy link
Collaborator

@ericharper ericharper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@ericharper ericharper merged commit 0beed05 into NVIDIA-NeMo:main Jul 8, 2025
224 checks passed
@gautham-kollu gautham-kollu mentioned this pull request Jul 9, 2025
8 tasks
AmirHussein96 pushed a commit to AmirHussein96/NeMo that referenced this pull request Jul 23, 2025
…4103)

Signed-off-by: Shunjia Ding <shunjiad@nvidia.com>
Signed-off-by: Amir Hussein <amhussein@nvidia.com>
@somay-jalan
Copy link

somay-jalan commented Aug 3, 2025

hey so while running the code i had a few errors in trying to use the validate_dataset_asset_accessibility function for msc path, as it created an msc object without suffices which turned out to be an issue, and morever even with providing the actual path with suffix to try it out, I found out that the if statement in line 109 was being triggered which checked it the provided file was readable or not using os and as it was a msc object it did not work and threw error. I was using ai store on local host to test things and I created a bucket and directly added my .bin and .idx file to it.
I was able to get the training to work for deepseekV3 model. Let me know if I did anything wrong or if any of my errors were due to some other error.
My fix to the function is as follows.

def validate_dataset_asset_accessibility(paths):
    """
    Validate the accessibility of the dataset assets.
    """
    if paths is None:
        raise ValueError("Expected path to have a value.")

    if isinstance(paths, tuple) or isinstance(paths, list):
        if is_zipped_list(paths):
            # remove weights from paths.
            paths = paths[1::2]
        for p in paths:
            validate_dataset_asset_accessibility(p)
        return
    elif isinstance(paths, dict):
        for p in paths.values():
            validate_dataset_asset_accessibility(p)
        return

    if not isinstance(paths, str) and not isinstance(paths, Path):
        raise ValueError("Expected path to be of string or Path type.")

    suffices = (".bin", ".idx")
    if is_multistorageclient_url(paths):
        try:
            for suffix in suffices:
                file_path = (paths + suffix)
                msc = import_multistorageclient()
                path = msc.Path(file_path)
            return
        except:
            raise FileNotFoundError(f"Expected {str(file_path)} to exist in the msc bucket..")
    else:
        path = Path(paths)


    if path.is_dir():
        if not os.access(path, os.R_OK):
            raise PermissionError(f"Expected {str(path)} to be readable.")
        # Will let the downstream class confirm contents are ok.
        return
    if path.exists():
        if not os.access(path, os.R_OK):
            raise PermissionError(f"Expected {str(path)} to be readable.")
        return
    for suffix in suffices:
        file_path = path.with_name(path.name + suffix)
        if not file_path.exists():
            raise FileNotFoundError(f"Expected {str(file_path)} to exist.")
        if not os.access(file_path, os.R_OK):
            raise PermissionError(f"Expected {str(file_path)} to be readable.")

AmirHussein96 pushed a commit to AmirHussein96/NeMo that referenced this pull request Aug 5, 2025
…4103)

Signed-off-by: Shunjia Ding <shunjiad@nvidia.com>
Signed-off-by: Amir Hussein <amhussein@nvidia.com>
AmirHussein96 pushed a commit to AmirHussein96/NeMo that referenced this pull request Aug 5, 2025
…4103)

Signed-off-by: Shunjia Ding <shunjiad@nvidia.com>
Signed-off-by: Amir Hussein <amhussein@nvidia.com>
nasretdinovr pushed a commit to nasretdinovr/NeMo that referenced this pull request Aug 8, 2025
guyueh1 pushed a commit to guyueh1/NeMo that referenced this pull request Aug 25, 2025
…4103)

Signed-off-by: Shunjia Ding <shunjiad@nvidia.com>
Signed-off-by: Guyue Huang <guyueh@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants