StackDataset batched sampling #110694
StackDataset batched sampling #110694
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/110694
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (2 Unrelated Failures)As of commit 7a31096 with merge base 65afa76 ( UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
ezyang
added
the
triaged
|
@ejguan will you be able to review this? |
Sure thing. I will review it later today. |
torch/utils/data/dataset.py
Outdated
| dict_batch: List[T_dict] = [{} for _ in indices] | ||
| for k, dataset in self.datasets.items(): | ||
| if callable(getattr(dataset, "__getitems__", None)): | ||
| for data, d_sample in zip(dataset.__getitems__(indices), dict_batch): # type: ignore[attr-defined] |
There was a problem hiding this comment.
Could you pls add an Error if the size of result from dataset.__getitems__(indices) is not correct?
torch/utils/data/dataset.py
Outdated
| list_batch: List[list] = [[] for _ in indices] | ||
| for dataset in self.datasets: | ||
| if callable(getattr(dataset, "__getitems__", None)): | ||
| for data, t_sample in zip(dataset.__getitems__(indices), list_batch): # type: ignore[attr-defined] |
| raise ValueError("Nested dataset's output size mismatch." | ||
| f" Expected {len(indices)}, got {len(items)}") |
There was a problem hiding this comment.
Can you pls add an unit test to validate the ValueError is raised as expectation?
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
5 participants
Optimization of loading minibatches