KEMBAR78
Add default PyTorch seeding and worker_init_fn to DataLoader by ssnl · Pull Request #4018 · pytorch/pytorch · GitHub
Skip to content

Add default PyTorch seeding and worker_init_fn to DataLoader #4018

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Dec 18, 2017
Merged

Add default PyTorch seeding and worker_init_fn to DataLoader #4018

merged 3 commits into from
Dec 18, 2017

Conversation

@ssnl
Copy link
Collaborator

@ssnl ssnl commented Dec 4, 2017

Fixes #3880 .

apaszke
apaszke reviewed Dec 5, 2017
View reviewed changes
Copy link
Contributor

@apaszke apaszke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That looks good, but we should be careful with passing functions around. lambdas aren't picklable

torch/utils/data/dataloader.py Outdated
_set_worker_signal_handlers()

torch.set_num_threads(1)
torch.manual_seed(torch.initial_seed() + id + 1)

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

torch/utils/data/dataloader.py Outdated


def _worker_loop(dataset, index_queue, data_queue, collate_fn):
def _worker_loop(dataset, index_queue, data_queue, collate_fn, init_fn, id):

This comment was marked as off-topic.

Sign in to view

torch/utils/data/dataloader.py Outdated
args=(self.dataset, self.index_queue, self.worker_result_queue, self.collate_fn))
for _ in range(self.num_workers)]
args=(self.dataset, self.index_queue, self.worker_result_queue, self.collate_fn,
self.worker_init_fn, i))

This comment was marked as off-topic.

Sign in to view

@netheril96
Copy link

Does this fix #3880 completely? As I understand it, at each epoch the worker processes are created fresh, so if the main process never calls random functions, torch.initial_seed() will always be the same in every worker at every epoch. In that case, the randomness is still restricted.

ssnl
ssnl commented Dec 5, 2017
View reviewed changes
torch/utils/data/dataloader.py Outdated
this value in :attr:`worker_init_fn`, which can be used to set other seeds
(e.g. NumPy) before data loading.
.. warning:: If ``spawn'' start method is used, :attr:`worker_init_fn` cannot be a lambda

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

apaszke
apaszke approved these changes Dec 5, 2017
View reviewed changes
Copy link
Contributor

@apaszke apaszke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but would be nice to have one more test.

torch/utils/data/dataloader.py Outdated
self.done_event.set()
# if worker_manager_thread is waiting to put
while not self.data_queue.empty():
while self.data_queue is not None and not self.data_queue.empty():

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

torch/utils/data/dataloader.py Outdated
this value in :attr:`worker_init_fn`, which can be used to set other seeds
(e.g. NumPy) before data loading.
.. warning:: If ``spawn'' start method is used, :attr:`worker_init_fn` cannot be a lambda

This comment was marked as off-topic.

Sign in to view

test/test_dataloader.py
worker_init_fn=init_fn)
for batch in dataloader:
self.assertEqual(12345, batch[0])
self.assertEqual(12345, batch[1])

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

ssnl
ssnl commented Dec 5, 2017
View reviewed changes
test/test_dataloader.py Outdated

This comment was marked as off-topic.

Sign in to view

ssnl
ssnl commented Dec 5, 2017
View reviewed changes
torch/utils/data/dataloader.py Outdated

This comment was marked as off-topic.

Sign in to view

ssnl
ssnl commented Dec 5, 2017
View reviewed changes
torch/utils/data/dataloader.py Outdated

This comment was marked as off-topic.

Sign in to view

@soumith soumith merged commit 5cc26c0 into pytorch:master Dec 18, 2017
@ssnl ssnl deleted the dl_seed branch December 18, 2017 16:23
@soumith soumith added the 0.3.1 label Feb 7, 2018
soumith pushed a commit that referenced this pull request Feb 7, 2018
@ssnl @soumith
Add default PyTorch seeding and worker_init_fn to DataLoader (#4018)
3156625 
* Add default PyTorch seeding and worker_init_fn to DataLoader

* generate seed using current RNG each time

* worker_seed <- main_proc_RNG_generated_seed + worker_id
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@apaszke apaszke apaszke approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

open source

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Reinitialize the random generator in worker processes of DataLoader

5 participants

@ssnl @netheril96 @apaszke @ezyang @soumith