[DDP] Custom buffer reduction #64513

rohan-varma · 2021-09-05T01:06:28Z

Stack from ghstack:

Proposal: #63041
Support custom buffer reduction in DDP via hook. This is required for an internal training use case and could be more broadly applicable to OSS scenarios.

Differential Revision: D30751152

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23

Proposal: #63041 Support custom buffer reduction in DDP via hook Differential Revision: [D30751152](https://our.internmc.facebook.com/intern/diff/D30751152/) [ghstack-poisoned]

facebook-github-bot · 2021-09-05T01:06:34Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/64513
📄 Preview docs built from this PR
📄 Preview C++ docs built from this PR
↩️ [fb-only] Re-run with SSH instructions
🔧 Opt-in to CIFlow to control what jobs run on your PRs

💊 CI failures summary and remediations

As of commit fe5d59b (more details on the Dr. CI page):

1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

linux-bionic-py3.8-gcc9-coverage / test (default, 1, 2, linux.2xlarge) (1/1)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-09-21T21:26:55.2910962Z CONTINUE_THROUGH_ERROR: false

  "oncall: distributed",
  "cla signed"
]
2021-09-21T21:26:55.2906556Z   GITHUB_TOKEN: ***
2021-09-21T21:26:55.2907472Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-py3.8-gcc9:74e757e8b0cf750d2f91db6aa4c29640abce32ea
2021-09-21T21:26:55.2908630Z   JOB_BASE_NAME: linux-bionic-py3.8-gcc9-coverage-test
2021-09-21T21:26:55.2909183Z   TEST_CONFIG: default
2021-09-21T21:26:55.2909496Z   SHARD_NUMBER: 1
2021-09-21T21:26:55.2909801Z   NUM_TEST_SHARDS: 2
2021-09-21T21:26:55.2910150Z   PYTORCH_IGNORE_DISABLED_ISSUES: 
2021-09-21T21:26:55.2910962Z   CONTINUE_THROUGH_ERROR: false
2021-09-21T21:26:55.2911284Z   SHM_SIZE: 1g
2021-09-21T21:26:55.2911567Z   PR_NUMBER: 64513
2021-09-21T21:26:55.2911826Z   IS_GHA: 1
2021-09-21T21:26:55.2912123Z   CIRCLE_BRANCH: pull/64513
2021-09-21T21:26:55.2912455Z   CIRCLE_PR_NUMBER: 64513
2021-09-21T21:26:55.2912925Z   CIRCLE_SHA1: fe5d59bf8fa514183e44f65287847a601cd643e0
2021-09-21T21:26:55.2913443Z   AWS_DEFAULT_REGION: us-east-1
2021-09-21T21:26:55.2913778Z ##[endgroup]
2021-09-21T21:27:08.4533513Z Processing ./dist/torch-1.10.0a0+gitb0b4960-cp38-cp38-linux_x86_64.whl
2021-09-21T21:27:08.4797762Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from torch==1.10.0a0+gitb0b4960) (3.10.0.2)

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@H-Huang

Proposal: #63041 Support custom buffer reduction in DDP via hook. This is required for an internal training use case and could be more broadly applicable to OSS scenarios. Differential Revision: [D30751152](https://our.internmc.facebook.com/intern/diff/D30751152/) cc pietern mrshenli pritamdamania87 zhaojuanmao satgera rohan-varma gqchen aazzolini osalpekar jiayisuse agolynski @SciPioneer @H-Huang mrzzd cbalioglu gcramer23 [ghstack-poisoned]

zhaojuanmao · 2021-09-10T20:22:37Z

torch/nn/parallel/distributed.py

+        state,
+        hook: callable,


nit: make state after hook? following up (func, args) convention?

Thanks for pointing this out!

I followed the same order as comm hook (state, hook): https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L1224.

Should we try to keep it the same or change the order here?

zhaojuanmao · 2021-09-10T20:24:05Z

torch/nn/parallel/distributed.py

+
+@dataclass
+class _BufferCommHook:
+    buffer_comm_hook: Callable


nit: document how the signature of a buffer_comm_hook looks like:

def buffer_comm_hook(state, self.name_buffers)

I added the documentation to the function in the PR at the top of this stack: https://github.com/pytorch/pytorch/pull/64515/files, will just leave it there if you're okay (plan is to land the whole stack together).

@H-Huang

Proposal: #63041 Support custom buffer reduction in DDP via hook. This is required for an internal training use case and could be more broadly applicable to OSS scenarios. Differential Revision: [D30751152](https://our.internmc.facebook.com/intern/diff/D30751152/) cc pietern mrshenli pritamdamania87 zhaojuanmao satgera rohan-varma gqchen aazzolini osalpekar jiayisuse agolynski @SciPioneer @H-Huang mrzzd cbalioglu gcramer23 [ghstack-poisoned]

@H-Huang

Proposal: #63041 Support custom buffer reduction in DDP via hook. This is required for an internal training use case and could be more broadly applicable to OSS scenarios. Differential Revision: [D30751152](https://our.internmc.facebook.com/intern/diff/D30751152/) cc pietern mrshenli pritamdamania87 zhaojuanmao satgera rohan-varma gqchen aazzolini osalpekar jiayisuse agolynski @SciPioneer @H-Huang mrzzd cbalioglu gcramer23 [ghstack-poisoned]

@H-Huang

Proposal: #63041 Support custom buffer reduction in DDP via hook. This is required for an internal training use case and could be more broadly applicable to OSS scenarios. Differential Revision: [D30751152](https://our.internmc.facebook.com/intern/diff/D30751152/) cc pietern mrshenli pritamdamania87 zhaojuanmao satgera rohan-varma gqchen aazzolini osalpekar jiayisuse agolynski @SciPioneer @H-Huang mrzzd cbalioglu gcramer23 [ghstack-poisoned]

rohan-varma · 2021-09-17T22:08:46Z

torch/nn/parallel/distributed.py


+            # sync params according to location (before/after forward) user
+            # specified as part of hook, if hook was specified.
+            if (


moved to helper fn in the next diff

@H-Huang

Proposal: #63041 Support custom buffer reduction in DDP via hook. This is required for an internal training use case and could be more broadly applicable to OSS scenarios. Differential Revision: [D30751152](https://our.internmc.facebook.com/intern/diff/D30751152/) cc pietern mrshenli pritamdamania87 zhaojuanmao satgera rohan-varma gqchen aazzolini osalpekar jiayisuse agolynski @SciPioneer @H-Huang mrzzd cbalioglu gcramer23 [ghstack-poisoned]

Pull Request resolved: #64513 Proposal: #63041 Support custom buffer reduction in DDP via hook ghstack-source-id: 138419449 Differential Revision: [D30751152](https://our.internmc.facebook.com/intern/diff/D30751152/)

@H-Huang

Proposal: #63041 Support custom buffer reduction in DDP via hook. This is required for an internal training use case and could be more broadly applicable to OSS scenarios. Differential Revision: [D30751152](https://our.internmc.facebook.com/intern/diff/D30751152/) cc pietern mrshenli pritamdamania87 zhaojuanmao satgera rohan-varma gqchen aazzolini osalpekar jiayisuse agolynski @SciPioneer @H-Huang mrzzd cbalioglu gcramer23 [ghstack-poisoned]

@H-Huang

Proposal: #63041 Support custom buffer reduction in DDP via hook. This is required for an internal training use case and could be more broadly applicable to OSS scenarios. Differential Revision: [D30751152](https://our.internmc.facebook.com/intern/diff/D30751152/) cc pietern mrshenli pritamdamania87 zhaojuanmao satgera rohan-varma gqchen aazzolini osalpekar jiayisuse agolynski @SciPioneer @H-Huang mrzzd cbalioglu gcramer23 [ghstack-poisoned]

facebook-github-bot · 2021-09-22T21:13:03Z

This pull request has been merged in ce5981e.

[DDP] Custom buffer reduction

31bb92a

Proposal: #63041 Support custom buffer reduction in DDP via hook Differential Revision: [D30751152](https://our.internmc.facebook.com/intern/diff/D30751152/) [ghstack-poisoned]

rohan-varma requested review from H-Huang, cbalioglu, mingzhe09088, mrshenli, pritamdamania87, wayi1 and zhaojuanmao as code owners September 5, 2021 01:06

facebook-github-bot added the cla signed label Sep 5, 2021

facebook-github-bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Sep 5, 2021

zhaojuanmao approved these changes Sep 10, 2021

View reviewed changes

This was referenced Sep 15, 2021

Back out "Revert D30745921: [DDP] Fix when buffers are reassigned in module" #64776

Closed

Back out "Revert D30745961: [DDP] Remove self.modules_params" #64777

Closed

Back out "Revert D30745960: [DDP] Remove SPMD from self.modules_buffers" #64778

Closed

rohan-varma added 2 commits September 16, 2021 21:33

rohan-varma commented Sep 17, 2021

View reviewed changes

rohan-varma added 2 commits September 20, 2021 18:22

rohan-varma mentioned this pull request Sep 21, 2021

Make DDP uneven inputs work with custom buffer reduction #65436

Open

facebook-github-bot closed this in ce5981e Sep 22, 2021

facebook-github-bot added the Merged label Sep 22, 2021

facebook-github-bot deleted the gh/rohan-varma/404/head branch September 26, 2021 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[DDP] Custom buffer reduction #64513

[DDP] Custom buffer reduction #64513

Uh oh!

rohan-varma commented Sep 5, 2021 •

edited

Loading

Uh oh!

facebook-github-bot commented Sep 5, 2021 •

edited

Loading

Uh oh!

zhaojuanmao Sep 10, 2021

Uh oh!

rohan-varma Sep 17, 2021

Uh oh!

zhaojuanmao Sep 10, 2021

Uh oh!

rohan-varma Sep 17, 2021 •

edited

Loading

Uh oh!

rohan-varma Sep 17, 2021

Uh oh!

facebook-github-bot commented Sep 22, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[DDP] Custom buffer reduction #64513

[DDP] Custom buffer reduction #64513

Uh oh!

Conversation

rohan-varma commented Sep 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Sep 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

🕵️ 1 new failure recognized by patterns

linux-bionic-py3.8-gcc9-coverage / test (default, 1, 2, linux.2xlarge) (1/1)

Uh oh!

zhaojuanmao Sep 10, 2021

Choose a reason for hiding this comment

Uh oh!

rohan-varma Sep 17, 2021

Choose a reason for hiding this comment

Uh oh!

zhaojuanmao Sep 10, 2021

Choose a reason for hiding this comment

Uh oh!

rohan-varma Sep 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohan-varma Sep 17, 2021

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Sep 22, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rohan-varma commented Sep 5, 2021 •

edited

Loading

facebook-github-bot commented Sep 5, 2021 •

edited

Loading

rohan-varma Sep 17, 2021 •

edited

Loading