swa avoid stream sync #157705
swa avoid stream sync #157705
Conversation
Summary:
When AveragedModel updates_parameters it calls self.n_averaged == 0 for each parameter, where n_averated is a buffer on GPU. Moving check before the cycle to call sync once
It improves update_parameter from 74ms to 57ms ~22% improvement
{F1980011097}
{F1980011111}
Test Plan:
CI
Rollback Plan:
Differential Revision: D77723025
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/157705
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit ed50d55 with merge base d27d361 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
This pull request was exported from Phabricator. Differential Revision: D77723025 |
There was a problem hiding this comment.
Thanks for the small fix, from a quick look, the whole n_averaged is pretty suspicious looking. I expect it would be much more efficient to always have it as an int that gets passed as a kernel parameter and can be checked for free on the cpu side.
If you plan on working on that, please open an issue to discuss strategies first!
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
6 participants
Summary:
When AveragedModel updates_parameters it calls self.n_averaged == 0 for each parameter, where n_averated is a buffer on GPU. Moving check before the cycle to call sync once
It improves update_parameter from 74ms to 57ms ~22% improvement
{F1980011097}
{F1980011111}
Test Plan:
CI
Rollback Plan:
Differential Revision: D77723025