[CI] Adds support for selecting experiments for workflows on runner determinator #137614
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/137614
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 3118452 with merge base b71d0ac ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
jeanschmidt
changed the title
| ) | ||
| parser.add_argument( | ||
| "--check-experiments", | ||
| type=_str_comma_separated_to_set, |
There was a problem hiding this comment.
TIL: you can pass in a function here!
| do_check = True | ||
| if check_experiments: | ||
| if experiment_name not in check_experiments: | ||
| exp_list = ", ".join(check_experiments) | ||
| log.info(f"Skipping experiment '{experiment_name}', as it is not in the check_experiments list: {exp_list}") | ||
| do_check = False |
There was a problem hiding this comment.
When an experiment is not a default experiment and not a check-experiment, even if the user is explicitly opted in the experiment should be disabled for them
There was a problem hiding this comment.
ohh good point...
| exp_list = ", ".join(check_experiments) | ||
| log.info(f"Skipping experiment '{experiment_name}', as it is not in the check_experiments list: {exp_list}") | ||
| do_check = False | ||
| elif not experiment_settings.default: |
There was a problem hiding this comment.
if check experiments is enabled, then we should ignore the default setting and only use the check experiments settings.
Else you'll pull in the lf prefix as well
| help="Current GitHub ref type, branch or tag", | ||
| ) | ||
| parser.add_argument( | ||
| "--check-experiments", |
There was a problem hiding this comment.
naming suggestion:
| "--check-experiments", | |
| "--eligible-experiments", |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| """ | ||
| prefix = rd.get_runner_prefix(settings_text, ["User2"], USER_BRANCH, {"otherExp"}) | ||
| self.assertEqual( | ||
| "otherExp.", prefix, "Runner prefix not correct for User2" | ||
| ) | ||
|
|
There was a problem hiding this comment.
| """ | |
| prefix = rd.get_runner_prefix(settings_text, ["User2"], USER_BRANCH, {"otherExp"}) | |
| self.assertEqual( | |
| "otherExp.", prefix, "Runner prefix not correct for User2" | |
| ) | |
| """ | |
| prefix = rd.get_runner_prefix( | |
| settings_text, ["User2"], USER_BRANCH, {"otherExp"} | |
| ) | |
| self.assertEqual("otherExp.", prefix, "Runner prefix not correct for User2") | |
| for experiment_name, experiment_settings in settings.experiments.items(): | ||
| enabled = False | ||
|
|
||
| if not experiment_settings.all_branches and is_exception_branch(branch): |
There was a problem hiding this comment.
| for experiment_name, experiment_settings in settings.experiments.items(): | |
| enabled = False | |
| if not experiment_settings.all_branches and is_exception_branch(branch): | |
| for experiment_name, experiment_settings in settings.experiments.items(): | |
| if not experiment_settings.all_branches and is_exception_branch(branch): |
jeanschmidt
changed the title
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
github-actions
bot
deleted the
jeanschmidt/runner_determinator_default
branch
4 participants
adds a
defaulttag to experiment configurations, allowing to remove some experiments by default on the random draw:and includes the configuration to filter what experiments are of interest for a particular workflow (comma separated):
The end goal, is to enable us to run multiple experiments, that are independent from one another. For example, while we still runs the LF infra experiment, we want to migrate other runners leveraging the current solution. A immediate UC is for the A100 instances, where we want to migrate to AWS.
Those new instances will during the migration period be labeled both
awsa100.linux.gcp.a100andlinux.aws.a100. Once the experiment ends, we will remove the first confusing one.