config: Add env_name_default and env_name_force to Config #138956

c00w · 2024-10-25T23:06:06Z

Stack from ghstack (oldest at bottom):

This allows Configs to handle setting their defaults (or overriding
themselves) via environment variables.

The environment variables are resolved at install time (which is usually
import time). This is done 1) to avoid any race conditions between
threads etc..., but 2) to help encourage people to just go modify the
configs directly, vs overriding environment variables to change
pytorch behaviour.

[ghstack-poisoned]

pytorch-bot · 2024-10-25T23:06:10Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138956

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 2ed5022 with merge base 5e4c8b6 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This allows Configs to handle setting their defaults (or overriding themselves) via environment variables. The environment variables are resolved at install time (which is usually import time). This is done 1) to avoid any race conditions between threads etc..., but 2) to help encourage people to just go modify the configs directly, vs overriding environment variables to change pytorch behaviour. ghstack-source-id: 75c4906 Pull Request resolved: #138956

[ghstack-poisoned]

This allows Configs to handle setting their defaults (or overriding themselves) via environment variables. The environment variables are resolved at install time (which is usually import time). This is done 1) to avoid any race conditions between threads etc..., but 2) to help encourage people to just go modify the configs directly, vs overriding environment variables to change pytorch behaviour. ghstack-source-id: 6011258 Pull Request resolved: #138956

[ghstack-poisoned]

This allows Configs to handle setting their defaults (or overriding themselves) via environment variables. The environment variables are resolved at install time (which is usually import time). This is done 1) to avoid any race conditions between threads etc..., but 2) to help encourage people to just go modify the configs directly, vs overriding environment variables to change pytorch behaviour. ghstack-source-id: c0a6d9a Pull Request resolved: #138956

[ghstack-poisoned]

This allows Configs to handle setting their defaults (or overriding themselves) via environment variables. The environment variables are resolved at install time (which is usually import time). This is done 1) to avoid any race conditions between threads etc..., but 2) to help encourage people to just go modify the configs directly, vs overriding environment variables to change pytorch behaviour. ghstack-source-id: c7881c9 Pull Request resolved: #138956

[ghstack-poisoned]

ezyang · 2024-10-31T02:30:49Z

torch/testing/_internal/fake_config_module.py

 e_jk_false = Config(justknob="does_not_exist", default=False)
+e_env_default = Config(env_name_default="ENV_TRUE", default=False)
+e_env_default_FALSE = Config(env_name_default="ENV_FALSE", default=True)
+e_env_force = Config(env_name_force="ENV_TRUE", default=False)


can we not name these something longer so there is no possibility of conflict lol

torch/utils/_config_module.py

ezyang · 2024-10-31T02:33:19Z

This is probably fine but the envvar parsing is technically BC breaking

[ghstack-poisoned]

ezyang · 2024-11-02T19:45:50Z

torch/utils/_config_module.py

    user_override: Any = _UNSET_SENTINEL
    # The justknob to check for this config
    justknob: Optional[str] = None
+    # environment variables are read at install time


BTW, this is a problem for future us, but handling TRITON_CACHE_DIR is going to be irritating because it is NOT read at install time and we regularly change the environ to make changes. It's better for the code we write to be patching envvar, but it may be more BC to continually read out of envvar.

ezyang

I'm going to approve this, but I'd feel much more comfortable if we're actually hooking this up to some config option, that will tell me if we /actually/ got the semantics right.

c00w · 2024-11-06T19:11:46Z

PLan is to stack a PR mograting some config options over (also need to chat with people who added the config options).

c00w · 2024-11-06T19:11:53Z

@pytorchbot merge

pytorchmergebot · 2024-11-06T19:13:39Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

This never ended up getting used, and instead we're doing this resolution within the configuration system. Removing these unused internal features. Pull Request resolved: #138767 Approved by: https://github.com/ezyang ghstack dependencies: #138766, #138956

…8956) This allows Configs to handle setting their defaults (or overriding themselves) via environment variables. The environment variables are resolved at install time (which is usually import time). This is done 1) to avoid any race conditions between threads etc..., but 2) to help encourage people to just go modify the configs directly, vs overriding environment variables to change pytorch behaviour. Pull Request resolved: pytorch#138956 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#138766

This never ended up getting used, and instead we're doing this resolution within the configuration system. Removing these unused internal features. Pull Request resolved: pytorch#138767 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#138766, pytorch#138956

…8956) This allows Configs to handle setting their defaults (or overriding themselves) via environment variables. The environment variables are resolved at install time (which is usually import time). This is done 1) to avoid any race conditions between threads etc..., but 2) to help encourage people to just go modify the configs directly, vs overriding environment variables to change pytorch behaviour. Pull Request resolved: pytorch#138956 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#138766

This never ended up getting used, and instead we're doing this resolution within the configuration system. Removing these unused internal features. Pull Request resolved: pytorch#138767 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#138766, pytorch#138956

…8956) This allows Configs to handle setting their defaults (or overriding themselves) via environment variables. The environment variables are resolved at install time (which is usually import time). This is done 1) to avoid any race conditions between threads etc..., but 2) to help encourage people to just go modify the configs directly, vs overriding environment variables to change pytorch behaviour. Pull Request resolved: pytorch#138956 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#138766

This never ended up getting used, and instead we're doing this resolution within the configuration system. Removing these unused internal features. Pull Request resolved: pytorch#138767 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#138766, pytorch#138956

…8956) This allows Configs to handle setting their defaults (or overriding themselves) via environment variables. The environment variables are resolved at install time (which is usually import time). This is done 1) to avoid any race conditions between threads etc..., but 2) to help encourage people to just go modify the configs directly, vs overriding environment variables to change pytorch behaviour. Pull Request resolved: pytorch#138956 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#138766

This never ended up getting used, and instead we're doing this resolution within the configuration system. Removing these unused internal features. Pull Request resolved: pytorch#138767 Approved by: https://github.com/ezyang ghstack dependencies: pytorch#138766, pytorch#138956

Update

0fd68e4

[ghstack-poisoned]

This was referenced Oct 25, 2024

config: Modify install_config_module to use a layered approach #138758

Closed

config: create Config objects with JK support #138766

Closed

justknobs: Remove JustKnobsConfig and justknobs_feature #138767

Closed

Update

132ae5d

[ghstack-poisoned]

Update

8767c27

[ghstack-poisoned]

Update

75ffb8a

[ghstack-poisoned]

Update

2071da8

[ghstack-poisoned]

c00w added the release notes: inductor label Oct 29, 2024

c00w requested review from ezyang and oulgen October 29, 2024 21:13

Update

899301d

[ghstack-poisoned]

ezyang reviewed Oct 31, 2024

View reviewed changes

torch/utils/_config_module.py Outdated Show resolved Hide resolved

ezyang reviewed Oct 31, 2024

View reviewed changes

torch/utils/_config_module.py Outdated Show resolved Hide resolved

ezyang reviewed Oct 31, 2024

View reviewed changes

torch/utils/_config_module.py Outdated Show resolved Hide resolved

c00w added 3 commits October 31, 2024 16:11

Update

1fe2b21

[ghstack-poisoned]

Update

fa3d866

[ghstack-poisoned]

Update

bb791e8

[ghstack-poisoned]

c00w requested a review from ezyang October 31, 2024 22:21

Update

2ed5022

[ghstack-poisoned]

c00w mentioned this pull request Nov 1, 2024

config: Throw if justknobs value is not a boolean #139488

Closed

ezyang reviewed Nov 2, 2024

View reviewed changes

ezyang approved these changes Nov 2, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 6, 2024

pytorchmergebot added the merging label Nov 6, 2024

pytorchmergebot added the Merged label Nov 6, 2024

pytorchmergebot closed this in 2a857e9 Nov 6, 2024

pytorchmergebot removed the merging label Nov 6, 2024

github-actions bot deleted the gh/c00w/4/head branch December 8, 2024 02:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

config: Add env_name_default and env_name_force to Config #138956

config: Add env_name_default and env_name_force to Config #138956

Uh oh!

c00w commented Oct 25, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Oct 25, 2024 •

edited

Loading

Uh oh!

ezyang Oct 31, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ezyang commented Oct 31, 2024

Uh oh!

ezyang Nov 2, 2024

Uh oh!

ezyang left a comment

Uh oh!

c00w commented Nov 6, 2024

Uh oh!

c00w commented Nov 6, 2024

Uh oh!

pytorchmergebot commented Nov 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

config: Add env_name_default and env_name_force to Config #138956

config: Add env_name_default and env_name_force to Config #138956

Uh oh!

Conversation

c00w commented Oct 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138956

✅ No Failures

Uh oh!

ezyang Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ezyang commented Oct 31, 2024

Uh oh!

ezyang Nov 2, 2024

Choose a reason for hiding this comment

Uh oh!

ezyang left a comment

Choose a reason for hiding this comment

Uh oh!

c00w commented Nov 6, 2024

Uh oh!

c00w commented Nov 6, 2024

Uh oh!

pytorchmergebot commented Nov 6, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

c00w commented Oct 25, 2024 •

edited

Loading

pytorch-bot bot commented Oct 25, 2024 •

edited

Loading