Update on "Set -DPy_LIMITED_API flag for py_limited_api=True extensions"

janeyx99 · janeyx99 · commit 70a52d44258d · 2025-01-28T09:06:11.000-08:00
This could be BC breaking, because there was a period of time when we use py_limited_api=True but don't enforce the flag, and now that we will start enforcing the flag, people's custom extensions may fail to build.

This is strictly still better behavior, as it is sketchy to claim CPython agnosticism without the flag, but calling this out as potential people yelling at us. Ways to mitigate this risk + reasons this may not be too big a deal:
- People haven't known about py_limited_api for extensions much due to lack of docs from python so usage is low right now
- My current tutorial is in store to make new users of py_limited_api pass this flag, so it'd be a noop for them.

Test plan:
* Locally i'm confident as I tried rebuilding ao with this change and it reliably failed (cuz importing torch/extension.h is a nono)
* Unit test wise, the normal python_agnostic one I added should work




[ghstack-poisoned]
diff --git a/torch/utils/cpp_extension.py b/torch/utils/cpp_extension.py
@@ -289,7 +289,7 @@ def _join_sycl_home(*paths) -> str:
     'win-amd64' : 'x86_amd64',
 }
 
-PY3_9_HEXCODE = "0x03090000"
+min_supported_cpython = "0x03090000"  # Python 3.9 hexcode
 
 def get_cxx_compiler():
     if IS_WINDOWS:
@@ -579,9 +579,9 @@ def build_extensions(self) -> None:
             if extension.py_limited_api:
                 # compile any extension that has passed in py_limited_api to the
                 # Extension constructor with the Py_LIMITED_API flag set to our
-                # min supported CPython version, here, 3.9.
+                # min supported CPython version.
                 # See https://docs.python.org/3/c-api/stable.html#c.Py_LIMITED_API
-                self._add_compile_flag(extension, f'-DPy_LIMITED_API={PY3_9_HEXCODE}')
+                self._add_compile_flag(extension, f'-DPy_LIMITED_API={min_supported_cpython}')
             else:
                 # pybind11 is not CPython API stable so don't add these flags used when
                 # compiling pybind11 when pybind11 is not even used. otherwise, the build
@@ -997,8 +997,10 @@ def CppExtension(name, sources, *args, **kwargs):
         Contrary to CPython setuptools, who does not define -DPy_LIMITED_API
         as a compile flag when py_limited_api is specified as an option for
         the "bdist_wheel" command in ``setup``, PyTorch does! We will specify
-        -DPy_LIMITED_API=<hexcode of the minimum supported Python version> to best
-        enforce consistency, safety, and sanity in order to encourage best practices.
+        -DPy_LIMITED_API=min_supported_cpython to best enforce consistency,
+        safety, and sanity in order to encourage best practices. To target a
+        different version, set min_supported_cpython to the hexcode of the
+        CPython version of choice.
 
     Example:
         >>> # xdoctest: +SKIP
@@ -1064,11 +1066,13 @@ def CUDAExtension(name, sources, *args, **kwargs):
         example, to give access to custom ops from python, the library should
         register the ops through the dispatcher.
 
-        Contray to CPython setuptools, who does not define -DPy_LIMITED_API
+        Contrary to CPython setuptools, who does not define -DPy_LIMITED_API
         as a compile flag when py_limited_api is specified as an option for
         the "bdist_wheel" command in ``setup``, PyTorch does! We will specify
-        -DPy_LIMITED_API=<hexcode of the minimum supported Python version> to best
-        enforce consistency, safety, and sanity in order to encourage best practices.
+        -DPy_LIMITED_API=min_supported_cpython to best enforce consistency,
+        safety, and sanity in order to encourage best practices. To target a
+        different version, set min_supported_cpython to the hexcode of the
+        CPython version of choice.
 
     Example:
         >>> # xdoctest: +SKIP
