KEMBAR78
Generate .pyi files in py_proto_library (#10366) by dws · Pull Request #21567 · protocolbuffers/protobuf · GitHub
Skip to content

Conversation

@dws
Copy link
Contributor

@dws dws commented May 2, 2025

We here adjust py_proto_library to produce .pyi files along with the .py files it already generates. This achieves the same effect as the gRPC py_proto_library.

Per reviewer feedback, we here propagate the paths to the .pyi
files via the direct_pyi_files and transitive_pyi_files members
of PyInfo which are available starting in rules_python 1.1.0.

Since this means that the files are not automatically in the runfiles
of the py_proto_library, tooling to run mypy will need to be
adjusted to add what is needed from these provider fields to the
runfiles if that is what is wanted.

Closes #10366.

@jasonkuster
Copy link

+1; with grpc moving to using protobuf as the default for py_proto_library it regresses the creation of pyi files; this is the approach that repo takes and it would be very appreciated to get something like this merged.

@anandolee anandolee self-requested a review May 12, 2025 16:52
@anandolee anandolee added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label May 12, 2025
@github-actions github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label May 12, 2025
@dws dws marked this pull request as ready for review May 13, 2025 18:27
@dws
Copy link
Contributor Author

dws commented May 13, 2025

@anandolee Thanks for adding the tag! It looks like my PR passes CI, so I look forward to your review.

@mkruskal-google mkruskal-google added python bazel 🅰️ safe for tests Mark a commit as safe to run presubmits over labels May 19, 2025
@github-actions github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label May 19, 2025
@mkruskal-google mkruskal-google requested a review from haberman May 19, 2025 20:55
dws added a commit to dws/protobuf that referenced this pull request Jun 3, 2025
We here update the minimum required version of `rules_python` from
1.0.0 to 1.1.0 in order to enable using the new `direct_pyi_files`
and `transitive_pyi_files` members of `PyInfo` in protocolbuffers#21567.

Closes protocolbuffers#22078.
@dws dws force-pushed the py_proto_library-pyi branch 3 times, most recently from 3c5da60 to a4bbe43 Compare June 3, 2025 14:49
@mkruskal-google mkruskal-google added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Jun 9, 2025
@github-actions github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Jun 9, 2025
copybara-service bot pushed a commit that referenced this pull request Jun 9, 2025
We here update the minimum required version of `rules_python` from 1.0.0 to 1.1.0 in order to enable using the new `direct_pyi_files` and `transitive_pyi_files` members of `PyInfo` in #21567.

Closes #22078.

Closes #22079

COPYBARA_INTEGRATE_REVIEW=#22079 from dws:rules_python-1.1.0 7d0dc78
PiperOrigin-RevId: 769258504
@dws
Copy link
Contributor Author

dws commented Jun 9, 2025

@mkruskal-google Now that the rules_python bump has landed, please try another round of CI here.

@dws
Copy link
Contributor Author

dws commented Jun 16, 2025

@mkruskal-google ping?

1 similar comment
@dws
Copy link
Contributor Author

dws commented Jun 24, 2025

@mkruskal-google ping?

@dws
Copy link
Contributor Author

dws commented Jun 24, 2025

@anandolee perhaps you could enable a new round of CI testing? I think that's all that should be needed to land this now that it is approved.

@mkruskal-google mkruskal-google added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Jul 1, 2025
@github-actions github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Jul 1, 2025
dws added 2 commits July 1, 2025 11:31
We here adjust py_proto_library to produce `.pyi` files along with
the `.py` files it already generates.  This achieves the same effect
as the gRPC py_proto_library.

Closes protocolbuffers#10366.
Per reviewer feedback, we here propagate the paths to the `.pyi`
files via the `direct_pyi_files` and `transitive_pyi_files` members
of `PyInfo` which are available starting in `rules_python` 1.1.0.

Since this means that the files are not automatically in the runfiles
of the `py_proto_library`, tooling to run mypy will need to be
adjusted to add what is needed from these provider fields to the
runfiles if that is what is wanted.
@dws dws force-pushed the py_proto_library-pyi branch from a4bbe43 to 37fbdc0 Compare July 1, 2025 15:32
@mkruskal-google mkruskal-google added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Jul 1, 2025
@github-actions github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Jul 1, 2025
@copybara-service copybara-service bot closed this in e3e56c7 Jul 1, 2025
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Aug 16, 2025
⚠️ **CAUTION: this is a major update, indicating a breaking change!** ⚠️

This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [protoc](https://github.com/protocolbuffers/protobuf) | major | `31.1` -> `32.0` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>protocolbuffers/protobuf (protoc)</summary>

### [`v32.0`](https://github.com/protocolbuffers/protobuf/releases/tag/v32.0): Protocol Buffers v32.0

[Compare Source](protocolbuffers/protobuf@v31.1...v32.0)

### Announcements

- [Protobuf News](https://protobuf.dev/news/) may include additional announcements or pre-announcements for upcoming changes.

### Bazel

- Migrates some Java test protos using java\_features.proto to edition 2024 (protocolbuffers/protobuf@6f4a990)
- Add generated message reflection unittest for `import option` and `option_deps` test proto in edition 2024. (protocolbuffers/protobuf@c51df73)
- Replace the usage of cc\_info.debug\_context with \_debug\_context (protocolbuffers/protobuf@776c5f2)
- Generate .pyi files in py\_proto\_library ([#&#8203;10366](protocolbuffers/protobuf#10366)) ([#&#8203;21567](protocolbuffers/protobuf#21567)) (protocolbuffers/protobuf@e3e56c7)
- Introduce denylisted\_protos as a preferred replacement for blacklisted\_protos attribute. (protocolbuffers/protobuf@795ab9d)
- Use native.proto\_library for Bazel 7 and explicitly error for Bazel 6 which is incompatible and out of support. (protocolbuffers/protobuf@da0077e)
- Require ProtoInfo provider instead of proto\_library rule for bazel cc\_proto\_library deps. (protocolbuffers/protobuf@639f1c9)

### Compiler

- Strip extensions from option imports that are known in CollectExtensions due to polluted pool from protoc parse when used with protoc full + java built in generator. (protocolbuffers/protobuf@ef3f9ca)
- Deprecate `google.protobuf.Method.syntax` and `google.protobuf.Method.edition` (protocolbuffers/protobuf@9ec241e)
- Enable edition 2024 support in protoc and language generators for cpp, hpb, java, kotlin, objectivec, php, python, pyi, rust. (protocolbuffers/protobuf@26843b3)
- Ban `import weak` and `weak` field option in edition 2024 in protoc (parser and c++ runtime). (protocolbuffers/protobuf@a2a0511)

### C++

- Strip extensions from option imports that are known in CollectExtensions due to polluted pool from protoc parse when used with protoc full + java built in generator. (protocolbuffers/protobuf@ef3f9ca)
- Manually migrate top-level protobuf unittest protos from edition 2023 to edition 2024. (protocolbuffers/protobuf@7359f75)
- Add generated message reflection unittest for `import option` and `option_deps` test proto in edition 2024. (protocolbuffers/protobuf@c51df73)
- Remove the flag requirement for Bazel+MSVC users. (protocolbuffers/protobuf@c6ba697)
- Enable edition 2024 support in protoc and language generators for cpp, hpb, java, kotlin, objectivec, php, python, pyi, rust. (protocolbuffers/protobuf@26843b3)
- Cast GetCachedTID() to unsigned to prevent left-shifting a negative number. (protocolbuffers/protobuf@3e935cf)
- Update comment at the top of message.h and message\_lite.h to document that it should only be extended by gencode and not user code. (protocolbuffers/protobuf@6a6ba52)
- Treat warnings encountered in chromium builds as errors. (protocolbuffers/protobuf@21c3c53)
- Ban `import weak` and `weak` field option in edition 2024 in protoc (parser and c++ runtime). (protocolbuffers/protobuf@a2a0511)

### Java

- Strip extensions from option imports that are known in CollectExtensions due to polluted pool from protoc parse when used with protoc full + java built in generator. (protocolbuffers/protobuf@ef3f9ca)
- Remove CollectExtensions check banning unknown custom options since unknown custom options are now expected when using `import option` and `option_deps` which exclude the options from the "builder" pool (aka "import" pool). (protocolbuffers/protobuf@694eedd)
- Manually migrate top-level protobuf unittest protos from edition 2023 to edition 2024. (protocolbuffers/protobuf@7359f75)
- Migrates some Java test protos using java\_features.proto to edition 2024 (protocolbuffers/protobuf@6f4a990)
- Restore compatibility of runtime with pre-3.22.x gencode impacted by CVE-2022-3171 (protocolbuffers/protobuf@adf556f)
- Update our compiled\_edition\_default's maximum\_edition to 2024 ([#&#8203;22870](protocolbuffers/protobuf#22870)) (protocolbuffers/protobuf@6216ae4)
- Restore the long-deprecated static functions on TextFormat. (protocolbuffers/protobuf@5777d3b)
- Add Kotlin class name helpers to names.h (protocolbuffers/protobuf@527b2ab)
- Enable edition 2024 support in protoc and language generators for cpp, hpb, java, kotlin, objectivec, php, python, pyi, rust. (protocolbuffers/protobuf@26843b3)
- Update Java gencode base class documentation to clarify that it is not supported to extend them. (protocolbuffers/protobuf@c3177fb)
- Add a FieldMaskUtil#trim overload that accepts TrimOptions and allows retaining unset primitive field state. (protocolbuffers/protobuf@84c5b63)

### Csharp

- Update staleness (protocolbuffers/protobuf@0014173)
- Manually migrate top-level protobuf unittest protos from edition 2023 to edition 2024. (protocolbuffers/protobuf@7359f75)
- Update our compiled\_edition\_default's maximum\_edition to 2024 ([#&#8203;22870](protocolbuffers/protobuf#22870)) (protocolbuffers/protobuf@6216ae4)
- Enable edition 2024 support in protoc and language generators for cpp, hpb, java, kotlin, objectivec, php, python, pyi, rust. (protocolbuffers/protobuf@26843b3)
- Include also ARM build of protoc in nuget package ([#&#8203;21568](protocolbuffers/protobuf#21568)) (protocolbuffers/protobuf@c2b4040)
- Improve C# handling of Any JSON serialization/deserialization (protocolbuffers/protobuf@dc4e429)

### Objective-C

- Enable edition 2024 support in protoc and language generators for cpp, hpb, java, kotlin, objectivec, php, python, pyi, rust. (protocolbuffers/protobuf@26843b3)

### Rust

- Rust: remove `DEP_UPB_VERSION` check in codegen crate ([#&#8203;22764](protocolbuffers/protobuf#22764)) (protocolbuffers/protobuf@6f6012a)
- Enable edition 2024 support in protoc and language generators for cpp, hpb, java, kotlin, objectivec, php, python, pyi, rust. (protocolbuffers/protobuf@26843b3)
- Update Rust generator to output minidescriptors instead of C minitables (protocolbuffers/protobuf@1b4b5fc)
- Add \[clear\_and\_]parse\_dont\_enforce\_required() to Rust protobuf. (protocolbuffers/protobuf@e2fd7a0)
- Add #\[non\_exhaustive] on the oneof-case enum (protocolbuffers/protobuf@aba2075)
- Impl Clone for RepeatedIter (protocolbuffers/protobuf@aca1ed5)
- *See also UPB changes below, which may affect Rust.*

### Python

- Strip extensions from option imports that are known in CollectExtensions due to polluted pool from protoc parse when used with protoc full + java built in generator. (protocolbuffers/protobuf@ef3f9ca)
- Manually migrate top-level protobuf unittest protos from edition 2023 to edition 2024. (protocolbuffers/protobuf@7359f75)
- Raise warnings for float\_precision from python json\_format. ([#&#8203;23042](protocolbuffers/protobuf#23042)) (protocolbuffers/protobuf@1a7e012)
- Raise warnings when assign bool to int/enum ([#&#8203;23030](protocolbuffers/protobuf#23030)) (protocolbuffers/protobuf@0b25f7b)
- Update python edition default maximum edition to 2024 ([#&#8203;22889](protocolbuffers/protobuf#22889)) (protocolbuffers/protobuf@72abf95)
- Make protobuf codebase compatible with `--incompatible_config_setting_private_default_visibility` ([#&#8203;22745](protocolbuffers/protobuf#22745)) (protocolbuffers/protobuf@5e2838e)
- Add Python deprecation warnings for Descriptor Label. (protocolbuffers/protobuf@64a76e6)
- Updating setuptools to patch CVE-2025-47273 ([#&#8203;22094](protocolbuffers/protobuf#22094)) (protocolbuffers/protobuf@508aba1)
- Enable edition 2024 support in protoc and language generators for cpp, hpb, java, kotlin, objectivec, php, python, pyi, rust. (protocolbuffers/protobuf@26843b3)
- Extend gencode compatibility support back to 3.20.0 (protocolbuffers/protobuf@1af7fd4)
- Change proto\_api work with custom pool for upb and pure python. (protocolbuffers/protobuf@49d9e2d)
- Python pyi print "import datetime" for Duration/Timestamp field (protocolbuffers/protobuf@f59b84a)
- Add reference leak check to Python well\_known\_types\_test. Fix two refleak bugs. (protocolbuffers/protobuf@e5f9ab4)
- Add reference leak check to Python timestamp\_test and duration\_test. Fix a refleak bug when assign datetime/duration to Timestamp/Duration (protocolbuffers/protobuf@e004c8b)
- Add recursion depth limits to pure python (protocolbuffers/protobuf@17838be)
- Remove calls to Label within Protobuf Python. (protocolbuffers/protobuf@f69180a)
- Change Python .pyi stubs to generated \_Optional\[] for bool fields same with (protocolbuffers/protobuf@6972464)
- Remove internal/api\_implementation.Version() (protocolbuffers/protobuf@cbad456)
- Fix python proto\_api to work with messages that required fields are not set. (protocolbuffers/protobuf@f63ed55)
- Fix python upb crashes on map/repeated reference stub destructor (protocolbuffers/protobuf@4274909)
- Mark Py JSON float\_precision deprecated. (protocolbuffers/protobuf@727a8ed)

### PHP

- Enable edition 2024 support in protoc and language generators for cpp, hpb, java, kotlin, objectivec, php, python, pyi, rust. (protocolbuffers/protobuf@26843b3)
- Fix(php): use count instead of ->count() to avoid bug in c-extension (protocolbuffers/protobuf@9fbce62)
- \[php]\[compiler]\[ext] - Add missing has\* methods and other well known types ([#&#8203;20636](protocolbuffers/protobuf#20636)) (protocolbuffers/protobuf@01a7429)
- Fix PHP 8.4 deprecation in GPBDecodeException ([#&#8203;21232](protocolbuffers/protobuf#21232)) (protocolbuffers/protobuf@676666e)

##### PHP C-Extension

- Update our compiled\_edition\_default's maximum\_edition to 2024 ([#&#8203;22870](protocolbuffers/protobuf#22870)) (protocolbuffers/protobuf@6216ae4)
- Change 'getLabel is deprecated' to E\_USER\_DEPRECATED (protocolbuffers/protobuf@158d8ac)
- \[php]\[compiler]\[ext] - Add missing has\* methods and other well known types ([#&#8203;20636](protocolbuffers/protobuf#20636)) (protocolbuffers/protobuf@01a7429)
- *See also UPB changes below, which may affect PHP C-Extension.*

### Ruby

- Make protobuf codebase compatible with `--incompatible_config_setting_private_default_visibility` ([#&#8203;22745](protocolbuffers/protobuf#22745)) (protocolbuffers/protobuf@5e2838e)
- Enable edition 2024 support in protoc and language generators for cpp, hpb, java, kotlin, objectivec, php, python, pyi, rust. (protocolbuffers/protobuf@26843b3)
- Fix silent failure of `rb_test` rules to run test ([#&#8203;21733](protocolbuffers/protobuf#21733)) (protocolbuffers/protobuf@4f74764)

##### Ruby C-Extension

- Update our compiled\_edition\_default's maximum\_edition to 2024 ([#&#8203;22870](protocolbuffers/protobuf#22870)) (protocolbuffers/protobuf@6216ae4)
- Removed `wrap_memcpy.c` compatibility shim for pre-2.14 glibc (protocolbuffers/protobuf@4ba9733)
- *See also UPB changes below, which may affect Ruby C-Extension.*

### UPB (Python/PHP/Ruby C-Extension)

- Update our compiled\_edition\_default's maximum\_edition to 2024 ([#&#8203;22870](protocolbuffers/protobuf#22870)) (protocolbuffers/protobuf@6216ae4)
- Make protobuf codebase compatible with `--incompatible_config_setting_private_default_visibility` ([#&#8203;22745](protocolbuffers/protobuf#22745)) (protocolbuffers/protobuf@5e2838e)
- Add upb\_Message\_SetExtensionMessage as a helper for setting a extension field that is a message. This is the extension equivalent of upb\_Message\_SetBaseFieldMessage. (protocolbuffers/protobuf@f15d28a)

### Other

- Update github major version tags to be formatted v<MAJOR>.<MINOR>.<MICRO>-<language> for clarity (protocolbuffers/protobuf@e001d57)
- Upgrade Abseil dependency to [`2025051`](https://github.com/protocolbuffers/protobuf/commit/20250512).1 (protocolbuffers/protobuf@ee68ed1)
- Patch `rules_ruby` to apply `neverlink = True` to the `jars` rule ([#&#8203;21416](protocolbuffers/protobuf#21416)) (protocolbuffers/protobuf@88e11bd)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MS43My4yIiwidXBkYXRlZEluVmVyIjoiNDEuNzMuMiIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiUmVub3ZhdGUgQm90Il19-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support pyi_out in py_proto_library

4 participants