Change-Id: I2fe82be1a65ae45971d47078368f30d57097b648
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/660316
Reviewed-by: Chressie Himpel <chressie@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

When looking into presence behavior, I noticed that this function
is rather hard to follow. Specifically:

1. usesPresenceArray was only used in the default case,
   but re-iterated all conditions of the non-default cases.
   usesPresenceArray was a convoluted way to write fd.HasPresence().

2. hasLazyField was a misnomer: the function does not look at entire
   messages (which can have a lazy field), but at an individual field.

3. Only messages can be lazy right now, so the definition and usage
   of isLazy can be moved into the message case.

4. Instead of the wall-of-text comment above the function,
   I have moved the comments to each individual case.

I have verified Google-internally that my new usePresenceForField()
yields exactly the same results as the old one on all our .protos
(see Google-internal CL 753969456 for details).

Change-Id: Ida598fda7f472c71e18802aecabfae3e16de321c
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/670275
Reviewed-by: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

The presence bitmap decision logic is based on the field descriptor,
and in a follow-up CL I want to make both, the runtime and the code
generator, use the same code, so the internal/filedesc package
seems like a good location.

Change-Id: I0216249e00295b3194aaeceb7144ee7cff8dd7ef
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/670515
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>

With this change, the generator and runtime use the same logic
to decide whether a presence bitmap entry is necessary,
instead of two (possibly diverging) copies of the logic.

I have verified Google-internally that sharing the logic
yields exactly the same results as the old logic on all our .protos
(see Google-internal CL 755276656 for details).

Change-Id: I21ed069b7efbc8114b23331b441e35ce86e5d38a
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/670516
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>

* Ignore the //clang-format off directive for copyright header checks.

* Run the conformance tests with the protolegacy build tag,
  because (with protobuf v31) the conformance tests now include
  a test for message sets (which are a protolegacy feature).

Change-Id: I23e649f433e9342020493180f20b92c1720da049
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/673495
Reviewed-by: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

The presence type is always constructed with a non-nil pointer,
so this check can never trigger.

I was not able to measure a performance difference either way.
(See Google-internal CL 759581451 for test results.)

Change-Id: I73c43fbca91ef59c3f96b43f4687e68813e58fa2
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/674015
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>

Only lazy fields need to be set/read atomically, and the
fieldInfoForMessageListOpaqueNoPresence function is used for non-lazy fields.
(In fact, protoc currently does not allow [lazy = true] on repeated fields.)

I was not able to measure a performance difference either way.
(See Google-internal CL 759558617 for test results.)

Change-Id: Ie4c08a1345b62ecbe0f005d6cc97be35a8fe4daf
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/674055
Reviewed-by: Lasse Folger <lassefolger@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cassondra Foesch <cfoesch@gmail.com>

SizeVarint is of strategic importance for Protobuf encoding,
but I want to be clear: This change, on its own, does not
measurably improve real-world Protobuf usages in my testing.
It does, however, improve performance within the context of
another, larger project. I don’t want to sequence this optimization
on the bigger project, but would rather test and submit it in isolation.

As the detailed comment in the source code explains,
this implementation follows C++ Protobuf’s approach.

For your convenience, here is a godbolt Compiler Explorer link
that shows what the Go compiler makes of the old and new version:
https://godbolt.org/z/4erW1EY4r

When compiling with GOAMD64=v1 (the default), the new version
is roughly performance-neutral (a little faster on some, a little
slower on other CPU architectures — probably within the noise floor):

.fullname: SizeVarint-4
                     │    head     │                micro                |
                     │   sec/op    │    sec/op     vs base               |
conan-altra            2.174µ ± 0%    2.156µ ± 0%  -0.83% (p=0.000 n=10)
arcadia-rome           3.519µ ± 2%    3.558µ ± 0%       ~ (p=0.060 n=10)
indus-skylake          2.143µ ± 3%    2.192µ ± 7%       ~ (p=0.448 n=10)
izumi-sapphirerapids   974.9n ± 0%   1020.0n ± 0%  +4.63% (p=0.000 n=10)
geomean                1.999µ         2.035µ       +1.78%

By setting GOAMD64=v3, we unlock the full feature set of our CPUs.

If we build the old version with GOAMD64=v3, we already see a -50% speed-up
on AMD Zen 2 CPUs (due to switching from the slow BSRQ to the fast LZCNTQ):

.fullname: SizeVarint-4
                     │    head     │           head-goamd64v3            │
                     │   sec/op    │   sec/op     vs base                │
conan-altra            2.174µ ± 0%   2.174µ ± 0%        ~ (p=1.000 n=10)
arcadia-rome           3.519µ ± 2%   1.789µ ± 0%  -49.15% (p=0.000 n=10)
indus-skylake          2.143µ ± 3%   2.165µ ± 9%        ~ (p=0.739 n=10)
izumi-sapphirerapids   974.9n ± 0%   980.5n ± 3%   +0.58% (p=0.007 n=10)
geomean                1.999µ        1.695µ       -15.22%

And if we benchmark the new version with GOAMD64=v3, we see a further speed-up
on ARM and Intel — as high as 20% on Skylake!

.fullname: SizeVarint-4
                     │ head-goamd64v3 │           micro-goamd64v3           │
                     │     sec/op     │   sec/op     vs base                │
conan-altra               2.174µ ± 0%   2.156µ ± 0%   -0.83% (p=0.000 n=10)
arcadia-rome              1.789µ ± 0%   1.836µ ± 1%   +2.63% (p=0.000 n=10)
indus-skylake             2.165µ ± 9%   1.753µ ± 7%  -19.05% (p=0.000 n=10)
izumi-sapphirerapids      980.5n ± 3%   959.1n ± 0%   -2.19% (p=0.000 n=10)
geomean                   1.695µ        1.606µ        -5.25%

In summary, I believe this version of SizeVarint is currently the fastest
on the relevant CPUs, and leaves the path open to squeeze out a little more
performance by changing the Go compiler.

Change-Id: Ibc2629f8dcf9f2f4eb0a09fe37f923829ee3165b
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/683955
Reviewed-by: Nicolas Hillegeer <aktau@google.com>
Auto-Submit: Nicolas Hillegeer <aktau@google.com>
Reviewed-by: Christian Höppner <hoeppi@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Commit-Queue: Nicolas Hillegeer <aktau@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

Change-Id: I5eb27252a147c357f1392a7e1006b15d4f2de866
Reviewed-on: https://go-review.googlesource.com/c/protobuf/+/693515
Reviewed-by: Lasse Folger <lassefolger@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>