When multiple requests use the same connection pool (same handler + same destination), they will hit a bunch of contention when acquiring and releasing the connection from the pool.

A quick example comparing the HttpClient benchmark using 1 vs 8 handlers, both with 256 worker tasks, shows a ~13% difference.

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --scenario httpclient-kestrel-get --profile aspnet-citrine-lin --variable useHttpMessageInvoker=true --variable concurrencyPerHttpClient=256 --variable numberOfHttpClients=1 --json 1x256.json
crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --scenario httpclient-kestrel-get --profile aspnet-citrine-lin --variable useHttpMessageInvoker=true --variable concurrencyPerHttpClient=32 --variable numberOfHttpClients=8 --json 8x32.json
crank compare .\1x256.json .\8x32.json

I saw similar numbers (a ~15% E2E difference) in my LLRP experiment when switching to using multiple handlers.

Approximate numbers from a benchmark doing minimal work outside of spamming the connection pool:

On my machine (8 core CPU) the Threads=6 scenario is spending 57 % of CPU on getting the lock in GetHttp11ConnectionAsync and ReturnHttp11Connection.

We should look into reducing the contention here if possible.