.NET 5 performance regression (~10%) from .NET Core 3.1 #36907
Description
Description
I have a .NET Standard 2.0 library implementing a high performance interpreter for the brainf*ck language. At the end of the day it is essentially just a Turing machine, so all the operations being performed are basically accesses to memory, incrementing values, looking up tables, etc. It's all "self-contained", with no external APIs being invoked, IO operations or anything.
I've run some benchmarks on .NET Core 2.1, .NET Core 3.1 and .NET 5 (Preview 4) and noticed that .NET 5 is consistently slower than .NET Core 3.1 in almost all cases. Especially in the mandlebrot test case, which is the most intensive, .NET 5 is almost one second slower, so about 10% slower.
I find this very surprising, as .NET 5 is supposed to be much more optimized than .NET Core 3.1.
NOTE: the solution is not open source for now, but I'd be more than happy to give access to the repo to anyone in the team wanting to take a look, just ping me and I'll send you the invite.
Pinging @EgorBo as you seemed interested in this when I shared some older benchmarks about this on Twitter (here) back when .NET 5 preview 1 had just been released, and cc. @tannergooding.
Configuration
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19619
Intel Core i7-8750H CPU 2.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100-preview.4.20258.7
[Host] : .NET Core 5.0.0 (CoreCLR 5.0.20.25106, CoreFX 5.0.20.25106), X64 RyuJIT
.NET Core 3.1 : .NET Core 3.1.4 (CoreCLR 4.700.20.20201, CoreFX 4.700.20.22101), X64 RyuJIT
.NET Core 5.0 : .NET Core 5.0.0 (CoreCLR 5.0.20.25106, CoreFX 5.0.20.25106), X64 RyuJIT
Regression?
This is a regression in speed from .NET Core 3.1 to .NET 5 Preview 4, on the same machine.
Data
| Runtime | Name | Mean | Error | StdDev |
|---|---|---|---|---|
| .NET Core 2.1 | Division | 56.82 us | 0.810 us | 0.758 us |
| .NET Core 3.1 | Division | 41.06 us | 0.588 us | 0.521 us |
| .NET Core 5.0 | Division | 42.95 us | 0.511 us | 0.453 us |
| .NET Core 2.1 | Fibonacci | 21,298.13 us | 398.437 us | 372.698 us |
| .NET Core 3.1 | Fibonacci | 13,197.50 us | 65.613 us | 58.164 us |
| .NET Core 5.0 | Fibonacci | 15,302.14 us | 83.934 us | 78.512 us |
| .NET Core 2.1 | HelloWorld | 17.61 us | 0.296 us | 0.341 us |
| .NET Core 3.1 | HelloWorld | 16.09 us | 0.311 us | 0.394 us |
| .NET Core 5.0 | HelloWorld | 15.68 us | 0.305 us | 0.510 us |
| .NET Core 2.1 | Mandelbrot | 10,221,974.39 us | 131,372.509 us | 116,458.307 us |
| .NET Core 3.1 | Mandelbrot | 7,799,742.74 us | 65,822.677 us | 58,350.088 us |
| .NET Core 5.0 | Mandelbrot | 8,544,272.25 us | 8,136.804 us | 7,611.172 us |
| .NET Core 2.1 | Multiply | 1,204.57 us | 14.566 us | 12.912 us |
| .NET Core 3.1 | Multiply | 792.31 us | 5.899 us | 4.926 us |
| .NET Core 5.0 | Multiply | 790.11 us | 1.301 us | 1.087 us |
| .NET Core 2.1 | Sum | 52.86 us | 0.590 us | 0.493 us |
| .NET Core 3.1 | Sum | 39.53 us | 0.620 us | 0.580 us |
| .NET Core 5.0 | Sum | 37.39 us | 0.470 us | 0.417 us |
category:cq
theme:jit-block-layout
skill-level:expert
cost:large